2024-06-04T04:07:55.496 INFO:root:teuthology version: 0.0.1.dev287+gb9e3da8 2024-06-04T04:07:55.496 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} --archive /home/teuthworker/archive/teuthology-2024-05-22_20:24:16-fs-main-distro-default-smithi/7721102 --verbose --owner scheduled_teuthology@teuthology --name teuthology-2024-05-22_20:24:16-fs-main-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-05-22_20:24:16-fs-main-distro-default-smithi/7721102/orig.config.yaml 2024-06-04T04:07:55.553 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-04T04:07:55.644 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-05-22_20:24:16-fs-main-distro-default-smithi/7721102 branch: main description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7721102' kernel: kdb: 1 sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: teuthology-2024-05-22_20:24:16-fs-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request sha1: b7328fed0e2e9ab66b09f8134f3cebfc71214504 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: b7328fed0e2e9ab66b09f8134f3cebfc71214504 kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 7aec00165b4b073a25396fd0279151d2b2fa23fe owner: scheduled_teuthology@teuthology priority: 950 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 6827 sha1: b7328fed0e2e9ab66b09f8134f3cebfc71214504 sleep_before_teardown: 0 subset: 382/512 suite: fs suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_7aec00165b4b073a25396fd0279151d2b2fa23fe/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7aec00165b4b073a25396fd0279151d2b2fa23fe syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi070.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH/PJhzZBxtfxs4iWEbiGnyHsf75ILWb3HRAUlO/C5WXzIL3nYFtlR/X5wyMYSN/hbJyzReY+QO43KHqyPp1PUc= smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKd22MHUAOy4PXi6MFxl2cuFHrVI4RjJeuddDZCnNHXQqK44DN6WUwlPYdRfw7UDVZ1zBytLMAXtgAnnk0GgIoo= smithi112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNwJqsirxVJVBTkKQYf3i+9qehJwpfvlYnWrQX/VjOPZ/TjJYL/NKjKA50jdOfQo8ftLpT6Lh8PBTxaSklPLmkI= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph config set mgr mgr/snap_schedule/log_level debug - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a timestamp: 2024-05-22_20:24:16 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-06-04T04:07:55.644 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7aec00165b4b073a25396fd0279151d2b2fa23fe/qa; will attempt to use it 2024-06-04T04:07:55.645 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7aec00165b4b073a25396fd0279151d2b2fa23fe/qa/tasks 2024-06-04T04:07:55.645 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-04T04:07:55.647 INFO:teuthology.task.internal:Checking packages... 2024-06-04T04:07:55.667 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b7328fed0e2e9ab66b09f8134f3cebfc71214504' 2024-06-04T04:07:55.667 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-04T04:07:55.667 INFO:teuthology.packaging:ref: None 2024-06-04T04:07:55.668 INFO:teuthology.packaging:tag: None 2024-06-04T04:07:55.668 INFO:teuthology.packaging:branch: main 2024-06-04T04:07:55.668 INFO:teuthology.packaging:sha1: b7328fed0e2e9ab66b09f8134f3cebfc71214504 2024-06-04T04:07:55.668 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-06-04T04:07:55.910 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-4027.g65533a90 2024-06-04T04:07:55.911 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-04T04:07:55.918 INFO:teuthology.task.internal:no buildpackages task found 2024-06-04T04:07:55.918 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-04T04:07:55.941 INFO:teuthology.task.internal:Saving configuration 2024-06-04T04:07:55.962 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-04T04:07:55.970 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-04T04:07:55.998 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-22_20:24:16-fs-main-distro-default-smithi/7721102', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-04 04:03:04.889842', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH/PJhzZBxtfxs4iWEbiGnyHsf75ILWb3HRAUlO/C5WXzIL3nYFtlR/X5wyMYSN/hbJyzReY+QO43KHqyPp1PUc='} 2024-06-04T04:07:56.021 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-22_20:24:16-fs-main-distro-default-smithi/7721102', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-04 04:03:04.887881', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKd22MHUAOy4PXi6MFxl2cuFHrVI4RjJeuddDZCnNHXQqK44DN6WUwlPYdRfw7UDVZ1zBytLMAXtgAnnk0GgIoo='} 2024-06-04T04:07:56.044 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-22_20:24:16-fs-main-distro-default-smithi/7721102', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-04 04:03:04.891290', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNwJqsirxVJVBTkKQYf3i+9qehJwpfvlYnWrQX/VjOPZ/TjJYL/NKjKA50jdOfQo8ftLpT6Lh8PBTxaSklPLmkI='} 2024-06-04T04:07:56.044 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-04T04:07:56.050 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-06-04T04:07:56.051 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-06-04T04:07:56.051 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-06-04T04:07:56.051 INFO:teuthology.run_tasks:Running task console_log... 2024-06-04T04:07:56.138 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4192b27be0>, signals=[15]) 2024-06-04T04:07:56.138 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-04T04:07:56.149 INFO:teuthology.task.internal:Opening connections... 2024-06-04T04:07:56.149 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2024-06-04T04:07:56.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-04T04:07:56.226 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-06-04T04:07:56.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-04T04:07:56.300 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2024-06-04T04:07:56.301 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-04T04:07:56.373 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-04T04:07:56.385 DEBUG:teuthology.orchestra.run.smithi070:> uname -m 2024-06-04T04:07:56.402 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2024-06-04T04:07:56.403 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2024-06-04T04:07:56.460 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="CentOS Stream" 2024-06-04T04:07:56.460 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="9" 2024-06-04T04:07:56.460 INFO:teuthology.orchestra.run.smithi070.stdout:ID="centos" 2024-06-04T04:07:56.460 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="rhel fedora" 2024-06-04T04:07:56.460 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="9" 2024-06-04T04:07:56.460 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el9" 2024-06-04T04:07:56.460 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-04T04:07:56.460 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2024-06-04T04:07:56.460 INFO:teuthology.orchestra.run.smithi070.stdout:LOGO="fedora-logo-icon" 2024-06-04T04:07:56.460 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-04T04:07:56.460 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://centos.org/" 2024-06-04T04:07:56.460 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-04T04:07:56.460 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-04T04:07:56.461 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-04T04:07:56.461 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2024-06-04T04:07:56.479 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-06-04T04:07:56.496 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-06-04T04:07:56.497 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-06-04T04:07:56.553 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-06-04T04:07:56.554 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-06-04T04:07:56.554 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-06-04T04:07:56.554 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-06-04T04:07:56.554 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-06-04T04:07:56.554 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-06-04T04:07:56.554 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-04T04:07:56.554 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-06-04T04:07:56.554 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-06-04T04:07:56.554 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-04T04:07:56.554 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-06-04T04:07:56.554 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-04T04:07:56.555 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-04T04:07:56.555 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-04T04:07:56.555 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-06-04T04:07:56.578 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2024-06-04T04:07:56.596 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2024-06-04T04:07:56.596 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2024-06-04T04:07:56.652 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="CentOS Stream" 2024-06-04T04:07:56.652 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="9" 2024-06-04T04:07:56.653 INFO:teuthology.orchestra.run.smithi112.stdout:ID="centos" 2024-06-04T04:07:56.653 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE="rhel fedora" 2024-06-04T04:07:56.653 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="9" 2024-06-04T04:07:56.653 INFO:teuthology.orchestra.run.smithi112.stdout:PLATFORM_ID="platform:el9" 2024-06-04T04:07:56.653 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-04T04:07:56.653 INFO:teuthology.orchestra.run.smithi112.stdout:ANSI_COLOR="0;31" 2024-06-04T04:07:56.653 INFO:teuthology.orchestra.run.smithi112.stdout:LOGO="fedora-logo-icon" 2024-06-04T04:07:56.653 INFO:teuthology.orchestra.run.smithi112.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-04T04:07:56.653 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://centos.org/" 2024-06-04T04:07:56.653 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-04T04:07:56.653 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-04T04:07:56.653 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-04T04:07:56.654 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2024-06-04T04:07:56.679 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-04T04:07:56.691 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-04T04:07:56.702 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-04T04:07:56.702 DEBUG:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2024-06-04T04:07:56.704 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-06-04T04:07:56.706 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2024-06-04T04:07:56.722 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-04T04:07:56.733 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-04T04:07:56.733 DEBUG:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2024-06-04T04:07:56.761 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-06-04T04:07:56.763 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2024-06-04T04:07:56.794 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-04T04:07:56.873 INFO:teuthology.run_tasks:Running task kernel... 2024-06-04T04:07:56.892 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-04T04:07:56.892 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-04T04:07:56.892 DEBUG:teuthology.orchestra.run.smithi070:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-04T04:07:56.894 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-04T04:07:56.894 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-04T04:07:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-04T04:07:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-04T04:07:56.911 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-06-04T04:07:56.911 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-06-04T04:07:56.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-04T04:07:56.912 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-06-04T04:07:56.967 INFO:teuthology.orchestra.run.smithi112.stdout:5.14.0-437.el9.x86_64 2024-06-04T04:07:56.967 INFO:teuthology.task.kernel:Running kernel on smithi112: 5.14.0-437.el9.x86_64 2024-06-04T04:07:56.967 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum install -y kernel 2024-06-04T04:07:56.968 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-437.el9.x86_64 2024-06-04T04:07:56.968 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-437.el9.x86_64 2024-06-04T04:07:56.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-06-04T04:07:56.970 INFO:teuthology.orchestra.run.smithi070.stdout:5.14.0-437.el9.x86_64 2024-06-04T04:07:56.970 INFO:teuthology.task.kernel:Running kernel on smithi070: 5.14.0-437.el9.x86_64 2024-06-04T04:07:56.970 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum install -y kernel 2024-06-04T04:12:18.808 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - BaseOS 32 kB/s | 8.1 MB 04:20 2024-06-04T04:12:21.586 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 31 kB/s | 8.1 MB 04:23 2024-06-04T04:12:21.987 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 31 kB/s | 8.1 MB 04:23 2024-06-04T04:16:41.419 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - AppStream 75 kB/s | 19 MB 04:21 2024-06-04T04:16:43.981 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 75 kB/s | 19 MB 04:21 2024-06-04T04:16:44.457 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 75 kB/s | 19 MB 04:21 2024-06-04T04:17:59.029 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - CRB 85 kB/s | 6.2 MB 01:14 2024-06-04T04:18:05.583 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 82 kB/s | 6.2 MB 01:17 2024-06-04T04:18:14.083 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 2.3 kB/s | 17 kB 00:07 2024-06-04T04:18:39.133 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - Extras packages 436 B/s | 17 kB 00:38 2024-06-04T04:19:03.173 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 47 kB/s | 6.2 MB 02:15 2024-06-04T04:19:11.373 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 2.4 kB/s | 17 kB 00:06 2024-06-04T04:19:16.505 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 584 kB/s | 21 MB 00:37 2024-06-04T04:19:21.785 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-06-04T04:19:23.791 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-04T04:19:23.791 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-04T04:19:23.891 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-06-04T04:19:23.896 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-06-04T04:19:23.896 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-06-04T04:19:23.897 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-06-04T04:19:23.897 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-06-04T04:19:23.897 INFO:teuthology.orchestra.run.smithi112.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-04T04:19:23.897 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-06-04T04:19:23.897 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-04T04:19:23.897 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-04T04:19:23.897 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-04T04:19:23.897 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:19:23.897 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-06-04T04:19:23.897 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-06-04T04:19:23.897 INFO:teuthology.orchestra.run.smithi112.stdout:Install 4 Packages 2024-06-04T04:19:23.897 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:19:23.898 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 80 M 2024-06-04T04:19:23.898 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 124 M 2024-06-04T04:19:23.898 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-06-04T04:19:27.323 INFO:teuthology.orchestra.run.smithi112.stdout:[MIRROR] kernel-core-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:19:27.366 INFO:teuthology.orchestra.run.smithi112.stdout:[MIRROR] kernel-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:19:27.679 INFO:teuthology.orchestra.run.smithi112.stdout:[MIRROR] kernel-modules-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:19:28.161 INFO:teuthology.orchestra.run.smithi112.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 217 kB/s | 893 kB 00:04 2024-06-04T04:19:29.478 INFO:teuthology.orchestra.run.smithi112.stdout:(2/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 6.4 MB/s | 35 MB 00:05 2024-06-04T04:19:30.495 INFO:teuthology.orchestra.run.smithi112.stdout:(3/4): kernel-modules-core-5.14.0-452.el9.x86_6 12 MB/s | 29 MB 00:02 2024-06-04T04:19:30.986 INFO:teuthology.orchestra.run.smithi112.stdout:(4/4): kernel-core-5.14.0-452.el9.x86_64.rpm 2.3 MB/s | 16 MB 00:06 2024-06-04T04:19:30.991 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-06-04T04:19:30.991 INFO:teuthology.orchestra.run.smithi112.stdout:Total 11 MB/s | 80 MB 00:07 2024-06-04T04:19:31.682 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-06-04T04:19:32.044 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-06-04T04:19:32.044 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-06-04T04:19:32.991 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-06-04T04:19:32.991 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-06-04T04:19:35.399 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-06-04T04:19:36.077 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-04T04:19:36.156 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-04T04:19:38.784 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-04T04:19:38.933 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-04T04:19:43.622 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-04T04:19:43.672 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:19:45.323 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 643 kB/s | 21 MB 00:33 2024-06-04T04:19:48.404 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:19:50.402 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-06-04T04:19:51.264 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 224 kB/s | 21 MB 01:37 2024-06-04T04:19:52.408 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-04T04:19:52.408 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-04T04:19:52.507 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-04T04:19:52.517 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-04T04:19:52.517 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-06-04T04:19:52.517 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-04T04:19:52.518 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-04T04:19:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-04T04:19:52.518 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-06-04T04:19:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-04T04:19:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-04T04:19:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-04T04:19:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:19:52.518 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-04T04:19:52.518 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-04T04:19:52.518 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-06-04T04:19:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:19:52.519 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 80 M 2024-06-04T04:19:52.519 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 124 M 2024-06-04T04:19:52.519 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-04T04:19:56.394 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-06-04T04:19:58.358 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-04T04:19:58.358 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-04T04:19:58.458 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-06-04T04:19:58.471 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-06-04T04:19:58.471 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-06-04T04:19:58.471 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-06-04T04:19:58.471 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-06-04T04:19:58.471 INFO:teuthology.orchestra.run.smithi070.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-04T04:19:58.471 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-06-04T04:19:58.471 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-04T04:19:58.471 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-04T04:19:58.471 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-04T04:19:58.471 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:19:58.471 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-06-04T04:19:58.472 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-06-04T04:19:58.472 INFO:teuthology.orchestra.run.smithi070.stdout:Install 4 Packages 2024-06-04T04:19:58.472 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:19:58.472 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 80 M 2024-06-04T04:19:58.472 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 124 M 2024-06-04T04:19:58.472 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-06-04T04:20:00.620 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:20:00.649 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:20:00.906 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:20:01.911 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 99 kB/s | 893 kB 00:09 2024-06-04T04:20:02.186 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 1.7 MB/s | 16 MB 00:09 2024-06-04T04:20:02.986 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] kernel-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:20:02.986 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] kernel-core-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:20:02.988 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] kernel-modules-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:20:03.062 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 3.4 MB/s | 35 MB 00:10 2024-06-04T04:20:03.311 INFO:teuthology.orchestra.run.smithi070.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 196 kB/s | 893 kB 00:04 2024-06-04T04:20:03.787 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): kernel-modules-core-5.14.0-452.el9.x86_6 15 MB/s | 29 MB 00:01 2024-06-04T04:20:03.790 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-04T04:20:03.790 INFO:teuthology.orchestra.run.smithi098.stdout:Total 7.1 MB/s | 80 MB 00:11 2024-06-04T04:20:04.487 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-04T04:20:04.824 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-04T04:20:04.824 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-04T04:20:05.788 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-04T04:20:05.789 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-04T04:20:05.969 INFO:teuthology.orchestra.run.smithi070.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 2.2 MB/s | 16 MB 00:07 2024-06-04T04:20:06.569 INFO:teuthology.orchestra.run.smithi070.stdout:(3/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 4.4 MB/s | 35 MB 00:07 2024-06-04T04:20:06.836 INFO:teuthology.orchestra.run.smithi070.stdout:(4/4): kernel-modules-core-5.14.0-452.el9.x86_6 8.2 MB/s | 29 MB 00:03 2024-06-04T04:20:06.838 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-06-04T04:20:06.838 INFO:teuthology.orchestra.run.smithi070.stdout:Total 9.6 MB/s | 80 MB 00:08 2024-06-04T04:20:07.529 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-06-04T04:20:07.881 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-06-04T04:20:07.881 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-06-04T04:20:08.193 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-04T04:20:08.831 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-06-04T04:20:08.832 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-06-04T04:20:08.844 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-04T04:20:08.942 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-04T04:20:11.251 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-06-04T04:20:11.580 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-04T04:20:11.736 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-04T04:20:11.940 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-04T04:20:12.037 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-04T04:20:14.695 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-04T04:20:14.847 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-04T04:20:16.447 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-04T04:20:16.508 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:20:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:20:17.736 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:20:18.022 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:20:18.022 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-04T04:20:18.022 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-04T04:20:18.022 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-04T04:20:18.770 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:20:18.770 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:20:18.770 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-06-04T04:20:18.770 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-04T04:20:18.770 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-04T04:20:18.770 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-04T04:20:18.771 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-04T04:20:18.771 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:20:18.771 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-06-04T04:20:19.148 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2024-06-04T04:20:19.231 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-04T04:20:19.234 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-04T04:20:19.234 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-04T04:20:19.235 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-04T04:20:19.235 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-04T04:20:19.235 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-06-04T04:20:19.235 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi112.front.sepia.ceph.com, path=None, version=distro) 2024-06-04T04:20:19.235 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2024-06-04T04:20:19.287 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-04T04:20:19.288 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-04T04:20:19.288 DEBUG:teuthology.orchestra.run.smithi112:> sudo rpm -qi grub2-tools 2024-06-04T04:20:19.318 INFO:teuthology.orchestra.run.smithi112.stdout:Name : grub2-tools 2024-06-04T04:20:19.318 INFO:teuthology.orchestra.run.smithi112.stdout:Epoch : 1 2024-06-04T04:20:19.318 INFO:teuthology.orchestra.run.smithi112.stdout:Version : 2.06 2024-06-04T04:20:19.318 INFO:teuthology.orchestra.run.smithi112.stdout:Release : 68.el9 2024-06-04T04:20:19.318 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: x86_64 2024-06-04T04:20:19.318 INFO:teuthology.orchestra.run.smithi112.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-04T04:20:19.318 INFO:teuthology.orchestra.run.smithi112.stdout:Group : Unspecified 2024-06-04T04:20:19.318 INFO:teuthology.orchestra.run.smithi112.stdout:Size : 8264254 2024-06-04T04:20:19.318 INFO:teuthology.orchestra.run.smithi112.stdout:License : GPLv3+ 2024-06-04T04:20:19.318 INFO:teuthology.orchestra.run.smithi112.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-04T04:20:19.318 INFO:teuthology.orchestra.run.smithi112.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-04T04:20:19.318 INFO:teuthology.orchestra.run.smithi112.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-04T04:20:19.319 INFO:teuthology.orchestra.run.smithi112.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-04T04:20:19.319 INFO:teuthology.orchestra.run.smithi112.stdout:Packager : builder@centos.org 2024-06-04T04:20:19.319 INFO:teuthology.orchestra.run.smithi112.stdout:Vendor : CentOS 2024-06-04T04:20:19.319 INFO:teuthology.orchestra.run.smithi112.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-04T04:20:19.319 INFO:teuthology.orchestra.run.smithi112.stdout:Summary : Support tools for GRUB. 2024-06-04T04:20:19.319 INFO:teuthology.orchestra.run.smithi112.stdout:Description : 2024-06-04T04:20:19.319 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:20:19.319 INFO:teuthology.orchestra.run.smithi112.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-04T04:20:19.319 INFO:teuthology.orchestra.run.smithi112.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-04T04:20:19.319 INFO:teuthology.orchestra.run.smithi112.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-04T04:20:19.319 INFO:teuthology.orchestra.run.smithi112.stdout:hardware devices. 2024-06-04T04:20:19.319 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:20:19.319 INFO:teuthology.orchestra.run.smithi112.stdout:This subpackage provides tools for support of all platforms. 2024-06-04T04:20:19.320 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-04T04:20:19.320 INFO:teuthology.task.kernel:Updating grub on smithi112 to boot 5.14.0-452.el9.x86_64 2024-06-04T04:20:19.320 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-04T04:20:19.568 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-04T04:20:19.685 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:20:19.920 INFO:teuthology.orchestra.run.smithi112.stderr:Generating grub configuration file ... 2024-06-04T04:20:21.247 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:20:21.353 INFO:teuthology.orchestra.run.smithi112.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-04T04:20:21.382 INFO:teuthology.orchestra.run.smithi112.stderr:done 2024-06-04T04:20:21.384 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-06-04T04:20:21.401 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.DU7WO6doiC 2024-06-04T04:20:21.402 DEBUG:teuthology.orchestra.run.smithi112:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.DU7WO6doiC 2024-06-04T04:20:21.467 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.DU7WO6doiC 2024-06-04T04:20:21.601 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.DU7WO6doiC is 6KB 2024-06-04T04:20:21.650 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.DU7WO6doiC 2024-06-04T04:20:21.664 DEBUG:teuthology.orchestra.run.smithi112:> sudo /bin/ls /boot/loader/entries || true 2024-06-04T04:20:21.735 INFO:teuthology.orchestra.run.smithi112.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-04T04:20:21.735 INFO:teuthology.orchestra.run.smithi112.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-04T04:20:21.735 INFO:teuthology.orchestra.run.smithi112.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-04T04:20:21.735 INFO:teuthology.orchestra.run.smithi112.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-04T04:20:21.736 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-04T04:20:21.882 DEBUG:teuthology.orchestra.run.smithi112:> sudo shutdown -r now 2024-06-04T04:20:24.391 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:20:50.807 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:20:50.847 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:20:51.201 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:20:51.201 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-04T04:20:51.201 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-04T04:20:51.201 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-04T04:20:51.885 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-06-04T04:20:51.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-06-04T04:20:51.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-04T04:20:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:20:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:20:52.001 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-04T04:20:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-04T04:20:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-04T04:20:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-04T04:20:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-04T04:20:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:20:52.001 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-04T04:20:52.371 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-06-04T04:20:52.448 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-04T04:20:52.448 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-04T04:20:52.448 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-04T04:20:52.449 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-04T04:20:52.449 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-04T04:20:52.449 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-06-04T04:20:52.449 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-06-04T04:20:52.449 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-06-04T04:20:52.501 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-04T04:20:52.502 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-04T04:20:52.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2024-06-04T04:20:52.542 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2024-06-04T04:20:52.542 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2024-06-04T04:20:52.542 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.06 2024-06-04T04:20:52.542 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 68.el9 2024-06-04T04:20:52.542 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2024-06-04T04:20:52.542 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-04T04:20:52.542 INFO:teuthology.orchestra.run.smithi098.stdout:Group : Unspecified 2024-06-04T04:20:52.542 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 8264254 2024-06-04T04:20:52.542 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2024-06-04T04:20:52.542 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-04T04:20:52.542 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-04T04:20:52.543 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-04T04:20:52.543 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-04T04:20:52.543 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : builder@centos.org 2024-06-04T04:20:52.543 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2024-06-04T04:20:52.543 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-04T04:20:52.543 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2024-06-04T04:20:52.543 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2024-06-04T04:20:52.543 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:20:52.543 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-04T04:20:52.543 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-04T04:20:52.543 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-04T04:20:52.543 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2024-06-04T04:20:52.543 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:20:52.543 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2024-06-04T04:20:52.544 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-04T04:20:52.544 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 5.14.0-452.el9.x86_64 2024-06-04T04:20:52.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-04T04:20:53.138 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-06-04T04:20:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:20:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:20:54.014 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:20:54.015 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-04T04:20:54.015 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-04T04:20:54.015 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-04T04:20:54.560 INFO:teuthology.orchestra.run.smithi098.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-04T04:20:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-06-04T04:20:54.595 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-06-04T04:20:54.610 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.KI42xjWOUU 2024-06-04T04:20:54.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KI42xjWOUU 2024-06-04T04:20:54.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.KI42xjWOUU 2024-06-04T04:20:54.813 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.KI42xjWOUU is 6KB 2024-06-04T04:20:54.833 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-04T04:20:54.833 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:20:54.833 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-06-04T04:20:54.833 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-04T04:20:54.833 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-04T04:20:54.834 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-04T04:20:54.834 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-04T04:20:54.834 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:20:54.834 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-06-04T04:20:54.863 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.KI42xjWOUU 2024-06-04T04:20:54.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2024-06-04T04:20:54.938 INFO:teuthology.orchestra.run.smithi098.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-04T04:20:54.938 INFO:teuthology.orchestra.run.smithi098.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-04T04:20:54.938 INFO:teuthology.orchestra.run.smithi098.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-04T04:20:54.938 INFO:teuthology.orchestra.run.smithi098.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-04T04:20:54.940 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-04T04:20:55.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-06-04T04:20:55.218 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-06-04T04:20:55.289 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-04T04:20:55.290 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-04T04:20:55.290 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-04T04:20:55.290 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-04T04:20:55.290 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-04T04:20:55.290 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-06-04T04:20:55.291 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi070.front.sepia.ceph.com, path=None, version=distro) 2024-06-04T04:20:55.291 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-06-04T04:20:55.342 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-04T04:20:55.343 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-04T04:20:55.343 DEBUG:teuthology.orchestra.run.smithi070:> sudo rpm -qi grub2-tools 2024-06-04T04:20:55.373 INFO:teuthology.orchestra.run.smithi070.stdout:Name : grub2-tools 2024-06-04T04:20:55.373 INFO:teuthology.orchestra.run.smithi070.stdout:Epoch : 1 2024-06-04T04:20:55.373 INFO:teuthology.orchestra.run.smithi070.stdout:Version : 2.06 2024-06-04T04:20:55.373 INFO:teuthology.orchestra.run.smithi070.stdout:Release : 68.el9 2024-06-04T04:20:55.373 INFO:teuthology.orchestra.run.smithi070.stdout:Architecture: x86_64 2024-06-04T04:20:55.373 INFO:teuthology.orchestra.run.smithi070.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-04T04:20:55.373 INFO:teuthology.orchestra.run.smithi070.stdout:Group : Unspecified 2024-06-04T04:20:55.373 INFO:teuthology.orchestra.run.smithi070.stdout:Size : 8264254 2024-06-04T04:20:55.374 INFO:teuthology.orchestra.run.smithi070.stdout:License : GPLv3+ 2024-06-04T04:20:55.374 INFO:teuthology.orchestra.run.smithi070.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-04T04:20:55.374 INFO:teuthology.orchestra.run.smithi070.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-04T04:20:55.374 INFO:teuthology.orchestra.run.smithi070.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-04T04:20:55.374 INFO:teuthology.orchestra.run.smithi070.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-04T04:20:55.374 INFO:teuthology.orchestra.run.smithi070.stdout:Packager : builder@centos.org 2024-06-04T04:20:55.374 INFO:teuthology.orchestra.run.smithi070.stdout:Vendor : CentOS 2024-06-04T04:20:55.374 INFO:teuthology.orchestra.run.smithi070.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-04T04:20:55.374 INFO:teuthology.orchestra.run.smithi070.stdout:Summary : Support tools for GRUB. 2024-06-04T04:20:55.374 INFO:teuthology.orchestra.run.smithi070.stdout:Description : 2024-06-04T04:20:55.374 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:20:55.374 INFO:teuthology.orchestra.run.smithi070.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-04T04:20:55.374 INFO:teuthology.orchestra.run.smithi070.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-04T04:20:55.375 INFO:teuthology.orchestra.run.smithi070.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-04T04:20:55.375 INFO:teuthology.orchestra.run.smithi070.stdout:hardware devices. 2024-06-04T04:20:55.375 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:20:55.375 INFO:teuthology.orchestra.run.smithi070.stdout:This subpackage provides tools for support of all platforms. 2024-06-04T04:20:55.375 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-04T04:20:55.376 INFO:teuthology.task.kernel:Updating grub on smithi070 to boot 5.14.0-452.el9.x86_64 2024-06-04T04:20:55.376 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-04T04:20:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:Generating grub configuration file ... 2024-06-04T04:20:57.370 INFO:teuthology.orchestra.run.smithi070.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-04T04:20:57.404 INFO:teuthology.orchestra.run.smithi070.stderr:done 2024-06-04T04:20:57.406 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-06-04T04:20:57.421 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.u9vAlTObFV 2024-06-04T04:20:57.421 DEBUG:teuthology.orchestra.run.smithi070:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.u9vAlTObFV 2024-06-04T04:20:57.486 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.u9vAlTObFV 2024-06-04T04:20:57.623 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.u9vAlTObFV is 6KB 2024-06-04T04:20:57.670 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.u9vAlTObFV 2024-06-04T04:20:57.685 DEBUG:teuthology.orchestra.run.smithi070:> sudo /bin/ls /boot/loader/entries || true 2024-06-04T04:20:57.746 INFO:teuthology.orchestra.run.smithi070.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-04T04:20:57.747 INFO:teuthology.orchestra.run.smithi070.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-04T04:20:57.747 INFO:teuthology.orchestra.run.smithi070.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-04T04:20:57.747 INFO:teuthology.orchestra.run.smithi070.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-04T04:20:57.748 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-04T04:20:57.883 DEBUG:teuthology.orchestra.run.smithi070:> sudo shutdown -r now 2024-06-04T04:21:10.303 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-06-04T04:21:19.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-06-04T04:21:19.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-04T04:21:22.370 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-06-04T04:21:25.092 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-06-04T04:21:25.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-06-04T04:21:25.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-04T04:21:27.887 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-06-04T04:21:27.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-06-04T04:21:27.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-04T04:21:34.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-06-04T04:21:34.378 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-04T04:21:43.582 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-06-04T04:21:46.398 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.70 2024-06-04T04:21:52.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-06-04T04:21:52.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-04T04:21:55.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-06-04T04:21:55.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-04T04:21:55.646 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-06-04T04:22:06.155 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-06-04T04:22:06.851 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-06-04T04:22:06.851 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-452.el9.x86_64"... 2024-06-04T04:22:06.852 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-06-04T04:22:06.900 INFO:teuthology.orchestra.run.smithi112.stdout:5.14.0-452.el9.x86_64 2024-06-04T04:22:06.901 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-04T04:22:06.901 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-04T04:22:06.901 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-04T04:22:07.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-06-04T04:22:07.648 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-04T04:22:07.901 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-06-04T04:22:07.902 DEBUG:teuthology.orchestra.run.smithi112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-04T04:22:08.209 INFO:teuthology.orchestra.run.smithi112.stdout:ttyS1 2024-06-04T04:22:08.247 DEBUG:teuthology.parallel:result is None 2024-06-04T04:22:39.519 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-06-04T04:22:39.930 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-06-04T04:22:39.930 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-452.el9.x86_64"... 2024-06-04T04:22:39.931 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-06-04T04:22:39.945 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-452.el9.x86_64 2024-06-04T04:22:39.945 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-04T04:22:39.945 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-04T04:22:39.945 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-04T04:22:40.946 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-06-04T04:22:40.946 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-04T04:22:41.029 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-06-04T04:22:41.069 DEBUG:teuthology.parallel:result is None 2024-06-04T04:22:55.402 DEBUG:teuthology.orchestra.remote:timed out 2024-06-04T04:23:07.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-06-04T04:23:07.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-04T04:23:07.790 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-06-04T04:23:08.177 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-06-04T04:23:08.177 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-452.el9.x86_64"... 2024-06-04T04:23:08.177 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-06-04T04:23:08.193 INFO:teuthology.orchestra.run.smithi070.stdout:5.14.0-452.el9.x86_64 2024-06-04T04:23:08.193 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-04T04:23:08.193 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-04T04:23:08.193 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-04T04:23:09.194 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-06-04T04:23:09.194 DEBUG:teuthology.orchestra.run.smithi070:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-04T04:23:09.290 INFO:teuthology.orchestra.run.smithi070.stdout:ttyS1 2024-06-04T04:23:09.320 DEBUG:teuthology.parallel:result is None 2024-06-04T04:23:09.320 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-04T04:23:09.331 INFO:teuthology.task.internal:Creating test directory... 2024-06-04T04:23:09.331 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-04T04:23:09.333 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-04T04:23:09.336 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-04T04:23:09.355 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-04T04:23:09.392 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-04T04:23:09.399 INFO:teuthology.task.internal:Creating archive directory... 2024-06-04T04:23:09.399 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-04T04:23:09.402 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-04T04:23:09.404 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-04T04:23:09.501 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-04T04:23:09.508 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-04T04:23:09.508 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-04T04:23:09.511 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-04T04:23:09.514 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-04T04:23:09.548 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-04T04:23:09.551 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-04T04:23:09.553 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-04T04:23:09.560 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-04T04:23:09.563 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-04T04:23:09.564 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-04T04:23:09.566 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-04T04:23:09.574 INFO:teuthology.task.internal:Configuring sudo... 2024-06-04T04:23:09.574 DEBUG:teuthology.orchestra.run.smithi070:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-04T04:23:09.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-04T04:23:09.608 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-04T04:23:09.636 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-04T04:23:09.645 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-04T04:23:09.645 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-04T04:23:09.673 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-04T04:23:09.676 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-04T04:23:09.693 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-04T04:23:09.773 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-04T04:23:09.823 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-04T04:23:09.907 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-04T04:23:09.947 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T04:23:09.947 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-04T04:23:10.012 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-04T04:23:10.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-04T04:23:10.116 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-04T04:23:10.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-04T04:23:10.244 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-04T04:23:10.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-04T04:23:10.312 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-04T04:23:10.358 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-04T04:23:10.409 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-04T04:23:10.494 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-04T04:23:10.535 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-06-04T04:23:10.535 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-04T04:23:10.602 DEBUG:teuthology.orchestra.run.smithi070:> sudo service rsyslog restart 2024-06-04T04:23:10.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-06-04T04:23:10.610 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2024-06-04T04:23:10.653 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-04T04:23:10.662 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-04T04:23:10.689 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-04T04:23:11.161 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-04T04:23:11.169 INFO:teuthology.task.internal:Starting timer... 2024-06-04T04:23:11.169 INFO:teuthology.run_tasks:Running task pcp... 2024-06-04T04:23:11.229 INFO:teuthology.run_tasks:Running task selinux... 2024-06-04T04:23:11.237 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-06-04T04:23:11.238 DEBUG:teuthology.orchestra.run.smithi070:> sudo service auditd rotate 2024-06-04T04:23:11.300 INFO:teuthology.orchestra.run.smithi070.stdout:Rotating logs: 2024-06-04T04:23:11.303 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-06-04T04:23:11.375 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-06-04T04:23:11.377 DEBUG:teuthology.orchestra.run.smithi112:> sudo service auditd rotate 2024-06-04T04:23:11.440 INFO:teuthology.orchestra.run.smithi112.stdout:Rotating logs: 2024-06-04T04:23:11.442 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-04T04:23:11.443 DEBUG:teuthology.orchestra.run.smithi070:> /usr/sbin/getenforce 2024-06-04T04:23:11.474 INFO:teuthology.orchestra.run.smithi070.stdout:Permissive 2024-06-04T04:23:11.474 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-06-04T04:23:11.512 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-06-04T04:23:11.512 DEBUG:teuthology.orchestra.run.smithi112:> /usr/sbin/getenforce 2024-06-04T04:23:11.550 INFO:teuthology.orchestra.run.smithi112.stdout:Permissive 2024-06-04T04:23:11.550 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi070.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi112.front.sepia.ceph.com': 'permissive'} 2024-06-04T04:23:11.551 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-04T04:23:11.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-04T04:23:11.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-04T04:23:11.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-04T04:23:11.609 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-04T04:23:11.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-04T04:23:11.638 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-04T04:23:11.638 DEBUG:teuthology.orchestra.run.smithi070:> sudo /usr/sbin/setenforce permissive 2024-06-04T04:23:11.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-06-04T04:23:11.706 DEBUG:teuthology.orchestra.run.smithi112:> sudo /usr/sbin/setenforce permissive 2024-06-04T04:23:11.733 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-04T04:23:11.743 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-04T04:23:11.754 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-04T04:23:11.755 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi070.front.sepia.ceph.com,smithi098.front.sepia.ceph.com,smithi112.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-04T04:30:07.147 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi070.front.sepia.ceph.com'), Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi112.front.sepia.ceph.com')] 2024-06-04T04:30:07.149 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-06-04T04:30:07.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-04T04:30:07.228 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-06-04T04:30:07.268 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-06-04T04:30:07.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-06-04T04:30:07.269 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-04T04:30:07.350 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-06-04T04:30:07.390 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-06-04T04:30:07.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-06-04T04:30:07.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-04T04:30:07.469 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-06-04T04:30:07.549 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-06-04T04:30:07.549 INFO:teuthology.run_tasks:Running task clock... 2024-06-04T04:30:07.561 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-04T04:30:07.561 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-04T04:30:07.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-04T04:30:07.564 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-04T04:30:07.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-04T04:30:07.567 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-04T04:30:07.567 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-04T04:30:07.591 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-04T04:30:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-04T04:30:07.605 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-04T04:30:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-04T04:30:07.618 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-04T04:30:07.629 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: ntpd: command not found 2024-06-04T04:30:07.633 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-04T04:30:07.633 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-06-04T04:30:07.640 INFO:teuthology.orchestra.run.smithi070.stdout:506 Cannot talk to daemon 2024-06-04T04:30:07.644 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-06-04T04:30:07.654 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-04T04:30:07.656 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: ntpd: command not found 2024-06-04T04:30:07.658 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-04T04:30:07.667 INFO:teuthology.orchestra.run.smithi112.stdout:506 Cannot talk to daemon 2024-06-04T04:30:07.668 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-04T04:30:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-04T04:30:07.681 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-04T04:30:07.695 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-04T04:30:07.721 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-06-04T04:30:07.726 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-04T04:30:07.726 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-06-04T04:30:07.726 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-04T04:30:07.726 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-04T04:30:07.726 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-04T04:30:07.726 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-04T04:30:07.733 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-06-04T04:30:07.737 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-04T04:30:07.737 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-06-04T04:30:07.737 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-04T04:30:07.737 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-04T04:30:07.737 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-04T04:30:07.737 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-04T04:30:07.752 INFO:teuthology.orchestra.run.smithi112.stderr:bash: line 1: ntpq: command not found 2024-06-04T04:30:07.756 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-04T04:30:07.756 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2024-06-04T04:30:07.756 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-04T04:30:07.757 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-04T04:30:07.757 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-04T04:30:07.757 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-04T04:30:07.757 INFO:teuthology.run_tasks:Running task install... 2024-06-04T04:30:07.765 DEBUG:teuthology.task.install:project ceph 2024-06-04T04:30:07.765 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b7328fed0e2e9ab66b09f8134f3cebfc71214504'}} 2024-06-04T04:30:07.765 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'b7328fed0e2e9ab66b09f8134f3cebfc71214504'} 2024-06-04T04:30:07.765 INFO:teuthology.task.install:Using flavor: default 2024-06-04T04:30:07.771 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-04T04:30:07.772 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-06-04T04:30:07.772 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b7328fed0e2e9ab66b09f8134f3cebfc71214504', 'tag': None, 'wait_for_package': False} 2024-06-04T04:30:07.773 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b7328fed0e2e9ab66b09f8134f3cebfc71214504 2024-06-04T04:30:07.775 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b7328fed0e2e9ab66b09f8134f3cebfc71214504', 'tag': None, 'wait_for_package': False} 2024-06-04T04:30:07.775 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b7328fed0e2e9ab66b09f8134f3cebfc71214504 2024-06-04T04:30:07.776 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b7328fed0e2e9ab66b09f8134f3cebfc71214504', 'tag': None, 'wait_for_package': False} 2024-06-04T04:30:07.776 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b7328fed0e2e9ab66b09f8134f3cebfc71214504 2024-06-04T04:30:07.944 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/flavors/default/ 2024-06-04T04:30:07.944 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3882.gb7328fed 2024-06-04T04:30:07.949 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/flavors/default/ 2024-06-04T04:30:07.949 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3882.gb7328fed 2024-06-04T04:30:07.952 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/flavors/default/ 2024-06-04T04:30:07.952 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3882.gb7328fed 2024-06-04T04:30:08.064 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-04T04:30:08.064 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-06-04T04:30:08.064 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-04T04:30:08.073 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-04T04:30:08.073 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T04:30:08.073 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-04T04:30:08.089 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-06-04T04:30:08.089 DEBUG:teuthology.orchestra.run.smithi112:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b7328fed0e2e9ab66b09f8134f3cebfc71214504/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-04T04:30:08.098 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-06-04T04:30:08.098 DEBUG:teuthology.orchestra.run.smithi070:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b7328fed0e2e9ab66b09f8134f3cebfc71214504/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-04T04:30:08.118 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-04T04:30:08.119 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-04T04:30:08.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-04T04:30:08.144 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-06-04T04:30:08.144 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b7328fed0e2e9ab66b09f8134f3cebfc71214504/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-04T04:30:08.155 DEBUG:teuthology.orchestra.run.smithi112:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-04T04:30:08.163 DEBUG:teuthology.orchestra.run.smithi070:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-04T04:30:08.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-04T04:30:08.229 DEBUG:teuthology.orchestra.run.smithi112:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-04T04:30:08.237 DEBUG:teuthology.orchestra.run.smithi070:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-04T04:30:08.287 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-04T04:30:08.293 INFO:teuthology.orchestra.run.smithi112.stdout:check_obsoletes = 1 2024-06-04T04:30:08.294 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum clean all 2024-06-04T04:30:08.301 INFO:teuthology.orchestra.run.smithi070.stdout:check_obsoletes = 1 2024-06-04T04:30:08.303 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum clean all 2024-06-04T04:30:08.352 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-06-04T04:30:08.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-06-04T04:30:08.520 INFO:teuthology.orchestra.run.smithi112.stdout:45 files removed 2024-06-04T04:30:08.534 INFO:teuthology.orchestra.run.smithi070.stdout:45 files removed 2024-06-04T04:30:08.547 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-06-04T04:30:08.560 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-06-04T04:30:08.580 INFO:teuthology.orchestra.run.smithi098.stdout:45 files removed 2024-06-04T04:30:08.607 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-06-04T04:30:09.261 INFO:teuthology.orchestra.run.smithi070.stdout:ceph packages for x86_64 196 kB/s | 84 kB 00:00 2024-06-04T04:30:09.267 INFO:teuthology.orchestra.run.smithi112.stdout:ceph packages for x86_64 188 kB/s | 84 kB 00:00 2024-06-04T04:30:09.284 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 208 kB/s | 84 kB 00:00 2024-06-04T04:30:09.642 INFO:teuthology.orchestra.run.smithi112.stdout:ceph noarch packages 36 kB/s | 12 kB 00:00 2024-06-04T04:30:09.711 INFO:teuthology.orchestra.run.smithi070.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-06-04T04:30:09.718 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-06-04T04:30:10.093 INFO:teuthology.orchestra.run.smithi112.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-06-04T04:30:10.103 INFO:teuthology.orchestra.run.smithi070.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2024-06-04T04:30:10.160 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-06-04T04:31:11.026 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - BaseOS 136 kB/s | 8.1 MB 01:00 2024-06-04T04:31:25.977 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 109 kB/s | 8.1 MB 01:15 2024-06-04T04:34:15.557 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - AppStream 107 kB/s | 19 MB 03:03 2024-06-04T04:34:26.112 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 32 kB/s | 8.1 MB 04:15 2024-06-04T04:34:51.201 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 814 kB/s | 19 MB 00:24 2024-06-04T04:34:51.401 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - CRB 196 kB/s | 6.2 MB 00:32 2024-06-04T04:35:03.160 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - Extras packages 1.6 kB/s | 17 kB 00:10 2024-06-04T04:35:50.131 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 74 kB/s | 19 MB 04:23 2024-06-04T04:37:13.688 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 46 kB/s | 6.2 MB 02:18 2024-06-04T04:37:19.928 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 159 kB/s | 21 MB 02:16 2024-06-04T04:37:24.994 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-06-04T04:37:27.412 INFO:teuthology.orchestra.run.smithi112.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-04T04:37:27.413 INFO:teuthology.orchestra.run.smithi112.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-04T04:37:27.424 INFO:teuthology.orchestra.run.smithi112.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-06-04T04:37:27.425 INFO:teuthology.orchestra.run.smithi112.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-06-04T04:37:27.426 INFO:teuthology.orchestra.run.smithi112.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-06-04T04:37:27.426 INFO:teuthology.orchestra.run.smithi112.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-06-04T04:37:27.428 INFO:teuthology.orchestra.run.smithi112.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-06-04T04:37:27.430 INFO:teuthology.orchestra.run.smithi112.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-06-04T04:37:27.431 INFO:teuthology.orchestra.run.smithi112.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-06-04T04:37:27.492 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-06-04T04:37:27.522 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-06-04T04:37:27.522 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-06-04T04:37:27.523 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-06-04T04:37:27.523 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-06-04T04:37:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-06-04T04:37:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: ceph x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 6.6 k 2024-06-04T04:37:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 5.4 M 2024-06-04T04:37:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 890 k 2024-06-04T04:37:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 145 k 2024-06-04T04:37:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 1.5 M 2024-06-04T04:37:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 143 k 2024-06-04T04:37:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 3.5 M 2024-06-04T04:37:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 7.4 M 2024-06-04T04:37:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 49 k 2024-06-04T04:37:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 11 M 2024-06-04T04:37:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 49 M 2024-06-04T04:37:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 273 k 2024-06-04T04:37:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 763 k 2024-06-04T04:37:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-mirror x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 225 k 2024-06-04T04:37:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-shell x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 27 k 2024-06-04T04:37:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-top noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 18 k 2024-06-04T04:37:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-06-04T04:37:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-06-04T04:37:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-06-04T04:37:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 32 k 2024-06-04T04:37:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2 x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 747 k 2024-06-04T04:37:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 127 k 2024-06-04T04:37:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-06-04T04:37:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-06-04T04:37:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-06-04T04:37:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-06-04T04:37:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-06-04T04:37:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-06-04T04:37:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 161 k 2024-06-04T04:37:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 320 k 2024-06-04T04:37:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 300 k 2024-06-04T04:37:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 100 k 2024-06-04T04:37:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 85 k 2024-06-04T04:37:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 3.1 M 2024-06-04T04:37:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 170 k 2024-06-04T04:37:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-06-04T04:37:27.526 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrading: 2024-06-04T04:37:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: NetworkManager x86_64 1:1.47.91-1.el9 baseos 2.3 M 2024-06-04T04:37:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: NetworkManager-libnm x86_64 1:1.47.91-1.el9 baseos 1.8 M 2024-06-04T04:37:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: NetworkManager-team x86_64 1:1.47.91-1.el9 baseos 41 k 2024-06-04T04:37:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: NetworkManager-tui x86_64 1:1.47.91-1.el9 baseos 248 k 2024-06-04T04:37:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-06-04T04:37:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-06-04T04:37:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-06-04T04:37:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: librados2 x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 3.4 M 2024-06-04T04:37:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1 x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 3.1 M 2024-06-04T04:37:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-06-04T04:37:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-06-04T04:37:27.527 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-06-04T04:37:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-04T04:37:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-06-04T04:37:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-04T04:37:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 21 M 2024-06-04T04:37:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 34 k 2024-06-04T04:37:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 2.4 M 2024-06-04T04:37:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 274 k 2024-06-04T04:37:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 4.7 M 2024-06-04T04:37:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 17 M 2024-06-04T04:37:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 16 k 2024-06-04T04:37:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 25 k 2024-06-04T04:37:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-04T04:37:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-04T04:37:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-04T04:37:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-04T04:37:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-04T04:37:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-04T04:37:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-04T04:37:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-04T04:37:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 163 k 2024-06-04T04:37:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-04T04:37:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-04T04:37:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-04T04:37:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-04T04:37:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-04T04:37:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1 x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 501 k 2024-06-04T04:37:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-04T04:37:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2 x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 5.4 M 2024-06-04T04:37:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-04T04:37:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-06-04T04:37:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-04T04:37:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-04T04:37:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-04T04:37:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-04T04:37:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-04T04:37:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-04T04:37:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-04T04:37:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-06-04T04:37:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-04T04:37:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-04T04:37:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-04T04:37:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-06-04T04:37:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-04T04:37:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-04T04:37:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-04T04:37:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 45 k 2024-06-04T04:37:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 135 k 2024-06-04T04:37:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-04T04:37:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-04T04:37:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-04T04:37:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-04T04:37:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-06-04T04:37:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-06-04T04:37:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-04T04:37:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-04T04:37:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-06-04T04:37:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-04T04:37:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-04T04:37:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-04T04:37:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-04T04:37:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-04T04:37:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-04T04:37:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-04T04:37:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-04T04:37:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-04T04:37:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-04T04:37:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-04T04:37:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-04T04:37:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-04T04:37:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-04T04:37:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-04T04:37:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-04T04:37:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-04T04:37:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-04T04:37:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-04T04:37:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-04T04:37:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-04T04:37:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-04T04:37:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-04T04:37:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-04T04:37:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-04T04:37:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-04T04:37:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-04T04:37:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-04T04:37:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-04T04:37:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-06-04T04:37:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-04T04:37:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-04T04:37:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-04T04:37:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-04T04:37:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-04T04:37:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-04T04:37:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-04T04:37:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-04T04:37:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-04T04:37:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-04T04:37:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-04T04:37:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-06-04T04:37:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-04T04:37:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-04T04:37:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-04T04:37:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-04T04:37:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-04T04:37:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-04T04:37:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-04T04:37:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-04T04:37:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-04T04:37:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-06-04T04:37:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-04T04:37:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-06-04T04:37:27.535 INFO:teuthology.orchestra.run.smithi112.stdout:Installing weak dependencies: 2024-06-04T04:37:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-04T04:37:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:37:27.536 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-06-04T04:37:27.536 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-06-04T04:37:27.536 INFO:teuthology.orchestra.run.smithi112.stdout:Install 148 Packages 2024-06-04T04:37:27.536 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrade 11 Packages 2024-06-04T04:37:27.536 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:37:27.536 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 227 M 2024-06-04T04:37:27.536 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-06-04T04:37:28.870 INFO:teuthology.orchestra.run.smithi112.stdout:(1/159): ceph-19.0.0-3882.gb7328fed.el9.x86_64. 11 kB/s | 6.6 kB 00:00 2024-06-04T04:37:29.112 INFO:teuthology.orchestra.run.smithi112.stdout:(2/159): ceph-base-19.0.0-3882.gb7328fed.el9.x8 6.5 MB/s | 5.4 MB 00:00 2024-06-04T04:37:29.188 INFO:teuthology.orchestra.run.smithi112.stdout:(3/159): ceph-immutable-object-cache-19.0.0-388 1.9 MB/s | 145 kB 00:00 2024-06-04T04:37:29.396 INFO:teuthology.orchestra.run.smithi112.stdout:(4/159): ceph-mds-19.0.0-3882.gb7328fed.el9.x86 11 MB/s | 2.4 MB 00:00 2024-06-04T04:37:29.430 INFO:teuthology.orchestra.run.smithi112.stdout:(5/159): ceph-fuse-19.0.0-3882.gb7328fed.el9.x8 1.6 MB/s | 890 kB 00:00 2024-06-04T04:37:29.555 INFO:teuthology.orchestra.run.smithi112.stdout:(6/159): ceph-mgr-19.0.0-3882.gb7328fed.el9.x86 9.6 MB/s | 1.5 MB 00:00 2024-06-04T04:37:30.556 INFO:teuthology.orchestra.run.smithi112.stdout:(7/159): ceph-osd-19.0.0-3882.gb7328fed.el9.x86 17 MB/s | 17 MB 00:01 2024-06-04T04:37:30.989 INFO:teuthology.orchestra.run.smithi112.stdout:(8/159): ceph-mon-19.0.0-3882.gb7328fed.el9.x86 3.0 MB/s | 4.7 MB 00:01 2024-06-04T04:37:31.056 INFO:teuthology.orchestra.run.smithi112.stdout:(9/159): ceph-selinux-19.0.0-3882.gb7328fed.el9 377 kB/s | 25 kB 00:00 2024-06-04T04:37:31.190 INFO:teuthology.orchestra.run.smithi112.stdout:(10/159): ceph-radosgw-19.0.0-3882.gb7328fed.el 17 MB/s | 11 MB 00:00 2024-06-04T04:37:31.266 INFO:teuthology.orchestra.run.smithi112.stdout:(11/159): cephfs-mirror-19.0.0-3882.gb7328fed.e 2.9 MB/s | 225 kB 00:00 2024-06-04T04:37:31.316 INFO:teuthology.orchestra.run.smithi112.stdout:(12/159): cephfs-shell-19.0.0-3882.gb7328fed.el 543 kB/s | 27 kB 00:00 2024-06-04T04:37:31.366 INFO:teuthology.orchestra.run.smithi112.stdout:(13/159): libcephfs-devel-19.0.0-3882.gb7328fed 633 kB/s | 32 kB 00:00 2024-06-04T04:37:31.516 INFO:teuthology.orchestra.run.smithi112.stdout:(14/159): libcephfs2-19.0.0-3882.gb7328fed.el9. 4.9 MB/s | 747 kB 00:00 2024-06-04T04:37:31.600 INFO:teuthology.orchestra.run.smithi112.stdout:(15/159): libcephsqlite-19.0.0-3882.gb7328fed.e 1.9 MB/s | 163 kB 00:00 2024-06-04T04:37:31.692 INFO:teuthology.orchestra.run.smithi112.stdout:(16/159): librados-devel-19.0.0-3882.gb7328fed. 1.3 MB/s | 127 kB 00:00 2024-06-04T04:37:31.784 INFO:teuthology.orchestra.run.smithi112.stdout:(17/159): libradosstriper1-19.0.0-3882.gb7328fe 5.3 MB/s | 501 kB 00:00 2024-06-04T04:37:32.284 INFO:teuthology.orchestra.run.smithi112.stdout:(18/159): librgw2-19.0.0-3882.gb7328fed.el9.x86 11 MB/s | 5.4 MB 00:00 2024-06-04T04:37:32.476 INFO:teuthology.orchestra.run.smithi112.stdout:(19/159): ceph-common-19.0.0-3882.gb7328fed.el9 5.0 MB/s | 21 MB 00:04 2024-06-04T04:37:32.493 INFO:teuthology.orchestra.run.smithi112.stdout:(20/159): python3-ceph-argparse-19.0.0-3882.gb7 216 kB/s | 45 kB 00:00 2024-06-04T04:37:32.627 INFO:teuthology.orchestra.run.smithi112.stdout:(21/159): python3-ceph-common-19.0.0-3882.gb732 903 kB/s | 135 kB 00:00 2024-06-04T04:37:32.669 INFO:teuthology.orchestra.run.smithi112.stdout:(22/159): python3-cephfs-19.0.0-3882.gb7328fed. 920 kB/s | 161 kB 00:00 2024-06-04T04:37:32.811 INFO:teuthology.orchestra.run.smithi112.stdout:(23/159): python3-rados-19.0.0-3882.gb7328fed.e 1.7 MB/s | 320 kB 00:00 2024-06-04T04:37:32.878 INFO:teuthology.orchestra.run.smithi112.stdout:(24/159): python3-rbd-19.0.0-3882.gb7328fed.el9 1.4 MB/s | 300 kB 00:00 2024-06-04T04:37:33.019 INFO:teuthology.orchestra.run.smithi112.stdout:(25/159): python3-rgw-19.0.0-3882.gb7328fed.el9 478 kB/s | 100 kB 00:00 2024-06-04T04:37:33.045 INFO:teuthology.orchestra.run.smithi112.stdout:(26/159): rbd-fuse-19.0.0-3882.gb7328fed.el9.x8 511 kB/s | 85 kB 00:00 2024-06-04T04:37:33.662 INFO:teuthology.orchestra.run.smithi112.stdout:(27/159): rbd-nbd-19.0.0-3882.gb7328fed.el9.x86 276 kB/s | 170 kB 00:00 2024-06-04T04:37:33.845 INFO:teuthology.orchestra.run.smithi112.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3882.g 187 kB/s | 34 kB 00:00 2024-06-04T04:37:33.979 INFO:teuthology.orchestra.run.smithi112.stdout:(29/159): ceph-mgr-cephadm-19.0.0-3882.gb7328fe 1.0 MB/s | 143 kB 00:00 2024-06-04T04:37:34.588 INFO:teuthology.orchestra.run.smithi112.stdout:(30/159): ceph-test-19.0.0-3882.gb7328fed.el9.x 14 MB/s | 49 MB 00:03 2024-06-04T04:37:34.646 INFO:teuthology.orchestra.run.smithi112.stdout:(31/159): rbd-mirror-19.0.0-3882.gb7328fed.el9. 1.9 MB/s | 3.1 MB 00:01 2024-06-04T04:37:34.721 INFO:teuthology.orchestra.run.smithi112.stdout:(32/159): ceph-mgr-dashboard-19.0.0-3882.gb7328 4.8 MB/s | 3.5 MB 00:00 2024-06-04T04:37:35.013 INFO:teuthology.orchestra.run.smithi112.stdout:(33/159): ceph-mgr-diskprediction-local-19.0.0- 17 MB/s | 7.4 MB 00:00 2024-06-04T04:37:35.039 INFO:teuthology.orchestra.run.smithi112.stdout:(34/159): ceph-mgr-rook-19.0.0-3882.gb7328fed.e 156 kB/s | 49 kB 00:00 2024-06-04T04:37:35.064 INFO:teuthology.orchestra.run.smithi112.stdout:(35/159): ceph-prometheus-alerts-19.0.0-3882.gb 329 kB/s | 16 kB 00:00 2024-06-04T04:37:35.089 INFO:teuthology.orchestra.run.smithi112.stdout:(36/159): ceph-mgr-modules-core-19.0.0-3882.gb7 620 kB/s | 274 kB 00:00 2024-06-04T04:37:35.114 INFO:teuthology.orchestra.run.smithi112.stdout:(37/159): ceph-volume-19.0.0-3882.gb7328fed.el9 3.5 MB/s | 273 kB 00:00 2024-06-04T04:37:35.157 INFO:teuthology.orchestra.run.smithi112.stdout:(38/159): cephadm-19.0.0-3882.gb7328fed.el9.noa 8.1 MB/s | 763 kB 00:00 2024-06-04T04:37:35.173 INFO:teuthology.orchestra.run.smithi112.stdout:(39/159): cephfs-top-19.0.0-3882.gb7328fed.el9. 217 kB/s | 18 kB 00:00 2024-06-04T04:37:39.724 INFO:teuthology.orchestra.run.smithi112.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:37:40.065 INFO:teuthology.orchestra.run.smithi112.stdout:(40/159): libgfortran-11.4.1-3.el9.x86_64.rpm 164 kB/s | 803 kB 00:04 2024-06-04T04:37:40.068 INFO:teuthology.orchestra.run.smithi112.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:37:40.132 INFO:teuthology.orchestra.run.smithi112.stdout:(41/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-06-04T04:37:40.135 INFO:teuthology.orchestra.run.smithi112.stdout:[MIRROR] python3-cryptography-36.0.1-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cryptography-36.0.1-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:37:40.574 INFO:teuthology.orchestra.run.smithi112.stdout:(42/159): python3-cryptography-36.0.1-4.el9.x86 2.8 MB/s | 1.2 MB 00:00 2024-06-04T04:37:40.666 INFO:teuthology.orchestra.run.smithi112.stdout:(43/159): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-06-04T04:37:40.799 INFO:teuthology.orchestra.run.smithi112.stdout:(44/159): python3-pycparser-2.20-6.el9.noarch.r 1.0 MB/s | 135 kB 00:00 2024-06-04T04:37:40.883 INFO:teuthology.orchestra.run.smithi112.stdout:(45/159): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-06-04T04:37:41.033 INFO:teuthology.orchestra.run.smithi112.stdout:(46/159): python3-urllib3-1.26.5-5.el9.noarch.r 1.4 MB/s | 215 kB 00:00 2024-06-04T04:37:41.036 INFO:teuthology.orchestra.run.smithi112.stdout:[MIRROR] automake-1.16.2-8.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/automake-1.16.2-8.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-04T04:37:41.208 INFO:teuthology.orchestra.run.smithi112.stdout:(47/159): automake-1.16.2-8.el9.noarch.rpm 3.8 MB/s | 689 kB 00:00 2024-06-04T04:37:41.211 INFO:teuthology.orchestra.run.smithi112.stdout:[MIRROR] bison-3.7.4-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/bison-3.7.4-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:37:41.484 INFO:teuthology.orchestra.run.smithi112.stdout:(48/159): bison-3.7.4-5.el9.x86_64.rpm 3.3 MB/s | 943 kB 00:00 2024-06-04T04:37:41.487 INFO:teuthology.orchestra.run.smithi112.stdout:[MIRROR] boost-program-options-1.75.0-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:37:41.526 INFO:teuthology.orchestra.run.smithi112.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 107 kB 00:00 2024-06-04T04:37:41.576 INFO:teuthology.orchestra.run.smithi112.stdout:(50/159): elfutils-libelf-devel-0.191-4.el9.x86 434 kB/s | 22 kB 00:00 2024-06-04T04:37:41.701 INFO:teuthology.orchestra.run.smithi112.stdout:(51/159): flex-2.6.4-9.el9.x86_64.rpm 2.5 MB/s | 313 kB 00:00 2024-06-04T04:37:41.752 INFO:teuthology.orchestra.run.smithi112.stdout:(52/159): flexiblas-3.0.4-8.el9.x86_64.rpm 653 kB/s | 33 kB 00:00 2024-06-04T04:37:41.793 INFO:teuthology.orchestra.run.smithi112.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:37:41.877 INFO:teuthology.orchestra.run.smithi112.stdout:(53/159): fuse-2.9.9-15.el9.x86_64.rpm 12 kB/s | 80 kB 00:06 2024-06-04T04:37:41.969 INFO:teuthology.orchestra.run.smithi112.stdout:(54/159): flexiblas-openblas-openmp-3.0.4-8.el9 199 kB/s | 18 kB 00:00 2024-06-04T04:37:42.019 INFO:teuthology.orchestra.run.smithi112.stdout:(55/159): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-06-04T04:37:42.144 INFO:teuthology.orchestra.run.smithi112.stdout:(56/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-06-04T04:37:42.186 INFO:teuthology.orchestra.run.smithi112.stdout:(57/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-06-04T04:37:42.286 INFO:teuthology.orchestra.run.smithi112.stdout:(58/159): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 662 kB 00:00 2024-06-04T04:37:42.462 INFO:teuthology.orchestra.run.smithi112.stdout:(59/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.2 MB/s | 3.0 MB 00:00 2024-06-04T04:37:42.487 INFO:teuthology.orchestra.run.smithi112.stdout:(60/159): libtool-2.4.6-45.el9.x86_64.rpm 2.8 MB/s | 582 kB 00:00 2024-06-04T04:37:42.512 INFO:teuthology.orchestra.run.smithi112.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 506 kB/s | 25 kB 00:00 2024-06-04T04:37:42.537 INFO:teuthology.orchestra.run.smithi112.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-06-04T04:37:42.621 INFO:teuthology.orchestra.run.smithi112.stdout:(63/159): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-06-04T04:37:42.646 INFO:teuthology.orchestra.run.smithi112.stdout:(64/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-06-04T04:37:42.672 INFO:teuthology.orchestra.run.smithi112.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 702 kB/s | 35 kB 00:00 2024-06-04T04:37:44.064 INFO:teuthology.orchestra.run.smithi112.stdout:(66/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.2 MB/s | 4.4 MB 00:01 2024-06-04T04:37:44.157 INFO:teuthology.orchestra.run.smithi112.stdout:(67/159): openblas-openmp-0.3.21-2.el9.x86_64.r 3.1 MB/s | 4.7 MB 00:01 2024-06-04T04:37:44.448 INFO:teuthology.orchestra.run.smithi112.stdout:(68/159): postgresql-contrib-13.14-1.el9.x86_64 3.1 MB/s | 911 kB 00:00 2024-06-04T04:37:44.540 INFO:teuthology.orchestra.run.smithi112.stdout:(69/159): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 133 kB 00:00 2024-06-04T04:37:44.673 INFO:teuthology.orchestra.run.smithi112.stdout:(70/159): postgresql-13.14-1.el9.x86_64.rpm 2.6 MB/s | 1.6 MB 00:00 2024-06-04T04:37:44.990 INFO:teuthology.orchestra.run.smithi112.stdout:(71/159): protobuf-3.14.0-13.el9.x86_64.rpm 3.2 MB/s | 1.0 MB 00:00 2024-06-04T04:37:45.049 INFO:teuthology.orchestra.run.smithi112.stdout:(72/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-06-04T04:37:46.282 INFO:teuthology.orchestra.run.smithi112.stdout:(73/159): postgresql-server-13.14-1.el9.x86_64. 3.3 MB/s | 5.8 MB 00:01 2024-06-04T04:37:46.316 INFO:teuthology.orchestra.run.smithi112.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.8 MB/s | 61 kB 00:00 2024-06-04T04:37:46.410 INFO:teuthology.orchestra.run.smithi112.stdout:(75/159): python3-devel-3.9.18-3.el9.x86_64.rpm 2.6 MB/s | 244 kB 00:00 2024-06-04T04:37:46.508 INFO:teuthology.orchestra.run.smithi112.stdout:(76/159): python3-jinja2-2.11.3-6.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-06-04T04:37:46.550 INFO:teuthology.orchestra.run.smithi112.stdout:(77/159): python3-jsonpatch-1.21-16.el9.noarch. 618 kB/s | 26 kB 00:00 2024-06-04T04:37:46.583 INFO:teuthology.orchestra.run.smithi112.stdout:(78/159): python3-jsonpointer-2.0-4.el9.noarch. 577 kB/s | 19 kB 00:00 2024-06-04T04:37:46.667 INFO:teuthology.orchestra.run.smithi112.stdout:(79/159): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-06-04T04:37:46.717 INFO:teuthology.orchestra.run.smithi112.stdout:(80/159): python3-markupsafe-1.1.1-12.el9.x86_6 695 kB/s | 35 kB 00:00 2024-06-04T04:37:46.901 INFO:teuthology.orchestra.run.smithi112.stdout:(81/159): python3-babel-2.9.1-2.el9.noarch.rpm 3.2 MB/s | 6.0 MB 00:01 2024-06-04T04:37:47.010 INFO:teuthology.orchestra.run.smithi112.stdout:(82/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-06-04T04:37:47.102 INFO:teuthology.orchestra.run.smithi112.stdout:(83/159): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-06-04T04:37:47.152 INFO:teuthology.orchestra.run.smithi112.stdout:(84/159): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-06-04T04:37:47.194 INFO:teuthology.orchestra.run.smithi112.stdout:(85/159): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-06-04T04:37:47.319 INFO:teuthology.orchestra.run.smithi112.stdout:(86/159): python3-protobuf-3.14.0-13.el9.noarch 2.1 MB/s | 269 kB 00:00 2024-06-04T04:37:47.394 INFO:teuthology.orchestra.run.smithi112.stdout:(87/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-06-04T04:37:47.503 INFO:teuthology.orchestra.run.smithi112.stdout:(88/159): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-06-04T04:37:47.545 INFO:teuthology.orchestra.run.smithi112.stdout:(89/159): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-06-04T04:37:47.595 INFO:teuthology.orchestra.run.smithi112.stdout:(90/159): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-06-04T04:37:48.354 INFO:teuthology.orchestra.run.smithi112.stdout:(91/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.1 MB/s | 5.1 MB 00:01 2024-06-04T04:37:48.404 INFO:teuthology.orchestra.run.smithi112.stdout:(92/159): python3-toml-0.10.2-6.el9.noarch.rpm 837 kB/s | 42 kB 00:00 2024-06-04T04:37:48.455 INFO:teuthology.orchestra.run.smithi112.stdout:(93/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 887 kB/s | 44 kB 00:00 2024-06-04T04:37:48.563 INFO:teuthology.orchestra.run.smithi112.stdout:(94/159): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-06-04T04:37:48.613 INFO:teuthology.orchestra.run.smithi112.stdout:(95/159): qatlib-service-24.02.0-1.el9.x86_64.r 717 kB/s | 36 kB 00:00 2024-06-04T04:37:48.656 INFO:teuthology.orchestra.run.smithi112.stdout:(96/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-06-04T04:37:48.764 INFO:teuthology.orchestra.run.smithi112.stdout:(97/159): socat-1.7.4.1-5.el9.x86_64.rpm 2.8 MB/s | 305 kB 00:00 2024-06-04T04:37:48.814 INFO:teuthology.orchestra.run.smithi112.stdout:(98/159): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-06-04T04:37:48.881 INFO:teuthology.orchestra.run.smithi112.stdout:(99/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.4 MB/s | 94 kB 00:00 2024-06-04T04:37:48.940 INFO:teuthology.orchestra.run.smithi112.stdout:(100/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-06-04T04:37:48.982 INFO:teuthology.orchestra.run.smithi112.stdout:(101/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-06-04T04:37:52.382 INFO:teuthology.orchestra.run.smithi112.stdout:(102/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.4 MB/s | 16 MB 00:04 2024-06-04T04:37:58.105 INFO:teuthology.orchestra.run.smithi112.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:37:58.217 INFO:teuthology.orchestra.run.smithi112.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Status code: 404 for https://mirror.grid.uchicago.edu/pub/linux/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm (IP: 192.170.231.249) 2024-06-04T04:37:58.274 INFO:teuthology.orchestra.run.smithi112.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Status code: 404 for http://mirror.grid.uchicago.edu/pub/linux/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm (IP: 192.170.231.249) 2024-06-04T04:37:59.507 INFO:teuthology.orchestra.run.smithi112.stdout:(103/159): lua-devel-5.4.4-4.el9.x86_64.rpm 2.1 kB/s | 22 kB 00:10 2024-06-04T04:38:00.632 INFO:teuthology.orchestra.run.smithi112.stdout:(104/159): abseil-cpp-20211102.0-3.el9.x86_64.r 487 kB/s | 548 kB 00:01 2024-06-04T04:38:01.274 INFO:teuthology.orchestra.run.smithi112.stdout:(105/159): gperftools-libs-2.9.1-2.el9.x86_64.r 481 kB/s | 309 kB 00:00 2024-06-04T04:38:01.332 INFO:teuthology.orchestra.run.smithi112.stdout:(106/159): grpc-data-1.46.7-10.el9.noarch.rpm 333 kB/s | 19 kB 00:00 2024-06-04T04:38:07.152 INFO:teuthology.orchestra.run.smithi112.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:38:07.374 INFO:teuthology.orchestra.run.smithi112.stdout:(107/159): libquadmath-11.4.1-3.el9.x86_64.rpm 6.1 kB/s | 195 kB 00:32 2024-06-04T04:38:07.466 INFO:teuthology.orchestra.run.smithi112.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-06-04T04:38:07.649 INFO:teuthology.orchestra.run.smithi112.stdout:(109/159): liboath-2.6.7-2.el9.x86_64.rpm 265 kB/s | 49 kB 00:00 2024-06-04T04:38:08.049 INFO:teuthology.orchestra.run.smithi112.stdout:(110/159): libunwind-1.6.2-1.el9.x86_64.rpm 168 kB/s | 67 kB 00:00 2024-06-04T04:38:08.550 INFO:teuthology.orchestra.run.smithi112.stdout:(111/159): luarocks-3.9.2-5.el9.noarch.rpm 303 kB/s | 151 kB 00:00 2024-06-04T04:38:10.133 INFO:teuthology.orchestra.run.smithi112.stdout:(112/159): libarrow-9.0.0-11.el9.x86_64.rpm 514 kB/s | 4.4 MB 00:08 2024-06-04T04:38:10.933 INFO:teuthology.orchestra.run.smithi112.stdout:(113/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 352 kB/s | 839 kB 00:02 2024-06-04T04:38:11.050 INFO:teuthology.orchestra.run.smithi112.stdout:(114/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 373 kB/s | 43 kB 00:00 2024-06-04T04:38:11.159 INFO:teuthology.orchestra.run.smithi112.stdout:(115/159): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-06-04T04:38:11.217 INFO:teuthology.orchestra.run.smithi112.stdout:(116/159): python3-certifi-2021.10.8-2.el9.noar 257 kB/s | 15 kB 00:00 2024-06-04T04:38:11.318 INFO:teuthology.orchestra.run.smithi112.stdout:(117/159): python3-asyncssh-2.13.2-5.el9.noarch 463 kB/s | 548 kB 00:01 2024-06-04T04:38:11.651 INFO:teuthology.orchestra.run.smithi112.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 396 kB/s | 172 kB 00:00 2024-06-04T04:38:12.085 INFO:teuthology.orchestra.run.smithi112.stdout:(119/159): python3-cherrypy-18.6.1-2.el9.noarch 467 kB/s | 358 kB 00:00 2024-06-04T04:38:12.393 INFO:teuthology.orchestra.run.smithi112.stdout:(120/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 409 kB/s | 303 kB 00:00 2024-06-04T04:38:12.535 INFO:teuthology.orchestra.run.smithi112.stdout:(121/159): python3-google-auth-2.29.0-1.el9.noa 456 kB/s | 206 kB 00:00 2024-06-04T04:38:12.844 INFO:teuthology.orchestra.run.smithi112.stdout:(122/159): python3-grpcio-tools-1.46.7-10.el9.x 469 kB/s | 144 kB 00:00 2024-06-04T04:38:12.894 INFO:teuthology.orchestra.run.smithi112.stdout:(123/159): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-06-04T04:38:12.953 INFO:teuthology.orchestra.run.smithi112.stdout:(124/159): python3-jaraco-classes-3.2.1-5.el9.n 303 kB/s | 18 kB 00:00 2024-06-04T04:38:13.020 INFO:teuthology.orchestra.run.smithi112.stdout:(125/159): python3-jaraco-collections-3.0.0-8.e 348 kB/s | 23 kB 00:00 2024-06-04T04:38:13.078 INFO:teuthology.orchestra.run.smithi112.stdout:(126/159): python3-jaraco-functools-3.5.0-2.el9 333 kB/s | 19 kB 00:00 2024-06-04T04:38:13.145 INFO:teuthology.orchestra.run.smithi112.stdout:(127/159): python3-jaraco-text-3.2.0-6.el9.noar 295 kB/s | 20 kB 00:00 2024-06-04T04:38:14.862 INFO:teuthology.orchestra.run.smithi112.stdout:(128/159): python3-kubernetes-26.1.0-2.el9.noar 609 kB/s | 1.0 MB 00:01 2024-06-04T04:38:14.987 INFO:teuthology.orchestra.run.smithi112.stdout:(129/159): python3-logutils-0.3.5-21.el9.noarch 371 kB/s | 46 kB 00:00 2024-06-04T04:38:15.171 INFO:teuthology.orchestra.run.smithi112.stdout:(130/159): python3-more-itertools-8.12.0-2.el9. 430 kB/s | 79 kB 00:00 2024-06-04T04:38:15.296 INFO:teuthology.orchestra.run.smithi112.stdout:(131/159): python3-natsort-7.1.1-5.el9.noarch.r 462 kB/s | 58 kB 00:00 2024-06-04T04:38:15.822 INFO:teuthology.orchestra.run.smithi112.stdout:(132/159): python3-pecan-1.4.2-3.el9.noarch.rpm 519 kB/s | 272 kB 00:00 2024-06-04T04:38:15.880 INFO:teuthology.orchestra.run.smithi112.stdout:(133/159): python3-portend-3.1.0-2.el9.noarch.r 282 kB/s | 16 kB 00:00 2024-06-04T04:38:16.080 INFO:teuthology.orchestra.run.smithi112.stdout:(134/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 449 kB/s | 90 kB 00:00 2024-06-04T04:38:16.164 INFO:teuthology.orchestra.run.smithi112.stdout:(135/159): python3-pyperclip-1.8.2-2.el9.noarch 339 kB/s | 28 kB 00:00 2024-06-04T04:38:16.256 INFO:teuthology.orchestra.run.smithi112.stdout:(136/159): python3-repoze-lru-0.7-16.el9.noarch 336 kB/s | 31 kB 00:00 2024-06-04T04:38:16.431 INFO:teuthology.orchestra.run.smithi112.stdout:(137/159): python3-grpcio-1.46.7-10.el9.x86_64. 517 kB/s | 2.0 MB 00:04 2024-06-04T04:38:16.557 INFO:teuthology.orchestra.run.smithi112.stdout:(138/159): python3-rsa-4.9-2.el9.noarch.rpm 473 kB/s | 59 kB 00:00 2024-06-04T04:38:16.649 INFO:teuthology.orchestra.run.smithi112.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 481 kB/s | 188 kB 00:00 2024-06-04T04:38:16.674 INFO:teuthology.orchestra.run.smithi112.stdout:(140/159): python3-tempora-5.0.0-2.el9.noarch.r 306 kB/s | 36 kB 00:00 2024-06-04T04:38:16.774 INFO:teuthology.orchestra.run.smithi112.stdout:(141/159): python3-typing-extensions-4.4.0-2.el 407 kB/s | 51 kB 00:00 2024-06-04T04:38:17.007 INFO:teuthology.orchestra.run.smithi112.stdout:(142/159): python3-websocket-client-1.2.3-2.el9 385 kB/s | 90 kB 00:00 2024-06-04T04:38:17.150 INFO:teuthology.orchestra.run.smithi112.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 483 kB/s | 230 kB 00:00 2024-06-04T04:38:17.225 INFO:teuthology.orchestra.run.smithi112.stdout:(144/159): python3-zc-lockfile-2.0-10.el9.noarc 266 kB/s | 20 kB 00:00 2024-06-04T04:38:17.634 INFO:teuthology.orchestra.run.smithi112.stdout:(145/159): re2-20211101-3.el9.x86_64.rpm 469 kB/s | 192 kB 00:00 2024-06-04T04:38:17.834 INFO:teuthology.orchestra.run.smithi112.stdout:(146/159): python3-werkzeug-2.0.3-3.el9.1.noarc 517 kB/s | 427 kB 00:00 2024-06-04T04:38:18.134 INFO:teuthology.orchestra.run.smithi112.stdout:(147/159): librados2-19.0.0-3882.gb7328fed.el9. 11 MB/s | 3.4 MB 00:00 2024-06-04T04:38:18.343 INFO:teuthology.orchestra.run.smithi112.stdout:(148/159): librbd1-19.0.0-3882.gb7328fed.el9.x8 15 MB/s | 3.1 MB 00:00 2024-06-04T04:38:19.076 INFO:teuthology.orchestra.run.smithi112.stdout:(149/159): NetworkManager-1.47.91-1.el9.x86_64. 3.2 MB/s | 2.3 MB 00:00 2024-06-04T04:38:20.043 INFO:teuthology.orchestra.run.smithi112.stdout:(150/159): NetworkManager-libnm-1.47.91-1.el9.x 1.9 MB/s | 1.8 MB 00:00 2024-06-04T04:38:20.093 INFO:teuthology.orchestra.run.smithi112.stdout:(151/159): NetworkManager-team-1.47.91-1.el9.x8 830 kB/s | 41 kB 00:00 2024-06-04T04:38:20.244 INFO:teuthology.orchestra.run.smithi112.stdout:(152/159): NetworkManager-tui-1.47.91-1.el9.x86 1.6 MB/s | 248 kB 00:00 2024-06-04T04:38:20.302 INFO:teuthology.orchestra.run.smithi112.stdout:(153/159): elfutils-debuginfod-client-0.191-4.e 628 kB/s | 37 kB 00:00 2024-06-04T04:38:20.453 INFO:teuthology.orchestra.run.smithi112.stdout:(154/159): elfutils-libelf-0.191-4.el9.x86_64.r 1.3 MB/s | 207 kB 00:00 2024-06-04T04:38:20.636 INFO:teuthology.orchestra.run.smithi112.stdout:(155/159): elfutils-libs-0.191-4.el9.x86_64.rpm 1.4 MB/s | 260 kB 00:00 2024-06-04T04:38:20.878 INFO:teuthology.orchestra.run.smithi112.stdout:(156/159): thrift-0.15.0-2.el9.x86_64.rpm 499 kB/s | 1.6 MB 00:03 2024-06-04T04:38:21.378 INFO:teuthology.orchestra.run.smithi112.stdout:(157/159): openssl-3.2.1-1.el9.x86_64.rpm 1.8 MB/s | 1.3 MB 00:00 2024-06-04T04:38:21.520 INFO:teuthology.orchestra.run.smithi112.stdout:(158/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 3.7 MB/s | 2.4 MB 00:00 2024-06-04T04:38:25.433 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 42 kB/s | 6.2 MB 02:31 2024-06-04T04:38:32.053 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 221 B/s | 17 kB 01:16 2024-06-04T04:39:01.515 INFO:teuthology.orchestra.run.smithi112.stdout:[MIRROR] protobuf-compiler-3.14.0-13.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-13.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:39:01.545 INFO:teuthology.orchestra.run.smithi112.stdout:[MIRROR] protobuf-compiler-3.14.0-13.el9.x86_64.rpm: Status code: 404 for https://mirror.grid.uchicago.edu/pub/linux/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-13.el9.x86_64.rpm (IP: 192.170.231.249) 2024-06-04T04:39:01.651 INFO:teuthology.orchestra.run.smithi112.stdout:[MIRROR] protobuf-compiler-3.14.0-13.el9.x86_64.rpm: Status code: 404 for http://mirror.grid.uchicago.edu/pub/linux/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-13.el9.x86_64.rpm (IP: 192.170.231.249) 2024-06-04T04:39:03.399 INFO:teuthology.orchestra.run.smithi112.stdout:(159/159): protobuf-compiler-3.14.0-13.el9.x86_ 12 kB/s | 863 kB 01:11 2024-06-04T04:39:03.402 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-06-04T04:39:03.402 INFO:teuthology.orchestra.run.smithi112.stdout:Total 2.4 MB/s | 227 MB 01:35 2024-06-04T04:39:04.573 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-06-04T04:39:04.730 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-06-04T04:39:04.730 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-06-04T04:39:06.984 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-06-04T04:39:06.984 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-06-04T04:39:13.199 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-06-04T04:39:13.279 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-06-04T04:39:13.316 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-06-04T04:39:13.371 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-06-04T04:39:13.404 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-06-04T04:39:13.429 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-06-04T04:39:13.489 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-06-04T04:39:13.819 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-06-04T04:39:13.930 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librados2-2:19.0.0-3882.gb7328fed.el9.x86_64 8/170 2024-06-04T04:39:14.053 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:19.0.0-3882.gb7328fed.el9.x86_64 8/170 2024-06-04T04:39:14.081 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs2-2:19.0.0-3882.gb7328fed.el9.x86_64 9/170 2024-06-04T04:39:14.177 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs2-2:19.0.0-3882.gb7328fed.el9.x86_64 9/170 2024-06-04T04:39:14.203 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rados-2:19.0.0-3882.gb7328fed.el9.x86_64 10/170 2024-06-04T04:39:14.241 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-06-04T04:39:14.430 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-06-04T04:39:14.487 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-06-04T04:39:14.512 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephsqlite-2:19.0.0-3882.gb7328fed.el9.x86_64 14/170 2024-06-04T04:39:14.615 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3882.gb7328fed.el9.x86_64 14/170 2024-06-04T04:39:14.638 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libradosstriper1-2:19.0.0-3882.gb7328fed.el9.x86 15/170 2024-06-04T04:39:14.701 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3882.gb7328fed.el9.x86 15/170 2024-06-04T04:39:14.722 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-04T04:39:14.742 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-04T04:39:14.933 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-04T04:39:15.117 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-04T04:39:15.236 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-04T04:39:15.330 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-04T04:39:15.418 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-04T04:39:15.498 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-04T04:39:15.705 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-04T04:39:15.819 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-06-04T04:39:15.852 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-06-04T04:39:15.886 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-06-04T04:39:15.951 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-06-04T04:39:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-06-04T04:39:16.037 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-06-04T04:39:16.070 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-06-04T04:39:16.097 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-06-04T04:39:16.174 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-06-04T04:39:16.227 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-06-04T04:39:16.268 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-06-04T04:39:16.311 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-06-04T04:39:16.386 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-06-04T04:39:16.422 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-common-2:19.0.0-3882.gb7328fed.el9. 33/170 2024-06-04T04:39:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-argparse-2:19.0.0-3882.gb7328fed.el 34/170 2024-06-04T04:39:16.494 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cephfs-2:19.0.0-3882.gb7328fed.el9.x86_6 35/170 2024-06-04T04:39:16.570 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-06-04T04:39:16.625 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-06-04T04:39:16.698 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-06-04T04:39:16.833 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-06-04T04:39:16.886 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-06-04T04:39:17.385 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 481 kB/s | 21 MB 00:45 2024-06-04T04:39:17.410 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-06-04T04:39:17.489 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-06-04T04:39:17.519 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-06-04T04:39:17.556 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-06-04T04:39:17.604 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-06-04T04:39:17.642 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librados-devel-2:19.0.0-3882.gb7328fed.el9.x86_6 46/170 2024-06-04T04:39:17.684 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-06-04T04:39:17.834 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-06-04T04:39:17.887 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-06-04T04:39:17.977 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-06-04T04:39:18.034 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-06-04T04:39:18.106 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-06-04T04:39:18.138 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-06-04T04:39:18.174 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-06-04T04:39:18.230 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-06-04T04:39:18.265 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-06-04T04:39:18.308 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-06-04T04:39:18.351 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-06-04T04:39:18.485 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-06-04T04:39:18.522 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-06-04T04:39:18.560 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-06-04T04:39:18.654 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-06-04T04:39:18.694 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-06-04T04:39:18.736 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-06-04T04:39:18.768 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-06-04T04:39:18.792 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-06-04T04:39:18.894 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-06-04T04:39:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-06-04T04:39:19.061 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-06-04T04:39:19.153 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-06-04T04:39:19.326 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-06-04T04:39:19.501 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-06-04T04:39:19.560 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-06-04T04:39:19.590 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-06-04T04:39:19.624 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-06-04T04:39:19.657 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-06-04T04:39:19.699 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-06-04T04:39:19.733 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-06-04T04:39:19.820 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-06-04T04:39:19.872 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-06-04T04:39:19.903 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-06-04T04:39:19.950 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-06-04T04:39:20.048 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-06-04T04:39:20.113 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-06-04T04:39:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-06-04T04:39:20.868 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-06-04T04:39:20.902 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-06-04T04:39:21.002 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-06-04T04:39:21.058 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-06-04T04:39:21.102 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-06-04T04:39:21.200 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-06-04T04:39:21.230 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-06-04T04:39:21.405 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-06-04T04:39:21.436 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-06-04T04:39:21.513 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-06-04T04:39:22.146 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-06-04T04:39:22.258 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-06-04T04:39:22.524 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-06-04T04:39:23.485 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-06-04T04:39:23.565 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-06-04T04:39:23.592 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-06-04T04:39:23.626 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-06-04T04:39:23.936 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-06-04T04:39:23.981 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librbd1-2:19.0.0-3882.gb7328fed.el9.x86_64 103/170 2024-06-04T04:39:24.054 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:19.0.0-3882.gb7328fed.el9.x86_64 103/170 2024-06-04T04:39:24.086 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rbd-2:19.0.0-3882.gb7328fed.el9.x86_64 104/170 2024-06-04T04:39:24.122 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-06-04T04:39:24.599 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-06-04T04:39:24.625 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librgw2-2:19.0.0-3882.gb7328fed.el9.x86_64 107/170 2024-06-04T04:39:24.696 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librgw2-2:19.0.0-3882.gb7328fed.el9.x86_64 107/170 2024-06-04T04:39:24.719 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rgw-2:19.0.0-3882.gb7328fed.el9.x86_64 108/170 2024-06-04T04:39:24.867 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-04T04:39:24.868 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-04T04:39:24.879 INFO:teuthology.orchestra.run.smithi098.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-06-04T04:39:24.879 INFO:teuthology.orchestra.run.smithi098.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-06-04T04:39:24.880 INFO:teuthology.orchestra.run.smithi098.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-06-04T04:39:24.881 INFO:teuthology.orchestra.run.smithi098.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-06-04T04:39:24.883 INFO:teuthology.orchestra.run.smithi098.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-06-04T04:39:24.885 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-06-04T04:39:24.886 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-06-04T04:39:24.948 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-04T04:39:24.966 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-06-04T04:39:24.966 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-06-04T04:39:24.966 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-06-04T04:39:24.966 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-04T04:39:24.966 INFO:teuthology.orchestra.run.smithi098.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-06-04T04:39:24.966 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 6.6 k 2024-06-04T04:39:24.966 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 5.4 M 2024-06-04T04:39:24.966 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 890 k 2024-06-04T04:39:24.966 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 145 k 2024-06-04T04:39:24.967 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 1.5 M 2024-06-04T04:39:24.967 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 143 k 2024-06-04T04:39:24.967 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 3.5 M 2024-06-04T04:39:24.967 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 7.4 M 2024-06-04T04:39:24.967 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 49 k 2024-06-04T04:39:24.967 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 11 M 2024-06-04T04:39:24.967 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 49 M 2024-06-04T04:39:24.967 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 273 k 2024-06-04T04:39:24.967 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 763 k 2024-06-04T04:39:24.967 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 225 k 2024-06-04T04:39:24.967 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 27 k 2024-06-04T04:39:24.968 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 18 k 2024-06-04T04:39:24.968 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-06-04T04:39:24.968 INFO:teuthology.orchestra.run.smithi098.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-06-04T04:39:24.968 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-06-04T04:39:24.968 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 32 k 2024-06-04T04:39:24.968 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 747 k 2024-06-04T04:39:24.968 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 127 k 2024-06-04T04:39:24.968 INFO:teuthology.orchestra.run.smithi098.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-06-04T04:39:24.968 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-06-04T04:39:24.968 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-06-04T04:39:24.968 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-06-04T04:39:24.969 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-06-04T04:39:24.969 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-06-04T04:39:24.969 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 161 k 2024-06-04T04:39:24.969 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 320 k 2024-06-04T04:39:24.969 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 300 k 2024-06-04T04:39:24.969 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 100 k 2024-06-04T04:39:24.969 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 85 k 2024-06-04T04:39:24.969 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 3.1 M 2024-06-04T04:39:24.969 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 170 k 2024-06-04T04:39:24.969 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-06-04T04:39:24.969 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-06-04T04:39:24.970 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager x86_64 1:1.47.91-1.el9 baseos 2.3 M 2024-06-04T04:39:24.970 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm x86_64 1:1.47.91-1.el9 baseos 1.8 M 2024-06-04T04:39:24.970 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team x86_64 1:1.47.91-1.el9 baseos 41 k 2024-06-04T04:39:24.970 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui x86_64 1:1.47.91-1.el9 baseos 248 k 2024-06-04T04:39:24.970 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-06-04T04:39:24.970 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-06-04T04:39:24.970 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-06-04T04:39:24.970 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 3.4 M 2024-06-04T04:39:24.970 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 3.1 M 2024-06-04T04:39:24.970 INFO:teuthology.orchestra.run.smithi098.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-06-04T04:39:24.970 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-06-04T04:39:24.971 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-06-04T04:39:24.971 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-04T04:39:24.971 INFO:teuthology.orchestra.run.smithi098.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-06-04T04:39:24.971 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-04T04:39:24.971 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 21 M 2024-06-04T04:39:24.971 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 34 k 2024-06-04T04:39:24.971 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 2.4 M 2024-06-04T04:39:24.971 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 274 k 2024-06-04T04:39:24.971 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 4.7 M 2024-06-04T04:39:24.971 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 17 M 2024-06-04T04:39:24.971 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 16 k 2024-06-04T04:39:24.971 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 25 k 2024-06-04T04:39:24.972 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-04T04:39:24.972 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-04T04:39:24.972 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-04T04:39:24.972 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-04T04:39:24.972 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-04T04:39:24.972 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-04T04:39:24.972 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-04T04:39:24.972 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-04T04:39:24.972 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 163 k 2024-06-04T04:39:24.972 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-04T04:39:24.972 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-04T04:39:24.973 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-04T04:39:24.973 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-04T04:39:24.973 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-04T04:39:24.973 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 501 k 2024-06-04T04:39:24.973 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-04T04:39:24.973 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 5.4 M 2024-06-04T04:39:24.973 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-04T04:39:24.973 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-06-04T04:39:24.973 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-04T04:39:24.973 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-04T04:39:24.973 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-04T04:39:24.973 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-04T04:39:24.973 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-04T04:39:24.973 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-04T04:39:24.974 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-04T04:39:24.974 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-06-04T04:39:24.974 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-04T04:39:24.974 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-04T04:39:24.974 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-04T04:39:24.974 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-06-04T04:39:24.974 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-04T04:39:24.974 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-04T04:39:24.974 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-04T04:39:24.974 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 45 k 2024-06-04T04:39:24.974 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 135 k 2024-06-04T04:39:24.974 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-04T04:39:24.975 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-04T04:39:24.975 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-04T04:39:24.975 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-04T04:39:24.975 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-06-04T04:39:24.975 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-06-04T04:39:24.975 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-04T04:39:24.975 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-04T04:39:24.975 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-06-04T04:39:24.975 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-04T04:39:24.975 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-04T04:39:24.975 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-04T04:39:24.976 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-04T04:39:24.976 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-04T04:39:24.976 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-04T04:39:24.976 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-04T04:39:24.976 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-04T04:39:24.976 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-04T04:39:24.976 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-04T04:39:24.976 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-04T04:39:24.976 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-04T04:39:24.976 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-04T04:39:24.976 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-04T04:39:24.976 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-04T04:39:24.976 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-04T04:39:24.976 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-04T04:39:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-04T04:39:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-04T04:39:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-04T04:39:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-04T04:39:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-04T04:39:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-04T04:39:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-04T04:39:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-04T04:39:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-04T04:39:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-04T04:39:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-04T04:39:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-04T04:39:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-06-04T04:39:24.978 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-04T04:39:24.978 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-04T04:39:24.978 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-04T04:39:24.978 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-04T04:39:24.978 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-04T04:39:24.978 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-04T04:39:24.978 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-04T04:39:24.978 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-04T04:39:24.978 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-04T04:39:24.978 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-04T04:39:24.978 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-04T04:39:24.978 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-06-04T04:39:24.978 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-04T04:39:24.979 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-04T04:39:24.979 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-04T04:39:24.979 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-04T04:39:24.979 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-04T04:39:24.979 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-04T04:39:24.979 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-04T04:39:24.979 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-04T04:39:24.979 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-04T04:39:24.979 INFO:teuthology.orchestra.run.smithi098.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-06-04T04:39:24.979 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-04T04:39:24.979 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-06-04T04:39:24.979 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-06-04T04:39:24.979 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-04T04:39:24.979 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:39:24.980 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-04T04:39:24.980 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-06-04T04:39:24.980 INFO:teuthology.orchestra.run.smithi098.stdout:Install 148 Packages 2024-06-04T04:39:24.980 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 11 Packages 2024-06-04T04:39:24.980 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:39:24.980 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 227 M 2024-06-04T04:39:24.980 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-04T04:39:25.659 INFO:teuthology.orchestra.run.smithi098.stdout:(1/159): ceph-19.0.0-3882.gb7328fed.el9.x86_64. 30 kB/s | 6.6 kB 00:00 2024-06-04T04:39:26.001 INFO:teuthology.orchestra.run.smithi098.stdout:(2/159): ceph-fuse-19.0.0-3882.gb7328fed.el9.x8 2.5 MB/s | 890 kB 00:00 2024-06-04T04:39:26.118 INFO:teuthology.orchestra.run.smithi098.stdout:(3/159): ceph-immutable-object-cache-19.0.0-388 1.2 MB/s | 145 kB 00:00 2024-06-04T04:39:26.230 INFO:teuthology.orchestra.run.smithi098.stdout:(4/159): ceph-base-19.0.0-3882.gb7328fed.el9.x8 6.8 MB/s | 5.4 MB 00:00 2024-06-04T04:39:26.468 INFO:teuthology.orchestra.run.smithi098.stdout:(5/159): ceph-mds-19.0.0-3882.gb7328fed.el9.x86 6.8 MB/s | 2.4 MB 00:00 2024-06-04T04:39:26.994 INFO:teuthology.orchestra.run.smithi098.stdout:(6/159): ceph-mgr-19.0.0-3882.gb7328fed.el9.x86 2.0 MB/s | 1.5 MB 00:00 2024-06-04T04:39:27.007 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 109/170 2024-06-04T04:39:27.040 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 109/170 2024-06-04T04:39:27.061 INFO:teuthology.orchestra.run.smithi098.stdout:(7/159): ceph-mon-19.0.0-3882.gb7328fed.el9.x86 7.9 MB/s | 4.7 MB 00:00 2024-06-04T04:39:27.403 INFO:teuthology.orchestra.run.smithi098.stdout:(8/159): ceph-common-19.0.0-3882.gb7328fed.el9. 11 MB/s | 21 MB 00:01 2024-06-04T04:39:27.461 INFO:teuthology.orchestra.run.smithi098.stdout:(9/159): ceph-selinux-19.0.0-3882.gb7328fed.el9 432 kB/s | 25 kB 00:00 2024-06-04T04:39:27.584 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 109/170 2024-06-04T04:39:27.619 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-base-2:19.0.0-3882.gb7328fed.el9.x86_64 110/170 2024-06-04T04:39:27.710 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-base-2:19.0.0-3882.gb7328fed.el9.x86_64 110/170 2024-06-04T04:39:27.710 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-04T04:39:27.710 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-04T04:39:27.710 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:39:27.737 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-selinux-2:19.0.0-3882.gb7328fed.el9.x86_64 111/170 2024-06-04T04:39:28.453 INFO:teuthology.orchestra.run.smithi098.stdout:(10/159): ceph-radosgw-19.0.0-3882.gb7328fed.el 7.7 MB/s | 11 MB 00:01 2024-06-04T04:39:28.553 INFO:teuthology.orchestra.run.smithi098.stdout:(11/159): cephfs-mirror-19.0.0-3882.gb7328fed.e 2.2 MB/s | 225 kB 00:00 2024-06-04T04:39:28.637 INFO:teuthology.orchestra.run.smithi098.stdout:(12/159): cephfs-shell-19.0.0-3882.gb7328fed.el 325 kB/s | 27 kB 00:00 2024-06-04T04:39:28.721 INFO:teuthology.orchestra.run.smithi098.stdout:(13/159): libcephfs-devel-19.0.0-3882.gb7328fed 381 kB/s | 32 kB 00:00 2024-06-04T04:39:28.904 INFO:teuthology.orchestra.run.smithi098.stdout:(14/159): libcephfs2-19.0.0-3882.gb7328fed.el9. 4.0 MB/s | 747 kB 00:00 2024-06-04T04:39:29.013 INFO:teuthology.orchestra.run.smithi098.stdout:(15/159): libcephsqlite-19.0.0-3882.gb7328fed.e 1.5 MB/s | 163 kB 00:00 2024-06-04T04:39:29.121 INFO:teuthology.orchestra.run.smithi098.stdout:(16/159): librados-devel-19.0.0-3882.gb7328fed. 1.1 MB/s | 127 kB 00:00 2024-06-04T04:39:29.569 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 271 B/s | 17 kB 01:02 2024-06-04T04:39:29.930 INFO:teuthology.orchestra.run.smithi098.stdout:(17/159): libradosstriper1-19.0.0-3882.gb7328fe 619 kB/s | 501 kB 00:00 2024-06-04T04:39:30.139 INFO:teuthology.orchestra.run.smithi098.stdout:(18/159): ceph-osd-19.0.0-3882.gb7328fed.el9.x8 5.4 MB/s | 17 MB 00:03 2024-06-04T04:39:30.214 INFO:teuthology.orchestra.run.smithi098.stdout:(19/159): python3-ceph-argparse-19.0.0-3882.gb7 603 kB/s | 45 kB 00:00 2024-06-04T04:39:30.289 INFO:teuthology.orchestra.run.smithi098.stdout:(20/159): python3-ceph-common-19.0.0-3882.gb732 1.8 MB/s | 135 kB 00:00 2024-06-04T04:39:30.381 INFO:teuthology.orchestra.run.smithi098.stdout:(21/159): python3-cephfs-19.0.0-3882.gb7328fed. 1.7 MB/s | 161 kB 00:00 2024-06-04T04:39:30.473 INFO:teuthology.orchestra.run.smithi098.stdout:(22/159): python3-rados-19.0.0-3882.gb7328fed.e 3.4 MB/s | 320 kB 00:00 2024-06-04T04:39:30.924 INFO:teuthology.orchestra.run.smithi098.stdout:(23/159): ceph-test-19.0.0-3882.gb7328fed.el9.x 14 MB/s | 49 MB 00:03 2024-06-04T04:39:30.949 INFO:teuthology.orchestra.run.smithi098.stdout:(24/159): python3-rbd-19.0.0-3882.gb7328fed.el9 630 kB/s | 300 kB 00:00 2024-06-04T04:39:31.025 INFO:teuthology.orchestra.run.smithi098.stdout:(25/159): librgw2-19.0.0-3882.gb7328fed.el9.x86 4.9 MB/s | 5.4 MB 00:01 2024-06-04T04:39:31.050 INFO:teuthology.orchestra.run.smithi098.stdout:(26/159): rbd-fuse-19.0.0-3882.gb7328fed.el9.x8 850 kB/s | 85 kB 00:00 2024-06-04T04:39:31.075 INFO:teuthology.orchestra.run.smithi098.stdout:(27/159): python3-rgw-19.0.0-3882.gb7328fed.el9 661 kB/s | 100 kB 00:00 2024-06-04T04:39:31.125 INFO:teuthology.orchestra.run.smithi098.stdout:(28/159): rbd-nbd-19.0.0-3882.gb7328fed.el9.x86 2.2 MB/s | 170 kB 00:00 2024-06-04T04:39:31.142 INFO:teuthology.orchestra.run.smithi098.stdout:(29/159): ceph-grafana-dashboards-19.0.0-3882.g 512 kB/s | 34 kB 00:00 2024-06-04T04:39:31.201 INFO:teuthology.orchestra.run.smithi098.stdout:(30/159): ceph-mgr-cephadm-19.0.0-3882.gb7328fe 1.9 MB/s | 143 kB 00:00 2024-06-04T04:39:31.460 INFO:teuthology.orchestra.run.smithi098.stdout:(31/159): ceph-mgr-dashboard-19.0.0-3882.gb7328 11 MB/s | 3.5 MB 00:00 2024-06-04T04:39:31.535 INFO:teuthology.orchestra.run.smithi098.stdout:(32/159): rbd-mirror-19.0.0-3882.gb7328fed.el9. 6.1 MB/s | 3.1 MB 00:00 2024-06-04T04:39:31.560 INFO:teuthology.orchestra.run.smithi098.stdout:(33/159): ceph-mgr-modules-core-19.0.0-3882.gb7 2.7 MB/s | 274 kB 00:00 2024-06-04T04:39:31.585 INFO:teuthology.orchestra.run.smithi098.stdout:(34/159): ceph-mgr-rook-19.0.0-3882.gb7328fed.e 981 kB/s | 49 kB 00:00 2024-06-04T04:39:31.611 INFO:teuthology.orchestra.run.smithi098.stdout:(35/159): ceph-prometheus-alerts-19.0.0-3882.gb 328 kB/s | 16 kB 00:00 2024-06-04T04:39:31.653 INFO:teuthology.orchestra.run.smithi098.stdout:(36/159): ceph-volume-19.0.0-3882.gb7328fed.el9 4.0 MB/s | 273 kB 00:00 2024-06-04T04:39:31.711 INFO:teuthology.orchestra.run.smithi098.stdout:(37/159): cephadm-19.0.0-3882.gb7328fed.el9.noa 7.4 MB/s | 763 kB 00:00 2024-06-04T04:39:31.728 INFO:teuthology.orchestra.run.smithi098.stdout:(38/159): cephfs-top-19.0.0-3882.gb7328fed.el9. 242 kB/s | 18 kB 00:00 2024-06-04T04:39:32.054 INFO:teuthology.orchestra.run.smithi098.stdout:(39/159): ceph-mgr-diskprediction-local-19.0.0- 8.7 MB/s | 7.4 MB 00:00 2024-06-04T04:39:38.644 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:39:38.757 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://mirror.grid.uchicago.edu/pub/linux/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 192.170.231.249) 2024-06-04T04:39:38.811 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for http://mirror.grid.uchicago.edu/pub/linux/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 192.170.231.249) 2024-06-04T04:39:39.053 INFO:teuthology.orchestra.run.smithi098.stdout:(40/159): libquadmath-11.4.1-3.el9.x86_64.rpm 28 kB/s | 195 kB 00:06 2024-06-04T04:39:39.056 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:39:39.084 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 404 for https://mirror.grid.uchicago.edu/pub/linux/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 192.170.231.249) 2024-06-04T04:39:39.111 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 404 for http://mirror.grid.uchicago.edu/pub/linux/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 192.170.231.249) 2024-06-04T04:39:39.218 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3882.gb7328fed.el9.x86_64 111/170 2024-06-04T04:39:39.218 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /sys 2024-06-04T04:39:39.218 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /proc 2024-06-04T04:39:39.218 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /mnt 2024-06-04T04:39:39.218 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /var/tmp 2024-06-04T04:39:39.218 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /home 2024-06-04T04:39:39.218 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /root 2024-06-04T04:39:39.218 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /tmp 2024-06-04T04:39:39.219 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:39:39.228 INFO:teuthology.orchestra.run.smithi098.stdout:(41/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-06-04T04:39:39.231 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-cryptography-36.0.1-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cryptography-36.0.1-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:39:39.259 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-cryptography-36.0.1-4.el9.x86_64.rpm: Status code: 404 for https://mirror.grid.uchicago.edu/pub/linux/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cryptography-36.0.1-4.el9.x86_64.rpm (IP: 192.170.231.249) 2024-06-04T04:39:39.287 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-cryptography-36.0.1-4.el9.x86_64.rpm: Status code: 404 for http://mirror.grid.uchicago.edu/pub/linux/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cryptography-36.0.1-4.el9.x86_64.rpm (IP: 192.170.231.249) 2024-06-04T04:39:39.383 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-volume-2:19.0.0-3882.gb7328fed.el9.noarch 112/170 2024-06-04T04:39:39.398 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-volume-2:19.0.0-3882.gb7328fed.el9.noarch 112/170 2024-06-04T04:39:39.398 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:39:39.399 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-04T04:39:39.399 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:39:39.579 INFO:teuthology.orchestra.run.smithi098.stdout:(42/159): python3-cryptography-36.0.1-4.el9.x86 3.5 MB/s | 1.2 MB 00:00 2024-06-04T04:39:39.606 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-ply-3.11-14.el9.noarch.rpm: Status code: 404 for https://mirror.grid.uchicago.edu/pub/linux/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-ply-3.11-14.el9.noarch.rpm (IP: 192.170.231.249) 2024-06-04T04:39:39.695 INFO:teuthology.orchestra.run.smithi098.stdout:(43/159): python3-ply-3.11-14.el9.noarch.rpm 913 kB/s | 106 kB 00:00 2024-06-04T04:39:39.821 INFO:teuthology.orchestra.run.smithi098.stdout:(44/159): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 135 kB 00:00 2024-06-04T04:39:39.904 INFO:teuthology.orchestra.run.smithi098.stdout:(45/159): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-06-04T04:39:40.055 INFO:teuthology.orchestra.run.smithi098.stdout:(46/159): python3-urllib3-1.26.5-5.el9.noarch.r 1.4 MB/s | 215 kB 00:00 2024-06-04T04:39:40.058 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] automake-1.16.2-8.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/automake-1.16.2-8.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-04T04:39:40.213 INFO:teuthology.orchestra.run.smithi098.stdout:(47/159): automake-1.16.2-8.el9.noarch.rpm 4.3 MB/s | 689 kB 00:00 2024-06-04T04:39:40.217 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] bison-3.7.4-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/bison-3.7.4-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:39:40.272 INFO:teuthology.orchestra.run.smithi098.stdout:(48/159): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 943 kB 00:00 2024-06-04T04:39:40.276 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] boost-program-options-1.75.0-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:39:40.322 INFO:teuthology.orchestra.run.smithi098.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-06-04T04:39:40.364 INFO:teuthology.orchestra.run.smithi098.stdout:(50/159): elfutils-libelf-devel-0.191-4.el9.x86 520 kB/s | 22 kB 00:00 2024-06-04T04:39:40.407 INFO:teuthology.orchestra.run.smithi098.stdout:(51/159): flex-2.6.4-9.el9.x86_64.rpm 7.3 MB/s | 313 kB 00:00 2024-06-04T04:39:40.448 INFO:teuthology.orchestra.run.smithi098.stdout:(52/159): flexiblas-3.0.4-8.el9.x86_64.rpm 785 kB/s | 33 kB 00:00 2024-06-04T04:39:40.549 INFO:teuthology.orchestra.run.smithi098.stdout:(53/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-06-04T04:39:40.582 INFO:teuthology.orchestra.run.smithi098.stdout:(54/159): flexiblas-openblas-openmp-3.0.4-8.el9 548 kB/s | 18 kB 00:00 2024-06-04T04:39:40.616 INFO:teuthology.orchestra.run.smithi098.stdout:(55/159): libaio-devel-0.3.111-13.el9.x86_64.rp 334 kB/s | 11 kB 00:00 2024-06-04T04:39:40.658 INFO:teuthology.orchestra.run.smithi098.stdout:(56/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 160 kB 00:00 2024-06-04T04:39:40.691 INFO:teuthology.orchestra.run.smithi098.stdout:(57/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-06-04T04:39:40.742 INFO:teuthology.orchestra.run.smithi098.stdout:(58/159): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-06-04T04:39:40.784 INFO:teuthology.orchestra.run.smithi098.stdout:(59/159): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 582 kB 00:00 2024-06-04T04:39:40.817 INFO:teuthology.orchestra.run.smithi098.stdout:(60/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 765 kB/s | 25 kB 00:00 2024-06-04T04:39:40.851 INFO:teuthology.orchestra.run.smithi098.stdout:(61/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-06-04T04:39:40.870 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-osd-2:19.0.0-3882.gb7328fed.el9.x86_64 113/170 2024-06-04T04:39:40.893 INFO:teuthology.orchestra.run.smithi098.stdout:(62/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.8 MB/s | 292 kB 00:00 2024-06-04T04:39:40.895 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-osd-2:19.0.0-3882.gb7328fed.el9.x86_64 113/170 2024-06-04T04:39:40.895 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:39:40.895 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-04T04:39:40.895 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-04T04:39:40.896 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-04T04:39:40.896 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:39:40.935 INFO:teuthology.orchestra.run.smithi098.stdout:(63/159): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 188 kB 00:00 2024-06-04T04:39:40.977 INFO:teuthology.orchestra.run.smithi098.stdout:(64/159): openblas-0.3.21-2.el9.x86_64.rpm 844 kB/s | 35 kB 00:00 2024-06-04T04:39:41.085 INFO:teuthology.orchestra.run.smithi098.stdout:(65/159): openblas-openmp-0.3.21-2.el9.x86_64.r 44 MB/s | 4.7 MB 00:00 2024-06-04T04:39:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mds-2:19.0.0-3882.gb7328fed.el9.x86_64 114/170 2024-06-04T04:39:41.144 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mds-2:19.0.0-3882.gb7328fed.el9.x86_64 114/170 2024-06-04T04:39:41.145 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:39:41.145 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-04T04:39:41.145 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-04T04:39:41.145 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-04T04:39:41.145 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:39:41.194 INFO:teuthology.orchestra.run.smithi098.stdout:(66/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-06-04T04:39:41.252 INFO:teuthology.orchestra.run.smithi098.stdout:(67/159): postgresql-13.14-1.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-06-04T04:39:41.303 INFO:teuthology.orchestra.run.smithi098.stdout:(68/159): postgresql-contrib-13.14-1.el9.x86_64 18 MB/s | 911 kB 00:00 2024-06-04T04:39:41.337 INFO:teuthology.orchestra.run.smithi098.stdout:(69/159): postgresql-private-libs-13.14-1.el9.x 3.9 MB/s | 133 kB 00:00 2024-06-04T04:39:41.462 INFO:teuthology.orchestra.run.smithi098.stdout:(70/159): postgresql-server-13.14-1.el9.x86_64. 46 MB/s | 5.8 MB 00:00 2024-06-04T04:39:41.512 INFO:teuthology.orchestra.run.smithi098.stdout:(71/159): protobuf-3.14.0-13.el9.x86_64.rpm 20 MB/s | 1.0 MB 00:00 2024-06-04T04:39:41.554 INFO:teuthology.orchestra.run.smithi098.stdout:(72/159): python3-attrs-20.3.0-7.el9.noarch.rpm 2.1 MB/s | 87 kB 00:00 2024-06-04T04:39:41.589 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mon-2:19.0.0-3882.gb7328fed.el9.x86_64 115/170 2024-06-04T04:39:41.612 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mon-2:19.0.0-3882.gb7328fed.el9.x86_64 115/170 2024-06-04T04:39:41.612 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:39:41.612 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-04T04:39:41.612 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-04T04:39:41.612 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-04T04:39:41.612 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:39:41.687 INFO:teuthology.orchestra.run.smithi098.stdout:(73/159): python3-babel-2.9.1-2.el9.noarch.rpm 45 MB/s | 6.0 MB 00:00 2024-06-04T04:39:41.719 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-06-04T04:39:41.729 INFO:teuthology.orchestra.run.smithi098.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-06-04T04:39:41.767 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-06-04T04:39:41.771 INFO:teuthology.orchestra.run.smithi098.stdout:(75/159): python3-devel-3.9.18-3.el9.x86_64.rpm 5.7 MB/s | 244 kB 00:00 2024-06-04T04:39:41.813 INFO:teuthology.orchestra.run.smithi098.stdout:(76/159): python3-jinja2-2.11.3-6.el9.noarch.rp 5.8 MB/s | 248 kB 00:00 2024-06-04T04:39:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-06-04T04:39:41.847 INFO:teuthology.orchestra.run.smithi098.stdout:(77/159): python3-jsonpatch-1.21-16.el9.noarch. 766 kB/s | 26 kB 00:00 2024-06-04T04:39:41.881 INFO:teuthology.orchestra.run.smithi098.stdout:(78/159): python3-jsonpointer-2.0-4.el9.noarch. 577 kB/s | 19 kB 00:00 2024-06-04T04:39:41.923 INFO:teuthology.orchestra.run.smithi098.stdout:(79/159): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-06-04T04:39:41.964 INFO:teuthology.orchestra.run.smithi098.stdout:(80/159): python3-markupsafe-1.1.1-12.el9.x86_6 837 kB/s | 35 kB 00:00 2024-06-04T04:39:42.007 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-06-04T04:39:42.065 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-06-04T04:39:42.090 INFO:teuthology.orchestra.run.smithi098.stdout:(81/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 5.1 MB 00:00 2024-06-04T04:39:42.132 INFO:teuthology.orchestra.run.smithi098.stdout:(82/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.9 MB/s | 250 kB 00:00 2024-06-04T04:39:42.141 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-06-04T04:39:42.174 INFO:teuthology.orchestra.run.smithi098.stdout:(83/159): python3-oauthlib-3.1.1-5.el9.noarch.r 5.2 MB/s | 222 kB 00:00 2024-06-04T04:39:42.216 INFO:teuthology.orchestra.run.smithi098.stdout:(84/159): python3-packaging-20.9-5.el9.noarch.r 1.8 MB/s | 77 kB 00:00 2024-06-04T04:39:42.230 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-06-04T04:39:42.258 INFO:teuthology.orchestra.run.smithi098.stdout:(85/159): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-06-04T04:39:42.300 INFO:teuthology.orchestra.run.smithi098.stdout:(86/159): python3-protobuf-3.14.0-13.el9.noarch 6.3 MB/s | 269 kB 00:00 2024-06-04T04:39:42.342 INFO:teuthology.orchestra.run.smithi098.stdout:(87/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.7 MB/s | 159 kB 00:00 2024-06-04T04:39:42.359 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-06-04T04:39:42.383 INFO:teuthology.orchestra.run.smithi098.stdout:(88/159): python3-pyasn1-modules-0.4.8-6.el9.no 6.6 MB/s | 279 kB 00:00 2024-06-04T04:39:42.415 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-06-04T04:39:42.425 INFO:teuthology.orchestra.run.smithi098.stdout:(89/159): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-06-04T04:39:42.456 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-06-04T04:39:42.467 INFO:teuthology.orchestra.run.smithi098.stdout:(90/159): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 54 kB 00:00 2024-06-04T04:39:42.484 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-06-04T04:39:42.785 INFO:teuthology.orchestra.run.smithi098.stdout:(91/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 51 MB/s | 16 MB 00:00 2024-06-04T04:39:42.826 INFO:teuthology.orchestra.run.smithi098.stdout:(92/159): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-06-04T04:39:42.868 INFO:teuthology.orchestra.run.smithi098.stdout:(93/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-06-04T04:39:42.910 INFO:teuthology.orchestra.run.smithi098.stdout:(94/159): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-06-04T04:39:42.946 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:19.0.0-3882.gb7328fed.el9.noarch 127/170 2024-06-04T04:39:42.952 INFO:teuthology.orchestra.run.smithi098.stdout:(95/159): qatlib-service-24.02.0-1.el9.x86_64.r 862 kB/s | 36 kB 00:00 2024-06-04T04:39:42.974 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephadm-2:19.0.0-3882.gb7328fed.el9.noarch 127/170 2024-06-04T04:39:42.994 INFO:teuthology.orchestra.run.smithi098.stdout:(96/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-06-04T04:39:43.006 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3882.gb7328fed.e 128/170 2024-06-04T04:39:43.036 INFO:teuthology.orchestra.run.smithi098.stdout:(97/159): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 305 kB 00:00 2024-06-04T04:39:43.067 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3882.gb7328fed. 129/170 2024-06-04T04:39:43.078 INFO:teuthology.orchestra.run.smithi098.stdout:(98/159): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-06-04T04:39:43.097 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3882.gb7328fed.el9.noa 130/170 2024-06-04T04:39:43.128 INFO:teuthology.orchestra.run.smithi098.stdout:(99/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-06-04T04:39:43.162 INFO:teuthology.orchestra.run.smithi098.stdout:(100/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-06-04T04:39:43.196 INFO:teuthology.orchestra.run.smithi098.stdout:(101/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-06-04T04:39:43.198 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:39:43.229 INFO:teuthology.orchestra.run.smithi098.stdout:(102/159): lua-devel-5.4.4-4.el9.x86_64.rpm 670 kB/s | 22 kB 00:00 2024-06-04T04:39:43.233 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] protobuf-compiler-3.14.0-13.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-13.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:39:43.288 INFO:teuthology.orchestra.run.smithi098.stdout:(103/159): protobuf-compiler-3.14.0-13.el9.x86_ 14 MB/s | 863 kB 00:00 2024-06-04T04:39:44.128 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3882.gb7328fed.el9.noa 130/170 2024-06-04T04:39:44.172 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3882.gb7328fed.el9.n 131/170 2024-06-04T04:39:44.196 INFO:teuthology.orchestra.run.smithi098.stdout:(104/159): abseil-cpp-20211102.0-3.el9.x86_64.r 603 kB/s | 548 kB 00:00 2024-06-04T04:39:44.730 INFO:teuthology.orchestra.run.smithi098.stdout:(105/159): gperftools-libs-2.9.1-2.el9.x86_64.r 579 kB/s | 309 kB 00:00 2024-06-04T04:39:44.780 INFO:teuthology.orchestra.run.smithi098.stdout:(106/159): grpc-data-1.46.7-10.el9.noarch.rpm 389 kB/s | 19 kB 00:00 2024-06-04T04:39:45.127 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3882.gb7328fed.el9.n 131/170 2024-06-04T04:39:45.158 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3882.gb73 132/170 2024-06-04T04:39:45.266 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3882.gb73 132/170 2024-06-04T04:39:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3882.gb7328fed.el 133/170 2024-06-04T04:39:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-2:19.0.0-3882.gb7328fed.el9.x86_64 134/170 2024-06-04T04:39:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3882.gb7328fed.el9.x86_64 134/170 2024-06-04T04:39:45.483 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:39:45.483 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-04T04:39:45.483 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-04T04:39:45.484 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-04T04:39:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:39:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-rook-2:19.0.0-3882.gb7328fed.el9.noarch 135/170 2024-06-04T04:39:45.538 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3882.gb7328fed.el9.noarch 135/170 2024-06-04T04:39:45.635 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-2:19.0.0-3882.gb7328fed.el9.x86_64 136/170 2024-06-04T04:39:45.689 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 137/170 2024-06-04T04:39:46.637 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-06-04T04:39:46.671 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-radosgw-2:19.0.0-3882.gb7328fed.el9.x86_64 139/170 2024-06-04T04:39:46.695 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3882.gb7328fed.el9.x86_64 139/170 2024-06-04T04:39:46.695 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:39:46.695 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-04T04:39:46.695 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-04T04:39:46.695 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-04T04:39:46.695 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:39:46.738 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3882.gb7328 140/170 2024-06-04T04:39:46.761 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3882.gb7328 140/170 2024-06-04T04:39:46.761 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:39:46.761 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-04T04:39:46.761 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:39:46.814 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephfs-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 141/170 2024-06-04T04:39:46.837 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 141/170 2024-06-04T04:39:46.837 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:39:46.837 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-06-04T04:39:46.837 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-04T04:39:46.837 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-04T04:39:46.837 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:39:47.140 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 142/170 2024-06-04T04:39:47.163 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 142/170 2024-06-04T04:39:47.163 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:39:47.163 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-04T04:39:47.163 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-04T04:39:47.163 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-04T04:39:47.163 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:39:51.735 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-test-2:19.0.0-3882.gb7328fed.el9.x86_64 143/170 2024-06-04T04:39:51.776 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 144/170 2024-06-04T04:39:51.817 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-nbd-2:19.0.0-3882.gb7328fed.el9.x86_64 145/170 2024-06-04T04:39:51.849 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-06-04T04:39:51.868 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-06-04T04:39:52.052 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephfs-shell-2:19.0.0-3882.gb7328fed.el9.x86_64 148/170 2024-06-04T04:39:52.089 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-06-04T04:39:52.121 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs-devel-2:19.0.0-3882.gb7328fed.el9.x86_ 150/170 2024-06-04T04:39:52.162 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-06-04T04:39:52.191 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : NetworkManager-team-1:1.47.91-1.el9.x86_64 152/170 2024-06-04T04:39:52.238 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : NetworkManager-tui-1:1.47.91-1.el9.x86_64 153/170 2024-06-04T04:39:52.355 INFO:teuthology.orchestra.run.smithi098.stdout:(107/159): libarrow-9.0.0-11.el9.x86_64.rpm 597 kB/s | 4.4 MB 00:07 2024-06-04T04:39:52.422 INFO:teuthology.orchestra.run.smithi098.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 391 kB/s | 26 kB 00:00 2024-06-04T04:39:52.530 INFO:teuthology.orchestra.run.smithi098.stdout:(109/159): liboath-2.6.7-2.el9.x86_64.rpm 447 kB/s | 49 kB 00:00 2024-06-04T04:39:52.672 INFO:teuthology.orchestra.run.smithi098.stdout:(110/159): libunwind-1.6.2-1.el9.x86_64.rpm 476 kB/s | 67 kB 00:00 2024-06-04T04:39:52.793 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-04T04:39:52.826 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-04T04:39:52.846 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-04T04:39:52.948 INFO:teuthology.orchestra.run.smithi098.stdout:(111/159): luarocks-3.9.2-5.el9.noarch.rpm 550 kB/s | 151 kB 00:00 2024-06-04T04:39:53.383 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:39:53.675 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephfs-top-2:19.0.0-3882.gb7328fed.el9.noarch 155/170 2024-06-04T04:39:53.756 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-06-04T04:39:53.773 INFO:teuthology.orchestra.run.smithi098.stdout:(112/159): fuse-2.9.9-15.el9.x86_64.rpm 3.6 kB/s | 80 kB 00:22 2024-06-04T04:39:53.778 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:39:53.797 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-06-04T04:39:53.891 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-06-04T04:39:53.926 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-06-04T04:39:53.950 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-04T04:39:53.976 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-06-04T04:39:54.003 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-06-04T04:39:54.003 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-04T04:39:54.006 INFO:teuthology.orchestra.run.smithi098.stdout:(113/159): libgfortran-11.4.1-3.el9.x86_64.rpm 36 kB/s | 803 kB 00:22 2024-06-04T04:39:54.087 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-04T04:39:54.098 INFO:teuthology.orchestra.run.smithi098.stdout:(114/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 473 kB/s | 43 kB 00:00 2024-06-04T04:39:54.112 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-06-04T04:39:54.112 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-04T04:39:54.165 INFO:teuthology.orchestra.run.smithi098.stdout:(115/159): python3-cachetools-4.2.4-1.el9.noarc 485 kB/s | 32 kB 00:00 2024-06-04T04:39:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-04T04:39:54.213 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-06-04T04:39:54.216 INFO:teuthology.orchestra.run.smithi098.stdout:(116/159): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-06-04T04:39:54.218 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-04T04:39:54.218 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-04T04:39:54.408 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-04T04:39:54.408 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-04T04:39:54.449 INFO:teuthology.orchestra.run.smithi098.stdout:(117/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 559 kB/s | 839 kB 00:01 2024-06-04T04:39:54.491 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-04T04:39:54.523 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-06-04T04:39:54.525 INFO:teuthology.orchestra.run.smithi098.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 557 kB/s | 172 kB 00:00 2024-06-04T04:39:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-04T04:39:54.691 INFO:teuthology.orchestra.run.smithi098.stdout:(119/159): python3-asyncssh-2.13.2-5.el9.noarch 598 kB/s | 548 kB 00:00 2024-06-04T04:39:55.033 INFO:teuthology.orchestra.run.smithi098.stdout:(120/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 596 kB/s | 303 kB 00:00 2024-06-04T04:39:55.059 INFO:teuthology.orchestra.run.smithi098.stdout:(121/159): python3-cherrypy-18.6.1-2.el9.noarch 589 kB/s | 358 kB 00:00 2024-06-04T04:39:55.092 INFO:teuthology.orchestra.run.smithi098.stdout:(122/159): python3-google-auth-2.29.0-1.el9.noa 513 kB/s | 206 kB 00:00 2024-06-04T04:39:55.126 INFO:teuthology.orchestra.run.smithi098.stdout:(123/159): python3-jaraco-8.2.1-3.el9.noarch.rp 323 kB/s | 11 kB 00:00 2024-06-04T04:39:55.168 INFO:teuthology.orchestra.run.smithi098.stdout:(124/159): python3-jaraco-classes-3.2.1-5.el9.n 426 kB/s | 18 kB 00:00 2024-06-04T04:39:55.235 INFO:teuthology.orchestra.run.smithi098.stdout:(125/159): python3-jaraco-collections-3.0.0-8.e 348 kB/s | 23 kB 00:00 2024-06-04T04:39:55.294 INFO:teuthology.orchestra.run.smithi098.stdout:(126/159): python3-jaraco-functools-3.5.0-2.el9 334 kB/s | 19 kB 00:00 2024-06-04T04:39:55.352 INFO:teuthology.orchestra.run.smithi098.stdout:(127/159): python3-jaraco-text-3.2.0-6.el9.noar 337 kB/s | 20 kB 00:00 2024-06-04T04:39:55.369 INFO:teuthology.orchestra.run.smithi098.stdout:(128/159): python3-grpcio-tools-1.46.7-10.el9.x 466 kB/s | 144 kB 00:00 2024-06-04T04:39:55.478 INFO:teuthology.orchestra.run.smithi098.stdout:(129/159): python3-logutils-0.3.5-21.el9.noarch 428 kB/s | 46 kB 00:00 2024-06-04T04:39:55.696 INFO:teuthology.orchestra.run.smithi098.stdout:(130/159): python3-more-itertools-8.12.0-2.el9. 361 kB/s | 79 kB 00:00 2024-06-04T04:39:55.820 INFO:teuthology.orchestra.run.smithi098.stdout:(131/159): python3-natsort-7.1.1-5.el9.noarch.r 467 kB/s | 58 kB 00:00 2024-06-04T04:39:56.295 INFO:teuthology.orchestra.run.smithi098.stdout:(132/159): python3-pecan-1.4.2-3.el9.noarch.rpm 573 kB/s | 272 kB 00:00 2024-06-04T04:39:56.346 INFO:teuthology.orchestra.run.smithi098.stdout:(133/159): python3-portend-3.1.0-2.el9.noarch.r 330 kB/s | 16 kB 00:00 2024-06-04T04:39:56.521 INFO:teuthology.orchestra.run.smithi098.stdout:(134/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 514 kB/s | 90 kB 00:00 2024-06-04T04:39:56.596 INFO:teuthology.orchestra.run.smithi098.stdout:(135/159): python3-pyperclip-1.8.2-2.el9.noarch 376 kB/s | 28 kB 00:00 2024-06-04T04:39:56.663 INFO:teuthology.orchestra.run.smithi098.stdout:(136/159): python3-repoze-lru-0.7-16.el9.noarch 459 kB/s | 31 kB 00:00 2024-06-04T04:39:56.997 INFO:teuthology.orchestra.run.smithi098.stdout:(137/159): python3-routes-2.5.1-5.el9.noarch.rp 566 kB/s | 188 kB 00:00 2024-06-04T04:39:57.131 INFO:teuthology.orchestra.run.smithi098.stdout:(138/159): python3-rsa-4.9-2.el9.noarch.rpm 443 kB/s | 59 kB 00:00 2024-06-04T04:39:57.239 INFO:teuthology.orchestra.run.smithi098.stdout:(139/159): python3-kubernetes-26.1.0-2.el9.noar 554 kB/s | 1.0 MB 00:01 2024-06-04T04:39:57.264 INFO:teuthology.orchestra.run.smithi098.stdout:(140/159): python3-tempora-5.0.0-2.el9.noarch.r 268 kB/s | 36 kB 00:00 2024-06-04T04:39:57.373 INFO:teuthology.orchestra.run.smithi098.stdout:(141/159): python3-typing-extensions-4.4.0-2.el 382 kB/s | 51 kB 00:00 2024-06-04T04:39:57.565 INFO:teuthology.orchestra.run.smithi098.stdout:(142/159): python3-websocket-client-1.2.3-2.el9 468 kB/s | 90 kB 00:00 2024-06-04T04:39:57.691 INFO:teuthology.orchestra.run.smithi098.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 539 kB/s | 230 kB 00:00 2024-06-04T04:39:57.749 INFO:teuthology.orchestra.run.smithi098.stdout:(144/159): python3-zc-lockfile-2.0-10.el9.noarc 343 kB/s | 20 kB 00:00 2024-06-04T04:39:58.116 INFO:teuthology.orchestra.run.smithi098.stdout:(145/159): re2-20211101-3.el9.x86_64.rpm 523 kB/s | 192 kB 00:00 2024-06-04T04:39:58.333 INFO:teuthology.orchestra.run.smithi098.stdout:(146/159): python3-werkzeug-2.0.3-3.el9.1.noarc 556 kB/s | 427 kB 00:00 2024-06-04T04:39:58.758 INFO:teuthology.orchestra.run.smithi098.stdout:(147/159): python3-grpcio-1.46.7-10.el9.x86_64. 560 kB/s | 2.0 MB 00:03 2024-06-04T04:39:59.250 INFO:teuthology.orchestra.run.smithi098.stdout:(148/159): librados2-19.0.0-3882.gb7328fed.el9. 3.7 MB/s | 3.4 MB 00:00 2024-06-04T04:39:59.517 INFO:teuthology.orchestra.run.smithi098.stdout:(149/159): librbd1-19.0.0-3882.gb7328fed.el9.x8 4.1 MB/s | 3.1 MB 00:00 2024-06-04T04:39:59.959 INFO:teuthology.orchestra.run.smithi098.stdout:(150/159): NetworkManager-libnm-1.47.91-1.el9.x 4.1 MB/s | 1.8 MB 00:00 2024-06-04T04:40:00.009 INFO:teuthology.orchestra.run.smithi098.stdout:(151/159): NetworkManager-team-1.47.91-1.el9.x8 830 kB/s | 41 kB 00:00 2024-06-04T04:40:00.110 INFO:teuthology.orchestra.run.smithi098.stdout:(152/159): NetworkManager-1.47.91-1.el9.x86_64. 2.7 MB/s | 2.3 MB 00:00 2024-06-04T04:40:00.160 INFO:teuthology.orchestra.run.smithi098.stdout:(153/159): NetworkManager-tui-1.47.91-1.el9.x86 1.6 MB/s | 248 kB 00:00 2024-06-04T04:40:00.188 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-04T04:40:00.188 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-2:19.0.0-3882.gb7328fed.el9.x86_64 1/170 2024-06-04T04:40:00.188 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-base-2:19.0.0-3882.gb7328fed.el9.x86_64 2/170 2024-06-04T04:40:00.189 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 3/170 2024-06-04T04:40:00.189 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 4/170 2024-06-04T04:40:00.189 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3882.gb7328 5/170 2024-06-04T04:40:00.189 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mds-2:19.0.0-3882.gb7328fed.el9.x86_64 6/170 2024-06-04T04:40:00.189 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-2:19.0.0-3882.gb7328fed.el9.x86_64 7/170 2024-06-04T04:40:00.189 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mon-2:19.0.0-3882.gb7328fed.el9.x86_64 8/170 2024-06-04T04:40:00.189 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-osd-2:19.0.0-3882.gb7328fed.el9.x86_64 9/170 2024-06-04T04:40:00.189 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-radosgw-2:19.0.0-3882.gb7328fed.el9.x86_64 10/170 2024-06-04T04:40:00.190 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-selinux-2:19.0.0-3882.gb7328fed.el9.x86_64 11/170 2024-06-04T04:40:00.190 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-test-2:19.0.0-3882.gb7328fed.el9.x86_64 12/170 2024-06-04T04:40:00.190 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephfs-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 13/170 2024-06-04T04:40:00.190 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephfs-shell-2:19.0.0-3882.gb7328fed.el9.x86_64 14/170 2024-06-04T04:40:00.190 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs-devel-2:19.0.0-3882.gb7328fed.el9.x86_ 15/170 2024-06-04T04:40:00.190 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs2-2:19.0.0-3882.gb7328fed.el9.x86_64 16/170 2024-06-04T04:40:00.190 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephsqlite-2:19.0.0-3882.gb7328fed.el9.x86_64 17/170 2024-06-04T04:40:00.190 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados-devel-2:19.0.0-3882.gb7328fed.el9.x86_6 18/170 2024-06-04T04:40:00.191 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libradosstriper1-2:19.0.0-3882.gb7328fed.el9.x86 19/170 2024-06-04T04:40:00.191 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librgw2-2:19.0.0-3882.gb7328fed.el9.x86_64 20/170 2024-06-04T04:40:00.191 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3882.gb7328fed.el 21/170 2024-06-04T04:40:00.191 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-common-2:19.0.0-3882.gb7328fed.el9. 22/170 2024-06-04T04:40:00.191 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cephfs-2:19.0.0-3882.gb7328fed.el9.x86_6 23/170 2024-06-04T04:40:00.191 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rados-2:19.0.0-3882.gb7328fed.el9.x86_64 24/170 2024-06-04T04:40:00.191 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rbd-2:19.0.0-3882.gb7328fed.el9.x86_64 25/170 2024-06-04T04:40:00.191 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rgw-2:19.0.0-3882.gb7328fed.el9.x86_64 26/170 2024-06-04T04:40:00.192 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 27/170 2024-06-04T04:40:00.192 INFO:teuthology.orchestra.run.smithi098.stdout:(154/159): elfutils-debuginfod-client-0.191-4.e 487 kB/s | 37 kB 00:00 2024-06-04T04:40:00.204 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 28/170 2024-06-04T04:40:00.204 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-nbd-2:19.0.0-3882.gb7328fed.el9.x86_64 29/170 2024-06-04T04:40:00.204 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3882.gb7328fed. 30/170 2024-06-04T04:40:00.204 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3882.gb7328fed.el9.noa 31/170 2024-06-04T04:40:00.204 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3882.gb7328fed.el9.n 32/170 2024-06-04T04:40:00.204 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3882.gb73 33/170 2024-06-04T04:40:00.204 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3882.gb7328fed.el 34/170 2024-06-04T04:40:00.204 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3882.gb7328fed.el9.noarch 35/170 2024-06-04T04:40:00.204 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3882.gb7328fed.e 36/170 2024-06-04T04:40:00.205 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-volume-2:19.0.0-3882.gb7328fed.el9.noarch 37/170 2024-06-04T04:40:00.205 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephadm-2:19.0.0-3882.gb7328fed.el9.noarch 38/170 2024-06-04T04:40:00.205 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephfs-top-2:19.0.0-3882.gb7328fed.el9.noarch 39/170 2024-06-04T04:40:00.205 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-06-04T04:40:00.205 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-06-04T04:40:00.205 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-06-04T04:40:00.205 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-06-04T04:40:00.205 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-06-04T04:40:00.205 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-06-04T04:40:00.206 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-06-04T04:40:00.206 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-06-04T04:40:00.206 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-06-04T04:40:00.206 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-06-04T04:40:00.206 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-06-04T04:40:00.206 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-06-04T04:40:00.206 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-06-04T04:40:00.206 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-06-04T04:40:00.206 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-06-04T04:40:00.206 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-06-04T04:40:00.206 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-06-04T04:40:00.206 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-06-04T04:40:00.206 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-06-04T04:40:00.206 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-06-04T04:40:00.207 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-06-04T04:40:00.207 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-06-04T04:40:00.207 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-06-04T04:40:00.207 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-06-04T04:40:00.207 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-06-04T04:40:00.207 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-06-04T04:40:00.207 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-06-04T04:40:00.207 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-06-04T04:40:00.207 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-06-04T04:40:00.207 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-06-04T04:40:00.207 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-06-04T04:40:00.207 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-06-04T04:40:00.208 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-06-04T04:40:00.208 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-06-04T04:40:00.208 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-06-04T04:40:00.208 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-06-04T04:40:00.208 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-06-04T04:40:00.208 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-06-04T04:40:00.208 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-06-04T04:40:00.208 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-06-04T04:40:00.208 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-06-04T04:40:00.208 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-06-04T04:40:00.208 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-06-04T04:40:00.208 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-06-04T04:40:00.208 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-06-04T04:40:00.208 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-06-04T04:40:00.209 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-06-04T04:40:00.209 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-06-04T04:40:00.209 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-06-04T04:40:00.209 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-06-04T04:40:00.209 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-06-04T04:40:00.209 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-06-04T04:40:00.209 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-06-04T04:40:00.209 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-06-04T04:40:00.209 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-06-04T04:40:00.209 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-06-04T04:40:00.209 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-06-04T04:40:00.209 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-06-04T04:40:00.209 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-06-04T04:40:00.210 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-06-04T04:40:00.210 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-06-04T04:40:00.210 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-06-04T04:40:00.210 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-06-04T04:40:00.210 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-06-04T04:40:00.210 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-06-04T04:40:00.210 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-06-04T04:40:00.210 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-06-04T04:40:00.210 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-06-04T04:40:00.210 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-06-04T04:40:00.210 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-06-04T04:40:00.210 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-06-04T04:40:00.211 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-06-04T04:40:00.211 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-06-04T04:40:00.211 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-06-04T04:40:00.211 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-06-04T04:40:00.211 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-06-04T04:40:00.211 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-06-04T04:40:00.211 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-06-04T04:40:00.211 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-06-04T04:40:00.211 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-06-04T04:40:00.211 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-06-04T04:40:00.211 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-06-04T04:40:00.211 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-06-04T04:40:00.211 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-06-04T04:40:00.211 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-06-04T04:40:00.212 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-06-04T04:40:00.212 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-06-04T04:40:00.212 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-06-04T04:40:00.212 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-06-04T04:40:00.212 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-06-04T04:40:00.212 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-06-04T04:40:00.212 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-06-04T04:40:00.212 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-06-04T04:40:00.212 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-06-04T04:40:00.212 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-06-04T04:40:00.212 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-06-04T04:40:00.212 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-06-04T04:40:00.212 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-06-04T04:40:00.213 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-06-04T04:40:00.213 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-06-04T04:40:00.213 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-06-04T04:40:00.213 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-06-04T04:40:00.213 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-06-04T04:40:00.213 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-06-04T04:40:00.213 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-06-04T04:40:00.213 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-06-04T04:40:00.213 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-06-04T04:40:00.213 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-06-04T04:40:00.213 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-06-04T04:40:00.213 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:19.0.0-3882.gb7328fed.el9.x86_64 149/170 2024-06-04T04:40:00.213 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-06-04T04:40:00.213 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:19.0.0-3882.gb7328fed.el9.x86_64 151/170 2024-06-04T04:40:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-06-04T04:40:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : NetworkManager-1:1.47.91-1.el9.x86_64 153/170 2024-06-04T04:40:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-06-04T04:40:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 155/170 2024-06-04T04:40:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-06-04T04:40:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : NetworkManager-team-1:1.47.91-1.el9.x86_64 157/170 2024-06-04T04:40:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-06-04T04:40:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : NetworkManager-tui-1:1.47.91-1.el9.x86_64 159/170 2024-06-04T04:40:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-04T04:40:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-06-04T04:40:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-06-04T04:40:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-06-04T04:40:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-06-04T04:40:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-06-04T04:40:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-06-04T04:40:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-06-04T04:40:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-06-04T04:40:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-06-04T04:40:00.336 INFO:teuthology.orchestra.run.smithi098.stdout:(155/159): elfutils-libelf-0.191-4.el9.x86_64.r 1.2 MB/s | 207 kB 00:00 2024-06-04T04:40:00.394 INFO:teuthology.orchestra.run.smithi098.stdout:(156/159): elfutils-libs-0.191-4.el9.x86_64.rpm 1.2 MB/s | 260 kB 00:00 2024-06-04T04:40:00.886 INFO:teuthology.orchestra.run.smithi098.stdout:(157/159): thrift-0.15.0-2.el9.x86_64.rpm 585 kB/s | 1.6 MB 00:02 2024-06-04T04:40:01.103 INFO:teuthology.orchestra.run.smithi098.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 1.8 MB/s | 1.3 MB 00:00 2024-06-04T04:40:01.678 INFO:teuthology.orchestra.run.smithi098.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 1.9 MB/s | 2.4 MB 00:01 2024-06-04T04:40:01.686 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-04T04:40:01.686 INFO:teuthology.orchestra.run.smithi098.stdout:Total 6.2 MB/s | 227 MB 00:36 2024-06-04T04:40:02.886 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-04T04:40:03.044 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-04T04:40:03.044 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-04T04:40:03.229 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-06-04T04:40:03.230 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:40:03.230 INFO:teuthology.orchestra.run.smithi112.stdout:Upgraded: 2024-06-04T04:40:03.230 INFO:teuthology.orchestra.run.smithi112.stdout: NetworkManager-1:1.47.91-1.el9.x86_64 2024-06-04T04:40:03.230 INFO:teuthology.orchestra.run.smithi112.stdout: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 2024-06-04T04:40:03.230 INFO:teuthology.orchestra.run.smithi112.stdout: NetworkManager-team-1:1.47.91-1.el9.x86_64 2024-06-04T04:40:03.230 INFO:teuthology.orchestra.run.smithi112.stdout: NetworkManager-tui-1:1.47.91-1.el9.x86_64 2024-06-04T04:40:03.230 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-06-04T04:40:03.230 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-06-04T04:40:03.230 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-06-04T04:40:03.230 INFO:teuthology.orchestra.run.smithi112.stdout: librados2-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.231 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.231 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-06-04T04:40:03.231 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-06-04T04:40:03.231 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-06-04T04:40:03.231 INFO:teuthology.orchestra.run.smithi112.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-04T04:40:03.231 INFO:teuthology.orchestra.run.smithi112.stdout: automake-1.16.2-8.el9.noarch 2024-06-04T04:40:03.231 INFO:teuthology.orchestra.run.smithi112.stdout: bison-3.7.4-5.el9.x86_64 2024-06-04T04:40:03.231 INFO:teuthology.orchestra.run.smithi112.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-04T04:40:03.231 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.231 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.231 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.231 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.231 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:40:03.232 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.232 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.232 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.232 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:40:03.232 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:40:03.232 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:40:03.232 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:40:03.232 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:40:03.232 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.232 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.232 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:40:03.232 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.232 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.233 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.233 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:40:03.233 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:40:03.233 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.233 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-shell-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.233 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-top-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:40:03.233 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-06-04T04:40:03.233 INFO:teuthology.orchestra.run.smithi112.stdout: flex-2.6.4-9.el9.x86_64 2024-06-04T04:40:03.233 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-04T04:40:03.233 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-04T04:40:03.233 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-04T04:40:03.233 INFO:teuthology.orchestra.run.smithi112.stdout: fuse-2.9.9-15.el9.x86_64 2024-06-04T04:40:03.234 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-04T04:40:03.234 INFO:teuthology.orchestra.run.smithi112.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-04T04:40:03.234 INFO:teuthology.orchestra.run.smithi112.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-06-04T04:40:03.234 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-04T04:40:03.234 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-04T04:40:03.234 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.234 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.234 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.234 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-04T04:40:03.234 INFO:teuthology.orchestra.run.smithi112.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-04T04:40:03.234 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-04T04:40:03.235 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-04T04:40:03.235 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-04T04:40:03.235 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.235 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.235 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-04T04:40:03.235 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.235 INFO:teuthology.orchestra.run.smithi112.stdout: libtool-2.4.6-45.el9.x86_64 2024-06-04T04:40:03.235 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-04T04:40:03.235 INFO:teuthology.orchestra.run.smithi112.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-06-04T04:40:03.235 INFO:teuthology.orchestra.run.smithi112.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-06-04T04:40:03.235 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-04T04:40:03.235 INFO:teuthology.orchestra.run.smithi112.stdout: lua-5.4.4-4.el9.x86_64 2024-06-04T04:40:03.236 INFO:teuthology.orchestra.run.smithi112.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-04T04:40:03.236 INFO:teuthology.orchestra.run.smithi112.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-04T04:40:03.236 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-04T04:40:03.236 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-04T04:40:03.236 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-06-04T04:40:03.236 INFO:teuthology.orchestra.run.smithi112.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-04T04:40:03.236 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-13.14-1.el9.x86_64 2024-06-04T04:40:03.236 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-06-04T04:40:03.236 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-06-04T04:40:03.236 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-server-13.14-1.el9.x86_64 2024-06-04T04:40:03.237 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-04T04:40:03.237 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-04T04:40:03.237 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-04T04:40:03.237 INFO:teuthology.orchestra.run.smithi112.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-06-04T04:40:03.237 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-04T04:40:03.237 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-04T04:40:03.237 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-04T04:40:03.237 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.237 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.237 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.237 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-04T04:40:03.237 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-04T04:40:03.237 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-04T04:40:03.238 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-04T04:40:03.238 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-06-04T04:40:03.238 INFO:teuthology.orchestra.run.smithi112.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-06-04T04:40:03.238 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-04T04:40:03.238 INFO:teuthology.orchestra.run.smithi112.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-04T04:40:03.239 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-06-04T04:40:03.239 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-04T04:40:03.239 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-04T04:40:03.239 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-04T04:40:03.239 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-04T04:40:03.239 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-04T04:40:03.239 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-04T04:40:03.239 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-04T04:40:03.239 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-04T04:40:03.239 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-04T04:40:03.240 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-04T04:40:03.240 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-04T04:40:03.240 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-04T04:40:03.240 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-04T04:40:03.240 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-04T04:40:03.240 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-04T04:40:03.240 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-04T04:40:03.240 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-04T04:40:03.240 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-04T04:40:03.240 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-04T04:40:03.240 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-04T04:40:03.240 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-04T04:40:03.240 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ply-3.11-14.el9.noarch 2024-06-04T04:40:03.240 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-04T04:40:03.240 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-04T04:40:03.241 INFO:teuthology.orchestra.run.smithi112.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-04T04:40:03.241 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-04T04:40:03.241 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-04T04:40:03.241 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-04T04:40:03.241 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-04T04:40:03.241 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-06-04T04:40:03.241 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-04T04:40:03.241 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.241 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.241 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-04T04:40:03.241 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-04T04:40:03.241 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-04T04:40:03.241 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.241 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-04T04:40:03.241 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-04T04:40:03.242 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-04T04:40:03.242 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-04T04:40:03.242 INFO:teuthology.orchestra.run.smithi112.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-04T04:40:03.242 INFO:teuthology.orchestra.run.smithi112.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-04T04:40:03.242 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-04T04:40:03.242 INFO:teuthology.orchestra.run.smithi112.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-06-04T04:40:03.242 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-04T04:40:03.242 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-04T04:40:03.242 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-04T04:40:03.242 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-04T04:40:03.242 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-04T04:40:03.242 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-04T04:40:03.243 INFO:teuthology.orchestra.run.smithi112.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-04T04:40:03.243 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.243 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.243 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:40:03.243 INFO:teuthology.orchestra.run.smithi112.stdout: re2-1:20211101-3.el9.x86_64 2024-06-04T04:40:03.243 INFO:teuthology.orchestra.run.smithi112.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-04T04:40:03.243 INFO:teuthology.orchestra.run.smithi112.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-04T04:40:03.243 INFO:teuthology.orchestra.run.smithi112.stdout: uuid-1.6.2-55.el9.x86_64 2024-06-04T04:40:03.243 INFO:teuthology.orchestra.run.smithi112.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-06-04T04:40:03.243 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-04T04:40:03.243 INFO:teuthology.orchestra.run.smithi112.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-06-04T04:40:03.244 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:40:03.244 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-06-04T04:40:03.660 DEBUG:teuthology.parallel:result is None 2024-06-04T04:40:05.199 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-04T04:40:05.199 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-04T04:40:11.476 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-04T04:40:11.558 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-06-04T04:40:11.600 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-06-04T04:40:11.659 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-06-04T04:40:11.692 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-06-04T04:40:11.717 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-06-04T04:40:11.767 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-06-04T04:40:12.110 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-06-04T04:40:12.210 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:19.0.0-3882.gb7328fed.el9.x86_64 8/170 2024-06-04T04:40:12.333 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:19.0.0-3882.gb7328fed.el9.x86_64 8/170 2024-06-04T04:40:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:19.0.0-3882.gb7328fed.el9.x86_64 9/170 2024-06-04T04:40:12.455 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:19.0.0-3882.gb7328fed.el9.x86_64 9/170 2024-06-04T04:40:12.485 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:19.0.0-3882.gb7328fed.el9.x86_64 10/170 2024-06-04T04:40:12.523 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-06-04T04:40:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-06-04T04:40:12.775 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-06-04T04:40:12.803 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:19.0.0-3882.gb7328fed.el9.x86_64 14/170 2024-06-04T04:40:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3882.gb7328fed.el9.x86_64 14/170 2024-06-04T04:40:12.928 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:19.0.0-3882.gb7328fed.el9.x86 15/170 2024-06-04T04:40:12.991 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3882.gb7328fed.el9.x86 15/170 2024-06-04T04:40:13.013 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-04T04:40:13.048 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-04T04:40:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-04T04:40:13.432 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-04T04:40:13.578 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-04T04:40:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-04T04:40:13.754 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-04T04:40:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-04T04:40:14.064 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-04T04:40:14.172 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-06-04T04:40:14.202 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-06-04T04:40:14.237 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-06-04T04:40:14.310 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-06-04T04:40:14.353 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-06-04T04:40:14.397 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-06-04T04:40:14.430 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-06-04T04:40:14.458 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-06-04T04:40:14.526 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-06-04T04:40:14.579 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-06-04T04:40:14.611 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-06-04T04:40:14.646 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-06-04T04:40:14.723 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-06-04T04:40:14.757 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:19.0.0-3882.gb7328fed.el9. 33/170 2024-06-04T04:40:14.800 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:19.0.0-3882.gb7328fed.el 34/170 2024-06-04T04:40:14.829 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:19.0.0-3882.gb7328fed.el9.x86_6 35/170 2024-06-04T04:40:14.899 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-06-04T04:40:14.946 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-06-04T04:40:15.019 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-06-04T04:40:15.152 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-06-04T04:40:15.205 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-06-04T04:40:15.748 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-06-04T04:40:15.826 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-06-04T04:40:15.855 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-06-04T04:40:15.892 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-06-04T04:40:15.940 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-06-04T04:40:15.978 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:19.0.0-3882.gb7328fed.el9.x86_6 46/170 2024-06-04T04:40:16.021 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-06-04T04:40:16.174 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-06-04T04:40:16.232 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-06-04T04:40:16.323 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-06-04T04:40:16.388 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-06-04T04:40:16.461 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-06-04T04:40:16.492 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-06-04T04:40:16.528 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-06-04T04:40:16.575 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-06-04T04:40:16.611 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-06-04T04:40:16.654 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-06-04T04:40:16.696 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-06-04T04:40:16.832 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-06-04T04:40:16.874 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-06-04T04:40:16.915 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-06-04T04:40:17.009 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-06-04T04:40:17.040 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-06-04T04:40:17.090 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-06-04T04:40:17.122 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-06-04T04:40:17.147 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-06-04T04:40:17.240 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-06-04T04:40:17.365 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-06-04T04:40:17.397 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-06-04T04:40:17.433 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-06-04T04:40:17.465 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-06-04T04:40:17.507 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-06-04T04:40:17.538 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-06-04T04:40:17.568 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-06-04T04:40:17.602 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-06-04T04:40:17.636 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-06-04T04:40:17.677 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-06-04T04:40:17.712 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-06-04T04:40:17.798 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-06-04T04:40:17.843 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-06-04T04:40:17.882 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-06-04T04:40:17.929 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-06-04T04:40:18.027 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-06-04T04:40:18.075 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-06-04T04:40:18.801 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-06-04T04:40:18.848 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-06-04T04:40:18.881 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-06-04T04:40:18.982 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-06-04T04:40:19.028 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-06-04T04:40:19.073 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-06-04T04:40:19.161 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-06-04T04:40:19.193 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-06-04T04:40:19.360 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-06-04T04:40:19.391 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-06-04T04:40:19.471 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-06-04T04:40:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-06-04T04:40:20.239 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-06-04T04:40:21.503 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-06-04T04:40:21.587 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-06-04T04:40:21.616 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-06-04T04:40:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-06-04T04:40:21.967 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-06-04T04:40:22.003 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:19.0.0-3882.gb7328fed.el9.x86_64 103/170 2024-06-04T04:40:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:19.0.0-3882.gb7328fed.el9.x86_64 103/170 2024-06-04T04:40:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:19.0.0-3882.gb7328fed.el9.x86_64 104/170 2024-06-04T04:40:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-06-04T04:40:22.617 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-06-04T04:40:22.648 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:19.0.0-3882.gb7328fed.el9.x86_64 107/170 2024-06-04T04:40:22.712 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:19.0.0-3882.gb7328fed.el9.x86_64 107/170 2024-06-04T04:40:22.734 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:19.0.0-3882.gb7328fed.el9.x86_64 108/170 2024-06-04T04:40:25.055 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 109/170 2024-06-04T04:40:25.097 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 109/170 2024-06-04T04:40:25.660 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 109/170 2024-06-04T04:40:25.700 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:19.0.0-3882.gb7328fed.el9.x86_64 110/170 2024-06-04T04:40:25.791 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:19.0.0-3882.gb7328fed.el9.x86_64 110/170 2024-06-04T04:40:25.792 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-04T04:40:25.792 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-04T04:40:25.792 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:40:25.819 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:19.0.0-3882.gb7328fed.el9.x86_64 111/170 2024-06-04T04:40:37.569 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3882.gb7328fed.el9.x86_64 111/170 2024-06-04T04:40:37.569 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-06-04T04:40:37.569 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-06-04T04:40:37.569 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-06-04T04:40:37.569 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-06-04T04:40:37.569 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-06-04T04:40:37.569 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-06-04T04:40:37.569 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-06-04T04:40:37.569 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:40:37.737 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:19.0.0-3882.gb7328fed.el9.noarch 112/170 2024-06-04T04:40:37.753 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:19.0.0-3882.gb7328fed.el9.noarch 112/170 2024-06-04T04:40:37.753 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:40:37.753 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-04T04:40:37.753 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:40:39.240 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:19.0.0-3882.gb7328fed.el9.x86_64 113/170 2024-06-04T04:40:39.271 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:19.0.0-3882.gb7328fed.el9.x86_64 113/170 2024-06-04T04:40:39.271 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:40:39.271 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-04T04:40:39.271 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-04T04:40:39.271 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-04T04:40:39.272 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:40:39.499 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:19.0.0-3882.gb7328fed.el9.x86_64 114/170 2024-06-04T04:40:39.522 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:19.0.0-3882.gb7328fed.el9.x86_64 114/170 2024-06-04T04:40:39.522 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:40:39.523 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-04T04:40:39.523 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-04T04:40:39.523 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-04T04:40:39.523 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:40:39.958 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:19.0.0-3882.gb7328fed.el9.x86_64 115/170 2024-06-04T04:40:39.981 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:19.0.0-3882.gb7328fed.el9.x86_64 115/170 2024-06-04T04:40:39.981 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:40:39.981 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-04T04:40:39.981 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-04T04:40:39.981 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-04T04:40:39.982 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:40:40.090 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-06-04T04:40:40.145 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-06-04T04:40:40.200 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-06-04T04:40:40.382 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-06-04T04:40:40.431 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-06-04T04:40:40.508 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-06-04T04:40:40.596 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-06-04T04:40:40.744 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-06-04T04:40:40.789 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-06-04T04:40:40.828 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-06-04T04:40:40.865 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-06-04T04:40:41.261 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-3882.gb7328fed.el9.noarch 127/170 2024-06-04T04:40:41.290 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:19.0.0-3882.gb7328fed.el9.noarch 127/170 2024-06-04T04:40:41.322 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3882.gb7328fed.e 128/170 2024-06-04T04:40:41.383 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3882.gb7328fed. 129/170 2024-06-04T04:40:41.445 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3882.gb7328fed.el9.noa 130/170 2024-06-04T04:40:42.480 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3882.gb7328fed.el9.noa 130/170 2024-06-04T04:40:42.523 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3882.gb7328fed.el9.n 131/170 2024-06-04T04:40:43.484 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3882.gb7328fed.el9.n 131/170 2024-06-04T04:40:43.523 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3882.gb73 132/170 2024-06-04T04:40:43.634 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3882.gb73 132/170 2024-06-04T04:40:43.794 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3882.gb7328fed.el 133/170 2024-06-04T04:40:43.826 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:19.0.0-3882.gb7328fed.el9.x86_64 134/170 2024-06-04T04:40:43.850 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3882.gb7328fed.el9.x86_64 134/170 2024-06-04T04:40:43.850 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:40:43.850 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-04T04:40:43.850 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-04T04:40:43.850 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-04T04:40:43.850 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:40:43.893 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:19.0.0-3882.gb7328fed.el9.noarch 135/170 2024-06-04T04:40:43.904 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3882.gb7328fed.el9.noarch 135/170 2024-06-04T04:40:44.004 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:19.0.0-3882.gb7328fed.el9.x86_64 136/170 2024-06-04T04:40:44.063 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 137/170 2024-06-04T04:40:45.003 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-06-04T04:40:45.038 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:19.0.0-3882.gb7328fed.el9.x86_64 139/170 2024-06-04T04:40:45.061 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3882.gb7328fed.el9.x86_64 139/170 2024-06-04T04:40:45.061 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:40:45.061 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-04T04:40:45.061 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-04T04:40:45.061 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-04T04:40:45.061 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:40:45.105 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3882.gb7328 140/170 2024-06-04T04:40:45.127 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3882.gb7328 140/170 2024-06-04T04:40:45.127 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:40:45.127 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-04T04:40:45.128 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:40:45.197 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 141/170 2024-06-04T04:40:45.220 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 141/170 2024-06-04T04:40:45.220 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:40:45.220 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-06-04T04:40:45.221 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-04T04:40:45.221 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-04T04:40:45.221 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:40:45.531 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 142/170 2024-06-04T04:40:45.554 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 142/170 2024-06-04T04:40:45.554 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:40:45.554 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-04T04:40:45.555 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-04T04:40:45.555 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-04T04:40:45.555 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:40:50.152 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:19.0.0-3882.gb7328fed.el9.x86_64 143/170 2024-06-04T04:40:50.193 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 144/170 2024-06-04T04:40:50.234 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:19.0.0-3882.gb7328fed.el9.x86_64 145/170 2024-06-04T04:40:50.266 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-06-04T04:40:50.294 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-06-04T04:40:50.486 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-shell-2:19.0.0-3882.gb7328fed.el9.x86_64 148/170 2024-06-04T04:40:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-06-04T04:40:50.564 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:19.0.0-3882.gb7328fed.el9.x86_ 150/170 2024-06-04T04:40:50.605 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-06-04T04:40:50.634 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-team-1:1.47.91-1.el9.x86_64 152/170 2024-06-04T04:40:50.681 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-tui-1:1.47.91-1.el9.x86_64 153/170 2024-06-04T04:40:51.282 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-04T04:40:51.321 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-04T04:40:51.341 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-04T04:40:52.169 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-top-2:19.0.0-3882.gb7328fed.el9.noarch 155/170 2024-06-04T04:40:52.268 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-06-04T04:40:52.310 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-06-04T04:40:52.393 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-06-04T04:40:52.421 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-06-04T04:40:52.446 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-04T04:40:52.472 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-06-04T04:40:52.498 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-06-04T04:40:52.498 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-04T04:40:52.566 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-04T04:40:52.599 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-06-04T04:40:52.599 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-04T04:40:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-04T04:40:52.717 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-06-04T04:40:52.723 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-04T04:40:52.723 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-04T04:40:52.911 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-04T04:40:52.911 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-04T04:40:52.996 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-04T04:40:53.028 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-06-04T04:40:53.094 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-04T04:40:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-04T04:40:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:19.0.0-3882.gb7328fed.el9.x86_64 1/170 2024-06-04T04:40:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:19.0.0-3882.gb7328fed.el9.x86_64 2/170 2024-06-04T04:40:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 3/170 2024-06-04T04:40:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 4/170 2024-06-04T04:40:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3882.gb7328 5/170 2024-06-04T04:40:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:19.0.0-3882.gb7328fed.el9.x86_64 6/170 2024-06-04T04:40:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:19.0.0-3882.gb7328fed.el9.x86_64 7/170 2024-06-04T04:40:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:19.0.0-3882.gb7328fed.el9.x86_64 8/170 2024-06-04T04:40:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:19.0.0-3882.gb7328fed.el9.x86_64 9/170 2024-06-04T04:40:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:19.0.0-3882.gb7328fed.el9.x86_64 10/170 2024-06-04T04:40:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:19.0.0-3882.gb7328fed.el9.x86_64 11/170 2024-06-04T04:40:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:19.0.0-3882.gb7328fed.el9.x86_64 12/170 2024-06-04T04:40:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 13/170 2024-06-04T04:40:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-shell-2:19.0.0-3882.gb7328fed.el9.x86_64 14/170 2024-06-04T04:40:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:19.0.0-3882.gb7328fed.el9.x86_ 15/170 2024-06-04T04:40:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:19.0.0-3882.gb7328fed.el9.x86_64 16/170 2024-06-04T04:40:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:19.0.0-3882.gb7328fed.el9.x86_64 17/170 2024-06-04T04:40:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:19.0.0-3882.gb7328fed.el9.x86_6 18/170 2024-06-04T04:40:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:19.0.0-3882.gb7328fed.el9.x86 19/170 2024-06-04T04:40:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:19.0.0-3882.gb7328fed.el9.x86_64 20/170 2024-06-04T04:40:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3882.gb7328fed.el 21/170 2024-06-04T04:40:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:19.0.0-3882.gb7328fed.el9. 22/170 2024-06-04T04:40:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:19.0.0-3882.gb7328fed.el9.x86_6 23/170 2024-06-04T04:40:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:19.0.0-3882.gb7328fed.el9.x86_64 24/170 2024-06-04T04:40:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:19.0.0-3882.gb7328fed.el9.x86_64 25/170 2024-06-04T04:40:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:19.0.0-3882.gb7328fed.el9.x86_64 26/170 2024-06-04T04:40:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 27/170 2024-06-04T04:40:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 28/170 2024-06-04T04:40:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:19.0.0-3882.gb7328fed.el9.x86_64 29/170 2024-06-04T04:40:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3882.gb7328fed. 30/170 2024-06-04T04:40:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3882.gb7328fed.el9.noa 31/170 2024-06-04T04:40:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3882.gb7328fed.el9.n 32/170 2024-06-04T04:40:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3882.gb73 33/170 2024-06-04T04:40:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3882.gb7328fed.el 34/170 2024-06-04T04:40:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3882.gb7328fed.el9.noarch 35/170 2024-06-04T04:40:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3882.gb7328fed.e 36/170 2024-06-04T04:40:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:19.0.0-3882.gb7328fed.el9.noarch 37/170 2024-06-04T04:40:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:19.0.0-3882.gb7328fed.el9.noarch 38/170 2024-06-04T04:40:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-top-2:19.0.0-3882.gb7328fed.el9.noarch 39/170 2024-06-04T04:40:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-06-04T04:40:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-06-04T04:40:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-06-04T04:40:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-06-04T04:40:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-06-04T04:40:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-06-04T04:40:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-06-04T04:40:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-06-04T04:40:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-06-04T04:40:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-06-04T04:40:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-06-04T04:40:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-06-04T04:40:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-06-04T04:40:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-06-04T04:40:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-06-04T04:40:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-06-04T04:40:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-06-04T04:40:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-06-04T04:40:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-06-04T04:40:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-06-04T04:40:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-06-04T04:40:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-06-04T04:40:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-06-04T04:40:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-06-04T04:40:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-06-04T04:40:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-06-04T04:40:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-06-04T04:40:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-06-04T04:40:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-06-04T04:40:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-06-04T04:40:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-06-04T04:40:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-06-04T04:40:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-06-04T04:40:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-06-04T04:40:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-06-04T04:40:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-06-04T04:40:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-06-04T04:40:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-06-04T04:40:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-06-04T04:40:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-06-04T04:40:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-06-04T04:40:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-06-04T04:40:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-06-04T04:40:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-06-04T04:40:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-06-04T04:40:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-06-04T04:40:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-06-04T04:40:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-06-04T04:40:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-06-04T04:40:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-06-04T04:40:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-06-04T04:40:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-06-04T04:40:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-06-04T04:40:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-06-04T04:40:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-06-04T04:40:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-06-04T04:40:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-06-04T04:40:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-06-04T04:40:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-06-04T04:40:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-06-04T04:40:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-06-04T04:40:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-06-04T04:40:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-06-04T04:40:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-06-04T04:40:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-06-04T04:40:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-06-04T04:40:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-06-04T04:40:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-06-04T04:40:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-06-04T04:40:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-06-04T04:40:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-06-04T04:40:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-06-04T04:40:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-06-04T04:40:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-06-04T04:40:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-06-04T04:40:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-06-04T04:40:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-06-04T04:40:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-06-04T04:40:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-06-04T04:40:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-06-04T04:40:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-06-04T04:40:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-06-04T04:40:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-06-04T04:40:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-06-04T04:40:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-06-04T04:40:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-06-04T04:40:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-06-04T04:40:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-06-04T04:40:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-06-04T04:40:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-06-04T04:40:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-06-04T04:40:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-06-04T04:40:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-06-04T04:40:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-06-04T04:40:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-06-04T04:40:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-06-04T04:40:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-06-04T04:40:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-06-04T04:40:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-06-04T04:40:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-06-04T04:40:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-06-04T04:40:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-06-04T04:40:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-06-04T04:40:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-06-04T04:40:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-06-04T04:40:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-06-04T04:40:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-06-04T04:40:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-06-04T04:40:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-06-04T04:40:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:19.0.0-3882.gb7328fed.el9.x86_64 149/170 2024-06-04T04:40:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-06-04T04:40:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:19.0.0-3882.gb7328fed.el9.x86_64 151/170 2024-06-04T04:40:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-06-04T04:40:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.47.91-1.el9.x86_64 153/170 2024-06-04T04:40:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-06-04T04:40:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 155/170 2024-06-04T04:40:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-06-04T04:40:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.47.91-1.el9.x86_64 157/170 2024-06-04T04:40:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-06-04T04:40:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.47.91-1.el9.x86_64 159/170 2024-06-04T04:40:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-04T04:40:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-06-04T04:40:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-06-04T04:40:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-06-04T04:40:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-06-04T04:40:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-06-04T04:40:58.919 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-06-04T04:40:58.919 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-06-04T04:40:58.919 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-06-04T04:40:58.919 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-06-04T04:41:01.683 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-06-04T04:41:01.684 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:41:01.684 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-06-04T04:41:01.684 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-1:1.47.91-1.el9.x86_64 2024-06-04T04:41:01.684 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 2024-06-04T04:41:01.684 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team-1:1.47.91-1.el9.x86_64 2024-06-04T04:41:01.684 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui-1:1.47.91-1.el9.x86_64 2024-06-04T04:41:01.684 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-06-04T04:41:01.684 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-06-04T04:41:01.685 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-06-04T04:41:01.685 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.685 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.685 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-06-04T04:41:01.685 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-06-04T04:41:01.685 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-04T04:41:01.685 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-04T04:41:01.685 INFO:teuthology.orchestra.run.smithi098.stdout: automake-1.16.2-8.el9.noarch 2024-06-04T04:41:01.685 INFO:teuthology.orchestra.run.smithi098.stdout: bison-3.7.4-5.el9.x86_64 2024-06-04T04:41:01.685 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-04T04:41:01.685 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.685 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.685 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.685 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.686 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:41:01.686 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.686 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.686 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.686 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:41:01.686 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:41:01.686 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:41:01.686 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:41:01.686 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:41:01.686 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.686 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.686 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:41:01.687 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.687 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.687 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.687 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:41:01.687 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:41:01.687 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.687 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.687 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:41:01.687 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-06-04T04:41:01.688 INFO:teuthology.orchestra.run.smithi098.stdout: flex-2.6.4-9.el9.x86_64 2024-06-04T04:41:01.688 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-04T04:41:01.688 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-04T04:41:01.688 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-04T04:41:01.688 INFO:teuthology.orchestra.run.smithi098.stdout: fuse-2.9.9-15.el9.x86_64 2024-06-04T04:41:01.688 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-04T04:41:01.688 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-04T04:41:01.688 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-06-04T04:41:01.688 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-04T04:41:01.688 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-04T04:41:01.688 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.688 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.688 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.688 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-04T04:41:01.688 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-04T04:41:01.688 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-04T04:41:01.689 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-04T04:41:01.689 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-04T04:41:01.689 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.689 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.689 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-04T04:41:01.689 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.689 INFO:teuthology.orchestra.run.smithi098.stdout: libtool-2.4.6-45.el9.x86_64 2024-06-04T04:41:01.689 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-04T04:41:01.689 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-06-04T04:41:01.689 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-06-04T04:41:01.689 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-04T04:41:01.689 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2024-06-04T04:41:01.689 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-04T04:41:01.689 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-04T04:41:01.689 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-04T04:41:01.690 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-04T04:41:01.690 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-06-04T04:41:01.690 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-04T04:41:01.690 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-13.14-1.el9.x86_64 2024-06-04T04:41:01.690 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-06-04T04:41:01.690 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-06-04T04:41:01.690 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server-13.14-1.el9.x86_64 2024-06-04T04:41:01.690 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-04T04:41:01.690 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-04T04:41:01.690 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-04T04:41:01.690 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-06-04T04:41:01.690 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-04T04:41:01.690 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-04T04:41:01.690 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-04T04:41:01.691 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.691 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.691 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.691 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-04T04:41:01.691 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-04T04:41:01.691 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-04T04:41:01.691 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-04T04:41:01.691 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-06-04T04:41:01.691 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-06-04T04:41:01.691 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-04T04:41:01.691 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-04T04:41:01.691 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-06-04T04:41:01.691 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-04T04:41:01.691 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-04T04:41:01.691 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-04T04:41:01.692 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-04T04:41:01.692 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-04T04:41:01.692 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-04T04:41:01.692 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-04T04:41:01.692 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-04T04:41:01.692 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-04T04:41:01.692 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-04T04:41:01.692 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-04T04:41:01.692 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-04T04:41:01.692 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-04T04:41:01.692 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-04T04:41:01.692 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-04T04:41:01.692 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-04T04:41:01.692 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-04T04:41:01.693 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-04T04:41:01.693 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-04T04:41:01.693 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-04T04:41:01.693 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-04T04:41:01.693 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-06-04T04:41:01.693 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-04T04:41:01.693 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-04T04:41:01.693 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-04T04:41:01.693 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-04T04:41:01.693 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-04T04:41:01.693 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-04T04:41:01.693 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-04T04:41:01.693 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-06-04T04:41:01.693 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-04T04:41:01.694 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.694 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.694 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-04T04:41:01.694 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-04T04:41:01.694 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-04T04:41:01.694 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.694 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-04T04:41:01.694 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-04T04:41:01.694 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-04T04:41:01.694 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-04T04:41:01.694 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-04T04:41:01.694 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-04T04:41:01.694 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-04T04:41:01.694 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-06-04T04:41:01.694 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-04T04:41:01.695 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-04T04:41:01.695 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-04T04:41:01.695 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-04T04:41:01.695 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-04T04:41:01.695 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-04T04:41:01.695 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-04T04:41:01.695 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.695 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.695 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:41:01.695 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-3.el9.x86_64 2024-06-04T04:41:01.695 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-04T04:41:01.695 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-04T04:41:01.695 INFO:teuthology.orchestra.run.smithi098.stdout: uuid-1.6.2-55.el9.x86_64 2024-06-04T04:41:01.695 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-06-04T04:41:01.695 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-04T04:41:01.696 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-06-04T04:41:01.696 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-04T04:41:01.696 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-04T04:41:02.152 DEBUG:teuthology.parallel:result is None 2024-06-04T04:41:30.161 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 180 kB/s | 21 MB 02:00 2024-06-04T04:41:35.285 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-06-04T04:41:37.636 INFO:teuthology.orchestra.run.smithi070.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-04T04:41:37.637 INFO:teuthology.orchestra.run.smithi070.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-04T04:41:37.647 INFO:teuthology.orchestra.run.smithi070.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-06-04T04:41:37.648 INFO:teuthology.orchestra.run.smithi070.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-06-04T04:41:37.649 INFO:teuthology.orchestra.run.smithi070.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-06-04T04:41:37.650 INFO:teuthology.orchestra.run.smithi070.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-06-04T04:41:37.651 INFO:teuthology.orchestra.run.smithi070.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-06-04T04:41:37.653 INFO:teuthology.orchestra.run.smithi070.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-06-04T04:41:37.657 INFO:teuthology.orchestra.run.smithi070.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-06-04T04:41:37.716 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-06-04T04:41:37.740 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================= 2024-06-04T04:41:37.740 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-06-04T04:41:37.740 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================= 2024-06-04T04:41:37.740 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-06-04T04:41:37.740 INFO:teuthology.orchestra.run.smithi070.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-06-04T04:41:37.740 INFO:teuthology.orchestra.run.smithi070.stdout: ceph x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 6.6 k 2024-06-04T04:41:37.741 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 5.4 M 2024-06-04T04:41:37.741 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 890 k 2024-06-04T04:41:37.741 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 145 k 2024-06-04T04:41:37.741 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 1.5 M 2024-06-04T04:41:37.741 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 143 k 2024-06-04T04:41:37.741 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 3.5 M 2024-06-04T04:41:37.741 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 7.4 M 2024-06-04T04:41:37.741 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 49 k 2024-06-04T04:41:37.741 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 11 M 2024-06-04T04:41:37.741 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 49 M 2024-06-04T04:41:37.741 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 273 k 2024-06-04T04:41:37.741 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 763 k 2024-06-04T04:41:37.741 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-mirror x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 225 k 2024-06-04T04:41:37.742 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-shell x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 27 k 2024-06-04T04:41:37.742 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-top noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 18 k 2024-06-04T04:41:37.742 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-06-04T04:41:37.742 INFO:teuthology.orchestra.run.smithi070.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-06-04T04:41:37.742 INFO:teuthology.orchestra.run.smithi070.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-06-04T04:41:37.742 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 32 k 2024-06-04T04:41:37.742 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2 x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 747 k 2024-06-04T04:41:37.742 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 127 k 2024-06-04T04:41:37.742 INFO:teuthology.orchestra.run.smithi070.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-06-04T04:41:37.742 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-06-04T04:41:37.742 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-06-04T04:41:37.742 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-06-04T04:41:37.743 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-06-04T04:41:37.743 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-06-04T04:41:37.743 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 161 k 2024-06-04T04:41:37.743 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 320 k 2024-06-04T04:41:37.743 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 300 k 2024-06-04T04:41:37.743 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 100 k 2024-06-04T04:41:37.743 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 85 k 2024-06-04T04:41:37.743 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 3.1 M 2024-06-04T04:41:37.743 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 170 k 2024-06-04T04:41:37.743 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-06-04T04:41:37.743 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrading: 2024-06-04T04:41:37.743 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager x86_64 1:1.47.91-1.el9 baseos 2.3 M 2024-06-04T04:41:37.743 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-libnm x86_64 1:1.47.91-1.el9 baseos 1.8 M 2024-06-04T04:41:37.744 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-team x86_64 1:1.47.91-1.el9 baseos 41 k 2024-06-04T04:41:37.744 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-tui x86_64 1:1.47.91-1.el9 baseos 248 k 2024-06-04T04:41:37.744 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-06-04T04:41:37.744 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-06-04T04:41:37.744 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-06-04T04:41:37.744 INFO:teuthology.orchestra.run.smithi070.stdout: librados2 x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 3.4 M 2024-06-04T04:41:37.744 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1 x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 3.1 M 2024-06-04T04:41:37.744 INFO:teuthology.orchestra.run.smithi070.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-06-04T04:41:37.744 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-06-04T04:41:37.744 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-06-04T04:41:37.744 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-04T04:41:37.744 INFO:teuthology.orchestra.run.smithi070.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-06-04T04:41:37.745 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-04T04:41:37.745 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 21 M 2024-06-04T04:41:37.745 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 34 k 2024-06-04T04:41:37.745 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 2.4 M 2024-06-04T04:41:37.745 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 274 k 2024-06-04T04:41:37.745 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 4.7 M 2024-06-04T04:41:37.745 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 17 M 2024-06-04T04:41:37.745 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3882.gb7328fed.el9 ceph-noarch 16 k 2024-06-04T04:41:37.745 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 25 k 2024-06-04T04:41:37.745 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-04T04:41:37.745 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-04T04:41:37.745 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-04T04:41:37.745 INFO:teuthology.orchestra.run.smithi070.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-04T04:41:37.745 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-04T04:41:37.746 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-04T04:41:37.746 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-04T04:41:37.746 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-04T04:41:37.746 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 163 k 2024-06-04T04:41:37.746 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-04T04:41:37.746 INFO:teuthology.orchestra.run.smithi070.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-04T04:41:37.746 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-04T04:41:37.746 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-04T04:41:37.746 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-04T04:41:37.746 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1 x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 501 k 2024-06-04T04:41:37.746 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-04T04:41:37.746 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2 x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 5.4 M 2024-06-04T04:41:37.747 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-04T04:41:37.747 INFO:teuthology.orchestra.run.smithi070.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-06-04T04:41:37.747 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-04T04:41:37.747 INFO:teuthology.orchestra.run.smithi070.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-04T04:41:37.747 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-04T04:41:37.747 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-04T04:41:37.747 INFO:teuthology.orchestra.run.smithi070.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-04T04:41:37.747 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-04T04:41:37.747 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-04T04:41:37.747 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-06-04T04:41:37.747 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-04T04:41:37.747 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-04T04:41:37.747 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-04T04:41:37.747 INFO:teuthology.orchestra.run.smithi070.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-06-04T04:41:37.748 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-04T04:41:37.748 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-04T04:41:37.748 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-04T04:41:37.748 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 45 k 2024-06-04T04:41:37.748 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common x86_64 2:19.0.0-3882.gb7328fed.el9 ceph 135 k 2024-06-04T04:41:37.748 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-04T04:41:37.748 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-04T04:41:37.748 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-04T04:41:37.748 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-04T04:41:37.748 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-06-04T04:41:37.748 INFO:teuthology.orchestra.run.smithi070.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-06-04T04:41:37.748 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-04T04:41:37.748 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-04T04:41:37.748 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-06-04T04:41:37.748 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-04T04:41:37.749 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-04T04:41:37.749 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-04T04:41:37.749 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-04T04:41:37.749 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-04T04:41:37.749 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-04T04:41:37.749 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-04T04:41:37.749 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-04T04:41:37.749 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-04T04:41:37.749 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-04T04:41:37.749 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-04T04:41:37.749 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-04T04:41:37.749 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-04T04:41:37.749 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-04T04:41:37.749 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-04T04:41:37.749 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-04T04:41:37.750 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-04T04:41:37.750 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-04T04:41:37.750 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-04T04:41:37.750 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-04T04:41:37.750 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-04T04:41:37.750 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-04T04:41:37.750 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-04T04:41:37.750 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-04T04:41:37.750 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-04T04:41:37.750 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-04T04:41:37.750 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-04T04:41:37.750 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-04T04:41:37.751 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-04T04:41:37.751 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-06-04T04:41:37.751 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-04T04:41:37.751 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-04T04:41:37.751 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-04T04:41:37.751 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-04T04:41:37.751 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-04T04:41:37.751 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-04T04:41:37.751 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-04T04:41:37.751 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-04T04:41:37.751 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-04T04:41:37.751 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-04T04:41:37.751 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-04T04:41:37.752 INFO:teuthology.orchestra.run.smithi070.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-06-04T04:41:37.752 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-04T04:41:37.752 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-04T04:41:37.752 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-04T04:41:37.752 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-04T04:41:37.752 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-04T04:41:37.752 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-04T04:41:37.752 INFO:teuthology.orchestra.run.smithi070.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-04T04:41:37.752 INFO:teuthology.orchestra.run.smithi070.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-04T04:41:37.752 INFO:teuthology.orchestra.run.smithi070.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-04T04:41:37.752 INFO:teuthology.orchestra.run.smithi070.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-06-04T04:41:37.752 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-04T04:41:37.752 INFO:teuthology.orchestra.run.smithi070.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-06-04T04:41:37.753 INFO:teuthology.orchestra.run.smithi070.stdout:Installing weak dependencies: 2024-06-04T04:41:37.753 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-04T04:41:37.753 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:41:37.753 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-06-04T04:41:37.753 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================= 2024-06-04T04:41:37.753 INFO:teuthology.orchestra.run.smithi070.stdout:Install 148 Packages 2024-06-04T04:41:37.753 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrade 11 Packages 2024-06-04T04:41:37.753 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:41:37.754 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 227 M 2024-06-04T04:41:37.754 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-06-04T04:41:38.545 INFO:teuthology.orchestra.run.smithi070.stdout:(1/159): ceph-19.0.0-3882.gb7328fed.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-06-04T04:41:39.428 INFO:teuthology.orchestra.run.smithi070.stdout:(2/159): ceph-fuse-19.0.0-3882.gb7328fed.el9.x8 1.0 MB/s | 890 kB 00:00 2024-06-04T04:41:39.812 INFO:teuthology.orchestra.run.smithi070.stdout:(3/159): ceph-immutable-object-cache-19.0.0-388 378 kB/s | 145 kB 00:00 2024-06-04T04:41:41.212 INFO:teuthology.orchestra.run.smithi070.stdout:(4/159): ceph-base-19.0.0-3882.gb7328fed.el9.x8 1.8 MB/s | 5.4 MB 00:02 2024-06-04T04:41:41.428 INFO:teuthology.orchestra.run.smithi070.stdout:(5/159): ceph-mds-19.0.0-3882.gb7328fed.el9.x86 1.5 MB/s | 2.4 MB 00:01 2024-06-04T04:41:41.895 INFO:teuthology.orchestra.run.smithi070.stdout:(6/159): ceph-mgr-19.0.0-3882.gb7328fed.el9.x86 2.2 MB/s | 1.5 MB 00:00 2024-06-04T04:41:44.304 INFO:teuthology.orchestra.run.smithi070.stdout:(7/159): ceph-mon-19.0.0-3882.gb7328fed.el9.x86 1.6 MB/s | 4.7 MB 00:02 2024-06-04T04:41:45.787 INFO:teuthology.orchestra.run.smithi070.stdout:(8/159): ceph-common-19.0.0-3882.gb7328fed.el9. 2.8 MB/s | 21 MB 00:07 2024-06-04T04:41:45.862 INFO:teuthology.orchestra.run.smithi070.stdout:(9/159): ceph-selinux-19.0.0-3882.gb7328fed.el9 336 kB/s | 25 kB 00:00 2024-06-04T04:41:46.229 INFO:teuthology.orchestra.run.smithi070.stdout:(10/159): ceph-osd-19.0.0-3882.gb7328fed.el9.x8 3.9 MB/s | 17 MB 00:04 2024-06-04T04:41:46.354 INFO:teuthology.orchestra.run.smithi070.stdout:(11/159): cephfs-mirror-19.0.0-3882.gb7328fed.e 1.8 MB/s | 225 kB 00:00 2024-06-04T04:41:46.413 INFO:teuthology.orchestra.run.smithi070.stdout:(12/159): cephfs-shell-19.0.0-3882.gb7328fed.el 466 kB/s | 27 kB 00:00 2024-06-04T04:41:46.471 INFO:teuthology.orchestra.run.smithi070.stdout:(13/159): libcephfs-devel-19.0.0-3882.gb7328fed 542 kB/s | 32 kB 00:00 2024-06-04T04:41:46.655 INFO:teuthology.orchestra.run.smithi070.stdout:(14/159): libcephfs2-19.0.0-3882.gb7328fed.el9. 4.0 MB/s | 747 kB 00:00 2024-06-04T04:41:46.722 INFO:teuthology.orchestra.run.smithi070.stdout:(15/159): libcephsqlite-19.0.0-3882.gb7328fed.e 2.4 MB/s | 163 kB 00:00 2024-06-04T04:41:46.789 INFO:teuthology.orchestra.run.smithi070.stdout:(16/159): librados-devel-19.0.0-3882.gb7328fed. 1.9 MB/s | 127 kB 00:00 2024-06-04T04:41:46.931 INFO:teuthology.orchestra.run.smithi070.stdout:(17/159): libradosstriper1-19.0.0-3882.gb7328fe 3.5 MB/s | 501 kB 00:00 2024-06-04T04:41:47.306 INFO:teuthology.orchestra.run.smithi070.stdout:(18/159): ceph-radosgw-19.0.0-3882.gb7328fed.el 3.6 MB/s | 11 MB 00:03 2024-06-04T04:41:47.373 INFO:teuthology.orchestra.run.smithi070.stdout:(19/159): python3-ceph-argparse-19.0.0-3882.gb7 679 kB/s | 45 kB 00:00 2024-06-04T04:41:47.440 INFO:teuthology.orchestra.run.smithi070.stdout:(20/159): python3-ceph-common-19.0.0-3882.gb732 2.0 MB/s | 135 kB 00:00 2024-06-04T04:41:47.507 INFO:teuthology.orchestra.run.smithi070.stdout:(21/159): python3-cephfs-19.0.0-3882.gb7328fed. 2.4 MB/s | 161 kB 00:00 2024-06-04T04:41:47.616 INFO:teuthology.orchestra.run.smithi070.stdout:(22/159): python3-rados-19.0.0-3882.gb7328fed.e 2.9 MB/s | 320 kB 00:00 2024-06-04T04:41:47.775 INFO:teuthology.orchestra.run.smithi070.stdout:(23/159): python3-rbd-19.0.0-3882.gb7328fed.el9 1.9 MB/s | 300 kB 00:00 2024-06-04T04:41:47.842 INFO:teuthology.orchestra.run.smithi070.stdout:(24/159): python3-rgw-19.0.0-3882.gb7328fed.el9 1.5 MB/s | 100 kB 00:00 2024-06-04T04:41:47.909 INFO:teuthology.orchestra.run.smithi070.stdout:(25/159): rbd-fuse-19.0.0-3882.gb7328fed.el9.x8 1.2 MB/s | 85 kB 00:00 2024-06-04T04:41:48.050 INFO:teuthology.orchestra.run.smithi070.stdout:(26/159): librgw2-19.0.0-3882.gb7328fed.el9.x86 4.8 MB/s | 5.4 MB 00:01 2024-06-04T04:41:48.092 INFO:teuthology.orchestra.run.smithi070.stdout:(27/159): rbd-nbd-19.0.0-3882.gb7328fed.el9.x86 4.0 MB/s | 170 kB 00:00 2024-06-04T04:41:48.126 INFO:teuthology.orchestra.run.smithi070.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3882.g 1.0 MB/s | 34 kB 00:00 2024-06-04T04:41:48.176 INFO:teuthology.orchestra.run.smithi070.stdout:(29/159): ceph-mgr-cephadm-19.0.0-3882.gb7328fe 2.8 MB/s | 143 kB 00:00 2024-06-04T04:41:48.494 INFO:teuthology.orchestra.run.smithi070.stdout:(30/159): rbd-mirror-19.0.0-3882.gb7328fed.el9. 5.3 MB/s | 3.1 MB 00:00 2024-06-04T04:41:48.727 INFO:teuthology.orchestra.run.smithi070.stdout:(31/159): ceph-mgr-dashboard-19.0.0-3882.gb7328 6.4 MB/s | 3.5 MB 00:00 2024-06-04T04:41:48.835 INFO:teuthology.orchestra.run.smithi070.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3882.gb7 2.5 MB/s | 274 kB 00:00 2024-06-04T04:41:48.894 INFO:teuthology.orchestra.run.smithi070.stdout:(33/159): ceph-mgr-rook-19.0.0-3882.gb7328fed.e 849 kB/s | 49 kB 00:00 2024-06-04T04:41:48.953 INFO:teuthology.orchestra.run.smithi070.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3882.gb 282 kB/s | 16 kB 00:00 2024-06-04T04:41:49.070 INFO:teuthology.orchestra.run.smithi070.stdout:(35/159): ceph-volume-19.0.0-3882.gb7328fed.el9 2.3 MB/s | 273 kB 00:00 2024-06-04T04:41:49.237 INFO:teuthology.orchestra.run.smithi070.stdout:(36/159): cephadm-19.0.0-3882.gb7328fed.el9.noa 4.5 MB/s | 763 kB 00:00 2024-06-04T04:41:49.287 INFO:teuthology.orchestra.run.smithi070.stdout:(37/159): cephfs-top-19.0.0-3882.gb7328fed.el9. 366 kB/s | 18 kB 00:00 2024-06-04T04:41:49.729 INFO:teuthology.orchestra.run.smithi070.stdout:(38/159): ceph-mgr-diskprediction-local-19.0.0- 6.0 MB/s | 7.4 MB 00:01 2024-06-04T04:41:56.653 INFO:teuthology.orchestra.run.smithi070.stdout:(39/159): ceph-test-19.0.0-3882.gb7328fed.el9.x 4.6 MB/s | 49 MB 00:10 2024-06-04T04:42:01.865 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:42:02.160 INFO:teuthology.orchestra.run.smithi070.stdout:(40/159): libquadmath-11.4.1-3.el9.x86_64.rpm 35 kB/s | 195 kB 00:05 2024-06-04T04:42:02.164 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:42:02.294 INFO:teuthology.orchestra.run.smithi070.stdout:(41/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-06-04T04:42:02.301 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] python3-cryptography-36.0.1-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cryptography-36.0.1-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:42:02.527 INFO:teuthology.orchestra.run.smithi070.stdout:(42/159): python3-cryptography-36.0.1-4.el9.x86 5.3 MB/s | 1.2 MB 00:00 2024-06-04T04:42:02.603 INFO:teuthology.orchestra.run.smithi070.stdout:(43/159): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-06-04T04:42:02.678 INFO:teuthology.orchestra.run.smithi070.stdout:(44/159): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 135 kB 00:00 2024-06-04T04:42:02.736 INFO:teuthology.orchestra.run.smithi070.stdout:(45/159): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-06-04T04:42:02.795 INFO:teuthology.orchestra.run.smithi070.stdout:(46/159): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-06-04T04:42:02.798 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] automake-1.16.2-8.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/automake-1.16.2-8.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-04T04:42:02.979 INFO:teuthology.orchestra.run.smithi070.stdout:(47/159): automake-1.16.2-8.el9.noarch.rpm 3.7 MB/s | 689 kB 00:00 2024-06-04T04:42:02.982 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] bison-3.7.4-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/bison-3.7.4-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:42:03.179 INFO:teuthology.orchestra.run.smithi070.stdout:(48/159): bison-3.7.4-5.el9.x86_64.rpm 4.6 MB/s | 943 kB 00:00 2024-06-04T04:42:03.183 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] boost-program-options-1.75.0-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:42:03.238 INFO:teuthology.orchestra.run.smithi070.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-06-04T04:42:03.280 INFO:teuthology.orchestra.run.smithi070.stdout:(50/159): elfutils-libelf-devel-0.191-4.el9.x86 522 kB/s | 22 kB 00:00 2024-06-04T04:42:03.389 INFO:teuthology.orchestra.run.smithi070.stdout:(51/159): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-06-04T04:42:03.430 INFO:teuthology.orchestra.run.smithi070.stdout:(52/159): flexiblas-3.0.4-8.el9.x86_64.rpm 788 kB/s | 33 kB 00:00 2024-06-04T04:42:04.339 INFO:teuthology.orchestra.run.smithi070.stdout:(53/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.3 MB/s | 3.0 MB 00:00 2024-06-04T04:42:04.372 INFO:teuthology.orchestra.run.smithi070.stdout:(54/159): flexiblas-openblas-openmp-3.0.4-8.el9 548 kB/s | 18 kB 00:00 2024-06-04T04:42:04.414 INFO:teuthology.orchestra.run.smithi070.stdout:(55/159): libaio-devel-0.3.111-13.el9.x86_64.rp 267 kB/s | 11 kB 00:00 2024-06-04T04:42:04.489 INFO:teuthology.orchestra.run.smithi070.stdout:(56/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-06-04T04:42:04.531 INFO:teuthology.orchestra.run.smithi070.stdout:(57/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-06-04T04:42:04.740 INFO:teuthology.orchestra.run.smithi070.stdout:(58/159): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-06-04T04:42:04.940 INFO:teuthology.orchestra.run.smithi070.stdout:(59/159): libtool-2.4.6-45.el9.x86_64.rpm 2.8 MB/s | 582 kB 00:00 2024-06-04T04:42:04.974 INFO:teuthology.orchestra.run.smithi070.stdout:(60/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 761 kB/s | 25 kB 00:00 2024-06-04T04:42:05.007 INFO:teuthology.orchestra.run.smithi070.stdout:(61/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-06-04T04:42:05.132 INFO:teuthology.orchestra.run.smithi070.stdout:(62/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-06-04T04:42:05.208 INFO:teuthology.orchestra.run.smithi070.stdout:(63/159): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 188 kB 00:00 2024-06-04T04:42:05.241 INFO:teuthology.orchestra.run.smithi070.stdout:(64/159): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 35 kB 00:00 2024-06-04T04:42:06.617 INFO:teuthology.orchestra.run.smithi070.stdout:(65/159): openblas-openmp-0.3.21-2.el9.x86_64.r 3.4 MB/s | 4.7 MB 00:01 2024-06-04T04:42:07.950 INFO:teuthology.orchestra.run.smithi070.stdout:(66/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.3 MB/s | 4.4 MB 00:01 2024-06-04T04:42:08.408 INFO:teuthology.orchestra.run.smithi070.stdout:(67/159): postgresql-13.14-1.el9.x86_64.rpm 3.5 MB/s | 1.6 MB 00:00 2024-06-04T04:42:08.692 INFO:teuthology.orchestra.run.smithi070.stdout:(68/159): postgresql-contrib-13.14-1.el9.x86_64 3.1 MB/s | 911 kB 00:00 2024-06-04T04:42:08.767 INFO:teuthology.orchestra.run.smithi070.stdout:(69/159): postgresql-private-libs-13.14-1.el9.x 1.7 MB/s | 133 kB 00:00 2024-06-04T04:42:10.409 INFO:teuthology.orchestra.run.smithi070.stdout:(70/159): postgresql-server-13.14-1.el9.x86_64. 3.5 MB/s | 5.8 MB 00:01 2024-06-04T04:42:10.726 INFO:teuthology.orchestra.run.smithi070.stdout:(71/159): protobuf-3.14.0-13.el9.x86_64.rpm 3.2 MB/s | 1.0 MB 00:00 2024-06-04T04:42:10.792 INFO:teuthology.orchestra.run.smithi070.stdout:(72/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-06-04T04:42:12.526 INFO:teuthology.orchestra.run.smithi070.stdout:(73/159): python3-babel-2.9.1-2.el9.noarch.rpm 3.4 MB/s | 6.0 MB 00:01 2024-06-04T04:42:12.568 INFO:teuthology.orchestra.run.smithi070.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-06-04T04:42:12.660 INFO:teuthology.orchestra.run.smithi070.stdout:(75/159): python3-devel-3.9.18-3.el9.x86_64.rpm 2.6 MB/s | 244 kB 00:00 2024-06-04T04:42:12.752 INFO:teuthology.orchestra.run.smithi070.stdout:(76/159): python3-jinja2-2.11.3-6.el9.noarch.rp 2.6 MB/s | 248 kB 00:00 2024-06-04T04:42:12.802 INFO:teuthology.orchestra.run.smithi070.stdout:(77/159): python3-jsonpatch-1.21-16.el9.noarch. 510 kB/s | 26 kB 00:00 2024-06-04T04:42:12.844 INFO:teuthology.orchestra.run.smithi070.stdout:(78/159): python3-jsonpointer-2.0-4.el9.noarch. 460 kB/s | 19 kB 00:00 2024-06-04T04:42:12.919 INFO:teuthology.orchestra.run.smithi070.stdout:(79/159): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-06-04T04:42:12.961 INFO:teuthology.orchestra.run.smithi070.stdout:(80/159): python3-markupsafe-1.1.1-12.el9.x86_6 836 kB/s | 35 kB 00:00 2024-06-04T04:42:14.436 INFO:teuthology.orchestra.run.smithi070.stdout:(81/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.5 MB/s | 5.1 MB 00:01 2024-06-04T04:42:14.528 INFO:teuthology.orchestra.run.smithi070.stdout:(82/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.7 MB/s | 250 kB 00:00 2024-06-04T04:42:14.620 INFO:teuthology.orchestra.run.smithi070.stdout:(83/159): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-06-04T04:42:14.679 INFO:teuthology.orchestra.run.smithi070.stdout:(84/159): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-06-04T04:42:14.721 INFO:teuthology.orchestra.run.smithi070.stdout:(85/159): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-06-04T04:42:14.837 INFO:teuthology.orchestra.run.smithi070.stdout:(86/159): python3-protobuf-3.14.0-13.el9.noarch 2.3 MB/s | 269 kB 00:00 2024-06-04T04:42:14.913 INFO:teuthology.orchestra.run.smithi070.stdout:(87/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-06-04T04:42:15.021 INFO:teuthology.orchestra.run.smithi070.stdout:(88/159): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-06-04T04:42:15.066 INFO:teuthology.orchestra.run.smithi070.stdout:(89/159): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-06-04T04:42:15.105 INFO:teuthology.orchestra.run.smithi070.stdout:(90/159): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 54 kB 00:00 2024-06-04T04:42:19.705 INFO:teuthology.orchestra.run.smithi070.stdout:(91/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.5 MB/s | 16 MB 00:04 2024-06-04T04:42:19.746 INFO:teuthology.orchestra.run.smithi070.stdout:(92/159): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-06-04T04:42:19.788 INFO:teuthology.orchestra.run.smithi070.stdout:(93/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-06-04T04:42:19.889 INFO:teuthology.orchestra.run.smithi070.stdout:(94/159): qatlib-24.02.0-1.el9.x86_64.rpm 2.2 MB/s | 221 kB 00:00 2024-06-04T04:42:19.930 INFO:teuthology.orchestra.run.smithi070.stdout:(95/159): qatlib-service-24.02.0-1.el9.x86_64.r 862 kB/s | 36 kB 00:00 2024-06-04T04:42:19.972 INFO:teuthology.orchestra.run.smithi070.stdout:(96/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-06-04T04:42:20.089 INFO:teuthology.orchestra.run.smithi070.stdout:(97/159): socat-1.7.4.1-5.el9.x86_64.rpm 2.6 MB/s | 305 kB 00:00 2024-06-04T04:42:20.131 INFO:teuthology.orchestra.run.smithi070.stdout:(98/159): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-06-04T04:42:20.190 INFO:teuthology.orchestra.run.smithi070.stdout:(99/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-06-04T04:42:20.249 INFO:teuthology.orchestra.run.smithi070.stdout:(100/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-06-04T04:42:20.290 INFO:teuthology.orchestra.run.smithi070.stdout:(101/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-06-04T04:42:56.594 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:42:56.866 INFO:teuthology.orchestra.run.smithi070.stdout:(102/159): lua-devel-5.4.4-4.el9.x86_64.rpm 623 B/s | 22 kB 00:36 2024-06-04T04:42:56.869 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] protobuf-compiler-3.14.0-13.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-13.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:42:56.975 INFO:teuthology.orchestra.run.smithi070.stdout:(103/159): protobuf-compiler-3.14.0-13.el9.x86_ 7.8 MB/s | 863 kB 00:00 2024-06-04T04:42:57.112 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:42:57.248 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-04T04:42:57.292 INFO:teuthology.orchestra.run.smithi070.stdout:(104/159): fuse-2.9.9-15.el9.x86_64.rpm 1.2 kB/s | 80 kB 01:08 2024-06-04T04:42:57.576 INFO:teuthology.orchestra.run.smithi070.stdout:(105/159): libgfortran-11.4.1-3.el9.x86_64.rpm 12 kB/s | 803 kB 01:07 2024-06-04T04:42:57.634 INFO:teuthology.orchestra.run.smithi070.stdout:(106/159): grpc-data-1.46.7-10.el9.noarch.rpm 335 kB/s | 19 kB 00:00 2024-06-04T04:42:57.876 INFO:teuthology.orchestra.run.smithi070.stdout:(107/159): gperftools-libs-2.9.1-2.el9.x86_64.r 529 kB/s | 309 kB 00:00 2024-06-04T04:42:57.943 INFO:teuthology.orchestra.run.smithi070.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 390 kB/s | 26 kB 00:00 2024-06-04T04:42:57.984 INFO:teuthology.orchestra.run.smithi070.stdout:(109/159): abseil-cpp-20211102.0-3.el9.x86_64.r 543 kB/s | 548 kB 00:01 2024-06-04T04:42:58.052 INFO:teuthology.orchestra.run.smithi070.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-06-04T04:42:58.152 INFO:teuthology.orchestra.run.smithi070.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 403 kB/s | 67 kB 00:00 2024-06-04T04:42:58.352 INFO:teuthology.orchestra.run.smithi070.stdout:(112/159): luarocks-3.9.2-5.el9.noarch.rpm 504 kB/s | 151 kB 00:00 2024-06-04T04:42:59.361 INFO:teuthology.orchestra.run.smithi070.stdout:(113/159): python3-asyncssh-2.13.2-5.el9.noarch 544 kB/s | 548 kB 00:01 2024-06-04T04:42:59.478 INFO:teuthology.orchestra.run.smithi070.stdout:(114/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 373 kB/s | 43 kB 00:00 2024-06-04T04:42:59.561 INFO:teuthology.orchestra.run.smithi070.stdout:(115/159): python3-cachetools-4.2.4-1.el9.noarc 386 kB/s | 32 kB 00:00 2024-06-04T04:42:59.628 INFO:teuthology.orchestra.run.smithi070.stdout:(116/159): python3-certifi-2021.10.8-2.el9.noar 225 kB/s | 15 kB 00:00 2024-06-04T04:42:59.737 INFO:teuthology.orchestra.run.smithi070.stdout:(117/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 529 kB/s | 839 kB 00:01 2024-06-04T04:42:59.929 INFO:teuthology.orchestra.run.smithi070.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 572 kB/s | 172 kB 00:00 2024-06-04T04:43:00.371 INFO:teuthology.orchestra.run.smithi070.stdout:(119/159): python3-cherrypy-18.6.1-2.el9.noarch 566 kB/s | 358 kB 00:00 2024-06-04T04:43:00.496 INFO:teuthology.orchestra.run.smithi070.stdout:(120/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 535 kB/s | 303 kB 00:00 2024-06-04T04:43:00.754 INFO:teuthology.orchestra.run.smithi070.stdout:(121/159): python3-google-auth-2.29.0-1.el9.noa 536 kB/s | 206 kB 00:00 2024-06-04T04:43:01.038 INFO:teuthology.orchestra.run.smithi070.stdout:(122/159): python3-grpcio-tools-1.46.7-10.el9.x 510 kB/s | 144 kB 00:00 2024-06-04T04:43:01.080 INFO:teuthology.orchestra.run.smithi070.stdout:(123/159): python3-jaraco-8.2.1-3.el9.noarch.rp 258 kB/s | 11 kB 00:00 2024-06-04T04:43:01.130 INFO:teuthology.orchestra.run.smithi070.stdout:(124/159): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-06-04T04:43:01.197 INFO:teuthology.orchestra.run.smithi070.stdout:(125/159): python3-jaraco-collections-3.0.0-8.e 348 kB/s | 23 kB 00:00 2024-06-04T04:43:01.256 INFO:teuthology.orchestra.run.smithi070.stdout:(126/159): python3-jaraco-functools-3.5.0-2.el9 334 kB/s | 19 kB 00:00 2024-06-04T04:43:01.314 INFO:teuthology.orchestra.run.smithi070.stdout:(127/159): python3-jaraco-text-3.2.0-6.el9.noar 338 kB/s | 20 kB 00:00 2024-06-04T04:43:03.164 INFO:teuthology.orchestra.run.smithi070.stdout:(128/159): python3-kubernetes-26.1.0-2.el9.noar 565 kB/s | 1.0 MB 00:01 2024-06-04T04:43:03.273 INFO:teuthology.orchestra.run.smithi070.stdout:(129/159): python3-logutils-0.3.5-21.el9.noarch 431 kB/s | 46 kB 00:00 2024-06-04T04:43:03.431 INFO:teuthology.orchestra.run.smithi070.stdout:(130/159): python3-more-itertools-8.12.0-2.el9. 498 kB/s | 79 kB 00:00 2024-06-04T04:43:03.790 INFO:teuthology.orchestra.run.smithi070.stdout:(131/159): python3-natsort-7.1.1-5.el9.noarch.r 161 kB/s | 58 kB 00:00 2024-06-04T04:43:04.232 INFO:teuthology.orchestra.run.smithi070.stdout:(132/159): python3-pecan-1.4.2-3.el9.noarch.rpm 617 kB/s | 272 kB 00:00 2024-06-04T04:43:04.291 INFO:teuthology.orchestra.run.smithi070.stdout:(133/159): python3-portend-3.1.0-2.el9.noarch.r 282 kB/s | 16 kB 00:00 2024-06-04T04:43:04.383 INFO:teuthology.orchestra.run.smithi070.stdout:(134/159): python3-grpcio-1.46.7-10.el9.x86_64. 537 kB/s | 2.0 MB 00:03 2024-06-04T04:43:04.466 INFO:teuthology.orchestra.run.smithi070.stdout:(135/159): python3-pyperclip-1.8.2-2.el9.noarch 339 kB/s | 28 kB 00:00 2024-06-04T04:43:04.492 INFO:teuthology.orchestra.run.smithi070.stdout:(136/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 448 kB/s | 90 kB 00:00 2024-06-04T04:43:04.550 INFO:teuthology.orchestra.run.smithi070.stdout:(137/159): python3-repoze-lru-0.7-16.el9.noarch 368 kB/s | 31 kB 00:00 2024-06-04T04:43:04.700 INFO:teuthology.orchestra.run.smithi070.stdout:(138/159): python3-rsa-4.9-2.el9.noarch.rpm 395 kB/s | 59 kB 00:00 2024-06-04T04:43:04.801 INFO:teuthology.orchestra.run.smithi070.stdout:(139/159): python3-tempora-5.0.0-2.el9.noarch.r 358 kB/s | 36 kB 00:00 2024-06-04T04:43:04.851 INFO:teuthology.orchestra.run.smithi070.stdout:(140/159): python3-routes-2.5.1-5.el9.noarch.rp 525 kB/s | 188 kB 00:00 2024-06-04T04:43:04.935 INFO:teuthology.orchestra.run.smithi070.stdout:(141/159): python3-typing-extensions-4.4.0-2.el 382 kB/s | 51 kB 00:00 2024-06-04T04:43:05.152 INFO:teuthology.orchestra.run.smithi070.stdout:(142/159): python3-websocket-client-1.2.3-2.el9 414 kB/s | 90 kB 00:00 2024-06-04T04:43:05.277 INFO:teuthology.orchestra.run.smithi070.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 540 kB/s | 230 kB 00:00 2024-06-04T04:43:05.335 INFO:teuthology.orchestra.run.smithi070.stdout:(144/159): python3-zc-lockfile-2.0-10.el9.noarc 343 kB/s | 20 kB 00:00 2024-06-04T04:43:05.628 INFO:teuthology.orchestra.run.smithi070.stdout:(145/159): libarrow-9.0.0-11.el9.x86_64.rpm 565 kB/s | 4.4 MB 00:07 2024-06-04T04:43:05.711 INFO:teuthology.orchestra.run.smithi070.stdout:(146/159): re2-20211101-3.el9.x86_64.rpm 511 kB/s | 192 kB 00:00 2024-06-04T04:43:06.328 INFO:teuthology.orchestra.run.smithi070.stdout:(147/159): python3-werkzeug-2.0.3-3.el9.1.noarc 363 kB/s | 427 kB 00:01 2024-06-04T04:43:07.403 INFO:teuthology.orchestra.run.smithi070.stdout:(148/159): librbd1-19.0.0-3882.gb7328fed.el9.x8 2.9 MB/s | 3.1 MB 00:01 2024-06-04T04:43:07.837 INFO:teuthology.orchestra.run.smithi070.stdout:(149/159): librados2-19.0.0-3882.gb7328fed.el9. 1.6 MB/s | 3.4 MB 00:02 2024-06-04T04:43:08.212 INFO:teuthology.orchestra.run.smithi070.stdout:(150/159): NetworkManager-1.47.91-1.el9.x86_64. 2.9 MB/s | 2.3 MB 00:00 2024-06-04T04:43:08.270 INFO:teuthology.orchestra.run.smithi070.stdout:(151/159): NetworkManager-team-1.47.91-1.el9.x8 712 kB/s | 41 kB 00:00 2024-06-04T04:43:08.387 INFO:teuthology.orchestra.run.smithi070.stdout:(152/159): NetworkManager-libnm-1.47.91-1.el9.x 3.3 MB/s | 1.8 MB 00:00 2024-06-04T04:43:08.421 INFO:teuthology.orchestra.run.smithi070.stdout:(153/159): NetworkManager-tui-1.47.91-1.el9.x86 1.6 MB/s | 248 kB 00:00 2024-06-04T04:43:08.446 INFO:teuthology.orchestra.run.smithi070.stdout:(154/159): elfutils-debuginfod-client-0.191-4.e 626 kB/s | 37 kB 00:00 2024-06-04T04:43:08.488 INFO:teuthology.orchestra.run.smithi070.stdout:(155/159): thrift-0.15.0-2.el9.x86_64.rpm 566 kB/s | 1.6 MB 00:02 2024-06-04T04:43:08.572 INFO:teuthology.orchestra.run.smithi070.stdout:(156/159): elfutils-libelf-0.191-4.el9.x86_64.r 1.3 MB/s | 207 kB 00:00 2024-06-04T04:43:08.622 INFO:teuthology.orchestra.run.smithi070.stdout:(157/159): elfutils-libs-0.191-4.el9.x86_64.rpm 1.4 MB/s | 260 kB 00:00 2024-06-04T04:43:08.814 INFO:teuthology.orchestra.run.smithi070.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 4.1 MB/s | 1.3 MB 00:00 2024-06-04T04:43:09.880 INFO:teuthology.orchestra.run.smithi070.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 1.8 MB/s | 2.4 MB 00:01 2024-06-04T04:43:09.886 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-06-04T04:43:09.886 INFO:teuthology.orchestra.run.smithi070.stdout:Total 2.5 MB/s | 227 MB 01:32 2024-06-04T04:43:11.056 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-06-04T04:43:11.212 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-06-04T04:43:11.213 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-06-04T04:43:13.345 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-06-04T04:43:13.346 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-06-04T04:43:19.534 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-06-04T04:43:19.643 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-06-04T04:43:19.692 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-06-04T04:43:19.782 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-06-04T04:43:19.818 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-06-04T04:43:19.843 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-06-04T04:43:19.899 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-06-04T04:43:20.238 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-06-04T04:43:20.352 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librados2-2:19.0.0-3882.gb7328fed.el9.x86_64 8/170 2024-06-04T04:43:20.473 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:19.0.0-3882.gb7328fed.el9.x86_64 8/170 2024-06-04T04:43:20.503 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs2-2:19.0.0-3882.gb7328fed.el9.x86_64 9/170 2024-06-04T04:43:20.594 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs2-2:19.0.0-3882.gb7328fed.el9.x86_64 9/170 2024-06-04T04:43:20.625 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rados-2:19.0.0-3882.gb7328fed.el9.x86_64 10/170 2024-06-04T04:43:20.666 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-06-04T04:43:20.854 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-06-04T04:43:20.915 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-06-04T04:43:20.942 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephsqlite-2:19.0.0-3882.gb7328fed.el9.x86_64 14/170 2024-06-04T04:43:21.044 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3882.gb7328fed.el9.x86_64 14/170 2024-06-04T04:43:21.068 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libradosstriper1-2:19.0.0-3882.gb7328fed.el9.x86 15/170 2024-06-04T04:43:21.131 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3882.gb7328fed.el9.x86 15/170 2024-06-04T04:43:21.152 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-04T04:43:21.174 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-04T04:43:21.362 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-04T04:43:21.557 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-04T04:43:21.692 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-04T04:43:21.760 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-04T04:43:21.864 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-04T04:43:21.944 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-04T04:43:22.144 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-04T04:43:22.262 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-06-04T04:43:22.294 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-06-04T04:43:22.333 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-06-04T04:43:22.397 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-06-04T04:43:22.433 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-06-04T04:43:22.476 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-06-04T04:43:22.509 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-06-04T04:43:22.536 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-06-04T04:43:22.604 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-06-04T04:43:22.666 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-06-04T04:43:22.697 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-06-04T04:43:22.741 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-06-04T04:43:22.816 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-06-04T04:43:22.852 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-common-2:19.0.0-3882.gb7328fed.el9. 33/170 2024-06-04T04:43:22.895 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-argparse-2:19.0.0-3882.gb7328fed.el 34/170 2024-06-04T04:43:22.924 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cephfs-2:19.0.0-3882.gb7328fed.el9.x86_6 35/170 2024-06-04T04:43:22.993 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-06-04T04:43:23.038 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-06-04T04:43:23.114 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-06-04T04:43:23.253 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-06-04T04:43:23.316 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-06-04T04:43:23.840 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-06-04T04:43:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-06-04T04:43:23.957 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-06-04T04:43:23.991 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-06-04T04:43:24.043 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-06-04T04:43:24.080 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librados-devel-2:19.0.0-3882.gb7328fed.el9.x86_6 46/170 2024-06-04T04:43:24.123 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-06-04T04:43:24.277 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-06-04T04:43:24.333 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-06-04T04:43:24.423 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-06-04T04:43:24.489 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-06-04T04:43:24.561 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-06-04T04:43:24.592 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-06-04T04:43:24.629 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-06-04T04:43:24.676 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-06-04T04:43:24.712 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-06-04T04:43:24.755 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-06-04T04:43:24.798 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-06-04T04:43:24.933 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-06-04T04:43:24.974 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-06-04T04:43:25.015 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-06-04T04:43:25.108 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-06-04T04:43:25.149 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-06-04T04:43:25.190 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-06-04T04:43:25.222 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-06-04T04:43:25.247 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-06-04T04:43:25.335 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-06-04T04:43:25.456 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-06-04T04:43:25.496 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-06-04T04:43:25.524 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-06-04T04:43:25.558 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-06-04T04:43:25.597 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-06-04T04:43:25.637 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-06-04T04:43:25.667 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-06-04T04:43:25.701 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-06-04T04:43:25.735 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-06-04T04:43:25.778 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-06-04T04:43:25.813 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-06-04T04:43:25.899 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-06-04T04:43:25.943 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-06-04T04:43:25.983 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-06-04T04:43:26.029 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-06-04T04:43:26.128 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-06-04T04:43:26.184 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-06-04T04:43:26.899 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-06-04T04:43:26.947 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-06-04T04:43:26.981 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-06-04T04:43:27.079 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-06-04T04:43:27.194 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-06-04T04:43:27.239 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-06-04T04:43:27.319 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-06-04T04:43:27.350 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-06-04T04:43:27.528 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-06-04T04:43:27.556 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-06-04T04:43:27.641 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-06-04T04:43:28.522 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-06-04T04:43:28.628 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-06-04T04:43:29.898 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-06-04T04:43:29.975 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-06-04T04:43:30.003 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-06-04T04:43:30.036 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-06-04T04:43:30.348 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-06-04T04:43:30.391 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librbd1-2:19.0.0-3882.gb7328fed.el9.x86_64 103/170 2024-06-04T04:43:30.463 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:19.0.0-3882.gb7328fed.el9.x86_64 103/170 2024-06-04T04:43:30.497 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rbd-2:19.0.0-3882.gb7328fed.el9.x86_64 104/170 2024-06-04T04:43:30.532 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-06-04T04:43:31.012 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-06-04T04:43:31.035 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librgw2-2:19.0.0-3882.gb7328fed.el9.x86_64 107/170 2024-06-04T04:43:31.106 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librgw2-2:19.0.0-3882.gb7328fed.el9.x86_64 107/170 2024-06-04T04:43:31.137 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rgw-2:19.0.0-3882.gb7328fed.el9.x86_64 108/170 2024-06-04T04:43:33.379 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 109/170 2024-06-04T04:43:33.416 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 109/170 2024-06-04T04:43:33.965 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 109/170 2024-06-04T04:43:34.002 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-base-2:19.0.0-3882.gb7328fed.el9.x86_64 110/170 2024-06-04T04:43:34.092 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-base-2:19.0.0-3882.gb7328fed.el9.x86_64 110/170 2024-06-04T04:43:34.093 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-04T04:43:34.093 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-04T04:43:34.093 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:43:34.120 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-selinux-2:19.0.0-3882.gb7328fed.el9.x86_64 111/170 2024-06-04T04:43:45.512 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3882.gb7328fed.el9.x86_64 111/170 2024-06-04T04:43:45.513 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /sys 2024-06-04T04:43:45.513 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /proc 2024-06-04T04:43:45.513 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /mnt 2024-06-04T04:43:45.513 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /var/tmp 2024-06-04T04:43:45.513 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /home 2024-06-04T04:43:45.513 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /root 2024-06-04T04:43:45.513 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /tmp 2024-06-04T04:43:45.513 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:43:45.690 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-volume-2:19.0.0-3882.gb7328fed.el9.noarch 112/170 2024-06-04T04:43:45.705 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-volume-2:19.0.0-3882.gb7328fed.el9.noarch 112/170 2024-06-04T04:43:45.705 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:43:45.705 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-04T04:43:45.705 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:43:47.159 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-osd-2:19.0.0-3882.gb7328fed.el9.x86_64 113/170 2024-06-04T04:43:47.185 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-osd-2:19.0.0-3882.gb7328fed.el9.x86_64 113/170 2024-06-04T04:43:47.185 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:43:47.185 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-04T04:43:47.185 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-04T04:43:47.185 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-04T04:43:47.186 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:43:47.418 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mds-2:19.0.0-3882.gb7328fed.el9.x86_64 114/170 2024-06-04T04:43:47.440 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mds-2:19.0.0-3882.gb7328fed.el9.x86_64 114/170 2024-06-04T04:43:47.440 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:43:47.440 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-04T04:43:47.440 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-04T04:43:47.441 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-04T04:43:47.441 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:43:47.877 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mon-2:19.0.0-3882.gb7328fed.el9.x86_64 115/170 2024-06-04T04:43:47.900 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mon-2:19.0.0-3882.gb7328fed.el9.x86_64 115/170 2024-06-04T04:43:47.900 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:43:47.900 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-04T04:43:47.900 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-04T04:43:47.900 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-04T04:43:47.900 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:43:48.007 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-06-04T04:43:48.055 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-06-04T04:43:48.110 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-06-04T04:43:48.295 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-06-04T04:43:48.353 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-06-04T04:43:48.422 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-06-04T04:43:48.510 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-06-04T04:43:48.650 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-06-04T04:43:48.694 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-06-04T04:43:48.741 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-06-04T04:43:48.770 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-06-04T04:43:49.230 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:19.0.0-3882.gb7328fed.el9.noarch 127/170 2024-06-04T04:43:49.257 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephadm-2:19.0.0-3882.gb7328fed.el9.noarch 127/170 2024-06-04T04:43:49.305 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3882.gb7328fed.e 128/170 2024-06-04T04:43:49.365 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3882.gb7328fed. 129/170 2024-06-04T04:43:49.395 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3882.gb7328fed.el9.noa 130/170 2024-06-04T04:43:50.431 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3882.gb7328fed.el9.noa 130/170 2024-06-04T04:43:50.464 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3882.gb7328fed.el9.n 131/170 2024-06-04T04:43:51.425 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3882.gb7328fed.el9.n 131/170 2024-06-04T04:43:51.456 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3882.gb73 132/170 2024-06-04T04:43:51.565 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3882.gb73 132/170 2024-06-04T04:43:51.733 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3882.gb7328fed.el 133/170 2024-06-04T04:43:51.775 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-2:19.0.0-3882.gb7328fed.el9.x86_64 134/170 2024-06-04T04:43:51.798 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3882.gb7328fed.el9.x86_64 134/170 2024-06-04T04:43:51.798 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:43:51.798 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-04T04:43:51.798 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-04T04:43:51.798 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-04T04:43:51.798 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:43:51.842 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-rook-2:19.0.0-3882.gb7328fed.el9.noarch 135/170 2024-06-04T04:43:51.853 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3882.gb7328fed.el9.noarch 135/170 2024-06-04T04:43:51.950 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-2:19.0.0-3882.gb7328fed.el9.x86_64 136/170 2024-06-04T04:43:52.002 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 137/170 2024-06-04T04:43:52.952 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-06-04T04:43:52.978 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-radosgw-2:19.0.0-3882.gb7328fed.el9.x86_64 139/170 2024-06-04T04:43:53.000 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3882.gb7328fed.el9.x86_64 139/170 2024-06-04T04:43:53.001 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:43:53.001 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-04T04:43:53.001 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-04T04:43:53.001 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-04T04:43:53.001 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:43:53.054 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3882.gb7328 140/170 2024-06-04T04:43:53.076 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3882.gb7328 140/170 2024-06-04T04:43:53.076 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:43:53.076 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-04T04:43:53.076 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:43:53.129 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 141/170 2024-06-04T04:43:53.151 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 141/170 2024-06-04T04:43:53.152 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:43:53.152 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-06-04T04:43:53.152 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-04T04:43:53.152 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-04T04:43:53.152 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:43:53.455 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 142/170 2024-06-04T04:43:53.478 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 142/170 2024-06-04T04:43:53.478 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-04T04:43:53.478 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-04T04:43:53.478 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-04T04:43:53.479 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-04T04:43:53.479 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:43:58.084 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-test-2:19.0.0-3882.gb7328fed.el9.x86_64 143/170 2024-06-04T04:43:58.124 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 144/170 2024-06-04T04:43:58.165 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-nbd-2:19.0.0-3882.gb7328fed.el9.x86_64 145/170 2024-06-04T04:43:58.197 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-06-04T04:43:58.225 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-06-04T04:43:58.419 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-shell-2:19.0.0-3882.gb7328fed.el9.x86_64 148/170 2024-06-04T04:43:58.463 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-06-04T04:43:58.495 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs-devel-2:19.0.0-3882.gb7328fed.el9.x86_ 150/170 2024-06-04T04:43:58.527 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-06-04T04:43:58.557 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : NetworkManager-team-1:1.47.91-1.el9.x86_64 152/170 2024-06-04T04:43:58.604 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : NetworkManager-tui-1:1.47.91-1.el9.x86_64 153/170 2024-06-04T04:43:59.239 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-04T04:43:59.277 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-04T04:43:59.296 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-04T04:44:00.120 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-top-2:19.0.0-3882.gb7328fed.el9.noarch 155/170 2024-06-04T04:44:00.215 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-06-04T04:44:00.257 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-06-04T04:44:00.361 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-06-04T04:44:00.394 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-06-04T04:44:00.419 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-04T04:44:00.444 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-06-04T04:44:00.471 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-06-04T04:44:00.471 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-04T04:44:00.555 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-04T04:44:00.605 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-06-04T04:44:00.605 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-04T04:44:00.689 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-04T04:44:00.714 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-06-04T04:44:00.720 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-04T04:44:00.720 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-04T04:44:00.900 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-04T04:44:00.900 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-04T04:44:00.985 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-04T04:44:01.017 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-06-04T04:44:01.083 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-04T04:44:06.827 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-04T04:44:06.828 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-2:19.0.0-3882.gb7328fed.el9.x86_64 1/170 2024-06-04T04:44:06.828 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-base-2:19.0.0-3882.gb7328fed.el9.x86_64 2/170 2024-06-04T04:44:06.828 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 3/170 2024-06-04T04:44:06.828 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 4/170 2024-06-04T04:44:06.828 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3882.gb7328 5/170 2024-06-04T04:44:06.828 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mds-2:19.0.0-3882.gb7328fed.el9.x86_64 6/170 2024-06-04T04:44:06.828 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-2:19.0.0-3882.gb7328fed.el9.x86_64 7/170 2024-06-04T04:44:06.828 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mon-2:19.0.0-3882.gb7328fed.el9.x86_64 8/170 2024-06-04T04:44:06.828 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-osd-2:19.0.0-3882.gb7328fed.el9.x86_64 9/170 2024-06-04T04:44:06.828 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-radosgw-2:19.0.0-3882.gb7328fed.el9.x86_64 10/170 2024-06-04T04:44:06.828 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-selinux-2:19.0.0-3882.gb7328fed.el9.x86_64 11/170 2024-06-04T04:44:06.828 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-test-2:19.0.0-3882.gb7328fed.el9.x86_64 12/170 2024-06-04T04:44:06.828 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 13/170 2024-06-04T04:44:06.829 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-shell-2:19.0.0-3882.gb7328fed.el9.x86_64 14/170 2024-06-04T04:44:06.829 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs-devel-2:19.0.0-3882.gb7328fed.el9.x86_ 15/170 2024-06-04T04:44:06.829 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs2-2:19.0.0-3882.gb7328fed.el9.x86_64 16/170 2024-06-04T04:44:06.829 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephsqlite-2:19.0.0-3882.gb7328fed.el9.x86_64 17/170 2024-06-04T04:44:06.829 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados-devel-2:19.0.0-3882.gb7328fed.el9.x86_6 18/170 2024-06-04T04:44:06.829 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libradosstriper1-2:19.0.0-3882.gb7328fed.el9.x86 19/170 2024-06-04T04:44:06.829 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librgw2-2:19.0.0-3882.gb7328fed.el9.x86_64 20/170 2024-06-04T04:44:06.829 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3882.gb7328fed.el 21/170 2024-06-04T04:44:06.829 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-common-2:19.0.0-3882.gb7328fed.el9. 22/170 2024-06-04T04:44:06.829 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cephfs-2:19.0.0-3882.gb7328fed.el9.x86_6 23/170 2024-06-04T04:44:06.829 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rados-2:19.0.0-3882.gb7328fed.el9.x86_64 24/170 2024-06-04T04:44:06.829 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rbd-2:19.0.0-3882.gb7328fed.el9.x86_64 25/170 2024-06-04T04:44:06.829 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rgw-2:19.0.0-3882.gb7328fed.el9.x86_64 26/170 2024-06-04T04:44:06.829 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 27/170 2024-06-04T04:44:06.829 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 28/170 2024-06-04T04:44:06.830 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-nbd-2:19.0.0-3882.gb7328fed.el9.x86_64 29/170 2024-06-04T04:44:06.830 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3882.gb7328fed. 30/170 2024-06-04T04:44:06.830 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3882.gb7328fed.el9.noa 31/170 2024-06-04T04:44:06.830 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3882.gb7328fed.el9.n 32/170 2024-06-04T04:44:06.830 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3882.gb73 33/170 2024-06-04T04:44:06.830 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3882.gb7328fed.el 34/170 2024-06-04T04:44:06.830 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3882.gb7328fed.el9.noarch 35/170 2024-06-04T04:44:06.830 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3882.gb7328fed.e 36/170 2024-06-04T04:44:06.830 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-volume-2:19.0.0-3882.gb7328fed.el9.noarch 37/170 2024-06-04T04:44:06.830 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephadm-2:19.0.0-3882.gb7328fed.el9.noarch 38/170 2024-06-04T04:44:06.830 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-top-2:19.0.0-3882.gb7328fed.el9.noarch 39/170 2024-06-04T04:44:06.830 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-06-04T04:44:06.830 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-06-04T04:44:06.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-06-04T04:44:06.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-06-04T04:44:06.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-06-04T04:44:06.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-06-04T04:44:06.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-06-04T04:44:06.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-06-04T04:44:06.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-06-04T04:44:06.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-06-04T04:44:06.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-06-04T04:44:06.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-06-04T04:44:06.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-06-04T04:44:06.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-06-04T04:44:06.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-06-04T04:44:06.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-06-04T04:44:06.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-06-04T04:44:06.832 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-06-04T04:44:06.832 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-06-04T04:44:06.832 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-06-04T04:44:06.832 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-06-04T04:44:06.832 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-06-04T04:44:06.832 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-06-04T04:44:06.832 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-06-04T04:44:06.832 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-06-04T04:44:06.832 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-06-04T04:44:06.832 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-06-04T04:44:06.832 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-06-04T04:44:06.832 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-06-04T04:44:06.832 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-06-04T04:44:06.833 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-06-04T04:44:06.833 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-06-04T04:44:06.833 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-06-04T04:44:06.833 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-06-04T04:44:06.833 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-06-04T04:44:06.833 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-06-04T04:44:06.833 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-06-04T04:44:06.833 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-06-04T04:44:06.833 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-06-04T04:44:06.833 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-06-04T04:44:06.833 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-06-04T04:44:06.833 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-06-04T04:44:06.833 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-06-04T04:44:06.834 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-06-04T04:44:06.834 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-06-04T04:44:06.834 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-06-04T04:44:06.834 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-06-04T04:44:06.834 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-06-04T04:44:06.834 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-06-04T04:44:06.834 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-06-04T04:44:06.834 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-06-04T04:44:06.834 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-06-04T04:44:06.834 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-06-04T04:44:06.834 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-06-04T04:44:06.835 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-06-04T04:44:06.835 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-06-04T04:44:06.835 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-06-04T04:44:06.835 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-06-04T04:44:06.835 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-06-04T04:44:06.835 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-06-04T04:44:06.835 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-06-04T04:44:06.835 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-06-04T04:44:06.835 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-06-04T04:44:06.835 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-06-04T04:44:06.835 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-06-04T04:44:06.836 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-06-04T04:44:06.836 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-06-04T04:44:06.836 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-06-04T04:44:06.836 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-06-04T04:44:06.836 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-06-04T04:44:06.836 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-06-04T04:44:06.836 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-06-04T04:44:06.836 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-06-04T04:44:06.836 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-06-04T04:44:06.836 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-06-04T04:44:06.836 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-06-04T04:44:06.836 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-06-04T04:44:06.836 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-06-04T04:44:06.837 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-06-04T04:44:06.837 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-06-04T04:44:06.837 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-06-04T04:44:06.837 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-06-04T04:44:06.837 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-06-04T04:44:06.837 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-06-04T04:44:06.837 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-06-04T04:44:06.837 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-06-04T04:44:06.837 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-06-04T04:44:06.837 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-06-04T04:44:06.837 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-06-04T04:44:06.837 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-06-04T04:44:06.837 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-06-04T04:44:06.837 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-06-04T04:44:06.837 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-06-04T04:44:06.838 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-06-04T04:44:06.838 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-06-04T04:44:06.838 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-06-04T04:44:06.838 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-06-04T04:44:06.838 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-06-04T04:44:06.838 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-06-04T04:44:06.838 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-06-04T04:44:06.838 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-06-04T04:44:06.838 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-06-04T04:44:06.838 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-06-04T04:44:06.838 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-06-04T04:44:06.838 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-06-04T04:44:06.839 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-06-04T04:44:06.839 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-06-04T04:44:06.839 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-06-04T04:44:06.839 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-06-04T04:44:06.839 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:19.0.0-3882.gb7328fed.el9.x86_64 149/170 2024-06-04T04:44:06.839 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-06-04T04:44:06.839 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:19.0.0-3882.gb7328fed.el9.x86_64 151/170 2024-06-04T04:44:06.839 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-06-04T04:44:06.839 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-1:1.47.91-1.el9.x86_64 153/170 2024-06-04T04:44:06.840 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-06-04T04:44:06.840 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 155/170 2024-06-04T04:44:06.840 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-06-04T04:44:06.840 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-team-1:1.47.91-1.el9.x86_64 157/170 2024-06-04T04:44:06.840 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-06-04T04:44:06.840 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-tui-1:1.47.91-1.el9.x86_64 159/170 2024-06-04T04:44:06.840 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-04T04:44:06.840 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-06-04T04:44:06.840 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-06-04T04:44:06.840 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-06-04T04:44:06.840 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-06-04T04:44:06.840 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-06-04T04:44:06.840 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-06-04T04:44:06.840 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-06-04T04:44:06.841 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-06-04T04:44:06.841 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-06-04T04:44:09.344 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-06-04T04:44:09.344 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:44:09.344 INFO:teuthology.orchestra.run.smithi070.stdout:Upgraded: 2024-06-04T04:44:09.344 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-1:1.47.91-1.el9.x86_64 2024-06-04T04:44:09.344 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 2024-06-04T04:44:09.344 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-team-1:1.47.91-1.el9.x86_64 2024-06-04T04:44:09.344 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-tui-1:1.47.91-1.el9.x86_64 2024-06-04T04:44:09.344 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-06-04T04:44:09.344 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-06-04T04:44:09.344 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-06-04T04:44:09.345 INFO:teuthology.orchestra.run.smithi070.stdout: librados2-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.345 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.345 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-06-04T04:44:09.345 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-06-04T04:44:09.345 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-06-04T04:44:09.345 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-04T04:44:09.345 INFO:teuthology.orchestra.run.smithi070.stdout: automake-1.16.2-8.el9.noarch 2024-06-04T04:44:09.345 INFO:teuthology.orchestra.run.smithi070.stdout: bison-3.7.4-5.el9.x86_64 2024-06-04T04:44:09.345 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-04T04:44:09.345 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.345 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.345 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.345 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.345 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:44:09.346 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.346 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.346 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.346 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:44:09.346 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:44:09.346 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:44:09.346 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:44:09.346 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:44:09.346 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.346 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.346 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:44:09.346 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.346 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.347 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.347 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:44:09.347 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:44:09.347 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.347 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-shell-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.347 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-top-2:19.0.0-3882.gb7328fed.el9.noarch 2024-06-04T04:44:09.347 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-06-04T04:44:09.347 INFO:teuthology.orchestra.run.smithi070.stdout: flex-2.6.4-9.el9.x86_64 2024-06-04T04:44:09.347 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-04T04:44:09.347 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-04T04:44:09.347 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-04T04:44:09.347 INFO:teuthology.orchestra.run.smithi070.stdout: fuse-2.9.9-15.el9.x86_64 2024-06-04T04:44:09.347 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-04T04:44:09.348 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-04T04:44:09.348 INFO:teuthology.orchestra.run.smithi070.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-06-04T04:44:09.348 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-04T04:44:09.348 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-04T04:44:09.348 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.348 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.348 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.348 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-04T04:44:09.348 INFO:teuthology.orchestra.run.smithi070.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-04T04:44:09.348 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-04T04:44:09.348 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-04T04:44:09.348 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-04T04:44:09.348 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.348 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.349 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-04T04:44:09.349 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.349 INFO:teuthology.orchestra.run.smithi070.stdout: libtool-2.4.6-45.el9.x86_64 2024-06-04T04:44:09.349 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-04T04:44:09.349 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-06-04T04:44:09.349 INFO:teuthology.orchestra.run.smithi070.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-06-04T04:44:09.349 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-04T04:44:09.349 INFO:teuthology.orchestra.run.smithi070.stdout: lua-5.4.4-4.el9.x86_64 2024-06-04T04:44:09.349 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-04T04:44:09.349 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-04T04:44:09.349 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-04T04:44:09.349 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-04T04:44:09.349 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-06-04T04:44:09.349 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-04T04:44:09.350 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-13.14-1.el9.x86_64 2024-06-04T04:44:09.350 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-06-04T04:44:09.350 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-06-04T04:44:09.350 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-server-13.14-1.el9.x86_64 2024-06-04T04:44:09.350 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-04T04:44:09.350 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-04T04:44:09.350 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-04T04:44:09.350 INFO:teuthology.orchestra.run.smithi070.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-06-04T04:44:09.350 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-04T04:44:09.350 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-04T04:44:09.350 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-04T04:44:09.350 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.351 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.351 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.351 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-04T04:44:09.351 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-04T04:44:09.351 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-04T04:44:09.351 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-04T04:44:09.351 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-06-04T04:44:09.351 INFO:teuthology.orchestra.run.smithi070.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-06-04T04:44:09.351 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-04T04:44:09.351 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-04T04:44:09.351 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-06-04T04:44:09.351 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-04T04:44:09.351 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-04T04:44:09.352 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-04T04:44:09.352 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-04T04:44:09.352 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-04T04:44:09.352 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-04T04:44:09.352 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-04T04:44:09.352 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-04T04:44:09.352 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-04T04:44:09.352 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-04T04:44:09.352 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-04T04:44:09.352 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-04T04:44:09.352 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-04T04:44:09.352 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-04T04:44:09.352 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-04T04:44:09.353 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-04T04:44:09.353 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-04T04:44:09.353 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-04T04:44:09.353 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-04T04:44:09.353 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-04T04:44:09.353 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-04T04:44:09.353 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply-3.11-14.el9.noarch 2024-06-04T04:44:09.353 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-04T04:44:09.353 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-04T04:44:09.353 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-04T04:44:09.353 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-04T04:44:09.353 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-04T04:44:09.353 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-04T04:44:09.353 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-04T04:44:09.354 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-06-04T04:44:09.354 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-04T04:44:09.354 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.354 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.354 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-04T04:44:09.354 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-04T04:44:09.354 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-04T04:44:09.354 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.354 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-04T04:44:09.354 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-04T04:44:09.354 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-04T04:44:09.354 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-04T04:44:09.355 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-04T04:44:09.355 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-04T04:44:09.355 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-04T04:44:09.355 INFO:teuthology.orchestra.run.smithi070.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-06-04T04:44:09.355 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-04T04:44:09.355 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-04T04:44:09.355 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-04T04:44:09.355 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-04T04:44:09.355 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-04T04:44:09.355 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-04T04:44:09.355 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-04T04:44:09.355 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.355 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.355 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd-2:19.0.0-3882.gb7328fed.el9.x86_64 2024-06-04T04:44:09.356 INFO:teuthology.orchestra.run.smithi070.stdout: re2-1:20211101-3.el9.x86_64 2024-06-04T04:44:09.356 INFO:teuthology.orchestra.run.smithi070.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-04T04:44:09.356 INFO:teuthology.orchestra.run.smithi070.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-04T04:44:09.356 INFO:teuthology.orchestra.run.smithi070.stdout: uuid-1.6.2-55.el9.x86_64 2024-06-04T04:44:09.356 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-06-04T04:44:09.356 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-04T04:44:09.356 INFO:teuthology.orchestra.run.smithi070.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-06-04T04:44:09.356 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:44:09.356 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-06-04T04:44:09.776 DEBUG:teuthology.parallel:result is None 2024-06-04T04:44:09.777 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b7328fed0e2e9ab66b09f8134f3cebfc71214504 2024-06-04T04:44:09.997 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-04T04:44:10.019 INFO:teuthology.orchestra.run.smithi070.stdout:19.0.0-3882.gb7328fed.el9 2024-06-04T04:44:10.020 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3882.gb7328fed.el9 2024-06-04T04:44:10.020 INFO:teuthology.task.install:The correct ceph version 19.0.0-3882.gb7328fed is installed. 2024-06-04T04:44:10.022 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b7328fed0e2e9ab66b09f8134f3cebfc71214504 2024-06-04T04:44:10.165 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-04T04:44:10.193 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-3882.gb7328fed.el9 2024-06-04T04:44:10.193 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3882.gb7328fed.el9 2024-06-04T04:44:10.194 INFO:teuthology.task.install:The correct ceph version 19.0.0-3882.gb7328fed is installed. 2024-06-04T04:44:10.195 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b7328fed0e2e9ab66b09f8134f3cebfc71214504 2024-06-04T04:44:10.410 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-04T04:44:10.438 INFO:teuthology.orchestra.run.smithi112.stdout:19.0.0-3882.gb7328fed.el9 2024-06-04T04:44:10.442 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3882.gb7328fed.el9 2024-06-04T04:44:10.442 INFO:teuthology.task.install:The correct ceph version 19.0.0-3882.gb7328fed is installed. 2024-06-04T04:44:10.444 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-04T04:44:10.444 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T04:44:10.445 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-04T04:44:10.474 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-04T04:44:10.474 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-04T04:44:10.506 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-06-04T04:44:10.506 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-04T04:44:10.538 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-04T04:44:10.538 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T04:44:10.538 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/daemon-helper 2024-06-04T04:44:10.566 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-04T04:44:10.632 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-04T04:44:10.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-06-04T04:44:10.662 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-04T04:44:10.731 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-06-04T04:44:10.731 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2024-06-04T04:44:10.760 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-04T04:44:10.829 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-04T04:44:10.829 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T04:44:10.829 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-04T04:44:10.856 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-04T04:44:10.922 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-04T04:44:10.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-04T04:44:10.947 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-04T04:44:11.012 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-06-04T04:44:11.012 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-04T04:44:11.037 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-04T04:44:11.102 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-04T04:44:11.102 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T04:44:11.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/stdin-killer 2024-06-04T04:44:11.129 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-04T04:44:11.195 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-04T04:44:11.195 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-06-04T04:44:11.219 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-04T04:44:11.284 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-06-04T04:44:11.284 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/stdin-killer 2024-06-04T04:44:11.309 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-04T04:44:11.373 INFO:teuthology.run_tasks:Running task cephadm... 2024-06-04T04:44:11.483 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'b7328fed0e2e9ab66b09f8134f3cebfc71214504', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-06-04T04:44:11.483 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 2024-06-04T04:44:11.484 INFO:tasks.cephadm:Cluster fsid is 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:44:11.484 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-06-04T04:44:11.484 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.70', 'mon.b': '172.21.15.98', 'mon.c': '172.21.15.112'} 2024-06-04T04:44:11.484 INFO:tasks.cephadm:First mon is mon.a on smithi070 2024-06-04T04:44:11.484 INFO:tasks.cephadm:First mgr is x 2024-06-04T04:44:11.484 INFO:tasks.cephadm:Normalizing hostnames... 2024-06-04T04:44:11.485 DEBUG:teuthology.orchestra.run.smithi070:> sudo hostname $(hostname -s) 2024-06-04T04:44:11.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname $(hostname -s) 2024-06-04T04:44:11.537 DEBUG:teuthology.orchestra.run.smithi112:> sudo hostname $(hostname -s) 2024-06-04T04:44:11.561 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-06-04T04:44:11.561 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b7328fed0e2e9ab66b09f8134f3cebfc71214504 2024-06-04T04:44:11.706 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'b7328fed0e2e9ab66b09f8134f3cebfc71214504', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/55203/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3882-gb7328fed', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3882.gb7328fed'}, 'url': 'https://3.chacra.ceph.com/r/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-22 17:15:03.629717', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-06-04T04:44:11.842 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref main, sha1 b7328fed0e2e9ab66b09f8134f3cebfc71214504 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=b7328fed0e2e9ab66b09f8134f3cebfc71214504 2024-06-04T04:44:11.843 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/x86_64/flavors/default/cephadm 2024-06-04T04:44:11.844 INFO:tasks.cephadm:Downloading cephadm from url: https://3.chacra.ceph.com/binaries/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/x86_64/flavors/default/cephadm 2024-06-04T04:44:11.844 DEBUG:teuthology.orchestra.run.smithi070:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-04T04:44:12.807 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-r--r--. 1 ubuntu ubuntu 782793 Jun 4 04:44 /home/ubuntu/cephtest/cephadm 2024-06-04T04:44:12.808 DEBUG:teuthology.orchestra.run.smithi098:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-04T04:44:13.755 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r--. 1 ubuntu ubuntu 782793 Jun 4 04:44 /home/ubuntu/cephtest/cephadm 2024-06-04T04:44:13.755 DEBUG:teuthology.orchestra.run.smithi112:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/b7328fed0e2e9ab66b09f8134f3cebfc71214504/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-04T04:44:14.524 INFO:teuthology.orchestra.run.smithi112.stdout:-rw-r--r--. 1 ubuntu ubuntu 782793 Jun 4 04:44 /home/ubuntu/cephtest/cephadm 2024-06-04T04:44:14.524 DEBUG:teuthology.orchestra.run.smithi070:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-04T04:44:14.544 DEBUG:teuthology.orchestra.run.smithi098:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-04T04:44:14.563 DEBUG:teuthology.orchestra.run.smithi112:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-04T04:44:14.597 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 on all hosts... 2024-06-04T04:44:14.598 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 pull 2024-06-04T04:44:14.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 pull 2024-06-04T04:44:14.606 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 pull 2024-06-04T04:44:15.368 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504... 2024-06-04T04:44:15.439 INFO:teuthology.orchestra.run.smithi112.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504... 2024-06-04T04:44:15.511 INFO:teuthology.orchestra.run.smithi070.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504... 2024-06-04T04:44:48.572 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:44:48.573 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-3882-gb7328fed (b7328fed0e2e9ab66b09f8134f3cebfc71214504) squid (dev)", 2024-06-04T04:44:48.573 INFO:teuthology.orchestra.run.smithi070.stdout: "image_id": "57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d", 2024-06-04T04:44:48.573 INFO:teuthology.orchestra.run.smithi070.stdout: "repo_digests": [ 2024-06-04T04:44:48.573 INFO:teuthology.orchestra.run.smithi070.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a" 2024-06-04T04:44:48.573 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-06-04T04:44:48.573 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:44:52.505 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-04T04:44:52.505 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-3882-gb7328fed (b7328fed0e2e9ab66b09f8134f3cebfc71214504) squid (dev)", 2024-06-04T04:44:52.505 INFO:teuthology.orchestra.run.smithi098.stdout: "image_id": "57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d", 2024-06-04T04:44:52.505 INFO:teuthology.orchestra.run.smithi098.stdout: "repo_digests": [ 2024-06-04T04:44:52.505 INFO:teuthology.orchestra.run.smithi098.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a" 2024-06-04T04:44:52.505 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-04T04:44:52.505 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-04T04:45:03.687 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-06-04T04:45:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_version": "ceph version 19.0.0-3882-gb7328fed (b7328fed0e2e9ab66b09f8134f3cebfc71214504) squid (dev)", 2024-06-04T04:45:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: "image_id": "57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d", 2024-06-04T04:45:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: "repo_digests": [ 2024-06-04T04:45:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a" 2024-06-04T04:45:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-06-04T04:45:03.688 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-06-04T04:45:03.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /etc/ceph 2024-06-04T04:45:03.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph 2024-06-04T04:45:03.779 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph 2024-06-04T04:45:03.807 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 777 /etc/ceph 2024-06-04T04:45:03.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /etc/ceph 2024-06-04T04:45:03.862 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /etc/ceph 2024-06-04T04:45:03.887 INFO:tasks.cephadm:Writing seed config... 2024-06-04T04:45:03.910 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-06-04T04:45:03.910 INFO:tasks.cephadm: override: [client] debug client = 20 2024-06-04T04:45:03.910 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-06-04T04:45:03.910 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-06-04T04:45:03.910 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-06-04T04:45:03.911 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-06-04T04:45:03.911 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-06-04T04:45:03.911 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-06-04T04:45:03.911 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-06-04T04:45:03.911 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-06-04T04:45:03.911 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-06-04T04:45:03.911 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-06-04T04:45:03.911 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-06-04T04:45:03.911 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-06-04T04:45:03.911 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-06-04T04:45:03.911 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-06-04T04:45:03.911 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-06-04T04:45:03.911 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-06-04T04:45:03.911 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-06-04T04:45:03.912 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-06-04T04:45:03.912 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-06-04T04:45:03.912 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-06-04T04:45:03.912 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-06-04T04:45:03.912 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-06-04T04:45:03.912 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-06-04T04:45:03.912 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-06-04T04:45:03.912 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-06-04T04:45:03.912 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-06-04T04:45:03.912 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-06-04T04:45:03.912 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-06-04T04:45:03.912 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-06-04T04:45:03.912 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-06-04T04:45:03.912 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-06-04T04:45:03.913 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-06-04T04:45:03.913 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-06-04T04:45:03.913 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-06-04T04:45:03.913 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-06-04T04:45:03.913 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-06-04T04:45:03.913 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-06-04T04:45:03.913 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-06-04T04:45:03.913 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-06-04T04:45:03.913 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-06-04T04:45:03.913 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-06-04T04:45:03.913 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-06-04T04:45:03.913 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-06-04T04:45:03.914 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T04:45:03.914 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-06-04T04:45:03.932 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 164dbaba-222d-11ef-bc9b-c7b262605968 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-06-04T04:45:03.932 DEBUG:teuthology.orchestra.run.smithi070:mon.a> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mon.a.service 2024-06-04T04:45:03.974 DEBUG:teuthology.orchestra.run.smithi070:mgr.x> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mgr.x.service 2024-06-04T04:45:04.017 INFO:tasks.cephadm:Bootstrapping... 2024-06-04T04:45:04.017 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 -v bootstrap --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.70 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-06-04T04:45:04.214 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-06-04T04:45:04.214 INFO:teuthology.orchestra.run.smithi070.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504', '-v', 'bootstrap', '--fsid', '164dbaba-222d-11ef-bc9b-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.70', '--skip-admin-label'] 2024-06-04T04:45:04.214 INFO:teuthology.orchestra.run.smithi070.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-06-04T04:45:04.214 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying podman|docker is present... 2024-06-04T04:45:04.241 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stdout 5.0.2 2024-06-04T04:45:04.241 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying lvm2 is present... 2024-06-04T04:45:04.241 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying time synchronization is in place... 2024-06-04T04:45:04.249 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-04T04:45:04.249 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-04T04:45:04.256 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-04T04:45:04.256 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-06-04T04:45:04.263 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout enabled 2024-06-04T04:45:04.270 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout active 2024-06-04T04:45:04.271 INFO:teuthology.orchestra.run.smithi070.stdout:Unit chronyd.service is enabled and running 2024-06-04T04:45:04.271 INFO:teuthology.orchestra.run.smithi070.stdout:Repeating the final host check... 2024-06-04T04:45:04.295 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stdout 5.0.2 2024-06-04T04:45:04.295 INFO:teuthology.orchestra.run.smithi070.stdout:podman (/bin/podman) version 5.0.2 is present 2024-06-04T04:45:04.295 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl is present 2024-06-04T04:45:04.295 INFO:teuthology.orchestra.run.smithi070.stdout:lvcreate is present 2024-06-04T04:45:04.302 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-04T04:45:04.303 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-04T04:45:04.309 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-04T04:45:04.309 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-06-04T04:45:04.317 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout enabled 2024-06-04T04:45:04.324 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout active 2024-06-04T04:45:04.324 INFO:teuthology.orchestra.run.smithi070.stdout:Unit chronyd.service is enabled and running 2024-06-04T04:45:04.325 INFO:teuthology.orchestra.run.smithi070.stdout:Host looks OK 2024-06-04T04:45:04.325 INFO:teuthology.orchestra.run.smithi070.stdout:Cluster fsid: 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:45:04.325 INFO:teuthology.orchestra.run.smithi070.stdout:Acquiring lock 140482707407056 on /run/cephadm/164dbaba-222d-11ef-bc9b-c7b262605968.lock 2024-06-04T04:45:04.325 INFO:teuthology.orchestra.run.smithi070.stdout:Lock 140482707407056 acquired on /run/cephadm/164dbaba-222d-11ef-bc9b-c7b262605968.lock 2024-06-04T04:45:04.325 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying IP 172.21.15.70 port 3300 ... 2024-06-04T04:45:04.325 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying IP 172.21.15.70 port 6789 ... 2024-06-04T04:45:04.326 INFO:teuthology.orchestra.run.smithi070.stdout:Base mon IP(s) is [172.21.15.70:3300, 172.21.15.70:6789], mon addrv is [v2:172.21.15.70:3300,v1:172.21.15.70:6789] 2024-06-04T04:45:04.330 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.70 metric 100 2024-06-04T04:45:04.330 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.70 metric 100 2024-06-04T04:45:04.334 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-06-04T04:45:04.335 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-06-04T04:45:04.335 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1786sec hoplimit 64 pref medium 2024-06-04T04:45:04.338 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-06-04T04:45:04.338 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-06-04T04:45:04.338 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-04T04:45:04.338 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-06-04T04:45:04.338 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-06-04T04:45:04.338 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-04T04:45:04.339 INFO:teuthology.orchestra.run.smithi070.stdout:Mon IP `172.21.15.70` is in CIDR network `172.21.0.0/20` 2024-06-04T04:45:04.339 INFO:teuthology.orchestra.run.smithi070.stdout:Mon IP `172.21.15.70` is in CIDR network `172.21.0.0/20` 2024-06-04T04:45:04.339 INFO:teuthology.orchestra.run.smithi070.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-06-04T04:45:04.341 INFO:teuthology.orchestra.run.smithi070.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-06-04T04:45:04.342 INFO:teuthology.orchestra.run.smithi070.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504... 2024-06-04T04:45:04.833 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stdout 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d 2024-06-04T04:45:04.833 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504... 2024-06-04T04:45:04.834 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Getting image source signatures 2024-06-04T04:45:04.834 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Copying blob sha256:af2bfe2195989993dd601658f16ff4b7ee99cf335930517d8b15388a4a91c26b 2024-06-04T04:45:04.834 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Copying blob sha256:6ac36e7c57f227110df5cd65cfdca1cc05e0b17df6748637a6aa04c5c80f89c6 2024-06-04T04:45:04.834 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Copying config sha256:57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d 2024-06-04T04:45:04.834 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Writing manifest to image destination 2024-06-04T04:45:05.763 INFO:teuthology.orchestra.run.smithi070.stdout:ceph: stdout ceph version 19.0.0-3882-gb7328fed (b7328fed0e2e9ab66b09f8134f3cebfc71214504) squid (dev) 2024-06-04T04:45:05.763 INFO:teuthology.orchestra.run.smithi070.stdout:Ceph version: ceph version 19.0.0-3882-gb7328fed (b7328fed0e2e9ab66b09f8134f3cebfc71214504) squid (dev) 2024-06-04T04:45:05.763 INFO:teuthology.orchestra.run.smithi070.stdout:Extracting ceph user uid/gid from container image... 2024-06-04T04:45:06.594 INFO:teuthology.orchestra.run.smithi070.stdout:stat: stdout 167 167 2024-06-04T04:45:06.594 INFO:teuthology.orchestra.run.smithi070.stdout:Creating initial keys... 2024-06-04T04:45:07.483 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph-authtool: stdout AQDSm15mCLjPNhAAa90G+dEkcMpLiSHOa1xIAg== 2024-06-04T04:45:08.398 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph-authtool: stdout AQDTm15mRS6yMBAA6Obgs2Em38Ija2FLCEjdxQ== 2024-06-04T04:45:09.328 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph-authtool: stdout AQDUm15maqgFKhAAtNu5P0RqYu1RImr2dt5PDg== 2024-06-04T04:45:09.329 INFO:teuthology.orchestra.run.smithi070.stdout:Creating initial monmap... 2024-06-04T04:45:12.036 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-04T04:45:12.036 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-06-04T04:45:12.036 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:45:12.037 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-04T04:45:12.037 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool for a [v2:172.21.15.70:3300,v1:172.21.15.70:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-04T04:45:12.037 INFO:teuthology.orchestra.run.smithi070.stdout:setting min_mon_release = quincy 2024-06-04T04:45:12.037 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: set fsid to 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:45:12.037 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-04T04:45:12.037 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:45:12.037 INFO:teuthology.orchestra.run.smithi070.stdout:Creating mon... 2024-06-04T04:45:20.955 INFO:teuthology.orchestra.run.smithi070.stdout:create mon.a on 2024-06-04T04:45:21.155 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-06-04T04:45:21.296 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-06-04T04:45:21.474 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-164dbaba-222d-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-164dbaba-222d-11ef-bc9b-c7b262605968.target. 2024-06-04T04:45:21.474 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-164dbaba-222d-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-164dbaba-222d-11ef-bc9b-c7b262605968.target. 2024-06-04T04:45:21.718 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mon.a 2024-06-04T04:45:21.718 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Failed to reset failed state of unit ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mon.a.service: Unit ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mon.a.service not loaded. 2024-06-04T04:45:21.878 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-164dbaba-222d-11ef-bc9b-c7b262605968.target.wants/ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mon.a.service → /etc/systemd/system/ceph-164dbaba-222d-11ef-bc9b-c7b262605968@.service. 2024-06-04T04:45:22.370 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-04T04:45:22.370 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout disabled 2024-06-04T04:45:22.376 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-04T04:45:22.376 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-06-04T04:45:22.376 INFO:teuthology.orchestra.run.smithi070.stdout:firewalld.service is not enabled 2024-06-04T04:45:22.376 INFO:teuthology.orchestra.run.smithi070.stdout:Not possible to enable service . firewalld.service is not available 2024-06-04T04:45:22.377 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mon to start... 2024-06-04T04:45:22.377 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mon... 2024-06-04T04:45:23.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:23 smithi070 ceph-mon[32259]: from='client.? 172.21.15.70:0/2383958151' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-06-04T04:45:23.941 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout cluster: 2024-06-04T04:45:23.947 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout id: 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:45:23.947 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-06-04T04:45:23.947 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-06-04T04:45:23.947 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout services: 2024-06-04T04:45:23.947 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.593022s) 2024-06-04T04:45:23.947 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-06-04T04:45:23.947 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-06-04T04:45:23.947 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-06-04T04:45:23.948 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout data: 2024-06-04T04:45:23.948 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-06-04T04:45:23.948 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-06-04T04:45:23.948 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-06-04T04:45:23.948 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout pgs: 2024-06-04T04:45:23.948 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-06-04T04:45:23.948 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.805+0000 7f51daa52700 1 Processor -- start 2024-06-04T04:45:23.948 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.805+0000 7f51daa52700 1 -- start start 2024-06-04T04:45:23.948 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.806+0000 7f51daa52700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51d4108ec0 0x7f51d41092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:23.948 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.806+0000 7f51daa52700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f51d4109820 con 0x7f51d4108ec0 2024-06-04T04:45:23.948 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.806+0000 7f51d3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51d4108ec0 0x7f51d41092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:23.948 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.806+0000 7f51d3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51d4108ec0 0x7f51d41092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55472/0 (socket says 172.21.15.70:55472) 2024-06-04T04:45:23.948 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.806+0000 7f51d3fff700 1 -- 172.21.15.70:0/2221484170 learned_addr learned my addr 172.21.15.70:0/2221484170 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:23.948 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.806+0000 7f51d3fff700 1 -- 172.21.15.70:0/2221484170 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d4109f80 con 0x7f51d4108ec0 2024-06-04T04:45:23.949 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.806+0000 7f51d3fff700 1 --2- 172.21.15.70:0/2221484170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51d4108ec0 0x7f51d41092e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f51c40098a0 tx=0x7f51c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6620735e206c242c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:23.949 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.807+0000 7f51d37fe700 1 -- 172.21.15.70:0/2221484170 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51c4004160 con 0x7f51d4108ec0 2024-06-04T04:45:23.949 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.807+0000 7f51d37fe700 1 -- 172.21.15.70:0/2221484170 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51c4004300 con 0x7f51d4108ec0 2024-06-04T04:45:23.949 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.807+0000 7f51d37fe700 1 -- 172.21.15.70:0/2221484170 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51c40036a0 con 0x7f51d4108ec0 2024-06-04T04:45:23.949 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.807+0000 7f51daa52700 1 -- 172.21.15.70:0/2221484170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51d4108ec0 msgr2=0x7f51d41092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:23.949 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.807+0000 7f51daa52700 1 --2- 172.21.15.70:0/2221484170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51d4108ec0 0x7f51d41092e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f51c40098a0 tx=0x7f51c4009bb0 comp rx=0 tx=0).stop 2024-06-04T04:45:23.949 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.807+0000 7f51daa52700 1 -- 172.21.15.70:0/2221484170 shutdown_connections 2024-06-04T04:45:23.949 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.807+0000 7f51daa52700 1 --2- 172.21.15.70:0/2221484170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51d4108ec0 0x7f51d41092e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:23.949 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.807+0000 7f51daa52700 1 -- 172.21.15.70:0/2221484170 >> 172.21.15.70:0/2221484170 conn(0x7f51d4104440 msgr2=0x7f51d41068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:23.949 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.807+0000 7f51daa52700 1 -- 172.21.15.70:0/2221484170 shutdown_connections 2024-06-04T04:45:23.949 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.807+0000 7f51daa52700 1 -- 172.21.15.70:0/2221484170 wait complete. 2024-06-04T04:45:23.949 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.808+0000 7f51daa52700 1 Processor -- start 2024-06-04T04:45:23.949 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.808+0000 7f51daa52700 1 -- start start 2024-06-04T04:45:23.949 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.808+0000 7f51daa52700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51d4108ec0 0x7f51d419c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:23.950 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.808+0000 7f51daa52700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f51d410a450 con 0x7f51d4108ec0 2024-06-04T04:45:23.950 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.808+0000 7f51d3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51d4108ec0 0x7f51d419c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:23.950 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.808+0000 7f51d3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51d4108ec0 0x7f51d419c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55476/0 (socket says 172.21.15.70:55476) 2024-06-04T04:45:23.950 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.808+0000 7f51d3fff700 1 -- 172.21.15.70:0/2383958151 learned_addr learned my addr 172.21.15.70:0/2383958151 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:23.950 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.808+0000 7f51d3fff700 1 -- 172.21.15.70:0/2383958151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d419cac0 con 0x7f51d4108ec0 2024-06-04T04:45:23.950 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.809+0000 7f51d3fff700 1 --2- 172.21.15.70:0/2383958151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51d4108ec0 0x7f51d419c580 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f51c4000c00 tx=0x7f51c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:23.950 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.809+0000 7f51d1ffb700 1 -- 172.21.15.70:0/2383958151 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51c4003db0 con 0x7f51d4108ec0 2024-06-04T04:45:23.950 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.809+0000 7f51d1ffb700 1 -- 172.21.15.70:0/2383958151 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51c4011630 con 0x7f51d4108ec0 2024-06-04T04:45:23.950 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.809+0000 7f51d1ffb700 1 -- 172.21.15.70:0/2383958151 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51c4011950 con 0x7f51d4108ec0 2024-06-04T04:45:23.950 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.809+0000 7f51daa52700 1 -- 172.21.15.70:0/2383958151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51d419cd50 con 0x7f51d4108ec0 2024-06-04T04:45:23.950 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.809+0000 7f51daa52700 1 -- 172.21.15.70:0/2383958151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51d419f9f0 con 0x7f51d4108ec0 2024-06-04T04:45:23.950 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.809+0000 7f51d1ffb700 1 -- 172.21.15.70:0/2383958151 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f51c4011af0 con 0x7f51d4108ec0 2024-06-04T04:45:23.950 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.809+0000 7f51d1ffb700 1 -- 172.21.15.70:0/2383958151 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f51c401c450 con 0x7f51d4108ec0 2024-06-04T04:45:23.951 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.810+0000 7f51daa52700 1 -- 172.21.15.70:0/2383958151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51d4064ba0 con 0x7f51d4108ec0 2024-06-04T04:45:23.951 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.812+0000 7f51d1ffb700 1 -- 172.21.15.70:0/2383958151 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f51c4004260 con 0x7f51d4108ec0 2024-06-04T04:45:23.951 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.888+0000 7f51daa52700 1 -- 172.21.15.70:0/2383958151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f51d402e600 con 0x7f51d4108ec0 2024-06-04T04:45:23.951 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.889+0000 7f51d1ffb700 1 -- 172.21.15.70:0/2383958151 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f51c401b450 con 0x7f51d4108ec0 2024-06-04T04:45:23.951 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.889+0000 7f51daa52700 1 -- 172.21.15.70:0/2383958151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51d4108ec0 msgr2=0x7f51d419c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:23.951 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.889+0000 7f51daa52700 1 --2- 172.21.15.70:0/2383958151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51d4108ec0 0x7f51d419c580 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f51c4000c00 tx=0x7f51c4004750 comp rx=0 tx=0).stop 2024-06-04T04:45:23.951 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.889+0000 7f51daa52700 1 -- 172.21.15.70:0/2383958151 shutdown_connections 2024-06-04T04:45:23.951 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.889+0000 7f51daa52700 1 --2- 172.21.15.70:0/2383958151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51d4108ec0 0x7f51d419c580 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:23.951 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.889+0000 7f51daa52700 1 -- 172.21.15.70:0/2383958151 >> 172.21.15.70:0/2383958151 conn(0x7f51d4104440 msgr2=0x7f51d4193070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:23.951 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.890+0000 7f51daa52700 1 -- 172.21.15.70:0/2383958151 shutdown_connections 2024-06-04T04:45:23.951 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:22.890+0000 7f51daa52700 1 -- 172.21.15.70:0/2383958151 wait complete. 2024-06-04T04:45:23.951 INFO:teuthology.orchestra.run.smithi070.stdout:mon is available 2024-06-04T04:45:23.951 INFO:teuthology.orchestra.run.smithi070.stdout:Assimilating anything we can from ceph.conf... 2024-06-04T04:45:24.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:24 smithi070 ceph-mon[32259]: from='client.? 172.21.15.70:0/1651045508' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-04T04:45:24.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:24 smithi070 ceph-mon[32259]: from='client.? 172.21.15.70:0/1651045508' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-04T04:45:24.994 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-06-04T04:45:24.994 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [global] 2024-06-04T04:45:24.994 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout fsid = 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:45:24.994 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-06-04T04:45:24.994 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.70:3300,v1:172.21.15.70:6789] 2024-06-04T04:45:24.995 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-04T04:45:24.995 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-04T04:45:24.995 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-04T04:45:24.995 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-04T04:45:24.995 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-06-04T04:45:24.995 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-04T04:45:24.995 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-04T04:45:24.995 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-06-04T04:45:24.995 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [osd] 2024-06-04T04:45:24.995 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-04T04:45:24.995 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-04T04:45:24.995 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-04T04:45:24.995 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-04T04:45:24.996 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-04T04:45:24.996 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.389+0000 7f2cac9c4700 1 Processor -- start 2024-06-04T04:45:24.996 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.390+0000 7f2cac9c4700 1 -- start start 2024-06-04T04:45:24.996 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.390+0000 7f2cac9c4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca8109980 0x7f2ca810bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:24.996 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.390+0000 7f2cac9c4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca807ce10 con 0x7f2ca8109980 2024-06-04T04:45:24.996 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.390+0000 7f2ca659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca8109980 0x7f2ca810bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:24.996 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.390+0000 7f2ca659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca8109980 0x7f2ca810bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55478/0 (socket says 172.21.15.70:55478) 2024-06-04T04:45:24.996 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.390+0000 7f2ca659c700 1 -- 172.21.15.70:0/1176728904 learned_addr learned my addr 172.21.15.70:0/1176728904 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:24.996 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.390+0000 7f2ca659c700 1 -- 172.21.15.70:0/1176728904 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca810c2b0 con 0x7f2ca8109980 2024-06-04T04:45:24.996 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.390+0000 7f2ca659c700 1 --2- 172.21.15.70:0/1176728904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca8109980 0x7f2ca810bd70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c0098a0 tx=0x7f2c9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ab03130801ab89b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:24.996 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.391+0000 7f2ca559a700 1 -- 172.21.15.70:0/1176728904 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c9c004160 con 0x7f2ca8109980 2024-06-04T04:45:24.996 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.391+0000 7f2ca559a700 1 -- 172.21.15.70:0/1176728904 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c9c004300 con 0x7f2ca8109980 2024-06-04T04:45:24.997 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.391+0000 7f2ca559a700 1 -- 172.21.15.70:0/1176728904 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c9c0036a0 con 0x7f2ca8109980 2024-06-04T04:45:24.997 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.391+0000 7f2cac9c4700 1 -- 172.21.15.70:0/1176728904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca8109980 msgr2=0x7f2ca810bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:24.997 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.391+0000 7f2cac9c4700 1 --2- 172.21.15.70:0/1176728904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca8109980 0x7f2ca810bd70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c0098a0 tx=0x7f2c9c009bb0 comp rx=0 tx=0).stop 2024-06-04T04:45:24.997 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.391+0000 7f2cac9c4700 1 -- 172.21.15.70:0/1176728904 shutdown_connections 2024-06-04T04:45:24.997 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.391+0000 7f2cac9c4700 1 --2- 172.21.15.70:0/1176728904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca8109980 0x7f2ca810bd70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:24.997 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.391+0000 7f2cac9c4700 1 -- 172.21.15.70:0/1176728904 >> 172.21.15.70:0/1176728904 conn(0x7f2ca81032f0 msgr2=0x7f2ca8105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:24.997 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.391+0000 7f2cac9c4700 1 -- 172.21.15.70:0/1176728904 shutdown_connections 2024-06-04T04:45:24.997 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.391+0000 7f2cac9c4700 1 -- 172.21.15.70:0/1176728904 wait complete. 2024-06-04T04:45:24.997 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.392+0000 7f2cac9c4700 1 Processor -- start 2024-06-04T04:45:24.997 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.392+0000 7f2cac9c4700 1 -- start start 2024-06-04T04:45:24.997 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.392+0000 7f2cac9c4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca8109980 0x7f2ca8109200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:24.997 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.392+0000 7f2cac9c4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca810c720 con 0x7f2ca8109980 2024-06-04T04:45:24.997 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.392+0000 7f2ca659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca8109980 0x7f2ca8109200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:24.997 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.392+0000 7f2ca659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca8109980 0x7f2ca8109200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55490/0 (socket says 172.21.15.70:55490) 2024-06-04T04:45:24.998 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.392+0000 7f2ca659c700 1 -- 172.21.15.70:0/1651045508 learned_addr learned my addr 172.21.15.70:0/1651045508 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:24.998 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.393+0000 7f2ca659c700 1 -- 172.21.15.70:0/1651045508 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca8109740 con 0x7f2ca8109980 2024-06-04T04:45:24.998 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.393+0000 7f2ca659c700 1 --2- 172.21.15.70:0/1651045508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca8109980 0x7f2ca8109200 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c000c00 tx=0x7f2c9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:24.998 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.393+0000 7f2c977fe700 1 -- 172.21.15.70:0/1651045508 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c9c003db0 con 0x7f2ca8109980 2024-06-04T04:45:24.998 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.393+0000 7f2c977fe700 1 -- 172.21.15.70:0/1651045508 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c9c011630 con 0x7f2ca8109980 2024-06-04T04:45:24.998 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.393+0000 7f2cac9c4700 1 -- 172.21.15.70:0/1651045508 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ca8107930 con 0x7f2ca8109980 2024-06-04T04:45:24.998 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.393+0000 7f2c977fe700 1 -- 172.21.15.70:0/1651045508 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c9c011950 con 0x7f2ca8109980 2024-06-04T04:45:24.998 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.393+0000 7f2cac9c4700 1 -- 172.21.15.70:0/1651045508 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ca8107e70 con 0x7f2ca8109980 2024-06-04T04:45:24.998 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.393+0000 7f2c977fe700 1 -- 172.21.15.70:0/1651045508 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2c9c011af0 con 0x7f2ca8109980 2024-06-04T04:45:24.998 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.393+0000 7f2c977fe700 1 -- 172.21.15.70:0/1651045508 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2c9c01c450 con 0x7f2ca8109980 2024-06-04T04:45:24.998 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.394+0000 7f2cac9c4700 1 -- 172.21.15.70:0/1651045508 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ca8064ba0 con 0x7f2ca8109980 2024-06-04T04:45:24.998 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.396+0000 7f2c977fe700 1 -- 172.21.15.70:0/1651045508 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f2c9c032020 con 0x7f2ca8109980 2024-06-04T04:45:24.998 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.463+0000 7f2cac9c4700 1 -- 172.21.15.70:0/1651045508 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f2ca8195dd0 con 0x7f2ca8109980 2024-06-04T04:45:24.998 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.466+0000 7f2c977fe700 1 -- 172.21.15.70:0/1651045508 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7f2c9c004260 con 0x7f2ca8109980 2024-06-04T04:45:24.999 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.466+0000 7f2c977fe700 1 -- 172.21.15.70:0/1651045508 <== mon.0 v2:172.21.15.70:3300/0 8 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f2c9c01b450 con 0x7f2ca8109980 2024-06-04T04:45:24.999 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.467+0000 7f2cac9c4700 1 -- 172.21.15.70:0/1651045508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca8109980 msgr2=0x7f2ca8109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:24.999 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.467+0000 7f2cac9c4700 1 --2- 172.21.15.70:0/1651045508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca8109980 0x7f2ca8109200 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c000c00 tx=0x7f2c9c004750 comp rx=0 tx=0).stop 2024-06-04T04:45:24.999 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.467+0000 7f2cac9c4700 1 -- 172.21.15.70:0/1651045508 shutdown_connections 2024-06-04T04:45:24.999 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.467+0000 7f2cac9c4700 1 --2- 172.21.15.70:0/1651045508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca8109980 0x7f2ca8109200 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:24.999 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.467+0000 7f2cac9c4700 1 -- 172.21.15.70:0/1651045508 >> 172.21.15.70:0/1651045508 conn(0x7f2ca81032f0 msgr2=0x7f2ca8105660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:24.999 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.467+0000 7f2cac9c4700 1 -- 172.21.15.70:0/1651045508 shutdown_connections 2024-06-04T04:45:24.999 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:24.467+0000 7f2cac9c4700 1 -- 172.21.15.70:0/1651045508 wait complete. 2024-06-04T04:45:24.999 INFO:teuthology.orchestra.run.smithi070.stdout:Generating new minimal ceph.conf... 2024-06-04T04:45:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:25 smithi070 ceph-mon[32259]: from='client.? 172.21.15.70:0/646770104' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:45:26.017 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.368+0000 7fea299c4700 1 Processor -- start 2024-06-04T04:45:26.017 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.368+0000 7fea299c4700 1 -- start start 2024-06-04T04:45:26.017 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.369+0000 7fea299c4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea24107d50 0x7fea24108170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:26.017 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.369+0000 7fea299c4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fea24108740 con 0x7fea24107d50 2024-06-04T04:45:26.017 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.369+0000 7fea22ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea24107d50 0x7fea24108170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:26.018 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.369+0000 7fea22ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea24107d50 0x7fea24108170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55504/0 (socket says 172.21.15.70:55504) 2024-06-04T04:45:26.018 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.369+0000 7fea22ffd700 1 -- 172.21.15.70:0/234921564 learned_addr learned my addr 172.21.15.70:0/234921564 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:26.018 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.369+0000 7fea22ffd700 1 -- 172.21.15.70:0/234921564 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea24108f50 con 0x7fea24107d50 2024-06-04T04:45:26.018 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.369+0000 7fea22ffd700 1 --2- 172.21.15.70:0/234921564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea24107d50 0x7fea24108170 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fea180098a0 tx=0x7fea18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c76d00006c859eb2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:26.018 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.369+0000 7fea21ffb700 1 -- 172.21.15.70:0/234921564 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea18004160 con 0x7fea24107d50 2024-06-04T04:45:26.018 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.369+0000 7fea21ffb700 1 -- 172.21.15.70:0/234921564 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fea18004300 con 0x7fea24107d50 2024-06-04T04:45:26.018 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.369+0000 7fea21ffb700 1 -- 172.21.15.70:0/234921564 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea180119e0 con 0x7fea24107d50 2024-06-04T04:45:26.018 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.370+0000 7fea299c4700 1 -- 172.21.15.70:0/234921564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea24107d50 msgr2=0x7fea24108170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:26.018 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.370+0000 7fea299c4700 1 --2- 172.21.15.70:0/234921564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea24107d50 0x7fea24108170 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fea180098a0 tx=0x7fea18009bb0 comp rx=0 tx=0).stop 2024-06-04T04:45:26.018 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.370+0000 7fea299c4700 1 -- 172.21.15.70:0/234921564 shutdown_connections 2024-06-04T04:45:26.018 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.370+0000 7fea299c4700 1 --2- 172.21.15.70:0/234921564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea24107d50 0x7fea24108170 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:26.018 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.370+0000 7fea299c4700 1 -- 172.21.15.70:0/234921564 >> 172.21.15.70:0/234921564 conn(0x7fea24103330 msgr2=0x7fea24105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:26.018 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.370+0000 7fea299c4700 1 -- 172.21.15.70:0/234921564 shutdown_connections 2024-06-04T04:45:26.019 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.370+0000 7fea299c4700 1 -- 172.21.15.70:0/234921564 wait complete. 2024-06-04T04:45:26.019 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.371+0000 7fea299c4700 1 Processor -- start 2024-06-04T04:45:26.019 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.371+0000 7fea299c4700 1 -- start start 2024-06-04T04:45:26.019 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.371+0000 7fea299c4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea24107d50 0x7fea2419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:26.019 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.371+0000 7fea299c4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fea2410c180 con 0x7fea24107d50 2024-06-04T04:45:26.019 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.371+0000 7fea22ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea24107d50 0x7fea2419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:26.019 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.371+0000 7fea22ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea24107d50 0x7fea2419cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55514/0 (socket says 172.21.15.70:55514) 2024-06-04T04:45:26.019 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.371+0000 7fea22ffd700 1 -- 172.21.15.70:0/646770104 learned_addr learned my addr 172.21.15.70:0/646770104 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:26.019 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.371+0000 7fea22ffd700 1 -- 172.21.15.70:0/646770104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2419d2f0 con 0x7fea24107d50 2024-06-04T04:45:26.019 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.371+0000 7fea22ffd700 1 --2- 172.21.15.70:0/646770104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea24107d50 0x7fea2419cdb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fea180119b0 tx=0x7fea18011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:26.019 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.371+0000 7fea289c2700 1 -- 172.21.15.70:0/646770104 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea18003b70 con 0x7fea24107d50 2024-06-04T04:45:26.019 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.372+0000 7fea289c2700 1 -- 172.21.15.70:0/646770104 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fea1801a8a0 con 0x7fea24107d50 2024-06-04T04:45:26.019 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.372+0000 7fea299c4700 1 -- 172.21.15.70:0/646770104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea2419d580 con 0x7fea24107d50 2024-06-04T04:45:26.020 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.372+0000 7fea289c2700 1 -- 172.21.15.70:0/646770104 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea18024830 con 0x7fea24107d50 2024-06-04T04:45:26.020 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.372+0000 7fea299c4700 1 -- 172.21.15.70:0/646770104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea241a0270 con 0x7fea24107d50 2024-06-04T04:45:26.020 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.372+0000 7fea289c2700 1 -- 172.21.15.70:0/646770104 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fea18022020 con 0x7fea24107d50 2024-06-04T04:45:26.020 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.372+0000 7fea289c2700 1 -- 172.21.15.70:0/646770104 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fea180193c0 con 0x7fea24107d50 2024-06-04T04:45:26.020 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.373+0000 7fea299c4700 1 -- 172.21.15.70:0/646770104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea24064ba0 con 0x7fea24107d50 2024-06-04T04:45:26.020 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.375+0000 7fea289c2700 1 -- 172.21.15.70:0/646770104 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fea1802dde0 con 0x7fea24107d50 2024-06-04T04:45:26.020 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.441+0000 7fea299c4700 1 -- 172.21.15.70:0/646770104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fea2402e6e0 con 0x7fea24107d50 2024-06-04T04:45:26.020 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.442+0000 7fea289c2700 1 -- 172.21.15.70:0/646770104 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fea1801aa50 con 0x7fea24107d50 2024-06-04T04:45:26.020 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.443+0000 7fea299c4700 1 -- 172.21.15.70:0/646770104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea24107d50 msgr2=0x7fea2419cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:26.020 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.443+0000 7fea299c4700 1 --2- 172.21.15.70:0/646770104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea24107d50 0x7fea2419cdb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fea180119b0 tx=0x7fea18011a90 comp rx=0 tx=0).stop 2024-06-04T04:45:26.020 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.443+0000 7fea299c4700 1 -- 172.21.15.70:0/646770104 shutdown_connections 2024-06-04T04:45:26.020 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.443+0000 7fea299c4700 1 --2- 172.21.15.70:0/646770104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea24107d50 0x7fea2419cdb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:26.020 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.443+0000 7fea299c4700 1 -- 172.21.15.70:0/646770104 >> 172.21.15.70:0/646770104 conn(0x7fea24103330 msgr2=0x7fea241041f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:26.020 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.443+0000 7fea299c4700 1 -- 172.21.15.70:0/646770104 shutdown_connections 2024-06-04T04:45:26.021 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:25.443+0000 7fea299c4700 1 -- 172.21.15.70:0/646770104 wait complete. 2024-06-04T04:45:26.021 INFO:teuthology.orchestra.run.smithi070.stdout:Restarting the monitor... 2024-06-04T04:45:26.271 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:26 smithi070 systemd[1]: Stopping Ceph mon.a for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T04:45:26.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:26 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-a[32255]: 2024-06-04T04:45:26.269+0000 7f6bdd18f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T04:45:26.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:26 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-a[32255]: 2024-06-04T04:45:26.269+0000 7f6bdd18f700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-06-04T04:45:26.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:26 smithi070 podman[32503]: 2024-06-04 04:45:26.422700124 +0000 UTC m=+0.252275294 container died db746f7081c2bc83949e75d5a94b09ce99985efbe93c74fc2d6de68325ee33df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-a, RELEASE=main-b7328fe, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-06-04T04:45:27.073 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:26 smithi070 podman[32503]: 2024-06-04 04:45:26.744173964 +0000 UTC m=+0.573749152 container cleanup db746f7081c2bc83949e75d5a94b09ce99985efbe93c74fc2d6de68325ee33df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-b7328fe) 2024-06-04T04:45:27.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:26 smithi070 bash[32503]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-a 2024-06-04T04:45:27.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:26 smithi070 podman[32514]: 2024-06-04 04:45:26.886762525 +0000 UTC m=+0.466473380 container remove db746f7081c2bc83949e75d5a94b09ce99985efbe93c74fc2d6de68325ee33df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-a, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-06-04T04:45:27.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 systemd[1]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mon.a.service: Deactivated successfully. 2024-06-04T04:45:27.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 systemd[1]: Stopped Ceph mon.a for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:45:27.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 systemd[1]: Starting Ceph mon.a for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T04:45:27.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 podman[32612]: 2024-06-04 04:45:27.20671614 +0000 UTC m=+0.017309399 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 2024-06-04T04:45:27.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 podman[32612]: 2024-06-04 04:45:27.304121092 +0000 UTC m=+0.114714314 container create e29249260accb47368f128a967c2d241c9c8843c9a20926d82ae02f15fb6bbe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-a, io.buildah.version=1.33.5, RELEASE=main-b7328fe, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-04T04:45:27.604 INFO:teuthology.orchestra.run.smithi070.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-06-04T04:45:27.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 podman[32612]: 2024-06-04 04:45:27.523530625 +0000 UTC m=+0.334123863 container init e29249260accb47368f128a967c2d241c9c8843c9a20926d82ae02f15fb6bbe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-b7328fe, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-06-04T04:45:27.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 podman[32612]: 2024-06-04 04:45:27.526663454 +0000 UTC m=+0.337256686 container start e29249260accb47368f128a967c2d241c9c8843c9a20926d82ae02f15fb6bbe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-a, org.label-schema.vendor=CentOS, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-06-04T04:45:27.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: set uid:gid to 167:167 (ceph:ceph) 2024-06-04T04:45:27.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: ceph version 19.0.0-3882-gb7328fed (b7328fed0e2e9ab66b09f8134f3cebfc71214504) squid (dev), process ceph-mon, pid 2 2024-06-04T04:45:27.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: pidfile_write: ignore empty --pid-file 2024-06-04T04:45:27.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: load: jerasure load: lrc 2024-06-04T04:45:27.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: RocksDB version: 7.9.2 2024-06-04T04:45:27.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Git sha 0 2024-06-04T04:45:27.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Compile date 2024-05-22 16:22:08 2024-06-04T04:45:27.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: DB SUMMARY 2024-06-04T04:45:27.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: DB Session ID: RNFNGMZ8YP31HETLWZGR 2024-06-04T04:45:27.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: CURRENT file: CURRENT 2024-06-04T04:45:27.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: IDENTITY file: IDENTITY 2024-06-04T04:45:27.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-06-04T04:45:27.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-06-04T04:45:27.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 103966 ; 2024-06-04T04:45:27.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.error_if_exists: 0 2024-06-04T04:45:27.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.create_if_missing: 0 2024-06-04T04:45:27.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.paranoid_checks: 1 2024-06-04T04:45:27.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-04T04:45:27.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-04T04:45:27.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-04T04:45:27.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.env: 0x55ef17440cc0 2024-06-04T04:45:27.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.fs: PosixFileSystem 2024-06-04T04:45:27.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.info_log: 0x55ef17eefda0 2024-06-04T04:45:27.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_file_opening_threads: 16 2024-06-04T04:45:27.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.statistics: (nil) 2024-06-04T04:45:27.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.use_fsync: 0 2024-06-04T04:45:27.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_log_file_size: 0 2024-06-04T04:45:27.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-04T04:45:27.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-04T04:45:27.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.keep_log_file_num: 1000 2024-06-04T04:45:27.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.recycle_log_file_num: 0 2024-06-04T04:45:27.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.allow_fallocate: 1 2024-06-04T04:45:27.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.allow_mmap_reads: 0 2024-06-04T04:45:27.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.allow_mmap_writes: 0 2024-06-04T04:45:27.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.use_direct_reads: 0 2024-06-04T04:45:27.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-04T04:45:27.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.create_missing_column_families: 0 2024-06-04T04:45:27.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.db_log_dir: 2024-06-04T04:45:27.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.wal_dir: 2024-06-04T04:45:27.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-04T04:45:27.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-04T04:45:27.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-04T04:45:27.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-04T04:45:27.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-04T04:45:27.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-04T04:45:27.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.advise_random_on_open: 1 2024-06-04T04:45:27.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.db_write_buffer_size: 0 2024-06-04T04:45:27.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.write_buffer_manager: 0x55ef17ec10e0 2024-06-04T04:45:27.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-04T04:45:27.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-04T04:45:27.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-04T04:45:27.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.rate_limiter: (nil) 2024-06-04T04:45:27.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-04T04:45:27.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.wal_recovery_mode: 2 2024-06-04T04:45:27.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.enable_thread_tracking: 0 2024-06-04T04:45:27.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.enable_pipelined_write: 0 2024-06-04T04:45:27.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.unordered_write: 0 2024-06-04T04:45:27.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-04T04:45:27.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-04T04:45:27.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-04T04:45:27.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-04T04:45:27.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.row_cache: None 2024-06-04T04:45:27.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.wal_filter: None 2024-06-04T04:45:27.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-04T04:45:27.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.allow_ingest_behind: 0 2024-06-04T04:45:27.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.two_write_queues: 0 2024-06-04T04:45:27.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.manual_wal_flush: 0 2024-06-04T04:45:27.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.wal_compression: 0 2024-06-04T04:45:27.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.atomic_flush: 0 2024-06-04T04:45:27.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-04T04:45:27.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-04T04:45:27.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-04T04:45:27.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.log_readahead_size: 0 2024-06-04T04:45:27.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-04T04:45:27.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.best_efforts_recovery: 0 2024-06-04T04:45:27.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-04T04:45:27.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-04T04:45:27.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.allow_data_in_errors: 0 2024-06-04T04:45:27.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.db_host_id: __hostname__ 2024-06-04T04:45:27.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-04T04:45:27.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_background_jobs: 2 2024-06-04T04:45:27.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_background_compactions: -1 2024-06-04T04:45:27.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_subcompactions: 1 2024-06-04T04:45:27.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-04T04:45:27.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-04T04:45:27.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-04T04:45:27.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_total_wal_size: 0 2024-06-04T04:45:27.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-04T04:45:27.734 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-04T04:45:27.734 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-04T04:45:27.734 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-04T04:45:27.734 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_open_files: -1 2024-06-04T04:45:27.734 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.bytes_per_sync: 0 2024-06-04T04:45:27.734 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-04T04:45:27.734 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-04T04:45:27.734 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compaction_readahead_size: 0 2024-06-04T04:45:27.734 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_background_flushes: -1 2024-06-04T04:45:27.734 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Compression algorithms supported: 2024-06-04T04:45:27.734 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-04T04:45:27.734 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: kZSTD supported: 0 2024-06-04T04:45:27.734 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: kXpressCompression supported: 0 2024-06-04T04:45:27.734 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: kLZ4HCCompression supported: 1 2024-06-04T04:45:27.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: kZlibCompression supported: 1 2024-06-04T04:45:27.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: kSnappyCompression supported: 1 2024-06-04T04:45:27.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: kLZ4Compression supported: 1 2024-06-04T04:45:27.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: kBZip2Compression supported: 0 2024-06-04T04:45:27.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-04T04:45:27.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-04T04:45:27.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-06-04T04:45:27.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-04T04:45:27.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-04T04:45:27.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.merge_operator: 2024-06-04T04:45:27.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compaction_filter: None 2024-06-04T04:45:27.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compaction_filter_factory: None 2024-06-04T04:45:27.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.sst_partitioner_factory: None 2024-06-04T04:45:27.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-04T04:45:27.736 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-04T04:45:27.736 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ef17eeff60) 2024-06-04T04:45:27.736 INFO:journalctl@ceph.mon.a.smithi070.stdout: cache_index_and_filter_blocks: 1 2024-06-04T04:45:27.736 INFO:journalctl@ceph.mon.a.smithi070.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-04T04:45:27.736 INFO:journalctl@ceph.mon.a.smithi070.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-04T04:45:27.736 INFO:journalctl@ceph.mon.a.smithi070.stdout: pin_top_level_index_and_filter: 1 2024-06-04T04:45:27.736 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_type: 0 2024-06-04T04:45:27.736 INFO:journalctl@ceph.mon.a.smithi070.stdout: data_block_index_type: 0 2024-06-04T04:45:27.736 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_shortening: 1 2024-06-04T04:45:27.736 INFO:journalctl@ceph.mon.a.smithi070.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-04T04:45:27.736 INFO:journalctl@ceph.mon.a.smithi070.stdout: checksum: 4 2024-06-04T04:45:27.736 INFO:journalctl@ceph.mon.a.smithi070.stdout: no_block_cache: 0 2024-06-04T04:45:27.736 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache: 0x55ef17ef31f0 2024-06-04T04:45:27.736 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_name: BinnedLRUCache 2024-06-04T04:45:27.737 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_options: 2024-06-04T04:45:27.737 INFO:journalctl@ceph.mon.a.smithi070.stdout: capacity : 536870912 2024-06-04T04:45:27.737 INFO:journalctl@ceph.mon.a.smithi070.stdout: num_shard_bits : 4 2024-06-04T04:45:27.737 INFO:journalctl@ceph.mon.a.smithi070.stdout: strict_capacity_limit : 0 2024-06-04T04:45:27.737 INFO:journalctl@ceph.mon.a.smithi070.stdout: high_pri_pool_ratio: 0.000 2024-06-04T04:45:27.737 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_compressed: (nil) 2024-06-04T04:45:27.737 INFO:journalctl@ceph.mon.a.smithi070.stdout: persistent_cache: (nil) 2024-06-04T04:45:27.737 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_size: 4096 2024-06-04T04:45:27.737 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_size_deviation: 10 2024-06-04T04:45:27.737 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_restart_interval: 16 2024-06-04T04:45:27.737 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_block_restart_interval: 1 2024-06-04T04:45:27.737 INFO:journalctl@ceph.mon.a.smithi070.stdout: metadata_block_size: 4096 2024-06-04T04:45:27.737 INFO:journalctl@ceph.mon.a.smithi070.stdout: partition_filters: 0 2024-06-04T04:45:27.737 INFO:journalctl@ceph.mon.a.smithi070.stdout: use_delta_encoding: 1 2024-06-04T04:45:27.738 INFO:journalctl@ceph.mon.a.smithi070.stdout: filter_policy: bloomfilter 2024-06-04T04:45:27.738 INFO:journalctl@ceph.mon.a.smithi070.stdout: whole_key_filtering: 1 2024-06-04T04:45:27.738 INFO:journalctl@ceph.mon.a.smithi070.stdout: verify_compression: 0 2024-06-04T04:45:27.738 INFO:journalctl@ceph.mon.a.smithi070.stdout: read_amp_bytes_per_bit: 0 2024-06-04T04:45:27.738 INFO:journalctl@ceph.mon.a.smithi070.stdout: format_version: 5 2024-06-04T04:45:27.738 INFO:journalctl@ceph.mon.a.smithi070.stdout: enable_index_compression: 1 2024-06-04T04:45:27.738 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_align: 0 2024-06-04T04:45:27.738 INFO:journalctl@ceph.mon.a.smithi070.stdout: max_auto_readahead_size: 262144 2024-06-04T04:45:27.738 INFO:journalctl@ceph.mon.a.smithi070.stdout: prepopulate_block_cache: 0 2024-06-04T04:45:27.738 INFO:journalctl@ceph.mon.a.smithi070.stdout: initial_auto_readahead_size: 8192 2024-06-04T04:45:27.738 INFO:journalctl@ceph.mon.a.smithi070.stdout: num_file_reads_for_auto_readahead: 2 2024-06-04T04:45:27.738 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.write_buffer_size: 33554432 2024-06-04T04:45:27.738 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_write_buffer_number: 2 2024-06-04T04:45:27.738 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compression: NoCompression 2024-06-04T04:45:27.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.bottommost_compression: Disabled 2024-06-04T04:45:27.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.prefix_extractor: nullptr 2024-06-04T04:45:27.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-04T04:45:27.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.num_levels: 7 2024-06-04T04:45:27.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-04T04:45:27.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-04T04:45:27.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-04T04:45:27.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-04T04:45:27.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-04T04:45:27.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-04T04:45:27.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-04T04:45:27.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-04T04:45:27.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-04T04:45:27.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-04T04:45:27.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-04T04:45:27.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-04T04:45:27.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-04T04:45:27.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compression_opts.level: 32767 2024-06-04T04:45:27.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compression_opts.strategy: 0 2024-06-04T04:45:27.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-04T04:45:27.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-04T04:45:27.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-04T04:45:27.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-04T04:45:27.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compression_opts.enabled: false 2024-06-04T04:45:27.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-04T04:45:27.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-04T04:45:27.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-04T04:45:27.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-04T04:45:27.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.target_file_size_base: 67108864 2024-06-04T04:45:27.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-04T04:45:27.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-04T04:45:27.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-04T04:45:27.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-04T04:45:27.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-04T04:45:27.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-04T04:45:27.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-04T04:45:27.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-04T04:45:27.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-04T04:45:27.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-04T04:45:27.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-04T04:45:27.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-04T04:45:27.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-04T04:45:27.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-04T04:45:27.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.arena_block_size: 1048576 2024-06-04T04:45:27.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-04T04:45:27.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-04T04:45:27.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.disable_auto_compactions: 0 2024-06-04T04:45:27.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-04T04:45:27.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-04T04:45:27.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-04T04:45:27.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-04T04:45:27.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-04T04:45:27.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-04T04:45:27.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-04T04:45:27.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-04T04:45:27.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-04T04:45:27.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-04T04:45:27.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-06-04T04:45:27.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.inplace_update_support: 0 2024-06-04T04:45:27.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-04T04:45:27.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-04T04:45:27.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-04T04:45:27.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-04T04:45:27.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.bloom_locality: 0 2024-06-04T04:45:27.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.max_successive_merges: 0 2024-06-04T04:45:27.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-04T04:45:27.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.paranoid_file_checks: 0 2024-06-04T04:45:27.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.force_consistency_checks: 1 2024-06-04T04:45:27.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.report_bg_io_stats: 0 2024-06-04T04:45:27.744 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.ttl: 2592000 2024-06-04T04:45:27.744 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-04T04:45:27.744 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-04T04:45:27.744 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-04T04:45:27.744 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.enable_blob_files: false 2024-06-04T04:45:27.744 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.min_blob_size: 0 2024-06-04T04:45:27.744 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.blob_file_size: 268435456 2024-06-04T04:45:27.744 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-04T04:45:27.744 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-04T04:45:27.744 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-04T04:45:27.744 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-04T04:45:27.744 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-04T04:45:27.744 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.blob_file_starting_level: 0 2024-06-04T04:45:27.744 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-04T04:45:27.745 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-06-04T04:45:27.745 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-06-04T04:45:27.745 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a9163f92-f6d8-4053-9c17-09e6ec1b4e75 2024-06-04T04:45:27.745 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717476327555695, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-06-04T04:45:27.745 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-06-04T04:45:27.745 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717476327556748, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99193, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 322, "table_properties": {"data_size": 97139, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13893, "raw_average_key_size": 49, "raw_value_size": 89558, "raw_average_value_size": 318, "num_data_blocks": 12, "num_entries": 281, "num_filter_entries": 281, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1717476327, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a9163f92-f6d8-4053-9c17-09e6ec1b4e75", "db_session_id": "RNFNGMZ8YP31HETLWZGR", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-06-04T04:45:27.745 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717476327556828, "job": 1, "event": "recovery_finished"} 2024-06-04T04:45:27.745 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-06-04T04:45:27.745 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-04T04:45:27.745 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ef18010000 2024-06-04T04:45:27.745 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: DB pointer 0x55ef17ffc000 2024-06-04T04:45:27.745 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-04T04:45:27.745 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-04T04:45:27.745 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** DB Stats ** 2024-06-04T04:45:27.746 INFO:journalctl@ceph.mon.a.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-04T04:45:27.746 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-04T04:45:27.746 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-04T04:45:27.746 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-04T04:45:27.746 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-04T04:45:27.746 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-04T04:45:27.746 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-04T04:45:27.746 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-06-04T04:45:27.746 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** Compaction Stats [default] ** 2024-06-04T04:45:27.746 INFO:journalctl@ceph.mon.a.smithi070.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-04T04:45:27.746 INFO:journalctl@ceph.mon.a.smithi070.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-04T04:45:27.746 INFO:journalctl@ceph.mon.a.smithi070.stdout: L0 2/0 98.73 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 140.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-04T04:45:27.746 INFO:journalctl@ceph.mon.a.smithi070.stdout: Sum 2/0 98.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 140.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-04T04:45:27.746 INFO:journalctl@ceph.mon.a.smithi070.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 140.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-04T04:45:27.747 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-06-04T04:45:27.747 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** Compaction Stats [default] ** 2024-06-04T04:45:27.747 INFO:journalctl@ceph.mon.a.smithi070.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-04T04:45:27.747 INFO:journalctl@ceph.mon.a.smithi070.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-04T04:45:27.747 INFO:journalctl@ceph.mon.a.smithi070.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 140.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-04T04:45:27.747 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-06-04T04:45:27.747 INFO:journalctl@ceph.mon.a.smithi070.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-04T04:45:27.747 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-06-04T04:45:27.747 INFO:journalctl@ceph.mon.a.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-04T04:45:27.747 INFO:journalctl@ceph.mon.a.smithi070.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-04T04:45:27.747 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-04T04:45:27.747 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-04T04:45:27.747 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-04T04:45:27.747 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-04T04:45:27.747 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative compaction: 0.00 GB write, 30.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-04T04:45:27.747 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval compaction: 0.00 GB write, 30.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-04T04:45:27.748 INFO:journalctl@ceph.mon.a.smithi070.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-04T04:45:27.748 INFO:journalctl@ceph.mon.a.smithi070.stdout: Block cache BinnedLRUCache@0x55ef17ef31f0#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-06-04T04:45:27.748 INFO:journalctl@ceph.mon.a.smithi070.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-06-04T04:45:27.748 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-06-04T04:45:27.748 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-04T04:45:27.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: starting mon.a rank 0 at public addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] at bind addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:45:27.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: mon.a@-1(???) e1 preinit fsid 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:45:27.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: mon.a@-1(???).mds e1 new map 2024-06-04T04:45:27.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: mon.a@-1(???).mds e1 print_map 2024-06-04T04:45:27.748 INFO:journalctl@ceph.mon.a.smithi070.stdout: e1 2024-06-04T04:45:27.748 INFO:journalctl@ceph.mon.a.smithi070.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-04T04:45:27.748 INFO:journalctl@ceph.mon.a.smithi070.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-06-04T04:45:27.748 INFO:journalctl@ceph.mon.a.smithi070.stdout: legacy client fscid: -1 2024-06-04T04:45:27.748 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-06-04T04:45:27.748 INFO:journalctl@ceph.mon.a.smithi070.stdout: No filesystems configured 2024-06-04T04:45:27.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-04T04:45:27.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-04T04:45:27.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-04T04:45:27.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-04T04:45:27.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 ceph-mon[32631]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-06-04T04:45:27.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 bash[32612]: e29249260accb47368f128a967c2d241c9c8843c9a20926d82ae02f15fb6bbe1 2024-06-04T04:45:27.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:27 smithi070 systemd[1]: Started Ceph mon.a for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:45:28.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:28 smithi070 ceph-mon[32631]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-04T04:45:28.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:28 smithi070 ceph-mon[32631]: monmap epoch 1 2024-06-04T04:45:28.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:28 smithi070 ceph-mon[32631]: fsid 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:45:28.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:28 smithi070 ceph-mon[32631]: last_changed 2024-06-04T04:45:09.662402+0000 2024-06-04T04:45:28.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:28 smithi070 ceph-mon[32631]: created 2024-06-04T04:45:09.662402+0000 2024-06-04T04:45:28.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:28 smithi070 ceph-mon[32631]: min_mon_release 19 (squid) 2024-06-04T04:45:28.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:28 smithi070 ceph-mon[32631]: election_strategy: 1 2024-06-04T04:45:28.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:28 smithi070 ceph-mon[32631]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-06-04T04:45:28.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:28 smithi070 ceph-mon[32631]: fsmap 2024-06-04T04:45:28.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:28 smithi070 ceph-mon[32631]: osdmap e1: 0 total, 0 up, 0 in 2024-06-04T04:45:28.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:28 smithi070 ceph-mon[32631]: mgrmap e1: no daemons active 2024-06-04T04:45:28.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:28 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3817208192' entity='client.admin' 2024-06-04T04:45:28.955 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.070+0000 7f36dfdec700 1 Processor -- start 2024-06-04T04:45:28.955 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.070+0000 7f36dfdec700 1 -- start start 2024-06-04T04:45:28.955 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.070+0000 7f36dfdec700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36d8107770 0x7f36d8107b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:28.955 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.070+0000 7f36dfdec700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f36d807ce10 con 0x7f36d8107770 2024-06-04T04:45:28.955 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.071+0000 7f36ddb88700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36d8107770 0x7f36d8107b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:28.955 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.071+0000 7f36ddb88700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36d8107770 0x7f36d8107b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36222/0 (socket says 172.21.15.70:36222) 2024-06-04T04:45:28.956 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.071+0000 7f36ddb88700 1 -- 172.21.15.70:0/3189277037 learned_addr learned my addr 172.21.15.70:0/3189277037 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:28.956 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.071+0000 7f36ddb88700 1 -- 172.21.15.70:0/3189277037 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d81080d0 con 0x7f36d8107770 2024-06-04T04:45:28.956 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.071+0000 7f36ddb88700 1 --2- 172.21.15.70:0/3189277037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36d8107770 0x7f36d8107b90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f36d4009e80 tx=0x7f36d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f12b01c9385c289d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:28.956 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.071+0000 7f36dcb86700 1 -- 172.21.15.70:0/3189277037 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36d4017070 con 0x7f36d8107770 2024-06-04T04:45:28.956 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.071+0000 7f36dcb86700 1 -- 172.21.15.70:0/3189277037 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f36d400b8c0 con 0x7f36d8107770 2024-06-04T04:45:28.956 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.071+0000 7f36dcb86700 1 -- 172.21.15.70:0/3189277037 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36d4004420 con 0x7f36d8107770 2024-06-04T04:45:28.956 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.072+0000 7f36dfdec700 1 -- 172.21.15.70:0/3189277037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36d8107770 msgr2=0x7f36d8107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:28.956 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.072+0000 7f36dfdec700 1 --2- 172.21.15.70:0/3189277037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36d8107770 0x7f36d8107b90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f36d4009e80 tx=0x7f36d400b330 comp rx=0 tx=0).stop 2024-06-04T04:45:28.956 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.072+0000 7f36dfdec700 1 -- 172.21.15.70:0/3189277037 shutdown_connections 2024-06-04T04:45:28.956 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.072+0000 7f36dfdec700 1 --2- 172.21.15.70:0/3189277037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36d8107770 0x7f36d8107b90 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:28.956 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.072+0000 7f36dfdec700 1 -- 172.21.15.70:0/3189277037 >> 172.21.15.70:0/3189277037 conn(0x7f36d8103310 msgr2=0x7f36d8105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:28.956 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.072+0000 7f36dfdec700 1 -- 172.21.15.70:0/3189277037 shutdown_connections 2024-06-04T04:45:28.956 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.072+0000 7f36dfdec700 1 -- 172.21.15.70:0/3189277037 wait complete. 2024-06-04T04:45:28.957 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.073+0000 7f36dfdec700 1 Processor -- start 2024-06-04T04:45:28.957 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.073+0000 7f36dfdec700 1 -- start start 2024-06-04T04:45:28.957 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.073+0000 7f36dfdec700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36d8107770 0x7f36d819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:28.957 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.073+0000 7f36dfdec700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f36d81090a0 con 0x7f36d8107770 2024-06-04T04:45:28.957 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.073+0000 7f36ddb88700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36d8107770 0x7f36d819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:28.957 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.073+0000 7f36ddb88700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36d8107770 0x7f36d819ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36228/0 (socket says 172.21.15.70:36228) 2024-06-04T04:45:28.957 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.073+0000 7f36ddb88700 1 -- 172.21.15.70:0/3817208192 learned_addr learned my addr 172.21.15.70:0/3817208192 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:28.957 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.073+0000 7f36ddb88700 1 -- 172.21.15.70:0/3817208192 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d819f4d0 con 0x7f36d8107770 2024-06-04T04:45:28.957 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.073+0000 7f36ddb88700 1 --2- 172.21.15.70:0/3817208192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36d8107770 0x7f36d819ef90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f36d4000f80 tx=0x7f36d4003ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:28.957 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.074+0000 7f36ceffd700 1 -- 172.21.15.70:0/3817208192 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36d4020020 con 0x7f36d8107770 2024-06-04T04:45:28.957 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.074+0000 7f36ceffd700 1 -- 172.21.15.70:0/3817208192 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f36d4003e20 con 0x7f36d8107770 2024-06-04T04:45:28.957 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.074+0000 7f36dfdec700 1 -- 172.21.15.70:0/3817208192 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36d819f760 con 0x7f36d8107770 2024-06-04T04:45:28.957 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.074+0000 7f36ceffd700 1 -- 172.21.15.70:0/3817208192 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36d4017040 con 0x7f36d8107770 2024-06-04T04:45:28.958 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.074+0000 7f36dfdec700 1 -- 172.21.15.70:0/3817208192 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36d81a2450 con 0x7f36d8107770 2024-06-04T04:45:28.958 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.074+0000 7f36ceffd700 1 -- 172.21.15.70:0/3817208192 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f36d4023020 con 0x7f36d8107770 2024-06-04T04:45:28.958 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.074+0000 7f36ceffd700 1 -- 172.21.15.70:0/3817208192 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f36d400f050 con 0x7f36d8107770 2024-06-04T04:45:28.958 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.075+0000 7f36dfdec700 1 -- 172.21.15.70:0/3817208192 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36d8064ba0 con 0x7f36d8107770 2024-06-04T04:45:28.958 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.077+0000 7f36ceffd700 1 -- 172.21.15.70:0/3817208192 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f36d402a020 con 0x7f36d8107770 2024-06-04T04:45:28.958 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.144+0000 7f36dfdec700 1 -- 172.21.15.70:0/3817208192 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f36d802e6d0 con 0x7f36d8107770 2024-06-04T04:45:28.958 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.146+0000 7f36ceffd700 1 -- 172.21.15.70:0/3817208192 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f36d400b860 con 0x7f36d8107770 2024-06-04T04:45:28.958 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.147+0000 7f36dfdec700 1 -- 172.21.15.70:0/3817208192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36d8107770 msgr2=0x7f36d819ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:28.958 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.147+0000 7f36dfdec700 1 --2- 172.21.15.70:0/3817208192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36d8107770 0x7f36d819ef90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f36d4000f80 tx=0x7f36d4003ad0 comp rx=0 tx=0).stop 2024-06-04T04:45:28.958 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.147+0000 7f36dfdec700 1 -- 172.21.15.70:0/3817208192 shutdown_connections 2024-06-04T04:45:28.958 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.147+0000 7f36dfdec700 1 --2- 172.21.15.70:0/3817208192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36d8107770 0x7f36d819ef90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:28.958 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.147+0000 7f36dfdec700 1 -- 172.21.15.70:0/3817208192 >> 172.21.15.70:0/3817208192 conn(0x7f36d8103310 msgr2=0x7f36d8195b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:28.959 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.147+0000 7f36dfdec700 1 -- 172.21.15.70:0/3817208192 shutdown_connections 2024-06-04T04:45:28.959 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:28.147+0000 7f36dfdec700 1 -- 172.21.15.70:0/3817208192 wait complete. 2024-06-04T04:45:28.959 INFO:teuthology.orchestra.run.smithi070.stdout:Wrote config to /etc/ceph/ceph.conf 2024-06-04T04:45:28.981 INFO:teuthology.orchestra.run.smithi070.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-06-04T04:45:28.982 INFO:teuthology.orchestra.run.smithi070.stdout:Creating mgr... 2024-06-04T04:45:28.982 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying port 0.0.0.0:9283 ... 2024-06-04T04:45:28.983 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying port 0.0.0.0:8765 ... 2024-06-04T04:45:29.241 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mgr.x 2024-06-04T04:45:29.241 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Failed to reset failed state of unit ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mgr.x.service: Unit ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mgr.x.service not loaded. 2024-06-04T04:45:29.385 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-164dbaba-222d-11ef-bc9b-c7b262605968.target.wants/ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-164dbaba-222d-11ef-bc9b-c7b262605968@.service. 2024-06-04T04:45:29.609 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:29 smithi070 systemd[1]: Starting Ceph mgr.x for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T04:45:29.851 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-04T04:45:29.851 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout disabled 2024-06-04T04:45:29.858 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-04T04:45:29.859 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-06-04T04:45:29.859 INFO:teuthology.orchestra.run.smithi070.stdout:firewalld.service is not enabled 2024-06-04T04:45:29.859 INFO:teuthology.orchestra.run.smithi070.stdout:Not possible to enable service . firewalld.service is not available 2024-06-04T04:45:29.867 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-04T04:45:29.867 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout disabled 2024-06-04T04:45:29.874 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-04T04:45:29.874 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-06-04T04:45:29.875 INFO:teuthology.orchestra.run.smithi070.stdout:firewalld.service is not enabled 2024-06-04T04:45:29.875 INFO:teuthology.orchestra.run.smithi070.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-06-04T04:45:29.875 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mgr to start... 2024-06-04T04:45:29.875 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mgr... 2024-06-04T04:45:29.934 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:29 smithi070 podman[32845]: 2024-06-04 04:45:29.607804948 +0000 UTC m=+0.106191818 container create b162b5a386994b5216cbdedc938e3e215a665f9b393d5542265d174dad9867e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-b7328fe, CEPH_POINT_RELEASE=) 2024-06-04T04:45:29.934 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:29 smithi070 podman[32845]: 2024-06-04 04:45:29.518450646 +0000 UTC m=+0.016837520 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 2024-06-04T04:45:29.934 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:29 smithi070 podman[32845]: 2024-06-04 04:45:29.768599084 +0000 UTC m=+0.266985973 container init b162b5a386994b5216cbdedc938e3e215a665f9b393d5542265d174dad9867e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-06-04T04:45:29.934 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:29 smithi070 podman[32845]: 2024-06-04 04:45:29.771333994 +0000 UTC m=+0.269720873 container start b162b5a386994b5216cbdedc938e3e215a665f9b393d5542265d174dad9867e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-04T04:45:29.934 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:29 smithi070 bash[32845]: b162b5a386994b5216cbdedc938e3e215a665f9b393d5542265d174dad9867e9 2024-06-04T04:45:29.934 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:29 smithi070 systemd[1]: Started Ceph mgr.x for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:45:29.935 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:29 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:29.931+0000 7f1ee3a86200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-04T04:45:30.185 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:30 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:30.121+0000 7f1ee3a86200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-04T04:45:30.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:30 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2001005908' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-04T04:45:30.546 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:30 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:30.224+0000 7f1ee3a86200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-04T04:45:30.547 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:30 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:30.543+0000 7f1ee3a86200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-04T04:45:30.859 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:30 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:30.631+0000 7f1ee3a86200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-04T04:45:30.966 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-06-04T04:45:30.966 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-06-04T04:45:30.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:45:30.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "health": { 2024-06-04T04:45:30.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-04T04:45:30.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-04T04:45:30.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-04T04:45:30.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:30.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-04T04:45:30.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-04T04:45:30.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 0 2024-06-04T04:45:30.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-06-04T04:45:30.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-04T04:45:30.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "a" 2024-06-04T04:45:30.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-06-04T04:45:30.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-06-04T04:45:30.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-04T04:45:30.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-04T04:45:30.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-06-04T04:45:30.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-04T04:45:30.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:30.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-04T04:45:30.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-04T04:45:30.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-04T04:45:30.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-04T04:45:30.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-04T04:45:30.976 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-04T04:45:30.976 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-04T04:45:30.976 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-04T04:45:30.976 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:30.976 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-04T04:45:30.976 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-04T04:45:30.976 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-04T04:45:30.977 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-04T04:45:30.977 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-04T04:45:30.977 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-04T04:45:30.977 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-04T04:45:30.977 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-04T04:45:30.977 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-04T04:45:30.977 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:30.977 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-04T04:45:30.977 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-04T04:45:30.977 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-04T04:45:30.977 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-04T04:45:30.977 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:30.977 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-04T04:45:30.978 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-04T04:45:30.978 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-04T04:45:30.978 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-04T04:45:30.978 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-04T04:45:30.978 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-04T04:45:30.978 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "restful" 2024-06-04T04:45:30.978 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-06-04T04:45:30.978 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-04T04:45:30.978 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:30.978 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-04T04:45:30.978 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-04T04:45:30.978 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modified": "2024-06-04T04:45:22.297998+0000", 2024-06-04T04:45:30.978 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-04T04:45:30.979 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:30.979 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-04T04:45:30.979 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-06-04T04:45:30.979 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.280+0000 7f3d22954700 1 Processor -- start 2024-06-04T04:45:30.979 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.281+0000 7f3d22954700 1 -- start start 2024-06-04T04:45:30.979 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.281+0000 7f3d22954700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d1c107710 0x7f3d1c107b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:30.979 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.281+0000 7f3d22954700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c07ce30 con 0x7f3d1c107710 2024-06-04T04:45:30.979 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.281+0000 7f3d1bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d1c107710 0x7f3d1c107b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:30.979 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.281+0000 7f3d1bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d1c107710 0x7f3d1c107b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36250/0 (socket says 172.21.15.70:36250) 2024-06-04T04:45:30.979 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.281+0000 7f3d1bfff700 1 -- 172.21.15.70:0/2500337524 learned_addr learned my addr 172.21.15.70:0/2500337524 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:30.979 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.281+0000 7f3d1bfff700 1 -- 172.21.15.70:0/2500337524 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1c108070 con 0x7f3d1c107710 2024-06-04T04:45:30.979 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.281+0000 7f3d1bfff700 1 --2- 172.21.15.70:0/2500337524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d1c107710 0x7f3d1c107b30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c0098a0 tx=0x7f3d0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc66ea281f963e3f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:30.979 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.282+0000 7f3d1affd700 1 -- 172.21.15.70:0/2500337524 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d0c004160 con 0x7f3d1c107710 2024-06-04T04:45:30.979 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.282+0000 7f3d1affd700 1 -- 172.21.15.70:0/2500337524 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f3d0c004300 con 0x7f3d1c107710 2024-06-04T04:45:30.980 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.282+0000 7f3d1affd700 1 -- 172.21.15.70:0/2500337524 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d0c0119e0 con 0x7f3d1c107710 2024-06-04T04:45:30.980 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.282+0000 7f3d22954700 1 -- 172.21.15.70:0/2500337524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d1c107710 msgr2=0x7f3d1c107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:30.980 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.282+0000 7f3d22954700 1 --2- 172.21.15.70:0/2500337524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d1c107710 0x7f3d1c107b30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c0098a0 tx=0x7f3d0c009bb0 comp rx=0 tx=0).stop 2024-06-04T04:45:30.980 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.282+0000 7f3d22954700 1 -- 172.21.15.70:0/2500337524 shutdown_connections 2024-06-04T04:45:30.980 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.282+0000 7f3d22954700 1 --2- 172.21.15.70:0/2500337524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d1c107710 0x7f3d1c107b30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:30.980 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.282+0000 7f3d22954700 1 -- 172.21.15.70:0/2500337524 >> 172.21.15.70:0/2500337524 conn(0x7f3d1c103330 msgr2=0x7f3d1c105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:30.980 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.282+0000 7f3d22954700 1 -- 172.21.15.70:0/2500337524 shutdown_connections 2024-06-04T04:45:30.980 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.282+0000 7f3d22954700 1 -- 172.21.15.70:0/2500337524 wait complete. 2024-06-04T04:45:30.980 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.283+0000 7f3d22954700 1 Processor -- start 2024-06-04T04:45:30.980 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.283+0000 7f3d22954700 1 -- start start 2024-06-04T04:45:30.980 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.283+0000 7f3d22954700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d1c107710 0x7f3d1c19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:30.980 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.283+0000 7f3d22954700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c109040 con 0x7f3d1c107710 2024-06-04T04:45:30.980 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.283+0000 7f3d1bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d1c107710 0x7f3d1c19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:30.980 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.284+0000 7f3d1bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d1c107710 0x7f3d1c19ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36254/0 (socket says 172.21.15.70:36254) 2024-06-04T04:45:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.284+0000 7f3d1bfff700 1 -- 172.21.15.70:0/2001005908 learned_addr learned my addr 172.21.15.70:0/2001005908 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.284+0000 7f3d1bfff700 1 -- 172.21.15.70:0/2001005908 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1c19f450 con 0x7f3d1c107710 2024-06-04T04:45:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.284+0000 7f3d1bfff700 1 --2- 172.21.15.70:0/2001005908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d1c107710 0x7f3d1c19ef10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c0119e0 tx=0x7f3d0c0046a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.284+0000 7f3d197fa700 1 -- 172.21.15.70:0/2001005908 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d0c003bf0 con 0x7f3d1c107710 2024-06-04T04:45:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.284+0000 7f3d197fa700 1 -- 172.21.15.70:0/2001005908 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f3d0c01a8a0 con 0x7f3d1c107710 2024-06-04T04:45:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.284+0000 7f3d22954700 1 -- 172.21.15.70:0/2001005908 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d1c19f6e0 con 0x7f3d1c107710 2024-06-04T04:45:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.284+0000 7f3d197fa700 1 -- 172.21.15.70:0/2001005908 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d0c0248c0 con 0x7f3d1c107710 2024-06-04T04:45:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.284+0000 7f3d22954700 1 -- 172.21.15.70:0/2001005908 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d1c1a23d0 con 0x7f3d1c107710 2024-06-04T04:45:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.284+0000 7f3d197fa700 1 -- 172.21.15.70:0/2001005908 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3d0c022020 con 0x7f3d1c107710 2024-06-04T04:45:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.284+0000 7f3d197fa700 1 -- 172.21.15.70:0/2001005908 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3d0c0193c0 con 0x7f3d1c107710 2024-06-04T04:45:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.285+0000 7f3d22954700 1 -- 172.21.15.70:0/2001005908 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d1c064ba0 con 0x7f3d1c107710 2024-06-04T04:45:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.287+0000 7f3d197fa700 1 -- 172.21.15.70:0/2001005908 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f3d0c043ec0 con 0x7f3d1c107710 2024-06-04T04:45:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.357+0000 7f3d22954700 1 -- 172.21.15.70:0/2001005908 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f3d1c02e600 con 0x7f3d1c107710 2024-06-04T04:45:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.358+0000 7f3d197fa700 1 -- 172.21.15.70:0/2001005908 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f3d0c032030 con 0x7f3d1c107710 2024-06-04T04:45:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.359+0000 7f3d22954700 1 -- 172.21.15.70:0/2001005908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d1c107710 msgr2=0x7f3d1c19ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.359+0000 7f3d22954700 1 --2- 172.21.15.70:0/2001005908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d1c107710 0x7f3d1c19ef10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c0119e0 tx=0x7f3d0c0046a0 comp rx=0 tx=0).stop 2024-06-04T04:45:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.359+0000 7f3d22954700 1 -- 172.21.15.70:0/2001005908 shutdown_connections 2024-06-04T04:45:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.359+0000 7f3d22954700 1 --2- 172.21.15.70:0/2001005908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d1c107710 0x7f3d1c19ef10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.359+0000 7f3d22954700 1 -- 172.21.15.70:0/2001005908 >> 172.21.15.70:0/2001005908 conn(0x7f3d1c103330 msgr2=0x7f3d1c195b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.359+0000 7f3d22954700 1 -- 172.21.15.70:0/2001005908 shutdown_connections 2024-06-04T04:45:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:30.359+0000 7f3d22954700 1 -- 172.21.15.70:0/2001005908 wait complete. 2024-06-04T04:45:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:mgr not available, waiting (1/15)... 2024-06-04T04:45:32.109 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:31 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:31.828+0000 7f1ee3a86200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-04T04:45:32.109 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:31 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:31.998+0000 7f1ee3a86200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-04T04:45:32.458 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:32 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:32.168+0000 7f1ee3a86200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-04T04:45:32.458 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:32 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:32.455+0000 7f1ee3a86200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-04T04:45:32.750 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:32 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:32.547+0000 7f1ee3a86200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-04T04:45:32.750 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:32 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:32.748+0000 7f1ee3a86200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-04T04:45:33.087 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:32 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:32.939+0000 7f1ee3a86200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-04T04:45:33.645 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:33 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1015785244' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-04T04:45:33.646 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:33 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:33.571+0000 7f1ee3a86200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-04T04:45:34.043 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:33 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:33.802+0000 7f1ee3a86200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-06-04T04:45:34.046 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-06-04T04:45:34.048 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-06-04T04:45:34.048 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:45:34.048 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "health": { 2024-06-04T04:45:34.048 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-04T04:45:34.048 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-04T04:45:34.049 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-04T04:45:34.049 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:34.049 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-04T04:45:34.049 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-04T04:45:34.049 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 0 2024-06-04T04:45:34.049 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-06-04T04:45:34.049 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-04T04:45:34.049 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "a" 2024-06-04T04:45:34.049 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-06-04T04:45:34.049 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-06-04T04:45:34.049 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-04T04:45:34.049 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-04T04:45:34.049 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-06-04T04:45:34.049 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-04T04:45:34.049 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:34.050 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-04T04:45:34.050 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-04T04:45:34.050 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-04T04:45:34.050 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-04T04:45:34.050 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-04T04:45:34.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-04T04:45:34.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-04T04:45:34.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-04T04:45:34.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:34.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-04T04:45:34.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-04T04:45:34.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-04T04:45:34.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-04T04:45:34.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-04T04:45:34.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-04T04:45:34.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-04T04:45:34.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-04T04:45:34.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-04T04:45:34.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:34.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-04T04:45:34.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-04T04:45:34.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-04T04:45:34.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-04T04:45:34.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:34.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-04T04:45:34.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-04T04:45:34.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-04T04:45:34.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-04T04:45:34.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-04T04:45:34.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-04T04:45:34.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "restful" 2024-06-04T04:45:34.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-06-04T04:45:34.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-04T04:45:34.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:34.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-04T04:45:34.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-04T04:45:34.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modified": "2024-06-04T04:45:22.297998+0000", 2024-06-04T04:45:34.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-04T04:45:34.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:34.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-04T04:45:34.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-06-04T04:45:34.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.389+0000 7f21b2dba700 1 Processor -- start 2024-06-04T04:45:34.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.390+0000 7f21b2dba700 1 -- start start 2024-06-04T04:45:34.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.390+0000 7f21b2dba700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac109950 0x7f21ac10bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:34.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.390+0000 7f21b2dba700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f21ac07ce50 con 0x7f21ac109950 2024-06-04T04:45:34.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.390+0000 7f21b0b56700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac109950 0x7f21ac10bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:34.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.390+0000 7f21b0b56700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac109950 0x7f21ac10bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36256/0 (socket says 172.21.15.70:36256) 2024-06-04T04:45:34.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.390+0000 7f21b0b56700 1 -- 172.21.15.70:0/360938040 learned_addr learned my addr 172.21.15.70:0/360938040 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:34.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.390+0000 7f21b0b56700 1 -- 172.21.15.70:0/360938040 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21ac10c280 con 0x7f21ac109950 2024-06-04T04:45:34.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.391+0000 7f21b0b56700 1 --2- 172.21.15.70:0/360938040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac109950 0x7f21ac10bd40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f21a00098a0 tx=0x7f21a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f956eaf58d79858 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:34.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.391+0000 7f21ab7fe700 1 -- 172.21.15.70:0/360938040 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21a0004160 con 0x7f21ac109950 2024-06-04T04:45:34.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.391+0000 7f21ab7fe700 1 -- 172.21.15.70:0/360938040 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f21a0004300 con 0x7f21ac109950 2024-06-04T04:45:34.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.391+0000 7f21ab7fe700 1 -- 172.21.15.70:0/360938040 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21a00119e0 con 0x7f21ac109950 2024-06-04T04:45:34.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.392+0000 7f21b2dba700 1 -- 172.21.15.70:0/360938040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac109950 msgr2=0x7f21ac10bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:34.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.392+0000 7f21b2dba700 1 --2- 172.21.15.70:0/360938040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac109950 0x7f21ac10bd40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f21a00098a0 tx=0x7f21a0009bb0 comp rx=0 tx=0).stop 2024-06-04T04:45:34.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.392+0000 7f21b2dba700 1 -- 172.21.15.70:0/360938040 shutdown_connections 2024-06-04T04:45:34.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.392+0000 7f21b2dba700 1 --2- 172.21.15.70:0/360938040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac109950 0x7f21ac10bd40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:34.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.392+0000 7f21b2dba700 1 -- 172.21.15.70:0/360938040 >> 172.21.15.70:0/360938040 conn(0x7f21ac103340 msgr2=0x7f21ac105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:34.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.392+0000 7f21b2dba700 1 -- 172.21.15.70:0/360938040 shutdown_connections 2024-06-04T04:45:34.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.392+0000 7f21b2dba700 1 -- 172.21.15.70:0/360938040 wait complete. 2024-06-04T04:45:34.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.392+0000 7f21b2dba700 1 Processor -- start 2024-06-04T04:45:34.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.393+0000 7f21b2dba700 1 -- start start 2024-06-04T04:45:34.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.393+0000 7f21b2dba700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac109950 0x7f21ac19cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:34.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.393+0000 7f21b2dba700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f21ac10d1e0 con 0x7f21ac109950 2024-06-04T04:45:34.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.393+0000 7f21b0b56700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac109950 0x7f21ac19cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:34.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.393+0000 7f21b0b56700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac109950 0x7f21ac19cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36262/0 (socket says 172.21.15.70:36262) 2024-06-04T04:45:34.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.393+0000 7f21b0b56700 1 -- 172.21.15.70:0/1015785244 learned_addr learned my addr 172.21.15.70:0/1015785244 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:34.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.393+0000 7f21b0b56700 1 -- 172.21.15.70:0/1015785244 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21ac19d270 con 0x7f21ac109950 2024-06-04T04:45:34.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.393+0000 7f21b0b56700 1 --2- 172.21.15.70:0/1015785244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac109950 0x7f21ac19cd30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f21a00119e0 tx=0x7f21a00046a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:34.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.394+0000 7f21a9ffb700 1 -- 172.21.15.70:0/1015785244 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21a0003bf0 con 0x7f21ac109950 2024-06-04T04:45:34.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.394+0000 7f21b2dba700 1 -- 172.21.15.70:0/1015785244 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21ac19d500 con 0x7f21ac109950 2024-06-04T04:45:34.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.394+0000 7f21a9ffb700 1 -- 172.21.15.70:0/1015785244 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f21a001a8a0 con 0x7f21ac109950 2024-06-04T04:45:34.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.394+0000 7f21a9ffb700 1 -- 172.21.15.70:0/1015785244 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21a00248c0 con 0x7f21ac109950 2024-06-04T04:45:34.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.394+0000 7f21b2dba700 1 -- 172.21.15.70:0/1015785244 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21ac1a01f0 con 0x7f21ac109950 2024-06-04T04:45:34.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.394+0000 7f21a9ffb700 1 -- 172.21.15.70:0/1015785244 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f21a0022020 con 0x7f21ac109950 2024-06-04T04:45:34.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.394+0000 7f21a9ffb700 1 -- 172.21.15.70:0/1015785244 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f21a00193c0 con 0x7f21ac109950 2024-06-04T04:45:34.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.395+0000 7f21b2dba700 1 -- 172.21.15.70:0/1015785244 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21ac064ba0 con 0x7f21ac109950 2024-06-04T04:45:34.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.399+0000 7f21a9ffb700 1 -- 172.21.15.70:0/1015785244 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f21a0043ec0 con 0x7f21ac109950 2024-06-04T04:45:34.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.471+0000 7f21b2dba700 1 -- 172.21.15.70:0/1015785244 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f21ac1a05f0 con 0x7f21ac109950 2024-06-04T04:45:34.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.472+0000 7f21a9ffb700 1 -- 172.21.15.70:0/1015785244 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f21a0032030 con 0x7f21ac109950 2024-06-04T04:45:34.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.473+0000 7f21b2dba700 1 -- 172.21.15.70:0/1015785244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac109950 msgr2=0x7f21ac19cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:34.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.473+0000 7f21b2dba700 1 --2- 172.21.15.70:0/1015785244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac109950 0x7f21ac19cd30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f21a00119e0 tx=0x7f21a00046a0 comp rx=0 tx=0).stop 2024-06-04T04:45:34.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.473+0000 7f21b2dba700 1 -- 172.21.15.70:0/1015785244 shutdown_connections 2024-06-04T04:45:34.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.473+0000 7f21b2dba700 1 --2- 172.21.15.70:0/1015785244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac109950 0x7f21ac19cd30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:34.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.473+0000 7f21b2dba700 1 -- 172.21.15.70:0/1015785244 >> 172.21.15.70:0/1015785244 conn(0x7f21ac103340 msgr2=0x7f21ac103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:34.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.473+0000 7f21b2dba700 1 -- 172.21.15.70:0/1015785244 shutdown_connections 2024-06-04T04:45:34.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:33.473+0000 7f21b2dba700 1 -- 172.21.15.70:0/1015785244 wait complete. 2024-06-04T04:45:34.063 INFO:teuthology.orchestra.run.smithi070.stdout:mgr not available, waiting (2/15)... 2024-06-04T04:45:34.333 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:34 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:34.234+0000 7f1ee3a86200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-04T04:45:34.333 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:34 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:34.331+0000 7f1ee3a86200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-04T04:45:35.047 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:34 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:34.698+0000 7f1ee3a86200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-04T04:45:35.047 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:35 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:35.045+0000 7f1ee3a86200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-04T04:45:35.359 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:35 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:35.136+0000 7f1ee3a86200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-04T04:45:35.859 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:35 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:35.375+0000 7f1ee3a86200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-04T04:45:35.859 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:35 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:35.505+0000 7f1ee3a86200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-04T04:45:35.859 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:35 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:35.599+0000 7f1ee3a86200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-04T04:45:36.177 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:35 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:35.918+0000 7f1ee3a86200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-04T04:45:36.177 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:36 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:36.008+0000 7f1ee3a86200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-04T04:45:36.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:36 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2457920701' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-04T04:45:36.859 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:36 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:36.641+0000 7f1ee3a86200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-04T04:45:37.169 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:36 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:36.864+0000 7f1ee3a86200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-04T04:45:37.173 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-06-04T04:45:37.173 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-06-04T04:45:37.173 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:45:37.173 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "health": { 2024-06-04T04:45:37.173 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-04T04:45:37.173 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-04T04:45:37.173 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-04T04:45:37.173 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:37.174 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-04T04:45:37.174 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-04T04:45:37.174 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 0 2024-06-04T04:45:37.174 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-06-04T04:45:37.174 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-04T04:45:37.174 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "a" 2024-06-04T04:45:37.174 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-06-04T04:45:37.174 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-06-04T04:45:37.174 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-04T04:45:37.174 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-04T04:45:37.174 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-06-04T04:45:37.174 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-04T04:45:37.174 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:37.174 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-04T04:45:37.175 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-04T04:45:37.175 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-04T04:45:37.175 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-04T04:45:37.182 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-04T04:45:37.182 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-04T04:45:37.182 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-04T04:45:37.182 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-04T04:45:37.182 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:37.182 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-04T04:45:37.182 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-04T04:45:37.182 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-04T04:45:37.182 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-04T04:45:37.182 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-04T04:45:37.182 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-04T04:45:37.182 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-04T04:45:37.183 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-04T04:45:37.183 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-04T04:45:37.183 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:37.183 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-04T04:45:37.183 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-04T04:45:37.183 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-04T04:45:37.183 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-04T04:45:37.183 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:37.183 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-04T04:45:37.183 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-04T04:45:37.183 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-04T04:45:37.183 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-04T04:45:37.183 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-04T04:45:37.183 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-04T04:45:37.184 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "restful" 2024-06-04T04:45:37.184 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-06-04T04:45:37.184 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-04T04:45:37.184 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:37.184 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-04T04:45:37.184 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-04T04:45:37.184 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modified": "2024-06-04T04:45:22.297998+0000", 2024-06-04T04:45:37.184 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-04T04:45:37.184 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:37.184 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-04T04:45:37.184 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-06-04T04:45:37.184 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.451+0000 7fb986268700 1 Processor -- start 2024-06-04T04:45:37.184 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.451+0000 7fb986268700 1 -- start start 2024-06-04T04:45:37.184 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.451+0000 7fb986268700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb980107db0 0x7fb9801081d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:37.184 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.451+0000 7fb986268700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9801087a0 con 0x7fb980107db0 2024-06-04T04:45:37.185 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.452+0000 7fb97ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb980107db0 0x7fb9801081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:37.185 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.452+0000 7fb97ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb980107db0 0x7fb9801081d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37806/0 (socket says 172.21.15.70:37806) 2024-06-04T04:45:37.185 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.452+0000 7fb97ffff700 1 -- 172.21.15.70:0/3831507127 learned_addr learned my addr 172.21.15.70:0/3831507127 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:37.185 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.452+0000 7fb97ffff700 1 -- 172.21.15.70:0/3831507127 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb980108fb0 con 0x7fb980107db0 2024-06-04T04:45:37.185 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.452+0000 7fb97ffff700 1 --2- 172.21.15.70:0/3831507127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb980107db0 0x7fb9801081d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb9740098a0 tx=0x7fb974009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a93f59ecd756bea8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:37.185 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.452+0000 7fb97effd700 1 -- 172.21.15.70:0/3831507127 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb974004160 con 0x7fb980107db0 2024-06-04T04:45:37.185 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.452+0000 7fb97effd700 1 -- 172.21.15.70:0/3831507127 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fb974004300 con 0x7fb980107db0 2024-06-04T04:45:37.185 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.452+0000 7fb97effd700 1 -- 172.21.15.70:0/3831507127 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9740119e0 con 0x7fb980107db0 2024-06-04T04:45:37.185 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.453+0000 7fb986268700 1 -- 172.21.15.70:0/3831507127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb980107db0 msgr2=0x7fb9801081d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:37.185 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.453+0000 7fb986268700 1 --2- 172.21.15.70:0/3831507127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb980107db0 0x7fb9801081d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb9740098a0 tx=0x7fb974009bb0 comp rx=0 tx=0).stop 2024-06-04T04:45:37.185 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.453+0000 7fb986268700 1 -- 172.21.15.70:0/3831507127 shutdown_connections 2024-06-04T04:45:37.185 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.453+0000 7fb986268700 1 --2- 172.21.15.70:0/3831507127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb980107db0 0x7fb9801081d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:37.185 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.453+0000 7fb986268700 1 -- 172.21.15.70:0/3831507127 >> 172.21.15.70:0/3831507127 conn(0x7fb980103370 msgr2=0x7fb980105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:37.185 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.453+0000 7fb986268700 1 -- 172.21.15.70:0/3831507127 shutdown_connections 2024-06-04T04:45:37.185 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.453+0000 7fb986268700 1 -- 172.21.15.70:0/3831507127 wait complete. 2024-06-04T04:45:37.186 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.454+0000 7fb986268700 1 Processor -- start 2024-06-04T04:45:37.186 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.454+0000 7fb986268700 1 -- start start 2024-06-04T04:45:37.186 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.454+0000 7fb986268700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb980107db0 0x7fb98019bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:37.186 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.454+0000 7fb986268700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb98010c1e0 con 0x7fb980107db0 2024-06-04T04:45:37.186 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.454+0000 7fb97ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb980107db0 0x7fb98019bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:37.186 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.454+0000 7fb97ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb980107db0 0x7fb98019bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37818/0 (socket says 172.21.15.70:37818) 2024-06-04T04:45:37.186 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.454+0000 7fb97ffff700 1 -- 172.21.15.70:0/2457920701 learned_addr learned my addr 172.21.15.70:0/2457920701 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:37.186 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.454+0000 7fb97ffff700 1 -- 172.21.15.70:0/2457920701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb98019c1e0 con 0x7fb980107db0 2024-06-04T04:45:37.186 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.455+0000 7fb97ffff700 1 --2- 172.21.15.70:0/2457920701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb980107db0 0x7fb98019bca0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb9740119b0 tx=0x7fb974011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:37.186 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.455+0000 7fb97d7fa700 1 -- 172.21.15.70:0/2457920701 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb974003b70 con 0x7fb980107db0 2024-06-04T04:45:37.186 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.455+0000 7fb97d7fa700 1 -- 172.21.15.70:0/2457920701 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fb97401a8a0 con 0x7fb980107db0 2024-06-04T04:45:37.186 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.455+0000 7fb986268700 1 -- 172.21.15.70:0/2457920701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb98019c470 con 0x7fb980107db0 2024-06-04T04:45:37.186 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.455+0000 7fb97d7fa700 1 -- 172.21.15.70:0/2457920701 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb974024830 con 0x7fb980107db0 2024-06-04T04:45:37.186 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.455+0000 7fb986268700 1 -- 172.21.15.70:0/2457920701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb98019f160 con 0x7fb980107db0 2024-06-04T04:45:37.187 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.455+0000 7fb97d7fa700 1 -- 172.21.15.70:0/2457920701 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb974022020 con 0x7fb980107db0 2024-06-04T04:45:37.187 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.455+0000 7fb97d7fa700 1 -- 172.21.15.70:0/2457920701 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb974019610 con 0x7fb980107db0 2024-06-04T04:45:37.187 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.456+0000 7fb986268700 1 -- 172.21.15.70:0/2457920701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb980064ba0 con 0x7fb980107db0 2024-06-04T04:45:37.187 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.458+0000 7fb97d7fa700 1 -- 172.21.15.70:0/2457920701 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fb97401aa50 con 0x7fb980107db0 2024-06-04T04:45:37.187 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.530+0000 7fb986268700 1 -- 172.21.15.70:0/2457920701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fb98019f5c0 con 0x7fb980107db0 2024-06-04T04:45:37.187 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.531+0000 7fb97d7fa700 1 -- 172.21.15.70:0/2457920701 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fb97401a430 con 0x7fb980107db0 2024-06-04T04:45:37.187 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.532+0000 7fb986268700 1 -- 172.21.15.70:0/2457920701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb980107db0 msgr2=0x7fb98019bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:37.187 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.532+0000 7fb986268700 1 --2- 172.21.15.70:0/2457920701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb980107db0 0x7fb98019bca0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb9740119b0 tx=0x7fb974011a90 comp rx=0 tx=0).stop 2024-06-04T04:45:37.187 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.532+0000 7fb986268700 1 -- 172.21.15.70:0/2457920701 shutdown_connections 2024-06-04T04:45:37.187 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.532+0000 7fb986268700 1 --2- 172.21.15.70:0/2457920701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb980107db0 0x7fb98019bca0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:37.187 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.532+0000 7fb986268700 1 -- 172.21.15.70:0/2457920701 >> 172.21.15.70:0/2457920701 conn(0x7fb980103370 msgr2=0x7fb980103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:37.187 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.532+0000 7fb986268700 1 -- 172.21.15.70:0/2457920701 shutdown_connections 2024-06-04T04:45:37.187 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:36.532+0000 7fb986268700 1 -- 172.21.15.70:0/2457920701 wait complete. 2024-06-04T04:45:37.187 INFO:teuthology.orchestra.run.smithi070.stdout:mgr not available, waiting (3/15)... 2024-06-04T04:45:37.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:37 smithi070 ceph-mon[32631]: Activating manager daemon x 2024-06-04T04:45:37.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:37 smithi070 ceph-mon[32631]: mgrmap e2: x(active, starting, since 0.0029293s) 2024-06-04T04:45:37.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:37 smithi070 ceph-mon[32631]: from='mgr.14100 172.21.15.70:0/2486080077' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-04T04:45:37.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:37 smithi070 ceph-mon[32631]: from='mgr.14100 172.21.15.70:0/2486080077' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-04T04:45:37.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:37 smithi070 ceph-mon[32631]: from='mgr.14100 172.21.15.70:0/2486080077' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-04T04:45:37.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:37 smithi070 ceph-mon[32631]: from='mgr.14100 172.21.15.70:0/2486080077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:45:37.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:37 smithi070 ceph-mon[32631]: from='mgr.14100 172.21.15.70:0/2486080077' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-04T04:45:37.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:37 smithi070 ceph-mon[32631]: Manager daemon x is now available 2024-06-04T04:45:37.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:37 smithi070 ceph-mon[32631]: from='mgr.14100 172.21.15.70:0/2486080077' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-04T04:45:37.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:37 smithi070 ceph-mon[32631]: from='mgr.14100 172.21.15.70:0/2486080077' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-04T04:45:37.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:37 smithi070 ceph-mon[32631]: from='mgr.14100 172.21.15.70:0/2486080077' entity='mgr.x' 2024-06-04T04:45:37.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:37 smithi070 ceph-mon[32631]: from='mgr.14100 172.21.15.70:0/2486080077' entity='mgr.x' 2024-06-04T04:45:37.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:37 smithi070 ceph-mon[32631]: from='mgr.14100 172.21.15.70:0/2486080077' entity='mgr.x' 2024-06-04T04:45:39.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:38 smithi070 ceph-mon[32631]: mgrmap e3: x(active, since 1.00762s) 2024-06-04T04:45:40.137 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:39 smithi070 ceph-mon[32631]: mgrmap e4: x(active, since 2s) 2024-06-04T04:45:40.137 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:39 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/966227611' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-04T04:45:40.475 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-06-04T04:45:40.475 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-06-04T04:45:40.475 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:45:40.475 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "health": { 2024-06-04T04:45:40.475 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-04T04:45:40.476 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-04T04:45:40.476 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-04T04:45:40.476 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:40.476 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-04T04:45:40.476 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-04T04:45:40.476 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 0 2024-06-04T04:45:40.476 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-06-04T04:45:40.476 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-04T04:45:40.476 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "a" 2024-06-04T04:45:40.476 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-06-04T04:45:40.476 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-06-04T04:45:40.476 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-04T04:45:40.476 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-04T04:45:40.476 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-06-04T04:45:40.476 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-04T04:45:40.476 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:40.477 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-04T04:45:40.477 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-04T04:45:40.477 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-04T04:45:40.477 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-04T04:45:40.477 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-04T04:45:40.486 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-04T04:45:40.486 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-04T04:45:40.486 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-04T04:45:40.486 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:40.486 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-04T04:45:40.486 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-04T04:45:40.486 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-04T04:45:40.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-04T04:45:40.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-04T04:45:40.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-04T04:45:40.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-04T04:45:40.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-04T04:45:40.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-04T04:45:40.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:40.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-04T04:45:40.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-04T04:45:40.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-04T04:45:40.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-04T04:45:40.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:40.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-04T04:45:40.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-04T04:45:40.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-04T04:45:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-04T04:45:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-04T04:45:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-04T04:45:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "restful" 2024-06-04T04:45:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-06-04T04:45:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-04T04:45:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-04T04:45:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-04T04:45:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modified": "2024-06-04T04:45:22.297998+0000", 2024-06-04T04:45:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-04T04:45:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-06-04T04:45:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-04T04:45:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-06-04T04:45:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.564+0000 7ff8afadb700 1 Processor -- start 2024-06-04T04:45:40.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.564+0000 7ff8afadb700 1 -- start start 2024-06-04T04:45:40.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.564+0000 7ff8afadb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8a8107770 0x7ff8a8107b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:40.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.564+0000 7ff8afadb700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a807ce10 con 0x7ff8a8107770 2024-06-04T04:45:40.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.564+0000 7ff8ad877700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8a8107770 0x7ff8a8107b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:40.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.564+0000 7ff8ad877700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8a8107770 0x7ff8a8107b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37906/0 (socket says 172.21.15.70:37906) 2024-06-04T04:45:40.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.564+0000 7ff8ad877700 1 -- 172.21.15.70:0/2639568426 learned_addr learned my addr 172.21.15.70:0/2639568426 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:40.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.564+0000 7ff8ad877700 1 -- 172.21.15.70:0/2639568426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a81080d0 con 0x7ff8a8107770 2024-06-04T04:45:40.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.565+0000 7ff8ad877700 1 --2- 172.21.15.70:0/2639568426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8a8107770 0x7ff8a8107b90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff8980098a0 tx=0x7ff898009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e1b996f738816 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:40.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.565+0000 7ff8ac875700 1 -- 172.21.15.70:0/2639568426 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff898004160 con 0x7ff8a8107770 2024-06-04T04:45:40.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.565+0000 7ff8ac875700 1 -- 172.21.15.70:0/2639568426 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7ff898004300 con 0x7ff8a8107770 2024-06-04T04:45:40.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.565+0000 7ff8ac875700 1 -- 172.21.15.70:0/2639568426 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff8980119e0 con 0x7ff8a8107770 2024-06-04T04:45:40.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.565+0000 7ff8afadb700 1 -- 172.21.15.70:0/2639568426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8a8107770 msgr2=0x7ff8a8107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:40.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.565+0000 7ff8afadb700 1 --2- 172.21.15.70:0/2639568426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8a8107770 0x7ff8a8107b90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff8980098a0 tx=0x7ff898009bb0 comp rx=0 tx=0).stop 2024-06-04T04:45:40.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.566+0000 7ff8afadb700 1 -- 172.21.15.70:0/2639568426 shutdown_connections 2024-06-04T04:45:40.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.566+0000 7ff8afadb700 1 --2- 172.21.15.70:0/2639568426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8a8107770 0x7ff8a8107b90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:40.490 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.566+0000 7ff8afadb700 1 -- 172.21.15.70:0/2639568426 >> 172.21.15.70:0/2639568426 conn(0x7ff8a81032f0 msgr2=0x7ff8a8105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:40.490 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.566+0000 7ff8afadb700 1 -- 172.21.15.70:0/2639568426 shutdown_connections 2024-06-04T04:45:40.490 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.566+0000 7ff8afadb700 1 -- 172.21.15.70:0/2639568426 wait complete. 2024-06-04T04:45:40.490 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.566+0000 7ff8afadb700 1 Processor -- start 2024-06-04T04:45:40.490 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.566+0000 7ff8afadb700 1 -- start start 2024-06-04T04:45:40.490 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.566+0000 7ff8afadb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8a8107770 0x7ff8a819eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:40.490 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.566+0000 7ff8afadb700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a81090a0 con 0x7ff8a8107770 2024-06-04T04:45:40.490 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.567+0000 7ff8ad877700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8a8107770 0x7ff8a819eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:40.490 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.567+0000 7ff8ad877700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8a8107770 0x7ff8a819eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37910/0 (socket says 172.21.15.70:37910) 2024-06-04T04:45:40.490 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.567+0000 7ff8ad877700 1 -- 172.21.15.70:0/966227611 learned_addr learned my addr 172.21.15.70:0/966227611 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:40.490 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.567+0000 7ff8ad877700 1 -- 172.21.15.70:0/966227611 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a819f400 con 0x7ff8a8107770 2024-06-04T04:45:40.490 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.567+0000 7ff8ad877700 1 --2- 172.21.15.70:0/966227611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8a8107770 0x7ff8a819eec0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff8980119b0 tx=0x7ff898011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:40.490 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.567+0000 7ff89effd700 1 -- 172.21.15.70:0/966227611 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff8980039f0 con 0x7ff8a8107770 2024-06-04T04:45:40.491 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.567+0000 7ff89effd700 1 -- 172.21.15.70:0/966227611 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7ff898003b90 con 0x7ff8a8107770 2024-06-04T04:45:40.491 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.567+0000 7ff89effd700 1 -- 172.21.15.70:0/966227611 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff89801a620 con 0x7ff8a8107770 2024-06-04T04:45:40.491 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.567+0000 7ff8afadb700 1 -- 172.21.15.70:0/966227611 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8a819f690 con 0x7ff8a8107770 2024-06-04T04:45:40.491 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.567+0000 7ff8afadb700 1 -- 172.21.15.70:0/966227611 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8a81a2380 con 0x7ff8a8107770 2024-06-04T04:45:40.491 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.568+0000 7ff89effd700 1 -- 172.21.15.70:0/966227611 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7ff898024a50 con 0x7ff8a8107770 2024-06-04T04:45:40.491 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.568+0000 7ff8afadb700 1 -- 172.21.15.70:0/966227611 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8a8064ba0 con 0x7ff8a8107770 2024-06-04T04:45:40.491 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.568+0000 7ff89effd700 1 --2- 172.21.15.70:0/966227611 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7ff89403e420 0x7ff8940408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:40.491 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.568+0000 7ff89effd700 1 -- 172.21.15.70:0/966227611 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff898053b40 con 0x7ff8a8107770 2024-06-04T04:45:40.491 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.569+0000 7ff8ad076700 1 --2- 172.21.15.70:0/966227611 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7ff89403e420 0x7ff8940408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:40.491 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.569+0000 7ff8ad076700 1 --2- 172.21.15.70:0/966227611 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7ff89403e420 0x7ff8940408e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4006fd0 tx=0x7ff8a4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:40.491 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.574+0000 7ff89effd700 1 -- 172.21.15.70:0/966227611 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff898019c70 con 0x7ff8a8107770 2024-06-04T04:45:40.491 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.858+0000 7ff8afadb700 1 -- 172.21.15.70:0/966227611 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff8a802e6e0 con 0x7ff8a8107770 2024-06-04T04:45:40.491 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.859+0000 7ff89effd700 1 -- 172.21.15.70:0/966227611 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff898053560 con 0x7ff8a8107770 2024-06-04T04:45:40.491 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.861+0000 7ff8afadb700 1 -- 172.21.15.70:0/966227611 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7ff89403e420 msgr2=0x7ff8940408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:40.492 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.861+0000 7ff8afadb700 1 --2- 172.21.15.70:0/966227611 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7ff89403e420 0x7ff8940408e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4006fd0 tx=0x7ff8a4006e40 comp rx=0 tx=0).stop 2024-06-04T04:45:40.492 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.861+0000 7ff8afadb700 1 -- 172.21.15.70:0/966227611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8a8107770 msgr2=0x7ff8a819eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:40.492 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.861+0000 7ff8afadb700 1 --2- 172.21.15.70:0/966227611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8a8107770 0x7ff8a819eec0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff8980119b0 tx=0x7ff898011a90 comp rx=0 tx=0).stop 2024-06-04T04:45:40.492 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.861+0000 7ff8afadb700 1 -- 172.21.15.70:0/966227611 shutdown_connections 2024-06-04T04:45:40.492 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.861+0000 7ff8afadb700 1 --2- 172.21.15.70:0/966227611 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7ff89403e420 0x7ff8940408e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:40.492 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.861+0000 7ff8afadb700 1 --2- 172.21.15.70:0/966227611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8a8107770 0x7ff8a819eec0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:40.492 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.861+0000 7ff8afadb700 1 -- 172.21.15.70:0/966227611 >> 172.21.15.70:0/966227611 conn(0x7ff8a81032f0 msgr2=0x7ff8a8195b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:40.492 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.861+0000 7ff8afadb700 1 -- 172.21.15.70:0/966227611 shutdown_connections 2024-06-04T04:45:40.492 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:39.861+0000 7ff8afadb700 1 -- 172.21.15.70:0/966227611 wait complete. 2024-06-04T04:45:40.492 INFO:teuthology.orchestra.run.smithi070.stdout:mgr is available 2024-06-04T04:45:41.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:41 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/391059848' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-04T04:45:41.685 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-06-04T04:45:41.686 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [global] 2024-06-04T04:45:41.686 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout fsid = 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:45:41.686 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-06-04T04:45:41.686 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-04T04:45:41.686 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-04T04:45:41.686 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-04T04:45:41.686 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-04T04:45:41.686 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-06-04T04:45:41.686 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-04T04:45:41.687 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-04T04:45:41.687 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-06-04T04:45:41.687 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [osd] 2024-06-04T04:45:41.687 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-04T04:45:41.687 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-04T04:45:41.687 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-04T04:45:41.687 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-04T04:45:41.687 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-04T04:45:41.687 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.893+0000 7fe0c93be700 1 Processor -- start 2024-06-04T04:45:41.687 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.893+0000 7fe0c93be700 1 -- start start 2024-06-04T04:45:41.687 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.894+0000 7fe0c93be700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0c4109420 0x7fe0c4109840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:41.687 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.894+0000 7fe0c93be700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c4109e10 con 0x7fe0c4109420 2024-06-04T04:45:41.687 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.894+0000 7fe0c2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0c4109420 0x7fe0c4109840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:41.688 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.894+0000 7fe0c2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0c4109420 0x7fe0c4109840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37924/0 (socket says 172.21.15.70:37924) 2024-06-04T04:45:41.688 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.894+0000 7fe0c2ffd700 1 -- 172.21.15.70:0/246577802 learned_addr learned my addr 172.21.15.70:0/246577802 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:41.688 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.894+0000 7fe0c2ffd700 1 -- 172.21.15.70:0/246577802 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c4107ff0 con 0x7fe0c4109420 2024-06-04T04:45:41.688 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.894+0000 7fe0c2ffd700 1 --2- 172.21.15.70:0/246577802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0c4109420 0x7fe0c4109840 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4009e80 tx=0x7fe0b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eecfe9686c8db8ce server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:41.688 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.894+0000 7fe0c1ffb700 1 -- 172.21.15.70:0/246577802 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0b4017070 con 0x7fe0c4109420 2024-06-04T04:45:41.688 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.894+0000 7fe0c1ffb700 1 -- 172.21.15.70:0/246577802 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fe0b400b8c0 con 0x7fe0c4109420 2024-06-04T04:45:41.688 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.894+0000 7fe0c1ffb700 1 -- 172.21.15.70:0/246577802 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0b4004420 con 0x7fe0c4109420 2024-06-04T04:45:41.688 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.895+0000 7fe0c93be700 1 -- 172.21.15.70:0/246577802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0c4109420 msgr2=0x7fe0c4109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:41.688 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.895+0000 7fe0c93be700 1 --2- 172.21.15.70:0/246577802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0c4109420 0x7fe0c4109840 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4009e80 tx=0x7fe0b400b330 comp rx=0 tx=0).stop 2024-06-04T04:45:41.688 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.895+0000 7fe0c93be700 1 -- 172.21.15.70:0/246577802 shutdown_connections 2024-06-04T04:45:41.688 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.895+0000 7fe0c93be700 1 --2- 172.21.15.70:0/246577802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0c4109420 0x7fe0c4109840 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:41.688 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.895+0000 7fe0c93be700 1 -- 172.21.15.70:0/246577802 >> 172.21.15.70:0/246577802 conn(0x7fe0c4102b20 msgr2=0x7fe0c4104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:41.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.895+0000 7fe0c93be700 1 -- 172.21.15.70:0/246577802 shutdown_connections 2024-06-04T04:45:41.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.895+0000 7fe0c93be700 1 -- 172.21.15.70:0/246577802 wait complete. 2024-06-04T04:45:41.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.896+0000 7fe0c93be700 1 Processor -- start 2024-06-04T04:45:41.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.896+0000 7fe0c93be700 1 -- start start 2024-06-04T04:45:41.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.896+0000 7fe0c93be700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0c4109420 0x7fe0c4107960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:41.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.896+0000 7fe0c93be700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c410c230 con 0x7fe0c4109420 2024-06-04T04:45:41.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.896+0000 7fe0c2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0c4109420 0x7fe0c4107960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:41.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.896+0000 7fe0c2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0c4109420 0x7fe0c4107960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37938/0 (socket says 172.21.15.70:37938) 2024-06-04T04:45:41.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.896+0000 7fe0c2ffd700 1 -- 172.21.15.70:0/391059848 learned_addr learned my addr 172.21.15.70:0/391059848 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:41.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.896+0000 7fe0c2ffd700 1 -- 172.21.15.70:0/391059848 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c419fab0 con 0x7fe0c4109420 2024-06-04T04:45:41.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.896+0000 7fe0c2ffd700 1 --2- 172.21.15.70:0/391059848 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0c4109420 0x7fe0c4107960 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4009e50 tx=0x7fe0b400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:41.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.897+0000 7fe0bbfff700 1 -- 172.21.15.70:0/391059848 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0b4020020 con 0x7fe0c4109420 2024-06-04T04:45:41.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.897+0000 7fe0c93be700 1 -- 172.21.15.70:0/391059848 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0c419fc80 con 0x7fe0c4109420 2024-06-04T04:45:41.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.897+0000 7fe0bbfff700 1 -- 172.21.15.70:0/391059848 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fe0b4012430 con 0x7fe0c4109420 2024-06-04T04:45:41.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.897+0000 7fe0bbfff700 1 -- 172.21.15.70:0/391059848 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0b4017040 con 0x7fe0c4109420 2024-06-04T04:45:41.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.897+0000 7fe0c93be700 1 -- 172.21.15.70:0/391059848 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0c41a00a0 con 0x7fe0c4109420 2024-06-04T04:45:41.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.898+0000 7fe0bbfff700 1 -- 172.21.15.70:0/391059848 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7fe0b4023020 con 0x7fe0c4109420 2024-06-04T04:45:41.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.898+0000 7fe0c93be700 1 -- 172.21.15.70:0/391059848 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0c4064ba0 con 0x7fe0c4109420 2024-06-04T04:45:41.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.898+0000 7fe0bbfff700 1 --2- 172.21.15.70:0/391059848 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fe0b003e420 0x7fe0b00408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:41.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.898+0000 7fe0bbfff700 1 -- 172.21.15.70:0/391059848 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe0b400f050 con 0x7fe0c4109420 2024-06-04T04:45:41.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.898+0000 7fe0c27fc700 1 --2- 172.21.15.70:0/391059848 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fe0b003e420 0x7fe0b00408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:41.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.899+0000 7fe0c27fc700 1 --2- 172.21.15.70:0/391059848 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fe0b003e420 0x7fe0b00408e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac006fd0 tx=0x7fe0ac006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:41.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:40.903+0000 7fe0bbfff700 1 -- 172.21.15.70:0/391059848 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe0b402a020 con 0x7fe0c4109420 2024-06-04T04:45:41.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:41.102+0000 7fe0c93be700 1 -- 172.21.15.70:0/391059848 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fe0c41a0270 con 0x7fe0c4109420 2024-06-04T04:45:41.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:41.103+0000 7fe0bbfff700 1 -- 172.21.15.70:0/391059848 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7fe0b400a040 con 0x7fe0c4109420 2024-06-04T04:45:41.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:41.106+0000 7fe0c93be700 1 -- 172.21.15.70:0/391059848 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fe0b003e420 msgr2=0x7fe0b00408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:41.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:41.106+0000 7fe0c93be700 1 --2- 172.21.15.70:0/391059848 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fe0b003e420 0x7fe0b00408e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac006fd0 tx=0x7fe0ac006e40 comp rx=0 tx=0).stop 2024-06-04T04:45:41.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:41.106+0000 7fe0c93be700 1 -- 172.21.15.70:0/391059848 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0c4109420 msgr2=0x7fe0c4107960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:41.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:41.106+0000 7fe0c93be700 1 --2- 172.21.15.70:0/391059848 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0c4109420 0x7fe0c4107960 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4009e50 tx=0x7fe0b400bd50 comp rx=0 tx=0).stop 2024-06-04T04:45:41.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:41.106+0000 7fe0c93be700 1 -- 172.21.15.70:0/391059848 shutdown_connections 2024-06-04T04:45:41.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:41.106+0000 7fe0c93be700 1 --2- 172.21.15.70:0/391059848 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fe0b003e420 0x7fe0b00408e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:41.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:41.106+0000 7fe0c93be700 1 --2- 172.21.15.70:0/391059848 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0c4109420 0x7fe0c4107960 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:41.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:41.106+0000 7fe0c93be700 1 -- 172.21.15.70:0/391059848 >> 172.21.15.70:0/391059848 conn(0x7fe0c4102b20 msgr2=0x7fe0c4103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:41.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:41.106+0000 7fe0c93be700 1 -- 172.21.15.70:0/391059848 shutdown_connections 2024-06-04T04:45:41.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:41.106+0000 7fe0c93be700 1 -- 172.21.15.70:0/391059848 wait complete. 2024-06-04T04:45:41.691 INFO:teuthology.orchestra.run.smithi070.stdout:Enabling cephadm module... 2024-06-04T04:45:42.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:42 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1741323205' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-06-04T04:45:42.609 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:42 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: ignoring --setuser ceph since I am not root 2024-06-04T04:45:42.609 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:42 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: ignoring --setgroup ceph since I am not root 2024-06-04T04:45:42.609 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:42 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:42.577+0000 7ff1be2e8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-04T04:45:43.050 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:42 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:42.766+0000 7ff1be2e8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-04T04:45:43.050 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:42 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:42.868+0000 7ff1be2e8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-04T04:45:43.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.085+0000 7f767aa22700 1 Processor -- start 2024-06-04T04:45:43.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.085+0000 7f767aa22700 1 -- start start 2024-06-04T04:45:43.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.086+0000 7f767aa22700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76741098e0 0x7f767410bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:43.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.086+0000 7f767aa22700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f767407ce70 con 0x7f76741098e0 2024-06-04T04:45:43.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.086+0000 7f7673fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76741098e0 0x7f767410bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:43.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.086+0000 7f7673fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76741098e0 0x7f767410bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37952/0 (socket says 172.21.15.70:37952) 2024-06-04T04:45:43.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.086+0000 7f7673fff700 1 -- 172.21.15.70:0/401138644 learned_addr learned my addr 172.21.15.70:0/401138644 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:43.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.086+0000 7f7673fff700 1 -- 172.21.15.70:0/401138644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f767410c210 con 0x7f76741098e0 2024-06-04T04:45:43.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.086+0000 7f7673fff700 1 --2- 172.21.15.70:0/401138644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76741098e0 0x7f767410bcd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f765c0098a0 tx=0x7f765c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b9951548ef06ff2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:43.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.086+0000 7f7672ffd700 1 -- 172.21.15.70:0/401138644 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f765c004160 con 0x7f76741098e0 2024-06-04T04:45:43.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.086+0000 7f7672ffd700 1 -- 172.21.15.70:0/401138644 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f765c004300 con 0x7f76741098e0 2024-06-04T04:45:43.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.086+0000 7f7672ffd700 1 -- 172.21.15.70:0/401138644 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f765c0119e0 con 0x7f76741098e0 2024-06-04T04:45:43.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.087+0000 7f767aa22700 1 -- 172.21.15.70:0/401138644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76741098e0 msgr2=0x7f767410bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:43.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.087+0000 7f767aa22700 1 --2- 172.21.15.70:0/401138644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76741098e0 0x7f767410bcd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f765c0098a0 tx=0x7f765c009bb0 comp rx=0 tx=0).stop 2024-06-04T04:45:43.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.087+0000 7f767aa22700 1 -- 172.21.15.70:0/401138644 shutdown_connections 2024-06-04T04:45:43.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.087+0000 7f767aa22700 1 --2- 172.21.15.70:0/401138644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76741098e0 0x7f767410bcd0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:43.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.087+0000 7f767aa22700 1 -- 172.21.15.70:0/401138644 >> 172.21.15.70:0/401138644 conn(0x7f7674103330 msgr2=0x7f7674105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:43.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.087+0000 7f767aa22700 1 -- 172.21.15.70:0/401138644 shutdown_connections 2024-06-04T04:45:43.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.087+0000 7f767aa22700 1 -- 172.21.15.70:0/401138644 wait complete. 2024-06-04T04:45:43.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.088+0000 7f767aa22700 1 Processor -- start 2024-06-04T04:45:43.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.088+0000 7f767aa22700 1 -- start start 2024-06-04T04:45:43.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.088+0000 7f767aa22700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76741098e0 0x7f767419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:43.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.088+0000 7f767aa22700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f767410d100 con 0x7f76741098e0 2024-06-04T04:45:43.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.088+0000 7f7673fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76741098e0 0x7f767419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:43.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.088+0000 7f7673fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76741098e0 0x7f767419cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37964/0 (socket says 172.21.15.70:37964) 2024-06-04T04:45:43.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.088+0000 7f7673fff700 1 -- 172.21.15.70:0/1741323205 learned_addr learned my addr 172.21.15.70:0/1741323205 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:43.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.088+0000 7f7673fff700 1 -- 172.21.15.70:0/1741323205 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f767419d220 con 0x7f76741098e0 2024-06-04T04:45:43.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.089+0000 7f7673fff700 1 --2- 172.21.15.70:0/1741323205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76741098e0 0x7f767419cce0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f765c0116a0 tx=0x7f765c0046a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:43.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.089+0000 7f76717fa700 1 -- 172.21.15.70:0/1741323205 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f765c003b70 con 0x7f76741098e0 2024-06-04T04:45:43.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.089+0000 7f76717fa700 1 -- 172.21.15.70:0/1741323205 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f765c01a8a0 con 0x7f76741098e0 2024-06-04T04:45:43.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.089+0000 7f767aa22700 1 -- 172.21.15.70:0/1741323205 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f767419d510 con 0x7f76741098e0 2024-06-04T04:45:43.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.089+0000 7f76717fa700 1 -- 172.21.15.70:0/1741323205 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f765c024890 con 0x7f76741098e0 2024-06-04T04:45:43.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.089+0000 7f767aa22700 1 -- 172.21.15.70:0/1741323205 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76741a0190 con 0x7f76741098e0 2024-06-04T04:45:43.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.090+0000 7f767aa22700 1 -- 172.21.15.70:0/1741323205 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7674064ba0 con 0x7f76741098e0 2024-06-04T04:45:43.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.090+0000 7f76717fa700 1 -- 172.21.15.70:0/1741323205 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f765c022020 con 0x7f76741098e0 2024-06-04T04:45:43.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.090+0000 7f76717fa700 1 --2- 172.21.15.70:0/1741323205 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7f766003e420 0x7f76600408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:43.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.090+0000 7f76717fa700 1 -- 172.21.15.70:0/1741323205 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f765c053950 con 0x7f76741098e0 2024-06-04T04:45:43.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.090+0000 7f76737fe700 1 --2- 172.21.15.70:0/1741323205 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7f766003e420 0x7f76600408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:43.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.090+0000 7f76737fe700 1 --2- 172.21.15.70:0/1741323205 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7f766003e420 0x7f76600408e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7664006fd0 tx=0x7f7664006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:43.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.095+0000 7f76717fa700 1 -- 172.21.15.70:0/1741323205 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f765c0536b0 con 0x7f76741098e0 2024-06-04T04:45:43.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.352+0000 7f767aa22700 1 -- 172.21.15.70:0/1741323205 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f76741a07c0 con 0x7f76741098e0 2024-06-04T04:45:43.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.404+0000 7f76717fa700 1 -- 172.21.15.70:0/1741323205 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f765c021d80 con 0x7f76741098e0 2024-06-04T04:45:43.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.405+0000 7f76717fa700 1 -- 172.21.15.70:0/1741323205 <== mon.0 v2:172.21.15.70:3300/0 8 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f765c018350 con 0x7f76741098e0 2024-06-04T04:45:43.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.407+0000 7f767aa22700 1 -- 172.21.15.70:0/1741323205 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7f766003e420 msgr2=0x7f76600408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:43.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.407+0000 7f767aa22700 1 --2- 172.21.15.70:0/1741323205 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7f766003e420 0x7f76600408e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7664006fd0 tx=0x7f7664006e40 comp rx=0 tx=0).stop 2024-06-04T04:45:43.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.407+0000 7f767aa22700 1 -- 172.21.15.70:0/1741323205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76741098e0 msgr2=0x7f767419cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:43.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.407+0000 7f767aa22700 1 --2- 172.21.15.70:0/1741323205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76741098e0 0x7f767419cce0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f765c0116a0 tx=0x7f765c0046a0 comp rx=0 tx=0).stop 2024-06-04T04:45:43.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.407+0000 7f767aa22700 1 -- 172.21.15.70:0/1741323205 shutdown_connections 2024-06-04T04:45:43.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.407+0000 7f767aa22700 1 --2- 172.21.15.70:0/1741323205 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7f766003e420 0x7f76600408e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:43.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.407+0000 7f767aa22700 1 --2- 172.21.15.70:0/1741323205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76741098e0 0x7f767419cce0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:43.063 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.407+0000 7f767aa22700 1 -- 172.21.15.70:0/1741323205 >> 172.21.15.70:0/1741323205 conn(0x7f7674103330 msgr2=0x7f7674103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:43.063 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.407+0000 7f767aa22700 1 -- 172.21.15.70:0/1741323205 shutdown_connections 2024-06-04T04:45:43.063 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:42.407+0000 7f767aa22700 1 -- 172.21.15.70:0/1741323205 wait complete. 2024-06-04T04:45:43.327 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:43 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:43.190+0000 7ff1be2e8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-04T04:45:43.327 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:43 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:43.279+0000 7ff1be2e8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-04T04:45:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:43 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1741323205' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-06-04T04:45:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:43 smithi070 ceph-mon[32631]: mgrmap e5: x(active, since 5s) 2024-06-04T04:45:44.313 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-06-04T04:45:44.313 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-06-04T04:45:44.313 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-04T04:45:44.313 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-06-04T04:45:44.313 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-04T04:45:44.313 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-06-04T04:45:44.313 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.423+0000 7fd6dec51700 1 Processor -- start 2024-06-04T04:45:44.313 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.423+0000 7fd6dec51700 1 -- start start 2024-06-04T04:45:44.314 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.423+0000 7fd6dec51700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6d8109340 0x7fd6d8109760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:44.314 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.423+0000 7fd6dec51700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d8109d30 con 0x7fd6d8109340 2024-06-04T04:45:44.314 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.423+0000 7fd6dc9ed700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6d8109340 0x7fd6d8109760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:44.314 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.423+0000 7fd6dc9ed700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6d8109340 0x7fd6d8109760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37992/0 (socket says 172.21.15.70:37992) 2024-06-04T04:45:44.314 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.423+0000 7fd6dc9ed700 1 -- 172.21.15.70:0/1153972677 learned_addr learned my addr 172.21.15.70:0/1153972677 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:44.314 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.424+0000 7fd6dc9ed700 1 -- 172.21.15.70:0/1153972677 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d8107f50 con 0x7fd6d8109340 2024-06-04T04:45:44.314 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.424+0000 7fd6dc9ed700 1 --2- 172.21.15.70:0/1153972677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6d8109340 0x7fd6d8109760 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0098a0 tx=0x7fd6cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=568bb091ffbd1f93 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:44.314 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.424+0000 7fd6d77fe700 1 -- 172.21.15.70:0/1153972677 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6cc004160 con 0x7fd6d8109340 2024-06-04T04:45:44.314 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.424+0000 7fd6d77fe700 1 -- 172.21.15.70:0/1153972677 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fd6cc004300 con 0x7fd6d8109340 2024-06-04T04:45:44.314 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.424+0000 7fd6d77fe700 1 -- 172.21.15.70:0/1153972677 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6cc0119e0 con 0x7fd6d8109340 2024-06-04T04:45:44.314 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.425+0000 7fd6dec51700 1 -- 172.21.15.70:0/1153972677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6d8109340 msgr2=0x7fd6d8109760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:44.314 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.425+0000 7fd6dec51700 1 --2- 172.21.15.70:0/1153972677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6d8109340 0x7fd6d8109760 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0098a0 tx=0x7fd6cc009bb0 comp rx=0 tx=0).stop 2024-06-04T04:45:44.315 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.425+0000 7fd6dec51700 1 -- 172.21.15.70:0/1153972677 shutdown_connections 2024-06-04T04:45:44.315 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.425+0000 7fd6dec51700 1 --2- 172.21.15.70:0/1153972677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6d8109340 0x7fd6d8109760 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:44.315 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.425+0000 7fd6dec51700 1 -- 172.21.15.70:0/1153972677 >> 172.21.15.70:0/1153972677 conn(0x7fd6d8102ae0 msgr2=0x7fd6d8104f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:44.315 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.425+0000 7fd6dec51700 1 -- 172.21.15.70:0/1153972677 shutdown_connections 2024-06-04T04:45:44.315 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.425+0000 7fd6dec51700 1 -- 172.21.15.70:0/1153972677 wait complete. 2024-06-04T04:45:44.315 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.425+0000 7fd6dec51700 1 Processor -- start 2024-06-04T04:45:44.315 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.425+0000 7fd6dec51700 1 -- start start 2024-06-04T04:45:44.315 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.426+0000 7fd6dec51700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6d8109340 0x7fd6d8107920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:44.315 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.426+0000 7fd6dec51700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d810c190 con 0x7fd6d8109340 2024-06-04T04:45:44.315 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.426+0000 7fd6dc9ed700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6d8109340 0x7fd6d8107920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:44.315 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.426+0000 7fd6dc9ed700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6d8109340 0x7fd6d8107920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38008/0 (socket says 172.21.15.70:38008) 2024-06-04T04:45:44.315 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.426+0000 7fd6dc9ed700 1 -- 172.21.15.70:0/4075149519 learned_addr learned my addr 172.21.15.70:0/4075149519 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:44.315 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.426+0000 7fd6dc9ed700 1 -- 172.21.15.70:0/4075149519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d819fd90 con 0x7fd6d8109340 2024-06-04T04:45:44.315 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.426+0000 7fd6dc9ed700 1 --2- 172.21.15.70:0/4075149519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6d8109340 0x7fd6d8107920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc006b20 tx=0x7fd6cc011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:44.315 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.426+0000 7fd6d5ffb700 1 -- 172.21.15.70:0/4075149519 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6cc003a60 con 0x7fd6d8109340 2024-06-04T04:45:44.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.426+0000 7fd6d5ffb700 1 -- 172.21.15.70:0/4075149519 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fd6cc003c00 con 0x7fd6d8109340 2024-06-04T04:45:44.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.426+0000 7fd6dec51700 1 -- 172.21.15.70:0/4075149519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d819ff60 con 0x7fd6d8109340 2024-06-04T04:45:44.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.426+0000 7fd6d5ffb700 1 -- 172.21.15.70:0/4075149519 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6cc024780 con 0x7fd6d8109340 2024-06-04T04:45:44.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.427+0000 7fd6dec51700 1 -- 172.21.15.70:0/4075149519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d81a0380 con 0x7fd6d8109340 2024-06-04T04:45:44.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.427+0000 7fd6dec51700 1 -- 172.21.15.70:0/4075149519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6d8064ba0 con 0x7fd6d8109340 2024-06-04T04:45:44.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.427+0000 7fd6d5ffb700 1 -- 172.21.15.70:0/4075149519 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7fd6cc022020 con 0x7fd6d8109340 2024-06-04T04:45:44.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.428+0000 7fd6d5ffb700 1 --2- 172.21.15.70:0/4075149519 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fd6c803e470 0x7fd6c8040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:44.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.428+0000 7fd6d5ffb700 1 -- 172.21.15.70:0/4075149519 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd6cc053790 con 0x7fd6d8109340 2024-06-04T04:45:44.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.428+0000 7fd6d7fff700 1 -- 172.21.15.70:0/4075149519 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fd6c803e470 msgr2=0x7fd6c8040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2087909610 2024-06-04T04:45:44.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.428+0000 7fd6d7fff700 1 --2- 172.21.15.70:0/4075149519 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fd6c803e470 0x7fd6c8040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-04T04:45:44.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.433+0000 7fd6d5ffb700 1 -- 172.21.15.70:0/4075149519 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd6cc00e260 con 0x7fd6d8109340 2024-06-04T04:45:44.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.629+0000 7fd6d7fff700 1 -- 172.21.15.70:0/4075149519 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fd6c803e470 msgr2=0x7fd6c8040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2087909610 2024-06-04T04:45:44.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.629+0000 7fd6d7fff700 1 --2- 172.21.15.70:0/4075149519 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fd6c803e470 0x7fd6c8040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-04T04:45:44.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.730+0000 7fd6dec51700 1 -- 172.21.15.70:0/4075149519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fd6d802e560 con 0x7fd6d8109340 2024-06-04T04:45:44.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.731+0000 7fd6d5ffb700 1 -- 172.21.15.70:0/4075149519 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fd6cc053470 con 0x7fd6d8109340 2024-06-04T04:45:44.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.733+0000 7fd6dec51700 1 -- 172.21.15.70:0/4075149519 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fd6c803e470 msgr2=0x7fd6c8040930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-04T04:45:44.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.733+0000 7fd6dec51700 1 --2- 172.21.15.70:0/4075149519 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fd6c803e470 0x7fd6c8040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:44.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.733+0000 7fd6dec51700 1 -- 172.21.15.70:0/4075149519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6d8109340 msgr2=0x7fd6d8107920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:44.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.733+0000 7fd6dec51700 1 --2- 172.21.15.70:0/4075149519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6d8109340 0x7fd6d8107920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc006b20 tx=0x7fd6cc011a60 comp rx=0 tx=0).stop 2024-06-04T04:45:44.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.733+0000 7fd6dec51700 1 -- 172.21.15.70:0/4075149519 shutdown_connections 2024-06-04T04:45:44.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.733+0000 7fd6dec51700 1 --2- 172.21.15.70:0/4075149519 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fd6c803e470 0x7fd6c8040930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:44.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.733+0000 7fd6dec51700 1 --2- 172.21.15.70:0/4075149519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6d8109340 0x7fd6d8107920 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:44.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.733+0000 7fd6dec51700 1 -- 172.21.15.70:0/4075149519 >> 172.21.15.70:0/4075149519 conn(0x7fd6d8102ae0 msgr2=0x7fd6d8103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:44.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.733+0000 7fd6dec51700 1 -- 172.21.15.70:0/4075149519 shutdown_connections 2024-06-04T04:45:44.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:43.734+0000 7fd6dec51700 1 -- 172.21.15.70:0/4075149519 wait complete. 2024-06-04T04:45:44.317 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for the mgr to restart... 2024-06-04T04:45:44.317 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mgr epoch 5... 2024-06-04T04:45:44.559 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:44 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/4075149519' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-04T04:45:44.560 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:44 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:44.515+0000 7ff1be2e8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-04T04:45:44.859 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:44 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:44.691+0000 7ff1be2e8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-04T04:45:45.159 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:44 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:44.866+0000 7ff1be2e8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-04T04:45:45.159 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:45 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:45.156+0000 7ff1be2e8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-04T04:45:45.455 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:45 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:45.250+0000 7ff1be2e8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-04T04:45:45.455 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:45 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:45.452+0000 7ff1be2e8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-04T04:45:45.859 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:45 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:45.644+0000 7ff1be2e8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-04T04:45:46.470 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:46 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:46.246+0000 7ff1be2e8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-04T04:45:46.470 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:46 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:46.467+0000 7ff1be2e8200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-06-04T04:45:47.356 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:46 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:46.891+0000 7ff1be2e8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-04T04:45:47.356 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:46 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:46.985+0000 7ff1be2e8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-04T04:45:47.357 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:47 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:47.354+0000 7ff1be2e8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-04T04:45:48.041 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:47 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:47.707+0000 7ff1be2e8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-04T04:45:48.041 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:47 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:47.800+0000 7ff1be2e8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-04T04:45:48.359 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:48 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:48.039+0000 7ff1be2e8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-04T04:45:48.359 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:48 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:48.173+0000 7ff1be2e8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-04T04:45:48.359 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:48 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:48.271+0000 7ff1be2e8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-04T04:45:48.859 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:48 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:48.593+0000 7ff1be2e8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-04T04:45:48.859 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:48 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:48.685+0000 7ff1be2e8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-04T04:45:49.554 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:49 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:49.283+0000 7ff1be2e8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-04T04:45:49.554 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:49 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:45:49.499+0000 7ff1be2e8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-04T04:45:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:49 smithi070 ceph-mon[32631]: Active manager daemon x restarted 2024-06-04T04:45:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:49 smithi070 ceph-mon[32631]: Activating manager daemon x 2024-06-04T04:45:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:49 smithi070 ceph-mon[32631]: osdmap e2: 0 total, 0 up, 0 in 2024-06-04T04:45:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:49 smithi070 ceph-mon[32631]: mgrmap e6: x(active, starting, since 0.0034946s) 2024-06-04T04:45:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:49 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:45:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:49 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-04T04:45:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:49 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-04T04:45:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:49 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-04T04:45:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:49 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-04T04:45:49.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:49 smithi070 ceph-mon[32631]: Manager daemon x is now available 2024-06-04T04:45:49.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:49 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' 2024-06-04T04:45:49.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:49 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' 2024-06-04T04:45:49.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:49 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:45:49.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:49 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:45:49.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:49 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-04T04:45:50.818 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:50 smithi070 ceph-mon[32631]: Found migration_current of "None". Setting to last migration. 2024-06-04T04:45:50.819 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:50 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-04T04:45:50.819 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:50 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' 2024-06-04T04:45:50.819 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:50 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' 2024-06-04T04:45:50.819 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:50 smithi070 ceph-mon[32631]: [04/Jun/2024:04:45:50] ENGINE Bus STARTING 2024-06-04T04:45:50.819 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:50 smithi070 ceph-mon[32631]: mgrmap e7: x(active, since 1.00549s) 2024-06-04T04:45:51.087 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-06-04T04:45:51.087 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-06-04T04:45:51.087 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-04T04:45:51.087 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-06-04T04:45:51.087 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.759+0000 7fc31ea98700 1 Processor -- start 2024-06-04T04:45:51.087 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.759+0000 7fc31ea98700 1 -- start start 2024-06-04T04:45:51.088 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.759+0000 7fc31ea98700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc318107150 0x7fc318107570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:51.088 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.759+0000 7fc31ea98700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc318107b40 con 0x7fc318107150 2024-06-04T04:45:51.088 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.759+0000 7fc31c834700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc318107150 0x7fc318107570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:51.088 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.759+0000 7fc31c834700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc318107150 0x7fc318107570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38010/0 (socket says 172.21.15.70:38010) 2024-06-04T04:45:51.088 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.759+0000 7fc31c834700 1 -- 172.21.15.70:0/626613232 learned_addr learned my addr 172.21.15.70:0/626613232 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:51.088 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.760+0000 7fc31c834700 1 -- 172.21.15.70:0/626613232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31810a0e0 con 0x7fc318107150 2024-06-04T04:45:51.088 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.760+0000 7fc31c834700 1 --2- 172.21.15.70:0/626613232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc318107150 0x7fc318107570 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0098a0 tx=0x7fc30c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b2e942bb17820a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:51.088 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.760+0000 7fc3177fe700 1 -- 172.21.15.70:0/626613232 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc30c004160 con 0x7fc318107150 2024-06-04T04:45:51.088 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.760+0000 7fc3177fe700 1 -- 172.21.15.70:0/626613232 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fc30c004300 con 0x7fc318107150 2024-06-04T04:45:51.088 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.760+0000 7fc3177fe700 1 -- 172.21.15.70:0/626613232 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc30c0119e0 con 0x7fc318107150 2024-06-04T04:45:51.088 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.761+0000 7fc31ea98700 1 -- 172.21.15.70:0/626613232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc318107150 msgr2=0x7fc318107570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:51.088 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.761+0000 7fc31ea98700 1 --2- 172.21.15.70:0/626613232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc318107150 0x7fc318107570 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0098a0 tx=0x7fc30c009bb0 comp rx=0 tx=0).stop 2024-06-04T04:45:51.088 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.761+0000 7fc31ea98700 1 -- 172.21.15.70:0/626613232 shutdown_connections 2024-06-04T04:45:51.088 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.761+0000 7fc31ea98700 1 --2- 172.21.15.70:0/626613232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc318107150 0x7fc318107570 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:51.089 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.761+0000 7fc31ea98700 1 -- 172.21.15.70:0/626613232 >> 172.21.15.70:0/626613232 conn(0x7fc318102ae0 msgr2=0x7fc318104f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:51.089 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.761+0000 7fc31ea98700 1 -- 172.21.15.70:0/626613232 shutdown_connections 2024-06-04T04:45:51.089 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.761+0000 7fc31ea98700 1 -- 172.21.15.70:0/626613232 wait complete. 2024-06-04T04:45:51.089 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.761+0000 7fc31ea98700 1 Processor -- start 2024-06-04T04:45:51.089 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.762+0000 7fc31ea98700 1 -- start start 2024-06-04T04:45:51.089 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.762+0000 7fc31ea98700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc318107150 0x7fc318109960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:51.089 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.762+0000 7fc31ea98700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc31810d310 con 0x7fc318107150 2024-06-04T04:45:51.089 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.762+0000 7fc31c834700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc318107150 0x7fc318109960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:51.089 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.762+0000 7fc31c834700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc318107150 0x7fc318109960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38012/0 (socket says 172.21.15.70:38012) 2024-06-04T04:45:51.089 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.762+0000 7fc31c834700 1 -- 172.21.15.70:0/1552526040 learned_addr learned my addr 172.21.15.70:0/1552526040 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:51.089 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.762+0000 7fc31c834700 1 -- 172.21.15.70:0/1552526040 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc318109ea0 con 0x7fc318107150 2024-06-04T04:45:51.089 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.762+0000 7fc31c834700 1 --2- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc318107150 0x7fc318109960 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0119b0 tx=0x7fc30c011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:51.089 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.762+0000 7fc315ffb700 1 -- 172.21.15.70:0/1552526040 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc30c003b70 con 0x7fc318107150 2024-06-04T04:45:51.089 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.762+0000 7fc31ea98700 1 -- 172.21.15.70:0/1552526040 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc318108090 con 0x7fc318107150 2024-06-04T04:45:51.089 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.762+0000 7fc315ffb700 1 -- 172.21.15.70:0/1552526040 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fc30c01a8a0 con 0x7fc318107150 2024-06-04T04:45:51.090 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.762+0000 7fc315ffb700 1 -- 172.21.15.70:0/1552526040 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc30c024830 con 0x7fc318107150 2024-06-04T04:45:51.090 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.763+0000 7fc31ea98700 1 -- 172.21.15.70:0/1552526040 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3181085d0 con 0x7fc318107150 2024-06-04T04:45:51.090 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.764+0000 7fc315ffb700 1 -- 172.21.15.70:0/1552526040 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7fc30c022020 con 0x7fc318107150 2024-06-04T04:45:51.090 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.764+0000 7fc315ffb700 1 --2- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fc30803e420 0x7fc3080408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:51.090 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.764+0000 7fc315ffb700 1 -- 172.21.15.70:0/1552526040 --> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc308040ff0 con 0x7fc30803e420 2024-06-04T04:45:51.090 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.764+0000 7fc315ffb700 1 -- 172.21.15.70:0/1552526040 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc30c054630 con 0x7fc318107150 2024-06-04T04:45:51.090 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.764+0000 7fc317fff700 1 -- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fc30803e420 msgr2=0x7fc3080408e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2087909610 2024-06-04T04:45:51.090 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.764+0000 7fc317fff700 1 --2- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fc30803e420 0x7fc3080408e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-04T04:45:51.090 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.965+0000 7fc317fff700 1 -- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fc30803e420 msgr2=0x7fc3080408e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2087909610 2024-06-04T04:45:51.090 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:44.965+0000 7fc317fff700 1 --2- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fc30803e420 0x7fc3080408e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-04T04:45:51.090 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:45.366+0000 7fc317fff700 1 -- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fc30803e420 msgr2=0x7fc3080408e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2087909610 2024-06-04T04:45:51.090 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:45.366+0000 7fc317fff700 1 --2- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fc30803e420 0x7fc3080408e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-04T04:45:51.090 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:46.167+0000 7fc317fff700 1 -- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fc30803e420 msgr2=0x7fc3080408e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2087909610 2024-06-04T04:45:51.091 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:46.167+0000 7fc317fff700 1 --2- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fc30803e420 0x7fc3080408e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-04T04:45:51.091 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:47.768+0000 7fc317fff700 1 -- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fc30803e420 msgr2=0x7fc3080408e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2087909610 2024-06-04T04:45:51.091 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:47.768+0000 7fc317fff700 1 --2- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fc30803e420 0x7fc3080408e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-06-04T04:45:51.091 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:49.503+0000 7fc315ffb700 1 -- 172.21.15.70:0/1552526040 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mgrmap(e 6) ==== 49989+0+0 (secure 0 0 0) 0x7fc30c02d5d0 con 0x7fc318107150 2024-06-04T04:45:51.091 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:49.503+0000 7fc315ffb700 1 -- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fc30803e420 msgr2=0x7fc3080408e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-04T04:45:51.091 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:49.503+0000 7fc315ffb700 1 --2- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fc30803e420 0x7fc3080408e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:51.091 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.504+0000 7fc315ffb700 1 -- 172.21.15.70:0/1552526040 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7fc30c055080 con 0x7fc318107150 2024-06-04T04:45:51.091 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.504+0000 7fc315ffb700 1 --2- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fc308041d10 0x7fc308044100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:51.091 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.504+0000 7fc315ffb700 1 -- 172.21.15.70:0/1552526040 --> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc30c00be90 con 0x7fc308041d10 2024-06-04T04:45:51.091 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.504+0000 7fc317fff700 1 --2- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fc308041d10 0x7fc308044100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:51.091 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.505+0000 7fc317fff700 1 --2- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fc308041d10 0x7fc308044100 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc304003a10 tx=0x7fc304009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:51.091 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.506+0000 7fc315ffb700 1 -- 172.21.15.70:0/1552526040 <== mgr.14120 v2:172.21.15.70:6800/1353782413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fc30c00be90 con 0x7fc308041d10 2024-06-04T04:45:51.091 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.514+0000 7fc31ea98700 1 -- 172.21.15.70:0/1552526040 --> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fc31810d490 con 0x7fc308041d10 2024-06-04T04:45:51.092 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.514+0000 7fc315ffb700 1 -- 172.21.15.70:0/1552526040 <== mgr.14120 v2:172.21.15.70:6800/1353782413 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fc31810d490 con 0x7fc308041d10 2024-06-04T04:45:51.092 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.514+0000 7fc31ea98700 1 -- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fc308041d10 msgr2=0x7fc308044100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:51.092 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.514+0000 7fc31ea98700 1 --2- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fc308041d10 0x7fc308044100 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc304003a10 tx=0x7fc304009290 comp rx=0 tx=0).stop 2024-06-04T04:45:51.092 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.514+0000 7fc31ea98700 1 -- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc318107150 msgr2=0x7fc318109960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:51.092 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.514+0000 7fc31ea98700 1 --2- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc318107150 0x7fc318109960 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0119b0 tx=0x7fc30c011a90 comp rx=0 tx=0).stop 2024-06-04T04:45:51.092 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.514+0000 7fc31ea98700 1 -- 172.21.15.70:0/1552526040 shutdown_connections 2024-06-04T04:45:51.092 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.514+0000 7fc31ea98700 1 --2- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fc308041d10 0x7fc308044100 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:51.092 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.514+0000 7fc31ea98700 1 --2- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc318107150 0x7fc318109960 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:51.092 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.514+0000 7fc31ea98700 1 --2- 172.21.15.70:0/1552526040 >> [v2:172.21.15.70:6800/2087909610,v1:172.21.15.70:6801/2087909610] conn(0x7fc30803e420 0x7fc3080408e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:51.092 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.514+0000 7fc31ea98700 1 -- 172.21.15.70:0/1552526040 >> 172.21.15.70:0/1552526040 conn(0x7fc318102ae0 msgr2=0x7fc318103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:51.092 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.514+0000 7fc31ea98700 1 -- 172.21.15.70:0/1552526040 shutdown_connections 2024-06-04T04:45:51.092 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:50.514+0000 7fc31ea98700 1 -- 172.21.15.70:0/1552526040 wait complete. 2024-06-04T04:45:51.092 INFO:teuthology.orchestra.run.smithi070.stdout:mgr epoch 5 is available 2024-06-04T04:45:51.092 INFO:teuthology.orchestra.run.smithi070.stdout:Setting orchestrator backend to cephadm... 2024-06-04T04:45:51.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:51 smithi070 ceph-mon[32631]: [04/Jun/2024:04:45:50] ENGINE Serving on http://172.21.15.70:8765 2024-06-04T04:45:51.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:51 smithi070 ceph-mon[32631]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-04T04:45:51.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:51 smithi070 ceph-mon[32631]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-04T04:45:51.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:51 smithi070 ceph-mon[32631]: [04/Jun/2024:04:45:50] ENGINE Serving on https://172.21.15.70:7150 2024-06-04T04:45:51.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:51 smithi070 ceph-mon[32631]: [04/Jun/2024:04:45:50] ENGINE Bus STARTED 2024-06-04T04:45:51.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:51 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:45:52.249 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.497+0000 7fdbc13b2700 1 Processor -- start 2024-06-04T04:45:52.250 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.498+0000 7fdbc13b2700 1 -- start start 2024-06-04T04:45:52.250 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.498+0000 7fdbc13b2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdbbc109900 0x7fdbbc10bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:52.250 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.498+0000 7fdbc13b2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdbbc07ce50 con 0x7fdbbc109900 2024-06-04T04:45:52.250 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.498+0000 7fdbbaffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdbbc109900 0x7fdbbc10bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:52.250 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.498+0000 7fdbbaffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdbbc109900 0x7fdbbc10bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57626/0 (socket says 172.21.15.70:57626) 2024-06-04T04:45:52.250 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.498+0000 7fdbbaffd700 1 -- 172.21.15.70:0/959886222 learned_addr learned my addr 172.21.15.70:0/959886222 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:52.250 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.498+0000 7fdbbaffd700 1 -- 172.21.15.70:0/959886222 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbbc10c230 con 0x7fdbbc109900 2024-06-04T04:45:52.250 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.498+0000 7fdbbaffd700 1 --2- 172.21.15.70:0/959886222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdbbc109900 0x7fdbbc10bcf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdba40098a0 tx=0x7fdba4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a977382c7e37bd62 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:52.250 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.498+0000 7fdbb9ffb700 1 -- 172.21.15.70:0/959886222 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdba4004160 con 0x7fdbbc109900 2024-06-04T04:45:52.250 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.498+0000 7fdbb9ffb700 1 -- 172.21.15.70:0/959886222 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fdba4004300 con 0x7fdbbc109900 2024-06-04T04:45:52.251 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.498+0000 7fdbb9ffb700 1 -- 172.21.15.70:0/959886222 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdba40119e0 con 0x7fdbbc109900 2024-06-04T04:45:52.251 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.499+0000 7fdbc13b2700 1 -- 172.21.15.70:0/959886222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdbbc109900 msgr2=0x7fdbbc10bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:52.251 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.499+0000 7fdbc13b2700 1 --2- 172.21.15.70:0/959886222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdbbc109900 0x7fdbbc10bcf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdba40098a0 tx=0x7fdba4009bb0 comp rx=0 tx=0).stop 2024-06-04T04:45:52.251 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.499+0000 7fdbc13b2700 1 -- 172.21.15.70:0/959886222 shutdown_connections 2024-06-04T04:45:52.251 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.499+0000 7fdbc13b2700 1 --2- 172.21.15.70:0/959886222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdbbc109900 0x7fdbbc10bcf0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:52.251 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.499+0000 7fdbc13b2700 1 -- 172.21.15.70:0/959886222 >> 172.21.15.70:0/959886222 conn(0x7fdbbc103330 msgr2=0x7fdbbc105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:52.251 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.499+0000 7fdbc13b2700 1 -- 172.21.15.70:0/959886222 shutdown_connections 2024-06-04T04:45:52.251 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.499+0000 7fdbc13b2700 1 -- 172.21.15.70:0/959886222 wait complete. 2024-06-04T04:45:52.251 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.500+0000 7fdbc13b2700 1 Processor -- start 2024-06-04T04:45:52.251 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.500+0000 7fdbc13b2700 1 -- start start 2024-06-04T04:45:52.251 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.500+0000 7fdbc13b2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdbbc109900 0x7fdbbc19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:52.251 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.500+0000 7fdbc13b2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdbbc10d190 con 0x7fdbbc109900 2024-06-04T04:45:52.251 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.500+0000 7fdbbaffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdbbc109900 0x7fdbbc19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:52.251 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.500+0000 7fdbbaffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdbbc109900 0x7fdbbc19cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57632/0 (socket says 172.21.15.70:57632) 2024-06-04T04:45:52.252 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.500+0000 7fdbbaffd700 1 -- 172.21.15.70:0/1416071544 learned_addr learned my addr 172.21.15.70:0/1416071544 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:52.252 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.500+0000 7fdbbaffd700 1 -- 172.21.15.70:0/1416071544 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbbc19d260 con 0x7fdbbc109900 2024-06-04T04:45:52.252 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.501+0000 7fdbbaffd700 1 --2- 172.21.15.70:0/1416071544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdbbc109900 0x7fdbbc19cd20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdba40119e0 tx=0x7fdba40046a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:52.252 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.501+0000 7fdbb3fff700 1 -- 172.21.15.70:0/1416071544 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdba4003bf0 con 0x7fdbbc109900 2024-06-04T04:45:52.252 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.501+0000 7fdbb3fff700 1 -- 172.21.15.70:0/1416071544 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fdba401a8a0 con 0x7fdbbc109900 2024-06-04T04:45:52.252 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.501+0000 7fdbc13b2700 1 -- 172.21.15.70:0/1416071544 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbbc19d4f0 con 0x7fdbbc109900 2024-06-04T04:45:52.252 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.501+0000 7fdbb3fff700 1 -- 172.21.15.70:0/1416071544 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdba40248c0 con 0x7fdbbc109900 2024-06-04T04:45:52.252 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.501+0000 7fdbc13b2700 1 -- 172.21.15.70:0/1416071544 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbbc1a01d0 con 0x7fdbbc109900 2024-06-04T04:45:52.252 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.502+0000 7fdbb3fff700 1 -- 172.21.15.70:0/1416071544 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7fdba4022020 con 0x7fdbbc109900 2024-06-04T04:45:52.252 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.502+0000 7fdbc13b2700 1 -- 172.21.15.70:0/1416071544 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbbc064ba0 con 0x7fdbbc109900 2024-06-04T04:45:52.252 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.502+0000 7fdbb3fff700 1 --2- 172.21.15.70:0/1416071544 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fdba803e310 0x7fdba80407d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:52.252 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.502+0000 7fdbb3fff700 1 -- 172.21.15.70:0/1416071544 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdba4053770 con 0x7fdbbc109900 2024-06-04T04:45:52.252 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.502+0000 7fdbba7fc700 1 --2- 172.21.15.70:0/1416071544 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fdba803e310 0x7fdba80407d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:52.252 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.502+0000 7fdbba7fc700 1 --2- 172.21.15.70:0/1416071544 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fdba803e310 0x7fdba80407d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdbac006fd0 tx=0x7fdbac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:52.253 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.507+0000 7fdbb3fff700 1 -- 172.21.15.70:0/1416071544 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdba402c350 con 0x7fdbbc109900 2024-06-04T04:45:52.253 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.555+0000 7fdbb3fff700 1 -- 172.21.15.70:0/1416071544 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fdba4055d00 con 0x7fdbbc109900 2024-06-04T04:45:52.253 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.725+0000 7fdbc13b2700 1 -- 172.21.15.70:0/1416071544 --> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fdbbc104570 con 0x7fdba803e310 2024-06-04T04:45:52.253 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.730+0000 7fdbb3fff700 1 -- 172.21.15.70:0/1416071544 <== mgr.14120 v2:172.21.15.70:6800/1353782413 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdbbc104570 con 0x7fdba803e310 2024-06-04T04:45:52.253 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.732+0000 7fdbc13b2700 1 -- 172.21.15.70:0/1416071544 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fdba803e310 msgr2=0x7fdba80407d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:52.253 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.732+0000 7fdbc13b2700 1 --2- 172.21.15.70:0/1416071544 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fdba803e310 0x7fdba80407d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdbac006fd0 tx=0x7fdbac006e40 comp rx=0 tx=0).stop 2024-06-04T04:45:52.253 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.732+0000 7fdbc13b2700 1 -- 172.21.15.70:0/1416071544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdbbc109900 msgr2=0x7fdbbc19cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:52.253 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.732+0000 7fdbc13b2700 1 --2- 172.21.15.70:0/1416071544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdbbc109900 0x7fdbbc19cd20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdba40119e0 tx=0x7fdba40046a0 comp rx=0 tx=0).stop 2024-06-04T04:45:52.253 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.732+0000 7fdbc13b2700 1 -- 172.21.15.70:0/1416071544 shutdown_connections 2024-06-04T04:45:52.253 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.733+0000 7fdbc13b2700 1 --2- 172.21.15.70:0/1416071544 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fdba803e310 0x7fdba80407d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:52.253 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.733+0000 7fdbc13b2700 1 --2- 172.21.15.70:0/1416071544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdbbc109900 0x7fdbbc19cd20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:52.253 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.733+0000 7fdbc13b2700 1 -- 172.21.15.70:0/1416071544 >> 172.21.15.70:0/1416071544 conn(0x7fdbbc103330 msgr2=0x7fdbbc103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:52.253 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.733+0000 7fdbc13b2700 1 -- 172.21.15.70:0/1416071544 shutdown_connections 2024-06-04T04:45:52.253 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:51.733+0000 7fdbc13b2700 1 -- 172.21.15.70:0/1416071544 wait complete. 2024-06-04T04:45:52.653 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:52 smithi070 ceph-mon[32631]: mgrmap e8: x(active, since 2s) 2024-06-04T04:45:52.653 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:52 smithi070 ceph-mon[32631]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:45:52.653 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:52 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' 2024-06-04T04:45:52.653 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:52 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:45:53.505 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout value unchanged 2024-06-04T04:45:53.505 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.668+0000 7fb58e1f9700 1 Processor -- start 2024-06-04T04:45:53.505 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.668+0000 7fb58e1f9700 1 -- start start 2024-06-04T04:45:53.505 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.669+0000 7fb58e1f9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb588107db0 0x7fb5881081d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:53.505 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.669+0000 7fb58e1f9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb5881087a0 con 0x7fb588107db0 2024-06-04T04:45:53.505 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.669+0000 7fb5877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb588107db0 0x7fb5881081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:53.505 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.669+0000 7fb5877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb588107db0 0x7fb5881081d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57640/0 (socket says 172.21.15.70:57640) 2024-06-04T04:45:53.506 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.669+0000 7fb5877fe700 1 -- 172.21.15.70:0/1677074283 learned_addr learned my addr 172.21.15.70:0/1677074283 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:53.506 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.669+0000 7fb5877fe700 1 -- 172.21.15.70:0/1677074283 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb588108fb0 con 0x7fb588107db0 2024-06-04T04:45:53.506 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.669+0000 7fb5877fe700 1 --2- 172.21.15.70:0/1677074283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb588107db0 0x7fb5881081d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb5780098a0 tx=0x7fb578009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e0dbab636aa75b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:53.506 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.669+0000 7fb586ffd700 1 -- 172.21.15.70:0/1677074283 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb578004160 con 0x7fb588107db0 2024-06-04T04:45:53.506 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.669+0000 7fb586ffd700 1 -- 172.21.15.70:0/1677074283 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fb578004300 con 0x7fb588107db0 2024-06-04T04:45:53.506 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.669+0000 7fb586ffd700 1 -- 172.21.15.70:0/1677074283 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5780119e0 con 0x7fb588107db0 2024-06-04T04:45:53.506 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.670+0000 7fb58e1f9700 1 -- 172.21.15.70:0/1677074283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb588107db0 msgr2=0x7fb5881081d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:53.506 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.670+0000 7fb58e1f9700 1 --2- 172.21.15.70:0/1677074283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb588107db0 0x7fb5881081d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb5780098a0 tx=0x7fb578009bb0 comp rx=0 tx=0).stop 2024-06-04T04:45:53.506 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.670+0000 7fb58e1f9700 1 -- 172.21.15.70:0/1677074283 shutdown_connections 2024-06-04T04:45:53.506 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.670+0000 7fb58e1f9700 1 --2- 172.21.15.70:0/1677074283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb588107db0 0x7fb5881081d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:53.506 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.670+0000 7fb58e1f9700 1 -- 172.21.15.70:0/1677074283 >> 172.21.15.70:0/1677074283 conn(0x7fb588103370 msgr2=0x7fb588105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:53.506 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.670+0000 7fb58e1f9700 1 -- 172.21.15.70:0/1677074283 shutdown_connections 2024-06-04T04:45:53.506 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.670+0000 7fb58e1f9700 1 -- 172.21.15.70:0/1677074283 wait complete. 2024-06-04T04:45:53.507 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.671+0000 7fb58e1f9700 1 Processor -- start 2024-06-04T04:45:53.507 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.671+0000 7fb58e1f9700 1 -- start start 2024-06-04T04:45:53.507 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.671+0000 7fb58e1f9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb588107db0 0x7fb588192e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:53.507 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.671+0000 7fb58e1f9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb58810c1e0 con 0x7fb588107db0 2024-06-04T04:45:53.507 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.671+0000 7fb5877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb588107db0 0x7fb588192e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:53.507 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.671+0000 7fb5877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb588107db0 0x7fb588192e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57644/0 (socket says 172.21.15.70:57644) 2024-06-04T04:45:53.507 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.671+0000 7fb5877fe700 1 -- 172.21.15.70:0/1325297003 learned_addr learned my addr 172.21.15.70:0/1325297003 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:53.507 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.671+0000 7fb5877fe700 1 -- 172.21.15.70:0/1325297003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5881933c0 con 0x7fb588107db0 2024-06-04T04:45:53.507 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.671+0000 7fb5877fe700 1 --2- 172.21.15.70:0/1325297003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb588107db0 0x7fb588192e80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb5780119b0 tx=0x7fb578011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:53.507 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.672+0000 7fb5857fa700 1 -- 172.21.15.70:0/1325297003 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb578003b70 con 0x7fb588107db0 2024-06-04T04:45:53.507 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.672+0000 7fb5857fa700 1 -- 172.21.15.70:0/1325297003 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fb57801a8a0 con 0x7fb588107db0 2024-06-04T04:45:53.507 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.672+0000 7fb58e1f9700 1 -- 172.21.15.70:0/1325297003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5881915b0 con 0x7fb588107db0 2024-06-04T04:45:53.507 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.672+0000 7fb5857fa700 1 -- 172.21.15.70:0/1325297003 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb578024830 con 0x7fb588107db0 2024-06-04T04:45:53.508 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.672+0000 7fb58e1f9700 1 -- 172.21.15.70:0/1325297003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb588191af0 con 0x7fb588107db0 2024-06-04T04:45:53.508 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.673+0000 7fb5857fa700 1 -- 172.21.15.70:0/1325297003 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fb578022020 con 0x7fb588107db0 2024-06-04T04:45:53.508 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.673+0000 7fb58e1f9700 1 -- 172.21.15.70:0/1325297003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb588064ba0 con 0x7fb588107db0 2024-06-04T04:45:53.508 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.673+0000 7fb5857fa700 1 --2- 172.21.15.70:0/1325297003 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fb57403e470 0x7fb574040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:53.508 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.673+0000 7fb5857fa700 1 -- 172.21.15.70:0/1325297003 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb578053a00 con 0x7fb588107db0 2024-06-04T04:45:53.508 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.673+0000 7fb57ffff700 1 --2- 172.21.15.70:0/1325297003 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fb57403e470 0x7fb574040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:53.508 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.673+0000 7fb57ffff700 1 --2- 172.21.15.70:0/1325297003 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fb57403e470 0x7fb574040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb570006fd0 tx=0x7fb570006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:53.508 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.678+0000 7fb5857fa700 1 -- 172.21.15.70:0/1325297003 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb57801aa50 con 0x7fb588107db0 2024-06-04T04:45:53.508 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.881+0000 7fb58e1f9700 1 -- 172.21.15.70:0/1325297003 --> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fb588192220 con 0x7fb57403e470 2024-06-04T04:45:53.508 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.882+0000 7fb5857fa700 1 -- 172.21.15.70:0/1325297003 <== mgr.14120 v2:172.21.15.70:6800/1353782413 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fb588192220 con 0x7fb57403e470 2024-06-04T04:45:53.508 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.884+0000 7fb58e1f9700 1 -- 172.21.15.70:0/1325297003 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fb57403e470 msgr2=0x7fb574040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:53.508 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.884+0000 7fb58e1f9700 1 --2- 172.21.15.70:0/1325297003 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fb57403e470 0x7fb574040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb570006fd0 tx=0x7fb570006e40 comp rx=0 tx=0).stop 2024-06-04T04:45:53.508 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.884+0000 7fb58e1f9700 1 -- 172.21.15.70:0/1325297003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb588107db0 msgr2=0x7fb588192e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:53.508 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.884+0000 7fb58e1f9700 1 --2- 172.21.15.70:0/1325297003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb588107db0 0x7fb588192e80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb5780119b0 tx=0x7fb578011a90 comp rx=0 tx=0).stop 2024-06-04T04:45:53.508 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.884+0000 7fb58e1f9700 1 -- 172.21.15.70:0/1325297003 shutdown_connections 2024-06-04T04:45:53.509 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.884+0000 7fb58e1f9700 1 --2- 172.21.15.70:0/1325297003 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fb57403e470 0x7fb574040930 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:53.509 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.884+0000 7fb58e1f9700 1 --2- 172.21.15.70:0/1325297003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb588107db0 0x7fb588192e80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:53.509 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.884+0000 7fb58e1f9700 1 -- 172.21.15.70:0/1325297003 >> 172.21.15.70:0/1325297003 conn(0x7fb588103370 msgr2=0x7fb588103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:53.509 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.884+0000 7fb58e1f9700 1 -- 172.21.15.70:0/1325297003 shutdown_connections 2024-06-04T04:45:53.509 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:52.884+0000 7fb58e1f9700 1 -- 172.21.15.70:0/1325297003 wait complete. 2024-06-04T04:45:53.509 INFO:teuthology.orchestra.run.smithi070.stdout:Generating ssh key... 2024-06-04T04:45:53.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:53 smithi070 ceph-mon[32631]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:45:54.609 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: Generating public/private rsa key pair. 2024-06-04T04:45:54.609 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: Your identification has been saved in /tmp/tmpqalsjm6b/key. 2024-06-04T04:45:54.610 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: Your public key has been saved in /tmp/tmpqalsjm6b/key.pub. 2024-06-04T04:45:54.610 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: The key fingerprint is: 2024-06-04T04:45:54.610 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: SHA256:UKboy2CRxHmj4FyqjxfLSsvLj/QQo+C4X8YSUQBrET4 ceph-164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:45:54.610 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: The key's randomart image is: 2024-06-04T04:45:54.610 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: +---[RSA 3072]----+ 2024-06-04T04:45:54.610 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: |.=+o. o | 2024-06-04T04:45:54.610 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: |oo++o. + | 2024-06-04T04:45:54.610 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: |+E*o..o | 2024-06-04T04:45:54.610 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: |.+o+ . | 2024-06-04T04:45:54.610 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: |oo+ . S | 2024-06-04T04:45:54.610 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: |*.+= . | 2024-06-04T04:45:54.610 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: |oO.o* | 2024-06-04T04:45:54.610 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: |*.X+ | 2024-06-04T04:45:54.611 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: |oX+o | 2024-06-04T04:45:54.611 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:45:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: +----[SHA256]-----+ 2024-06-04T04:45:54.894 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.946+0000 7efc3c8e9700 1 Processor -- start 2024-06-04T04:45:54.895 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.946+0000 7efc3c8e9700 1 -- start start 2024-06-04T04:45:54.895 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.946+0000 7efc3c8e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc38109960 0x7efc3810bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:54.895 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.946+0000 7efc3c8e9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efc3807ce30 con 0x7efc38109960 2024-06-04T04:45:54.895 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.947+0000 7efc3659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc38109960 0x7efc3810bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:54.895 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.947+0000 7efc3659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc38109960 0x7efc3810bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57646/0 (socket says 172.21.15.70:57646) 2024-06-04T04:45:54.895 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.947+0000 7efc3659c700 1 -- 172.21.15.70:0/62877472 learned_addr learned my addr 172.21.15.70:0/62877472 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:54.895 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.947+0000 7efc3659c700 1 -- 172.21.15.70:0/62877472 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc3810c290 con 0x7efc38109960 2024-06-04T04:45:54.895 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.947+0000 7efc3659c700 1 --2- 172.21.15.70:0/62877472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc38109960 0x7efc3810bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efc28009e80 tx=0x7efc2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91e44f7200a9f67 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:54.895 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.947+0000 7efc3559a700 1 -- 172.21.15.70:0/62877472 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc28017070 con 0x7efc38109960 2024-06-04T04:45:54.895 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.947+0000 7efc3559a700 1 -- 172.21.15.70:0/62877472 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7efc2800b8c0 con 0x7efc38109960 2024-06-04T04:45:54.896 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.947+0000 7efc3559a700 1 -- 172.21.15.70:0/62877472 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc28004420 con 0x7efc38109960 2024-06-04T04:45:54.896 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.948+0000 7efc3c8e9700 1 -- 172.21.15.70:0/62877472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc38109960 msgr2=0x7efc3810bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:54.896 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.948+0000 7efc3c8e9700 1 --2- 172.21.15.70:0/62877472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc38109960 0x7efc3810bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efc28009e80 tx=0x7efc2800b330 comp rx=0 tx=0).stop 2024-06-04T04:45:54.896 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.948+0000 7efc3c8e9700 1 -- 172.21.15.70:0/62877472 shutdown_connections 2024-06-04T04:45:54.896 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.948+0000 7efc3c8e9700 1 --2- 172.21.15.70:0/62877472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc38109960 0x7efc3810bd50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:54.896 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.948+0000 7efc3c8e9700 1 -- 172.21.15.70:0/62877472 >> 172.21.15.70:0/62877472 conn(0x7efc381032f0 msgr2=0x7efc38105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:54.896 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.948+0000 7efc3c8e9700 1 -- 172.21.15.70:0/62877472 shutdown_connections 2024-06-04T04:45:54.896 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.948+0000 7efc3c8e9700 1 -- 172.21.15.70:0/62877472 wait complete. 2024-06-04T04:45:54.896 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.949+0000 7efc3c8e9700 1 Processor -- start 2024-06-04T04:45:54.896 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.949+0000 7efc3c8e9700 1 -- start start 2024-06-04T04:45:54.896 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.949+0000 7efc3c8e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc38109960 0x7efc3819ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:54.897 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.949+0000 7efc3c8e9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efc3810d1a0 con 0x7efc38109960 2024-06-04T04:45:54.897 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.949+0000 7efc3659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc38109960 0x7efc3819ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:54.897 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.949+0000 7efc3659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc38109960 0x7efc3819ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57656/0 (socket says 172.21.15.70:57656) 2024-06-04T04:45:54.897 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.949+0000 7efc3659c700 1 -- 172.21.15.70:0/4151764413 learned_addr learned my addr 172.21.15.70:0/4151764413 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:54.897 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.949+0000 7efc3659c700 1 -- 172.21.15.70:0/4151764413 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc3819d230 con 0x7efc38109960 2024-06-04T04:45:54.897 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.949+0000 7efc3659c700 1 --2- 172.21.15.70:0/4151764413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc38109960 0x7efc3819ccf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efc28007130 tx=0x7efc2800bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:54.897 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.950+0000 7efc2f7fe700 1 -- 172.21.15.70:0/4151764413 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc28020020 con 0x7efc38109960 2024-06-04T04:45:54.897 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.950+0000 7efc2f7fe700 1 -- 172.21.15.70:0/4151764413 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7efc28003bd0 con 0x7efc38109960 2024-06-04T04:45:54.897 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.950+0000 7efc3c8e9700 1 -- 172.21.15.70:0/4151764413 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc3819d4c0 con 0x7efc38109960 2024-06-04T04:45:54.897 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.950+0000 7efc2f7fe700 1 -- 172.21.15.70:0/4151764413 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc28017040 con 0x7efc38109960 2024-06-04T04:45:54.898 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.950+0000 7efc3c8e9700 1 -- 172.21.15.70:0/4151764413 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc381a01a0 con 0x7efc38109960 2024-06-04T04:45:54.898 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.951+0000 7efc2f7fe700 1 -- 172.21.15.70:0/4151764413 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7efc28023020 con 0x7efc38109960 2024-06-04T04:45:54.898 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.951+0000 7efc3c8e9700 1 -- 172.21.15.70:0/4151764413 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc38064ba0 con 0x7efc38109960 2024-06-04T04:45:54.898 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.951+0000 7efc2f7fe700 1 --2- 172.21.15.70:0/4151764413 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7efc2403e470 0x7efc24040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:54.898 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.951+0000 7efc2f7fe700 1 -- 172.21.15.70:0/4151764413 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7efc2800f050 con 0x7efc38109960 2024-06-04T04:45:54.898 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.951+0000 7efc35d9b700 1 --2- 172.21.15.70:0/4151764413 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7efc2403e470 0x7efc24040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:54.898 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.951+0000 7efc35d9b700 1 --2- 172.21.15.70:0/4151764413 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7efc2403e470 0x7efc24040930 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efc20006fd0 tx=0x7efc20006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:54.898 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:53.956+0000 7efc2f7fe700 1 -- 172.21.15.70:0/4151764413 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7efc2802a020 con 0x7efc38109960 2024-06-04T04:45:54.898 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:54.161+0000 7efc3c8e9700 1 -- 172.21.15.70:0/4151764413 --> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7efc381044b0 con 0x7efc2403e470 2024-06-04T04:45:54.898 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:54.326+0000 7efc2f7fe700 1 -- 172.21.15.70:0/4151764413 <== mgr.14120 v2:172.21.15.70:6800/1353782413 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7efc381044b0 con 0x7efc2403e470 2024-06-04T04:45:54.899 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:54.328+0000 7efc3c8e9700 1 -- 172.21.15.70:0/4151764413 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7efc2403e470 msgr2=0x7efc24040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:54.899 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:54.328+0000 7efc3c8e9700 1 --2- 172.21.15.70:0/4151764413 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7efc2403e470 0x7efc24040930 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efc20006fd0 tx=0x7efc20006e40 comp rx=0 tx=0).stop 2024-06-04T04:45:54.899 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:54.328+0000 7efc3c8e9700 1 -- 172.21.15.70:0/4151764413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc38109960 msgr2=0x7efc3819ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:54.899 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:54.328+0000 7efc3c8e9700 1 --2- 172.21.15.70:0/4151764413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc38109960 0x7efc3819ccf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efc28007130 tx=0x7efc2800bdc0 comp rx=0 tx=0).stop 2024-06-04T04:45:54.899 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:54.329+0000 7efc3c8e9700 1 -- 172.21.15.70:0/4151764413 shutdown_connections 2024-06-04T04:45:54.899 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:54.329+0000 7efc3c8e9700 1 --2- 172.21.15.70:0/4151764413 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7efc2403e470 0x7efc24040930 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:54.899 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:54.329+0000 7efc3c8e9700 1 --2- 172.21.15.70:0/4151764413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc38109960 0x7efc3819ccf0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:54.899 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:54.329+0000 7efc3c8e9700 1 -- 172.21.15.70:0/4151764413 >> 172.21.15.70:0/4151764413 conn(0x7efc381032f0 msgr2=0x7efc38103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:54.899 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:54.329+0000 7efc3c8e9700 1 -- 172.21.15.70:0/4151764413 shutdown_connections 2024-06-04T04:45:54.899 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:54.329+0000 7efc3c8e9700 1 -- 172.21.15.70:0/4151764413 wait complete. 2024-06-04T04:45:55.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:55 smithi070 ceph-mon[32631]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:45:55.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:55 smithi070 ceph-mon[32631]: Generating ssh key... 2024-06-04T04:45:55.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:55 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' 2024-06-04T04:45:55.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:55 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' 2024-06-04T04:45:56.024 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg5CwZGXZLnV1TcuIPz0z5UDu4ADBqOutmIkMwVsZ4wRJEDtP8oTRYtubDPA2pMHnfTbE6VbeV/E65ir1mONox67UC/ks3B/sVFdKm7h63Dwd8k0yP0gg5k6zvzU3lUD45T2XeUlnuM1pjJJPHSixOJ7Of2T5faYtRGZMkPB/MWkicagFj9oSMNyci7VljguWMARuXsOnfcq8irgmUB24p3g4JjDsq+SkTVxj0c6XzoFgpE+NTvF1jlsINnKZT4Pgw9R5vfu9lFIpRDT7A+HYDAw7ZcrcxqJdIVo18pchKZj0PvXc1nPln4x7E/19BgpJyeujhEqXuVQAEH5B7Uq40uZ+TENdWWy+kp2O1eFUBM5ryzgWTLFCsGt7A/g9VCsBoAhDWY9PXbOIDb0RdyRDK/xXs+Ryv8oio6/dWD6YPB3b/ynhmQyGl+xvHcv1mCyoMWSj2SGrrXWgArIYya115d6J8CfmnTWODJ6rHjngENspR+Zc9L6znbKFExZWJPFM= ceph-164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:45:56.025 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.291+0000 7f6425e40700 1 Processor -- start 2024-06-04T04:45:56.025 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.291+0000 7f6425e40700 1 -- start start 2024-06-04T04:45:56.025 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.291+0000 7f6425e40700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6420108ea0 0x7f64201092c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:56.025 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.291+0000 7f6425e40700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6420109800 con 0x7f6420108ea0 2024-06-04T04:45:56.025 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.291+0000 7f641f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6420108ea0 0x7f64201092c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:56.025 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.291+0000 7f641f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6420108ea0 0x7f64201092c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57658/0 (socket says 172.21.15.70:57658) 2024-06-04T04:45:56.025 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.291+0000 7f641f7fe700 1 -- 172.21.15.70:0/142447494 learned_addr learned my addr 172.21.15.70:0/142447494 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:56.025 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.291+0000 7f641f7fe700 1 -- 172.21.15.70:0/142447494 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6420109980 con 0x7f6420108ea0 2024-06-04T04:45:56.025 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.291+0000 7f641f7fe700 1 --2- 172.21.15.70:0/142447494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6420108ea0 0x7f64201092c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f64100098a0 tx=0x7f6410009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9ca8b2bb9c67a2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:56.025 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.292+0000 7f641effd700 1 -- 172.21.15.70:0/142447494 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6410004160 con 0x7f6420108ea0 2024-06-04T04:45:56.025 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.292+0000 7f641effd700 1 -- 172.21.15.70:0/142447494 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f6410004300 con 0x7f6420108ea0 2024-06-04T04:45:56.026 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.292+0000 7f641effd700 1 -- 172.21.15.70:0/142447494 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64100119e0 con 0x7f6420108ea0 2024-06-04T04:45:56.026 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.292+0000 7f6425e40700 1 -- 172.21.15.70:0/142447494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6420108ea0 msgr2=0x7f64201092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:56.026 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.292+0000 7f6425e40700 1 --2- 172.21.15.70:0/142447494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6420108ea0 0x7f64201092c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f64100098a0 tx=0x7f6410009bb0 comp rx=0 tx=0).stop 2024-06-04T04:45:56.026 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.292+0000 7f6425e40700 1 -- 172.21.15.70:0/142447494 shutdown_connections 2024-06-04T04:45:56.026 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.292+0000 7f6425e40700 1 --2- 172.21.15.70:0/142447494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6420108ea0 0x7f64201092c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:56.026 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.292+0000 7f6425e40700 1 -- 172.21.15.70:0/142447494 >> 172.21.15.70:0/142447494 conn(0x7f64201044e0 msgr2=0x7f6420106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:56.026 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.292+0000 7f6425e40700 1 -- 172.21.15.70:0/142447494 shutdown_connections 2024-06-04T04:45:56.026 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.292+0000 7f6425e40700 1 -- 172.21.15.70:0/142447494 wait complete. 2024-06-04T04:45:56.026 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.293+0000 7f6425e40700 1 Processor -- start 2024-06-04T04:45:56.027 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.293+0000 7f6425e40700 1 -- start start 2024-06-04T04:45:56.027 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.293+0000 7f6425e40700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6420108ea0 0x7f642007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:56.027 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.293+0000 7f6425e40700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f642010adf0 con 0x7f6420108ea0 2024-06-04T04:45:56.027 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.293+0000 7f641f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6420108ea0 0x7f642007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:56.027 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.293+0000 7f641f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6420108ea0 0x7f642007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57672/0 (socket says 172.21.15.70:57672) 2024-06-04T04:45:56.027 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.293+0000 7f641f7fe700 1 -- 172.21.15.70:0/3960259938 learned_addr learned my addr 172.21.15.70:0/3960259938 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:45:56.027 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.293+0000 7f641f7fe700 1 -- 172.21.15.70:0/3960259938 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f642007f090 con 0x7f6420108ea0 2024-06-04T04:45:56.027 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.294+0000 7f641f7fe700 1 --2- 172.21.15.70:0/3960259938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6420108ea0 0x7f642007eb50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6410009130 tx=0x7f6410011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:56.027 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.294+0000 7f641d7fa700 1 -- 172.21.15.70:0/3960259938 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6410003b20 con 0x7f6420108ea0 2024-06-04T04:45:56.027 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.294+0000 7f6425e40700 1 -- 172.21.15.70:0/3960259938 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f642007d280 con 0x7f6420108ea0 2024-06-04T04:45:56.028 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.294+0000 7f641d7fa700 1 -- 172.21.15.70:0/3960259938 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f641001a8a0 con 0x7f6420108ea0 2024-06-04T04:45:56.028 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.294+0000 7f641d7fa700 1 -- 172.21.15.70:0/3960259938 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6410024810 con 0x7f6420108ea0 2024-06-04T04:45:56.028 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.294+0000 7f6425e40700 1 -- 172.21.15.70:0/3960259938 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f642007d7c0 con 0x7f6420108ea0 2024-06-04T04:45:56.028 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.295+0000 7f641d7fa700 1 -- 172.21.15.70:0/3960259938 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f6410022020 con 0x7f6420108ea0 2024-06-04T04:45:56.028 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.295+0000 7f6425e40700 1 -- 172.21.15.70:0/3960259938 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6420064ba0 con 0x7f6420108ea0 2024-06-04T04:45:56.028 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.295+0000 7f641d7fa700 1 --2- 172.21.15.70:0/3960259938 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f640c03e470 0x7f640c040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:45:56.028 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.295+0000 7f641d7fa700 1 -- 172.21.15.70:0/3960259938 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6410053910 con 0x7f6420108ea0 2024-06-04T04:45:56.028 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.295+0000 7f6417fff700 1 --2- 172.21.15.70:0/3960259938 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f640c03e470 0x7f640c040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:45:56.028 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.295+0000 7f6417fff700 1 --2- 172.21.15.70:0/3960259938 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f640c03e470 0x7f640c040930 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6408006fd0 tx=0x7f6408006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:45:56.028 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.300+0000 7f641d7fa700 1 -- 172.21.15.70:0/3960259938 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6410005a20 con 0x7f6420108ea0 2024-06-04T04:45:56.028 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.504+0000 7f6425e40700 1 -- 172.21.15.70:0/3960259938 --> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f6420105690 con 0x7f640c03e470 2024-06-04T04:45:56.028 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.504+0000 7f641d7fa700 1 -- 172.21.15.70:0/3960259938 <== mgr.14120 v2:172.21.15.70:6800/1353782413 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f6420105690 con 0x7f640c03e470 2024-06-04T04:45:56.028 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.507+0000 7f6425e40700 1 -- 172.21.15.70:0/3960259938 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f640c03e470 msgr2=0x7f640c040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:56.028 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.507+0000 7f6425e40700 1 --2- 172.21.15.70:0/3960259938 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f640c03e470 0x7f640c040930 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6408006fd0 tx=0x7f6408006e40 comp rx=0 tx=0).stop 2024-06-04T04:45:56.029 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.507+0000 7f6425e40700 1 -- 172.21.15.70:0/3960259938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6420108ea0 msgr2=0x7f642007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:45:56.029 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.507+0000 7f6425e40700 1 --2- 172.21.15.70:0/3960259938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6420108ea0 0x7f642007eb50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6410009130 tx=0x7f6410011a60 comp rx=0 tx=0).stop 2024-06-04T04:45:56.029 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.507+0000 7f6425e40700 1 -- 172.21.15.70:0/3960259938 shutdown_connections 2024-06-04T04:45:56.029 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.507+0000 7f6425e40700 1 --2- 172.21.15.70:0/3960259938 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f640c03e470 0x7f640c040930 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:56.029 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.507+0000 7f6425e40700 1 --2- 172.21.15.70:0/3960259938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6420108ea0 0x7f642007eb50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:45:56.029 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.507+0000 7f6425e40700 1 -- 172.21.15.70:0/3960259938 >> 172.21.15.70:0/3960259938 conn(0x7f64201044e0 msgr2=0x7f6420104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:45:56.029 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.507+0000 7f6425e40700 1 -- 172.21.15.70:0/3960259938 shutdown_connections 2024-06-04T04:45:56.029 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:55.507+0000 7f6425e40700 1 -- 172.21.15.70:0/3960259938 wait complete. 2024-06-04T04:45:56.029 INFO:teuthology.orchestra.run.smithi070.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-06-04T04:45:56.029 INFO:teuthology.orchestra.run.smithi070.stdout:Adding key to root@localhost authorized_keys... 2024-06-04T04:45:56.029 INFO:teuthology.orchestra.run.smithi070.stdout:Adding host smithi070... 2024-06-04T04:45:56.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:56 smithi070 ceph-mon[32631]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:45:57.563 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:57 smithi070 ceph-mon[32631]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi070", "addr": "172.21.15.70", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:45:57.564 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:45:57 smithi070 ceph-mon[32631]: Deploying cephadm binary to smithi070 2024-06-04T04:46:01.374 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout Added host 'smithi070' with addr '172.21.15.70' 2024-06-04T04:46:01.374 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.452+0000 7f9ffc9d4700 1 Processor -- start 2024-06-04T04:46:01.374 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.452+0000 7f9ffc9d4700 1 -- start start 2024-06-04T04:46:01.375 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.452+0000 7f9ffc9d4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ff8109980 0x7f9ff810bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:01.375 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.452+0000 7f9ffc9d4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff807ce10 con 0x7f9ff8109980 2024-06-04T04:46:01.375 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.452+0000 7f9ff659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ff8109980 0x7f9ff810bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:01.375 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.452+0000 7f9ff659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ff8109980 0x7f9ff810bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44494/0 (socket says 172.21.15.70:44494) 2024-06-04T04:46:01.375 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.452+0000 7f9ff659c700 1 -- 172.21.15.70:0/3971562661 learned_addr learned my addr 172.21.15.70:0/3971562661 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:01.375 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.453+0000 7f9ff659c700 1 -- 172.21.15.70:0/3971562661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff810c2b0 con 0x7f9ff8109980 2024-06-04T04:46:01.375 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.453+0000 7f9ff659c700 1 --2- 172.21.15.70:0/3971562661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ff8109980 0x7f9ff810bd70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9fe80098a0 tx=0x7f9fe8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929463444db32133 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:01.375 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.453+0000 7f9ff559a700 1 -- 172.21.15.70:0/3971562661 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fe8004160 con 0x7f9ff8109980 2024-06-04T04:46:01.375 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.453+0000 7f9ff559a700 1 -- 172.21.15.70:0/3971562661 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f9fe8004300 con 0x7f9ff8109980 2024-06-04T04:46:01.375 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.453+0000 7f9ff559a700 1 -- 172.21.15.70:0/3971562661 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fe80119e0 con 0x7f9ff8109980 2024-06-04T04:46:01.375 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.453+0000 7f9ffc9d4700 1 -- 172.21.15.70:0/3971562661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ff8109980 msgr2=0x7f9ff810bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:01.375 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.453+0000 7f9ffc9d4700 1 --2- 172.21.15.70:0/3971562661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ff8109980 0x7f9ff810bd70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9fe80098a0 tx=0x7f9fe8009bb0 comp rx=0 tx=0).stop 2024-06-04T04:46:01.376 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.454+0000 7f9ffc9d4700 1 -- 172.21.15.70:0/3971562661 shutdown_connections 2024-06-04T04:46:01.376 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.454+0000 7f9ffc9d4700 1 --2- 172.21.15.70:0/3971562661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ff8109980 0x7f9ff810bd70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:01.376 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.454+0000 7f9ffc9d4700 1 -- 172.21.15.70:0/3971562661 >> 172.21.15.70:0/3971562661 conn(0x7f9ff81032f0 msgr2=0x7f9ff8105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:01.376 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.454+0000 7f9ffc9d4700 1 -- 172.21.15.70:0/3971562661 shutdown_connections 2024-06-04T04:46:01.376 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.454+0000 7f9ffc9d4700 1 -- 172.21.15.70:0/3971562661 wait complete. 2024-06-04T04:46:01.376 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.454+0000 7f9ffc9d4700 1 Processor -- start 2024-06-04T04:46:01.376 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.454+0000 7f9ffc9d4700 1 -- start start 2024-06-04T04:46:01.376 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.455+0000 7f9ffc9d4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ff8109980 0x7f9ff8109200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:01.376 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.455+0000 7f9ffc9d4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff810d1b0 con 0x7f9ff8109980 2024-06-04T04:46:01.376 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.455+0000 7f9ff659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ff8109980 0x7f9ff8109200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:01.376 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.455+0000 7f9ff659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ff8109980 0x7f9ff8109200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44498/0 (socket says 172.21.15.70:44498) 2024-06-04T04:46:01.376 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.455+0000 7f9ff659c700 1 -- 172.21.15.70:0/4279899199 learned_addr learned my addr 172.21.15.70:0/4279899199 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:01.376 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.455+0000 7f9ff659c700 1 -- 172.21.15.70:0/4279899199 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff8109740 con 0x7f9ff8109980 2024-06-04T04:46:01.377 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.455+0000 7f9ff659c700 1 --2- 172.21.15.70:0/4279899199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ff8109980 0x7f9ff8109200 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9fe80119b0 tx=0x7f9fe8011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:01.377 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.455+0000 7f9fe77fe700 1 -- 172.21.15.70:0/4279899199 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fe80039f0 con 0x7f9ff8109980 2024-06-04T04:46:01.377 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.455+0000 7f9ffc9d4700 1 -- 172.21.15.70:0/4279899199 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ff8107930 con 0x7f9ff8109980 2024-06-04T04:46:01.377 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.455+0000 7f9fe77fe700 1 -- 172.21.15.70:0/4279899199 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f9fe8003b90 con 0x7f9ff8109980 2024-06-04T04:46:01.377 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.455+0000 7f9fe77fe700 1 -- 172.21.15.70:0/4279899199 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fe801a620 con 0x7f9ff8109980 2024-06-04T04:46:01.377 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.455+0000 7f9ffc9d4700 1 -- 172.21.15.70:0/4279899199 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ff8107e70 con 0x7f9ff8109980 2024-06-04T04:46:01.377 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.456+0000 7f9fe77fe700 1 -- 172.21.15.70:0/4279899199 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f9fe801a920 con 0x7f9ff8109980 2024-06-04T04:46:01.377 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.456+0000 7f9ffc9d4700 1 -- 172.21.15.70:0/4279899199 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ff8064ba0 con 0x7f9ff8109980 2024-06-04T04:46:01.377 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.456+0000 7f9fe77fe700 1 --2- 172.21.15.70:0/4279899199 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f9fe003e420 0x7f9fe00408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:01.377 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.456+0000 7f9fe77fe700 1 -- 172.21.15.70:0/4279899199 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9fe8053c10 con 0x7f9ff8109980 2024-06-04T04:46:01.377 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.457+0000 7f9ff5d9b700 1 --2- 172.21.15.70:0/4279899199 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f9fe003e420 0x7f9fe00408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:01.377 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.457+0000 7f9ff5d9b700 1 --2- 172.21.15.70:0/4279899199 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f9fe003e420 0x7f9fe00408e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9fec006fd0 tx=0x7f9fec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:01.377 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.461+0000 7f9fe77fe700 1 -- 172.21.15.70:0/4279899199 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9fe802c350 con 0x7f9ff8109980 2024-06-04T04:46:01.378 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:45:56.673+0000 7f9ffc9d4700 1 -- 172.21.15.70:0/4279899199 --> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi070", "addr": "172.21.15.70", "target": ["mon-mgr", ""]}) -- 0x7f9ff81044e0 con 0x7f9fe003e420 2024-06-04T04:46:01.378 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:00.812+0000 7f9fe77fe700 1 -- 172.21.15.70:0/4279899199 <== mgr.14120 v2:172.21.15.70:6800/1353782413 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f9ff81044e0 con 0x7f9fe003e420 2024-06-04T04:46:01.378 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:00.814+0000 7f9ffc9d4700 1 -- 172.21.15.70:0/4279899199 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f9fe003e420 msgr2=0x7f9fe00408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:01.378 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:00.814+0000 7f9ffc9d4700 1 --2- 172.21.15.70:0/4279899199 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f9fe003e420 0x7f9fe00408e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9fec006fd0 tx=0x7f9fec006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:01.378 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:00.814+0000 7f9ffc9d4700 1 -- 172.21.15.70:0/4279899199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ff8109980 msgr2=0x7f9ff8109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:01.378 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:00.814+0000 7f9ffc9d4700 1 --2- 172.21.15.70:0/4279899199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ff8109980 0x7f9ff8109200 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9fe80119b0 tx=0x7f9fe8011a90 comp rx=0 tx=0).stop 2024-06-04T04:46:01.378 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:00.814+0000 7f9ffc9d4700 1 -- 172.21.15.70:0/4279899199 shutdown_connections 2024-06-04T04:46:01.378 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:00.814+0000 7f9ffc9d4700 1 --2- 172.21.15.70:0/4279899199 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f9fe003e420 0x7f9fe00408e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:01.378 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:00.814+0000 7f9ffc9d4700 1 --2- 172.21.15.70:0/4279899199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ff8109980 0x7f9ff8109200 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:01.378 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:00.814+0000 7f9ffc9d4700 1 -- 172.21.15.70:0/4279899199 >> 172.21.15.70:0/4279899199 conn(0x7f9ff81032f0 msgr2=0x7f9ff8103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:01.378 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:00.815+0000 7f9ffc9d4700 1 -- 172.21.15.70:0/4279899199 shutdown_connections 2024-06-04T04:46:01.378 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:00.815+0000 7f9ffc9d4700 1 -- 172.21.15.70:0/4279899199 wait complete. 2024-06-04T04:46:01.379 INFO:teuthology.orchestra.run.smithi070.stdout:Deploying unmanaged mon service... 2024-06-04T04:46:02.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:01 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' 2024-06-04T04:46:02.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:01 smithi070 ceph-mon[32631]: Added host smithi070 2024-06-04T04:46:02.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:01 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:46:03.564 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:03 smithi070 ceph-mon[32631]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:46:03.564 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:03 smithi070 ceph-mon[32631]: Saving service mon spec with placement count:5 2024-06-04T04:46:03.565 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:03 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' 2024-06-04T04:46:03.565 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:03 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' 2024-06-04T04:46:03.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-06-04T04:46:03.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.105+0000 7f489d68b700 1 Processor -- start 2024-06-04T04:46:03.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.105+0000 7f489d68b700 1 -- start start 2024-06-04T04:46:03.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.106+0000 7f489d68b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4898108260 0x7f4898108680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:03.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.106+0000 7f489d68b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4898108c50 con 0x7f4898108260 2024-06-04T04:46:03.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.106+0000 7f4896ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4898108260 0x7f4898108680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:03.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.106+0000 7f4896ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4898108260 0x7f4898108680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44502/0 (socket says 172.21.15.70:44502) 2024-06-04T04:46:03.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.106+0000 7f4896ffd700 1 -- 172.21.15.70:0/1139722358 learned_addr learned my addr 172.21.15.70:0/1139722358 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:03.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.106+0000 7f4896ffd700 1 -- 172.21.15.70:0/1139722358 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4898108f80 con 0x7f4898108260 2024-06-04T04:46:03.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.106+0000 7f4896ffd700 1 --2- 172.21.15.70:0/1139722358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4898108260 0x7f4898108680 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f488800bb40 tx=0x7f488800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11204b6479ea8bc5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:03.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.106+0000 7f4895ffb700 1 -- 172.21.15.70:0/1139722358 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f488800dbe0 con 0x7f4898108260 2024-06-04T04:46:03.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.106+0000 7f4895ffb700 1 -- 172.21.15.70:0/1139722358 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f48880044a0 con 0x7f4898108260 2024-06-04T04:46:03.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.106+0000 7f4895ffb700 1 -- 172.21.15.70:0/1139722358 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4888003e30 con 0x7f4898108260 2024-06-04T04:46:03.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.107+0000 7f489d68b700 1 -- 172.21.15.70:0/1139722358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4898108260 msgr2=0x7f4898108680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:03.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.107+0000 7f489d68b700 1 --2- 172.21.15.70:0/1139722358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4898108260 0x7f4898108680 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f488800bb40 tx=0x7f488800be50 comp rx=0 tx=0).stop 2024-06-04T04:46:03.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.107+0000 7f489d68b700 1 -- 172.21.15.70:0/1139722358 shutdown_connections 2024-06-04T04:46:03.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.107+0000 7f489d68b700 1 --2- 172.21.15.70:0/1139722358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4898108260 0x7f4898108680 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:03.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.107+0000 7f489d68b700 1 -- 172.21.15.70:0/1139722358 >> 172.21.15.70:0/1139722358 conn(0x7f4898102ae0 msgr2=0x7f4898104f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:03.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.107+0000 7f489d68b700 1 -- 172.21.15.70:0/1139722358 shutdown_connections 2024-06-04T04:46:03.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.107+0000 7f489d68b700 1 -- 172.21.15.70:0/1139722358 wait complete. 2024-06-04T04:46:03.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.108+0000 7f489d68b700 1 Processor -- start 2024-06-04T04:46:03.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.108+0000 7f489d68b700 1 -- start start 2024-06-04T04:46:03.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.108+0000 7f489d68b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4898108260 0x7f489819ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:03.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.108+0000 7f489d68b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4898106dc0 con 0x7f4898108260 2024-06-04T04:46:03.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.108+0000 7f4896ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4898108260 0x7f489819ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:03.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.108+0000 7f4896ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4898108260 0x7f489819ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44512/0 (socket says 172.21.15.70:44512) 2024-06-04T04:46:03.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.108+0000 7f4896ffd700 1 -- 172.21.15.70:0/2711535347 learned_addr learned my addr 172.21.15.70:0/2711535347 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:03.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.108+0000 7f4896ffd700 1 -- 172.21.15.70:0/2711535347 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f489819b350 con 0x7f4898108260 2024-06-04T04:46:03.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.108+0000 7f4896ffd700 1 --2- 172.21.15.70:0/2711535347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4898108260 0x7f489819ae10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4888003bc0 tx=0x7f4888003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:03.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.108+0000 7f488ffff700 1 -- 172.21.15.70:0/2711535347 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4888027070 con 0x7f4898108260 2024-06-04T04:46:03.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.109+0000 7f488ffff700 1 -- 172.21.15.70:0/2711535347 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f48880109f0 con 0x7f4898108260 2024-06-04T04:46:03.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.109+0000 7f489d68b700 1 -- 172.21.15.70:0/2711535347 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f489819b5e0 con 0x7f4898108260 2024-06-04T04:46:03.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.109+0000 7f488ffff700 1 -- 172.21.15.70:0/2711535347 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48880229c0 con 0x7f4898108260 2024-06-04T04:46:03.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.109+0000 7f489d68b700 1 -- 172.21.15.70:0/2711535347 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f489819e2d0 con 0x7f4898108260 2024-06-04T04:46:03.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.109+0000 7f488ffff700 1 -- 172.21.15.70:0/2711535347 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f488802a070 con 0x7f4898108260 2024-06-04T04:46:03.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.109+0000 7f489d68b700 1 -- 172.21.15.70:0/2711535347 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4898064ba0 con 0x7f4898108260 2024-06-04T04:46:03.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.110+0000 7f488ffff700 1 --2- 172.21.15.70:0/2711535347 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f488403e470 0x7f4884040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:03.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.110+0000 7f488ffff700 1 -- 172.21.15.70:0/2711535347 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f488801f070 con 0x7f4898108260 2024-06-04T04:46:03.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.110+0000 7f48967fc700 1 --2- 172.21.15.70:0/2711535347 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f488403e470 0x7f4884040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:03.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.110+0000 7f48967fc700 1 --2- 172.21.15.70:0/2711535347 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f488403e470 0x7f4884040930 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4880006fd0 tx=0x7f4880006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:03.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.115+0000 7f488ffff700 1 -- 172.21.15.70:0/2711535347 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4888015070 con 0x7f4898108260 2024-06-04T04:46:03.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.341+0000 7f489d68b700 1 -- 172.21.15.70:0/2711535347 --> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f4898063980 con 0x7f488403e470 2024-06-04T04:46:03.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.345+0000 7f488ffff700 1 -- 172.21.15.70:0/2711535347 <== mgr.14120 v2:172.21.15.70:6800/1353782413 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4898063980 con 0x7f488403e470 2024-06-04T04:46:03.826 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.348+0000 7f489d68b700 1 -- 172.21.15.70:0/2711535347 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f488403e470 msgr2=0x7f4884040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:03.826 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.348+0000 7f489d68b700 1 --2- 172.21.15.70:0/2711535347 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f488403e470 0x7f4884040930 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4880006fd0 tx=0x7f4880006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:03.826 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.348+0000 7f489d68b700 1 -- 172.21.15.70:0/2711535347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4898108260 msgr2=0x7f489819ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:03.826 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.348+0000 7f489d68b700 1 --2- 172.21.15.70:0/2711535347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4898108260 0x7f489819ae10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4888003bc0 tx=0x7f4888003f80 comp rx=0 tx=0).stop 2024-06-04T04:46:03.826 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.348+0000 7f489d68b700 1 -- 172.21.15.70:0/2711535347 shutdown_connections 2024-06-04T04:46:03.826 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.348+0000 7f489d68b700 1 --2- 172.21.15.70:0/2711535347 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f488403e470 0x7f4884040930 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:03.826 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.348+0000 7f489d68b700 1 --2- 172.21.15.70:0/2711535347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4898108260 0x7f489819ae10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:03.826 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.348+0000 7f489d68b700 1 -- 172.21.15.70:0/2711535347 >> 172.21.15.70:0/2711535347 conn(0x7f4898102ae0 msgr2=0x7f4898191450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:03.826 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.348+0000 7f489d68b700 1 -- 172.21.15.70:0/2711535347 shutdown_connections 2024-06-04T04:46:03.826 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:02.348+0000 7f489d68b700 1 -- 172.21.15.70:0/2711535347 wait complete. 2024-06-04T04:46:03.826 INFO:teuthology.orchestra.run.smithi070.stdout:Deploying unmanaged mgr service... 2024-06-04T04:46:05.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:04 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' 2024-06-04T04:46:05.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:04 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' 2024-06-04T04:46:05.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-06-04T04:46:05.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.246+0000 7fe0a8621700 1 Processor -- start 2024-06-04T04:46:05.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.246+0000 7fe0a8621700 1 -- start start 2024-06-04T04:46:05.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.246+0000 7fe0a8621700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0a0107150 0x7fe0a0107570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:05.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.246+0000 7fe0a8621700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a0107b40 con 0x7fe0a0107150 2024-06-04T04:46:05.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.246+0000 7fe0a63bd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0a0107150 0x7fe0a0107570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:05.689 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.246+0000 7fe0a63bd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0a0107150 0x7fe0a0107570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44520/0 (socket says 172.21.15.70:44520) 2024-06-04T04:46:05.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.246+0000 7fe0a63bd700 1 -- 172.21.15.70:0/2086788290 learned_addr learned my addr 172.21.15.70:0/2086788290 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:05.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.247+0000 7fe0a63bd700 1 -- 172.21.15.70:0/2086788290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a010a0e0 con 0x7fe0a0107150 2024-06-04T04:46:05.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.247+0000 7fe0a63bd700 1 --2- 172.21.15.70:0/2086788290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0a0107150 0x7fe0a0107570 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe09c0098a0 tx=0x7fe09c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca476258fff958b4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:05.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.247+0000 7fe0a53bb700 1 -- 172.21.15.70:0/2086788290 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe09c004160 con 0x7fe0a0107150 2024-06-04T04:46:05.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.247+0000 7fe0a53bb700 1 -- 172.21.15.70:0/2086788290 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe09c004300 con 0x7fe0a0107150 2024-06-04T04:46:05.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.247+0000 7fe0a53bb700 1 -- 172.21.15.70:0/2086788290 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe09c011960 con 0x7fe0a0107150 2024-06-04T04:46:05.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.248+0000 7fe0a8621700 1 -- 172.21.15.70:0/2086788290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0a0107150 msgr2=0x7fe0a0107570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:05.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.248+0000 7fe0a8621700 1 --2- 172.21.15.70:0/2086788290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0a0107150 0x7fe0a0107570 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe09c0098a0 tx=0x7fe09c009bb0 comp rx=0 tx=0).stop 2024-06-04T04:46:05.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.248+0000 7fe0a8621700 1 -- 172.21.15.70:0/2086788290 shutdown_connections 2024-06-04T04:46:05.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.248+0000 7fe0a8621700 1 --2- 172.21.15.70:0/2086788290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0a0107150 0x7fe0a0107570 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:05.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.248+0000 7fe0a8621700 1 -- 172.21.15.70:0/2086788290 >> 172.21.15.70:0/2086788290 conn(0x7fe0a0102ae0 msgr2=0x7fe0a0104f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:05.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.248+0000 7fe0a8621700 1 -- 172.21.15.70:0/2086788290 shutdown_connections 2024-06-04T04:46:05.690 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.248+0000 7fe0a8621700 1 -- 172.21.15.70:0/2086788290 wait complete. 2024-06-04T04:46:05.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.248+0000 7fe0a8621700 1 Processor -- start 2024-06-04T04:46:05.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.248+0000 7fe0a8621700 1 -- start start 2024-06-04T04:46:05.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.249+0000 7fe0a8621700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0a0107150 0x7fe0a0109910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:05.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.249+0000 7fe0a8621700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a010d310 con 0x7fe0a0107150 2024-06-04T04:46:05.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.249+0000 7fe0a63bd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0a0107150 0x7fe0a0109910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:05.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.249+0000 7fe0a63bd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0a0107150 0x7fe0a0109910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44526/0 (socket says 172.21.15.70:44526) 2024-06-04T04:46:05.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.249+0000 7fe0a63bd700 1 -- 172.21.15.70:0/4094422163 learned_addr learned my addr 172.21.15.70:0/4094422163 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:05.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.249+0000 7fe0a63bd700 1 -- 172.21.15.70:0/4094422163 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a0109e50 con 0x7fe0a0107150 2024-06-04T04:46:05.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.249+0000 7fe0a63bd700 1 --2- 172.21.15.70:0/4094422163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0a0107150 0x7fe0a0109910 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe09c011670 tx=0x7fe09c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:05.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.249+0000 7fe0977fe700 1 -- 172.21.15.70:0/4094422163 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe09c003aa0 con 0x7fe0a0107150 2024-06-04T04:46:05.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.249+0000 7fe0977fe700 1 -- 172.21.15.70:0/4094422163 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe09c01b8d0 con 0x7fe0a0107150 2024-06-04T04:46:05.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.249+0000 7fe0a8621700 1 -- 172.21.15.70:0/4094422163 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a0108090 con 0x7fe0a0107150 2024-06-04T04:46:05.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.250+0000 7fe0977fe700 1 -- 172.21.15.70:0/4094422163 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe09c0197f0 con 0x7fe0a0107150 2024-06-04T04:46:05.691 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.250+0000 7fe0a8621700 1 -- 172.21.15.70:0/4094422163 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a0108570 con 0x7fe0a0107150 2024-06-04T04:46:05.692 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.250+0000 7fe0a8621700 1 -- 172.21.15.70:0/4094422163 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0a010d490 con 0x7fe0a0107150 2024-06-04T04:46:05.692 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.251+0000 7fe0977fe700 1 -- 172.21.15.70:0/4094422163 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fe09c01b440 con 0x7fe0a0107150 2024-06-04T04:46:05.692 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.251+0000 7fe0977fe700 1 --2- 172.21.15.70:0/4094422163 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fe08c03e420 0x7fe08c0408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:05.692 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.251+0000 7fe0977fe700 1 -- 172.21.15.70:0/4094422163 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe09c053e70 con 0x7fe0a0107150 2024-06-04T04:46:05.692 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.251+0000 7fe0a5bbc700 1 --2- 172.21.15.70:0/4094422163 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fe08c03e420 0x7fe08c0408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:05.692 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.251+0000 7fe0a5bbc700 1 --2- 172.21.15.70:0/4094422163 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fe08c03e420 0x7fe08c0408e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe090006fd0 tx=0x7fe090006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:05.692 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.260+0000 7fe0977fe700 1 -- 172.21.15.70:0/4094422163 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe09c01f030 con 0x7fe0a0107150 2024-06-04T04:46:05.692 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.512+0000 7fe0a8621700 1 -- 172.21.15.70:0/4094422163 --> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fe0a0196710 con 0x7fe08c03e420 2024-06-04T04:46:05.692 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.517+0000 7fe0977fe700 1 -- 172.21.15.70:0/4094422163 <== mgr.14120 v2:172.21.15.70:6800/1353782413 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe0a0196710 con 0x7fe08c03e420 2024-06-04T04:46:05.692 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.519+0000 7fe0a8621700 1 -- 172.21.15.70:0/4094422163 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fe08c03e420 msgr2=0x7fe08c0408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:05.692 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.519+0000 7fe0a8621700 1 --2- 172.21.15.70:0/4094422163 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fe08c03e420 0x7fe08c0408e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe090006fd0 tx=0x7fe090006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:05.692 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.519+0000 7fe0a8621700 1 -- 172.21.15.70:0/4094422163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0a0107150 msgr2=0x7fe0a0109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:05.692 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.519+0000 7fe0a8621700 1 --2- 172.21.15.70:0/4094422163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0a0107150 0x7fe0a0109910 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe09c011670 tx=0x7fe09c011a30 comp rx=0 tx=0).stop 2024-06-04T04:46:05.693 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.519+0000 7fe0a8621700 1 -- 172.21.15.70:0/4094422163 shutdown_connections 2024-06-04T04:46:05.693 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.519+0000 7fe0a8621700 1 --2- 172.21.15.70:0/4094422163 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fe08c03e420 0x7fe08c0408e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:05.693 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.519+0000 7fe0a8621700 1 --2- 172.21.15.70:0/4094422163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0a0107150 0x7fe0a0109910 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:05.693 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.519+0000 7fe0a8621700 1 -- 172.21.15.70:0/4094422163 >> 172.21.15.70:0/4094422163 conn(0x7fe0a0102ae0 msgr2=0x7fe0a0103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:05.693 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.519+0000 7fe0a8621700 1 -- 172.21.15.70:0/4094422163 shutdown_connections 2024-06-04T04:46:05.693 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:04.519+0000 7fe0a8621700 1 -- 172.21.15.70:0/4094422163 wait complete. 2024-06-04T04:46:06.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:06 smithi070 ceph-mon[32631]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:46:06.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:06 smithi070 ceph-mon[32631]: Saving service mgr spec with placement count:2 2024-06-04T04:46:06.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:06 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' 2024-06-04T04:46:06.961 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.156+0000 7fa22589f700 1 Processor -- start 2024-06-04T04:46:06.962 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.156+0000 7fa22589f700 1 -- start start 2024-06-04T04:46:06.962 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.156+0000 7fa22589f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa220108ea0 0x7fa2201092c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:06.962 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.156+0000 7fa22589f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa220109800 con 0x7fa220108ea0 2024-06-04T04:46:06.962 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.156+0000 7fa21effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa220108ea0 0x7fa2201092c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:06.962 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.157+0000 7fa21effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa220108ea0 0x7fa2201092c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60934/0 (socket says 172.21.15.70:60934) 2024-06-04T04:46:06.962 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.157+0000 7fa21effd700 1 -- 172.21.15.70:0/4294756035 learned_addr learned my addr 172.21.15.70:0/4294756035 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:06.962 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.157+0000 7fa21effd700 1 -- 172.21.15.70:0/4294756035 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa220109fb0 con 0x7fa220108ea0 2024-06-04T04:46:06.962 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.157+0000 7fa21effd700 1 --2- 172.21.15.70:0/4294756035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa220108ea0 0x7fa2201092c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa2100098a0 tx=0x7fa210009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1df0ba40d7213a4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:06.962 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.157+0000 7fa21dffb700 1 -- 172.21.15.70:0/4294756035 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa210004160 con 0x7fa220108ea0 2024-06-04T04:46:06.962 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.157+0000 7fa21dffb700 1 -- 172.21.15.70:0/4294756035 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa210004300 con 0x7fa220108ea0 2024-06-04T04:46:06.963 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.157+0000 7fa21dffb700 1 -- 172.21.15.70:0/4294756035 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa210011960 con 0x7fa220108ea0 2024-06-04T04:46:06.963 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.158+0000 7fa22589f700 1 -- 172.21.15.70:0/4294756035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa220108ea0 msgr2=0x7fa2201092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:06.963 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.158+0000 7fa22589f700 1 --2- 172.21.15.70:0/4294756035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa220108ea0 0x7fa2201092c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa2100098a0 tx=0x7fa210009bb0 comp rx=0 tx=0).stop 2024-06-04T04:46:06.963 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.158+0000 7fa22589f700 1 -- 172.21.15.70:0/4294756035 shutdown_connections 2024-06-04T04:46:06.963 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.158+0000 7fa22589f700 1 --2- 172.21.15.70:0/4294756035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa220108ea0 0x7fa2201092c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:06.963 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.158+0000 7fa22589f700 1 -- 172.21.15.70:0/4294756035 >> 172.21.15.70:0/4294756035 conn(0x7fa220104460 msgr2=0x7fa220106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:06.963 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.158+0000 7fa22589f700 1 -- 172.21.15.70:0/4294756035 shutdown_connections 2024-06-04T04:46:06.963 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.158+0000 7fa22589f700 1 -- 172.21.15.70:0/4294756035 wait complete. 2024-06-04T04:46:06.963 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.159+0000 7fa22589f700 1 Processor -- start 2024-06-04T04:46:06.963 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.159+0000 7fa22589f700 1 -- start start 2024-06-04T04:46:06.963 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.159+0000 7fa22589f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa220108ea0 0x7fa22007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:06.963 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.159+0000 7fa22589f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa22010d270 con 0x7fa220108ea0 2024-06-04T04:46:06.963 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.159+0000 7fa21effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa220108ea0 0x7fa22007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:06.964 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.159+0000 7fa21effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa220108ea0 0x7fa22007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60948/0 (socket says 172.21.15.70:60948) 2024-06-04T04:46:06.964 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.159+0000 7fa21effd700 1 -- 172.21.15.70:0/3238350664 learned_addr learned my addr 172.21.15.70:0/3238350664 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:06.964 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.160+0000 7fa21effd700 1 -- 172.21.15.70:0/3238350664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa22007f090 con 0x7fa220108ea0 2024-06-04T04:46:06.964 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.160+0000 7fa21effd700 1 --2- 172.21.15.70:0/3238350664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa220108ea0 0x7fa22007eb50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa210003e50 tx=0x7fa210011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:06.964 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.160+0000 7fa22489d700 1 -- 172.21.15.70:0/3238350664 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa210003a60 con 0x7fa220108ea0 2024-06-04T04:46:06.964 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.160+0000 7fa22489d700 1 -- 172.21.15.70:0/3238350664 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa21001b8d0 con 0x7fa220108ea0 2024-06-04T04:46:06.964 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.160+0000 7fa22589f700 1 -- 172.21.15.70:0/3238350664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa22007d280 con 0x7fa220108ea0 2024-06-04T04:46:06.964 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.160+0000 7fa22489d700 1 -- 172.21.15.70:0/3238350664 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2100237d0 con 0x7fa220108ea0 2024-06-04T04:46:06.964 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.160+0000 7fa22589f700 1 -- 172.21.15.70:0/3238350664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa22007d7c0 con 0x7fa220108ea0 2024-06-04T04:46:06.964 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.161+0000 7fa22589f700 1 -- 172.21.15.70:0/3238350664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa220064ba0 con 0x7fa220108ea0 2024-06-04T04:46:06.964 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.161+0000 7fa22489d700 1 -- 172.21.15.70:0/3238350664 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fa21002c3f0 con 0x7fa220108ea0 2024-06-04T04:46:06.964 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.161+0000 7fa22489d700 1 --2- 172.21.15.70:0/3238350664 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fa20803e470 0x7fa208040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:06.964 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.162+0000 7fa22489d700 1 -- 172.21.15.70:0/3238350664 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa210054030 con 0x7fa220108ea0 2024-06-04T04:46:06.965 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.162+0000 7fa21e7fc700 1 --2- 172.21.15.70:0/3238350664 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fa20803e470 0x7fa208040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:06.965 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.162+0000 7fa21e7fc700 1 --2- 172.21.15.70:0/3238350664 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fa20803e470 0x7fa208040930 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa214006fd0 tx=0x7fa214006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:06.965 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.166+0000 7fa22489d700 1 -- 172.21.15.70:0/3238350664 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa21001f030 con 0x7fa220108ea0 2024-06-04T04:46:06.965 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.384+0000 7fa22589f700 1 -- 172.21.15.70:0/3238350664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fa22007dc50 con 0x7fa220108ea0 2024-06-04T04:46:06.965 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.386+0000 7fa22489d700 1 -- 172.21.15.70:0/3238350664 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fa210017e90 con 0x7fa220108ea0 2024-06-04T04:46:06.965 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.389+0000 7fa22589f700 1 -- 172.21.15.70:0/3238350664 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fa20803e470 msgr2=0x7fa208040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:06.965 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.389+0000 7fa22589f700 1 --2- 172.21.15.70:0/3238350664 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fa20803e470 0x7fa208040930 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa214006fd0 tx=0x7fa214006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:06.965 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.389+0000 7fa22589f700 1 -- 172.21.15.70:0/3238350664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa220108ea0 msgr2=0x7fa22007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:06.965 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.389+0000 7fa22589f700 1 --2- 172.21.15.70:0/3238350664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa220108ea0 0x7fa22007eb50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa210003e50 tx=0x7fa210011a30 comp rx=0 tx=0).stop 2024-06-04T04:46:06.965 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.389+0000 7fa22589f700 1 -- 172.21.15.70:0/3238350664 shutdown_connections 2024-06-04T04:46:06.965 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.389+0000 7fa22589f700 1 --2- 172.21.15.70:0/3238350664 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7fa20803e470 0x7fa208040930 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:06.965 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.389+0000 7fa22589f700 1 --2- 172.21.15.70:0/3238350664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa220108ea0 0x7fa22007eb50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:06.965 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.389+0000 7fa22589f700 1 -- 172.21.15.70:0/3238350664 >> 172.21.15.70:0/3238350664 conn(0x7fa220104460 msgr2=0x7fa220104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:06.966 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.389+0000 7fa22589f700 1 -- 172.21.15.70:0/3238350664 shutdown_connections 2024-06-04T04:46:06.966 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:06.389+0000 7fa22589f700 1 -- 172.21.15.70:0/3238350664 wait complete. 2024-06-04T04:46:07.594 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:07 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3238350664' entity='client.admin' 2024-06-04T04:46:07.594 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:07 smithi070 ceph-mon[32631]: from='mgr.14120 172.21.15.70:0/3089107753' entity='mgr.x' 2024-06-04T04:46:08.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:08 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1049790520' entity='client.admin' 2024-06-04T04:46:08.916 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.373+0000 7f133cb47700 1 Processor -- start 2024-06-04T04:46:08.916 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.373+0000 7f133cb47700 1 -- start start 2024-06-04T04:46:08.916 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.373+0000 7f133cb47700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1338107150 0x7f1338107570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:08.916 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.373+0000 7f133cb47700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1338107b40 con 0x7f1338107150 2024-06-04T04:46:08.916 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.374+0000 7f133659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1338107150 0x7f1338107570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:08.917 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.374+0000 7f133659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1338107150 0x7f1338107570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60956/0 (socket says 172.21.15.70:60956) 2024-06-04T04:46:08.917 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.374+0000 7f133659c700 1 -- 172.21.15.70:0/1840308227 learned_addr learned my addr 172.21.15.70:0/1840308227 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:08.917 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.374+0000 7f133659c700 1 -- 172.21.15.70:0/1840308227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f133810a0e0 con 0x7f1338107150 2024-06-04T04:46:08.917 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.374+0000 7f133659c700 1 --2- 172.21.15.70:0/1840308227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1338107150 0x7f1338107570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f13200098a0 tx=0x7f1320009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6414874b715f92 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:08.917 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.374+0000 7f133559a700 1 -- 172.21.15.70:0/1840308227 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1320004160 con 0x7f1338107150 2024-06-04T04:46:08.917 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.374+0000 7f133559a700 1 -- 172.21.15.70:0/1840308227 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1320004300 con 0x7f1338107150 2024-06-04T04:46:08.917 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.374+0000 7f133559a700 1 -- 172.21.15.70:0/1840308227 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1320011960 con 0x7f1338107150 2024-06-04T04:46:08.917 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.375+0000 7f133cb47700 1 -- 172.21.15.70:0/1840308227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1338107150 msgr2=0x7f1338107570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:08.917 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.375+0000 7f133cb47700 1 --2- 172.21.15.70:0/1840308227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1338107150 0x7f1338107570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f13200098a0 tx=0x7f1320009bb0 comp rx=0 tx=0).stop 2024-06-04T04:46:08.917 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.375+0000 7f133cb47700 1 -- 172.21.15.70:0/1840308227 shutdown_connections 2024-06-04T04:46:08.917 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.375+0000 7f133cb47700 1 --2- 172.21.15.70:0/1840308227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1338107150 0x7f1338107570 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:08.917 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.375+0000 7f133cb47700 1 -- 172.21.15.70:0/1840308227 >> 172.21.15.70:0/1840308227 conn(0x7f1338102ae0 msgr2=0x7f1338104f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:08.917 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.375+0000 7f133cb47700 1 -- 172.21.15.70:0/1840308227 shutdown_connections 2024-06-04T04:46:08.917 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.375+0000 7f133cb47700 1 -- 172.21.15.70:0/1840308227 wait complete. 2024-06-04T04:46:08.918 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.376+0000 7f133cb47700 1 Processor -- start 2024-06-04T04:46:08.918 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.376+0000 7f133cb47700 1 -- start start 2024-06-04T04:46:08.918 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.376+0000 7f133cb47700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1338107150 0x7f1338109910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:08.918 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.376+0000 7f133cb47700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f133810d310 con 0x7f1338107150 2024-06-04T04:46:08.918 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.376+0000 7f133659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1338107150 0x7f1338109910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:08.918 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.376+0000 7f133659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1338107150 0x7f1338109910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60968/0 (socket says 172.21.15.70:60968) 2024-06-04T04:46:08.918 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.376+0000 7f133659c700 1 -- 172.21.15.70:0/1049790520 learned_addr learned my addr 172.21.15.70:0/1049790520 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:08.918 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.377+0000 7f133659c700 1 -- 172.21.15.70:0/1049790520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1338109e50 con 0x7f1338107150 2024-06-04T04:46:08.918 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.377+0000 7f133659c700 1 --2- 172.21.15.70:0/1049790520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1338107150 0x7f1338109910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1320011670 tx=0x7f1320011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:08.918 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.377+0000 7f132f7fe700 1 -- 172.21.15.70:0/1049790520 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1320003aa0 con 0x7f1338107150 2024-06-04T04:46:08.918 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.377+0000 7f132f7fe700 1 -- 172.21.15.70:0/1049790520 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f132001b8d0 con 0x7f1338107150 2024-06-04T04:46:08.918 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.377+0000 7f132f7fe700 1 -- 172.21.15.70:0/1049790520 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13200197f0 con 0x7f1338107150 2024-06-04T04:46:08.919 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.377+0000 7f133cb47700 1 -- 172.21.15.70:0/1049790520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1338108090 con 0x7f1338107150 2024-06-04T04:46:08.919 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.377+0000 7f133cb47700 1 -- 172.21.15.70:0/1049790520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1338108570 con 0x7f1338107150 2024-06-04T04:46:08.919 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.378+0000 7f132f7fe700 1 -- 172.21.15.70:0/1049790520 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f1320019990 con 0x7f1338107150 2024-06-04T04:46:08.919 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.378+0000 7f132f7fe700 1 --2- 172.21.15.70:0/1049790520 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f132403e470 0x7f1324040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:08.919 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.378+0000 7f133cb47700 1 -- 172.21.15.70:0/1049790520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f133810d490 con 0x7f1338107150 2024-06-04T04:46:08.919 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.378+0000 7f132f7fe700 1 -- 172.21.15.70:0/1049790520 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f132001bd80 con 0x7f1338107150 2024-06-04T04:46:08.919 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.378+0000 7f1335d9b700 1 --2- 172.21.15.70:0/1049790520 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f132403e470 0x7f1324040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:08.919 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.379+0000 7f1335d9b700 1 --2- 172.21.15.70:0/1049790520 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f132403e470 0x7f1324040930 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1328006fd0 tx=0x7f1328006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:08.919 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.384+0000 7f132f7fe700 1 -- 172.21.15.70:0/1049790520 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f132001f030 con 0x7f1338107150 2024-06-04T04:46:08.919 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.592+0000 7f133cb47700 1 -- 172.21.15.70:0/1049790520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f13381966a0 con 0x7f1338107150 2024-06-04T04:46:08.919 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.594+0000 7f132f7fe700 1 -- 172.21.15.70:0/1049790520 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f132000e260 con 0x7f1338107150 2024-06-04T04:46:08.919 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.596+0000 7f133cb47700 1 -- 172.21.15.70:0/1049790520 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f132403e470 msgr2=0x7f1324040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:08.919 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.596+0000 7f133cb47700 1 --2- 172.21.15.70:0/1049790520 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f132403e470 0x7f1324040930 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1328006fd0 tx=0x7f1328006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:08.919 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.596+0000 7f133cb47700 1 -- 172.21.15.70:0/1049790520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1338107150 msgr2=0x7f1338109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:08.920 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.596+0000 7f133cb47700 1 --2- 172.21.15.70:0/1049790520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1338107150 0x7f1338109910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1320011670 tx=0x7f1320011a30 comp rx=0 tx=0).stop 2024-06-04T04:46:08.920 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.596+0000 7f133cb47700 1 -- 172.21.15.70:0/1049790520 shutdown_connections 2024-06-04T04:46:08.920 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.596+0000 7f133cb47700 1 --2- 172.21.15.70:0/1049790520 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f132403e470 0x7f1324040930 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:08.920 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.596+0000 7f133cb47700 1 --2- 172.21.15.70:0/1049790520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1338107150 0x7f1338109910 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:08.920 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.596+0000 7f133cb47700 1 -- 172.21.15.70:0/1049790520 >> 172.21.15.70:0/1049790520 conn(0x7f1338102ae0 msgr2=0x7f1338103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:08.920 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.597+0000 7f133cb47700 1 -- 172.21.15.70:0/1049790520 shutdown_connections 2024-06-04T04:46:08.920 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:07.597+0000 7f133cb47700 1 -- 172.21.15.70:0/1049790520 wait complete. 2024-06-04T04:46:08.920 INFO:teuthology.orchestra.run.smithi070.stdout:Enabling the dashboard module... 2024-06-04T04:46:10.830 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:10 smithi070 ceph-mon[32631]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:46:10.830 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:10 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2171655133' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-06-04T04:46:10.831 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: ignoring --setuser ceph since I am not root 2024-06-04T04:46:10.831 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: ignoring --setgroup ceph since I am not root 2024-06-04T04:46:10.831 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:10.693+0000 7f7b0fe00200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-04T04:46:11.109 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:10.882+0000 7f7b0fe00200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-04T04:46:11.109 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:10.985+0000 7f7b0fe00200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-04T04:46:11.122 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.341+0000 7f4837f1d700 1 Processor -- start 2024-06-04T04:46:11.123 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.341+0000 7f4837f1d700 1 -- start start 2024-06-04T04:46:11.123 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.341+0000 7f4837f1d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4830109340 0x7f4830109760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:11.123 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.342+0000 7f4837f1d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4830109d30 con 0x7f4830109340 2024-06-04T04:46:11.124 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.342+0000 7f4835cb9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4830109340 0x7f4830109760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:11.124 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.342+0000 7f4835cb9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4830109340 0x7f4830109760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60974/0 (socket says 172.21.15.70:60974) 2024-06-04T04:46:11.124 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.342+0000 7f4835cb9700 1 -- 172.21.15.70:0/4200069933 learned_addr learned my addr 172.21.15.70:0/4200069933 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:11.124 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.342+0000 7f4835cb9700 1 -- 172.21.15.70:0/4200069933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4830107f50 con 0x7f4830109340 2024-06-04T04:46:11.124 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.342+0000 7f4835cb9700 1 --2- 172.21.15.70:0/4200069933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4830109340 0x7f4830109760 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f482c0098a0 tx=0x7f482c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21df6fb020bdfeeb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:11.124 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.342+0000 7f4834cb7700 1 -- 172.21.15.70:0/4200069933 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f482c004160 con 0x7f4830109340 2024-06-04T04:46:11.124 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.342+0000 7f4834cb7700 1 -- 172.21.15.70:0/4200069933 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f482c004300 con 0x7f4830109340 2024-06-04T04:46:11.124 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.342+0000 7f4834cb7700 1 -- 172.21.15.70:0/4200069933 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f482c011960 con 0x7f4830109340 2024-06-04T04:46:11.124 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.343+0000 7f4837f1d700 1 -- 172.21.15.70:0/4200069933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4830109340 msgr2=0x7f4830109760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:11.124 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.343+0000 7f4837f1d700 1 --2- 172.21.15.70:0/4200069933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4830109340 0x7f4830109760 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f482c0098a0 tx=0x7f482c009bb0 comp rx=0 tx=0).stop 2024-06-04T04:46:11.124 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.343+0000 7f4837f1d700 1 -- 172.21.15.70:0/4200069933 shutdown_connections 2024-06-04T04:46:11.124 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.343+0000 7f4837f1d700 1 --2- 172.21.15.70:0/4200069933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4830109340 0x7f4830109760 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:11.125 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.343+0000 7f4837f1d700 1 -- 172.21.15.70:0/4200069933 >> 172.21.15.70:0/4200069933 conn(0x7f4830102ae0 msgr2=0x7f4830104f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:11.125 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.343+0000 7f4837f1d700 1 -- 172.21.15.70:0/4200069933 shutdown_connections 2024-06-04T04:46:11.125 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.343+0000 7f4837f1d700 1 -- 172.21.15.70:0/4200069933 wait complete. 2024-06-04T04:46:11.125 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.344+0000 7f4837f1d700 1 Processor -- start 2024-06-04T04:46:11.125 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.344+0000 7f4837f1d700 1 -- start start 2024-06-04T04:46:11.125 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.344+0000 7f4837f1d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4830109340 0x7f483019d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:11.125 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.344+0000 7f4837f1d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f483010c190 con 0x7f4830109340 2024-06-04T04:46:11.125 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.344+0000 7f4835cb9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4830109340 0x7f483019d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:11.125 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.344+0000 7f4835cb9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4830109340 0x7f483019d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60978/0 (socket says 172.21.15.70:60978) 2024-06-04T04:46:11.125 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.344+0000 7f4835cb9700 1 -- 172.21.15.70:0/2171655133 learned_addr learned my addr 172.21.15.70:0/2171655133 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:11.125 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.344+0000 7f4835cb9700 1 -- 172.21.15.70:0/2171655133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f483019d6e0 con 0x7f4830109340 2024-06-04T04:46:11.125 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.344+0000 7f4835cb9700 1 --2- 172.21.15.70:0/2171655133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4830109340 0x7f483019d1a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f482c011670 tx=0x7f482c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:11.125 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.344+0000 7f4826ffd700 1 -- 172.21.15.70:0/2171655133 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f482c003aa0 con 0x7f4830109340 2024-06-04T04:46:11.125 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.345+0000 7f4826ffd700 1 -- 172.21.15.70:0/2171655133 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f482c01b8d0 con 0x7f4830109340 2024-06-04T04:46:11.126 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.345+0000 7f4837f1d700 1 -- 172.21.15.70:0/2171655133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f483019d970 con 0x7f4830109340 2024-06-04T04:46:11.126 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.345+0000 7f4826ffd700 1 -- 172.21.15.70:0/2171655133 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f482c0197f0 con 0x7f4830109340 2024-06-04T04:46:11.126 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.345+0000 7f4837f1d700 1 -- 172.21.15.70:0/2171655133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48301a05c0 con 0x7f4830109340 2024-06-04T04:46:11.126 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.345+0000 7f4826ffd700 1 -- 172.21.15.70:0/2171655133 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f482c01b440 con 0x7f4830109340 2024-06-04T04:46:11.126 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.346+0000 7f4837f1d700 1 -- 172.21.15.70:0/2171655133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f483010c310 con 0x7f4830109340 2024-06-04T04:46:11.126 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.346+0000 7f4826ffd700 1 --2- 172.21.15.70:0/2171655133 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f481c03e470 0x7f481c040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:11.126 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.346+0000 7f4826ffd700 1 -- 172.21.15.70:0/2171655133 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f482c053e50 con 0x7f4830109340 2024-06-04T04:46:11.126 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.346+0000 7f48354b8700 1 --2- 172.21.15.70:0/2171655133 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f481c03e470 0x7f481c040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:11.126 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.346+0000 7f48354b8700 1 --2- 172.21.15.70:0/2171655133 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f481c03e470 0x7f481c040930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4820006fd0 tx=0x7f4820006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:11.126 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.351+0000 7f4826ffd700 1 -- 172.21.15.70:0/2171655133 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f482c01f030 con 0x7f4830109340 2024-06-04T04:46:11.127 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:09.616+0000 7f4837f1d700 1 -- 172.21.15.70:0/2171655133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f483006c960 con 0x7f4830109340 2024-06-04T04:46:11.127 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:10.513+0000 7f4826ffd700 1 -- 172.21.15.70:0/2171655133 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f482c055020 con 0x7f4830109340 2024-06-04T04:46:11.127 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:10.515+0000 7f4826ffd700 1 -- 172.21.15.70:0/2171655133 <== mon.0 v2:172.21.15.70:3300/0 8 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f482c01a3d0 con 0x7f4830109340 2024-06-04T04:46:11.127 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:10.516+0000 7f4837f1d700 1 -- 172.21.15.70:0/2171655133 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f481c03e470 msgr2=0x7f481c040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:11.127 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:10.516+0000 7f4837f1d700 1 --2- 172.21.15.70:0/2171655133 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f481c03e470 0x7f481c040930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4820006fd0 tx=0x7f4820006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:11.127 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:10.516+0000 7f4837f1d700 1 -- 172.21.15.70:0/2171655133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4830109340 msgr2=0x7f483019d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:11.127 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:10.516+0000 7f4837f1d700 1 --2- 172.21.15.70:0/2171655133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4830109340 0x7f483019d1a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f482c011670 tx=0x7f482c011a30 comp rx=0 tx=0).stop 2024-06-04T04:46:11.127 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:10.516+0000 7f4837f1d700 1 -- 172.21.15.70:0/2171655133 shutdown_connections 2024-06-04T04:46:11.127 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:10.516+0000 7f4837f1d700 1 --2- 172.21.15.70:0/2171655133 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f481c03e470 0x7f481c040930 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:11.127 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:10.517+0000 7f4837f1d700 1 --2- 172.21.15.70:0/2171655133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4830109340 0x7f483019d1a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:11.127 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:10.517+0000 7f4837f1d700 1 -- 172.21.15.70:0/2171655133 >> 172.21.15.70:0/2171655133 conn(0x7f4830102ae0 msgr2=0x7f48301039c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:11.127 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:10.517+0000 7f4837f1d700 1 -- 172.21.15.70:0/2171655133 shutdown_connections 2024-06-04T04:46:11.127 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:10.517+0000 7f4837f1d700 1 -- 172.21.15.70:0/2171655133 wait complete. 2024-06-04T04:46:11.394 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:11 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:11.303+0000 7f7b0fe00200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-04T04:46:11.395 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:11 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:11.392+0000 7f7b0fe00200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-04T04:46:11.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:11 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2171655133' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-06-04T04:46:11.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:11 smithi070 ceph-mon[32631]: mgrmap e9: x(active, since 21s) 2024-06-04T04:46:12.420 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-06-04T04:46:12.420 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-06-04T04:46:12.420 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-04T04:46:12.420 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-06-04T04:46:12.420 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-04T04:46:12.420 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-06-04T04:46:12.420 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.527+0000 7f0e61b31700 1 Processor -- start 2024-06-04T04:46:12.420 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.528+0000 7f0e61b31700 1 -- start start 2024-06-04T04:46:12.420 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.528+0000 7f0e61b31700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c109900 0x7f0e5c10bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:12.420 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.528+0000 7f0e61b31700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c07ce50 con 0x7f0e5c109900 2024-06-04T04:46:12.420 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.528+0000 7f0e5b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c109900 0x7f0e5c10bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:12.421 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.528+0000 7f0e5b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c109900 0x7f0e5c10bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60994/0 (socket says 172.21.15.70:60994) 2024-06-04T04:46:12.421 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.528+0000 7f0e5b7fe700 1 -- 172.21.15.70:0/3248530439 learned_addr learned my addr 172.21.15.70:0/3248530439 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:12.421 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.528+0000 7f0e5b7fe700 1 -- 172.21.15.70:0/3248530439 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e5c10c230 con 0x7f0e5c109900 2024-06-04T04:46:12.421 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.528+0000 7f0e5b7fe700 1 --2- 172.21.15.70:0/3248530439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c109900 0x7f0e5c10bcf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c0098a0 tx=0x7f0e4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9968b0386a7242bc server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:12.421 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.528+0000 7f0e5a7fc700 1 -- 172.21.15.70:0/3248530439 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e4c004160 con 0x7f0e5c109900 2024-06-04T04:46:12.421 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.528+0000 7f0e5a7fc700 1 -- 172.21.15.70:0/3248530439 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0e4c004300 con 0x7f0e5c109900 2024-06-04T04:46:12.421 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.528+0000 7f0e5a7fc700 1 -- 172.21.15.70:0/3248530439 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e4c011960 con 0x7f0e5c109900 2024-06-04T04:46:12.421 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.529+0000 7f0e61b31700 1 -- 172.21.15.70:0/3248530439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c109900 msgr2=0x7f0e5c10bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:12.421 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.529+0000 7f0e61b31700 1 --2- 172.21.15.70:0/3248530439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c109900 0x7f0e5c10bcf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c0098a0 tx=0x7f0e4c009bb0 comp rx=0 tx=0).stop 2024-06-04T04:46:12.421 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.529+0000 7f0e61b31700 1 -- 172.21.15.70:0/3248530439 shutdown_connections 2024-06-04T04:46:12.421 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.529+0000 7f0e61b31700 1 --2- 172.21.15.70:0/3248530439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c109900 0x7f0e5c10bcf0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:12.421 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.529+0000 7f0e61b31700 1 -- 172.21.15.70:0/3248530439 >> 172.21.15.70:0/3248530439 conn(0x7f0e5c103330 msgr2=0x7f0e5c105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:12.421 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.529+0000 7f0e61b31700 1 -- 172.21.15.70:0/3248530439 shutdown_connections 2024-06-04T04:46:12.422 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.529+0000 7f0e61b31700 1 -- 172.21.15.70:0/3248530439 wait complete. 2024-06-04T04:46:12.422 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.530+0000 7f0e61b31700 1 Processor -- start 2024-06-04T04:46:12.422 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.530+0000 7f0e61b31700 1 -- start start 2024-06-04T04:46:12.422 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.530+0000 7f0e61b31700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c109900 0x7f0e5c19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:12.422 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.530+0000 7f0e61b31700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c10d1d0 con 0x7f0e5c109900 2024-06-04T04:46:12.422 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.530+0000 7f0e5b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c109900 0x7f0e5c19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:12.422 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.530+0000 7f0e5b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c109900 0x7f0e5c19cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32772/0 (socket says 172.21.15.70:32772) 2024-06-04T04:46:12.422 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.530+0000 7f0e5b7fe700 1 -- 172.21.15.70:0/1619528601 learned_addr learned my addr 172.21.15.70:0/1619528601 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:12.422 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.530+0000 7f0e5b7fe700 1 -- 172.21.15.70:0/1619528601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e5c19d280 con 0x7f0e5c109900 2024-06-04T04:46:12.422 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.531+0000 7f0e5b7fe700 1 --2- 172.21.15.70:0/1619528601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c109900 0x7f0e5c19cd40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c004660 tx=0x7f0e4c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:12.422 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.531+0000 7f0e58ff9700 1 -- 172.21.15.70:0/1619528601 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e4c003aa0 con 0x7f0e5c109900 2024-06-04T04:46:12.422 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.531+0000 7f0e58ff9700 1 -- 172.21.15.70:0/1619528601 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0e4c01b8d0 con 0x7f0e5c109900 2024-06-04T04:46:12.422 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.531+0000 7f0e61b31700 1 -- 172.21.15.70:0/1619528601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e5c19d510 con 0x7f0e5c109900 2024-06-04T04:46:12.423 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.531+0000 7f0e58ff9700 1 -- 172.21.15.70:0/1619528601 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e4c023820 con 0x7f0e5c109900 2024-06-04T04:46:12.423 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.531+0000 7f0e61b31700 1 -- 172.21.15.70:0/1619528601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e5c1a0200 con 0x7f0e5c109900 2024-06-04T04:46:12.423 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.532+0000 7f0e61b31700 1 -- 172.21.15.70:0/1619528601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e5c064ba0 con 0x7f0e5c109900 2024-06-04T04:46:12.423 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.532+0000 7f0e58ff9700 1 -- 172.21.15.70:0/1619528601 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f0e4c01b440 con 0x7f0e5c109900 2024-06-04T04:46:12.423 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.532+0000 7f0e58ff9700 1 --2- 172.21.15.70:0/1619528601 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f0e4403e4d0 0x7f0e44040990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:12.423 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.532+0000 7f0e58ff9700 1 -- 172.21.15.70:0/1619528601 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0e4c053e30 con 0x7f0e5c109900 2024-06-04T04:46:12.423 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.532+0000 7f0e5affd700 1 -- 172.21.15.70:0/1619528601 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f0e4403e4d0 msgr2=0x7f0e44040990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/1353782413 2024-06-04T04:46:12.423 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.532+0000 7f0e5affd700 1 --2- 172.21.15.70:0/1619528601 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f0e4403e4d0 0x7f0e44040990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-04T04:46:12.423 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.537+0000 7f0e58ff9700 1 -- 172.21.15.70:0/1619528601 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0e4c01f030 con 0x7f0e5c109900 2024-06-04T04:46:12.423 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.733+0000 7f0e5affd700 1 -- 172.21.15.70:0/1619528601 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f0e4403e4d0 msgr2=0x7f0e44040990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/1353782413 2024-06-04T04:46:12.423 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.733+0000 7f0e5affd700 1 --2- 172.21.15.70:0/1619528601 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f0e4403e4d0 0x7f0e44040990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-04T04:46:12.423 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.834+0000 7f0e61b31700 1 -- 172.21.15.70:0/1619528601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f0e5c1a0640 con 0x7f0e5c109900 2024-06-04T04:46:12.423 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.836+0000 7f0e58ff9700 1 -- 172.21.15.70:0/1619528601 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f0e4c00edf0 con 0x7f0e5c109900 2024-06-04T04:46:12.423 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.839+0000 7f0e61b31700 1 -- 172.21.15.70:0/1619528601 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f0e4403e4d0 msgr2=0x7f0e44040990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-04T04:46:12.424 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.839+0000 7f0e61b31700 1 --2- 172.21.15.70:0/1619528601 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f0e4403e4d0 0x7f0e44040990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:12.424 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.839+0000 7f0e61b31700 1 -- 172.21.15.70:0/1619528601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c109900 msgr2=0x7f0e5c19cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:12.424 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.839+0000 7f0e61b31700 1 --2- 172.21.15.70:0/1619528601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c109900 0x7f0e5c19cd40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c004660 tx=0x7f0e4c011a30 comp rx=0 tx=0).stop 2024-06-04T04:46:12.424 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.839+0000 7f0e61b31700 1 -- 172.21.15.70:0/1619528601 shutdown_connections 2024-06-04T04:46:12.424 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.839+0000 7f0e61b31700 1 --2- 172.21.15.70:0/1619528601 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f0e4403e4d0 0x7f0e44040990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:12.424 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.839+0000 7f0e61b31700 1 --2- 172.21.15.70:0/1619528601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c109900 0x7f0e5c19cd40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:12.424 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.839+0000 7f0e61b31700 1 -- 172.21.15.70:0/1619528601 >> 172.21.15.70:0/1619528601 conn(0x7f0e5c103330 msgr2=0x7f0e5c103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:12.424 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.839+0000 7f0e61b31700 1 -- 172.21.15.70:0/1619528601 shutdown_connections 2024-06-04T04:46:12.424 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:11.839+0000 7f0e61b31700 1 -- 172.21.15.70:0/1619528601 wait complete. 2024-06-04T04:46:12.424 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for the mgr to restart... 2024-06-04T04:46:12.424 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mgr epoch 9... 2024-06-04T04:46:12.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:12 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1619528601' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-04T04:46:12.731 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:12 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:12.624+0000 7f7b0fe00200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-04T04:46:13.041 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:12 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:12.801+0000 7f7b0fe00200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-04T04:46:13.041 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:12 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:12.975+0000 7f7b0fe00200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-04T04:46:13.358 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:13 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:13.263+0000 7f7b0fe00200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-04T04:46:13.358 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:13 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:13.355+0000 7f7b0fe00200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-04T04:46:13.609 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:13 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:13.558+0000 7f7b0fe00200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-04T04:46:14.109 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:13 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:13.751+0000 7f7b0fe00200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-04T04:46:14.579 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:14 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:14.352+0000 7f7b0fe00200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-04T04:46:14.579 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:14 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:14.575+0000 7f7b0fe00200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-06-04T04:46:15.359 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:15 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:15.004+0000 7f7b0fe00200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-04T04:46:15.359 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:15 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:15.102+0000 7f7b0fe00200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-04T04:46:15.836 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:15 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:15.479+0000 7f7b0fe00200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-04T04:46:15.836 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:15 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:15.833+0000 7f7b0fe00200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-04T04:46:16.109 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:15 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:15.927+0000 7f7b0fe00200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-04T04:46:16.609 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:16 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:16.174+0000 7f7b0fe00200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-04T04:46:16.609 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:16 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:16.305+0000 7f7b0fe00200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-04T04:46:16.609 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:16 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:16.402+0000 7f7b0fe00200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-04T04:46:17.109 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:16 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:16.724+0000 7f7b0fe00200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-04T04:46:17.109 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:16 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:16.817+0000 7f7b0fe00200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-04T04:46:17.690 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:17 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:17.415+0000 7f7b0fe00200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-04T04:46:17.690 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:46:17 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:46:17.633+0000 7f7b0fe00200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-04T04:46:18.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:17 smithi070 ceph-mon[32631]: Active manager daemon x restarted 2024-06-04T04:46:18.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:17 smithi070 ceph-mon[32631]: Activating manager daemon x 2024-06-04T04:46:18.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:17 smithi070 ceph-mon[32631]: osdmap e3: 0 total, 0 up, 0 in 2024-06-04T04:46:18.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:17 smithi070 ceph-mon[32631]: mgrmap e10: x(active, starting, since 0.00346654s) 2024-06-04T04:46:18.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:17 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:46:18.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:17 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-04T04:46:18.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:17 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-04T04:46:18.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:17 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-04T04:46:18.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:17 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-04T04:46:18.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:17 smithi070 ceph-mon[32631]: Manager daemon x is now available 2024-06-04T04:46:18.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:17 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:46:18.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:17 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-04T04:46:18.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-04T04:46:18.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:18 smithi070 ceph-mon[32631]: mgrmap e11: x(active, since 1.007s) 2024-06-04T04:46:19.612 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-06-04T04:46:19.612 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-06-04T04:46:19.612 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-04T04:46:19.613 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-06-04T04:46:19.613 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.833+0000 7f8e3f5a7700 1 Processor -- start 2024-06-04T04:46:19.613 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.833+0000 7f8e3f5a7700 1 -- start start 2024-06-04T04:46:19.613 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.833+0000 7f8e3f5a7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38109960 0x7f8e3810bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:19.613 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.833+0000 7f8e3f5a7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3807ce30 con 0x7f8e38109960 2024-06-04T04:46:19.613 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.833+0000 7f8e3d343700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38109960 0x7f8e3810bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:19.613 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.833+0000 7f8e3d343700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38109960 0x7f8e3810bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32788/0 (socket says 172.21.15.70:32788) 2024-06-04T04:46:19.613 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.833+0000 7f8e3d343700 1 -- 172.21.15.70:0/634136997 learned_addr learned my addr 172.21.15.70:0/634136997 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:19.613 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.834+0000 7f8e3d343700 1 -- 172.21.15.70:0/634136997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e3810c290 con 0x7f8e38109960 2024-06-04T04:46:19.613 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.834+0000 7f8e3d343700 1 --2- 172.21.15.70:0/634136997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38109960 0x7f8e3810bd50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8e340098a0 tx=0x7f8e34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ec8f33d9a9ba9e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:19.613 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.834+0000 7f8e2ffff700 1 -- 172.21.15.70:0/634136997 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e34004160 con 0x7f8e38109960 2024-06-04T04:46:19.613 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.834+0000 7f8e2ffff700 1 -- 172.21.15.70:0/634136997 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e34004300 con 0x7f8e38109960 2024-06-04T04:46:19.613 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.834+0000 7f8e2ffff700 1 -- 172.21.15.70:0/634136997 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e34011960 con 0x7f8e38109960 2024-06-04T04:46:19.613 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.835+0000 7f8e3f5a7700 1 -- 172.21.15.70:0/634136997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38109960 msgr2=0x7f8e3810bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:19.614 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.835+0000 7f8e3f5a7700 1 --2- 172.21.15.70:0/634136997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38109960 0x7f8e3810bd50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8e340098a0 tx=0x7f8e34009bb0 comp rx=0 tx=0).stop 2024-06-04T04:46:19.614 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.835+0000 7f8e3f5a7700 1 -- 172.21.15.70:0/634136997 shutdown_connections 2024-06-04T04:46:19.614 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.835+0000 7f8e3f5a7700 1 --2- 172.21.15.70:0/634136997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38109960 0x7f8e3810bd50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:19.614 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.835+0000 7f8e3f5a7700 1 -- 172.21.15.70:0/634136997 >> 172.21.15.70:0/634136997 conn(0x7f8e38103310 msgr2=0x7f8e38105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:19.614 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.835+0000 7f8e3f5a7700 1 -- 172.21.15.70:0/634136997 shutdown_connections 2024-06-04T04:46:19.614 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.835+0000 7f8e3f5a7700 1 -- 172.21.15.70:0/634136997 wait complete. 2024-06-04T04:46:19.614 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.836+0000 7f8e3f5a7700 1 Processor -- start 2024-06-04T04:46:19.614 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.836+0000 7f8e3f5a7700 1 -- start start 2024-06-04T04:46:19.614 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.836+0000 7f8e3f5a7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38109960 0x7f8e3819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:19.614 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.836+0000 7f8e3f5a7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3810d1e0 con 0x7f8e38109960 2024-06-04T04:46:19.614 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.836+0000 7f8e3d343700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38109960 0x7f8e3819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:19.614 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.836+0000 7f8e3d343700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38109960 0x7f8e3819cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32802/0 (socket says 172.21.15.70:32802) 2024-06-04T04:46:19.614 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.836+0000 7f8e3d343700 1 -- 172.21.15.70:0/907881314 learned_addr learned my addr 172.21.15.70:0/907881314 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:19.614 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.836+0000 7f8e3d343700 1 -- 172.21.15.70:0/907881314 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e3819d2e0 con 0x7f8e38109960 2024-06-04T04:46:19.614 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.837+0000 7f8e3d343700 1 --2- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38109960 0x7f8e3819cda0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8e34011670 tx=0x7f8e34011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:19.615 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.837+0000 7f8e2e7fc700 1 -- 172.21.15.70:0/907881314 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e3401f070 con 0x7f8e38109960 2024-06-04T04:46:19.615 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.837+0000 7f8e3f5a7700 1 -- 172.21.15.70:0/907881314 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e3819d570 con 0x7f8e38109960 2024-06-04T04:46:19.615 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.837+0000 7f8e2e7fc700 1 -- 172.21.15.70:0/907881314 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e34003b20 con 0x7f8e38109960 2024-06-04T04:46:19.615 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.837+0000 7f8e2e7fc700 1 -- 172.21.15.70:0/907881314 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e340195f0 con 0x7f8e38109960 2024-06-04T04:46:19.615 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.837+0000 7f8e3f5a7700 1 -- 172.21.15.70:0/907881314 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e381a0220 con 0x7f8e38109960 2024-06-04T04:46:19.615 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.838+0000 7f8e2e7fc700 1 -- 172.21.15.70:0/907881314 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f8e34022070 con 0x7f8e38109960 2024-06-04T04:46:19.615 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.838+0000 7f8e2e7fc700 1 --2- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f8e2403e520 0x7f8e240409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:19.615 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.838+0000 7f8e2e7fc700 1 -- 172.21.15.70:0/907881314 --> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8e240410f0 con 0x7f8e2403e520 2024-06-04T04:46:19.615 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.838+0000 7f8e2e7fc700 1 -- 172.21.15.70:0/907881314 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8e34054930 con 0x7f8e38109960 2024-06-04T04:46:19.615 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.838+0000 7f8e3cb42700 1 -- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f8e2403e520 msgr2=0x7f8e240409e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/1353782413 2024-06-04T04:46:19.615 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:12.838+0000 7f8e3cb42700 1 --2- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f8e2403e520 0x7f8e240409e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-04T04:46:19.615 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:13.039+0000 7f8e3cb42700 1 -- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f8e2403e520 msgr2=0x7f8e240409e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/1353782413 2024-06-04T04:46:19.615 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:13.039+0000 7f8e3cb42700 1 --2- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f8e2403e520 0x7f8e240409e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-04T04:46:19.616 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:13.439+0000 7f8e3cb42700 1 -- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f8e2403e520 msgr2=0x7f8e240409e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/1353782413 2024-06-04T04:46:19.616 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:13.439+0000 7f8e3cb42700 1 --2- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f8e2403e520 0x7f8e240409e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-04T04:46:19.616 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:14.240+0000 7f8e3cb42700 1 -- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f8e2403e520 msgr2=0x7f8e240409e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/1353782413 2024-06-04T04:46:19.616 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:14.240+0000 7f8e3cb42700 1 --2- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f8e2403e520 0x7f8e240409e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-04T04:46:19.616 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:15.841+0000 7f8e3cb42700 1 -- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f8e2403e520 msgr2=0x7f8e240409e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/1353782413 2024-06-04T04:46:19.616 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:15.841+0000 7f8e3cb42700 1 --2- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f8e2403e520 0x7f8e240409e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-06-04T04:46:19.616 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:17.637+0000 7f8e2e7fc700 1 -- 172.21.15.70:0/907881314 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mgrmap(e 10) ==== 50002+0+0 (secure 0 0 0) 0x7f8e34019820 con 0x7f8e38109960 2024-06-04T04:46:19.616 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:17.637+0000 7f8e2e7fc700 1 -- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f8e2403e520 msgr2=0x7f8e240409e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-04T04:46:19.616 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:17.637+0000 7f8e2e7fc700 1 --2- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f8e2403e520 0x7f8e240409e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:19.616 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.640+0000 7f8e2e7fc700 1 -- 172.21.15.70:0/907881314 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 11) ==== 50129+0+0 (secure 0 0 0) 0x7f8e340552c0 con 0x7f8e38109960 2024-06-04T04:46:19.616 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.640+0000 7f8e2e7fc700 1 --2- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8e24041fa0 0x7f8e24044390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:19.616 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.640+0000 7f8e2e7fc700 1 -- 172.21.15.70:0/907881314 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8e34017e60 con 0x7f8e24041fa0 2024-06-04T04:46:19.616 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.640+0000 7f8e3cb42700 1 --2- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8e24041fa0 0x7f8e24044390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:19.617 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.641+0000 7f8e3cb42700 1 --2- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8e24041fa0 0x7f8e24044390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8e28003a10 tx=0x7f8e28009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:19.617 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.641+0000 7f8e2e7fc700 1 -- 172.21.15.70:0/907881314 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f8e34017e60 con 0x7f8e24041fa0 2024-06-04T04:46:19.617 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.656+0000 7f8e3f5a7700 1 -- 172.21.15.70:0/907881314 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f8e1c002630 con 0x7f8e24041fa0 2024-06-04T04:46:19.617 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.656+0000 7f8e2e7fc700 1 -- 172.21.15.70:0/907881314 <== mgr.14152 v2:172.21.15.70:6800/2752966840 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f8e1c002630 con 0x7f8e24041fa0 2024-06-04T04:46:19.617 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.656+0000 7f8e3f5a7700 1 -- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8e24041fa0 msgr2=0x7f8e24044390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:19.617 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.656+0000 7f8e3f5a7700 1 --2- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8e24041fa0 0x7f8e24044390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8e28003a10 tx=0x7f8e28009270 comp rx=0 tx=0).stop 2024-06-04T04:46:19.617 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.656+0000 7f8e3f5a7700 1 -- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38109960 msgr2=0x7f8e3819cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:19.617 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.656+0000 7f8e3f5a7700 1 --2- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38109960 0x7f8e3819cda0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8e34011670 tx=0x7f8e34011a30 comp rx=0 tx=0).stop 2024-06-04T04:46:19.617 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.656+0000 7f8e3f5a7700 1 -- 172.21.15.70:0/907881314 shutdown_connections 2024-06-04T04:46:19.617 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.656+0000 7f8e3f5a7700 1 --2- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8e24041fa0 0x7f8e24044390 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:19.617 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.656+0000 7f8e3f5a7700 1 --2- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38109960 0x7f8e3819cda0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:19.617 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.656+0000 7f8e3f5a7700 1 --2- 172.21.15.70:0/907881314 >> [v2:172.21.15.70:6800/1353782413,v1:172.21.15.70:6801/1353782413] conn(0x7f8e2403e520 0x7f8e240409e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:19.617 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.657+0000 7f8e3f5a7700 1 -- 172.21.15.70:0/907881314 >> 172.21.15.70:0/907881314 conn(0x7f8e38103310 msgr2=0x7f8e38103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:19.617 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.657+0000 7f8e3f5a7700 1 -- 172.21.15.70:0/907881314 shutdown_connections 2024-06-04T04:46:19.617 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:18.657+0000 7f8e3f5a7700 1 -- 172.21.15.70:0/907881314 wait complete. 2024-06-04T04:46:19.618 INFO:teuthology.orchestra.run.smithi070.stdout:mgr epoch 9 is available 2024-06-04T04:46:19.618 INFO:teuthology.orchestra.run.smithi070.stdout:Generating a dashboard self-signed certificate... 2024-06-04T04:46:20.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:19 smithi070 ceph-mon[32631]: [04/Jun/2024:04:46:18] ENGINE Bus STARTING 2024-06-04T04:46:20.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:20.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:19 smithi070 ceph-mon[32631]: [04/Jun/2024:04:46:19] ENGINE Serving on http://172.21.15.70:8765 2024-06-04T04:46:20.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:19 smithi070 ceph-mon[32631]: [04/Jun/2024:04:46:19] ENGINE Serving on https://172.21.15.70:7150 2024-06-04T04:46:20.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:19 smithi070 ceph-mon[32631]: [04/Jun/2024:04:46:19] ENGINE Bus STARTED 2024-06-04T04:46:20.883 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-06-04T04:46:20.884 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:19.999+0000 7f41659f9700 1 Processor -- start 2024-06-04T04:46:20.884 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:19.999+0000 7f41659f9700 1 -- start start 2024-06-04T04:46:20.884 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.000+0000 7f41659f9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4160108ee0 0x7f4160109300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:20.884 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.000+0000 7f41659f9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4160109840 con 0x7f4160108ee0 2024-06-04T04:46:20.884 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.000+0000 7f415effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4160108ee0 0x7f4160109300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:20.884 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.000+0000 7f415effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4160108ee0 0x7f4160109300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52842/0 (socket says 172.21.15.70:52842) 2024-06-04T04:46:20.884 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.000+0000 7f415effd700 1 -- 172.21.15.70:0/617951300 learned_addr learned my addr 172.21.15.70:0/617951300 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:20.884 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.000+0000 7f415effd700 1 -- 172.21.15.70:0/617951300 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4160109fa0 con 0x7f4160108ee0 2024-06-04T04:46:20.884 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.000+0000 7f415effd700 1 --2- 172.21.15.70:0/617951300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4160108ee0 0x7f4160109300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f41540098a0 tx=0x7f4154009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d14fe3c0653883 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:20.884 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.000+0000 7f415dffb700 1 -- 172.21.15.70:0/617951300 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4154004160 con 0x7f4160108ee0 2024-06-04T04:46:20.884 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.000+0000 7f415dffb700 1 -- 172.21.15.70:0/617951300 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4154004300 con 0x7f4160108ee0 2024-06-04T04:46:20.884 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.000+0000 7f415dffb700 1 -- 172.21.15.70:0/617951300 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4154011960 con 0x7f4160108ee0 2024-06-04T04:46:20.885 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.001+0000 7f41659f9700 1 -- 172.21.15.70:0/617951300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4160108ee0 msgr2=0x7f4160109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:20.885 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.001+0000 7f41659f9700 1 --2- 172.21.15.70:0/617951300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4160108ee0 0x7f4160109300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f41540098a0 tx=0x7f4154009bb0 comp rx=0 tx=0).stop 2024-06-04T04:46:20.885 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.001+0000 7f41659f9700 1 -- 172.21.15.70:0/617951300 shutdown_connections 2024-06-04T04:46:20.885 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.001+0000 7f41659f9700 1 --2- 172.21.15.70:0/617951300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4160108ee0 0x7f4160109300 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:20.885 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.001+0000 7f41659f9700 1 -- 172.21.15.70:0/617951300 >> 172.21.15.70:0/617951300 conn(0x7f4160104460 msgr2=0x7f41601068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:20.885 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.001+0000 7f41659f9700 1 -- 172.21.15.70:0/617951300 shutdown_connections 2024-06-04T04:46:20.885 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.001+0000 7f41659f9700 1 -- 172.21.15.70:0/617951300 wait complete. 2024-06-04T04:46:20.885 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.001+0000 7f41659f9700 1 Processor -- start 2024-06-04T04:46:20.885 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.002+0000 7f41659f9700 1 -- start start 2024-06-04T04:46:20.886 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.002+0000 7f41659f9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4160108ee0 0x7f416019ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:20.886 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.002+0000 7f41659f9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f416010d260 con 0x7f4160108ee0 2024-06-04T04:46:20.886 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.002+0000 7f415effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4160108ee0 0x7f416019ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:20.886 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.002+0000 7f415effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4160108ee0 0x7f416019ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52850/0 (socket says 172.21.15.70:52850) 2024-06-04T04:46:20.886 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.002+0000 7f415effd700 1 -- 172.21.15.70:0/2212519164 learned_addr learned my addr 172.21.15.70:0/2212519164 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:20.886 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.002+0000 7f415effd700 1 -- 172.21.15.70:0/2212519164 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f416019d380 con 0x7f4160108ee0 2024-06-04T04:46:20.886 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.002+0000 7f415effd700 1 --2- 172.21.15.70:0/2212519164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4160108ee0 0x7f416019ce40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4154011670 tx=0x7f4154011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:20.886 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.002+0000 7f41649f7700 1 -- 172.21.15.70:0/2212519164 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4154003a60 con 0x7f4160108ee0 2024-06-04T04:46:20.886 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.002+0000 7f41659f9700 1 -- 172.21.15.70:0/2212519164 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f416019d610 con 0x7f4160108ee0 2024-06-04T04:46:20.886 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.003+0000 7f41649f7700 1 -- 172.21.15.70:0/2212519164 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f415401b8d0 con 0x7f4160108ee0 2024-06-04T04:46:20.886 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.003+0000 7f41659f9700 1 -- 172.21.15.70:0/2212519164 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41601a0300 con 0x7f4160108ee0 2024-06-04T04:46:20.886 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.003+0000 7f41649f7700 1 -- 172.21.15.70:0/2212519164 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4154023790 con 0x7f4160108ee0 2024-06-04T04:46:20.886 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.003+0000 7f41467fc700 1 -- 172.21.15.70:0/2212519164 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4140005330 con 0x7f4160108ee0 2024-06-04T04:46:20.887 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.003+0000 7f41649f7700 1 -- 172.21.15.70:0/2212519164 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f4154023a60 con 0x7f4160108ee0 2024-06-04T04:46:20.887 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.004+0000 7f41649f7700 1 --2- 172.21.15.70:0/2212519164 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f414803e520 0x7f41480409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:20.887 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.004+0000 7f41649f7700 1 -- 172.21.15.70:0/2212519164 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4154054070 con 0x7f4160108ee0 2024-06-04T04:46:20.887 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.004+0000 7f415e7fc700 1 --2- 172.21.15.70:0/2212519164 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f414803e520 0x7f41480409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:20.887 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.004+0000 7f415e7fc700 1 --2- 172.21.15.70:0/2212519164 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f414803e520 0x7f41480409e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4150006fd0 tx=0x7f4150006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:20.887 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.009+0000 7f41649f7700 1 -- 172.21.15.70:0/2212519164 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f415401f030 con 0x7f4160108ee0 2024-06-04T04:46:20.887 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.226+0000 7f41467fc700 1 -- 172.21.15.70:0/2212519164 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f4140000bc0 con 0x7f414803e520 2024-06-04T04:46:20.887 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.342+0000 7f41649f7700 1 -- 172.21.15.70:0/2212519164 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f4140000bc0 con 0x7f414803e520 2024-06-04T04:46:20.887 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.345+0000 7f41467fc700 1 -- 172.21.15.70:0/2212519164 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f414803e520 msgr2=0x7f41480409e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:20.887 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.345+0000 7f41467fc700 1 --2- 172.21.15.70:0/2212519164 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f414803e520 0x7f41480409e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4150006fd0 tx=0x7f4150006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:20.887 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.345+0000 7f41467fc700 1 -- 172.21.15.70:0/2212519164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4160108ee0 msgr2=0x7f416019ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:20.887 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.345+0000 7f41467fc700 1 --2- 172.21.15.70:0/2212519164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4160108ee0 0x7f416019ce40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4154011670 tx=0x7f4154011a30 comp rx=0 tx=0).stop 2024-06-04T04:46:20.887 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.345+0000 7f41467fc700 1 -- 172.21.15.70:0/2212519164 shutdown_connections 2024-06-04T04:46:20.888 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.345+0000 7f41467fc700 1 --2- 172.21.15.70:0/2212519164 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f414803e520 0x7f41480409e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:20.888 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.345+0000 7f41467fc700 1 --2- 172.21.15.70:0/2212519164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4160108ee0 0x7f416019ce40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:20.888 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.345+0000 7f41467fc700 1 -- 172.21.15.70:0/2212519164 >> 172.21.15.70:0/2212519164 conn(0x7f4160104460 msgr2=0x7f4160104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:20.888 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.345+0000 7f41467fc700 1 -- 172.21.15.70:0/2212519164 shutdown_connections 2024-06-04T04:46:20.888 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:20.345+0000 7f41467fc700 1 -- 172.21.15.70:0/2212519164 wait complete. 2024-06-04T04:46:20.888 INFO:teuthology.orchestra.run.smithi070.stdout:Creating initial admin user... 2024-06-04T04:46:21.249 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:20 smithi070 ceph-mon[32631]: mgrmap e12: x(active, since 2s) 2024-06-04T04:46:21.249 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:20 smithi070 ceph-mon[32631]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:46:21.250 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:20 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:21.250 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:20 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:21.250 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:20 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:22.669 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$zhPCQp.wl8pH0lgaDflUx.b65tQU5lOrrTxvWypT8W/7zRFKVn2YK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1717476381, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-06-04T04:46:22.669 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.351+0000 7fb111d02700 1 Processor -- start 2024-06-04T04:46:22.669 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.351+0000 7fb111d02700 1 -- start start 2024-06-04T04:46:22.670 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.351+0000 7fb111d02700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c108ed0 0x7fb10c1092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:22.670 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.351+0000 7fb111d02700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c109830 con 0x7fb10c108ed0 2024-06-04T04:46:22.670 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.351+0000 7fb10b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c108ed0 0x7fb10c1092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:22.670 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.351+0000 7fb10b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c108ed0 0x7fb10c1092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52860/0 (socket says 172.21.15.70:52860) 2024-06-04T04:46:22.670 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.351+0000 7fb10b7fe700 1 -- 172.21.15.70:0/2828788027 learned_addr learned my addr 172.21.15.70:0/2828788027 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:22.670 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.351+0000 7fb10b7fe700 1 -- 172.21.15.70:0/2828788027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10c109f90 con 0x7fb10c108ed0 2024-06-04T04:46:22.670 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.352+0000 7fb10b7fe700 1 --2- 172.21.15.70:0/2828788027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c108ed0 0x7fb10c1092f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc0098a0 tx=0x7fb0fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3bd99a83a730de4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:22.670 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.352+0000 7fb10a7fc700 1 -- 172.21.15.70:0/2828788027 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0fc004160 con 0x7fb10c108ed0 2024-06-04T04:46:22.670 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.352+0000 7fb10a7fc700 1 -- 172.21.15.70:0/2828788027 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb0fc004300 con 0x7fb10c108ed0 2024-06-04T04:46:22.670 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.352+0000 7fb10a7fc700 1 -- 172.21.15.70:0/2828788027 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0fc011960 con 0x7fb10c108ed0 2024-06-04T04:46:22.670 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.352+0000 7fb111d02700 1 -- 172.21.15.70:0/2828788027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c108ed0 msgr2=0x7fb10c1092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:22.670 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.352+0000 7fb111d02700 1 --2- 172.21.15.70:0/2828788027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c108ed0 0x7fb10c1092f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc0098a0 tx=0x7fb0fc009bb0 comp rx=0 tx=0).stop 2024-06-04T04:46:22.671 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.352+0000 7fb111d02700 1 -- 172.21.15.70:0/2828788027 shutdown_connections 2024-06-04T04:46:22.671 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.353+0000 7fb111d02700 1 --2- 172.21.15.70:0/2828788027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c108ed0 0x7fb10c1092f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:22.671 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.353+0000 7fb111d02700 1 -- 172.21.15.70:0/2828788027 >> 172.21.15.70:0/2828788027 conn(0x7fb10c104470 msgr2=0x7fb10c1068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:22.671 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.353+0000 7fb111d02700 1 -- 172.21.15.70:0/2828788027 shutdown_connections 2024-06-04T04:46:22.671 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.353+0000 7fb111d02700 1 -- 172.21.15.70:0/2828788027 wait complete. 2024-06-04T04:46:22.671 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.353+0000 7fb111d02700 1 Processor -- start 2024-06-04T04:46:22.671 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.353+0000 7fb111d02700 1 -- start start 2024-06-04T04:46:22.671 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.354+0000 7fb111d02700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c108ed0 0x7fb10c19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:22.671 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.354+0000 7fb111d02700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c10d250 con 0x7fb10c108ed0 2024-06-04T04:46:22.671 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.354+0000 7fb10b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c108ed0 0x7fb10c19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:22.671 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.354+0000 7fb10b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c108ed0 0x7fb10c19f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52862/0 (socket says 172.21.15.70:52862) 2024-06-04T04:46:22.671 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.354+0000 7fb10b7fe700 1 -- 172.21.15.70:0/2800053004 learned_addr learned my addr 172.21.15.70:0/2800053004 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:22.671 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.354+0000 7fb10b7fe700 1 -- 172.21.15.70:0/2800053004 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10c19f5b0 con 0x7fb10c108ed0 2024-06-04T04:46:22.671 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.354+0000 7fb10b7fe700 1 --2- 172.21.15.70:0/2800053004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c108ed0 0x7fb10c19f070 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc003e50 tx=0x7fb0fc011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:22.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.355+0000 7fb108ff9700 1 -- 172.21.15.70:0/2800053004 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0fc003a60 con 0x7fb10c108ed0 2024-06-04T04:46:22.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.355+0000 7fb108ff9700 1 -- 172.21.15.70:0/2800053004 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb0fc01b8d0 con 0x7fb10c108ed0 2024-06-04T04:46:22.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.355+0000 7fb108ff9700 1 -- 172.21.15.70:0/2800053004 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0fc0237d0 con 0x7fb10c108ed0 2024-06-04T04:46:22.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.355+0000 7fb111d02700 1 -- 172.21.15.70:0/2800053004 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb10c19f840 con 0x7fb10c108ed0 2024-06-04T04:46:22.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.355+0000 7fb111d02700 1 -- 172.21.15.70:0/2800053004 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb10c1a2530 con 0x7fb10c108ed0 2024-06-04T04:46:22.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.356+0000 7fb108ff9700 1 -- 172.21.15.70:0/2800053004 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7fb0fc01b440 con 0x7fb10c108ed0 2024-06-04T04:46:22.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.356+0000 7fb111d02700 1 -- 172.21.15.70:0/2800053004 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb10c064ba0 con 0x7fb10c108ed0 2024-06-04T04:46:22.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.356+0000 7fb108ff9700 1 --2- 172.21.15.70:0/2800053004 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb0f403e4d0 0x7fb0f4040990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:22.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.356+0000 7fb108ff9700 1 -- 172.21.15.70:0/2800053004 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb0fc0541f0 con 0x7fb10c108ed0 2024-06-04T04:46:22.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.356+0000 7fb10affd700 1 --2- 172.21.15.70:0/2800053004 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb0f403e4d0 0x7fb0f4040990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:22.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.357+0000 7fb10affd700 1 --2- 172.21.15.70:0/2800053004 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb0f403e4d0 0x7fb0f4040990 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb100006fd0 tx=0x7fb100006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:22.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.362+0000 7fb108ff9700 1 -- 172.21.15.70:0/2800053004 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb0fc01f030 con 0x7fb10c108ed0 2024-06-04T04:46:22.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.594+0000 7fb111d02700 1 -- 172.21.15.70:0/2800053004 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fb10c10d3d0 con 0x7fb0f403e4d0 2024-06-04T04:46:22.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.819+0000 7fb108ff9700 1 -- 172.21.15.70:0/2800053004 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fb10c10d3d0 con 0x7fb0f403e4d0 2024-06-04T04:46:22.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.821+0000 7fb111d02700 1 -- 172.21.15.70:0/2800053004 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb0f403e4d0 msgr2=0x7fb0f4040990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:22.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.821+0000 7fb111d02700 1 --2- 172.21.15.70:0/2800053004 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb0f403e4d0 0x7fb0f4040990 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb100006fd0 tx=0x7fb100006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:22.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.821+0000 7fb111d02700 1 -- 172.21.15.70:0/2800053004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c108ed0 msgr2=0x7fb10c19f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:22.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.821+0000 7fb111d02700 1 --2- 172.21.15.70:0/2800053004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c108ed0 0x7fb10c19f070 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc003e50 tx=0x7fb0fc011a30 comp rx=0 tx=0).stop 2024-06-04T04:46:22.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.821+0000 7fb111d02700 1 -- 172.21.15.70:0/2800053004 shutdown_connections 2024-06-04T04:46:22.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.821+0000 7fb111d02700 1 --2- 172.21.15.70:0/2800053004 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb0f403e4d0 0x7fb0f4040990 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:22.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.821+0000 7fb111d02700 1 --2- 172.21.15.70:0/2800053004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c108ed0 0x7fb10c19f070 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:22.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.821+0000 7fb111d02700 1 -- 172.21.15.70:0/2800053004 >> 172.21.15.70:0/2800053004 conn(0x7fb10c104470 msgr2=0x7fb10c104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:22.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.821+0000 7fb111d02700 1 -- 172.21.15.70:0/2800053004 shutdown_connections 2024-06-04T04:46:22.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:21.821+0000 7fb111d02700 1 -- 172.21.15.70:0/2800053004 wait complete. 2024-06-04T04:46:22.674 INFO:teuthology.orchestra.run.smithi070.stdout:Fetching dashboard port number... 2024-06-04T04:46:22.912 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:22 smithi070 ceph-mon[32631]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:46:22.912 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:24.010 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 8443 2024-06-04T04:46:24.010 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.105+0000 7f62fd968700 1 Processor -- start 2024-06-04T04:46:24.010 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.105+0000 7f62fd968700 1 -- start start 2024-06-04T04:46:24.010 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.105+0000 7f62fd968700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62f807ed50 0x7f62f807d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:24.010 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.105+0000 7f62fd968700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62f807d800 con 0x7f62f807ed50 2024-06-04T04:46:24.011 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.105+0000 7f62f6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62f807ed50 0x7f62f807d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:24.011 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.105+0000 7f62f6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62f807ed50 0x7f62f807d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52868/0 (socket says 172.21.15.70:52868) 2024-06-04T04:46:24.011 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.105+0000 7f62f6ffd700 1 -- 172.21.15.70:0/3352995239 learned_addr learned my addr 172.21.15.70:0/3352995239 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:24.011 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.105+0000 7f62f6ffd700 1 -- 172.21.15.70:0/3352995239 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f807d980 con 0x7f62f807ed50 2024-06-04T04:46:24.011 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.105+0000 7f62f6ffd700 1 --2- 172.21.15.70:0/3352995239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62f807ed50 0x7f62f807d2c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f62e80098a0 tx=0x7f62e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0c1b0b0359e35b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:24.011 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.106+0000 7f62f5ffb700 1 -- 172.21.15.70:0/3352995239 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62e8004160 con 0x7f62f807ed50 2024-06-04T04:46:24.011 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.106+0000 7f62f5ffb700 1 -- 172.21.15.70:0/3352995239 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f62e8004300 con 0x7f62f807ed50 2024-06-04T04:46:24.011 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.106+0000 7f62f5ffb700 1 -- 172.21.15.70:0/3352995239 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62e8011960 con 0x7f62f807ed50 2024-06-04T04:46:24.011 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.106+0000 7f62fd968700 1 -- 172.21.15.70:0/3352995239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62f807ed50 msgr2=0x7f62f807d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:24.011 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.106+0000 7f62fd968700 1 --2- 172.21.15.70:0/3352995239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62f807ed50 0x7f62f807d2c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f62e80098a0 tx=0x7f62e8009bb0 comp rx=0 tx=0).stop 2024-06-04T04:46:24.011 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.106+0000 7f62fd968700 1 -- 172.21.15.70:0/3352995239 shutdown_connections 2024-06-04T04:46:24.011 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.106+0000 7f62fd968700 1 --2- 172.21.15.70:0/3352995239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62f807ed50 0x7f62f807d2c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:24.012 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.106+0000 7f62fd968700 1 -- 172.21.15.70:0/3352995239 >> 172.21.15.70:0/3352995239 conn(0x7f62f81044b0 msgr2=0x7f62f81068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:24.012 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.106+0000 7f62fd968700 1 -- 172.21.15.70:0/3352995239 shutdown_connections 2024-06-04T04:46:24.012 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.106+0000 7f62fd968700 1 -- 172.21.15.70:0/3352995239 wait complete. 2024-06-04T04:46:24.012 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.107+0000 7f62fd968700 1 Processor -- start 2024-06-04T04:46:24.012 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.107+0000 7f62fd968700 1 -- start start 2024-06-04T04:46:24.012 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.107+0000 7f62fd968700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62f807ed50 0x7f62f819eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:24.012 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.107+0000 7f62fd968700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62f810f2e0 con 0x7f62f807ed50 2024-06-04T04:46:24.012 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.107+0000 7f62f6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62f807ed50 0x7f62f819eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:24.012 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.107+0000 7f62f6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62f807ed50 0x7f62f819eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52870/0 (socket says 172.21.15.70:52870) 2024-06-04T04:46:24.012 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.107+0000 7f62f6ffd700 1 -- 172.21.15.70:0/2740292605 learned_addr learned my addr 172.21.15.70:0/2740292605 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:24.012 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.108+0000 7f62f6ffd700 1 -- 172.21.15.70:0/2740292605 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f819f3f0 con 0x7f62f807ed50 2024-06-04T04:46:24.012 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.108+0000 7f62f6ffd700 1 --2- 172.21.15.70:0/2740292605 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62f807ed50 0x7f62f819eeb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f62e8011670 tx=0x7f62e8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:24.012 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.108+0000 7f62fc966700 1 -- 172.21.15.70:0/2740292605 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62e8003a60 con 0x7f62f807ed50 2024-06-04T04:46:24.012 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.108+0000 7f62fc966700 1 -- 172.21.15.70:0/2740292605 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f62e801b8d0 con 0x7f62f807ed50 2024-06-04T04:46:24.013 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.108+0000 7f62fd968700 1 -- 172.21.15.70:0/2740292605 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62f819f680 con 0x7f62f807ed50 2024-06-04T04:46:24.013 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.108+0000 7f62fc966700 1 -- 172.21.15.70:0/2740292605 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62e8023790 con 0x7f62f807ed50 2024-06-04T04:46:24.013 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.108+0000 7f62fd968700 1 -- 172.21.15.70:0/2740292605 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62f81a2370 con 0x7f62f807ed50 2024-06-04T04:46:24.013 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.109+0000 7f62fc966700 1 -- 172.21.15.70:0/2740292605 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f62e8023a60 con 0x7f62f807ed50 2024-06-04T04:46:24.013 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.109+0000 7f62fc966700 1 --2- 172.21.15.70:0/2740292605 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f62e003e520 0x7f62e00409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:24.013 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.109+0000 7f62fc966700 1 -- 172.21.15.70:0/2740292605 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f62e8054050 con 0x7f62f807ed50 2024-06-04T04:46:24.013 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.109+0000 7f62fd968700 1 -- 172.21.15.70:0/2740292605 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62f8064ba0 con 0x7f62f807ed50 2024-06-04T04:46:24.013 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.109+0000 7f62f67fc700 1 --2- 172.21.15.70:0/2740292605 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f62e003e520 0x7f62e00409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:24.013 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.110+0000 7f62f67fc700 1 --2- 172.21.15.70:0/2740292605 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f62e003e520 0x7f62e00409e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f62ec006fd0 tx=0x7f62ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:24.013 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.114+0000 7f62fc966700 1 -- 172.21.15.70:0/2740292605 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f62e801f030 con 0x7f62f807ed50 2024-06-04T04:46:24.013 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.329+0000 7f62fd968700 1 -- 172.21.15.70:0/2740292605 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f62f81a2720 con 0x7f62f807ed50 2024-06-04T04:46:24.013 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.330+0000 7f62fc966700 1 -- 172.21.15.70:0/2740292605 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f62e8019e20 con 0x7f62f807ed50 2024-06-04T04:46:24.014 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.332+0000 7f62fd968700 1 -- 172.21.15.70:0/2740292605 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f62e003e520 msgr2=0x7f62e00409e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:24.014 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.332+0000 7f62fd968700 1 --2- 172.21.15.70:0/2740292605 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f62e003e520 0x7f62e00409e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f62ec006fd0 tx=0x7f62ec006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:24.014 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.332+0000 7f62fd968700 1 -- 172.21.15.70:0/2740292605 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62f807ed50 msgr2=0x7f62f819eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:24.014 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.332+0000 7f62fd968700 1 --2- 172.21.15.70:0/2740292605 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62f807ed50 0x7f62f819eeb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f62e8011670 tx=0x7f62e8011a30 comp rx=0 tx=0).stop 2024-06-04T04:46:24.014 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.332+0000 7f62fd968700 1 -- 172.21.15.70:0/2740292605 shutdown_connections 2024-06-04T04:46:24.014 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.332+0000 7f62fd968700 1 --2- 172.21.15.70:0/2740292605 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f62e003e520 0x7f62e00409e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:24.014 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.332+0000 7f62fd968700 1 --2- 172.21.15.70:0/2740292605 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62f807ed50 0x7f62f819eeb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:24.014 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.332+0000 7f62fd968700 1 -- 172.21.15.70:0/2740292605 >> 172.21.15.70:0/2740292605 conn(0x7f62f81044b0 msgr2=0x7f62f8104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:24.014 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.332+0000 7f62fd968700 1 -- 172.21.15.70:0/2740292605 shutdown_connections 2024-06-04T04:46:24.014 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:23.332+0000 7f62fd968700 1 -- 172.21.15.70:0/2740292605 wait complete. 2024-06-04T04:46:24.017 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-04T04:46:24.017 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout disabled 2024-06-04T04:46:24.024 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-04T04:46:24.025 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-06-04T04:46:24.025 INFO:teuthology.orchestra.run.smithi070.stdout:firewalld.service is not enabled 2024-06-04T04:46:24.025 INFO:teuthology.orchestra.run.smithi070.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-06-04T04:46:24.026 INFO:teuthology.orchestra.run.smithi070.stdout:Ceph Dashboard is now available at: 2024-06-04T04:46:24.026 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:46:24.027 INFO:teuthology.orchestra.run.smithi070.stdout: URL: https://smithi070.front.sepia.ceph.com:8443/ 2024-06-04T04:46:24.027 INFO:teuthology.orchestra.run.smithi070.stdout: User: admin 2024-06-04T04:46:24.027 INFO:teuthology.orchestra.run.smithi070.stdout: Password: 67u6826aou 2024-06-04T04:46:24.027 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:46:24.028 INFO:teuthology.orchestra.run.smithi070.stdout:Saving cluster configuration to /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config directory 2024-06-04T04:46:24.028 INFO:teuthology.orchestra.run.smithi070.stdout:Enabling autotune for osd_memory_target 2024-06-04T04:46:24.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:23 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2740292605' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-06-04T04:46:24.942 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:24 smithi070 ceph-mon[32631]: mgrmap e13: x(active, since 6s) 2024-06-04T04:46:25.207 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.431+0000 7fa96e839700 1 Processor -- start 2024-06-04T04:46:25.207 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.431+0000 7fa96e839700 1 -- start start 2024-06-04T04:46:25.207 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.431+0000 7fa96e839700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa968108ea0 0x7fa9681092c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:25.207 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.431+0000 7fa96e839700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa968109800 con 0x7fa968108ea0 2024-06-04T04:46:25.207 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.431+0000 7fa967fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa968108ea0 0x7fa9681092c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:25.207 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.431+0000 7fa967fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa968108ea0 0x7fa9681092c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52872/0 (socket says 172.21.15.70:52872) 2024-06-04T04:46:25.208 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.431+0000 7fa967fff700 1 -- 172.21.15.70:0/509341296 learned_addr learned my addr 172.21.15.70:0/509341296 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:25.208 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.432+0000 7fa967fff700 1 -- 172.21.15.70:0/509341296 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa968109980 con 0x7fa968108ea0 2024-06-04T04:46:25.208 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.432+0000 7fa967fff700 1 --2- 172.21.15.70:0/509341296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa968108ea0 0x7fa9681092c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa9580098a0 tx=0x7fa958009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1709b1a69ee285c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:25.208 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.432+0000 7fa966ffd700 1 -- 172.21.15.70:0/509341296 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa958004160 con 0x7fa968108ea0 2024-06-04T04:46:25.208 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.432+0000 7fa966ffd700 1 -- 172.21.15.70:0/509341296 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa958004300 con 0x7fa968108ea0 2024-06-04T04:46:25.208 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.432+0000 7fa966ffd700 1 -- 172.21.15.70:0/509341296 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa958011960 con 0x7fa968108ea0 2024-06-04T04:46:25.208 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.432+0000 7fa96e839700 1 -- 172.21.15.70:0/509341296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa968108ea0 msgr2=0x7fa9681092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:25.208 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.432+0000 7fa96e839700 1 --2- 172.21.15.70:0/509341296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa968108ea0 0x7fa9681092c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa9580098a0 tx=0x7fa958009bb0 comp rx=0 tx=0).stop 2024-06-04T04:46:25.208 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.433+0000 7fa96e839700 1 -- 172.21.15.70:0/509341296 shutdown_connections 2024-06-04T04:46:25.208 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.433+0000 7fa96e839700 1 --2- 172.21.15.70:0/509341296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa968108ea0 0x7fa9681092c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:25.208 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.433+0000 7fa96e839700 1 -- 172.21.15.70:0/509341296 >> 172.21.15.70:0/509341296 conn(0x7fa9681044e0 msgr2=0x7fa968106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:25.208 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.433+0000 7fa96e839700 1 -- 172.21.15.70:0/509341296 shutdown_connections 2024-06-04T04:46:25.208 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.433+0000 7fa96e839700 1 -- 172.21.15.70:0/509341296 wait complete. 2024-06-04T04:46:25.209 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.433+0000 7fa96e839700 1 Processor -- start 2024-06-04T04:46:25.209 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.433+0000 7fa96e839700 1 -- start start 2024-06-04T04:46:25.209 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.434+0000 7fa96e839700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa968108ea0 0x7fa96819cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:25.209 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.434+0000 7fa96e839700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa96810ae80 con 0x7fa968108ea0 2024-06-04T04:46:25.209 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.434+0000 7fa967fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa968108ea0 0x7fa96819cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:25.209 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.434+0000 7fa967fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa968108ea0 0x7fa96819cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52876/0 (socket says 172.21.15.70:52876) 2024-06-04T04:46:25.209 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.434+0000 7fa967fff700 1 -- 172.21.15.70:0/2729168016 learned_addr learned my addr 172.21.15.70:0/2729168016 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:25.209 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.434+0000 7fa967fff700 1 -- 172.21.15.70:0/2729168016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa96819d1b0 con 0x7fa968108ea0 2024-06-04T04:46:25.209 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.434+0000 7fa967fff700 1 --2- 172.21.15.70:0/2729168016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa968108ea0 0x7fa96819cc70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa958011980 tx=0x7fa958011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:25.209 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.434+0000 7fa9657fa700 1 -- 172.21.15.70:0/2729168016 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa9580039e0 con 0x7fa968108ea0 2024-06-04T04:46:25.209 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.434+0000 7fa9657fa700 1 -- 172.21.15.70:0/2729168016 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa95801b8d0 con 0x7fa968108ea0 2024-06-04T04:46:25.209 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.434+0000 7fa96e839700 1 -- 172.21.15.70:0/2729168016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa96819d4a0 con 0x7fa968108ea0 2024-06-04T04:46:25.209 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.434+0000 7fa9657fa700 1 -- 172.21.15.70:0/2729168016 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa958023790 con 0x7fa968108ea0 2024-06-04T04:46:25.210 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.434+0000 7fa96e839700 1 -- 172.21.15.70:0/2729168016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9681a0130 con 0x7fa968108ea0 2024-06-04T04:46:25.210 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.435+0000 7fa9657fa700 1 -- 172.21.15.70:0/2729168016 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fa958023930 con 0x7fa968108ea0 2024-06-04T04:46:25.210 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.435+0000 7fa9657fa700 1 --2- 172.21.15.70:0/2729168016 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa95003e570 0x7fa950040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:25.210 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.436+0000 7fa9657fa700 1 -- 172.21.15.70:0/2729168016 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa95801ba80 con 0x7fa968108ea0 2024-06-04T04:46:25.210 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.436+0000 7fa96e839700 1 -- 172.21.15.70:0/2729168016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa96810d310 con 0x7fa968108ea0 2024-06-04T04:46:25.210 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.436+0000 7fa9677fe700 1 --2- 172.21.15.70:0/2729168016 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa95003e570 0x7fa950040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:25.210 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.436+0000 7fa9677fe700 1 --2- 172.21.15.70:0/2729168016 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa95003e570 0x7fa950040a30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa95c006fd0 tx=0x7fa95c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:25.210 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.441+0000 7fa9657fa700 1 -- 172.21.15.70:0/2729168016 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa95801f030 con 0x7fa968108ea0 2024-06-04T04:46:25.210 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.641+0000 7fa96e839700 1 -- 172.21.15.70:0/2729168016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fa968064ba0 con 0x7fa968108ea0 2024-06-04T04:46:25.210 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.642+0000 7fa9657fa700 1 -- 172.21.15.70:0/2729168016 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fa958018df0 con 0x7fa968108ea0 2024-06-04T04:46:25.210 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.644+0000 7fa96e839700 1 -- 172.21.15.70:0/2729168016 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa95003e570 msgr2=0x7fa950040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:25.210 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.644+0000 7fa96e839700 1 --2- 172.21.15.70:0/2729168016 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa95003e570 0x7fa950040a30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa95c006fd0 tx=0x7fa95c006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:25.210 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.644+0000 7fa96e839700 1 -- 172.21.15.70:0/2729168016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa968108ea0 msgr2=0x7fa96819cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:25.211 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.644+0000 7fa96e839700 1 --2- 172.21.15.70:0/2729168016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa968108ea0 0x7fa96819cc70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa958011980 tx=0x7fa958011a60 comp rx=0 tx=0).stop 2024-06-04T04:46:25.211 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.644+0000 7fa96e839700 1 -- 172.21.15.70:0/2729168016 shutdown_connections 2024-06-04T04:46:25.211 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.644+0000 7fa96e839700 1 --2- 172.21.15.70:0/2729168016 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa95003e570 0x7fa950040a30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:25.211 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.644+0000 7fa96e839700 1 --2- 172.21.15.70:0/2729168016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa968108ea0 0x7fa96819cc70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:25.211 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.644+0000 7fa96e839700 1 -- 172.21.15.70:0/2729168016 >> 172.21.15.70:0/2729168016 conn(0x7fa9681044e0 msgr2=0x7fa968104f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:25.211 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.644+0000 7fa96e839700 1 -- 172.21.15.70:0/2729168016 shutdown_connections 2024-06-04T04:46:25.211 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:24.644+0000 7fa96e839700 1 -- 172.21.15.70:0/2729168016 wait complete. 2024-06-04T04:46:26.582 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.669+0000 7f20b8cfb700 1 Processor -- start 2024-06-04T04:46:26.582 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.669+0000 7f20b8cfb700 1 -- start start 2024-06-04T04:46:26.582 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.669+0000 7f20b8cfb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20b4108ee0 0x7f20b4109300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:26.582 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.669+0000 7f20b8cfb700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f20b4109840 con 0x7f20b4108ee0 2024-06-04T04:46:26.582 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.669+0000 7f20b259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20b4108ee0 0x7f20b4109300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:26.582 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.669+0000 7f20b259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20b4108ee0 0x7f20b4109300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52882/0 (socket says 172.21.15.70:52882) 2024-06-04T04:46:26.582 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.669+0000 7f20b259c700 1 -- 172.21.15.70:0/949150668 learned_addr learned my addr 172.21.15.70:0/949150668 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:26.582 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.670+0000 7f20b259c700 1 -- 172.21.15.70:0/949150668 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b4109fa0 con 0x7f20b4108ee0 2024-06-04T04:46:26.582 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.670+0000 7f20b259c700 1 --2- 172.21.15.70:0/949150668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20b4108ee0 0x7f20b4109300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f20a40098a0 tx=0x7f20a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ced4ed0cc19f4a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:26.582 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.670+0000 7f20b159a700 1 -- 172.21.15.70:0/949150668 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20a4004160 con 0x7f20b4108ee0 2024-06-04T04:46:26.582 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.670+0000 7f20b159a700 1 -- 172.21.15.70:0/949150668 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f20a4004300 con 0x7f20b4108ee0 2024-06-04T04:46:26.583 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.670+0000 7f20b159a700 1 -- 172.21.15.70:0/949150668 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20a4011960 con 0x7f20b4108ee0 2024-06-04T04:46:26.583 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.670+0000 7f20b8cfb700 1 -- 172.21.15.70:0/949150668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20b4108ee0 msgr2=0x7f20b4109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:26.583 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.670+0000 7f20b8cfb700 1 --2- 172.21.15.70:0/949150668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20b4108ee0 0x7f20b4109300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f20a40098a0 tx=0x7f20a4009bb0 comp rx=0 tx=0).stop 2024-06-04T04:46:26.583 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.671+0000 7f20b8cfb700 1 -- 172.21.15.70:0/949150668 shutdown_connections 2024-06-04T04:46:26.583 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.671+0000 7f20b8cfb700 1 --2- 172.21.15.70:0/949150668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20b4108ee0 0x7f20b4109300 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:26.583 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.671+0000 7f20b8cfb700 1 -- 172.21.15.70:0/949150668 >> 172.21.15.70:0/949150668 conn(0x7f20b4104460 msgr2=0x7f20b41068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:26.583 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.671+0000 7f20b8cfb700 1 -- 172.21.15.70:0/949150668 shutdown_connections 2024-06-04T04:46:26.583 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.671+0000 7f20b8cfb700 1 -- 172.21.15.70:0/949150668 wait complete. 2024-06-04T04:46:26.583 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.671+0000 7f20b8cfb700 1 Processor -- start 2024-06-04T04:46:26.583 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.671+0000 7f20b8cfb700 1 -- start start 2024-06-04T04:46:26.583 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.672+0000 7f20b8cfb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20b4108ee0 0x7f20b419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:26.584 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.672+0000 7f20b8cfb700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f20b410d260 con 0x7f20b4108ee0 2024-06-04T04:46:26.584 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.672+0000 7f20b259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20b4108ee0 0x7f20b419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:26.584 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.672+0000 7f20b259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20b4108ee0 0x7f20b419f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52886/0 (socket says 172.21.15.70:52886) 2024-06-04T04:46:26.584 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.672+0000 7f20b259c700 1 -- 172.21.15.70:0/4234174579 learned_addr learned my addr 172.21.15.70:0/4234174579 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:26.584 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.672+0000 7f20b259c700 1 -- 172.21.15.70:0/4234174579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b419f560 con 0x7f20b4108ee0 2024-06-04T04:46:26.584 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.672+0000 7f20b259c700 1 --2- 172.21.15.70:0/4234174579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20b4108ee0 0x7f20b419f020 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f20a4003e50 tx=0x7f20a4011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:26.584 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.672+0000 7f20a37fe700 1 -- 172.21.15.70:0/4234174579 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20a4003a60 con 0x7f20b4108ee0 2024-06-04T04:46:26.584 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.672+0000 7f20a37fe700 1 -- 172.21.15.70:0/4234174579 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f20a401b8d0 con 0x7f20b4108ee0 2024-06-04T04:46:26.584 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.672+0000 7f20b8cfb700 1 -- 172.21.15.70:0/4234174579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20b419f7f0 con 0x7f20b4108ee0 2024-06-04T04:46:26.584 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.672+0000 7f20a37fe700 1 -- 172.21.15.70:0/4234174579 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20a40237d0 con 0x7f20b4108ee0 2024-06-04T04:46:26.584 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.672+0000 7f20b8cfb700 1 -- 172.21.15.70:0/4234174579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20b41a24e0 con 0x7f20b4108ee0 2024-06-04T04:46:26.584 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.673+0000 7f20a37fe700 1 -- 172.21.15.70:0/4234174579 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f20a401b440 con 0x7f20b4108ee0 2024-06-04T04:46:26.584 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.673+0000 7f20b8cfb700 1 -- 172.21.15.70:0/4234174579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20b4064ba0 con 0x7f20b4108ee0 2024-06-04T04:46:26.585 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.673+0000 7f20a37fe700 1 --2- 172.21.15.70:0/4234174579 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f209c03e570 0x7f209c040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:26.585 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.674+0000 7f20a37fe700 1 -- 172.21.15.70:0/4234174579 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f20a40541f0 con 0x7f20b4108ee0 2024-06-04T04:46:26.585 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.674+0000 7f20b1d9b700 1 --2- 172.21.15.70:0/4234174579 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f209c03e570 0x7f209c040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:26.585 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.674+0000 7f20b1d9b700 1 --2- 172.21.15.70:0/4234174579 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f209c03e570 0x7f209c040a30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f20a8006fd0 tx=0x7f20a8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:26.585 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.678+0000 7f20a37fe700 1 -- 172.21.15.70:0/4234174579 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f20a401f030 con 0x7f20b4108ee0 2024-06-04T04:46:26.585 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.992+0000 7f20b8cfb700 1 -- 172.21.15.70:0/4234174579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f20b41a27c0 con 0x7f20b4108ee0 2024-06-04T04:46:26.585 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.995+0000 7f20a37fe700 1 -- 172.21.15.70:0/4234174579 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f20a4017e90 con 0x7f20b4108ee0 2024-06-04T04:46:26.585 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.997+0000 7f20b8cfb700 1 -- 172.21.15.70:0/4234174579 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f209c03e570 msgr2=0x7f209c040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:26.585 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.997+0000 7f20b8cfb700 1 --2- 172.21.15.70:0/4234174579 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f209c03e570 0x7f209c040a30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f20a8006fd0 tx=0x7f20a8006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:26.585 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.997+0000 7f20b8cfb700 1 -- 172.21.15.70:0/4234174579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20b4108ee0 msgr2=0x7f20b419f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:26.585 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.997+0000 7f20b8cfb700 1 --2- 172.21.15.70:0/4234174579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20b4108ee0 0x7f20b419f020 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f20a4003e50 tx=0x7f20a4011a30 comp rx=0 tx=0).stop 2024-06-04T04:46:26.585 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.997+0000 7f20b8cfb700 1 -- 172.21.15.70:0/4234174579 shutdown_connections 2024-06-04T04:46:26.585 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.997+0000 7f20b8cfb700 1 --2- 172.21.15.70:0/4234174579 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f209c03e570 0x7f209c040a30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:26.586 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.997+0000 7f20b8cfb700 1 --2- 172.21.15.70:0/4234174579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20b4108ee0 0x7f20b419f020 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:26.586 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.997+0000 7f20b8cfb700 1 -- 172.21.15.70:0/4234174579 >> 172.21.15.70:0/4234174579 conn(0x7f20b4104460 msgr2=0x7f20b4104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:26.586 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.997+0000 7f20b8cfb700 1 -- 172.21.15.70:0/4234174579 shutdown_connections 2024-06-04T04:46:26.586 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-06-04T04:46:25.997+0000 7f20b8cfb700 1 -- 172.21.15.70:0/4234174579 wait complete. 2024-06-04T04:46:26.586 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-06-04T04:46:26.586 INFO:teuthology.orchestra.run.smithi070.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-06-04T04:46:26.586 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:46:26.586 INFO:teuthology.orchestra.run.smithi070.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-06-04T04:46:26.586 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:46:26.586 INFO:teuthology.orchestra.run.smithi070.stdout:Or, if you are only running a single cluster on this host: 2024-06-04T04:46:26.586 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:46:26.586 INFO:teuthology.orchestra.run.smithi070.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-06-04T04:46:26.586 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:46:26.586 INFO:teuthology.orchestra.run.smithi070.stdout:Please consider enabling telemetry to help improve Ceph: 2024-06-04T04:46:26.587 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:46:26.587 INFO:teuthology.orchestra.run.smithi070.stdout: ceph telemetry on 2024-06-04T04:46:26.587 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:46:26.587 INFO:teuthology.orchestra.run.smithi070.stdout:For more information see: 2024-06-04T04:46:26.587 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:46:26.587 INFO:teuthology.orchestra.run.smithi070.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-06-04T04:46:26.587 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:46:26.587 INFO:teuthology.orchestra.run.smithi070.stdout:Bootstrap complete. 2024-06-04T04:46:26.626 INFO:tasks.cephadm:Fetching config... 2024-06-04T04:46:26.627 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T04:46:26.627 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-06-04T04:46:26.643 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-06-04T04:46:26.643 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T04:46:26.643 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-06-04T04:46:26.700 INFO:tasks.cephadm:Fetching mon keyring... 2024-06-04T04:46:26.700 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T04:46:26.700 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/keyring of=/dev/stdout 2024-06-04T04:46:26.765 INFO:tasks.cephadm:Fetching pub ssh key... 2024-06-04T04:46:26.765 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T04:46:26.765 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-06-04T04:46:26.822 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-06-04T04:46:26.822 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg5CwZGXZLnV1TcuIPz0z5UDu4ADBqOutmIkMwVsZ4wRJEDtP8oTRYtubDPA2pMHnfTbE6VbeV/E65ir1mONox67UC/ks3B/sVFdKm7h63Dwd8k0yP0gg5k6zvzU3lUD45T2XeUlnuM1pjJJPHSixOJ7Of2T5faYtRGZMkPB/MWkicagFj9oSMNyci7VljguWMARuXsOnfcq8irgmUB24p3g4JjDsq+SkTVxj0c6XzoFgpE+NTvF1jlsINnKZT4Pgw9R5vfu9lFIpRDT7A+HYDAw7ZcrcxqJdIVo18pchKZj0PvXc1nPln4x7E/19BgpJyeujhEqXuVQAEH5B7Uq40uZ+TENdWWy+kp2O1eFUBM5ryzgWTLFCsGt7A/g9VCsBoAhDWY9PXbOIDb0RdyRDK/xXs+Ryv8oio6/dWD6YPB3b/ynhmQyGl+xvHcv1mCyoMWSj2SGrrXWgArIYya115d6J8CfmnTWODJ6rHjngENspR+Zc9L6znbKFExZWJPFM= ceph-164dbaba-222d-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-04T04:46:26.898 INFO:teuthology.orchestra.run.smithi070.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg5CwZGXZLnV1TcuIPz0z5UDu4ADBqOutmIkMwVsZ4wRJEDtP8oTRYtubDPA2pMHnfTbE6VbeV/E65ir1mONox67UC/ks3B/sVFdKm7h63Dwd8k0yP0gg5k6zvzU3lUD45T2XeUlnuM1pjJJPHSixOJ7Of2T5faYtRGZMkPB/MWkicagFj9oSMNyci7VljguWMARuXsOnfcq8irgmUB24p3g4JjDsq+SkTVxj0c6XzoFgpE+NTvF1jlsINnKZT4Pgw9R5vfu9lFIpRDT7A+HYDAw7ZcrcxqJdIVo18pchKZj0PvXc1nPln4x7E/19BgpJyeujhEqXuVQAEH5B7Uq40uZ+TENdWWy+kp2O1eFUBM5ryzgWTLFCsGt7A/g9VCsBoAhDWY9PXbOIDb0RdyRDK/xXs+Ryv8oio6/dWD6YPB3b/ynhmQyGl+xvHcv1mCyoMWSj2SGrrXWgArIYya115d6J8CfmnTWODJ6rHjngENspR+Zc9L6znbKFExZWJPFM= ceph-164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:46:26.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg5CwZGXZLnV1TcuIPz0z5UDu4ADBqOutmIkMwVsZ4wRJEDtP8oTRYtubDPA2pMHnfTbE6VbeV/E65ir1mONox67UC/ks3B/sVFdKm7h63Dwd8k0yP0gg5k6zvzU3lUD45T2XeUlnuM1pjJJPHSixOJ7Of2T5faYtRGZMkPB/MWkicagFj9oSMNyci7VljguWMARuXsOnfcq8irgmUB24p3g4JjDsq+SkTVxj0c6XzoFgpE+NTvF1jlsINnKZT4Pgw9R5vfu9lFIpRDT7A+HYDAw7ZcrcxqJdIVo18pchKZj0PvXc1nPln4x7E/19BgpJyeujhEqXuVQAEH5B7Uq40uZ+TENdWWy+kp2O1eFUBM5ryzgWTLFCsGt7A/g9VCsBoAhDWY9PXbOIDb0RdyRDK/xXs+Ryv8oio6/dWD6YPB3b/ynhmQyGl+xvHcv1mCyoMWSj2SGrrXWgArIYya115d6J8CfmnTWODJ6rHjngENspR+Zc9L6znbKFExZWJPFM= ceph-164dbaba-222d-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-04T04:46:26.950 INFO:teuthology.orchestra.run.smithi098.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg5CwZGXZLnV1TcuIPz0z5UDu4ADBqOutmIkMwVsZ4wRJEDtP8oTRYtubDPA2pMHnfTbE6VbeV/E65ir1mONox67UC/ks3B/sVFdKm7h63Dwd8k0yP0gg5k6zvzU3lUD45T2XeUlnuM1pjJJPHSixOJ7Of2T5faYtRGZMkPB/MWkicagFj9oSMNyci7VljguWMARuXsOnfcq8irgmUB24p3g4JjDsq+SkTVxj0c6XzoFgpE+NTvF1jlsINnKZT4Pgw9R5vfu9lFIpRDT7A+HYDAw7ZcrcxqJdIVo18pchKZj0PvXc1nPln4x7E/19BgpJyeujhEqXuVQAEH5B7Uq40uZ+TENdWWy+kp2O1eFUBM5ryzgWTLFCsGt7A/g9VCsBoAhDWY9PXbOIDb0RdyRDK/xXs+Ryv8oio6/dWD6YPB3b/ynhmQyGl+xvHcv1mCyoMWSj2SGrrXWgArIYya115d6J8CfmnTWODJ6rHjngENspR+Zc9L6znbKFExZWJPFM= ceph-164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:46:26.962 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg5CwZGXZLnV1TcuIPz0z5UDu4ADBqOutmIkMwVsZ4wRJEDtP8oTRYtubDPA2pMHnfTbE6VbeV/E65ir1mONox67UC/ks3B/sVFdKm7h63Dwd8k0yP0gg5k6zvzU3lUD45T2XeUlnuM1pjJJPHSixOJ7Of2T5faYtRGZMkPB/MWkicagFj9oSMNyci7VljguWMARuXsOnfcq8irgmUB24p3g4JjDsq+SkTVxj0c6XzoFgpE+NTvF1jlsINnKZT4Pgw9R5vfu9lFIpRDT7A+HYDAw7ZcrcxqJdIVo18pchKZj0PvXc1nPln4x7E/19BgpJyeujhEqXuVQAEH5B7Uq40uZ+TENdWWy+kp2O1eFUBM5ryzgWTLFCsGt7A/g9VCsBoAhDWY9PXbOIDb0RdyRDK/xXs+Ryv8oio6/dWD6YPB3b/ynhmQyGl+xvHcv1mCyoMWSj2SGrrXWgArIYya115d6J8CfmnTWODJ6rHjngENspR+Zc9L6znbKFExZWJPFM= ceph-164dbaba-222d-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-04T04:46:27.003 INFO:teuthology.orchestra.run.smithi112.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg5CwZGXZLnV1TcuIPz0z5UDu4ADBqOutmIkMwVsZ4wRJEDtP8oTRYtubDPA2pMHnfTbE6VbeV/E65ir1mONox67UC/ks3B/sVFdKm7h63Dwd8k0yP0gg5k6zvzU3lUD45T2XeUlnuM1pjJJPHSixOJ7Of2T5faYtRGZMkPB/MWkicagFj9oSMNyci7VljguWMARuXsOnfcq8irgmUB24p3g4JjDsq+SkTVxj0c6XzoFgpE+NTvF1jlsINnKZT4Pgw9R5vfu9lFIpRDT7A+HYDAw7ZcrcxqJdIVo18pchKZj0PvXc1nPln4x7E/19BgpJyeujhEqXuVQAEH5B7Uq40uZ+TENdWWy+kp2O1eFUBM5ryzgWTLFCsGt7A/g9VCsBoAhDWY9PXbOIDb0RdyRDK/xXs+Ryv8oio6/dWD6YPB3b/ynhmQyGl+xvHcv1mCyoMWSj2SGrrXWgArIYya115d6J8CfmnTWODJ6rHjngENspR+Zc9L6znbKFExZWJPFM= ceph-164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:46:27.016 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-06-04T04:46:27.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:26 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/4234174579' entity='client.admin' 2024-06-04T04:46:27.255 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:46:29.722 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:29 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:29.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:29 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:29.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:29 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi070", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:46:29.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:29 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:29.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:29 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:46:29.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:29 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:29.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.832+0000 7fe2862bf700 1 -- 172.21.15.70:0/2741912485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe280102fa0 msgr2=0x7fe2801033c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:29.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.832+0000 7fe2862bf700 1 --2- 172.21.15.70:0/2741912485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe280102fa0 0x7fe2801033c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe270009910 tx=0x7fe270009c20 comp rx=0 tx=0).stop 2024-06-04T04:46:29.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.832+0000 7fe2862bf700 1 -- 172.21.15.70:0/2741912485 shutdown_connections 2024-06-04T04:46:29.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.832+0000 7fe2862bf700 1 --2- 172.21.15.70:0/2741912485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe280102fa0 0x7fe2801033c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:29.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.832+0000 7fe2862bf700 1 -- 172.21.15.70:0/2741912485 >> 172.21.15.70:0/2741912485 conn(0x7fe2800fe540 msgr2=0x7fe280100980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:29.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.832+0000 7fe2862bf700 1 -- 172.21.15.70:0/2741912485 shutdown_connections 2024-06-04T04:46:29.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.832+0000 7fe2862bf700 1 -- 172.21.15.70:0/2741912485 wait complete. 2024-06-04T04:46:29.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.832+0000 7fe2862bf700 1 Processor -- start 2024-06-04T04:46:29.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.832+0000 7fe2862bf700 1 -- start start 2024-06-04T04:46:29.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.833+0000 7fe2862bf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe280102fa0 0x7fe280198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:29.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.833+0000 7fe2862bf700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe280108d90 con 0x7fe280102fa0 2024-06-04T04:46:29.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.833+0000 7fe27ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe280102fa0 0x7fe280198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:29.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.833+0000 7fe27ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe280102fa0 0x7fe280198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48424/0 (socket says 172.21.15.70:48424) 2024-06-04T04:46:29.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.833+0000 7fe27ffff700 1 -- 172.21.15.70:0/172946956 learned_addr learned my addr 172.21.15.70:0/172946956 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:29.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.833+0000 7fe27ffff700 1 -- 172.21.15.70:0/172946956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe280198a50 con 0x7fe280102fa0 2024-06-04T04:46:29.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.833+0000 7fe27ffff700 1 --2- 172.21.15.70:0/172946956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe280102fa0 0x7fe280198510 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe270005790 tx=0x7fe2700057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:29.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.833+0000 7fe27d7fa700 1 -- 172.21.15.70:0/172946956 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe270027070 con 0x7fe280102fa0 2024-06-04T04:46:29.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.834+0000 7fe27d7fa700 1 -- 172.21.15.70:0/172946956 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe27000bdf0 con 0x7fe280102fa0 2024-06-04T04:46:29.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.834+0000 7fe2862bf700 1 -- 172.21.15.70:0/172946956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe280198ce0 con 0x7fe280102fa0 2024-06-04T04:46:29.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.834+0000 7fe27d7fa700 1 -- 172.21.15.70:0/172946956 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe27000fe50 con 0x7fe280102fa0 2024-06-04T04:46:29.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.834+0000 7fe2862bf700 1 -- 172.21.15.70:0/172946956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe28019b9d0 con 0x7fe280102fa0 2024-06-04T04:46:29.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.834+0000 7fe27d7fa700 1 -- 172.21.15.70:0/172946956 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fe27002a070 con 0x7fe280102fa0 2024-06-04T04:46:29.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.835+0000 7fe27d7fa700 1 --2- 172.21.15.70:0/172946956 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe26803e6b0 0x7fe268040b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:29.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.835+0000 7fe27d7fa700 1 -- 172.21.15.70:0/172946956 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe270053960 con 0x7fe280102fa0 2024-06-04T04:46:29.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.835+0000 7fe27f7fe700 1 --2- 172.21.15.70:0/172946956 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe26803e6b0 0x7fe268040b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:29.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.835+0000 7fe2862bf700 1 -- 172.21.15.70:0/172946956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe280109510 con 0x7fe280102fa0 2024-06-04T04:46:29.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.835+0000 7fe27f7fe700 1 --2- 172.21.15.70:0/172946956 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe26803e6b0 0x7fe268040b70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe274006fd0 tx=0x7fe274006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:29.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:29.840+0000 7fe27d7fa700 1 -- 172.21.15.70:0/172946956 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe270031070 con 0x7fe280102fa0 2024-06-04T04:46:30.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:30.101+0000 7fe2862bf700 1 -- 172.21.15.70:0/172946956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fe28002e560 con 0x7fe280102fa0 2024-06-04T04:46:30.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:30.103+0000 7fe27d7fa700 1 -- 172.21.15.70:0/172946956 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fe270013070 con 0x7fe280102fa0 2024-06-04T04:46:30.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:30.106+0000 7fe2862bf700 1 -- 172.21.15.70:0/172946956 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe26803e6b0 msgr2=0x7fe268040b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:30.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:30.106+0000 7fe2862bf700 1 --2- 172.21.15.70:0/172946956 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe26803e6b0 0x7fe268040b70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe274006fd0 tx=0x7fe274006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:30.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:30.106+0000 7fe2862bf700 1 -- 172.21.15.70:0/172946956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe280102fa0 msgr2=0x7fe280198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:30.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:30.106+0000 7fe2862bf700 1 --2- 172.21.15.70:0/172946956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe280102fa0 0x7fe280198510 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe270005790 tx=0x7fe2700057c0 comp rx=0 tx=0).stop 2024-06-04T04:46:30.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:30.106+0000 7fe2862bf700 1 -- 172.21.15.70:0/172946956 shutdown_connections 2024-06-04T04:46:30.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:30.106+0000 7fe2862bf700 1 --2- 172.21.15.70:0/172946956 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe26803e6b0 0x7fe268040b70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:30.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:30.106+0000 7fe2862bf700 1 --2- 172.21.15.70:0/172946956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe280102fa0 0x7fe280198510 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:30.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:30.106+0000 7fe2862bf700 1 -- 172.21.15.70:0/172946956 >> 172.21.15.70:0/172946956 conn(0x7fe2800fe540 msgr2=0x7fe2800ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:30.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:30.106+0000 7fe2862bf700 1 -- 172.21.15.70:0/172946956 shutdown_connections 2024-06-04T04:46:30.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:30.106+0000 7fe2862bf700 1 -- 172.21.15.70:0/172946956 wait complete. 2024-06-04T04:46:30.744 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-06-04T04:46:30.744 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-06-04T04:46:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:46:31.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:31 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/172946956' entity='client.admin' 2024-06-04T04:46:31.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:31 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:46:31.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:31 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:32.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.671+0000 7f845551d700 1 -- 172.21.15.70:0/4156714132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8450103a40 msgr2=0x7f8450105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:32.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.671+0000 7f845551d700 1 --2- 172.21.15.70:0/4156714132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8450103a40 0x7f8450105e30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8440009910 tx=0x7f8440009c20 comp rx=0 tx=0).stop 2024-06-04T04:46:32.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.671+0000 7f845551d700 1 -- 172.21.15.70:0/4156714132 shutdown_connections 2024-06-04T04:46:32.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.671+0000 7f845551d700 1 --2- 172.21.15.70:0/4156714132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8450103a40 0x7f8450105e30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:32.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.671+0000 7f845551d700 1 -- 172.21.15.70:0/4156714132 >> 172.21.15.70:0/4156714132 conn(0x7f84500fd450 msgr2=0x7f84500ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:32.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.671+0000 7f845551d700 1 -- 172.21.15.70:0/4156714132 shutdown_connections 2024-06-04T04:46:32.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.671+0000 7f845551d700 1 -- 172.21.15.70:0/4156714132 wait complete. 2024-06-04T04:46:32.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.671+0000 7f845551d700 1 Processor -- start 2024-06-04T04:46:32.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.672+0000 7f845551d700 1 -- start start 2024-06-04T04:46:32.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.672+0000 7f845551d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8450103a40 0x7f84501984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:32.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.672+0000 7f845551d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8450108d40 con 0x7f8450103a40 2024-06-04T04:46:32.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.672+0000 7f844effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8450103a40 0x7f84501984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:32.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.672+0000 7f844effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8450103a40 0x7f84501984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48450/0 (socket says 172.21.15.70:48450) 2024-06-04T04:46:32.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.672+0000 7f844effd700 1 -- 172.21.15.70:0/1965471553 learned_addr learned my addr 172.21.15.70:0/1965471553 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:32.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.672+0000 7f844effd700 1 -- 172.21.15.70:0/1965471553 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84501989e0 con 0x7f8450103a40 2024-06-04T04:46:32.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.672+0000 7f844effd700 1 --2- 172.21.15.70:0/1965471553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8450103a40 0x7f84501984a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f844000b5c0 tx=0x7f8440005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:32.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.672+0000 7f844cff9700 1 -- 172.21.15.70:0/1965471553 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8440027070 con 0x7f8450103a40 2024-06-04T04:46:32.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.672+0000 7f844cff9700 1 -- 172.21.15.70:0/1965471553 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f844000f460 con 0x7f8450103a40 2024-06-04T04:46:32.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.672+0000 7f845551d700 1 -- 172.21.15.70:0/1965471553 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8450198cd0 con 0x7f8450103a40 2024-06-04T04:46:32.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.672+0000 7f844cff9700 1 -- 172.21.15.70:0/1965471553 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84400175d0 con 0x7f8450103a40 2024-06-04T04:46:32.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.673+0000 7f845551d700 1 -- 172.21.15.70:0/1965471553 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f845019b960 con 0x7f8450103a40 2024-06-04T04:46:32.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.673+0000 7f844cff9700 1 -- 172.21.15.70:0/1965471553 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f844002a070 con 0x7f8450103a40 2024-06-04T04:46:32.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.674+0000 7f845551d700 1 -- 172.21.15.70:0/1965471553 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84501094c0 con 0x7f8450103a40 2024-06-04T04:46:32.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.674+0000 7f844cff9700 1 --2- 172.21.15.70:0/1965471553 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f843c03e570 0x7f843c040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:32.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.674+0000 7f844cff9700 1 -- 172.21.15.70:0/1965471553 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8440053970 con 0x7f8450103a40 2024-06-04T04:46:32.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.674+0000 7f84467fc700 1 --2- 172.21.15.70:0/1965471553 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f843c03e570 0x7f843c040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:32.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.674+0000 7f84467fc700 1 --2- 172.21.15.70:0/1965471553 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f843c03e570 0x7f843c040a30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8438006fd0 tx=0x7f8438006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:32.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.679+0000 7f844cff9700 1 -- 172.21.15.70:0/1965471553 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8440013070 con 0x7f8450103a40 2024-06-04T04:46:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.887+0000 7f845551d700 1 -- 172.21.15.70:0/1965471553 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f845002e750 con 0x7f843c03e570 2024-06-04T04:46:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.890+0000 7f844cff9700 1 -- 172.21.15.70:0/1965471553 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f845002e750 con 0x7f843c03e570 2024-06-04T04:46:32.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.893+0000 7f845551d700 1 -- 172.21.15.70:0/1965471553 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f843c03e570 msgr2=0x7f843c040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:32.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.893+0000 7f845551d700 1 --2- 172.21.15.70:0/1965471553 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f843c03e570 0x7f843c040a30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8438006fd0 tx=0x7f8438006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:32.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.893+0000 7f845551d700 1 -- 172.21.15.70:0/1965471553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8450103a40 msgr2=0x7f84501984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:32.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.893+0000 7f845551d700 1 --2- 172.21.15.70:0/1965471553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8450103a40 0x7f84501984a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f844000b5c0 tx=0x7f8440005810 comp rx=0 tx=0).stop 2024-06-04T04:46:32.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.893+0000 7f845551d700 1 -- 172.21.15.70:0/1965471553 shutdown_connections 2024-06-04T04:46:32.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.893+0000 7f845551d700 1 --2- 172.21.15.70:0/1965471553 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f843c03e570 0x7f843c040a30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:32.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.893+0000 7f845551d700 1 --2- 172.21.15.70:0/1965471553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8450103a40 0x7f84501984a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:32.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.893+0000 7f845551d700 1 -- 172.21.15.70:0/1965471553 >> 172.21.15.70:0/1965471553 conn(0x7f84500fd450 msgr2=0x7f84500fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:32.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.893+0000 7f845551d700 1 -- 172.21.15.70:0/1965471553 shutdown_connections 2024-06-04T04:46:32.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:32.893+0000 7f845551d700 1 -- 172.21.15.70:0/1965471553 wait complete. 2024-06-04T04:46:33.811 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi098 2024-06-04T04:46:33.811 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-04T04:46:33.812 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/etc/ceph/ceph.conf 2024-06-04T04:46:33.830 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-04T04:46:33.830 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:46:33.888 INFO:tasks.cephadm:Adding host smithi098 to orchestrator... 2024-06-04T04:46:33.888 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch host add smithi098 2024-06-04T04:46:34.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:33 smithi070 ceph-mon[32631]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:46:34.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:34.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:46:34.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:46:34.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:46:34.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:33 smithi070 ceph-mon[32631]: Updating smithi070:/etc/ceph/ceph.conf 2024-06-04T04:46:34.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:33 smithi070 ceph-mon[32631]: Updating smithi070:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:46:34.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:33 smithi070 ceph-mon[32631]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:46:34.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:34.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:34.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:34.120 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:46:35.243 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:34 smithi070 ceph-mon[32631]: Updating smithi070:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-04T04:46:35.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.582+0000 7fc3953ce700 1 -- 172.21.15.70:0/1897854500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3901031e0 msgr2=0x7fc3901035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:35.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.582+0000 7fc3953ce700 1 --2- 172.21.15.70:0/1897854500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3901031e0 0x7fc3901035c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc380009910 tx=0x7fc380009c20 comp rx=0 tx=0).stop 2024-06-04T04:46:35.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.582+0000 7fc3953ce700 1 -- 172.21.15.70:0/1897854500 shutdown_connections 2024-06-04T04:46:35.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.582+0000 7fc3953ce700 1 --2- 172.21.15.70:0/1897854500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3901031e0 0x7fc3901035c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:35.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.582+0000 7fc3953ce700 1 -- 172.21.15.70:0/1897854500 >> 172.21.15.70:0/1897854500 conn(0x7fc3900fea70 msgr2=0x7fc390100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:35.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.582+0000 7fc3953ce700 1 -- 172.21.15.70:0/1897854500 shutdown_connections 2024-06-04T04:46:35.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.582+0000 7fc3953ce700 1 -- 172.21.15.70:0/1897854500 wait complete. 2024-06-04T04:46:35.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.583+0000 7fc3953ce700 1 Processor -- start 2024-06-04T04:46:35.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.583+0000 7fc3953ce700 1 -- start start 2024-06-04T04:46:35.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.583+0000 7fc3953ce700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3901031e0 0x7fc39019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:35.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.583+0000 7fc3953ce700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc39010d310 con 0x7fc3901031e0 2024-06-04T04:46:35.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.584+0000 7fc38effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3901031e0 0x7fc39019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:35.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.584+0000 7fc38effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3901031e0 0x7fc39019e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48472/0 (socket says 172.21.15.70:48472) 2024-06-04T04:46:35.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.584+0000 7fc38effd700 1 -- 172.21.15.70:0/946144892 learned_addr learned my addr 172.21.15.70:0/946144892 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:35.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.584+0000 7fc38effd700 1 -- 172.21.15.70:0/946144892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc39019eb80 con 0x7fc3901031e0 2024-06-04T04:46:35.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.584+0000 7fc38effd700 1 --2- 172.21.15.70:0/946144892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3901031e0 0x7fc39019e640 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc380005660 tx=0x7fc380005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:35.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.584+0000 7fc377fff700 1 -- 172.21.15.70:0/946144892 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc380026070 con 0x7fc3901031e0 2024-06-04T04:46:35.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.584+0000 7fc3953ce700 1 -- 172.21.15.70:0/946144892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc39019ee70 con 0x7fc3901031e0 2024-06-04T04:46:35.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.584+0000 7fc377fff700 1 -- 172.21.15.70:0/946144892 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc38000b6f0 con 0x7fc3901031e0 2024-06-04T04:46:35.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.584+0000 7fc377fff700 1 -- 172.21.15.70:0/946144892 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3800037b0 con 0x7fc3901031e0 2024-06-04T04:46:35.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.584+0000 7fc3953ce700 1 -- 172.21.15.70:0/946144892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3901988e0 con 0x7fc3901031e0 2024-06-04T04:46:35.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.585+0000 7fc377fff700 1 -- 172.21.15.70:0/946144892 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fc380029070 con 0x7fc3901031e0 2024-06-04T04:46:35.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.585+0000 7fc3953ce700 1 -- 172.21.15.70:0/946144892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3901046c0 con 0x7fc3901031e0 2024-06-04T04:46:35.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.585+0000 7fc377fff700 1 --2- 172.21.15.70:0/946144892 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc37803e510 0x7fc3780409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:35.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.586+0000 7fc377fff700 1 -- 172.21.15.70:0/946144892 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc38001e070 con 0x7fc3901031e0 2024-06-04T04:46:35.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.586+0000 7fc38e7fc700 1 --2- 172.21.15.70:0/946144892 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc37803e510 0x7fc3780409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:35.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.586+0000 7fc38e7fc700 1 --2- 172.21.15.70:0/946144892 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc37803e510 0x7fc3780409d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc384006fd0 tx=0x7fc384006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:35.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.591+0000 7fc377fff700 1 -- 172.21.15.70:0/946144892 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc380013070 con 0x7fc3901031e0 2024-06-04T04:46:35.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:35.798+0000 7fc3953ce700 1 -- 172.21.15.70:0/946144892 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi098", "target": ["mon-mgr", ""]}) -- 0x7fc390199cb0 con 0x7fc37803e510 2024-06-04T04:46:37.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:36 smithi070 ceph-mon[32631]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:46:37.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:36 smithi070 ceph-mon[32631]: Deploying cephadm binary to smithi098 2024-06-04T04:46:39.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:38 smithi070 ceph-mon[32631]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:46:39.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:39.897+0000 7fc377fff700 1 -- 172.21.15.70:0/946144892 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fc390199cb0 con 0x7fc37803e510 2024-06-04T04:46:39.900 INFO:teuthology.orchestra.run.smithi070.stdout:Added host 'smithi098' with addr '172.21.15.98' 2024-06-04T04:46:39.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:39.900+0000 7fc3953ce700 1 -- 172.21.15.70:0/946144892 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc37803e510 msgr2=0x7fc3780409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:39.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:39.900+0000 7fc3953ce700 1 --2- 172.21.15.70:0/946144892 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc37803e510 0x7fc3780409d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc384006fd0 tx=0x7fc384006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:39.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:39.900+0000 7fc3953ce700 1 -- 172.21.15.70:0/946144892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3901031e0 msgr2=0x7fc39019e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:39.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:39.900+0000 7fc3953ce700 1 --2- 172.21.15.70:0/946144892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3901031e0 0x7fc39019e640 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc380005660 tx=0x7fc380005740 comp rx=0 tx=0).stop 2024-06-04T04:46:39.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:39.900+0000 7fc3953ce700 1 -- 172.21.15.70:0/946144892 shutdown_connections 2024-06-04T04:46:39.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:39.900+0000 7fc3953ce700 1 --2- 172.21.15.70:0/946144892 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc37803e510 0x7fc3780409d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:39.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:39.900+0000 7fc3953ce700 1 --2- 172.21.15.70:0/946144892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3901031e0 0x7fc39019e640 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:39.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:39.900+0000 7fc3953ce700 1 -- 172.21.15.70:0/946144892 >> 172.21.15.70:0/946144892 conn(0x7fc3900fea70 msgr2=0x7fc390104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:39.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:39.900+0000 7fc3953ce700 1 -- 172.21.15.70:0/946144892 shutdown_connections 2024-06-04T04:46:39.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:39.900+0000 7fc3953ce700 1 -- 172.21.15.70:0/946144892 wait complete. 2024-06-04T04:46:40.571 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-06-04T04:46:40.808 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:46:41.187 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:40 smithi070 ceph-mon[32631]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:46:41.187 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:40 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:41.188 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:40 smithi070 ceph-mon[32631]: Added host smithi098 2024-06-04T04:46:41.188 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:40 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:46:41.188 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:40 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:42.190 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:41 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:42.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.196+0000 7f5d5abb2700 1 -- 172.21.15.70:0/1330766815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d541020b0 msgr2=0x7f5d54102490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:42.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.196+0000 7f5d5abb2700 1 --2- 172.21.15.70:0/1330766815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d541020b0 0x7f5d54102490 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5d48009910 tx=0x7f5d48009c20 comp rx=0 tx=0).stop 2024-06-04T04:46:42.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.196+0000 7f5d5abb2700 1 -- 172.21.15.70:0/1330766815 shutdown_connections 2024-06-04T04:46:42.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.196+0000 7f5d5abb2700 1 --2- 172.21.15.70:0/1330766815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d541020b0 0x7f5d54102490 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:42.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.196+0000 7f5d5abb2700 1 -- 172.21.15.70:0/1330766815 >> 172.21.15.70:0/1330766815 conn(0x7f5d540fd960 msgr2=0x7f5d540ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:42.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.196+0000 7f5d5abb2700 1 -- 172.21.15.70:0/1330766815 shutdown_connections 2024-06-04T04:46:42.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.196+0000 7f5d5abb2700 1 -- 172.21.15.70:0/1330766815 wait complete. 2024-06-04T04:46:42.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.196+0000 7f5d5abb2700 1 Processor -- start 2024-06-04T04:46:42.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.196+0000 7f5d5abb2700 1 -- start start 2024-06-04T04:46:42.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.197+0000 7f5d5abb2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d541020b0 0x7f5d54199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:42.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.197+0000 7f5d5abb2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5410c1e0 con 0x7f5d541020b0 2024-06-04T04:46:42.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.197+0000 7f5d5894e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d541020b0 0x7f5d54199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:42.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.197+0000 7f5d5894e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d541020b0 0x7f5d54199750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35920/0 (socket says 172.21.15.70:35920) 2024-06-04T04:46:42.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.197+0000 7f5d5894e700 1 -- 172.21.15.70:0/2876917070 learned_addr learned my addr 172.21.15.70:0/2876917070 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:42.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.197+0000 7f5d5894e700 1 -- 172.21.15.70:0/2876917070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d54199c90 con 0x7f5d541020b0 2024-06-04T04:46:42.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.197+0000 7f5d5894e700 1 --2- 172.21.15.70:0/2876917070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d541020b0 0x7f5d54199750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5d48005660 tx=0x7f5d48005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:42.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.197+0000 7f5d51ffb700 1 -- 172.21.15.70:0/2876917070 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d48026070 con 0x7f5d541020b0 2024-06-04T04:46:42.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.197+0000 7f5d51ffb700 1 -- 172.21.15.70:0/2876917070 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d4800b6f0 con 0x7f5d541020b0 2024-06-04T04:46:42.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.197+0000 7f5d5abb2700 1 -- 172.21.15.70:0/2876917070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d54199f80 con 0x7f5d541020b0 2024-06-04T04:46:42.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.197+0000 7f5d51ffb700 1 -- 172.21.15.70:0/2876917070 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d480037b0 con 0x7f5d541020b0 2024-06-04T04:46:42.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.197+0000 7f5d5abb2700 1 -- 172.21.15.70:0/2876917070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d5419a440 con 0x7f5d541020b0 2024-06-04T04:46:42.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.198+0000 7f5d51ffb700 1 -- 172.21.15.70:0/2876917070 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f5d48029070 con 0x7f5d541020b0 2024-06-04T04:46:42.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.198+0000 7f5d5abb2700 1 -- 172.21.15.70:0/2876917070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d54103590 con 0x7f5d541020b0 2024-06-04T04:46:42.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.199+0000 7f5d51ffb700 1 --2- 172.21.15.70:0/2876917070 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f5d4403e510 0x7f5d440409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:42.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.199+0000 7f5d51ffb700 1 -- 172.21.15.70:0/2876917070 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5d4801e070 con 0x7f5d541020b0 2024-06-04T04:46:42.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.199+0000 7f5d53fff700 1 --2- 172.21.15.70:0/2876917070 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f5d4403e510 0x7f5d440409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:42.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.199+0000 7f5d53fff700 1 --2- 172.21.15.70:0/2876917070 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f5d4403e510 0x7f5d440409d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5d40006fd0 tx=0x7f5d40006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:42.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.204+0000 7f5d51ffb700 1 -- 172.21.15.70:0/2876917070 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5d48013070 con 0x7f5d541020b0 2024-06-04T04:46:42.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.412+0000 7f5d5abb2700 1 -- 172.21.15.70:0/2876917070 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5d5406b9c0 con 0x7f5d4403e510 2024-06-04T04:46:42.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.413+0000 7f5d51ffb700 1 -- 172.21.15.70:0/2876917070 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f5d5406b9c0 con 0x7f5d4403e510 2024-06-04T04:46:42.418 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:46:42.418 INFO:teuthology.orchestra.run.smithi070.stdout:[{"addr": "172.21.15.70", "hostname": "smithi070", "labels": [], "status": ""}, {"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}] 2024-06-04T04:46:42.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.415+0000 7f5d5abb2700 1 -- 172.21.15.70:0/2876917070 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f5d4403e510 msgr2=0x7f5d440409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:42.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.415+0000 7f5d5abb2700 1 --2- 172.21.15.70:0/2876917070 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f5d4403e510 0x7f5d440409d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5d40006fd0 tx=0x7f5d40006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:42.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.415+0000 7f5d5abb2700 1 -- 172.21.15.70:0/2876917070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d541020b0 msgr2=0x7f5d54199750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:42.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.415+0000 7f5d5abb2700 1 --2- 172.21.15.70:0/2876917070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d541020b0 0x7f5d54199750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5d48005660 tx=0x7f5d48005740 comp rx=0 tx=0).stop 2024-06-04T04:46:42.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.416+0000 7f5d5abb2700 1 -- 172.21.15.70:0/2876917070 shutdown_connections 2024-06-04T04:46:42.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.416+0000 7f5d5abb2700 1 --2- 172.21.15.70:0/2876917070 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f5d4403e510 0x7f5d440409d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:42.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.416+0000 7f5d5abb2700 1 --2- 172.21.15.70:0/2876917070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d541020b0 0x7f5d54199750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:42.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.416+0000 7f5d5abb2700 1 -- 172.21.15.70:0/2876917070 >> 172.21.15.70:0/2876917070 conn(0x7f5d540fd960 msgr2=0x7f5d54103a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:42.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.416+0000 7f5d5abb2700 1 -- 172.21.15.70:0/2876917070 shutdown_connections 2024-06-04T04:46:42.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:42.416+0000 7f5d5abb2700 1 -- 172.21.15.70:0/2876917070 wait complete. 2024-06-04T04:46:43.014 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi112 2024-06-04T04:46:43.014 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-06-04T04:46:43.014 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/etc/ceph/ceph.conf 2024-06-04T04:46:43.034 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-06-04T04:46:43.034 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:46:43.092 INFO:tasks.cephadm:Adding host smithi112 to orchestrator... 2024-06-04T04:46:43.092 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch host add smithi112 2024-06-04T04:46:43.324 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:46:43.721 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:43 smithi070 ceph-mon[32631]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:46:43.721 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:43 smithi070 ceph-mon[32631]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:46:43.721 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:44.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.853+0000 7f563efd4700 1 -- 172.21.15.70:0/3310005343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56381031b0 msgr2=0x7f5638103590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:44.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.853+0000 7f563efd4700 1 --2- 172.21.15.70:0/3310005343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56381031b0 0x7f5638103590 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f562c009910 tx=0x7f562c009c20 comp rx=0 tx=0).stop 2024-06-04T04:46:44.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.853+0000 7f563efd4700 1 -- 172.21.15.70:0/3310005343 shutdown_connections 2024-06-04T04:46:44.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.853+0000 7f563efd4700 1 --2- 172.21.15.70:0/3310005343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56381031b0 0x7f5638103590 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:44.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.853+0000 7f563efd4700 1 -- 172.21.15.70:0/3310005343 >> 172.21.15.70:0/3310005343 conn(0x7f56380fea80 msgr2=0x7f5638100ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:44.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.853+0000 7f563efd4700 1 -- 172.21.15.70:0/3310005343 shutdown_connections 2024-06-04T04:46:44.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.853+0000 7f563efd4700 1 -- 172.21.15.70:0/3310005343 wait complete. 2024-06-04T04:46:44.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.854+0000 7f563efd4700 1 Processor -- start 2024-06-04T04:46:44.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.854+0000 7f563efd4700 1 -- start start 2024-06-04T04:46:44.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.854+0000 7f563efd4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56381031b0 0x7f563819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.854+0000 7f563efd4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f563810d2e0 con 0x7f56381031b0 2024-06-04T04:46:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.854+0000 7f563cd70700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56381031b0 0x7f563819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.854+0000 7f563cd70700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56381031b0 0x7f563819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35936/0 (socket says 172.21.15.70:35936) 2024-06-04T04:46:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.854+0000 7f563cd70700 1 -- 172.21.15.70:0/329823051 learned_addr learned my addr 172.21.15.70:0/329823051 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.855+0000 7f563cd70700 1 -- 172.21.15.70:0/329823051 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f563819eb90 con 0x7f56381031b0 2024-06-04T04:46:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.855+0000 7f563cd70700 1 --2- 172.21.15.70:0/329823051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56381031b0 0x7f563819e650 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f562c005660 tx=0x7f562c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.855+0000 7f5635ffb700 1 -- 172.21.15.70:0/329823051 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f562c026070 con 0x7f56381031b0 2024-06-04T04:46:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.855+0000 7f5635ffb700 1 -- 172.21.15.70:0/329823051 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f562c00b6f0 con 0x7f56381031b0 2024-06-04T04:46:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.855+0000 7f563efd4700 1 -- 172.21.15.70:0/329823051 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f563819ee80 con 0x7f56381031b0 2024-06-04T04:46:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.855+0000 7f5635ffb700 1 -- 172.21.15.70:0/329823051 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f562c0037b0 con 0x7f56381031b0 2024-06-04T04:46:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.855+0000 7f563efd4700 1 -- 172.21.15.70:0/329823051 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56381987d0 con 0x7f56381031b0 2024-06-04T04:46:44.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.856+0000 7f563efd4700 1 -- 172.21.15.70:0/329823051 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5638104690 con 0x7f56381031b0 2024-06-04T04:46:44.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.856+0000 7f5635ffb700 1 -- 172.21.15.70:0/329823051 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f562c029070 con 0x7f56381031b0 2024-06-04T04:46:44.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.856+0000 7f5635ffb700 1 --2- 172.21.15.70:0/329823051 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f562803e570 0x7f5628040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:44.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.856+0000 7f5635ffb700 1 -- 172.21.15.70:0/329823051 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f562c01e070 con 0x7f56381031b0 2024-06-04T04:46:44.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.856+0000 7f5637fff700 1 --2- 172.21.15.70:0/329823051 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f562803e570 0x7f5628040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:44.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.857+0000 7f5637fff700 1 --2- 172.21.15.70:0/329823051 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f562803e570 0x7f5628040a30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5624006fd0 tx=0x7f5624006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:44.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:44.861+0000 7f5635ffb700 1 -- 172.21.15.70:0/329823051 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f562c013070 con 0x7f56381031b0 2024-06-04T04:46:45.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:45.069+0000 7f563efd4700 1 -- 172.21.15.70:0/329823051 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi112", "target": ["mon-mgr", ""]}) -- 0x7f5638199d00 con 0x7f562803e570 2024-06-04T04:46:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:45 smithi070 ceph-mon[32631]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:46:46.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:46 smithi070 ceph-mon[32631]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi112", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:46:46.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:46.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:46.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:46.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:46.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:46:46.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:46:46.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:46:47.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:47 smithi070 ceph-mon[32631]: Updating smithi098:/etc/ceph/ceph.conf 2024-06-04T04:46:47.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:47 smithi070 ceph-mon[32631]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:46:47.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:47 smithi070 ceph-mon[32631]: Deploying cephadm binary to smithi112 2024-06-04T04:46:47.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:47 smithi070 ceph-mon[32631]: Updating smithi098:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:46:47.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:47 smithi070 ceph-mon[32631]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:46:49.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:49.267+0000 7f5635ffb700 1 -- 172.21.15.70:0/329823051 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f5638199d00 con 0x7f562803e570 2024-06-04T04:46:49.271 INFO:teuthology.orchestra.run.smithi070.stdout:Added host 'smithi112' with addr '172.21.15.112' 2024-06-04T04:46:49.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:49.270+0000 7f563efd4700 1 -- 172.21.15.70:0/329823051 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f562803e570 msgr2=0x7f5628040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:49.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:49.270+0000 7f563efd4700 1 --2- 172.21.15.70:0/329823051 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f562803e570 0x7f5628040a30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5624006fd0 tx=0x7f5624006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:49.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:49.270+0000 7f563efd4700 1 -- 172.21.15.70:0/329823051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56381031b0 msgr2=0x7f563819e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:49.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:49.270+0000 7f563efd4700 1 --2- 172.21.15.70:0/329823051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56381031b0 0x7f563819e650 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f562c005660 tx=0x7f562c005740 comp rx=0 tx=0).stop 2024-06-04T04:46:49.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:49.270+0000 7f563efd4700 1 -- 172.21.15.70:0/329823051 shutdown_connections 2024-06-04T04:46:49.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:49.270+0000 7f563efd4700 1 --2- 172.21.15.70:0/329823051 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f562803e570 0x7f5628040a30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:49.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:49.270+0000 7f563efd4700 1 --2- 172.21.15.70:0/329823051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56381031b0 0x7f563819e650 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:49.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:49.270+0000 7f563efd4700 1 -- 172.21.15.70:0/329823051 >> 172.21.15.70:0/329823051 conn(0x7f56380fea80 msgr2=0x7f563810ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:49.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:49.270+0000 7f563efd4700 1 -- 172.21.15.70:0/329823051 shutdown_connections 2024-06-04T04:46:49.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:49.270+0000 7f563efd4700 1 -- 172.21.15.70:0/329823051 wait complete. 2024-06-04T04:46:49.571 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:49 smithi070 ceph-mon[32631]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:46:49.571 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:49 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:49.571 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:49 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:49.571 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:49 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:49.571 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:49 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:49.571 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:49 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:46:49.917 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-06-04T04:46:50.149 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:46:50.574 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:50 smithi070 ceph-mon[32631]: Updating smithi098:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-04T04:46:50.575 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:50 smithi070 ceph-mon[32631]: Added host smithi112 2024-06-04T04:46:50.575 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:50 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:50.575 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:50 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:51.515 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:51 smithi070 ceph-mon[32631]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:46:51.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.613+0000 7f5f382b9700 1 -- 172.21.15.70:0/4136375932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f301020f0 msgr2=0x7f5f301024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:51.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.613+0000 7f5f382b9700 1 --2- 172.21.15.70:0/4136375932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f301020f0 0x7f5f301024d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009910 tx=0x7f5f20009c20 comp rx=0 tx=0).stop 2024-06-04T04:46:51.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.613+0000 7f5f382b9700 1 -- 172.21.15.70:0/4136375932 shutdown_connections 2024-06-04T04:46:51.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.613+0000 7f5f382b9700 1 --2- 172.21.15.70:0/4136375932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f301020f0 0x7f5f301024d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:51.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.613+0000 7f5f382b9700 1 -- 172.21.15.70:0/4136375932 >> 172.21.15.70:0/4136375932 conn(0x7f5f300779d0 msgr2=0x7f5f30077de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:51.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.613+0000 7f5f382b9700 1 -- 172.21.15.70:0/4136375932 shutdown_connections 2024-06-04T04:46:51.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.613+0000 7f5f382b9700 1 -- 172.21.15.70:0/4136375932 wait complete. 2024-06-04T04:46:51.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.614+0000 7f5f382b9700 1 Processor -- start 2024-06-04T04:46:51.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.614+0000 7f5f382b9700 1 -- start start 2024-06-04T04:46:51.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.614+0000 7f5f382b9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f301020f0 0x7f5f3019d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:51.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.614+0000 7f5f382b9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3010c200 con 0x7f5f301020f0 2024-06-04T04:46:51.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.614+0000 7f5f36055700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f301020f0 0x7f5f3019d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:51.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.614+0000 7f5f36055700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f301020f0 0x7f5f3019d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60664/0 (socket says 172.21.15.70:60664) 2024-06-04T04:46:51.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.614+0000 7f5f36055700 1 -- 172.21.15.70:0/3761053440 learned_addr learned my addr 172.21.15.70:0/3761053440 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:51.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.614+0000 7f5f36055700 1 -- 172.21.15.70:0/3761053440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f3019d960 con 0x7f5f301020f0 2024-06-04T04:46:51.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.614+0000 7f5f36055700 1 --2- 172.21.15.70:0/3761053440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f301020f0 0x7f5f3019d420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5f20005660 tx=0x7f5f20005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:51.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.615+0000 7f5f277fe700 1 -- 172.21.15.70:0/3761053440 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5f20026070 con 0x7f5f301020f0 2024-06-04T04:46:51.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.615+0000 7f5f277fe700 1 -- 172.21.15.70:0/3761053440 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5f2000b6f0 con 0x7f5f301020f0 2024-06-04T04:46:51.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.615+0000 7f5f382b9700 1 -- 172.21.15.70:0/3761053440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f3019dc50 con 0x7f5f301020f0 2024-06-04T04:46:51.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.615+0000 7f5f277fe700 1 -- 172.21.15.70:0/3761053440 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5f200037b0 con 0x7f5f301020f0 2024-06-04T04:46:51.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.615+0000 7f5f382b9700 1 -- 172.21.15.70:0/3761053440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f301975e0 con 0x7f5f301020f0 2024-06-04T04:46:51.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.616+0000 7f5f277fe700 1 -- 172.21.15.70:0/3761053440 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f5f20029070 con 0x7f5f301020f0 2024-06-04T04:46:51.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.616+0000 7f5f277fe700 1 --2- 172.21.15.70:0/3761053440 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f5f1c03e510 0x7f5f1c0409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:51.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.616+0000 7f5f277fe700 1 -- 172.21.15.70:0/3761053440 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5f2001e070 con 0x7f5f301020f0 2024-06-04T04:46:51.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.616+0000 7f5f382b9700 1 -- 172.21.15.70:0/3761053440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f301035b0 con 0x7f5f301020f0 2024-06-04T04:46:51.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.616+0000 7f5f35854700 1 --2- 172.21.15.70:0/3761053440 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f5f1c03e510 0x7f5f1c0409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:51.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.616+0000 7f5f35854700 1 --2- 172.21.15.70:0/3761053440 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f5f1c03e510 0x7f5f1c0409d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c006fd0 tx=0x7f5f2c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:51.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.622+0000 7f5f277fe700 1 -- 172.21.15.70:0/3761053440 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5f20030070 con 0x7f5f301020f0 2024-06-04T04:46:51.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.829+0000 7f5f382b9700 1 -- 172.21.15.70:0/3761053440 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5f30064ba0 con 0x7f5f1c03e510 2024-06-04T04:46:51.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.830+0000 7f5f277fe700 1 -- 172.21.15.70:0/3761053440 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f5f30064ba0 con 0x7f5f1c03e510 2024-06-04T04:46:51.833 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:46:51.834 INFO:teuthology.orchestra.run.smithi070.stdout:[{"addr": "172.21.15.70", "hostname": "smithi070", "labels": [], "status": ""}, {"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.112", "hostname": "smithi112", "labels": [], "status": ""}] 2024-06-04T04:46:51.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.833+0000 7f5f382b9700 1 -- 172.21.15.70:0/3761053440 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f5f1c03e510 msgr2=0x7f5f1c0409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:51.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.833+0000 7f5f382b9700 1 --2- 172.21.15.70:0/3761053440 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f5f1c03e510 0x7f5f1c0409d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c006fd0 tx=0x7f5f2c006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:51.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.833+0000 7f5f382b9700 1 -- 172.21.15.70:0/3761053440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f301020f0 msgr2=0x7f5f3019d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:51.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.833+0000 7f5f382b9700 1 --2- 172.21.15.70:0/3761053440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f301020f0 0x7f5f3019d420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5f20005660 tx=0x7f5f20005740 comp rx=0 tx=0).stop 2024-06-04T04:46:51.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.833+0000 7f5f382b9700 1 -- 172.21.15.70:0/3761053440 shutdown_connections 2024-06-04T04:46:51.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.833+0000 7f5f382b9700 1 --2- 172.21.15.70:0/3761053440 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f5f1c03e510 0x7f5f1c0409d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:51.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.833+0000 7f5f382b9700 1 --2- 172.21.15.70:0/3761053440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f301020f0 0x7f5f3019d420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:51.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.833+0000 7f5f382b9700 1 -- 172.21.15.70:0/3761053440 >> 172.21.15.70:0/3761053440 conn(0x7f5f300779d0 msgr2=0x7f5f30103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:51.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.833+0000 7f5f382b9700 1 -- 172.21.15.70:0/3761053440 shutdown_connections 2024-06-04T04:46:51.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:51.833+0000 7f5f382b9700 1 -- 172.21.15.70:0/3761053440 wait complete. 2024-06-04T04:46:52.393 INFO:tasks.cephadm:Setting crush tunables to default 2024-06-04T04:46:52.394 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd crush tunables default 2024-06-04T04:46:52.626 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:46:53.054 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:52 smithi070 ceph-mon[32631]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:46:53.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:52 smithi070 ceph-mon[32631]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:46:53.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:52 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:54.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.169+0000 7f26ccd25700 1 -- 172.21.15.70:0/2525731899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26c81040f0 msgr2=0x7f26c81044d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:54.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.169+0000 7f26ccd25700 1 --2- 172.21.15.70:0/2525731899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26c81040f0 0x7f26c81044d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f26b8009910 tx=0x7f26b8009c20 comp rx=0 tx=0).stop 2024-06-04T04:46:54.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.169+0000 7f26ccd25700 1 -- 172.21.15.70:0/2525731899 shutdown_connections 2024-06-04T04:46:54.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.169+0000 7f26ccd25700 1 --2- 172.21.15.70:0/2525731899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26c81040f0 0x7f26c81044d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:54.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.169+0000 7f26ccd25700 1 -- 172.21.15.70:0/2525731899 >> 172.21.15.70:0/2525731899 conn(0x7f26c80fd940 msgr2=0x7f26c80ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:54.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.169+0000 7f26ccd25700 1 -- 172.21.15.70:0/2525731899 shutdown_connections 2024-06-04T04:46:54.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.169+0000 7f26ccd25700 1 -- 172.21.15.70:0/2525731899 wait complete. 2024-06-04T04:46:54.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.170+0000 7f26ccd25700 1 Processor -- start 2024-06-04T04:46:54.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.170+0000 7f26ccd25700 1 -- start start 2024-06-04T04:46:54.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.170+0000 7f26ccd25700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26c81040f0 0x7f26c819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:54.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.170+0000 7f26ccd25700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f26c810d340 con 0x7f26c81040f0 2024-06-04T04:46:54.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.170+0000 7f26c659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26c81040f0 0x7f26c819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:54.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.170+0000 7f26c659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26c81040f0 0x7f26c819e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60672/0 (socket says 172.21.15.70:60672) 2024-06-04T04:46:54.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.170+0000 7f26c659c700 1 -- 172.21.15.70:0/3744553518 learned_addr learned my addr 172.21.15.70:0/3744553518 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:46:54.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.171+0000 7f26c659c700 1 -- 172.21.15.70:0/3744553518 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26c819eba0 con 0x7f26c81040f0 2024-06-04T04:46:54.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.171+0000 7f26c659c700 1 --2- 172.21.15.70:0/3744553518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26c81040f0 0x7f26c819e660 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f26b8005060 tx=0x7f26b8005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:54.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.171+0000 7f26bffff700 1 -- 172.21.15.70:0/3744553518 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26b8026070 con 0x7f26c81040f0 2024-06-04T04:46:54.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.171+0000 7f26bffff700 1 -- 172.21.15.70:0/3744553518 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f26b800b750 con 0x7f26c81040f0 2024-06-04T04:46:54.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.171+0000 7f26ccd25700 1 -- 172.21.15.70:0/3744553518 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26c819ee30 con 0x7f26c81040f0 2024-06-04T04:46:54.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.171+0000 7f26bffff700 1 -- 172.21.15.70:0/3744553518 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26b8003850 con 0x7f26c81040f0 2024-06-04T04:46:54.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.171+0000 7f26ccd25700 1 -- 172.21.15.70:0/3744553518 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26c81987e0 con 0x7f26c81040f0 2024-06-04T04:46:54.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.172+0000 7f26bffff700 1 -- 172.21.15.70:0/3744553518 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f26b8029070 con 0x7f26c81040f0 2024-06-04T04:46:54.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.172+0000 7f26ccd25700 1 -- 172.21.15.70:0/3744553518 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26c806bad0 con 0x7f26c81040f0 2024-06-04T04:46:54.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.172+0000 7f26bffff700 1 --2- 172.21.15.70:0/3744553518 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f26b403e510 0x7f26b40409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:54.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.172+0000 7f26bffff700 1 -- 172.21.15.70:0/3744553518 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f26b801e070 con 0x7f26c81040f0 2024-06-04T04:46:54.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.172+0000 7f26bdd9b700 1 --2- 172.21.15.70:0/3744553518 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f26b403e510 0x7f26b40409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:54.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.172+0000 7f26bdd9b700 1 --2- 172.21.15.70:0/3744553518 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f26b403e510 0x7f26b40409d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f26b0006fd0 tx=0x7f26b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:54.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.177+0000 7f26bffff700 1 -- 172.21.15.70:0/3744553518 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f26b8013070 con 0x7f26c81040f0 2024-06-04T04:46:54.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.385+0000 7f26ccd25700 1 -- 172.21.15.70:0/3744553518 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f26c8064ba0 con 0x7f26c81040f0 2024-06-04T04:46:54.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.852+0000 7f26bffff700 1 -- 172.21.15.70:0/3744553518 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f26b802ada0 con 0x7f26c81040f0 2024-06-04T04:46:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.855+0000 7f26ccd25700 1 -- 172.21.15.70:0/3744553518 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f26b403e510 msgr2=0x7f26b40409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.855+0000 7f26ccd25700 1 --2- 172.21.15.70:0/3744553518 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f26b403e510 0x7f26b40409d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f26b0006fd0 tx=0x7f26b0006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.855+0000 7f26ccd25700 1 -- 172.21.15.70:0/3744553518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26c81040f0 msgr2=0x7f26c819e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.855+0000 7f26ccd25700 1 --2- 172.21.15.70:0/3744553518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26c81040f0 0x7f26c819e660 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f26b8005060 tx=0x7f26b8005710 comp rx=0 tx=0).stop 2024-06-04T04:46:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.855+0000 7f26ccd25700 1 -- 172.21.15.70:0/3744553518 shutdown_connections 2024-06-04T04:46:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.855+0000 7f26ccd25700 1 --2- 172.21.15.70:0/3744553518 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f26b403e510 0x7f26b40409d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.855+0000 7f26ccd25700 1 --2- 172.21.15.70:0/3744553518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26c81040f0 0x7f26c819e660 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.855+0000 7f26ccd25700 1 -- 172.21.15.70:0/3744553518 >> 172.21.15.70:0/3744553518 conn(0x7f26c80fd940 msgr2=0x7f26c8106eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.855+0000 7f26ccd25700 1 -- 172.21.15.70:0/3744553518 shutdown_connections 2024-06-04T04:46:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:46:54.855+0000 7f26ccd25700 1 -- 172.21.15.70:0/3744553518 wait complete. 2024-06-04T04:46:54.858 INFO:teuthology.orchestra.run.smithi070.stderr:adjusted tunables profile to default 2024-06-04T04:46:55.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:54 smithi070 ceph-mon[32631]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:46:55.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:54 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3744553518' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-04T04:46:55.582 INFO:tasks.cephadm:Adding mon.a on smithi070 2024-06-04T04:46:55.582 INFO:tasks.cephadm:Adding mon.b on smithi098 2024-06-04T04:46:55.582 INFO:tasks.cephadm:Adding mon.c on smithi112 2024-06-04T04:46:55.582 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch apply mon '3;smithi070:172.21.15.70=a;smithi098:172.21.15.98=b;smithi112:172.21.15.112=c' 2024-06-04T04:46:55.816 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:46:56.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:55 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3744553518' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-04T04:46:56.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:55 smithi070 ceph-mon[32631]: osdmap e4: 0 total, 0 up, 0 in 2024-06-04T04:46:56.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:55 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:56.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:55 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:56.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:55 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:56.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:55 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:56.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:55 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:46:56.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:55 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:46:56.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:55 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:46:56.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:55 smithi070 ceph-mon[32631]: Updating smithi112:/etc/ceph/ceph.conf 2024-06-04T04:46:56.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:55 smithi070 ceph-mon[32631]: Updating smithi112:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:46:56.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:55 smithi070 ceph-mon[32631]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:46:57.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.350+0000 7fac78adf700 1 -- 172.21.15.112:0/1118681975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac741031b0 msgr2=0x7fac74103590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:57.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.350+0000 7fac78adf700 1 --2- 172.21.15.112:0/1118681975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac741031b0 0x7fac74103590 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fac68009910 tx=0x7fac68009c20 comp rx=0 tx=0).stop 2024-06-04T04:46:57.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.350+0000 7fac78adf700 1 -- 172.21.15.112:0/1118681975 shutdown_connections 2024-06-04T04:46:57.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.350+0000 7fac78adf700 1 --2- 172.21.15.112:0/1118681975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac741031b0 0x7fac74103590 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:57.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.350+0000 7fac78adf700 1 -- 172.21.15.112:0/1118681975 >> 172.21.15.112:0/1118681975 conn(0x7fac740fea80 msgr2=0x7fac74100ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:57.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.350+0000 7fac78adf700 1 -- 172.21.15.112:0/1118681975 shutdown_connections 2024-06-04T04:46:57.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.350+0000 7fac78adf700 1 -- 172.21.15.112:0/1118681975 wait complete. 2024-06-04T04:46:57.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.351+0000 7fac78adf700 1 Processor -- start 2024-06-04T04:46:57.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.351+0000 7fac78adf700 1 -- start start 2024-06-04T04:46:57.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.351+0000 7fac78adf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac741031b0 0x7fac7419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.351+0000 7fac78adf700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fac7410d2e0 con 0x7fac741031b0 2024-06-04T04:46:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.351+0000 7fac7259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac741031b0 0x7fac7419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.351+0000 7fac7259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac741031b0 0x7fac7419e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.112:49424/0 (socket says 172.21.15.112:49424) 2024-06-04T04:46:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.351+0000 7fac7259c700 1 -- 172.21.15.112:0/2905981766 learned_addr learned my addr 172.21.15.112:0/2905981766 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-06-04T04:46:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.352+0000 7fac7259c700 1 -- 172.21.15.112:0/2905981766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac7419eb50 con 0x7fac741031b0 2024-06-04T04:46:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.352+0000 7fac7259c700 1 --2- 172.21.15.112:0/2905981766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac741031b0 0x7fac7419e610 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fac68005660 tx=0x7fac68005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.352+0000 7fac637fe700 1 -- 172.21.15.112:0/2905981766 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac68026070 con 0x7fac741031b0 2024-06-04T04:46:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.352+0000 7fac78adf700 1 -- 172.21.15.112:0/2905981766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac7419ee40 con 0x7fac741031b0 2024-06-04T04:46:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.352+0000 7fac637fe700 1 -- 172.21.15.112:0/2905981766 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fac6800b6f0 con 0x7fac741031b0 2024-06-04T04:46:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.352+0000 7fac637fe700 1 -- 172.21.15.112:0/2905981766 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac680037b0 con 0x7fac741031b0 2024-06-04T04:46:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.352+0000 7fac78adf700 1 -- 172.21.15.112:0/2905981766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac741988b0 con 0x7fac741031b0 2024-06-04T04:46:57.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.353+0000 7fac78adf700 1 -- 172.21.15.112:0/2905981766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac74104690 con 0x7fac741031b0 2024-06-04T04:46:57.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.353+0000 7fac637fe700 1 -- 172.21.15.112:0/2905981766 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fac68029070 con 0x7fac741031b0 2024-06-04T04:46:57.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.353+0000 7fac637fe700 1 --2- 172.21.15.112:0/2905981766 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fac5c03e510 0x7fac5c0409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:46:57.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.353+0000 7fac637fe700 1 -- 172.21.15.112:0/2905981766 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fac6801e070 con 0x7fac741031b0 2024-06-04T04:46:57.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.354+0000 7fac71d9b700 1 --2- 172.21.15.112:0/2905981766 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fac5c03e510 0x7fac5c0409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:46:57.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.354+0000 7fac71d9b700 1 --2- 172.21.15.112:0/2905981766 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fac5c03e510 0x7fac5c0409d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fac64006fd0 tx=0x7fac64006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:46:57.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.359+0000 7fac637fe700 1 -- 172.21.15.112:0/2905981766 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fac68013070 con 0x7fac741031b0 2024-06-04T04:46:57.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:56 smithi070 ceph-mon[32631]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:46:57.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:56 smithi070 ceph-mon[32631]: Updating smithi112:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-04T04:46:57.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:57.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:57.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:57.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.574+0000 7fac78adf700 1 -- 172.21.15.112:0/2905981766 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi070:172.21.15.70=a;smithi098:172.21.15.98=b;smithi112:172.21.15.112=c", "target": ["mon-mgr", ""]}) -- 0x7fac740008d0 con 0x7fac5c03e510 2024-06-04T04:46:57.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.580+0000 7fac637fe700 1 -- 172.21.15.112:0/2905981766 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fac740008d0 con 0x7fac5c03e510 2024-06-04T04:46:57.583 INFO:teuthology.orchestra.run.smithi112.stdout:Scheduled mon update... 2024-06-04T04:46:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.582+0000 7fac78adf700 1 -- 172.21.15.112:0/2905981766 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fac5c03e510 msgr2=0x7fac5c0409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.582+0000 7fac78adf700 1 --2- 172.21.15.112:0/2905981766 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fac5c03e510 0x7fac5c0409d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fac64006fd0 tx=0x7fac64006e40 comp rx=0 tx=0).stop 2024-06-04T04:46:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.582+0000 7fac78adf700 1 -- 172.21.15.112:0/2905981766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac741031b0 msgr2=0x7fac7419e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:46:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.582+0000 7fac78adf700 1 --2- 172.21.15.112:0/2905981766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac741031b0 0x7fac7419e610 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fac68005660 tx=0x7fac68005740 comp rx=0 tx=0).stop 2024-06-04T04:46:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.583+0000 7fac78adf700 1 -- 172.21.15.112:0/2905981766 shutdown_connections 2024-06-04T04:46:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.583+0000 7fac78adf700 1 --2- 172.21.15.112:0/2905981766 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fac5c03e510 0x7fac5c0409d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.583+0000 7fac78adf700 1 --2- 172.21.15.112:0/2905981766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac741031b0 0x7fac7419e610 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:46:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.583+0000 7fac78adf700 1 -- 172.21.15.112:0/2905981766 >> 172.21.15.112:0/2905981766 conn(0x7fac740fea80 msgr2=0x7fac74104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:46:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.583+0000 7fac78adf700 1 -- 172.21.15.112:0/2905981766 shutdown_connections 2024-06-04T04:46:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:46:57.583+0000 7fac78adf700 1 -- 172.21.15.112:0/2905981766 wait complete. 2024-06-04T04:46:58.187 DEBUG:teuthology.orchestra.run.smithi098:mon.b> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mon.b.service 2024-06-04T04:46:58.190 DEBUG:teuthology.orchestra.run.smithi112:mon.c> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mon.c.service 2024-06-04T04:46:58.192 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-04T04:46:58.192 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-04T04:46:58.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:58 smithi070 ceph-mon[32631]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi070:172.21.15.70=a;smithi098:172.21.15.98=b;smithi112:172.21.15.112=c", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:46:58.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:58 smithi070 ceph-mon[32631]: Saving service mon spec with placement smithi070:172.21.15.70=a;smithi098:172.21.15.98=b;smithi112:172.21.15.112=c;count:3 2024-06-04T04:46:58.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:58 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:58.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:58 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:46:58.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:58 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:46:58.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:58 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:46:58.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:58 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:46:58.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:58 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-04T04:46:58.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:58 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:46:59.149 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:46:59.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:59 smithi070 ceph-mon[32631]: Deploying daemon mon.c on smithi112 2024-06-04T04:46:59.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:46:59 smithi070 ceph-mon[32631]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:00.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.739+0000 7f48d438f700 1 -- 172.21.15.112:0/1232593713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48cc1020f0 msgr2=0x7f48cc1024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:00.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.739+0000 7f48d438f700 1 --2- 172.21.15.112:0/1232593713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48cc1020f0 0x7f48cc1024d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f48c400b0a0 tx=0x7f48c400b3b0 comp rx=0 tx=0).stop 2024-06-04T04:47:00.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.739+0000 7f48d438f700 1 -- 172.21.15.112:0/1232593713 shutdown_connections 2024-06-04T04:47:00.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.739+0000 7f48d438f700 1 --2- 172.21.15.112:0/1232593713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48cc1020f0 0x7f48cc1024d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:00.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.739+0000 7f48d438f700 1 -- 172.21.15.112:0/1232593713 >> 172.21.15.112:0/1232593713 conn(0x7f48cc0779d0 msgr2=0x7f48cc077de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:47:00.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.739+0000 7f48d438f700 1 -- 172.21.15.112:0/1232593713 shutdown_connections 2024-06-04T04:47:00.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.739+0000 7f48d438f700 1 -- 172.21.15.112:0/1232593713 wait complete. 2024-06-04T04:47:00.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.740+0000 7f48d438f700 1 Processor -- start 2024-06-04T04:47:00.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.740+0000 7f48d438f700 1 -- start start 2024-06-04T04:47:00.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.740+0000 7f48d438f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48cc1020f0 0x7f48cc19d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:00.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.740+0000 7f48d438f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc10c200 con 0x7f48cc1020f0 2024-06-04T04:47:00.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.741+0000 7f48d212b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48cc1020f0 0x7f48cc19d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:00.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.741+0000 7f48d212b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48cc1020f0 0x7f48cc19d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.112:49454/0 (socket says 172.21.15.112:49454) 2024-06-04T04:47:00.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.741+0000 7f48d212b700 1 -- 172.21.15.112:0/719642409 learned_addr learned my addr 172.21.15.112:0/719642409 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-06-04T04:47:00.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.741+0000 7f48d212b700 1 -- 172.21.15.112:0/719642409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48cc19dae0 con 0x7f48cc1020f0 2024-06-04T04:47:00.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.741+0000 7f48d212b700 1 --2- 172.21.15.112:0/719642409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48cc1020f0 0x7f48cc19d5a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f48c400b830 tx=0x7f48c400bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:47:00.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.741+0000 7f48cb7fe700 1 -- 172.21.15.112:0/719642409 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48c4003a00 con 0x7f48cc1020f0 2024-06-04T04:47:00.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.741+0000 7f48cb7fe700 1 -- 172.21.15.112:0/719642409 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f48c4003ba0 con 0x7f48cc1020f0 2024-06-04T04:47:00.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.741+0000 7f48d438f700 1 -- 172.21.15.112:0/719642409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48cc19dd70 con 0x7f48cc1020f0 2024-06-04T04:47:00.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.741+0000 7f48cb7fe700 1 -- 172.21.15.112:0/719642409 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48c4004960 con 0x7f48cc1020f0 2024-06-04T04:47:00.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.741+0000 7f48d438f700 1 -- 172.21.15.112:0/719642409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48cc197700 con 0x7f48cc1020f0 2024-06-04T04:47:00.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.742+0000 7f48cb7fe700 1 -- 172.21.15.112:0/719642409 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f48c4024510 con 0x7f48cc1020f0 2024-06-04T04:47:00.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.742+0000 7f48d438f700 1 -- 172.21.15.112:0/719642409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48cc1035b0 con 0x7f48cc1020f0 2024-06-04T04:47:00.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.743+0000 7f48cb7fe700 1 --2- 172.21.15.112:0/719642409 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f48b403e510 0x7f48b40409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:00.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.743+0000 7f48cb7fe700 1 -- 172.21.15.112:0/719642409 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f48c400f050 con 0x7f48cc1020f0 2024-06-04T04:47:00.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.743+0000 7f48d192a700 1 --2- 172.21.15.112:0/719642409 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f48b403e510 0x7f48b40409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:00.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.743+0000 7f48d192a700 1 --2- 172.21.15.112:0/719642409 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f48b403e510 0x7f48b40409d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f48c0006fd0 tx=0x7f48c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:47:00.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:00.748+0000 7f48cb7fe700 1 -- 172.21.15.112:0/719642409 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f48c4023ca0 con 0x7f48cc1020f0 2024-06-04T04:47:01.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:01.049+0000 7f48d438f700 1 -- 172.21.15.112:0/719642409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f48cc064ba0 con 0x7f48cc1020f0 2024-06-04T04:47:01.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:01.050+0000 7f48cb7fe700 1 -- 172.21.15.112:0/719642409 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f48c401f230 con 0x7f48cc1020f0 2024-06-04T04:47:01.052 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:47:01.053 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","modified":"2024-06-04T04:45:09.662402Z","created":"2024-06-04T04:45:09.662402Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-04T04:47:01.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:01.053+0000 7f48d438f700 1 -- 172.21.15.112:0/719642409 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f48b403e510 msgr2=0x7f48b40409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:01.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:01.053+0000 7f48d438f700 1 --2- 172.21.15.112:0/719642409 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f48b403e510 0x7f48b40409d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f48c0006fd0 tx=0x7f48c0006e40 comp rx=0 tx=0).stop 2024-06-04T04:47:01.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:01.053+0000 7f48d438f700 1 -- 172.21.15.112:0/719642409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48cc1020f0 msgr2=0x7f48cc19d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:01.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:01.053+0000 7f48d438f700 1 --2- 172.21.15.112:0/719642409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48cc1020f0 0x7f48cc19d5a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f48c400b830 tx=0x7f48c400bdb0 comp rx=0 tx=0).stop 2024-06-04T04:47:01.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:01.053+0000 7f48d438f700 1 -- 172.21.15.112:0/719642409 shutdown_connections 2024-06-04T04:47:01.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:01.053+0000 7f48d438f700 1 --2- 172.21.15.112:0/719642409 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f48b403e510 0x7f48b40409d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:01.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:01.053+0000 7f48d438f700 1 --2- 172.21.15.112:0/719642409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48cc1020f0 0x7f48cc19d5a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:01.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:01.053+0000 7f48d438f700 1 -- 172.21.15.112:0/719642409 >> 172.21.15.112:0/719642409 conn(0x7f48cc0779d0 msgr2=0x7f48cc103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:47:01.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:01.053+0000 7f48d438f700 1 -- 172.21.15.112:0/719642409 shutdown_connections 2024-06-04T04:47:01.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:01.053+0000 7f48d438f700 1 -- 172.21.15.112:0/719642409 wait complete. 2024-06-04T04:47:01.056 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-06-04T04:47:01.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:01 smithi070 ceph-mon[32631]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:01.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:01 smithi070 ceph-mon[32631]: from='client.? 172.21.15.112:0/719642409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-04T04:47:02.641 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-04T04:47:02.641 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-04T04:47:03.017 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.c/config 2024-06-04T04:47:03.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:02 smithi070 ceph-mon[32631]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:03.142 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:03 smithi112 systemd[1]: Starting Ceph mon.c for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T04:47:04.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.493+0000 7f353ef23700 1 -- 172.21.15.112:0/1276261542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3538102190 msgr2=0x7f3538102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:04.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.493+0000 7f353ef23700 1 --2- 172.21.15.112:0/1276261542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3538102190 0x7f3538102570 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3530009910 tx=0x7f3530009c20 comp rx=0 tx=0).stop 2024-06-04T04:47:04.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.493+0000 7f353ef23700 1 -- 172.21.15.112:0/1276261542 shutdown_connections 2024-06-04T04:47:04.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.493+0000 7f353ef23700 1 --2- 172.21.15.112:0/1276261542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3538102190 0x7f3538102570 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:04.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.493+0000 7f353ef23700 1 -- 172.21.15.112:0/1276261542 >> 172.21.15.112:0/1276261542 conn(0x7f35380fda20 msgr2=0x7f35380ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:47:04.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.493+0000 7f353ef23700 1 -- 172.21.15.112:0/1276261542 shutdown_connections 2024-06-04T04:47:04.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.493+0000 7f353ef23700 1 -- 172.21.15.112:0/1276261542 wait complete. 2024-06-04T04:47:04.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.493+0000 7f353ef23700 1 Processor -- start 2024-06-04T04:47:04.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.493+0000 7f353ef23700 1 -- start start 2024-06-04T04:47:04.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.494+0000 7f353ef23700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3538102190 0x7f3538103c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:04.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.494+0000 7f353ef23700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f353810c2c0 con 0x7f3538102190 2024-06-04T04:47:04.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.494+0000 7f353ccbf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3538102190 0x7f3538103c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:04.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.494+0000 7f353ccbf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3538102190 0x7f3538103c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.112:36410/0 (socket says 172.21.15.112:36410) 2024-06-04T04:47:04.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.494+0000 7f353ccbf700 1 -- 172.21.15.112:0/2869857187 learned_addr learned my addr 172.21.15.112:0/2869857187 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-06-04T04:47:04.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.494+0000 7f353ccbf700 1 -- 172.21.15.112:0/2869857187 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3538105b20 con 0x7f3538102190 2024-06-04T04:47:04.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.494+0000 7f353ccbf700 1 --2- 172.21.15.112:0/2869857187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3538102190 0x7f3538103c30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3530005660 tx=0x7f3530005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:47:04.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.495+0000 7f3535ffb700 1 -- 172.21.15.112:0/2869857187 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3530026070 con 0x7f3538102190 2024-06-04T04:47:04.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.495+0000 7f353ef23700 1 -- 172.21.15.112:0/2869857187 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35381041d0 con 0x7f3538102190 2024-06-04T04:47:04.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.495+0000 7f3535ffb700 1 -- 172.21.15.112:0/2869857187 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f353000b750 con 0x7f3538102190 2024-06-04T04:47:04.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.495+0000 7f3535ffb700 1 -- 172.21.15.112:0/2869857187 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35300037b0 con 0x7f3538102190 2024-06-04T04:47:04.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.495+0000 7f353ef23700 1 -- 172.21.15.112:0/2869857187 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3538104690 con 0x7f3538102190 2024-06-04T04:47:04.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.496+0000 7f353ef23700 1 -- 172.21.15.112:0/2869857187 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3538103670 con 0x7f3538102190 2024-06-04T04:47:04.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.496+0000 7f3535ffb700 1 -- 172.21.15.112:0/2869857187 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f3530029070 con 0x7f3538102190 2024-06-04T04:47:04.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.496+0000 7f3535ffb700 1 --2- 172.21.15.112:0/2869857187 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f352403e510 0x7f35240409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:04.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.496+0000 7f3535ffb700 1 -- 172.21.15.112:0/2869857187 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f353001e070 con 0x7f3538102190 2024-06-04T04:47:04.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.496+0000 7f3537fff700 1 --2- 172.21.15.112:0/2869857187 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f352403e510 0x7f35240409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:04.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.497+0000 7f3537fff700 1 --2- 172.21.15.112:0/2869857187 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f352403e510 0x7f35240409d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f352800ad30 tx=0x7f35280093f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:47:04.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.502+0000 7f3535ffb700 1 -- 172.21.15.112:0/2869857187 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3530013070 con 0x7f3538102190 2024-06-04T04:47:04.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.805+0000 7f353ef23700 1 -- 172.21.15.112:0/2869857187 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f3538105200 con 0x7f3538102190 2024-06-04T04:47:04.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.806+0000 7f3535ffb700 1 -- 172.21.15.112:0/2869857187 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f3530030090 con 0x7f3538102190 2024-06-04T04:47:04.808 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:47:04.808 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","modified":"2024-06-04T04:45:09.662402Z","created":"2024-06-04T04:45:09.662402Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-04T04:47:04.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.808+0000 7f353ef23700 1 -- 172.21.15.112:0/2869857187 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f352403e510 msgr2=0x7f35240409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:04.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.808+0000 7f353ef23700 1 --2- 172.21.15.112:0/2869857187 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f352403e510 0x7f35240409d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f352800ad30 tx=0x7f35280093f0 comp rx=0 tx=0).stop 2024-06-04T04:47:04.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.808+0000 7f353ef23700 1 -- 172.21.15.112:0/2869857187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3538102190 msgr2=0x7f3538103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:04.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.808+0000 7f353ef23700 1 --2- 172.21.15.112:0/2869857187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3538102190 0x7f3538103c30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3530005660 tx=0x7f3530005740 comp rx=0 tx=0).stop 2024-06-04T04:47:04.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.808+0000 7f353ef23700 1 -- 172.21.15.112:0/2869857187 shutdown_connections 2024-06-04T04:47:04.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.808+0000 7f353ef23700 1 --2- 172.21.15.112:0/2869857187 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f352403e510 0x7f35240409d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:04.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.808+0000 7f353ef23700 1 --2- 172.21.15.112:0/2869857187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3538102190 0x7f3538103c30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:04.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.808+0000 7f353ef23700 1 -- 172.21.15.112:0/2869857187 >> 172.21.15.112:0/2869857187 conn(0x7f35380fda20 msgr2=0x7f353806aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:47:04.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.808+0000 7f353ef23700 1 -- 172.21.15.112:0/2869857187 shutdown_connections 2024-06-04T04:47:04.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:04.808+0000 7f353ef23700 1 -- 172.21.15.112:0/2869857187 wait complete. 2024-06-04T04:47:04.812 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-06-04T04:47:04.865 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 podman[36399]: 2024-06-04 04:47:04.591136735 +0000 UTC m=+0.285491676 container create ddd0d2b2cd3f9e471da04690bd0e0c608fc3cb988e25a4182ce74a8bfdeebd2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-c, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-b7328fe, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-06-04T04:47:04.865 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 podman[36399]: 2024-06-04 04:47:04.501907842 +0000 UTC m=+0.196262795 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T04:47:04.865 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 podman[36399]: 2024-06-04 04:47:04.835855631 +0000 UTC m=+0.530210589 container init ddd0d2b2cd3f9e471da04690bd0e0c608fc3cb988e25a4182ce74a8bfdeebd2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-b7328fe, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-06-04T04:47:04.865 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 podman[36399]: 2024-06-04 04:47:04.839648158 +0000 UTC m=+0.534003112 container start ddd0d2b2cd3f9e471da04690bd0e0c608fc3cb988e25a4182ce74a8bfdeebd2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-06-04T04:47:04.865 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: set uid:gid to 167:167 (ceph:ceph) 2024-06-04T04:47:04.865 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: ceph version 19.0.0-3882-gb7328fed (b7328fed0e2e9ab66b09f8134f3cebfc71214504) squid (dev), process ceph-mon, pid 2 2024-06-04T04:47:04.865 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: pidfile_write: ignore empty --pid-file 2024-06-04T04:47:05.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:04 smithi070 ceph-mon[32631]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:05.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: load: jerasure load: lrc 2024-06-04T04:47:05.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: RocksDB version: 7.9.2 2024-06-04T04:47:05.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Git sha 0 2024-06-04T04:47:05.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Compile date 2024-05-22 16:22:08 2024-06-04T04:47:05.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: DB SUMMARY 2024-06-04T04:47:05.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: DB Session ID: CNFJTX5A1GYFDUKQNDYE 2024-06-04T04:47:05.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: CURRENT file: CURRENT 2024-06-04T04:47:05.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: IDENTITY file: IDENTITY 2024-06-04T04:47:05.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-06-04T04:47:05.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-06-04T04:47:05.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-06-04T04:47:05.188 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.error_if_exists: 0 2024-06-04T04:47:05.188 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.create_if_missing: 0 2024-06-04T04:47:05.188 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.paranoid_checks: 1 2024-06-04T04:47:05.188 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-04T04:47:05.188 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-04T04:47:05.188 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-04T04:47:05.188 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.env: 0x564b9d22fcc0 2024-06-04T04:47:05.188 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.fs: PosixFileSystem 2024-06-04T04:47:05.188 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.info_log: 0x564b9e7a5da0 2024-06-04T04:47:05.188 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_file_opening_threads: 16 2024-06-04T04:47:05.188 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.statistics: (nil) 2024-06-04T04:47:05.188 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.use_fsync: 0 2024-06-04T04:47:05.188 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_log_file_size: 0 2024-06-04T04:47:05.189 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-04T04:47:05.189 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-04T04:47:05.189 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.keep_log_file_num: 1000 2024-06-04T04:47:05.189 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.recycle_log_file_num: 0 2024-06-04T04:47:05.189 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.allow_fallocate: 1 2024-06-04T04:47:05.189 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.allow_mmap_reads: 0 2024-06-04T04:47:05.189 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.allow_mmap_writes: 0 2024-06-04T04:47:05.189 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.use_direct_reads: 0 2024-06-04T04:47:05.189 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-04T04:47:05.189 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.create_missing_column_families: 0 2024-06-04T04:47:05.189 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.db_log_dir: 2024-06-04T04:47:05.189 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.wal_dir: 2024-06-04T04:47:05.189 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-04T04:47:05.190 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-04T04:47:05.190 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-04T04:47:05.190 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-04T04:47:05.190 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-04T04:47:05.190 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-04T04:47:05.190 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.advise_random_on_open: 1 2024-06-04T04:47:05.190 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.db_write_buffer_size: 0 2024-06-04T04:47:05.190 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.write_buffer_manager: 0x564b9e7770e0 2024-06-04T04:47:05.190 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-04T04:47:05.190 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-04T04:47:05.190 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-04T04:47:05.190 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.rate_limiter: (nil) 2024-06-04T04:47:05.190 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-04T04:47:05.190 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.wal_recovery_mode: 2 2024-06-04T04:47:05.191 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.enable_thread_tracking: 0 2024-06-04T04:47:05.191 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.enable_pipelined_write: 0 2024-06-04T04:47:05.191 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.unordered_write: 0 2024-06-04T04:47:05.191 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-04T04:47:05.191 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-04T04:47:05.191 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-04T04:47:05.191 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-04T04:47:05.191 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.row_cache: None 2024-06-04T04:47:05.191 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.wal_filter: None 2024-06-04T04:47:05.191 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-04T04:47:05.191 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.allow_ingest_behind: 0 2024-06-04T04:47:05.191 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.two_write_queues: 0 2024-06-04T04:47:05.191 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.manual_wal_flush: 0 2024-06-04T04:47:05.191 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.wal_compression: 0 2024-06-04T04:47:05.192 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.atomic_flush: 0 2024-06-04T04:47:05.192 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-04T04:47:05.192 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-04T04:47:05.192 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-04T04:47:05.192 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.log_readahead_size: 0 2024-06-04T04:47:05.192 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-04T04:47:05.192 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.best_efforts_recovery: 0 2024-06-04T04:47:05.192 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-04T04:47:05.192 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-04T04:47:05.192 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.allow_data_in_errors: 0 2024-06-04T04:47:05.192 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.db_host_id: __hostname__ 2024-06-04T04:47:05.192 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-04T04:47:05.192 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_background_jobs: 2 2024-06-04T04:47:05.192 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_background_compactions: -1 2024-06-04T04:47:05.193 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_subcompactions: 1 2024-06-04T04:47:05.194 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-04T04:47:05.194 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-04T04:47:05.194 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-04T04:47:05.194 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_total_wal_size: 0 2024-06-04T04:47:05.194 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-04T04:47:05.195 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-04T04:47:05.195 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-04T04:47:05.195 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-04T04:47:05.195 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_open_files: -1 2024-06-04T04:47:05.195 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.bytes_per_sync: 0 2024-06-04T04:47:05.195 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-04T04:47:05.195 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-04T04:47:05.195 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compaction_readahead_size: 0 2024-06-04T04:47:05.195 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_background_flushes: -1 2024-06-04T04:47:05.195 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Compression algorithms supported: 2024-06-04T04:47:05.195 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-04T04:47:05.195 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: kZSTD supported: 0 2024-06-04T04:47:05.195 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: kXpressCompression supported: 0 2024-06-04T04:47:05.195 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: kLZ4HCCompression supported: 1 2024-06-04T04:47:05.196 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: kZlibCompression supported: 1 2024-06-04T04:47:05.196 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: kSnappyCompression supported: 1 2024-06-04T04:47:05.196 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: kLZ4Compression supported: 1 2024-06-04T04:47:05.196 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: kBZip2Compression supported: 0 2024-06-04T04:47:05.196 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-04T04:47:05.196 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-04T04:47:05.196 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-06-04T04:47:05.196 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-04T04:47:05.196 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-04T04:47:05.196 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.merge_operator: 2024-06-04T04:47:05.196 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compaction_filter: None 2024-06-04T04:47:05.196 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compaction_filter_factory: None 2024-06-04T04:47:05.196 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.sst_partitioner_factory: None 2024-06-04T04:47:05.196 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-04T04:47:05.197 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-04T04:47:05.197 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564b9e7a5f60) 2024-06-04T04:47:05.197 INFO:journalctl@ceph.mon.c.smithi112.stdout: cache_index_and_filter_blocks: 1 2024-06-04T04:47:05.197 INFO:journalctl@ceph.mon.c.smithi112.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-04T04:47:05.197 INFO:journalctl@ceph.mon.c.smithi112.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-04T04:47:05.197 INFO:journalctl@ceph.mon.c.smithi112.stdout: pin_top_level_index_and_filter: 1 2024-06-04T04:47:05.197 INFO:journalctl@ceph.mon.c.smithi112.stdout: index_type: 0 2024-06-04T04:47:05.197 INFO:journalctl@ceph.mon.c.smithi112.stdout: data_block_index_type: 0 2024-06-04T04:47:05.197 INFO:journalctl@ceph.mon.c.smithi112.stdout: index_shortening: 1 2024-06-04T04:47:05.197 INFO:journalctl@ceph.mon.c.smithi112.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-04T04:47:05.197 INFO:journalctl@ceph.mon.c.smithi112.stdout: checksum: 4 2024-06-04T04:47:05.197 INFO:journalctl@ceph.mon.c.smithi112.stdout: no_block_cache: 0 2024-06-04T04:47:05.197 INFO:journalctl@ceph.mon.c.smithi112.stdout: block_cache: 0x564b9e7a91f0 2024-06-04T04:47:05.198 INFO:journalctl@ceph.mon.c.smithi112.stdout: block_cache_name: BinnedLRUCache 2024-06-04T04:47:05.198 INFO:journalctl@ceph.mon.c.smithi112.stdout: block_cache_options: 2024-06-04T04:47:05.198 INFO:journalctl@ceph.mon.c.smithi112.stdout: capacity : 536870912 2024-06-04T04:47:05.198 INFO:journalctl@ceph.mon.c.smithi112.stdout: num_shard_bits : 4 2024-06-04T04:47:05.198 INFO:journalctl@ceph.mon.c.smithi112.stdout: strict_capacity_limit : 0 2024-06-04T04:47:05.198 INFO:journalctl@ceph.mon.c.smithi112.stdout: high_pri_pool_ratio: 0.000 2024-06-04T04:47:05.198 INFO:journalctl@ceph.mon.c.smithi112.stdout: block_cache_compressed: (nil) 2024-06-04T04:47:05.198 INFO:journalctl@ceph.mon.c.smithi112.stdout: persistent_cache: (nil) 2024-06-04T04:47:05.198 INFO:journalctl@ceph.mon.c.smithi112.stdout: block_size: 4096 2024-06-04T04:47:05.198 INFO:journalctl@ceph.mon.c.smithi112.stdout: block_size_deviation: 10 2024-06-04T04:47:05.198 INFO:journalctl@ceph.mon.c.smithi112.stdout: block_restart_interval: 16 2024-06-04T04:47:05.198 INFO:journalctl@ceph.mon.c.smithi112.stdout: index_block_restart_interval: 1 2024-06-04T04:47:05.198 INFO:journalctl@ceph.mon.c.smithi112.stdout: metadata_block_size: 4096 2024-06-04T04:47:05.199 INFO:journalctl@ceph.mon.c.smithi112.stdout: partition_filters: 0 2024-06-04T04:47:05.199 INFO:journalctl@ceph.mon.c.smithi112.stdout: use_delta_encoding: 1 2024-06-04T04:47:05.199 INFO:journalctl@ceph.mon.c.smithi112.stdout: filter_policy: bloomfilter 2024-06-04T04:47:05.199 INFO:journalctl@ceph.mon.c.smithi112.stdout: whole_key_filtering: 1 2024-06-04T04:47:05.199 INFO:journalctl@ceph.mon.c.smithi112.stdout: verify_compression: 0 2024-06-04T04:47:05.199 INFO:journalctl@ceph.mon.c.smithi112.stdout: read_amp_bytes_per_bit: 0 2024-06-04T04:47:05.199 INFO:journalctl@ceph.mon.c.smithi112.stdout: format_version: 5 2024-06-04T04:47:05.199 INFO:journalctl@ceph.mon.c.smithi112.stdout: enable_index_compression: 1 2024-06-04T04:47:05.199 INFO:journalctl@ceph.mon.c.smithi112.stdout: block_align: 0 2024-06-04T04:47:05.199 INFO:journalctl@ceph.mon.c.smithi112.stdout: max_auto_readahead_size: 262144 2024-06-04T04:47:05.199 INFO:journalctl@ceph.mon.c.smithi112.stdout: prepopulate_block_cache: 0 2024-06-04T04:47:05.199 INFO:journalctl@ceph.mon.c.smithi112.stdout: initial_auto_readahead_size: 8192 2024-06-04T04:47:05.199 INFO:journalctl@ceph.mon.c.smithi112.stdout: num_file_reads_for_auto_readahead: 2 2024-06-04T04:47:05.199 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.write_buffer_size: 33554432 2024-06-04T04:47:05.200 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_write_buffer_number: 2 2024-06-04T04:47:05.200 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compression: NoCompression 2024-06-04T04:47:05.200 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.bottommost_compression: Disabled 2024-06-04T04:47:05.200 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.prefix_extractor: nullptr 2024-06-04T04:47:05.200 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-04T04:47:05.200 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.num_levels: 7 2024-06-04T04:47:05.200 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-04T04:47:05.200 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-04T04:47:05.200 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-04T04:47:05.200 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-04T04:47:05.200 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-04T04:47:05.200 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-04T04:47:05.200 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-04T04:47:05.200 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-04T04:47:05.200 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-04T04:47:05.201 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-04T04:47:05.201 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-04T04:47:05.201 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-04T04:47:05.201 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-04T04:47:05.201 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compression_opts.level: 32767 2024-06-04T04:47:05.201 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compression_opts.strategy: 0 2024-06-04T04:47:05.201 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-04T04:47:05.201 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-04T04:47:05.201 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-04T04:47:05.201 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-04T04:47:05.201 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compression_opts.enabled: false 2024-06-04T04:47:05.201 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-04T04:47:05.201 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-04T04:47:05.201 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-04T04:47:05.202 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-04T04:47:05.202 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.target_file_size_base: 67108864 2024-06-04T04:47:05.202 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-04T04:47:05.202 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-04T04:47:05.202 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-04T04:47:05.202 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-04T04:47:05.202 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-04T04:47:05.202 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-04T04:47:05.202 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-04T04:47:05.202 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-04T04:47:05.202 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-04T04:47:05.202 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-04T04:47:05.202 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-04T04:47:05.202 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-04T04:47:05.203 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-04T04:47:05.203 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-04T04:47:05.203 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.arena_block_size: 1048576 2024-06-04T04:47:05.203 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-04T04:47:05.203 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-04T04:47:05.203 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.disable_auto_compactions: 0 2024-06-04T04:47:05.203 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-04T04:47:05.203 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-04T04:47:05.203 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-04T04:47:05.203 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-04T04:47:05.203 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-04T04:47:05.203 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-04T04:47:05.203 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-04T04:47:05.203 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-04T04:47:05.204 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-04T04:47:05.204 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-04T04:47:05.204 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-06-04T04:47:05.204 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.inplace_update_support: 0 2024-06-04T04:47:05.204 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-04T04:47:05.204 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-04T04:47:05.204 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-04T04:47:05.204 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-04T04:47:05.204 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.bloom_locality: 0 2024-06-04T04:47:05.204 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.max_successive_merges: 0 2024-06-04T04:47:05.204 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-04T04:47:05.204 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.paranoid_file_checks: 0 2024-06-04T04:47:05.204 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.force_consistency_checks: 1 2024-06-04T04:47:05.204 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.report_bg_io_stats: 0 2024-06-04T04:47:05.205 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.ttl: 2592000 2024-06-04T04:47:05.205 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-04T04:47:05.205 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-04T04:47:05.205 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-04T04:47:05.205 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.enable_blob_files: false 2024-06-04T04:47:05.205 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.min_blob_size: 0 2024-06-04T04:47:05.205 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.blob_file_size: 268435456 2024-06-04T04:47:05.205 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-04T04:47:05.205 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-04T04:47:05.205 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-04T04:47:05.205 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-04T04:47:05.205 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-04T04:47:05.205 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.blob_file_starting_level: 0 2024-06-04T04:47:05.206 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-04T04:47:05.206 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-04T04:47:05.206 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-06-04T04:47:05.206 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 502ba25b-7bef-40df-9aea-8c94523bcbe2 2024-06-04T04:47:05.206 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717476424870516, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-06-04T04:47:05.206 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-06-04T04:47:05.206 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717476424871015, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1717476424, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "502ba25b-7bef-40df-9aea-8c94523bcbe2", "db_session_id": "CNFJTX5A1GYFDUKQNDYE", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-06-04T04:47:05.206 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717476424871110, "job": 1, "event": "recovery_finished"} 2024-06-04T04:47:05.206 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-06-04T04:47:05.206 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-04T04:47:05.206 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564b9e8c6000 2024-06-04T04:47:05.206 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: DB pointer 0x564b9e8b2000 2024-06-04T04:47:05.206 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-04T04:47:05.207 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-04T04:47:05.207 INFO:journalctl@ceph.mon.c.smithi112.stdout: ** DB Stats ** 2024-06-04T04:47:05.207 INFO:journalctl@ceph.mon.c.smithi112.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-04T04:47:05.207 INFO:journalctl@ceph.mon.c.smithi112.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-04T04:47:05.207 INFO:journalctl@ceph.mon.c.smithi112.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-04T04:47:05.207 INFO:journalctl@ceph.mon.c.smithi112.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-04T04:47:05.207 INFO:journalctl@ceph.mon.c.smithi112.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-04T04:47:05.207 INFO:journalctl@ceph.mon.c.smithi112.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-04T04:47:05.207 INFO:journalctl@ceph.mon.c.smithi112.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-04T04:47:05.207 INFO:journalctl@ceph.mon.c.smithi112.stdout: 2024-06-04T04:47:05.207 INFO:journalctl@ceph.mon.c.smithi112.stdout: ** Compaction Stats [default] ** 2024-06-04T04:47:05.207 INFO:journalctl@ceph.mon.c.smithi112.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-04T04:47:05.207 INFO:journalctl@ceph.mon.c.smithi112.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-04T04:47:05.207 INFO:journalctl@ceph.mon.c.smithi112.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-06-04T04:47:05.208 INFO:journalctl@ceph.mon.c.smithi112.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-06-04T04:47:05.208 INFO:journalctl@ceph.mon.c.smithi112.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-06-04T04:47:05.208 INFO:journalctl@ceph.mon.c.smithi112.stdout: 2024-06-04T04:47:05.208 INFO:journalctl@ceph.mon.c.smithi112.stdout: ** Compaction Stats [default] ** 2024-06-04T04:47:05.208 INFO:journalctl@ceph.mon.c.smithi112.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-04T04:47:05.208 INFO:journalctl@ceph.mon.c.smithi112.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-04T04:47:05.208 INFO:journalctl@ceph.mon.c.smithi112.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-06-04T04:47:05.208 INFO:journalctl@ceph.mon.c.smithi112.stdout: 2024-06-04T04:47:05.208 INFO:journalctl@ceph.mon.c.smithi112.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-04T04:47:05.208 INFO:journalctl@ceph.mon.c.smithi112.stdout: 2024-06-04T04:47:05.208 INFO:journalctl@ceph.mon.c.smithi112.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-04T04:47:05.208 INFO:journalctl@ceph.mon.c.smithi112.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-04T04:47:05.208 INFO:journalctl@ceph.mon.c.smithi112.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-04T04:47:05.208 INFO:journalctl@ceph.mon.c.smithi112.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-04T04:47:05.209 INFO:journalctl@ceph.mon.c.smithi112.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-04T04:47:05.209 INFO:journalctl@ceph.mon.c.smithi112.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-04T04:47:05.209 INFO:journalctl@ceph.mon.c.smithi112.stdout: Cumulative compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-04T04:47:05.209 INFO:journalctl@ceph.mon.c.smithi112.stdout: Interval compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-04T04:47:05.209 INFO:journalctl@ceph.mon.c.smithi112.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-04T04:47:05.209 INFO:journalctl@ceph.mon.c.smithi112.stdout: Block cache BinnedLRUCache@0x564b9e7a91f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-06-04T04:47:05.209 INFO:journalctl@ceph.mon.c.smithi112.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-06-04T04:47:05.209 INFO:journalctl@ceph.mon.c.smithi112.stdout: 2024-06-04T04:47:05.209 INFO:journalctl@ceph.mon.c.smithi112.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-04T04:47:05.209 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-06-04T04:47:05.209 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: using public_addr v2:172.21.15.112:0/0 -> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-06-04T04:47:05.209 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: starting mon.c rank -1 at public addrs [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] at bind addrs [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:47:05.209 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: mon.c@-1(???) e0 preinit fsid 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:47:05.209 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: mon.c@-1(synchronizing).mds e1 new map 2024-06-04T04:47:05.210 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: mon.c@-1(synchronizing).mds e1 print_map 2024-06-04T04:47:05.210 INFO:journalctl@ceph.mon.c.smithi112.stdout: e1 2024-06-04T04:47:05.210 INFO:journalctl@ceph.mon.c.smithi112.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-04T04:47:05.210 INFO:journalctl@ceph.mon.c.smithi112.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-06-04T04:47:05.210 INFO:journalctl@ceph.mon.c.smithi112.stdout: legacy client fscid: -1 2024-06-04T04:47:05.210 INFO:journalctl@ceph.mon.c.smithi112.stdout: 2024-06-04T04:47:05.210 INFO:journalctl@ceph.mon.c.smithi112.stdout: No filesystems configured 2024-06-04T04:47:05.210 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-06-04T04:47:05.210 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-06-04T04:47:05.210 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-06-04T04:47:05.210 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-06-04T04:47:05.210 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-06-04T04:47:05.210 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-06-04T04:47:05.211 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-04T04:47:05.211 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-04T04:47:05.211 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-04T04:47:05.211 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-04T04:47:05.211 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi112", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:47:05.211 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.211 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.211 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.211 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.211 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:47:05.211 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:05.211 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:05.211 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: Updating smithi098:/etc/ceph/ceph.conf 2024-06-04T04:47:05.211 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:05.212 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: Deploying cephadm binary to smithi112 2024-06-04T04:47:05.212 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: Updating smithi098:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:47:05.212 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:47:05.212 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:05.212 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.212 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.212 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.212 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.212 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:05.212 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: Updating smithi098:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-04T04:47:05.212 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: Added host smithi112 2024-06-04T04:47:05.212 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.212 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.212 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:05.212 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:05.213 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:47:05.213 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.213 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:05.213 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3744553518' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-04T04:47:05.213 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3744553518' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-04T04:47:05.213 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: osdmap e4: 0 total, 0 up, 0 in 2024-06-04T04:47:05.213 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.213 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.213 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.213 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.213 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:47:05.213 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:05.213 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:05.213 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: Updating smithi112:/etc/ceph/ceph.conf 2024-06-04T04:47:05.213 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: Updating smithi112:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:47:05.214 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:47:05.214 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:05.214 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: Updating smithi112:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-04T04:47:05.214 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.214 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.214 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.214 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi070:172.21.15.70=a;smithi098:172.21.15.98=b;smithi112:172.21.15.112=c", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:47:05.214 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: Saving service mon spec with placement smithi070:172.21.15.70=a;smithi098:172.21.15.98=b;smithi112:172.21.15.112=c;count:3 2024-06-04T04:47:05.214 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.214 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:05.214 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:05.214 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:05.214 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:05.215 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-04T04:47:05.215 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:05.215 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: Deploying daemon mon.c on smithi112 2024-06-04T04:47:05.215 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:05.215 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:05.215 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: from='client.? 172.21.15.112:0/719642409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-04T04:47:05.215 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:05.215 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:05.215 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:04 smithi112 ceph-mon[36453]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-06-04T04:47:05.532 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:05 smithi112 bash[36399]: ddd0d2b2cd3f9e471da04690bd0e0c608fc3cb988e25a4182ce74a8bfdeebd2c 2024-06-04T04:47:05.532 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:05 smithi112 systemd[1]: Started Ceph mon.c for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:47:06.560 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-04T04:47:06.560 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-04T04:47:06.804 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.c/config 2024-06-04T04:47:10.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:47:10.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:10.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: mon.a calling monitor election 2024-06-04T04:47:10.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:10.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:10.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:10.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: mon.c calling monitor election 2024-06-04T04:47:10.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:10.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:10.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:10.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:10.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-04T04:47:10.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: monmap epoch 2 2024-06-04T04:47:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: fsid 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:47:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: last_changed 2024-06-04T04:47:04.895907+0000 2024-06-04T04:47:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: created 2024-06-04T04:45:09.662402+0000 2024-06-04T04:47:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: min_mon_release 19 (squid) 2024-06-04T04:47:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: election_strategy: 1 2024-06-04T04:47:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-06-04T04:47:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: 1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.c 2024-06-04T04:47:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: fsmap 2024-06-04T04:47:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: osdmap e4: 0 total, 0 up, 0 in 2024-06-04T04:47:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: mgrmap e13: x(active, since 52s) 2024-06-04T04:47:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: overall HEALTH_OK 2024-06-04T04:47:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:10.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-04T04:47:10.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:10.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:47:10.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: mon.a calling monitor election 2024-06-04T04:47:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: mon.c calling monitor election 2024-06-04T04:47:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-04T04:47:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: monmap epoch 2 2024-06-04T04:47:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: fsid 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:47:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: last_changed 2024-06-04T04:47:04.895907+0000 2024-06-04T04:47:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: created 2024-06-04T04:45:09.662402+0000 2024-06-04T04:47:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: min_mon_release 19 (squid) 2024-06-04T04:47:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: election_strategy: 1 2024-06-04T04:47:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-06-04T04:47:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: 1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.c 2024-06-04T04:47:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: fsmap 2024-06-04T04:47:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: osdmap e4: 0 total, 0 up, 0 in 2024-06-04T04:47:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: mgrmap e13: x(active, since 52s) 2024-06-04T04:47:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: overall HEALTH_OK 2024-06-04T04:47:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:10.439 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-04T04:47:10.439 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:11.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.273+0000 7fc1533ae700 1 -- 172.21.15.112:0/2631084124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1340035c0 msgr2=0x7fc134005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:11.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.273+0000 7fc1533ae700 1 --2- 172.21.15.112:0/2631084124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1340035c0 0x7fc134005a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc14c06af00 tx=0x7fc144009e80 comp rx=0 tx=0).stop 2024-06-04T04:47:11.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.273+0000 7fc1533ae700 1 -- 172.21.15.112:0/2631084124 shutdown_connections 2024-06-04T04:47:11.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.273+0000 7fc1533ae700 1 --2- 172.21.15.112:0/2631084124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1340035c0 0x7fc134005a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:11.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.273+0000 7fc1533ae700 1 -- 172.21.15.112:0/2631084124 >> 172.21.15.112:0/2631084124 conn(0x7fc14c0fda20 msgr2=0x7fc14c0ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:47:11.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.273+0000 7fc1533ae700 1 -- 172.21.15.112:0/2631084124 shutdown_connections 2024-06-04T04:47:11.276 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:10 smithi112 ceph-mon[36453]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:11.276 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:10 smithi112 ceph-mon[36453]: Deploying daemon mon.b on smithi098 2024-06-04T04:47:11.276 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:10 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:11.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.273+0000 7fc1533ae700 1 -- 172.21.15.112:0/2631084124 wait complete. 2024-06-04T04:47:11.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.274+0000 7fc1533ae700 1 Processor -- start 2024-06-04T04:47:11.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.274+0000 7fc1533ae700 1 -- start start 2024-06-04T04:47:11.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.275+0000 7fc1533ae700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1340035c0 0x7fc14c19e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:11.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.275+0000 7fc1533ae700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc14c19ec60 0x7fc14c1988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:11.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.275+0000 7fc1533ae700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc14c10ae30 con 0x7fc14c19ec60 2024-06-04T04:47:11.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.275+0000 7fc1533ae700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc14c10acb0 con 0x7fc1340035c0 2024-06-04T04:47:11.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.275+0000 7fc15114a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1340035c0 0x7fc14c19e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:11.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.275+0000 7fc15114a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1340035c0 0x7fc14c19e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:56476/0 (socket says 172.21.15.112:56476) 2024-06-04T04:47:11.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.275+0000 7fc15114a700 1 -- 172.21.15.112:0/4231489375 learned_addr learned my addr 172.21.15.112:0/4231489375 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-06-04T04:47:11.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.275+0000 7fc150949700 1 --2- 172.21.15.112:0/4231489375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc14c19ec60 0x7fc14c1988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:11.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.275+0000 7fc15114a700 1 -- 172.21.15.112:0/4231489375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1340035c0 msgr2=0x7fc14c19e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-06-04T04:47:11.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.275+0000 7fc15114a700 1 -- 172.21.15.112:0/4231489375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1340035c0 msgr2=0x7fc14c19e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-04T04:47:11.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.275+0000 7fc15114a700 1 --2- 172.21.15.112:0/4231489375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1340035c0 0x7fc14c19e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-04T04:47:11.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.275+0000 7fc15114a700 1 --2- 172.21.15.112:0/4231489375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1340035c0 0x7fc14c19e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-04T04:47:11.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.275+0000 7fc150949700 1 -- 172.21.15.112:0/4231489375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1340035c0 msgr2=0x7fc14c19e720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-04T04:47:11.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.275+0000 7fc150949700 1 --2- 172.21.15.112:0/4231489375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1340035c0 0x7fc14c19e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:11.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.275+0000 7fc150949700 1 -- 172.21.15.112:0/4231489375 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14c19f2a0 con 0x7fc14c19ec60 2024-06-04T04:47:11.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.275+0000 7fc150949700 1 --2- 172.21.15.112:0/4231489375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc14c19ec60 0x7fc14c1988e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc14000b4d0 tx=0x7fc14000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:47:11.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.276+0000 7fc14a7fc700 1 -- 172.21.15.112:0/4231489375 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc140011630 con 0x7fc14c19ec60 2024-06-04T04:47:11.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.276+0000 7fc14a7fc700 1 -- 172.21.15.112:0/4231489375 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc140011c60 con 0x7fc14c19ec60 2024-06-04T04:47:11.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.276+0000 7fc1533ae700 1 -- 172.21.15.112:0/4231489375 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc14c198e20 con 0x7fc14c19ec60 2024-06-04T04:47:11.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.276+0000 7fc14a7fc700 1 -- 172.21.15.112:0/4231489375 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc140010c90 con 0x7fc14c19ec60 2024-06-04T04:47:11.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.276+0000 7fc1533ae700 1 -- 172.21.15.112:0/4231489375 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc14c1993b0 con 0x7fc14c19ec60 2024-06-04T04:47:11.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.277+0000 7fc14a7fc700 1 -- 172.21.15.112:0/4231489375 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fc140018020 con 0x7fc14c19ec60 2024-06-04T04:47:11.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.277+0000 7fc1533ae700 1 -- 172.21.15.112:0/4231489375 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc14c199720 con 0x7fc14c19ec60 2024-06-04T04:47:11.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.277+0000 7fc14a7fc700 1 --2- 172.21.15.112:0/4231489375 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc13803e320 0x7fc1380407e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:11.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.277+0000 7fc14a7fc700 1 -- 172.21.15.112:0/4231489375 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc1400520e0 con 0x7fc14c19ec60 2024-06-04T04:47:11.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.277+0000 7fc15114a700 1 --2- 172.21.15.112:0/4231489375 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc13803e320 0x7fc1380407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:11.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.278+0000 7fc15114a700 1 --2- 172.21.15.112:0/4231489375 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc13803e320 0x7fc1380407e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc144007590 tx=0x7fc1440074e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:47:11.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.283+0000 7fc14a7fc700 1 -- 172.21.15.112:0/4231489375 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc14001f020 con 0x7fc14c19ec60 2024-06-04T04:47:11.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:10 smithi070 ceph-mon[32631]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:11.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:10 smithi070 ceph-mon[32631]: Deploying daemon mon.b on smithi098 2024-06-04T04:47:11.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:10 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:11.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.571+0000 7fc1533ae700 1 -- 172.21.15.112:0/4231489375 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc14c1a35e0 con 0x7fc14c19ec60 2024-06-04T04:47:11.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.572+0000 7fc14a7fc700 1 -- 172.21.15.112:0/4231489375 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fc1400187b0 con 0x7fc14c19ec60 2024-06-04T04:47:11.574 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:47:11.575 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":2,"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","modified":"2024-06-04T04:47:04.895907Z","created":"2024-06-04T04:45:09.662402Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-04T04:47:11.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.575+0000 7fc1533ae700 1 -- 172.21.15.112:0/4231489375 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc13803e320 msgr2=0x7fc1380407e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:11.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.575+0000 7fc1533ae700 1 --2- 172.21.15.112:0/4231489375 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc13803e320 0x7fc1380407e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc144007590 tx=0x7fc1440074e0 comp rx=0 tx=0).stop 2024-06-04T04:47:11.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.575+0000 7fc1533ae700 1 -- 172.21.15.112:0/4231489375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc14c19ec60 msgr2=0x7fc14c1988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:11.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.575+0000 7fc1533ae700 1 --2- 172.21.15.112:0/4231489375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc14c19ec60 0x7fc14c1988e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc14000b4d0 tx=0x7fc14000b890 comp rx=0 tx=0).stop 2024-06-04T04:47:11.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.575+0000 7fc1533ae700 1 -- 172.21.15.112:0/4231489375 shutdown_connections 2024-06-04T04:47:11.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.575+0000 7fc1533ae700 1 --2- 172.21.15.112:0/4231489375 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc13803e320 0x7fc1380407e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:11.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.575+0000 7fc1533ae700 1 --2- 172.21.15.112:0/4231489375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1340035c0 0x7fc14c19e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:11.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.575+0000 7fc1533ae700 1 --2- 172.21.15.112:0/4231489375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc14c19ec60 0x7fc14c1988e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:11.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.575+0000 7fc1533ae700 1 -- 172.21.15.112:0/4231489375 >> 172.21.15.112:0/4231489375 conn(0x7fc14c0fda20 msgr2=0x7fc14c105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:47:11.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.575+0000 7fc1533ae700 1 -- 172.21.15.112:0/4231489375 shutdown_connections 2024-06-04T04:47:11.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:11.575+0000 7fc1533ae700 1 -- 172.21.15.112:0/4231489375 wait complete. 2024-06-04T04:47:11.578 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 2 2024-06-04T04:47:12.152 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:11 smithi112 ceph-mon[36453]: from='client.? 172.21.15.112:0/4231489375' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-04T04:47:12.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:11 smithi070 ceph-mon[32631]: from='client.? 172.21.15.112:0/4231489375' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-04T04:47:13.179 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-04T04:47:13.179 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-04T04:47:13.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:12 smithi070 ceph-mon[32631]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:13.417 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.c/config 2024-06-04T04:47:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:12 smithi112 ceph-mon[36453]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:13.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 podman[35550]: 2024-06-04 04:47:13.637227161 +0000 UTC m=+0.118097793 container create f688c5292625b8c15e5207a27b04de0676ba1a58702cb1f3472c29579d6ebff4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-b, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-b7328fe, GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-06-04T04:47:14.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 podman[35550]: 2024-06-04 04:47:13.839457235 +0000 UTC m=+0.320327814 container init f688c5292625b8c15e5207a27b04de0676ba1a58702cb1f3472c29579d6ebff4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=main-b7328fe, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-06-04T04:47:14.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 podman[35550]: 2024-06-04 04:47:13.842589987 +0000 UTC m=+0.323460576 container start f688c5292625b8c15e5207a27b04de0676ba1a58702cb1f3472c29579d6ebff4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-b, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-06-04T04:47:14.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: set uid:gid to 167:167 (ceph:ceph) 2024-06-04T04:47:14.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: ceph version 19.0.0-3882-gb7328fed (b7328fed0e2e9ab66b09f8134f3cebfc71214504) squid (dev), process ceph-mon, pid 2 2024-06-04T04:47:14.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: pidfile_write: ignore empty --pid-file 2024-06-04T04:47:14.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: load: jerasure load: lrc 2024-06-04T04:47:14.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: RocksDB version: 7.9.2 2024-06-04T04:47:14.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Git sha 0 2024-06-04T04:47:14.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Compile date 2024-05-22 16:22:08 2024-06-04T04:47:14.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: DB SUMMARY 2024-06-04T04:47:14.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: DB Session ID: 49ENZ9T9WTIQKIN5SNYT 2024-06-04T04:47:14.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: CURRENT file: CURRENT 2024-06-04T04:47:14.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: IDENTITY file: IDENTITY 2024-06-04T04:47:14.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-06-04T04:47:14.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-06-04T04:47:14.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-06-04T04:47:14.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.error_if_exists: 0 2024-06-04T04:47:14.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.create_if_missing: 0 2024-06-04T04:47:14.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.paranoid_checks: 1 2024-06-04T04:47:14.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-04T04:47:14.248 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-04T04:47:14.248 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-04T04:47:14.248 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.env: 0x559a43846cc0 2024-06-04T04:47:14.248 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.fs: PosixFileSystem 2024-06-04T04:47:14.248 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.info_log: 0x559a45423da0 2024-06-04T04:47:14.248 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_file_opening_threads: 16 2024-06-04T04:47:14.248 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.statistics: (nil) 2024-06-04T04:47:14.248 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.use_fsync: 0 2024-06-04T04:47:14.248 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_log_file_size: 0 2024-06-04T04:47:14.248 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-04T04:47:14.248 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-04T04:47:14.248 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.keep_log_file_num: 1000 2024-06-04T04:47:14.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.recycle_log_file_num: 0 2024-06-04T04:47:14.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.allow_fallocate: 1 2024-06-04T04:47:14.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.allow_mmap_reads: 0 2024-06-04T04:47:14.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.allow_mmap_writes: 0 2024-06-04T04:47:14.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.use_direct_reads: 0 2024-06-04T04:47:14.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-04T04:47:14.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.create_missing_column_families: 0 2024-06-04T04:47:14.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.db_log_dir: 2024-06-04T04:47:14.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.wal_dir: 2024-06-04T04:47:14.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-04T04:47:14.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-04T04:47:14.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-04T04:47:14.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-04T04:47:14.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-04T04:47:14.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-04T04:47:14.250 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.advise_random_on_open: 1 2024-06-04T04:47:14.250 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.db_write_buffer_size: 0 2024-06-04T04:47:14.250 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.write_buffer_manager: 0x559a453f50e0 2024-06-04T04:47:14.250 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-04T04:47:14.250 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-04T04:47:14.250 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-04T04:47:14.250 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.rate_limiter: (nil) 2024-06-04T04:47:14.250 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-04T04:47:14.250 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.wal_recovery_mode: 2 2024-06-04T04:47:14.250 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.enable_thread_tracking: 0 2024-06-04T04:47:14.250 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.enable_pipelined_write: 0 2024-06-04T04:47:14.250 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.unordered_write: 0 2024-06-04T04:47:14.250 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-04T04:47:14.251 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-04T04:47:14.251 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-04T04:47:14.251 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-04T04:47:14.251 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.row_cache: None 2024-06-04T04:47:14.251 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.wal_filter: None 2024-06-04T04:47:14.252 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-04T04:47:14.252 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.allow_ingest_behind: 0 2024-06-04T04:47:14.252 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.two_write_queues: 0 2024-06-04T04:47:14.252 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.manual_wal_flush: 0 2024-06-04T04:47:14.252 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.wal_compression: 0 2024-06-04T04:47:14.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.atomic_flush: 0 2024-06-04T04:47:14.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-04T04:47:14.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-04T04:47:14.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-04T04:47:14.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.log_readahead_size: 0 2024-06-04T04:47:14.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-04T04:47:14.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.best_efforts_recovery: 0 2024-06-04T04:47:14.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-04T04:47:14.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-04T04:47:14.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.allow_data_in_errors: 0 2024-06-04T04:47:14.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.db_host_id: __hostname__ 2024-06-04T04:47:14.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-04T04:47:14.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_background_jobs: 2 2024-06-04T04:47:14.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_background_compactions: -1 2024-06-04T04:47:14.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_subcompactions: 1 2024-06-04T04:47:14.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-04T04:47:14.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-04T04:47:14.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-04T04:47:14.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_total_wal_size: 0 2024-06-04T04:47:14.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-04T04:47:14.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-04T04:47:14.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-04T04:47:14.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-04T04:47:14.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_open_files: -1 2024-06-04T04:47:14.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.bytes_per_sync: 0 2024-06-04T04:47:14.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-04T04:47:14.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-04T04:47:14.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compaction_readahead_size: 0 2024-06-04T04:47:14.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_background_flushes: -1 2024-06-04T04:47:14.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Compression algorithms supported: 2024-06-04T04:47:14.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-04T04:47:14.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: kZSTD supported: 0 2024-06-04T04:47:14.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: kXpressCompression supported: 0 2024-06-04T04:47:14.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: kLZ4HCCompression supported: 1 2024-06-04T04:47:14.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: kZlibCompression supported: 1 2024-06-04T04:47:14.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: kSnappyCompression supported: 1 2024-06-04T04:47:14.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: kLZ4Compression supported: 1 2024-06-04T04:47:14.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: kBZip2Compression supported: 0 2024-06-04T04:47:14.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-04T04:47:14.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-04T04:47:14.256 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-06-04T04:47:14.256 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-04T04:47:14.256 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-04T04:47:14.256 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.merge_operator: 2024-06-04T04:47:14.256 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compaction_filter: None 2024-06-04T04:47:14.256 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compaction_filter_factory: None 2024-06-04T04:47:14.256 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.sst_partitioner_factory: None 2024-06-04T04:47:14.256 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-04T04:47:14.256 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-04T04:47:14.256 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559a45423f60) 2024-06-04T04:47:14.256 INFO:journalctl@ceph.mon.b.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-06-04T04:47:14.256 INFO:journalctl@ceph.mon.b.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-04T04:47:14.256 INFO:journalctl@ceph.mon.b.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-04T04:47:14.256 INFO:journalctl@ceph.mon.b.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-06-04T04:47:14.256 INFO:journalctl@ceph.mon.b.smithi098.stdout: index_type: 0 2024-06-04T04:47:14.257 INFO:journalctl@ceph.mon.b.smithi098.stdout: data_block_index_type: 0 2024-06-04T04:47:14.257 INFO:journalctl@ceph.mon.b.smithi098.stdout: index_shortening: 1 2024-06-04T04:47:14.257 INFO:journalctl@ceph.mon.b.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-04T04:47:14.257 INFO:journalctl@ceph.mon.b.smithi098.stdout: checksum: 4 2024-06-04T04:47:14.257 INFO:journalctl@ceph.mon.b.smithi098.stdout: no_block_cache: 0 2024-06-04T04:47:14.257 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_cache: 0x559a454271f0 2024-06-04T04:47:14.257 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-06-04T04:47:14.257 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_cache_options: 2024-06-04T04:47:14.257 INFO:journalctl@ceph.mon.b.smithi098.stdout: capacity : 536870912 2024-06-04T04:47:14.257 INFO:journalctl@ceph.mon.b.smithi098.stdout: num_shard_bits : 4 2024-06-04T04:47:14.257 INFO:journalctl@ceph.mon.b.smithi098.stdout: strict_capacity_limit : 0 2024-06-04T04:47:14.257 INFO:journalctl@ceph.mon.b.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-06-04T04:47:14.257 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_cache_compressed: (nil) 2024-06-04T04:47:14.257 INFO:journalctl@ceph.mon.b.smithi098.stdout: persistent_cache: (nil) 2024-06-04T04:47:14.258 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_size: 4096 2024-06-04T04:47:14.258 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_size_deviation: 10 2024-06-04T04:47:14.258 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_restart_interval: 16 2024-06-04T04:47:14.258 INFO:journalctl@ceph.mon.b.smithi098.stdout: index_block_restart_interval: 1 2024-06-04T04:47:14.258 INFO:journalctl@ceph.mon.b.smithi098.stdout: metadata_block_size: 4096 2024-06-04T04:47:14.258 INFO:journalctl@ceph.mon.b.smithi098.stdout: partition_filters: 0 2024-06-04T04:47:14.258 INFO:journalctl@ceph.mon.b.smithi098.stdout: use_delta_encoding: 1 2024-06-04T04:47:14.258 INFO:journalctl@ceph.mon.b.smithi098.stdout: filter_policy: bloomfilter 2024-06-04T04:47:14.258 INFO:journalctl@ceph.mon.b.smithi098.stdout: whole_key_filtering: 1 2024-06-04T04:47:14.258 INFO:journalctl@ceph.mon.b.smithi098.stdout: verify_compression: 0 2024-06-04T04:47:14.258 INFO:journalctl@ceph.mon.b.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-06-04T04:47:14.258 INFO:journalctl@ceph.mon.b.smithi098.stdout: format_version: 5 2024-06-04T04:47:14.258 INFO:journalctl@ceph.mon.b.smithi098.stdout: enable_index_compression: 1 2024-06-04T04:47:14.258 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_align: 0 2024-06-04T04:47:14.259 INFO:journalctl@ceph.mon.b.smithi098.stdout: max_auto_readahead_size: 262144 2024-06-04T04:47:14.259 INFO:journalctl@ceph.mon.b.smithi098.stdout: prepopulate_block_cache: 0 2024-06-04T04:47:14.259 INFO:journalctl@ceph.mon.b.smithi098.stdout: initial_auto_readahead_size: 8192 2024-06-04T04:47:14.259 INFO:journalctl@ceph.mon.b.smithi098.stdout: num_file_reads_for_auto_readahead: 2 2024-06-04T04:47:14.259 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.write_buffer_size: 33554432 2024-06-04T04:47:14.259 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_write_buffer_number: 2 2024-06-04T04:47:14.259 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compression: NoCompression 2024-06-04T04:47:14.259 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.bottommost_compression: Disabled 2024-06-04T04:47:14.259 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.prefix_extractor: nullptr 2024-06-04T04:47:14.259 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-04T04:47:14.259 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.num_levels: 7 2024-06-04T04:47:14.259 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-04T04:47:14.259 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-04T04:47:14.259 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-04T04:47:14.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-04T04:47:14.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-04T04:47:14.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-04T04:47:14.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-04T04:47:14.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-04T04:47:14.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-04T04:47:14.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-04T04:47:14.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-04T04:47:14.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-04T04:47:14.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-04T04:47:14.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compression_opts.level: 32767 2024-06-04T04:47:14.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compression_opts.strategy: 0 2024-06-04T04:47:14.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-04T04:47:14.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-04T04:47:14.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-04T04:47:14.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-04T04:47:14.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compression_opts.enabled: false 2024-06-04T04:47:14.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-04T04:47:14.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-04T04:47:14.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-04T04:47:14.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-04T04:47:14.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.target_file_size_base: 67108864 2024-06-04T04:47:14.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-04T04:47:14.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-04T04:47:14.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-04T04:47:14.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-04T04:47:14.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-04T04:47:14.262 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-04T04:47:14.262 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-04T04:47:14.262 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-04T04:47:14.262 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-04T04:47:14.262 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-04T04:47:14.262 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-04T04:47:14.262 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-04T04:47:14.262 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-04T04:47:14.262 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-04T04:47:14.262 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.arena_block_size: 1048576 2024-06-04T04:47:14.262 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-04T04:47:14.262 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-04T04:47:14.262 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.disable_auto_compactions: 0 2024-06-04T04:47:14.263 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-04T04:47:14.263 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-04T04:47:14.263 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-04T04:47:14.263 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-04T04:47:14.263 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-04T04:47:14.263 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-04T04:47:14.263 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-04T04:47:14.263 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-04T04:47:14.263 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-04T04:47:14.263 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-04T04:47:14.263 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-06-04T04:47:14.263 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.inplace_update_support: 0 2024-06-04T04:47:14.263 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-04T04:47:14.263 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-04T04:47:14.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-04T04:47:14.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-04T04:47:14.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.bloom_locality: 0 2024-06-04T04:47:14.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.max_successive_merges: 0 2024-06-04T04:47:14.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-04T04:47:14.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.paranoid_file_checks: 0 2024-06-04T04:47:14.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.force_consistency_checks: 1 2024-06-04T04:47:14.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.report_bg_io_stats: 0 2024-06-04T04:47:14.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.ttl: 2592000 2024-06-04T04:47:14.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-04T04:47:14.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-04T04:47:14.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-04T04:47:14.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.enable_blob_files: false 2024-06-04T04:47:14.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.min_blob_size: 0 2024-06-04T04:47:14.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.blob_file_size: 268435456 2024-06-04T04:47:14.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-04T04:47:14.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-04T04:47:14.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-04T04:47:14.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-04T04:47:14.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-04T04:47:14.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.blob_file_starting_level: 0 2024-06-04T04:47:14.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-04T04:47:14.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-04T04:47:14.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-06-04T04:47:14.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e5ee9605-3937-49ea-a518-633c14fc05ce 2024-06-04T04:47:14.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717476433872687, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-06-04T04:47:14.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-06-04T04:47:14.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717476433873101, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1717476433, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e5ee9605-3937-49ea-a518-633c14fc05ce", "db_session_id": "49ENZ9T9WTIQKIN5SNYT", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-06-04T04:47:14.266 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717476433873180, "job": 1, "event": "recovery_finished"} 2024-06-04T04:47:14.266 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-06-04T04:47:14.266 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-04T04:47:14.266 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559a45544000 2024-06-04T04:47:14.266 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: DB pointer 0x559a45530000 2024-06-04T04:47:14.266 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-04T04:47:14.266 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-04T04:47:14.266 INFO:journalctl@ceph.mon.b.smithi098.stdout: ** DB Stats ** 2024-06-04T04:47:14.266 INFO:journalctl@ceph.mon.b.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-04T04:47:14.266 INFO:journalctl@ceph.mon.b.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-04T04:47:14.267 INFO:journalctl@ceph.mon.b.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-04T04:47:14.267 INFO:journalctl@ceph.mon.b.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-04T04:47:14.267 INFO:journalctl@ceph.mon.b.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-04T04:47:14.267 INFO:journalctl@ceph.mon.b.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-04T04:47:14.267 INFO:journalctl@ceph.mon.b.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-04T04:47:14.267 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-06-04T04:47:14.267 INFO:journalctl@ceph.mon.b.smithi098.stdout: ** Compaction Stats [default] ** 2024-06-04T04:47:14.267 INFO:journalctl@ceph.mon.b.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-04T04:47:14.267 INFO:journalctl@ceph.mon.b.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-04T04:47:14.268 INFO:journalctl@ceph.mon.b.smithi098.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-06-04T04:47:14.268 INFO:journalctl@ceph.mon.b.smithi098.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-06-04T04:47:14.268 INFO:journalctl@ceph.mon.b.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-06-04T04:47:14.268 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-06-04T04:47:14.268 INFO:journalctl@ceph.mon.b.smithi098.stdout: ** Compaction Stats [default] ** 2024-06-04T04:47:14.268 INFO:journalctl@ceph.mon.b.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-04T04:47:14.268 INFO:journalctl@ceph.mon.b.smithi098.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-04T04:47:14.268 INFO:journalctl@ceph.mon.b.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-06-04T04:47:14.268 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-06-04T04:47:14.268 INFO:journalctl@ceph.mon.b.smithi098.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-04T04:47:14.268 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-06-04T04:47:14.268 INFO:journalctl@ceph.mon.b.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-04T04:47:14.268 INFO:journalctl@ceph.mon.b.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-04T04:47:14.269 INFO:journalctl@ceph.mon.b.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-04T04:47:14.269 INFO:journalctl@ceph.mon.b.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-04T04:47:14.269 INFO:journalctl@ceph.mon.b.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-04T04:47:14.269 INFO:journalctl@ceph.mon.b.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-04T04:47:14.269 INFO:journalctl@ceph.mon.b.smithi098.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-04T04:47:14.269 INFO:journalctl@ceph.mon.b.smithi098.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-04T04:47:14.269 INFO:journalctl@ceph.mon.b.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-04T04:47:14.269 INFO:journalctl@ceph.mon.b.smithi098.stdout: Block cache BinnedLRUCache@0x559a454271f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-06-04T04:47:14.269 INFO:journalctl@ceph.mon.b.smithi098.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-06-04T04:47:14.269 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-06-04T04:47:14.269 INFO:journalctl@ceph.mon.b.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-04T04:47:14.269 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-06-04T04:47:14.269 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: using public_addr v2:172.21.15.98:0/0 -> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-04T04:47:14.269 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: starting mon.b rank -1 at public addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] at bind addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:47:14.270 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.b@-1(???) e0 preinit fsid 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:47:14.270 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.b@-1(synchronizing).mds e1 new map 2024-06-04T04:47:14.270 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.b@-1(synchronizing).mds e1 print_map 2024-06-04T04:47:14.270 INFO:journalctl@ceph.mon.b.smithi098.stdout: e1 2024-06-04T04:47:14.270 INFO:journalctl@ceph.mon.b.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-04T04:47:14.270 INFO:journalctl@ceph.mon.b.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-06-04T04:47:14.270 INFO:journalctl@ceph.mon.b.smithi098.stdout: legacy client fscid: -1 2024-06-04T04:47:14.270 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-06-04T04:47:14.270 INFO:journalctl@ceph.mon.b.smithi098.stdout: No filesystems configured 2024-06-04T04:47:14.270 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-06-04T04:47:14.270 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-06-04T04:47:14.270 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-06-04T04:47:14.270 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-06-04T04:47:14.270 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-06-04T04:47:14.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-06-04T04:47:14.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-04T04:47:14.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-04T04:47:14.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-04T04:47:14.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-04T04:47:14.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi112", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:47:14.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:47:14.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:14.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:14.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: Updating smithi098:/etc/ceph/ceph.conf 2024-06-04T04:47:14.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:14.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: Deploying cephadm binary to smithi112 2024-06-04T04:47:14.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: Updating smithi098:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:47:14.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:47:14.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:14.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:14.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: Updating smithi098:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-04T04:47:14.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: Added host smithi112 2024-06-04T04:47:14.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.273 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:14.273 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:14.273 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:47:14.273 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.273 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:14.273 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3744553518' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-04T04:47:14.273 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3744553518' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-04T04:47:14.273 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: osdmap e4: 0 total, 0 up, 0 in 2024-06-04T04:47:14.273 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.273 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.273 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.273 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.273 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:47:14.273 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:14.273 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:14.274 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: Updating smithi112:/etc/ceph/ceph.conf 2024-06-04T04:47:14.274 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: Updating smithi112:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:47:14.274 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:47:14.274 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:14.274 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: Updating smithi112:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-04T04:47:14.274 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.274 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.274 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.274 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi070:172.21.15.70=a;smithi098:172.21.15.98=b;smithi112:172.21.15.112=c", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:47:14.274 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: Saving service mon spec with placement smithi070:172.21.15.70=a;smithi098:172.21.15.98=b;smithi112:172.21.15.112=c;count:3 2024-06-04T04:47:14.274 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.274 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:14.274 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:14.274 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:14.275 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.275 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-04T04:47:14.275 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:14.275 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: Deploying daemon mon.c on smithi112 2024-06-04T04:47:14.275 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:14.275 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:14.275 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='client.? 172.21.15.112:0/719642409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-04T04:47:14.275 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:14.275 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:14.275 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:47:14.275 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:14.275 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.a calling monitor election 2024-06-04T04:47:14.275 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:14.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:14.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:14.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.c calling monitor election 2024-06-04T04:47:14.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:14.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:14.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:14.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:14.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-04T04:47:14.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: monmap epoch 2 2024-06-04T04:47:14.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: fsid 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:47:14.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: last_changed 2024-06-04T04:47:04.895907+0000 2024-06-04T04:47:14.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: created 2024-06-04T04:45:09.662402+0000 2024-06-04T04:47:14.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: min_mon_release 19 (squid) 2024-06-04T04:47:14.277 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: election_strategy: 1 2024-06-04T04:47:14.277 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-06-04T04:47:14.277 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: 1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.c 2024-06-04T04:47:14.277 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: fsmap 2024-06-04T04:47:14.277 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: osdmap e4: 0 total, 0 up, 0 in 2024-06-04T04:47:14.277 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mgrmap e13: x(active, since 52s) 2024-06-04T04:47:14.277 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: overall HEALTH_OK 2024-06-04T04:47:14.277 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.277 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.277 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:14.277 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-04T04:47:14.277 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:14.277 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:14.278 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: Deploying daemon mon.b on smithi098 2024-06-04T04:47:14.278 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:14.278 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: from='client.? 172.21.15.112:0/4231489375' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-04T04:47:14.278 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:14.278 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 ceph-mon[35569]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-06-04T04:47:14.278 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 bash[35550]: f688c5292625b8c15e5207a27b04de0676ba1a58702cb1f3472c29579d6ebff4 2024-06-04T04:47:14.278 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:13 smithi098 systemd[1]: Started Ceph mon.b for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:47:19.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.137+0000 7f875e833700 1 -- 172.21.15.112:0/89322000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87400035c0 msgr2=0x7f8740005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:19.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.137+0000 7f875e833700 1 --2- 172.21.15.112:0/89322000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87400035c0 0x7f8740005a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8758075810 tx=0x7f874800fe60 comp rx=0 tx=0).stop 2024-06-04T04:47:19.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.137+0000 7f875e833700 1 -- 172.21.15.112:0/89322000 shutdown_connections 2024-06-04T04:47:19.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.137+0000 7f875e833700 1 --2- 172.21.15.112:0/89322000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87400035c0 0x7f8740005a50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:19.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.137+0000 7f875e833700 1 -- 172.21.15.112:0/89322000 >> 172.21.15.112:0/89322000 conn(0x7f87580feb50 msgr2=0x7f8758100f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:47:19.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.137+0000 7f875e833700 1 -- 172.21.15.112:0/89322000 shutdown_connections 2024-06-04T04:47:19.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.137+0000 7f875e833700 1 -- 172.21.15.112:0/89322000 wait complete. 2024-06-04T04:47:19.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.138+0000 7f875e833700 1 Processor -- start 2024-06-04T04:47:19.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.138+0000 7f875e833700 1 -- start start 2024-06-04T04:47:19.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.138+0000 7f875e833700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87400035c0 0x7f875819f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:19.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.138+0000 7f875e833700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f875819f6f0 0x7f87581a3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:19.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.138+0000 7f875e833700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87581a3fc0 0x7f87581a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:19.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.138+0000 7f875e833700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8758111410 con 0x7f87581a3fc0 2024-06-04T04:47:19.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.138+0000 7f875e833700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8758111290 con 0x7f87400035c0 2024-06-04T04:47:19.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.138+0000 7f875e833700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8758111590 con 0x7f875819f6f0 2024-06-04T04:47:19.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.138+0000 7f87577fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f875819f6f0 0x7f87581a3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:19.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.138+0000 7f875cdd0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87581a3fc0 0x7f87581a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:19.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f875cdd0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87581a3fc0 0x7f87581a4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.112:35496/0 (socket says 172.21.15.112:35496) 2024-06-04T04:47:19.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f87577fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f875819f6f0 0x7f87581a3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54310/0 (socket says 172.21.15.112:54310) 2024-06-04T04:47:19.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f875cdd0700 1 -- 172.21.15.112:0/1830638352 learned_addr learned my addr 172.21.15.112:0/1830638352 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-06-04T04:47:19.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f87577fe700 1 -- 172.21.15.112:0/1830638352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f875819f6f0 msgr2=0x7f87581a3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-06-04T04:47:19.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f87577fe700 1 -- 172.21.15.112:0/1830638352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f875819f6f0 msgr2=0x7f87581a3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-04T04:47:19.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f87577fe700 1 --2- 172.21.15.112:0/1830638352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f875819f6f0 0x7f87581a3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-04T04:47:19.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f87577fe700 1 --2- 172.21.15.112:0/1830638352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f875819f6f0 0x7f87581a3a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-04T04:47:19.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f8757fff700 1 --2- 172.21.15.112:0/1830638352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87400035c0 0x7f875819f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:19.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f875cdd0700 1 -- 172.21.15.112:0/1830638352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f875819f6f0 msgr2=0x7f87581a3a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-04T04:47:19.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f875cdd0700 1 --2- 172.21.15.112:0/1830638352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f875819f6f0 0x7f87581a3a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:19.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f875cdd0700 1 -- 172.21.15.112:0/1830638352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87400035c0 msgr2=0x7f875819f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:19.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f875cdd0700 1 --2- 172.21.15.112:0/1830638352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87400035c0 0x7f875819f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:19.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f875cdd0700 1 -- 172.21.15.112:0/1830638352 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87581abd30 con 0x7f87581a3fc0 2024-06-04T04:47:19.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f875cdd0700 1 --2- 172.21.15.112:0/1830638352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87581a3fc0 0x7f87581a4470 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f875000ba50 tx=0x7f875000be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:47:19.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f87557fa700 1 -- 172.21.15.112:0/1830638352 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8750019070 con 0x7f87581a3fc0 2024-06-04T04:47:19.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f87557fa700 1 -- 172.21.15.112:0/1830638352 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8750014440 con 0x7f87581a3fc0 2024-06-04T04:47:19.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f875e833700 1 -- 172.21.15.112:0/1830638352 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87581abf60 con 0x7f87581a3fc0 2024-06-04T04:47:19.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f87557fa700 1 -- 172.21.15.112:0/1830638352 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87500075b0 con 0x7f87581a3fc0 2024-06-04T04:47:19.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.139+0000 7f875e833700 1 -- 172.21.15.112:0/1830638352 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87581ac440 con 0x7f87581a3fc0 2024-06-04T04:47:19.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.140+0000 7f87557fa700 1 -- 172.21.15.112:0/1830638352 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f875001c070 con 0x7f87581a3fc0 2024-06-04T04:47:19.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.140+0000 7f87557fa700 1 --2- 172.21.15.112:0/1830638352 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f874403e9a0 0x7f8744040e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:19.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.141+0000 7f87557fa700 1 -- 172.21.15.112:0/1830638352 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8750052cd0 con 0x7f87581a3fc0 2024-06-04T04:47:19.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.141+0000 7f875e833700 1 -- 172.21.15.112:0/1830638352 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8758064ba0 con 0x7f87581a3fc0 2024-06-04T04:47:19.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.141+0000 7f8757fff700 1 --2- 172.21.15.112:0/1830638352 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f874403e9a0 0x7f8744040e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:19.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.141+0000 7f8757fff700 1 --2- 172.21.15.112:0/1830638352 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f874403e9a0 0x7f8744040e60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8748020040 tx=0x7f8748001e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:47:19.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.146+0000 7f87557fa700 1 -- 172.21.15.112:0/1830638352 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8750022170 con 0x7f87581a3fc0 2024-06-04T04:47:19.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:19.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:47:19.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: mon.a calling monitor election 2024-06-04T04:47:19.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:19.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: mon.c calling monitor election 2024-06-04T04:47:19.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:19.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: mon.b calling monitor election 2024-06-04T04:47:19.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:19.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-04T04:47:19.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: monmap epoch 3 2024-06-04T04:47:19.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: fsid 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:47:19.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: last_changed 2024-06-04T04:47:13.896327+0000 2024-06-04T04:47:19.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: created 2024-06-04T04:45:09.662402+0000 2024-06-04T04:47:19.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: min_mon_release 19 (squid) 2024-06-04T04:47:19.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: election_strategy: 1 2024-06-04T04:47:19.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-06-04T04:47:19.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: 1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.c 2024-06-04T04:47:19.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: 2: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.b 2024-06-04T04:47:19.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: fsmap 2024-06-04T04:47:19.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: osdmap e4: 0 total, 0 up, 0 in 2024-06-04T04:47:19.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: mgrmap e13: x(active, since 61s) 2024-06-04T04:47:19.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: overall HEALTH_OK 2024-06-04T04:47:19.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:19.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:19.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:19.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:47:19.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: mon.a calling monitor election 2024-06-04T04:47:19.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:19.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: mon.c calling monitor election 2024-06-04T04:47:19.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:19.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: mon.b calling monitor election 2024-06-04T04:47:19.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:19.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-04T04:47:19.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: monmap epoch 3 2024-06-04T04:47:19.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: fsid 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:47:19.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: last_changed 2024-06-04T04:47:13.896327+0000 2024-06-04T04:47:19.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: created 2024-06-04T04:45:09.662402+0000 2024-06-04T04:47:19.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: min_mon_release 19 (squid) 2024-06-04T04:47:19.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: election_strategy: 1 2024-06-04T04:47:19.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-06-04T04:47:19.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: 1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.c 2024-06-04T04:47:19.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: 2: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.b 2024-06-04T04:47:19.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: fsmap 2024-06-04T04:47:19.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: osdmap e4: 0 total, 0 up, 0 in 2024-06-04T04:47:19.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: mgrmap e13: x(active, since 61s) 2024-06-04T04:47:19.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: overall HEALTH_OK 2024-06-04T04:47:19.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:19.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:19.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:19.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:47:19.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: mon.a calling monitor election 2024-06-04T04:47:19.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:47:19.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: mon.c calling monitor election 2024-06-04T04:47:19.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:19.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: mon.b calling monitor election 2024-06-04T04:47:19.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:19.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:19.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-04T04:47:19.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: monmap epoch 3 2024-06-04T04:47:19.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: fsid 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:47:19.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: last_changed 2024-06-04T04:47:13.896327+0000 2024-06-04T04:47:19.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: created 2024-06-04T04:45:09.662402+0000 2024-06-04T04:47:19.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: min_mon_release 19 (squid) 2024-06-04T04:47:19.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: election_strategy: 1 2024-06-04T04:47:19.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-06-04T04:47:19.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: 1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.c 2024-06-04T04:47:19.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: 2: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.b 2024-06-04T04:47:19.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: fsmap 2024-06-04T04:47:19.439 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: osdmap e4: 0 total, 0 up, 0 in 2024-06-04T04:47:19.439 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: mgrmap e13: x(active, since 61s) 2024-06-04T04:47:19.439 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: overall HEALTH_OK 2024-06-04T04:47:19.439 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:19.439 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:19.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.448+0000 7f875e833700 1 -- 172.21.15.112:0/1830638352 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f875802e560 con 0x7f87581a3fc0 2024-06-04T04:47:19.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.449+0000 7f87557fa700 1 -- 172.21.15.112:0/1830638352 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1281 (secure 0 0 0) 0x7f8750012c40 con 0x7f87581a3fc0 2024-06-04T04:47:19.451 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-06-04T04:47:19.451 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":3,"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","modified":"2024-06-04T04:47:13.896327Z","created":"2024-06-04T04:45:09.662402Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-06-04T04:47:19.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.451+0000 7f875e833700 1 -- 172.21.15.112:0/1830638352 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f874403e9a0 msgr2=0x7f8744040e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:19.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.451+0000 7f875e833700 1 --2- 172.21.15.112:0/1830638352 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f874403e9a0 0x7f8744040e60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8748020040 tx=0x7f8748001e50 comp rx=0 tx=0).stop 2024-06-04T04:47:19.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.451+0000 7f875e833700 1 -- 172.21.15.112:0/1830638352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87581a3fc0 msgr2=0x7f87581a4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:19.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.451+0000 7f875e833700 1 --2- 172.21.15.112:0/1830638352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87581a3fc0 0x7f87581a4470 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f875000ba50 tx=0x7f875000be10 comp rx=0 tx=0).stop 2024-06-04T04:47:19.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.451+0000 7f875e833700 1 -- 172.21.15.112:0/1830638352 shutdown_connections 2024-06-04T04:47:19.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.451+0000 7f875e833700 1 --2- 172.21.15.112:0/1830638352 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f874403e9a0 0x7f8744040e60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:19.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.451+0000 7f875e833700 1 --2- 172.21.15.112:0/1830638352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87581a3fc0 0x7f87581a4470 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:19.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.451+0000 7f875e833700 1 --2- 172.21.15.112:0/1830638352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87400035c0 0x7f875819f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:19.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.451+0000 7f875e833700 1 --2- 172.21.15.112:0/1830638352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f875819f6f0 0x7f87581a3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:19.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.451+0000 7f875e833700 1 -- 172.21.15.112:0/1830638352 >> 172.21.15.112:0/1830638352 conn(0x7f87580feb50 msgr2=0x7f8758106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:47:19.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.451+0000 7f875e833700 1 -- 172.21.15.112:0/1830638352 shutdown_connections 2024-06-04T04:47:19.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:19.451+0000 7f875e833700 1 -- 172.21.15.112:0/1830638352 wait complete. 2024-06-04T04:47:19.455 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 3 2024-06-04T04:47:20.064 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:20.064 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:20.064 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:20.064 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:19 smithi112 ceph-mon[36453]: from='client.? 172.21.15.112:0/1830638352' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-04T04:47:20.064 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:20.095 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-06-04T04:47:20.096 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph config generate-minimal-conf 2024-06-04T04:47:20.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:20.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:20.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:20.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:19 smithi098 ceph-mon[35569]: from='client.? 172.21.15.112:0/1830638352' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-04T04:47:20.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:20.342 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:47:20.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:20.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:20.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:20.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:19 smithi070 ceph-mon[32631]: from='client.? 172.21.15.112:0/1830638352' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-04T04:47:20.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:47:21.568 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:21 smithi070 ceph-mon[32631]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:21.568 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:21 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:21.568 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:21 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:21.568 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:21 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:21.568 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:21 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:21.569 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:21 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:21.569 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:21 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:21.569 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:21 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:21.569 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:21 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:21 smithi112 ceph-mon[36453]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:21 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:21 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:21 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:21 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:21.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:21 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:21.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:21 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:21.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:21 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:21.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:21 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:21 smithi098 ceph-mon[35569]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:21 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:21 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:21 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:21.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:21 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:21.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:21 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:21.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:21 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:21.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:21 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:21.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:21 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.064+0000 7f1e4de9b700 1 -- 172.21.15.70:0/3125758311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e481090d0 msgr2=0x7f1e481094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:22.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.064+0000 7f1e4de9b700 1 --2- 172.21.15.70:0/3125758311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e481090d0 0x7f1e481094b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009910 tx=0x7f1e38009c20 comp rx=0 tx=0).stop 2024-06-04T04:47:22.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.064+0000 7f1e4de9b700 1 -- 172.21.15.70:0/3125758311 shutdown_connections 2024-06-04T04:47:22.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.064+0000 7f1e4de9b700 1 --2- 172.21.15.70:0/3125758311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e481090d0 0x7f1e481094b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:22.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.064+0000 7f1e4de9b700 1 -- 172.21.15.70:0/3125758311 >> 172.21.15.70:0/3125758311 conn(0x7f1e480785c0 msgr2=0x7f1e480789d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:47:22.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.064+0000 7f1e4de9b700 1 -- 172.21.15.70:0/3125758311 shutdown_connections 2024-06-04T04:47:22.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.064+0000 7f1e4de9b700 1 -- 172.21.15.70:0/3125758311 wait complete. 2024-06-04T04:47:22.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.065+0000 7f1e4de9b700 1 Processor -- start 2024-06-04T04:47:22.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.065+0000 7f1e4de9b700 1 -- start start 2024-06-04T04:47:22.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.065+0000 7f1e4de9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e481090d0 0x7f1e4819e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:22.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.065+0000 7f1e4de9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e4819ee40 0x7f1e48198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:22.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.065+0000 7f1e4de9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e48199000 0x7f1e481994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:22.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.066+0000 7f1e4de9b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4810d3e0 con 0x7f1e4819ee40 2024-06-04T04:47:22.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.066+0000 7f1e4de9b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4810d260 con 0x7f1e481090d0 2024-06-04T04:47:22.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.066+0000 7f1e4de9b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4810d560 con 0x7f1e48199000 2024-06-04T04:47:22.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.066+0000 7f1e46ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e4819ee40 0x7f1e48198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:22.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.066+0000 7f1e46ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e4819ee40 0x7f1e48198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60778/0 (socket says 172.21.15.70:60778) 2024-06-04T04:47:22.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.066+0000 7f1e46ffd700 1 -- 172.21.15.70:0/2953432825 learned_addr learned my addr 172.21.15.70:0/2953432825 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:47:22.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.066+0000 7f1e477fe700 1 --2- 172.21.15.70:0/2953432825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e481090d0 0x7f1e4819e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:22.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.066+0000 7f1e47fff700 1 --2- 172.21.15.70:0/2953432825 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e48199000 0x7f1e481994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:22.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.066+0000 7f1e46ffd700 1 -- 172.21.15.70:0/2953432825 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e48199000 msgr2=0x7f1e481994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:22.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.066+0000 7f1e46ffd700 1 --2- 172.21.15.70:0/2953432825 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e48199000 0x7f1e481994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:22.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.066+0000 7f1e46ffd700 1 -- 172.21.15.70:0/2953432825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e481090d0 msgr2=0x7f1e4819e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:22.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.066+0000 7f1e46ffd700 1 --2- 172.21.15.70:0/2953432825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e481090d0 0x7f1e4819e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:22.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.066+0000 7f1e46ffd700 1 -- 172.21.15.70:0/2953432825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e481a57d0 con 0x7f1e4819ee40 2024-06-04T04:47:22.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.066+0000 7f1e46ffd700 1 --2- 172.21.15.70:0/2953432825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e4819ee40 0x7f1e48198ac0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c00b780 tx=0x7f1e3c00ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:47:22.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.067+0000 7f1e44ff9700 1 -- 172.21.15.70:0/2953432825 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c004230 con 0x7f1e4819ee40 2024-06-04T04:47:22.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.067+0000 7f1e44ff9700 1 -- 172.21.15.70:0/2953432825 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e3c0043d0 con 0x7f1e4819ee40 2024-06-04T04:47:22.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.067+0000 7f1e44ff9700 1 -- 172.21.15.70:0/2953432825 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c011a60 con 0x7f1e4819ee40 2024-06-04T04:47:22.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.067+0000 7f1e4de9b700 1 -- 172.21.15.70:0/2953432825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e481a5a00 con 0x7f1e4819ee40 2024-06-04T04:47:22.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.067+0000 7f1e4de9b700 1 -- 172.21.15.70:0/2953432825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e481a5ef0 con 0x7f1e4819ee40 2024-06-04T04:47:22.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.068+0000 7f1e4de9b700 1 -- 172.21.15.70:0/2953432825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e48064ba0 con 0x7f1e4819ee40 2024-06-04T04:47:22.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.068+0000 7f1e44ff9700 1 -- 172.21.15.70:0/2953432825 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f1e3c011c00 con 0x7f1e4819ee40 2024-06-04T04:47:22.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.068+0000 7f1e44ff9700 1 --2- 172.21.15.70:0/2953432825 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1e3003e9a0 0x7f1e30040e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:22.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.068+0000 7f1e44ff9700 1 -- 172.21.15.70:0/2953432825 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1e3c0523d0 con 0x7f1e4819ee40 2024-06-04T04:47:22.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.068+0000 7f1e477fe700 1 --2- 172.21.15.70:0/2953432825 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1e3003e9a0 0x7f1e30040e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:22.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.069+0000 7f1e477fe700 1 --2- 172.21.15.70:0/2953432825 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1e3003e9a0 0x7f1e30040e60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e38004690 tx=0x7f1e3801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:47:22.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.074+0000 7f1e44ff9700 1 -- 172.21.15.70:0/2953432825 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1e3c020740 con 0x7f1e4819ee40 2024-06-04T04:47:22.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.296+0000 7f1e4de9b700 1 -- 172.21.15.70:0/2953432825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f1e481a61b0 con 0x7f1e4819ee40 2024-06-04T04:47:22.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.296+0000 7f1e44ff9700 1 -- 172.21.15.70:0/2953432825 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f1e3c017d50 con 0x7f1e4819ee40 2024-06-04T04:47:22.300 INFO:teuthology.orchestra.run.smithi070.stdout:# minimal ceph.conf for 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:47:22.300 INFO:teuthology.orchestra.run.smithi070.stdout:[global] 2024-06-04T04:47:22.301 INFO:teuthology.orchestra.run.smithi070.stdout: fsid = 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:47:22.301 INFO:teuthology.orchestra.run.smithi070.stdout: mon_host = [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-06-04T04:47:22.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.299+0000 7f1e4de9b700 1 -- 172.21.15.70:0/2953432825 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1e3003e9a0 msgr2=0x7f1e30040e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:22.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.299+0000 7f1e4de9b700 1 --2- 172.21.15.70:0/2953432825 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1e3003e9a0 0x7f1e30040e60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e38004690 tx=0x7f1e3801a040 comp rx=0 tx=0).stop 2024-06-04T04:47:22.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.299+0000 7f1e4de9b700 1 -- 172.21.15.70:0/2953432825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e4819ee40 msgr2=0x7f1e48198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:22.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.299+0000 7f1e4de9b700 1 --2- 172.21.15.70:0/2953432825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e4819ee40 0x7f1e48198ac0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c00b780 tx=0x7f1e3c00ba90 comp rx=0 tx=0).stop 2024-06-04T04:47:22.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.299+0000 7f1e4de9b700 1 -- 172.21.15.70:0/2953432825 shutdown_connections 2024-06-04T04:47:22.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.299+0000 7f1e4de9b700 1 --2- 172.21.15.70:0/2953432825 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1e3003e9a0 0x7f1e30040e60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:22.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.299+0000 7f1e4de9b700 1 --2- 172.21.15.70:0/2953432825 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e48199000 0x7f1e481994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:22.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.299+0000 7f1e4de9b700 1 --2- 172.21.15.70:0/2953432825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e481090d0 0x7f1e4819e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:22.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.299+0000 7f1e4de9b700 1 --2- 172.21.15.70:0/2953432825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e4819ee40 0x7f1e48198ac0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:22.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.299+0000 7f1e4de9b700 1 -- 172.21.15.70:0/2953432825 >> 172.21.15.70:0/2953432825 conn(0x7f1e480785c0 msgr2=0x7f1e48104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:47:22.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.299+0000 7f1e4de9b700 1 -- 172.21.15.70:0/2953432825 shutdown_connections 2024-06-04T04:47:22.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:22.299+0000 7f1e4de9b700 1 -- 172.21.15.70:0/2953432825 wait complete. 2024-06-04T04:47:22.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: Updating smithi070:/etc/ceph/ceph.conf 2024-06-04T04:47:22.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: Updating smithi098:/etc/ceph/ceph.conf 2024-06-04T04:47:22.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: Updating smithi112:/etc/ceph/ceph.conf 2024-06-04T04:47:22.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: Updating smithi098:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:47:22.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: Updating smithi112:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:47:22.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: Updating smithi070:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:47:22.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-04T04:47:22.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-04T04:47:22.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:22.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:22 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2953432825' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:22.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: Updating smithi070:/etc/ceph/ceph.conf 2024-06-04T04:47:22.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: Updating smithi098:/etc/ceph/ceph.conf 2024-06-04T04:47:22.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: Updating smithi112:/etc/ceph/ceph.conf 2024-06-04T04:47:22.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: Updating smithi098:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:47:22.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: Updating smithi112:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:47:22.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: Updating smithi070:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:47:22.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-04T04:47:22.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-04T04:47:22.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:22.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:22 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2953432825' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:22.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: Updating smithi070:/etc/ceph/ceph.conf 2024-06-04T04:47:22.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: Updating smithi098:/etc/ceph/ceph.conf 2024-06-04T04:47:22.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: Updating smithi112:/etc/ceph/ceph.conf 2024-06-04T04:47:22.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: Updating smithi098:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:47:22.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: Updating smithi112:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:47:22.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: Updating smithi070:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:47:22.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:22.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-04T04:47:22.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-04T04:47:22.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:22.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:22 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2953432825' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:22.996 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-06-04T04:47:22.997 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T04:47:22.997 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.conf 2024-06-04T04:47:23.023 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T04:47:23.024 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:47:23.089 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-04T04:47:23.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.conf 2024-06-04T04:47:23.117 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-04T04:47:23.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:47:23.183 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-06-04T04:47:23.183 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.conf 2024-06-04T04:47:23.215 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-06-04T04:47:23.215 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:47:23.285 INFO:tasks.cephadm:Adding mgr.x on smithi070 2024-06-04T04:47:23.285 INFO:tasks.cephadm:Adding mgr.y on smithi098 2024-06-04T04:47:23.285 INFO:tasks.cephadm:Adding mgr.z on smithi112 2024-06-04T04:47:23.285 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch apply mgr '3;smithi070=x;smithi098=y;smithi112=z' 2024-06-04T04:47:23.562 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.c/config 2024-06-04T04:47:23.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:23 smithi070 ceph-mon[32631]: Reconfiguring mon.a (unknown last config time)... 2024-06-04T04:47:23.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:23 smithi070 ceph-mon[32631]: Reconfiguring daemon mon.a on smithi070 2024-06-04T04:47:23.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:23 smithi070 ceph-mon[32631]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:23.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:23 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:23.683 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:23 smithi112 ceph-mon[36453]: Reconfiguring mon.a (unknown last config time)... 2024-06-04T04:47:23.684 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:23 smithi112 ceph-mon[36453]: Reconfiguring daemon mon.a on smithi070 2024-06-04T04:47:23.684 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:23 smithi112 ceph-mon[36453]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:23.684 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:23 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:23.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:23 smithi098 ceph-mon[35569]: Reconfiguring mon.a (unknown last config time)... 2024-06-04T04:47:23.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:23 smithi098 ceph-mon[35569]: Reconfiguring daemon mon.a on smithi070 2024-06-04T04:47:23.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:23 smithi098 ceph-mon[35569]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:23.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:23 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:24.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.979+0000 7f0144159700 1 -- 172.21.15.112:0/1230766127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f013c1030d0 msgr2=0x7f013c1034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:24.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.979+0000 7f0144159700 1 --2- 172.21.15.112:0/1230766127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f013c1030d0 0x7f013c1034f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0138009910 tx=0x7f0138009c20 comp rx=0 tx=0).stop 2024-06-04T04:47:24.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.979+0000 7f0144159700 1 -- 172.21.15.112:0/1230766127 shutdown_connections 2024-06-04T04:47:24.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.980+0000 7f0144159700 1 --2- 172.21.15.112:0/1230766127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f013c1030d0 0x7f013c1034f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:24.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.980+0000 7f0144159700 1 -- 172.21.15.112:0/1230766127 >> 172.21.15.112:0/1230766127 conn(0x7f013c0fe670 msgr2=0x7f013c100ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:47:24.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.980+0000 7f0144159700 1 -- 172.21.15.112:0/1230766127 shutdown_connections 2024-06-04T04:47:24.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.980+0000 7f0144159700 1 -- 172.21.15.112:0/1230766127 wait complete. 2024-06-04T04:47:24.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.980+0000 7f0144159700 1 Processor -- start 2024-06-04T04:47:24.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.980+0000 7f0144159700 1 -- start start 2024-06-04T04:47:24.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.980+0000 7f0144159700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f013c1030d0 0x7f013c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:24.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.980+0000 7f0144159700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f013c198c90 0x7f013c19dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:24.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.980+0000 7f0144159700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f013c199110 0x7f013c1a0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:24.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.980+0000 7f0144159700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f013c108e70 con 0x7f013c198c90 2024-06-04T04:47:24.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.980+0000 7f0144159700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f013c108cf0 con 0x7f013c199110 2024-06-04T04:47:24.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.980+0000 7f0144159700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f013c108ff0 con 0x7f013c1030d0 2024-06-04T04:47:24.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.981+0000 7f0141ef5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f013c1030d0 0x7f013c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:24.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.981+0000 7f0141ef5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f013c1030d0 0x7f013c198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:34176/0 (socket says 172.21.15.112:34176) 2024-06-04T04:47:24.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.981+0000 7f0141ef5700 1 -- 172.21.15.112:0/3445612510 learned_addr learned my addr 172.21.15.112:0/3445612510 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-06-04T04:47:24.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.981+0000 7f01416f4700 1 --2- 172.21.15.112:0/3445612510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f013c198c90 0x7f013c19dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:24.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.981+0000 7f01426f6700 1 --2- 172.21.15.112:0/3445612510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f013c199110 0x7f013c1a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:24.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.981+0000 7f0141ef5700 1 -- 172.21.15.112:0/3445612510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f013c199110 msgr2=0x7f013c1a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:24.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.981+0000 7f0141ef5700 1 --2- 172.21.15.112:0/3445612510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f013c199110 0x7f013c1a0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:24.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.981+0000 7f0141ef5700 1 -- 172.21.15.112:0/3445612510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f013c198c90 msgr2=0x7f013c19dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:24.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.981+0000 7f0141ef5700 1 --2- 172.21.15.112:0/3445612510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f013c198c90 0x7f013c19dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:24.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.981+0000 7f0141ef5700 1 -- 172.21.15.112:0/3445612510 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013c1a0b60 con 0x7f013c1030d0 2024-06-04T04:47:24.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.981+0000 7f01416f4700 1 --2- 172.21.15.112:0/3445612510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f013c198c90 0x7f013c19dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:47:24.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.981+0000 7f0141ef5700 1 --2- 172.21.15.112:0/3445612510 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f013c1030d0 0x7f013c198750 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0138005000 tx=0x7f0138005030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:47:24.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.981+0000 7f0132ffd700 1 -- 172.21.15.112:0/3445612510 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138027070 con 0x7f013c1030d0 2024-06-04T04:47:24.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.981+0000 7f0132ffd700 1 -- 172.21.15.112:0/3445612510 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f013800bdf0 con 0x7f013c1030d0 2024-06-04T04:47:24.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.981+0000 7f0144159700 1 -- 172.21.15.112:0/3445612510 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f013c1a0df0 con 0x7f013c1030d0 2024-06-04T04:47:24.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.981+0000 7f0132ffd700 1 -- 172.21.15.112:0/3445612510 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138022cf0 con 0x7f013c1030d0 2024-06-04T04:47:24.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.981+0000 7f0144159700 1 -- 172.21.15.112:0/3445612510 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f013c1a1290 con 0x7f013c1030d0 2024-06-04T04:47:24.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.982+0000 7f0132ffd700 1 -- 172.21.15.112:0/3445612510 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f013802a070 con 0x7f013c1030d0 2024-06-04T04:47:24.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.982+0000 7f0144159700 1 -- 172.21.15.112:0/3445612510 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f013c064ba0 con 0x7f013c1030d0 2024-06-04T04:47:24.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.982+0000 7f0132ffd700 1 --2- 172.21.15.112:0/3445612510 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f012803e9a0 0x7f0128040e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:24.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.982+0000 7f0132ffd700 1 -- 172.21.15.112:0/3445612510 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f01380539b0 con 0x7f013c1030d0 2024-06-04T04:47:24.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.983+0000 7f01416f4700 1 --2- 172.21.15.112:0/3445612510 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f012803e9a0 0x7f0128040e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:24.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.983+0000 7f01416f4700 1 --2- 172.21.15.112:0/3445612510 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f012803e9a0 0x7f0128040e60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f012c009ec0 tx=0x7f012c009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:47:24.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:24.988+0000 7f0132ffd700 1 -- 172.21.15.112:0/3445612510 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0138013070 con 0x7f013c1030d0 2024-06-04T04:47:25.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:24 smithi112 ceph-mon[36453]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:25.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:24 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:25.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:24 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:25.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:24 smithi112 ceph-mon[36453]: Reconfiguring mon.b (monmap changed)... 2024-06-04T04:47:25.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:24 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-04T04:47:25.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:24 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-04T04:47:25.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:24 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:25.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:24 smithi112 ceph-mon[36453]: Reconfiguring daemon mon.b on smithi098 2024-06-04T04:47:25.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:25.200+0000 7f0144159700 1 -- 172.21.15.112:0/3445612510 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi070=x;smithi098=y;smithi112=z", "target": ["mon-mgr", ""]}) -- 0x7f013c100680 con 0x7f012803e9a0 2024-06-04T04:47:25.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:25.207+0000 7f0132ffd700 1 -- 172.21.15.112:0/3445612510 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f013c100680 con 0x7f012803e9a0 2024-06-04T04:47:25.210 INFO:teuthology.orchestra.run.smithi112.stdout:Scheduled mgr update... 2024-06-04T04:47:25.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:25.210+0000 7f0144159700 1 -- 172.21.15.112:0/3445612510 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f012803e9a0 msgr2=0x7f0128040e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:25.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:25.210+0000 7f0144159700 1 --2- 172.21.15.112:0/3445612510 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f012803e9a0 0x7f0128040e60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f012c009ec0 tx=0x7f012c009210 comp rx=0 tx=0).stop 2024-06-04T04:47:25.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:25.210+0000 7f0144159700 1 -- 172.21.15.112:0/3445612510 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f013c1030d0 msgr2=0x7f013c198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:25.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:25.210+0000 7f0144159700 1 --2- 172.21.15.112:0/3445612510 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f013c1030d0 0x7f013c198750 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0138005000 tx=0x7f0138005030 comp rx=0 tx=0).stop 2024-06-04T04:47:25.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:25.210+0000 7f0144159700 1 -- 172.21.15.112:0/3445612510 shutdown_connections 2024-06-04T04:47:25.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:25.210+0000 7f0144159700 1 --2- 172.21.15.112:0/3445612510 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f012803e9a0 0x7f0128040e60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:25.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:25.210+0000 7f0144159700 1 --2- 172.21.15.112:0/3445612510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f013c199110 0x7f013c1a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:25.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:25.210+0000 7f0144159700 1 --2- 172.21.15.112:0/3445612510 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f013c1030d0 0x7f013c198750 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:25.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:25.210+0000 7f0144159700 1 --2- 172.21.15.112:0/3445612510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f013c198c90 0x7f013c19dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:25.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:25.210+0000 7f0144159700 1 -- 172.21.15.112:0/3445612510 >> 172.21.15.112:0/3445612510 conn(0x7f013c0fe670 msgr2=0x7f013c0fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:47:25.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:25.210+0000 7f0144159700 1 -- 172.21.15.112:0/3445612510 shutdown_connections 2024-06-04T04:47:25.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:47:25.210+0000 7f0144159700 1 -- 172.21.15.112:0/3445612510 wait complete. 2024-06-04T04:47:25.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:24 smithi098 ceph-mon[35569]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:25.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:24 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:25.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:24 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:25.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:24 smithi098 ceph-mon[35569]: Reconfiguring mon.b (monmap changed)... 2024-06-04T04:47:25.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:24 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-04T04:47:25.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:24 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-04T04:47:25.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:24 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:25.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:24 smithi098 ceph-mon[35569]: Reconfiguring daemon mon.b on smithi098 2024-06-04T04:47:25.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:24 smithi070 ceph-mon[32631]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:25.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:24 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:25.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:24 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:25.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:24 smithi070 ceph-mon[32631]: Reconfiguring mon.b (monmap changed)... 2024-06-04T04:47:25.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:24 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-04T04:47:25.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:24 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-04T04:47:25.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:24 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:25.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:24 smithi070 ceph-mon[32631]: Reconfiguring daemon mon.b on smithi098 2024-06-04T04:47:25.809 DEBUG:teuthology.orchestra.run.smithi098:mgr.y> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mgr.y.service 2024-06-04T04:47:25.812 DEBUG:teuthology.orchestra.run.smithi112:mgr.z> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mgr.z.service 2024-06-04T04:47:25.815 INFO:tasks.cephadm:Deploying OSDs... 2024-06-04T04:47:25.815 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T04:47:25.815 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/scratch_devs of=/dev/stdout 2024-06-04T04:47:25.835 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-04T04:47:25.835 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_1 2024-06-04T04:47:25.895 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-04T04:47:25.895 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-04T04:47:25.895 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-06-04T04:47:25.895 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-04T04:47:25.895 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-06-04T04:47:25.895 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-06-04 04:46:29.823502643 +0000 2024-06-04T04:47:25.895 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-06-04 04:27:27.506443091 +0000 2024-06-04T04:47:25.895 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-06-04 04:27:27.506443091 +0000 2024-06-04T04:47:25.895 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-06-04 04:27:27.506443091 +0000 2024-06-04T04:47:25.896 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-04T04:47:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-06-04T04:47:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-06-04T04:47:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000105645 s, 4.8 MB/s 2024-06-04T04:47:25.964 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-04T04:47:26.023 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_2 2024-06-04T04:47:26.080 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-04T04:47:26.080 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-04T04:47:26.081 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-06-04T04:47:26.081 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-04T04:47:26.081 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-06-04T04:47:26.081 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-06-04 04:46:29.853501762 +0000 2024-06-04T04:47:26.081 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-06-04 04:27:27.838433290 +0000 2024-06-04T04:47:26.081 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-06-04 04:27:27.838433290 +0000 2024-06-04T04:47:26.081 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-06-04 04:27:27.838433290 +0000 2024-06-04T04:47:26.081 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-04T04:47:26.146 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-06-04T04:47:26.146 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-06-04T04:47:26.146 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000123018 s, 4.2 MB/s 2024-06-04T04:47:26.147 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-04T04:47:26.205 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_3 2024-06-04T04:47:26.263 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-04T04:47:26.263 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-04T04:47:26.263 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-06-04T04:47:26.263 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-04T04:47:26.263 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-06-04T04:47:26.263 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-06-04 04:46:29.881500939 +0000 2024-06-04T04:47:26.263 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-06-04 04:27:28.156423903 +0000 2024-06-04T04:47:26.263 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-06-04 04:27:28.156423903 +0000 2024-06-04T04:47:26.263 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-06-04 04:27:28.156423903 +0000 2024-06-04T04:47:26.264 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-04T04:47:26.326 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:26 smithi070 ceph-mon[32631]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi070=x;smithi098=y;smithi112=z", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:47:26.326 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:26 smithi070 ceph-mon[32631]: Saving service mgr spec with placement smithi070=x;smithi098=y;smithi112=z;count:3 2024-06-04T04:47:26.326 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:26.326 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:26.326 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:26.326 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:26 smithi070 ceph-mon[32631]: Reconfiguring mon.c (monmap changed)... 2024-06-04T04:47:26.326 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-04T04:47:26.326 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-04T04:47:26.326 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:26.327 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:26 smithi070 ceph-mon[32631]: Reconfiguring daemon mon.c on smithi112 2024-06-04T04:47:26.328 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-06-04T04:47:26.328 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-06-04T04:47:26.328 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000114278 s, 4.5 MB/s 2024-06-04T04:47:26.329 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-04T04:47:26.387 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_4 2024-06-04T04:47:26.393 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:26 smithi112 ceph-mon[36453]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi070=x;smithi098=y;smithi112=z", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:47:26.393 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:26 smithi112 ceph-mon[36453]: Saving service mgr spec with placement smithi070=x;smithi098=y;smithi112=z;count:3 2024-06-04T04:47:26.393 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:26.393 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:26.393 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:26.394 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:26 smithi112 ceph-mon[36453]: Reconfiguring mon.c (monmap changed)... 2024-06-04T04:47:26.394 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-04T04:47:26.394 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-04T04:47:26.394 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:26.394 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:26 smithi112 ceph-mon[36453]: Reconfiguring daemon mon.c on smithi112 2024-06-04T04:47:26.443 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-04T04:47:26.443 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-04T04:47:26.443 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-06-04T04:47:26.443 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-04T04:47:26.443 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-06-04T04:47:26.443 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-06-04 04:46:29.916499910 +0000 2024-06-04T04:47:26.443 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-06-04 04:27:28.484414220 +0000 2024-06-04T04:47:26.443 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-06-04 04:27:28.484414220 +0000 2024-06-04T04:47:26.444 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-06-04 04:27:28.484414220 +0000 2024-06-04T04:47:26.444 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-04T04:47:26.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:26 smithi098 ceph-mon[35569]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi070=x;smithi098=y;smithi112=z", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:47:26.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:26 smithi098 ceph-mon[35569]: Saving service mgr spec with placement smithi070=x;smithi098=y;smithi112=z;count:3 2024-06-04T04:47:26.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:26.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:26.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:26.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:26 smithi098 ceph-mon[35569]: Reconfiguring mon.c (monmap changed)... 2024-06-04T04:47:26.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-04T04:47:26.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-04T04:47:26.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:26.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:26 smithi098 ceph-mon[35569]: Reconfiguring daemon mon.c on smithi112 2024-06-04T04:47:26.507 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-06-04T04:47:26.507 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-06-04T04:47:26.507 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000132997 s, 3.8 MB/s 2024-06-04T04:47:26.508 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-04T04:47:26.566 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-04T04:47:26.566 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-06-04T04:47:26.585 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-04T04:47:26.586 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-06-04T04:47:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-04T04:47:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-04T04:47:26.645 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-06-04T04:47:26.645 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-04T04:47:26.645 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-06-04T04:47:26.645 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-06-04 04:47:13.343440843 +0000 2024-06-04T04:47:26.645 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-06-04 04:29:12.485637247 +0000 2024-06-04T04:47:26.645 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-06-04 04:29:12.485637247 +0000 2024-06-04T04:47:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-06-04 04:29:12.485637247 +0000 2024-06-04T04:47:26.646 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-04T04:47:26.713 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-06-04T04:47:26.713 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-06-04T04:47:26.713 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.00026372 s, 1.9 MB/s 2024-06-04T04:47:26.714 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-04T04:47:26.774 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-06-04T04:47:26.834 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-04T04:47:26.834 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-04T04:47:26.835 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-06-04T04:47:26.835 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-04T04:47:26.835 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-06-04T04:47:26.835 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-06-04 04:47:13.344440829 +0000 2024-06-04T04:47:26.835 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-06-04 04:29:12.831632903 +0000 2024-06-04T04:47:26.835 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-06-04 04:29:12.831632903 +0000 2024-06-04T04:47:26.836 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-06-04 04:29:12.831632903 +0000 2024-06-04T04:47:26.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-04T04:47:26.901 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-06-04T04:47:26.901 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-06-04T04:47:26.901 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000222212 s, 2.3 MB/s 2024-06-04T04:47:26.903 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-04T04:47:26.962 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-06-04T04:47:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-04T04:47:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-04T04:47:27.019 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-06-04T04:47:27.019 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-04T04:47:27.019 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-06-04T04:47:27.019 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-06-04 04:47:13.344440829 +0000 2024-06-04T04:47:27.019 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-06-04 04:29:13.155628835 +0000 2024-06-04T04:47:27.019 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-06-04 04:29:13.155628835 +0000 2024-06-04T04:47:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-06-04 04:29:13.155628835 +0000 2024-06-04T04:47:27.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-04T04:47:27.084 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-06-04T04:47:27.084 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-06-04T04:47:27.085 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000280694 s, 1.8 MB/s 2024-06-04T04:47:27.086 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-04T04:47:27.143 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-06-04T04:47:27.199 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-04T04:47:27.199 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-04T04:47:27.199 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-06-04T04:47:27.199 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-04T04:47:27.199 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-06-04T04:47:27.199 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-06-04 04:47:13.344440829 +0000 2024-06-04T04:47:27.199 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-06-04 04:29:13.493624591 +0000 2024-06-04T04:47:27.199 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-06-04 04:29:13.493624591 +0000 2024-06-04T04:47:27.199 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-06-04 04:29:13.493624591 +0000 2024-06-04T04:47:27.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-04T04:47:27.263 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-06-04T04:47:27.263 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-06-04T04:47:27.263 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000193936 s, 2.6 MB/s 2024-06-04T04:47:27.264 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-04T04:47:27.321 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-06-04T04:47:27.321 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/scratch_devs of=/dev/stdout 2024-06-04T04:47:27.340 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-04T04:47:27.341 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_1 2024-06-04T04:47:27.399 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-04T04:47:27.399 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-04T04:47:27.399 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 672 Links: 1 2024-06-04T04:47:27.400 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-04T04:47:27.400 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-06-04T04:47:27.400 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-06-04 04:47:02.934202840 +0000 2024-06-04T04:47:27.400 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-06-04 04:27:27.523498544 +0000 2024-06-04T04:47:27.400 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-06-04 04:27:27.523498544 +0000 2024-06-04T04:47:27.400 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-06-04 04:27:27.523498544 +0000 2024-06-04T04:47:27.400 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-04T04:47:27.468 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-06-04T04:47:27.468 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-06-04T04:47:27.468 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000227168 s, 2.3 MB/s 2024-06-04T04:47:27.470 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-04T04:47:27.528 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_2 2024-06-04T04:47:27.585 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-04T04:47:27.585 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-04T04:47:27.586 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-06-04T04:47:27.586 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-04T04:47:27.586 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-06-04T04:47:27.586 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-06-04 04:47:02.934202840 +0000 2024-06-04T04:47:27.586 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-06-04 04:27:27.849489688 +0000 2024-06-04T04:47:27.586 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-06-04 04:27:27.849489688 +0000 2024-06-04T04:47:27.586 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-06-04 04:27:27.849489688 +0000 2024-06-04T04:47:27.586 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-04T04:47:27.651 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-06-04T04:47:27.651 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-06-04T04:47:27.651 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000224612 s, 2.3 MB/s 2024-06-04T04:47:27.652 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-04T04:47:27.710 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_3 2024-06-04T04:47:27.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:27 smithi098 ceph-mon[35569]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:27.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:27 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:27.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:27 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:27.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:27 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:27.765 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-04T04:47:27.765 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-04T04:47:27.765 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 710 Links: 1 2024-06-04T04:47:27.765 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-04T04:47:27.765 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-06-04T04:47:27.765 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-06-04 04:47:02.935202813 +0000 2024-06-04T04:47:27.765 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-06-04 04:27:28.161481365 +0000 2024-06-04T04:47:27.765 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-06-04 04:27:28.161481365 +0000 2024-06-04T04:47:27.765 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-06-04 04:27:28.161481365 +0000 2024-06-04T04:47:27.766 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-04T04:47:27.826 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:27 smithi112 ceph-mon[36453]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:27.826 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:27 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:27.826 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:27 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:27.826 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:27 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:27.829 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-06-04T04:47:27.829 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-06-04T04:47:27.829 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000171198 s, 3.0 MB/s 2024-06-04T04:47:27.830 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-04T04:47:27.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:27 smithi070 ceph-mon[32631]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:27.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:27 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:27.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:27 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:27.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:27 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:27.886 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_4 2024-06-04T04:47:27.942 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-04T04:47:27.942 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-04T04:47:27.942 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 731 Links: 1 2024-06-04T04:47:27.942 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-04T04:47:27.942 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-06-04T04:47:27.942 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-06-04 04:47:02.935202813 +0000 2024-06-04T04:47:27.942 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-06-04 04:27:28.494472623 +0000 2024-06-04T04:47:27.942 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-06-04 04:27:28.494472623 +0000 2024-06-04T04:47:27.942 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-06-04 04:27:28.494472623 +0000 2024-06-04T04:47:27.943 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-04T04:47:28.007 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-06-04T04:47:28.008 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-06-04T04:47:28.008 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000233974 s, 2.2 MB/s 2024-06-04T04:47:28.009 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-04T04:47:28.067 INFO:tasks.cephadm:Deploying osd.0 on smithi070 with /dev/vg_nvme/lv_4... 2024-06-04T04:47:28.068 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-04T04:47:28.298 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:47:28.583 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:28 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:28.583 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:28 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:28.583 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:28 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:28.583 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:28 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-04T04:47:28.583 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:28 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-04T04:47:28.584 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:28 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-04T04:47:28.584 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:28 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:28.672 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:28 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:28.672 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:28 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:28.672 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:28 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:28.672 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:28 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-04T04:47:28.672 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:28 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-04T04:47:28.672 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:28 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-04T04:47:28.672 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:28 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:28.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:28 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:28.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:28 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:28.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:28 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:28.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:28 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-04T04:47:28.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:28 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-04T04:47:28.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:28 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-04T04:47:28.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:28 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:29.464 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:29 smithi112 ceph-mon[36453]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:29.464 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:29 smithi112 ceph-mon[36453]: Deploying daemon mgr.z on smithi112 2024-06-04T04:47:29.499 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:29 smithi070 ceph-mon[32631]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:29.499 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:29 smithi070 ceph-mon[32631]: Deploying daemon mgr.z on smithi112 2024-06-04T04:47:29.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:29 smithi098 ceph-mon[35569]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:29.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:29 smithi098 ceph-mon[35569]: Deploying daemon mgr.z on smithi112 2024-06-04T04:47:29.814 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:29 smithi112 systemd[1]: Starting Ceph mgr.z for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T04:47:30.064 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-04T04:47:30.065 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-04T04:47:30.065 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-06-04T04:47:30.065 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-06-04T04:47:30.065 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00848601 s, 1.2 GB/s 2024-06-04T04:47:30.065 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-06-04T04:47:30.065 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:29 smithi112 podman[38282]: 2024-06-04 04:47:29.812678622 +0000 UTC m=+0.090194100 container create 0b79d87cdfcaf8c83847990679ce64829fa7767a65a4f3963748534a4f387fd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, org.label-schema.build-date=20240521) 2024-06-04T04:47:30.066 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:29 smithi112 podman[38282]: 2024-06-04 04:47:29.73977485 +0000 UTC m=+0.017290305 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T04:47:30.066 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:30 smithi112 podman[38282]: 2024-06-04 04:47:30.006487088 +0000 UTC m=+0.284002523 container init 0b79d87cdfcaf8c83847990679ce64829fa7767a65a4f3963748534a4f387fd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=main-b7328fe, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-06-04T04:47:30.066 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:30 smithi112 podman[38282]: 2024-06-04 04:47:30.009105602 +0000 UTC m=+0.286621041 container start 0b79d87cdfcaf8c83847990679ce64829fa7767a65a4f3963748534a4f387fd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, ceph=True, RELEASE=main-b7328fe, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-06-04T04:47:30.066 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:30 smithi112 ceph-mgr[38301]: -- 172.21.15.112:0/2015952251 <== mon.1 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555e5734c5a0 con 0x555e57320800 2024-06-04T04:47:30.363 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:30 smithi112 bash[38282]: 0b79d87cdfcaf8c83847990679ce64829fa7767a65a4f3963748534a4f387fd2 2024-06-04T04:47:30.363 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:30 smithi112 systemd[1]: Started Ceph mgr.z for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:47:30.364 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:30 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:30.176+0000 7f5be6b00200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-04T04:47:30.364 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:30 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:30.361+0000 7f5be6b00200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-04T04:47:30.686 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:30 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:30.465+0000 7f5be6b00200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-04T04:47:30.687 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi070:vg_nvme/lv_4 2024-06-04T04:47:30.920 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:47:31.142 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:30 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:30.773+0000 7f5be6b00200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-04T04:47:31.142 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:30 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:30.858+0000 7f5be6b00200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-04T04:47:31.334 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:31 smithi070 ceph-mon[32631]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:31.335 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:31 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:31.335 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:31 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:31.335 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:31 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:31.335 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:31 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-04T04:47:31.335 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:31 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-04T04:47:31.335 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:31 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-04T04:47:31.335 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:31 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:31.335 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:31 smithi070 ceph-mon[32631]: Deploying daemon mgr.y on smithi098 2024-06-04T04:47:31.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:31 smithi112 ceph-mon[36453]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:31.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:31 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:31.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:31 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:31.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:31 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:31.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:31 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-04T04:47:31.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:31 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-04T04:47:31.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:31 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-04T04:47:31.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:31 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:31.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:31 smithi112 ceph-mon[36453]: Deploying daemon mgr.y on smithi098 2024-06-04T04:47:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:31 smithi098 ceph-mon[35569]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:31 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:31 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:31 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:31 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-04T04:47:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:31 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-04T04:47:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:31 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-04T04:47:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:31 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:31.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:31 smithi098 ceph-mon[35569]: Deploying daemon mgr.y on smithi098 2024-06-04T04:47:32.245 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:32 smithi098 systemd[1]: Starting Ceph mgr.y for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T04:47:32.390 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:32 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:32.051+0000 7f5be6b00200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-04T04:47:32.391 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:32 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:32.219+0000 7f5be6b00200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-04T04:47:32.391 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:32 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:32.388+0000 7f5be6b00200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-04T04:47:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.511+0000 7f0579f7f700 1 -- 172.21.15.70:0/1091304033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0574104040 msgr2=0x7f057410a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.511+0000 7f0579f7f700 1 --2- 172.21.15.70:0/1091304033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0574104040 0x7f057410a8f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f056c0099c0 tx=0x7f056c009cd0 comp rx=0 tx=0).stop 2024-06-04T04:47:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.511+0000 7f0579f7f700 1 -- 172.21.15.70:0/1091304033 shutdown_connections 2024-06-04T04:47:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.511+0000 7f0579f7f700 1 --2- 172.21.15.70:0/1091304033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0574104040 0x7f057410a8f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:32.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.511+0000 7f0579f7f700 1 --2- 172.21.15.70:0/1091304033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05741024a0 0x7f05741028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:32.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.511+0000 7f0579f7f700 1 --2- 172.21.15.70:0/1091304033 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05741036a0 0x7f0574103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:32.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.511+0000 7f0579f7f700 1 -- 172.21.15.70:0/1091304033 >> 172.21.15.70:0/1091304033 conn(0x7f05740fda60 msgr2=0x7f05740ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:47:32.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.511+0000 7f0579f7f700 1 -- 172.21.15.70:0/1091304033 shutdown_connections 2024-06-04T04:47:32.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.511+0000 7f0579f7f700 1 -- 172.21.15.70:0/1091304033 wait complete. 2024-06-04T04:47:32.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.511+0000 7f0579f7f700 1 Processor -- start 2024-06-04T04:47:32.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f0579f7f700 1 -- start start 2024-06-04T04:47:32.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f0579f7f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05741024a0 0x7f057406c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:32.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f0579f7f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05741036a0 0x7f057419e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:32.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f0579f7f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0574104040 0x7f05741a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:32.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f0579f7f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f057410ddf0 con 0x7f05741036a0 2024-06-04T04:47:32.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f0579f7f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f057410dc70 con 0x7f0574104040 2024-06-04T04:47:32.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f0579f7f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f057410df70 con 0x7f05741024a0 2024-06-04T04:47:32.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f0572ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05741036a0 0x7f057419e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:32.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f0572ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05741036a0 0x7f057419e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51226/0 (socket says 172.21.15.70:51226) 2024-06-04T04:47:32.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f0572ffd700 1 -- 172.21.15.70:0/1957755525 learned_addr learned my addr 172.21.15.70:0/1957755525 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:47:32.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f05737fe700 1 --2- 172.21.15.70:0/1957755525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05741024a0 0x7f057406c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:32.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f0573fff700 1 --2- 172.21.15.70:0/1957755525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0574104040 0x7f05741a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:32.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f0572ffd700 1 -- 172.21.15.70:0/1957755525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05741024a0 msgr2=0x7f057406c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:32.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f0572ffd700 1 --2- 172.21.15.70:0/1957755525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05741024a0 0x7f057406c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:32.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f0572ffd700 1 -- 172.21.15.70:0/1957755525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0574104040 msgr2=0x7f05741a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:32.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f0572ffd700 1 --2- 172.21.15.70:0/1957755525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0574104040 0x7f05741a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:32.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f0572ffd700 1 -- 172.21.15.70:0/1957755525 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05741a1200 con 0x7f05741036a0 2024-06-04T04:47:32.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.512+0000 7f05737fe700 1 --2- 172.21.15.70:0/1957755525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05741024a0 0x7f057406c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:47:32.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.513+0000 7f0572ffd700 1 --2- 172.21.15.70:0/1957755525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05741036a0 0x7f057419e2a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f056800e9c0 tx=0x7f056800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:47:32.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.513+0000 7f0570ff9700 1 -- 172.21.15.70:0/1957755525 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056800cbd0 con 0x7f05741036a0 2024-06-04T04:47:32.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.513+0000 7f0579f7f700 1 -- 172.21.15.70:0/1957755525 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05741a14f0 con 0x7f05741036a0 2024-06-04T04:47:32.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.513+0000 7f0570ff9700 1 -- 172.21.15.70:0/1957755525 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f056800ee60 con 0x7f05741036a0 2024-06-04T04:47:32.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.513+0000 7f0570ff9700 1 -- 172.21.15.70:0/1957755525 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05680054b0 con 0x7f05741036a0 2024-06-04T04:47:32.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.513+0000 7f0579f7f700 1 -- 172.21.15.70:0/1957755525 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05741a1a30 con 0x7f05741036a0 2024-06-04T04:47:32.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.514+0000 7f0579f7f700 1 -- 172.21.15.70:0/1957755525 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f057406a230 con 0x7f05741036a0 2024-06-04T04:47:32.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.514+0000 7f0570ff9700 1 -- 172.21.15.70:0/1957755525 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f0568021440 con 0x7f05741036a0 2024-06-04T04:47:32.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.514+0000 7f0570ff9700 1 --2- 172.21.15.70:0/1957755525 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f055c03e750 0x7f055c040c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:32.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.514+0000 7f0570ff9700 1 -- 172.21.15.70:0/1957755525 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f0568052c30 con 0x7f05741036a0 2024-06-04T04:47:32.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.514+0000 7f05737fe700 1 --2- 172.21.15.70:0/1957755525 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f055c03e750 0x7f055c040c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:32.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.514+0000 7f05737fe700 1 --2- 172.21.15.70:0/1957755525 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f055c03e750 0x7f055c040c10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0574103500 tx=0x7f05640091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:47:32.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.519+0000 7f0570ff9700 1 -- 172.21.15.70:0/1957755525 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0568016590 con 0x7f05741036a0 2024-06-04T04:47:32.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:32 smithi098 podman[36930]: 2024-06-04 04:47:32.19420794 +0000 UTC m=+0.016889048 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T04:47:32.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:32 smithi098 podman[36930]: 2024-06-04 04:47:32.303190185 +0000 UTC m=+0.125871303 container create 6d890c31030897bee872995330708a4ab7e5de8fad69dca0954ec6e89d6f28cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-b7328fe, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521) 2024-06-04T04:47:32.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:32 smithi098 podman[36930]: 2024-06-04 04:47:32.512975749 +0000 UTC m=+0.335656845 container init 6d890c31030897bee872995330708a4ab7e5de8fad69dca0954ec6e89d6f28cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.build-date=20240521, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-b7328fe, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-06-04T04:47:32.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:32 smithi098 podman[36930]: 2024-06-04 04:47:32.515778531 +0000 UTC m=+0.338459634 container start 6d890c31030897bee872995330708a4ab7e5de8fad69dca0954ec6e89d6f28cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-b7328fe, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-06-04T04:47:32.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:32 smithi098 ceph-mgr[36948]: -- 172.21.15.98:0/2981509357 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562fcbe0e5a0 con 0x562fcbde2800 2024-06-04T04:47:32.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:32 smithi098 bash[36930]: 6d890c31030897bee872995330708a4ab7e5de8fad69dca0954ec6e89d6f28cc 2024-06-04T04:47:32.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:32.740+0000 7f0579f7f700 1 -- 172.21.15.70:0/1957755525 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f0574063980 con 0x7f055c03e750 2024-06-04T04:47:32.860 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:32 smithi098 systemd[1]: Started Ceph mgr.y for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:47:32.860 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:32 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:32.668+0000 7fc7cf009200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-04T04:47:32.860 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:32 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:32.857+0000 7fc7cf009200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-04T04:47:32.936 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:32 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:32.681+0000 7f5be6b00200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-04T04:47:32.936 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:32 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:32.774+0000 7f5be6b00200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-04T04:47:33.144 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:32 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:32.964+0000 7fc7cf009200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-04T04:47:33.398 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:33 smithi070 ceph-mon[32631]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:33.398 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:33.398 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:33.398 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:33.399 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:33.399 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:33.399 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:47:33.399 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:47:33.399 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:33.399 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:33.436 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:32 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:32.982+0000 7f5be6b00200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-04T04:47:33.436 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:33 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:33.183+0000 7f5be6b00200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-04T04:47:33.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:33 smithi112 ceph-mon[36453]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:33.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:33 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:33.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:33 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:33.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:33 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:33.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:33 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:33.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:33 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:33.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:33 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:47:33.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:33 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:47:33.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:33 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:33.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:33 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:33.495 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:33 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:33.284+0000 7fc7cf009200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-04T04:47:33.495 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:33 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:33.382+0000 7fc7cf009200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-04T04:47:33.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:33 smithi098 ceph-mon[35569]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:33.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:33 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:33.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:33 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:33.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:33 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:33.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:33 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:33.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:33 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:33.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:33 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:47:33.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:33 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:47:33.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:33 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:33.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:33 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:34.069 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:33 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:33.824+0000 7f5be6b00200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-04T04:47:34.069 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:34 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:34.066+0000 7f5be6b00200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-06-04T04:47:34.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:34 smithi070 ceph-mon[32631]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:47:34.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:34.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:34.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:34 smithi112 ceph-mon[36453]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:47:34.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:34.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:34.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:34 smithi098 ceph-mon[35569]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:47:34.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:34.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:34.936 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:34 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:34.498+0000 7f5be6b00200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-04T04:47:34.936 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:34 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:34.593+0000 7f5be6b00200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-04T04:47:34.949 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:34 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:34.607+0000 7fc7cf009200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-04T04:47:34.949 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:34 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:34.777+0000 7fc7cf009200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-04T04:47:34.949 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:34 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:34.947+0000 7fc7cf009200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-04T04:47:35.237 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:35 smithi098 ceph-mon[35569]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:35.237 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:35 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:35.237 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:35 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:35.237 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:35 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:35.237 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:35 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:35.237 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:35 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:35.237 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:35 smithi098 ceph-mon[35569]: Reconfiguring mgr.x (unknown last config time)... 2024-06-04T04:47:35.237 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:35 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-04T04:47:35.237 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:35 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-04T04:47:35.237 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:35 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:35.237 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:35 smithi098 ceph-mon[35569]: Reconfiguring daemon mgr.x on smithi070 2024-06-04T04:47:35.315 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:34 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:34.961+0000 7f5be6b00200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-04T04:47:35.316 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:35 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:35.313+0000 7f5be6b00200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-04T04:47:35.316 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:35 smithi112 ceph-mon[36453]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:35.316 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:35 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:35.316 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:35 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:35.316 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:35 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:35.316 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:35 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:35.316 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:35 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:35.316 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:35 smithi112 ceph-mon[36453]: Reconfiguring mgr.x (unknown last config time)... 2024-06-04T04:47:35.316 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:35 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-04T04:47:35.316 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:35 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-04T04:47:35.316 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:35 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:35.317 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:35 smithi112 ceph-mon[36453]: Reconfiguring daemon mgr.x on smithi070 2024-06-04T04:47:35.495 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:35 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:35.234+0000 7fc7cf009200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-04T04:47:35.496 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:35 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:35.327+0000 7fc7cf009200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-04T04:47:35.606 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:35 smithi070 ceph-mon[32631]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:35.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:35 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:35.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:35 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:35.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:35 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:35.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:35 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:35.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:35 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:35.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:35 smithi070 ceph-mon[32631]: Reconfiguring mgr.x (unknown last config time)... 2024-06-04T04:47:35.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:35 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-04T04:47:35.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:35 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-04T04:47:35.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:35 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:35.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:35 smithi070 ceph-mon[32631]: Reconfiguring daemon mgr.x on smithi070 2024-06-04T04:47:35.646 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:35 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:35.405+0000 7f5be6b00200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-04T04:47:35.936 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:35 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:35.644+0000 7f5be6b00200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-04T04:47:35.936 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:35 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:35.777+0000 7f5be6b00200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-04T04:47:35.936 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:35 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:35.871+0000 7f5be6b00200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-04T04:47:35.995 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:35 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:35.529+0000 7fc7cf009200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-04T04:47:35.995 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:35 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:35.722+0000 7fc7cf009200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-04T04:47:36.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:36 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1846880323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff491ea2-1906-48e4-883e-bc78e778260e"}]: dispatch 2024-06-04T04:47:36.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:36 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1846880323' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff491ea2-1906-48e4-883e-bc78e778260e"}]': finished 2024-06-04T04:47:36.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:36 smithi112 ceph-mon[36453]: osdmap e5: 1 total, 0 up, 1 in 2024-06-04T04:47:36.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:36 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:36.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:36 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3420327660' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:47:36.436 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:36 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:36.195+0000 7f5be6b00200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-04T04:47:36.437 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:36 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:36.286+0000 7f5be6b00200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-04T04:47:36.495 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:36 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:36.323+0000 7fc7cf009200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-04T04:47:36.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:36 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1846880323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff491ea2-1906-48e4-883e-bc78e778260e"}]: dispatch 2024-06-04T04:47:36.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:36 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1846880323' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff491ea2-1906-48e4-883e-bc78e778260e"}]': finished 2024-06-04T04:47:36.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:36 smithi098 ceph-mon[35569]: osdmap e5: 1 total, 0 up, 1 in 2024-06-04T04:47:36.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:36 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:36.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:36 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3420327660' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:47:36.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:36 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1846880323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff491ea2-1906-48e4-883e-bc78e778260e"}]: dispatch 2024-06-04T04:47:36.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:36 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1846880323' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff491ea2-1906-48e4-883e-bc78e778260e"}]': finished 2024-06-04T04:47:36.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:36 smithi070 ceph-mon[32631]: osdmap e5: 1 total, 0 up, 1 in 2024-06-04T04:47:36.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:36 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:36.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:36 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3420327660' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:47:36.969 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:36 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:36.544+0000 7fc7cf009200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-06-04T04:47:37.186 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:36 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:36.882+0000 7f5be6b00200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-04T04:47:37.186 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:47:37 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:47:37.095+0000 7f5be6b00200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-04T04:47:37.245 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:36 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:36.967+0000 7fc7cf009200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-04T04:47:37.245 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:37 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:37.063+0000 7fc7cf009200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-04T04:47:37.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:37.462+0000 7f0570ff9700 1 -- 172.21.15.70:0/1957755525 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 14) ==== 99937+0+0 (secure 0 0 0) 0x7f05680219b0 con 0x7f05741036a0 2024-06-04T04:47:37.745 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:37 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:37.431+0000 7fc7cf009200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-04T04:47:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:37 smithi098 ceph-mon[35569]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:37.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:37 smithi098 ceph-mon[35569]: Standby manager daemon z started 2024-06-04T04:47:37.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:37 smithi098 ceph-mon[35569]: from='mgr.? 172.21.15.112:0/1220163678' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-04T04:47:37.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:37 smithi098 ceph-mon[35569]: from='mgr.? 172.21.15.112:0/1220163678' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-04T04:47:37.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:37 smithi098 ceph-mon[35569]: from='mgr.? 172.21.15.112:0/1220163678' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-04T04:47:37.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:37 smithi098 ceph-mon[35569]: from='mgr.? 172.21.15.112:0/1220163678' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-04T04:47:37.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:37 smithi070 ceph-mon[32631]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:37.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:37 smithi070 ceph-mon[32631]: Standby manager daemon z started 2024-06-04T04:47:37.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:37 smithi070 ceph-mon[32631]: from='mgr.? 172.21.15.112:0/1220163678' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-04T04:47:37.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:37 smithi070 ceph-mon[32631]: from='mgr.? 172.21.15.112:0/1220163678' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-04T04:47:37.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:37 smithi070 ceph-mon[32631]: from='mgr.? 172.21.15.112:0/1220163678' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-04T04:47:37.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:37 smithi070 ceph-mon[32631]: from='mgr.? 172.21.15.112:0/1220163678' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-04T04:47:37.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:37 smithi112 ceph-mon[36453]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:37.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:37 smithi112 ceph-mon[36453]: Standby manager daemon z started 2024-06-04T04:47:37.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:37 smithi112 ceph-mon[36453]: from='mgr.? 172.21.15.112:0/1220163678' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-04T04:47:37.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:37 smithi112 ceph-mon[36453]: from='mgr.? 172.21.15.112:0/1220163678' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-04T04:47:37.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:37 smithi112 ceph-mon[36453]: from='mgr.? 172.21.15.112:0/1220163678' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-04T04:47:37.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:37 smithi112 ceph-mon[36453]: from='mgr.? 172.21.15.112:0/1220163678' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-04T04:47:38.120 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:37 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:37.783+0000 7fc7cf009200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-04T04:47:38.120 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:37 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:37.874+0000 7fc7cf009200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-04T04:47:38.472 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:38 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:38.117+0000 7fc7cf009200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-04T04:47:38.472 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:38 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:38.249+0000 7fc7cf009200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-04T04:47:38.473 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:38 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:38.344+0000 7fc7cf009200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-04T04:47:38.745 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:38 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:38.667+0000 7fc7cf009200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-04T04:47:38.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:38 smithi098 ceph-mon[35569]: mgrmap e14: x(active, since 79s), standbys: z 2024-06-04T04:47:38.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:38 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-04T04:47:38.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:38 smithi070 ceph-mon[32631]: mgrmap e14: x(active, since 79s), standbys: z 2024-06-04T04:47:38.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:38 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-04T04:47:38.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:38 smithi112 ceph-mon[36453]: mgrmap e14: x(active, since 79s), standbys: z 2024-06-04T04:47:38.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:38 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-04T04:47:39.245 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:38 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:38.759+0000 7fc7cf009200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-04T04:47:39.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:39 smithi098 ceph-mon[35569]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:39.746 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:39 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:39.355+0000 7fc7cf009200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-04T04:47:39.746 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:47:39 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:47:39.571+0000 7fc7cf009200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-04T04:47:39.809 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:39 smithi070 ceph-mon[32631]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:39.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:39 smithi112 ceph-mon[36453]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:40.483+0000 7f0570ff9700 1 -- 172.21.15.70:0/1957755525 <== mon.0 v2:172.21.15.70:3300/0 8 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7f056800fcc0 con 0x7f05741036a0 2024-06-04T04:47:40.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:40 smithi098 ceph-mon[35569]: Standby manager daemon y started 2024-06-04T04:47:40.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:40 smithi098 ceph-mon[35569]: from='mgr.? 172.21.15.98:0/2269193100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-04T04:47:40.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:40 smithi098 ceph-mon[35569]: from='mgr.? 172.21.15.98:0/2269193100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-04T04:47:40.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:40 smithi098 ceph-mon[35569]: from='mgr.? 172.21.15.98:0/2269193100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-04T04:47:40.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:40 smithi098 ceph-mon[35569]: from='mgr.? 172.21.15.98:0/2269193100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-04T04:47:40.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:40 smithi070 ceph-mon[32631]: Standby manager daemon y started 2024-06-04T04:47:40.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:40 smithi070 ceph-mon[32631]: from='mgr.? 172.21.15.98:0/2269193100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-04T04:47:40.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:40 smithi070 ceph-mon[32631]: from='mgr.? 172.21.15.98:0/2269193100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-04T04:47:40.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:40 smithi070 ceph-mon[32631]: from='mgr.? 172.21.15.98:0/2269193100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-04T04:47:40.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:40 smithi070 ceph-mon[32631]: from='mgr.? 172.21.15.98:0/2269193100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-04T04:47:40.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:40 smithi112 ceph-mon[36453]: Standby manager daemon y started 2024-06-04T04:47:40.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:40 smithi112 ceph-mon[36453]: from='mgr.? 172.21.15.98:0/2269193100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-04T04:47:40.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:40 smithi112 ceph-mon[36453]: from='mgr.? 172.21.15.98:0/2269193100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-04T04:47:40.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:40 smithi112 ceph-mon[36453]: from='mgr.? 172.21.15.98:0/2269193100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-04T04:47:40.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:40 smithi112 ceph-mon[36453]: from='mgr.? 172.21.15.98:0/2269193100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-04T04:47:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:41 smithi070 ceph-mon[32631]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:41 smithi070 ceph-mon[32631]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-06-04T04:47:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:41 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-04T04:47:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:41 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:41 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:41.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:41 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:41.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:41 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:41.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:41 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:41.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:41 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:41 smithi098 ceph-mon[35569]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:41 smithi098 ceph-mon[35569]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-06-04T04:47:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:41 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-04T04:47:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:41 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:41.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:41 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:41.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:41 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:41.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:41 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:41.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:41 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:41.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:41 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:41.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:41 smithi112 ceph-mon[36453]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:41.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:41 smithi112 ceph-mon[36453]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-06-04T04:47:41.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:41 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-04T04:47:41.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:41 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:41.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:41 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:41.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:41 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:41.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:41 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:41.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:41 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:41.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:41 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:43 smithi098 ceph-mon[35569]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-04T04:47:43.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:43.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:43 smithi070 ceph-mon[32631]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:43.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-04T04:47:43.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:43.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:43 smithi112 ceph-mon[36453]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:43.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-04T04:47:43.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:44.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:44 smithi070 ceph-mon[32631]: Deploying daemon osd.0 on smithi070 2024-06-04T04:47:44.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:44 smithi098 ceph-mon[35569]: Deploying daemon osd.0 on smithi070 2024-06-04T04:47:44.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:44 smithi112 ceph-mon[36453]: Deploying daemon osd.0 on smithi070 2024-06-04T04:47:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:45 smithi070 ceph-mon[32631]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:45.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:45 smithi098 ceph-mon[35569]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:45.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:45 smithi112 ceph-mon[36453]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:47 smithi098 ceph-mon[35569]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:47.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:47 smithi070 ceph-mon[32631]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:47 smithi112 ceph-mon[36453]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:49.746 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:49 smithi070 ceph-mon[32631]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:49.746 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:49 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:49.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:49 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:49.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:49 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:49 smithi112 ceph-mon[36453]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:49 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:49 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:49 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:49 smithi098 ceph-mon[35569]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:49 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:49 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:49 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:51 smithi070 ceph-mon[32631]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:51 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:51 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:51 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:51 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:51 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:51 smithi098 ceph-mon[35569]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:51 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:51 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:51 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:51 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:51 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:51.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:51 smithi112 ceph-mon[36453]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:51.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:51 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:51.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:51 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:51.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:51 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:51.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:51 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:51.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:51 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:52.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:52.466+0000 7f0570ff9700 1 -- 172.21.15.70:0/1957755525 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0574063980 con 0x7f055c03e750 2024-06-04T04:47:52.469 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 0 on host 'smithi070' 2024-06-04T04:47:52.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:52.468+0000 7f0579f7f700 1 -- 172.21.15.70:0/1957755525 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f055c03e750 msgr2=0x7f055c040c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:52.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:52.468+0000 7f0579f7f700 1 --2- 172.21.15.70:0/1957755525 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f055c03e750 0x7f055c040c10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0574103500 tx=0x7f05640091b0 comp rx=0 tx=0).stop 2024-06-04T04:47:52.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:52.468+0000 7f0579f7f700 1 -- 172.21.15.70:0/1957755525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05741036a0 msgr2=0x7f057419e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:52.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:52.468+0000 7f0579f7f700 1 --2- 172.21.15.70:0/1957755525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05741036a0 0x7f057419e2a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f056800e9c0 tx=0x7f056800ecd0 comp rx=0 tx=0).stop 2024-06-04T04:47:52.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:52.468+0000 7f0579f7f700 1 -- 172.21.15.70:0/1957755525 shutdown_connections 2024-06-04T04:47:52.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:52.468+0000 7f0579f7f700 1 --2- 172.21.15.70:0/1957755525 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f055c03e750 0x7f055c040c10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:52.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:52.468+0000 7f0579f7f700 1 --2- 172.21.15.70:0/1957755525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0574104040 0x7f05741a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:52.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:52.468+0000 7f0579f7f700 1 --2- 172.21.15.70:0/1957755525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05741024a0 0x7f057406c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:52.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:52.468+0000 7f0579f7f700 1 --2- 172.21.15.70:0/1957755525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05741036a0 0x7f057419e2a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:52.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:52.468+0000 7f0579f7f700 1 -- 172.21.15.70:0/1957755525 >> 172.21.15.70:0/1957755525 conn(0x7f05740fda60 msgr2=0x7f0574106ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:47:52.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:52.469+0000 7f0579f7f700 1 -- 172.21.15.70:0/1957755525 shutdown_connections 2024-06-04T04:47:52.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:47:52.469+0000 7f0579f7f700 1 -- 172.21.15.70:0/1957755525 wait complete. 2024-06-04T04:47:53.131 DEBUG:teuthology.orchestra.run.smithi070:osd.0> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.0.service 2024-06-04T04:47:53.133 INFO:tasks.cephadm:Deploying osd.1 on smithi098 with /dev/vg_nvme/lv_4... 2024-06-04T04:47:53.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-04T04:47:53.375 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.b/config 2024-06-04T04:47:53.737 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:53 smithi098 ceph-mon[35569]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:53.737 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:53 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:53.738 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:53 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:53.738 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:53 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:53.738 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:53 smithi098 ceph-mon[35569]: from='osd.0 [v2:172.21.15.70:6802/824119100,v1:172.21.15.70:6803/824119100]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-04T04:47:53.738 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:53 smithi098 ceph-mon[35569]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-04T04:47:53.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:53 smithi070 ceph-mon[32631]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:53.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:53 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:53.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:53 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:53.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:53 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:53.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:53 smithi070 ceph-mon[32631]: from='osd.0 [v2:172.21.15.70:6802/824119100,v1:172.21.15.70:6803/824119100]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-04T04:47:53.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:53 smithi070 ceph-mon[32631]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-04T04:47:53.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:53 smithi112 ceph-mon[36453]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:53.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:53 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:53.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:53 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:53.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:53 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:47:53.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:53 smithi112 ceph-mon[36453]: from='osd.0 [v2:172.21.15.70:6802/824119100,v1:172.21.15.70:6803/824119100]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-04T04:47:53.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:53 smithi112 ceph-mon[36453]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-04T04:47:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:54 smithi070 ceph-mon[32631]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-04T04:47:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:54 smithi070 ceph-mon[32631]: osdmap e6: 1 total, 0 up, 1 in 2024-06-04T04:47:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:54 smithi070 ceph-mon[32631]: from='osd.0 [v2:172.21.15.70:6802/824119100,v1:172.21.15.70:6803/824119100]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-06-04T04:47:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:54 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:54.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:54 smithi070 ceph-mon[32631]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-06-04T04:47:54.610 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 04:47:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-0[47458]: 2024-06-04T04:47:54.514+0000 7f9f1f5cb700 -1 osd.0 0 waiting for initial osdmap 2024-06-04T04:47:54.610 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 04:47:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-0[47458]: 2024-06-04T04:47:54.517+0000 7f9f1b3be700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-04T04:47:54.652 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:54 smithi098 ceph-mon[35569]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-04T04:47:54.652 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:54 smithi098 ceph-mon[35569]: osdmap e6: 1 total, 0 up, 1 in 2024-06-04T04:47:54.652 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:54 smithi098 ceph-mon[35569]: from='osd.0 [v2:172.21.15.70:6802/824119100,v1:172.21.15.70:6803/824119100]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-06-04T04:47:54.652 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:54 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:54.653 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:54 smithi098 ceph-mon[35569]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-06-04T04:47:54.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:54 smithi112 ceph-mon[36453]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-04T04:47:54.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:54 smithi112 ceph-mon[36453]: osdmap e6: 1 total, 0 up, 1 in 2024-06-04T04:47:54.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:54 smithi112 ceph-mon[36453]: from='osd.0 [v2:172.21.15.70:6802/824119100,v1:172.21.15.70:6803/824119100]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-06-04T04:47:54.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:54 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:54.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:54 smithi112 ceph-mon[36453]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-06-04T04:47:55.248 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-04T04:47:55.248 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-04T04:47:55.248 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-06-04T04:47:55.248 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-06-04T04:47:55.248 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113397 s, 925 MB/s 2024-06-04T04:47:55.248 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-06-04T04:47:55.766 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:55 smithi098 ceph-mon[35569]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:55.766 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:55 smithi098 ceph-mon[35569]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-06-04T04:47:55.766 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:55 smithi098 ceph-mon[35569]: osdmap e7: 1 total, 0 up, 1 in 2024-06-04T04:47:55.766 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:55 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:55.766 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:55 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:55.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_4 2024-06-04T04:47:55.805 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:55 smithi070 ceph-mon[32631]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:55.805 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:55 smithi070 ceph-mon[32631]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-06-04T04:47:55.805 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:55 smithi070 ceph-mon[32631]: osdmap e7: 1 total, 0 up, 1 in 2024-06-04T04:47:55.805 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:55 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:55.806 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:55 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:55 smithi112 ceph-mon[36453]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:55 smithi112 ceph-mon[36453]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-06-04T04:47:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:55 smithi112 ceph-mon[36453]: osdmap e7: 1 total, 0 up, 1 in 2024-06-04T04:47:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:55 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:55 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:56.015 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.b/config 2024-06-04T04:47:56.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:56 smithi098 ceph-mon[35569]: purged_snaps scrub starts 2024-06-04T04:47:56.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:56 smithi098 ceph-mon[35569]: purged_snaps scrub ok 2024-06-04T04:47:56.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:56 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:56.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:56 smithi098 ceph-mon[35569]: osd.0 [v2:172.21.15.70:6802/824119100,v1:172.21.15.70:6803/824119100] boot 2024-06-04T04:47:56.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:56 smithi098 ceph-mon[35569]: osdmap e8: 1 total, 1 up, 1 in 2024-06-04T04:47:56.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:56 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:56.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:56 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:56.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:56 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:56.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:56 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:47:56.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:56 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:56.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:56 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:56.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:56 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:56.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:56 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:56.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:56 smithi070 ceph-mon[32631]: purged_snaps scrub starts 2024-06-04T04:47:56.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:56 smithi070 ceph-mon[32631]: purged_snaps scrub ok 2024-06-04T04:47:56.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:56.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:56 smithi070 ceph-mon[32631]: osd.0 [v2:172.21.15.70:6802/824119100,v1:172.21.15.70:6803/824119100] boot 2024-06-04T04:47:56.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:56 smithi070 ceph-mon[32631]: osdmap e8: 1 total, 1 up, 1 in 2024-06-04T04:47:56.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:56.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:56.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:56.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:47:56.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:56.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:56.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:56.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:56.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:56 smithi112 ceph-mon[36453]: purged_snaps scrub starts 2024-06-04T04:47:56.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:56 smithi112 ceph-mon[36453]: purged_snaps scrub ok 2024-06-04T04:47:56.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:56 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:56.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:56 smithi112 ceph-mon[36453]: osd.0 [v2:172.21.15.70:6802/824119100,v1:172.21.15.70:6803/824119100] boot 2024-06-04T04:47:56.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:56 smithi112 ceph-mon[36453]: osdmap e8: 1 total, 1 up, 1 in 2024-06-04T04:47:56.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:56 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:47:56.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:56 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:56.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:56 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:56.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:56 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:47:56.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:56 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:56.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:56 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:56.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:56 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:47:56.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:56 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:47:57.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.610+0000 7f4622361700 1 -- 172.21.15.98:0/1092007589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f461c1036a0 msgr2=0x7f461c103b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:57.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.610+0000 7f4622361700 1 --2- 172.21.15.98:0/1092007589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f461c1036a0 0x7f461c103b00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f46040099c0 tx=0x7f4604009cd0 comp rx=0 tx=0).stop 2024-06-04T04:47:57.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.611+0000 7f4622361700 1 -- 172.21.15.98:0/1092007589 shutdown_connections 2024-06-04T04:47:57.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.611+0000 7f4622361700 1 --2- 172.21.15.98:0/1092007589 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f461c104040 0x7f461c10a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:57.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.611+0000 7f4622361700 1 --2- 172.21.15.98:0/1092007589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f461c1024a0 0x7f461c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:57.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.611+0000 7f4622361700 1 --2- 172.21.15.98:0/1092007589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f461c1036a0 0x7f461c103b00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:57.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.611+0000 7f4622361700 1 -- 172.21.15.98:0/1092007589 >> 172.21.15.98:0/1092007589 conn(0x7f461c0fda40 msgr2=0x7f461c0ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:47:57.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.611+0000 7f4622361700 1 -- 172.21.15.98:0/1092007589 shutdown_connections 2024-06-04T04:47:57.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.611+0000 7f4622361700 1 -- 172.21.15.98:0/1092007589 wait complete. 2024-06-04T04:47:57.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.611+0000 7f4622361700 1 Processor -- start 2024-06-04T04:47:57.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.611+0000 7f4622361700 1 -- start start 2024-06-04T04:47:57.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.611+0000 7f4622361700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f461c1024a0 0x7f461c06c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:57.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.611+0000 7f4622361700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f461c1036a0 0x7f461c19e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:57.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.611+0000 7f4622361700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f461c104040 0x7f461c1a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.611+0000 7f4622361700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f461c10de40 con 0x7f461c104040 2024-06-04T04:47:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.611+0000 7f4622361700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f461c10dcc0 con 0x7f461c1036a0 2024-06-04T04:47:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.611+0000 7f4622361700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f461c10dfc0 con 0x7f461c1024a0 2024-06-04T04:47:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.612+0000 7f4613fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f461c1036a0 0x7f461c19e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.612+0000 7f4613fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f461c1036a0 0x7f461c19e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51390/0 (socket says 172.21.15.98:51390) 2024-06-04T04:47:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.612+0000 7f4613fff700 1 -- 172.21.15.98:0/1796432708 learned_addr learned my addr 172.21.15.98:0/1796432708 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-04T04:47:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.612+0000 7f461bfff700 1 --2- 172.21.15.98:0/1796432708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f461c1024a0 0x7f461c06c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.612+0000 7f46208fe700 1 --2- 172.21.15.98:0/1796432708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f461c104040 0x7f461c1a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.612+0000 7f4613fff700 1 -- 172.21.15.98:0/1796432708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f461c1024a0 msgr2=0x7f461c06c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.612+0000 7f4613fff700 1 --2- 172.21.15.98:0/1796432708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f461c1024a0 0x7f461c06c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.612+0000 7f4613fff700 1 -- 172.21.15.98:0/1796432708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f461c104040 msgr2=0x7f461c1a0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:47:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.612+0000 7f4613fff700 1 --2- 172.21.15.98:0/1796432708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f461c104040 0x7f461c1a0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:47:57.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.612+0000 7f4613fff700 1 -- 172.21.15.98:0/1796432708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461c1a12e0 con 0x7f461c1036a0 2024-06-04T04:47:57.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.612+0000 7f46208fe700 1 --2- 172.21.15.98:0/1796432708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f461c104040 0x7f461c1a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:47:57.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.612+0000 7f4613fff700 1 --2- 172.21.15.98:0/1796432708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f461c1036a0 0x7f461c19e3d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f46040098e0 tx=0x7f460400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:47:57.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.612+0000 7f4619ffb700 1 -- 172.21.15.98:0/1796432708 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4604027070 con 0x7f461c1036a0 2024-06-04T04:47:57.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.612+0000 7f4619ffb700 1 -- 172.21.15.98:0/1796432708 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f460400bd10 con 0x7f461c1036a0 2024-06-04T04:47:57.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.612+0000 7f4622361700 1 -- 172.21.15.98:0/1796432708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461c1a1570 con 0x7f461c1036a0 2024-06-04T04:47:57.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.612+0000 7f4619ffb700 1 -- 172.21.15.98:0/1796432708 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4604022e20 con 0x7f461c1036a0 2024-06-04T04:47:57.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.612+0000 7f4622361700 1 -- 172.21.15.98:0/1796432708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f461c1a1a20 con 0x7f461c1036a0 2024-06-04T04:47:57.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.613+0000 7f4622361700 1 -- 172.21.15.98:0/1796432708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f461c06a230 con 0x7f461c1036a0 2024-06-04T04:47:57.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.615+0000 7f4619ffb700 1 -- 172.21.15.98:0/1796432708 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7f460402a070 con 0x7f461c1036a0 2024-06-04T04:47:57.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.615+0000 7f4619ffb700 1 --2- 172.21.15.98:0/1796432708 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f46080b2870 0x7f46080b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:47:57.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.615+0000 7f4619ffb700 1 -- 172.21.15.98:0/1796432708 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f46040e1670 con 0x7f461c1036a0 2024-06-04T04:47:57.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.616+0000 7f461bfff700 1 --2- 172.21.15.98:0/1796432708 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f46080b2870 0x7f46080b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:47:57.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.616+0000 7f461bfff700 1 --2- 172.21.15.98:0/1796432708 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f46080b2870 0x7f46080b4d30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f461c103500 tx=0x7f460c007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:47:57.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.619+0000 7f4619ffb700 1 -- 172.21.15.98:0/1796432708 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4604013070 con 0x7f461c1036a0 2024-06-04T04:47:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:57 smithi098 ceph-mon[35569]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:57 smithi098 ceph-mon[35569]: Detected new or changed devices on smithi070 2024-06-04T04:47:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:57 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi070 to 17088M 2024-06-04T04:47:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:57 smithi098 ceph-mon[35569]: osdmap e9: 1 total, 1 up, 1 in 2024-06-04T04:47:57.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:47:57.831+0000 7f4622361700 1 -- 172.21.15.98:0/1796432708 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f461c063980 con 0x7f46080b2870 2024-06-04T04:47:57.837 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:57 smithi112 ceph-mon[36453]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:57.837 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:57 smithi112 ceph-mon[36453]: Detected new or changed devices on smithi070 2024-06-04T04:47:57.837 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:57 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi070 to 17088M 2024-06-04T04:47:57.837 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:57 smithi112 ceph-mon[36453]: osdmap e9: 1 total, 1 up, 1 in 2024-06-04T04:47:57.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:57 smithi070 ceph-mon[32631]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-04T04:47:57.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:57 smithi070 ceph-mon[32631]: Detected new or changed devices on smithi070 2024-06-04T04:47:57.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:57 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi070 to 17088M 2024-06-04T04:47:57.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:57 smithi070 ceph-mon[32631]: osdmap e9: 1 total, 1 up, 1 in 2024-06-04T04:47:58.736 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:58 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:47:58.736 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:58 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:47:58.736 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:58 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:58.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:58 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:47:58.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:58 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:47:58.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:58 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:58.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:58 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:47:58.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:58 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:47:58.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:58 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:47:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:59 smithi098 ceph-mon[35569]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:47:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:47:59 smithi098 ceph-mon[35569]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:47:59.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:59 smithi070 ceph-mon[32631]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:47:59.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:47:59 smithi070 ceph-mon[32631]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:47:59.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:59 smithi112 ceph-mon[36453]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:47:59.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:47:59 smithi112 ceph-mon[36453]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:48:00.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:00 smithi070 ceph-mon[32631]: from='client.? 172.21.15.98:0/74694962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd5e4806-535b-49b5-ba6b-93228da35a84"}]: dispatch 2024-06-04T04:48:00.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:00 smithi070 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd5e4806-535b-49b5-ba6b-93228da35a84"}]: dispatch 2024-06-04T04:48:00.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:00 smithi070 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd5e4806-535b-49b5-ba6b-93228da35a84"}]': finished 2024-06-04T04:48:00.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:00 smithi070 ceph-mon[32631]: osdmap e10: 2 total, 1 up, 2 in 2024-06-04T04:48:00.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:00 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:00.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:00 smithi112 ceph-mon[36453]: from='client.? 172.21.15.98:0/74694962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd5e4806-535b-49b5-ba6b-93228da35a84"}]: dispatch 2024-06-04T04:48:00.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:00 smithi112 ceph-mon[36453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd5e4806-535b-49b5-ba6b-93228da35a84"}]: dispatch 2024-06-04T04:48:00.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:00 smithi112 ceph-mon[36453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd5e4806-535b-49b5-ba6b-93228da35a84"}]': finished 2024-06-04T04:48:00.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:00 smithi112 ceph-mon[36453]: osdmap e10: 2 total, 1 up, 2 in 2024-06-04T04:48:00.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:00 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:00.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:00 smithi098 ceph-mon[35569]: from='client.? 172.21.15.98:0/74694962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd5e4806-535b-49b5-ba6b-93228da35a84"}]: dispatch 2024-06-04T04:48:00.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:00 smithi098 ceph-mon[35569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd5e4806-535b-49b5-ba6b-93228da35a84"}]: dispatch 2024-06-04T04:48:00.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:00 smithi098 ceph-mon[35569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd5e4806-535b-49b5-ba6b-93228da35a84"}]': finished 2024-06-04T04:48:00.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:00 smithi098 ceph-mon[35569]: osdmap e10: 2 total, 1 up, 2 in 2024-06-04T04:48:00.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:00 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:01.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:01 smithi070 ceph-mon[32631]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:01.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:01 smithi070 ceph-mon[32631]: from='client.? 172.21.15.98:0/4120165356' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:48:01.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:01 smithi112 ceph-mon[36453]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:01.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:01 smithi112 ceph-mon[36453]: from='client.? 172.21.15.98:0/4120165356' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:48:01.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:01 smithi098 ceph-mon[35569]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:01.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:01 smithi098 ceph-mon[35569]: from='client.? 172.21.15.98:0/4120165356' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:48:03.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:03 smithi070 ceph-mon[32631]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:03.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:03 smithi112 ceph-mon[36453]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:03.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:03 smithi098 ceph-mon[35569]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:05.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:05 smithi070 ceph-mon[32631]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:05.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:05 smithi112 ceph-mon[36453]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:05.980 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:05 smithi098 ceph-mon[35569]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:07.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:07 smithi070 ceph-mon[32631]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:07.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:07 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-04T04:48:07.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:07 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:07.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:07 smithi112 ceph-mon[36453]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:07.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:07 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-04T04:48:07.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:07 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:07.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:07 smithi098 ceph-mon[35569]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:07.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:07 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-04T04:48:07.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:07 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:08.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:08 smithi098 ceph-mon[35569]: Deploying daemon osd.1 on smithi098 2024-06-04T04:48:08.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:08 smithi070 ceph-mon[32631]: Deploying daemon osd.1 on smithi098 2024-06-04T04:48:08.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:08 smithi112 ceph-mon[36453]: Deploying daemon osd.1 on smithi098 2024-06-04T04:48:09.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:09 smithi070 ceph-mon[32631]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:09.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:09 smithi112 ceph-mon[36453]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:09.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:09 smithi098 ceph-mon[35569]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:11.713 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:11 smithi098 ceph-mon[35569]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:11.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:11 smithi070 ceph-mon[32631]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:11 smithi112 ceph-mon[36453]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:13.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:13 smithi070 ceph-mon[32631]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:13.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:13 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:13.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:13 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:13.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:13 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:48:13.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:13 smithi098 ceph-mon[35569]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:13.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:13.882 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:13.882 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:48:13.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:13 smithi112 ceph-mon[36453]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:13.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:13 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:13.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:13 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:13.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:13 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:48:15.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:15 smithi098 ceph-mon[35569]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:15.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:15 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:15.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:15 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:15.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:15 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:15.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:15 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:48:15.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:15 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:15.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:15 smithi070 ceph-mon[32631]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:15.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:15 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:15.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:15 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:15.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:15 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:15.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:15 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:48:15.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:15 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:15.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:15 smithi112 ceph-mon[36453]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:15.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:15 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:15.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:15 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:15.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:15 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:15.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:15 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:48:15.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:15 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:16.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:48:16.595+0000 7f4619ffb700 1 -- 172.21.15.98:0/1796432708 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f461c063980 con 0x7f46080b2870 2024-06-04T04:48:16.598 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 1 on host 'smithi098' 2024-06-04T04:48:16.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:48:16.598+0000 7f4622361700 1 -- 172.21.15.98:0/1796432708 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f46080b2870 msgr2=0x7f46080b4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:48:16.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:48:16.598+0000 7f4622361700 1 --2- 172.21.15.98:0/1796432708 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f46080b2870 0x7f46080b4d30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f461c103500 tx=0x7f460c007400 comp rx=0 tx=0).stop 2024-06-04T04:48:16.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:48:16.598+0000 7f4622361700 1 -- 172.21.15.98:0/1796432708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f461c1036a0 msgr2=0x7f461c19e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:48:16.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:48:16.598+0000 7f4622361700 1 --2- 172.21.15.98:0/1796432708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f461c1036a0 0x7f461c19e3d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f46040098e0 tx=0x7f460400f690 comp rx=0 tx=0).stop 2024-06-04T04:48:16.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:48:16.598+0000 7f4622361700 1 -- 172.21.15.98:0/1796432708 shutdown_connections 2024-06-04T04:48:16.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:48:16.598+0000 7f4622361700 1 --2- 172.21.15.98:0/1796432708 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f46080b2870 0x7f46080b4d30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:16.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:48:16.598+0000 7f4622361700 1 --2- 172.21.15.98:0/1796432708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f461c104040 0x7f461c1a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:16.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:48:16.598+0000 7f4622361700 1 --2- 172.21.15.98:0/1796432708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f461c1024a0 0x7f461c06c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:16.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:48:16.598+0000 7f4622361700 1 --2- 172.21.15.98:0/1796432708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f461c1036a0 0x7f461c19e3d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:16.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:48:16.598+0000 7f4622361700 1 -- 172.21.15.98:0/1796432708 >> 172.21.15.98:0/1796432708 conn(0x7f461c0fda40 msgr2=0x7f461c106ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:48:16.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:48:16.598+0000 7f4622361700 1 -- 172.21.15.98:0/1796432708 shutdown_connections 2024-06-04T04:48:16.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:48:16.598+0000 7f4622361700 1 -- 172.21.15.98:0/1796432708 wait complete. 2024-06-04T04:48:17.442 DEBUG:teuthology.orchestra.run.smithi098:osd.1> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.1.service 2024-06-04T04:48:17.445 INFO:tasks.cephadm:Deploying osd.2 on smithi112 with /dev/vg_nvme/lv_4... 2024-06-04T04:48:17.445 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-04T04:48:17.692 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.c/config 2024-06-04T04:48:17.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:17 smithi098 ceph-mon[35569]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:17.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:17 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:17.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:17 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:17.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:17 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:48:17.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:17 smithi098 ceph-mon[35569]: from='osd.1 [v2:172.21.15.98:6800/2744598742,v1:172.21.15.98:6801/2744598742]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-04T04:48:17.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:17 smithi098 ceph-mon[35569]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-04T04:48:17.813 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:17 smithi112 ceph-mon[36453]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:17.813 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:17 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:17.814 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:17 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:17.814 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:17 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:48:17.814 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:17 smithi112 ceph-mon[36453]: from='osd.1 [v2:172.21.15.98:6800/2744598742,v1:172.21.15.98:6801/2744598742]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-04T04:48:17.814 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:17 smithi112 ceph-mon[36453]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-04T04:48:17.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:17 smithi070 ceph-mon[32631]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:17.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:17 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:17.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:17 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:17.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:17 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:48:17.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:17 smithi070 ceph-mon[32631]: from='osd.1 [v2:172.21.15.98:6800/2744598742,v1:172.21.15.98:6801/2744598742]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-04T04:48:17.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:17 smithi070 ceph-mon[32631]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-04T04:48:18.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:18 smithi098 ceph-mon[35569]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-04T04:48:18.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:18 smithi098 ceph-mon[35569]: from='osd.1 [v2:172.21.15.98:6800/2744598742,v1:172.21.15.98:6801/2744598742]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:48:18.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:18 smithi098 ceph-mon[35569]: osdmap e11: 2 total, 1 up, 2 in 2024-06-04T04:48:18.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:18.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:18 smithi098 ceph-mon[35569]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:48:18.836 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:18 smithi112 ceph-mon[36453]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-04T04:48:18.836 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:18 smithi112 ceph-mon[36453]: from='osd.1 [v2:172.21.15.98:6800/2744598742,v1:172.21.15.98:6801/2744598742]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:48:18.836 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:18 smithi112 ceph-mon[36453]: osdmap e11: 2 total, 1 up, 2 in 2024-06-04T04:48:18.836 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:18.836 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:18 smithi112 ceph-mon[36453]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:48:18.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:18 smithi070 ceph-mon[32631]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-04T04:48:18.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:18 smithi070 ceph-mon[32631]: from='osd.1 [v2:172.21.15.98:6800/2744598742,v1:172.21.15.98:6801/2744598742]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:48:18.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:18 smithi070 ceph-mon[32631]: osdmap e11: 2 total, 1 up, 2 in 2024-06-04T04:48:18.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:18.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:18 smithi070 ceph-mon[32631]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:48:19.246 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 04:48:18 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-1[40165]: 2024-06-04T04:48:18.994+0000 7f76f3d59700 -1 osd.1 0 waiting for initial osdmap 2024-06-04T04:48:19.246 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 04:48:18 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-1[40165]: 2024-06-04T04:48:18.997+0000 7f76ebb44700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-04T04:48:19.554 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-04T04:48:19.554 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-04T04:48:19.555 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2024-06-04T04:48:19.555 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2024-06-04T04:48:19.555 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114322 s, 917 MB/s 2024-06-04T04:48:19.555 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2024-06-04T04:48:19.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:19 smithi070 ceph-mon[32631]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:19.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:19 smithi070 ceph-mon[32631]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-06-04T04:48:19.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:19 smithi070 ceph-mon[32631]: osdmap e12: 2 total, 1 up, 2 in 2024-06-04T04:48:19.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:19.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:19.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:19 smithi070 ceph-mon[32631]: from='osd.1 ' entity='osd.1' 2024-06-04T04:48:19.928 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:19 smithi112 ceph-mon[36453]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:19.928 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:19 smithi112 ceph-mon[36453]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-06-04T04:48:19.928 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:19 smithi112 ceph-mon[36453]: osdmap e12: 2 total, 1 up, 2 in 2024-06-04T04:48:19.928 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:19.928 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:19.928 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:19 smithi112 ceph-mon[36453]: from='osd.1 ' entity='osd.1' 2024-06-04T04:48:19.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:19 smithi098 ceph-mon[35569]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:19.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:19 smithi098 ceph-mon[35569]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-06-04T04:48:19.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:19 smithi098 ceph-mon[35569]: osdmap e12: 2 total, 1 up, 2 in 2024-06-04T04:48:19.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:19.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:19.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:19 smithi098 ceph-mon[35569]: from='osd.1 ' entity='osd.1' 2024-06-04T04:48:20.212 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_4 2024-06-04T04:48:20.456 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.c/config 2024-06-04T04:48:20.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:20 smithi098 ceph-mon[35569]: purged_snaps scrub starts 2024-06-04T04:48:20.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:20 smithi098 ceph-mon[35569]: purged_snaps scrub ok 2024-06-04T04:48:20.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:20 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:20.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:20 smithi098 ceph-mon[35569]: osd.1 [v2:172.21.15.98:6800/2744598742,v1:172.21.15.98:6801/2744598742] boot 2024-06-04T04:48:20.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:20 smithi098 ceph-mon[35569]: osdmap e13: 2 total, 2 up, 2 in 2024-06-04T04:48:20.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:20 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:20.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:20 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:20.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:20 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:20.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:20 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:48:20.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:20 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:20.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:20 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:20.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:20 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:48:20.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:20 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:20.782 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:20 smithi112 ceph-mon[36453]: purged_snaps scrub starts 2024-06-04T04:48:20.782 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:20 smithi112 ceph-mon[36453]: purged_snaps scrub ok 2024-06-04T04:48:20.782 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:20 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:20.782 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:20 smithi112 ceph-mon[36453]: osd.1 [v2:172.21.15.98:6800/2744598742,v1:172.21.15.98:6801/2744598742] boot 2024-06-04T04:48:20.783 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:20 smithi112 ceph-mon[36453]: osdmap e13: 2 total, 2 up, 2 in 2024-06-04T04:48:20.783 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:20 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:20.783 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:20 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:20.783 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:20 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:20.783 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:20 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:48:20.783 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:20 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:20.783 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:20 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:20.783 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:20 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:48:20.783 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:20 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:20.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:20 smithi070 ceph-mon[32631]: purged_snaps scrub starts 2024-06-04T04:48:20.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:20 smithi070 ceph-mon[32631]: purged_snaps scrub ok 2024-06-04T04:48:20.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:20 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:20.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:20 smithi070 ceph-mon[32631]: osd.1 [v2:172.21.15.98:6800/2744598742,v1:172.21.15.98:6801/2744598742] boot 2024-06-04T04:48:20.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:20 smithi070 ceph-mon[32631]: osdmap e13: 2 total, 2 up, 2 in 2024-06-04T04:48:20.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:20 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:48:20.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:20 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:20.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:20 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:20.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:20 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:48:20.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:20 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:20.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:20 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:20.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:20 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:48:20.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:20 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:21.648 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:21 smithi112 ceph-mon[36453]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:21.649 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:21 smithi112 ceph-mon[36453]: Detected new or changed devices on smithi098 2024-06-04T04:48:21.649 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:21 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi098 to 17088M 2024-06-04T04:48:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:21 smithi070 ceph-mon[32631]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:21 smithi070 ceph-mon[32631]: Detected new or changed devices on smithi098 2024-06-04T04:48:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:21 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi098 to 17088M 2024-06-04T04:48:21.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.883+0000 7fef76163700 1 -- 172.21.15.112:0/3349808853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70104030 msgr2=0x7fef7010a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:48:21.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.883+0000 7fef76163700 1 --2- 172.21.15.112:0/3349808853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70104030 0x7fef7010a8e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fef68009990 tx=0x7fef68009ca0 comp rx=0 tx=0).stop 2024-06-04T04:48:21.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.883+0000 7fef76163700 1 -- 172.21.15.112:0/3349808853 shutdown_connections 2024-06-04T04:48:21.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.883+0000 7fef76163700 1 --2- 172.21.15.112:0/3349808853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70104030 0x7fef7010a8e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:21.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.883+0000 7fef76163700 1 --2- 172.21.15.112:0/3349808853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef70102490 0x7fef701028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:21.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.883+0000 7fef76163700 1 --2- 172.21.15.112:0/3349808853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef70103690 0x7fef70103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:21.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.883+0000 7fef76163700 1 -- 172.21.15.112:0/3349808853 >> 172.21.15.112:0/3349808853 conn(0x7fef700fda30 msgr2=0x7fef700ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:48:21.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.883+0000 7fef76163700 1 -- 172.21.15.112:0/3349808853 shutdown_connections 2024-06-04T04:48:21.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.883+0000 7fef76163700 1 -- 172.21.15.112:0/3349808853 wait complete. 2024-06-04T04:48:21.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef76163700 1 Processor -- start 2024-06-04T04:48:21.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef76163700 1 -- start start 2024-06-04T04:48:21.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef76163700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef70102490 0x7fef70198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:48:21.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef76163700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef70103690 0x7fef70198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:48:21.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef76163700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70104030 0x7fef7019fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:48:21.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef76163700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fef7010de30 con 0x7fef70103690 2024-06-04T04:48:21.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef76163700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef7010dcb0 con 0x7fef70102490 2024-06-04T04:48:21.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef76163700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fef7010dfb0 con 0x7fef70104030 2024-06-04T04:48:21.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef6ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70104030 0x7fef7019fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:48:21.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef6ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70104030 0x7fef7019fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:49918/0 (socket says 172.21.15.112:49918) 2024-06-04T04:48:21.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef6ffff700 1 -- 172.21.15.112:0/2424955214 learned_addr learned my addr 172.21.15.112:0/2424955214 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-06-04T04:48:21.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef6effd700 1 --2- 172.21.15.112:0/2424955214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef70103690 0x7fef70198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:48:21.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef6f7fe700 1 --2- 172.21.15.112:0/2424955214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef70102490 0x7fef70198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:48:21.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef6ffff700 1 -- 172.21.15.112:0/2424955214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef70102490 msgr2=0x7fef70198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:48:21.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef6ffff700 1 --2- 172.21.15.112:0/2424955214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef70102490 0x7fef70198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:21.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef6ffff700 1 -- 172.21.15.112:0/2424955214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef70103690 msgr2=0x7fef70198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:48:21.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef6ffff700 1 --2- 172.21.15.112:0/2424955214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef70103690 0x7fef70198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:21.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef6ffff700 1 -- 172.21.15.112:0/2424955214 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef701a00a0 con 0x7fef70104030 2024-06-04T04:48:21.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.884+0000 7fef6f7fe700 1 --2- 172.21.15.112:0/2424955214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef70102490 0x7fef70198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:48:21.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.885+0000 7fef6ffff700 1 --2- 172.21.15.112:0/2424955214 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70104030 0x7fef7019fa00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fef680047c0 tx=0x7fef68009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:48:21.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.885+0000 7fef6cff9700 1 -- 172.21.15.112:0/2424955214 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6802a070 con 0x7fef70104030 2024-06-04T04:48:21.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.885+0000 7fef6cff9700 1 -- 172.21.15.112:0/2424955214 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef6801ac60 con 0x7fef70104030 2024-06-04T04:48:21.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.885+0000 7fef76163700 1 -- 172.21.15.112:0/2424955214 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef7006bcf0 con 0x7fef70104030 2024-06-04T04:48:21.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.885+0000 7fef6cff9700 1 -- 172.21.15.112:0/2424955214 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef680041c0 con 0x7fef70104030 2024-06-04T04:48:21.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.885+0000 7fef76163700 1 -- 172.21.15.112:0/2424955214 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef7006c1d0 con 0x7fef70104030 2024-06-04T04:48:21.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.886+0000 7fef76163700 1 -- 172.21.15.112:0/2424955214 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef7006a230 con 0x7fef70104030 2024-06-04T04:48:21.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.887+0000 7fef6cff9700 1 -- 172.21.15.112:0/2424955214 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7fef6802d070 con 0x7fef70104030 2024-06-04T04:48:21.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.888+0000 7fef6cff9700 1 --2- 172.21.15.112:0/2424955214 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fef580b27f0 0x7fef580b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:48:21.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.888+0000 7fef6f7fe700 1 --2- 172.21.15.112:0/2424955214 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fef580b27f0 0x7fef580b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:48:21.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.888+0000 7fef6cff9700 1 -- 172.21.15.112:0/2424955214 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fef680e3e20 con 0x7fef70104030 2024-06-04T04:48:21.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.889+0000 7fef6f7fe700 1 --2- 172.21.15.112:0/2424955214 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fef580b27f0 0x7fef580b4cb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fef701034f0 tx=0x7fef60009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:48:21.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:21.892+0000 7fef6cff9700 1 -- 172.21.15.112:0/2424955214 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fef68034020 con 0x7fef70104030 2024-06-04T04:48:21.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:21 smithi098 ceph-mon[35569]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-04T04:48:21.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:21 smithi098 ceph-mon[35569]: Detected new or changed devices on smithi098 2024-06-04T04:48:21.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:21 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi098 to 17088M 2024-06-04T04:48:22.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:22.103+0000 7fef76163700 1 -- 172.21.15.112:0/2424955214 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fef7006c4f0 con 0x7fef580b27f0 2024-06-04T04:48:22.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:22 smithi098 ceph-mon[35569]: osdmap e14: 2 total, 2 up, 2 in 2024-06-04T04:48:22.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:48:22.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:48:22.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:22.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:22 smithi070 ceph-mon[32631]: osdmap e14: 2 total, 2 up, 2 in 2024-06-04T04:48:22.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:48:22.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:48:22.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:22.865 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:22 smithi112 ceph-mon[36453]: osdmap e14: 2 total, 2 up, 2 in 2024-06-04T04:48:22.865 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:48:22.865 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:48:22.865 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:23.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:23 smithi070 ceph-mon[32631]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:23.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:23 smithi070 ceph-mon[32631]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:48:23.914 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:23 smithi112 ceph-mon[36453]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:23.915 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:23 smithi112 ceph-mon[36453]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:48:23.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:23 smithi098 ceph-mon[35569]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:23.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:23 smithi098 ceph-mon[35569]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:48:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:25 smithi070 ceph-mon[32631]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:25 smithi070 ceph-mon[32631]: from='client.? 172.21.15.112:0/3433577841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8456fb55-910a-4a59-b84b-2c0f68346ac0"}]: dispatch 2024-06-04T04:48:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:25 smithi070 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8456fb55-910a-4a59-b84b-2c0f68346ac0"}]: dispatch 2024-06-04T04:48:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:25 smithi070 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8456fb55-910a-4a59-b84b-2c0f68346ac0"}]': finished 2024-06-04T04:48:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:25 smithi070 ceph-mon[32631]: osdmap e15: 3 total, 2 up, 3 in 2024-06-04T04:48:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:25 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:48:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:25 smithi070 ceph-mon[32631]: from='client.? 172.21.15.112:0/1626202455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:48:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:25 smithi112 ceph-mon[36453]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:25 smithi112 ceph-mon[36453]: from='client.? 172.21.15.112:0/3433577841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8456fb55-910a-4a59-b84b-2c0f68346ac0"}]: dispatch 2024-06-04T04:48:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:25 smithi112 ceph-mon[36453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8456fb55-910a-4a59-b84b-2c0f68346ac0"}]: dispatch 2024-06-04T04:48:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:25 smithi112 ceph-mon[36453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8456fb55-910a-4a59-b84b-2c0f68346ac0"}]': finished 2024-06-04T04:48:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:25 smithi112 ceph-mon[36453]: osdmap e15: 3 total, 2 up, 3 in 2024-06-04T04:48:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:25 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:48:25.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:25 smithi112 ceph-mon[36453]: from='client.? 172.21.15.112:0/1626202455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:48:25.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:25 smithi098 ceph-mon[35569]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:25.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:25 smithi098 ceph-mon[35569]: from='client.? 172.21.15.112:0/3433577841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8456fb55-910a-4a59-b84b-2c0f68346ac0"}]: dispatch 2024-06-04T04:48:25.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:25 smithi098 ceph-mon[35569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8456fb55-910a-4a59-b84b-2c0f68346ac0"}]: dispatch 2024-06-04T04:48:25.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:25 smithi098 ceph-mon[35569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8456fb55-910a-4a59-b84b-2c0f68346ac0"}]': finished 2024-06-04T04:48:25.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:25 smithi098 ceph-mon[35569]: osdmap e15: 3 total, 2 up, 3 in 2024-06-04T04:48:25.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:25 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:48:25.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:25 smithi098 ceph-mon[35569]: from='client.? 172.21.15.112:0/1626202455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:48:27.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:27 smithi070 ceph-mon[32631]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:27.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:27 smithi112 ceph-mon[36453]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:27.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:27 smithi098 ceph-mon[35569]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:28.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:28 smithi112 ceph-mon[36453]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:28.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:28 smithi070 ceph-mon[32631]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:28.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:28 smithi098 ceph-mon[35569]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:30.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:30 smithi098 ceph-mon[35569]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:31.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:30 smithi070 ceph-mon[32631]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:31.151 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:30 smithi112 ceph-mon[36453]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:31.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:31 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-04T04:48:31.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:31 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:31.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:31 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-04T04:48:31.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:31 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:32.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:31 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-04T04:48:32.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:31 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:32.912 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:32 smithi112 ceph-mon[36453]: Deploying daemon osd.2 on smithi112 2024-06-04T04:48:32.913 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:32 smithi112 ceph-mon[36453]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:32.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:32 smithi098 ceph-mon[35569]: Deploying daemon osd.2 on smithi112 2024-06-04T04:48:32.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:32 smithi098 ceph-mon[35569]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:33.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:32 smithi070 ceph-mon[32631]: Deploying daemon osd.2 on smithi112 2024-06-04T04:48:33.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:32 smithi070 ceph-mon[32631]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:34.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:34 smithi112 ceph-mon[36453]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:34.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:34 smithi098 ceph-mon[35569]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:34 smithi070 ceph-mon[32631]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:36.787 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:36 smithi112 ceph-mon[36453]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:36.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:36 smithi098 ceph-mon[35569]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:37.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:36 smithi070 ceph-mon[32631]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:38.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:38 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:38.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:38 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:38.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:38 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:48:38.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:38 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:38.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:38 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:38.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:38 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:48:38.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:38 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:38.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:38 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:38.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:38 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:48:39.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:39 smithi112 ceph-mon[36453]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:39.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:39 smithi070 ceph-mon[32631]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:39.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:39 smithi098 ceph-mon[35569]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:40.620 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:40 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:40.620 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:40 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:40.620 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:40 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:40.620 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:40 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:48:40.620 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:40 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:40.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:40.617+0000 7fef6cff9700 1 -- 172.21.15.112:0/2424955214 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fef7006c4f0 con 0x7fef580b27f0 2024-06-04T04:48:40.620 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 2 on host 'smithi112' 2024-06-04T04:48:40.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:40.620+0000 7fef76163700 1 -- 172.21.15.112:0/2424955214 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fef580b27f0 msgr2=0x7fef580b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:48:40.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:40.620+0000 7fef76163700 1 --2- 172.21.15.112:0/2424955214 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fef580b27f0 0x7fef580b4cb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fef701034f0 tx=0x7fef60009040 comp rx=0 tx=0).stop 2024-06-04T04:48:40.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:40.620+0000 7fef76163700 1 -- 172.21.15.112:0/2424955214 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70104030 msgr2=0x7fef7019fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:48:40.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:40.620+0000 7fef76163700 1 --2- 172.21.15.112:0/2424955214 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70104030 0x7fef7019fa00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fef680047c0 tx=0x7fef68009580 comp rx=0 tx=0).stop 2024-06-04T04:48:40.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:40.620+0000 7fef76163700 1 -- 172.21.15.112:0/2424955214 shutdown_connections 2024-06-04T04:48:40.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:40.620+0000 7fef76163700 1 --2- 172.21.15.112:0/2424955214 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fef580b27f0 0x7fef580b4cb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:40.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:40.620+0000 7fef76163700 1 --2- 172.21.15.112:0/2424955214 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70104030 0x7fef7019fa00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:40.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:40.620+0000 7fef76163700 1 --2- 172.21.15.112:0/2424955214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef70102490 0x7fef70198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:40.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:40.620+0000 7fef76163700 1 --2- 172.21.15.112:0/2424955214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef70103690 0x7fef70198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:40.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:40.620+0000 7fef76163700 1 -- 172.21.15.112:0/2424955214 >> 172.21.15.112:0/2424955214 conn(0x7fef700fda30 msgr2=0x7fef70106990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:48:40.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:40.620+0000 7fef76163700 1 -- 172.21.15.112:0/2424955214 shutdown_connections 2024-06-04T04:48:40.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:48:40.620+0000 7fef76163700 1 -- 172.21.15.112:0/2424955214 wait complete. 2024-06-04T04:48:40.861 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:40 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:40.861 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:40 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:40.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:40 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:40.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:40 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:48:40.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:40 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:40.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:40 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:40.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:40 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:40.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:40 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:40.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:40 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:48:40.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:40 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:41.196 DEBUG:teuthology.orchestra.run.smithi112:osd.2> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.2.service 2024-06-04T04:48:41.199 INFO:tasks.cephadm:Deploying osd.3 on smithi070 with /dev/vg_nvme/lv_3... 2024-06-04T04:48:41.199 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-04T04:48:41.442 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:48:41.816 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:41 smithi070 ceph-mon[32631]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:41.816 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:41 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:41.816 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:41 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:41.816 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:41 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:48:41.817 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:41 smithi070 ceph-mon[32631]: from='osd.2 [v2:172.21.15.112:6800/1328098370,v1:172.21.15.112:6801/1328098370]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-04T04:48:41.817 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:41 smithi070 ceph-mon[32631]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-04T04:48:41.915 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:41 smithi112 ceph-mon[36453]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:41.915 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:41 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:41.916 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:41 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:41.916 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:41 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:48:41.916 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:41 smithi112 ceph-mon[36453]: from='osd.2 [v2:172.21.15.112:6800/1328098370,v1:172.21.15.112:6801/1328098370]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-04T04:48:41.916 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:41 smithi112 ceph-mon[36453]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-04T04:48:41.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:41 smithi098 ceph-mon[35569]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:41.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:41 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:41.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:41 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:41.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:41 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:48:41.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:41 smithi098 ceph-mon[35569]: from='osd.2 [v2:172.21.15.112:6800/1328098370,v1:172.21.15.112:6801/1328098370]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-04T04:48:41.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:41 smithi098 ceph-mon[35569]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-04T04:48:42.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:42 smithi070 ceph-mon[32631]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-04T04:48:42.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:42 smithi070 ceph-mon[32631]: osdmap e16: 3 total, 2 up, 3 in 2024-06-04T04:48:42.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:42 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:48:42.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:42 smithi070 ceph-mon[32631]: from='osd.2 [v2:172.21.15.112:6800/1328098370,v1:172.21.15.112:6801/1328098370]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:48:42.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:42 smithi070 ceph-mon[32631]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:48:42.851 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:42 smithi070 ceph-mon[32631]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:42.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:42 smithi112 ceph-mon[36453]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-04T04:48:42.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:42 smithi112 ceph-mon[36453]: osdmap e16: 3 total, 2 up, 3 in 2024-06-04T04:48:42.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:42 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:48:42.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:42 smithi112 ceph-mon[36453]: from='osd.2 [v2:172.21.15.112:6800/1328098370,v1:172.21.15.112:6801/1328098370]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:48:42.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:42 smithi112 ceph-mon[36453]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:48:42.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:42 smithi112 ceph-mon[36453]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:42.937 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 04:48:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-2[41553]: 2024-06-04T04:48:42.628+0000 7f3e88a66700 -1 osd.2 0 waiting for initial osdmap 2024-06-04T04:48:42.937 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 04:48:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-2[41553]: 2024-06-04T04:48:42.632+0000 7f3e84859700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-04T04:48:42.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:42 smithi098 ceph-mon[35569]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-04T04:48:42.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:42 smithi098 ceph-mon[35569]: osdmap e16: 3 total, 2 up, 3 in 2024-06-04T04:48:42.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:42 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:48:42.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:42 smithi098 ceph-mon[35569]: from='osd.2 [v2:172.21.15.112:6800/1328098370,v1:172.21.15.112:6801/1328098370]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:48:42.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:42 smithi098 ceph-mon[35569]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:48:42.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:42 smithi098 ceph-mon[35569]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-04T04:48:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-04T04:48:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-04T04:48:43.534 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-06-04T04:48:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-06-04T04:48:43.534 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0087994 s, 1.2 GB/s 2024-06-04T04:48:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-06-04T04:48:43.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:43 smithi112 ceph-mon[36453]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-06-04T04:48:43.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:43 smithi112 ceph-mon[36453]: osdmap e17: 3 total, 2 up, 3 in 2024-06-04T04:48:43.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:48:43.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:48:43.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:43 smithi098 ceph-mon[35569]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-06-04T04:48:43.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:43 smithi098 ceph-mon[35569]: osdmap e17: 3 total, 2 up, 3 in 2024-06-04T04:48:43.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:48:43.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:48:44.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:43 smithi070 ceph-mon[32631]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-06-04T04:48:44.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:43 smithi070 ceph-mon[32631]: osdmap e17: 3 total, 2 up, 3 in 2024-06-04T04:48:44.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:48:44.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:48:44.184 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi070:vg_nvme/lv_3 2024-06-04T04:48:44.423 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:48:44.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:44 smithi070 ceph-mon[32631]: purged_snaps scrub starts 2024-06-04T04:48:44.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:44 smithi070 ceph-mon[32631]: purged_snaps scrub ok 2024-06-04T04:48:44.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:44 smithi070 ceph-mon[32631]: osd.2 [v2:172.21.15.112:6800/1328098370,v1:172.21.15.112:6801/1328098370] boot 2024-06-04T04:48:44.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:44 smithi070 ceph-mon[32631]: osdmap e18: 3 total, 3 up, 3 in 2024-06-04T04:48:44.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:44 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:48:44.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:44 smithi070 ceph-mon[32631]: pgmap v81: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:44.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:44 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-04T04:48:44.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:44 smithi070 ceph-mon[32631]: Detected new or changed devices on smithi112 2024-06-04T04:48:44.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:44 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:44.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:44 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:44.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:44 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:48:44.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:44 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi112 to 17088M 2024-06-04T04:48:44.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:44 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:44.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:44 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:44.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:44 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:48:44.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:44 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:44.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:44 smithi112 ceph-mon[36453]: purged_snaps scrub starts 2024-06-04T04:48:44.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:44 smithi112 ceph-mon[36453]: purged_snaps scrub ok 2024-06-04T04:48:44.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:44 smithi112 ceph-mon[36453]: osd.2 [v2:172.21.15.112:6800/1328098370,v1:172.21.15.112:6801/1328098370] boot 2024-06-04T04:48:44.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:44 smithi112 ceph-mon[36453]: osdmap e18: 3 total, 3 up, 3 in 2024-06-04T04:48:44.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:44 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:48:44.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:44 smithi112 ceph-mon[36453]: pgmap v81: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:44.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:44 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-04T04:48:44.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:44 smithi112 ceph-mon[36453]: Detected new or changed devices on smithi112 2024-06-04T04:48:44.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:44 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:44.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:44 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:44.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:44 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:48:44.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:44 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi112 to 17088M 2024-06-04T04:48:44.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:44 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:44.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:44 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:44.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:44 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:48:44.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:44 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:44.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:44 smithi098 ceph-mon[35569]: purged_snaps scrub starts 2024-06-04T04:48:44.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:44 smithi098 ceph-mon[35569]: purged_snaps scrub ok 2024-06-04T04:48:44.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:44 smithi098 ceph-mon[35569]: osd.2 [v2:172.21.15.112:6800/1328098370,v1:172.21.15.112:6801/1328098370] boot 2024-06-04T04:48:44.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:44 smithi098 ceph-mon[35569]: osdmap e18: 3 total, 3 up, 3 in 2024-06-04T04:48:44.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:44 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:48:44.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:44 smithi098 ceph-mon[35569]: pgmap v81: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:44.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:44 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-04T04:48:44.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:44 smithi098 ceph-mon[35569]: Detected new or changed devices on smithi112 2024-06-04T04:48:44.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:44 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:44.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:44 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:44.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:44 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:48:44.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:44 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi112 to 17088M 2024-06-04T04:48:44.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:44 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:44.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:44 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:44.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:44 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:48:44.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:44 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:45.818 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:45 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-04T04:48:45.818 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:45 smithi070 ceph-mon[32631]: osdmap e19: 3 total, 3 up, 3 in 2024-06-04T04:48:45.818 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:45 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-04T04:48:45.902 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 04:48:45 smithi098 sudo[43850]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-04T04:48:45.902 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 04:48:45 smithi098 sudo[43850]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-04T04:48:45.902 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 04:48:45 smithi098 sudo[43850]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-04T04:48:45.903 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 04:48:45 smithi098 sudo[43850]: pam_unix(sudo:session): session closed for user root 2024-06-04T04:48:45.903 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:45 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-04T04:48:45.903 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:45 smithi098 ceph-mon[35569]: osdmap e19: 3 total, 3 up, 3 in 2024-06-04T04:48:45.903 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:45 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-04T04:48:45.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:45 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-04T04:48:45.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:45 smithi112 ceph-mon[36453]: osdmap e19: 3 total, 3 up, 3 in 2024-06-04T04:48:45.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:45 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-04T04:48:46.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.042+0000 7f2851b96700 1 -- 172.21.15.70:0/3823943672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f284c105210 msgr2=0x7f284c10bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:48:46.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.042+0000 7f2851b96700 1 --2- 172.21.15.70:0/3823943672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f284c105210 0x7f284c10bac0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f284400b0a0 tx=0x7f284400b3b0 comp rx=0 tx=0).stop 2024-06-04T04:48:46.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.043+0000 7f2851b96700 1 -- 172.21.15.70:0/3823943672 shutdown_connections 2024-06-04T04:48:46.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.043+0000 7f2851b96700 1 --2- 172.21.15.70:0/3823943672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f284c105210 0x7f284c10bac0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:46.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.043+0000 7f2851b96700 1 --2- 172.21.15.70:0/3823943672 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f284c101890 0x7f284c101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:46.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.043+0000 7f2851b96700 1 --2- 172.21.15.70:0/3823943672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f284c104870 0x7f284c104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:46.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.043+0000 7f2851b96700 1 -- 172.21.15.70:0/3823943672 >> 172.21.15.70:0/3823943672 conn(0x7f284c0fd1f0 msgr2=0x7f284c0ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:48:46.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.043+0000 7f2851b96700 1 -- 172.21.15.70:0/3823943672 shutdown_connections 2024-06-04T04:48:46.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.043+0000 7f2851b96700 1 -- 172.21.15.70:0/3823943672 wait complete. 2024-06-04T04:48:46.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.043+0000 7f2851b96700 1 Processor -- start 2024-06-04T04:48:46.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f2851b96700 1 -- start start 2024-06-04T04:48:46.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f2851b96700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f284c101890 0x7f284c1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:48:46.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f2851b96700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f284c104870 0x7f284c199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:48:46.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f2851b96700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f284c105210 0x7f284c1a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:48:46.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f2851b96700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f284c10ef80 con 0x7f284c105210 2024-06-04T04:48:46.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f2851b96700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f284c10ee00 con 0x7f284c104870 2024-06-04T04:48:46.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f2851b96700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f284c10f100 con 0x7f284c101890 2024-06-04T04:48:46.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f284bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f284c105210 0x7f284c1a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:48:46.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f284bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f284c105210 0x7f284c1a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34328/0 (socket says 172.21.15.70:34328) 2024-06-04T04:48:46.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f284bfff700 1 -- 172.21.15.70:0/103038855 learned_addr learned my addr 172.21.15.70:0/103038855 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:48:46.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f284b7fe700 1 --2- 172.21.15.70:0/103038855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f284c101890 0x7f284c1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:48:46.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f284affd700 1 --2- 172.21.15.70:0/103038855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f284c104870 0x7f284c199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:48:46.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f284bfff700 1 -- 172.21.15.70:0/103038855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f284c101890 msgr2=0x7f284c1995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:48:46.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f284bfff700 1 --2- 172.21.15.70:0/103038855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f284c101890 0x7f284c1995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:46.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f284bfff700 1 -- 172.21.15.70:0/103038855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f284c104870 msgr2=0x7f284c199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:48:46.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f284bfff700 1 --2- 172.21.15.70:0/103038855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f284c104870 0x7f284c199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:48:46.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f284bfff700 1 -- 172.21.15.70:0/103038855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f284c1a1240 con 0x7f284c105210 2024-06-04T04:48:46.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f284bfff700 1 --2- 172.21.15.70:0/103038855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f284c105210 0x7f284c1a0ba0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f284400b070 tx=0x7f284400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:48:46.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.044+0000 7f2848ff9700 1 -- 172.21.15.70:0/103038855 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284401f030 con 0x7f284c105210 2024-06-04T04:48:46.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.045+0000 7f2848ff9700 1 -- 172.21.15.70:0/103038855 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2844007bc0 con 0x7f284c105210 2024-06-04T04:48:46.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.045+0000 7f2851b96700 1 -- 172.21.15.70:0/103038855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f284c1a14d0 con 0x7f284c105210 2024-06-04T04:48:46.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.045+0000 7f2848ff9700 1 -- 172.21.15.70:0/103038855 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844005040 con 0x7f284c105210 2024-06-04T04:48:46.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.045+0000 7f2851b96700 1 -- 172.21.15.70:0/103038855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f284c1a1a00 con 0x7f284c105210 2024-06-04T04:48:46.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.045+0000 7f2851b96700 1 -- 172.21.15.70:0/103038855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f284c06a230 con 0x7f284c105210 2024-06-04T04:48:46.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.047+0000 7f2848ff9700 1 -- 172.21.15.70:0/103038855 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7f2844003a00 con 0x7f284c105210 2024-06-04T04:48:46.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.048+0000 7f2848ff9700 1 --2- 172.21.15.70:0/103038855 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f28340b27f0 0x7f28340b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:48:46.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.048+0000 7f284b7fe700 1 --2- 172.21.15.70:0/103038855 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f28340b27f0 0x7f28340b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:48:46.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.048+0000 7f2848ff9700 1 -- 172.21.15.70:0/103038855 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7f284400f020 con 0x7f284c105210 2024-06-04T04:48:46.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.048+0000 7f284b7fe700 1 --2- 172.21.15.70:0/103038855 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f28340b27f0 0x7f28340b4cb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f283c006fd0 tx=0x7f283c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:48:46.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.052+0000 7f2848ff9700 1 -- 172.21.15.70:0/103038855 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2844029020 con 0x7f284c105210 2024-06-04T04:48:46.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 sudo[51640]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-04T04:48:46.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 sudo[51640]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-04T04:48:46.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 sudo[51640]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-04T04:48:46.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 sudo[51640]: pam_unix(sudo:session): session closed for user root 2024-06-04T04:48:46.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 sudo[51645]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-04T04:48:46.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 sudo[51645]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-04T04:48:46.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 sudo[51645]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-04T04:48:46.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 sudo[51645]: pam_unix(sudo:session): session closed for user root 2024-06-04T04:48:46.110 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 04:48:45 smithi070 sudo[51629]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-04T04:48:46.110 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 04:48:45 smithi070 sudo[51629]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-04T04:48:46.110 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 04:48:45 smithi070 sudo[51629]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-04T04:48:46.110 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 04:48:45 smithi070 sudo[51629]: pam_unix(sudo:session): session closed for user root 2024-06-04T04:48:46.110 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 04:48:45 smithi070 sudo[51635]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-04T04:48:46.110 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 04:48:45 smithi070 sudo[51635]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-04T04:48:46.110 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 04:48:45 smithi070 sudo[51635]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-04T04:48:46.110 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 04:48:45 smithi070 sudo[51635]: pam_unix(sudo:session): session closed for user root 2024-06-04T04:48:46.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 sudo[43856]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-04T04:48:46.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 sudo[43856]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-04T04:48:46.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 sudo[43856]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-04T04:48:46.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 sudo[43856]: pam_unix(sudo:session): session closed for user root 2024-06-04T04:48:46.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 sudo[43859]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-04T04:48:46.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 sudo[43859]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-04T04:48:46.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 sudo[43859]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-04T04:48:46.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 sudo[43859]: pam_unix(sudo:session): session closed for user root 2024-06-04T04:48:46.246 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 04:48:45 smithi098 sudo[43853]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-04T04:48:46.246 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 04:48:45 smithi098 sudo[43853]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-04T04:48:46.246 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 04:48:45 smithi098 sudo[43853]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-04T04:48:46.246 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 04:48:45 smithi098 sudo[43853]: pam_unix(sudo:session): session closed for user root 2024-06-04T04:48:46.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:46.267+0000 7f2851b96700 1 -- 172.21.15.70:0/103038855 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f284c063980 con 0x7f28340b27f0 2024-06-04T04:48:46.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 sudo[45579]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-04T04:48:46.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 sudo[45579]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-04T04:48:46.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 sudo[45579]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-04T04:48:46.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 sudo[45579]: pam_unix(sudo:session): session closed for user root 2024-06-04T04:48:46.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 sudo[45582]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-04T04:48:46.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 sudo[45582]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-04T04:48:46.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 sudo[45582]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-04T04:48:46.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 sudo[45582]: pam_unix(sudo:session): session closed for user root 2024-06-04T04:48:46.437 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 04:48:45 smithi112 sudo[45573]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-04T04:48:46.437 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 04:48:45 smithi112 sudo[45573]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-04T04:48:46.437 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 04:48:45 smithi112 sudo[45573]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-04T04:48:46.437 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 04:48:45 smithi112 sudo[45573]: pam_unix(sudo:session): session closed for user root 2024-06-04T04:48:46.437 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 04:48:45 smithi112 sudo[45576]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-04T04:48:46.437 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 04:48:45 smithi112 sudo[45576]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-04T04:48:46.437 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 04:48:45 smithi112 sudo[45576]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-04T04:48:46.437 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 04:48:45 smithi112 sudo[45576]: pam_unix(sudo:session): session closed for user root 2024-06-04T04:48:46.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-04T04:48:46.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: osdmap e20: 3 total, 3 up, 3 in 2024-06-04T04:48:46.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:46.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:46.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:46.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:46.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-04T04:48:46.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-04T04:48:46.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:48:46.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:48:46.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:48:46.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:48:46.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-04T04:48:46.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:48:46.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:48:46.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-04T04:48:46.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-04T04:48:46.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:48:46.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:48:46.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:48:46.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-04T04:48:46.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:48:46.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:48:46.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:48:46.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:46.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-04T04:48:46.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: osdmap e20: 3 total, 3 up, 3 in 2024-06-04T04:48:46.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:46.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:46.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:46.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:46.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-04T04:48:46.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-04T04:48:46.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:48:46.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:48:46.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:48:46.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:48:46.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-04T04:48:46.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:48:46.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:48:46.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-04T04:48:46.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-04T04:48:46.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:48:46.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:48:46.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:48:46.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-04T04:48:46.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:48:46.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:48:46.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:48:46.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:47.002 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-04T04:48:47.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: osdmap e20: 3 total, 3 up, 3 in 2024-06-04T04:48:47.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:47.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:47.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:47.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:48:47.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-04T04:48:47.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-04T04:48:47.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:48:47.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:48:47.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:48:47.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:48:47.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-04T04:48:47.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:48:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:48:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-04T04:48:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-04T04:48:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:48:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:48:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:48:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-04T04:48:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:48:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:48:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:48:47.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:47 smithi112 ceph-mon[36453]: osdmap e21: 3 total, 3 up, 3 in 2024-06-04T04:48:47.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:47 smithi098 ceph-mon[35569]: osdmap e21: 3 total, 3 up, 3 in 2024-06-04T04:48:48.069 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:47 smithi070 ceph-mon[32631]: osdmap e21: 3 total, 3 up, 3 in 2024-06-04T04:48:48.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:48:48.664+0000 7f2848ff9700 1 -- 172.21.15.70:0/103038855 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f28440a7490 con 0x7f284c105210 2024-06-04T04:48:48.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:48 smithi112 ceph-mon[36453]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:48.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:48 smithi098 ceph-mon[35569]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:49.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:48 smithi070 ceph-mon[32631]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:49.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:49 smithi070 ceph-mon[32631]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-06-04T04:48:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:49 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1270049286' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8fbc440-bf70-42d3-b01e-777ccfa7900b"}]: dispatch 2024-06-04T04:48:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:49 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1270049286' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8fbc440-bf70-42d3-b01e-777ccfa7900b"}]': finished 2024-06-04T04:48:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:49 smithi070 ceph-mon[32631]: osdmap e22: 4 total, 3 up, 4 in 2024-06-04T04:48:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:49 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:48:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:49 smithi112 ceph-mon[36453]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-06-04T04:48:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:49 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1270049286' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8fbc440-bf70-42d3-b01e-777ccfa7900b"}]: dispatch 2024-06-04T04:48:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:49 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1270049286' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8fbc440-bf70-42d3-b01e-777ccfa7900b"}]': finished 2024-06-04T04:48:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:49 smithi112 ceph-mon[36453]: osdmap e22: 4 total, 3 up, 4 in 2024-06-04T04:48:49.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:49 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:48:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:49 smithi098 ceph-mon[35569]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-06-04T04:48:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:49 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1270049286' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8fbc440-bf70-42d3-b01e-777ccfa7900b"}]: dispatch 2024-06-04T04:48:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:49 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1270049286' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8fbc440-bf70-42d3-b01e-777ccfa7900b"}]': finished 2024-06-04T04:48:49.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:49 smithi098 ceph-mon[35569]: osdmap e22: 4 total, 3 up, 4 in 2024-06-04T04:48:49.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:49 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:48:50.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:50 smithi112 ceph-mon[36453]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:50.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:50 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1974418177' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:48:50.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:50 smithi098 ceph-mon[35569]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:50.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:50 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1974418177' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:48:51.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:50 smithi070 ceph-mon[32631]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:51.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:50 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1974418177' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:48:52.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:52 smithi098 ceph-mon[35569]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:53.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:52 smithi070 ceph-mon[32631]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:53.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:52 smithi112 ceph-mon[36453]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:54.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:54 smithi070 ceph-mon[32631]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:54.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:54 smithi098 ceph-mon[35569]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:55.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:54 smithi112 ceph-mon[36453]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:56.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:56 smithi070 ceph-mon[32631]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:56.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-04T04:48:56.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:56.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:56 smithi070 ceph-mon[32631]: Deploying daemon osd.3 on smithi070 2024-06-04T04:48:56.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:56 smithi098 ceph-mon[35569]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:56.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:56 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-04T04:48:56.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:56 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:56.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:56 smithi098 ceph-mon[35569]: Deploying daemon osd.3 on smithi070 2024-06-04T04:48:57.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:56 smithi112 ceph-mon[36453]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:57.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:56 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-04T04:48:57.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:56 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:48:57.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:56 smithi112 ceph-mon[36453]: Deploying daemon osd.3 on smithi070 2024-06-04T04:48:58.988 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:48:58 smithi112 ceph-mon[36453]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:58.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:48:58 smithi098 ceph-mon[35569]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:48:59.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:48:58 smithi070 ceph-mon[32631]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:49:00.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:00 smithi098 ceph-mon[35569]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:49:01.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:00 smithi070 ceph-mon[32631]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:49:01.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:00 smithi112 ceph-mon[36453]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:49:03.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:03 smithi112 ceph-mon[36453]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:49:03.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:03 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:03.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:03 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:03.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:03 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:03.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:03 smithi098 ceph-mon[35569]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:49:03.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:03 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:03.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:03 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:03.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:03 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:03.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:03 smithi070 ceph-mon[32631]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:49:03.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:03 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:03.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:03 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:03.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:03 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:05.336 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:05 smithi070 ceph-mon[32631]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:49:05.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:49:05.386+0000 7f2848ff9700 1 -- 172.21.15.70:0/103038855 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f284c063980 con 0x7f28340b27f0 2024-06-04T04:49:05.389 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 3 on host 'smithi070' 2024-06-04T04:49:05.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:49:05.389+0000 7f2851b96700 1 -- 172.21.15.70:0/103038855 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f28340b27f0 msgr2=0x7f28340b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:49:05.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:49:05.389+0000 7f2851b96700 1 --2- 172.21.15.70:0/103038855 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f28340b27f0 0x7f28340b4cb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f283c006fd0 tx=0x7f283c008040 comp rx=0 tx=0).stop 2024-06-04T04:49:05.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:49:05.389+0000 7f2851b96700 1 -- 172.21.15.70:0/103038855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f284c105210 msgr2=0x7f284c1a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:49:05.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:49:05.389+0000 7f2851b96700 1 --2- 172.21.15.70:0/103038855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f284c105210 0x7f284c1a0ba0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f284400b070 tx=0x7f284400bd70 comp rx=0 tx=0).stop 2024-06-04T04:49:05.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:49:05.389+0000 7f2851b96700 1 -- 172.21.15.70:0/103038855 shutdown_connections 2024-06-04T04:49:05.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:49:05.389+0000 7f2851b96700 1 --2- 172.21.15.70:0/103038855 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f28340b27f0 0x7f28340b4cb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:05.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:49:05.389+0000 7f2851b96700 1 --2- 172.21.15.70:0/103038855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f284c105210 0x7f284c1a0ba0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:05.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:49:05.389+0000 7f2851b96700 1 --2- 172.21.15.70:0/103038855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f284c101890 0x7f284c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:05.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:49:05.389+0000 7f2851b96700 1 --2- 172.21.15.70:0/103038855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f284c104870 0x7f284c199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:05.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:49:05.389+0000 7f2851b96700 1 -- 172.21.15.70:0/103038855 >> 172.21.15.70:0/103038855 conn(0x7f284c0fd1f0 msgr2=0x7f284c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:49:05.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:49:05.389+0000 7f2851b96700 1 -- 172.21.15.70:0/103038855 shutdown_connections 2024-06-04T04:49:05.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:49:05.389+0000 7f2851b96700 1 -- 172.21.15.70:0/103038855 wait complete. 2024-06-04T04:49:05.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:05 smithi112 ceph-mon[36453]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:49:05.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:05 smithi098 ceph-mon[35569]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:49:06.059 DEBUG:teuthology.orchestra.run.smithi070:osd.3> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.3.service 2024-06-04T04:49:06.062 INFO:tasks.cephadm:Deploying osd.4 on smithi098 with /dev/vg_nvme/lv_3... 2024-06-04T04:49:06.062 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-04T04:49:06.307 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.b/config 2024-06-04T04:49:06.377 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 04:49:06 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-3[53903]: 2024-06-04T04:49:06.109+0000 7f6736bb97c0 -1 osd.3 0 log_to_monitors true 2024-06-04T04:49:06.682 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:06 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:06.682 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:06 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:06.682 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:06 smithi098 ceph-mon[35569]: from='osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-04T04:49:06.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:06 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:06.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:06 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:06.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:06 smithi112 ceph-mon[36453]: from='osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-04T04:49:06.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:06 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:06.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:06 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:06.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:06 smithi070 ceph-mon[32631]: from='osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-04T04:49:07.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:07 smithi070 ceph-mon[32631]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:49:07.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:07 smithi070 ceph-mon[32631]: from='osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-04T04:49:07.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:07 smithi070 ceph-mon[32631]: osdmap e23: 4 total, 3 up, 4 in 2024-06-04T04:49:07.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:07 smithi070 ceph-mon[32631]: from='osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-06-04T04:49:07.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:07 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:07.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:07 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:07.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:07 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:07 smithi112 ceph-mon[36453]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:49:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:07 smithi112 ceph-mon[36453]: from='osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-04T04:49:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:07 smithi112 ceph-mon[36453]: osdmap e23: 4 total, 3 up, 4 in 2024-06-04T04:49:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:07 smithi112 ceph-mon[36453]: from='osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-06-04T04:49:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:07 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:07 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:07 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:07.690 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:07 smithi098 ceph-mon[35569]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:49:07.690 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:07 smithi098 ceph-mon[35569]: from='osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-04T04:49:07.690 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:07 smithi098 ceph-mon[35569]: osdmap e23: 4 total, 3 up, 4 in 2024-06-04T04:49:07.690 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:07 smithi098 ceph-mon[35569]: from='osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-06-04T04:49:07.690 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:07 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:07.690 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:07 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:07.691 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:07 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:08.109 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 04:49:07 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-3[53903]: 2024-06-04T04:49:07.874+0000 7f672c21d700 -1 osd.3 0 waiting for initial osdmap 2024-06-04T04:49:08.109 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 04:49:07 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-3[53903]: 2024-06-04T04:49:07.879+0000 7f6724008700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-04T04:49:08.277 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-04T04:49:08.277 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-04T04:49:08.277 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-06-04T04:49:08.277 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-06-04T04:49:08.277 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113036 s, 928 MB/s 2024-06-04T04:49:08.277 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-06-04T04:49:08.510 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:08 smithi070 ceph-mon[32631]: from='osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-06-04T04:49:08.510 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:08 smithi070 ceph-mon[32631]: osdmap e24: 4 total, 3 up, 4 in 2024-06-04T04:49:08.510 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:08 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:08.511 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:08 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:08.511 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:08 smithi070 ceph-mon[32631]: osdmap e25: 4 total, 3 up, 4 in 2024-06-04T04:49:08.511 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:08 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:08.511 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:08 smithi070 ceph-mon[32631]: from='osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346]' entity='osd.3' 2024-06-04T04:49:08.572 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:08 smithi098 ceph-mon[35569]: from='osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-06-04T04:49:08.572 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:08 smithi098 ceph-mon[35569]: osdmap e24: 4 total, 3 up, 4 in 2024-06-04T04:49:08.572 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:08 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:08.573 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:08 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:08.573 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:08 smithi098 ceph-mon[35569]: osdmap e25: 4 total, 3 up, 4 in 2024-06-04T04:49:08.573 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:08 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:08.573 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:08 smithi098 ceph-mon[35569]: from='osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346]' entity='osd.3' 2024-06-04T04:49:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:08 smithi112 ceph-mon[36453]: from='osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-06-04T04:49:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:08 smithi112 ceph-mon[36453]: osdmap e24: 4 total, 3 up, 4 in 2024-06-04T04:49:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:08 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:08 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:08 smithi112 ceph-mon[36453]: osdmap e25: 4 total, 3 up, 4 in 2024-06-04T04:49:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:08 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:08 smithi112 ceph-mon[36453]: from='osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346]' entity='osd.3' 2024-06-04T04:49:08.914 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_3 2024-06-04T04:49:09.154 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.b/config 2024-06-04T04:49:09.548 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:09 smithi098 ceph-mon[35569]: purged_snaps scrub starts 2024-06-04T04:49:09.548 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:09 smithi098 ceph-mon[35569]: purged_snaps scrub ok 2024-06-04T04:49:09.548 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:09 smithi098 ceph-mon[35569]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:49:09.548 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:09 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:09.548 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:09 smithi098 ceph-mon[35569]: osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346] boot 2024-06-04T04:49:09.548 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:09 smithi098 ceph-mon[35569]: osdmap e26: 4 total, 4 up, 4 in 2024-06-04T04:49:09.548 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:09 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:09.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:09 smithi070 ceph-mon[32631]: purged_snaps scrub starts 2024-06-04T04:49:09.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:09 smithi070 ceph-mon[32631]: purged_snaps scrub ok 2024-06-04T04:49:09.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:09 smithi070 ceph-mon[32631]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:49:09.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:09.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:09 smithi070 ceph-mon[32631]: osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346] boot 2024-06-04T04:49:09.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:09 smithi070 ceph-mon[32631]: osdmap e26: 4 total, 4 up, 4 in 2024-06-04T04:49:09.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:09 smithi112 ceph-mon[36453]: purged_snaps scrub starts 2024-06-04T04:49:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:09 smithi112 ceph-mon[36453]: purged_snaps scrub ok 2024-06-04T04:49:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:09 smithi112 ceph-mon[36453]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-06-04T04:49:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:09 smithi112 ceph-mon[36453]: osd.3 [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346] boot 2024-06-04T04:49:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:09 smithi112 ceph-mon[36453]: osdmap e26: 4 total, 4 up, 4 in 2024-06-04T04:49:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:49:10.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.765+0000 7f74f81b9700 1 -- 172.21.15.98:0/1630909181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74f01024b0 msgr2=0x7f74f01028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:49:10.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.765+0000 7f74f81b9700 1 --2- 172.21.15.98:0/1630909181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74f01024b0 0x7f74f01028d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f74ec00b0d0 tx=0x7f74ec00b3e0 comp rx=0 tx=0).stop 2024-06-04T04:49:10.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.765+0000 7f74f81b9700 1 -- 172.21.15.98:0/1630909181 shutdown_connections 2024-06-04T04:49:10.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.765+0000 7f74f81b9700 1 --2- 172.21.15.98:0/1630909181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74f0104050 0x7f74f010a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:10.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.765+0000 7f74f81b9700 1 --2- 172.21.15.98:0/1630909181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74f01024b0 0x7f74f01028d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:10.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.765+0000 7f74f81b9700 1 --2- 172.21.15.98:0/1630909181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74f01036b0 0x7f74f0103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:10.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.765+0000 7f74f81b9700 1 -- 172.21.15.98:0/1630909181 >> 172.21.15.98:0/1630909181 conn(0x7f74f00fda70 msgr2=0x7f74f00ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:49:10.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.765+0000 7f74f81b9700 1 -- 172.21.15.98:0/1630909181 shutdown_connections 2024-06-04T04:49:10.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.765+0000 7f74f81b9700 1 -- 172.21.15.98:0/1630909181 wait complete. 2024-06-04T04:49:10.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.765+0000 7f74f81b9700 1 Processor -- start 2024-06-04T04:49:10.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.765+0000 7f74f81b9700 1 -- start start 2024-06-04T04:49:10.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.765+0000 7f74f81b9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74f01024b0 0x7f74f0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:49:10.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.765+0000 7f74f81b9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74f01036b0 0x7f74f0198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:49:10.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.765+0000 7f74f81b9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74f0104050 0x7f74f019fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:49:10.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.765+0000 7f74f81b9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f74f010de10 con 0x7f74f01036b0 2024-06-04T04:49:10.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.765+0000 7f74f81b9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f74f010dc90 con 0x7f74f0104050 2024-06-04T04:49:10.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.765+0000 7f74f81b9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f74f010df90 con 0x7f74f01024b0 2024-06-04T04:49:10.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.766+0000 7f74f6756700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74f0104050 0x7f74f019fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:49:10.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.766+0000 7f74f6756700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74f0104050 0x7f74f019fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58894/0 (socket says 172.21.15.98:58894) 2024-06-04T04:49:10.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.766+0000 7f74f6756700 1 -- 172.21.15.98:0/1865485165 learned_addr learned my addr 172.21.15.98:0/1865485165 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-04T04:49:10.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.766+0000 7f74f5f55700 1 --2- 172.21.15.98:0/1865485165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74f01024b0 0x7f74f0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:49:10.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.766+0000 7f74f5754700 1 --2- 172.21.15.98:0/1865485165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74f01036b0 0x7f74f0198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:49:10.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.766+0000 7f74f6756700 1 -- 172.21.15.98:0/1865485165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74f01024b0 msgr2=0x7f74f0198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:49:10.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.766+0000 7f74f6756700 1 --2- 172.21.15.98:0/1865485165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74f01024b0 0x7f74f0198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:10.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.766+0000 7f74f6756700 1 -- 172.21.15.98:0/1865485165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74f01036b0 msgr2=0x7f74f0198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:49:10.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.766+0000 7f74f6756700 1 --2- 172.21.15.98:0/1865485165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74f01036b0 0x7f74f0198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:10.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.766+0000 7f74f6756700 1 -- 172.21.15.98:0/1865485165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74f01a01b0 con 0x7f74f0104050 2024-06-04T04:49:10.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.766+0000 7f74f6756700 1 --2- 172.21.15.98:0/1865485165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74f0104050 0x7f74f019fb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f74e800c900 tx=0x7f74e800cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:49:10.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.766+0000 7f74e6ffd700 1 -- 172.21.15.98:0/1865485165 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e80078d0 con 0x7f74f0104050 2024-06-04T04:49:10.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.766+0000 7f74f81b9700 1 -- 172.21.15.98:0/1865485165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74f006bcf0 con 0x7f74f0104050 2024-06-04T04:49:10.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.766+0000 7f74e6ffd700 1 -- 172.21.15.98:0/1865485165 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f74e800f450 con 0x7f74f0104050 2024-06-04T04:49:10.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.766+0000 7f74e6ffd700 1 -- 172.21.15.98:0/1865485165 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e8018660 con 0x7f74f0104050 2024-06-04T04:49:10.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.766+0000 7f74f81b9700 1 -- 172.21.15.98:0/1865485165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74f006c230 con 0x7f74f0104050 2024-06-04T04:49:10.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.767+0000 7f74f81b9700 1 -- 172.21.15.98:0/1865485165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74f006a230 con 0x7f74f0104050 2024-06-04T04:49:10.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.769+0000 7f74e6ffd700 1 -- 172.21.15.98:0/1865485165 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f74e8007a70 con 0x7f74f0104050 2024-06-04T04:49:10.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.769+0000 7f74e6ffd700 1 --2- 172.21.15.98:0/1865485165 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f74dc0b2880 0x7f74dc0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:49:10.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.769+0000 7f74e6ffd700 1 -- 172.21.15.98:0/1865485165 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7f74e80e0ff0 con 0x7f74f0104050 2024-06-04T04:49:10.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.769+0000 7f74f5f55700 1 --2- 172.21.15.98:0/1865485165 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f74dc0b2880 0x7f74dc0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:49:10.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.770+0000 7f74f5f55700 1 --2- 172.21.15.98:0/1865485165 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f74dc0b2880 0x7f74dc0b4d40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f74ec009fd0 tx=0x7f74ec01c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:49:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.773+0000 7f74e6ffd700 1 -- 172.21.15.98:0/1865485165 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f74e8054df0 con 0x7f74f0104050 2024-06-04T04:49:10.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:10 smithi112 ceph-mon[36453]: osdmap e27: 4 total, 4 up, 4 in 2024-06-04T04:49:10.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:10 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:10.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:10 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:10.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:10 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:49:10.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:10 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:49:10.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:10 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:10.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:10 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:10.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:10 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:49:10.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:10 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:10.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:10 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:10.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:10 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:10.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:10 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:49:10.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:10 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:10.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:10.982+0000 7f74f81b9700 1 -- 172.21.15.98:0/1865485165 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f74f006c4f0 con 0x7f74dc0b2880 2024-06-04T04:49:10.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:10 smithi098 ceph-mon[35569]: osdmap e27: 4 total, 4 up, 4 in 2024-06-04T04:49:10.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:10 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:10.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:10 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:10.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:10 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:49:10.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:10 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:49:10.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:10 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:10.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:10 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:10.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:10 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:49:10.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:10 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:10.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:10 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:10.986 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:10 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:10.986 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:10 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:49:10.986 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:10 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:11.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:10 smithi070 ceph-mon[32631]: osdmap e27: 4 total, 4 up, 4 in 2024-06-04T04:49:11.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:10 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:11.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:10 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:11.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:10 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:49:11.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:10 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:49:11.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:10 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:11.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:10 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:11.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:10 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:49:11.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:10 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:11.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:10 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:11.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:10 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:11.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:10 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:49:11.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:10 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:11.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:11 smithi098 ceph-mon[35569]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:11.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:11 smithi098 ceph-mon[35569]: Detected new or changed devices on smithi070 2024-06-04T04:49:11.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:11 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi070 to 8544M 2024-06-04T04:49:11.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:11 smithi098 ceph-mon[35569]: osdmap e28: 4 total, 4 up, 4 in 2024-06-04T04:49:11.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:11 smithi098 ceph-mon[35569]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:49:11.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:11 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:49:11.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:11 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:49:11.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:11 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:11 smithi112 ceph-mon[36453]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:11 smithi112 ceph-mon[36453]: Detected new or changed devices on smithi070 2024-06-04T04:49:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:11 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi070 to 8544M 2024-06-04T04:49:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:11 smithi112 ceph-mon[36453]: osdmap e28: 4 total, 4 up, 4 in 2024-06-04T04:49:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:11 smithi112 ceph-mon[36453]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:49:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:11 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:49:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:11 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:49:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:11 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:12.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:11 smithi070 ceph-mon[32631]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:12.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:11 smithi070 ceph-mon[32631]: Detected new or changed devices on smithi070 2024-06-04T04:49:12.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:11 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi070 to 8544M 2024-06-04T04:49:12.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:11 smithi070 ceph-mon[32631]: osdmap e28: 4 total, 4 up, 4 in 2024-06-04T04:49:12.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:11 smithi070 ceph-mon[32631]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:49:12.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:11 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:49:12.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:11 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:49:12.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:11 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:12.903 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:12 smithi098 ceph-mon[35569]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:12.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:12 smithi112 ceph-mon[36453]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:13.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:12 smithi070 ceph-mon[32631]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:14.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:14 smithi098 ceph-mon[35569]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:14.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:14 smithi098 ceph-mon[35569]: from='client.? 172.21.15.98:0/1093997884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "549c8c70-8be8-46ae-bcb6-0fde32b40cd9"}]: dispatch 2024-06-04T04:49:14.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:14 smithi098 ceph-mon[35569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "549c8c70-8be8-46ae-bcb6-0fde32b40cd9"}]: dispatch 2024-06-04T04:49:14.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:14 smithi098 ceph-mon[35569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "549c8c70-8be8-46ae-bcb6-0fde32b40cd9"}]': finished 2024-06-04T04:49:14.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:14 smithi098 ceph-mon[35569]: osdmap e29: 5 total, 4 up, 5 in 2024-06-04T04:49:14.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:14 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:14.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:14 smithi098 ceph-mon[35569]: from='client.? 172.21.15.98:0/326711793' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:49:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:14 smithi070 ceph-mon[32631]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:14 smithi070 ceph-mon[32631]: from='client.? 172.21.15.98:0/1093997884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "549c8c70-8be8-46ae-bcb6-0fde32b40cd9"}]: dispatch 2024-06-04T04:49:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:14 smithi070 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "549c8c70-8be8-46ae-bcb6-0fde32b40cd9"}]: dispatch 2024-06-04T04:49:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:14 smithi070 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "549c8c70-8be8-46ae-bcb6-0fde32b40cd9"}]': finished 2024-06-04T04:49:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:14 smithi070 ceph-mon[32631]: osdmap e29: 5 total, 4 up, 5 in 2024-06-04T04:49:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:14 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:14 smithi070 ceph-mon[32631]: from='client.? 172.21.15.98:0/326711793' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:49:15.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:14 smithi112 ceph-mon[36453]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:15.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:14 smithi112 ceph-mon[36453]: from='client.? 172.21.15.98:0/1093997884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "549c8c70-8be8-46ae-bcb6-0fde32b40cd9"}]: dispatch 2024-06-04T04:49:15.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:14 smithi112 ceph-mon[36453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "549c8c70-8be8-46ae-bcb6-0fde32b40cd9"}]: dispatch 2024-06-04T04:49:15.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:14 smithi112 ceph-mon[36453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "549c8c70-8be8-46ae-bcb6-0fde32b40cd9"}]': finished 2024-06-04T04:49:15.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:14 smithi112 ceph-mon[36453]: osdmap e29: 5 total, 4 up, 5 in 2024-06-04T04:49:15.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:14 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:15.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:14 smithi112 ceph-mon[36453]: from='client.? 172.21.15.98:0/326711793' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:49:16.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:16 smithi098 ceph-mon[35569]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-06-04T04:49:17.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:16 smithi070 ceph-mon[32631]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-06-04T04:49:17.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:16 smithi112 ceph-mon[36453]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-06-04T04:49:18.957 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:18 smithi098 ceph-mon[35569]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-04T04:49:19.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:18 smithi070 ceph-mon[32631]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-04T04:49:19.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:18 smithi112 ceph-mon[36453]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-04T04:49:20.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:20 smithi098 ceph-mon[35569]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-06-04T04:49:21.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:20 smithi070 ceph-mon[32631]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-06-04T04:49:21.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:20 smithi112 ceph-mon[36453]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-06-04T04:49:21.947 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:21 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-04T04:49:21.947 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:21 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:21.947 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:21 smithi098 ceph-mon[35569]: Deploying daemon osd.4 on smithi098 2024-06-04T04:49:22.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:21 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-04T04:49:22.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:21 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:22.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:21 smithi070 ceph-mon[32631]: Deploying daemon osd.4 on smithi098 2024-06-04T04:49:22.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:21 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-04T04:49:22.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:21 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:22.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:21 smithi112 ceph-mon[36453]: Deploying daemon osd.4 on smithi098 2024-06-04T04:49:22.919 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:22 smithi098 ceph-mon[35569]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-04T04:49:23.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:22 smithi070 ceph-mon[32631]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-04T04:49:23.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:22 smithi112 ceph-mon[36453]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-04T04:49:24.939 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:24 smithi098 ceph-mon[35569]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-04T04:49:25.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:24 smithi070 ceph-mon[32631]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-04T04:49:25.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:24 smithi112 ceph-mon[36453]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-04T04:49:26.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:26 smithi098 ceph-mon[35569]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-06-04T04:49:26.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:26.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:27.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:26 smithi070 ceph-mon[32631]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-06-04T04:49:27.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:27.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:27.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:26 smithi112 ceph-mon[36453]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-06-04T04:49:27.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:27.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:27.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:27 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:28.087 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:27 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:28.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:27 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:28.988 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:28 smithi098 ceph-mon[35569]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:29.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:28 smithi070 ceph-mon[32631]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:29.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:28 smithi112 ceph-mon[36453]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:30.702 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 4 on host 'smithi098' 2024-06-04T04:49:30.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:30.700+0000 7f74e6ffd700 1 -- 172.21.15.98:0/1865485165 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f74f006c4f0 con 0x7f74dc0b2880 2024-06-04T04:49:30.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:30.702+0000 7f74f81b9700 1 -- 172.21.15.98:0/1865485165 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f74dc0b2880 msgr2=0x7f74dc0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:49:30.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:30.702+0000 7f74f81b9700 1 --2- 172.21.15.98:0/1865485165 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f74dc0b2880 0x7f74dc0b4d40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f74ec009fd0 tx=0x7f74ec01c040 comp rx=0 tx=0).stop 2024-06-04T04:49:30.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:30.702+0000 7f74f81b9700 1 -- 172.21.15.98:0/1865485165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74f0104050 msgr2=0x7f74f019fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:49:30.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:30.702+0000 7f74f81b9700 1 --2- 172.21.15.98:0/1865485165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74f0104050 0x7f74f019fb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f74e800c900 tx=0x7f74e800cc10 comp rx=0 tx=0).stop 2024-06-04T04:49:30.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:30.702+0000 7f74f81b9700 1 -- 172.21.15.98:0/1865485165 shutdown_connections 2024-06-04T04:49:30.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:30.702+0000 7f74f81b9700 1 --2- 172.21.15.98:0/1865485165 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f74dc0b2880 0x7f74dc0b4d40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:30.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:30.702+0000 7f74f81b9700 1 --2- 172.21.15.98:0/1865485165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74f0104050 0x7f74f019fb10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:30.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:30.702+0000 7f74f81b9700 1 --2- 172.21.15.98:0/1865485165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74f01024b0 0x7f74f0198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:30.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:30.702+0000 7f74f81b9700 1 --2- 172.21.15.98:0/1865485165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74f01036b0 0x7f74f0198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:30.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:30.702+0000 7f74f81b9700 1 -- 172.21.15.98:0/1865485165 >> 172.21.15.98:0/1865485165 conn(0x7f74f00fda70 msgr2=0x7f74f0106b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:49:30.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:30.702+0000 7f74f81b9700 1 -- 172.21.15.98:0/1865485165 shutdown_connections 2024-06-04T04:49:30.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:49:30.702+0000 7f74f81b9700 1 -- 172.21.15.98:0/1865485165 wait complete. 2024-06-04T04:49:30.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:30 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:30.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:30 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:30.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:30 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:30.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:30 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:49:30.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:30 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:30.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:30 smithi112 ceph-mon[36453]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:30.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:30 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:30.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:30 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:30.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:30 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:30.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:30 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:49:30.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:30 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:30.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:30 smithi098 ceph-mon[35569]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:31.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:30 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:31.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:30 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:31.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:30 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:31.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:30 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:49:31.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:30 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:31.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:30 smithi070 ceph-mon[32631]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:31.361 DEBUG:teuthology.orchestra.run.smithi098:osd.4> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.4.service 2024-06-04T04:49:31.363 INFO:tasks.cephadm:Deploying osd.5 on smithi112 with /dev/vg_nvme/lv_3... 2024-06-04T04:49:31.364 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-04T04:49:31.608 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.c/config 2024-06-04T04:49:31.895 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:31 smithi112 ceph-mon[36453]: from='osd.4 [v2:172.21.15.98:6808/3652775814,v1:172.21.15.98:6809/3652775814]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-04T04:49:31.895 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:31 smithi112 ceph-mon[36453]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-04T04:49:31.895 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:31 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:31.895 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:31 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:31.895 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:31 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:31.948 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:31 smithi098 ceph-mon[35569]: from='osd.4 [v2:172.21.15.98:6808/3652775814,v1:172.21.15.98:6809/3652775814]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-04T04:49:31.948 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:31 smithi098 ceph-mon[35569]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-04T04:49:31.948 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:31 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:31.948 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:31 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:31.949 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:31 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:32.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:31 smithi070 ceph-mon[32631]: from='osd.4 [v2:172.21.15.98:6808/3652775814,v1:172.21.15.98:6809/3652775814]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-04T04:49:32.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:31 smithi070 ceph-mon[32631]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-04T04:49:32.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:31 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:32.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:31 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:32.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:31 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:32.912 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:32 smithi112 ceph-mon[36453]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-04T04:49:32.912 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:32 smithi112 ceph-mon[36453]: osdmap e30: 5 total, 4 up, 5 in 2024-06-04T04:49:32.912 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:32 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:32.912 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:32 smithi112 ceph-mon[36453]: from='osd.4 [v2:172.21.15.98:6808/3652775814,v1:172.21.15.98:6809/3652775814]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:49:32.912 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:32 smithi112 ceph-mon[36453]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:49:32.912 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:32 smithi112 ceph-mon[36453]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:32.925 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:32 smithi098 ceph-mon[35569]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-04T04:49:32.925 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:32 smithi098 ceph-mon[35569]: osdmap e30: 5 total, 4 up, 5 in 2024-06-04T04:49:32.925 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:32 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:32.925 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:32 smithi098 ceph-mon[35569]: from='osd.4 [v2:172.21.15.98:6808/3652775814,v1:172.21.15.98:6809/3652775814]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:49:32.925 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:32 smithi098 ceph-mon[35569]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:49:32.925 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:32 smithi098 ceph-mon[35569]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:33.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:32 smithi070 ceph-mon[32631]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-04T04:49:33.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:32 smithi070 ceph-mon[32631]: osdmap e30: 5 total, 4 up, 5 in 2024-06-04T04:49:33.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:32 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:33.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:32 smithi070 ceph-mon[32631]: from='osd.4 [v2:172.21.15.98:6808/3652775814,v1:172.21.15.98:6809/3652775814]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:49:33.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:32 smithi070 ceph-mon[32631]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:49:33.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:32 smithi070 ceph-mon[32631]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:33.245 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 04:49:33 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-4[46658]: 2024-06-04T04:49:33.136+0000 7f6cd39e7700 -1 osd.4 0 waiting for initial osdmap 2024-06-04T04:49:33.245 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 04:49:33 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-4[46658]: 2024-06-04T04:49:33.142+0000 7f6ccb7d2700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-04T04:49:33.612 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-04T04:49:33.612 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-04T04:49:33.612 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2024-06-04T04:49:33.612 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2024-06-04T04:49:33.612 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115334 s, 909 MB/s 2024-06-04T04:49:33.612 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2024-06-04T04:49:33.926 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:33 smithi112 ceph-mon[36453]: purged_snaps scrub starts 2024-06-04T04:49:33.926 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:33 smithi112 ceph-mon[36453]: purged_snaps scrub ok 2024-06-04T04:49:33.926 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:33 smithi112 ceph-mon[36453]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-06-04T04:49:33.926 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:33 smithi112 ceph-mon[36453]: osdmap e31: 5 total, 4 up, 5 in 2024-06-04T04:49:33.926 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:33 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:33.927 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:33 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:33.927 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:33 smithi112 ceph-mon[36453]: from='osd.4 ' entity='osd.4' 2024-06-04T04:49:33.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:33 smithi098 ceph-mon[35569]: purged_snaps scrub starts 2024-06-04T04:49:33.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:33 smithi098 ceph-mon[35569]: purged_snaps scrub ok 2024-06-04T04:49:33.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:33 smithi098 ceph-mon[35569]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-06-04T04:49:33.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:33 smithi098 ceph-mon[35569]: osdmap e31: 5 total, 4 up, 5 in 2024-06-04T04:49:33.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:33 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:33.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:33 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:33.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:33 smithi098 ceph-mon[35569]: from='osd.4 ' entity='osd.4' 2024-06-04T04:49:34.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:33 smithi070 ceph-mon[32631]: purged_snaps scrub starts 2024-06-04T04:49:34.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:33 smithi070 ceph-mon[32631]: purged_snaps scrub ok 2024-06-04T04:49:34.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:33 smithi070 ceph-mon[32631]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-06-04T04:49:34.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:33 smithi070 ceph-mon[32631]: osdmap e31: 5 total, 4 up, 5 in 2024-06-04T04:49:34.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:34.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:34.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:33 smithi070 ceph-mon[32631]: from='osd.4 ' entity='osd.4' 2024-06-04T04:49:34.256 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_3 2024-06-04T04:49:34.498 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.c/config 2024-06-04T04:49:34.922 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:34.922 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:34 smithi112 ceph-mon[36453]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:34.922 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:34 smithi112 ceph-mon[36453]: osd.4 [v2:172.21.15.98:6808/3652775814,v1:172.21.15.98:6809/3652775814] boot 2024-06-04T04:49:34.922 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:34 smithi112 ceph-mon[36453]: osdmap e32: 5 total, 5 up, 5 in 2024-06-04T04:49:34.922 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:34.922 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:34 smithi112 ceph-mon[36453]: Detected new or changed devices on smithi098 2024-06-04T04:49:34.922 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:34.922 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:34.923 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:49:34.923 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:49:34.923 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:34 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi098 to 8544M 2024-06-04T04:49:34.923 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:34.923 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:34.923 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:49:34.923 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:34.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:34 smithi098 ceph-mon[35569]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:34 smithi098 ceph-mon[35569]: osd.4 [v2:172.21.15.98:6808/3652775814,v1:172.21.15.98:6809/3652775814] boot 2024-06-04T04:49:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:34 smithi098 ceph-mon[35569]: osdmap e32: 5 total, 5 up, 5 in 2024-06-04T04:49:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:34 smithi098 ceph-mon[35569]: Detected new or changed devices on smithi098 2024-06-04T04:49:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:49:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:49:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:34 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi098 to 8544M 2024-06-04T04:49:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:49:34.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:34 smithi070 ceph-mon[32631]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-04T04:49:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:34 smithi070 ceph-mon[32631]: osd.4 [v2:172.21.15.98:6808/3652775814,v1:172.21.15.98:6809/3652775814] boot 2024-06-04T04:49:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:34 smithi070 ceph-mon[32631]: osdmap e32: 5 total, 5 up, 5 in 2024-06-04T04:49:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:49:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:34 smithi070 ceph-mon[32631]: Detected new or changed devices on smithi098 2024-06-04T04:49:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:49:35.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:49:35.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:34 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi098 to 8544M 2024-06-04T04:49:35.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:35.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:35.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:49:35.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:36.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.125+0000 7f36aa7f4700 1 -- 172.21.15.112:0/2654485049 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36a4104040 msgr2=0x7f36a410a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:49:36.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.125+0000 7f36aa7f4700 1 --2- 172.21.15.112:0/2654485049 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36a4104040 0x7f36a410a8f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f369c00b0a0 tx=0x7f369c00b3b0 comp rx=0 tx=0).stop 2024-06-04T04:49:36.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.125+0000 7f36aa7f4700 1 -- 172.21.15.112:0/2654485049 shutdown_connections 2024-06-04T04:49:36.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.125+0000 7f36aa7f4700 1 --2- 172.21.15.112:0/2654485049 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36a4104040 0x7f36a410a8f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:36.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.125+0000 7f36aa7f4700 1 --2- 172.21.15.112:0/2654485049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36a41024a0 0x7f36a41028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:36.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.125+0000 7f36aa7f4700 1 --2- 172.21.15.112:0/2654485049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a41036a0 0x7f36a4103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:36.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.125+0000 7f36aa7f4700 1 -- 172.21.15.112:0/2654485049 >> 172.21.15.112:0/2654485049 conn(0x7f36a40fda60 msgr2=0x7f36a40ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:49:36.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.125+0000 7f36aa7f4700 1 -- 172.21.15.112:0/2654485049 shutdown_connections 2024-06-04T04:49:36.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.125+0000 7f36aa7f4700 1 -- 172.21.15.112:0/2654485049 wait complete. 2024-06-04T04:49:36.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.126+0000 7f36aa7f4700 1 Processor -- start 2024-06-04T04:49:36.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.126+0000 7f36aa7f4700 1 -- start start 2024-06-04T04:49:36.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.126+0000 7f36aa7f4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a41024a0 0x7f36a4106430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:49:36.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.126+0000 7f36aa7f4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36a41036a0 0x7f36a4104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:49:36.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.126+0000 7f36aa7f4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36a4104040 0x7f36a4104fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:49:36.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.126+0000 7f36aa7f4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f36a410de70 con 0x7f36a41036a0 2024-06-04T04:49:36.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.126+0000 7f36aa7f4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f36a410dcf0 con 0x7f36a41024a0 2024-06-04T04:49:36.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.126+0000 7f36aa7f4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f36a410dff0 con 0x7f36a4104040 2024-06-04T04:49:36.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.126+0000 7f36a8d91700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36a4104040 0x7f36a4104fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:49:36.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.126+0000 7f36a8d91700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36a4104040 0x7f36a4104fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:41986/0 (socket says 172.21.15.112:41986) 2024-06-04T04:49:36.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.126+0000 7f36a8d91700 1 -- 172.21.15.112:0/810597195 learned_addr learned my addr 172.21.15.112:0/810597195 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-06-04T04:49:36.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.126+0000 7f36a37fe700 1 --2- 172.21.15.112:0/810597195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36a41036a0 0x7f36a4104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:49:36.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.126+0000 7f36a3fff700 1 --2- 172.21.15.112:0/810597195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a41024a0 0x7f36a4106430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:49:36.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.127+0000 7f36a8d91700 1 -- 172.21.15.112:0/810597195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a41024a0 msgr2=0x7f36a4106430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:49:36.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.127+0000 7f36a8d91700 1 --2- 172.21.15.112:0/810597195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a41024a0 0x7f36a4106430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:36.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.127+0000 7f36a8d91700 1 -- 172.21.15.112:0/810597195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36a41036a0 msgr2=0x7f36a4104a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:49:36.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.127+0000 7f36a8d91700 1 --2- 172.21.15.112:0/810597195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36a41036a0 0x7f36a4104a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:36.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.127+0000 7f36a8d91700 1 -- 172.21.15.112:0/810597195 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a4105870 con 0x7f36a4104040 2024-06-04T04:49:36.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.127+0000 7f36a8d91700 1 --2- 172.21.15.112:0/810597195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36a4104040 0x7f36a4104fc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f369c00b070 tx=0x7f369c007d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:49:36.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.127+0000 7f36a17fa700 1 -- 172.21.15.112:0/810597195 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369c003e10 con 0x7f36a4104040 2024-06-04T04:49:36.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.127+0000 7f36a17fa700 1 -- 172.21.15.112:0/810597195 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f369c022070 con 0x7f36a4104040 2024-06-04T04:49:36.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.127+0000 7f36a17fa700 1 -- 172.21.15.112:0/810597195 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369c01b860 con 0x7f36a4104040 2024-06-04T04:49:36.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.127+0000 7f36aa7f4700 1 -- 172.21.15.112:0/810597195 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36a41a36b0 con 0x7f36a4104040 2024-06-04T04:49:36.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.127+0000 7f36aa7f4700 1 -- 172.21.15.112:0/810597195 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36a41a3b30 con 0x7f36a4104040 2024-06-04T04:49:36.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.128+0000 7f36aa7f4700 1 -- 172.21.15.112:0/810597195 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36a406a230 con 0x7f36a4104040 2024-06-04T04:49:36.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.129+0000 7f36a17fa700 1 -- 172.21.15.112:0/810597195 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f369c01ba00 con 0x7f36a4104040 2024-06-04T04:49:36.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.130+0000 7f36a17fa700 1 --2- 172.21.15.112:0/810597195 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f36900b2880 0x7f36900b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:49:36.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.130+0000 7f36a17fa700 1 -- 172.21.15.112:0/810597195 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4101+0+0 (secure 0 0 0) 0x7f369c00f020 con 0x7f36a4104040 2024-06-04T04:49:36.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.130+0000 7f36a3fff700 1 --2- 172.21.15.112:0/810597195 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f36900b2880 0x7f36900b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:49:36.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.131+0000 7f36a3fff700 1 --2- 172.21.15.112:0/810597195 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f36900b2880 0x7f36900b4d40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f368c006fd0 tx=0x7f368c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:49:36.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.134+0000 7f36a17fa700 1 -- 172.21.15.112:0/810597195 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f369c0ab700 con 0x7f36a4104040 2024-06-04T04:49:36.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:36.344+0000 7f36aa7f4700 1 -- 172.21.15.112:0/810597195 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f36a40ffbe0 con 0x7f36900b2880 2024-06-04T04:49:36.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:36 smithi112 ceph-mon[36453]: osdmap e33: 5 total, 5 up, 5 in 2024-06-04T04:49:36.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:36 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:49:36.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:36 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:49:36.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:36 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:36.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:36 smithi098 ceph-mon[35569]: osdmap e33: 5 total, 5 up, 5 in 2024-06-04T04:49:36.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:36 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:49:36.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:36 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:49:36.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:36 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:36.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:36 smithi070 ceph-mon[32631]: osdmap e33: 5 total, 5 up, 5 in 2024-06-04T04:49:36.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:36 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:49:36.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:36 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:49:36.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:36 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:37.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:37 smithi070 ceph-mon[32631]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:37.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:37 smithi070 ceph-mon[32631]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:49:37.634 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:37 smithi112 ceph-mon[36453]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:37.634 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:37 smithi112 ceph-mon[36453]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:49:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:37 smithi098 ceph-mon[35569]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:37 smithi098 ceph-mon[35569]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:49:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:39 smithi112 ceph-mon[36453]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:39 smithi112 ceph-mon[36453]: from='client.? 172.21.15.112:0/840353387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51d86af1-0d6c-4ae0-b090-00414fc1bf7a"}]: dispatch 2024-06-04T04:49:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:39 smithi112 ceph-mon[36453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51d86af1-0d6c-4ae0-b090-00414fc1bf7a"}]: dispatch 2024-06-04T04:49:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:39 smithi112 ceph-mon[36453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51d86af1-0d6c-4ae0-b090-00414fc1bf7a"}]': finished 2024-06-04T04:49:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:39 smithi112 ceph-mon[36453]: osdmap e34: 6 total, 5 up, 6 in 2024-06-04T04:49:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:39 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:49:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:39 smithi098 ceph-mon[35569]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:39 smithi098 ceph-mon[35569]: from='client.? 172.21.15.112:0/840353387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51d86af1-0d6c-4ae0-b090-00414fc1bf7a"}]: dispatch 2024-06-04T04:49:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:39 smithi098 ceph-mon[35569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51d86af1-0d6c-4ae0-b090-00414fc1bf7a"}]: dispatch 2024-06-04T04:49:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:39 smithi098 ceph-mon[35569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51d86af1-0d6c-4ae0-b090-00414fc1bf7a"}]': finished 2024-06-04T04:49:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:39 smithi098 ceph-mon[35569]: osdmap e34: 6 total, 5 up, 6 in 2024-06-04T04:49:39.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:39 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:49:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:39 smithi070 ceph-mon[32631]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:39 smithi070 ceph-mon[32631]: from='client.? 172.21.15.112:0/840353387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51d86af1-0d6c-4ae0-b090-00414fc1bf7a"}]: dispatch 2024-06-04T04:49:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:39 smithi070 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51d86af1-0d6c-4ae0-b090-00414fc1bf7a"}]: dispatch 2024-06-04T04:49:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:39 smithi070 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51d86af1-0d6c-4ae0-b090-00414fc1bf7a"}]': finished 2024-06-04T04:49:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:39 smithi070 ceph-mon[32631]: osdmap e34: 6 total, 5 up, 6 in 2024-06-04T04:49:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:39 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:49:40.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:40 smithi098 ceph-mon[35569]: from='client.? 172.21.15.112:0/1287573340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:49:40.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:40 smithi112 ceph-mon[36453]: from='client.? 172.21.15.112:0/1287573340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:49:40.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:40 smithi070 ceph-mon[32631]: from='client.? 172.21.15.112:0/1287573340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:49:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:41 smithi112 ceph-mon[36453]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:41 smithi098 ceph-mon[35569]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:41.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:41 smithi070 ceph-mon[32631]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:43.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:43 smithi112 ceph-mon[36453]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:43 smithi098 ceph-mon[35569]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:43.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:43 smithi070 ceph-mon[32631]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:45.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:45 smithi112 ceph-mon[36453]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:45 smithi098 ceph-mon[35569]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:45.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:45 smithi070 ceph-mon[32631]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:47.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:47 smithi112 ceph-mon[36453]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:47.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-04T04:49:47.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:47.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:47 smithi112 ceph-mon[36453]: Deploying daemon osd.5 on smithi112 2024-06-04T04:49:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:47 smithi098 ceph-mon[35569]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-04T04:49:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:47 smithi098 ceph-mon[35569]: Deploying daemon osd.5 on smithi112 2024-06-04T04:49:47.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:47 smithi070 ceph-mon[32631]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:47.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-04T04:49:47.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:47.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:47 smithi070 ceph-mon[32631]: Deploying daemon osd.5 on smithi112 2024-06-04T04:49:49.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:49 smithi112 ceph-mon[36453]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:49 smithi098 ceph-mon[35569]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:49 smithi070 ceph-mon[32631]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:51.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:51 smithi112 ceph-mon[36453]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:51 smithi098 ceph-mon[35569]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:51.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:51 smithi070 ceph-mon[32631]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:53.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:52 smithi098 ceph-mon[35569]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:53.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:52 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:53.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:52 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:53.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:52 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:53.299 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:52 smithi112 ceph-mon[36453]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:53.299 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:52 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:53.299 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:52 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:53.299 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:52 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:53.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:52 smithi070 ceph-mon[32631]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:53.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:52 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:53.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:52 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:53.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:52 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:55.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:54 smithi098 ceph-mon[35569]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:55.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:54 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:55.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:54 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:55.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:54 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:55.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:54 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:49:55.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:54 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:55.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:54 smithi070 ceph-mon[32631]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:55.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:54 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:55.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:54 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:55.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:54 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:55.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:54 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:49:55.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:54 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:55.368 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:54 smithi112 ceph-mon[36453]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:55.368 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:54 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:55.369 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:54 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:55.369 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:54 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:55.369 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:54 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:49:55.369 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:54 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:55.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:55.711+0000 7f36a17fa700 1 -- 172.21.15.112:0/810597195 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f36a40ffbe0 con 0x7f36900b2880 2024-06-04T04:49:55.713 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 5 on host 'smithi112' 2024-06-04T04:49:55.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:55.713+0000 7f36aa7f4700 1 -- 172.21.15.112:0/810597195 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f36900b2880 msgr2=0x7f36900b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:49:55.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:55.713+0000 7f36aa7f4700 1 --2- 172.21.15.112:0/810597195 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f36900b2880 0x7f36900b4d40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f368c006fd0 tx=0x7f368c008040 comp rx=0 tx=0).stop 2024-06-04T04:49:55.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:55.713+0000 7f36aa7f4700 1 -- 172.21.15.112:0/810597195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36a4104040 msgr2=0x7f36a4104fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:49:55.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:55.713+0000 7f36aa7f4700 1 --2- 172.21.15.112:0/810597195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36a4104040 0x7f36a4104fc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f369c00b070 tx=0x7f369c007d10 comp rx=0 tx=0).stop 2024-06-04T04:49:55.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:55.713+0000 7f36aa7f4700 1 -- 172.21.15.112:0/810597195 shutdown_connections 2024-06-04T04:49:55.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:55.713+0000 7f36aa7f4700 1 --2- 172.21.15.112:0/810597195 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f36900b2880 0x7f36900b4d40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:55.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:55.713+0000 7f36aa7f4700 1 --2- 172.21.15.112:0/810597195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36a4104040 0x7f36a4104fc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:55.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:55.713+0000 7f36aa7f4700 1 --2- 172.21.15.112:0/810597195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a41024a0 0x7f36a4106430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:55.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:55.713+0000 7f36aa7f4700 1 --2- 172.21.15.112:0/810597195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36a41036a0 0x7f36a4104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:49:55.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:55.713+0000 7f36aa7f4700 1 -- 172.21.15.112:0/810597195 >> 172.21.15.112:0/810597195 conn(0x7f36a40fda60 msgr2=0x7f36a40ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:49:55.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:55.713+0000 7f36aa7f4700 1 -- 172.21.15.112:0/810597195 shutdown_connections 2024-06-04T04:49:55.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:49:55.713+0000 7f36aa7f4700 1 -- 172.21.15.112:0/810597195 wait complete. 2024-06-04T04:49:56.403 DEBUG:teuthology.orchestra.run.smithi112:osd.5> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.5.service 2024-06-04T04:49:56.406 INFO:tasks.cephadm:Deploying osd.6 on smithi070 with /dev/vg_nvme/lv_2... 2024-06-04T04:49:56.406 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-04T04:49:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:49:56.995 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:56 smithi070 ceph-mon[32631]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:56.996 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:56.996 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:56.996 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:56.996 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:56 smithi070 ceph-mon[32631]: from='osd.5 [v2:172.21.15.112:6808/1857257899,v1:172.21.15.112:6809/1857257899]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-04T04:49:56.996 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:56 smithi070 ceph-mon[32631]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-04T04:49:56.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:56 smithi098 ceph-mon[35569]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:56.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:56 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:56.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:56 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:56.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:56 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:56.998 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:56 smithi098 ceph-mon[35569]: from='osd.5 [v2:172.21.15.112:6808/1857257899,v1:172.21.15.112:6809/1857257899]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-04T04:49:56.998 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:56 smithi098 ceph-mon[35569]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-04T04:49:57.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:56 smithi112 ceph-mon[36453]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:57.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:56 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:57.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:56 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:57.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:56 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:49:57.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:56 smithi112 ceph-mon[36453]: from='osd.5 [v2:172.21.15.112:6808/1857257899,v1:172.21.15.112:6809/1857257899]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-04T04:49:57.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:56 smithi112 ceph-mon[36453]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-04T04:49:57.869 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:57 smithi112 ceph-mon[36453]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-04T04:49:57.870 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:57 smithi112 ceph-mon[36453]: osdmap e35: 6 total, 5 up, 6 in 2024-06-04T04:49:57.870 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:57 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:49:57.870 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:57 smithi112 ceph-mon[36453]: from='osd.5 [v2:172.21.15.112:6808/1857257899,v1:172.21.15.112:6809/1857257899]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:49:57.870 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:57 smithi112 ceph-mon[36453]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:49:57.870 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 04:49:57 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T04:49:57.698+0000 7fbeb34c0700 -1 osd.5 0 waiting for initial osdmap 2024-06-04T04:49:57.870 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 04:49:57 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T04:49:57.704+0000 7fbeaf2b3700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-04T04:49:57.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:57 smithi098 ceph-mon[35569]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-04T04:49:57.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:57 smithi098 ceph-mon[35569]: osdmap e35: 6 total, 5 up, 6 in 2024-06-04T04:49:57.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:57 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:49:57.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:57 smithi098 ceph-mon[35569]: from='osd.5 [v2:172.21.15.112:6808/1857257899,v1:172.21.15.112:6809/1857257899]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:49:57.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:57 smithi098 ceph-mon[35569]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:49:58.092 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:57 smithi070 ceph-mon[32631]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-04T04:49:58.092 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:57 smithi070 ceph-mon[32631]: osdmap e35: 6 total, 5 up, 6 in 2024-06-04T04:49:58.092 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:57 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:49:58.092 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:57 smithi070 ceph-mon[32631]: from='osd.5 [v2:172.21.15.112:6808/1857257899,v1:172.21.15.112:6809/1857257899]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:49:58.092 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:57 smithi070 ceph-mon[32631]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:49:58.715 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:58 smithi112 ceph-mon[36453]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:58.716 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:58 smithi112 ceph-mon[36453]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-06-04T04:49:58.716 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:58 smithi112 ceph-mon[36453]: osdmap e36: 6 total, 5 up, 6 in 2024-06-04T04:49:58.716 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:58 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:49:58.716 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:58 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:49:58.778 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-04T04:49:58.779 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-04T04:49:58.779 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-06-04T04:49:58.779 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-06-04T04:49:58.779 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00859866 s, 1.2 GB/s 2024-06-04T04:49:58.779 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-06-04T04:49:58.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:58 smithi098 ceph-mon[35569]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:58.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:58 smithi098 ceph-mon[35569]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-06-04T04:49:58.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:58 smithi098 ceph-mon[35569]: osdmap e36: 6 total, 5 up, 6 in 2024-06-04T04:49:58.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:58 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:49:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:58 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:49:59.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:58 smithi070 ceph-mon[32631]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:49:59.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:58 smithi070 ceph-mon[32631]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-06-04T04:49:59.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:58 smithi070 ceph-mon[32631]: osdmap e36: 6 total, 5 up, 6 in 2024-06-04T04:49:59.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:58 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:49:59.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:58 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:49:59.448 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi070:vg_nvme/lv_2 2024-06-04T04:49:59.698 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:49:59.962 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:59 smithi070 ceph-mon[32631]: purged_snaps scrub starts 2024-06-04T04:49:59.963 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:59 smithi070 ceph-mon[32631]: purged_snaps scrub ok 2024-06-04T04:49:59.963 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:59 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:49:59.963 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:59 smithi070 ceph-mon[32631]: osd.5 [v2:172.21.15.112:6808/1857257899,v1:172.21.15.112:6809/1857257899] boot 2024-06-04T04:49:59.963 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:59 smithi070 ceph-mon[32631]: osdmap e37: 6 total, 6 up, 6 in 2024-06-04T04:49:59.963 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:59 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:49:59.963 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:59 smithi070 ceph-mon[32631]: Detected new or changed devices on smithi112 2024-06-04T04:49:59.963 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:59 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:59.963 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:59 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:59.963 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:59 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:49:59.963 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:59 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:49:59.963 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:59 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi112 to 8544M 2024-06-04T04:49:59.963 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:59 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:59.963 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:59 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:59.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:59 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:49:59.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:49:59 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:59.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:59 smithi098 ceph-mon[35569]: purged_snaps scrub starts 2024-06-04T04:49:59.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:59 smithi098 ceph-mon[35569]: purged_snaps scrub ok 2024-06-04T04:49:59.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:59 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:49:59.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:59 smithi098 ceph-mon[35569]: osd.5 [v2:172.21.15.112:6808/1857257899,v1:172.21.15.112:6809/1857257899] boot 2024-06-04T04:49:59.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:59 smithi098 ceph-mon[35569]: osdmap e37: 6 total, 6 up, 6 in 2024-06-04T04:49:59.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:59 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:49:59.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:59 smithi098 ceph-mon[35569]: Detected new or changed devices on smithi112 2024-06-04T04:49:59.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:59 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:59.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:59 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:59.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:59 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:49:59.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:59 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:49:59.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:59 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi112 to 8544M 2024-06-04T04:49:59.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:59 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:49:59.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:59 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:49:59.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:59 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:49:59.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:49:59 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:00.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:59 smithi112 ceph-mon[36453]: purged_snaps scrub starts 2024-06-04T04:50:00.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:59 smithi112 ceph-mon[36453]: purged_snaps scrub ok 2024-06-04T04:50:00.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:59 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:50:00.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:59 smithi112 ceph-mon[36453]: osd.5 [v2:172.21.15.112:6808/1857257899,v1:172.21.15.112:6809/1857257899] boot 2024-06-04T04:50:00.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:59 smithi112 ceph-mon[36453]: osdmap e37: 6 total, 6 up, 6 in 2024-06-04T04:50:00.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:59 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:50:00.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:59 smithi112 ceph-mon[36453]: Detected new or changed devices on smithi112 2024-06-04T04:50:00.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:59 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:00.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:59 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:00.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:59 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:00.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:59 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:00.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:59 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi112 to 8544M 2024-06-04T04:50:00.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:59 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:00.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:59 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:00.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:59 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:50:00.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:49:59 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:00.932 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:00 smithi070 ceph-mon[32631]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:50:00.933 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:00 smithi070 ceph-mon[32631]: osdmap e38: 6 total, 6 up, 6 in 2024-06-04T04:50:00.933 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:00 smithi070 ceph-mon[32631]: overall HEALTH_OK 2024-06-04T04:50:00.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:00 smithi098 ceph-mon[35569]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:50:00.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:00 smithi098 ceph-mon[35569]: osdmap e38: 6 total, 6 up, 6 in 2024-06-04T04:50:00.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:00 smithi098 ceph-mon[35569]: overall HEALTH_OK 2024-06-04T04:50:01.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:00 smithi112 ceph-mon[36453]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-04T04:50:01.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:00 smithi112 ceph-mon[36453]: osdmap e38: 6 total, 6 up, 6 in 2024-06-04T04:50:01.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:00 smithi112 ceph-mon[36453]: overall HEALTH_OK 2024-06-04T04:50:01.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.447+0000 7eff69d46700 1 -- 172.21.15.70:0/445338165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff641024a0 msgr2=0x7eff641028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:50:01.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.447+0000 7eff69d46700 1 --2- 172.21.15.70:0/445338165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff641024a0 0x7eff641028c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7eff5400b0d0 tx=0x7eff5400b3e0 comp rx=0 tx=0).stop 2024-06-04T04:50:01.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.447+0000 7eff69d46700 1 -- 172.21.15.70:0/445338165 shutdown_connections 2024-06-04T04:50:01.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.447+0000 7eff69d46700 1 --2- 172.21.15.70:0/445338165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff64104040 0x7eff6410a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:01.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.447+0000 7eff69d46700 1 --2- 172.21.15.70:0/445338165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff641024a0 0x7eff641028c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:01.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.447+0000 7eff69d46700 1 --2- 172.21.15.70:0/445338165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff641036a0 0x7eff64103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:01.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.447+0000 7eff69d46700 1 -- 172.21.15.70:0/445338165 >> 172.21.15.70:0/445338165 conn(0x7eff640fda40 msgr2=0x7eff640ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:50:01.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.447+0000 7eff69d46700 1 -- 172.21.15.70:0/445338165 shutdown_connections 2024-06-04T04:50:01.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.447+0000 7eff69d46700 1 -- 172.21.15.70:0/445338165 wait complete. 2024-06-04T04:50:01.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff69d46700 1 Processor -- start 2024-06-04T04:50:01.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff69d46700 1 -- start start 2024-06-04T04:50:01.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff69d46700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff641024a0 0x7eff641063e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:50:01.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff69d46700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff641036a0 0x7eff64104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:50:01.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff69d46700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff64104040 0x7eff64104fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:50:01.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff69d46700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7eff6410de70 con 0x7eff64104040 2024-06-04T04:50:01.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff69d46700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff6410dcf0 con 0x7eff641024a0 2024-06-04T04:50:01.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff69d46700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7eff6410dff0 con 0x7eff641036a0 2024-06-04T04:50:01.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff63fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff64104040 0x7eff64104fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:50:01.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff63fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff64104040 0x7eff64104fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57032/0 (socket says 172.21.15.70:57032) 2024-06-04T04:50:01.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff63fff700 1 -- 172.21.15.70:0/2923439538 learned_addr learned my addr 172.21.15.70:0/2923439538 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:50:01.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff637fe700 1 --2- 172.21.15.70:0/2923439538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff641024a0 0x7eff641063e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:50:01.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff62ffd700 1 --2- 172.21.15.70:0/2923439538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff641036a0 0x7eff64104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:50:01.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff63fff700 1 -- 172.21.15.70:0/2923439538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff641036a0 msgr2=0x7eff64104a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:50:01.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff63fff700 1 --2- 172.21.15.70:0/2923439538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff641036a0 0x7eff64104a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:01.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff63fff700 1 -- 172.21.15.70:0/2923439538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff641024a0 msgr2=0x7eff641063e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:50:01.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff63fff700 1 --2- 172.21.15.70:0/2923439538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff641024a0 0x7eff641063e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:01.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff63fff700 1 -- 172.21.15.70:0/2923439538 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff64105890 con 0x7eff64104040 2024-06-04T04:50:01.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.448+0000 7eff637fe700 1 --2- 172.21.15.70:0/2923439538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff641024a0 0x7eff641063e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:50:01.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.449+0000 7eff63fff700 1 --2- 172.21.15.70:0/2923439538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff64104040 0x7eff64104fc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00b550 tx=0x7eff5c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:50:01.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.449+0000 7eff60ff9700 1 -- 172.21.15.70:0/2923439538 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c015020 con 0x7eff64104040 2024-06-04T04:50:01.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.449+0000 7eff69d46700 1 -- 172.21.15.70:0/2923439538 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff641a3850 con 0x7eff64104040 2024-06-04T04:50:01.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.449+0000 7eff60ff9700 1 -- 172.21.15.70:0/2923439538 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7eff5c011cc0 con 0x7eff64104040 2024-06-04T04:50:01.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.449+0000 7eff60ff9700 1 -- 172.21.15.70:0/2923439538 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c010c70 con 0x7eff64104040 2024-06-04T04:50:01.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.449+0000 7eff69d46700 1 -- 172.21.15.70:0/2923439538 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff641a3de0 con 0x7eff64104040 2024-06-04T04:50:01.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.450+0000 7eff69d46700 1 -- 172.21.15.70:0/2923439538 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff6406a230 con 0x7eff64104040 2024-06-04T04:50:01.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.451+0000 7eff60ff9700 1 -- 172.21.15.70:0/2923439538 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7eff5c018020 con 0x7eff64104040 2024-06-04T04:50:01.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.452+0000 7eff60ff9700 1 --2- 172.21.15.70:0/2923439538 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7eff500b2880 0x7eff500b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:50:01.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.452+0000 7eff637fe700 1 --2- 172.21.15.70:0/2923439538 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7eff500b2880 0x7eff500b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:50:01.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.452+0000 7eff60ff9700 1 -- 172.21.15.70:0/2923439538 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4533+0+0 (secure 0 0 0) 0x7eff5c0e1170 con 0x7eff64104040 2024-06-04T04:50:01.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.452+0000 7eff637fe700 1 --2- 172.21.15.70:0/2923439538 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7eff500b2880 0x7eff500b4d40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7eff54009fd0 tx=0x7eff5400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:50:01.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.456+0000 7eff60ff9700 1 -- 172.21.15.70:0/2923439538 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7eff5c01f020 con 0x7eff64104040 2024-06-04T04:50:01.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:01.671+0000 7eff69d46700 1 -- 172.21.15.70:0/2923439538 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7eff641a3fb0 con 0x7eff500b2880 2024-06-04T04:50:01.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:01 smithi098 ceph-mon[35569]: osdmap e39: 6 total, 6 up, 6 in 2024-06-04T04:50:01.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:01 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:50:01.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:01 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:50:01.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:01 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:02.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:01 smithi070 ceph-mon[32631]: osdmap e39: 6 total, 6 up, 6 in 2024-06-04T04:50:02.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:01 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:50:02.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:01 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:50:02.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:01 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:02.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:01 smithi112 ceph-mon[36453]: osdmap e39: 6 total, 6 up, 6 in 2024-06-04T04:50:02.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:01 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:50:02.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:01 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:50:02.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:01 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:02.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:02 smithi070 ceph-mon[32631]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:50:02.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:02 smithi070 ceph-mon[32631]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:02.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:02 smithi098 ceph-mon[35569]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:50:02.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:02 smithi098 ceph-mon[35569]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:03.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:02 smithi112 ceph-mon[36453]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:50:03.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:02 smithi112 ceph-mon[36453]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:04.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:04 smithi098 ceph-mon[35569]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:04.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:04 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2319184719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "526a8138-bfa8-419e-a5b5-e6b80f0ae004"}]: dispatch 2024-06-04T04:50:04.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:04 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2319184719' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "526a8138-bfa8-419e-a5b5-e6b80f0ae004"}]': finished 2024-06-04T04:50:04.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:04 smithi098 ceph-mon[35569]: osdmap e40: 7 total, 6 up, 7 in 2024-06-04T04:50:04.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:04 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:05.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:04 smithi070 ceph-mon[32631]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:05.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:04 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2319184719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "526a8138-bfa8-419e-a5b5-e6b80f0ae004"}]: dispatch 2024-06-04T04:50:05.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:04 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2319184719' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "526a8138-bfa8-419e-a5b5-e6b80f0ae004"}]': finished 2024-06-04T04:50:05.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:04 smithi070 ceph-mon[32631]: osdmap e40: 7 total, 6 up, 7 in 2024-06-04T04:50:05.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:04 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:05.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:04 smithi112 ceph-mon[36453]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:05.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:04 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2319184719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "526a8138-bfa8-419e-a5b5-e6b80f0ae004"}]: dispatch 2024-06-04T04:50:05.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:04 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2319184719' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "526a8138-bfa8-419e-a5b5-e6b80f0ae004"}]': finished 2024-06-04T04:50:05.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:04 smithi112 ceph-mon[36453]: osdmap e40: 7 total, 6 up, 7 in 2024-06-04T04:50:05.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:04 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:05.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:05 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2272654948' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:50:06.013 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:05 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2272654948' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:50:06.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:05 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2272654948' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:50:06.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:06 smithi098 ceph-mon[35569]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-04T04:50:07.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:06 smithi070 ceph-mon[32631]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-04T04:50:07.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:06 smithi112 ceph-mon[36453]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-04T04:50:08.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:08 smithi070 ceph-mon[32631]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-04T04:50:08.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:08 smithi098 ceph-mon[35569]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-04T04:50:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:08 smithi112 ceph-mon[36453]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-04T04:50:10.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:10 smithi070 ceph-mon[32631]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-06-04T04:50:10.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:10 smithi098 ceph-mon[35569]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-06-04T04:50:11.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:10 smithi112 ceph-mon[36453]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-06-04T04:50:12.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:12 smithi098 ceph-mon[35569]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-04T04:50:12.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:12 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-04T04:50:12.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:12 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:12.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:12 smithi098 ceph-mon[35569]: Deploying daemon osd.6 on smithi070 2024-06-04T04:50:13.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:12 smithi070 ceph-mon[32631]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-04T04:50:13.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:12 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-04T04:50:13.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:12 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:13.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:12 smithi070 ceph-mon[32631]: Deploying daemon osd.6 on smithi070 2024-06-04T04:50:13.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:12 smithi112 ceph-mon[36453]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-04T04:50:13.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:12 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-04T04:50:13.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:12 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:13.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:12 smithi112 ceph-mon[36453]: Deploying daemon osd.6 on smithi070 2024-06-04T04:50:14.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:14 smithi098 ceph-mon[35569]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-04T04:50:15.009 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:14 smithi070 ceph-mon[32631]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-04T04:50:15.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:14 smithi112 ceph-mon[36453]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-04T04:50:16.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:16 smithi070 ceph-mon[32631]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-06-04T04:50:16.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:16 smithi098 ceph-mon[35569]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-06-04T04:50:17.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:16 smithi112 ceph-mon[36453]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-06-04T04:50:18.911 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:18 smithi070 ceph-mon[32631]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:18.912 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:18.912 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:18.912 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:50:19.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:18 smithi112 ceph-mon[36453]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:19.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:19.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:19.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:50:19.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:18 smithi098 ceph-mon[35569]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:19.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:19.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:19.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:50:21.182 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:20 smithi070 ceph-mon[32631]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:21.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:20 smithi112 ceph-mon[36453]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:21.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:20 smithi098 ceph-mon[35569]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:21.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:21.558+0000 7eff60ff9700 1 -- 172.21.15.70:0/2923439538 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7eff641a3fb0 con 0x7eff500b2880 2024-06-04T04:50:21.561 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 6 on host 'smithi070' 2024-06-04T04:50:21.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:21.561+0000 7eff69d46700 1 -- 172.21.15.70:0/2923439538 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7eff500b2880 msgr2=0x7eff500b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:50:21.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:21.561+0000 7eff69d46700 1 --2- 172.21.15.70:0/2923439538 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7eff500b2880 0x7eff500b4d40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7eff54009fd0 tx=0x7eff5400b040 comp rx=0 tx=0).stop 2024-06-04T04:50:21.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:21.561+0000 7eff69d46700 1 -- 172.21.15.70:0/2923439538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff64104040 msgr2=0x7eff64104fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:50:21.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:21.561+0000 7eff69d46700 1 --2- 172.21.15.70:0/2923439538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff64104040 0x7eff64104fc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00b550 tx=0x7eff5c00b910 comp rx=0 tx=0).stop 2024-06-04T04:50:21.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:21.561+0000 7eff69d46700 1 -- 172.21.15.70:0/2923439538 shutdown_connections 2024-06-04T04:50:21.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:21.561+0000 7eff69d46700 1 --2- 172.21.15.70:0/2923439538 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7eff500b2880 0x7eff500b4d40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:21.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:21.561+0000 7eff69d46700 1 --2- 172.21.15.70:0/2923439538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff64104040 0x7eff64104fc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:21.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:21.561+0000 7eff69d46700 1 --2- 172.21.15.70:0/2923439538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff641024a0 0x7eff641063e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:21.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:21.561+0000 7eff69d46700 1 --2- 172.21.15.70:0/2923439538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff641036a0 0x7eff64104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:21.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:21.561+0000 7eff69d46700 1 -- 172.21.15.70:0/2923439538 >> 172.21.15.70:0/2923439538 conn(0x7eff640fda40 msgr2=0x7eff6406b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:50:21.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:21.561+0000 7eff69d46700 1 -- 172.21.15.70:0/2923439538 shutdown_connections 2024-06-04T04:50:21.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:50:21.561+0000 7eff69d46700 1 -- 172.21.15.70:0/2923439538 wait complete. 2024-06-04T04:50:22.315 DEBUG:teuthology.orchestra.run.smithi070:osd.6> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.6.service 2024-06-04T04:50:22.318 INFO:tasks.cephadm:Deploying osd.7 on smithi098 with /dev/vg_nvme/lv_2... 2024-06-04T04:50:22.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-04T04:50:22.563 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.b/config 2024-06-04T04:50:22.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:22.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:22.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:22 smithi070 ceph-mon[32631]: from='osd.6 [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-04T04:50:22.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:22.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:22.685 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:22.685 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:22.685 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:22 smithi098 ceph-mon[35569]: from='osd.6 [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-04T04:50:22.685 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:22.685 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:22.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:22.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:22.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:22 smithi112 ceph-mon[36453]: from='osd.6 [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-04T04:50:22.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:22.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:23.818 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:23 smithi098 ceph-mon[35569]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:23.818 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:23 smithi098 ceph-mon[35569]: from='osd.6 [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-04T04:50:23.818 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:23 smithi098 ceph-mon[35569]: osdmap e41: 7 total, 6 up, 7 in 2024-06-04T04:50:23.818 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:23 smithi098 ceph-mon[35569]: from='osd.6 [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-06-04T04:50:23.818 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:23 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:23.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:23 smithi070 ceph-mon[32631]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:23.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:23 smithi070 ceph-mon[32631]: from='osd.6 [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-04T04:50:23.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:23 smithi070 ceph-mon[32631]: osdmap e41: 7 total, 6 up, 7 in 2024-06-04T04:50:23.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:23 smithi070 ceph-mon[32631]: from='osd.6 [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-06-04T04:50:23.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:23 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:23.859 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 04:50:23 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T04:50:23.534+0000 7ff1d9d1e700 -1 osd.6 0 waiting for initial osdmap 2024-06-04T04:50:23.859 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 04:50:23 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T04:50:23.542+0000 7ff1d5b11700 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-04T04:50:23.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:23 smithi112 ceph-mon[36453]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:23.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:23 smithi112 ceph-mon[36453]: from='osd.6 [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-04T04:50:23.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:23 smithi112 ceph-mon[36453]: osdmap e41: 7 total, 6 up, 7 in 2024-06-04T04:50:23.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:23 smithi112 ceph-mon[36453]: from='osd.6 [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-06-04T04:50:23.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:23 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:24.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:24 smithi070 ceph-mon[32631]: from='osd.6 [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-06-04T04:50:24.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:24 smithi070 ceph-mon[32631]: osdmap e42: 7 total, 6 up, 7 in 2024-06-04T04:50:24.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:24 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:24.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:24 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:24.671 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:24 smithi098 ceph-mon[35569]: from='osd.6 [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-06-04T04:50:24.671 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:24 smithi098 ceph-mon[35569]: osdmap e42: 7 total, 6 up, 7 in 2024-06-04T04:50:24.671 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:24 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:24.671 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:24 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:24.755 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-04T04:50:24.756 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-04T04:50:24.756 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-06-04T04:50:24.756 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-06-04T04:50:24.756 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134357 s, 780 MB/s 2024-06-04T04:50:24.756 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-06-04T04:50:24.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:24 smithi112 ceph-mon[36453]: from='osd.6 [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-06-04T04:50:24.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:24 smithi112 ceph-mon[36453]: osdmap e42: 7 total, 6 up, 7 in 2024-06-04T04:50:24.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:24 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:24.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:24 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:25.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_2 2024-06-04T04:50:25.613 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.b/config 2024-06-04T04:50:25.633 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:25 smithi098 ceph-mon[35569]: purged_snaps scrub starts 2024-06-04T04:50:25.633 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:25 smithi098 ceph-mon[35569]: purged_snaps scrub ok 2024-06-04T04:50:25.634 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:25 smithi098 ceph-mon[35569]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:25.634 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:25 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:25.634 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:25 smithi098 ceph-mon[35569]: osd.6 [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496] boot 2024-06-04T04:50:25.634 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:25 smithi098 ceph-mon[35569]: osdmap e43: 7 total, 7 up, 7 in 2024-06-04T04:50:25.634 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:25 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:25 smithi070 ceph-mon[32631]: purged_snaps scrub starts 2024-06-04T04:50:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:25 smithi070 ceph-mon[32631]: purged_snaps scrub ok 2024-06-04T04:50:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:25 smithi070 ceph-mon[32631]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:25 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:25 smithi070 ceph-mon[32631]: osd.6 [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496] boot 2024-06-04T04:50:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:25 smithi070 ceph-mon[32631]: osdmap e43: 7 total, 7 up, 7 in 2024-06-04T04:50:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:25 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:25 smithi112 ceph-mon[36453]: purged_snaps scrub starts 2024-06-04T04:50:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:25 smithi112 ceph-mon[36453]: purged_snaps scrub ok 2024-06-04T04:50:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:25 smithi112 ceph-mon[36453]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-06-04T04:50:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:25 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:25 smithi112 ceph-mon[36453]: osd.6 [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496] boot 2024-06-04T04:50:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:25 smithi112 ceph-mon[36453]: osdmap e43: 7 total, 7 up, 7 in 2024-06-04T04:50:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:25 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:50:26.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:26 smithi070 ceph-mon[32631]: osdmap e44: 7 total, 7 up, 7 in 2024-06-04T04:50:26.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:26.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:26.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:26.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:26.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:26.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:26.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:26.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:50:26.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:26.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:50:26.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:26.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:50:26.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:26.880 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:26 smithi098 ceph-mon[35569]: osdmap e44: 7 total, 7 up, 7 in 2024-06-04T04:50:26.880 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:26.880 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:26.880 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:26.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:26.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:26.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:26.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:26.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:50:26.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:26.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:50:26.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:26.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:50:26.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:26.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:26 smithi112 ceph-mon[36453]: osdmap e44: 7 total, 7 up, 7 in 2024-06-04T04:50:26.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:26.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:26.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:26.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:26.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:26.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:26.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:26.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:50:26.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:26.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:50:26.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:26.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:50:26.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:27.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.225+0000 7f9f1dc70700 1 -- 172.21.15.98:0/4173327533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f181024e0 msgr2=0x7f9f18102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:50:27.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.225+0000 7f9f1dc70700 1 --2- 172.21.15.98:0/4173327533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f181024e0 0x7f9f18102900 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800b0d0 tx=0x7f9f0800b3e0 comp rx=0 tx=0).stop 2024-06-04T04:50:27.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.225+0000 7f9f1dc70700 1 -- 172.21.15.98:0/4173327533 shutdown_connections 2024-06-04T04:50:27.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.225+0000 7f9f1dc70700 1 --2- 172.21.15.98:0/4173327533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f18104080 0x7f9f1810a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:27.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.225+0000 7f9f1dc70700 1 --2- 172.21.15.98:0/4173327533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f181024e0 0x7f9f18102900 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:27.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.225+0000 7f9f1dc70700 1 --2- 172.21.15.98:0/4173327533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f181036e0 0x7f9f18103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:27.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.225+0000 7f9f1dc70700 1 -- 172.21.15.98:0/4173327533 >> 172.21.15.98:0/4173327533 conn(0x7f9f180fdaa0 msgr2=0x7f9f180ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:50:27.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.225+0000 7f9f1dc70700 1 -- 172.21.15.98:0/4173327533 shutdown_connections 2024-06-04T04:50:27.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.225+0000 7f9f1dc70700 1 -- 172.21.15.98:0/4173327533 wait complete. 2024-06-04T04:50:27.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.225+0000 7f9f1dc70700 1 Processor -- start 2024-06-04T04:50:27.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.226+0000 7f9f1dc70700 1 -- start start 2024-06-04T04:50:27.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.226+0000 7f9f1dc70700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f181024e0 0x7f9f181985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:50:27.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.226+0000 7f9f1dc70700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f181036e0 0x7f9f18198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:50:27.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.226+0000 7f9f1dc70700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f18104080 0x7f9f1819fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:50:27.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.226+0000 7f9f1dc70700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1810de40 con 0x7f9f181036e0 2024-06-04T04:50:27.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.226+0000 7f9f1dc70700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1810dcc0 con 0x7f9f18104080 2024-06-04T04:50:27.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.226+0000 7f9f1dc70700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1810dfc0 con 0x7f9f181024e0 2024-06-04T04:50:27.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.226+0000 7f9f17fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f18104080 0x7f9f1819fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:50:27.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.226+0000 7f9f17fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f18104080 0x7f9f1819fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40930/0 (socket says 172.21.15.98:40930) 2024-06-04T04:50:27.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.226+0000 7f9f17fff700 1 -- 172.21.15.98:0/666612391 learned_addr learned my addr 172.21.15.98:0/666612391 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-04T04:50:27.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.226+0000 7f9f177fe700 1 --2- 172.21.15.98:0/666612391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f181024e0 0x7f9f181985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:50:27.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.226+0000 7f9f16ffd700 1 --2- 172.21.15.98:0/666612391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f181036e0 0x7f9f18198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:50:27.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.226+0000 7f9f17fff700 1 -- 172.21.15.98:0/666612391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f181024e0 msgr2=0x7f9f181985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:50:27.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.226+0000 7f9f17fff700 1 --2- 172.21.15.98:0/666612391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f181024e0 0x7f9f181985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:27.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.226+0000 7f9f17fff700 1 -- 172.21.15.98:0/666612391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f181036e0 msgr2=0x7f9f18198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:50:27.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.226+0000 7f9f17fff700 1 --2- 172.21.15.98:0/666612391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f181036e0 0x7f9f18198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:27.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.226+0000 7f9f17fff700 1 -- 172.21.15.98:0/666612391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f181a0230 con 0x7f9f18104080 2024-06-04T04:50:27.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.227+0000 7f9f17fff700 1 --2- 172.21.15.98:0/666612391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f18104080 0x7f9f1819fb90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9f10007920 tx=0x7f9f10007c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:50:27.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.227+0000 7f9f14ff9700 1 -- 172.21.15.98:0/666612391 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f10019020 con 0x7f9f18104080 2024-06-04T04:50:27.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.227+0000 7f9f14ff9700 1 -- 172.21.15.98:0/666612391 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9f10004cc0 con 0x7f9f18104080 2024-06-04T04:50:27.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.227+0000 7f9f1dc70700 1 -- 172.21.15.98:0/666612391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f1806bcf0 con 0x7f9f18104080 2024-06-04T04:50:27.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.227+0000 7f9f14ff9700 1 -- 172.21.15.98:0/666612391 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1000a440 con 0x7f9f18104080 2024-06-04T04:50:27.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.227+0000 7f9f1dc70700 1 -- 172.21.15.98:0/666612391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f1806c1e0 con 0x7f9f18104080 2024-06-04T04:50:27.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.228+0000 7f9f1dc70700 1 -- 172.21.15.98:0/666612391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f1806a230 con 0x7f9f18104080 2024-06-04T04:50:27.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.229+0000 7f9f14ff9700 1 -- 172.21.15.98:0/666612391 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f9f10004750 con 0x7f9f18104080 2024-06-04T04:50:27.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.230+0000 7f9f14ff9700 1 --2- 172.21.15.98:0/666612391 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f9f040b2880 0x7f9f040b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:50:27.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.230+0000 7f9f14ff9700 1 -- 172.21.15.98:0/666612391 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4965+0+0 (secure 0 0 0) 0x7f9f100e2880 con 0x7f9f18104080 2024-06-04T04:50:27.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.230+0000 7f9f177fe700 1 --2- 172.21.15.98:0/666612391 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f9f040b2880 0x7f9f040b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:50:27.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.231+0000 7f9f177fe700 1 --2- 172.21.15.98:0/666612391 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f9f040b2880 0x7f9f040b4d40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009fd0 tx=0x7f9f0800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:50:27.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.234+0000 7f9f14ff9700 1 -- 172.21.15.98:0/666612391 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9f10023020 con 0x7f9f18104080 2024-06-04T04:50:27.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:27.448+0000 7f9f1dc70700 1 -- 172.21.15.98:0/666612391 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f9f1806c440 con 0x7f9f040b2880 2024-06-04T04:50:27.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:27 smithi070 ceph-mon[32631]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:27.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:27 smithi070 ceph-mon[32631]: Detected new or changed devices on smithi070 2024-06-04T04:50:27.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:27 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi070 to 5696M 2024-06-04T04:50:27.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:27 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:50:27.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:27 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:50:27.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:27 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:27.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:27 smithi112 ceph-mon[36453]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:27.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:27 smithi112 ceph-mon[36453]: Detected new or changed devices on smithi070 2024-06-04T04:50:27.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:27 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi070 to 5696M 2024-06-04T04:50:27.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:27 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:50:27.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:27 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:50:27.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:27 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:27.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:27 smithi098 ceph-mon[35569]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:27.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:27 smithi098 ceph-mon[35569]: Detected new or changed devices on smithi070 2024-06-04T04:50:27.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:27 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi070 to 5696M 2024-06-04T04:50:27.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:27 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:50:27.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:27 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:50:27.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:27 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:28.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:28 smithi098 ceph-mon[35569]: from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:50:28.837 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:28 smithi112 ceph-mon[36453]: from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:50:28.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:28 smithi070 ceph-mon[32631]: from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:50:29.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:29 smithi098 ceph-mon[35569]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:29.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:29 smithi070 ceph-mon[32631]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:29.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:29 smithi112 ceph-mon[36453]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:30.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:30 smithi098 ceph-mon[35569]: from='client.? 172.21.15.98:0/3514278446' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1173f785-6ed1-47ef-9265-2e233ea9ee02"}]: dispatch 2024-06-04T04:50:30.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:30 smithi098 ceph-mon[35569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1173f785-6ed1-47ef-9265-2e233ea9ee02"}]: dispatch 2024-06-04T04:50:30.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:30 smithi098 ceph-mon[35569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1173f785-6ed1-47ef-9265-2e233ea9ee02"}]': finished 2024-06-04T04:50:30.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:30 smithi098 ceph-mon[35569]: osdmap e45: 8 total, 7 up, 8 in 2024-06-04T04:50:30.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:30 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:50:30.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:30 smithi070 ceph-mon[32631]: from='client.? 172.21.15.98:0/3514278446' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1173f785-6ed1-47ef-9265-2e233ea9ee02"}]: dispatch 2024-06-04T04:50:30.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:30 smithi070 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1173f785-6ed1-47ef-9265-2e233ea9ee02"}]: dispatch 2024-06-04T04:50:30.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:30 smithi070 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1173f785-6ed1-47ef-9265-2e233ea9ee02"}]': finished 2024-06-04T04:50:30.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:30 smithi070 ceph-mon[32631]: osdmap e45: 8 total, 7 up, 8 in 2024-06-04T04:50:30.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:30 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:50:30.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:30 smithi112 ceph-mon[36453]: from='client.? 172.21.15.98:0/3514278446' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1173f785-6ed1-47ef-9265-2e233ea9ee02"}]: dispatch 2024-06-04T04:50:30.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:30 smithi112 ceph-mon[36453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1173f785-6ed1-47ef-9265-2e233ea9ee02"}]: dispatch 2024-06-04T04:50:30.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:30 smithi112 ceph-mon[36453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1173f785-6ed1-47ef-9265-2e233ea9ee02"}]': finished 2024-06-04T04:50:30.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:30 smithi112 ceph-mon[36453]: osdmap e45: 8 total, 7 up, 8 in 2024-06-04T04:50:30.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:30 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:50:31.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:31 smithi070 ceph-mon[32631]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:31.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:31 smithi070 ceph-mon[32631]: from='client.? 172.21.15.98:0/3545513220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:50:31.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:31 smithi112 ceph-mon[36453]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:31.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:31 smithi112 ceph-mon[36453]: from='client.? 172.21.15.98:0/3545513220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:50:31.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:31 smithi098 ceph-mon[35569]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:31.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:31 smithi098 ceph-mon[35569]: from='client.? 172.21.15.98:0/3545513220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:50:33.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:33 smithi070 ceph-mon[32631]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:33.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:33 smithi112 ceph-mon[36453]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:33.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:33 smithi098 ceph-mon[35569]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:35.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:35 smithi098 ceph-mon[35569]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:35.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:35 smithi070 ceph-mon[32631]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:35 smithi112 ceph-mon[36453]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:37.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:37 smithi070 ceph-mon[32631]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:37.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:37 smithi112 ceph-mon[36453]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:37.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:37 smithi098 ceph-mon[35569]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:38.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:38 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-04T04:50:38.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:38 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:38.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:38 smithi070 ceph-mon[32631]: Deploying daemon osd.7 on smithi098 2024-06-04T04:50:38.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:38 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-04T04:50:38.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:38 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:38.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:38 smithi112 ceph-mon[36453]: Deploying daemon osd.7 on smithi098 2024-06-04T04:50:38.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:38 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-04T04:50:38.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:38 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:38.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:38 smithi098 ceph-mon[35569]: Deploying daemon osd.7 on smithi098 2024-06-04T04:50:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:39 smithi070 ceph-mon[32631]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:39.926 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:39 smithi098 ceph-mon[35569]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:39.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:39 smithi112 ceph-mon[36453]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:41.720 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:41 smithi098 ceph-mon[35569]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:41.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:41 smithi070 ceph-mon[32631]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:41.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:41 smithi112 ceph-mon[36453]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:43.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:43 smithi070 ceph-mon[32631]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:43.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:43.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:43.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:50:43.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:43 smithi112 ceph-mon[36453]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:43.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:43.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:43.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:50:43.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:43 smithi098 ceph-mon[35569]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:43.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:43.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:43.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:50:45.623 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:45 smithi098 ceph-mon[35569]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:45 smithi070 ceph-mon[32631]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:45.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:45 smithi112 ceph-mon[36453]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:47.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:47.365+0000 7f9f14ff9700 1 -- 172.21.15.98:0/666612391 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9f1806c440 con 0x7f9f040b2880 2024-06-04T04:50:47.368 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 7 on host 'smithi098' 2024-06-04T04:50:47.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:47.368+0000 7f9f1dc70700 1 -- 172.21.15.98:0/666612391 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f9f040b2880 msgr2=0x7f9f040b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:50:47.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:47.368+0000 7f9f1dc70700 1 --2- 172.21.15.98:0/666612391 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f9f040b2880 0x7f9f040b4d40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009fd0 tx=0x7f9f0800b040 comp rx=0 tx=0).stop 2024-06-04T04:50:47.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:47.368+0000 7f9f1dc70700 1 -- 172.21.15.98:0/666612391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f18104080 msgr2=0x7f9f1819fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:50:47.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:47.368+0000 7f9f1dc70700 1 --2- 172.21.15.98:0/666612391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f18104080 0x7f9f1819fb90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9f10007920 tx=0x7f9f10007c30 comp rx=0 tx=0).stop 2024-06-04T04:50:47.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:47.368+0000 7f9f1dc70700 1 -- 172.21.15.98:0/666612391 shutdown_connections 2024-06-04T04:50:47.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:47.368+0000 7f9f1dc70700 1 --2- 172.21.15.98:0/666612391 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f9f040b2880 0x7f9f040b4d40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:47.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:47.368+0000 7f9f1dc70700 1 --2- 172.21.15.98:0/666612391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f18104080 0x7f9f1819fb90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:47.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:47.368+0000 7f9f1dc70700 1 --2- 172.21.15.98:0/666612391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f181024e0 0x7f9f181985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:47.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:47.368+0000 7f9f1dc70700 1 --2- 172.21.15.98:0/666612391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f181036e0 0x7f9f18198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:47.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:47.368+0000 7f9f1dc70700 1 -- 172.21.15.98:0/666612391 >> 172.21.15.98:0/666612391 conn(0x7f9f180fdaa0 msgr2=0x7f9f18106af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:50:47.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:47.368+0000 7f9f1dc70700 1 -- 172.21.15.98:0/666612391 shutdown_connections 2024-06-04T04:50:47.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:50:47.368+0000 7f9f1dc70700 1 -- 172.21.15.98:0/666612391 wait complete. 2024-06-04T04:50:47.677 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:47 smithi098 ceph-mon[35569]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:47.677 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:47.677 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:47.677 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:47.678 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:50:47.678 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:47.678 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:47 smithi098 ceph-mon[35569]: from='osd.7 [v2:172.21.15.98:6816/709593020,v1:172.21.15.98:6817/709593020]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-04T04:50:47.678 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:47 smithi098 ceph-mon[35569]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-04T04:50:47.678 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:47.678 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:47.678 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:50:47.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:47 smithi070 ceph-mon[32631]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:47.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:47.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:47.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:47.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:50:47.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:47.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:47 smithi070 ceph-mon[32631]: from='osd.7 [v2:172.21.15.98:6816/709593020,v1:172.21.15.98:6817/709593020]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-04T04:50:47.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:47 smithi070 ceph-mon[32631]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-04T04:50:47.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:47.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:47.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:50:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:47 smithi112 ceph-mon[36453]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:50:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:47 smithi112 ceph-mon[36453]: from='osd.7 [v2:172.21.15.98:6816/709593020,v1:172.21.15.98:6817/709593020]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-04T04:50:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:47 smithi112 ceph-mon[36453]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-04T04:50:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:50:48.023 DEBUG:teuthology.orchestra.run.smithi098:osd.7> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.7.service 2024-06-04T04:50:48.026 INFO:tasks.cephadm:Deploying osd.8 on smithi112 with /dev/vg_nvme/lv_2... 2024-06-04T04:50:48.027 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-04T04:50:48.274 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.c/config 2024-06-04T04:50:48.664 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:48 smithi112 ceph-mon[36453]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-04T04:50:48.665 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:48 smithi112 ceph-mon[36453]: osdmap e46: 8 total, 7 up, 8 in 2024-06-04T04:50:48.665 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:48 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:50:48.665 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:48 smithi112 ceph-mon[36453]: from='osd.7 [v2:172.21.15.98:6816/709593020,v1:172.21.15.98:6817/709593020]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:50:48.665 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:48 smithi112 ceph-mon[36453]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:50:48.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:48 smithi070 ceph-mon[32631]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-04T04:50:48.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:48 smithi070 ceph-mon[32631]: osdmap e46: 8 total, 7 up, 8 in 2024-06-04T04:50:48.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:48 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:50:48.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:48 smithi070 ceph-mon[32631]: from='osd.7 [v2:172.21.15.98:6816/709593020,v1:172.21.15.98:6817/709593020]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:50:48.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:48 smithi070 ceph-mon[32631]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:50:48.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:48 smithi098 ceph-mon[35569]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-04T04:50:48.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:48 smithi098 ceph-mon[35569]: osdmap e46: 8 total, 7 up, 8 in 2024-06-04T04:50:48.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:48 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:50:48.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:48 smithi098 ceph-mon[35569]: from='osd.7 [v2:172.21.15.98:6816/709593020,v1:172.21.15.98:6817/709593020]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:50:48.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:48 smithi098 ceph-mon[35569]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:50:48.996 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 04:50:48 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T04:50:48.599+0000 7fae55e41700 -1 osd.7 0 waiting for initial osdmap 2024-06-04T04:50:48.996 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 04:50:48 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T04:50:48.607+0000 7fae51c34700 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-04T04:50:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:49 smithi112 ceph-mon[36453]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:49 smithi112 ceph-mon[36453]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-06-04T04:50:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:49 smithi112 ceph-mon[36453]: osdmap e47: 8 total, 7 up, 8 in 2024-06-04T04:50:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:49 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:50:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:49 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:50:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:49 smithi070 ceph-mon[32631]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:49 smithi070 ceph-mon[32631]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-06-04T04:50:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:49 smithi070 ceph-mon[32631]: osdmap e47: 8 total, 7 up, 8 in 2024-06-04T04:50:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:49 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:50:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:49 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:50:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:49 smithi098 ceph-mon[35569]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-04T04:50:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:49 smithi098 ceph-mon[35569]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-06-04T04:50:49.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:49 smithi098 ceph-mon[35569]: osdmap e47: 8 total, 7 up, 8 in 2024-06-04T04:50:49.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:49 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:50:49.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:49 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:50:50.423 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-04T04:50:50.423 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-04T04:50:50.423 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2024-06-04T04:50:50.423 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2024-06-04T04:50:50.424 INFO:teuthology.orchestra.run.smithi112.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0149149 s, 703 MB/s 2024-06-04T04:50:50.424 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 2024-06-04T04:50:50.424 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2024-06-04T04:50:50.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:50 smithi098 ceph-mon[35569]: purged_snaps scrub starts 2024-06-04T04:50:50.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:50 smithi098 ceph-mon[35569]: purged_snaps scrub ok 2024-06-04T04:50:50.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:50 smithi098 ceph-mon[35569]: osd.7 [v2:172.21.15.98:6816/709593020,v1:172.21.15.98:6817/709593020] boot 2024-06-04T04:50:50.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:50 smithi098 ceph-mon[35569]: osdmap e48: 8 total, 8 up, 8 in 2024-06-04T04:50:50.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:50 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:50:50.836 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:50 smithi112 ceph-mon[36453]: purged_snaps scrub starts 2024-06-04T04:50:50.836 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:50 smithi112 ceph-mon[36453]: purged_snaps scrub ok 2024-06-04T04:50:50.836 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:50 smithi112 ceph-mon[36453]: osd.7 [v2:172.21.15.98:6816/709593020,v1:172.21.15.98:6817/709593020] boot 2024-06-04T04:50:50.836 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:50 smithi112 ceph-mon[36453]: osdmap e48: 8 total, 8 up, 8 in 2024-06-04T04:50:50.836 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:50 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:50:50.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:50 smithi070 ceph-mon[32631]: purged_snaps scrub starts 2024-06-04T04:50:50.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:50 smithi070 ceph-mon[32631]: purged_snaps scrub ok 2024-06-04T04:50:50.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:50 smithi070 ceph-mon[32631]: osd.7 [v2:172.21.15.98:6816/709593020,v1:172.21.15.98:6817/709593020] boot 2024-06-04T04:50:50.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:50 smithi070 ceph-mon[32631]: osdmap e48: 8 total, 8 up, 8 in 2024-06-04T04:50:50.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:50 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:50:51.138 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_2 2024-06-04T04:50:51.383 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.c/config 2024-06-04T04:50:51.747 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:51 smithi112 ceph-mon[36453]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:50:51.747 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:51 smithi112 ceph-mon[36453]: osdmap e49: 8 total, 8 up, 8 in 2024-06-04T04:50:51.747 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:51 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:51.747 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:51 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:51.747 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:51 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:51.748 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:51 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:51.748 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:51 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:51.748 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:51 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:51.748 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:51 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:51.748 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:51 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:50:51.748 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:51 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:51.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:51 smithi070 ceph-mon[32631]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:50:51.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:51 smithi070 ceph-mon[32631]: osdmap e49: 8 total, 8 up, 8 in 2024-06-04T04:50:51.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:51 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:51.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:51 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:51.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:51 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:51.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:51 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:51.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:51 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:51.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:51 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:51.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:51 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:51.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:51 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:50:51.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:51 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:51.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:51 smithi098 ceph-mon[35569]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:50:51.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:51 smithi098 ceph-mon[35569]: osdmap e49: 8 total, 8 up, 8 in 2024-06-04T04:50:51.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:51 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:51.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:51 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:51.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:51 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:51.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:51 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:51.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:51 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:50:51.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:51 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:51.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:51 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:51.882 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:51 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:50:51.882 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:51 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:50:52.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:52 smithi098 ceph-mon[35569]: Detected new or changed devices on smithi098 2024-06-04T04:50:52.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:52 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi098 to 5696M 2024-06-04T04:50:52.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:52 smithi098 ceph-mon[35569]: osdmap e50: 8 total, 8 up, 8 in 2024-06-04T04:50:52.784 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:52 smithi112 ceph-mon[36453]: Detected new or changed devices on smithi098 2024-06-04T04:50:52.785 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:52 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi098 to 5696M 2024-06-04T04:50:52.785 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:52 smithi112 ceph-mon[36453]: osdmap e50: 8 total, 8 up, 8 in 2024-06-04T04:50:53.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.012+0000 7f0a93ac5700 1 -- 172.21.15.112:0/1654359313 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a8c1092b0 msgr2=0x7f0a8c10b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:50:53.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.012+0000 7f0a93ac5700 1 --2- 172.21.15.112:0/1654359313 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a8c1092b0 0x7f0a8c10b720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400b0a0 tx=0x7f0a8400b3b0 comp rx=0 tx=0).stop 2024-06-04T04:50:53.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.012+0000 7f0a93ac5700 1 -- 172.21.15.112:0/1654359313 shutdown_connections 2024-06-04T04:50:53.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.012+0000 7f0a93ac5700 1 --2- 172.21.15.112:0/1654359313 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a8c1092b0 0x7f0a8c10b720 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:53.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.012+0000 7f0a93ac5700 1 --2- 172.21.15.112:0/1654359313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a8c104050 0x7f0a8c106440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:53.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.012+0000 7f0a93ac5700 1 --2- 172.21.15.112:0/1654359313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a8c106980 0x7f0a8c108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:53.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.012+0000 7f0a93ac5700 1 -- 172.21.15.112:0/1654359313 >> 172.21.15.112:0/1654359313 conn(0x7f0a8c0fdaa0 msgr2=0x7f0a8c0ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:50:53.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.013+0000 7f0a93ac5700 1 -- 172.21.15.112:0/1654359313 shutdown_connections 2024-06-04T04:50:53.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.013+0000 7f0a93ac5700 1 -- 172.21.15.112:0/1654359313 wait complete. 2024-06-04T04:50:53.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.013+0000 7f0a93ac5700 1 Processor -- start 2024-06-04T04:50:53.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.013+0000 7f0a93ac5700 1 -- start start 2024-06-04T04:50:53.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.013+0000 7f0a93ac5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a8c104050 0x7f0a8c1039b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:50:53.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.013+0000 7f0a93ac5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a8c106980 0x7f0a8c102000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:50:53.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.013+0000 7f0a93ac5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a8c1092b0 0x7f0a8c102540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:50:53.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.013+0000 7f0a93ac5700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c10ef50 con 0x7f0a8c106980 2024-06-04T04:50:53.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.013+0000 7f0a93ac5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c10edd0 con 0x7f0a8c104050 2024-06-04T04:50:53.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.013+0000 7f0a93ac5700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c10f0d0 con 0x7f0a8c1092b0 2024-06-04T04:50:53.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.013+0000 7f0a92062700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a8c1092b0 0x7f0a8c102540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:50:53.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.013+0000 7f0a92062700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a8c1092b0 0x7f0a8c102540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:57098/0 (socket says 172.21.15.112:57098) 2024-06-04T04:50:53.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.013+0000 7f0a92062700 1 -- 172.21.15.112:0/937337032 learned_addr learned my addr 172.21.15.112:0/937337032 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-06-04T04:50:53.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.014+0000 7f0a91060700 1 --2- 172.21.15.112:0/937337032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a8c106980 0x7f0a8c102000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:50:53.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.014+0000 7f0a91861700 1 --2- 172.21.15.112:0/937337032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a8c104050 0x7f0a8c1039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:50:53.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.014+0000 7f0a92062700 1 -- 172.21.15.112:0/937337032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a8c104050 msgr2=0x7f0a8c1039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:50:53.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.014+0000 7f0a92062700 1 --2- 172.21.15.112:0/937337032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a8c104050 0x7f0a8c1039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:53.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.014+0000 7f0a92062700 1 -- 172.21.15.112:0/937337032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a8c106980 msgr2=0x7f0a8c102000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:50:53.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.014+0000 7f0a92062700 1 --2- 172.21.15.112:0/937337032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a8c106980 0x7f0a8c102000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:50:53.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.014+0000 7f0a92062700 1 -- 172.21.15.112:0/937337032 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c102df0 con 0x7f0a8c1092b0 2024-06-04T04:50:53.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.014+0000 7f0a91060700 1 --2- 172.21.15.112:0/937337032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a8c106980 0x7f0a8c102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:50:53.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.014+0000 7f0a92062700 1 --2- 172.21.15.112:0/937337032 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a8c1092b0 0x7f0a8c102540 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0a840062a0 tx=0x7f0a8400bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:50:53.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.014+0000 7f0a82ffd700 1 -- 172.21.15.112:0/937337032 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8401f030 con 0x7f0a8c1092b0 2024-06-04T04:50:53.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.014+0000 7f0a93ac5700 1 -- 172.21.15.112:0/937337032 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8c1a36c0 con 0x7f0a8c1092b0 2024-06-04T04:50:53.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.014+0000 7f0a82ffd700 1 -- 172.21.15.112:0/937337032 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a84007b40 con 0x7f0a8c1092b0 2024-06-04T04:50:53.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.014+0000 7f0a82ffd700 1 -- 172.21.15.112:0/937337032 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a840047f0 con 0x7f0a8c1092b0 2024-06-04T04:50:53.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.014+0000 7f0a93ac5700 1 -- 172.21.15.112:0/937337032 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a8c1a3ba0 con 0x7f0a8c1092b0 2024-06-04T04:50:53.019 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.015+0000 7f0a93ac5700 1 -- 172.21.15.112:0/937337032 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a8c06a230 con 0x7f0a8c1092b0 2024-06-04T04:50:53.019 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.016+0000 7f0a82ffd700 1 -- 172.21.15.112:0/937337032 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f0a84003a00 con 0x7f0a8c1092b0 2024-06-04T04:50:53.019 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.017+0000 7f0a82ffd700 1 --2- 172.21.15.112:0/937337032 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0a780b2880 0x7f0a780b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:50:53.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.018+0000 7f0a82ffd700 1 -- 172.21.15.112:0/937337032 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5397+0+0 (secure 0 0 0) 0x7f0a840e26e0 con 0x7f0a8c1092b0 2024-06-04T04:50:53.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.018+0000 7f0a91861700 1 --2- 172.21.15.112:0/937337032 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0a780b2880 0x7f0a780b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:50:53.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.018+0000 7f0a91861700 1 --2- 172.21.15.112:0/937337032 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0a780b2880 0x7f0a780b4d40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0a88005ea0 tx=0x7f0a88005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:50:53.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.022+0000 7f0a82ffd700 1 -- 172.21.15.112:0/937337032 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0a840aaff0 con 0x7f0a8c1092b0 2024-06-04T04:50:53.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:52 smithi070 ceph-mon[32631]: Detected new or changed devices on smithi098 2024-06-04T04:50:53.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:52 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi098 to 5696M 2024-06-04T04:50:53.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:52 smithi070 ceph-mon[32631]: osdmap e50: 8 total, 8 up, 8 in 2024-06-04T04:50:53.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:50:53.235+0000 7f0a93ac5700 1 -- 172.21.15.112:0/937337032 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f0a8c063980 con 0x7f0a780b2880 2024-06-04T04:50:53.875 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:53 smithi112 ceph-mon[36453]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:50:53.875 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:53 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:50:53.876 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:53 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:50:53.876 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:53 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:53.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:53 smithi098 ceph-mon[35569]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:50:53.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:53 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:50:53.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:53 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:50:53.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:53 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:54.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:53 smithi070 ceph-mon[32631]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:50:54.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:53 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:50:54.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:53 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:50:54.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:53 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:50:54.879 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:54 smithi112 ceph-mon[36453]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:50:54.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:54 smithi098 ceph-mon[35569]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:50:55.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:54 smithi070 ceph-mon[32631]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:50:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:55 smithi112 ceph-mon[36453]: pgmap v178: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:50:55.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:55 smithi098 ceph-mon[35569]: pgmap v178: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:50:56.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:55 smithi070 ceph-mon[32631]: pgmap v178: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:50:56.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:56 smithi112 ceph-mon[36453]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-06-04T04:50:56.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:56 smithi112 ceph-mon[36453]: from='client.? 172.21.15.112:0/2304005360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6424cc6e-24a7-4b81-9ce6-39203fecc7ae"}]: dispatch 2024-06-04T04:50:56.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:56 smithi112 ceph-mon[36453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6424cc6e-24a7-4b81-9ce6-39203fecc7ae"}]: dispatch 2024-06-04T04:50:56.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:56 smithi112 ceph-mon[36453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6424cc6e-24a7-4b81-9ce6-39203fecc7ae"}]': finished 2024-06-04T04:50:56.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:56 smithi112 ceph-mon[36453]: osdmap e51: 9 total, 8 up, 9 in 2024-06-04T04:50:56.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:56 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:50:56.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:56 smithi098 ceph-mon[35569]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-06-04T04:50:56.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:56 smithi098 ceph-mon[35569]: from='client.? 172.21.15.112:0/2304005360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6424cc6e-24a7-4b81-9ce6-39203fecc7ae"}]: dispatch 2024-06-04T04:50:56.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:56 smithi098 ceph-mon[35569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6424cc6e-24a7-4b81-9ce6-39203fecc7ae"}]: dispatch 2024-06-04T04:50:56.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:56 smithi098 ceph-mon[35569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6424cc6e-24a7-4b81-9ce6-39203fecc7ae"}]': finished 2024-06-04T04:50:56.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:56 smithi098 ceph-mon[35569]: osdmap e51: 9 total, 8 up, 9 in 2024-06-04T04:50:56.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:56 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:50:57.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:56 smithi070 ceph-mon[32631]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-06-04T04:50:57.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:56 smithi070 ceph-mon[32631]: from='client.? 172.21.15.112:0/2304005360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6424cc6e-24a7-4b81-9ce6-39203fecc7ae"}]: dispatch 2024-06-04T04:50:57.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:56 smithi070 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6424cc6e-24a7-4b81-9ce6-39203fecc7ae"}]: dispatch 2024-06-04T04:50:57.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:56 smithi070 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6424cc6e-24a7-4b81-9ce6-39203fecc7ae"}]': finished 2024-06-04T04:50:57.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:56 smithi070 ceph-mon[32631]: osdmap e51: 9 total, 8 up, 9 in 2024-06-04T04:50:57.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:50:57.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:57 smithi112 ceph-mon[36453]: from='client.? 172.21.15.112:0/945925295' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:50:57.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:57 smithi098 ceph-mon[35569]: from='client.? 172.21.15.112:0/945925295' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:50:58.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:57 smithi070 ceph-mon[32631]: from='client.? 172.21.15.112:0/945925295' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:50:58.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:50:58 smithi112 ceph-mon[36453]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s, 0 objects/s recovering 2024-06-04T04:50:58.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:50:58 smithi098 ceph-mon[35569]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s, 0 objects/s recovering 2024-06-04T04:50:59.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:50:58 smithi070 ceph-mon[32631]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s, 0 objects/s recovering 2024-06-04T04:51:01.061 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:00 smithi112 ceph-mon[36453]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-06-04T04:51:01.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:00 smithi070 ceph-mon[32631]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-06-04T04:51:01.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:00 smithi098 ceph-mon[35569]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-06-04T04:51:03.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:02 smithi070 ceph-mon[32631]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-06-04T04:51:03.123 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:02 smithi112 ceph-mon[36453]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-06-04T04:51:03.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:02 smithi098 ceph-mon[35569]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-06-04T04:51:04.100 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:03 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-06-04T04:51:04.100 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:03 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:04.101 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:03 smithi112 ceph-mon[36453]: Deploying daemon osd.8 on smithi112 2024-06-04T04:51:04.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:03 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-06-04T04:51:04.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:03 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:04.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:03 smithi070 ceph-mon[32631]: Deploying daemon osd.8 on smithi112 2024-06-04T04:51:04.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:03 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-06-04T04:51:04.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:03 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:04.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:03 smithi098 ceph-mon[35569]: Deploying daemon osd.8 on smithi112 2024-06-04T04:51:05.081 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:04 smithi112 ceph-mon[36453]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-06-04T04:51:05.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:04 smithi070 ceph-mon[32631]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-06-04T04:51:05.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:04 smithi098 ceph-mon[35569]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-06-04T04:51:07.057 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:06 smithi112 ceph-mon[36453]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:07.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:06 smithi070 ceph-mon[32631]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:07.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:06 smithi098 ceph-mon[35569]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:09.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:08 smithi070 ceph-mon[32631]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:08 smithi112 ceph-mon[36453]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:08 smithi098 ceph-mon[35569]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:10.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:10 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:10.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:10 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:10.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:10 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:51:10.384 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:10 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:10.384 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:10 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:10.384 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:10 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:51:10.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:10 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:10.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:10 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:10.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:10 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:51:11.280 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:11 smithi112 ceph-mon[36453]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:11.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:11 smithi070 ceph-mon[32631]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:11.423 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:11 smithi098 ceph-mon[35569]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:12.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:51:12.855+0000 7f0a82ffd700 1 -- 172.21.15.112:0/937337032 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0a8c063980 con 0x7f0a780b2880 2024-06-04T04:51:12.858 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 8 on host 'smithi112' 2024-06-04T04:51:12.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:51:12.858+0000 7f0a93ac5700 1 -- 172.21.15.112:0/937337032 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0a780b2880 msgr2=0x7f0a780b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:51:12.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:51:12.858+0000 7f0a93ac5700 1 --2- 172.21.15.112:0/937337032 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0a780b2880 0x7f0a780b4d40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0a88005ea0 tx=0x7f0a88005e30 comp rx=0 tx=0).stop 2024-06-04T04:51:12.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:51:12.858+0000 7f0a93ac5700 1 -- 172.21.15.112:0/937337032 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a8c1092b0 msgr2=0x7f0a8c102540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:51:12.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:51:12.858+0000 7f0a93ac5700 1 --2- 172.21.15.112:0/937337032 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a8c1092b0 0x7f0a8c102540 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0a840062a0 tx=0x7f0a8400bd70 comp rx=0 tx=0).stop 2024-06-04T04:51:12.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:51:12.858+0000 7f0a93ac5700 1 -- 172.21.15.112:0/937337032 shutdown_connections 2024-06-04T04:51:12.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:51:12.858+0000 7f0a93ac5700 1 --2- 172.21.15.112:0/937337032 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0a780b2880 0x7f0a780b4d40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:12.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:51:12.858+0000 7f0a93ac5700 1 --2- 172.21.15.112:0/937337032 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a8c1092b0 0x7f0a8c102540 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:12.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:51:12.858+0000 7f0a93ac5700 1 --2- 172.21.15.112:0/937337032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a8c104050 0x7f0a8c1039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:12.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:51:12.858+0000 7f0a93ac5700 1 --2- 172.21.15.112:0/937337032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a8c106980 0x7f0a8c102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:12.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:51:12.858+0000 7f0a93ac5700 1 -- 172.21.15.112:0/937337032 >> 172.21.15.112:0/937337032 conn(0x7f0a8c0fdaa0 msgr2=0x7f0a8c104950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:51:12.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:51:12.858+0000 7f0a93ac5700 1 -- 172.21.15.112:0/937337032 shutdown_connections 2024-06-04T04:51:12.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:51:12.858+0000 7f0a93ac5700 1 -- 172.21.15.112:0/937337032 wait complete. 2024-06-04T04:51:13.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:12 smithi112 ceph-mon[36453]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:13.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:12 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:13.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:12 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:13.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:12 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:13.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:12 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:51:13.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:12 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:13.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:12 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:13.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:12 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:13.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:12 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:51:13.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:12 smithi112 ceph-mon[36453]: from='osd.8 [v2:172.21.15.112:6816/2250225413,v1:172.21.15.112:6817/2250225413]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-04T04:51:13.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:12 smithi112 ceph-mon[36453]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-04T04:51:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:12 smithi070 ceph-mon[32631]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:12 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:12 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:12 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:12 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:51:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:12 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:12 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:13.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:12 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:13.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:12 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:51:13.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:12 smithi070 ceph-mon[32631]: from='osd.8 [v2:172.21.15.112:6816/2250225413,v1:172.21.15.112:6817/2250225413]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-04T04:51:13.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:12 smithi070 ceph-mon[32631]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-04T04:51:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:12 smithi098 ceph-mon[35569]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:12 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:12 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:12 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:13.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:12 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:51:13.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:12 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:13.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:12 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:13.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:12 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:13.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:12 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:51:13.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:12 smithi098 ceph-mon[35569]: from='osd.8 [v2:172.21.15.112:6816/2250225413,v1:172.21.15.112:6817/2250225413]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-04T04:51:13.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:12 smithi098 ceph-mon[35569]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-04T04:51:13.552 DEBUG:teuthology.orchestra.run.smithi112:osd.8> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.8.service 2024-06-04T04:51:13.554 INFO:tasks.cephadm:Deploying osd.9 on smithi070 with /dev/vg_nvme/lv_1... 2024-06-04T04:51:13.554 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-04T04:51:13.801 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:51:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:14 smithi070 ceph-mon[32631]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:14 smithi070 ceph-mon[32631]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-06-04T04:51:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:14 smithi070 ceph-mon[32631]: osdmap e52: 9 total, 8 up, 9 in 2024-06-04T04:51:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:14 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:51:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:14 smithi070 ceph-mon[32631]: from='osd.8 [v2:172.21.15.112:6816/2250225413,v1:172.21.15.112:6817/2250225413]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:51:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:14 smithi070 ceph-mon[32631]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:51:15.137 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:14 smithi112 ceph-mon[36453]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:15.137 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:14 smithi112 ceph-mon[36453]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-06-04T04:51:15.137 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:14 smithi112 ceph-mon[36453]: osdmap e52: 9 total, 8 up, 9 in 2024-06-04T04:51:15.138 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:14 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:51:15.138 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:14 smithi112 ceph-mon[36453]: from='osd.8 [v2:172.21.15.112:6816/2250225413,v1:172.21.15.112:6817/2250225413]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:51:15.138 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:14 smithi112 ceph-mon[36453]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:51:15.138 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 04:51:14 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T04:51:14.862+0000 7f0993296700 -1 osd.8 0 waiting for initial osdmap 2024-06-04T04:51:15.138 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 04:51:14 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T04:51:14.871+0000 7f098f089700 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-04T04:51:15.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:14 smithi098 ceph-mon[35569]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:15.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:14 smithi098 ceph-mon[35569]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-06-04T04:51:15.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:14 smithi098 ceph-mon[35569]: osdmap e52: 9 total, 8 up, 9 in 2024-06-04T04:51:15.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:14 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:51:15.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:14 smithi098 ceph-mon[35569]: from='osd.8 [v2:172.21.15.112:6816/2250225413,v1:172.21.15.112:6817/2250225413]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:51:15.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:14 smithi098 ceph-mon[35569]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:51:16.096 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-04T04:51:16.096 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-04T04:51:16.096 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-06-04T04:51:16.096 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-06-04T04:51:16.096 INFO:teuthology.orchestra.run.smithi070.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00825421 s, 1.3 GB/s 2024-06-04T04:51:16.096 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-06-04T04:51:16.120 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:15 smithi070 ceph-mon[32631]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-06-04T04:51:16.120 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:15 smithi070 ceph-mon[32631]: osdmap e53: 9 total, 8 up, 9 in 2024-06-04T04:51:16.120 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:15 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:51:16.121 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:15 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:51:16.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:15 smithi112 ceph-mon[36453]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-06-04T04:51:16.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:15 smithi112 ceph-mon[36453]: osdmap e53: 9 total, 8 up, 9 in 2024-06-04T04:51:16.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:15 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:51:16.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:15 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:51:16.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:15 smithi098 ceph-mon[35569]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-06-04T04:51:16.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:15 smithi098 ceph-mon[35569]: osdmap e53: 9 total, 8 up, 9 in 2024-06-04T04:51:16.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:15 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:51:16.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:15 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:51:16.782 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi070:vg_nvme/lv_1 2024-06-04T04:51:17.022 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:51:17.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: purged_snaps scrub starts 2024-06-04T04:51:17.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: purged_snaps scrub ok 2024-06-04T04:51:17.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:17.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:51:17.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: osd.8 [v2:172.21.15.112:6816/2250225413,v1:172.21.15.112:6817/2250225413] boot 2024-06-04T04:51:17.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: osdmap e54: 9 total, 9 up, 9 in 2024-06-04T04:51:17.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:51:17.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: Detected new or changed devices on smithi112 2024-06-04T04:51:17.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:17.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:17.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:17.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:17.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:17.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi112 to 5696M 2024-06-04T04:51:17.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:17.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:17.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:51:17.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:16 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:17.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: purged_snaps scrub starts 2024-06-04T04:51:17.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: purged_snaps scrub ok 2024-06-04T04:51:17.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:17.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:51:17.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: osd.8 [v2:172.21.15.112:6816/2250225413,v1:172.21.15.112:6817/2250225413] boot 2024-06-04T04:51:17.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: osdmap e54: 9 total, 9 up, 9 in 2024-06-04T04:51:17.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:51:17.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: Detected new or changed devices on smithi112 2024-06-04T04:51:17.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:17.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:17.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:17.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:17.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:17.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi112 to 5696M 2024-06-04T04:51:17.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:17.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:17.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:51:17.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:16 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:17.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: purged_snaps scrub starts 2024-06-04T04:51:17.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: purged_snaps scrub ok 2024-06-04T04:51:17.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-04T04:51:17.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:51:17.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: osd.8 [v2:172.21.15.112:6816/2250225413,v1:172.21.15.112:6817/2250225413] boot 2024-06-04T04:51:17.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: osdmap e54: 9 total, 9 up, 9 in 2024-06-04T04:51:17.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:51:17.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: Detected new or changed devices on smithi112 2024-06-04T04:51:17.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:17.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:17.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:17.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:17.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:17.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi112 to 5696M 2024-06-04T04:51:17.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:17.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:17.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:51:17.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:16 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:18.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:17 smithi070 ceph-mon[32631]: osdmap e55: 9 total, 9 up, 9 in 2024-06-04T04:51:18.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:17 smithi112 ceph-mon[36453]: osdmap e55: 9 total, 9 up, 9 in 2024-06-04T04:51:18.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:17 smithi098 ceph-mon[35569]: osdmap e55: 9 total, 9 up, 9 in 2024-06-04T04:51:18.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.644+0000 7fed93041700 1 -- 172.21.15.70:0/239112253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed8c1047a0 msgr2=0x7fed8c104c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:51:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.644+0000 7fed93041700 1 --2- 172.21.15.70:0/239112253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed8c1047a0 0x7fed8c104c00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fed800099c0 tx=0x7fed80009cd0 comp rx=0 tx=0).stop 2024-06-04T04:51:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.644+0000 7fed93041700 1 -- 172.21.15.70:0/239112253 shutdown_connections 2024-06-04T04:51:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.644+0000 7fed93041700 1 --2- 172.21.15.70:0/239112253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed8c105140 0x7fed8c10b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.644+0000 7fed93041700 1 --2- 172.21.15.70:0/239112253 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed8c1035a0 0x7fed8c1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.644+0000 7fed93041700 1 --2- 172.21.15.70:0/239112253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed8c1047a0 0x7fed8c104c00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.644+0000 7fed93041700 1 -- 172.21.15.70:0/239112253 >> 172.21.15.70:0/239112253 conn(0x7fed8c0feb40 msgr2=0x7fed8c100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:51:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.644+0000 7fed93041700 1 -- 172.21.15.70:0/239112253 shutdown_connections 2024-06-04T04:51:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.644+0000 7fed93041700 1 -- 172.21.15.70:0/239112253 wait complete. 2024-06-04T04:51:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed93041700 1 Processor -- start 2024-06-04T04:51:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed93041700 1 -- start start 2024-06-04T04:51:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed93041700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed8c1035a0 0x7fed8c1996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:51:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed93041700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed8c1047a0 0x7fed8c199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:51:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed93041700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed8c105140 0x7fed8c1a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:51:18.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed93041700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c10eee0 con 0x7fed8c1047a0 2024-06-04T04:51:18.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed93041700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c10ed60 con 0x7fed8c105140 2024-06-04T04:51:18.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed93041700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c10f060 con 0x7fed8c1035a0 2024-06-04T04:51:18.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed8bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed8c1047a0 0x7fed8c199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:51:18.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed8bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed8c1047a0 0x7fed8c199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49252/0 (socket says 172.21.15.70:49252) 2024-06-04T04:51:18.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed8bfff700 1 -- 172.21.15.70:0/982762950 learned_addr learned my addr 172.21.15.70:0/982762950 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:51:18.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed90ddd700 1 --2- 172.21.15.70:0/982762950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed8c1035a0 0x7fed8c1996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:51:18.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed915de700 1 --2- 172.21.15.70:0/982762950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed8c105140 0x7fed8c1a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:51:18.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed8bfff700 1 -- 172.21.15.70:0/982762950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed8c1035a0 msgr2=0x7fed8c1996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:51:18.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed8bfff700 1 --2- 172.21.15.70:0/982762950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed8c1035a0 0x7fed8c1996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:18.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed8bfff700 1 -- 172.21.15.70:0/982762950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed8c105140 msgr2=0x7fed8c1a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:51:18.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed8bfff700 1 --2- 172.21.15.70:0/982762950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed8c105140 0x7fed8c1a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:18.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed8bfff700 1 -- 172.21.15.70:0/982762950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8c1a12b0 con 0x7fed8c1047a0 2024-06-04T04:51:18.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.645+0000 7fed8bfff700 1 --2- 172.21.15.70:0/982762950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed8c1047a0 0x7fed8c199be0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fed8000f690 tx=0x7fed8000f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:51:18.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.646+0000 7fed89ffb700 1 -- 172.21.15.70:0/982762950 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed80027070 con 0x7fed8c1047a0 2024-06-04T04:51:18.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.646+0000 7fed89ffb700 1 -- 172.21.15.70:0/982762950 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed8000fbd0 con 0x7fed8c1047a0 2024-06-04T04:51:18.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.646+0000 7fed93041700 1 -- 172.21.15.70:0/982762950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed8c1a1540 con 0x7fed8c1047a0 2024-06-04T04:51:18.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.646+0000 7fed89ffb700 1 -- 172.21.15.70:0/982762950 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed800037f0 con 0x7fed8c1047a0 2024-06-04T04:51:18.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.646+0000 7fed93041700 1 -- 172.21.15.70:0/982762950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed8c1a1a20 con 0x7fed8c1047a0 2024-06-04T04:51:18.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.647+0000 7fed93041700 1 -- 172.21.15.70:0/982762950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed8c06a230 con 0x7fed8c1047a0 2024-06-04T04:51:18.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.649+0000 7fed89ffb700 1 -- 172.21.15.70:0/982762950 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fed8002a070 con 0x7fed8c1047a0 2024-06-04T04:51:18.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.650+0000 7fed89ffb700 1 --2- 172.21.15.70:0/982762950 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fed740b2880 0x7fed740b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:51:18.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.650+0000 7fed90ddd700 1 --2- 172.21.15.70:0/982762950 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fed740b2880 0x7fed740b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:51:18.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.650+0000 7fed89ffb700 1 -- 172.21.15.70:0/982762950 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5829+0+0 (secure 0 0 0) 0x7fed800e2d80 con 0x7fed8c1047a0 2024-06-04T04:51:18.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.650+0000 7fed90ddd700 1 --2- 172.21.15.70:0/982762950 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fed740b2880 0x7fed740b4d40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fed7c005fd0 tx=0x7fed7c00a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:51:18.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.653+0000 7fed89ffb700 1 -- 172.21.15.70:0/982762950 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fed80013070 con 0x7fed8c1047a0 2024-06-04T04:51:18.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:18.869+0000 7fed93041700 1 -- 172.21.15.70:0/982762950 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fed8c063980 con 0x7fed740b2880 2024-06-04T04:51:19.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:18 smithi112 ceph-mon[36453]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:19.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:18 smithi112 ceph-mon[36453]: osdmap e56: 9 total, 9 up, 9 in 2024-06-04T04:51:19.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:18 smithi098 ceph-mon[35569]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:19.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:18 smithi098 ceph-mon[35569]: osdmap e56: 9 total, 9 up, 9 in 2024-06-04T04:51:19.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:18 smithi070 ceph-mon[32631]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:19.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:18 smithi070 ceph-mon[32631]: osdmap e56: 9 total, 9 up, 9 in 2024-06-04T04:51:20.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:19 smithi070 ceph-mon[32631]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:51:20.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:51:20.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:51:20.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:20.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:19 smithi112 ceph-mon[36453]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:51:20.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:51:20.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:51:20.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:20.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:19 smithi098 ceph-mon[35569]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:51:20.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:51:20.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:51:20.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:21.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:20 smithi070 ceph-mon[32631]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-06-04T04:51:21.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:20 smithi112 ceph-mon[36453]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-06-04T04:51:21.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:20 smithi098 ceph-mon[35569]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-06-04T04:51:22.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:21 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/358803290' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d666db8-8831-4d63-9e3c-a4bf2504149e"}]: dispatch 2024-06-04T04:51:22.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:21 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/358803290' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d666db8-8831-4d63-9e3c-a4bf2504149e"}]': finished 2024-06-04T04:51:22.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:21 smithi112 ceph-mon[36453]: osdmap e57: 10 total, 9 up, 10 in 2024-06-04T04:51:22.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:21 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:22.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:21 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/358803290' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d666db8-8831-4d63-9e3c-a4bf2504149e"}]: dispatch 2024-06-04T04:51:22.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:21 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/358803290' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d666db8-8831-4d63-9e3c-a4bf2504149e"}]': finished 2024-06-04T04:51:22.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:21 smithi098 ceph-mon[35569]: osdmap e57: 10 total, 9 up, 10 in 2024-06-04T04:51:22.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:21 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:22.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:21 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/358803290' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d666db8-8831-4d63-9e3c-a4bf2504149e"}]: dispatch 2024-06-04T04:51:22.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:21 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/358803290' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d666db8-8831-4d63-9e3c-a4bf2504149e"}]': finished 2024-06-04T04:51:22.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:21 smithi070 ceph-mon[32631]: osdmap e57: 10 total, 9 up, 10 in 2024-06-04T04:51:22.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:21 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:23.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:22 smithi112 ceph-mon[36453]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-04T04:51:23.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:22 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/4234315790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:51:23.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:22 smithi098 ceph-mon[35569]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-04T04:51:23.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:22 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/4234315790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:51:23.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:22 smithi070 ceph-mon[32631]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-04T04:51:23.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:22 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/4234315790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:51:25.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:24 smithi112 ceph-mon[36453]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-06-04T04:51:25.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:24 smithi098 ceph-mon[35569]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-06-04T04:51:25.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:24 smithi070 ceph-mon[32631]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-06-04T04:51:27.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:26 smithi070 ceph-mon[32631]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-04T04:51:27.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:26 smithi112 ceph-mon[36453]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-04T04:51:27.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:26 smithi098 ceph-mon[35569]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-04T04:51:29.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:28 smithi112 ceph-mon[36453]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-06-04T04:51:29.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:28 smithi098 ceph-mon[35569]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-06-04T04:51:29.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:28 smithi070 ceph-mon[32631]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-06-04T04:51:30.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:29 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-06-04T04:51:30.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:29 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:30.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:29 smithi112 ceph-mon[36453]: Deploying daemon osd.9 on smithi070 2024-06-04T04:51:30.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:29 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-06-04T04:51:30.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:29 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:30.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:29 smithi098 ceph-mon[35569]: Deploying daemon osd.9 on smithi070 2024-06-04T04:51:30.257 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:29 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-06-04T04:51:30.257 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:29 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:30.257 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:29 smithi070 ceph-mon[32631]: Deploying daemon osd.9 on smithi070 2024-06-04T04:51:31.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:30 smithi070 ceph-mon[32631]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:31.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:30 smithi112 ceph-mon[36453]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:31.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:30 smithi098 ceph-mon[35569]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:32.998 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:32 smithi070 ceph-mon[32631]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:33.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:32 smithi112 ceph-mon[36453]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:33.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:32 smithi098 ceph-mon[35569]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:34 smithi070 ceph-mon[32631]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:35.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:34 smithi112 ceph-mon[36453]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:35.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:34 smithi098 ceph-mon[35569]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:36.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:36 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:36.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:36 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:36.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:36 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:51:36.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:36 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:36.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:36 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:36.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:36 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:51:36.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:36 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:36.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:36 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:36.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:36 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:51:37.708 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:37 smithi070 ceph-mon[32631]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:37 smithi098 ceph-mon[35569]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:37.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:37 smithi112 ceph-mon[36453]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:39.348 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 9 on host 'smithi070' 2024-06-04T04:51:39.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:39.345+0000 7fed89ffb700 1 -- 172.21.15.70:0/982762950 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fed8c063980 con 0x7fed740b2880 2024-06-04T04:51:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:39.348+0000 7fed93041700 1 -- 172.21.15.70:0/982762950 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fed740b2880 msgr2=0x7fed740b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:51:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:39.348+0000 7fed93041700 1 --2- 172.21.15.70:0/982762950 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fed740b2880 0x7fed740b4d40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fed7c005fd0 tx=0x7fed7c00a560 comp rx=0 tx=0).stop 2024-06-04T04:51:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:39.348+0000 7fed93041700 1 -- 172.21.15.70:0/982762950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed8c1047a0 msgr2=0x7fed8c199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:51:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:39.348+0000 7fed93041700 1 --2- 172.21.15.70:0/982762950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed8c1047a0 0x7fed8c199be0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fed8000f690 tx=0x7fed8000f770 comp rx=0 tx=0).stop 2024-06-04T04:51:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:39.348+0000 7fed93041700 1 -- 172.21.15.70:0/982762950 shutdown_connections 2024-06-04T04:51:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:39.348+0000 7fed93041700 1 --2- 172.21.15.70:0/982762950 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fed740b2880 0x7fed740b4d40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:39.348+0000 7fed93041700 1 --2- 172.21.15.70:0/982762950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed8c105140 0x7fed8c1a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:39.348+0000 7fed93041700 1 --2- 172.21.15.70:0/982762950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed8c1035a0 0x7fed8c1996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:39.348+0000 7fed93041700 1 --2- 172.21.15.70:0/982762950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed8c1047a0 0x7fed8c199be0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:39.348+0000 7fed93041700 1 -- 172.21.15.70:0/982762950 >> 172.21.15.70:0/982762950 conn(0x7fed8c0feb40 msgr2=0x7fed8c107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:51:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:39.349+0000 7fed93041700 1 -- 172.21.15.70:0/982762950 shutdown_connections 2024-06-04T04:51:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:51:39.349+0000 7fed93041700 1 -- 172.21.15.70:0/982762950 wait complete. 2024-06-04T04:51:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:39 smithi070 ceph-mon[32631]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:39 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:39 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:39 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:39 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:51:39.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:39 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:39.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:39 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:39.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:39 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:39.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:39 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:51:39.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:39 smithi070 ceph-mon[32631]: from='osd.9 [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-04T04:51:39.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:39 smithi112 ceph-mon[36453]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:39.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:39 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:39.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:39 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:39.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:39 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:39.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:39 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:51:39.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:39 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:39.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:39 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:39.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:39 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:39.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:39 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:51:39.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:39 smithi112 ceph-mon[36453]: from='osd.9 [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-04T04:51:39.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:39 smithi098 ceph-mon[35569]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:39.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:39 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:39.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:39 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:39.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:39 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:39.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:39 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:51:39.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:39 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:39.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:39 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:39.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:39 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:39.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:39 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:51:39.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:39 smithi098 ceph-mon[35569]: from='osd.9 [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-04T04:51:40.030 DEBUG:teuthology.orchestra.run.smithi070:osd.9> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.9.service 2024-06-04T04:51:40.032 INFO:tasks.cephadm:Deploying osd.10 on smithi098 with /dev/vg_nvme/lv_1... 2024-06-04T04:51:40.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-04T04:51:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.b/config 2024-06-04T04:51:41.594 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:41 smithi098 ceph-mon[35569]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:41.595 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:41 smithi098 ceph-mon[35569]: from='osd.9 [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-06-04T04:51:41.595 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:41 smithi098 ceph-mon[35569]: osdmap e58: 10 total, 9 up, 10 in 2024-06-04T04:51:41.595 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:41 smithi098 ceph-mon[35569]: from='osd.9 [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-06-04T04:51:41.595 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:41 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:41 smithi070 ceph-mon[32631]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:41 smithi070 ceph-mon[32631]: from='osd.9 [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-06-04T04:51:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:41 smithi070 ceph-mon[32631]: osdmap e58: 10 total, 9 up, 10 in 2024-06-04T04:51:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:41 smithi070 ceph-mon[32631]: from='osd.9 [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-06-04T04:51:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:41 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:41.609 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 04:51:41 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T04:51:41.334+0000 7f1fb1121700 -1 osd.9 0 waiting for initial osdmap 2024-06-04T04:51:41.609 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 04:51:41 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T04:51:41.345+0000 7f1facf14700 -1 osd.9 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-04T04:51:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:41 smithi112 ceph-mon[36453]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:41 smithi112 ceph-mon[36453]: from='osd.9 [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-06-04T04:51:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:41 smithi112 ceph-mon[36453]: osdmap e58: 10 total, 9 up, 10 in 2024-06-04T04:51:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:41 smithi112 ceph-mon[36453]: from='osd.9 [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-06-04T04:51:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:41 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:42.349 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-04T04:51:42.349 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-04T04:51:42.349 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-06-04T04:51:42.349 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-06-04T04:51:42.349 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0118391 s, 886 MB/s 2024-06-04T04:51:42.349 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 2024-06-04T04:51:42.350 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-06-04T04:51:42.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:42 smithi070 ceph-mon[32631]: from='osd.9 [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-06-04T04:51:42.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:42 smithi070 ceph-mon[32631]: osdmap e59: 10 total, 9 up, 10 in 2024-06-04T04:51:42.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:42 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:42.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:42 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:42.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:42 smithi112 ceph-mon[36453]: from='osd.9 [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-06-04T04:51:42.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:42 smithi112 ceph-mon[36453]: osdmap e59: 10 total, 9 up, 10 in 2024-06-04T04:51:42.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:42 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:42.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:42 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:42.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:42 smithi098 ceph-mon[35569]: from='osd.9 [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-06-04T04:51:42.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:42 smithi098 ceph-mon[35569]: osdmap e59: 10 total, 9 up, 10 in 2024-06-04T04:51:42.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:42 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:42.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:42 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:43.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_1 2024-06-04T04:51:43.267 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.b/config 2024-06-04T04:51:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:43 smithi070 ceph-mon[32631]: purged_snaps scrub starts 2024-06-04T04:51:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:43 smithi070 ceph-mon[32631]: purged_snaps scrub ok 2024-06-04T04:51:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:43 smithi070 ceph-mon[32631]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:43 smithi070 ceph-mon[32631]: osd.9 [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707] boot 2024-06-04T04:51:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:43 smithi070 ceph-mon[32631]: osdmap e60: 10 total, 10 up, 10 in 2024-06-04T04:51:43.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:43.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:43.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:43.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:43.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:43.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:43.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:43.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:43.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:43.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:51:43.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:43 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:43.681 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:43 smithi098 ceph-mon[35569]: purged_snaps scrub starts 2024-06-04T04:51:43.681 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:43 smithi098 ceph-mon[35569]: purged_snaps scrub ok 2024-06-04T04:51:43.682 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:43 smithi098 ceph-mon[35569]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:43.682 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:43.682 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:43 smithi098 ceph-mon[35569]: osd.9 [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707] boot 2024-06-04T04:51:43.682 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:43 smithi098 ceph-mon[35569]: osdmap e60: 10 total, 10 up, 10 in 2024-06-04T04:51:43.682 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:43.682 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:43.682 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:43.682 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:43.682 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:43.682 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:43.682 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:43.682 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:43.683 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:43.683 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:51:43.683 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:43 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:43 smithi112 ceph-mon[36453]: purged_snaps scrub starts 2024-06-04T04:51:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:43 smithi112 ceph-mon[36453]: purged_snaps scrub ok 2024-06-04T04:51:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:43 smithi112 ceph-mon[36453]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-04T04:51:43.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:43.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:43 smithi112 ceph-mon[36453]: osd.9 [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707] boot 2024-06-04T04:51:43.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:43 smithi112 ceph-mon[36453]: osdmap e60: 10 total, 10 up, 10 in 2024-06-04T04:51:43.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:51:43.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:43.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:43.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:43.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:43.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:43.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:51:43.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:43.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:43.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:51:43.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:43 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:51:44.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:44 smithi070 ceph-mon[32631]: Detected new or changed devices on smithi070 2024-06-04T04:51:44.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:44 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi070 to 4272M 2024-06-04T04:51:44.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:44 smithi070 ceph-mon[32631]: osdmap e61: 10 total, 10 up, 10 in 2024-06-04T04:51:44.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:44 smithi112 ceph-mon[36453]: Detected new or changed devices on smithi070 2024-06-04T04:51:44.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:44 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi070 to 4272M 2024-06-04T04:51:44.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:44 smithi112 ceph-mon[36453]: osdmap e61: 10 total, 10 up, 10 in 2024-06-04T04:51:44.695 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:44 smithi098 ceph-mon[35569]: Detected new or changed devices on smithi070 2024-06-04T04:51:44.695 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:44 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi070 to 4272M 2024-06-04T04:51:44.695 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:44 smithi098 ceph-mon[35569]: osdmap e61: 10 total, 10 up, 10 in 2024-06-04T04:51:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.086+0000 7faefef3e700 1 -- 172.21.15.98:0/3703328317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef81035a0 msgr2=0x7faef81039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:51:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.086+0000 7faefef3e700 1 --2- 172.21.15.98:0/3703328317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef81035a0 0x7faef81039c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7faeec0099c0 tx=0x7faeec009cd0 comp rx=0 tx=0).stop 2024-06-04T04:51:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.086+0000 7faefef3e700 1 -- 172.21.15.98:0/3703328317 shutdown_connections 2024-06-04T04:51:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.086+0000 7faefef3e700 1 --2- 172.21.15.98:0/3703328317 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faef8105140 0x7faef810b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.086+0000 7faefef3e700 1 --2- 172.21.15.98:0/3703328317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef81035a0 0x7faef81039c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.086+0000 7faefef3e700 1 --2- 172.21.15.98:0/3703328317 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faef81047a0 0x7faef8104c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.086+0000 7faefef3e700 1 -- 172.21.15.98:0/3703328317 >> 172.21.15.98:0/3703328317 conn(0x7faef80feb40 msgr2=0x7faef8100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:51:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.087+0000 7faefef3e700 1 -- 172.21.15.98:0/3703328317 shutdown_connections 2024-06-04T04:51:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.087+0000 7faefef3e700 1 -- 172.21.15.98:0/3703328317 wait complete. 2024-06-04T04:51:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.087+0000 7faefef3e700 1 Processor -- start 2024-06-04T04:51:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.087+0000 7faefef3e700 1 -- start start 2024-06-04T04:51:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.087+0000 7faefef3e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faef81035a0 0x7faef8199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:51:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.087+0000 7faefef3e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef81047a0 0x7faef8199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:51:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.087+0000 7faefef3e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faef8105140 0x7faef81a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:51:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.087+0000 7faefef3e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faef810eef0 con 0x7faef8105140 2024-06-04T04:51:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.087+0000 7faefef3e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faef810ed70 con 0x7faef81047a0 2024-06-04T04:51:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.087+0000 7faefef3e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7faef810f070 con 0x7faef81035a0 2024-06-04T04:51:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.087+0000 7faef7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef81047a0 0x7faef8199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:51:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.087+0000 7faef7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef81047a0 0x7faef8199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40224/0 (socket says 172.21.15.98:40224) 2024-06-04T04:51:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.087+0000 7faef7fff700 1 -- 172.21.15.98:0/3446114095 learned_addr learned my addr 172.21.15.98:0/3446114095 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-04T04:51:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.088+0000 7faefccda700 1 --2- 172.21.15.98:0/3446114095 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faef81035a0 0x7faef8199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:51:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.088+0000 7faefd4db700 1 --2- 172.21.15.98:0/3446114095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faef8105140 0x7faef81a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:51:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.088+0000 7faef7fff700 1 -- 172.21.15.98:0/3446114095 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faef81035a0 msgr2=0x7faef8199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:51:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.088+0000 7faef7fff700 1 --2- 172.21.15.98:0/3446114095 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faef81035a0 0x7faef8199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.088+0000 7faef7fff700 1 -- 172.21.15.98:0/3446114095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faef8105140 msgr2=0x7faef81a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:51:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.088+0000 7faef7fff700 1 --2- 172.21.15.98:0/3446114095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faef8105140 0x7faef81a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:51:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.088+0000 7faef7fff700 1 -- 172.21.15.98:0/3446114095 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef81a12f0 con 0x7faef81047a0 2024-06-04T04:51:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.088+0000 7faef7fff700 1 --2- 172.21.15.98:0/3446114095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef81047a0 0x7faef8199bd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faee400c970 tx=0x7faee400cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:51:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.088+0000 7faef5ffb700 1 -- 172.21.15.98:0/3446114095 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee40078d0 con 0x7faef81047a0 2024-06-04T04:51:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.088+0000 7faef5ffb700 1 -- 172.21.15.98:0/3446114095 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faee4018480 con 0x7faef81047a0 2024-06-04T04:51:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.088+0000 7faef5ffb700 1 -- 172.21.15.98:0/3446114095 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee400f6b0 con 0x7faef81047a0 2024-06-04T04:51:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.088+0000 7faefef3e700 1 -- 172.21.15.98:0/3446114095 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faef81a15e0 con 0x7faef81047a0 2024-06-04T04:51:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.088+0000 7faefef3e700 1 -- 172.21.15.98:0/3446114095 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faef81a1ad0 con 0x7faef81047a0 2024-06-04T04:51:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.089+0000 7faefef3e700 1 -- 172.21.15.98:0/3446114095 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faef806a230 con 0x7faef81047a0 2024-06-04T04:51:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.090+0000 7faef5ffb700 1 -- 172.21.15.98:0/3446114095 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7faee4007a70 con 0x7faef81047a0 2024-06-04T04:51:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.091+0000 7faef5ffb700 1 --2- 172.21.15.98:0/3446114095 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7faee80b2900 0x7faee80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:51:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.091+0000 7faefccda700 1 --2- 172.21.15.98:0/3446114095 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7faee80b2900 0x7faee80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:51:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.091+0000 7faef5ffb700 1 -- 172.21.15.98:0/3446114095 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6261+0+0 (secure 0 0 0) 0x7faee40e1a80 con 0x7faef81047a0 2024-06-04T04:51:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.092+0000 7faefccda700 1 --2- 172.21.15.98:0/3446114095 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7faee80b2900 0x7faee80b4dc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7faeec00b580 tx=0x7faeec005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:51:45.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.096+0000 7faef5ffb700 1 -- 172.21.15.98:0/3446114095 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7faee40aa030 con 0x7faef81047a0 2024-06-04T04:51:45.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:51:45.305+0000 7faefef3e700 1 -- 172.21.15.98:0/3446114095 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7faef8063980 con 0x7faee80b2900 2024-06-04T04:51:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:45 smithi070 ceph-mon[32631]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:45 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:51:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:45 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:51:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:45 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:45 smithi112 ceph-mon[36453]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:45 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:51:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:45 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:51:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:45 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:45 smithi098 ceph-mon[35569]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:45 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:51:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:45 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:51:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:45 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:46.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:46 smithi070 ceph-mon[32631]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:51:46.631 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:46 smithi098 ceph-mon[35569]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:51:46.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:46 smithi112 ceph-mon[36453]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:51:47.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:47 smithi070 ceph-mon[32631]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:47 smithi112 ceph-mon[36453]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:47 smithi098 ceph-mon[35569]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:48.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:48 smithi112 ceph-mon[36453]: from='client.? 172.21.15.98:0/1652642985' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ecb925d2-0263-4dfc-a2fd-9e27856022d5"}]: dispatch 2024-06-04T04:51:48.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:48 smithi112 ceph-mon[36453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ecb925d2-0263-4dfc-a2fd-9e27856022d5"}]: dispatch 2024-06-04T04:51:48.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:48 smithi112 ceph-mon[36453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ecb925d2-0263-4dfc-a2fd-9e27856022d5"}]': finished 2024-06-04T04:51:48.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:48 smithi112 ceph-mon[36453]: osdmap e62: 11 total, 10 up, 11 in 2024-06-04T04:51:48.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:48 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:51:48.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:48 smithi098 ceph-mon[35569]: from='client.? 172.21.15.98:0/1652642985' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ecb925d2-0263-4dfc-a2fd-9e27856022d5"}]: dispatch 2024-06-04T04:51:48.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:48 smithi098 ceph-mon[35569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ecb925d2-0263-4dfc-a2fd-9e27856022d5"}]: dispatch 2024-06-04T04:51:48.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:48 smithi098 ceph-mon[35569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ecb925d2-0263-4dfc-a2fd-9e27856022d5"}]': finished 2024-06-04T04:51:48.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:48 smithi098 ceph-mon[35569]: osdmap e62: 11 total, 10 up, 11 in 2024-06-04T04:51:48.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:48 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:51:48.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:48 smithi070 ceph-mon[32631]: from='client.? 172.21.15.98:0/1652642985' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ecb925d2-0263-4dfc-a2fd-9e27856022d5"}]: dispatch 2024-06-04T04:51:48.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:48 smithi070 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ecb925d2-0263-4dfc-a2fd-9e27856022d5"}]: dispatch 2024-06-04T04:51:48.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:48 smithi070 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ecb925d2-0263-4dfc-a2fd-9e27856022d5"}]': finished 2024-06-04T04:51:48.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:48 smithi070 ceph-mon[32631]: osdmap e62: 11 total, 10 up, 11 in 2024-06-04T04:51:48.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:48 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:51:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:49 smithi112 ceph-mon[36453]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:49 smithi112 ceph-mon[36453]: from='client.? 172.21.15.98:0/1683145642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:51:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:49 smithi098 ceph-mon[35569]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:49 smithi098 ceph-mon[35569]: from='client.? 172.21.15.98:0/1683145642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:51:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:49 smithi070 ceph-mon[32631]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:49 smithi070 ceph-mon[32631]: from='client.? 172.21.15.98:0/1683145642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:51:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:51 smithi112 ceph-mon[36453]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:51 smithi098 ceph-mon[35569]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:51.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:51 smithi070 ceph-mon[32631]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:53.463 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:53 smithi098 ceph-mon[35569]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:53 smithi112 ceph-mon[36453]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:53.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:53 smithi070 ceph-mon[32631]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:55.593 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:55 smithi098 ceph-mon[35569]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:55 smithi112 ceph-mon[36453]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:55.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:55 smithi070 ceph-mon[32631]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:56.579 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:56 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-06-04T04:51:56.579 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:56 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:56.579 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:56 smithi098 ceph-mon[35569]: Deploying daemon osd.10 on smithi098 2024-06-04T04:51:56.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:56 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-06-04T04:51:56.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:56 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:56.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:56 smithi112 ceph-mon[36453]: Deploying daemon osd.10 on smithi098 2024-06-04T04:51:56.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-06-04T04:51:56.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:56 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:51:56.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:56 smithi070 ceph-mon[32631]: Deploying daemon osd.10 on smithi098 2024-06-04T04:51:57.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:57 smithi098 ceph-mon[35569]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:57 smithi112 ceph-mon[36453]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:57.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:57 smithi070 ceph-mon[32631]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:59.639 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:51:59 smithi098 ceph-mon[35569]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:51:59 smithi112 ceph-mon[36453]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:51:59.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:51:59 smithi070 ceph-mon[32631]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:52:01.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:01 smithi112 ceph-mon[36453]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:52:01.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:01 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:01.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:01 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:01.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:01 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:52:01.707 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:01 smithi098 ceph-mon[35569]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:52:01.707 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:01 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:01.707 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:01 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:01.707 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:01 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:52:01.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:01 smithi070 ceph-mon[32631]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:52:01.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:01 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:01.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:01 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:01.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:01 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:52:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:03 smithi112 ceph-mon[36453]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:52:03.689 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:03 smithi098 ceph-mon[35569]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:52:03.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:03 smithi070 ceph-mon[32631]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:52:05.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:52:05.272+0000 7faef5ffb700 1 -- 172.21.15.98:0/3446114095 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7faef8063980 con 0x7faee80b2900 2024-06-04T04:52:05.274 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 10 on host 'smithi098' 2024-06-04T04:52:05.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:52:05.274+0000 7faefef3e700 1 -- 172.21.15.98:0/3446114095 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7faee80b2900 msgr2=0x7faee80b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:05.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:52:05.274+0000 7faefef3e700 1 --2- 172.21.15.98:0/3446114095 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7faee80b2900 0x7faee80b4dc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7faeec00b580 tx=0x7faeec005fb0 comp rx=0 tx=0).stop 2024-06-04T04:52:05.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:52:05.274+0000 7faefef3e700 1 -- 172.21.15.98:0/3446114095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef81047a0 msgr2=0x7faef8199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:05.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:52:05.274+0000 7faefef3e700 1 --2- 172.21.15.98:0/3446114095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef81047a0 0x7faef8199bd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faee400c970 tx=0x7faee400cc80 comp rx=0 tx=0).stop 2024-06-04T04:52:05.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:52:05.274+0000 7faefef3e700 1 -- 172.21.15.98:0/3446114095 shutdown_connections 2024-06-04T04:52:05.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:52:05.274+0000 7faefef3e700 1 --2- 172.21.15.98:0/3446114095 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7faee80b2900 0x7faee80b4dc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:05.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:52:05.274+0000 7faefef3e700 1 --2- 172.21.15.98:0/3446114095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faef8105140 0x7faef81a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:05.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:52:05.274+0000 7faefef3e700 1 --2- 172.21.15.98:0/3446114095 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faef81035a0 0x7faef8199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:05.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:52:05.274+0000 7faefef3e700 1 --2- 172.21.15.98:0/3446114095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef81047a0 0x7faef8199bd0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:05.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:52:05.274+0000 7faefef3e700 1 -- 172.21.15.98:0/3446114095 >> 172.21.15.98:0/3446114095 conn(0x7faef80feb40 msgr2=0x7faef8107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:52:05.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:52:05.274+0000 7faefef3e700 1 -- 172.21.15.98:0/3446114095 shutdown_connections 2024-06-04T04:52:05.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-04T04:52:05.274+0000 7faefef3e700 1 -- 172.21.15.98:0/3446114095 wait complete. 2024-06-04T04:52:05.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:05 smithi070 ceph-mon[32631]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:52:05.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:05 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:05.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:05 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:05.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:05 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:05.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:05 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:52:05.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:05 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:05.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:05 smithi070 ceph-mon[32631]: from='osd.10 [v2:172.21.15.98:6824/1896493718,v1:172.21.15.98:6825/1896493718]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-04T04:52:05.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:05 smithi070 ceph-mon[32631]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-04T04:52:05.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:05 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:05.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:05 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:05.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:05 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:52:05.919 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:05 smithi098 ceph-mon[35569]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:52:05.920 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:05 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:05.920 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:05 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:05.920 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:05 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:05.920 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:05 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:52:05.920 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:05 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:05.920 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:05 smithi098 ceph-mon[35569]: from='osd.10 [v2:172.21.15.98:6824/1896493718,v1:172.21.15.98:6825/1896493718]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-04T04:52:05.920 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:05 smithi098 ceph-mon[35569]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-04T04:52:05.920 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:05 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:05.920 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:05 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:05.920 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:05 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:52:05.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:05 smithi112 ceph-mon[36453]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:52:05.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:05 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:05.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:05 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:05.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:05 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:05.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:05 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:52:05.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:05 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:05.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:05 smithi112 ceph-mon[36453]: from='osd.10 [v2:172.21.15.98:6824/1896493718,v1:172.21.15.98:6825/1896493718]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-04T04:52:05.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:05 smithi112 ceph-mon[36453]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-04T04:52:05.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:05 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:05.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:05 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:05.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:05 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:52:05.943 DEBUG:teuthology.orchestra.run.smithi098:osd.10> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.10.service 2024-06-04T04:52:05.946 INFO:tasks.cephadm:Deploying osd.11 on smithi112 with /dev/vg_nvme/lv_1... 2024-06-04T04:52:05.946 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-04T04:52:06.194 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.c/config 2024-06-04T04:52:06.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:06 smithi098 ceph-mon[35569]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-04T04:52:06.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:06 smithi098 ceph-mon[35569]: osdmap e63: 11 total, 10 up, 11 in 2024-06-04T04:52:06.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:06 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:52:06.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:06 smithi098 ceph-mon[35569]: from='osd.10 [v2:172.21.15.98:6824/1896493718,v1:172.21.15.98:6825/1896493718]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:52:06.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:06 smithi098 ceph-mon[35569]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:52:06.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:06 smithi070 ceph-mon[32631]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-04T04:52:06.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:06 smithi070 ceph-mon[32631]: osdmap e63: 11 total, 10 up, 11 in 2024-06-04T04:52:06.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:06 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:52:06.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:06 smithi070 ceph-mon[32631]: from='osd.10 [v2:172.21.15.98:6824/1896493718,v1:172.21.15.98:6825/1896493718]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:52:06.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:06 smithi070 ceph-mon[32631]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:52:06.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:06 smithi112 ceph-mon[36453]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-04T04:52:06.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:06 smithi112 ceph-mon[36453]: osdmap e63: 11 total, 10 up, 11 in 2024-06-04T04:52:06.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:06 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:52:06.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:06 smithi112 ceph-mon[36453]: from='osd.10 [v2:172.21.15.98:6824/1896493718,v1:172.21.15.98:6825/1896493718]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:52:06.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:06 smithi112 ceph-mon[36453]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-06-04T04:52:07.122 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 04:52:07 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T04:52:07.028+0000 7efd4d5bd700 -1 osd.10 0 waiting for initial osdmap 2024-06-04T04:52:07.122 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 04:52:07 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T04:52:07.040+0000 7efd453a8700 -1 osd.10 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-04T04:52:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:07 smithi098 ceph-mon[35569]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:52:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:07 smithi098 ceph-mon[35569]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-06-04T04:52:07.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:07 smithi098 ceph-mon[35569]: osdmap e64: 11 total, 10 up, 11 in 2024-06-04T04:52:07.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:07 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:52:07.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:07 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:52:07.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:07 smithi098 ceph-mon[35569]: from='osd.10 ' entity='osd.10' 2024-06-04T04:52:07.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:07 smithi070 ceph-mon[32631]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:52:07.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:07 smithi070 ceph-mon[32631]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-06-04T04:52:07.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:07 smithi070 ceph-mon[32631]: osdmap e64: 11 total, 10 up, 11 in 2024-06-04T04:52:07.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:07 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:52:07.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:07 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:52:07.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:07 smithi070 ceph-mon[32631]: from='osd.10 ' entity='osd.10' 2024-06-04T04:52:07.898 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:07 smithi112 ceph-mon[36453]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-06-04T04:52:07.899 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:07 smithi112 ceph-mon[36453]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-06-04T04:52:07.899 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:07 smithi112 ceph-mon[36453]: osdmap e64: 11 total, 10 up, 11 in 2024-06-04T04:52:07.899 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:07 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:52:07.899 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:07 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:52:07.899 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:07 smithi112 ceph-mon[36453]: from='osd.10 ' entity='osd.10' 2024-06-04T04:52:08.532 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-04T04:52:08.532 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-04T04:52:08.532 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2024-06-04T04:52:08.532 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2024-06-04T04:52:08.532 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115752 s, 906 MB/s 2024-06-04T04:52:08.532 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2024-06-04T04:52:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:08 smithi112 ceph-mon[36453]: purged_snaps scrub starts 2024-06-04T04:52:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:08 smithi112 ceph-mon[36453]: purged_snaps scrub ok 2024-06-04T04:52:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:08 smithi112 ceph-mon[36453]: osd.10 [v2:172.21.15.98:6824/1896493718,v1:172.21.15.98:6825/1896493718] boot 2024-06-04T04:52:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:08 smithi112 ceph-mon[36453]: osdmap e65: 11 total, 11 up, 11 in 2024-06-04T04:52:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:08 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:52:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:08 smithi112 ceph-mon[36453]: osdmap e66: 11 total, 11 up, 11 in 2024-06-04T04:52:08.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:08 smithi070 ceph-mon[32631]: purged_snaps scrub starts 2024-06-04T04:52:08.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:08 smithi070 ceph-mon[32631]: purged_snaps scrub ok 2024-06-04T04:52:08.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:08 smithi070 ceph-mon[32631]: osd.10 [v2:172.21.15.98:6824/1896493718,v1:172.21.15.98:6825/1896493718] boot 2024-06-04T04:52:08.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:08 smithi070 ceph-mon[32631]: osdmap e65: 11 total, 11 up, 11 in 2024-06-04T04:52:08.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:08 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:52:08.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:08 smithi070 ceph-mon[32631]: osdmap e66: 11 total, 11 up, 11 in 2024-06-04T04:52:08.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:08 smithi098 ceph-mon[35569]: purged_snaps scrub starts 2024-06-04T04:52:08.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:08 smithi098 ceph-mon[35569]: purged_snaps scrub ok 2024-06-04T04:52:08.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:08 smithi098 ceph-mon[35569]: osd.10 [v2:172.21.15.98:6824/1896493718,v1:172.21.15.98:6825/1896493718] boot 2024-06-04T04:52:08.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:08 smithi098 ceph-mon[35569]: osdmap e65: 11 total, 11 up, 11 in 2024-06-04T04:52:08.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:08 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:52:08.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:08 smithi098 ceph-mon[35569]: osdmap e66: 11 total, 11 up, 11 in 2024-06-04T04:52:09.293 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_1 2024-06-04T04:52:09.532 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.c/config 2024-06-04T04:52:09.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:09 smithi070 ceph-mon[32631]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:09.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:09 smithi070 ceph-mon[32631]: osdmap e67: 11 total, 11 up, 11 in 2024-06-04T04:52:09.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:09.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:09.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:09.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:09.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:09.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:09.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:09.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:09.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:52:09.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:09 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:09.861 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:09 smithi112 ceph-mon[36453]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:09.861 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:09 smithi112 ceph-mon[36453]: osdmap e67: 11 total, 11 up, 11 in 2024-06-04T04:52:09.861 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:09.861 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:09.861 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:09.861 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:09.861 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:09.861 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:09.861 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:09.861 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:09.862 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:52:09.862 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:09 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:09.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:09 smithi098 ceph-mon[35569]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:09.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:09 smithi098 ceph-mon[35569]: osdmap e67: 11 total, 11 up, 11 in 2024-06-04T04:52:09.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:09 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:09.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:09 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:09.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:09 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:09.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:09 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:09.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:09 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:09.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:09 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:09.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:09 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:09.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:09 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:09.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:09 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:52:09.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:09 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:10.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:10 smithi070 ceph-mon[32631]: Detected new or changed devices on smithi098 2024-06-04T04:52:10.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:10 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi098 to 4272M 2024-06-04T04:52:10.878 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:10 smithi112 ceph-mon[36453]: Detected new or changed devices on smithi098 2024-06-04T04:52:10.878 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:10 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi098 to 4272M 2024-06-04T04:52:10.897 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:10 smithi098 ceph-mon[35569]: Detected new or changed devices on smithi098 2024-06-04T04:52:10.897 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:10 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi098 to 4272M 2024-06-04T04:52:11.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.127+0000 7f4ced77e700 1 -- 172.21.15.112:0/687347706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ce8102490 msgr2=0x7f4ce81028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:11.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.127+0000 7f4ced77e700 1 --2- 172.21.15.112:0/687347706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ce8102490 0x7f4ce81028b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009990 tx=0x7f4cd8009ca0 comp rx=0 tx=0).stop 2024-06-04T04:52:11.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.127+0000 7f4ced77e700 1 -- 172.21.15.112:0/687347706 shutdown_connections 2024-06-04T04:52:11.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.127+0000 7f4ced77e700 1 --2- 172.21.15.112:0/687347706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ce8104030 0x7f4ce810a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:11.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.127+0000 7f4ced77e700 1 --2- 172.21.15.112:0/687347706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ce8102490 0x7f4ce81028b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:11.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.127+0000 7f4ced77e700 1 --2- 172.21.15.112:0/687347706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ce8103690 0x7f4ce8103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:11.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.127+0000 7f4ced77e700 1 -- 172.21.15.112:0/687347706 >> 172.21.15.112:0/687347706 conn(0x7f4ce80fda30 msgr2=0x7f4ce80ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:52:11.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.127+0000 7f4ced77e700 1 -- 172.21.15.112:0/687347706 shutdown_connections 2024-06-04T04:52:11.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.127+0000 7f4ced77e700 1 -- 172.21.15.112:0/687347706 wait complete. 2024-06-04T04:52:11.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.128+0000 7f4ced77e700 1 Processor -- start 2024-06-04T04:52:11.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.128+0000 7f4ced77e700 1 -- start start 2024-06-04T04:52:11.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.128+0000 7f4ced77e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ce8102490 0x7f4ce8199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:11.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.128+0000 7f4ced77e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ce8103690 0x7f4ce8199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:11.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.128+0000 7f4ced77e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ce8104030 0x7f4ce81a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:11.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.128+0000 7f4ced77e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce810de30 con 0x7f4ce8104030 2024-06-04T04:52:11.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.128+0000 7f4ced77e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce810dcb0 con 0x7f4ce8103690 2024-06-04T04:52:11.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.128+0000 7f4ced77e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce810dfb0 con 0x7f4ce8102490 2024-06-04T04:52:11.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.128+0000 7f4ce6ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ce8102490 0x7f4ce8199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:11.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.128+0000 7f4ce6ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ce8102490 0x7f4ce8199680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:34720/0 (socket says 172.21.15.112:34720) 2024-06-04T04:52:11.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.128+0000 7f4ce6ffd700 1 -- 172.21.15.112:0/207321630 learned_addr learned my addr 172.21.15.112:0/207321630 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-06-04T04:52:11.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.128+0000 7f4cde7fc700 1 --2- 172.21.15.112:0/207321630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ce8103690 0x7f4ce8199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:11.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.129+0000 7f4ce77fe700 1 --2- 172.21.15.112:0/207321630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ce8104030 0x7f4ce81a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:11.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.129+0000 7f4ce6ffd700 1 -- 172.21.15.112:0/207321630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ce8103690 msgr2=0x7f4ce8199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:11.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.129+0000 7f4ce6ffd700 1 --2- 172.21.15.112:0/207321630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ce8103690 0x7f4ce8199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:11.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.129+0000 7f4ce6ffd700 1 -- 172.21.15.112:0/207321630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ce8104030 msgr2=0x7f4ce81a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:11.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.129+0000 7f4ce6ffd700 1 --2- 172.21.15.112:0/207321630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ce8104030 0x7f4ce81a0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:11.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.129+0000 7f4ce6ffd700 1 -- 172.21.15.112:0/207321630 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce81a12e0 con 0x7f4ce8102490 2024-06-04T04:52:11.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.129+0000 7f4ce6ffd700 1 --2- 172.21.15.112:0/207321630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ce8102490 0x7f4ce8199680 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8000c00 tx=0x7f4cd80047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:52:11.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.129+0000 7f4ce4ff9700 1 -- 172.21.15.112:0/207321630 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8004830 con 0x7f4ce8102490 2024-06-04T04:52:11.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.129+0000 7f4ced77e700 1 -- 172.21.15.112:0/207321630 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ce81a1570 con 0x7f4ce8102490 2024-06-04T04:52:11.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.129+0000 7f4ce4ff9700 1 -- 172.21.15.112:0/207321630 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4cd8003b90 con 0x7f4ce8102490 2024-06-04T04:52:11.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.129+0000 7f4ce4ff9700 1 -- 172.21.15.112:0/207321630 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8005280 con 0x7f4ce8102490 2024-06-04T04:52:11.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.129+0000 7f4ced77e700 1 -- 172.21.15.112:0/207321630 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ce81a1a40 con 0x7f4ce8102490 2024-06-04T04:52:11.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.130+0000 7f4ced77e700 1 -- 172.21.15.112:0/207321630 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ce806a230 con 0x7f4ce8102490 2024-06-04T04:52:11.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.131+0000 7f4ce4ff9700 1 -- 172.21.15.112:0/207321630 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f4cd8005420 con 0x7f4ce8102490 2024-06-04T04:52:11.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.132+0000 7f4ce4ff9700 1 --2- 172.21.15.112:0/207321630 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f4cd40b2880 0x7f4cd40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:11.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.132+0000 7f4ce4ff9700 1 -- 172.21.15.112:0/207321630 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6694+0+0 (secure 0 0 0) 0x7f4cd80e30d0 con 0x7f4ce8102490 2024-06-04T04:52:11.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.132+0000 7f4cde7fc700 1 --2- 172.21.15.112:0/207321630 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f4cd40b2880 0x7f4cd40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:11.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.133+0000 7f4cde7fc700 1 --2- 172.21.15.112:0/207321630 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f4cd40b2880 0x7f4cd40b4d40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009a10 tx=0x7f4cd0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:52:11.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.135+0000 7f4ce4ff9700 1 -- 172.21.15.112:0/207321630 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4cd8013070 con 0x7f4ce8102490 2024-06-04T04:52:11.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:11.347+0000 7f4ced77e700 1 -- 172.21.15.112:0/207321630 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f4ce806c1a0 con 0x7f4cd40b2880 2024-06-04T04:52:11.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:11 smithi070 ceph-mon[32631]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-06-04T04:52:11.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:11 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:52:11.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:11 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:52:11.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:11 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:11 smithi112 ceph-mon[36453]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-06-04T04:52:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:11 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:52:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:11 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:52:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:11 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:11.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:11 smithi098 ceph-mon[35569]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-06-04T04:52:11.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:11 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-04T04:52:11.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:11 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-04T04:52:11.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:11 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:12.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:12 smithi112 ceph-mon[36453]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:52:12.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:12 smithi098 ceph-mon[35569]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:52:13.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:12 smithi070 ceph-mon[32631]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:52:13.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:13 smithi112 ceph-mon[36453]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-06-04T04:52:13.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:13 smithi098 ceph-mon[35569]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-06-04T04:52:14.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:13 smithi070 ceph-mon[32631]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-06-04T04:52:14.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:14 smithi112 ceph-mon[36453]: from='client.? 172.21.15.112:0/2489096639' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8e3230c-bc3a-40bd-8dd0-5fef56e4df97"}]: dispatch 2024-06-04T04:52:14.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:14 smithi112 ceph-mon[36453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8e3230c-bc3a-40bd-8dd0-5fef56e4df97"}]: dispatch 2024-06-04T04:52:14.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:14 smithi112 ceph-mon[36453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8e3230c-bc3a-40bd-8dd0-5fef56e4df97"}]': finished 2024-06-04T04:52:14.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:14 smithi112 ceph-mon[36453]: osdmap e68: 12 total, 11 up, 12 in 2024-06-04T04:52:14.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:14 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:14.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:14 smithi098 ceph-mon[35569]: from='client.? 172.21.15.112:0/2489096639' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8e3230c-bc3a-40bd-8dd0-5fef56e4df97"}]: dispatch 2024-06-04T04:52:14.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:14 smithi098 ceph-mon[35569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8e3230c-bc3a-40bd-8dd0-5fef56e4df97"}]: dispatch 2024-06-04T04:52:14.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:14 smithi098 ceph-mon[35569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8e3230c-bc3a-40bd-8dd0-5fef56e4df97"}]': finished 2024-06-04T04:52:14.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:14 smithi098 ceph-mon[35569]: osdmap e68: 12 total, 11 up, 12 in 2024-06-04T04:52:14.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:14 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:14 smithi070 ceph-mon[32631]: from='client.? 172.21.15.112:0/2489096639' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8e3230c-bc3a-40bd-8dd0-5fef56e4df97"}]: dispatch 2024-06-04T04:52:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:14 smithi070 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8e3230c-bc3a-40bd-8dd0-5fef56e4df97"}]: dispatch 2024-06-04T04:52:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:14 smithi070 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8e3230c-bc3a-40bd-8dd0-5fef56e4df97"}]': finished 2024-06-04T04:52:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:14 smithi070 ceph-mon[32631]: osdmap e68: 12 total, 11 up, 12 in 2024-06-04T04:52:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:14 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:15.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:15 smithi112 ceph-mon[36453]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-06-04T04:52:15.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:15 smithi112 ceph-mon[36453]: from='client.? 172.21.15.112:0/2629644384' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:52:15.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:15 smithi098 ceph-mon[35569]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-06-04T04:52:15.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:15 smithi098 ceph-mon[35569]: from='client.? 172.21.15.112:0/2629644384' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:52:16.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:15 smithi070 ceph-mon[32631]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-06-04T04:52:16.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:15 smithi070 ceph-mon[32631]: from='client.? 172.21.15.112:0/2629644384' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-04T04:52:17.880 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:17 smithi112 ceph-mon[36453]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-04T04:52:17.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:17 smithi098 ceph-mon[35569]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-04T04:52:18.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:17 smithi070 ceph-mon[32631]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-04T04:52:19.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:19 smithi070 ceph-mon[32631]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-06-04T04:52:19.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:19 smithi112 ceph-mon[36453]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-06-04T04:52:19.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:19 smithi098 ceph-mon[35569]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-06-04T04:52:21.915 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:21 smithi112 ceph-mon[36453]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:21.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:21 smithi098 ceph-mon[35569]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:22.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:21 smithi070 ceph-mon[32631]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:22.819 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-06-04T04:52:22.819 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:22.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-06-04T04:52:22.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:23.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-06-04T04:52:23.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:23.921 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:23 smithi112 ceph-mon[36453]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:23.921 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:23 smithi112 ceph-mon[36453]: Deploying daemon osd.11 on smithi112 2024-06-04T04:52:23.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:23 smithi098 ceph-mon[35569]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:23.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:23 smithi098 ceph-mon[35569]: Deploying daemon osd.11 on smithi112 2024-06-04T04:52:24.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:23 smithi070 ceph-mon[32631]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:24.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:23 smithi070 ceph-mon[32631]: Deploying daemon osd.11 on smithi112 2024-06-04T04:52:24.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:24 smithi112 ceph-mon[36453]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:24.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:24 smithi098 ceph-mon[35569]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:25.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:24 smithi070 ceph-mon[32631]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:26.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:26 smithi112 ceph-mon[36453]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:27.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:26 smithi070 ceph-mon[32631]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:27.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:26 smithi098 ceph-mon[35569]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:29.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:29 smithi112 ceph-mon[36453]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:29.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:29 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:29.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:29 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:29.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:29 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:52:29.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:29 smithi070 ceph-mon[32631]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:29.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:29 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:29.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:29 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:29.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:29 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:52:29.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:29 smithi098 ceph-mon[35569]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:29.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:29 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:29.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:29 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:29.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:29 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:52:31.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:31 smithi070 ceph-mon[32631]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:31.391 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:31 smithi112 ceph-mon[36453]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:31 smithi098 ceph-mon[35569]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:32.119 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:32.116+0000 7f4ce4ff9700 1 -- 172.21.15.112:0/207321630 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f4ce806c1a0 con 0x7f4cd40b2880 2024-06-04T04:52:32.119 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 11 on host 'smithi112' 2024-06-04T04:52:32.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:32.118+0000 7f4ced77e700 1 -- 172.21.15.112:0/207321630 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f4cd40b2880 msgr2=0x7f4cd40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:32.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:32.118+0000 7f4ced77e700 1 --2- 172.21.15.112:0/207321630 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f4cd40b2880 0x7f4cd40b4d40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009a10 tx=0x7f4cd0009190 comp rx=0 tx=0).stop 2024-06-04T04:52:32.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:32.118+0000 7f4ced77e700 1 -- 172.21.15.112:0/207321630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ce8102490 msgr2=0x7f4ce8199680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:32.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:32.118+0000 7f4ced77e700 1 --2- 172.21.15.112:0/207321630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ce8102490 0x7f4ce8199680 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8000c00 tx=0x7f4cd80047b0 comp rx=0 tx=0).stop 2024-06-04T04:52:32.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:32.118+0000 7f4ced77e700 1 -- 172.21.15.112:0/207321630 shutdown_connections 2024-06-04T04:52:32.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:32.118+0000 7f4ced77e700 1 --2- 172.21.15.112:0/207321630 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f4cd40b2880 0x7f4cd40b4d40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:32.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:32.118+0000 7f4ced77e700 1 --2- 172.21.15.112:0/207321630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ce8104030 0x7f4ce81a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:32.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:32.118+0000 7f4ced77e700 1 --2- 172.21.15.112:0/207321630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ce8102490 0x7f4ce8199680 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:32.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:32.118+0000 7f4ced77e700 1 --2- 172.21.15.112:0/207321630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ce8103690 0x7f4ce8199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:32.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:32.118+0000 7f4ced77e700 1 -- 172.21.15.112:0/207321630 >> 172.21.15.112:0/207321630 conn(0x7f4ce80fda30 msgr2=0x7f4ce806ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:52:32.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:32.118+0000 7f4ced77e700 1 -- 172.21.15.112:0/207321630 shutdown_connections 2024-06-04T04:52:32.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:32.118+0000 7f4ced77e700 1 -- 172.21.15.112:0/207321630 wait complete. 2024-06-04T04:52:32.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:32 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:32.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:32 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:32.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:32 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:32.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:32 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:52:32.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:32 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:32.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:32 smithi112 ceph-mon[36453]: from='osd.11 [v2:172.21.15.112:6824/1400793137,v1:172.21.15.112:6825/1400793137]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-04T04:52:32.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:32 smithi112 ceph-mon[36453]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-04T04:52:32.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:32 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:32.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:32 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:32.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:32 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:52:32.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:32 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:32.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:32 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:32.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:32 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:32.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:32 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:52:32.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:32 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:32.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:32 smithi098 ceph-mon[35569]: from='osd.11 [v2:172.21.15.112:6824/1400793137,v1:172.21.15.112:6825/1400793137]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-04T04:52:32.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:32 smithi098 ceph-mon[35569]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-04T04:52:32.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:32 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:32.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:32 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:32.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:32 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:52:32.791 DEBUG:teuthology.orchestra.run.smithi112:osd.11> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.11.service 2024-06-04T04:52:32.793 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-06-04T04:52:32.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-06-04T04:52:32.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:32 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:32.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:32 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:32.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:32 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:32.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:32 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:52:32.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:32 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:32.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:32 smithi070 ceph-mon[32631]: from='osd.11 [v2:172.21.15.112:6824/1400793137,v1:172.21.15.112:6825/1400793137]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-04T04:52:32.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:32 smithi070 ceph-mon[32631]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-04T04:52:32.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:32 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:32.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:32 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:32.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:32 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:52:33.042 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:52:33.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:33 smithi070 ceph-mon[32631]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:33.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:33 smithi070 ceph-mon[32631]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-06-04T04:52:33.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:33 smithi070 ceph-mon[32631]: osdmap e69: 12 total, 11 up, 12 in 2024-06-04T04:52:33.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:33 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:33.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:33 smithi070 ceph-mon[32631]: from='osd.11 [v2:172.21.15.112:6824/1400793137,v1:172.21.15.112:6825/1400793137]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:52:33.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:33 smithi070 ceph-mon[32631]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:52:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:33 smithi112 ceph-mon[36453]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:33 smithi112 ceph-mon[36453]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-06-04T04:52:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:33 smithi112 ceph-mon[36453]: osdmap e69: 12 total, 11 up, 12 in 2024-06-04T04:52:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:33 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:33 smithi112 ceph-mon[36453]: from='osd.11 [v2:172.21.15.112:6824/1400793137,v1:172.21.15.112:6825/1400793137]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:52:33.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:33 smithi112 ceph-mon[36453]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:52:33.687 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 04:52:33 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T04:52:33.390+0000 7f0ee52dc700 -1 osd.11 0 waiting for initial osdmap 2024-06-04T04:52:33.687 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 04:52:33 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T04:52:33.406+0000 7f0ee10cf700 -1 osd.11 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-04T04:52:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:33 smithi098 ceph-mon[35569]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:33 smithi098 ceph-mon[35569]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-06-04T04:52:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:33 smithi098 ceph-mon[35569]: osdmap e69: 12 total, 11 up, 12 in 2024-06-04T04:52:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:33 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:33 smithi098 ceph-mon[35569]: from='osd.11 [v2:172.21.15.112:6824/1400793137,v1:172.21.15.112:6825/1400793137]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:52:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:33 smithi098 ceph-mon[35569]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-06-04T04:52:34.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:34 smithi070 ceph-mon[32631]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-06-04T04:52:34.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:34 smithi070 ceph-mon[32631]: osdmap e70: 12 total, 11 up, 12 in 2024-06-04T04:52:34.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:34.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:34.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:34 smithi112 ceph-mon[36453]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-06-04T04:52:34.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:34 smithi112 ceph-mon[36453]: osdmap e70: 12 total, 11 up, 12 in 2024-06-04T04:52:34.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:34.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:34.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.721+0000 7fe4b4b30700 1 -- 172.21.15.70:0/943582142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4b0106960 msgr2=0x7fe4b0108d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:34.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.721+0000 7fe4b4b30700 1 --2- 172.21.15.70:0/943582142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4b0106960 0x7fe4b0108d50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4009990 tx=0x7fe4a4009ca0 comp rx=0 tx=0).stop 2024-06-04T04:52:34.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.721+0000 7fe4b4b30700 1 -- 172.21.15.70:0/943582142 shutdown_connections 2024-06-04T04:52:34.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.721+0000 7fe4b4b30700 1 --2- 172.21.15.70:0/943582142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4b0109290 0x7fe4b010b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:34.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.721+0000 7fe4b4b30700 1 --2- 172.21.15.70:0/943582142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b0104030 0x7fe4b0106420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:34.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.721+0000 7fe4b4b30700 1 --2- 172.21.15.70:0/943582142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4b0106960 0x7fe4b0108d50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:34.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.721+0000 7fe4b4b30700 1 -- 172.21.15.70:0/943582142 >> 172.21.15.70:0/943582142 conn(0x7fe4b00fda40 msgr2=0x7fe4b00ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:52:34.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.721+0000 7fe4b4b30700 1 -- 172.21.15.70:0/943582142 shutdown_connections 2024-06-04T04:52:34.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.721+0000 7fe4b4b30700 1 -- 172.21.15.70:0/943582142 wait complete. 2024-06-04T04:52:34.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.721+0000 7fe4b4b30700 1 Processor -- start 2024-06-04T04:52:34.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.721+0000 7fe4b4b30700 1 -- start start 2024-06-04T04:52:34.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4b4b30700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4b0104030 0x7fe4b01995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:34.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4b4b30700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4b0106960 0x7fe4b0199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:34.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4b4b30700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b0109290 0x7fe4b01a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:34.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4b4b30700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b010ef10 con 0x7fe4b0104030 2024-06-04T04:52:34.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4b4b30700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b010ed90 con 0x7fe4b0106960 2024-06-04T04:52:34.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4b4b30700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b010f090 con 0x7fe4b0109290 2024-06-04T04:52:34.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4ae59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4b0104030 0x7fe4b01995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:34.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4ae59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4b0104030 0x7fe4b01995c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51320/0 (socket says 172.21.15.70:51320) 2024-06-04T04:52:34.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4ae59c700 1 -- 172.21.15.70:0/4106373037 learned_addr learned my addr 172.21.15.70:0/4106373037 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:52:34.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4aed9d700 1 --2- 172.21.15.70:0/4106373037 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b0109290 0x7fe4b01a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:34.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4add9b700 1 --2- 172.21.15.70:0/4106373037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4b0106960 0x7fe4b0199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:34.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4ae59c700 1 -- 172.21.15.70:0/4106373037 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b0109290 msgr2=0x7fe4b01a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:34.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4ae59c700 1 --2- 172.21.15.70:0/4106373037 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b0109290 0x7fe4b01a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:34.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4ae59c700 1 -- 172.21.15.70:0/4106373037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4b0106960 msgr2=0x7fe4b0199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:34.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4ae59c700 1 --2- 172.21.15.70:0/4106373037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4b0106960 0x7fe4b0199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:34.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4ae59c700 1 -- 172.21.15.70:0/4106373037 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b01a1220 con 0x7fe4b0104030 2024-06-04T04:52:34.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4add9b700 1 --2- 172.21.15.70:0/4106373037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4b0106960 0x7fe4b0199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:52:34.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4ae59c700 1 --2- 172.21.15.70:0/4106373037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4b0104030 0x7fe4b01995c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000e9f0 tx=0x7fe4a000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:52:34.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe49f7fe700 1 -- 172.21.15.70:0/4106373037 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a000cbd0 con 0x7fe4b0104030 2024-06-04T04:52:34.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe4b4b30700 1 -- 172.21.15.70:0/4106373037 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4b01a1510 con 0x7fe4b0104030 2024-06-04T04:52:34.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe49f7fe700 1 -- 172.21.15.70:0/4106373037 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe4a000cd70 con 0x7fe4b0104030 2024-06-04T04:52:34.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.722+0000 7fe49f7fe700 1 -- 172.21.15.70:0/4106373037 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0005570 con 0x7fe4b0104030 2024-06-04T04:52:34.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.723+0000 7fe4b4b30700 1 -- 172.21.15.70:0/4106373037 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4b01a1a50 con 0x7fe4b0104030 2024-06-04T04:52:34.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.723+0000 7fe4b4b30700 1 -- 172.21.15.70:0/4106373037 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4b006a230 con 0x7fe4b0104030 2024-06-04T04:52:34.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.725+0000 7fe49f7fe700 1 -- 172.21.15.70:0/4106373037 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fe4a0004750 con 0x7fe4b0104030 2024-06-04T04:52:34.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.725+0000 7fe49f7fe700 1 --2- 172.21.15.70:0/4106373037 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe4980b2880 0x7fe4980b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:34.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.725+0000 7fe4add9b700 1 --2- 172.21.15.70:0/4106373037 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe4980b2880 0x7fe4980b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:34.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.726+0000 7fe49f7fe700 1 -- 172.21.15.70:0/4106373037 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fe4a00e2220 con 0x7fe4b0104030 2024-06-04T04:52:34.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.726+0000 7fe4add9b700 1 --2- 172.21.15.70:0/4106373037 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe4980b2880 0x7fe4980b4d40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe4a400b2e0 tx=0x7fe4a400b6d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:52:34.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.729+0000 7fe49f7fe700 1 -- 172.21.15.70:0/4106373037 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe4a0054df0 con 0x7fe4b0104030 2024-06-04T04:52:34.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:34 smithi098 ceph-mon[35569]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-06-04T04:52:34.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:34 smithi098 ceph-mon[35569]: osdmap e70: 12 total, 11 up, 12 in 2024-06-04T04:52:34.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:34.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:34.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.931+0000 7fe4b4b30700 1 -- 172.21.15.70:0/4106373037 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fe4b002e560 con 0x7fe4b0104030 2024-06-04T04:52:34.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.932+0000 7fe49f7fe700 1 -- 172.21.15.70:0/4106373037 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v71) ==== 74+0+133 (secure 0 0 0) 0x7fe4a00aa3e0 con 0x7fe4b0104030 2024-06-04T04:52:34.934 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:52:34.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.934+0000 7fe4b4b30700 1 -- 172.21.15.70:0/4106373037 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe4980b2880 msgr2=0x7fe4980b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:34.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.934+0000 7fe4b4b30700 1 --2- 172.21.15.70:0/4106373037 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe4980b2880 0x7fe4980b4d40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe4a400b2e0 tx=0x7fe4a400b6d0 comp rx=0 tx=0).stop 2024-06-04T04:52:34.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.934+0000 7fe4b4b30700 1 -- 172.21.15.70:0/4106373037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4b0104030 msgr2=0x7fe4b01995c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:34.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.934+0000 7fe4b4b30700 1 --2- 172.21.15.70:0/4106373037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4b0104030 0x7fe4b01995c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000e9f0 tx=0x7fe4a000ed00 comp rx=0 tx=0).stop 2024-06-04T04:52:34.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.934+0000 7fe4b4b30700 1 -- 172.21.15.70:0/4106373037 shutdown_connections 2024-06-04T04:52:34.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.934+0000 7fe4b4b30700 1 --2- 172.21.15.70:0/4106373037 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe4980b2880 0x7fe4980b4d40 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:34.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.934+0000 7fe4b4b30700 1 --2- 172.21.15.70:0/4106373037 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b0109290 0x7fe4b01a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:34.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.934+0000 7fe4b4b30700 1 --2- 172.21.15.70:0/4106373037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4b0104030 0x7fe4b01995c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:34.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.934+0000 7fe4b4b30700 1 --2- 172.21.15.70:0/4106373037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4b0106960 0x7fe4b0199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:34.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.934+0000 7fe4b4b30700 1 -- 172.21.15.70:0/4106373037 >> 172.21.15.70:0/4106373037 conn(0x7fe4b00fda40 msgr2=0x7fe4b0104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:52:34.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.934+0000 7fe4b4b30700 1 -- 172.21.15.70:0/4106373037 shutdown_connections 2024-06-04T04:52:34.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:34.934+0000 7fe4b4b30700 1 -- 172.21.15.70:0/4106373037 wait complete. 2024-06-04T04:52:35.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:35 smithi112 ceph-mon[36453]: purged_snaps scrub starts 2024-06-04T04:52:35.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:35 smithi112 ceph-mon[36453]: purged_snaps scrub ok 2024-06-04T04:52:35.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:35 smithi112 ceph-mon[36453]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:35.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:35 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:35.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:35 smithi112 ceph-mon[36453]: osd.11 [v2:172.21.15.112:6824/1400793137,v1:172.21.15.112:6825/1400793137] boot 2024-06-04T04:52:35.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:35 smithi112 ceph-mon[36453]: osdmap e71: 12 total, 12 up, 12 in 2024-06-04T04:52:35.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:35 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:35.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:35 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/4106373037' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-04T04:52:35.585 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:35 smithi070 ceph-mon[32631]: purged_snaps scrub starts 2024-06-04T04:52:35.585 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:35 smithi070 ceph-mon[32631]: purged_snaps scrub ok 2024-06-04T04:52:35.585 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:35 smithi070 ceph-mon[32631]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:35.585 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:35 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:35.585 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:35 smithi070 ceph-mon[32631]: osd.11 [v2:172.21.15.112:6824/1400793137,v1:172.21.15.112:6825/1400793137] boot 2024-06-04T04:52:35.585 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:35 smithi070 ceph-mon[32631]: osdmap e71: 12 total, 12 up, 12 in 2024-06-04T04:52:35.585 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:35 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:35.585 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:35 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/4106373037' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-04T04:52:35.613 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":71,"num_osds":12,"num_up_osds":12,"osd_up_since":1717476754,"num_in_osds":12,"osd_in_since":1717476734,"num_remapped_pgs":0} 2024-06-04T04:52:35.614 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-04T04:52:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:35 smithi098 ceph-mon[35569]: purged_snaps scrub starts 2024-06-04T04:52:35.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:35 smithi098 ceph-mon[35569]: purged_snaps scrub ok 2024-06-04T04:52:35.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:35 smithi098 ceph-mon[35569]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-06-04T04:52:35.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:35 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:35.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:35 smithi098 ceph-mon[35569]: osd.11 [v2:172.21.15.112:6824/1400793137,v1:172.21.15.112:6825/1400793137] boot 2024-06-04T04:52:35.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:35 smithi098 ceph-mon[35569]: osdmap e71: 12 total, 12 up, 12 in 2024-06-04T04:52:35.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:35 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:52:35.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:35 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/4106373037' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-04T04:52:35.900 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:52:36.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:36 smithi112 ceph-mon[36453]: osdmap e72: 12 total, 12 up, 12 in 2024-06-04T04:52:36.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:36 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:36.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:36 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:36.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:36 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:36.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:36 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:36.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:36 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:36.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:36 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:36.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:36 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:36.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:36 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:36.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:36 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:52:36.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:36 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:36.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:36 smithi070 ceph-mon[32631]: osdmap e72: 12 total, 12 up, 12 in 2024-06-04T04:52:36.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:36 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:36.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:36 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:36.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:36 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:36.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:36 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:36.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:36 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:36.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:36 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:36.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:36 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:36.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:36 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:36.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:36 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:52:36.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:36 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:36.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:36 smithi098 ceph-mon[35569]: osdmap e72: 12 total, 12 up, 12 in 2024-06-04T04:52:36.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:36 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:36.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:36 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:36.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:36 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:36.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:36 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:36.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:36 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:36.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:36 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:52:36.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:36 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:36.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:36 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:36.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:36 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:52:36.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:36 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:37.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.548+0000 7fc103734700 1 -- 172.21.15.70:0/3164497239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0fc108470 msgr2=0x7fc0fc06bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:37.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.548+0000 7fc103734700 1 --2- 172.21.15.70:0/3164497239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0fc108470 0x7fc0fc06bd00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009990 tx=0x7fc0f8009ca0 comp rx=0 tx=0).stop 2024-06-04T04:52:37.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.548+0000 7fc103734700 1 -- 172.21.15.70:0/3164497239 shutdown_connections 2024-06-04T04:52:37.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.548+0000 7fc103734700 1 --2- 172.21.15.70:0/3164497239 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0fc105f80 0x7fc0fc106410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:37.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.548+0000 7fc103734700 1 --2- 172.21.15.70:0/3164497239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0fc108470 0x7fc0fc06bd00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:37.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.548+0000 7fc103734700 1 --2- 172.21.15.70:0/3164497239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0fc06c240 0x7fc0fc06c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:37.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.548+0000 7fc103734700 1 -- 172.21.15.70:0/3164497239 >> 172.21.15.70:0/3164497239 conn(0x7fc0fc0793f0 msgr2=0x7fc0fc079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:52:37.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.548+0000 7fc103734700 1 -- 172.21.15.70:0/3164497239 shutdown_connections 2024-06-04T04:52:37.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.548+0000 7fc103734700 1 -- 172.21.15.70:0/3164497239 wait complete. 2024-06-04T04:52:37.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.548+0000 7fc103734700 1 Processor -- start 2024-06-04T04:52:37.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc103734700 1 -- start start 2024-06-04T04:52:37.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc103734700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0fc06c240 0x7fc0fc1a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:37.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc103734700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0fc105f80 0x7fc0fc1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:37.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc103734700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0fc108470 0x7fc0fc1a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:37.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc103734700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc0fc119be0 con 0x7fc0fc06c240 2024-06-04T04:52:37.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc103734700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc0fc119a60 con 0x7fc0fc108470 2024-06-04T04:52:37.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc103734700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc0fc119d60 con 0x7fc0fc105f80 2024-06-04T04:52:37.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc1014d0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0fc06c240 0x7fc0fc1a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:37.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc1014d0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0fc06c240 0x7fc0fc1a0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44238/0 (socket says 172.21.15.70:44238) 2024-06-04T04:52:37.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc1014d0700 1 -- 172.21.15.70:0/3372946944 learned_addr learned my addr 172.21.15.70:0/3372946944 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:52:37.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc101cd1700 1 --2- 172.21.15.70:0/3372946944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0fc108470 0x7fc0fc1a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:37.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc100ccf700 1 --2- 172.21.15.70:0/3372946944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0fc105f80 0x7fc0fc1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:37.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc1014d0700 1 -- 172.21.15.70:0/3372946944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0fc105f80 msgr2=0x7fc0fc1a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:37.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc1014d0700 1 --2- 172.21.15.70:0/3372946944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0fc105f80 0x7fc0fc1a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:37.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc1014d0700 1 -- 172.21.15.70:0/3372946944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0fc108470 msgr2=0x7fc0fc1a49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:37.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc1014d0700 1 --2- 172.21.15.70:0/3372946944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0fc108470 0x7fc0fc1a49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:37.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc1014d0700 1 -- 172.21.15.70:0/3372946944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0fc1a5120 con 0x7fc0fc06c240 2024-06-04T04:52:37.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc100ccf700 1 --2- 172.21.15.70:0/3372946944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0fc105f80 0x7fc0fc1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:52:37.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.549+0000 7fc1014d0700 1 --2- 172.21.15.70:0/3372946944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0fc06c240 0x7fc0fc1a0110 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8000c00 tx=0x7fc0f80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:52:37.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.550+0000 7fc0f27fc700 1 -- 172.21.15.70:0/3372946944 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8027070 con 0x7fc0fc06c240 2024-06-04T04:52:37.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.550+0000 7fc0f27fc700 1 -- 172.21.15.70:0/3372946944 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc0f800fd00 con 0x7fc0fc06c240 2024-06-04T04:52:37.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.550+0000 7fc103734700 1 -- 172.21.15.70:0/3372946944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0fc1a53b0 con 0x7fc0fc06c240 2024-06-04T04:52:37.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.550+0000 7fc0f27fc700 1 -- 172.21.15.70:0/3372946944 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8022e20 con 0x7fc0fc06c240 2024-06-04T04:52:37.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.550+0000 7fc103734700 1 -- 172.21.15.70:0/3372946944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0fc1acc90 con 0x7fc0fc06c240 2024-06-04T04:52:37.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.551+0000 7fc103734700 1 -- 172.21.15.70:0/3372946944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0fc102a60 con 0x7fc0fc06c240 2024-06-04T04:52:37.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.552+0000 7fc0f27fc700 1 -- 172.21.15.70:0/3372946944 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fc0f802a070 con 0x7fc0fc06c240 2024-06-04T04:52:37.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.553+0000 7fc0f27fc700 1 --2- 172.21.15.70:0/3372946944 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc0e80b2880 0x7fc0e80b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:37.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.553+0000 7fc100ccf700 1 --2- 172.21.15.70:0/3372946944 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc0e80b2880 0x7fc0e80b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:37.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.553+0000 7fc0f27fc700 1 -- 172.21.15.70:0/3372946944 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fc0f80e2db0 con 0x7fc0fc06c240 2024-06-04T04:52:37.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.553+0000 7fc100ccf700 1 --2- 172.21.15.70:0/3372946944 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc0e80b2880 0x7fc0e80b4d40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec005fd0 tx=0x7fc0ec005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:52:37.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.556+0000 7fc0f27fc700 1 -- 172.21.15.70:0/3372946944 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc0f8013070 con 0x7fc0fc06c240 2024-06-04T04:52:37.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:37 smithi070 ceph-mon[32631]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:37.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:37 smithi070 ceph-mon[32631]: Detected new or changed devices on smithi112 2024-06-04T04:52:37.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:37 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi112 to 4272M 2024-06-04T04:52:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:37 smithi112 ceph-mon[36453]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:37 smithi112 ceph-mon[36453]: Detected new or changed devices on smithi112 2024-06-04T04:52:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:37 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi112 to 4272M 2024-06-04T04:52:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:37 smithi098 ceph-mon[35569]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:37.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:37 smithi098 ceph-mon[35569]: Detected new or changed devices on smithi112 2024-06-04T04:52:37.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:37 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi112 to 4272M 2024-06-04T04:52:37.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.765+0000 7fc103734700 1 -- 172.21.15.70:0/3372946944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc0fc06a230 con 0x7fc0fc06c240 2024-06-04T04:52:37.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.766+0000 7fc0f27fc700 1 -- 172.21.15.70:0/3372946944 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18420 (secure 0 0 0) 0x7fc0fc06a230 con 0x7fc0fc06c240 2024-06-04T04:52:37.768 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:52:37.768 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":72,"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","created":"2024-06-04T04:45:22.297536+0000","modified":"2024-06-04T04:52:35.393927+0000","last_up_change":"2024-06-04T04:52:34.388708+0000","last_in_change":"2024-06-04T04:52:14.197780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-04T04:48:43.724356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ff491ea2-1906-48e4-883e-bc78e778260e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6803","nonce":824119100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6805","nonce":824119100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6809","nonce":824119100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6807","nonce":824119100}]},"public_addr":"172.21.15.70:6803/824119100","cluster_addr":"172.21.15.70:6805/824119100","heartbeat_back_addr":"172.21.15.70:6809/824119100","heartbeat_front_addr":"172.21.15.70:6807/824119100","state":["exists","up"]},{"osd":1,"uuid":"bd5e4806-535b-49b5-ba6b-93228da35a84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6801","nonce":2744598742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6803","nonce":2744598742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6807","nonce":2744598742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6805","nonce":2744598742}]},"public_addr":"172.21.15.98:6801/2744598742","cluster_addr":"172.21.15.98:6803/2744598742","heartbeat_back_addr":"172.21.15.98:6807/2744598742","heartbeat_front_addr":"172.21.15.98:6805/2744598742","state":["exists","up"]},{"osd":2,"uuid":"8456fb55-910a-4a59-b84b-2c0f68346ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6801","nonce":1328098370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6803","nonce":1328098370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6807","nonce":1328098370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6805","nonce":1328098370}]},"public_addr":"172.21.15.112:6801/1328098370","cluster_addr":"172.21.15.112:6803/1328098370","heartbeat_back_addr":"172.21.15.112:6807/1328098370","heartbeat_front_addr":"172.21.15.112:6805/1328098370","state":["exists","up"]},{"osd":3,"uuid":"c8fbc440-bf70-42d3-b01e-777ccfa7900b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6811","nonce":4041566346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6813","nonce":4041566346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6817","nonce":4041566346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6815","nonce":4041566346}]},"public_addr":"172.21.15.70:6811/4041566346","cluster_addr":"172.21.15.70:6813/4041566346","heartbeat_back_addr":"172.21.15.70:6817/4041566346","heartbeat_front_addr":"172.21.15.70:6815/4041566346","state":["exists","up"]},{"osd":4,"uuid":"549c8c70-8be8-46ae-bcb6-0fde32b40cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6809","nonce":3652775814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6811","nonce":3652775814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6815","nonce":3652775814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6813","nonce":3652775814}]},"public_addr":"172.21.15.98:6809/3652775814","cluster_addr":"172.21.15.98:6811/3652775814","heartbeat_back_addr":"172.21.15.98:6815/3652775814","heartbeat_front_addr":"172.21.15.98:6813/3652775814","state":["exists","up"]},{"osd":5,"uuid":"51d86af1-0d6c-4ae0-b090-00414fc1bf7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6809","nonce":1857257899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6811","nonce":1857257899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6815","nonce":1857257899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6813","nonce":1857257899}]},"public_addr":"172.21.15.112:6809/1857257899","cluster_addr":"172.21.15.112:6811/1857257899","heartbeat_back_addr":"172.21.15.112:6815/1857257899","heartbeat_front_addr":"172.21.15.112:6813/1857257899","state":["exists","up"]},{"osd":6,"uuid":"526a8138-bfa8-419e-a5b5-e6b80f0ae004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6819","nonce":1264676496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6821","nonce":1264676496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6825","nonce":1264676496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6823","nonce":1264676496}]},"public_addr":"172.21.15.70:6819/1264676496","cluster_addr":"172.21.15.70:6821/1264676496","heartbeat_back_addr":"172.21.15.70:6825/1264676496","heartbeat_front_addr":"172.21.15.70:6823/1264676496","state":["exists","up"]},{"osd":7,"uuid":"1173f785-6ed1-47ef-9265-2e233ea9ee02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6817","nonce":709593020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6819","nonce":709593020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6823","nonce":709593020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6821","nonce":709593020}]},"public_addr":"172.21.15.98:6817/709593020","cluster_addr":"172.21.15.98:6819/709593020","heartbeat_back_addr":"172.21.15.98:6823/709593020","heartbeat_front_addr":"172.21.15.98:6821/709593020","state":["exists","up"]},{"osd":8,"uuid":"6424cc6e-24a7-4b81-9ce6-39203fecc7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6817","nonce":2250225413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6819","nonce":2250225413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6823","nonce":2250225413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6821","nonce":2250225413}]},"public_addr":"172.21.15.112:6817/2250225413","cluster_addr":"172.21.15.112:6819/2250225413","heartbeat_back_addr":"172.21.15.112:6823/2250225413","heartbeat_front_addr":"172.21.15.112:6821/2250225413","state":["exists","up"]},{"osd":9,"uuid":"8d666db8-8831-4d63-9e3c-a4bf2504149e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6827","nonce":3557688707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6829","nonce":3557688707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6833","nonce":3557688707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6831","nonce":3557688707}]},"public_addr":"172.21.15.70:6827/3557688707","cluster_addr":"172.21.15.70:6829/3557688707","heartbeat_back_addr":"172.21.15.70:6833/3557688707","heartbeat_front_addr":"172.21.15.70:6831/3557688707","state":["exists","up"]},{"osd":10,"uuid":"ecb925d2-0263-4dfc-a2fd-9e27856022d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6825","nonce":1896493718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6827","nonce":1896493718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6831","nonce":1896493718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6829","nonce":1896493718}]},"public_addr":"172.21.15.98:6825/1896493718","cluster_addr":"172.21.15.98:6827/1896493718","heartbeat_back_addr":"172.21.15.98:6831/1896493718","heartbeat_front_addr":"172.21.15.98:6829/1896493718","state":["exists","up"]},{"osd":11,"uuid":"c8e3230c-bc3a-40bd-8dd0-5fef56e4df97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6825","nonce":1400793137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6827","nonce":1400793137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6831","nonce":1400793137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6829","nonce":1400793137}]},"public_addr":"172.21.15.112:6825/1400793137","cluster_addr":"172.21.15.112:6827/1400793137","heartbeat_back_addr":"172.21.15.112:6831/1400793137","heartbeat_front_addr":"172.21.15.112:6829/1400793137","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:47:53.626608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:17.784505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:41.971969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:07.130699+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:31.615496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:56.898607+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:22.739991+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:48.319727+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:13.902063+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:40.389470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:06.142092+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:32.949383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/3442074771":"2024-06-05T04:46:17.636907+0000","172.21.15.70:6801/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/1442095493":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/2453733268":"2024-06-05T04:45:49.501864+0000","172.21.15.70:0/1369781447":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6801/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/915186456":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/860033006":"2024-06-05T04:45:49.501864+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-04T04:52:37.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.768+0000 7fc103734700 1 -- 172.21.15.70:0/3372946944 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc0e80b2880 msgr2=0x7fc0e80b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:37.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.768+0000 7fc103734700 1 --2- 172.21.15.70:0/3372946944 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc0e80b2880 0x7fc0e80b4d40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec005fd0 tx=0x7fc0ec005e20 comp rx=0 tx=0).stop 2024-06-04T04:52:37.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.768+0000 7fc103734700 1 -- 172.21.15.70:0/3372946944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0fc06c240 msgr2=0x7fc0fc1a0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:37.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.768+0000 7fc103734700 1 --2- 172.21.15.70:0/3372946944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0fc06c240 0x7fc0fc1a0110 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8000c00 tx=0x7fc0f80047b0 comp rx=0 tx=0).stop 2024-06-04T04:52:37.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.768+0000 7fc103734700 1 -- 172.21.15.70:0/3372946944 shutdown_connections 2024-06-04T04:52:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.768+0000 7fc103734700 1 --2- 172.21.15.70:0/3372946944 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc0e80b2880 0x7fc0e80b4d40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.768+0000 7fc103734700 1 --2- 172.21.15.70:0/3372946944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0fc108470 0x7fc0fc1a49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.768+0000 7fc103734700 1 --2- 172.21.15.70:0/3372946944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0fc06c240 0x7fc0fc1a0110 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.768+0000 7fc103734700 1 --2- 172.21.15.70:0/3372946944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0fc105f80 0x7fc0fc1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.768+0000 7fc103734700 1 -- 172.21.15.70:0/3372946944 >> 172.21.15.70:0/3372946944 conn(0x7fc0fc0793f0 msgr2=0x7fc0fc104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:52:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.768+0000 7fc103734700 1 -- 172.21.15.70:0/3372946944 shutdown_connections 2024-06-04T04:52:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:37.768+0000 7fc103734700 1 -- 172.21.15.70:0/3372946944 wait complete. 2024-06-04T04:52:38.483 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-04T04:48:43.724356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-04T04:52:38.484 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-06-04T04:52:38.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:38 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3372946944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:52:38.719 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:52:38.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:38 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3372946944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:52:38.841 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:38 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3372946944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:52:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:39 smithi112 ceph-mon[36453]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:39 smithi098 ceph-mon[35569]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:39 smithi070 ceph-mon[32631]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:40.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.447+0000 7f577a138700 1 -- 172.21.15.70:0/1639444088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57741088f0 msgr2=0x7f5774110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:40.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.447+0000 7f577a138700 1 --2- 172.21.15.70:0/1639444088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57741088f0 0x7f5774110470 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f576c00b0d0 tx=0x7f576c00b3e0 comp rx=0 tx=0).stop 2024-06-04T04:52:40.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.447+0000 7f577a138700 1 -- 172.21.15.70:0/1639444088 shutdown_connections 2024-06-04T04:52:40.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.447+0000 7f577a138700 1 --2- 172.21.15.70:0/1639444088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57741088f0 0x7f5774110470 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:40.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.447+0000 7f577a138700 1 --2- 172.21.15.70:0/1639444088 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57741037f0 0x7f5774103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:40.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.447+0000 7f577a138700 1 --2- 172.21.15.70:0/1639444088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57741041a0 0x7f57741081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:40.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.447+0000 7f577a138700 1 -- 172.21.15.70:0/1639444088 >> 172.21.15.70:0/1639444088 conn(0x7f577407a6a0 msgr2=0x7f577407aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:52:40.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.447+0000 7f577a138700 1 -- 172.21.15.70:0/1639444088 shutdown_connections 2024-06-04T04:52:40.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.447+0000 7f577a138700 1 -- 172.21.15.70:0/1639444088 wait complete. 2024-06-04T04:52:40.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.448+0000 7f577a138700 1 Processor -- start 2024-06-04T04:52:40.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.448+0000 7f577a138700 1 -- start start 2024-06-04T04:52:40.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.448+0000 7f577a138700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57741037f0 0x7f577419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:40.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.448+0000 7f577a138700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57741041a0 0x7f577419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:40.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.448+0000 7f577a138700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57741088f0 0x7f57741998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:40.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.448+0000 7f577a138700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5774113570 con 0x7f57741041a0 2024-06-04T04:52:40.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.448+0000 7f577a138700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f57741133f0 con 0x7f57741037f0 2024-06-04T04:52:40.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.448+0000 7f577a138700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f57741136f0 con 0x7f57741088f0 2024-06-04T04:52:40.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.448+0000 7f576affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57741041a0 0x7f577419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:40.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.448+0000 7f576affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57741041a0 0x7f577419fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44242/0 (socket says 172.21.15.70:44242) 2024-06-04T04:52:40.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.448+0000 7f576affd700 1 -- 172.21.15.70:0/144238843 learned_addr learned my addr 172.21.15.70:0/144238843 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:52:40.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.448+0000 7f57737fe700 1 --2- 172.21.15.70:0/144238843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57741037f0 0x7f577419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:40.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.448+0000 7f5773fff700 1 --2- 172.21.15.70:0/144238843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57741088f0 0x7f57741998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:40.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.449+0000 7f576affd700 1 -- 172.21.15.70:0/144238843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57741088f0 msgr2=0x7f57741998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:40.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.449+0000 7f576affd700 1 --2- 172.21.15.70:0/144238843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57741088f0 0x7f57741998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:40.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.449+0000 7f576affd700 1 -- 172.21.15.70:0/144238843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57741037f0 msgr2=0x7f577419f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:40.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.449+0000 7f576affd700 1 --2- 172.21.15.70:0/144238843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57741037f0 0x7f577419f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:40.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.449+0000 7f576affd700 1 -- 172.21.15.70:0/144238843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f577419a010 con 0x7f57741041a0 2024-06-04T04:52:40.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.449+0000 7f576affd700 1 --2- 172.21.15.70:0/144238843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57741041a0 0x7f577419fbe0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f575c00d950 tx=0x7f575c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:52:40.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.449+0000 7f57717fa700 1 -- 172.21.15.70:0/144238843 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c016070 con 0x7f57741041a0 2024-06-04T04:52:40.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.449+0000 7f577a138700 1 -- 172.21.15.70:0/144238843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f577419a300 con 0x7f57741041a0 2024-06-04T04:52:40.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.449+0000 7f57717fa700 1 -- 172.21.15.70:0/144238843 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f575c00fd20 con 0x7f57741041a0 2024-06-04T04:52:40.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.449+0000 7f57717fa700 1 -- 172.21.15.70:0/144238843 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c00e3b0 con 0x7f57741041a0 2024-06-04T04:52:40.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.449+0000 7f577a138700 1 -- 172.21.15.70:0/144238843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57741a6570 con 0x7f57741041a0 2024-06-04T04:52:40.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.450+0000 7f577a138700 1 -- 172.21.15.70:0/144238843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f577406a230 con 0x7f57741041a0 2024-06-04T04:52:40.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.451+0000 7f57717fa700 1 -- 172.21.15.70:0/144238843 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f575c019070 con 0x7f57741041a0 2024-06-04T04:52:40.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.452+0000 7f57717fa700 1 --2- 172.21.15.70:0/144238843 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f57600b2880 0x7f57600b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:40.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.452+0000 7f57737fe700 1 --2- 172.21.15.70:0/144238843 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f57600b2880 0x7f57600b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:40.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.452+0000 7f57717fa700 1 -- 172.21.15.70:0/144238843 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f575c0e17d0 con 0x7f57741041a0 2024-06-04T04:52:40.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.452+0000 7f57737fe700 1 --2- 172.21.15.70:0/144238843 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f57600b2880 0x7f57600b4d40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5764005950 tx=0x7f576400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:52:40.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.455+0000 7f57717fa700 1 -- 172.21.15.70:0/144238843 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f575c0a9990 con 0x7f57741041a0 2024-06-04T04:52:40.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.662+0000 7f577a138700 1 -- 172.21.15.70:0/144238843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f577402e560 con 0x7f57741041a0 2024-06-04T04:52:40.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.662+0000 7f57717fa700 1 -- 172.21.15.70:0/144238843 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7f575c0a9290 con 0x7f57741041a0 2024-06-04T04:52:40.665 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-06-04T04:52:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.665+0000 7f577a138700 1 -- 172.21.15.70:0/144238843 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f57600b2880 msgr2=0x7f57600b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.665+0000 7f577a138700 1 --2- 172.21.15.70:0/144238843 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f57600b2880 0x7f57600b4d40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5764005950 tx=0x7f576400a300 comp rx=0 tx=0).stop 2024-06-04T04:52:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.665+0000 7f577a138700 1 -- 172.21.15.70:0/144238843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57741041a0 msgr2=0x7f577419fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.665+0000 7f577a138700 1 --2- 172.21.15.70:0/144238843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57741041a0 0x7f577419fbe0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f575c00d950 tx=0x7f575c00dc60 comp rx=0 tx=0).stop 2024-06-04T04:52:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.665+0000 7f577a138700 1 -- 172.21.15.70:0/144238843 shutdown_connections 2024-06-04T04:52:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.665+0000 7f577a138700 1 --2- 172.21.15.70:0/144238843 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f57600b2880 0x7f57600b4d40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.665+0000 7f577a138700 1 --2- 172.21.15.70:0/144238843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57741088f0 0x7f57741998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.665+0000 7f577a138700 1 --2- 172.21.15.70:0/144238843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57741037f0 0x7f577419f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.665+0000 7f577a138700 1 --2- 172.21.15.70:0/144238843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57741041a0 0x7f577419fbe0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:40.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.665+0000 7f577a138700 1 -- 172.21.15.70:0/144238843 >> 172.21.15.70:0/144238843 conn(0x7f577407a6a0 msgr2=0x7f5774101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:52:40.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.665+0000 7f577a138700 1 -- 172.21.15.70:0/144238843 shutdown_connections 2024-06-04T04:52:40.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:40.665+0000 7f577a138700 1 -- 172.21.15.70:0/144238843 wait complete. 2024-06-04T04:52:41.486 INFO:tasks.cephadm:Adding mds.a on smithi070 2024-06-04T04:52:41.487 INFO:tasks.cephadm:Adding mds.d on smithi070 2024-06-04T04:52:41.487 INFO:tasks.cephadm:Adding mds.g on smithi070 2024-06-04T04:52:41.487 INFO:tasks.cephadm:Adding mds.j on smithi070 2024-06-04T04:52:41.487 INFO:tasks.cephadm:Adding mds.b on smithi098 2024-06-04T04:52:41.487 INFO:tasks.cephadm:Adding mds.e on smithi098 2024-06-04T04:52:41.487 INFO:tasks.cephadm:Adding mds.h on smithi098 2024-06-04T04:52:41.487 INFO:tasks.cephadm:Adding mds.k on smithi098 2024-06-04T04:52:41.487 INFO:tasks.cephadm:Adding mds.c on smithi112 2024-06-04T04:52:41.487 INFO:tasks.cephadm:Adding mds.f on smithi112 2024-06-04T04:52:41.487 INFO:tasks.cephadm:Adding mds.i on smithi112 2024-06-04T04:52:41.487 INFO:tasks.cephadm:Adding mds.l on smithi112 2024-06-04T04:52:41.488 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph orch apply mds all '12;smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi112=c;smithi112=f;smithi112=i;smithi112=l' 2024-06-04T04:52:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:41 smithi112 ceph-mon[36453]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:41 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/144238843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-04T04:52:41.744 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.c/config 2024-06-04T04:52:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:41 smithi098 ceph-mon[35569]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:41 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/144238843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-04T04:52:41.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:41 smithi070 ceph-mon[32631]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:41.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:41 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/144238843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-04T04:52:43.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.391+0000 7f1c4cfbc700 1 -- 172.21.15.112:0/3155854795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c48101890 msgr2=0x7f1c48101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:43.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.391+0000 7f1c4cfbc700 1 --2- 172.21.15.112:0/3155854795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c48101890 0x7f1c48101cb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009990 tx=0x7f1c38009ca0 comp rx=0 tx=0).stop 2024-06-04T04:52:43.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.392+0000 7f1c4cfbc700 1 -- 172.21.15.112:0/3155854795 shutdown_connections 2024-06-04T04:52:43.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.392+0000 7f1c4cfbc700 1 --2- 172.21.15.112:0/3155854795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c481052b0 0x7f1c4810bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:43.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.392+0000 7f1c4cfbc700 1 --2- 172.21.15.112:0/3155854795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c48101890 0x7f1c48101cb0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:43.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.392+0000 7f1c4cfbc700 1 --2- 172.21.15.112:0/3155854795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c48104910 0x7f1c48104d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:43.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.392+0000 7f1c4cfbc700 1 -- 172.21.15.112:0/3155854795 >> 172.21.15.112:0/3155854795 conn(0x7f1c480fd1b0 msgr2=0x7f1c480ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:52:43.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.392+0000 7f1c4cfbc700 1 -- 172.21.15.112:0/3155854795 shutdown_connections 2024-06-04T04:52:43.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.392+0000 7f1c4cfbc700 1 -- 172.21.15.112:0/3155854795 wait complete. 2024-06-04T04:52:43.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.392+0000 7f1c4cfbc700 1 Processor -- start 2024-06-04T04:52:43.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.392+0000 7f1c4cfbc700 1 -- start start 2024-06-04T04:52:43.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c4cfbc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c48101890 0x7f1c48104270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:43.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c4cfbc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c48104910 0x7f1c481028c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:43.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c4cfbc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c481052b0 0x7f1c48102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:43.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c4cfbc700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4810ef80 con 0x7f1c48101890 2024-06-04T04:52:43.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c4cfbc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4810ee00 con 0x7f1c48104910 2024-06-04T04:52:43.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c4cfbc700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4810f100 con 0x7f1c481052b0 2024-06-04T04:52:43.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c46d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c481052b0 0x7f1c48102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:43.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c46d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c481052b0 0x7f1c48102e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:37280/0 (socket says 172.21.15.112:37280) 2024-06-04T04:52:43.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c46d9d700 1 -- 172.21.15.112:0/4005179391 learned_addr learned my addr 172.21.15.112:0/4005179391 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-06-04T04:52:43.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c4659c700 1 --2- 172.21.15.112:0/4005179391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c48101890 0x7f1c48104270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:43.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c45d9b700 1 --2- 172.21.15.112:0/4005179391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c48104910 0x7f1c481028c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:43.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c46d9d700 1 -- 172.21.15.112:0/4005179391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c48104910 msgr2=0x7f1c481028c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:43.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c46d9d700 1 --2- 172.21.15.112:0/4005179391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c48104910 0x7f1c481028c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:43.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c46d9d700 1 -- 172.21.15.112:0/4005179391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c48101890 msgr2=0x7f1c48104270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:43.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c46d9d700 1 --2- 172.21.15.112:0/4005179391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c48101890 0x7f1c48104270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:43.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c46d9d700 1 -- 172.21.15.112:0/4005179391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c481036b0 con 0x7f1c481052b0 2024-06-04T04:52:43.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c46d9d700 1 --2- 172.21.15.112:0/4005179391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c481052b0 0x7f1c48102e00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000ada0 tx=0x7f1c4000c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:52:43.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c377fe700 1 -- 172.21.15.112:0/4005179391 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c40017020 con 0x7f1c481052b0 2024-06-04T04:52:43.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c377fe700 1 -- 172.21.15.112:0/4005179391 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1c4000ce10 con 0x7f1c481052b0 2024-06-04T04:52:43.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c4cfbc700 1 -- 172.21.15.112:0/4005179391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c481a3840 con 0x7f1c481052b0 2024-06-04T04:52:43.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c377fe700 1 -- 172.21.15.112:0/4005179391 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c40005280 con 0x7f1c481052b0 2024-06-04T04:52:43.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.393+0000 7f1c4cfbc700 1 -- 172.21.15.112:0/4005179391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c481a3e00 con 0x7f1c481052b0 2024-06-04T04:52:43.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.394+0000 7f1c4cfbc700 1 -- 172.21.15.112:0/4005179391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c4806a230 con 0x7f1c481052b0 2024-06-04T04:52:43.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.396+0000 7f1c377fe700 1 -- 172.21.15.112:0/4005179391 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f1c4000c980 con 0x7f1c481052b0 2024-06-04T04:52:43.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.396+0000 7f1c377fe700 1 --2- 172.21.15.112:0/4005179391 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1c300b2900 0x7f1c300b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:43.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.396+0000 7f1c377fe700 1 -- 172.21.15.112:0/4005179391 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f1c400e19e0 con 0x7f1c481052b0 2024-06-04T04:52:43.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.397+0000 7f1c4659c700 1 --2- 172.21.15.112:0/4005179391 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1c300b2900 0x7f1c300b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:43.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.397+0000 7f1c4659c700 1 --2- 172.21.15.112:0/4005179391 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1c300b2900 0x7f1c300b4dc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1c380036f0 tx=0x7f1c38005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:52:43.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.400+0000 7f1c377fe700 1 -- 172.21.15.112:0/4005179391 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1c40021020 con 0x7f1c481052b0 2024-06-04T04:52:43.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.616+0000 7f1c4cfbc700 1 -- 172.21.15.112:0/4005179391 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi112=c;smithi112=f;smithi112=i;smithi112=l", "target": ["mon-mgr", ""]}) -- 0x7f1c48063b80 con 0x7f1c300b2900 2024-06-04T04:52:43.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.624+0000 7f1c377fe700 1 -- 172.21.15.112:0/4005179391 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f1c48063b80 con 0x7f1c300b2900 2024-06-04T04:52:43.626 INFO:teuthology.orchestra.run.smithi112.stdout:Scheduled mds.all update... 2024-06-04T04:52:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.626+0000 7f1c4cfbc700 1 -- 172.21.15.112:0/4005179391 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1c300b2900 msgr2=0x7f1c300b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.626+0000 7f1c4cfbc700 1 --2- 172.21.15.112:0/4005179391 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1c300b2900 0x7f1c300b4dc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1c380036f0 tx=0x7f1c38005fd0 comp rx=0 tx=0).stop 2024-06-04T04:52:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.626+0000 7f1c4cfbc700 1 -- 172.21.15.112:0/4005179391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c481052b0 msgr2=0x7f1c48102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.626+0000 7f1c4cfbc700 1 --2- 172.21.15.112:0/4005179391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c481052b0 0x7f1c48102e00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000ada0 tx=0x7f1c4000c330 comp rx=0 tx=0).stop 2024-06-04T04:52:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.626+0000 7f1c4cfbc700 1 -- 172.21.15.112:0/4005179391 shutdown_connections 2024-06-04T04:52:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.626+0000 7f1c4cfbc700 1 --2- 172.21.15.112:0/4005179391 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1c300b2900 0x7f1c300b4dc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.626+0000 7f1c4cfbc700 1 --2- 172.21.15.112:0/4005179391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c481052b0 0x7f1c48102e00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.626+0000 7f1c4cfbc700 1 --2- 172.21.15.112:0/4005179391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c48101890 0x7f1c48104270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.626+0000 7f1c4cfbc700 1 --2- 172.21.15.112:0/4005179391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c48104910 0x7f1c481028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.626+0000 7f1c4cfbc700 1 -- 172.21.15.112:0/4005179391 >> 172.21.15.112:0/4005179391 conn(0x7f1c480fd1b0 msgr2=0x7f1c48107d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:52:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.627+0000 7f1c4cfbc700 1 -- 172.21.15.112:0/4005179391 shutdown_connections 2024-06-04T04:52:43.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-06-04T04:52:43.627+0000 7f1c4cfbc700 1 -- 172.21.15.112:0/4005179391 wait complete. 2024-06-04T04:52:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:43 smithi112 ceph-mon[36453]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:43 smithi098 ceph-mon[35569]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:43.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:43 smithi070 ceph-mon[32631]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:44.255 DEBUG:teuthology.orchestra.run.smithi070:mds.a> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.a.service 2024-06-04T04:52:44.258 DEBUG:teuthology.orchestra.run.smithi070:mds.d> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.d.service 2024-06-04T04:52:44.301 DEBUG:teuthology.orchestra.run.smithi070:mds.g> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.g.service 2024-06-04T04:52:44.344 DEBUG:teuthology.orchestra.run.smithi070:mds.j> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.j.service 2024-06-04T04:52:44.387 DEBUG:teuthology.orchestra.run.smithi098:mds.b> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.b.service 2024-06-04T04:52:44.389 DEBUG:teuthology.orchestra.run.smithi098:mds.e> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.e.service 2024-06-04T04:52:44.432 DEBUG:teuthology.orchestra.run.smithi098:mds.h> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.h.service 2024-06-04T04:52:44.474 DEBUG:teuthology.orchestra.run.smithi098:mds.k> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.k.service 2024-06-04T04:52:44.517 DEBUG:teuthology.orchestra.run.smithi112:mds.c> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.c.service 2024-06-04T04:52:44.520 DEBUG:teuthology.orchestra.run.smithi112:mds.f> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.f.service 2024-06-04T04:52:44.563 DEBUG:teuthology.orchestra.run.smithi112:mds.i> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.i.service 2024-06-04T04:52:44.607 DEBUG:teuthology.orchestra.run.smithi112:mds.l> sudo journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.l.service 2024-06-04T04:52:44.651 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-06-04T04:52:44.651 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:52:44.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:44 smithi098 ceph-mon[35569]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi112=c;smithi112=f;smithi112=i;smithi112=l", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:52:44.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:44 smithi098 ceph-mon[35569]: Saving service mds.all spec with placement smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi112=c;smithi112=f;smithi112=i;smithi112=l;count:12 2024-06-04T04:52:44.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:44 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:44.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:44 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:52:44.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:44 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:44.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:44 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:52:44.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:44 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:44.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:44 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:52:44.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:44 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:52:44.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:44 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:44.886 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:52:44.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:44 smithi112 ceph-mon[36453]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi112=c;smithi112=f;smithi112=i;smithi112=l", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:52:44.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:44 smithi112 ceph-mon[36453]: Saving service mds.all spec with placement smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi112=c;smithi112=f;smithi112=i;smithi112=l;count:12 2024-06-04T04:52:44.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:44 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:44.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:44 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:52:44.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:44 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:44.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:44 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:52:44.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:44 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:44.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:44 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:52:44.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:44 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:52:44.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:44 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:45.008 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:44 smithi070 ceph-mon[32631]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi112=c;smithi112=f;smithi112=i;smithi112=l", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:52:45.008 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:44 smithi070 ceph-mon[32631]: Saving service mds.all spec with placement smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi112=c;smithi112=f;smithi112=i;smithi112=l;count:12 2024-06-04T04:52:45.008 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:44 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:45.008 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:44 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:52:45.008 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:44 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:45.008 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:44 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:52:45.008 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:44 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:45.008 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:44 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:52:45.008 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:44 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:52:45.008 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:44 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:45.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:45 smithi112 ceph-mon[36453]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:45.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:45 smithi112 ceph-mon[36453]: Deploying daemon mds.l on smithi112 2024-06-04T04:52:45.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:45 smithi098 ceph-mon[35569]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:45.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:45 smithi098 ceph-mon[35569]: Deploying daemon mds.l on smithi112 2024-06-04T04:52:46.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:45 smithi070 ceph-mon[32631]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:46.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:45 smithi070 ceph-mon[32631]: Deploying daemon mds.l on smithi112 2024-06-04T04:52:46.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.680+0000 7f7ed102c700 1 -- 172.21.15.70:0/4050569644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ecc06c8c0 msgr2=0x7f7ecc108180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:46.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.680+0000 7f7ed102c700 1 --2- 172.21.15.70:0/4050569644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ecc06c8c0 0x7f7ecc108180 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7ec00099c0 tx=0x7f7ec0009cd0 comp rx=0 tx=0).stop 2024-06-04T04:52:46.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.680+0000 7f7ed102c700 1 -- 172.21.15.70:0/4050569644 shutdown_connections 2024-06-04T04:52:46.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.680+0000 7f7ed102c700 1 --2- 172.21.15.70:0/4050569644 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ecc108860 0x7f7ecc110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:46.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.680+0000 7f7ed102c700 1 --2- 172.21.15.70:0/4050569644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ecc06bf10 0x7f7ecc06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:46.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.680+0000 7f7ed102c700 1 --2- 172.21.15.70:0/4050569644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ecc06c8c0 0x7f7ecc108180 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:46.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.680+0000 7f7ed102c700 1 -- 172.21.15.70:0/4050569644 >> 172.21.15.70:0/4050569644 conn(0x7f7ecc078490 msgr2=0x7f7ecc0788a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:52:46.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.681+0000 7f7ed102c700 1 -- 172.21.15.70:0/4050569644 shutdown_connections 2024-06-04T04:52:46.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.681+0000 7f7ed102c700 1 -- 172.21.15.70:0/4050569644 wait complete. 2024-06-04T04:52:46.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.681+0000 7f7ed102c700 1 Processor -- start 2024-06-04T04:52:46.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7ed102c700 1 -- start start 2024-06-04T04:52:46.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7ed102c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ecc06bf10 0x7f7ecc19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:46.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7ed102c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ecc06c8c0 0x7f7ecc19fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:46.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7ed102c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ecc108860 0x7f7ecc1999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:46.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7ed102c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc113600 con 0x7f7ecc06bf10 2024-06-04T04:52:46.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7ed102c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc113480 con 0x7f7ecc06c8c0 2024-06-04T04:52:46.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7ed102c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc113780 con 0x7f7ecc108860 2024-06-04T04:52:46.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7ecad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ecc06bf10 0x7f7ecc19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:46.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7ecad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ecc06bf10 0x7f7ecc19f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53652/0 (socket says 172.21.15.70:53652) 2024-06-04T04:52:46.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7ecad9d700 1 -- 172.21.15.70:0/103948554 learned_addr learned my addr 172.21.15.70:0/103948554 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:52:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7eca59c700 1 --2- 172.21.15.70:0/103948554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ecc06c8c0 0x7f7ecc19fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7ecb59e700 1 --2- 172.21.15.70:0/103948554 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ecc108860 0x7f7ecc1999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7ecad9d700 1 -- 172.21.15.70:0/103948554 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ecc108860 msgr2=0x7f7ecc1999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7ecad9d700 1 --2- 172.21.15.70:0/103948554 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ecc108860 0x7f7ecc1999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7ecad9d700 1 -- 172.21.15.70:0/103948554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ecc06c8c0 msgr2=0x7f7ecc19fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7ecad9d700 1 --2- 172.21.15.70:0/103948554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ecc06c8c0 0x7f7ecc19fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7ecad9d700 1 -- 172.21.15.70:0/103948554 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ecc19a120 con 0x7f7ecc06bf10 2024-06-04T04:52:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7ecad9d700 1 --2- 172.21.15.70:0/103948554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ecc06bf10 0x7f7ecc19f7b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc00c920 tx=0x7f7ebc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:52:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.682+0000 7f7eb3fff700 1 -- 172.21.15.70:0/103948554 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc004a30 con 0x7f7ecc06bf10 2024-06-04T04:52:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.683+0000 7f7ed102c700 1 -- 172.21.15.70:0/103948554 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ecc19a410 con 0x7f7ecc06bf10 2024-06-04T04:52:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.683+0000 7f7eb3fff700 1 -- 172.21.15.70:0/103948554 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7ebc0054b0 con 0x7f7ecc06bf10 2024-06-04T04:52:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.683+0000 7f7eb3fff700 1 -- 172.21.15.70:0/103948554 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc007b70 con 0x7f7ecc06bf10 2024-06-04T04:52:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.683+0000 7f7ed102c700 1 -- 172.21.15.70:0/103948554 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ecc1a6680 con 0x7f7ecc06bf10 2024-06-04T04:52:46.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.684+0000 7f7ed102c700 1 -- 172.21.15.70:0/103948554 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ecc06a230 con 0x7f7ecc06bf10 2024-06-04T04:52:46.688 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 04:52:46 smithi112 systemd[1]: Starting Ceph mds.l for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T04:52:46.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.686+0000 7f7eb3fff700 1 -- 172.21.15.70:0/103948554 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f7ebc007d10 con 0x7f7ecc06bf10 2024-06-04T04:52:46.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.686+0000 7f7eb3fff700 1 --2- 172.21.15.70:0/103948554 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f7eb40b2880 0x7f7eb40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:46.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.687+0000 7f7eca59c700 1 --2- 172.21.15.70:0/103948554 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f7eb40b2880 0x7f7eb40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:46.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.687+0000 7f7eb3fff700 1 -- 172.21.15.70:0/103948554 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f7ebc0e1c70 con 0x7f7ecc06bf10 2024-06-04T04:52:46.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.687+0000 7f7eca59c700 1 --2- 172.21.15.70:0/103948554 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f7eb40b2880 0x7f7eb40b4d40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0000c00 tx=0x7f7ec0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:52:46.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.690+0000 7f7eb3fff700 1 -- 172.21.15.70:0/103948554 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7ebc014030 con 0x7f7ecc06bf10 2024-06-04T04:52:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.937+0000 7f7ed102c700 1 -- 172.21.15.70:0/103948554 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ecc19aea0 con 0x7f7ecc06bf10 2024-06-04T04:52:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.938+0000 7f7eb3fff700 1 -- 172.21.15.70:0/103948554 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f7ebc005660 con 0x7f7ecc06bf10 2024-06-04T04:52:46.940 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:52:46.940 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-06-04T04:52:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.940+0000 7f7ed102c700 1 -- 172.21.15.70:0/103948554 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f7eb40b2880 msgr2=0x7f7eb40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.940+0000 7f7ed102c700 1 --2- 172.21.15.70:0/103948554 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f7eb40b2880 0x7f7eb40b4d40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0000c00 tx=0x7f7ec0005fb0 comp rx=0 tx=0).stop 2024-06-04T04:52:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.940+0000 7f7ed102c700 1 -- 172.21.15.70:0/103948554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ecc06bf10 msgr2=0x7f7ecc19f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.940+0000 7f7ed102c700 1 --2- 172.21.15.70:0/103948554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ecc06bf10 0x7f7ecc19f7b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc00c920 tx=0x7f7ebc00cc30 comp rx=0 tx=0).stop 2024-06-04T04:52:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.940+0000 7f7ed102c700 1 -- 172.21.15.70:0/103948554 shutdown_connections 2024-06-04T04:52:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.940+0000 7f7ed102c700 1 --2- 172.21.15.70:0/103948554 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f7eb40b2880 0x7f7eb40b4d40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.940+0000 7f7ed102c700 1 --2- 172.21.15.70:0/103948554 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ecc108860 0x7f7ecc1999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.940+0000 7f7ed102c700 1 --2- 172.21.15.70:0/103948554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ecc06bf10 0x7f7ecc19f7b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.940+0000 7f7ed102c700 1 --2- 172.21.15.70:0/103948554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ecc06c8c0 0x7f7ecc19fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.940+0000 7f7ed102c700 1 -- 172.21.15.70:0/103948554 >> 172.21.15.70:0/103948554 conn(0x7f7ecc078490 msgr2=0x7f7ecc107540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:52:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.940+0000 7f7ed102c700 1 -- 172.21.15.70:0/103948554 shutdown_connections 2024-06-04T04:52:46.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:46.940+0000 7f7ed102c700 1 -- 172.21.15.70:0/103948554 wait complete. 2024-06-04T04:52:46.944 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 1 2024-06-04T04:52:47.049 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 04:52:46 smithi112 podman[66869]: 2024-06-04 04:52:46.680189046 +0000 UTC m=+0.017099206 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T04:52:47.049 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 04:52:46 smithi112 podman[66869]: 2024-06-04 04:52:46.803908959 +0000 UTC m=+0.140819142 container create 4730b993ef36ac73daa6386462d16f572151a5963ddd90ab6c598e4522be1ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-06-04T04:52:47.049 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 04:52:47 smithi112 podman[66869]: 2024-06-04 04:52:47.04855824 +0000 UTC m=+0.385468395 container init 4730b993ef36ac73daa6386462d16f572151a5963ddd90ab6c598e4522be1ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-06-04T04:52:47.436 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 04:52:47 smithi112 podman[66869]: 2024-06-04 04:52:47.051467661 +0000 UTC m=+0.388377816 container start 4730b993ef36ac73daa6386462d16f572151a5963ddd90ab6c598e4522be1ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-l, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , RELEASE=main-b7328fe, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-06-04T04:52:47.436 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 04:52:47 smithi112 ceph-mds[66889]: -- 172.21.15.112:0/2936563564 <== mon.1 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f9bd3b1a40 con 0x55f9bd385000 2024-06-04T04:52:47.436 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 04:52:47 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-l[66885]: starting mds.l at 2024-06-04T04:52:47.436 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 04:52:47 smithi112 bash[66869]: 4730b993ef36ac73daa6386462d16f572151a5963ddd90ab6c598e4522be1ca1 2024-06-04T04:52:47.436 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 04:52:47 smithi112 systemd[1]: Started Ceph mds.l for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:52:47.887 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:47 smithi098 ceph-mon[35569]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:47.887 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:47 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/103948554' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:52:47.887 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:47 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.112:6832/1359739019,v1:172.21.15.112:6833/1359739019] up:boot 2024-06-04T04:52:47.887 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:47 smithi098 ceph-mon[35569]: fsmap 1 up:standby 2024-06-04T04:52:47.887 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-04T04:52:47.887 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:47.887 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:47.887 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:47.887 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:52:47.887 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:52:47.887 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:47 smithi112 ceph-mon[36453]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:47 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/103948554' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:52:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:47 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.112:6832/1359739019,v1:172.21.15.112:6833/1359739019] up:boot 2024-06-04T04:52:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:47 smithi112 ceph-mon[36453]: fsmap 1 up:standby 2024-06-04T04:52:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-04T04:52:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:52:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:52:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:48.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:47 smithi070 ceph-mon[32631]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:48.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:47 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/103948554' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:52:48.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:47 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.112:6832/1359739019,v1:172.21.15.112:6833/1359739019] up:boot 2024-06-04T04:52:48.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:47 smithi070 ceph-mon[32631]: fsmap 1 up:standby 2024-06-04T04:52:48.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-04T04:52:48.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:48.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:48.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:48.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:52:48.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:52:48.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:48.902 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:48 smithi098 ceph-mon[35569]: Deploying daemon mds.k on smithi098 2024-06-04T04:52:48.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:48 smithi112 ceph-mon[36453]: Deploying daemon mds.k on smithi098 2024-06-04T04:52:49.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:48 smithi070 ceph-mon[32631]: Deploying daemon mds.k on smithi098 2024-06-04T04:52:49.749 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:52:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:49 smithi112 ceph-mon[36453]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:49 smithi098 ceph-mon[35569]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:49.995 INFO:journalctl@ceph.mds.k.smithi098.stdout:Jun 04 04:52:49 smithi098 systemd[1]: Started Ceph mds.k for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:52:49.999 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:52:50.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:49 smithi070 ceph-mon[32631]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:51.160 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:50 smithi070 ceph-mon[32631]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:51.160 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:50 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:51.160 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:50 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:51.160 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:50 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:51.160 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:50 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:52:51.160 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:50 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:52:51.160 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:50 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:51.160 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:50 smithi070 ceph-mon[32631]: Deploying daemon mds.j on smithi070 2024-06-04T04:52:51.160 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:50 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.98:6832/1896647585,v1:172.21.15.98:6833/1896647585] up:boot 2024-06-04T04:52:51.160 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:50 smithi070 ceph-mon[32631]: fsmap 2 up:standby 2024-06-04T04:52:51.161 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:50 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-04T04:52:51.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:50 smithi112 ceph-mon[36453]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:51.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:50 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:51.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:50 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:51.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:50 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:51.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:50 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:52:51.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:50 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:52:51.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:50 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:51.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:50 smithi112 ceph-mon[36453]: Deploying daemon mds.j on smithi070 2024-06-04T04:52:51.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:50 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.98:6832/1896647585,v1:172.21.15.98:6833/1896647585] up:boot 2024-06-04T04:52:51.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:50 smithi112 ceph-mon[36453]: fsmap 2 up:standby 2024-06-04T04:52:51.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:50 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-04T04:52:51.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:50 smithi098 ceph-mon[35569]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:51.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:50 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:51.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:50 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:51.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:50 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:51.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:50 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:52:51.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:50 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:52:51.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:50 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:51.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:50 smithi098 ceph-mon[35569]: Deploying daemon mds.j on smithi070 2024-06-04T04:52:51.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:50 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.98:6832/1896647585,v1:172.21.15.98:6833/1896647585] up:boot 2024-06-04T04:52:51.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:50 smithi098 ceph-mon[35569]: fsmap 2 up:standby 2024-06-04T04:52:51.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:50 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-04T04:52:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.445+0000 7f3013eb6700 1 -- 172.21.15.70:0/201715976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f300c06c470 msgr2=0x7f300c1103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.445+0000 7f3013eb6700 1 --2- 172.21.15.70:0/201715976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f300c06c470 0x7f300c1103c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f300400b0a0 tx=0x7f300400b3b0 comp rx=0 tx=0).stop 2024-06-04T04:52:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.445+0000 7f3013eb6700 1 -- 172.21.15.70:0/201715976 shutdown_connections 2024-06-04T04:52:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.445+0000 7f3013eb6700 1 --2- 172.21.15.70:0/201715976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f300c06c470 0x7f300c1103c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.445+0000 7f3013eb6700 1 --2- 172.21.15.70:0/201715976 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f300c1095a0 0x7f300c109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.445+0000 7f3013eb6700 1 --2- 172.21.15.70:0/201715976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f300c06bab0 0x7f300c06bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.445+0000 7f3013eb6700 1 -- 172.21.15.70:0/201715976 >> 172.21.15.70:0/201715976 conn(0x7f300c0793b0 msgr2=0x7f300c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:52:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.445+0000 7f3013eb6700 1 -- 172.21.15.70:0/201715976 shutdown_connections 2024-06-04T04:52:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.445+0000 7f3013eb6700 1 -- 172.21.15.70:0/201715976 wait complete. 2024-06-04T04:52:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.445+0000 7f3013eb6700 1 Processor -- start 2024-06-04T04:52:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.445+0000 7f3013eb6700 1 -- start start 2024-06-04T04:52:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3013eb6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f300c06bab0 0x7f300c102500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3013eb6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f300c06c470 0x7f300c102a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3013eb6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f300c1095a0 0x7f300c102f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3013eb6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f300c113580 con 0x7f300c06c470 2024-06-04T04:52:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3013eb6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f300c113400 con 0x7f300c06bab0 2024-06-04T04:52:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3013eb6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f300c113700 con 0x7f300c1095a0 2024-06-04T04:52:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3011451700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f300c06c470 0x7f300c102a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3011451700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f300c06c470 0x7f300c102a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53666/0 (socket says 172.21.15.70:53666) 2024-06-04T04:52:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3011451700 1 -- 172.21.15.70:0/1033610448 learned_addr learned my addr 172.21.15.70:0/1033610448 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:52:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3012453700 1 --2- 172.21.15.70:0/1033610448 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f300c1095a0 0x7f300c102f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3011c52700 1 --2- 172.21.15.70:0/1033610448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f300c06bab0 0x7f300c102500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3011451700 1 -- 172.21.15.70:0/1033610448 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f300c1095a0 msgr2=0x7f300c102f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:52.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3011451700 1 --2- 172.21.15.70:0/1033610448 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f300c1095a0 0x7f300c102f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:52.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3011451700 1 -- 172.21.15.70:0/1033610448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f300c06bab0 msgr2=0x7f300c102500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:52.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3011451700 1 --2- 172.21.15.70:0/1033610448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f300c06bab0 0x7f300c102500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:52.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3011451700 1 -- 172.21.15.70:0/1033610448 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f300c1a8000 con 0x7f300c06c470 2024-06-04T04:52:52.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3011451700 1 --2- 172.21.15.70:0/1033610448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f300c06c470 0x7f300c102a40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00c920 tx=0x7f2ffc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:52:52.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3002ffd700 1 -- 172.21.15.70:0/1033610448 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc004a30 con 0x7f300c06c470 2024-06-04T04:52:52.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3013eb6700 1 -- 172.21.15.70:0/1033610448 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f300c1a8290 con 0x7f300c06c470 2024-06-04T04:52:52.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.446+0000 7f3002ffd700 1 -- 172.21.15.70:0/1033610448 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2ffc0054b0 con 0x7f300c06c470 2024-06-04T04:52:52.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.447+0000 7f3013eb6700 1 -- 172.21.15.70:0/1033610448 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f300c1a8820 con 0x7f300c06c470 2024-06-04T04:52:52.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.447+0000 7f3002ffd700 1 -- 172.21.15.70:0/1033610448 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc007b70 con 0x7f300c06c470 2024-06-04T04:52:52.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.447+0000 7f3013eb6700 1 -- 172.21.15.70:0/1033610448 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f300c10ba80 con 0x7f300c06c470 2024-06-04T04:52:52.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.449+0000 7f3002ffd700 1 -- 172.21.15.70:0/1033610448 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f2ffc007d10 con 0x7f300c06c470 2024-06-04T04:52:52.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.449+0000 7f3002ffd700 1 --2- 172.21.15.70:0/1033610448 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2ff80b2880 0x7f2ff80b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:52.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.450+0000 7f3011c52700 1 --2- 172.21.15.70:0/1033610448 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2ff80b2880 0x7f2ff80b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:52.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.450+0000 7f3002ffd700 1 -- 172.21.15.70:0/1033610448 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f2ffc0e1c90 con 0x7f300c06c470 2024-06-04T04:52:52.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.450+0000 7f3011c52700 1 --2- 172.21.15.70:0/1033610448 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2ff80b2880 0x7f2ff80b4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3008005950 tx=0x7f300800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:52:52.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.453+0000 7f3002ffd700 1 -- 172.21.15.70:0/1033610448 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2ffc014030 con 0x7f300c06c470 2024-06-04T04:52:52.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.708+0000 7f3013eb6700 1 -- 172.21.15.70:0/1033610448 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f300c06a230 con 0x7f300c06c470 2024-06-04T04:52:52.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.708+0000 7f3002ffd700 1 -- 172.21.15.70:0/1033610448 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f2ffc005660 con 0x7f300c06c470 2024-06-04T04:52:52.710 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:52:52.711 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-06-04T04:52:52.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.710+0000 7f3013eb6700 1 -- 172.21.15.70:0/1033610448 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2ff80b2880 msgr2=0x7f2ff80b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:52.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.710+0000 7f3013eb6700 1 --2- 172.21.15.70:0/1033610448 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2ff80b2880 0x7f2ff80b4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3008005950 tx=0x7f300800a300 comp rx=0 tx=0).stop 2024-06-04T04:52:52.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.710+0000 7f3013eb6700 1 -- 172.21.15.70:0/1033610448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f300c06c470 msgr2=0x7f300c102a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:52.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.710+0000 7f3013eb6700 1 --2- 172.21.15.70:0/1033610448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f300c06c470 0x7f300c102a40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00c920 tx=0x7f2ffc00cc30 comp rx=0 tx=0).stop 2024-06-04T04:52:52.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.710+0000 7f3013eb6700 1 -- 172.21.15.70:0/1033610448 shutdown_connections 2024-06-04T04:52:52.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.710+0000 7f3013eb6700 1 --2- 172.21.15.70:0/1033610448 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2ff80b2880 0x7f2ff80b4d40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:52.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.710+0000 7f3013eb6700 1 --2- 172.21.15.70:0/1033610448 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f300c1095a0 0x7f300c102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:52.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.710+0000 7f3013eb6700 1 --2- 172.21.15.70:0/1033610448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f300c06bab0 0x7f300c102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:52.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.710+0000 7f3013eb6700 1 --2- 172.21.15.70:0/1033610448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f300c06c470 0x7f300c102a40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:52.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.710+0000 7f3013eb6700 1 -- 172.21.15.70:0/1033610448 >> 172.21.15.70:0/1033610448 conn(0x7f300c0793b0 msgr2=0x7f300c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:52:52.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.710+0000 7f3013eb6700 1 -- 172.21.15.70:0/1033610448 shutdown_connections 2024-06-04T04:52:52.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:52.710+0000 7f3013eb6700 1 -- 172.21.15.70:0/1033610448 wait complete. 2024-06-04T04:52:52.713 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 3 2024-06-04T04:52:52.976 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:52 smithi070 ceph-mon[32631]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:52.976 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:52 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1033610448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:52:53.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:52 smithi112 ceph-mon[36453]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:53.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:52 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1033610448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:52:53.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:52 smithi098 ceph-mon[35569]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:53.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:52 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1033610448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:52:53.540 INFO:journalctl@ceph.mds.j.smithi070.stdout:Jun 04 04:52:53 smithi070 systemd[1]: Starting Ceph mds.j for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T04:52:54.323 INFO:journalctl@ceph.mds.j.smithi070.stdout:Jun 04 04:52:54 smithi070 podman[72375]: 2024-06-04 04:52:54.090824777 +0000 UTC m=+0.017222493 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T04:52:54.323 INFO:journalctl@ceph.mds.j.smithi070.stdout:Jun 04 04:52:54 smithi070 podman[72375]: 2024-06-04 04:52:54.191124193 +0000 UTC m=+0.117521947 container create 92fa0d8e8b6b560c03c378d2769e8e2bc3b98ffb39fed5eb4fd8f55d15f0d493 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-j, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-b7328fe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-06-04T04:52:54.859 INFO:journalctl@ceph.mds.j.smithi070.stdout:Jun 04 04:52:54 smithi070 podman[72375]: 2024-06-04 04:52:54.469686017 +0000 UTC m=+0.396083723 container init 92fa0d8e8b6b560c03c378d2769e8e2bc3b98ffb39fed5eb4fd8f55d15f0d493 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521) 2024-06-04T04:52:54.859 INFO:journalctl@ceph.mds.j.smithi070.stdout:Jun 04 04:52:54 smithi070 podman[72375]: 2024-06-04 04:52:54.472548634 +0000 UTC m=+0.398946358 container start 92fa0d8e8b6b560c03c378d2769e8e2bc3b98ffb39fed5eb4fd8f55d15f0d493 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-j, RELEASE=main-b7328fe, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-06-04T04:52:54.859 INFO:journalctl@ceph.mds.j.smithi070.stdout:Jun 04 04:52:54 smithi070 ceph-mds[72394]: -- 172.21.15.70:0/3899678165 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5651c77c1a40 con 0x5651c7794800 2024-06-04T04:52:54.859 INFO:journalctl@ceph.mds.j.smithi070.stdout:Jun 04 04:52:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-j[72390]: starting mds.j at 2024-06-04T04:52:54.859 INFO:journalctl@ceph.mds.j.smithi070.stdout:Jun 04 04:52:54 smithi070 bash[72375]: 92fa0d8e8b6b560c03c378d2769e8e2bc3b98ffb39fed5eb4fd8f55d15f0d493 2024-06-04T04:52:54.859 INFO:journalctl@ceph.mds.j.smithi070.stdout:Jun 04 04:52:54 smithi070 systemd[1]: Started Ceph mds.j for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:52:55.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:55 smithi070 ceph-mon[32631]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:55.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:55 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:55.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:55 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:55.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:55 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:55.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:55 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:52:55.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:55 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:52:55.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:55 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:55.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:55 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.70:6834/2694011970,v1:172.21.15.70:6835/2694011970] up:boot 2024-06-04T04:52:55.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:55 smithi070 ceph-mon[32631]: fsmap 3 up:standby 2024-06-04T04:52:55.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:55 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-04T04:52:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:55 smithi112 ceph-mon[36453]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:55 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:55 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:55 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:55.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:55 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:52:55.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:55 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:52:55.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:55 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:55.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:55 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.70:6834/2694011970,v1:172.21.15.70:6835/2694011970] up:boot 2024-06-04T04:52:55.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:55 smithi112 ceph-mon[36453]: fsmap 3 up:standby 2024-06-04T04:52:55.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:55 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-04T04:52:55.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:52:55.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:55 smithi098 ceph-mon[35569]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:55.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:55 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:55.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:55 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:55.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:55 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:55.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:55 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:52:55.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:55 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:52:55.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:55 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:55.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:55 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.70:6834/2694011970,v1:172.21.15.70:6835/2694011970] up:boot 2024-06-04T04:52:55.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:55 smithi098 ceph-mon[35569]: fsmap 3 up:standby 2024-06-04T04:52:55.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:55 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-04T04:52:56.448 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:52:56.658 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:56 smithi070 ceph-mon[32631]: Deploying daemon mds.d on smithi070 2024-06-04T04:52:56.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:56 smithi112 ceph-mon[36453]: Deploying daemon mds.d on smithi070 2024-06-04T04:52:56.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:56 smithi098 ceph-mon[35569]: Deploying daemon mds.d on smithi070 2024-06-04T04:52:57.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:57 smithi112 ceph-mon[36453]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:57.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:57 smithi098 ceph-mon[35569]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:58.010 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:57 smithi070 ceph-mon[32631]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:58.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.262+0000 7f93ecfe0700 1 -- 172.21.15.70:0/1329834374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93e8102160 msgr2=0x7f93e8102540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.262+0000 7f93ecfe0700 1 --2- 172.21.15.70:0/1329834374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93e8102160 0x7f93e8102540 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f93d8009990 tx=0x7f93d8009ca0 comp rx=0 tx=0).stop 2024-06-04T04:52:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.263+0000 7f93ecfe0700 1 -- 172.21.15.70:0/1329834374 shutdown_connections 2024-06-04T04:52:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.263+0000 7f93ecfe0700 1 --2- 172.21.15.70:0/1329834374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93e8107000 0x7f93e810eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.263+0000 7f93ecfe0700 1 --2- 172.21.15.70:0/1329834374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93e8102160 0x7f93e8102540 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.263+0000 7f93ecfe0700 1 --2- 172.21.15.70:0/1329834374 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93e8102b10 0x7f93e81068d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.263+0000 7f93ecfe0700 1 -- 172.21.15.70:0/1329834374 >> 172.21.15.70:0/1329834374 conn(0x7f93e8077780 msgr2=0x7f93e8077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:52:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.263+0000 7f93ecfe0700 1 -- 172.21.15.70:0/1329834374 shutdown_connections 2024-06-04T04:52:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.263+0000 7f93ecfe0700 1 -- 172.21.15.70:0/1329834374 wait complete. 2024-06-04T04:52:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.263+0000 7f93ecfe0700 1 Processor -- start 2024-06-04T04:52:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.263+0000 7f93ecfe0700 1 -- start start 2024-06-04T04:52:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93ecfe0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93e8102160 0x7f93e819bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93ecfe0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93e8102b10 0x7f93e819c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93ecfe0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93e8107000 0x7f93e81a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93ecfe0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f93e8111460 con 0x7f93e8107000 2024-06-04T04:52:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93ecfe0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f93e81112e0 con 0x7f93e8102b10 2024-06-04T04:52:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93ecfe0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f93e81115e0 con 0x7f93e8102160 2024-06-04T04:52:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93e6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93e8107000 0x7f93e81a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93e6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93e8107000 0x7f93e81a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57820/0 (socket says 172.21.15.70:57820) 2024-06-04T04:52:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93e6d9d700 1 -- 172.21.15.70:0/115590045 learned_addr learned my addr 172.21.15.70:0/115590045 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:52:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93e659c700 1 --2- 172.21.15.70:0/115590045 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93e8102160 0x7f93e819bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93e5d9b700 1 --2- 172.21.15.70:0/115590045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93e8102b10 0x7f93e819c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93e6d9d700 1 -- 172.21.15.70:0/115590045 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93e8102160 msgr2=0x7f93e819bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93e6d9d700 1 --2- 172.21.15.70:0/115590045 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93e8102160 0x7f93e819bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:58.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93e6d9d700 1 -- 172.21.15.70:0/115590045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93e8102b10 msgr2=0x7f93e819c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:58.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93e6d9d700 1 --2- 172.21.15.70:0/115590045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93e8102b10 0x7f93e819c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:58.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93e6d9d700 1 -- 172.21.15.70:0/115590045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e81a0da0 con 0x7f93e8107000 2024-06-04T04:52:58.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93e6d9d700 1 --2- 172.21.15.70:0/115590045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93e8107000 0x7f93e81a0660 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f93e000d370 tx=0x7f93e000d680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:52:58.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93d77fe700 1 -- 172.21.15.70:0/115590045 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e000ddf0 con 0x7f93e8107000 2024-06-04T04:52:58.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93ecfe0700 1 -- 172.21.15.70:0/115590045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93e81a1030 con 0x7f93e8107000 2024-06-04T04:52:58.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93d77fe700 1 -- 172.21.15.70:0/115590045 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f93e0003a00 con 0x7f93e8107000 2024-06-04T04:52:58.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.264+0000 7f93d77fe700 1 -- 172.21.15.70:0/115590045 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e0008ce0 con 0x7f93e8107000 2024-06-04T04:52:58.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.265+0000 7f93ecfe0700 1 -- 172.21.15.70:0/115590045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93e81a8910 con 0x7f93e8107000 2024-06-04T04:52:58.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.265+0000 7f93ecfe0700 1 -- 172.21.15.70:0/115590045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93e806a230 con 0x7f93e8107000 2024-06-04T04:52:58.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.267+0000 7f93d77fe700 1 -- 172.21.15.70:0/115590045 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f93e001a020 con 0x7f93e8107000 2024-06-04T04:52:58.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.267+0000 7f93d77fe700 1 --2- 172.21.15.70:0/115590045 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f93d00b2880 0x7f93d00b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:52:58.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.268+0000 7f93e659c700 1 --2- 172.21.15.70:0/115590045 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f93d00b2880 0x7f93d00b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:52:58.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.268+0000 7f93d77fe700 1 -- 172.21.15.70:0/115590045 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f93e00e1f60 con 0x7f93e8107000 2024-06-04T04:52:58.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.268+0000 7f93e659c700 1 --2- 172.21.15.70:0/115590045 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f93d00b2880 0x7f93d00b4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f93d80098e0 tx=0x7f93d800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:52:58.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.271+0000 7f93d77fe700 1 -- 172.21.15.70:0/115590045 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f93e00aa120 con 0x7f93e8107000 2024-06-04T04:52:58.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.523+0000 7f93ecfe0700 1 -- 172.21.15.70:0/115590045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93e81a8ae0 con 0x7f93e8107000 2024-06-04T04:52:58.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.524+0000 7f93d77fe700 1 -- 172.21.15.70:0/115590045 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7f93e00a9a20 con 0x7f93e8107000 2024-06-04T04:52:58.526 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:52:58.527 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-06-04T04:52:58.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.526+0000 7f93ecfe0700 1 -- 172.21.15.70:0/115590045 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f93d00b2880 msgr2=0x7f93d00b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:58.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.526+0000 7f93ecfe0700 1 --2- 172.21.15.70:0/115590045 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f93d00b2880 0x7f93d00b4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f93d80098e0 tx=0x7f93d800b540 comp rx=0 tx=0).stop 2024-06-04T04:52:58.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.526+0000 7f93ecfe0700 1 -- 172.21.15.70:0/115590045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93e8107000 msgr2=0x7f93e81a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:52:58.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.526+0000 7f93ecfe0700 1 --2- 172.21.15.70:0/115590045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93e8107000 0x7f93e81a0660 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f93e000d370 tx=0x7f93e000d680 comp rx=0 tx=0).stop 2024-06-04T04:52:58.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.526+0000 7f93ecfe0700 1 -- 172.21.15.70:0/115590045 shutdown_connections 2024-06-04T04:52:58.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.526+0000 7f93ecfe0700 1 --2- 172.21.15.70:0/115590045 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f93d00b2880 0x7f93d00b4d40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:58.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.526+0000 7f93ecfe0700 1 --2- 172.21.15.70:0/115590045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93e8107000 0x7f93e81a0660 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:58.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.526+0000 7f93ecfe0700 1 --2- 172.21.15.70:0/115590045 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93e8102160 0x7f93e819bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:58.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.526+0000 7f93ecfe0700 1 --2- 172.21.15.70:0/115590045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93e8102b10 0x7f93e819c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:52:58.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.526+0000 7f93ecfe0700 1 -- 172.21.15.70:0/115590045 >> 172.21.15.70:0/115590045 conn(0x7f93e8077780 msgr2=0x7f93e8100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:52:58.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.526+0000 7f93ecfe0700 1 -- 172.21.15.70:0/115590045 shutdown_connections 2024-06-04T04:52:58.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:52:58.527+0000 7f93ecfe0700 1 -- 172.21.15.70:0/115590045 wait complete. 2024-06-04T04:52:58.530 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 4 2024-06-04T04:52:58.787 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:58 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/115590045' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:52:58.788 INFO:journalctl@ceph.mds.d.smithi070.stdout:Jun 04 04:52:58 smithi070 systemd[1]: Started Ceph mds.d for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:52:58.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:58 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/115590045' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:52:58.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:58 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/115590045' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:52:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:59 smithi098 ceph-mon[35569]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:59.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:59 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:59.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:59 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:59.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:59 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:59.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:59 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:52:59.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:59 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:52:59.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:52:59 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:52:59.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:59 smithi112 ceph-mon[36453]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:52:59.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:59 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:59.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:59 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:59.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:59 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:52:59.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:59 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:52:59.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:59 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:52:59.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:52:59 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:00.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:59 smithi070 ceph-mon[32631]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:00.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:59 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:00.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:59 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:00.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:59 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:00.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:59 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:00.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:59 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:00.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:52:59 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:00.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:00 smithi098 ceph-mon[35569]: Deploying daemon mds.e on smithi098 2024-06-04T04:53:00.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:00 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.70:6836/729112700,v1:172.21.15.70:6837/729112700] up:boot 2024-06-04T04:53:00.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:00 smithi098 ceph-mon[35569]: fsmap 4 up:standby 2024-06-04T04:53:00.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:00 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-04T04:53:00.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:00 smithi098 ceph-mon[35569]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:00.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:00 smithi112 ceph-mon[36453]: Deploying daemon mds.e on smithi098 2024-06-04T04:53:00.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:00 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.70:6836/729112700,v1:172.21.15.70:6837/729112700] up:boot 2024-06-04T04:53:00.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:00 smithi112 ceph-mon[36453]: fsmap 4 up:standby 2024-06-04T04:53:00.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:00 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-04T04:53:00.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:00 smithi112 ceph-mon[36453]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:01.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:00 smithi070 ceph-mon[32631]: Deploying daemon mds.e on smithi098 2024-06-04T04:53:01.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:00 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.70:6836/729112700,v1:172.21.15.70:6837/729112700] up:boot 2024-06-04T04:53:01.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:00 smithi070 ceph-mon[32631]: fsmap 4 up:standby 2024-06-04T04:53:01.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:00 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-04T04:53:01.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:00 smithi070 ceph-mon[32631]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:01.231 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 04:53:01 smithi098 systemd[1]: Starting Ceph mds.e for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T04:53:01.409 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:53:01.490 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 04:53:01 smithi098 podman[64845]: 2024-06-04 04:53:01.131758901 +0000 UTC m=+0.016919264 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T04:53:01.490 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 04:53:01 smithi098 podman[64845]: 2024-06-04 04:53:01.261277146 +0000 UTC m=+0.146437485 container create 14db21803650c377a6f8d81dd01b9c85f758bcecc06c994077d317e01f497cec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-b7328fe, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-06-04T04:53:01.490 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 04:53:01 smithi098 podman[64845]: 2024-06-04 04:53:01.489075632 +0000 UTC m=+0.374235987 container init 14db21803650c377a6f8d81dd01b9c85f758bcecc06c994077d317e01f497cec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-b7328fe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-04T04:53:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:53:01.745 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 04:53:01 smithi098 podman[64845]: 2024-06-04 04:53:01.491765757 +0000 UTC m=+0.376926104 container start 14db21803650c377a6f8d81dd01b9c85f758bcecc06c994077d317e01f497cec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-e, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-b7328fe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-06-04T04:53:01.745 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 04:53:01 smithi098 ceph-mds[64864]: -- 172.21.15.98:0/2793388565 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e81fe95a40 con 0x55e81fe69000 2024-06-04T04:53:01.745 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 04:53:01 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-e[64860]: starting mds.e at 2024-06-04T04:53:01.746 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 04:53:01 smithi098 bash[64845]: 14db21803650c377a6f8d81dd01b9c85f758bcecc06c994077d317e01f497cec 2024-06-04T04:53:01.746 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 04:53:01 smithi098 systemd[1]: Started Ceph mds.e for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:53:02.798 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:02 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.98:6834/3148471566,v1:172.21.15.98:6835/3148471566] up:boot 2024-06-04T04:53:02.798 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:02 smithi070 ceph-mon[32631]: fsmap 5 up:standby 2024-06-04T04:53:02.798 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:02 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-04T04:53:02.798 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:02 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:02.798 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:02 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:02.799 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:02 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:02.799 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:02 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:02.799 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:02 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:02.799 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:02 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:02.799 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:02 smithi070 ceph-mon[32631]: Deploying daemon mds.f on smithi112 2024-06-04T04:53:02.799 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:02 smithi070 ceph-mon[32631]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:02.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:02 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.98:6834/3148471566,v1:172.21.15.98:6835/3148471566] up:boot 2024-06-04T04:53:02.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:02 smithi112 ceph-mon[36453]: fsmap 5 up:standby 2024-06-04T04:53:02.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:02 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-04T04:53:02.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:02 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:02.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:02 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:02.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:02 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:02.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:02 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:02.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:02 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:02.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:02 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:02.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:02 smithi112 ceph-mon[36453]: Deploying daemon mds.f on smithi112 2024-06-04T04:53:02.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:02 smithi112 ceph-mon[36453]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:02.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:02 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.98:6834/3148471566,v1:172.21.15.98:6835/3148471566] up:boot 2024-06-04T04:53:02.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:02 smithi098 ceph-mon[35569]: fsmap 5 up:standby 2024-06-04T04:53:02.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:02 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-04T04:53:02.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:02 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:02.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:02 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:02.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:02 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:02.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:02 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:02.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:02 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:02.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:02 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:02.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:02 smithi098 ceph-mon[35569]: Deploying daemon mds.f on smithi112 2024-06-04T04:53:02.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:02 smithi098 ceph-mon[35569]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:03.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.324+0000 7fb4f6161700 1 -- 172.21.15.70:0/2732535723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4f0102430 msgr2=0x7fb4f01028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:03.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.324+0000 7fb4f6161700 1 --2- 172.21.15.70:0/2732535723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4f0102430 0x7fb4f01028b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0009990 tx=0x7fb4e0009ca0 comp rx=0 tx=0).stop 2024-06-04T04:53:03.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.324+0000 7fb4f6161700 1 -- 172.21.15.70:0/2732535723 shutdown_connections 2024-06-04T04:53:03.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.324+0000 7fb4f6161700 1 --2- 172.21.15.70:0/2732535723 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4f0102df0 0x7fb4f010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:03.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.324+0000 7fb4f6161700 1 --2- 172.21.15.70:0/2732535723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4f0108510 0x7fb4f01088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:03.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.324+0000 7fb4f6161700 1 --2- 172.21.15.70:0/2732535723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4f0102430 0x7fb4f01028b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:03.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.324+0000 7fb4f6161700 1 -- 172.21.15.70:0/2732535723 >> 172.21.15.70:0/2732535723 conn(0x7fb4f00793b0 msgr2=0x7fb4f00797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:03.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.324+0000 7fb4f6161700 1 -- 172.21.15.70:0/2732535723 shutdown_connections 2024-06-04T04:53:03.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.324+0000 7fb4f6161700 1 -- 172.21.15.70:0/2732535723 wait complete. 2024-06-04T04:53:03.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.325+0000 7fb4f6161700 1 Processor -- start 2024-06-04T04:53:03.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.325+0000 7fb4f6161700 1 -- start start 2024-06-04T04:53:03.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.325+0000 7fb4f6161700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4f0102430 0x7fb4f019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:03.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.325+0000 7fb4f6161700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4f0102df0 0x7fb4f019fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:03.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.325+0000 7fb4f6161700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4f0108510 0x7fb4f0199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:03.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.325+0000 7fb4f6161700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f0112490 con 0x7fb4f0108510 2024-06-04T04:53:03.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.325+0000 7fb4f6161700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f0112310 con 0x7fb4f0102df0 2024-06-04T04:53:03.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.325+0000 7fb4f6161700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f0112610 con 0x7fb4f0102430 2024-06-04T04:53:03.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.325+0000 7fb4effff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4f0108510 0x7fb4f0199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:03.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.325+0000 7fb4ef7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4f0102430 0x7fb4f019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:03.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.326+0000 7fb4ef7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4f0102430 0x7fb4f019f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36148/0 (socket says 172.21.15.70:36148) 2024-06-04T04:53:03.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.326+0000 7fb4ef7fe700 1 -- 172.21.15.70:0/604013941 learned_addr learned my addr 172.21.15.70:0/604013941 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:53:03.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.326+0000 7fb4eeffd700 1 --2- 172.21.15.70:0/604013941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4f0102df0 0x7fb4f019fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:03.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.326+0000 7fb4effff700 1 -- 172.21.15.70:0/604013941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4f0102430 msgr2=0x7fb4f019f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:03.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.326+0000 7fb4effff700 1 --2- 172.21.15.70:0/604013941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4f0102430 0x7fb4f019f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:03.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.326+0000 7fb4effff700 1 -- 172.21.15.70:0/604013941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4f0102df0 msgr2=0x7fb4f019fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:03.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.326+0000 7fb4effff700 1 --2- 172.21.15.70:0/604013941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4f0102df0 0x7fb4f019fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:03.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.326+0000 7fb4effff700 1 -- 172.21.15.70:0/604013941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f019a090 con 0x7fb4f0108510 2024-06-04T04:53:03.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.326+0000 7fb4effff700 1 --2- 172.21.15.70:0/604013941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4f0108510 0x7fb4f0199930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8007f10 tx=0x7fb4e800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:03.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.326+0000 7fb4ecff9700 1 -- 172.21.15.70:0/604013941 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8004db0 con 0x7fb4f0108510 2024-06-04T04:53:03.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.326+0000 7fb4f6161700 1 -- 172.21.15.70:0/604013941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4f019a380 con 0x7fb4f0108510 2024-06-04T04:53:03.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.326+0000 7fb4ecff9700 1 -- 172.21.15.70:0/604013941 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb4e8009e20 con 0x7fb4f0108510 2024-06-04T04:53:03.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.326+0000 7fb4ecff9700 1 -- 172.21.15.70:0/604013941 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8015400 con 0x7fb4f0108510 2024-06-04T04:53:03.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.326+0000 7fb4f6161700 1 -- 172.21.15.70:0/604013941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4f01a6640 con 0x7fb4f0108510 2024-06-04T04:53:03.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.327+0000 7fb4f6161700 1 -- 172.21.15.70:0/604013941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4f006a230 con 0x7fb4f0108510 2024-06-04T04:53:03.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.328+0000 7fb4ecff9700 1 -- 172.21.15.70:0/604013941 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fb4e8005240 con 0x7fb4f0108510 2024-06-04T04:53:03.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.329+0000 7fb4ecff9700 1 --2- 172.21.15.70:0/604013941 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb4dc0b2880 0x7fb4dc0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:03.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.329+0000 7fb4ef7fe700 1 --2- 172.21.15.70:0/604013941 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb4dc0b2880 0x7fb4dc0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:03.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.329+0000 7fb4ecff9700 1 -- 172.21.15.70:0/604013941 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fb4e80e1f20 con 0x7fb4f0108510 2024-06-04T04:53:03.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.329+0000 7fb4ef7fe700 1 --2- 172.21.15.70:0/604013941 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb4dc0b2880 0x7fb4dc0b4d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb4f019aa70 tx=0x7fb4d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:03.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.332+0000 7fb4ecff9700 1 -- 172.21.15.70:0/604013941 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb4e8010070 con 0x7fb4f0108510 2024-06-04T04:53:03.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.579+0000 7fb4f6161700 1 -- 172.21.15.70:0/604013941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4f019ae90 con 0x7fb4f0108510 2024-06-04T04:53:03.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.580+0000 7fb4ecff9700 1 -- 172.21.15.70:0/604013941 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4693 (secure 0 0 0) 0x7fb4f019ae90 con 0x7fb4f0108510 2024-06-04T04:53:03.583 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:53:03.583 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24386,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-06-04T04:53:03.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.582+0000 7fb4f6161700 1 -- 172.21.15.70:0/604013941 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb4dc0b2880 msgr2=0x7fb4dc0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:03.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.582+0000 7fb4f6161700 1 --2- 172.21.15.70:0/604013941 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb4dc0b2880 0x7fb4dc0b4d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb4f019aa70 tx=0x7fb4d8009260 comp rx=0 tx=0).stop 2024-06-04T04:53:03.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.582+0000 7fb4f6161700 1 -- 172.21.15.70:0/604013941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4f0108510 msgr2=0x7fb4f0199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:03.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.582+0000 7fb4f6161700 1 --2- 172.21.15.70:0/604013941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4f0108510 0x7fb4f0199930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8007f10 tx=0x7fb4e800d3b0 comp rx=0 tx=0).stop 2024-06-04T04:53:03.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.582+0000 7fb4f6161700 1 -- 172.21.15.70:0/604013941 shutdown_connections 2024-06-04T04:53:03.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.582+0000 7fb4f6161700 1 --2- 172.21.15.70:0/604013941 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb4dc0b2880 0x7fb4dc0b4d40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:03.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.582+0000 7fb4f6161700 1 --2- 172.21.15.70:0/604013941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4f0108510 0x7fb4f0199930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:03.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.582+0000 7fb4f6161700 1 --2- 172.21.15.70:0/604013941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4f0102430 0x7fb4f019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:03.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.582+0000 7fb4f6161700 1 --2- 172.21.15.70:0/604013941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4f0102df0 0x7fb4f019fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:03.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.582+0000 7fb4f6161700 1 -- 172.21.15.70:0/604013941 >> 172.21.15.70:0/604013941 conn(0x7fb4f00793b0 msgr2=0x7fb4f006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:03.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.582+0000 7fb4f6161700 1 -- 172.21.15.70:0/604013941 shutdown_connections 2024-06-04T04:53:03.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:03.582+0000 7fb4f6161700 1 -- 172.21.15.70:0/604013941 wait complete. 2024-06-04T04:53:03.587 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 6 2024-06-04T04:53:03.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:03 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/604013941' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:53:03.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:03 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/604013941' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:53:03.975 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 04:53:03 smithi112 systemd[1]: Starting Ceph mds.f for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T04:53:03.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:03 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/604013941' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:53:04.257 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 04:53:03 smithi112 podman[67181]: 2024-06-04 04:53:03.860945988 +0000 UTC m=+0.017170487 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T04:53:04.257 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 04:53:03 smithi112 podman[67181]: 2024-06-04 04:53:03.994232712 +0000 UTC m=+0.150457185 container create 76244bb7a9e717c7950bc3562dbd85ac644cd274b55f676b0dd82add97cc6a34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-b7328fe, ceph=True) 2024-06-04T04:53:04.257 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 04:53:04 smithi112 podman[67181]: 2024-06-04 04:53:04.255945029 +0000 UTC m=+0.412169517 container init 76244bb7a9e717c7950bc3562dbd85ac644cd274b55f676b0dd82add97cc6a34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-f, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-06-04T04:53:04.686 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 04:53:04 smithi112 podman[67181]: 2024-06-04 04:53:04.258902105 +0000 UTC m=+0.415126588 container start 76244bb7a9e717c7950bc3562dbd85ac644cd274b55f676b0dd82add97cc6a34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-f, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, RELEASE=main-b7328fe, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-06-04T04:53:04.686 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 04:53:04 smithi112 ceph-mds[67201]: -- 172.21.15.112:0/3687559113 <== mon.1 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f0368bba40 con 0x55f03688e800 2024-06-04T04:53:04.686 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 04:53:04 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-f[67196]: starting mds.f at 2024-06-04T04:53:04.686 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 04:53:04 smithi112 bash[67181]: 76244bb7a9e717c7950bc3562dbd85ac644cd274b55f676b0dd82add97cc6a34 2024-06-04T04:53:04.686 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 04:53:04 smithi112 systemd[1]: Started Ceph mds.f for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:53:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:05 smithi112 ceph-mon[36453]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:05 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:05 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:05 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:05 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:05 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:05 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:05.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:05 smithi112 ceph-mon[36453]: Deploying daemon mds.c on smithi112 2024-06-04T04:53:05.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:05 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] up:boot 2024-06-04T04:53:05.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:05 smithi112 ceph-mon[36453]: fsmap 6 up:standby 2024-06-04T04:53:05.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:05 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-04T04:53:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:05 smithi098 ceph-mon[35569]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:05 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:05 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:05 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:05.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:05 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:05.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:05 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:05.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:05 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:05.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:05 smithi098 ceph-mon[35569]: Deploying daemon mds.c on smithi112 2024-06-04T04:53:05.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:05 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] up:boot 2024-06-04T04:53:05.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:05 smithi098 ceph-mon[35569]: fsmap 6 up:standby 2024-06-04T04:53:05.748 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:05 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-04T04:53:05.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:05 smithi070 ceph-mon[32631]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:05.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:05 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:05.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:05 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:05.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:05 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:05.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:05 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:05.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:05 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:05.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:05 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:05.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:05 smithi070 ceph-mon[32631]: Deploying daemon mds.c on smithi112 2024-06-04T04:53:05.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:05 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] up:boot 2024-06-04T04:53:05.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:05 smithi070 ceph-mon[32631]: fsmap 6 up:standby 2024-06-04T04:53:05.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:05 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-04T04:53:06.494 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:53:06.686 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 04:53:06 smithi112 systemd[1]: Starting Ceph mds.c for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T04:53:06.746 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:53:06.984 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 04:53:06 smithi112 podman[67486]: 2024-06-04 04:53:06.624483342 +0000 UTC m=+0.016872245 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T04:53:06.984 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 04:53:06 smithi112 podman[67486]: 2024-06-04 04:53:06.733895129 +0000 UTC m=+0.126284024 container create abd1e979f2cc5bf11f1cfb25d6be62126dcd964a5ec4880fd096317802a28291 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, RELEASE=main-b7328fe) 2024-06-04T04:53:06.985 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 04:53:06 smithi112 podman[67486]: 2024-06-04 04:53:06.983006832 +0000 UTC m=+0.375395717 container init abd1e979f2cc5bf11f1cfb25d6be62126dcd964a5ec4880fd096317802a28291 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-06-04T04:53:07.417 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 04:53:06 smithi112 podman[67486]: 2024-06-04 04:53:06.986078313 +0000 UTC m=+0.378467217 container start abd1e979f2cc5bf11f1cfb25d6be62126dcd964a5ec4880fd096317802a28291 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-c, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-04T04:53:07.417 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 04:53:07 smithi112 ceph-mds[67507]: -- 172.21.15.112:0/2359378998 <== mon.1 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55de63c0ba40 con 0x55de63bdec00 2024-06-04T04:53:07.418 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 04:53:07 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-c[67503]: starting mds.c at 2024-06-04T04:53:07.418 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 04:53:07 smithi112 bash[67486]: abd1e979f2cc5bf11f1cfb25d6be62126dcd964a5ec4880fd096317802a28291 2024-06-04T04:53:07.418 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 04:53:07 smithi112 systemd[1]: Started Ceph mds.c for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:53:07.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:07 smithi070 ceph-mon[32631]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:07.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:07 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:07.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:07 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:07.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:07 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:07.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:07 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:07.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:07 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:07.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:07 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:07 smithi112 ceph-mon[36453]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:07 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:07 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:07 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:07 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:07 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:07.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:07 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:07 smithi098 ceph-mon[35569]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:07 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:07.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:07 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:07.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:07 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:07.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:07 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:07.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:07 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:07.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:07 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:08.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.325+0000 7f8a5115d700 1 -- 172.21.15.70:0/283809907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a4c06c470 msgr2=0x7f8a4c1103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:08.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.325+0000 7f8a5115d700 1 --2- 172.21.15.70:0/283809907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a4c06c470 0x7f8a4c1103c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009990 tx=0x7f8a3c009ca0 comp rx=0 tx=0).stop 2024-06-04T04:53:08.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.325+0000 7f8a5115d700 1 -- 172.21.15.70:0/283809907 shutdown_connections 2024-06-04T04:53:08.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.325+0000 7f8a5115d700 1 --2- 172.21.15.70:0/283809907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a4c06c470 0x7f8a4c1103c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:08.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.325+0000 7f8a5115d700 1 --2- 172.21.15.70:0/283809907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c1095a0 0x7f8a4c109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:08.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.325+0000 7f8a5115d700 1 --2- 172.21.15.70:0/283809907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a4c06bab0 0x7f8a4c06bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:08.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.325+0000 7f8a5115d700 1 -- 172.21.15.70:0/283809907 >> 172.21.15.70:0/283809907 conn(0x7f8a4c0793b0 msgr2=0x7f8a4c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:08.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.325+0000 7f8a5115d700 1 -- 172.21.15.70:0/283809907 shutdown_connections 2024-06-04T04:53:08.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.325+0000 7f8a5115d700 1 -- 172.21.15.70:0/283809907 wait complete. 2024-06-04T04:53:08.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.326+0000 7f8a5115d700 1 Processor -- start 2024-06-04T04:53:08.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.326+0000 7f8a5115d700 1 -- start start 2024-06-04T04:53:08.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.326+0000 7f8a5115d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a4c06bab0 0x7f8a4c19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.326+0000 7f8a5115d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c06c470 0x7f8a4c19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.326+0000 7f8a5115d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a4c1095a0 0x7f8a4c1998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.326+0000 7f8a5115d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c113570 con 0x7f8a4c06bab0 2024-06-04T04:53:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.326+0000 7f8a5115d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c1133f0 con 0x7f8a4c06c470 2024-06-04T04:53:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.326+0000 7f8a5115d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c1136f0 con 0x7f8a4c1095a0 2024-06-04T04:53:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.326+0000 7f8a4ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a4c06bab0 0x7f8a4c19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.326+0000 7f8a4ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a4c06bab0 0x7f8a4c19f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34950/0 (socket says 172.21.15.70:34950) 2024-06-04T04:53:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.326+0000 7f8a4ad9d700 1 -- 172.21.15.70:0/3002199823 learned_addr learned my addr 172.21.15.70:0/3002199823 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:53:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.326+0000 7f8a4a59c700 1 --2- 172.21.15.70:0/3002199823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c06c470 0x7f8a4c19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.326+0000 7f8a4b59e700 1 --2- 172.21.15.70:0/3002199823 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a4c1095a0 0x7f8a4c1998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.327+0000 7f8a4ad9d700 1 -- 172.21.15.70:0/3002199823 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a4c1095a0 msgr2=0x7f8a4c1998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.327+0000 7f8a4ad9d700 1 --2- 172.21.15.70:0/3002199823 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a4c1095a0 0x7f8a4c1998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.327+0000 7f8a4ad9d700 1 -- 172.21.15.70:0/3002199823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c06c470 msgr2=0x7f8a4c19fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.327+0000 7f8a4ad9d700 1 --2- 172.21.15.70:0/3002199823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c06c470 0x7f8a4c19fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:08.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.327+0000 7f8a4ad9d700 1 -- 172.21.15.70:0/3002199823 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c19a050 con 0x7f8a4c06bab0 2024-06-04T04:53:08.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.327+0000 7f8a4ad9d700 1 --2- 172.21.15.70:0/3002199823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a4c06bab0 0x7f8a4c19f730 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8a440060b0 tx=0x7f8a4400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:08.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.327+0000 7f8a43fff700 1 -- 172.21.15.70:0/3002199823 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4401a070 con 0x7f8a4c06bab0 2024-06-04T04:53:08.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.327+0000 7f8a43fff700 1 -- 172.21.15.70:0/3002199823 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a4400ddd0 con 0x7f8a4c06bab0 2024-06-04T04:53:08.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.327+0000 7f8a5115d700 1 -- 172.21.15.70:0/3002199823 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a4c19a340 con 0x7f8a4c06bab0 2024-06-04T04:53:08.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.327+0000 7f8a43fff700 1 -- 172.21.15.70:0/3002199823 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a44014690 con 0x7f8a4c06bab0 2024-06-04T04:53:08.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.327+0000 7f8a5115d700 1 -- 172.21.15.70:0/3002199823 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a4c1a6600 con 0x7f8a4c06bab0 2024-06-04T04:53:08.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.328+0000 7f8a5115d700 1 -- 172.21.15.70:0/3002199823 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a4c06a230 con 0x7f8a4c06bab0 2024-06-04T04:53:08.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.329+0000 7f8a43fff700 1 -- 172.21.15.70:0/3002199823 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f8a4401d070 con 0x7f8a4c06bab0 2024-06-04T04:53:08.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.330+0000 7f8a43fff700 1 --2- 172.21.15.70:0/3002199823 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8a380b2900 0x7f8a380b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:08.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.330+0000 7f8a4a59c700 1 --2- 172.21.15.70:0/3002199823 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8a380b2900 0x7f8a380b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:08.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.330+0000 7f8a43fff700 1 -- 172.21.15.70:0/3002199823 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f8a440e14a0 con 0x7f8a4c06bab0 2024-06-04T04:53:08.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.330+0000 7f8a4a59c700 1 --2- 172.21.15.70:0/3002199823 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8a380b2900 0x7f8a380b4dc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c079a00 tx=0x7f8a3400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:08.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.334+0000 7f8a43fff700 1 -- 172.21.15.70:0/3002199823 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8a44010070 con 0x7f8a4c06bab0 2024-06-04T04:53:08.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.592+0000 7f8a5115d700 1 -- 172.21.15.70:0/3002199823 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a4c02e560 con 0x7f8a4c06bab0 2024-06-04T04:53:08.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.592+0000 7f8a43fff700 1 -- 172.21.15.70:0/3002199823 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6346 (secure 0 0 0) 0x7f8a440a9660 con 0x7f8a4c06bab0 2024-06-04T04:53:08.595 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:53:08.595 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24386,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24436,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-06-04T04:53:08.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.595+0000 7f8a5115d700 1 -- 172.21.15.70:0/3002199823 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8a380b2900 msgr2=0x7f8a380b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:08.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.595+0000 7f8a5115d700 1 --2- 172.21.15.70:0/3002199823 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8a380b2900 0x7f8a380b4dc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c079a00 tx=0x7f8a3400a300 comp rx=0 tx=0).stop 2024-06-04T04:53:08.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.595+0000 7f8a5115d700 1 -- 172.21.15.70:0/3002199823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a4c06bab0 msgr2=0x7f8a4c19f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:08.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.595+0000 7f8a5115d700 1 --2- 172.21.15.70:0/3002199823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a4c06bab0 0x7f8a4c19f730 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8a440060b0 tx=0x7f8a4400d3b0 comp rx=0 tx=0).stop 2024-06-04T04:53:08.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.595+0000 7f8a5115d700 1 -- 172.21.15.70:0/3002199823 shutdown_connections 2024-06-04T04:53:08.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.595+0000 7f8a5115d700 1 --2- 172.21.15.70:0/3002199823 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8a380b2900 0x7f8a380b4dc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:08.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.595+0000 7f8a5115d700 1 --2- 172.21.15.70:0/3002199823 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a4c1095a0 0x7f8a4c1998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:08.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.595+0000 7f8a5115d700 1 --2- 172.21.15.70:0/3002199823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a4c06bab0 0x7f8a4c19f730 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:08.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.595+0000 7f8a5115d700 1 --2- 172.21.15.70:0/3002199823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c06c470 0x7f8a4c19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:08.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.595+0000 7f8a5115d700 1 -- 172.21.15.70:0/3002199823 >> 172.21.15.70:0/3002199823 conn(0x7f8a4c0793b0 msgr2=0x7f8a4c101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:08.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.595+0000 7f8a5115d700 1 -- 172.21.15.70:0/3002199823 shutdown_connections 2024-06-04T04:53:08.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:08.595+0000 7f8a5115d700 1 -- 172.21.15.70:0/3002199823 wait complete. 2024-06-04T04:53:08.599 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 8 2024-06-04T04:53:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:08 smithi112 ceph-mon[36453]: Deploying daemon mds.h on smithi098 2024-06-04T04:53:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:08 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] up:boot 2024-06-04T04:53:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:08 smithi112 ceph-mon[36453]: fsmap 7 up:standby 2024-06-04T04:53:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:08 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-04T04:53:08.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:08 smithi098 ceph-mon[35569]: Deploying daemon mds.h on smithi098 2024-06-04T04:53:08.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:08 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] up:boot 2024-06-04T04:53:08.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:08 smithi098 ceph-mon[35569]: fsmap 7 up:standby 2024-06-04T04:53:08.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:08 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-04T04:53:08.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:08 smithi070 ceph-mon[32631]: Deploying daemon mds.h on smithi098 2024-06-04T04:53:08.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:08 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] up:boot 2024-06-04T04:53:08.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:08 smithi070 ceph-mon[32631]: fsmap 7 up:standby 2024-06-04T04:53:08.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:08 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-04T04:53:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:09 smithi098 ceph-mon[35569]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:09.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:09 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3002199823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:53:09.516 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:09 smithi070 ceph-mon[32631]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:09.516 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:09 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3002199823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:53:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:09 smithi112 ceph-mon[36453]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:09 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3002199823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:53:10.245 INFO:journalctl@ceph.mds.h.smithi098.stdout:Jun 04 04:53:10 smithi098 systemd[1]: Started Ceph mds.h for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:53:11.255 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:53:11.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:11 smithi070 ceph-mon[32631]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:11.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:11 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:11.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:11 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:11.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:11 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:11.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:11 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:11.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:11 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:11.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:11 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:11.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:11 smithi070 ceph-mon[32631]: Deploying daemon mds.i on smithi112 2024-06-04T04:53:11.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:11 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] up:boot 2024-06-04T04:53:11.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:11 smithi070 ceph-mon[32631]: fsmap 8 up:standby 2024-06-04T04:53:11.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:11 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-04T04:53:11.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:11 smithi112 ceph-mon[36453]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:11.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:11 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:11.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:11 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:11.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:11 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:11.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:11 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:11.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:11 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:11.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:11 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:11.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:11 smithi112 ceph-mon[36453]: Deploying daemon mds.i on smithi112 2024-06-04T04:53:11.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:11 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] up:boot 2024-06-04T04:53:11.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:11 smithi112 ceph-mon[36453]: fsmap 8 up:standby 2024-06-04T04:53:11.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:11 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-04T04:53:11.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:11 smithi098 ceph-mon[35569]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:11.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:11 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:11.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:11 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:11 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:11 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:11 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:11 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:11 smithi098 ceph-mon[35569]: Deploying daemon mds.i on smithi112 2024-06-04T04:53:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:11 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] up:boot 2024-06-04T04:53:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:11 smithi098 ceph-mon[35569]: fsmap 8 up:standby 2024-06-04T04:53:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:11 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-04T04:53:11.509 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:53:12.436 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 04:53:12 smithi112 systemd[1]: Starting Ceph mds.i for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T04:53:12.763 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 04:53:12 smithi112 podman[67788]: 2024-06-04 04:53:12.392331307 +0000 UTC m=+0.017100575 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T04:53:12.763 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 04:53:12 smithi112 podman[67788]: 2024-06-04 04:53:12.499797152 +0000 UTC m=+0.124566412 container create ac055dfba103d64848825cd2a86f4a38a2700256fe7404114c0d0ee379b7ac87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-06-04T04:53:12.763 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 04:53:12 smithi112 podman[67788]: 2024-06-04 04:53:12.762111542 +0000 UTC m=+0.386880810 container init ac055dfba103d64848825cd2a86f4a38a2700256fe7404114c0d0ee379b7ac87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-06-04T04:53:13.116 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 04:53:12 smithi112 podman[67788]: 2024-06-04 04:53:12.764938502 +0000 UTC m=+0.389707771 container start ac055dfba103d64848825cd2a86f4a38a2700256fe7404114c0d0ee379b7ac87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-b7328fe, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-06-04T04:53:13.116 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 04:53:12 smithi112 ceph-mds[67808]: -- 172.21.15.112:0/3268989859 <== mon.1 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d6ddaa9a40 con 0x55d6dda7cc00 2024-06-04T04:53:13.116 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 04:53:12 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: starting mds.i at 2024-06-04T04:53:13.116 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 04:53:12 smithi112 bash[67788]: ac055dfba103d64848825cd2a86f4a38a2700256fe7404114c0d0ee379b7ac87 2024-06-04T04:53:13.116 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 04:53:12 smithi112 systemd[1]: Started Ceph mds.i for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:53:13.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.222+0000 7f0897b6a700 1 -- 172.21.15.70:0/2975266298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890104200 msgr2=0x7f08901081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:13.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.222+0000 7f0897b6a700 1 --2- 172.21.15.70:0/2975266298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890104200 0x7f08901081f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f08800099c0 tx=0x7f0880009cd0 comp rx=0 tx=0).stop 2024-06-04T04:53:13.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.222+0000 7f0897b6a700 1 -- 172.21.15.70:0/2975266298 shutdown_connections 2024-06-04T04:53:13.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.222+0000 7f0897b6a700 1 --2- 172.21.15.70:0/2975266298 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0890108920 0x7f08901104a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:13.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.222+0000 7f0897b6a700 1 --2- 172.21.15.70:0/2975266298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0890103850 0x7f0890103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:13.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.222+0000 7f0897b6a700 1 --2- 172.21.15.70:0/2975266298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890104200 0x7f08901081f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:13.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.222+0000 7f0897b6a700 1 -- 172.21.15.70:0/2975266298 >> 172.21.15.70:0/2975266298 conn(0x7f089007a5d0 msgr2=0x7f089007a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:13.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.222+0000 7f0897b6a700 1 -- 172.21.15.70:0/2975266298 shutdown_connections 2024-06-04T04:53:13.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.222+0000 7f0897b6a700 1 -- 172.21.15.70:0/2975266298 wait complete. 2024-06-04T04:53:13.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.223+0000 7f0897b6a700 1 Processor -- start 2024-06-04T04:53:13.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.223+0000 7f0897b6a700 1 -- start start 2024-06-04T04:53:13.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.223+0000 7f0897b6a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0890103850 0x7f089019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:13.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.223+0000 7f0897b6a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0890104200 0x7f089019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:13.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.223+0000 7f0897b6a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890108920 0x7f0890199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:13.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.223+0000 7f0897b6a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0890113610 con 0x7f0890108920 2024-06-04T04:53:13.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.223+0000 7f0897b6a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0890113490 con 0x7f0890103850 2024-06-04T04:53:13.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.223+0000 7f0897b6a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0890113790 con 0x7f0890104200 2024-06-04T04:53:13.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.223+0000 7f0895906700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0890103850 0x7f089019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:13.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.223+0000 7f0896107700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890108920 0x7f0890199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:13.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.223+0000 7f0895906700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0890103850 0x7f089019f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:43314/0 (socket says 172.21.15.70:43314) 2024-06-04T04:53:13.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.223+0000 7f0895906700 1 -- 172.21.15.70:0/2443491148 learned_addr learned my addr 172.21.15.70:0/2443491148 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:53:13.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.223+0000 7f0895105700 1 --2- 172.21.15.70:0/2443491148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0890104200 0x7f089019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:13.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.223+0000 7f0895906700 1 -- 172.21.15.70:0/2443491148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0890104200 msgr2=0x7f089019fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:13.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.223+0000 7f0895906700 1 --2- 172.21.15.70:0/2443491148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0890104200 0x7f089019fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:13.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.224+0000 7f0895906700 1 -- 172.21.15.70:0/2443491148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890108920 msgr2=0x7f0890199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:13.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.224+0000 7f0895906700 1 --2- 172.21.15.70:0/2443491148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890108920 0x7f0890199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:13.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.224+0000 7f0895906700 1 -- 172.21.15.70:0/2443491148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f089019a070 con 0x7f0890103850 2024-06-04T04:53:13.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.224+0000 7f0896107700 1 --2- 172.21.15.70:0/2443491148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890108920 0x7f0890199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:53:13.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.224+0000 7f0895906700 1 --2- 172.21.15.70:0/2443491148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0890103850 0x7f089019f750 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f088c00d980 tx=0x7f088c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:13.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.224+0000 7f0886ffd700 1 -- 172.21.15.70:0/2443491148 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c016070 con 0x7f0890103850 2024-06-04T04:53:13.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.224+0000 7f0897b6a700 1 -- 172.21.15.70:0/2443491148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f089019a360 con 0x7f0890103850 2024-06-04T04:53:13.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.224+0000 7f0886ffd700 1 -- 172.21.15.70:0/2443491148 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f088c004830 con 0x7f0890103850 2024-06-04T04:53:13.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.224+0000 7f0886ffd700 1 -- 172.21.15.70:0/2443491148 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c0054c0 con 0x7f0890103850 2024-06-04T04:53:13.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.224+0000 7f0897b6a700 1 -- 172.21.15.70:0/2443491148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08901a6620 con 0x7f0890103850 2024-06-04T04:53:13.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.225+0000 7f0897b6a700 1 -- 172.21.15.70:0/2443491148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f089006a230 con 0x7f0890103850 2024-06-04T04:53:13.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.226+0000 7f0886ffd700 1 -- 172.21.15.70:0/2443491148 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f088c004220 con 0x7f0890103850 2024-06-04T04:53:13.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.227+0000 7f0886ffd700 1 --2- 172.21.15.70:0/2443491148 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f087c0b2880 0x7f087c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:13.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.227+0000 7f0895105700 1 --2- 172.21.15.70:0/2443491148 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f087c0b2880 0x7f087c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:13.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.227+0000 7f0886ffd700 1 -- 172.21.15.70:0/2443491148 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f088c0e21b0 con 0x7f0890103850 2024-06-04T04:53:13.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.227+0000 7f0895105700 1 --2- 172.21.15.70:0/2443491148 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f087c0b2880 0x7f087c0b4d40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0880000c00 tx=0x7f0880005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:13.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.230+0000 7f0886ffd700 1 -- 172.21.15.70:0/2443491148 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f088c01f020 con 0x7f0890103850 2024-06-04T04:53:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:13 smithi070 ceph-mon[32631]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:13 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:13 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:13 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:13 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:13 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:13 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:13 smithi112 ceph-mon[36453]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:13 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:13 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:13 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:13 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:13 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:13 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:13.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.488+0000 7f0897b6a700 1 -- 172.21.15.70:0/2443491148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f089002e560 con 0x7f0890103850 2024-06-04T04:53:13.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.489+0000 7f0886ffd700 1 -- 172.21.15.70:0/2443491148 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7f088c0aa2d0 con 0x7f0890103850 2024-06-04T04:53:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:53:13.492 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24386,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24410,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24436,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24448,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-06-04T04:53:13.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.492+0000 7f0897b6a700 1 -- 172.21.15.70:0/2443491148 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f087c0b2880 msgr2=0x7f087c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:13.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.492+0000 7f0897b6a700 1 --2- 172.21.15.70:0/2443491148 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f087c0b2880 0x7f087c0b4d40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0880000c00 tx=0x7f0880005fb0 comp rx=0 tx=0).stop 2024-06-04T04:53:13.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.492+0000 7f0897b6a700 1 -- 172.21.15.70:0/2443491148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0890103850 msgr2=0x7f089019f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:13.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.492+0000 7f0897b6a700 1 --2- 172.21.15.70:0/2443491148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0890103850 0x7f089019f750 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f088c00d980 tx=0x7f088c00dc90 comp rx=0 tx=0).stop 2024-06-04T04:53:13.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.492+0000 7f0897b6a700 1 -- 172.21.15.70:0/2443491148 shutdown_connections 2024-06-04T04:53:13.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.492+0000 7f0897b6a700 1 --2- 172.21.15.70:0/2443491148 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f087c0b2880 0x7f087c0b4d40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:13.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.492+0000 7f0897b6a700 1 --2- 172.21.15.70:0/2443491148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890108920 0x7f0890199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:13.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.492+0000 7f0897b6a700 1 --2- 172.21.15.70:0/2443491148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0890103850 0x7f089019f750 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:13.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.492+0000 7f0897b6a700 1 --2- 172.21.15.70:0/2443491148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0890104200 0x7f089019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:13.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.492+0000 7f0897b6a700 1 -- 172.21.15.70:0/2443491148 >> 172.21.15.70:0/2443491148 conn(0x7f089007a5d0 msgr2=0x7f0890101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:13.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.492+0000 7f0897b6a700 1 -- 172.21.15.70:0/2443491148 shutdown_connections 2024-06-04T04:53:13.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:13.492+0000 7f0897b6a700 1 -- 172.21.15.70:0/2443491148 wait complete. 2024-06-04T04:53:13.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:13 smithi098 ceph-mon[35569]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:13.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:13.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:13.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:13.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:13.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:13.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:13 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:13.496 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 10 2024-06-04T04:53:14.372 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:14 smithi070 ceph-mon[32631]: Deploying daemon mds.g on smithi070 2024-06-04T04:53:14.372 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:14 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] up:boot 2024-06-04T04:53:14.372 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:14 smithi070 ceph-mon[32631]: fsmap 9 up:standby 2024-06-04T04:53:14.372 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:14 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-04T04:53:14.372 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:14 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2443491148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:53:14.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:14 smithi112 ceph-mon[36453]: Deploying daemon mds.g on smithi070 2024-06-04T04:53:14.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:14 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] up:boot 2024-06-04T04:53:14.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:14 smithi112 ceph-mon[36453]: fsmap 9 up:standby 2024-06-04T04:53:14.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:14 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-04T04:53:14.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:14 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2443491148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:53:14.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:14 smithi098 ceph-mon[35569]: Deploying daemon mds.g on smithi070 2024-06-04T04:53:14.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:14 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] up:boot 2024-06-04T04:53:14.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:14 smithi098 ceph-mon[35569]: fsmap 9 up:standby 2024-06-04T04:53:14.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:14 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-04T04:53:14.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:14 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2443491148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:53:15.259 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:15 smithi070 ceph-mon[32631]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:15.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:15 smithi112 ceph-mon[36453]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:15.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:15 smithi098 ceph-mon[35569]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:15.859 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 04:53:15 smithi070 systemd[1]: Starting Ceph mds.g for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T04:53:16.145 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:53:16.232 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 04:53:15 smithi070 podman[73629]: 2024-06-04 04:53:15.859790636 +0000 UTC m=+0.017567960 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T04:53:16.232 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 04:53:16 smithi070 podman[73629]: 2024-06-04 04:53:16.001801684 +0000 UTC m=+0.159578995 container create ee01603f0be4382400ef24bb04a4ed8c3e3cd036a36119d0fcd9e630c8849192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-g, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240521) 2024-06-04T04:53:16.232 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 04:53:16 smithi070 podman[73629]: 2024-06-04 04:53:16.230121987 +0000 UTC m=+0.387899316 container init ee01603f0be4382400ef24bb04a4ed8c3e3cd036a36119d0fcd9e630c8849192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-06-04T04:53:16.410 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:53:16.532 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 04:53:16 smithi070 podman[73629]: 2024-06-04 04:53:16.233491512 +0000 UTC m=+0.391268839 container start ee01603f0be4382400ef24bb04a4ed8c3e3cd036a36119d0fcd9e630c8849192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-g, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-b7328fe, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-04T04:53:16.533 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 04:53:16 smithi070 ceph-mds[73666]: -- 172.21.15.70:0/3795260840 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b69ad53a40 con 0x55b69ad26800 2024-06-04T04:53:16.533 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 04:53:16 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-g[73644]: starting mds.g at 2024-06-04T04:53:16.533 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 04:53:16 smithi070 bash[73629]: ee01603f0be4382400ef24bb04a4ed8c3e3cd036a36119d0fcd9e630c8849192 2024-06-04T04:53:16.533 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 04:53:16 smithi070 systemd[1]: Started Ceph mds.g for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:53:17.523 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:17 smithi070 ceph-mon[32631]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:17.523 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:17 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:17.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:17 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:17.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:17 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:17.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:17 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:17.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:17 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:17.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:17 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:17.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:17 smithi070 ceph-mon[32631]: Deploying daemon mds.a on smithi070 2024-06-04T04:53:17.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:17 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.70:6838/3817259285,v1:172.21.15.70:6839/3817259285] up:boot 2024-06-04T04:53:17.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:17 smithi070 ceph-mon[32631]: fsmap 10 up:standby 2024-06-04T04:53:17.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:17 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-04T04:53:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:17 smithi112 ceph-mon[36453]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:17 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:17 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:17 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:17 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:17.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:17 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:17.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:17 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:17.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:17 smithi112 ceph-mon[36453]: Deploying daemon mds.a on smithi070 2024-06-04T04:53:17.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:17 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.70:6838/3817259285,v1:172.21.15.70:6839/3817259285] up:boot 2024-06-04T04:53:17.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:17 smithi112 ceph-mon[36453]: fsmap 10 up:standby 2024-06-04T04:53:17.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:17 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-04T04:53:17.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:17 smithi098 ceph-mon[35569]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:17.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:17 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:17.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:17 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:17.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:17 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:17.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:17 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:17.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:17 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:17.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:17 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:17.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:17 smithi098 ceph-mon[35569]: Deploying daemon mds.a on smithi070 2024-06-04T04:53:17.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:17 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.70:6838/3817259285,v1:172.21.15.70:6839/3817259285] up:boot 2024-06-04T04:53:17.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:17 smithi098 ceph-mon[35569]: fsmap 10 up:standby 2024-06-04T04:53:17.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:17 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-04T04:53:18.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.070+0000 7fe0ebfe6700 1 -- 172.21.15.70:0/3370385207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e41041a0 msgr2=0x7fe0e41081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:18.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.070+0000 7fe0ebfe6700 1 --2- 172.21.15.70:0/3370385207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e41041a0 0x7fe0e41081c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe0d40099c0 tx=0x7fe0d4009cd0 comp rx=0 tx=0).stop 2024-06-04T04:53:18.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.070+0000 7fe0ebfe6700 1 -- 172.21.15.70:0/3370385207 shutdown_connections 2024-06-04T04:53:18.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.070+0000 7fe0ebfe6700 1 --2- 172.21.15.70:0/3370385207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0e41088f0 0x7fe0e4110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:18.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.070+0000 7fe0ebfe6700 1 --2- 172.21.15.70:0/3370385207 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe0e41037f0 0x7fe0e4103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:18.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.070+0000 7fe0ebfe6700 1 --2- 172.21.15.70:0/3370385207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e41041a0 0x7fe0e41081c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:18.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.070+0000 7fe0ebfe6700 1 -- 172.21.15.70:0/3370385207 >> 172.21.15.70:0/3370385207 conn(0x7fe0e407a6a0 msgr2=0x7fe0e407aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:18.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.070+0000 7fe0ebfe6700 1 -- 172.21.15.70:0/3370385207 shutdown_connections 2024-06-04T04:53:18.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.070+0000 7fe0ebfe6700 1 -- 172.21.15.70:0/3370385207 wait complete. 2024-06-04T04:53:18.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0ebfe6700 1 Processor -- start 2024-06-04T04:53:18.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0ebfe6700 1 -- start start 2024-06-04T04:53:18.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0ebfe6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0e41037f0 0x7fe0e419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:18.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0ebfe6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e41041a0 0x7fe0e419fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:18.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0ebfe6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe0e41088f0 0x7fe0e41998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:18.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0ebfe6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e4113580 con 0x7fe0e41041a0 2024-06-04T04:53:18.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0ebfe6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e4113400 con 0x7fe0e41037f0 2024-06-04T04:53:18.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0ebfe6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e4113700 con 0x7fe0e41088f0 2024-06-04T04:53:18.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0e9581700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e41041a0 0x7fe0e419fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:18.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0e9581700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e41041a0 0x7fe0e419fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39550/0 (socket says 172.21.15.70:39550) 2024-06-04T04:53:18.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0e9581700 1 -- 172.21.15.70:0/2077414487 learned_addr learned my addr 172.21.15.70:0/2077414487 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:53:18.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0e9d82700 1 --2- 172.21.15.70:0/2077414487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0e41037f0 0x7fe0e419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:18.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0ea583700 1 --2- 172.21.15.70:0/2077414487 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe0e41088f0 0x7fe0e41998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:18.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0e9581700 1 -- 172.21.15.70:0/2077414487 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe0e41088f0 msgr2=0x7fe0e41998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:18.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0e9581700 1 --2- 172.21.15.70:0/2077414487 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe0e41088f0 0x7fe0e41998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:18.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0e9581700 1 -- 172.21.15.70:0/2077414487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0e41037f0 msgr2=0x7fe0e419f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:18.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0e9581700 1 --2- 172.21.15.70:0/2077414487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0e41037f0 0x7fe0e419f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:18.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0e9581700 1 -- 172.21.15.70:0/2077414487 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e419a040 con 0x7fe0e41041a0 2024-06-04T04:53:18.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.071+0000 7fe0ea583700 1 --2- 172.21.15.70:0/2077414487 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe0e41088f0 0x7fe0e41998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:53:18.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.072+0000 7fe0e9581700 1 --2- 172.21.15.70:0/2077414487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e41041a0 0x7fe0e419fc60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4000c00 tx=0x7fe0d400f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:18.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.072+0000 7fe0daffd700 1 -- 172.21.15.70:0/2077414487 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d400f940 con 0x7fe0e41041a0 2024-06-04T04:53:18.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.072+0000 7fe0daffd700 1 -- 172.21.15.70:0/2077414487 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0d4004830 con 0x7fe0e41041a0 2024-06-04T04:53:18.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.072+0000 7fe0ebfe6700 1 -- 172.21.15.70:0/2077414487 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0e419a2d0 con 0x7fe0e41041a0 2024-06-04T04:53:18.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.072+0000 7fe0ebfe6700 1 -- 172.21.15.70:0/2077414487 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0e4075770 con 0x7fe0e41041a0 2024-06-04T04:53:18.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.072+0000 7fe0daffd700 1 -- 172.21.15.70:0/2077414487 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d40054b0 con 0x7fe0e41041a0 2024-06-04T04:53:18.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.073+0000 7fe0ebfe6700 1 -- 172.21.15.70:0/2077414487 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0e410bac0 con 0x7fe0e41041a0 2024-06-04T04:53:18.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.074+0000 7fe0daffd700 1 -- 172.21.15.70:0/2077414487 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fe0d4029060 con 0x7fe0e41041a0 2024-06-04T04:53:18.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.075+0000 7fe0daffd700 1 --2- 172.21.15.70:0/2077414487 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe0d00b2900 0x7fe0d00b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:18.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.075+0000 7fe0e9d82700 1 --2- 172.21.15.70:0/2077414487 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe0d00b2900 0x7fe0d00b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:18.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.075+0000 7fe0daffd700 1 -- 172.21.15.70:0/2077414487 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fe0d40e2ad0 con 0x7fe0e41041a0 2024-06-04T04:53:18.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.075+0000 7fe0e9d82700 1 --2- 172.21.15.70:0/2077414487 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe0d00b2900 0x7fe0d00b4dc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0005950 tx=0x7fe0e0005eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:18.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.078+0000 7fe0daffd700 1 -- 172.21.15.70:0/2077414487 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe0d4013070 con 0x7fe0e41041a0 2024-06-04T04:53:18.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.333+0000 7fe0ebfe6700 1 -- 172.21.15.70:0/2077414487 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0e4104c80 con 0x7fe0e41041a0 2024-06-04T04:53:18.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.333+0000 7fe0daffd700 1 -- 172.21.15.70:0/2077414487 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7fe0d40aad20 con 0x7fe0e41041a0 2024-06-04T04:53:18.336 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:53:18.336 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24386,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24410,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24436,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24448,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-06-04T04:53:18.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.336+0000 7fe0ebfe6700 1 -- 172.21.15.70:0/2077414487 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe0d00b2900 msgr2=0x7fe0d00b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:18.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.336+0000 7fe0ebfe6700 1 --2- 172.21.15.70:0/2077414487 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe0d00b2900 0x7fe0d00b4dc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0005950 tx=0x7fe0e0005eb0 comp rx=0 tx=0).stop 2024-06-04T04:53:18.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.336+0000 7fe0ebfe6700 1 -- 172.21.15.70:0/2077414487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e41041a0 msgr2=0x7fe0e419fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:18.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.336+0000 7fe0ebfe6700 1 --2- 172.21.15.70:0/2077414487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e41041a0 0x7fe0e419fc60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4000c00 tx=0x7fe0d400f510 comp rx=0 tx=0).stop 2024-06-04T04:53:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.336+0000 7fe0ebfe6700 1 -- 172.21.15.70:0/2077414487 shutdown_connections 2024-06-04T04:53:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.336+0000 7fe0ebfe6700 1 --2- 172.21.15.70:0/2077414487 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe0d00b2900 0x7fe0d00b4dc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.336+0000 7fe0ebfe6700 1 --2- 172.21.15.70:0/2077414487 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe0e41088f0 0x7fe0e41998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.336+0000 7fe0ebfe6700 1 --2- 172.21.15.70:0/2077414487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0e41037f0 0x7fe0e419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.336+0000 7fe0ebfe6700 1 --2- 172.21.15.70:0/2077414487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e41041a0 0x7fe0e419fc60 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.336+0000 7fe0ebfe6700 1 -- 172.21.15.70:0/2077414487 >> 172.21.15.70:0/2077414487 conn(0x7fe0e407a6a0 msgr2=0x7fe0e4101b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.336+0000 7fe0ebfe6700 1 -- 172.21.15.70:0/2077414487 shutdown_connections 2024-06-04T04:53:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:18.336+0000 7fe0ebfe6700 1 -- 172.21.15.70:0/2077414487 wait complete. 2024-06-04T04:53:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 11 2024-06-04T04:53:18.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:18 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2077414487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:53:18.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:18 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2077414487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:53:18.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:18 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2077414487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:53:19.601 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:19 smithi070 ceph-mon[32631]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:19 smithi112 ceph-mon[36453]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:19.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:19 smithi098 ceph-mon[35569]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:19.859 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 04:53:19 smithi070 systemd[1]: Starting Ceph mds.a for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T04:53:20.452 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 04:53:20 smithi070 podman[74088]: 2024-06-04 04:53:20.334391323 +0000 UTC m=+0.017094315 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T04:53:20.453 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 04:53:20 smithi070 podman[74088]: 2024-06-04 04:53:20.451332698 +0000 UTC m=+0.134035676 container create 2227476cb2266f698108b64582e84028ea20f8a8cecf3c3ee28bb1c1c6092be6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=main-b7328fe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521) 2024-06-04T04:53:20.717 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 04:53:20 smithi070 podman[74088]: 2024-06-04 04:53:20.687636051 +0000 UTC m=+0.370339032 container init 2227476cb2266f698108b64582e84028ea20f8a8cecf3c3ee28bb1c1c6092be6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-b7328fe) 2024-06-04T04:53:20.717 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 04:53:20 smithi070 podman[74088]: 2024-06-04 04:53:20.69156784 +0000 UTC m=+0.374270825 container start 2227476cb2266f698108b64582e84028ea20f8a8cecf3c3ee28bb1c1c6092be6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-a, ceph=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=main-b7328fe) 2024-06-04T04:53:21.109 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 04:53:20 smithi070 ceph-mds[74107]: -- 172.21.15.70:0/3010154636 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5568ab2cfa40 con 0x5568ab2a2800 2024-06-04T04:53:21.109 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 04:53:20 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-a[74103]: starting mds.a at 2024-06-04T04:53:21.109 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 04:53:20 smithi070 bash[74088]: 2227476cb2266f698108b64582e84028ea20f8a8cecf3c3ee28bb1c1c6092be6 2024-06-04T04:53:21.109 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 04:53:20 smithi070 systemd[1]: Started Ceph mds.a for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:53:21.662 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:21 smithi098 ceph-mon[35569]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:21.662 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:21 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:21.663 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:21 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:21.663 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:21 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:21.663 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:21 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:21.663 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:21 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:21.663 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:21 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:21 smithi112 ceph-mon[36453]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:21 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:21 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:21 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:21 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:21 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:21 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:21 smithi070 ceph-mon[32631]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:21 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:21 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:21 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:21 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-04T04:53:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:21 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-04T04:53:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:21 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:22.212 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:53:22.467 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:53:22.589 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:22 smithi070 ceph-mon[32631]: Deploying daemon mds.b on smithi098 2024-06-04T04:53:22.589 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:22 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.70:6840/2748767095,v1:172.21.15.70:6841/2748767095] up:boot 2024-06-04T04:53:22.589 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:22 smithi070 ceph-mon[32631]: fsmap 11 up:standby 2024-06-04T04:53:22.590 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:22 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-04T04:53:22.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:22 smithi112 ceph-mon[36453]: Deploying daemon mds.b on smithi098 2024-06-04T04:53:22.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:22 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.70:6840/2748767095,v1:172.21.15.70:6841/2748767095] up:boot 2024-06-04T04:53:22.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:22 smithi112 ceph-mon[36453]: fsmap 11 up:standby 2024-06-04T04:53:22.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:22 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-04T04:53:22.716 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:22 smithi098 ceph-mon[35569]: Deploying daemon mds.b on smithi098 2024-06-04T04:53:22.716 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:22 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.70:6840/2748767095,v1:172.21.15.70:6841/2748767095] up:boot 2024-06-04T04:53:22.716 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:22 smithi098 ceph-mon[35569]: fsmap 11 up:standby 2024-06-04T04:53:22.716 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:22 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-04T04:53:23.408 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 04:53:23 smithi098 podman[65466]: 2024-06-04 04:53:23.326342581 +0000 UTC m=+0.137348039 container create 5d68ed213d1f94a119bf9ca1d2c65f897bf33a49879c1b061f4d258ed145656c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-b7328fe, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-06-04T04:53:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:23 smithi112 ceph-mon[36453]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:23.697 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:23 smithi098 ceph-mon[35569]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:23.698 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 04:53:23 smithi098 podman[65466]: 2024-06-04 04:53:23.6053165 +0000 UTC m=+0.416321900 container init 5d68ed213d1f94a119bf9ca1d2c65f897bf33a49879c1b061f4d258ed145656c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-b7328fe, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-06-04T04:53:23.698 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 04:53:23 smithi098 podman[65466]: 2024-06-04 04:53:23.608204665 +0000 UTC m=+0.419210060 container start 5d68ed213d1f94a119bf9ca1d2c65f897bf33a49879c1b061f4d258ed145656c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-06-04T04:53:23.698 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 04:53:23 smithi098 ceph-mds[65485]: -- 172.21.15.98:0/504933102 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ff6b485a40 con 0x55ff6b458c00 2024-06-04T04:53:23.698 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 04:53:23 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-b[65481]: starting mds.b at 2024-06-04T04:53:23.698 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 04:53:23 smithi098 bash[65466]: 5d68ed213d1f94a119bf9ca1d2c65f897bf33a49879c1b061f4d258ed145656c 2024-06-04T04:53:23.810 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:23 smithi070 ceph-mon[32631]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:23.995 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 04:53:23 smithi098 systemd[1]: Started Ceph mds.b for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T04:53:24.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.165+0000 7ffad479c700 1 -- 172.21.15.70:0/1373894581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffacc109680 msgr2=0x7ffacc109a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:24.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.165+0000 7ffad479c700 1 --2- 172.21.15.70:0/1373894581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffacc109680 0x7ffacc109a60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffac80099c0 tx=0x7ffac8009cd0 comp rx=0 tx=0).stop 2024-06-04T04:53:24.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.165+0000 7ffad479c700 1 -- 172.21.15.70:0/1373894581 shutdown_connections 2024-06-04T04:53:24.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.165+0000 7ffad479c700 1 --2- 172.21.15.70:0/1373894581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffacc103fa0 0x7ffacc110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:24.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.165+0000 7ffad479c700 1 --2- 172.21.15.70:0/1373894581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffacc109680 0x7ffacc109a60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:24.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.165+0000 7ffad479c700 1 --2- 172.21.15.70:0/1373894581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffacc1035e0 0x7ffacc103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:24.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.165+0000 7ffad479c700 1 -- 172.21.15.70:0/1373894581 >> 172.21.15.70:0/1373894581 conn(0x7ffacc07a6f0 msgr2=0x7ffacc07ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:24.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.165+0000 7ffad479c700 1 -- 172.21.15.70:0/1373894581 shutdown_connections 2024-06-04T04:53:24.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.165+0000 7ffad479c700 1 -- 172.21.15.70:0/1373894581 wait complete. 2024-06-04T04:53:24.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.165+0000 7ffad479c700 1 Processor -- start 2024-06-04T04:53:24.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.165+0000 7ffad479c700 1 -- start start 2024-06-04T04:53:24.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad479c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffacc1035e0 0x7ffacc19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:24.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad479c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffacc103fa0 0x7ffacc19fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:24.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad479c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffacc109680 0x7ffacc199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:24.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad479c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc113630 con 0x7ffacc103fa0 2024-06-04T04:53:24.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad479c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc1134b0 con 0x7ffacc109680 2024-06-04T04:53:24.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad479c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc1137b0 con 0x7ffacc1035e0 2024-06-04T04:53:24.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad1d37700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffacc103fa0 0x7ffacc19fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:24.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad2538700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffacc1035e0 0x7ffacc19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:24.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad2538700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffacc1035e0 0x7ffacc19f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:58272/0 (socket says 172.21.15.70:58272) 2024-06-04T04:53:24.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad1d37700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffacc103fa0 0x7ffacc19fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39590/0 (socket says 172.21.15.70:39590) 2024-06-04T04:53:24.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad2538700 1 -- 172.21.15.70:0/3732471056 learned_addr learned my addr 172.21.15.70:0/3732471056 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:53:24.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad2d39700 1 --2- 172.21.15.70:0/3732471056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffacc109680 0x7ffacc199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:24.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad1d37700 1 -- 172.21.15.70:0/3732471056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffacc1035e0 msgr2=0x7ffacc19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:24.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad1d37700 1 --2- 172.21.15.70:0/3732471056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffacc1035e0 0x7ffacc19f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:24.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad1d37700 1 -- 172.21.15.70:0/3732471056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffacc109680 msgr2=0x7ffacc199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:24.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad1d37700 1 --2- 172.21.15.70:0/3732471056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffacc109680 0x7ffacc199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:24.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad1d37700 1 -- 172.21.15.70:0/3732471056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffacc19a0d0 con 0x7ffacc103fa0 2024-06-04T04:53:24.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad2538700 1 --2- 172.21.15.70:0/3732471056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffacc1035e0 0x7ffacc19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:53:24.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad1d37700 1 --2- 172.21.15.70:0/3732471056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffacc103fa0 0x7ffacc19fcf0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00d750 tx=0x7ffabc00da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:24.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffac37fe700 1 -- 172.21.15.70:0/3732471056 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffabc014030 con 0x7ffacc103fa0 2024-06-04T04:53:24.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad479c700 1 -- 172.21.15.70:0/3732471056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffacc19a3c0 con 0x7ffacc103fa0 2024-06-04T04:53:24.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffac37fe700 1 -- 172.21.15.70:0/3732471056 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffabc0044b0 con 0x7ffacc103fa0 2024-06-04T04:53:24.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffac37fe700 1 -- 172.21.15.70:0/3732471056 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffabc005280 con 0x7ffacc103fa0 2024-06-04T04:53:24.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.166+0000 7ffad479c700 1 -- 172.21.15.70:0/3732471056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffacc1a6680 con 0x7ffacc103fa0 2024-06-04T04:53:24.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.168+0000 7ffad479c700 1 -- 172.21.15.70:0/3732471056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffacc10bb10 con 0x7ffacc103fa0 2024-06-04T04:53:24.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.169+0000 7ffac37fe700 1 -- 172.21.15.70:0/3732471056 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7ffabc0049e0 con 0x7ffacc103fa0 2024-06-04T04:53:24.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.169+0000 7ffac37fe700 1 --2- 172.21.15.70:0/3732471056 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ffab80b2880 0x7ffab80b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:24.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.170+0000 7ffad2538700 1 --2- 172.21.15.70:0/3732471056 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ffab80b2880 0x7ffab80b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:24.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.170+0000 7ffac37fe700 1 -- 172.21.15.70:0/3732471056 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7ffabc0e1610 con 0x7ffacc103fa0 2024-06-04T04:53:24.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.170+0000 7ffad2538700 1 --2- 172.21.15.70:0/3732471056 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ffab80b2880 0x7ffab80b4d40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffac8009f50 tx=0x7ffac801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:24.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.174+0000 7ffac37fe700 1 -- 172.21.15.70:0/3732471056 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ffabc0a97d0 con 0x7ffacc103fa0 2024-06-04T04:53:24.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.433+0000 7ffad479c700 1 -- 172.21.15.70:0/3732471056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffacc19aea0 con 0x7ffacc103fa0 2024-06-04T04:53:24.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.434+0000 7ffac37fe700 1 -- 172.21.15.70:0/3732471056 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10479 (secure 0 0 0) 0x7ffabc0a90d0 con 0x7ffacc103fa0 2024-06-04T04:53:24.448 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:53:24.448 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24386,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24410,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24436,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24440,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24448,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-06-04T04:53:24.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.437+0000 7ffad479c700 1 -- 172.21.15.70:0/3732471056 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ffab80b2880 msgr2=0x7ffab80b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:24.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.437+0000 7ffad479c700 1 --2- 172.21.15.70:0/3732471056 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ffab80b2880 0x7ffab80b4d40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffac8009f50 tx=0x7ffac801a040 comp rx=0 tx=0).stop 2024-06-04T04:53:24.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.437+0000 7ffad479c700 1 -- 172.21.15.70:0/3732471056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffacc103fa0 msgr2=0x7ffacc19fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.437+0000 7ffad479c700 1 --2- 172.21.15.70:0/3732471056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffacc103fa0 0x7ffacc19fcf0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00d750 tx=0x7ffabc00da60 comp rx=0 tx=0).stop 2024-06-04T04:53:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.437+0000 7ffad479c700 1 -- 172.21.15.70:0/3732471056 shutdown_connections 2024-06-04T04:53:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.437+0000 7ffad479c700 1 --2- 172.21.15.70:0/3732471056 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ffab80b2880 0x7ffab80b4d40 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.437+0000 7ffad479c700 1 --2- 172.21.15.70:0/3732471056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffacc109680 0x7ffacc199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.437+0000 7ffad479c700 1 --2- 172.21.15.70:0/3732471056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffacc1035e0 0x7ffacc19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.437+0000 7ffad479c700 1 --2- 172.21.15.70:0/3732471056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffacc103fa0 0x7ffacc19fcf0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.437+0000 7ffad479c700 1 -- 172.21.15.70:0/3732471056 >> 172.21.15.70:0/3732471056 conn(0x7ffacc07a6f0 msgr2=0x7ffacc102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.437+0000 7ffad479c700 1 -- 172.21.15.70:0/3732471056 shutdown_connections 2024-06-04T04:53:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:24.438+0000 7ffad479c700 1 -- 172.21.15.70:0/3732471056 wait complete. 2024-06-04T04:53:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 13 2024-06-04T04:53:25.035 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:24 smithi098 ceph-mon[35569]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:25.035 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:24 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:25.035 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:24 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:25.035 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:24 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:25.035 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:24 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:25.036 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:24 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:25.036 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:24 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:53:25.036 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:24 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] up:boot 2024-06-04T04:53:25.036 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:24 smithi098 ceph-mon[35569]: fsmap 12 up:standby 2024-06-04T04:53:25.036 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:24 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-04T04:53:25.036 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:24 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3732471056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:53:25.050 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:24 smithi112 ceph-mon[36453]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:25.051 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:24 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:25.051 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:24 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:25.051 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:24 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:25.051 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:24 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:25.051 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:24 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:25.051 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:24 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:53:25.051 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:24 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] up:boot 2024-06-04T04:53:25.051 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:24 smithi112 ceph-mon[36453]: fsmap 12 up:standby 2024-06-04T04:53:25.051 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:24 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-04T04:53:25.051 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:24 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3732471056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:53:25.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:24 smithi070 ceph-mon[32631]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:25.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:24 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:25.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:24 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:25.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:24 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:25.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:24 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:25.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:24 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:25.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:24 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:53:25.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:24 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] up:boot 2024-06-04T04:53:25.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:24 smithi070 ceph-mon[32631]: fsmap 12 up:standby 2024-06-04T04:53:25.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:24 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-04T04:53:25.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:24 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3732471056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:53:25.106 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-06-04T04:53:26.003 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:53:26.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:26.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:26.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:26 smithi112 ceph-mon[36453]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:26.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:26.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:26 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:26.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:26.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:26.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:26 smithi098 ceph-mon[35569]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:26.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:26.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:26 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:27.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:27.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:27.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:26 smithi070 ceph-mon[32631]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:27.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:27.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:26 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:27.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.658+0000 7f2cf0d06700 1 -- 172.21.15.70:0/3152142839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cec06c670 msgr2=0x7f2cec110420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:27.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.658+0000 7f2cf0d06700 1 --2- 172.21.15.70:0/3152142839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cec06c670 0x7f2cec110420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2ce400b0a0 tx=0x7f2ce400b3b0 comp rx=0 tx=0).stop 2024-06-04T04:53:27.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.658+0000 7f2cf0d06700 1 -- 172.21.15.70:0/3152142839 shutdown_connections 2024-06-04T04:53:27.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.658+0000 7f2cf0d06700 1 --2- 172.21.15.70:0/3152142839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cec06c670 0x7f2cec110420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:27.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.658+0000 7f2cf0d06700 1 --2- 172.21.15.70:0/3152142839 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cec1095e0 0x7f2cec1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:27.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.658+0000 7f2cf0d06700 1 --2- 172.21.15.70:0/3152142839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cec06bcb0 0x7f2cec06c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:27.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.658+0000 7f2cf0d06700 1 -- 172.21.15.70:0/3152142839 >> 172.21.15.70:0/3152142839 conn(0x7f2cec0784b0 msgr2=0x7f2cec0788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:27.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.659+0000 7f2cf0d06700 1 -- 172.21.15.70:0/3152142839 shutdown_connections 2024-06-04T04:53:27.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.659+0000 7f2cf0d06700 1 -- 172.21.15.70:0/3152142839 wait complete. 2024-06-04T04:53:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.659+0000 7f2cf0d06700 1 Processor -- start 2024-06-04T04:53:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.659+0000 7f2cf0d06700 1 -- start start 2024-06-04T04:53:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2cf0d06700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cec06bcb0 0x7f2cec102540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2cf0d06700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cec06c670 0x7f2cec102a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2cf0d06700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cec1095e0 0x7f2cec102fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2cf0d06700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec113660 con 0x7f2cec06c670 2024-06-04T04:53:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2cf0d06700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec1134e0 con 0x7f2cec06bcb0 2024-06-04T04:53:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2cf0d06700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec1137e0 con 0x7f2cec1095e0 2024-06-04T04:53:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2ce9d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cec06c670 0x7f2cec102a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2ce9d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cec06c670 0x7f2cec102a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50640/0 (socket says 172.21.15.70:50640) 2024-06-04T04:53:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2ce9d9b700 1 -- 172.21.15.70:0/334370092 learned_addr learned my addr 172.21.15.70:0/334370092 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:53:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2cead9d700 1 --2- 172.21.15.70:0/334370092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cec1095e0 0x7f2cec102fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2cea59c700 1 --2- 172.21.15.70:0/334370092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cec06bcb0 0x7f2cec102540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2ce9d9b700 1 -- 172.21.15.70:0/334370092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cec1095e0 msgr2=0x7f2cec102fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2ce9d9b700 1 --2- 172.21.15.70:0/334370092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cec1095e0 0x7f2cec102fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2ce9d9b700 1 -- 172.21.15.70:0/334370092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cec06bcb0 msgr2=0x7f2cec102540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2ce9d9b700 1 --2- 172.21.15.70:0/334370092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cec06bcb0 0x7f2cec102540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2ce9d9b700 1 -- 172.21.15.70:0/334370092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cec1a8100 con 0x7f2cec06c670 2024-06-04T04:53:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2ce9d9b700 1 --2- 172.21.15.70:0/334370092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cec06c670 0x7f2cec102a80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2cd400b530 tx=0x7f2cd400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2ce37fe700 1 -- 172.21.15.70:0/334370092 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4015020 con 0x7f2cec06c670 2024-06-04T04:53:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2ce37fe700 1 -- 172.21.15.70:0/334370092 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2cd4011cc0 con 0x7f2cec06c670 2024-06-04T04:53:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.660+0000 7f2cf0d06700 1 -- 172.21.15.70:0/334370092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cec1a8390 con 0x7f2cec06c670 2024-06-04T04:53:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.661+0000 7f2ce37fe700 1 -- 172.21.15.70:0/334370092 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4010c80 con 0x7f2cec06c670 2024-06-04T04:53:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.661+0000 7f2cf0d06700 1 -- 172.21.15.70:0/334370092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cec1a8920 con 0x7f2cec06c670 2024-06-04T04:53:27.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.661+0000 7f2cf0d06700 1 -- 172.21.15.70:0/334370092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cec10bae0 con 0x7f2cec06c670 2024-06-04T04:53:27.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.663+0000 7f2ce37fe700 1 -- 172.21.15.70:0/334370092 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f2cd4018020 con 0x7f2cec06c670 2024-06-04T04:53:27.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.663+0000 7f2ce37fe700 1 --2- 172.21.15.70:0/334370092 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2cd80b2900 0x7f2cd80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:27.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.663+0000 7f2cea59c700 1 --2- 172.21.15.70:0/334370092 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2cd80b2900 0x7f2cd80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:27.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.663+0000 7f2ce37fe700 1 -- 172.21.15.70:0/334370092 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f2cd40e1960 con 0x7f2cec06c670 2024-06-04T04:53:27.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.664+0000 7f2cea59c700 1 --2- 172.21.15.70:0/334370092 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2cd80b2900 0x7f2cd80b4dc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc004630 tx=0x7f2cdc00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:27.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.667+0000 7f2ce37fe700 1 -- 172.21.15.70:0/334370092 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2cd401f020 con 0x7f2cec06c670 2024-06-04T04:53:27.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.888+0000 7f2cf0d06700 1 -- 172.21.15.70:0/334370092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f2cec10b4c0 con 0x7f2cec06c670 2024-06-04T04:53:27.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.888+0000 7f2ce37fe700 1 -- 172.21.15.70:0/334370092 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) ==== 84+0+65 (secure 0 0 0) 0x7f2cd40a9b20 con 0x7f2cec06c670 2024-06-04T04:53:27.891 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:53:27.891 INFO:teuthology.orchestra.run.smithi070.stdout:[ 2024-06-04T04:53:27.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T04:53:27.892 INFO:teuthology.orchestra.run.smithi070.stdout: "poolnum": 1, 2024-06-04T04:53:27.892 INFO:teuthology.orchestra.run.smithi070.stdout: "poolname": ".mgr" 2024-06-04T04:53:27.892 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T04:53:27.892 INFO:teuthology.orchestra.run.smithi070.stdout:] 2024-06-04T04:53:27.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.890+0000 7f2cf0d06700 1 -- 172.21.15.70:0/334370092 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2cd80b2900 msgr2=0x7f2cd80b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:27.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.890+0000 7f2cf0d06700 1 --2- 172.21.15.70:0/334370092 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2cd80b2900 0x7f2cd80b4dc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc004630 tx=0x7f2cdc00b3c0 comp rx=0 tx=0).stop 2024-06-04T04:53:27.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.890+0000 7f2cf0d06700 1 -- 172.21.15.70:0/334370092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cec06c670 msgr2=0x7f2cec102a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:27.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.890+0000 7f2cf0d06700 1 --2- 172.21.15.70:0/334370092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cec06c670 0x7f2cec102a80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2cd400b530 tx=0x7f2cd400b840 comp rx=0 tx=0).stop 2024-06-04T04:53:27.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.890+0000 7f2cf0d06700 1 -- 172.21.15.70:0/334370092 shutdown_connections 2024-06-04T04:53:27.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.890+0000 7f2cf0d06700 1 --2- 172.21.15.70:0/334370092 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2cd80b2900 0x7f2cd80b4dc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:27.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.890+0000 7f2cf0d06700 1 --2- 172.21.15.70:0/334370092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cec1095e0 0x7f2cec102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:27.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.890+0000 7f2cf0d06700 1 --2- 172.21.15.70:0/334370092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cec06bcb0 0x7f2cec102540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:27.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.890+0000 7f2cf0d06700 1 --2- 172.21.15.70:0/334370092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cec06c670 0x7f2cec102a80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:27.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.890+0000 7f2cf0d06700 1 -- 172.21.15.70:0/334370092 >> 172.21.15.70:0/334370092 conn(0x7f2cec0784b0 msgr2=0x7f2cec101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:27.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.891+0000 7f2cf0d06700 1 -- 172.21.15.70:0/334370092 shutdown_connections 2024-06-04T04:53:27.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:27.891+0000 7f2cf0d06700 1 -- 172.21.15.70:0/334370092 wait complete. 2024-06-04T04:53:28.562 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-04T04:53:28.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-04T04:53:28.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:28 smithi070 ceph-mon[32631]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:28.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:28 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:28.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:28 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:28.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:28 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:28.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:28 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:28.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:28 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:53:28.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:28 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/334370092' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-06-04T04:53:28.861 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:28 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:28.861 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:28 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:53:28.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:28 smithi098 ceph-mon[35569]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:28.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:28 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:28.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:28 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:28.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:28 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:28.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:28 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:28.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:28 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:53:28.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:28 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/334370092' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-06-04T04:53:28.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:28 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:28.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:28 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:53:29.061 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:28 smithi112 ceph-mon[36453]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:29.061 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:28 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:29.062 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:28 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:29.062 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:28 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:29.062 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:28 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:29.062 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:28 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:53:29.062 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:28 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/334370092' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-06-04T04:53:29.062 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:28 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:29.062 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:28 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:53:29.485 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:53:30.930 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:30 smithi070 ceph-mon[32631]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:30.930 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:30 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:30.931 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:30 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:30.931 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:30 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:30.931 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:30 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:31.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.159+0000 7f0f887a9700 1 -- 172.21.15.70:0/2351757554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f80102e30 msgr2=0x7f0f8010f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:31.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.159+0000 7f0f887a9700 1 --2- 172.21.15.70:0/2351757554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f80102e30 0x7f0f8010f320 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0f7800b0d0 tx=0x7f0f7800b3e0 comp rx=0 tx=0).stop 2024-06-04T04:53:31.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.159+0000 7f0f887a9700 1 -- 172.21.15.70:0/2351757554 shutdown_connections 2024-06-04T04:53:31.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.159+0000 7f0f887a9700 1 --2- 172.21.15.70:0/2351757554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f80102e30 0x7f0f8010f320 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:31.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.159+0000 7f0f887a9700 1 --2- 172.21.15.70:0/2351757554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f80108550 0x7f0f80108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:31.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.159+0000 7f0f887a9700 1 --2- 172.21.15.70:0/2351757554 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f80102470 0x7f0f801028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:31.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.159+0000 7f0f887a9700 1 -- 172.21.15.70:0/2351757554 >> 172.21.15.70:0/2351757554 conn(0x7f0f80079480 msgr2=0x7f0f80079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:31.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.159+0000 7f0f887a9700 1 -- 172.21.15.70:0/2351757554 shutdown_connections 2024-06-04T04:53:31.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.159+0000 7f0f887a9700 1 -- 172.21.15.70:0/2351757554 wait complete. 2024-06-04T04:53:31.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.160+0000 7f0f887a9700 1 Processor -- start 2024-06-04T04:53:31.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.160+0000 7f0f887a9700 1 -- start start 2024-06-04T04:53:31.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.160+0000 7f0f887a9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f80102470 0x7f0f801040b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:31.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.160+0000 7f0f887a9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f80102e30 0x7f0f801045f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:31.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.160+0000 7f0f887a9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f80108550 0x7f0f80104b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:31.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.160+0000 7f0f887a9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0f801124f0 con 0x7f0f80108550 2024-06-04T04:53:31.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.160+0000 7f0f887a9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0f80112370 con 0x7f0f80102470 2024-06-04T04:53:31.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.160+0000 7f0f887a9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0f80112670 con 0x7f0f80102e30 2024-06-04T04:53:31.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.160+0000 7f0f86d46700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f80108550 0x7f0f80104b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:31.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.160+0000 7f0f86d46700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f80108550 0x7f0f80104b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50652/0 (socket says 172.21.15.70:50652) 2024-06-04T04:53:31.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.160+0000 7f0f86d46700 1 -- 172.21.15.70:0/3824971866 learned_addr learned my addr 172.21.15.70:0/3824971866 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:53:31.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.160+0000 7f0f86545700 1 --2- 172.21.15.70:0/3824971866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f80102470 0x7f0f801040b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:31.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.161+0000 7f0f85d44700 1 --2- 172.21.15.70:0/3824971866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f80102e30 0x7f0f801045f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:31.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.161+0000 7f0f86d46700 1 -- 172.21.15.70:0/3824971866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f80102e30 msgr2=0x7f0f801045f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:31.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.161+0000 7f0f86d46700 1 --2- 172.21.15.70:0/3824971866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f80102e30 0x7f0f801045f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:31.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.161+0000 7f0f86d46700 1 -- 172.21.15.70:0/3824971866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f80102470 msgr2=0x7f0f801040b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:31.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.161+0000 7f0f86d46700 1 --2- 172.21.15.70:0/3824971866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f80102470 0x7f0f801040b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:31.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.161+0000 7f0f86d46700 1 -- 172.21.15.70:0/3824971866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f801a7ff0 con 0x7f0f80108550 2024-06-04T04:53:31.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.161+0000 7f0f85d44700 1 --2- 172.21.15.70:0/3824971866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f80102e30 0x7f0f801045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:53:31.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.161+0000 7f0f86d46700 1 --2- 172.21.15.70:0/3824971866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f80108550 0x7f0f80104b30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0f78009850 tx=0x7f0f78007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:31.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.161+0000 7f0f777fe700 1 -- 172.21.15.70:0/3824971866 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7801f030 con 0x7f0f80108550 2024-06-04T04:53:31.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.161+0000 7f0f887a9700 1 -- 172.21.15.70:0/3824971866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f801a8280 con 0x7f0f80108550 2024-06-04T04:53:31.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.161+0000 7f0f777fe700 1 -- 172.21.15.70:0/3824971866 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0f7801bad0 con 0x7f0f80108550 2024-06-04T04:53:31.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.161+0000 7f0f777fe700 1 -- 172.21.15.70:0/3824971866 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f78012bb0 con 0x7f0f80108550 2024-06-04T04:53:31.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.161+0000 7f0f887a9700 1 -- 172.21.15.70:0/3824971866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f801a87c0 con 0x7f0f80108550 2024-06-04T04:53:31.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.162+0000 7f0f887a9700 1 -- 172.21.15.70:0/3824971866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f80103b90 con 0x7f0f80108550 2024-06-04T04:53:31.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.163+0000 7f0f777fe700 1 -- 172.21.15.70:0/3824971866 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f0f78022030 con 0x7f0f80108550 2024-06-04T04:53:31.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.164+0000 7f0f777fe700 1 --2- 172.21.15.70:0/3824971866 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0f6c0b2880 0x7f0f6c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:31.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.164+0000 7f0f86545700 1 --2- 172.21.15.70:0/3824971866 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0f6c0b2880 0x7f0f6c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:31.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.164+0000 7f0f777fe700 1 -- 172.21.15.70:0/3824971866 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f0f7800f020 con 0x7f0f80108550 2024-06-04T04:53:31.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.164+0000 7f0f86545700 1 --2- 172.21.15.70:0/3824971866 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0f6c0b2880 0x7f0f6c0b4d40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0f70005fd0 tx=0x7f0f7000af20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:31.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.168+0000 7f0f777fe700 1 -- 172.21.15.70:0/3824971866 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0f78029020 con 0x7f0f80108550 2024-06-04T04:53:31.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:30 smithi112 ceph-mon[36453]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:31.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:30 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:31.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:30 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:31.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:30 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:31.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:30 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:31.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:30 smithi098 ceph-mon[35569]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:31.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:30 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:31.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:30 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:31.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:30 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:31.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:30 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:31.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:31.393+0000 7f0f887a9700 1 -- 172.21.15.70:0/3824971866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f0f8002e780 con 0x7f0f80108550 2024-06-04T04:53:32.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:32.327+0000 7f0f777fe700 1 -- 172.21.15.70:0/3824971866 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) ==== 138+0+0 (secure 0 0 0) 0x7f0f780ab240 con 0x7f0f80108550 2024-06-04T04:53:32.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:32.330+0000 7f0f887a9700 1 -- 172.21.15.70:0/3824971866 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0f6c0b2880 msgr2=0x7f0f6c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:32.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:32.330+0000 7f0f887a9700 1 --2- 172.21.15.70:0/3824971866 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0f6c0b2880 0x7f0f6c0b4d40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0f70005fd0 tx=0x7f0f7000af20 comp rx=0 tx=0).stop 2024-06-04T04:53:32.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:32.330+0000 7f0f887a9700 1 -- 172.21.15.70:0/3824971866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f80108550 msgr2=0x7f0f80104b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:32.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:32.330+0000 7f0f887a9700 1 --2- 172.21.15.70:0/3824971866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f80108550 0x7f0f80104b30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0f78009850 tx=0x7f0f78007e40 comp rx=0 tx=0).stop 2024-06-04T04:53:32.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:32.330+0000 7f0f887a9700 1 -- 172.21.15.70:0/3824971866 shutdown_connections 2024-06-04T04:53:32.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:32.330+0000 7f0f887a9700 1 --2- 172.21.15.70:0/3824971866 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0f6c0b2880 0x7f0f6c0b4d40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:32.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:32.330+0000 7f0f887a9700 1 --2- 172.21.15.70:0/3824971866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f80108550 0x7f0f80104b30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:32.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:32.330+0000 7f0f887a9700 1 --2- 172.21.15.70:0/3824971866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f80102470 0x7f0f801040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:32.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:32.330+0000 7f0f887a9700 1 --2- 172.21.15.70:0/3824971866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f80102e30 0x7f0f801045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:32.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:32.330+0000 7f0f887a9700 1 -- 172.21.15.70:0/3824971866 >> 172.21.15.70:0/3824971866 conn(0x7f0f80079480 msgr2=0x7f0f8006bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:32.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:32.330+0000 7f0f887a9700 1 -- 172.21.15.70:0/3824971866 shutdown_connections 2024-06-04T04:53:32.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:32.331+0000 7f0f887a9700 1 -- 172.21.15.70:0/3824971866 wait complete. 2024-06-04T04:53:32.335 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' created 2024-06-04T04:53:32.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:32 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:32.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:32 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:32.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:32 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3824971866' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-06-04T04:53:32.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:32 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:32.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:32 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:53:32.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:32 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:32.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:32 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:32.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:32 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:32.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:32 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3824971866' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-06-04T04:53:32.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:32 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:32.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:32 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:53:32.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:32 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:32.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:32 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:32.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:32 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:32.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:32 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3824971866' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-06-04T04:53:32.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:32 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:53:32.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:32 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:53:32.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:32 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:53:32.997 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-04T04:53:33.235 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:53:33.606 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:33 smithi070 ceph-mon[32631]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:33.606 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:33 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3824971866' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-06-04T04:53:33.606 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:33 smithi070 ceph-mon[32631]: osdmap e73: 12 total, 12 up, 12 in 2024-06-04T04:53:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:33 smithi112 ceph-mon[36453]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:33 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3824971866' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-06-04T04:53:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:33 smithi112 ceph-mon[36453]: osdmap e73: 12 total, 12 up, 12 in 2024-06-04T04:53:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:33 smithi098 ceph-mon[35569]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:33 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3824971866' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-06-04T04:53:33.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:33 smithi098 ceph-mon[35569]: osdmap e73: 12 total, 12 up, 12 in 2024-06-04T04:53:34.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:34 smithi070 ceph-mon[32631]: osdmap e74: 12 total, 12 up, 12 in 2024-06-04T04:53:34.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:34 smithi112 ceph-mon[36453]: osdmap e74: 12 total, 12 up, 12 in 2024-06-04T04:53:34.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:34 smithi098 ceph-mon[35569]: osdmap e74: 12 total, 12 up, 12 in 2024-06-04T04:53:35.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.082+0000 7f14d2c9a700 1 -- 172.21.15.70:0/1291362539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14cc1028c0 msgr2=0x7f14cc110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:35.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.082+0000 7f14d2c9a700 1 --2- 172.21.15.70:0/1291362539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14cc1028c0 0x7f14cc110330 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f14c400b0a0 tx=0x7f14c400b3b0 comp rx=0 tx=0).stop 2024-06-04T04:53:35.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.082+0000 7f14d2c9a700 1 -- 172.21.15.70:0/1291362539 shutdown_connections 2024-06-04T04:53:35.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.082+0000 7f14d2c9a700 1 --2- 172.21.15.70:0/1291362539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14cc1028c0 0x7f14cc110330 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:35.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.082+0000 7f14d2c9a700 1 --2- 172.21.15.70:0/1291362539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14cc108620 0x7f14cc108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:35.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.082+0000 7f14d2c9a700 1 --2- 172.21.15.70:0/1291362539 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14cc101f00 0x7f14cc102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:35.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.082+0000 7f14d2c9a700 1 -- 172.21.15.70:0/1291362539 >> 172.21.15.70:0/1291362539 conn(0x7f14cc077780 msgr2=0x7f14cc077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:35.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.082+0000 7f14d2c9a700 1 -- 172.21.15.70:0/1291362539 shutdown_connections 2024-06-04T04:53:35.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.082+0000 7f14d2c9a700 1 -- 172.21.15.70:0/1291362539 wait complete. 2024-06-04T04:53:35.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.082+0000 7f14d2c9a700 1 Processor -- start 2024-06-04T04:53:35.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.082+0000 7f14d2c9a700 1 -- start start 2024-06-04T04:53:35.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d2c9a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14cc101f00 0x7f14cc19f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:35.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d2c9a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14cc1028c0 0x7f14cc19fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:35.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d2c9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14cc108620 0x7f14cc199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:35.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d2c9a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc1136e0 con 0x7f14cc101f00 2024-06-04T04:53:35.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d2c9a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc113560 con 0x7f14cc108620 2024-06-04T04:53:35.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d2c9a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc113860 con 0x7f14cc1028c0 2024-06-04T04:53:35.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d0a36700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14cc101f00 0x7f14cc19f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:35.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d0a36700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14cc101f00 0x7f14cc19f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50672/0 (socket says 172.21.15.70:50672) 2024-06-04T04:53:35.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d0a36700 1 -- 172.21.15.70:0/1804918177 learned_addr learned my addr 172.21.15.70:0/1804918177 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:53:35.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14cbfff700 1 --2- 172.21.15.70:0/1804918177 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14cc1028c0 0x7f14cc19fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:35.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d1237700 1 --2- 172.21.15.70:0/1804918177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14cc108620 0x7f14cc199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:35.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d0a36700 1 -- 172.21.15.70:0/1804918177 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14cc1028c0 msgr2=0x7f14cc19fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:35.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d0a36700 1 --2- 172.21.15.70:0/1804918177 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14cc1028c0 0x7f14cc19fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:35.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d0a36700 1 -- 172.21.15.70:0/1804918177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14cc108620 msgr2=0x7f14cc199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:35.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d0a36700 1 --2- 172.21.15.70:0/1804918177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14cc108620 0x7f14cc199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:35.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d0a36700 1 -- 172.21.15.70:0/1804918177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14cc19a1a0 con 0x7f14cc101f00 2024-06-04T04:53:35.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d1237700 1 --2- 172.21.15.70:0/1804918177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14cc108620 0x7f14cc199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:53:35.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d0a36700 1 --2- 172.21.15.70:0/1804918177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14cc101f00 0x7f14cc19f880 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f14c000b500 tx=0x7f14c000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:35.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14c9ffb700 1 -- 172.21.15.70:0/1804918177 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c0015020 con 0x7f14cc101f00 2024-06-04T04:53:35.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14c9ffb700 1 -- 172.21.15.70:0/1804918177 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f14c0011cc0 con 0x7f14cc101f00 2024-06-04T04:53:35.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14d2c9a700 1 -- 172.21.15.70:0/1804918177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14cc19a430 con 0x7f14cc101f00 2024-06-04T04:53:35.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.083+0000 7f14c9ffb700 1 -- 172.21.15.70:0/1804918177 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c0010ca0 con 0x7f14cc101f00 2024-06-04T04:53:35.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.084+0000 7f14d2c9a700 1 -- 172.21.15.70:0/1804918177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14cc1a6750 con 0x7f14cc101f00 2024-06-04T04:53:35.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.084+0000 7f14d2c9a700 1 -- 172.21.15.70:0/1804918177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14cc06a230 con 0x7f14cc101f00 2024-06-04T04:53:35.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.086+0000 7f14c9ffb700 1 -- 172.21.15.70:0/1804918177 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f14c0018020 con 0x7f14cc101f00 2024-06-04T04:53:35.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.086+0000 7f14c9ffb700 1 --2- 172.21.15.70:0/1804918177 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f14bc0b2880 0x7f14bc0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:35.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.087+0000 7f14cbfff700 1 --2- 172.21.15.70:0/1804918177 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f14bc0b2880 0x7f14bc0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:35.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.087+0000 7f14c9ffb700 1 -- 172.21.15.70:0/1804918177 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7497+0+0 (secure 0 0 0) 0x7f14c00e1c00 con 0x7f14cc101f00 2024-06-04T04:53:35.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.087+0000 7f14cbfff700 1 --2- 172.21.15.70:0/1804918177 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f14bc0b2880 0x7f14bc0b4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f14cc077d60 tx=0x7f14b8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:35.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.090+0000 7f14c9ffb700 1 -- 172.21.15.70:0/1804918177 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f14c001f020 con 0x7f14cc101f00 2024-06-04T04:53:35.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.293+0000 7f14d2c9a700 1 -- 172.21.15.70:0/1804918177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f14cc02e560 con 0x7f14cc101f00 2024-06-04T04:53:35.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.346+0000 7f14c9ffb700 1 -- 172.21.15.70:0/1804918177 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v76) ==== 170+0+0 (secure 0 0 0) 0x7f14c00a9c50 con 0x7f14cc101f00 2024-06-04T04:53:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.349+0000 7f14d2c9a700 1 -- 172.21.15.70:0/1804918177 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f14bc0b2880 msgr2=0x7f14bc0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.349+0000 7f14d2c9a700 1 --2- 172.21.15.70:0/1804918177 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f14bc0b2880 0x7f14bc0b4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f14cc077d60 tx=0x7f14b8009190 comp rx=0 tx=0).stop 2024-06-04T04:53:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.349+0000 7f14d2c9a700 1 -- 172.21.15.70:0/1804918177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14cc101f00 msgr2=0x7f14cc19f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.349+0000 7f14d2c9a700 1 --2- 172.21.15.70:0/1804918177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14cc101f00 0x7f14cc19f880 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f14c000b500 tx=0x7f14c000b810 comp rx=0 tx=0).stop 2024-06-04T04:53:35.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.350+0000 7f14d2c9a700 1 -- 172.21.15.70:0/1804918177 shutdown_connections 2024-06-04T04:53:35.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.350+0000 7f14d2c9a700 1 --2- 172.21.15.70:0/1804918177 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f14bc0b2880 0x7f14bc0b4d40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:35.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.350+0000 7f14d2c9a700 1 --2- 172.21.15.70:0/1804918177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14cc108620 0x7f14cc199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:35.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.350+0000 7f14d2c9a700 1 --2- 172.21.15.70:0/1804918177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14cc101f00 0x7f14cc19f880 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:35.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.350+0000 7f14d2c9a700 1 --2- 172.21.15.70:0/1804918177 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14cc1028c0 0x7f14cc19fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:35.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.350+0000 7f14d2c9a700 1 -- 172.21.15.70:0/1804918177 >> 172.21.15.70:0/1804918177 conn(0x7f14cc077780 msgr2=0x7f14cc100b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:35.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.350+0000 7f14d2c9a700 1 -- 172.21.15.70:0/1804918177 shutdown_connections 2024-06-04T04:53:35.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:35.350+0000 7f14d2c9a700 1 -- 172.21.15.70:0/1804918177 wait complete. 2024-06-04T04:53:35.355 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' created 2024-06-04T04:53:35.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:35 smithi070 ceph-mon[32631]: pgmap v282: 33 pgs: 2 creating+activating, 3 creating+peering, 5 active+clean, 23 unknown; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:35.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:35 smithi070 ceph-mon[32631]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-04T04:53:35.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:35 smithi070 ceph-mon[32631]: osdmap e75: 12 total, 12 up, 12 in 2024-06-04T04:53:35.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:35 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1804918177' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-06-04T04:53:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:35 smithi112 ceph-mon[36453]: pgmap v282: 33 pgs: 2 creating+activating, 3 creating+peering, 5 active+clean, 23 unknown; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:35 smithi112 ceph-mon[36453]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-04T04:53:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:35 smithi112 ceph-mon[36453]: osdmap e75: 12 total, 12 up, 12 in 2024-06-04T04:53:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:35 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1804918177' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-06-04T04:53:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:35 smithi098 ceph-mon[35569]: pgmap v282: 33 pgs: 2 creating+activating, 3 creating+peering, 5 active+clean, 23 unknown; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:35 smithi098 ceph-mon[35569]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-04T04:53:35.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:35 smithi098 ceph-mon[35569]: osdmap e75: 12 total, 12 up, 12 in 2024-06-04T04:53:35.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:35 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1804918177' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-06-04T04:53:35.980 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-04T04:53:36.218 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:53:36.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:36 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1804918177' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-06-04T04:53:36.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:36 smithi070 ceph-mon[32631]: osdmap e76: 12 total, 12 up, 12 in 2024-06-04T04:53:36.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:36 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1804918177' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-06-04T04:53:36.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:36 smithi112 ceph-mon[36453]: osdmap e76: 12 total, 12 up, 12 in 2024-06-04T04:53:36.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:36 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1804918177' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-06-04T04:53:36.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:36 smithi098 ceph-mon[35569]: osdmap e76: 12 total, 12 up, 12 in 2024-06-04T04:53:37.419 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:37 smithi070 ceph-mon[32631]: pgmap v285: 97 pgs: 2 creating+activating, 22 creating+peering, 22 active+clean, 51 unknown; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:37.419 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:37 smithi070 ceph-mon[32631]: osdmap e77: 12 total, 12 up, 12 in 2024-06-04T04:53:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:37 smithi112 ceph-mon[36453]: pgmap v285: 97 pgs: 2 creating+activating, 22 creating+peering, 22 active+clean, 51 unknown; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:37 smithi112 ceph-mon[36453]: osdmap e77: 12 total, 12 up, 12 in 2024-06-04T04:53:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:37 smithi098 ceph-mon[35569]: pgmap v285: 97 pgs: 2 creating+activating, 22 creating+peering, 22 active+clean, 51 unknown; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:37 smithi098 ceph-mon[35569]: osdmap e77: 12 total, 12 up, 12 in 2024-06-04T04:53:37.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.976+0000 7fdc0f5bd700 1 -- 172.21.15.70:0/3358435157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc08102440 msgr2=0x7fdc081028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:37.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.976+0000 7fdc0f5bd700 1 --2- 172.21.15.70:0/3358435157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc08102440 0x7fdc081028c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8009990 tx=0x7fdbf8009ca0 comp rx=0 tx=0).stop 2024-06-04T04:53:37.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.976+0000 7fdc0f5bd700 1 -- 172.21.15.70:0/3358435157 shutdown_connections 2024-06-04T04:53:37.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.976+0000 7fdc0f5bd700 1 --2- 172.21.15.70:0/3358435157 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc08102e00 0x7fdc0810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:37.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.976+0000 7fdc0f5bd700 1 --2- 172.21.15.70:0/3358435157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc08108520 0x7fdc08108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:37.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.976+0000 7fdc0f5bd700 1 --2- 172.21.15.70:0/3358435157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc08102440 0x7fdc081028c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:37.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.976+0000 7fdc0f5bd700 1 -- 172.21.15.70:0/3358435157 >> 172.21.15.70:0/3358435157 conn(0x7fdc080793f0 msgr2=0x7fdc08079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:37.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.976+0000 7fdc0f5bd700 1 -- 172.21.15.70:0/3358435157 shutdown_connections 2024-06-04T04:53:37.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.976+0000 7fdc0f5bd700 1 -- 172.21.15.70:0/3358435157 wait complete. 2024-06-04T04:53:37.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.976+0000 7fdc0f5bd700 1 Processor -- start 2024-06-04T04:53:37.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0f5bd700 1 -- start start 2024-06-04T04:53:37.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0f5bd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc08102440 0x7fdc0819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:37.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0f5bd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc08102e00 0x7fdc0819ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:37.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0f5bd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc08108520 0x7fdc08198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:37.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0f5bd700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc08112500 con 0x7fdc08102e00 2024-06-04T04:53:37.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0f5bd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc08112380 con 0x7fdc08108520 2024-06-04T04:53:37.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0f5bd700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdc08112680 con 0x7fdc08102440 2024-06-04T04:53:37.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0cb58700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc08102e00 0x7fdc0819ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:37.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0cb58700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc08102e00 0x7fdc0819ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40434/0 (socket says 172.21.15.70:40434) 2024-06-04T04:53:37.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0cb58700 1 -- 172.21.15.70:0/1785137495 learned_addr learned my addr 172.21.15.70:0/1785137495 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:53:37.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0db5a700 1 --2- 172.21.15.70:0/1785137495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc08108520 0x7fdc08198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:37.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0d359700 1 --2- 172.21.15.70:0/1785137495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc08102440 0x7fdc0819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:37.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0cb58700 1 -- 172.21.15.70:0/1785137495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc08102440 msgr2=0x7fdc0819e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:37.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0cb58700 1 --2- 172.21.15.70:0/1785137495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc08102440 0x7fdc0819e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:37.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0cb58700 1 -- 172.21.15.70:0/1785137495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc08108520 msgr2=0x7fdc08198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:37.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0cb58700 1 --2- 172.21.15.70:0/1785137495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc08108520 0x7fdc08198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:37.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0cb58700 1 -- 172.21.15.70:0/1785137495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc08199060 con 0x7fdc08102e00 2024-06-04T04:53:37.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0db5a700 1 --2- 172.21.15.70:0/1785137495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc08108520 0x7fdc08198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:53:37.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0d359700 1 --2- 172.21.15.70:0/1785137495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc08102440 0x7fdc0819e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:53:37.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0cb58700 1 --2- 172.21.15.70:0/1785137495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc08102e00 0x7fdc0819ebf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8000c00 tx=0x7fdbf80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:37.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdbfe7fc700 1 -- 172.21.15.70:0/1785137495 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf802a070 con 0x7fdc08102e00 2024-06-04T04:53:37.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdbfe7fc700 1 -- 172.21.15.70:0/1785137495 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdbf801ad50 con 0x7fdc08102e00 2024-06-04T04:53:37.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdc0f5bd700 1 -- 172.21.15.70:0/1785137495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc081992f0 con 0x7fdc08102e00 2024-06-04T04:53:37.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.977+0000 7fdbfe7fc700 1 -- 172.21.15.70:0/1785137495 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf800fe60 con 0x7fdc08102e00 2024-06-04T04:53:37.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.978+0000 7fdc0f5bd700 1 -- 172.21.15.70:0/1785137495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc0806b220 con 0x7fdc08102e00 2024-06-04T04:53:37.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.978+0000 7fdc0f5bd700 1 -- 172.21.15.70:0/1785137495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc08103b60 con 0x7fdc08102e00 2024-06-04T04:53:37.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.980+0000 7fdbfe7fc700 1 -- 172.21.15.70:0/1785137495 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fdbf802d070 con 0x7fdc08102e00 2024-06-04T04:53:37.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.981+0000 7fdbfe7fc700 1 --2- 172.21.15.70:0/1785137495 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fdbf40b2880 0x7fdbf40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:37.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.981+0000 7fdc0d359700 1 --2- 172.21.15.70:0/1785137495 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fdbf40b2880 0x7fdbf40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:37.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.981+0000 7fdbfe7fc700 1 -- 172.21.15.70:0/1785137495 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7879+0+0 (secure 0 0 0) 0x7fdbf80e56c0 con 0x7fdc08102e00 2024-06-04T04:53:37.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.981+0000 7fdc0d359700 1 --2- 172.21.15.70:0/1785137495 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fdbf40b2880 0x7fdbf40b4d40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdc04005fd0 tx=0x7fdc04005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:37.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:37.985+0000 7fdbfe7fc700 1 -- 172.21.15.70:0/1785137495 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdbf8034070 con 0x7fdc08102e00 2024-06-04T04:53:38.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:38.235+0000 7fdc0f5bd700 1 -- 172.21.15.70:0/1785137495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fdc0806bc50 con 0x7fdc08102e00 2024-06-04T04:53:38.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:38.364+0000 7fdbfe7fc700 1 -- 172.21.15.70:0/1785137495 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fdbf8013070 con 0x7fdc08102e00 2024-06-04T04:53:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:38.367+0000 7fdc0f5bd700 1 -- 172.21.15.70:0/1785137495 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fdbf40b2880 msgr2=0x7fdbf40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:38.367+0000 7fdc0f5bd700 1 --2- 172.21.15.70:0/1785137495 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fdbf40b2880 0x7fdbf40b4d40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdc04005fd0 tx=0x7fdc04005e30 comp rx=0 tx=0).stop 2024-06-04T04:53:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:38.367+0000 7fdc0f5bd700 1 -- 172.21.15.70:0/1785137495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc08102e00 msgr2=0x7fdc0819ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:38.367+0000 7fdc0f5bd700 1 --2- 172.21.15.70:0/1785137495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc08102e00 0x7fdc0819ebf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8000c00 tx=0x7fdbf80047b0 comp rx=0 tx=0).stop 2024-06-04T04:53:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:38.367+0000 7fdc0f5bd700 1 -- 172.21.15.70:0/1785137495 shutdown_connections 2024-06-04T04:53:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:38.367+0000 7fdc0f5bd700 1 --2- 172.21.15.70:0/1785137495 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fdbf40b2880 0x7fdbf40b4d40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:38.367+0000 7fdc0f5bd700 1 --2- 172.21.15.70:0/1785137495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc08108520 0x7fdc08198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:38.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:38.367+0000 7fdc0f5bd700 1 --2- 172.21.15.70:0/1785137495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc08102440 0x7fdc0819e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:38.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:38.367+0000 7fdc0f5bd700 1 --2- 172.21.15.70:0/1785137495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc08102e00 0x7fdc0819ebf0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:38.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:38.367+0000 7fdc0f5bd700 1 -- 172.21.15.70:0/1785137495 >> 172.21.15.70:0/1785137495 conn(0x7fdc080793f0 msgr2=0x7fdc08100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:38.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:38.367+0000 7fdc0f5bd700 1 -- 172.21.15.70:0/1785137495 shutdown_connections 2024-06-04T04:53:38.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:38.367+0000 7fdc0f5bd700 1 -- 172.21.15.70:0/1785137495 wait complete. 2024-06-04T04:53:38.371 INFO:teuthology.orchestra.run.smithi070.stderr:new fs with metadata pool 2 and data pool 3 2024-06-04T04:53:38.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:38 smithi070 ceph-mon[32631]: osdmap e78: 12 total, 12 up, 12 in 2024-06-04T04:53:38.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:38 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1785137495' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-06-04T04:53:38.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:38 smithi070 ceph-mon[32631]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-06-04T04:53:38.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:38 smithi112 ceph-mon[36453]: osdmap e78: 12 total, 12 up, 12 in 2024-06-04T04:53:38.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:38 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1785137495' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-06-04T04:53:38.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:38 smithi112 ceph-mon[36453]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-06-04T04:53:38.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:38 smithi098 ceph-mon[35569]: osdmap e78: 12 total, 12 up, 12 in 2024-06-04T04:53:38.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:38 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1785137495' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-06-04T04:53:38.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:38 smithi098 ceph-mon[35569]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-06-04T04:53:39.057 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-04T04:53:39.297 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:53:39.681 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:39 smithi070 ceph-mon[32631]: pgmap v288: 97 pgs: 2 creating+activating, 19 creating+peering, 47 active+clean, 29 unknown; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:39.681 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:39 smithi070 ceph-mon[32631]: osdmap e79: 12 total, 12 up, 12 in 2024-06-04T04:53:39.681 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:39 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1785137495' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-06-04T04:53:39.682 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:39 smithi070 ceph-mon[32631]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-06-04T04:53:39.682 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:39 smithi070 ceph-mon[32631]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-06-04T04:53:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:39 smithi112 ceph-mon[36453]: pgmap v288: 97 pgs: 2 creating+activating, 19 creating+peering, 47 active+clean, 29 unknown; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:39 smithi112 ceph-mon[36453]: osdmap e79: 12 total, 12 up, 12 in 2024-06-04T04:53:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:39 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1785137495' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-06-04T04:53:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:39 smithi112 ceph-mon[36453]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-06-04T04:53:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:39 smithi112 ceph-mon[36453]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-06-04T04:53:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:39 smithi098 ceph-mon[35569]: pgmap v288: 97 pgs: 2 creating+activating, 19 creating+peering, 47 active+clean, 29 unknown; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:53:39.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:39 smithi098 ceph-mon[35569]: osdmap e79: 12 total, 12 up, 12 in 2024-06-04T04:53:39.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:39 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1785137495' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-06-04T04:53:39.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:39 smithi098 ceph-mon[35569]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-06-04T04:53:39.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:39 smithi098 ceph-mon[35569]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-06-04T04:53:40.673 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:40 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] up:active 2024-06-04T04:53:40.673 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:40 smithi070 ceph-mon[32631]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-04T04:53:40.673 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:40 smithi070 ceph-mon[32631]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-04T04:53:40.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:40 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] up:active 2024-06-04T04:53:40.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:40 smithi112 ceph-mon[36453]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-04T04:53:40.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:40 smithi112 ceph-mon[36453]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-04T04:53:40.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:40 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] up:active 2024-06-04T04:53:40.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:40 smithi098 ceph-mon[35569]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-04T04:53:40.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:40 smithi098 ceph-mon[35569]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-04T04:53:40.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.926+0000 7efc877ba700 1 -- 172.21.15.70:0/514805757 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc801084c0 msgr2=0x7efc801088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:40.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.926+0000 7efc877ba700 1 --2- 172.21.15.70:0/514805757 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc801084c0 0x7efc801088a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efc70009990 tx=0x7efc70009ca0 comp rx=0 tx=0).stop 2024-06-04T04:53:40.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.926+0000 7efc877ba700 1 -- 172.21.15.70:0/514805757 shutdown_connections 2024-06-04T04:53:40.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.926+0000 7efc877ba700 1 --2- 172.21.15.70:0/514805757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc80103600 0x7efc8010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:40.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.926+0000 7efc877ba700 1 --2- 172.21.15.70:0/514805757 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc801084c0 0x7efc801088a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:40.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.926+0000 7efc877ba700 1 --2- 172.21.15.70:0/514805757 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc80102c40 0x7efc801030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:40.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.926+0000 7efc877ba700 1 -- 172.21.15.70:0/514805757 >> 172.21.15.70:0/514805757 conn(0x7efc80078ab0 msgr2=0x7efc800ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:40.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.926+0000 7efc877ba700 1 -- 172.21.15.70:0/514805757 shutdown_connections 2024-06-04T04:53:40.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.926+0000 7efc877ba700 1 -- 172.21.15.70:0/514805757 wait complete. 2024-06-04T04:53:40.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.927+0000 7efc877ba700 1 Processor -- start 2024-06-04T04:53:40.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.927+0000 7efc877ba700 1 -- start start 2024-06-04T04:53:40.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.927+0000 7efc877ba700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc80102c40 0x7efc8019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.927+0000 7efc877ba700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc80103600 0x7efc8019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.927+0000 7efc877ba700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc801084c0 0x7efc80198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.928+0000 7efc877ba700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efc801124a0 con 0x7efc80102c40 2024-06-04T04:53:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.928+0000 7efc877ba700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efc80112320 con 0x7efc80103600 2024-06-04T04:53:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.928+0000 7efc877ba700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efc80112620 con 0x7efc801084c0 2024-06-04T04:53:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.928+0000 7efc85556700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc80102c40 0x7efc8019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.928+0000 7efc85556700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc80102c40 0x7efc8019e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40446/0 (socket says 172.21.15.70:40446) 2024-06-04T04:53:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.928+0000 7efc85556700 1 -- 172.21.15.70:0/2855701772 learned_addr learned my addr 172.21.15.70:0/2855701772 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:53:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.928+0000 7efc84d55700 1 --2- 172.21.15.70:0/2855701772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc80103600 0x7efc8019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.928+0000 7efc85d57700 1 --2- 172.21.15.70:0/2855701772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc801084c0 0x7efc80198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.928+0000 7efc85556700 1 -- 172.21.15.70:0/2855701772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc801084c0 msgr2=0x7efc80198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.928+0000 7efc85556700 1 --2- 172.21.15.70:0/2855701772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc801084c0 0x7efc80198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.928+0000 7efc85556700 1 -- 172.21.15.70:0/2855701772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc80103600 msgr2=0x7efc8019ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:40.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.928+0000 7efc85556700 1 --2- 172.21.15.70:0/2855701772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc80103600 0x7efc8019ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:40.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.928+0000 7efc85556700 1 -- 172.21.15.70:0/2855701772 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc80198fc0 con 0x7efc80102c40 2024-06-04T04:53:40.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.928+0000 7efc85556700 1 --2- 172.21.15.70:0/2855701772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc80102c40 0x7efc8019e6a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efc70000c00 tx=0x7efc700047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:40.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.928+0000 7efc767fc700 1 -- 172.21.15.70:0/2855701772 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc70027070 con 0x7efc80102c40 2024-06-04T04:53:40.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.928+0000 7efc877ba700 1 -- 172.21.15.70:0/2855701772 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc80199250 con 0x7efc80102c40 2024-06-04T04:53:40.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.928+0000 7efc877ba700 1 -- 172.21.15.70:0/2855701772 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc801a5570 con 0x7efc80102c40 2024-06-04T04:53:40.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.929+0000 7efc767fc700 1 -- 172.21.15.70:0/2855701772 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc7000fb00 con 0x7efc80102c40 2024-06-04T04:53:40.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.929+0000 7efc767fc700 1 -- 172.21.15.70:0/2855701772 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc70022c40 con 0x7efc80102c40 2024-06-04T04:53:40.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.930+0000 7efc877ba700 1 -- 172.21.15.70:0/2855701772 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc64005330 con 0x7efc80102c40 2024-06-04T04:53:40.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.931+0000 7efc767fc700 1 -- 172.21.15.70:0/2855701772 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7efc7002a070 con 0x7efc80102c40 2024-06-04T04:53:40.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.931+0000 7efc767fc700 1 --2- 172.21.15.70:0/2855701772 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7efc6c0b2900 0x7efc6c0b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:40.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.932+0000 7efc84d55700 1 --2- 172.21.15.70:0/2855701772 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7efc6c0b2900 0x7efc6c0b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:40.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.932+0000 7efc767fc700 1 -- 172.21.15.70:0/2855701772 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7efc700e2ce0 con 0x7efc80102c40 2024-06-04T04:53:40.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.932+0000 7efc84d55700 1 --2- 172.21.15.70:0/2855701772 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7efc6c0b2900 0x7efc6c0b4dc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efc80107c00 tx=0x7efc7c00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:40.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:40.936+0000 7efc767fc700 1 -- 172.21.15.70:0/2855701772 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7efc7005fdf0 con 0x7efc80102c40 2024-06-04T04:53:41.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:41.149+0000 7efc877ba700 1 -- 172.21.15.70:0/2855701772 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc640058f0 con 0x7efc80102c40 2024-06-04T04:53:41.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:41.151+0000 7efc767fc700 1 -- 172.21.15.70:0/2855701772 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22183 (secure 0 0 0) 0x7efc70013070 con 0x7efc80102c40 2024-06-04T04:53:41.153 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:53:41.153 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":79,"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","created":"2024-06-04T04:45:22.297536+0000","modified":"2024-06-04T04:53:38.238030+0000","last_up_change":"2024-06-04T04:52:34.388708+0000","last_in_change":"2024-06-04T04:52:14.197780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-04T04:48:43.724356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-04T04:53:31.398455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-04T04:53:35.298617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ff491ea2-1906-48e4-883e-bc78e778260e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6803","nonce":824119100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6805","nonce":824119100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6809","nonce":824119100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6807","nonce":824119100}]},"public_addr":"172.21.15.70:6803/824119100","cluster_addr":"172.21.15.70:6805/824119100","heartbeat_back_addr":"172.21.15.70:6809/824119100","heartbeat_front_addr":"172.21.15.70:6807/824119100","state":["exists","up"]},{"osd":1,"uuid":"bd5e4806-535b-49b5-ba6b-93228da35a84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6801","nonce":2744598742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6803","nonce":2744598742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6807","nonce":2744598742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6805","nonce":2744598742}]},"public_addr":"172.21.15.98:6801/2744598742","cluster_addr":"172.21.15.98:6803/2744598742","heartbeat_back_addr":"172.21.15.98:6807/2744598742","heartbeat_front_addr":"172.21.15.98:6805/2744598742","state":["exists","up"]},{"osd":2,"uuid":"8456fb55-910a-4a59-b84b-2c0f68346ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6801","nonce":1328098370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6803","nonce":1328098370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6807","nonce":1328098370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6805","nonce":1328098370}]},"public_addr":"172.21.15.112:6801/1328098370","cluster_addr":"172.21.15.112:6803/1328098370","heartbeat_back_addr":"172.21.15.112:6807/1328098370","heartbeat_front_addr":"172.21.15.112:6805/1328098370","state":["exists","up"]},{"osd":3,"uuid":"c8fbc440-bf70-42d3-b01e-777ccfa7900b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6811","nonce":4041566346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6813","nonce":4041566346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6817","nonce":4041566346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6815","nonce":4041566346}]},"public_addr":"172.21.15.70:6811/4041566346","cluster_addr":"172.21.15.70:6813/4041566346","heartbeat_back_addr":"172.21.15.70:6817/4041566346","heartbeat_front_addr":"172.21.15.70:6815/4041566346","state":["exists","up"]},{"osd":4,"uuid":"549c8c70-8be8-46ae-bcb6-0fde32b40cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6809","nonce":3652775814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6811","nonce":3652775814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6815","nonce":3652775814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6813","nonce":3652775814}]},"public_addr":"172.21.15.98:6809/3652775814","cluster_addr":"172.21.15.98:6811/3652775814","heartbeat_back_addr":"172.21.15.98:6815/3652775814","heartbeat_front_addr":"172.21.15.98:6813/3652775814","state":["exists","up"]},{"osd":5,"uuid":"51d86af1-0d6c-4ae0-b090-00414fc1bf7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6809","nonce":1857257899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6811","nonce":1857257899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6815","nonce":1857257899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6813","nonce":1857257899}]},"public_addr":"172.21.15.112:6809/1857257899","cluster_addr":"172.21.15.112:6811/1857257899","heartbeat_back_addr":"172.21.15.112:6815/1857257899","heartbeat_front_addr":"172.21.15.112:6813/1857257899","state":["exists","up"]},{"osd":6,"uuid":"526a8138-bfa8-419e-a5b5-e6b80f0ae004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6819","nonce":1264676496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6821","nonce":1264676496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6825","nonce":1264676496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6823","nonce":1264676496}]},"public_addr":"172.21.15.70:6819/1264676496","cluster_addr":"172.21.15.70:6821/1264676496","heartbeat_back_addr":"172.21.15.70:6825/1264676496","heartbeat_front_addr":"172.21.15.70:6823/1264676496","state":["exists","up"]},{"osd":7,"uuid":"1173f785-6ed1-47ef-9265-2e233ea9ee02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6817","nonce":709593020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6819","nonce":709593020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6823","nonce":709593020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6821","nonce":709593020}]},"public_addr":"172.21.15.98:6817/709593020","cluster_addr":"172.21.15.98:6819/709593020","heartbeat_back_addr":"172.21.15.98:6823/709593020","heartbeat_front_addr":"172.21.15.98:6821/709593020","state":["exists","up"]},{"osd":8,"uuid":"6424cc6e-24a7-4b81-9ce6-39203fecc7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6817","nonce":2250225413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6819","nonce":2250225413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6823","nonce":2250225413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6821","nonce":2250225413}]},"public_addr":"172.21.15.112:6817/2250225413","cluster_addr":"172.21.15.112:6819/2250225413","heartbeat_back_addr":"172.21.15.112:6823/2250225413","heartbeat_front_addr":"172.21.15.112:6821/2250225413","state":["exists","up"]},{"osd":9,"uuid":"8d666db8-8831-4d63-9e3c-a4bf2504149e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6827","nonce":3557688707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6829","nonce":3557688707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6833","nonce":3557688707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6831","nonce":3557688707}]},"public_addr":"172.21.15.70:6827/3557688707","cluster_addr":"172.21.15.70:6829/3557688707","heartbeat_back_addr":"172.21.15.70:6833/3557688707","heartbeat_front_addr":"172.21.15.70:6831/3557688707","state":["exists","up"]},{"osd":10,"uuid":"ecb925d2-0263-4dfc-a2fd-9e27856022d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6825","nonce":1896493718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6827","nonce":1896493718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6831","nonce":1896493718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6829","nonce":1896493718}]},"public_addr":"172.21.15.98:6825/1896493718","cluster_addr":"172.21.15.98:6827/1896493718","heartbeat_back_addr":"172.21.15.98:6831/1896493718","heartbeat_front_addr":"172.21.15.98:6829/1896493718","state":["exists","up"]},{"osd":11,"uuid":"c8e3230c-bc3a-40bd-8dd0-5fef56e4df97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6825","nonce":1400793137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6827","nonce":1400793137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6831","nonce":1400793137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6829","nonce":1400793137}]},"public_addr":"172.21.15.112:6825/1400793137","cluster_addr":"172.21.15.112:6827/1400793137","heartbeat_back_addr":"172.21.15.112:6831/1400793137","heartbeat_front_addr":"172.21.15.112:6829/1400793137","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:47:53.626608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:17.784505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:41.971969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:07.130699+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:31.615496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:56.898607+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:22.739991+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:48.319727+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:13.902063+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:40.389470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:06.142092+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:32.949383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/3442074771":"2024-06-05T04:46:17.636907+0000","172.21.15.70:6801/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/1442095493":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/2453733268":"2024-06-05T04:45:49.501864+0000","172.21.15.70:0/1369781447":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6801/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/915186456":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/860033006":"2024-06-05T04:45:49.501864+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-04T04:53:41.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:41.153+0000 7efc877ba700 1 -- 172.21.15.70:0/2855701772 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7efc6c0b2900 msgr2=0x7efc6c0b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:41.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:41.153+0000 7efc877ba700 1 --2- 172.21.15.70:0/2855701772 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7efc6c0b2900 0x7efc6c0b4dc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efc80107c00 tx=0x7efc7c00b3c0 comp rx=0 tx=0).stop 2024-06-04T04:53:41.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:41.153+0000 7efc877ba700 1 -- 172.21.15.70:0/2855701772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc80102c40 msgr2=0x7efc8019e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:41.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:41.153+0000 7efc877ba700 1 --2- 172.21.15.70:0/2855701772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc80102c40 0x7efc8019e6a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efc70000c00 tx=0x7efc700047b0 comp rx=0 tx=0).stop 2024-06-04T04:53:41.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:41.153+0000 7efc877ba700 1 -- 172.21.15.70:0/2855701772 shutdown_connections 2024-06-04T04:53:41.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:41.153+0000 7efc877ba700 1 --2- 172.21.15.70:0/2855701772 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7efc6c0b2900 0x7efc6c0b4dc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:41.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:41.153+0000 7efc877ba700 1 --2- 172.21.15.70:0/2855701772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc801084c0 0x7efc80198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:41.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:41.153+0000 7efc877ba700 1 --2- 172.21.15.70:0/2855701772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc80102c40 0x7efc8019e6a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:41.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:41.153+0000 7efc877ba700 1 --2- 172.21.15.70:0/2855701772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc80103600 0x7efc8019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:41.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:41.153+0000 7efc877ba700 1 -- 172.21.15.70:0/2855701772 >> 172.21.15.70:0/2855701772 conn(0x7efc80078ab0 msgr2=0x7efc801002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:41.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:41.153+0000 7efc877ba700 1 -- 172.21.15.70:0/2855701772 shutdown_connections 2024-06-04T04:53:41.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:41.153+0000 7efc877ba700 1 -- 172.21.15.70:0/2855701772 wait complete. 2024-06-04T04:53:41.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:41 smithi098 ceph-mon[35569]: pgmap v290: 97 pgs: 19 creating+peering, 78 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 2 op/s 2024-06-04T04:53:41.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:41 smithi098 ceph-mon[35569]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-06-04T04:53:41.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:41 smithi098 ceph-mon[35569]: Cluster is now healthy 2024-06-04T04:53:41.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:41 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2855701772' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:53:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:41 smithi112 ceph-mon[36453]: pgmap v290: 97 pgs: 19 creating+peering, 78 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 2 op/s 2024-06-04T04:53:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:41 smithi112 ceph-mon[36453]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-06-04T04:53:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:41 smithi112 ceph-mon[36453]: Cluster is now healthy 2024-06-04T04:53:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:41 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2855701772' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:53:41.853 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:41 smithi070 ceph-mon[32631]: pgmap v290: 97 pgs: 19 creating+peering, 78 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 2 op/s 2024-06-04T04:53:41.853 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:41 smithi070 ceph-mon[32631]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-06-04T04:53:41.853 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:41 smithi070 ceph-mon[32631]: Cluster is now healthy 2024-06-04T04:53:41.853 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:41 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2855701772' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:53:41.880 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-04T04:53:42.124 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:53:43.378 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 04:53:43 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T04:53:43.116+0000 7f03b194b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-04T04:53:43.637 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:43 smithi070 ceph-mon[32631]: pgmap v291: 97 pgs: 97 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-06-04T04:53:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:43 smithi112 ceph-mon[36453]: pgmap v291: 97 pgs: 97 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-06-04T04:53:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:43 smithi098 ceph-mon[35569]: pgmap v291: 97 pgs: 97 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-06-04T04:53:43.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.989+0000 7f4561367700 1 -- 172.21.15.70:0/581725557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f455c102a80 msgr2=0x7f455c102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:43.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.989+0000 7f4561367700 1 --2- 172.21.15.70:0/581725557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f455c102a80 0x7f455c102e60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f45500099c0 tx=0x7f4550009cd0 comp rx=0 tx=0).stop 2024-06-04T04:53:43.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.990+0000 7f4561367700 1 -- 172.21.15.70:0/581725557 shutdown_connections 2024-06-04T04:53:43.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.990+0000 7f4561367700 1 --2- 172.21.15.70:0/581725557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f455c1077d0 0x7f455c10f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:43.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.990+0000 7f4561367700 1 --2- 172.21.15.70:0/581725557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f455c102a80 0x7f455c102e60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:43.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.990+0000 7f4561367700 1 --2- 172.21.15.70:0/581725557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f455c103430 0x7f455c1070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:43.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.990+0000 7f4561367700 1 -- 172.21.15.70:0/581725557 >> 172.21.15.70:0/581725557 conn(0x7f455c0793f0 msgr2=0x7f455c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:43.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.990+0000 7f4561367700 1 -- 172.21.15.70:0/581725557 shutdown_connections 2024-06-04T04:53:43.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.990+0000 7f4561367700 1 -- 172.21.15.70:0/581725557 wait complete. 2024-06-04T04:53:43.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.990+0000 7f4561367700 1 Processor -- start 2024-06-04T04:53:43.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.990+0000 7f4561367700 1 -- start start 2024-06-04T04:53:43.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f4561367700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f455c102a80 0x7f455c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:43.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f4561367700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f455c103430 0x7f455c19fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:43.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f4561367700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f455c1077d0 0x7f455c1999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:43.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f4561367700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f455c112500 con 0x7f455c102a80 2024-06-04T04:53:43.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f4561367700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f455c112380 con 0x7f455c103430 2024-06-04T04:53:43.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f4561367700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f455c112680 con 0x7f455c1077d0 2024-06-04T04:53:43.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f455affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f455c102a80 0x7f455c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:43.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f455affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f455c102a80 0x7f455c19f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40456/0 (socket says 172.21.15.70:40456) 2024-06-04T04:53:43.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f455affd700 1 -- 172.21.15.70:0/2893939645 learned_addr learned my addr 172.21.15.70:0/2893939645 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:53:43.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f455b7fe700 1 --2- 172.21.15.70:0/2893939645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f455c1077d0 0x7f455c1999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:43.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f455a7fc700 1 --2- 172.21.15.70:0/2893939645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f455c103430 0x7f455c19fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:43.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f455affd700 1 -- 172.21.15.70:0/2893939645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f455c1077d0 msgr2=0x7f455c1999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:43.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f455affd700 1 --2- 172.21.15.70:0/2893939645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f455c1077d0 0x7f455c1999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:43.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f455affd700 1 -- 172.21.15.70:0/2893939645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f455c103430 msgr2=0x7f455c19fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:43.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f455affd700 1 --2- 172.21.15.70:0/2893939645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f455c103430 0x7f455c19fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:43.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f455affd700 1 -- 172.21.15.70:0/2893939645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f455c19a120 con 0x7f455c102a80 2024-06-04T04:53:43.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f455a7fc700 1 --2- 172.21.15.70:0/2893939645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f455c103430 0x7f455c19fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:53:43.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f455affd700 1 --2- 172.21.15.70:0/2893939645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f455c102a80 0x7f455c19f7b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4550000c00 tx=0x7f45500047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:43.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f4543fff700 1 -- 172.21.15.70:0/2893939645 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f455000f8a0 con 0x7f455c102a80 2024-06-04T04:53:43.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f4543fff700 1 -- 172.21.15.70:0/2893939645 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4550004830 con 0x7f455c102a80 2024-06-04T04:53:43.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f4561367700 1 -- 172.21.15.70:0/2893939645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f455c19a3b0 con 0x7f455c102a80 2024-06-04T04:53:43.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f4543fff700 1 -- 172.21.15.70:0/2893939645 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4550005410 con 0x7f455c102a80 2024-06-04T04:53:43.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.991+0000 7f4561367700 1 -- 172.21.15.70:0/2893939645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f455c1a6680 con 0x7f455c102a80 2024-06-04T04:53:43.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.992+0000 7f4561367700 1 -- 172.21.15.70:0/2893939645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f455c06b4c0 con 0x7f455c102a80 2024-06-04T04:53:43.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.994+0000 7f4543fff700 1 -- 172.21.15.70:0/2893939645 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f4550029060 con 0x7f455c102a80 2024-06-04T04:53:43.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.995+0000 7f4543fff700 1 --2- 172.21.15.70:0/2893939645 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f45440b2880 0x7f45440b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:43.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.995+0000 7f455a7fc700 1 --2- 172.21.15.70:0/2893939645 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f45440b2880 0x7f45440b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:43.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.995+0000 7f4543fff700 1 -- 172.21.15.70:0/2893939645 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f45500e2e80 con 0x7f455c102a80 2024-06-04T04:53:43.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.995+0000 7f455a7fc700 1 --2- 172.21.15.70:0/2893939645 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f45440b2880 0x7f45440b4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f455c06ccf0 tx=0x7f454c00b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:44.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:43.998+0000 7f4543fff700 1 -- 172.21.15.70:0/2893939645 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4550013070 con 0x7f455c102a80 2024-06-04T04:53:44.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:44.204+0000 7f4561367700 1 -- 172.21.15.70:0/2893939645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f455c1a6850 con 0x7f455c102a80 2024-06-04T04:53:44.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:44.206+0000 7f4543fff700 1 -- 172.21.15.70:0/2893939645 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22183 (secure 0 0 0) 0x7f45500aace0 con 0x7f455c102a80 2024-06-04T04:53:44.208 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:53:44.208 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":79,"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","created":"2024-06-04T04:45:22.297536+0000","modified":"2024-06-04T04:53:38.238030+0000","last_up_change":"2024-06-04T04:52:34.388708+0000","last_in_change":"2024-06-04T04:52:14.197780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-04T04:48:43.724356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-04T04:53:31.398455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-04T04:53:35.298617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ff491ea2-1906-48e4-883e-bc78e778260e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6803","nonce":824119100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6805","nonce":824119100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6809","nonce":824119100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6807","nonce":824119100}]},"public_addr":"172.21.15.70:6803/824119100","cluster_addr":"172.21.15.70:6805/824119100","heartbeat_back_addr":"172.21.15.70:6809/824119100","heartbeat_front_addr":"172.21.15.70:6807/824119100","state":["exists","up"]},{"osd":1,"uuid":"bd5e4806-535b-49b5-ba6b-93228da35a84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6801","nonce":2744598742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6803","nonce":2744598742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6807","nonce":2744598742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6805","nonce":2744598742}]},"public_addr":"172.21.15.98:6801/2744598742","cluster_addr":"172.21.15.98:6803/2744598742","heartbeat_back_addr":"172.21.15.98:6807/2744598742","heartbeat_front_addr":"172.21.15.98:6805/2744598742","state":["exists","up"]},{"osd":2,"uuid":"8456fb55-910a-4a59-b84b-2c0f68346ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6801","nonce":1328098370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6803","nonce":1328098370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6807","nonce":1328098370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6805","nonce":1328098370}]},"public_addr":"172.21.15.112:6801/1328098370","cluster_addr":"172.21.15.112:6803/1328098370","heartbeat_back_addr":"172.21.15.112:6807/1328098370","heartbeat_front_addr":"172.21.15.112:6805/1328098370","state":["exists","up"]},{"osd":3,"uuid":"c8fbc440-bf70-42d3-b01e-777ccfa7900b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6811","nonce":4041566346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6813","nonce":4041566346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6817","nonce":4041566346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6815","nonce":4041566346}]},"public_addr":"172.21.15.70:6811/4041566346","cluster_addr":"172.21.15.70:6813/4041566346","heartbeat_back_addr":"172.21.15.70:6817/4041566346","heartbeat_front_addr":"172.21.15.70:6815/4041566346","state":["exists","up"]},{"osd":4,"uuid":"549c8c70-8be8-46ae-bcb6-0fde32b40cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6809","nonce":3652775814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6811","nonce":3652775814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6815","nonce":3652775814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6813","nonce":3652775814}]},"public_addr":"172.21.15.98:6809/3652775814","cluster_addr":"172.21.15.98:6811/3652775814","heartbeat_back_addr":"172.21.15.98:6815/3652775814","heartbeat_front_addr":"172.21.15.98:6813/3652775814","state":["exists","up"]},{"osd":5,"uuid":"51d86af1-0d6c-4ae0-b090-00414fc1bf7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6809","nonce":1857257899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6811","nonce":1857257899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6815","nonce":1857257899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6813","nonce":1857257899}]},"public_addr":"172.21.15.112:6809/1857257899","cluster_addr":"172.21.15.112:6811/1857257899","heartbeat_back_addr":"172.21.15.112:6815/1857257899","heartbeat_front_addr":"172.21.15.112:6813/1857257899","state":["exists","up"]},{"osd":6,"uuid":"526a8138-bfa8-419e-a5b5-e6b80f0ae004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6819","nonce":1264676496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6821","nonce":1264676496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6825","nonce":1264676496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6823","nonce":1264676496}]},"public_addr":"172.21.15.70:6819/1264676496","cluster_addr":"172.21.15.70:6821/1264676496","heartbeat_back_addr":"172.21.15.70:6825/1264676496","heartbeat_front_addr":"172.21.15.70:6823/1264676496","state":["exists","up"]},{"osd":7,"uuid":"1173f785-6ed1-47ef-9265-2e233ea9ee02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6817","nonce":709593020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6819","nonce":709593020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6823","nonce":709593020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6821","nonce":709593020}]},"public_addr":"172.21.15.98:6817/709593020","cluster_addr":"172.21.15.98:6819/709593020","heartbeat_back_addr":"172.21.15.98:6823/709593020","heartbeat_front_addr":"172.21.15.98:6821/709593020","state":["exists","up"]},{"osd":8,"uuid":"6424cc6e-24a7-4b81-9ce6-39203fecc7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6817","nonce":2250225413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6819","nonce":2250225413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6823","nonce":2250225413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6821","nonce":2250225413}]},"public_addr":"172.21.15.112:6817/2250225413","cluster_addr":"172.21.15.112:6819/2250225413","heartbeat_back_addr":"172.21.15.112:6823/2250225413","heartbeat_front_addr":"172.21.15.112:6821/2250225413","state":["exists","up"]},{"osd":9,"uuid":"8d666db8-8831-4d63-9e3c-a4bf2504149e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6827","nonce":3557688707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6829","nonce":3557688707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6833","nonce":3557688707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6831","nonce":3557688707}]},"public_addr":"172.21.15.70:6827/3557688707","cluster_addr":"172.21.15.70:6829/3557688707","heartbeat_back_addr":"172.21.15.70:6833/3557688707","heartbeat_front_addr":"172.21.15.70:6831/3557688707","state":["exists","up"]},{"osd":10,"uuid":"ecb925d2-0263-4dfc-a2fd-9e27856022d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6825","nonce":1896493718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6827","nonce":1896493718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6831","nonce":1896493718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6829","nonce":1896493718}]},"public_addr":"172.21.15.98:6825/1896493718","cluster_addr":"172.21.15.98:6827/1896493718","heartbeat_back_addr":"172.21.15.98:6831/1896493718","heartbeat_front_addr":"172.21.15.98:6829/1896493718","state":["exists","up"]},{"osd":11,"uuid":"c8e3230c-bc3a-40bd-8dd0-5fef56e4df97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6825","nonce":1400793137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6827","nonce":1400793137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6831","nonce":1400793137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6829","nonce":1400793137}]},"public_addr":"172.21.15.112:6825/1400793137","cluster_addr":"172.21.15.112:6827/1400793137","heartbeat_back_addr":"172.21.15.112:6831/1400793137","heartbeat_front_addr":"172.21.15.112:6829/1400793137","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:47:53.626608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:17.784505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:41.971969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:07.130699+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:31.615496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:56.898607+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:22.739991+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:48.319727+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:13.902063+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:40.389470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:06.142092+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:32.949383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/3442074771":"2024-06-05T04:46:17.636907+0000","172.21.15.70:6801/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/1442095493":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/2453733268":"2024-06-05T04:45:49.501864+0000","172.21.15.70:0/1369781447":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6801/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/915186456":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/860033006":"2024-06-05T04:45:49.501864+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-04T04:53:44.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:44.208+0000 7f4561367700 1 -- 172.21.15.70:0/2893939645 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f45440b2880 msgr2=0x7f45440b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:44.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:44.208+0000 7f4561367700 1 --2- 172.21.15.70:0/2893939645 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f45440b2880 0x7f45440b4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f455c06ccf0 tx=0x7f454c00b2c0 comp rx=0 tx=0).stop 2024-06-04T04:53:44.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:44.208+0000 7f4561367700 1 -- 172.21.15.70:0/2893939645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f455c102a80 msgr2=0x7f455c19f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:44.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:44.208+0000 7f4561367700 1 --2- 172.21.15.70:0/2893939645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f455c102a80 0x7f455c19f7b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4550000c00 tx=0x7f45500047b0 comp rx=0 tx=0).stop 2024-06-04T04:53:44.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:44.208+0000 7f4561367700 1 -- 172.21.15.70:0/2893939645 shutdown_connections 2024-06-04T04:53:44.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:44.208+0000 7f4561367700 1 --2- 172.21.15.70:0/2893939645 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f45440b2880 0x7f45440b4d40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:44.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:44.208+0000 7f4561367700 1 --2- 172.21.15.70:0/2893939645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f455c1077d0 0x7f455c1999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:44.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:44.208+0000 7f4561367700 1 --2- 172.21.15.70:0/2893939645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f455c102a80 0x7f455c19f7b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:44.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:44.208+0000 7f4561367700 1 --2- 172.21.15.70:0/2893939645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f455c103430 0x7f455c19fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:44.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:44.208+0000 7f4561367700 1 -- 172.21.15.70:0/2893939645 >> 172.21.15.70:0/2893939645 conn(0x7f455c0793f0 msgr2=0x7f455c106500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:44.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:44.209+0000 7f4561367700 1 -- 172.21.15.70:0/2893939645 shutdown_connections 2024-06-04T04:53:44.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:44.209+0000 7f4561367700 1 -- 172.21.15.70:0/2893939645 wait complete. 2024-06-04T04:53:44.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:44 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2893939645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:53:44.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:44 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2893939645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:53:44.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:44 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2893939645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:53:44.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-06-04T04:53:45.214 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:53:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:45 smithi070 ceph-mon[32631]: pgmap v292: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-06-04T04:53:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:45 smithi112 ceph-mon[36453]: pgmap v292: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-06-04T04:53:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:45 smithi098 ceph-mon[35569]: pgmap v292: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-06-04T04:53:46.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.899+0000 7f3a48abe700 1 -- 172.21.15.70:0/2895643830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a44103850 msgr2=0x7f3a44103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:46.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.899+0000 7f3a48abe700 1 --2- 172.21.15.70:0/2895643830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a44103850 0x7f3a44103c30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c0099c0 tx=0x7f3a2c009cd0 comp rx=0 tx=0).stop 2024-06-04T04:53:46.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.899+0000 7f3a48abe700 1 -- 172.21.15.70:0/2895643830 shutdown_connections 2024-06-04T04:53:46.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.899+0000 7f3a48abe700 1 --2- 172.21.15.70:0/2895643830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a44108920 0x7f3a441104a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:46.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.899+0000 7f3a48abe700 1 --2- 172.21.15.70:0/2895643830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a44103850 0x7f3a44103c30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:46.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.899+0000 7f3a48abe700 1 --2- 172.21.15.70:0/2895643830 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a44104200 0x7f3a441081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:46.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.899+0000 7f3a48abe700 1 -- 172.21.15.70:0/2895643830 >> 172.21.15.70:0/2895643830 conn(0x7f3a4407a620 msgr2=0x7f3a4407aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:46.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.899+0000 7f3a48abe700 1 -- 172.21.15.70:0/2895643830 shutdown_connections 2024-06-04T04:53:46.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.899+0000 7f3a48abe700 1 -- 172.21.15.70:0/2895643830 wait complete. 2024-06-04T04:53:46.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.900+0000 7f3a48abe700 1 Processor -- start 2024-06-04T04:53:46.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.900+0000 7f3a48abe700 1 -- start start 2024-06-04T04:53:46.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.900+0000 7f3a48abe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a44103850 0x7f3a4419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:46.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a48abe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a44104200 0x7f3a4419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:46.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a48abe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a44108920 0x7f3a44199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:46.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a48abe700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3a441135d0 con 0x7f3a44108920 2024-06-04T04:53:46.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a48abe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3a44113450 con 0x7f3a44103850 2024-06-04T04:53:46.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a48abe700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3a44113750 con 0x7f3a44104200 2024-06-04T04:53:46.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a42d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a44108920 0x7f3a44199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:46.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a42d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a44108920 0x7f3a44199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42522/0 (socket says 172.21.15.70:42522) 2024-06-04T04:53:46.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a42d9d700 1 -- 172.21.15.70:0/1707529086 learned_addr learned my addr 172.21.15.70:0/1707529086 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:53:46.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a41d9b700 1 --2- 172.21.15.70:0/1707529086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a44104200 0x7f3a4419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:46.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a4259c700 1 --2- 172.21.15.70:0/1707529086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a44103850 0x7f3a4419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:46.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a42d9d700 1 -- 172.21.15.70:0/1707529086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a44104200 msgr2=0x7f3a4419fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:46.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a42d9d700 1 --2- 172.21.15.70:0/1707529086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a44104200 0x7f3a4419fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:46.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a42d9d700 1 -- 172.21.15.70:0/1707529086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a44103850 msgr2=0x7f3a4419f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:46.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a42d9d700 1 --2- 172.21.15.70:0/1707529086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a44103850 0x7f3a4419f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:46.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a42d9d700 1 -- 172.21.15.70:0/1707529086 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a44199fd0 con 0x7f3a44108920 2024-06-04T04:53:46.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a42d9d700 1 --2- 172.21.15.70:0/1707529086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a44108920 0x7f3a44199870 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c007d70 tx=0x7f3a3c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:46.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a3b7fe700 1 -- 172.21.15.70:0/1707529086 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c016020 con 0x7f3a44108920 2024-06-04T04:53:46.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a3b7fe700 1 -- 172.21.15.70:0/1707529086 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3a3c00ac90 con 0x7f3a44108920 2024-06-04T04:53:46.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a48abe700 1 -- 172.21.15.70:0/1707529086 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a4419a2c0 con 0x7f3a44108920 2024-06-04T04:53:46.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a3b7fe700 1 -- 172.21.15.70:0/1707529086 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c0115e0 con 0x7f3a44108920 2024-06-04T04:53:46.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.901+0000 7f3a48abe700 1 -- 172.21.15.70:0/1707529086 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a44075770 con 0x7f3a44108920 2024-06-04T04:53:46.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.903+0000 7f3a48abe700 1 -- 172.21.15.70:0/1707529086 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a4410bb60 con 0x7f3a44108920 2024-06-04T04:53:46.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.904+0000 7f3a3b7fe700 1 -- 172.21.15.70:0/1707529086 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f3a3c019020 con 0x7f3a44108920 2024-06-04T04:53:46.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.904+0000 7f3a3b7fe700 1 --2- 172.21.15.70:0/1707529086 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3a300b2880 0x7f3a300b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:46.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.905+0000 7f3a4259c700 1 --2- 172.21.15.70:0/1707529086 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3a300b2880 0x7f3a300b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:46.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.905+0000 7f3a3b7fe700 1 -- 172.21.15.70:0/1707529086 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f3a3c0e25d0 con 0x7f3a44108920 2024-06-04T04:53:46.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.905+0000 7f3a4259c700 1 --2- 172.21.15.70:0/1707529086 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3a300b2880 0x7f3a300b4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c009990 tx=0x7f3a2c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:46.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:46.908+0000 7f3a3b7fe700 1 -- 172.21.15.70:0/1707529086 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3a3c020020 con 0x7f3a44108920 2024-06-04T04:53:47.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:47.161+0000 7f3a48abe700 1 -- 172.21.15.70:0/1707529086 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f3a4419ada0 con 0x7f3a44108920 2024-06-04T04:53:47.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:47.385+0000 7f3a3b7fe700 1 -- 172.21.15.70:0/1707529086 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f3a3c0aa440 con 0x7f3a44108920 2024-06-04T04:53:47.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:47.388+0000 7f3a48abe700 1 -- 172.21.15.70:0/1707529086 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3a300b2880 msgr2=0x7f3a300b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:47.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:47.388+0000 7f3a48abe700 1 --2- 172.21.15.70:0/1707529086 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3a300b2880 0x7f3a300b4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c009990 tx=0x7f3a2c01a040 comp rx=0 tx=0).stop 2024-06-04T04:53:47.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:47.388+0000 7f3a48abe700 1 -- 172.21.15.70:0/1707529086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a44108920 msgr2=0x7f3a44199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:47.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:47.388+0000 7f3a48abe700 1 --2- 172.21.15.70:0/1707529086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a44108920 0x7f3a44199870 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c007d70 tx=0x7f3a3c00a340 comp rx=0 tx=0).stop 2024-06-04T04:53:47.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:47.388+0000 7f3a48abe700 1 -- 172.21.15.70:0/1707529086 shutdown_connections 2024-06-04T04:53:47.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:47.388+0000 7f3a48abe700 1 --2- 172.21.15.70:0/1707529086 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3a300b2880 0x7f3a300b4d40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:47.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:47.388+0000 7f3a48abe700 1 --2- 172.21.15.70:0/1707529086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a44108920 0x7f3a44199870 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:47.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:47.388+0000 7f3a48abe700 1 --2- 172.21.15.70:0/1707529086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a44103850 0x7f3a4419f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:47.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:47.388+0000 7f3a48abe700 1 --2- 172.21.15.70:0/1707529086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a44104200 0x7f3a4419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:47.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:47.388+0000 7f3a48abe700 1 -- 172.21.15.70:0/1707529086 >> 172.21.15.70:0/1707529086 conn(0x7f3a4407a620 msgr2=0x7f3a44109a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:47.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:47.388+0000 7f3a48abe700 1 -- 172.21.15.70:0/1707529086 shutdown_connections 2024-06-04T04:53:47.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:47.388+0000 7f3a48abe700 1 -- 172.21.15.70:0/1707529086 wait complete. 2024-06-04T04:53:47.426 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:47 smithi070 ceph-mon[32631]: pgmap v293: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-06-04T04:53:47.426 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:47 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1707529086' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-06-04T04:53:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:47 smithi112 ceph-mon[36453]: pgmap v293: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-06-04T04:53:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:47 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1707529086' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-06-04T04:53:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:47 smithi098 ceph-mon[35569]: pgmap v293: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-06-04T04:53:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:47 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1707529086' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-06-04T04:53:48.185 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-06-04T04:53:48.186 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-06-04T04:53:48.430 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:53:48.547 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:48 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1707529086' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-06-04T04:53:48.547 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:48 smithi070 ceph-mon[32631]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-04T04:53:48.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:48 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1707529086' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-06-04T04:53:48.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:48 smithi112 ceph-mon[36453]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-04T04:53:48.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:48 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1707529086' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-06-04T04:53:48.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:48 smithi098 ceph-mon[35569]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-04T04:53:49.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:49 smithi070 ceph-mon[32631]: pgmap v294: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-06-04T04:53:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:49 smithi112 ceph-mon[36453]: pgmap v294: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-06-04T04:53:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:49 smithi098 ceph-mon[35569]: pgmap v294: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-06-04T04:53:50.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.045+0000 7f0588db3700 1 -- 172.21.15.70:0/3347320087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f058406c670 msgr2=0x7f0584110410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:50.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.045+0000 7f0588db3700 1 --2- 172.21.15.70:0/3347320087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f058406c670 0x7f0584110410 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f057c00b0a0 tx=0x7f057c00b3b0 comp rx=0 tx=0).stop 2024-06-04T04:53:50.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.045+0000 7f0588db3700 1 -- 172.21.15.70:0/3347320087 shutdown_connections 2024-06-04T04:53:50.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.045+0000 7f0588db3700 1 --2- 172.21.15.70:0/3347320087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f058406c670 0x7f0584110410 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:50.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.045+0000 7f0588db3700 1 --2- 172.21.15.70:0/3347320087 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0584109620 0x7f0584109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:50.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.045+0000 7f0588db3700 1 --2- 172.21.15.70:0/3347320087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058406bcb0 0x7f058406c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:50.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.045+0000 7f0588db3700 1 -- 172.21.15.70:0/3347320087 >> 172.21.15.70:0/3347320087 conn(0x7f0584078490 msgr2=0x7f05840788a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:50.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.045+0000 7f0588db3700 1 -- 172.21.15.70:0/3347320087 shutdown_connections 2024-06-04T04:53:50.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.045+0000 7f0588db3700 1 -- 172.21.15.70:0/3347320087 wait complete. 2024-06-04T04:53:50.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.045+0000 7f0588db3700 1 Processor -- start 2024-06-04T04:53:50.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.045+0000 7f0588db3700 1 -- start start 2024-06-04T04:53:50.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f0588db3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058406bcb0 0x7f058419f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:50.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f0588db3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f058406c670 0x7f058419fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:50.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f0588db3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0584109620 0x7f0584199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:50.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f0588db3700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0584113630 con 0x7f0584109620 2024-06-04T04:53:50.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f0588db3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f05841134b0 con 0x7f058406bcb0 2024-06-04T04:53:50.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f0588db3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f05841137b0 con 0x7f058406c670 2024-06-04T04:53:50.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f0582d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0584109620 0x7f0584199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:50.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f0582d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0584109620 0x7f0584199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42536/0 (socket says 172.21.15.70:42536) 2024-06-04T04:53:50.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f0582d9d700 1 -- 172.21.15.70:0/2826799318 learned_addr learned my addr 172.21.15.70:0/2826799318 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:53:50.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f0581d9b700 1 --2- 172.21.15.70:0/2826799318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f058406c670 0x7f058419fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:50.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f058259c700 1 --2- 172.21.15.70:0/2826799318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058406bcb0 0x7f058419f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:50.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f0582d9d700 1 -- 172.21.15.70:0/2826799318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f058406c670 msgr2=0x7f058419fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:50.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f0582d9d700 1 --2- 172.21.15.70:0/2826799318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f058406c670 0x7f058419fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:50.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f0582d9d700 1 -- 172.21.15.70:0/2826799318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058406bcb0 msgr2=0x7f058419f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:50.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f0582d9d700 1 --2- 172.21.15.70:0/2826799318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058406bcb0 0x7f058419f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:50.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f0582d9d700 1 -- 172.21.15.70:0/2826799318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058419a030 con 0x7f0584109620 2024-06-04T04:53:50.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f058259c700 1 --2- 172.21.15.70:0/2826799318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058406bcb0 0x7f058419f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:53:50.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f0582d9d700 1 --2- 172.21.15.70:0/2826799318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0584109620 0x7f0584199850 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f057c00b830 tx=0x7f057c004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:50.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f05737fe700 1 -- 172.21.15.70:0/2826799318 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c021070 con 0x7f0584109620 2024-06-04T04:53:50.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f05737fe700 1 -- 172.21.15.70:0/2826799318 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f057c007b80 con 0x7f0584109620 2024-06-04T04:53:50.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f0588db3700 1 -- 172.21.15.70:0/2826799318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f058419a2c0 con 0x7f0584109620 2024-06-04T04:53:50.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.046+0000 7f05737fe700 1 -- 172.21.15.70:0/2826799318 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c0052a0 con 0x7f0584109620 2024-06-04T04:53:50.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.047+0000 7f0588db3700 1 -- 172.21.15.70:0/2826799318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05841a6530 con 0x7f0584109620 2024-06-04T04:53:50.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.047+0000 7f0588db3700 1 -- 172.21.15.70:0/2826799318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f058410bad0 con 0x7f0584109620 2024-06-04T04:53:50.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.049+0000 7f05737fe700 1 -- 172.21.15.70:0/2826799318 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f057c024070 con 0x7f0584109620 2024-06-04T04:53:50.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.050+0000 7f05737fe700 1 --2- 172.21.15.70:0/2826799318 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f056c0b2880 0x7f056c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:50.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.050+0000 7f058259c700 1 --2- 172.21.15.70:0/2826799318 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f056c0b2880 0x7f056c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:50.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.050+0000 7f05737fe700 1 -- 172.21.15.70:0/2826799318 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f057c0e3630 con 0x7f0584109620 2024-06-04T04:53:50.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.050+0000 7f058259c700 1 --2- 172.21.15.70:0/2826799318 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f056c0b2880 0x7f056c0b4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f057400bcf0 tx=0x7f057400b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:50.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.053+0000 7f05737fe700 1 -- 172.21.15.70:0/2826799318 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f057c0ab530 con 0x7f0584109620 2024-06-04T04:53:50.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.301+0000 7f0588db3700 1 -- 172.21.15.70:0/2826799318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f058419ad80 con 0x7f0584109620 2024-06-04T04:53:50.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.404+0000 7f05737fe700 1 -- 172.21.15.70:0/2826799318 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f057c0aae30 con 0x7f0584109620 2024-06-04T04:53:50.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.407+0000 7f0588db3700 1 -- 172.21.15.70:0/2826799318 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f056c0b2880 msgr2=0x7f056c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:50.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.407+0000 7f0588db3700 1 --2- 172.21.15.70:0/2826799318 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f056c0b2880 0x7f056c0b4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f057400bcf0 tx=0x7f057400b2c0 comp rx=0 tx=0).stop 2024-06-04T04:53:50.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.407+0000 7f0588db3700 1 -- 172.21.15.70:0/2826799318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0584109620 msgr2=0x7f0584199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:50.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.407+0000 7f0588db3700 1 --2- 172.21.15.70:0/2826799318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0584109620 0x7f0584199850 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f057c00b830 tx=0x7f057c004d00 comp rx=0 tx=0).stop 2024-06-04T04:53:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.407+0000 7f0588db3700 1 -- 172.21.15.70:0/2826799318 shutdown_connections 2024-06-04T04:53:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.407+0000 7f0588db3700 1 --2- 172.21.15.70:0/2826799318 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f056c0b2880 0x7f056c0b4d40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.407+0000 7f0588db3700 1 --2- 172.21.15.70:0/2826799318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0584109620 0x7f0584199850 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.407+0000 7f0588db3700 1 --2- 172.21.15.70:0/2826799318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058406bcb0 0x7f058419f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.407+0000 7f0588db3700 1 --2- 172.21.15.70:0/2826799318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f058406c670 0x7f058419fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.407+0000 7f0588db3700 1 -- 172.21.15.70:0/2826799318 >> 172.21.15.70:0/2826799318 conn(0x7f0584078490 msgr2=0x7f05841010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.407+0000 7f0588db3700 1 -- 172.21.15.70:0/2826799318 shutdown_connections 2024-06-04T04:53:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:50.407+0000 7f0588db3700 1 -- 172.21.15.70:0/2826799318 wait complete. 2024-06-04T04:53:50.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:50 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2826799318' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-06-04T04:53:50.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:50 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2826799318' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-06-04T04:53:50.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:50 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2826799318' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-06-04T04:53:51.143 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-06-04T04:53:51.385 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:53:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:51 smithi112 ceph-mon[36453]: pgmap v295: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-06-04T04:53:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:51 smithi112 ceph-mon[36453]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-06-04T04:53:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:51 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2826799318' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-06-04T04:53:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:51 smithi112 ceph-mon[36453]: daemon mds.c assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-06-04T04:53:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:51 smithi112 ceph-mon[36453]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-04T04:53:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:51 smithi112 ceph-mon[36453]: fsmap cephfs:2 {0=i=up:active,1=c=up:creating} 10 up:standby 2024-06-04T04:53:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:51 smithi112 ceph-mon[36453]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-06-04T04:53:51.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:51 smithi070 ceph-mon[32631]: pgmap v295: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-06-04T04:53:51.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:51 smithi070 ceph-mon[32631]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-06-04T04:53:51.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:51 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2826799318' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-06-04T04:53:51.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:51 smithi070 ceph-mon[32631]: daemon mds.c assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-06-04T04:53:51.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:51 smithi070 ceph-mon[32631]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-04T04:53:51.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:51 smithi070 ceph-mon[32631]: fsmap cephfs:2 {0=i=up:active,1=c=up:creating} 10 up:standby 2024-06-04T04:53:51.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:51 smithi070 ceph-mon[32631]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-06-04T04:53:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:51 smithi098 ceph-mon[35569]: pgmap v295: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-06-04T04:53:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:51 smithi098 ceph-mon[35569]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-06-04T04:53:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:51 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2826799318' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-06-04T04:53:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:51 smithi098 ceph-mon[35569]: daemon mds.c assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-06-04T04:53:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:51 smithi098 ceph-mon[35569]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-04T04:53:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:51 smithi098 ceph-mon[35569]: fsmap cephfs:2 {0=i=up:active,1=c=up:creating} 10 up:standby 2024-06-04T04:53:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:51 smithi098 ceph-mon[35569]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-06-04T04:53:52.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:52 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] up:active 2024-06-04T04:53:52.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:52 smithi112 ceph-mon[36453]: daemon mds.b assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-06-04T04:53:52.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:52 smithi112 ceph-mon[36453]: fsmap cephfs:2 {0=i=up:active,1=c=up:active} 10 up:standby 2024-06-04T04:53:52.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:52 smithi112 ceph-mon[36453]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=b=up:creating} 9 up:standby 2024-06-04T04:53:52.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:52 smithi112 ceph-mon[36453]: daemon mds.b is now active in filesystem cephfs as rank 2 2024-06-04T04:53:52.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:52 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] up:active 2024-06-04T04:53:52.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:52 smithi098 ceph-mon[35569]: daemon mds.b assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-06-04T04:53:52.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:52 smithi098 ceph-mon[35569]: fsmap cephfs:2 {0=i=up:active,1=c=up:active} 10 up:standby 2024-06-04T04:53:52.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:52 smithi098 ceph-mon[35569]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=b=up:creating} 9 up:standby 2024-06-04T04:53:52.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:52 smithi098 ceph-mon[35569]: daemon mds.b is now active in filesystem cephfs as rank 2 2024-06-04T04:53:52.783 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:52 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] up:active 2024-06-04T04:53:52.783 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:52 smithi070 ceph-mon[32631]: daemon mds.b assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-06-04T04:53:52.783 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:52 smithi070 ceph-mon[32631]: fsmap cephfs:2 {0=i=up:active,1=c=up:active} 10 up:standby 2024-06-04T04:53:52.783 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:52 smithi070 ceph-mon[32631]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=b=up:creating} 9 up:standby 2024-06-04T04:53:52.783 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:52 smithi070 ceph-mon[32631]: daemon mds.b is now active in filesystem cephfs as rank 2 2024-06-04T04:53:53.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.161+0000 7f09c4873700 1 -- 172.21.15.70:0/17702246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09c01084b0 msgr2=0x7f09c0108890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:53.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.161+0000 7f09c4873700 1 --2- 172.21.15.70:0/17702246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09c01084b0 0x7f09c0108890 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009990 tx=0x7f09b0009ca0 comp rx=0 tx=0).stop 2024-06-04T04:53:53.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.162+0000 7f09c4873700 1 -- 172.21.15.70:0/17702246 shutdown_connections 2024-06-04T04:53:53.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.162+0000 7f09c4873700 1 --2- 172.21.15.70:0/17702246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09c0102e30 0x7f09c010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:53.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.162+0000 7f09c4873700 1 --2- 172.21.15.70:0/17702246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09c01084b0 0x7f09c0108890 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:53.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.162+0000 7f09c4873700 1 --2- 172.21.15.70:0/17702246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09c0102470 0x7f09c01028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:53.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.162+0000 7f09c4873700 1 -- 172.21.15.70:0/17702246 >> 172.21.15.70:0/17702246 conn(0x7f09c00793f0 msgr2=0x7f09c0079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:53.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.162+0000 7f09c4873700 1 -- 172.21.15.70:0/17702246 shutdown_connections 2024-06-04T04:53:53.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.162+0000 7f09c4873700 1 -- 172.21.15.70:0/17702246 wait complete. 2024-06-04T04:53:53.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.162+0000 7f09c4873700 1 Processor -- start 2024-06-04T04:53:53.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.162+0000 7f09c4873700 1 -- start start 2024-06-04T04:53:53.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09c4873700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09c0102470 0x7f09c0104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:53.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09c4873700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09c0102e30 0x7f09c01045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:53.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09c4873700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09c01084b0 0x7f09c0104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:53.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09c4873700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f09c0112490 con 0x7f09c0102470 2024-06-04T04:53:53.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09c4873700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09c0112310 con 0x7f09c01084b0 2024-06-04T04:53:53.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09c4873700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f09c0112610 con 0x7f09c0102e30 2024-06-04T04:53:53.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09be59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09c0102470 0x7f09c0104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:53.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09be59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09c0102470 0x7f09c0104080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42556/0 (socket says 172.21.15.70:42556) 2024-06-04T04:53:53.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09be59c700 1 -- 172.21.15.70:0/1008322069 learned_addr learned my addr 172.21.15.70:0/1008322069 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:53:53.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09bdd9b700 1 --2- 172.21.15.70:0/1008322069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09c0102e30 0x7f09c01045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:53.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09bed9d700 1 --2- 172.21.15.70:0/1008322069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09c01084b0 0x7f09c0104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:53.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09be59c700 1 -- 172.21.15.70:0/1008322069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09c0102e30 msgr2=0x7f09c01045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:53.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09be59c700 1 --2- 172.21.15.70:0/1008322069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09c0102e30 0x7f09c01045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:53.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09be59c700 1 -- 172.21.15.70:0/1008322069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09c01084b0 msgr2=0x7f09c0104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:53.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09be59c700 1 --2- 172.21.15.70:0/1008322069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09c01084b0 0x7f09c0104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:53.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09be59c700 1 -- 172.21.15.70:0/1008322069 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c01a80e0 con 0x7f09c0102470 2024-06-04T04:53:53.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09be59c700 1 --2- 172.21.15.70:0/1008322069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09c0102470 0x7f09c0104080 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f09b0000c00 tx=0x7f09b00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:53.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09af7fe700 1 -- 172.21.15.70:0/1008322069 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b0027070 con 0x7f09c0102470 2024-06-04T04:53:53.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09c4873700 1 -- 172.21.15.70:0/1008322069 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c01a8370 con 0x7f09c0102470 2024-06-04T04:53:53.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09af7fe700 1 -- 172.21.15.70:0/1008322069 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f09b000fb00 con 0x7f09c0102470 2024-06-04T04:53:53.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.163+0000 7f09af7fe700 1 -- 172.21.15.70:0/1008322069 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b0022c40 con 0x7f09c0102470 2024-06-04T04:53:53.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.164+0000 7f09c4873700 1 -- 172.21.15.70:0/1008322069 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c01a8820 con 0x7f09c0102470 2024-06-04T04:53:53.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.164+0000 7f09c4873700 1 -- 172.21.15.70:0/1008322069 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c006a230 con 0x7f09c0102470 2024-06-04T04:53:53.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.166+0000 7f09af7fe700 1 -- 172.21.15.70:0/1008322069 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f09b002a070 con 0x7f09c0102470 2024-06-04T04:53:53.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.167+0000 7f09af7fe700 1 --2- 172.21.15.70:0/1008322069 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f09a80b2880 0x7f09a80b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:53.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.167+0000 7f09bdd9b700 1 --2- 172.21.15.70:0/1008322069 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f09a80b2880 0x7f09a80b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:53.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.167+0000 7f09af7fe700 1 -- 172.21.15.70:0/1008322069 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f09b00e33c0 con 0x7f09c0102470 2024-06-04T04:53:53.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.167+0000 7f09bdd9b700 1 --2- 172.21.15.70:0/1008322069 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f09a80b2880 0x7f09a80b4d40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f09c0079ba0 tx=0x7f09b40073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:53.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.171+0000 7f09af7fe700 1 -- 172.21.15.70:0/1008322069 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f09b0031070 con 0x7f09c0102470 2024-06-04T04:53:53.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.423+0000 7f09c4873700 1 -- 172.21.15.70:0/1008322069 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f09c002e560 con 0x7f09c0102470 2024-06-04T04:53:53.428 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 04:53:53 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T04:53:53.117+0000 7f03b194b700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-06-04T04:53:53.428 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 04:53:53 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T04:53:53.117+0000 7f03b194b700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-06-04T04:53:53.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.430+0000 7f09af7fe700 1 -- 172.21.15.70:0/1008322069 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v24) ==== 121+0+0 (secure 0 0 0) 0x7f09b0013070 con 0x7f09c0102470 2024-06-04T04:53:53.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.433+0000 7f09c4873700 1 -- 172.21.15.70:0/1008322069 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f09a80b2880 msgr2=0x7f09a80b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:53.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.433+0000 7f09c4873700 1 --2- 172.21.15.70:0/1008322069 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f09a80b2880 0x7f09a80b4d40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f09c0079ba0 tx=0x7f09b40073d0 comp rx=0 tx=0).stop 2024-06-04T04:53:53.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.433+0000 7f09c4873700 1 -- 172.21.15.70:0/1008322069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09c0102470 msgr2=0x7f09c0104080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:53.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.433+0000 7f09c4873700 1 --2- 172.21.15.70:0/1008322069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09c0102470 0x7f09c0104080 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f09b0000c00 tx=0x7f09b00047b0 comp rx=0 tx=0).stop 2024-06-04T04:53:53.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.433+0000 7f09c4873700 1 -- 172.21.15.70:0/1008322069 shutdown_connections 2024-06-04T04:53:53.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.433+0000 7f09c4873700 1 --2- 172.21.15.70:0/1008322069 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f09a80b2880 0x7f09a80b4d40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:53.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.433+0000 7f09c4873700 1 --2- 172.21.15.70:0/1008322069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09c01084b0 0x7f09c0104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:53.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.433+0000 7f09c4873700 1 --2- 172.21.15.70:0/1008322069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09c0102470 0x7f09c0104080 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:53.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.433+0000 7f09c4873700 1 --2- 172.21.15.70:0/1008322069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09c0102e30 0x7f09c01045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:53.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.433+0000 7f09c4873700 1 -- 172.21.15.70:0/1008322069 >> 172.21.15.70:0/1008322069 conn(0x7f09c00793f0 msgr2=0x7f09c006c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:53.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.433+0000 7f09c4873700 1 -- 172.21.15.70:0/1008322069 shutdown_connections 2024-06-04T04:53:53.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:53.433+0000 7f09c4873700 1 -- 172.21.15.70:0/1008322069 wait complete. 2024-06-04T04:53:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:53 smithi112 ceph-mon[36453]: pgmap v296: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 3 op/s 2024-06-04T04:53:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:53 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] up:active 2024-06-04T04:53:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:53 smithi112 ceph-mon[36453]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-06-04T04:53:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:53 smithi112 ceph-mon[36453]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=b=up:active} 9 up:standby 2024-06-04T04:53:53.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:53 smithi112 ceph-mon[36453]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=b=up:active,3=f=up:creating} 8 up:standby 2024-06-04T04:53:53.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:53 smithi112 ceph-mon[36453]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-06-04T04:53:53.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:53 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1008322069' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-06-04T04:53:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:53 smithi098 ceph-mon[35569]: pgmap v296: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 3 op/s 2024-06-04T04:53:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:53 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] up:active 2024-06-04T04:53:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:53 smithi098 ceph-mon[35569]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-06-04T04:53:53.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:53 smithi098 ceph-mon[35569]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=b=up:active} 9 up:standby 2024-06-04T04:53:53.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:53 smithi098 ceph-mon[35569]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=b=up:active,3=f=up:creating} 8 up:standby 2024-06-04T04:53:53.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:53 smithi098 ceph-mon[35569]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-06-04T04:53:53.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:53 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1008322069' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-06-04T04:53:53.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:53 smithi070 ceph-mon[32631]: pgmap v296: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 3 op/s 2024-06-04T04:53:53.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:53 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] up:active 2024-06-04T04:53:53.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:53 smithi070 ceph-mon[32631]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-06-04T04:53:53.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:53 smithi070 ceph-mon[32631]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=b=up:active} 9 up:standby 2024-06-04T04:53:53.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:53 smithi070 ceph-mon[32631]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=b=up:active,3=f=up:creating} 8 up:standby 2024-06-04T04:53:53.814 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:53 smithi070 ceph-mon[32631]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-06-04T04:53:53.814 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:53 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1008322069' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-06-04T04:53:54.145 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-06-04T04:53:54.391 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:53:54.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:54 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1008322069' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-06-04T04:53:54.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:54 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] up:active 2024-06-04T04:53:54.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:54 smithi112 ceph-mon[36453]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-06-04T04:53:54.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:54 smithi112 ceph-mon[36453]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-06-04T04:53:54.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:54 smithi112 ceph-mon[36453]: Cluster is now healthy 2024-06-04T04:53:54.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:54 smithi112 ceph-mon[36453]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=b=up:active,3=f=up:active} 8 up:standby 2024-06-04T04:53:54.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:54 smithi112 ceph-mon[36453]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-06-04T04:53:54.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:54 smithi112 ceph-mon[36453]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-06-04T04:53:54.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:54 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1008322069' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-06-04T04:53:54.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:54 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] up:active 2024-06-04T04:53:54.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:54 smithi070 ceph-mon[32631]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-06-04T04:53:54.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:54 smithi070 ceph-mon[32631]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-06-04T04:53:54.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:54 smithi070 ceph-mon[32631]: Cluster is now healthy 2024-06-04T04:53:54.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:54 smithi070 ceph-mon[32631]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=b=up:active,3=f=up:active} 8 up:standby 2024-06-04T04:53:54.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:54 smithi070 ceph-mon[32631]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-06-04T04:53:54.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:54 smithi070 ceph-mon[32631]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-06-04T04:53:54.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:54 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1008322069' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-06-04T04:53:54.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:54 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] up:active 2024-06-04T04:53:54.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:54 smithi098 ceph-mon[35569]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-06-04T04:53:54.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:54 smithi098 ceph-mon[35569]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-06-04T04:53:54.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:54 smithi098 ceph-mon[35569]: Cluster is now healthy 2024-06-04T04:53:54.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:54 smithi098 ceph-mon[35569]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=b=up:active,3=f=up:active} 8 up:standby 2024-06-04T04:53:54.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:54 smithi098 ceph-mon[35569]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-06-04T04:53:54.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:54 smithi098 ceph-mon[35569]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-06-04T04:53:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:55 smithi098 ceph-mon[35569]: pgmap v297: 97 pgs: 97 active+clean; 580 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 8 op/s 2024-06-04T04:53:55.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:55 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] up:active 2024-06-04T04:53:55.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:55 smithi098 ceph-mon[35569]: fsmap cephfs:5 7 up:standby 5 up:active 2024-06-04T04:53:55.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:55 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T04:53:55.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:55 smithi070 ceph-mon[32631]: pgmap v297: 97 pgs: 97 active+clean; 580 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 8 op/s 2024-06-04T04:53:55.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:55 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] up:active 2024-06-04T04:53:55.775 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:55 smithi070 ceph-mon[32631]: fsmap cephfs:5 7 up:standby 5 up:active 2024-06-04T04:53:55.775 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:55 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T04:53:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:55 smithi112 ceph-mon[36453]: pgmap v297: 97 pgs: 97 active+clean; 580 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 8 op/s 2024-06-04T04:53:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:55 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] up:active 2024-06-04T04:53:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:55 smithi112 ceph-mon[36453]: fsmap cephfs:5 7 up:standby 5 up:active 2024-06-04T04:53:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:55 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T04:53:56.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.179+0000 7efc25bf4700 1 -- 172.21.15.70:0/2608827812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc2006bbe0 msgr2=0x7efc201103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:56.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.179+0000 7efc25bf4700 1 --2- 172.21.15.70:0/2608827812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc2006bbe0 0x7efc201103d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efc1800b0a0 tx=0x7efc1800b3b0 comp rx=0 tx=0).stop 2024-06-04T04:53:56.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.179+0000 7efc25bf4700 1 -- 172.21.15.70:0/2608827812 shutdown_connections 2024-06-04T04:53:56.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.179+0000 7efc25bf4700 1 --2- 172.21.15.70:0/2608827812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc2006bbe0 0x7efc201103d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:56.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.179+0000 7efc25bf4700 1 --2- 172.21.15.70:0/2608827812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc201095b0 0x7efc20109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:56.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.179+0000 7efc25bf4700 1 --2- 172.21.15.70:0/2608827812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc2006b220 0x7efc2006b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:56.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.179+0000 7efc25bf4700 1 -- 172.21.15.70:0/2608827812 >> 172.21.15.70:0/2608827812 conn(0x7efc200793f0 msgr2=0x7efc20079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:56.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.179+0000 7efc25bf4700 1 -- 172.21.15.70:0/2608827812 shutdown_connections 2024-06-04T04:53:56.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.179+0000 7efc25bf4700 1 -- 172.21.15.70:0/2608827812 wait complete. 2024-06-04T04:53:56.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.180+0000 7efc25bf4700 1 Processor -- start 2024-06-04T04:53:56.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.180+0000 7efc25bf4700 1 -- start start 2024-06-04T04:53:56.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.180+0000 7efc25bf4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc2006b220 0x7efc20102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:56.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.180+0000 7efc25bf4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc2006bbe0 0x7efc20102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:56.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.180+0000 7efc25bf4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc201095b0 0x7efc20102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:56.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.180+0000 7efc25bf4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efc201135d0 con 0x7efc2006bbe0 2024-06-04T04:53:56.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.180+0000 7efc25bf4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efc20113450 con 0x7efc201095b0 2024-06-04T04:53:56.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.180+0000 7efc25bf4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efc20113750 con 0x7efc2006b220 2024-06-04T04:53:56.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.180+0000 7efc17fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc2006bbe0 0x7efc20102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:56.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.180+0000 7efc17fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc2006bbe0 0x7efc20102a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46274/0 (socket says 172.21.15.70:46274) 2024-06-04T04:53:56.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.180+0000 7efc17fff700 1 -- 172.21.15.70:0/2942825072 learned_addr learned my addr 172.21.15.70:0/2942825072 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:53:56.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.180+0000 7efc1f7fe700 1 --2- 172.21.15.70:0/2942825072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc2006b220 0x7efc20102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:56.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.180+0000 7efc1ffff700 1 --2- 172.21.15.70:0/2942825072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc201095b0 0x7efc20102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:56.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.181+0000 7efc17fff700 1 -- 172.21.15.70:0/2942825072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc2006b220 msgr2=0x7efc20102510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:56.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.181+0000 7efc17fff700 1 --2- 172.21.15.70:0/2942825072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc2006b220 0x7efc20102510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:56.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.181+0000 7efc17fff700 1 -- 172.21.15.70:0/2942825072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc201095b0 msgr2=0x7efc20102f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:56.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.181+0000 7efc17fff700 1 --2- 172.21.15.70:0/2942825072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc201095b0 0x7efc20102f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:56.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.181+0000 7efc17fff700 1 -- 172.21.15.70:0/2942825072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc201a7f50 con 0x7efc2006bbe0 2024-06-04T04:53:56.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.181+0000 7efc1ffff700 1 --2- 172.21.15.70:0/2942825072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc201095b0 0x7efc20102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:53:56.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.181+0000 7efc17fff700 1 --2- 172.21.15.70:0/2942825072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc2006bbe0 0x7efc20102a50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efc0800c9b0 tx=0x7efc0800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:56.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.181+0000 7efc1d7fa700 1 -- 172.21.15.70:0/2942825072 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc08007b00 con 0x7efc2006bbe0 2024-06-04T04:53:56.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.181+0000 7efc1d7fa700 1 -- 172.21.15.70:0/2942825072 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc08007ca0 con 0x7efc2006bbe0 2024-06-04T04:53:56.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.181+0000 7efc25bf4700 1 -- 172.21.15.70:0/2942825072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc201a8240 con 0x7efc2006bbe0 2024-06-04T04:53:56.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.181+0000 7efc1d7fa700 1 -- 172.21.15.70:0/2942825072 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc08018660 con 0x7efc2006bbe0 2024-06-04T04:53:56.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.181+0000 7efc25bf4700 1 -- 172.21.15.70:0/2942825072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc201a8780 con 0x7efc2006bbe0 2024-06-04T04:53:56.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.182+0000 7efc25bf4700 1 -- 172.21.15.70:0/2942825072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc2006a230 con 0x7efc2006bbe0 2024-06-04T04:53:56.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.183+0000 7efc1d7fa700 1 -- 172.21.15.70:0/2942825072 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7efc0800f450 con 0x7efc2006bbe0 2024-06-04T04:53:56.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.184+0000 7efc1d7fa700 1 --2- 172.21.15.70:0/2942825072 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7efc0c0b2880 0x7efc0c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:56.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.184+0000 7efc1f7fe700 1 --2- 172.21.15.70:0/2942825072 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7efc0c0b2880 0x7efc0c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:56.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.185+0000 7efc1d7fa700 1 -- 172.21.15.70:0/2942825072 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7efc080e2400 con 0x7efc2006bbe0 2024-06-04T04:53:56.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.185+0000 7efc1f7fe700 1 --2- 172.21.15.70:0/2942825072 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7efc0c0b2880 0x7efc0c0b4d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efc1000abd0 tx=0x7efc1000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:56.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.188+0000 7efc1d7fa700 1 -- 172.21.15.70:0/2942825072 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7efc08014030 con 0x7efc2006bbe0 2024-06-04T04:53:56.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.440+0000 7efc25bf4700 1 -- 172.21.15.70:0/2942825072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7efc2002e560 con 0x7efc2006bbe0 2024-06-04T04:53:56.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.448+0000 7efc1d7fa700 1 -- 172.21.15.70:0/2942825072 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v28) ==== 115+0+0 (secure 0 0 0) 0x7efc080aa260 con 0x7efc2006bbe0 2024-06-04T04:53:56.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.451+0000 7efc25bf4700 1 -- 172.21.15.70:0/2942825072 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7efc0c0b2880 msgr2=0x7efc0c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:56.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.451+0000 7efc25bf4700 1 --2- 172.21.15.70:0/2942825072 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7efc0c0b2880 0x7efc0c0b4d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efc1000abd0 tx=0x7efc1000a380 comp rx=0 tx=0).stop 2024-06-04T04:53:56.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.451+0000 7efc25bf4700 1 -- 172.21.15.70:0/2942825072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc2006bbe0 msgr2=0x7efc20102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:56.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.451+0000 7efc25bf4700 1 --2- 172.21.15.70:0/2942825072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc2006bbe0 0x7efc20102a50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efc0800c9b0 tx=0x7efc0800ccc0 comp rx=0 tx=0).stop 2024-06-04T04:53:56.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.451+0000 7efc25bf4700 1 -- 172.21.15.70:0/2942825072 shutdown_connections 2024-06-04T04:53:56.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.451+0000 7efc25bf4700 1 --2- 172.21.15.70:0/2942825072 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7efc0c0b2880 0x7efc0c0b4d40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:56.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.451+0000 7efc25bf4700 1 --2- 172.21.15.70:0/2942825072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc201095b0 0x7efc20102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:56.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.451+0000 7efc25bf4700 1 --2- 172.21.15.70:0/2942825072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc2006b220 0x7efc20102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:56.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.451+0000 7efc25bf4700 1 --2- 172.21.15.70:0/2942825072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc2006bbe0 0x7efc20102a50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:56.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.451+0000 7efc25bf4700 1 -- 172.21.15.70:0/2942825072 >> 172.21.15.70:0/2942825072 conn(0x7efc200793f0 msgr2=0x7efc201010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:56.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.451+0000 7efc25bf4700 1 -- 172.21.15.70:0/2942825072 shutdown_connections 2024-06-04T04:53:56.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:56.451+0000 7efc25bf4700 1 -- 172.21.15.70:0/2942825072 wait complete. 2024-06-04T04:53:56.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:56 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2942825072' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-06-04T04:53:56.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:56 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2942825072' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-06-04T04:53:56.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:56 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2942825072' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-06-04T04:53:57.142 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-06-04T04:53:57.142 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-06-04T04:53:57.382 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:53:57.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:57 smithi070 ceph-mon[32631]: pgmap v298: 97 pgs: 97 active+clean; 584 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3.7 KiB/s wr, 13 op/s 2024-06-04T04:53:57.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:57 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2942825072' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-06-04T04:53:57.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:57 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T04:53:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:57 smithi098 ceph-mon[35569]: pgmap v298: 97 pgs: 97 active+clean; 584 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3.7 KiB/s wr, 13 op/s 2024-06-04T04:53:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:57 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2942825072' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-06-04T04:53:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:57 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T04:53:57.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:57 smithi112 ceph-mon[36453]: pgmap v298: 97 pgs: 97 active+clean; 584 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3.7 KiB/s wr, 13 op/s 2024-06-04T04:53:57.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:57 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2942825072' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-06-04T04:53:57.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:57 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T04:53:58.436 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 04:53:58 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T04:53:58.117+0000 7f03b194b700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-06-04T04:53:59.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.153+0000 7f987617e700 1 -- 172.21.15.70:0/2763757698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9870101f00 msgr2=0x7f9870102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:59.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.153+0000 7f987617e700 1 --2- 172.21.15.70:0/2763757698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9870101f00 0x7f9870102380 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9864009990 tx=0x7f9864009ca0 comp rx=0 tx=0).stop 2024-06-04T04:53:59.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.153+0000 7f987617e700 1 -- 172.21.15.70:0/2763757698 shutdown_connections 2024-06-04T04:53:59.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.153+0000 7f987617e700 1 --2- 172.21.15.70:0/2763757698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98701028c0 0x7f9870110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:59.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.153+0000 7f987617e700 1 --2- 172.21.15.70:0/2763757698 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9870108620 0x7f9870108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:59.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.153+0000 7f987617e700 1 --2- 172.21.15.70:0/2763757698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9870101f00 0x7f9870102380 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:59.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.153+0000 7f987617e700 1 -- 172.21.15.70:0/2763757698 >> 172.21.15.70:0/2763757698 conn(0x7f9870077780 msgr2=0x7f9870077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:59.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.153+0000 7f987617e700 1 -- 172.21.15.70:0/2763757698 shutdown_connections 2024-06-04T04:53:59.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.153+0000 7f987617e700 1 -- 172.21.15.70:0/2763757698 wait complete. 2024-06-04T04:53:59.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.154+0000 7f987617e700 1 Processor -- start 2024-06-04T04:53:59.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.154+0000 7f987617e700 1 -- start start 2024-06-04T04:53:59.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f987617e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9870101f00 0x7f987019f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:59.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f987617e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98701028c0 0x7f987019fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:59.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f987617e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9870108620 0x7f98701999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f987617e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9870113660 con 0x7f9870101f00 2024-06-04T04:53:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f987617e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f98701134e0 con 0x7f98701028c0 2024-06-04T04:53:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f987617e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f98701137e0 con 0x7f9870108620 2024-06-04T04:53:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f986f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9870101f00 0x7f987019f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f986f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9870101f00 0x7f987019f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46300/0 (socket says 172.21.15.70:46300) 2024-06-04T04:53:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f986f7fe700 1 -- 172.21.15.70:0/1142919804 learned_addr learned my addr 172.21.15.70:0/1142919804 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:53:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f986effd700 1 --2- 172.21.15.70:0/1142919804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98701028c0 0x7f987019fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f986ffff700 1 --2- 172.21.15.70:0/1142919804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9870108620 0x7f98701999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f986f7fe700 1 -- 172.21.15.70:0/1142919804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9870108620 msgr2=0x7f98701999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f986f7fe700 1 --2- 172.21.15.70:0/1142919804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9870108620 0x7f98701999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f986f7fe700 1 -- 172.21.15.70:0/1142919804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98701028c0 msgr2=0x7f987019fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f986f7fe700 1 --2- 172.21.15.70:0/1142919804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98701028c0 0x7f987019fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f986f7fe700 1 -- 172.21.15.70:0/1142919804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987019a120 con 0x7f9870101f00 2024-06-04T04:53:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f986f7fe700 1 --2- 172.21.15.70:0/1142919804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9870101f00 0x7f987019f800 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f986000b500 tx=0x7f986000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f986cff9700 1 -- 172.21.15.70:0/1142919804 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860015020 con 0x7f9870101f00 2024-06-04T04:53:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f987617e700 1 -- 172.21.15.70:0/1142919804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f987019a410 con 0x7f9870101f00 2024-06-04T04:53:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.155+0000 7f986cff9700 1 -- 172.21.15.70:0/1142919804 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9860011cc0 con 0x7f9870101f00 2024-06-04T04:53:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.156+0000 7f986cff9700 1 -- 172.21.15.70:0/1142919804 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860010c10 con 0x7f9870101f00 2024-06-04T04:53:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.156+0000 7f987617e700 1 -- 172.21.15.70:0/1142919804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98701a66d0 con 0x7f9870101f00 2024-06-04T04:53:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.157+0000 7f987617e700 1 -- 172.21.15.70:0/1142919804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f987006a230 con 0x7f9870101f00 2024-06-04T04:53:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.158+0000 7f986cff9700 1 -- 172.21.15.70:0/1142919804 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f9860018020 con 0x7f9870101f00 2024-06-04T04:53:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.158+0000 7f986cff9700 1 --2- 172.21.15.70:0/1142919804 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f98580b2880 0x7f98580b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:53:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.158+0000 7f986effd700 1 --2- 172.21.15.70:0/1142919804 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f98580b2880 0x7f98580b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:53:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.159+0000 7f986cff9700 1 -- 172.21.15.70:0/1142919804 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f98600e17e0 con 0x7f9870101f00 2024-06-04T04:53:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.159+0000 7f986effd700 1 --2- 172.21.15.70:0/1142919804 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f98580b2880 0x7f98580b4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f986400b5c0 tx=0x7f9864005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:53:59.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.163+0000 7f986cff9700 1 -- 172.21.15.70:0/1142919804 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f986001f020 con 0x7f9870101f00 2024-06-04T04:53:59.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.398+0000 7f987617e700 1 -- 172.21.15.70:0/1142919804 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f9870063980 con 0x7f98580b2880 2024-06-04T04:53:59.406 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:53:59 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:53:59.402+0000 7f7a710da700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:53:59.406 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:53:59 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:53:59.402+0000 7f7a710da700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:53:59.406 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:53:59 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:53:59.402+0000 7f7a710da700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:53:59.406 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:53:59 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:53:59.402+0000 7f7a710da700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:53:59.406 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:53:59 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:53:59.402+0000 7f7a710da700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:53:59.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.421+0000 7f986cff9700 1 -- 172.21.15.70:0/1142919804 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9870063980 con 0x7f98580b2880 2024-06-04T04:53:59.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.423+0000 7f987617e700 1 -- 172.21.15.70:0/1142919804 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f98580b2880 msgr2=0x7f98580b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:59.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.423+0000 7f987617e700 1 --2- 172.21.15.70:0/1142919804 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f98580b2880 0x7f98580b4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f986400b5c0 tx=0x7f9864005960 comp rx=0 tx=0).stop 2024-06-04T04:53:59.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.423+0000 7f987617e700 1 -- 172.21.15.70:0/1142919804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9870101f00 msgr2=0x7f987019f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:53:59.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.423+0000 7f987617e700 1 --2- 172.21.15.70:0/1142919804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9870101f00 0x7f987019f800 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f986000b500 tx=0x7f986000b810 comp rx=0 tx=0).stop 2024-06-04T04:53:59.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.423+0000 7f987617e700 1 -- 172.21.15.70:0/1142919804 shutdown_connections 2024-06-04T04:53:59.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.423+0000 7f987617e700 1 --2- 172.21.15.70:0/1142919804 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f98580b2880 0x7f98580b4d40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:59.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.423+0000 7f987617e700 1 --2- 172.21.15.70:0/1142919804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9870108620 0x7f98701999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:59.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.423+0000 7f987617e700 1 --2- 172.21.15.70:0/1142919804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9870101f00 0x7f987019f800 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:59.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.423+0000 7f987617e700 1 --2- 172.21.15.70:0/1142919804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98701028c0 0x7f987019fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:53:59.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.423+0000 7f987617e700 1 -- 172.21.15.70:0/1142919804 >> 172.21.15.70:0/1142919804 conn(0x7f9870077780 msgr2=0x7f9870100b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:53:59.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.424+0000 7f987617e700 1 -- 172.21.15.70:0/1142919804 shutdown_connections 2024-06-04T04:53:59.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:53:59.424+0000 7f987617e700 1 -- 172.21.15.70:0/1142919804 wait complete. 2024-06-04T04:53:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:53:59 smithi098 ceph-mon[35569]: pgmap v299: 97 pgs: 97 active+clean; 584 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 3.9 KiB/s wr, 17 op/s 2024-06-04T04:53:59.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:53:59 smithi070 ceph-mon[32631]: pgmap v299: 97 pgs: 97 active+clean; 584 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 3.9 KiB/s wr, 17 op/s 2024-06-04T04:53:59.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:53:59 smithi112 ceph-mon[36453]: pgmap v299: 97 pgs: 97 active+clean; 584 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 3.9 KiB/s wr, 17 op/s 2024-06-04T04:54:00.135 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-06-04T04:54:00.378 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:00.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:00 smithi112 ceph-mon[36453]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:54:00.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:00 smithi070 ceph-mon[32631]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:54:00.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:00 smithi098 ceph-mon[35569]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:54:01.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:01 smithi070 ceph-mon[32631]: pgmap v300: 97 pgs: 97 active+clean; 585 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.1 KiB/s wr, 19 op/s 2024-06-04T04:54:01.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:01 smithi070 ceph-mon[32631]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-06-04T04:54:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:01 smithi098 ceph-mon[35569]: pgmap v300: 97 pgs: 97 active+clean; 585 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.1 KiB/s wr, 19 op/s 2024-06-04T04:54:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:01 smithi098 ceph-mon[35569]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-06-04T04:54:01.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:01 smithi112 ceph-mon[36453]: pgmap v300: 97 pgs: 97 active+clean; 585 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.1 KiB/s wr, 19 op/s 2024-06-04T04:54:01.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:01 smithi112 ceph-mon[36453]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-06-04T04:54:02.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.140+0000 7f9ed44a4700 1 -- 172.21.15.70:0/4083292386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ecc102a80 msgr2=0x7f9ecc102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:02.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.140+0000 7f9ed44a4700 1 --2- 172.21.15.70:0/4083292386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ecc102a80 0x7f9ecc102e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9ec80099c0 tx=0x7f9ec8009cd0 comp rx=0 tx=0).stop 2024-06-04T04:54:02.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.140+0000 7f9ed44a4700 1 -- 172.21.15.70:0/4083292386 shutdown_connections 2024-06-04T04:54:02.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.140+0000 7f9ed44a4700 1 --2- 172.21.15.70:0/4083292386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ecc1077d0 0x7f9ecc10f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:02.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.140+0000 7f9ed44a4700 1 --2- 172.21.15.70:0/4083292386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ecc102a80 0x7f9ecc102e60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:02.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.140+0000 7f9ed44a4700 1 --2- 172.21.15.70:0/4083292386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ecc103430 0x7f9ecc1070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:02.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.140+0000 7f9ed44a4700 1 -- 172.21.15.70:0/4083292386 >> 172.21.15.70:0/4083292386 conn(0x7f9ecc0793f0 msgr2=0x7f9ecc079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:02.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.140+0000 7f9ed44a4700 1 -- 172.21.15.70:0/4083292386 shutdown_connections 2024-06-04T04:54:02.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.140+0000 7f9ed44a4700 1 -- 172.21.15.70:0/4083292386 wait complete. 2024-06-04T04:54:02.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.140+0000 7f9ed44a4700 1 Processor -- start 2024-06-04T04:54:02.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.140+0000 7f9ed44a4700 1 -- start start 2024-06-04T04:54:02.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed44a4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ecc102a80 0x7f9ecc19e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:02.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed44a4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ecc103430 0x7f9ecc19ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:02.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed44a4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ecc1077d0 0x7f9ecc1988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:02.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed44a4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc112510 con 0x7f9ecc102a80 2024-06-04T04:54:02.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed44a4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc112390 con 0x7f9ecc103430 2024-06-04T04:54:02.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed44a4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc112690 con 0x7f9ecc1077d0 2024-06-04T04:54:02.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed2240700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ecc102a80 0x7f9ecc19e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:02.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed2240700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ecc102a80 0x7f9ecc19e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46334/0 (socket says 172.21.15.70:46334) 2024-06-04T04:54:02.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed2240700 1 -- 172.21.15.70:0/920938358 learned_addr learned my addr 172.21.15.70:0/920938358 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:02.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed2a41700 1 --2- 172.21.15.70:0/920938358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ecc1077d0 0x7f9ecc1988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:02.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed1a3f700 1 --2- 172.21.15.70:0/920938358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ecc103430 0x7f9ecc19ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:02.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed2240700 1 -- 172.21.15.70:0/920938358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ecc1077d0 msgr2=0x7f9ecc1988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:02.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed2240700 1 --2- 172.21.15.70:0/920938358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ecc1077d0 0x7f9ecc1988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:02.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed2240700 1 -- 172.21.15.70:0/920938358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ecc103430 msgr2=0x7f9ecc19ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:02.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed2240700 1 --2- 172.21.15.70:0/920938358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ecc103430 0x7f9ecc19ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:02.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed2240700 1 -- 172.21.15.70:0/920938358 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ecc199010 con 0x7f9ecc102a80 2024-06-04T04:54:02.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed2a41700 1 --2- 172.21.15.70:0/920938358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ecc1077d0 0x7f9ecc1988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:54:02.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed1a3f700 1 --2- 172.21.15.70:0/920938358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ecc103430 0x7f9ecc19ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:54:02.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed2240700 1 --2- 172.21.15.70:0/920938358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ecc102a80 0x7f9ecc19e6f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8000c00 tx=0x7f9ec800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:02.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ec37fe700 1 -- 172.21.15.70:0/920938358 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8026070 con 0x7f9ecc102a80 2024-06-04T04:54:02.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ec37fe700 1 -- 172.21.15.70:0/920938358 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9ec80054c0 con 0x7f9ecc102a80 2024-06-04T04:54:02.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.141+0000 7f9ed44a4700 1 -- 172.21.15.70:0/920938358 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ecc1992a0 con 0x7f9ecc102a80 2024-06-04T04:54:02.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.142+0000 7f9ec37fe700 1 -- 172.21.15.70:0/920938358 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8004c20 con 0x7f9ecc102a80 2024-06-04T04:54:02.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.142+0000 7f9ed44a4700 1 -- 172.21.15.70:0/920938358 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ecc06bc10 con 0x7f9ecc102a80 2024-06-04T04:54:02.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.142+0000 7f9ed44a4700 1 -- 172.21.15.70:0/920938358 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ecc103b80 con 0x7f9ecc102a80 2024-06-04T04:54:02.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.144+0000 7f9ec37fe700 1 -- 172.21.15.70:0/920938358 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f9ec8029070 con 0x7f9ecc102a80 2024-06-04T04:54:02.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.144+0000 7f9ec37fe700 1 --2- 172.21.15.70:0/920938358 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f9eb80b4ba0 0x7f9eb80b7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:02.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.144+0000 7f9ed1a3f700 1 --2- 172.21.15.70:0/920938358 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f9eb80b4ba0 0x7f9eb80b7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:02.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.145+0000 7f9ec37fe700 1 -- 172.21.15.70:0/920938358 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f9ec80e3030 con 0x7f9ecc102a80 2024-06-04T04:54:02.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.145+0000 7f9ed1a3f700 1 --2- 172.21.15.70:0/920938358 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f9eb80b4ba0 0x7f9eb80b7060 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc005fd0 tx=0x7f9ebc005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:02.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.148+0000 7f9ec37fe700 1 -- 172.21.15.70:0/920938358 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9ec8030020 con 0x7f9ecc102a80 2024-06-04T04:54:02.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.385+0000 7f9ed44a4700 1 -- 172.21.15.70:0/920938358 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f9ecc06be70 con 0x7f9eb80b4ba0 2024-06-04T04:54:02.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.411+0000 7f9ec37fe700 1 -- 172.21.15.70:0/920938358 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9ecc06be70 con 0x7f9eb80b4ba0 2024-06-04T04:54:02.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.413+0000 7f9ed44a4700 1 -- 172.21.15.70:0/920938358 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f9eb80b4ba0 msgr2=0x7f9eb80b7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:02.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.413+0000 7f9ed44a4700 1 --2- 172.21.15.70:0/920938358 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f9eb80b4ba0 0x7f9eb80b7060 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc005fd0 tx=0x7f9ebc005dc0 comp rx=0 tx=0).stop 2024-06-04T04:54:02.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.413+0000 7f9ed44a4700 1 -- 172.21.15.70:0/920938358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ecc102a80 msgr2=0x7f9ecc19e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:02.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.413+0000 7f9ed44a4700 1 --2- 172.21.15.70:0/920938358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ecc102a80 0x7f9ecc19e6f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8000c00 tx=0x7f9ec800f740 comp rx=0 tx=0).stop 2024-06-04T04:54:02.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.413+0000 7f9ed44a4700 1 -- 172.21.15.70:0/920938358 shutdown_connections 2024-06-04T04:54:02.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.413+0000 7f9ed44a4700 1 --2- 172.21.15.70:0/920938358 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f9eb80b4ba0 0x7f9eb80b7060 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:02.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.413+0000 7f9ed44a4700 1 --2- 172.21.15.70:0/920938358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ecc1077d0 0x7f9ecc1988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:02.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.413+0000 7f9ed44a4700 1 --2- 172.21.15.70:0/920938358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ecc102a80 0x7f9ecc19e6f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:02.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.413+0000 7f9ed44a4700 1 --2- 172.21.15.70:0/920938358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ecc103430 0x7f9ecc19ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:02.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.413+0000 7f9ed44a4700 1 -- 172.21.15.70:0/920938358 >> 172.21.15.70:0/920938358 conn(0x7f9ecc0793f0 msgr2=0x7f9ecc100c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:02.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.414+0000 7f9ed44a4700 1 -- 172.21.15.70:0/920938358 shutdown_connections 2024-06-04T04:54:02.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:02.414+0000 7f9ed44a4700 1 -- 172.21.15.70:0/920938358 wait complete. 2024-06-04T04:54:03.143 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-06-04T04:54:03.383 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:03.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:03 smithi070 ceph-mon[32631]: pgmap v301: 97 pgs: 97 active+clean; 585 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.1 KiB/s wr, 20 op/s 2024-06-04T04:54:03.736 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:03 smithi070 ceph-mon[32631]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:54:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:03 smithi098 ceph-mon[35569]: pgmap v301: 97 pgs: 97 active+clean; 585 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.1 KiB/s wr, 20 op/s 2024-06-04T04:54:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:03 smithi098 ceph-mon[35569]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:54:03.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:03 smithi112 ceph-mon[36453]: pgmap v301: 97 pgs: 97 active+clean; 585 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.1 KiB/s wr, 20 op/s 2024-06-04T04:54:03.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:03 smithi112 ceph-mon[36453]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:54:05.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.053+0000 7f44fe222700 1 -- 172.21.15.70:0/342466267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f8103600 msgr2=0x7f44f8103a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:05.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.053+0000 7f44fe222700 1 --2- 172.21.15.70:0/342466267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f8103600 0x7f44f8103a80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f44e00099c0 tx=0x7f44e0009cd0 comp rx=0 tx=0).stop 2024-06-04T04:54:05.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.053+0000 7f44fe222700 1 -- 172.21.15.70:0/342466267 shutdown_connections 2024-06-04T04:54:05.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.053+0000 7f44fe222700 1 --2- 172.21.15.70:0/342466267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f8103fc0 0x7f44f81103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:05.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.053+0000 7f44fe222700 1 --2- 172.21.15.70:0/342466267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44f81095e0 0x7f44f81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:05.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.053+0000 7f44fe222700 1 --2- 172.21.15.70:0/342466267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f8103600 0x7f44f8103a80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:05.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.053+0000 7f44fe222700 1 -- 172.21.15.70:0/342466267 >> 172.21.15.70:0/342466267 conn(0x7f44f807a700 msgr2=0x7f44f807ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.053+0000 7f44fe222700 1 -- 172.21.15.70:0/342466267 shutdown_connections 2024-06-04T04:54:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.053+0000 7f44fe222700 1 -- 172.21.15.70:0/342466267 wait complete. 2024-06-04T04:54:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.053+0000 7f44fe222700 1 Processor -- start 2024-06-04T04:54:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.053+0000 7f44fe222700 1 -- start start 2024-06-04T04:54:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.054+0000 7f44fe222700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f8103600 0x7f44f819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.054+0000 7f44fe222700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44f8103fc0 0x7f44f819fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.054+0000 7f44fe222700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f81095e0 0x7f44f8199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.054+0000 7f44fe222700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f44f81135f0 con 0x7f44f81095e0 2024-06-04T04:54:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.054+0000 7f44fe222700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f44f8113470 con 0x7f44f8103600 2024-06-04T04:54:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.054+0000 7f44fe222700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f44f8113770 con 0x7f44f8103fc0 2024-06-04T04:54:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.054+0000 7f44f7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f81095e0 0x7f44f8199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.054+0000 7f44f7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f81095e0 0x7f44f8199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46348/0 (socket says 172.21.15.70:46348) 2024-06-04T04:54:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.054+0000 7f44f7fff700 1 -- 172.21.15.70:0/3090209146 learned_addr learned my addr 172.21.15.70:0/3090209146 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.054+0000 7f44eeffd700 1 --2- 172.21.15.70:0/3090209146 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44f8103fc0 0x7f44f819fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.054+0000 7f44f77fe700 1 --2- 172.21.15.70:0/3090209146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f8103600 0x7f44f819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.054+0000 7f44f7fff700 1 -- 172.21.15.70:0/3090209146 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44f8103fc0 msgr2=0x7f44f819fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.054+0000 7f44f7fff700 1 --2- 172.21.15.70:0/3090209146 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44f8103fc0 0x7f44f819fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.054+0000 7f44f7fff700 1 -- 172.21.15.70:0/3090209146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f8103600 msgr2=0x7f44f819f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.054+0000 7f44f7fff700 1 --2- 172.21.15.70:0/3090209146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f8103600 0x7f44f819f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.054+0000 7f44f7fff700 1 -- 172.21.15.70:0/3090209146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f819a030 con 0x7f44f81095e0 2024-06-04T04:54:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.054+0000 7f44f7fff700 1 --2- 172.21.15.70:0/3090209146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f81095e0 0x7f44f8199850 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f44f0007d70 tx=0x7f44f000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.055+0000 7f44f57fa700 1 -- 172.21.15.70:0/3090209146 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f0016020 con 0x7f44f81095e0 2024-06-04T04:54:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.055+0000 7f44fe222700 1 -- 172.21.15.70:0/3090209146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f819a320 con 0x7f44f81095e0 2024-06-04T04:54:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.055+0000 7f44f57fa700 1 -- 172.21.15.70:0/3090209146 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f44f000ad40 con 0x7f44f81095e0 2024-06-04T04:54:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.055+0000 7f44f57fa700 1 -- 172.21.15.70:0/3090209146 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f00115e0 con 0x7f44f81095e0 2024-06-04T04:54:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.055+0000 7f44fe222700 1 -- 172.21.15.70:0/3090209146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f81a64e0 con 0x7f44f81095e0 2024-06-04T04:54:05.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.056+0000 7f44fe222700 1 -- 172.21.15.70:0/3090209146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44f8104c50 con 0x7f44f81095e0 2024-06-04T04:54:05.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.057+0000 7f44f57fa700 1 -- 172.21.15.70:0/3090209146 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f44f0019020 con 0x7f44f81095e0 2024-06-04T04:54:05.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.058+0000 7f44f57fa700 1 --2- 172.21.15.70:0/3090209146 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f44e40b2910 0x7f44e40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:05.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.058+0000 7f44f77fe700 1 --2- 172.21.15.70:0/3090209146 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f44e40b2910 0x7f44e40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:05.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.058+0000 7f44f57fa700 1 -- 172.21.15.70:0/3090209146 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f44f00e2580 con 0x7f44f81095e0 2024-06-04T04:54:05.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.058+0000 7f44f77fe700 1 --2- 172.21.15.70:0/3090209146 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f44e40b2910 0x7f44e40b4dd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f44e8005950 tx=0x7f44e80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:05.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.061+0000 7f44f57fa700 1 -- 172.21.15.70:0/3090209146 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f44f0020020 con 0x7f44f81095e0 2024-06-04T04:54:05.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.300+0000 7f44fe222700 1 -- 172.21.15.70:0/3090209146 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f44f8063980 con 0x7f44e40b2910 2024-06-04T04:54:05.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.305+0000 7f44f57fa700 1 -- 172.21.15.70:0/3090209146 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f44f8063980 con 0x7f44e40b2910 2024-06-04T04:54:05.307 INFO:teuthology.orchestra.run.smithi070.stdout:/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b 2024-06-04T04:54:05.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.308+0000 7f44fe222700 1 -- 172.21.15.70:0/3090209146 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f44e40b2910 msgr2=0x7f44e40b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:05.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.308+0000 7f44fe222700 1 --2- 172.21.15.70:0/3090209146 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f44e40b2910 0x7f44e40b4dd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f44e8005950 tx=0x7f44e80058e0 comp rx=0 tx=0).stop 2024-06-04T04:54:05.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.308+0000 7f44fe222700 1 -- 172.21.15.70:0/3090209146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f81095e0 msgr2=0x7f44f8199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:05.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.308+0000 7f44fe222700 1 --2- 172.21.15.70:0/3090209146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f81095e0 0x7f44f8199850 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f44f0007d70 tx=0x7f44f000a340 comp rx=0 tx=0).stop 2024-06-04T04:54:05.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.308+0000 7f44fe222700 1 -- 172.21.15.70:0/3090209146 shutdown_connections 2024-06-04T04:54:05.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.308+0000 7f44fe222700 1 --2- 172.21.15.70:0/3090209146 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f44e40b2910 0x7f44e40b4dd0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:05.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.308+0000 7f44fe222700 1 --2- 172.21.15.70:0/3090209146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f81095e0 0x7f44f8199850 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:05.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.308+0000 7f44fe222700 1 --2- 172.21.15.70:0/3090209146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f8103600 0x7f44f819f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:05.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.308+0000 7f44fe222700 1 --2- 172.21.15.70:0/3090209146 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44f8103fc0 0x7f44f819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:05.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.308+0000 7f44fe222700 1 -- 172.21.15.70:0/3090209146 >> 172.21.15.70:0/3090209146 conn(0x7f44f807a700 msgr2=0x7f44f8102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:05.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.308+0000 7f44fe222700 1 -- 172.21.15.70:0/3090209146 shutdown_connections 2024-06-04T04:54:05.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:05.308+0000 7f44fe222700 1 -- 172.21.15.70:0/3090209146 wait complete. 2024-06-04T04:54:05.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:05 smithi070 ceph-mon[32631]: pgmap v302: 97 pgs: 97 active+clean; 605 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.2 KiB/s wr, 23 op/s 2024-06-04T04:54:05.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:05 smithi098 ceph-mon[35569]: pgmap v302: 97 pgs: 97 active+clean; 605 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.2 KiB/s wr, 23 op/s 2024-06-04T04:54:05.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:05 smithi112 ceph-mon[36453]: pgmap v302: 97 pgs: 97 active+clean; 605 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.2 KiB/s wr, 23 op/s 2024-06-04T04:54:06.028 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-06-04T04:54:06.267 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:06.640 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:06 smithi070 ceph-mon[32631]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:54:06.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:06 smithi098 ceph-mon[35569]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:54:06.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:06 smithi112 ceph-mon[36453]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:54:07.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:07 smithi070 ceph-mon[32631]: pgmap v303: 97 pgs: 97 active+clean; 605 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 5.0 KiB/s wr, 19 op/s 2024-06-04T04:54:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:07 smithi098 ceph-mon[35569]: pgmap v303: 97 pgs: 97 active+clean; 605 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 5.0 KiB/s wr, 19 op/s 2024-06-04T04:54:07.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:07 smithi112 ceph-mon[36453]: pgmap v303: 97 pgs: 97 active+clean; 605 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 5.0 KiB/s wr, 19 op/s 2024-06-04T04:54:08.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.182+0000 7f62b700f700 1 -- 172.21.15.70:0/1844543050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b01095c0 msgr2=0x7f62b01099a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:08.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.182+0000 7f62b700f700 1 --2- 172.21.15.70:0/1844543050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b01095c0 0x7f62b01099a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6298009990 tx=0x7f6298009ca0 comp rx=0 tx=0).stop 2024-06-04T04:54:08.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.182+0000 7f62b700f700 1 -- 172.21.15.70:0/1844543050 shutdown_connections 2024-06-04T04:54:08.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.182+0000 7f62b700f700 1 --2- 172.21.15.70:0/1844543050 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b006c5d0 0x7f62b01103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:08.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.182+0000 7f62b700f700 1 --2- 172.21.15.70:0/1844543050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b01095c0 0x7f62b01099a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:08.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.182+0000 7f62b700f700 1 --2- 172.21.15.70:0/1844543050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62b006bc10 0x7f62b006c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:08.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.182+0000 7f62b700f700 1 -- 172.21.15.70:0/1844543050 >> 172.21.15.70:0/1844543050 conn(0x7f62b00793f0 msgr2=0x7f62b0079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:08.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.182+0000 7f62b700f700 1 -- 172.21.15.70:0/1844543050 shutdown_connections 2024-06-04T04:54:08.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.182+0000 7f62b700f700 1 -- 172.21.15.70:0/1844543050 wait complete. 2024-06-04T04:54:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62b700f700 1 Processor -- start 2024-06-04T04:54:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62b700f700 1 -- start start 2024-06-04T04:54:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62b700f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b006bc10 0x7f62b0074e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62b700f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b006c5d0 0x7f62b006df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62b700f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62b01095c0 0x7f62b006e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62b700f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62b0113680 con 0x7f62b006c5d0 2024-06-04T04:54:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62b700f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62b0113500 con 0x7f62b01095c0 2024-06-04T04:54:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62b700f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f62b0113800 con 0x7f62b006bc10 2024-06-04T04:54:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62affff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b006c5d0 0x7f62b006df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62affff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b006c5d0 0x7f62b006df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33156/0 (socket says 172.21.15.70:33156) 2024-06-04T04:54:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62affff700 1 -- 172.21.15.70:0/2858393763 learned_addr learned my addr 172.21.15.70:0/2858393763 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62b4dab700 1 --2- 172.21.15.70:0/2858393763 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b006bc10 0x7f62b0074e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62b55ac700 1 --2- 172.21.15.70:0/2858393763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62b01095c0 0x7f62b006e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62affff700 1 -- 172.21.15.70:0/2858393763 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b006bc10 msgr2=0x7f62b0074e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62affff700 1 --2- 172.21.15.70:0/2858393763 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b006bc10 0x7f62b0074e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62affff700 1 -- 172.21.15.70:0/2858393763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62b01095c0 msgr2=0x7f62b006e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62affff700 1 --2- 172.21.15.70:0/2858393763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62b01095c0 0x7f62b006e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62affff700 1 -- 172.21.15.70:0/2858393763 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b006eb80 con 0x7f62b006c5d0 2024-06-04T04:54:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62b4dab700 1 --2- 172.21.15.70:0/2858393763 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b006bc10 0x7f62b0074e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:54:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62b55ac700 1 --2- 172.21.15.70:0/2858393763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62b01095c0 0x7f62b006e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:54:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.183+0000 7f62affff700 1 --2- 172.21.15.70:0/2858393763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b006c5d0 0x7f62b006df00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f62a000ba50 tx=0x7f62a000be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.184+0000 7f62adffb700 1 -- 172.21.15.70:0/2858393763 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a0019070 con 0x7f62b006c5d0 2024-06-04T04:54:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.184+0000 7f62adffb700 1 -- 172.21.15.70:0/2858393763 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f62a0014440 con 0x7f62b006c5d0 2024-06-04T04:54:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.184+0000 7f62b700f700 1 -- 172.21.15.70:0/2858393763 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62b00729a0 con 0x7f62b006c5d0 2024-06-04T04:54:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.184+0000 7f62adffb700 1 -- 172.21.15.70:0/2858393763 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a00075d0 con 0x7f62b006c5d0 2024-06-04T04:54:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.184+0000 7f62b700f700 1 -- 172.21.15.70:0/2858393763 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62b0072ee0 con 0x7f62b006c5d0 2024-06-04T04:54:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.185+0000 7f62b700f700 1 -- 172.21.15.70:0/2858393763 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62b006b4c0 con 0x7f62b006c5d0 2024-06-04T04:54:08.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.186+0000 7f62adffb700 1 -- 172.21.15.70:0/2858393763 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f62a001c070 con 0x7f62b006c5d0 2024-06-04T04:54:08.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.187+0000 7f62adffb700 1 --2- 172.21.15.70:0/2858393763 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f629c0b2910 0x7f629c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:08.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.187+0000 7f62b4dab700 1 --2- 172.21.15.70:0/2858393763 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f629c0b2910 0x7f629c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:08.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.187+0000 7f62adffb700 1 -- 172.21.15.70:0/2858393763 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f62a00e17a0 con 0x7f62b006c5d0 2024-06-04T04:54:08.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.187+0000 7f62b4dab700 1 --2- 172.21.15.70:0/2858393763 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f629c0b2910 0x7f629c0b4dd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6298003760 tx=0x7f629801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:08.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.191+0000 7f62adffb700 1 -- 172.21.15.70:0/2858393763 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f62a0051df0 con 0x7f62b006c5d0 2024-06-04T04:54:08.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.433+0000 7f62b700f700 1 -- 172.21.15.70:0/2858393763 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f62b00731a0 con 0x7f629c0b2910 2024-06-04T04:54:08.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.458+0000 7f62adffb700 1 -- 172.21.15.70:0/2858393763 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f62b00731a0 con 0x7f629c0b2910 2024-06-04T04:54:08.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.460+0000 7f62b700f700 1 -- 172.21.15.70:0/2858393763 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f629c0b2910 msgr2=0x7f629c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:08.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.460+0000 7f62b700f700 1 --2- 172.21.15.70:0/2858393763 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f629c0b2910 0x7f629c0b4dd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6298003760 tx=0x7f629801a040 comp rx=0 tx=0).stop 2024-06-04T04:54:08.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.460+0000 7f62b700f700 1 -- 172.21.15.70:0/2858393763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b006c5d0 msgr2=0x7f62b006df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:08.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.460+0000 7f62b700f700 1 --2- 172.21.15.70:0/2858393763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b006c5d0 0x7f62b006df00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f62a000ba50 tx=0x7f62a000be10 comp rx=0 tx=0).stop 2024-06-04T04:54:08.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.460+0000 7f62b700f700 1 -- 172.21.15.70:0/2858393763 shutdown_connections 2024-06-04T04:54:08.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.460+0000 7f62b700f700 1 --2- 172.21.15.70:0/2858393763 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f629c0b2910 0x7f629c0b4dd0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:08.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.460+0000 7f62b700f700 1 --2- 172.21.15.70:0/2858393763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62b01095c0 0x7f62b006e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:08.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.460+0000 7f62b700f700 1 --2- 172.21.15.70:0/2858393763 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b006bc10 0x7f62b0074e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:08.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.460+0000 7f62b700f700 1 --2- 172.21.15.70:0/2858393763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b006c5d0 0x7f62b006df00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:08.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.460+0000 7f62b700f700 1 -- 172.21.15.70:0/2858393763 >> 172.21.15.70:0/2858393763 conn(0x7f62b00793f0 msgr2=0x7f62b0101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:08.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.460+0000 7f62b700f700 1 -- 172.21.15.70:0/2858393763 shutdown_connections 2024-06-04T04:54:08.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:08.460+0000 7f62b700f700 1 -- 172.21.15.70:0/2858393763 wait complete. 2024-06-04T04:54:09.112 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-06-04T04:54:09.357 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:09.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:09 smithi070 ceph-mon[32631]: pgmap v304: 97 pgs: 97 active+clean; 610 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 2.8 KiB/s wr, 14 op/s 2024-06-04T04:54:09.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:09 smithi070 ceph-mon[32631]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:54:09.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:09 smithi098 ceph-mon[35569]: pgmap v304: 97 pgs: 97 active+clean; 610 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 2.8 KiB/s wr, 14 op/s 2024-06-04T04:54:09.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:09 smithi098 ceph-mon[35569]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:54:09.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:09 smithi112 ceph-mon[36453]: pgmap v304: 97 pgs: 97 active+clean; 610 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 2.8 KiB/s wr, 14 op/s 2024-06-04T04:54:09.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:09 smithi112 ceph-mon[36453]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:54:11.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.100+0000 7f6458cb7700 1 -- 172.21.15.70:0/2586463115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6454102810 msgr2=0x7f6454106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:11.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.100+0000 7f6458cb7700 1 --2- 172.21.15.70:0/2586463115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6454102810 0x7f6454106810 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f644800aad0 tx=0x7f644800ade0 comp rx=0 tx=0).stop 2024-06-04T04:54:11.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.100+0000 7f6458cb7700 1 -- 172.21.15.70:0/2586463115 shutdown_connections 2024-06-04T04:54:11.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.100+0000 7f6458cb7700 1 --2- 172.21.15.70:0/2586463115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6454106ef0 0x7f645410eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:11.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.100+0000 7f6458cb7700 1 --2- 172.21.15.70:0/2586463115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6454101e60 0x7f6454102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:11.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.100+0000 7f6458cb7700 1 --2- 172.21.15.70:0/2586463115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6454102810 0x7f6454106810 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:11.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.100+0000 7f6458cb7700 1 -- 172.21.15.70:0/2586463115 >> 172.21.15.70:0/2586463115 conn(0x7f6454078ab0 msgr2=0x7f64540ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:11.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.100+0000 7f6458cb7700 1 -- 172.21.15.70:0/2586463115 shutdown_connections 2024-06-04T04:54:11.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.100+0000 7f6458cb7700 1 -- 172.21.15.70:0/2586463115 wait complete. 2024-06-04T04:54:11.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.101+0000 7f6458cb7700 1 Processor -- start 2024-06-04T04:54:11.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.101+0000 7f6458cb7700 1 -- start start 2024-06-04T04:54:11.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6458cb7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6454101e60 0x7f645419f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:11.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6458cb7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6454102810 0x7f645419fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:11.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6458cb7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6454106ef0 0x7f6454199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:11.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6458cb7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6454111460 con 0x7f6454106ef0 2024-06-04T04:54:11.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6458cb7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f64541112e0 con 0x7f6454101e60 2024-06-04T04:54:11.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6458cb7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f64541115e0 con 0x7f6454102810 2024-06-04T04:54:11.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6452d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6454106ef0 0x7f6454199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:11.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6452d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6454106ef0 0x7f6454199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33174/0 (socket says 172.21.15.70:33174) 2024-06-04T04:54:11.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6452d9d700 1 -- 172.21.15.70:0/93384913 learned_addr learned my addr 172.21.15.70:0/93384913 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:11.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f645259c700 1 --2- 172.21.15.70:0/93384913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6454101e60 0x7f645419f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:11.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6451d9b700 1 --2- 172.21.15.70:0/93384913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6454102810 0x7f645419fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:11.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6452d9d700 1 -- 172.21.15.70:0/93384913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6454102810 msgr2=0x7f645419fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:11.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6452d9d700 1 --2- 172.21.15.70:0/93384913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6454102810 0x7f645419fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:11.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6452d9d700 1 -- 172.21.15.70:0/93384913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6454101e60 msgr2=0x7f645419f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:11.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6452d9d700 1 --2- 172.21.15.70:0/93384913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6454101e60 0x7f645419f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:11.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6452d9d700 1 -- 172.21.15.70:0/93384913 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f645419a200 con 0x7f6454106ef0 2024-06-04T04:54:11.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6451d9b700 1 --2- 172.21.15.70:0/93384913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6454102810 0x7f645419fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:54:11.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6452d9d700 1 --2- 172.21.15.70:0/93384913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6454106ef0 0x7f6454199aa0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f644c007ee0 tx=0x7f644c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:11.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f64437fe700 1 -- 172.21.15.70:0/93384913 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644c00dbd0 con 0x7f6454106ef0 2024-06-04T04:54:11.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6458cb7700 1 -- 172.21.15.70:0/93384913 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f645419a4f0 con 0x7f6454106ef0 2024-06-04T04:54:11.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f64437fe700 1 -- 172.21.15.70:0/93384913 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f644c00dd70 con 0x7f6454106ef0 2024-06-04T04:54:11.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f64437fe700 1 -- 172.21.15.70:0/93384913 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644c0058d0 con 0x7f6454106ef0 2024-06-04T04:54:11.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.102+0000 7f6458cb7700 1 -- 172.21.15.70:0/93384913 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64541a6760 con 0x7f6454106ef0 2024-06-04T04:54:11.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.103+0000 7f6458cb7700 1 -- 172.21.15.70:0/93384913 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f645406a230 con 0x7f6454106ef0 2024-06-04T04:54:11.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.104+0000 7f64437fe700 1 -- 172.21.15.70:0/93384913 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f644c01a020 con 0x7f6454106ef0 2024-06-04T04:54:11.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.105+0000 7f64437fe700 1 --2- 172.21.15.70:0/93384913 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f643c0b2910 0x7f643c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:11.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.105+0000 7f645259c700 1 --2- 172.21.15.70:0/93384913 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f643c0b2910 0x7f643c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:11.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.105+0000 7f64437fe700 1 -- 172.21.15.70:0/93384913 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f644c0e19d0 con 0x7f6454106ef0 2024-06-04T04:54:11.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.106+0000 7f645259c700 1 --2- 172.21.15.70:0/93384913 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f643c0b2910 0x7f643c0b4dd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f64541085c0 tx=0x7f6444009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:11.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.110+0000 7f64437fe700 1 -- 172.21.15.70:0/93384913 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f644c010070 con 0x7f6454106ef0 2024-06-04T04:54:11.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.352+0000 7f6458cb7700 1 -- 172.21.15.70:0/93384913 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f6454063980 con 0x7f643c0b2910 2024-06-04T04:54:11.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.357+0000 7f64437fe700 1 -- 172.21.15.70:0/93384913 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f6454063980 con 0x7f643c0b2910 2024-06-04T04:54:11.361 INFO:teuthology.orchestra.run.smithi070.stdout:/volumes/qa/sv_1/b5ee1386-9dd6-43c8-b377-703fc03c0249 2024-06-04T04:54:11.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.360+0000 7f6458cb7700 1 -- 172.21.15.70:0/93384913 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f643c0b2910 msgr2=0x7f643c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:11.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.360+0000 7f6458cb7700 1 --2- 172.21.15.70:0/93384913 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f643c0b2910 0x7f643c0b4dd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f64541085c0 tx=0x7f6444009190 comp rx=0 tx=0).stop 2024-06-04T04:54:11.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.360+0000 7f6458cb7700 1 -- 172.21.15.70:0/93384913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6454106ef0 msgr2=0x7f6454199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:11.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.360+0000 7f6458cb7700 1 --2- 172.21.15.70:0/93384913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6454106ef0 0x7f6454199aa0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f644c007ee0 tx=0x7f644c00d3b0 comp rx=0 tx=0).stop 2024-06-04T04:54:11.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.360+0000 7f6458cb7700 1 -- 172.21.15.70:0/93384913 shutdown_connections 2024-06-04T04:54:11.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.360+0000 7f6458cb7700 1 --2- 172.21.15.70:0/93384913 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f643c0b2910 0x7f643c0b4dd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:11.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.360+0000 7f6458cb7700 1 --2- 172.21.15.70:0/93384913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6454106ef0 0x7f6454199aa0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:11.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.360+0000 7f6458cb7700 1 --2- 172.21.15.70:0/93384913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6454101e60 0x7f645419f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:11.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.360+0000 7f6458cb7700 1 --2- 172.21.15.70:0/93384913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6454102810 0x7f645419fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:11.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.360+0000 7f6458cb7700 1 -- 172.21.15.70:0/93384913 >> 172.21.15.70:0/93384913 conn(0x7f6454078ab0 msgr2=0x7f64540ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:11.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.360+0000 7f6458cb7700 1 -- 172.21.15.70:0/93384913 shutdown_connections 2024-06-04T04:54:11.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:11.360+0000 7f6458cb7700 1 -- 172.21.15.70:0/93384913 wait complete. 2024-06-04T04:54:11.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:11 smithi070 ceph-mon[32631]: pgmap v305: 97 pgs: 97 active+clean; 610 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-06-04T04:54:11.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:11 smithi098 ceph-mon[35569]: pgmap v305: 97 pgs: 97 active+clean; 610 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-06-04T04:54:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:11 smithi112 ceph-mon[36453]: pgmap v305: 97 pgs: 97 active+clean; 610 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-06-04T04:54:12.073 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:54:12.313 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:12.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:12 smithi098 ceph-mon[35569]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:54:12.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:12 smithi070 ceph-mon[32631]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:54:12.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:12 smithi112 ceph-mon[36453]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:54:13.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:13 smithi098 ceph-mon[35569]: pgmap v306: 97 pgs: 97 active+clean; 610 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-06-04T04:54:13.751 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:13 smithi070 ceph-mon[32631]: pgmap v306: 97 pgs: 97 active+clean; 610 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-06-04T04:54:13.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:13 smithi112 ceph-mon[36453]: pgmap v306: 97 pgs: 97 active+clean; 610 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-06-04T04:54:14.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:13.998+0000 7f69e54f7700 1 -- 172.21.15.70:0/691185158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e01037f0 msgr2=0x7f69e0103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:14.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:13.998+0000 7f69e54f7700 1 --2- 172.21.15.70:0/691185158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e01037f0 0x7f69e0103bd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f69d00099c0 tx=0x7f69d0009cd0 comp rx=0 tx=0).stop 2024-06-04T04:54:14.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:13.998+0000 7f69e54f7700 1 -- 172.21.15.70:0/691185158 shutdown_connections 2024-06-04T04:54:14.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:13.998+0000 7f69e54f7700 1 --2- 172.21.15.70:0/691185158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e01088f0 0x7f69e0110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:14.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:13.998+0000 7f69e54f7700 1 --2- 172.21.15.70:0/691185158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e01037f0 0x7f69e0103bd0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:14.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:13.998+0000 7f69e54f7700 1 --2- 172.21.15.70:0/691185158 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69e01041a0 0x7f69e01081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:14.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:13.998+0000 7f69e54f7700 1 -- 172.21.15.70:0/691185158 >> 172.21.15.70:0/691185158 conn(0x7f69e007a6a0 msgr2=0x7f69e007aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:14.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:13.998+0000 7f69e54f7700 1 -- 172.21.15.70:0/691185158 shutdown_connections 2024-06-04T04:54:14.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:13.998+0000 7f69e54f7700 1 -- 172.21.15.70:0/691185158 wait complete. 2024-06-04T04:54:14.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:13.999+0000 7f69e54f7700 1 Processor -- start 2024-06-04T04:54:14.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:13.999+0000 7f69e54f7700 1 -- start start 2024-06-04T04:54:14.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69e54f7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e01037f0 0x7f69e019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:14.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69e54f7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69e01041a0 0x7f69e019fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:14.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69e54f7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e01088f0 0x7f69e0199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:14.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69e54f7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f69e0113590 con 0x7f69e01037f0 2024-06-04T04:54:14.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69e54f7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f69e0113410 con 0x7f69e01088f0 2024-06-04T04:54:14.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69e54f7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f69e0113710 con 0x7f69e01041a0 2024-06-04T04:54:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69deffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e01037f0 0x7f69e019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69deffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e01037f0 0x7f69e019f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33200/0 (socket says 172.21.15.70:33200) 2024-06-04T04:54:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69deffd700 1 -- 172.21.15.70:0/2591083016 learned_addr learned my addr 172.21.15.70:0/2591083016 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69de7fc700 1 --2- 172.21.15.70:0/2591083016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69e01041a0 0x7f69e019fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69deffd700 1 -- 172.21.15.70:0/2591083016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69e01041a0 msgr2=0x7f69e019fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69deffd700 1 --2- 172.21.15.70:0/2591083016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69e01041a0 0x7f69e019fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69df7fe700 1 --2- 172.21.15.70:0/2591083016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e01088f0 0x7f69e0199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69deffd700 1 -- 172.21.15.70:0/2591083016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e01088f0 msgr2=0x7f69e0199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69deffd700 1 --2- 172.21.15.70:0/2591083016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e01088f0 0x7f69e0199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69deffd700 1 -- 172.21.15.70:0/2591083016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69e0199fd0 con 0x7f69e01037f0 2024-06-04T04:54:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69df7fe700 1 --2- 172.21.15.70:0/2591083016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e01088f0 0x7f69e0199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:54:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69deffd700 1 --2- 172.21.15.70:0/2591083016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e01037f0 0x7f69e019f6b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f69d0000c00 tx=0x7f69d000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69c7fff700 1 -- 172.21.15.70:0/2591083016 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d0026070 con 0x7f69e01037f0 2024-06-04T04:54:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69e54f7700 1 -- 172.21.15.70:0/2591083016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69e019a260 con 0x7f69e01037f0 2024-06-04T04:54:14.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.001+0000 7f69e54f7700 1 -- 172.21.15.70:0/2591083016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69e01a6580 con 0x7f69e01037f0 2024-06-04T04:54:14.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.000+0000 7f69c7fff700 1 -- 172.21.15.70:0/2591083016 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69d00054c0 con 0x7f69e01037f0 2024-06-04T04:54:14.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.001+0000 7f69c7fff700 1 -- 172.21.15.70:0/2591083016 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d0004c20 con 0x7f69e01037f0 2024-06-04T04:54:14.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.002+0000 7f69e54f7700 1 -- 172.21.15.70:0/2591083016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69e006a230 con 0x7f69e01037f0 2024-06-04T04:54:14.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.003+0000 7f69c7fff700 1 -- 172.21.15.70:0/2591083016 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f69d0029070 con 0x7f69e01037f0 2024-06-04T04:54:14.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.004+0000 7f69c7fff700 1 --2- 172.21.15.70:0/2591083016 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f69c80b2910 0x7f69c80b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:14.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.004+0000 7f69de7fc700 1 --2- 172.21.15.70:0/2591083016 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f69c80b2910 0x7f69c80b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:14.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.004+0000 7f69c7fff700 1 -- 172.21.15.70:0/2591083016 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f69d00e3110 con 0x7f69e01037f0 2024-06-04T04:54:14.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.004+0000 7f69de7fc700 1 --2- 172.21.15.70:0/2591083016 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f69c80b2910 0x7f69c80b4dd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f69d4005fd0 tx=0x7f69d400a540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:14.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.007+0000 7f69c7fff700 1 -- 172.21.15.70:0/2591083016 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f69d0030020 con 0x7f69e01037f0 2024-06-04T04:54:14.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.254+0000 7f69e54f7700 1 -- 172.21.15.70:0/2591083016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69e002e560 con 0x7f69e01037f0 2024-06-04T04:54:14.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.255+0000 7f69c7fff700 1 -- 172.21.15.70:0/2591083016 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11929 (secure 0 0 0) 0x7f69d0013070 con 0x7f69e01037f0 2024-06-04T04:54:14.258 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:54:14.259 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:53:56.441682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T04:54:14.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.258+0000 7f69e54f7700 1 -- 172.21.15.70:0/2591083016 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f69c80b2910 msgr2=0x7f69c80b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:14.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.258+0000 7f69e54f7700 1 --2- 172.21.15.70:0/2591083016 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f69c80b2910 0x7f69c80b4dd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f69d4005fd0 tx=0x7f69d400a540 comp rx=0 tx=0).stop 2024-06-04T04:54:14.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.258+0000 7f69e54f7700 1 -- 172.21.15.70:0/2591083016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e01037f0 msgr2=0x7f69e019f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:14.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.258+0000 7f69e54f7700 1 --2- 172.21.15.70:0/2591083016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e01037f0 0x7f69e019f6b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f69d0000c00 tx=0x7f69d000f740 comp rx=0 tx=0).stop 2024-06-04T04:54:14.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.258+0000 7f69e54f7700 1 -- 172.21.15.70:0/2591083016 shutdown_connections 2024-06-04T04:54:14.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.258+0000 7f69e54f7700 1 --2- 172.21.15.70:0/2591083016 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f69c80b2910 0x7f69c80b4dd0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:14.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.258+0000 7f69e54f7700 1 --2- 172.21.15.70:0/2591083016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e01088f0 0x7f69e0199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:14.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.258+0000 7f69e54f7700 1 --2- 172.21.15.70:0/2591083016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e01037f0 0x7f69e019f6b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:14.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.258+0000 7f69e54f7700 1 --2- 172.21.15.70:0/2591083016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69e01041a0 0x7f69e019fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:14.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.258+0000 7f69e54f7700 1 -- 172.21.15.70:0/2591083016 >> 172.21.15.70:0/2591083016 conn(0x7f69e007a6a0 msgr2=0x7f69e01018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:14.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.258+0000 7f69e54f7700 1 -- 172.21.15.70:0/2591083016 shutdown_connections 2024-06-04T04:54:14.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:14.258+0000 7f69e54f7700 1 -- 172.21.15.70:0/2591083016 wait complete. 2024-06-04T04:54:14.263 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 28 2024-06-04T04:54:14.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:14 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2591083016' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:54:14.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:14 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2591083016' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:54:14.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:14 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2591083016' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:54:14.973 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-04T04:54:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:15.586 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:15 smithi070 ceph-mon[32631]: pgmap v307: 97 pgs: 97 active+clean; 627 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 4.4 KiB/s wr, 13 op/s 2024-06-04T04:54:15.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:15 smithi098 ceph-mon[35569]: pgmap v307: 97 pgs: 97 active+clean; 627 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 4.4 KiB/s wr, 13 op/s 2024-06-04T04:54:15.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:15 smithi112 ceph-mon[36453]: pgmap v307: 97 pgs: 97 active+clean; 627 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 4.4 KiB/s wr, 13 op/s 2024-06-04T04:54:16.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.890+0000 7f0a25b03700 1 -- 172.21.15.70:0/439270678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a20108520 msgr2=0x7f0a20108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:16.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.890+0000 7f0a25b03700 1 --2- 172.21.15.70:0/439270678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a20108520 0x7f0a20108900 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009990 tx=0x7f0a10009ca0 comp rx=0 tx=0).stop 2024-06-04T04:54:16.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.890+0000 7f0a25b03700 1 -- 172.21.15.70:0/439270678 shutdown_connections 2024-06-04T04:54:16.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.890+0000 7f0a25b03700 1 --2- 172.21.15.70:0/439270678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a20102e00 0x7f0a2010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:16.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.890+0000 7f0a25b03700 1 --2- 172.21.15.70:0/439270678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a20108520 0x7f0a20108900 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:16.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.890+0000 7f0a25b03700 1 --2- 172.21.15.70:0/439270678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a20102440 0x7f0a201028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:16.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.890+0000 7f0a25b03700 1 -- 172.21.15.70:0/439270678 >> 172.21.15.70:0/439270678 conn(0x7f0a200793f0 msgr2=0x7f0a20079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:16.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.890+0000 7f0a25b03700 1 -- 172.21.15.70:0/439270678 shutdown_connections 2024-06-04T04:54:16.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.890+0000 7f0a25b03700 1 -- 172.21.15.70:0/439270678 wait complete. 2024-06-04T04:54:16.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a25b03700 1 Processor -- start 2024-06-04T04:54:16.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a25b03700 1 -- start start 2024-06-04T04:54:16.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a25b03700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a20102440 0x7f0a2019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:16.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a25b03700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a20102e00 0x7f0a2019fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:16.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a25b03700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a20108520 0x7f0a20199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:16.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a25b03700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0a201124a0 con 0x7f0a20102e00 2024-06-04T04:54:16.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a25b03700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a20112320 con 0x7f0a20102440 2024-06-04T04:54:16.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a25b03700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0a20112620 con 0x7f0a20108520 2024-06-04T04:54:16.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a16ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a20102e00 0x7f0a2019fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:16.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a16ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a20102e00 0x7f0a2019fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40336/0 (socket says 172.21.15.70:40336) 2024-06-04T04:54:16.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a16ffd700 1 -- 172.21.15.70:0/3723287680 learned_addr learned my addr 172.21.15.70:0/3723287680 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:16.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a1ffff700 1 --2- 172.21.15.70:0/3723287680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a20108520 0x7f0a20199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:16.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a16ffd700 1 -- 172.21.15.70:0/3723287680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a20108520 msgr2=0x7f0a20199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:16.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a1f7fe700 1 --2- 172.21.15.70:0/3723287680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a20102440 0x7f0a2019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:16.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a16ffd700 1 --2- 172.21.15.70:0/3723287680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a20108520 0x7f0a20199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:16.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a16ffd700 1 -- 172.21.15.70:0/3723287680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a20102440 msgr2=0x7f0a2019f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:16.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a16ffd700 1 --2- 172.21.15.70:0/3723287680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a20102440 0x7f0a2019f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:16.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a16ffd700 1 -- 172.21.15.70:0/3723287680 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a2019a0a0 con 0x7f0a20102e00 2024-06-04T04:54:16.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.891+0000 7f0a1f7fe700 1 --2- 172.21.15.70:0/3723287680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a20102440 0x7f0a2019f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:54:16.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.892+0000 7f0a16ffd700 1 --2- 172.21.15.70:0/3723287680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a20102e00 0x7f0a2019fc70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800c9b0 tx=0x7f0a0800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.892+0000 7f0a1d7fa700 1 -- 172.21.15.70:0/3723287680 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a08007b00 con 0x7f0a20102e00 2024-06-04T04:54:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.892+0000 7f0a25b03700 1 -- 172.21.15.70:0/3723287680 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a2019a390 con 0x7f0a20102e00 2024-06-04T04:54:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.892+0000 7f0a1d7fa700 1 -- 172.21.15.70:0/3723287680 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a08007ca0 con 0x7f0a20102e00 2024-06-04T04:54:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.892+0000 7f0a1d7fa700 1 -- 172.21.15.70:0/3723287680 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a08018660 con 0x7f0a20102e00 2024-06-04T04:54:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.892+0000 7f0a25b03700 1 -- 172.21.15.70:0/3723287680 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a201a6600 con 0x7f0a20102e00 2024-06-04T04:54:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.893+0000 7f0a25b03700 1 -- 172.21.15.70:0/3723287680 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a2006a230 con 0x7f0a20102e00 2024-06-04T04:54:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.894+0000 7f0a1d7fa700 1 -- 172.21.15.70:0/3723287680 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f0a0800f450 con 0x7f0a20102e00 2024-06-04T04:54:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.895+0000 7f0a1d7fa700 1 --2- 172.21.15.70:0/3723287680 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0a0c0b2910 0x7f0a0c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.895+0000 7f0a1f7fe700 1 --2- 172.21.15.70:0/3723287680 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0a0c0b2910 0x7f0a0c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.895+0000 7f0a1d7fa700 1 -- 172.21.15.70:0/3723287680 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f0a080e2640 con 0x7f0a20102e00 2024-06-04T04:54:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.895+0000 7f0a1f7fe700 1 --2- 172.21.15.70:0/3723287680 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0a0c0b2910 0x7f0a0c0b4dd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0a1000b2e0 tx=0x7f0a10005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:16.899+0000 7f0a1d7fa700 1 -- 172.21.15.70:0/3723287680 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0a08014030 con 0x7f0a20102e00 2024-06-04T04:54:17.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:17.108+0000 7f0a25b03700 1 -- 172.21.15.70:0/3723287680 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0a201a6890 con 0x7f0a20102e00 2024-06-04T04:54:17.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:17.109+0000 7f0a1d7fa700 1 -- 172.21.15.70:0/3723287680 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22183 (secure 0 0 0) 0x7f0a080aa340 con 0x7f0a20102e00 2024-06-04T04:54:17.112 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:54:17.112 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":79,"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","created":"2024-06-04T04:45:22.297536+0000","modified":"2024-06-04T04:53:38.238030+0000","last_up_change":"2024-06-04T04:52:34.388708+0000","last_in_change":"2024-06-04T04:52:14.197780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-04T04:48:43.724356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-04T04:53:31.398455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-04T04:53:35.298617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ff491ea2-1906-48e4-883e-bc78e778260e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6803","nonce":824119100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6805","nonce":824119100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6809","nonce":824119100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6807","nonce":824119100}]},"public_addr":"172.21.15.70:6803/824119100","cluster_addr":"172.21.15.70:6805/824119100","heartbeat_back_addr":"172.21.15.70:6809/824119100","heartbeat_front_addr":"172.21.15.70:6807/824119100","state":["exists","up"]},{"osd":1,"uuid":"bd5e4806-535b-49b5-ba6b-93228da35a84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6801","nonce":2744598742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6803","nonce":2744598742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6807","nonce":2744598742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6805","nonce":2744598742}]},"public_addr":"172.21.15.98:6801/2744598742","cluster_addr":"172.21.15.98:6803/2744598742","heartbeat_back_addr":"172.21.15.98:6807/2744598742","heartbeat_front_addr":"172.21.15.98:6805/2744598742","state":["exists","up"]},{"osd":2,"uuid":"8456fb55-910a-4a59-b84b-2c0f68346ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6801","nonce":1328098370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6803","nonce":1328098370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6807","nonce":1328098370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6805","nonce":1328098370}]},"public_addr":"172.21.15.112:6801/1328098370","cluster_addr":"172.21.15.112:6803/1328098370","heartbeat_back_addr":"172.21.15.112:6807/1328098370","heartbeat_front_addr":"172.21.15.112:6805/1328098370","state":["exists","up"]},{"osd":3,"uuid":"c8fbc440-bf70-42d3-b01e-777ccfa7900b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6811","nonce":4041566346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6813","nonce":4041566346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6817","nonce":4041566346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6815","nonce":4041566346}]},"public_addr":"172.21.15.70:6811/4041566346","cluster_addr":"172.21.15.70:6813/4041566346","heartbeat_back_addr":"172.21.15.70:6817/4041566346","heartbeat_front_addr":"172.21.15.70:6815/4041566346","state":["exists","up"]},{"osd":4,"uuid":"549c8c70-8be8-46ae-bcb6-0fde32b40cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6809","nonce":3652775814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6811","nonce":3652775814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6815","nonce":3652775814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6813","nonce":3652775814}]},"public_addr":"172.21.15.98:6809/3652775814","cluster_addr":"172.21.15.98:6811/3652775814","heartbeat_back_addr":"172.21.15.98:6815/3652775814","heartbeat_front_addr":"172.21.15.98:6813/3652775814","state":["exists","up"]},{"osd":5,"uuid":"51d86af1-0d6c-4ae0-b090-00414fc1bf7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6809","nonce":1857257899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6811","nonce":1857257899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6815","nonce":1857257899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6813","nonce":1857257899}]},"public_addr":"172.21.15.112:6809/1857257899","cluster_addr":"172.21.15.112:6811/1857257899","heartbeat_back_addr":"172.21.15.112:6815/1857257899","heartbeat_front_addr":"172.21.15.112:6813/1857257899","state":["exists","up"]},{"osd":6,"uuid":"526a8138-bfa8-419e-a5b5-e6b80f0ae004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6819","nonce":1264676496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6821","nonce":1264676496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6825","nonce":1264676496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6823","nonce":1264676496}]},"public_addr":"172.21.15.70:6819/1264676496","cluster_addr":"172.21.15.70:6821/1264676496","heartbeat_back_addr":"172.21.15.70:6825/1264676496","heartbeat_front_addr":"172.21.15.70:6823/1264676496","state":["exists","up"]},{"osd":7,"uuid":"1173f785-6ed1-47ef-9265-2e233ea9ee02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6817","nonce":709593020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6819","nonce":709593020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6823","nonce":709593020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6821","nonce":709593020}]},"public_addr":"172.21.15.98:6817/709593020","cluster_addr":"172.21.15.98:6819/709593020","heartbeat_back_addr":"172.21.15.98:6823/709593020","heartbeat_front_addr":"172.21.15.98:6821/709593020","state":["exists","up"]},{"osd":8,"uuid":"6424cc6e-24a7-4b81-9ce6-39203fecc7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6817","nonce":2250225413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6819","nonce":2250225413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6823","nonce":2250225413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6821","nonce":2250225413}]},"public_addr":"172.21.15.112:6817/2250225413","cluster_addr":"172.21.15.112:6819/2250225413","heartbeat_back_addr":"172.21.15.112:6823/2250225413","heartbeat_front_addr":"172.21.15.112:6821/2250225413","state":["exists","up"]},{"osd":9,"uuid":"8d666db8-8831-4d63-9e3c-a4bf2504149e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6827","nonce":3557688707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6829","nonce":3557688707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6833","nonce":3557688707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6831","nonce":3557688707}]},"public_addr":"172.21.15.70:6827/3557688707","cluster_addr":"172.21.15.70:6829/3557688707","heartbeat_back_addr":"172.21.15.70:6833/3557688707","heartbeat_front_addr":"172.21.15.70:6831/3557688707","state":["exists","up"]},{"osd":10,"uuid":"ecb925d2-0263-4dfc-a2fd-9e27856022d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6825","nonce":1896493718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6827","nonce":1896493718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6831","nonce":1896493718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6829","nonce":1896493718}]},"public_addr":"172.21.15.98:6825/1896493718","cluster_addr":"172.21.15.98:6827/1896493718","heartbeat_back_addr":"172.21.15.98:6831/1896493718","heartbeat_front_addr":"172.21.15.98:6829/1896493718","state":["exists","up"]},{"osd":11,"uuid":"c8e3230c-bc3a-40bd-8dd0-5fef56e4df97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6825","nonce":1400793137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6827","nonce":1400793137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6831","nonce":1400793137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6829","nonce":1400793137}]},"public_addr":"172.21.15.112:6825/1400793137","cluster_addr":"172.21.15.112:6827/1400793137","heartbeat_back_addr":"172.21.15.112:6831/1400793137","heartbeat_front_addr":"172.21.15.112:6829/1400793137","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:47:53.626608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:17.784505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:41.971969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:07.130699+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:31.615496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:56.898607+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:22.739991+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:48.319727+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:13.902063+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:40.389470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:06.142092+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:32.949383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/3442074771":"2024-06-05T04:46:17.636907+0000","172.21.15.70:6801/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/1442095493":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/2453733268":"2024-06-05T04:45:49.501864+0000","172.21.15.70:0/1369781447":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6801/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/915186456":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/860033006":"2024-06-05T04:45:49.501864+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-04T04:54:17.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:17.112+0000 7f0a25b03700 1 -- 172.21.15.70:0/3723287680 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0a0c0b2910 msgr2=0x7f0a0c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:17.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:17.112+0000 7f0a25b03700 1 --2- 172.21.15.70:0/3723287680 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0a0c0b2910 0x7f0a0c0b4dd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0a1000b2e0 tx=0x7f0a10005e80 comp rx=0 tx=0).stop 2024-06-04T04:54:17.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:17.112+0000 7f0a25b03700 1 -- 172.21.15.70:0/3723287680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a20102e00 msgr2=0x7f0a2019fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:17.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:17.112+0000 7f0a25b03700 1 --2- 172.21.15.70:0/3723287680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a20102e00 0x7f0a2019fc70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800c9b0 tx=0x7f0a0800cd70 comp rx=0 tx=0).stop 2024-06-04T04:54:17.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:17.112+0000 7f0a25b03700 1 -- 172.21.15.70:0/3723287680 shutdown_connections 2024-06-04T04:54:17.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:17.112+0000 7f0a25b03700 1 --2- 172.21.15.70:0/3723287680 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0a0c0b2910 0x7f0a0c0b4dd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:17.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:17.112+0000 7f0a25b03700 1 --2- 172.21.15.70:0/3723287680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a20108520 0x7f0a20199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:17.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:17.112+0000 7f0a25b03700 1 --2- 172.21.15.70:0/3723287680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a20102440 0x7f0a2019f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:17.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:17.112+0000 7f0a25b03700 1 --2- 172.21.15.70:0/3723287680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a20102e00 0x7f0a2019fc70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:17.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:17.112+0000 7f0a25b03700 1 -- 172.21.15.70:0/3723287680 >> 172.21.15.70:0/3723287680 conn(0x7f0a200793f0 msgr2=0x7f0a2006b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:17.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:17.112+0000 7f0a25b03700 1 -- 172.21.15.70:0/3723287680 shutdown_connections 2024-06-04T04:54:17.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:17.112+0000 7f0a25b03700 1 -- 172.21.15.70:0/3723287680 wait complete. 2024-06-04T04:54:17.738 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:17 smithi070 ceph-mon[32631]: pgmap v308: 97 pgs: 97 active+clean; 627 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-06-04T04:54:17.738 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:17 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3723287680' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:54:17.765 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-04T04:54:17.765 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-04T04:54:17.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:17 smithi112 ceph-mon[36453]: pgmap v308: 97 pgs: 97 active+clean; 627 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-06-04T04:54:17.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:17 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3723287680' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:54:17.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:17 smithi098 ceph-mon[35569]: pgmap v308: 97 pgs: 97 active+clean; 627 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-06-04T04:54:17.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:17 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3723287680' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:54:18.002 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:18.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-06-04T04:54:18.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-06-04T04:54:18.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-06-04T04:54:18.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-06-04T04:54:18.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-06-04T04:54:18.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-06-04T04:54:18.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-06-04T04:54:18.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-06-04T04:54:18.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-06-04T04:54:18.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:18 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]: dispatch 2024-06-04T04:54:18.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-06-04T04:54:18.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-06-04T04:54:18.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-06-04T04:54:18.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-06-04T04:54:18.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-06-04T04:54:18.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-06-04T04:54:18.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-06-04T04:54:18.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-06-04T04:54:18.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-06-04T04:54:18.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:18 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]: dispatch 2024-06-04T04:54:18.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-06-04T04:54:18.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-06-04T04:54:18.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-06-04T04:54:18.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-06-04T04:54:18.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-06-04T04:54:18.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-06-04T04:54:18.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-06-04T04:54:18.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-06-04T04:54:18.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-06-04T04:54:18.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:18 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]: dispatch 2024-06-04T04:54:19.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.728+0000 7f17f6b8b700 1 -- 172.21.15.70:0/2676048315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f0101b30 msgr2=0x7f17f0101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:19.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.728+0000 7f17f6b8b700 1 --2- 172.21.15.70:0/2676048315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f0101b30 0x7f17f0101fb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f17e00099c0 tx=0x7f17e0009cd0 comp rx=0 tx=0).stop 2024-06-04T04:54:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.728+0000 7f17f6b8b700 1 -- 172.21.15.70:0/2676048315 shutdown_connections 2024-06-04T04:54:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.728+0000 7f17f6b8b700 1 --2- 172.21.15.70:0/2676048315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f01024f0 0x7f17f01103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.728+0000 7f17f6b8b700 1 --2- 172.21.15.70:0/2676048315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f0109520 0x7f17f0109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.728+0000 7f17f6b8b700 1 --2- 172.21.15.70:0/2676048315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f0101b30 0x7f17f0101fb0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.728+0000 7f17f6b8b700 1 -- 172.21.15.70:0/2676048315 >> 172.21.15.70:0/2676048315 conn(0x7f17f0078ab0 msgr2=0x7f17f00ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.728+0000 7f17f6b8b700 1 -- 172.21.15.70:0/2676048315 shutdown_connections 2024-06-04T04:54:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.728+0000 7f17f6b8b700 1 -- 172.21.15.70:0/2676048315 wait complete. 2024-06-04T04:54:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f6b8b700 1 Processor -- start 2024-06-04T04:54:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f6b8b700 1 -- start start 2024-06-04T04:54:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f6b8b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f0101b30 0x7f17f019f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f6b8b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f01024f0 0x7f17f019fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f6b8b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f0109520 0x7f17f0199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f6b8b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0113610 con 0x7f17f0109520 2024-06-04T04:54:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f6b8b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0113490 con 0x7f17f0101b30 2024-06-04T04:54:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f6b8b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0113790 con 0x7f17f01024f0 2024-06-04T04:54:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f5128700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f0109520 0x7f17f0199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f5128700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f0109520 0x7f17f0199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40360/0 (socket says 172.21.15.70:40360) 2024-06-04T04:54:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f5128700 1 -- 172.21.15.70:0/3514073594 learned_addr learned my addr 172.21.15.70:0/3514073594 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f4927700 1 --2- 172.21.15.70:0/3514073594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f0101b30 0x7f17f019f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17e7fff700 1 --2- 172.21.15.70:0/3514073594 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f01024f0 0x7f17f019fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f5128700 1 -- 172.21.15.70:0/3514073594 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f01024f0 msgr2=0x7f17f019fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f5128700 1 --2- 172.21.15.70:0/3514073594 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f01024f0 0x7f17f019fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f5128700 1 -- 172.21.15.70:0/3514073594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f0101b30 msgr2=0x7f17f019f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f5128700 1 --2- 172.21.15.70:0/3514073594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f0101b30 0x7f17f019f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f5128700 1 -- 172.21.15.70:0/3514073594 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f019a0f0 con 0x7f17f0109520 2024-06-04T04:54:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.729+0000 7f17f5128700 1 --2- 172.21.15.70:0/3514073594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f0109520 0x7f17f0199990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f17e800b970 tx=0x7f17e800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.730+0000 7f17e5ffb700 1 -- 172.21.15.70:0/3514073594 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e800d860 con 0x7f17f0109520 2024-06-04T04:54:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.730+0000 7f17e5ffb700 1 -- 172.21.15.70:0/3514073594 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f17e800e430 con 0x7f17f0109520 2024-06-04T04:54:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.730+0000 7f17e5ffb700 1 -- 172.21.15.70:0/3514073594 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8017660 con 0x7f17f0109520 2024-06-04T04:54:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.730+0000 7f17f6b8b700 1 -- 172.21.15.70:0/3514073594 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f019a3e0 con 0x7f17f0109520 2024-06-04T04:54:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.730+0000 7f17f6b8b700 1 -- 172.21.15.70:0/3514073594 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f01a66a0 con 0x7f17f0109520 2024-06-04T04:54:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.731+0000 7f17f6b8b700 1 -- 172.21.15.70:0/3514073594 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17f006a230 con 0x7f17f0109520 2024-06-04T04:54:19.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.732+0000 7f17e5ffb700 1 -- 172.21.15.70:0/3514073594 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f17e800da00 con 0x7f17f0109520 2024-06-04T04:54:19.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.733+0000 7f17e5ffb700 1 --2- 172.21.15.70:0/3514073594 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f17d80b2910 0x7f17d80b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:19.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.733+0000 7f17f4927700 1 --2- 172.21.15.70:0/3514073594 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f17d80b2910 0x7f17d80b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:19.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.733+0000 7f17e5ffb700 1 -- 172.21.15.70:0/3514073594 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f17e80e1f50 con 0x7f17f0109520 2024-06-04T04:54:19.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.733+0000 7f17f4927700 1 --2- 172.21.15.70:0/3514073594 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f17d80b2910 0x7f17d80b4dd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f17f019aad0 tx=0x7f17ec0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:19.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.737+0000 7f17e5ffb700 1 -- 172.21.15.70:0/3514073594 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f17e8013030 con 0x7f17f0109520 2024-06-04T04:54:19.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:19 smithi070 ceph-mon[32631]: pgmap v309: 97 pgs: 97 active+clean; 637 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-06-04T04:54:19.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-06-04T04:54:19.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-06-04T04:54:19.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-06-04T04:54:19.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-06-04T04:54:19.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-06-04T04:54:19.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-06-04T04:54:19.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-06-04T04:54:19.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-06-04T04:54:19.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-06-04T04:54:19.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:19 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]': finished 2024-06-04T04:54:19.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:19 smithi070 ceph-mon[32631]: osdmap e80: 12 total, 12 up, 12 in 2024-06-04T04:54:19.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:19 smithi112 ceph-mon[36453]: pgmap v309: 97 pgs: 97 active+clean; 637 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-06-04T04:54:19.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-06-04T04:54:19.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-06-04T04:54:19.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-06-04T04:54:19.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-06-04T04:54:19.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-06-04T04:54:19.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-06-04T04:54:19.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-06-04T04:54:19.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-06-04T04:54:19.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-06-04T04:54:19.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:19 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]': finished 2024-06-04T04:54:19.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:19 smithi112 ceph-mon[36453]: osdmap e80: 12 total, 12 up, 12 in 2024-06-04T04:54:19.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.949+0000 7f17f6b8b700 1 -- 172.21.15.70:0/3514073594 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f17f0063980 con 0x7f17d80b2910 2024-06-04T04:54:19.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.957+0000 7f17e5ffb700 1 -- 172.21.15.70:0/3514073594 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258894 (secure 0 0 0) 0x7f17f0063980 con 0x7f17d80b2910 2024-06-04T04:54:19.963 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:54:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.960+0000 7f17f6b8b700 1 -- 172.21.15.70:0/3514073594 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f17d80b2910 msgr2=0x7f17d80b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.960+0000 7f17f6b8b700 1 --2- 172.21.15.70:0/3514073594 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f17d80b2910 0x7f17d80b4dd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f17f019aad0 tx=0x7f17ec0073d0 comp rx=0 tx=0).stop 2024-06-04T04:54:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.960+0000 7f17f6b8b700 1 -- 172.21.15.70:0/3514073594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f0109520 msgr2=0x7f17f0199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.960+0000 7f17f6b8b700 1 --2- 172.21.15.70:0/3514073594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f0109520 0x7f17f0199990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f17e800b970 tx=0x7f17e800bc80 comp rx=0 tx=0).stop 2024-06-04T04:54:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.960+0000 7f17f6b8b700 1 -- 172.21.15.70:0/3514073594 shutdown_connections 2024-06-04T04:54:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.960+0000 7f17f6b8b700 1 --2- 172.21.15.70:0/3514073594 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f17d80b2910 0x7f17d80b4dd0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.960+0000 7f17f6b8b700 1 --2- 172.21.15.70:0/3514073594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f0109520 0x7f17f0199990 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.960+0000 7f17f6b8b700 1 --2- 172.21.15.70:0/3514073594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f0101b30 0x7f17f019f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.960+0000 7f17f6b8b700 1 --2- 172.21.15.70:0/3514073594 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f01024f0 0x7f17f019fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.960+0000 7f17f6b8b700 1 -- 172.21.15.70:0/3514073594 >> 172.21.15.70:0/3514073594 conn(0x7f17f0078ab0 msgr2=0x7f17f0100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.960+0000 7f17f6b8b700 1 -- 172.21.15.70:0/3514073594 shutdown_connections 2024-06-04T04:54:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:19.960+0000 7f17f6b8b700 1 -- 172.21.15.70:0/3514073594 wait complete. 2024-06-04T04:54:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-06-04T04:54:19.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:19 smithi098 ceph-mon[35569]: pgmap v309: 97 pgs: 97 active+clean; 637 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-06-04T04:54:19.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-06-04T04:54:19.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-06-04T04:54:19.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-06-04T04:54:19.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-06-04T04:54:19.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-06-04T04:54:19.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-06-04T04:54:19.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-06-04T04:54:19.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-06-04T04:54:19.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-06-04T04:54:19.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:19 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]': finished 2024-06-04T04:54:19.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:19 smithi098 ceph-mon[35569]: osdmap e80: 12 total, 12 up, 12 in 2024-06-04T04:54:20.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:20 smithi070 ceph-mon[32631]: osdmap e81: 12 total, 12 up, 12 in 2024-06-04T04:54:20.683 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2024-06-04T04:54:19.731552+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651787,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":364,"num_read_kb":274,"num_write":363,"num_write_kb":2277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":298,"ondisk_log_size":298,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":292,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":355648,"kb_used_data":10640,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124586176,"statfs":{"total":1151940427776,"available":1151576244224,"internally_reserved":0,"allocated":10895360,"data_stored":6934002,"data_compressed":125279,"data_compressed_allocated":163840,"data_compressed_original":1826816,"omap_allocated":19138,"internal_metadata":353154366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27427,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":97,"num_read_kb":73,"num_write":19,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001912"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.380206+0000","last_change":"2024-06-04T04:53:36.383755+0000","last_active":"2024-06-04T04:53:38.380206+0000","last_peered":"2024-06-04T04:53:38.380206+0000","last_clean":"2024-06-04T04:53:38.380206+0000","last_became_active":"2024-06-04T04:53:36.378274+0000","last_became_peered":"2024-06-04T04:53:36.378274+0000","last_unstale":"2024-06-04T04:53:38.380206+0000","last_undegraded":"2024-06-04T04:53:38.380206+0000","last_fullsized":"2024-06-04T04:53:38.380206+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:30:19.685994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.376152+0000","last_change":"2024-06-04T04:53:36.372436+0000","last_active":"2024-06-04T04:53:38.376152+0000","last_peered":"2024-06-04T04:53:38.376152+0000","last_clean":"2024-06-04T04:53:38.376152+0000","last_became_active":"2024-06-04T04:53:36.372100+0000","last_became_peered":"2024-06-04T04:53:36.372100+0000","last_unstale":"2024-06-04T04:53:38.376152+0000","last_undegraded":"2024-06-04T04:53:38.376152+0000","last_fullsized":"2024-06-04T04:53:38.376152+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:42:31.646754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.375389+0000","last_change":"2024-06-04T04:53:36.366915+0000","last_active":"2024-06-04T04:53:38.375389+0000","last_peered":"2024-06-04T04:53:38.375389+0000","last_clean":"2024-06-04T04:53:38.375389+0000","last_became_active":"2024-06-04T04:53:36.366611+0000","last_became_peered":"2024-06-04T04:53:36.366611+0000","last_unstale":"2024-06-04T04:53:38.375389+0000","last_undegraded":"2024-06-04T04:53:38.375389+0000","last_fullsized":"2024-06-04T04:53:38.375389+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:17:48.775938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631647+0000","last_change":"2024-06-04T04:53:36.374486+0000","last_active":"2024-06-04T04:54:19.631647+0000","last_peered":"2024-06-04T04:54:19.631647+0000","last_clean":"2024-06-04T04:54:19.631647+0000","last_became_active":"2024-06-04T04:53:36.374178+0000","last_became_peered":"2024-06-04T04:53:36.374178+0000","last_unstale":"2024-06-04T04:54:19.631647+0000","last_undegraded":"2024-06-04T04:54:19.631647+0000","last_fullsized":"2024-06-04T04:54:19.631647+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:30:28.287070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.364249+0000","last_change":"2024-06-04T04:53:36.376183+0000","last_active":"2024-06-04T04:53:38.364249+0000","last_peered":"2024-06-04T04:53:38.364249+0000","last_clean":"2024-06-04T04:53:38.364249+0000","last_became_active":"2024-06-04T04:53:36.375790+0000","last_became_peered":"2024-06-04T04:53:36.375790+0000","last_unstale":"2024-06-04T04:53:38.364249+0000","last_undegraded":"2024-06-04T04:53:38.364249+0000","last_fullsized":"2024-06-04T04:53:38.364249+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:58.697440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.374957+0000","last_change":"2024-06-04T04:53:36.370249+0000","last_active":"2024-06-04T04:53:38.374957+0000","last_peered":"2024-06-04T04:53:38.374957+0000","last_clean":"2024-06-04T04:53:38.374957+0000","last_became_active":"2024-06-04T04:53:36.369916+0000","last_became_peered":"2024-06-04T04:53:36.369916+0000","last_unstale":"2024-06-04T04:53:38.374957+0000","last_undegraded":"2024-06-04T04:53:38.374957+0000","last_fullsized":"2024-06-04T04:53:38.374957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:02:41.739806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510237+0000","last_change":"2024-06-04T04:53:36.384329+0000","last_active":"2024-06-04T04:54:19.510237+0000","last_peered":"2024-06-04T04:54:19.510237+0000","last_clean":"2024-06-04T04:54:19.510237+0000","last_became_active":"2024-06-04T04:53:36.382504+0000","last_became_peered":"2024-06-04T04:53:36.382504+0000","last_unstale":"2024-06-04T04:54:19.510237+0000","last_undegraded":"2024-06-04T04:54:19.510237+0000","last_fullsized":"2024-06-04T04:54:19.510237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:51:07.604691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504427+0000","last_change":"2024-06-04T04:53:36.372319+0000","last_active":"2024-06-04T04:54:18.504427+0000","last_peered":"2024-06-04T04:54:18.504427+0000","last_clean":"2024-06-04T04:54:18.504427+0000","last_became_active":"2024-06-04T04:53:36.372000+0000","last_became_peered":"2024-06-04T04:53:36.372000+0000","last_unstale":"2024-06-04T04:54:18.504427+0000","last_undegraded":"2024-06-04T04:54:18.504427+0000","last_fullsized":"2024-06-04T04:54:18.504427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:15:56.906996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.364499+0000","last_change":"2024-06-04T04:53:36.376051+0000","last_active":"2024-06-04T04:53:38.364499+0000","last_peered":"2024-06-04T04:53:38.364499+0000","last_clean":"2024-06-04T04:53:38.364499+0000","last_became_active":"2024-06-04T04:53:36.375726+0000","last_became_peered":"2024-06-04T04:53:36.375726+0000","last_unstale":"2024-06-04T04:53:38.364499+0000","last_undegraded":"2024-06-04T04:53:38.364499+0000","last_fullsized":"2024-06-04T04:53:38.364499+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:09:08.542764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.378366+0000","last_change":"2024-06-04T04:53:36.373266+0000","last_active":"2024-06-04T04:53:38.378366+0000","last_peered":"2024-06-04T04:53:38.378366+0000","last_clean":"2024-06-04T04:53:38.378366+0000","last_became_active":"2024-06-04T04:53:36.372898+0000","last_became_peered":"2024-06-04T04:53:36.372898+0000","last_unstale":"2024-06-04T04:53:38.378366+0000","last_undegraded":"2024-06-04T04:53:38.378366+0000","last_fullsized":"2024-06-04T04:53:38.378366+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:41:43.436781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510727+0000","last_change":"2024-06-04T04:53:36.381344+0000","last_active":"2024-06-04T04:54:19.510727+0000","last_peered":"2024-06-04T04:54:19.510727+0000","last_clean":"2024-06-04T04:54:19.510727+0000","last_became_active":"2024-06-04T04:53:36.380736+0000","last_became_peered":"2024-06-04T04:53:36.380736+0000","last_unstale":"2024-06-04T04:54:19.510727+0000","last_undegraded":"2024-06-04T04:54:19.510727+0000","last_fullsized":"2024-06-04T04:54:19.510727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:23:01.607272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631528+0000","last_change":"2024-06-04T04:53:36.374035+0000","last_active":"2024-06-04T04:54:19.631528+0000","last_peered":"2024-06-04T04:54:19.631528+0000","last_clean":"2024-06-04T04:54:19.631528+0000","last_became_active":"2024-06-04T04:53:36.373667+0000","last_became_peered":"2024-06-04T04:53:36.373667+0000","last_unstale":"2024-06-04T04:54:19.631528+0000","last_undegraded":"2024-06-04T04:54:19.631528+0000","last_fullsized":"2024-06-04T04:54:19.631528+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:23:07.880369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.377099+0000","last_change":"2024-06-04T04:53:36.369613+0000","last_active":"2024-06-04T04:53:38.377099+0000","last_peered":"2024-06-04T04:53:38.377099+0000","last_clean":"2024-06-04T04:53:38.377099+0000","last_became_active":"2024-06-04T04:53:36.369277+0000","last_became_peered":"2024-06-04T04:53:36.369277+0000","last_unstale":"2024-06-04T04:53:38.377099+0000","last_undegraded":"2024-06-04T04:53:38.377099+0000","last_fullsized":"2024-06-04T04:53:38.377099+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:13:33.756609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.375109+0000","last_change":"2024-06-04T04:53:36.368094+0000","last_active":"2024-06-04T04:53:38.375109+0000","last_peered":"2024-06-04T04:53:38.375109+0000","last_clean":"2024-06-04T04:53:38.375109+0000","last_became_active":"2024-06-04T04:53:36.367717+0000","last_became_peered":"2024-06-04T04:53:36.367717+0000","last_unstale":"2024-06-04T04:53:38.375109+0000","last_undegraded":"2024-06-04T04:53:38.375109+0000","last_fullsized":"2024-06-04T04:53:38.375109+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:39:56.305078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.375998+0000","last_change":"2024-06-04T04:53:36.366765+0000","last_active":"2024-06-04T04:53:38.375998+0000","last_peered":"2024-06-04T04:53:38.375998+0000","last_clean":"2024-06-04T04:53:38.375998+0000","last_became_active":"2024-06-04T04:53:36.366438+0000","last_became_peered":"2024-06-04T04:53:36.366438+0000","last_unstale":"2024-06-04T04:53:38.375998+0000","last_undegraded":"2024-06-04T04:53:38.375998+0000","last_fullsized":"2024-06-04T04:53:38.375998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:38:37.313926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.363674+0000","last_change":"2024-06-04T04:53:36.380088+0000","last_active":"2024-06-04T04:53:38.363674+0000","last_peered":"2024-06-04T04:53:38.363674+0000","last_clean":"2024-06-04T04:53:38.363674+0000","last_became_active":"2024-06-04T04:53:36.379467+0000","last_became_peered":"2024-06-04T04:53:36.379467+0000","last_unstale":"2024-06-04T04:53:38.363674+0000","last_undegraded":"2024-06-04T04:53:38.363674+0000","last_fullsized":"2024-06-04T04:53:38.363674+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:46:53.236419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.380579+0000","last_change":"2024-06-04T04:53:36.381126+0000","last_active":"2024-06-04T04:53:38.380579+0000","last_peered":"2024-06-04T04:53:38.380579+0000","last_clean":"2024-06-04T04:53:38.380579+0000","last_became_active":"2024-06-04T04:53:36.378833+0000","last_became_peered":"2024-06-04T04:53:36.378833+0000","last_unstale":"2024-06-04T04:53:38.380579+0000","last_undegraded":"2024-06-04T04:53:38.380579+0000","last_fullsized":"2024-06-04T04:53:38.380579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:20:28.639273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.374064+0000","last_change":"2024-06-04T04:53:36.369692+0000","last_active":"2024-06-04T04:53:38.374064+0000","last_peered":"2024-06-04T04:53:38.374064+0000","last_clean":"2024-06-04T04:53:38.374064+0000","last_became_active":"2024-06-04T04:53:36.369370+0000","last_became_peered":"2024-06-04T04:53:36.369370+0000","last_unstale":"2024-06-04T04:53:38.374064+0000","last_undegraded":"2024-06-04T04:53:38.374064+0000","last_fullsized":"2024-06-04T04:53:38.374064+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:55.275465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.509473+0000","last_change":"2024-06-04T04:53:36.383639+0000","last_active":"2024-06-04T04:54:18.509473+0000","last_peered":"2024-06-04T04:54:18.509473+0000","last_clean":"2024-06-04T04:54:18.509473+0000","last_became_active":"2024-06-04T04:53:36.383303+0000","last_became_peered":"2024-06-04T04:53:36.383303+0000","last_unstale":"2024-06-04T04:54:18.509473+0000","last_undegraded":"2024-06-04T04:54:18.509473+0000","last_fullsized":"2024-06-04T04:54:18.509473+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:26:31.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":60,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:54:16.484354+0000","last_change":"2024-06-04T04:53:33.344120+0000","last_active":"2024-06-04T04:54:16.484354+0000","last_peered":"2024-06-04T04:54:16.484354+0000","last_clean":"2024-06-04T04:54:16.484354+0000","last_became_active":"2024-06-04T04:53:33.343743+0000","last_became_peered":"2024-06-04T04:53:33.343743+0000","last_unstale":"2024-06-04T04:54:16.484354+0000","last_undegraded":"2024-06-04T04:54:16.484354+0000","last_fullsized":"2024-06-04T04:54:16.484354+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:57:46.585195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504978+0000","last_change":"2024-06-04T04:53:36.371827+0000","last_active":"2024-06-04T04:54:18.504978+0000","last_peered":"2024-06-04T04:54:18.504978+0000","last_clean":"2024-06-04T04:54:18.504978+0000","last_became_active":"2024-06-04T04:53:36.371509+0000","last_became_peered":"2024-06-04T04:53:36.371509+0000","last_unstale":"2024-06-04T04:54:18.504978+0000","last_undegraded":"2024-06-04T04:54:18.504978+0000","last_fullsized":"2024-06-04T04:54:18.504978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:54:51.589008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.505150+0000","last_change":"2024-06-04T04:53:33.353976+0000","last_active":"2024-06-04T04:54:18.505150+0000","last_peered":"2024-06-04T04:54:18.505150+0000","last_clean":"2024-06-04T04:54:18.505150+0000","last_became_active":"2024-06-04T04:53:33.353586+0000","last_became_peered":"2024-06-04T04:53:33.353586+0000","last_unstale":"2024-06-04T04:54:18.505150+0000","last_undegraded":"2024-06-04T04:54:18.505150+0000","last_fullsized":"2024-06-04T04:54:18.505150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:17:10.378008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.376909+0000","last_change":"2024-06-04T04:53:36.367806+0000","last_active":"2024-06-04T04:53:38.376909+0000","last_peered":"2024-06-04T04:53:38.376909+0000","last_clean":"2024-06-04T04:53:38.376909+0000","last_became_active":"2024-06-04T04:53:36.367484+0000","last_became_peered":"2024-06-04T04:53:36.367484+0000","last_unstale":"2024-06-04T04:53:38.376909+0000","last_undegraded":"2024-06-04T04:53:38.376909+0000","last_fullsized":"2024-06-04T04:53:38.376909+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:25:36.194746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":35,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:54.451561+0000","last_change":"2024-06-04T04:53:33.348902+0000","last_active":"2024-06-04T04:53:54.451561+0000","last_peered":"2024-06-04T04:53:54.451561+0000","last_clean":"2024-06-04T04:53:54.451561+0000","last_became_active":"2024-06-04T04:53:33.348441+0000","last_became_peered":"2024-06-04T04:53:33.348441+0000","last_unstale":"2024-06-04T04:53:54.451561+0000","last_undegraded":"2024-06-04T04:53:54.451561+0000","last_fullsized":"2024-06-04T04:53:54.451561+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:22:28.428303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.364094+0000","last_change":"2024-06-04T04:53:36.376867+0000","last_active":"2024-06-04T04:53:38.364094+0000","last_peered":"2024-06-04T04:53:38.364094+0000","last_clean":"2024-06-04T04:53:38.364094+0000","last_became_active":"2024-06-04T04:53:36.376517+0000","last_became_peered":"2024-06-04T04:53:36.376517+0000","last_unstale":"2024-06-04T04:53:38.364094+0000","last_undegraded":"2024-06-04T04:53:38.364094+0000","last_fullsized":"2024-06-04T04:53:38.364094+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:21:59.494321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":28,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:53.447130+0000","last_change":"2024-06-04T04:53:33.347993+0000","last_active":"2024-06-04T04:53:53.447130+0000","last_peered":"2024-06-04T04:53:53.447130+0000","last_clean":"2024-06-04T04:53:53.447130+0000","last_became_active":"2024-06-04T04:53:33.347656+0000","last_became_peered":"2024-06-04T04:53:33.347656+0000","last_unstale":"2024-06-04T04:53:53.447130+0000","last_undegraded":"2024-06-04T04:53:53.447130+0000","last_fullsized":"2024-06-04T04:53:53.447130+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:21:26.806557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.632790+0000","last_change":"2024-06-04T04:54:19.632790+0000","last_active":"2024-06-04T04:54:19.632790+0000","last_peered":"2024-06-04T04:54:19.632790+0000","last_clean":"2024-06-04T04:54:19.632790+0000","last_became_active":"2024-06-04T04:54:19.632338+0000","last_became_peered":"2024-06-04T04:54:19.632338+0000","last_unstale":"2024-06-04T04:54:19.632790+0000","last_undegraded":"2024-06-04T04:54:19.632790+0000","last_fullsized":"2024-06-04T04:54:19.632790+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:44:00.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'5","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512034+0000","last_change":"2024-06-04T04:53:33.346111+0000","last_active":"2024-06-04T04:54:19.512034+0000","last_peered":"2024-06-04T04:54:19.512034+0000","last_clean":"2024-06-04T04:54:19.512034+0000","last_became_active":"2024-06-04T04:53:33.345773+0000","last_became_peered":"2024-06-04T04:53:33.345773+0000","last_unstale":"2024-06-04T04:54:19.512034+0000","last_undegraded":"2024-06-04T04:54:19.512034+0000","last_fullsized":"2024-06-04T04:54:19.512034+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:18.565583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.364617+0000","last_change":"2024-06-04T04:53:36.373369+0000","last_active":"2024-06-04T04:53:38.364617+0000","last_peered":"2024-06-04T04:53:38.364617+0000","last_clean":"2024-06-04T04:53:38.364617+0000","last_became_active":"2024-06-04T04:53:36.373001+0000","last_became_peered":"2024-06-04T04:53:36.373001+0000","last_unstale":"2024-06-04T04:53:38.364617+0000","last_undegraded":"2024-06-04T04:53:38.364617+0000","last_fullsized":"2024-06-04T04:53:38.364617+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:02:22.012643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":28,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:52.442028+0000","last_change":"2024-06-04T04:53:33.344358+0000","last_active":"2024-06-04T04:53:52.442028+0000","last_peered":"2024-06-04T04:53:52.442028+0000","last_clean":"2024-06-04T04:53:52.442028+0000","last_became_active":"2024-06-04T04:53:33.343980+0000","last_became_peered":"2024-06-04T04:53:33.343980+0000","last_unstale":"2024-06-04T04:53:52.442028+0000","last_undegraded":"2024-06-04T04:53:52.442028+0000","last_fullsized":"2024-06-04T04:53:52.442028+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:14.493138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512293+0000","last_change":"2024-06-04T04:53:36.371239+0000","last_active":"2024-06-04T04:54:19.512293+0000","last_peered":"2024-06-04T04:54:19.512293+0000","last_clean":"2024-06-04T04:54:19.512293+0000","last_became_active":"2024-06-04T04:53:36.370885+0000","last_became_peered":"2024-06-04T04:53:36.370885+0000","last_unstale":"2024-06-04T04:54:19.512293+0000","last_undegraded":"2024-06-04T04:54:19.512293+0000","last_fullsized":"2024-06-04T04:54:19.512293+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T04:54:29.753796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:51.429310+0000","last_change":"2024-06-04T04:53:33.338512+0000","last_active":"2024-06-04T04:53:51.429310+0000","last_peered":"2024-06-04T04:53:51.429310+0000","last_clean":"2024-06-04T04:53:51.429310+0000","last_became_active":"2024-06-04T04:53:33.338180+0000","last_became_peered":"2024-06-04T04:53:33.338180+0000","last_unstale":"2024-06-04T04:53:51.429310+0000","last_undegraded":"2024-06-04T04:53:51.429310+0000","last_fullsized":"2024-06-04T04:53:51.429310+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:57:13.406479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504521+0000","last_change":"2024-06-04T04:53:36.377316+0000","last_active":"2024-06-04T04:54:18.504521+0000","last_peered":"2024-06-04T04:54:18.504521+0000","last_clean":"2024-06-04T04:54:18.504521+0000","last_became_active":"2024-06-04T04:53:36.376992+0000","last_became_peered":"2024-06-04T04:53:36.376992+0000","last_unstale":"2024-06-04T04:54:18.504521+0000","last_undegraded":"2024-06-04T04:54:18.504521+0000","last_fullsized":"2024-06-04T04:54:18.504521+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:33.145631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511696+0000","last_change":"2024-06-04T04:53:33.348305+0000","last_active":"2024-06-04T04:54:19.511696+0000","last_peered":"2024-06-04T04:54:19.511696+0000","last_clean":"2024-06-04T04:54:19.511696+0000","last_became_active":"2024-06-04T04:53:33.347948+0000","last_became_peered":"2024-06-04T04:53:33.347948+0000","last_unstale":"2024-06-04T04:54:19.511696+0000","last_undegraded":"2024-06-04T04:54:19.511696+0000","last_fullsized":"2024-06-04T04:54:19.511696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:04:13.669274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.374246+0000","last_change":"2024-06-04T04:53:36.367955+0000","last_active":"2024-06-04T04:53:38.374246+0000","last_peered":"2024-06-04T04:53:38.374246+0000","last_clean":"2024-06-04T04:53:38.374246+0000","last_became_active":"2024-06-04T04:53:36.367628+0000","last_became_peered":"2024-06-04T04:53:36.367628+0000","last_unstale":"2024-06-04T04:53:38.374246+0000","last_undegraded":"2024-06-04T04:53:38.374246+0000","last_fullsized":"2024-06-04T04:53:38.374246+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:33:26.185498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'4","reported_seq":73,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:54:17.486475+0000","last_change":"2024-06-04T04:53:33.349471+0000","last_active":"2024-06-04T04:54:17.486475+0000","last_peered":"2024-06-04T04:54:17.486475+0000","last_clean":"2024-06-04T04:54:17.486475+0000","last_became_active":"2024-06-04T04:53:33.349105+0000","last_became_peered":"2024-06-04T04:53:33.349105+0000","last_unstale":"2024-06-04T04:54:17.486475+0000","last_undegraded":"2024-06-04T04:54:17.486475+0000","last_fullsized":"2024-06-04T04:54:17.486475+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:31:35.584275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":48,"num_read_kb":48,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.364416+0000","last_change":"2024-06-04T04:53:36.367249+0000","last_active":"2024-06-04T04:53:38.364416+0000","last_peered":"2024-06-04T04:53:38.364416+0000","last_clean":"2024-06-04T04:53:38.364416+0000","last_became_active":"2024-06-04T04:53:36.366863+0000","last_became_peered":"2024-06-04T04:53:36.366863+0000","last_unstale":"2024-06-04T04:53:38.364416+0000","last_undegraded":"2024-06-04T04:53:38.364416+0000","last_fullsized":"2024-06-04T04:53:38.364416+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:13:17.308321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510836+0000","last_change":"2024-06-04T04:53:33.346778+0000","last_active":"2024-06-04T04:54:19.510836+0000","last_peered":"2024-06-04T04:54:19.510836+0000","last_clean":"2024-06-04T04:54:19.510836+0000","last_became_active":"2024-06-04T04:53:33.346376+0000","last_became_peered":"2024-06-04T04:53:33.346376+0000","last_unstale":"2024-06-04T04:54:19.510836+0000","last_undegraded":"2024-06-04T04:54:19.510836+0000","last_fullsized":"2024-06-04T04:54:19.510836+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:03:16.973822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.376706+0000","last_change":"2024-06-04T04:53:36.367653+0000","last_active":"2024-06-04T04:53:38.376706+0000","last_peered":"2024-06-04T04:53:38.376706+0000","last_clean":"2024-06-04T04:53:38.376706+0000","last_became_active":"2024-06-04T04:53:36.367175+0000","last_became_peered":"2024-06-04T04:53:36.367175+0000","last_unstale":"2024-06-04T04:53:38.376706+0000","last_undegraded":"2024-06-04T04:53:38.376706+0000","last_fullsized":"2024-06-04T04:53:38.376706+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:39:48.165273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512326+0000","last_change":"2024-06-04T04:53:33.354989+0000","last_active":"2024-06-04T04:54:19.512326+0000","last_peered":"2024-06-04T04:54:19.512326+0000","last_clean":"2024-06-04T04:54:19.512326+0000","last_became_active":"2024-06-04T04:53:33.354613+0000","last_became_peered":"2024-06-04T04:53:33.354613+0000","last_unstale":"2024-06-04T04:54:19.512326+0000","last_undegraded":"2024-06-04T04:54:19.512326+0000","last_fullsized":"2024-06-04T04:54:19.512326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:19:59.398763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.364157+0000","last_change":"2024-06-04T04:53:36.372216+0000","last_active":"2024-06-04T04:53:38.364157+0000","last_peered":"2024-06-04T04:53:38.364157+0000","last_clean":"2024-06-04T04:53:38.364157+0000","last_became_active":"2024-06-04T04:53:36.371820+0000","last_became_peered":"2024-06-04T04:53:36.371820+0000","last_unstale":"2024-06-04T04:53:38.364157+0000","last_undegraded":"2024-06-04T04:53:38.364157+0000","last_fullsized":"2024-06-04T04:53:38.364157+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:55:59.887963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"79'27","reported_seq":103,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:54:17.490520+0000","last_change":"2024-06-04T04:53:33.345493+0000","last_active":"2024-06-04T04:54:17.490520+0000","last_peered":"2024-06-04T04:54:17.490520+0000","last_clean":"2024-06-04T04:54:17.490520+0000","last_became_active":"2024-06-04T04:53:33.345102+0000","last_became_peered":"2024-06-04T04:53:33.345102+0000","last_unstale":"2024-06-04T04:54:17.490520+0000","last_undegraded":"2024-06-04T04:54:17.490520+0000","last_fullsized":"2024-06-04T04:54:17.490520+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:50:27.853391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54372,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":55,"num_read_kb":80,"num_write":29,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.364618+0000","last_change":"2024-06-04T04:53:36.382070+0000","last_active":"2024-06-04T04:53:38.364618+0000","last_peered":"2024-06-04T04:53:38.364618+0000","last_clean":"2024-06-04T04:53:38.364618+0000","last_became_active":"2024-06-04T04:53:36.379144+0000","last_became_peered":"2024-06-04T04:53:36.379144+0000","last_unstale":"2024-06-04T04:53:38.364618+0000","last_undegraded":"2024-06-04T04:53:38.364618+0000","last_fullsized":"2024-06-04T04:53:38.364618+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:16.702709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.508806+0000","last_change":"2024-06-04T04:53:33.344726+0000","last_active":"2024-06-04T04:54:18.508806+0000","last_peered":"2024-06-04T04:54:18.508806+0000","last_clean":"2024-06-04T04:54:18.508806+0000","last_became_active":"2024-06-04T04:53:33.344360+0000","last_became_peered":"2024-06-04T04:53:33.344360+0000","last_unstale":"2024-06-04T04:54:18.508806+0000","last_undegraded":"2024-06-04T04:54:18.508806+0000","last_fullsized":"2024-06-04T04:54:18.508806+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:44:56.784808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.375657+0000","last_change":"2024-06-04T04:53:36.379036+0000","last_active":"2024-06-04T04:53:38.375657+0000","last_peered":"2024-06-04T04:53:38.375657+0000","last_clean":"2024-06-04T04:53:38.375657+0000","last_became_active":"2024-06-04T04:53:36.378681+0000","last_became_peered":"2024-06-04T04:53:36.378681+0000","last_unstale":"2024-06-04T04:53:38.375657+0000","last_undegraded":"2024-06-04T04:53:38.375657+0000","last_fullsized":"2024-06-04T04:53:38.375657+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:07:49.704731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511967+0000","last_change":"2024-06-04T04:53:33.345915+0000","last_active":"2024-06-04T04:54:19.511967+0000","last_peered":"2024-06-04T04:54:19.511967+0000","last_clean":"2024-06-04T04:54:19.511967+0000","last_became_active":"2024-06-04T04:53:33.345537+0000","last_became_peered":"2024-06-04T04:53:33.345537+0000","last_unstale":"2024-06-04T04:54:19.511967+0000","last_undegraded":"2024-06-04T04:54:19.511967+0000","last_fullsized":"2024-06-04T04:54:19.511967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:06:44.086968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510431+0000","last_change":"2024-06-04T04:53:36.379094+0000","last_active":"2024-06-04T04:54:19.510431+0000","last_peered":"2024-06-04T04:54:19.510431+0000","last_clean":"2024-06-04T04:54:19.510431+0000","last_became_active":"2024-06-04T04:53:36.378763+0000","last_became_peered":"2024-06-04T04:53:36.378763+0000","last_unstale":"2024-06-04T04:54:19.510431+0000","last_undegraded":"2024-06-04T04:54:19.510431+0000","last_fullsized":"2024-06-04T04:54:19.510431+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:16.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.630622+0000","last_change":"2024-06-04T04:53:33.341342+0000","last_active":"2024-06-04T04:54:19.630622+0000","last_peered":"2024-06-04T04:54:19.630622+0000","last_clean":"2024-06-04T04:54:19.630622+0000","last_became_active":"2024-06-04T04:53:33.340862+0000","last_became_peered":"2024-06-04T04:53:33.340862+0000","last_unstale":"2024-06-04T04:54:19.630622+0000","last_undegraded":"2024-06-04T04:54:19.630622+0000","last_fullsized":"2024-06-04T04:54:19.630622+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:26.818063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510293+0000","last_change":"2024-06-04T04:53:36.383124+0000","last_active":"2024-06-04T04:54:19.510293+0000","last_peered":"2024-06-04T04:54:19.510293+0000","last_clean":"2024-06-04T04:54:19.510293+0000","last_became_active":"2024-06-04T04:53:36.382449+0000","last_became_peered":"2024-06-04T04:53:36.382449+0000","last_unstale":"2024-06-04T04:54:19.510293+0000","last_undegraded":"2024-06-04T04:54:19.510293+0000","last_fullsized":"2024-06-04T04:54:19.510293+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:52:40.776003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:53.444957+0000","last_change":"2024-06-04T04:53:33.354556+0000","last_active":"2024-06-04T04:53:53.444957+0000","last_peered":"2024-06-04T04:53:53.444957+0000","last_clean":"2024-06-04T04:53:53.444957+0000","last_became_active":"2024-06-04T04:53:33.354238+0000","last_became_peered":"2024-06-04T04:53:33.354238+0000","last_unstale":"2024-06-04T04:53:53.444957+0000","last_undegraded":"2024-06-04T04:53:53.444957+0000","last_fullsized":"2024-06-04T04:53:53.444957+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:54.697374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511906+0000","last_change":"2024-06-04T04:53:36.371387+0000","last_active":"2024-06-04T04:54:19.511906+0000","last_peered":"2024-06-04T04:54:19.511906+0000","last_clean":"2024-06-04T04:54:19.511906+0000","last_became_active":"2024-06-04T04:53:36.370852+0000","last_became_peered":"2024-06-04T04:53:36.370852+0000","last_unstale":"2024-06-04T04:54:19.511906+0000","last_undegraded":"2024-06-04T04:54:19.511906+0000","last_fullsized":"2024-06-04T04:54:19.511906+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:44:06.956210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":25,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:53.426358+0000","last_change":"2024-06-04T04:53:33.349982+0000","last_active":"2024-06-04T04:53:53.426358+0000","last_peered":"2024-06-04T04:53:53.426358+0000","last_clean":"2024-06-04T04:53:53.426358+0000","last_became_active":"2024-06-04T04:53:33.349633+0000","last_became_peered":"2024-06-04T04:53:33.349633+0000","last_unstale":"2024-06-04T04:53:53.426358+0000","last_undegraded":"2024-06-04T04:53:53.426358+0000","last_fullsized":"2024-06-04T04:53:53.426358+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:40.107426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.380135+0000","last_change":"2024-06-04T04:53:36.382313+0000","last_active":"2024-06-04T04:53:38.380135+0000","last_peered":"2024-06-04T04:53:38.380135+0000","last_clean":"2024-06-04T04:53:38.380135+0000","last_became_active":"2024-06-04T04:53:36.379638+0000","last_became_peered":"2024-06-04T04:53:36.379638+0000","last_unstale":"2024-06-04T04:53:38.380135+0000","last_undegraded":"2024-06-04T04:53:38.380135+0000","last_fullsized":"2024-06-04T04:53:38.380135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:43:42.527262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.374221+0000","last_change":"2024-06-04T04:53:33.354766+0000","last_active":"2024-06-04T04:53:38.374221+0000","last_peered":"2024-06-04T04:53:38.374221+0000","last_clean":"2024-06-04T04:53:38.374221+0000","last_became_active":"2024-06-04T04:53:33.354424+0000","last_became_peered":"2024-06-04T04:53:33.354424+0000","last_unstale":"2024-06-04T04:53:38.374221+0000","last_undegraded":"2024-06-04T04:53:38.374221+0000","last_fullsized":"2024-06-04T04:53:38.374221+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:01:06.706010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.378389+0000","last_change":"2024-06-04T04:53:36.369904+0000","last_active":"2024-06-04T04:53:38.378389+0000","last_peered":"2024-06-04T04:53:38.378389+0000","last_clean":"2024-06-04T04:53:38.378389+0000","last_became_active":"2024-06-04T04:53:36.367929+0000","last_became_peered":"2024-06-04T04:53:36.367929+0000","last_unstale":"2024-06-04T04:53:38.378389+0000","last_undegraded":"2024-06-04T04:53:38.378389+0000","last_fullsized":"2024-06-04T04:53:38.378389+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:37:08.721569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'6","reported_seq":60,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510585+0000","last_change":"2024-06-04T04:53:33.342723+0000","last_active":"2024-06-04T04:54:19.510585+0000","last_peered":"2024-06-04T04:54:19.510585+0000","last_clean":"2024-06-04T04:54:19.510585+0000","last_became_active":"2024-06-04T04:53:33.342344+0000","last_became_peered":"2024-06-04T04:53:33.342344+0000","last_unstale":"2024-06-04T04:54:19.510585+0000","last_undegraded":"2024-06-04T04:54:19.510585+0000","last_fullsized":"2024-06-04T04:54:19.510585+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:59:50.147517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.376913+0000","last_change":"2024-06-04T04:53:36.372516+0000","last_active":"2024-06-04T04:53:38.376913+0000","last_peered":"2024-06-04T04:53:38.376913+0000","last_clean":"2024-06-04T04:53:38.376913+0000","last_became_active":"2024-06-04T04:53:36.370698+0000","last_became_peered":"2024-06-04T04:53:36.370698+0000","last_unstale":"2024-06-04T04:53:38.376913+0000","last_undegraded":"2024-06-04T04:53:38.376913+0000","last_fullsized":"2024-06-04T04:53:38.376913+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:29:10.449002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512091+0000","last_change":"2024-06-04T04:53:36.371236+0000","last_active":"2024-06-04T04:54:19.512091+0000","last_peered":"2024-06-04T04:54:19.512091+0000","last_clean":"2024-06-04T04:54:19.512091+0000","last_became_active":"2024-06-04T04:53:36.370876+0000","last_became_peered":"2024-06-04T04:53:36.370876+0000","last_unstale":"2024-06-04T04:54:19.512091+0000","last_undegraded":"2024-06-04T04:54:19.512091+0000","last_fullsized":"2024-06-04T04:54:19.512091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:15:56.751317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":49,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:54:14.482055+0000","last_change":"2024-06-04T04:53:33.355642+0000","last_active":"2024-06-04T04:54:14.482055+0000","last_peered":"2024-06-04T04:54:14.482055+0000","last_clean":"2024-06-04T04:54:14.482055+0000","last_became_active":"2024-06-04T04:53:33.355187+0000","last_became_peered":"2024-06-04T04:53:33.355187+0000","last_unstale":"2024-06-04T04:54:14.482055+0000","last_undegraded":"2024-06-04T04:54:14.482055+0000","last_fullsized":"2024-06-04T04:54:14.482055+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:32:28.331518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.362788+0000","last_change":"2024-06-04T04:53:36.370656+0000","last_active":"2024-06-04T04:53:38.362788+0000","last_peered":"2024-06-04T04:53:38.362788+0000","last_clean":"2024-06-04T04:53:38.362788+0000","last_became_active":"2024-06-04T04:53:36.370350+0000","last_became_peered":"2024-06-04T04:53:36.370350+0000","last_unstale":"2024-06-04T04:53:38.362788+0000","last_undegraded":"2024-06-04T04:53:38.362788+0000","last_fullsized":"2024-06-04T04:53:38.362788+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:03:16.609933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631469+0000","last_change":"2024-06-04T04:53:33.340653+0000","last_active":"2024-06-04T04:54:19.631469+0000","last_peered":"2024-06-04T04:54:19.631469+0000","last_clean":"2024-06-04T04:54:19.631469+0000","last_became_active":"2024-06-04T04:53:33.340250+0000","last_became_peered":"2024-06-04T04:53:33.340250+0000","last_unstale":"2024-06-04T04:54:19.631469+0000","last_undegraded":"2024-06-04T04:54:19.631469+0000","last_fullsized":"2024-06-04T04:54:19.631469+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:46:20.329514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":99,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510637+0000","last_change":"2024-06-04T04:52:09.181155+0000","last_active":"2024-06-04T04:54:19.510637+0000","last_peered":"2024-06-04T04:54:19.510637+0000","last_clean":"2024-06-04T04:54:19.510637+0000","last_became_active":"2024-06-04T04:52:09.177002+0000","last_became_peered":"2024-06-04T04:52:09.177002+0000","last_unstale":"2024-06-04T04:54:19.510637+0000","last_undegraded":"2024-06-04T04:54:19.510637+0000","last_fullsized":"2024-06-04T04:54:19.510637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_clean_scrub_stamp":"2024-06-04T04:48:44.631948+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:46:57.735302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.375147+0000","last_change":"2024-06-04T04:53:36.372634+0000","last_active":"2024-06-04T04:53:38.375147+0000","last_peered":"2024-06-04T04:53:38.375147+0000","last_clean":"2024-06-04T04:53:38.375147+0000","last_became_active":"2024-06-04T04:53:36.372290+0000","last_became_peered":"2024-06-04T04:53:36.372290+0000","last_unstale":"2024-06-04T04:53:38.375147+0000","last_undegraded":"2024-06-04T04:53:38.375147+0000","last_fullsized":"2024-06-04T04:53:38.375147+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:14:07.460382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511010+0000","last_change":"2024-06-04T04:53:33.348760+0000","last_active":"2024-06-04T04:54:19.511010+0000","last_peered":"2024-06-04T04:54:19.511010+0000","last_clean":"2024-06-04T04:54:19.511010+0000","last_became_active":"2024-06-04T04:53:33.348347+0000","last_became_peered":"2024-06-04T04:53:33.348347+0000","last_unstale":"2024-06-04T04:54:19.511010+0000","last_undegraded":"2024-06-04T04:54:19.511010+0000","last_fullsized":"2024-06-04T04:54:19.511010+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:36:20.641061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:53.447658+0000","last_change":"2024-06-04T04:53:33.354840+0000","last_active":"2024-06-04T04:53:53.447658+0000","last_peered":"2024-06-04T04:53:53.447658+0000","last_clean":"2024-06-04T04:53:53.447658+0000","last_became_active":"2024-06-04T04:53:33.354287+0000","last_became_peered":"2024-06-04T04:53:33.354287+0000","last_unstale":"2024-06-04T04:53:53.447658+0000","last_undegraded":"2024-06-04T04:53:53.447658+0000","last_fullsized":"2024-06-04T04:53:53.447658+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:42.550098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.375469+0000","last_change":"2024-06-04T04:53:36.372678+0000","last_active":"2024-06-04T04:53:38.375469+0000","last_peered":"2024-06-04T04:53:38.375469+0000","last_clean":"2024-06-04T04:53:38.375469+0000","last_became_active":"2024-06-04T04:53:36.372312+0000","last_became_peered":"2024-06-04T04:53:36.372312+0000","last_unstale":"2024-06-04T04:53:38.375469+0000","last_undegraded":"2024-06-04T04:53:38.375469+0000","last_fullsized":"2024-06-04T04:53:38.375469+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:34:41.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"79'14","reported_seq":36,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:54:15.377798+0000","last_change":"2024-06-04T04:53:33.348440+0000","last_active":"2024-06-04T04:54:15.377798+0000","last_peered":"2024-06-04T04:54:15.377798+0000","last_clean":"2024-06-04T04:54:15.377798+0000","last_became_active":"2024-06-04T04:53:33.348131+0000","last_became_peered":"2024-06-04T04:53:33.348131+0000","last_unstale":"2024-06-04T04:54:15.377798+0000","last_undegraded":"2024-06-04T04:54:15.377798+0000","last_fullsized":"2024-06-04T04:54:15.377798+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:14:11.552286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.505140+0000","last_change":"2024-06-04T04:53:36.378669+0000","last_active":"2024-06-04T04:54:18.505140+0000","last_peered":"2024-06-04T04:54:18.505140+0000","last_clean":"2024-06-04T04:54:18.505140+0000","last_became_active":"2024-06-04T04:53:36.378233+0000","last_became_peered":"2024-06-04T04:53:36.378233+0000","last_unstale":"2024-06-04T04:54:18.505140+0000","last_undegraded":"2024-06-04T04:54:18.505140+0000","last_fullsized":"2024-06-04T04:54:18.505140+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:36:45.870924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631647+0000","last_change":"2024-06-04T04:53:33.345078+0000","last_active":"2024-06-04T04:54:19.631647+0000","last_peered":"2024-06-04T04:54:19.631647+0000","last_clean":"2024-06-04T04:54:19.631647+0000","last_became_active":"2024-06-04T04:53:33.344708+0000","last_became_peered":"2024-06-04T04:53:33.344708+0000","last_unstale":"2024-06-04T04:54:19.631647+0000","last_undegraded":"2024-06-04T04:54:19.631647+0000","last_fullsized":"2024-06-04T04:54:19.631647+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:45:14.010758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.376260+0000","last_change":"2024-06-04T04:53:36.364817+0000","last_active":"2024-06-04T04:53:38.376260+0000","last_peered":"2024-06-04T04:53:38.376260+0000","last_clean":"2024-06-04T04:53:38.376260+0000","last_became_active":"2024-06-04T04:53:36.364499+0000","last_became_peered":"2024-06-04T04:53:36.364499+0000","last_unstale":"2024-06-04T04:53:38.376260+0000","last_undegraded":"2024-06-04T04:53:38.376260+0000","last_fullsized":"2024-06-04T04:53:38.376260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:39:30.518028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:50.422786+0000","last_change":"2024-06-04T04:53:33.354527+0000","last_active":"2024-06-04T04:53:50.422786+0000","last_peered":"2024-06-04T04:53:50.422786+0000","last_clean":"2024-06-04T04:53:50.422786+0000","last_became_active":"2024-06-04T04:53:33.354175+0000","last_became_peered":"2024-06-04T04:53:33.354175+0000","last_unstale":"2024-06-04T04:53:50.422786+0000","last_undegraded":"2024-06-04T04:53:50.422786+0000","last_fullsized":"2024-06-04T04:53:50.422786+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:33:08.125428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.364210+0000","last_change":"2024-06-04T04:53:36.373248+0000","last_active":"2024-06-04T04:53:38.364210+0000","last_peered":"2024-06-04T04:53:38.364210+0000","last_clean":"2024-06-04T04:53:38.364210+0000","last_became_active":"2024-06-04T04:53:36.372915+0000","last_became_peered":"2024-06-04T04:53:36.372915+0000","last_unstale":"2024-06-04T04:53:38.364210+0000","last_undegraded":"2024-06-04T04:53:38.364210+0000","last_fullsized":"2024-06-04T04:53:38.364210+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:23:11.689349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504542+0000","last_change":"2024-06-04T04:53:33.340427+0000","last_active":"2024-06-04T04:54:18.504542+0000","last_peered":"2024-06-04T04:54:18.504542+0000","last_clean":"2024-06-04T04:54:18.504542+0000","last_became_active":"2024-06-04T04:53:33.340022+0000","last_became_peered":"2024-06-04T04:53:33.340022+0000","last_unstale":"2024-06-04T04:54:18.504542+0000","last_undegraded":"2024-06-04T04:54:18.504542+0000","last_fullsized":"2024-06-04T04:54:18.504542+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:56:16.496471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.379360+0000","last_change":"2024-06-04T04:53:36.380114+0000","last_active":"2024-06-04T04:53:38.379360+0000","last_peered":"2024-06-04T04:53:38.379360+0000","last_clean":"2024-06-04T04:53:38.379360+0000","last_became_active":"2024-06-04T04:53:36.379392+0000","last_became_peered":"2024-06-04T04:53:36.379392+0000","last_unstale":"2024-06-04T04:53:38.379360+0000","last_undegraded":"2024-06-04T04:53:38.379360+0000","last_fullsized":"2024-06-04T04:53:38.379360+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:41:48.776439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:52.442028+0000","last_change":"2024-06-04T04:53:33.347565+0000","last_active":"2024-06-04T04:53:52.442028+0000","last_peered":"2024-06-04T04:53:52.442028+0000","last_clean":"2024-06-04T04:53:52.442028+0000","last_became_active":"2024-06-04T04:53:33.347172+0000","last_became_peered":"2024-06-04T04:53:33.347172+0000","last_unstale":"2024-06-04T04:53:52.442028+0000","last_undegraded":"2024-06-04T04:53:52.442028+0000","last_fullsized":"2024-06-04T04:53:52.442028+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:17:11.295293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504890+0000","last_change":"2024-06-04T04:53:36.377420+0000","last_active":"2024-06-04T04:54:18.504890+0000","last_peered":"2024-06-04T04:54:18.504890+0000","last_clean":"2024-06-04T04:54:18.504890+0000","last_became_active":"2024-06-04T04:53:36.377109+0000","last_became_peered":"2024-06-04T04:53:36.377109+0000","last_unstale":"2024-06-04T04:54:18.504890+0000","last_undegraded":"2024-06-04T04:54:18.504890+0000","last_fullsized":"2024-06-04T04:54:18.504890+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:47:12.102396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:50.421065+0000","last_change":"2024-06-04T04:53:33.349285+0000","last_active":"2024-06-04T04:53:50.421065+0000","last_peered":"2024-06-04T04:53:50.421065+0000","last_clean":"2024-06-04T04:53:50.421065+0000","last_became_active":"2024-06-04T04:53:33.348929+0000","last_became_peered":"2024-06-04T04:53:33.348929+0000","last_unstale":"2024-06-04T04:53:50.421065+0000","last_undegraded":"2024-06-04T04:53:50.421065+0000","last_fullsized":"2024-06-04T04:53:50.421065+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:15:32.933106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510562+0000","last_change":"2024-06-04T04:53:36.381187+0000","last_active":"2024-06-04T04:54:19.510562+0000","last_peered":"2024-06-04T04:54:19.510562+0000","last_clean":"2024-06-04T04:54:19.510562+0000","last_became_active":"2024-06-04T04:53:36.380591+0000","last_became_peered":"2024-06-04T04:53:36.380591+0000","last_unstale":"2024-06-04T04:54:19.510562+0000","last_undegraded":"2024-06-04T04:54:19.510562+0000","last_fullsized":"2024-06-04T04:54:19.510562+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:26:43.122339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'5","reported_seq":55,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.509038+0000","last_change":"2024-06-04T04:53:33.355908+0000","last_active":"2024-06-04T04:54:18.509038+0000","last_peered":"2024-06-04T04:54:18.509038+0000","last_clean":"2024-06-04T04:54:18.509038+0000","last_became_active":"2024-06-04T04:53:33.355522+0000","last_became_peered":"2024-06-04T04:53:33.355522+0000","last_unstale":"2024-06-04T04:54:18.509038+0000","last_undegraded":"2024-06-04T04:54:18.509038+0000","last_fullsized":"2024-06-04T04:54:18.509038+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:16:44.639431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510919+0000","last_change":"2024-06-04T04:53:36.382737+0000","last_active":"2024-06-04T04:54:19.510919+0000","last_peered":"2024-06-04T04:54:19.510919+0000","last_clean":"2024-06-04T04:54:19.510919+0000","last_became_active":"2024-06-04T04:53:36.382353+0000","last_became_peered":"2024-06-04T04:53:36.382353+0000","last_unstale":"2024-06-04T04:54:19.510919+0000","last_undegraded":"2024-06-04T04:54:19.510919+0000","last_fullsized":"2024-06-04T04:54:19.510919+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:18:01.476058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512086+0000","last_change":"2024-06-04T04:53:33.347273+0000","last_active":"2024-06-04T04:54:19.512086+0000","last_peered":"2024-06-04T04:54:19.512086+0000","last_clean":"2024-06-04T04:54:19.512086+0000","last_became_active":"2024-06-04T04:53:33.346915+0000","last_became_peered":"2024-06-04T04:53:33.346915+0000","last_unstale":"2024-06-04T04:54:19.512086+0000","last_undegraded":"2024-06-04T04:54:19.512086+0000","last_fullsized":"2024-06-04T04:54:19.512086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:17.235484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"79'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510494+0000","last_change":"2024-06-04T04:53:36.383454+0000","last_active":"2024-06-04T04:54:19.510494+0000","last_peered":"2024-06-04T04:54:19.510494+0000","last_clean":"2024-06-04T04:54:19.510494+0000","last_became_active":"2024-06-04T04:53:36.383105+0000","last_became_peered":"2024-06-04T04:53:36.383105+0000","last_unstale":"2024-06-04T04:54:19.510494+0000","last_undegraded":"2024-06-04T04:54:19.510494+0000","last_fullsized":"2024-06-04T04:54:19.510494+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:04:02.147688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:53.444425+0000","last_change":"2024-06-04T04:53:33.345435+0000","last_active":"2024-06-04T04:53:53.444425+0000","last_peered":"2024-06-04T04:53:53.444425+0000","last_clean":"2024-06-04T04:53:53.444425+0000","last_became_active":"2024-06-04T04:53:33.345077+0000","last_became_peered":"2024-06-04T04:53:33.345077+0000","last_unstale":"2024-06-04T04:53:53.444425+0000","last_undegraded":"2024-06-04T04:53:53.444425+0000","last_fullsized":"2024-06-04T04:53:53.444425+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:20:13.992259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510565+0000","last_change":"2024-06-04T04:53:36.385238+0000","last_active":"2024-06-04T04:54:19.510565+0000","last_peered":"2024-06-04T04:54:19.510565+0000","last_clean":"2024-06-04T04:54:19.510565+0000","last_became_active":"2024-06-04T04:53:36.384899+0000","last_became_peered":"2024-06-04T04:53:36.384899+0000","last_unstale":"2024-06-04T04:54:19.510565+0000","last_undegraded":"2024-06-04T04:54:19.510565+0000","last_fullsized":"2024-06-04T04:54:19.510565+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:09:42.629340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.379437+0000","last_change":"2024-06-04T04:53:36.383450+0000","last_active":"2024-06-04T04:53:38.379437+0000","last_peered":"2024-06-04T04:53:38.379437+0000","last_clean":"2024-06-04T04:53:38.379437+0000","last_became_active":"2024-06-04T04:53:36.382678+0000","last_became_peered":"2024-06-04T04:53:36.382678+0000","last_unstale":"2024-06-04T04:53:38.379437+0000","last_undegraded":"2024-06-04T04:53:38.379437+0000","last_fullsized":"2024-06-04T04:53:38.379437+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:36:28.932636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":79,"state":"peering","last_fresh":"2024-06-04T04:53:38.658184+0000","last_change":"2024-06-04T04:54:18.509509+0000","last_active":"2024-06-04T04:54:18.509509+0000","last_peered":"2024-06-04T04:53:38.658184+0000","last_clean":"2024-06-04T04:53:38.658184+0000","last_became_active":"2024-06-04T04:53:36.375049+0000","last_became_peered":"2024-06-04T04:53:36.375049+0000","last_unstale":"2024-06-04T04:53:38.658184+0000","last_undegraded":"2024-06-04T04:53:38.658184+0000","last_fullsized":"2024-06-04T04:53:38.658184+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5,9],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.508282+0000","last_change":"2024-06-04T04:53:36.381408+0000","last_active":"2024-06-04T04:54:18.508282+0000","last_peered":"2024-06-04T04:54:18.508282+0000","last_clean":"2024-06-04T04:54:18.508282+0000","last_became_active":"2024-06-04T04:53:36.381078+0000","last_became_peered":"2024-06-04T04:53:36.381078+0000","last_unstale":"2024-06-04T04:54:18.508282+0000","last_undegraded":"2024-06-04T04:54:18.508282+0000","last_fullsized":"2024-06-04T04:54:18.508282+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:28:48.988591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.375256+0000","last_change":"2024-06-04T04:53:36.360168+0000","last_active":"2024-06-04T04:53:38.375256+0000","last_peered":"2024-06-04T04:53:38.375256+0000","last_clean":"2024-06-04T04:53:38.375256+0000","last_became_active":"2024-06-04T04:53:36.359691+0000","last_became_peered":"2024-06-04T04:53:36.359691+0000","last_unstale":"2024-06-04T04:53:38.375256+0000","last_undegraded":"2024-06-04T04:53:38.375256+0000","last_fullsized":"2024-06-04T04:53:38.375256+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:55:17.418339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.376555+0000","last_change":"2024-06-04T04:53:36.363018+0000","last_active":"2024-06-04T04:53:38.376555+0000","last_peered":"2024-06-04T04:53:38.376555+0000","last_clean":"2024-06-04T04:53:38.376555+0000","last_became_active":"2024-06-04T04:53:36.361595+0000","last_became_peered":"2024-06-04T04:53:36.361595+0000","last_unstale":"2024-06-04T04:53:38.376555+0000","last_undegraded":"2024-06-04T04:53:38.376555+0000","last_fullsized":"2024-06-04T04:53:38.376555+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:00:21.377322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.380037+0000","last_change":"2024-06-04T04:53:36.375377+0000","last_active":"2024-06-04T04:53:38.380037+0000","last_peered":"2024-06-04T04:53:38.380037+0000","last_clean":"2024-06-04T04:53:38.380037+0000","last_became_active":"2024-06-04T04:53:36.373459+0000","last_became_peered":"2024-06-04T04:53:36.373459+0000","last_unstale":"2024-06-04T04:53:38.380037+0000","last_undegraded":"2024-06-04T04:53:38.380037+0000","last_fullsized":"2024-06-04T04:53:38.380037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:51:31.686969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.509760+0000","last_change":"2024-06-04T04:53:36.373850+0000","last_active":"2024-06-04T04:54:18.509760+0000","last_peered":"2024-06-04T04:54:18.509760+0000","last_clean":"2024-06-04T04:54:18.509760+0000","last_became_active":"2024-06-04T04:53:36.371951+0000","last_became_peered":"2024-06-04T04:53:36.371951+0000","last_unstale":"2024-06-04T04:54:18.509760+0000","last_undegraded":"2024-06-04T04:54:18.509760+0000","last_fullsized":"2024-06-04T04:54:18.509760+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:06.452703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.505025+0000","last_change":"2024-06-04T04:53:36.374871+0000","last_active":"2024-06-04T04:54:18.505025+0000","last_peered":"2024-06-04T04:54:18.505025+0000","last_clean":"2024-06-04T04:54:18.505025+0000","last_became_active":"2024-06-04T04:53:36.374486+0000","last_became_peered":"2024-06-04T04:53:36.374486+0000","last_unstale":"2024-06-04T04:54:18.505025+0000","last_undegraded":"2024-06-04T04:54:18.505025+0000","last_fullsized":"2024-06-04T04:54:18.505025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:15:13.337374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.514431+0000","last_change":"2024-06-04T04:54:19.513540+0000","last_active":"2024-06-04T04:54:19.514431+0000","last_peered":"2024-06-04T04:54:19.514431+0000","last_clean":"2024-06-04T04:54:19.514431+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:54:19.514431+0000","last_undegraded":"2024-06-04T04:54:19.514431+0000","last_fullsized":"2024-06-04T04:54:19.514431+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:09:27.993585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504476+0000","last_change":"2024-06-04T04:53:36.373055+0000","last_active":"2024-06-04T04:54:18.504476+0000","last_peered":"2024-06-04T04:54:18.504476+0000","last_clean":"2024-06-04T04:54:18.504476+0000","last_became_active":"2024-06-04T04:53:36.372754+0000","last_became_peered":"2024-06-04T04:53:36.372754+0000","last_unstale":"2024-06-04T04:54:18.504476+0000","last_undegraded":"2024-06-04T04:54:18.504476+0000","last_fullsized":"2024-06-04T04:54:18.504476+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:22:27.554200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.364208+0000","last_change":"2024-06-04T04:53:36.378897+0000","last_active":"2024-06-04T04:53:38.364208+0000","last_peered":"2024-06-04T04:53:38.364208+0000","last_clean":"2024-06-04T04:53:38.364208+0000","last_became_active":"2024-06-04T04:53:36.378607+0000","last_became_peered":"2024-06-04T04:53:36.378607+0000","last_unstale":"2024-06-04T04:53:38.364208+0000","last_undegraded":"2024-06-04T04:53:38.364208+0000","last_fullsized":"2024-06-04T04:53:38.364208+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:03.395491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.376889+0000","last_change":"2024-06-04T04:53:36.378240+0000","last_active":"2024-06-04T04:53:38.376889+0000","last_peered":"2024-06-04T04:53:38.376889+0000","last_clean":"2024-06-04T04:53:38.376889+0000","last_became_active":"2024-06-04T04:53:36.377909+0000","last_became_peered":"2024-06-04T04:53:36.377909+0000","last_unstale":"2024-06-04T04:53:38.376889+0000","last_undegraded":"2024-06-04T04:53:38.376889+0000","last_fullsized":"2024-06-04T04:53:38.376889+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:15:25.288340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.379740+0000","last_change":"2024-06-04T04:53:36.379792+0000","last_active":"2024-06-04T04:53:38.379740+0000","last_peered":"2024-06-04T04:53:38.379740+0000","last_clean":"2024-06-04T04:53:38.379740+0000","last_became_active":"2024-06-04T04:53:36.379438+0000","last_became_peered":"2024-06-04T04:53:36.379438+0000","last_unstale":"2024-06-04T04:53:38.379740+0000","last_undegraded":"2024-06-04T04:53:38.379740+0000","last_fullsized":"2024-06-04T04:53:38.379740+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:23:57.617877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":214930,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61185,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":283,"num_read_kb":205,"num_write":238,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":194248,"data_compressed":5499,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":204,"ondisk_log_size":204,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678039,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27852,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717300,"statfs":{"total":95995035648,"available":95966515200,"internally_reserved":0,"allocated":856064,"data_stored":412268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874268,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":1057669,"data_compressed":37925,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037794,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":880640,"data_stored":412198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234023,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":412576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430251,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":1013853,"data_compressed":38968,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593777,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":412306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":467591,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":413254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411397,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":478113,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":892928,"data_stored":420499,"data_compressed":2876,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953532,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":876544,"data_stored":420535,"data_compressed":2876,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790007,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":1013140,"data_compressed":38968,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34470,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":56515,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":67037,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":56108,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-04T04:54:20.687 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-04T04:54:20.926 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:20.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:20 smithi112 ceph-mon[36453]: osdmap e81: 12 total, 12 up, 12 in 2024-06-04T04:54:20.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:20 smithi098 ceph-mon[35569]: osdmap e81: 12 total, 12 up, 12 in 2024-06-04T04:54:21.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:21 smithi070 ceph-mon[32631]: pgmap v312: 97 pgs: 1 peering, 96 active+clean; 637 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 4.0 KiB/s wr, 14 op/s 2024-06-04T04:54:21.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:21 smithi070 ceph-mon[32631]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:54:21.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:21 smithi112 ceph-mon[36453]: pgmap v312: 97 pgs: 1 peering, 96 active+clean; 637 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 4.0 KiB/s wr, 14 op/s 2024-06-04T04:54:21.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:21 smithi112 ceph-mon[36453]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:54:21.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:21 smithi098 ceph-mon[35569]: pgmap v312: 97 pgs: 1 peering, 96 active+clean; 637 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 4.0 KiB/s wr, 14 op/s 2024-06-04T04:54:21.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:21 smithi098 ceph-mon[35569]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:54:22.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.602+0000 7fda11057700 1 -- 172.21.15.70:0/1857131559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda0c103600 msgr2=0x7fda0c10f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:22.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.602+0000 7fda11057700 1 --2- 172.21.15.70:0/1857131559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda0c103600 0x7fda0c10f2e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fda0400b0a0 tx=0x7fda0400b3b0 comp rx=0 tx=0).stop 2024-06-04T04:54:22.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.602+0000 7fda11057700 1 -- 172.21.15.70:0/1857131559 shutdown_connections 2024-06-04T04:54:22.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.602+0000 7fda11057700 1 --2- 172.21.15.70:0/1857131559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda0c103600 0x7fda0c10f2e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:22.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.602+0000 7fda11057700 1 --2- 172.21.15.70:0/1857131559 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda0c1084c0 0x7fda0c1088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:22.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.602+0000 7fda11057700 1 --2- 172.21.15.70:0/1857131559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda0c102c40 0x7fda0c1030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:22.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.602+0000 7fda11057700 1 -- 172.21.15.70:0/1857131559 >> 172.21.15.70:0/1857131559 conn(0x7fda0c078ab0 msgr2=0x7fda0c0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:22.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.602+0000 7fda11057700 1 -- 172.21.15.70:0/1857131559 shutdown_connections 2024-06-04T04:54:22.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.602+0000 7fda11057700 1 -- 172.21.15.70:0/1857131559 wait complete. 2024-06-04T04:54:22.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.603+0000 7fda11057700 1 Processor -- start 2024-06-04T04:54:22.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.603+0000 7fda11057700 1 -- start start 2024-06-04T04:54:22.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.603+0000 7fda11057700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda0c102c40 0x7fda0c104110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:22.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.603+0000 7fda11057700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda0c103600 0x7fda0c104650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:22.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.603+0000 7fda11057700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda0c1084c0 0x7fda0c104b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:22.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.603+0000 7fda11057700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fda0c112480 con 0x7fda0c102c40 2024-06-04T04:54:22.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.603+0000 7fda11057700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fda0c112300 con 0x7fda0c103600 2024-06-04T04:54:22.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.603+0000 7fda11057700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fda0c112600 con 0x7fda0c1084c0 2024-06-04T04:54:22.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.603+0000 7fda0ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda0c102c40 0x7fda0c104110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:22.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.604+0000 7fda0ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda0c102c40 0x7fda0c104110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40372/0 (socket says 172.21.15.70:40372) 2024-06-04T04:54:22.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.604+0000 7fda0ad9d700 1 -- 172.21.15.70:0/994558584 learned_addr learned my addr 172.21.15.70:0/994558584 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:22.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.604+0000 7fda0a59c700 1 --2- 172.21.15.70:0/994558584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda0c103600 0x7fda0c104650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:22.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.604+0000 7fda0b59e700 1 --2- 172.21.15.70:0/994558584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda0c1084c0 0x7fda0c104b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:22.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.604+0000 7fda0ad9d700 1 -- 172.21.15.70:0/994558584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda0c1084c0 msgr2=0x7fda0c104b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:22.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.604+0000 7fda0ad9d700 1 --2- 172.21.15.70:0/994558584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda0c1084c0 0x7fda0c104b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:22.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.604+0000 7fda0ad9d700 1 -- 172.21.15.70:0/994558584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda0c103600 msgr2=0x7fda0c104650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:22.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.604+0000 7fda0ad9d700 1 --2- 172.21.15.70:0/994558584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda0c103600 0x7fda0c104650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:22.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.604+0000 7fda0ad9d700 1 -- 172.21.15.70:0/994558584 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda0c1a7f40 con 0x7fda0c102c40 2024-06-04T04:54:22.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.604+0000 7fda0a59c700 1 --2- 172.21.15.70:0/994558584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda0c103600 0x7fda0c104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:54:22.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.604+0000 7fda0ad9d700 1 --2- 172.21.15.70:0/994558584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda0c102c40 0x7fda0c104110 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fda0000d980 tx=0x7fda0000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:22.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.604+0000 7fd9f3fff700 1 -- 172.21.15.70:0/994558584 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00017030 con 0x7fda0c102c40 2024-06-04T04:54:22.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.604+0000 7fda11057700 1 -- 172.21.15.70:0/994558584 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda0c1a81d0 con 0x7fda0c102c40 2024-06-04T04:54:22.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.604+0000 7fd9f3fff700 1 -- 172.21.15.70:0/994558584 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fda000044b0 con 0x7fda0c102c40 2024-06-04T04:54:22.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.604+0000 7fd9f3fff700 1 -- 172.21.15.70:0/994558584 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00003970 con 0x7fda0c102c40 2024-06-04T04:54:22.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.604+0000 7fda11057700 1 -- 172.21.15.70:0/994558584 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda0c1a8710 con 0x7fda0c102c40 2024-06-04T04:54:22.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.605+0000 7fda11057700 1 -- 172.21.15.70:0/994558584 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda0c103c50 con 0x7fda0c102c40 2024-06-04T04:54:22.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.607+0000 7fd9f3fff700 1 -- 172.21.15.70:0/994558584 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fda0001a070 con 0x7fda0c102c40 2024-06-04T04:54:22.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.608+0000 7fd9f3fff700 1 --2- 172.21.15.70:0/994558584 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd9f40b2910 0x7fd9f40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:22.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.608+0000 7fda0a59c700 1 --2- 172.21.15.70:0/994558584 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd9f40b2910 0x7fd9f40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:22.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.608+0000 7fd9f3fff700 1 -- 172.21.15.70:0/994558584 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fda000e2100 con 0x7fda0c102c40 2024-06-04T04:54:22.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.608+0000 7fda0a59c700 1 --2- 172.21.15.70:0/994558584 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd9f40b2910 0x7fd9f40b4dd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc005fd0 tx=0x7fd9fc005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:22.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.611+0000 7fd9f3fff700 1 -- 172.21.15.70:0/994558584 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fda000a9ca0 con 0x7fda0c102c40 2024-06-04T04:54:22.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.811+0000 7fda11057700 1 -- 172.21.15.70:0/994558584 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fda0c106250 con 0x7fd9f40b2910 2024-06-04T04:54:22.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.818+0000 7fd9f3fff700 1 -- 172.21.15.70:0/994558584 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258928 (secure 0 0 0) 0x7fda0c106250 con 0x7fd9f40b2910 2024-06-04T04:54:22.823 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:54:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.821+0000 7fda11057700 1 -- 172.21.15.70:0/994558584 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd9f40b2910 msgr2=0x7fd9f40b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.821+0000 7fda11057700 1 --2- 172.21.15.70:0/994558584 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd9f40b2910 0x7fd9f40b4dd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc005fd0 tx=0x7fd9fc005ee0 comp rx=0 tx=0).stop 2024-06-04T04:54:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.821+0000 7fda11057700 1 -- 172.21.15.70:0/994558584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda0c102c40 msgr2=0x7fda0c104110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.821+0000 7fda11057700 1 --2- 172.21.15.70:0/994558584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda0c102c40 0x7fda0c104110 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fda0000d980 tx=0x7fda0000dd40 comp rx=0 tx=0).stop 2024-06-04T04:54:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.821+0000 7fda11057700 1 -- 172.21.15.70:0/994558584 shutdown_connections 2024-06-04T04:54:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.821+0000 7fda11057700 1 --2- 172.21.15.70:0/994558584 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd9f40b2910 0x7fd9f40b4dd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.821+0000 7fda11057700 1 --2- 172.21.15.70:0/994558584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda0c1084c0 0x7fda0c104b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.821+0000 7fda11057700 1 --2- 172.21.15.70:0/994558584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda0c102c40 0x7fda0c104110 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.821+0000 7fda11057700 1 --2- 172.21.15.70:0/994558584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda0c103600 0x7fda0c104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.821+0000 7fda11057700 1 -- 172.21.15.70:0/994558584 >> 172.21.15.70:0/994558584 conn(0x7fda0c078ab0 msgr2=0x7fda0c1007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.822+0000 7fda11057700 1 -- 172.21.15.70:0/994558584 shutdown_connections 2024-06-04T04:54:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:22.822+0000 7fda11057700 1 -- 172.21.15.70:0/994558584 wait complete. 2024-06-04T04:54:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-06-04T04:54:23.559 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-06-04T04:54:21.732009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651787,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":369,"num_read_kb":279,"num_write":363,"num_write_kb":2277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":298,"ondisk_log_size":298,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":295,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":355784,"kb_used_data":10748,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124586040,"statfs":{"total":1151940427776,"available":1151576104960,"internally_reserved":0,"allocated":11005952,"data_stored":7014445,"data_compressed":148315,"data_compressed_allocated":196608,"data_compressed_original":1892352,"omap_allocated":19138,"internal_metadata":353154366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9441,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":71,"num_read_kb":40,"num_write":7,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001959"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509657+0000","last_change":"2024-06-04T04:53:36.383755+0000","last_active":"2024-06-04T04:54:19.509657+0000","last_peered":"2024-06-04T04:54:19.509657+0000","last_clean":"2024-06-04T04:54:19.509657+0000","last_became_active":"2024-06-04T04:53:36.378274+0000","last_became_peered":"2024-06-04T04:53:36.378274+0000","last_unstale":"2024-06-04T04:54:19.509657+0000","last_undegraded":"2024-06-04T04:54:19.509657+0000","last_fullsized":"2024-06-04T04:54:19.509657+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:30:19.685994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510014+0000","last_change":"2024-06-04T04:53:36.372436+0000","last_active":"2024-06-04T04:54:19.510014+0000","last_peered":"2024-06-04T04:54:19.510014+0000","last_clean":"2024-06-04T04:54:19.510014+0000","last_became_active":"2024-06-04T04:53:36.372100+0000","last_became_peered":"2024-06-04T04:53:36.372100+0000","last_unstale":"2024-06-04T04:54:19.510014+0000","last_undegraded":"2024-06-04T04:54:19.510014+0000","last_fullsized":"2024-06-04T04:54:19.510014+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:42:31.646754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510448+0000","last_change":"2024-06-04T04:53:36.366915+0000","last_active":"2024-06-04T04:54:19.510448+0000","last_peered":"2024-06-04T04:54:19.510448+0000","last_clean":"2024-06-04T04:54:19.510448+0000","last_became_active":"2024-06-04T04:53:36.366611+0000","last_became_peered":"2024-06-04T04:53:36.366611+0000","last_unstale":"2024-06-04T04:54:19.510448+0000","last_undegraded":"2024-06-04T04:54:19.510448+0000","last_fullsized":"2024-06-04T04:54:19.510448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:17:48.775938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631647+0000","last_change":"2024-06-04T04:53:36.374486+0000","last_active":"2024-06-04T04:54:19.631647+0000","last_peered":"2024-06-04T04:54:19.631647+0000","last_clean":"2024-06-04T04:54:19.631647+0000","last_became_active":"2024-06-04T04:53:36.374178+0000","last_became_peered":"2024-06-04T04:53:36.374178+0000","last_unstale":"2024-06-04T04:54:19.631647+0000","last_undegraded":"2024-06-04T04:54:19.631647+0000","last_fullsized":"2024-06-04T04:54:19.631647+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:30:28.287070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.364249+0000","last_change":"2024-06-04T04:53:36.376183+0000","last_active":"2024-06-04T04:53:38.364249+0000","last_peered":"2024-06-04T04:53:38.364249+0000","last_clean":"2024-06-04T04:53:38.364249+0000","last_became_active":"2024-06-04T04:53:36.375790+0000","last_became_peered":"2024-06-04T04:53:36.375790+0000","last_unstale":"2024-06-04T04:53:38.364249+0000","last_undegraded":"2024-06-04T04:53:38.364249+0000","last_fullsized":"2024-06-04T04:53:38.364249+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:58.697440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.374957+0000","last_change":"2024-06-04T04:53:36.370249+0000","last_active":"2024-06-04T04:53:38.374957+0000","last_peered":"2024-06-04T04:53:38.374957+0000","last_clean":"2024-06-04T04:53:38.374957+0000","last_became_active":"2024-06-04T04:53:36.369916+0000","last_became_peered":"2024-06-04T04:53:36.369916+0000","last_unstale":"2024-06-04T04:53:38.374957+0000","last_undegraded":"2024-06-04T04:53:38.374957+0000","last_fullsized":"2024-06-04T04:53:38.374957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:02:41.739806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510237+0000","last_change":"2024-06-04T04:53:36.384329+0000","last_active":"2024-06-04T04:54:19.510237+0000","last_peered":"2024-06-04T04:54:19.510237+0000","last_clean":"2024-06-04T04:54:19.510237+0000","last_became_active":"2024-06-04T04:53:36.382504+0000","last_became_peered":"2024-06-04T04:53:36.382504+0000","last_unstale":"2024-06-04T04:54:19.510237+0000","last_undegraded":"2024-06-04T04:54:19.510237+0000","last_fullsized":"2024-06-04T04:54:19.510237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:51:07.604691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504427+0000","last_change":"2024-06-04T04:53:36.372319+0000","last_active":"2024-06-04T04:54:18.504427+0000","last_peered":"2024-06-04T04:54:18.504427+0000","last_clean":"2024-06-04T04:54:18.504427+0000","last_became_active":"2024-06-04T04:53:36.372000+0000","last_became_peered":"2024-06-04T04:53:36.372000+0000","last_unstale":"2024-06-04T04:54:18.504427+0000","last_undegraded":"2024-06-04T04:54:18.504427+0000","last_fullsized":"2024-06-04T04:54:18.504427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:15:56.906996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631893+0000","last_change":"2024-06-04T04:53:36.376051+0000","last_active":"2024-06-04T04:54:19.631893+0000","last_peered":"2024-06-04T04:54:19.631893+0000","last_clean":"2024-06-04T04:54:19.631893+0000","last_became_active":"2024-06-04T04:53:36.375726+0000","last_became_peered":"2024-06-04T04:53:36.375726+0000","last_unstale":"2024-06-04T04:54:19.631893+0000","last_undegraded":"2024-06-04T04:54:19.631893+0000","last_fullsized":"2024-06-04T04:54:19.631893+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:09:08.542764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.378366+0000","last_change":"2024-06-04T04:53:36.373266+0000","last_active":"2024-06-04T04:53:38.378366+0000","last_peered":"2024-06-04T04:53:38.378366+0000","last_clean":"2024-06-04T04:53:38.378366+0000","last_became_active":"2024-06-04T04:53:36.372898+0000","last_became_peered":"2024-06-04T04:53:36.372898+0000","last_unstale":"2024-06-04T04:53:38.378366+0000","last_undegraded":"2024-06-04T04:53:38.378366+0000","last_fullsized":"2024-06-04T04:53:38.378366+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:41:43.436781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510727+0000","last_change":"2024-06-04T04:53:36.381344+0000","last_active":"2024-06-04T04:54:19.510727+0000","last_peered":"2024-06-04T04:54:19.510727+0000","last_clean":"2024-06-04T04:54:19.510727+0000","last_became_active":"2024-06-04T04:53:36.380736+0000","last_became_peered":"2024-06-04T04:53:36.380736+0000","last_unstale":"2024-06-04T04:54:19.510727+0000","last_undegraded":"2024-06-04T04:54:19.510727+0000","last_fullsized":"2024-06-04T04:54:19.510727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:23:01.607272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631528+0000","last_change":"2024-06-04T04:53:36.374035+0000","last_active":"2024-06-04T04:54:19.631528+0000","last_peered":"2024-06-04T04:54:19.631528+0000","last_clean":"2024-06-04T04:54:19.631528+0000","last_became_active":"2024-06-04T04:53:36.373667+0000","last_became_peered":"2024-06-04T04:53:36.373667+0000","last_unstale":"2024-06-04T04:54:19.631528+0000","last_undegraded":"2024-06-04T04:54:19.631528+0000","last_fullsized":"2024-06-04T04:54:19.631528+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:23:07.880369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505537+0000","last_change":"2024-06-04T04:53:36.369613+0000","last_active":"2024-06-04T04:54:19.505537+0000","last_peered":"2024-06-04T04:54:19.505537+0000","last_clean":"2024-06-04T04:54:19.505537+0000","last_became_active":"2024-06-04T04:53:36.369277+0000","last_became_peered":"2024-06-04T04:53:36.369277+0000","last_unstale":"2024-06-04T04:54:19.505537+0000","last_undegraded":"2024-06-04T04:54:19.505537+0000","last_fullsized":"2024-06-04T04:54:19.505537+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:13:33.756609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510066+0000","last_change":"2024-06-04T04:53:36.368094+0000","last_active":"2024-06-04T04:54:19.510066+0000","last_peered":"2024-06-04T04:54:19.510066+0000","last_clean":"2024-06-04T04:54:19.510066+0000","last_became_active":"2024-06-04T04:53:36.367717+0000","last_became_peered":"2024-06-04T04:53:36.367717+0000","last_unstale":"2024-06-04T04:54:19.510066+0000","last_undegraded":"2024-06-04T04:54:19.510066+0000","last_fullsized":"2024-06-04T04:54:19.510066+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:39:56.305078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509882+0000","last_change":"2024-06-04T04:53:36.366765+0000","last_active":"2024-06-04T04:54:19.509882+0000","last_peered":"2024-06-04T04:54:19.509882+0000","last_clean":"2024-06-04T04:54:19.509882+0000","last_became_active":"2024-06-04T04:53:36.366438+0000","last_became_peered":"2024-06-04T04:53:36.366438+0000","last_unstale":"2024-06-04T04:54:19.509882+0000","last_undegraded":"2024-06-04T04:54:19.509882+0000","last_fullsized":"2024-06-04T04:54:19.509882+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:38:37.313926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631086+0000","last_change":"2024-06-04T04:53:36.380088+0000","last_active":"2024-06-04T04:54:19.631086+0000","last_peered":"2024-06-04T04:54:19.631086+0000","last_clean":"2024-06-04T04:54:19.631086+0000","last_became_active":"2024-06-04T04:53:36.379467+0000","last_became_peered":"2024-06-04T04:53:36.379467+0000","last_unstale":"2024-06-04T04:54:19.631086+0000","last_undegraded":"2024-06-04T04:54:19.631086+0000","last_fullsized":"2024-06-04T04:54:19.631086+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:46:53.236419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509925+0000","last_change":"2024-06-04T04:53:36.381126+0000","last_active":"2024-06-04T04:54:19.509925+0000","last_peered":"2024-06-04T04:54:19.509925+0000","last_clean":"2024-06-04T04:54:19.509925+0000","last_became_active":"2024-06-04T04:53:36.378833+0000","last_became_peered":"2024-06-04T04:53:36.378833+0000","last_unstale":"2024-06-04T04:54:19.509925+0000","last_undegraded":"2024-06-04T04:54:19.509925+0000","last_fullsized":"2024-06-04T04:54:19.509925+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:20:28.639273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.374064+0000","last_change":"2024-06-04T04:53:36.369692+0000","last_active":"2024-06-04T04:53:38.374064+0000","last_peered":"2024-06-04T04:53:38.374064+0000","last_clean":"2024-06-04T04:53:38.374064+0000","last_became_active":"2024-06-04T04:53:36.369370+0000","last_became_peered":"2024-06-04T04:53:36.369370+0000","last_unstale":"2024-06-04T04:53:38.374064+0000","last_undegraded":"2024-06-04T04:53:38.374064+0000","last_fullsized":"2024-06-04T04:53:38.374064+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:55.275465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.509473+0000","last_change":"2024-06-04T04:53:36.383639+0000","last_active":"2024-06-04T04:54:18.509473+0000","last_peered":"2024-06-04T04:54:18.509473+0000","last_clean":"2024-06-04T04:54:18.509473+0000","last_became_active":"2024-06-04T04:53:36.383303+0000","last_became_peered":"2024-06-04T04:53:36.383303+0000","last_unstale":"2024-06-04T04:54:18.509473+0000","last_undegraded":"2024-06-04T04:54:18.509473+0000","last_fullsized":"2024-06-04T04:54:18.509473+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:26:31.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":60,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:54:16.484354+0000","last_change":"2024-06-04T04:53:33.344120+0000","last_active":"2024-06-04T04:54:16.484354+0000","last_peered":"2024-06-04T04:54:16.484354+0000","last_clean":"2024-06-04T04:54:16.484354+0000","last_became_active":"2024-06-04T04:53:33.343743+0000","last_became_peered":"2024-06-04T04:53:33.343743+0000","last_unstale":"2024-06-04T04:54:16.484354+0000","last_undegraded":"2024-06-04T04:54:16.484354+0000","last_fullsized":"2024-06-04T04:54:16.484354+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:57:46.585195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504978+0000","last_change":"2024-06-04T04:53:36.371827+0000","last_active":"2024-06-04T04:54:18.504978+0000","last_peered":"2024-06-04T04:54:18.504978+0000","last_clean":"2024-06-04T04:54:18.504978+0000","last_became_active":"2024-06-04T04:53:36.371509+0000","last_became_peered":"2024-06-04T04:53:36.371509+0000","last_unstale":"2024-06-04T04:54:18.504978+0000","last_undegraded":"2024-06-04T04:54:18.504978+0000","last_fullsized":"2024-06-04T04:54:18.504978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:54:51.589008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.505150+0000","last_change":"2024-06-04T04:53:33.353976+0000","last_active":"2024-06-04T04:54:18.505150+0000","last_peered":"2024-06-04T04:54:18.505150+0000","last_clean":"2024-06-04T04:54:18.505150+0000","last_became_active":"2024-06-04T04:53:33.353586+0000","last_became_peered":"2024-06-04T04:53:33.353586+0000","last_unstale":"2024-06-04T04:54:18.505150+0000","last_undegraded":"2024-06-04T04:54:18.505150+0000","last_fullsized":"2024-06-04T04:54:18.505150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:17:10.378008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510428+0000","last_change":"2024-06-04T04:53:36.367806+0000","last_active":"2024-06-04T04:54:19.510428+0000","last_peered":"2024-06-04T04:54:19.510428+0000","last_clean":"2024-06-04T04:54:19.510428+0000","last_became_active":"2024-06-04T04:53:36.367484+0000","last_became_peered":"2024-06-04T04:53:36.367484+0000","last_unstale":"2024-06-04T04:54:19.510428+0000","last_undegraded":"2024-06-04T04:54:19.510428+0000","last_fullsized":"2024-06-04T04:54:19.510428+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:25:36.194746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":35,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:54.451561+0000","last_change":"2024-06-04T04:53:33.348902+0000","last_active":"2024-06-04T04:53:54.451561+0000","last_peered":"2024-06-04T04:53:54.451561+0000","last_clean":"2024-06-04T04:53:54.451561+0000","last_became_active":"2024-06-04T04:53:33.348441+0000","last_became_peered":"2024-06-04T04:53:33.348441+0000","last_unstale":"2024-06-04T04:53:54.451561+0000","last_undegraded":"2024-06-04T04:53:54.451561+0000","last_fullsized":"2024-06-04T04:53:54.451561+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:22:28.428303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631324+0000","last_change":"2024-06-04T04:53:36.376867+0000","last_active":"2024-06-04T04:54:19.631324+0000","last_peered":"2024-06-04T04:54:19.631324+0000","last_clean":"2024-06-04T04:54:19.631324+0000","last_became_active":"2024-06-04T04:53:36.376517+0000","last_became_peered":"2024-06-04T04:53:36.376517+0000","last_unstale":"2024-06-04T04:54:19.631324+0000","last_undegraded":"2024-06-04T04:54:19.631324+0000","last_fullsized":"2024-06-04T04:54:19.631324+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:21:59.494321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":28,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:53.447130+0000","last_change":"2024-06-04T04:53:33.347993+0000","last_active":"2024-06-04T04:53:53.447130+0000","last_peered":"2024-06-04T04:53:53.447130+0000","last_clean":"2024-06-04T04:53:53.447130+0000","last_became_active":"2024-06-04T04:53:33.347656+0000","last_became_peered":"2024-06-04T04:53:33.347656+0000","last_unstale":"2024-06-04T04:53:53.447130+0000","last_undegraded":"2024-06-04T04:53:53.447130+0000","last_fullsized":"2024-06-04T04:53:53.447130+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:21:26.806557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.632790+0000","last_change":"2024-06-04T04:54:19.632790+0000","last_active":"2024-06-04T04:54:19.632790+0000","last_peered":"2024-06-04T04:54:19.632790+0000","last_clean":"2024-06-04T04:54:19.632790+0000","last_became_active":"2024-06-04T04:54:19.632338+0000","last_became_peered":"2024-06-04T04:54:19.632338+0000","last_unstale":"2024-06-04T04:54:19.632790+0000","last_undegraded":"2024-06-04T04:54:19.632790+0000","last_fullsized":"2024-06-04T04:54:19.632790+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:44:00.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'5","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512034+0000","last_change":"2024-06-04T04:53:33.346111+0000","last_active":"2024-06-04T04:54:19.512034+0000","last_peered":"2024-06-04T04:54:19.512034+0000","last_clean":"2024-06-04T04:54:19.512034+0000","last_became_active":"2024-06-04T04:53:33.345773+0000","last_became_peered":"2024-06-04T04:53:33.345773+0000","last_unstale":"2024-06-04T04:54:19.512034+0000","last_undegraded":"2024-06-04T04:54:19.512034+0000","last_fullsized":"2024-06-04T04:54:19.512034+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:18.565583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.364617+0000","last_change":"2024-06-04T04:53:36.373369+0000","last_active":"2024-06-04T04:53:38.364617+0000","last_peered":"2024-06-04T04:53:38.364617+0000","last_clean":"2024-06-04T04:53:38.364617+0000","last_became_active":"2024-06-04T04:53:36.373001+0000","last_became_peered":"2024-06-04T04:53:36.373001+0000","last_unstale":"2024-06-04T04:53:38.364617+0000","last_undegraded":"2024-06-04T04:53:38.364617+0000","last_fullsized":"2024-06-04T04:53:38.364617+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:02:22.012643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":28,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:52.442028+0000","last_change":"2024-06-04T04:53:33.344358+0000","last_active":"2024-06-04T04:53:52.442028+0000","last_peered":"2024-06-04T04:53:52.442028+0000","last_clean":"2024-06-04T04:53:52.442028+0000","last_became_active":"2024-06-04T04:53:33.343980+0000","last_became_peered":"2024-06-04T04:53:33.343980+0000","last_unstale":"2024-06-04T04:53:52.442028+0000","last_undegraded":"2024-06-04T04:53:52.442028+0000","last_fullsized":"2024-06-04T04:53:52.442028+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:14.493138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512293+0000","last_change":"2024-06-04T04:53:36.371239+0000","last_active":"2024-06-04T04:54:19.512293+0000","last_peered":"2024-06-04T04:54:19.512293+0000","last_clean":"2024-06-04T04:54:19.512293+0000","last_became_active":"2024-06-04T04:53:36.370885+0000","last_became_peered":"2024-06-04T04:53:36.370885+0000","last_unstale":"2024-06-04T04:54:19.512293+0000","last_undegraded":"2024-06-04T04:54:19.512293+0000","last_fullsized":"2024-06-04T04:54:19.512293+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T04:54:29.753796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505354+0000","last_change":"2024-06-04T04:53:33.338512+0000","last_active":"2024-06-04T04:54:19.505354+0000","last_peered":"2024-06-04T04:54:19.505354+0000","last_clean":"2024-06-04T04:54:19.505354+0000","last_became_active":"2024-06-04T04:53:33.338180+0000","last_became_peered":"2024-06-04T04:53:33.338180+0000","last_unstale":"2024-06-04T04:54:19.505354+0000","last_undegraded":"2024-06-04T04:54:19.505354+0000","last_fullsized":"2024-06-04T04:54:19.505354+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:57:13.406479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504521+0000","last_change":"2024-06-04T04:53:36.377316+0000","last_active":"2024-06-04T04:54:18.504521+0000","last_peered":"2024-06-04T04:54:18.504521+0000","last_clean":"2024-06-04T04:54:18.504521+0000","last_became_active":"2024-06-04T04:53:36.376992+0000","last_became_peered":"2024-06-04T04:53:36.376992+0000","last_unstale":"2024-06-04T04:54:18.504521+0000","last_undegraded":"2024-06-04T04:54:18.504521+0000","last_fullsized":"2024-06-04T04:54:18.504521+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:33.145631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511696+0000","last_change":"2024-06-04T04:53:33.348305+0000","last_active":"2024-06-04T04:54:19.511696+0000","last_peered":"2024-06-04T04:54:19.511696+0000","last_clean":"2024-06-04T04:54:19.511696+0000","last_became_active":"2024-06-04T04:53:33.347948+0000","last_became_peered":"2024-06-04T04:53:33.347948+0000","last_unstale":"2024-06-04T04:54:19.511696+0000","last_undegraded":"2024-06-04T04:54:19.511696+0000","last_fullsized":"2024-06-04T04:54:19.511696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:04:13.669274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.374246+0000","last_change":"2024-06-04T04:53:36.367955+0000","last_active":"2024-06-04T04:53:38.374246+0000","last_peered":"2024-06-04T04:53:38.374246+0000","last_clean":"2024-06-04T04:53:38.374246+0000","last_became_active":"2024-06-04T04:53:36.367628+0000","last_became_peered":"2024-06-04T04:53:36.367628+0000","last_unstale":"2024-06-04T04:53:38.374246+0000","last_undegraded":"2024-06-04T04:53:38.374246+0000","last_fullsized":"2024-06-04T04:53:38.374246+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:33:26.185498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'4","reported_seq":73,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:54:17.486475+0000","last_change":"2024-06-04T04:53:33.349471+0000","last_active":"2024-06-04T04:54:17.486475+0000","last_peered":"2024-06-04T04:54:17.486475+0000","last_clean":"2024-06-04T04:54:17.486475+0000","last_became_active":"2024-06-04T04:53:33.349105+0000","last_became_peered":"2024-06-04T04:53:33.349105+0000","last_unstale":"2024-06-04T04:54:17.486475+0000","last_undegraded":"2024-06-04T04:54:17.486475+0000","last_fullsized":"2024-06-04T04:54:17.486475+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:31:35.584275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":48,"num_read_kb":48,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631859+0000","last_change":"2024-06-04T04:53:36.367249+0000","last_active":"2024-06-04T04:54:19.631859+0000","last_peered":"2024-06-04T04:54:19.631859+0000","last_clean":"2024-06-04T04:54:19.631859+0000","last_became_active":"2024-06-04T04:53:36.366863+0000","last_became_peered":"2024-06-04T04:53:36.366863+0000","last_unstale":"2024-06-04T04:54:19.631859+0000","last_undegraded":"2024-06-04T04:54:19.631859+0000","last_fullsized":"2024-06-04T04:54:19.631859+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:13:17.308321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510836+0000","last_change":"2024-06-04T04:53:33.346778+0000","last_active":"2024-06-04T04:54:19.510836+0000","last_peered":"2024-06-04T04:54:19.510836+0000","last_clean":"2024-06-04T04:54:19.510836+0000","last_became_active":"2024-06-04T04:53:33.346376+0000","last_became_peered":"2024-06-04T04:53:33.346376+0000","last_unstale":"2024-06-04T04:54:19.510836+0000","last_undegraded":"2024-06-04T04:54:19.510836+0000","last_fullsized":"2024-06-04T04:54:19.510836+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:03:16.973822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.514491+0000","last_change":"2024-06-04T04:54:19.514491+0000","last_active":"2024-06-04T04:54:19.514491+0000","last_peered":"2024-06-04T04:54:19.514491+0000","last_clean":"2024-06-04T04:54:19.514491+0000","last_became_active":"2024-06-04T04:54:19.514148+0000","last_became_peered":"2024-06-04T04:54:19.514148+0000","last_unstale":"2024-06-04T04:54:19.514491+0000","last_undegraded":"2024-06-04T04:54:19.514491+0000","last_fullsized":"2024-06-04T04:54:19.514491+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:09:59.641041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512326+0000","last_change":"2024-06-04T04:53:33.354989+0000","last_active":"2024-06-04T04:54:19.512326+0000","last_peered":"2024-06-04T04:54:19.512326+0000","last_clean":"2024-06-04T04:54:19.512326+0000","last_became_active":"2024-06-04T04:53:33.354613+0000","last_became_peered":"2024-06-04T04:53:33.354613+0000","last_unstale":"2024-06-04T04:54:19.512326+0000","last_undegraded":"2024-06-04T04:54:19.512326+0000","last_fullsized":"2024-06-04T04:54:19.512326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:19:59.398763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631419+0000","last_change":"2024-06-04T04:53:36.372216+0000","last_active":"2024-06-04T04:54:19.631419+0000","last_peered":"2024-06-04T04:54:19.631419+0000","last_clean":"2024-06-04T04:54:19.631419+0000","last_became_active":"2024-06-04T04:53:36.371820+0000","last_became_peered":"2024-06-04T04:53:36.371820+0000","last_unstale":"2024-06-04T04:54:19.631419+0000","last_undegraded":"2024-06-04T04:54:19.631419+0000","last_fullsized":"2024-06-04T04:54:19.631419+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:55:59.887963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"79'27","reported_seq":103,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:54:17.490520+0000","last_change":"2024-06-04T04:53:33.345493+0000","last_active":"2024-06-04T04:54:17.490520+0000","last_peered":"2024-06-04T04:54:17.490520+0000","last_clean":"2024-06-04T04:54:17.490520+0000","last_became_active":"2024-06-04T04:53:33.345102+0000","last_became_peered":"2024-06-04T04:53:33.345102+0000","last_unstale":"2024-06-04T04:54:17.490520+0000","last_undegraded":"2024-06-04T04:54:17.490520+0000","last_fullsized":"2024-06-04T04:54:17.490520+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:50:27.853391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54372,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":55,"num_read_kb":80,"num_write":29,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.364618+0000","last_change":"2024-06-04T04:53:36.382070+0000","last_active":"2024-06-04T04:53:38.364618+0000","last_peered":"2024-06-04T04:53:38.364618+0000","last_clean":"2024-06-04T04:53:38.364618+0000","last_became_active":"2024-06-04T04:53:36.379144+0000","last_became_peered":"2024-06-04T04:53:36.379144+0000","last_unstale":"2024-06-04T04:53:38.364618+0000","last_undegraded":"2024-06-04T04:53:38.364618+0000","last_fullsized":"2024-06-04T04:53:38.364618+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:16.702709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.508806+0000","last_change":"2024-06-04T04:53:33.344726+0000","last_active":"2024-06-04T04:54:18.508806+0000","last_peered":"2024-06-04T04:54:18.508806+0000","last_clean":"2024-06-04T04:54:18.508806+0000","last_became_active":"2024-06-04T04:53:33.344360+0000","last_became_peered":"2024-06-04T04:53:33.344360+0000","last_unstale":"2024-06-04T04:54:18.508806+0000","last_undegraded":"2024-06-04T04:54:18.508806+0000","last_fullsized":"2024-06-04T04:54:18.508806+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:44:56.784808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505351+0000","last_change":"2024-06-04T04:53:36.379036+0000","last_active":"2024-06-04T04:54:19.505351+0000","last_peered":"2024-06-04T04:54:19.505351+0000","last_clean":"2024-06-04T04:54:19.505351+0000","last_became_active":"2024-06-04T04:53:36.378681+0000","last_became_peered":"2024-06-04T04:53:36.378681+0000","last_unstale":"2024-06-04T04:54:19.505351+0000","last_undegraded":"2024-06-04T04:54:19.505351+0000","last_fullsized":"2024-06-04T04:54:19.505351+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:07:49.704731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511967+0000","last_change":"2024-06-04T04:53:33.345915+0000","last_active":"2024-06-04T04:54:19.511967+0000","last_peered":"2024-06-04T04:54:19.511967+0000","last_clean":"2024-06-04T04:54:19.511967+0000","last_became_active":"2024-06-04T04:53:33.345537+0000","last_became_peered":"2024-06-04T04:53:33.345537+0000","last_unstale":"2024-06-04T04:54:19.511967+0000","last_undegraded":"2024-06-04T04:54:19.511967+0000","last_fullsized":"2024-06-04T04:54:19.511967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:06:44.086968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510431+0000","last_change":"2024-06-04T04:53:36.379094+0000","last_active":"2024-06-04T04:54:19.510431+0000","last_peered":"2024-06-04T04:54:19.510431+0000","last_clean":"2024-06-04T04:54:19.510431+0000","last_became_active":"2024-06-04T04:53:36.378763+0000","last_became_peered":"2024-06-04T04:53:36.378763+0000","last_unstale":"2024-06-04T04:54:19.510431+0000","last_undegraded":"2024-06-04T04:54:19.510431+0000","last_fullsized":"2024-06-04T04:54:19.510431+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:16.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.630622+0000","last_change":"2024-06-04T04:53:33.341342+0000","last_active":"2024-06-04T04:54:19.630622+0000","last_peered":"2024-06-04T04:54:19.630622+0000","last_clean":"2024-06-04T04:54:19.630622+0000","last_became_active":"2024-06-04T04:53:33.340862+0000","last_became_peered":"2024-06-04T04:53:33.340862+0000","last_unstale":"2024-06-04T04:54:19.630622+0000","last_undegraded":"2024-06-04T04:54:19.630622+0000","last_fullsized":"2024-06-04T04:54:19.630622+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:26.818063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510293+0000","last_change":"2024-06-04T04:53:36.383124+0000","last_active":"2024-06-04T04:54:19.510293+0000","last_peered":"2024-06-04T04:54:19.510293+0000","last_clean":"2024-06-04T04:54:19.510293+0000","last_became_active":"2024-06-04T04:53:36.382449+0000","last_became_peered":"2024-06-04T04:53:36.382449+0000","last_unstale":"2024-06-04T04:54:19.510293+0000","last_undegraded":"2024-06-04T04:54:19.510293+0000","last_fullsized":"2024-06-04T04:54:19.510293+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:52:40.776003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:53.444957+0000","last_change":"2024-06-04T04:53:33.354556+0000","last_active":"2024-06-04T04:53:53.444957+0000","last_peered":"2024-06-04T04:53:53.444957+0000","last_clean":"2024-06-04T04:53:53.444957+0000","last_became_active":"2024-06-04T04:53:33.354238+0000","last_became_peered":"2024-06-04T04:53:33.354238+0000","last_unstale":"2024-06-04T04:53:53.444957+0000","last_undegraded":"2024-06-04T04:53:53.444957+0000","last_fullsized":"2024-06-04T04:53:53.444957+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:54.697374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511906+0000","last_change":"2024-06-04T04:53:36.371387+0000","last_active":"2024-06-04T04:54:19.511906+0000","last_peered":"2024-06-04T04:54:19.511906+0000","last_clean":"2024-06-04T04:54:19.511906+0000","last_became_active":"2024-06-04T04:53:36.370852+0000","last_became_peered":"2024-06-04T04:53:36.370852+0000","last_unstale":"2024-06-04T04:54:19.511906+0000","last_undegraded":"2024-06-04T04:54:19.511906+0000","last_fullsized":"2024-06-04T04:54:19.511906+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:44:06.956210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":25,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:53.426358+0000","last_change":"2024-06-04T04:53:33.349982+0000","last_active":"2024-06-04T04:53:53.426358+0000","last_peered":"2024-06-04T04:53:53.426358+0000","last_clean":"2024-06-04T04:53:53.426358+0000","last_became_active":"2024-06-04T04:53:33.349633+0000","last_became_peered":"2024-06-04T04:53:33.349633+0000","last_unstale":"2024-06-04T04:53:53.426358+0000","last_undegraded":"2024-06-04T04:53:53.426358+0000","last_fullsized":"2024-06-04T04:53:53.426358+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:40.107426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509399+0000","last_change":"2024-06-04T04:53:36.382313+0000","last_active":"2024-06-04T04:54:19.509399+0000","last_peered":"2024-06-04T04:54:19.509399+0000","last_clean":"2024-06-04T04:54:19.509399+0000","last_became_active":"2024-06-04T04:53:36.379638+0000","last_became_peered":"2024-06-04T04:53:36.379638+0000","last_unstale":"2024-06-04T04:54:19.509399+0000","last_undegraded":"2024-06-04T04:54:19.509399+0000","last_fullsized":"2024-06-04T04:54:19.509399+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:43:42.527262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.374221+0000","last_change":"2024-06-04T04:53:33.354766+0000","last_active":"2024-06-04T04:53:38.374221+0000","last_peered":"2024-06-04T04:53:38.374221+0000","last_clean":"2024-06-04T04:53:38.374221+0000","last_became_active":"2024-06-04T04:53:33.354424+0000","last_became_peered":"2024-06-04T04:53:33.354424+0000","last_unstale":"2024-06-04T04:53:38.374221+0000","last_undegraded":"2024-06-04T04:53:38.374221+0000","last_fullsized":"2024-06-04T04:53:38.374221+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:01:06.706010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.378389+0000","last_change":"2024-06-04T04:53:36.369904+0000","last_active":"2024-06-04T04:53:38.378389+0000","last_peered":"2024-06-04T04:53:38.378389+0000","last_clean":"2024-06-04T04:53:38.378389+0000","last_became_active":"2024-06-04T04:53:36.367929+0000","last_became_peered":"2024-06-04T04:53:36.367929+0000","last_unstale":"2024-06-04T04:53:38.378389+0000","last_undegraded":"2024-06-04T04:53:38.378389+0000","last_fullsized":"2024-06-04T04:53:38.378389+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:37:08.721569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'6","reported_seq":60,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510585+0000","last_change":"2024-06-04T04:53:33.342723+0000","last_active":"2024-06-04T04:54:19.510585+0000","last_peered":"2024-06-04T04:54:19.510585+0000","last_clean":"2024-06-04T04:54:19.510585+0000","last_became_active":"2024-06-04T04:53:33.342344+0000","last_became_peered":"2024-06-04T04:53:33.342344+0000","last_unstale":"2024-06-04T04:54:19.510585+0000","last_undegraded":"2024-06-04T04:54:19.510585+0000","last_fullsized":"2024-06-04T04:54:19.510585+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:59:50.147517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.376913+0000","last_change":"2024-06-04T04:53:36.372516+0000","last_active":"2024-06-04T04:53:38.376913+0000","last_peered":"2024-06-04T04:53:38.376913+0000","last_clean":"2024-06-04T04:53:38.376913+0000","last_became_active":"2024-06-04T04:53:36.370698+0000","last_became_peered":"2024-06-04T04:53:36.370698+0000","last_unstale":"2024-06-04T04:53:38.376913+0000","last_undegraded":"2024-06-04T04:53:38.376913+0000","last_fullsized":"2024-06-04T04:53:38.376913+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:29:10.449002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512091+0000","last_change":"2024-06-04T04:53:36.371236+0000","last_active":"2024-06-04T04:54:19.512091+0000","last_peered":"2024-06-04T04:54:19.512091+0000","last_clean":"2024-06-04T04:54:19.512091+0000","last_became_active":"2024-06-04T04:53:36.370876+0000","last_became_peered":"2024-06-04T04:53:36.370876+0000","last_unstale":"2024-06-04T04:54:19.512091+0000","last_undegraded":"2024-06-04T04:54:19.512091+0000","last_fullsized":"2024-06-04T04:54:19.512091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:15:56.751317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":58,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505309+0000","last_change":"2024-06-04T04:53:33.355642+0000","last_active":"2024-06-04T04:54:19.505309+0000","last_peered":"2024-06-04T04:54:19.505309+0000","last_clean":"2024-06-04T04:54:19.505309+0000","last_became_active":"2024-06-04T04:53:33.355187+0000","last_became_peered":"2024-06-04T04:53:33.355187+0000","last_unstale":"2024-06-04T04:54:19.505309+0000","last_undegraded":"2024-06-04T04:54:19.505309+0000","last_fullsized":"2024-06-04T04:54:19.505309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:32:28.331518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.362788+0000","last_change":"2024-06-04T04:53:36.370656+0000","last_active":"2024-06-04T04:53:38.362788+0000","last_peered":"2024-06-04T04:53:38.362788+0000","last_clean":"2024-06-04T04:53:38.362788+0000","last_became_active":"2024-06-04T04:53:36.370350+0000","last_became_peered":"2024-06-04T04:53:36.370350+0000","last_unstale":"2024-06-04T04:53:38.362788+0000","last_undegraded":"2024-06-04T04:53:38.362788+0000","last_fullsized":"2024-06-04T04:53:38.362788+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:03:16.609933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631469+0000","last_change":"2024-06-04T04:53:33.340653+0000","last_active":"2024-06-04T04:54:19.631469+0000","last_peered":"2024-06-04T04:54:19.631469+0000","last_clean":"2024-06-04T04:54:19.631469+0000","last_became_active":"2024-06-04T04:53:33.340250+0000","last_became_peered":"2024-06-04T04:53:33.340250+0000","last_unstale":"2024-06-04T04:54:19.631469+0000","last_undegraded":"2024-06-04T04:54:19.631469+0000","last_fullsized":"2024-06-04T04:54:19.631469+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:46:20.329514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":99,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510637+0000","last_change":"2024-06-04T04:52:09.181155+0000","last_active":"2024-06-04T04:54:19.510637+0000","last_peered":"2024-06-04T04:54:19.510637+0000","last_clean":"2024-06-04T04:54:19.510637+0000","last_became_active":"2024-06-04T04:52:09.177002+0000","last_became_peered":"2024-06-04T04:52:09.177002+0000","last_unstale":"2024-06-04T04:54:19.510637+0000","last_undegraded":"2024-06-04T04:54:19.510637+0000","last_fullsized":"2024-06-04T04:54:19.510637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_clean_scrub_stamp":"2024-06-04T04:48:44.631948+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:46:57.735302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505220+0000","last_change":"2024-06-04T04:53:36.372634+0000","last_active":"2024-06-04T04:54:19.505220+0000","last_peered":"2024-06-04T04:54:19.505220+0000","last_clean":"2024-06-04T04:54:19.505220+0000","last_became_active":"2024-06-04T04:53:36.372290+0000","last_became_peered":"2024-06-04T04:53:36.372290+0000","last_unstale":"2024-06-04T04:54:19.505220+0000","last_undegraded":"2024-06-04T04:54:19.505220+0000","last_fullsized":"2024-06-04T04:54:19.505220+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:14:07.460382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511010+0000","last_change":"2024-06-04T04:53:33.348760+0000","last_active":"2024-06-04T04:54:19.511010+0000","last_peered":"2024-06-04T04:54:19.511010+0000","last_clean":"2024-06-04T04:54:19.511010+0000","last_became_active":"2024-06-04T04:53:33.348347+0000","last_became_peered":"2024-06-04T04:53:33.348347+0000","last_unstale":"2024-06-04T04:54:19.511010+0000","last_undegraded":"2024-06-04T04:54:19.511010+0000","last_fullsized":"2024-06-04T04:54:19.511010+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:36:20.641061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:53.447658+0000","last_change":"2024-06-04T04:53:33.354840+0000","last_active":"2024-06-04T04:53:53.447658+0000","last_peered":"2024-06-04T04:53:53.447658+0000","last_clean":"2024-06-04T04:53:53.447658+0000","last_became_active":"2024-06-04T04:53:33.354287+0000","last_became_peered":"2024-06-04T04:53:33.354287+0000","last_unstale":"2024-06-04T04:53:53.447658+0000","last_undegraded":"2024-06-04T04:53:53.447658+0000","last_fullsized":"2024-06-04T04:53:53.447658+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:42.550098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505559+0000","last_change":"2024-06-04T04:53:36.372678+0000","last_active":"2024-06-04T04:54:19.505559+0000","last_peered":"2024-06-04T04:54:19.505559+0000","last_clean":"2024-06-04T04:54:19.505559+0000","last_became_active":"2024-06-04T04:53:36.372312+0000","last_became_peered":"2024-06-04T04:53:36.372312+0000","last_unstale":"2024-06-04T04:54:19.505559+0000","last_undegraded":"2024-06-04T04:54:19.505559+0000","last_fullsized":"2024-06-04T04:54:19.505559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:34:41.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"79'14","reported_seq":36,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:54:15.377798+0000","last_change":"2024-06-04T04:53:33.348440+0000","last_active":"2024-06-04T04:54:15.377798+0000","last_peered":"2024-06-04T04:54:15.377798+0000","last_clean":"2024-06-04T04:54:15.377798+0000","last_became_active":"2024-06-04T04:53:33.348131+0000","last_became_peered":"2024-06-04T04:53:33.348131+0000","last_unstale":"2024-06-04T04:54:15.377798+0000","last_undegraded":"2024-06-04T04:54:15.377798+0000","last_fullsized":"2024-06-04T04:54:15.377798+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:14:11.552286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.505140+0000","last_change":"2024-06-04T04:53:36.378669+0000","last_active":"2024-06-04T04:54:18.505140+0000","last_peered":"2024-06-04T04:54:18.505140+0000","last_clean":"2024-06-04T04:54:18.505140+0000","last_became_active":"2024-06-04T04:53:36.378233+0000","last_became_peered":"2024-06-04T04:53:36.378233+0000","last_unstale":"2024-06-04T04:54:18.505140+0000","last_undegraded":"2024-06-04T04:54:18.505140+0000","last_fullsized":"2024-06-04T04:54:18.505140+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:36:45.870924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631647+0000","last_change":"2024-06-04T04:53:33.345078+0000","last_active":"2024-06-04T04:54:19.631647+0000","last_peered":"2024-06-04T04:54:19.631647+0000","last_clean":"2024-06-04T04:54:19.631647+0000","last_became_active":"2024-06-04T04:53:33.344708+0000","last_became_peered":"2024-06-04T04:53:33.344708+0000","last_unstale":"2024-06-04T04:54:19.631647+0000","last_undegraded":"2024-06-04T04:54:19.631647+0000","last_fullsized":"2024-06-04T04:54:19.631647+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:45:14.010758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510104+0000","last_change":"2024-06-04T04:53:36.364817+0000","last_active":"2024-06-04T04:54:19.510104+0000","last_peered":"2024-06-04T04:54:19.510104+0000","last_clean":"2024-06-04T04:54:19.510104+0000","last_became_active":"2024-06-04T04:53:36.364499+0000","last_became_peered":"2024-06-04T04:53:36.364499+0000","last_unstale":"2024-06-04T04:54:19.510104+0000","last_undegraded":"2024-06-04T04:54:19.510104+0000","last_fullsized":"2024-06-04T04:54:19.510104+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:39:30.518028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.630871+0000","last_change":"2024-06-04T04:53:33.354527+0000","last_active":"2024-06-04T04:54:19.630871+0000","last_peered":"2024-06-04T04:54:19.630871+0000","last_clean":"2024-06-04T04:54:19.630871+0000","last_became_active":"2024-06-04T04:53:33.354175+0000","last_became_peered":"2024-06-04T04:53:33.354175+0000","last_unstale":"2024-06-04T04:54:19.630871+0000","last_undegraded":"2024-06-04T04:54:19.630871+0000","last_fullsized":"2024-06-04T04:54:19.630871+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:33:08.125428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:38.364210+0000","last_change":"2024-06-04T04:53:36.373248+0000","last_active":"2024-06-04T04:53:38.364210+0000","last_peered":"2024-06-04T04:53:38.364210+0000","last_clean":"2024-06-04T04:53:38.364210+0000","last_became_active":"2024-06-04T04:53:36.372915+0000","last_became_peered":"2024-06-04T04:53:36.372915+0000","last_unstale":"2024-06-04T04:53:38.364210+0000","last_undegraded":"2024-06-04T04:53:38.364210+0000","last_fullsized":"2024-06-04T04:53:38.364210+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:23:11.689349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504542+0000","last_change":"2024-06-04T04:53:33.340427+0000","last_active":"2024-06-04T04:54:18.504542+0000","last_peered":"2024-06-04T04:54:18.504542+0000","last_clean":"2024-06-04T04:54:18.504542+0000","last_became_active":"2024-06-04T04:53:33.340022+0000","last_became_peered":"2024-06-04T04:53:33.340022+0000","last_unstale":"2024-06-04T04:54:18.504542+0000","last_undegraded":"2024-06-04T04:54:18.504542+0000","last_fullsized":"2024-06-04T04:54:18.504542+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:56:16.496471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509628+0000","last_change":"2024-06-04T04:53:36.380114+0000","last_active":"2024-06-04T04:54:19.509628+0000","last_peered":"2024-06-04T04:54:19.509628+0000","last_clean":"2024-06-04T04:54:19.509628+0000","last_became_active":"2024-06-04T04:53:36.379392+0000","last_became_peered":"2024-06-04T04:53:36.379392+0000","last_unstale":"2024-06-04T04:54:19.509628+0000","last_undegraded":"2024-06-04T04:54:19.509628+0000","last_fullsized":"2024-06-04T04:54:19.509628+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:41:48.776439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.520195+0000","last_change":"2024-06-04T04:54:19.519755+0000","last_active":"2024-06-04T04:54:19.520195+0000","last_peered":"2024-06-04T04:54:19.520195+0000","last_clean":"2024-06-04T04:54:19.520195+0000","last_became_active":"2024-06-04T04:54:19.513992+0000","last_became_peered":"2024-06-04T04:54:19.513992+0000","last_unstale":"2024-06-04T04:54:19.520195+0000","last_undegraded":"2024-06-04T04:54:19.520195+0000","last_fullsized":"2024-06-04T04:54:19.520195+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:01.951935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504890+0000","last_change":"2024-06-04T04:53:36.377420+0000","last_active":"2024-06-04T04:54:18.504890+0000","last_peered":"2024-06-04T04:54:18.504890+0000","last_clean":"2024-06-04T04:54:18.504890+0000","last_became_active":"2024-06-04T04:53:36.377109+0000","last_became_peered":"2024-06-04T04:53:36.377109+0000","last_unstale":"2024-06-04T04:54:18.504890+0000","last_undegraded":"2024-06-04T04:54:18.504890+0000","last_fullsized":"2024-06-04T04:54:18.504890+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:47:12.102396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-06-04T04:53:50.421065+0000","last_change":"2024-06-04T04:53:33.349285+0000","last_active":"2024-06-04T04:53:50.421065+0000","last_peered":"2024-06-04T04:53:50.421065+0000","last_clean":"2024-06-04T04:53:50.421065+0000","last_became_active":"2024-06-04T04:53:33.348929+0000","last_became_peered":"2024-06-04T04:53:33.348929+0000","last_unstale":"2024-06-04T04:53:50.421065+0000","last_undegraded":"2024-06-04T04:53:50.421065+0000","last_fullsized":"2024-06-04T04:53:50.421065+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:15:32.933106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510562+0000","last_change":"2024-06-04T04:53:36.381187+0000","last_active":"2024-06-04T04:54:19.510562+0000","last_peered":"2024-06-04T04:54:19.510562+0000","last_clean":"2024-06-04T04:54:19.510562+0000","last_became_active":"2024-06-04T04:53:36.380591+0000","last_became_peered":"2024-06-04T04:53:36.380591+0000","last_unstale":"2024-06-04T04:54:19.510562+0000","last_undegraded":"2024-06-04T04:54:19.510562+0000","last_fullsized":"2024-06-04T04:54:19.510562+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:26:43.122339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'5","reported_seq":55,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.509038+0000","last_change":"2024-06-04T04:53:33.355908+0000","last_active":"2024-06-04T04:54:18.509038+0000","last_peered":"2024-06-04T04:54:18.509038+0000","last_clean":"2024-06-04T04:54:18.509038+0000","last_became_active":"2024-06-04T04:53:33.355522+0000","last_became_peered":"2024-06-04T04:53:33.355522+0000","last_unstale":"2024-06-04T04:54:18.509038+0000","last_undegraded":"2024-06-04T04:54:18.509038+0000","last_fullsized":"2024-06-04T04:54:18.509038+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:16:44.639431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510919+0000","last_change":"2024-06-04T04:53:36.382737+0000","last_active":"2024-06-04T04:54:19.510919+0000","last_peered":"2024-06-04T04:54:19.510919+0000","last_clean":"2024-06-04T04:54:19.510919+0000","last_became_active":"2024-06-04T04:53:36.382353+0000","last_became_peered":"2024-06-04T04:53:36.382353+0000","last_unstale":"2024-06-04T04:54:19.510919+0000","last_undegraded":"2024-06-04T04:54:19.510919+0000","last_fullsized":"2024-06-04T04:54:19.510919+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:18:01.476058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512086+0000","last_change":"2024-06-04T04:53:33.347273+0000","last_active":"2024-06-04T04:54:19.512086+0000","last_peered":"2024-06-04T04:54:19.512086+0000","last_clean":"2024-06-04T04:54:19.512086+0000","last_became_active":"2024-06-04T04:53:33.346915+0000","last_became_peered":"2024-06-04T04:53:33.346915+0000","last_unstale":"2024-06-04T04:54:19.512086+0000","last_undegraded":"2024-06-04T04:54:19.512086+0000","last_fullsized":"2024-06-04T04:54:19.512086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:17.235484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"79'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510494+0000","last_change":"2024-06-04T04:53:36.383454+0000","last_active":"2024-06-04T04:54:19.510494+0000","last_peered":"2024-06-04T04:54:19.510494+0000","last_clean":"2024-06-04T04:54:19.510494+0000","last_became_active":"2024-06-04T04:53:36.383105+0000","last_became_peered":"2024-06-04T04:53:36.383105+0000","last_unstale":"2024-06-04T04:54:19.510494+0000","last_undegraded":"2024-06-04T04:54:19.510494+0000","last_fullsized":"2024-06-04T04:54:19.510494+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:04:02.147688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505624+0000","last_change":"2024-06-04T04:53:33.345435+0000","last_active":"2024-06-04T04:54:19.505624+0000","last_peered":"2024-06-04T04:54:19.505624+0000","last_clean":"2024-06-04T04:54:19.505624+0000","last_became_active":"2024-06-04T04:53:33.345077+0000","last_became_peered":"2024-06-04T04:53:33.345077+0000","last_unstale":"2024-06-04T04:54:19.505624+0000","last_undegraded":"2024-06-04T04:54:19.505624+0000","last_fullsized":"2024-06-04T04:54:19.505624+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:20:13.992259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510565+0000","last_change":"2024-06-04T04:53:36.385238+0000","last_active":"2024-06-04T04:54:19.510565+0000","last_peered":"2024-06-04T04:54:19.510565+0000","last_clean":"2024-06-04T04:54:19.510565+0000","last_became_active":"2024-06-04T04:53:36.384899+0000","last_became_peered":"2024-06-04T04:53:36.384899+0000","last_unstale":"2024-06-04T04:54:19.510565+0000","last_undegraded":"2024-06-04T04:54:19.510565+0000","last_fullsized":"2024-06-04T04:54:19.510565+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:09:42.629340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509793+0000","last_change":"2024-06-04T04:53:36.383450+0000","last_active":"2024-06-04T04:54:19.509793+0000","last_peered":"2024-06-04T04:54:19.509793+0000","last_clean":"2024-06-04T04:54:19.509793+0000","last_became_active":"2024-06-04T04:53:36.382678+0000","last_became_peered":"2024-06-04T04:53:36.382678+0000","last_unstale":"2024-06-04T04:54:19.509793+0000","last_undegraded":"2024-06-04T04:54:19.509793+0000","last_fullsized":"2024-06-04T04:54:19.509793+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:36:28.932636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":79,"state":"peering","last_fresh":"2024-06-04T04:53:38.658184+0000","last_change":"2024-06-04T04:54:18.509509+0000","last_active":"2024-06-04T04:54:18.509509+0000","last_peered":"2024-06-04T04:53:38.658184+0000","last_clean":"2024-06-04T04:53:38.658184+0000","last_became_active":"2024-06-04T04:53:36.375049+0000","last_became_peered":"2024-06-04T04:53:36.375049+0000","last_unstale":"2024-06-04T04:53:38.658184+0000","last_undegraded":"2024-06-04T04:53:38.658184+0000","last_fullsized":"2024-06-04T04:53:38.658184+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5,9],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.508282+0000","last_change":"2024-06-04T04:53:36.381408+0000","last_active":"2024-06-04T04:54:18.508282+0000","last_peered":"2024-06-04T04:54:18.508282+0000","last_clean":"2024-06-04T04:54:18.508282+0000","last_became_active":"2024-06-04T04:53:36.381078+0000","last_became_peered":"2024-06-04T04:53:36.381078+0000","last_unstale":"2024-06-04T04:54:18.508282+0000","last_undegraded":"2024-06-04T04:54:18.508282+0000","last_fullsized":"2024-06-04T04:54:18.508282+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:28:48.988591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510556+0000","last_change":"2024-06-04T04:53:36.360168+0000","last_active":"2024-06-04T04:54:19.510556+0000","last_peered":"2024-06-04T04:54:19.510556+0000","last_clean":"2024-06-04T04:54:19.510556+0000","last_became_active":"2024-06-04T04:53:36.359691+0000","last_became_peered":"2024-06-04T04:53:36.359691+0000","last_unstale":"2024-06-04T04:54:19.510556+0000","last_undegraded":"2024-06-04T04:54:19.510556+0000","last_fullsized":"2024-06-04T04:54:19.510556+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:55:17.418339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505605+0000","last_change":"2024-06-04T04:53:36.363018+0000","last_active":"2024-06-04T04:54:19.505605+0000","last_peered":"2024-06-04T04:54:19.505605+0000","last_clean":"2024-06-04T04:54:19.505605+0000","last_became_active":"2024-06-04T04:53:36.361595+0000","last_became_peered":"2024-06-04T04:53:36.361595+0000","last_unstale":"2024-06-04T04:54:19.505605+0000","last_undegraded":"2024-06-04T04:54:19.505605+0000","last_fullsized":"2024-06-04T04:54:19.505605+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:00:21.377322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509182+0000","last_change":"2024-06-04T04:53:36.375377+0000","last_active":"2024-06-04T04:54:19.509182+0000","last_peered":"2024-06-04T04:54:19.509182+0000","last_clean":"2024-06-04T04:54:19.509182+0000","last_became_active":"2024-06-04T04:53:36.373459+0000","last_became_peered":"2024-06-04T04:53:36.373459+0000","last_unstale":"2024-06-04T04:54:19.509182+0000","last_undegraded":"2024-06-04T04:54:19.509182+0000","last_fullsized":"2024-06-04T04:54:19.509182+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:51:31.686969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.509760+0000","last_change":"2024-06-04T04:53:36.373850+0000","last_active":"2024-06-04T04:54:18.509760+0000","last_peered":"2024-06-04T04:54:18.509760+0000","last_clean":"2024-06-04T04:54:18.509760+0000","last_became_active":"2024-06-04T04:53:36.371951+0000","last_became_peered":"2024-06-04T04:53:36.371951+0000","last_unstale":"2024-06-04T04:54:18.509760+0000","last_undegraded":"2024-06-04T04:54:18.509760+0000","last_fullsized":"2024-06-04T04:54:18.509760+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:06.452703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.505025+0000","last_change":"2024-06-04T04:53:36.374871+0000","last_active":"2024-06-04T04:54:18.505025+0000","last_peered":"2024-06-04T04:54:18.505025+0000","last_clean":"2024-06-04T04:54:18.505025+0000","last_became_active":"2024-06-04T04:53:36.374486+0000","last_became_peered":"2024-06-04T04:53:36.374486+0000","last_unstale":"2024-06-04T04:54:18.505025+0000","last_undegraded":"2024-06-04T04:54:18.505025+0000","last_fullsized":"2024-06-04T04:54:18.505025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:15:13.337374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.514431+0000","last_change":"2024-06-04T04:54:19.513540+0000","last_active":"2024-06-04T04:54:19.514431+0000","last_peered":"2024-06-04T04:54:19.514431+0000","last_clean":"2024-06-04T04:54:19.514431+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:54:19.514431+0000","last_undegraded":"2024-06-04T04:54:19.514431+0000","last_fullsized":"2024-06-04T04:54:19.514431+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:09:27.993585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504476+0000","last_change":"2024-06-04T04:53:36.373055+0000","last_active":"2024-06-04T04:54:18.504476+0000","last_peered":"2024-06-04T04:54:18.504476+0000","last_clean":"2024-06-04T04:54:18.504476+0000","last_became_active":"2024-06-04T04:53:36.372754+0000","last_became_peered":"2024-06-04T04:53:36.372754+0000","last_unstale":"2024-06-04T04:54:18.504476+0000","last_undegraded":"2024-06-04T04:54:18.504476+0000","last_fullsized":"2024-06-04T04:54:18.504476+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:22:27.554200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631332+0000","last_change":"2024-06-04T04:53:36.378897+0000","last_active":"2024-06-04T04:54:19.631332+0000","last_peered":"2024-06-04T04:54:19.631332+0000","last_clean":"2024-06-04T04:54:19.631332+0000","last_became_active":"2024-06-04T04:53:36.378607+0000","last_became_peered":"2024-06-04T04:53:36.378607+0000","last_unstale":"2024-06-04T04:54:19.631332+0000","last_undegraded":"2024-06-04T04:54:19.631332+0000","last_fullsized":"2024-06-04T04:54:19.631332+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:03.395491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505499+0000","last_change":"2024-06-04T04:53:36.378240+0000","last_active":"2024-06-04T04:54:19.505499+0000","last_peered":"2024-06-04T04:54:19.505499+0000","last_clean":"2024-06-04T04:54:19.505499+0000","last_became_active":"2024-06-04T04:53:36.377909+0000","last_became_peered":"2024-06-04T04:53:36.377909+0000","last_unstale":"2024-06-04T04:54:19.505499+0000","last_undegraded":"2024-06-04T04:54:19.505499+0000","last_fullsized":"2024-06-04T04:54:19.505499+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:15:25.288340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510525+0000","last_change":"2024-06-04T04:53:36.379792+0000","last_active":"2024-06-04T04:54:19.510525+0000","last_peered":"2024-06-04T04:54:19.510525+0000","last_clean":"2024-06-04T04:54:19.510525+0000","last_became_active":"2024-06-04T04:53:36.379438+0000","last_became_peered":"2024-06-04T04:53:36.379438+0000","last_unstale":"2024-06-04T04:54:19.510525+0000","last_undegraded":"2024-06-04T04:54:19.510525+0000","last_fullsized":"2024-06-04T04:54:19.510525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:23:57.617877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":214930,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61185,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":288,"num_read_kb":210,"num_write":238,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":205613,"data_compressed":5499,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":204,"ondisk_log_size":204,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678040,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":880640,"data_stored":429560,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874268,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":1057669,"data_compressed":37925,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037794,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":880640,"data_stored":412198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234023,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":412576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430251,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":1013853,"data_compressed":38968,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593778,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":888832,"data_stored":429598,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":496158,"data_compressed":7592,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":909312,"data_stored":430546,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411397,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":478113,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":892928,"data_stored":420499,"data_compressed":2876,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953532,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":876544,"data_stored":420535,"data_compressed":2876,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790007,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":1013140,"data_compressed":38968,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34470,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":67880,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":67037,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":56108,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-04T04:54:23.563 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-04T04:54:23.804 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:23.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:23 smithi070 ceph-mon[32631]: pgmap v313: 97 pgs: 1 peering, 96 active+clean; 637 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-06-04T04:54:23.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:23 smithi112 ceph-mon[36453]: pgmap v313: 97 pgs: 1 peering, 96 active+clean; 637 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-06-04T04:54:23.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:23 smithi098 ceph-mon[35569]: pgmap v313: 97 pgs: 1 peering, 96 active+clean; 637 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-06-04T04:54:24.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:24 smithi070 ceph-mon[32631]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:54:24.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:24 smithi112 ceph-mon[36453]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:54:24.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:24 smithi098 ceph-mon[35569]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:54:25.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.451+0000 7ff1abb7b700 1 -- 172.21.15.70:0/3499715401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1a4108520 msgr2=0x7ff1a4108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:25.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.451+0000 7ff1abb7b700 1 --2- 172.21.15.70:0/3499715401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1a4108520 0x7ff1a4108900 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff194009990 tx=0x7ff194009ca0 comp rx=0 tx=0).stop 2024-06-04T04:54:25.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.451+0000 7ff1abb7b700 1 -- 172.21.15.70:0/3499715401 shutdown_connections 2024-06-04T04:54:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.451+0000 7ff1abb7b700 1 --2- 172.21.15.70:0/3499715401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a4102e00 0x7ff1a410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.451+0000 7ff1abb7b700 1 --2- 172.21.15.70:0/3499715401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1a4108520 0x7ff1a4108900 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.451+0000 7ff1abb7b700 1 --2- 172.21.15.70:0/3499715401 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1a4102440 0x7ff1a41028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.451+0000 7ff1abb7b700 1 -- 172.21.15.70:0/3499715401 >> 172.21.15.70:0/3499715401 conn(0x7ff1a40793f0 msgr2=0x7ff1a4079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.451+0000 7ff1abb7b700 1 -- 172.21.15.70:0/3499715401 shutdown_connections 2024-06-04T04:54:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.451+0000 7ff1abb7b700 1 -- 172.21.15.70:0/3499715401 wait complete. 2024-06-04T04:54:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1abb7b700 1 Processor -- start 2024-06-04T04:54:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1abb7b700 1 -- start start 2024-06-04T04:54:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1abb7b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1a4102440 0x7ff1a419a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1abb7b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a4102e00 0x7ff1a419aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1abb7b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1a4108520 0x7ff1a41a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1abb7b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a41124a0 con 0x7ff1a4108520 2024-06-04T04:54:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1abb7b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a4112320 con 0x7ff1a4102e00 2024-06-04T04:54:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1abb7b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a4112620 con 0x7ff1a4102440 2024-06-04T04:54:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1aa118700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1a4108520 0x7ff1a41a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1aa118700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1a4108520 0x7ff1a41a5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40380/0 (socket says 172.21.15.70:40380) 2024-06-04T04:54:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1a9917700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1a4102440 0x7ff1a419a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1a9116700 1 --2- 172.21.15.70:0/1993044024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a4102e00 0x7ff1a419aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1aa118700 1 -- 172.21.15.70:0/1993044024 learned_addr learned my addr 172.21.15.70:0/1993044024 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1aa118700 1 -- 172.21.15.70:0/1993044024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1a4102440 msgr2=0x7ff1a419a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1aa118700 1 --2- 172.21.15.70:0/1993044024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1a4102440 0x7ff1a419a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1aa118700 1 -- 172.21.15.70:0/1993044024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a4102e00 msgr2=0x7ff1a419aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1aa118700 1 --2- 172.21.15.70:0/1993044024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a4102e00 0x7ff1a419aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1aa118700 1 -- 172.21.15.70:0/1993044024 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a41a5f60 con 0x7ff1a4108520 2024-06-04T04:54:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff1aa118700 1 --2- 172.21.15.70:0/1993044024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1a4108520 0x7ff1a41a5a20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00ada0 tx=0x7ff19c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.452+0000 7ff19affd700 1 -- 172.21.15.70:0/1993044024 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c017020 con 0x7ff1a4108520 2024-06-04T04:54:25.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.453+0000 7ff1abb7b700 1 -- 172.21.15.70:0/1993044024 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1a41a6190 con 0x7ff1a4108520 2024-06-04T04:54:25.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.453+0000 7ff19affd700 1 -- 172.21.15.70:0/1993044024 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff19c00cd60 con 0x7ff1a4108520 2024-06-04T04:54:25.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.453+0000 7ff19affd700 1 -- 172.21.15.70:0/1993044024 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c005280 con 0x7ff1a4108520 2024-06-04T04:54:25.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.453+0000 7ff1abb7b700 1 -- 172.21.15.70:0/1993044024 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1a41a6650 con 0x7ff1a4108520 2024-06-04T04:54:25.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.453+0000 7ff1abb7b700 1 -- 172.21.15.70:0/1993044024 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1a410a9b0 con 0x7ff1a4108520 2024-06-04T04:54:25.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.455+0000 7ff19affd700 1 -- 172.21.15.70:0/1993044024 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7ff19c00c8d0 con 0x7ff1a4108520 2024-06-04T04:54:25.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.455+0000 7ff19affd700 1 --2- 172.21.15.70:0/1993044024 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff1900b2910 0x7ff1900b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:25.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.455+0000 7ff1a9917700 1 --2- 172.21.15.70:0/1993044024 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff1900b2910 0x7ff1900b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:25.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.456+0000 7ff19affd700 1 -- 172.21.15.70:0/1993044024 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7ff19c0e1fd0 con 0x7ff1a4108520 2024-06-04T04:54:25.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.456+0000 7ff1a9917700 1 --2- 172.21.15.70:0/1993044024 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff1900b2910 0x7ff1900b4dd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff19400b5c0 tx=0x7ff194005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:25.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.459+0000 7ff19affd700 1 -- 172.21.15.70:0/1993044024 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff19c021020 con 0x7ff1a4108520 2024-06-04T04:54:25.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.662+0000 7ff1abb7b700 1 -- 172.21.15.70:0/1993044024 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff1a4063980 con 0x7ff1900b2910 2024-06-04T04:54:25.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.669+0000 7ff19affd700 1 -- 172.21.15.70:0/1993044024 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259024 (secure 0 0 0) 0x7ff1a4063980 con 0x7ff1900b2910 2024-06-04T04:54:25.675 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:54:25.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.672+0000 7ff1abb7b700 1 -- 172.21.15.70:0/1993044024 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff1900b2910 msgr2=0x7ff1900b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:25.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.672+0000 7ff1abb7b700 1 --2- 172.21.15.70:0/1993044024 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff1900b2910 0x7ff1900b4dd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff19400b5c0 tx=0x7ff194005960 comp rx=0 tx=0).stop 2024-06-04T04:54:25.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.672+0000 7ff1abb7b700 1 -- 172.21.15.70:0/1993044024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1a4108520 msgr2=0x7ff1a41a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:25.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.672+0000 7ff1abb7b700 1 --2- 172.21.15.70:0/1993044024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1a4108520 0x7ff1a41a5a20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00ada0 tx=0x7ff19c00c330 comp rx=0 tx=0).stop 2024-06-04T04:54:25.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.672+0000 7ff1abb7b700 1 -- 172.21.15.70:0/1993044024 shutdown_connections 2024-06-04T04:54:25.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.672+0000 7ff1abb7b700 1 --2- 172.21.15.70:0/1993044024 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff1900b2910 0x7ff1900b4dd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:25.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.672+0000 7ff1abb7b700 1 --2- 172.21.15.70:0/1993044024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1a4108520 0x7ff1a41a5a20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:25.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.672+0000 7ff1abb7b700 1 --2- 172.21.15.70:0/1993044024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1a4102440 0x7ff1a419a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:25.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.672+0000 7ff1abb7b700 1 --2- 172.21.15.70:0/1993044024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a4102e00 0x7ff1a419aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:25.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.672+0000 7ff1abb7b700 1 -- 172.21.15.70:0/1993044024 >> 172.21.15.70:0/1993044024 conn(0x7ff1a40793f0 msgr2=0x7ff1a4101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:25.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.672+0000 7ff1abb7b700 1 -- 172.21.15.70:0/1993044024 shutdown_connections 2024-06-04T04:54:25.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:25.672+0000 7ff1abb7b700 1 -- 172.21.15.70:0/1993044024 wait complete. 2024-06-04T04:54:25.678 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-06-04T04:54:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:25 smithi070 ceph-mon[32631]: pgmap v314: 97 pgs: 97 active+clean; 647 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 83 B/s, 1 objects/s recovering 2024-06-04T04:54:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:25 smithi112 ceph-mon[36453]: pgmap v314: 97 pgs: 97 active+clean; 647 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 83 B/s, 1 objects/s recovering 2024-06-04T04:54:25.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:25 smithi098 ceph-mon[35569]: pgmap v314: 97 pgs: 97 active+clean; 647 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 83 B/s, 1 objects/s recovering 2024-06-04T04:54:26.344 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-06-04T04:54:23.732443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662480,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":394,"num_read_kb":304,"num_write":369,"num_write_kb":2292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2952512,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":304,"ondisk_log_size":304,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":296,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":355904,"kb_used_data":10844,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585920,"statfs":{"total":1151940427776,"available":1151575982080,"internally_reserved":0,"allocated":11104256,"data_stored":7093634,"data_compressed":171351,"data_compressed_allocated":229376,"data_compressed_original":1957888,"omap_allocated":19138,"internal_metadata":353154366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20134,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":60,"num_write":13,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001902"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509657+0000","last_change":"2024-06-04T04:53:36.383755+0000","last_active":"2024-06-04T04:54:19.509657+0000","last_peered":"2024-06-04T04:54:19.509657+0000","last_clean":"2024-06-04T04:54:19.509657+0000","last_became_active":"2024-06-04T04:53:36.378274+0000","last_became_peered":"2024-06-04T04:53:36.378274+0000","last_unstale":"2024-06-04T04:54:19.509657+0000","last_undegraded":"2024-06-04T04:54:19.509657+0000","last_fullsized":"2024-06-04T04:54:19.509657+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:30:19.685994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510014+0000","last_change":"2024-06-04T04:53:36.372436+0000","last_active":"2024-06-04T04:54:19.510014+0000","last_peered":"2024-06-04T04:54:19.510014+0000","last_clean":"2024-06-04T04:54:19.510014+0000","last_became_active":"2024-06-04T04:53:36.372100+0000","last_became_peered":"2024-06-04T04:53:36.372100+0000","last_unstale":"2024-06-04T04:54:19.510014+0000","last_undegraded":"2024-06-04T04:54:19.510014+0000","last_fullsized":"2024-06-04T04:54:19.510014+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:42:31.646754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510448+0000","last_change":"2024-06-04T04:53:36.366915+0000","last_active":"2024-06-04T04:54:19.510448+0000","last_peered":"2024-06-04T04:54:19.510448+0000","last_clean":"2024-06-04T04:54:19.510448+0000","last_became_active":"2024-06-04T04:53:36.366611+0000","last_became_peered":"2024-06-04T04:53:36.366611+0000","last_unstale":"2024-06-04T04:54:19.510448+0000","last_undegraded":"2024-06-04T04:54:19.510448+0000","last_fullsized":"2024-06-04T04:54:19.510448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:17:48.775938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631647+0000","last_change":"2024-06-04T04:53:36.374486+0000","last_active":"2024-06-04T04:54:19.631647+0000","last_peered":"2024-06-04T04:54:19.631647+0000","last_clean":"2024-06-04T04:54:19.631647+0000","last_became_active":"2024-06-04T04:53:36.374178+0000","last_became_peered":"2024-06-04T04:53:36.374178+0000","last_unstale":"2024-06-04T04:54:19.631647+0000","last_undegraded":"2024-06-04T04:54:19.631647+0000","last_fullsized":"2024-06-04T04:54:19.631647+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:30:28.287070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508460+0000","last_change":"2024-06-04T04:53:36.376183+0000","last_active":"2024-06-04T04:54:19.508460+0000","last_peered":"2024-06-04T04:54:19.508460+0000","last_clean":"2024-06-04T04:54:19.508460+0000","last_became_active":"2024-06-04T04:53:36.375790+0000","last_became_peered":"2024-06-04T04:53:36.375790+0000","last_unstale":"2024-06-04T04:54:19.508460+0000","last_undegraded":"2024-06-04T04:54:19.508460+0000","last_fullsized":"2024-06-04T04:54:19.508460+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:58.697440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509028+0000","last_change":"2024-06-04T04:53:36.370249+0000","last_active":"2024-06-04T04:54:19.509028+0000","last_peered":"2024-06-04T04:54:19.509028+0000","last_clean":"2024-06-04T04:54:19.509028+0000","last_became_active":"2024-06-04T04:53:36.369916+0000","last_became_peered":"2024-06-04T04:53:36.369916+0000","last_unstale":"2024-06-04T04:54:19.509028+0000","last_undegraded":"2024-06-04T04:54:19.509028+0000","last_fullsized":"2024-06-04T04:54:19.509028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:02:41.739806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510237+0000","last_change":"2024-06-04T04:53:36.384329+0000","last_active":"2024-06-04T04:54:19.510237+0000","last_peered":"2024-06-04T04:54:19.510237+0000","last_clean":"2024-06-04T04:54:19.510237+0000","last_became_active":"2024-06-04T04:53:36.382504+0000","last_became_peered":"2024-06-04T04:53:36.382504+0000","last_unstale":"2024-06-04T04:54:19.510237+0000","last_undegraded":"2024-06-04T04:54:19.510237+0000","last_fullsized":"2024-06-04T04:54:19.510237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:51:07.604691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504427+0000","last_change":"2024-06-04T04:53:36.372319+0000","last_active":"2024-06-04T04:54:18.504427+0000","last_peered":"2024-06-04T04:54:18.504427+0000","last_clean":"2024-06-04T04:54:18.504427+0000","last_became_active":"2024-06-04T04:53:36.372000+0000","last_became_peered":"2024-06-04T04:53:36.372000+0000","last_unstale":"2024-06-04T04:54:18.504427+0000","last_undegraded":"2024-06-04T04:54:18.504427+0000","last_fullsized":"2024-06-04T04:54:18.504427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:15:56.906996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631893+0000","last_change":"2024-06-04T04:53:36.376051+0000","last_active":"2024-06-04T04:54:19.631893+0000","last_peered":"2024-06-04T04:54:19.631893+0000","last_clean":"2024-06-04T04:54:19.631893+0000","last_became_active":"2024-06-04T04:53:36.375726+0000","last_became_peered":"2024-06-04T04:53:36.375726+0000","last_unstale":"2024-06-04T04:54:19.631893+0000","last_undegraded":"2024-06-04T04:54:19.631893+0000","last_fullsized":"2024-06-04T04:54:19.631893+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:09:08.542764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509369+0000","last_change":"2024-06-04T04:53:36.373266+0000","last_active":"2024-06-04T04:54:19.509369+0000","last_peered":"2024-06-04T04:54:19.509369+0000","last_clean":"2024-06-04T04:54:19.509369+0000","last_became_active":"2024-06-04T04:53:36.372898+0000","last_became_peered":"2024-06-04T04:53:36.372898+0000","last_unstale":"2024-06-04T04:54:19.509369+0000","last_undegraded":"2024-06-04T04:54:19.509369+0000","last_fullsized":"2024-06-04T04:54:19.509369+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:41:43.436781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510727+0000","last_change":"2024-06-04T04:53:36.381344+0000","last_active":"2024-06-04T04:54:19.510727+0000","last_peered":"2024-06-04T04:54:19.510727+0000","last_clean":"2024-06-04T04:54:19.510727+0000","last_became_active":"2024-06-04T04:53:36.380736+0000","last_became_peered":"2024-06-04T04:53:36.380736+0000","last_unstale":"2024-06-04T04:54:19.510727+0000","last_undegraded":"2024-06-04T04:54:19.510727+0000","last_fullsized":"2024-06-04T04:54:19.510727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:23:01.607272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631528+0000","last_change":"2024-06-04T04:53:36.374035+0000","last_active":"2024-06-04T04:54:19.631528+0000","last_peered":"2024-06-04T04:54:19.631528+0000","last_clean":"2024-06-04T04:54:19.631528+0000","last_became_active":"2024-06-04T04:53:36.373667+0000","last_became_peered":"2024-06-04T04:53:36.373667+0000","last_unstale":"2024-06-04T04:54:19.631528+0000","last_undegraded":"2024-06-04T04:54:19.631528+0000","last_fullsized":"2024-06-04T04:54:19.631528+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:23:07.880369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505537+0000","last_change":"2024-06-04T04:53:36.369613+0000","last_active":"2024-06-04T04:54:19.505537+0000","last_peered":"2024-06-04T04:54:19.505537+0000","last_clean":"2024-06-04T04:54:19.505537+0000","last_became_active":"2024-06-04T04:53:36.369277+0000","last_became_peered":"2024-06-04T04:53:36.369277+0000","last_unstale":"2024-06-04T04:54:19.505537+0000","last_undegraded":"2024-06-04T04:54:19.505537+0000","last_fullsized":"2024-06-04T04:54:19.505537+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:13:33.756609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510066+0000","last_change":"2024-06-04T04:53:36.368094+0000","last_active":"2024-06-04T04:54:19.510066+0000","last_peered":"2024-06-04T04:54:19.510066+0000","last_clean":"2024-06-04T04:54:19.510066+0000","last_became_active":"2024-06-04T04:53:36.367717+0000","last_became_peered":"2024-06-04T04:53:36.367717+0000","last_unstale":"2024-06-04T04:54:19.510066+0000","last_undegraded":"2024-06-04T04:54:19.510066+0000","last_fullsized":"2024-06-04T04:54:19.510066+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:39:56.305078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509882+0000","last_change":"2024-06-04T04:53:36.366765+0000","last_active":"2024-06-04T04:54:19.509882+0000","last_peered":"2024-06-04T04:54:19.509882+0000","last_clean":"2024-06-04T04:54:19.509882+0000","last_became_active":"2024-06-04T04:53:36.366438+0000","last_became_peered":"2024-06-04T04:53:36.366438+0000","last_unstale":"2024-06-04T04:54:19.509882+0000","last_undegraded":"2024-06-04T04:54:19.509882+0000","last_fullsized":"2024-06-04T04:54:19.509882+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:38:37.313926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631086+0000","last_change":"2024-06-04T04:53:36.380088+0000","last_active":"2024-06-04T04:54:19.631086+0000","last_peered":"2024-06-04T04:54:19.631086+0000","last_clean":"2024-06-04T04:54:19.631086+0000","last_became_active":"2024-06-04T04:53:36.379467+0000","last_became_peered":"2024-06-04T04:53:36.379467+0000","last_unstale":"2024-06-04T04:54:19.631086+0000","last_undegraded":"2024-06-04T04:54:19.631086+0000","last_fullsized":"2024-06-04T04:54:19.631086+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:46:53.236419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509925+0000","last_change":"2024-06-04T04:53:36.381126+0000","last_active":"2024-06-04T04:54:19.509925+0000","last_peered":"2024-06-04T04:54:19.509925+0000","last_clean":"2024-06-04T04:54:19.509925+0000","last_became_active":"2024-06-04T04:53:36.378833+0000","last_became_peered":"2024-06-04T04:53:36.378833+0000","last_unstale":"2024-06-04T04:54:19.509925+0000","last_undegraded":"2024-06-04T04:54:19.509925+0000","last_fullsized":"2024-06-04T04:54:19.509925+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:20:28.639273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508689+0000","last_change":"2024-06-04T04:53:36.369692+0000","last_active":"2024-06-04T04:54:19.508689+0000","last_peered":"2024-06-04T04:54:19.508689+0000","last_clean":"2024-06-04T04:54:19.508689+0000","last_became_active":"2024-06-04T04:53:36.369370+0000","last_became_peered":"2024-06-04T04:53:36.369370+0000","last_unstale":"2024-06-04T04:54:19.508689+0000","last_undegraded":"2024-06-04T04:54:19.508689+0000","last_fullsized":"2024-06-04T04:54:19.508689+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:55.275465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.506778+0000","last_change":"2024-06-04T04:53:36.383639+0000","last_active":"2024-06-04T04:54:19.506778+0000","last_peered":"2024-06-04T04:54:19.506778+0000","last_clean":"2024-06-04T04:54:19.506778+0000","last_became_active":"2024-06-04T04:53:36.383303+0000","last_became_peered":"2024-06-04T04:53:36.383303+0000","last_unstale":"2024-06-04T04:54:19.506778+0000","last_undegraded":"2024-06-04T04:54:19.506778+0000","last_fullsized":"2024-06-04T04:54:19.506778+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:26:31.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":89,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:21.635720+0000","last_change":"2024-06-04T04:54:19.641520+0000","last_active":"2024-06-04T04:54:21.635720+0000","last_peered":"2024-06-04T04:54:21.635720+0000","last_clean":"2024-06-04T04:54:21.635720+0000","last_became_active":"2024-06-04T04:54:19.632433+0000","last_became_peered":"2024-06-04T04:54:19.632433+0000","last_unstale":"2024-06-04T04:54:21.635720+0000","last_undegraded":"2024-06-04T04:54:21.635720+0000","last_fullsized":"2024-06-04T04:54:21.635720+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:26:05.993919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504978+0000","last_change":"2024-06-04T04:53:36.371827+0000","last_active":"2024-06-04T04:54:18.504978+0000","last_peered":"2024-06-04T04:54:18.504978+0000","last_clean":"2024-06-04T04:54:18.504978+0000","last_became_active":"2024-06-04T04:53:36.371509+0000","last_became_peered":"2024-06-04T04:53:36.371509+0000","last_unstale":"2024-06-04T04:54:18.504978+0000","last_undegraded":"2024-06-04T04:54:18.504978+0000","last_fullsized":"2024-06-04T04:54:18.504978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:54:51.589008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.505150+0000","last_change":"2024-06-04T04:53:33.353976+0000","last_active":"2024-06-04T04:54:18.505150+0000","last_peered":"2024-06-04T04:54:18.505150+0000","last_clean":"2024-06-04T04:54:18.505150+0000","last_became_active":"2024-06-04T04:53:33.353586+0000","last_became_peered":"2024-06-04T04:53:33.353586+0000","last_unstale":"2024-06-04T04:54:18.505150+0000","last_undegraded":"2024-06-04T04:54:18.505150+0000","last_fullsized":"2024-06-04T04:54:18.505150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:17:10.378008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510428+0000","last_change":"2024-06-04T04:53:36.367806+0000","last_active":"2024-06-04T04:54:19.510428+0000","last_peered":"2024-06-04T04:54:19.510428+0000","last_clean":"2024-06-04T04:54:19.510428+0000","last_became_active":"2024-06-04T04:53:36.367484+0000","last_became_peered":"2024-06-04T04:53:36.367484+0000","last_unstale":"2024-06-04T04:54:19.510428+0000","last_undegraded":"2024-06-04T04:54:19.510428+0000","last_fullsized":"2024-06-04T04:54:19.510428+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:25:36.194746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509723+0000","last_change":"2024-06-04T04:53:33.348902+0000","last_active":"2024-06-04T04:54:19.509723+0000","last_peered":"2024-06-04T04:54:19.509723+0000","last_clean":"2024-06-04T04:54:19.509723+0000","last_became_active":"2024-06-04T04:53:33.348441+0000","last_became_peered":"2024-06-04T04:53:33.348441+0000","last_unstale":"2024-06-04T04:54:19.509723+0000","last_undegraded":"2024-06-04T04:54:19.509723+0000","last_fullsized":"2024-06-04T04:54:19.509723+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:22:28.428303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631324+0000","last_change":"2024-06-04T04:53:36.376867+0000","last_active":"2024-06-04T04:54:19.631324+0000","last_peered":"2024-06-04T04:54:19.631324+0000","last_clean":"2024-06-04T04:54:19.631324+0000","last_became_active":"2024-06-04T04:53:36.376517+0000","last_became_peered":"2024-06-04T04:53:36.376517+0000","last_unstale":"2024-06-04T04:54:19.631324+0000","last_undegraded":"2024-06-04T04:54:19.631324+0000","last_fullsized":"2024-06-04T04:54:19.631324+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:21:59.494321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509187+0000","last_change":"2024-06-04T04:53:33.347993+0000","last_active":"2024-06-04T04:54:19.509187+0000","last_peered":"2024-06-04T04:54:19.509187+0000","last_clean":"2024-06-04T04:54:19.509187+0000","last_became_active":"2024-06-04T04:53:33.347656+0000","last_became_peered":"2024-06-04T04:53:33.347656+0000","last_unstale":"2024-06-04T04:54:19.509187+0000","last_undegraded":"2024-06-04T04:54:19.509187+0000","last_fullsized":"2024-06-04T04:54:19.509187+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:21:26.806557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.632790+0000","last_change":"2024-06-04T04:54:19.632790+0000","last_active":"2024-06-04T04:54:19.632790+0000","last_peered":"2024-06-04T04:54:19.632790+0000","last_clean":"2024-06-04T04:54:19.632790+0000","last_became_active":"2024-06-04T04:54:19.632338+0000","last_became_peered":"2024-06-04T04:54:19.632338+0000","last_unstale":"2024-06-04T04:54:19.632790+0000","last_undegraded":"2024-06-04T04:54:19.632790+0000","last_fullsized":"2024-06-04T04:54:19.632790+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:44:00.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'5","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512034+0000","last_change":"2024-06-04T04:53:33.346111+0000","last_active":"2024-06-04T04:54:19.512034+0000","last_peered":"2024-06-04T04:54:19.512034+0000","last_clean":"2024-06-04T04:54:19.512034+0000","last_became_active":"2024-06-04T04:53:33.345773+0000","last_became_peered":"2024-06-04T04:53:33.345773+0000","last_unstale":"2024-06-04T04:54:19.512034+0000","last_undegraded":"2024-06-04T04:54:19.512034+0000","last_fullsized":"2024-06-04T04:54:19.512034+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:18.565583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508684+0000","last_change":"2024-06-04T04:53:36.373369+0000","last_active":"2024-06-04T04:54:19.508684+0000","last_peered":"2024-06-04T04:54:19.508684+0000","last_clean":"2024-06-04T04:54:19.508684+0000","last_became_active":"2024-06-04T04:53:36.373001+0000","last_became_peered":"2024-06-04T04:53:36.373001+0000","last_unstale":"2024-06-04T04:54:19.508684+0000","last_undegraded":"2024-06-04T04:54:19.508684+0000","last_fullsized":"2024-06-04T04:54:19.508684+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:02:22.012643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508915+0000","last_change":"2024-06-04T04:53:33.344358+0000","last_active":"2024-06-04T04:54:19.508915+0000","last_peered":"2024-06-04T04:54:19.508915+0000","last_clean":"2024-06-04T04:54:19.508915+0000","last_became_active":"2024-06-04T04:53:33.343980+0000","last_became_peered":"2024-06-04T04:53:33.343980+0000","last_unstale":"2024-06-04T04:54:19.508915+0000","last_undegraded":"2024-06-04T04:54:19.508915+0000","last_fullsized":"2024-06-04T04:54:19.508915+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:14.493138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512293+0000","last_change":"2024-06-04T04:53:36.371239+0000","last_active":"2024-06-04T04:54:19.512293+0000","last_peered":"2024-06-04T04:54:19.512293+0000","last_clean":"2024-06-04T04:54:19.512293+0000","last_became_active":"2024-06-04T04:53:36.370885+0000","last_became_peered":"2024-06-04T04:53:36.370885+0000","last_unstale":"2024-06-04T04:54:19.512293+0000","last_undegraded":"2024-06-04T04:54:19.512293+0000","last_fullsized":"2024-06-04T04:54:19.512293+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T04:54:29.753796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505354+0000","last_change":"2024-06-04T04:53:33.338512+0000","last_active":"2024-06-04T04:54:19.505354+0000","last_peered":"2024-06-04T04:54:19.505354+0000","last_clean":"2024-06-04T04:54:19.505354+0000","last_became_active":"2024-06-04T04:53:33.338180+0000","last_became_peered":"2024-06-04T04:53:33.338180+0000","last_unstale":"2024-06-04T04:54:19.505354+0000","last_undegraded":"2024-06-04T04:54:19.505354+0000","last_fullsized":"2024-06-04T04:54:19.505354+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:57:13.406479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504521+0000","last_change":"2024-06-04T04:53:36.377316+0000","last_active":"2024-06-04T04:54:18.504521+0000","last_peered":"2024-06-04T04:54:18.504521+0000","last_clean":"2024-06-04T04:54:18.504521+0000","last_became_active":"2024-06-04T04:53:36.376992+0000","last_became_peered":"2024-06-04T04:53:36.376992+0000","last_unstale":"2024-06-04T04:54:18.504521+0000","last_undegraded":"2024-06-04T04:54:18.504521+0000","last_fullsized":"2024-06-04T04:54:18.504521+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:33.145631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511696+0000","last_change":"2024-06-04T04:53:33.348305+0000","last_active":"2024-06-04T04:54:19.511696+0000","last_peered":"2024-06-04T04:54:19.511696+0000","last_clean":"2024-06-04T04:54:19.511696+0000","last_became_active":"2024-06-04T04:53:33.347948+0000","last_became_peered":"2024-06-04T04:53:33.347948+0000","last_unstale":"2024-06-04T04:54:19.511696+0000","last_undegraded":"2024-06-04T04:54:19.511696+0000","last_fullsized":"2024-06-04T04:54:19.511696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:04:13.669274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508429+0000","last_change":"2024-06-04T04:53:36.367955+0000","last_active":"2024-06-04T04:54:19.508429+0000","last_peered":"2024-06-04T04:54:19.508429+0000","last_clean":"2024-06-04T04:54:19.508429+0000","last_became_active":"2024-06-04T04:53:36.367628+0000","last_became_peered":"2024-06-04T04:53:36.367628+0000","last_unstale":"2024-06-04T04:54:19.508429+0000","last_undegraded":"2024-06-04T04:54:19.508429+0000","last_fullsized":"2024-06-04T04:54:19.508429+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:33:26.185498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'4","reported_seq":87,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:22.493752+0000","last_change":"2024-06-04T04:53:33.349471+0000","last_active":"2024-06-04T04:54:22.493752+0000","last_peered":"2024-06-04T04:54:22.493752+0000","last_clean":"2024-06-04T04:54:22.493752+0000","last_became_active":"2024-06-04T04:53:33.349105+0000","last_became_peered":"2024-06-04T04:53:33.349105+0000","last_unstale":"2024-06-04T04:54:22.493752+0000","last_undegraded":"2024-06-04T04:54:22.493752+0000","last_fullsized":"2024-06-04T04:54:22.493752+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:31:35.584275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":58,"num_read_kb":58,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631859+0000","last_change":"2024-06-04T04:53:36.367249+0000","last_active":"2024-06-04T04:54:19.631859+0000","last_peered":"2024-06-04T04:54:19.631859+0000","last_clean":"2024-06-04T04:54:19.631859+0000","last_became_active":"2024-06-04T04:53:36.366863+0000","last_became_peered":"2024-06-04T04:53:36.366863+0000","last_unstale":"2024-06-04T04:54:19.631859+0000","last_undegraded":"2024-06-04T04:54:19.631859+0000","last_fullsized":"2024-06-04T04:54:19.631859+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:13:17.308321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510836+0000","last_change":"2024-06-04T04:53:33.346778+0000","last_active":"2024-06-04T04:54:19.510836+0000","last_peered":"2024-06-04T04:54:19.510836+0000","last_clean":"2024-06-04T04:54:19.510836+0000","last_became_active":"2024-06-04T04:53:33.346376+0000","last_became_peered":"2024-06-04T04:53:33.346376+0000","last_unstale":"2024-06-04T04:54:19.510836+0000","last_undegraded":"2024-06-04T04:54:19.510836+0000","last_fullsized":"2024-06-04T04:54:19.510836+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:03:16.973822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.514491+0000","last_change":"2024-06-04T04:54:19.514491+0000","last_active":"2024-06-04T04:54:19.514491+0000","last_peered":"2024-06-04T04:54:19.514491+0000","last_clean":"2024-06-04T04:54:19.514491+0000","last_became_active":"2024-06-04T04:54:19.514148+0000","last_became_peered":"2024-06-04T04:54:19.514148+0000","last_unstale":"2024-06-04T04:54:19.514491+0000","last_undegraded":"2024-06-04T04:54:19.514491+0000","last_fullsized":"2024-06-04T04:54:19.514491+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:09:59.641041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512326+0000","last_change":"2024-06-04T04:53:33.354989+0000","last_active":"2024-06-04T04:54:19.512326+0000","last_peered":"2024-06-04T04:54:19.512326+0000","last_clean":"2024-06-04T04:54:19.512326+0000","last_became_active":"2024-06-04T04:53:33.354613+0000","last_became_peered":"2024-06-04T04:53:33.354613+0000","last_unstale":"2024-06-04T04:54:19.512326+0000","last_undegraded":"2024-06-04T04:54:19.512326+0000","last_fullsized":"2024-06-04T04:54:19.512326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:19:59.398763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631419+0000","last_change":"2024-06-04T04:53:36.372216+0000","last_active":"2024-06-04T04:54:19.631419+0000","last_peered":"2024-06-04T04:54:19.631419+0000","last_clean":"2024-06-04T04:54:19.631419+0000","last_became_active":"2024-06-04T04:53:36.371820+0000","last_became_peered":"2024-06-04T04:53:36.371820+0000","last_unstale":"2024-06-04T04:54:19.631419+0000","last_undegraded":"2024-06-04T04:54:19.631419+0000","last_fullsized":"2024-06-04T04:54:19.631419+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:55:59.887963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"79'33","reported_seq":124,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:22.499064+0000","last_change":"2024-06-04T04:53:33.345493+0000","last_active":"2024-06-04T04:54:22.499064+0000","last_peered":"2024-06-04T04:54:22.499064+0000","last_clean":"2024-06-04T04:54:22.499064+0000","last_became_active":"2024-06-04T04:53:33.345102+0000","last_became_peered":"2024-06-04T04:53:33.345102+0000","last_unstale":"2024-06-04T04:54:22.499064+0000","last_undegraded":"2024-06-04T04:54:22.499064+0000","last_fullsized":"2024-06-04T04:54:22.499064+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:50:27.853391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65065,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":66,"num_read_kb":96,"num_write":35,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509448+0000","last_change":"2024-06-04T04:53:36.382070+0000","last_active":"2024-06-04T04:54:19.509448+0000","last_peered":"2024-06-04T04:54:19.509448+0000","last_clean":"2024-06-04T04:54:19.509448+0000","last_became_active":"2024-06-04T04:53:36.379144+0000","last_became_peered":"2024-06-04T04:53:36.379144+0000","last_unstale":"2024-06-04T04:54:19.509448+0000","last_undegraded":"2024-06-04T04:54:19.509448+0000","last_fullsized":"2024-06-04T04:54:19.509448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:16.702709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.506537+0000","last_change":"2024-06-04T04:53:33.344726+0000","last_active":"2024-06-04T04:54:19.506537+0000","last_peered":"2024-06-04T04:54:19.506537+0000","last_clean":"2024-06-04T04:54:19.506537+0000","last_became_active":"2024-06-04T04:53:33.344360+0000","last_became_peered":"2024-06-04T04:53:33.344360+0000","last_unstale":"2024-06-04T04:54:19.506537+0000","last_undegraded":"2024-06-04T04:54:19.506537+0000","last_fullsized":"2024-06-04T04:54:19.506537+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:44:56.784808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505351+0000","last_change":"2024-06-04T04:53:36.379036+0000","last_active":"2024-06-04T04:54:19.505351+0000","last_peered":"2024-06-04T04:54:19.505351+0000","last_clean":"2024-06-04T04:54:19.505351+0000","last_became_active":"2024-06-04T04:53:36.378681+0000","last_became_peered":"2024-06-04T04:53:36.378681+0000","last_unstale":"2024-06-04T04:54:19.505351+0000","last_undegraded":"2024-06-04T04:54:19.505351+0000","last_fullsized":"2024-06-04T04:54:19.505351+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:07:49.704731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511967+0000","last_change":"2024-06-04T04:53:33.345915+0000","last_active":"2024-06-04T04:54:19.511967+0000","last_peered":"2024-06-04T04:54:19.511967+0000","last_clean":"2024-06-04T04:54:19.511967+0000","last_became_active":"2024-06-04T04:53:33.345537+0000","last_became_peered":"2024-06-04T04:53:33.345537+0000","last_unstale":"2024-06-04T04:54:19.511967+0000","last_undegraded":"2024-06-04T04:54:19.511967+0000","last_fullsized":"2024-06-04T04:54:19.511967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:06:44.086968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510431+0000","last_change":"2024-06-04T04:53:36.379094+0000","last_active":"2024-06-04T04:54:19.510431+0000","last_peered":"2024-06-04T04:54:19.510431+0000","last_clean":"2024-06-04T04:54:19.510431+0000","last_became_active":"2024-06-04T04:53:36.378763+0000","last_became_peered":"2024-06-04T04:53:36.378763+0000","last_unstale":"2024-06-04T04:54:19.510431+0000","last_undegraded":"2024-06-04T04:54:19.510431+0000","last_fullsized":"2024-06-04T04:54:19.510431+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:16.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.630622+0000","last_change":"2024-06-04T04:53:33.341342+0000","last_active":"2024-06-04T04:54:19.630622+0000","last_peered":"2024-06-04T04:54:19.630622+0000","last_clean":"2024-06-04T04:54:19.630622+0000","last_became_active":"2024-06-04T04:53:33.340862+0000","last_became_peered":"2024-06-04T04:53:33.340862+0000","last_unstale":"2024-06-04T04:54:19.630622+0000","last_undegraded":"2024-06-04T04:54:19.630622+0000","last_fullsized":"2024-06-04T04:54:19.630622+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:26.818063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510293+0000","last_change":"2024-06-04T04:53:36.383124+0000","last_active":"2024-06-04T04:54:19.510293+0000","last_peered":"2024-06-04T04:54:19.510293+0000","last_clean":"2024-06-04T04:54:19.510293+0000","last_became_active":"2024-06-04T04:53:36.382449+0000","last_became_peered":"2024-06-04T04:53:36.382449+0000","last_unstale":"2024-06-04T04:54:19.510293+0000","last_undegraded":"2024-06-04T04:54:19.510293+0000","last_fullsized":"2024-06-04T04:54:19.510293+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:52:40.776003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":44,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508738+0000","last_change":"2024-06-04T04:53:33.354556+0000","last_active":"2024-06-04T04:54:19.508738+0000","last_peered":"2024-06-04T04:54:19.508738+0000","last_clean":"2024-06-04T04:54:19.508738+0000","last_became_active":"2024-06-04T04:53:33.354238+0000","last_became_peered":"2024-06-04T04:53:33.354238+0000","last_unstale":"2024-06-04T04:54:19.508738+0000","last_undegraded":"2024-06-04T04:54:19.508738+0000","last_fullsized":"2024-06-04T04:54:19.508738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:54.697374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511906+0000","last_change":"2024-06-04T04:53:36.371387+0000","last_active":"2024-06-04T04:54:19.511906+0000","last_peered":"2024-06-04T04:54:19.511906+0000","last_clean":"2024-06-04T04:54:19.511906+0000","last_became_active":"2024-06-04T04:53:36.370852+0000","last_became_peered":"2024-06-04T04:53:36.370852+0000","last_unstale":"2024-06-04T04:54:19.511906+0000","last_undegraded":"2024-06-04T04:54:19.511906+0000","last_fullsized":"2024-06-04T04:54:19.511906+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:44:06.956210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508796+0000","last_change":"2024-06-04T04:53:33.349982+0000","last_active":"2024-06-04T04:54:19.508796+0000","last_peered":"2024-06-04T04:54:19.508796+0000","last_clean":"2024-06-04T04:54:19.508796+0000","last_became_active":"2024-06-04T04:53:33.349633+0000","last_became_peered":"2024-06-04T04:53:33.349633+0000","last_unstale":"2024-06-04T04:54:19.508796+0000","last_undegraded":"2024-06-04T04:54:19.508796+0000","last_fullsized":"2024-06-04T04:54:19.508796+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:40.107426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509399+0000","last_change":"2024-06-04T04:53:36.382313+0000","last_active":"2024-06-04T04:54:19.509399+0000","last_peered":"2024-06-04T04:54:19.509399+0000","last_clean":"2024-06-04T04:54:19.509399+0000","last_became_active":"2024-06-04T04:53:36.379638+0000","last_became_peered":"2024-06-04T04:53:36.379638+0000","last_unstale":"2024-06-04T04:54:19.509399+0000","last_undegraded":"2024-06-04T04:54:19.509399+0000","last_fullsized":"2024-06-04T04:54:19.509399+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:43:42.527262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.513581+0000","last_change":"2024-06-04T04:54:19.513581+0000","last_active":"2024-06-04T04:54:19.513581+0000","last_peered":"2024-06-04T04:54:19.513581+0000","last_clean":"2024-06-04T04:54:19.513581+0000","last_became_active":"2024-06-04T04:54:19.513111+0000","last_became_peered":"2024-06-04T04:54:19.513111+0000","last_unstale":"2024-06-04T04:54:19.513581+0000","last_undegraded":"2024-06-04T04:54:19.513581+0000","last_fullsized":"2024-06-04T04:54:19.513581+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:57.028083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508983+0000","last_change":"2024-06-04T04:53:36.369904+0000","last_active":"2024-06-04T04:54:19.508983+0000","last_peered":"2024-06-04T04:54:19.508983+0000","last_clean":"2024-06-04T04:54:19.508983+0000","last_became_active":"2024-06-04T04:53:36.367929+0000","last_became_peered":"2024-06-04T04:53:36.367929+0000","last_unstale":"2024-06-04T04:54:19.508983+0000","last_undegraded":"2024-06-04T04:54:19.508983+0000","last_fullsized":"2024-06-04T04:54:19.508983+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:37:08.721569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'6","reported_seq":60,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510585+0000","last_change":"2024-06-04T04:53:33.342723+0000","last_active":"2024-06-04T04:54:19.510585+0000","last_peered":"2024-06-04T04:54:19.510585+0000","last_clean":"2024-06-04T04:54:19.510585+0000","last_became_active":"2024-06-04T04:53:33.342344+0000","last_became_peered":"2024-06-04T04:53:33.342344+0000","last_unstale":"2024-06-04T04:54:19.510585+0000","last_undegraded":"2024-06-04T04:54:19.510585+0000","last_fullsized":"2024-06-04T04:54:19.510585+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:59:50.147517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.513534+0000","last_change":"2024-06-04T04:54:19.513534+0000","last_active":"2024-06-04T04:54:19.513534+0000","last_peered":"2024-06-04T04:54:19.513534+0000","last_clean":"2024-06-04T04:54:19.513534+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:54:19.513534+0000","last_undegraded":"2024-06-04T04:54:19.513534+0000","last_fullsized":"2024-06-04T04:54:19.513534+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:16:32.240932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512091+0000","last_change":"2024-06-04T04:53:36.371236+0000","last_active":"2024-06-04T04:54:19.512091+0000","last_peered":"2024-06-04T04:54:19.512091+0000","last_clean":"2024-06-04T04:54:19.512091+0000","last_became_active":"2024-06-04T04:53:36.370876+0000","last_became_peered":"2024-06-04T04:53:36.370876+0000","last_unstale":"2024-06-04T04:54:19.512091+0000","last_undegraded":"2024-06-04T04:54:19.512091+0000","last_fullsized":"2024-06-04T04:54:19.512091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:15:56.751317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":58,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505309+0000","last_change":"2024-06-04T04:53:33.355642+0000","last_active":"2024-06-04T04:54:19.505309+0000","last_peered":"2024-06-04T04:54:19.505309+0000","last_clean":"2024-06-04T04:54:19.505309+0000","last_became_active":"2024-06-04T04:53:33.355187+0000","last_became_peered":"2024-06-04T04:53:33.355187+0000","last_unstale":"2024-06-04T04:54:19.505309+0000","last_undegraded":"2024-06-04T04:54:19.505309+0000","last_fullsized":"2024-06-04T04:54:19.505309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:32:28.331518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.632155+0000","last_change":"2024-06-04T04:54:19.631750+0000","last_active":"2024-06-04T04:54:19.632155+0000","last_peered":"2024-06-04T04:54:19.632155+0000","last_clean":"2024-06-04T04:54:19.632155+0000","last_became_active":"2024-06-04T04:54:19.631485+0000","last_became_peered":"2024-06-04T04:54:19.631485+0000","last_unstale":"2024-06-04T04:54:19.632155+0000","last_undegraded":"2024-06-04T04:54:19.632155+0000","last_fullsized":"2024-06-04T04:54:19.632155+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:29:44.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631469+0000","last_change":"2024-06-04T04:53:33.340653+0000","last_active":"2024-06-04T04:54:19.631469+0000","last_peered":"2024-06-04T04:54:19.631469+0000","last_clean":"2024-06-04T04:54:19.631469+0000","last_became_active":"2024-06-04T04:53:33.340250+0000","last_became_peered":"2024-06-04T04:53:33.340250+0000","last_unstale":"2024-06-04T04:54:19.631469+0000","last_undegraded":"2024-06-04T04:54:19.631469+0000","last_fullsized":"2024-06-04T04:54:19.631469+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:46:20.329514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":99,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510637+0000","last_change":"2024-06-04T04:52:09.181155+0000","last_active":"2024-06-04T04:54:19.510637+0000","last_peered":"2024-06-04T04:54:19.510637+0000","last_clean":"2024-06-04T04:54:19.510637+0000","last_became_active":"2024-06-04T04:52:09.177002+0000","last_became_peered":"2024-06-04T04:52:09.177002+0000","last_unstale":"2024-06-04T04:54:19.510637+0000","last_undegraded":"2024-06-04T04:54:19.510637+0000","last_fullsized":"2024-06-04T04:54:19.510637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_clean_scrub_stamp":"2024-06-04T04:48:44.631948+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:46:57.735302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505220+0000","last_change":"2024-06-04T04:53:36.372634+0000","last_active":"2024-06-04T04:54:19.505220+0000","last_peered":"2024-06-04T04:54:19.505220+0000","last_clean":"2024-06-04T04:54:19.505220+0000","last_became_active":"2024-06-04T04:53:36.372290+0000","last_became_peered":"2024-06-04T04:53:36.372290+0000","last_unstale":"2024-06-04T04:54:19.505220+0000","last_undegraded":"2024-06-04T04:54:19.505220+0000","last_fullsized":"2024-06-04T04:54:19.505220+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:14:07.460382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511010+0000","last_change":"2024-06-04T04:53:33.348760+0000","last_active":"2024-06-04T04:54:19.511010+0000","last_peered":"2024-06-04T04:54:19.511010+0000","last_clean":"2024-06-04T04:54:19.511010+0000","last_became_active":"2024-06-04T04:53:33.348347+0000","last_became_peered":"2024-06-04T04:53:33.348347+0000","last_unstale":"2024-06-04T04:54:19.511010+0000","last_undegraded":"2024-06-04T04:54:19.511010+0000","last_fullsized":"2024-06-04T04:54:19.511010+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:36:20.641061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509558+0000","last_change":"2024-06-04T04:53:33.354840+0000","last_active":"2024-06-04T04:54:19.509558+0000","last_peered":"2024-06-04T04:54:19.509558+0000","last_clean":"2024-06-04T04:54:19.509558+0000","last_became_active":"2024-06-04T04:53:33.354287+0000","last_became_peered":"2024-06-04T04:53:33.354287+0000","last_unstale":"2024-06-04T04:54:19.509558+0000","last_undegraded":"2024-06-04T04:54:19.509558+0000","last_fullsized":"2024-06-04T04:54:19.509558+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:42.550098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505559+0000","last_change":"2024-06-04T04:53:36.372678+0000","last_active":"2024-06-04T04:54:19.505559+0000","last_peered":"2024-06-04T04:54:19.505559+0000","last_clean":"2024-06-04T04:54:19.505559+0000","last_became_active":"2024-06-04T04:53:36.372312+0000","last_became_peered":"2024-06-04T04:53:36.372312+0000","last_unstale":"2024-06-04T04:54:19.505559+0000","last_undegraded":"2024-06-04T04:54:19.505559+0000","last_fullsized":"2024-06-04T04:54:19.505559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:34:41.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"79'14","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508647+0000","last_change":"2024-06-04T04:53:33.348440+0000","last_active":"2024-06-04T04:54:19.508647+0000","last_peered":"2024-06-04T04:54:19.508647+0000","last_clean":"2024-06-04T04:54:19.508647+0000","last_became_active":"2024-06-04T04:53:33.348131+0000","last_became_peered":"2024-06-04T04:53:33.348131+0000","last_unstale":"2024-06-04T04:54:19.508647+0000","last_undegraded":"2024-06-04T04:54:19.508647+0000","last_fullsized":"2024-06-04T04:54:19.508647+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:14:11.552286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.505140+0000","last_change":"2024-06-04T04:53:36.378669+0000","last_active":"2024-06-04T04:54:18.505140+0000","last_peered":"2024-06-04T04:54:18.505140+0000","last_clean":"2024-06-04T04:54:18.505140+0000","last_became_active":"2024-06-04T04:53:36.378233+0000","last_became_peered":"2024-06-04T04:53:36.378233+0000","last_unstale":"2024-06-04T04:54:18.505140+0000","last_undegraded":"2024-06-04T04:54:18.505140+0000","last_fullsized":"2024-06-04T04:54:18.505140+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:36:45.870924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631647+0000","last_change":"2024-06-04T04:53:33.345078+0000","last_active":"2024-06-04T04:54:19.631647+0000","last_peered":"2024-06-04T04:54:19.631647+0000","last_clean":"2024-06-04T04:54:19.631647+0000","last_became_active":"2024-06-04T04:53:33.344708+0000","last_became_peered":"2024-06-04T04:53:33.344708+0000","last_unstale":"2024-06-04T04:54:19.631647+0000","last_undegraded":"2024-06-04T04:54:19.631647+0000","last_fullsized":"2024-06-04T04:54:19.631647+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:45:14.010758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510104+0000","last_change":"2024-06-04T04:53:36.364817+0000","last_active":"2024-06-04T04:54:19.510104+0000","last_peered":"2024-06-04T04:54:19.510104+0000","last_clean":"2024-06-04T04:54:19.510104+0000","last_became_active":"2024-06-04T04:53:36.364499+0000","last_became_peered":"2024-06-04T04:53:36.364499+0000","last_unstale":"2024-06-04T04:54:19.510104+0000","last_undegraded":"2024-06-04T04:54:19.510104+0000","last_fullsized":"2024-06-04T04:54:19.510104+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:39:30.518028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.630871+0000","last_change":"2024-06-04T04:53:33.354527+0000","last_active":"2024-06-04T04:54:19.630871+0000","last_peered":"2024-06-04T04:54:19.630871+0000","last_clean":"2024-06-04T04:54:19.630871+0000","last_became_active":"2024-06-04T04:53:33.354175+0000","last_became_peered":"2024-06-04T04:53:33.354175+0000","last_unstale":"2024-06-04T04:54:19.630871+0000","last_undegraded":"2024-06-04T04:54:19.630871+0000","last_fullsized":"2024-06-04T04:54:19.630871+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:33:08.125428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508458+0000","last_change":"2024-06-04T04:53:36.373248+0000","last_active":"2024-06-04T04:54:19.508458+0000","last_peered":"2024-06-04T04:54:19.508458+0000","last_clean":"2024-06-04T04:54:19.508458+0000","last_became_active":"2024-06-04T04:53:36.372915+0000","last_became_peered":"2024-06-04T04:53:36.372915+0000","last_unstale":"2024-06-04T04:54:19.508458+0000","last_undegraded":"2024-06-04T04:54:19.508458+0000","last_fullsized":"2024-06-04T04:54:19.508458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:23:11.689349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504542+0000","last_change":"2024-06-04T04:53:33.340427+0000","last_active":"2024-06-04T04:54:18.504542+0000","last_peered":"2024-06-04T04:54:18.504542+0000","last_clean":"2024-06-04T04:54:18.504542+0000","last_became_active":"2024-06-04T04:53:33.340022+0000","last_became_peered":"2024-06-04T04:53:33.340022+0000","last_unstale":"2024-06-04T04:54:18.504542+0000","last_undegraded":"2024-06-04T04:54:18.504542+0000","last_fullsized":"2024-06-04T04:54:18.504542+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:56:16.496471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509628+0000","last_change":"2024-06-04T04:53:36.380114+0000","last_active":"2024-06-04T04:54:19.509628+0000","last_peered":"2024-06-04T04:54:19.509628+0000","last_clean":"2024-06-04T04:54:19.509628+0000","last_became_active":"2024-06-04T04:53:36.379392+0000","last_became_peered":"2024-06-04T04:53:36.379392+0000","last_unstale":"2024-06-04T04:54:19.509628+0000","last_undegraded":"2024-06-04T04:54:19.509628+0000","last_fullsized":"2024-06-04T04:54:19.509628+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:41:48.776439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.520195+0000","last_change":"2024-06-04T04:54:19.519755+0000","last_active":"2024-06-04T04:54:19.520195+0000","last_peered":"2024-06-04T04:54:19.520195+0000","last_clean":"2024-06-04T04:54:19.520195+0000","last_became_active":"2024-06-04T04:54:19.513992+0000","last_became_peered":"2024-06-04T04:54:19.513992+0000","last_unstale":"2024-06-04T04:54:19.520195+0000","last_undegraded":"2024-06-04T04:54:19.520195+0000","last_fullsized":"2024-06-04T04:54:19.520195+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:01.951935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504890+0000","last_change":"2024-06-04T04:53:36.377420+0000","last_active":"2024-06-04T04:54:18.504890+0000","last_peered":"2024-06-04T04:54:18.504890+0000","last_clean":"2024-06-04T04:54:18.504890+0000","last_became_active":"2024-06-04T04:53:36.377109+0000","last_became_peered":"2024-06-04T04:53:36.377109+0000","last_unstale":"2024-06-04T04:54:18.504890+0000","last_undegraded":"2024-06-04T04:54:18.504890+0000","last_fullsized":"2024-06-04T04:54:18.504890+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:47:12.102396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":43,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.645328+0000","last_change":"2024-06-04T04:54:19.645328+0000","last_active":"2024-06-04T04:54:19.645328+0000","last_peered":"2024-06-04T04:54:19.645328+0000","last_clean":"2024-06-04T04:54:19.645328+0000","last_became_active":"2024-06-04T04:54:19.633416+0000","last_became_peered":"2024-06-04T04:54:19.633416+0000","last_unstale":"2024-06-04T04:54:19.645328+0000","last_undegraded":"2024-06-04T04:54:19.645328+0000","last_fullsized":"2024-06-04T04:54:19.645328+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:04:20.494641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510562+0000","last_change":"2024-06-04T04:53:36.381187+0000","last_active":"2024-06-04T04:54:19.510562+0000","last_peered":"2024-06-04T04:54:19.510562+0000","last_clean":"2024-06-04T04:54:19.510562+0000","last_became_active":"2024-06-04T04:53:36.380591+0000","last_became_peered":"2024-06-04T04:53:36.380591+0000","last_unstale":"2024-06-04T04:54:19.510562+0000","last_undegraded":"2024-06-04T04:54:19.510562+0000","last_fullsized":"2024-06-04T04:54:19.510562+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:26:43.122339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'5","reported_seq":62,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:22.494583+0000","last_change":"2024-06-04T04:53:33.355908+0000","last_active":"2024-06-04T04:54:22.494583+0000","last_peered":"2024-06-04T04:54:22.494583+0000","last_clean":"2024-06-04T04:54:22.494583+0000","last_became_active":"2024-06-04T04:53:33.355522+0000","last_became_peered":"2024-06-04T04:53:33.355522+0000","last_unstale":"2024-06-04T04:54:22.494583+0000","last_undegraded":"2024-06-04T04:54:22.494583+0000","last_fullsized":"2024-06-04T04:54:22.494583+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:16:44.639431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510919+0000","last_change":"2024-06-04T04:53:36.382737+0000","last_active":"2024-06-04T04:54:19.510919+0000","last_peered":"2024-06-04T04:54:19.510919+0000","last_clean":"2024-06-04T04:54:19.510919+0000","last_became_active":"2024-06-04T04:53:36.382353+0000","last_became_peered":"2024-06-04T04:53:36.382353+0000","last_unstale":"2024-06-04T04:54:19.510919+0000","last_undegraded":"2024-06-04T04:54:19.510919+0000","last_fullsized":"2024-06-04T04:54:19.510919+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:18:01.476058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512086+0000","last_change":"2024-06-04T04:53:33.347273+0000","last_active":"2024-06-04T04:54:19.512086+0000","last_peered":"2024-06-04T04:54:19.512086+0000","last_clean":"2024-06-04T04:54:19.512086+0000","last_became_active":"2024-06-04T04:53:33.346915+0000","last_became_peered":"2024-06-04T04:53:33.346915+0000","last_unstale":"2024-06-04T04:54:19.512086+0000","last_undegraded":"2024-06-04T04:54:19.512086+0000","last_fullsized":"2024-06-04T04:54:19.512086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:17.235484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"79'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510494+0000","last_change":"2024-06-04T04:53:36.383454+0000","last_active":"2024-06-04T04:54:19.510494+0000","last_peered":"2024-06-04T04:54:19.510494+0000","last_clean":"2024-06-04T04:54:19.510494+0000","last_became_active":"2024-06-04T04:53:36.383105+0000","last_became_peered":"2024-06-04T04:53:36.383105+0000","last_unstale":"2024-06-04T04:54:19.510494+0000","last_undegraded":"2024-06-04T04:54:19.510494+0000","last_fullsized":"2024-06-04T04:54:19.510494+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:04:02.147688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505624+0000","last_change":"2024-06-04T04:53:33.345435+0000","last_active":"2024-06-04T04:54:19.505624+0000","last_peered":"2024-06-04T04:54:19.505624+0000","last_clean":"2024-06-04T04:54:19.505624+0000","last_became_active":"2024-06-04T04:53:33.345077+0000","last_became_peered":"2024-06-04T04:53:33.345077+0000","last_unstale":"2024-06-04T04:54:19.505624+0000","last_undegraded":"2024-06-04T04:54:19.505624+0000","last_fullsized":"2024-06-04T04:54:19.505624+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:20:13.992259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510565+0000","last_change":"2024-06-04T04:53:36.385238+0000","last_active":"2024-06-04T04:54:19.510565+0000","last_peered":"2024-06-04T04:54:19.510565+0000","last_clean":"2024-06-04T04:54:19.510565+0000","last_became_active":"2024-06-04T04:53:36.384899+0000","last_became_peered":"2024-06-04T04:53:36.384899+0000","last_unstale":"2024-06-04T04:54:19.510565+0000","last_undegraded":"2024-06-04T04:54:19.510565+0000","last_fullsized":"2024-06-04T04:54:19.510565+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:09:42.629340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509793+0000","last_change":"2024-06-04T04:53:36.383450+0000","last_active":"2024-06-04T04:54:19.509793+0000","last_peered":"2024-06-04T04:54:19.509793+0000","last_clean":"2024-06-04T04:54:19.509793+0000","last_became_active":"2024-06-04T04:53:36.382678+0000","last_became_peered":"2024-06-04T04:53:36.382678+0000","last_unstale":"2024-06-04T04:54:19.509793+0000","last_undegraded":"2024-06-04T04:54:19.509793+0000","last_fullsized":"2024-06-04T04:54:19.509793+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:36:28.932636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.632563+0000","last_change":"2024-06-04T04:54:19.632563+0000","last_active":"2024-06-04T04:54:19.632563+0000","last_peered":"2024-06-04T04:54:19.632563+0000","last_clean":"2024-06-04T04:54:19.632563+0000","last_became_active":"2024-06-04T04:54:19.632070+0000","last_became_peered":"2024-06-04T04:54:19.632070+0000","last_unstale":"2024-06-04T04:54:19.632563+0000","last_undegraded":"2024-06-04T04:54:19.632563+0000","last_fullsized":"2024-06-04T04:54:19.632563+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:07:16.552958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.506413+0000","last_change":"2024-06-04T04:53:36.381408+0000","last_active":"2024-06-04T04:54:19.506413+0000","last_peered":"2024-06-04T04:54:19.506413+0000","last_clean":"2024-06-04T04:54:19.506413+0000","last_became_active":"2024-06-04T04:53:36.381078+0000","last_became_peered":"2024-06-04T04:53:36.381078+0000","last_unstale":"2024-06-04T04:54:19.506413+0000","last_undegraded":"2024-06-04T04:54:19.506413+0000","last_fullsized":"2024-06-04T04:54:19.506413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:28:48.988591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510556+0000","last_change":"2024-06-04T04:53:36.360168+0000","last_active":"2024-06-04T04:54:19.510556+0000","last_peered":"2024-06-04T04:54:19.510556+0000","last_clean":"2024-06-04T04:54:19.510556+0000","last_became_active":"2024-06-04T04:53:36.359691+0000","last_became_peered":"2024-06-04T04:53:36.359691+0000","last_unstale":"2024-06-04T04:54:19.510556+0000","last_undegraded":"2024-06-04T04:54:19.510556+0000","last_fullsized":"2024-06-04T04:54:19.510556+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:55:17.418339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505605+0000","last_change":"2024-06-04T04:53:36.363018+0000","last_active":"2024-06-04T04:54:19.505605+0000","last_peered":"2024-06-04T04:54:19.505605+0000","last_clean":"2024-06-04T04:54:19.505605+0000","last_became_active":"2024-06-04T04:53:36.361595+0000","last_became_peered":"2024-06-04T04:53:36.361595+0000","last_unstale":"2024-06-04T04:54:19.505605+0000","last_undegraded":"2024-06-04T04:54:19.505605+0000","last_fullsized":"2024-06-04T04:54:19.505605+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:00:21.377322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509182+0000","last_change":"2024-06-04T04:53:36.375377+0000","last_active":"2024-06-04T04:54:19.509182+0000","last_peered":"2024-06-04T04:54:19.509182+0000","last_clean":"2024-06-04T04:54:19.509182+0000","last_became_active":"2024-06-04T04:53:36.373459+0000","last_became_peered":"2024-06-04T04:53:36.373459+0000","last_unstale":"2024-06-04T04:54:19.509182+0000","last_undegraded":"2024-06-04T04:54:19.509182+0000","last_fullsized":"2024-06-04T04:54:19.509182+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:51:31.686969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.506738+0000","last_change":"2024-06-04T04:53:36.373850+0000","last_active":"2024-06-04T04:54:19.506738+0000","last_peered":"2024-06-04T04:54:19.506738+0000","last_clean":"2024-06-04T04:54:19.506738+0000","last_became_active":"2024-06-04T04:53:36.371951+0000","last_became_peered":"2024-06-04T04:53:36.371951+0000","last_unstale":"2024-06-04T04:54:19.506738+0000","last_undegraded":"2024-06-04T04:54:19.506738+0000","last_fullsized":"2024-06-04T04:54:19.506738+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:06.452703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.505025+0000","last_change":"2024-06-04T04:53:36.374871+0000","last_active":"2024-06-04T04:54:18.505025+0000","last_peered":"2024-06-04T04:54:18.505025+0000","last_clean":"2024-06-04T04:54:18.505025+0000","last_became_active":"2024-06-04T04:53:36.374486+0000","last_became_peered":"2024-06-04T04:53:36.374486+0000","last_unstale":"2024-06-04T04:54:18.505025+0000","last_undegraded":"2024-06-04T04:54:18.505025+0000","last_fullsized":"2024-06-04T04:54:18.505025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:15:13.337374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.514431+0000","last_change":"2024-06-04T04:54:19.513540+0000","last_active":"2024-06-04T04:54:19.514431+0000","last_peered":"2024-06-04T04:54:19.514431+0000","last_clean":"2024-06-04T04:54:19.514431+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:54:19.514431+0000","last_undegraded":"2024-06-04T04:54:19.514431+0000","last_fullsized":"2024-06-04T04:54:19.514431+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:09:27.993585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-06-04T04:54:18.504476+0000","last_change":"2024-06-04T04:53:36.373055+0000","last_active":"2024-06-04T04:54:18.504476+0000","last_peered":"2024-06-04T04:54:18.504476+0000","last_clean":"2024-06-04T04:54:18.504476+0000","last_became_active":"2024-06-04T04:53:36.372754+0000","last_became_peered":"2024-06-04T04:53:36.372754+0000","last_unstale":"2024-06-04T04:54:18.504476+0000","last_undegraded":"2024-06-04T04:54:18.504476+0000","last_fullsized":"2024-06-04T04:54:18.504476+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:22:27.554200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631332+0000","last_change":"2024-06-04T04:53:36.378897+0000","last_active":"2024-06-04T04:54:19.631332+0000","last_peered":"2024-06-04T04:54:19.631332+0000","last_clean":"2024-06-04T04:54:19.631332+0000","last_became_active":"2024-06-04T04:53:36.378607+0000","last_became_peered":"2024-06-04T04:53:36.378607+0000","last_unstale":"2024-06-04T04:54:19.631332+0000","last_undegraded":"2024-06-04T04:54:19.631332+0000","last_fullsized":"2024-06-04T04:54:19.631332+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:03.395491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505499+0000","last_change":"2024-06-04T04:53:36.378240+0000","last_active":"2024-06-04T04:54:19.505499+0000","last_peered":"2024-06-04T04:54:19.505499+0000","last_clean":"2024-06-04T04:54:19.505499+0000","last_became_active":"2024-06-04T04:53:36.377909+0000","last_became_peered":"2024-06-04T04:53:36.377909+0000","last_unstale":"2024-06-04T04:54:19.505499+0000","last_undegraded":"2024-06-04T04:54:19.505499+0000","last_fullsized":"2024-06-04T04:54:19.505499+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:15:25.288340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510525+0000","last_change":"2024-06-04T04:53:36.379792+0000","last_active":"2024-06-04T04:54:19.510525+0000","last_peered":"2024-06-04T04:54:19.510525+0000","last_clean":"2024-06-04T04:54:19.510525+0000","last_became_active":"2024-06-04T04:53:36.379438+0000","last_became_peered":"2024-06-04T04:53:36.379438+0000","last_unstale":"2024-06-04T04:54:19.510525+0000","last_undegraded":"2024-06-04T04:54:19.510525+0000","last_fullsized":"2024-06-04T04:54:19.510525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:23:57.617877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":214930,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71878,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":313,"num_read_kb":235,"num_write":244,"num_write_kb":140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":215634,"data_compressed":5499,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678040,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":880640,"data_stored":429560,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874269,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":1085654,"data_compressed":43684,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037795,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":905216,"data_stored":429490,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234024,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":892928,"data_stored":429196,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430251,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":1013853,"data_compressed":38968,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593778,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":888832,"data_stored":429598,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":496158,"data_compressed":7592,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":909312,"data_stored":430546,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":495405,"data_compressed":7592,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":892928,"data_stored":420499,"data_compressed":2876,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953532,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":876544,"data_stored":420535,"data_compressed":2876,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790007,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":1013140,"data_compressed":38968,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34470,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":67880,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":67037,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":66801,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-04T04:54:26.350 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph status --format=json 2024-06-04T04:54:26.590 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:27.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:27 smithi070 ceph-mon[32631]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:54:27.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:27 smithi070 ceph-mon[32631]: pgmap v315: 97 pgs: 97 active+clean; 647 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 10 op/s; 83 B/s, 1 objects/s recovering 2024-06-04T04:54:27.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:27 smithi112 ceph-mon[36453]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:54:27.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:27 smithi112 ceph-mon[36453]: pgmap v315: 97 pgs: 97 active+clean; 647 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 10 op/s; 83 B/s, 1 objects/s recovering 2024-06-04T04:54:27.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:27 smithi098 ceph-mon[35569]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:54:27.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:27 smithi098 ceph-mon[35569]: pgmap v315: 97 pgs: 97 active+clean; 647 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 10 op/s; 83 B/s, 1 objects/s recovering 2024-06-04T04:54:28.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.301+0000 7f1231119700 1 -- 172.21.15.70:0/1602860261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f122c101ee0 msgr2=0x7f122c102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:28.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.301+0000 7f1231119700 1 --2- 172.21.15.70:0/1602860261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f122c101ee0 0x7f122c102360 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1220009990 tx=0x7f1220009ca0 comp rx=0 tx=0).stop 2024-06-04T04:54:28.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.301+0000 7f1231119700 1 -- 172.21.15.70:0/1602860261 shutdown_connections 2024-06-04T04:54:28.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.301+0000 7f1231119700 1 --2- 172.21.15.70:0/1602860261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f122c1028a0 0x7f122c110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:28.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.301+0000 7f1231119700 1 --2- 172.21.15.70:0/1602860261 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f122c108600 0x7f122c1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:28.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.301+0000 7f1231119700 1 --2- 172.21.15.70:0/1602860261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f122c101ee0 0x7f122c102360 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:28.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.301+0000 7f1231119700 1 -- 172.21.15.70:0/1602860261 >> 172.21.15.70:0/1602860261 conn(0x7f122c077760 msgr2=0x7f122c077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:28.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.301+0000 7f1231119700 1 -- 172.21.15.70:0/1602860261 shutdown_connections 2024-06-04T04:54:28.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.301+0000 7f1231119700 1 -- 172.21.15.70:0/1602860261 wait complete. 2024-06-04T04:54:28.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.302+0000 7f1231119700 1 Processor -- start 2024-06-04T04:54:28.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.302+0000 7f1231119700 1 -- start start 2024-06-04T04:54:28.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.302+0000 7f1231119700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f122c101ee0 0x7f122c19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:28.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.302+0000 7f1231119700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f122c1028a0 0x7f122c19fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:28.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.302+0000 7f1231119700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f122c108600 0x7f122c1999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:28.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.302+0000 7f1231119700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f122c113640 con 0x7f122c1028a0 2024-06-04T04:54:28.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.302+0000 7f1231119700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f122c1134c0 con 0x7f122c108600 2024-06-04T04:54:28.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.302+0000 7f1231119700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f122c1137c0 con 0x7f122c101ee0 2024-06-04T04:54:28.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.302+0000 7f122a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f122c1028a0 0x7f122c19fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:28.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.302+0000 7f122a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f122c1028a0 0x7f122c19fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54430/0 (socket says 172.21.15.70:54430) 2024-06-04T04:54:28.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.302+0000 7f122a59c700 1 -- 172.21.15.70:0/3607137064 learned_addr learned my addr 172.21.15.70:0/3607137064 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:28.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.303+0000 7f122b59e700 1 --2- 172.21.15.70:0/3607137064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f122c108600 0x7f122c1999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:28.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.303+0000 7f122ad9d700 1 --2- 172.21.15.70:0/3607137064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f122c101ee0 0x7f122c19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:28.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.303+0000 7f122a59c700 1 -- 172.21.15.70:0/3607137064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f122c101ee0 msgr2=0x7f122c19f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:28.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.303+0000 7f122a59c700 1 --2- 172.21.15.70:0/3607137064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f122c101ee0 0x7f122c19f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:28.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.303+0000 7f122a59c700 1 -- 172.21.15.70:0/3607137064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f122c108600 msgr2=0x7f122c1999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:28.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.303+0000 7f122a59c700 1 --2- 172.21.15.70:0/3607137064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f122c108600 0x7f122c1999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:28.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.303+0000 7f122a59c700 1 -- 172.21.15.70:0/3607137064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122c19a120 con 0x7f122c1028a0 2024-06-04T04:54:28.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.303+0000 7f122ad9d700 1 --2- 172.21.15.70:0/3607137064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f122c101ee0 0x7f122c19f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:54:28.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.303+0000 7f122b59e700 1 --2- 172.21.15.70:0/3607137064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f122c108600 0x7f122c1999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:54:28.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.303+0000 7f122a59c700 1 --2- 172.21.15.70:0/3607137064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f122c1028a0 0x7f122c19fd40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1220000c00 tx=0x7f1220009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:28.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.303+0000 7f1213fff700 1 -- 172.21.15.70:0/3607137064 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220027070 con 0x7f122c1028a0 2024-06-04T04:54:28.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.303+0000 7f1213fff700 1 -- 172.21.15.70:0/3607137064 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f122000fc70 con 0x7f122c1028a0 2024-06-04T04:54:28.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.303+0000 7f1231119700 1 -- 172.21.15.70:0/3607137064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f122c19a3b0 con 0x7f122c1028a0 2024-06-04T04:54:28.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.303+0000 7f1213fff700 1 -- 172.21.15.70:0/3607137064 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220022d40 con 0x7f122c1028a0 2024-06-04T04:54:28.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.303+0000 7f1231119700 1 -- 172.21.15.70:0/3607137064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f122c1a66d0 con 0x7f122c1028a0 2024-06-04T04:54:28.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.304+0000 7f1231119700 1 -- 172.21.15.70:0/3607137064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f122c06a230 con 0x7f122c1028a0 2024-06-04T04:54:28.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.306+0000 7f1213fff700 1 -- 172.21.15.70:0/3607137064 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f122002a070 con 0x7f122c1028a0 2024-06-04T04:54:28.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.307+0000 7f1213fff700 1 --2- 172.21.15.70:0/3607137064 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f12140b2910 0x7f12140b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:28.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.307+0000 7f122ad9d700 1 --2- 172.21.15.70:0/3607137064 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f12140b2910 0x7f12140b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:28.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.307+0000 7f1213fff700 1 -- 172.21.15.70:0/3607137064 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f12200e3580 con 0x7f122c1028a0 2024-06-04T04:54:28.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.307+0000 7f122ad9d700 1 --2- 172.21.15.70:0/3607137064 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f12140b2910 0x7f12140b4dd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f122c19ab00 tx=0x7f121c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:28.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.311+0000 7f1213fff700 1 -- 172.21.15.70:0/3607137064 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1220031070 con 0x7f122c1028a0 2024-06-04T04:54:28.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.604+0000 7f1231119700 1 -- 172.21.15.70:0/3607137064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f122c02e560 con 0x7f122c1028a0 2024-06-04T04:54:28.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.604+0000 7f1213fff700 1 -- 172.21.15.70:0/3607137064 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+3161 (secure 0 0 0) 0x7f1220013070 con 0x7f122c1028a0 2024-06-04T04:54:28.607 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:54:28.607 INFO:teuthology.orchestra.run.smithi070.stdout:{"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":429,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1717476754,"num_in_osds":12,"osd_in_since":1717476734,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":102,"data_bytes":662480,"bytes_used":364511232,"bytes_avail":1151575916544,"bytes_total":1151940427776,"recovering_objects_per_sec":1,"recovering_bytes_per_sec":72,"recovering_keys_per_sec":0,"num_objects_recovered":10,"num_bytes_recovered":672,"num_keys_recovered":0,"read_bytes_sec":5547,"write_bytes_sec":1664,"read_op_per_sec":9,"write_op_per_sec":0},"fsmap":{"epoch":28,"id":1,"up":5,"in":5,"max":5,"by_rank":[{"filesystem_id":1,"rank":4,"name":"g","status":"up:standby-replay","gid":14580},{"filesystem_id":1,"rank":3,"name":"a","status":"up:standby-replay","gid":14592},{"filesystem_id":1,"rank":2,"name":"k","status":"up:standby-replay","gid":24371},{"filesystem_id":1,"rank":1,"name":"e","status":"up:standby-replay","gid":24386},{"filesystem_id":1,"rank":0,"name":"l","status":"up:standby-replay","gid":24400},{"filesystem_id":1,"rank":4,"name":"h","status":"up:active","gid":24410},{"filesystem_id":1,"rank":3,"name":"f","status":"up:active","gid":24436},{"filesystem_id":1,"rank":2,"name":"b","status":"up:active","gid":24440},{"filesystem_id":1,"rank":1,"name":"c","status":"up:active","gid":24442},{"filesystem_id":1,"rank":0,"name":"i","status":"up:active","gid":24448}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":2,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.70:8443/"}},"servicemap":{"epoch":22,"modified":"2024-06-04T04:53:59.726790+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"e":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"f":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"g":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"h":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"i":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"l":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-06-04T04:54:28.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.606+0000 7f1231119700 1 -- 172.21.15.70:0/3607137064 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f12140b2910 msgr2=0x7f12140b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:28.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.606+0000 7f1231119700 1 --2- 172.21.15.70:0/3607137064 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f12140b2910 0x7f12140b4dd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f122c19ab00 tx=0x7f121c008040 comp rx=0 tx=0).stop 2024-06-04T04:54:28.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.606+0000 7f1231119700 1 -- 172.21.15.70:0/3607137064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f122c1028a0 msgr2=0x7f122c19fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:28.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.606+0000 7f1231119700 1 --2- 172.21.15.70:0/3607137064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f122c1028a0 0x7f122c19fd40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1220000c00 tx=0x7f1220009f90 comp rx=0 tx=0).stop 2024-06-04T04:54:28.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.606+0000 7f1231119700 1 -- 172.21.15.70:0/3607137064 shutdown_connections 2024-06-04T04:54:28.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.606+0000 7f1231119700 1 --2- 172.21.15.70:0/3607137064 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f12140b2910 0x7f12140b4dd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:28.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.606+0000 7f1231119700 1 --2- 172.21.15.70:0/3607137064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f122c108600 0x7f122c1999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:28.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.606+0000 7f1231119700 1 --2- 172.21.15.70:0/3607137064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f122c101ee0 0x7f122c19f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:28.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.606+0000 7f1231119700 1 --2- 172.21.15.70:0/3607137064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f122c1028a0 0x7f122c19fd40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:28.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.606+0000 7f1231119700 1 -- 172.21.15.70:0/3607137064 >> 172.21.15.70:0/3607137064 conn(0x7f122c077760 msgr2=0x7f122c100b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:28.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.606+0000 7f1231119700 1 -- 172.21.15.70:0/3607137064 shutdown_connections 2024-06-04T04:54:28.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:28.607+0000 7f1231119700 1 -- 172.21.15.70:0/3607137064 wait complete. 2024-06-04T04:54:29.260 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-06-04T04:54:29.261 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-04T04:54:29.503 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:29.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:29 smithi070 ceph-mon[32631]: pgmap v316: 97 pgs: 97 active+clean; 647 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.6 KiB/s wr, 10 op/s; 72 B/s, 1 objects/s recovering 2024-06-04T04:54:29.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:29 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3607137064' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-06-04T04:54:29.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:29 smithi112 ceph-mon[36453]: pgmap v316: 97 pgs: 97 active+clean; 647 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.6 KiB/s wr, 10 op/s; 72 B/s, 1 objects/s recovering 2024-06-04T04:54:29.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:29 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3607137064' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-06-04T04:54:29.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:29 smithi098 ceph-mon[35569]: pgmap v316: 97 pgs: 97 active+clean; 647 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.6 KiB/s wr, 10 op/s; 72 B/s, 1 objects/s recovering 2024-06-04T04:54:29.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:29 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3607137064' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-06-04T04:54:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.210+0000 7ff231cab700 1 -- 172.21.15.70:0/944502013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff22c1024f0 msgr2=0x7ff22c1103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.210+0000 7ff231cab700 1 --2- 172.21.15.70:0/944502013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff22c1024f0 0x7ff22c1103d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00b0d0 tx=0x7ff21c00b3e0 comp rx=0 tx=0).stop 2024-06-04T04:54:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.210+0000 7ff231cab700 1 -- 172.21.15.70:0/944502013 shutdown_connections 2024-06-04T04:54:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.210+0000 7ff231cab700 1 --2- 172.21.15.70:0/944502013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff22c1024f0 0x7ff22c1103d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.210+0000 7ff231cab700 1 --2- 172.21.15.70:0/944502013 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff22c109520 0x7ff22c109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.210+0000 7ff231cab700 1 --2- 172.21.15.70:0/944502013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff22c101b30 0x7ff22c101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.210+0000 7ff231cab700 1 -- 172.21.15.70:0/944502013 >> 172.21.15.70:0/944502013 conn(0x7ff22c078ab0 msgr2=0x7ff22c0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.210+0000 7ff231cab700 1 -- 172.21.15.70:0/944502013 shutdown_connections 2024-06-04T04:54:31.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.210+0000 7ff231cab700 1 -- 172.21.15.70:0/944502013 wait complete. 2024-06-04T04:54:31.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff231cab700 1 Processor -- start 2024-06-04T04:54:31.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff231cab700 1 -- start start 2024-06-04T04:54:31.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff231cab700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff22c101b30 0x7ff22c102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:31.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff231cab700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff22c1024f0 0x7ff22c103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:31.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff231cab700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff22c109520 0x7ff22c103790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:31.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff231cab700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c113630 con 0x7ff22c109520 2024-06-04T04:54:31.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff231cab700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1134b0 con 0x7ff22c101b30 2024-06-04T04:54:31.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff231cab700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1137b0 con 0x7ff22c1024f0 2024-06-04T04:54:31.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff22bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff22c109520 0x7ff22c103790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:31.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff22bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff22c109520 0x7ff22c103790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54446/0 (socket says 172.21.15.70:54446) 2024-06-04T04:54:31.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff22bfff700 1 -- 172.21.15.70:0/99373867 learned_addr learned my addr 172.21.15.70:0/99373867 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:31.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff22b7fe700 1 --2- 172.21.15.70:0/99373867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff22c101b30 0x7ff22c102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:31.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff22bfff700 1 -- 172.21.15.70:0/99373867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff22c1024f0 msgr2=0x7ff22c103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:31.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff22affd700 1 --2- 172.21.15.70:0/99373867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff22c1024f0 0x7ff22c103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:31.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff22bfff700 1 --2- 172.21.15.70:0/99373867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff22c1024f0 0x7ff22c103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:31.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff22bfff700 1 -- 172.21.15.70:0/99373867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff22c101b30 msgr2=0x7ff22c102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:31.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff22bfff700 1 --2- 172.21.15.70:0/99373867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff22c101b30 0x7ff22c102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:31.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff22bfff700 1 -- 172.21.15.70:0/99373867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff22c1a80b0 con 0x7ff22c109520 2024-06-04T04:54:31.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff22b7fe700 1 --2- 172.21.15.70:0/99373867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff22c101b30 0x7ff22c102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:54:31.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.211+0000 7ff22affd700 1 --2- 172.21.15.70:0/99373867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff22c1024f0 0x7ff22c103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:54:31.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.212+0000 7ff22bfff700 1 --2- 172.21.15.70:0/99373867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff22c109520 0x7ff22c103790 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff21c007d90 tx=0x7ff21c007e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:31.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.212+0000 7ff228ff9700 1 -- 172.21.15.70:0/99373867 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c01f030 con 0x7ff22c109520 2024-06-04T04:54:31.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.212+0000 7ff231cab700 1 -- 172.21.15.70:0/99373867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff22c1a8340 con 0x7ff22c109520 2024-06-04T04:54:31.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.212+0000 7ff228ff9700 1 -- 172.21.15.70:0/99373867 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff21c01bb90 con 0x7ff22c109520 2024-06-04T04:54:31.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.212+0000 7ff228ff9700 1 -- 172.21.15.70:0/99373867 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c012c90 con 0x7ff22c109520 2024-06-04T04:54:31.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.212+0000 7ff231cab700 1 -- 172.21.15.70:0/99373867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff22c1a87f0 con 0x7ff22c109520 2024-06-04T04:54:31.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.213+0000 7ff231cab700 1 -- 172.21.15.70:0/99373867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff22c104e50 con 0x7ff22c109520 2024-06-04T04:54:31.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.215+0000 7ff228ff9700 1 -- 172.21.15.70:0/99373867 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7ff21c022030 con 0x7ff22c109520 2024-06-04T04:54:31.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.215+0000 7ff228ff9700 1 --2- 172.21.15.70:0/99373867 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff2180b2910 0x7ff2180b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:31.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.216+0000 7ff22b7fe700 1 --2- 172.21.15.70:0/99373867 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff2180b2910 0x7ff2180b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:31.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.216+0000 7ff228ff9700 1 -- 172.21.15.70:0/99373867 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7ff21c00f020 con 0x7ff22c109520 2024-06-04T04:54:31.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.216+0000 7ff22b7fe700 1 --2- 172.21.15.70:0/99373867 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff2180b2910 0x7ff2180b4dd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff224005fd0 tx=0x7ff224005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:31.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.219+0000 7ff228ff9700 1 -- 172.21.15.70:0/99373867 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff21c029020 con 0x7ff22c109520 2024-06-04T04:54:31.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.419+0000 7ff231cab700 1 -- 172.21.15.70:0/99373867 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff22c063980 con 0x7ff2180b2910 2024-06-04T04:54:31.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.426+0000 7ff228ff9700 1 -- 172.21.15.70:0/99373867 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259004 (secure 0 0 0) 0x7ff22c063980 con 0x7ff2180b2910 2024-06-04T04:54:31.432 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:54:31.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.429+0000 7ff231cab700 1 -- 172.21.15.70:0/99373867 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff2180b2910 msgr2=0x7ff2180b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:31.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.429+0000 7ff231cab700 1 --2- 172.21.15.70:0/99373867 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff2180b2910 0x7ff2180b4dd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff224005fd0 tx=0x7ff224005dc0 comp rx=0 tx=0).stop 2024-06-04T04:54:31.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.429+0000 7ff231cab700 1 -- 172.21.15.70:0/99373867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff22c109520 msgr2=0x7ff22c103790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.429+0000 7ff231cab700 1 --2- 172.21.15.70:0/99373867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff22c109520 0x7ff22c103790 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff21c007d90 tx=0x7ff21c007e70 comp rx=0 tx=0).stop 2024-06-04T04:54:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.429+0000 7ff231cab700 1 -- 172.21.15.70:0/99373867 shutdown_connections 2024-06-04T04:54:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.429+0000 7ff231cab700 1 --2- 172.21.15.70:0/99373867 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff2180b2910 0x7ff2180b4dd0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.429+0000 7ff231cab700 1 --2- 172.21.15.70:0/99373867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff22c109520 0x7ff22c103790 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.429+0000 7ff231cab700 1 --2- 172.21.15.70:0/99373867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff22c101b30 0x7ff22c102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.429+0000 7ff231cab700 1 --2- 172.21.15.70:0/99373867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff22c1024f0 0x7ff22c103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.429+0000 7ff231cab700 1 -- 172.21.15.70:0/99373867 >> 172.21.15.70:0/99373867 conn(0x7ff22c078ab0 msgr2=0x7ff22c100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.429+0000 7ff231cab700 1 -- 172.21.15.70:0/99373867 shutdown_connections 2024-06-04T04:54:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:31.429+0000 7ff231cab700 1 -- 172.21.15.70:0/99373867 wait complete. 2024-06-04T04:54:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-06-04T04:54:31.782 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:31 smithi070 ceph-mon[32631]: pgmap v317: 97 pgs: 97 active+clean; 647 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.5 KiB/s wr, 11 op/s; 65 B/s, 0 objects/s recovering 2024-06-04T04:54:31.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:31 smithi112 ceph-mon[36453]: pgmap v317: 97 pgs: 97 active+clean; 647 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.5 KiB/s wr, 11 op/s; 65 B/s, 0 objects/s recovering 2024-06-04T04:54:31.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:31 smithi098 ceph-mon[35569]: pgmap v317: 97 pgs: 97 active+clean; 647 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.5 KiB/s wr, 11 op/s; 65 B/s, 0 objects/s recovering 2024-06-04T04:54:32.114 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-06-04T04:54:29.734011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662480,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":458,"num_read_kb":329,"num_write":369,"num_write_kb":2292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2952512,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":304,"ondisk_log_size":304,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":355968,"kb_used_data":10896,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585856,"statfs":{"total":1151940427776,"available":1151575916544,"internally_reserved":0,"allocated":11157504,"data_stored":7128994,"data_compressed":182883,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19138,"internal_metadata":353154366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10693,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":109,"num_read_kb":55,"num_write":6,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.229870"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509657+0000","last_change":"2024-06-04T04:53:36.383755+0000","last_active":"2024-06-04T04:54:19.509657+0000","last_peered":"2024-06-04T04:54:19.509657+0000","last_clean":"2024-06-04T04:54:19.509657+0000","last_became_active":"2024-06-04T04:53:36.378274+0000","last_became_peered":"2024-06-04T04:53:36.378274+0000","last_unstale":"2024-06-04T04:54:19.509657+0000","last_undegraded":"2024-06-04T04:54:19.509657+0000","last_fullsized":"2024-06-04T04:54:19.509657+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:30:19.685994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510014+0000","last_change":"2024-06-04T04:53:36.372436+0000","last_active":"2024-06-04T04:54:19.510014+0000","last_peered":"2024-06-04T04:54:19.510014+0000","last_clean":"2024-06-04T04:54:19.510014+0000","last_became_active":"2024-06-04T04:53:36.372100+0000","last_became_peered":"2024-06-04T04:53:36.372100+0000","last_unstale":"2024-06-04T04:54:19.510014+0000","last_undegraded":"2024-06-04T04:54:19.510014+0000","last_fullsized":"2024-06-04T04:54:19.510014+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:42:31.646754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510448+0000","last_change":"2024-06-04T04:53:36.366915+0000","last_active":"2024-06-04T04:54:19.510448+0000","last_peered":"2024-06-04T04:54:19.510448+0000","last_clean":"2024-06-04T04:54:19.510448+0000","last_became_active":"2024-06-04T04:53:36.366611+0000","last_became_peered":"2024-06-04T04:53:36.366611+0000","last_unstale":"2024-06-04T04:54:19.510448+0000","last_undegraded":"2024-06-04T04:54:19.510448+0000","last_fullsized":"2024-06-04T04:54:19.510448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:17:48.775938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631647+0000","last_change":"2024-06-04T04:53:36.374486+0000","last_active":"2024-06-04T04:54:19.631647+0000","last_peered":"2024-06-04T04:54:19.631647+0000","last_clean":"2024-06-04T04:54:19.631647+0000","last_became_active":"2024-06-04T04:53:36.374178+0000","last_became_peered":"2024-06-04T04:53:36.374178+0000","last_unstale":"2024-06-04T04:54:19.631647+0000","last_undegraded":"2024-06-04T04:54:19.631647+0000","last_fullsized":"2024-06-04T04:54:19.631647+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:30:28.287070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508460+0000","last_change":"2024-06-04T04:53:36.376183+0000","last_active":"2024-06-04T04:54:19.508460+0000","last_peered":"2024-06-04T04:54:19.508460+0000","last_clean":"2024-06-04T04:54:19.508460+0000","last_became_active":"2024-06-04T04:53:36.375790+0000","last_became_peered":"2024-06-04T04:53:36.375790+0000","last_unstale":"2024-06-04T04:54:19.508460+0000","last_undegraded":"2024-06-04T04:54:19.508460+0000","last_fullsized":"2024-06-04T04:54:19.508460+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:58.697440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509028+0000","last_change":"2024-06-04T04:53:36.370249+0000","last_active":"2024-06-04T04:54:19.509028+0000","last_peered":"2024-06-04T04:54:19.509028+0000","last_clean":"2024-06-04T04:54:19.509028+0000","last_became_active":"2024-06-04T04:53:36.369916+0000","last_became_peered":"2024-06-04T04:53:36.369916+0000","last_unstale":"2024-06-04T04:54:19.509028+0000","last_undegraded":"2024-06-04T04:54:19.509028+0000","last_fullsized":"2024-06-04T04:54:19.509028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:02:41.739806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510237+0000","last_change":"2024-06-04T04:53:36.384329+0000","last_active":"2024-06-04T04:54:19.510237+0000","last_peered":"2024-06-04T04:54:19.510237+0000","last_clean":"2024-06-04T04:54:19.510237+0000","last_became_active":"2024-06-04T04:53:36.382504+0000","last_became_peered":"2024-06-04T04:53:36.382504+0000","last_unstale":"2024-06-04T04:54:19.510237+0000","last_undegraded":"2024-06-04T04:54:19.510237+0000","last_fullsized":"2024-06-04T04:54:19.510237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:51:07.604691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511986+0000","last_change":"2024-06-04T04:53:36.372319+0000","last_active":"2024-06-04T04:54:19.511986+0000","last_peered":"2024-06-04T04:54:19.511986+0000","last_clean":"2024-06-04T04:54:19.511986+0000","last_became_active":"2024-06-04T04:53:36.372000+0000","last_became_peered":"2024-06-04T04:53:36.372000+0000","last_unstale":"2024-06-04T04:54:19.511986+0000","last_undegraded":"2024-06-04T04:54:19.511986+0000","last_fullsized":"2024-06-04T04:54:19.511986+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:15:56.906996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631893+0000","last_change":"2024-06-04T04:53:36.376051+0000","last_active":"2024-06-04T04:54:19.631893+0000","last_peered":"2024-06-04T04:54:19.631893+0000","last_clean":"2024-06-04T04:54:19.631893+0000","last_became_active":"2024-06-04T04:53:36.375726+0000","last_became_peered":"2024-06-04T04:53:36.375726+0000","last_unstale":"2024-06-04T04:54:19.631893+0000","last_undegraded":"2024-06-04T04:54:19.631893+0000","last_fullsized":"2024-06-04T04:54:19.631893+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:09:08.542764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509369+0000","last_change":"2024-06-04T04:53:36.373266+0000","last_active":"2024-06-04T04:54:19.509369+0000","last_peered":"2024-06-04T04:54:19.509369+0000","last_clean":"2024-06-04T04:54:19.509369+0000","last_became_active":"2024-06-04T04:53:36.372898+0000","last_became_peered":"2024-06-04T04:53:36.372898+0000","last_unstale":"2024-06-04T04:54:19.509369+0000","last_undegraded":"2024-06-04T04:54:19.509369+0000","last_fullsized":"2024-06-04T04:54:19.509369+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:41:43.436781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510727+0000","last_change":"2024-06-04T04:53:36.381344+0000","last_active":"2024-06-04T04:54:19.510727+0000","last_peered":"2024-06-04T04:54:19.510727+0000","last_clean":"2024-06-04T04:54:19.510727+0000","last_became_active":"2024-06-04T04:53:36.380736+0000","last_became_peered":"2024-06-04T04:53:36.380736+0000","last_unstale":"2024-06-04T04:54:19.510727+0000","last_undegraded":"2024-06-04T04:54:19.510727+0000","last_fullsized":"2024-06-04T04:54:19.510727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:23:01.607272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631528+0000","last_change":"2024-06-04T04:53:36.374035+0000","last_active":"2024-06-04T04:54:19.631528+0000","last_peered":"2024-06-04T04:54:19.631528+0000","last_clean":"2024-06-04T04:54:19.631528+0000","last_became_active":"2024-06-04T04:53:36.373667+0000","last_became_peered":"2024-06-04T04:53:36.373667+0000","last_unstale":"2024-06-04T04:54:19.631528+0000","last_undegraded":"2024-06-04T04:54:19.631528+0000","last_fullsized":"2024-06-04T04:54:19.631528+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:23:07.880369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505537+0000","last_change":"2024-06-04T04:53:36.369613+0000","last_active":"2024-06-04T04:54:19.505537+0000","last_peered":"2024-06-04T04:54:19.505537+0000","last_clean":"2024-06-04T04:54:19.505537+0000","last_became_active":"2024-06-04T04:53:36.369277+0000","last_became_peered":"2024-06-04T04:53:36.369277+0000","last_unstale":"2024-06-04T04:54:19.505537+0000","last_undegraded":"2024-06-04T04:54:19.505537+0000","last_fullsized":"2024-06-04T04:54:19.505537+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:13:33.756609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510066+0000","last_change":"2024-06-04T04:53:36.368094+0000","last_active":"2024-06-04T04:54:19.510066+0000","last_peered":"2024-06-04T04:54:19.510066+0000","last_clean":"2024-06-04T04:54:19.510066+0000","last_became_active":"2024-06-04T04:53:36.367717+0000","last_became_peered":"2024-06-04T04:53:36.367717+0000","last_unstale":"2024-06-04T04:54:19.510066+0000","last_undegraded":"2024-06-04T04:54:19.510066+0000","last_fullsized":"2024-06-04T04:54:19.510066+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:39:56.305078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509882+0000","last_change":"2024-06-04T04:53:36.366765+0000","last_active":"2024-06-04T04:54:19.509882+0000","last_peered":"2024-06-04T04:54:19.509882+0000","last_clean":"2024-06-04T04:54:19.509882+0000","last_became_active":"2024-06-04T04:53:36.366438+0000","last_became_peered":"2024-06-04T04:53:36.366438+0000","last_unstale":"2024-06-04T04:54:19.509882+0000","last_undegraded":"2024-06-04T04:54:19.509882+0000","last_fullsized":"2024-06-04T04:54:19.509882+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:38:37.313926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631086+0000","last_change":"2024-06-04T04:53:36.380088+0000","last_active":"2024-06-04T04:54:19.631086+0000","last_peered":"2024-06-04T04:54:19.631086+0000","last_clean":"2024-06-04T04:54:19.631086+0000","last_became_active":"2024-06-04T04:53:36.379467+0000","last_became_peered":"2024-06-04T04:53:36.379467+0000","last_unstale":"2024-06-04T04:54:19.631086+0000","last_undegraded":"2024-06-04T04:54:19.631086+0000","last_fullsized":"2024-06-04T04:54:19.631086+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:46:53.236419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509925+0000","last_change":"2024-06-04T04:53:36.381126+0000","last_active":"2024-06-04T04:54:19.509925+0000","last_peered":"2024-06-04T04:54:19.509925+0000","last_clean":"2024-06-04T04:54:19.509925+0000","last_became_active":"2024-06-04T04:53:36.378833+0000","last_became_peered":"2024-06-04T04:53:36.378833+0000","last_unstale":"2024-06-04T04:54:19.509925+0000","last_undegraded":"2024-06-04T04:54:19.509925+0000","last_fullsized":"2024-06-04T04:54:19.509925+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:20:28.639273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508689+0000","last_change":"2024-06-04T04:53:36.369692+0000","last_active":"2024-06-04T04:54:19.508689+0000","last_peered":"2024-06-04T04:54:19.508689+0000","last_clean":"2024-06-04T04:54:19.508689+0000","last_became_active":"2024-06-04T04:53:36.369370+0000","last_became_peered":"2024-06-04T04:53:36.369370+0000","last_unstale":"2024-06-04T04:54:19.508689+0000","last_undegraded":"2024-06-04T04:54:19.508689+0000","last_fullsized":"2024-06-04T04:54:19.508689+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:55.275465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.506778+0000","last_change":"2024-06-04T04:53:36.383639+0000","last_active":"2024-06-04T04:54:19.506778+0000","last_peered":"2024-06-04T04:54:19.506778+0000","last_clean":"2024-06-04T04:54:19.506778+0000","last_became_active":"2024-06-04T04:53:36.383303+0000","last_became_peered":"2024-06-04T04:53:36.383303+0000","last_unstale":"2024-06-04T04:54:19.506778+0000","last_undegraded":"2024-06-04T04:54:19.506778+0000","last_fullsized":"2024-06-04T04:54:19.506778+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:26:31.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":94,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:26.642848+0000","last_change":"2024-06-04T04:54:19.641520+0000","last_active":"2024-06-04T04:54:26.642848+0000","last_peered":"2024-06-04T04:54:26.642848+0000","last_clean":"2024-06-04T04:54:26.642848+0000","last_became_active":"2024-06-04T04:54:19.632433+0000","last_became_peered":"2024-06-04T04:54:19.632433+0000","last_unstale":"2024-06-04T04:54:26.642848+0000","last_undegraded":"2024-06-04T04:54:26.642848+0000","last_fullsized":"2024-06-04T04:54:26.642848+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:26:05.993919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512831+0000","last_change":"2024-06-04T04:53:36.371827+0000","last_active":"2024-06-04T04:54:19.512831+0000","last_peered":"2024-06-04T04:54:19.512831+0000","last_clean":"2024-06-04T04:54:19.512831+0000","last_became_active":"2024-06-04T04:53:36.371509+0000","last_became_peered":"2024-06-04T04:53:36.371509+0000","last_unstale":"2024-06-04T04:54:19.512831+0000","last_undegraded":"2024-06-04T04:54:19.512831+0000","last_fullsized":"2024-06-04T04:54:19.512831+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:54:51.589008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.513177+0000","last_change":"2024-06-04T04:53:33.353976+0000","last_active":"2024-06-04T04:54:19.513177+0000","last_peered":"2024-06-04T04:54:19.513177+0000","last_clean":"2024-06-04T04:54:19.513177+0000","last_became_active":"2024-06-04T04:53:33.353586+0000","last_became_peered":"2024-06-04T04:53:33.353586+0000","last_unstale":"2024-06-04T04:54:19.513177+0000","last_undegraded":"2024-06-04T04:54:19.513177+0000","last_fullsized":"2024-06-04T04:54:19.513177+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:17:10.378008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510428+0000","last_change":"2024-06-04T04:53:36.367806+0000","last_active":"2024-06-04T04:54:19.510428+0000","last_peered":"2024-06-04T04:54:19.510428+0000","last_clean":"2024-06-04T04:54:19.510428+0000","last_became_active":"2024-06-04T04:53:36.367484+0000","last_became_peered":"2024-06-04T04:53:36.367484+0000","last_unstale":"2024-06-04T04:54:19.510428+0000","last_undegraded":"2024-06-04T04:54:19.510428+0000","last_fullsized":"2024-06-04T04:54:19.510428+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:25:36.194746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509723+0000","last_change":"2024-06-04T04:53:33.348902+0000","last_active":"2024-06-04T04:54:19.509723+0000","last_peered":"2024-06-04T04:54:19.509723+0000","last_clean":"2024-06-04T04:54:19.509723+0000","last_became_active":"2024-06-04T04:53:33.348441+0000","last_became_peered":"2024-06-04T04:53:33.348441+0000","last_unstale":"2024-06-04T04:54:19.509723+0000","last_undegraded":"2024-06-04T04:54:19.509723+0000","last_fullsized":"2024-06-04T04:54:19.509723+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:22:28.428303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631324+0000","last_change":"2024-06-04T04:53:36.376867+0000","last_active":"2024-06-04T04:54:19.631324+0000","last_peered":"2024-06-04T04:54:19.631324+0000","last_clean":"2024-06-04T04:54:19.631324+0000","last_became_active":"2024-06-04T04:53:36.376517+0000","last_became_peered":"2024-06-04T04:53:36.376517+0000","last_unstale":"2024-06-04T04:54:19.631324+0000","last_undegraded":"2024-06-04T04:54:19.631324+0000","last_fullsized":"2024-06-04T04:54:19.631324+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:21:59.494321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509187+0000","last_change":"2024-06-04T04:53:33.347993+0000","last_active":"2024-06-04T04:54:19.509187+0000","last_peered":"2024-06-04T04:54:19.509187+0000","last_clean":"2024-06-04T04:54:19.509187+0000","last_became_active":"2024-06-04T04:53:33.347656+0000","last_became_peered":"2024-06-04T04:53:33.347656+0000","last_unstale":"2024-06-04T04:54:19.509187+0000","last_undegraded":"2024-06-04T04:54:19.509187+0000","last_fullsized":"2024-06-04T04:54:19.509187+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:21:26.806557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.632790+0000","last_change":"2024-06-04T04:54:19.632790+0000","last_active":"2024-06-04T04:54:19.632790+0000","last_peered":"2024-06-04T04:54:19.632790+0000","last_clean":"2024-06-04T04:54:19.632790+0000","last_became_active":"2024-06-04T04:54:19.632338+0000","last_became_peered":"2024-06-04T04:54:19.632338+0000","last_unstale":"2024-06-04T04:54:19.632790+0000","last_undegraded":"2024-06-04T04:54:19.632790+0000","last_fullsized":"2024-06-04T04:54:19.632790+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:44:00.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'5","reported_seq":69,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:28.503575+0000","last_change":"2024-06-04T04:53:33.346111+0000","last_active":"2024-06-04T04:54:28.503575+0000","last_peered":"2024-06-04T04:54:28.503575+0000","last_clean":"2024-06-04T04:54:28.503575+0000","last_became_active":"2024-06-04T04:53:33.345773+0000","last_became_peered":"2024-06-04T04:53:33.345773+0000","last_unstale":"2024-06-04T04:54:28.503575+0000","last_undegraded":"2024-06-04T04:54:28.503575+0000","last_fullsized":"2024-06-04T04:54:28.503575+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:18.565583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508684+0000","last_change":"2024-06-04T04:53:36.373369+0000","last_active":"2024-06-04T04:54:19.508684+0000","last_peered":"2024-06-04T04:54:19.508684+0000","last_clean":"2024-06-04T04:54:19.508684+0000","last_became_active":"2024-06-04T04:53:36.373001+0000","last_became_peered":"2024-06-04T04:53:36.373001+0000","last_unstale":"2024-06-04T04:54:19.508684+0000","last_undegraded":"2024-06-04T04:54:19.508684+0000","last_fullsized":"2024-06-04T04:54:19.508684+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:02:22.012643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508915+0000","last_change":"2024-06-04T04:53:33.344358+0000","last_active":"2024-06-04T04:54:19.508915+0000","last_peered":"2024-06-04T04:54:19.508915+0000","last_clean":"2024-06-04T04:54:19.508915+0000","last_became_active":"2024-06-04T04:53:33.343980+0000","last_became_peered":"2024-06-04T04:53:33.343980+0000","last_unstale":"2024-06-04T04:54:19.508915+0000","last_undegraded":"2024-06-04T04:54:19.508915+0000","last_fullsized":"2024-06-04T04:54:19.508915+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:14.493138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512293+0000","last_change":"2024-06-04T04:53:36.371239+0000","last_active":"2024-06-04T04:54:19.512293+0000","last_peered":"2024-06-04T04:54:19.512293+0000","last_clean":"2024-06-04T04:54:19.512293+0000","last_became_active":"2024-06-04T04:53:36.370885+0000","last_became_peered":"2024-06-04T04:53:36.370885+0000","last_unstale":"2024-06-04T04:54:19.512293+0000","last_undegraded":"2024-06-04T04:54:19.512293+0000","last_fullsized":"2024-06-04T04:54:19.512293+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T04:54:29.753796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505354+0000","last_change":"2024-06-04T04:53:33.338512+0000","last_active":"2024-06-04T04:54:19.505354+0000","last_peered":"2024-06-04T04:54:19.505354+0000","last_clean":"2024-06-04T04:54:19.505354+0000","last_became_active":"2024-06-04T04:53:33.338180+0000","last_became_peered":"2024-06-04T04:53:33.338180+0000","last_unstale":"2024-06-04T04:54:19.505354+0000","last_undegraded":"2024-06-04T04:54:19.505354+0000","last_fullsized":"2024-06-04T04:54:19.505354+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:57:13.406479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512117+0000","last_change":"2024-06-04T04:53:36.377316+0000","last_active":"2024-06-04T04:54:19.512117+0000","last_peered":"2024-06-04T04:54:19.512117+0000","last_clean":"2024-06-04T04:54:19.512117+0000","last_became_active":"2024-06-04T04:53:36.376992+0000","last_became_peered":"2024-06-04T04:53:36.376992+0000","last_unstale":"2024-06-04T04:54:19.512117+0000","last_undegraded":"2024-06-04T04:54:19.512117+0000","last_fullsized":"2024-06-04T04:54:19.512117+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:33.145631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511696+0000","last_change":"2024-06-04T04:53:33.348305+0000","last_active":"2024-06-04T04:54:19.511696+0000","last_peered":"2024-06-04T04:54:19.511696+0000","last_clean":"2024-06-04T04:54:19.511696+0000","last_became_active":"2024-06-04T04:53:33.347948+0000","last_became_peered":"2024-06-04T04:53:33.347948+0000","last_unstale":"2024-06-04T04:54:19.511696+0000","last_undegraded":"2024-06-04T04:54:19.511696+0000","last_fullsized":"2024-06-04T04:54:19.511696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:04:13.669274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508429+0000","last_change":"2024-06-04T04:53:36.367955+0000","last_active":"2024-06-04T04:54:19.508429+0000","last_peered":"2024-06-04T04:54:19.508429+0000","last_clean":"2024-06-04T04:54:19.508429+0000","last_became_active":"2024-06-04T04:53:36.367628+0000","last_became_peered":"2024-06-04T04:53:36.367628+0000","last_unstale":"2024-06-04T04:54:19.508429+0000","last_undegraded":"2024-06-04T04:54:19.508429+0000","last_fullsized":"2024-06-04T04:54:19.508429+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:33:26.185498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'4","reported_seq":97,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:27.501030+0000","last_change":"2024-06-04T04:53:33.349471+0000","last_active":"2024-06-04T04:54:27.501030+0000","last_peered":"2024-06-04T04:54:27.501030+0000","last_clean":"2024-06-04T04:54:27.501030+0000","last_became_active":"2024-06-04T04:53:33.349105+0000","last_became_peered":"2024-06-04T04:53:33.349105+0000","last_unstale":"2024-06-04T04:54:27.501030+0000","last_undegraded":"2024-06-04T04:54:27.501030+0000","last_fullsized":"2024-06-04T04:54:27.501030+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:31:35.584275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631859+0000","last_change":"2024-06-04T04:53:36.367249+0000","last_active":"2024-06-04T04:54:19.631859+0000","last_peered":"2024-06-04T04:54:19.631859+0000","last_clean":"2024-06-04T04:54:19.631859+0000","last_became_active":"2024-06-04T04:53:36.366863+0000","last_became_peered":"2024-06-04T04:53:36.366863+0000","last_unstale":"2024-06-04T04:54:19.631859+0000","last_undegraded":"2024-06-04T04:54:19.631859+0000","last_fullsized":"2024-06-04T04:54:19.631859+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:13:17.308321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510836+0000","last_change":"2024-06-04T04:53:33.346778+0000","last_active":"2024-06-04T04:54:19.510836+0000","last_peered":"2024-06-04T04:54:19.510836+0000","last_clean":"2024-06-04T04:54:19.510836+0000","last_became_active":"2024-06-04T04:53:33.346376+0000","last_became_peered":"2024-06-04T04:53:33.346376+0000","last_unstale":"2024-06-04T04:54:19.510836+0000","last_undegraded":"2024-06-04T04:54:19.510836+0000","last_fullsized":"2024-06-04T04:54:19.510836+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:03:16.973822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.514491+0000","last_change":"2024-06-04T04:54:19.514491+0000","last_active":"2024-06-04T04:54:19.514491+0000","last_peered":"2024-06-04T04:54:19.514491+0000","last_clean":"2024-06-04T04:54:19.514491+0000","last_became_active":"2024-06-04T04:54:19.514148+0000","last_became_peered":"2024-06-04T04:54:19.514148+0000","last_unstale":"2024-06-04T04:54:19.514491+0000","last_undegraded":"2024-06-04T04:54:19.514491+0000","last_fullsized":"2024-06-04T04:54:19.514491+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:09:59.641041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512326+0000","last_change":"2024-06-04T04:53:33.354989+0000","last_active":"2024-06-04T04:54:19.512326+0000","last_peered":"2024-06-04T04:54:19.512326+0000","last_clean":"2024-06-04T04:54:19.512326+0000","last_became_active":"2024-06-04T04:53:33.354613+0000","last_became_peered":"2024-06-04T04:53:33.354613+0000","last_unstale":"2024-06-04T04:54:19.512326+0000","last_undegraded":"2024-06-04T04:54:19.512326+0000","last_fullsized":"2024-06-04T04:54:19.512326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:19:59.398763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631419+0000","last_change":"2024-06-04T04:53:36.372216+0000","last_active":"2024-06-04T04:54:19.631419+0000","last_peered":"2024-06-04T04:54:19.631419+0000","last_clean":"2024-06-04T04:54:19.631419+0000","last_became_active":"2024-06-04T04:53:36.371820+0000","last_became_peered":"2024-06-04T04:53:36.371820+0000","last_unstale":"2024-06-04T04:54:19.631419+0000","last_undegraded":"2024-06-04T04:54:19.631419+0000","last_fullsized":"2024-06-04T04:54:19.631419+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:55:59.887963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"79'33","reported_seq":134,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:27.505946+0000","last_change":"2024-06-04T04:53:33.345493+0000","last_active":"2024-06-04T04:54:27.505946+0000","last_peered":"2024-06-04T04:54:27.505946+0000","last_clean":"2024-06-04T04:54:27.505946+0000","last_became_active":"2024-06-04T04:53:33.345102+0000","last_became_peered":"2024-06-04T04:53:33.345102+0000","last_unstale":"2024-06-04T04:54:27.505946+0000","last_undegraded":"2024-06-04T04:54:27.505946+0000","last_fullsized":"2024-06-04T04:54:27.505946+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:50:27.853391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65065,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":76,"num_read_kb":101,"num_write":35,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509448+0000","last_change":"2024-06-04T04:53:36.382070+0000","last_active":"2024-06-04T04:54:19.509448+0000","last_peered":"2024-06-04T04:54:19.509448+0000","last_clean":"2024-06-04T04:54:19.509448+0000","last_became_active":"2024-06-04T04:53:36.379144+0000","last_became_peered":"2024-06-04T04:53:36.379144+0000","last_unstale":"2024-06-04T04:54:19.509448+0000","last_undegraded":"2024-06-04T04:54:19.509448+0000","last_fullsized":"2024-06-04T04:54:19.509448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:16.702709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.506537+0000","last_change":"2024-06-04T04:53:33.344726+0000","last_active":"2024-06-04T04:54:19.506537+0000","last_peered":"2024-06-04T04:54:19.506537+0000","last_clean":"2024-06-04T04:54:19.506537+0000","last_became_active":"2024-06-04T04:53:33.344360+0000","last_became_peered":"2024-06-04T04:53:33.344360+0000","last_unstale":"2024-06-04T04:54:19.506537+0000","last_undegraded":"2024-06-04T04:54:19.506537+0000","last_fullsized":"2024-06-04T04:54:19.506537+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:44:56.784808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505351+0000","last_change":"2024-06-04T04:53:36.379036+0000","last_active":"2024-06-04T04:54:19.505351+0000","last_peered":"2024-06-04T04:54:19.505351+0000","last_clean":"2024-06-04T04:54:19.505351+0000","last_became_active":"2024-06-04T04:53:36.378681+0000","last_became_peered":"2024-06-04T04:53:36.378681+0000","last_unstale":"2024-06-04T04:54:19.505351+0000","last_undegraded":"2024-06-04T04:54:19.505351+0000","last_fullsized":"2024-06-04T04:54:19.505351+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:07:49.704731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511967+0000","last_change":"2024-06-04T04:53:33.345915+0000","last_active":"2024-06-04T04:54:19.511967+0000","last_peered":"2024-06-04T04:54:19.511967+0000","last_clean":"2024-06-04T04:54:19.511967+0000","last_became_active":"2024-06-04T04:53:33.345537+0000","last_became_peered":"2024-06-04T04:53:33.345537+0000","last_unstale":"2024-06-04T04:54:19.511967+0000","last_undegraded":"2024-06-04T04:54:19.511967+0000","last_fullsized":"2024-06-04T04:54:19.511967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:06:44.086968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510431+0000","last_change":"2024-06-04T04:53:36.379094+0000","last_active":"2024-06-04T04:54:19.510431+0000","last_peered":"2024-06-04T04:54:19.510431+0000","last_clean":"2024-06-04T04:54:19.510431+0000","last_became_active":"2024-06-04T04:53:36.378763+0000","last_became_peered":"2024-06-04T04:53:36.378763+0000","last_unstale":"2024-06-04T04:54:19.510431+0000","last_undegraded":"2024-06-04T04:54:19.510431+0000","last_fullsized":"2024-06-04T04:54:19.510431+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:16.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":69,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:28.502638+0000","last_change":"2024-06-04T04:53:33.341342+0000","last_active":"2024-06-04T04:54:28.502638+0000","last_peered":"2024-06-04T04:54:28.502638+0000","last_clean":"2024-06-04T04:54:28.502638+0000","last_became_active":"2024-06-04T04:53:33.340862+0000","last_became_peered":"2024-06-04T04:53:33.340862+0000","last_unstale":"2024-06-04T04:54:28.502638+0000","last_undegraded":"2024-06-04T04:54:28.502638+0000","last_fullsized":"2024-06-04T04:54:28.502638+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:26.818063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":39,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510293+0000","last_change":"2024-06-04T04:53:36.383124+0000","last_active":"2024-06-04T04:54:19.510293+0000","last_peered":"2024-06-04T04:54:19.510293+0000","last_clean":"2024-06-04T04:54:19.510293+0000","last_became_active":"2024-06-04T04:53:36.382449+0000","last_became_peered":"2024-06-04T04:53:36.382449+0000","last_unstale":"2024-06-04T04:54:19.510293+0000","last_undegraded":"2024-06-04T04:54:19.510293+0000","last_fullsized":"2024-06-04T04:54:19.510293+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:52:40.776003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":44,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508738+0000","last_change":"2024-06-04T04:53:33.354556+0000","last_active":"2024-06-04T04:54:19.508738+0000","last_peered":"2024-06-04T04:54:19.508738+0000","last_clean":"2024-06-04T04:54:19.508738+0000","last_became_active":"2024-06-04T04:53:33.354238+0000","last_became_peered":"2024-06-04T04:53:33.354238+0000","last_unstale":"2024-06-04T04:54:19.508738+0000","last_undegraded":"2024-06-04T04:54:19.508738+0000","last_fullsized":"2024-06-04T04:54:19.508738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:54.697374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511906+0000","last_change":"2024-06-04T04:53:36.371387+0000","last_active":"2024-06-04T04:54:19.511906+0000","last_peered":"2024-06-04T04:54:19.511906+0000","last_clean":"2024-06-04T04:54:19.511906+0000","last_became_active":"2024-06-04T04:53:36.370852+0000","last_became_peered":"2024-06-04T04:53:36.370852+0000","last_unstale":"2024-06-04T04:54:19.511906+0000","last_undegraded":"2024-06-04T04:54:19.511906+0000","last_fullsized":"2024-06-04T04:54:19.511906+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:44:06.956210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508796+0000","last_change":"2024-06-04T04:53:33.349982+0000","last_active":"2024-06-04T04:54:19.508796+0000","last_peered":"2024-06-04T04:54:19.508796+0000","last_clean":"2024-06-04T04:54:19.508796+0000","last_became_active":"2024-06-04T04:53:33.349633+0000","last_became_peered":"2024-06-04T04:53:33.349633+0000","last_unstale":"2024-06-04T04:54:19.508796+0000","last_undegraded":"2024-06-04T04:54:19.508796+0000","last_fullsized":"2024-06-04T04:54:19.508796+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:40.107426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509399+0000","last_change":"2024-06-04T04:53:36.382313+0000","last_active":"2024-06-04T04:54:19.509399+0000","last_peered":"2024-06-04T04:54:19.509399+0000","last_clean":"2024-06-04T04:54:19.509399+0000","last_became_active":"2024-06-04T04:53:36.379638+0000","last_became_peered":"2024-06-04T04:53:36.379638+0000","last_unstale":"2024-06-04T04:54:19.509399+0000","last_undegraded":"2024-06-04T04:54:19.509399+0000","last_fullsized":"2024-06-04T04:54:19.509399+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:43:42.527262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.513581+0000","last_change":"2024-06-04T04:54:19.513581+0000","last_active":"2024-06-04T04:54:19.513581+0000","last_peered":"2024-06-04T04:54:19.513581+0000","last_clean":"2024-06-04T04:54:19.513581+0000","last_became_active":"2024-06-04T04:54:19.513111+0000","last_became_peered":"2024-06-04T04:54:19.513111+0000","last_unstale":"2024-06-04T04:54:19.513581+0000","last_undegraded":"2024-06-04T04:54:19.513581+0000","last_fullsized":"2024-06-04T04:54:19.513581+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:57.028083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508983+0000","last_change":"2024-06-04T04:53:36.369904+0000","last_active":"2024-06-04T04:54:19.508983+0000","last_peered":"2024-06-04T04:54:19.508983+0000","last_clean":"2024-06-04T04:54:19.508983+0000","last_became_active":"2024-06-04T04:53:36.367929+0000","last_became_peered":"2024-06-04T04:53:36.367929+0000","last_unstale":"2024-06-04T04:54:19.508983+0000","last_undegraded":"2024-06-04T04:54:19.508983+0000","last_fullsized":"2024-06-04T04:54:19.508983+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:37:08.721569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'6","reported_seq":69,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:27.644667+0000","last_change":"2024-06-04T04:53:33.342723+0000","last_active":"2024-06-04T04:54:27.644667+0000","last_peered":"2024-06-04T04:54:27.644667+0000","last_clean":"2024-06-04T04:54:27.644667+0000","last_became_active":"2024-06-04T04:53:33.342344+0000","last_became_peered":"2024-06-04T04:53:33.342344+0000","last_unstale":"2024-06-04T04:54:27.644667+0000","last_undegraded":"2024-06-04T04:54:27.644667+0000","last_fullsized":"2024-06-04T04:54:27.644667+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:59:50.147517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.513534+0000","last_change":"2024-06-04T04:54:19.513534+0000","last_active":"2024-06-04T04:54:19.513534+0000","last_peered":"2024-06-04T04:54:19.513534+0000","last_clean":"2024-06-04T04:54:19.513534+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:54:19.513534+0000","last_undegraded":"2024-06-04T04:54:19.513534+0000","last_fullsized":"2024-06-04T04:54:19.513534+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:16:32.240932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512091+0000","last_change":"2024-06-04T04:53:36.371236+0000","last_active":"2024-06-04T04:54:19.512091+0000","last_peered":"2024-06-04T04:54:19.512091+0000","last_clean":"2024-06-04T04:54:19.512091+0000","last_became_active":"2024-06-04T04:53:36.370876+0000","last_became_peered":"2024-06-04T04:53:36.370876+0000","last_unstale":"2024-06-04T04:54:19.512091+0000","last_undegraded":"2024-06-04T04:54:19.512091+0000","last_fullsized":"2024-06-04T04:54:19.512091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:15:56.751317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":63,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:24.495996+0000","last_change":"2024-06-04T04:53:33.355642+0000","last_active":"2024-06-04T04:54:24.495996+0000","last_peered":"2024-06-04T04:54:24.495996+0000","last_clean":"2024-06-04T04:54:24.495996+0000","last_became_active":"2024-06-04T04:53:33.355187+0000","last_became_peered":"2024-06-04T04:53:33.355187+0000","last_unstale":"2024-06-04T04:54:24.495996+0000","last_undegraded":"2024-06-04T04:54:24.495996+0000","last_fullsized":"2024-06-04T04:54:24.495996+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:32:28.331518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.632155+0000","last_change":"2024-06-04T04:54:19.631750+0000","last_active":"2024-06-04T04:54:19.632155+0000","last_peered":"2024-06-04T04:54:19.632155+0000","last_clean":"2024-06-04T04:54:19.632155+0000","last_became_active":"2024-06-04T04:54:19.631485+0000","last_became_peered":"2024-06-04T04:54:19.631485+0000","last_unstale":"2024-06-04T04:54:19.632155+0000","last_undegraded":"2024-06-04T04:54:19.632155+0000","last_fullsized":"2024-06-04T04:54:19.632155+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:29:44.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631469+0000","last_change":"2024-06-04T04:53:33.340653+0000","last_active":"2024-06-04T04:54:19.631469+0000","last_peered":"2024-06-04T04:54:19.631469+0000","last_clean":"2024-06-04T04:54:19.631469+0000","last_became_active":"2024-06-04T04:53:33.340250+0000","last_became_peered":"2024-06-04T04:53:33.340250+0000","last_unstale":"2024-06-04T04:54:19.631469+0000","last_undegraded":"2024-06-04T04:54:19.631469+0000","last_fullsized":"2024-06-04T04:54:19.631469+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:46:20.329514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":99,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510637+0000","last_change":"2024-06-04T04:52:09.181155+0000","last_active":"2024-06-04T04:54:19.510637+0000","last_peered":"2024-06-04T04:54:19.510637+0000","last_clean":"2024-06-04T04:54:19.510637+0000","last_became_active":"2024-06-04T04:52:09.177002+0000","last_became_peered":"2024-06-04T04:52:09.177002+0000","last_unstale":"2024-06-04T04:54:19.510637+0000","last_undegraded":"2024-06-04T04:54:19.510637+0000","last_fullsized":"2024-06-04T04:54:19.510637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_clean_scrub_stamp":"2024-06-04T04:48:44.631948+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:46:57.735302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505220+0000","last_change":"2024-06-04T04:53:36.372634+0000","last_active":"2024-06-04T04:54:19.505220+0000","last_peered":"2024-06-04T04:54:19.505220+0000","last_clean":"2024-06-04T04:54:19.505220+0000","last_became_active":"2024-06-04T04:53:36.372290+0000","last_became_peered":"2024-06-04T04:53:36.372290+0000","last_unstale":"2024-06-04T04:54:19.505220+0000","last_undegraded":"2024-06-04T04:54:19.505220+0000","last_fullsized":"2024-06-04T04:54:19.505220+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:14:07.460382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511010+0000","last_change":"2024-06-04T04:53:33.348760+0000","last_active":"2024-06-04T04:54:19.511010+0000","last_peered":"2024-06-04T04:54:19.511010+0000","last_clean":"2024-06-04T04:54:19.511010+0000","last_became_active":"2024-06-04T04:53:33.348347+0000","last_became_peered":"2024-06-04T04:53:33.348347+0000","last_unstale":"2024-06-04T04:54:19.511010+0000","last_undegraded":"2024-06-04T04:54:19.511010+0000","last_fullsized":"2024-06-04T04:54:19.511010+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:36:20.641061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509558+0000","last_change":"2024-06-04T04:53:33.354840+0000","last_active":"2024-06-04T04:54:19.509558+0000","last_peered":"2024-06-04T04:54:19.509558+0000","last_clean":"2024-06-04T04:54:19.509558+0000","last_became_active":"2024-06-04T04:53:33.354287+0000","last_became_peered":"2024-06-04T04:53:33.354287+0000","last_unstale":"2024-06-04T04:54:19.509558+0000","last_undegraded":"2024-06-04T04:54:19.509558+0000","last_fullsized":"2024-06-04T04:54:19.509558+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:42.550098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505559+0000","last_change":"2024-06-04T04:53:36.372678+0000","last_active":"2024-06-04T04:54:19.505559+0000","last_peered":"2024-06-04T04:54:19.505559+0000","last_clean":"2024-06-04T04:54:19.505559+0000","last_became_active":"2024-06-04T04:53:36.372312+0000","last_became_peered":"2024-06-04T04:53:36.372312+0000","last_unstale":"2024-06-04T04:54:19.505559+0000","last_undegraded":"2024-06-04T04:54:19.505559+0000","last_fullsized":"2024-06-04T04:54:19.505559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:34:41.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"79'14","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508647+0000","last_change":"2024-06-04T04:53:33.348440+0000","last_active":"2024-06-04T04:54:19.508647+0000","last_peered":"2024-06-04T04:54:19.508647+0000","last_clean":"2024-06-04T04:54:19.508647+0000","last_became_active":"2024-06-04T04:53:33.348131+0000","last_became_peered":"2024-06-04T04:53:33.348131+0000","last_unstale":"2024-06-04T04:54:19.508647+0000","last_undegraded":"2024-06-04T04:54:19.508647+0000","last_fullsized":"2024-06-04T04:54:19.508647+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:14:11.552286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.513045+0000","last_change":"2024-06-04T04:53:36.378669+0000","last_active":"2024-06-04T04:54:19.513045+0000","last_peered":"2024-06-04T04:54:19.513045+0000","last_clean":"2024-06-04T04:54:19.513045+0000","last_became_active":"2024-06-04T04:53:36.378233+0000","last_became_peered":"2024-06-04T04:53:36.378233+0000","last_unstale":"2024-06-04T04:54:19.513045+0000","last_undegraded":"2024-06-04T04:54:19.513045+0000","last_fullsized":"2024-06-04T04:54:19.513045+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:36:45.870924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631647+0000","last_change":"2024-06-04T04:53:33.345078+0000","last_active":"2024-06-04T04:54:19.631647+0000","last_peered":"2024-06-04T04:54:19.631647+0000","last_clean":"2024-06-04T04:54:19.631647+0000","last_became_active":"2024-06-04T04:53:33.344708+0000","last_became_peered":"2024-06-04T04:53:33.344708+0000","last_unstale":"2024-06-04T04:54:19.631647+0000","last_undegraded":"2024-06-04T04:54:19.631647+0000","last_fullsized":"2024-06-04T04:54:19.631647+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:45:14.010758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510104+0000","last_change":"2024-06-04T04:53:36.364817+0000","last_active":"2024-06-04T04:54:19.510104+0000","last_peered":"2024-06-04T04:54:19.510104+0000","last_clean":"2024-06-04T04:54:19.510104+0000","last_became_active":"2024-06-04T04:53:36.364499+0000","last_became_peered":"2024-06-04T04:53:36.364499+0000","last_unstale":"2024-06-04T04:54:19.510104+0000","last_undegraded":"2024-06-04T04:54:19.510104+0000","last_fullsized":"2024-06-04T04:54:19.510104+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:39:30.518028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.630871+0000","last_change":"2024-06-04T04:53:33.354527+0000","last_active":"2024-06-04T04:54:19.630871+0000","last_peered":"2024-06-04T04:54:19.630871+0000","last_clean":"2024-06-04T04:54:19.630871+0000","last_became_active":"2024-06-04T04:53:33.354175+0000","last_became_peered":"2024-06-04T04:53:33.354175+0000","last_unstale":"2024-06-04T04:54:19.630871+0000","last_undegraded":"2024-06-04T04:54:19.630871+0000","last_fullsized":"2024-06-04T04:54:19.630871+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:33:08.125428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508458+0000","last_change":"2024-06-04T04:53:36.373248+0000","last_active":"2024-06-04T04:54:19.508458+0000","last_peered":"2024-06-04T04:54:19.508458+0000","last_clean":"2024-06-04T04:54:19.508458+0000","last_became_active":"2024-06-04T04:53:36.372915+0000","last_became_peered":"2024-06-04T04:53:36.372915+0000","last_unstale":"2024-06-04T04:54:19.508458+0000","last_undegraded":"2024-06-04T04:54:19.508458+0000","last_fullsized":"2024-06-04T04:54:19.508458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:23:11.689349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512413+0000","last_change":"2024-06-04T04:53:33.340427+0000","last_active":"2024-06-04T04:54:19.512413+0000","last_peered":"2024-06-04T04:54:19.512413+0000","last_clean":"2024-06-04T04:54:19.512413+0000","last_became_active":"2024-06-04T04:53:33.340022+0000","last_became_peered":"2024-06-04T04:53:33.340022+0000","last_unstale":"2024-06-04T04:54:19.512413+0000","last_undegraded":"2024-06-04T04:54:19.512413+0000","last_fullsized":"2024-06-04T04:54:19.512413+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:56:16.496471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509628+0000","last_change":"2024-06-04T04:53:36.380114+0000","last_active":"2024-06-04T04:54:19.509628+0000","last_peered":"2024-06-04T04:54:19.509628+0000","last_clean":"2024-06-04T04:54:19.509628+0000","last_became_active":"2024-06-04T04:53:36.379392+0000","last_became_peered":"2024-06-04T04:53:36.379392+0000","last_unstale":"2024-06-04T04:54:19.509628+0000","last_undegraded":"2024-06-04T04:54:19.509628+0000","last_fullsized":"2024-06-04T04:54:19.509628+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:41:48.776439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.520195+0000","last_change":"2024-06-04T04:54:19.519755+0000","last_active":"2024-06-04T04:54:19.520195+0000","last_peered":"2024-06-04T04:54:19.520195+0000","last_clean":"2024-06-04T04:54:19.520195+0000","last_became_active":"2024-06-04T04:54:19.513992+0000","last_became_peered":"2024-06-04T04:54:19.513992+0000","last_unstale":"2024-06-04T04:54:19.520195+0000","last_undegraded":"2024-06-04T04:54:19.520195+0000","last_fullsized":"2024-06-04T04:54:19.520195+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:01.951935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512566+0000","last_change":"2024-06-04T04:53:36.377420+0000","last_active":"2024-06-04T04:54:19.512566+0000","last_peered":"2024-06-04T04:54:19.512566+0000","last_clean":"2024-06-04T04:54:19.512566+0000","last_became_active":"2024-06-04T04:53:36.377109+0000","last_became_peered":"2024-06-04T04:53:36.377109+0000","last_unstale":"2024-06-04T04:54:19.512566+0000","last_undegraded":"2024-06-04T04:54:19.512566+0000","last_fullsized":"2024-06-04T04:54:19.512566+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:47:12.102396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":43,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.645328+0000","last_change":"2024-06-04T04:54:19.645328+0000","last_active":"2024-06-04T04:54:19.645328+0000","last_peered":"2024-06-04T04:54:19.645328+0000","last_clean":"2024-06-04T04:54:19.645328+0000","last_became_active":"2024-06-04T04:54:19.633416+0000","last_became_peered":"2024-06-04T04:54:19.633416+0000","last_unstale":"2024-06-04T04:54:19.645328+0000","last_undegraded":"2024-06-04T04:54:19.645328+0000","last_fullsized":"2024-06-04T04:54:19.645328+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:04:20.494641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510562+0000","last_change":"2024-06-04T04:53:36.381187+0000","last_active":"2024-06-04T04:54:19.510562+0000","last_peered":"2024-06-04T04:54:19.510562+0000","last_clean":"2024-06-04T04:54:19.510562+0000","last_became_active":"2024-06-04T04:53:36.380591+0000","last_became_peered":"2024-06-04T04:53:36.380591+0000","last_unstale":"2024-06-04T04:54:19.510562+0000","last_undegraded":"2024-06-04T04:54:19.510562+0000","last_fullsized":"2024-06-04T04:54:19.510562+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:26:43.122339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'5","reported_seq":67,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:27.501962+0000","last_change":"2024-06-04T04:53:33.355908+0000","last_active":"2024-06-04T04:54:27.501962+0000","last_peered":"2024-06-04T04:54:27.501962+0000","last_clean":"2024-06-04T04:54:27.501962+0000","last_became_active":"2024-06-04T04:53:33.355522+0000","last_became_peered":"2024-06-04T04:53:33.355522+0000","last_unstale":"2024-06-04T04:54:27.501962+0000","last_undegraded":"2024-06-04T04:54:27.501962+0000","last_fullsized":"2024-06-04T04:54:27.501962+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:16:44.639431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510919+0000","last_change":"2024-06-04T04:53:36.382737+0000","last_active":"2024-06-04T04:54:19.510919+0000","last_peered":"2024-06-04T04:54:19.510919+0000","last_clean":"2024-06-04T04:54:19.510919+0000","last_became_active":"2024-06-04T04:53:36.382353+0000","last_became_peered":"2024-06-04T04:53:36.382353+0000","last_unstale":"2024-06-04T04:54:19.510919+0000","last_undegraded":"2024-06-04T04:54:19.510919+0000","last_fullsized":"2024-06-04T04:54:19.510919+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:18:01.476058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512086+0000","last_change":"2024-06-04T04:53:33.347273+0000","last_active":"2024-06-04T04:54:19.512086+0000","last_peered":"2024-06-04T04:54:19.512086+0000","last_clean":"2024-06-04T04:54:19.512086+0000","last_became_active":"2024-06-04T04:53:33.346915+0000","last_became_peered":"2024-06-04T04:53:33.346915+0000","last_unstale":"2024-06-04T04:54:19.512086+0000","last_undegraded":"2024-06-04T04:54:19.512086+0000","last_fullsized":"2024-06-04T04:54:19.512086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:17.235484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"79'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510494+0000","last_change":"2024-06-04T04:53:36.383454+0000","last_active":"2024-06-04T04:54:19.510494+0000","last_peered":"2024-06-04T04:54:19.510494+0000","last_clean":"2024-06-04T04:54:19.510494+0000","last_became_active":"2024-06-04T04:53:36.383105+0000","last_became_peered":"2024-06-04T04:53:36.383105+0000","last_unstale":"2024-06-04T04:54:19.510494+0000","last_undegraded":"2024-06-04T04:54:19.510494+0000","last_fullsized":"2024-06-04T04:54:19.510494+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:04:02.147688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505624+0000","last_change":"2024-06-04T04:53:33.345435+0000","last_active":"2024-06-04T04:54:19.505624+0000","last_peered":"2024-06-04T04:54:19.505624+0000","last_clean":"2024-06-04T04:54:19.505624+0000","last_became_active":"2024-06-04T04:53:33.345077+0000","last_became_peered":"2024-06-04T04:53:33.345077+0000","last_unstale":"2024-06-04T04:54:19.505624+0000","last_undegraded":"2024-06-04T04:54:19.505624+0000","last_fullsized":"2024-06-04T04:54:19.505624+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:20:13.992259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510565+0000","last_change":"2024-06-04T04:53:36.385238+0000","last_active":"2024-06-04T04:54:19.510565+0000","last_peered":"2024-06-04T04:54:19.510565+0000","last_clean":"2024-06-04T04:54:19.510565+0000","last_became_active":"2024-06-04T04:53:36.384899+0000","last_became_peered":"2024-06-04T04:53:36.384899+0000","last_unstale":"2024-06-04T04:54:19.510565+0000","last_undegraded":"2024-06-04T04:54:19.510565+0000","last_fullsized":"2024-06-04T04:54:19.510565+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:09:42.629340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509793+0000","last_change":"2024-06-04T04:53:36.383450+0000","last_active":"2024-06-04T04:54:19.509793+0000","last_peered":"2024-06-04T04:54:19.509793+0000","last_clean":"2024-06-04T04:54:19.509793+0000","last_became_active":"2024-06-04T04:53:36.382678+0000","last_became_peered":"2024-06-04T04:53:36.382678+0000","last_unstale":"2024-06-04T04:54:19.509793+0000","last_undegraded":"2024-06-04T04:54:19.509793+0000","last_fullsized":"2024-06-04T04:54:19.509793+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:36:28.932636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.632563+0000","last_change":"2024-06-04T04:54:19.632563+0000","last_active":"2024-06-04T04:54:19.632563+0000","last_peered":"2024-06-04T04:54:19.632563+0000","last_clean":"2024-06-04T04:54:19.632563+0000","last_became_active":"2024-06-04T04:54:19.632070+0000","last_became_peered":"2024-06-04T04:54:19.632070+0000","last_unstale":"2024-06-04T04:54:19.632563+0000","last_undegraded":"2024-06-04T04:54:19.632563+0000","last_fullsized":"2024-06-04T04:54:19.632563+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:07:16.552958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.506413+0000","last_change":"2024-06-04T04:53:36.381408+0000","last_active":"2024-06-04T04:54:19.506413+0000","last_peered":"2024-06-04T04:54:19.506413+0000","last_clean":"2024-06-04T04:54:19.506413+0000","last_became_active":"2024-06-04T04:53:36.381078+0000","last_became_peered":"2024-06-04T04:53:36.381078+0000","last_unstale":"2024-06-04T04:54:19.506413+0000","last_undegraded":"2024-06-04T04:54:19.506413+0000","last_fullsized":"2024-06-04T04:54:19.506413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:28:48.988591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510556+0000","last_change":"2024-06-04T04:53:36.360168+0000","last_active":"2024-06-04T04:54:19.510556+0000","last_peered":"2024-06-04T04:54:19.510556+0000","last_clean":"2024-06-04T04:54:19.510556+0000","last_became_active":"2024-06-04T04:53:36.359691+0000","last_became_peered":"2024-06-04T04:53:36.359691+0000","last_unstale":"2024-06-04T04:54:19.510556+0000","last_undegraded":"2024-06-04T04:54:19.510556+0000","last_fullsized":"2024-06-04T04:54:19.510556+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:55:17.418339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505605+0000","last_change":"2024-06-04T04:53:36.363018+0000","last_active":"2024-06-04T04:54:19.505605+0000","last_peered":"2024-06-04T04:54:19.505605+0000","last_clean":"2024-06-04T04:54:19.505605+0000","last_became_active":"2024-06-04T04:53:36.361595+0000","last_became_peered":"2024-06-04T04:53:36.361595+0000","last_unstale":"2024-06-04T04:54:19.505605+0000","last_undegraded":"2024-06-04T04:54:19.505605+0000","last_fullsized":"2024-06-04T04:54:19.505605+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:00:21.377322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509182+0000","last_change":"2024-06-04T04:53:36.375377+0000","last_active":"2024-06-04T04:54:19.509182+0000","last_peered":"2024-06-04T04:54:19.509182+0000","last_clean":"2024-06-04T04:54:19.509182+0000","last_became_active":"2024-06-04T04:53:36.373459+0000","last_became_peered":"2024-06-04T04:53:36.373459+0000","last_unstale":"2024-06-04T04:54:19.509182+0000","last_undegraded":"2024-06-04T04:54:19.509182+0000","last_fullsized":"2024-06-04T04:54:19.509182+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:51:31.686969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.506738+0000","last_change":"2024-06-04T04:53:36.373850+0000","last_active":"2024-06-04T04:54:19.506738+0000","last_peered":"2024-06-04T04:54:19.506738+0000","last_clean":"2024-06-04T04:54:19.506738+0000","last_became_active":"2024-06-04T04:53:36.371951+0000","last_became_peered":"2024-06-04T04:53:36.371951+0000","last_unstale":"2024-06-04T04:54:19.506738+0000","last_undegraded":"2024-06-04T04:54:19.506738+0000","last_fullsized":"2024-06-04T04:54:19.506738+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:06.452703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512871+0000","last_change":"2024-06-04T04:53:36.374871+0000","last_active":"2024-06-04T04:54:19.512871+0000","last_peered":"2024-06-04T04:54:19.512871+0000","last_clean":"2024-06-04T04:54:19.512871+0000","last_became_active":"2024-06-04T04:53:36.374486+0000","last_became_peered":"2024-06-04T04:53:36.374486+0000","last_unstale":"2024-06-04T04:54:19.512871+0000","last_undegraded":"2024-06-04T04:54:19.512871+0000","last_fullsized":"2024-06-04T04:54:19.512871+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:15:13.337374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.514431+0000","last_change":"2024-06-04T04:54:19.513540+0000","last_active":"2024-06-04T04:54:19.514431+0000","last_peered":"2024-06-04T04:54:19.514431+0000","last_clean":"2024-06-04T04:54:19.514431+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:54:19.514431+0000","last_undegraded":"2024-06-04T04:54:19.514431+0000","last_fullsized":"2024-06-04T04:54:19.514431+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:09:27.993585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512321+0000","last_change":"2024-06-04T04:53:36.373055+0000","last_active":"2024-06-04T04:54:19.512321+0000","last_peered":"2024-06-04T04:54:19.512321+0000","last_clean":"2024-06-04T04:54:19.512321+0000","last_became_active":"2024-06-04T04:53:36.372754+0000","last_became_peered":"2024-06-04T04:53:36.372754+0000","last_unstale":"2024-06-04T04:54:19.512321+0000","last_undegraded":"2024-06-04T04:54:19.512321+0000","last_fullsized":"2024-06-04T04:54:19.512321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:22:27.554200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631332+0000","last_change":"2024-06-04T04:53:36.378897+0000","last_active":"2024-06-04T04:54:19.631332+0000","last_peered":"2024-06-04T04:54:19.631332+0000","last_clean":"2024-06-04T04:54:19.631332+0000","last_became_active":"2024-06-04T04:53:36.378607+0000","last_became_peered":"2024-06-04T04:53:36.378607+0000","last_unstale":"2024-06-04T04:54:19.631332+0000","last_undegraded":"2024-06-04T04:54:19.631332+0000","last_fullsized":"2024-06-04T04:54:19.631332+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:03.395491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505499+0000","last_change":"2024-06-04T04:53:36.378240+0000","last_active":"2024-06-04T04:54:19.505499+0000","last_peered":"2024-06-04T04:54:19.505499+0000","last_clean":"2024-06-04T04:54:19.505499+0000","last_became_active":"2024-06-04T04:53:36.377909+0000","last_became_peered":"2024-06-04T04:53:36.377909+0000","last_unstale":"2024-06-04T04:54:19.505499+0000","last_undegraded":"2024-06-04T04:54:19.505499+0000","last_fullsized":"2024-06-04T04:54:19.505499+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:15:25.288340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510525+0000","last_change":"2024-06-04T04:53:36.379792+0000","last_active":"2024-06-04T04:54:19.510525+0000","last_peered":"2024-06-04T04:54:19.510525+0000","last_clean":"2024-06-04T04:54:19.510525+0000","last_became_active":"2024-06-04T04:53:36.379438+0000","last_became_peered":"2024-06-04T04:53:36.379438+0000","last_unstale":"2024-06-04T04:54:19.510525+0000","last_undegraded":"2024-06-04T04:54:19.510525+0000","last_fullsized":"2024-06-04T04:54:19.510525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:23:57.617877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":214930,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71878,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":377,"num_read_kb":260,"num_write":244,"num_write_kb":140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":216772,"data_compressed":5499,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678041,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":880640,"data_stored":429560,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874270,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":1085654,"data_compressed":43684,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037796,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":905216,"data_stored":429490,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234025,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":892928,"data_stored":429196,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430253,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":1022386,"data_compressed":41851,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593779,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":888832,"data_stored":429598,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":496248,"data_compressed":7592,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":909312,"data_stored":430546,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":496543,"data_compressed":7592,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36228,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708924,"statfs":{"total":95995035648,"available":95957938176,"internally_reserved":0,"allocated":905216,"data_stored":429032,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953534,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":888832,"data_stored":429068,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790009,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":1021673,"data_compressed":41851,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34470,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":67880,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":68175,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":66801,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-04T04:54:32.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:32 smithi070 ceph-mon[32631]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:54:32.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:32 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:54:32.876 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:32 smithi112 ceph-mon[36453]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:54:32.876 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:32 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:54:32.893 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:32 smithi098 ceph-mon[35569]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:54:32.893 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:32 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:54:33.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:33 smithi070 ceph-mon[32631]: pgmap v318: 97 pgs: 97 active+clean; 647 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.2 KiB/s wr, 8 op/s; 55 B/s, 0 objects/s recovering 2024-06-04T04:54:33.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:33 smithi112 ceph-mon[36453]: pgmap v318: 97 pgs: 97 active+clean; 647 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.2 KiB/s wr, 8 op/s; 55 B/s, 0 objects/s recovering 2024-06-04T04:54:33.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:33 smithi098 ceph-mon[35569]: pgmap v318: 97 pgs: 97 active+clean; 647 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.2 KiB/s wr, 8 op/s; 55 B/s, 0 objects/s recovering 2024-06-04T04:54:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:34 smithi070 ceph-mon[32631]: pgmap v319: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.4 KiB/s wr, 10 op/s; 55 B/s, 0 objects/s recovering 2024-06-04T04:54:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:54:35.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:54:35.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.118 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-04T04:54:35.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:34 smithi112 ceph-mon[36453]: pgmap v319: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.4 KiB/s wr, 10 op/s; 55 B/s, 0 objects/s recovering 2024-06-04T04:54:35.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:54:35.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:54:35.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:34 smithi098 ceph-mon[35569]: pgmap v319: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.4 KiB/s wr, 10 op/s; 55 B/s, 0 objects/s recovering 2024-06-04T04:54:35.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:54:35.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:54:35.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:54:35.359 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:37.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:36 smithi070 ceph-mon[32631]: pgmap v320: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-06-04T04:54:37.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.116+0000 7f326cffc700 1 -- 172.21.15.70:0/2689061617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3268102e30 msgr2=0x7f326810f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:37.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.116+0000 7f326cffc700 1 --2- 172.21.15.70:0/2689061617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3268102e30 0x7f326810f2f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f326000b0a0 tx=0x7f326000b3b0 comp rx=0 tx=0).stop 2024-06-04T04:54:37.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.116+0000 7f326cffc700 1 -- 172.21.15.70:0/2689061617 shutdown_connections 2024-06-04T04:54:37.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.116+0000 7f326cffc700 1 --2- 172.21.15.70:0/2689061617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3268102e30 0x7f326810f2f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:37.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.116+0000 7f326cffc700 1 --2- 172.21.15.70:0/2689061617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f32681084b0 0x7f3268108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:37.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.116+0000 7f326cffc700 1 --2- 172.21.15.70:0/2689061617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3268102470 0x7f32681028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:37.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.116+0000 7f326cffc700 1 -- 172.21.15.70:0/2689061617 >> 172.21.15.70:0/2689061617 conn(0x7f32680793f0 msgr2=0x7f3268079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:37.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.117+0000 7f326cffc700 1 -- 172.21.15.70:0/2689061617 shutdown_connections 2024-06-04T04:54:37.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.117+0000 7f326cffc700 1 -- 172.21.15.70:0/2689061617 wait complete. 2024-06-04T04:54:37.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.117+0000 7f326cffc700 1 Processor -- start 2024-06-04T04:54:37.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.117+0000 7f326cffc700 1 -- start start 2024-06-04T04:54:37.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.117+0000 7f326cffc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3268102470 0x7f3268104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:37.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.117+0000 7f326cffc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3268102e30 0x7f32681045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:37.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.117+0000 7f326cffc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32681084b0 0x7f3268104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:37.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.117+0000 7f326cffc700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3268112490 con 0x7f3268102470 2024-06-04T04:54:37.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.117+0000 7f326cffc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3268112310 con 0x7f32681084b0 2024-06-04T04:54:37.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.117+0000 7f326cffc700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3268112610 con 0x7f3268102e30 2024-06-04T04:54:37.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.118+0000 7f326659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3268102470 0x7f3268104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:37.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.118+0000 7f326659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3268102470 0x7f3268104080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47256/0 (socket says 172.21.15.70:47256) 2024-06-04T04:54:37.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.118+0000 7f326659c700 1 -- 172.21.15.70:0/437922377 learned_addr learned my addr 172.21.15.70:0/437922377 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:37.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.118+0000 7f3265d9b700 1 --2- 172.21.15.70:0/437922377 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3268102e30 0x7f32681045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:37.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.118+0000 7f3266d9d700 1 --2- 172.21.15.70:0/437922377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32681084b0 0x7f3268104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:37.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.118+0000 7f326659c700 1 -- 172.21.15.70:0/437922377 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3268102e30 msgr2=0x7f32681045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:37.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.118+0000 7f326659c700 1 --2- 172.21.15.70:0/437922377 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3268102e30 0x7f32681045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:37.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.118+0000 7f326659c700 1 -- 172.21.15.70:0/437922377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32681084b0 msgr2=0x7f3268104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:37.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.118+0000 7f326659c700 1 --2- 172.21.15.70:0/437922377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32681084b0 0x7f3268104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:37.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.118+0000 7f326659c700 1 -- 172.21.15.70:0/437922377 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32681a8080 con 0x7f3268102470 2024-06-04T04:54:37.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.118+0000 7f326659c700 1 --2- 172.21.15.70:0/437922377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3268102470 0x7f3268104080 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f325800d980 tx=0x7f325800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:37.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.118+0000 7f32577fe700 1 -- 172.21.15.70:0/437922377 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258017070 con 0x7f3268102470 2024-06-04T04:54:37.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.118+0000 7f32577fe700 1 -- 172.21.15.70:0/437922377 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3258005020 con 0x7f3268102470 2024-06-04T04:54:37.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.118+0000 7f32577fe700 1 -- 172.21.15.70:0/437922377 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258009ab0 con 0x7f3268102470 2024-06-04T04:54:37.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.118+0000 7f326cffc700 1 -- 172.21.15.70:0/437922377 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32681a8370 con 0x7f3268102470 2024-06-04T04:54:37.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.118+0000 7f326cffc700 1 -- 172.21.15.70:0/437922377 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32681a88b0 con 0x7f3268102470 2024-06-04T04:54:37.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.119+0000 7f326cffc700 1 -- 172.21.15.70:0/437922377 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f326806a230 con 0x7f3268102470 2024-06-04T04:54:37.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.121+0000 7f32577fe700 1 -- 172.21.15.70:0/437922377 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f32580051d0 con 0x7f3268102470 2024-06-04T04:54:37.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.121+0000 7f32577fe700 1 --2- 172.21.15.70:0/437922377 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f32500b2990 0x7f32500b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:37.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.121+0000 7f3265d9b700 1 --2- 172.21.15.70:0/437922377 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f32500b2990 0x7f32500b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:37.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.121+0000 7f32577fe700 1 -- 172.21.15.70:0/437922377 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f32580e2600 con 0x7f3268102470 2024-06-04T04:54:37.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.122+0000 7f3265d9b700 1 --2- 172.21.15.70:0/437922377 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f32500b2990 0x7f32500b4e50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3268079b80 tx=0x7f325c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:37.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.125+0000 7f32577fe700 1 -- 172.21.15.70:0/437922377 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3258020020 con 0x7f3268102470 2024-06-04T04:54:37.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:36 smithi112 ceph-mon[36453]: pgmap v320: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-06-04T04:54:37.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:36 smithi098 ceph-mon[35569]: pgmap v320: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-06-04T04:54:37.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.324+0000 7f326cffc700 1 -- 172.21.15.70:0/437922377 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3268063980 con 0x7f32500b2990 2024-06-04T04:54:37.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.331+0000 7f32577fe700 1 -- 172.21.15.70:0/437922377 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259000 (secure 0 0 0) 0x7f3268063980 con 0x7f32500b2990 2024-06-04T04:54:37.337 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:54:37.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.334+0000 7f326cffc700 1 -- 172.21.15.70:0/437922377 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f32500b2990 msgr2=0x7f32500b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:37.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.334+0000 7f326cffc700 1 --2- 172.21.15.70:0/437922377 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f32500b2990 0x7f32500b4e50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3268079b80 tx=0x7f325c00b410 comp rx=0 tx=0).stop 2024-06-04T04:54:37.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.334+0000 7f326cffc700 1 -- 172.21.15.70:0/437922377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3268102470 msgr2=0x7f3268104080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:37.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.334+0000 7f326cffc700 1 --2- 172.21.15.70:0/437922377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3268102470 0x7f3268104080 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f325800d980 tx=0x7f325800dc90 comp rx=0 tx=0).stop 2024-06-04T04:54:37.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.334+0000 7f326cffc700 1 -- 172.21.15.70:0/437922377 shutdown_connections 2024-06-04T04:54:37.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.334+0000 7f326cffc700 1 --2- 172.21.15.70:0/437922377 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f32500b2990 0x7f32500b4e50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:37.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.334+0000 7f326cffc700 1 --2- 172.21.15.70:0/437922377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32681084b0 0x7f3268104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:37.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.334+0000 7f326cffc700 1 --2- 172.21.15.70:0/437922377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3268102470 0x7f3268104080 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:37.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.334+0000 7f326cffc700 1 --2- 172.21.15.70:0/437922377 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3268102e30 0x7f32681045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:37.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.334+0000 7f326cffc700 1 -- 172.21.15.70:0/437922377 >> 172.21.15.70:0/437922377 conn(0x7f32680793f0 msgr2=0x7f326806c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:37.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.335+0000 7f326cffc700 1 -- 172.21.15.70:0/437922377 shutdown_connections 2024-06-04T04:54:37.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:37.335+0000 7f326cffc700 1 -- 172.21.15.70:0/437922377 wait complete. 2024-06-04T04:54:37.339 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-06-04T04:54:38.032 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:37 smithi070 ceph-mon[32631]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:54:38.057 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-06-04T04:54:35.735404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663618,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":514,"num_read_kb":361,"num_write":370,"num_write_kb":2294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2952512,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":355968,"kb_used_data":10896,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585856,"statfs":{"total":1151940427776,"available":1151575916544,"internally_reserved":0,"allocated":11157504,"data_stored":7131270,"data_compressed":182883,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19138,"internal_metadata":353154366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1138,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":120,"num_read_kb":57,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002961"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509657+0000","last_change":"2024-06-04T04:53:36.383755+0000","last_active":"2024-06-04T04:54:19.509657+0000","last_peered":"2024-06-04T04:54:19.509657+0000","last_clean":"2024-06-04T04:54:19.509657+0000","last_became_active":"2024-06-04T04:53:36.378274+0000","last_became_peered":"2024-06-04T04:53:36.378274+0000","last_unstale":"2024-06-04T04:54:19.509657+0000","last_undegraded":"2024-06-04T04:54:19.509657+0000","last_fullsized":"2024-06-04T04:54:19.509657+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:30:19.685994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510014+0000","last_change":"2024-06-04T04:53:36.372436+0000","last_active":"2024-06-04T04:54:19.510014+0000","last_peered":"2024-06-04T04:54:19.510014+0000","last_clean":"2024-06-04T04:54:19.510014+0000","last_became_active":"2024-06-04T04:53:36.372100+0000","last_became_peered":"2024-06-04T04:53:36.372100+0000","last_unstale":"2024-06-04T04:54:19.510014+0000","last_undegraded":"2024-06-04T04:54:19.510014+0000","last_fullsized":"2024-06-04T04:54:19.510014+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:42:31.646754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510448+0000","last_change":"2024-06-04T04:53:36.366915+0000","last_active":"2024-06-04T04:54:19.510448+0000","last_peered":"2024-06-04T04:54:19.510448+0000","last_clean":"2024-06-04T04:54:19.510448+0000","last_became_active":"2024-06-04T04:53:36.366611+0000","last_became_peered":"2024-06-04T04:53:36.366611+0000","last_unstale":"2024-06-04T04:54:19.510448+0000","last_undegraded":"2024-06-04T04:54:19.510448+0000","last_fullsized":"2024-06-04T04:54:19.510448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:17:48.775938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631647+0000","last_change":"2024-06-04T04:53:36.374486+0000","last_active":"2024-06-04T04:54:19.631647+0000","last_peered":"2024-06-04T04:54:19.631647+0000","last_clean":"2024-06-04T04:54:19.631647+0000","last_became_active":"2024-06-04T04:53:36.374178+0000","last_became_peered":"2024-06-04T04:53:36.374178+0000","last_unstale":"2024-06-04T04:54:19.631647+0000","last_undegraded":"2024-06-04T04:54:19.631647+0000","last_fullsized":"2024-06-04T04:54:19.631647+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:30:28.287070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508460+0000","last_change":"2024-06-04T04:53:36.376183+0000","last_active":"2024-06-04T04:54:19.508460+0000","last_peered":"2024-06-04T04:54:19.508460+0000","last_clean":"2024-06-04T04:54:19.508460+0000","last_became_active":"2024-06-04T04:53:36.375790+0000","last_became_peered":"2024-06-04T04:53:36.375790+0000","last_unstale":"2024-06-04T04:54:19.508460+0000","last_undegraded":"2024-06-04T04:54:19.508460+0000","last_fullsized":"2024-06-04T04:54:19.508460+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:58.697440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509028+0000","last_change":"2024-06-04T04:53:36.370249+0000","last_active":"2024-06-04T04:54:19.509028+0000","last_peered":"2024-06-04T04:54:19.509028+0000","last_clean":"2024-06-04T04:54:19.509028+0000","last_became_active":"2024-06-04T04:53:36.369916+0000","last_became_peered":"2024-06-04T04:53:36.369916+0000","last_unstale":"2024-06-04T04:54:19.509028+0000","last_undegraded":"2024-06-04T04:54:19.509028+0000","last_fullsized":"2024-06-04T04:54:19.509028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:02:41.739806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510237+0000","last_change":"2024-06-04T04:53:36.384329+0000","last_active":"2024-06-04T04:54:19.510237+0000","last_peered":"2024-06-04T04:54:19.510237+0000","last_clean":"2024-06-04T04:54:19.510237+0000","last_became_active":"2024-06-04T04:53:36.382504+0000","last_became_peered":"2024-06-04T04:53:36.382504+0000","last_unstale":"2024-06-04T04:54:19.510237+0000","last_undegraded":"2024-06-04T04:54:19.510237+0000","last_fullsized":"2024-06-04T04:54:19.510237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:51:07.604691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511986+0000","last_change":"2024-06-04T04:53:36.372319+0000","last_active":"2024-06-04T04:54:19.511986+0000","last_peered":"2024-06-04T04:54:19.511986+0000","last_clean":"2024-06-04T04:54:19.511986+0000","last_became_active":"2024-06-04T04:53:36.372000+0000","last_became_peered":"2024-06-04T04:53:36.372000+0000","last_unstale":"2024-06-04T04:54:19.511986+0000","last_undegraded":"2024-06-04T04:54:19.511986+0000","last_fullsized":"2024-06-04T04:54:19.511986+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:15:56.906996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631893+0000","last_change":"2024-06-04T04:53:36.376051+0000","last_active":"2024-06-04T04:54:19.631893+0000","last_peered":"2024-06-04T04:54:19.631893+0000","last_clean":"2024-06-04T04:54:19.631893+0000","last_became_active":"2024-06-04T04:53:36.375726+0000","last_became_peered":"2024-06-04T04:53:36.375726+0000","last_unstale":"2024-06-04T04:54:19.631893+0000","last_undegraded":"2024-06-04T04:54:19.631893+0000","last_fullsized":"2024-06-04T04:54:19.631893+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:09:08.542764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509369+0000","last_change":"2024-06-04T04:53:36.373266+0000","last_active":"2024-06-04T04:54:19.509369+0000","last_peered":"2024-06-04T04:54:19.509369+0000","last_clean":"2024-06-04T04:54:19.509369+0000","last_became_active":"2024-06-04T04:53:36.372898+0000","last_became_peered":"2024-06-04T04:53:36.372898+0000","last_unstale":"2024-06-04T04:54:19.509369+0000","last_undegraded":"2024-06-04T04:54:19.509369+0000","last_fullsized":"2024-06-04T04:54:19.509369+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:41:43.436781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510727+0000","last_change":"2024-06-04T04:53:36.381344+0000","last_active":"2024-06-04T04:54:19.510727+0000","last_peered":"2024-06-04T04:54:19.510727+0000","last_clean":"2024-06-04T04:54:19.510727+0000","last_became_active":"2024-06-04T04:53:36.380736+0000","last_became_peered":"2024-06-04T04:53:36.380736+0000","last_unstale":"2024-06-04T04:54:19.510727+0000","last_undegraded":"2024-06-04T04:54:19.510727+0000","last_fullsized":"2024-06-04T04:54:19.510727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:23:01.607272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631528+0000","last_change":"2024-06-04T04:53:36.374035+0000","last_active":"2024-06-04T04:54:19.631528+0000","last_peered":"2024-06-04T04:54:19.631528+0000","last_clean":"2024-06-04T04:54:19.631528+0000","last_became_active":"2024-06-04T04:53:36.373667+0000","last_became_peered":"2024-06-04T04:53:36.373667+0000","last_unstale":"2024-06-04T04:54:19.631528+0000","last_undegraded":"2024-06-04T04:54:19.631528+0000","last_fullsized":"2024-06-04T04:54:19.631528+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:23:07.880369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505537+0000","last_change":"2024-06-04T04:53:36.369613+0000","last_active":"2024-06-04T04:54:19.505537+0000","last_peered":"2024-06-04T04:54:19.505537+0000","last_clean":"2024-06-04T04:54:19.505537+0000","last_became_active":"2024-06-04T04:53:36.369277+0000","last_became_peered":"2024-06-04T04:53:36.369277+0000","last_unstale":"2024-06-04T04:54:19.505537+0000","last_undegraded":"2024-06-04T04:54:19.505537+0000","last_fullsized":"2024-06-04T04:54:19.505537+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:13:33.756609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510066+0000","last_change":"2024-06-04T04:53:36.368094+0000","last_active":"2024-06-04T04:54:19.510066+0000","last_peered":"2024-06-04T04:54:19.510066+0000","last_clean":"2024-06-04T04:54:19.510066+0000","last_became_active":"2024-06-04T04:53:36.367717+0000","last_became_peered":"2024-06-04T04:53:36.367717+0000","last_unstale":"2024-06-04T04:54:19.510066+0000","last_undegraded":"2024-06-04T04:54:19.510066+0000","last_fullsized":"2024-06-04T04:54:19.510066+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:39:56.305078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509882+0000","last_change":"2024-06-04T04:53:36.366765+0000","last_active":"2024-06-04T04:54:19.509882+0000","last_peered":"2024-06-04T04:54:19.509882+0000","last_clean":"2024-06-04T04:54:19.509882+0000","last_became_active":"2024-06-04T04:53:36.366438+0000","last_became_peered":"2024-06-04T04:53:36.366438+0000","last_unstale":"2024-06-04T04:54:19.509882+0000","last_undegraded":"2024-06-04T04:54:19.509882+0000","last_fullsized":"2024-06-04T04:54:19.509882+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:38:37.313926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631086+0000","last_change":"2024-06-04T04:53:36.380088+0000","last_active":"2024-06-04T04:54:19.631086+0000","last_peered":"2024-06-04T04:54:19.631086+0000","last_clean":"2024-06-04T04:54:19.631086+0000","last_became_active":"2024-06-04T04:53:36.379467+0000","last_became_peered":"2024-06-04T04:53:36.379467+0000","last_unstale":"2024-06-04T04:54:19.631086+0000","last_undegraded":"2024-06-04T04:54:19.631086+0000","last_fullsized":"2024-06-04T04:54:19.631086+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:46:53.236419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509925+0000","last_change":"2024-06-04T04:53:36.381126+0000","last_active":"2024-06-04T04:54:19.509925+0000","last_peered":"2024-06-04T04:54:19.509925+0000","last_clean":"2024-06-04T04:54:19.509925+0000","last_became_active":"2024-06-04T04:53:36.378833+0000","last_became_peered":"2024-06-04T04:53:36.378833+0000","last_unstale":"2024-06-04T04:54:19.509925+0000","last_undegraded":"2024-06-04T04:54:19.509925+0000","last_fullsized":"2024-06-04T04:54:19.509925+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:20:28.639273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508689+0000","last_change":"2024-06-04T04:53:36.369692+0000","last_active":"2024-06-04T04:54:19.508689+0000","last_peered":"2024-06-04T04:54:19.508689+0000","last_clean":"2024-06-04T04:54:19.508689+0000","last_became_active":"2024-06-04T04:53:36.369370+0000","last_became_peered":"2024-06-04T04:53:36.369370+0000","last_unstale":"2024-06-04T04:54:19.508689+0000","last_undegraded":"2024-06-04T04:54:19.508689+0000","last_fullsized":"2024-06-04T04:54:19.508689+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:55.275465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.506778+0000","last_change":"2024-06-04T04:53:36.383639+0000","last_active":"2024-06-04T04:54:19.506778+0000","last_peered":"2024-06-04T04:54:19.506778+0000","last_clean":"2024-06-04T04:54:19.506778+0000","last_became_active":"2024-06-04T04:53:36.383303+0000","last_became_peered":"2024-06-04T04:53:36.383303+0000","last_unstale":"2024-06-04T04:54:19.506778+0000","last_undegraded":"2024-06-04T04:54:19.506778+0000","last_fullsized":"2024-06-04T04:54:19.506778+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:26:31.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":99,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:31.649573+0000","last_change":"2024-06-04T04:54:19.641520+0000","last_active":"2024-06-04T04:54:31.649573+0000","last_peered":"2024-06-04T04:54:31.649573+0000","last_clean":"2024-06-04T04:54:31.649573+0000","last_became_active":"2024-06-04T04:54:19.632433+0000","last_became_peered":"2024-06-04T04:54:19.632433+0000","last_unstale":"2024-06-04T04:54:31.649573+0000","last_undegraded":"2024-06-04T04:54:31.649573+0000","last_fullsized":"2024-06-04T04:54:31.649573+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:26:05.993919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512831+0000","last_change":"2024-06-04T04:53:36.371827+0000","last_active":"2024-06-04T04:54:19.512831+0000","last_peered":"2024-06-04T04:54:19.512831+0000","last_clean":"2024-06-04T04:54:19.512831+0000","last_became_active":"2024-06-04T04:53:36.371509+0000","last_became_peered":"2024-06-04T04:53:36.371509+0000","last_unstale":"2024-06-04T04:54:19.512831+0000","last_undegraded":"2024-06-04T04:54:19.512831+0000","last_fullsized":"2024-06-04T04:54:19.512831+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:54:51.589008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.513177+0000","last_change":"2024-06-04T04:53:33.353976+0000","last_active":"2024-06-04T04:54:19.513177+0000","last_peered":"2024-06-04T04:54:19.513177+0000","last_clean":"2024-06-04T04:54:19.513177+0000","last_became_active":"2024-06-04T04:53:33.353586+0000","last_became_peered":"2024-06-04T04:53:33.353586+0000","last_unstale":"2024-06-04T04:54:19.513177+0000","last_undegraded":"2024-06-04T04:54:19.513177+0000","last_fullsized":"2024-06-04T04:54:19.513177+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:17:10.378008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510428+0000","last_change":"2024-06-04T04:53:36.367806+0000","last_active":"2024-06-04T04:54:19.510428+0000","last_peered":"2024-06-04T04:54:19.510428+0000","last_clean":"2024-06-04T04:54:19.510428+0000","last_became_active":"2024-06-04T04:53:36.367484+0000","last_became_peered":"2024-06-04T04:53:36.367484+0000","last_unstale":"2024-06-04T04:54:19.510428+0000","last_undegraded":"2024-06-04T04:54:19.510428+0000","last_fullsized":"2024-06-04T04:54:19.510428+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:25:36.194746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509723+0000","last_change":"2024-06-04T04:53:33.348902+0000","last_active":"2024-06-04T04:54:19.509723+0000","last_peered":"2024-06-04T04:54:19.509723+0000","last_clean":"2024-06-04T04:54:19.509723+0000","last_became_active":"2024-06-04T04:53:33.348441+0000","last_became_peered":"2024-06-04T04:53:33.348441+0000","last_unstale":"2024-06-04T04:54:19.509723+0000","last_undegraded":"2024-06-04T04:54:19.509723+0000","last_fullsized":"2024-06-04T04:54:19.509723+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:22:28.428303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631324+0000","last_change":"2024-06-04T04:53:36.376867+0000","last_active":"2024-06-04T04:54:19.631324+0000","last_peered":"2024-06-04T04:54:19.631324+0000","last_clean":"2024-06-04T04:54:19.631324+0000","last_became_active":"2024-06-04T04:53:36.376517+0000","last_became_peered":"2024-06-04T04:53:36.376517+0000","last_unstale":"2024-06-04T04:54:19.631324+0000","last_undegraded":"2024-06-04T04:54:19.631324+0000","last_fullsized":"2024-06-04T04:54:19.631324+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:21:59.494321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509187+0000","last_change":"2024-06-04T04:53:33.347993+0000","last_active":"2024-06-04T04:54:19.509187+0000","last_peered":"2024-06-04T04:54:19.509187+0000","last_clean":"2024-06-04T04:54:19.509187+0000","last_became_active":"2024-06-04T04:53:33.347656+0000","last_became_peered":"2024-06-04T04:53:33.347656+0000","last_unstale":"2024-06-04T04:54:19.509187+0000","last_undegraded":"2024-06-04T04:54:19.509187+0000","last_fullsized":"2024-06-04T04:54:19.509187+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:21:26.806557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.632790+0000","last_change":"2024-06-04T04:54:19.632790+0000","last_active":"2024-06-04T04:54:19.632790+0000","last_peered":"2024-06-04T04:54:19.632790+0000","last_clean":"2024-06-04T04:54:19.632790+0000","last_became_active":"2024-06-04T04:54:19.632338+0000","last_became_peered":"2024-06-04T04:54:19.632338+0000","last_unstale":"2024-06-04T04:54:19.632790+0000","last_undegraded":"2024-06-04T04:54:19.632790+0000","last_fullsized":"2024-06-04T04:54:19.632790+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:44:00.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'5","reported_seq":74,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:33.510236+0000","last_change":"2024-06-04T04:53:33.346111+0000","last_active":"2024-06-04T04:54:33.510236+0000","last_peered":"2024-06-04T04:54:33.510236+0000","last_clean":"2024-06-04T04:54:33.510236+0000","last_became_active":"2024-06-04T04:53:33.345773+0000","last_became_peered":"2024-06-04T04:53:33.345773+0000","last_unstale":"2024-06-04T04:54:33.510236+0000","last_undegraded":"2024-06-04T04:54:33.510236+0000","last_fullsized":"2024-06-04T04:54:33.510236+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:18.565583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508684+0000","last_change":"2024-06-04T04:53:36.373369+0000","last_active":"2024-06-04T04:54:19.508684+0000","last_peered":"2024-06-04T04:54:19.508684+0000","last_clean":"2024-06-04T04:54:19.508684+0000","last_became_active":"2024-06-04T04:53:36.373001+0000","last_became_peered":"2024-06-04T04:53:36.373001+0000","last_unstale":"2024-06-04T04:54:19.508684+0000","last_undegraded":"2024-06-04T04:54:19.508684+0000","last_fullsized":"2024-06-04T04:54:19.508684+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:02:22.012643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508915+0000","last_change":"2024-06-04T04:53:33.344358+0000","last_active":"2024-06-04T04:54:19.508915+0000","last_peered":"2024-06-04T04:54:19.508915+0000","last_clean":"2024-06-04T04:54:19.508915+0000","last_became_active":"2024-06-04T04:53:33.343980+0000","last_became_peered":"2024-06-04T04:53:33.343980+0000","last_unstale":"2024-06-04T04:54:19.508915+0000","last_undegraded":"2024-06-04T04:54:19.508915+0000","last_fullsized":"2024-06-04T04:54:19.508915+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:14.493138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512293+0000","last_change":"2024-06-04T04:53:36.371239+0000","last_active":"2024-06-04T04:54:19.512293+0000","last_peered":"2024-06-04T04:54:19.512293+0000","last_clean":"2024-06-04T04:54:19.512293+0000","last_became_active":"2024-06-04T04:53:36.370885+0000","last_became_peered":"2024-06-04T04:53:36.370885+0000","last_unstale":"2024-06-04T04:54:19.512293+0000","last_undegraded":"2024-06-04T04:54:19.512293+0000","last_fullsized":"2024-06-04T04:54:19.512293+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T04:54:29.753796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505354+0000","last_change":"2024-06-04T04:53:33.338512+0000","last_active":"2024-06-04T04:54:19.505354+0000","last_peered":"2024-06-04T04:54:19.505354+0000","last_clean":"2024-06-04T04:54:19.505354+0000","last_became_active":"2024-06-04T04:53:33.338180+0000","last_became_peered":"2024-06-04T04:53:33.338180+0000","last_unstale":"2024-06-04T04:54:19.505354+0000","last_undegraded":"2024-06-04T04:54:19.505354+0000","last_fullsized":"2024-06-04T04:54:19.505354+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:57:13.406479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512117+0000","last_change":"2024-06-04T04:53:36.377316+0000","last_active":"2024-06-04T04:54:19.512117+0000","last_peered":"2024-06-04T04:54:19.512117+0000","last_clean":"2024-06-04T04:54:19.512117+0000","last_became_active":"2024-06-04T04:53:36.376992+0000","last_became_peered":"2024-06-04T04:53:36.376992+0000","last_unstale":"2024-06-04T04:54:19.512117+0000","last_undegraded":"2024-06-04T04:54:19.512117+0000","last_fullsized":"2024-06-04T04:54:19.512117+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:33.145631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511696+0000","last_change":"2024-06-04T04:53:33.348305+0000","last_active":"2024-06-04T04:54:19.511696+0000","last_peered":"2024-06-04T04:54:19.511696+0000","last_clean":"2024-06-04T04:54:19.511696+0000","last_became_active":"2024-06-04T04:53:33.347948+0000","last_became_peered":"2024-06-04T04:53:33.347948+0000","last_unstale":"2024-06-04T04:54:19.511696+0000","last_undegraded":"2024-06-04T04:54:19.511696+0000","last_fullsized":"2024-06-04T04:54:19.511696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:04:13.669274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508429+0000","last_change":"2024-06-04T04:53:36.367955+0000","last_active":"2024-06-04T04:54:19.508429+0000","last_peered":"2024-06-04T04:54:19.508429+0000","last_clean":"2024-06-04T04:54:19.508429+0000","last_became_active":"2024-06-04T04:53:36.367628+0000","last_became_peered":"2024-06-04T04:53:36.367628+0000","last_unstale":"2024-06-04T04:54:19.508429+0000","last_undegraded":"2024-06-04T04:54:19.508429+0000","last_fullsized":"2024-06-04T04:54:19.508429+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:33:26.185498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'4","reported_seq":107,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:32.508115+0000","last_change":"2024-06-04T04:53:33.349471+0000","last_active":"2024-06-04T04:54:32.508115+0000","last_peered":"2024-06-04T04:54:32.508115+0000","last_clean":"2024-06-04T04:54:32.508115+0000","last_became_active":"2024-06-04T04:53:33.349105+0000","last_became_peered":"2024-06-04T04:53:33.349105+0000","last_unstale":"2024-06-04T04:54:32.508115+0000","last_undegraded":"2024-06-04T04:54:32.508115+0000","last_fullsized":"2024-06-04T04:54:32.508115+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:31:35.584275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":78,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631859+0000","last_change":"2024-06-04T04:53:36.367249+0000","last_active":"2024-06-04T04:54:19.631859+0000","last_peered":"2024-06-04T04:54:19.631859+0000","last_clean":"2024-06-04T04:54:19.631859+0000","last_became_active":"2024-06-04T04:53:36.366863+0000","last_became_peered":"2024-06-04T04:53:36.366863+0000","last_unstale":"2024-06-04T04:54:19.631859+0000","last_undegraded":"2024-06-04T04:54:19.631859+0000","last_fullsized":"2024-06-04T04:54:19.631859+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:13:17.308321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510836+0000","last_change":"2024-06-04T04:53:33.346778+0000","last_active":"2024-06-04T04:54:19.510836+0000","last_peered":"2024-06-04T04:54:19.510836+0000","last_clean":"2024-06-04T04:54:19.510836+0000","last_became_active":"2024-06-04T04:53:33.346376+0000","last_became_peered":"2024-06-04T04:53:33.346376+0000","last_unstale":"2024-06-04T04:54:19.510836+0000","last_undegraded":"2024-06-04T04:54:19.510836+0000","last_fullsized":"2024-06-04T04:54:19.510836+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:03:16.973822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.514491+0000","last_change":"2024-06-04T04:54:19.514491+0000","last_active":"2024-06-04T04:54:19.514491+0000","last_peered":"2024-06-04T04:54:19.514491+0000","last_clean":"2024-06-04T04:54:19.514491+0000","last_became_active":"2024-06-04T04:54:19.514148+0000","last_became_peered":"2024-06-04T04:54:19.514148+0000","last_unstale":"2024-06-04T04:54:19.514491+0000","last_undegraded":"2024-06-04T04:54:19.514491+0000","last_fullsized":"2024-06-04T04:54:19.514491+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:09:59.641041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512326+0000","last_change":"2024-06-04T04:53:33.354989+0000","last_active":"2024-06-04T04:54:19.512326+0000","last_peered":"2024-06-04T04:54:19.512326+0000","last_clean":"2024-06-04T04:54:19.512326+0000","last_became_active":"2024-06-04T04:53:33.354613+0000","last_became_peered":"2024-06-04T04:53:33.354613+0000","last_unstale":"2024-06-04T04:54:19.512326+0000","last_undegraded":"2024-06-04T04:54:19.512326+0000","last_fullsized":"2024-06-04T04:54:19.512326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:19:59.398763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631419+0000","last_change":"2024-06-04T04:53:36.372216+0000","last_active":"2024-06-04T04:54:19.631419+0000","last_peered":"2024-06-04T04:54:19.631419+0000","last_clean":"2024-06-04T04:54:19.631419+0000","last_became_active":"2024-06-04T04:53:36.371820+0000","last_became_peered":"2024-06-04T04:53:36.371820+0000","last_unstale":"2024-06-04T04:54:19.631419+0000","last_undegraded":"2024-06-04T04:54:19.631419+0000","last_fullsized":"2024-06-04T04:54:19.631419+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:55:59.887963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'34","reported_seq":146,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:32.513330+0000","last_change":"2024-06-04T04:53:33.345493+0000","last_active":"2024-06-04T04:54:32.513330+0000","last_peered":"2024-06-04T04:54:32.513330+0000","last_clean":"2024-06-04T04:54:32.513330+0000","last_became_active":"2024-06-04T04:53:33.345102+0000","last_became_peered":"2024-06-04T04:53:33.345102+0000","last_unstale":"2024-06-04T04:54:32.513330+0000","last_undegraded":"2024-06-04T04:54:32.513330+0000","last_fullsized":"2024-06-04T04:54:32.513330+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:50:27.853391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66203,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":87,"num_read_kb":108,"num_write":36,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509448+0000","last_change":"2024-06-04T04:53:36.382070+0000","last_active":"2024-06-04T04:54:19.509448+0000","last_peered":"2024-06-04T04:54:19.509448+0000","last_clean":"2024-06-04T04:54:19.509448+0000","last_became_active":"2024-06-04T04:53:36.379144+0000","last_became_peered":"2024-06-04T04:53:36.379144+0000","last_unstale":"2024-06-04T04:54:19.509448+0000","last_undegraded":"2024-06-04T04:54:19.509448+0000","last_fullsized":"2024-06-04T04:54:19.509448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:16.702709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.506537+0000","last_change":"2024-06-04T04:53:33.344726+0000","last_active":"2024-06-04T04:54:19.506537+0000","last_peered":"2024-06-04T04:54:19.506537+0000","last_clean":"2024-06-04T04:54:19.506537+0000","last_became_active":"2024-06-04T04:53:33.344360+0000","last_became_peered":"2024-06-04T04:53:33.344360+0000","last_unstale":"2024-06-04T04:54:19.506537+0000","last_undegraded":"2024-06-04T04:54:19.506537+0000","last_fullsized":"2024-06-04T04:54:19.506537+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:44:56.784808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505351+0000","last_change":"2024-06-04T04:53:36.379036+0000","last_active":"2024-06-04T04:54:19.505351+0000","last_peered":"2024-06-04T04:54:19.505351+0000","last_clean":"2024-06-04T04:54:19.505351+0000","last_became_active":"2024-06-04T04:53:36.378681+0000","last_became_peered":"2024-06-04T04:53:36.378681+0000","last_unstale":"2024-06-04T04:54:19.505351+0000","last_undegraded":"2024-06-04T04:54:19.505351+0000","last_fullsized":"2024-06-04T04:54:19.505351+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:07:49.704731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511967+0000","last_change":"2024-06-04T04:53:33.345915+0000","last_active":"2024-06-04T04:54:19.511967+0000","last_peered":"2024-06-04T04:54:19.511967+0000","last_clean":"2024-06-04T04:54:19.511967+0000","last_became_active":"2024-06-04T04:53:33.345537+0000","last_became_peered":"2024-06-04T04:53:33.345537+0000","last_unstale":"2024-06-04T04:54:19.511967+0000","last_undegraded":"2024-06-04T04:54:19.511967+0000","last_fullsized":"2024-06-04T04:54:19.511967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:06:44.086968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510431+0000","last_change":"2024-06-04T04:53:36.379094+0000","last_active":"2024-06-04T04:54:19.510431+0000","last_peered":"2024-06-04T04:54:19.510431+0000","last_clean":"2024-06-04T04:54:19.510431+0000","last_became_active":"2024-06-04T04:53:36.378763+0000","last_became_peered":"2024-06-04T04:53:36.378763+0000","last_unstale":"2024-06-04T04:54:19.510431+0000","last_undegraded":"2024-06-04T04:54:19.510431+0000","last_fullsized":"2024-06-04T04:54:19.510431+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:16.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":74,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:33.508954+0000","last_change":"2024-06-04T04:53:33.341342+0000","last_active":"2024-06-04T04:54:33.508954+0000","last_peered":"2024-06-04T04:54:33.508954+0000","last_clean":"2024-06-04T04:54:33.508954+0000","last_became_active":"2024-06-04T04:53:33.340862+0000","last_became_peered":"2024-06-04T04:53:33.340862+0000","last_unstale":"2024-06-04T04:54:33.508954+0000","last_undegraded":"2024-06-04T04:54:33.508954+0000","last_fullsized":"2024-06-04T04:54:33.508954+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:26.818063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":44,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510293+0000","last_change":"2024-06-04T04:53:36.383124+0000","last_active":"2024-06-04T04:54:19.510293+0000","last_peered":"2024-06-04T04:54:19.510293+0000","last_clean":"2024-06-04T04:54:19.510293+0000","last_became_active":"2024-06-04T04:53:36.382449+0000","last_became_peered":"2024-06-04T04:53:36.382449+0000","last_unstale":"2024-06-04T04:54:19.510293+0000","last_undegraded":"2024-06-04T04:54:19.510293+0000","last_fullsized":"2024-06-04T04:54:19.510293+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:52:40.776003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":44,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508738+0000","last_change":"2024-06-04T04:53:33.354556+0000","last_active":"2024-06-04T04:54:19.508738+0000","last_peered":"2024-06-04T04:54:19.508738+0000","last_clean":"2024-06-04T04:54:19.508738+0000","last_became_active":"2024-06-04T04:53:33.354238+0000","last_became_peered":"2024-06-04T04:53:33.354238+0000","last_unstale":"2024-06-04T04:54:19.508738+0000","last_undegraded":"2024-06-04T04:54:19.508738+0000","last_fullsized":"2024-06-04T04:54:19.508738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:54.697374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511906+0000","last_change":"2024-06-04T04:53:36.371387+0000","last_active":"2024-06-04T04:54:19.511906+0000","last_peered":"2024-06-04T04:54:19.511906+0000","last_clean":"2024-06-04T04:54:19.511906+0000","last_became_active":"2024-06-04T04:53:36.370852+0000","last_became_peered":"2024-06-04T04:53:36.370852+0000","last_unstale":"2024-06-04T04:54:19.511906+0000","last_undegraded":"2024-06-04T04:54:19.511906+0000","last_fullsized":"2024-06-04T04:54:19.511906+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:44:06.956210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508796+0000","last_change":"2024-06-04T04:53:33.349982+0000","last_active":"2024-06-04T04:54:19.508796+0000","last_peered":"2024-06-04T04:54:19.508796+0000","last_clean":"2024-06-04T04:54:19.508796+0000","last_became_active":"2024-06-04T04:53:33.349633+0000","last_became_peered":"2024-06-04T04:53:33.349633+0000","last_unstale":"2024-06-04T04:54:19.508796+0000","last_undegraded":"2024-06-04T04:54:19.508796+0000","last_fullsized":"2024-06-04T04:54:19.508796+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:40.107426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509399+0000","last_change":"2024-06-04T04:53:36.382313+0000","last_active":"2024-06-04T04:54:19.509399+0000","last_peered":"2024-06-04T04:54:19.509399+0000","last_clean":"2024-06-04T04:54:19.509399+0000","last_became_active":"2024-06-04T04:53:36.379638+0000","last_became_peered":"2024-06-04T04:53:36.379638+0000","last_unstale":"2024-06-04T04:54:19.509399+0000","last_undegraded":"2024-06-04T04:54:19.509399+0000","last_fullsized":"2024-06-04T04:54:19.509399+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:43:42.527262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.513581+0000","last_change":"2024-06-04T04:54:19.513581+0000","last_active":"2024-06-04T04:54:19.513581+0000","last_peered":"2024-06-04T04:54:19.513581+0000","last_clean":"2024-06-04T04:54:19.513581+0000","last_became_active":"2024-06-04T04:54:19.513111+0000","last_became_peered":"2024-06-04T04:54:19.513111+0000","last_unstale":"2024-06-04T04:54:19.513581+0000","last_undegraded":"2024-06-04T04:54:19.513581+0000","last_fullsized":"2024-06-04T04:54:19.513581+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:57.028083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508983+0000","last_change":"2024-06-04T04:53:36.369904+0000","last_active":"2024-06-04T04:54:19.508983+0000","last_peered":"2024-06-04T04:54:19.508983+0000","last_clean":"2024-06-04T04:54:19.508983+0000","last_became_active":"2024-06-04T04:53:36.367929+0000","last_became_peered":"2024-06-04T04:53:36.367929+0000","last_unstale":"2024-06-04T04:54:19.508983+0000","last_undegraded":"2024-06-04T04:54:19.508983+0000","last_fullsized":"2024-06-04T04:54:19.508983+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:37:08.721569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'6","reported_seq":74,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:32.651358+0000","last_change":"2024-06-04T04:53:33.342723+0000","last_active":"2024-06-04T04:54:32.651358+0000","last_peered":"2024-06-04T04:54:32.651358+0000","last_clean":"2024-06-04T04:54:32.651358+0000","last_became_active":"2024-06-04T04:53:33.342344+0000","last_became_peered":"2024-06-04T04:53:33.342344+0000","last_unstale":"2024-06-04T04:54:32.651358+0000","last_undegraded":"2024-06-04T04:54:32.651358+0000","last_fullsized":"2024-06-04T04:54:32.651358+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:59:50.147517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.513534+0000","last_change":"2024-06-04T04:54:19.513534+0000","last_active":"2024-06-04T04:54:19.513534+0000","last_peered":"2024-06-04T04:54:19.513534+0000","last_clean":"2024-06-04T04:54:19.513534+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:54:19.513534+0000","last_undegraded":"2024-06-04T04:54:19.513534+0000","last_fullsized":"2024-06-04T04:54:19.513534+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:16:32.240932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512091+0000","last_change":"2024-06-04T04:53:36.371236+0000","last_active":"2024-06-04T04:54:19.512091+0000","last_peered":"2024-06-04T04:54:19.512091+0000","last_clean":"2024-06-04T04:54:19.512091+0000","last_became_active":"2024-06-04T04:53:36.370876+0000","last_became_peered":"2024-06-04T04:53:36.370876+0000","last_unstale":"2024-06-04T04:54:19.512091+0000","last_undegraded":"2024-06-04T04:54:19.512091+0000","last_fullsized":"2024-06-04T04:54:19.512091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:15:56.751317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":73,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:34.509562+0000","last_change":"2024-06-04T04:53:33.355642+0000","last_active":"2024-06-04T04:54:34.509562+0000","last_peered":"2024-06-04T04:54:34.509562+0000","last_clean":"2024-06-04T04:54:34.509562+0000","last_became_active":"2024-06-04T04:53:33.355187+0000","last_became_peered":"2024-06-04T04:53:33.355187+0000","last_unstale":"2024-06-04T04:54:34.509562+0000","last_undegraded":"2024-06-04T04:54:34.509562+0000","last_fullsized":"2024-06-04T04:54:34.509562+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:32:28.331518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":43,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.632155+0000","last_change":"2024-06-04T04:54:19.631750+0000","last_active":"2024-06-04T04:54:19.632155+0000","last_peered":"2024-06-04T04:54:19.632155+0000","last_clean":"2024-06-04T04:54:19.632155+0000","last_became_active":"2024-06-04T04:54:19.631485+0000","last_became_peered":"2024-06-04T04:54:19.631485+0000","last_unstale":"2024-06-04T04:54:19.632155+0000","last_undegraded":"2024-06-04T04:54:19.632155+0000","last_fullsized":"2024-06-04T04:54:19.632155+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:29:44.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631469+0000","last_change":"2024-06-04T04:53:33.340653+0000","last_active":"2024-06-04T04:54:19.631469+0000","last_peered":"2024-06-04T04:54:19.631469+0000","last_clean":"2024-06-04T04:54:19.631469+0000","last_became_active":"2024-06-04T04:53:33.340250+0000","last_became_peered":"2024-06-04T04:53:33.340250+0000","last_unstale":"2024-06-04T04:54:19.631469+0000","last_undegraded":"2024-06-04T04:54:19.631469+0000","last_fullsized":"2024-06-04T04:54:19.631469+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:46:20.329514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":99,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510637+0000","last_change":"2024-06-04T04:52:09.181155+0000","last_active":"2024-06-04T04:54:19.510637+0000","last_peered":"2024-06-04T04:54:19.510637+0000","last_clean":"2024-06-04T04:54:19.510637+0000","last_became_active":"2024-06-04T04:52:09.177002+0000","last_became_peered":"2024-06-04T04:52:09.177002+0000","last_unstale":"2024-06-04T04:54:19.510637+0000","last_undegraded":"2024-06-04T04:54:19.510637+0000","last_fullsized":"2024-06-04T04:54:19.510637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_clean_scrub_stamp":"2024-06-04T04:48:44.631948+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:46:57.735302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505220+0000","last_change":"2024-06-04T04:53:36.372634+0000","last_active":"2024-06-04T04:54:19.505220+0000","last_peered":"2024-06-04T04:54:19.505220+0000","last_clean":"2024-06-04T04:54:19.505220+0000","last_became_active":"2024-06-04T04:53:36.372290+0000","last_became_peered":"2024-06-04T04:53:36.372290+0000","last_unstale":"2024-06-04T04:54:19.505220+0000","last_undegraded":"2024-06-04T04:54:19.505220+0000","last_fullsized":"2024-06-04T04:54:19.505220+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:14:07.460382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511010+0000","last_change":"2024-06-04T04:53:33.348760+0000","last_active":"2024-06-04T04:54:19.511010+0000","last_peered":"2024-06-04T04:54:19.511010+0000","last_clean":"2024-06-04T04:54:19.511010+0000","last_became_active":"2024-06-04T04:53:33.348347+0000","last_became_peered":"2024-06-04T04:53:33.348347+0000","last_unstale":"2024-06-04T04:54:19.511010+0000","last_undegraded":"2024-06-04T04:54:19.511010+0000","last_fullsized":"2024-06-04T04:54:19.511010+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:36:20.641061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509558+0000","last_change":"2024-06-04T04:53:33.354840+0000","last_active":"2024-06-04T04:54:19.509558+0000","last_peered":"2024-06-04T04:54:19.509558+0000","last_clean":"2024-06-04T04:54:19.509558+0000","last_became_active":"2024-06-04T04:53:33.354287+0000","last_became_peered":"2024-06-04T04:53:33.354287+0000","last_unstale":"2024-06-04T04:54:19.509558+0000","last_undegraded":"2024-06-04T04:54:19.509558+0000","last_fullsized":"2024-06-04T04:54:19.509558+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:42.550098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505559+0000","last_change":"2024-06-04T04:53:36.372678+0000","last_active":"2024-06-04T04:54:19.505559+0000","last_peered":"2024-06-04T04:54:19.505559+0000","last_clean":"2024-06-04T04:54:19.505559+0000","last_became_active":"2024-06-04T04:53:36.372312+0000","last_became_peered":"2024-06-04T04:53:36.372312+0000","last_unstale":"2024-06-04T04:54:19.505559+0000","last_undegraded":"2024-06-04T04:54:19.505559+0000","last_fullsized":"2024-06-04T04:54:19.505559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:34:41.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"79'14","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508647+0000","last_change":"2024-06-04T04:53:33.348440+0000","last_active":"2024-06-04T04:54:19.508647+0000","last_peered":"2024-06-04T04:54:19.508647+0000","last_clean":"2024-06-04T04:54:19.508647+0000","last_became_active":"2024-06-04T04:53:33.348131+0000","last_became_peered":"2024-06-04T04:53:33.348131+0000","last_unstale":"2024-06-04T04:54:19.508647+0000","last_undegraded":"2024-06-04T04:54:19.508647+0000","last_fullsized":"2024-06-04T04:54:19.508647+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:14:11.552286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.513045+0000","last_change":"2024-06-04T04:53:36.378669+0000","last_active":"2024-06-04T04:54:19.513045+0000","last_peered":"2024-06-04T04:54:19.513045+0000","last_clean":"2024-06-04T04:54:19.513045+0000","last_became_active":"2024-06-04T04:53:36.378233+0000","last_became_peered":"2024-06-04T04:53:36.378233+0000","last_unstale":"2024-06-04T04:54:19.513045+0000","last_undegraded":"2024-06-04T04:54:19.513045+0000","last_fullsized":"2024-06-04T04:54:19.513045+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:36:45.870924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631647+0000","last_change":"2024-06-04T04:53:33.345078+0000","last_active":"2024-06-04T04:54:19.631647+0000","last_peered":"2024-06-04T04:54:19.631647+0000","last_clean":"2024-06-04T04:54:19.631647+0000","last_became_active":"2024-06-04T04:53:33.344708+0000","last_became_peered":"2024-06-04T04:53:33.344708+0000","last_unstale":"2024-06-04T04:54:19.631647+0000","last_undegraded":"2024-06-04T04:54:19.631647+0000","last_fullsized":"2024-06-04T04:54:19.631647+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:45:14.010758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510104+0000","last_change":"2024-06-04T04:53:36.364817+0000","last_active":"2024-06-04T04:54:19.510104+0000","last_peered":"2024-06-04T04:54:19.510104+0000","last_clean":"2024-06-04T04:54:19.510104+0000","last_became_active":"2024-06-04T04:53:36.364499+0000","last_became_peered":"2024-06-04T04:53:36.364499+0000","last_unstale":"2024-06-04T04:54:19.510104+0000","last_undegraded":"2024-06-04T04:54:19.510104+0000","last_fullsized":"2024-06-04T04:54:19.510104+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:39:30.518028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.630871+0000","last_change":"2024-06-04T04:53:33.354527+0000","last_active":"2024-06-04T04:54:19.630871+0000","last_peered":"2024-06-04T04:54:19.630871+0000","last_clean":"2024-06-04T04:54:19.630871+0000","last_became_active":"2024-06-04T04:53:33.354175+0000","last_became_peered":"2024-06-04T04:53:33.354175+0000","last_unstale":"2024-06-04T04:54:19.630871+0000","last_undegraded":"2024-06-04T04:54:19.630871+0000","last_fullsized":"2024-06-04T04:54:19.630871+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:33:08.125428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508458+0000","last_change":"2024-06-04T04:53:36.373248+0000","last_active":"2024-06-04T04:54:19.508458+0000","last_peered":"2024-06-04T04:54:19.508458+0000","last_clean":"2024-06-04T04:54:19.508458+0000","last_became_active":"2024-06-04T04:53:36.372915+0000","last_became_peered":"2024-06-04T04:53:36.372915+0000","last_unstale":"2024-06-04T04:54:19.508458+0000","last_undegraded":"2024-06-04T04:54:19.508458+0000","last_fullsized":"2024-06-04T04:54:19.508458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:23:11.689349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512413+0000","last_change":"2024-06-04T04:53:33.340427+0000","last_active":"2024-06-04T04:54:19.512413+0000","last_peered":"2024-06-04T04:54:19.512413+0000","last_clean":"2024-06-04T04:54:19.512413+0000","last_became_active":"2024-06-04T04:53:33.340022+0000","last_became_peered":"2024-06-04T04:53:33.340022+0000","last_unstale":"2024-06-04T04:54:19.512413+0000","last_undegraded":"2024-06-04T04:54:19.512413+0000","last_fullsized":"2024-06-04T04:54:19.512413+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:56:16.496471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509628+0000","last_change":"2024-06-04T04:53:36.380114+0000","last_active":"2024-06-04T04:54:19.509628+0000","last_peered":"2024-06-04T04:54:19.509628+0000","last_clean":"2024-06-04T04:54:19.509628+0000","last_became_active":"2024-06-04T04:53:36.379392+0000","last_became_peered":"2024-06-04T04:53:36.379392+0000","last_unstale":"2024-06-04T04:54:19.509628+0000","last_undegraded":"2024-06-04T04:54:19.509628+0000","last_fullsized":"2024-06-04T04:54:19.509628+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:41:48.776439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.520195+0000","last_change":"2024-06-04T04:54:19.519755+0000","last_active":"2024-06-04T04:54:19.520195+0000","last_peered":"2024-06-04T04:54:19.520195+0000","last_clean":"2024-06-04T04:54:19.520195+0000","last_became_active":"2024-06-04T04:54:19.513992+0000","last_became_peered":"2024-06-04T04:54:19.513992+0000","last_unstale":"2024-06-04T04:54:19.520195+0000","last_undegraded":"2024-06-04T04:54:19.520195+0000","last_fullsized":"2024-06-04T04:54:19.520195+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:01.951935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512566+0000","last_change":"2024-06-04T04:53:36.377420+0000","last_active":"2024-06-04T04:54:19.512566+0000","last_peered":"2024-06-04T04:54:19.512566+0000","last_clean":"2024-06-04T04:54:19.512566+0000","last_became_active":"2024-06-04T04:53:36.377109+0000","last_became_peered":"2024-06-04T04:53:36.377109+0000","last_unstale":"2024-06-04T04:54:19.512566+0000","last_undegraded":"2024-06-04T04:54:19.512566+0000","last_fullsized":"2024-06-04T04:54:19.512566+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:47:12.102396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":43,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.645328+0000","last_change":"2024-06-04T04:54:19.645328+0000","last_active":"2024-06-04T04:54:19.645328+0000","last_peered":"2024-06-04T04:54:19.645328+0000","last_clean":"2024-06-04T04:54:19.645328+0000","last_became_active":"2024-06-04T04:54:19.633416+0000","last_became_peered":"2024-06-04T04:54:19.633416+0000","last_unstale":"2024-06-04T04:54:19.645328+0000","last_undegraded":"2024-06-04T04:54:19.645328+0000","last_fullsized":"2024-06-04T04:54:19.645328+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:04:20.494641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510562+0000","last_change":"2024-06-04T04:53:36.381187+0000","last_active":"2024-06-04T04:54:19.510562+0000","last_peered":"2024-06-04T04:54:19.510562+0000","last_clean":"2024-06-04T04:54:19.510562+0000","last_became_active":"2024-06-04T04:53:36.380591+0000","last_became_peered":"2024-06-04T04:53:36.380591+0000","last_unstale":"2024-06-04T04:54:19.510562+0000","last_undegraded":"2024-06-04T04:54:19.510562+0000","last_fullsized":"2024-06-04T04:54:19.510562+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:26:43.122339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'5","reported_seq":72,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:32.508922+0000","last_change":"2024-06-04T04:53:33.355908+0000","last_active":"2024-06-04T04:54:32.508922+0000","last_peered":"2024-06-04T04:54:32.508922+0000","last_clean":"2024-06-04T04:54:32.508922+0000","last_became_active":"2024-06-04T04:53:33.355522+0000","last_became_peered":"2024-06-04T04:53:33.355522+0000","last_unstale":"2024-06-04T04:54:32.508922+0000","last_undegraded":"2024-06-04T04:54:32.508922+0000","last_fullsized":"2024-06-04T04:54:32.508922+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:16:44.639431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":42,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510919+0000","last_change":"2024-06-04T04:53:36.382737+0000","last_active":"2024-06-04T04:54:19.510919+0000","last_peered":"2024-06-04T04:54:19.510919+0000","last_clean":"2024-06-04T04:54:19.510919+0000","last_became_active":"2024-06-04T04:53:36.382353+0000","last_became_peered":"2024-06-04T04:53:36.382353+0000","last_unstale":"2024-06-04T04:54:19.510919+0000","last_undegraded":"2024-06-04T04:54:19.510919+0000","last_fullsized":"2024-06-04T04:54:19.510919+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:18:01.476058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512086+0000","last_change":"2024-06-04T04:53:33.347273+0000","last_active":"2024-06-04T04:54:19.512086+0000","last_peered":"2024-06-04T04:54:19.512086+0000","last_clean":"2024-06-04T04:54:19.512086+0000","last_became_active":"2024-06-04T04:53:33.346915+0000","last_became_peered":"2024-06-04T04:53:33.346915+0000","last_unstale":"2024-06-04T04:54:19.512086+0000","last_undegraded":"2024-06-04T04:54:19.512086+0000","last_fullsized":"2024-06-04T04:54:19.512086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:17.235484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"79'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510494+0000","last_change":"2024-06-04T04:53:36.383454+0000","last_active":"2024-06-04T04:54:19.510494+0000","last_peered":"2024-06-04T04:54:19.510494+0000","last_clean":"2024-06-04T04:54:19.510494+0000","last_became_active":"2024-06-04T04:53:36.383105+0000","last_became_peered":"2024-06-04T04:53:36.383105+0000","last_unstale":"2024-06-04T04:54:19.510494+0000","last_undegraded":"2024-06-04T04:54:19.510494+0000","last_fullsized":"2024-06-04T04:54:19.510494+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:04:02.147688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505624+0000","last_change":"2024-06-04T04:53:33.345435+0000","last_active":"2024-06-04T04:54:19.505624+0000","last_peered":"2024-06-04T04:54:19.505624+0000","last_clean":"2024-06-04T04:54:19.505624+0000","last_became_active":"2024-06-04T04:53:33.345077+0000","last_became_peered":"2024-06-04T04:53:33.345077+0000","last_unstale":"2024-06-04T04:54:19.505624+0000","last_undegraded":"2024-06-04T04:54:19.505624+0000","last_fullsized":"2024-06-04T04:54:19.505624+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:20:13.992259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510565+0000","last_change":"2024-06-04T04:53:36.385238+0000","last_active":"2024-06-04T04:54:19.510565+0000","last_peered":"2024-06-04T04:54:19.510565+0000","last_clean":"2024-06-04T04:54:19.510565+0000","last_became_active":"2024-06-04T04:53:36.384899+0000","last_became_peered":"2024-06-04T04:53:36.384899+0000","last_unstale":"2024-06-04T04:54:19.510565+0000","last_undegraded":"2024-06-04T04:54:19.510565+0000","last_fullsized":"2024-06-04T04:54:19.510565+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:09:42.629340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509793+0000","last_change":"2024-06-04T04:53:36.383450+0000","last_active":"2024-06-04T04:54:19.509793+0000","last_peered":"2024-06-04T04:54:19.509793+0000","last_clean":"2024-06-04T04:54:19.509793+0000","last_became_active":"2024-06-04T04:53:36.382678+0000","last_became_peered":"2024-06-04T04:53:36.382678+0000","last_unstale":"2024-06-04T04:54:19.509793+0000","last_undegraded":"2024-06-04T04:54:19.509793+0000","last_fullsized":"2024-06-04T04:54:19.509793+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:36:28.932636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.632563+0000","last_change":"2024-06-04T04:54:19.632563+0000","last_active":"2024-06-04T04:54:19.632563+0000","last_peered":"2024-06-04T04:54:19.632563+0000","last_clean":"2024-06-04T04:54:19.632563+0000","last_became_active":"2024-06-04T04:54:19.632070+0000","last_became_peered":"2024-06-04T04:54:19.632070+0000","last_unstale":"2024-06-04T04:54:19.632563+0000","last_undegraded":"2024-06-04T04:54:19.632563+0000","last_fullsized":"2024-06-04T04:54:19.632563+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:07:16.552958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.506413+0000","last_change":"2024-06-04T04:53:36.381408+0000","last_active":"2024-06-04T04:54:19.506413+0000","last_peered":"2024-06-04T04:54:19.506413+0000","last_clean":"2024-06-04T04:54:19.506413+0000","last_became_active":"2024-06-04T04:53:36.381078+0000","last_became_peered":"2024-06-04T04:53:36.381078+0000","last_unstale":"2024-06-04T04:54:19.506413+0000","last_undegraded":"2024-06-04T04:54:19.506413+0000","last_fullsized":"2024-06-04T04:54:19.506413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:28:48.988591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510556+0000","last_change":"2024-06-04T04:53:36.360168+0000","last_active":"2024-06-04T04:54:19.510556+0000","last_peered":"2024-06-04T04:54:19.510556+0000","last_clean":"2024-06-04T04:54:19.510556+0000","last_became_active":"2024-06-04T04:53:36.359691+0000","last_became_peered":"2024-06-04T04:53:36.359691+0000","last_unstale":"2024-06-04T04:54:19.510556+0000","last_undegraded":"2024-06-04T04:54:19.510556+0000","last_fullsized":"2024-06-04T04:54:19.510556+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:55:17.418339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505605+0000","last_change":"2024-06-04T04:53:36.363018+0000","last_active":"2024-06-04T04:54:19.505605+0000","last_peered":"2024-06-04T04:54:19.505605+0000","last_clean":"2024-06-04T04:54:19.505605+0000","last_became_active":"2024-06-04T04:53:36.361595+0000","last_became_peered":"2024-06-04T04:53:36.361595+0000","last_unstale":"2024-06-04T04:54:19.505605+0000","last_undegraded":"2024-06-04T04:54:19.505605+0000","last_fullsized":"2024-06-04T04:54:19.505605+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:00:21.377322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509182+0000","last_change":"2024-06-04T04:53:36.375377+0000","last_active":"2024-06-04T04:54:19.509182+0000","last_peered":"2024-06-04T04:54:19.509182+0000","last_clean":"2024-06-04T04:54:19.509182+0000","last_became_active":"2024-06-04T04:53:36.373459+0000","last_became_peered":"2024-06-04T04:53:36.373459+0000","last_unstale":"2024-06-04T04:54:19.509182+0000","last_undegraded":"2024-06-04T04:54:19.509182+0000","last_fullsized":"2024-06-04T04:54:19.509182+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:51:31.686969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.506738+0000","last_change":"2024-06-04T04:53:36.373850+0000","last_active":"2024-06-04T04:54:19.506738+0000","last_peered":"2024-06-04T04:54:19.506738+0000","last_clean":"2024-06-04T04:54:19.506738+0000","last_became_active":"2024-06-04T04:53:36.371951+0000","last_became_peered":"2024-06-04T04:53:36.371951+0000","last_unstale":"2024-06-04T04:54:19.506738+0000","last_undegraded":"2024-06-04T04:54:19.506738+0000","last_fullsized":"2024-06-04T04:54:19.506738+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:06.452703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512871+0000","last_change":"2024-06-04T04:53:36.374871+0000","last_active":"2024-06-04T04:54:19.512871+0000","last_peered":"2024-06-04T04:54:19.512871+0000","last_clean":"2024-06-04T04:54:19.512871+0000","last_became_active":"2024-06-04T04:53:36.374486+0000","last_became_peered":"2024-06-04T04:53:36.374486+0000","last_unstale":"2024-06-04T04:54:19.512871+0000","last_undegraded":"2024-06-04T04:54:19.512871+0000","last_fullsized":"2024-06-04T04:54:19.512871+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:15:13.337374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.514431+0000","last_change":"2024-06-04T04:54:19.513540+0000","last_active":"2024-06-04T04:54:19.514431+0000","last_peered":"2024-06-04T04:54:19.514431+0000","last_clean":"2024-06-04T04:54:19.514431+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:54:19.514431+0000","last_undegraded":"2024-06-04T04:54:19.514431+0000","last_fullsized":"2024-06-04T04:54:19.514431+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:09:27.993585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512321+0000","last_change":"2024-06-04T04:53:36.373055+0000","last_active":"2024-06-04T04:54:19.512321+0000","last_peered":"2024-06-04T04:54:19.512321+0000","last_clean":"2024-06-04T04:54:19.512321+0000","last_became_active":"2024-06-04T04:53:36.372754+0000","last_became_peered":"2024-06-04T04:53:36.372754+0000","last_unstale":"2024-06-04T04:54:19.512321+0000","last_undegraded":"2024-06-04T04:54:19.512321+0000","last_fullsized":"2024-06-04T04:54:19.512321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:22:27.554200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631332+0000","last_change":"2024-06-04T04:53:36.378897+0000","last_active":"2024-06-04T04:54:19.631332+0000","last_peered":"2024-06-04T04:54:19.631332+0000","last_clean":"2024-06-04T04:54:19.631332+0000","last_became_active":"2024-06-04T04:53:36.378607+0000","last_became_peered":"2024-06-04T04:53:36.378607+0000","last_unstale":"2024-06-04T04:54:19.631332+0000","last_undegraded":"2024-06-04T04:54:19.631332+0000","last_fullsized":"2024-06-04T04:54:19.631332+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:03.395491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505499+0000","last_change":"2024-06-04T04:53:36.378240+0000","last_active":"2024-06-04T04:54:19.505499+0000","last_peered":"2024-06-04T04:54:19.505499+0000","last_clean":"2024-06-04T04:54:19.505499+0000","last_became_active":"2024-06-04T04:53:36.377909+0000","last_became_peered":"2024-06-04T04:53:36.377909+0000","last_unstale":"2024-06-04T04:54:19.505499+0000","last_undegraded":"2024-06-04T04:54:19.505499+0000","last_fullsized":"2024-06-04T04:54:19.505499+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:15:25.288340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510525+0000","last_change":"2024-06-04T04:53:36.379792+0000","last_active":"2024-06-04T04:54:19.510525+0000","last_peered":"2024-06-04T04:54:19.510525+0000","last_clean":"2024-06-04T04:54:19.510525+0000","last_became_active":"2024-06-04T04:53:36.379438+0000","last_became_peered":"2024-06-04T04:53:36.379438+0000","last_unstale":"2024-06-04T04:54:19.510525+0000","last_undegraded":"2024-06-04T04:54:19.510525+0000","last_fullsized":"2024-06-04T04:54:19.510525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:23:57.617877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":214930,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73016,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":433,"num_read_kb":292,"num_write":245,"num_write_kb":142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":219048,"data_compressed":5499,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678043,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":880640,"data_stored":429560,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874271,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":1086792,"data_compressed":43684,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037797,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":905216,"data_stored":429490,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234026,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":892928,"data_stored":429196,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430254,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":1022386,"data_compressed":41851,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593781,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":888832,"data_stored":429598,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":497386,"data_compressed":7592,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":909312,"data_stored":430546,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":496543,"data_compressed":7592,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149764,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36228,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708924,"statfs":{"total":95995035648,"available":95957938176,"internally_reserved":0,"allocated":905216,"data_stored":429032,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953535,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":888832,"data_stored":429068,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790010,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":1021673,"data_compressed":41851,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34470,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69018,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":68175,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":67939,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-04T04:54:38.061 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-04T04:54:38.062 INFO:tasks.cephadm:Setting up client nodes... 2024-06-04T04:54:38.062 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-04T04:54:38.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:37 smithi112 ceph-mon[36453]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:54:38.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:37 smithi098 ceph-mon[35569]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:54:38.299 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:39.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:38 smithi070 ceph-mon[32631]: pgmap v321: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-06-04T04:54:39.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:38 smithi112 ceph-mon[36453]: pgmap v321: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-06-04T04:54:39.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:38 smithi098 ceph-mon[35569]: pgmap v321: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-06-04T04:54:40.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.026+0000 7f8b436bc700 1 -- 172.21.15.70:0/3565867182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b3c102e70 msgr2=0x7f8b3c1032d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:40.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.026+0000 7f8b436bc700 1 --2- 172.21.15.70:0/3565867182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b3c102e70 0x7f8b3c1032d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8b3000aad0 tx=0x7f8b3000ade0 comp rx=0 tx=0).stop 2024-06-04T04:54:40.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.026+0000 7f8b436bc700 1 -- 172.21.15.70:0/3565867182 shutdown_connections 2024-06-04T04:54:40.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.026+0000 7f8b436bc700 1 --2- 172.21.15.70:0/3565867182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b3c103810 0x7f8b3c10a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:40.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.026+0000 7f8b436bc700 1 --2- 172.21.15.70:0/3565867182 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8b3c101c70 0x7f8b3c102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:40.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.026+0000 7f8b436bc700 1 --2- 172.21.15.70:0/3565867182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b3c102e70 0x7f8b3c1032d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:40.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.026+0000 7f8b436bc700 1 -- 172.21.15.70:0/3565867182 >> 172.21.15.70:0/3565867182 conn(0x7f8b3c0fd210 msgr2=0x7f8b3c0ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:40.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.026+0000 7f8b436bc700 1 -- 172.21.15.70:0/3565867182 shutdown_connections 2024-06-04T04:54:40.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.026+0000 7f8b436bc700 1 -- 172.21.15.70:0/3565867182 wait complete. 2024-06-04T04:54:40.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.027+0000 7f8b436bc700 1 Processor -- start 2024-06-04T04:54:40.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.027+0000 7f8b436bc700 1 -- start start 2024-06-04T04:54:40.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.027+0000 7f8b436bc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b3c101c70 0x7f8b3c199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:40.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.027+0000 7f8b436bc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8b3c102e70 0x7f8b3c199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:40.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.027+0000 7f8b436bc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b3c103810 0x7f8b3c1a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:40.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.027+0000 7f8b436bc700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3c10cd30 con 0x7f8b3c101c70 2024-06-04T04:54:40.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.027+0000 7f8b436bc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3c10cbb0 con 0x7f8b3c103810 2024-06-04T04:54:40.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.027+0000 7f8b436bc700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3c10ceb0 con 0x7f8b3c102e70 2024-06-04T04:54:40.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.027+0000 7f8b41458700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b3c101c70 0x7f8b3c199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:40.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.027+0000 7f8b41458700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b3c101c70 0x7f8b3c199550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47282/0 (socket says 172.21.15.70:47282) 2024-06-04T04:54:40.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.027+0000 7f8b41458700 1 -- 172.21.15.70:0/3591488641 learned_addr learned my addr 172.21.15.70:0/3591488641 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:40.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.027+0000 7f8b40c57700 1 --2- 172.21.15.70:0/3591488641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8b3c102e70 0x7f8b3c199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:40.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.027+0000 7f8b41c59700 1 --2- 172.21.15.70:0/3591488641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b3c103810 0x7f8b3c1a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:40.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.028+0000 7f8b41458700 1 -- 172.21.15.70:0/3591488641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8b3c102e70 msgr2=0x7f8b3c199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:40.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.028+0000 7f8b41458700 1 --2- 172.21.15.70:0/3591488641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8b3c102e70 0x7f8b3c199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:40.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.028+0000 7f8b41458700 1 -- 172.21.15.70:0/3591488641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b3c103810 msgr2=0x7f8b3c1a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:40.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.028+0000 7f8b41458700 1 --2- 172.21.15.70:0/3591488641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b3c103810 0x7f8b3c1a0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:40.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.028+0000 7f8b41458700 1 -- 172.21.15.70:0/3591488641 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b3c1a11b0 con 0x7f8b3c101c70 2024-06-04T04:54:40.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.028+0000 7f8b41458700 1 --2- 172.21.15.70:0/3591488641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b3c101c70 0x7f8b3c199550 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400e980 tx=0x7f8b3400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:40.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.028+0000 7f8b3a7fc700 1 -- 172.21.15.70:0/3591488641 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b34004230 con 0x7f8b3c101c70 2024-06-04T04:54:40.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.028+0000 7f8b3a7fc700 1 -- 172.21.15.70:0/3591488641 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8b340043d0 con 0x7f8b3c101c70 2024-06-04T04:54:40.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.029+0000 7f8b3a7fc700 1 -- 172.21.15.70:0/3591488641 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b34003c80 con 0x7f8b3c101c70 2024-06-04T04:54:40.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.029+0000 7f8b436bc700 1 -- 172.21.15.70:0/3591488641 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b3c1a14a0 con 0x7f8b3c101c70 2024-06-04T04:54:40.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.029+0000 7f8b436bc700 1 -- 172.21.15.70:0/3591488641 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b3c1a19e0 con 0x7f8b3c101c70 2024-06-04T04:54:40.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.030+0000 7f8b436bc700 1 -- 172.21.15.70:0/3591488641 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b3c051a80 con 0x7f8b3c101c70 2024-06-04T04:54:40.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.031+0000 7f8b3a7fc700 1 -- 172.21.15.70:0/3591488641 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f8b3400c810 con 0x7f8b3c101c70 2024-06-04T04:54:40.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.032+0000 7f8b3a7fc700 1 --2- 172.21.15.70:0/3591488641 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8b240b2a30 0x7f8b240b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:40.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.032+0000 7f8b40c57700 1 --2- 172.21.15.70:0/3591488641 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8b240b2a30 0x7f8b240b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:40.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.032+0000 7f8b3a7fc700 1 -- 172.21.15.70:0/3591488641 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f8b340e2520 con 0x7f8b3c101c70 2024-06-04T04:54:40.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.032+0000 7f8b40c57700 1 --2- 172.21.15.70:0/3591488641 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8b240b2a30 0x7f8b240b4ef0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8b30000c00 tx=0x7f8b30005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:40.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.036+0000 7f8b3a7fc700 1 -- 172.21.15.70:0/3591488641 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8b340aa0c0 con 0x7f8b3c101c70 2024-06-04T04:54:40.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.340+0000 7f8b436bc700 1 -- 172.21.15.70:0/3591488641 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f8b3c1a1c40 con 0x7f8b3c101c70 2024-06-04T04:54:40.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.342+0000 7f8b3a7fc700 1 -- 172.21.15.70:0/3591488641 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f8b340a99c0 con 0x7f8b3c101c70 2024-06-04T04:54:40.344 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-06-04T04:54:40.344 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQAQnl5mvlJaFBAA5whK+emxMmmeoClN3XjY5w== 2024-06-04T04:54:40.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.344+0000 7f8b436bc700 1 -- 172.21.15.70:0/3591488641 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8b240b2a30 msgr2=0x7f8b240b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:40.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.344+0000 7f8b436bc700 1 --2- 172.21.15.70:0/3591488641 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8b240b2a30 0x7f8b240b4ef0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8b30000c00 tx=0x7f8b30005fb0 comp rx=0 tx=0).stop 2024-06-04T04:54:40.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.344+0000 7f8b436bc700 1 -- 172.21.15.70:0/3591488641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b3c101c70 msgr2=0x7f8b3c199550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:40.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.344+0000 7f8b436bc700 1 --2- 172.21.15.70:0/3591488641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b3c101c70 0x7f8b3c199550 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400e980 tx=0x7f8b3400ec90 comp rx=0 tx=0).stop 2024-06-04T04:54:40.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.344+0000 7f8b436bc700 1 -- 172.21.15.70:0/3591488641 shutdown_connections 2024-06-04T04:54:40.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.344+0000 7f8b436bc700 1 --2- 172.21.15.70:0/3591488641 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8b240b2a30 0x7f8b240b4ef0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:40.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.344+0000 7f8b436bc700 1 --2- 172.21.15.70:0/3591488641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b3c103810 0x7f8b3c1a0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:40.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.344+0000 7f8b436bc700 1 --2- 172.21.15.70:0/3591488641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b3c101c70 0x7f8b3c199550 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:40.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.344+0000 7f8b436bc700 1 --2- 172.21.15.70:0/3591488641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8b3c102e70 0x7f8b3c199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:40.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.344+0000 7f8b436bc700 1 -- 172.21.15.70:0/3591488641 >> 172.21.15.70:0/3591488641 conn(0x7f8b3c0fd210 msgr2=0x7f8b3c1082b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:40.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.344+0000 7f8b436bc700 1 -- 172.21.15.70:0/3591488641 shutdown_connections 2024-06-04T04:54:40.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:40.344+0000 7f8b436bc700 1 -- 172.21.15.70:0/3591488641 wait complete. 2024-06-04T04:54:40.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:40 smithi070 ceph-mon[32631]: pgmap v322: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-06-04T04:54:40.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:40 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3591488641' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-04T04:54:40.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:40 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3591488641' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-04T04:54:41.001 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T04:54:41.001 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-06-04T04:54:41.001 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-04T04:54:41.039 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-04T04:54:41.039 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-04T04:54:41.039 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph mgr dump --format=json 2024-06-04T04:54:41.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:40 smithi112 ceph-mon[36453]: pgmap v322: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-06-04T04:54:41.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:40 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3591488641' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-04T04:54:41.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:40 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3591488641' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-04T04:54:41.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:40 smithi098 ceph-mon[35569]: pgmap v322: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-06-04T04:54:41.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:40 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3591488641' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-04T04:54:41.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:40 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3591488641' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-04T04:54:41.315 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:42.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.952+0000 7f43e3313700 1 -- 172.21.15.70:0/1735960019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43dc1041d0 msgr2=0x7f43dc1081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:42.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.952+0000 7f43e3313700 1 --2- 172.21.15.70:0/1735960019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43dc1041d0 0x7f43dc1081e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f43cc0099c0 tx=0x7f43cc009cd0 comp rx=0 tx=0).stop 2024-06-04T04:54:42.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.952+0000 7f43e3313700 1 -- 172.21.15.70:0/1735960019 shutdown_connections 2024-06-04T04:54:42.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.952+0000 7f43e3313700 1 --2- 172.21.15.70:0/1735960019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43dc108910 0x7f43dc110490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:42.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.952+0000 7f43e3313700 1 --2- 172.21.15.70:0/1735960019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43dc103820 0x7f43dc103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:42.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.952+0000 7f43e3313700 1 --2- 172.21.15.70:0/1735960019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43dc1041d0 0x7f43dc1081e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.952+0000 7f43e3313700 1 -- 172.21.15.70:0/1735960019 >> 172.21.15.70:0/1735960019 conn(0x7f43dc07a6b0 msgr2=0x7f43dc07aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.952+0000 7f43e3313700 1 -- 172.21.15.70:0/1735960019 shutdown_connections 2024-06-04T04:54:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.952+0000 7f43e3313700 1 -- 172.21.15.70:0/1735960019 wait complete. 2024-06-04T04:54:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.952+0000 7f43e3313700 1 Processor -- start 2024-06-04T04:54:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.952+0000 7f43e3313700 1 -- start start 2024-06-04T04:54:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e3313700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43dc103820 0x7f43dc075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e3313700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43dc1041d0 0x7f43dc075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e3313700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43dc108910 0x7f43dc0762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e3313700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc113580 con 0x7f43dc108910 2024-06-04T04:54:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e3313700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc113400 con 0x7f43dc103820 2024-06-04T04:54:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e3313700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc113700 con 0x7f43dc1041d0 2024-06-04T04:54:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e18b0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43dc108910 0x7f43dc0762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:42.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e18b0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43dc108910 0x7f43dc0762c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47298/0 (socket says 172.21.15.70:47298) 2024-06-04T04:54:42.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e18b0700 1 -- 172.21.15.70:0/2496519450 learned_addr learned my addr 172.21.15.70:0/2496519450 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:42.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e10af700 1 --2- 172.21.15.70:0/2496519450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43dc103820 0x7f43dc075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:42.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e08ae700 1 --2- 172.21.15.70:0/2496519450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43dc1041d0 0x7f43dc075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:42.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e18b0700 1 -- 172.21.15.70:0/2496519450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43dc1041d0 msgr2=0x7f43dc075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:42.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e18b0700 1 --2- 172.21.15.70:0/2496519450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43dc1041d0 0x7f43dc075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:42.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e18b0700 1 -- 172.21.15.70:0/2496519450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43dc103820 msgr2=0x7f43dc075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:42.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e18b0700 1 --2- 172.21.15.70:0/2496519450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43dc103820 0x7f43dc075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:42.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e18b0700 1 -- 172.21.15.70:0/2496519450 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43dc1a7fc0 con 0x7f43dc108910 2024-06-04T04:54:42.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e18b0700 1 --2- 172.21.15.70:0/2496519450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43dc108910 0x7f43dc0762c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f43d4007ee0 tx=0x7f43d400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:42.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43d27fc700 1 -- 172.21.15.70:0/2496519450 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d400dbd0 con 0x7f43dc108910 2024-06-04T04:54:42.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43d27fc700 1 -- 172.21.15.70:0/2496519450 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43d400dd70 con 0x7f43dc108910 2024-06-04T04:54:42.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e3313700 1 -- 172.21.15.70:0/2496519450 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43dc1a82b0 con 0x7f43dc108910 2024-06-04T04:54:42.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43d27fc700 1 -- 172.21.15.70:0/2496519450 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d40058d0 con 0x7f43dc108910 2024-06-04T04:54:42.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.953+0000 7f43e3313700 1 -- 172.21.15.70:0/2496519450 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43dc1a8770 con 0x7f43dc108910 2024-06-04T04:54:42.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.954+0000 7f43e3313700 1 -- 172.21.15.70:0/2496519450 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43dc06a230 con 0x7f43dc108910 2024-06-04T04:54:42.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.956+0000 7f43d27fc700 1 -- 172.21.15.70:0/2496519450 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f43d401a020 con 0x7f43dc108910 2024-06-04T04:54:42.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.956+0000 7f43d27fc700 1 --2- 172.21.15.70:0/2496519450 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f43c80b2910 0x7f43c80b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:42.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.956+0000 7f43e10af700 1 --2- 172.21.15.70:0/2496519450 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f43c80b2910 0x7f43c80b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:42.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.956+0000 7f43d27fc700 1 -- 172.21.15.70:0/2496519450 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f43d40e2170 con 0x7f43dc108910 2024-06-04T04:54:42.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.957+0000 7f43e10af700 1 --2- 172.21.15.70:0/2496519450 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f43c80b2910 0x7f43c80b4dd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f43d8005950 tx=0x7f43d800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:42.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:42.960+0000 7f43d27fc700 1 -- 172.21.15.70:0/2496519450 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f43d4010070 con 0x7f43dc108910 2024-06-04T04:54:43.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:42 smithi070 ceph-mon[32631]: pgmap v323: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 8 op/s 2024-06-04T04:54:43.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:42 smithi112 ceph-mon[36453]: pgmap v323: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 8 op/s 2024-06-04T04:54:43.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:43.224+0000 7f43e3313700 1 -- 172.21.15.70:0/2496519450 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f43dc02e7c0 con 0x7f43dc108910 2024-06-04T04:54:43.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:43.229+0000 7f43d27fc700 1 -- 172.21.15.70:0/2496519450 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+288645 (secure 0 0 0) 0x7f43d40a9d10 con 0x7f43dc108910 2024-06-04T04:54:43.236 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:54:43.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:43.232+0000 7f43e3313700 1 -- 172.21.15.70:0/2496519450 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f43c80b2910 msgr2=0x7f43c80b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:43.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:43.232+0000 7f43e3313700 1 --2- 172.21.15.70:0/2496519450 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f43c80b2910 0x7f43c80b4dd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f43d8005950 tx=0x7f43d800b410 comp rx=0 tx=0).stop 2024-06-04T04:54:43.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:43.232+0000 7f43e3313700 1 -- 172.21.15.70:0/2496519450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43dc108910 msgr2=0x7f43dc0762c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:43.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:43.232+0000 7f43e3313700 1 --2- 172.21.15.70:0/2496519450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43dc108910 0x7f43dc0762c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f43d4007ee0 tx=0x7f43d400d3b0 comp rx=0 tx=0).stop 2024-06-04T04:54:43.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:43.232+0000 7f43e3313700 1 -- 172.21.15.70:0/2496519450 shutdown_connections 2024-06-04T04:54:43.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:43.232+0000 7f43e3313700 1 --2- 172.21.15.70:0/2496519450 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f43c80b2910 0x7f43c80b4dd0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:43.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:43.232+0000 7f43e3313700 1 --2- 172.21.15.70:0/2496519450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43dc108910 0x7f43dc0762c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:43.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:43.232+0000 7f43e3313700 1 --2- 172.21.15.70:0/2496519450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43dc103820 0x7f43dc075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:43.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:43.232+0000 7f43e3313700 1 --2- 172.21.15.70:0/2496519450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43dc1041d0 0x7f43dc075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:43.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:43.232+0000 7f43e3313700 1 -- 172.21.15.70:0/2496519450 >> 172.21.15.70:0/2496519450 conn(0x7f43dc07a6b0 msgr2=0x7f43dc101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:43.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:43.232+0000 7f43e3313700 1 -- 172.21.15.70:0/2496519450 shutdown_connections 2024-06-04T04:54:43.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:43.232+0000 7f43e3313700 1 -- 172.21.15.70:0/2496519450 wait complete. 2024-06-04T04:54:43.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:42 smithi098 ceph-mon[35569]: pgmap v323: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 8 op/s 2024-06-04T04:54:43.990 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:43 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2496519450' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-04T04:54:44.019 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2752966840},{"type":"v1","addr":"172.21.15.70:6801","nonce":2752966840}]},"active_addr":"172.21.15.70:6801/2752966840","active_change":"2024-06-04T04:46:17.637054+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.70:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":145533746}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":3584175673}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":3720789940}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":4291129442}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":981951158}]}]} 2024-06-04T04:54:44.024 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-04T04:54:44.024 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-04T04:54:44.024 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-04T04:54:44.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:43 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2496519450' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-04T04:54:44.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:43 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2496519450' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-04T04:54:44.266 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:45.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:44 smithi070 ceph-mon[32631]: pgmap v324: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 10 op/s 2024-06-04T04:54:45.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:44 smithi112 ceph-mon[36453]: pgmap v324: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 10 op/s 2024-06-04T04:54:45.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:44 smithi098 ceph-mon[35569]: pgmap v324: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 10 op/s 2024-06-04T04:54:45.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.980+0000 7fecd1c37700 1 -- 172.21.15.70:0/1248754910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feccc1037f0 msgr2=0x7feccc103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:45.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.980+0000 7fecd1c37700 1 --2- 172.21.15.70:0/1248754910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feccc1037f0 0x7feccc103bd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0099c0 tx=0x7fecbc009cd0 comp rx=0 tx=0).stop 2024-06-04T04:54:45.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.980+0000 7fecd1c37700 1 -- 172.21.15.70:0/1248754910 shutdown_connections 2024-06-04T04:54:45.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.980+0000 7fecd1c37700 1 --2- 172.21.15.70:0/1248754910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feccc1088f0 0x7feccc110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:45.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.980+0000 7fecd1c37700 1 --2- 172.21.15.70:0/1248754910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feccc1037f0 0x7feccc103bd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:45.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.980+0000 7fecd1c37700 1 --2- 172.21.15.70:0/1248754910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feccc1041a0 0x7feccc1081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:45.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.981+0000 7fecd1c37700 1 -- 172.21.15.70:0/1248754910 >> 172.21.15.70:0/1248754910 conn(0x7feccc07a6a0 msgr2=0x7feccc07aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:45.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.981+0000 7fecd1c37700 1 -- 172.21.15.70:0/1248754910 shutdown_connections 2024-06-04T04:54:45.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.981+0000 7fecd1c37700 1 -- 172.21.15.70:0/1248754910 wait complete. 2024-06-04T04:54:45.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.981+0000 7fecd1c37700 1 Processor -- start 2024-06-04T04:54:45.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.981+0000 7fecd1c37700 1 -- start start 2024-06-04T04:54:45.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7fecd1c37700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feccc1037f0 0x7feccc19bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:45.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7fecd1c37700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feccc1041a0 0x7feccc19c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:45.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7fecd1c37700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feccc1088f0 0x7feccc1a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:45.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7fecd1c37700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7feccc113600 con 0x7feccc1037f0 2024-06-04T04:54:45.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7fecd1c37700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feccc113480 con 0x7feccc1088f0 2024-06-04T04:54:45.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7fecd1c37700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7feccc113780 con 0x7feccc1041a0 2024-06-04T04:54:45.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7feccb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feccc1037f0 0x7feccc19bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:45.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7feccb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feccc1037f0 0x7feccc19bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43788/0 (socket says 172.21.15.70:43788) 2024-06-04T04:54:45.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7feccb7fe700 1 -- 172.21.15.70:0/3916886620 learned_addr learned my addr 172.21.15.70:0/3916886620 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:45.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7feccaffd700 1 --2- 172.21.15.70:0/3916886620 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feccc1041a0 0x7feccc19c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:45.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7feccbfff700 1 --2- 172.21.15.70:0/3916886620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feccc1088f0 0x7feccc1a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:45.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7feccb7fe700 1 -- 172.21.15.70:0/3916886620 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feccc1041a0 msgr2=0x7feccc19c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:45.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7feccb7fe700 1 --2- 172.21.15.70:0/3916886620 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feccc1041a0 0x7feccc19c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:45.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7feccb7fe700 1 -- 172.21.15.70:0/3916886620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feccc1088f0 msgr2=0x7feccc1a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:45.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7feccb7fe700 1 --2- 172.21.15.70:0/3916886620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feccc1088f0 0x7feccc1a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:45.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7feccb7fe700 1 -- 172.21.15.70:0/3916886620 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feccc1a0d60 con 0x7feccc1037f0 2024-06-04T04:54:45.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7feccaffd700 1 --2- 172.21.15.70:0/3916886620 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feccc1041a0 0x7feccc19c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:54:45.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7feccbfff700 1 --2- 172.21.15.70:0/3916886620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feccc1088f0 0x7feccc1a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:54:45.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.982+0000 7feccb7fe700 1 --2- 172.21.15.70:0/3916886620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feccc1037f0 0x7feccc19bd50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fecbc006010 tx=0x7fecbc004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:45.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.983+0000 7fecc8ff9700 1 -- 172.21.15.70:0/3916886620 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc026070 con 0x7feccc1037f0 2024-06-04T04:54:45.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.983+0000 7fecc8ff9700 1 -- 172.21.15.70:0/3916886620 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fecbc00f460 con 0x7feccc1037f0 2024-06-04T04:54:45.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.983+0000 7fecd1c37700 1 -- 172.21.15.70:0/3916886620 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feccc1a0ff0 con 0x7feccc1037f0 2024-06-04T04:54:45.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.983+0000 7fecc8ff9700 1 -- 172.21.15.70:0/3916886620 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc009e60 con 0x7feccc1037f0 2024-06-04T04:54:45.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.983+0000 7fecd1c37700 1 -- 172.21.15.70:0/3916886620 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feccc1a88d0 con 0x7feccc1037f0 2024-06-04T04:54:45.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.984+0000 7fecd1c37700 1 -- 172.21.15.70:0/3916886620 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feccc06a230 con 0x7feccc1037f0 2024-06-04T04:54:45.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.985+0000 7fecc8ff9700 1 -- 172.21.15.70:0/3916886620 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fecbc029070 con 0x7feccc1037f0 2024-06-04T04:54:45.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.986+0000 7fecc8ff9700 1 --2- 172.21.15.70:0/3916886620 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fecb40b2910 0x7fecb40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:45.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.986+0000 7feccaffd700 1 --2- 172.21.15.70:0/3916886620 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fecb40b2910 0x7fecb40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:45.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.986+0000 7fecc8ff9700 1 -- 172.21.15.70:0/3916886620 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fecbc0e2f60 con 0x7feccc1037f0 2024-06-04T04:54:45.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.986+0000 7feccaffd700 1 --2- 172.21.15.70:0/3916886620 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fecb40b2910 0x7fecb40b4dd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7feccc103370 tx=0x7fecc0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:45.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:45.989+0000 7fecc8ff9700 1 -- 172.21.15.70:0/3916886620 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fecbc030080 con 0x7feccc1037f0 2024-06-04T04:54:46.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:46.194+0000 7fecd1c37700 1 -- 172.21.15.70:0/3916886620 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feccc1a8aa0 con 0x7feccc1037f0 2024-06-04T04:54:46.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:46.196+0000 7fecc8ff9700 1 -- 172.21.15.70:0/3916886620 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22708 (secure 0 0 0) 0x7fecbc013070 con 0x7feccc1037f0 2024-06-04T04:54:46.199 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:54:46.199 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":81,"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","created":"2024-06-04T04:45:22.297536+0000","modified":"2024-06-04T04:54:19.502410+0000","last_up_change":"2024-06-04T04:52:34.388708+0000","last_in_change":"2024-06-04T04:52:14.197780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-04T04:48:43.724356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-04T04:53:31.398455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-04T04:53:35.298617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ff491ea2-1906-48e4-883e-bc78e778260e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6803","nonce":824119100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6805","nonce":824119100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6809","nonce":824119100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6807","nonce":824119100}]},"public_addr":"172.21.15.70:6803/824119100","cluster_addr":"172.21.15.70:6805/824119100","heartbeat_back_addr":"172.21.15.70:6809/824119100","heartbeat_front_addr":"172.21.15.70:6807/824119100","state":["exists","up"]},{"osd":1,"uuid":"bd5e4806-535b-49b5-ba6b-93228da35a84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6801","nonce":2744598742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6803","nonce":2744598742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6807","nonce":2744598742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6805","nonce":2744598742}]},"public_addr":"172.21.15.98:6801/2744598742","cluster_addr":"172.21.15.98:6803/2744598742","heartbeat_back_addr":"172.21.15.98:6807/2744598742","heartbeat_front_addr":"172.21.15.98:6805/2744598742","state":["exists","up"]},{"osd":2,"uuid":"8456fb55-910a-4a59-b84b-2c0f68346ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6801","nonce":1328098370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6803","nonce":1328098370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6807","nonce":1328098370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6805","nonce":1328098370}]},"public_addr":"172.21.15.112:6801/1328098370","cluster_addr":"172.21.15.112:6803/1328098370","heartbeat_back_addr":"172.21.15.112:6807/1328098370","heartbeat_front_addr":"172.21.15.112:6805/1328098370","state":["exists","up"]},{"osd":3,"uuid":"c8fbc440-bf70-42d3-b01e-777ccfa7900b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6811","nonce":4041566346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6813","nonce":4041566346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6817","nonce":4041566346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6815","nonce":4041566346}]},"public_addr":"172.21.15.70:6811/4041566346","cluster_addr":"172.21.15.70:6813/4041566346","heartbeat_back_addr":"172.21.15.70:6817/4041566346","heartbeat_front_addr":"172.21.15.70:6815/4041566346","state":["exists","up"]},{"osd":4,"uuid":"549c8c70-8be8-46ae-bcb6-0fde32b40cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6809","nonce":3652775814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6811","nonce":3652775814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6815","nonce":3652775814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6813","nonce":3652775814}]},"public_addr":"172.21.15.98:6809/3652775814","cluster_addr":"172.21.15.98:6811/3652775814","heartbeat_back_addr":"172.21.15.98:6815/3652775814","heartbeat_front_addr":"172.21.15.98:6813/3652775814","state":["exists","up"]},{"osd":5,"uuid":"51d86af1-0d6c-4ae0-b090-00414fc1bf7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6809","nonce":1857257899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6811","nonce":1857257899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6815","nonce":1857257899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6813","nonce":1857257899}]},"public_addr":"172.21.15.112:6809/1857257899","cluster_addr":"172.21.15.112:6811/1857257899","heartbeat_back_addr":"172.21.15.112:6815/1857257899","heartbeat_front_addr":"172.21.15.112:6813/1857257899","state":["exists","up"]},{"osd":6,"uuid":"526a8138-bfa8-419e-a5b5-e6b80f0ae004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6819","nonce":1264676496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6821","nonce":1264676496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6825","nonce":1264676496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6823","nonce":1264676496}]},"public_addr":"172.21.15.70:6819/1264676496","cluster_addr":"172.21.15.70:6821/1264676496","heartbeat_back_addr":"172.21.15.70:6825/1264676496","heartbeat_front_addr":"172.21.15.70:6823/1264676496","state":["exists","up"]},{"osd":7,"uuid":"1173f785-6ed1-47ef-9265-2e233ea9ee02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6817","nonce":709593020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6819","nonce":709593020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6823","nonce":709593020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6821","nonce":709593020}]},"public_addr":"172.21.15.98:6817/709593020","cluster_addr":"172.21.15.98:6819/709593020","heartbeat_back_addr":"172.21.15.98:6823/709593020","heartbeat_front_addr":"172.21.15.98:6821/709593020","state":["exists","up"]},{"osd":8,"uuid":"6424cc6e-24a7-4b81-9ce6-39203fecc7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6817","nonce":2250225413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6819","nonce":2250225413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6823","nonce":2250225413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6821","nonce":2250225413}]},"public_addr":"172.21.15.112:6817/2250225413","cluster_addr":"172.21.15.112:6819/2250225413","heartbeat_back_addr":"172.21.15.112:6823/2250225413","heartbeat_front_addr":"172.21.15.112:6821/2250225413","state":["exists","up"]},{"osd":9,"uuid":"8d666db8-8831-4d63-9e3c-a4bf2504149e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6827","nonce":3557688707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6829","nonce":3557688707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6833","nonce":3557688707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6831","nonce":3557688707}]},"public_addr":"172.21.15.70:6827/3557688707","cluster_addr":"172.21.15.70:6829/3557688707","heartbeat_back_addr":"172.21.15.70:6833/3557688707","heartbeat_front_addr":"172.21.15.70:6831/3557688707","state":["exists","up"]},{"osd":10,"uuid":"ecb925d2-0263-4dfc-a2fd-9e27856022d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6825","nonce":1896493718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6827","nonce":1896493718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6831","nonce":1896493718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6829","nonce":1896493718}]},"public_addr":"172.21.15.98:6825/1896493718","cluster_addr":"172.21.15.98:6827/1896493718","heartbeat_back_addr":"172.21.15.98:6831/1896493718","heartbeat_front_addr":"172.21.15.98:6829/1896493718","state":["exists","up"]},{"osd":11,"uuid":"c8e3230c-bc3a-40bd-8dd0-5fef56e4df97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6825","nonce":1400793137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6827","nonce":1400793137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6831","nonce":1400793137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6829","nonce":1400793137}]},"public_addr":"172.21.15.112:6825/1400793137","cluster_addr":"172.21.15.112:6827/1400793137","heartbeat_back_addr":"172.21.15.112:6831/1400793137","heartbeat_front_addr":"172.21.15.112:6829/1400793137","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:47:53.626608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:17.784505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:41.971969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:07.130699+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:31.615496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:56.898607+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:22.739991+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:48.319727+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:13.902063+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:40.389470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:06.142092+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:32.949383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/3442074771":"2024-06-05T04:46:17.636907+0000","172.21.15.70:6801/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/1442095493":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/2453733268":"2024-06-05T04:45:49.501864+0000","172.21.15.70:0/1369781447":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6801/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/915186456":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/860033006":"2024-06-05T04:45:49.501864+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-04T04:54:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:46.198+0000 7fecd1c37700 1 -- 172.21.15.70:0/3916886620 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fecb40b2910 msgr2=0x7fecb40b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:46.198+0000 7fecd1c37700 1 --2- 172.21.15.70:0/3916886620 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fecb40b2910 0x7fecb40b4dd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7feccc103370 tx=0x7fecc0008040 comp rx=0 tx=0).stop 2024-06-04T04:54:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:46.198+0000 7fecd1c37700 1 -- 172.21.15.70:0/3916886620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feccc1037f0 msgr2=0x7feccc19bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:46.198+0000 7fecd1c37700 1 --2- 172.21.15.70:0/3916886620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feccc1037f0 0x7feccc19bd50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fecbc006010 tx=0x7fecbc004690 comp rx=0 tx=0).stop 2024-06-04T04:54:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:46.198+0000 7fecd1c37700 1 -- 172.21.15.70:0/3916886620 shutdown_connections 2024-06-04T04:54:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:46.198+0000 7fecd1c37700 1 --2- 172.21.15.70:0/3916886620 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fecb40b2910 0x7fecb40b4dd0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:46.198+0000 7fecd1c37700 1 --2- 172.21.15.70:0/3916886620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feccc1088f0 0x7feccc1a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:46.198+0000 7fecd1c37700 1 --2- 172.21.15.70:0/3916886620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feccc1037f0 0x7feccc19bd50 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:46.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:46.198+0000 7fecd1c37700 1 --2- 172.21.15.70:0/3916886620 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feccc1041a0 0x7feccc19c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:46.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:46.198+0000 7fecd1c37700 1 -- 172.21.15.70:0/3916886620 >> 172.21.15.70:0/3916886620 conn(0x7feccc07a6a0 msgr2=0x7feccc101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:46.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:46.198+0000 7fecd1c37700 1 -- 172.21.15.70:0/3916886620 shutdown_connections 2024-06-04T04:54:46.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:46.198+0000 7fecd1c37700 1 -- 172.21.15.70:0/3916886620 wait complete. 2024-06-04T04:54:46.929 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-04T04:54:46.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-04T04:54:47.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:46 smithi070 ceph-mon[32631]: pgmap v325: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:54:47.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:46 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3916886620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:54:47.172 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:47.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:46 smithi112 ceph-mon[36453]: pgmap v325: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:54:47.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:46 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3916886620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:54:47.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:46 smithi098 ceph-mon[35569]: pgmap v325: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:54:47.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:46 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3916886620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:54:48.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.838+0000 7f3514503700 1 -- 172.21.15.70:0/1782761065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f350c103850 msgr2=0x7f350c103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:48.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.838+0000 7f3514503700 1 --2- 172.21.15.70:0/1782761065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f350c103850 0x7f350c103c30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f35080099c0 tx=0x7f3508009cd0 comp rx=0 tx=0).stop 2024-06-04T04:54:48.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.838+0000 7f3514503700 1 -- 172.21.15.70:0/1782761065 shutdown_connections 2024-06-04T04:54:48.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.838+0000 7f3514503700 1 --2- 172.21.15.70:0/1782761065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f350c108920 0x7f350c1104a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:48.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.838+0000 7f3514503700 1 --2- 172.21.15.70:0/1782761065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f350c103850 0x7f350c103c30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:48.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.838+0000 7f3514503700 1 --2- 172.21.15.70:0/1782761065 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f350c104200 0x7f350c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:48.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.838+0000 7f3514503700 1 -- 172.21.15.70:0/1782761065 >> 172.21.15.70:0/1782761065 conn(0x7f350c07a5d0 msgr2=0x7f350c07a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:48.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.839+0000 7f3514503700 1 -- 172.21.15.70:0/1782761065 shutdown_connections 2024-06-04T04:54:48.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.839+0000 7f3514503700 1 -- 172.21.15.70:0/1782761065 wait complete. 2024-06-04T04:54:48.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.839+0000 7f3514503700 1 Processor -- start 2024-06-04T04:54:48.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.839+0000 7f3514503700 1 -- start start 2024-06-04T04:54:48.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.839+0000 7f3514503700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f350c103850 0x7f350c19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:48.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.839+0000 7f3514503700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f350c104200 0x7f350c19fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:48.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.839+0000 7f3514503700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f350c108920 0x7f350c199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:48.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.839+0000 7f3514503700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f350c1135b0 con 0x7f350c108920 2024-06-04T04:54:48.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.839+0000 7f3514503700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f350c113430 con 0x7f350c104200 2024-06-04T04:54:48.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.839+0000 7f3514503700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f350c113730 con 0x7f350c103850 2024-06-04T04:54:48.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.839+0000 7f3512aa0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f350c108920 0x7f350c199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:48.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.840+0000 7f3512aa0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f350c108920 0x7f350c199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43806/0 (socket says 172.21.15.70:43806) 2024-06-04T04:54:48.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.840+0000 7f3512aa0700 1 -- 172.21.15.70:0/2872497895 learned_addr learned my addr 172.21.15.70:0/2872497895 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:48.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.840+0000 7f351229f700 1 --2- 172.21.15.70:0/2872497895 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f350c103850 0x7f350c19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:48.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.840+0000 7f3512aa0700 1 -- 172.21.15.70:0/2872497895 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f350c103850 msgr2=0x7f350c19f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:48.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.840+0000 7f3512aa0700 1 --2- 172.21.15.70:0/2872497895 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f350c103850 0x7f350c19f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:48.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.840+0000 7f3512aa0700 1 -- 172.21.15.70:0/2872497895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f350c104200 msgr2=0x7f350c19fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:48.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.840+0000 7f3512aa0700 1 --2- 172.21.15.70:0/2872497895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f350c104200 0x7f350c19fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:48.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.840+0000 7f3512aa0700 1 -- 172.21.15.70:0/2872497895 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350c199ff0 con 0x7f350c108920 2024-06-04T04:54:48.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.840+0000 7f3512aa0700 1 --2- 172.21.15.70:0/2872497895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f350c108920 0x7f350c199810 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f350400ada0 tx=0x7f350400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:48.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.840+0000 7f35037fe700 1 -- 172.21.15.70:0/2872497895 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3504017020 con 0x7f350c108920 2024-06-04T04:54:48.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.840+0000 7f35037fe700 1 -- 172.21.15.70:0/2872497895 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f350400ce10 con 0x7f350c108920 2024-06-04T04:54:48.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.840+0000 7f35037fe700 1 -- 172.21.15.70:0/2872497895 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3504005440 con 0x7f350c108920 2024-06-04T04:54:48.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.841+0000 7f3514503700 1 -- 172.21.15.70:0/2872497895 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f350c19a2e0 con 0x7f350c108920 2024-06-04T04:54:48.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.841+0000 7f3514503700 1 -- 172.21.15.70:0/2872497895 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f350c1a64f0 con 0x7f350c108920 2024-06-04T04:54:48.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.841+0000 7f3514503700 1 -- 172.21.15.70:0/2872497895 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f350c10bb60 con 0x7f350c108920 2024-06-04T04:54:48.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.843+0000 7f35037fe700 1 -- 172.21.15.70:0/2872497895 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f3504004ad0 con 0x7f350c108920 2024-06-04T04:54:48.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.843+0000 7f35037fe700 1 --2- 172.21.15.70:0/2872497895 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f34f80b2910 0x7f34f80b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:48.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.844+0000 7f351229f700 1 --2- 172.21.15.70:0/2872497895 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f34f80b2910 0x7f34f80b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:48.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.844+0000 7f35037fe700 1 -- 172.21.15.70:0/2872497895 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f35040e1f10 con 0x7f350c108920 2024-06-04T04:54:48.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.844+0000 7f351229f700 1 --2- 172.21.15.70:0/2872497895 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f34f80b2910 0x7f34f80b4dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f35080058e0 tx=0x7f350801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:48.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:48.847+0000 7f35037fe700 1 -- 172.21.15.70:0/2872497895 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3504021060 con 0x7f350c108920 2024-06-04T04:54:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:49.051+0000 7f3514503700 1 -- 172.21.15.70:0/2872497895 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f350c10b540 con 0x7f350c108920 2024-06-04T04:54:49.053 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:48 smithi070 ceph-mon[32631]: pgmap v326: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:54:49.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:49.053+0000 7f35037fe700 1 -- 172.21.15.70:0/2872497895 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22708 (secure 0 0 0) 0x7f35040a9b40 con 0x7f350c108920 2024-06-04T04:54:49.055 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:54:49.055 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":81,"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","created":"2024-06-04T04:45:22.297536+0000","modified":"2024-06-04T04:54:19.502410+0000","last_up_change":"2024-06-04T04:52:34.388708+0000","last_in_change":"2024-06-04T04:52:14.197780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-04T04:48:43.724356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-04T04:53:31.398455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-04T04:53:35.298617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ff491ea2-1906-48e4-883e-bc78e778260e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6803","nonce":824119100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6805","nonce":824119100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6809","nonce":824119100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6807","nonce":824119100}]},"public_addr":"172.21.15.70:6803/824119100","cluster_addr":"172.21.15.70:6805/824119100","heartbeat_back_addr":"172.21.15.70:6809/824119100","heartbeat_front_addr":"172.21.15.70:6807/824119100","state":["exists","up"]},{"osd":1,"uuid":"bd5e4806-535b-49b5-ba6b-93228da35a84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6801","nonce":2744598742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6803","nonce":2744598742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6807","nonce":2744598742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6805","nonce":2744598742}]},"public_addr":"172.21.15.98:6801/2744598742","cluster_addr":"172.21.15.98:6803/2744598742","heartbeat_back_addr":"172.21.15.98:6807/2744598742","heartbeat_front_addr":"172.21.15.98:6805/2744598742","state":["exists","up"]},{"osd":2,"uuid":"8456fb55-910a-4a59-b84b-2c0f68346ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6801","nonce":1328098370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6803","nonce":1328098370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6807","nonce":1328098370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6805","nonce":1328098370}]},"public_addr":"172.21.15.112:6801/1328098370","cluster_addr":"172.21.15.112:6803/1328098370","heartbeat_back_addr":"172.21.15.112:6807/1328098370","heartbeat_front_addr":"172.21.15.112:6805/1328098370","state":["exists","up"]},{"osd":3,"uuid":"c8fbc440-bf70-42d3-b01e-777ccfa7900b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6811","nonce":4041566346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6813","nonce":4041566346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6817","nonce":4041566346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6815","nonce":4041566346}]},"public_addr":"172.21.15.70:6811/4041566346","cluster_addr":"172.21.15.70:6813/4041566346","heartbeat_back_addr":"172.21.15.70:6817/4041566346","heartbeat_front_addr":"172.21.15.70:6815/4041566346","state":["exists","up"]},{"osd":4,"uuid":"549c8c70-8be8-46ae-bcb6-0fde32b40cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6809","nonce":3652775814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6811","nonce":3652775814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6815","nonce":3652775814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6813","nonce":3652775814}]},"public_addr":"172.21.15.98:6809/3652775814","cluster_addr":"172.21.15.98:6811/3652775814","heartbeat_back_addr":"172.21.15.98:6815/3652775814","heartbeat_front_addr":"172.21.15.98:6813/3652775814","state":["exists","up"]},{"osd":5,"uuid":"51d86af1-0d6c-4ae0-b090-00414fc1bf7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6809","nonce":1857257899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6811","nonce":1857257899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6815","nonce":1857257899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6813","nonce":1857257899}]},"public_addr":"172.21.15.112:6809/1857257899","cluster_addr":"172.21.15.112:6811/1857257899","heartbeat_back_addr":"172.21.15.112:6815/1857257899","heartbeat_front_addr":"172.21.15.112:6813/1857257899","state":["exists","up"]},{"osd":6,"uuid":"526a8138-bfa8-419e-a5b5-e6b80f0ae004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6819","nonce":1264676496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6821","nonce":1264676496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6825","nonce":1264676496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6823","nonce":1264676496}]},"public_addr":"172.21.15.70:6819/1264676496","cluster_addr":"172.21.15.70:6821/1264676496","heartbeat_back_addr":"172.21.15.70:6825/1264676496","heartbeat_front_addr":"172.21.15.70:6823/1264676496","state":["exists","up"]},{"osd":7,"uuid":"1173f785-6ed1-47ef-9265-2e233ea9ee02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6817","nonce":709593020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6819","nonce":709593020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6823","nonce":709593020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6821","nonce":709593020}]},"public_addr":"172.21.15.98:6817/709593020","cluster_addr":"172.21.15.98:6819/709593020","heartbeat_back_addr":"172.21.15.98:6823/709593020","heartbeat_front_addr":"172.21.15.98:6821/709593020","state":["exists","up"]},{"osd":8,"uuid":"6424cc6e-24a7-4b81-9ce6-39203fecc7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6817","nonce":2250225413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6819","nonce":2250225413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6823","nonce":2250225413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6821","nonce":2250225413}]},"public_addr":"172.21.15.112:6817/2250225413","cluster_addr":"172.21.15.112:6819/2250225413","heartbeat_back_addr":"172.21.15.112:6823/2250225413","heartbeat_front_addr":"172.21.15.112:6821/2250225413","state":["exists","up"]},{"osd":9,"uuid":"8d666db8-8831-4d63-9e3c-a4bf2504149e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6827","nonce":3557688707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6829","nonce":3557688707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6833","nonce":3557688707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6831","nonce":3557688707}]},"public_addr":"172.21.15.70:6827/3557688707","cluster_addr":"172.21.15.70:6829/3557688707","heartbeat_back_addr":"172.21.15.70:6833/3557688707","heartbeat_front_addr":"172.21.15.70:6831/3557688707","state":["exists","up"]},{"osd":10,"uuid":"ecb925d2-0263-4dfc-a2fd-9e27856022d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6825","nonce":1896493718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6827","nonce":1896493718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6831","nonce":1896493718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6829","nonce":1896493718}]},"public_addr":"172.21.15.98:6825/1896493718","cluster_addr":"172.21.15.98:6827/1896493718","heartbeat_back_addr":"172.21.15.98:6831/1896493718","heartbeat_front_addr":"172.21.15.98:6829/1896493718","state":["exists","up"]},{"osd":11,"uuid":"c8e3230c-bc3a-40bd-8dd0-5fef56e4df97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6825","nonce":1400793137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6827","nonce":1400793137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6831","nonce":1400793137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6829","nonce":1400793137}]},"public_addr":"172.21.15.112:6825/1400793137","cluster_addr":"172.21.15.112:6827/1400793137","heartbeat_back_addr":"172.21.15.112:6831/1400793137","heartbeat_front_addr":"172.21.15.112:6829/1400793137","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:47:53.626608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:17.784505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:41.971969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:07.130699+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:31.615496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:56.898607+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:22.739991+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:48.319727+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:13.902063+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:40.389470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:06.142092+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:32.949383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/3442074771":"2024-06-05T04:46:17.636907+0000","172.21.15.70:6801/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/1442095493":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/2453733268":"2024-06-05T04:45:49.501864+0000","172.21.15.70:0/1369781447":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6801/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/915186456":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/860033006":"2024-06-05T04:45:49.501864+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-04T04:54:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:49.055+0000 7f3514503700 1 -- 172.21.15.70:0/2872497895 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f34f80b2910 msgr2=0x7f34f80b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:49.055+0000 7f3514503700 1 --2- 172.21.15.70:0/2872497895 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f34f80b2910 0x7f34f80b4dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f35080058e0 tx=0x7f350801a040 comp rx=0 tx=0).stop 2024-06-04T04:54:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:49.055+0000 7f3514503700 1 -- 172.21.15.70:0/2872497895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f350c108920 msgr2=0x7f350c199810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:49.055+0000 7f3514503700 1 --2- 172.21.15.70:0/2872497895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f350c108920 0x7f350c199810 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f350400ada0 tx=0x7f350400c330 comp rx=0 tx=0).stop 2024-06-04T04:54:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:49.055+0000 7f3514503700 1 -- 172.21.15.70:0/2872497895 shutdown_connections 2024-06-04T04:54:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:49.055+0000 7f3514503700 1 --2- 172.21.15.70:0/2872497895 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f34f80b2910 0x7f34f80b4dd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:49.055+0000 7f3514503700 1 --2- 172.21.15.70:0/2872497895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f350c108920 0x7f350c199810 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:49.055+0000 7f3514503700 1 --2- 172.21.15.70:0/2872497895 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f350c103850 0x7f350c19f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:49.055+0000 7f3514503700 1 --2- 172.21.15.70:0/2872497895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f350c104200 0x7f350c19fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:49.055+0000 7f3514503700 1 -- 172.21.15.70:0/2872497895 >> 172.21.15.70:0/2872497895 conn(0x7f350c07a5d0 msgr2=0x7f350c109a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:49.055+0000 7f3514503700 1 -- 172.21.15.70:0/2872497895 shutdown_connections 2024-06-04T04:54:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:49.055+0000 7f3514503700 1 -- 172.21.15.70:0/2872497895 wait complete. 2024-06-04T04:54:49.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:48 smithi112 ceph-mon[36453]: pgmap v326: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:54:49.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:48 smithi098 ceph-mon[35569]: pgmap v326: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:54:49.710 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-04T04:54:49.710 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-04T04:54:49.711 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-04T04:54:49.711 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-04T04:54:49.711 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-04T04:54:49.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-04T04:54:49.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-06-04T04:54:49.713 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-06-04T04:54:49.713 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-06-04T04:54:49.714 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-06-04T04:54:49.714 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-06-04T04:54:49.715 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-06-04T04:54:50.111 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:49 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2872497895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:54:50.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:49 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2872497895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:54:50.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:49 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2872497895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:54:50.396 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:50.400 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:50.407 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:50.407 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:50.407 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:50.476 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:50.710 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:50.717 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:50.719 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:51.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:50 smithi070 ceph-mon[32631]: pgmap v327: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:54:51.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:50 smithi112 ceph-mon[36453]: pgmap v327: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:54:51.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:50 smithi098 ceph-mon[35569]: pgmap v327: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:54:53.072 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:52 smithi070 ceph-mon[32631]: pgmap v328: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:54:53.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:52 smithi112 ceph-mon[36453]: pgmap v328: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:54:53.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:52 smithi098 ceph-mon[35569]: pgmap v328: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:54:55.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:54 smithi070 ceph-mon[32631]: pgmap v329: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:54:55.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:54 smithi112 ceph-mon[36453]: pgmap v329: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:54:55.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:54 smithi098 ceph-mon[35569]: pgmap v329: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:54:55.694 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:54:56.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.339+0000 7f0eae38e700 1 -- 172.21.15.70:0/2539654832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ea81041a0 msgr2=0x7f0ea81081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:56.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.339+0000 7f0eae38e700 1 --2- 172.21.15.70:0/2539654832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ea81041a0 0x7f0ea81081c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0099c0 tx=0x7f0e9c009cd0 comp rx=0 tx=0).stop 2024-06-04T04:54:56.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.339+0000 7f0eae38e700 1 -- 172.21.15.70:0/2539654832 shutdown_connections 2024-06-04T04:54:56.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.339+0000 7f0eae38e700 1 --2- 172.21.15.70:0/2539654832 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ea81088f0 0x7f0ea8110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:56.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.339+0000 7f0eae38e700 1 --2- 172.21.15.70:0/2539654832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ea81037f0 0x7f0ea8103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:56.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.339+0000 7f0eae38e700 1 --2- 172.21.15.70:0/2539654832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ea81041a0 0x7f0ea81081c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:56.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.339+0000 7f0eae38e700 1 -- 172.21.15.70:0/2539654832 >> 172.21.15.70:0/2539654832 conn(0x7f0ea807a6a0 msgr2=0x7f0ea807aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:56.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.339+0000 7f0eae38e700 1 -- 172.21.15.70:0/2539654832 shutdown_connections 2024-06-04T04:54:56.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.339+0000 7f0eae38e700 1 -- 172.21.15.70:0/2539654832 wait complete. 2024-06-04T04:54:56.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.340+0000 7f0eae38e700 1 Processor -- start 2024-06-04T04:54:56.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.340+0000 7f0eae38e700 1 -- start start 2024-06-04T04:54:56.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.340+0000 7f0eae38e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ea81037f0 0x7f0ea819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:56.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.340+0000 7f0eae38e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ea81041a0 0x7f0ea819fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:56.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.340+0000 7f0eae38e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ea81088f0 0x7f0ea81999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:56.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.340+0000 7f0eae38e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea8113660 con 0x7f0ea81041a0 2024-06-04T04:54:56.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.340+0000 7f0eae38e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea81134e0 con 0x7f0ea81088f0 2024-06-04T04:54:56.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.340+0000 7f0eae38e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea81137e0 con 0x7f0ea81037f0 2024-06-04T04:54:56.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.340+0000 7f0ea77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ea81041a0 0x7f0ea819fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:56.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.341+0000 7f0ea77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ea81041a0 0x7f0ea819fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57938/0 (socket says 172.21.15.70:57938) 2024-06-04T04:54:56.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.341+0000 7f0ea77fe700 1 -- 172.21.15.70:0/2713244991 learned_addr learned my addr 172.21.15.70:0/2713244991 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:56.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.341+0000 7f0ea7fff700 1 --2- 172.21.15.70:0/2713244991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ea81037f0 0x7f0ea819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:56.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.341+0000 7f0eac92b700 1 --2- 172.21.15.70:0/2713244991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ea81088f0 0x7f0ea81999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:56.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.341+0000 7f0ea77fe700 1 -- 172.21.15.70:0/2713244991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ea81037f0 msgr2=0x7f0ea819f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:56.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.341+0000 7f0ea77fe700 1 --2- 172.21.15.70:0/2713244991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ea81037f0 0x7f0ea819f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:56.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.341+0000 7f0ea77fe700 1 -- 172.21.15.70:0/2713244991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ea81088f0 msgr2=0x7f0ea81999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:56.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.341+0000 7f0ea77fe700 1 --2- 172.21.15.70:0/2713244991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ea81088f0 0x7f0ea81999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:56.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.341+0000 7f0ea77fe700 1 -- 172.21.15.70:0/2713244991 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea819a100 con 0x7f0ea81041a0 2024-06-04T04:54:56.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.341+0000 7f0ea77fe700 1 --2- 172.21.15.70:0/2713244991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ea81041a0 0x7f0ea819fd20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c000c00 tx=0x7f0e9c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:56.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.341+0000 7f0ea57fa700 1 -- 172.21.15.70:0/2713244991 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c026070 con 0x7f0ea81041a0 2024-06-04T04:54:56.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.341+0000 7f0ea57fa700 1 -- 172.21.15.70:0/2713244991 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0e9c0054c0 con 0x7f0ea81041a0 2024-06-04T04:54:56.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.341+0000 7f0eae38e700 1 -- 172.21.15.70:0/2713244991 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ea819a390 con 0x7f0ea81041a0 2024-06-04T04:54:56.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.341+0000 7f0ea57fa700 1 -- 172.21.15.70:0/2713244991 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c004c20 con 0x7f0ea81041a0 2024-06-04T04:54:56.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.341+0000 7f0eae38e700 1 -- 172.21.15.70:0/2713244991 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ea81a66b0 con 0x7f0ea81041a0 2024-06-04T04:54:56.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.342+0000 7f0eae38e700 1 -- 172.21.15.70:0/2713244991 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0ea8063cd0 con 0x7f0ea81041a0 2024-06-04T04:54:56.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.343+0000 7f0ea57fa700 1 -- 172.21.15.70:0/2713244991 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f0e9c029070 con 0x7f0ea81041a0 2024-06-04T04:54:56.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.344+0000 7f0ea57fa700 1 --2- 172.21.15.70:0/2713244991 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0e900b2910 0x7f0e900b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:56.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.344+0000 7f0ea7fff700 1 --2- 172.21.15.70:0/2713244991 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0e900b2910 0x7f0e900b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:56.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.344+0000 7f0ea57fa700 1 -- 172.21.15.70:0/2713244991 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f0e9c0e36f0 con 0x7f0ea81041a0 2024-06-04T04:54:56.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.344+0000 7f0ea57fa700 1 --2- 172.21.15.70:0/2713244991 >> [v2:172.21.15.112:6800/1328098370,v1:172.21.15.112:6801/1328098370] conn(0x7f0e900bc550 0x7f0e900be9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:56.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.344+0000 7f0ea57fa700 1 -- 172.21.15.70:0/2713244991 --> [v2:172.21.15.112:6800/1328098370,v1:172.21.15.112:6801/1328098370] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0e9c00b0b0 con 0x7f0e900bc550 2024-06-04T04:54:56.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.344+0000 7f0ea57fa700 1 -- 172.21.15.70:0/2713244991 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f0e9c0e3b40 con 0x7f0ea81041a0 2024-06-04T04:54:56.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.345+0000 7f0ea7fff700 1 --2- 172.21.15.70:0/2713244991 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0e900b2910 0x7f0e900b4dd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0e98006fd0 tx=0x7f0e98008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:56.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.345+0000 7f0eac92b700 1 --2- 172.21.15.70:0/2713244991 >> [v2:172.21.15.112:6800/1328098370,v1:172.21.15.112:6801/1328098370] conn(0x7f0e900bc550 0x7f0e900be9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:56.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.345+0000 7f0eac92b700 1 --2- 172.21.15.70:0/2713244991 >> [v2:172.21.15.112:6800/1328098370,v1:172.21.15.112:6801/1328098370] conn(0x7f0e900bc550 0x7f0e900be9f0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:56.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.346+0000 7f0ea57fa700 1 -- 172.21.15.70:0/2713244991 <== osd.2 v2:172.21.15.112:6800/1328098370 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0e9c00b0b0 con 0x7f0e900bc550 2024-06-04T04:54:56.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.363+0000 7f0eae38e700 1 -- 172.21.15.70:0/2713244991 --> [v2:172.21.15.112:6800/1328098370,v1:172.21.15.112:6801/1328098370] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0ea8064150 con 0x7f0e900bc550 2024-06-04T04:54:56.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.363+0000 7f0ea57fa700 1 -- 172.21.15.70:0/2713244991 <== osd.2 v2:172.21.15.112:6800/1328098370 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0ea8064150 con 0x7f0e900bc550 2024-06-04T04:54:56.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.364+0000 7f0eae38e700 1 -- 172.21.15.70:0/2713244991 >> [v2:172.21.15.112:6800/1328098370,v1:172.21.15.112:6801/1328098370] conn(0x7f0e900bc550 msgr2=0x7f0e900be9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:56.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.364+0000 7f0eae38e700 1 --2- 172.21.15.70:0/2713244991 >> [v2:172.21.15.112:6800/1328098370,v1:172.21.15.112:6801/1328098370] conn(0x7f0e900bc550 0x7f0e900be9f0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:56.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.364+0000 7f0eae38e700 1 -- 172.21.15.70:0/2713244991 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0e900b2910 msgr2=0x7f0e900b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:56.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.364+0000 7f0eae38e700 1 --2- 172.21.15.70:0/2713244991 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0e900b2910 0x7f0e900b4dd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0e98006fd0 tx=0x7f0e98008040 comp rx=0 tx=0).stop 2024-06-04T04:54:56.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.364+0000 7f0eae38e700 1 -- 172.21.15.70:0/2713244991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ea81041a0 msgr2=0x7f0ea819fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:56.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.364+0000 7f0eae38e700 1 --2- 172.21.15.70:0/2713244991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ea81041a0 0x7f0ea819fd20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c000c00 tx=0x7f0e9c00f740 comp rx=0 tx=0).stop 2024-06-04T04:54:56.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.364+0000 7f0eac92b700 1 -- 172.21.15.70:0/2713244991 reap_dead start 2024-06-04T04:54:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.364+0000 7f0eae38e700 1 -- 172.21.15.70:0/2713244991 shutdown_connections 2024-06-04T04:54:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.364+0000 7f0eae38e700 1 -- 172.21.15.70:0/2713244991 >> 172.21.15.70:0/2713244991 conn(0x7f0ea807a6a0 msgr2=0x7f0ea8101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.364+0000 7f0eae38e700 1 -- 172.21.15.70:0/2713244991 shutdown_connections 2024-06-04T04:54:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:56.364+0000 7f0eae38e700 1 -- 172.21.15.70:0/2713244991 wait complete. 2024-06-04T04:54:57.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:56 smithi070 ceph-mon[32631]: pgmap v330: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:54:57.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:56 smithi112 ceph-mon[36453]: pgmap v330: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:54:57.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:56 smithi098 ceph-mon[35569]: pgmap v330: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:54:58.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.353+0000 7f9aa776c700 1 -- 172.21.15.70:0/2061066718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9aa0076090 msgr2=0x7f9aa01141d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:58.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.353+0000 7f9aa776c700 1 --2- 172.21.15.70:0/2061066718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9aa0076090 0x7f9aa01141d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009990 tx=0x7f9a90009ca0 comp rx=0 tx=0).stop 2024-06-04T04:54:58.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.353+0000 7f9aa776c700 1 -- 172.21.15.70:0/2061066718 shutdown_connections 2024-06-04T04:54:58.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.353+0000 7f9aa776c700 1 --2- 172.21.15.70:0/2061066718 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9aa0114710 0x7f9aa0116b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.353+0000 7f9aa776c700 1 --2- 172.21.15.70:0/2061066718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa0075770 0x7f9aa0075b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.353+0000 7f9aa776c700 1 --2- 172.21.15.70:0/2061066718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9aa0076090 0x7f9aa01141d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.353+0000 7f9aa776c700 1 -- 172.21.15.70:0/2061066718 >> 172.21.15.70:0/2061066718 conn(0x7f9aa007a6a0 msgr2=0x7f9aa007aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.354+0000 7f9aa776c700 1 -- 172.21.15.70:0/2061066718 shutdown_connections 2024-06-04T04:54:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.354+0000 7f9aa776c700 1 -- 172.21.15.70:0/2061066718 wait complete. 2024-06-04T04:54:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.354+0000 7f9aa776c700 1 Processor -- start 2024-06-04T04:54:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.354+0000 7f9aa776c700 1 -- start start 2024-06-04T04:54:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.354+0000 7f9aa776c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9aa0075770 0x7f9aa01a02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.354+0000 7f9aa776c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9aa0076090 0x7f9aa01a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.354+0000 7f9aa776c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa0114710 0x7f9aa01a4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.354+0000 7f9aa776c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa0119ca0 con 0x7f9aa0076090 2024-06-04T04:54:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.354+0000 7f9aa776c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa0119b20 con 0x7f9aa0114710 2024-06-04T04:54:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.354+0000 7f9aa776c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa0119e20 con 0x7f9aa0075770 2024-06-04T04:54:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.355+0000 7f9aa4d07700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9aa0076090 0x7f9aa01a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.355+0000 7f9aa4d07700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9aa0076090 0x7f9aa01a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57962/0 (socket says 172.21.15.70:57962) 2024-06-04T04:54:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.355+0000 7f9aa4d07700 1 -- 172.21.15.70:0/2546421391 learned_addr learned my addr 172.21.15.70:0/2546421391 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:54:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.355+0000 7f9aa5508700 1 --2- 172.21.15.70:0/2546421391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9aa0075770 0x7f9aa01a02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.355+0000 7f9aa4d07700 1 -- 172.21.15.70:0/2546421391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9aa0075770 msgr2=0x7f9aa01a02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.355+0000 7f9aa4d07700 1 --2- 172.21.15.70:0/2546421391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9aa0075770 0x7f9aa01a02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.355+0000 7f9aa4d07700 1 -- 172.21.15.70:0/2546421391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa0114710 msgr2=0x7f9aa01a4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.355+0000 7f9aa5d09700 1 --2- 172.21.15.70:0/2546421391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa0114710 0x7f9aa01a4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.355+0000 7f9aa4d07700 1 --2- 172.21.15.70:0/2546421391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa0114710 0x7f9aa01a4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.355+0000 7f9aa4d07700 1 -- 172.21.15.70:0/2546421391 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa01a52f0 con 0x7f9aa0076090 2024-06-04T04:54:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.355+0000 7f9aa4d07700 1 --2- 172.21.15.70:0/2546421391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9aa0076090 0x7f9aa01a0820 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9a900057f0 tx=0x7f9a90009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.355+0000 7f9a967fc700 1 -- 172.21.15.70:0/2546421391 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a90027070 con 0x7f9aa0076090 2024-06-04T04:54:58.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.355+0000 7f9a967fc700 1 -- 172.21.15.70:0/2546421391 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9a9000fd60 con 0x7f9aa0076090 2024-06-04T04:54:58.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.355+0000 7f9aa776c700 1 -- 172.21.15.70:0/2546421391 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa01a5580 con 0x7f9aa0076090 2024-06-04T04:54:58.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.355+0000 7f9a967fc700 1 -- 172.21.15.70:0/2546421391 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a90022d70 con 0x7f9aa0076090 2024-06-04T04:54:58.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.355+0000 7f9aa776c700 1 -- 172.21.15.70:0/2546421391 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa01ace60 con 0x7f9aa0076090 2024-06-04T04:54:58.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.356+0000 7f9aa776c700 1 -- 172.21.15.70:0/2546421391 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9aa0110bd0 con 0x7f9aa0076090 2024-06-04T04:54:58.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.357+0000 7f9a967fc700 1 -- 172.21.15.70:0/2546421391 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f9a9002a070 con 0x7f9aa0076090 2024-06-04T04:54:58.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.358+0000 7f9a967fc700 1 --2- 172.21.15.70:0/2546421391 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f9a8c0b2990 0x7f9a8c0b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.358+0000 7f9aa5508700 1 --2- 172.21.15.70:0/2546421391 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f9a8c0b2990 0x7f9a8c0b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.358+0000 7f9a967fc700 1 -- 172.21.15.70:0/2546421391 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f9a900e36c0 con 0x7f9aa0076090 2024-06-04T04:54:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.358+0000 7f9a967fc700 1 --2- 172.21.15.70:0/2546421391 >> [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346] conn(0x7f9a8c0bc5d0 0x7f9a8c0bea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:54:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.358+0000 7f9a967fc700 1 -- 172.21.15.70:0/2546421391 --> [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a8c0bf140 con 0x7f9a8c0bc5d0 2024-06-04T04:54:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.358+0000 7f9a967fc700 1 -- 172.21.15.70:0/2546421391 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f9a900e3b10 con 0x7f9aa0076090 2024-06-04T04:54:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.359+0000 7f9aa5d09700 1 --2- 172.21.15.70:0/2546421391 >> [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346] conn(0x7f9a8c0bc5d0 0x7f9a8c0bea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:54:58.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.359+0000 7f9aa5508700 1 --2- 172.21.15.70:0/2546421391 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f9a8c0b2990 0x7f9a8c0b4e50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c006fd0 tx=0x7f9a9c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:58.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.359+0000 7f9aa5d09700 1 --2- 172.21.15.70:0/2546421391 >> [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346] conn(0x7f9a8c0bc5d0 0x7f9a8c0bea70 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:54:58.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.360+0000 7f9a967fc700 1 -- 172.21.15.70:0/2546421391 <== osd.3 v2:172.21.15.70:6810/4041566346 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a8c0bf140 con 0x7f9a8c0bc5d0 2024-06-04T04:54:58.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.376+0000 7f9aa776c700 1 -- 172.21.15.70:0/2546421391 --> [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9aa0076e60 con 0x7f9a8c0bc5d0 2024-06-04T04:54:58.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.376+0000 7f9a967fc700 1 -- 172.21.15.70:0/2546421391 <== osd.3 v2:172.21.15.70:6810/4041566346 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9aa0076e60 con 0x7f9a8c0bc5d0 2024-06-04T04:54:58.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.377+0000 7f9aa776c700 1 -- 172.21.15.70:0/2546421391 >> [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346] conn(0x7f9a8c0bc5d0 msgr2=0x7f9a8c0bea70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:58.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.377+0000 7f9aa776c700 1 --2- 172.21.15.70:0/2546421391 >> [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346] conn(0x7f9a8c0bc5d0 0x7f9a8c0bea70 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:54:58.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.377+0000 7f9aa776c700 1 -- 172.21.15.70:0/2546421391 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f9a8c0b2990 msgr2=0x7f9a8c0b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:58.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.377+0000 7f9aa776c700 1 --2- 172.21.15.70:0/2546421391 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f9a8c0b2990 0x7f9a8c0b4e50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c006fd0 tx=0x7f9a9c008040 comp rx=0 tx=0).stop 2024-06-04T04:54:58.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.377+0000 7f9aa776c700 1 -- 172.21.15.70:0/2546421391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9aa0076090 msgr2=0x7f9aa01a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:54:58.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.377+0000 7f9aa776c700 1 --2- 172.21.15.70:0/2546421391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9aa0076090 0x7f9aa01a0820 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9a900057f0 tx=0x7f9a90009f90 comp rx=0 tx=0).stop 2024-06-04T04:54:58.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.377+0000 7f9aa5d09700 1 -- 172.21.15.70:0/2546421391 reap_dead start 2024-06-04T04:54:58.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.377+0000 7f9aa776c700 1 -- 172.21.15.70:0/2546421391 shutdown_connections 2024-06-04T04:54:58.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.377+0000 7f9aa776c700 1 -- 172.21.15.70:0/2546421391 >> 172.21.15.70:0/2546421391 conn(0x7f9aa007a6a0 msgr2=0x7f9aa01101b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:54:58.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.377+0000 7f9aa776c700 1 -- 172.21.15.70:0/2546421391 shutdown_connections 2024-06-04T04:54:58.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:54:58.377+0000 7f9aa776c700 1 -- 172.21.15.70:0/2546421391 wait complete. 2024-06-04T04:54:59.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:54:58 smithi070 ceph-mon[32631]: pgmap v331: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-04T04:54:59.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:54:58 smithi112 ceph-mon[36453]: pgmap v331: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-04T04:54:59.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:54:58 smithi098 ceph-mon[35569]: pgmap v331: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-04T04:55:00.019 INFO:teuthology.orchestra.run.smithi070.stdout:111669149769 2024-06-04T04:55:00.020 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-04T04:55:01.092 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:00 smithi070 ceph-mon[32631]: pgmap v332: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:55:01.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:00 smithi112 ceph-mon[36453]: pgmap v332: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:55:01.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:00 smithi098 ceph-mon[35569]: pgmap v332: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:55:03.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:02 smithi070 ceph-mon[32631]: pgmap v333: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:55:03.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:02 smithi112 ceph-mon[36453]: pgmap v333: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:55:03.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:02 smithi098 ceph-mon[35569]: pgmap v333: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:55:04.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.078+0000 7febf6c7b700 1 -- 172.21.15.70:0/2932508630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf01035e0 msgr2=0x7febf0103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:04.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.078+0000 7febf6c7b700 1 --2- 172.21.15.70:0/2932508630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf01035e0 0x7febf0103a60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7febdc0099c0 tx=0x7febdc009cd0 comp rx=0 tx=0).stop 2024-06-04T04:55:04.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.078+0000 7febf6c7b700 1 -- 172.21.15.70:0/2932508630 shutdown_connections 2024-06-04T04:55:04.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.078+0000 7febf6c7b700 1 --2- 172.21.15.70:0/2932508630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf0103fa0 0x7febf0110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:04.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.078+0000 7febf6c7b700 1 --2- 172.21.15.70:0/2932508630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7febf0109680 0x7febf0109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:04.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.078+0000 7febf6c7b700 1 --2- 172.21.15.70:0/2932508630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf01035e0 0x7febf0103a60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:04.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.078+0000 7febf6c7b700 1 -- 172.21.15.70:0/2932508630 >> 172.21.15.70:0/2932508630 conn(0x7febf007a6f0 msgr2=0x7febf007ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.078+0000 7febf6c7b700 1 -- 172.21.15.70:0/2932508630 shutdown_connections 2024-06-04T04:55:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.078+0000 7febf6c7b700 1 -- 172.21.15.70:0/2932508630 wait complete. 2024-06-04T04:55:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.078+0000 7febf6c7b700 1 Processor -- start 2024-06-04T04:55:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf6c7b700 1 -- start start 2024-06-04T04:55:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf6c7b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf01035e0 0x7febf019f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf6c7b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf0103fa0 0x7febf019fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf6c7b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7febf0109680 0x7febf0199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf6c7b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7febf0113690 con 0x7febf01035e0 2024-06-04T04:55:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf6c7b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7febf0113510 con 0x7febf0103fa0 2024-06-04T04:55:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf6c7b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7febf0113810 con 0x7febf0109680 2024-06-04T04:55:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf4a17700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf01035e0 0x7febf019f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf4a17700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf01035e0 0x7febf019f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57980/0 (socket says 172.21.15.70:57980) 2024-06-04T04:55:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf4a17700 1 -- 172.21.15.70:0/4078515852 learned_addr learned my addr 172.21.15.70:0/4078515852 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febeffff700 1 --2- 172.21.15.70:0/4078515852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf0103fa0 0x7febf019fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf5218700 1 --2- 172.21.15.70:0/4078515852 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7febf0109680 0x7febf0199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf4a17700 1 -- 172.21.15.70:0/4078515852 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7febf0109680 msgr2=0x7febf0199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf4a17700 1 --2- 172.21.15.70:0/4078515852 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7febf0109680 0x7febf0199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf4a17700 1 -- 172.21.15.70:0/4078515852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf0103fa0 msgr2=0x7febf019fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf4a17700 1 --2- 172.21.15.70:0/4078515852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf0103fa0 0x7febf019fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf4a17700 1 -- 172.21.15.70:0/4078515852 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febf019a0f0 con 0x7febf01035e0 2024-06-04T04:55:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf5218700 1 --2- 172.21.15.70:0/4078515852 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7febf0109680 0x7febf0199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:55:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf4a17700 1 --2- 172.21.15.70:0/4078515852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf01035e0 0x7febf019f7d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7febe400c920 tx=0x7febe400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febedffb700 1 -- 172.21.15.70:0/4078515852 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe4007b00 con 0x7febf01035e0 2024-06-04T04:55:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febedffb700 1 -- 172.21.15.70:0/4078515852 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7febe4004cc0 con 0x7febf01035e0 2024-06-04T04:55:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febf6c7b700 1 -- 172.21.15.70:0/4078515852 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febf019a3e0 con 0x7febf01035e0 2024-06-04T04:55:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.079+0000 7febedffb700 1 -- 172.21.15.70:0/4078515852 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe4005440 con 0x7febf01035e0 2024-06-04T04:55:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.080+0000 7febf6c7b700 1 -- 172.21.15.70:0/4078515852 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febf01a66a0 con 0x7febf01035e0 2024-06-04T04:55:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.080+0000 7febf6c7b700 1 -- 172.21.15.70:0/4078515852 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7febf0063c50 con 0x7febf01035e0 2024-06-04T04:55:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.082+0000 7febedffb700 1 -- 172.21.15.70:0/4078515852 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7febe4004750 con 0x7febf01035e0 2024-06-04T04:55:04.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.082+0000 7febedffb700 1 --2- 172.21.15.70:0/4078515852 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7febe00b2990 0x7febe00b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:04.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.082+0000 7febeffff700 1 --2- 172.21.15.70:0/4078515852 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7febe00b2990 0x7febe00b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:04.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.083+0000 7febedffb700 1 -- 172.21.15.70:0/4078515852 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7febe40e22b0 con 0x7febf01035e0 2024-06-04T04:55:04.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.083+0000 7febedffb700 1 --2- 172.21.15.70:0/4078515852 >> [v2:172.21.15.112:6824/1400793137,v1:172.21.15.112:6825/1400793137] conn(0x7febe00bc5d0 0x7febe00bea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:04.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.083+0000 7febedffb700 1 -- 172.21.15.70:0/4078515852 --> [v2:172.21.15.112:6824/1400793137,v1:172.21.15.112:6825/1400793137] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7febe00bf140 con 0x7febe00bc5d0 2024-06-04T04:55:04.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.083+0000 7febedffb700 1 -- 172.21.15.70:0/4078515852 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7febe40e2700 con 0x7febf01035e0 2024-06-04T04:55:04.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.083+0000 7febeffff700 1 --2- 172.21.15.70:0/4078515852 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7febe00b2990 0x7febe00b4e50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7febdc000c00 tx=0x7febdc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:04.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.083+0000 7febf5218700 1 --2- 172.21.15.70:0/4078515852 >> [v2:172.21.15.112:6824/1400793137,v1:172.21.15.112:6825/1400793137] conn(0x7febe00bc5d0 0x7febe00bea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:04.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.083+0000 7febf5218700 1 --2- 172.21.15.70:0/4078515852 >> [v2:172.21.15.112:6824/1400793137,v1:172.21.15.112:6825/1400793137] conn(0x7febe00bc5d0 0x7febe00bea70 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:04.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.084+0000 7febedffb700 1 -- 172.21.15.70:0/4078515852 <== osd.11 v2:172.21.15.112:6824/1400793137 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7febe00bf140 con 0x7febe00bc5d0 2024-06-04T04:55:04.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.105+0000 7febf6c7b700 1 -- 172.21.15.70:0/4078515852 --> [v2:172.21.15.112:6824/1400793137,v1:172.21.15.112:6825/1400793137] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7febf00640d0 con 0x7febe00bc5d0 2024-06-04T04:55:04.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.106+0000 7febedffb700 1 -- 172.21.15.70:0/4078515852 <== osd.11 v2:172.21.15.112:6824/1400793137 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7febf00640d0 con 0x7febe00bc5d0 2024-06-04T04:55:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.106+0000 7febf6c7b700 1 -- 172.21.15.70:0/4078515852 >> [v2:172.21.15.112:6824/1400793137,v1:172.21.15.112:6825/1400793137] conn(0x7febe00bc5d0 msgr2=0x7febe00bea70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.106+0000 7febf6c7b700 1 --2- 172.21.15.70:0/4078515852 >> [v2:172.21.15.112:6824/1400793137,v1:172.21.15.112:6825/1400793137] conn(0x7febe00bc5d0 0x7febe00bea70 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.106+0000 7febf6c7b700 1 -- 172.21.15.70:0/4078515852 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7febe00b2990 msgr2=0x7febe00b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.106+0000 7febf6c7b700 1 --2- 172.21.15.70:0/4078515852 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7febe00b2990 0x7febe00b4e50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7febdc000c00 tx=0x7febdc005fb0 comp rx=0 tx=0).stop 2024-06-04T04:55:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.106+0000 7febf6c7b700 1 -- 172.21.15.70:0/4078515852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf01035e0 msgr2=0x7febf019f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.106+0000 7febf6c7b700 1 --2- 172.21.15.70:0/4078515852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf01035e0 0x7febf019f7d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7febe400c920 tx=0x7febe400cc30 comp rx=0 tx=0).stop 2024-06-04T04:55:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.106+0000 7febf5218700 1 -- 172.21.15.70:0/4078515852 reap_dead start 2024-06-04T04:55:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.106+0000 7febf6c7b700 1 -- 172.21.15.70:0/4078515852 shutdown_connections 2024-06-04T04:55:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.106+0000 7febf6c7b700 1 -- 172.21.15.70:0/4078515852 >> 172.21.15.70:0/4078515852 conn(0x7febf007a6f0 msgr2=0x7febf0102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.106+0000 7febf6c7b700 1 -- 172.21.15.70:0/4078515852 shutdown_connections 2024-06-04T04:55:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:04.106+0000 7febf6c7b700 1 -- 172.21.15.70:0/4078515852 wait complete. 2024-06-04T04:55:05.096 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:04 smithi070 ceph-mon[32631]: pgmap v334: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:55:05.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:04 smithi112 ceph-mon[36453]: pgmap v334: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:55:05.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.193+0000 7f1717eff700 1 -- 172.21.15.70:0/2467647508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f171006c6d0 msgr2=0x7f1710110400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:05.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.193+0000 7f1717eff700 1 --2- 172.21.15.70:0/2467647508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f171006c6d0 0x7f1710110400 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f170800b0d0 tx=0x7f170800b3e0 comp rx=0 tx=0).stop 2024-06-04T04:55:05.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.193+0000 7f1717eff700 1 -- 172.21.15.70:0/2467647508 shutdown_connections 2024-06-04T04:55:05.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.193+0000 7f1717eff700 1 --2- 172.21.15.70:0/2467647508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f171006c6d0 0x7f1710110400 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:05.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.193+0000 7f1717eff700 1 --2- 172.21.15.70:0/2467647508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1710109630 0x7f1710109a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:05.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.193+0000 7f1717eff700 1 --2- 172.21.15.70:0/2467647508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f171006bd10 0x7f171006c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:05.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.193+0000 7f1717eff700 1 -- 172.21.15.70:0/2467647508 >> 172.21.15.70:0/2467647508 conn(0x7f17100794e0 msgr2=0x7f17100798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:05.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.193+0000 7f1717eff700 1 -- 172.21.15.70:0/2467647508 shutdown_connections 2024-06-04T04:55:05.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.193+0000 7f1717eff700 1 -- 172.21.15.70:0/2467647508 wait complete. 2024-06-04T04:55:05.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.193+0000 7f1717eff700 1 Processor -- start 2024-06-04T04:55:05.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.193+0000 7f1717eff700 1 -- start start 2024-06-04T04:55:05.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f1717eff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f171006bd10 0x7f171019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:05.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f1717eff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f171006c6d0 0x7f171019fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:05.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f1717eff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1710109630 0x7f1710199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:05.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f1717eff700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1710113600 con 0x7f1710109630 2024-06-04T04:55:05.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f1717eff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1710113480 con 0x7f171006c6d0 2024-06-04T04:55:05.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f1717eff700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1710113780 con 0x7f171006bd10 2024-06-04T04:55:05.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f171649c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1710109630 0x7f1710199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:05.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f171649c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1710109630 0x7f1710199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58002/0 (socket says 172.21.15.70:58002) 2024-06-04T04:55:05.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f171649c700 1 -- 172.21.15.70:0/73985732 learned_addr learned my addr 172.21.15.70:0/73985732 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:05.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f171549a700 1 --2- 172.21.15.70:0/73985732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f171006c6d0 0x7f171019fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:05.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f1715c9b700 1 --2- 172.21.15.70:0/73985732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f171006bd10 0x7f171019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:05.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f171649c700 1 -- 172.21.15.70:0/73985732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f171006bd10 msgr2=0x7f171019f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:05.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f171649c700 1 --2- 172.21.15.70:0/73985732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f171006bd10 0x7f171019f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:05.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f171649c700 1 -- 172.21.15.70:0/73985732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f171006c6d0 msgr2=0x7f171019fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:05.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f171649c700 1 --2- 172.21.15.70:0/73985732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f171006c6d0 0x7f171019fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:05.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f171649c700 1 -- 172.21.15.70:0/73985732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171019a080 con 0x7f1710109630 2024-06-04T04:55:05.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f171649c700 1 --2- 172.21.15.70:0/73985732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1710109630 0x7f1710199920 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1708009850 tx=0x7f1708007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:05.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f1706ffd700 1 -- 172.21.15.70:0/73985732 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170801f070 con 0x7f1710109630 2024-06-04T04:55:05.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f1706ffd700 1 -- 172.21.15.70:0/73985732 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1708005500 con 0x7f1710109630 2024-06-04T04:55:05.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f1717eff700 1 -- 172.21.15.70:0/73985732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f171019a310 con 0x7f1710109630 2024-06-04T04:55:05.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.194+0000 7f1706ffd700 1 -- 172.21.15.70:0/73985732 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170801b9a0 con 0x7f1710109630 2024-06-04T04:55:05.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.195+0000 7f1717eff700 1 -- 172.21.15.70:0/73985732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17101a6630 con 0x7f1710109630 2024-06-04T04:55:05.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.195+0000 7f1717eff700 1 -- 172.21.15.70:0/73985732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1710063cd0 con 0x7f1710109630 2024-06-04T04:55:05.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.197+0000 7f1706ffd700 1 -- 172.21.15.70:0/73985732 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f1708022070 con 0x7f1710109630 2024-06-04T04:55:05.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.197+0000 7f1706ffd700 1 --2- 172.21.15.70:0/73985732 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f16fc0b2910 0x7f16fc0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:05.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.198+0000 7f1715c9b700 1 --2- 172.21.15.70:0/73985732 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f16fc0b2910 0x7f16fc0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:05.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.198+0000 7f1706ffd700 1 -- 172.21.15.70:0/73985732 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f170800f020 con 0x7f1710109630 2024-06-04T04:55:05.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.198+0000 7f1706ffd700 1 --2- 172.21.15.70:0/73985732 >> [v2:172.21.15.98:6816/709593020,v1:172.21.15.98:6817/709593020] conn(0x7f16fc0bc550 0x7f16fc0be9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:05.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.198+0000 7f1706ffd700 1 -- 172.21.15.70:0/73985732 --> [v2:172.21.15.98:6816/709593020,v1:172.21.15.98:6817/709593020] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16fc0bf0c0 con 0x7f16fc0bc550 2024-06-04T04:55:05.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.198+0000 7f1706ffd700 1 -- 172.21.15.70:0/73985732 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f17080e3f30 con 0x7f1710109630 2024-06-04T04:55:05.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.198+0000 7f1715c9b700 1 --2- 172.21.15.70:0/73985732 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f16fc0b2910 0x7f16fc0b4dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1700005e10 tx=0x7f1700005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:05.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.198+0000 7f171549a700 1 --2- 172.21.15.70:0/73985732 >> [v2:172.21.15.98:6816/709593020,v1:172.21.15.98:6817/709593020] conn(0x7f16fc0bc550 0x7f16fc0be9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:05.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.198+0000 7f171549a700 1 --2- 172.21.15.70:0/73985732 >> [v2:172.21.15.98:6816/709593020,v1:172.21.15.98:6817/709593020] conn(0x7f16fc0bc550 0x7f16fc0be9f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:05.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.199+0000 7f1706ffd700 1 -- 172.21.15.70:0/73985732 <== osd.7 v2:172.21.15.98:6816/709593020 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16fc0bf0c0 con 0x7f16fc0bc550 2024-06-04T04:55:05.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.215+0000 7f1717eff700 1 -- 172.21.15.70:0/73985732 --> [v2:172.21.15.98:6816/709593020,v1:172.21.15.98:6817/709593020] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1710064150 con 0x7f16fc0bc550 2024-06-04T04:55:05.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.216+0000 7f1706ffd700 1 -- 172.21.15.70:0/73985732 <== osd.7 v2:172.21.15.98:6816/709593020 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1710064150 con 0x7f16fc0bc550 2024-06-04T04:55:05.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.216+0000 7f1717eff700 1 -- 172.21.15.70:0/73985732 >> [v2:172.21.15.98:6816/709593020,v1:172.21.15.98:6817/709593020] conn(0x7f16fc0bc550 msgr2=0x7f16fc0be9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:05.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.216+0000 7f1717eff700 1 --2- 172.21.15.70:0/73985732 >> [v2:172.21.15.98:6816/709593020,v1:172.21.15.98:6817/709593020] conn(0x7f16fc0bc550 0x7f16fc0be9f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:05.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.216+0000 7f1717eff700 1 -- 172.21.15.70:0/73985732 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f16fc0b2910 msgr2=0x7f16fc0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:05.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.216+0000 7f1717eff700 1 --2- 172.21.15.70:0/73985732 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f16fc0b2910 0x7f16fc0b4dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1700005e10 tx=0x7f1700005d40 comp rx=0 tx=0).stop 2024-06-04T04:55:05.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.216+0000 7f1717eff700 1 -- 172.21.15.70:0/73985732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1710109630 msgr2=0x7f1710199920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:05.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.216+0000 7f1717eff700 1 --2- 172.21.15.70:0/73985732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1710109630 0x7f1710199920 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1708009850 tx=0x7f1708007d90 comp rx=0 tx=0).stop 2024-06-04T04:55:05.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.216+0000 7f171649c700 1 -- 172.21.15.70:0/73985732 reap_dead start 2024-06-04T04:55:05.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.216+0000 7f1717eff700 1 -- 172.21.15.70:0/73985732 shutdown_connections 2024-06-04T04:55:05.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.217+0000 7f1717eff700 1 -- 172.21.15.70:0/73985732 >> 172.21.15.70:0/73985732 conn(0x7f17100794e0 msgr2=0x7f1710101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:05.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.217+0000 7f1717eff700 1 -- 172.21.15.70:0/73985732 shutdown_connections 2024-06-04T04:55:05.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:05.217+0000 7f1717eff700 1 -- 172.21.15.70:0/73985732 wait complete. 2024-06-04T04:55:05.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:04 smithi098 ceph-mon[35569]: pgmap v334: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:55:06.825 INFO:teuthology.orchestra.run.smithi070.stdout:77309411405 2024-06-04T04:55:06.826 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-04T04:55:07.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.016+0000 7f9683f6b700 1 -- 172.21.15.70:0/2762532100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f967c109680 msgr2=0x7f967c109a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:07.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.016+0000 7f9683f6b700 1 --2- 172.21.15.70:0/2762532100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f967c109680 0x7f967c109a60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9678009990 tx=0x7f9678009ca0 comp rx=0 tx=0).stop 2024-06-04T04:55:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.016+0000 7f9683f6b700 1 -- 172.21.15.70:0/2762532100 shutdown_connections 2024-06-04T04:55:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.016+0000 7f9683f6b700 1 --2- 172.21.15.70:0/2762532100 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f967c104220 0x7f967c110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.016+0000 7f9683f6b700 1 --2- 172.21.15.70:0/2762532100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f967c109680 0x7f967c109a60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.016+0000 7f9683f6b700 1 --2- 172.21.15.70:0/2762532100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f967c103860 0x7f967c103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.016+0000 7f9683f6b700 1 -- 172.21.15.70:0/2762532100 >> 172.21.15.70:0/2762532100 conn(0x7f967c07b4d0 msgr2=0x7f967c07b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.016+0000 7f9683f6b700 1 -- 172.21.15.70:0/2762532100 shutdown_connections 2024-06-04T04:55:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.016+0000 7f9683f6b700 1 -- 172.21.15.70:0/2762532100 wait complete. 2024-06-04T04:55:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.017+0000 7f9683f6b700 1 Processor -- start 2024-06-04T04:55:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.017+0000 7f9683f6b700 1 -- start start 2024-06-04T04:55:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.017+0000 7f9683f6b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f967c103860 0x7f967c19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.017+0000 7f9683f6b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f967c104220 0x7f967c19fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.017+0000 7f9683f6b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f967c109680 0x7f967c1999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.017+0000 7f9683f6b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f967c113660 con 0x7f967c103860 2024-06-04T04:55:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.017+0000 7f9683f6b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f967c1134e0 con 0x7f967c104220 2024-06-04T04:55:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.017+0000 7f9683f6b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f967c1137e0 con 0x7f967c109680 2024-06-04T04:55:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.017+0000 7f9681d07700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f967c103860 0x7f967c19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.017+0000 7f9681d07700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f967c103860 0x7f967c19f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42200/0 (socket says 172.21.15.70:42200) 2024-06-04T04:55:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.017+0000 7f9681d07700 1 -- 172.21.15.70:0/830456891 learned_addr learned my addr 172.21.15.70:0/830456891 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.017+0000 7f9681506700 1 --2- 172.21.15.70:0/830456891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f967c104220 0x7f967c19fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.018+0000 7f9682508700 1 --2- 172.21.15.70:0/830456891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f967c109680 0x7f967c1999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.018+0000 7f9681d07700 1 -- 172.21.15.70:0/830456891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f967c109680 msgr2=0x7f967c1999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.018+0000 7f9681d07700 1 --2- 172.21.15.70:0/830456891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f967c109680 0x7f967c1999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.018+0000 7f9681d07700 1 -- 172.21.15.70:0/830456891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f967c104220 msgr2=0x7f967c19fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.018+0000 7f9681d07700 1 --2- 172.21.15.70:0/830456891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f967c104220 0x7f967c19fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.018+0000 7f9681d07700 1 -- 172.21.15.70:0/830456891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f967c19a140 con 0x7f967c103860 2024-06-04T04:55:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.018+0000 7f9682508700 1 --2- 172.21.15.70:0/830456891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f967c109680 0x7f967c1999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:55:07.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.018+0000 7f9681d07700 1 --2- 172.21.15.70:0/830456891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f967c103860 0x7f967c19f820 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9678000c00 tx=0x7f96780047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:07.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.018+0000 7f9672ffd700 1 -- 172.21.15.70:0/830456891 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9678027070 con 0x7f967c103860 2024-06-04T04:55:07.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.018+0000 7f9672ffd700 1 -- 172.21.15.70:0/830456891 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f967800fb80 con 0x7f967c103860 2024-06-04T04:55:07.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.018+0000 7f9683f6b700 1 -- 172.21.15.70:0/830456891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f967c19a3d0 con 0x7f967c103860 2024-06-04T04:55:07.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.018+0000 7f9672ffd700 1 -- 172.21.15.70:0/830456891 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9678022d00 con 0x7f967c103860 2024-06-04T04:55:07.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.018+0000 7f9683f6b700 1 -- 172.21.15.70:0/830456891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f967c1a66f0 con 0x7f967c103860 2024-06-04T04:55:07.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.019+0000 7f9683f6b700 1 -- 172.21.15.70:0/830456891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f967c063c50 con 0x7f967c103860 2024-06-04T04:55:07.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.021+0000 7f9672ffd700 1 -- 172.21.15.70:0/830456891 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f967802a070 con 0x7f967c103860 2024-06-04T04:55:07.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.022+0000 7f9672ffd700 1 --2- 172.21.15.70:0/830456891 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f96680b2910 0x7f96680b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:07.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.022+0000 7f9681506700 1 --2- 172.21.15.70:0/830456891 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f96680b2910 0x7f96680b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:07.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.022+0000 7f9672ffd700 1 -- 172.21.15.70:0/830456891 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f96780e38e0 con 0x7f967c103860 2024-06-04T04:55:07.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.022+0000 7f9672ffd700 1 --2- 172.21.15.70:0/830456891 >> [v2:172.21.15.98:6800/2744598742,v1:172.21.15.98:6801/2744598742] conn(0x7f96680bc550 0x7f96680be9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:07.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.022+0000 7f9672ffd700 1 -- 172.21.15.70:0/830456891 --> [v2:172.21.15.98:6800/2744598742,v1:172.21.15.98:6801/2744598742] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f96780095d0 con 0x7f96680bc550 2024-06-04T04:55:07.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.022+0000 7f9672ffd700 1 -- 172.21.15.70:0/830456891 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f96780e3d30 con 0x7f967c103860 2024-06-04T04:55:07.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.022+0000 7f9681506700 1 --2- 172.21.15.70:0/830456891 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f96680b2910 0x7f96680b4dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f967c07bbb0 tx=0x7f966c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:07.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.023+0000 7f9682508700 1 --2- 172.21.15.70:0/830456891 >> [v2:172.21.15.98:6800/2744598742,v1:172.21.15.98:6801/2744598742] conn(0x7f96680bc550 0x7f96680be9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:07.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.023+0000 7f9682508700 1 --2- 172.21.15.70:0/830456891 >> [v2:172.21.15.98:6800/2744598742,v1:172.21.15.98:6801/2744598742] conn(0x7f96680bc550 0x7f96680be9f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:07.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.024+0000 7f9672ffd700 1 -- 172.21.15.70:0/830456891 <== osd.1 v2:172.21.15.98:6800/2744598742 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96780095d0 con 0x7f96680bc550 2024-06-04T04:55:07.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.046+0000 7f9683f6b700 1 -- 172.21.15.70:0/830456891 --> [v2:172.21.15.98:6800/2744598742,v1:172.21.15.98:6801/2744598742] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f967c0640d0 con 0x7f96680bc550 2024-06-04T04:55:07.049 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:06 smithi070 ceph-mon[32631]: pgmap v335: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:55:07.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.047+0000 7f9672ffd700 1 -- 172.21.15.70:0/830456891 <== osd.1 v2:172.21.15.98:6800/2744598742 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f967c0640d0 con 0x7f96680bc550 2024-06-04T04:55:07.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.047+0000 7f9683f6b700 1 -- 172.21.15.70:0/830456891 >> [v2:172.21.15.98:6800/2744598742,v1:172.21.15.98:6801/2744598742] conn(0x7f96680bc550 msgr2=0x7f96680be9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:07.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.047+0000 7f9683f6b700 1 --2- 172.21.15.70:0/830456891 >> [v2:172.21.15.98:6800/2744598742,v1:172.21.15.98:6801/2744598742] conn(0x7f96680bc550 0x7f96680be9f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:07.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.048+0000 7f9683f6b700 1 -- 172.21.15.70:0/830456891 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f96680b2910 msgr2=0x7f96680b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:07.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.048+0000 7f9683f6b700 1 --2- 172.21.15.70:0/830456891 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f96680b2910 0x7f96680b4dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f967c07bbb0 tx=0x7f966c00a300 comp rx=0 tx=0).stop 2024-06-04T04:55:07.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.048+0000 7f9683f6b700 1 -- 172.21.15.70:0/830456891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f967c103860 msgr2=0x7f967c19f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:07.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.048+0000 7f9683f6b700 1 --2- 172.21.15.70:0/830456891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f967c103860 0x7f967c19f820 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9678000c00 tx=0x7f96780047b0 comp rx=0 tx=0).stop 2024-06-04T04:55:07.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.048+0000 7f9682508700 1 -- 172.21.15.70:0/830456891 reap_dead start 2024-06-04T04:55:07.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.048+0000 7f9683f6b700 1 -- 172.21.15.70:0/830456891 shutdown_connections 2024-06-04T04:55:07.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.048+0000 7f9683f6b700 1 -- 172.21.15.70:0/830456891 >> 172.21.15.70:0/830456891 conn(0x7f967c07b4d0 msgr2=0x7f967c1026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:07.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.048+0000 7f9683f6b700 1 -- 172.21.15.70:0/830456891 shutdown_connections 2024-06-04T04:55:07.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.048+0000 7f9683f6b700 1 -- 172.21.15.70:0/830456891 wait complete. 2024-06-04T04:55:07.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:06 smithi112 ceph-mon[36453]: pgmap v335: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:55:07.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:06 smithi098 ceph-mon[35569]: pgmap v335: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:55:07.851 INFO:teuthology.orchestra.run.smithi070.stdout:206158430261 2024-06-04T04:55:07.851 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-06-04T04:55:08.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.147+0000 7efea8218700 1 -- 172.21.15.70:0/3302568232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efea0108520 msgr2=0x7efea0108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:08.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.147+0000 7efea8218700 1 --2- 172.21.15.70:0/3302568232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efea0108520 0x7efea0108900 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efe9c009990 tx=0x7efe9c009ca0 comp rx=0 tx=0).stop 2024-06-04T04:55:08.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.147+0000 7efea8218700 1 -- 172.21.15.70:0/3302568232 shutdown_connections 2024-06-04T04:55:08.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.147+0000 7efea8218700 1 --2- 172.21.15.70:0/3302568232 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efea0102e00 0x7efea010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:08.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.147+0000 7efea8218700 1 --2- 172.21.15.70:0/3302568232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efea0108520 0x7efea0108900 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:08.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.147+0000 7efea8218700 1 --2- 172.21.15.70:0/3302568232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efea0102440 0x7efea01028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:08.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.147+0000 7efea8218700 1 -- 172.21.15.70:0/3302568232 >> 172.21.15.70:0/3302568232 conn(0x7efea00793f0 msgr2=0x7efea0079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:08.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.147+0000 7efea8218700 1 -- 172.21.15.70:0/3302568232 shutdown_connections 2024-06-04T04:55:08.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.147+0000 7efea8218700 1 -- 172.21.15.70:0/3302568232 wait complete. 2024-06-04T04:55:08.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea8218700 1 Processor -- start 2024-06-04T04:55:08.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea8218700 1 -- start start 2024-06-04T04:55:08.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea8218700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efea0102440 0x7efea019f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:08.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea8218700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efea0102e00 0x7efea019fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:08.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea8218700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efea0108520 0x7efea0199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:08.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea8218700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efea0113670 con 0x7efea0108520 2024-06-04T04:55:08.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea8218700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efea01134f0 con 0x7efea0102e00 2024-06-04T04:55:08.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea8218700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efea01137f0 con 0x7efea0102440 2024-06-04T04:55:08.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea67b5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efea0108520 0x7efea0199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:08.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea67b5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efea0108520 0x7efea0199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42248/0 (socket says 172.21.15.70:42248) 2024-06-04T04:55:08.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea67b5700 1 -- 172.21.15.70:0/1129893835 learned_addr learned my addr 172.21.15.70:0/1129893835 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:08.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea5fb4700 1 --2- 172.21.15.70:0/1129893835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efea0102440 0x7efea019f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:08.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea67b5700 1 -- 172.21.15.70:0/1129893835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efea0102440 msgr2=0x7efea019f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:08.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea57b3700 1 --2- 172.21.15.70:0/1129893835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efea0102e00 0x7efea019fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:08.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea67b5700 1 --2- 172.21.15.70:0/1129893835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efea0102440 0x7efea019f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:08.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea67b5700 1 -- 172.21.15.70:0/1129893835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efea0102e00 msgr2=0x7efea019fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:08.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea67b5700 1 --2- 172.21.15.70:0/1129893835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efea0102e00 0x7efea019fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:08.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea67b5700 1 -- 172.21.15.70:0/1129893835 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea019a230 con 0x7efea0108520 2024-06-04T04:55:08.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea57b3700 1 --2- 172.21.15.70:0/1129893835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efea0102e00 0x7efea019fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:55:08.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.148+0000 7efea67b5700 1 --2- 172.21.15.70:0/1129893835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efea0108520 0x7efea0199ad0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efe9800b970 tx=0x7efe9800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:08.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.149+0000 7efe96ffd700 1 -- 172.21.15.70:0/1129893835 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe98019070 con 0x7efea0108520 2024-06-04T04:55:08.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.149+0000 7efe96ffd700 1 -- 172.21.15.70:0/1129893835 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efe9800ce40 con 0x7efea0108520 2024-06-04T04:55:08.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.149+0000 7efea8218700 1 -- 172.21.15.70:0/1129893835 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efea019a520 con 0x7efea0108520 2024-06-04T04:55:08.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.149+0000 7efe96ffd700 1 -- 172.21.15.70:0/1129893835 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe98007370 con 0x7efea0108520 2024-06-04T04:55:08.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.149+0000 7efea8218700 1 -- 172.21.15.70:0/1129893835 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efea01a67e0 con 0x7efea0108520 2024-06-04T04:55:08.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.150+0000 7efea8218700 1 -- 172.21.15.70:0/1129893835 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efea0063cd0 con 0x7efea0108520 2024-06-04T04:55:08.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.151+0000 7efe96ffd700 1 -- 172.21.15.70:0/1129893835 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7efe9801c070 con 0x7efea0108520 2024-06-04T04:55:08.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.152+0000 7efe96ffd700 1 --2- 172.21.15.70:0/1129893835 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7efe8c0b2910 0x7efe8c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:08.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.152+0000 7efea5fb4700 1 --2- 172.21.15.70:0/1129893835 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7efe8c0b2910 0x7efe8c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:08.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.152+0000 7efe96ffd700 1 -- 172.21.15.70:0/1129893835 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7efe980e2330 con 0x7efea0108520 2024-06-04T04:55:08.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.152+0000 7efe96ffd700 1 --2- 172.21.15.70:0/1129893835 >> [v2:172.21.15.112:6808/1857257899,v1:172.21.15.112:6809/1857257899] conn(0x7efe8c0bc550 0x7efe8c0be9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:08.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.152+0000 7efe96ffd700 1 -- 172.21.15.70:0/1129893835 --> [v2:172.21.15.112:6808/1857257899,v1:172.21.15.112:6809/1857257899] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efe98004860 con 0x7efe8c0bc550 2024-06-04T04:55:08.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.152+0000 7efe96ffd700 1 -- 172.21.15.70:0/1129893835 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7efe980e2780 con 0x7efea0108520 2024-06-04T04:55:08.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.152+0000 7efea5fb4700 1 --2- 172.21.15.70:0/1129893835 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7efe8c0b2910 0x7efe8c0b4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efe9c005cd0 tx=0x7efe9c005c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:08.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.152+0000 7efea57b3700 1 --2- 172.21.15.70:0/1129893835 >> [v2:172.21.15.112:6808/1857257899,v1:172.21.15.112:6809/1857257899] conn(0x7efe8c0bc550 0x7efe8c0be9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:08.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.153+0000 7efea57b3700 1 --2- 172.21.15.70:0/1129893835 >> [v2:172.21.15.112:6808/1857257899,v1:172.21.15.112:6809/1857257899] conn(0x7efe8c0bc550 0x7efe8c0be9f0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:08.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.154+0000 7efe96ffd700 1 -- 172.21.15.70:0/1129893835 <== osd.5 v2:172.21.15.112:6808/1857257899 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efe98004860 con 0x7efe8c0bc550 2024-06-04T04:55:08.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.170+0000 7efea8218700 1 -- 172.21.15.70:0/1129893835 --> [v2:172.21.15.112:6808/1857257899,v1:172.21.15.112:6809/1857257899] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efea0064150 con 0x7efe8c0bc550 2024-06-04T04:55:08.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.171+0000 7efe96ffd700 1 -- 172.21.15.70:0/1129893835 <== osd.5 v2:172.21.15.112:6808/1857257899 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7efea0064150 con 0x7efe8c0bc550 2024-06-04T04:55:08.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.171+0000 7efea8218700 1 -- 172.21.15.70:0/1129893835 >> [v2:172.21.15.112:6808/1857257899,v1:172.21.15.112:6809/1857257899] conn(0x7efe8c0bc550 msgr2=0x7efe8c0be9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:08.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.171+0000 7efea8218700 1 --2- 172.21.15.70:0/1129893835 >> [v2:172.21.15.112:6808/1857257899,v1:172.21.15.112:6809/1857257899] conn(0x7efe8c0bc550 0x7efe8c0be9f0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:08.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.171+0000 7efea8218700 1 -- 172.21.15.70:0/1129893835 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7efe8c0b2910 msgr2=0x7efe8c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:08.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.171+0000 7efea8218700 1 --2- 172.21.15.70:0/1129893835 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7efe8c0b2910 0x7efe8c0b4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efe9c005cd0 tx=0x7efe9c005c40 comp rx=0 tx=0).stop 2024-06-04T04:55:08.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.171+0000 7efea8218700 1 -- 172.21.15.70:0/1129893835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efea0108520 msgr2=0x7efea0199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:08.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.171+0000 7efea8218700 1 --2- 172.21.15.70:0/1129893835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efea0108520 0x7efea0199ad0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efe9800b970 tx=0x7efe9800bd30 comp rx=0 tx=0).stop 2024-06-04T04:55:08.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.171+0000 7efea67b5700 1 -- 172.21.15.70:0/1129893835 reap_dead start 2024-06-04T04:55:08.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.171+0000 7efea8218700 1 -- 172.21.15.70:0/1129893835 shutdown_connections 2024-06-04T04:55:08.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.171+0000 7efea8218700 1 -- 172.21.15.70:0/1129893835 >> 172.21.15.70:0/1129893835 conn(0x7efea00793f0 msgr2=0x7efea0101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:08.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.171+0000 7efea8218700 1 -- 172.21.15.70:0/1129893835 shutdown_connections 2024-06-04T04:55:08.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:08.171+0000 7efea8218700 1 -- 172.21.15.70:0/1129893835 wait complete. 2024-06-04T04:55:09.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:08 smithi070 ceph-mon[32631]: pgmap v336: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-04T04:55:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:08 smithi112 ceph-mon[36453]: pgmap v336: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-04T04:55:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:08 smithi098 ceph-mon[35569]: pgmap v336: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-04T04:55:09.626 INFO:teuthology.orchestra.run.smithi070.stdout:304942678049 2024-06-04T04:55:09.627 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-06-04T04:55:10.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.259+0000 7f0f7beb6700 1 -- 172.21.15.70:0/2241887741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f741037f0 msgr2=0x7f0f74103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:10.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.259+0000 7f0f7beb6700 1 --2- 172.21.15.70:0/2241887741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f741037f0 0x7f0f74103bd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0f700099c0 tx=0x7f0f70009cd0 comp rx=0 tx=0).stop 2024-06-04T04:55:10.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.259+0000 7f0f7beb6700 1 -- 172.21.15.70:0/2241887741 shutdown_connections 2024-06-04T04:55:10.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.259+0000 7f0f7beb6700 1 --2- 172.21.15.70:0/2241887741 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f741088f0 0x7f0f74110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:10.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.259+0000 7f0f7beb6700 1 --2- 172.21.15.70:0/2241887741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f741037f0 0x7f0f74103bd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:10.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.259+0000 7f0f7beb6700 1 --2- 172.21.15.70:0/2241887741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f741041a0 0x7f0f741081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:10.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.259+0000 7f0f7beb6700 1 -- 172.21.15.70:0/2241887741 >> 172.21.15.70:0/2241887741 conn(0x7f0f7407a6a0 msgr2=0x7f0f7407aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:10.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.259+0000 7f0f7beb6700 1 -- 172.21.15.70:0/2241887741 shutdown_connections 2024-06-04T04:55:10.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.260+0000 7f0f7beb6700 1 -- 172.21.15.70:0/2241887741 wait complete. 2024-06-04T04:55:10.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.260+0000 7f0f7beb6700 1 Processor -- start 2024-06-04T04:55:10.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.260+0000 7f0f7beb6700 1 -- start start 2024-06-04T04:55:10.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.260+0000 7f0f7beb6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f741037f0 0x7f0f7419f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:10.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.260+0000 7f0f7beb6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f741041a0 0x7f0f7419fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:10.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.260+0000 7f0f7beb6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f741088f0 0x7f0f741999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:10.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.260+0000 7f0f7beb6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0f74113600 con 0x7f0f741037f0 2024-06-04T04:55:10.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.260+0000 7f0f7beb6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0f74113480 con 0x7f0f741088f0 2024-06-04T04:55:10.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.260+0000 7f0f7beb6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0f74113780 con 0x7f0f741041a0 2024-06-04T04:55:10.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.260+0000 7f0f79c52700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f741037f0 0x7f0f7419f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:10.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.260+0000 7f0f79c52700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f741037f0 0x7f0f7419f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42230/0 (socket says 172.21.15.70:42230) 2024-06-04T04:55:10.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.260+0000 7f0f79c52700 1 -- 172.21.15.70:0/660022631 learned_addr learned my addr 172.21.15.70:0/660022631 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:10.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.261+0000 7f0f7a453700 1 --2- 172.21.15.70:0/660022631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f741088f0 0x7f0f741999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:10.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.261+0000 7f0f79451700 1 --2- 172.21.15.70:0/660022631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f741041a0 0x7f0f7419fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:10.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.261+0000 7f0f79c52700 1 -- 172.21.15.70:0/660022631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f741041a0 msgr2=0x7f0f7419fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:10.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.261+0000 7f0f79c52700 1 --2- 172.21.15.70:0/660022631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f741041a0 0x7f0f7419fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.261+0000 7f0f79c52700 1 -- 172.21.15.70:0/660022631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f741088f0 msgr2=0x7f0f741999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.261+0000 7f0f79c52700 1 --2- 172.21.15.70:0/660022631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f741088f0 0x7f0f741999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.261+0000 7f0f79c52700 1 -- 172.21.15.70:0/660022631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f7419a100 con 0x7f0f741037f0 2024-06-04T04:55:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.261+0000 7f0f79451700 1 --2- 172.21.15.70:0/660022631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f741041a0 0x7f0f7419fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:55:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.261+0000 7f0f79c52700 1 --2- 172.21.15.70:0/660022631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f741037f0 0x7f0f7419f7e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0f70000c00 tx=0x7f0f7000f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.261+0000 7f0f6affd700 1 -- 172.21.15.70:0/660022631 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7000f940 con 0x7f0f741037f0 2024-06-04T04:55:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.261+0000 7f0f7beb6700 1 -- 172.21.15.70:0/660022631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f7419a390 con 0x7f0f741037f0 2024-06-04T04:55:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.261+0000 7f0f6affd700 1 -- 172.21.15.70:0/660022631 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0f70004830 con 0x7f0f741037f0 2024-06-04T04:55:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.261+0000 7f0f7beb6700 1 -- 172.21.15.70:0/660022631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f741a66b0 con 0x7f0f741037f0 2024-06-04T04:55:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.261+0000 7f0f6affd700 1 -- 172.21.15.70:0/660022631 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f700054b0 con 0x7f0f741037f0 2024-06-04T04:55:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.262+0000 7f0f7beb6700 1 -- 172.21.15.70:0/660022631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0f74063cd0 con 0x7f0f741037f0 2024-06-04T04:55:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.263+0000 7f0f6affd700 1 -- 172.21.15.70:0/660022631 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f0f70029060 con 0x7f0f741037f0 2024-06-04T04:55:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.264+0000 7f0f6affd700 1 --2- 172.21.15.70:0/660022631 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0f600b2910 0x7f0f600b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.264+0000 7f0f79451700 1 --2- 172.21.15.70:0/660022631 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0f600b2910 0x7f0f600b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.264+0000 7f0f6affd700 1 -- 172.21.15.70:0/660022631 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f0f700e33c0 con 0x7f0f741037f0 2024-06-04T04:55:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.264+0000 7f0f6affd700 1 --2- 172.21.15.70:0/660022631 >> [v2:172.21.15.112:6816/2250225413,v1:172.21.15.112:6817/2250225413] conn(0x7f0f600bc550 0x7f0f600be9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.264+0000 7f0f6affd700 1 -- 172.21.15.70:0/660022631 --> [v2:172.21.15.112:6816/2250225413,v1:172.21.15.112:6817/2250225413] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f600bf0c0 con 0x7f0f600bc550 2024-06-04T04:55:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.264+0000 7f0f6affd700 1 -- 172.21.15.70:0/660022631 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f0f700e3810 con 0x7f0f741037f0 2024-06-04T04:55:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.265+0000 7f0f79451700 1 --2- 172.21.15.70:0/660022631 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0f600b2910 0x7f0f600b4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0f741031d0 tx=0x7f0f64008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.265+0000 7f0f7a453700 1 --2- 172.21.15.70:0/660022631 >> [v2:172.21.15.112:6816/2250225413,v1:172.21.15.112:6817/2250225413] conn(0x7f0f600bc550 0x7f0f600be9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.265+0000 7f0f7a453700 1 --2- 172.21.15.70:0/660022631 >> [v2:172.21.15.112:6816/2250225413,v1:172.21.15.112:6817/2250225413] conn(0x7f0f600bc550 0x7f0f600be9f0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.266+0000 7f0f6affd700 1 -- 172.21.15.70:0/660022631 <== osd.8 v2:172.21.15.112:6816/2250225413 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0f600bf0c0 con 0x7f0f600bc550 2024-06-04T04:55:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.282+0000 7f0f7beb6700 1 -- 172.21.15.70:0/660022631 --> [v2:172.21.15.112:6816/2250225413,v1:172.21.15.112:6817/2250225413] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0f740640d0 con 0x7f0f600bc550 2024-06-04T04:55:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.283+0000 7f0f6affd700 1 -- 172.21.15.70:0/660022631 <== osd.8 v2:172.21.15.112:6816/2250225413 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0f740640d0 con 0x7f0f600bc550 2024-06-04T04:55:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.283+0000 7f0f7beb6700 1 -- 172.21.15.70:0/660022631 >> [v2:172.21.15.112:6816/2250225413,v1:172.21.15.112:6817/2250225413] conn(0x7f0f600bc550 msgr2=0x7f0f600be9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.283+0000 7f0f7beb6700 1 --2- 172.21.15.70:0/660022631 >> [v2:172.21.15.112:6816/2250225413,v1:172.21.15.112:6817/2250225413] conn(0x7f0f600bc550 0x7f0f600be9f0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.283+0000 7f0f7beb6700 1 -- 172.21.15.70:0/660022631 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0f600b2910 msgr2=0x7f0f600b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.283+0000 7f0f7beb6700 1 --2- 172.21.15.70:0/660022631 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0f600b2910 0x7f0f600b4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0f741031d0 tx=0x7f0f64008040 comp rx=0 tx=0).stop 2024-06-04T04:55:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.283+0000 7f0f7beb6700 1 -- 172.21.15.70:0/660022631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f741037f0 msgr2=0x7f0f7419f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.283+0000 7f0f7beb6700 1 --2- 172.21.15.70:0/660022631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f741037f0 0x7f0f7419f7e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0f70000c00 tx=0x7f0f7000f510 comp rx=0 tx=0).stop 2024-06-04T04:55:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.283+0000 7f0f7a453700 1 -- 172.21.15.70:0/660022631 reap_dead start 2024-06-04T04:55:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.283+0000 7f0f7beb6700 1 -- 172.21.15.70:0/660022631 shutdown_connections 2024-06-04T04:55:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.283+0000 7f0f7beb6700 1 -- 172.21.15.70:0/660022631 >> 172.21.15.70:0/660022631 conn(0x7f0f7407a6a0 msgr2=0x7f0f74101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.283+0000 7f0f7beb6700 1 -- 172.21.15.70:0/660022631 shutdown_connections 2024-06-04T04:55:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:07.283+0000 7f0f7beb6700 1 -- 172.21.15.70:0/660022631 wait complete. 2024-06-04T04:55:10.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.475+0000 7f59a5f47700 1 -- 172.21.15.70:0/1806641720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a0109520 msgr2=0x7f59a0109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:10.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.475+0000 7f59a5f47700 1 --2- 172.21.15.70:0/1806641720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a0109520 0x7f59a0109900 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f59900099c0 tx=0x7f5990009cd0 comp rx=0 tx=0).stop 2024-06-04T04:55:10.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.475+0000 7f59a5f47700 1 -- 172.21.15.70:0/1806641720 shutdown_connections 2024-06-04T04:55:10.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.475+0000 7f59a5f47700 1 --2- 172.21.15.70:0/1806641720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59a01024f0 0x7f59a01103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:10.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.475+0000 7f59a5f47700 1 --2- 172.21.15.70:0/1806641720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a0109520 0x7f59a0109900 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:10.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.475+0000 7f59a5f47700 1 --2- 172.21.15.70:0/1806641720 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59a0101b30 0x7f59a0101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:10.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.475+0000 7f59a5f47700 1 -- 172.21.15.70:0/1806641720 >> 172.21.15.70:0/1806641720 conn(0x7f59a0078ab0 msgr2=0x7f59a00ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:10.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.475+0000 7f59a5f47700 1 -- 172.21.15.70:0/1806641720 shutdown_connections 2024-06-04T04:55:10.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.475+0000 7f59a5f47700 1 -- 172.21.15.70:0/1806641720 wait complete. 2024-06-04T04:55:10.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.475+0000 7f59a5f47700 1 Processor -- start 2024-06-04T04:55:10.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.476+0000 7f59a5f47700 1 -- start start 2024-06-04T04:55:10.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.476+0000 7f59a5f47700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59a0101b30 0x7f59a019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:10.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.476+0000 7f59a5f47700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59a01024f0 0x7f59a019fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:10.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.476+0000 7f59a5f47700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a0109520 0x7f59a0199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:10.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.476+0000 7f59a5f47700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f59a01135b0 con 0x7f59a0109520 2024-06-04T04:55:10.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.476+0000 7f59a5f47700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59a0113430 con 0x7f59a0101b30 2024-06-04T04:55:10.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.476+0000 7f59a5f47700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f59a0113730 con 0x7f59a01024f0 2024-06-04T04:55:10.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.476+0000 7f599ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a0109520 0x7f59a0199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:10.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.476+0000 7f599ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a0109520 0x7f59a0199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42266/0 (socket says 172.21.15.70:42266) 2024-06-04T04:55:10.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.476+0000 7f599ffff700 1 -- 172.21.15.70:0/3002089958 learned_addr learned my addr 172.21.15.70:0/3002089958 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:10.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.476+0000 7f599effd700 1 --2- 172.21.15.70:0/3002089958 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59a01024f0 0x7f59a019fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:10.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.476+0000 7f599f7fe700 1 --2- 172.21.15.70:0/3002089958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59a0101b30 0x7f59a019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:10.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.476+0000 7f599ffff700 1 -- 172.21.15.70:0/3002089958 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59a01024f0 msgr2=0x7f59a019fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:10.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.476+0000 7f599ffff700 1 --2- 172.21.15.70:0/3002089958 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59a01024f0 0x7f59a019fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:10.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.477+0000 7f599ffff700 1 -- 172.21.15.70:0/3002089958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59a0101b30 msgr2=0x7f59a019f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:10.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.477+0000 7f599ffff700 1 --2- 172.21.15.70:0/3002089958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59a0101b30 0x7f59a019f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:10.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.477+0000 7f599ffff700 1 -- 172.21.15.70:0/3002089958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a019a0b0 con 0x7f59a0109520 2024-06-04T04:55:10.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.477+0000 7f599f7fe700 1 --2- 172.21.15.70:0/3002089958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59a0101b30 0x7f59a019f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:55:10.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.477+0000 7f599ffff700 1 --2- 172.21.15.70:0/3002089958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a0109520 0x7f59a0199950 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5998007ee0 tx=0x7f599800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:10.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.477+0000 7f599cff9700 1 -- 172.21.15.70:0/3002089958 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599800dbd0 con 0x7f59a0109520 2024-06-04T04:55:10.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.477+0000 7f599cff9700 1 -- 172.21.15.70:0/3002089958 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f599800dd70 con 0x7f59a0109520 2024-06-04T04:55:10.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.477+0000 7f599cff9700 1 -- 172.21.15.70:0/3002089958 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59980058d0 con 0x7f59a0109520 2024-06-04T04:55:10.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.477+0000 7f59a5f47700 1 -- 172.21.15.70:0/3002089958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59a019a3a0 con 0x7f59a0109520 2024-06-04T04:55:10.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.477+0000 7f59a5f47700 1 -- 172.21.15.70:0/3002089958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59a01a6660 con 0x7f59a0109520 2024-06-04T04:55:10.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.478+0000 7f59a5f47700 1 -- 172.21.15.70:0/3002089958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f59a0063c90 con 0x7f59a0109520 2024-06-04T04:55:10.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.481+0000 7f599cff9700 1 -- 172.21.15.70:0/3002089958 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f599801a020 con 0x7f59a0109520 2024-06-04T04:55:10.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.481+0000 7f599cff9700 1 --2- 172.21.15.70:0/3002089958 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f59880b2990 0x7f59880b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:10.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.481+0000 7f599cff9700 1 -- 172.21.15.70:0/3002089958 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f59980e2220 con 0x7f59a0109520 2024-06-04T04:55:10.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.481+0000 7f599f7fe700 1 --2- 172.21.15.70:0/3002089958 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f59880b2990 0x7f59880b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:10.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.481+0000 7f599cff9700 1 --2- 172.21.15.70:0/3002089958 >> [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707] conn(0x7f59880bc5d0 0x7f59880bea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:10.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.481+0000 7f599cff9700 1 -- 172.21.15.70:0/3002089958 --> [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f59880bf140 con 0x7f59880bc5d0 2024-06-04T04:55:10.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.481+0000 7f599cff9700 1 -- 172.21.15.70:0/3002089958 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f59980e2670 con 0x7f59a0109520 2024-06-04T04:55:10.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.482+0000 7f599effd700 1 --2- 172.21.15.70:0/3002089958 >> [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707] conn(0x7f59880bc5d0 0x7f59880bea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:10.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.482+0000 7f599f7fe700 1 --2- 172.21.15.70:0/3002089958 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f59880b2990 0x7f59880b4e50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5990009990 tx=0x7f599001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:10.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.482+0000 7f599effd700 1 --2- 172.21.15.70:0/3002089958 >> [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707] conn(0x7f59880bc5d0 0x7f59880bea70 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:10.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.483+0000 7f599cff9700 1 -- 172.21.15.70:0/3002089958 <== osd.9 v2:172.21.15.70:6826/3557688707 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f59880bf140 con 0x7f59880bc5d0 2024-06-04T04:55:10.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.500+0000 7f59a5f47700 1 -- 172.21.15.70:0/3002089958 --> [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f59a0064110 con 0x7f59880bc5d0 2024-06-04T04:55:10.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.501+0000 7f599cff9700 1 -- 172.21.15.70:0/3002089958 <== osd.9 v2:172.21.15.70:6826/3557688707 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f59a0064110 con 0x7f59880bc5d0 2024-06-04T04:55:10.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.501+0000 7f59a5f47700 1 -- 172.21.15.70:0/3002089958 >> [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707] conn(0x7f59880bc5d0 msgr2=0x7f59880bea70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:10.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.501+0000 7f59a5f47700 1 --2- 172.21.15.70:0/3002089958 >> [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707] conn(0x7f59880bc5d0 0x7f59880bea70 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:10.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.501+0000 7f59a5f47700 1 -- 172.21.15.70:0/3002089958 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f59880b2990 msgr2=0x7f59880b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:10.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.501+0000 7f59a5f47700 1 --2- 172.21.15.70:0/3002089958 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f59880b2990 0x7f59880b4e50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5990009990 tx=0x7f599001a040 comp rx=0 tx=0).stop 2024-06-04T04:55:10.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.501+0000 7f59a5f47700 1 -- 172.21.15.70:0/3002089958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a0109520 msgr2=0x7f59a0199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:10.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.501+0000 7f59a5f47700 1 --2- 172.21.15.70:0/3002089958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a0109520 0x7f59a0199950 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5998007ee0 tx=0x7f599800d3b0 comp rx=0 tx=0).stop 2024-06-04T04:55:10.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.501+0000 7f599ffff700 1 -- 172.21.15.70:0/3002089958 reap_dead start 2024-06-04T04:55:10.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.501+0000 7f59a5f47700 1 -- 172.21.15.70:0/3002089958 shutdown_connections 2024-06-04T04:55:10.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.501+0000 7f59a5f47700 1 -- 172.21.15.70:0/3002089958 >> 172.21.15.70:0/3002089958 conn(0x7f59a0078ab0 msgr2=0x7f59a01003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:10.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.501+0000 7f59a5f47700 1 -- 172.21.15.70:0/3002089958 shutdown_connections 2024-06-04T04:55:10.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:10.502+0000 7f59a5f47700 1 -- 172.21.15.70:0/3002089958 wait complete. 2024-06-04T04:55:11.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:10 smithi070 ceph-mon[32631]: pgmap v337: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:55:11.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:10 smithi112 ceph-mon[36453]: pgmap v337: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:55:11.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:10 smithi098 ceph-mon[35569]: pgmap v337: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:55:11.431 INFO:teuthology.orchestra.run.smithi070.stdout:158913790017 2024-06-04T04:55:11.432 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-04T04:55:12.079 INFO:teuthology.orchestra.run.smithi070.stdout:231928234034 2024-06-04T04:55:12.079 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-06-04T04:55:12.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.276+0000 7f2ffcc00700 1 -- 172.21.15.70:0/1061871498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff806bbe0 msgr2=0x7f2ff81103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:12.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.276+0000 7f2ffcc00700 1 --2- 172.21.15.70:0/1061871498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff806bbe0 0x7f2ff81103d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000b0a0 tx=0x7f2ff000b3b0 comp rx=0 tx=0).stop 2024-06-04T04:55:12.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.276+0000 7f2ffcc00700 1 -- 172.21.15.70:0/1061871498 shutdown_connections 2024-06-04T04:55:12.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.276+0000 7f2ffcc00700 1 --2- 172.21.15.70:0/1061871498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff806bbe0 0x7f2ff81103d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:12.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.276+0000 7f2ffcc00700 1 --2- 172.21.15.70:0/1061871498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff81095b0 0x7f2ff8109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:12.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.276+0000 7f2ffcc00700 1 --2- 172.21.15.70:0/1061871498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff806b220 0x7f2ff806b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:12.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.276+0000 7f2ffcc00700 1 -- 172.21.15.70:0/1061871498 >> 172.21.15.70:0/1061871498 conn(0x7f2ff80793f0 msgr2=0x7f2ff8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:12.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.276+0000 7f2ffcc00700 1 -- 172.21.15.70:0/1061871498 shutdown_connections 2024-06-04T04:55:12.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.276+0000 7f2ffcc00700 1 -- 172.21.15.70:0/1061871498 wait complete. 2024-06-04T04:55:12.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.277+0000 7f2ffcc00700 1 Processor -- start 2024-06-04T04:55:12.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.277+0000 7f2ffcc00700 1 -- start start 2024-06-04T04:55:12.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.277+0000 7f2ffcc00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff806b220 0x7f2ff819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:12.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.277+0000 7f2ffcc00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff806bbe0 0x7f2ff819fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:12.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.277+0000 7f2ffcc00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff81095b0 0x7f2ff8199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:12.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.277+0000 7f2ffcc00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff8113570 con 0x7f2ff806bbe0 2024-06-04T04:55:12.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.277+0000 7f2ffcc00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff81133f0 con 0x7f2ff81095b0 2024-06-04T04:55:12.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.277+0000 7f2ffcc00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff81136f0 con 0x7f2ff806b220 2024-06-04T04:55:12.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.277+0000 7f2ff6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff81095b0 0x7f2ff8199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:12.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.277+0000 7f2ff5d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff806bbe0 0x7f2ff819fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.277+0000 7f2ff6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff81095b0 0x7f2ff8199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:38006/0 (socket says 172.21.15.70:38006) 2024-06-04T04:55:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.278+0000 7f2ff6d9d700 1 -- 172.21.15.70:0/3956482316 learned_addr learned my addr 172.21.15.70:0/3956482316 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.278+0000 7f2ff659c700 1 --2- 172.21.15.70:0/3956482316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff806b220 0x7f2ff819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.278+0000 7f2ff6d9d700 1 -- 172.21.15.70:0/3956482316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff806b220 msgr2=0x7f2ff819f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.278+0000 7f2ff6d9d700 1 --2- 172.21.15.70:0/3956482316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff806b220 0x7f2ff819f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.278+0000 7f2ff6d9d700 1 -- 172.21.15.70:0/3956482316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff806bbe0 msgr2=0x7f2ff819fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.278+0000 7f2ff6d9d700 1 --2- 172.21.15.70:0/3956482316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff806bbe0 0x7f2ff819fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.278+0000 7f2ff6d9d700 1 -- 172.21.15.70:0/3956482316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff819a070 con 0x7f2ff81095b0 2024-06-04T04:55:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.278+0000 7f2ff5d9b700 1 --2- 172.21.15.70:0/3956482316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff806bbe0 0x7f2ff819fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:55:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.278+0000 7f2ff6d9d700 1 --2- 172.21.15.70:0/3956482316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff81095b0 0x7f2ff8199910 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00062a0 tx=0x7f2ff0004e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.278+0000 7f2fe77fe700 1 -- 172.21.15.70:0/3956482316 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0021070 con 0x7f2ff81095b0 2024-06-04T04:55:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.278+0000 7f2ffcc00700 1 -- 172.21.15.70:0/3956482316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ff819a300 con 0x7f2ff81095b0 2024-06-04T04:55:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.278+0000 7f2fe77fe700 1 -- 172.21.15.70:0/3956482316 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2ff0007c00 con 0x7f2ff81095b0 2024-06-04T04:55:12.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.278+0000 7f2fe77fe700 1 -- 172.21.15.70:0/3956482316 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff00052a0 con 0x7f2ff81095b0 2024-06-04T04:55:12.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.278+0000 7f2ffcc00700 1 -- 172.21.15.70:0/3956482316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ff81a6620 con 0x7f2ff81095b0 2024-06-04T04:55:12.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.279+0000 7f2ffcc00700 1 -- 172.21.15.70:0/3956482316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2ff8063c50 con 0x7f2ff81095b0 2024-06-04T04:55:12.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.281+0000 7f2fe77fe700 1 -- 172.21.15.70:0/3956482316 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f2ff0024070 con 0x7f2ff81095b0 2024-06-04T04:55:12.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.281+0000 7f2fe77fe700 1 --2- 172.21.15.70:0/3956482316 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2fe00b2910 0x7f2fe00b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:12.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.282+0000 7f2fe77fe700 1 -- 172.21.15.70:0/3956482316 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f2ff000f020 con 0x7f2ff81095b0 2024-06-04T04:55:12.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.282+0000 7f2ff659c700 1 --2- 172.21.15.70:0/3956482316 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2fe00b2910 0x7f2fe00b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:12.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.282+0000 7f2fe77fe700 1 --2- 172.21.15.70:0/3956482316 >> [v2:172.21.15.70:6802/824119100,v1:172.21.15.70:6803/824119100] conn(0x7f2fe00bc5b0 0x7f2fe00bea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:12.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.282+0000 7f2fe77fe700 1 -- 172.21.15.70:0/3956482316 --> [v2:172.21.15.70:6802/824119100,v1:172.21.15.70:6803/824119100] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2fe00bf120 con 0x7f2fe00bc5b0 2024-06-04T04:55:12.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.282+0000 7f2fe77fe700 1 -- 172.21.15.70:0/3956482316 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f2ff00e3fb0 con 0x7f2ff81095b0 2024-06-04T04:55:12.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.282+0000 7f2ff5d9b700 1 --2- 172.21.15.70:0/3956482316 >> [v2:172.21.15.70:6802/824119100,v1:172.21.15.70:6803/824119100] conn(0x7f2fe00bc5b0 0x7f2fe00bea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:12.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.282+0000 7f2ff659c700 1 --2- 172.21.15.70:0/3956482316 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2fe00b2910 0x7f2fe00b4dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2ff819aa50 tx=0x7f2fe800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:12.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.282+0000 7f2ff5d9b700 1 --2- 172.21.15.70:0/3956482316 >> [v2:172.21.15.70:6802/824119100,v1:172.21.15.70:6803/824119100] conn(0x7f2fe00bc5b0 0x7f2fe00bea50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:12.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.283+0000 7f2fe77fe700 1 -- 172.21.15.70:0/3956482316 <== osd.0 v2:172.21.15.70:6802/824119100 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2fe00bf120 con 0x7f2fe00bc5b0 2024-06-04T04:55:12.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.300+0000 7f2ffcc00700 1 -- 172.21.15.70:0/3956482316 --> [v2:172.21.15.70:6802/824119100,v1:172.21.15.70:6803/824119100] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2ff8064050 con 0x7f2fe00bc5b0 2024-06-04T04:55:12.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.300+0000 7f2fe77fe700 1 -- 172.21.15.70:0/3956482316 <== osd.0 v2:172.21.15.70:6802/824119100 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2ff8064050 con 0x7f2fe00bc5b0 2024-06-04T04:55:12.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.301+0000 7f2ffcc00700 1 -- 172.21.15.70:0/3956482316 >> [v2:172.21.15.70:6802/824119100,v1:172.21.15.70:6803/824119100] conn(0x7f2fe00bc5b0 msgr2=0x7f2fe00bea50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:12.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.301+0000 7f2ffcc00700 1 --2- 172.21.15.70:0/3956482316 >> [v2:172.21.15.70:6802/824119100,v1:172.21.15.70:6803/824119100] conn(0x7f2fe00bc5b0 0x7f2fe00bea50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:12.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.301+0000 7f2ffcc00700 1 -- 172.21.15.70:0/3956482316 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2fe00b2910 msgr2=0x7f2fe00b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:12.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.301+0000 7f2ffcc00700 1 --2- 172.21.15.70:0/3956482316 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2fe00b2910 0x7f2fe00b4dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2ff819aa50 tx=0x7f2fe800a380 comp rx=0 tx=0).stop 2024-06-04T04:55:12.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.301+0000 7f2ffcc00700 1 -- 172.21.15.70:0/3956482316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff81095b0 msgr2=0x7f2ff8199910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:12.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.301+0000 7f2ffcc00700 1 --2- 172.21.15.70:0/3956482316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff81095b0 0x7f2ff8199910 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00062a0 tx=0x7f2ff0004e20 comp rx=0 tx=0).stop 2024-06-04T04:55:12.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.301+0000 7f2ff6d9d700 1 -- 172.21.15.70:0/3956482316 reap_dead start 2024-06-04T04:55:12.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.301+0000 7f2ffcc00700 1 -- 172.21.15.70:0/3956482316 shutdown_connections 2024-06-04T04:55:12.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.301+0000 7f2ffcc00700 1 -- 172.21.15.70:0/3956482316 >> 172.21.15.70:0/3956482316 conn(0x7f2ff80793f0 msgr2=0x7f2ff8101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:12.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.301+0000 7f2ffcc00700 1 -- 172.21.15.70:0/3956482316 shutdown_connections 2024-06-04T04:55:12.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:12.301+0000 7f2ffcc00700 1 -- 172.21.15.70:0/3956482316 wait complete. 2024-06-04T04:55:13.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:12 smithi070 ceph-mon[32631]: pgmap v338: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-06-04T04:55:13.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:12 smithi112 ceph-mon[36453]: pgmap v338: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-06-04T04:55:13.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:12 smithi098 ceph-mon[35569]: pgmap v338: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-06-04T04:55:13.549 INFO:teuthology.orchestra.run.smithi070.stdout:34359738459 2024-06-04T04:55:13.549 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-04T04:55:13.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.746+0000 7fc4c425d700 1 -- 172.21.15.70:0/3389483146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4bc108500 msgr2=0x7fc4bc1088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:13.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.746+0000 7fc4c425d700 1 --2- 172.21.15.70:0/3389483146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4bc108500 0x7fc4bc1088e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8009990 tx=0x7fc4b8009ca0 comp rx=0 tx=0).stop 2024-06-04T04:55:13.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.746+0000 7fc4c425d700 1 -- 172.21.15.70:0/3389483146 shutdown_connections 2024-06-04T04:55:13.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.746+0000 7fc4c425d700 1 --2- 172.21.15.70:0/3389483146 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4bc103f90 0x7fc4bc110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:13.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.746+0000 7fc4c425d700 1 --2- 172.21.15.70:0/3389483146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4bc108500 0x7fc4bc1088e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:13.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.746+0000 7fc4c425d700 1 --2- 172.21.15.70:0/3389483146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4bc1035f0 0x7fc4bc103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:13.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.746+0000 7fc4c425d700 1 -- 172.21.15.70:0/3389483146 >> 172.21.15.70:0/3389483146 conn(0x7fc4bc07a6a0 msgr2=0x7fc4bc07aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:13.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.746+0000 7fc4c425d700 1 -- 172.21.15.70:0/3389483146 shutdown_connections 2024-06-04T04:55:13.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.746+0000 7fc4c425d700 1 -- 172.21.15.70:0/3389483146 wait complete. 2024-06-04T04:55:13.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.747+0000 7fc4c425d700 1 Processor -- start 2024-06-04T04:55:13.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.747+0000 7fc4c425d700 1 -- start start 2024-06-04T04:55:13.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.747+0000 7fc4c425d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4bc1035f0 0x7fc4bc19f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:13.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.747+0000 7fc4c425d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4bc103f90 0x7fc4bc19fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:13.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.747+0000 7fc4c425d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4bc108500 0x7fc4bc199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:13.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.747+0000 7fc4c425d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc113640 con 0x7fc4bc1035f0 2024-06-04T04:55:13.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.747+0000 7fc4c425d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc1134c0 con 0x7fc4bc108500 2024-06-04T04:55:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.747+0000 7fc4c425d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc1137c0 con 0x7fc4bc103f90 2024-06-04T04:55:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.748+0000 7fc4c1ff9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4bc1035f0 0x7fc4bc19f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.748+0000 7fc4c1ff9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4bc1035f0 0x7fc4bc19f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42304/0 (socket says 172.21.15.70:42304) 2024-06-04T04:55:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.748+0000 7fc4c1ff9700 1 -- 172.21.15.70:0/2831945956 learned_addr learned my addr 172.21.15.70:0/2831945956 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.748+0000 7fc4c27fa700 1 --2- 172.21.15.70:0/2831945956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4bc108500 0x7fc4bc199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.748+0000 7fc4c17f8700 1 --2- 172.21.15.70:0/2831945956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4bc103f90 0x7fc4bc19fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.748+0000 7fc4c1ff9700 1 -- 172.21.15.70:0/2831945956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4bc103f90 msgr2=0x7fc4bc19fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.748+0000 7fc4c1ff9700 1 --2- 172.21.15.70:0/2831945956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4bc103f90 0x7fc4bc19fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.748+0000 7fc4c1ff9700 1 -- 172.21.15.70:0/2831945956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4bc108500 msgr2=0x7fc4bc199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.748+0000 7fc4c1ff9700 1 --2- 172.21.15.70:0/2831945956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4bc108500 0x7fc4bc199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.748+0000 7fc4c1ff9700 1 -- 172.21.15.70:0/2831945956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4bc19a160 con 0x7fc4bc1035f0 2024-06-04T04:55:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.748+0000 7fc4c17f8700 1 --2- 172.21.15.70:0/2831945956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4bc103f90 0x7fc4bc19fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:55:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.748+0000 7fc4c1ff9700 1 --2- 172.21.15.70:0/2831945956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4bc1035f0 0x7fc4bc19f840 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80047b0 tx=0x7fc4b800bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:13.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.748+0000 7fc4b2ffd700 1 -- 172.21.15.70:0/2831945956 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8026070 con 0x7fc4bc1035f0 2024-06-04T04:55:13.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.749+0000 7fc4c425d700 1 -- 172.21.15.70:0/2831945956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4bc19a3f0 con 0x7fc4bc1035f0 2024-06-04T04:55:13.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.749+0000 7fc4b2ffd700 1 -- 172.21.15.70:0/2831945956 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc4b8021410 con 0x7fc4bc1035f0 2024-06-04T04:55:13.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.749+0000 7fc4b2ffd700 1 -- 172.21.15.70:0/2831945956 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8017650 con 0x7fc4bc1035f0 2024-06-04T04:55:13.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.749+0000 7fc4c425d700 1 -- 172.21.15.70:0/2831945956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4bc1a6710 con 0x7fc4bc1035f0 2024-06-04T04:55:13.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.750+0000 7fc4c425d700 1 -- 172.21.15.70:0/2831945956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc4bc063cd0 con 0x7fc4bc1035f0 2024-06-04T04:55:13.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.752+0000 7fc4b2ffd700 1 -- 172.21.15.70:0/2831945956 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fc4b8029070 con 0x7fc4bc1035f0 2024-06-04T04:55:13.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.753+0000 7fc4b2ffd700 1 --2- 172.21.15.70:0/2831945956 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc4a80b2990 0x7fc4a80b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:13.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.753+0000 7fc4c17f8700 1 --2- 172.21.15.70:0/2831945956 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc4a80b2990 0x7fc4a80b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:13.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.754+0000 7fc4b2ffd700 1 -- 172.21.15.70:0/2831945956 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fc4b80e3d40 con 0x7fc4bc1035f0 2024-06-04T04:55:13.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.754+0000 7fc4b2ffd700 1 --2- 172.21.15.70:0/2831945956 >> [v2:172.21.15.98:6808/3652775814,v1:172.21.15.98:6809/3652775814] conn(0x7fc4a80bc5d0 0x7fc4a80bea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:13.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.754+0000 7fc4b2ffd700 1 -- 172.21.15.70:0/2831945956 --> [v2:172.21.15.98:6808/3652775814,v1:172.21.15.98:6809/3652775814] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc4a80bf140 con 0x7fc4a80bc5d0 2024-06-04T04:55:13.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.754+0000 7fc4b2ffd700 1 -- 172.21.15.70:0/2831945956 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fc4b80e4190 con 0x7fc4bc1035f0 2024-06-04T04:55:13.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.754+0000 7fc4c17f8700 1 --2- 172.21.15.70:0/2831945956 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc4a80b2990 0x7fc4a80b4e50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc07ad80 tx=0x7fc4ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:13.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.754+0000 7fc4c27fa700 1 --2- 172.21.15.70:0/2831945956 >> [v2:172.21.15.98:6808/3652775814,v1:172.21.15.98:6809/3652775814] conn(0x7fc4a80bc5d0 0x7fc4a80bea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:13.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.755+0000 7fc4c27fa700 1 --2- 172.21.15.70:0/2831945956 >> [v2:172.21.15.98:6808/3652775814,v1:172.21.15.98:6809/3652775814] conn(0x7fc4a80bc5d0 0x7fc4a80bea70 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:13.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.755+0000 7fc4b2ffd700 1 -- 172.21.15.70:0/2831945956 <== osd.4 v2:172.21.15.98:6808/3652775814 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc4a80bf140 con 0x7fc4a80bc5d0 2024-06-04T04:55:13.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.779+0000 7fc4c425d700 1 -- 172.21.15.70:0/2831945956 --> [v2:172.21.15.98:6808/3652775814,v1:172.21.15.98:6809/3652775814] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc4bc0640d0 con 0x7fc4a80bc5d0 2024-06-04T04:55:13.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.780+0000 7fc4b2ffd700 1 -- 172.21.15.70:0/2831945956 <== osd.4 v2:172.21.15.98:6808/3652775814 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc4bc0640d0 con 0x7fc4a80bc5d0 2024-06-04T04:55:13.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.780+0000 7fc4c425d700 1 -- 172.21.15.70:0/2831945956 >> [v2:172.21.15.98:6808/3652775814,v1:172.21.15.98:6809/3652775814] conn(0x7fc4a80bc5d0 msgr2=0x7fc4a80bea70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:13.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.780+0000 7fc4c425d700 1 --2- 172.21.15.70:0/2831945956 >> [v2:172.21.15.98:6808/3652775814,v1:172.21.15.98:6809/3652775814] conn(0x7fc4a80bc5d0 0x7fc4a80bea70 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:13.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.780+0000 7fc4c425d700 1 -- 172.21.15.70:0/2831945956 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc4a80b2990 msgr2=0x7fc4a80b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:13.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.780+0000 7fc4c425d700 1 --2- 172.21.15.70:0/2831945956 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc4a80b2990 0x7fc4a80b4e50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc07ad80 tx=0x7fc4ac009260 comp rx=0 tx=0).stop 2024-06-04T04:55:13.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.780+0000 7fc4c425d700 1 -- 172.21.15.70:0/2831945956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4bc1035f0 msgr2=0x7fc4bc19f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:13.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.780+0000 7fc4c425d700 1 --2- 172.21.15.70:0/2831945956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4bc1035f0 0x7fc4bc19f840 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80047b0 tx=0x7fc4b800bfa0 comp rx=0 tx=0).stop 2024-06-04T04:55:13.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.780+0000 7fc4c27fa700 1 -- 172.21.15.70:0/2831945956 reap_dead start 2024-06-04T04:55:13.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.780+0000 7fc4c425d700 1 -- 172.21.15.70:0/2831945956 shutdown_connections 2024-06-04T04:55:13.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.780+0000 7fc4c425d700 1 -- 172.21.15.70:0/2831945956 >> 172.21.15.70:0/2831945956 conn(0x7fc4bc07a6a0 msgr2=0x7fc4bc1017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:13.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.781+0000 7fc4c425d700 1 -- 172.21.15.70:0/2831945956 shutdown_connections 2024-06-04T04:55:13.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:13.781+0000 7fc4c425d700 1 -- 172.21.15.70:0/2831945956 wait complete. 2024-06-04T04:55:14.814 INFO:teuthology.orchestra.run.smithi070.stdout:137438953543 2024-06-04T04:55:14.815 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-04T04:55:15.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:14 smithi070 ceph-mon[32631]: pgmap v339: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-06-04T04:55:15.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:14 smithi112 ceph-mon[36453]: pgmap v339: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-06-04T04:55:15.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:14 smithi098 ceph-mon[35569]: pgmap v339: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-06-04T04:55:15.495 INFO:teuthology.orchestra.run.smithi070.stdout:55834574933 2024-06-04T04:55:15.496 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-04T04:55:15.964 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:16.005 INFO:teuthology.orchestra.run.smithi070.stdout:257698037805 2024-06-04T04:55:16.005 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-06-04T04:55:16.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.243+0000 7fa104714700 1 -- 172.21.15.70:0/286113541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0fc102c40 msgr2=0x7fa0fc1030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:16.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.243+0000 7fa104714700 1 --2- 172.21.15.70:0/286113541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0fc102c40 0x7fa0fc1030c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec009990 tx=0x7fa0ec009ca0 comp rx=0 tx=0).stop 2024-06-04T04:55:16.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.243+0000 7fa104714700 1 -- 172.21.15.70:0/286113541 shutdown_connections 2024-06-04T04:55:16.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.243+0000 7fa104714700 1 --2- 172.21.15.70:0/286113541 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0fc103600 0x7fa0fc10f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:16.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.243+0000 7fa104714700 1 --2- 172.21.15.70:0/286113541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0fc1084c0 0x7fa0fc1088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:16.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.243+0000 7fa104714700 1 --2- 172.21.15.70:0/286113541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0fc102c40 0x7fa0fc1030c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:16.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.243+0000 7fa104714700 1 -- 172.21.15.70:0/286113541 >> 172.21.15.70:0/286113541 conn(0x7fa0fc078ab0 msgr2=0x7fa0fc0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:16.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.243+0000 7fa104714700 1 -- 172.21.15.70:0/286113541 shutdown_connections 2024-06-04T04:55:16.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.243+0000 7fa104714700 1 -- 172.21.15.70:0/286113541 wait complete. 2024-06-04T04:55:16.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.243+0000 7fa104714700 1 Processor -- start 2024-06-04T04:55:16.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.243+0000 7fa104714700 1 -- start start 2024-06-04T04:55:16.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa104714700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0fc102c40 0x7fa0fc19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:16.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa104714700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0fc103600 0x7fa0fc19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:16.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa104714700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0fc1084c0 0x7fa0fc1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:16.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa104714700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa0fc1124a0 con 0x7fa0fc102c40 2024-06-04T04:55:16.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa104714700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa0fc112320 con 0x7fa0fc1084c0 2024-06-04T04:55:16.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa104714700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa0fc112620 con 0x7fa0fc103600 2024-06-04T04:55:16.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa1024b0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0fc102c40 0x7fa0fc19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:16.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa1024b0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0fc102c40 0x7fa0fc19e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51328/0 (socket says 172.21.15.70:51328) 2024-06-04T04:55:16.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa1024b0700 1 -- 172.21.15.70:0/3861250586 learned_addr learned my addr 172.21.15.70:0/3861250586 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:16.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa101caf700 1 --2- 172.21.15.70:0/3861250586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0fc103600 0x7fa0fc19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:16.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa102cb1700 1 --2- 172.21.15.70:0/3861250586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0fc1084c0 0x7fa0fc1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:16.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa1024b0700 1 -- 172.21.15.70:0/3861250586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0fc103600 msgr2=0x7fa0fc19eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:16.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa1024b0700 1 --2- 172.21.15.70:0/3861250586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0fc103600 0x7fa0fc19eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa1024b0700 1 -- 172.21.15.70:0/3861250586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0fc1084c0 msgr2=0x7fa0fc1987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa1024b0700 1 --2- 172.21.15.70:0/3861250586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0fc1084c0 0x7fa0fc1987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa1024b0700 1 -- 172.21.15.70:0/3861250586 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0fc198fc0 con 0x7fa0fc102c40 2024-06-04T04:55:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa102cb1700 1 --2- 172.21.15.70:0/3861250586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0fc1084c0 0x7fa0fc1987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:55:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa1024b0700 1 --2- 172.21.15.70:0/3861250586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0fc102c40 0x7fa0fc19e660 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800d980 tx=0x7fa0f800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa0f37fe700 1 -- 172.21.15.70:0/3861250586 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8017030 con 0x7fa0fc102c40 2024-06-04T04:55:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa0f37fe700 1 -- 172.21.15.70:0/3861250586 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa0f80044b0 con 0x7fa0fc102c40 2024-06-04T04:55:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa104714700 1 -- 172.21.15.70:0/3861250586 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0fc1992b0 con 0x7fa0fc102c40 2024-06-04T04:55:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.244+0000 7fa0f37fe700 1 -- 172.21.15.70:0/3861250586 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f80038e0 con 0x7fa0fc102c40 2024-06-04T04:55:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.245+0000 7fa104714700 1 -- 172.21.15.70:0/3861250586 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0fc101b30 con 0x7fa0fc102c40 2024-06-04T04:55:16.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.245+0000 7fa104714700 1 -- 172.21.15.70:0/3861250586 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa0fc102110 con 0x7fa0fc102c40 2024-06-04T04:55:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.247+0000 7fa0f37fe700 1 -- 172.21.15.70:0/3861250586 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fa0f801a070 con 0x7fa0fc102c40 2024-06-04T04:55:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.247+0000 7fa0f37fe700 1 --2- 172.21.15.70:0/3861250586 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa0e80b2990 0x7fa0e80b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.248+0000 7fa101caf700 1 --2- 172.21.15.70:0/3861250586 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa0e80b2990 0x7fa0e80b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.248+0000 7fa0f37fe700 1 -- 172.21.15.70:0/3861250586 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fa0f80e2440 con 0x7fa0fc102c40 2024-06-04T04:55:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.248+0000 7fa0f37fe700 1 --2- 172.21.15.70:0/3861250586 >> [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496] conn(0x7fa0e80bc5d0 0x7fa0e80bea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.248+0000 7fa0f37fe700 1 -- 172.21.15.70:0/3861250586 --> [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa0f800d130 con 0x7fa0e80bc5d0 2024-06-04T04:55:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.248+0000 7fa0f37fe700 1 -- 172.21.15.70:0/3861250586 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fa0f80e2890 con 0x7fa0fc102c40 2024-06-04T04:55:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.248+0000 7fa102cb1700 1 --2- 172.21.15.70:0/3861250586 >> [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496] conn(0x7fa0e80bc5d0 0x7fa0e80bea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.248+0000 7fa101caf700 1 --2- 172.21.15.70:0/3861250586 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa0e80b2990 0x7fa0e80b4e50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec00b5c0 tx=0x7fa0ec005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.248+0000 7fa102cb1700 1 --2- 172.21.15.70:0/3861250586 >> [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496] conn(0x7fa0e80bc5d0 0x7fa0e80bea70 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:16.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.249+0000 7fa0f37fe700 1 -- 172.21.15.70:0/3861250586 <== osd.6 v2:172.21.15.70:6818/1264676496 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa0f800d130 con 0x7fa0e80bc5d0 2024-06-04T04:55:16.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.267+0000 7fa104714700 1 -- 172.21.15.70:0/3861250586 --> [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa0fc102590 con 0x7fa0e80bc5d0 2024-06-04T04:55:16.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.267+0000 7fa0f37fe700 1 -- 172.21.15.70:0/3861250586 <== osd.6 v2:172.21.15.70:6818/1264676496 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa0fc102590 con 0x7fa0e80bc5d0 2024-06-04T04:55:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.268+0000 7fa104714700 1 -- 172.21.15.70:0/3861250586 >> [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496] conn(0x7fa0e80bc5d0 msgr2=0x7fa0e80bea70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.268+0000 7fa104714700 1 --2- 172.21.15.70:0/3861250586 >> [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496] conn(0x7fa0e80bc5d0 0x7fa0e80bea70 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.268+0000 7fa104714700 1 -- 172.21.15.70:0/3861250586 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa0e80b2990 msgr2=0x7fa0e80b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.268+0000 7fa104714700 1 --2- 172.21.15.70:0/3861250586 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa0e80b2990 0x7fa0e80b4e50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec00b5c0 tx=0x7fa0ec005960 comp rx=0 tx=0).stop 2024-06-04T04:55:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.268+0000 7fa104714700 1 -- 172.21.15.70:0/3861250586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0fc102c40 msgr2=0x7fa0fc19e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.268+0000 7fa104714700 1 --2- 172.21.15.70:0/3861250586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0fc102c40 0x7fa0fc19e660 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800d980 tx=0x7fa0f800dc90 comp rx=0 tx=0).stop 2024-06-04T04:55:16.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.268+0000 7fa102cb1700 1 -- 172.21.15.70:0/3861250586 reap_dead start 2024-06-04T04:55:16.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.268+0000 7fa104714700 1 -- 172.21.15.70:0/3861250586 shutdown_connections 2024-06-04T04:55:16.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.268+0000 7fa104714700 1 -- 172.21.15.70:0/3861250586 >> 172.21.15.70:0/3861250586 conn(0x7fa0fc078ab0 msgr2=0x7fa0fc1002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:16.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.268+0000 7fa104714700 1 -- 172.21.15.70:0/3861250586 shutdown_connections 2024-06-04T04:55:16.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:16.268+0000 7fa104714700 1 -- 172.21.15.70:0/3861250586 wait complete. 2024-06-04T04:55:17.144 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:16 smithi070 ceph-mon[32631]: pgmap v340: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-06-04T04:55:17.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:16 smithi112 ceph-mon[36453]: pgmap v340: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-06-04T04:55:17.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:16 smithi098 ceph-mon[35569]: pgmap v340: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-06-04T04:55:17.344 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:17.345 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:17.347 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:17.348 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:17.381 INFO:teuthology.orchestra.run.smithi070.stdout:184683593790 2024-06-04T04:55:17.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-06-04T04:55:17.676 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:17.677 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.611+0000 7f61319f9700 1 -- 172.21.15.70:0/2750265836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f612c1095a0 msgr2=0x7f612c109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.611+0000 7f61319f9700 1 --2- 172.21.15.70:0/2750265836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f612c1095a0 0x7f612c109980 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f611c009990 tx=0x7f611c009ca0 comp rx=0 tx=0).stop 2024-06-04T04:55:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.611+0000 7f61319f9700 1 -- 172.21.15.70:0/2750265836 shutdown_connections 2024-06-04T04:55:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.611+0000 7f61319f9700 1 --2- 172.21.15.70:0/2750265836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f612c06c470 0x7f612c1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.611+0000 7f61319f9700 1 --2- 172.21.15.70:0/2750265836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f612c1095a0 0x7f612c109980 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.611+0000 7f61319f9700 1 --2- 172.21.15.70:0/2750265836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f612c06bab0 0x7f612c06bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.611+0000 7f61319f9700 1 -- 172.21.15.70:0/2750265836 >> 172.21.15.70:0/2750265836 conn(0x7f612c0793b0 msgr2=0x7f612c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.611+0000 7f61319f9700 1 -- 172.21.15.70:0/2750265836 shutdown_connections 2024-06-04T04:55:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.611+0000 7f61319f9700 1 -- 172.21.15.70:0/2750265836 wait complete. 2024-06-04T04:55:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f61319f9700 1 Processor -- start 2024-06-04T04:55:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f61319f9700 1 -- start start 2024-06-04T04:55:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f61319f9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f612c06bab0 0x7f612c102500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f61319f9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f612c06c470 0x7f612c102a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f61319f9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f612c1095a0 0x7f612c102f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f61319f9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f612c113600 con 0x7f612c06c470 2024-06-04T04:55:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f61319f9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f612c113480 con 0x7f612c1095a0 2024-06-04T04:55:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f61319f9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f612c113780 con 0x7f612c06bab0 2024-06-04T04:55:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f61227fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f612c06c470 0x7f612c102a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f61227fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f612c06c470 0x7f612c102a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51338/0 (socket says 172.21.15.70:51338) 2024-06-04T04:55:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f61227fc700 1 -- 172.21.15.70:0/4007110959 learned_addr learned my addr 172.21.15.70:0/4007110959 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f612affd700 1 --2- 172.21.15.70:0/4007110959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f612c06bab0 0x7f612c102500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f612b7fe700 1 --2- 172.21.15.70:0/4007110959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f612c1095a0 0x7f612c102f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f61227fc700 1 -- 172.21.15.70:0/4007110959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f612c06bab0 msgr2=0x7f612c102500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f61227fc700 1 --2- 172.21.15.70:0/4007110959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f612c06bab0 0x7f612c102500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f61227fc700 1 -- 172.21.15.70:0/4007110959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f612c1095a0 msgr2=0x7f612c102f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f61227fc700 1 --2- 172.21.15.70:0/4007110959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f612c1095a0 0x7f612c102f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.612+0000 7f61227fc700 1 -- 172.21.15.70:0/4007110959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612c1a80c0 con 0x7f612c06c470 2024-06-04T04:55:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.613+0000 7f61227fc700 1 --2- 172.21.15.70:0/4007110959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f612c06c470 0x7f612c102a40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f611400d980 tx=0x7f611400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.613+0000 7f61309f7700 1 -- 172.21.15.70:0/4007110959 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6114017070 con 0x7f612c06c470 2024-06-04T04:55:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.613+0000 7f61319f9700 1 -- 172.21.15.70:0/4007110959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f612c1a83b0 con 0x7f612c06c470 2024-06-04T04:55:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.613+0000 7f61309f7700 1 -- 172.21.15.70:0/4007110959 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6114005020 con 0x7f612c06c470 2024-06-04T04:55:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.613+0000 7f61309f7700 1 -- 172.21.15.70:0/4007110959 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6114009ab0 con 0x7f612c06c470 2024-06-04T04:55:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.613+0000 7f61319f9700 1 -- 172.21.15.70:0/4007110959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f612c1a88a0 con 0x7f612c06c470 2024-06-04T04:55:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.613+0000 7f61319f9700 1 -- 172.21.15.70:0/4007110959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f612c063c50 con 0x7f612c06c470 2024-06-04T04:55:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.615+0000 7f61309f7700 1 -- 172.21.15.70:0/4007110959 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f61140051d0 con 0x7f612c06c470 2024-06-04T04:55:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.616+0000 7f61309f7700 1 --2- 172.21.15.70:0/4007110959 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f61180b2880 0x7f61180b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.616+0000 7f612affd700 1 --2- 172.21.15.70:0/4007110959 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f61180b2880 0x7f61180b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.616+0000 7f61309f7700 1 -- 172.21.15.70:0/4007110959 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f61140e26a0 con 0x7f612c06c470 2024-06-04T04:55:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.616+0000 7f61309f7700 1 --2- 172.21.15.70:0/4007110959 >> [v2:172.21.15.98:6824/1896493718,v1:172.21.15.98:6825/1896493718] conn(0x7f61180bc4c0 0x7f61180be960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.616+0000 7f61309f7700 1 -- 172.21.15.70:0/4007110959 --> [v2:172.21.15.98:6824/1896493718,v1:172.21.15.98:6825/1896493718] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f61180bf030 con 0x7f61180bc4c0 2024-06-04T04:55:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.616+0000 7f61309f7700 1 -- 172.21.15.70:0/4007110959 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f61140e2af0 con 0x7f612c06c470 2024-06-04T04:55:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.616+0000 7f612affd700 1 --2- 172.21.15.70:0/4007110959 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f61180b2880 0x7f61180b4d40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f611c00b5c0 tx=0x7f611c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.616+0000 7f612b7fe700 1 --2- 172.21.15.70:0/4007110959 >> [v2:172.21.15.98:6824/1896493718,v1:172.21.15.98:6825/1896493718] conn(0x7f61180bc4c0 0x7f61180be960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:18.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.617+0000 7f612b7fe700 1 --2- 172.21.15.70:0/4007110959 >> [v2:172.21.15.98:6824/1896493718,v1:172.21.15.98:6825/1896493718] conn(0x7f61180bc4c0 0x7f61180be960 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:18.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.617+0000 7f61309f7700 1 -- 172.21.15.70:0/4007110959 <== osd.10 v2:172.21.15.98:6824/1896493718 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61180bf030 con 0x7f61180bc4c0 2024-06-04T04:55:18.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.634+0000 7f61319f9700 1 -- 172.21.15.70:0/4007110959 --> [v2:172.21.15.98:6824/1896493718,v1:172.21.15.98:6825/1896493718] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f612c064050 con 0x7f61180bc4c0 2024-06-04T04:55:18.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.634+0000 7f61309f7700 1 -- 172.21.15.70:0/4007110959 <== osd.10 v2:172.21.15.98:6824/1896493718 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f612c064050 con 0x7f61180bc4c0 2024-06-04T04:55:18.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.634+0000 7f61319f9700 1 -- 172.21.15.70:0/4007110959 >> [v2:172.21.15.98:6824/1896493718,v1:172.21.15.98:6825/1896493718] conn(0x7f61180bc4c0 msgr2=0x7f61180be960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:18.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.634+0000 7f61319f9700 1 --2- 172.21.15.70:0/4007110959 >> [v2:172.21.15.98:6824/1896493718,v1:172.21.15.98:6825/1896493718] conn(0x7f61180bc4c0 0x7f61180be960 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:18.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.634+0000 7f61319f9700 1 -- 172.21.15.70:0/4007110959 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f61180b2880 msgr2=0x7f61180b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:18.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.634+0000 7f61319f9700 1 --2- 172.21.15.70:0/4007110959 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f61180b2880 0x7f61180b4d40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f611c00b5c0 tx=0x7f611c005960 comp rx=0 tx=0).stop 2024-06-04T04:55:18.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.634+0000 7f61319f9700 1 -- 172.21.15.70:0/4007110959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f612c06c470 msgr2=0x7f612c102a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:18.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.634+0000 7f61319f9700 1 --2- 172.21.15.70:0/4007110959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f612c06c470 0x7f612c102a40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f611400d980 tx=0x7f611400dc90 comp rx=0 tx=0).stop 2024-06-04T04:55:18.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.634+0000 7f612b7fe700 1 -- 172.21.15.70:0/4007110959 reap_dead start 2024-06-04T04:55:18.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.635+0000 7f61319f9700 1 -- 172.21.15.70:0/4007110959 shutdown_connections 2024-06-04T04:55:18.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.635+0000 7f61319f9700 1 -- 172.21.15.70:0/4007110959 >> 172.21.15.70:0/4007110959 conn(0x7f612c0793b0 msgr2=0x7f612c101330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:18.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.635+0000 7f61319f9700 1 -- 172.21.15.70:0/4007110959 shutdown_connections 2024-06-04T04:55:18.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:18.635+0000 7f61319f9700 1 -- 172.21.15.70:0/4007110959 wait complete. 2024-06-04T04:55:19.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:18 smithi070 ceph-mon[32631]: pgmap v341: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:55:19.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:18 smithi070 ceph-mon[32631]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-06-04T04:55:19.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:18 smithi112 ceph-mon[36453]: pgmap v341: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:55:19.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:18 smithi112 ceph-mon[36453]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-06-04T04:55:19.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:18 smithi098 ceph-mon[35569]: pgmap v341: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:55:19.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:18 smithi098 ceph-mon[35569]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-06-04T04:55:20.456 INFO:teuthology.orchestra.run.smithi070.stdout:279172874281 2024-06-04T04:55:20.456 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-06-04T04:55:21.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:20 smithi070 ceph-mon[32631]: pgmap v342: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-06-04T04:55:21.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:20 smithi112 ceph-mon[36453]: pgmap v342: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-06-04T04:55:21.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:20 smithi098 ceph-mon[35569]: pgmap v342: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-06-04T04:55:22.339 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:22.339 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:23.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:22 smithi070 ceph-mon[32631]: pgmap v343: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-06-04T04:55:23.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:22 smithi112 ceph-mon[36453]: pgmap v343: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-06-04T04:55:23.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:22 smithi098 ceph-mon[35569]: pgmap v343: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-06-04T04:55:25.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:24 smithi070 ceph-mon[32631]: pgmap v344: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 426 B/s wr, 11 op/s 2024-06-04T04:55:25.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:24 smithi112 ceph-mon[36453]: pgmap v344: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 426 B/s wr, 11 op/s 2024-06-04T04:55:25.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:24 smithi098 ceph-mon[35569]: pgmap v344: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 426 B/s wr, 11 op/s 2024-06-04T04:55:26.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.505+0000 7f33e913c700 1 -- 172.21.15.70:0/2128930174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33e406c470 msgr2=0x7f33e41103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:26.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.505+0000 7f33e913c700 1 --2- 172.21.15.70:0/2128930174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33e406c470 0x7f33e41103c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f33d400b0a0 tx=0x7f33d400b3b0 comp rx=0 tx=0).stop 2024-06-04T04:55:26.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.505+0000 7f33e913c700 1 -- 172.21.15.70:0/2128930174 shutdown_connections 2024-06-04T04:55:26.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.505+0000 7f33e913c700 1 --2- 172.21.15.70:0/2128930174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33e406c470 0x7f33e41103c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:26.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.505+0000 7f33e913c700 1 --2- 172.21.15.70:0/2128930174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33e41095a0 0x7f33e4109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:26.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.505+0000 7f33e913c700 1 --2- 172.21.15.70:0/2128930174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33e406bab0 0x7f33e406bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:26.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.505+0000 7f33e913c700 1 -- 172.21.15.70:0/2128930174 >> 172.21.15.70:0/2128930174 conn(0x7f33e40793b0 msgr2=0x7f33e40797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:26.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.505+0000 7f33e913c700 1 -- 172.21.15.70:0/2128930174 shutdown_connections 2024-06-04T04:55:26.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.505+0000 7f33e913c700 1 -- 172.21.15.70:0/2128930174 wait complete. 2024-06-04T04:55:26.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.506+0000 7f33e913c700 1 Processor -- start 2024-06-04T04:55:26.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.506+0000 7f33e913c700 1 -- start start 2024-06-04T04:55:26.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.506+0000 7f33e913c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33e406bab0 0x7f33e419f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:26.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.506+0000 7f33e913c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33e406c470 0x7f33e419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:26.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.506+0000 7f33e913c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33e41095a0 0x7f33e4199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:26.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.506+0000 7f33e913c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33e4113580 con 0x7f33e406bab0 2024-06-04T04:55:26.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.506+0000 7f33e913c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f33e4113400 con 0x7f33e406c470 2024-06-04T04:55:26.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.506+0000 7f33e913c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f33e4113700 con 0x7f33e41095a0 2024-06-04T04:55:26.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.506+0000 7f33e2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33e406bab0 0x7f33e419f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:26.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.506+0000 7f33e2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33e406bab0 0x7f33e419f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50368/0 (socket says 172.21.15.70:50368) 2024-06-04T04:55:26.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.506+0000 7f33e2d9d700 1 -- 172.21.15.70:0/3317405922 learned_addr learned my addr 172.21.15.70:0/3317405922 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:26.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.506+0000 7f33e359e700 1 --2- 172.21.15.70:0/3317405922 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33e41095a0 0x7f33e4199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:26.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.506+0000 7f33e259c700 1 --2- 172.21.15.70:0/3317405922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33e406c470 0x7f33e419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:26.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.507+0000 7f33e2d9d700 1 -- 172.21.15.70:0/3317405922 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33e41095a0 msgr2=0x7f33e4199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:26.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.507+0000 7f33e2d9d700 1 --2- 172.21.15.70:0/3317405922 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33e41095a0 0x7f33e4199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:26.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.507+0000 7f33e2d9d700 1 -- 172.21.15.70:0/3317405922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33e406c470 msgr2=0x7f33e419fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:26.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.507+0000 7f33e2d9d700 1 --2- 172.21.15.70:0/3317405922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33e406c470 0x7f33e419fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:26.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.507+0000 7f33e2d9d700 1 -- 172.21.15.70:0/3317405922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e419a080 con 0x7f33e406bab0 2024-06-04T04:55:26.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.507+0000 7f33e2d9d700 1 --2- 172.21.15.70:0/3317405922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33e406bab0 0x7f33e419f760 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f33dc00d980 tx=0x7f33dc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:26.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.507+0000 7f33dbfff700 1 -- 172.21.15.70:0/3317405922 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33dc017070 con 0x7f33e406bab0 2024-06-04T04:55:26.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.507+0000 7f33e913c700 1 -- 172.21.15.70:0/3317405922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33e419a310 con 0x7f33e406bab0 2024-06-04T04:55:26.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.507+0000 7f33dbfff700 1 -- 172.21.15.70:0/3317405922 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f33dc005020 con 0x7f33e406bab0 2024-06-04T04:55:26.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.507+0000 7f33dbfff700 1 -- 172.21.15.70:0/3317405922 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33dc009ab0 con 0x7f33e406bab0 2024-06-04T04:55:26.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.507+0000 7f33e913c700 1 -- 172.21.15.70:0/3317405922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33e41a6630 con 0x7f33e406bab0 2024-06-04T04:55:26.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.508+0000 7f33e913c700 1 -- 172.21.15.70:0/3317405922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33e406a230 con 0x7f33e406bab0 2024-06-04T04:55:26.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.510+0000 7f33dbfff700 1 -- 172.21.15.70:0/3317405922 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f33dc0051d0 con 0x7f33e406bab0 2024-06-04T04:55:26.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.511+0000 7f33dbfff700 1 --2- 172.21.15.70:0/3317405922 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f33d00b2880 0x7f33d00b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:26.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.511+0000 7f33e259c700 1 --2- 172.21.15.70:0/3317405922 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f33d00b2880 0x7f33d00b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:26.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.511+0000 7f33dbfff700 1 -- 172.21.15.70:0/3317405922 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f33dc0e24b0 con 0x7f33e406bab0 2024-06-04T04:55:26.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.511+0000 7f33e259c700 1 --2- 172.21.15.70:0/3317405922 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f33d00b2880 0x7f33d00b4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f33cc005950 tx=0x7f33cc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:26.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.515+0000 7f33dbfff700 1 -- 172.21.15.70:0/3317405922 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f33dc020020 con 0x7f33e406bab0 2024-06-04T04:55:26.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.724+0000 7f33e913c700 1 -- 172.21.15.70:0/3317405922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f33e402e560 con 0x7f33e406bab0 2024-06-04T04:55:26.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.724+0000 7f33dbfff700 1 -- 172.21.15.70:0/3317405922 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f33dc0aa1b0 con 0x7f33e406bab0 2024-06-04T04:55:26.726 INFO:teuthology.orchestra.run.smithi070.stdout:111669149775 2024-06-04T04:55:26.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.726+0000 7f33e913c700 1 -- 172.21.15.70:0/3317405922 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f33d00b2880 msgr2=0x7f33d00b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:26.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.726+0000 7f33e913c700 1 --2- 172.21.15.70:0/3317405922 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f33d00b2880 0x7f33d00b4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f33cc005950 tx=0x7f33cc0058e0 comp rx=0 tx=0).stop 2024-06-04T04:55:26.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.726+0000 7f33e913c700 1 -- 172.21.15.70:0/3317405922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33e406bab0 msgr2=0x7f33e419f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:26.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.726+0000 7f33e913c700 1 --2- 172.21.15.70:0/3317405922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33e406bab0 0x7f33e419f760 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f33dc00d980 tx=0x7f33dc00dc90 comp rx=0 tx=0).stop 2024-06-04T04:55:26.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.727+0000 7f33e913c700 1 -- 172.21.15.70:0/3317405922 shutdown_connections 2024-06-04T04:55:26.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.727+0000 7f33e913c700 1 --2- 172.21.15.70:0/3317405922 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f33d00b2880 0x7f33d00b4d40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:26.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.727+0000 7f33e913c700 1 --2- 172.21.15.70:0/3317405922 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33e41095a0 0x7f33e4199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:26.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.727+0000 7f33e913c700 1 --2- 172.21.15.70:0/3317405922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33e406bab0 0x7f33e419f760 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:26.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.727+0000 7f33e913c700 1 --2- 172.21.15.70:0/3317405922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33e406c470 0x7f33e419fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:26.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.727+0000 7f33e913c700 1 -- 172.21.15.70:0/3317405922 >> 172.21.15.70:0/3317405922 conn(0x7f33e40793b0 msgr2=0x7f33e4101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:26.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.727+0000 7f33e913c700 1 -- 172.21.15.70:0/3317405922 shutdown_connections 2024-06-04T04:55:26.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:26.727+0000 7f33e913c700 1 -- 172.21.15.70:0/3317405922 wait complete. 2024-06-04T04:55:27.034 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:26 smithi070 ceph-mon[32631]: pgmap v345: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-06-04T04:55:27.035 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:26 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3317405922' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-04T04:55:27.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:26 smithi112 ceph-mon[36453]: pgmap v345: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-06-04T04:55:27.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:26 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3317405922' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-04T04:55:27.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:26 smithi098 ceph-mon[35569]: pgmap v345: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-06-04T04:55:27.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:26 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3317405922' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-04T04:55:27.685 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:28.966 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:28 smithi070 ceph-mon[32631]: pgmap v346: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 10 op/s 2024-06-04T04:55:28.994 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149769 got 111669149775 for osd.3 2024-06-04T04:55:28.994 DEBUG:teuthology.parallel:result is None 2024-06-04T04:55:29.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:28 smithi112 ceph-mon[36453]: pgmap v346: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 10 op/s 2024-06-04T04:55:29.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:28 smithi098 ceph-mon[35569]: pgmap v346: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 10 op/s 2024-06-04T04:55:29.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.415+0000 7f31b1063700 1 -- 172.21.15.70:0/1970368851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31ac1035d0 msgr2=0x7f31ac103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:29.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.415+0000 7f31b1063700 1 --2- 172.21.15.70:0/1970368851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31ac1035d0 0x7f31ac103a50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f31940099c0 tx=0x7f3194009cd0 comp rx=0 tx=0).stop 2024-06-04T04:55:29.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.415+0000 7f31b1063700 1 -- 172.21.15.70:0/1970368851 shutdown_connections 2024-06-04T04:55:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.415+0000 7f31b1063700 1 --2- 172.21.15.70:0/1970368851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ac103f90 0x7f31ac110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.415+0000 7f31b1063700 1 --2- 172.21.15.70:0/1970368851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31ac109670 0x7f31ac109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.415+0000 7f31b1063700 1 --2- 172.21.15.70:0/1970368851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31ac1035d0 0x7f31ac103a50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.415+0000 7f31b1063700 1 -- 172.21.15.70:0/1970368851 >> 172.21.15.70:0/1970368851 conn(0x7f31ac07a730 msgr2=0x7f31ac07ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.415+0000 7f31b1063700 1 -- 172.21.15.70:0/1970368851 shutdown_connections 2024-06-04T04:55:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.415+0000 7f31b1063700 1 -- 172.21.15.70:0/1970368851 wait complete. 2024-06-04T04:55:29.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.416+0000 7f31b1063700 1 Processor -- start 2024-06-04T04:55:29.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.416+0000 7f31b1063700 1 -- start start 2024-06-04T04:55:29.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.416+0000 7f31b1063700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31ac1035d0 0x7f31ac19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:29.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.416+0000 7f31b1063700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ac103f90 0x7f31ac19fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:29.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.416+0000 7f31b1063700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31ac109670 0x7f31ac199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:29.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.416+0000 7f31b1063700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f31ac113680 con 0x7f31ac109670 2024-06-04T04:55:29.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.416+0000 7f31b1063700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f31ac113500 con 0x7f31ac1035d0 2024-06-04T04:55:29.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.416+0000 7f31b1063700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f31ac113800 con 0x7f31ac103f90 2024-06-04T04:55:29.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.416+0000 7f31ab59e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31ac109670 0x7f31ac199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:29.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.417+0000 7f31ab59e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31ac109670 0x7f31ac199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50372/0 (socket says 172.21.15.70:50372) 2024-06-04T04:55:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.417+0000 7f31ab59e700 1 -- 172.21.15.70:0/2606347567 learned_addr learned my addr 172.21.15.70:0/2606347567 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.417+0000 7f31aad9d700 1 --2- 172.21.15.70:0/2606347567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31ac1035d0 0x7f31ac19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.417+0000 7f31aa59c700 1 --2- 172.21.15.70:0/2606347567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ac103f90 0x7f31ac19fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.417+0000 7f31ab59e700 1 -- 172.21.15.70:0/2606347567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ac103f90 msgr2=0x7f31ac19fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.417+0000 7f31ab59e700 1 --2- 172.21.15.70:0/2606347567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ac103f90 0x7f31ac19fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.417+0000 7f31ab59e700 1 -- 172.21.15.70:0/2606347567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31ac1035d0 msgr2=0x7f31ac19f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.417+0000 7f31ab59e700 1 --2- 172.21.15.70:0/2606347567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31ac1035d0 0x7f31ac19f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.417+0000 7f31ab59e700 1 -- 172.21.15.70:0/2606347567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31ac19a180 con 0x7f31ac109670 2024-06-04T04:55:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.417+0000 7f31ab59e700 1 --2- 172.21.15.70:0/2606347567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31ac109670 0x7f31ac199a20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f319c007ee0 tx=0x7f319c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.417+0000 7f31a3fff700 1 -- 172.21.15.70:0/2606347567 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319c00dd20 con 0x7f31ac109670 2024-06-04T04:55:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.417+0000 7f31b1063700 1 -- 172.21.15.70:0/2606347567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31ac19a470 con 0x7f31ac109670 2024-06-04T04:55:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.417+0000 7f31a3fff700 1 -- 172.21.15.70:0/2606347567 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f319c004020 con 0x7f31ac109670 2024-06-04T04:55:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.417+0000 7f31a3fff700 1 -- 172.21.15.70:0/2606347567 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319c015660 con 0x7f31ac109670 2024-06-04T04:55:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.417+0000 7f31b1063700 1 -- 172.21.15.70:0/2606347567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31ac1a6730 con 0x7f31ac109670 2024-06-04T04:55:29.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.418+0000 7f31b1063700 1 -- 172.21.15.70:0/2606347567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31ac06a230 con 0x7f31ac109670 2024-06-04T04:55:29.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.420+0000 7f31a3fff700 1 -- 172.21.15.70:0/2606347567 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f319c0041d0 con 0x7f31ac109670 2024-06-04T04:55:29.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.421+0000 7f31a3fff700 1 --2- 172.21.15.70:0/2606347567 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f31980b2880 0x7f31980b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:29.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.421+0000 7f31aad9d700 1 --2- 172.21.15.70:0/2606347567 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f31980b2880 0x7f31980b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:29.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.421+0000 7f31a3fff700 1 -- 172.21.15.70:0/2606347567 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f319c0e2460 con 0x7f31ac109670 2024-06-04T04:55:29.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.422+0000 7f31aad9d700 1 --2- 172.21.15.70:0/2606347567 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f31980b2880 0x7f31980b4d40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f31a4005950 tx=0x7f31a400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:29.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.425+0000 7f31a3fff700 1 -- 172.21.15.70:0/2606347567 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f319c010070 con 0x7f31ac109670 2024-06-04T04:55:29.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.635+0000 7f31b1063700 1 -- 172.21.15.70:0/2606347567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f31ac02e560 con 0x7f31ac109670 2024-06-04T04:55:29.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.635+0000 7f31a3fff700 1 -- 172.21.15.70:0/2606347567 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f319c0aa160 con 0x7f31ac109670 2024-06-04T04:55:29.638 INFO:teuthology.orchestra.run.smithi070.stdout:158913790021 2024-06-04T04:55:29.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.637+0000 7f31b1063700 1 -- 172.21.15.70:0/2606347567 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f31980b2880 msgr2=0x7f31980b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:29.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.637+0000 7f31b1063700 1 --2- 172.21.15.70:0/2606347567 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f31980b2880 0x7f31980b4d40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f31a4005950 tx=0x7f31a400a300 comp rx=0 tx=0).stop 2024-06-04T04:55:29.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.637+0000 7f31b1063700 1 -- 172.21.15.70:0/2606347567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31ac109670 msgr2=0x7f31ac199a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:29.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.637+0000 7f31b1063700 1 --2- 172.21.15.70:0/2606347567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31ac109670 0x7f31ac199a20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f319c007ee0 tx=0x7f319c00d3b0 comp rx=0 tx=0).stop 2024-06-04T04:55:29.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.638+0000 7f31b1063700 1 -- 172.21.15.70:0/2606347567 shutdown_connections 2024-06-04T04:55:29.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.638+0000 7f31b1063700 1 --2- 172.21.15.70:0/2606347567 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f31980b2880 0x7f31980b4d40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:29.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.638+0000 7f31b1063700 1 --2- 172.21.15.70:0/2606347567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31ac109670 0x7f31ac199a20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:29.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.638+0000 7f31b1063700 1 --2- 172.21.15.70:0/2606347567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31ac1035d0 0x7f31ac19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:29.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.638+0000 7f31b1063700 1 --2- 172.21.15.70:0/2606347567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ac103f90 0x7f31ac19fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:29.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.638+0000 7f31b1063700 1 -- 172.21.15.70:0/2606347567 >> 172.21.15.70:0/2606347567 conn(0x7f31ac07a730 msgr2=0x7f31ac1024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:29.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.638+0000 7f31b1063700 1 -- 172.21.15.70:0/2606347567 shutdown_connections 2024-06-04T04:55:29.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.638+0000 7f31b1063700 1 -- 172.21.15.70:0/2606347567 wait complete. 2024-06-04T04:55:29.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.849+0000 7fed7a82a700 1 -- 172.21.15.70:0/982975016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed74108930 msgr2=0x7fed741104b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:29.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.849+0000 7fed7a82a700 1 --2- 172.21.15.70:0/982975016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed74108930 0x7fed741104b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00b0d0 tx=0x7fed6c00b3e0 comp rx=0 tx=0).stop 2024-06-04T04:55:29.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.849+0000 7fed7a82a700 1 -- 172.21.15.70:0/982975016 shutdown_connections 2024-06-04T04:55:29.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.849+0000 7fed7a82a700 1 --2- 172.21.15.70:0/982975016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed74108930 0x7fed741104b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:29.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.849+0000 7fed7a82a700 1 --2- 172.21.15.70:0/982975016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed74101cf0 0x7fed741020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:29.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.849+0000 7fed7a82a700 1 --2- 172.21.15.70:0/982975016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed741026a0 0x7fed74108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:29.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.849+0000 7fed7a82a700 1 -- 172.21.15.70:0/982975016 >> 172.21.15.70:0/982975016 conn(0x7fed74078a40 msgr2=0x7fed740ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:29.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.849+0000 7fed7a82a700 1 -- 172.21.15.70:0/982975016 shutdown_connections 2024-06-04T04:55:29.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.849+0000 7fed7a82a700 1 -- 172.21.15.70:0/982975016 wait complete. 2024-06-04T04:55:29.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.849+0000 7fed7a82a700 1 Processor -- start 2024-06-04T04:55:29.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.849+0000 7fed7a82a700 1 -- start start 2024-06-04T04:55:29.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.850+0000 7fed7a82a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed74101cf0 0x7fed7419f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:29.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.850+0000 7fed7a82a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed741026a0 0x7fed7419fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:29.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.850+0000 7fed7a82a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed74108930 0x7fed741997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:29.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.850+0000 7fed7a82a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fed741135d0 con 0x7fed74108930 2024-06-04T04:55:29.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.850+0000 7fed7a82a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed74113450 con 0x7fed74101cf0 2024-06-04T04:55:29.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.850+0000 7fed7a82a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fed74113750 con 0x7fed741026a0 2024-06-04T04:55:29.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.850+0000 7fed78dc7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed74108930 0x7fed741997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:29.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.850+0000 7fed78dc7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed74108930 0x7fed741997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50380/0 (socket says 172.21.15.70:50380) 2024-06-04T04:55:29.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.850+0000 7fed78dc7700 1 -- 172.21.15.70:0/302185336 learned_addr learned my addr 172.21.15.70:0/302185336 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:29.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.850+0000 7fed73fff700 1 --2- 172.21.15.70:0/302185336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed74101cf0 0x7fed7419f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:29.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.850+0000 7fed737fe700 1 --2- 172.21.15.70:0/302185336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed741026a0 0x7fed7419fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:29.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.850+0000 7fed78dc7700 1 -- 172.21.15.70:0/302185336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed741026a0 msgr2=0x7fed7419fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:29.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.850+0000 7fed78dc7700 1 --2- 172.21.15.70:0/302185336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed741026a0 0x7fed7419fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:29.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.850+0000 7fed78dc7700 1 -- 172.21.15.70:0/302185336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed74101cf0 msgr2=0x7fed7419f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:29.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.850+0000 7fed78dc7700 1 --2- 172.21.15.70:0/302185336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed74101cf0 0x7fed7419f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:29.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.850+0000 7fed78dc7700 1 -- 172.21.15.70:0/302185336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed74199fb0 con 0x7fed74108930 2024-06-04T04:55:29.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.850+0000 7fed78dc7700 1 --2- 172.21.15.70:0/302185336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed74108930 0x7fed741997d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fed6c007d90 tx=0x7fed6c007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:29.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.851+0000 7fed717fa700 1 -- 172.21.15.70:0/302185336 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c01f070 con 0x7fed74108930 2024-06-04T04:55:29.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.851+0000 7fed7a82a700 1 -- 172.21.15.70:0/302185336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed7419a240 con 0x7fed74108930 2024-06-04T04:55:29.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.851+0000 7fed717fa700 1 -- 172.21.15.70:0/302185336 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed6c005500 con 0x7fed74108930 2024-06-04T04:55:29.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.851+0000 7fed717fa700 1 -- 172.21.15.70:0/302185336 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c01b830 con 0x7fed74108930 2024-06-04T04:55:29.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.851+0000 7fed7a82a700 1 -- 172.21.15.70:0/302185336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed741a64b0 con 0x7fed74108930 2024-06-04T04:55:29.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.852+0000 7fed7a82a700 1 -- 172.21.15.70:0/302185336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed7410bb00 con 0x7fed74108930 2024-06-04T04:55:29.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.853+0000 7fed717fa700 1 -- 172.21.15.70:0/302185336 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fed6c022070 con 0x7fed74108930 2024-06-04T04:55:29.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.854+0000 7fed717fa700 1 --2- 172.21.15.70:0/302185336 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fed5c0b2880 0x7fed5c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:29.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.854+0000 7fed73fff700 1 --2- 172.21.15.70:0/302185336 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fed5c0b2880 0x7fed5c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:29.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.854+0000 7fed717fa700 1 -- 172.21.15.70:0/302185336 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fed6c00f020 con 0x7fed74108930 2024-06-04T04:55:29.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.854+0000 7fed73fff700 1 --2- 172.21.15.70:0/302185336 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fed5c0b2880 0x7fed5c0b4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fed64005950 tx=0x7fed6400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:29.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:29.857+0000 7fed717fa700 1 -- 172.21.15.70:0/302185336 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fed6c029020 con 0x7fed74108930 2024-06-04T04:55:29.916 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:29 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2606347567' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-04T04:55:30.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.074+0000 7fed7a82a700 1 -- 172.21.15.70:0/302185336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fed7419ad00 con 0x7fed74108930 2024-06-04T04:55:30.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.074+0000 7fed717fa700 1 -- 172.21.15.70:0/302185336 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fed6c0ab7f0 con 0x7fed74108930 2024-06-04T04:55:30.077 INFO:teuthology.orchestra.run.smithi070.stdout:206158430266 2024-06-04T04:55:30.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.077+0000 7fed7a82a700 1 -- 172.21.15.70:0/302185336 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fed5c0b2880 msgr2=0x7fed5c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:30.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.077+0000 7fed7a82a700 1 --2- 172.21.15.70:0/302185336 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fed5c0b2880 0x7fed5c0b4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fed64005950 tx=0x7fed6400a300 comp rx=0 tx=0).stop 2024-06-04T04:55:30.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.077+0000 7fed7a82a700 1 -- 172.21.15.70:0/302185336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed74108930 msgr2=0x7fed741997d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:30.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.077+0000 7fed7a82a700 1 --2- 172.21.15.70:0/302185336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed74108930 0x7fed741997d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fed6c007d90 tx=0x7fed6c007dc0 comp rx=0 tx=0).stop 2024-06-04T04:55:30.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.077+0000 7fed7a82a700 1 -- 172.21.15.70:0/302185336 shutdown_connections 2024-06-04T04:55:30.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.077+0000 7fed7a82a700 1 --2- 172.21.15.70:0/302185336 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fed5c0b2880 0x7fed5c0b4d40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:30.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.077+0000 7fed7a82a700 1 --2- 172.21.15.70:0/302185336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed74108930 0x7fed741997d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:30.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.077+0000 7fed7a82a700 1 --2- 172.21.15.70:0/302185336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed74101cf0 0x7fed7419f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:30.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.077+0000 7fed7a82a700 1 --2- 172.21.15.70:0/302185336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed741026a0 0x7fed7419fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:30.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.077+0000 7fed7a82a700 1 -- 172.21.15.70:0/302185336 >> 172.21.15.70:0/302185336 conn(0x7fed74078a40 msgr2=0x7fed741070a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:30.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.077+0000 7fed7a82a700 1 -- 172.21.15.70:0/302185336 shutdown_connections 2024-06-04T04:55:30.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.077+0000 7fed7a82a700 1 -- 172.21.15.70:0/302185336 wait complete. 2024-06-04T04:55:30.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:29 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2606347567' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-04T04:55:30.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:29 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2606347567' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-04T04:55:31.096 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:30 smithi070 ceph-mon[32631]: pgmap v347: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-06-04T04:55:31.096 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:30 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/302185336' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-04T04:55:31.096 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:30 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2097748519' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-04T04:55:31.097 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790017 got 158913790021 for osd.5 2024-06-04T04:55:31.097 DEBUG:teuthology.parallel:result is None 2024-06-04T04:55:31.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:30 smithi112 ceph-mon[36453]: pgmap v347: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-06-04T04:55:31.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:30 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/302185336' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-04T04:55:31.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:30 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2097748519' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-04T04:55:31.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:30 smithi098 ceph-mon[35569]: pgmap v347: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-06-04T04:55:31.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:30 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/302185336' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-04T04:55:31.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:30 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2097748519' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-04T04:55:32.163 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430261 got 206158430266 for osd.7 2024-06-04T04:55:32.163 DEBUG:teuthology.parallel:result is None 2024-06-04T04:55:33.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:32 smithi070 ceph-mon[32631]: pgmap v348: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-06-04T04:55:33.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.131+0000 7f8f949b6700 1 -- 172.21.15.70:0/1549799120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f90102440 msgr2=0x7f8f901028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:33.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.131+0000 7f8f949b6700 1 --2- 172.21.15.70:0/1549799120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f90102440 0x7f8f901028c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8f84009990 tx=0x7f8f84009ca0 comp rx=0 tx=0).stop 2024-06-04T04:55:33.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.131+0000 7f8f949b6700 1 -- 172.21.15.70:0/1549799120 shutdown_connections 2024-06-04T04:55:33.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.131+0000 7f8f949b6700 1 --2- 172.21.15.70:0/1549799120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f90102e00 0x7f8f9010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:33.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.131+0000 7f8f949b6700 1 --2- 172.21.15.70:0/1549799120 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f90108520 0x7f8f90108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:33.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.131+0000 7f8f949b6700 1 --2- 172.21.15.70:0/1549799120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f90102440 0x7f8f901028c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:33.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.131+0000 7f8f949b6700 1 -- 172.21.15.70:0/1549799120 >> 172.21.15.70:0/1549799120 conn(0x7f8f900793f0 msgr2=0x7f8f90079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:33.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.131+0000 7f8f949b6700 1 -- 172.21.15.70:0/1549799120 shutdown_connections 2024-06-04T04:55:33.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.131+0000 7f8f949b6700 1 -- 172.21.15.70:0/1549799120 wait complete. 2024-06-04T04:55:33.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.131+0000 7f8f949b6700 1 Processor -- start 2024-06-04T04:55:33.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.131+0000 7f8f949b6700 1 -- start start 2024-06-04T04:55:33.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f949b6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f90102440 0x7f8f9019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:33.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f949b6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f90102e00 0x7f8f9019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:33.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f949b6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f90108520 0x7f8f901988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:33.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f949b6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8f901124a0 con 0x7f8f90102e00 2024-06-04T04:55:33.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f949b6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8f90112320 con 0x7f8f90102440 2024-06-04T04:55:33.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f949b6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8f90112620 con 0x7f8f90108520 2024-06-04T04:55:33.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f8dd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f90102e00 0x7f8f9019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:33.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f8dd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f90102e00 0x7f8f9019ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50398/0 (socket says 172.21.15.70:50398) 2024-06-04T04:55:33.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f8dd9b700 1 -- 172.21.15.70:0/2097748519 learned_addr learned my addr 172.21.15.70:0/2097748519 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:33.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f8e59c700 1 --2- 172.21.15.70:0/2097748519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f90102440 0x7f8f9019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:33.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f8ed9d700 1 --2- 172.21.15.70:0/2097748519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f90108520 0x7f8f901988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:33.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f8dd9b700 1 -- 172.21.15.70:0/2097748519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f90108520 msgr2=0x7f8f901988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:33.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f8dd9b700 1 --2- 172.21.15.70:0/2097748519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f90108520 0x7f8f901988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:33.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f8dd9b700 1 -- 172.21.15.70:0/2097748519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f90102440 msgr2=0x7f8f9019e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:33.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f8dd9b700 1 --2- 172.21.15.70:0/2097748519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f90102440 0x7f8f9019e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:33.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f8dd9b700 1 -- 172.21.15.70:0/2097748519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f90199000 con 0x7f8f90102e00 2024-06-04T04:55:33.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f8ed9d700 1 --2- 172.21.15.70:0/2097748519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f90108520 0x7f8f901988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:55:33.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f8dd9b700 1 --2- 172.21.15.70:0/2097748519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f90102e00 0x7f8f9019ec20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8f84000c00 tx=0x7f8f840047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:33.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f7f7fe700 1 -- 172.21.15.70:0/2097748519 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f84027070 con 0x7f8f90102e00 2024-06-04T04:55:33.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.132+0000 7f8f949b6700 1 -- 172.21.15.70:0/2097748519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f90199290 con 0x7f8f90102e00 2024-06-04T04:55:33.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.133+0000 7f8f7f7fe700 1 -- 172.21.15.70:0/2097748519 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8f8400fd00 con 0x7f8f90102e00 2024-06-04T04:55:33.167 INFO:teuthology.orchestra.run.smithi070.stdout:34359738462 2024-06-04T04:55:33.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.133+0000 7f8f7f7fe700 1 -- 172.21.15.70:0/2097748519 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f84022e20 con 0x7f8f90102e00 2024-06-04T04:55:33.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.133+0000 7f8f949b6700 1 -- 172.21.15.70:0/2097748519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f901a55b0 con 0x7f8f90102e00 2024-06-04T04:55:33.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.133+0000 7f8f949b6700 1 -- 172.21.15.70:0/2097748519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f9010a9b0 con 0x7f8f90102e00 2024-06-04T04:55:33.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.135+0000 7f8f7f7fe700 1 -- 172.21.15.70:0/2097748519 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f8f8402a070 con 0x7f8f90102e00 2024-06-04T04:55:33.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.136+0000 7f8f7f7fe700 1 --2- 172.21.15.70:0/2097748519 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8f780b2880 0x7f8f780b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:33.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.136+0000 7f8f8e59c700 1 --2- 172.21.15.70:0/2097748519 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8f780b2880 0x7f8f780b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:33.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.136+0000 7f8f7f7fe700 1 -- 172.21.15.70:0/2097748519 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f8f840e32c0 con 0x7f8f90102e00 2024-06-04T04:55:33.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.136+0000 7f8f8e59c700 1 --2- 172.21.15.70:0/2097748519 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8f780b2880 0x7f8f780b4d40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8f80005950 tx=0x7f8f8000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:33.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.139+0000 7f8f7f7fe700 1 -- 172.21.15.70:0/2097748519 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8f84013070 con 0x7f8f90102e00 2024-06-04T04:55:33.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.345+0000 7f8f949b6700 1 -- 172.21.15.70:0/2097748519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f8f90199dd0 con 0x7f8f90102e00 2024-06-04T04:55:33.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.346+0000 7f8f7f7fe700 1 -- 172.21.15.70:0/2097748519 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8f840aaff0 con 0x7f8f90102e00 2024-06-04T04:55:33.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.348+0000 7f8f949b6700 1 -- 172.21.15.70:0/2097748519 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8f780b2880 msgr2=0x7f8f780b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:33.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.348+0000 7f8f949b6700 1 --2- 172.21.15.70:0/2097748519 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8f780b2880 0x7f8f780b4d40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8f80005950 tx=0x7f8f8000a300 comp rx=0 tx=0).stop 2024-06-04T04:55:33.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.348+0000 7f8f949b6700 1 -- 172.21.15.70:0/2097748519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f90102e00 msgr2=0x7f8f9019ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:33.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.348+0000 7f8f949b6700 1 --2- 172.21.15.70:0/2097748519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f90102e00 0x7f8f9019ec20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8f84000c00 tx=0x7f8f840047b0 comp rx=0 tx=0).stop 2024-06-04T04:55:33.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.348+0000 7f8f949b6700 1 -- 172.21.15.70:0/2097748519 shutdown_connections 2024-06-04T04:55:33.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.348+0000 7f8f949b6700 1 --2- 172.21.15.70:0/2097748519 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8f780b2880 0x7f8f780b4d40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:33.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.348+0000 7f8f949b6700 1 --2- 172.21.15.70:0/2097748519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f90108520 0x7f8f901988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:33.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.348+0000 7f8f949b6700 1 --2- 172.21.15.70:0/2097748519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f90102440 0x7f8f9019e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:33.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.348+0000 7f8f949b6700 1 --2- 172.21.15.70:0/2097748519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f90102e00 0x7f8f9019ec20 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:33.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.348+0000 7f8f949b6700 1 -- 172.21.15.70:0/2097748519 >> 172.21.15.70:0/2097748519 conn(0x7f8f900793f0 msgr2=0x7f8f90100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:33.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.348+0000 7f8f949b6700 1 -- 172.21.15.70:0/2097748519 shutdown_connections 2024-06-04T04:55:33.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:30.348+0000 7f8f949b6700 1 -- 172.21.15.70:0/2097748519 wait complete. 2024-06-04T04:55:33.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:32 smithi112 ceph-mon[36453]: pgmap v348: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-06-04T04:55:33.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:32 smithi098 ceph-mon[35569]: pgmap v348: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-06-04T04:55:33.758 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738459 got 34359738462 for osd.0 2024-06-04T04:55:33.758 DEBUG:teuthology.parallel:result is None 2024-06-04T04:55:33.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.993+0000 7f9062aa0700 1 -- 172.21.15.70:0/1538101799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f905c108520 msgr2=0x7f905c108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.993+0000 7f9062aa0700 1 --2- 172.21.15.70:0/1538101799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f905c108520 0x7f905c108900 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9050009990 tx=0x7f9050009ca0 comp rx=0 tx=0).stop 2024-06-04T04:55:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.993+0000 7f9062aa0700 1 -- 172.21.15.70:0/1538101799 shutdown_connections 2024-06-04T04:55:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.993+0000 7f9062aa0700 1 --2- 172.21.15.70:0/1538101799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f905c102e00 0x7f905c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.993+0000 7f9062aa0700 1 --2- 172.21.15.70:0/1538101799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f905c108520 0x7f905c108900 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.993+0000 7f9062aa0700 1 --2- 172.21.15.70:0/1538101799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f905c102440 0x7f905c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.993+0000 7f9062aa0700 1 -- 172.21.15.70:0/1538101799 >> 172.21.15.70:0/1538101799 conn(0x7f905c0793f0 msgr2=0x7f905c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.993+0000 7f9062aa0700 1 -- 172.21.15.70:0/1538101799 shutdown_connections 2024-06-04T04:55:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.993+0000 7f9062aa0700 1 -- 172.21.15.70:0/1538101799 wait complete. 2024-06-04T04:55:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f9062aa0700 1 Processor -- start 2024-06-04T04:55:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f9062aa0700 1 -- start start 2024-06-04T04:55:33.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f9062aa0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f905c102440 0x7f905c104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:33.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f9062aa0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f905c102e00 0x7f905c1045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:33.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f9062aa0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f905c108520 0x7f905c104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:33.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f9062aa0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f905c112580 con 0x7f905c102440 2024-06-04T04:55:33.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f9062aa0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f905c112400 con 0x7f905c108520 2024-06-04T04:55:33.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f9062aa0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f905c112700 con 0x7f905c102e00 2024-06-04T04:55:33.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f906083c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f905c102440 0x7f905c104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:33.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f906083c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f905c102440 0x7f905c104080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50406/0 (socket says 172.21.15.70:50406) 2024-06-04T04:55:33.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f906083c700 1 -- 172.21.15.70:0/4060772603 learned_addr learned my addr 172.21.15.70:0/4060772603 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:33.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f906103d700 1 --2- 172.21.15.70:0/4060772603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f905c108520 0x7f905c104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:33.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f905bfff700 1 --2- 172.21.15.70:0/4060772603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f905c102e00 0x7f905c1045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:33.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f906083c700 1 -- 172.21.15.70:0/4060772603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f905c102e00 msgr2=0x7f905c1045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f906083c700 1 --2- 172.21.15.70:0/4060772603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f905c102e00 0x7f905c1045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f906083c700 1 -- 172.21.15.70:0/4060772603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f905c108520 msgr2=0x7f905c104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f906083c700 1 --2- 172.21.15.70:0/4060772603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f905c108520 0x7f905c104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f906083c700 1 -- 172.21.15.70:0/4060772603 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f905c1a81d0 con 0x7f905c102440 2024-06-04T04:55:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.994+0000 7f905bfff700 1 --2- 172.21.15.70:0/4060772603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f905c102e00 0x7f905c1045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:55:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.995+0000 7f906083c700 1 --2- 172.21.15.70:0/4060772603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f905c102440 0x7f905c104080 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9050000c00 tx=0x7f90500047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.995+0000 7f9059ffb700 1 -- 172.21.15.70:0/4060772603 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9050027070 con 0x7f905c102440 2024-06-04T04:55:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.995+0000 7f9059ffb700 1 -- 172.21.15.70:0/4060772603 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f905000fd00 con 0x7f905c102440 2024-06-04T04:55:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.995+0000 7f9062aa0700 1 -- 172.21.15.70:0/4060772603 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f905c1a8460 con 0x7f905c102440 2024-06-04T04:55:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.995+0000 7f9059ffb700 1 -- 172.21.15.70:0/4060772603 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9050022e20 con 0x7f905c102440 2024-06-04T04:55:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.995+0000 7f9062aa0700 1 -- 172.21.15.70:0/4060772603 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f905c1a88b0 con 0x7f905c102440 2024-06-04T04:55:34.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.996+0000 7f9062aa0700 1 -- 172.21.15.70:0/4060772603 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f905c06a230 con 0x7f905c102440 2024-06-04T04:55:34.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.998+0000 7f9059ffb700 1 -- 172.21.15.70:0/4060772603 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f905002a070 con 0x7f905c102440 2024-06-04T04:55:34.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.999+0000 7f9059ffb700 1 --2- 172.21.15.70:0/4060772603 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f904c0b2880 0x7f904c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:34.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.999+0000 7f905bfff700 1 --2- 172.21.15.70:0/4060772603 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f904c0b2880 0x7f904c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:34.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.999+0000 7f9059ffb700 1 -- 172.21.15.70:0/4060772603 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f90500e3260 con 0x7f905c102440 2024-06-04T04:55:34.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:33.999+0000 7f905bfff700 1 --2- 172.21.15.70:0/4060772603 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f904c0b2880 0x7f904c0b4d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f905c079910 tx=0x7f9048008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:34.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.003+0000 7f9059ffb700 1 -- 172.21.15.70:0/4060772603 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9050013070 con 0x7f905c102440 2024-06-04T04:55:34.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.212+0000 7f9062aa0700 1 -- 172.21.15.70:0/4060772603 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f905c1a8bd0 con 0x7f905c102440 2024-06-04T04:55:34.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.212+0000 7f9059ffb700 1 -- 172.21.15.70:0/4060772603 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f90500aaff0 con 0x7f905c102440 2024-06-04T04:55:34.215 INFO:teuthology.orchestra.run.smithi070.stdout:304942678055 2024-06-04T04:55:34.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.215+0000 7f9062aa0700 1 -- 172.21.15.70:0/4060772603 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f904c0b2880 msgr2=0x7f904c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:34.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.215+0000 7f9062aa0700 1 --2- 172.21.15.70:0/4060772603 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f904c0b2880 0x7f904c0b4d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f905c079910 tx=0x7f9048008040 comp rx=0 tx=0).stop 2024-06-04T04:55:34.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.215+0000 7f9062aa0700 1 -- 172.21.15.70:0/4060772603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f905c102440 msgr2=0x7f905c104080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:34.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.215+0000 7f9062aa0700 1 --2- 172.21.15.70:0/4060772603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f905c102440 0x7f905c104080 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9050000c00 tx=0x7f90500047b0 comp rx=0 tx=0).stop 2024-06-04T04:55:34.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.215+0000 7f9062aa0700 1 -- 172.21.15.70:0/4060772603 shutdown_connections 2024-06-04T04:55:34.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.215+0000 7f9062aa0700 1 --2- 172.21.15.70:0/4060772603 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f904c0b2880 0x7f904c0b4d40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:34.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.215+0000 7f9062aa0700 1 --2- 172.21.15.70:0/4060772603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f905c108520 0x7f905c104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:34.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.215+0000 7f9062aa0700 1 --2- 172.21.15.70:0/4060772603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f905c102440 0x7f905c104080 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:34.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.215+0000 7f9062aa0700 1 --2- 172.21.15.70:0/4060772603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f905c102e00 0x7f905c1045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:34.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.215+0000 7f9062aa0700 1 -- 172.21.15.70:0/4060772603 >> 172.21.15.70:0/4060772603 conn(0x7f905c0793f0 msgr2=0x7f905c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:34.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.215+0000 7f9062aa0700 1 -- 172.21.15.70:0/4060772603 shutdown_connections 2024-06-04T04:55:34.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.215+0000 7f9062aa0700 1 -- 172.21.15.70:0/4060772603 wait complete. 2024-06-04T04:55:34.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.219+0000 7f5660443700 1 -- 172.21.15.70:0/401944066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5658102440 msgr2=0x7f56581028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.219+0000 7f5660443700 1 --2- 172.21.15.70:0/401944066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5658102440 0x7f56581028c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5648009990 tx=0x7f5648009ca0 comp rx=0 tx=0).stop 2024-06-04T04:55:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.219+0000 7f5660443700 1 -- 172.21.15.70:0/401944066 shutdown_connections 2024-06-04T04:55:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.219+0000 7f5660443700 1 --2- 172.21.15.70:0/401944066 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5658102e00 0x7f565810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.219+0000 7f5660443700 1 --2- 172.21.15.70:0/401944066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5658108520 0x7f5658108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.219+0000 7f5660443700 1 --2- 172.21.15.70:0/401944066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5658102440 0x7f56581028c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.219+0000 7f5660443700 1 -- 172.21.15.70:0/401944066 >> 172.21.15.70:0/401944066 conn(0x7f56580793f0 msgr2=0x7f5658079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.219+0000 7f5660443700 1 -- 172.21.15.70:0/401944066 shutdown_connections 2024-06-04T04:55:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.219+0000 7f5660443700 1 -- 172.21.15.70:0/401944066 wait complete. 2024-06-04T04:55:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.220+0000 7f5660443700 1 Processor -- start 2024-06-04T04:55:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.220+0000 7f5660443700 1 -- start start 2024-06-04T04:55:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f5660443700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5658102440 0x7f565819e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f5660443700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5658102e00 0x7f565819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f5660443700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5658108520 0x7f56581988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f5660443700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5658112500 con 0x7f5658108520 2024-06-04T04:55:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f5660443700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5658112380 con 0x7f5658102440 2024-06-04T04:55:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f5660443700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5658112680 con 0x7f5658102e00 2024-06-04T04:55:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f565e9e0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5658108520 0x7f56581988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f565e9e0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5658108520 0x7f56581988a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50424/0 (socket says 172.21.15.70:50424) 2024-06-04T04:55:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f565e9e0700 1 -- 172.21.15.70:0/1441911819 learned_addr learned my addr 172.21.15.70:0/1441911819 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f565d9de700 1 --2- 172.21.15.70:0/1441911819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5658102e00 0x7f565819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f565e1df700 1 --2- 172.21.15.70:0/1441911819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5658102440 0x7f565819e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f565e9e0700 1 -- 172.21.15.70:0/1441911819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5658102e00 msgr2=0x7f565819ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f565e9e0700 1 --2- 172.21.15.70:0/1441911819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5658102e00 0x7f565819ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f565e9e0700 1 -- 172.21.15.70:0/1441911819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5658102440 msgr2=0x7f565819e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f565e9e0700 1 --2- 172.21.15.70:0/1441911819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5658102440 0x7f565819e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f565e9e0700 1 -- 172.21.15.70:0/1441911819 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5658199000 con 0x7f5658108520 2024-06-04T04:55:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f565e9e0700 1 --2- 172.21.15.70:0/1441911819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5658108520 0x7f56581988a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f565000b970 tx=0x7f565000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f564f7fe700 1 -- 172.21.15.70:0/1441911819 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565000d860 con 0x7f5658108520 2024-06-04T04:55:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f5660443700 1 -- 172.21.15.70:0/1441911819 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56581992f0 con 0x7f5658108520 2024-06-04T04:55:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f564f7fe700 1 -- 172.21.15.70:0/1441911819 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f565000e430 con 0x7f5658108520 2024-06-04T04:55:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f564f7fe700 1 -- 172.21.15.70:0/1441911819 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5650017660 con 0x7f5658108520 2024-06-04T04:55:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.221+0000 7f5660443700 1 -- 172.21.15.70:0/1441911819 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56581a55b0 con 0x7f5658108520 2024-06-04T04:55:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.222+0000 7f5660443700 1 -- 172.21.15.70:0/1441911819 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f565806a230 con 0x7f5658108520 2024-06-04T04:55:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.224+0000 7f564f7fe700 1 -- 172.21.15.70:0/1441911819 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f565000da00 con 0x7f5658108520 2024-06-04T04:55:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.225+0000 7f564f7fe700 1 --2- 172.21.15.70:0/1441911819 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f56440b2880 0x7f56440b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.225+0000 7f565e1df700 1 --2- 172.21.15.70:0/1441911819 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f56440b2880 0x7f56440b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.225+0000 7f564f7fe700 1 -- 172.21.15.70:0/1441911819 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f56500e1f90 con 0x7f5658108520 2024-06-04T04:55:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.226+0000 7f565e1df700 1 --2- 172.21.15.70:0/1441911819 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f56440b2880 0x7f56440b4d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5654005950 tx=0x7f565400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:34.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.231+0000 7f564f7fe700 1 -- 172.21.15.70:0/1441911819 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5650053df0 con 0x7f5658108520 2024-06-04T04:55:34.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.443+0000 7f5660443700 1 -- 172.21.15.70:0/1441911819 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f5658199dd0 con 0x7f5658108520 2024-06-04T04:55:34.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.443+0000 7f564f7fe700 1 -- 172.21.15.70:0/1441911819 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5650013030 con 0x7f5658108520 2024-06-04T04:55:34.446 INFO:teuthology.orchestra.run.smithi070.stdout:77309411413 2024-06-04T04:55:34.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.446+0000 7f5660443700 1 -- 172.21.15.70:0/1441911819 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f56440b2880 msgr2=0x7f56440b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:34.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.446+0000 7f5660443700 1 --2- 172.21.15.70:0/1441911819 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f56440b2880 0x7f56440b4d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5654005950 tx=0x7f565400a300 comp rx=0 tx=0).stop 2024-06-04T04:55:34.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.446+0000 7f5660443700 1 -- 172.21.15.70:0/1441911819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5658108520 msgr2=0x7f56581988a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:34.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.446+0000 7f5660443700 1 --2- 172.21.15.70:0/1441911819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5658108520 0x7f56581988a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f565000b970 tx=0x7f565000bc80 comp rx=0 tx=0).stop 2024-06-04T04:55:34.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.446+0000 7f5660443700 1 -- 172.21.15.70:0/1441911819 shutdown_connections 2024-06-04T04:55:34.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.446+0000 7f5660443700 1 --2- 172.21.15.70:0/1441911819 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f56440b2880 0x7f56440b4d40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:34.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.446+0000 7f5660443700 1 --2- 172.21.15.70:0/1441911819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5658108520 0x7f56581988a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:34.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.446+0000 7f5660443700 1 --2- 172.21.15.70:0/1441911819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5658102440 0x7f565819e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:34.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.446+0000 7f5660443700 1 --2- 172.21.15.70:0/1441911819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5658102e00 0x7f565819ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:34.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.446+0000 7f5660443700 1 -- 172.21.15.70:0/1441911819 >> 172.21.15.70:0/1441911819 conn(0x7f56580793f0 msgr2=0x7f5658100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:34.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.446+0000 7f5660443700 1 -- 172.21.15.70:0/1441911819 shutdown_connections 2024-06-04T04:55:34.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:34.446+0000 7f5660443700 1 -- 172.21.15.70:0/1441911819 wait complete. 2024-06-04T04:55:35.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:34 smithi070 ceph-mon[32631]: pgmap v349: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-04T04:55:35.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:34 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/4060772603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-04T04:55:35.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:34 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1441911819' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-04T04:55:35.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:34 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:55:35.043 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678049 got 304942678055 for osd.11 2024-06-04T04:55:35.043 DEBUG:teuthology.parallel:result is None 2024-06-04T04:55:35.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:34 smithi112 ceph-mon[36453]: pgmap v349: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-04T04:55:35.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:34 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/4060772603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-04T04:55:35.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:34 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1441911819' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-04T04:55:35.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:34 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:55:35.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:34 smithi098 ceph-mon[35569]: pgmap v349: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-04T04:55:35.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:34 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/4060772603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-04T04:55:35.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:34 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1441911819' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-04T04:55:35.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:34 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:55:35.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.614+0000 7f8b775be700 1 -- 172.21.15.70:0/1812413003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b701096a0 msgr2=0x7f8b70109a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.614+0000 7f8b775be700 1 --2- 172.21.15.70:0/1812413003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b701096a0 0x7f8b70109a80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c0098f0 tx=0x7f8b6c009c00 comp rx=0 tx=0).stop 2024-06-04T04:55:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.614+0000 7f8b775be700 1 -- 172.21.15.70:0/1812413003 shutdown_connections 2024-06-04T04:55:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.614+0000 7f8b775be700 1 --2- 172.21.15.70:0/1812413003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b70103f50 0x7f8b70110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.614+0000 7f8b775be700 1 --2- 172.21.15.70:0/1812413003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b701096a0 0x7f8b70109a80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.614+0000 7f8b775be700 1 --2- 172.21.15.70:0/1812413003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8b70103590 0x7f8b70103a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.614+0000 7f8b775be700 1 -- 172.21.15.70:0/1812413003 >> 172.21.15.70:0/1812413003 conn(0x7f8b7007a6f0 msgr2=0x7f8b7007ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.614+0000 7f8b775be700 1 -- 172.21.15.70:0/1812413003 shutdown_connections 2024-06-04T04:55:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.614+0000 7f8b775be700 1 -- 172.21.15.70:0/1812413003 wait complete. 2024-06-04T04:55:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.614+0000 7f8b775be700 1 Processor -- start 2024-06-04T04:55:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.614+0000 7f8b775be700 1 -- start start 2024-06-04T04:55:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b775be700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b70103590 0x7f8b7019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b775be700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b70103f50 0x7f8b7019fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b775be700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8b701096a0 0x7f8b70199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b775be700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8b701135e0 con 0x7f8b70103f50 2024-06-04T04:55:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b775be700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b70113460 con 0x7f8b70103590 2024-06-04T04:55:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b775be700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8b70113760 con 0x7f8b701096a0 2024-06-04T04:55:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b74b59700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b70103f50 0x7f8b7019fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b74b59700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b70103f50 0x7f8b7019fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50476/0 (socket says 172.21.15.70:50476) 2024-06-04T04:55:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b74b59700 1 -- 172.21.15.70:0/3850535827 learned_addr learned my addr 172.21.15.70:0/3850535827 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b7535a700 1 --2- 172.21.15.70:0/3850535827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b70103590 0x7f8b7019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b75b5b700 1 --2- 172.21.15.70:0/3850535827 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8b701096a0 0x7f8b70199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b74b59700 1 -- 172.21.15.70:0/3850535827 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8b701096a0 msgr2=0x7f8b70199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b74b59700 1 --2- 172.21.15.70:0/3850535827 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8b701096a0 0x7f8b70199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b74b59700 1 -- 172.21.15.70:0/3850535827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b70103590 msgr2=0x7f8b7019f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b74b59700 1 --2- 172.21.15.70:0/3850535827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b70103590 0x7f8b7019f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b74b59700 1 -- 172.21.15.70:0/3850535827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b7019a080 con 0x7f8b70103f50 2024-06-04T04:55:35.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b74b59700 1 --2- 172.21.15.70:0/3850535827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b70103f50 0x7f8b7019fca0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8b6000d980 tx=0x7f8b6000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:35.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b667fc700 1 -- 172.21.15.70:0/3850535827 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b60017070 con 0x7f8b70103f50 2024-06-04T04:55:35.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b775be700 1 -- 172.21.15.70:0/3850535827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b7019a370 con 0x7f8b70103f50 2024-06-04T04:55:35.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.615+0000 7f8b667fc700 1 -- 172.21.15.70:0/3850535827 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8b60005020 con 0x7f8b70103f50 2024-06-04T04:55:35.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.616+0000 7f8b667fc700 1 -- 172.21.15.70:0/3850535827 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b60009ab0 con 0x7f8b70103f50 2024-06-04T04:55:35.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.616+0000 7f8b775be700 1 -- 172.21.15.70:0/3850535827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b701a6630 con 0x7f8b70103f50 2024-06-04T04:55:35.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.617+0000 7f8b775be700 1 -- 172.21.15.70:0/3850535827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b7006a230 con 0x7f8b70103f50 2024-06-04T04:55:35.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.619+0000 7f8b667fc700 1 -- 172.21.15.70:0/3850535827 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f8b600051d0 con 0x7f8b70103f50 2024-06-04T04:55:35.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.620+0000 7f8b667fc700 1 --2- 172.21.15.70:0/3850535827 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8b5c0b2880 0x7f8b5c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:35.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.620+0000 7f8b667fc700 1 -- 172.21.15.70:0/3850535827 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f8b600e2330 con 0x7f8b70103f50 2024-06-04T04:55:35.625 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:35.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.623+0000 7f8b7535a700 1 --2- 172.21.15.70:0/3850535827 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8b5c0b2880 0x7f8b5c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:35.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.623+0000 7f8b7535a700 1 --2- 172.21.15.70:0/3850535827 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8b5c0b2880 0x7f8b5c0b4d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c0098c0 tx=0x7f8b6c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:35.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.627+0000 7f8b667fc700 1 -- 172.21.15.70:0/3850535827 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8b60020030 con 0x7f8b70103f50 2024-06-04T04:55:35.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.858+0000 7f8b775be700 1 -- 172.21.15.70:0/3850535827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f8b7002e560 con 0x7f8b70103f50 2024-06-04T04:55:35.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.858+0000 7f8b667fc700 1 -- 172.21.15.70:0/3850535827 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8b600aa030 con 0x7f8b70103f50 2024-06-04T04:55:35.861 INFO:teuthology.orchestra.run.smithi070.stdout:231928234039 2024-06-04T04:55:35.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.861+0000 7f8b775be700 1 -- 172.21.15.70:0/3850535827 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8b5c0b2880 msgr2=0x7f8b5c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:35.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.861+0000 7f8b775be700 1 --2- 172.21.15.70:0/3850535827 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8b5c0b2880 0x7f8b5c0b4d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c0098c0 tx=0x7f8b6c019040 comp rx=0 tx=0).stop 2024-06-04T04:55:35.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.861+0000 7f8b775be700 1 -- 172.21.15.70:0/3850535827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b70103f50 msgr2=0x7f8b7019fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:35.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.861+0000 7f8b775be700 1 --2- 172.21.15.70:0/3850535827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b70103f50 0x7f8b7019fca0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8b6000d980 tx=0x7f8b6000dc90 comp rx=0 tx=0).stop 2024-06-04T04:55:35.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.861+0000 7f8b775be700 1 -- 172.21.15.70:0/3850535827 shutdown_connections 2024-06-04T04:55:35.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.861+0000 7f8b775be700 1 --2- 172.21.15.70:0/3850535827 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8b5c0b2880 0x7f8b5c0b4d40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:35.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.861+0000 7f8b775be700 1 --2- 172.21.15.70:0/3850535827 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8b701096a0 0x7f8b70199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:35.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.861+0000 7f8b775be700 1 --2- 172.21.15.70:0/3850535827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b70103590 0x7f8b7019f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:35.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.861+0000 7f8b775be700 1 --2- 172.21.15.70:0/3850535827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b70103f50 0x7f8b7019fca0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:35.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.861+0000 7f8b775be700 1 -- 172.21.15.70:0/3850535827 >> 172.21.15.70:0/3850535827 conn(0x7f8b7007a6f0 msgr2=0x7f8b701022d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:35.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.861+0000 7f8b775be700 1 -- 172.21.15.70:0/3850535827 shutdown_connections 2024-06-04T04:55:35.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.861+0000 7f8b775be700 1 -- 172.21.15.70:0/3850535827 wait complete. 2024-06-04T04:55:35.971 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:35 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2239591422' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-04T04:55:35.972 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:35 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3850535827' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-04T04:55:36.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:35 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2239591422' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-04T04:55:36.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:35 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3850535827' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-04T04:55:36.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:35 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2239591422' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-04T04:55:36.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:35 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3850535827' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-04T04:55:36.327 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411405 got 77309411413 for osd.2 2024-06-04T04:55:36.327 DEBUG:teuthology.parallel:result is None 2024-06-04T04:55:36.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.504+0000 7fed1277b700 1 -- 172.21.15.70:0/2700917116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed0c102df0 msgr2=0x7fed0c10f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:36.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.504+0000 7fed1277b700 1 --2- 172.21.15.70:0/2700917116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed0c102df0 0x7fed0c10f2e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fed0400b0a0 tx=0x7fed0400b3b0 comp rx=0 tx=0).stop 2024-06-04T04:55:36.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.504+0000 7fed1277b700 1 -- 172.21.15.70:0/2700917116 shutdown_connections 2024-06-04T04:55:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.504+0000 7fed1277b700 1 --2- 172.21.15.70:0/2700917116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed0c102df0 0x7fed0c10f2e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.504+0000 7fed1277b700 1 --2- 172.21.15.70:0/2700917116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed0c108510 0x7fed0c1088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.504+0000 7fed1277b700 1 --2- 172.21.15.70:0/2700917116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed0c102430 0x7fed0c1028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.504+0000 7fed1277b700 1 -- 172.21.15.70:0/2700917116 >> 172.21.15.70:0/2700917116 conn(0x7fed0c0793b0 msgr2=0x7fed0c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.504+0000 7fed1277b700 1 -- 172.21.15.70:0/2700917116 shutdown_connections 2024-06-04T04:55:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.504+0000 7fed1277b700 1 -- 172.21.15.70:0/2700917116 wait complete. 2024-06-04T04:55:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.504+0000 7fed1277b700 1 Processor -- start 2024-06-04T04:55:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.504+0000 7fed1277b700 1 -- start start 2024-06-04T04:55:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed1277b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed0c102430 0x7fed0c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed1277b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed0c102df0 0x7fed0c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed1277b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed0c108510 0x7fed0c1987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed1277b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c112470 con 0x7fed0c108510 2024-06-04T04:55:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed1277b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c1122f0 con 0x7fed0c102430 2024-06-04T04:55:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed1277b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c1125f0 con 0x7fed0c102df0 2024-06-04T04:55:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed10d18700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed0c108510 0x7fed0c1987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:36.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed10d18700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed0c108510 0x7fed0c1987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50446/0 (socket says 172.21.15.70:50446) 2024-06-04T04:55:36.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed10d18700 1 -- 172.21.15.70:0/2239591422 learned_addr learned my addr 172.21.15.70:0/2239591422 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:36.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed0bfff700 1 --2- 172.21.15.70:0/2239591422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed0c102430 0x7fed0c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:36.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed0b7fe700 1 --2- 172.21.15.70:0/2239591422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed0c102df0 0x7fed0c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:36.624 INFO:teuthology.orchestra.run.smithi070.stdout:55834574938 2024-06-04T04:55:36.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed10d18700 1 -- 172.21.15.70:0/2239591422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed0c102df0 msgr2=0x7fed0c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:36.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed10d18700 1 --2- 172.21.15.70:0/2239591422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed0c102df0 0x7fed0c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:36.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed10d18700 1 -- 172.21.15.70:0/2239591422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed0c102430 msgr2=0x7fed0c19e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:36.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed10d18700 1 --2- 172.21.15.70:0/2239591422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed0c102430 0x7fed0c19e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed10d18700 1 -- 172.21.15.70:0/2239591422 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed0c198fb0 con 0x7fed0c108510 2024-06-04T04:55:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed0bfff700 1 --2- 172.21.15.70:0/2239591422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed0c102430 0x7fed0c19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:55:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed10d18700 1 --2- 172.21.15.70:0/2239591422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed0c108510 0x7fed0c1987d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fed0400b780 tx=0x7fed04004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.505+0000 7fed097fa700 1 -- 172.21.15.70:0/2239591422 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed04021070 con 0x7fed0c108510 2024-06-04T04:55:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.506+0000 7fed1277b700 1 -- 172.21.15.70:0/2239591422 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed0c199240 con 0x7fed0c108510 2024-06-04T04:55:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.506+0000 7fed097fa700 1 -- 172.21.15.70:0/2239591422 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed04007b80 con 0x7fed0c108510 2024-06-04T04:55:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.506+0000 7fed097fa700 1 -- 172.21.15.70:0/2239591422 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed040052a0 con 0x7fed0c108510 2024-06-04T04:55:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.506+0000 7fed1277b700 1 -- 172.21.15.70:0/2239591422 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed0c1a54b0 con 0x7fed0c108510 2024-06-04T04:55:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.507+0000 7fed1277b700 1 -- 172.21.15.70:0/2239591422 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed0c103b50 con 0x7fed0c108510 2024-06-04T04:55:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.508+0000 7fed097fa700 1 -- 172.21.15.70:0/2239591422 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fed04024070 con 0x7fed0c108510 2024-06-04T04:55:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.508+0000 7fed097fa700 1 --2- 172.21.15.70:0/2239591422 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fecf40b2880 0x7fecf40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.509+0000 7fed0bfff700 1 --2- 172.21.15.70:0/2239591422 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fecf40b2880 0x7fecf40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.509+0000 7fed097fa700 1 -- 172.21.15.70:0/2239591422 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fed0400f020 con 0x7fed0c108510 2024-06-04T04:55:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.509+0000 7fed0bfff700 1 --2- 172.21.15.70:0/2239591422 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fecf40b2880 0x7fecf40b4d40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fecfc007f50 tx=0x7fecfc0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.513+0000 7fed097fa700 1 -- 172.21.15.70:0/2239591422 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fed0402b020 con 0x7fed0c108510 2024-06-04T04:55:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.738+0000 7fed1277b700 1 -- 172.21.15.70:0/2239591422 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fed0c02e560 con 0x7fed0c108510 2024-06-04T04:55:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.738+0000 7fed097fa700 1 -- 172.21.15.70:0/2239591422 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fed040ab4b0 con 0x7fed0c108510 2024-06-04T04:55:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.741+0000 7fed1277b700 1 -- 172.21.15.70:0/2239591422 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fecf40b2880 msgr2=0x7fecf40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.741+0000 7fed1277b700 1 --2- 172.21.15.70:0/2239591422 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fecf40b2880 0x7fecf40b4d40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fecfc007f50 tx=0x7fecfc0073d0 comp rx=0 tx=0).stop 2024-06-04T04:55:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.741+0000 7fed1277b700 1 -- 172.21.15.70:0/2239591422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed0c108510 msgr2=0x7fed0c1987d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.741+0000 7fed1277b700 1 --2- 172.21.15.70:0/2239591422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed0c108510 0x7fed0c1987d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fed0400b780 tx=0x7fed04004d00 comp rx=0 tx=0).stop 2024-06-04T04:55:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.741+0000 7fed1277b700 1 -- 172.21.15.70:0/2239591422 shutdown_connections 2024-06-04T04:55:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.741+0000 7fed1277b700 1 --2- 172.21.15.70:0/2239591422 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fecf40b2880 0x7fecf40b4d40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.741+0000 7fed1277b700 1 --2- 172.21.15.70:0/2239591422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed0c108510 0x7fed0c1987d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.741+0000 7fed1277b700 1 --2- 172.21.15.70:0/2239591422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed0c102430 0x7fed0c19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.741+0000 7fed1277b700 1 --2- 172.21.15.70:0/2239591422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed0c102df0 0x7fed0c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.741+0000 7fed1277b700 1 -- 172.21.15.70:0/2239591422 >> 172.21.15.70:0/2239591422 conn(0x7fed0c0793b0 msgr2=0x7fed0c1005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.741+0000 7fed1277b700 1 -- 172.21.15.70:0/2239591422 shutdown_connections 2024-06-04T04:55:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:35.741+0000 7fed1277b700 1 -- 172.21.15.70:0/2239591422 wait complete. 2024-06-04T04:55:37.305 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574933 got 55834574938 for osd.1 2024-06-04T04:55:37.306 DEBUG:teuthology.parallel:result is None 2024-06-04T04:55:37.454 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:37 smithi070 ceph-mon[32631]: pgmap v350: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-06-04T04:55:37.454 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:37 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:37.454 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:37 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:37.454 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:37 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:37.454 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:37 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:37.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.553+0000 7fc627b53700 1 -- 172.21.15.70:0/2195248384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6201041a0 msgr2=0x7fc6201081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:37.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.553+0000 7fc627b53700 1 --2- 172.21.15.70:0/2195248384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6201041a0 0x7fc6201081c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0099c0 tx=0x7fc61c009cd0 comp rx=0 tx=0).stop 2024-06-04T04:55:37.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.553+0000 7fc627b53700 1 -- 172.21.15.70:0/2195248384 shutdown_connections 2024-06-04T04:55:37.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.553+0000 7fc627b53700 1 --2- 172.21.15.70:0/2195248384 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6201088f0 0x7fc620110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:37.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.553+0000 7fc627b53700 1 --2- 172.21.15.70:0/2195248384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6201037f0 0x7fc620103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:37.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.553+0000 7fc627b53700 1 --2- 172.21.15.70:0/2195248384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6201041a0 0x7fc6201081c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:37.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.553+0000 7fc627b53700 1 -- 172.21.15.70:0/2195248384 >> 172.21.15.70:0/2195248384 conn(0x7fc62007a6a0 msgr2=0x7fc62007aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:37.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.553+0000 7fc627b53700 1 -- 172.21.15.70:0/2195248384 shutdown_connections 2024-06-04T04:55:37.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.554+0000 7fc627b53700 1 -- 172.21.15.70:0/2195248384 wait complete. 2024-06-04T04:55:37.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.554+0000 7fc627b53700 1 Processor -- start 2024-06-04T04:55:37.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.554+0000 7fc627b53700 1 -- start start 2024-06-04T04:55:37.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.554+0000 7fc627b53700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6201037f0 0x7fc62019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:37.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.554+0000 7fc627b53700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6201041a0 0x7fc62019fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:37.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.554+0000 7fc627b53700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6201088f0 0x7fc620199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:37.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.554+0000 7fc627b53700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc6201135e0 con 0x7fc6201037f0 2024-06-04T04:55:37.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.554+0000 7fc627b53700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc620113460 con 0x7fc6201041a0 2024-06-04T04:55:37.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.554+0000 7fc627b53700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc620113760 con 0x7fc6201088f0 2024-06-04T04:55:37.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.554+0000 7fc6258ef700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6201037f0 0x7fc62019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:37.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.554+0000 7fc6258ef700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6201037f0 0x7fc62019f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54650/0 (socket says 172.21.15.70:54650) 2024-06-04T04:55:37.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.554+0000 7fc6258ef700 1 -- 172.21.15.70:0/1113400492 learned_addr learned my addr 172.21.15.70:0/1113400492 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:37.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.555+0000 7fc6258ef700 1 -- 172.21.15.70:0/1113400492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6201088f0 msgr2=0x7fc620199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:37.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.555+0000 7fc6258ef700 1 --2- 172.21.15.70:0/1113400492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6201088f0 0x7fc620199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:37.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.555+0000 7fc6258ef700 1 -- 172.21.15.70:0/1113400492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6201041a0 msgr2=0x7fc62019fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:37.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.555+0000 7fc6258ef700 1 --2- 172.21.15.70:0/1113400492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6201041a0 0x7fc62019fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:37.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.555+0000 7fc6258ef700 1 -- 172.21.15.70:0/1113400492 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc62019a0a0 con 0x7fc6201037f0 2024-06-04T04:55:37.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.555+0000 7fc6258ef700 1 --2- 172.21.15.70:0/1113400492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6201037f0 0x7fc62019f780 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc61000c920 tx=0x7fc61000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:37.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.555+0000 7fc616ffd700 1 -- 172.21.15.70:0/1113400492 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc610007b00 con 0x7fc6201037f0 2024-06-04T04:55:37.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.555+0000 7fc616ffd700 1 -- 172.21.15.70:0/1113400492 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc610004cc0 con 0x7fc6201037f0 2024-06-04T04:55:37.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.555+0000 7fc616ffd700 1 -- 172.21.15.70:0/1113400492 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6100054f0 con 0x7fc6201037f0 2024-06-04T04:55:37.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.555+0000 7fc627b53700 1 -- 172.21.15.70:0/1113400492 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc62019a390 con 0x7fc6201037f0 2024-06-04T04:55:37.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.555+0000 7fc627b53700 1 -- 172.21.15.70:0/1113400492 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6201a6650 con 0x7fc6201037f0 2024-06-04T04:55:37.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.556+0000 7fc627b53700 1 -- 172.21.15.70:0/1113400492 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc62006a230 con 0x7fc6201037f0 2024-06-04T04:55:37.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.557+0000 7fc616ffd700 1 -- 172.21.15.70:0/1113400492 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fc610007ca0 con 0x7fc6201037f0 2024-06-04T04:55:37.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.558+0000 7fc616ffd700 1 --2- 172.21.15.70:0/1113400492 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc60c0b2900 0x7fc60c0b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:37.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.558+0000 7fc6250ee700 1 --2- 172.21.15.70:0/1113400492 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc60c0b2900 0x7fc60c0b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:37.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.558+0000 7fc616ffd700 1 -- 172.21.15.70:0/1113400492 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fc6100e1e00 con 0x7fc6201037f0 2024-06-04T04:55:37.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.558+0000 7fc6250ee700 1 --2- 172.21.15.70:0/1113400492 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc60c0b2900 0x7fc60c0b4dc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0098e0 tx=0x7fc61c009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:37.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.562+0000 7fc616ffd700 1 -- 172.21.15.70:0/1113400492 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc610014030 con 0x7fc6201037f0 2024-06-04T04:55:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:37 smithi112 ceph-mon[36453]: pgmap v350: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-06-04T04:55:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:37 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:37 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:37 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:37 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:37 smithi098 ceph-mon[35569]: pgmap v350: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-06-04T04:55:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:37 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:37 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:37 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:37 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.776+0000 7fc627b53700 1 -- 172.21.15.70:0/1113400492 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fc62002e560 con 0x7fc6201037f0 2024-06-04T04:55:37.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.777+0000 7fc616ffd700 1 -- 172.21.15.70:0/1113400492 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc6100a9b00 con 0x7fc6201037f0 2024-06-04T04:55:37.779 INFO:teuthology.orchestra.run.smithi070.stdout:184683593794 2024-06-04T04:55:37.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.780+0000 7fc627b53700 1 -- 172.21.15.70:0/1113400492 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc60c0b2900 msgr2=0x7fc60c0b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.780+0000 7fc627b53700 1 --2- 172.21.15.70:0/1113400492 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc60c0b2900 0x7fc60c0b4dc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0098e0 tx=0x7fc61c009f10 comp rx=0 tx=0).stop 2024-06-04T04:55:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.780+0000 7fc627b53700 1 -- 172.21.15.70:0/1113400492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6201037f0 msgr2=0x7fc62019f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.780+0000 7fc627b53700 1 --2- 172.21.15.70:0/1113400492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6201037f0 0x7fc62019f780 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc61000c920 tx=0x7fc61000cce0 comp rx=0 tx=0).stop 2024-06-04T04:55:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.780+0000 7fc627b53700 1 -- 172.21.15.70:0/1113400492 shutdown_connections 2024-06-04T04:55:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.780+0000 7fc627b53700 1 --2- 172.21.15.70:0/1113400492 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc60c0b2900 0x7fc60c0b4dc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.780+0000 7fc627b53700 1 --2- 172.21.15.70:0/1113400492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6201088f0 0x7fc620199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.780+0000 7fc627b53700 1 --2- 172.21.15.70:0/1113400492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6201037f0 0x7fc62019f780 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.780+0000 7fc627b53700 1 --2- 172.21.15.70:0/1113400492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6201041a0 0x7fc62019fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.780+0000 7fc627b53700 1 -- 172.21.15.70:0/1113400492 >> 172.21.15.70:0/1113400492 conn(0x7fc62007a6a0 msgr2=0x7fc620101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.780+0000 7fc627b53700 1 -- 172.21.15.70:0/1113400492 shutdown_connections 2024-06-04T04:55:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:37.780+0000 7fc627b53700 1 -- 172.21.15.70:0/1113400492 wait complete. 2024-06-04T04:55:38.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:38 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1113400492' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-04T04:55:38.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:38 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1113400492' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-04T04:55:38.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:38 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1113400492' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-04T04:55:39.190 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:39.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.223+0000 7fb82c821700 1 -- 172.21.15.70:0/95133524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb82806b220 msgr2=0x7fb82806b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:39.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.223+0000 7fb82c821700 1 --2- 172.21.15.70:0/95133524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb82806b220 0x7fb82806b6a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb81c0098f0 tx=0x7fb81c009c00 comp rx=0 tx=0).stop 2024-06-04T04:55:39.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.223+0000 7fb82c821700 1 -- 172.21.15.70:0/95133524 shutdown_connections 2024-06-04T04:55:39.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.223+0000 7fb82c821700 1 --2- 172.21.15.70:0/95133524 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb82806bbe0 0x7fb8281103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:39.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.223+0000 7fb82c821700 1 --2- 172.21.15.70:0/95133524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8281095b0 0x7fb828109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:39.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.223+0000 7fb82c821700 1 --2- 172.21.15.70:0/95133524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb82806b220 0x7fb82806b6a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:39.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.223+0000 7fb82c821700 1 -- 172.21.15.70:0/95133524 >> 172.21.15.70:0/95133524 conn(0x7fb8280793f0 msgr2=0x7fb828079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:39.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.223+0000 7fb82c821700 1 -- 172.21.15.70:0/95133524 shutdown_connections 2024-06-04T04:55:39.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.223+0000 7fb82c821700 1 -- 172.21.15.70:0/95133524 wait complete. 2024-06-04T04:55:39.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.224+0000 7fb82c821700 1 Processor -- start 2024-06-04T04:55:39.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.224+0000 7fb82c821700 1 -- start start 2024-06-04T04:55:39.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.224+0000 7fb82c821700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb82806b220 0x7fb82819f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:39.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.224+0000 7fb82c821700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb82806bbe0 0x7fb82819fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:39.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.224+0000 7fb82c821700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8281095b0 0x7fb828199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:39.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.224+0000 7fb82c821700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb828113570 con 0x7fb82806bbe0 2024-06-04T04:55:39.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.224+0000 7fb82c821700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb8281133f0 con 0x7fb82806b220 2024-06-04T04:55:39.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.224+0000 7fb82c821700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb8281136f0 con 0x7fb8281095b0 2024-06-04T04:55:39.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.224+0000 7fb825d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb82806bbe0 0x7fb82819fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:39.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.224+0000 7fb825d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb82806bbe0 0x7fb82819fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54664/0 (socket says 172.21.15.70:54664) 2024-06-04T04:55:39.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.224+0000 7fb825d9b700 1 -- 172.21.15.70:0/1116556195 learned_addr learned my addr 172.21.15.70:0/1116556195 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:39.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.224+0000 7fb82659c700 1 --2- 172.21.15.70:0/1116556195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb82806b220 0x7fb82819f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:39.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.224+0000 7fb826d9d700 1 --2- 172.21.15.70:0/1116556195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8281095b0 0x7fb828199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:39.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.225+0000 7fb825d9b700 1 -- 172.21.15.70:0/1116556195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8281095b0 msgr2=0x7fb828199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:39.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.225+0000 7fb825d9b700 1 --2- 172.21.15.70:0/1116556195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8281095b0 0x7fb828199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:39.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.225+0000 7fb825d9b700 1 -- 172.21.15.70:0/1116556195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb82806b220 msgr2=0x7fb82819f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:39.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.225+0000 7fb825d9b700 1 --2- 172.21.15.70:0/1116556195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb82806b220 0x7fb82819f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:39.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.225+0000 7fb825d9b700 1 -- 172.21.15.70:0/1116556195 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb82819a190 con 0x7fb82806bbe0 2024-06-04T04:55:39.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.225+0000 7fb825d9b700 1 --2- 172.21.15.70:0/1116556195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb82806bbe0 0x7fb82819fd60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00fa00 tx=0x7fb81c00fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:39.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.225+0000 7fb8177fe700 1 -- 172.21.15.70:0/1116556195 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c027070 con 0x7fb82806bbe0 2024-06-04T04:55:39.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.225+0000 7fb8177fe700 1 -- 172.21.15.70:0/1116556195 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb81c00b9f0 con 0x7fb82806bbe0 2024-06-04T04:55:39.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.225+0000 7fb82c821700 1 -- 172.21.15.70:0/1116556195 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb82819a420 con 0x7fb82806bbe0 2024-06-04T04:55:39.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.225+0000 7fb8177fe700 1 -- 172.21.15.70:0/1116556195 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c0046d0 con 0x7fb82806bbe0 2024-06-04T04:55:39.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.225+0000 7fb82c821700 1 -- 172.21.15.70:0/1116556195 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8281a66f0 con 0x7fb82806bbe0 2024-06-04T04:55:39.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.226+0000 7fb82c821700 1 -- 172.21.15.70:0/1116556195 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb828104c70 con 0x7fb82806bbe0 2024-06-04T04:55:39.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.228+0000 7fb8177fe700 1 -- 172.21.15.70:0/1116556195 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fb81c02a070 con 0x7fb82806bbe0 2024-06-04T04:55:39.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.228+0000 7fb8177fe700 1 --2- 172.21.15.70:0/1116556195 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb8100b2880 0x7fb8100b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:39.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.228+0000 7fb8177fe700 1 -- 172.21.15.70:0/1116556195 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fb81c0e32a0 con 0x7fb82806bbe0 2024-06-04T04:55:39.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.228+0000 7fb82659c700 1 --2- 172.21.15.70:0/1116556195 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb8100b2880 0x7fb8100b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:39.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.229+0000 7fb82659c700 1 --2- 172.21.15.70:0/1116556195 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb8100b2880 0x7fb8100b4d40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb818009680 tx=0x7fb818009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:39.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.232+0000 7fb8177fe700 1 -- 172.21.15.70:0/1116556195 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb81c031070 con 0x7fb82806bbe0 2024-06-04T04:55:39.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.448+0000 7fb82c821700 1 -- 172.21.15.70:0/1116556195 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fb82802e560 con 0x7fb82806bbe0 2024-06-04T04:55:39.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.448+0000 7fb8177fe700 1 -- 172.21.15.70:0/1116556195 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb81c013030 con 0x7fb82806bbe0 2024-06-04T04:55:39.451 INFO:teuthology.orchestra.run.smithi070.stdout:257698037810 2024-06-04T04:55:39.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.451+0000 7fb82c821700 1 -- 172.21.15.70:0/1116556195 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb8100b2880 msgr2=0x7fb8100b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:39.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.451+0000 7fb82c821700 1 --2- 172.21.15.70:0/1116556195 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb8100b2880 0x7fb8100b4d40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb818009680 tx=0x7fb818009260 comp rx=0 tx=0).stop 2024-06-04T04:55:39.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.451+0000 7fb82c821700 1 -- 172.21.15.70:0/1116556195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb82806bbe0 msgr2=0x7fb82819fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:39.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.451+0000 7fb82c821700 1 --2- 172.21.15.70:0/1116556195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb82806bbe0 0x7fb82819fd60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00fa00 tx=0x7fb81c00fa30 comp rx=0 tx=0).stop 2024-06-04T04:55:39.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.451+0000 7fb82c821700 1 -- 172.21.15.70:0/1116556195 shutdown_connections 2024-06-04T04:55:39.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.451+0000 7fb82c821700 1 --2- 172.21.15.70:0/1116556195 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fb8100b2880 0x7fb8100b4d40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:39.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.451+0000 7fb82c821700 1 --2- 172.21.15.70:0/1116556195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8281095b0 0x7fb828199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:39.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.451+0000 7fb82c821700 1 --2- 172.21.15.70:0/1116556195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb82806b220 0x7fb82819f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:39.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.451+0000 7fb82c821700 1 --2- 172.21.15.70:0/1116556195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb82806bbe0 0x7fb82819fd60 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:39.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.451+0000 7fb82c821700 1 -- 172.21.15.70:0/1116556195 >> 172.21.15.70:0/1116556195 conn(0x7fb8280793f0 msgr2=0x7fb828101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:39.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.451+0000 7fb82c821700 1 -- 172.21.15.70:0/1116556195 shutdown_connections 2024-06-04T04:55:39.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:39.451+0000 7fb82c821700 1 -- 172.21.15.70:0/1116556195 wait complete. 2024-06-04T04:55:39.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:39 smithi070 ceph-mon[32631]: pgmap v351: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-06-04T04:55:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:39 smithi112 ceph-mon[36453]: pgmap v351: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-06-04T04:55:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:39 smithi098 ceph-mon[35569]: pgmap v351: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-06-04T04:55:40.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.338+0000 7f3aae39f700 1 -- 172.21.15.70:0/957596340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3aa8102e00 msgr2=0x7f3aa810f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:40.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.338+0000 7f3aae39f700 1 --2- 172.21.15.70:0/957596340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3aa8102e00 0x7f3aa810f2f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3aa000b0a0 tx=0x7f3aa000b3b0 comp rx=0 tx=0).stop 2024-06-04T04:55:40.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.338+0000 7f3aae39f700 1 -- 172.21.15.70:0/957596340 shutdown_connections 2024-06-04T04:55:40.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.338+0000 7f3aae39f700 1 --2- 172.21.15.70:0/957596340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3aa8102e00 0x7f3aa810f2f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:40.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.338+0000 7f3aae39f700 1 --2- 172.21.15.70:0/957596340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3aa8108520 0x7f3aa8108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:40.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.338+0000 7f3aae39f700 1 --2- 172.21.15.70:0/957596340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3aa8102440 0x7f3aa81028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:40.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.338+0000 7f3aae39f700 1 -- 172.21.15.70:0/957596340 >> 172.21.15.70:0/957596340 conn(0x7f3aa80793f0 msgr2=0x7f3aa8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:40.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.338+0000 7f3aae39f700 1 -- 172.21.15.70:0/957596340 shutdown_connections 2024-06-04T04:55:40.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.338+0000 7f3aae39f700 1 -- 172.21.15.70:0/957596340 wait complete. 2024-06-04T04:55:40.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.339+0000 7f3aae39f700 1 Processor -- start 2024-06-04T04:55:40.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.339+0000 7f3aae39f700 1 -- start start 2024-06-04T04:55:40.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.339+0000 7f3aae39f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3aa8102440 0x7f3aa819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:40.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.339+0000 7f3aae39f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3aa8102e00 0x7f3aa819ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:40.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.339+0000 7f3aae39f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3aa8108520 0x7f3aa81a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:40.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.339+0000 7f3aae39f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa8112480 con 0x7f3aa8108520 2024-06-04T04:55:40.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.339+0000 7f3aae39f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa8112300 con 0x7f3aa8102e00 2024-06-04T04:55:40.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.339+0000 7f3aae39f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa8112600 con 0x7f3aa8102440 2024-06-04T04:55:40.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.340+0000 7f3aac93c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3aa8108520 0x7f3aa81a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:40.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.340+0000 7f3aac93c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3aa8108520 0x7f3aa81a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54682/0 (socket says 172.21.15.70:54682) 2024-06-04T04:55:40.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.340+0000 7f3aac93c700 1 -- 172.21.15.70:0/1517897369 learned_addr learned my addr 172.21.15.70:0/1517897369 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:40.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.340+0000 7f3aa7fff700 1 --2- 172.21.15.70:0/1517897369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3aa8102440 0x7f3aa819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:40.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.340+0000 7f3a9ffff700 1 --2- 172.21.15.70:0/1517897369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3aa8102e00 0x7f3aa819ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:40.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.340+0000 7f3aac93c700 1 -- 172.21.15.70:0/1517897369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3aa8102440 msgr2=0x7f3aa819a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:40.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.340+0000 7f3aac93c700 1 --2- 172.21.15.70:0/1517897369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3aa8102440 0x7f3aa819a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:40.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.340+0000 7f3aac93c700 1 -- 172.21.15.70:0/1517897369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3aa8102e00 msgr2=0x7f3aa819ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:40.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.340+0000 7f3aac93c700 1 --2- 172.21.15.70:0/1517897369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3aa8102e00 0x7f3aa819ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:40.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.340+0000 7f3aac93c700 1 -- 172.21.15.70:0/1517897369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa81a5ee0 con 0x7f3aa8108520 2024-06-04T04:55:40.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.340+0000 7f3a9ffff700 1 --2- 172.21.15.70:0/1517897369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3aa8102e00 0x7f3aa819ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:55:40.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.340+0000 7f3aac93c700 1 --2- 172.21.15.70:0/1517897369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3aa8108520 0x7f3aa81a59a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3aa000b830 tx=0x7f3aa0004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:40.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.340+0000 7f3aa5ffb700 1 -- 172.21.15.70:0/1517897369 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0021070 con 0x7f3aa8108520 2024-06-04T04:55:40.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.340+0000 7f3aae39f700 1 -- 172.21.15.70:0/1517897369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aa81a60b0 con 0x7f3aa8108520 2024-06-04T04:55:40.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.340+0000 7f3aa5ffb700 1 -- 172.21.15.70:0/1517897369 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3aa0007b80 con 0x7f3aa8108520 2024-06-04T04:55:40.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.340+0000 7f3aa5ffb700 1 -- 172.21.15.70:0/1517897369 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa00052a0 con 0x7f3aa8108520 2024-06-04T04:55:40.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.340+0000 7f3aae39f700 1 -- 172.21.15.70:0/1517897369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aa81988e0 con 0x7f3aa8108520 2024-06-04T04:55:40.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.342+0000 7f3aae39f700 1 -- 172.21.15.70:0/1517897369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3aa806a230 con 0x7f3aa8108520 2024-06-04T04:55:40.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.343+0000 7f3aa5ffb700 1 -- 172.21.15.70:0/1517897369 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f3aa0024070 con 0x7f3aa8108520 2024-06-04T04:55:40.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.344+0000 7f3aa5ffb700 1 --2- 172.21.15.70:0/1517897369 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3a940b6c90 0x7f3a940b9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:40.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.344+0000 7f3aa7fff700 1 --2- 172.21.15.70:0/1517897369 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3a940b6c90 0x7f3a940b9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:40.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.344+0000 7f3aa5ffb700 1 -- 172.21.15.70:0/1517897369 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f3aa000f020 con 0x7f3aa8108520 2024-06-04T04:55:40.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.344+0000 7f3aa7fff700 1 --2- 172.21.15.70:0/1517897369 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3a940b6c90 0x7f3a940b9150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3a98005950 tx=0x7f3a9800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:40.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.348+0000 7f3aa5ffb700 1 -- 172.21.15.70:0/1517897369 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3aa002b020 con 0x7f3aa8108520 2024-06-04T04:55:40.478 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:40 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1116556195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-04T04:55:40.505 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593790 got 184683593794 for osd.6 2024-06-04T04:55:40.505 DEBUG:teuthology.parallel:result is None 2024-06-04T04:55:40.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.563+0000 7f3aae39f700 1 -- 172.21.15.70:0/1517897369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f3aa802e560 con 0x7f3aa8108520 2024-06-04T04:55:40.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.563+0000 7f3aa5ffb700 1 -- 172.21.15.70:0/1517897369 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3aa00ab600 con 0x7f3aa8108520 2024-06-04T04:55:40.566 INFO:teuthology.orchestra.run.smithi070.stdout:137438953549 2024-06-04T04:55:40.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.566+0000 7f3aae39f700 1 -- 172.21.15.70:0/1517897369 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3a940b6c90 msgr2=0x7f3a940b9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:40.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.566+0000 7f3aae39f700 1 --2- 172.21.15.70:0/1517897369 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3a940b6c90 0x7f3a940b9150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3a98005950 tx=0x7f3a9800b410 comp rx=0 tx=0).stop 2024-06-04T04:55:40.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.566+0000 7f3aae39f700 1 -- 172.21.15.70:0/1517897369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3aa8108520 msgr2=0x7f3aa81a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:40.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.566+0000 7f3aae39f700 1 --2- 172.21.15.70:0/1517897369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3aa8108520 0x7f3aa81a59a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3aa000b830 tx=0x7f3aa0004d00 comp rx=0 tx=0).stop 2024-06-04T04:55:40.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.566+0000 7f3aae39f700 1 -- 172.21.15.70:0/1517897369 shutdown_connections 2024-06-04T04:55:40.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.566+0000 7f3aae39f700 1 --2- 172.21.15.70:0/1517897369 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3a940b6c90 0x7f3a940b9150 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:40.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.566+0000 7f3aae39f700 1 --2- 172.21.15.70:0/1517897369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3aa8108520 0x7f3aa81a59a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:40.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.566+0000 7f3aae39f700 1 --2- 172.21.15.70:0/1517897369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3aa8102440 0x7f3aa819a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:40.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.566+0000 7f3aae39f700 1 --2- 172.21.15.70:0/1517897369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3aa8102e00 0x7f3aa819ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:40.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.566+0000 7f3aae39f700 1 -- 172.21.15.70:0/1517897369 >> 172.21.15.70:0/1517897369 conn(0x7f3aa80793f0 msgr2=0x7f3aa8100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:40.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.566+0000 7f3aae39f700 1 -- 172.21.15.70:0/1517897369 shutdown_connections 2024-06-04T04:55:40.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:40.566+0000 7f3aae39f700 1 -- 172.21.15.70:0/1517897369 wait complete. 2024-06-04T04:55:40.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:40 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1116556195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-04T04:55:40.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:40 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1116556195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-04T04:55:41.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:41 smithi070 ceph-mon[32631]: pgmap v352: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:55:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:41 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1517897369' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-04T04:55:41.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:41 smithi098 ceph-mon[35569]: pgmap v352: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:55:41.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:41 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1517897369' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-04T04:55:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:41 smithi112 ceph-mon[36453]: pgmap v352: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:55:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:41 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1517897369' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-04T04:55:41.700 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953543 got 137438953549 for osd.4 2024-06-04T04:55:41.700 DEBUG:teuthology.parallel:result is None 2024-06-04T04:55:42.458 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234034 got 231928234039 for osd.8 2024-06-04T04:55:42.458 DEBUG:teuthology.parallel:result is None 2024-06-04T04:55:43.416 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037805 got 257698037810 for osd.9 2024-06-04T04:55:43.416 DEBUG:teuthology.parallel:result is None 2024-06-04T04:55:43.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:43 smithi070 ceph-mon[32631]: pgmap v353: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:55:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:43 smithi112 ceph-mon[36453]: pgmap v353: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:55:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:43 smithi098 ceph-mon[35569]: pgmap v353: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:55:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.072+0000 7f0b27d33700 1 -- 172.21.15.70:0/3552507562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b20103850 msgr2=0x7f0b20103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.072+0000 7f0b27d33700 1 --2- 172.21.15.70:0/3552507562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b20103850 0x7f0b20103c30 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0b1000b0d0 tx=0x7f0b1000b3e0 comp rx=0 tx=0).stop 2024-06-04T04:55:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.072+0000 7f0b27d33700 1 -- 172.21.15.70:0/3552507562 shutdown_connections 2024-06-04T04:55:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.072+0000 7f0b27d33700 1 --2- 172.21.15.70:0/3552507562 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0b20108920 0x7f0b201104a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.072+0000 7f0b27d33700 1 --2- 172.21.15.70:0/3552507562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b20103850 0x7f0b20103c30 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.072+0000 7f0b27d33700 1 --2- 172.21.15.70:0/3552507562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b20104200 0x7f0b201081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.072+0000 7f0b27d33700 1 -- 172.21.15.70:0/3552507562 >> 172.21.15.70:0/3552507562 conn(0x7f0b2007a5d0 msgr2=0x7f0b2007a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.072+0000 7f0b27d33700 1 -- 172.21.15.70:0/3552507562 shutdown_connections 2024-06-04T04:55:44.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.072+0000 7f0b27d33700 1 -- 172.21.15.70:0/3552507562 wait complete. 2024-06-04T04:55:44.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.073+0000 7f0b27d33700 1 Processor -- start 2024-06-04T04:55:44.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.073+0000 7f0b27d33700 1 -- start start 2024-06-04T04:55:44.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.073+0000 7f0b27d33700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0b20103850 0x7f0b2019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:44.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.073+0000 7f0b27d33700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b20104200 0x7f0b2019fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:44.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.073+0000 7f0b27d33700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b20108920 0x7f0b201998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:44.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.073+0000 7f0b27d33700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0b201135c0 con 0x7f0b20104200 2024-06-04T04:55:44.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.073+0000 7f0b27d33700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0b20113440 con 0x7f0b20108920 2024-06-04T04:55:44.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.073+0000 7f0b27d33700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0b20113740 con 0x7f0b20103850 2024-06-04T04:55:44.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.073+0000 7f0b252ce700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b20104200 0x7f0b2019fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:44.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.073+0000 7f0b252ce700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b20104200 0x7f0b2019fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54690/0 (socket says 172.21.15.70:54690) 2024-06-04T04:55:44.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.073+0000 7f0b252ce700 1 -- 172.21.15.70:0/3069288338 learned_addr learned my addr 172.21.15.70:0/3069288338 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:44.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.074+0000 7f0b25acf700 1 --2- 172.21.15.70:0/3069288338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0b20103850 0x7f0b2019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:44.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.074+0000 7f0b262d0700 1 --2- 172.21.15.70:0/3069288338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b20108920 0x7f0b201998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:44.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.074+0000 7f0b252ce700 1 -- 172.21.15.70:0/3069288338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0b20103850 msgr2=0x7f0b2019f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:44.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.074+0000 7f0b252ce700 1 --2- 172.21.15.70:0/3069288338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0b20103850 0x7f0b2019f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:44.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.074+0000 7f0b252ce700 1 -- 172.21.15.70:0/3069288338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b20108920 msgr2=0x7f0b201998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:44.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.074+0000 7f0b252ce700 1 --2- 172.21.15.70:0/3069288338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b20108920 0x7f0b201998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:44.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.074+0000 7f0b252ce700 1 -- 172.21.15.70:0/3069288338 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b2019a020 con 0x7f0b20104200 2024-06-04T04:55:44.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.074+0000 7f0b252ce700 1 --2- 172.21.15.70:0/3069288338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b20104200 0x7f0b2019fbf0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0b1400c9b0 tx=0x7f0b1400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:44.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.074+0000 7f0b1effd700 1 -- 172.21.15.70:0/3069288338 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b14007b00 con 0x7f0b20104200 2024-06-04T04:55:44.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.074+0000 7f0b1effd700 1 -- 172.21.15.70:0/3069288338 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0b14007ca0 con 0x7f0b20104200 2024-06-04T04:55:44.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.075+0000 7f0b1effd700 1 -- 172.21.15.70:0/3069288338 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b14018660 con 0x7f0b20104200 2024-06-04T04:55:44.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.075+0000 7f0b27d33700 1 -- 172.21.15.70:0/3069288338 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b2019a310 con 0x7f0b20104200 2024-06-04T04:55:44.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.075+0000 7f0b27d33700 1 -- 172.21.15.70:0/3069288338 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b201a6580 con 0x7f0b20104200 2024-06-04T04:55:44.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.076+0000 7f0b27d33700 1 -- 172.21.15.70:0/3069288338 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b2010bb60 con 0x7f0b20104200 2024-06-04T04:55:44.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.077+0000 7f0b1effd700 1 -- 172.21.15.70:0/3069288338 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f0b1400f450 con 0x7f0b20104200 2024-06-04T04:55:44.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.078+0000 7f0b1effd700 1 --2- 172.21.15.70:0/3069288338 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0b080b2880 0x7f0b080b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:44.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.078+0000 7f0b25acf700 1 --2- 172.21.15.70:0/3069288338 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0b080b2880 0x7f0b080b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:44.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.078+0000 7f0b1effd700 1 -- 172.21.15.70:0/3069288338 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f0b140e22c0 con 0x7f0b20104200 2024-06-04T04:55:44.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.078+0000 7f0b25acf700 1 --2- 172.21.15.70:0/3069288338 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0b080b2880 0x7f0b080b4d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0b10009fd0 tx=0x7f0b1000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:44.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.083+0000 7f0b1effd700 1 -- 172.21.15.70:0/3069288338 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0b14014030 con 0x7f0b20104200 2024-06-04T04:55:44.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.303+0000 7f0b27d33700 1 -- 172.21.15.70:0/3069288338 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f0b2019ada0 con 0x7f0b20104200 2024-06-04T04:55:44.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.303+0000 7f0b1effd700 1 -- 172.21.15.70:0/3069288338 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f0b140a9fc0 con 0x7f0b20104200 2024-06-04T04:55:44.306 INFO:teuthology.orchestra.run.smithi070.stdout:279172874285 2024-06-04T04:55:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.305+0000 7f0b27d33700 1 -- 172.21.15.70:0/3069288338 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0b080b2880 msgr2=0x7f0b080b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.305+0000 7f0b27d33700 1 --2- 172.21.15.70:0/3069288338 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0b080b2880 0x7f0b080b4d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0b10009fd0 tx=0x7f0b1000b040 comp rx=0 tx=0).stop 2024-06-04T04:55:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.305+0000 7f0b27d33700 1 -- 172.21.15.70:0/3069288338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b20104200 msgr2=0x7f0b2019fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.305+0000 7f0b27d33700 1 --2- 172.21.15.70:0/3069288338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b20104200 0x7f0b2019fbf0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0b1400c9b0 tx=0x7f0b1400ccc0 comp rx=0 tx=0).stop 2024-06-04T04:55:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.305+0000 7f0b27d33700 1 -- 172.21.15.70:0/3069288338 shutdown_connections 2024-06-04T04:55:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.305+0000 7f0b27d33700 1 --2- 172.21.15.70:0/3069288338 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0b080b2880 0x7f0b080b4d40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.305+0000 7f0b27d33700 1 --2- 172.21.15.70:0/3069288338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b20108920 0x7f0b201998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.305+0000 7f0b27d33700 1 --2- 172.21.15.70:0/3069288338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0b20103850 0x7f0b2019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.305+0000 7f0b27d33700 1 --2- 172.21.15.70:0/3069288338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b20104200 0x7f0b2019fbf0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:44.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.305+0000 7f0b27d33700 1 -- 172.21.15.70:0/3069288338 >> 172.21.15.70:0/3069288338 conn(0x7f0b2007a5d0 msgr2=0x7f0b20101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:44.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.306+0000 7f0b27d33700 1 -- 172.21.15.70:0/3069288338 shutdown_connections 2024-06-04T04:55:44.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:44.306+0000 7f0b27d33700 1 -- 172.21.15.70:0/3069288338 wait complete. 2024-06-04T04:55:44.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:44 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3069288338' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-04T04:55:44.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:44 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3069288338' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-04T04:55:44.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:44 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3069288338' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-04T04:55:45.018 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874281 got 279172874285 for osd.10 2024-06-04T04:55:45.019 DEBUG:teuthology.parallel:result is None 2024-06-04T04:55:45.019 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-04T04:55:45.019 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-04T04:55:45.275 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:45 smithi070 ceph-mon[32631]: pgmap v354: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-06-04T04:55:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:45 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:45 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:45 smithi112 ceph-mon[36453]: pgmap v354: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-06-04T04:55:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:45 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:45 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:45 smithi098 ceph-mon[35569]: pgmap v354: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-06-04T04:55:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:45 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:45 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:46.419 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:55:46.419 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:55:46.419 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:46.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:55:46.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:55:46.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:46.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:55:46.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:55:46.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:55:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.941+0000 7ff84f7d0700 1 -- 172.21.15.70:0/3938897565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff848102470 msgr2=0x7ff8481028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.941+0000 7ff84f7d0700 1 --2- 172.21.15.70:0/3938897565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff848102470 0x7ff8481028f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff83c009990 tx=0x7ff83c009ca0 comp rx=0 tx=0).stop 2024-06-04T04:55:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.941+0000 7ff84f7d0700 1 -- 172.21.15.70:0/3938897565 shutdown_connections 2024-06-04T04:55:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.941+0000 7ff84f7d0700 1 --2- 172.21.15.70:0/3938897565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff848102e30 0x7ff84810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.941+0000 7ff84f7d0700 1 --2- 172.21.15.70:0/3938897565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8481084b0 0x7ff848108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.941+0000 7ff84f7d0700 1 --2- 172.21.15.70:0/3938897565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff848102470 0x7ff8481028f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.941+0000 7ff84f7d0700 1 -- 172.21.15.70:0/3938897565 >> 172.21.15.70:0/3938897565 conn(0x7ff8480784b0 msgr2=0x7ff8480788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.941+0000 7ff84f7d0700 1 -- 172.21.15.70:0/3938897565 shutdown_connections 2024-06-04T04:55:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.941+0000 7ff84f7d0700 1 -- 172.21.15.70:0/3938897565 wait complete. 2024-06-04T04:55:46.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.941+0000 7ff84f7d0700 1 Processor -- start 2024-06-04T04:55:46.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.941+0000 7ff84f7d0700 1 -- start start 2024-06-04T04:55:46.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84f7d0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff848102470 0x7ff848104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:46.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84f7d0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff848102e30 0x7ff8481045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:46.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84f7d0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8481084b0 0x7ff848104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:46.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84f7d0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff848112470 con 0x7ff8481084b0 2024-06-04T04:55:46.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84f7d0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8481122f0 con 0x7ff848102e30 2024-06-04T04:55:46.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84f7d0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff8481125f0 con 0x7ff848102470 2024-06-04T04:55:46.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84dd6d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8481084b0 0x7ff848104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:46.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84dd6d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8481084b0 0x7ff848104b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34474/0 (socket says 172.21.15.70:34474) 2024-06-04T04:55:46.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84dd6d700 1 -- 172.21.15.70:0/50468659 learned_addr learned my addr 172.21.15.70:0/50468659 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:46.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84d56c700 1 --2- 172.21.15.70:0/50468659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff848102470 0x7ff848104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:46.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84cd6b700 1 --2- 172.21.15.70:0/50468659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff848102e30 0x7ff8481045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:46.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84dd6d700 1 -- 172.21.15.70:0/50468659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff848102470 msgr2=0x7ff848104080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:46.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84dd6d700 1 --2- 172.21.15.70:0/50468659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff848102470 0x7ff848104080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:46.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84dd6d700 1 -- 172.21.15.70:0/50468659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff848102e30 msgr2=0x7ff8481045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:46.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84dd6d700 1 --2- 172.21.15.70:0/50468659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff848102e30 0x7ff8481045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:46.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84dd6d700 1 -- 172.21.15.70:0/50468659 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8481a8090 con 0x7ff8481084b0 2024-06-04T04:55:46.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84dd6d700 1 --2- 172.21.15.70:0/50468659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8481084b0 0x7ff848104b00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff84000e940 tx=0x7ff84000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:46.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff8467fc700 1 -- 172.21.15.70:0/50468659 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff840004a30 con 0x7ff8481084b0 2024-06-04T04:55:46.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff84f7d0700 1 -- 172.21.15.70:0/50468659 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8481a8380 con 0x7ff8481084b0 2024-06-04T04:55:46.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff8467fc700 1 -- 172.21.15.70:0/50468659 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff8400054b0 con 0x7ff8481084b0 2024-06-04T04:55:46.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.942+0000 7ff8467fc700 1 -- 172.21.15.70:0/50468659 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84000cbe0 con 0x7ff8481084b0 2024-06-04T04:55:46.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.943+0000 7ff84f7d0700 1 -- 172.21.15.70:0/50468659 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8481a8890 con 0x7ff8481084b0 2024-06-04T04:55:46.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.944+0000 7ff84f7d0700 1 -- 172.21.15.70:0/50468659 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff848103b60 con 0x7ff8481084b0 2024-06-04T04:55:46.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.945+0000 7ff8467fc700 1 -- 172.21.15.70:0/50468659 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7ff840005000 con 0x7ff8481084b0 2024-06-04T04:55:46.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.946+0000 7ff8467fc700 1 --2- 172.21.15.70:0/50468659 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff8300b2880 0x7ff8300b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:46.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.946+0000 7ff84d56c700 1 --2- 172.21.15.70:0/50468659 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff8300b2880 0x7ff8300b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:46.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.946+0000 7ff8467fc700 1 -- 172.21.15.70:0/50468659 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7ff8400e2600 con 0x7ff8481084b0 2024-06-04T04:55:46.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.946+0000 7ff84d56c700 1 --2- 172.21.15.70:0/50468659 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff8300b2880 0x7ff8300b4d40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff838009550 tx=0x7ff8380091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:46.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:46.949+0000 7ff8467fc700 1 -- 172.21.15.70:0/50468659 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff840054df0 con 0x7ff8481084b0 2024-06-04T04:55:47.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:47.150+0000 7ff84f7d0700 1 -- 172.21.15.70:0/50468659 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff848063980 con 0x7ff8300b2880 2024-06-04T04:55:47.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:47.158+0000 7ff8467fc700 1 -- 172.21.15.70:0/50468659 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259123 (secure 0 0 0) 0x7ff848063980 con 0x7ff8300b2880 2024-06-04T04:55:47.163 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:55:47.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:47.161+0000 7ff84f7d0700 1 -- 172.21.15.70:0/50468659 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff8300b2880 msgr2=0x7ff8300b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:47.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:47.161+0000 7ff84f7d0700 1 --2- 172.21.15.70:0/50468659 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff8300b2880 0x7ff8300b4d40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff838009550 tx=0x7ff8380091b0 comp rx=0 tx=0).stop 2024-06-04T04:55:47.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:47.161+0000 7ff84f7d0700 1 -- 172.21.15.70:0/50468659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8481084b0 msgr2=0x7ff848104b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:47.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:47.161+0000 7ff84f7d0700 1 --2- 172.21.15.70:0/50468659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8481084b0 0x7ff848104b00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff84000e940 tx=0x7ff84000ec50 comp rx=0 tx=0).stop 2024-06-04T04:55:47.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:47.161+0000 7ff84f7d0700 1 -- 172.21.15.70:0/50468659 shutdown_connections 2024-06-04T04:55:47.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:47.161+0000 7ff84f7d0700 1 --2- 172.21.15.70:0/50468659 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ff8300b2880 0x7ff8300b4d40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:47.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:47.161+0000 7ff84f7d0700 1 --2- 172.21.15.70:0/50468659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8481084b0 0x7ff848104b00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:47.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:47.161+0000 7ff84f7d0700 1 --2- 172.21.15.70:0/50468659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff848102470 0x7ff848104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:47.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:47.161+0000 7ff84f7d0700 1 --2- 172.21.15.70:0/50468659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff848102e30 0x7ff8481045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:47.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:47.161+0000 7ff84f7d0700 1 -- 172.21.15.70:0/50468659 >> 172.21.15.70:0/50468659 conn(0x7ff8480784b0 msgr2=0x7ff84806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:47.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:47.161+0000 7ff84f7d0700 1 -- 172.21.15.70:0/50468659 shutdown_connections 2024-06-04T04:55:47.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:47.161+0000 7ff84f7d0700 1 -- 172.21.15.70:0/50468659 wait complete. 2024-06-04T04:55:47.167 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-06-04T04:55:47.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:47 smithi070 ceph-mon[32631]: pgmap v355: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-06-04T04:55:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:47 smithi112 ceph-mon[36453]: pgmap v355: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-06-04T04:55:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:47 smithi098 ceph-mon[35569]: pgmap v355: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-06-04T04:55:47.789 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":355,"stamp":"2024-06-04T04:55:45.753007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664008,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1212,"num_read_kb":710,"num_write":377,"num_write_kb":2299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2952512,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":811,"ondisk_log_size":811,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":355900,"kb_used_data":10908,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585924,"statfs":{"total":1151940427776,"available":1151575986176,"internally_reserved":0,"allocated":11169792,"data_stored":7132440,"data_compressed":182883,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19138,"internal_metadata":353154366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":54,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002842"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.759859+0000","last_change":"2024-06-04T04:53:36.383755+0000","last_active":"2024-06-04T04:55:17.759859+0000","last_peered":"2024-06-04T04:55:17.759859+0000","last_clean":"2024-06-04T04:55:17.759859+0000","last_became_active":"2024-06-04T04:53:36.378274+0000","last_became_peered":"2024-06-04T04:53:36.378274+0000","last_unstale":"2024-06-04T04:55:17.759859+0000","last_undegraded":"2024-06-04T04:55:17.759859+0000","last_fullsized":"2024-06-04T04:55:17.759859+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:30:19.685994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758036+0000","last_change":"2024-06-04T04:53:36.372436+0000","last_active":"2024-06-04T04:55:17.758036+0000","last_peered":"2024-06-04T04:55:17.758036+0000","last_clean":"2024-06-04T04:55:17.758036+0000","last_became_active":"2024-06-04T04:53:36.372100+0000","last_became_peered":"2024-06-04T04:53:36.372100+0000","last_unstale":"2024-06-04T04:55:17.758036+0000","last_undegraded":"2024-06-04T04:55:17.758036+0000","last_fullsized":"2024-06-04T04:55:17.758036+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:42:31.646754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758260+0000","last_change":"2024-06-04T04:53:36.366915+0000","last_active":"2024-06-04T04:55:17.758260+0000","last_peered":"2024-06-04T04:55:17.758260+0000","last_clean":"2024-06-04T04:55:17.758260+0000","last_became_active":"2024-06-04T04:53:36.366611+0000","last_became_peered":"2024-06-04T04:53:36.366611+0000","last_unstale":"2024-06-04T04:55:17.758260+0000","last_undegraded":"2024-06-04T04:55:17.758260+0000","last_fullsized":"2024-06-04T04:55:17.758260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:17:48.775938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.747879+0000","last_change":"2024-06-04T04:53:36.374486+0000","last_active":"2024-06-04T04:55:17.747879+0000","last_peered":"2024-06-04T04:55:17.747879+0000","last_clean":"2024-06-04T04:55:17.747879+0000","last_became_active":"2024-06-04T04:53:36.374178+0000","last_became_peered":"2024-06-04T04:53:36.374178+0000","last_unstale":"2024-06-04T04:55:17.747879+0000","last_undegraded":"2024-06-04T04:55:17.747879+0000","last_fullsized":"2024-06-04T04:55:17.747879+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:30:28.287070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.748587+0000","last_change":"2024-06-04T04:53:36.376183+0000","last_active":"2024-06-04T04:55:17.748587+0000","last_peered":"2024-06-04T04:55:17.748587+0000","last_clean":"2024-06-04T04:55:17.748587+0000","last_became_active":"2024-06-04T04:53:36.375790+0000","last_became_peered":"2024-06-04T04:53:36.375790+0000","last_unstale":"2024-06-04T04:55:17.748587+0000","last_undegraded":"2024-06-04T04:55:17.748587+0000","last_fullsized":"2024-06-04T04:55:17.748587+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:58.697440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.744541+0000","last_change":"2024-06-04T04:53:36.370249+0000","last_active":"2024-06-04T04:55:17.744541+0000","last_peered":"2024-06-04T04:55:17.744541+0000","last_clean":"2024-06-04T04:55:17.744541+0000","last_became_active":"2024-06-04T04:53:36.369916+0000","last_became_peered":"2024-06-04T04:53:36.369916+0000","last_unstale":"2024-06-04T04:55:17.744541+0000","last_undegraded":"2024-06-04T04:55:17.744541+0000","last_fullsized":"2024-06-04T04:55:17.744541+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:02:41.739806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.755306+0000","last_change":"2024-06-04T04:53:36.384329+0000","last_active":"2024-06-04T04:55:17.755306+0000","last_peered":"2024-06-04T04:55:17.755306+0000","last_clean":"2024-06-04T04:55:17.755306+0000","last_became_active":"2024-06-04T04:53:36.382504+0000","last_became_peered":"2024-06-04T04:53:36.382504+0000","last_unstale":"2024-06-04T04:55:17.755306+0000","last_undegraded":"2024-06-04T04:55:17.755306+0000","last_fullsized":"2024-06-04T04:55:17.755306+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:51:07.604691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.745731+0000","last_change":"2024-06-04T04:53:36.372319+0000","last_active":"2024-06-04T04:55:17.745731+0000","last_peered":"2024-06-04T04:55:17.745731+0000","last_clean":"2024-06-04T04:55:17.745731+0000","last_became_active":"2024-06-04T04:53:36.372000+0000","last_became_peered":"2024-06-04T04:53:36.372000+0000","last_unstale":"2024-06-04T04:55:17.745731+0000","last_undegraded":"2024-06-04T04:55:17.745731+0000","last_fullsized":"2024-06-04T04:55:17.745731+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:15:56.906996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.752310+0000","last_change":"2024-06-04T04:53:36.376051+0000","last_active":"2024-06-04T04:55:17.752310+0000","last_peered":"2024-06-04T04:55:17.752310+0000","last_clean":"2024-06-04T04:55:17.752310+0000","last_became_active":"2024-06-04T04:53:36.375726+0000","last_became_peered":"2024-06-04T04:53:36.375726+0000","last_unstale":"2024-06-04T04:55:17.752310+0000","last_undegraded":"2024-06-04T04:55:17.752310+0000","last_fullsized":"2024-06-04T04:55:17.752310+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:09:08.542764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.745381+0000","last_change":"2024-06-04T04:53:36.373266+0000","last_active":"2024-06-04T04:55:17.745381+0000","last_peered":"2024-06-04T04:55:17.745381+0000","last_clean":"2024-06-04T04:55:17.745381+0000","last_became_active":"2024-06-04T04:53:36.372898+0000","last_became_peered":"2024-06-04T04:53:36.372898+0000","last_unstale":"2024-06-04T04:55:17.745381+0000","last_undegraded":"2024-06-04T04:55:17.745381+0000","last_fullsized":"2024-06-04T04:55:17.745381+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:41:43.436781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.759967+0000","last_change":"2024-06-04T04:53:36.381344+0000","last_active":"2024-06-04T04:55:17.759967+0000","last_peered":"2024-06-04T04:55:17.759967+0000","last_clean":"2024-06-04T04:55:17.759967+0000","last_became_active":"2024-06-04T04:53:36.380736+0000","last_became_peered":"2024-06-04T04:53:36.380736+0000","last_unstale":"2024-06-04T04:55:17.759967+0000","last_undegraded":"2024-06-04T04:55:17.759967+0000","last_fullsized":"2024-06-04T04:55:17.759967+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:23:01.607272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.736749+0000","last_change":"2024-06-04T04:53:36.374035+0000","last_active":"2024-06-04T04:55:17.736749+0000","last_peered":"2024-06-04T04:55:17.736749+0000","last_clean":"2024-06-04T04:55:17.736749+0000","last_became_active":"2024-06-04T04:53:36.373667+0000","last_became_peered":"2024-06-04T04:53:36.373667+0000","last_unstale":"2024-06-04T04:55:17.736749+0000","last_undegraded":"2024-06-04T04:55:17.736749+0000","last_fullsized":"2024-06-04T04:55:17.736749+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:23:07.880369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.746826+0000","last_change":"2024-06-04T04:53:36.369613+0000","last_active":"2024-06-04T04:55:17.746826+0000","last_peered":"2024-06-04T04:55:17.746826+0000","last_clean":"2024-06-04T04:55:17.746826+0000","last_became_active":"2024-06-04T04:53:36.369277+0000","last_became_peered":"2024-06-04T04:53:36.369277+0000","last_unstale":"2024-06-04T04:55:17.746826+0000","last_undegraded":"2024-06-04T04:55:17.746826+0000","last_fullsized":"2024-06-04T04:55:17.746826+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:13:33.756609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.754113+0000","last_change":"2024-06-04T04:53:36.368094+0000","last_active":"2024-06-04T04:55:17.754113+0000","last_peered":"2024-06-04T04:55:17.754113+0000","last_clean":"2024-06-04T04:55:17.754113+0000","last_became_active":"2024-06-04T04:53:36.367717+0000","last_became_peered":"2024-06-04T04:53:36.367717+0000","last_unstale":"2024-06-04T04:55:17.754113+0000","last_undegraded":"2024-06-04T04:55:17.754113+0000","last_fullsized":"2024-06-04T04:55:17.754113+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:39:56.305078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.757070+0000","last_change":"2024-06-04T04:53:36.366765+0000","last_active":"2024-06-04T04:55:17.757070+0000","last_peered":"2024-06-04T04:55:17.757070+0000","last_clean":"2024-06-04T04:55:17.757070+0000","last_became_active":"2024-06-04T04:53:36.366438+0000","last_became_peered":"2024-06-04T04:53:36.366438+0000","last_unstale":"2024-06-04T04:55:17.757070+0000","last_undegraded":"2024-06-04T04:55:17.757070+0000","last_fullsized":"2024-06-04T04:55:17.757070+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:38:37.313926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758873+0000","last_change":"2024-06-04T04:53:36.380088+0000","last_active":"2024-06-04T04:55:17.758873+0000","last_peered":"2024-06-04T04:55:17.758873+0000","last_clean":"2024-06-04T04:55:17.758873+0000","last_became_active":"2024-06-04T04:53:36.379467+0000","last_became_peered":"2024-06-04T04:53:36.379467+0000","last_unstale":"2024-06-04T04:55:17.758873+0000","last_undegraded":"2024-06-04T04:55:17.758873+0000","last_fullsized":"2024-06-04T04:55:17.758873+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:46:53.236419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.759763+0000","last_change":"2024-06-04T04:53:36.381126+0000","last_active":"2024-06-04T04:55:17.759763+0000","last_peered":"2024-06-04T04:55:17.759763+0000","last_clean":"2024-06-04T04:55:17.759763+0000","last_became_active":"2024-06-04T04:53:36.378833+0000","last_became_peered":"2024-06-04T04:53:36.378833+0000","last_unstale":"2024-06-04T04:55:17.759763+0000","last_undegraded":"2024-06-04T04:55:17.759763+0000","last_fullsized":"2024-06-04T04:55:17.759763+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:20:28.639273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.753871+0000","last_change":"2024-06-04T04:53:36.369692+0000","last_active":"2024-06-04T04:55:17.753871+0000","last_peered":"2024-06-04T04:55:17.753871+0000","last_clean":"2024-06-04T04:55:17.753871+0000","last_became_active":"2024-06-04T04:53:36.369370+0000","last_became_peered":"2024-06-04T04:53:36.369370+0000","last_unstale":"2024-06-04T04:55:17.753871+0000","last_undegraded":"2024-06-04T04:55:17.753871+0000","last_fullsized":"2024-06-04T04:55:17.753871+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:55.275465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.749875+0000","last_change":"2024-06-04T04:53:36.383639+0000","last_active":"2024-06-04T04:55:17.749875+0000","last_peered":"2024-06-04T04:55:17.749875+0000","last_clean":"2024-06-04T04:55:17.749875+0000","last_became_active":"2024-06-04T04:53:36.383303+0000","last_became_peered":"2024-06-04T04:53:36.383303+0000","last_unstale":"2024-06-04T04:55:17.749875+0000","last_undegraded":"2024-06-04T04:55:17.749875+0000","last_fullsized":"2024-06-04T04:55:17.749875+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:26:31.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'14","reported_seq":171,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:41.744731+0000","last_change":"2024-06-04T04:54:19.641520+0000","last_active":"2024-06-04T04:55:41.744731+0000","last_peered":"2024-06-04T04:55:41.744731+0000","last_clean":"2024-06-04T04:55:41.744731+0000","last_became_active":"2024-06-04T04:54:19.632433+0000","last_became_peered":"2024-06-04T04:54:19.632433+0000","last_unstale":"2024-06-04T04:55:41.744731+0000","last_undegraded":"2024-06-04T04:55:41.744731+0000","last_fullsized":"2024-06-04T04:55:41.744731+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:26:05.993919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.744423+0000","last_change":"2024-06-04T04:53:36.371827+0000","last_active":"2024-06-04T04:55:17.744423+0000","last_peered":"2024-06-04T04:55:17.744423+0000","last_clean":"2024-06-04T04:55:17.744423+0000","last_became_active":"2024-06-04T04:53:36.371509+0000","last_became_peered":"2024-06-04T04:53:36.371509+0000","last_unstale":"2024-06-04T04:55:17.744423+0000","last_undegraded":"2024-06-04T04:55:17.744423+0000","last_fullsized":"2024-06-04T04:55:17.744423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:54:51.589008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.513230+0000","last_change":"2024-06-04T04:53:33.353976+0000","last_active":"2024-06-04T04:54:19.513230+0000","last_peered":"2024-06-04T04:54:19.513230+0000","last_clean":"2024-06-04T04:54:19.513230+0000","last_became_active":"2024-06-04T04:53:33.353586+0000","last_became_peered":"2024-06-04T04:53:33.353586+0000","last_unstale":"2024-06-04T04:54:19.513230+0000","last_undegraded":"2024-06-04T04:54:19.513230+0000","last_fullsized":"2024-06-04T04:54:19.513230+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:17:10.378008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.759061+0000","last_change":"2024-06-04T04:53:36.367806+0000","last_active":"2024-06-04T04:55:17.759061+0000","last_peered":"2024-06-04T04:55:17.759061+0000","last_clean":"2024-06-04T04:55:17.759061+0000","last_became_active":"2024-06-04T04:53:36.367484+0000","last_became_peered":"2024-06-04T04:53:36.367484+0000","last_unstale":"2024-06-04T04:55:17.759061+0000","last_undegraded":"2024-06-04T04:55:17.759061+0000","last_fullsized":"2024-06-04T04:55:17.759061+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:25:36.194746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509723+0000","last_change":"2024-06-04T04:53:33.348902+0000","last_active":"2024-06-04T04:54:19.509723+0000","last_peered":"2024-06-04T04:54:19.509723+0000","last_clean":"2024-06-04T04:54:19.509723+0000","last_became_active":"2024-06-04T04:53:33.348441+0000","last_became_peered":"2024-06-04T04:53:33.348441+0000","last_unstale":"2024-06-04T04:54:19.509723+0000","last_undegraded":"2024-06-04T04:54:19.509723+0000","last_fullsized":"2024-06-04T04:54:19.509723+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:22:28.428303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.748336+0000","last_change":"2024-06-04T04:53:36.376867+0000","last_active":"2024-06-04T04:55:17.748336+0000","last_peered":"2024-06-04T04:55:17.748336+0000","last_clean":"2024-06-04T04:55:17.748336+0000","last_became_active":"2024-06-04T04:53:36.376517+0000","last_became_peered":"2024-06-04T04:53:36.376517+0000","last_unstale":"2024-06-04T04:55:17.748336+0000","last_undegraded":"2024-06-04T04:55:17.748336+0000","last_fullsized":"2024-06-04T04:55:17.748336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:21:59.494321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509187+0000","last_change":"2024-06-04T04:53:33.347993+0000","last_active":"2024-06-04T04:54:19.509187+0000","last_peered":"2024-06-04T04:54:19.509187+0000","last_clean":"2024-06-04T04:54:19.509187+0000","last_became_active":"2024-06-04T04:53:33.347656+0000","last_became_peered":"2024-06-04T04:53:33.347656+0000","last_unstale":"2024-06-04T04:54:19.509187+0000","last_undegraded":"2024-06-04T04:54:19.509187+0000","last_fullsized":"2024-06-04T04:54:19.509187+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:21:26.806557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.751286+0000","last_change":"2024-06-04T04:54:19.632790+0000","last_active":"2024-06-04T04:55:17.751286+0000","last_peered":"2024-06-04T04:55:17.751286+0000","last_clean":"2024-06-04T04:55:17.751286+0000","last_became_active":"2024-06-04T04:54:19.632338+0000","last_became_peered":"2024-06-04T04:54:19.632338+0000","last_unstale":"2024-06-04T04:55:17.751286+0000","last_undegraded":"2024-06-04T04:55:17.751286+0000","last_fullsized":"2024-06-04T04:55:17.751286+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:44:00.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'5","reported_seq":144,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:43.611373+0000","last_change":"2024-06-04T04:53:33.346111+0000","last_active":"2024-06-04T04:55:43.611373+0000","last_peered":"2024-06-04T04:55:43.611373+0000","last_clean":"2024-06-04T04:55:43.611373+0000","last_became_active":"2024-06-04T04:53:33.345773+0000","last_became_peered":"2024-06-04T04:53:33.345773+0000","last_unstale":"2024-06-04T04:55:43.611373+0000","last_undegraded":"2024-06-04T04:55:43.611373+0000","last_fullsized":"2024-06-04T04:55:43.611373+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:18.565583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.738921+0000","last_change":"2024-06-04T04:53:36.373369+0000","last_active":"2024-06-04T04:55:17.738921+0000","last_peered":"2024-06-04T04:55:17.738921+0000","last_clean":"2024-06-04T04:55:17.738921+0000","last_became_active":"2024-06-04T04:53:36.373001+0000","last_became_peered":"2024-06-04T04:53:36.373001+0000","last_unstale":"2024-06-04T04:55:17.738921+0000","last_undegraded":"2024-06-04T04:55:17.738921+0000","last_fullsized":"2024-06-04T04:55:17.738921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:02:22.012643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508915+0000","last_change":"2024-06-04T04:53:33.344358+0000","last_active":"2024-06-04T04:54:19.508915+0000","last_peered":"2024-06-04T04:54:19.508915+0000","last_clean":"2024-06-04T04:54:19.508915+0000","last_became_active":"2024-06-04T04:53:33.343980+0000","last_became_peered":"2024-06-04T04:53:33.343980+0000","last_unstale":"2024-06-04T04:54:19.508915+0000","last_undegraded":"2024-06-04T04:54:19.508915+0000","last_fullsized":"2024-06-04T04:54:19.508915+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:14.493138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.745556+0000","last_change":"2024-06-04T04:53:36.371239+0000","last_active":"2024-06-04T04:55:17.745556+0000","last_peered":"2024-06-04T04:55:17.745556+0000","last_clean":"2024-06-04T04:55:17.745556+0000","last_became_active":"2024-06-04T04:53:36.370885+0000","last_became_peered":"2024-06-04T04:53:36.370885+0000","last_unstale":"2024-06-04T04:55:17.745556+0000","last_undegraded":"2024-06-04T04:55:17.745556+0000","last_fullsized":"2024-06-04T04:55:17.745556+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T04:54:29.753796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505354+0000","last_change":"2024-06-04T04:53:33.338512+0000","last_active":"2024-06-04T04:54:19.505354+0000","last_peered":"2024-06-04T04:54:19.505354+0000","last_clean":"2024-06-04T04:54:19.505354+0000","last_became_active":"2024-06-04T04:53:33.338180+0000","last_became_peered":"2024-06-04T04:53:33.338180+0000","last_unstale":"2024-06-04T04:54:19.505354+0000","last_undegraded":"2024-06-04T04:54:19.505354+0000","last_fullsized":"2024-06-04T04:54:19.505354+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:57:13.406479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.744719+0000","last_change":"2024-06-04T04:53:36.377316+0000","last_active":"2024-06-04T04:55:17.744719+0000","last_peered":"2024-06-04T04:55:17.744719+0000","last_clean":"2024-06-04T04:55:17.744719+0000","last_became_active":"2024-06-04T04:53:36.376992+0000","last_became_peered":"2024-06-04T04:53:36.376992+0000","last_unstale":"2024-06-04T04:55:17.744719+0000","last_undegraded":"2024-06-04T04:55:17.744719+0000","last_fullsized":"2024-06-04T04:55:17.744719+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:33.145631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511696+0000","last_change":"2024-06-04T04:53:33.348305+0000","last_active":"2024-06-04T04:54:19.511696+0000","last_peered":"2024-06-04T04:54:19.511696+0000","last_clean":"2024-06-04T04:54:19.511696+0000","last_became_active":"2024-06-04T04:53:33.347948+0000","last_became_peered":"2024-06-04T04:53:33.347948+0000","last_unstale":"2024-06-04T04:54:19.511696+0000","last_undegraded":"2024-06-04T04:54:19.511696+0000","last_fullsized":"2024-06-04T04:54:19.511696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:04:13.669274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.752489+0000","last_change":"2024-06-04T04:53:36.367955+0000","last_active":"2024-06-04T04:55:17.752489+0000","last_peered":"2024-06-04T04:55:17.752489+0000","last_clean":"2024-06-04T04:55:17.752489+0000","last_became_active":"2024-06-04T04:53:36.367628+0000","last_became_peered":"2024-06-04T04:53:36.367628+0000","last_unstale":"2024-06-04T04:55:17.752489+0000","last_undegraded":"2024-06-04T04:55:17.752489+0000","last_fullsized":"2024-06-04T04:55:17.752489+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:33:26.185498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'4","reported_seq":246,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:41.646374+0000","last_change":"2024-06-04T04:53:33.349471+0000","last_active":"2024-06-04T04:55:41.646374+0000","last_peered":"2024-06-04T04:55:41.646374+0000","last_clean":"2024-06-04T04:55:41.646374+0000","last_became_active":"2024-06-04T04:53:33.349105+0000","last_became_peered":"2024-06-04T04:53:33.349105+0000","last_unstale":"2024-06-04T04:55:41.646374+0000","last_undegraded":"2024-06-04T04:55:41.646374+0000","last_fullsized":"2024-06-04T04:55:41.646374+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:31:35.584275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":217,"num_read_kb":217,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.744175+0000","last_change":"2024-06-04T04:53:36.367249+0000","last_active":"2024-06-04T04:55:17.744175+0000","last_peered":"2024-06-04T04:55:17.744175+0000","last_clean":"2024-06-04T04:55:17.744175+0000","last_became_active":"2024-06-04T04:53:36.366863+0000","last_became_peered":"2024-06-04T04:53:36.366863+0000","last_unstale":"2024-06-04T04:55:17.744175+0000","last_undegraded":"2024-06-04T04:55:17.744175+0000","last_fullsized":"2024-06-04T04:55:17.744175+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:13:17.308321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510836+0000","last_change":"2024-06-04T04:53:33.346778+0000","last_active":"2024-06-04T04:54:19.510836+0000","last_peered":"2024-06-04T04:54:19.510836+0000","last_clean":"2024-06-04T04:54:19.510836+0000","last_became_active":"2024-06-04T04:53:33.346376+0000","last_became_peered":"2024-06-04T04:53:33.346376+0000","last_unstale":"2024-06-04T04:54:19.510836+0000","last_undegraded":"2024-06-04T04:54:19.510836+0000","last_fullsized":"2024-06-04T04:54:19.510836+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:03:16.973822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758785+0000","last_change":"2024-06-04T04:54:19.514491+0000","last_active":"2024-06-04T04:55:17.758785+0000","last_peered":"2024-06-04T04:55:17.758785+0000","last_clean":"2024-06-04T04:55:17.758785+0000","last_became_active":"2024-06-04T04:54:19.514148+0000","last_became_peered":"2024-06-04T04:54:19.514148+0000","last_unstale":"2024-06-04T04:55:17.758785+0000","last_undegraded":"2024-06-04T04:55:17.758785+0000","last_fullsized":"2024-06-04T04:55:17.758785+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:09:59.641041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512326+0000","last_change":"2024-06-04T04:53:33.354989+0000","last_active":"2024-06-04T04:54:19.512326+0000","last_peered":"2024-06-04T04:54:19.512326+0000","last_clean":"2024-06-04T04:54:19.512326+0000","last_became_active":"2024-06-04T04:53:33.354613+0000","last_became_peered":"2024-06-04T04:53:33.354613+0000","last_unstale":"2024-06-04T04:54:19.512326+0000","last_undegraded":"2024-06-04T04:54:19.512326+0000","last_fullsized":"2024-06-04T04:54:19.512326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:19:59.398763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.752224+0000","last_change":"2024-06-04T04:53:36.372216+0000","last_active":"2024-06-04T04:55:17.752224+0000","last_peered":"2024-06-04T04:55:17.752224+0000","last_clean":"2024-06-04T04:55:17.752224+0000","last_became_active":"2024-06-04T04:53:36.371820+0000","last_became_peered":"2024-06-04T04:53:36.371820+0000","last_unstale":"2024-06-04T04:55:17.752224+0000","last_undegraded":"2024-06-04T04:55:17.752224+0000","last_fullsized":"2024-06-04T04:55:17.752224+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:55:59.887963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'37","reported_seq":288,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:41.621268+0000","last_change":"2024-06-04T04:53:33.345493+0000","last_active":"2024-06-04T04:55:41.621268+0000","last_peered":"2024-06-04T04:55:41.621268+0000","last_clean":"2024-06-04T04:55:41.621268+0000","last_became_active":"2024-06-04T04:53:33.345102+0000","last_became_peered":"2024-06-04T04:53:33.345102+0000","last_unstale":"2024-06-04T04:55:41.621268+0000","last_undegraded":"2024-06-04T04:55:41.621268+0000","last_fullsized":"2024-06-04T04:55:41.621268+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:50:27.853391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66452,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":226,"num_read_kb":178,"num_write":39,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758459+0000","last_change":"2024-06-04T04:53:36.382070+0000","last_active":"2024-06-04T04:55:17.758459+0000","last_peered":"2024-06-04T04:55:17.758459+0000","last_clean":"2024-06-04T04:55:17.758459+0000","last_became_active":"2024-06-04T04:53:36.379144+0000","last_became_peered":"2024-06-04T04:53:36.379144+0000","last_unstale":"2024-06-04T04:55:17.758459+0000","last_undegraded":"2024-06-04T04:55:17.758459+0000","last_fullsized":"2024-06-04T04:55:17.758459+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:16.702709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.506537+0000","last_change":"2024-06-04T04:53:33.344726+0000","last_active":"2024-06-04T04:54:19.506537+0000","last_peered":"2024-06-04T04:54:19.506537+0000","last_clean":"2024-06-04T04:54:19.506537+0000","last_became_active":"2024-06-04T04:53:33.344360+0000","last_became_peered":"2024-06-04T04:53:33.344360+0000","last_unstale":"2024-06-04T04:54:19.506537+0000","last_undegraded":"2024-06-04T04:54:19.506537+0000","last_fullsized":"2024-06-04T04:54:19.506537+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:44:56.784808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.751057+0000","last_change":"2024-06-04T04:53:36.379036+0000","last_active":"2024-06-04T04:55:17.751057+0000","last_peered":"2024-06-04T04:55:17.751057+0000","last_clean":"2024-06-04T04:55:17.751057+0000","last_became_active":"2024-06-04T04:53:36.378681+0000","last_became_peered":"2024-06-04T04:53:36.378681+0000","last_unstale":"2024-06-04T04:55:17.751057+0000","last_undegraded":"2024-06-04T04:55:17.751057+0000","last_fullsized":"2024-06-04T04:55:17.751057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:07:49.704731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511967+0000","last_change":"2024-06-04T04:53:33.345915+0000","last_active":"2024-06-04T04:54:19.511967+0000","last_peered":"2024-06-04T04:54:19.511967+0000","last_clean":"2024-06-04T04:54:19.511967+0000","last_became_active":"2024-06-04T04:53:33.345537+0000","last_became_peered":"2024-06-04T04:53:33.345537+0000","last_unstale":"2024-06-04T04:54:19.511967+0000","last_undegraded":"2024-06-04T04:54:19.511967+0000","last_fullsized":"2024-06-04T04:54:19.511967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:06:44.086968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.757276+0000","last_change":"2024-06-04T04:53:36.379094+0000","last_active":"2024-06-04T04:55:17.757276+0000","last_peered":"2024-06-04T04:55:17.757276+0000","last_clean":"2024-06-04T04:55:17.757276+0000","last_became_active":"2024-06-04T04:53:36.378763+0000","last_became_peered":"2024-06-04T04:53:36.378763+0000","last_unstale":"2024-06-04T04:55:17.757276+0000","last_undegraded":"2024-06-04T04:55:17.757276+0000","last_fullsized":"2024-06-04T04:55:17.757276+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:16.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":144,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:43.610328+0000","last_change":"2024-06-04T04:53:33.341342+0000","last_active":"2024-06-04T04:55:43.610328+0000","last_peered":"2024-06-04T04:55:43.610328+0000","last_clean":"2024-06-04T04:55:43.610328+0000","last_became_active":"2024-06-04T04:53:33.340862+0000","last_became_peered":"2024-06-04T04:53:33.340862+0000","last_unstale":"2024-06-04T04:55:43.610328+0000","last_undegraded":"2024-06-04T04:55:43.610328+0000","last_fullsized":"2024-06-04T04:55:43.610328+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:26.818063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.755138+0000","last_change":"2024-06-04T04:53:36.383124+0000","last_active":"2024-06-04T04:55:17.755138+0000","last_peered":"2024-06-04T04:55:17.755138+0000","last_clean":"2024-06-04T04:55:17.755138+0000","last_became_active":"2024-06-04T04:53:36.382449+0000","last_became_peered":"2024-06-04T04:53:36.382449+0000","last_unstale":"2024-06-04T04:55:17.755138+0000","last_undegraded":"2024-06-04T04:55:17.755138+0000","last_fullsized":"2024-06-04T04:55:17.755138+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:52:40.776003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":44,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508738+0000","last_change":"2024-06-04T04:53:33.354556+0000","last_active":"2024-06-04T04:54:19.508738+0000","last_peered":"2024-06-04T04:54:19.508738+0000","last_clean":"2024-06-04T04:54:19.508738+0000","last_became_active":"2024-06-04T04:53:33.354238+0000","last_became_peered":"2024-06-04T04:53:33.354238+0000","last_unstale":"2024-06-04T04:54:19.508738+0000","last_undegraded":"2024-06-04T04:54:19.508738+0000","last_fullsized":"2024-06-04T04:54:19.508738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:54.697374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.750667+0000","last_change":"2024-06-04T04:53:36.371387+0000","last_active":"2024-06-04T04:55:17.750667+0000","last_peered":"2024-06-04T04:55:17.750667+0000","last_clean":"2024-06-04T04:55:17.750667+0000","last_became_active":"2024-06-04T04:53:36.370852+0000","last_became_peered":"2024-06-04T04:53:36.370852+0000","last_unstale":"2024-06-04T04:55:17.750667+0000","last_undegraded":"2024-06-04T04:55:17.750667+0000","last_fullsized":"2024-06-04T04:55:17.750667+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:44:06.956210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508796+0000","last_change":"2024-06-04T04:53:33.349982+0000","last_active":"2024-06-04T04:54:19.508796+0000","last_peered":"2024-06-04T04:54:19.508796+0000","last_clean":"2024-06-04T04:54:19.508796+0000","last_became_active":"2024-06-04T04:53:33.349633+0000","last_became_peered":"2024-06-04T04:53:33.349633+0000","last_unstale":"2024-06-04T04:54:19.508796+0000","last_undegraded":"2024-06-04T04:54:19.508796+0000","last_fullsized":"2024-06-04T04:54:19.508796+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:40.107426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.755048+0000","last_change":"2024-06-04T04:53:36.382313+0000","last_active":"2024-06-04T04:55:17.755048+0000","last_peered":"2024-06-04T04:55:17.755048+0000","last_clean":"2024-06-04T04:55:17.755048+0000","last_became_active":"2024-06-04T04:53:36.379638+0000","last_became_peered":"2024-06-04T04:53:36.379638+0000","last_unstale":"2024-06-04T04:55:17.755048+0000","last_undegraded":"2024-06-04T04:55:17.755048+0000","last_fullsized":"2024-06-04T04:55:17.755048+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:43:42.527262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.515392+0000","last_change":"2024-06-04T04:54:19.513581+0000","last_active":"2024-06-04T04:54:19.515392+0000","last_peered":"2024-06-04T04:54:19.515392+0000","last_clean":"2024-06-04T04:54:19.515392+0000","last_became_active":"2024-06-04T04:54:19.513111+0000","last_became_peered":"2024-06-04T04:54:19.513111+0000","last_unstale":"2024-06-04T04:54:19.515392+0000","last_undegraded":"2024-06-04T04:54:19.515392+0000","last_fullsized":"2024-06-04T04:54:19.515392+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:57.028083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.740728+0000","last_change":"2024-06-04T04:53:36.369904+0000","last_active":"2024-06-04T04:55:17.740728+0000","last_peered":"2024-06-04T04:55:17.740728+0000","last_clean":"2024-06-04T04:55:17.740728+0000","last_became_active":"2024-06-04T04:53:36.367929+0000","last_became_peered":"2024-06-04T04:53:36.367929+0000","last_unstale":"2024-06-04T04:55:17.740728+0000","last_undegraded":"2024-06-04T04:55:17.740728+0000","last_fullsized":"2024-06-04T04:55:17.740728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:37:08.721569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"81'7","reported_seq":147,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:42.746577+0000","last_change":"2024-06-04T04:53:33.342723+0000","last_active":"2024-06-04T04:55:42.746577+0000","last_peered":"2024-06-04T04:55:42.746577+0000","last_clean":"2024-06-04T04:55:42.746577+0000","last_became_active":"2024-06-04T04:53:33.342344+0000","last_became_peered":"2024-06-04T04:53:33.342344+0000","last_unstale":"2024-06-04T04:55:42.746577+0000","last_undegraded":"2024-06-04T04:55:42.746577+0000","last_fullsized":"2024-06-04T04:55:42.746577+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:59:50.147517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.745755+0000","last_change":"2024-06-04T04:54:19.513534+0000","last_active":"2024-06-04T04:55:17.745755+0000","last_peered":"2024-06-04T04:55:17.745755+0000","last_clean":"2024-06-04T04:55:17.745755+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:55:17.745755+0000","last_undegraded":"2024-06-04T04:55:17.745755+0000","last_fullsized":"2024-06-04T04:55:17.745755+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:16:32.240932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.741074+0000","last_change":"2024-06-04T04:53:36.371236+0000","last_active":"2024-06-04T04:55:17.741074+0000","last_peered":"2024-06-04T04:55:17.741074+0000","last_clean":"2024-06-04T04:55:17.741074+0000","last_became_active":"2024-06-04T04:53:36.370876+0000","last_became_peered":"2024-06-04T04:53:36.370876+0000","last_unstale":"2024-06-04T04:55:17.741074+0000","last_undegraded":"2024-06-04T04:55:17.741074+0000","last_fullsized":"2024-06-04T04:55:17.741074+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:15:56.751317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":143,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:43.609770+0000","last_change":"2024-06-04T04:53:33.355642+0000","last_active":"2024-06-04T04:55:43.609770+0000","last_peered":"2024-06-04T04:55:43.609770+0000","last_clean":"2024-06-04T04:55:43.609770+0000","last_became_active":"2024-06-04T04:53:33.355187+0000","last_became_peered":"2024-06-04T04:53:33.355187+0000","last_unstale":"2024-06-04T04:55:43.609770+0000","last_undegraded":"2024-06-04T04:55:43.609770+0000","last_fullsized":"2024-06-04T04:55:43.609770+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:32:28.331518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.754076+0000","last_change":"2024-06-04T04:54:19.631750+0000","last_active":"2024-06-04T04:55:17.754076+0000","last_peered":"2024-06-04T04:55:17.754076+0000","last_clean":"2024-06-04T04:55:17.754076+0000","last_became_active":"2024-06-04T04:54:19.631485+0000","last_became_peered":"2024-06-04T04:54:19.631485+0000","last_unstale":"2024-06-04T04:55:17.754076+0000","last_undegraded":"2024-06-04T04:55:17.754076+0000","last_fullsized":"2024-06-04T04:55:17.754076+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:29:44.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631560+0000","last_change":"2024-06-04T04:53:33.340653+0000","last_active":"2024-06-04T04:54:19.631560+0000","last_peered":"2024-06-04T04:54:19.631560+0000","last_clean":"2024-06-04T04:54:19.631560+0000","last_became_active":"2024-06-04T04:53:33.340250+0000","last_became_peered":"2024-06-04T04:53:33.340250+0000","last_unstale":"2024-06-04T04:54:19.631560+0000","last_undegraded":"2024-06-04T04:54:19.631560+0000","last_fullsized":"2024-06-04T04:54:19.631560+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:46:20.329514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":100,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510709+0000","last_change":"2024-06-04T04:52:09.181155+0000","last_active":"2024-06-04T04:54:19.510709+0000","last_peered":"2024-06-04T04:54:19.510709+0000","last_clean":"2024-06-04T04:54:19.510709+0000","last_became_active":"2024-06-04T04:52:09.177002+0000","last_became_peered":"2024-06-04T04:52:09.177002+0000","last_unstale":"2024-06-04T04:54:19.510709+0000","last_undegraded":"2024-06-04T04:54:19.510709+0000","last_fullsized":"2024-06-04T04:54:19.510709+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_clean_scrub_stamp":"2024-06-04T04:48:44.631948+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:46:57.735302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.755531+0000","last_change":"2024-06-04T04:53:36.372634+0000","last_active":"2024-06-04T04:55:17.755531+0000","last_peered":"2024-06-04T04:55:17.755531+0000","last_clean":"2024-06-04T04:55:17.755531+0000","last_became_active":"2024-06-04T04:53:36.372290+0000","last_became_peered":"2024-06-04T04:53:36.372290+0000","last_unstale":"2024-06-04T04:55:17.755531+0000","last_undegraded":"2024-06-04T04:55:17.755531+0000","last_fullsized":"2024-06-04T04:55:17.755531+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:14:07.460382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511064+0000","last_change":"2024-06-04T04:53:33.348760+0000","last_active":"2024-06-04T04:54:19.511064+0000","last_peered":"2024-06-04T04:54:19.511064+0000","last_clean":"2024-06-04T04:54:19.511064+0000","last_became_active":"2024-06-04T04:53:33.348347+0000","last_became_peered":"2024-06-04T04:53:33.348347+0000","last_unstale":"2024-06-04T04:54:19.511064+0000","last_undegraded":"2024-06-04T04:54:19.511064+0000","last_fullsized":"2024-06-04T04:54:19.511064+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:36:20.641061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509608+0000","last_change":"2024-06-04T04:53:33.354840+0000","last_active":"2024-06-04T04:54:19.509608+0000","last_peered":"2024-06-04T04:54:19.509608+0000","last_clean":"2024-06-04T04:54:19.509608+0000","last_became_active":"2024-06-04T04:53:33.354287+0000","last_became_peered":"2024-06-04T04:53:33.354287+0000","last_unstale":"2024-06-04T04:54:19.509608+0000","last_undegraded":"2024-06-04T04:54:19.509608+0000","last_fullsized":"2024-06-04T04:54:19.509608+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:42.550098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.755774+0000","last_change":"2024-06-04T04:53:36.372678+0000","last_active":"2024-06-04T04:55:17.755774+0000","last_peered":"2024-06-04T04:55:17.755774+0000","last_clean":"2024-06-04T04:55:17.755774+0000","last_became_active":"2024-06-04T04:53:36.372312+0000","last_became_peered":"2024-06-04T04:53:36.372312+0000","last_unstale":"2024-06-04T04:55:17.755774+0000","last_undegraded":"2024-06-04T04:55:17.755774+0000","last_fullsized":"2024-06-04T04:55:17.755774+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:34:41.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'15","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:18.386842+0000","last_change":"2024-06-04T04:53:33.348440+0000","last_active":"2024-06-04T04:55:18.386842+0000","last_peered":"2024-06-04T04:55:18.386842+0000","last_clean":"2024-06-04T04:55:18.386842+0000","last_became_active":"2024-06-04T04:53:33.348131+0000","last_became_peered":"2024-06-04T04:53:33.348131+0000","last_unstale":"2024-06-04T04:55:18.386842+0000","last_undegraded":"2024-06-04T04:55:18.386842+0000","last_fullsized":"2024-06-04T04:55:18.386842+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:14:11.552286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.759679+0000","last_change":"2024-06-04T04:53:36.378669+0000","last_active":"2024-06-04T04:55:17.759679+0000","last_peered":"2024-06-04T04:55:17.759679+0000","last_clean":"2024-06-04T04:55:17.759679+0000","last_became_active":"2024-06-04T04:53:36.378233+0000","last_became_peered":"2024-06-04T04:53:36.378233+0000","last_unstale":"2024-06-04T04:55:17.759679+0000","last_undegraded":"2024-06-04T04:55:17.759679+0000","last_fullsized":"2024-06-04T04:55:17.759679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:36:45.870924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631707+0000","last_change":"2024-06-04T04:53:33.345078+0000","last_active":"2024-06-04T04:54:19.631707+0000","last_peered":"2024-06-04T04:54:19.631707+0000","last_clean":"2024-06-04T04:54:19.631707+0000","last_became_active":"2024-06-04T04:53:33.344708+0000","last_became_peered":"2024-06-04T04:53:33.344708+0000","last_unstale":"2024-06-04T04:54:19.631707+0000","last_undegraded":"2024-06-04T04:54:19.631707+0000","last_fullsized":"2024-06-04T04:54:19.631707+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:45:14.010758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.757819+0000","last_change":"2024-06-04T04:53:36.364817+0000","last_active":"2024-06-04T04:55:17.757819+0000","last_peered":"2024-06-04T04:55:17.757819+0000","last_clean":"2024-06-04T04:55:17.757819+0000","last_became_active":"2024-06-04T04:53:36.364499+0000","last_became_peered":"2024-06-04T04:53:36.364499+0000","last_unstale":"2024-06-04T04:55:17.757819+0000","last_undegraded":"2024-06-04T04:55:17.757819+0000","last_fullsized":"2024-06-04T04:55:17.757819+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:39:30.518028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.630933+0000","last_change":"2024-06-04T04:53:33.354527+0000","last_active":"2024-06-04T04:54:19.630933+0000","last_peered":"2024-06-04T04:54:19.630933+0000","last_clean":"2024-06-04T04:54:19.630933+0000","last_became_active":"2024-06-04T04:53:33.354175+0000","last_became_peered":"2024-06-04T04:53:33.354175+0000","last_unstale":"2024-06-04T04:54:19.630933+0000","last_undegraded":"2024-06-04T04:54:19.630933+0000","last_fullsized":"2024-06-04T04:54:19.630933+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:33:08.125428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.708672+0000","last_change":"2024-06-04T04:53:36.373248+0000","last_active":"2024-06-04T04:55:17.708672+0000","last_peered":"2024-06-04T04:55:17.708672+0000","last_clean":"2024-06-04T04:55:17.708672+0000","last_became_active":"2024-06-04T04:53:36.372915+0000","last_became_peered":"2024-06-04T04:53:36.372915+0000","last_unstale":"2024-06-04T04:55:17.708672+0000","last_undegraded":"2024-06-04T04:55:17.708672+0000","last_fullsized":"2024-06-04T04:55:17.708672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:23:11.689349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512461+0000","last_change":"2024-06-04T04:53:33.340427+0000","last_active":"2024-06-04T04:54:19.512461+0000","last_peered":"2024-06-04T04:54:19.512461+0000","last_clean":"2024-06-04T04:54:19.512461+0000","last_became_active":"2024-06-04T04:53:33.340022+0000","last_became_peered":"2024-06-04T04:53:33.340022+0000","last_unstale":"2024-06-04T04:54:19.512461+0000","last_undegraded":"2024-06-04T04:54:19.512461+0000","last_fullsized":"2024-06-04T04:54:19.512461+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:56:16.496471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.759040+0000","last_change":"2024-06-04T04:53:36.380114+0000","last_active":"2024-06-04T04:55:17.759040+0000","last_peered":"2024-06-04T04:55:17.759040+0000","last_clean":"2024-06-04T04:55:17.759040+0000","last_became_active":"2024-06-04T04:53:36.379392+0000","last_became_peered":"2024-06-04T04:53:36.379392+0000","last_unstale":"2024-06-04T04:55:17.759040+0000","last_undegraded":"2024-06-04T04:55:17.759040+0000","last_fullsized":"2024-06-04T04:55:17.759040+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:41:48.776439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.520195+0000","last_change":"2024-06-04T04:54:19.519755+0000","last_active":"2024-06-04T04:54:19.520195+0000","last_peered":"2024-06-04T04:54:19.520195+0000","last_clean":"2024-06-04T04:54:19.520195+0000","last_became_active":"2024-06-04T04:54:19.513992+0000","last_became_peered":"2024-06-04T04:54:19.513992+0000","last_unstale":"2024-06-04T04:54:19.520195+0000","last_undegraded":"2024-06-04T04:54:19.520195+0000","last_fullsized":"2024-06-04T04:54:19.520195+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:01.951935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.753820+0000","last_change":"2024-06-04T04:53:36.377420+0000","last_active":"2024-06-04T04:55:17.753820+0000","last_peered":"2024-06-04T04:55:17.753820+0000","last_clean":"2024-06-04T04:55:17.753820+0000","last_became_active":"2024-06-04T04:53:36.377109+0000","last_became_peered":"2024-06-04T04:53:36.377109+0000","last_unstale":"2024-06-04T04:55:17.753820+0000","last_undegraded":"2024-06-04T04:55:17.753820+0000","last_fullsized":"2024-06-04T04:55:17.753820+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:47:12.102396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":43,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.645328+0000","last_change":"2024-06-04T04:54:19.645328+0000","last_active":"2024-06-04T04:54:19.645328+0000","last_peered":"2024-06-04T04:54:19.645328+0000","last_clean":"2024-06-04T04:54:19.645328+0000","last_became_active":"2024-06-04T04:54:19.633416+0000","last_became_peered":"2024-06-04T04:54:19.633416+0000","last_unstale":"2024-06-04T04:54:19.645328+0000","last_undegraded":"2024-06-04T04:54:19.645328+0000","last_fullsized":"2024-06-04T04:54:19.645328+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:04:20.494641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.759725+0000","last_change":"2024-06-04T04:53:36.381187+0000","last_active":"2024-06-04T04:55:17.759725+0000","last_peered":"2024-06-04T04:55:17.759725+0000","last_clean":"2024-06-04T04:55:17.759725+0000","last_became_active":"2024-06-04T04:53:36.380591+0000","last_became_peered":"2024-06-04T04:53:36.380591+0000","last_unstale":"2024-06-04T04:55:17.759725+0000","last_undegraded":"2024-06-04T04:55:17.759725+0000","last_fullsized":"2024-06-04T04:55:17.759725+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:26:43.122339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'5","reported_seq":142,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:42.649365+0000","last_change":"2024-06-04T04:53:33.355908+0000","last_active":"2024-06-04T04:55:42.649365+0000","last_peered":"2024-06-04T04:55:42.649365+0000","last_clean":"2024-06-04T04:55:42.649365+0000","last_became_active":"2024-06-04T04:53:33.355522+0000","last_became_peered":"2024-06-04T04:53:33.355522+0000","last_unstale":"2024-06-04T04:55:42.649365+0000","last_undegraded":"2024-06-04T04:55:42.649365+0000","last_fullsized":"2024-06-04T04:55:42.649365+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:16:44.639431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.759847+0000","last_change":"2024-06-04T04:53:36.382737+0000","last_active":"2024-06-04T04:55:17.759847+0000","last_peered":"2024-06-04T04:55:17.759847+0000","last_clean":"2024-06-04T04:55:17.759847+0000","last_became_active":"2024-06-04T04:53:36.382353+0000","last_became_peered":"2024-06-04T04:53:36.382353+0000","last_unstale":"2024-06-04T04:55:17.759847+0000","last_undegraded":"2024-06-04T04:55:17.759847+0000","last_fullsized":"2024-06-04T04:55:17.759847+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:18:01.476058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512086+0000","last_change":"2024-06-04T04:53:33.347273+0000","last_active":"2024-06-04T04:54:19.512086+0000","last_peered":"2024-06-04T04:54:19.512086+0000","last_clean":"2024-06-04T04:54:19.512086+0000","last_became_active":"2024-06-04T04:53:33.346915+0000","last_became_peered":"2024-06-04T04:53:33.346915+0000","last_unstale":"2024-06-04T04:54:19.512086+0000","last_undegraded":"2024-06-04T04:54:19.512086+0000","last_fullsized":"2024-06-04T04:54:19.512086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:17.235484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.759127+0000","last_change":"2024-06-04T04:53:36.383454+0000","last_active":"2024-06-04T04:55:17.759127+0000","last_peered":"2024-06-04T04:55:17.759127+0000","last_clean":"2024-06-04T04:55:17.759127+0000","last_became_active":"2024-06-04T04:53:36.383105+0000","last_became_peered":"2024-06-04T04:53:36.383105+0000","last_unstale":"2024-06-04T04:55:17.759127+0000","last_undegraded":"2024-06-04T04:55:17.759127+0000","last_fullsized":"2024-06-04T04:55:17.759127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:04:02.147688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505624+0000","last_change":"2024-06-04T04:53:33.345435+0000","last_active":"2024-06-04T04:54:19.505624+0000","last_peered":"2024-06-04T04:54:19.505624+0000","last_clean":"2024-06-04T04:54:19.505624+0000","last_became_active":"2024-06-04T04:53:33.345077+0000","last_became_peered":"2024-06-04T04:53:33.345077+0000","last_unstale":"2024-06-04T04:54:19.505624+0000","last_undegraded":"2024-06-04T04:54:19.505624+0000","last_fullsized":"2024-06-04T04:54:19.505624+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:20:13.992259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.760007+0000","last_change":"2024-06-04T04:53:36.385238+0000","last_active":"2024-06-04T04:55:17.760007+0000","last_peered":"2024-06-04T04:55:17.760007+0000","last_clean":"2024-06-04T04:55:17.760007+0000","last_became_active":"2024-06-04T04:53:36.384899+0000","last_became_peered":"2024-06-04T04:53:36.384899+0000","last_unstale":"2024-06-04T04:55:17.760007+0000","last_undegraded":"2024-06-04T04:55:17.760007+0000","last_fullsized":"2024-06-04T04:55:17.760007+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:09:42.629340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.751843+0000","last_change":"2024-06-04T04:53:36.383450+0000","last_active":"2024-06-04T04:55:17.751843+0000","last_peered":"2024-06-04T04:55:17.751843+0000","last_clean":"2024-06-04T04:55:17.751843+0000","last_became_active":"2024-06-04T04:53:36.382678+0000","last_became_peered":"2024-06-04T04:53:36.382678+0000","last_unstale":"2024-06-04T04:55:17.751843+0000","last_undegraded":"2024-06-04T04:55:17.751843+0000","last_fullsized":"2024-06-04T04:55:17.751843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:36:28.932636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.752940+0000","last_change":"2024-06-04T04:54:19.632563+0000","last_active":"2024-06-04T04:55:17.752940+0000","last_peered":"2024-06-04T04:55:17.752940+0000","last_clean":"2024-06-04T04:55:17.752940+0000","last_became_active":"2024-06-04T04:54:19.632070+0000","last_became_peered":"2024-06-04T04:54:19.632070+0000","last_unstale":"2024-06-04T04:55:17.752940+0000","last_undegraded":"2024-06-04T04:55:17.752940+0000","last_fullsized":"2024-06-04T04:55:17.752940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:07:16.552958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758655+0000","last_change":"2024-06-04T04:53:36.381408+0000","last_active":"2024-06-04T04:55:17.758655+0000","last_peered":"2024-06-04T04:55:17.758655+0000","last_clean":"2024-06-04T04:55:17.758655+0000","last_became_active":"2024-06-04T04:53:36.381078+0000","last_became_peered":"2024-06-04T04:53:36.381078+0000","last_unstale":"2024-06-04T04:55:17.758655+0000","last_undegraded":"2024-06-04T04:55:17.758655+0000","last_fullsized":"2024-06-04T04:55:17.758655+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:28:48.988591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758212+0000","last_change":"2024-06-04T04:53:36.360168+0000","last_active":"2024-06-04T04:55:17.758212+0000","last_peered":"2024-06-04T04:55:17.758212+0000","last_clean":"2024-06-04T04:55:17.758212+0000","last_became_active":"2024-06-04T04:53:36.359691+0000","last_became_peered":"2024-06-04T04:53:36.359691+0000","last_unstale":"2024-06-04T04:55:17.758212+0000","last_undegraded":"2024-06-04T04:55:17.758212+0000","last_fullsized":"2024-06-04T04:55:17.758212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:55:17.418339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758241+0000","last_change":"2024-06-04T04:53:36.363018+0000","last_active":"2024-06-04T04:55:17.758241+0000","last_peered":"2024-06-04T04:55:17.758241+0000","last_clean":"2024-06-04T04:55:17.758241+0000","last_became_active":"2024-06-04T04:53:36.361595+0000","last_became_peered":"2024-06-04T04:53:36.361595+0000","last_unstale":"2024-06-04T04:55:17.758241+0000","last_undegraded":"2024-06-04T04:55:17.758241+0000","last_fullsized":"2024-06-04T04:55:17.758241+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:00:21.377322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.751679+0000","last_change":"2024-06-04T04:53:36.375377+0000","last_active":"2024-06-04T04:55:17.751679+0000","last_peered":"2024-06-04T04:55:17.751679+0000","last_clean":"2024-06-04T04:55:17.751679+0000","last_became_active":"2024-06-04T04:53:36.373459+0000","last_became_peered":"2024-06-04T04:53:36.373459+0000","last_unstale":"2024-06-04T04:55:17.751679+0000","last_undegraded":"2024-06-04T04:55:17.751679+0000","last_fullsized":"2024-06-04T04:55:17.751679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:51:31.686969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.744737+0000","last_change":"2024-06-04T04:53:36.373850+0000","last_active":"2024-06-04T04:55:17.744737+0000","last_peered":"2024-06-04T04:55:17.744737+0000","last_clean":"2024-06-04T04:55:17.744737+0000","last_became_active":"2024-06-04T04:53:36.371951+0000","last_became_peered":"2024-06-04T04:53:36.371951+0000","last_unstale":"2024-06-04T04:55:17.744737+0000","last_undegraded":"2024-06-04T04:55:17.744737+0000","last_fullsized":"2024-06-04T04:55:17.744737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:06.452703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.743863+0000","last_change":"2024-06-04T04:53:36.374871+0000","last_active":"2024-06-04T04:55:17.743863+0000","last_peered":"2024-06-04T04:55:17.743863+0000","last_clean":"2024-06-04T04:55:17.743863+0000","last_became_active":"2024-06-04T04:53:36.374486+0000","last_became_peered":"2024-06-04T04:53:36.374486+0000","last_unstale":"2024-06-04T04:55:17.743863+0000","last_undegraded":"2024-06-04T04:55:17.743863+0000","last_fullsized":"2024-06-04T04:55:17.743863+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:15:13.337374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.757968+0000","last_change":"2024-06-04T04:54:19.513540+0000","last_active":"2024-06-04T04:55:17.757968+0000","last_peered":"2024-06-04T04:55:17.757968+0000","last_clean":"2024-06-04T04:55:17.757968+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:55:17.757968+0000","last_undegraded":"2024-06-04T04:55:17.757968+0000","last_fullsized":"2024-06-04T04:55:17.757968+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:09:27.993585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.744742+0000","last_change":"2024-06-04T04:53:36.373055+0000","last_active":"2024-06-04T04:55:17.744742+0000","last_peered":"2024-06-04T04:55:17.744742+0000","last_clean":"2024-06-04T04:55:17.744742+0000","last_became_active":"2024-06-04T04:53:36.372754+0000","last_became_peered":"2024-06-04T04:53:36.372754+0000","last_unstale":"2024-06-04T04:55:17.744742+0000","last_undegraded":"2024-06-04T04:55:17.744742+0000","last_fullsized":"2024-06-04T04:55:17.744742+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:22:27.554200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758984+0000","last_change":"2024-06-04T04:53:36.378897+0000","last_active":"2024-06-04T04:55:17.758984+0000","last_peered":"2024-06-04T04:55:17.758984+0000","last_clean":"2024-06-04T04:55:17.758984+0000","last_became_active":"2024-06-04T04:53:36.378607+0000","last_became_peered":"2024-06-04T04:53:36.378607+0000","last_unstale":"2024-06-04T04:55:17.758984+0000","last_undegraded":"2024-06-04T04:55:17.758984+0000","last_fullsized":"2024-06-04T04:55:17.758984+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:03.395491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.756938+0000","last_change":"2024-06-04T04:53:36.378240+0000","last_active":"2024-06-04T04:55:17.756938+0000","last_peered":"2024-06-04T04:55:17.756938+0000","last_clean":"2024-06-04T04:55:17.756938+0000","last_became_active":"2024-06-04T04:53:36.377909+0000","last_became_peered":"2024-06-04T04:53:36.377909+0000","last_unstale":"2024-06-04T04:55:17.756938+0000","last_undegraded":"2024-06-04T04:55:17.756938+0000","last_fullsized":"2024-06-04T04:55:17.756938+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:15:25.288340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.754518+0000","last_change":"2024-06-04T04:53:36.379792+0000","last_active":"2024-06-04T04:55:17.754518+0000","last_peered":"2024-06-04T04:55:17.754518+0000","last_clean":"2024-06-04T04:55:17.754518+0000","last_became_active":"2024-06-04T04:53:36.379438+0000","last_became_peered":"2024-06-04T04:53:36.379438+0000","last_unstale":"2024-06-04T04:55:17.754518+0000","last_undegraded":"2024-06-04T04:55:17.754518+0000","last_fullsized":"2024-06-04T04:55:17.754518+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:23:57.617877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":214930,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73406,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1131,"num_read_kb":641,"num_write":252,"num_write_kb":147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":220218,"data_compressed":5499,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":217,"ondisk_log_size":217,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678058,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":880640,"data_stored":429560,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874286,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":1087041,"data_compressed":43684,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037812,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":429490,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234041,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":429196,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430269,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":1022527,"data_compressed":41851,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593796,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":429739,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":497776,"data_compressed":7592,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":909312,"data_stored":430546,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":496792,"data_compressed":7592,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36220,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708932,"statfs":{"total":95995035648,"available":95957946368,"internally_reserved":0,"allocated":905216,"data_stored":429032,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953550,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32044,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713108,"statfs":{"total":95995035648,"available":95962222592,"internally_reserved":0,"allocated":888832,"data_stored":429068,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790025,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":1021673,"data_compressed":41851,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34470,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":69408,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":68424,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":68188,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-04T04:55:47.793 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-04T04:55:48.032 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:48.506 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:48 smithi070 ceph-mon[32631]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:55:48.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:48 smithi112 ceph-mon[36453]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:55:48.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:48 smithi098 ceph-mon[35569]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:55:49.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:49 smithi070 ceph-mon[32631]: pgmap v356: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-06-04T04:55:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:49 smithi112 ceph-mon[36453]: pgmap v356: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-06-04T04:55:49.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.731+0000 7fd2fc14a700 1 -- 172.21.15.70:0/2966784787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f4102df0 msgr2=0x7fd2f410f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:49.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.731+0000 7fd2fc14a700 1 --2- 172.21.15.70:0/2966784787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f4102df0 0x7fd2f410f2e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00b0a0 tx=0x7fd2ec00b3b0 comp rx=0 tx=0).stop 2024-06-04T04:55:49.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.731+0000 7fd2fc14a700 1 -- 172.21.15.70:0/2966784787 shutdown_connections 2024-06-04T04:55:49.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.731+0000 7fd2fc14a700 1 --2- 172.21.15.70:0/2966784787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f4102df0 0x7fd2f410f2e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:49.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.731+0000 7fd2fc14a700 1 --2- 172.21.15.70:0/2966784787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2f4108510 0x7fd2f41088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:49.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.731+0000 7fd2fc14a700 1 --2- 172.21.15.70:0/2966784787 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2f4102430 0x7fd2f41028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:49.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.731+0000 7fd2fc14a700 1 -- 172.21.15.70:0/2966784787 >> 172.21.15.70:0/2966784787 conn(0x7fd2f40793b0 msgr2=0x7fd2f40797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:49.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.732+0000 7fd2fc14a700 1 -- 172.21.15.70:0/2966784787 shutdown_connections 2024-06-04T04:55:49.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.732+0000 7fd2fc14a700 1 -- 172.21.15.70:0/2966784787 wait complete. 2024-06-04T04:55:49.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.732+0000 7fd2fc14a700 1 Processor -- start 2024-06-04T04:55:49.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.732+0000 7fd2fc14a700 1 -- start start 2024-06-04T04:55:49.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.732+0000 7fd2fc14a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2f4102430 0x7fd2f419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:49.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.732+0000 7fd2fc14a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f4102df0 0x7fd2f419ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:49.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.732+0000 7fd2fc14a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2f4108510 0x7fd2f4198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:49.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.732+0000 7fd2fc14a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f41124f0 con 0x7fd2f4102df0 2024-06-04T04:55:49.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.732+0000 7fd2fc14a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f4112370 con 0x7fd2f4108510 2024-06-04T04:55:49.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.732+0000 7fd2fc14a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f4112670 con 0x7fd2f4102430 2024-06-04T04:55:49.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.733+0000 7fd2f96e5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f4102df0 0x7fd2f419ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:49.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.733+0000 7fd2f96e5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f4102df0 0x7fd2f419ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34496/0 (socket says 172.21.15.70:34496) 2024-06-04T04:55:49.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.733+0000 7fd2f96e5700 1 -- 172.21.15.70:0/662294277 learned_addr learned my addr 172.21.15.70:0/662294277 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:49.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.733+0000 7fd2f9ee6700 1 --2- 172.21.15.70:0/662294277 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2f4102430 0x7fd2f419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:49.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.733+0000 7fd2fa6e7700 1 --2- 172.21.15.70:0/662294277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2f4108510 0x7fd2f4198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:49.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.733+0000 7fd2f96e5700 1 -- 172.21.15.70:0/662294277 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2f4102430 msgr2=0x7fd2f419e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:49.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.733+0000 7fd2f96e5700 1 --2- 172.21.15.70:0/662294277 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2f4102430 0x7fd2f419e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:49.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.733+0000 7fd2f96e5700 1 -- 172.21.15.70:0/662294277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2f4108510 msgr2=0x7fd2f4198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:49.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.733+0000 7fd2f96e5700 1 --2- 172.21.15.70:0/662294277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2f4108510 0x7fd2f4198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:49.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.733+0000 7fd2f96e5700 1 -- 172.21.15.70:0/662294277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f4198ff0 con 0x7fd2f4102df0 2024-06-04T04:55:49.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.733+0000 7fd2f96e5700 1 --2- 172.21.15.70:0/662294277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f4102df0 0x7fd2f419ebd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd2f000d950 tx=0x7fd2f000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:49.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.733+0000 7fd2eaffd700 1 -- 172.21.15.70:0/662294277 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0016070 con 0x7fd2f4102df0 2024-06-04T04:55:49.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.733+0000 7fd2eaffd700 1 -- 172.21.15.70:0/662294277 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd2f000fd20 con 0x7fd2f4102df0 2024-06-04T04:55:49.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.733+0000 7fd2fc14a700 1 -- 172.21.15.70:0/662294277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2f4199280 con 0x7fd2f4102df0 2024-06-04T04:55:49.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.733+0000 7fd2eaffd700 1 -- 172.21.15.70:0/662294277 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f000e3b0 con 0x7fd2f4102df0 2024-06-04T04:55:49.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.733+0000 7fd2fc14a700 1 -- 172.21.15.70:0/662294277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2f406bab0 con 0x7fd2f4102df0 2024-06-04T04:55:49.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.734+0000 7fd2fc14a700 1 -- 172.21.15.70:0/662294277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2f4103b50 con 0x7fd2f4102df0 2024-06-04T04:55:49.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.735+0000 7fd2eaffd700 1 -- 172.21.15.70:0/662294277 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fd2f0019070 con 0x7fd2f4102df0 2024-06-04T04:55:49.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.736+0000 7fd2eaffd700 1 --2- 172.21.15.70:0/662294277 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd2e00b2900 0x7fd2e00b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:49.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.736+0000 7fd2eaffd700 1 -- 172.21.15.70:0/662294277 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fd2f00e20d0 con 0x7fd2f4102df0 2024-06-04T04:55:49.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.736+0000 7fd2f9ee6700 1 --2- 172.21.15.70:0/662294277 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd2e00b2900 0x7fd2e00b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:49.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.737+0000 7fd2f9ee6700 1 --2- 172.21.15.70:0/662294277 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd2e00b2900 0x7fd2e00b4dc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4009a10 tx=0x7fd2e4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:49.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.740+0000 7fd2eaffd700 1 -- 172.21.15.70:0/662294277 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd2f00a9dd0 con 0x7fd2f4102df0 2024-06-04T04:55:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:49 smithi098 ceph-mon[35569]: pgmap v356: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-06-04T04:55:49.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.941+0000 7fd2fc14a700 1 -- 172.21.15.70:0/662294277 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd2f406c6a0 con 0x7fd2e00b2900 2024-06-04T04:55:49.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.948+0000 7fd2eaffd700 1 -- 172.21.15.70:0/662294277 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259123 (secure 0 0 0) 0x7fd2f406c6a0 con 0x7fd2e00b2900 2024-06-04T04:55:49.955 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:55:49.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.952+0000 7fd2fc14a700 1 -- 172.21.15.70:0/662294277 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd2e00b2900 msgr2=0x7fd2e00b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:49.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.952+0000 7fd2fc14a700 1 --2- 172.21.15.70:0/662294277 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd2e00b2900 0x7fd2e00b4dc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4009a10 tx=0x7fd2e4009190 comp rx=0 tx=0).stop 2024-06-04T04:55:49.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.952+0000 7fd2fc14a700 1 -- 172.21.15.70:0/662294277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f4102df0 msgr2=0x7fd2f419ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:49.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.952+0000 7fd2fc14a700 1 --2- 172.21.15.70:0/662294277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f4102df0 0x7fd2f419ebd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd2f000d950 tx=0x7fd2f000dc60 comp rx=0 tx=0).stop 2024-06-04T04:55:49.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.952+0000 7fd2fc14a700 1 -- 172.21.15.70:0/662294277 shutdown_connections 2024-06-04T04:55:49.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.952+0000 7fd2fc14a700 1 --2- 172.21.15.70:0/662294277 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd2e00b2900 0x7fd2e00b4dc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:49.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.952+0000 7fd2fc14a700 1 --2- 172.21.15.70:0/662294277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2f4108510 0x7fd2f4198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:49.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.952+0000 7fd2fc14a700 1 --2- 172.21.15.70:0/662294277 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2f4102430 0x7fd2f419e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:49.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.952+0000 7fd2fc14a700 1 --2- 172.21.15.70:0/662294277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f4102df0 0x7fd2f419ebd0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:49.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.952+0000 7fd2fc14a700 1 -- 172.21.15.70:0/662294277 >> 172.21.15.70:0/662294277 conn(0x7fd2f40793b0 msgr2=0x7fd2f4100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:49.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.952+0000 7fd2fc14a700 1 -- 172.21.15.70:0/662294277 shutdown_connections 2024-06-04T04:55:49.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:49.952+0000 7fd2fc14a700 1 -- 172.21.15.70:0/662294277 wait complete. 2024-06-04T04:55:49.957 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-06-04T04:55:50.673 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-06-04T04:55:49.754151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664008,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1257,"num_read_kb":730,"num_write":377,"num_write_kb":2299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2952512,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":811,"ondisk_log_size":811,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":355900,"kb_used_data":10908,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585924,"statfs":{"total":1151940427776,"available":1151575986176,"internally_reserved":0,"allocated":11169792,"data_stored":7132440,"data_compressed":182883,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19138,"internal_metadata":353154366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":123,"num_read_kb":54,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002906"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.759859+0000","last_change":"2024-06-04T04:53:36.383755+0000","last_active":"2024-06-04T04:55:17.759859+0000","last_peered":"2024-06-04T04:55:17.759859+0000","last_clean":"2024-06-04T04:55:17.759859+0000","last_became_active":"2024-06-04T04:53:36.378274+0000","last_became_peered":"2024-06-04T04:53:36.378274+0000","last_unstale":"2024-06-04T04:55:17.759859+0000","last_undegraded":"2024-06-04T04:55:17.759859+0000","last_fullsized":"2024-06-04T04:55:17.759859+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:30:19.685994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758036+0000","last_change":"2024-06-04T04:53:36.372436+0000","last_active":"2024-06-04T04:55:17.758036+0000","last_peered":"2024-06-04T04:55:17.758036+0000","last_clean":"2024-06-04T04:55:17.758036+0000","last_became_active":"2024-06-04T04:53:36.372100+0000","last_became_peered":"2024-06-04T04:53:36.372100+0000","last_unstale":"2024-06-04T04:55:17.758036+0000","last_undegraded":"2024-06-04T04:55:17.758036+0000","last_fullsized":"2024-06-04T04:55:17.758036+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:42:31.646754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758260+0000","last_change":"2024-06-04T04:53:36.366915+0000","last_active":"2024-06-04T04:55:17.758260+0000","last_peered":"2024-06-04T04:55:17.758260+0000","last_clean":"2024-06-04T04:55:17.758260+0000","last_became_active":"2024-06-04T04:53:36.366611+0000","last_became_peered":"2024-06-04T04:53:36.366611+0000","last_unstale":"2024-06-04T04:55:17.758260+0000","last_undegraded":"2024-06-04T04:55:17.758260+0000","last_fullsized":"2024-06-04T04:55:17.758260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:17:48.775938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.747879+0000","last_change":"2024-06-04T04:53:36.374486+0000","last_active":"2024-06-04T04:55:17.747879+0000","last_peered":"2024-06-04T04:55:17.747879+0000","last_clean":"2024-06-04T04:55:17.747879+0000","last_became_active":"2024-06-04T04:53:36.374178+0000","last_became_peered":"2024-06-04T04:53:36.374178+0000","last_unstale":"2024-06-04T04:55:17.747879+0000","last_undegraded":"2024-06-04T04:55:17.747879+0000","last_fullsized":"2024-06-04T04:55:17.747879+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:30:28.287070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.748587+0000","last_change":"2024-06-04T04:53:36.376183+0000","last_active":"2024-06-04T04:55:17.748587+0000","last_peered":"2024-06-04T04:55:17.748587+0000","last_clean":"2024-06-04T04:55:17.748587+0000","last_became_active":"2024-06-04T04:53:36.375790+0000","last_became_peered":"2024-06-04T04:53:36.375790+0000","last_unstale":"2024-06-04T04:55:17.748587+0000","last_undegraded":"2024-06-04T04:55:17.748587+0000","last_fullsized":"2024-06-04T04:55:17.748587+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:58.697440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.744541+0000","last_change":"2024-06-04T04:53:36.370249+0000","last_active":"2024-06-04T04:55:17.744541+0000","last_peered":"2024-06-04T04:55:17.744541+0000","last_clean":"2024-06-04T04:55:17.744541+0000","last_became_active":"2024-06-04T04:53:36.369916+0000","last_became_peered":"2024-06-04T04:53:36.369916+0000","last_unstale":"2024-06-04T04:55:17.744541+0000","last_undegraded":"2024-06-04T04:55:17.744541+0000","last_fullsized":"2024-06-04T04:55:17.744541+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:02:41.739806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.755306+0000","last_change":"2024-06-04T04:53:36.384329+0000","last_active":"2024-06-04T04:55:17.755306+0000","last_peered":"2024-06-04T04:55:17.755306+0000","last_clean":"2024-06-04T04:55:17.755306+0000","last_became_active":"2024-06-04T04:53:36.382504+0000","last_became_peered":"2024-06-04T04:53:36.382504+0000","last_unstale":"2024-06-04T04:55:17.755306+0000","last_undegraded":"2024-06-04T04:55:17.755306+0000","last_fullsized":"2024-06-04T04:55:17.755306+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:51:07.604691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.745731+0000","last_change":"2024-06-04T04:53:36.372319+0000","last_active":"2024-06-04T04:55:17.745731+0000","last_peered":"2024-06-04T04:55:17.745731+0000","last_clean":"2024-06-04T04:55:17.745731+0000","last_became_active":"2024-06-04T04:53:36.372000+0000","last_became_peered":"2024-06-04T04:53:36.372000+0000","last_unstale":"2024-06-04T04:55:17.745731+0000","last_undegraded":"2024-06-04T04:55:17.745731+0000","last_fullsized":"2024-06-04T04:55:17.745731+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:15:56.906996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.752310+0000","last_change":"2024-06-04T04:53:36.376051+0000","last_active":"2024-06-04T04:55:17.752310+0000","last_peered":"2024-06-04T04:55:17.752310+0000","last_clean":"2024-06-04T04:55:17.752310+0000","last_became_active":"2024-06-04T04:53:36.375726+0000","last_became_peered":"2024-06-04T04:53:36.375726+0000","last_unstale":"2024-06-04T04:55:17.752310+0000","last_undegraded":"2024-06-04T04:55:17.752310+0000","last_fullsized":"2024-06-04T04:55:17.752310+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:09:08.542764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.745381+0000","last_change":"2024-06-04T04:53:36.373266+0000","last_active":"2024-06-04T04:55:17.745381+0000","last_peered":"2024-06-04T04:55:17.745381+0000","last_clean":"2024-06-04T04:55:17.745381+0000","last_became_active":"2024-06-04T04:53:36.372898+0000","last_became_peered":"2024-06-04T04:53:36.372898+0000","last_unstale":"2024-06-04T04:55:17.745381+0000","last_undegraded":"2024-06-04T04:55:17.745381+0000","last_fullsized":"2024-06-04T04:55:17.745381+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:41:43.436781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.759967+0000","last_change":"2024-06-04T04:53:36.381344+0000","last_active":"2024-06-04T04:55:17.759967+0000","last_peered":"2024-06-04T04:55:17.759967+0000","last_clean":"2024-06-04T04:55:17.759967+0000","last_became_active":"2024-06-04T04:53:36.380736+0000","last_became_peered":"2024-06-04T04:53:36.380736+0000","last_unstale":"2024-06-04T04:55:17.759967+0000","last_undegraded":"2024-06-04T04:55:17.759967+0000","last_fullsized":"2024-06-04T04:55:17.759967+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:23:01.607272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.736749+0000","last_change":"2024-06-04T04:53:36.374035+0000","last_active":"2024-06-04T04:55:17.736749+0000","last_peered":"2024-06-04T04:55:17.736749+0000","last_clean":"2024-06-04T04:55:17.736749+0000","last_became_active":"2024-06-04T04:53:36.373667+0000","last_became_peered":"2024-06-04T04:53:36.373667+0000","last_unstale":"2024-06-04T04:55:17.736749+0000","last_undegraded":"2024-06-04T04:55:17.736749+0000","last_fullsized":"2024-06-04T04:55:17.736749+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:23:07.880369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.746826+0000","last_change":"2024-06-04T04:53:36.369613+0000","last_active":"2024-06-04T04:55:17.746826+0000","last_peered":"2024-06-04T04:55:17.746826+0000","last_clean":"2024-06-04T04:55:17.746826+0000","last_became_active":"2024-06-04T04:53:36.369277+0000","last_became_peered":"2024-06-04T04:53:36.369277+0000","last_unstale":"2024-06-04T04:55:17.746826+0000","last_undegraded":"2024-06-04T04:55:17.746826+0000","last_fullsized":"2024-06-04T04:55:17.746826+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:13:33.756609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.754113+0000","last_change":"2024-06-04T04:53:36.368094+0000","last_active":"2024-06-04T04:55:17.754113+0000","last_peered":"2024-06-04T04:55:17.754113+0000","last_clean":"2024-06-04T04:55:17.754113+0000","last_became_active":"2024-06-04T04:53:36.367717+0000","last_became_peered":"2024-06-04T04:53:36.367717+0000","last_unstale":"2024-06-04T04:55:17.754113+0000","last_undegraded":"2024-06-04T04:55:17.754113+0000","last_fullsized":"2024-06-04T04:55:17.754113+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:39:56.305078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.757070+0000","last_change":"2024-06-04T04:53:36.366765+0000","last_active":"2024-06-04T04:55:17.757070+0000","last_peered":"2024-06-04T04:55:17.757070+0000","last_clean":"2024-06-04T04:55:17.757070+0000","last_became_active":"2024-06-04T04:53:36.366438+0000","last_became_peered":"2024-06-04T04:53:36.366438+0000","last_unstale":"2024-06-04T04:55:17.757070+0000","last_undegraded":"2024-06-04T04:55:17.757070+0000","last_fullsized":"2024-06-04T04:55:17.757070+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:38:37.313926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758873+0000","last_change":"2024-06-04T04:53:36.380088+0000","last_active":"2024-06-04T04:55:17.758873+0000","last_peered":"2024-06-04T04:55:17.758873+0000","last_clean":"2024-06-04T04:55:17.758873+0000","last_became_active":"2024-06-04T04:53:36.379467+0000","last_became_peered":"2024-06-04T04:53:36.379467+0000","last_unstale":"2024-06-04T04:55:17.758873+0000","last_undegraded":"2024-06-04T04:55:17.758873+0000","last_fullsized":"2024-06-04T04:55:17.758873+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:46:53.236419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.759763+0000","last_change":"2024-06-04T04:53:36.381126+0000","last_active":"2024-06-04T04:55:17.759763+0000","last_peered":"2024-06-04T04:55:17.759763+0000","last_clean":"2024-06-04T04:55:17.759763+0000","last_became_active":"2024-06-04T04:53:36.378833+0000","last_became_peered":"2024-06-04T04:53:36.378833+0000","last_unstale":"2024-06-04T04:55:17.759763+0000","last_undegraded":"2024-06-04T04:55:17.759763+0000","last_fullsized":"2024-06-04T04:55:17.759763+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:20:28.639273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.753871+0000","last_change":"2024-06-04T04:53:36.369692+0000","last_active":"2024-06-04T04:55:17.753871+0000","last_peered":"2024-06-04T04:55:17.753871+0000","last_clean":"2024-06-04T04:55:17.753871+0000","last_became_active":"2024-06-04T04:53:36.369370+0000","last_became_peered":"2024-06-04T04:53:36.369370+0000","last_unstale":"2024-06-04T04:55:17.753871+0000","last_undegraded":"2024-06-04T04:55:17.753871+0000","last_fullsized":"2024-06-04T04:55:17.753871+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:55.275465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.749875+0000","last_change":"2024-06-04T04:53:36.383639+0000","last_active":"2024-06-04T04:55:17.749875+0000","last_peered":"2024-06-04T04:55:17.749875+0000","last_clean":"2024-06-04T04:55:17.749875+0000","last_became_active":"2024-06-04T04:53:36.383303+0000","last_became_peered":"2024-06-04T04:53:36.383303+0000","last_unstale":"2024-06-04T04:55:17.749875+0000","last_undegraded":"2024-06-04T04:55:17.749875+0000","last_fullsized":"2024-06-04T04:55:17.749875+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:26:31.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'14","reported_seq":176,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:46.751236+0000","last_change":"2024-06-04T04:54:19.641520+0000","last_active":"2024-06-04T04:55:46.751236+0000","last_peered":"2024-06-04T04:55:46.751236+0000","last_clean":"2024-06-04T04:55:46.751236+0000","last_became_active":"2024-06-04T04:54:19.632433+0000","last_became_peered":"2024-06-04T04:54:19.632433+0000","last_unstale":"2024-06-04T04:55:46.751236+0000","last_undegraded":"2024-06-04T04:55:46.751236+0000","last_fullsized":"2024-06-04T04:55:46.751236+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:26:05.993919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":109,"num_read_kb":109,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.744423+0000","last_change":"2024-06-04T04:53:36.371827+0000","last_active":"2024-06-04T04:55:17.744423+0000","last_peered":"2024-06-04T04:55:17.744423+0000","last_clean":"2024-06-04T04:55:17.744423+0000","last_became_active":"2024-06-04T04:53:36.371509+0000","last_became_peered":"2024-06-04T04:53:36.371509+0000","last_unstale":"2024-06-04T04:55:17.744423+0000","last_undegraded":"2024-06-04T04:55:17.744423+0000","last_fullsized":"2024-06-04T04:55:17.744423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:54:51.589008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.513230+0000","last_change":"2024-06-04T04:53:33.353976+0000","last_active":"2024-06-04T04:54:19.513230+0000","last_peered":"2024-06-04T04:54:19.513230+0000","last_clean":"2024-06-04T04:54:19.513230+0000","last_became_active":"2024-06-04T04:53:33.353586+0000","last_became_peered":"2024-06-04T04:53:33.353586+0000","last_unstale":"2024-06-04T04:54:19.513230+0000","last_undegraded":"2024-06-04T04:54:19.513230+0000","last_fullsized":"2024-06-04T04:54:19.513230+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:17:10.378008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.759061+0000","last_change":"2024-06-04T04:53:36.367806+0000","last_active":"2024-06-04T04:55:17.759061+0000","last_peered":"2024-06-04T04:55:17.759061+0000","last_clean":"2024-06-04T04:55:17.759061+0000","last_became_active":"2024-06-04T04:53:36.367484+0000","last_became_peered":"2024-06-04T04:53:36.367484+0000","last_unstale":"2024-06-04T04:55:17.759061+0000","last_undegraded":"2024-06-04T04:55:17.759061+0000","last_fullsized":"2024-06-04T04:55:17.759061+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:25:36.194746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509723+0000","last_change":"2024-06-04T04:53:33.348902+0000","last_active":"2024-06-04T04:54:19.509723+0000","last_peered":"2024-06-04T04:54:19.509723+0000","last_clean":"2024-06-04T04:54:19.509723+0000","last_became_active":"2024-06-04T04:53:33.348441+0000","last_became_peered":"2024-06-04T04:53:33.348441+0000","last_unstale":"2024-06-04T04:54:19.509723+0000","last_undegraded":"2024-06-04T04:54:19.509723+0000","last_fullsized":"2024-06-04T04:54:19.509723+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:22:28.428303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.748336+0000","last_change":"2024-06-04T04:53:36.376867+0000","last_active":"2024-06-04T04:55:17.748336+0000","last_peered":"2024-06-04T04:55:17.748336+0000","last_clean":"2024-06-04T04:55:17.748336+0000","last_became_active":"2024-06-04T04:53:36.376517+0000","last_became_peered":"2024-06-04T04:53:36.376517+0000","last_unstale":"2024-06-04T04:55:17.748336+0000","last_undegraded":"2024-06-04T04:55:17.748336+0000","last_fullsized":"2024-06-04T04:55:17.748336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:21:59.494321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509187+0000","last_change":"2024-06-04T04:53:33.347993+0000","last_active":"2024-06-04T04:54:19.509187+0000","last_peered":"2024-06-04T04:54:19.509187+0000","last_clean":"2024-06-04T04:54:19.509187+0000","last_became_active":"2024-06-04T04:53:33.347656+0000","last_became_peered":"2024-06-04T04:53:33.347656+0000","last_unstale":"2024-06-04T04:54:19.509187+0000","last_undegraded":"2024-06-04T04:54:19.509187+0000","last_fullsized":"2024-06-04T04:54:19.509187+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:21:26.806557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.751286+0000","last_change":"2024-06-04T04:54:19.632790+0000","last_active":"2024-06-04T04:55:17.751286+0000","last_peered":"2024-06-04T04:55:17.751286+0000","last_clean":"2024-06-04T04:55:17.751286+0000","last_became_active":"2024-06-04T04:54:19.632338+0000","last_became_peered":"2024-06-04T04:54:19.632338+0000","last_unstale":"2024-06-04T04:55:17.751286+0000","last_undegraded":"2024-06-04T04:55:17.751286+0000","last_fullsized":"2024-06-04T04:55:17.751286+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:44:00.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'5","reported_seq":149,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:48.618267+0000","last_change":"2024-06-04T04:53:33.346111+0000","last_active":"2024-06-04T04:55:48.618267+0000","last_peered":"2024-06-04T04:55:48.618267+0000","last_clean":"2024-06-04T04:55:48.618267+0000","last_became_active":"2024-06-04T04:53:33.345773+0000","last_became_peered":"2024-06-04T04:53:33.345773+0000","last_unstale":"2024-06-04T04:55:48.618267+0000","last_undegraded":"2024-06-04T04:55:48.618267+0000","last_fullsized":"2024-06-04T04:55:48.618267+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:18.565583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.738921+0000","last_change":"2024-06-04T04:53:36.373369+0000","last_active":"2024-06-04T04:55:17.738921+0000","last_peered":"2024-06-04T04:55:17.738921+0000","last_clean":"2024-06-04T04:55:17.738921+0000","last_became_active":"2024-06-04T04:53:36.373001+0000","last_became_peered":"2024-06-04T04:53:36.373001+0000","last_unstale":"2024-06-04T04:55:17.738921+0000","last_undegraded":"2024-06-04T04:55:17.738921+0000","last_fullsized":"2024-06-04T04:55:17.738921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:02:22.012643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508915+0000","last_change":"2024-06-04T04:53:33.344358+0000","last_active":"2024-06-04T04:54:19.508915+0000","last_peered":"2024-06-04T04:54:19.508915+0000","last_clean":"2024-06-04T04:54:19.508915+0000","last_became_active":"2024-06-04T04:53:33.343980+0000","last_became_peered":"2024-06-04T04:53:33.343980+0000","last_unstale":"2024-06-04T04:54:19.508915+0000","last_undegraded":"2024-06-04T04:54:19.508915+0000","last_fullsized":"2024-06-04T04:54:19.508915+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:14.493138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.745556+0000","last_change":"2024-06-04T04:53:36.371239+0000","last_active":"2024-06-04T04:55:17.745556+0000","last_peered":"2024-06-04T04:55:17.745556+0000","last_clean":"2024-06-04T04:55:17.745556+0000","last_became_active":"2024-06-04T04:53:36.370885+0000","last_became_peered":"2024-06-04T04:53:36.370885+0000","last_unstale":"2024-06-04T04:55:17.745556+0000","last_undegraded":"2024-06-04T04:55:17.745556+0000","last_fullsized":"2024-06-04T04:55:17.745556+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T04:54:29.753796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505354+0000","last_change":"2024-06-04T04:53:33.338512+0000","last_active":"2024-06-04T04:54:19.505354+0000","last_peered":"2024-06-04T04:54:19.505354+0000","last_clean":"2024-06-04T04:54:19.505354+0000","last_became_active":"2024-06-04T04:53:33.338180+0000","last_became_peered":"2024-06-04T04:53:33.338180+0000","last_unstale":"2024-06-04T04:54:19.505354+0000","last_undegraded":"2024-06-04T04:54:19.505354+0000","last_fullsized":"2024-06-04T04:54:19.505354+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:57:13.406479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.744719+0000","last_change":"2024-06-04T04:53:36.377316+0000","last_active":"2024-06-04T04:55:17.744719+0000","last_peered":"2024-06-04T04:55:17.744719+0000","last_clean":"2024-06-04T04:55:17.744719+0000","last_became_active":"2024-06-04T04:53:36.376992+0000","last_became_peered":"2024-06-04T04:53:36.376992+0000","last_unstale":"2024-06-04T04:55:17.744719+0000","last_undegraded":"2024-06-04T04:55:17.744719+0000","last_fullsized":"2024-06-04T04:55:17.744719+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:33.145631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511696+0000","last_change":"2024-06-04T04:53:33.348305+0000","last_active":"2024-06-04T04:54:19.511696+0000","last_peered":"2024-06-04T04:54:19.511696+0000","last_clean":"2024-06-04T04:54:19.511696+0000","last_became_active":"2024-06-04T04:53:33.347948+0000","last_became_peered":"2024-06-04T04:53:33.347948+0000","last_unstale":"2024-06-04T04:54:19.511696+0000","last_undegraded":"2024-06-04T04:54:19.511696+0000","last_fullsized":"2024-06-04T04:54:19.511696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:04:13.669274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.752489+0000","last_change":"2024-06-04T04:53:36.367955+0000","last_active":"2024-06-04T04:55:17.752489+0000","last_peered":"2024-06-04T04:55:17.752489+0000","last_clean":"2024-06-04T04:55:17.752489+0000","last_became_active":"2024-06-04T04:53:36.367628+0000","last_became_peered":"2024-06-04T04:53:36.367628+0000","last_unstale":"2024-06-04T04:55:17.752489+0000","last_undegraded":"2024-06-04T04:55:17.752489+0000","last_fullsized":"2024-06-04T04:55:17.752489+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:33:26.185498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'4","reported_seq":256,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:46.655165+0000","last_change":"2024-06-04T04:53:33.349471+0000","last_active":"2024-06-04T04:55:46.655165+0000","last_peered":"2024-06-04T04:55:46.655165+0000","last_clean":"2024-06-04T04:55:46.655165+0000","last_became_active":"2024-06-04T04:53:33.349105+0000","last_became_peered":"2024-06-04T04:53:33.349105+0000","last_unstale":"2024-06-04T04:55:46.655165+0000","last_undegraded":"2024-06-04T04:55:46.655165+0000","last_fullsized":"2024-06-04T04:55:46.655165+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:31:35.584275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":227,"num_read_kb":227,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.744175+0000","last_change":"2024-06-04T04:53:36.367249+0000","last_active":"2024-06-04T04:55:17.744175+0000","last_peered":"2024-06-04T04:55:17.744175+0000","last_clean":"2024-06-04T04:55:17.744175+0000","last_became_active":"2024-06-04T04:53:36.366863+0000","last_became_peered":"2024-06-04T04:53:36.366863+0000","last_unstale":"2024-06-04T04:55:17.744175+0000","last_undegraded":"2024-06-04T04:55:17.744175+0000","last_fullsized":"2024-06-04T04:55:17.744175+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:13:17.308321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510836+0000","last_change":"2024-06-04T04:53:33.346778+0000","last_active":"2024-06-04T04:54:19.510836+0000","last_peered":"2024-06-04T04:54:19.510836+0000","last_clean":"2024-06-04T04:54:19.510836+0000","last_became_active":"2024-06-04T04:53:33.346376+0000","last_became_peered":"2024-06-04T04:53:33.346376+0000","last_unstale":"2024-06-04T04:54:19.510836+0000","last_undegraded":"2024-06-04T04:54:19.510836+0000","last_fullsized":"2024-06-04T04:54:19.510836+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:03:16.973822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758785+0000","last_change":"2024-06-04T04:54:19.514491+0000","last_active":"2024-06-04T04:55:17.758785+0000","last_peered":"2024-06-04T04:55:17.758785+0000","last_clean":"2024-06-04T04:55:17.758785+0000","last_became_active":"2024-06-04T04:54:19.514148+0000","last_became_peered":"2024-06-04T04:54:19.514148+0000","last_unstale":"2024-06-04T04:55:17.758785+0000","last_undegraded":"2024-06-04T04:55:17.758785+0000","last_fullsized":"2024-06-04T04:55:17.758785+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:09:59.641041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512326+0000","last_change":"2024-06-04T04:53:33.354989+0000","last_active":"2024-06-04T04:54:19.512326+0000","last_peered":"2024-06-04T04:54:19.512326+0000","last_clean":"2024-06-04T04:54:19.512326+0000","last_became_active":"2024-06-04T04:53:33.354613+0000","last_became_peered":"2024-06-04T04:53:33.354613+0000","last_unstale":"2024-06-04T04:54:19.512326+0000","last_undegraded":"2024-06-04T04:54:19.512326+0000","last_fullsized":"2024-06-04T04:54:19.512326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:19:59.398763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.752224+0000","last_change":"2024-06-04T04:53:36.372216+0000","last_active":"2024-06-04T04:55:17.752224+0000","last_peered":"2024-06-04T04:55:17.752224+0000","last_clean":"2024-06-04T04:55:17.752224+0000","last_became_active":"2024-06-04T04:53:36.371820+0000","last_became_peered":"2024-06-04T04:53:36.371820+0000","last_unstale":"2024-06-04T04:55:17.752224+0000","last_undegraded":"2024-06-04T04:55:17.752224+0000","last_fullsized":"2024-06-04T04:55:17.752224+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:55:59.887963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'37","reported_seq":298,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:46.627820+0000","last_change":"2024-06-04T04:53:33.345493+0000","last_active":"2024-06-04T04:55:46.627820+0000","last_peered":"2024-06-04T04:55:46.627820+0000","last_clean":"2024-06-04T04:55:46.627820+0000","last_became_active":"2024-06-04T04:53:33.345102+0000","last_became_peered":"2024-06-04T04:53:33.345102+0000","last_unstale":"2024-06-04T04:55:46.627820+0000","last_undegraded":"2024-06-04T04:55:46.627820+0000","last_fullsized":"2024-06-04T04:55:46.627820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:50:27.853391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66452,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":236,"num_read_kb":183,"num_write":39,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758459+0000","last_change":"2024-06-04T04:53:36.382070+0000","last_active":"2024-06-04T04:55:17.758459+0000","last_peered":"2024-06-04T04:55:17.758459+0000","last_clean":"2024-06-04T04:55:17.758459+0000","last_became_active":"2024-06-04T04:53:36.379144+0000","last_became_peered":"2024-06-04T04:53:36.379144+0000","last_unstale":"2024-06-04T04:55:17.758459+0000","last_undegraded":"2024-06-04T04:55:17.758459+0000","last_fullsized":"2024-06-04T04:55:17.758459+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:16.702709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.506537+0000","last_change":"2024-06-04T04:53:33.344726+0000","last_active":"2024-06-04T04:54:19.506537+0000","last_peered":"2024-06-04T04:54:19.506537+0000","last_clean":"2024-06-04T04:54:19.506537+0000","last_became_active":"2024-06-04T04:53:33.344360+0000","last_became_peered":"2024-06-04T04:53:33.344360+0000","last_unstale":"2024-06-04T04:54:19.506537+0000","last_undegraded":"2024-06-04T04:54:19.506537+0000","last_fullsized":"2024-06-04T04:54:19.506537+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:44:56.784808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.751057+0000","last_change":"2024-06-04T04:53:36.379036+0000","last_active":"2024-06-04T04:55:17.751057+0000","last_peered":"2024-06-04T04:55:17.751057+0000","last_clean":"2024-06-04T04:55:17.751057+0000","last_became_active":"2024-06-04T04:53:36.378681+0000","last_became_peered":"2024-06-04T04:53:36.378681+0000","last_unstale":"2024-06-04T04:55:17.751057+0000","last_undegraded":"2024-06-04T04:55:17.751057+0000","last_fullsized":"2024-06-04T04:55:17.751057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:07:49.704731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511967+0000","last_change":"2024-06-04T04:53:33.345915+0000","last_active":"2024-06-04T04:54:19.511967+0000","last_peered":"2024-06-04T04:54:19.511967+0000","last_clean":"2024-06-04T04:54:19.511967+0000","last_became_active":"2024-06-04T04:53:33.345537+0000","last_became_peered":"2024-06-04T04:53:33.345537+0000","last_unstale":"2024-06-04T04:54:19.511967+0000","last_undegraded":"2024-06-04T04:54:19.511967+0000","last_fullsized":"2024-06-04T04:54:19.511967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:06:44.086968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.757276+0000","last_change":"2024-06-04T04:53:36.379094+0000","last_active":"2024-06-04T04:55:17.757276+0000","last_peered":"2024-06-04T04:55:17.757276+0000","last_clean":"2024-06-04T04:55:17.757276+0000","last_became_active":"2024-06-04T04:53:36.378763+0000","last_became_peered":"2024-06-04T04:53:36.378763+0000","last_unstale":"2024-06-04T04:55:17.757276+0000","last_undegraded":"2024-06-04T04:55:17.757276+0000","last_fullsized":"2024-06-04T04:55:17.757276+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:16.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":149,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:48.617422+0000","last_change":"2024-06-04T04:53:33.341342+0000","last_active":"2024-06-04T04:55:48.617422+0000","last_peered":"2024-06-04T04:55:48.617422+0000","last_clean":"2024-06-04T04:55:48.617422+0000","last_became_active":"2024-06-04T04:53:33.340862+0000","last_became_peered":"2024-06-04T04:53:33.340862+0000","last_unstale":"2024-06-04T04:55:48.617422+0000","last_undegraded":"2024-06-04T04:55:48.617422+0000","last_fullsized":"2024-06-04T04:55:48.617422+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:26.818063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":119,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.755138+0000","last_change":"2024-06-04T04:53:36.383124+0000","last_active":"2024-06-04T04:55:17.755138+0000","last_peered":"2024-06-04T04:55:17.755138+0000","last_clean":"2024-06-04T04:55:17.755138+0000","last_became_active":"2024-06-04T04:53:36.382449+0000","last_became_peered":"2024-06-04T04:53:36.382449+0000","last_unstale":"2024-06-04T04:55:17.755138+0000","last_undegraded":"2024-06-04T04:55:17.755138+0000","last_fullsized":"2024-06-04T04:55:17.755138+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:52:40.776003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":44,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508738+0000","last_change":"2024-06-04T04:53:33.354556+0000","last_active":"2024-06-04T04:54:19.508738+0000","last_peered":"2024-06-04T04:54:19.508738+0000","last_clean":"2024-06-04T04:54:19.508738+0000","last_became_active":"2024-06-04T04:53:33.354238+0000","last_became_peered":"2024-06-04T04:53:33.354238+0000","last_unstale":"2024-06-04T04:54:19.508738+0000","last_undegraded":"2024-06-04T04:54:19.508738+0000","last_fullsized":"2024-06-04T04:54:19.508738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:54.697374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.750667+0000","last_change":"2024-06-04T04:53:36.371387+0000","last_active":"2024-06-04T04:55:17.750667+0000","last_peered":"2024-06-04T04:55:17.750667+0000","last_clean":"2024-06-04T04:55:17.750667+0000","last_became_active":"2024-06-04T04:53:36.370852+0000","last_became_peered":"2024-06-04T04:53:36.370852+0000","last_unstale":"2024-06-04T04:55:17.750667+0000","last_undegraded":"2024-06-04T04:55:17.750667+0000","last_fullsized":"2024-06-04T04:55:17.750667+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:44:06.956210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.508796+0000","last_change":"2024-06-04T04:53:33.349982+0000","last_active":"2024-06-04T04:54:19.508796+0000","last_peered":"2024-06-04T04:54:19.508796+0000","last_clean":"2024-06-04T04:54:19.508796+0000","last_became_active":"2024-06-04T04:53:33.349633+0000","last_became_peered":"2024-06-04T04:53:33.349633+0000","last_unstale":"2024-06-04T04:54:19.508796+0000","last_undegraded":"2024-06-04T04:54:19.508796+0000","last_fullsized":"2024-06-04T04:54:19.508796+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:40.107426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.755048+0000","last_change":"2024-06-04T04:53:36.382313+0000","last_active":"2024-06-04T04:55:17.755048+0000","last_peered":"2024-06-04T04:55:17.755048+0000","last_clean":"2024-06-04T04:55:17.755048+0000","last_became_active":"2024-06-04T04:53:36.379638+0000","last_became_peered":"2024-06-04T04:53:36.379638+0000","last_unstale":"2024-06-04T04:55:17.755048+0000","last_undegraded":"2024-06-04T04:55:17.755048+0000","last_fullsized":"2024-06-04T04:55:17.755048+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:43:42.527262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.515392+0000","last_change":"2024-06-04T04:54:19.513581+0000","last_active":"2024-06-04T04:54:19.515392+0000","last_peered":"2024-06-04T04:54:19.515392+0000","last_clean":"2024-06-04T04:54:19.515392+0000","last_became_active":"2024-06-04T04:54:19.513111+0000","last_became_peered":"2024-06-04T04:54:19.513111+0000","last_unstale":"2024-06-04T04:54:19.515392+0000","last_undegraded":"2024-06-04T04:54:19.515392+0000","last_fullsized":"2024-06-04T04:54:19.515392+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:57.028083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.740728+0000","last_change":"2024-06-04T04:53:36.369904+0000","last_active":"2024-06-04T04:55:17.740728+0000","last_peered":"2024-06-04T04:55:17.740728+0000","last_clean":"2024-06-04T04:55:17.740728+0000","last_became_active":"2024-06-04T04:53:36.367929+0000","last_became_peered":"2024-06-04T04:53:36.367929+0000","last_unstale":"2024-06-04T04:55:17.740728+0000","last_undegraded":"2024-06-04T04:55:17.740728+0000","last_fullsized":"2024-06-04T04:55:17.740728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:37:08.721569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"81'7","reported_seq":152,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:47.788916+0000","last_change":"2024-06-04T04:53:33.342723+0000","last_active":"2024-06-04T04:55:47.788916+0000","last_peered":"2024-06-04T04:55:47.788916+0000","last_clean":"2024-06-04T04:55:47.788916+0000","last_became_active":"2024-06-04T04:53:33.342344+0000","last_became_peered":"2024-06-04T04:53:33.342344+0000","last_unstale":"2024-06-04T04:55:47.788916+0000","last_undegraded":"2024-06-04T04:55:47.788916+0000","last_fullsized":"2024-06-04T04:55:47.788916+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:59:50.147517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.745755+0000","last_change":"2024-06-04T04:54:19.513534+0000","last_active":"2024-06-04T04:55:17.745755+0000","last_peered":"2024-06-04T04:55:17.745755+0000","last_clean":"2024-06-04T04:55:17.745755+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:55:17.745755+0000","last_undegraded":"2024-06-04T04:55:17.745755+0000","last_fullsized":"2024-06-04T04:55:17.745755+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:16:32.240932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.741074+0000","last_change":"2024-06-04T04:53:36.371236+0000","last_active":"2024-06-04T04:55:17.741074+0000","last_peered":"2024-06-04T04:55:17.741074+0000","last_clean":"2024-06-04T04:55:17.741074+0000","last_became_active":"2024-06-04T04:53:36.370876+0000","last_became_peered":"2024-06-04T04:53:36.370876+0000","last_unstale":"2024-06-04T04:55:17.741074+0000","last_undegraded":"2024-06-04T04:55:17.741074+0000","last_fullsized":"2024-06-04T04:55:17.741074+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:15:56.751317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":143,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:43.609770+0000","last_change":"2024-06-04T04:53:33.355642+0000","last_active":"2024-06-04T04:55:43.609770+0000","last_peered":"2024-06-04T04:55:43.609770+0000","last_clean":"2024-06-04T04:55:43.609770+0000","last_became_active":"2024-06-04T04:53:33.355187+0000","last_became_peered":"2024-06-04T04:53:33.355187+0000","last_unstale":"2024-06-04T04:55:43.609770+0000","last_undegraded":"2024-06-04T04:55:43.609770+0000","last_fullsized":"2024-06-04T04:55:43.609770+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:32:28.331518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.754076+0000","last_change":"2024-06-04T04:54:19.631750+0000","last_active":"2024-06-04T04:55:17.754076+0000","last_peered":"2024-06-04T04:55:17.754076+0000","last_clean":"2024-06-04T04:55:17.754076+0000","last_became_active":"2024-06-04T04:54:19.631485+0000","last_became_peered":"2024-06-04T04:54:19.631485+0000","last_unstale":"2024-06-04T04:55:17.754076+0000","last_undegraded":"2024-06-04T04:55:17.754076+0000","last_fullsized":"2024-06-04T04:55:17.754076+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:29:44.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631560+0000","last_change":"2024-06-04T04:53:33.340653+0000","last_active":"2024-06-04T04:54:19.631560+0000","last_peered":"2024-06-04T04:54:19.631560+0000","last_clean":"2024-06-04T04:54:19.631560+0000","last_became_active":"2024-06-04T04:53:33.340250+0000","last_became_peered":"2024-06-04T04:53:33.340250+0000","last_unstale":"2024-06-04T04:54:19.631560+0000","last_undegraded":"2024-06-04T04:54:19.631560+0000","last_fullsized":"2024-06-04T04:54:19.631560+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:46:20.329514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":100,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.510709+0000","last_change":"2024-06-04T04:52:09.181155+0000","last_active":"2024-06-04T04:54:19.510709+0000","last_peered":"2024-06-04T04:54:19.510709+0000","last_clean":"2024-06-04T04:54:19.510709+0000","last_became_active":"2024-06-04T04:52:09.177002+0000","last_became_peered":"2024-06-04T04:52:09.177002+0000","last_unstale":"2024-06-04T04:54:19.510709+0000","last_undegraded":"2024-06-04T04:54:19.510709+0000","last_fullsized":"2024-06-04T04:54:19.510709+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_clean_scrub_stamp":"2024-06-04T04:48:44.631948+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:46:57.735302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.755531+0000","last_change":"2024-06-04T04:53:36.372634+0000","last_active":"2024-06-04T04:55:17.755531+0000","last_peered":"2024-06-04T04:55:17.755531+0000","last_clean":"2024-06-04T04:55:17.755531+0000","last_became_active":"2024-06-04T04:53:36.372290+0000","last_became_peered":"2024-06-04T04:53:36.372290+0000","last_unstale":"2024-06-04T04:55:17.755531+0000","last_undegraded":"2024-06-04T04:55:17.755531+0000","last_fullsized":"2024-06-04T04:55:17.755531+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:14:07.460382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.511064+0000","last_change":"2024-06-04T04:53:33.348760+0000","last_active":"2024-06-04T04:54:19.511064+0000","last_peered":"2024-06-04T04:54:19.511064+0000","last_clean":"2024-06-04T04:54:19.511064+0000","last_became_active":"2024-06-04T04:53:33.348347+0000","last_became_peered":"2024-06-04T04:53:33.348347+0000","last_unstale":"2024-06-04T04:54:19.511064+0000","last_undegraded":"2024-06-04T04:54:19.511064+0000","last_fullsized":"2024-06-04T04:54:19.511064+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:36:20.641061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.509608+0000","last_change":"2024-06-04T04:53:33.354840+0000","last_active":"2024-06-04T04:54:19.509608+0000","last_peered":"2024-06-04T04:54:19.509608+0000","last_clean":"2024-06-04T04:54:19.509608+0000","last_became_active":"2024-06-04T04:53:33.354287+0000","last_became_peered":"2024-06-04T04:53:33.354287+0000","last_unstale":"2024-06-04T04:54:19.509608+0000","last_undegraded":"2024-06-04T04:54:19.509608+0000","last_fullsized":"2024-06-04T04:54:19.509608+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:42.550098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.755774+0000","last_change":"2024-06-04T04:53:36.372678+0000","last_active":"2024-06-04T04:55:17.755774+0000","last_peered":"2024-06-04T04:55:17.755774+0000","last_clean":"2024-06-04T04:55:17.755774+0000","last_became_active":"2024-06-04T04:53:36.372312+0000","last_became_peered":"2024-06-04T04:53:36.372312+0000","last_unstale":"2024-06-04T04:55:17.755774+0000","last_undegraded":"2024-06-04T04:55:17.755774+0000","last_fullsized":"2024-06-04T04:55:17.755774+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:34:41.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'15","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:18.386842+0000","last_change":"2024-06-04T04:53:33.348440+0000","last_active":"2024-06-04T04:55:18.386842+0000","last_peered":"2024-06-04T04:55:18.386842+0000","last_clean":"2024-06-04T04:55:18.386842+0000","last_became_active":"2024-06-04T04:53:33.348131+0000","last_became_peered":"2024-06-04T04:53:33.348131+0000","last_unstale":"2024-06-04T04:55:18.386842+0000","last_undegraded":"2024-06-04T04:55:18.386842+0000","last_fullsized":"2024-06-04T04:55:18.386842+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:14:11.552286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.760097+0000","last_change":"2024-06-04T04:53:36.378669+0000","last_active":"2024-06-04T04:55:17.760097+0000","last_peered":"2024-06-04T04:55:17.760097+0000","last_clean":"2024-06-04T04:55:17.760097+0000","last_became_active":"2024-06-04T04:53:36.378233+0000","last_became_peered":"2024-06-04T04:53:36.378233+0000","last_unstale":"2024-06-04T04:55:17.760097+0000","last_undegraded":"2024-06-04T04:55:17.760097+0000","last_fullsized":"2024-06-04T04:55:17.760097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:36:45.870924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.631707+0000","last_change":"2024-06-04T04:53:33.345078+0000","last_active":"2024-06-04T04:54:19.631707+0000","last_peered":"2024-06-04T04:54:19.631707+0000","last_clean":"2024-06-04T04:54:19.631707+0000","last_became_active":"2024-06-04T04:53:33.344708+0000","last_became_peered":"2024-06-04T04:53:33.344708+0000","last_unstale":"2024-06-04T04:54:19.631707+0000","last_undegraded":"2024-06-04T04:54:19.631707+0000","last_fullsized":"2024-06-04T04:54:19.631707+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:45:14.010758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758144+0000","last_change":"2024-06-04T04:53:36.364817+0000","last_active":"2024-06-04T04:55:17.758144+0000","last_peered":"2024-06-04T04:55:17.758144+0000","last_clean":"2024-06-04T04:55:17.758144+0000","last_became_active":"2024-06-04T04:53:36.364499+0000","last_became_peered":"2024-06-04T04:53:36.364499+0000","last_unstale":"2024-06-04T04:55:17.758144+0000","last_undegraded":"2024-06-04T04:55:17.758144+0000","last_fullsized":"2024-06-04T04:55:17.758144+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:39:30.518028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.630933+0000","last_change":"2024-06-04T04:53:33.354527+0000","last_active":"2024-06-04T04:54:19.630933+0000","last_peered":"2024-06-04T04:54:19.630933+0000","last_clean":"2024-06-04T04:54:19.630933+0000","last_became_active":"2024-06-04T04:53:33.354175+0000","last_became_peered":"2024-06-04T04:53:33.354175+0000","last_unstale":"2024-06-04T04:54:19.630933+0000","last_undegraded":"2024-06-04T04:54:19.630933+0000","last_fullsized":"2024-06-04T04:54:19.630933+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:33:08.125428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.734552+0000","last_change":"2024-06-04T04:53:36.373248+0000","last_active":"2024-06-04T04:55:17.734552+0000","last_peered":"2024-06-04T04:55:17.734552+0000","last_clean":"2024-06-04T04:55:17.734552+0000","last_became_active":"2024-06-04T04:53:36.372915+0000","last_became_peered":"2024-06-04T04:53:36.372915+0000","last_unstale":"2024-06-04T04:55:17.734552+0000","last_undegraded":"2024-06-04T04:55:17.734552+0000","last_fullsized":"2024-06-04T04:55:17.734552+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:23:11.689349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512461+0000","last_change":"2024-06-04T04:53:33.340427+0000","last_active":"2024-06-04T04:54:19.512461+0000","last_peered":"2024-06-04T04:54:19.512461+0000","last_clean":"2024-06-04T04:54:19.512461+0000","last_became_active":"2024-06-04T04:53:33.340022+0000","last_became_peered":"2024-06-04T04:53:33.340022+0000","last_unstale":"2024-06-04T04:54:19.512461+0000","last_undegraded":"2024-06-04T04:54:19.512461+0000","last_fullsized":"2024-06-04T04:54:19.512461+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:56:16.496471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.759040+0000","last_change":"2024-06-04T04:53:36.380114+0000","last_active":"2024-06-04T04:55:17.759040+0000","last_peered":"2024-06-04T04:55:17.759040+0000","last_clean":"2024-06-04T04:55:17.759040+0000","last_became_active":"2024-06-04T04:53:36.379392+0000","last_became_peered":"2024-06-04T04:53:36.379392+0000","last_unstale":"2024-06-04T04:55:17.759040+0000","last_undegraded":"2024-06-04T04:55:17.759040+0000","last_fullsized":"2024-06-04T04:55:17.759040+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:41:48.776439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.520195+0000","last_change":"2024-06-04T04:54:19.519755+0000","last_active":"2024-06-04T04:54:19.520195+0000","last_peered":"2024-06-04T04:54:19.520195+0000","last_clean":"2024-06-04T04:54:19.520195+0000","last_became_active":"2024-06-04T04:54:19.513992+0000","last_became_peered":"2024-06-04T04:54:19.513992+0000","last_unstale":"2024-06-04T04:54:19.520195+0000","last_undegraded":"2024-06-04T04:54:19.520195+0000","last_fullsized":"2024-06-04T04:54:19.520195+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:01.951935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.753820+0000","last_change":"2024-06-04T04:53:36.377420+0000","last_active":"2024-06-04T04:55:17.753820+0000","last_peered":"2024-06-04T04:55:17.753820+0000","last_clean":"2024-06-04T04:55:17.753820+0000","last_became_active":"2024-06-04T04:53:36.377109+0000","last_became_peered":"2024-06-04T04:53:36.377109+0000","last_unstale":"2024-06-04T04:55:17.753820+0000","last_undegraded":"2024-06-04T04:55:17.753820+0000","last_fullsized":"2024-06-04T04:55:17.753820+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:47:12.102396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":43,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.645328+0000","last_change":"2024-06-04T04:54:19.645328+0000","last_active":"2024-06-04T04:54:19.645328+0000","last_peered":"2024-06-04T04:54:19.645328+0000","last_clean":"2024-06-04T04:54:19.645328+0000","last_became_active":"2024-06-04T04:54:19.633416+0000","last_became_peered":"2024-06-04T04:54:19.633416+0000","last_unstale":"2024-06-04T04:54:19.645328+0000","last_undegraded":"2024-06-04T04:54:19.645328+0000","last_fullsized":"2024-06-04T04:54:19.645328+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:04:20.494641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.759937+0000","last_change":"2024-06-04T04:53:36.381187+0000","last_active":"2024-06-04T04:55:17.759937+0000","last_peered":"2024-06-04T04:55:17.759937+0000","last_clean":"2024-06-04T04:55:17.759937+0000","last_became_active":"2024-06-04T04:53:36.380591+0000","last_became_peered":"2024-06-04T04:53:36.380591+0000","last_unstale":"2024-06-04T04:55:17.759937+0000","last_undegraded":"2024-06-04T04:55:17.759937+0000","last_fullsized":"2024-06-04T04:55:17.759937+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:26:43.122339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'5","reported_seq":148,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:48.659840+0000","last_change":"2024-06-04T04:53:33.355908+0000","last_active":"2024-06-04T04:55:48.659840+0000","last_peered":"2024-06-04T04:55:48.659840+0000","last_clean":"2024-06-04T04:55:48.659840+0000","last_became_active":"2024-06-04T04:53:33.355522+0000","last_became_peered":"2024-06-04T04:53:33.355522+0000","last_unstale":"2024-06-04T04:55:48.659840+0000","last_undegraded":"2024-06-04T04:55:48.659840+0000","last_fullsized":"2024-06-04T04:55:48.659840+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:16:44.639431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.759847+0000","last_change":"2024-06-04T04:53:36.382737+0000","last_active":"2024-06-04T04:55:17.759847+0000","last_peered":"2024-06-04T04:55:17.759847+0000","last_clean":"2024-06-04T04:55:17.759847+0000","last_became_active":"2024-06-04T04:53:36.382353+0000","last_became_peered":"2024-06-04T04:53:36.382353+0000","last_unstale":"2024-06-04T04:55:17.759847+0000","last_undegraded":"2024-06-04T04:55:17.759847+0000","last_fullsized":"2024-06-04T04:55:17.759847+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:18:01.476058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.512174+0000","last_change":"2024-06-04T04:53:33.347273+0000","last_active":"2024-06-04T04:54:19.512174+0000","last_peered":"2024-06-04T04:54:19.512174+0000","last_clean":"2024-06-04T04:54:19.512174+0000","last_became_active":"2024-06-04T04:53:33.346915+0000","last_became_peered":"2024-06-04T04:53:33.346915+0000","last_unstale":"2024-06-04T04:54:19.512174+0000","last_undegraded":"2024-06-04T04:54:19.512174+0000","last_fullsized":"2024-06-04T04:54:19.512174+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:17.235484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.759127+0000","last_change":"2024-06-04T04:53:36.383454+0000","last_active":"2024-06-04T04:55:17.759127+0000","last_peered":"2024-06-04T04:55:17.759127+0000","last_clean":"2024-06-04T04:55:17.759127+0000","last_became_active":"2024-06-04T04:53:36.383105+0000","last_became_peered":"2024-06-04T04:53:36.383105+0000","last_unstale":"2024-06-04T04:55:17.759127+0000","last_undegraded":"2024-06-04T04:55:17.759127+0000","last_fullsized":"2024-06-04T04:55:17.759127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:04:02.147688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:54:19.505624+0000","last_change":"2024-06-04T04:53:33.345435+0000","last_active":"2024-06-04T04:54:19.505624+0000","last_peered":"2024-06-04T04:54:19.505624+0000","last_clean":"2024-06-04T04:54:19.505624+0000","last_became_active":"2024-06-04T04:53:33.345077+0000","last_became_peered":"2024-06-04T04:53:33.345077+0000","last_unstale":"2024-06-04T04:54:19.505624+0000","last_undegraded":"2024-06-04T04:54:19.505624+0000","last_fullsized":"2024-06-04T04:54:19.505624+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:20:13.992259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.760007+0000","last_change":"2024-06-04T04:53:36.385238+0000","last_active":"2024-06-04T04:55:17.760007+0000","last_peered":"2024-06-04T04:55:17.760007+0000","last_clean":"2024-06-04T04:55:17.760007+0000","last_became_active":"2024-06-04T04:53:36.384899+0000","last_became_peered":"2024-06-04T04:53:36.384899+0000","last_unstale":"2024-06-04T04:55:17.760007+0000","last_undegraded":"2024-06-04T04:55:17.760007+0000","last_fullsized":"2024-06-04T04:55:17.760007+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:09:42.629340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.751843+0000","last_change":"2024-06-04T04:53:36.383450+0000","last_active":"2024-06-04T04:55:17.751843+0000","last_peered":"2024-06-04T04:55:17.751843+0000","last_clean":"2024-06-04T04:55:17.751843+0000","last_became_active":"2024-06-04T04:53:36.382678+0000","last_became_peered":"2024-06-04T04:53:36.382678+0000","last_unstale":"2024-06-04T04:55:17.751843+0000","last_undegraded":"2024-06-04T04:55:17.751843+0000","last_fullsized":"2024-06-04T04:55:17.751843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:36:28.932636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.752940+0000","last_change":"2024-06-04T04:54:19.632563+0000","last_active":"2024-06-04T04:55:17.752940+0000","last_peered":"2024-06-04T04:55:17.752940+0000","last_clean":"2024-06-04T04:55:17.752940+0000","last_became_active":"2024-06-04T04:54:19.632070+0000","last_became_peered":"2024-06-04T04:54:19.632070+0000","last_unstale":"2024-06-04T04:55:17.752940+0000","last_undegraded":"2024-06-04T04:55:17.752940+0000","last_fullsized":"2024-06-04T04:55:17.752940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:07:16.552958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758655+0000","last_change":"2024-06-04T04:53:36.381408+0000","last_active":"2024-06-04T04:55:17.758655+0000","last_peered":"2024-06-04T04:55:17.758655+0000","last_clean":"2024-06-04T04:55:17.758655+0000","last_became_active":"2024-06-04T04:53:36.381078+0000","last_became_peered":"2024-06-04T04:53:36.381078+0000","last_unstale":"2024-06-04T04:55:17.758655+0000","last_undegraded":"2024-06-04T04:55:17.758655+0000","last_fullsized":"2024-06-04T04:55:17.758655+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:28:48.988591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758212+0000","last_change":"2024-06-04T04:53:36.360168+0000","last_active":"2024-06-04T04:55:17.758212+0000","last_peered":"2024-06-04T04:55:17.758212+0000","last_clean":"2024-06-04T04:55:17.758212+0000","last_became_active":"2024-06-04T04:53:36.359691+0000","last_became_peered":"2024-06-04T04:53:36.359691+0000","last_unstale":"2024-06-04T04:55:17.758212+0000","last_undegraded":"2024-06-04T04:55:17.758212+0000","last_fullsized":"2024-06-04T04:55:17.758212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:55:17.418339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758241+0000","last_change":"2024-06-04T04:53:36.363018+0000","last_active":"2024-06-04T04:55:17.758241+0000","last_peered":"2024-06-04T04:55:17.758241+0000","last_clean":"2024-06-04T04:55:17.758241+0000","last_became_active":"2024-06-04T04:53:36.361595+0000","last_became_peered":"2024-06-04T04:53:36.361595+0000","last_unstale":"2024-06-04T04:55:17.758241+0000","last_undegraded":"2024-06-04T04:55:17.758241+0000","last_fullsized":"2024-06-04T04:55:17.758241+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:00:21.377322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.751679+0000","last_change":"2024-06-04T04:53:36.375377+0000","last_active":"2024-06-04T04:55:17.751679+0000","last_peered":"2024-06-04T04:55:17.751679+0000","last_clean":"2024-06-04T04:55:17.751679+0000","last_became_active":"2024-06-04T04:53:36.373459+0000","last_became_peered":"2024-06-04T04:53:36.373459+0000","last_unstale":"2024-06-04T04:55:17.751679+0000","last_undegraded":"2024-06-04T04:55:17.751679+0000","last_fullsized":"2024-06-04T04:55:17.751679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:51:31.686969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.745493+0000","last_change":"2024-06-04T04:53:36.373850+0000","last_active":"2024-06-04T04:55:17.745493+0000","last_peered":"2024-06-04T04:55:17.745493+0000","last_clean":"2024-06-04T04:55:17.745493+0000","last_became_active":"2024-06-04T04:53:36.371951+0000","last_became_peered":"2024-06-04T04:53:36.371951+0000","last_unstale":"2024-06-04T04:55:17.745493+0000","last_undegraded":"2024-06-04T04:55:17.745493+0000","last_fullsized":"2024-06-04T04:55:17.745493+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:06.452703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.744089+0000","last_change":"2024-06-04T04:53:36.374871+0000","last_active":"2024-06-04T04:55:17.744089+0000","last_peered":"2024-06-04T04:55:17.744089+0000","last_clean":"2024-06-04T04:55:17.744089+0000","last_became_active":"2024-06-04T04:53:36.374486+0000","last_became_peered":"2024-06-04T04:53:36.374486+0000","last_unstale":"2024-06-04T04:55:17.744089+0000","last_undegraded":"2024-06-04T04:55:17.744089+0000","last_fullsized":"2024-06-04T04:55:17.744089+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:15:13.337374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.757968+0000","last_change":"2024-06-04T04:54:19.513540+0000","last_active":"2024-06-04T04:55:17.757968+0000","last_peered":"2024-06-04T04:55:17.757968+0000","last_clean":"2024-06-04T04:55:17.757968+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:55:17.757968+0000","last_undegraded":"2024-06-04T04:55:17.757968+0000","last_fullsized":"2024-06-04T04:55:17.757968+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:09:27.993585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.744742+0000","last_change":"2024-06-04T04:53:36.373055+0000","last_active":"2024-06-04T04:55:17.744742+0000","last_peered":"2024-06-04T04:55:17.744742+0000","last_clean":"2024-06-04T04:55:17.744742+0000","last_became_active":"2024-06-04T04:53:36.372754+0000","last_became_peered":"2024-06-04T04:53:36.372754+0000","last_unstale":"2024-06-04T04:55:17.744742+0000","last_undegraded":"2024-06-04T04:55:17.744742+0000","last_fullsized":"2024-06-04T04:55:17.744742+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:22:27.554200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.758984+0000","last_change":"2024-06-04T04:53:36.378897+0000","last_active":"2024-06-04T04:55:17.758984+0000","last_peered":"2024-06-04T04:55:17.758984+0000","last_clean":"2024-06-04T04:55:17.758984+0000","last_became_active":"2024-06-04T04:53:36.378607+0000","last_became_peered":"2024-06-04T04:53:36.378607+0000","last_unstale":"2024-06-04T04:55:17.758984+0000","last_undegraded":"2024-06-04T04:55:17.758984+0000","last_fullsized":"2024-06-04T04:55:17.758984+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:03.395491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.756938+0000","last_change":"2024-06-04T04:53:36.378240+0000","last_active":"2024-06-04T04:55:17.756938+0000","last_peered":"2024-06-04T04:55:17.756938+0000","last_clean":"2024-06-04T04:55:17.756938+0000","last_became_active":"2024-06-04T04:53:36.377909+0000","last_became_peered":"2024-06-04T04:53:36.377909+0000","last_unstale":"2024-06-04T04:55:17.756938+0000","last_undegraded":"2024-06-04T04:55:17.756938+0000","last_fullsized":"2024-06-04T04:55:17.756938+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:15:25.288340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-06-04T04:55:17.754518+0000","last_change":"2024-06-04T04:53:36.379792+0000","last_active":"2024-06-04T04:55:17.754518+0000","last_peered":"2024-06-04T04:55:17.754518+0000","last_clean":"2024-06-04T04:55:17.754518+0000","last_became_active":"2024-06-04T04:53:36.379438+0000","last_became_peered":"2024-06-04T04:53:36.379438+0000","last_unstale":"2024-06-04T04:55:17.754518+0000","last_undegraded":"2024-06-04T04:55:17.754518+0000","last_fullsized":"2024-06-04T04:55:17.754518+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:23:57.617877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":214930,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73406,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1176,"num_read_kb":661,"num_write":252,"num_write_kb":147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":220218,"data_compressed":5499,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":217,"ondisk_log_size":217,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678058,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":880640,"data_stored":429560,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874287,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":1087041,"data_compressed":43684,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037813,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":429490,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234042,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":429196,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430270,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":1022527,"data_compressed":41851,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593796,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":429739,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":497776,"data_compressed":7592,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":909312,"data_stored":430546,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":496792,"data_compressed":7592,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36220,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708932,"statfs":{"total":95995035648,"available":95957946368,"internally_reserved":0,"allocated":905216,"data_stored":429032,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953551,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32044,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713108,"statfs":{"total":95995035648,"available":95962222592,"internally_reserved":0,"allocated":888832,"data_stored":429068,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790026,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":1021673,"data_compressed":41851,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34470,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":69408,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":68424,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":68188,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-04T04:55:50.677 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-04T04:55:50.677 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-04T04:55:50.677 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-04T04:55:50.677 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-04T04:55:50.916 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:51 smithi070 ceph-mon[32631]: pgmap v357: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-06-04T04:55:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:51 smithi070 ceph-mon[32631]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:55:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:51 smithi112 ceph-mon[36453]: pgmap v357: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-06-04T04:55:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:51 smithi112 ceph-mon[36453]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:55:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:51 smithi098 ceph-mon[35569]: pgmap v357: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-06-04T04:55:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:51 smithi098 ceph-mon[35569]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:55:52.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.440+0000 7fdf89845700 1 -- 172.21.15.70:0/1392110620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf841022d0 msgr2=0x7fdf84109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:52.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.440+0000 7fdf89845700 1 --2- 172.21.15.70:0/1392110620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf841022d0 0x7fdf84109320 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdf780099c0 tx=0x7fdf78009cd0 comp rx=0 tx=0).stop 2024-06-04T04:55:52.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.440+0000 7fdf89845700 1 -- 172.21.15.70:0/1392110620 shutdown_connections 2024-06-04T04:55:52.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.440+0000 7fdf89845700 1 --2- 172.21.15.70:0/1392110620 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf84109a00 0x7fdf84110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:52.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.440+0000 7fdf89845700 1 --2- 172.21.15.70:0/1392110620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf84101920 0x7fdf84101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:52.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.440+0000 7fdf89845700 1 --2- 172.21.15.70:0/1392110620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf841022d0 0x7fdf84109320 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:52.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.440+0000 7fdf89845700 1 -- 172.21.15.70:0/1392110620 >> 172.21.15.70:0/1392110620 conn(0x7fdf84078640 msgr2=0x7fdf840ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:52.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.440+0000 7fdf89845700 1 -- 172.21.15.70:0/1392110620 shutdown_connections 2024-06-04T04:55:52.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.440+0000 7fdf89845700 1 -- 172.21.15.70:0/1392110620 wait complete. 2024-06-04T04:55:52.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.440+0000 7fdf89845700 1 Processor -- start 2024-06-04T04:55:52.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf89845700 1 -- start start 2024-06-04T04:55:52.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf89845700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf84101920 0x7fdf84103130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:52.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf89845700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf841022d0 0x7fdf84103670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf89845700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf84109a00 0x7fdf84103bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf89845700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdf841135a0 con 0x7fdf841022d0 2024-06-04T04:55:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf89845700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf84113420 con 0x7fdf84101920 2024-06-04T04:55:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf89845700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdf84113720 con 0x7fdf84109a00 2024-06-04T04:55:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf827fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf841022d0 0x7fdf84103670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf827fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf841022d0 0x7fdf84103670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34512/0 (socket says 172.21.15.70:34512) 2024-06-04T04:55:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf827fc700 1 -- 172.21.15.70:0/2147857850 learned_addr learned my addr 172.21.15.70:0/2147857850 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf82ffd700 1 --2- 172.21.15.70:0/2147857850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf84101920 0x7fdf84103130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf837fe700 1 --2- 172.21.15.70:0/2147857850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf84109a00 0x7fdf84103bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf827fc700 1 -- 172.21.15.70:0/2147857850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf84109a00 msgr2=0x7fdf84103bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf827fc700 1 --2- 172.21.15.70:0/2147857850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf84109a00 0x7fdf84103bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf827fc700 1 -- 172.21.15.70:0/2147857850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf84101920 msgr2=0x7fdf84103130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf827fc700 1 --2- 172.21.15.70:0/2147857850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf84101920 0x7fdf84103130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf827fc700 1 -- 172.21.15.70:0/2147857850 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf841a8020 con 0x7fdf841022d0 2024-06-04T04:55:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.441+0000 7fdf837fe700 1 --2- 172.21.15.70:0/2147857850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf84109a00 0x7fdf84103bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:55:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.442+0000 7fdf827fc700 1 --2- 172.21.15.70:0/2147857850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf841022d0 0x7fdf84103670 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdf7800b580 tx=0x7fdf78004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.442+0000 7fdf88843700 1 -- 172.21.15.70:0/2147857850 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7800f8a0 con 0x7fdf841022d0 2024-06-04T04:55:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.442+0000 7fdf88843700 1 -- 172.21.15.70:0/2147857850 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdf78004830 con 0x7fdf841022d0 2024-06-04T04:55:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.442+0000 7fdf89845700 1 -- 172.21.15.70:0/2147857850 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf841a82b0 con 0x7fdf841022d0 2024-06-04T04:55:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.442+0000 7fdf88843700 1 -- 172.21.15.70:0/2147857850 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78005350 con 0x7fdf841022d0 2024-06-04T04:55:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.442+0000 7fdf89845700 1 -- 172.21.15.70:0/2147857850 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf841a86a0 con 0x7fdf841022d0 2024-06-04T04:55:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.443+0000 7fdf89845700 1 -- 172.21.15.70:0/2147857850 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf84102a50 con 0x7fdf841022d0 2024-06-04T04:55:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.444+0000 7fdf88843700 1 -- 172.21.15.70:0/2147857850 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fdf78029060 con 0x7fdf841022d0 2024-06-04T04:55:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.445+0000 7fdf88843700 1 --2- 172.21.15.70:0/2147857850 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fdf6c0b2880 0x7fdf6c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.445+0000 7fdf82ffd700 1 --2- 172.21.15.70:0/2147857850 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fdf6c0b2880 0x7fdf6c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.445+0000 7fdf88843700 1 -- 172.21.15.70:0/2147857850 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fdf780e2f70 con 0x7fdf841022d0 2024-06-04T04:55:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.445+0000 7fdf82ffd700 1 --2- 172.21.15.70:0/2147857850 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fdf6c0b2880 0x7fdf6c0b4d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400ac50 tx=0x7fdf7400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:52.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.448+0000 7fdf88843700 1 -- 172.21.15.70:0/2147857850 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdf78013070 con 0x7fdf841022d0 2024-06-04T04:55:52.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.701+0000 7fdf89845700 1 -- 172.21.15.70:0/2147857850 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fdf8402e560 con 0x7fdf841022d0 2024-06-04T04:55:52.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.702+0000 7fdf88843700 1 -- 172.21.15.70:0/2147857850 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fdf780aad00 con 0x7fdf841022d0 2024-06-04T04:55:52.705 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:55:52.705 INFO:teuthology.orchestra.run.smithi070.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-06-04T04:55:52.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.704+0000 7fdf89845700 1 -- 172.21.15.70:0/2147857850 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fdf6c0b2880 msgr2=0x7fdf6c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:52.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.704+0000 7fdf89845700 1 --2- 172.21.15.70:0/2147857850 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fdf6c0b2880 0x7fdf6c0b4d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400ac50 tx=0x7fdf7400a380 comp rx=0 tx=0).stop 2024-06-04T04:55:52.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.704+0000 7fdf89845700 1 -- 172.21.15.70:0/2147857850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf841022d0 msgr2=0x7fdf84103670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:52.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.704+0000 7fdf89845700 1 --2- 172.21.15.70:0/2147857850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf841022d0 0x7fdf84103670 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdf7800b580 tx=0x7fdf78004790 comp rx=0 tx=0).stop 2024-06-04T04:55:52.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.704+0000 7fdf89845700 1 -- 172.21.15.70:0/2147857850 shutdown_connections 2024-06-04T04:55:52.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.704+0000 7fdf89845700 1 --2- 172.21.15.70:0/2147857850 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fdf6c0b2880 0x7fdf6c0b4d40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:52.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.704+0000 7fdf89845700 1 --2- 172.21.15.70:0/2147857850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf84109a00 0x7fdf84103bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:52.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.704+0000 7fdf89845700 1 --2- 172.21.15.70:0/2147857850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf84101920 0x7fdf84103130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:52.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.704+0000 7fdf89845700 1 --2- 172.21.15.70:0/2147857850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf841022d0 0x7fdf84103670 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:52.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.704+0000 7fdf89845700 1 -- 172.21.15.70:0/2147857850 >> 172.21.15.70:0/2147857850 conn(0x7fdf84078640 msgr2=0x7fdf841002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:52.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.704+0000 7fdf89845700 1 -- 172.21.15.70:0/2147857850 shutdown_connections 2024-06-04T04:55:52.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:52.704+0000 7fdf89845700 1 -- 172.21.15.70:0/2147857850 wait complete. 2024-06-04T04:55:53.416 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-06-04T04:55:53.417 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:55:53.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:53 smithi070 ceph-mon[32631]: pgmap v358: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-06-04T04:55:53.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:53 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2147857850' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-04T04:55:53.657 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:53 smithi112 ceph-mon[36453]: pgmap v358: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-06-04T04:55:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:53 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2147857850' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-04T04:55:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:53 smithi098 ceph-mon[35569]: pgmap v358: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-06-04T04:55:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:53 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2147857850' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-04T04:55:55.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.308+0000 7fca0065b700 1 -- 172.21.15.70:0/3244583654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9f81037f0 msgr2=0x7fc9f8103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:55.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.308+0000 7fca0065b700 1 --2- 172.21.15.70:0/3244583654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9f81037f0 0x7fc9f8103bd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc9f40099c0 tx=0x7fc9f4009cd0 comp rx=0 tx=0).stop 2024-06-04T04:55:55.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.308+0000 7fca0065b700 1 -- 172.21.15.70:0/3244583654 shutdown_connections 2024-06-04T04:55:55.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.308+0000 7fca0065b700 1 --2- 172.21.15.70:0/3244583654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9f81088f0 0x7fc9f8110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:55.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.308+0000 7fca0065b700 1 --2- 172.21.15.70:0/3244583654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9f81037f0 0x7fc9f8103bd0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:55.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.308+0000 7fca0065b700 1 --2- 172.21.15.70:0/3244583654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9f81041a0 0x7fc9f81081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:55.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.308+0000 7fca0065b700 1 -- 172.21.15.70:0/3244583654 >> 172.21.15.70:0/3244583654 conn(0x7fc9f807a6a0 msgr2=0x7fc9f807aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:55.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.308+0000 7fca0065b700 1 -- 172.21.15.70:0/3244583654 shutdown_connections 2024-06-04T04:55:55.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.308+0000 7fca0065b700 1 -- 172.21.15.70:0/3244583654 wait complete. 2024-06-04T04:55:55.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.308+0000 7fca0065b700 1 Processor -- start 2024-06-04T04:55:55.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.308+0000 7fca0065b700 1 -- start start 2024-06-04T04:55:55.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fca0065b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9f81037f0 0x7fc9f8075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:55.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fca0065b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9f81041a0 0x7fc9f8075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:55.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fca0065b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9f81088f0 0x7fc9f80762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:55.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fca0065b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f81135f0 con 0x7fc9f81037f0 2024-06-04T04:55:55.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fca0065b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f8113470 con 0x7fc9f81041a0 2024-06-04T04:55:55.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fca0065b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f8113770 con 0x7fc9f81088f0 2024-06-04T04:55:55.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fc9fe3f7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9f81037f0 0x7fc9f8075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:55.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fc9fe3f7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9f81037f0 0x7fc9f8075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34532/0 (socket says 172.21.15.70:34532) 2024-06-04T04:55:55.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fc9fe3f7700 1 -- 172.21.15.70:0/2677884030 learned_addr learned my addr 172.21.15.70:0/2677884030 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:55.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fc9febf8700 1 --2- 172.21.15.70:0/2677884030 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9f81088f0 0x7fc9f80762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:55.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fc9fdbf6700 1 --2- 172.21.15.70:0/2677884030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9f81041a0 0x7fc9f8075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:55.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fc9fe3f7700 1 -- 172.21.15.70:0/2677884030 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9f81088f0 msgr2=0x7fc9f80762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:55.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fc9fe3f7700 1 --2- 172.21.15.70:0/2677884030 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9f81088f0 0x7fc9f80762c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:55.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fc9fe3f7700 1 -- 172.21.15.70:0/2677884030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9f81041a0 msgr2=0x7fc9f8075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:55.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fc9fe3f7700 1 --2- 172.21.15.70:0/2677884030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9f81041a0 0x7fc9f8075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:55.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fc9fe3f7700 1 -- 172.21.15.70:0/2677884030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f81a8070 con 0x7fc9f81037f0 2024-06-04T04:55:55.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fc9fdbf6700 1 --2- 172.21.15.70:0/2677884030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9f81041a0 0x7fc9f8075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:55:55.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fc9fe3f7700 1 --2- 172.21.15.70:0/2677884030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9f81037f0 0x7fc9f8075840 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4005f50 tx=0x7fc9f400f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:55.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fc9ef7fe700 1 -- 172.21.15.70:0/2677884030 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4027070 con 0x7fc9f81037f0 2024-06-04T04:55:55.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fca0065b700 1 -- 172.21.15.70:0/2677884030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f81a8300 con 0x7fc9f81037f0 2024-06-04T04:55:55.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fc9ef7fe700 1 -- 172.21.15.70:0/2677884030 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9f400fdd0 con 0x7fc9f81037f0 2024-06-04T04:55:55.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fc9ef7fe700 1 -- 172.21.15.70:0/2677884030 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f40173f0 con 0x7fc9f81037f0 2024-06-04T04:55:55.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.309+0000 7fca0065b700 1 -- 172.21.15.70:0/2677884030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f81a87b0 con 0x7fc9f81037f0 2024-06-04T04:55:55.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.310+0000 7fca0065b700 1 -- 172.21.15.70:0/2677884030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9f810bb30 con 0x7fc9f81037f0 2024-06-04T04:55:55.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.312+0000 7fc9ef7fe700 1 -- 172.21.15.70:0/2677884030 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fc9f402a070 con 0x7fc9f81037f0 2024-06-04T04:55:55.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.312+0000 7fc9ef7fe700 1 --2- 172.21.15.70:0/2677884030 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc9e40b2880 0x7fc9e40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:55.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.312+0000 7fc9fdbf6700 1 --2- 172.21.15.70:0/2677884030 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc9e40b2880 0x7fc9e40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:55.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.312+0000 7fc9ef7fe700 1 -- 172.21.15.70:0/2677884030 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fc9f40e2fd0 con 0x7fc9f81037f0 2024-06-04T04:55:55.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.313+0000 7fc9fdbf6700 1 --2- 172.21.15.70:0/2677884030 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc9e40b2880 0x7fc9e40b4d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8005fd0 tx=0x7fc9e8005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:55.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.317+0000 7fc9ef7fe700 1 -- 172.21.15.70:0/2677884030 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc9f405fdf0 con 0x7fc9f81037f0 2024-06-04T04:55:55.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.566+0000 7fca0065b700 1 -- 172.21.15.70:0/2677884030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9f810b510 con 0x7fc9f81037f0 2024-06-04T04:55:55.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.566+0000 7fc9ef7fe700 1 -- 172.21.15.70:0/2677884030 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11929 (secure 0 0 0) 0x7fc9f40a4e20 con 0x7fc9f81037f0 2024-06-04T04:55:55.569 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:55:55.569 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:53:56.441682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T04:55:55.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.569+0000 7fca0065b700 1 -- 172.21.15.70:0/2677884030 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc9e40b2880 msgr2=0x7fc9e40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:55.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.569+0000 7fca0065b700 1 --2- 172.21.15.70:0/2677884030 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc9e40b2880 0x7fc9e40b4d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8005fd0 tx=0x7fc9e8005ee0 comp rx=0 tx=0).stop 2024-06-04T04:55:55.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.569+0000 7fca0065b700 1 -- 172.21.15.70:0/2677884030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9f81037f0 msgr2=0x7fc9f8075840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:55.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.569+0000 7fca0065b700 1 --2- 172.21.15.70:0/2677884030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9f81037f0 0x7fc9f8075840 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4005f50 tx=0x7fc9f400f460 comp rx=0 tx=0).stop 2024-06-04T04:55:55.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.569+0000 7fca0065b700 1 -- 172.21.15.70:0/2677884030 shutdown_connections 2024-06-04T04:55:55.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.569+0000 7fca0065b700 1 --2- 172.21.15.70:0/2677884030 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fc9e40b2880 0x7fc9e40b4d40 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:55.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.569+0000 7fca0065b700 1 --2- 172.21.15.70:0/2677884030 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9f81088f0 0x7fc9f80762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:55.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.569+0000 7fca0065b700 1 --2- 172.21.15.70:0/2677884030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9f81037f0 0x7fc9f8075840 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:55.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.569+0000 7fca0065b700 1 --2- 172.21.15.70:0/2677884030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9f81041a0 0x7fc9f8075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:55.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.569+0000 7fca0065b700 1 -- 172.21.15.70:0/2677884030 >> 172.21.15.70:0/2677884030 conn(0x7fc9f807a6a0 msgr2=0x7fc9f8101970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:55.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.569+0000 7fca0065b700 1 -- 172.21.15.70:0/2677884030 shutdown_connections 2024-06-04T04:55:55.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:55.569+0000 7fca0065b700 1 -- 172.21.15.70:0/2677884030 wait complete. 2024-06-04T04:55:55.572 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 28 2024-06-04T04:55:55.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:55 smithi070 ceph-mon[32631]: pgmap v359: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-06-04T04:55:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:55 smithi112 ceph-mon[36453]: pgmap v359: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-06-04T04:55:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:55 smithi098 ceph-mon[35569]: pgmap v359: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-06-04T04:55:56.264 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:55:56.504 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:56.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:56 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2677884030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:55:56.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:56 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2677884030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:55:56.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:56 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2677884030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:55:57.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:57 smithi070 ceph-mon[32631]: pgmap v360: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-06-04T04:55:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:57 smithi112 ceph-mon[36453]: pgmap v360: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-06-04T04:55:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:57 smithi098 ceph-mon[35569]: pgmap v360: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-06-04T04:55:58.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.138+0000 7ffb8794f700 1 -- 172.21.15.70:0/1223578798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb80102160 msgr2=0x7ffb80102540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:58.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.138+0000 7ffb8794f700 1 --2- 172.21.15.70:0/1223578798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb80102160 0x7ffb80102540 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00b0a0 tx=0x7ffb7c00b3b0 comp rx=0 tx=0).stop 2024-06-04T04:55:58.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.138+0000 7ffb8794f700 1 -- 172.21.15.70:0/1223578798 shutdown_connections 2024-06-04T04:55:58.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.138+0000 7ffb8794f700 1 --2- 172.21.15.70:0/1223578798 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb80107000 0x7ffb8010eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:58.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.138+0000 7ffb8794f700 1 --2- 172.21.15.70:0/1223578798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb80102160 0x7ffb80102540 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:58.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.138+0000 7ffb8794f700 1 --2- 172.21.15.70:0/1223578798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb80102b10 0x7ffb801068d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:58.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.138+0000 7ffb8794f700 1 -- 172.21.15.70:0/1223578798 >> 172.21.15.70:0/1223578798 conn(0x7ffb80077780 msgr2=0x7ffb80077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:58.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.138+0000 7ffb8794f700 1 -- 172.21.15.70:0/1223578798 shutdown_connections 2024-06-04T04:55:58.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.138+0000 7ffb8794f700 1 -- 172.21.15.70:0/1223578798 wait complete. 2024-06-04T04:55:58.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb8794f700 1 Processor -- start 2024-06-04T04:55:58.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb8794f700 1 -- start start 2024-06-04T04:55:58.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb8794f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb80102160 0x7ffb8019a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:58.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb8794f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb80102b10 0x7ffb8019af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:58.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb8794f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb80107000 0x7ffb801a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:58.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb8794f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffb80111d10 con 0x7ffb80102160 2024-06-04T04:55:58.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb8794f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffb80111b90 con 0x7ffb80102b10 2024-06-04T04:55:58.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb8794f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ffb80111e90 con 0x7ffb80107000 2024-06-04T04:55:58.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb856eb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb80102160 0x7ffb8019a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:58.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb856eb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb80102160 0x7ffb8019a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53732/0 (socket says 172.21.15.70:53732) 2024-06-04T04:55:58.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb856eb700 1 -- 172.21.15.70:0/821607563 learned_addr learned my addr 172.21.15.70:0/821607563 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:55:58.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb84eea700 1 --2- 172.21.15.70:0/821607563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb80102b10 0x7ffb8019af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:58.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb85eec700 1 --2- 172.21.15.70:0/821607563 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb80107000 0x7ffb801a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:58.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb856eb700 1 -- 172.21.15.70:0/821607563 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb80107000 msgr2=0x7ffb801a5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:58.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb856eb700 1 --2- 172.21.15.70:0/821607563 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb80107000 0x7ffb801a5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:58.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb856eb700 1 -- 172.21.15.70:0/821607563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb80102b10 msgr2=0x7ffb8019af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:58.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb856eb700 1 --2- 172.21.15.70:0/821607563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb80102b10 0x7ffb8019af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:58.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb856eb700 1 -- 172.21.15.70:0/821607563 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb801a5fb0 con 0x7ffb80102160 2024-06-04T04:55:58.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb85eec700 1 --2- 172.21.15.70:0/821607563 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb80107000 0x7ffb801a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:55:58.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.139+0000 7ffb84eea700 1 --2- 172.21.15.70:0/821607563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb80102b10 0x7ffb8019af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:55:58.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.140+0000 7ffb856eb700 1 --2- 172.21.15.70:0/821607563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb80102160 0x7ffb8019a9f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c006130 tx=0x7ffb7c00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:58.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.140+0000 7ffb767fc700 1 -- 172.21.15.70:0/821607563 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c021030 con 0x7ffb80102160 2024-06-04T04:55:58.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.140+0000 7ffb767fc700 1 -- 172.21.15.70:0/821607563 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb7c007bb0 con 0x7ffb80102160 2024-06-04T04:55:58.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.140+0000 7ffb8794f700 1 -- 172.21.15.70:0/821607563 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb801a6180 con 0x7ffb80102160 2024-06-04T04:55:58.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.140+0000 7ffb767fc700 1 -- 172.21.15.70:0/821607563 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c004730 con 0x7ffb80102160 2024-06-04T04:55:58.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.140+0000 7ffb8794f700 1 -- 172.21.15.70:0/821607563 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb801a6630 con 0x7ffb80102160 2024-06-04T04:55:58.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.141+0000 7ffb8794f700 1 -- 172.21.15.70:0/821607563 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb8006a230 con 0x7ffb80102160 2024-06-04T04:55:58.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.142+0000 7ffb767fc700 1 -- 172.21.15.70:0/821607563 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7ffb7c024070 con 0x7ffb80102160 2024-06-04T04:55:58.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.143+0000 7ffb767fc700 1 --2- 172.21.15.70:0/821607563 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ffb6c0b2880 0x7ffb6c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:55:58.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.143+0000 7ffb84eea700 1 --2- 172.21.15.70:0/821607563 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ffb6c0b2880 0x7ffb6c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:55:58.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.143+0000 7ffb767fc700 1 -- 172.21.15.70:0/821607563 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7ffb7c00f020 con 0x7ffb80102160 2024-06-04T04:55:58.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.143+0000 7ffb84eea700 1 --2- 172.21.15.70:0/821607563 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ffb6c0b2880 0x7ffb6c0b4d40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ffb80105df0 tx=0x7ffb70008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:55:58.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.146+0000 7ffb767fc700 1 -- 172.21.15.70:0/821607563 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ffb7c0ab2b0 con 0x7ffb80102160 2024-06-04T04:55:58.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.397+0000 7ffb8794f700 1 -- 172.21.15.70:0/821607563 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb801a6960 con 0x7ffb80102160 2024-06-04T04:55:58.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.397+0000 7ffb767fc700 1 -- 172.21.15.70:0/821607563 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11929 (secure 0 0 0) 0x7ffb7c0b32a0 con 0x7ffb80102160 2024-06-04T04:55:58.400 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:55:58.400 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:53:56.441682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T04:55:58.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.399+0000 7ffb8794f700 1 -- 172.21.15.70:0/821607563 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ffb6c0b2880 msgr2=0x7ffb6c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:58.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.400+0000 7ffb8794f700 1 --2- 172.21.15.70:0/821607563 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ffb6c0b2880 0x7ffb6c0b4d40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ffb80105df0 tx=0x7ffb70008040 comp rx=0 tx=0).stop 2024-06-04T04:55:58.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.400+0000 7ffb8794f700 1 -- 172.21.15.70:0/821607563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb80102160 msgr2=0x7ffb8019a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:55:58.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.400+0000 7ffb8794f700 1 --2- 172.21.15.70:0/821607563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb80102160 0x7ffb8019a9f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c006130 tx=0x7ffb7c00cf80 comp rx=0 tx=0).stop 2024-06-04T04:55:58.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.400+0000 7ffb8794f700 1 -- 172.21.15.70:0/821607563 shutdown_connections 2024-06-04T04:55:58.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.400+0000 7ffb8794f700 1 --2- 172.21.15.70:0/821607563 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7ffb6c0b2880 0x7ffb6c0b4d40 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:58.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.400+0000 7ffb8794f700 1 --2- 172.21.15.70:0/821607563 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb80107000 0x7ffb801a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:58.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.400+0000 7ffb8794f700 1 --2- 172.21.15.70:0/821607563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb80102160 0x7ffb8019a9f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:58.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.400+0000 7ffb8794f700 1 --2- 172.21.15.70:0/821607563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb80102b10 0x7ffb8019af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:55:58.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.400+0000 7ffb8794f700 1 -- 172.21.15.70:0/821607563 >> 172.21.15.70:0/821607563 conn(0x7ffb80077780 msgr2=0x7ffb80100ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:55:58.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.400+0000 7ffb8794f700 1 -- 172.21.15.70:0/821607563 shutdown_connections 2024-06-04T04:55:58.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:55:58.400+0000 7ffb8794f700 1 -- 172.21.15.70:0/821607563 wait complete. 2024-06-04T04:55:58.403 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 28 2024-06-04T04:55:59.190 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-04T04:55:59.431 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:55:59.553 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:59 smithi070 ceph-mon[32631]: pgmap v361: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-06-04T04:55:59.553 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:55:59 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/821607563' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:55:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:59 smithi112 ceph-mon[36453]: pgmap v361: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-06-04T04:55:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:55:59 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/821607563' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:55:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:59 smithi098 ceph-mon[35569]: pgmap v361: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-06-04T04:55:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:55:59 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/821607563' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:56:01.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.098+0000 7f59fb212700 1 -- 172.21.15.70:0/473589187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59f4108520 msgr2=0x7f59f4108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:01.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.098+0000 7f59fb212700 1 --2- 172.21.15.70:0/473589187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59f4108520 0x7f59f4108900 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f59e8009990 tx=0x7f59e8009ca0 comp rx=0 tx=0).stop 2024-06-04T04:56:01.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.098+0000 7f59fb212700 1 -- 172.21.15.70:0/473589187 shutdown_connections 2024-06-04T04:56:01.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.098+0000 7f59fb212700 1 --2- 172.21.15.70:0/473589187 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59f4102e00 0x7f59f410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:01.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.098+0000 7f59fb212700 1 --2- 172.21.15.70:0/473589187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59f4108520 0x7f59f4108900 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:01.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.098+0000 7f59fb212700 1 --2- 172.21.15.70:0/473589187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59f4102440 0x7f59f41028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:01.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.098+0000 7f59fb212700 1 -- 172.21.15.70:0/473589187 >> 172.21.15.70:0/473589187 conn(0x7f59f40793f0 msgr2=0x7f59f4079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:01.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.098+0000 7f59fb212700 1 -- 172.21.15.70:0/473589187 shutdown_connections 2024-06-04T04:56:01.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.098+0000 7f59fb212700 1 -- 172.21.15.70:0/473589187 wait complete. 2024-06-04T04:56:01.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59fb212700 1 Processor -- start 2024-06-04T04:56:01.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59fb212700 1 -- start start 2024-06-04T04:56:01.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59fb212700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59f4102440 0x7f59f4104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:01.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59fb212700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59f4102e00 0x7f59f41045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:01.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59fb212700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59f4108520 0x7f59f4104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:01.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59fb212700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f59f4112500 con 0x7f59f4102e00 2024-06-04T04:56:01.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59fb212700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59f4112380 con 0x7f59f4108520 2024-06-04T04:56:01.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59fb212700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f59f4112680 con 0x7f59f4102440 2024-06-04T04:56:01.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59f3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59f4102e00 0x7f59f41045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:01.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59f3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59f4102e00 0x7f59f41045c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53762/0 (socket says 172.21.15.70:53762) 2024-06-04T04:56:01.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59f3fff700 1 -- 172.21.15.70:0/3072557207 learned_addr learned my addr 172.21.15.70:0/3072557207 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:01.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59f8fae700 1 --2- 172.21.15.70:0/3072557207 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59f4102440 0x7f59f4104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:01.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59f3fff700 1 -- 172.21.15.70:0/3072557207 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59f4102440 msgr2=0x7f59f4104080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:01.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59f3fff700 1 --2- 172.21.15.70:0/3072557207 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59f4102440 0x7f59f4104080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:01.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59f3fff700 1 -- 172.21.15.70:0/3072557207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59f4108520 msgr2=0x7f59f4104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59f97af700 1 --2- 172.21.15.70:0/3072557207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59f4108520 0x7f59f4104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59f3fff700 1 --2- 172.21.15.70:0/3072557207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59f4108520 0x7f59f4104b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.099+0000 7f59f3fff700 1 -- 172.21.15.70:0/3072557207 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f41a8120 con 0x7f59f4102e00 2024-06-04T04:56:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.100+0000 7f59f3fff700 1 --2- 172.21.15.70:0/3072557207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59f4102e00 0x7f59f41045c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f59e000d980 tx=0x7f59e000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.100+0000 7f59f1ffb700 1 -- 172.21.15.70:0/3072557207 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e0016070 con 0x7f59f4102e00 2024-06-04T04:56:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.100+0000 7f59f1ffb700 1 -- 172.21.15.70:0/3072557207 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f59e0004020 con 0x7f59f4102e00 2024-06-04T04:56:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.100+0000 7f59f1ffb700 1 -- 172.21.15.70:0/3072557207 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e0004bb0 con 0x7f59f4102e00 2024-06-04T04:56:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.100+0000 7f59fb212700 1 -- 172.21.15.70:0/3072557207 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59f41a8410 con 0x7f59f4102e00 2024-06-04T04:56:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.100+0000 7f59fb212700 1 -- 172.21.15.70:0/3072557207 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59f41a89a0 con 0x7f59f4102e00 2024-06-04T04:56:01.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.101+0000 7f59fb212700 1 -- 172.21.15.70:0/3072557207 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59f406a230 con 0x7f59f4102e00 2024-06-04T04:56:01.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.102+0000 7f59f1ffb700 1 -- 172.21.15.70:0/3072557207 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f59e000bb60 con 0x7f59f4102e00 2024-06-04T04:56:01.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.103+0000 7f59f1ffb700 1 --2- 172.21.15.70:0/3072557207 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f59e40b2900 0x7f59e40b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:01.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.103+0000 7f59f8fae700 1 --2- 172.21.15.70:0/3072557207 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f59e40b2900 0x7f59e40b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:01.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.103+0000 7f59f1ffb700 1 -- 172.21.15.70:0/3072557207 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f59e00e2540 con 0x7f59f4102e00 2024-06-04T04:56:01.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.104+0000 7f59f8fae700 1 --2- 172.21.15.70:0/3072557207 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f59e40b2900 0x7f59e40b4dc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f59e8005cd0 tx=0x7f59e8005c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:01.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.107+0000 7f59f1ffb700 1 -- 172.21.15.70:0/3072557207 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f59e001f020 con 0x7f59f4102e00 2024-06-04T04:56:01.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.313+0000 7f59fb212700 1 -- 172.21.15.70:0/3072557207 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f59f41a8c00 con 0x7f59f4102e00 2024-06-04T04:56:01.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.315+0000 7f59f1ffb700 1 -- 172.21.15.70:0/3072557207 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22708 (secure 0 0 0) 0x7f59e00aa240 con 0x7f59f4102e00 2024-06-04T04:56:01.317 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:56:01.318 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":81,"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","created":"2024-06-04T04:45:22.297536+0000","modified":"2024-06-04T04:54:19.502410+0000","last_up_change":"2024-06-04T04:52:34.388708+0000","last_in_change":"2024-06-04T04:52:14.197780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-04T04:48:43.724356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-04T04:53:31.398455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-04T04:53:35.298617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ff491ea2-1906-48e4-883e-bc78e778260e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6803","nonce":824119100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6805","nonce":824119100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6809","nonce":824119100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6807","nonce":824119100}]},"public_addr":"172.21.15.70:6803/824119100","cluster_addr":"172.21.15.70:6805/824119100","heartbeat_back_addr":"172.21.15.70:6809/824119100","heartbeat_front_addr":"172.21.15.70:6807/824119100","state":["exists","up"]},{"osd":1,"uuid":"bd5e4806-535b-49b5-ba6b-93228da35a84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6801","nonce":2744598742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6803","nonce":2744598742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6807","nonce":2744598742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6805","nonce":2744598742}]},"public_addr":"172.21.15.98:6801/2744598742","cluster_addr":"172.21.15.98:6803/2744598742","heartbeat_back_addr":"172.21.15.98:6807/2744598742","heartbeat_front_addr":"172.21.15.98:6805/2744598742","state":["exists","up"]},{"osd":2,"uuid":"8456fb55-910a-4a59-b84b-2c0f68346ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6801","nonce":1328098370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6803","nonce":1328098370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6807","nonce":1328098370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6805","nonce":1328098370}]},"public_addr":"172.21.15.112:6801/1328098370","cluster_addr":"172.21.15.112:6803/1328098370","heartbeat_back_addr":"172.21.15.112:6807/1328098370","heartbeat_front_addr":"172.21.15.112:6805/1328098370","state":["exists","up"]},{"osd":3,"uuid":"c8fbc440-bf70-42d3-b01e-777ccfa7900b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6811","nonce":4041566346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6813","nonce":4041566346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6817","nonce":4041566346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6815","nonce":4041566346}]},"public_addr":"172.21.15.70:6811/4041566346","cluster_addr":"172.21.15.70:6813/4041566346","heartbeat_back_addr":"172.21.15.70:6817/4041566346","heartbeat_front_addr":"172.21.15.70:6815/4041566346","state":["exists","up"]},{"osd":4,"uuid":"549c8c70-8be8-46ae-bcb6-0fde32b40cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6809","nonce":3652775814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6811","nonce":3652775814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6815","nonce":3652775814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6813","nonce":3652775814}]},"public_addr":"172.21.15.98:6809/3652775814","cluster_addr":"172.21.15.98:6811/3652775814","heartbeat_back_addr":"172.21.15.98:6815/3652775814","heartbeat_front_addr":"172.21.15.98:6813/3652775814","state":["exists","up"]},{"osd":5,"uuid":"51d86af1-0d6c-4ae0-b090-00414fc1bf7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6809","nonce":1857257899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6811","nonce":1857257899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6815","nonce":1857257899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6813","nonce":1857257899}]},"public_addr":"172.21.15.112:6809/1857257899","cluster_addr":"172.21.15.112:6811/1857257899","heartbeat_back_addr":"172.21.15.112:6815/1857257899","heartbeat_front_addr":"172.21.15.112:6813/1857257899","state":["exists","up"]},{"osd":6,"uuid":"526a8138-bfa8-419e-a5b5-e6b80f0ae004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6819","nonce":1264676496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6821","nonce":1264676496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6825","nonce":1264676496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6823","nonce":1264676496}]},"public_addr":"172.21.15.70:6819/1264676496","cluster_addr":"172.21.15.70:6821/1264676496","heartbeat_back_addr":"172.21.15.70:6825/1264676496","heartbeat_front_addr":"172.21.15.70:6823/1264676496","state":["exists","up"]},{"osd":7,"uuid":"1173f785-6ed1-47ef-9265-2e233ea9ee02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6817","nonce":709593020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6819","nonce":709593020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6823","nonce":709593020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6821","nonce":709593020}]},"public_addr":"172.21.15.98:6817/709593020","cluster_addr":"172.21.15.98:6819/709593020","heartbeat_back_addr":"172.21.15.98:6823/709593020","heartbeat_front_addr":"172.21.15.98:6821/709593020","state":["exists","up"]},{"osd":8,"uuid":"6424cc6e-24a7-4b81-9ce6-39203fecc7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6817","nonce":2250225413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6819","nonce":2250225413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6823","nonce":2250225413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6821","nonce":2250225413}]},"public_addr":"172.21.15.112:6817/2250225413","cluster_addr":"172.21.15.112:6819/2250225413","heartbeat_back_addr":"172.21.15.112:6823/2250225413","heartbeat_front_addr":"172.21.15.112:6821/2250225413","state":["exists","up"]},{"osd":9,"uuid":"8d666db8-8831-4d63-9e3c-a4bf2504149e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6827","nonce":3557688707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6829","nonce":3557688707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6833","nonce":3557688707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6831","nonce":3557688707}]},"public_addr":"172.21.15.70:6827/3557688707","cluster_addr":"172.21.15.70:6829/3557688707","heartbeat_back_addr":"172.21.15.70:6833/3557688707","heartbeat_front_addr":"172.21.15.70:6831/3557688707","state":["exists","up"]},{"osd":10,"uuid":"ecb925d2-0263-4dfc-a2fd-9e27856022d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6825","nonce":1896493718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6827","nonce":1896493718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6831","nonce":1896493718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6829","nonce":1896493718}]},"public_addr":"172.21.15.98:6825/1896493718","cluster_addr":"172.21.15.98:6827/1896493718","heartbeat_back_addr":"172.21.15.98:6831/1896493718","heartbeat_front_addr":"172.21.15.98:6829/1896493718","state":["exists","up"]},{"osd":11,"uuid":"c8e3230c-bc3a-40bd-8dd0-5fef56e4df97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6825","nonce":1400793137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6827","nonce":1400793137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6831","nonce":1400793137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6829","nonce":1400793137}]},"public_addr":"172.21.15.112:6825/1400793137","cluster_addr":"172.21.15.112:6827/1400793137","heartbeat_back_addr":"172.21.15.112:6831/1400793137","heartbeat_front_addr":"172.21.15.112:6829/1400793137","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:47:53.626608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:17.784505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:41.971969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:07.130699+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:31.615496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:56.898607+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:22.739991+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:48.319727+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:13.902063+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:40.389470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:06.142092+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:32.949383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/3442074771":"2024-06-05T04:46:17.636907+0000","172.21.15.70:6801/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/1442095493":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/2453733268":"2024-06-05T04:45:49.501864+0000","172.21.15.70:0/1369781447":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6801/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/915186456":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/860033006":"2024-06-05T04:45:49.501864+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-04T04:56:01.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.318+0000 7f59fb212700 1 -- 172.21.15.70:0/3072557207 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f59e40b2900 msgr2=0x7f59e40b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:01.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.318+0000 7f59fb212700 1 --2- 172.21.15.70:0/3072557207 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f59e40b2900 0x7f59e40b4dc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f59e8005cd0 tx=0x7f59e8005c40 comp rx=0 tx=0).stop 2024-06-04T04:56:01.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.318+0000 7f59fb212700 1 -- 172.21.15.70:0/3072557207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59f4102e00 msgr2=0x7f59f41045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:01.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.318+0000 7f59fb212700 1 --2- 172.21.15.70:0/3072557207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59f4102e00 0x7f59f41045c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f59e000d980 tx=0x7f59e000dd40 comp rx=0 tx=0).stop 2024-06-04T04:56:01.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.318+0000 7f59fb212700 1 -- 172.21.15.70:0/3072557207 shutdown_connections 2024-06-04T04:56:01.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.318+0000 7f59fb212700 1 --2- 172.21.15.70:0/3072557207 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f59e40b2900 0x7f59e40b4dc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:01.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.318+0000 7f59fb212700 1 --2- 172.21.15.70:0/3072557207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59f4108520 0x7f59f4104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:01.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.318+0000 7f59fb212700 1 --2- 172.21.15.70:0/3072557207 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59f4102440 0x7f59f4104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:01.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.318+0000 7f59fb212700 1 --2- 172.21.15.70:0/3072557207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59f4102e00 0x7f59f41045c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:01.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.318+0000 7f59fb212700 1 -- 172.21.15.70:0/3072557207 >> 172.21.15.70:0/3072557207 conn(0x7f59f40793f0 msgr2=0x7f59f406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:01.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.318+0000 7f59fb212700 1 -- 172.21.15.70:0/3072557207 shutdown_connections 2024-06-04T04:56:01.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:01.319+0000 7f59fb212700 1 -- 172.21.15.70:0/3072557207 wait complete. 2024-06-04T04:56:01.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:01 smithi070 ceph-mon[32631]: pgmap v362: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-06-04T04:56:01.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:01 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3072557207' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:56:01.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:01 smithi112 ceph-mon[36453]: pgmap v362: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-06-04T04:56:01.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:01 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3072557207' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:56:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:01 smithi098 ceph-mon[35569]: pgmap v362: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-06-04T04:56:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:01 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3072557207' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:56:02.035 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:56:02.274 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:03.457 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:03 smithi070 ceph-mon[32631]: pgmap v363: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-06-04T04:56:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:03 smithi112 ceph-mon[36453]: pgmap v363: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-06-04T04:56:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:03 smithi098 ceph-mon[35569]: pgmap v363: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-06-04T04:56:04.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.002+0000 7f8fde998700 1 -- 172.21.15.70:0/565551010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fd8108560 msgr2=0x7f8fd8108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:04.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.002+0000 7f8fde998700 1 --2- 172.21.15.70:0/565551010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fd8108560 0x7f8fd8108940 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80099c0 tx=0x7f8fc8009cd0 comp rx=0 tx=0).stop 2024-06-04T04:56:04.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.002+0000 7f8fde998700 1 -- 172.21.15.70:0/565551010 shutdown_connections 2024-06-04T04:56:04.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.002+0000 7f8fde998700 1 --2- 172.21.15.70:0/565551010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fd8102f30 0x7f8fd810f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:04.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.002+0000 7f8fde998700 1 --2- 172.21.15.70:0/565551010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fd8108560 0x7f8fd8108940 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:04.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.002+0000 7f8fde998700 1 --2- 172.21.15.70:0/565551010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fd8102570 0x7f8fd81029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:04.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.002+0000 7f8fde998700 1 -- 172.21.15.70:0/565551010 >> 172.21.15.70:0/565551010 conn(0x7f8fd8078350 msgr2=0x7f8fd8078760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:04.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.002+0000 7f8fde998700 1 -- 172.21.15.70:0/565551010 shutdown_connections 2024-06-04T04:56:04.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.002+0000 7f8fde998700 1 -- 172.21.15.70:0/565551010 wait complete. 2024-06-04T04:56:04.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.003+0000 7f8fde998700 1 Processor -- start 2024-06-04T04:56:04.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.003+0000 7f8fde998700 1 -- start start 2024-06-04T04:56:04.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.003+0000 7f8fde998700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fd8102570 0x7f8fd819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:04.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.003+0000 7f8fde998700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fd8102f30 0x7f8fd819fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:04.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.003+0000 7f8fde998700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fd8108560 0x7f8fd8199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:04.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.003+0000 7f8fde998700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd81124f0 con 0x7f8fd8102570 2024-06-04T04:56:04.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.003+0000 7f8fde998700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd8112370 con 0x7f8fd8102f30 2024-06-04T04:56:04.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.003+0000 7f8fde998700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd8112670 con 0x7f8fd8108560 2024-06-04T04:56:04.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.003+0000 7f8fd7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fd8102570 0x7f8fd819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:04.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.003+0000 7f8fd7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fd8102570 0x7f8fd819f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53768/0 (socket says 172.21.15.70:53768) 2024-06-04T04:56:04.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.003+0000 7f8fd7fff700 1 -- 172.21.15.70:0/1229913798 learned_addr learned my addr 172.21.15.70:0/1229913798 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:04.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.004+0000 7f8fd77fe700 1 --2- 172.21.15.70:0/1229913798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fd8102f30 0x7f8fd819fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:04.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.004+0000 7f8fdcf35700 1 --2- 172.21.15.70:0/1229913798 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fd8108560 0x7f8fd8199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:04.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.004+0000 7f8fd7fff700 1 -- 172.21.15.70:0/1229913798 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fd8108560 msgr2=0x7f8fd8199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:04.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.004+0000 7f8fd7fff700 1 --2- 172.21.15.70:0/1229913798 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fd8108560 0x7f8fd8199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:04.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.004+0000 7f8fd7fff700 1 -- 172.21.15.70:0/1229913798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fd8102f30 msgr2=0x7f8fd819fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:04.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.004+0000 7f8fd7fff700 1 --2- 172.21.15.70:0/1229913798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fd8102f30 0x7f8fd819fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:04.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.004+0000 7f8fd7fff700 1 -- 172.21.15.70:0/1229913798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd819a0f0 con 0x7f8fd8102570 2024-06-04T04:56:04.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.004+0000 7f8fd77fe700 1 --2- 172.21.15.70:0/1229913798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fd8102f30 0x7f8fd819fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:04.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.004+0000 7f8fd7fff700 1 --2- 172.21.15.70:0/1229913798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fd8102570 0x7f8fd819f780 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8000c00 tx=0x7f8fc800f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:04.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.004+0000 7f8fd57fa700 1 -- 172.21.15.70:0/1229913798 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc800f940 con 0x7f8fd8102570 2024-06-04T04:56:04.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.004+0000 7f8fd57fa700 1 -- 172.21.15.70:0/1229913798 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8fc8004830 con 0x7f8fd8102570 2024-06-04T04:56:04.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.004+0000 7f8fde998700 1 -- 172.21.15.70:0/1229913798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd819a380 con 0x7f8fd8102570 2024-06-04T04:56:04.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.004+0000 7f8fd57fa700 1 -- 172.21.15.70:0/1229913798 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80054b0 con 0x7f8fd8102570 2024-06-04T04:56:04.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.004+0000 7f8fde998700 1 -- 172.21.15.70:0/1229913798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd81a6650 con 0x7f8fd8102570 2024-06-04T04:56:04.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.005+0000 7f8fde998700 1 -- 172.21.15.70:0/1229913798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fd8103bd0 con 0x7f8fd8102570 2024-06-04T04:56:04.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.006+0000 7f8fd57fa700 1 -- 172.21.15.70:0/1229913798 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f8fc8029060 con 0x7f8fd8102570 2024-06-04T04:56:04.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.007+0000 7f8fd57fa700 1 --2- 172.21.15.70:0/1229913798 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8fc00b2880 0x7f8fc00b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:04.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.007+0000 7f8fd77fe700 1 --2- 172.21.15.70:0/1229913798 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8fc00b2880 0x7f8fc00b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:04.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.007+0000 7f8fd57fa700 1 -- 172.21.15.70:0/1229913798 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f8fc80e2fc0 con 0x7f8fd8102570 2024-06-04T04:56:04.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.007+0000 7f8fd77fe700 1 --2- 172.21.15.70:0/1229913798 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8fc00b2880 0x7f8fc00b4d40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8fd806cdf0 tx=0x7f8fcc005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:04.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.011+0000 7f8fd57fa700 1 -- 172.21.15.70:0/1229913798 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8fc8013070 con 0x7f8fd8102570 2024-06-04T04:56:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.258+0000 7f8fde998700 1 -- 172.21.15.70:0/1229913798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8fd81a6820 con 0x7f8fd8102570 2024-06-04T04:56:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.259+0000 7f8fd57fa700 1 -- 172.21.15.70:0/1229913798 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11929 (secure 0 0 0) 0x7f8fc80aad50 con 0x7f8fd8102570 2024-06-04T04:56:04.261 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:56:04.261 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:53:56.441682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T04:56:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.261+0000 7f8fde998700 1 -- 172.21.15.70:0/1229913798 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8fc00b2880 msgr2=0x7f8fc00b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.261+0000 7f8fde998700 1 --2- 172.21.15.70:0/1229913798 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8fc00b2880 0x7f8fc00b4d40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8fd806cdf0 tx=0x7f8fcc005f50 comp rx=0 tx=0).stop 2024-06-04T04:56:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.261+0000 7f8fde998700 1 -- 172.21.15.70:0/1229913798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fd8102570 msgr2=0x7f8fd819f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.261+0000 7f8fde998700 1 --2- 172.21.15.70:0/1229913798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fd8102570 0x7f8fd819f780 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8000c00 tx=0x7f8fc800f510 comp rx=0 tx=0).stop 2024-06-04T04:56:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.261+0000 7f8fde998700 1 -- 172.21.15.70:0/1229913798 shutdown_connections 2024-06-04T04:56:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.261+0000 7f8fde998700 1 --2- 172.21.15.70:0/1229913798 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f8fc00b2880 0x7f8fc00b4d40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.261+0000 7f8fde998700 1 --2- 172.21.15.70:0/1229913798 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fd8108560 0x7f8fd8199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.261+0000 7f8fde998700 1 --2- 172.21.15.70:0/1229913798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fd8102570 0x7f8fd819f780 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.261+0000 7f8fde998700 1 --2- 172.21.15.70:0/1229913798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fd8102f30 0x7f8fd819fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.261+0000 7f8fde998700 1 -- 172.21.15.70:0/1229913798 >> 172.21.15.70:0/1229913798 conn(0x7f8fd8078350 msgr2=0x7f8fd81067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.261+0000 7f8fde998700 1 -- 172.21.15.70:0/1229913798 shutdown_connections 2024-06-04T04:56:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:04.261+0000 7f8fde998700 1 -- 172.21.15.70:0/1229913798 wait complete. 2024-06-04T04:56:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 28 2024-06-04T04:56:04.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:04 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1229913798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:56:04.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:04 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1229913798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:56:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:04 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1229913798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:56:04.995 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-06-04T04:53:38.237993+0000', 'modified': '2024-06-04T04:53:56.441682+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24448, 'mds_1': 24442, 'mds_2': 24440, 'mds_3': 24436, 'mds_4': 24410}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14580': {'gid': 14580, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6839/3817259285', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6838', 'nonce': 3817259285}, {'type': 'v1', 'addr': '172.21.15.70:6839', 'nonce': 3817259285}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14592': {'gid': 14592, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6841/2748767095', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6840', 'nonce': 2748767095}, {'type': 'v1', 'addr': '172.21.15.70:6841', 'nonce': 2748767095}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24371': {'gid': 24371, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6833/1896647585', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6832', 'nonce': 1896647585}, {'type': 'v1', 'addr': '172.21.15.98:6833', 'nonce': 1896647585}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24386': {'gid': 24386, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6835/3148471566', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 3148471566}, {'type': 'v1', 'addr': '172.21.15.98:6835', 'nonce': 3148471566}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24400': {'gid': 24400, 'name': 'l', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6833/1359739019', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6832', 'nonce': 1359739019}, {'type': 'v1', 'addr': '172.21.15.112:6833', 'nonce': 1359739019}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24410': {'gid': 24410, 'name': 'h', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.98:6837/3160044154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6836', 'nonce': 3160044154}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3160044154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24436': {'gid': 24436, 'name': 'f', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6835/242926435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6834', 'nonce': 242926435}, {'type': 'v1', 'addr': '172.21.15.112:6835', 'nonce': 242926435}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'b', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.98:6839/1361276220', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6838', 'nonce': 1361276220}, {'type': 'v1', 'addr': '172.21.15.98:6839', 'nonce': 1361276220}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'c', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.112:6837/3596492666', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6836', 'nonce': 3596492666}, {'type': 'v1', 'addr': '172.21.15.112:6837', 'nonce': 3596492666}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24448': {'gid': 24448, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.112:6839/2996116087', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6838', 'nonce': 2996116087}, {'type': 'v1', 'addr': '172.21.15.112:6839', 'nonce': 2996116087}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24448, 'qdb_cluster': [24410, 24436, 24440, 24442, 24448]} 2024-06-04T04:56:04.995 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-06-04T04:56:04.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.h status 2024-06-04T04:56:05.237 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:05.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:05 smithi070 ceph-mon[32631]: pgmap v364: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-06-04T04:56:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:05 smithi112 ceph-mon[36453]: pgmap v364: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-06-04T04:56:05.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:05 smithi098 ceph-mon[35569]: pgmap v364: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-06-04T04:56:06.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.949+0000 7f7c41afa700 1 -- 172.21.15.70:0/1299151110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c3c102440 msgr2=0x7f7c3c1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:06.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.949+0000 7f7c41afa700 1 --2- 172.21.15.70:0/1299151110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c3c102440 0x7f7c3c1028c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009990 tx=0x7f7c24009ca0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.949+0000 7f7c41afa700 1 -- 172.21.15.70:0/1299151110 shutdown_connections 2024-06-04T04:56:06.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.949+0000 7f7c41afa700 1 --2- 172.21.15.70:0/1299151110 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c3c102e00 0x7f7c3c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.949+0000 7f7c41afa700 1 --2- 172.21.15.70:0/1299151110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c3c108520 0x7f7c3c108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.949+0000 7f7c41afa700 1 --2- 172.21.15.70:0/1299151110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c3c102440 0x7f7c3c1028c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.949+0000 7f7c41afa700 1 -- 172.21.15.70:0/1299151110 >> 172.21.15.70:0/1299151110 conn(0x7f7c3c0793f0 msgr2=0x7f7c3c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:06.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.949+0000 7f7c41afa700 1 -- 172.21.15.70:0/1299151110 shutdown_connections 2024-06-04T04:56:06.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.949+0000 7f7c41afa700 1 -- 172.21.15.70:0/1299151110 wait complete. 2024-06-04T04:56:06.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.949+0000 7f7c41afa700 1 Processor -- start 2024-06-04T04:56:06.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.950+0000 7f7c41afa700 1 -- start start 2024-06-04T04:56:06.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.950+0000 7f7c41afa700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c3c102440 0x7f7c3c104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.950+0000 7f7c41afa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c3c102e00 0x7f7c3c1045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.950+0000 7f7c41afa700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c3c108520 0x7f7c3c104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.950+0000 7f7c41afa700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7c3c112480 con 0x7f7c3c102440 2024-06-04T04:56:06.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.950+0000 7f7c41afa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c3c112300 con 0x7f7c3c102e00 2024-06-04T04:56:06.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.950+0000 7f7c41afa700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7c3c112600 con 0x7f7c3c108520 2024-06-04T04:56:06.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.950+0000 7f7c3b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c3c102440 0x7f7c3c104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:06.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.950+0000 7f7c3b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c3c102440 0x7f7c3c104080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34230/0 (socket says 172.21.15.70:34230) 2024-06-04T04:56:06.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.950+0000 7f7c3b7fe700 1 -- 172.21.15.70:0/3259779510 learned_addr learned my addr 172.21.15.70:0/3259779510 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:06.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.950+0000 7f7c32ffd700 1 --2- 172.21.15.70:0/3259779510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c3c102e00 0x7f7c3c1045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:06.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.950+0000 7f7c3bfff700 1 --2- 172.21.15.70:0/3259779510 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c3c108520 0x7f7c3c104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:06.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.950+0000 7f7c3b7fe700 1 -- 172.21.15.70:0/3259779510 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c3c108520 msgr2=0x7f7c3c104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:06.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.950+0000 7f7c3b7fe700 1 --2- 172.21.15.70:0/3259779510 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c3c108520 0x7f7c3c104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.950+0000 7f7c3b7fe700 1 -- 172.21.15.70:0/3259779510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c3c102e00 msgr2=0x7f7c3c1045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:06.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.950+0000 7f7c3b7fe700 1 --2- 172.21.15.70:0/3259779510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c3c102e00 0x7f7c3c1045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.950+0000 7f7c3b7fe700 1 -- 172.21.15.70:0/3259779510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c3c1a8070 con 0x7f7c3c102440 2024-06-04T04:56:06.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.951+0000 7f7c3b7fe700 1 --2- 172.21.15.70:0/3259779510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c3c102440 0x7f7c3c104080 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c00c9b0 tx=0x7f7c2c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.951+0000 7f7c397fa700 1 -- 172.21.15.70:0/3259779510 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c007b00 con 0x7f7c3c102440 2024-06-04T04:56:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.951+0000 7f7c397fa700 1 -- 172.21.15.70:0/3259779510 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c2c00ce50 con 0x7f7c3c102440 2024-06-04T04:56:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.951+0000 7f7c41afa700 1 -- 172.21.15.70:0/3259779510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c3c1a8360 con 0x7f7c3c102440 2024-06-04T04:56:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.951+0000 7f7c397fa700 1 -- 172.21.15.70:0/3259779510 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c018660 con 0x7f7c3c102440 2024-06-04T04:56:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.951+0000 7f7c41afa700 1 -- 172.21.15.70:0/3259779510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c3c1a8870 con 0x7f7c3c102440 2024-06-04T04:56:06.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.953+0000 7f7c397fa700 1 -- 172.21.15.70:0/3259779510 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f7c2c00f450 con 0x7f7c3c102440 2024-06-04T04:56:06.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.954+0000 7f7c397fa700 1 --2- 172.21.15.70:0/3259779510 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f7c280b2880 0x7f7c280b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.954+0000 7f7c32ffd700 1 --2- 172.21.15.70:0/3259779510 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f7c280b2880 0x7f7c280b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:06.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.955+0000 7f7c397fa700 1 -- 172.21.15.70:0/3259779510 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f7c2c0e2130 con 0x7f7c3c102440 2024-06-04T04:56:06.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.955+0000 7f7c32ffd700 1 --2- 172.21.15.70:0/3259779510 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f7c280b2880 0x7f7c280b4d40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400b5c0 tx=0x7f7c24005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:06.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.955+0000 7f7c4c38fb80 1 Processor -- start 2024-06-04T04:56:06.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.955+0000 7f7c4c38fb80 1 -- start start 2024-06-04T04:56:06.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.955+0000 7f7c4c38fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c52d0 0x562f1f0c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.955+0000 7f7c4c38fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0bf850 0x562f1f0bfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.955+0000 7f7c4c38fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0c0210 0x562f1f0cc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.955+0000 7f7c4c38fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562f1f01dd20 con 0x562f1f0c52d0 2024-06-04T04:56:06.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.955+0000 7f7c4c38fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562f1f00d8c0 con 0x562f1f0bf850 2024-06-04T04:56:06.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.955+0000 7f7c4c38fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562f1f011900 con 0x562f1f0c0210 2024-06-04T04:56:06.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.955+0000 7f7c3b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c52d0 0x562f1f0c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:06.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.955+0000 7f7c3b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c52d0 0x562f1f0c56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34242/0 (socket says 172.21.15.70:34242) 2024-06-04T04:56:06.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.955+0000 7f7c3b7fe700 1 -- 172.21.15.70:0/1666036684 learned_addr learned my addr 172.21.15.70:0/1666036684 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:06.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.955+0000 7f7c3bfff700 1 --2- 172.21.15.70:0/1666036684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0c0210 0x562f1f0cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:06.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.956+0000 7f7c32ffd700 1 --2- 172.21.15.70:0/1666036684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0bf850 0x562f1f0bfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:06.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.956+0000 7f7c3b7fe700 1 -- 172.21.15.70:0/1666036684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0c0210 msgr2=0x562f1f0cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.956+0000 7f7c3b7fe700 1 --2- 172.21.15.70:0/1666036684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0c0210 0x562f1f0cc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.956+0000 7f7c3b7fe700 1 -- 172.21.15.70:0/1666036684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0bf850 msgr2=0x562f1f0bfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.956+0000 7f7c3b7fe700 1 --2- 172.21.15.70:0/1666036684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0bf850 0x562f1f0bfcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.956+0000 7f7c3b7fe700 1 -- 172.21.15.70:0/1666036684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f1efb6f60 con 0x562f1f0c52d0 2024-06-04T04:56:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.956+0000 7f7c3bfff700 1 --2- 172.21.15.70:0/1666036684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0c0210 0x562f1f0cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.956+0000 7f7c3b7fe700 1 --2- 172.21.15.70:0/1666036684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c52d0 0x562f1f0c56b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c0a5750 tx=0x7f7c2c0a91e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b69fa6847fc65c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.956+0000 7f7c31ffb700 1 -- 172.21.15.70:0/1666036684 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c0a96d0 con 0x562f1f0c52d0 2024-06-04T04:56:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.956+0000 7f7c31ffb700 1 -- 172.21.15.70:0/1666036684 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c2c0ada80 con 0x562f1f0c52d0 2024-06-04T04:56:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.956+0000 7f7c31ffb700 1 -- 172.21.15.70:0/1666036684 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c0910b0 con 0x562f1f0c52d0 2024-06-04T04:56:06.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.957+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/1666036684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c52d0 msgr2=0x562f1f0c56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:06.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.957+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/1666036684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c52d0 0x562f1f0c56b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c0a5750 tx=0x7f7c2c0a91e0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.957+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/1666036684 shutdown_connections 2024-06-04T04:56:06.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.957+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/1666036684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0c0210 0x562f1f0cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.957+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/1666036684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c52d0 0x562f1f0c56b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.957+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/1666036684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0bf850 0x562f1f0bfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.957+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/1666036684 >> 172.21.15.70:0/1666036684 conn(0x562f1efeb1f0 msgr2=0x562f1f0ba390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:06.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.957+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/1666036684 shutdown_connections 2024-06-04T04:56:06.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.957+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/1666036684 wait complete. 2024-06-04T04:56:06.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.957+0000 7f7c4c38fb80 1 -- start start 2024-06-04T04:56:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c4c38fb80 1 Processor -- start 2024-06-04T04:56:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c4c38fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0bf850 0x562f1f0cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c4c38fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0c0210 0x562f1f0c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c4c38fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0c52d0 0x562f1f0c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c4c38fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562f1ef13f50 con 0x562f1f0bf850 2024-06-04T04:56:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c4c38fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562f1ef75d50 con 0x562f1f0c0210 2024-06-04T04:56:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c4c38fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562f1f0cc690 con 0x562f1f0c52d0 2024-06-04T04:56:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c4c38fb80 10 client.0 _ll_register_callbacks cb 0x562f1efa8900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c3b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0bf850 0x562f1f0cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c3b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0bf850 0x562f1f0cbca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34258/0 (socket says 172.21.15.70:34258) 2024-06-04T04:56:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c3b7fe700 1 -- 172.21.15.70:0/2369253917 learned_addr learned my addr 172.21.15.70:0/2369253917 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c32ffd700 1 --2- 172.21.15.70:0/2369253917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0c0210 0x562f1f0c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c3bfff700 1 --2- 172.21.15.70:0/2369253917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0c52d0 0x562f1f0c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c3b7fe700 1 -- 172.21.15.70:0/2369253917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0c52d0 msgr2=0x562f1f0c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c3b7fe700 1 --2- 172.21.15.70:0/2369253917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0c52d0 0x562f1f0c94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c3b7fe700 1 -- 172.21.15.70:0/2369253917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0c0210 msgr2=0x562f1f0c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c3b7fe700 1 --2- 172.21.15.70:0/2369253917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0c0210 0x562f1f0c8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c3b7fe700 1 -- 172.21.15.70:0/2369253917 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f1ef5d880 con 0x562f1f0bf850 2024-06-04T04:56:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.958+0000 7f7c32ffd700 1 --2- 172.21.15.70:0/2369253917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0c0210 0x562f1f0c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.959+0000 7f7c3b7fe700 1 --2- 172.21.15.70:0/2369253917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0bf850 0x562f1f0cbca0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c000c00 tx=0x7f7c2c0b2660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ad3b7281818a0a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.959+0000 7f7c1affd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.959+0000 7f7c1affd700 1 -- 172.21.15.70:0/2369253917 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c0c8060 con 0x562f1f0bf850 2024-06-04T04:56:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.959+0000 7f7c1affd700 1 -- 172.21.15.70:0/2369253917 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f7c2c007ca0 con 0x562f1f0bf850 2024-06-04T04:56:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.959+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/2369253917 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c2c0ada80 con 0x562f1f0bf850 2024-06-04T04:56:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.959+0000 7f7c1affd700 1 -- 172.21.15.70:0/2369253917 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7c2c088050 con 0x562f1f0bf850 2024-06-04T04:56:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.959+0000 7f7c4c38fb80 10 client.14967 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.959+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/2369253917 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562f1efc7710 con 0x562f1f0bf850 2024-06-04T04:56:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.959+0000 7f7c1affd700 1 -- 172.21.15.70:0/2369253917 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c2c0ada80 con 0x562f1f0bf850 2024-06-04T04:56:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.959+0000 7f7c1affd700 1 -- 172.21.15.70:0/2369253917 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f7c2c0beb20 con 0x562f1f0bf850 2024-06-04T04:56:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.959+0000 7f7c4c38fb80 10 client.14967 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.959+0000 7f7c4c38fb80 10 client.14967 resolve_mds: resolved name 'h' to daemon mds.h 2024-06-04T04:56:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.959+0000 7f7c4c38fb80 20 client.14967 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.959+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/2369253917 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x562f1f0d7ee0 0x562f1f0da0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.959+0000 7f7c4c38fb80 4 client.14967 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:56:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.959+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/2369253917 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f1f01fc70 con 0x562f1f0d7ee0 2024-06-04T04:56:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.960+0000 7f7c32ffd700 1 --2- 172.21.15.70:0/2369253917 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x562f1f0d7ee0 0x562f1f0da0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.960+0000 7f7c1affd700 10 client.14967 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T04:56:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.960+0000 7f7c32ffd700 1 --2- 172.21.15.70:0/2369253917 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x562f1f0d7ee0 0x562f1f0da0d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=528f10cc12f49a40 server_cookie=eb28f49cef06cfbc in_seq=0 out_seq=0 2024-06-04T04:56:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.961+0000 7f7c1affd700 1 -- 172.21.15.70:0/2369253917 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562f1f01fc70 con 0x562f1f0d7ee0 2024-06-04T04:56:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.961+0000 7f7c1affd700 10 client.14967 handle_command_reply: tid=0 2024-06-04T04:56:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.961+0000 7f7c4c38fb80 1 client.14967 shutdown 2024-06-04T04:56:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.961+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/2369253917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0bf850 msgr2=0x562f1f0cbca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.961+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/2369253917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0bf850 0x562f1f0cbca0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c000c00 tx=0x7f7c2c0b2660 comp rx=0 tx=0).stop 2024-06-04T04:56:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.961+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/2369253917 shutdown_connections 2024-06-04T04:56:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.961+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/2369253917 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x562f1f0d7ee0 0x562f1f0da0d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.961+0000 7f7c1affd700 0 client.14967 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T04:56:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.961+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/2369253917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0c52d0 0x562f1f0c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.961+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/2369253917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0bf850 0x562f1f0cbca0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.961+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/2369253917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0c0210 0x562f1f0c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.961+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/2369253917 >> 172.21.15.70:0/2369253917 conn(0x562f1efeb1f0 msgr2=0x562f1f0b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.961+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/2369253917 shutdown_connections 2024-06-04T04:56:06.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.961+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/2369253917 wait complete. 2024-06-04T04:56:06.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.962+0000 7f7c4c38fb80 20 client.14967 trim_cache size 0 max 16384 2024-06-04T04:56:06.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c4c38fb80 1 Processor -- start 2024-06-04T04:56:06.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c4c38fb80 1 -- start start 2024-06-04T04:56:06.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c4c38fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0bf850 0x562f1f0ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c4c38fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c0210 0x562f1f0caee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c4c38fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0c52d0 0x562f1f0c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c4c38fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562f1f0d8410 con 0x562f1f0c0210 2024-06-04T04:56:06.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c4c38fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562f1f0d8ae0 con 0x562f1f0bf850 2024-06-04T04:56:06.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c4c38fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562f1f0d91b0 con 0x562f1f0c52d0 2024-06-04T04:56:06.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c32ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c0210 0x562f1f0caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:06.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c32ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c0210 0x562f1f0caee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34272/0 (socket says 172.21.15.70:34272) 2024-06-04T04:56:06.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c32ffd700 1 -- 172.21.15.70:0/1773634600 learned_addr learned my addr 172.21.15.70:0/1773634600 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:06.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c32ffd700 1 -- 172.21.15.70:0/1773634600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0c52d0 msgr2=0x562f1f0c5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:06.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c32ffd700 1 --2- 172.21.15.70:0/1773634600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0c52d0 0x562f1f0c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c32ffd700 1 -- 172.21.15.70:0/1773634600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0bf850 msgr2=0x562f1f0ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:06.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c32ffd700 1 --2- 172.21.15.70:0/1773634600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0bf850 0x562f1f0ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c32ffd700 1 -- 172.21.15.70:0/1773634600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f1f0cb8e0 con 0x562f1f0c0210 2024-06-04T04:56:06.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c32ffd700 1 --2- 172.21.15.70:0/1773634600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c0210 0x562f1f0caee0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7c24003730 tx=0x7f7c24024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52fc446c13f066f0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:06.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c317fa700 1 -- 172.21.15.70:0/1773634600 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2400f840 con 0x562f1f0c0210 2024-06-04T04:56:06.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c317fa700 1 -- 172.21.15.70:0/1773634600 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c24024420 con 0x562f1f0c0210 2024-06-04T04:56:06.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.984+0000 7f7c317fa700 1 -- 172.21.15.70:0/1773634600 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c24030650 con 0x562f1f0c0210 2024-06-04T04:56:06.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.985+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/1773634600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c0210 msgr2=0x562f1f0caee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:06.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.985+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/1773634600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c0210 0x562f1f0caee0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7c24003730 tx=0x7f7c24024000 comp rx=0 tx=0).stop 2024-06-04T04:56:06.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.985+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/1773634600 shutdown_connections 2024-06-04T04:56:06.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.985+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/1773634600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0c52d0 0x562f1f0c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.985+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/1773634600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0bf850 0x562f1f0ba220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.985+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/1773634600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c0210 0x562f1f0caee0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.985+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/1773634600 >> 172.21.15.70:0/1773634600 conn(0x562f1efeb1f0 msgr2=0x562f1f0ca7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:06.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.985+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/1773634600 shutdown_connections 2024-06-04T04:56:06.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.985+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/1773634600 wait complete. 2024-06-04T04:56:06.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.986+0000 7f7c4c38fb80 1 -- start start 2024-06-04T04:56:06.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.986+0000 7f7c4c38fb80 1 Processor -- start 2024-06-04T04:56:06.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.986+0000 7f7c4c38fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0bf850 0x562f1f0e3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.986+0000 7f7c4c38fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0c0210 0x562f1f0e3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.986+0000 7f7c4c38fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c52d0 0x562f1f0dc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.986+0000 7f7c4c38fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562f1f0d9f50 con 0x562f1f0c52d0 2024-06-04T04:56:06.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.986+0000 7f7c4c38fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562f1f0d9880 con 0x562f1f0c0210 2024-06-04T04:56:06.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.986+0000 7f7c4c38fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562f1f0da620 con 0x562f1f0bf850 2024-06-04T04:56:06.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.986+0000 7f7c4c38fb80 10 client.0 _ll_register_callbacks cb 0x562f1efa8900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:06.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.986+0000 7f7c3bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c52d0 0x562f1f0dc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:06.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.986+0000 7f7c3bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c52d0 0x562f1f0dc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34278/0 (socket says 172.21.15.70:34278) 2024-06-04T04:56:06.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.986+0000 7f7c3bfff700 1 -- 172.21.15.70:0/2410427815 learned_addr learned my addr 172.21.15.70:0/2410427815 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:06.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.986+0000 7f7c3b7fe700 1 --2- 172.21.15.70:0/2410427815 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0bf850 0x562f1f0e3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:06.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.986+0000 7f7c32ffd700 1 --2- 172.21.15.70:0/2410427815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0c0210 0x562f1f0e3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:06.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.987+0000 7f7c3bfff700 1 -- 172.21.15.70:0/2410427815 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0bf850 msgr2=0x562f1f0e3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:06.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.987+0000 7f7c3bfff700 1 --2- 172.21.15.70:0/2410427815 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0bf850 0x562f1f0e3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.987+0000 7f7c3bfff700 1 -- 172.21.15.70:0/2410427815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0c0210 msgr2=0x562f1f0e3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:06.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.987+0000 7f7c3bfff700 1 --2- 172.21.15.70:0/2410427815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0c0210 0x562f1f0e3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.987+0000 7f7c3bfff700 1 -- 172.21.15.70:0/2410427815 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f1f0dcc40 con 0x562f1f0c52d0 2024-06-04T04:56:06.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.987+0000 7f7c3bfff700 1 --2- 172.21.15.70:0/2410427815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c52d0 0x562f1f0dc6d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7c34004b10 tx=0x7f7c340124b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4ad0ff0c9647ba server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:06.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.987+0000 7f7c1affd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:06.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.987+0000 7f7c1affd700 1 -- 172.21.15.70:0/2410427815 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3401b020 con 0x562f1f0c52d0 2024-06-04T04:56:06.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.987+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/2410427815 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c24024420 con 0x562f1f0c52d0 2024-06-04T04:56:06.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.987+0000 7f7c1affd700 1 -- 172.21.15.70:0/2410427815 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f7c34012a40 con 0x562f1f0c52d0 2024-06-04T04:56:06.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.987+0000 7f7c1affd700 1 -- 172.21.15.70:0/2410427815 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7c34026050 con 0x562f1f0c52d0 2024-06-04T04:56:06.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.987+0000 7f7c4c38fb80 10 client.14973 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:06.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.987+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/2410427815 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562f1f144e70 con 0x562f1f0c52d0 2024-06-04T04:56:06.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.987+0000 7f7c1affd700 1 -- 172.21.15.70:0/2410427815 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c24024420 con 0x562f1f0c52d0 2024-06-04T04:56:06.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.987+0000 7f7c1affd700 1 -- 172.21.15.70:0/2410427815 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f7c34016460 con 0x562f1f0c52d0 2024-06-04T04:56:06.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.987+0000 7f7c4c38fb80 10 client.14973 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:06.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.987+0000 7f7c4c38fb80 10 client.14973 resolve_mds: resolved name 'h' to daemon mds.h 2024-06-04T04:56:06.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.988+0000 7f7c4c38fb80 20 client.14973 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:06.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.988+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/2410427815 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x562f1f0d7ee0 0x562f1f1487c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:06.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.988+0000 7f7c4c38fb80 4 client.14973 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-04T04:56:06.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.988+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/2410427815 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "status"}) -- 0x562f1f019450 con 0x562f1f0d7ee0 2024-06-04T04:56:06.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.988+0000 7f7c3b7fe700 1 --2- 172.21.15.70:0/2410427815 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x562f1f0d7ee0 0x562f1f1487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:06.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.988+0000 7f7c3b7fe700 1 --2- 172.21.15.70:0/2410427815 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x562f1f0d7ee0 0x562f1f1487c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=79d23cce4be50465 server_cookie=dbee354e78260646 in_seq=0 out_seq=0 2024-06-04T04:56:06.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.988+0000 7f7c1affd700 10 client.14973 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T04:56:06.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.988+0000 7f7c1affd700 1 -- 172.21.15.70:0/2410427815 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x562f1f019450 con 0x562f1f0d7ee0 2024-06-04T04:56:06.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.988+0000 7f7c1affd700 10 client.14973 handle_command_reply: tid=0 2024-06-04T04:56:06.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.988+0000 7f7c4c38fb80 1 client.14973 shutdown 2024-06-04T04:56:06.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/2410427815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c52d0 msgr2=0x562f1f0dc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/2410427815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c52d0 0x562f1f0dc6d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7c34004b10 tx=0x7f7c340124b0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/2410427815 shutdown_connections 2024-06-04T04:56:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/2410427815 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x562f1f0d7ee0 0x562f1f1487c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/2410427815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1f0c52d0 0x562f1f0dc6d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/2410427815 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f1f0bf850 0x562f1f0e3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c1affd700 0 client.14973 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T04:56:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c4c38fb80 1 --2- 172.21.15.70:0/2410427815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f1f0c0210 0x562f1f0e3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/2410427815 >> 172.21.15.70:0/2410427815 conn(0x562f1efeb1f0 msgr2=0x562f1f0bf0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/2410427815 shutdown_connections 2024-06-04T04:56:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c4c38fb80 1 -- 172.21.15.70:0/2410427815 wait complete. 2024-06-04T04:56:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c4c38fb80 20 client.14973 trim_cache size 0 max 16384 2024-06-04T04:56:06.996 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:56:06.997 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:56:06.997 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 4, 2024-06-04T04:56:06.997 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24410, 2024-06-04T04:56:06.997 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-06-04T04:56:06.997 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-06-04T04:56:06.997 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-06-04T04:56:06.997 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 133.552747205, 2024-06-04T04:56:06.997 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-06-04T04:56:06.997 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 81, 2024-06-04T04:56:06.997 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 79, 2024-06-04T04:56:06.997 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 177.014136038 2024-06-04T04:56:06.998 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:56:06.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c41afa700 1 -- 172.21.15.70:0/3259779510 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f7c280b2880 msgr2=0x7f7c280b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:06.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c41afa700 1 --2- 172.21.15.70:0/3259779510 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f7c280b2880 0x7f7c280b4d40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400b5c0 tx=0x7f7c24005960 comp rx=0 tx=0).stop 2024-06-04T04:56:06.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c41afa700 1 -- 172.21.15.70:0/3259779510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c3c102440 msgr2=0x7f7c3c104080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:06.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c41afa700 1 --2- 172.21.15.70:0/3259779510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c3c102440 0x7f7c3c104080 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c00c9b0 tx=0x7f7c2c00ccc0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c41afa700 1 -- 172.21.15.70:0/3259779510 shutdown_connections 2024-06-04T04:56:06.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c41afa700 1 --2- 172.21.15.70:0/3259779510 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f7c280b2880 0x7f7c280b4d40 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c41afa700 1 --2- 172.21.15.70:0/3259779510 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c3c108520 0x7f7c3c104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c41afa700 1 --2- 172.21.15.70:0/3259779510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c3c102440 0x7f7c3c104080 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c41afa700 1 --2- 172.21.15.70:0/3259779510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c3c102e00 0x7f7c3c1045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:06.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.989+0000 7f7c41afa700 1 -- 172.21.15.70:0/3259779510 >> 172.21.15.70:0/3259779510 conn(0x7f7c3c0793f0 msgr2=0x7f7c3c06c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:06.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.990+0000 7f7c41afa700 1 -- 172.21.15.70:0/3259779510 shutdown_connections 2024-06-04T04:56:06.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:06.990+0000 7f7c41afa700 1 -- 172.21.15.70:0/3259779510 wait complete. 2024-06-04T04:56:07.629 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.f status 2024-06-04T04:56:07.652 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:07 smithi070 ceph-mon[32631]: pgmap v365: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:07 smithi112 ceph-mon[36453]: pgmap v365: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:07 smithi098 ceph-mon[35569]: pgmap v365: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:07.871 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:09.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.620+0000 7fcbf25fe700 1 -- 172.21.15.70:0/1900882809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec102e30 msgr2=0x7fcbec10f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:09.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.620+0000 7fcbf25fe700 1 --2- 172.21.15.70:0/1900882809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec102e30 0x7fcbec10f320 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400b0d0 tx=0x7fcbe400b3e0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.620+0000 7fcbf25fe700 1 -- 172.21.15.70:0/1900882809 shutdown_connections 2024-06-04T04:56:09.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.620+0000 7fcbf25fe700 1 --2- 172.21.15.70:0/1900882809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec102e30 0x7fcbec10f320 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.620+0000 7fcbf25fe700 1 --2- 172.21.15.70:0/1900882809 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcbec108550 0x7fcbec108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.620+0000 7fcbf25fe700 1 --2- 172.21.15.70:0/1900882809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbec102470 0x7fcbec1028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.620+0000 7fcbf25fe700 1 -- 172.21.15.70:0/1900882809 >> 172.21.15.70:0/1900882809 conn(0x7fcbec079480 msgr2=0x7fcbec079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:09.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.620+0000 7fcbf25fe700 1 -- 172.21.15.70:0/1900882809 shutdown_connections 2024-06-04T04:56:09.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.620+0000 7fcbf25fe700 1 -- 172.21.15.70:0/1900882809 wait complete. 2024-06-04T04:56:09.624 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:09 smithi070 ceph-mon[32631]: pgmap v366: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:56:09.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.621+0000 7fcbf25fe700 1 Processor -- start 2024-06-04T04:56:09.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.621+0000 7fcbf25fe700 1 -- start start 2024-06-04T04:56:09.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.622+0000 7fcbf25fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcbec102470 0x7fcbec19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.622+0000 7fcbf25fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbec102e30 0x7fcbec19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.622+0000 7fcbf25fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec108550 0x7fcbec198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.622+0000 7fcbf25fe700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec112570 con 0x7fcbec108550 2024-06-04T04:56:09.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.622+0000 7fcbf25fe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec1123f0 con 0x7fcbec102e30 2024-06-04T04:56:09.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.622+0000 7fcbf25fe700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec1126f0 con 0x7fcbec102470 2024-06-04T04:56:09.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.622+0000 7fcbebfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcbec102470 0x7fcbec19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.622+0000 7fcbf0b9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec108550 0x7fcbec198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.622+0000 7fcbebfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcbec102470 0x7fcbec19e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41844/0 (socket says 172.21.15.70:41844) 2024-06-04T04:56:09.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.622+0000 7fcbebfff700 1 -- 172.21.15.70:0/2276778386 learned_addr learned my addr 172.21.15.70:0/2276778386 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:09.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.622+0000 7fcbeb7fe700 1 --2- 172.21.15.70:0/2276778386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbec102e30 0x7fcbec19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.622+0000 7fcbebfff700 1 -- 172.21.15.70:0/2276778386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbec102e30 msgr2=0x7fcbec19ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:09.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.622+0000 7fcbebfff700 1 --2- 172.21.15.70:0/2276778386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbec102e30 0x7fcbec19ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.622+0000 7fcbebfff700 1 -- 172.21.15.70:0/2276778386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec108550 msgr2=0x7fcbec198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:09.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.622+0000 7fcbebfff700 1 --2- 172.21.15.70:0/2276778386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec108550 0x7fcbec198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.622+0000 7fcbebfff700 1 -- 172.21.15.70:0/2276778386 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbec198ff0 con 0x7fcbec102470 2024-06-04T04:56:09.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.622+0000 7fcbf0b9b700 1 --2- 172.21.15.70:0/2276778386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec108550 0x7fcbec198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:56:09.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.623+0000 7fcbebfff700 1 --2- 172.21.15.70:0/2276778386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcbec102470 0x7fcbec19e6d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00d980 tx=0x7fcbdc00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:09.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.623+0000 7fcbe97fa700 1 -- 172.21.15.70:0/2276778386 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc017070 con 0x7fcbec102470 2024-06-04T04:56:09.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.623+0000 7fcbe97fa700 1 -- 172.21.15.70:0/2276778386 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbdc005020 con 0x7fcbec102470 2024-06-04T04:56:09.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.623+0000 7fcbf25fe700 1 -- 172.21.15.70:0/2276778386 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbec1992e0 con 0x7fcbec102470 2024-06-04T04:56:09.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.623+0000 7fcbe97fa700 1 -- 172.21.15.70:0/2276778386 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc00f980 con 0x7fcbec102470 2024-06-04T04:56:09.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.623+0000 7fcbf25fe700 1 -- 172.21.15.70:0/2276778386 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbec06bc10 con 0x7fcbec102470 2024-06-04T04:56:09.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.626+0000 7fcbe97fa700 1 -- 172.21.15.70:0/2276778386 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fcbdc004a30 con 0x7fcbec102470 2024-06-04T04:56:09.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.626+0000 7fcbe97fa700 1 --2- 172.21.15.70:0/2276778386 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fcbd40b2880 0x7fcbd40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.626+0000 7fcbeb7fe700 1 --2- 172.21.15.70:0/2276778386 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fcbd40b2880 0x7fcbd40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.627+0000 7fcbe97fa700 1 -- 172.21.15.70:0/2276778386 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fcbdc0e2000 con 0x7fcbec102470 2024-06-04T04:56:09.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.627+0000 7fcbeb7fe700 1 --2- 172.21.15.70:0/2276778386 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fcbd40b2880 0x7fcbd40b4d40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcbec079b60 tx=0x7fcbe0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:09.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.627+0000 7fcbfce93b80 1 Processor -- start 2024-06-04T04:56:09.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.627+0000 7fcbfce93b80 1 -- start start 2024-06-04T04:56:09.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.627+0000 7fcbfce93b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d1402586b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.627+0000 7fcbfce93b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140252850 0x55d140252cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.627+0000 7fcbfce93b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d140253210 0x55d14025f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.627+0000 7fcbfce93b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d1401b0d20 con 0x55d1402582d0 2024-06-04T04:56:09.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.627+0000 7fcbfce93b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d1401a08c0 con 0x55d140253210 2024-06-04T04:56:09.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.627+0000 7fcbfce93b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d1401a4900 con 0x55d140252850 2024-06-04T04:56:09.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.627+0000 7fcbebfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d1402586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.627+0000 7fcbebfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d1402586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34300/0 (socket says 172.21.15.70:34300) 2024-06-04T04:56:09.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.627+0000 7fcbebfff700 1 -- 172.21.15.70:0/1615478768 learned_addr learned my addr 172.21.15.70:0/1615478768 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:09.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.628+0000 7fcbeb7fe700 1 --2- 172.21.15.70:0/1615478768 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140252850 0x55d140252cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.628+0000 7fcbebfff700 1 -- 172.21.15.70:0/1615478768 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140252850 msgr2=0x55d140252cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:09.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.628+0000 7fcbebfff700 1 --2- 172.21.15.70:0/1615478768 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140252850 0x55d140252cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.628+0000 7fcbf0b9b700 1 --2- 172.21.15.70:0/1615478768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d140253210 0x55d14025f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.628+0000 7fcbebfff700 1 -- 172.21.15.70:0/1615478768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d140253210 msgr2=0x55d14025f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:09.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.628+0000 7fcbebfff700 1 --2- 172.21.15.70:0/1615478768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d140253210 0x55d14025f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.628+0000 7fcbebfff700 1 -- 172.21.15.70:0/1615478768 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d140149f60 con 0x55d1402582d0 2024-06-04T04:56:09.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.628+0000 7fcbeb7fe700 1 --2- 172.21.15.70:0/1615478768 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140252850 0x55d140252cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:56:09.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.628+0000 7fcbebfff700 1 --2- 172.21.15.70:0/1615478768 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d1402586b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc0a93a0 tx=0x7fcbdc004e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f4a41c86b5c6d4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:09.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.628+0000 7fcbd27fc700 1 -- 172.21.15.70:0/1615478768 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc0a96a0 con 0x55d1402582d0 2024-06-04T04:56:09.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.628+0000 7fcbd27fc700 1 -- 172.21.15.70:0/1615478768 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbdc0ada50 con 0x55d1402582d0 2024-06-04T04:56:09.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.628+0000 7fcbd27fc700 1 -- 172.21.15.70:0/1615478768 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc0b2070 con 0x55d1402582d0 2024-06-04T04:56:09.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.629+0000 7fcbfce93b80 1 -- 172.21.15.70:0/1615478768 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 msgr2=0x55d1402586b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:09.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.629+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/1615478768 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d1402586b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc0a93a0 tx=0x7fcbdc004e50 comp rx=0 tx=0).stop 2024-06-04T04:56:09.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.629+0000 7fcbfce93b80 1 -- 172.21.15.70:0/1615478768 shutdown_connections 2024-06-04T04:56:09.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.629+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/1615478768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d140253210 0x55d14025f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.629+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/1615478768 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d1402586b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.629+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/1615478768 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140252850 0x55d140252cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.629+0000 7fcbfce93b80 1 -- 172.21.15.70:0/1615478768 >> 172.21.15.70:0/1615478768 conn(0x55d14017e1f0 msgr2=0x55d14024d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:09.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.629+0000 7fcbfce93b80 1 -- 172.21.15.70:0/1615478768 shutdown_connections 2024-06-04T04:56:09.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.629+0000 7fcbfce93b80 1 -- 172.21.15.70:0/1615478768 wait complete. 2024-06-04T04:56:09.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.629+0000 7fcbfce93b80 1 -- start start 2024-06-04T04:56:09.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.629+0000 7fcbfce93b80 1 Processor -- start 2024-06-04T04:56:09.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbfce93b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140252850 0x55d14025ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbfce93b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d140253210 0x55d14025c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbfce93b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d14025c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbfce93b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d1400a6f50 con 0x55d1402582d0 2024-06-04T04:56:09.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbfce93b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d140108d50 con 0x55d140253210 2024-06-04T04:56:09.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbfce93b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d14025f5d0 con 0x55d140252850 2024-06-04T04:56:09.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbfce93b80 10 client.0 _ll_register_callbacks cb 0x55d14013b900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:09.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbf0b9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d14025c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbebfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140252850 0x55d14025ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbeb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d140253210 0x55d14025c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbf0b9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d14025c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34304/0 (socket says 172.21.15.70:34304) 2024-06-04T04:56:09.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbf0b9b700 1 -- 172.21.15.70:0/1036514530 learned_addr learned my addr 172.21.15.70:0/1036514530 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:09.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbebfff700 1 --2- 172.21.15.70:0/1036514530 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140252850 0x55d14025ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41864/0 (socket says 172.21.15.70:41864) 2024-06-04T04:56:09.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbf0b9b700 1 -- 172.21.15.70:0/1036514530 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140252850 msgr2=0x55d14025ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:09.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbf0b9b700 1 --2- 172.21.15.70:0/1036514530 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140252850 0x55d14025ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbf0b9b700 1 -- 172.21.15.70:0/1036514530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d140253210 msgr2=0x55d14025c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:09.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbf0b9b700 1 --2- 172.21.15.70:0/1036514530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d140253210 0x55d14025c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbf0b9b700 1 -- 172.21.15.70:0/1036514530 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1400f0880 con 0x55d1402582d0 2024-06-04T04:56:09.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbf0b9b700 1 --2- 172.21.15.70:0/1036514530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d14025c570 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4003e00 tx=0x7fcbe4003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5c9f31ae3eb34d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:09.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.630+0000 7fcbcb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:09.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.631+0000 7fcbcb7fe700 1 -- 172.21.15.70:0/1036514530 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4007970 con 0x55d1402582d0 2024-06-04T04:56:09.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.631+0000 7fcbcb7fe700 1 -- 172.21.15.70:0/1036514530 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fcbe4007b10 con 0x55d1402582d0 2024-06-04T04:56:09.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.631+0000 7fcbfce93b80 1 -- 172.21.15.70:0/1036514530 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbdc0ada50 con 0x55d1402582d0 2024-06-04T04:56:09.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.631+0000 7fcbcb7fe700 1 -- 172.21.15.70:0/1036514530 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcbe402c070 con 0x55d1402582d0 2024-06-04T04:56:09.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.631+0000 7fcbfce93b80 10 client.14985 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:09.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.631+0000 7fcbfce93b80 1 -- 172.21.15.70:0/1036514530 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d14015a710 con 0x55d1402582d0 2024-06-04T04:56:09.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.631+0000 7fcbcb7fe700 1 -- 172.21.15.70:0/1036514530 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbdc0ada50 con 0x55d1402582d0 2024-06-04T04:56:09.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.631+0000 7fcbcb7fe700 1 -- 172.21.15.70:0/1036514530 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fcbe4011070 con 0x55d1402582d0 2024-06-04T04:56:09.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.631+0000 7fcbfce93b80 10 client.14985 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:09.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.631+0000 7fcbfce93b80 10 client.14985 resolve_mds: resolved name 'f' to daemon mds.f 2024-06-04T04:56:09.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.631+0000 7fcbfce93b80 20 client.14985 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:09.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.631+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/1036514530 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d14026c130 0x55d14026d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.631+0000 7fcbfce93b80 4 client.14985 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:56:09.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.631+0000 7fcbfce93b80 1 -- 172.21.15.70:0/1036514530 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1401b2c70 con 0x55d14026c130 2024-06-04T04:56:09.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.632+0000 7fcbebfff700 1 --2- 172.21.15.70:0/1036514530 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d14026c130 0x55d14026d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.632+0000 7fcbebfff700 1 --2- 172.21.15.70:0/1036514530 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d14026c130 0x55d14026d5a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=96b32c173f5d2b6b server_cookie=e49cef57e3dab26d in_seq=0 out_seq=0 2024-06-04T04:56:09.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.632+0000 7fcbcb7fe700 10 client.14985 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T04:56:09.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.633+0000 7fcbcb7fe700 1 -- 172.21.15.70:0/1036514530 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d1401b2c70 con 0x55d14026c130 2024-06-04T04:56:09.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.633+0000 7fcbcb7fe700 10 client.14985 handle_command_reply: tid=0 2024-06-04T04:56:09.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.633+0000 7fcbfce93b80 1 client.14985 shutdown 2024-06-04T04:56:09.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.633+0000 7fcbfce93b80 1 -- 172.21.15.70:0/1036514530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 msgr2=0x55d14025c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:09.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.633+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/1036514530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d14025c570 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4003e00 tx=0x7fcbe4003e30 comp rx=0 tx=0).stop 2024-06-04T04:56:09.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.633+0000 7fcbfce93b80 1 -- 172.21.15.70:0/1036514530 shutdown_connections 2024-06-04T04:56:09.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.633+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/1036514530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d14025c570 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.633+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/1036514530 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d14026c130 0x55d14026d5a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.633+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/1036514530 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140252850 0x55d14025ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.633+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/1036514530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d140253210 0x55d14025c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.633+0000 7fcbcb7fe700 0 client.14985 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T04:56:09.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.633+0000 7fcbfce93b80 1 -- 172.21.15.70:0/1036514530 >> 172.21.15.70:0/1036514530 conn(0x55d14017e1f0 msgr2=0x55d14024caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:09.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.633+0000 7fcbfce93b80 1 -- 172.21.15.70:0/1036514530 shutdown_connections 2024-06-04T04:56:09.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.633+0000 7fcbfce93b80 1 -- 172.21.15.70:0/1036514530 wait complete. 2024-06-04T04:56:09.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.633+0000 7fcbfce93b80 20 client.14985 trim_cache size 0 max 16384 2024-06-04T04:56:09.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.654+0000 7fcbfce93b80 1 Processor -- start 2024-06-04T04:56:09.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.654+0000 7fcbfce93b80 1 -- start start 2024-06-04T04:56:09.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbfce93b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d140252850 0x55d14024d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbfce93b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140253210 0x55d140256410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbfce93b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d140276a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbfce93b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d14026b8e0 con 0x55d1402582d0 2024-06-04T04:56:09.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbfce93b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d14026bfb0 con 0x55d140252850 2024-06-04T04:56:09.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbfce93b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d14026c680 con 0x55d140253210 2024-06-04T04:56:09.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbf0b9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d140276a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbf0b9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d140276a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34320/0 (socket says 172.21.15.70:34320) 2024-06-04T04:56:09.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbf0b9b700 1 -- 172.21.15.70:0/287048714 learned_addr learned my addr 172.21.15.70:0/287048714 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:09.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbeb7fe700 1 --2- 172.21.15.70:0/287048714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140253210 0x55d140256410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbebfff700 1 --2- 172.21.15.70:0/287048714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d140252850 0x55d14024d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbf0b9b700 1 -- 172.21.15.70:0/287048714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140253210 msgr2=0x55d140256410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbf0b9b700 1 --2- 172.21.15.70:0/287048714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140253210 0x55d140256410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbf0b9b700 1 -- 172.21.15.70:0/287048714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d140252850 msgr2=0x55d14024d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbf0b9b700 1 --2- 172.21.15.70:0/287048714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d140252850 0x55d14024d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbf0b9b700 1 -- 172.21.15.70:0/287048714 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d14025f580 con 0x55d1402582d0 2024-06-04T04:56:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbeb7fe700 1 --2- 172.21.15.70:0/287048714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140253210 0x55d140256410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.655+0000 7fcbf0b9b700 1 --2- 172.21.15.70:0/287048714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d140276a00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4005ce0 tx=0x7fcbe4005d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b13cc397554b91 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.656+0000 7fcbd1ffb700 1 -- 172.21.15.70:0/287048714 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4036070 con 0x55d1402582d0 2024-06-04T04:56:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.656+0000 7fcbd1ffb700 1 -- 172.21.15.70:0/287048714 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbe4021030 con 0x55d1402582d0 2024-06-04T04:56:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.656+0000 7fcbd1ffb700 1 -- 172.21.15.70:0/287048714 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4007af0 con 0x55d1402582d0 2024-06-04T04:56:09.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.656+0000 7fcbfce93b80 1 -- 172.21.15.70:0/287048714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 msgr2=0x55d140276a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:09.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.656+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/287048714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d140276a00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4005ce0 tx=0x7fcbe4005d10 comp rx=0 tx=0).stop 2024-06-04T04:56:09.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.656+0000 7fcbfce93b80 1 -- 172.21.15.70:0/287048714 shutdown_connections 2024-06-04T04:56:09.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.656+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/287048714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1402582d0 0x55d140276a00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.656+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/287048714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d140252850 0x55d14024d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.656+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/287048714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140253210 0x55d140256410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.656+0000 7fcbfce93b80 1 -- 172.21.15.70:0/287048714 >> 172.21.15.70:0/287048714 conn(0x55d14017e1f0 msgr2=0x55d140255d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:09.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.656+0000 7fcbfce93b80 1 -- 172.21.15.70:0/287048714 shutdown_connections 2024-06-04T04:56:09.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.656+0000 7fcbfce93b80 1 -- 172.21.15.70:0/287048714 wait complete. 2024-06-04T04:56:09.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.657+0000 7fcbfce93b80 1 -- start start 2024-06-04T04:56:09.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.657+0000 7fcbfce93b80 1 Processor -- start 2024-06-04T04:56:09.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.657+0000 7fcbfce93b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140252850 0x55d140276a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.657+0000 7fcbfce93b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d140253210 0x55d140276f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.657+0000 7fcbfce93b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1402582d0 0x55d140272eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.657+0000 7fcbfce93b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d14026d420 con 0x55d140253210 2024-06-04T04:56:09.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.657+0000 7fcbfce93b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d14026cd50 con 0x55d1402582d0 2024-06-04T04:56:09.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.657+0000 7fcbfce93b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d14026daf0 con 0x55d140252850 2024-06-04T04:56:09.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.657+0000 7fcbfce93b80 10 client.0 _ll_register_callbacks cb 0x55d14013b900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:09.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.657+0000 7fcbeb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d140253210 0x55d140276f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.657+0000 7fcbeb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d140253210 0x55d140276f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34334/0 (socket says 172.21.15.70:34334) 2024-06-04T04:56:09.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.657+0000 7fcbeb7fe700 1 -- 172.21.15.70:0/3112059079 learned_addr learned my addr 172.21.15.70:0/3112059079 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:09.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.658+0000 7fcbf0b9b700 1 --2- 172.21.15.70:0/3112059079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1402582d0 0x55d140272eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.658+0000 7fcbebfff700 1 --2- 172.21.15.70:0/3112059079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140252850 0x55d140276a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.658+0000 7fcbeb7fe700 1 -- 172.21.15.70:0/3112059079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140252850 msgr2=0x55d140276a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:09.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.658+0000 7fcbeb7fe700 1 --2- 172.21.15.70:0/3112059079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140252850 0x55d140276a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.658+0000 7fcbeb7fe700 1 -- 172.21.15.70:0/3112059079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1402582d0 msgr2=0x55d140272eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:09.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.658+0000 7fcbeb7fe700 1 --2- 172.21.15.70:0/3112059079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1402582d0 0x55d140272eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.658+0000 7fcbeb7fe700 1 -- 172.21.15.70:0/3112059079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1402736c0 con 0x55d140253210 2024-06-04T04:56:09.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.658+0000 7fcbeb7fe700 1 --2- 172.21.15.70:0/3112059079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d140253210 0x55d140276f40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0006ed0 tx=0x7fcbe0004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c694e22a96c87bf8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:09.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.658+0000 7fcbcb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:09.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.658+0000 7fcbcb7fe700 1 -- 172.21.15.70:0/3112059079 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0026070 con 0x55d140253210 2024-06-04T04:56:09.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.658+0000 7fcbcb7fe700 1 -- 172.21.15.70:0/3112059079 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fcbe000f040 con 0x55d140253210 2024-06-04T04:56:09.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.658+0000 7fcbfce93b80 1 -- 172.21.15.70:0/3112059079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbe4021030 con 0x55d140253210 2024-06-04T04:56:09.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.658+0000 7fcbcb7fe700 1 -- 172.21.15.70:0/3112059079 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcbe0015c90 con 0x55d140253210 2024-06-04T04:56:09.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.658+0000 7fcbfce93b80 10 client.14991 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:09.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.658+0000 7fcbfce93b80 1 -- 172.21.15.70:0/3112059079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d1402d8380 con 0x55d140253210 2024-06-04T04:56:09.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.658+0000 7fcbcb7fe700 1 -- 172.21.15.70:0/3112059079 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbe4021030 con 0x55d140253210 2024-06-04T04:56:09.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.659+0000 7fcbcb7fe700 1 -- 172.21.15.70:0/3112059079 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fcbe0022430 con 0x55d140253210 2024-06-04T04:56:09.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.659+0000 7fcbfce93b80 10 client.14991 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:09.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.659+0000 7fcbfce93b80 10 client.14991 resolve_mds: resolved name 'f' to daemon mds.f 2024-06-04T04:56:09.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.659+0000 7fcbfce93b80 20 client.14991 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:09.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.659+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/3112059079 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d1402dcf60 0x55d1402dbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:09.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.659+0000 7fcbfce93b80 4 client.14991 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-04T04:56:09.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.659+0000 7fcbfce93b80 1 -- 172.21.15.70:0/3112059079 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "status"}) -- 0x55d1401ac450 con 0x55d1402dcf60 2024-06-04T04:56:09.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.659+0000 7fcbebfff700 1 --2- 172.21.15.70:0/3112059079 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d1402dcf60 0x55d1402dbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:09.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.659+0000 7fcbebfff700 1 --2- 172.21.15.70:0/3112059079 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d1402dcf60 0x55d1402dbc70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=78ae279e009f11b4 server_cookie=2212226fa0dda8fa in_seq=0 out_seq=0 2024-06-04T04:56:09.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.659+0000 7fcbcb7fe700 10 client.14991 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T04:56:09.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.659+0000 7fcbcb7fe700 1 -- 172.21.15.70:0/3112059079 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x55d1401ac450 con 0x55d1402dcf60 2024-06-04T04:56:09.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.659+0000 7fcbcb7fe700 10 client.14991 handle_command_reply: tid=0 2024-06-04T04:56:09.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.660+0000 7fcbfce93b80 1 client.14991 shutdown 2024-06-04T04:56:09.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.660+0000 7fcbfce93b80 1 -- 172.21.15.70:0/3112059079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d140253210 msgr2=0x55d140276f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:09.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.660+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/3112059079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d140253210 0x55d140276f40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0006ed0 tx=0x7fcbe0004860 comp rx=0 tx=0).stop 2024-06-04T04:56:09.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.660+0000 7fcbfce93b80 1 -- 172.21.15.70:0/3112059079 shutdown_connections 2024-06-04T04:56:09.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.660+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/3112059079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1402582d0 0x55d140272eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.660+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/3112059079 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d1402dcf60 0x55d1402dbc70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.660+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/3112059079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d140252850 0x55d140276a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.660+0000 7fcbfce93b80 1 --2- 172.21.15.70:0/3112059079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d140253210 0x55d140276f40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.660+0000 7fcbfce93b80 1 -- 172.21.15.70:0/3112059079 >> 172.21.15.70:0/3112059079 conn(0x55d14017e1f0 msgr2=0x55d140256590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:09.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.660+0000 7fcbcb7fe700 0 client.14991 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T04:56:09.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.660+0000 7fcbfce93b80 1 -- 172.21.15.70:0/3112059079 shutdown_connections 2024-06-04T04:56:09.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.660+0000 7fcbfce93b80 1 -- 172.21.15.70:0/3112059079 wait complete. 2024-06-04T04:56:09.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.660+0000 7fcbfce93b80 20 client.14991 trim_cache size 0 max 16384 2024-06-04T04:56:09.667 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:56:09.668 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:56:09.668 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 3, 2024-06-04T04:56:09.668 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24436, 2024-06-04T04:56:09.668 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-06-04T04:56:09.668 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-06-04T04:56:09.668 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-06-04T04:56:09.668 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 137.22894176, 2024-06-04T04:56:09.668 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-06-04T04:56:09.668 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 81, 2024-06-04T04:56:09.668 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 79, 2024-06-04T04:56:09.668 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 185.376082271 2024-06-04T04:56:09.668 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:56:09.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.660+0000 7fcbf25fe700 1 -- 172.21.15.70:0/2276778386 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fcbd40b2880 msgr2=0x7fcbd40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:09.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.660+0000 7fcbf25fe700 1 --2- 172.21.15.70:0/2276778386 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fcbd40b2880 0x7fcbd40b4d40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcbec079b60 tx=0x7fcbe0009190 comp rx=0 tx=0).stop 2024-06-04T04:56:09.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.660+0000 7fcbf25fe700 1 -- 172.21.15.70:0/2276778386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcbec102470 msgr2=0x7fcbec19e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:09.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.660+0000 7fcbf25fe700 1 --2- 172.21.15.70:0/2276778386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcbec102470 0x7fcbec19e6d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00d980 tx=0x7fcbdc00dc90 comp rx=0 tx=0).stop 2024-06-04T04:56:09.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.661+0000 7fcbf25fe700 1 -- 172.21.15.70:0/2276778386 shutdown_connections 2024-06-04T04:56:09.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.661+0000 7fcbf25fe700 1 --2- 172.21.15.70:0/2276778386 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fcbd40b2880 0x7fcbd40b4d40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.661+0000 7fcbf25fe700 1 --2- 172.21.15.70:0/2276778386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec108550 0x7fcbec198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.661+0000 7fcbf25fe700 1 --2- 172.21.15.70:0/2276778386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcbec102470 0x7fcbec19e6d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.661+0000 7fcbf25fe700 1 --2- 172.21.15.70:0/2276778386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbec102e30 0x7fcbec19ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:09.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.661+0000 7fcbf25fe700 1 -- 172.21.15.70:0/2276778386 >> 172.21.15.70:0/2276778386 conn(0x7fcbec079480 msgr2=0x7fcbec100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:09.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.661+0000 7fcbf25fe700 1 -- 172.21.15.70:0/2276778386 shutdown_connections 2024-06-04T04:56:09.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:09.661+0000 7fcbf25fe700 1 -- 172.21.15.70:0/2276778386 wait complete. 2024-06-04T04:56:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:09 smithi112 ceph-mon[36453]: pgmap v366: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:56:09.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:09 smithi098 ceph-mon[35569]: pgmap v366: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:56:10.400 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-06-04T04:56:10.684 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:11.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:11 smithi112 ceph-mon[36453]: pgmap v367: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:56:11.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:11 smithi098 ceph-mon[35569]: pgmap v367: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:56:11.838 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:11 smithi070 ceph-mon[32631]: pgmap v367: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:56:12.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.351+0000 7fd9840ec700 1 -- 172.21.15.70:0/452128923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd97c1088f0 msgr2=0x7fd97c110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:12.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.351+0000 7fd9840ec700 1 --2- 172.21.15.70:0/452128923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd97c1088f0 0x7fd97c110470 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd97400b0d0 tx=0x7fd97400b3e0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.351+0000 7fd9840ec700 1 -- 172.21.15.70:0/452128923 shutdown_connections 2024-06-04T04:56:12.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.351+0000 7fd9840ec700 1 --2- 172.21.15.70:0/452128923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd97c1088f0 0x7fd97c110470 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.351+0000 7fd9840ec700 1 --2- 172.21.15.70:0/452128923 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd97c1037f0 0x7fd97c103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.351+0000 7fd9840ec700 1 --2- 172.21.15.70:0/452128923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd97c1041a0 0x7fd97c1081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.351+0000 7fd9840ec700 1 -- 172.21.15.70:0/452128923 >> 172.21.15.70:0/452128923 conn(0x7fd97c07a6a0 msgr2=0x7fd97c07aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:12.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.352+0000 7fd9840ec700 1 -- 172.21.15.70:0/452128923 shutdown_connections 2024-06-04T04:56:12.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.352+0000 7fd9840ec700 1 -- 172.21.15.70:0/452128923 wait complete. 2024-06-04T04:56:12.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.352+0000 7fd9840ec700 1 Processor -- start 2024-06-04T04:56:12.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.352+0000 7fd9840ec700 1 -- start start 2024-06-04T04:56:12.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.352+0000 7fd9840ec700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd97c1037f0 0x7fd97c075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.352+0000 7fd9840ec700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd97c1041a0 0x7fd97c075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.352+0000 7fd9840ec700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd97c1088f0 0x7fd97c0762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.352+0000 7fd9840ec700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd97c113660 con 0x7fd97c1088f0 2024-06-04T04:56:12.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.352+0000 7fd9840ec700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd97c1134e0 con 0x7fd97c1037f0 2024-06-04T04:56:12.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.352+0000 7fd9840ec700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd97c1137e0 con 0x7fd97c1041a0 2024-06-04T04:56:12.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd982689700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd97c1088f0 0x7fd97c0762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd982689700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd97c1088f0 0x7fd97c0762c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34350/0 (socket says 172.21.15.70:34350) 2024-06-04T04:56:12.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd982689700 1 -- 172.21.15.70:0/2058650537 learned_addr learned my addr 172.21.15.70:0/2058650537 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:12.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd981e88700 1 --2- 172.21.15.70:0/2058650537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd97c1037f0 0x7fd97c075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd981687700 1 --2- 172.21.15.70:0/2058650537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd97c1041a0 0x7fd97c075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd982689700 1 -- 172.21.15.70:0/2058650537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd97c1041a0 msgr2=0x7fd97c075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:12.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd982689700 1 --2- 172.21.15.70:0/2058650537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd97c1041a0 0x7fd97c075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd982689700 1 -- 172.21.15.70:0/2058650537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd97c1037f0 msgr2=0x7fd97c075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:12.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd982689700 1 --2- 172.21.15.70:0/2058650537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd97c1037f0 0x7fd97c075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd982689700 1 -- 172.21.15.70:0/2058650537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd97c1a8100 con 0x7fd97c1088f0 2024-06-04T04:56:12.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd981e88700 1 --2- 172.21.15.70:0/2058650537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd97c1037f0 0x7fd97c075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:12.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd981687700 1 --2- 172.21.15.70:0/2058650537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd97c1041a0 0x7fd97c075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:12.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd982689700 1 --2- 172.21.15.70:0/2058650537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd97c1088f0 0x7fd97c0762c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd974007d90 tx=0x7fd974007e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:12.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd972ffd700 1 -- 172.21.15.70:0/2058650537 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97401f030 con 0x7fd97c1088f0 2024-06-04T04:56:12.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd972ffd700 1 -- 172.21.15.70:0/2058650537 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd97401bb90 con 0x7fd97c1088f0 2024-06-04T04:56:12.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd9840ec700 1 -- 172.21.15.70:0/2058650537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd97c1a8390 con 0x7fd97c1088f0 2024-06-04T04:56:12.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd972ffd700 1 -- 172.21.15.70:0/2058650537 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd974012c90 con 0x7fd97c1088f0 2024-06-04T04:56:12.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.353+0000 7fd9840ec700 1 -- 172.21.15.70:0/2058650537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd97c1a8890 con 0x7fd97c1088f0 2024-06-04T04:56:12.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.356+0000 7fd972ffd700 1 -- 172.21.15.70:0/2058650537 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fd974022030 con 0x7fd97c1088f0 2024-06-04T04:56:12.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.357+0000 7fd972ffd700 1 --2- 172.21.15.70:0/2058650537 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd9680b2900 0x7fd9680b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.357+0000 7fd981e88700 1 --2- 172.21.15.70:0/2058650537 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd9680b2900 0x7fd9680b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.357+0000 7fd98e981b80 1 Processor -- start 2024-06-04T04:56:12.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.357+0000 7fd972ffd700 1 -- 172.21.15.70:0/2058650537 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fd97400f020 con 0x7fd97c1088f0 2024-06-04T04:56:12.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd98e981b80 1 -- start start 2024-06-04T04:56:12.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd981e88700 1 --2- 172.21.15.70:0/2058650537 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd9680b2900 0x7fd9680b4dc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd9780099d0 tx=0x7fd978008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:12.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd98e981b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b7217b2d0 0x562b7217b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd98e981b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b72175850 0x562b72175cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd98e981b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72176210 0x562b72182030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd98e981b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562b720d3d20 con 0x562b72176210 2024-06-04T04:56:12.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd98e981b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562b720c38c0 con 0x562b7217b2d0 2024-06-04T04:56:12.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd98e981b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562b720c7900 con 0x562b72175850 2024-06-04T04:56:12.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd982689700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72176210 0x562b72182030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd982689700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72176210 0x562b72182030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34358/0 (socket says 172.21.15.70:34358) 2024-06-04T04:56:12.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd982689700 1 -- 172.21.15.70:0/1567936145 learned_addr learned my addr 172.21.15.70:0/1567936145 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:12.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd981e88700 1 --2- 172.21.15.70:0/1567936145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b7217b2d0 0x562b7217b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd981687700 1 --2- 172.21.15.70:0/1567936145 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b72175850 0x562b72175cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd982689700 1 -- 172.21.15.70:0/1567936145 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b72175850 msgr2=0x562b72175cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:12.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd982689700 1 --2- 172.21.15.70:0/1567936145 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b72175850 0x562b72175cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd982689700 1 -- 172.21.15.70:0/1567936145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b7217b2d0 msgr2=0x562b7217b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:12.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd982689700 1 --2- 172.21.15.70:0/1567936145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b7217b2d0 0x562b7217b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd982689700 1 -- 172.21.15.70:0/1567936145 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b7206cf60 con 0x562b72176210 2024-06-04T04:56:12.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd981687700 1 --2- 172.21.15.70:0/1567936145 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b72175850 0x562b72175cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:56:12.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.358+0000 7fd982689700 1 --2- 172.21.15.70:0/1567936145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72176210 0x562b72182030 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd9740aa880 tx=0x7fd9740099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2301699ad7733d5d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:12.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.359+0000 7fd967fff700 1 -- 172.21.15.70:0/1567936145 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9740aa690 con 0x562b72176210 2024-06-04T04:56:12.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.359+0000 7fd967fff700 1 -- 172.21.15.70:0/1567936145 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd9740b3280 con 0x562b72176210 2024-06-04T04:56:12.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.359+0000 7fd967fff700 1 -- 172.21.15.70:0/1567936145 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9740075d0 con 0x562b72176210 2024-06-04T04:56:12.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.359+0000 7fd98e981b80 1 -- 172.21.15.70:0/1567936145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72176210 msgr2=0x562b72182030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:12.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.359+0000 7fd98e981b80 1 --2- 172.21.15.70:0/1567936145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72176210 0x562b72182030 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd9740aa880 tx=0x7fd9740099e0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.359+0000 7fd98e981b80 1 -- 172.21.15.70:0/1567936145 shutdown_connections 2024-06-04T04:56:12.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.359+0000 7fd98e981b80 1 --2- 172.21.15.70:0/1567936145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72176210 0x562b72182030 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.359+0000 7fd98e981b80 1 --2- 172.21.15.70:0/1567936145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b7217b2d0 0x562b7217b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.359+0000 7fd98e981b80 1 --2- 172.21.15.70:0/1567936145 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b72175850 0x562b72175cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.359+0000 7fd98e981b80 1 -- 172.21.15.70:0/1567936145 >> 172.21.15.70:0/1567936145 conn(0x562b720a11f0 msgr2=0x562b72170390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:12.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.359+0000 7fd98e981b80 1 -- 172.21.15.70:0/1567936145 shutdown_connections 2024-06-04T04:56:12.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.359+0000 7fd98e981b80 1 -- 172.21.15.70:0/1567936145 wait complete. 2024-06-04T04:56:12.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.360+0000 7fd98e981b80 1 -- start start 2024-06-04T04:56:12.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.360+0000 7fd98e981b80 1 Processor -- start 2024-06-04T04:56:12.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.360+0000 7fd98e981b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b72175850 0x562b72184c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.360+0000 7fd98e981b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b72176210 0x562b721851d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.360+0000 7fd98e981b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b7217b2d0 0x562b7217f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.360+0000 7fd98e981b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562b71fc9f50 con 0x562b7217b2d0 2024-06-04T04:56:12.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.360+0000 7fd98e981b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562b7202bd50 con 0x562b72176210 2024-06-04T04:56:12.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.360+0000 7fd98e981b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562b720d3d20 con 0x562b72175850 2024-06-04T04:56:12.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.360+0000 7fd98e981b80 10 client.0 _ll_register_callbacks cb 0x562b7205e900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:12.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd982689700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b7217b2d0 0x562b7217f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd982689700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b7217b2d0 0x562b7217f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34364/0 (socket says 172.21.15.70:34364) 2024-06-04T04:56:12.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd982689700 1 -- 172.21.15.70:0/1577812247 learned_addr learned my addr 172.21.15.70:0/1577812247 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:12.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd981e88700 1 --2- 172.21.15.70:0/1577812247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b72175850 0x562b72184c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd981687700 1 --2- 172.21.15.70:0/1577812247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b72176210 0x562b721851d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd982689700 1 -- 172.21.15.70:0/1577812247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b72175850 msgr2=0x562b72184c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:12.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd982689700 1 --2- 172.21.15.70:0/1577812247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b72175850 0x562b72184c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd982689700 1 -- 172.21.15.70:0/1577812247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b72176210 msgr2=0x562b721851d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:12.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd982689700 1 --2- 172.21.15.70:0/1577812247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b72176210 0x562b721851d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd982689700 1 -- 172.21.15.70:0/1577812247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b72013880 con 0x562b7217b2d0 2024-06-04T04:56:12.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd981687700 1 --2- 172.21.15.70:0/1577812247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b72176210 0x562b721851d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:56:12.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd982689700 1 --2- 172.21.15.70:0/1577812247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b7217b2d0 0x562b7217f010 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd97401bfd0 tx=0x7fd9740aa690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90581b774f8876a6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:12.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd9657fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:12.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd9657fa700 1 -- 172.21.15.70:0/1577812247 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9740c3030 con 0x562b7217b2d0 2024-06-04T04:56:12.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd98e981b80 1 -- 172.21.15.70:0/1577812247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9740b3280 con 0x562b7217b2d0 2024-06-04T04:56:12.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd9657fa700 1 -- 172.21.15.70:0/1577812247 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fd9740a7170 con 0x562b7217b2d0 2024-06-04T04:56:12.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd9657fa700 1 -- 172.21.15.70:0/1577812247 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd9740ab280 con 0x562b7217b2d0 2024-06-04T04:56:12.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd98e981b80 10 client.15003 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:12.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd98e981b80 1 -- 172.21.15.70:0/1577812247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562b7207d710 con 0x562b7217b2d0 2024-06-04T04:56:12.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.361+0000 7fd9657fa700 1 -- 172.21.15.70:0/1577812247 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd9740b3280 con 0x562b7217b2d0 2024-06-04T04:56:12.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.362+0000 7fd9657fa700 1 -- 172.21.15.70:0/1577812247 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fd974011070 con 0x562b7217b2d0 2024-06-04T04:56:12.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.362+0000 7fd98e981b80 10 client.15003 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:12.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.362+0000 7fd98e981b80 10 client.15003 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-04T04:56:12.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.362+0000 7fd98e981b80 20 client.15003 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:12.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.362+0000 7fd98e981b80 1 --2- 172.21.15.70:0/1577812247 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x562b7218eb20 0x562b7218ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.362+0000 7fd98e981b80 4 client.15003 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:56:12.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.362+0000 7fd98e981b80 1 -- 172.21.15.70:0/1577812247 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b720d5c70 con 0x562b7218eb20 2024-06-04T04:56:12.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.362+0000 7fd981e88700 1 --2- 172.21.15.70:0/1577812247 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x562b7218eb20 0x562b7218ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.362+0000 7fd9657fa700 10 client.15003 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T04:56:12.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.362+0000 7fd981e88700 1 --2- 172.21.15.70:0/1577812247 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x562b7218eb20 0x562b7218ff90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=207119c3c0e12196 server_cookie=fb05c9fe860d72b0 in_seq=0 out_seq=0 2024-06-04T04:56:12.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.363+0000 7fd9657fa700 1 -- 172.21.15.70:0/1577812247 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562b720d5c70 con 0x562b7218eb20 2024-06-04T04:56:12.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.363+0000 7fd9657fa700 10 client.15003 handle_command_reply: tid=0 2024-06-04T04:56:12.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.363+0000 7fd98e981b80 1 client.15003 shutdown 2024-06-04T04:56:12.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.363+0000 7fd98e981b80 1 -- 172.21.15.70:0/1577812247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b7217b2d0 msgr2=0x562b7217f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:12.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.363+0000 7fd98e981b80 1 --2- 172.21.15.70:0/1577812247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b7217b2d0 0x562b7217f010 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd97401bfd0 tx=0x7fd9740aa690 comp rx=0 tx=0).stop 2024-06-04T04:56:12.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.364+0000 7fd98e981b80 1 -- 172.21.15.70:0/1577812247 shutdown_connections 2024-06-04T04:56:12.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.364+0000 7fd98e981b80 1 --2- 172.21.15.70:0/1577812247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b7217b2d0 0x562b7217f010 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.364+0000 7fd98e981b80 1 --2- 172.21.15.70:0/1577812247 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x562b7218eb20 0x562b7218ff90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.364+0000 7fd98e981b80 1 --2- 172.21.15.70:0/1577812247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b72175850 0x562b72184c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.364+0000 7fd98e981b80 1 --2- 172.21.15.70:0/1577812247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b72176210 0x562b721851d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.364+0000 7fd9657fa700 0 client.15003 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T04:56:12.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.364+0000 7fd98e981b80 1 -- 172.21.15.70:0/1577812247 >> 172.21.15.70:0/1577812247 conn(0x562b720a11f0 msgr2=0x562b7216faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:12.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.364+0000 7fd98e981b80 1 -- 172.21.15.70:0/1577812247 shutdown_connections 2024-06-04T04:56:12.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.364+0000 7fd98e981b80 1 -- 172.21.15.70:0/1577812247 wait complete. 2024-06-04T04:56:12.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.364+0000 7fd98e981b80 20 client.15003 trim_cache size 0 max 16384 2024-06-04T04:56:12.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.385+0000 7fd98e981b80 1 Processor -- start 2024-06-04T04:56:12.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.385+0000 7fd98e981b80 1 -- start start 2024-06-04T04:56:12.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.385+0000 7fd98e981b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72175850 0x562b7217efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.385+0000 7fd98e981b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b72176210 0x562b72170220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.385+0000 7fd98e981b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b7217b2d0 0x562b721993f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.385+0000 7fd98e981b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562b7218e2d0 con 0x562b72175850 2024-06-04T04:56:12.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.385+0000 7fd98e981b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562b7218e9a0 con 0x562b72176210 2024-06-04T04:56:12.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.385+0000 7fd98e981b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562b7218f070 con 0x562b7217b2d0 2024-06-04T04:56:12.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.386+0000 7fd981e88700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72175850 0x562b7217efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.386+0000 7fd981e88700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72175850 0x562b7217efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34366/0 (socket says 172.21.15.70:34366) 2024-06-04T04:56:12.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.386+0000 7fd981e88700 1 -- 172.21.15.70:0/4275400247 learned_addr learned my addr 172.21.15.70:0/4275400247 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:12.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.386+0000 7fd982689700 1 --2- 172.21.15.70:0/4275400247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b7217b2d0 0x562b721993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.386+0000 7fd981687700 1 --2- 172.21.15.70:0/4275400247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b72176210 0x562b72170220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.386+0000 7fd981e88700 1 -- 172.21.15.70:0/4275400247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b7217b2d0 msgr2=0x562b721993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:12.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.386+0000 7fd981e88700 1 --2- 172.21.15.70:0/4275400247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b7217b2d0 0x562b721993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.386+0000 7fd981e88700 1 -- 172.21.15.70:0/4275400247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b72176210 msgr2=0x562b72170220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:12.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.386+0000 7fd981e88700 1 --2- 172.21.15.70:0/4275400247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b72176210 0x562b72170220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.386+0000 7fd981e88700 1 -- 172.21.15.70:0/4275400247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b7207d710 con 0x562b72175850 2024-06-04T04:56:12.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.386+0000 7fd981e88700 1 --2- 172.21.15.70:0/4275400247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72175850 0x562b7217efd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd978009370 tx=0x7fd978013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802f0e7522021a4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.387+0000 7fd9677fe700 1 -- 172.21.15.70:0/4275400247 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd978014790 con 0x562b72175850 2024-06-04T04:56:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.387+0000 7fd9677fe700 1 -- 172.21.15.70:0/4275400247 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd97802e070 con 0x562b72175850 2024-06-04T04:56:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.387+0000 7fd9677fe700 1 -- 172.21.15.70:0/4275400247 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9780285b0 con 0x562b72175850 2024-06-04T04:56:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.388+0000 7fd98e981b80 1 -- 172.21.15.70:0/4275400247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72175850 msgr2=0x562b7217efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.388+0000 7fd98e981b80 1 --2- 172.21.15.70:0/4275400247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72175850 0x562b7217efd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd978009370 tx=0x7fd978013000 comp rx=0 tx=0).stop 2024-06-04T04:56:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.388+0000 7fd98e981b80 1 -- 172.21.15.70:0/4275400247 shutdown_connections 2024-06-04T04:56:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.388+0000 7fd98e981b80 1 --2- 172.21.15.70:0/4275400247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b7217b2d0 0x562b721993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.388+0000 7fd98e981b80 1 --2- 172.21.15.70:0/4275400247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72175850 0x562b7217efd0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.388+0000 7fd98e981b80 1 --2- 172.21.15.70:0/4275400247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b72176210 0x562b72170220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.388+0000 7fd98e981b80 1 -- 172.21.15.70:0/4275400247 >> 172.21.15.70:0/4275400247 conn(0x562b720a11f0 msgr2=0x562b7217e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.388+0000 7fd98e981b80 1 -- 172.21.15.70:0/4275400247 shutdown_connections 2024-06-04T04:56:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.388+0000 7fd98e981b80 1 -- 172.21.15.70:0/4275400247 wait complete. 2024-06-04T04:56:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.388+0000 7fd98e981b80 1 -- start start 2024-06-04T04:56:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.388+0000 7fd98e981b80 1 Processor -- start 2024-06-04T04:56:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.388+0000 7fd98e981b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b72175850 0x562b721879c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd98e981b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72176210 0x562b72187f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd98e981b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b7217b2d0 0x562b721f92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd98e981b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562b7218fe10 con 0x562b72176210 2024-06-04T04:56:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd98e981b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562b7218f740 con 0x562b7217b2d0 2024-06-04T04:56:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd98e981b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562b721904e0 con 0x562b72175850 2024-06-04T04:56:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd98e981b80 10 client.0 _ll_register_callbacks cb 0x562b7205e900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd981687700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72176210 0x562b72187f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd981687700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72176210 0x562b72187f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34374/0 (socket says 172.21.15.70:34374) 2024-06-04T04:56:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd981e88700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b72175850 0x562b721879c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd981687700 1 -- 172.21.15.70:0/267873575 learned_addr learned my addr 172.21.15.70:0/267873575 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd982689700 1 --2- 172.21.15.70:0/267873575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b7217b2d0 0x562b721f92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd981687700 1 -- 172.21.15.70:0/267873575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b72175850 msgr2=0x562b721879c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd981687700 1 --2- 172.21.15.70:0/267873575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b72175850 0x562b721879c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd981687700 1 -- 172.21.15.70:0/267873575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b7217b2d0 msgr2=0x562b721f92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd981687700 1 --2- 172.21.15.70:0/267873575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b7217b2d0 0x562b721f92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd981687700 1 -- 172.21.15.70:0/267873575 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b72171dc0 con 0x562b72176210 2024-06-04T04:56:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd981687700 1 --2- 172.21.15.70:0/267873575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72176210 0x562b72187f00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd97c103210 tx=0x7fd96c012830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988f6a0efcbcbd5e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd9657fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd9657fa700 1 -- 172.21.15.70:0/267873575 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96c01a070 con 0x562b72176210 2024-06-04T04:56:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd9657fa700 1 -- 172.21.15.70:0/267873575 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fd96c015450 con 0x562b72176210 2024-06-04T04:56:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd98e981b80 1 -- 172.21.15.70:0/267873575 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd97802e070 con 0x562b72176210 2024-06-04T04:56:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd9657fa700 1 -- 172.21.15.70:0/267873575 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd96c026050 con 0x562b72176210 2024-06-04T04:56:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd98e981b80 10 client.15009 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.389+0000 7fd98e981b80 1 -- 172.21.15.70:0/267873575 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562b721fa7a0 con 0x562b72176210 2024-06-04T04:56:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.390+0000 7fd9657fa700 1 -- 172.21.15.70:0/267873575 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd97802e070 con 0x562b72176210 2024-06-04T04:56:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.390+0000 7fd9657fa700 1 -- 172.21.15.70:0/267873575 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fd96c01e6b0 con 0x562b72176210 2024-06-04T04:56:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.390+0000 7fd98e981b80 10 client.15009 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.390+0000 7fd98e981b80 10 client.15009 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-04T04:56:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.390+0000 7fd98e981b80 20 client.15009 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.390+0000 7fd98e981b80 1 --2- 172.21.15.70:0/267873575 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x562b721ff3d0 0x562b721fe0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.390+0000 7fd98e981b80 4 client.15009 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-04T04:56:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.390+0000 7fd98e981b80 1 -- 172.21.15.70:0/267873575 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "status"}) -- 0x562b720cf450 con 0x562b721ff3d0 2024-06-04T04:56:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.390+0000 7fd981e88700 1 --2- 172.21.15.70:0/267873575 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x562b721ff3d0 0x562b721fe0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.391+0000 7fd9657fa700 10 client.15009 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T04:56:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.391+0000 7fd981e88700 1 --2- 172.21.15.70:0/267873575 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x562b721ff3d0 0x562b721fe0e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3c9f6a619985a7d0 server_cookie=922b7a2400141be2 in_seq=0 out_seq=0 2024-06-04T04:56:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.391+0000 7fd9657fa700 1 -- 172.21.15.70:0/267873575 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x562b720cf450 con 0x562b721ff3d0 2024-06-04T04:56:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.391+0000 7fd9657fa700 10 client.15009 handle_command_reply: tid=0 2024-06-04T04:56:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.391+0000 7fd98e981b80 1 client.15009 shutdown 2024-06-04T04:56:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.391+0000 7fd98e981b80 1 -- 172.21.15.70:0/267873575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72176210 msgr2=0x562b72187f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.391+0000 7fd98e981b80 1 --2- 172.21.15.70:0/267873575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72176210 0x562b72187f00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd97c103210 tx=0x7fd96c012830 comp rx=0 tx=0).stop 2024-06-04T04:56:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.391+0000 7fd98e981b80 1 -- 172.21.15.70:0/267873575 shutdown_connections 2024-06-04T04:56:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.391+0000 7fd98e981b80 1 --2- 172.21.15.70:0/267873575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b7217b2d0 0x562b721f92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.391+0000 7fd98e981b80 1 --2- 172.21.15.70:0/267873575 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x562b721ff3d0 0x562b721fe0e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.391+0000 7fd98e981b80 1 --2- 172.21.15.70:0/267873575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b72175850 0x562b721879c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.391+0000 7fd98e981b80 1 --2- 172.21.15.70:0/267873575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b72176210 0x562b72187f00 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.391+0000 7fd9657fa700 0 client.15009 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T04:56:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.391+0000 7fd98e981b80 1 -- 172.21.15.70:0/267873575 >> 172.21.15.70:0/267873575 conn(0x562b720a11f0 msgr2=0x562b7217d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.391+0000 7fd98e981b80 1 -- 172.21.15.70:0/267873575 shutdown_connections 2024-06-04T04:56:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.391+0000 7fd98e981b80 1 -- 172.21.15.70:0/267873575 wait complete. 2024-06-04T04:56:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.391+0000 7fd98e981b80 20 client.15009 trim_cache size 0 max 16384 2024-06-04T04:56:12.398 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:56:12.398 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:56:12.398 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 2, 2024-06-04T04:56:12.398 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24440, 2024-06-04T04:56:12.398 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-06-04T04:56:12.398 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-06-04T04:56:12.398 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-06-04T04:56:12.399 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 140.971039596, 2024-06-04T04:56:12.399 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-06-04T04:56:12.399 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 81, 2024-06-04T04:56:12.399 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 79, 2024-06-04T04:56:12.399 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 168.757086273 2024-06-04T04:56:12.399 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:56:12.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.392+0000 7fd9840ec700 1 -- 172.21.15.70:0/2058650537 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd9680b2900 msgr2=0x7fd9680b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.392+0000 7fd9840ec700 1 --2- 172.21.15.70:0/2058650537 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd9680b2900 0x7fd9680b4dc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd9780099d0 tx=0x7fd978008040 comp rx=0 tx=0).stop 2024-06-04T04:56:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.392+0000 7fd9840ec700 1 -- 172.21.15.70:0/2058650537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd97c1088f0 msgr2=0x7fd97c0762c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.392+0000 7fd9840ec700 1 --2- 172.21.15.70:0/2058650537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd97c1088f0 0x7fd97c0762c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd974007d90 tx=0x7fd974007e70 comp rx=0 tx=0).stop 2024-06-04T04:56:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.392+0000 7fd9840ec700 1 -- 172.21.15.70:0/2058650537 shutdown_connections 2024-06-04T04:56:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.392+0000 7fd9840ec700 1 --2- 172.21.15.70:0/2058650537 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd9680b2900 0x7fd9680b4dc0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.392+0000 7fd9840ec700 1 --2- 172.21.15.70:0/2058650537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd97c1088f0 0x7fd97c0762c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.392+0000 7fd9840ec700 1 --2- 172.21.15.70:0/2058650537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd97c1037f0 0x7fd97c075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.392+0000 7fd9840ec700 1 --2- 172.21.15.70:0/2058650537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd97c1041a0 0x7fd97c075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.392+0000 7fd9840ec700 1 -- 172.21.15.70:0/2058650537 >> 172.21.15.70:0/2058650537 conn(0x7fd97c07a6a0 msgr2=0x7fd97c101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.392+0000 7fd9840ec700 1 -- 172.21.15.70:0/2058650537 shutdown_connections 2024-06-04T04:56:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:12.392+0000 7fd9840ec700 1 -- 172.21.15.70:0/2058650537 wait complete. 2024-06-04T04:56:13.163 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.c status 2024-06-04T04:56:13.407 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:13.681 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:13 smithi070 ceph-mon[32631]: pgmap v368: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:56:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:13 smithi112 ceph-mon[36453]: pgmap v368: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:56:13.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:13 smithi098 ceph-mon[35569]: pgmap v368: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:56:15.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.165+0000 7f3523880700 1 -- 172.21.15.70:0/1716847034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f351c1088f0 msgr2=0x7f351c110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:15.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.165+0000 7f3523880700 1 --2- 172.21.15.70:0/1716847034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f351c1088f0 0x7f351c110470 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f351400b0d0 tx=0x7f351400b3e0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.165+0000 7f3523880700 1 -- 172.21.15.70:0/1716847034 shutdown_connections 2024-06-04T04:56:15.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.165+0000 7f3523880700 1 --2- 172.21.15.70:0/1716847034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f351c1088f0 0x7f351c110470 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.165+0000 7f3523880700 1 --2- 172.21.15.70:0/1716847034 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f351c1037f0 0x7f351c103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.165+0000 7f3523880700 1 --2- 172.21.15.70:0/1716847034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f351c1041a0 0x7f351c1081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.165+0000 7f3523880700 1 -- 172.21.15.70:0/1716847034 >> 172.21.15.70:0/1716847034 conn(0x7f351c07a6a0 msgr2=0x7f351c07aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:15.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.165+0000 7f3523880700 1 -- 172.21.15.70:0/1716847034 shutdown_connections 2024-06-04T04:56:15.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.165+0000 7f3523880700 1 -- 172.21.15.70:0/1716847034 wait complete. 2024-06-04T04:56:15.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.165+0000 7f3523880700 1 Processor -- start 2024-06-04T04:56:15.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.165+0000 7f3523880700 1 -- start start 2024-06-04T04:56:15.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3523880700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f351c1037f0 0x7f351c19f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3523880700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f351c1041a0 0x7f351c19fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3523880700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f351c1088f0 0x7f351c1997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3523880700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f351c113590 con 0x7f351c1088f0 2024-06-04T04:56:15.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3523880700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f351c113410 con 0x7f351c1037f0 2024-06-04T04:56:15.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3523880700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f351c113710 con 0x7f351c1041a0 2024-06-04T04:56:15.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3521e1d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f351c1088f0 0x7f351c1997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3521e1d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f351c1088f0 0x7f351c1997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34396/0 (socket says 172.21.15.70:34396) 2024-06-04T04:56:15.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3521e1d700 1 -- 172.21.15.70:0/1544941792 learned_addr learned my addr 172.21.15.70:0/1544941792 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:15.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3520e1b700 1 --2- 172.21.15.70:0/1544941792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f351c1041a0 0x7f351c19fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f352161c700 1 --2- 172.21.15.70:0/1544941792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f351c1037f0 0x7f351c19f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3521e1d700 1 -- 172.21.15.70:0/1544941792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f351c1041a0 msgr2=0x7f351c19fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:15.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3521e1d700 1 --2- 172.21.15.70:0/1544941792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f351c1041a0 0x7f351c19fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3521e1d700 1 -- 172.21.15.70:0/1544941792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f351c1037f0 msgr2=0x7f351c19f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:15.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3521e1d700 1 --2- 172.21.15.70:0/1544941792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f351c1037f0 0x7f351c19f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3521e1d700 1 -- 172.21.15.70:0/1544941792 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f351c199fb0 con 0x7f351c1088f0 2024-06-04T04:56:15.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3521e1d700 1 --2- 172.21.15.70:0/1544941792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f351c1088f0 0x7f351c1997d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3514009850 tx=0x7f3514007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:15.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f35127fc700 1 -- 172.21.15.70:0/1544941792 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351401f070 con 0x7f351c1088f0 2024-06-04T04:56:15.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f35127fc700 1 -- 172.21.15.70:0/1544941792 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3514005500 con 0x7f351c1088f0 2024-06-04T04:56:15.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3523880700 1 -- 172.21.15.70:0/1544941792 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f351c19a240 con 0x7f351c1088f0 2024-06-04T04:56:15.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f35127fc700 1 -- 172.21.15.70:0/1544941792 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351401b8f0 con 0x7f351c1088f0 2024-06-04T04:56:15.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.166+0000 7f3523880700 1 -- 172.21.15.70:0/1544941792 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f351c1a64b0 con 0x7f351c1088f0 2024-06-04T04:56:15.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.169+0000 7f35127fc700 1 -- 172.21.15.70:0/1544941792 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f3514022070 con 0x7f351c1088f0 2024-06-04T04:56:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.170+0000 7f35127fc700 1 --2- 172.21.15.70:0/1544941792 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f35080b2880 0x7f35080b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.170+0000 7f352161c700 1 --2- 172.21.15.70:0/1544941792 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f35080b2880 0x7f35080b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.170+0000 7f35127fc700 1 -- 172.21.15.70:0/1544941792 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f35140e3150 con 0x7f351c1088f0 2024-06-04T04:56:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.170+0000 7f352e115b80 1 Processor -- start 2024-06-04T04:56:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.170+0000 7f352161c700 1 --2- 172.21.15.70:0/1544941792 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f35080b2880 0x7f35080b4d40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3518004630 tx=0x7f351800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.170+0000 7f352e115b80 1 -- start start 2024-06-04T04:56:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f352e115b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c10810 0x556954c10bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f352e115b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c111c0 0x556954c148f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f352e115b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c14f10 0x556954c15350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f352e115b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556954b6fc40 con 0x556954c14f10 2024-06-04T04:56:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f352e115b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556954b5f7e0 con 0x556954c111c0 2024-06-04T04:56:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f352e115b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556954b63820 con 0x556954c10810 2024-06-04T04:56:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f3521e1d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c14f10 0x556954c15350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f3521e1d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c14f10 0x556954c15350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34406/0 (socket says 172.21.15.70:34406) 2024-06-04T04:56:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f3521e1d700 1 -- 172.21.15.70:0/329242406 learned_addr learned my addr 172.21.15.70:0/329242406 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f352161c700 1 --2- 172.21.15.70:0/329242406 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c10810 0x556954c10bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f3520e1b700 1 --2- 172.21.15.70:0/329242406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c111c0 0x556954c148f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f3521e1d700 1 -- 172.21.15.70:0/329242406 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c10810 msgr2=0x556954c10bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f3521e1d700 1 --2- 172.21.15.70:0/329242406 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c10810 0x556954c10bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f3521e1d700 1 -- 172.21.15.70:0/329242406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c111c0 msgr2=0x556954c148f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f3521e1d700 1 --2- 172.21.15.70:0/329242406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c111c0 0x556954c148f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f3521e1d700 1 -- 172.21.15.70:0/329242406 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556954aaf880 con 0x556954c14f10 2024-06-04T04:56:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f3521e1d700 1 --2- 172.21.15.70:0/329242406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c14f10 0x556954c15350 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f35140a6860 tx=0x7f35140aa050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399ac8415b3825d2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f35077fe700 1 -- 172.21.15.70:0/329242406 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514029070 con 0x556954c14f10 2024-06-04T04:56:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f35077fe700 1 -- 172.21.15.70:0/329242406 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35140b10e0 con 0x556954c14f10 2024-06-04T04:56:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.171+0000 7f35077fe700 1 -- 172.21.15.70:0/329242406 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35140074e0 con 0x556954c14f10 2024-06-04T04:56:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.172+0000 7f352e115b80 1 -- 172.21.15.70:0/329242406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c14f10 msgr2=0x556954c15350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.172+0000 7f352e115b80 1 --2- 172.21.15.70:0/329242406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c14f10 0x556954c15350 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f35140a6860 tx=0x7f35140aa050 comp rx=0 tx=0).stop 2024-06-04T04:56:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.172+0000 7f352e115b80 1 -- 172.21.15.70:0/329242406 shutdown_connections 2024-06-04T04:56:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.172+0000 7f352e115b80 1 --2- 172.21.15.70:0/329242406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c14f10 0x556954c15350 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.172+0000 7f352e115b80 1 --2- 172.21.15.70:0/329242406 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c10810 0x556954c10bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.172+0000 7f352e115b80 1 --2- 172.21.15.70:0/329242406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c111c0 0x556954c148f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.172+0000 7f352e115b80 1 -- 172.21.15.70:0/329242406 >> 172.21.15.70:0/329242406 conn(0x556954b3e030 msgr2=0x556954c0ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.172+0000 7f352e115b80 1 -- 172.21.15.70:0/329242406 shutdown_connections 2024-06-04T04:56:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.172+0000 7f352e115b80 1 -- 172.21.15.70:0/329242406 wait complete. 2024-06-04T04:56:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.172+0000 7f352e115b80 1 -- start start 2024-06-04T04:56:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f352e115b80 1 Processor -- start 2024-06-04T04:56:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f352e115b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c10810 0x556954c1c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f352e115b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c111c0 0x556954c19660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f352e115b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c14f10 0x556954c19ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f352e115b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556954a65f50 con 0x556954c10810 2024-06-04T04:56:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f352e115b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556954b01770 con 0x556954c14f10 2024-06-04T04:56:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f352e115b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556954c1ce90 con 0x556954c111c0 2024-06-04T04:56:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f352e115b80 10 client.0 _ll_register_callbacks cb 0x556954afa8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f352161c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c10810 0x556954c1c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f352161c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c10810 0x556954c1c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34420/0 (socket says 172.21.15.70:34420) 2024-06-04T04:56:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f352161c700 1 -- 172.21.15.70:0/3098880658 learned_addr learned my addr 172.21.15.70:0/3098880658 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f3521e1d700 1 --2- 172.21.15.70:0/3098880658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c14f10 0x556954c19ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f3520e1b700 1 --2- 172.21.15.70:0/3098880658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c111c0 0x556954c19660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f352161c700 1 -- 172.21.15.70:0/3098880658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c111c0 msgr2=0x556954c19660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f352161c700 1 --2- 172.21.15.70:0/3098880658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c111c0 0x556954c19660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f352161c700 1 -- 172.21.15.70:0/3098880658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c14f10 msgr2=0x556954c19ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:15.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f352161c700 1 --2- 172.21.15.70:0/3098880658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c14f10 0x556954c19ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f352161c700 1 -- 172.21.15.70:0/3098880658 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556954afe640 con 0x556954c10810 2024-06-04T04:56:15.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f3520e1b700 1 --2- 172.21.15.70:0/3098880658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c111c0 0x556954c19660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:56:15.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f3521e1d700 1 --2- 172.21.15.70:0/3098880658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c14f10 0x556954c19ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:15.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f352161c700 1 --2- 172.21.15.70:0/3098880658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c10810 0x556954c1c360 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3518003e70 tx=0x7f35180047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c8e451f6e806ebb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:15.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.173+0000 7f3504ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:15.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.174+0000 7f3504ff9700 1 -- 172.21.15.70:0/3098880658 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3518011ce0 con 0x556954c10810 2024-06-04T04:56:15.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.174+0000 7f3504ff9700 1 -- 172.21.15.70:0/3098880658 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f3518028470 con 0x556954c10810 2024-06-04T04:56:15.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.174+0000 7f352e115b80 1 -- 172.21.15.70:0/3098880658 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35140b10e0 con 0x556954c10810 2024-06-04T04:56:15.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.174+0000 7f3504ff9700 1 -- 172.21.15.70:0/3098880658 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3518028670 con 0x556954c10810 2024-06-04T04:56:15.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.174+0000 7f352e115b80 10 client.15021 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:15.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.174+0000 7f352e115b80 1 -- 172.21.15.70:0/3098880658 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556954c27530 con 0x556954c10810 2024-06-04T04:56:15.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.174+0000 7f3504ff9700 1 -- 172.21.15.70:0/3098880658 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35140b10e0 con 0x556954c10810 2024-06-04T04:56:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.174+0000 7f3504ff9700 1 -- 172.21.15.70:0/3098880658 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f351801e430 con 0x556954c10810 2024-06-04T04:56:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.174+0000 7f352e115b80 10 client.15021 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.174+0000 7f352e115b80 10 client.15021 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-04T04:56:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.174+0000 7f352e115b80 20 client.15021 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.174+0000 7f352e115b80 1 --2- 172.21.15.70:0/3098880658 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x556954c29bb0 0x556954c2b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.174+0000 7f352e115b80 4 client.15021 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:56:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.174+0000 7f352e115b80 1 -- 172.21.15.70:0/3098880658 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556954bdc990 con 0x556954c29bb0 2024-06-04T04:56:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.174+0000 7f3520e1b700 1 --2- 172.21.15.70:0/3098880658 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x556954c29bb0 0x556954c2b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.175+0000 7f3520e1b700 1 --2- 172.21.15.70:0/3098880658 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x556954c29bb0 0x556954c2b020 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=41b3bf26f8c73e9e server_cookie=fae6003b389e5732 in_seq=0 out_seq=0 2024-06-04T04:56:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.175+0000 7f3504ff9700 10 client.15021 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T04:56:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.176+0000 7f3504ff9700 1 -- 172.21.15.70:0/3098880658 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556954bdc990 con 0x556954c29bb0 2024-06-04T04:56:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.176+0000 7f3504ff9700 10 client.15021 handle_command_reply: tid=0 2024-06-04T04:56:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.176+0000 7f352e115b80 1 client.15021 shutdown 2024-06-04T04:56:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.176+0000 7f352e115b80 1 -- 172.21.15.70:0/3098880658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c10810 msgr2=0x556954c1c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.176+0000 7f352e115b80 1 --2- 172.21.15.70:0/3098880658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c10810 0x556954c1c360 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3518003e70 tx=0x7f35180047c0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.176+0000 7f352e115b80 1 -- 172.21.15.70:0/3098880658 shutdown_connections 2024-06-04T04:56:15.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.176+0000 7f352e115b80 1 --2- 172.21.15.70:0/3098880658 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x556954c29bb0 0x556954c2b020 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.176+0000 7f352e115b80 1 --2- 172.21.15.70:0/3098880658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c14f10 0x556954c19ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.176+0000 7f352e115b80 1 --2- 172.21.15.70:0/3098880658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c10810 0x556954c1c360 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.176+0000 7f3504ff9700 0 client.15021 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T04:56:15.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.176+0000 7f352e115b80 1 --2- 172.21.15.70:0/3098880658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c111c0 0x556954c19660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.176+0000 7f352e115b80 1 -- 172.21.15.70:0/3098880658 >> 172.21.15.70:0/3098880658 conn(0x556954b3e030 msgr2=0x556954c0a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:15.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.176+0000 7f352e115b80 1 -- 172.21.15.70:0/3098880658 shutdown_connections 2024-06-04T04:56:15.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.176+0000 7f352e115b80 1 -- 172.21.15.70:0/3098880658 wait complete. 2024-06-04T04:56:15.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.176+0000 7f352e115b80 20 client.15021 trim_cache size 0 max 16384 2024-06-04T04:56:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.202+0000 7f352e115b80 1 Processor -- start 2024-06-04T04:56:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.202+0000 7f352e115b80 1 -- start start 2024-06-04T04:56:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.202+0000 7f352e115b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c10810 0x556954c13f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.202+0000 7f352e115b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c111c0 0x556954c0ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.202+0000 7f352e115b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c14f10 0x556954c34480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.202+0000 7f352e115b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556954c29360 con 0x556954c14f10 2024-06-04T04:56:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.202+0000 7f352e115b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556954c29a30 con 0x556954c111c0 2024-06-04T04:56:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.202+0000 7f352e115b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556954c2a100 con 0x556954c10810 2024-06-04T04:56:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.203+0000 7f3521e1d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c14f10 0x556954c34480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.203+0000 7f3521e1d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c14f10 0x556954c34480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34424/0 (socket says 172.21.15.70:34424) 2024-06-04T04:56:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.203+0000 7f3521e1d700 1 -- 172.21.15.70:0/3169671593 learned_addr learned my addr 172.21.15.70:0/3169671593 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.203+0000 7f352161c700 1 --2- 172.21.15.70:0/3169671593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c10810 0x556954c13f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.203+0000 7f3520e1b700 1 --2- 172.21.15.70:0/3169671593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c111c0 0x556954c0ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.203+0000 7f3521e1d700 1 -- 172.21.15.70:0/3169671593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c10810 msgr2=0x556954c13f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.203+0000 7f3521e1d700 1 --2- 172.21.15.70:0/3169671593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c10810 0x556954c13f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.203+0000 7f3521e1d700 1 -- 172.21.15.70:0/3169671593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c111c0 msgr2=0x556954c0ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.203+0000 7f3521e1d700 1 --2- 172.21.15.70:0/3169671593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c111c0 0x556954c0ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.203+0000 7f3521e1d700 1 -- 172.21.15.70:0/3169671593 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556954c260f0 con 0x556954c14f10 2024-06-04T04:56:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.203+0000 7f3521e1d700 1 --2- 172.21.15.70:0/3169671593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c14f10 0x556954c34480 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f35140aa4e0 tx=0x7f35140b1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a1067912dc1fc3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.203+0000 7f3506ffd700 1 -- 172.21.15.70:0/3169671593 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514029070 con 0x556954c14f10 2024-06-04T04:56:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.203+0000 7f3506ffd700 1 -- 172.21.15.70:0/3169671593 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35140aa7e0 con 0x556954c14f10 2024-06-04T04:56:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.203+0000 7f3506ffd700 1 -- 172.21.15.70:0/3169671593 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514005a40 con 0x556954c14f10 2024-06-04T04:56:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.204+0000 7f352e115b80 1 -- 172.21.15.70:0/3169671593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c14f10 msgr2=0x556954c34480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.204+0000 7f352e115b80 1 --2- 172.21.15.70:0/3169671593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c14f10 0x556954c34480 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f35140aa4e0 tx=0x7f35140b1040 comp rx=0 tx=0).stop 2024-06-04T04:56:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.204+0000 7f352e115b80 1 -- 172.21.15.70:0/3169671593 shutdown_connections 2024-06-04T04:56:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.204+0000 7f352e115b80 1 --2- 172.21.15.70:0/3169671593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c14f10 0x556954c34480 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.204+0000 7f352e115b80 1 --2- 172.21.15.70:0/3169671593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c10810 0x556954c13f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.204+0000 7f352e115b80 1 --2- 172.21.15.70:0/3169671593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c111c0 0x556954c0ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.204+0000 7f352e115b80 1 -- 172.21.15.70:0/3169671593 >> 172.21.15.70:0/3169671593 conn(0x556954b3e030 msgr2=0x556954c13860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.204+0000 7f352e115b80 1 -- 172.21.15.70:0/3169671593 shutdown_connections 2024-06-04T04:56:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.204+0000 7f352e115b80 1 -- 172.21.15.70:0/3169671593 wait complete. 2024-06-04T04:56:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.204+0000 7f352e115b80 1 -- start start 2024-06-04T04:56:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.204+0000 7f352e115b80 1 Processor -- start 2024-06-04T04:56:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.205+0000 7f352e115b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c10810 0x556954c30390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.205+0000 7f352e115b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c111c0 0x556954c308d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.205+0000 7f352e115b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c14f10 0x556954c30e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.205+0000 7f352e115b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556954c2aea0 con 0x556954c10810 2024-06-04T04:56:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.205+0000 7f352e115b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556954c2a7d0 con 0x556954c111c0 2024-06-04T04:56:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.205+0000 7f352e115b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556954c2b570 con 0x556954c14f10 2024-06-04T04:56:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.205+0000 7f352e115b80 10 client.0 _ll_register_callbacks cb 0x556954afa8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.205+0000 7f352161c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c10810 0x556954c30390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.205+0000 7f352161c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c10810 0x556954c30390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34434/0 (socket says 172.21.15.70:34434) 2024-06-04T04:56:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.205+0000 7f3520e1b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c111c0 0x556954c308d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.205+0000 7f352161c700 1 -- 172.21.15.70:0/1531655707 learned_addr learned my addr 172.21.15.70:0/1531655707 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.205+0000 7f3521e1d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c14f10 0x556954c30e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.206+0000 7f352161c700 1 -- 172.21.15.70:0/1531655707 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c14f10 msgr2=0x556954c30e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.206+0000 7f352161c700 1 --2- 172.21.15.70:0/1531655707 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c14f10 0x556954c30e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.206+0000 7f352161c700 1 -- 172.21.15.70:0/1531655707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c111c0 msgr2=0x556954c308d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.206+0000 7f352161c700 1 --2- 172.21.15.70:0/1531655707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c111c0 0x556954c308d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.206+0000 7f352161c700 1 -- 172.21.15.70:0/1531655707 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556954c945a0 con 0x556954c10810 2024-06-04T04:56:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.206+0000 7f3520e1b700 1 --2- 172.21.15.70:0/1531655707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c111c0 0x556954c308d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:56:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.206+0000 7f352161c700 1 --2- 172.21.15.70:0/1531655707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c10810 0x556954c30390 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f351801e8d0 tx=0x7f351800b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a541ea0d7fd5471b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.206+0000 7f3504ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.206+0000 7f3504ff9700 1 -- 172.21.15.70:0/1531655707 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3518040060 con 0x556954c10810 2024-06-04T04:56:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.206+0000 7f3504ff9700 1 -- 172.21.15.70:0/1531655707 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f351801ecd0 con 0x556954c10810 2024-06-04T04:56:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.206+0000 7f352e115b80 1 -- 172.21.15.70:0/1531655707 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556954b71b90 con 0x556954c10810 2024-06-04T04:56:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.206+0000 7f3504ff9700 1 -- 172.21.15.70:0/1531655707 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3518011720 con 0x556954c10810 2024-06-04T04:56:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.206+0000 7f352e115b80 10 client.15027 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.206+0000 7f352e115b80 1 -- 172.21.15.70:0/1531655707 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556954c95840 con 0x556954c10810 2024-06-04T04:56:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.206+0000 7f3504ff9700 1 -- 172.21.15.70:0/1531655707 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x556954b71b90 con 0x556954c10810 2024-06-04T04:56:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.207+0000 7f3504ff9700 1 -- 172.21.15.70:0/1531655707 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f351801f450 con 0x556954c10810 2024-06-04T04:56:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.207+0000 7f352e115b80 10 client.15027 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.207+0000 7f352e115b80 10 client.15027 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-04T04:56:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.207+0000 7f352e115b80 20 client.15027 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.207+0000 7f352e115b80 1 --2- 172.21.15.70:0/1531655707 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x556954c9a460 0x556954c99170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.207+0000 7f352e115b80 4 client.15027 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-04T04:56:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.207+0000 7f352e115b80 1 -- 172.21.15.70:0/1531655707 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "status"}) -- 0x556954b6b370 con 0x556954c9a460 2024-06-04T04:56:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.207+0000 7f3520e1b700 1 --2- 172.21.15.70:0/1531655707 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x556954c9a460 0x556954c99170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.207+0000 7f3520e1b700 1 --2- 172.21.15.70:0/1531655707 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x556954c9a460 0x556954c99170 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5c8bab2bb8f8643f server_cookie=29173be4445da95a in_seq=0 out_seq=0 2024-06-04T04:56:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.207+0000 7f3504ff9700 10 client.15027 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T04:56:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.208+0000 7f3504ff9700 1 -- 172.21.15.70:0/1531655707 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x556954b6b370 con 0x556954c9a460 2024-06-04T04:56:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.208+0000 7f3504ff9700 10 client.15027 handle_command_reply: tid=0 2024-06-04T04:56:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.208+0000 7f352e115b80 1 client.15027 shutdown 2024-06-04T04:56:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.208+0000 7f352e115b80 1 -- 172.21.15.70:0/1531655707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c10810 msgr2=0x556954c30390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.208+0000 7f352e115b80 1 --2- 172.21.15.70:0/1531655707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c10810 0x556954c30390 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f351801e8d0 tx=0x7f351800b650 comp rx=0 tx=0).stop 2024-06-04T04:56:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.208+0000 7f352e115b80 1 -- 172.21.15.70:0/1531655707 shutdown_connections 2024-06-04T04:56:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.208+0000 7f352e115b80 1 --2- 172.21.15.70:0/1531655707 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556954c14f10 0x556954c30e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.208+0000 7f352e115b80 1 --2- 172.21.15.70:0/1531655707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556954c10810 0x556954c30390 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.208+0000 7f352e115b80 1 --2- 172.21.15.70:0/1531655707 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x556954c9a460 0x556954c99170 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.208+0000 7f352e115b80 1 --2- 172.21.15.70:0/1531655707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556954c111c0 0x556954c308d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.208+0000 7f352e115b80 1 -- 172.21.15.70:0/1531655707 >> 172.21.15.70:0/1531655707 conn(0x556954b3e030 msgr2=0x556954c140f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.208+0000 7f3504ff9700 0 client.15027 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T04:56:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.208+0000 7f352e115b80 1 -- 172.21.15.70:0/1531655707 shutdown_connections 2024-06-04T04:56:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.208+0000 7f352e115b80 1 -- 172.21.15.70:0/1531655707 wait complete. 2024-06-04T04:56:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.208+0000 7f352e115b80 20 client.15027 trim_cache size 0 max 16384 2024-06-04T04:56:15.218 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:56:15.218 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:56:15.218 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 1, 2024-06-04T04:56:15.218 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24442, 2024-06-04T04:56:15.218 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-06-04T04:56:15.218 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-06-04T04:56:15.218 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-06-04T04:56:15.218 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 144.796523425, 2024-06-04T04:56:15.218 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-06-04T04:56:15.219 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 81, 2024-06-04T04:56:15.219 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 79, 2024-06-04T04:56:15.219 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 188.196909844 2024-06-04T04:56:15.219 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:56:15.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.209+0000 7f3523880700 1 -- 172.21.15.70:0/1544941792 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f35080b2880 msgr2=0x7f35080b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:15.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.209+0000 7f3523880700 1 --2- 172.21.15.70:0/1544941792 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f35080b2880 0x7f35080b4d40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3518004630 tx=0x7f351800b3c0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.209+0000 7f3523880700 1 -- 172.21.15.70:0/1544941792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f351c1088f0 msgr2=0x7f351c1997d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.209+0000 7f3523880700 1 --2- 172.21.15.70:0/1544941792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f351c1088f0 0x7f351c1997d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3514009850 tx=0x7f3514007d90 comp rx=0 tx=0).stop 2024-06-04T04:56:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.209+0000 7f3523880700 1 -- 172.21.15.70:0/1544941792 shutdown_connections 2024-06-04T04:56:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.209+0000 7f3523880700 1 --2- 172.21.15.70:0/1544941792 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f35080b2880 0x7f35080b4d40 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.209+0000 7f3523880700 1 --2- 172.21.15.70:0/1544941792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f351c1088f0 0x7f351c1997d0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.209+0000 7f3523880700 1 --2- 172.21.15.70:0/1544941792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f351c1037f0 0x7f351c19f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.209+0000 7f3523880700 1 --2- 172.21.15.70:0/1544941792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f351c1041a0 0x7f351c19fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.209+0000 7f3523880700 1 -- 172.21.15.70:0/1544941792 >> 172.21.15.70:0/1544941792 conn(0x7f351c07a6a0 msgr2=0x7f351c101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.209+0000 7f3523880700 1 -- 172.21.15.70:0/1544941792 shutdown_connections 2024-06-04T04:56:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:15.209+0000 7f3523880700 1 -- 172.21.15.70:0/1544941792 wait complete. 2024-06-04T04:56:15.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:15 smithi070 ceph-mon[32631]: pgmap v369: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:56:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:15 smithi112 ceph-mon[36453]: pgmap v369: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:56:15.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:15 smithi098 ceph-mon[35569]: pgmap v369: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:56:15.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-06-04T04:56:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:17 smithi112 ceph-mon[36453]: pgmap v370: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:17.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:17 smithi070 ceph-mon[32631]: pgmap v370: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:17.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:17 smithi098 ceph-mon[35569]: pgmap v370: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:17.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.916+0000 7f0cbf37e700 1 -- 172.21.15.70:0/56408301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0cb806bc90 msgr2=0x7f0cb806c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:17.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.916+0000 7f0cbf37e700 1 --2- 172.21.15.70:0/56408301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0cb806bc90 0x7f0cb806c110 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4009990 tx=0x7f0cb4009ca0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.916+0000 7f0cbf37e700 1 -- 172.21.15.70:0/56408301 shutdown_connections 2024-06-04T04:56:17.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.916+0000 7f0cbf37e700 1 --2- 172.21.15.70:0/56408301 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0cb806c650 0x7f0cb81103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.916+0000 7f0cbf37e700 1 --2- 172.21.15.70:0/56408301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb81095b0 0x7f0cb8109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.916+0000 7f0cbf37e700 1 --2- 172.21.15.70:0/56408301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0cb806bc90 0x7f0cb806c110 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.916+0000 7f0cbf37e700 1 -- 172.21.15.70:0/56408301 >> 172.21.15.70:0/56408301 conn(0x7f0cb8079410 msgr2=0x7f0cb8079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:17.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.916+0000 7f0cbf37e700 1 -- 172.21.15.70:0/56408301 shutdown_connections 2024-06-04T04:56:17.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.916+0000 7f0cbf37e700 1 -- 172.21.15.70:0/56408301 wait complete. 2024-06-04T04:56:17.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.917+0000 7f0cbf37e700 1 Processor -- start 2024-06-04T04:56:17.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.917+0000 7f0cbf37e700 1 -- start start 2024-06-04T04:56:17.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.917+0000 7f0cbf37e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb806bc90 0x7f0cb819f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.917+0000 7f0cbf37e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0cb806c650 0x7f0cb819fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.917+0000 7f0cbf37e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0cb81095b0 0x7f0cb81999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.917+0000 7f0cbf37e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb8113610 con 0x7f0cb806c650 2024-06-04T04:56:17.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.917+0000 7f0cbf37e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb8113490 con 0x7f0cb806bc90 2024-06-04T04:56:17.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.917+0000 7f0cbf37e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb8113790 con 0x7f0cb81095b0 2024-06-04T04:56:17.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.917+0000 7f0cbc919700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0cb806c650 0x7f0cb819fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:17.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.917+0000 7f0cbc919700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0cb806c650 0x7f0cb819fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59880/0 (socket says 172.21.15.70:59880) 2024-06-04T04:56:17.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.917+0000 7f0cbc919700 1 -- 172.21.15.70:0/1971277216 learned_addr learned my addr 172.21.15.70:0/1971277216 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:17.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.918+0000 7f0cbd11a700 1 --2- 172.21.15.70:0/1971277216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb806bc90 0x7f0cb819f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:17.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.918+0000 7f0cbc919700 1 -- 172.21.15.70:0/1971277216 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0cb81095b0 msgr2=0x7f0cb81999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:17.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.918+0000 7f0cbd91b700 1 --2- 172.21.15.70:0/1971277216 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0cb81095b0 0x7f0cb81999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:17.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.918+0000 7f0cbc919700 1 --2- 172.21.15.70:0/1971277216 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0cb81095b0 0x7f0cb81999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.918+0000 7f0cbc919700 1 -- 172.21.15.70:0/1971277216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb806bc90 msgr2=0x7f0cb819f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:17.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.918+0000 7f0cbc919700 1 --2- 172.21.15.70:0/1971277216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb806bc90 0x7f0cb819f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.918+0000 7f0cbc919700 1 -- 172.21.15.70:0/1971277216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb819a150 con 0x7f0cb806c650 2024-06-04T04:56:17.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.918+0000 7f0cbd11a700 1 --2- 172.21.15.70:0/1971277216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb806bc90 0x7f0cb819f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:56:17.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.918+0000 7f0cbc919700 1 --2- 172.21.15.70:0/1971277216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0cb806c650 0x7f0cb819fd70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4000c00 tx=0x7f0cb40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:17.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.918+0000 7f0caa7fc700 1 -- 172.21.15.70:0/1971277216 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb4027070 con 0x7f0cb806c650 2024-06-04T04:56:17.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.918+0000 7f0caa7fc700 1 -- 172.21.15.70:0/1971277216 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0cb400fdb0 con 0x7f0cb806c650 2024-06-04T04:56:17.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.918+0000 7f0caa7fc700 1 -- 172.21.15.70:0/1971277216 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb40173f0 con 0x7f0cb806c650 2024-06-04T04:56:17.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.918+0000 7f0cbf37e700 1 -- 172.21.15.70:0/1971277216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cb819a3e0 con 0x7f0cb806c650 2024-06-04T04:56:17.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.918+0000 7f0cbf37e700 1 -- 172.21.15.70:0/1971277216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cb81a6700 con 0x7f0cb806c650 2024-06-04T04:56:17.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.920+0000 7f0caa7fc700 1 -- 172.21.15.70:0/1971277216 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f0cb402a070 con 0x7f0cb806c650 2024-06-04T04:56:17.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.921+0000 7f0caa7fc700 1 --2- 172.21.15.70:0/1971277216 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0ca40b2880 0x7f0ca40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.921+0000 7f0cbd11a700 1 --2- 172.21.15.70:0/1971277216 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0ca40b2880 0x7f0ca40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:17.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.921+0000 7f0caa7fc700 1 -- 172.21.15.70:0/1971277216 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f0cb40e3310 con 0x7f0cb806c650 2024-06-04T04:56:17.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.922+0000 7f0cbd11a700 1 --2- 172.21.15.70:0/1971277216 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0ca40b2880 0x7f0ca40b4d40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0cb819ab30 tx=0x7f0cac008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:17.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.922+0000 7f0cc9c13b80 1 Processor -- start 2024-06-04T04:56:17.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.922+0000 7f0cc9c13b80 1 -- start start 2024-06-04T04:56:17.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.922+0000 7f0cc9c13b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123c9810 0x5559123c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.922+0000 7f0cc9c13b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.922+0000 7f0cc9c13b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123cdf10 0x5559123ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.922+0000 7f0cc9c13b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555912328c40 con 0x5559123ca1c0 2024-06-04T04:56:17.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.922+0000 7f0cc9c13b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5559123187e0 con 0x5559123cdf10 2024-06-04T04:56:17.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.922+0000 7f0cc9c13b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55591231c820 con 0x5559123c9810 2024-06-04T04:56:17.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.922+0000 7f0cbc919700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:17.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.922+0000 7f0cbc919700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59882/0 (socket says 172.21.15.70:59882) 2024-06-04T04:56:17.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.922+0000 7f0cbc919700 1 -- 172.21.15.70:0/2341731730 learned_addr learned my addr 172.21.15.70:0/2341731730 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:17.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.922+0000 7f0cbd11a700 1 --2- 172.21.15.70:0/2341731730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123c9810 0x5559123c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:17.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.923+0000 7f0cbc919700 1 -- 172.21.15.70:0/2341731730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123c9810 msgr2=0x5559123c9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:17.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.923+0000 7f0cbc919700 1 --2- 172.21.15.70:0/2341731730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123c9810 0x5559123c9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.923+0000 7f0cbc919700 1 -- 172.21.15.70:0/2341731730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123cdf10 msgr2=0x5559123ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:17.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.923+0000 7f0cbc919700 1 --2- 172.21.15.70:0/2341731730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123cdf10 0x5559123ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.923+0000 7f0cbc919700 1 -- 172.21.15.70:0/2341731730 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555912268880 con 0x5559123ca1c0 2024-06-04T04:56:17.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.923+0000 7f0cbc919700 1 --2- 172.21.15.70:0/2341731730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123cd8f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40a6a20 tx=0x7f0cb400ff60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9846a48aaaa118ec server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:17.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.923+0000 7f0ca37fe700 1 -- 172.21.15.70:0/2341731730 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb40ab0a0 con 0x5559123ca1c0 2024-06-04T04:56:17.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.923+0000 7f0ca37fe700 1 -- 172.21.15.70:0/2341731730 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0cb40aa9a0 con 0x5559123ca1c0 2024-06-04T04:56:17.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.923+0000 7f0ca37fe700 1 -- 172.21.15.70:0/2341731730 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb40b10d0 con 0x5559123ca1c0 2024-06-04T04:56:17.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.924+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/2341731730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 msgr2=0x5559123cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:17.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.924+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/2341731730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123cd8f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40a6a20 tx=0x7f0cb400ff60 comp rx=0 tx=0).stop 2024-06-04T04:56:17.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.924+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/2341731730 shutdown_connections 2024-06-04T04:56:17.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.924+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/2341731730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123cdf10 0x5559123ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.924+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/2341731730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123c9810 0x5559123c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.924+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/2341731730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123cd8f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.924+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/2341731730 >> 172.21.15.70:0/2341731730 conn(0x5559122f7030 msgr2=0x5559123c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:17.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.924+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/2341731730 shutdown_connections 2024-06-04T04:56:17.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.924+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/2341731730 wait complete. 2024-06-04T04:56:17.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.924+0000 7f0cc9c13b80 1 -- start start 2024-06-04T04:56:17.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.924+0000 7f0cc9c13b80 1 Processor -- start 2024-06-04T04:56:17.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cc9c13b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123c9810 0x5559123d5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cc9c13b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123d2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cc9c13b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123cdf10 0x5559123d2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cc9c13b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55591221ef50 con 0x5559123ca1c0 2024-06-04T04:56:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cc9c13b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5559122ba770 con 0x5559123c9810 2024-06-04T04:56:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cc9c13b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5559123d5e90 con 0x5559123cdf10 2024-06-04T04:56:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cc9c13b80 10 client.0 _ll_register_callbacks cb 0x5559122b38d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cbc919700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123d2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cbc919700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123d2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59886/0 (socket says 172.21.15.70:59886) 2024-06-04T04:56:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cbc919700 1 -- 172.21.15.70:0/1781921621 learned_addr learned my addr 172.21.15.70:0/1781921621 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cbd11a700 1 --2- 172.21.15.70:0/1781921621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123c9810 0x5559123d5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cbd91b700 1 --2- 172.21.15.70:0/1781921621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123cdf10 0x5559123d2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cbc919700 1 -- 172.21.15.70:0/1781921621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123cdf10 msgr2=0x5559123d2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cbc919700 1 --2- 172.21.15.70:0/1781921621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123cdf10 0x5559123d2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cbc919700 1 -- 172.21.15.70:0/1781921621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123c9810 msgr2=0x5559123d5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:17.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cbc919700 1 --2- 172.21.15.70:0/1781921621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123c9810 0x5559123d5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cbc919700 1 -- 172.21.15.70:0/1781921621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559122b7640 con 0x5559123ca1c0 2024-06-04T04:56:17.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cbd11a700 1 --2- 172.21.15.70:0/1781921621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123c9810 0x5559123d5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:17.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cbc919700 1 --2- 172.21.15.70:0/1781921621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123d2760 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40aa930 tx=0x7f0cb4009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef0857985d464a8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:17.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0ca0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:17.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0ca0ff9700 1 -- 172.21.15.70:0/1781921621 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb40c3070 con 0x5559123ca1c0 2024-06-04T04:56:17.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0ca0ff9700 1 -- 172.21.15.70:0/1781921621 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f0cb40a6f90 con 0x5559123ca1c0 2024-06-04T04:56:17.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/1781921621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cb40aa9a0 con 0x5559123ca1c0 2024-06-04T04:56:17.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0ca0ff9700 1 -- 172.21.15.70:0/1781921621 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0cb40b32e0 con 0x5559123ca1c0 2024-06-04T04:56:17.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.925+0000 7f0cc9c13b80 10 client.15039 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:17.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.926+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/1781921621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5559123e0600 con 0x5559123ca1c0 2024-06-04T04:56:17.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.926+0000 7f0ca0ff9700 1 -- 172.21.15.70:0/1781921621 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0cb40aa9a0 con 0x5559123ca1c0 2024-06-04T04:56:17.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.926+0000 7f0ca0ff9700 1 -- 172.21.15.70:0/1781921621 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f0cb40cbbc0 con 0x5559123ca1c0 2024-06-04T04:56:17.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.926+0000 7f0cc9c13b80 10 client.15039 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:17.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.926+0000 7f0cc9c13b80 10 client.15039 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-04T04:56:17.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.926+0000 7f0cc9c13b80 20 client.15039 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:17.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.926+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/1781921621 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5559123e2a10 0x5559123e3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.926+0000 7f0cc9c13b80 4 client.15039 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:56:17.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.926+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/1781921621 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555912395990 con 0x5559123e2a10 2024-06-04T04:56:17.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.926+0000 7f0cbd11a700 1 --2- 172.21.15.70:0/1781921621 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5559123e2a10 0x5559123e3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:17.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.927+0000 7f0ca0ff9700 10 client.15039 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:56:17.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.927+0000 7f0cbd11a700 1 --2- 172.21.15.70:0/1781921621 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5559123e2a10 0x5559123e3e80 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=346a886daa9c3137 server_cookie=c7d969fe0618fae8 in_seq=0 out_seq=0 2024-06-04T04:56:17.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.928+0000 7f0ca0ff9700 1 -- 172.21.15.70:0/1781921621 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555912395990 con 0x5559123e2a10 2024-06-04T04:56:17.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.928+0000 7f0ca0ff9700 10 client.15039 handle_command_reply: tid=0 2024-06-04T04:56:17.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.928+0000 7f0cc9c13b80 1 client.15039 shutdown 2024-06-04T04:56:17.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.928+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/1781921621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 msgr2=0x5559123d2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:17.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.928+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/1781921621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123d2760 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40aa930 tx=0x7f0cb4009510 comp rx=0 tx=0).stop 2024-06-04T04:56:17.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.928+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/1781921621 shutdown_connections 2024-06-04T04:56:17.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.928+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/1781921621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123cdf10 0x5559123d2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.928+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/1781921621 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5559123e2a10 0x5559123e3e80 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.928+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/1781921621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123c9810 0x5559123d5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.928+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/1781921621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123d2760 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.928+0000 7f0ca0ff9700 0 client.15039 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:56:17.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.928+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/1781921621 >> 172.21.15.70:0/1781921621 conn(0x5559122f7030 msgr2=0x5559123cf300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:17.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.928+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/1781921621 shutdown_connections 2024-06-04T04:56:17.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.928+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/1781921621 wait complete. 2024-06-04T04:56:17.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.928+0000 7f0cc9c13b80 20 client.15039 trim_cache size 0 max 16384 2024-06-04T04:56:17.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.949+0000 7f0cc9c13b80 1 Processor -- start 2024-06-04T04:56:17.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.949+0000 7f0cc9c13b80 1 -- start start 2024-06-04T04:56:17.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.949+0000 7f0cc9c13b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123c9810 0x5559123d18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.949+0000 7f0cc9c13b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.949+0000 7f0cc9c13b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123cdf10 0x5559123ed6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.949+0000 7f0cc9c13b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5559123e21c0 con 0x5559123ca1c0 2024-06-04T04:56:17.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.949+0000 7f0cc9c13b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5559123e2890 con 0x5559123c9810 2024-06-04T04:56:17.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.949+0000 7f0cc9c13b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5559123e2f60 con 0x5559123cdf10 2024-06-04T04:56:17.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.950+0000 7f0cbc919700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:17.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.950+0000 7f0cbc919700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59900/0 (socket says 172.21.15.70:59900) 2024-06-04T04:56:17.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.950+0000 7f0cbc919700 1 -- 172.21.15.70:0/1635422598 learned_addr learned my addr 172.21.15.70:0/1635422598 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:17.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.950+0000 7f0cbd11a700 1 --2- 172.21.15.70:0/1635422598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123c9810 0x5559123d18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:17.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.950+0000 7f0cbd91b700 1 --2- 172.21.15.70:0/1635422598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123cdf10 0x5559123ed6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:17.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.950+0000 7f0cbc919700 1 -- 172.21.15.70:0/1635422598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123cdf10 msgr2=0x5559123ed6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:17.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.950+0000 7f0cbc919700 1 --2- 172.21.15.70:0/1635422598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123cdf10 0x5559123ed6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.950+0000 7f0cbc919700 1 -- 172.21.15.70:0/1635422598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123c9810 msgr2=0x5559123d18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:17.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.950+0000 7f0cbc919700 1 --2- 172.21.15.70:0/1635422598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123c9810 0x5559123d18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.950+0000 7f0cbc919700 1 -- 172.21.15.70:0/1635422598 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559123c3b50 con 0x5559123ca1c0 2024-06-04T04:56:17.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.950+0000 7f0cbd11a700 1 --2- 172.21.15.70:0/1635422598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123c9810 0x5559123d18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:17.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.950+0000 7f0cbc919700 1 --2- 172.21.15.70:0/1635422598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123cd480 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40a6a20 tx=0x7f0cb40a6860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b37de0e54c7e8da server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:17.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.950+0000 7f0ca2ffd700 1 -- 172.21.15.70:0/1635422598 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb40c3070 con 0x5559123ca1c0 2024-06-04T04:56:17.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.950+0000 7f0ca2ffd700 1 -- 172.21.15.70:0/1635422598 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0cb40b3770 con 0x5559123ca1c0 2024-06-04T04:56:17.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.950+0000 7f0ca2ffd700 1 -- 172.21.15.70:0/1635422598 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb40702b0 con 0x5559123ca1c0 2024-06-04T04:56:17.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.951+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/1635422598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 msgr2=0x5559123cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:17.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.951+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/1635422598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123cd480 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40a6a20 tx=0x7f0cb40a6860 comp rx=0 tx=0).stop 2024-06-04T04:56:17.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.951+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/1635422598 shutdown_connections 2024-06-04T04:56:17.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.951+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/1635422598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123cdf10 0x5559123ed6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.951+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/1635422598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123c9810 0x5559123d18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.951+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/1635422598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123ca1c0 0x5559123cd480 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.951+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/1635422598 >> 172.21.15.70:0/1635422598 conn(0x5559122f7030 msgr2=0x5559123ccd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:17.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.951+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/1635422598 shutdown_connections 2024-06-04T04:56:17.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.951+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/1635422598 wait complete. 2024-06-04T04:56:17.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.951+0000 7f0cc9c13b80 1 -- start start 2024-06-04T04:56:17.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.951+0000 7f0cc9c13b80 1 Processor -- start 2024-06-04T04:56:17.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.952+0000 7f0cc9c13b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123c9810 0x5559123ed960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.952+0000 7f0cc9c13b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123ca1c0 0x5559123e9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.952+0000 7f0cc9c13b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123cdf10 0x5559123ea230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.952+0000 7f0cc9c13b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5559123e3d00 con 0x5559123c9810 2024-06-04T04:56:17.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.952+0000 7f0cc9c13b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5559123e3630 con 0x5559123ca1c0 2024-06-04T04:56:17.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.952+0000 7f0cc9c13b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5559123e43d0 con 0x5559123cdf10 2024-06-04T04:56:17.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.952+0000 7f0cc9c13b80 10 client.0 _ll_register_callbacks cb 0x5559122b38d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:17.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.952+0000 7f0cbd11a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123c9810 0x5559123ed960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:17.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.952+0000 7f0cbd11a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123c9810 0x5559123ed960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59914/0 (socket says 172.21.15.70:59914) 2024-06-04T04:56:17.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.952+0000 7f0cbd11a700 1 -- 172.21.15.70:0/566847884 learned_addr learned my addr 172.21.15.70:0/566847884 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:17.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.952+0000 7f0cbc919700 1 --2- 172.21.15.70:0/566847884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123ca1c0 0x5559123e9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:17.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.952+0000 7f0cbd91b700 1 --2- 172.21.15.70:0/566847884 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123cdf10 0x5559123ea230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:17.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.952+0000 7f0cbd11a700 1 -- 172.21.15.70:0/566847884 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123cdf10 msgr2=0x5559123ea230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:17.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.952+0000 7f0cbd11a700 1 --2- 172.21.15.70:0/566847884 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123cdf10 0x5559123ea230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.952+0000 7f0cbd11a700 1 -- 172.21.15.70:0/566847884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123ca1c0 msgr2=0x5559123e9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:17.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.952+0000 7f0cbd11a700 1 --2- 172.21.15.70:0/566847884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123ca1c0 0x5559123e9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.952+0000 7f0cbd11a700 1 -- 172.21.15.70:0/566847884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559123eaa70 con 0x5559123c9810 2024-06-04T04:56:17.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.953+0000 7f0cbd11a700 1 --2- 172.21.15.70:0/566847884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123c9810 0x5559123ed960 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x5559123d5fa0 tx=0x7f0cac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ded1e7a5d24d571 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:17.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.953+0000 7f0ca0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:17.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.953+0000 7f0ca0ff9700 1 -- 172.21.15.70:0/566847884 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac030060 con 0x5559123c9810 2024-06-04T04:56:17.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.953+0000 7f0ca0ff9700 1 -- 172.21.15.70:0/566847884 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f0cac013770 con 0x5559123c9810 2024-06-04T04:56:17.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.953+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/566847884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cb40b3770 con 0x5559123c9810 2024-06-04T04:56:17.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.953+0000 7f0ca0ff9700 1 -- 172.21.15.70:0/566847884 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0cac013970 con 0x5559123c9810 2024-06-04T04:56:17.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.953+0000 7f0cc9c13b80 10 client.15045 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:17.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.953+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/566847884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555912457a30 con 0x5559123c9810 2024-06-04T04:56:17.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.953+0000 7f0ca0ff9700 1 -- 172.21.15.70:0/566847884 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0cb40b3770 con 0x5559123c9810 2024-06-04T04:56:17.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.953+0000 7f0ca0ff9700 1 -- 172.21.15.70:0/566847884 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f0cac02bbe0 con 0x5559123c9810 2024-06-04T04:56:17.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.953+0000 7f0cc9c13b80 10 client.15045 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:17.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.953+0000 7f0cc9c13b80 10 client.15045 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-04T04:56:17.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.953+0000 7f0cc9c13b80 20 client.15045 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:17.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.953+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/566847884 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55591245c670 0x55591245b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:17.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.953+0000 7f0cc9c13b80 4 client.15045 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-04T04:56:17.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.953+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/566847884 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "status"}) -- 0x55591232ab90 con 0x55591245c670 2024-06-04T04:56:17.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.954+0000 7f0cbc919700 1 --2- 172.21.15.70:0/566847884 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55591245c670 0x55591245b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:17.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.954+0000 7f0cbc919700 1 --2- 172.21.15.70:0/566847884 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55591245c670 0x55591245b380 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b18d1411040f433a server_cookie=79abb6bf8a2896e7 in_seq=0 out_seq=0 2024-06-04T04:56:17.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.954+0000 7f0ca0ff9700 10 client.15045 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:56:17.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.954+0000 7f0ca0ff9700 1 -- 172.21.15.70:0/566847884 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55591232ab90 con 0x55591245c670 2024-06-04T04:56:17.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.954+0000 7f0ca0ff9700 10 client.15045 handle_command_reply: tid=0 2024-06-04T04:56:17.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.954+0000 7f0cc9c13b80 1 client.15045 shutdown 2024-06-04T04:56:17.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.954+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/566847884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123c9810 msgr2=0x5559123ed960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:17.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.954+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/566847884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123c9810 0x5559123ed960 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x5559123d5fa0 tx=0x7f0cac004750 comp rx=0 tx=0).stop 2024-06-04T04:56:17.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.954+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/566847884 shutdown_connections 2024-06-04T04:56:17.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.954+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/566847884 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559123cdf10 0x5559123ea230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.954+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/566847884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559123c9810 0x5559123ed960 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.954+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/566847884 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55591245c670 0x55591245b380 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.954+0000 7f0cc9c13b80 1 --2- 172.21.15.70:0/566847884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559123ca1c0 0x5559123e9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.954+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/566847884 >> 172.21.15.70:0/566847884 conn(0x5559122f7030 msgr2=0x5559123ccd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:17.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.954+0000 7f0ca0ff9700 0 client.15045 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:56:17.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.955+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/566847884 shutdown_connections 2024-06-04T04:56:17.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.955+0000 7f0cc9c13b80 1 -- 172.21.15.70:0/566847884 wait complete. 2024-06-04T04:56:17.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.955+0000 7f0cc9c13b80 20 client.15045 trim_cache size 0 max 16384 2024-06-04T04:56:17.962 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:56:17.962 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:56:17.962 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-06-04T04:56:17.962 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24448, 2024-06-04T04:56:17.962 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-06-04T04:56:17.962 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-06-04T04:56:17.962 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-06-04T04:56:17.963 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 159.58726117099999, 2024-06-04T04:56:17.963 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-06-04T04:56:17.963 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 81, 2024-06-04T04:56:17.963 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 79, 2024-06-04T04:56:17.963 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 185.164500613 2024-06-04T04:56:17.963 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:56:17.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.955+0000 7f0cbf37e700 1 -- 172.21.15.70:0/1971277216 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0ca40b2880 msgr2=0x7f0ca40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:17.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.955+0000 7f0cbf37e700 1 --2- 172.21.15.70:0/1971277216 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0ca40b2880 0x7f0ca40b4d40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0cb819ab30 tx=0x7f0cac008040 comp rx=0 tx=0).stop 2024-06-04T04:56:17.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.955+0000 7f0cbf37e700 1 -- 172.21.15.70:0/1971277216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0cb806c650 msgr2=0x7f0cb819fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:17.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.955+0000 7f0cbf37e700 1 --2- 172.21.15.70:0/1971277216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0cb806c650 0x7f0cb819fd70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4000c00 tx=0x7f0cb40047b0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.955+0000 7f0cbf37e700 1 -- 172.21.15.70:0/1971277216 shutdown_connections 2024-06-04T04:56:17.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.955+0000 7f0cbf37e700 1 --2- 172.21.15.70:0/1971277216 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0ca40b2880 0x7f0ca40b4d40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.955+0000 7f0cbf37e700 1 --2- 172.21.15.70:0/1971277216 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0cb81095b0 0x7f0cb81999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.955+0000 7f0cbf37e700 1 --2- 172.21.15.70:0/1971277216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb806bc90 0x7f0cb819f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.955+0000 7f0cbf37e700 1 --2- 172.21.15.70:0/1971277216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0cb806c650 0x7f0cb819fd70 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:17.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.956+0000 7f0cbf37e700 1 -- 172.21.15.70:0/1971277216 >> 172.21.15.70:0/1971277216 conn(0x7f0cb8079410 msgr2=0x7f0cb81013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:17.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.956+0000 7f0cbf37e700 1 -- 172.21.15.70:0/1971277216 shutdown_connections 2024-06-04T04:56:17.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:17.956+0000 7f0cbf37e700 1 -- 172.21.15.70:0/1971277216 wait complete. 2024-06-04T04:56:18.631 INFO:tasks.cephadm:Setup complete, yielding 2024-06-04T04:56:18.632 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-06-04T04:56:18.641 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi070.front.sepia.ceph.com 2024-06-04T04:56:18.641 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch status' 2024-06-04T04:56:18.881 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:19.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:19 smithi070 ceph-mon[32631]: pgmap v371: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:56:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:19 smithi112 ceph-mon[36453]: pgmap v371: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:56:19.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:19 smithi098 ceph-mon[35569]: pgmap v371: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:56:20.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.611+0000 7f16d5e7f700 1 -- 172.21.15.70:0/3689275676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16d01040e0 msgr2=0x7f16d01064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:20.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.611+0000 7f16d5e7f700 1 --2- 172.21.15.70:0/3689275676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16d01040e0 0x7f16d01064d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f16c0009990 tx=0x7f16c0009ca0 comp rx=0 tx=0).stop 2024-06-04T04:56:20.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.611+0000 7f16d5e7f700 1 -- 172.21.15.70:0/3689275676 shutdown_connections 2024-06-04T04:56:20.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.611+0000 7f16d5e7f700 1 --2- 172.21.15.70:0/3689275676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16d0109340 0x7f16d010b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:20.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.611+0000 7f16d5e7f700 1 --2- 172.21.15.70:0/3689275676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16d01040e0 0x7f16d01064d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:20.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.611+0000 7f16d5e7f700 1 --2- 172.21.15.70:0/3689275676 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16d0106a10 0x7f16d0108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:20.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.611+0000 7f16d5e7f700 1 -- 172.21.15.70:0/3689275676 >> 172.21.15.70:0/3689275676 conn(0x7f16d00fda70 msgr2=0x7f16d00ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:20.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.611+0000 7f16d5e7f700 1 -- 172.21.15.70:0/3689275676 shutdown_connections 2024-06-04T04:56:20.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.611+0000 7f16d5e7f700 1 -- 172.21.15.70:0/3689275676 wait complete. 2024-06-04T04:56:20.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.612+0000 7f16d5e7f700 1 Processor -- start 2024-06-04T04:56:20.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.612+0000 7f16d5e7f700 1 -- start start 2024-06-04T04:56:20.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.612+0000 7f16d5e7f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16d01040e0 0x7f16d0199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:20.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.612+0000 7f16d5e7f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16d0106a10 0x7f16d0199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:20.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.612+0000 7f16d5e7f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16d0109340 0x7f16d01a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:20.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.612+0000 7f16d5e7f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f16d010eec0 con 0x7f16d0109340 2024-06-04T04:56:20.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.612+0000 7f16d5e7f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f16d010ed40 con 0x7f16d0106a10 2024-06-04T04:56:20.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.613+0000 7f16d5e7f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f16d010f040 con 0x7f16d01040e0 2024-06-04T04:56:20.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.613+0000 7f16cffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16d0109340 0x7f16d01a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:20.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.613+0000 7f16cffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16d0109340 0x7f16d01a0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59930/0 (socket says 172.21.15.70:59930) 2024-06-04T04:56:20.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.613+0000 7f16cffff700 1 -- 172.21.15.70:0/3213063275 learned_addr learned my addr 172.21.15.70:0/3213063275 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:20.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.613+0000 7f16cf7fe700 1 --2- 172.21.15.70:0/3213063275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16d01040e0 0x7f16d0199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:20.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.613+0000 7f16ceffd700 1 --2- 172.21.15.70:0/3213063275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16d0106a10 0x7f16d0199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:20.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.613+0000 7f16cffff700 1 -- 172.21.15.70:0/3213063275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16d01040e0 msgr2=0x7f16d0199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:20.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.613+0000 7f16cffff700 1 --2- 172.21.15.70:0/3213063275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16d01040e0 0x7f16d0199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:20.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.613+0000 7f16cffff700 1 -- 172.21.15.70:0/3213063275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16d0106a10 msgr2=0x7f16d0199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:20.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.613+0000 7f16cffff700 1 --2- 172.21.15.70:0/3213063275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16d0106a10 0x7f16d0199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:20.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.613+0000 7f16cffff700 1 -- 172.21.15.70:0/3213063275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d0101ed0 con 0x7f16d0109340 2024-06-04T04:56:20.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.613+0000 7f16cffff700 1 --2- 172.21.15.70:0/3213063275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16d0109340 0x7f16d01a0c40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f16c8007ee0 tx=0x7f16c800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:20.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.613+0000 7f16ccff9700 1 -- 172.21.15.70:0/3213063275 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8004db0 con 0x7f16d0109340 2024-06-04T04:56:20.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.613+0000 7f16ccff9700 1 -- 172.21.15.70:0/3213063275 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f16c8005020 con 0x7f16d0109340 2024-06-04T04:56:20.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.613+0000 7f16d5e7f700 1 -- 172.21.15.70:0/3213063275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d01021c0 con 0x7f16d0109340 2024-06-04T04:56:20.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.613+0000 7f16ccff9700 1 -- 172.21.15.70:0/3213063275 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8015660 con 0x7f16d0109340 2024-06-04T04:56:20.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.613+0000 7f16d5e7f700 1 -- 172.21.15.70:0/3213063275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d0102700 con 0x7f16d0109340 2024-06-04T04:56:20.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.614+0000 7f16d5e7f700 1 -- 172.21.15.70:0/3213063275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d006a230 con 0x7f16d0109340 2024-06-04T04:56:20.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.616+0000 7f16ccff9700 1 -- 172.21.15.70:0/3213063275 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f16c80051d0 con 0x7f16d0109340 2024-06-04T04:56:20.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.616+0000 7f16ccff9700 1 --2- 172.21.15.70:0/3213063275 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f16b80b2900 0x7f16b80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:20.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.617+0000 7f16cf7fe700 1 --2- 172.21.15.70:0/3213063275 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f16b80b2900 0x7f16b80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:20.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.617+0000 7f16ccff9700 1 -- 172.21.15.70:0/3213063275 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f16c80e2360 con 0x7f16d0109340 2024-06-04T04:56:20.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.617+0000 7f16cf7fe700 1 --2- 172.21.15.70:0/3213063275 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f16b80b2900 0x7f16b80b4dc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f16c0006010 tx=0x7f16c00037c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:20.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.620+0000 7f16ccff9700 1 -- 172.21.15.70:0/3213063275 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f16c8010070 con 0x7f16d0109340 2024-06-04T04:56:20.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.845+0000 7f16d5e7f700 1 -- 172.21.15.70:0/3213063275 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f16d00ff140 con 0x7f16b80b2900 2024-06-04T04:56:20.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.846+0000 7f16ccff9700 1 -- 172.21.15.70:0/3213063275 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f16d00ff140 con 0x7f16b80b2900 2024-06-04T04:56:20.848 INFO:teuthology.orchestra.run.smithi070.stdout:Backend: cephadm 2024-06-04T04:56:20.848 INFO:teuthology.orchestra.run.smithi070.stdout:Available: Yes 2024-06-04T04:56:20.848 INFO:teuthology.orchestra.run.smithi070.stdout:Paused: No 2024-06-04T04:56:20.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.848+0000 7f16d5e7f700 1 -- 172.21.15.70:0/3213063275 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f16b80b2900 msgr2=0x7f16b80b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:20.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.848+0000 7f16d5e7f700 1 --2- 172.21.15.70:0/3213063275 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f16b80b2900 0x7f16b80b4dc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f16c0006010 tx=0x7f16c00037c0 comp rx=0 tx=0).stop 2024-06-04T04:56:20.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.848+0000 7f16d5e7f700 1 -- 172.21.15.70:0/3213063275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16d0109340 msgr2=0x7f16d01a0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:20.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.848+0000 7f16d5e7f700 1 --2- 172.21.15.70:0/3213063275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16d0109340 0x7f16d01a0c40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f16c8007ee0 tx=0x7f16c800d3b0 comp rx=0 tx=0).stop 2024-06-04T04:56:20.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.848+0000 7f16d5e7f700 1 -- 172.21.15.70:0/3213063275 shutdown_connections 2024-06-04T04:56:20.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.848+0000 7f16d5e7f700 1 --2- 172.21.15.70:0/3213063275 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f16b80b2900 0x7f16b80b4dc0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:20.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.848+0000 7f16d5e7f700 1 --2- 172.21.15.70:0/3213063275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16d0109340 0x7f16d01a0c40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:20.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.848+0000 7f16d5e7f700 1 --2- 172.21.15.70:0/3213063275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16d01040e0 0x7f16d0199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:20.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.848+0000 7f16d5e7f700 1 --2- 172.21.15.70:0/3213063275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16d0106a10 0x7f16d0199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:20.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.848+0000 7f16d5e7f700 1 -- 172.21.15.70:0/3213063275 >> 172.21.15.70:0/3213063275 conn(0x7f16d00fda70 msgr2=0x7f16d0105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:20.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.849+0000 7f16d5e7f700 1 -- 172.21.15.70:0/3213063275 shutdown_connections 2024-06-04T04:56:20.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:20.849+0000 7f16d5e7f700 1 -- 172.21.15.70:0/3213063275 wait complete. 2024-06-04T04:56:21.515 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:21 smithi070 ceph-mon[32631]: pgmap v372: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:56:21.543 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ps' 2024-06-04T04:56:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:21 smithi112 ceph-mon[36453]: pgmap v372: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:56:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:21 smithi098 ceph-mon[35569]: pgmap v372: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:56:21.782 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:22.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:22 smithi070 ceph-mon[32631]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:56:22.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:22 smithi112 ceph-mon[36453]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:56:22.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:22 smithi098 ceph-mon[35569]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:56:23.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.439+0000 7f1cc9a95700 1 -- 172.21.15.70:0/3082186442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cc41036b0 msgr2=0x7f1cc4103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:23.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.439+0000 7f1cc9a95700 1 --2- 172.21.15.70:0/3082186442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cc41036b0 0x7f1cc4103b10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80099c0 tx=0x7f1cb8009cd0 comp rx=0 tx=0).stop 2024-06-04T04:56:23.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.439+0000 7f1cc9a95700 1 -- 172.21.15.70:0/3082186442 shutdown_connections 2024-06-04T04:56:23.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.439+0000 7f1cc9a95700 1 --2- 172.21.15.70:0/3082186442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cc4104050 0x7f1cc410a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:23.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.439+0000 7f1cc9a95700 1 --2- 172.21.15.70:0/3082186442 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1cc41024b0 0x7f1cc41028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:23.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.439+0000 7f1cc9a95700 1 --2- 172.21.15.70:0/3082186442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cc41036b0 0x7f1cc4103b10 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.439+0000 7f1cc9a95700 1 -- 172.21.15.70:0/3082186442 >> 172.21.15.70:0/3082186442 conn(0x7f1cc40fda70 msgr2=0x7f1cc40ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.439+0000 7f1cc9a95700 1 -- 172.21.15.70:0/3082186442 shutdown_connections 2024-06-04T04:56:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.439+0000 7f1cc9a95700 1 -- 172.21.15.70:0/3082186442 wait complete. 2024-06-04T04:56:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.439+0000 7f1cc9a95700 1 Processor -- start 2024-06-04T04:56:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc9a95700 1 -- start start 2024-06-04T04:56:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc9a95700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cc41024b0 0x7f1cc41063f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc9a95700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1cc41036b0 0x7f1cc4104a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc9a95700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cc4104050 0x7f1cc4104fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc9a95700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc410de80 con 0x7f1cc41024b0 2024-06-04T04:56:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc9a95700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc410dd00 con 0x7f1cc4104050 2024-06-04T04:56:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc9a95700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc410e000 con 0x7f1cc41036b0 2024-06-04T04:56:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cc41024b0 0x7f1cc41063f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cc41024b0 0x7f1cc41063f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59948/0 (socket says 172.21.15.70:59948) 2024-06-04T04:56:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc37fe700 1 -- 172.21.15.70:0/83625505 learned_addr learned my addr 172.21.15.70:0/83625505 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc3fff700 1 --2- 172.21.15.70:0/83625505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cc4104050 0x7f1cc4104fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc2ffd700 1 --2- 172.21.15.70:0/83625505 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1cc41036b0 0x7f1cc4104a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc37fe700 1 -- 172.21.15.70:0/83625505 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1cc41036b0 msgr2=0x7f1cc4104a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc37fe700 1 --2- 172.21.15.70:0/83625505 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1cc41036b0 0x7f1cc4104a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc37fe700 1 -- 172.21.15.70:0/83625505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cc4104050 msgr2=0x7f1cc4104fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc37fe700 1 --2- 172.21.15.70:0/83625505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cc4104050 0x7f1cc4104fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc37fe700 1 -- 172.21.15.70:0/83625505 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc41058a0 con 0x7f1cc41024b0 2024-06-04T04:56:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc37fe700 1 --2- 172.21.15.70:0/83625505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cc41024b0 0x7f1cc41063f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1cb400b9c0 tx=0x7f1cb400bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.440+0000 7f1cc0ff9700 1 -- 172.21.15.70:0/83625505 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4019070 con 0x7f1cc41024b0 2024-06-04T04:56:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.441+0000 7f1cc0ff9700 1 -- 172.21.15.70:0/83625505 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1cb400cde0 con 0x7f1cc41024b0 2024-06-04T04:56:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.441+0000 7f1cc0ff9700 1 -- 172.21.15.70:0/83625505 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4012340 con 0x7f1cc41024b0 2024-06-04T04:56:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.441+0000 7f1cc9a95700 1 -- 172.21.15.70:0/83625505 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cc41a38e0 con 0x7f1cc41024b0 2024-06-04T04:56:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.441+0000 7f1cc9a95700 1 -- 172.21.15.70:0/83625505 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cc41a3da0 con 0x7f1cc41024b0 2024-06-04T04:56:23.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.442+0000 7f1cc9a95700 1 -- 172.21.15.70:0/83625505 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cc406a230 con 0x7f1cc41024b0 2024-06-04T04:56:23.446 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:23 smithi070 ceph-mon[32631]: pgmap v373: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:56:23.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.443+0000 7f1cc0ff9700 1 -- 172.21.15.70:0/83625505 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f1cb401c070 con 0x7f1cc41024b0 2024-06-04T04:56:23.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.444+0000 7f1cc0ff9700 1 --2- 172.21.15.70:0/83625505 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1cac0b2900 0x7f1cac0b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:23.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.444+0000 7f1cc0ff9700 1 -- 172.21.15.70:0/83625505 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f1cb40e1f00 con 0x7f1cc41024b0 2024-06-04T04:56:23.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.444+0000 7f1cc2ffd700 1 --2- 172.21.15.70:0/83625505 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1cac0b2900 0x7f1cac0b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:23.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.444+0000 7f1cc2ffd700 1 --2- 172.21.15.70:0/83625505 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1cac0b2900 0x7f1cac0b4dc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8000c00 tx=0x7f1cb801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:23.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.448+0000 7f1cc0ff9700 1 -- 172.21.15.70:0/83625505 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1cb4023070 con 0x7f1cc41024b0 2024-06-04T04:56:23.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.664+0000 7f1cc9a95700 1 -- 172.21.15.70:0/83625505 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f1cc406c3f0 con 0x7f1cac0b2900 2024-06-04T04:56:23.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.677+0000 7f1cc0ff9700 1 -- 172.21.15.70:0/83625505 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f1cc406c3f0 con 0x7f1cac0b2900 2024-06-04T04:56:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-04T04:56:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:mds.a smithi070 running (3m) 38s ago 3m 20.9M - 19.0.0-3882-gb7328fed 57e49aa75a3f 2227476cb226 2024-06-04T04:56:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:mds.b smithi098 running (3m) 47s ago 2m 21.0M - 19.0.0-3882-gb7328fed 57e49aa75a3f 5d68ed213d1f 2024-06-04T04:56:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:mds.c smithi112 running (3m) 47s ago 3m 22.2M - 19.0.0-3882-gb7328fed 57e49aa75a3f abd1e979f2cc 2024-06-04T04:56:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:mds.d smithi070 running (3m) 38s ago 3m 14.6M - 19.0.0-3882-gb7328fed 57e49aa75a3f 26ade5db2dba 2024-06-04T04:56:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:mds.e smithi098 running (3m) 47s ago 3m 20.9M - 19.0.0-3882-gb7328fed 57e49aa75a3f 14db21803650 2024-06-04T04:56:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:mds.f smithi112 running (3m) 47s ago 3m 20.6M - 19.0.0-3882-gb7328fed 57e49aa75a3f 76244bb7a9e7 2024-06-04T04:56:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:mds.g smithi070 running (3m) 38s ago 3m 19.5M - 19.0.0-3882-gb7328fed 57e49aa75a3f ee01603f0be4 2024-06-04T04:56:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:mds.h smithi098 running (3m) 47s ago 3m 19.7M - 19.0.0-3882-gb7328fed 57e49aa75a3f 87dd7687abc5 2024-06-04T04:56:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:mds.i smithi112 running (3m) 47s ago 3m 27.6M - 19.0.0-3882-gb7328fed 57e49aa75a3f ac055dfba103 2024-06-04T04:56:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:mds.j smithi070 running (3m) 38s ago 3m 16.9M - 19.0.0-3882-gb7328fed 57e49aa75a3f 92fa0d8e8b6b 2024-06-04T04:56:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:mds.k smithi098 running (3m) 47s ago 3m 21.8M - 19.0.0-3882-gb7328fed 57e49aa75a3f 99a968947a48 2024-06-04T04:56:23.681 INFO:teuthology.orchestra.run.smithi070.stdout:mds.l smithi112 running (3m) 47s ago 3m 20.8M - 19.0.0-3882-gb7328fed 57e49aa75a3f 4730b993ef36 2024-06-04T04:56:23.681 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi070 *:9283,8765 running (10m) 38s ago 10m 531M - 19.0.0-3882-gb7328fed 57e49aa75a3f b162b5a38699 2024-06-04T04:56:23.681 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi098 *:8443,8765 running (8m) 47s ago 8m 470M - 19.0.0-3882-gb7328fed 57e49aa75a3f 6d890c310308 2024-06-04T04:56:23.681 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.z smithi112 *:8443,8765 running (8m) 47s ago 8m 467M - 19.0.0-3882-gb7328fed 57e49aa75a3f 0b79d87cdfca 2024-06-04T04:56:23.681 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (10m) 38s ago 11m 53.0M 2048M 19.0.0-3882-gb7328fed 57e49aa75a3f e29249260acc 2024-06-04T04:56:23.681 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi098 running (9m) 47s ago 9m 43.7M 2048M 19.0.0-3882-gb7328fed 57e49aa75a3f f688c5292625 2024-06-04T04:56:23.681 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi112 running (9m) 47s ago 9m 41.4M 2048M 19.0.0-3882-gb7328fed 57e49aa75a3f ddd0d2b2cd3f 2024-06-04T04:56:23.681 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (8m) 38s ago 8m 46.2M 4272M 19.0.0-3882-gb7328fed 57e49aa75a3f cb47a300e8af 2024-06-04T04:56:23.681 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi098 running (8m) 47s ago 8m 58.4M 4272M 19.0.0-3882-gb7328fed 57e49aa75a3f a5622784960a 2024-06-04T04:56:23.681 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi112 running (7m) 47s ago 7m 48.4M 4272M 19.0.0-3882-gb7328fed 57e49aa75a3f 8d000e948ead 2024-06-04T04:56:23.681 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (7m) 38s ago 7m 59.4M 4272M 19.0.0-3882-gb7328fed 57e49aa75a3f be0a6cb91d7d 2024-06-04T04:56:23.681 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi098 running (6m) 47s ago 6m 55.1M 4272M 19.0.0-3882-gb7328fed 57e49aa75a3f e421053b69e0 2024-06-04T04:56:23.681 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi112 running (6m) 47s ago 6m 43.4M 4272M 19.0.0-3882-gb7328fed 57e49aa75a3f 5bc6749a8232 2024-06-04T04:56:23.681 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi070 running (6m) 38s ago 6m 42.8M 4272M 19.0.0-3882-gb7328fed 57e49aa75a3f 0fca30c8f3f1 2024-06-04T04:56:23.681 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi098 running (5m) 47s ago 5m 45.0M 4272M 19.0.0-3882-gb7328fed 57e49aa75a3f 02ab71a6d6d6 2024-06-04T04:56:23.682 INFO:teuthology.orchestra.run.smithi070.stdout:osd.8 smithi112 running (5m) 47s ago 5m 47.0M 4272M 19.0.0-3882-gb7328fed 57e49aa75a3f 562b972b6f59 2024-06-04T04:56:23.682 INFO:teuthology.orchestra.run.smithi070.stdout:osd.9 smithi070 running (4m) 38s ago 4m 43.5M 4272M 19.0.0-3882-gb7328fed 57e49aa75a3f 8ac04e70817f 2024-06-04T04:56:23.682 INFO:teuthology.orchestra.run.smithi070.stdout:osd.10 smithi098 running (4m) 47s ago 4m 55.4M 4272M 19.0.0-3882-gb7328fed 57e49aa75a3f 1432d7446edb 2024-06-04T04:56:23.682 INFO:teuthology.orchestra.run.smithi070.stdout:osd.11 smithi112 running (3m) 47s ago 3m 45.4M 4272M 19.0.0-3882-gb7328fed 57e49aa75a3f ce1ba8899258 2024-06-04T04:56:23.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.680+0000 7f1cc9a95700 1 -- 172.21.15.70:0/83625505 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1cac0b2900 msgr2=0x7f1cac0b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:23.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.680+0000 7f1cc9a95700 1 --2- 172.21.15.70:0/83625505 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1cac0b2900 0x7f1cac0b4dc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8000c00 tx=0x7f1cb801a040 comp rx=0 tx=0).stop 2024-06-04T04:56:23.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.680+0000 7f1cc9a95700 1 -- 172.21.15.70:0/83625505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cc41024b0 msgr2=0x7f1cc41063f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:23.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.680+0000 7f1cc9a95700 1 --2- 172.21.15.70:0/83625505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cc41024b0 0x7f1cc41063f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1cb400b9c0 tx=0x7f1cb400bcd0 comp rx=0 tx=0).stop 2024-06-04T04:56:23.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.680+0000 7f1cc9a95700 1 -- 172.21.15.70:0/83625505 shutdown_connections 2024-06-04T04:56:23.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.680+0000 7f1cc9a95700 1 --2- 172.21.15.70:0/83625505 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1cac0b2900 0x7f1cac0b4dc0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:23.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.680+0000 7f1cc9a95700 1 --2- 172.21.15.70:0/83625505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cc4104050 0x7f1cc4104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:23.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.680+0000 7f1cc9a95700 1 --2- 172.21.15.70:0/83625505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cc41024b0 0x7f1cc41063f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:23.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.680+0000 7f1cc9a95700 1 --2- 172.21.15.70:0/83625505 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1cc41036b0 0x7f1cc4104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:23.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.680+0000 7f1cc9a95700 1 -- 172.21.15.70:0/83625505 >> 172.21.15.70:0/83625505 conn(0x7f1cc40fda70 msgr2=0x7f1cc406bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:23.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.680+0000 7f1cc9a95700 1 -- 172.21.15.70:0/83625505 shutdown_connections 2024-06-04T04:56:23.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:23.680+0000 7f1cc9a95700 1 -- 172.21.15.70:0/83625505 wait complete. 2024-06-04T04:56:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:23 smithi112 ceph-mon[36453]: pgmap v373: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:56:23.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:23 smithi098 ceph-mon[35569]: pgmap v373: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:56:24.412 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ls' 2024-06-04T04:56:24.651 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:25 smithi112 ceph-mon[36453]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:56:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:25 smithi112 ceph-mon[36453]: pgmap v374: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:56:25.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:25 smithi098 ceph-mon[35569]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:56:25.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:25 smithi098 ceph-mon[35569]: pgmap v374: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:56:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:25 smithi070 ceph-mon[32631]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:56:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:25 smithi070 ceph-mon[32631]: pgmap v374: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:56:26.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.406+0000 7fec33ae8700 1 -- 172.21.15.70:0/1624774355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec2c101860 msgr2=0x7fec2c101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:26.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.406+0000 7fec33ae8700 1 --2- 172.21.15.70:0/1624774355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec2c101860 0x7fec2c101c80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fec2400b0a0 tx=0x7fec2400b3b0 comp rx=0 tx=0).stop 2024-06-04T04:56:26.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.406+0000 7fec33ae8700 1 -- 172.21.15.70:0/1624774355 shutdown_connections 2024-06-04T04:56:26.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.406+0000 7fec33ae8700 1 --2- 172.21.15.70:0/1624774355 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec2c102f90 0x7fec2c107830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:26.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.406+0000 7fec33ae8700 1 --2- 172.21.15.70:0/1624774355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec2c101860 0x7fec2c101c80 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:26.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.406+0000 7fec33ae8700 1 --2- 172.21.15.70:0/1624774355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec2c1025d0 0x7fec2c102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:26.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.406+0000 7fec33ae8700 1 -- 172.21.15.70:0/1624774355 >> 172.21.15.70:0/1624774355 conn(0x7fec2c0fd1a0 msgr2=0x7fec2c0ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:26.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.406+0000 7fec33ae8700 1 -- 172.21.15.70:0/1624774355 shutdown_connections 2024-06-04T04:56:26.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.406+0000 7fec33ae8700 1 -- 172.21.15.70:0/1624774355 wait complete. 2024-06-04T04:56:26.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec33ae8700 1 Processor -- start 2024-06-04T04:56:26.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec33ae8700 1 -- start start 2024-06-04T04:56:26.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec33ae8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec2c101860 0x7fec2c19fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:26.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec33ae8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec2c1025d0 0x7fec2c1a00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:26.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec33ae8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec2c102f90 0x7fec2c1a7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:26.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec33ae8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fec2c115510 con 0x7fec2c101860 2024-06-04T04:56:26.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec33ae8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fec2c115390 con 0x7fec2c102f90 2024-06-04T04:56:26.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec33ae8700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fec2c115690 con 0x7fec2c1025d0 2024-06-04T04:56:26.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec31884700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec2c101860 0x7fec2c19fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:26.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec31884700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec2c101860 0x7fec2c19fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35812/0 (socket says 172.21.15.70:35812) 2024-06-04T04:56:26.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec31884700 1 -- 172.21.15.70:0/1058262023 learned_addr learned my addr 172.21.15.70:0/1058262023 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:26.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec32085700 1 --2- 172.21.15.70:0/1058262023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec2c102f90 0x7fec2c1a7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:26.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec31083700 1 --2- 172.21.15.70:0/1058262023 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec2c1025d0 0x7fec2c1a00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:26.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec31884700 1 -- 172.21.15.70:0/1058262023 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec2c1025d0 msgr2=0x7fec2c1a00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:26.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec31884700 1 --2- 172.21.15.70:0/1058262023 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec2c1025d0 0x7fec2c1a00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:26.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec31884700 1 -- 172.21.15.70:0/1058262023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec2c102f90 msgr2=0x7fec2c1a7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:26.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec31884700 1 --2- 172.21.15.70:0/1058262023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec2c102f90 0x7fec2c1a7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:26.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec31884700 1 -- 172.21.15.70:0/1058262023 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec2c1a77e0 con 0x7fec2c101860 2024-06-04T04:56:26.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.407+0000 7fec31884700 1 --2- 172.21.15.70:0/1058262023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec2c101860 0x7fec2c19fb80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fec2400b070 tx=0x7fec24009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:26.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.408+0000 7fec2affd700 1 -- 172.21.15.70:0/1058262023 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2401f030 con 0x7fec2c101860 2024-06-04T04:56:26.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.408+0000 7fec33ae8700 1 -- 172.21.15.70:0/1058262023 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec2c1a7a70 con 0x7fec2c101860 2024-06-04T04:56:26.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.408+0000 7fec2affd700 1 -- 172.21.15.70:0/1058262023 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fec24003a00 con 0x7fec2c101860 2024-06-04T04:56:26.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.408+0000 7fec2affd700 1 -- 172.21.15.70:0/1058262023 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec24004680 con 0x7fec2c101860 2024-06-04T04:56:26.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.409+0000 7fec33ae8700 1 -- 172.21.15.70:0/1058262023 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec2c1a7f70 con 0x7fec2c101860 2024-06-04T04:56:26.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.410+0000 7fec33ae8700 1 -- 172.21.15.70:0/1058262023 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec2c06a230 con 0x7fec2c101860 2024-06-04T04:56:26.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.411+0000 7fec2affd700 1 -- 172.21.15.70:0/1058262023 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fec24003bb0 con 0x7fec2c101860 2024-06-04T04:56:26.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.411+0000 7fec2affd700 1 --2- 172.21.15.70:0/1058262023 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fec140b2900 0x7fec140b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:26.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.411+0000 7fec31083700 1 --2- 172.21.15.70:0/1058262023 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fec140b2900 0x7fec140b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:26.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.412+0000 7fec2affd700 1 -- 172.21.15.70:0/1058262023 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fec2400f020 con 0x7fec2c101860 2024-06-04T04:56:26.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.412+0000 7fec31083700 1 --2- 172.21.15.70:0/1058262023 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fec140b2900 0x7fec140b4dc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fec2000ac50 tx=0x7fec2000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:26.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.416+0000 7fec2affd700 1 -- 172.21.15.70:0/1058262023 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fec24029020 con 0x7fec2c101860 2024-06-04T04:56:26.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.633+0000 7fec33ae8700 1 -- 172.21.15.70:0/1058262023 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fec2c0ff530 con 0x7fec140b2900 2024-06-04T04:56:26.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.639+0000 7fec2affd700 1 -- 172.21.15.70:0/1058262023 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fec2c0ff530 con 0x7fec140b2900 2024-06-04T04:56:26.642 INFO:teuthology.orchestra.run.smithi070.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-06-04T04:56:26.642 INFO:teuthology.orchestra.run.smithi070.stdout:mds.all 12/12 50s ago 3m smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi112=c;smithi112=f;smithi112=i;smithi112=l;count:12 2024-06-04T04:56:26.642 INFO:teuthology.orchestra.run.smithi070.stdout:mgr 3/3 50s ago 9m smithi070=x;smithi098=y;smithi112=z;count:3 2024-06-04T04:56:26.642 INFO:teuthology.orchestra.run.smithi070.stdout:mon 3/3 50s ago 9m smithi070:172.21.15.70=a;smithi098:172.21.15.98=b;smithi112:172.21.15.112=c;count:3 2024-06-04T04:56:26.642 INFO:teuthology.orchestra.run.smithi070.stdout:osd 12 50s ago - 2024-06-04T04:56:26.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.641+0000 7fec33ae8700 1 -- 172.21.15.70:0/1058262023 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fec140b2900 msgr2=0x7fec140b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:26.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.641+0000 7fec33ae8700 1 --2- 172.21.15.70:0/1058262023 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fec140b2900 0x7fec140b4dc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fec2000ac50 tx=0x7fec2000a380 comp rx=0 tx=0).stop 2024-06-04T04:56:26.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.641+0000 7fec33ae8700 1 -- 172.21.15.70:0/1058262023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec2c101860 msgr2=0x7fec2c19fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:26.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.641+0000 7fec33ae8700 1 --2- 172.21.15.70:0/1058262023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec2c101860 0x7fec2c19fb80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fec2400b070 tx=0x7fec24009c00 comp rx=0 tx=0).stop 2024-06-04T04:56:26.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.641+0000 7fec33ae8700 1 -- 172.21.15.70:0/1058262023 shutdown_connections 2024-06-04T04:56:26.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.641+0000 7fec33ae8700 1 --2- 172.21.15.70:0/1058262023 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fec140b2900 0x7fec140b4dc0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:26.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.641+0000 7fec33ae8700 1 --2- 172.21.15.70:0/1058262023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec2c102f90 0x7fec2c1a7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:26.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.641+0000 7fec33ae8700 1 --2- 172.21.15.70:0/1058262023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec2c101860 0x7fec2c19fb80 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:26.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.641+0000 7fec33ae8700 1 --2- 172.21.15.70:0/1058262023 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec2c1025d0 0x7fec2c1a00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:26.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.641+0000 7fec33ae8700 1 -- 172.21.15.70:0/1058262023 >> 172.21.15.70:0/1058262023 conn(0x7fec2c0fd1a0 msgr2=0x7fec2c0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:26.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.641+0000 7fec33ae8700 1 -- 172.21.15.70:0/1058262023 shutdown_connections 2024-06-04T04:56:26.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:26.641+0000 7fec33ae8700 1 -- 172.21.15.70:0/1058262023 wait complete. 2024-06-04T04:56:27.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch host ls' 2024-06-04T04:56:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:27 smithi112 ceph-mon[36453]: pgmap v375: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:27 smithi112 ceph-mon[36453]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:56:27.696 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:27.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:27 smithi098 ceph-mon[35569]: pgmap v375: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:27.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:27 smithi098 ceph-mon[35569]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:56:27.818 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:27 smithi070 ceph-mon[32631]: pgmap v375: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:27.818 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:27 smithi070 ceph-mon[32631]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:56:29.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.479+0000 7fd34604e700 1 -- 172.21.15.70:0/2561600557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd340104060 msgr2=0x7fd34010a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:29.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.479+0000 7fd34604e700 1 --2- 172.21.15.70:0/2561600557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd340104060 0x7fd34010a910 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd33800d3d0 tx=0x7fd33800d6e0 comp rx=0 tx=0).stop 2024-06-04T04:56:29.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.479+0000 7fd34604e700 1 -- 172.21.15.70:0/2561600557 shutdown_connections 2024-06-04T04:56:29.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.479+0000 7fd34604e700 1 --2- 172.21.15.70:0/2561600557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd340104060 0x7fd34010a910 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:29.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.479+0000 7fd34604e700 1 --2- 172.21.15.70:0/2561600557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3401024c0 0x7fd3401028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:29.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.479+0000 7fd34604e700 1 --2- 172.21.15.70:0/2561600557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3401036c0 0x7fd340103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:29.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.479+0000 7fd34604e700 1 -- 172.21.15.70:0/2561600557 >> 172.21.15.70:0/2561600557 conn(0x7fd3400fdaa0 msgr2=0x7fd3400ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:29.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.479+0000 7fd34604e700 1 -- 172.21.15.70:0/2561600557 shutdown_connections 2024-06-04T04:56:29.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.479+0000 7fd34604e700 1 -- 172.21.15.70:0/2561600557 wait complete. 2024-06-04T04:56:29.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.480+0000 7fd34604e700 1 Processor -- start 2024-06-04T04:56:29.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.480+0000 7fd34604e700 1 -- start start 2024-06-04T04:56:29.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.481+0000 7fd34604e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3401024c0 0x7fd3401973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:29.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.481+0000 7fd34604e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3401036c0 0x7fd3401978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:29.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.481+0000 7fd34604e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd340104060 0x7fd34019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:29.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.481+0000 7fd34604e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd34010cd90 con 0x7fd3401024c0 2024-06-04T04:56:29.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.481+0000 7fd34604e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd34010cc10 con 0x7fd340104060 2024-06-04T04:56:29.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.481+0000 7fd34604e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd34010cf10 con 0x7fd3401036c0 2024-06-04T04:56:29.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.481+0000 7fd33f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3401024c0 0x7fd3401973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:29.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.481+0000 7fd33f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3401024c0 0x7fd3401973a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35832/0 (socket says 172.21.15.70:35832) 2024-06-04T04:56:29.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.481+0000 7fd33f7fe700 1 -- 172.21.15.70:0/2675602080 learned_addr learned my addr 172.21.15.70:0/2675602080 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:29.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.481+0000 7fd33effd700 1 --2- 172.21.15.70:0/2675602080 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3401036c0 0x7fd3401978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:29.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.481+0000 7fd33ffff700 1 --2- 172.21.15.70:0/2675602080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd340104060 0x7fd34019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:29.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.481+0000 7fd33f7fe700 1 -- 172.21.15.70:0/2675602080 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3401036c0 msgr2=0x7fd3401978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:29.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.481+0000 7fd33f7fe700 1 --2- 172.21.15.70:0/2675602080 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3401036c0 0x7fd3401978e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:29.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.481+0000 7fd33f7fe700 1 -- 172.21.15.70:0/2675602080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd340104060 msgr2=0x7fd34019e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:29.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.481+0000 7fd33f7fe700 1 --2- 172.21.15.70:0/2675602080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd340104060 0x7fd34019e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:29.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.481+0000 7fd33f7fe700 1 -- 172.21.15.70:0/2675602080 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd34019f000 con 0x7fd3401024c0 2024-06-04T04:56:29.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.481+0000 7fd33f7fe700 1 --2- 172.21.15.70:0/2675602080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3401024c0 0x7fd3401973a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd33000d9b0 tx=0x7fd33000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:29.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.482+0000 7fd33cff9700 1 -- 172.21.15.70:0/2675602080 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd330017070 con 0x7fd3401024c0 2024-06-04T04:56:29.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.482+0000 7fd34604e700 1 -- 172.21.15.70:0/2675602080 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd34019f2f0 con 0x7fd3401024c0 2024-06-04T04:56:29.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.482+0000 7fd33cff9700 1 -- 172.21.15.70:0/2675602080 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd330009dc0 con 0x7fd3401024c0 2024-06-04T04:56:29.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.482+0000 7fd33cff9700 1 -- 172.21.15.70:0/2675602080 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd330010de0 con 0x7fd3401024c0 2024-06-04T04:56:29.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.482+0000 7fd34604e700 1 -- 172.21.15.70:0/2675602080 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd34019f830 con 0x7fd3401024c0 2024-06-04T04:56:29.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.483+0000 7fd34604e700 1 -- 172.21.15.70:0/2675602080 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd34006a230 con 0x7fd3401024c0 2024-06-04T04:56:29.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.484+0000 7fd33cff9700 1 -- 172.21.15.70:0/2675602080 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fd330010460 con 0x7fd3401024c0 2024-06-04T04:56:29.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.485+0000 7fd33cff9700 1 --2- 172.21.15.70:0/2675602080 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd3280b2880 0x7fd3280b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:29.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.485+0000 7fd33effd700 1 --2- 172.21.15.70:0/2675602080 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd3280b2880 0x7fd3280b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:29.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.485+0000 7fd33cff9700 1 -- 172.21.15.70:0/2675602080 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fd3300e1cf0 con 0x7fd3401024c0 2024-06-04T04:56:29.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.485+0000 7fd33effd700 1 --2- 172.21.15.70:0/2675602080 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd3280b2880 0x7fd3280b4d40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd334007be0 tx=0x7fd3340073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:29.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.488+0000 7fd33cff9700 1 -- 172.21.15.70:0/2675602080 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd3300a99f0 con 0x7fd3401024c0 2024-06-04T04:56:29.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:29 smithi070 ceph-mon[32631]: pgmap v376: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:56:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:29 smithi112 ceph-mon[36453]: pgmap v376: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:56:29.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.704+0000 7fd34604e700 1 -- 172.21.15.70:0/2675602080 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fd340063980 con 0x7fd3280b2880 2024-06-04T04:56:29.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.706+0000 7fd33cff9700 1 -- 172.21.15.70:0/2675602080 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fd340063980 con 0x7fd3280b2880 2024-06-04T04:56:29.708 INFO:teuthology.orchestra.run.smithi070.stdout:HOST ADDR LABELS STATUS 2024-06-04T04:56:29.708 INFO:teuthology.orchestra.run.smithi070.stdout:smithi070 172.21.15.70 2024-06-04T04:56:29.708 INFO:teuthology.orchestra.run.smithi070.stdout:smithi098 172.21.15.98 2024-06-04T04:56:29.709 INFO:teuthology.orchestra.run.smithi070.stdout:smithi112 172.21.15.112 2024-06-04T04:56:29.709 INFO:teuthology.orchestra.run.smithi070.stdout:3 hosts in cluster 2024-06-04T04:56:29.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.708+0000 7fd34604e700 1 -- 172.21.15.70:0/2675602080 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd3280b2880 msgr2=0x7fd3280b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:29.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.708+0000 7fd34604e700 1 --2- 172.21.15.70:0/2675602080 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd3280b2880 0x7fd3280b4d40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd334007be0 tx=0x7fd3340073d0 comp rx=0 tx=0).stop 2024-06-04T04:56:29.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.708+0000 7fd34604e700 1 -- 172.21.15.70:0/2675602080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3401024c0 msgr2=0x7fd3401973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:29.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.708+0000 7fd34604e700 1 --2- 172.21.15.70:0/2675602080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3401024c0 0x7fd3401973a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd33000d9b0 tx=0x7fd33000dcc0 comp rx=0 tx=0).stop 2024-06-04T04:56:29.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.708+0000 7fd34604e700 1 -- 172.21.15.70:0/2675602080 shutdown_connections 2024-06-04T04:56:29.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.708+0000 7fd34604e700 1 --2- 172.21.15.70:0/2675602080 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd3280b2880 0x7fd3280b4d40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:29.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.708+0000 7fd34604e700 1 --2- 172.21.15.70:0/2675602080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd340104060 0x7fd34019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:29.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.708+0000 7fd34604e700 1 --2- 172.21.15.70:0/2675602080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3401024c0 0x7fd3401973a0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:29.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.708+0000 7fd34604e700 1 --2- 172.21.15.70:0/2675602080 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3401036c0 0x7fd3401978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:29.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.708+0000 7fd34604e700 1 -- 172.21.15.70:0/2675602080 >> 172.21.15.70:0/2675602080 conn(0x7fd3400fdaa0 msgr2=0x7fd340106a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:29.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.708+0000 7fd34604e700 1 -- 172.21.15.70:0/2675602080 shutdown_connections 2024-06-04T04:56:29.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:29.708+0000 7fd34604e700 1 -- 172.21.15.70:0/2675602080 wait complete. 2024-06-04T04:56:29.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:29 smithi098 ceph-mon[35569]: pgmap v376: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:56:30.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch device ls' 2024-06-04T04:56:30.786 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:31.590 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:31 smithi070 ceph-mon[32631]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:56:31.590 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:31 smithi070 ceph-mon[32631]: pgmap v377: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:56:31.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:31 smithi112 ceph-mon[36453]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:56:31.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:31 smithi112 ceph-mon[36453]: pgmap v377: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:56:31.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:31 smithi098 ceph-mon[35569]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:56:31.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:31 smithi098 ceph-mon[35569]: pgmap v377: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:56:32.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.516+0000 7f4119f69700 1 -- 172.21.15.70:0/19352297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4114102490 msgr2=0x7f41141028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:32.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.516+0000 7f4119f69700 1 --2- 172.21.15.70:0/19352297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4114102490 0x7f41141028b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f410400b0d0 tx=0x7f410400b3e0 comp rx=0 tx=0).stop 2024-06-04T04:56:32.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.516+0000 7f4119f69700 1 -- 172.21.15.70:0/19352297 shutdown_connections 2024-06-04T04:56:32.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.516+0000 7f4119f69700 1 --2- 172.21.15.70:0/19352297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4114104030 0x7f411410a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:32.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.516+0000 7f4119f69700 1 --2- 172.21.15.70:0/19352297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4114102490 0x7f41141028b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:32.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.516+0000 7f4119f69700 1 --2- 172.21.15.70:0/19352297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4114103690 0x7f4114103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:32.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.516+0000 7f4119f69700 1 -- 172.21.15.70:0/19352297 >> 172.21.15.70:0/19352297 conn(0x7f41140fda70 msgr2=0x7f41140ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:32.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.516+0000 7f4119f69700 1 -- 172.21.15.70:0/19352297 shutdown_connections 2024-06-04T04:56:32.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.516+0000 7f4119f69700 1 -- 172.21.15.70:0/19352297 wait complete. 2024-06-04T04:56:32.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.516+0000 7f4119f69700 1 Processor -- start 2024-06-04T04:56:32.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.516+0000 7f4119f69700 1 -- start start 2024-06-04T04:56:32.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f4119f69700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4114102490 0x7f4114198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:32.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f4119f69700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4114103690 0x7f4114198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:32.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f4119f69700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4114104030 0x7f411419fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:32.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f4119f69700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f411410de30 con 0x7f4114102490 2024-06-04T04:56:32.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f4119f69700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f411410dcb0 con 0x7f4114104030 2024-06-04T04:56:32.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f4119f69700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f411410dfb0 con 0x7f4114103690 2024-06-04T04:56:32.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f41137fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4114102490 0x7f4114198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:32.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f41137fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4114102490 0x7f4114198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35850/0 (socket says 172.21.15.70:35850) 2024-06-04T04:56:32.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f41137fe700 1 -- 172.21.15.70:0/2359415237 learned_addr learned my addr 172.21.15.70:0/2359415237 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:32.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f4112ffd700 1 --2- 172.21.15.70:0/2359415237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4114103690 0x7f4114198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:32.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f4113fff700 1 --2- 172.21.15.70:0/2359415237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4114104030 0x7f411419fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:32.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f41137fe700 1 -- 172.21.15.70:0/2359415237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4114103690 msgr2=0x7f4114198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:32.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f41137fe700 1 --2- 172.21.15.70:0/2359415237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4114103690 0x7f4114198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:32.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f41137fe700 1 -- 172.21.15.70:0/2359415237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4114104030 msgr2=0x7f411419fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:32.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f41137fe700 1 --2- 172.21.15.70:0/2359415237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4114104030 0x7f411419fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:32.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f41137fe700 1 -- 172.21.15.70:0/2359415237 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41141a01b0 con 0x7f4114102490 2024-06-04T04:56:32.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f4112ffd700 1 --2- 172.21.15.70:0/2359415237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4114103690 0x7f4114198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:32.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f41137fe700 1 --2- 172.21.15.70:0/2359415237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4114102490 0x7f4114198550 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4104009c00 tx=0x7f410400bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:32.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f4110ff9700 1 -- 172.21.15.70:0/2359415237 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104027070 con 0x7f4114102490 2024-06-04T04:56:32.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.517+0000 7f4119f69700 1 -- 172.21.15.70:0/2359415237 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f411406bcb0 con 0x7f4114102490 2024-06-04T04:56:32.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.518+0000 7f4110ff9700 1 -- 172.21.15.70:0/2359415237 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f410401a950 con 0x7f4114102490 2024-06-04T04:56:32.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.518+0000 7f4110ff9700 1 -- 172.21.15.70:0/2359415237 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104005480 con 0x7f4114102490 2024-06-04T04:56:32.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.518+0000 7f4119f69700 1 -- 172.21.15.70:0/2359415237 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f411406c2c0 con 0x7f4114102490 2024-06-04T04:56:32.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.518+0000 7f4119f69700 1 -- 172.21.15.70:0/2359415237 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f411406a230 con 0x7f4114102490 2024-06-04T04:56:32.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.520+0000 7f4110ff9700 1 -- 172.21.15.70:0/2359415237 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f410401ab00 con 0x7f4114102490 2024-06-04T04:56:32.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.520+0000 7f4110ff9700 1 --2- 172.21.15.70:0/2359415237 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f40fc0b2880 0x7f40fc0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:32.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.520+0000 7f4112ffd700 1 --2- 172.21.15.70:0/2359415237 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f40fc0b2880 0x7f40fc0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:32.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.520+0000 7f4110ff9700 1 -- 172.21.15.70:0/2359415237 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f410400f020 con 0x7f4114102490 2024-06-04T04:56:32.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.521+0000 7f4112ffd700 1 --2- 172.21.15.70:0/2359415237 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f40fc0b2880 0x7f40fc0b4d40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f41080099d0 tx=0x7f4108008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:32.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.524+0000 7f4110ff9700 1 -- 172.21.15.70:0/2359415237 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4104030020 con 0x7f4114102490 2024-06-04T04:56:32.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.741+0000 7f4119f69700 1 -- 172.21.15.70:0/2359415237 --> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f411406c520 con 0x7f40fc0b2880 2024-06-04T04:56:32.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.742+0000 7f4110ff9700 1 -- 172.21.15.70:0/2359415237 <== mgr.14152 v2:172.21.15.70:6800/2752966840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f411406c520 con 0x7f40fc0b2880 2024-06-04T04:56:32.744 INFO:teuthology.orchestra.run.smithi070.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-06-04T04:56:32.745 INFO:teuthology.orchestra.run.smithi070.stdout:smithi070 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513002P480DGN 447G No 4m ago Has a FileSystem, LVM detected 2024-06-04T04:56:32.745 INFO:teuthology.orchestra.run.smithi070.stdout:smithi098 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010H400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-06-04T04:56:32.745 INFO:teuthology.orchestra.run.smithi070.stdout:smithi112 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NV400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-06-04T04:56:32.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.745+0000 7f4119f69700 1 -- 172.21.15.70:0/2359415237 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f40fc0b2880 msgr2=0x7f40fc0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:32.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.745+0000 7f4119f69700 1 --2- 172.21.15.70:0/2359415237 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f40fc0b2880 0x7f40fc0b4d40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f41080099d0 tx=0x7f4108008040 comp rx=0 tx=0).stop 2024-06-04T04:56:32.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.745+0000 7f4119f69700 1 -- 172.21.15.70:0/2359415237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4114102490 msgr2=0x7f4114198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:32.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.745+0000 7f4119f69700 1 --2- 172.21.15.70:0/2359415237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4114102490 0x7f4114198550 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4104009c00 tx=0x7f410400bd60 comp rx=0 tx=0).stop 2024-06-04T04:56:32.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.745+0000 7f4119f69700 1 -- 172.21.15.70:0/2359415237 shutdown_connections 2024-06-04T04:56:32.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.745+0000 7f4119f69700 1 --2- 172.21.15.70:0/2359415237 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f40fc0b2880 0x7f40fc0b4d40 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:32.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.745+0000 7f4119f69700 1 --2- 172.21.15.70:0/2359415237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4114104030 0x7f411419fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:32.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.745+0000 7f4119f69700 1 --2- 172.21.15.70:0/2359415237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4114102490 0x7f4114198550 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:32.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.745+0000 7f4119f69700 1 --2- 172.21.15.70:0/2359415237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4114103690 0x7f4114198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:32.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.745+0000 7f4119f69700 1 -- 172.21.15.70:0/2359415237 >> 172.21.15.70:0/2359415237 conn(0x7f41140fda70 msgr2=0x7f4114106ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:32.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.745+0000 7f4119f69700 1 -- 172.21.15.70:0/2359415237 shutdown_connections 2024-06-04T04:56:32.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:32.745+0000 7f4119f69700 1 -- 172.21.15.70:0/2359415237 wait complete. 2024-06-04T04:56:33.429 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:33 smithi070 ceph-mon[32631]: pgmap v378: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:56:33.430 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-06-04T04:56:33.439 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi070.front.sepia.ceph.com 2024-06-04T04:56:33.439 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- bash -c 'ceph fs dump' 2024-06-04T04:56:33.679 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:33 smithi112 ceph-mon[36453]: pgmap v378: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:56:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:33 smithi098 ceph-mon[35569]: pgmap v378: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:56:34.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:34 smithi112 ceph-mon[36453]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:56:34.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:34 smithi098 ceph-mon[35569]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:56:34.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:34 smithi070 ceph-mon[32631]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:56:35.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.443+0000 7fd1d7b8e700 1 -- 172.21.15.70:0/2422578041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1d0101920 msgr2=0x7fd1d0101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:35.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.443+0000 7fd1d7b8e700 1 --2- 172.21.15.70:0/2422578041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1d0101920 0x7fd1d0101d40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800b0a0 tx=0x7fd1c800b3b0 comp rx=0 tx=0).stop 2024-06-04T04:56:35.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.443+0000 7fd1d7b8e700 1 -- 172.21.15.70:0/2422578041 shutdown_connections 2024-06-04T04:56:35.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.443+0000 7fd1d7b8e700 1 --2- 172.21.15.70:0/2422578041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1d01052a0 0x7fd1d010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:35.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.443+0000 7fd1d7b8e700 1 --2- 172.21.15.70:0/2422578041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1d0101920 0x7fd1d0101d40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:35.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.443+0000 7fd1d7b8e700 1 --2- 172.21.15.70:0/2422578041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d0104900 0x7fd1d0104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:35.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.443+0000 7fd1d7b8e700 1 -- 172.21.15.70:0/2422578041 >> 172.21.15.70:0/2422578041 conn(0x7fd1d00fd280 msgr2=0x7fd1d00ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:35.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.443+0000 7fd1d7b8e700 1 -- 172.21.15.70:0/2422578041 shutdown_connections 2024-06-04T04:56:35.447 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:35 smithi070 ceph-mon[32631]: pgmap v379: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:56:35.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.443+0000 7fd1d7b8e700 1 -- 172.21.15.70:0/2422578041 wait complete. 2024-06-04T04:56:35.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.444+0000 7fd1d7b8e700 1 Processor -- start 2024-06-04T04:56:35.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.444+0000 7fd1d7b8e700 1 -- start start 2024-06-04T04:56:35.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.444+0000 7fd1d7b8e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d0101920 0x7fd1d0199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:35.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.444+0000 7fd1d7b8e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1d0104900 0x7fd1d0199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:35.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.444+0000 7fd1d7b8e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1d01052a0 0x7fd1d01a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:35.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.444+0000 7fd1d7b8e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d010eff0 con 0x7fd1d01052a0 2024-06-04T04:56:35.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.444+0000 7fd1d7b8e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d010ee70 con 0x7fd1d0101920 2024-06-04T04:56:35.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.444+0000 7fd1d7b8e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d010f170 con 0x7fd1d0104900 2024-06-04T04:56:35.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.444+0000 7fd1d612b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1d01052a0 0x7fd1d01a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:35.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.445+0000 7fd1d612b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1d01052a0 0x7fd1d01a0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35870/0 (socket says 172.21.15.70:35870) 2024-06-04T04:56:35.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.445+0000 7fd1d612b700 1 -- 172.21.15.70:0/544216054 learned_addr learned my addr 172.21.15.70:0/544216054 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:35.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.445+0000 7fd1d592a700 1 --2- 172.21.15.70:0/544216054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d0101920 0x7fd1d0199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:35.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.445+0000 7fd1d5129700 1 --2- 172.21.15.70:0/544216054 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1d0104900 0x7fd1d0199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:35.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.445+0000 7fd1d612b700 1 -- 172.21.15.70:0/544216054 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1d0104900 msgr2=0x7fd1d0199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:35.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.445+0000 7fd1d612b700 1 --2- 172.21.15.70:0/544216054 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1d0104900 0x7fd1d0199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:35.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.445+0000 7fd1d612b700 1 -- 172.21.15.70:0/544216054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d0101920 msgr2=0x7fd1d0199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:35.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.445+0000 7fd1d612b700 1 --2- 172.21.15.70:0/544216054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d0101920 0x7fd1d0199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:35.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.445+0000 7fd1d612b700 1 -- 172.21.15.70:0/544216054 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d01a1340 con 0x7fd1d01052a0 2024-06-04T04:56:35.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.445+0000 7fd1d5129700 1 --2- 172.21.15.70:0/544216054 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1d0104900 0x7fd1d0199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:35.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.445+0000 7fd1d612b700 1 --2- 172.21.15.70:0/544216054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1d01052a0 0x7fd1d01a0ca0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd1c000b580 tx=0x7fd1c000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:35.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.445+0000 7fd1ceffd700 1 -- 172.21.15.70:0/544216054 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0015020 con 0x7fd1d01052a0 2024-06-04T04:56:35.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.445+0000 7fd1d7b8e700 1 -- 172.21.15.70:0/544216054 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d01a1630 con 0x7fd1d01052a0 2024-06-04T04:56:35.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.445+0000 7fd1ceffd700 1 -- 172.21.15.70:0/544216054 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd1c0011cc0 con 0x7fd1d01052a0 2024-06-04T04:56:35.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.445+0000 7fd1ceffd700 1 -- 172.21.15.70:0/544216054 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0010c80 con 0x7fd1d01052a0 2024-06-04T04:56:35.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.445+0000 7fd1d7b8e700 1 -- 172.21.15.70:0/544216054 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d01a1bc0 con 0x7fd1d01052a0 2024-06-04T04:56:35.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.446+0000 7fd1d7b8e700 1 -- 172.21.15.70:0/544216054 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1d006a230 con 0x7fd1d01052a0 2024-06-04T04:56:35.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.448+0000 7fd1ceffd700 1 -- 172.21.15.70:0/544216054 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fd1c0018020 con 0x7fd1d01052a0 2024-06-04T04:56:35.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.449+0000 7fd1ceffd700 1 --2- 172.21.15.70:0/544216054 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd1b80b2880 0x7fd1b80b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:35.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.449+0000 7fd1d592a700 1 --2- 172.21.15.70:0/544216054 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd1b80b2880 0x7fd1b80b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:35.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.449+0000 7fd1ceffd700 1 -- 172.21.15.70:0/544216054 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fd1c00e1eb0 con 0x7fd1d01052a0 2024-06-04T04:56:35.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.450+0000 7fd1d592a700 1 --2- 172.21.15.70:0/544216054 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd1b80b2880 0x7fd1b80b4d40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8007ed0 tx=0x7fd1c800b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:35.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.453+0000 7fd1ceffd700 1 -- 172.21.15.70:0/544216054 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd1c0111df0 con 0x7fd1d01052a0 2024-06-04T04:56:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:35 smithi112 ceph-mon[36453]: pgmap v379: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:56:35.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.701+0000 7fd1d7b8e700 1 -- 172.21.15.70:0/544216054 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fd1d002e560 con 0x7fd1d01052a0 2024-06-04T04:56:35.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.702+0000 7fd1ceffd700 1 -- 172.21.15.70:0/544216054 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3089 (secure 0 0 0) 0x7fd1c001f020 con 0x7fd1d01052a0 2024-06-04T04:56:35.706 INFO:teuthology.orchestra.run.smithi070.stdout:e28 2024-06-04T04:56:35.706 INFO:teuthology.orchestra.run.smithi070.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-06-04T04:56:35.706 INFO:teuthology.orchestra.run.smithi070.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-06-04T04:56:35.706 INFO:teuthology.orchestra.run.smithi070.stdout:legacy client fscid: 1 2024-06-04T04:56:35.706 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:56:35.706 INFO:teuthology.orchestra.run.smithi070.stdout:Filesystem 'cephfs' (1) 2024-06-04T04:56:35.706 INFO:teuthology.orchestra.run.smithi070.stdout:fs_name cephfs 2024-06-04T04:56:35.706 INFO:teuthology.orchestra.run.smithi070.stdout:epoch 28 2024-06-04T04:56:35.707 INFO:teuthology.orchestra.run.smithi070.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-06-04T04:56:35.707 INFO:teuthology.orchestra.run.smithi070.stdout:created 2024-06-04T04:53:38.237993+0000 2024-06-04T04:56:35.707 INFO:teuthology.orchestra.run.smithi070.stdout:modified 2024-06-04T04:53:56.441682+0000 2024-06-04T04:56:35.707 INFO:teuthology.orchestra.run.smithi070.stdout:tableserver 0 2024-06-04T04:56:35.707 INFO:teuthology.orchestra.run.smithi070.stdout:root 0 2024-06-04T04:56:35.707 INFO:teuthology.orchestra.run.smithi070.stdout:session_timeout 300 2024-06-04T04:56:35.707 INFO:teuthology.orchestra.run.smithi070.stdout:session_autoclose 300 2024-06-04T04:56:35.707 INFO:teuthology.orchestra.run.smithi070.stdout:max_file_size 1099511627776 2024-06-04T04:56:35.707 INFO:teuthology.orchestra.run.smithi070.stdout:max_xattr_size 65536 2024-06-04T04:56:35.707 INFO:teuthology.orchestra.run.smithi070.stdout:required_client_features {} 2024-06-04T04:56:35.707 INFO:teuthology.orchestra.run.smithi070.stdout:last_failure 0 2024-06-04T04:56:35.707 INFO:teuthology.orchestra.run.smithi070.stdout:last_failure_osd_epoch 0 2024-06-04T04:56:35.707 INFO:teuthology.orchestra.run.smithi070.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-06-04T04:56:35.707 INFO:teuthology.orchestra.run.smithi070.stdout:max_mds 5 2024-06-04T04:56:35.707 INFO:teuthology.orchestra.run.smithi070.stdout:in 0,1,2,3,4 2024-06-04T04:56:35.708 INFO:teuthology.orchestra.run.smithi070.stdout:up {0=24448,1=24442,2=24440,3=24436,4=24410} 2024-06-04T04:56:35.708 INFO:teuthology.orchestra.run.smithi070.stdout:failed 2024-06-04T04:56:35.708 INFO:teuthology.orchestra.run.smithi070.stdout:damaged 2024-06-04T04:56:35.708 INFO:teuthology.orchestra.run.smithi070.stdout:stopped 2024-06-04T04:56:35.708 INFO:teuthology.orchestra.run.smithi070.stdout:data_pools [3] 2024-06-04T04:56:35.708 INFO:teuthology.orchestra.run.smithi070.stdout:metadata_pool 2 2024-06-04T04:56:35.708 INFO:teuthology.orchestra.run.smithi070.stdout:inline_data disabled 2024-06-04T04:56:35.708 INFO:teuthology.orchestra.run.smithi070.stdout:balancer 2024-06-04T04:56:35.708 INFO:teuthology.orchestra.run.smithi070.stdout:bal_rank_mask -1 2024-06-04T04:56:35.708 INFO:teuthology.orchestra.run.smithi070.stdout:standby_count_wanted 0 2024-06-04T04:56:35.708 INFO:teuthology.orchestra.run.smithi070.stdout:qdb_cluster leader: 24448 members: 24410,24436,24440,24442,24448 2024-06-04T04:56:35.708 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.i{0:24448} state up:active seq 8 addr [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] compat {c=[1],r=[1],i=[1fff]}] 2024-06-04T04:56:35.708 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.l{0:24400} state up:standby-replay seq 1 addr [v2:172.21.15.112:6832/1359739019,v1:172.21.15.112:6833/1359739019] compat {c=[1],r=[1],i=[1fff]}] 2024-06-04T04:56:35.708 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.c{1:24442} state up:active seq 12 addr [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] compat {c=[1],r=[1],i=[1fff]}] 2024-06-04T04:56:35.708 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.e{1:24386} state up:standby-replay seq 1 addr [v2:172.21.15.98:6834/3148471566,v1:172.21.15.98:6835/3148471566] compat {c=[1],r=[1],i=[1fff]}] 2024-06-04T04:56:35.709 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.b{2:24440} state up:active seq 8 addr [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] compat {c=[1],r=[1],i=[1fff]}] 2024-06-04T04:56:35.709 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.k{2:24371} state up:standby-replay seq 1 addr [v2:172.21.15.98:6832/1896647585,v1:172.21.15.98:6833/1896647585] compat {c=[1],r=[1],i=[1fff]}] 2024-06-04T04:56:35.709 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.f{3:24436} state up:active seq 14 addr [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] compat {c=[1],r=[1],i=[1fff]}] 2024-06-04T04:56:35.709 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.a{3:14592} state up:standby-replay seq 1 addr [v2:172.21.15.70:6840/2748767095,v1:172.21.15.70:6841/2748767095] compat {c=[1],r=[1],i=[1fff]}] 2024-06-04T04:56:35.709 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.h{4:24410} state up:active seq 12 addr [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] compat {c=[1],r=[1],i=[1fff]}] 2024-06-04T04:56:35.709 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.g{4:14580} state up:standby-replay seq 1 addr [v2:172.21.15.70:6838/3817259285,v1:172.21.15.70:6839/3817259285] compat {c=[1],r=[1],i=[1fff]}] 2024-06-04T04:56:35.709 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:56:35.709 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:56:35.709 INFO:teuthology.orchestra.run.smithi070.stdout:Standby daemons: 2024-06-04T04:56:35.709 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:56:35.709 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.j{-1:14526} state up:standby seq 1 addr [v2:172.21.15.70:6834/2694011970,v1:172.21.15.70:6835/2694011970] compat {c=[1],r=[1],i=[1fff]}] 2024-06-04T04:56:35.709 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.d{-1:14538} state up:standby seq 1 addr [v2:172.21.15.70:6836/729112700,v1:172.21.15.70:6837/729112700] compat {c=[1],r=[1],i=[1fff]}] 2024-06-04T04:56:35.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.704+0000 7fd1d7b8e700 1 -- 172.21.15.70:0/544216054 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd1b80b2880 msgr2=0x7fd1b80b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:35.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.704+0000 7fd1d7b8e700 1 --2- 172.21.15.70:0/544216054 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd1b80b2880 0x7fd1b80b4d40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8007ed0 tx=0x7fd1c800b7b0 comp rx=0 tx=0).stop 2024-06-04T04:56:35.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.704+0000 7fd1d7b8e700 1 -- 172.21.15.70:0/544216054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1d01052a0 msgr2=0x7fd1d01a0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:35.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.704+0000 7fd1d7b8e700 1 --2- 172.21.15.70:0/544216054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1d01052a0 0x7fd1d01a0ca0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd1c000b580 tx=0x7fd1c000b890 comp rx=0 tx=0).stop 2024-06-04T04:56:35.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.704+0000 7fd1d7b8e700 1 -- 172.21.15.70:0/544216054 shutdown_connections 2024-06-04T04:56:35.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.704+0000 7fd1d7b8e700 1 --2- 172.21.15.70:0/544216054 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fd1b80b2880 0x7fd1b80b4d40 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:35.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.704+0000 7fd1d7b8e700 1 --2- 172.21.15.70:0/544216054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1d01052a0 0x7fd1d01a0ca0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:35.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.704+0000 7fd1d7b8e700 1 --2- 172.21.15.70:0/544216054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d0101920 0x7fd1d0199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:35.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.704+0000 7fd1d7b8e700 1 --2- 172.21.15.70:0/544216054 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1d0104900 0x7fd1d0199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:35.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.704+0000 7fd1d7b8e700 1 -- 172.21.15.70:0/544216054 >> 172.21.15.70:0/544216054 conn(0x7fd1d00fd280 msgr2=0x7fd1d0107d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:35.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.704+0000 7fd1d7b8e700 1 -- 172.21.15.70:0/544216054 shutdown_connections 2024-06-04T04:56:35.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:35.705+0000 7fd1d7b8e700 1 -- 172.21.15.70:0/544216054 wait complete. 2024-06-04T04:56:35.711 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 28 2024-06-04T04:56:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:35 smithi098 ceph-mon[35569]: pgmap v379: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:56:36.524 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- bash -c 'ceph osd dump' 2024-06-04T04:56:36.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:36 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/544216054' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-06-04T04:56:36.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:36 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/544216054' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-06-04T04:56:36.767 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:36.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:36 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/544216054' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-06-04T04:56:37.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:37 smithi070 ceph-mon[32631]: pgmap v380: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:37 smithi112 ceph-mon[36453]: pgmap v380: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:37 smithi098 ceph-mon[35569]: pgmap v380: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:38.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.433+0000 7f2de4254700 1 -- 172.21.15.70:0/1256885297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ddc1024a0 msgr2=0x7f2ddc1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:38.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.433+0000 7f2de4254700 1 --2- 172.21.15.70:0/1256885297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ddc1024a0 0x7f2ddc1028c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0099c0 tx=0x7f2dcc009cd0 comp rx=0 tx=0).stop 2024-06-04T04:56:38.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.433+0000 7f2de4254700 1 -- 172.21.15.70:0/1256885297 shutdown_connections 2024-06-04T04:56:38.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.433+0000 7f2de4254700 1 --2- 172.21.15.70:0/1256885297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ddc104040 0x7f2ddc10a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:38.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.433+0000 7f2de4254700 1 --2- 172.21.15.70:0/1256885297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ddc1024a0 0x7f2ddc1028c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:38.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.433+0000 7f2de4254700 1 --2- 172.21.15.70:0/1256885297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ddc1036a0 0x7f2ddc103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:38.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.434+0000 7f2de4254700 1 -- 172.21.15.70:0/1256885297 >> 172.21.15.70:0/1256885297 conn(0x7f2ddc0fda40 msgr2=0x7f2ddc0ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:38.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.434+0000 7f2de4254700 1 -- 172.21.15.70:0/1256885297 shutdown_connections 2024-06-04T04:56:38.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.434+0000 7f2de4254700 1 -- 172.21.15.70:0/1256885297 wait complete. 2024-06-04T04:56:38.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.434+0000 7f2de4254700 1 Processor -- start 2024-06-04T04:56:38.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.434+0000 7f2de4254700 1 -- start start 2024-06-04T04:56:38.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.434+0000 7f2de4254700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ddc1024a0 0x7f2ddc1997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:38.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.434+0000 7f2de4254700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ddc1036a0 0x7f2ddc199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:38.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.434+0000 7f2de4254700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ddc104040 0x7f2ddc1a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:38.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.434+0000 7f2de4254700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2ddc10dec0 con 0x7f2ddc104040 2024-06-04T04:56:38.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.434+0000 7f2de4254700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ddc10dd40 con 0x7f2ddc1036a0 2024-06-04T04:56:38.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.434+0000 7f2de4254700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2ddc10e040 con 0x7f2ddc1024a0 2024-06-04T04:56:38.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.435+0000 7f2de27f1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ddc104040 0x7f2ddc1a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:38.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.435+0000 7f2de27f1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ddc104040 0x7f2ddc1a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42296/0 (socket says 172.21.15.70:42296) 2024-06-04T04:56:38.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.435+0000 7f2de27f1700 1 -- 172.21.15.70:0/3504175109 learned_addr learned my addr 172.21.15.70:0/3504175109 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:38.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.435+0000 7f2de1ff0700 1 --2- 172.21.15.70:0/3504175109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ddc1024a0 0x7f2ddc1997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:38.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.435+0000 7f2de17ef700 1 --2- 172.21.15.70:0/3504175109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ddc1036a0 0x7f2ddc199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:38.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.435+0000 7f2de27f1700 1 -- 172.21.15.70:0/3504175109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ddc1024a0 msgr2=0x7f2ddc1997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:38.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.435+0000 7f2de27f1700 1 --2- 172.21.15.70:0/3504175109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ddc1024a0 0x7f2ddc1997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:38.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.435+0000 7f2de27f1700 1 -- 172.21.15.70:0/3504175109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ddc1036a0 msgr2=0x7f2ddc199ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:38.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.435+0000 7f2de27f1700 1 --2- 172.21.15.70:0/3504175109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ddc1036a0 0x7f2ddc199ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:38.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.435+0000 7f2de27f1700 1 -- 172.21.15.70:0/3504175109 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ddc1a13b0 con 0x7f2ddc104040 2024-06-04T04:56:38.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.435+0000 7f2de27f1700 1 --2- 172.21.15.70:0/3504175109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ddc104040 0x7f2ddc1a0d10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400b9a0 tx=0x7f2dd400bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:38.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.435+0000 7f2dd2ffd700 1 -- 172.21.15.70:0/3504175109 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd400d850 con 0x7f2ddc104040 2024-06-04T04:56:38.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.435+0000 7f2de4254700 1 -- 172.21.15.70:0/3504175109 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ddc1a16a0 con 0x7f2ddc104040 2024-06-04T04:56:38.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.435+0000 7f2dd2ffd700 1 -- 172.21.15.70:0/3504175109 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2dd400e420 con 0x7f2ddc104040 2024-06-04T04:56:38.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.435+0000 7f2dd2ffd700 1 -- 172.21.15.70:0/3504175109 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4017660 con 0x7f2ddc104040 2024-06-04T04:56:38.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.435+0000 7f2de4254700 1 -- 172.21.15.70:0/3504175109 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ddc1a1c30 con 0x7f2ddc104040 2024-06-04T04:56:38.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.436+0000 7f2de4254700 1 -- 172.21.15.70:0/3504175109 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ddc06a230 con 0x7f2ddc104040 2024-06-04T04:56:38.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.437+0000 7f2dd2ffd700 1 -- 172.21.15.70:0/3504175109 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f2dd400d9f0 con 0x7f2ddc104040 2024-06-04T04:56:38.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.438+0000 7f2dd2ffd700 1 --2- 172.21.15.70:0/3504175109 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2dc80b2880 0x7f2dc80b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:38.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.438+0000 7f2de1ff0700 1 --2- 172.21.15.70:0/3504175109 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2dc80b2880 0x7f2dc80b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:38.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.438+0000 7f2dd2ffd700 1 -- 172.21.15.70:0/3504175109 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f2dd40e21a0 con 0x7f2ddc104040 2024-06-04T04:56:38.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.439+0000 7f2de1ff0700 1 --2- 172.21.15.70:0/3504175109 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2dc80b2880 0x7f2dc80b4d40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009990 tx=0x7f2dcc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:38.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.442+0000 7f2dd2ffd700 1 -- 172.21.15.70:0/3504175109 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2dd4013030 con 0x7f2ddc104040 2024-06-04T04:56:38.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.653+0000 7f2de4254700 1 -- 172.21.15.70:0/3504175109 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f2ddc1a1f50 con 0x7f2ddc104040 2024-06-04T04:56:38.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.654+0000 7f2dd2ffd700 1 -- 172.21.15.70:0/3504175109 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v81) ==== 56+0+5321 (secure 0 0 0) 0x7f2dd40a9ea0 con 0x7f2ddc104040 2024-06-04T04:56:38.656 INFO:teuthology.orchestra.run.smithi070.stdout:epoch 81 2024-06-04T04:56:38.656 INFO:teuthology.orchestra.run.smithi070.stdout:fsid 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T04:56:38.656 INFO:teuthology.orchestra.run.smithi070.stdout:created 2024-06-04T04:45:22.297536+0000 2024-06-04T04:56:38.656 INFO:teuthology.orchestra.run.smithi070.stdout:modified 2024-06-04T04:54:19.502410+0000 2024-06-04T04:56:38.656 INFO:teuthology.orchestra.run.smithi070.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-06-04T04:56:38.656 INFO:teuthology.orchestra.run.smithi070.stdout:crush_version 26 2024-06-04T04:56:38.656 INFO:teuthology.orchestra.run.smithi070.stdout:full_ratio 0.95 2024-06-04T04:56:38.656 INFO:teuthology.orchestra.run.smithi070.stdout:backfillfull_ratio 0.9 2024-06-04T04:56:38.656 INFO:teuthology.orchestra.run.smithi070.stdout:nearfull_ratio 0.85 2024-06-04T04:56:38.657 INFO:teuthology.orchestra.run.smithi070.stdout:require_min_compat_client luminous 2024-06-04T04:56:38.657 INFO:teuthology.orchestra.run.smithi070.stdout:min_compat_client luminous 2024-06-04T04:56:38.657 INFO:teuthology.orchestra.run.smithi070.stdout:require_osd_release squid 2024-06-04T04:56:38.657 INFO:teuthology.orchestra.run.smithi070.stdout:stretch_mode_enabled false 2024-06-04T04:56:38.657 INFO:teuthology.orchestra.run.smithi070.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-06-04T04:56:38.657 INFO:teuthology.orchestra.run.smithi070.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-06-04T04:56:38.657 INFO:teuthology.orchestra.run.smithi070.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.69 2024-06-04T04:56:38.657 INFO:teuthology.orchestra.run.smithi070.stdout:max_osd 12 2024-06-04T04:56:38.657 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 up in weight 1 up_from 8 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6802/824119100,v1:172.21.15.70:6803/824119100] [v2:172.21.15.70:6804/824119100,v1:172.21.15.70:6805/824119100] exists,up ff491ea2-1906-48e4-883e-bc78e778260e 2024-06-04T04:56:38.657 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 up in weight 1 up_from 13 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6800/2744598742,v1:172.21.15.98:6801/2744598742] [v2:172.21.15.98:6802/2744598742,v1:172.21.15.98:6803/2744598742] exists,up bd5e4806-535b-49b5-ba6b-93228da35a84 2024-06-04T04:56:38.657 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 up in weight 1 up_from 18 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6800/1328098370,v1:172.21.15.112:6801/1328098370] [v2:172.21.15.112:6802/1328098370,v1:172.21.15.112:6803/1328098370] exists,up 8456fb55-910a-4a59-b84b-2c0f68346ac0 2024-06-04T04:56:38.657 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 up in weight 1 up_from 26 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6810/4041566346,v1:172.21.15.70:6811/4041566346] [v2:172.21.15.70:6812/4041566346,v1:172.21.15.70:6813/4041566346] exists,up c8fbc440-bf70-42d3-b01e-777ccfa7900b 2024-06-04T04:56:38.657 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 up in weight 1 up_from 32 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6808/3652775814,v1:172.21.15.98:6809/3652775814] [v2:172.21.15.98:6810/3652775814,v1:172.21.15.98:6811/3652775814] exists,up 549c8c70-8be8-46ae-bcb6-0fde32b40cd9 2024-06-04T04:56:38.658 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 up in weight 1 up_from 37 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6808/1857257899,v1:172.21.15.112:6809/1857257899] [v2:172.21.15.112:6810/1857257899,v1:172.21.15.112:6811/1857257899] exists,up 51d86af1-0d6c-4ae0-b090-00414fc1bf7a 2024-06-04T04:56:38.658 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 up in weight 1 up_from 43 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6818/1264676496,v1:172.21.15.70:6819/1264676496] [v2:172.21.15.70:6820/1264676496,v1:172.21.15.70:6821/1264676496] exists,up 526a8138-bfa8-419e-a5b5-e6b80f0ae004 2024-06-04T04:56:38.658 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 up in weight 1 up_from 48 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6816/709593020,v1:172.21.15.98:6817/709593020] [v2:172.21.15.98:6818/709593020,v1:172.21.15.98:6819/709593020] exists,up 1173f785-6ed1-47ef-9265-2e233ea9ee02 2024-06-04T04:56:38.658 INFO:teuthology.orchestra.run.smithi070.stdout:osd.8 up in weight 1 up_from 54 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6816/2250225413,v1:172.21.15.112:6817/2250225413] [v2:172.21.15.112:6818/2250225413,v1:172.21.15.112:6819/2250225413] exists,up 6424cc6e-24a7-4b81-9ce6-39203fecc7ae 2024-06-04T04:56:38.658 INFO:teuthology.orchestra.run.smithi070.stdout:osd.9 up in weight 1 up_from 60 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6826/3557688707,v1:172.21.15.70:6827/3557688707] [v2:172.21.15.70:6828/3557688707,v1:172.21.15.70:6829/3557688707] exists,up 8d666db8-8831-4d63-9e3c-a4bf2504149e 2024-06-04T04:56:38.658 INFO:teuthology.orchestra.run.smithi070.stdout:osd.10 up in weight 1 up_from 65 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6824/1896493718,v1:172.21.15.98:6825/1896493718] [v2:172.21.15.98:6826/1896493718,v1:172.21.15.98:6827/1896493718] exists,up ecb925d2-0263-4dfc-a2fd-9e27856022d5 2024-06-04T04:56:38.658 INFO:teuthology.orchestra.run.smithi070.stdout:osd.11 up in weight 1 up_from 71 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6824/1400793137,v1:172.21.15.112:6825/1400793137] [v2:172.21.15.112:6826/1400793137,v1:172.21.15.112:6827/1400793137] exists,up c8e3230c-bc3a-40bd-8dd0-5fef56e4df97 2024-06-04T04:56:38.658 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 2.4 [2,11] 2024-06-04T04:56:38.658 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 2.a [2,11] 2024-06-04T04:56:38.658 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 2.b [2,4] 2024-06-04T04:56:38.658 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 2.1f [8,1] 2024-06-04T04:56:38.658 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 3.2 [3,9] 2024-06-04T04:56:38.658 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 3.3 [4,9] 2024-06-04T04:56:38.659 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 3.17 [4,6] 2024-06-04T04:56:38.659 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 3.1d [4,8] 2024-06-04T04:56:38.659 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 3.21 [5,9] 2024-06-04T04:56:38.659 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 3.28 [5,7] 2024-06-04T04:56:38.659 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/3442074771 expires 2024-06-05T04:46:17.636907+0000 2024-06-04T04:56:38.659 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6801/1353782413 expires 2024-06-05T04:46:17.636907+0000 2024-06-04T04:56:38.659 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/1442095493 expires 2024-06-05T04:46:17.636907+0000 2024-06-04T04:56:38.659 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/2453733268 expires 2024-06-05T04:45:49.501864+0000 2024-06-04T04:56:38.659 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/1369781447 expires 2024-06-05T04:45:49.501864+0000 2024-06-04T04:56:38.659 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6800/2087909610 expires 2024-06-05T04:45:49.501864+0000 2024-06-04T04:56:38.659 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6801/2087909610 expires 2024-06-05T04:45:49.501864+0000 2024-06-04T04:56:38.659 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6800/1353782413 expires 2024-06-05T04:46:17.636907+0000 2024-06-04T04:56:38.660 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/915186456 expires 2024-06-05T04:46:17.636907+0000 2024-06-04T04:56:38.660 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/860033006 expires 2024-06-05T04:45:49.501864+0000 2024-06-04T04:56:38.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.656+0000 7f2de4254700 1 -- 172.21.15.70:0/3504175109 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2dc80b2880 msgr2=0x7f2dc80b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:38.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.656+0000 7f2de4254700 1 --2- 172.21.15.70:0/3504175109 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2dc80b2880 0x7f2dc80b4d40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009990 tx=0x7f2dcc005fb0 comp rx=0 tx=0).stop 2024-06-04T04:56:38.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.656+0000 7f2de4254700 1 -- 172.21.15.70:0/3504175109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ddc104040 msgr2=0x7f2ddc1a0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:38.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.656+0000 7f2de4254700 1 --2- 172.21.15.70:0/3504175109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ddc104040 0x7f2ddc1a0d10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400b9a0 tx=0x7f2dd400bcb0 comp rx=0 tx=0).stop 2024-06-04T04:56:38.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.656+0000 7f2de4254700 1 -- 172.21.15.70:0/3504175109 shutdown_connections 2024-06-04T04:56:38.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.656+0000 7f2de4254700 1 --2- 172.21.15.70:0/3504175109 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2dc80b2880 0x7f2dc80b4d40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:38.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.656+0000 7f2de4254700 1 --2- 172.21.15.70:0/3504175109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ddc104040 0x7f2ddc1a0d10 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:38.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.656+0000 7f2de4254700 1 --2- 172.21.15.70:0/3504175109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ddc1024a0 0x7f2ddc1997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:38.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.656+0000 7f2de4254700 1 --2- 172.21.15.70:0/3504175109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ddc1036a0 0x7f2ddc199ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:38.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.656+0000 7f2de4254700 1 -- 172.21.15.70:0/3504175109 >> 172.21.15.70:0/3504175109 conn(0x7f2ddc0fda40 msgr2=0x7f2ddc06b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:38.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.656+0000 7f2de4254700 1 -- 172.21.15.70:0/3504175109 shutdown_connections 2024-06-04T04:56:38.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:38.656+0000 7f2de4254700 1 -- 172.21.15.70:0/3504175109 wait complete. 2024-06-04T04:56:39.567 INFO:teuthology.run_tasks:Running task fs.ready... 2024-06-04T04:56:39.600 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:56:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:39 smithi112 ceph-mon[36453]: pgmap v381: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:56:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:39 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3504175109' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-04T04:56:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:39 smithi098 ceph-mon[35569]: pgmap v381: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:56:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:39 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3504175109' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-04T04:56:39.850 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:39.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:39 smithi070 ceph-mon[32631]: pgmap v381: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:56:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:39 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3504175109' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-04T04:56:41.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.657+0000 7f2f64f2b700 1 -- 172.21.15.70:0/3065368337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f60102470 msgr2=0x7f2f601028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:41.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.657+0000 7f2f64f2b700 1 --2- 172.21.15.70:0/3065368337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f60102470 0x7f2f601028f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009990 tx=0x7f2f50009ca0 comp rx=0 tx=0).stop 2024-06-04T04:56:41.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.657+0000 7f2f64f2b700 1 -- 172.21.15.70:0/3065368337 shutdown_connections 2024-06-04T04:56:41.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.657+0000 7f2f64f2b700 1 --2- 172.21.15.70:0/3065368337 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f60102e30 0x7f2f6010f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:41.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.657+0000 7f2f64f2b700 1 --2- 172.21.15.70:0/3065368337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f601042f0 0x7f2f601046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:41.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.657+0000 7f2f64f2b700 1 --2- 172.21.15.70:0/3065368337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f60102470 0x7f2f601028f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:41.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.657+0000 7f2f64f2b700 1 -- 172.21.15.70:0/3065368337 >> 172.21.15.70:0/3065368337 conn(0x7f2f600784b0 msgr2=0x7f2f600788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:41.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.657+0000 7f2f64f2b700 1 -- 172.21.15.70:0/3065368337 shutdown_connections 2024-06-04T04:56:41.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.657+0000 7f2f64f2b700 1 -- 172.21.15.70:0/3065368337 wait complete. 2024-06-04T04:56:41.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.658+0000 7f2f64f2b700 1 Processor -- start 2024-06-04T04:56:41.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.658+0000 7f2f64f2b700 1 -- start start 2024-06-04T04:56:41.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.658+0000 7f2f64f2b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f60102470 0x7f2f6019e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:41.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f64f2b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f60102e30 0x7f2f6019ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:41.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f64f2b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f601042f0 0x7f2f601989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:41.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f64f2b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2f601125f0 con 0x7f2f60102470 2024-06-04T04:56:41.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f64f2b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2f60112470 con 0x7f2f60102e30 2024-06-04T04:56:41.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f64f2b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2f60112770 con 0x7f2f601042f0 2024-06-04T04:56:41.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f5e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f60102470 0x7f2f6019e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:41.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f5e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f60102470 0x7f2f6019e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42306/0 (socket says 172.21.15.70:42306) 2024-06-04T04:56:41.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f5e59c700 1 -- 172.21.15.70:0/1269717671 learned_addr learned my addr 172.21.15.70:0/1269717671 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:41.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f5dd9b700 1 --2- 172.21.15.70:0/1269717671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f60102e30 0x7f2f6019ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:41.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f5ed9d700 1 --2- 172.21.15.70:0/1269717671 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f601042f0 0x7f2f601989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:41.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f5e59c700 1 -- 172.21.15.70:0/1269717671 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f601042f0 msgr2=0x7f2f601989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:41.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f5e59c700 1 --2- 172.21.15.70:0/1269717671 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f601042f0 0x7f2f601989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:41.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f5e59c700 1 -- 172.21.15.70:0/1269717671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f60102e30 msgr2=0x7f2f6019ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:41.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f5e59c700 1 --2- 172.21.15.70:0/1269717671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f60102e30 0x7f2f6019ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:41.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f5e59c700 1 -- 172.21.15.70:0/1269717671 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f60199150 con 0x7f2f60102470 2024-06-04T04:56:41.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f5dd9b700 1 --2- 172.21.15.70:0/1269717671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f60102e30 0x7f2f6019ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:41.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f5e59c700 1 --2- 172.21.15.70:0/1269717671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f60102470 0x7f2f6019e7e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2f4800c920 tx=0x7f2f4800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:41.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f577fe700 1 -- 172.21.15.70:0/1269717671 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f48007b00 con 0x7f2f60102470 2024-06-04T04:56:41.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f577fe700 1 -- 172.21.15.70:0/1269717671 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2f48004cc0 con 0x7f2f60102470 2024-06-04T04:56:41.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f64f2b700 1 -- 172.21.15.70:0/1269717671 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f60199440 con 0x7f2f60102470 2024-06-04T04:56:41.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f577fe700 1 -- 172.21.15.70:0/1269717671 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f480054f0 con 0x7f2f60102470 2024-06-04T04:56:41.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.659+0000 7f2f64f2b700 1 -- 172.21.15.70:0/1269717671 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f601a56b0 con 0x7f2f60102470 2024-06-04T04:56:41.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.660+0000 7f2f64f2b700 1 -- 172.21.15.70:0/1269717671 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f6006a230 con 0x7f2f60102470 2024-06-04T04:56:41.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.662+0000 7f2f577fe700 1 -- 172.21.15.70:0/1269717671 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f2f48007ca0 con 0x7f2f60102470 2024-06-04T04:56:41.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.662+0000 7f2f577fe700 1 --2- 172.21.15.70:0/1269717671 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2f4c0b2980 0x7f2f4c0b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:41.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.663+0000 7f2f5dd9b700 1 --2- 172.21.15.70:0/1269717671 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2f4c0b2980 0x7f2f4c0b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:41.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.663+0000 7f2f577fe700 1 -- 172.21.15.70:0/1269717671 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f2f480e2240 con 0x7f2f60102470 2024-06-04T04:56:41.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.663+0000 7f2f5dd9b700 1 --2- 172.21.15.70:0/1269717671 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2f4c0b2980 0x7f2f4c0b4e40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009750 tx=0x7f2f5001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:41.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.666+0000 7f2f577fe700 1 -- 172.21.15.70:0/1269717671 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2f48014030 con 0x7f2f60102470 2024-06-04T04:56:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:41 smithi098 ceph-mon[35569]: pgmap v382: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:56:41.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:41 smithi070 ceph-mon[32631]: pgmap v382: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:56:41.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.912+0000 7f2f64f2b700 1 -- 172.21.15.70:0/1269717671 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f6002e560 con 0x7f2f60102470 2024-06-04T04:56:41.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.913+0000 7f2f577fe700 1 -- 172.21.15.70:0/1269717671 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11929 (secure 0 0 0) 0x7f2f480a9f60 con 0x7f2f60102470 2024-06-04T04:56:41.915 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:56:41.915 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:53:56.441682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T04:56:41.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.915+0000 7f2f64f2b700 1 -- 172.21.15.70:0/1269717671 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2f4c0b2980 msgr2=0x7f2f4c0b4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:41.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.915+0000 7f2f64f2b700 1 --2- 172.21.15.70:0/1269717671 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2f4c0b2980 0x7f2f4c0b4e40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009750 tx=0x7f2f5001a040 comp rx=0 tx=0).stop 2024-06-04T04:56:41.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.915+0000 7f2f64f2b700 1 -- 172.21.15.70:0/1269717671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f60102470 msgr2=0x7f2f6019e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:41.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.915+0000 7f2f64f2b700 1 --2- 172.21.15.70:0/1269717671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f60102470 0x7f2f6019e7e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2f4800c920 tx=0x7f2f4800cce0 comp rx=0 tx=0).stop 2024-06-04T04:56:41.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.915+0000 7f2f64f2b700 1 -- 172.21.15.70:0/1269717671 shutdown_connections 2024-06-04T04:56:41.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.915+0000 7f2f64f2b700 1 --2- 172.21.15.70:0/1269717671 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f2f4c0b2980 0x7f2f4c0b4e40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:41.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.915+0000 7f2f64f2b700 1 --2- 172.21.15.70:0/1269717671 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f601042f0 0x7f2f601989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:41.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.915+0000 7f2f64f2b700 1 --2- 172.21.15.70:0/1269717671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f60102470 0x7f2f6019e7e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:41.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.915+0000 7f2f64f2b700 1 --2- 172.21.15.70:0/1269717671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f60102e30 0x7f2f6019ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:41.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.915+0000 7f2f64f2b700 1 -- 172.21.15.70:0/1269717671 >> 172.21.15.70:0/1269717671 conn(0x7f2f600784b0 msgr2=0x7f2f601073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:41.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.915+0000 7f2f64f2b700 1 -- 172.21.15.70:0/1269717671 shutdown_connections 2024-06-04T04:56:41.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:41.915+0000 7f2f64f2b700 1 -- 172.21.15.70:0/1269717671 wait complete. 2024-06-04T04:56:41.919 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 28 2024-06-04T04:56:41.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:41 smithi112 ceph-mon[36453]: pgmap v382: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:56:42.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:42 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1269717671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:56:42.692 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:56:42.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:42 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1269717671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:56:42.935 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:42.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:42 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1269717671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:56:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:43 smithi070 ceph-mon[32631]: pgmap v383: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:56:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:43 smithi098 ceph-mon[35569]: pgmap v383: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:56:43.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:43 smithi112 ceph-mon[36453]: pgmap v383: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:56:44.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.575+0000 7f980998a700 1 -- 172.21.15.70:0/1144304135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f980406bc10 msgr2=0x7f980406c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:44.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.575+0000 7f980998a700 1 --2- 172.21.15.70:0/1144304135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f980406bc10 0x7f980406c090 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009990 tx=0x7f97f4009ca0 comp rx=0 tx=0).stop 2024-06-04T04:56:44.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.575+0000 7f980998a700 1 -- 172.21.15.70:0/1144304135 shutdown_connections 2024-06-04T04:56:44.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.575+0000 7f980998a700 1 --2- 172.21.15.70:0/1144304135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980406c5d0 0x7f98041103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:44.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.575+0000 7f980998a700 1 --2- 172.21.15.70:0/1144304135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98041095c0 0x7f98041099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:44.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.575+0000 7f980998a700 1 --2- 172.21.15.70:0/1144304135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f980406bc10 0x7f980406c090 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:44.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.575+0000 7f980998a700 1 -- 172.21.15.70:0/1144304135 >> 172.21.15.70:0/1144304135 conn(0x7f98040793f0 msgr2=0x7f9804079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:44.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.575+0000 7f980998a700 1 -- 172.21.15.70:0/1144304135 shutdown_connections 2024-06-04T04:56:44.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.575+0000 7f980998a700 1 -- 172.21.15.70:0/1144304135 wait complete. 2024-06-04T04:56:44.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.575+0000 7f980998a700 1 Processor -- start 2024-06-04T04:56:44.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.575+0000 7f980998a700 1 -- start start 2024-06-04T04:56:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f980998a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f980406bc10 0x7f98041a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f980998a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980406c5d0 0x7f98041a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f980998a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98041095c0 0x7f980419bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f980998a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f98041135a0 con 0x7f980406bc10 2024-06-04T04:56:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f980998a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9804113420 con 0x7f98041095c0 2024-06-04T04:56:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f980998a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9804113720 con 0x7f980406c5d0 2024-06-04T04:56:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f9802ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f980406bc10 0x7f98041a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f9802ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f980406bc10 0x7f98041a1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42328/0 (socket says 172.21.15.70:42328) 2024-06-04T04:56:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f9802ffd700 1 -- 172.21.15.70:0/2372052788 learned_addr learned my addr 172.21.15.70:0/2372052788 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f98037fe700 1 --2- 172.21.15.70:0/2372052788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98041095c0 0x7f980419bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f98027fc700 1 --2- 172.21.15.70:0/2372052788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980406c5d0 0x7f98041a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f9802ffd700 1 -- 172.21.15.70:0/2372052788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980406c5d0 msgr2=0x7f98041a1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f9802ffd700 1 --2- 172.21.15.70:0/2372052788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980406c5d0 0x7f98041a1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f9802ffd700 1 -- 172.21.15.70:0/2372052788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98041095c0 msgr2=0x7f980419bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f9802ffd700 1 --2- 172.21.15.70:0/2372052788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98041095c0 0x7f980419bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f9802ffd700 1 -- 172.21.15.70:0/2372052788 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f980419c260 con 0x7f980406bc10 2024-06-04T04:56:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f9802ffd700 1 --2- 172.21.15.70:0/2372052788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f980406bc10 0x7f98041a1940 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00c9b0 tx=0x7f97ec00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f9808988700 1 -- 172.21.15.70:0/2372052788 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ec007b00 con 0x7f980406bc10 2024-06-04T04:56:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f980998a700 1 -- 172.21.15.70:0/2372052788 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f980419c550 con 0x7f980406bc10 2024-06-04T04:56:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f9808988700 1 -- 172.21.15.70:0/2372052788 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97ec007ca0 con 0x7f980406bc10 2024-06-04T04:56:44.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.576+0000 7f9808988700 1 -- 172.21.15.70:0/2372052788 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ec018660 con 0x7f980406bc10 2024-06-04T04:56:44.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.577+0000 7f980998a700 1 -- 172.21.15.70:0/2372052788 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98041a8810 con 0x7f980406bc10 2024-06-04T04:56:44.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.577+0000 7f980998a700 1 -- 172.21.15.70:0/2372052788 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9804104c90 con 0x7f980406bc10 2024-06-04T04:56:44.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.579+0000 7f9808988700 1 -- 172.21.15.70:0/2372052788 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f97ec00f450 con 0x7f980406bc10 2024-06-04T04:56:44.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.579+0000 7f9808988700 1 --2- 172.21.15.70:0/2372052788 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f97f00b2880 0x7f97f00b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:44.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.579+0000 7f98027fc700 1 --2- 172.21.15.70:0/2372052788 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f97f00b2880 0x7f97f00b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:44.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.580+0000 7f9808988700 1 -- 172.21.15.70:0/2372052788 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f97ec0e2420 con 0x7f980406bc10 2024-06-04T04:56:44.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.580+0000 7f98027fc700 1 --2- 172.21.15.70:0/2372052788 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f97f00b2880 0x7f97f00b4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f97f4005cd0 tx=0x7f97f4005c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:44.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.583+0000 7f9808988700 1 -- 172.21.15.70:0/2372052788 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f97ec014030 con 0x7f980406bc10 2024-06-04T04:56:44.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.833+0000 7f980998a700 1 -- 172.21.15.70:0/2372052788 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f980406a230 con 0x7f980406bc10 2024-06-04T04:56:44.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.834+0000 7f9808988700 1 -- 172.21.15.70:0/2372052788 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11929 (secure 0 0 0) 0x7f97ec0aa1b0 con 0x7f980406bc10 2024-06-04T04:56:44.836 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:56:44.837 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:53:56.441682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T04:56:44.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.836+0000 7f980998a700 1 -- 172.21.15.70:0/2372052788 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f97f00b2880 msgr2=0x7f97f00b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:44.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.836+0000 7f980998a700 1 --2- 172.21.15.70:0/2372052788 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f97f00b2880 0x7f97f00b4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f97f4005cd0 tx=0x7f97f4005c40 comp rx=0 tx=0).stop 2024-06-04T04:56:44.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.836+0000 7f980998a700 1 -- 172.21.15.70:0/2372052788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f980406bc10 msgr2=0x7f98041a1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:44.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.836+0000 7f980998a700 1 --2- 172.21.15.70:0/2372052788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f980406bc10 0x7f98041a1940 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00c9b0 tx=0x7f97ec00ccc0 comp rx=0 tx=0).stop 2024-06-04T04:56:44.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.836+0000 7f980998a700 1 -- 172.21.15.70:0/2372052788 shutdown_connections 2024-06-04T04:56:44.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.836+0000 7f980998a700 1 --2- 172.21.15.70:0/2372052788 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f97f00b2880 0x7f97f00b4d40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:44.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.836+0000 7f980998a700 1 --2- 172.21.15.70:0/2372052788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98041095c0 0x7f980419bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:44.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.836+0000 7f980998a700 1 --2- 172.21.15.70:0/2372052788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f980406bc10 0x7f98041a1940 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:44.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.836+0000 7f980998a700 1 --2- 172.21.15.70:0/2372052788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980406c5d0 0x7f98041a1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:44.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.836+0000 7f980998a700 1 -- 172.21.15.70:0/2372052788 >> 172.21.15.70:0/2372052788 conn(0x7f98040793f0 msgr2=0x7f980410e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:44.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.837+0000 7f980998a700 1 -- 172.21.15.70:0/2372052788 shutdown_connections 2024-06-04T04:56:44.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:44.837+0000 7f980998a700 1 -- 172.21.15.70:0/2372052788 wait complete. 2024-06-04T04:56:44.840 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 28 2024-06-04T04:56:45.591 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-04T04:56:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:45 smithi098 ceph-mon[35569]: pgmap v384: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:56:45.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:45 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2372052788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:56:45.831 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:45.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:45 smithi070 ceph-mon[32631]: pgmap v384: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:56:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:45 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2372052788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:56:45.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:45 smithi112 ceph-mon[36453]: pgmap v384: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:56:45.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:45 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2372052788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:56:46.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:46 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:56:46.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:46 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:56:46.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:46 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:56:47.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.503+0000 7f0e2d95a700 1 -- 172.21.15.70:0/4252208042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e28103850 msgr2=0x7f0e28103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:47.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.503+0000 7f0e2d95a700 1 --2- 172.21.15.70:0/4252208042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e28103850 0x7f0e28103c30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0e180099c0 tx=0x7f0e18009cd0 comp rx=0 tx=0).stop 2024-06-04T04:56:47.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.503+0000 7f0e2d95a700 1 -- 172.21.15.70:0/4252208042 shutdown_connections 2024-06-04T04:56:47.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.503+0000 7f0e2d95a700 1 --2- 172.21.15.70:0/4252208042 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e28108920 0x7f0e281104a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:47.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.503+0000 7f0e2d95a700 1 --2- 172.21.15.70:0/4252208042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e28103850 0x7f0e28103c30 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:47.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.503+0000 7f0e2d95a700 1 --2- 172.21.15.70:0/4252208042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e28104200 0x7f0e281081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:47.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.503+0000 7f0e2d95a700 1 -- 172.21.15.70:0/4252208042 >> 172.21.15.70:0/4252208042 conn(0x7f0e2807a5d0 msgr2=0x7f0e2807a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:47.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.503+0000 7f0e2d95a700 1 -- 172.21.15.70:0/4252208042 shutdown_connections 2024-06-04T04:56:47.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.503+0000 7f0e2d95a700 1 -- 172.21.15.70:0/4252208042 wait complete. 2024-06-04T04:56:47.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.504+0000 7f0e2d95a700 1 Processor -- start 2024-06-04T04:56:47.508 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:47 smithi070 ceph-mon[32631]: pgmap v385: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:47.508 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:56:47.508 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:56:47.508 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:56:47.508 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:47 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:56:47.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.504+0000 7f0e2d95a700 1 -- start start 2024-06-04T04:56:47.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.505+0000 7f0e2d95a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e28103850 0x7f0e281a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:47.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.505+0000 7f0e2d95a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e28104200 0x7f0e281a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:47.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.505+0000 7f0e2d95a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e28108920 0x7f0e2819bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:47.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.505+0000 7f0e267fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e28104200 0x7f0e281a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:47.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.505+0000 7f0e267fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e28104200 0x7f0e281a1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:57736/0 (socket says 172.21.15.70:57736) 2024-06-04T04:56:47.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.505+0000 7f0e277fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e28108920 0x7f0e2819bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:47.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.505+0000 7f0e2d95a700 1 -- 172.21.15.70:0/701119336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0e281135b0 con 0x7f0e28108920 2024-06-04T04:56:47.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.505+0000 7f0e267fc700 1 -- 172.21.15.70:0/701119336 learned_addr learned my addr 172.21.15.70:0/701119336 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:47.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.505+0000 7f0e2d95a700 1 -- 172.21.15.70:0/701119336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0e28113430 con 0x7f0e28104200 2024-06-04T04:56:47.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.505+0000 7f0e2d95a700 1 -- 172.21.15.70:0/701119336 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0e28113730 con 0x7f0e28103850 2024-06-04T04:56:47.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.505+0000 7f0e26ffd700 1 --2- 172.21.15.70:0/701119336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e28103850 0x7f0e281a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:47.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.505+0000 7f0e267fc700 1 -- 172.21.15.70:0/701119336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e28103850 msgr2=0x7f0e281a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:47.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.505+0000 7f0e267fc700 1 --2- 172.21.15.70:0/701119336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e28103850 0x7f0e281a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:47.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.505+0000 7f0e267fc700 1 -- 172.21.15.70:0/701119336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e28108920 msgr2=0x7f0e2819bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:47.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.505+0000 7f0e267fc700 1 --2- 172.21.15.70:0/701119336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e28108920 0x7f0e2819bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:47.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.505+0000 7f0e267fc700 1 -- 172.21.15.70:0/701119336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e2819c260 con 0x7f0e28104200 2024-06-04T04:56:47.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.505+0000 7f0e277fe700 1 --2- 172.21.15.70:0/701119336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e28108920 0x7f0e2819bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:56:47.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.506+0000 7f0e267fc700 1 --2- 172.21.15.70:0/701119336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e28104200 0x7f0e281a1e30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00c9b0 tx=0x7f0e1c00ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:47.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.506+0000 7f0e2c958700 1 -- 172.21.15.70:0/701119336 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c007b00 con 0x7f0e28104200 2024-06-04T04:56:47.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.506+0000 7f0e2c958700 1 -- 172.21.15.70:0/701119336 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0e1c00ce50 con 0x7f0e28104200 2024-06-04T04:56:47.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.506+0000 7f0e2d95a700 1 -- 172.21.15.70:0/701119336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e2819c550 con 0x7f0e28104200 2024-06-04T04:56:47.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.506+0000 7f0e2c958700 1 -- 172.21.15.70:0/701119336 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c018660 con 0x7f0e28104200 2024-06-04T04:56:47.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.506+0000 7f0e2d95a700 1 -- 172.21.15.70:0/701119336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e281a87c0 con 0x7f0e28104200 2024-06-04T04:56:47.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.507+0000 7f0e2d95a700 1 -- 172.21.15.70:0/701119336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e2806a230 con 0x7f0e28104200 2024-06-04T04:56:47.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.508+0000 7f0e2c958700 1 -- 172.21.15.70:0/701119336 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f0e1c004830 con 0x7f0e28104200 2024-06-04T04:56:47.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.509+0000 7f0e2c958700 1 --2- 172.21.15.70:0/701119336 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0e100b2900 0x7f0e100b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:47.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.509+0000 7f0e26ffd700 1 --2- 172.21.15.70:0/701119336 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0e100b2900 0x7f0e100b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:47.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.509+0000 7f0e2c958700 1 -- 172.21.15.70:0/701119336 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f0e1c0e2170 con 0x7f0e28104200 2024-06-04T04:56:47.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.510+0000 7f0e26ffd700 1 --2- 172.21.15.70:0/701119336 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0e100b2900 0x7f0e100b4dc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0e18009990 tx=0x7f0e18005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:47.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.514+0000 7f0e2c958700 1 -- 172.21.15.70:0/701119336 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0e1c014030 con 0x7f0e28104200 2024-06-04T04:56:47.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.727+0000 7f0e2d95a700 1 -- 172.21.15.70:0/701119336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0e281a8a50 con 0x7f0e28104200 2024-06-04T04:56:47.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.728+0000 7f0e2c958700 1 -- 172.21.15.70:0/701119336 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22708 (secure 0 0 0) 0x7f0e1c0a9e70 con 0x7f0e28104200 2024-06-04T04:56:47.731 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:56:47.731 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":81,"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","created":"2024-06-04T04:45:22.297536+0000","modified":"2024-06-04T04:54:19.502410+0000","last_up_change":"2024-06-04T04:52:34.388708+0000","last_in_change":"2024-06-04T04:52:14.197780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-04T04:48:43.724356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-04T04:53:31.398455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-04T04:53:35.298617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ff491ea2-1906-48e4-883e-bc78e778260e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6803","nonce":824119100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6805","nonce":824119100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6809","nonce":824119100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6807","nonce":824119100}]},"public_addr":"172.21.15.70:6803/824119100","cluster_addr":"172.21.15.70:6805/824119100","heartbeat_back_addr":"172.21.15.70:6809/824119100","heartbeat_front_addr":"172.21.15.70:6807/824119100","state":["exists","up"]},{"osd":1,"uuid":"bd5e4806-535b-49b5-ba6b-93228da35a84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6801","nonce":2744598742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6803","nonce":2744598742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6807","nonce":2744598742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6805","nonce":2744598742}]},"public_addr":"172.21.15.98:6801/2744598742","cluster_addr":"172.21.15.98:6803/2744598742","heartbeat_back_addr":"172.21.15.98:6807/2744598742","heartbeat_front_addr":"172.21.15.98:6805/2744598742","state":["exists","up"]},{"osd":2,"uuid":"8456fb55-910a-4a59-b84b-2c0f68346ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6801","nonce":1328098370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6803","nonce":1328098370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6807","nonce":1328098370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6805","nonce":1328098370}]},"public_addr":"172.21.15.112:6801/1328098370","cluster_addr":"172.21.15.112:6803/1328098370","heartbeat_back_addr":"172.21.15.112:6807/1328098370","heartbeat_front_addr":"172.21.15.112:6805/1328098370","state":["exists","up"]},{"osd":3,"uuid":"c8fbc440-bf70-42d3-b01e-777ccfa7900b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6811","nonce":4041566346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6813","nonce":4041566346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6817","nonce":4041566346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6815","nonce":4041566346}]},"public_addr":"172.21.15.70:6811/4041566346","cluster_addr":"172.21.15.70:6813/4041566346","heartbeat_back_addr":"172.21.15.70:6817/4041566346","heartbeat_front_addr":"172.21.15.70:6815/4041566346","state":["exists","up"]},{"osd":4,"uuid":"549c8c70-8be8-46ae-bcb6-0fde32b40cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6809","nonce":3652775814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6811","nonce":3652775814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6815","nonce":3652775814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6813","nonce":3652775814}]},"public_addr":"172.21.15.98:6809/3652775814","cluster_addr":"172.21.15.98:6811/3652775814","heartbeat_back_addr":"172.21.15.98:6815/3652775814","heartbeat_front_addr":"172.21.15.98:6813/3652775814","state":["exists","up"]},{"osd":5,"uuid":"51d86af1-0d6c-4ae0-b090-00414fc1bf7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6809","nonce":1857257899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6811","nonce":1857257899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6815","nonce":1857257899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6813","nonce":1857257899}]},"public_addr":"172.21.15.112:6809/1857257899","cluster_addr":"172.21.15.112:6811/1857257899","heartbeat_back_addr":"172.21.15.112:6815/1857257899","heartbeat_front_addr":"172.21.15.112:6813/1857257899","state":["exists","up"]},{"osd":6,"uuid":"526a8138-bfa8-419e-a5b5-e6b80f0ae004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6819","nonce":1264676496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6821","nonce":1264676496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6825","nonce":1264676496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6823","nonce":1264676496}]},"public_addr":"172.21.15.70:6819/1264676496","cluster_addr":"172.21.15.70:6821/1264676496","heartbeat_back_addr":"172.21.15.70:6825/1264676496","heartbeat_front_addr":"172.21.15.70:6823/1264676496","state":["exists","up"]},{"osd":7,"uuid":"1173f785-6ed1-47ef-9265-2e233ea9ee02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6817","nonce":709593020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6819","nonce":709593020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6823","nonce":709593020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6821","nonce":709593020}]},"public_addr":"172.21.15.98:6817/709593020","cluster_addr":"172.21.15.98:6819/709593020","heartbeat_back_addr":"172.21.15.98:6823/709593020","heartbeat_front_addr":"172.21.15.98:6821/709593020","state":["exists","up"]},{"osd":8,"uuid":"6424cc6e-24a7-4b81-9ce6-39203fecc7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6817","nonce":2250225413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6819","nonce":2250225413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6823","nonce":2250225413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6821","nonce":2250225413}]},"public_addr":"172.21.15.112:6817/2250225413","cluster_addr":"172.21.15.112:6819/2250225413","heartbeat_back_addr":"172.21.15.112:6823/2250225413","heartbeat_front_addr":"172.21.15.112:6821/2250225413","state":["exists","up"]},{"osd":9,"uuid":"8d666db8-8831-4d63-9e3c-a4bf2504149e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6827","nonce":3557688707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6829","nonce":3557688707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6833","nonce":3557688707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6831","nonce":3557688707}]},"public_addr":"172.21.15.70:6827/3557688707","cluster_addr":"172.21.15.70:6829/3557688707","heartbeat_back_addr":"172.21.15.70:6833/3557688707","heartbeat_front_addr":"172.21.15.70:6831/3557688707","state":["exists","up"]},{"osd":10,"uuid":"ecb925d2-0263-4dfc-a2fd-9e27856022d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6825","nonce":1896493718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6827","nonce":1896493718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6831","nonce":1896493718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6829","nonce":1896493718}]},"public_addr":"172.21.15.98:6825/1896493718","cluster_addr":"172.21.15.98:6827/1896493718","heartbeat_back_addr":"172.21.15.98:6831/1896493718","heartbeat_front_addr":"172.21.15.98:6829/1896493718","state":["exists","up"]},{"osd":11,"uuid":"c8e3230c-bc3a-40bd-8dd0-5fef56e4df97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6825","nonce":1400793137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6827","nonce":1400793137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6831","nonce":1400793137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6829","nonce":1400793137}]},"public_addr":"172.21.15.112:6825/1400793137","cluster_addr":"172.21.15.112:6827/1400793137","heartbeat_back_addr":"172.21.15.112:6831/1400793137","heartbeat_front_addr":"172.21.15.112:6829/1400793137","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:47:53.626608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:17.784505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:41.971969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:07.130699+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:31.615496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:56.898607+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:22.739991+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:48.319727+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:13.902063+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:40.389470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:06.142092+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:32.949383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/3442074771":"2024-06-05T04:46:17.636907+0000","172.21.15.70:6801/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/1442095493":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/2453733268":"2024-06-05T04:45:49.501864+0000","172.21.15.70:0/1369781447":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6801/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/915186456":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/860033006":"2024-06-05T04:45:49.501864+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-04T04:56:47.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.730+0000 7f0e2d95a700 1 -- 172.21.15.70:0/701119336 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0e100b2900 msgr2=0x7f0e100b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:47.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.730+0000 7f0e2d95a700 1 --2- 172.21.15.70:0/701119336 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0e100b2900 0x7f0e100b4dc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0e18009990 tx=0x7f0e18005fb0 comp rx=0 tx=0).stop 2024-06-04T04:56:47.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.730+0000 7f0e2d95a700 1 -- 172.21.15.70:0/701119336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e28104200 msgr2=0x7f0e281a1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:47.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.731+0000 7f0e2d95a700 1 --2- 172.21.15.70:0/701119336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e28104200 0x7f0e281a1e30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00c9b0 tx=0x7f0e1c00ccc0 comp rx=0 tx=0).stop 2024-06-04T04:56:47.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.731+0000 7f0e2d95a700 1 -- 172.21.15.70:0/701119336 shutdown_connections 2024-06-04T04:56:47.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.731+0000 7f0e2d95a700 1 --2- 172.21.15.70:0/701119336 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f0e100b2900 0x7f0e100b4dc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:47.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.731+0000 7f0e2d95a700 1 --2- 172.21.15.70:0/701119336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e28108920 0x7f0e2819bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:47.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.731+0000 7f0e2d95a700 1 --2- 172.21.15.70:0/701119336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e28103850 0x7f0e281a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:47.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.731+0000 7f0e2d95a700 1 --2- 172.21.15.70:0/701119336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e28104200 0x7f0e281a1e30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:47.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.731+0000 7f0e2d95a700 1 -- 172.21.15.70:0/701119336 >> 172.21.15.70:0/701119336 conn(0x7f0e2807a5d0 msgr2=0x7f0e28101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:47.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.731+0000 7f0e2d95a700 1 -- 172.21.15.70:0/701119336 shutdown_connections 2024-06-04T04:56:47.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:47.731+0000 7f0e2d95a700 1 -- 172.21.15.70:0/701119336 wait complete. 2024-06-04T04:56:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:47 smithi098 ceph-mon[35569]: pgmap v385: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:56:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:56:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:56:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:47 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:56:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:47 smithi112 ceph-mon[36453]: pgmap v385: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:56:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:56:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:56:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:47 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:56:48.441 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-06-04T04:53:38.237993+0000', 'modified': '2024-06-04T04:53:56.441682+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24448, 'mds_1': 24442, 'mds_2': 24440, 'mds_3': 24436, 'mds_4': 24410}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14580': {'gid': 14580, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6839/3817259285', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6838', 'nonce': 3817259285}, {'type': 'v1', 'addr': '172.21.15.70:6839', 'nonce': 3817259285}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14592': {'gid': 14592, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6841/2748767095', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6840', 'nonce': 2748767095}, {'type': 'v1', 'addr': '172.21.15.70:6841', 'nonce': 2748767095}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24371': {'gid': 24371, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6833/1896647585', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6832', 'nonce': 1896647585}, {'type': 'v1', 'addr': '172.21.15.98:6833', 'nonce': 1896647585}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24386': {'gid': 24386, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6835/3148471566', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 3148471566}, {'type': 'v1', 'addr': '172.21.15.98:6835', 'nonce': 3148471566}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24400': {'gid': 24400, 'name': 'l', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6833/1359739019', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6832', 'nonce': 1359739019}, {'type': 'v1', 'addr': '172.21.15.112:6833', 'nonce': 1359739019}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24410': {'gid': 24410, 'name': 'h', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.98:6837/3160044154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6836', 'nonce': 3160044154}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3160044154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24436': {'gid': 24436, 'name': 'f', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6835/242926435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6834', 'nonce': 242926435}, {'type': 'v1', 'addr': '172.21.15.112:6835', 'nonce': 242926435}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'b', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.98:6839/1361276220', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6838', 'nonce': 1361276220}, {'type': 'v1', 'addr': '172.21.15.98:6839', 'nonce': 1361276220}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'c', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.112:6837/3596492666', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6836', 'nonce': 3596492666}, {'type': 'v1', 'addr': '172.21.15.112:6837', 'nonce': 3596492666}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24448': {'gid': 24448, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.112:6839/2996116087', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6838', 'nonce': 2996116087}, {'type': 'v1', 'addr': '172.21.15.112:6839', 'nonce': 2996116087}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24448, 'qdb_cluster': [24410, 24436, 24440, 24442, 24448]} 2024-06-04T04:56:48.442 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-06-04T04:56:48.442 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.h status 2024-06-04T04:56:48.684 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:48.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:48 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:56:48.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:48 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:56:48.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:48 smithi098 ceph-mon[35569]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:56:48.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:48 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/701119336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:56:48.806 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:48 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:56:48.806 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:48 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:56:48.806 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:48 smithi070 ceph-mon[32631]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:56:48.806 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:48 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/701119336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:56:48.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:48 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:56:48.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:48 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:56:48.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:48 smithi112 ceph-mon[36453]: from='mgr.14152 172.21.15.70:0/918034828' entity='mgr.x' 2024-06-04T04:56:48.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:48 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/701119336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:56:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:49 smithi098 ceph-mon[35569]: pgmap v386: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:56:49.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:49 smithi070 ceph-mon[32631]: pgmap v386: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:56:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:49 smithi112 ceph-mon[36453]: pgmap v386: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:56:50.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.349+0000 7fe66bdfe700 1 -- 172.21.15.70:0/294717102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe664108520 msgr2=0x7fe664108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:50.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.349+0000 7fe66bdfe700 1 --2- 172.21.15.70:0/294717102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe664108520 0x7fe664108900 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe66000b0a0 tx=0x7fe66000b3b0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.349+0000 7fe66bdfe700 1 -- 172.21.15.70:0/294717102 shutdown_connections 2024-06-04T04:56:50.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.349+0000 7fe66bdfe700 1 --2- 172.21.15.70:0/294717102 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe664102e00 0x7fe66410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.349+0000 7fe66bdfe700 1 --2- 172.21.15.70:0/294717102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe664108520 0x7fe664108900 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.349+0000 7fe66bdfe700 1 --2- 172.21.15.70:0/294717102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe664102440 0x7fe6641028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.349+0000 7fe66bdfe700 1 -- 172.21.15.70:0/294717102 >> 172.21.15.70:0/294717102 conn(0x7fe6640793f0 msgr2=0x7fe664079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:50.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.349+0000 7fe66bdfe700 1 -- 172.21.15.70:0/294717102 shutdown_connections 2024-06-04T04:56:50.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.349+0000 7fe66bdfe700 1 -- 172.21.15.70:0/294717102 wait complete. 2024-06-04T04:56:50.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.350+0000 7fe66bdfe700 1 Processor -- start 2024-06-04T04:56:50.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.350+0000 7fe66bdfe700 1 -- start start 2024-06-04T04:56:50.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.350+0000 7fe66bdfe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe664102440 0x7fe664104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.350+0000 7fe66bdfe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe664102e00 0x7fe6641045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.350+0000 7fe66bdfe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe664108520 0x7fe664104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.350+0000 7fe66bdfe700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe6641124a0 con 0x7fe664102e00 2024-06-04T04:56:50.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.350+0000 7fe66bdfe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe664112320 con 0x7fe664108520 2024-06-04T04:56:50.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.350+0000 7fe66bdfe700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe664112620 con 0x7fe664102440 2024-06-04T04:56:50.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.350+0000 7fe669399700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe664102e00 0x7fe6641045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:50.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.350+0000 7fe669399700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe664102e00 0x7fe6641045c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44526/0 (socket says 172.21.15.70:44526) 2024-06-04T04:56:50.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.350+0000 7fe669399700 1 -- 172.21.15.70:0/2747629183 learned_addr learned my addr 172.21.15.70:0/2747629183 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:50.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.350+0000 7fe669b9a700 1 --2- 172.21.15.70:0/2747629183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe664102440 0x7fe664104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:50.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.351+0000 7fe669399700 1 -- 172.21.15.70:0/2747629183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe664102440 msgr2=0x7fe664104080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:50.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.351+0000 7fe669399700 1 --2- 172.21.15.70:0/2747629183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe664102440 0x7fe664104080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.351+0000 7fe669399700 1 -- 172.21.15.70:0/2747629183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe664108520 msgr2=0x7fe664104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:50.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.351+0000 7fe669399700 1 --2- 172.21.15.70:0/2747629183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe664108520 0x7fe664104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.351+0000 7fe669399700 1 -- 172.21.15.70:0/2747629183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6641a7fc0 con 0x7fe664102e00 2024-06-04T04:56:50.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.351+0000 7fe669399700 1 --2- 172.21.15.70:0/2747629183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe664102e00 0x7fe6641045c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe65400c9b0 tx=0x7fe65400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:50.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.351+0000 7fe65affd700 1 -- 172.21.15.70:0/2747629183 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe654007b00 con 0x7fe664102e00 2024-06-04T04:56:50.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.351+0000 7fe66bdfe700 1 -- 172.21.15.70:0/2747629183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6641a8250 con 0x7fe664102e00 2024-06-04T04:56:50.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.351+0000 7fe65affd700 1 -- 172.21.15.70:0/2747629183 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe654007ca0 con 0x7fe664102e00 2024-06-04T04:56:50.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.351+0000 7fe65affd700 1 -- 172.21.15.70:0/2747629183 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe654018660 con 0x7fe664102e00 2024-06-04T04:56:50.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.351+0000 7fe66bdfe700 1 -- 172.21.15.70:0/2747629183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6641a87e0 con 0x7fe664102e00 2024-06-04T04:56:50.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.353+0000 7fe65affd700 1 -- 172.21.15.70:0/2747629183 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fe65400f450 con 0x7fe664102e00 2024-06-04T04:56:50.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.354+0000 7fe65affd700 1 --2- 172.21.15.70:0/2747629183 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe6500b2880 0x7fe6500b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.354+0000 7fe669b9a700 1 --2- 172.21.15.70:0/2747629183 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe6500b2880 0x7fe6500b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:50.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.354+0000 7fe65affd700 1 -- 172.21.15.70:0/2747629183 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fe6540e2210 con 0x7fe664102e00 2024-06-04T04:56:50.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.354+0000 7fe669b9a700 1 --2- 172.21.15.70:0/2747629183 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe6500b2880 0x7fe6500b4d40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe660009e60 tx=0x7fe66000b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:50.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe676693b80 1 Processor -- start 2024-06-04T04:56:50.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe676693b80 1 -- start start 2024-06-04T04:56:50.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe676693b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87ee2d0 0x5635d87ee6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe676693b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87e8850 0x5635d87e8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe676693b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87e9210 0x5635d87f5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe676693b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5635d8746d20 con 0x5635d87ee2d0 2024-06-04T04:56:50.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe676693b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5635d87368c0 con 0x5635d87e8850 2024-06-04T04:56:50.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe676693b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5635d873a900 con 0x5635d87e9210 2024-06-04T04:56:50.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe669b9a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87ee2d0 0x5635d87ee6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:50.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe669b9a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87ee2d0 0x5635d87ee6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44532/0 (socket says 172.21.15.70:44532) 2024-06-04T04:56:50.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe669b9a700 1 -- 172.21.15.70:0/1329994926 learned_addr learned my addr 172.21.15.70:0/1329994926 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:50.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe669399700 1 --2- 172.21.15.70:0/1329994926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87e8850 0x5635d87e8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:50.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe66a39b700 1 --2- 172.21.15.70:0/1329994926 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87e9210 0x5635d87f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:50.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe669b9a700 1 -- 172.21.15.70:0/1329994926 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87e9210 msgr2=0x5635d87f5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:50.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe669b9a700 1 --2- 172.21.15.70:0/1329994926 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87e9210 0x5635d87f5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe669b9a700 1 -- 172.21.15.70:0/1329994926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87e8850 msgr2=0x5635d87e8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:50.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe669b9a700 1 --2- 172.21.15.70:0/1329994926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87e8850 0x5635d87e8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe669b9a700 1 -- 172.21.15.70:0/1329994926 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635d86dff60 con 0x5635d87ee2d0 2024-06-04T04:56:50.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.355+0000 7fe66a39b700 1 --2- 172.21.15.70:0/1329994926 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87e9210 0x5635d87f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:50.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.356+0000 7fe669b9a700 1 --2- 172.21.15.70:0/1329994926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87ee2d0 0x5635d87ee6b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe660015000 tx=0x7fe66001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2490420263af970 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:50.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.356+0000 7fe64ffff700 1 -- 172.21.15.70:0/1329994926 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66002c030 con 0x5635d87ee2d0 2024-06-04T04:56:50.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.356+0000 7fe64ffff700 1 -- 172.21.15.70:0/1329994926 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe6600079f0 con 0x5635d87ee2d0 2024-06-04T04:56:50.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.356+0000 7fe64ffff700 1 -- 172.21.15.70:0/1329994926 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe660028b10 con 0x5635d87ee2d0 2024-06-04T04:56:50.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.356+0000 7fe676693b80 1 -- 172.21.15.70:0/1329994926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87ee2d0 msgr2=0x5635d87ee6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:50.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.356+0000 7fe676693b80 1 --2- 172.21.15.70:0/1329994926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87ee2d0 0x5635d87ee6b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe660015000 tx=0x7fe66001d000 comp rx=0 tx=0).stop 2024-06-04T04:56:50.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.357+0000 7fe676693b80 1 -- 172.21.15.70:0/1329994926 shutdown_connections 2024-06-04T04:56:50.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.357+0000 7fe676693b80 1 --2- 172.21.15.70:0/1329994926 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87e9210 0x5635d87f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.357+0000 7fe676693b80 1 --2- 172.21.15.70:0/1329994926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87ee2d0 0x5635d87ee6b0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.357+0000 7fe676693b80 1 --2- 172.21.15.70:0/1329994926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87e8850 0x5635d87e8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.357+0000 7fe676693b80 1 -- 172.21.15.70:0/1329994926 >> 172.21.15.70:0/1329994926 conn(0x5635d87141f0 msgr2=0x5635d87e3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:50.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.357+0000 7fe676693b80 1 -- 172.21.15.70:0/1329994926 shutdown_connections 2024-06-04T04:56:50.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.357+0000 7fe676693b80 1 -- 172.21.15.70:0/1329994926 wait complete. 2024-06-04T04:56:50.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.357+0000 7fe676693b80 1 -- start start 2024-06-04T04:56:50.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.357+0000 7fe676693b80 1 Processor -- start 2024-06-04T04:56:50.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.357+0000 7fe676693b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87e8850 0x5635d87f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.357+0000 7fe676693b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87e9210 0x5635d87f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe676693b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87ee2d0 0x5635d87f2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe676693b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5635d863cf50 con 0x5635d87e8850 2024-06-04T04:56:50.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe676693b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5635d869ed50 con 0x5635d87ee2d0 2024-06-04T04:56:50.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe676693b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5635d87f5630 con 0x5635d87e9210 2024-06-04T04:56:50.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe676693b80 10 client.0 _ll_register_callbacks cb 0x5635d86d1900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:50.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe669b9a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87e8850 0x5635d87f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:50.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe669b9a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87e8850 0x5635d87f4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44546/0 (socket says 172.21.15.70:44546) 2024-06-04T04:56:50.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe669b9a700 1 -- 172.21.15.70:0/1367480712 learned_addr learned my addr 172.21.15.70:0/1367480712 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:50.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe66a39b700 1 --2- 172.21.15.70:0/1367480712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87ee2d0 0x5635d87f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:50.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe669399700 1 --2- 172.21.15.70:0/1367480712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87e9210 0x5635d87f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:50.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe669b9a700 1 -- 172.21.15.70:0/1367480712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87e9210 msgr2=0x5635d87f1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:50.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe669b9a700 1 --2- 172.21.15.70:0/1367480712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87e9210 0x5635d87f1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe669b9a700 1 -- 172.21.15.70:0/1367480712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87ee2d0 msgr2=0x5635d87f2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:50.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe669b9a700 1 --2- 172.21.15.70:0/1367480712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87ee2d0 0x5635d87f2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe669b9a700 1 -- 172.21.15.70:0/1367480712 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635d8686880 con 0x5635d87e8850 2024-06-04T04:56:50.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe669b9a700 1 --2- 172.21.15.70:0/1367480712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87e8850 0x5635d87f4bf0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe660007790 tx=0x7fe6600077c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97bfaa92f1b3b3b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:50.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe64d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:50.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe64d7fa700 1 -- 172.21.15.70:0/1367480712 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66002c050 con 0x5635d87e8850 2024-06-04T04:56:50.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe676693b80 1 -- 172.21.15.70:0/1367480712 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6600079f0 con 0x5635d87e8850 2024-06-04T04:56:50.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.358+0000 7fe64d7fa700 1 -- 172.21.15.70:0/1367480712 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fe660028ba0 con 0x5635d87e8850 2024-06-04T04:56:50.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.359+0000 7fe64d7fa700 1 -- 172.21.15.70:0/1367480712 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe660028da0 con 0x5635d87e8850 2024-06-04T04:56:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.359+0000 7fe676693b80 10 client.15117 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.359+0000 7fe676693b80 1 -- 172.21.15.70:0/1367480712 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5635d86f0710 con 0x5635d87e8850 2024-06-04T04:56:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.359+0000 7fe64d7fa700 1 -- 172.21.15.70:0/1367480712 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe6600079f0 con 0x5635d87e8850 2024-06-04T04:56:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.359+0000 7fe64d7fa700 1 -- 172.21.15.70:0/1367480712 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fe660011070 con 0x5635d87e8850 2024-06-04T04:56:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.359+0000 7fe676693b80 10 client.15117 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.359+0000 7fe676693b80 10 client.15117 resolve_mds: resolved name 'h' to daemon mds.h 2024-06-04T04:56:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.359+0000 7fe676693b80 20 client.15117 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.359+0000 7fe676693b80 1 --2- 172.21.15.70:0/1367480712 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5635d8800e10 0x5635d8802f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.359+0000 7fe676693b80 4 client.15117 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:56:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.359+0000 7fe676693b80 1 -- 172.21.15.70:0/1367480712 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635d8748c70 con 0x5635d8800e10 2024-06-04T04:56:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.359+0000 7fe669399700 1 --2- 172.21.15.70:0/1367480712 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5635d8800e10 0x5635d8802f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.360+0000 7fe64d7fa700 10 client.15117 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T04:56:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.360+0000 7fe669399700 1 --2- 172.21.15.70:0/1367480712 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5635d8800e10 0x5635d8802f80 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e1bebb33daf13114 server_cookie=cd9e0d4dd08225f3 in_seq=0 out_seq=0 2024-06-04T04:56:50.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.360+0000 7fe64d7fa700 1 -- 172.21.15.70:0/1367480712 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5635d8748c70 con 0x5635d8800e10 2024-06-04T04:56:50.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.360+0000 7fe64d7fa700 10 client.15117 handle_command_reply: tid=0 2024-06-04T04:56:50.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.360+0000 7fe676693b80 1 client.15117 shutdown 2024-06-04T04:56:50.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.361+0000 7fe676693b80 1 -- 172.21.15.70:0/1367480712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87e8850 msgr2=0x5635d87f4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:50.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.361+0000 7fe676693b80 1 --2- 172.21.15.70:0/1367480712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87e8850 0x5635d87f4bf0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe660007790 tx=0x7fe6600077c0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.361+0000 7fe676693b80 1 -- 172.21.15.70:0/1367480712 shutdown_connections 2024-06-04T04:56:50.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.361+0000 7fe676693b80 1 --2- 172.21.15.70:0/1367480712 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5635d8800e10 0x5635d8802f80 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.361+0000 7fe676693b80 1 --2- 172.21.15.70:0/1367480712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87ee2d0 0x5635d87f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.361+0000 7fe64d7fa700 0 client.15117 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T04:56:50.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.361+0000 7fe676693b80 1 --2- 172.21.15.70:0/1367480712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87e8850 0x5635d87f4bf0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.361+0000 7fe676693b80 1 --2- 172.21.15.70:0/1367480712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87e9210 0x5635d87f1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.361+0000 7fe676693b80 1 -- 172.21.15.70:0/1367480712 >> 172.21.15.70:0/1367480712 conn(0x5635d87141f0 msgr2=0x5635d87e2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:50.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.361+0000 7fe676693b80 1 -- 172.21.15.70:0/1367480712 shutdown_connections 2024-06-04T04:56:50.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.361+0000 7fe676693b80 1 -- 172.21.15.70:0/1367480712 wait complete. 2024-06-04T04:56:50.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.361+0000 7fe676693b80 20 client.15117 trim_cache size 0 max 16384 2024-06-04T04:56:50.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.382+0000 7fe676693b80 1 Processor -- start 2024-06-04T04:56:50.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.382+0000 7fe676693b80 1 -- start start 2024-06-04T04:56:50.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.382+0000 7fe676693b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87e8850 0x5635d87e8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.382+0000 7fe676693b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87e9210 0x5635d87e7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.382+0000 7fe676693b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87ee2d0 0x5635d87e7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.382+0000 7fe676693b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5635d88012c0 con 0x5635d87ee2d0 2024-06-04T04:56:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.382+0000 7fe676693b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5635d8801990 con 0x5635d87e8850 2024-06-04T04:56:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.382+0000 7fe676693b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5635d8802060 con 0x5635d87e9210 2024-06-04T04:56:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.382+0000 7fe66a39b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87ee2d0 0x5635d87e7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.382+0000 7fe66a39b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87ee2d0 0x5635d87e7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44550/0 (socket says 172.21.15.70:44550) 2024-06-04T04:56:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.382+0000 7fe66a39b700 1 -- 172.21.15.70:0/3023182238 learned_addr learned my addr 172.21.15.70:0/3023182238 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.382+0000 7fe669399700 1 --2- 172.21.15.70:0/3023182238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87e9210 0x5635d87e7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.383+0000 7fe669b9a700 1 --2- 172.21.15.70:0/3023182238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87e8850 0x5635d87e8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.383+0000 7fe66a39b700 1 -- 172.21.15.70:0/3023182238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87e9210 msgr2=0x5635d87e7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.383+0000 7fe66a39b700 1 --2- 172.21.15.70:0/3023182238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87e9210 0x5635d87e7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.383+0000 7fe66a39b700 1 -- 172.21.15.70:0/3023182238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87e8850 msgr2=0x5635d87e8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.383+0000 7fe66a39b700 1 --2- 172.21.15.70:0/3023182238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87e8850 0x5635d87e8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.383+0000 7fe66a39b700 1 -- 172.21.15.70:0/3023182238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635d87fe8f0 con 0x5635d87ee2d0 2024-06-04T04:56:50.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.383+0000 7fe66a39b700 1 --2- 172.21.15.70:0/3023182238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87ee2d0 0x5635d87e7d80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00dc70 tx=0x7fe65c00df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d59bfcea91a27a9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:50.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.383+0000 7fe64f7fe700 1 -- 172.21.15.70:0/3023182238 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe65c012c50 con 0x5635d87ee2d0 2024-06-04T04:56:50.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.383+0000 7fe64f7fe700 1 -- 172.21.15.70:0/3023182238 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe65c012df0 con 0x5635d87ee2d0 2024-06-04T04:56:50.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.383+0000 7fe64f7fe700 1 -- 172.21.15.70:0/3023182238 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe65c00f3f0 con 0x5635d87ee2d0 2024-06-04T04:56:50.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.384+0000 7fe676693b80 1 -- 172.21.15.70:0/3023182238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87ee2d0 msgr2=0x5635d87e7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.384+0000 7fe676693b80 1 --2- 172.21.15.70:0/3023182238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87ee2d0 0x5635d87e7d80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00dc70 tx=0x7fe65c00df80 comp rx=0 tx=0).stop 2024-06-04T04:56:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.384+0000 7fe676693b80 1 -- 172.21.15.70:0/3023182238 shutdown_connections 2024-06-04T04:56:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.384+0000 7fe676693b80 1 --2- 172.21.15.70:0/3023182238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87ee2d0 0x5635d87e7d80 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.384+0000 7fe676693b80 1 --2- 172.21.15.70:0/3023182238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87e8850 0x5635d87e8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.384+0000 7fe676693b80 1 --2- 172.21.15.70:0/3023182238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87e9210 0x5635d87e7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.384+0000 7fe676693b80 1 -- 172.21.15.70:0/3023182238 >> 172.21.15.70:0/3023182238 conn(0x5635d87141f0 msgr2=0x5635d87f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.384+0000 7fe676693b80 1 -- 172.21.15.70:0/3023182238 shutdown_connections 2024-06-04T04:56:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.384+0000 7fe676693b80 1 -- 172.21.15.70:0/3023182238 wait complete. 2024-06-04T04:56:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.384+0000 7fe676693b80 1 -- start start 2024-06-04T04:56:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.384+0000 7fe676693b80 1 Processor -- start 2024-06-04T04:56:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe676693b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87e8850 0x5635d8805580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe676693b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87e9210 0x5635d8805ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe676693b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87ee2d0 0x5635d880a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe676693b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5635d8802e00 con 0x5635d87e9210 2024-06-04T04:56:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe676693b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5635d8802730 con 0x5635d87e8850 2024-06-04T04:56:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe676693b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5635d88034d0 con 0x5635d87ee2d0 2024-06-04T04:56:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe676693b80 10 client.0 _ll_register_callbacks cb 0x5635d86d1900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe669399700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87e9210 0x5635d8805ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe669399700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87e9210 0x5635d8805ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44558/0 (socket says 172.21.15.70:44558) 2024-06-04T04:56:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe669399700 1 -- 172.21.15.70:0/4017214293 learned_addr learned my addr 172.21.15.70:0/4017214293 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe66a39b700 1 --2- 172.21.15.70:0/4017214293 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87ee2d0 0x5635d880a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe669b9a700 1 --2- 172.21.15.70:0/4017214293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87e8850 0x5635d8805580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe669399700 1 -- 172.21.15.70:0/4017214293 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87ee2d0 msgr2=0x5635d880a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe669399700 1 --2- 172.21.15.70:0/4017214293 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87ee2d0 0x5635d880a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe669399700 1 -- 172.21.15.70:0/4017214293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87e8850 msgr2=0x5635d8805580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe669399700 1 --2- 172.21.15.70:0/4017214293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87e8850 0x5635d8805580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe669399700 1 -- 172.21.15.70:0/4017214293 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635d880ab30 con 0x5635d87e9210 2024-06-04T04:56:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe669399700 1 --2- 172.21.15.70:0/4017214293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87e9210 0x5635d8805ac0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe6540e0060 tx=0x7fe65400ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfbe802188a593c4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.385+0000 7fe64d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.386+0000 7fe64d7fa700 1 -- 172.21.15.70:0/4017214293 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6540a9810 con 0x5635d87e9210 2024-06-04T04:56:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.386+0000 7fe64d7fa700 1 -- 172.21.15.70:0/4017214293 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fe65406f0b0 con 0x5635d87e9210 2024-06-04T04:56:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.386+0000 7fe676693b80 1 -- 172.21.15.70:0/4017214293 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe65c012df0 con 0x5635d87e9210 2024-06-04T04:56:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.386+0000 7fe64d7fa700 1 -- 172.21.15.70:0/4017214293 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe6540adbc0 con 0x5635d87e9210 2024-06-04T04:56:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.386+0000 7fe676693b80 10 client.15123 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.386+0000 7fe676693b80 1 -- 172.21.15.70:0/4017214293 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5635d880bee0 con 0x5635d87e9210 2024-06-04T04:56:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.386+0000 7fe64d7fa700 1 -- 172.21.15.70:0/4017214293 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe65c012df0 con 0x5635d87e9210 2024-06-04T04:56:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.386+0000 7fe64d7fa700 1 -- 172.21.15.70:0/4017214293 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fe6540a9b00 con 0x5635d87e9210 2024-06-04T04:56:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.386+0000 7fe676693b80 10 client.15123 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:50.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.386+0000 7fe676693b80 10 client.15123 resolve_mds: resolved name 'h' to daemon mds.h 2024-06-04T04:56:50.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.386+0000 7fe676693b80 20 client.15123 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:50.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.386+0000 7fe676693b80 1 --2- 172.21.15.70:0/4017214293 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5635d8872bc0 0x5635d88718d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:50.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.386+0000 7fe676693b80 4 client.15123 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-04T04:56:50.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.386+0000 7fe676693b80 1 -- 172.21.15.70:0/4017214293 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "status"}) -- 0x5635d8742450 con 0x5635d8872bc0 2024-06-04T04:56:50.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.387+0000 7fe669b9a700 1 --2- 172.21.15.70:0/4017214293 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5635d8872bc0 0x5635d88718d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:50.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.387+0000 7fe669b9a700 1 --2- 172.21.15.70:0/4017214293 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5635d8872bc0 0x5635d88718d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dcd673b493368456 server_cookie=7d9fa6f702987180 in_seq=0 out_seq=0 2024-06-04T04:56:50.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.387+0000 7fe64d7fa700 10 client.15123 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T04:56:50.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.387+0000 7fe64d7fa700 1 -- 172.21.15.70:0/4017214293 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5635d8742450 con 0x5635d8872bc0 2024-06-04T04:56:50.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.387+0000 7fe64d7fa700 10 client.15123 handle_command_reply: tid=0 2024-06-04T04:56:50.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.387+0000 7fe676693b80 1 client.15123 shutdown 2024-06-04T04:56:50.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.387+0000 7fe676693b80 1 -- 172.21.15.70:0/4017214293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87e9210 msgr2=0x5635d8805ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:50.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.387+0000 7fe676693b80 1 --2- 172.21.15.70:0/4017214293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87e9210 0x5635d8805ac0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe6540e0060 tx=0x7fe65400ff90 comp rx=0 tx=0).stop 2024-06-04T04:56:50.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.387+0000 7fe676693b80 1 -- 172.21.15.70:0/4017214293 shutdown_connections 2024-06-04T04:56:50.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.387+0000 7fe676693b80 1 --2- 172.21.15.70:0/4017214293 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5635d8872bc0 0x5635d88718d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.387+0000 7fe676693b80 1 --2- 172.21.15.70:0/4017214293 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635d87ee2d0 0x5635d880a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.387+0000 7fe676693b80 1 --2- 172.21.15.70:0/4017214293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635d87e8850 0x5635d8805580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.387+0000 7fe676693b80 1 --2- 172.21.15.70:0/4017214293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635d87e9210 0x5635d8805ac0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.387+0000 7fe64d7fa700 0 client.15123 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T04:56:50.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.387+0000 7fe676693b80 1 -- 172.21.15.70:0/4017214293 >> 172.21.15.70:0/4017214293 conn(0x5635d87141f0 msgr2=0x5635d87ebc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:50.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.388+0000 7fe676693b80 1 -- 172.21.15.70:0/4017214293 shutdown_connections 2024-06-04T04:56:50.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.388+0000 7fe676693b80 1 -- 172.21.15.70:0/4017214293 wait complete. 2024-06-04T04:56:50.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.388+0000 7fe676693b80 20 client.15123 trim_cache size 0 max 16384 2024-06-04T04:56:50.395 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:56:50.395 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:56:50.395 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 4, 2024-06-04T04:56:50.396 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24410, 2024-06-04T04:56:50.396 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-06-04T04:56:50.396 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-06-04T04:56:50.396 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-06-04T04:56:50.396 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 176.951560736, 2024-06-04T04:56:50.396 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-06-04T04:56:50.396 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 81, 2024-06-04T04:56:50.396 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 79, 2024-06-04T04:56:50.396 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 220.41294965200001 2024-06-04T04:56:50.396 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:56:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.388+0000 7fe66bdfe700 1 -- 172.21.15.70:0/2747629183 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe6500b2880 msgr2=0x7fe6500b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.388+0000 7fe66bdfe700 1 --2- 172.21.15.70:0/2747629183 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe6500b2880 0x7fe6500b4d40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe660009e60 tx=0x7fe66000b7b0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.388+0000 7fe66bdfe700 1 -- 172.21.15.70:0/2747629183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe664102e00 msgr2=0x7fe6641045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.388+0000 7fe66bdfe700 1 --2- 172.21.15.70:0/2747629183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe664102e00 0x7fe6641045c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe65400c9b0 tx=0x7fe65400ccc0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.388+0000 7fe66bdfe700 1 -- 172.21.15.70:0/2747629183 shutdown_connections 2024-06-04T04:56:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.388+0000 7fe66bdfe700 1 --2- 172.21.15.70:0/2747629183 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fe6500b2880 0x7fe6500b4d40 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.388+0000 7fe66bdfe700 1 --2- 172.21.15.70:0/2747629183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe664108520 0x7fe664104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.388+0000 7fe66bdfe700 1 --2- 172.21.15.70:0/2747629183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe664102440 0x7fe664104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.388+0000 7fe66bdfe700 1 --2- 172.21.15.70:0/2747629183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe664102e00 0x7fe6641045c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.388+0000 7fe66bdfe700 1 -- 172.21.15.70:0/2747629183 >> 172.21.15.70:0/2747629183 conn(0x7fe6640793f0 msgr2=0x7fe66406b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.388+0000 7fe66bdfe700 1 -- 172.21.15.70:0/2747629183 shutdown_connections 2024-06-04T04:56:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:50.388+0000 7fe66bdfe700 1 -- 172.21.15.70:0/2747629183 wait complete. 2024-06-04T04:56:51.083 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.f status 2024-06-04T04:56:51.322 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:51 smithi098 ceph-mon[35569]: pgmap v387: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:56:51.784 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:51 smithi070 ceph-mon[32631]: pgmap v387: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:56:51.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:51 smithi112 ceph-mon[36453]: pgmap v387: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:56:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.983+0000 7fa7a59df700 1 -- 172.21.15.70:0/674804617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a0108520 msgr2=0x7fa7a0108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.983+0000 7fa7a59df700 1 --2- 172.21.15.70:0/674804617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a0108520 0x7fa7a0108900 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa790009990 tx=0x7fa790009ca0 comp rx=0 tx=0).stop 2024-06-04T04:56:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.983+0000 7fa7a59df700 1 -- 172.21.15.70:0/674804617 shutdown_connections 2024-06-04T04:56:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.983+0000 7fa7a59df700 1 --2- 172.21.15.70:0/674804617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0102e00 0x7fa7a010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.983+0000 7fa7a59df700 1 --2- 172.21.15.70:0/674804617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a0108520 0x7fa7a0108900 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.983+0000 7fa7a59df700 1 --2- 172.21.15.70:0/674804617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7a0102440 0x7fa7a01028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.983+0000 7fa7a59df700 1 -- 172.21.15.70:0/674804617 >> 172.21.15.70:0/674804617 conn(0x7fa7a00793f0 msgr2=0x7fa7a0079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.984+0000 7fa7a59df700 1 -- 172.21.15.70:0/674804617 shutdown_connections 2024-06-04T04:56:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.984+0000 7fa7a59df700 1 -- 172.21.15.70:0/674804617 wait complete. 2024-06-04T04:56:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.985+0000 7fa7a59df700 1 Processor -- start 2024-06-04T04:56:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.985+0000 7fa7a59df700 1 -- start start 2024-06-04T04:56:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.985+0000 7fa7a59df700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a0102440 0x7fa7a019aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.985+0000 7fa7a59df700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7a0102e00 0x7fa7a019b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.985+0000 7fa7a59df700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0108520 0x7fa7a01a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.985+0000 7fa7a59df700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a0112580 con 0x7fa7a0102440 2024-06-04T04:56:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.985+0000 7fa7a59df700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a0112400 con 0x7fa7a0108520 2024-06-04T04:56:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.985+0000 7fa7a59df700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a0112700 con 0x7fa7a0102e00 2024-06-04T04:56:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.985+0000 7fa79effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a0102440 0x7fa7a019aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.985+0000 7fa79effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a0102440 0x7fa7a019aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44574/0 (socket says 172.21.15.70:44574) 2024-06-04T04:56:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.985+0000 7fa79effd700 1 -- 172.21.15.70:0/2715266617 learned_addr learned my addr 172.21.15.70:0/2715266617 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.985+0000 7fa79e7fc700 1 --2- 172.21.15.70:0/2715266617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7a0102e00 0x7fa7a019b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.986+0000 7fa79f7fe700 1 --2- 172.21.15.70:0/2715266617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0108520 0x7fa7a01a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.986+0000 7fa79effd700 1 -- 172.21.15.70:0/2715266617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7a0102e00 msgr2=0x7fa7a019b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.986+0000 7fa79effd700 1 --2- 172.21.15.70:0/2715266617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7a0102e00 0x7fa7a019b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.986+0000 7fa79effd700 1 -- 172.21.15.70:0/2715266617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0108520 msgr2=0x7fa7a01a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.986+0000 7fa79effd700 1 --2- 172.21.15.70:0/2715266617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0108520 0x7fa7a01a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.986+0000 7fa79effd700 1 -- 172.21.15.70:0/2715266617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a01a6090 con 0x7fa7a0102440 2024-06-04T04:56:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.986+0000 7fa79e7fc700 1 --2- 172.21.15.70:0/2715266617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7a0102e00 0x7fa7a019b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.986+0000 7fa79f7fe700 1 --2- 172.21.15.70:0/2715266617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0108520 0x7fa7a01a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:56:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.986+0000 7fa79effd700 1 --2- 172.21.15.70:0/2715266617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a0102440 0x7fa7a019aad0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa7900057f0 tx=0x7fa790009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.986+0000 7fa7a49dd700 1 -- 172.21.15.70:0/2715266617 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790027070 con 0x7fa7a0102440 2024-06-04T04:56:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.986+0000 7fa7a59df700 1 -- 172.21.15.70:0/2715266617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a01a6260 con 0x7fa7a0102440 2024-06-04T04:56:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.986+0000 7fa7a49dd700 1 -- 172.21.15.70:0/2715266617 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa79000fd60 con 0x7fa7a0102440 2024-06-04T04:56:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.986+0000 7fa7a49dd700 1 -- 172.21.15.70:0/2715266617 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790022d70 con 0x7fa7a0102440 2024-06-04T04:56:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.986+0000 7fa7a59df700 1 -- 172.21.15.70:0/2715266617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a01a6790 con 0x7fa7a0102440 2024-06-04T04:56:52.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.988+0000 7fa7a49dd700 1 -- 172.21.15.70:0/2715266617 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fa79002a070 con 0x7fa7a0102440 2024-06-04T04:56:52.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.989+0000 7fa7a49dd700 1 --2- 172.21.15.70:0/2715266617 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa7880b2880 0x7fa7880b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.989+0000 7fa79e7fc700 1 --2- 172.21.15.70:0/2715266617 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa7880b2880 0x7fa7880b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.989+0000 7fa7a49dd700 1 -- 172.21.15.70:0/2715266617 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fa7900e3210 con 0x7fa7a0102440 2024-06-04T04:56:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.989+0000 7fa7b0274b80 1 Processor -- start 2024-06-04T04:56:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa7b0274b80 1 -- start start 2024-06-04T04:56:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa79e7fc700 1 --2- 172.21.15.70:0/2715266617 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa7880b2880 0x7fa7880b4d40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0079c60 tx=0x7fa794008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa7b0274b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e801675c0 0x562e801679a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa7b0274b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e80167f70 0x562e80161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa7b0274b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e80162280 0x562e801626e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa7b0274b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562e800bbd20 con 0x562e80167f70 2024-06-04T04:56:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa7b0274b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562e800ab8c0 con 0x562e80162280 2024-06-04T04:56:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa7b0274b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562e800af900 con 0x562e801675c0 2024-06-04T04:56:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa79e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e80167f70 0x562e80161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa79e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e80167f70 0x562e80161c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44586/0 (socket says 172.21.15.70:44586) 2024-06-04T04:56:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa79e7fc700 1 -- 172.21.15.70:0/3487097254 learned_addr learned my addr 172.21.15.70:0/3487097254 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa79effd700 1 --2- 172.21.15.70:0/3487097254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e801675c0 0x562e801679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa79e7fc700 1 -- 172.21.15.70:0/3487097254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e801675c0 msgr2=0x562e801679a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa79e7fc700 1 --2- 172.21.15.70:0/3487097254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e801675c0 0x562e801679a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa79e7fc700 1 -- 172.21.15.70:0/3487097254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e80162280 msgr2=0x562e801626e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa79e7fc700 1 --2- 172.21.15.70:0/3487097254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e80162280 0x562e801626e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa79e7fc700 1 -- 172.21.15.70:0/3487097254 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e80054f60 con 0x562e80167f70 2024-06-04T04:56:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa79e7fc700 1 --2- 172.21.15.70:0/3487097254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e80167f70 0x562e80161c60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa79400d750 tx=0x7fa794010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff7f1d97e6e72e26 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.990+0000 7fa785ffb700 1 -- 172.21.15.70:0/3487097254 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa794024070 con 0x562e80167f70 2024-06-04T04:56:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.991+0000 7fa785ffb700 1 -- 172.21.15.70:0/3487097254 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa794014ad0 con 0x562e80167f70 2024-06-04T04:56:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.991+0000 7fa785ffb700 1 -- 172.21.15.70:0/3487097254 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79401fa70 con 0x562e80167f70 2024-06-04T04:56:52.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.991+0000 7fa7b0274b80 1 -- 172.21.15.70:0/3487097254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e80167f70 msgr2=0x562e80161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:52.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.991+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/3487097254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e80167f70 0x562e80161c60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa79400d750 tx=0x7fa794010000 comp rx=0 tx=0).stop 2024-06-04T04:56:52.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.991+0000 7fa7b0274b80 1 -- 172.21.15.70:0/3487097254 shutdown_connections 2024-06-04T04:56:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.991+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/3487097254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e80162280 0x562e801626e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.991+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/3487097254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e801675c0 0x562e801679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.991+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/3487097254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e80167f70 0x562e80161c60 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.991+0000 7fa7b0274b80 1 -- 172.21.15.70:0/3487097254 >> 172.21.15.70:0/3487097254 conn(0x562e800891f0 msgr2=0x562e80158390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.991+0000 7fa7b0274b80 1 -- 172.21.15.70:0/3487097254 shutdown_connections 2024-06-04T04:56:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.991+0000 7fa7b0274b80 1 -- 172.21.15.70:0/3487097254 wait complete. 2024-06-04T04:56:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.992+0000 7fa7b0274b80 1 -- start start 2024-06-04T04:56:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.992+0000 7fa7b0274b80 1 Processor -- start 2024-06-04T04:56:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.992+0000 7fa7b0274b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e80162280 0x562e8016dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.992+0000 7fa7b0274b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e801675c0 0x562e8016b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.992+0000 7fa7b0274b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e80167f70 0x562e8016b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.992+0000 7fa7b0274b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562e7ffb1f50 con 0x562e801675c0 2024-06-04T04:56:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.992+0000 7fa7b0274b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562e80013d50 con 0x562e80167f70 2024-06-04T04:56:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.992+0000 7fa7b0274b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562e8016ea50 con 0x562e80162280 2024-06-04T04:56:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.992+0000 7fa79e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e801675c0 0x562e8016b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.992+0000 7fa7b0274b80 10 client.0 _ll_register_callbacks cb 0x562e80046900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.992+0000 7fa79e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e801675c0 0x562e8016b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44600/0 (socket says 172.21.15.70:44600) 2024-06-04T04:56:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.992+0000 7fa79e7fc700 1 -- 172.21.15.70:0/1766148701 learned_addr learned my addr 172.21.15.70:0/1766148701 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.992+0000 7fa79effd700 1 --2- 172.21.15.70:0/1766148701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e80162280 0x562e8016dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.992+0000 7fa79f7fe700 1 --2- 172.21.15.70:0/1766148701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e80167f70 0x562e8016b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.993+0000 7fa79e7fc700 1 -- 172.21.15.70:0/1766148701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e80162280 msgr2=0x562e8016dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.993+0000 7fa79e7fc700 1 --2- 172.21.15.70:0/1766148701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e80162280 0x562e8016dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.993+0000 7fa79e7fc700 1 -- 172.21.15.70:0/1766148701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e80167f70 msgr2=0x562e8016b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.993+0000 7fa79e7fc700 1 --2- 172.21.15.70:0/1766148701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e80167f70 0x562e8016b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.993+0000 7fa79e7fc700 1 -- 172.21.15.70:0/1766148701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e7fffb880 con 0x562e801675c0 2024-06-04T04:56:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.993+0000 7fa79e7fc700 1 --2- 172.21.15.70:0/1766148701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e801675c0 0x562e8016b0a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa794004790 tx=0x7fa7940146f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be29bf2a54e6d209 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.993+0000 7fa77effd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.993+0000 7fa77effd700 1 -- 172.21.15.70:0/1766148701 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79402f060 con 0x562e801675c0 2024-06-04T04:56:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.993+0000 7fa77effd700 1 -- 172.21.15.70:0/1766148701 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fa79401f6f0 con 0x562e801675c0 2024-06-04T04:56:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.993+0000 7fa7b0274b80 1 -- 172.21.15.70:0/1766148701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa794014ad0 con 0x562e801675c0 2024-06-04T04:56:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.993+0000 7fa77effd700 1 -- 172.21.15.70:0/1766148701 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa794024040 con 0x562e801675c0 2024-06-04T04:56:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.993+0000 7fa7b0274b80 10 client.15135 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.993+0000 7fa7b0274b80 1 -- 172.21.15.70:0/1766148701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562e80065710 con 0x562e801675c0 2024-06-04T04:56:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.993+0000 7fa77effd700 1 -- 172.21.15.70:0/1766148701 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa794014ad0 con 0x562e801675c0 2024-06-04T04:56:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.993+0000 7fa77effd700 1 -- 172.21.15.70:0/1766148701 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa794013070 con 0x562e801675c0 2024-06-04T04:56:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.993+0000 7fa7b0274b80 10 client.15135 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.994+0000 7fa7b0274b80 10 client.15135 resolve_mds: resolved name 'f' to daemon mds.f 2024-06-04T04:56:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.994+0000 7fa7b0274b80 20 client.15135 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.994+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/1766148701 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x562e8017b370 0x562e8017c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.994+0000 7fa7b0274b80 4 client.15135 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:56:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.994+0000 7fa7b0274b80 1 -- 172.21.15.70:0/1766148701 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e800bdc70 con 0x562e8017b370 2024-06-04T04:56:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.994+0000 7fa79effd700 1 --2- 172.21.15.70:0/1766148701 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x562e8017b370 0x562e8017c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.994+0000 7fa79effd700 1 --2- 172.21.15.70:0/1766148701 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x562e8017b370 0x562e8017c7e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=371ca991190ef2be server_cookie=38acc51ecad09565 in_seq=0 out_seq=0 2024-06-04T04:56:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.994+0000 7fa77effd700 10 client.15135 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T04:56:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.995+0000 7fa77effd700 1 -- 172.21.15.70:0/1766148701 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562e800bdc70 con 0x562e8017b370 2024-06-04T04:56:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.995+0000 7fa77effd700 10 client.15135 handle_command_reply: tid=0 2024-06-04T04:56:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.995+0000 7fa7b0274b80 1 client.15135 shutdown 2024-06-04T04:56:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.995+0000 7fa7b0274b80 1 -- 172.21.15.70:0/1766148701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e801675c0 msgr2=0x562e8016b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.995+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/1766148701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e801675c0 0x562e8016b0a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa794004790 tx=0x7fa7940146f0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.995+0000 7fa7b0274b80 1 -- 172.21.15.70:0/1766148701 shutdown_connections 2024-06-04T04:56:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.995+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/1766148701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e80167f70 0x562e8016b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.995+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/1766148701 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x562e8017b370 0x562e8017c7e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.995+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/1766148701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e80162280 0x562e8016dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.995+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/1766148701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e801675c0 0x562e8016b0a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.995+0000 7fa7b0274b80 1 -- 172.21.15.70:0/1766148701 >> 172.21.15.70:0/1766148701 conn(0x562e800891f0 msgr2=0x562e80157aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.995+0000 7fa77effd700 0 client.15135 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T04:56:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.995+0000 7fa7b0274b80 1 -- 172.21.15.70:0/1766148701 shutdown_connections 2024-06-04T04:56:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.995+0000 7fa7b0274b80 1 -- 172.21.15.70:0/1766148701 wait complete. 2024-06-04T04:56:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:52.995+0000 7fa7b0274b80 20 client.15135 trim_cache size 0 max 16384 2024-06-04T04:56:53.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa7b0274b80 1 Processor -- start 2024-06-04T04:56:53.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa7b0274b80 1 -- start start 2024-06-04T04:56:53.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa7b0274b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e80162280 0x562e80162660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:53.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa7b0274b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e801675c0 0x562e80158220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:53.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa7b0274b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e80167f70 0x562e80168350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:53.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa7b0274b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562e8017ab20 con 0x562e80167f70 2024-06-04T04:56:53.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa7b0274b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562e8017b1f0 con 0x562e801675c0 2024-06-04T04:56:53.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa7b0274b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562e8017b8c0 con 0x562e80162280 2024-06-04T04:56:53.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa79f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e80167f70 0x562e80168350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:53.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa79f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e80167f70 0x562e80168350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44616/0 (socket says 172.21.15.70:44616) 2024-06-04T04:56:53.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa79f7fe700 1 -- 172.21.15.70:0/2092093932 learned_addr learned my addr 172.21.15.70:0/2092093932 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:53.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa79e7fc700 1 --2- 172.21.15.70:0/2092093932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e801675c0 0x562e80158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:53.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa79effd700 1 --2- 172.21.15.70:0/2092093932 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e80162280 0x562e80162660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:53.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa79f7fe700 1 -- 172.21.15.70:0/2092093932 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e80162280 msgr2=0x562e80162660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:53.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa79f7fe700 1 --2- 172.21.15.70:0/2092093932 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e80162280 0x562e80162660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa79f7fe700 1 -- 172.21.15.70:0/2092093932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e801675c0 msgr2=0x562e80158220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:53.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa79f7fe700 1 --2- 172.21.15.70:0/2092093932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e801675c0 0x562e80158220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa79f7fe700 1 -- 172.21.15.70:0/2092093932 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e80159c40 con 0x562e80167f70 2024-06-04T04:56:53.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.016+0000 7fa79f7fe700 1 --2- 172.21.15.70:0/2092093932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e80167f70 0x562e80168350 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa798007eb0 tx=0x7fa79800e620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1192fb694b5f7b2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:53.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.017+0000 7fa7857fa700 1 -- 172.21.15.70:0/2092093932 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa798014070 con 0x562e80167f70 2024-06-04T04:56:53.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.017+0000 7fa7857fa700 1 -- 172.21.15.70:0/2092093932 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa7980198f0 con 0x562e80167f70 2024-06-04T04:56:53.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.017+0000 7fa7857fa700 1 -- 172.21.15.70:0/2092093932 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7980109e0 con 0x562e80167f70 2024-06-04T04:56:53.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.017+0000 7fa7b0274b80 1 -- 172.21.15.70:0/2092093932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e80167f70 msgr2=0x562e80168350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:53.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.017+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/2092093932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e80167f70 0x562e80168350 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa798007eb0 tx=0x7fa79800e620 comp rx=0 tx=0).stop 2024-06-04T04:56:53.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.017+0000 7fa7b0274b80 1 -- 172.21.15.70:0/2092093932 shutdown_connections 2024-06-04T04:56:53.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.017+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/2092093932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e80167f70 0x562e80168350 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.017+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/2092093932 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e80162280 0x562e80162660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.017+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/2092093932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e801675c0 0x562e80158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.017+0000 7fa7b0274b80 1 -- 172.21.15.70:0/2092093932 >> 172.21.15.70:0/2092093932 conn(0x562e800891f0 msgr2=0x562e80154710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:53.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.017+0000 7fa7b0274b80 1 -- 172.21.15.70:0/2092093932 shutdown_connections 2024-06-04T04:56:53.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.017+0000 7fa7b0274b80 1 -- 172.21.15.70:0/2092093932 wait complete. 2024-06-04T04:56:53.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.018+0000 7fa7b0274b80 1 -- start start 2024-06-04T04:56:53.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.018+0000 7fa7b0274b80 1 Processor -- start 2024-06-04T04:56:53.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.018+0000 7fa7b0274b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e80162280 0x562e80185c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:53.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.018+0000 7fa7b0274b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e801675c0 0x562e801861a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:53.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.018+0000 7fa7b0274b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e80167f70 0x562e80180c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:53.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.018+0000 7fa7b0274b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562e8017c660 con 0x562e801675c0 2024-06-04T04:56:53.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.018+0000 7fa7b0274b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562e8017bf90 con 0x562e80162280 2024-06-04T04:56:53.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.018+0000 7fa7b0274b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562e8017cd30 con 0x562e80167f70 2024-06-04T04:56:53.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.018+0000 7fa7b0274b80 10 client.0 _ll_register_callbacks cb 0x562e80046900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:53.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.018+0000 7fa79e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e801675c0 0x562e801861a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:53.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.018+0000 7fa79e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e801675c0 0x562e801861a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44632/0 (socket says 172.21.15.70:44632) 2024-06-04T04:56:53.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.018+0000 7fa79e7fc700 1 -- 172.21.15.70:0/3128820569 learned_addr learned my addr 172.21.15.70:0/3128820569 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:53.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.018+0000 7fa79f7fe700 1 --2- 172.21.15.70:0/3128820569 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e80167f70 0x562e80180c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:53.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.019+0000 7fa79effd700 1 --2- 172.21.15.70:0/3128820569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e80162280 0x562e80185c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.019+0000 7fa79e7fc700 1 -- 172.21.15.70:0/3128820569 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e80167f70 msgr2=0x562e80180c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.019+0000 7fa79e7fc700 1 --2- 172.21.15.70:0/3128820569 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e80167f70 0x562e80180c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.019+0000 7fa79e7fc700 1 -- 172.21.15.70:0/3128820569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e80162280 msgr2=0x562e80185c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.019+0000 7fa79e7fc700 1 --2- 172.21.15.70:0/3128820569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e80162280 0x562e80185c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.019+0000 7fa79e7fc700 1 -- 172.21.15.70:0/3128820569 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e8017d740 con 0x562e801675c0 2024-06-04T04:56:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.019+0000 7fa79effd700 1 --2- 172.21.15.70:0/3128820569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e80162280 0x562e80185c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:56:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.019+0000 7fa79f7fe700 1 --2- 172.21.15.70:0/3128820569 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e80167f70 0x562e80180c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.019+0000 7fa79e7fc700 1 --2- 172.21.15.70:0/3128820569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e801675c0 0x562e801861a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa7940123e0 tx=0x7fa794014db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=182546230daa8989 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.019+0000 7fa77effd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.019+0000 7fa77effd700 1 -- 172.21.15.70:0/3128820569 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79402f040 con 0x562e801675c0 2024-06-04T04:56:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.019+0000 7fa7b0274b80 1 -- 172.21.15.70:0/3128820569 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7980198f0 con 0x562e801675c0 2024-06-04T04:56:53.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.019+0000 7fa77effd700 1 -- 172.21.15.70:0/3128820569 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fa79401f680 con 0x562e801675c0 2024-06-04T04:56:53.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.019+0000 7fa77effd700 1 -- 172.21.15.70:0/3128820569 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa794009710 con 0x562e801675c0 2024-06-04T04:56:53.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.019+0000 7fa7b0274b80 10 client.15141 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:53.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.019+0000 7fa7b0274b80 1 -- 172.21.15.70:0/3128820569 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562e801e6f80 con 0x562e801675c0 2024-06-04T04:56:53.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.019+0000 7fa77effd700 1 -- 172.21.15.70:0/3128820569 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa7980198f0 con 0x562e801675c0 2024-06-04T04:56:53.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.020+0000 7fa77effd700 1 -- 172.21.15.70:0/3128820569 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa79403e590 con 0x562e801675c0 2024-06-04T04:56:53.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.020+0000 7fa7b0274b80 10 client.15141 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:53.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.020+0000 7fa7b0274b80 10 client.15141 resolve_mds: resolved name 'f' to daemon mds.f 2024-06-04T04:56:53.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.020+0000 7fa7b0274b80 20 client.15141 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:53.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.020+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/3128820569 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x562e801ebb50 0x562e801ea860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:53.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.020+0000 7fa7b0274b80 4 client.15141 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-04T04:56:53.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.020+0000 7fa7b0274b80 1 -- 172.21.15.70:0/3128820569 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "status"}) -- 0x562e800b7450 con 0x562e801ebb50 2024-06-04T04:56:53.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.020+0000 7fa79effd700 1 --2- 172.21.15.70:0/3128820569 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x562e801ebb50 0x562e801ea860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:53.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.020+0000 7fa79effd700 1 --2- 172.21.15.70:0/3128820569 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x562e801ebb50 0x562e801ea860 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1a3c69557f5d8fc7 server_cookie=73977f7ef00a9686 in_seq=0 out_seq=0 2024-06-04T04:56:53.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.020+0000 7fa77effd700 10 client.15141 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T04:56:53.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.021+0000 7fa77effd700 1 -- 172.21.15.70:0/3128820569 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x562e800b7450 con 0x562e801ebb50 2024-06-04T04:56:53.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.021+0000 7fa77effd700 10 client.15141 handle_command_reply: tid=0 2024-06-04T04:56:53.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.021+0000 7fa7b0274b80 1 client.15141 shutdown 2024-06-04T04:56:53.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.021+0000 7fa7b0274b80 1 -- 172.21.15.70:0/3128820569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e801675c0 msgr2=0x562e801861a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:53.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.021+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/3128820569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e801675c0 0x562e801861a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa7940123e0 tx=0x7fa794014db0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.021+0000 7fa7b0274b80 1 -- 172.21.15.70:0/3128820569 shutdown_connections 2024-06-04T04:56:53.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.021+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/3128820569 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x562e801ebb50 0x562e801ea860 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.021+0000 7fa77effd700 0 client.15141 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T04:56:53.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.021+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/3128820569 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e80167f70 0x562e80180c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.021+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/3128820569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e80162280 0x562e80185c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.021+0000 7fa7b0274b80 1 --2- 172.21.15.70:0/3128820569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e801675c0 0x562e801861a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.021+0000 7fa7b0274b80 1 -- 172.21.15.70:0/3128820569 >> 172.21.15.70:0/3128820569 conn(0x562e800891f0 msgr2=0x562e80168350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:53.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.021+0000 7fa7b0274b80 1 -- 172.21.15.70:0/3128820569 shutdown_connections 2024-06-04T04:56:53.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.021+0000 7fa7b0274b80 1 -- 172.21.15.70:0/3128820569 wait complete. 2024-06-04T04:56:53.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.021+0000 7fa7b0274b80 20 client.15141 trim_cache size 0 max 16384 2024-06-04T04:56:53.029 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:56:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:56:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 3, 2024-06-04T04:56:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24436, 2024-06-04T04:56:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-06-04T04:56:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-06-04T04:56:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-06-04T04:56:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 180.58996750899999, 2024-06-04T04:56:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-06-04T04:56:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 81, 2024-06-04T04:56:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 79, 2024-06-04T04:56:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 228.737108068 2024-06-04T04:56:53.030 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:56:53.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.021+0000 7fa7a59df700 1 -- 172.21.15.70:0/2715266617 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa7880b2880 msgr2=0x7fa7880b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:53.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.021+0000 7fa7a59df700 1 --2- 172.21.15.70:0/2715266617 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa7880b2880 0x7fa7880b4d40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0079c60 tx=0x7fa794008040 comp rx=0 tx=0).stop 2024-06-04T04:56:53.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.022+0000 7fa7a59df700 1 -- 172.21.15.70:0/2715266617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a0102440 msgr2=0x7fa7a019aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:53.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.022+0000 7fa7a59df700 1 --2- 172.21.15.70:0/2715266617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a0102440 0x7fa7a019aad0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa7900057f0 tx=0x7fa790009f90 comp rx=0 tx=0).stop 2024-06-04T04:56:53.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.022+0000 7fa7a59df700 1 -- 172.21.15.70:0/2715266617 shutdown_connections 2024-06-04T04:56:53.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.022+0000 7fa7a59df700 1 --2- 172.21.15.70:0/2715266617 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa7880b2880 0x7fa7880b4d40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.022+0000 7fa7a59df700 1 --2- 172.21.15.70:0/2715266617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0108520 0x7fa7a01a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.022+0000 7fa7a59df700 1 --2- 172.21.15.70:0/2715266617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a0102440 0x7fa7a019aad0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.022+0000 7fa7a59df700 1 --2- 172.21.15.70:0/2715266617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7a0102e00 0x7fa7a019b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:53.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.022+0000 7fa7a59df700 1 -- 172.21.15.70:0/2715266617 >> 172.21.15.70:0/2715266617 conn(0x7fa7a00793f0 msgr2=0x7fa7a01014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:53.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.022+0000 7fa7a59df700 1 -- 172.21.15.70:0/2715266617 shutdown_connections 2024-06-04T04:56:53.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:53.022+0000 7fa7a59df700 1 -- 172.21.15.70:0/2715266617 wait complete. 2024-06-04T04:56:53.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:53 smithi070 ceph-mon[32631]: pgmap v388: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:56:53.765 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-06-04T04:56:53.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:53 smithi112 ceph-mon[36453]: pgmap v388: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:56:53.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:53 smithi098 ceph-mon[35569]: pgmap v388: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-06-04T04:56:54.004 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:55.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.661+0000 7f1ac3b7f700 1 -- 172.21.15.70:0/1977001384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1abc108930 msgr2=0x7f1abc1104b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:55.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.661+0000 7f1ac3b7f700 1 --2- 172.21.15.70:0/1977001384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1abc108930 0x7f1abc1104b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40099c0 tx=0x7f1ab4009cd0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.661+0000 7f1ac3b7f700 1 -- 172.21.15.70:0/1977001384 shutdown_connections 2024-06-04T04:56:55.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.661+0000 7f1ac3b7f700 1 --2- 172.21.15.70:0/1977001384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1abc108930 0x7f1abc1104b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.661+0000 7f1ac3b7f700 1 --2- 172.21.15.70:0/1977001384 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1abc101cf0 0x7f1abc1020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.661+0000 7f1ac3b7f700 1 --2- 172.21.15.70:0/1977001384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc1026a0 0x7f1abc108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.661+0000 7f1ac3b7f700 1 -- 172.21.15.70:0/1977001384 >> 172.21.15.70:0/1977001384 conn(0x7f1abc078a40 msgr2=0x7f1abc0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:55.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.661+0000 7f1ac3b7f700 1 -- 172.21.15.70:0/1977001384 shutdown_connections 2024-06-04T04:56:55.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.661+0000 7f1ac3b7f700 1 -- 172.21.15.70:0/1977001384 wait complete. 2024-06-04T04:56:55.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac3b7f700 1 Processor -- start 2024-06-04T04:56:55.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac3b7f700 1 -- start start 2024-06-04T04:56:55.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac3b7f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1abc101cf0 0x7f1abc19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac3b7f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc1026a0 0x7f1abc19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac3b7f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1abc108930 0x7f1abc1998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac3b7f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc1135b0 con 0x7f1abc101cf0 2024-06-04T04:56:55.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac3b7f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc113430 con 0x7f1abc1026a0 2024-06-04T04:56:55.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac3b7f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc113730 con 0x7f1abc108930 2024-06-04T04:56:55.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac191b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1abc101cf0 0x7f1abc19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:55.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac191b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1abc101cf0 0x7f1abc19f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44654/0 (socket says 172.21.15.70:44654) 2024-06-04T04:56:55.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac191b700 1 -- 172.21.15.70:0/3013492454 learned_addr learned my addr 172.21.15.70:0/3013492454 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:55.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac211c700 1 --2- 172.21.15.70:0/3013492454 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1abc108930 0x7f1abc1998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:55.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac111a700 1 --2- 172.21.15.70:0/3013492454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc1026a0 0x7f1abc19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:55.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac191b700 1 -- 172.21.15.70:0/3013492454 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1abc108930 msgr2=0x7f1abc1998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:55.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac191b700 1 --2- 172.21.15.70:0/3013492454 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1abc108930 0x7f1abc1998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac191b700 1 -- 172.21.15.70:0/3013492454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc1026a0 msgr2=0x7f1abc19fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:55.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac191b700 1 --2- 172.21.15.70:0/3013492454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc1026a0 0x7f1abc19fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.662+0000 7f1ac191b700 1 -- 172.21.15.70:0/3013492454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1abc19a050 con 0x7f1abc101cf0 2024-06-04T04:56:55.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.663+0000 7f1ac111a700 1 --2- 172.21.15.70:0/3013492454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc1026a0 0x7f1abc19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:56:55.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.663+0000 7f1ac191b700 1 --2- 172.21.15.70:0/3013492454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1abc101cf0 0x7f1abc19f730 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80060b0 tx=0x7f1ab800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:55.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.663+0000 7f1ab2ffd700 1 -- 172.21.15.70:0/3013492454 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab801a070 con 0x7f1abc101cf0 2024-06-04T04:56:55.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.663+0000 7f1ac3b7f700 1 -- 172.21.15.70:0/3013492454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1abc19a340 con 0x7f1abc101cf0 2024-06-04T04:56:55.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.663+0000 7f1ab2ffd700 1 -- 172.21.15.70:0/3013492454 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1ab800ddd0 con 0x7f1abc101cf0 2024-06-04T04:56:55.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.663+0000 7f1ab2ffd700 1 -- 172.21.15.70:0/3013492454 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8014690 con 0x7f1abc101cf0 2024-06-04T04:56:55.669 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:55 smithi070 ceph-mon[32631]: pgmap v389: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:56:55.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.663+0000 7f1ac3b7f700 1 -- 172.21.15.70:0/3013492454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1abc1a6600 con 0x7f1abc101cf0 2024-06-04T04:56:55.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.665+0000 7f1ab2ffd700 1 -- 172.21.15.70:0/3013492454 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f1ab801d070 con 0x7f1abc101cf0 2024-06-04T04:56:55.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.666+0000 7f1ab2ffd700 1 --2- 172.21.15.70:0/3013492454 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1aa80b2880 0x7f1aa80b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.666+0000 7f1ac111a700 1 --2- 172.21.15.70:0/3013492454 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1aa80b2880 0x7f1aa80b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:55.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.666+0000 7f1ab2ffd700 1 -- 172.21.15.70:0/3013492454 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f1ab80e1a80 con 0x7f1abc101cf0 2024-06-04T04:56:55.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.666+0000 7f1ac111a700 1 --2- 172.21.15.70:0/3013492454 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1aa80b2880 0x7f1aa80b4d40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1abc100f60 tx=0x7f1aac009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:55.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ace414b80 1 Processor -- start 2024-06-04T04:56:55.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ace414b80 1 -- start start 2024-06-04T04:56:55.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ace414b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cad42d0 0x55920cad46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ace414b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cace850 0x55920cacecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ace414b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cacf210 0x55920cadb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ace414b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55920ca2cd20 con 0x55920cad42d0 2024-06-04T04:56:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ace414b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55920ca1c8c0 con 0x55920cace850 2024-06-04T04:56:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ace414b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55920ca20900 con 0x55920cacf210 2024-06-04T04:56:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ac191b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cad42d0 0x55920cad46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ac191b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cad42d0 0x55920cad46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44668/0 (socket says 172.21.15.70:44668) 2024-06-04T04:56:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ac191b700 1 -- 172.21.15.70:0/29509537 learned_addr learned my addr 172.21.15.70:0/29509537 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ac211c700 1 --2- 172.21.15.70:0/29509537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cacf210 0x55920cadb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ac191b700 1 -- 172.21.15.70:0/29509537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cacf210 msgr2=0x55920cadb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ac111a700 1 --2- 172.21.15.70:0/29509537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cace850 0x55920cacecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:55.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ac191b700 1 --2- 172.21.15.70:0/29509537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cacf210 0x55920cadb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ac191b700 1 -- 172.21.15.70:0/29509537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cace850 msgr2=0x55920cacecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:55.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ac191b700 1 --2- 172.21.15.70:0/29509537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cace850 0x55920cacecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ac191b700 1 -- 172.21.15.70:0/29509537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55920c9c5f60 con 0x55920cad42d0 2024-06-04T04:56:55.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.668+0000 7f1ac211c700 1 --2- 172.21.15.70:0/29509537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cacf210 0x55920cadb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:56:55.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.669+0000 7f1ac191b700 1 --2- 172.21.15.70:0/29509537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cad42d0 0x55920cad46b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80a9010 tx=0x7f1ab80a88f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc67cefcd4f9fa1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:55.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.669+0000 7f1aa7fff700 1 -- 172.21.15.70:0/29509537 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab80ad430 con 0x55920cad42d0 2024-06-04T04:56:55.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.669+0000 7f1aa7fff700 1 -- 172.21.15.70:0/29509537 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1ab80b1ee0 con 0x55920cad42d0 2024-06-04T04:56:55.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.669+0000 7f1aa7fff700 1 -- 172.21.15.70:0/29509537 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab80b1a50 con 0x55920cad42d0 2024-06-04T04:56:55.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.670+0000 7f1ace414b80 1 -- 172.21.15.70:0/29509537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cad42d0 msgr2=0x55920cad46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.670+0000 7f1ace414b80 1 --2- 172.21.15.70:0/29509537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cad42d0 0x55920cad46b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80a9010 tx=0x7f1ab80a88f0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.670+0000 7f1ace414b80 1 -- 172.21.15.70:0/29509537 shutdown_connections 2024-06-04T04:56:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.670+0000 7f1ace414b80 1 --2- 172.21.15.70:0/29509537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cacf210 0x55920cadb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.670+0000 7f1ace414b80 1 --2- 172.21.15.70:0/29509537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cad42d0 0x55920cad46b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.670+0000 7f1ace414b80 1 --2- 172.21.15.70:0/29509537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cace850 0x55920cacecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.670+0000 7f1ace414b80 1 -- 172.21.15.70:0/29509537 >> 172.21.15.70:0/29509537 conn(0x55920c9fa1f0 msgr2=0x55920cac9390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.670+0000 7f1ace414b80 1 -- 172.21.15.70:0/29509537 shutdown_connections 2024-06-04T04:56:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.670+0000 7f1ace414b80 1 -- 172.21.15.70:0/29509537 wait complete. 2024-06-04T04:56:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.670+0000 7f1ace414b80 1 -- start start 2024-06-04T04:56:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.670+0000 7f1ace414b80 1 Processor -- start 2024-06-04T04:56:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ace414b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cace850 0x55920cadac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ace414b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cacf210 0x55920cad7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ace414b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cad42d0 0x55920cad84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ace414b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55920c922f50 con 0x55920cace850 2024-06-04T04:56:55.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ace414b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55920c984d50 con 0x55920cacf210 2024-06-04T04:56:55.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ace414b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55920cadb690 con 0x55920cad42d0 2024-06-04T04:56:55.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ace414b80 10 client.0 _ll_register_callbacks cb 0x55920c9b7900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:55.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ac191b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cace850 0x55920cadac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:55.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ac191b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cace850 0x55920cadac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44682/0 (socket says 172.21.15.70:44682) 2024-06-04T04:56:55.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ac191b700 1 -- 172.21.15.70:0/3129111168 learned_addr learned my addr 172.21.15.70:0/3129111168 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:55.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ac211c700 1 --2- 172.21.15.70:0/3129111168 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cad42d0 0x55920cad84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:55.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ac111a700 1 --2- 172.21.15.70:0/3129111168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cacf210 0x55920cad7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:55.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ac191b700 1 -- 172.21.15.70:0/3129111168 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cad42d0 msgr2=0x55920cad84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:55.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ac191b700 1 --2- 172.21.15.70:0/3129111168 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cad42d0 0x55920cad84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ac191b700 1 -- 172.21.15.70:0/3129111168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cacf210 msgr2=0x55920cad7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:55.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ac191b700 1 --2- 172.21.15.70:0/3129111168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cacf210 0x55920cad7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ac191b700 1 -- 172.21.15.70:0/3129111168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55920c96c880 con 0x55920cace850 2024-06-04T04:56:55.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ac191b700 1 --2- 172.21.15.70:0/3129111168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cace850 0x55920cadac80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80df5b0 tx=0x7f1ab8015c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd22b5212ebbf381 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:55.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1aa57fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:55.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1aa57fa700 1 -- 172.21.15.70:0/3129111168 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab80b1a50 con 0x55920cace850 2024-06-04T04:56:55.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1aa57fa700 1 -- 172.21.15.70:0/3129111168 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f1ab80b1bf0 con 0x55920cace850 2024-06-04T04:56:55.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.671+0000 7f1ace414b80 1 -- 172.21.15.70:0/3129111168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ab80b1ee0 con 0x55920cace850 2024-06-04T04:56:55.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.672+0000 7f1aa57fa700 1 -- 172.21.15.70:0/3129111168 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1ab8089070 con 0x55920cace850 2024-06-04T04:56:55.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.672+0000 7f1ace414b80 10 client.15153 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:55.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.672+0000 7f1ace414b80 1 -- 172.21.15.70:0/3129111168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55920c9d6710 con 0x55920cace850 2024-06-04T04:56:55.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.672+0000 7f1aa57fa700 1 -- 172.21.15.70:0/3129111168 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1ab80b1ee0 con 0x55920cace850 2024-06-04T04:56:55.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.672+0000 7f1aa57fa700 1 -- 172.21.15.70:0/3129111168 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1ab80ae5c0 con 0x55920cace850 2024-06-04T04:56:55.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.672+0000 7f1ace414b80 10 client.15153 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.672+0000 7f1ace414b80 10 client.15153 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-04T04:56:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.672+0000 7f1ace414b80 20 client.15153 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.672+0000 7f1ace414b80 1 --2- 172.21.15.70:0/3129111168 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55920caeb240 0x55920caed410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.672+0000 7f1ace414b80 4 client.15153 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:56:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.672+0000 7f1ace414b80 1 -- 172.21.15.70:0/3129111168 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55920ca2ec70 con 0x55920caeb240 2024-06-04T04:56:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.672+0000 7f1ac111a700 1 --2- 172.21.15.70:0/3129111168 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55920caeb240 0x55920caed410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.673+0000 7f1ac111a700 1 --2- 172.21.15.70:0/3129111168 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55920caeb240 0x55920caed410 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2e389be0d421e4c0 server_cookie=ad9bf09f7b37ebe0 in_seq=0 out_seq=0 2024-06-04T04:56:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.673+0000 7f1aa57fa700 10 client.15153 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T04:56:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.673+0000 7f1aa57fa700 1 -- 172.21.15.70:0/3129111168 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55920ca2ec70 con 0x55920caeb240 2024-06-04T04:56:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.673+0000 7f1aa57fa700 10 client.15153 handle_command_reply: tid=0 2024-06-04T04:56:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.673+0000 7f1ace414b80 1 client.15153 shutdown 2024-06-04T04:56:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.674+0000 7f1ace414b80 1 -- 172.21.15.70:0/3129111168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cace850 msgr2=0x55920cadac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.674+0000 7f1ace414b80 1 --2- 172.21.15.70:0/3129111168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cace850 0x55920cadac80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80df5b0 tx=0x7f1ab8015c30 comp rx=0 tx=0).stop 2024-06-04T04:56:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.674+0000 7f1ace414b80 1 -- 172.21.15.70:0/3129111168 shutdown_connections 2024-06-04T04:56:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.674+0000 7f1ace414b80 1 --2- 172.21.15.70:0/3129111168 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55920caeb240 0x55920caed410 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.674+0000 7f1ace414b80 1 --2- 172.21.15.70:0/3129111168 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cad42d0 0x55920cad84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.674+0000 7f1ace414b80 1 --2- 172.21.15.70:0/3129111168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cace850 0x55920cadac80 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.674+0000 7f1ace414b80 1 --2- 172.21.15.70:0/3129111168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cacf210 0x55920cad7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.674+0000 7f1aa57fa700 0 client.15153 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T04:56:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.674+0000 7f1ace414b80 1 -- 172.21.15.70:0/3129111168 >> 172.21.15.70:0/3129111168 conn(0x55920c9fa1f0 msgr2=0x55920cac8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.674+0000 7f1ace414b80 1 -- 172.21.15.70:0/3129111168 shutdown_connections 2024-06-04T04:56:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.674+0000 7f1ace414b80 1 -- 172.21.15.70:0/3129111168 wait complete. 2024-06-04T04:56:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.674+0000 7f1ace414b80 20 client.15153 trim_cache size 0 max 16384 2024-06-04T04:56:55.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.694+0000 7f1ace414b80 1 Processor -- start 2024-06-04T04:56:55.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.694+0000 7f1ace414b80 1 -- start start 2024-06-04T04:56:55.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.694+0000 7f1ace414b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cace850 0x55920caecd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.694+0000 7f1ace414b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cacf210 0x55920cac9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.695+0000 7f1ace414b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cad42d0 0x55920cad9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.695+0000 7f1ace414b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55920caeb750 con 0x55920cacf210 2024-06-04T04:56:55.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.695+0000 7f1ace414b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55920caebe20 con 0x55920cace850 2024-06-04T04:56:55.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.695+0000 7f1ace414b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55920caec4f0 con 0x55920cad42d0 2024-06-04T04:56:55.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.695+0000 7f1ac111a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cacf210 0x55920cac9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:55.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.695+0000 7f1ac111a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cacf210 0x55920cac9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44696/0 (socket says 172.21.15.70:44696) 2024-06-04T04:56:55.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.695+0000 7f1ac111a700 1 -- 172.21.15.70:0/1721327265 learned_addr learned my addr 172.21.15.70:0/1721327265 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:55.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.695+0000 7f1ac191b700 1 --2- 172.21.15.70:0/1721327265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cace850 0x55920caecd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:55.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.695+0000 7f1ac111a700 1 -- 172.21.15.70:0/1721327265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cad42d0 msgr2=0x55920cad9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:55.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.695+0000 7f1ac111a700 1 --2- 172.21.15.70:0/1721327265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cad42d0 0x55920cad9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.695+0000 7f1ac111a700 1 -- 172.21.15.70:0/1721327265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cace850 msgr2=0x55920caecd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:55.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.695+0000 7f1ac111a700 1 --2- 172.21.15.70:0/1721327265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cace850 0x55920caecd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.695+0000 7f1ac111a700 1 -- 172.21.15.70:0/1721327265 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55920caed780 con 0x55920cacf210 2024-06-04T04:56:55.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.695+0000 7f1ac111a700 1 --2- 172.21.15.70:0/1721327265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cacf210 0x55920cac9220 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009490 tx=0x7f1aac014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa63184a1616b65c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:55.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.695+0000 7f1aa77fe700 1 -- 172.21.15.70:0/1721327265 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aac00ee40 con 0x55920cacf210 2024-06-04T04:56:55.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.695+0000 7f1aa77fe700 1 -- 172.21.15.70:0/1721327265 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1aac017040 con 0x55920cacf210 2024-06-04T04:56:55.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.695+0000 7f1aa77fe700 1 -- 172.21.15.70:0/1721327265 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aac005310 con 0x55920cacf210 2024-06-04T04:56:55.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.696+0000 7f1ace414b80 1 -- 172.21.15.70:0/1721327265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cacf210 msgr2=0x55920cac9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:55.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.696+0000 7f1ace414b80 1 --2- 172.21.15.70:0/1721327265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cacf210 0x55920cac9220 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009490 tx=0x7f1aac014000 comp rx=0 tx=0).stop 2024-06-04T04:56:55.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.696+0000 7f1ace414b80 1 -- 172.21.15.70:0/1721327265 shutdown_connections 2024-06-04T04:56:55.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.696+0000 7f1ace414b80 1 --2- 172.21.15.70:0/1721327265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cad42d0 0x55920cad9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.696+0000 7f1ace414b80 1 --2- 172.21.15.70:0/1721327265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cace850 0x55920caecd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.696+0000 7f1ace414b80 1 --2- 172.21.15.70:0/1721327265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cacf210 0x55920cac9220 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.696+0000 7f1ace414b80 1 -- 172.21.15.70:0/1721327265 >> 172.21.15.70:0/1721327265 conn(0x55920c9fa1f0 msgr2=0x55920cad97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:55.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.696+0000 7f1ace414b80 1 -- 172.21.15.70:0/1721327265 shutdown_connections 2024-06-04T04:56:55.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.696+0000 7f1ace414b80 1 -- 172.21.15.70:0/1721327265 wait complete. 2024-06-04T04:56:55.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.696+0000 7f1ace414b80 1 -- start start 2024-06-04T04:56:55.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.696+0000 7f1ace414b80 1 Processor -- start 2024-06-04T04:56:55.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ace414b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cace850 0x55920caf6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ace414b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cacf210 0x55920caf0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ace414b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cad42d0 0x55920caf1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ace414b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55920caed290 con 0x55920cad42d0 2024-06-04T04:56:55.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ace414b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55920caecbc0 con 0x55920cacf210 2024-06-04T04:56:55.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ace414b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55920caed960 con 0x55920cace850 2024-06-04T04:56:55.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ace414b80 10 client.0 _ll_register_callbacks cb 0x55920c9b7900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:55.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ac211c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cad42d0 0x55920caf1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:55.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ac211c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cad42d0 0x55920caf1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44708/0 (socket says 172.21.15.70:44708) 2024-06-04T04:56:55.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ac211c700 1 -- 172.21.15.70:0/2924361194 learned_addr learned my addr 172.21.15.70:0/2924361194 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:55.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ac111a700 1 --2- 172.21.15.70:0/2924361194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cacf210 0x55920caf0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:55.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ac191b700 1 --2- 172.21.15.70:0/2924361194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cace850 0x55920caf6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:55.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ac211c700 1 -- 172.21.15.70:0/2924361194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cace850 msgr2=0x55920caf6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:55.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ac211c700 1 --2- 172.21.15.70:0/2924361194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cace850 0x55920caf6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ac211c700 1 -- 172.21.15.70:0/2924361194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cacf210 msgr2=0x55920caf0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:55.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ac211c700 1 --2- 172.21.15.70:0/2924361194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cacf210 0x55920caf0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ac211c700 1 -- 172.21.15.70:0/2924361194 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55920caf19d0 con 0x55920cad42d0 2024-06-04T04:56:55.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ac191b700 1 --2- 172.21.15.70:0/2924361194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cace850 0x55920caf6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:55.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1ac211c700 1 --2- 172.21.15.70:0/2924361194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cad42d0 0x55920caf1490 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40099c0 tx=0x7f1ab4005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0f32d3996a9e79 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:55.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1aa57fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:55.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.697+0000 7f1aa57fa700 1 -- 172.21.15.70:0/2924361194 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab400bba0 con 0x55920cad42d0 2024-06-04T04:56:55.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.698+0000 7f1aa57fa700 1 -- 172.21.15.70:0/2924361194 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f1ab400bd40 con 0x55920cad42d0 2024-06-04T04:56:55.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.698+0000 7f1ace414b80 1 -- 172.21.15.70:0/2924361194 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1aac017040 con 0x55920cad42d0 2024-06-04T04:56:55.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.698+0000 7f1aa57fa700 1 -- 172.21.15.70:0/2924361194 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1ab40095b0 con 0x55920cad42d0 2024-06-04T04:56:55.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.698+0000 7f1ace414b80 10 client.15159 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:55.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.698+0000 7f1ace414b80 1 -- 172.21.15.70:0/2924361194 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55920cb58400 con 0x55920cad42d0 2024-06-04T04:56:55.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.698+0000 7f1aa57fa700 1 -- 172.21.15.70:0/2924361194 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1aac017040 con 0x55920cad42d0 2024-06-04T04:56:55.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.698+0000 7f1aa57fa700 1 -- 172.21.15.70:0/2924361194 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1ab4024710 con 0x55920cad42d0 2024-06-04T04:56:55.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.698+0000 7f1ace414b80 10 client.15159 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:55.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.698+0000 7f1ace414b80 10 client.15159 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-04T04:56:55.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.698+0000 7f1ace414b80 20 client.15159 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:55.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.698+0000 7f1ace414b80 1 --2- 172.21.15.70:0/2924361194 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55920cb5d0a0 0x55920cb5bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:55.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.698+0000 7f1ace414b80 4 client.15159 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-04T04:56:55.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.698+0000 7f1ace414b80 1 -- 172.21.15.70:0/2924361194 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "status"}) -- 0x55920ca28450 con 0x55920cb5d0a0 2024-06-04T04:56:55.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.698+0000 7f1ac191b700 1 --2- 172.21.15.70:0/2924361194 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55920cb5d0a0 0x55920cb5bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:55.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.699+0000 7f1ac191b700 1 --2- 172.21.15.70:0/2924361194 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55920cb5d0a0 0x55920cb5bdb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b74d579690301e37 server_cookie=3bfed9f44133df04 in_seq=0 out_seq=0 2024-06-04T04:56:55.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.699+0000 7f1aa57fa700 10 client.15159 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T04:56:55.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.699+0000 7f1aa57fa700 1 -- 172.21.15.70:0/2924361194 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55920ca28450 con 0x55920cb5d0a0 2024-06-04T04:56:55.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.699+0000 7f1aa57fa700 10 client.15159 handle_command_reply: tid=0 2024-06-04T04:56:55.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.699+0000 7f1ace414b80 1 client.15159 shutdown 2024-06-04T04:56:55.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.699+0000 7f1ace414b80 1 -- 172.21.15.70:0/2924361194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cad42d0 msgr2=0x55920caf1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:55.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.699+0000 7f1ace414b80 1 --2- 172.21.15.70:0/2924361194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cad42d0 0x55920caf1490 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40099c0 tx=0x7f1ab4005f00 comp rx=0 tx=0).stop 2024-06-04T04:56:55.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.699+0000 7f1ace414b80 1 -- 172.21.15.70:0/2924361194 shutdown_connections 2024-06-04T04:56:55.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.699+0000 7f1ace414b80 1 --2- 172.21.15.70:0/2924361194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55920cad42d0 0x55920caf1490 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.699+0000 7f1ace414b80 1 --2- 172.21.15.70:0/2924361194 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55920cb5d0a0 0x55920cb5bdb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.699+0000 7f1ace414b80 1 --2- 172.21.15.70:0/2924361194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55920cace850 0x55920caf6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.699+0000 7f1ace414b80 1 --2- 172.21.15.70:0/2924361194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55920cacf210 0x55920caf0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.699+0000 7f1aa57fa700 0 client.15159 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T04:56:55.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.699+0000 7f1ace414b80 1 -- 172.21.15.70:0/2924361194 >> 172.21.15.70:0/2924361194 conn(0x55920c9fa1f0 msgr2=0x55920cad5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:55.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.699+0000 7f1ace414b80 1 -- 172.21.15.70:0/2924361194 shutdown_connections 2024-06-04T04:56:55.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.699+0000 7f1ace414b80 1 -- 172.21.15.70:0/2924361194 wait complete. 2024-06-04T04:56:55.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.699+0000 7f1ace414b80 20 client.15159 trim_cache size 0 max 16384 2024-06-04T04:56:55.708 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:56:55.708 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:56:55.708 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 2, 2024-06-04T04:56:55.708 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24440, 2024-06-04T04:56:55.708 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-06-04T04:56:55.708 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-06-04T04:56:55.708 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-06-04T04:56:55.708 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 184.279247158, 2024-06-04T04:56:55.708 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-06-04T04:56:55.708 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 81, 2024-06-04T04:56:55.708 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 79, 2024-06-04T04:56:55.708 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 212.06529297599999 2024-06-04T04:56:55.708 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:56:55.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.700+0000 7f1ac3b7f700 1 -- 172.21.15.70:0/3013492454 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1aa80b2880 msgr2=0x7f1aa80b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:55.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.700+0000 7f1ac3b7f700 1 --2- 172.21.15.70:0/3013492454 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1aa80b2880 0x7f1aa80b4d40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1abc100f60 tx=0x7f1aac009550 comp rx=0 tx=0).stop 2024-06-04T04:56:55.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.700+0000 7f1ac3b7f700 1 -- 172.21.15.70:0/3013492454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1abc101cf0 msgr2=0x7f1abc19f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:55.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.700+0000 7f1ac3b7f700 1 --2- 172.21.15.70:0/3013492454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1abc101cf0 0x7f1abc19f730 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80060b0 tx=0x7f1ab800d3b0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.700+0000 7f1ac3b7f700 1 -- 172.21.15.70:0/3013492454 shutdown_connections 2024-06-04T04:56:55.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.700+0000 7f1ac3b7f700 1 --2- 172.21.15.70:0/3013492454 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1aa80b2880 0x7f1aa80b4d40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.700+0000 7f1ac3b7f700 1 --2- 172.21.15.70:0/3013492454 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1abc108930 0x7f1abc1998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.700+0000 7f1ac3b7f700 1 --2- 172.21.15.70:0/3013492454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1abc101cf0 0x7f1abc19f730 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.700+0000 7f1ac3b7f700 1 --2- 172.21.15.70:0/3013492454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc1026a0 0x7f1abc19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:55.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.700+0000 7f1ac3b7f700 1 -- 172.21.15.70:0/3013492454 >> 172.21.15.70:0/3013492454 conn(0x7f1abc078a40 msgr2=0x7f1abc1070a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:55.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.700+0000 7f1ac3b7f700 1 -- 172.21.15.70:0/3013492454 shutdown_connections 2024-06-04T04:56:55.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:55.700+0000 7f1ac3b7f700 1 -- 172.21.15.70:0/3013492454 wait complete. 2024-06-04T04:56:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:55 smithi112 ceph-mon[36453]: pgmap v389: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:56:55.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:55 smithi098 ceph-mon[35569]: pgmap v389: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-04T04:56:56.452 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.c status 2024-06-04T04:56:56.693 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:57.831 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:57 smithi070 ceph-mon[32631]: pgmap v390: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:57.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:57 smithi112 ceph-mon[36453]: pgmap v390: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:57.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:57 smithi098 ceph-mon[35569]: pgmap v390: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-04T04:56:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.351+0000 7f1c0a19a700 1 -- 172.21.15.70:0/3513847651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c04108520 msgr2=0x7f1c04108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.351+0000 7f1c0a19a700 1 --2- 172.21.15.70:0/3513847651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c04108520 0x7f1c04108900 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8009990 tx=0x7f1bf8009ca0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.351+0000 7f1c0a19a700 1 -- 172.21.15.70:0/3513847651 shutdown_connections 2024-06-04T04:56:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.351+0000 7f1c0a19a700 1 --2- 172.21.15.70:0/3513847651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c04102e00 0x7f1c0410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.351+0000 7f1c0a19a700 1 --2- 172.21.15.70:0/3513847651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c04108520 0x7f1c04108900 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.351+0000 7f1c0a19a700 1 --2- 172.21.15.70:0/3513847651 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c04102440 0x7f1c041028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.351+0000 7f1c0a19a700 1 -- 172.21.15.70:0/3513847651 >> 172.21.15.70:0/3513847651 conn(0x7f1c040793f0 msgr2=0x7f1c04079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.351+0000 7f1c0a19a700 1 -- 172.21.15.70:0/3513847651 shutdown_connections 2024-06-04T04:56:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.351+0000 7f1c0a19a700 1 -- 172.21.15.70:0/3513847651 wait complete. 2024-06-04T04:56:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.352+0000 7f1c0a19a700 1 Processor -- start 2024-06-04T04:56:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.352+0000 7f1c0a19a700 1 -- start start 2024-06-04T04:56:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.352+0000 7f1c0a19a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c04102440 0x7f1c0419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.352+0000 7f1c0a19a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c04102e00 0x7f1c0419fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.352+0000 7f1c0a19a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c04108520 0x7f1c04199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.352+0000 7f1c0a19a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1c041124a0 con 0x7f1c04108520 2024-06-04T04:56:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.352+0000 7f1c0a19a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1c04112320 con 0x7f1c04102e00 2024-06-04T04:56:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.352+0000 7f1c0a19a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1c04112620 con 0x7f1c04102440 2024-06-04T04:56:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.352+0000 7f1c03fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c04108520 0x7f1c04199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.353+0000 7f1c03fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c04108520 0x7f1c04199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48706/0 (socket says 172.21.15.70:48706) 2024-06-04T04:56:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.353+0000 7f1c03fff700 1 -- 172.21.15.70:0/357463959 learned_addr learned my addr 172.21.15.70:0/357463959 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.353+0000 7f1c02ffd700 1 --2- 172.21.15.70:0/357463959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c04102e00 0x7f1c0419fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.353+0000 7f1c037fe700 1 --2- 172.21.15.70:0/357463959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c04102440 0x7f1c0419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.353+0000 7f1c03fff700 1 -- 172.21.15.70:0/357463959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c04102440 msgr2=0x7f1c0419f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.353+0000 7f1c03fff700 1 --2- 172.21.15.70:0/357463959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c04102440 0x7f1c0419f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.353+0000 7f1c03fff700 1 -- 172.21.15.70:0/357463959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c04102e00 msgr2=0x7f1c0419fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.353+0000 7f1c03fff700 1 --2- 172.21.15.70:0/357463959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c04102e00 0x7f1c0419fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.353+0000 7f1c03fff700 1 -- 172.21.15.70:0/357463959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0419a090 con 0x7f1c04108520 2024-06-04T04:56:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.353+0000 7f1c03fff700 1 --2- 172.21.15.70:0/357463959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c04108520 0x7f1c04199930 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc00ada0 tx=0x7f1bfc00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.353+0000 7f1c00ff9700 1 -- 172.21.15.70:0/357463959 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc017020 con 0x7f1c04108520 2024-06-04T04:56:58.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.353+0000 7f1c00ff9700 1 -- 172.21.15.70:0/357463959 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1bfc00ce10 con 0x7f1c04108520 2024-06-04T04:56:58.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.353+0000 7f1c0a19a700 1 -- 172.21.15.70:0/357463959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c0419a380 con 0x7f1c04108520 2024-06-04T04:56:58.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.353+0000 7f1c00ff9700 1 -- 172.21.15.70:0/357463959 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc005280 con 0x7f1c04108520 2024-06-04T04:56:58.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.353+0000 7f1c0a19a700 1 -- 172.21.15.70:0/357463959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c041a65f0 con 0x7f1c04108520 2024-06-04T04:56:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.356+0000 7f1c00ff9700 1 -- 172.21.15.70:0/357463959 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f1bfc00c980 con 0x7f1c04108520 2024-06-04T04:56:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.356+0000 7f1c00ff9700 1 --2- 172.21.15.70:0/357463959 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1bec0b2880 0x7f1bec0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.356+0000 7f1c037fe700 1 --2- 172.21.15.70:0/357463959 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1bec0b2880 0x7f1bec0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.356+0000 7f1c00ff9700 1 -- 172.21.15.70:0/357463959 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f1bfc0e1920 con 0x7f1c04108520 2024-06-04T04:56:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c14a2fb80 1 Processor -- start 2024-06-04T04:56:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c037fe700 1 --2- 172.21.15.70:0/357463959 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1bec0b2880 0x7f1bec0b4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8005850 tx=0x7f1bf801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c14a2fb80 1 -- start start 2024-06-04T04:56:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c14a2fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b211242d0 0x555b211246b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c14a2fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b2111e850 0x555b2111ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c14a2fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111f210 0x555b2112b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c14a2fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555b2107cd20 con 0x555b2111f210 2024-06-04T04:56:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c14a2fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555b2106c8c0 con 0x555b2111e850 2024-06-04T04:56:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c14a2fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555b21070900 con 0x555b211242d0 2024-06-04T04:56:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c03fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111f210 0x555b2112b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c03fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111f210 0x555b2112b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48722/0 (socket says 172.21.15.70:48722) 2024-06-04T04:56:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c03fff700 1 -- 172.21.15.70:0/1247043869 learned_addr learned my addr 172.21.15.70:0/1247043869 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c02ffd700 1 --2- 172.21.15.70:0/1247043869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b2111e850 0x555b2111ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c037fe700 1 --2- 172.21.15.70:0/1247043869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b211242d0 0x555b211246b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c03fff700 1 -- 172.21.15.70:0/1247043869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b211242d0 msgr2=0x555b211246b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c03fff700 1 --2- 172.21.15.70:0/1247043869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b211242d0 0x555b211246b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c03fff700 1 -- 172.21.15.70:0/1247043869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b2111e850 msgr2=0x555b2111ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c03fff700 1 --2- 172.21.15.70:0/1247043869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b2111e850 0x555b2111ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c03fff700 1 -- 172.21.15.70:0/1247043869 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b21015f60 con 0x555b2111f210 2024-06-04T04:56:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c02ffd700 1 --2- 172.21.15.70:0/1247043869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b2111e850 0x555b2111ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.357+0000 7f1c037fe700 1 --2- 172.21.15.70:0/1247043869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b211242d0 0x555b211246b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:56:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.358+0000 7f1c03fff700 1 --2- 172.21.15.70:0/1247043869 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111f210 0x555b2112b030 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc0a6220 tx=0x7f1bfc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d87e1e761e0ee050 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.358+0000 7f1be9ffb700 1 -- 172.21.15.70:0/1247043869 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc0a8ac0 con 0x555b2111f210 2024-06-04T04:56:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.358+0000 7f1be9ffb700 1 -- 172.21.15.70:0/1247043869 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1bfc0a96b0 con 0x555b2111f210 2024-06-04T04:56:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.358+0000 7f1be9ffb700 1 -- 172.21.15.70:0/1247043869 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc0a9470 con 0x555b2111f210 2024-06-04T04:56:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.358+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/1247043869 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111f210 msgr2=0x555b2112b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.358+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/1247043869 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111f210 0x555b2112b030 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc0a6220 tx=0x7f1bfc00cc70 comp rx=0 tx=0).stop 2024-06-04T04:56:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.358+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/1247043869 shutdown_connections 2024-06-04T04:56:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.358+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/1247043869 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111f210 0x555b2112b030 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.358+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/1247043869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b211242d0 0x555b211246b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.358+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/1247043869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b2111e850 0x555b2111ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.358+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/1247043869 >> 172.21.15.70:0/1247043869 conn(0x555b2104a1f0 msgr2=0x555b21119390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.359+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/1247043869 shutdown_connections 2024-06-04T04:56:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.359+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/1247043869 wait complete. 2024-06-04T04:56:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.359+0000 7f1c14a2fb80 1 -- start start 2024-06-04T04:56:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.359+0000 7f1c14a2fb80 1 Processor -- start 2024-06-04T04:56:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.359+0000 7f1c14a2fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111e850 0x555b2112acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.359+0000 7f1c14a2fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b2111f210 0x555b21127fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.359+0000 7f1c14a2fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b211242d0 0x555b21128500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.359+0000 7f1c14a2fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555b20f72f50 con 0x555b2111e850 2024-06-04T04:56:58.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.359+0000 7f1c14a2fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555b20fd4d50 con 0x555b2111f210 2024-06-04T04:56:58.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.359+0000 7f1c14a2fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555b2112b690 con 0x555b211242d0 2024-06-04T04:56:58.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.359+0000 7f1c037fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111e850 0x555b2112acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.359+0000 7f1c14a2fb80 10 client.0 _ll_register_callbacks cb 0x555b21007900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:58.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.359+0000 7f1c037fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111e850 0x555b2112acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48724/0 (socket says 172.21.15.70:48724) 2024-06-04T04:56:58.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.359+0000 7f1c037fe700 1 -- 172.21.15.70:0/1966826944 learned_addr learned my addr 172.21.15.70:0/1966826944 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:58.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.360+0000 7f1c03fff700 1 --2- 172.21.15.70:0/1966826944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b211242d0 0x555b21128500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.360+0000 7f1c02ffd700 1 --2- 172.21.15.70:0/1966826944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b2111f210 0x555b21127fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.360+0000 7f1c037fe700 1 -- 172.21.15.70:0/1966826944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b211242d0 msgr2=0x555b21128500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:58.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.360+0000 7f1c037fe700 1 --2- 172.21.15.70:0/1966826944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b211242d0 0x555b21128500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.360+0000 7f1c037fe700 1 -- 172.21.15.70:0/1966826944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b2111f210 msgr2=0x555b21127fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:58.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.360+0000 7f1c037fe700 1 --2- 172.21.15.70:0/1966826944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b2111f210 0x555b21127fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.360+0000 7f1c037fe700 1 -- 172.21.15.70:0/1966826944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b20fbc880 con 0x555b2111e850 2024-06-04T04:56:58.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.360+0000 7f1c037fe700 1 --2- 172.21.15.70:0/1966826944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111e850 0x555b2112acc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8000c00 tx=0x7f1bf8009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd909778495e97b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:58.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.360+0000 7f1be2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:58.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.360+0000 7f1be2ffd700 1 -- 172.21.15.70:0/1966826944 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8033060 con 0x555b2111e850 2024-06-04T04:56:58.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.360+0000 7f1be2ffd700 1 -- 172.21.15.70:0/1966826944 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f1bf8023820 con 0x555b2111e850 2024-06-04T04:56:58.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.360+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/1966826944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bfc0a96b0 con 0x555b2111e850 2024-06-04T04:56:58.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.360+0000 7f1be2ffd700 1 -- 172.21.15.70:0/1966826944 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1bf8023a20 con 0x555b2111e850 2024-06-04T04:56:58.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.360+0000 7f1c14a2fb80 10 client.15171 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:58.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.360+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/1966826944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555b21026710 con 0x555b2111e850 2024-06-04T04:56:58.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.360+0000 7f1be2ffd700 1 -- 172.21.15.70:0/1966826944 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1bfc0a96b0 con 0x555b2111e850 2024-06-04T04:56:58.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.361+0000 7f1be2ffd700 1 -- 172.21.15.70:0/1966826944 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1bf802ebb0 con 0x555b2111e850 2024-06-04T04:56:58.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.361+0000 7f1c14a2fb80 10 client.15171 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:58.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.361+0000 7f1c14a2fb80 10 client.15171 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-04T04:56:58.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.361+0000 7f1c14a2fb80 20 client.15171 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:58.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.361+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/1966826944 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x555b211381c0 0x555b21139630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.361+0000 7f1c14a2fb80 4 client.15171 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:56:58.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.361+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/1966826944 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b2107ec70 con 0x555b211381c0 2024-06-04T04:56:58.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.361+0000 7f1c02ffd700 1 --2- 172.21.15.70:0/1966826944 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x555b211381c0 0x555b21139630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.361+0000 7f1c02ffd700 1 --2- 172.21.15.70:0/1966826944 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x555b211381c0 0x555b21139630 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b51c515cd7d4e2d9 server_cookie=40af04bb42045800 in_seq=0 out_seq=0 2024-06-04T04:56:58.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.361+0000 7f1be2ffd700 10 client.15171 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T04:56:58.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.362+0000 7f1be2ffd700 1 -- 172.21.15.70:0/1966826944 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555b2107ec70 con 0x555b211381c0 2024-06-04T04:56:58.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.362+0000 7f1be2ffd700 10 client.15171 handle_command_reply: tid=0 2024-06-04T04:56:58.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.362+0000 7f1c14a2fb80 1 client.15171 shutdown 2024-06-04T04:56:58.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.362+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/1966826944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111e850 msgr2=0x555b2112acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:58.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.362+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/1966826944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111e850 0x555b2112acc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8000c00 tx=0x7f1bf8009ca0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.363+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/1966826944 shutdown_connections 2024-06-04T04:56:58.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.363+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/1966826944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b211242d0 0x555b21128500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.363+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/1966826944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111e850 0x555b2112acc0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.363+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/1966826944 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x555b211381c0 0x555b21139630 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.363+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/1966826944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b2111f210 0x555b21127fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.363+0000 7f1be2ffd700 0 client.15171 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T04:56:58.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.363+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/1966826944 >> 172.21.15.70:0/1966826944 conn(0x555b2104a1f0 msgr2=0x555b21118aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:58.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.363+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/1966826944 shutdown_connections 2024-06-04T04:56:58.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.363+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/1966826944 wait complete. 2024-06-04T04:56:58.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.363+0000 7f1c14a2fb80 20 client.15171 trim_cache size 0 max 16384 2024-06-04T04:56:58.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c14a2fb80 1 Processor -- start 2024-06-04T04:56:58.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c14a2fb80 1 -- start start 2024-06-04T04:56:58.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c14a2fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111e850 0x555b2111ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c14a2fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b2111f210 0x555b21123800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c14a2fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b211242d0 0x555b21124fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c14a2fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555b21137970 con 0x555b2111e850 2024-06-04T04:56:58.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c14a2fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555b21138040 con 0x555b211242d0 2024-06-04T04:56:58.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c14a2fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555b21138710 con 0x555b2111f210 2024-06-04T04:56:58.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c037fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111e850 0x555b2111ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c037fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111e850 0x555b2111ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48728/0 (socket says 172.21.15.70:48728) 2024-06-04T04:56:58.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c037fe700 1 -- 172.21.15.70:0/3078966103 learned_addr learned my addr 172.21.15.70:0/3078966103 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:58.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c02ffd700 1 --2- 172.21.15.70:0/3078966103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b2111f210 0x555b21123800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c03fff700 1 --2- 172.21.15.70:0/3078966103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b211242d0 0x555b21124fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c037fe700 1 -- 172.21.15.70:0/3078966103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b2111f210 msgr2=0x555b21123800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:58.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c037fe700 1 --2- 172.21.15.70:0/3078966103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b2111f210 0x555b21123800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c037fe700 1 -- 172.21.15.70:0/3078966103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b211242d0 msgr2=0x555b21124fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:58.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c037fe700 1 --2- 172.21.15.70:0/3078966103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b211242d0 0x555b21124fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c037fe700 1 -- 172.21.15.70:0/3078966103 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b21126550 con 0x555b2111e850 2024-06-04T04:56:58.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c03fff700 1 --2- 172.21.15.70:0/3078966103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b211242d0 0x555b21124fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:56:58.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.384+0000 7f1c02ffd700 1 --2- 172.21.15.70:0/3078966103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b2111f210 0x555b21123800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:56:58.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.385+0000 7f1c037fe700 1 --2- 172.21.15.70:0/3078966103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111e850 0x555b2111ec30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1bf802efd0 tx=0x7f1bf8039750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f67155e0bbb8c4f7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:58.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.385+0000 7f1be97fa700 1 -- 172.21.15.70:0/3078966103 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8033040 con 0x555b2111e850 2024-06-04T04:56:58.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.385+0000 7f1be97fa700 1 -- 172.21.15.70:0/3078966103 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1bf8023820 con 0x555b2111e850 2024-06-04T04:56:58.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.385+0000 7f1be97fa700 1 -- 172.21.15.70:0/3078966103 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8005290 con 0x555b2111e850 2024-06-04T04:56:58.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.386+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/3078966103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111e850 msgr2=0x555b2111ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:58.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.386+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/3078966103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111e850 0x555b2111ec30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1bf802efd0 tx=0x7f1bf8039750 comp rx=0 tx=0).stop 2024-06-04T04:56:58.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.386+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/3078966103 shutdown_connections 2024-06-04T04:56:58.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.386+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/3078966103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b211242d0 0x555b21124fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.386+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/3078966103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b2111e850 0x555b2111ec30 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.386+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/3078966103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b2111f210 0x555b21123800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.386+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/3078966103 >> 172.21.15.70:0/3078966103 conn(0x555b2104a1f0 msgr2=0x555b211230f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:58.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.386+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/3078966103 shutdown_connections 2024-06-04T04:56:58.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.386+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/3078966103 wait complete. 2024-06-04T04:56:58.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.386+0000 7f1c14a2fb80 1 -- start start 2024-06-04T04:56:58.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.386+0000 7f1c14a2fb80 1 Processor -- start 2024-06-04T04:56:58.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c14a2fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b2111e850 0x555b21142e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c14a2fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b2111f210 0x555b2113fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c14a2fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b211242d0 0x555b211403a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c14a2fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555b211394b0 con 0x555b211242d0 2024-06-04T04:56:58.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c14a2fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555b21138de0 con 0x555b2111e850 2024-06-04T04:56:58.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c14a2fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555b21139b80 con 0x555b2111f210 2024-06-04T04:56:58.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c14a2fb80 10 client.0 _ll_register_callbacks cb 0x555b21007900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:56:58.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c03fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b211242d0 0x555b211403a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c03fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b211242d0 0x555b211403a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48738/0 (socket says 172.21.15.70:48738) 2024-06-04T04:56:58.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c03fff700 1 -- 172.21.15.70:0/3683982431 learned_addr learned my addr 172.21.15.70:0/3683982431 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:56:58.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c037fe700 1 --2- 172.21.15.70:0/3683982431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b2111e850 0x555b21142e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c02ffd700 1 --2- 172.21.15.70:0/3683982431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b2111f210 0x555b2113fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c03fff700 1 -- 172.21.15.70:0/3683982431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b2111f210 msgr2=0x555b2113fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:58.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c03fff700 1 --2- 172.21.15.70:0/3683982431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b2111f210 0x555b2113fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c03fff700 1 -- 172.21.15.70:0/3683982431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b2111e850 msgr2=0x555b21142e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:56:58.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c03fff700 1 --2- 172.21.15.70:0/3683982431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b2111e850 0x555b21142e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c03fff700 1 -- 172.21.15.70:0/3683982431 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b21140910 con 0x555b211242d0 2024-06-04T04:56:58.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c037fe700 1 --2- 172.21.15.70:0/3683982431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b2111e850 0x555b21142e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:56:58.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1c03fff700 1 --2- 172.21.15.70:0/3683982431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b211242d0 0x555b211403a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc0a8f40 tx=0x7f1bfc0a94c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe532a9612eba0c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:56:58.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.387+0000 7f1be2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:56:58.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.388+0000 7f1be2ffd700 1 -- 172.21.15.70:0/3683982431 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc0a96b0 con 0x555b211242d0 2024-06-04T04:56:58.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.388+0000 7f1be2ffd700 1 -- 172.21.15.70:0/3683982431 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f1bfc0a55a0 con 0x555b211242d0 2024-06-04T04:56:58.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.388+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/3683982431 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bf8023820 con 0x555b211242d0 2024-06-04T04:56:58.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.388+0000 7f1be2ffd700 1 -- 172.21.15.70:0/3683982431 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1bfc0ad360 con 0x555b211242d0 2024-06-04T04:56:58.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.388+0000 7f1c14a2fb80 10 client.15177 fetch_fsmap learned FSMap version 28 2024-06-04T04:56:58.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.388+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/3683982431 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555b211a8670 con 0x555b211242d0 2024-06-04T04:56:58.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.388+0000 7f1be2ffd700 1 -- 172.21.15.70:0/3683982431 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1bf8023820 con 0x555b211242d0 2024-06-04T04:56:58.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.388+0000 7f1be2ffd700 1 -- 172.21.15.70:0/3683982431 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1bfc012070 con 0x555b211242d0 2024-06-04T04:56:58.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.388+0000 7f1c14a2fb80 10 client.15177 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:56:58.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.388+0000 7f1c14a2fb80 10 client.15177 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-04T04:56:58.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.388+0000 7f1c14a2fb80 20 client.15177 populate_metadata read hostname 'smithi070' 2024-06-04T04:56:58.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.388+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/3683982431 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x555b211ace00 0x555b211abb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:56:58.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.388+0000 7f1c14a2fb80 4 client.15177 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-04T04:56:58.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.388+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/3683982431 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "status"}) -- 0x555b21078450 con 0x555b211ace00 2024-06-04T04:56:58.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.389+0000 7f1c037fe700 1 --2- 172.21.15.70:0/3683982431 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x555b211ace00 0x555b211abb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:56:58.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.389+0000 7f1c037fe700 1 --2- 172.21.15.70:0/3683982431 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x555b211ace00 0x555b211abb10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7b0e62656e84284f server_cookie=9f85154fb662884c in_seq=0 out_seq=0 2024-06-04T04:56:58.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.389+0000 7f1be2ffd700 10 client.15177 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T04:56:58.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.389+0000 7f1be2ffd700 1 -- 172.21.15.70:0/3683982431 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x555b21078450 con 0x555b211ace00 2024-06-04T04:56:58.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.389+0000 7f1be2ffd700 10 client.15177 handle_command_reply: tid=0 2024-06-04T04:56:58.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.389+0000 7f1c14a2fb80 1 client.15177 shutdown 2024-06-04T04:56:58.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.389+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/3683982431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b211242d0 msgr2=0x555b211403a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:58.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.389+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/3683982431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b211242d0 0x555b211403a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc0a8f40 tx=0x7f1bfc0a94c0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.389+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/3683982431 shutdown_connections 2024-06-04T04:56:58.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.389+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/3683982431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b211242d0 0x555b211403a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.389+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/3683982431 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x555b211ace00 0x555b211abb10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.389+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/3683982431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b2111e850 0x555b21142e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.389+0000 7f1be2ffd700 0 client.15177 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T04:56:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.389+0000 7f1c14a2fb80 1 --2- 172.21.15.70:0/3683982431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b2111f210 0x555b2113fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.389+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/3683982431 >> 172.21.15.70:0/3683982431 conn(0x555b2104a1f0 msgr2=0x555b21121c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.390+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/3683982431 shutdown_connections 2024-06-04T04:56:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.390+0000 7f1c14a2fb80 1 -- 172.21.15.70:0/3683982431 wait complete. 2024-06-04T04:56:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.390+0000 7f1c14a2fb80 20 client.15177 trim_cache size 0 max 16384 2024-06-04T04:56:58.398 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:56:58.399 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:56:58.399 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 1, 2024-06-04T04:56:58.399 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24442, 2024-06-04T04:56:58.399 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-06-04T04:56:58.399 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-06-04T04:56:58.399 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-06-04T04:56:58.399 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 187.979071017, 2024-06-04T04:56:58.399 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-06-04T04:56:58.399 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 81, 2024-06-04T04:56:58.399 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 79, 2024-06-04T04:56:58.399 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 231.37944702300001 2024-06-04T04:56:58.399 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:56:58.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.390+0000 7f1c0a19a700 1 -- 172.21.15.70:0/357463959 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1bec0b2880 msgr2=0x7f1bec0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:58.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.390+0000 7f1c0a19a700 1 --2- 172.21.15.70:0/357463959 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1bec0b2880 0x7f1bec0b4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8005850 tx=0x7f1bf801a040 comp rx=0 tx=0).stop 2024-06-04T04:56:58.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.390+0000 7f1c0a19a700 1 -- 172.21.15.70:0/357463959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c04108520 msgr2=0x7f1c04199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:56:58.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.390+0000 7f1c0a19a700 1 --2- 172.21.15.70:0/357463959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c04108520 0x7f1c04199930 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc00ada0 tx=0x7f1bfc00c330 comp rx=0 tx=0).stop 2024-06-04T04:56:58.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.391+0000 7f1c0a19a700 1 -- 172.21.15.70:0/357463959 shutdown_connections 2024-06-04T04:56:58.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.391+0000 7f1c0a19a700 1 --2- 172.21.15.70:0/357463959 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f1bec0b2880 0x7f1bec0b4d40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.391+0000 7f1c0a19a700 1 --2- 172.21.15.70:0/357463959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c04108520 0x7f1c04199930 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.391+0000 7f1c0a19a700 1 --2- 172.21.15.70:0/357463959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c04102440 0x7f1c0419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.391+0000 7f1c0a19a700 1 --2- 172.21.15.70:0/357463959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c04102e00 0x7f1c0419fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:56:58.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.391+0000 7f1c0a19a700 1 -- 172.21.15.70:0/357463959 >> 172.21.15.70:0/357463959 conn(0x7f1c040793f0 msgr2=0x7f1c0406b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:56:58.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.391+0000 7f1c0a19a700 1 -- 172.21.15.70:0/357463959 shutdown_connections 2024-06-04T04:56:58.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:56:58.391+0000 7f1c0a19a700 1 -- 172.21.15.70:0/357463959 wait complete. 2024-06-04T04:56:59.086 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-06-04T04:56:59.327 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:56:59.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:56:59 smithi070 ceph-mon[32631]: pgmap v391: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:56:59.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:56:59 smithi112 ceph-mon[36453]: pgmap v391: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:56:59.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:56:59 smithi098 ceph-mon[35569]: pgmap v391: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-06-04T04:57:01.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.026+0000 7f3f41b73700 1 -- 172.21.15.70:0/1925124380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f3c102440 msgr2=0x7f3f3c1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:01.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.026+0000 7f3f41b73700 1 --2- 172.21.15.70:0/1925124380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f3c102440 0x7f3f3c1028c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3f30009990 tx=0x7f3f30009ca0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.026+0000 7f3f41b73700 1 -- 172.21.15.70:0/1925124380 shutdown_connections 2024-06-04T04:57:01.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.026+0000 7f3f41b73700 1 --2- 172.21.15.70:0/1925124380 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f3c102e00 0x7f3f3c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.027+0000 7f3f41b73700 1 --2- 172.21.15.70:0/1925124380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f3c108520 0x7f3f3c108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.027+0000 7f3f41b73700 1 --2- 172.21.15.70:0/1925124380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f3c102440 0x7f3f3c1028c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.027+0000 7f3f41b73700 1 -- 172.21.15.70:0/1925124380 >> 172.21.15.70:0/1925124380 conn(0x7f3f3c0793f0 msgr2=0x7f3f3c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:01.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.027+0000 7f3f41b73700 1 -- 172.21.15.70:0/1925124380 shutdown_connections 2024-06-04T04:57:01.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.027+0000 7f3f41b73700 1 -- 172.21.15.70:0/1925124380 wait complete. 2024-06-04T04:57:01.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.027+0000 7f3f41b73700 1 Processor -- start 2024-06-04T04:57:01.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.027+0000 7f3f41b73700 1 -- start start 2024-06-04T04:57:01.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.027+0000 7f3f41b73700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f3c102440 0x7f3f3c19a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.027+0000 7f3f41b73700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f3c102e00 0x7f3f3c19aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f41b73700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f3c108520 0x7f3f3c1a5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f41b73700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c1124f0 con 0x7f3f3c102e00 2024-06-04T04:57:01.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f41b73700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c112370 con 0x7f3f3c102440 2024-06-04T04:57:01.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f41b73700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c112670 con 0x7f3f3c108520 2024-06-04T04:57:01.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f3affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f3c102e00 0x7f3f3c19aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f3affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f3c102e00 0x7f3f3c19aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48746/0 (socket says 172.21.15.70:48746) 2024-06-04T04:57:01.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f3affd700 1 -- 172.21.15.70:0/174662903 learned_addr learned my addr 172.21.15.70:0/174662903 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:01.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f3bfff700 1 --2- 172.21.15.70:0/174662903 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f3c108520 0x7f3f3c1a5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f3b7fe700 1 --2- 172.21.15.70:0/174662903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f3c102440 0x7f3f3c19a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f3affd700 1 -- 172.21.15.70:0/174662903 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f3c108520 msgr2=0x7f3f3c1a5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:01.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f3affd700 1 --2- 172.21.15.70:0/174662903 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f3c108520 0x7f3f3c1a5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f3affd700 1 -- 172.21.15.70:0/174662903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f3c102440 msgr2=0x7f3f3c19a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:01.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f3affd700 1 --2- 172.21.15.70:0/174662903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f3c102440 0x7f3f3c19a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f3affd700 1 -- 172.21.15.70:0/174662903 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3c1a5f70 con 0x7f3f3c102e00 2024-06-04T04:57:01.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f3b7fe700 1 --2- 172.21.15.70:0/174662903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f3c102440 0x7f3f3c19a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:01.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f3affd700 1 --2- 172.21.15.70:0/174662903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f3c102e00 0x7f3f3c19aef0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3f30005910 tx=0x7f3f300047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:01.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f38ff9700 1 -- 172.21.15.70:0/174662903 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f30027070 con 0x7f3f3c102e00 2024-06-04T04:57:01.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f41b73700 1 -- 172.21.15.70:0/174662903 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f3c1a6140 con 0x7f3f3c102e00 2024-06-04T04:57:01.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f38ff9700 1 -- 172.21.15.70:0/174662903 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3f3000fc80 con 0x7f3f3c102e00 2024-06-04T04:57:01.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f38ff9700 1 -- 172.21.15.70:0/174662903 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f300173f0 con 0x7f3f3c102e00 2024-06-04T04:57:01.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.028+0000 7f3f41b73700 1 -- 172.21.15.70:0/174662903 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f3c1a65f0 con 0x7f3f3c102e00 2024-06-04T04:57:01.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.032+0000 7f3f38ff9700 1 -- 172.21.15.70:0/174662903 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f3f3002a070 con 0x7f3f3c102e00 2024-06-04T04:57:01.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.032+0000 7f3f38ff9700 1 --2- 172.21.15.70:0/174662903 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3f240b2880 0x7f3f240b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.032+0000 7f3f4c408b80 1 Processor -- start 2024-06-04T04:57:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f3b7fe700 1 --2- 172.21.15.70:0/174662903 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3f240b2880 0x7f3f240b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f4c408b80 1 -- start start 2024-06-04T04:57:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f38ff9700 1 -- 172.21.15.70:0/174662903 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f3f300e32a0 con 0x7f3f3c102e00 2024-06-04T04:57:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f4c408b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc2cc20 0x55d96cc2d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f4c408b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2d5d0 0x55d96cc30d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f4c408b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 0x55d96cc31760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f4c408b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d96cb87c40 con 0x55d96cc31320 2024-06-04T04:57:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f4c408b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d96cb777e0 con 0x55d96cc2d5d0 2024-06-04T04:57:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f4c408b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d96cb7b820 con 0x55d96cc2cc20 2024-06-04T04:57:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f3bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 0x55d96cc31760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f3b7fe700 1 --2- 172.21.15.70:0/174662903 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3f240b2880 0x7f3f240b4d40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00baa0 tx=0x7f3f2c005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f3bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 0x55d96cc31760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48756/0 (socket says 172.21.15.70:48756) 2024-06-04T04:57:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f3affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2d5d0 0x55d96cc30d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f3bfff700 1 -- 172.21.15.70:0/680551149 learned_addr learned my addr 172.21.15.70:0/680551149 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:01.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f3b7fe700 1 --2- 172.21.15.70:0/680551149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc2cc20 0x55d96cc2d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f3bfff700 1 -- 172.21.15.70:0/680551149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc2cc20 msgr2=0x55d96cc2d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:01.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f3bfff700 1 --2- 172.21.15.70:0/680551149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc2cc20 0x55d96cc2d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f3bfff700 1 -- 172.21.15.70:0/680551149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2d5d0 msgr2=0x55d96cc30d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:01.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f3bfff700 1 --2- 172.21.15.70:0/680551149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2d5d0 0x55d96cc30d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f3bfff700 1 -- 172.21.15.70:0/680551149 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d96cac7880 con 0x55d96cc31320 2024-06-04T04:57:01.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.033+0000 7f3f3bfff700 1 --2- 172.21.15.70:0/680551149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 0x55d96cc31760 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400a730 tx=0x7f3f340078d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c6a770c91dae3a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:01.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.034+0000 7f3f21ffb700 1 -- 172.21.15.70:0/680551149 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34003a00 con 0x55d96cc31320 2024-06-04T04:57:01.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.034+0000 7f3f21ffb700 1 -- 172.21.15.70:0/680551149 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3f34003ba0 con 0x55d96cc31320 2024-06-04T04:57:01.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.034+0000 7f3f21ffb700 1 -- 172.21.15.70:0/680551149 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34014660 con 0x55d96cc31320 2024-06-04T04:57:01.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.034+0000 7f3f4c408b80 1 -- 172.21.15.70:0/680551149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 msgr2=0x55d96cc31760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:01.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.034+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/680551149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 0x55d96cc31760 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400a730 tx=0x7f3f340078d0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.034+0000 7f3f4c408b80 1 -- 172.21.15.70:0/680551149 shutdown_connections 2024-06-04T04:57:01.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.034+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/680551149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 0x55d96cc31760 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.034+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/680551149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc2cc20 0x55d96cc2d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.034+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/680551149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2d5d0 0x55d96cc30d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.034+0000 7f3f4c408b80 1 -- 172.21.15.70:0/680551149 >> 172.21.15.70:0/680551149 conn(0x55d96cb56030 msgr2=0x55d96cc22e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:01.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.034+0000 7f3f4c408b80 1 -- 172.21.15.70:0/680551149 shutdown_connections 2024-06-04T04:57:01.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.034+0000 7f3f4c408b80 1 -- 172.21.15.70:0/680551149 wait complete. 2024-06-04T04:57:01.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.035+0000 7f3f4c408b80 1 -- start start 2024-06-04T04:57:01.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.035+0000 7f3f4c408b80 1 Processor -- start 2024-06-04T04:57:01.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.035+0000 7f3f4c408b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2cc20 0x55d96cc386e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.035+0000 7f3f4c408b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc2d5d0 0x55d96cc359e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.035+0000 7f3f4c408b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 0x55d96cc35f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.035+0000 7f3f4c408b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d96ca7df50 con 0x55d96cc31320 2024-06-04T04:57:01.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.035+0000 7f3f4c408b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d96cb19770 con 0x55d96cc2cc20 2024-06-04T04:57:01.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.035+0000 7f3f4c408b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d96cc39480 con 0x55d96cc2d5d0 2024-06-04T04:57:01.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.035+0000 7f3f4c408b80 10 client.0 _ll_register_callbacks cb 0x55d96cb128d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:01.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.035+0000 7f3f3bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 0x55d96cc35f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.035+0000 7f3f3bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 0x55d96cc35f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48766/0 (socket says 172.21.15.70:48766) 2024-06-04T04:57:01.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.035+0000 7f3f3affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc2d5d0 0x55d96cc359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.035+0000 7f3f3bfff700 1 -- 172.21.15.70:0/928396819 learned_addr learned my addr 172.21.15.70:0/928396819 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:01.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.036+0000 7f3f3b7fe700 1 --2- 172.21.15.70:0/928396819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2cc20 0x55d96cc386e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.036+0000 7f3f3bfff700 1 -- 172.21.15.70:0/928396819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc2d5d0 msgr2=0x55d96cc359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:01.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.036+0000 7f3f3bfff700 1 --2- 172.21.15.70:0/928396819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc2d5d0 0x55d96cc359e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.036+0000 7f3f3bfff700 1 -- 172.21.15.70:0/928396819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2cc20 msgr2=0x55d96cc386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:01.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.036+0000 7f3f3bfff700 1 --2- 172.21.15.70:0/928396819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2cc20 0x55d96cc386e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.036+0000 7f3f3bfff700 1 -- 172.21.15.70:0/928396819 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d96cb16640 con 0x55d96cc31320 2024-06-04T04:57:01.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.036+0000 7f3f3b7fe700 1 --2- 172.21.15.70:0/928396819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2cc20 0x55d96cc386e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:01.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.036+0000 7f3f3affd700 1 --2- 172.21.15.70:0/928396819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc2d5d0 0x55d96cc359e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:57:01.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.036+0000 7f3f3bfff700 1 --2- 172.21.15.70:0/928396819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 0x55d96cc35f20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3f34007b80 tx=0x7f3f34004450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17100edde2403f7b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:01.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.036+0000 7f3f1affd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:57:01.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.036+0000 7f3f1affd700 1 -- 172.21.15.70:0/928396819 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34022070 con 0x55d96cc31320 2024-06-04T04:57:01.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.036+0000 7f3f1affd700 1 -- 172.21.15.70:0/928396819 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f3f34004ad0 con 0x55d96cc31320 2024-06-04T04:57:01.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.036+0000 7f3f4c408b80 1 -- 172.21.15.70:0/928396819 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f34003ba0 con 0x55d96cc31320 2024-06-04T04:57:01.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.036+0000 7f3f1affd700 1 -- 172.21.15.70:0/928396819 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3f34003a00 con 0x55d96cc31320 2024-06-04T04:57:01.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.036+0000 7f3f4c408b80 10 client.15189 fetch_fsmap learned FSMap version 28 2024-06-04T04:57:01.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.036+0000 7f3f4c408b80 1 -- 172.21.15.70:0/928396819 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d96cc43b60 con 0x55d96cc31320 2024-06-04T04:57:01.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.036+0000 7f3f1affd700 1 -- 172.21.15.70:0/928396819 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3f34003ba0 con 0x55d96cc31320 2024-06-04T04:57:01.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.037+0000 7f3f1affd700 1 -- 172.21.15.70:0/928396819 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3f3401eb30 con 0x55d96cc31320 2024-06-04T04:57:01.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.037+0000 7f3f4c408b80 10 client.15189 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:57:01.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.037+0000 7f3f4c408b80 10 client.15189 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-04T04:57:01.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.037+0000 7f3f4c408b80 20 client.15189 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:01.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.037+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/928396819 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d96cc45dd0 0x55d96cc47240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.037+0000 7f3f4c408b80 4 client.15189 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:57:01.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.037+0000 7f3f4c408b80 1 -- 172.21.15.70:0/928396819 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d96cbf4990 con 0x55d96cc45dd0 2024-06-04T04:57:01.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.037+0000 7f3f3b7fe700 1 --2- 172.21.15.70:0/928396819 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d96cc45dd0 0x55d96cc47240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.037+0000 7f3f1affd700 10 client.15189 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:01.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.037+0000 7f3f3b7fe700 1 --2- 172.21.15.70:0/928396819 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d96cc45dd0 0x55d96cc47240 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fb6a0183eb849c5 server_cookie=c1f5a34a439368d5 in_seq=0 out_seq=0 2024-06-04T04:57:01.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.038+0000 7f3f1affd700 1 -- 172.21.15.70:0/928396819 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d96cbf4990 con 0x55d96cc45dd0 2024-06-04T04:57:01.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.038+0000 7f3f1affd700 10 client.15189 handle_command_reply: tid=0 2024-06-04T04:57:01.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.038+0000 7f3f4c408b80 1 client.15189 shutdown 2024-06-04T04:57:01.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.038+0000 7f3f4c408b80 1 -- 172.21.15.70:0/928396819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 msgr2=0x55d96cc35f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:01.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.038+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/928396819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 0x55d96cc35f20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3f34007b80 tx=0x7f3f34004450 comp rx=0 tx=0).stop 2024-06-04T04:57:01.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.038+0000 7f3f4c408b80 1 -- 172.21.15.70:0/928396819 shutdown_connections 2024-06-04T04:57:01.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.038+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/928396819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 0x55d96cc35f20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.038+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/928396819 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d96cc45dd0 0x55d96cc47240 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.038+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/928396819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2cc20 0x55d96cc386e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.038+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/928396819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc2d5d0 0x55d96cc359e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.039+0000 7f3f4c408b80 1 -- 172.21.15.70:0/928396819 >> 172.21.15.70:0/928396819 conn(0x55d96cb56030 msgr2=0x55d96cc224c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:01.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.039+0000 7f3f1affd700 0 client.15189 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:01.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.039+0000 7f3f4c408b80 1 -- 172.21.15.70:0/928396819 shutdown_connections 2024-06-04T04:57:01.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.039+0000 7f3f4c408b80 1 -- 172.21.15.70:0/928396819 wait complete. 2024-06-04T04:57:01.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.039+0000 7f3f4c408b80 20 client.15189 trim_cache size 0 max 16384 2024-06-04T04:57:01.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.060+0000 7f3f4c408b80 1 Processor -- start 2024-06-04T04:57:01.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.060+0000 7f3f4c408b80 1 -- start start 2024-06-04T04:57:01.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.060+0000 7f3f4c408b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc2cc20 0x55d96cc302e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.060+0000 7f3f4c408b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2d5d0 0x55d96cc34b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.060+0000 7f3f4c408b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc31320 0x55d96cc37920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.060+0000 7f3f4c408b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d96cc45580 con 0x55d96cc2cc20 2024-06-04T04:57:01.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.060+0000 7f3f4c408b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d96cc45c50 con 0x55d96cc2d5d0 2024-06-04T04:57:01.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.060+0000 7f3f4c408b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d96cc46320 con 0x55d96cc31320 2024-06-04T04:57:01.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.060+0000 7f3f3b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc2cc20 0x55d96cc302e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.060+0000 7f3f3b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc2cc20 0x55d96cc302e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48770/0 (socket says 172.21.15.70:48770) 2024-06-04T04:57:01.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.060+0000 7f3f3b7fe700 1 -- 172.21.15.70:0/1473110566 learned_addr learned my addr 172.21.15.70:0/1473110566 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:01.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.060+0000 7f3f3bfff700 1 --2- 172.21.15.70:0/1473110566 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc31320 0x55d96cc37920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.061+0000 7f3f3b7fe700 1 -- 172.21.15.70:0/1473110566 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc31320 msgr2=0x55d96cc37920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:01.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.061+0000 7f3f3b7fe700 1 --2- 172.21.15.70:0/1473110566 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc31320 0x55d96cc37920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.061+0000 7f3f3b7fe700 1 -- 172.21.15.70:0/1473110566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2d5d0 msgr2=0x55d96cc34b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:01.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.061+0000 7f3f3b7fe700 1 --2- 172.21.15.70:0/1473110566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2d5d0 0x55d96cc34b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.061+0000 7f3f3b7fe700 1 -- 172.21.15.70:0/1473110566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d96cc238f0 con 0x55d96cc2cc20 2024-06-04T04:57:01.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.061+0000 7f3f3b7fe700 1 --2- 172.21.15.70:0/1473110566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc2cc20 0x55d96cc302e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00de70 tx=0x7f3f2c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1894347ef51477 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:01.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.061+0000 7f3f217fa700 1 -- 172.21.15.70:0/1473110566 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c01ead0 con 0x55d96cc2cc20 2024-06-04T04:57:01.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.061+0000 7f3f217fa700 1 -- 172.21.15.70:0/1473110566 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3f2c004cc0 con 0x55d96cc2cc20 2024-06-04T04:57:01.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.061+0000 7f3f217fa700 1 -- 172.21.15.70:0/1473110566 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c005570 con 0x55d96cc2cc20 2024-06-04T04:57:01.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.062+0000 7f3f4c408b80 1 -- 172.21.15.70:0/1473110566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc2cc20 msgr2=0x55d96cc302e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:01.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.062+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/1473110566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc2cc20 0x55d96cc302e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00de70 tx=0x7f3f2c013000 comp rx=0 tx=0).stop 2024-06-04T04:57:01.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.062+0000 7f3f4c408b80 1 -- 172.21.15.70:0/1473110566 shutdown_connections 2024-06-04T04:57:01.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.062+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/1473110566 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc31320 0x55d96cc37920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.062+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/1473110566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc2cc20 0x55d96cc302e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.062+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/1473110566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2d5d0 0x55d96cc34b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.062+0000 7f3f4c408b80 1 -- 172.21.15.70:0/1473110566 >> 172.21.15.70:0/1473110566 conn(0x55d96cb56030 msgr2=0x55d96cc37210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.062+0000 7f3f4c408b80 1 -- 172.21.15.70:0/1473110566 shutdown_connections 2024-06-04T04:57:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.062+0000 7f3f4c408b80 1 -- 172.21.15.70:0/1473110566 wait complete. 2024-06-04T04:57:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.062+0000 7f3f4c408b80 1 -- start start 2024-06-04T04:57:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.062+0000 7f3f4c408b80 1 Processor -- start 2024-06-04T04:57:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f4c408b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2cc20 0x55d96cc4c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f4c408b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc2d5d0 0x55d96cc4cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f4c408b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 0x55d96cc4d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f4c408b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d96cc470c0 con 0x55d96cc31320 2024-06-04T04:57:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f4c408b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d96cc469f0 con 0x55d96cc2cc20 2024-06-04T04:57:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f4c408b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d96cc47790 con 0x55d96cc2d5d0 2024-06-04T04:57:01.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f3bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 0x55d96cc4d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f3bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 0x55d96cc4d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48784/0 (socket says 172.21.15.70:48784) 2024-06-04T04:57:01.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f3bfff700 1 -- 172.21.15.70:0/4125587354 learned_addr learned my addr 172.21.15.70:0/4125587354 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:01.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f4c408b80 10 client.0 _ll_register_callbacks cb 0x55d96cb128d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:01.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f3affd700 1 --2- 172.21.15.70:0/4125587354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc2d5d0 0x55d96cc4cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f3b7fe700 1 --2- 172.21.15.70:0/4125587354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2cc20 0x55d96cc4c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f3affd700 1 -- 172.21.15.70:0/4125587354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2cc20 msgr2=0x55d96cc4c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:01.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f3affd700 1 --2- 172.21.15.70:0/4125587354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2cc20 0x55d96cc4c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f3affd700 1 -- 172.21.15.70:0/4125587354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 msgr2=0x55d96cc4d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:01.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f3affd700 1 --2- 172.21.15.70:0/4125587354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 0x55d96cc4d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f3affd700 1 -- 172.21.15.70:0/4125587354 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d96ccb0840 con 0x55d96cc2d5d0 2024-06-04T04:57:01.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f3affd700 1 --2- 172.21.15.70:0/4125587354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc2d5d0 0x55d96cc4cbf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000fc50 tx=0x7f3f300cd330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d332cd9c0bffba0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:01.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.063+0000 7f3f1affd700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:57:01.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.064+0000 7f3f1affd700 1 -- 172.21.15.70:0/4125587354 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f300aa0a0 con 0x55d96cc2d5d0 2024-06-04T04:57:01.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.064+0000 7f3f1affd700 1 -- 172.21.15.70:0/4125587354 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f3f300aada0 con 0x55d96cc2d5d0 2024-06-04T04:57:01.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.064+0000 7f3f4c408b80 1 -- 172.21.15.70:0/4125587354 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f2c004cc0 con 0x55d96cc2d5d0 2024-06-04T04:57:01.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.064+0000 7f3f1affd700 1 -- 172.21.15.70:0/4125587354 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3f300a60c0 con 0x55d96cc2d5d0 2024-06-04T04:57:01.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.064+0000 7f3f4c408b80 10 client.24871 fetch_fsmap learned FSMap version 28 2024-06-04T04:57:01.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.064+0000 7f3f4c408b80 1 -- 172.21.15.70:0/4125587354 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d96ccb1b40 con 0x55d96cc2d5d0 2024-06-04T04:57:01.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.064+0000 7f3f1affd700 1 -- 172.21.15.70:0/4125587354 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3f2c004cc0 con 0x55d96cc2d5d0 2024-06-04T04:57:01.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.064+0000 7f3f1affd700 1 -- 172.21.15.70:0/4125587354 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3f30070230 con 0x55d96cc2d5d0 2024-06-04T04:57:01.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.064+0000 7f3f4c408b80 10 client.24871 fetch_fsmap finished waiting for FSMap version 28 2024-06-04T04:57:01.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.064+0000 7f3f4c408b80 10 client.24871 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-04T04:57:01.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.064+0000 7f3f4c408b80 20 client.24871 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:01.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.064+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/4125587354 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d96ccb6770 0x55d96ccb5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.064+0000 7f3f4c408b80 4 client.24871 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-04T04:57:01.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.064+0000 7f3f4c408b80 1 -- 172.21.15.70:0/4125587354 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "status"}) -- 0x55d96cb89b90 con 0x55d96ccb6770 2024-06-04T04:57:01.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.065+0000 7f3f3b7fe700 1 --2- 172.21.15.70:0/4125587354 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d96ccb6770 0x55d96ccb5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.065+0000 7f3f3b7fe700 1 --2- 172.21.15.70:0/4125587354 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d96ccb6770 0x55d96ccb5480 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=105f6ed5914ea35 server_cookie=22f80f8c8baa0b48 in_seq=0 out_seq=0 2024-06-04T04:57:01.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.065+0000 7f3f1affd700 10 client.24871 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:01.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.065+0000 7f3f1affd700 1 -- 172.21.15.70:0/4125587354 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55d96cb89b90 con 0x55d96ccb6770 2024-06-04T04:57:01.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.065+0000 7f3f1affd700 10 client.24871 handle_command_reply: tid=0 2024-06-04T04:57:01.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.065+0000 7f3f4c408b80 1 client.24871 shutdown 2024-06-04T04:57:01.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.065+0000 7f3f4c408b80 1 -- 172.21.15.70:0/4125587354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc2d5d0 msgr2=0x55d96cc4cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:01.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.065+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/4125587354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc2d5d0 0x55d96cc4cbf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000fc50 tx=0x7f3f300cd330 comp rx=0 tx=0).stop 2024-06-04T04:57:01.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.065+0000 7f3f4c408b80 1 -- 172.21.15.70:0/4125587354 shutdown_connections 2024-06-04T04:57:01.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.065+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/4125587354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d96cc31320 0x55d96cc4d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.065+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/4125587354 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d96ccb6770 0x55d96ccb5480 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.065+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/4125587354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d96cc2cc20 0x55d96cc4c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.065+0000 7f3f4c408b80 1 --2- 172.21.15.70:0/4125587354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d96cc2d5d0 0x55d96cc4cbf0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.065+0000 7f3f4c408b80 1 -- 172.21.15.70:0/4125587354 >> 172.21.15.70:0/4125587354 conn(0x55d96cb56030 msgr2=0x55d96cc27eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:01.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.065+0000 7f3f1affd700 0 client.24871 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:01.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.066+0000 7f3f4c408b80 1 -- 172.21.15.70:0/4125587354 shutdown_connections 2024-06-04T04:57:01.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.066+0000 7f3f4c408b80 1 -- 172.21.15.70:0/4125587354 wait complete. 2024-06-04T04:57:01.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.066+0000 7f3f4c408b80 20 client.24871 trim_cache size 0 max 16384 2024-06-04T04:57:01.074 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:57:01.074 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:57:01.074 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-06-04T04:57:01.074 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24448, 2024-06-04T04:57:01.074 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-06-04T04:57:01.074 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-06-04T04:57:01.074 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-06-04T04:57:01.074 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 202.699142825, 2024-06-04T04:57:01.074 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-06-04T04:57:01.075 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 81, 2024-06-04T04:57:01.075 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 79, 2024-06-04T04:57:01.075 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 228.27638207499999 2024-06-04T04:57:01.075 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:57:01.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.066+0000 7f3f41b73700 1 -- 172.21.15.70:0/174662903 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3f240b2880 msgr2=0x7f3f240b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:01.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.066+0000 7f3f41b73700 1 --2- 172.21.15.70:0/174662903 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3f240b2880 0x7f3f240b4d40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00baa0 tx=0x7f3f2c005db0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.066+0000 7f3f41b73700 1 -- 172.21.15.70:0/174662903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f3c102e00 msgr2=0x7f3f3c19aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:01.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.066+0000 7f3f41b73700 1 --2- 172.21.15.70:0/174662903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f3c102e00 0x7f3f3c19aef0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3f30005910 tx=0x7f3f300047b0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.066+0000 7f3f41b73700 1 -- 172.21.15.70:0/174662903 shutdown_connections 2024-06-04T04:57:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.066+0000 7f3f41b73700 1 --2- 172.21.15.70:0/174662903 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3f240b2880 0x7f3f240b4d40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.066+0000 7f3f41b73700 1 --2- 172.21.15.70:0/174662903 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f3c108520 0x7f3f3c1a5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.066+0000 7f3f41b73700 1 --2- 172.21.15.70:0/174662903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f3c102440 0x7f3f3c19a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.066+0000 7f3f41b73700 1 --2- 172.21.15.70:0/174662903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f3c102e00 0x7f3f3c19aef0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.066+0000 7f3f41b73700 1 -- 172.21.15.70:0/174662903 >> 172.21.15.70:0/174662903 conn(0x7f3f3c0793f0 msgr2=0x7f3f3c100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.067+0000 7f3f41b73700 1 -- 172.21.15.70:0/174662903 shutdown_connections 2024-06-04T04:57:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.067+0000 7f3f41b73700 1 -- 172.21.15.70:0/174662903 wait complete. 2024-06-04T04:57:01.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:01 smithi070 ceph-mon[32631]: pgmap v392: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:57:01.801 INFO:teuthology.run_tasks:Running task sequential... 2024-06-04T04:57:01.810 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-06-04T04:57:01.810 INFO:teuthology.task.sequential:In sequential, running task print... 2024-06-04T04:57:01.811 INFO:teuthology.task.print:Enabling mgr modules 2024-06-04T04:57:01.811 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-06-04T04:57:01.812 INFO:teuthology.task.exec:Executing custom commands... 2024-06-04T04:57:01.812 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi070.front.sepia.ceph.com 2024-06-04T04:57:01.812 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-06-04T04:57:01.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.911+0000 7fa68b274640 1 -- 172.21.15.70:0/4004868060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa684102bd0 msgr2=0x7fa684103050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:01.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.911+0000 7fa68b274640 1 --2- 172.21.15.70:0/4004868060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa684102bd0 0x7fa684103050 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa678009900 tx=0x7fa67802a410 comp rx=0 tx=0).stop 2024-06-04T04:57:01.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.911+0000 7fa68b274640 1 -- 172.21.15.70:0/4004868060 shutdown_connections 2024-06-04T04:57:01.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.911+0000 7fa68b274640 1 --2- 172.21.15.70:0/4004868060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa684103590 0x7fa684109e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.911+0000 7fa68b274640 1 --2- 172.21.15.70:0/4004868060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa684102bd0 0x7fa684103050 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.911+0000 7fa68b274640 1 --2- 172.21.15.70:0/4004868060 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6841019e0 0x7fa684101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.911+0000 7fa68b274640 1 -- 172.21.15.70:0/4004868060 >> 172.21.15.70:0/4004868060 conn(0x7fa6840fcf50 msgr2=0x7fa6840ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:01.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.911+0000 7fa68b274640 1 -- 172.21.15.70:0/4004868060 shutdown_connections 2024-06-04T04:57:01.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.911+0000 7fa68b274640 1 -- 172.21.15.70:0/4004868060 wait complete. 2024-06-04T04:57:01.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa68b274640 1 Processor -- start 2024-06-04T04:57:01.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa68b274640 1 -- start start 2024-06-04T04:57:01.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa68b274640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6841019e0 0x7fa684197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa68b274640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa684102bd0 0x7fa684198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa68b274640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa684103590 0x7fa68419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa68b274640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa68410d410 con 0x7fa6841019e0 2024-06-04T04:57:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa68b274640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa68410d290 con 0x7fa684103590 2024-06-04T04:57:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa68b274640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa68410d590 con 0x7fa684102bd0 2024-06-04T04:57:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa689010640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6841019e0 0x7fa684197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa689010640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6841019e0 0x7fa684197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48806/0 (socket says 172.21.15.70:48806) 2024-06-04T04:57:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa689010640 1 -- 172.21.15.70:0/1102172615 learned_addr learned my addr 172.21.15.70:0/1102172615 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa689811640 1 --2- 172.21.15.70:0/1102172615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa684103590 0x7fa68419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa68880f640 1 --2- 172.21.15.70:0/1102172615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa684102bd0 0x7fa684198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa689010640 1 -- 172.21.15.70:0/1102172615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa684102bd0 msgr2=0x7fa684198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa689010640 1 --2- 172.21.15.70:0/1102172615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa684102bd0 0x7fa684198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa689010640 1 -- 172.21.15.70:0/1102172615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa684103590 msgr2=0x7fa68419f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa689010640 1 --2- 172.21.15.70:0/1102172615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa684103590 0x7fa68419f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa689010640 1 -- 172.21.15.70:0/1102172615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa68419f6e0 con 0x7fa6841019e0 2024-06-04T04:57:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa68880f640 1 --2- 172.21.15.70:0/1102172615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa684102bd0 0x7fa684198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa689811640 1 --2- 172.21.15.70:0/1102172615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa684103590 0x7fa68419f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.912+0000 7fa689010640 1 --2- 172.21.15.70:0/1102172615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6841019e0 0x7fa684197ac0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa67c009850 tx=0x7fa67c009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.913+0000 7fa6827fc640 1 -- 172.21.15.70:0/1102172615 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67c01d070 con 0x7fa6841019e0 2024-06-04T04:57:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.913+0000 7fa6827fc640 1 -- 172.21.15.70:0/1102172615 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa67c007c30 con 0x7fa6841019e0 2024-06-04T04:57:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.913+0000 7fa68b274640 1 -- 172.21.15.70:0/1102172615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa68406b300 con 0x7fa6841019e0 2024-06-04T04:57:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.913+0000 7fa6827fc640 1 -- 172.21.15.70:0/1102172615 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67c018c80 con 0x7fa6841019e0 2024-06-04T04:57:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.913+0000 7fa68b274640 1 -- 172.21.15.70:0/1102172615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa68406b800 con 0x7fa6841019e0 2024-06-04T04:57:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.914+0000 7fa68b274640 1 -- 172.21.15.70:0/1102172615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6840507e0 con 0x7fa6841019e0 2024-06-04T04:57:01.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.915+0000 7fa6827fc640 1 -- 172.21.15.70:0/1102172615 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fa67c020070 con 0x7fa6841019e0 2024-06-04T04:57:01.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.915+0000 7fa6827fc640 1 --2- 172.21.15.70:0/1102172615 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa66c0b28a0 0x7fa66c0b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:01.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.916+0000 7fa68880f640 1 --2- 172.21.15.70:0/1102172615 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa66c0b28a0 0x7fa66c0b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:01.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.916+0000 7fa6827fc640 1 -- 172.21.15.70:0/1102172615 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fa67c0e3f60 con 0x7fa6841019e0 2024-06-04T04:57:01.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.916+0000 7fa68880f640 1 --2- 172.21.15.70:0/1102172615 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa66c0b28a0 0x7fa66c0b4d60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa67802a930 tx=0x7fa678000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:01.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:01.919+0000 7fa6827fc640 1 -- 172.21.15.70:0/1102172615 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa67c027070 con 0x7fa6841019e0 2024-06-04T04:57:01.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:01 smithi112 ceph-mon[36453]: pgmap v392: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:57:01.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:01 smithi098 ceph-mon[35569]: pgmap v392: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-06-04T04:57:02.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:02.152+0000 7fa68b274640 1 -- 172.21.15.70:0/1102172615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fa684108a40 con 0x7fa6841019e0 2024-06-04T04:57:02.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:02.509+0000 7fa6827fc640 1 -- 172.21.15.70:0/1102172615 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fa67c018e20 con 0x7fa6841019e0 2024-06-04T04:57:02.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:02.512+0000 7fa68b274640 1 -- 172.21.15.70:0/1102172615 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa66c0b28a0 msgr2=0x7fa66c0b4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:02.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:02.512+0000 7fa68b274640 1 --2- 172.21.15.70:0/1102172615 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa66c0b28a0 0x7fa66c0b4d60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa67802a930 tx=0x7fa678000bc0 comp rx=0 tx=0).stop 2024-06-04T04:57:02.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:02.512+0000 7fa68b274640 1 -- 172.21.15.70:0/1102172615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6841019e0 msgr2=0x7fa684197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:02.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:02.512+0000 7fa68b274640 1 --2- 172.21.15.70:0/1102172615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6841019e0 0x7fa684197ac0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa67c009850 tx=0x7fa67c009d10 comp rx=0 tx=0).stop 2024-06-04T04:57:02.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:02.512+0000 7fa68b274640 1 -- 172.21.15.70:0/1102172615 shutdown_connections 2024-06-04T04:57:02.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:02.512+0000 7fa68b274640 1 --2- 172.21.15.70:0/1102172615 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7fa66c0b28a0 0x7fa66c0b4d60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:02.512+0000 7fa68b274640 1 --2- 172.21.15.70:0/1102172615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa684103590 0x7fa68419f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:02.512+0000 7fa68b274640 1 --2- 172.21.15.70:0/1102172615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa684102bd0 0x7fa684198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:02.512+0000 7fa68b274640 1 --2- 172.21.15.70:0/1102172615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6841019e0 0x7fa684197ac0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:02.512+0000 7fa68b274640 1 -- 172.21.15.70:0/1102172615 >> 172.21.15.70:0/1102172615 conn(0x7fa6840fcf50 msgr2=0x7fa684105ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:02.512+0000 7fa68b274640 1 -- 172.21.15.70:0/1102172615 shutdown_connections 2024-06-04T04:57:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:02.512+0000 7fa68b274640 1 -- 172.21.15.70:0/1102172615 wait complete. 2024-06-04T04:57:02.533 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-06-04T04:57:02.559 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-06-04T04:57:02.560 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-06-04T04:57:02.560 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-06-04T04:57:02.560 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-06-04T04:57:02.560 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-04T04:57:02.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T04:57:02.560 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-06-04T04:57:02.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T04:57:02.578 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link delete ceph-brx 2024-06-04T04:57:02.644 INFO:teuthology.orchestra.run.smithi070.stderr:Cannot find device "ceph-brx" 2024-06-04T04:57:02.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-04T04:57:02.646 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-06-04T04:57:02.646 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs ls 2024-06-04T04:57:02.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:02 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1102172615' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-06-04T04:57:02.859 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:02 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: ignoring --setuser ceph since I am not root 2024-06-04T04:57:02.859 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:02 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: ignoring --setgroup ceph since I am not root 2024-06-04T04:57:02.859 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:02 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:02.698+0000 7fba37ef9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-04T04:57:02.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:02 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1102172615' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-06-04T04:57:02.883 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:02 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: ignoring --setuser ceph since I am not root 2024-06-04T04:57:02.883 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:02 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: ignoring --setgroup ceph since I am not root 2024-06-04T04:57:02.883 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:02 smithi098 ceph-mgr[36948]: -- 172.21.15.98:0/1625654986 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55de5d0325a0 con 0x55de5d007000 2024-06-04T04:57:02.883 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:02 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:02.695+0000 7efc54c26200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-04T04:57:02.884 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:02 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1102172615' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-06-04T04:57:02.884 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:02 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: ignoring --setuser ceph since I am not root 2024-06-04T04:57:02.884 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:02 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: ignoring --setgroup ceph since I am not root 2024-06-04T04:57:02.884 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:02 smithi112 ceph-mgr[38301]: -- 172.21.15.112:0/3016517522 <== mon.1 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a4237705a0 con 0x55a423744c00 2024-06-04T04:57:02.884 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:02 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:02.695+0000 7fc35e2f7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-04T04:57:02.884 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:02 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:02.881+0000 7fc35e2f7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-04T04:57:02.932 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:03.186 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:02 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:02.982+0000 7fc35e2f7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-04T04:57:03.188 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:02 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:02.893+0000 7fba37ef9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-04T04:57:03.188 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:03 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:03.010+0000 7fba37ef9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-04T04:57:03.245 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:02 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:02.880+0000 7efc54c26200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-04T04:57:03.245 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:02 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:02.983+0000 7efc54c26200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-04T04:57:03.513 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:03 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:03.329+0000 7fba37ef9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-04T04:57:03.514 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:03 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:03.415+0000 7fba37ef9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-04T04:57:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:03 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1102172615' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-06-04T04:57:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:03 smithi112 ceph-mon[36453]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-06-04T04:57:03.686 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:03 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:03.287+0000 7fc35e2f7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-04T04:57:03.686 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:03 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:03.370+0000 7fc35e2f7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-04T04:57:03.745 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:03 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:03.295+0000 7efc54c26200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-04T04:57:03.745 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:03 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:03.381+0000 7efc54c26200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-04T04:57:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:03 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1102172615' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-06-04T04:57:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:03 smithi098 ceph-mon[35569]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-06-04T04:57:03.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:03 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1102172615' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-06-04T04:57:03.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:03 smithi070 ceph-mon[32631]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-06-04T04:57:04.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.639+0000 7f0542b7c700 1 -- 172.21.15.70:0/3851357919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f053c1095a0 msgr2=0x7f053c109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:04.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.639+0000 7f0542b7c700 1 --2- 172.21.15.70:0/3851357919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f053c1095a0 0x7f053c109980 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0530009990 tx=0x7f0530009ca0 comp rx=0 tx=0).stop 2024-06-04T04:57:04.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.639+0000 7f0542b7c700 1 -- 172.21.15.70:0/3851357919 shutdown_connections 2024-06-04T04:57:04.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.639+0000 7f0542b7c700 1 --2- 172.21.15.70:0/3851357919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f053c06c470 0x7f053c1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:04.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.639+0000 7f0542b7c700 1 --2- 172.21.15.70:0/3851357919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f053c1095a0 0x7f053c109980 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:04.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.639+0000 7f0542b7c700 1 --2- 172.21.15.70:0/3851357919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f053c06bab0 0x7f053c06bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:04.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.639+0000 7f0542b7c700 1 -- 172.21.15.70:0/3851357919 >> 172.21.15.70:0/3851357919 conn(0x7f053c0793b0 msgr2=0x7f053c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:04.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.639+0000 7f0542b7c700 1 -- 172.21.15.70:0/3851357919 shutdown_connections 2024-06-04T04:57:04.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.639+0000 7f0542b7c700 1 -- 172.21.15.70:0/3851357919 wait complete. 2024-06-04T04:57:04.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.639+0000 7f0542b7c700 1 Processor -- start 2024-06-04T04:57:04.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.639+0000 7f0542b7c700 1 -- start start 2024-06-04T04:57:04.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0542b7c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f053c06bab0 0x7f053c19bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:04.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0542b7c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f053c06c470 0x7f053c19c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:04.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0542b7c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f053c1095a0 0x7f053c1a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:04.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0542b7c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f053c113580 con 0x7f053c06c470 2024-06-04T04:57:04.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0542b7c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f053c113400 con 0x7f053c1095a0 2024-06-04T04:57:04.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0542b7c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f053c113700 con 0x7f053c06bab0 2024-06-04T04:57:04.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f053bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f053c06c470 0x7f053c19c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:04.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0540918700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f053c06bab0 0x7f053c19bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:04.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0540918700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f053c06bab0 0x7f053c19bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48734/0 (socket says 172.21.15.70:48734) 2024-06-04T04:57:04.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0540918700 1 -- 172.21.15.70:0/2115594438 learned_addr learned my addr 172.21.15.70:0/2115594438 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:04.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0541119700 1 --2- 172.21.15.70:0/2115594438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f053c1095a0 0x7f053c1a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:04.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0540918700 1 -- 172.21.15.70:0/2115594438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f053c1095a0 msgr2=0x7f053c1a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:04.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0540918700 1 --2- 172.21.15.70:0/2115594438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f053c1095a0 0x7f053c1a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:04.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0540918700 1 -- 172.21.15.70:0/2115594438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f053c06c470 msgr2=0x7f053c19c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:04.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0540918700 1 --2- 172.21.15.70:0/2115594438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f053c06c470 0x7f053c19c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:04.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0540918700 1 -- 172.21.15.70:0/2115594438 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f053c1a0d40 con 0x7f053c06bab0 2024-06-04T04:57:04.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0540918700 1 --2- 172.21.15.70:0/2115594438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f053c06bab0 0x7f053c19bd30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0530000c00 tx=0x7f05300047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:04.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0539ffb700 1 -- 172.21.15.70:0/2115594438 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0530027070 con 0x7f053c06bab0 2024-06-04T04:57:04.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0542b7c700 1 -- 172.21.15.70:0/2115594438 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f053c1a0fd0 con 0x7f053c06bab0 2024-06-04T04:57:04.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0539ffb700 1 -- 172.21.15.70:0/2115594438 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f053000fb00 con 0x7f053c06bab0 2024-06-04T04:57:04.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.640+0000 7f0539ffb700 1 -- 172.21.15.70:0/2115594438 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0530022c40 con 0x7f053c06bab0 2024-06-04T04:57:04.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.641+0000 7f0542b7c700 1 -- 172.21.15.70:0/2115594438 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f053c1a88b0 con 0x7f053c06bab0 2024-06-04T04:57:04.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.641+0000 7f0542b7c700 1 -- 172.21.15.70:0/2115594438 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f053c06a230 con 0x7f053c06bab0 2024-06-04T04:57:04.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.643+0000 7f0539ffb700 1 -- 172.21.15.70:0/2115594438 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7f053002a070 con 0x7f053c06bab0 2024-06-04T04:57:04.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.644+0000 7f0539ffb700 1 --2- 172.21.15.70:0/2115594438 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f052c0b28d0 0x7f052c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:04.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.644+0000 7f053bfff700 1 -- 172.21.15.70:0/2115594438 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f052c0b28d0 msgr2=0x7f052c0b4d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2752966840 2024-06-04T04:57:04.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.644+0000 7f0539ffb700 1 -- 172.21.15.70:0/2115594438 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f05300e3590 con 0x7f053c06bab0 2024-06-04T04:57:04.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.644+0000 7f053bfff700 1 --2- 172.21.15.70:0/2115594438 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f052c0b28d0 0x7f052c0b4d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-04T04:57:04.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.647+0000 7f0539ffb700 1 -- 172.21.15.70:0/2115594438 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0530031070 con 0x7f053c06bab0 2024-06-04T04:57:04.821 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:04 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:04.643+0000 7fba37ef9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-04T04:57:04.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.845+0000 7f053bfff700 1 -- 172.21.15.70:0/2115594438 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f052c0b28d0 msgr2=0x7f052c0b4d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2752966840 2024-06-04T04:57:04.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.845+0000 7f053bfff700 1 --2- 172.21.15.70:0/2115594438 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f052c0b28d0 0x7f052c0b4d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-04T04:57:04.885 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:04 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:04.547+0000 7fc35e2f7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-04T04:57:04.885 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:04 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:04.715+0000 7fc35e2f7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-04T04:57:04.885 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:04 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:04.882+0000 7fc35e2f7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-04T04:57:04.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.903+0000 7f0542b7c700 1 -- 172.21.15.70:0/2115594438 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f053c1a8a80 con 0x7f053c06bab0 2024-06-04T04:57:04.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.903+0000 7f0539ffb700 1 -- 172.21.15.70:0/2115594438 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+73 (secure 0 0 0) 0x7f0530013070 con 0x7f053c06bab0 2024-06-04T04:57:04.905 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-04T04:57:04.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.905+0000 7f0542b7c700 1 -- 172.21.15.70:0/2115594438 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f052c0b28d0 msgr2=0x7f052c0b4d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-04T04:57:04.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.905+0000 7f0542b7c700 1 --2- 172.21.15.70:0/2115594438 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f052c0b28d0 0x7f052c0b4d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:04.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.905+0000 7f0542b7c700 1 -- 172.21.15.70:0/2115594438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f053c06bab0 msgr2=0x7f053c19bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:04.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.905+0000 7f0542b7c700 1 --2- 172.21.15.70:0/2115594438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f053c06bab0 0x7f053c19bd30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0530000c00 tx=0x7f05300047b0 comp rx=0 tx=0).stop 2024-06-04T04:57:04.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.905+0000 7f0542b7c700 1 -- 172.21.15.70:0/2115594438 shutdown_connections 2024-06-04T04:57:04.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.905+0000 7f0542b7c700 1 --2- 172.21.15.70:0/2115594438 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f052c0b28d0 0x7f052c0b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:04.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.905+0000 7f0542b7c700 1 --2- 172.21.15.70:0/2115594438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f053c1095a0 0x7f053c1a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:04.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.905+0000 7f0542b7c700 1 --2- 172.21.15.70:0/2115594438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f053c06bab0 0x7f053c19bd30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:04.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.905+0000 7f0542b7c700 1 --2- 172.21.15.70:0/2115594438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f053c06c470 0x7f053c19c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:04.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.906+0000 7f0542b7c700 1 -- 172.21.15.70:0/2115594438 >> 172.21.15.70:0/2115594438 conn(0x7f053c0793b0 msgr2=0x7f053c10e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:04.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.906+0000 7f0542b7c700 1 -- 172.21.15.70:0/2115594438 shutdown_connections 2024-06-04T04:57:04.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:04.906+0000 7f0542b7c700 1 -- 172.21.15.70:0/2115594438 wait complete. 2024-06-04T04:57:04.921 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:04 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:04.577+0000 7efc54c26200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-04T04:57:04.921 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:04 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:04.748+0000 7efc54c26200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-04T04:57:04.921 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:04 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:04.919+0000 7efc54c26200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-04T04:57:05.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:04 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2115594438' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-06-04T04:57:05.109 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:04 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:04.819+0000 7fba37ef9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-04T04:57:05.109 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:04 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:04.996+0000 7fba37ef9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-04T04:57:05.171 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:04 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2115594438' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-06-04T04:57:05.213 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:04 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2115594438' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-06-04T04:57:05.436 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:05 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:05.169+0000 7fc35e2f7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-04T04:57:05.436 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:05 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:05.260+0000 7fc35e2f7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-04T04:57:05.495 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:05 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:05.211+0000 7efc54c26200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-04T04:57:05.495 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:05 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:05.305+0000 7efc54c26200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-04T04:57:05.550 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:05 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:05.290+0000 7fba37ef9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-04T04:57:05.550 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:05 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:05.383+0000 7fba37ef9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-04T04:57:05.579 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-06-04T04:57:05.579 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-04T04:57:05.579 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-06-04T04:57:05.579 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-04T04:57:05.579 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-04T04:57:05.579 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-06-04T04:57:05.579 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-06-04T04:57:05.580 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-04T04:57:05.580 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-04T04:57:05.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T04:57:05.580 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-06-04T04:57:05.596 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-04T04:57:05.596 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-04T04:57:05.596 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-06-04T04:57:05.597 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-06-04T04:57:05.597 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-06-04T04:57:05.597 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-06-04T04:57:05.597 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-04T04:57:05.597 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-06-04T04:57:05.597 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-06-04T04:57:05.597 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-04T04:57:05.597 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-06-04T04:57:05.597 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-06-04T04:57:05.597 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-04T04:57:05.597 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-06-04T04:57:05.597 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-04T04:57:05.598 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-06-04T04:57:05.598 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-04T04:57:05.598 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 41129sec preferred_lft 41129sec 2024-06-04T04:57:05.598 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-06-04T04:57:05.598 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-06-04T04:57:05.598 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-06-04T04:57:05.599 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-04T04:57:05.599 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-06-04T04:57:05.599 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link add name ceph-brx type bridge 2024-06-04T04:57:05.599 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip addr flush dev ceph-brx 2024-06-04T04:57:05.599 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set ceph-brx up 2024-06-04T04:57:05.599 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-06-04T04:57:05.599 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-06-04T04:57:05.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-04T04:57:05.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-04T04:57:05.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T04:57:05.801 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-04T04:57:05.809 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:05 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:05.591+0000 7fba37ef9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-04T04:57:05.826 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-06-04T04:57:05.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T04:57:05.828 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-06-04T04:57:05.895 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-06-04T04:57:05.896 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-04T04:57:05.896 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-04T04:57:05.896 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-04T04:57:05.896 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-04T04:57:05.896 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-04T04:57:05.896 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-06-04T04:57:05.896 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-04T04:57:05.896 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-04T04:57:05.896 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-04T04:57:05.897 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-06-04T04:57:05.936 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:05 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:05.460+0000 7fc35e2f7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-04T04:57:05.936 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:05 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:05.654+0000 7fc35e2f7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-04T04:57:05.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-04T04:57:05.995 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:05 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:05.510+0000 7efc54c26200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-04T04:57:05.995 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:05 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:05.708+0000 7efc54c26200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-04T04:57:06.091 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:05 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:05.807+0000 7fba37ef9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-04T04:57:06.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-04T04:57:06.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T04:57:06.183 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-06-04T04:57:06.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T04:57:06.199 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list-id 2024-06-04T04:57:06.254 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-04T04:57:06.255 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-06-04T04:57:06.255 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-04T04:57:06.255 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-06-04T04:57:06.255 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-06-04T04:57:06.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-04T04:57:06.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-04T04:57:06.378 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-06-04T04:57:06.378 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-04T04:57:06.378 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-06-04T04:57:06.378 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-06-04T04:57:06.378 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-06-04T04:57:06.379 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-06-04T04:57:06.379 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-06-04T04:57:06.379 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-06-04T04:57:06.379 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-06-04T04:57:06.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-04T04:57:06.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-04T04:57:06.509 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-04T04:57:06.509 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-06-04T04:57:06.509 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.0 up 2024-06-04T04:57:06.509 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set dev brx.0 master ceph-brx 2024-06-04T04:57:06.509 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-06-04T04:57:06.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-04T04:57:06.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-04T04:57:06.599 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-06-04T04:57:06.600 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-04T04:57:06.600 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-04T04:57:06.608 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:06 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:06.450+0000 7fba37ef9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-04T04:57:06.624 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-04T04:57:06.624 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-04T04:57:06.624 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-04T04:57:06.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-06-04T04:57:06.686 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:06 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:06.247+0000 7fc35e2f7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-04T04:57:06.686 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:06 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:06.465+0000 7fc35e2f7200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-06-04T04:57:06.745 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:06.314+0000 7efc54c26200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-04T04:57:06.745 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:06.538+0000 7efc54c26200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-06-04T04:57:06.747 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-06-04T04:57:06.803 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-06-04T04:57:06.804 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-06-04T04:57:06.804 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-06-04T04:57:06.804 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-06-04T04:57:06.804 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-06-04T04:57:06.804 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-06-04T04:57:06.804 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-06-04T04:57:06.804 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-06-04T04:57:06.804 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-06-04T04:57:06.804 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-06-04T04:57:06.804 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-06-04T04:57:06.804 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-06-04T04:57:06.804 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-06-04T04:57:06.805 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-06-04T04:57:06.805 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-06-04T04:57:06.805 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-06-04T04:57:06.805 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-06-04T04:57:06.805 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-06-04T04:57:06.805 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-06-04T04:57:06.805 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-06-04T04:57:06.805 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-06-04T04:57:06.805 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-06-04T04:57:06.805 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-06-04T04:57:06.805 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-06-04T04:57:06.805 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-06-04T04:57:06.805 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay 2024-06-04T04:57:06.805 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/a30d25f206e05366c5311467dcc5da4b46700bfc07551cc1e02162aeff2add6f/merged 2024-06-04T04:57:06.806 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/76230bbb5f5c6344c671898d5e5e67c5149113ad70c852073ee3d1319398618f/merged 2024-06-04T04:57:06.806 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/0 2024-06-04T04:57:06.806 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-06-04T04:57:06.806 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/9fe685e367bdef3db1f1780632f9703aefd4501ab6a08ee4990dff95c4dbcb06/merged 2024-06-04T04:57:06.806 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/dc02fb10687fd276576419e7ebd62db45c8dd8cb3d572884f7e6b50fd67bebc6/merged 2024-06-04T04:57:06.806 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/0c4fd2341916a74b777ca98ee122fcecbf4442a31a740c46febafefcf7a9411e/merged 2024-06-04T04:57:06.806 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/cebbb738f189048d692508d71ad2b172df42e57e64b2851476d8a15ece35fd6d/merged 2024-06-04T04:57:06.806 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/2fd9197cedd5f5482fbf34a0530be73490decfe25e047dce523eaf24db1700e4/merged 2024-06-04T04:57:06.806 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/6511e386fa02fcc9376fd660bf15970e6819bbab8b8a79668c64efee26cb44b6/merged 2024-06-04T04:57:06.806 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/14a326c35f71a498147b50b597df3cca43ffdf5ba2710ff3fa38c6a328c5d43c/merged 2024-06-04T04:57:06.806 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/299d0bee4bedbec3f24f58ba4f1d52d545383ab3e3d1c6676701e572d79c4027/merged 2024-06-04T04:57:06.806 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-06-04T04:57:06.806 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-04T04:57:06.806 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-04T04:57:06.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T04:57:06.807 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-06-04T04:57:06.859 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-06-04T04:57:06.859 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b --client_fs=cephfs) 2024-06-04T04:57:06.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-06-04T04:57:06.926 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-06-04T04:57:06.983 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-06-04T04:57:06.984 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-06-04T04:57:06.984 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-06-04T04:57:06.984 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-06-04T04:57:06.984 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-06-04T04:57:06.984 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-06-04T04:57:06.984 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-06-04T04:57:06.984 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-06-04T04:57:06.984 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-06-04T04:57:06.984 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-06-04T04:57:06.984 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-06-04T04:57:06.984 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-06-04T04:57:06.984 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-06-04T04:57:06.984 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-06-04T04:57:06.984 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-06-04T04:57:06.985 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-06-04T04:57:06.985 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-06-04T04:57:06.985 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-06-04T04:57:06.985 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-06-04T04:57:06.985 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-06-04T04:57:06.985 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-06-04T04:57:06.985 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-06-04T04:57:06.985 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-06-04T04:57:06.985 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-06-04T04:57:06.985 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-06-04T04:57:06.985 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay 2024-06-04T04:57:06.985 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/a30d25f206e05366c5311467dcc5da4b46700bfc07551cc1e02162aeff2add6f/merged 2024-06-04T04:57:06.985 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/76230bbb5f5c6344c671898d5e5e67c5149113ad70c852073ee3d1319398618f/merged 2024-06-04T04:57:06.985 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/0 2024-06-04T04:57:06.985 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-06-04T04:57:06.985 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/9fe685e367bdef3db1f1780632f9703aefd4501ab6a08ee4990dff95c4dbcb06/merged 2024-06-04T04:57:06.986 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/dc02fb10687fd276576419e7ebd62db45c8dd8cb3d572884f7e6b50fd67bebc6/merged 2024-06-04T04:57:06.986 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/0c4fd2341916a74b777ca98ee122fcecbf4442a31a740c46febafefcf7a9411e/merged 2024-06-04T04:57:06.986 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/cebbb738f189048d692508d71ad2b172df42e57e64b2851476d8a15ece35fd6d/merged 2024-06-04T04:57:06.986 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/2fd9197cedd5f5482fbf34a0530be73490decfe25e047dce523eaf24db1700e4/merged 2024-06-04T04:57:06.986 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/6511e386fa02fcc9376fd660bf15970e6819bbab8b8a79668c64efee26cb44b6/merged 2024-06-04T04:57:06.986 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/14a326c35f71a498147b50b597df3cca43ffdf5ba2710ff3fa38c6a328c5d43c/merged 2024-06-04T04:57:06.986 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/299d0bee4bedbec3f24f58ba4f1d52d545383ab3e3d1c6676701e572d79c4027/merged 2024-06-04T04:57:06.986 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-06-04T04:57:06.986 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-04T04:57:06.986 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-04T04:57:06.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T04:57:06.986 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-06-04T04:57:07.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:2024-06-04T04:57:07.001+0000 7f79f5367480 -1 init, newargv = 0x5576c3155250 newargc=15 2024-06-04T04:57:07.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[89512]: starting ceph client 2024-06-04T04:57:07.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[89512]: starting fuse 2024-06-04T04:57:07.108 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:06 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:06.704+0000 7fba37ef9200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-06-04T04:57:07.186 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:06 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:06.887+0000 7fc35e2f7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-04T04:57:07.186 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:06 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:06.982+0000 7fc35e2f7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-04T04:57:07.245 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:06.966+0000 7efc54c26200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-04T04:57:07.245 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:07 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:07.067+0000 7efc54c26200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-04T04:57:07.610 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:07 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:07.156+0000 7fba37ef9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-04T04:57:07.611 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:07 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:07.255+0000 7fba37ef9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-04T04:57:07.686 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:07 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:07.353+0000 7fc35e2f7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-04T04:57:07.745 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:07 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:07.436+0000 7efc54c26200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-04T04:57:07.993 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:07 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:07.631+0000 7fba37ef9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-04T04:57:08.042 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:07 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:07.708+0000 7fc35e2f7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-04T04:57:08.042 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:07 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:07.800+0000 7fc35e2f7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-04T04:57:08.121 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-06-04T04:57:08.135 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:07 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:07.796+0000 7efc54c26200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-04T04:57:08.135 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:07 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:07.888+0000 7efc54c26200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-04T04:57:08.148 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-06-04T04:57:08.204 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-06-04T04:57:08.204 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-06-04T04:57:08.204 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-06-04T04:57:08.204 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-06-04T04:57:08.204 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-06-04T04:57:08.204 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-06-04T04:57:08.204 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-06-04T04:57:08.205 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-06-04T04:57:08.205 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-06-04T04:57:08.205 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-06-04T04:57:08.205 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-06-04T04:57:08.205 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-06-04T04:57:08.205 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-06-04T04:57:08.205 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-06-04T04:57:08.205 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-06-04T04:57:08.205 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-06-04T04:57:08.205 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-06-04T04:57:08.205 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-06-04T04:57:08.205 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-06-04T04:57:08.205 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-06-04T04:57:08.205 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-06-04T04:57:08.206 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-06-04T04:57:08.206 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-06-04T04:57:08.206 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-06-04T04:57:08.206 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-06-04T04:57:08.206 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay 2024-06-04T04:57:08.206 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/a30d25f206e05366c5311467dcc5da4b46700bfc07551cc1e02162aeff2add6f/merged 2024-06-04T04:57:08.206 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/76230bbb5f5c6344c671898d5e5e67c5149113ad70c852073ee3d1319398618f/merged 2024-06-04T04:57:08.206 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/0 2024-06-04T04:57:08.206 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-06-04T04:57:08.206 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/9fe685e367bdef3db1f1780632f9703aefd4501ab6a08ee4990dff95c4dbcb06/merged 2024-06-04T04:57:08.206 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/dc02fb10687fd276576419e7ebd62db45c8dd8cb3d572884f7e6b50fd67bebc6/merged 2024-06-04T04:57:08.206 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/0c4fd2341916a74b777ca98ee122fcecbf4442a31a740c46febafefcf7a9411e/merged 2024-06-04T04:57:08.206 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/cebbb738f189048d692508d71ad2b172df42e57e64b2851476d8a15ece35fd6d/merged 2024-06-04T04:57:08.206 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/2fd9197cedd5f5482fbf34a0530be73490decfe25e047dce523eaf24db1700e4/merged 2024-06-04T04:57:08.206 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/6511e386fa02fcc9376fd660bf15970e6819bbab8b8a79668c64efee26cb44b6/merged 2024-06-04T04:57:08.207 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/14a326c35f71a498147b50b597df3cca43ffdf5ba2710ff3fa38c6a328c5d43c/merged 2024-06-04T04:57:08.207 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/299d0bee4bedbec3f24f58ba4f1d52d545383ab3e3d1c6676701e572d79c4027/merged 2024-06-04T04:57:08.207 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-06-04T04:57:08.207 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-04T04:57:08.207 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-04T04:57:08.207 INFO:teuthology.orchestra.run.smithi070.stdout:/home/ubuntu/cephtest/mnt.0 2024-06-04T04:57:08.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T04:57:08.207 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-06-04T04:57:08.259 INFO:teuthology.orchestra.run.smithi070.stdout:119 2024-06-04T04:57:08.259 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-06-04T04:57:08.259 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-06-04T04:57:08.259 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-06-04T04:57:08.260 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-06-04T04:57:08.260 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-06-04T04:57:08.260 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-06-04T04:57:08.260 DEBUG:teuthology.orchestra.run.smithi070:> 2024-06-04T04:57:08.260 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-06-04T04:57:08.260 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-06-04T04:57:08.260 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-06-04T04:57:08.260 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-06-04T04:57:08.260 DEBUG:teuthology.orchestra.run.smithi070:> 2024-06-04T04:57:08.260 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-06-04T04:57:08.260 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-06-04T04:57:08.260 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-06-04T04:57:08.260 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-06-04T04:57:08.260 DEBUG:teuthology.orchestra.run.smithi070:> 2024-06-04T04:57:08.261 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-06-04T04:57:08.261 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-06-04T04:57:08.261 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-06-04T04:57:08.261 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-06-04T04:57:08.261 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-06-04T04:57:08.261 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-06-04T04:57:08.261 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-06-04T04:57:08.261 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-06-04T04:57:08.261 DEBUG:teuthology.orchestra.run.smithi070:> 2024-06-04T04:57:08.261 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.0")) 2024-06-04T04:57:08.261 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-06-04T04:57:08.320 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:07 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:07.991+0000 7fba37ef9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-04T04:57:08.320 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:08 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:08.086+0000 7fba37ef9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-04T04:57:08.379 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.0.89512.asok 2024-06-04T04:57:08.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-04T04:57:08.391 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.89512.asok 2024-06-04T04:57:08.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T04:57:08.391 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.89512.asok status 2024-06-04T04:57:08.436 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:08 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:08.040+0000 7fc35e2f7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-04T04:57:08.436 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:08 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:08.170+0000 7fc35e2f7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-04T04:57:08.436 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:08 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:08.264+0000 7fc35e2f7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-04T04:57:08.495 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:08 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:08.132+0000 7efc54c26200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-04T04:57:08.495 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:08 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:08.266+0000 7efc54c26200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-04T04:57:08.495 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:08 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:08.362+0000 7efc54c26200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-04T04:57:08.528 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:57:08.528 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-06-04T04:57:08.528 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "b7328fed0e2e9ab66b09f8134f3cebfc71214504", 2024-06-04T04:57:08.529 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-3882-gb7328fed (b7328fed0e2e9ab66b09f8134f3cebfc71214504) squid (dev)", 2024-06-04T04:57:08.529 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "0", 2024-06-04T04:57:08.529 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-06-04T04:57:08.529 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-06-04T04:57:08.529 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "89512", 2024-06-04T04:57:08.529 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b" 2024-06-04T04:57:08.529 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T04:57:08.529 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-06-04T04:57:08.529 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-06-04T04:57:08.529 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 15231, 2024-06-04T04:57:08.529 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-06-04T04:57:08.529 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-06-04T04:57:08.529 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-06-04T04:57:08.529 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 15231 2024-06-04T04:57:08.530 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T04:57:08.530 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-06-04T04:57:08.530 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-06-04T04:57:08.530 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-06-04T04:57:08.530 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3317513393 2024-06-04T04:57:08.530 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T04:57:08.530 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T04:57:08.530 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-06-04T04:57:08.530 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-06-04T04:57:08.530 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-06-04T04:57:08.530 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3317513393 2024-06-04T04:57:08.530 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T04:57:08.530 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.15231 192.168.0.1:0/3317513393", 2024-06-04T04:57:08.531 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.1:0/3317513393", 2024-06-04T04:57:08.531 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 5, 2024-06-04T04:57:08.531 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 28, 2024-06-04T04:57:08.531 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 81, 2024-06-04T04:57:08.531 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-06-04T04:57:08.531 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-06-04T04:57:08.531 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-06-04T04:57:08.531 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:57:08.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T04:57:08.537 DEBUG:teuthology.orchestra.run.smithi070:> stat --file-system '--printf=%T 2024-06-04T04:57:08.538 DEBUG:teuthology.orchestra.run.smithi070:> ' -- /home/ubuntu/cephtest/mnt.0 2024-06-04T04:57:08.553 INFO:teuthology.orchestra.run.smithi070.stdout:fuseblk 2024-06-04T04:57:08.553 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-06-04T04:57:08.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T04:57:08.553 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-04T04:57:08.596 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:08 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:08.343+0000 7fba37ef9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-04T04:57:08.596 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:08 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:08.488+0000 7fba37ef9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-04T04:57:08.621 INFO:teuthology.run_tasks:Running task exec... 2024-06-04T04:57:08.633 INFO:teuthology.task.exec:Executing custom commands... 2024-06-04T04:57:08.634 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi070.front.sepia.ceph.com 2024-06-04T04:57:08.634 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-06-04T04:57:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.772+0000 7f66e0a62640 1 -- 172.21.15.70:0/2455877216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66dc102bc0 msgr2=0x7f66dc103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.772+0000 7f66e0a62640 1 --2- 172.21.15.70:0/2455877216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66dc102bc0 0x7f66dc103040 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f66cc009900 tx=0x7f66cc02a410 comp rx=0 tx=0).stop 2024-06-04T04:57:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.772+0000 7f66e0a62640 1 -- 172.21.15.70:0/2455877216 shutdown_connections 2024-06-04T04:57:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.772+0000 7f66e0a62640 1 --2- 172.21.15.70:0/2455877216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66dc103580 0x7f66dc109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.772+0000 7f66e0a62640 1 --2- 172.21.15.70:0/2455877216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66dc102bc0 0x7f66dc103040 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.772+0000 7f66e0a62640 1 --2- 172.21.15.70:0/2455877216 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66dc1019d0 0x7f66dc101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:08.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.772+0000 7f66e0a62640 1 -- 172.21.15.70:0/2455877216 >> 172.21.15.70:0/2455877216 conn(0x7f66dc0fcf40 msgr2=0x7f66dc0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:08.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.772+0000 7f66e0a62640 1 -- 172.21.15.70:0/2455877216 shutdown_connections 2024-06-04T04:57:08.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.772+0000 7f66e0a62640 1 -- 172.21.15.70:0/2455877216 wait complete. 2024-06-04T04:57:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66e0a62640 1 Processor -- start 2024-06-04T04:57:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66e0a62640 1 -- start start 2024-06-04T04:57:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66e0a62640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66dc1019d0 0x7f66dc06bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66e0a62640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66dc102bc0 0x7f66dc19d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66e0a62640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66dc103580 0x7f66dc19ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66e0a62640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f66dc10d3b0 con 0x7f66dc102bc0 2024-06-04T04:57:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66e0a62640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66dc10d230 con 0x7f66dc1019d0 2024-06-04T04:57:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66e0a62640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f66dc10d530 con 0x7f66dc103580 2024-06-04T04:57:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66d9d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66dc102bc0 0x7f66dc19d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66da59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66dc1019d0 0x7f66dc06bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66dad9d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66dc103580 0x7f66dc19ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66d9d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66dc102bc0 0x7f66dc19d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33302/0 (socket says 172.21.15.70:33302) 2024-06-04T04:57:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66da59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66dc1019d0 0x7f66dc06bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:60968/0 (socket says 172.21.15.70:60968) 2024-06-04T04:57:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66d9d9b640 1 -- 172.21.15.70:0/1145659502 learned_addr learned my addr 172.21.15.70:0/1145659502 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66da59c640 1 -- 172.21.15.70:0/1145659502 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66dc103580 msgr2=0x7f66dc19ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66da59c640 1 --2- 172.21.15.70:0/1145659502 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66dc103580 0x7f66dc19ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66da59c640 1 -- 172.21.15.70:0/1145659502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66dc102bc0 msgr2=0x7f66dc19d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66da59c640 1 --2- 172.21.15.70:0/1145659502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66dc102bc0 0x7f66dc19d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66da59c640 1 -- 172.21.15.70:0/1145659502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66dc197690 con 0x7f66dc1019d0 2024-06-04T04:57:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.773+0000 7f66dad9d640 1 --2- 172.21.15.70:0/1145659502 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66dc103580 0x7f66dc19ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:57:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.774+0000 7f66da59c640 1 --2- 172.21.15.70:0/1145659502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66dc1019d0 0x7f66dc06bdb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f66c400e9c0 tx=0x7f66c400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.774+0000 7f66d37fe640 1 -- 172.21.15.70:0/1145659502 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c400ccc0 con 0x7f66dc1019d0 2024-06-04T04:57:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.774+0000 7f66d37fe640 1 -- 172.21.15.70:0/1145659502 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f66c40035f0 con 0x7f66dc1019d0 2024-06-04T04:57:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.774+0000 7f66e0a62640 1 -- 172.21.15.70:0/1145659502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66dc197980 con 0x7f66dc1019d0 2024-06-04T04:57:08.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.774+0000 7f66d37fe640 1 -- 172.21.15.70:0/1145659502 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c400ce60 con 0x7f66dc1019d0 2024-06-04T04:57:08.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.774+0000 7f66e0a62640 1 -- 172.21.15.70:0/1145659502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66dc197e80 con 0x7f66dc1019d0 2024-06-04T04:57:08.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.775+0000 7f66e0a62640 1 -- 172.21.15.70:0/1145659502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66dc02dba0 con 0x7f66dc1019d0 2024-06-04T04:57:08.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.776+0000 7f66d37fe640 1 -- 172.21.15.70:0/1145659502 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7f66c4004000 con 0x7f66dc1019d0 2024-06-04T04:57:08.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.777+0000 7f66d37fe640 1 --2- 172.21.15.70:0/1145659502 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f66c80b28f0 0x7f66c80b4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:08.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.777+0000 7f66d9d9b640 1 -- 172.21.15.70:0/1145659502 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f66c80b28f0 msgr2=0x7f66c80b4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2752966840 2024-06-04T04:57:08.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.777+0000 7f66d37fe640 1 -- 172.21.15.70:0/1145659502 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f66c40e2620 con 0x7f66dc1019d0 2024-06-04T04:57:08.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.777+0000 7f66d9d9b640 1 --2- 172.21.15.70:0/1145659502 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f66c80b28f0 0x7f66c80b4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-04T04:57:08.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.780+0000 7f66d37fe640 1 -- 172.21.15.70:0/1145659502 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f66c40aa2c0 con 0x7f66dc1019d0 2024-06-04T04:57:08.858 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:08 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:08.592+0000 7fba37ef9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-04T04:57:08.936 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:08 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:08.586+0000 7fc35e2f7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-04T04:57:08.936 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:08 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:08.677+0000 7fc35e2f7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-04T04:57:08.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.978+0000 7f66d9d9b640 1 -- 172.21.15.70:0/1145659502 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f66c80b28f0 msgr2=0x7f66c80b4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2752966840 2024-06-04T04:57:08.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:08.978+0000 7f66d9d9b640 1 --2- 172.21.15.70:0/1145659502 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f66c80b28f0 0x7f66c80b4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-04T04:57:08.995 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:08 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:08.688+0000 7efc54c26200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-04T04:57:08.995 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:08 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:08.782+0000 7efc54c26200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-04T04:57:09.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.011+0000 7f66e0a62640 1 -- 172.21.15.70:0/1145659502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f66dc1088c0 con 0x7f66dc1019d0 2024-06-04T04:57:09.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.066+0000 7f66d37fe640 1 -- 172.21.15.70:0/1145659502 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v29) ==== 117+0+0 (secure 0 0 0) 0x7f66c40a9bc0 con 0x7f66dc1019d0 2024-06-04T04:57:09.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.070+0000 7f66e0a62640 1 -- 172.21.15.70:0/1145659502 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f66c80b28f0 msgr2=0x7f66c80b4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-04T04:57:09.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.070+0000 7f66e0a62640 1 --2- 172.21.15.70:0/1145659502 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f66c80b28f0 0x7f66c80b4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.070+0000 7f66e0a62640 1 -- 172.21.15.70:0/1145659502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66dc1019d0 msgr2=0x7f66dc06bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:09.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.070+0000 7f66e0a62640 1 --2- 172.21.15.70:0/1145659502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66dc1019d0 0x7f66dc06bdb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f66c400e9c0 tx=0x7f66c400ee80 comp rx=0 tx=0).stop 2024-06-04T04:57:09.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.070+0000 7f66e0a62640 1 -- 172.21.15.70:0/1145659502 shutdown_connections 2024-06-04T04:57:09.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.070+0000 7f66e0a62640 1 --2- 172.21.15.70:0/1145659502 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f66c80b28f0 0x7f66c80b4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.070+0000 7f66e0a62640 1 --2- 172.21.15.70:0/1145659502 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66dc103580 0x7f66dc19ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.070+0000 7f66e0a62640 1 --2- 172.21.15.70:0/1145659502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66dc102bc0 0x7f66dc19d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.070+0000 7f66e0a62640 1 --2- 172.21.15.70:0/1145659502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66dc1019d0 0x7f66dc06bdb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.070+0000 7f66e0a62640 1 -- 172.21.15.70:0/1145659502 >> 172.21.15.70:0/1145659502 conn(0x7f66dc0fcf40 msgr2=0x7f66dc105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:09.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.070+0000 7f66e0a62640 1 -- 172.21.15.70:0/1145659502 shutdown_connections 2024-06-04T04:57:09.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.070+0000 7f66e0a62640 1 -- 172.21.15.70:0/1145659502 wait complete. 2024-06-04T04:57:09.083 INFO:teuthology.run_tasks:Running task check-counter... 2024-06-04T04:57:09.108 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-06-04T04:57:09.119 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-04T04:57:09.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.221+0000 7f3e9aef1640 1 -- 172.21.15.70:0/2070062949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e94105e40 msgr2=0x7f3e94108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:09.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.221+0000 7f3e9aef1640 1 --2- 172.21.15.70:0/2070062949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e94105e40 0x7f3e94108230 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009900 tx=0x7f3e8002a410 comp rx=0 tx=0).stop 2024-06-04T04:57:09.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.221+0000 7f3e9aef1640 1 -- 172.21.15.70:0/2070062949 shutdown_connections 2024-06-04T04:57:09.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.221+0000 7f3e9aef1640 1 --2- 172.21.15.70:0/2070062949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e94108770 0x7f3e9410abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.221+0000 7f3e9aef1640 1 --2- 172.21.15.70:0/2070062949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e94105e40 0x7f3e94108230 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.221+0000 7f3e9aef1640 1 --2- 172.21.15.70:0/2070062949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e94103510 0x7f3e94105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.221+0000 7f3e9aef1640 1 -- 172.21.15.70:0/2070062949 >> 172.21.15.70:0/2070062949 conn(0x7f3e940fcf00 msgr2=0x7f3e940ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:09.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.221+0000 7f3e9aef1640 1 -- 172.21.15.70:0/2070062949 shutdown_connections 2024-06-04T04:57:09.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.221+0000 7f3e9aef1640 1 -- 172.21.15.70:0/2070062949 wait complete. 2024-06-04T04:57:09.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.222+0000 7f3e9aef1640 1 Processor -- start 2024-06-04T04:57:09.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.222+0000 7f3e9aef1640 1 -- start start 2024-06-04T04:57:09.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.222+0000 7f3e9aef1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e94103510 0x7f3e94198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:09.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.222+0000 7f3e9aef1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e94105e40 0x7f3e94199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:09.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.222+0000 7f3e9aef1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e94108770 0x7f3e941a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:09.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.222+0000 7f3e9aef1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9410e460 con 0x7f3e94105e40 2024-06-04T04:57:09.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.222+0000 7f3e9aef1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9410e2e0 con 0x7f3e94108770 2024-06-04T04:57:09.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.222+0000 7f3e9aef1640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9410e5e0 con 0x7f3e94103510 2024-06-04T04:57:09.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.223+0000 7f3e93fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e94105e40 0x7f3e94199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:09.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.223+0000 7f3e93fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e94105e40 0x7f3e94199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33328/0 (socket says 172.21.15.70:33328) 2024-06-04T04:57:09.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.223+0000 7f3e93fff640 1 -- 172.21.15.70:0/2167914169 learned_addr learned my addr 172.21.15.70:0/2167914169 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:09.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.223+0000 7f3e98c8d640 1 --2- 172.21.15.70:0/2167914169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e94103510 0x7f3e94198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:09.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.223+0000 7f3e9948e640 1 --2- 172.21.15.70:0/2167914169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e94108770 0x7f3e941a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:09.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.223+0000 7f3e93fff640 1 -- 172.21.15.70:0/2167914169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e94103510 msgr2=0x7f3e94198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:09.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.223+0000 7f3e93fff640 1 --2- 172.21.15.70:0/2167914169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e94103510 0x7f3e94198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.223+0000 7f3e93fff640 1 -- 172.21.15.70:0/2167914169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e94108770 msgr2=0x7f3e941a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:09.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.223+0000 7f3e93fff640 1 --2- 172.21.15.70:0/2167914169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e94108770 0x7f3e941a01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.223+0000 7f3e93fff640 1 -- 172.21.15.70:0/2167914169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e941a0810 con 0x7f3e94105e40 2024-06-04T04:57:09.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.223+0000 7f3e9948e640 1 --2- 172.21.15.70:0/2167914169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e94108770 0x7f3e941a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:57:09.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.223+0000 7f3e98c8d640 1 --2- 172.21.15.70:0/2167914169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e94103510 0x7f3e94198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:57:09.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.223+0000 7f3e93fff640 1 --2- 172.21.15.70:0/2167914169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e94105e40 0x7f3e94199130 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3e80005ab0 tx=0x7f3e800042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:09.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.223+0000 7f3e91ffb640 1 -- 172.21.15.70:0/2167914169 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8003f030 con 0x7f3e94105e40 2024-06-04T04:57:09.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.223+0000 7f3e91ffb640 1 -- 172.21.15.70:0/2167914169 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e80032e10 con 0x7f3e94105e40 2024-06-04T04:57:09.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.223+0000 7f3e9aef1640 1 -- 172.21.15.70:0/2167914169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e941a0aa0 con 0x7f3e94105e40 2024-06-04T04:57:09.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.223+0000 7f3e91ffb640 1 -- 172.21.15.70:0/2167914169 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e80035070 con 0x7f3e94105e40 2024-06-04T04:57:09.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.224+0000 7f3e9aef1640 1 -- 172.21.15.70:0/2167914169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e941a0f90 con 0x7f3e94105e40 2024-06-04T04:57:09.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.225+0000 7f3e9aef1640 1 -- 172.21.15.70:0/2167914169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e940fe9f0 con 0x7f3e94105e40 2024-06-04T04:57:09.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.226+0000 7f3e91ffb640 1 -- 172.21.15.70:0/2167914169 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7f3e80042030 con 0x7f3e94105e40 2024-06-04T04:57:09.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.227+0000 7f3e91ffb640 1 --2- 172.21.15.70:0/2167914169 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3e840b2840 0x7f3e840b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:09.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.227+0000 7f3e98c8d640 1 -- 172.21.15.70:0/2167914169 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3e840b2840 msgr2=0x7f3e840b4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2752966840 2024-06-04T04:57:09.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.227+0000 7f3e91ffb640 1 -- 172.21.15.70:0/2167914169 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f3e80102700 con 0x7f3e94105e40 2024-06-04T04:57:09.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.227+0000 7f3e98c8d640 1 --2- 172.21.15.70:0/2167914169 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3e840b2840 0x7f3e840b4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-04T04:57:09.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.231+0000 7f3e91ffb640 1 -- 172.21.15.70:0/2167914169 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3e800c9d90 con 0x7f3e94105e40 2024-06-04T04:57:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:09 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1145659502' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-06-04T04:57:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:09 smithi070 ceph-mon[32631]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-06-04T04:57:09.359 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:08 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:08.944+0000 7fba37ef9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-04T04:57:09.359 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:09 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:09.043+0000 7fba37ef9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-04T04:57:09.394 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:09 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1145659502' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-06-04T04:57:09.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:09 smithi098 ceph-mon[35569]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-06-04T04:57:09.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.425+0000 7f3e9aef1640 1 -- 172.21.15.70:0/2167914169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3e94063650 con 0x7f3e94105e40 2024-06-04T04:57:09.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.427+0000 7f3e91ffb640 1 -- 172.21.15.70:0/2167914169 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22708 (secure 0 0 0) 0x7f3e800ce090 con 0x7f3e94105e40 2024-06-04T04:57:09.429 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:57:09.430 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":81,"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","created":"2024-06-04T04:45:22.297536+0000","modified":"2024-06-04T04:54:19.502410+0000","last_up_change":"2024-06-04T04:52:34.388708+0000","last_in_change":"2024-06-04T04:52:14.197780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-04T04:48:43.724356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-04T04:53:31.398455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-04T04:53:35.298617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ff491ea2-1906-48e4-883e-bc78e778260e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6803","nonce":824119100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6805","nonce":824119100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6809","nonce":824119100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6807","nonce":824119100}]},"public_addr":"172.21.15.70:6803/824119100","cluster_addr":"172.21.15.70:6805/824119100","heartbeat_back_addr":"172.21.15.70:6809/824119100","heartbeat_front_addr":"172.21.15.70:6807/824119100","state":["exists","up"]},{"osd":1,"uuid":"bd5e4806-535b-49b5-ba6b-93228da35a84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6801","nonce":2744598742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6803","nonce":2744598742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6807","nonce":2744598742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6805","nonce":2744598742}]},"public_addr":"172.21.15.98:6801/2744598742","cluster_addr":"172.21.15.98:6803/2744598742","heartbeat_back_addr":"172.21.15.98:6807/2744598742","heartbeat_front_addr":"172.21.15.98:6805/2744598742","state":["exists","up"]},{"osd":2,"uuid":"8456fb55-910a-4a59-b84b-2c0f68346ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6801","nonce":1328098370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6803","nonce":1328098370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6807","nonce":1328098370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6805","nonce":1328098370}]},"public_addr":"172.21.15.112:6801/1328098370","cluster_addr":"172.21.15.112:6803/1328098370","heartbeat_back_addr":"172.21.15.112:6807/1328098370","heartbeat_front_addr":"172.21.15.112:6805/1328098370","state":["exists","up"]},{"osd":3,"uuid":"c8fbc440-bf70-42d3-b01e-777ccfa7900b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6811","nonce":4041566346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6813","nonce":4041566346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6817","nonce":4041566346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6815","nonce":4041566346}]},"public_addr":"172.21.15.70:6811/4041566346","cluster_addr":"172.21.15.70:6813/4041566346","heartbeat_back_addr":"172.21.15.70:6817/4041566346","heartbeat_front_addr":"172.21.15.70:6815/4041566346","state":["exists","up"]},{"osd":4,"uuid":"549c8c70-8be8-46ae-bcb6-0fde32b40cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6809","nonce":3652775814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6811","nonce":3652775814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6815","nonce":3652775814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6813","nonce":3652775814}]},"public_addr":"172.21.15.98:6809/3652775814","cluster_addr":"172.21.15.98:6811/3652775814","heartbeat_back_addr":"172.21.15.98:6815/3652775814","heartbeat_front_addr":"172.21.15.98:6813/3652775814","state":["exists","up"]},{"osd":5,"uuid":"51d86af1-0d6c-4ae0-b090-00414fc1bf7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6809","nonce":1857257899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6811","nonce":1857257899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6815","nonce":1857257899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6813","nonce":1857257899}]},"public_addr":"172.21.15.112:6809/1857257899","cluster_addr":"172.21.15.112:6811/1857257899","heartbeat_back_addr":"172.21.15.112:6815/1857257899","heartbeat_front_addr":"172.21.15.112:6813/1857257899","state":["exists","up"]},{"osd":6,"uuid":"526a8138-bfa8-419e-a5b5-e6b80f0ae004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6819","nonce":1264676496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6821","nonce":1264676496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6825","nonce":1264676496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6823","nonce":1264676496}]},"public_addr":"172.21.15.70:6819/1264676496","cluster_addr":"172.21.15.70:6821/1264676496","heartbeat_back_addr":"172.21.15.70:6825/1264676496","heartbeat_front_addr":"172.21.15.70:6823/1264676496","state":["exists","up"]},{"osd":7,"uuid":"1173f785-6ed1-47ef-9265-2e233ea9ee02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6817","nonce":709593020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6819","nonce":709593020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6823","nonce":709593020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6821","nonce":709593020}]},"public_addr":"172.21.15.98:6817/709593020","cluster_addr":"172.21.15.98:6819/709593020","heartbeat_back_addr":"172.21.15.98:6823/709593020","heartbeat_front_addr":"172.21.15.98:6821/709593020","state":["exists","up"]},{"osd":8,"uuid":"6424cc6e-24a7-4b81-9ce6-39203fecc7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6817","nonce":2250225413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6819","nonce":2250225413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6823","nonce":2250225413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6821","nonce":2250225413}]},"public_addr":"172.21.15.112:6817/2250225413","cluster_addr":"172.21.15.112:6819/2250225413","heartbeat_back_addr":"172.21.15.112:6823/2250225413","heartbeat_front_addr":"172.21.15.112:6821/2250225413","state":["exists","up"]},{"osd":9,"uuid":"8d666db8-8831-4d63-9e3c-a4bf2504149e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6827","nonce":3557688707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6829","nonce":3557688707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6833","nonce":3557688707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6831","nonce":3557688707}]},"public_addr":"172.21.15.70:6827/3557688707","cluster_addr":"172.21.15.70:6829/3557688707","heartbeat_back_addr":"172.21.15.70:6833/3557688707","heartbeat_front_addr":"172.21.15.70:6831/3557688707","state":["exists","up"]},{"osd":10,"uuid":"ecb925d2-0263-4dfc-a2fd-9e27856022d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6825","nonce":1896493718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6827","nonce":1896493718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6831","nonce":1896493718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6829","nonce":1896493718}]},"public_addr":"172.21.15.98:6825/1896493718","cluster_addr":"172.21.15.98:6827/1896493718","heartbeat_back_addr":"172.21.15.98:6831/1896493718","heartbeat_front_addr":"172.21.15.98:6829/1896493718","state":["exists","up"]},{"osd":11,"uuid":"c8e3230c-bc3a-40bd-8dd0-5fef56e4df97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6825","nonce":1400793137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6827","nonce":1400793137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6831","nonce":1400793137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6829","nonce":1400793137}]},"public_addr":"172.21.15.112:6825/1400793137","cluster_addr":"172.21.15.112:6827/1400793137","heartbeat_back_addr":"172.21.15.112:6831/1400793137","heartbeat_front_addr":"172.21.15.112:6829/1400793137","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:47:53.626608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:17.784505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:41.971969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:07.130699+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:31.615496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:56.898607+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:22.739991+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:48.319727+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:13.902063+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:40.389470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:06.142092+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:32.949383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/3442074771":"2024-06-05T04:46:17.636907+0000","172.21.15.70:6801/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/1442095493":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/2453733268":"2024-06-05T04:45:49.501864+0000","172.21.15.70:0/1369781447":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6801/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/915186456":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/860033006":"2024-06-05T04:45:49.501864+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-04T04:57:09.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.428+0000 7f3e98c8d640 1 -- 172.21.15.70:0/2167914169 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3e840b2840 msgr2=0x7f3e840b4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2752966840 2024-06-04T04:57:09.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.428+0000 7f3e98c8d640 1 --2- 172.21.15.70:0/2167914169 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3e840b2840 0x7f3e840b4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-04T04:57:09.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.430+0000 7f3e9aef1640 1 -- 172.21.15.70:0/2167914169 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3e840b2840 msgr2=0x7f3e840b4d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-04T04:57:09.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.430+0000 7f3e9aef1640 1 --2- 172.21.15.70:0/2167914169 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3e840b2840 0x7f3e840b4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.430+0000 7f3e9aef1640 1 -- 172.21.15.70:0/2167914169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e94105e40 msgr2=0x7f3e94199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:09.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.430+0000 7f3e9aef1640 1 --2- 172.21.15.70:0/2167914169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e94105e40 0x7f3e94199130 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3e80005ab0 tx=0x7f3e800042f0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.430+0000 7f3e9aef1640 1 -- 172.21.15.70:0/2167914169 shutdown_connections 2024-06-04T04:57:09.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.430+0000 7f3e9aef1640 1 --2- 172.21.15.70:0/2167914169 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f3e840b2840 0x7f3e840b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.430+0000 7f3e9aef1640 1 --2- 172.21.15.70:0/2167914169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e94108770 0x7f3e941a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.430+0000 7f3e9aef1640 1 --2- 172.21.15.70:0/2167914169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e94105e40 0x7f3e94199130 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.430+0000 7f3e9aef1640 1 --2- 172.21.15.70:0/2167914169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e94103510 0x7f3e94198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.430+0000 7f3e9aef1640 1 -- 172.21.15.70:0/2167914169 >> 172.21.15.70:0/2167914169 conn(0x7f3e940fcf00 msgr2=0x7f3e941053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:09.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.430+0000 7f3e9aef1640 1 -- 172.21.15.70:0/2167914169 shutdown_connections 2024-06-04T04:57:09.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.430+0000 7f3e9aef1640 1 -- 172.21.15.70:0/2167914169 wait complete. 2024-06-04T04:57:09.436 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:09 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:09.273+0000 7fc35e2f7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-04T04:57:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:09 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1145659502' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-06-04T04:57:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:09 smithi112 ceph-mon[36453]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-06-04T04:57:09.447 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-04T04:48:43.724356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-04T04:53:31.398455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-04T04:53:35.298617+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-04T04:57:09.447 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-04T04:57:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.551+0000 7f44f6647640 1 -- 172.21.15.70:0/785573660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f0105e40 msgr2=0x7f44f0108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.551+0000 7f44f6647640 1 --2- 172.21.15.70:0/785573660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f0105e40 0x7f44f0108230 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f44e4009900 tx=0x7f44e402a410 comp rx=0 tx=0).stop 2024-06-04T04:57:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.551+0000 7f44f6647640 1 -- 172.21.15.70:0/785573660 shutdown_connections 2024-06-04T04:57:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.551+0000 7f44f6647640 1 --2- 172.21.15.70:0/785573660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f0108770 0x7f44f010abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.551+0000 7f44f6647640 1 --2- 172.21.15.70:0/785573660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f0105e40 0x7f44f0108230 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.551+0000 7f44f6647640 1 --2- 172.21.15.70:0/785573660 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44f0103510 0x7f44f0105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.551+0000 7f44f6647640 1 -- 172.21.15.70:0/785573660 >> 172.21.15.70:0/785573660 conn(0x7f44f00fcf00 msgr2=0x7f44f00ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.551+0000 7f44f6647640 1 -- 172.21.15.70:0/785573660 shutdown_connections 2024-06-04T04:57:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.551+0000 7f44f6647640 1 -- 172.21.15.70:0/785573660 wait complete. 2024-06-04T04:57:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.552+0000 7f44f6647640 1 Processor -- start 2024-06-04T04:57:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.552+0000 7f44f6647640 1 -- start start 2024-06-04T04:57:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.552+0000 7f44f6647640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f0103510 0x7f44f0198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.552+0000 7f44f6647640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f0105e40 0x7f44f01991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.552+0000 7f44f6647640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44f0108770 0x7f44f01a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.552+0000 7f44f6647640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f44f010e480 con 0x7f44f0103510 2024-06-04T04:57:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.552+0000 7f44f6647640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f44f010e300 con 0x7f44f0105e40 2024-06-04T04:57:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.552+0000 7f44f6647640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f44f010e600 con 0x7f44f0108770 2024-06-04T04:57:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.552+0000 7f44effff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f0103510 0x7f44f0198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.552+0000 7f44ef7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f0105e40 0x7f44f01991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.552+0000 7f44ef7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f0105e40 0x7f44f01991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:32782/0 (socket says 172.21.15.70:32782) 2024-06-04T04:57:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.552+0000 7f44ef7fe640 1 -- 172.21.15.70:0/2063776545 learned_addr learned my addr 172.21.15.70:0/2063776545 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.553+0000 7f44f4be4640 1 --2- 172.21.15.70:0/2063776545 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44f0108770 0x7f44f01a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.553+0000 7f44ef7fe640 1 -- 172.21.15.70:0/2063776545 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44f0108770 msgr2=0x7f44f01a0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.553+0000 7f44ef7fe640 1 --2- 172.21.15.70:0/2063776545 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44f0108770 0x7f44f01a0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.553+0000 7f44ef7fe640 1 -- 172.21.15.70:0/2063776545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f0103510 msgr2=0x7f44f0198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.553+0000 7f44ef7fe640 1 --2- 172.21.15.70:0/2063776545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f0103510 0x7f44f0198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.553+0000 7f44ef7fe640 1 -- 172.21.15.70:0/2063776545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f01a08b0 con 0x7f44f0105e40 2024-06-04T04:57:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.553+0000 7f44ef7fe640 1 --2- 172.21.15.70:0/2063776545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f0105e40 0x7f44f01991b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f44e4009230 tx=0x7f44e40040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.553+0000 7f44ed7fa640 1 -- 172.21.15.70:0/2063776545 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e403f030 con 0x7f44f0105e40 2024-06-04T04:57:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.553+0000 7f44f6647640 1 -- 172.21.15.70:0/2063776545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f01a0b40 con 0x7f44f0105e40 2024-06-04T04:57:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.553+0000 7f44ed7fa640 1 -- 172.21.15.70:0/2063776545 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f44e40044c0 con 0x7f44f0105e40 2024-06-04T04:57:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.553+0000 7f44ed7fa640 1 -- 172.21.15.70:0/2063776545 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e4035070 con 0x7f44f0105e40 2024-06-04T04:57:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.553+0000 7f44f6647640 1 -- 172.21.15.70:0/2063776545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f01a1030 con 0x7f44f0105e40 2024-06-04T04:57:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.554+0000 7f44f6647640 1 -- 172.21.15.70:0/2063776545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44f00fe9f0 con 0x7f44f0105e40 2024-06-04T04:57:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.556+0000 7f44ed7fa640 1 -- 172.21.15.70:0/2063776545 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7f44e4042030 con 0x7f44f0105e40 2024-06-04T04:57:09.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.557+0000 7f44ed7fa640 1 --2- 172.21.15.70:0/2063776545 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f44d80b2840 0x7f44d80b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:09.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.557+0000 7f44effff640 1 -- 172.21.15.70:0/2063776545 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f44d80b2840 msgr2=0x7f44d80b4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2752966840 2024-06-04T04:57:09.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.557+0000 7f44ed7fa640 1 -- 172.21.15.70:0/2063776545 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f44e4102b40 con 0x7f44f0105e40 2024-06-04T04:57:09.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.557+0000 7f44effff640 1 --2- 172.21.15.70:0/2063776545 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f44d80b2840 0x7f44d80b4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-04T04:57:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.560+0000 7f44ed7fa640 1 -- 172.21.15.70:0/2063776545 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f44e40ca1f0 con 0x7f44f0105e40 2024-06-04T04:57:09.745 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:09 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:09.392+0000 7efc54c26200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-04T04:57:09.745 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 04:57:09 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y[36944]: 2024-06-04T04:57:09.611+0000 7efc54c26200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-04T04:57:09.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.745+0000 7f44f6647640 1 -- 172.21.15.70:0/2063776545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f44f002dda0 con 0x7f44f0105e40 2024-06-04T04:57:09.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.746+0000 7f44ed7fa640 1 -- 172.21.15.70:0/2063776545 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f44e40ce4f0 con 0x7f44f0105e40 2024-06-04T04:57:09.748 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-06-04T04:57:09.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.748+0000 7f44f6647640 1 -- 172.21.15.70:0/2063776545 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f44d80b2840 msgr2=0x7f44d80b4d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-04T04:57:09.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.748+0000 7f44f6647640 1 --2- 172.21.15.70:0/2063776545 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f44d80b2840 0x7f44d80b4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.748+0000 7f44f6647640 1 -- 172.21.15.70:0/2063776545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f0105e40 msgr2=0x7f44f01991b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:09.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.748+0000 7f44f6647640 1 --2- 172.21.15.70:0/2063776545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f0105e40 0x7f44f01991b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f44e4009230 tx=0x7f44e40040c0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.748+0000 7f44f6647640 1 -- 172.21.15.70:0/2063776545 shutdown_connections 2024-06-04T04:57:09.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.748+0000 7f44f6647640 1 --2- 172.21.15.70:0/2063776545 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f44d80b2840 0x7f44d80b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.748+0000 7f44f6647640 1 --2- 172.21.15.70:0/2063776545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f0105e40 0x7f44f01991b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.748+0000 7f44f6647640 1 --2- 172.21.15.70:0/2063776545 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44f0108770 0x7f44f01a0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.748+0000 7f44f6647640 1 --2- 172.21.15.70:0/2063776545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44f0103510 0x7f44f0198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.748+0000 7f44f6647640 1 -- 172.21.15.70:0/2063776545 >> 172.21.15.70:0/2063776545 conn(0x7f44f00fcf00 msgr2=0x7f44f0105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:09.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.748+0000 7f44f6647640 1 -- 172.21.15.70:0/2063776545 shutdown_connections 2024-06-04T04:57:09.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.748+0000 7f44f6647640 1 -- 172.21.15.70:0/2063776545 wait complete. 2024-06-04T04:57:09.761 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-04T04:57:09.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.864+0000 7f19e44e1640 1 -- 172.21.15.70:0/2175047803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19dc101990 msgr2=0x7f19dc101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:09.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.864+0000 7f19e44e1640 1 --2- 172.21.15.70:0/2175047803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19dc101990 0x7f19dc101db0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f19d8009900 tx=0x7f19d802a410 comp rx=0 tx=0).stop 2024-06-04T04:57:09.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.864+0000 7f19e44e1640 1 -- 172.21.15.70:0/2175047803 shutdown_connections 2024-06-04T04:57:09.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.864+0000 7f19e44e1640 1 --2- 172.21.15.70:0/2175047803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19dc103540 0x7f19dc109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.864+0000 7f19e44e1640 1 --2- 172.21.15.70:0/2175047803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19dc102b80 0x7f19dc103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.864+0000 7f19e44e1640 1 --2- 172.21.15.70:0/2175047803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19dc101990 0x7f19dc101db0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.864+0000 7f19e44e1640 1 -- 172.21.15.70:0/2175047803 >> 172.21.15.70:0/2175047803 conn(0x7f19dc0fcf00 msgr2=0x7f19dc0ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:09.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.864+0000 7f19e44e1640 1 -- 172.21.15.70:0/2175047803 shutdown_connections 2024-06-04T04:57:09.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.864+0000 7f19e44e1640 1 -- 172.21.15.70:0/2175047803 wait complete. 2024-06-04T04:57:09.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e44e1640 1 Processor -- start 2024-06-04T04:57:09.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e44e1640 1 -- start start 2024-06-04T04:57:09.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e44e1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19dc101990 0x7f19dc06a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:09.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e44e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19dc102b80 0x7f19dc19fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:09.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e44e1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19dc103540 0x7f19dc1a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:09.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e44e1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f19dc10d360 con 0x7f19dc103540 2024-06-04T04:57:09.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e44e1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19dc10d1e0 con 0x7f19dc102b80 2024-06-04T04:57:09.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e44e1640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f19dc10d4e0 con 0x7f19dc101990 2024-06-04T04:57:09.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e2a7e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19dc103540 0x7f19dc1a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:09.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e227d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19dc101990 0x7f19dc06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:09.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e2a7e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19dc103540 0x7f19dc1a2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33372/0 (socket says 172.21.15.70:33372) 2024-06-04T04:57:09.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e2a7e640 1 -- 172.21.15.70:0/382646356 learned_addr learned my addr 172.21.15.70:0/382646356 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:09.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e1a7c640 1 --2- 172.21.15.70:0/382646356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19dc102b80 0x7f19dc19fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:09.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e2a7e640 1 -- 172.21.15.70:0/382646356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19dc101990 msgr2=0x7f19dc06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:09.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e2a7e640 1 --2- 172.21.15.70:0/382646356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19dc101990 0x7f19dc06a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e2a7e640 1 -- 172.21.15.70:0/382646356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19dc102b80 msgr2=0x7f19dc19fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:09.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e2a7e640 1 --2- 172.21.15.70:0/382646356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19dc102b80 0x7f19dc19fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:09.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e2a7e640 1 -- 172.21.15.70:0/382646356 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19dc1a2b20 con 0x7f19dc103540 2024-06-04T04:57:09.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.865+0000 7f19e2a7e640 1 --2- 172.21.15.70:0/382646356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19dc103540 0x7f19dc1a2330 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009fd0 tx=0x7f19d400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:09.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.866+0000 7f19d37fe640 1 -- 172.21.15.70:0/382646356 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d4017070 con 0x7f19dc103540 2024-06-04T04:57:09.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.866+0000 7f19e44e1640 1 -- 172.21.15.70:0/382646356 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19dc1a2db0 con 0x7f19dc103540 2024-06-04T04:57:09.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.866+0000 7f19d37fe640 1 -- 172.21.15.70:0/382646356 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f19d4003e00 con 0x7f19dc103540 2024-06-04T04:57:09.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.866+0000 7f19d37fe640 1 -- 172.21.15.70:0/382646356 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d4010460 con 0x7f19dc103540 2024-06-04T04:57:09.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.866+0000 7f19e44e1640 1 -- 172.21.15.70:0/382646356 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19dc1a3220 con 0x7f19dc103540 2024-06-04T04:57:09.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.867+0000 7f19e44e1640 1 -- 172.21.15.70:0/382646356 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19dc108eb0 con 0x7f19dc103540 2024-06-04T04:57:09.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.868+0000 7f19d37fe640 1 -- 172.21.15.70:0/382646356 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7f19d4003970 con 0x7f19dc103540 2024-06-04T04:57:09.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.869+0000 7f19d37fe640 1 --2- 172.21.15.70:0/382646356 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f19c80b2870 0x7f19c80b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:09.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.869+0000 7f19e227d640 1 -- 172.21.15.70:0/382646356 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f19c80b2870 msgr2=0x7f19c80b4d30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2752966840 2024-06-04T04:57:09.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.869+0000 7f19d37fe640 1 -- 172.21.15.70:0/382646356 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f19d40e2050 con 0x7f19dc103540 2024-06-04T04:57:09.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.869+0000 7f19e227d640 1 --2- 172.21.15.70:0/382646356 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f19c80b2870 0x7f19c80b4d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-04T04:57:09.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.873+0000 7f19d37fe640 1 -- 172.21.15.70:0/382646356 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f19d40a9d50 con 0x7f19dc103540 2024-06-04T04:57:09.936 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 04:57:09 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z[38297]: 2024-06-04T04:57:09.489+0000 7fc35e2f7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-04T04:57:09.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.948+0000 7f19d37fe640 1 -- 172.21.15.70:0/382646356 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 20) ==== 149337+0+0 (secure 0 0 0) 0x7f19d40a58e0 con 0x7f19dc103540 2024-06-04T04:57:09.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.948+0000 7f19d37fe640 1 -- 172.21.15.70:0/382646356 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f19c80b2870 msgr2=0x7f19c80b4d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-04T04:57:09.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:09.948+0000 7f19d37fe640 1 --2- 172.21.15.70:0/382646356 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f19c80b2870 0x7f19c80b4d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:10.073 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:09 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:09.703+0000 7fba37ef9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-04T04:57:10.073 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:09 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:09.934+0000 7fba37ef9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-04T04:57:10.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.093+0000 7f19e44e1640 1 -- 172.21.15.70:0/382646356 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f19dc063650 con 0x7f19dc103540 2024-06-04T04:57:10.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.093+0000 7f19d37fe640 1 -- 172.21.15.70:0/382646356 <== mon.0 v2:172.21.15.70:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) ==== 104+0+11 (secure 0 0 0) 0x7f19d4099190 con 0x7f19dc103540 2024-06-04T04:57:10.096 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-06-04T04:57:10.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.096+0000 7f19e44e1640 1 -- 172.21.15.70:0/382646356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19dc103540 msgr2=0x7f19dc1a2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:10.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.096+0000 7f19e44e1640 1 --2- 172.21.15.70:0/382646356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19dc103540 0x7f19dc1a2330 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009fd0 tx=0x7f19d400b260 comp rx=0 tx=0).stop 2024-06-04T04:57:10.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.096+0000 7f19e44e1640 1 -- 172.21.15.70:0/382646356 shutdown_connections 2024-06-04T04:57:10.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.096+0000 7f19e44e1640 1 --2- 172.21.15.70:0/382646356 >> [v2:172.21.15.70:6800/2752966840,v1:172.21.15.70:6801/2752966840] conn(0x7f19c80b2870 0x7f19c80b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:10.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.096+0000 7f19e44e1640 1 --2- 172.21.15.70:0/382646356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19dc102b80 0x7f19dc19fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:10.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.096+0000 7f19e44e1640 1 --2- 172.21.15.70:0/382646356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19dc103540 0x7f19dc1a2330 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:10.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.096+0000 7f19e44e1640 1 --2- 172.21.15.70:0/382646356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19dc101990 0x7f19dc06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:10.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.096+0000 7f19e44e1640 1 -- 172.21.15.70:0/382646356 >> 172.21.15.70:0/382646356 conn(0x7f19dc0fcf00 msgr2=0x7f19dc106080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:10.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.096+0000 7f19e44e1640 1 -- 172.21.15.70:0/382646356 shutdown_connections 2024-06-04T04:57:10.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.096+0000 7f19e44e1640 1 -- 172.21.15.70:0/382646356 wait complete. 2024-06-04T04:57:10.114 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-04T04:57:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.242+0000 7f6b9c859640 1 -- 172.21.15.70:0/1824240772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b98105e40 msgr2=0x7f6b98108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.242+0000 7f6b9c859640 1 --2- 172.21.15.70:0/1824240772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b98105e40 0x7f6b98108230 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c009900 tx=0x7f6b8c02a410 comp rx=0 tx=0).stop 2024-06-04T04:57:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.242+0000 7f6b9c859640 1 -- 172.21.15.70:0/1824240772 shutdown_connections 2024-06-04T04:57:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.242+0000 7f6b9c859640 1 --2- 172.21.15.70:0/1824240772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b98108770 0x7f6b9810abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.242+0000 7f6b9c859640 1 --2- 172.21.15.70:0/1824240772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b98105e40 0x7f6b98108230 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.242+0000 7f6b9c859640 1 --2- 172.21.15.70:0/1824240772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6b98103510 0x7f6b98105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.242+0000 7f6b9c859640 1 -- 172.21.15.70:0/1824240772 >> 172.21.15.70:0/1824240772 conn(0x7f6b980fcf00 msgr2=0x7f6b980ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.242+0000 7f6b9c859640 1 -- 172.21.15.70:0/1824240772 shutdown_connections 2024-06-04T04:57:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.242+0000 7f6b9c859640 1 -- 172.21.15.70:0/1824240772 wait complete. 2024-06-04T04:57:10.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.243+0000 7f6b9c859640 1 Processor -- start 2024-06-04T04:57:10.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.243+0000 7f6b9c859640 1 -- start start 2024-06-04T04:57:10.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.243+0000 7f6b9c859640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6b98103510 0x7f6b98102e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:10.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.243+0000 7f6b9c859640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b98105e40 0x7f6b981014c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:10.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.243+0000 7f6b9c859640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b98108770 0x7f6b98101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:10.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.243+0000 7f6b9c859640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9810e480 con 0x7f6b98108770 2024-06-04T04:57:10.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.243+0000 7f6b9c859640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9810e300 con 0x7f6b98105e40 2024-06-04T04:57:10.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.243+0000 7f6b9c859640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9810e600 con 0x7f6b98103510 2024-06-04T04:57:10.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.243+0000 7f6b96d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b98108770 0x7f6b98101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:10.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.243+0000 7f6b96d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b98108770 0x7f6b98101a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33538/0 (socket says 172.21.15.70:33538) 2024-06-04T04:57:10.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.243+0000 7f6b96d9d640 1 -- 172.21.15.70:0/2049335740 learned_addr learned my addr 172.21.15.70:0/2049335740 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:10.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.243+0000 7f6b9659c640 1 --2- 172.21.15.70:0/2049335740 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6b98103510 0x7f6b98102e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:10.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.243+0000 7f6b95d9b640 1 --2- 172.21.15.70:0/2049335740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b98105e40 0x7f6b981014c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:10.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.244+0000 7f6b96d9d640 1 -- 172.21.15.70:0/2049335740 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6b98103510 msgr2=0x7f6b98102e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:10.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.244+0000 7f6b96d9d640 1 --2- 172.21.15.70:0/2049335740 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6b98103510 0x7f6b98102e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:10.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.244+0000 7f6b96d9d640 1 -- 172.21.15.70:0/2049335740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b98105e40 msgr2=0x7f6b981014c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:10.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.244+0000 7f6b96d9d640 1 --2- 172.21.15.70:0/2049335740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b98105e40 0x7f6b981014c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:10.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.244+0000 7f6b96d9d640 1 -- 172.21.15.70:0/2049335740 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b98102290 con 0x7f6b98108770 2024-06-04T04:57:10.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.244+0000 7f6b9659c640 1 --2- 172.21.15.70:0/2049335740 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6b98103510 0x7f6b98102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:57:10.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.244+0000 7f6b96d9d640 1 --2- 172.21.15.70:0/2049335740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b98108770 0x7f6b98101a00 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000eb10 tx=0x7f6b9000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:10.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.244+0000 7f6b877fe640 1 -- 172.21.15.70:0/2049335740 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b90004b20 con 0x7f6b98108770 2024-06-04T04:57:10.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.244+0000 7f6b9c859640 1 -- 172.21.15.70:0/2049335740 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b981a2d90 con 0x7f6b98108770 2024-06-04T04:57:10.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.244+0000 7f6b877fe640 1 -- 172.21.15.70:0/2049335740 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6b90004cc0 con 0x7f6b98108770 2024-06-04T04:57:10.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.244+0000 7f6b877fe640 1 -- 172.21.15.70:0/2049335740 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b90010690 con 0x7f6b98108770 2024-06-04T04:57:10.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.244+0000 7f6b9c859640 1 -- 172.21.15.70:0/2049335740 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b981a31a0 con 0x7f6b98108770 2024-06-04T04:57:10.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.245+0000 7f6b9c859640 1 -- 172.21.15.70:0/2049335740 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b980fe9f0 con 0x7f6b98108770 2024-06-04T04:57:10.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.247+0000 7f6b877fe640 1 -- 172.21.15.70:0/2049335740 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 149337+0+0 (secure 0 0 0) 0x7f6b90005020 con 0x7f6b98108770 2024-06-04T04:57:10.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.248+0000 7f6b877fe640 1 -- 172.21.15.70:0/2049335740 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f6b900e2370 con 0x7f6b98108770 2024-06-04T04:57:10.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.252+0000 7f6b877fe640 1 -- 172.21.15.70:0/2049335740 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6b900aa370 con 0x7f6b98108770 2024-06-04T04:57:10.359 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:10.207+0000 7fb994f35700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:57:10.359 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:10.207+0000 7fb994f35700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:57:10.359 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:10.207+0000 7fb994f35700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:57:10.359 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:10.207+0000 7fb994f35700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:57:10.360 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:10.207+0000 7fb994f35700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:57:10.360 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:10.208+0000 7fb99873c700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:57:10.360 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:10.208+0000 7fb99873c700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:57:10.360 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:10.208+0000 7fb99873c700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:57:10.360 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:10.208+0000 7fb99873c700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:57:10.360 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:57:10.208+0000 7fb99873c700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:57:10.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-06-04T04:57:10.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T04:57:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2167914169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:57:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: Standby manager daemon z restarted 2024-06-04T04:57:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: Standby manager daemon z started 2024-06-04T04:57:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.? 172.21.15.112:0/108468049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-04T04:57:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.? 172.21.15.112:0/108468049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-04T04:57:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.? 172.21.15.112:0/108468049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-04T04:57:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.? 172.21.15.112:0/108468049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-04T04:57:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: Standby manager daemon y restarted 2024-06-04T04:57:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: Standby manager daemon y started 2024-06-04T04:57:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.? 172.21.15.98:0/3691281541' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-04T04:57:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.? 172.21.15.98:0/3691281541' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-04T04:57:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.? 172.21.15.98:0/3691281541' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-04T04:57:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.? 172.21.15.98:0/3691281541' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-04T04:57:10.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2063776545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-04T04:57:10.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: Active manager daemon x restarted 2024-06-04T04:57:10.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: Activating manager daemon x 2024-06-04T04:57:10.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: osdmap e82: 12 total, 12 up, 12 in 2024-06-04T04:57:10.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: mgrmap e20: x(active, starting, since 0.0124181s), standbys: z, y 2024-06-04T04:57:10.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:57:10.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:57:10.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:57:10.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-04T04:57:10.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-04T04:57:10.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-04T04:57:10.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-04T04:57:10.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-04T04:57:10.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-04T04:57:10.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-04T04:57:10.363 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-04T04:57:10.363 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-04T04:57:10.363 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-04T04:57:10.363 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-04T04:57:10.363 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-04T04:57:10.363 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-04T04:57:10.363 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-04T04:57:10.363 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-04T04:57:10.363 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:57:10.363 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:57:10.363 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:57:10.363 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:57:10.363 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:57:10.363 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:57:10.363 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:57:10.363 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:57:10.364 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:57:10.364 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:57:10.364 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:57:10.364 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:57:10.364 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-04T04:57:10.364 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-04T04:57:10.364 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-04T04:57:10.364 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: Manager daemon x is now available 2024-06-04T04:57:10.364 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:57:10.364 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-04T04:57:10.364 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:10 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-04T04:57:10.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-06-04T04:57:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T04:57:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2167914169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:57:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: Standby manager daemon z restarted 2024-06-04T04:57:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: Standby manager daemon z started 2024-06-04T04:57:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.? 172.21.15.112:0/108468049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-04T04:57:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.? 172.21.15.112:0/108468049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-04T04:57:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.? 172.21.15.112:0/108468049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-04T04:57:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.? 172.21.15.112:0/108468049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-04T04:57:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: Standby manager daemon y restarted 2024-06-04T04:57:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: Standby manager daemon y started 2024-06-04T04:57:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.? 172.21.15.98:0/3691281541' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-04T04:57:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.? 172.21.15.98:0/3691281541' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-04T04:57:10.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.? 172.21.15.98:0/3691281541' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-04T04:57:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.? 172.21.15.98:0/3691281541' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-04T04:57:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2063776545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-04T04:57:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: Active manager daemon x restarted 2024-06-04T04:57:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: Activating manager daemon x 2024-06-04T04:57:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: osdmap e82: 12 total, 12 up, 12 in 2024-06-04T04:57:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: mgrmap e20: x(active, starting, since 0.0124181s), standbys: z, y 2024-06-04T04:57:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:57:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:57:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:57:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-04T04:57:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-04T04:57:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-04T04:57:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-04T04:57:10.438 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-04T04:57:10.439 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-04T04:57:10.439 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-04T04:57:10.439 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-04T04:57:10.439 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-04T04:57:10.439 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-04T04:57:10.439 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-04T04:57:10.439 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-04T04:57:10.439 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-04T04:57:10.439 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-04T04:57:10.440 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-04T04:57:10.440 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:57:10.440 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:57:10.440 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:57:10.440 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:57:10.440 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:57:10.440 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:57:10.440 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:57:10.440 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:57:10.440 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:57:10.440 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:57:10.440 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:57:10.440 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:57:10.441 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-04T04:57:10.441 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-04T04:57:10.441 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-04T04:57:10.441 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: Manager daemon x is now available 2024-06-04T04:57:10.441 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:57:10.441 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-04T04:57:10.441 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:10 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-04T04:57:10.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.444+0000 7f6b9c859640 1 -- 172.21.15.70:0/2049335740 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6b98063650 con 0x7f6b98108770 2024-06-04T04:57:10.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.444+0000 7f6b877fe640 1 -- 172.21.15.70:0/2049335740 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) ==== 100+0+11 (secure 0 0 0) 0x7f6b900b23f0 con 0x7f6b98108770 2024-06-04T04:57:10.446 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-06-04T04:57:10.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.446+0000 7f6b9c859640 1 -- 172.21.15.70:0/2049335740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b98108770 msgr2=0x7f6b98101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:10.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.446+0000 7f6b9c859640 1 --2- 172.21.15.70:0/2049335740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b98108770 0x7f6b98101a00 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000eb10 tx=0x7f6b9000c2d0 comp rx=0 tx=0).stop 2024-06-04T04:57:10.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.446+0000 7f6b9c859640 1 -- 172.21.15.70:0/2049335740 shutdown_connections 2024-06-04T04:57:10.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.446+0000 7f6b9c859640 1 --2- 172.21.15.70:0/2049335740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b98105e40 0x7f6b981014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:10.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.446+0000 7f6b9c859640 1 --2- 172.21.15.70:0/2049335740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b98108770 0x7f6b98101a00 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:10.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.446+0000 7f6b9c859640 1 --2- 172.21.15.70:0/2049335740 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6b98103510 0x7f6b98102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:10.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.446+0000 7f6b9c859640 1 -- 172.21.15.70:0/2049335740 >> 172.21.15.70:0/2049335740 conn(0x7f6b980fcf00 msgr2=0x7f6b98105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:10.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.446+0000 7f6b9c859640 1 -- 172.21.15.70:0/2049335740 shutdown_connections 2024-06-04T04:57:10.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:10.446+0000 7f6b9c859640 1 -- 172.21.15.70:0/2049335740 wait complete. 2024-06-04T04:57:10.459 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-06-04T04:57:10.459 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:57:10.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-06-04T04:57:10.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T04:57:10.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2167914169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:57:10.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: Standby manager daemon z restarted 2024-06-04T04:57:10.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: Standby manager daemon z started 2024-06-04T04:57:10.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.? 172.21.15.112:0/108468049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-04T04:57:10.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.? 172.21.15.112:0/108468049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-04T04:57:10.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.? 172.21.15.112:0/108468049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-04T04:57:10.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.? 172.21.15.112:0/108468049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-04T04:57:10.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: Standby manager daemon y restarted 2024-06-04T04:57:10.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: Standby manager daemon y started 2024-06-04T04:57:10.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.? 172.21.15.98:0/3691281541' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-04T04:57:10.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.? 172.21.15.98:0/3691281541' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-04T04:57:10.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.? 172.21.15.98:0/3691281541' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-04T04:57:10.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.? 172.21.15.98:0/3691281541' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-04T04:57:10.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2063776545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-04T04:57:10.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: Active manager daemon x restarted 2024-06-04T04:57:10.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: Activating manager daemon x 2024-06-04T04:57:10.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: osdmap e82: 12 total, 12 up, 12 in 2024-06-04T04:57:10.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: mgrmap e20: x(active, starting, since 0.0124181s), standbys: z, y 2024-06-04T04:57:10.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-04T04:57:10.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-04T04:57:10.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-04T04:57:10.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-04T04:57:10.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-04T04:57:10.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-04T04:57:10.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-04T04:57:10.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-04T04:57:10.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-04T04:57:10.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-04T04:57:10.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-04T04:57:10.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-04T04:57:10.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-04T04:57:10.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-04T04:57:10.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-04T04:57:10.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-04T04:57:10.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-04T04:57:10.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-04T04:57:10.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-04T04:57:10.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-04T04:57:10.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-04T04:57:10.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-04T04:57:10.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-04T04:57:10.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-04T04:57:10.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-04T04:57:10.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-04T04:57:10.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-04T04:57:10.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-04T04:57:10.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-04T04:57:10.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-04T04:57:10.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-04T04:57:10.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-04T04:57:10.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-04T04:57:10.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: Manager daemon x is now available 2024-06-04T04:57:10.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:57:10.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-04T04:57:10.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:10 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-04T04:57:10.723 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:11.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:11 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/382646356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-04T04:57:11.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:11 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2049335740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-04T04:57:11.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:11 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:11.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:11 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:11.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:11 smithi070 ceph-mon[32631]: mgrmap e21: x(active, since 1.01959s), standbys: z, y 2024-06-04T04:57:11.399 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:11 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/382646356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-04T04:57:11.399 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:11 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2049335740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-04T04:57:11.399 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:11 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:11.399 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:11 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:11.399 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:11 smithi098 ceph-mon[35569]: mgrmap e21: x(active, since 1.01959s), standbys: z, y 2024-06-04T04:57:11.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:11 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/382646356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-04T04:57:11.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:11 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2049335740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-04T04:57:11.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:11 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:11.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:11 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:11.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:11 smithi112 ceph-mon[36453]: mgrmap e21: x(active, since 1.01959s), standbys: z, y 2024-06-04T04:57:12.194 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:12 smithi070 ceph-mon[32631]: [04/Jun/2024:04:57:10] ENGINE Bus STARTING 2024-06-04T04:57:12.195 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:12 smithi070 ceph-mon[32631]: [04/Jun/2024:04:57:11] ENGINE Serving on http://172.21.15.70:8765 2024-06-04T04:57:12.195 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:12 smithi070 ceph-mon[32631]: [04/Jun/2024:04:57:11] ENGINE Serving on https://172.21.15.70:7150 2024-06-04T04:57:12.195 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:12 smithi070 ceph-mon[32631]: [04/Jun/2024:04:57:11] ENGINE Bus STARTED 2024-06-04T04:57:12.195 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:12 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:12.195 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:12 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:12.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:12 smithi098 ceph-mon[35569]: [04/Jun/2024:04:57:10] ENGINE Bus STARTING 2024-06-04T04:57:12.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:12 smithi098 ceph-mon[35569]: [04/Jun/2024:04:57:11] ENGINE Serving on http://172.21.15.70:8765 2024-06-04T04:57:12.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:12 smithi098 ceph-mon[35569]: [04/Jun/2024:04:57:11] ENGINE Serving on https://172.21.15.70:7150 2024-06-04T04:57:12.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:12 smithi098 ceph-mon[35569]: [04/Jun/2024:04:57:11] ENGINE Bus STARTED 2024-06-04T04:57:12.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:12 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:12.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:12 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.426+0000 7f067b55c700 1 -- 172.21.15.70:0/1566896420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674102e30 msgr2=0x7f067410f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.426+0000 7f067b55c700 1 --2- 172.21.15.70:0/1566896420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674102e30 0x7f067410f2f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f066c00b0a0 tx=0x7f066c00b3b0 comp rx=0 tx=0).stop 2024-06-04T04:57:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.426+0000 7f067b55c700 1 -- 172.21.15.70:0/1566896420 shutdown_connections 2024-06-04T04:57:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.426+0000 7f067b55c700 1 --2- 172.21.15.70:0/1566896420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674102e30 0x7f067410f2f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.426+0000 7f067b55c700 1 --2- 172.21.15.70:0/1566896420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06741084b0 0x7f0674108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.426+0000 7f067b55c700 1 --2- 172.21.15.70:0/1566896420 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0674102470 0x7f06741028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.426+0000 7f067b55c700 1 -- 172.21.15.70:0/1566896420 >> 172.21.15.70:0/1566896420 conn(0x7f0674079410 msgr2=0x7f0674079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.426+0000 7f067b55c700 1 -- 172.21.15.70:0/1566896420 shutdown_connections 2024-06-04T04:57:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.426+0000 7f067b55c700 1 -- 172.21.15.70:0/1566896420 wait complete. 2024-06-04T04:57:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.427+0000 7f067b55c700 1 Processor -- start 2024-06-04T04:57:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.427+0000 7f067b55c700 1 -- start start 2024-06-04T04:57:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.427+0000 7f067b55c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0674102470 0x7f067419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.427+0000 7f067b55c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674102e30 0x7f067419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.427+0000 7f067b55c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06741084b0 0x7f0674198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.427+0000 7f067b55c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f06741124a0 con 0x7f0674102e30 2024-06-04T04:57:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.427+0000 7f067b55c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0674112320 con 0x7f0674102470 2024-06-04T04:57:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.427+0000 7f067b55c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0674112620 con 0x7f06741084b0 2024-06-04T04:57:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.427+0000 7f0678af7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674102e30 0x7f067419ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.427+0000 7f0678af7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674102e30 0x7f067419ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33574/0 (socket says 172.21.15.70:33574) 2024-06-04T04:57:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.427+0000 7f0678af7700 1 -- 172.21.15.70:0/29051470 learned_addr learned my addr 172.21.15.70:0/29051470 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.427+0000 7f06792f8700 1 --2- 172.21.15.70:0/29051470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0674102470 0x7f067419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.427+0000 7f0679af9700 1 --2- 172.21.15.70:0/29051470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06741084b0 0x7f0674198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.428+0000 7f0678af7700 1 -- 172.21.15.70:0/29051470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06741084b0 msgr2=0x7f0674198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.428+0000 7f0678af7700 1 --2- 172.21.15.70:0/29051470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06741084b0 0x7f0674198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:12.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.428+0000 7f0678af7700 1 -- 172.21.15.70:0/29051470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0674102470 msgr2=0x7f067419e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:12.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.428+0000 7f0678af7700 1 --2- 172.21.15.70:0/29051470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0674102470 0x7f067419e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:12.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.428+0000 7f0678af7700 1 -- 172.21.15.70:0/29051470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0674199020 con 0x7f0674102e30 2024-06-04T04:57:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.428+0000 7f0678af7700 1 --2- 172.21.15.70:0/29051470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674102e30 0x7f067419ec00 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f067000e9c0 tx=0x7f067000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.428+0000 7f06667fc700 1 -- 172.21.15.70:0/29051470 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067000cbd0 con 0x7f0674102e30 2024-06-04T04:57:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.428+0000 7f06667fc700 1 -- 172.21.15.70:0/29051470 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f067000ee60 con 0x7f0674102e30 2024-06-04T04:57:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.428+0000 7f067b55c700 1 -- 172.21.15.70:0/29051470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06741992b0 con 0x7f0674102e30 2024-06-04T04:57:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.428+0000 7f06667fc700 1 -- 172.21.15.70:0/29051470 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06700054b0 con 0x7f0674102e30 2024-06-04T04:57:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.428+0000 7f067b55c700 1 -- 172.21.15.70:0/29051470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06741a5520 con 0x7f0674102e30 2024-06-04T04:57:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.429+0000 7f067b55c700 1 -- 172.21.15.70:0/29051470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0674103b60 con 0x7f0674102e30 2024-06-04T04:57:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.431+0000 7f06667fc700 1 -- 172.21.15.70:0/29051470 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 22) ==== 149772+0+0 (secure 0 0 0) 0x7f0670004750 con 0x7f0674102e30 2024-06-04T04:57:12.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.431+0000 7f06667fc700 1 --2- 172.21.15.70:0/29051470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f06600b29f0 0x7f06600b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:12.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.431+0000 7f06792f8700 1 --2- 172.21.15.70:0/29051470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f06600b29f0 0x7f06600b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:12.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.432+0000 7f06667fc700 1 -- 172.21.15.70:0/29051470 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f06700e2b50 con 0x7f0674102e30 2024-06-04T04:57:12.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.432+0000 7f06792f8700 1 --2- 172.21.15.70:0/29051470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f06600b29f0 0x7f06600b4eb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0668009be0 tx=0x7f0668009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:12.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:12 smithi112 ceph-mon[36453]: [04/Jun/2024:04:57:10] ENGINE Bus STARTING 2024-06-04T04:57:12.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:12 smithi112 ceph-mon[36453]: [04/Jun/2024:04:57:11] ENGINE Serving on http://172.21.15.70:8765 2024-06-04T04:57:12.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:12 smithi112 ceph-mon[36453]: [04/Jun/2024:04:57:11] ENGINE Serving on https://172.21.15.70:7150 2024-06-04T04:57:12.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:12 smithi112 ceph-mon[36453]: [04/Jun/2024:04:57:11] ENGINE Bus STARTED 2024-06-04T04:57:12.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:12 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:12.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:12 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:12.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.436+0000 7f06667fc700 1 -- 172.21.15.70:0/29051470 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0670054df0 con 0x7f0674102e30 2024-06-04T04:57:12.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.702+0000 7f067b55c700 1 -- 172.21.15.70:0/29051470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f067402e680 con 0x7f0674102e30 2024-06-04T04:57:12.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.703+0000 7f06667fc700 1 -- 172.21.15.70:0/29051470 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f06700aa530 con 0x7f0674102e30 2024-06-04T04:57:12.706 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:57:12.706 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:57:09.013178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T04:57:12.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.705+0000 7f067b55c700 1 -- 172.21.15.70:0/29051470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f06600b29f0 msgr2=0x7f06600b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:12.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.705+0000 7f067b55c700 1 --2- 172.21.15.70:0/29051470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f06600b29f0 0x7f06600b4eb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0668009be0 tx=0x7f0668009260 comp rx=0 tx=0).stop 2024-06-04T04:57:12.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.705+0000 7f067b55c700 1 -- 172.21.15.70:0/29051470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674102e30 msgr2=0x7f067419ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:12.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.705+0000 7f067b55c700 1 --2- 172.21.15.70:0/29051470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674102e30 0x7f067419ec00 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f067000e9c0 tx=0x7f067000ecd0 comp rx=0 tx=0).stop 2024-06-04T04:57:12.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.705+0000 7f067b55c700 1 -- 172.21.15.70:0/29051470 shutdown_connections 2024-06-04T04:57:12.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.705+0000 7f067b55c700 1 --2- 172.21.15.70:0/29051470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f06600b29f0 0x7f06600b4eb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:12.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.705+0000 7f067b55c700 1 --2- 172.21.15.70:0/29051470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06741084b0 0x7f0674198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:12.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.705+0000 7f067b55c700 1 --2- 172.21.15.70:0/29051470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0674102470 0x7f067419e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:12.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.705+0000 7f067b55c700 1 --2- 172.21.15.70:0/29051470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674102e30 0x7f067419ec00 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:12.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.705+0000 7f067b55c700 1 -- 172.21.15.70:0/29051470 >> 172.21.15.70:0/29051470 conn(0x7f0674079410 msgr2=0x7f0674100580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:12.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.706+0000 7f067b55c700 1 -- 172.21.15.70:0/29051470 shutdown_connections 2024-06-04T04:57:12.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:12.706+0000 7f067b55c700 1 -- 172.21.15.70:0/29051470 wait complete. 2024-06-04T04:57:12.709 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-06-04T04:57:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:13 smithi070 ceph-mon[32631]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:57:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:13 smithi070 ceph-mon[32631]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-06-04T04:57:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:13 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:13 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:13 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/29051470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:57:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:13 smithi112 ceph-mon[36453]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:57:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:13 smithi112 ceph-mon[36453]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-06-04T04:57:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:13 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:13 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:13 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/29051470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:57:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:13 smithi098 ceph-mon[35569]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:57:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:13 smithi098 ceph-mon[35569]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-06-04T04:57:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:13 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:13 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:13 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/29051470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:57:13.794 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-06-04T04:57:13.794 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-06-04T04:57:13.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-04T04:57:13.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.902+0000 7f82b10da640 1 -- 172.21.15.70:0/3781968097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82ac102b80 msgr2=0x7f82ac103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.902+0000 7f82b10da640 1 --2- 172.21.15.70:0/3781968097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82ac102b80 0x7f82ac103000 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009900 tx=0x7f82a002a410 comp rx=0 tx=0).stop 2024-06-04T04:57:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.902+0000 7f82b10da640 1 -- 172.21.15.70:0/3781968097 shutdown_connections 2024-06-04T04:57:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.902+0000 7f82b10da640 1 --2- 172.21.15.70:0/3781968097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82ac103540 0x7f82ac109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.902+0000 7f82b10da640 1 --2- 172.21.15.70:0/3781968097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82ac102b80 0x7f82ac103000 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.902+0000 7f82b10da640 1 --2- 172.21.15.70:0/3781968097 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82ac101990 0x7f82ac101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.902+0000 7f82b10da640 1 -- 172.21.15.70:0/3781968097 >> 172.21.15.70:0/3781968097 conn(0x7f82ac0fcf00 msgr2=0x7f82ac0ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.902+0000 7f82b10da640 1 -- 172.21.15.70:0/3781968097 shutdown_connections 2024-06-04T04:57:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.902+0000 7f82b10da640 1 -- 172.21.15.70:0/3781968097 wait complete. 2024-06-04T04:57:13.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.903+0000 7f82b10da640 1 Processor -- start 2024-06-04T04:57:13.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.903+0000 7f82b10da640 1 -- start start 2024-06-04T04:57:13.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.903+0000 7f82b10da640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82ac101990 0x7f82ac10c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:13.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.903+0000 7f82b10da640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82ac102b80 0x7f82ac10a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:13.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.903+0000 7f82b10da640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82ac103540 0x7f82ac10ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:13.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.903+0000 7f82b10da640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f82ac10e490 con 0x7f82ac103540 2024-06-04T04:57:13.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.903+0000 7f82b10da640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f82ac10e310 con 0x7f82ac101990 2024-06-04T04:57:13.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.903+0000 7f82b10da640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f82ac10e610 con 0x7f82ac102b80 2024-06-04T04:57:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.904+0000 7f82ab59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82ac103540 0x7f82ac10ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.904+0000 7f82aad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82ac101990 0x7f82ac10c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.904+0000 7f82ab59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82ac103540 0x7f82ac10ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33588/0 (socket says 172.21.15.70:33588) 2024-06-04T04:57:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.904+0000 7f82aa59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82ac102b80 0x7f82ac10a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.904+0000 7f82ab59e640 1 -- 172.21.15.70:0/1908210777 learned_addr learned my addr 172.21.15.70:0/1908210777 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.904+0000 7f82aa59c640 1 -- 172.21.15.70:0/1908210777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82ac101990 msgr2=0x7f82ac10c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.904+0000 7f82aa59c640 1 --2- 172.21.15.70:0/1908210777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82ac101990 0x7f82ac10c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.904+0000 7f82aa59c640 1 -- 172.21.15.70:0/1908210777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82ac103540 msgr2=0x7f82ac10ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.904+0000 7f82aa59c640 1 --2- 172.21.15.70:0/1908210777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82ac103540 0x7f82ac10ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.904+0000 7f82aa59c640 1 -- 172.21.15.70:0/1908210777 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82ac10b720 con 0x7f82ac102b80 2024-06-04T04:57:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.904+0000 7f82ab59e640 1 --2- 172.21.15.70:0/1908210777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82ac103540 0x7f82ac10ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:57:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.904+0000 7f82aa59c640 1 --2- 172.21.15.70:0/1908210777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82ac102b80 0x7f82ac10a950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f82a0005ab0 tx=0x7f82a00042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.904+0000 7f8293fff640 1 -- 172.21.15.70:0/1908210777 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a003f070 con 0x7f82ac102b80 2024-06-04T04:57:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.904+0000 7f82b10da640 1 -- 172.21.15.70:0/1908210777 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82ac1a2cc0 con 0x7f82ac102b80 2024-06-04T04:57:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.904+0000 7f8293fff640 1 -- 172.21.15.70:0/1908210777 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f82a0032e40 con 0x7f82ac102b80 2024-06-04T04:57:13.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.904+0000 7f8293fff640 1 -- 172.21.15.70:0/1908210777 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a0035070 con 0x7f82ac102b80 2024-06-04T04:57:13.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.904+0000 7f82b10da640 1 -- 172.21.15.70:0/1908210777 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82ac1a31b0 con 0x7f82ac102b80 2024-06-04T04:57:13.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.905+0000 7f82b10da640 1 -- 172.21.15.70:0/1908210777 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82ac108da0 con 0x7f82ac102b80 2024-06-04T04:57:13.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.907+0000 7f8293fff640 1 -- 172.21.15.70:0/1908210777 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 22) ==== 149772+0+0 (secure 0 0 0) 0x7f82a003b460 con 0x7f82ac102b80 2024-06-04T04:57:13.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.908+0000 7f8293fff640 1 --2- 172.21.15.70:0/1908210777 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f82940b2960 0x7f82940b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:13.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.908+0000 7f82aad9d640 1 --2- 172.21.15.70:0/1908210777 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f82940b2960 0x7f82940b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:13.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.909+0000 7f8293fff640 1 -- 172.21.15.70:0/1908210777 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f82a01033a0 con 0x7f82ac102b80 2024-06-04T04:57:13.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.909+0000 7f82aad9d640 1 --2- 172.21.15.70:0/1908210777 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f82940b2960 0x7f82940b4e20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f82ac1029e0 tx=0x7f829c00b390 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:13.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:13.911+0000 7f8293fff640 1 -- 172.21.15.70:0/1908210777 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f82a00ca730 con 0x7f82ac102b80 2024-06-04T04:57:14.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.105+0000 7f82b10da640 1 -- 172.21.15.70:0/1908210777 --> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f82ac062e90 con 0x7f82940b2960 2024-06-04T04:57:14.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.113+0000 7f8293fff640 1 -- 172.21.15.70:0/1908210777 <== mgr.24806 v2:172.21.15.70:6800/542036627 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258068 (secure 0 0 0) 0x7f82ac062e90 con 0x7f82940b2960 2024-06-04T04:57:14.118 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:57:14.119 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-06-04T04:57:14.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.116+0000 7f82b10da640 1 -- 172.21.15.70:0/1908210777 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f82940b2960 msgr2=0x7f82940b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:14.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.117+0000 7f82b10da640 1 --2- 172.21.15.70:0/1908210777 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f82940b2960 0x7f82940b4e20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f82ac1029e0 tx=0x7f829c00b390 comp rx=0 tx=0).stop 2024-06-04T04:57:14.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.117+0000 7f82b10da640 1 -- 172.21.15.70:0/1908210777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82ac102b80 msgr2=0x7f82ac10a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:14.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.117+0000 7f82b10da640 1 --2- 172.21.15.70:0/1908210777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82ac102b80 0x7f82ac10a950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f82a0005ab0 tx=0x7f82a00042f0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.117+0000 7f82b10da640 1 -- 172.21.15.70:0/1908210777 shutdown_connections 2024-06-04T04:57:14.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.117+0000 7f82b10da640 1 --2- 172.21.15.70:0/1908210777 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f82940b2960 0x7f82940b4e20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.117+0000 7f82b10da640 1 --2- 172.21.15.70:0/1908210777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82ac103540 0x7f82ac10ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.117+0000 7f82b10da640 1 --2- 172.21.15.70:0/1908210777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82ac102b80 0x7f82ac10a950 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.117+0000 7f82b10da640 1 --2- 172.21.15.70:0/1908210777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82ac101990 0x7f82ac10c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.117+0000 7f82b10da640 1 -- 172.21.15.70:0/1908210777 >> 172.21.15.70:0/1908210777 conn(0x7f82ac0fcf00 msgr2=0x7f82ac105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:14.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.117+0000 7f82b10da640 1 -- 172.21.15.70:0/1908210777 shutdown_connections 2024-06-04T04:57:14.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.117+0000 7f82b10da640 1 -- 172.21.15.70:0/1908210777 wait complete. 2024-06-04T04:57:14.130 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-06-04T04:57:13.959890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1196776,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2186,"num_read_kb":1240,"num_write":510,"num_write_kb":3628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2952512,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":902,"ondisk_log_size":902,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356092,"kb_used_data":11072,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585732,"statfs":{"total":1151940427776,"available":1151575789568,"internally_reserved":0,"allocated":11337728,"data_stored":8248864,"data_compressed":250839,"data_compressed_allocated":335872,"data_compressed_original":3072000,"omap_allocated":19138,"internal_metadata":353154366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.993238"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466420+0000","last_change":"2024-06-04T04:53:36.383755+0000","last_active":"2024-06-04T04:57:10.466420+0000","last_peered":"2024-06-04T04:57:10.466420+0000","last_clean":"2024-06-04T04:57:10.466420+0000","last_became_active":"2024-06-04T04:53:36.378274+0000","last_became_peered":"2024-06-04T04:53:36.378274+0000","last_unstale":"2024-06-04T04:57:10.466420+0000","last_undegraded":"2024-06-04T04:57:10.466420+0000","last_fullsized":"2024-06-04T04:57:10.466420+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:30:19.685994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466173+0000","last_change":"2024-06-04T04:53:36.372436+0000","last_active":"2024-06-04T04:57:10.466173+0000","last_peered":"2024-06-04T04:57:10.466173+0000","last_clean":"2024-06-04T04:57:10.466173+0000","last_became_active":"2024-06-04T04:53:36.372100+0000","last_became_peered":"2024-06-04T04:53:36.372100+0000","last_unstale":"2024-06-04T04:57:10.466173+0000","last_undegraded":"2024-06-04T04:57:10.466173+0000","last_fullsized":"2024-06-04T04:57:10.466173+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:42:31.646754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466063+0000","last_change":"2024-06-04T04:53:36.366915+0000","last_active":"2024-06-04T04:57:10.466063+0000","last_peered":"2024-06-04T04:57:10.466063+0000","last_clean":"2024-06-04T04:57:10.466063+0000","last_became_active":"2024-06-04T04:53:36.366611+0000","last_became_peered":"2024-06-04T04:53:36.366611+0000","last_unstale":"2024-06-04T04:57:10.466063+0000","last_undegraded":"2024-06-04T04:57:10.466063+0000","last_fullsized":"2024-06-04T04:57:10.466063+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:17:48.775938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466271+0000","last_change":"2024-06-04T04:53:36.374486+0000","last_active":"2024-06-04T04:57:10.466271+0000","last_peered":"2024-06-04T04:57:10.466271+0000","last_clean":"2024-06-04T04:57:10.466271+0000","last_became_active":"2024-06-04T04:53:36.374178+0000","last_became_peered":"2024-06-04T04:53:36.374178+0000","last_unstale":"2024-06-04T04:57:10.466271+0000","last_undegraded":"2024-06-04T04:57:10.466271+0000","last_fullsized":"2024-06-04T04:57:10.466271+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:30:28.287070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965508+0000","last_change":"2024-06-04T04:53:36.376183+0000","last_active":"2024-06-04T04:57:09.965508+0000","last_peered":"2024-06-04T04:57:09.965508+0000","last_clean":"2024-06-04T04:57:09.965508+0000","last_became_active":"2024-06-04T04:53:36.375790+0000","last_became_peered":"2024-06-04T04:53:36.375790+0000","last_unstale":"2024-06-04T04:57:09.965508+0000","last_undegraded":"2024-06-04T04:57:09.965508+0000","last_fullsized":"2024-06-04T04:57:09.965508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:58.697440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946349+0000","last_change":"2024-06-04T04:53:36.370249+0000","last_active":"2024-06-04T04:57:09.946349+0000","last_peered":"2024-06-04T04:57:09.946349+0000","last_clean":"2024-06-04T04:57:09.946349+0000","last_became_active":"2024-06-04T04:53:36.369916+0000","last_became_peered":"2024-06-04T04:53:36.369916+0000","last_unstale":"2024-06-04T04:57:09.946349+0000","last_undegraded":"2024-06-04T04:57:09.946349+0000","last_fullsized":"2024-06-04T04:57:09.946349+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:02:41.739806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213216+0000","last_change":"2024-06-04T04:53:36.384329+0000","last_active":"2024-06-04T04:57:10.213216+0000","last_peered":"2024-06-04T04:57:10.213216+0000","last_clean":"2024-06-04T04:57:10.213216+0000","last_became_active":"2024-06-04T04:53:36.382504+0000","last_became_peered":"2024-06-04T04:53:36.382504+0000","last_unstale":"2024-06-04T04:57:10.213216+0000","last_undegraded":"2024-06-04T04:57:10.213216+0000","last_fullsized":"2024-06-04T04:57:10.213216+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:51:07.604691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942376+0000","last_change":"2024-06-04T04:53:36.372319+0000","last_active":"2024-06-04T04:57:09.942376+0000","last_peered":"2024-06-04T04:57:09.942376+0000","last_clean":"2024-06-04T04:57:09.942376+0000","last_became_active":"2024-06-04T04:53:36.372000+0000","last_became_peered":"2024-06-04T04:53:36.372000+0000","last_unstale":"2024-06-04T04:57:09.942376+0000","last_undegraded":"2024-06-04T04:57:09.942376+0000","last_fullsized":"2024-06-04T04:57:09.942376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:15:56.906996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215880+0000","last_change":"2024-06-04T04:53:36.376051+0000","last_active":"2024-06-04T04:57:10.215880+0000","last_peered":"2024-06-04T04:57:10.215880+0000","last_clean":"2024-06-04T04:57:10.215880+0000","last_became_active":"2024-06-04T04:53:36.375726+0000","last_became_peered":"2024-06-04T04:53:36.375726+0000","last_unstale":"2024-06-04T04:57:10.215880+0000","last_undegraded":"2024-06-04T04:57:10.215880+0000","last_fullsized":"2024-06-04T04:57:10.215880+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:09:08.542764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946736+0000","last_change":"2024-06-04T04:53:36.373266+0000","last_active":"2024-06-04T04:57:09.946736+0000","last_peered":"2024-06-04T04:57:09.946736+0000","last_clean":"2024-06-04T04:57:09.946736+0000","last_became_active":"2024-06-04T04:53:36.372898+0000","last_became_peered":"2024-06-04T04:53:36.372898+0000","last_unstale":"2024-06-04T04:57:09.946736+0000","last_undegraded":"2024-06-04T04:57:09.946736+0000","last_fullsized":"2024-06-04T04:57:09.946736+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:41:43.436781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213610+0000","last_change":"2024-06-04T04:53:36.381344+0000","last_active":"2024-06-04T04:57:10.213610+0000","last_peered":"2024-06-04T04:57:10.213610+0000","last_clean":"2024-06-04T04:57:10.213610+0000","last_became_active":"2024-06-04T04:53:36.380736+0000","last_became_peered":"2024-06-04T04:53:36.380736+0000","last_unstale":"2024-06-04T04:57:10.213610+0000","last_undegraded":"2024-06-04T04:57:10.213610+0000","last_fullsized":"2024-06-04T04:57:10.213610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:23:01.607272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466124+0000","last_change":"2024-06-04T04:53:36.374035+0000","last_active":"2024-06-04T04:57:10.466124+0000","last_peered":"2024-06-04T04:57:10.466124+0000","last_clean":"2024-06-04T04:57:10.466124+0000","last_became_active":"2024-06-04T04:53:36.373667+0000","last_became_peered":"2024-06-04T04:53:36.373667+0000","last_unstale":"2024-06-04T04:57:10.466124+0000","last_undegraded":"2024-06-04T04:57:10.466124+0000","last_fullsized":"2024-06-04T04:57:10.466124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:23:07.880369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215527+0000","last_change":"2024-06-04T04:53:36.369613+0000","last_active":"2024-06-04T04:57:10.215527+0000","last_peered":"2024-06-04T04:57:10.215527+0000","last_clean":"2024-06-04T04:57:10.215527+0000","last_became_active":"2024-06-04T04:53:36.369277+0000","last_became_peered":"2024-06-04T04:53:36.369277+0000","last_unstale":"2024-06-04T04:57:10.215527+0000","last_undegraded":"2024-06-04T04:57:10.215527+0000","last_fullsized":"2024-06-04T04:57:10.215527+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:13:33.756609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.465960+0000","last_change":"2024-06-04T04:53:36.368094+0000","last_active":"2024-06-04T04:57:10.465960+0000","last_peered":"2024-06-04T04:57:10.465960+0000","last_clean":"2024-06-04T04:57:10.465960+0000","last_became_active":"2024-06-04T04:53:36.367717+0000","last_became_peered":"2024-06-04T04:53:36.367717+0000","last_unstale":"2024-06-04T04:57:10.465960+0000","last_undegraded":"2024-06-04T04:57:10.465960+0000","last_fullsized":"2024-06-04T04:57:10.465960+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:39:56.305078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.465864+0000","last_change":"2024-06-04T04:53:36.366765+0000","last_active":"2024-06-04T04:57:10.465864+0000","last_peered":"2024-06-04T04:57:10.465864+0000","last_clean":"2024-06-04T04:57:10.465864+0000","last_became_active":"2024-06-04T04:53:36.366438+0000","last_became_peered":"2024-06-04T04:53:36.366438+0000","last_unstale":"2024-06-04T04:57:10.465864+0000","last_undegraded":"2024-06-04T04:57:10.465864+0000","last_fullsized":"2024-06-04T04:57:10.465864+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:38:37.313926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215176+0000","last_change":"2024-06-04T04:53:36.380088+0000","last_active":"2024-06-04T04:57:10.215176+0000","last_peered":"2024-06-04T04:57:10.215176+0000","last_clean":"2024-06-04T04:57:10.215176+0000","last_became_active":"2024-06-04T04:53:36.379467+0000","last_became_peered":"2024-06-04T04:53:36.379467+0000","last_unstale":"2024-06-04T04:57:10.215176+0000","last_undegraded":"2024-06-04T04:57:10.215176+0000","last_fullsized":"2024-06-04T04:57:10.215176+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:46:53.236419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466550+0000","last_change":"2024-06-04T04:53:36.381126+0000","last_active":"2024-06-04T04:57:10.466550+0000","last_peered":"2024-06-04T04:57:10.466550+0000","last_clean":"2024-06-04T04:57:10.466550+0000","last_became_active":"2024-06-04T04:53:36.378833+0000","last_became_peered":"2024-06-04T04:53:36.378833+0000","last_unstale":"2024-06-04T04:57:10.466550+0000","last_undegraded":"2024-06-04T04:57:10.466550+0000","last_fullsized":"2024-06-04T04:57:10.466550+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:20:28.639273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.048240+0000","last_change":"2024-06-04T04:53:36.369692+0000","last_active":"2024-06-04T04:57:10.048240+0000","last_peered":"2024-06-04T04:57:10.048240+0000","last_clean":"2024-06-04T04:57:10.048240+0000","last_became_active":"2024-06-04T04:53:36.369370+0000","last_became_peered":"2024-06-04T04:53:36.369370+0000","last_unstale":"2024-06-04T04:57:10.048240+0000","last_undegraded":"2024-06-04T04:57:10.048240+0000","last_fullsized":"2024-06-04T04:57:10.048240+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:55.275465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219675+0000","last_change":"2024-06-04T04:53:36.383639+0000","last_active":"2024-06-04T04:57:10.219675+0000","last_peered":"2024-06-04T04:57:10.219675+0000","last_clean":"2024-06-04T04:57:10.219675+0000","last_became_active":"2024-06-04T04:53:36.383303+0000","last_became_peered":"2024-06-04T04:53:36.383303+0000","last_unstale":"2024-06-04T04:57:10.219675+0000","last_undegraded":"2024-06-04T04:57:10.219675+0000","last_fullsized":"2024-06-04T04:57:10.219675+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:26:31.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'15","reported_seq":262,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.048139+0000","last_change":"2024-06-04T04:54:19.641520+0000","last_active":"2024-06-04T04:57:10.048139+0000","last_peered":"2024-06-04T04:57:10.048139+0000","last_clean":"2024-06-04T04:57:10.048139+0000","last_became_active":"2024-06-04T04:54:19.632433+0000","last_became_peered":"2024-06-04T04:54:19.632433+0000","last_unstale":"2024-06-04T04:57:10.048139+0000","last_undegraded":"2024-06-04T04:57:10.048139+0000","last_fullsized":"2024-06-04T04:57:10.048139+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:26:05.993919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":192,"num_read_kb":192,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942727+0000","last_change":"2024-06-04T04:53:36.371827+0000","last_active":"2024-06-04T04:57:09.942727+0000","last_peered":"2024-06-04T04:57:09.942727+0000","last_clean":"2024-06-04T04:57:09.942727+0000","last_became_active":"2024-06-04T04:53:36.371509+0000","last_became_peered":"2024-06-04T04:53:36.371509+0000","last_unstale":"2024-06-04T04:57:09.942727+0000","last_undegraded":"2024-06-04T04:57:09.942727+0000","last_fullsized":"2024-06-04T04:57:09.942727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:54:51.589008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.943009+0000","last_change":"2024-06-04T04:53:33.353976+0000","last_active":"2024-06-04T04:57:09.943009+0000","last_peered":"2024-06-04T04:57:09.943009+0000","last_clean":"2024-06-04T04:57:09.943009+0000","last_became_active":"2024-06-04T04:53:33.353586+0000","last_became_peered":"2024-06-04T04:53:33.353586+0000","last_unstale":"2024-06-04T04:57:09.943009+0000","last_undegraded":"2024-06-04T04:57:09.943009+0000","last_fullsized":"2024-06-04T04:57:09.943009+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:17:10.378008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466390+0000","last_change":"2024-06-04T04:53:36.367806+0000","last_active":"2024-06-04T04:57:10.466390+0000","last_peered":"2024-06-04T04:57:10.466390+0000","last_clean":"2024-06-04T04:57:10.466390+0000","last_became_active":"2024-06-04T04:53:36.367484+0000","last_became_peered":"2024-06-04T04:53:36.367484+0000","last_unstale":"2024-06-04T04:57:10.466390+0000","last_undegraded":"2024-06-04T04:57:10.466390+0000","last_fullsized":"2024-06-04T04:57:10.466390+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:25:36.194746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"82'13","reported_seq":43,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965867+0000","last_change":"2024-06-04T04:53:33.348902+0000","last_active":"2024-06-04T04:57:09.965867+0000","last_peered":"2024-06-04T04:57:09.965867+0000","last_clean":"2024-06-04T04:57:09.965867+0000","last_became_active":"2024-06-04T04:53:33.348441+0000","last_became_peered":"2024-06-04T04:53:33.348441+0000","last_unstale":"2024-06-04T04:57:09.965867+0000","last_undegraded":"2024-06-04T04:57:09.965867+0000","last_fullsized":"2024-06-04T04:57:09.965867+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:22:28.428303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215265+0000","last_change":"2024-06-04T04:53:36.376867+0000","last_active":"2024-06-04T04:57:10.215265+0000","last_peered":"2024-06-04T04:57:10.215265+0000","last_clean":"2024-06-04T04:57:10.215265+0000","last_became_active":"2024-06-04T04:53:36.376517+0000","last_became_peered":"2024-06-04T04:53:36.376517+0000","last_unstale":"2024-06-04T04:57:10.215265+0000","last_undegraded":"2024-06-04T04:57:10.215265+0000","last_fullsized":"2024-06-04T04:57:10.215265+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:21:59.494321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965814+0000","last_change":"2024-06-04T04:53:33.347993+0000","last_active":"2024-06-04T04:57:09.965814+0000","last_peered":"2024-06-04T04:57:09.965814+0000","last_clean":"2024-06-04T04:57:09.965814+0000","last_became_active":"2024-06-04T04:53:33.347656+0000","last_became_peered":"2024-06-04T04:53:33.347656+0000","last_unstale":"2024-06-04T04:57:09.965814+0000","last_undegraded":"2024-06-04T04:57:09.965814+0000","last_fullsized":"2024-06-04T04:57:09.965814+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:21:26.806557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":44,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946660+0000","last_change":"2024-06-04T04:54:19.632790+0000","last_active":"2024-06-04T04:57:09.946660+0000","last_peered":"2024-06-04T04:57:09.946660+0000","last_clean":"2024-06-04T04:57:09.946660+0000","last_became_active":"2024-06-04T04:54:19.632338+0000","last_became_peered":"2024-06-04T04:54:19.632338+0000","last_unstale":"2024-06-04T04:57:09.946660+0000","last_undegraded":"2024-06-04T04:57:09.946660+0000","last_fullsized":"2024-06-04T04:57:09.946660+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:44:00.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"81'6","reported_seq":235,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946559+0000","last_change":"2024-06-04T04:53:33.346111+0000","last_active":"2024-06-04T04:57:09.946559+0000","last_peered":"2024-06-04T04:57:09.946559+0000","last_clean":"2024-06-04T04:57:09.946559+0000","last_became_active":"2024-06-04T04:53:33.345773+0000","last_became_peered":"2024-06-04T04:53:33.345773+0000","last_unstale":"2024-06-04T04:57:09.946559+0000","last_undegraded":"2024-06-04T04:57:09.946559+0000","last_fullsized":"2024-06-04T04:57:09.946559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:18.565583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1115,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":200,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965482+0000","last_change":"2024-06-04T04:53:36.373369+0000","last_active":"2024-06-04T04:57:09.965482+0000","last_peered":"2024-06-04T04:57:09.965482+0000","last_clean":"2024-06-04T04:57:09.965482+0000","last_became_active":"2024-06-04T04:53:36.373001+0000","last_became_peered":"2024-06-04T04:53:36.373001+0000","last_unstale":"2024-06-04T04:57:09.965482+0000","last_undegraded":"2024-06-04T04:57:09.965482+0000","last_fullsized":"2024-06-04T04:57:09.965482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:02:22.012643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946291+0000","last_change":"2024-06-04T04:53:33.344358+0000","last_active":"2024-06-04T04:57:09.946291+0000","last_peered":"2024-06-04T04:57:09.946291+0000","last_clean":"2024-06-04T04:57:09.946291+0000","last_became_active":"2024-06-04T04:53:33.343980+0000","last_became_peered":"2024-06-04T04:53:33.343980+0000","last_unstale":"2024-06-04T04:57:09.946291+0000","last_undegraded":"2024-06-04T04:57:09.946291+0000","last_fullsized":"2024-06-04T04:57:09.946291+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:14.493138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946579+0000","last_change":"2024-06-04T04:53:36.371239+0000","last_active":"2024-06-04T04:57:09.946579+0000","last_peered":"2024-06-04T04:57:09.946579+0000","last_clean":"2024-06-04T04:57:09.946579+0000","last_became_active":"2024-06-04T04:53:36.370885+0000","last_became_peered":"2024-06-04T04:53:36.370885+0000","last_unstale":"2024-06-04T04:57:09.946579+0000","last_undegraded":"2024-06-04T04:57:09.946579+0000","last_fullsized":"2024-06-04T04:57:09.946579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T04:54:29.753796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215252+0000","last_change":"2024-06-04T04:53:33.338512+0000","last_active":"2024-06-04T04:57:10.215252+0000","last_peered":"2024-06-04T04:57:10.215252+0000","last_clean":"2024-06-04T04:57:10.215252+0000","last_became_active":"2024-06-04T04:53:33.338180+0000","last_became_peered":"2024-06-04T04:53:33.338180+0000","last_unstale":"2024-06-04T04:57:10.215252+0000","last_undegraded":"2024-06-04T04:57:10.215252+0000","last_fullsized":"2024-06-04T04:57:10.215252+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:57:13.406479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942451+0000","last_change":"2024-06-04T04:53:36.377316+0000","last_active":"2024-06-04T04:57:09.942451+0000","last_peered":"2024-06-04T04:57:09.942451+0000","last_clean":"2024-06-04T04:57:09.942451+0000","last_became_active":"2024-06-04T04:53:36.376992+0000","last_became_peered":"2024-06-04T04:53:36.376992+0000","last_unstale":"2024-06-04T04:57:09.942451+0000","last_undegraded":"2024-06-04T04:57:09.942451+0000","last_fullsized":"2024-06-04T04:57:09.942451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:33.145631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946281+0000","last_change":"2024-06-04T04:53:33.348305+0000","last_active":"2024-06-04T04:57:09.946281+0000","last_peered":"2024-06-04T04:57:09.946281+0000","last_clean":"2024-06-04T04:57:09.946281+0000","last_became_active":"2024-06-04T04:53:33.347948+0000","last_became_peered":"2024-06-04T04:53:33.347948+0000","last_unstale":"2024-06-04T04:57:09.946281+0000","last_undegraded":"2024-06-04T04:57:09.946281+0000","last_fullsized":"2024-06-04T04:57:09.946281+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:04:13.669274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.047728+0000","last_change":"2024-06-04T04:53:36.367955+0000","last_active":"2024-06-04T04:57:10.047728+0000","last_peered":"2024-06-04T04:57:10.047728+0000","last_clean":"2024-06-04T04:57:10.047728+0000","last_became_active":"2024-06-04T04:53:36.367628+0000","last_became_peered":"2024-06-04T04:53:36.367628+0000","last_unstale":"2024-06-04T04:57:10.047728+0000","last_undegraded":"2024-06-04T04:57:10.047728+0000","last_fullsized":"2024-06-04T04:57:10.047728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:33:26.185498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":428,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.732694+0000","last_change":"2024-06-04T04:53:33.349471+0000","last_active":"2024-06-04T04:57:10.732694+0000","last_peered":"2024-06-04T04:57:10.732694+0000","last_clean":"2024-06-04T04:57:10.732694+0000","last_became_active":"2024-06-04T04:53:33.349105+0000","last_became_peered":"2024-06-04T04:53:33.349105+0000","last_unstale":"2024-06-04T04:57:10.732694+0000","last_undegraded":"2024-06-04T04:57:10.732694+0000","last_fullsized":"2024-06-04T04:57:10.732694+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:31:35.584275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":394,"num_read_kb":394,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215768+0000","last_change":"2024-06-04T04:53:36.367249+0000","last_active":"2024-06-04T04:57:10.215768+0000","last_peered":"2024-06-04T04:57:10.215768+0000","last_clean":"2024-06-04T04:57:10.215768+0000","last_became_active":"2024-06-04T04:53:36.366863+0000","last_became_peered":"2024-06-04T04:53:36.366863+0000","last_unstale":"2024-06-04T04:57:10.215768+0000","last_undegraded":"2024-06-04T04:57:10.215768+0000","last_fullsized":"2024-06-04T04:57:10.215768+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:13:17.308321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213781+0000","last_change":"2024-06-04T04:53:33.346778+0000","last_active":"2024-06-04T04:57:10.213781+0000","last_peered":"2024-06-04T04:57:10.213781+0000","last_clean":"2024-06-04T04:57:10.213781+0000","last_became_active":"2024-06-04T04:53:33.346376+0000","last_became_peered":"2024-06-04T04:53:33.346376+0000","last_unstale":"2024-06-04T04:57:10.213781+0000","last_undegraded":"2024-06-04T04:57:10.213781+0000","last_fullsized":"2024-06-04T04:57:10.213781+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:03:16.973822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466320+0000","last_change":"2024-06-04T04:54:19.514491+0000","last_active":"2024-06-04T04:57:10.466320+0000","last_peered":"2024-06-04T04:57:10.466320+0000","last_clean":"2024-06-04T04:57:10.466320+0000","last_became_active":"2024-06-04T04:54:19.514148+0000","last_became_peered":"2024-06-04T04:54:19.514148+0000","last_unstale":"2024-06-04T04:57:10.466320+0000","last_undegraded":"2024-06-04T04:57:10.466320+0000","last_fullsized":"2024-06-04T04:57:10.466320+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:09:59.641041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946792+0000","last_change":"2024-06-04T04:53:33.354989+0000","last_active":"2024-06-04T04:57:09.946792+0000","last_peered":"2024-06-04T04:57:09.946792+0000","last_clean":"2024-06-04T04:57:09.946792+0000","last_became_active":"2024-06-04T04:53:33.354613+0000","last_became_peered":"2024-06-04T04:53:33.354613+0000","last_unstale":"2024-06-04T04:57:09.946792+0000","last_undegraded":"2024-06-04T04:57:09.946792+0000","last_fullsized":"2024-06-04T04:57:09.946792+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:19:59.398763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215492+0000","last_change":"2024-06-04T04:53:36.372216+0000","last_active":"2024-06-04T04:57:10.215492+0000","last_peered":"2024-06-04T04:57:10.215492+0000","last_clean":"2024-06-04T04:57:10.215492+0000","last_became_active":"2024-06-04T04:53:36.371820+0000","last_became_peered":"2024-06-04T04:53:36.371820+0000","last_unstale":"2024-06-04T04:57:10.215492+0000","last_undegraded":"2024-06-04T04:57:10.215492+0000","last_fullsized":"2024-06-04T04:57:10.215492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:55:59.887963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'40","reported_seq":474,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.753604+0000","last_change":"2024-06-04T04:53:33.345493+0000","last_active":"2024-06-04T04:57:10.753604+0000","last_peered":"2024-06-04T04:57:10.753604+0000","last_clean":"2024-06-04T04:57:10.753604+0000","last_became_active":"2024-06-04T04:53:33.345102+0000","last_became_peered":"2024-06-04T04:53:33.345102+0000","last_unstale":"2024-06-04T04:57:10.753604+0000","last_undegraded":"2024-06-04T04:57:10.753604+0000","last_fullsized":"2024-06-04T04:57:10.753604+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:50:27.853391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71650,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":406,"num_read_kb":273,"num_write":42,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965646+0000","last_change":"2024-06-04T04:53:36.382070+0000","last_active":"2024-06-04T04:57:09.965646+0000","last_peered":"2024-06-04T04:57:09.965646+0000","last_clean":"2024-06-04T04:57:09.965646+0000","last_became_active":"2024-06-04T04:53:36.379144+0000","last_became_peered":"2024-06-04T04:53:36.379144+0000","last_unstale":"2024-06-04T04:57:09.965646+0000","last_undegraded":"2024-06-04T04:57:09.965646+0000","last_fullsized":"2024-06-04T04:57:09.965646+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:16.702709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219432+0000","last_change":"2024-06-04T04:53:33.344726+0000","last_active":"2024-06-04T04:57:10.219432+0000","last_peered":"2024-06-04T04:57:10.219432+0000","last_clean":"2024-06-04T04:57:10.219432+0000","last_became_active":"2024-06-04T04:53:33.344360+0000","last_became_peered":"2024-06-04T04:53:33.344360+0000","last_unstale":"2024-06-04T04:57:10.219432+0000","last_undegraded":"2024-06-04T04:57:10.219432+0000","last_fullsized":"2024-06-04T04:57:10.219432+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:44:56.784808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215391+0000","last_change":"2024-06-04T04:53:36.379036+0000","last_active":"2024-06-04T04:57:10.215391+0000","last_peered":"2024-06-04T04:57:10.215391+0000","last_clean":"2024-06-04T04:57:10.215391+0000","last_became_active":"2024-06-04T04:53:36.378681+0000","last_became_peered":"2024-06-04T04:53:36.378681+0000","last_unstale":"2024-06-04T04:57:10.215391+0000","last_undegraded":"2024-06-04T04:57:10.215391+0000","last_fullsized":"2024-06-04T04:57:10.215391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:07:49.704731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946436+0000","last_change":"2024-06-04T04:53:33.345915+0000","last_active":"2024-06-04T04:57:09.946436+0000","last_peered":"2024-06-04T04:57:09.946436+0000","last_clean":"2024-06-04T04:57:09.946436+0000","last_became_active":"2024-06-04T04:53:33.345537+0000","last_became_peered":"2024-06-04T04:53:33.345537+0000","last_unstale":"2024-06-04T04:57:09.946436+0000","last_undegraded":"2024-06-04T04:57:09.946436+0000","last_fullsized":"2024-06-04T04:57:09.946436+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:06:44.086968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213503+0000","last_change":"2024-06-04T04:53:36.379094+0000","last_active":"2024-06-04T04:57:10.213503+0000","last_peered":"2024-06-04T04:57:10.213503+0000","last_clean":"2024-06-04T04:57:10.213503+0000","last_became_active":"2024-06-04T04:53:36.378763+0000","last_became_peered":"2024-06-04T04:53:36.378763+0000","last_unstale":"2024-06-04T04:57:10.213503+0000","last_undegraded":"2024-06-04T04:57:10.213503+0000","last_fullsized":"2024-06-04T04:57:10.213503+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:16.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'6","reported_seq":235,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.465965+0000","last_change":"2024-06-04T04:53:33.341342+0000","last_active":"2024-06-04T04:57:10.465965+0000","last_peered":"2024-06-04T04:57:10.465965+0000","last_clean":"2024-06-04T04:57:10.465965+0000","last_became_active":"2024-06-04T04:53:33.340862+0000","last_became_peered":"2024-06-04T04:53:33.340862+0000","last_unstale":"2024-06-04T04:57:10.465965+0000","last_undegraded":"2024-06-04T04:57:10.465965+0000","last_fullsized":"2024-06-04T04:57:10.465965+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:26.818063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":201,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213135+0000","last_change":"2024-06-04T04:53:36.383124+0000","last_active":"2024-06-04T04:57:10.213135+0000","last_peered":"2024-06-04T04:57:10.213135+0000","last_clean":"2024-06-04T04:57:10.213135+0000","last_became_active":"2024-06-04T04:53:36.382449+0000","last_became_peered":"2024-06-04T04:53:36.382449+0000","last_unstale":"2024-06-04T04:57:10.213135+0000","last_undegraded":"2024-06-04T04:57:10.213135+0000","last_fullsized":"2024-06-04T04:57:10.213135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:52:40.776003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":47,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965391+0000","last_change":"2024-06-04T04:53:33.354556+0000","last_active":"2024-06-04T04:57:09.965391+0000","last_peered":"2024-06-04T04:57:09.965391+0000","last_clean":"2024-06-04T04:57:09.965391+0000","last_became_active":"2024-06-04T04:53:33.354238+0000","last_became_peered":"2024-06-04T04:53:33.354238+0000","last_unstale":"2024-06-04T04:57:09.965391+0000","last_undegraded":"2024-06-04T04:57:09.965391+0000","last_fullsized":"2024-06-04T04:57:09.965391+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:54.697374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946345+0000","last_change":"2024-06-04T04:53:36.371387+0000","last_active":"2024-06-04T04:57:09.946345+0000","last_peered":"2024-06-04T04:57:09.946345+0000","last_clean":"2024-06-04T04:57:09.946345+0000","last_became_active":"2024-06-04T04:53:36.370852+0000","last_became_peered":"2024-06-04T04:53:36.370852+0000","last_unstale":"2024-06-04T04:57:09.946345+0000","last_undegraded":"2024-06-04T04:57:09.946345+0000","last_fullsized":"2024-06-04T04:57:09.946345+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:44:06.956210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'3","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965694+0000","last_change":"2024-06-04T04:53:33.349982+0000","last_active":"2024-06-04T04:57:09.965694+0000","last_peered":"2024-06-04T04:57:09.965694+0000","last_clean":"2024-06-04T04:57:09.965694+0000","last_became_active":"2024-06-04T04:53:33.349633+0000","last_became_peered":"2024-06-04T04:53:33.349633+0000","last_unstale":"2024-06-04T04:57:09.965694+0000","last_undegraded":"2024-06-04T04:57:09.965694+0000","last_fullsized":"2024-06-04T04:57:09.965694+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:40.107426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466355+0000","last_change":"2024-06-04T04:53:36.382313+0000","last_active":"2024-06-04T04:57:10.466355+0000","last_peered":"2024-06-04T04:57:10.466355+0000","last_clean":"2024-06-04T04:57:10.466355+0000","last_became_active":"2024-06-04T04:53:36.379638+0000","last_became_peered":"2024-06-04T04:53:36.379638+0000","last_unstale":"2024-06-04T04:57:10.466355+0000","last_undegraded":"2024-06-04T04:57:10.466355+0000","last_fullsized":"2024-06-04T04:57:10.466355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:43:42.527262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.048030+0000","last_change":"2024-06-04T04:54:19.513581+0000","last_active":"2024-06-04T04:57:10.048030+0000","last_peered":"2024-06-04T04:57:10.048030+0000","last_clean":"2024-06-04T04:57:10.048030+0000","last_became_active":"2024-06-04T04:54:19.513111+0000","last_became_peered":"2024-06-04T04:54:19.513111+0000","last_unstale":"2024-06-04T04:57:10.048030+0000","last_undegraded":"2024-06-04T04:57:10.048030+0000","last_fullsized":"2024-06-04T04:57:10.048030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:57.028083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946526+0000","last_change":"2024-06-04T04:53:36.369904+0000","last_active":"2024-06-04T04:57:09.946526+0000","last_peered":"2024-06-04T04:57:09.946526+0000","last_clean":"2024-06-04T04:57:09.946526+0000","last_became_active":"2024-06-04T04:53:36.367929+0000","last_became_peered":"2024-06-04T04:53:36.367929+0000","last_unstale":"2024-06-04T04:57:09.946526+0000","last_undegraded":"2024-06-04T04:57:09.946526+0000","last_fullsized":"2024-06-04T04:57:09.946526+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:37:08.721569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"82'10","reported_seq":241,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.900365+0000","last_change":"2024-06-04T04:53:33.342723+0000","last_active":"2024-06-04T04:57:10.900365+0000","last_peered":"2024-06-04T04:57:10.900365+0000","last_clean":"2024-06-04T04:57:10.900365+0000","last_became_active":"2024-06-04T04:53:33.342344+0000","last_became_peered":"2024-06-04T04:53:33.342344+0000","last_unstale":"2024-06-04T04:57:10.900365+0000","last_undegraded":"2024-06-04T04:57:10.900365+0000","last_fullsized":"2024-06-04T04:57:10.900365+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:59:50.147517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":202,"num_read_kb":5,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946315+0000","last_change":"2024-06-04T04:54:19.513534+0000","last_active":"2024-06-04T04:57:09.946315+0000","last_peered":"2024-06-04T04:57:09.946315+0000","last_clean":"2024-06-04T04:57:09.946315+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:57:09.946315+0000","last_undegraded":"2024-06-04T04:57:09.946315+0000","last_fullsized":"2024-06-04T04:57:09.946315+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:16:32.240932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946648+0000","last_change":"2024-06-04T04:53:36.371236+0000","last_active":"2024-06-04T04:57:09.946648+0000","last_peered":"2024-06-04T04:57:09.946648+0000","last_clean":"2024-06-04T04:57:09.946648+0000","last_became_active":"2024-06-04T04:53:36.370876+0000","last_became_peered":"2024-06-04T04:53:36.370876+0000","last_unstale":"2024-06-04T04:57:09.946648+0000","last_undegraded":"2024-06-04T04:57:09.946648+0000","last_fullsized":"2024-06-04T04:57:09.946648+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:15:56.751317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":232,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215195+0000","last_change":"2024-06-04T04:53:33.355642+0000","last_active":"2024-06-04T04:57:10.215195+0000","last_peered":"2024-06-04T04:57:10.215195+0000","last_clean":"2024-06-04T04:57:10.215195+0000","last_became_active":"2024-06-04T04:53:33.355187+0000","last_became_peered":"2024-06-04T04:53:33.355187+0000","last_unstale":"2024-06-04T04:57:10.215195+0000","last_undegraded":"2024-06-04T04:57:10.215195+0000","last_fullsized":"2024-06-04T04:57:10.215195+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:32:28.331518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":198,"num_read_kb":198,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.047902+0000","last_change":"2024-06-04T04:54:19.631750+0000","last_active":"2024-06-04T04:57:10.047902+0000","last_peered":"2024-06-04T04:57:10.047902+0000","last_clean":"2024-06-04T04:57:10.047902+0000","last_became_active":"2024-06-04T04:54:19.631485+0000","last_became_peered":"2024-06-04T04:54:19.631485+0000","last_unstale":"2024-06-04T04:57:10.047902+0000","last_undegraded":"2024-06-04T04:57:10.047902+0000","last_fullsized":"2024-06-04T04:57:10.047902+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:29:44.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466037+0000","last_change":"2024-06-04T04:53:33.340653+0000","last_active":"2024-06-04T04:57:10.466037+0000","last_peered":"2024-06-04T04:57:10.466037+0000","last_clean":"2024-06-04T04:57:10.466037+0000","last_became_active":"2024-06-04T04:53:33.340250+0000","last_became_peered":"2024-06-04T04:53:33.340250+0000","last_unstale":"2024-06-04T04:57:10.466037+0000","last_undegraded":"2024-06-04T04:57:10.466037+0000","last_fullsized":"2024-06-04T04:57:10.466037+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:46:20.329514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":103,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213299+0000","last_change":"2024-06-04T04:52:09.181155+0000","last_active":"2024-06-04T04:57:10.213299+0000","last_peered":"2024-06-04T04:57:10.213299+0000","last_clean":"2024-06-04T04:57:10.213299+0000","last_became_active":"2024-06-04T04:52:09.177002+0000","last_became_peered":"2024-06-04T04:52:09.177002+0000","last_unstale":"2024-06-04T04:57:10.213299+0000","last_undegraded":"2024-06-04T04:57:10.213299+0000","last_fullsized":"2024-06-04T04:57:10.213299+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_clean_scrub_stamp":"2024-06-04T04:48:44.631948+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:46:57.735302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215170+0000","last_change":"2024-06-04T04:53:36.372634+0000","last_active":"2024-06-04T04:57:10.215170+0000","last_peered":"2024-06-04T04:57:10.215170+0000","last_clean":"2024-06-04T04:57:10.215170+0000","last_became_active":"2024-06-04T04:53:36.372290+0000","last_became_peered":"2024-06-04T04:53:36.372290+0000","last_unstale":"2024-06-04T04:57:10.215170+0000","last_undegraded":"2024-06-04T04:57:10.215170+0000","last_fullsized":"2024-06-04T04:57:10.215170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:14:07.460382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213392+0000","last_change":"2024-06-04T04:53:33.348760+0000","last_active":"2024-06-04T04:57:10.213392+0000","last_peered":"2024-06-04T04:57:10.213392+0000","last_clean":"2024-06-04T04:57:10.213392+0000","last_became_active":"2024-06-04T04:53:33.348347+0000","last_became_peered":"2024-06-04T04:53:33.348347+0000","last_unstale":"2024-06-04T04:57:10.213392+0000","last_undegraded":"2024-06-04T04:57:10.213392+0000","last_fullsized":"2024-06-04T04:57:10.213392+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:36:20.641061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965676+0000","last_change":"2024-06-04T04:53:33.354840+0000","last_active":"2024-06-04T04:57:09.965676+0000","last_peered":"2024-06-04T04:57:09.965676+0000","last_clean":"2024-06-04T04:57:09.965676+0000","last_became_active":"2024-06-04T04:53:33.354287+0000","last_became_peered":"2024-06-04T04:53:33.354287+0000","last_unstale":"2024-06-04T04:57:09.965676+0000","last_undegraded":"2024-06-04T04:57:09.965676+0000","last_fullsized":"2024-06-04T04:57:09.965676+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:42.550098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215686+0000","last_change":"2024-06-04T04:53:36.372678+0000","last_active":"2024-06-04T04:57:10.215686+0000","last_peered":"2024-06-04T04:57:10.215686+0000","last_clean":"2024-06-04T04:57:10.215686+0000","last_became_active":"2024-06-04T04:53:36.372312+0000","last_became_peered":"2024-06-04T04:53:36.372312+0000","last_unstale":"2024-06-04T04:57:10.215686+0000","last_undegraded":"2024-06-04T04:57:10.215686+0000","last_fullsized":"2024-06-04T04:57:10.215686+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:34:41.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'51","reported_seq":103,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.173057+0000","last_change":"2024-06-04T04:53:33.348440+0000","last_active":"2024-06-04T04:57:10.173057+0000","last_peered":"2024-06-04T04:57:10.173057+0000","last_clean":"2024-06-04T04:57:10.173057+0000","last_became_active":"2024-06-04T04:53:33.348131+0000","last_became_peered":"2024-06-04T04:53:33.348131+0000","last_unstale":"2024-06-04T04:57:10.173057+0000","last_undegraded":"2024-06-04T04:57:10.173057+0000","last_fullsized":"2024-06-04T04:57:10.173057+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:14:11.552286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":91,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942741+0000","last_change":"2024-06-04T04:53:36.378669+0000","last_active":"2024-06-04T04:57:09.942741+0000","last_peered":"2024-06-04T04:57:09.942741+0000","last_clean":"2024-06-04T04:57:09.942741+0000","last_became_active":"2024-06-04T04:53:36.378233+0000","last_became_peered":"2024-06-04T04:53:36.378233+0000","last_unstale":"2024-06-04T04:57:09.942741+0000","last_undegraded":"2024-06-04T04:57:09.942741+0000","last_fullsized":"2024-06-04T04:57:09.942741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:36:45.870924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466139+0000","last_change":"2024-06-04T04:53:33.345078+0000","last_active":"2024-06-04T04:57:10.466139+0000","last_peered":"2024-06-04T04:57:10.466139+0000","last_clean":"2024-06-04T04:57:10.466139+0000","last_became_active":"2024-06-04T04:53:33.344708+0000","last_became_peered":"2024-06-04T04:53:33.344708+0000","last_unstale":"2024-06-04T04:57:10.466139+0000","last_undegraded":"2024-06-04T04:57:10.466139+0000","last_fullsized":"2024-06-04T04:57:10.466139+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:45:14.010758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466300+0000","last_change":"2024-06-04T04:53:36.364817+0000","last_active":"2024-06-04T04:57:10.466300+0000","last_peered":"2024-06-04T04:57:10.466300+0000","last_clean":"2024-06-04T04:57:10.466300+0000","last_became_active":"2024-06-04T04:53:36.364499+0000","last_became_peered":"2024-06-04T04:53:36.364499+0000","last_unstale":"2024-06-04T04:57:10.466300+0000","last_undegraded":"2024-06-04T04:57:10.466300+0000","last_fullsized":"2024-06-04T04:57:10.466300+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:39:30.518028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215210+0000","last_change":"2024-06-04T04:53:33.354527+0000","last_active":"2024-06-04T04:57:10.215210+0000","last_peered":"2024-06-04T04:57:10.215210+0000","last_clean":"2024-06-04T04:57:10.215210+0000","last_became_active":"2024-06-04T04:53:33.354175+0000","last_became_peered":"2024-06-04T04:53:33.354175+0000","last_unstale":"2024-06-04T04:57:10.215210+0000","last_undegraded":"2024-06-04T04:57:10.215210+0000","last_fullsized":"2024-06-04T04:57:10.215210+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:33:08.125428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965391+0000","last_change":"2024-06-04T04:53:36.373248+0000","last_active":"2024-06-04T04:57:09.965391+0000","last_peered":"2024-06-04T04:57:09.965391+0000","last_clean":"2024-06-04T04:57:09.965391+0000","last_became_active":"2024-06-04T04:53:36.372915+0000","last_became_peered":"2024-06-04T04:53:36.372915+0000","last_unstale":"2024-06-04T04:57:09.965391+0000","last_undegraded":"2024-06-04T04:57:09.965391+0000","last_fullsized":"2024-06-04T04:57:09.965391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:23:11.689349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942495+0000","last_change":"2024-06-04T04:53:33.340427+0000","last_active":"2024-06-04T04:57:09.942495+0000","last_peered":"2024-06-04T04:57:09.942495+0000","last_clean":"2024-06-04T04:57:09.942495+0000","last_became_active":"2024-06-04T04:53:33.340022+0000","last_became_peered":"2024-06-04T04:53:33.340022+0000","last_unstale":"2024-06-04T04:57:09.942495+0000","last_undegraded":"2024-06-04T04:57:09.942495+0000","last_fullsized":"2024-06-04T04:57:09.942495+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:56:16.496471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466096+0000","last_change":"2024-06-04T04:53:36.380114+0000","last_active":"2024-06-04T04:57:10.466096+0000","last_peered":"2024-06-04T04:57:10.466096+0000","last_clean":"2024-06-04T04:57:10.466096+0000","last_became_active":"2024-06-04T04:53:36.379392+0000","last_became_peered":"2024-06-04T04:53:36.379392+0000","last_unstale":"2024-06-04T04:57:10.466096+0000","last_undegraded":"2024-06-04T04:57:10.466096+0000","last_fullsized":"2024-06-04T04:57:10.466096+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:41:48.776439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"81'5","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466122+0000","last_change":"2024-06-04T04:54:19.519755+0000","last_active":"2024-06-04T04:57:10.466122+0000","last_peered":"2024-06-04T04:57:10.466122+0000","last_clean":"2024-06-04T04:57:10.466122+0000","last_became_active":"2024-06-04T04:54:19.513992+0000","last_became_peered":"2024-06-04T04:54:19.513992+0000","last_unstale":"2024-06-04T04:57:10.466122+0000","last_undegraded":"2024-06-04T04:57:10.466122+0000","last_fullsized":"2024-06-04T04:57:10.466122+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:01.951935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942779+0000","last_change":"2024-06-04T04:53:36.377420+0000","last_active":"2024-06-04T04:57:09.942779+0000","last_peered":"2024-06-04T04:57:09.942779+0000","last_clean":"2024-06-04T04:57:09.942779+0000","last_became_active":"2024-06-04T04:53:36.377109+0000","last_became_peered":"2024-06-04T04:53:36.377109+0000","last_unstale":"2024-06-04T04:57:09.942779+0000","last_undegraded":"2024-06-04T04:57:09.942779+0000","last_fullsized":"2024-06-04T04:57:09.942779+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:47:12.102396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965575+0000","last_change":"2024-06-04T04:54:19.645328+0000","last_active":"2024-06-04T04:57:09.965575+0000","last_peered":"2024-06-04T04:57:09.965575+0000","last_clean":"2024-06-04T04:57:09.965575+0000","last_became_active":"2024-06-04T04:54:19.633416+0000","last_became_peered":"2024-06-04T04:54:19.633416+0000","last_unstale":"2024-06-04T04:57:09.965575+0000","last_undegraded":"2024-06-04T04:57:09.965575+0000","last_fullsized":"2024-06-04T04:57:09.965575+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:04:20.494641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213349+0000","last_change":"2024-06-04T04:53:36.381187+0000","last_active":"2024-06-04T04:57:10.213349+0000","last_peered":"2024-06-04T04:57:10.213349+0000","last_clean":"2024-06-04T04:57:10.213349+0000","last_became_active":"2024-06-04T04:53:36.380591+0000","last_became_peered":"2024-06-04T04:53:36.380591+0000","last_unstale":"2024-06-04T04:57:10.213349+0000","last_undegraded":"2024-06-04T04:57:10.213349+0000","last_fullsized":"2024-06-04T04:57:10.213349+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:26:43.122339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'6","reported_seq":234,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219295+0000","last_change":"2024-06-04T04:53:33.355908+0000","last_active":"2024-06-04T04:57:10.219295+0000","last_peered":"2024-06-04T04:57:10.219295+0000","last_clean":"2024-06-04T04:57:10.219295+0000","last_became_active":"2024-06-04T04:53:33.355522+0000","last_became_peered":"2024-06-04T04:53:33.355522+0000","last_unstale":"2024-06-04T04:57:10.219295+0000","last_undegraded":"2024-06-04T04:57:10.219295+0000","last_fullsized":"2024-06-04T04:57:10.219295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:16:44.639431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":200,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213745+0000","last_change":"2024-06-04T04:53:36.382737+0000","last_active":"2024-06-04T04:57:10.213745+0000","last_peered":"2024-06-04T04:57:10.213745+0000","last_clean":"2024-06-04T04:57:10.213745+0000","last_became_active":"2024-06-04T04:53:36.382353+0000","last_became_peered":"2024-06-04T04:53:36.382353+0000","last_unstale":"2024-06-04T04:57:10.213745+0000","last_undegraded":"2024-06-04T04:57:10.213745+0000","last_fullsized":"2024-06-04T04:57:10.213745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:18:01.476058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'41","reported_seq":78,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.170281+0000","last_change":"2024-06-04T04:53:33.347273+0000","last_active":"2024-06-04T04:57:10.170281+0000","last_peered":"2024-06-04T04:57:10.170281+0000","last_clean":"2024-06-04T04:57:10.170281+0000","last_became_active":"2024-06-04T04:53:33.346915+0000","last_became_peered":"2024-06-04T04:53:33.346915+0000","last_unstale":"2024-06-04T04:57:10.170281+0000","last_undegraded":"2024-06-04T04:57:10.170281+0000","last_fullsized":"2024-06-04T04:57:10.170281+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:17.235484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213559+0000","last_change":"2024-06-04T04:53:36.383454+0000","last_active":"2024-06-04T04:57:10.213559+0000","last_peered":"2024-06-04T04:57:10.213559+0000","last_clean":"2024-06-04T04:57:10.213559+0000","last_became_active":"2024-06-04T04:53:36.383105+0000","last_became_peered":"2024-06-04T04:53:36.383105+0000","last_unstale":"2024-06-04T04:57:10.213559+0000","last_undegraded":"2024-06-04T04:57:10.213559+0000","last_fullsized":"2024-06-04T04:57:10.213559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:04:02.147688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215701+0000","last_change":"2024-06-04T04:53:33.345435+0000","last_active":"2024-06-04T04:57:10.215701+0000","last_peered":"2024-06-04T04:57:10.215701+0000","last_clean":"2024-06-04T04:57:10.215701+0000","last_became_active":"2024-06-04T04:53:33.345077+0000","last_became_peered":"2024-06-04T04:53:33.345077+0000","last_unstale":"2024-06-04T04:57:10.215701+0000","last_undegraded":"2024-06-04T04:57:10.215701+0000","last_fullsized":"2024-06-04T04:57:10.215701+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:20:13.992259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213743+0000","last_change":"2024-06-04T04:53:36.385238+0000","last_active":"2024-06-04T04:57:10.213743+0000","last_peered":"2024-06-04T04:57:10.213743+0000","last_clean":"2024-06-04T04:57:10.213743+0000","last_became_active":"2024-06-04T04:53:36.384899+0000","last_became_peered":"2024-06-04T04:53:36.384899+0000","last_unstale":"2024-06-04T04:57:10.213743+0000","last_undegraded":"2024-06-04T04:57:10.213743+0000","last_fullsized":"2024-06-04T04:57:10.213743+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:09:42.629340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466183+0000","last_change":"2024-06-04T04:53:36.383450+0000","last_active":"2024-06-04T04:57:10.466183+0000","last_peered":"2024-06-04T04:57:10.466183+0000","last_clean":"2024-06-04T04:57:10.466183+0000","last_became_active":"2024-06-04T04:53:36.382678+0000","last_became_peered":"2024-06-04T04:53:36.382678+0000","last_unstale":"2024-06-04T04:57:10.466183+0000","last_undegraded":"2024-06-04T04:57:10.466183+0000","last_fullsized":"2024-06-04T04:57:10.466183+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:36:28.932636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219063+0000","last_change":"2024-06-04T04:54:19.632563+0000","last_active":"2024-06-04T04:57:10.219063+0000","last_peered":"2024-06-04T04:57:10.219063+0000","last_clean":"2024-06-04T04:57:10.219063+0000","last_became_active":"2024-06-04T04:54:19.632070+0000","last_became_peered":"2024-06-04T04:54:19.632070+0000","last_unstale":"2024-06-04T04:57:10.219063+0000","last_undegraded":"2024-06-04T04:57:10.219063+0000","last_fullsized":"2024-06-04T04:57:10.219063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:07:16.552958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219185+0000","last_change":"2024-06-04T04:53:36.381408+0000","last_active":"2024-06-04T04:57:10.219185+0000","last_peered":"2024-06-04T04:57:10.219185+0000","last_clean":"2024-06-04T04:57:10.219185+0000","last_became_active":"2024-06-04T04:53:36.381078+0000","last_became_peered":"2024-06-04T04:53:36.381078+0000","last_unstale":"2024-06-04T04:57:10.219185+0000","last_undegraded":"2024-06-04T04:57:10.219185+0000","last_fullsized":"2024-06-04T04:57:10.219185+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:28:48.988591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466464+0000","last_change":"2024-06-04T04:53:36.360168+0000","last_active":"2024-06-04T04:57:10.466464+0000","last_peered":"2024-06-04T04:57:10.466464+0000","last_clean":"2024-06-04T04:57:10.466464+0000","last_became_active":"2024-06-04T04:53:36.359691+0000","last_became_peered":"2024-06-04T04:53:36.359691+0000","last_unstale":"2024-06-04T04:57:10.466464+0000","last_undegraded":"2024-06-04T04:57:10.466464+0000","last_fullsized":"2024-06-04T04:57:10.466464+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:55:17.418339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215688+0000","last_change":"2024-06-04T04:53:36.363018+0000","last_active":"2024-06-04T04:57:10.215688+0000","last_peered":"2024-06-04T04:57:10.215688+0000","last_clean":"2024-06-04T04:57:10.215688+0000","last_became_active":"2024-06-04T04:53:36.361595+0000","last_became_peered":"2024-06-04T04:53:36.361595+0000","last_unstale":"2024-06-04T04:57:10.215688+0000","last_undegraded":"2024-06-04T04:57:10.215688+0000","last_fullsized":"2024-06-04T04:57:10.215688+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:00:21.377322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466289+0000","last_change":"2024-06-04T04:53:36.375377+0000","last_active":"2024-06-04T04:57:10.466289+0000","last_peered":"2024-06-04T04:57:10.466289+0000","last_clean":"2024-06-04T04:57:10.466289+0000","last_became_active":"2024-06-04T04:53:36.373459+0000","last_became_peered":"2024-06-04T04:53:36.373459+0000","last_unstale":"2024-06-04T04:57:10.466289+0000","last_undegraded":"2024-06-04T04:57:10.466289+0000","last_fullsized":"2024-06-04T04:57:10.466289+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:51:31.686969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219589+0000","last_change":"2024-06-04T04:53:36.373850+0000","last_active":"2024-06-04T04:57:10.219589+0000","last_peered":"2024-06-04T04:57:10.219589+0000","last_clean":"2024-06-04T04:57:10.219589+0000","last_became_active":"2024-06-04T04:53:36.371951+0000","last_became_peered":"2024-06-04T04:53:36.371951+0000","last_unstale":"2024-06-04T04:57:10.219589+0000","last_undegraded":"2024-06-04T04:57:10.219589+0000","last_fullsized":"2024-06-04T04:57:10.219589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:06.452703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942719+0000","last_change":"2024-06-04T04:53:36.374871+0000","last_active":"2024-06-04T04:57:09.942719+0000","last_peered":"2024-06-04T04:57:09.942719+0000","last_clean":"2024-06-04T04:57:09.942719+0000","last_became_active":"2024-06-04T04:53:36.374486+0000","last_became_peered":"2024-06-04T04:53:36.374486+0000","last_unstale":"2024-06-04T04:57:09.942719+0000","last_undegraded":"2024-06-04T04:57:09.942719+0000","last_fullsized":"2024-06-04T04:57:09.942719+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:15:13.337374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213230+0000","last_change":"2024-06-04T04:54:19.513540+0000","last_active":"2024-06-04T04:57:10.213230+0000","last_peered":"2024-06-04T04:57:10.213230+0000","last_clean":"2024-06-04T04:57:10.213230+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:57:10.213230+0000","last_undegraded":"2024-06-04T04:57:10.213230+0000","last_fullsized":"2024-06-04T04:57:10.213230+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:09:27.993585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942434+0000","last_change":"2024-06-04T04:53:36.373055+0000","last_active":"2024-06-04T04:57:09.942434+0000","last_peered":"2024-06-04T04:57:09.942434+0000","last_clean":"2024-06-04T04:57:09.942434+0000","last_became_active":"2024-06-04T04:53:36.372754+0000","last_became_peered":"2024-06-04T04:53:36.372754+0000","last_unstale":"2024-06-04T04:57:09.942434+0000","last_undegraded":"2024-06-04T04:57:09.942434+0000","last_fullsized":"2024-06-04T04:57:09.942434+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:22:27.554200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215415+0000","last_change":"2024-06-04T04:53:36.378897+0000","last_active":"2024-06-04T04:57:10.215415+0000","last_peered":"2024-06-04T04:57:10.215415+0000","last_clean":"2024-06-04T04:57:10.215415+0000","last_became_active":"2024-06-04T04:53:36.378607+0000","last_became_peered":"2024-06-04T04:53:36.378607+0000","last_unstale":"2024-06-04T04:57:10.215415+0000","last_undegraded":"2024-06-04T04:57:10.215415+0000","last_fullsized":"2024-06-04T04:57:10.215415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:03.395491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215502+0000","last_change":"2024-06-04T04:53:36.378240+0000","last_active":"2024-06-04T04:57:10.215502+0000","last_peered":"2024-06-04T04:57:10.215502+0000","last_clean":"2024-06-04T04:57:10.215502+0000","last_became_active":"2024-06-04T04:53:36.377909+0000","last_became_peered":"2024-06-04T04:53:36.377909+0000","last_unstale":"2024-06-04T04:57:10.215502+0000","last_undegraded":"2024-06-04T04:57:10.215502+0000","last_fullsized":"2024-06-04T04:57:10.215502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:15:25.288340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466255+0000","last_change":"2024-06-04T04:53:36.379792+0000","last_active":"2024-06-04T04:57:10.466255+0000","last_peered":"2024-06-04T04:57:10.466255+0000","last_clean":"2024-06-04T04:57:10.466255+0000","last_became_active":"2024-06-04T04:53:36.379438+0000","last_became_peered":"2024-06-04T04:53:36.379438+0000","last_unstale":"2024-06-04T04:57:10.466255+0000","last_undegraded":"2024-06-04T04:57:10.466255+0000","last_fullsized":"2024-06-04T04:57:10.466255+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:23:57.617877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108276,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":606174,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2105,"num_read_kb":1171,"num_write":385,"num_write_kb":1476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":724992,"data_stored":1818522,"data_compressed":81405,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":308,"ondisk_log_size":308,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678074,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":880640,"data_stored":430053,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874302,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1052672,"data_stored":1097012,"data_compressed":46576,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037828,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":429490,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234058,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":831397,"data_compressed":27797,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430285,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":1024006,"data_compressed":41851,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593812,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":901120,"data_stored":439999,"data_compressed":8651,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":498762,"data_compressed":7592,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738482,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":909312,"data_stored":431039,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411431,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":511961,"data_compressed":10484,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":962833,"data_compressed":33953,"data_compressed_allocated":45056,"data_compressed_original":540672,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953566,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32044,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713108,"statfs":{"total":95995035648,"available":95962222592,"internally_reserved":0,"allocated":888832,"data_stored":429561,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790041,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":999424,"data_stored":1162751,"data_compressed":50899,"data_compressed_allocated":65536,"data_compressed_original":737280,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":75881,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":74608,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":134913,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":394044,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":205479,"data_compressed":7989,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-04T04:57:14.134 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-04T04:57:14.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.273+0000 7f11894da640 1 -- 172.21.15.70:0/775827434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11841035b0 msgr2=0x7f11841059a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:14.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.273+0000 7f11894da640 1 --2- 172.21.15.70:0/775827434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11841035b0 0x7f11841059a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f117c009900 tx=0x7f117c02a410 comp rx=0 tx=0).stop 2024-06-04T04:57:14.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.273+0000 7f11894da640 1 -- 172.21.15.70:0/775827434 shutdown_connections 2024-06-04T04:57:14.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.273+0000 7f11894da640 1 --2- 172.21.15.70:0/775827434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1184105ee0 0x7f11841082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.273+0000 7f11894da640 1 --2- 172.21.15.70:0/775827434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1184108810 0x7f118410ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.273+0000 7f11894da640 1 --2- 172.21.15.70:0/775827434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11841035b0 0x7f11841059a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.273+0000 7f11894da640 1 -- 172.21.15.70:0/775827434 >> 172.21.15.70:0/775827434 conn(0x7f11840fcf40 msgr2=0x7f11840ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:14.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.273+0000 7f11894da640 1 -- 172.21.15.70:0/775827434 shutdown_connections 2024-06-04T04:57:14.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.273+0000 7f11894da640 1 -- 172.21.15.70:0/775827434 wait complete. 2024-06-04T04:57:14.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.273+0000 7f11894da640 1 Processor -- start 2024-06-04T04:57:14.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.273+0000 7f11894da640 1 -- start start 2024-06-04T04:57:14.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f11894da640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11841035b0 0x7f1184198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:14.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f11894da640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1184105ee0 0x7f11841991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:14.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f11894da640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1184108810 0x7f11841a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:14.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f11894da640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f118410e480 con 0x7f11841035b0 2024-06-04T04:57:14.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f11894da640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f118410e300 con 0x7f1184108810 2024-06-04T04:57:14.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f11894da640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f118410e600 con 0x7f1184105ee0 2024-06-04T04:57:14.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f1182ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11841035b0 0x7f1184198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:14.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f1182ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11841035b0 0x7f1184198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33614/0 (socket says 172.21.15.70:33614) 2024-06-04T04:57:14.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f1182ffd640 1 -- 172.21.15.70:0/1030170580 learned_addr learned my addr 172.21.15.70:0/1030170580 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:14.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f11827fc640 1 --2- 172.21.15.70:0/1030170580 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1184105ee0 0x7f11841991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:14.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f11837fe640 1 --2- 172.21.15.70:0/1030170580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1184108810 0x7f11841a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:14.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f1182ffd640 1 -- 172.21.15.70:0/1030170580 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1184105ee0 msgr2=0x7f11841991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:14.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f1182ffd640 1 --2- 172.21.15.70:0/1030170580 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1184105ee0 0x7f11841991a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f1182ffd640 1 -- 172.21.15.70:0/1030170580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1184108810 msgr2=0x7f11841a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:14.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f1182ffd640 1 --2- 172.21.15.70:0/1030170580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1184108810 0x7f11841a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f1182ffd640 1 -- 172.21.15.70:0/1030170580 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11841a0870 con 0x7f11841035b0 2024-06-04T04:57:14.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f1182ffd640 1 --2- 172.21.15.70:0/1030170580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11841035b0 0x7f1184198c60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f117c009a20 tx=0x7f117c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:14.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f117bfff640 1 -- 172.21.15.70:0/1030170580 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117c03f030 con 0x7f11841035b0 2024-06-04T04:57:14.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f11894da640 1 -- 172.21.15.70:0/1030170580 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11841a0b00 con 0x7f11841035b0 2024-06-04T04:57:14.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f117bfff640 1 -- 172.21.15.70:0/1030170580 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f117c003a80 con 0x7f11841035b0 2024-06-04T04:57:14.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.274+0000 7f117bfff640 1 -- 172.21.15.70:0/1030170580 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117c035070 con 0x7f11841035b0 2024-06-04T04:57:14.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.275+0000 7f11894da640 1 -- 172.21.15.70:0/1030170580 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11841a0ff0 con 0x7f11841035b0 2024-06-04T04:57:14.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.276+0000 7f11894da640 1 -- 172.21.15.70:0/1030170580 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11840fea30 con 0x7f11841035b0 2024-06-04T04:57:14.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.277+0000 7f117bfff640 1 -- 172.21.15.70:0/1030170580 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 22) ==== 149772+0+0 (secure 0 0 0) 0x7f117c042030 con 0x7f11841035b0 2024-06-04T04:57:14.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.278+0000 7f117bfff640 1 --2- 172.21.15.70:0/1030170580 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f11700b2960 0x7f11700b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:14.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.278+0000 7f11827fc640 1 --2- 172.21.15.70:0/1030170580 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f11700b2960 0x7f11700b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:14.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.278+0000 7f117bfff640 1 -- 172.21.15.70:0/1030170580 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f117c102ff0 con 0x7f11841035b0 2024-06-04T04:57:14.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.279+0000 7f11827fc640 1 --2- 172.21.15.70:0/1030170580 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f11700b2960 0x7f11700b4e20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f118406bbd0 tx=0x7f116c00b390 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:14.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.282+0000 7f117bfff640 1 -- 172.21.15.70:0/1030170580 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f117c0ca380 con 0x7f11841035b0 2024-06-04T04:57:14.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.469+0000 7f11894da640 1 -- 172.21.15.70:0/1030170580 --> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1184062e90 con 0x7f11700b2960 2024-06-04T04:57:14.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.476+0000 7f117bfff640 1 -- 172.21.15.70:0/1030170580 <== mgr.24806 v2:172.21.15.70:6800/542036627 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258068 (secure 0 0 0) 0x7f1184062e90 con 0x7f11700b2960 2024-06-04T04:57:14.481 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:57:14.482 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-06-04T04:57:14.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.479+0000 7f11894da640 1 -- 172.21.15.70:0/1030170580 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f11700b2960 msgr2=0x7f11700b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:14.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.479+0000 7f11894da640 1 --2- 172.21.15.70:0/1030170580 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f11700b2960 0x7f11700b4e20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f118406bbd0 tx=0x7f116c00b390 comp rx=0 tx=0).stop 2024-06-04T04:57:14.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.479+0000 7f11894da640 1 -- 172.21.15.70:0/1030170580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11841035b0 msgr2=0x7f1184198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:14.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.479+0000 7f11894da640 1 --2- 172.21.15.70:0/1030170580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11841035b0 0x7f1184198c60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f117c009a20 tx=0x7f117c0040c0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.479+0000 7f11894da640 1 -- 172.21.15.70:0/1030170580 shutdown_connections 2024-06-04T04:57:14.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.479+0000 7f11894da640 1 --2- 172.21.15.70:0/1030170580 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f11700b2960 0x7f11700b4e20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.479+0000 7f11894da640 1 --2- 172.21.15.70:0/1030170580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1184108810 0x7f11841a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.479+0000 7f11894da640 1 --2- 172.21.15.70:0/1030170580 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1184105ee0 0x7f11841991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.479+0000 7f11894da640 1 --2- 172.21.15.70:0/1030170580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11841035b0 0x7f1184198c60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.479+0000 7f11894da640 1 -- 172.21.15.70:0/1030170580 >> 172.21.15.70:0/1030170580 conn(0x7f11840fcf40 msgr2=0x7f1184105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:14.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.480+0000 7f11894da640 1 -- 172.21.15.70:0/1030170580 shutdown_connections 2024-06-04T04:57:14.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.480+0000 7f11894da640 1 -- 172.21.15.70:0/1030170580 wait complete. 2024-06-04T04:57:14.492 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:14 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.492 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:14 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.492 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:14 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.492 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:14 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.492 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:14 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:14 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:14 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi098 to 176.1M 2024-06-04T04:57:14.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:14 smithi070 ceph-mon[32631]: Unable to set osd_memory_target on smithi098 to 184732979: error parsing value: Value '184732979' is below minimum 939524096 2024-06-04T04:57:14.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:14 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:14 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:14 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:14 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:14 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:14 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:14 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:14 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.494 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-06-04T04:57:13.959890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1196776,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2186,"num_read_kb":1240,"num_write":510,"num_write_kb":3628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2952512,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":902,"ondisk_log_size":902,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356092,"kb_used_data":11072,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585732,"statfs":{"total":1151940427776,"available":1151575789568,"internally_reserved":0,"allocated":11337728,"data_stored":8248864,"data_compressed":250839,"data_compressed_allocated":335872,"data_compressed_original":3072000,"omap_allocated":19138,"internal_metadata":353154366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.993238"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466420+0000","last_change":"2024-06-04T04:53:36.383755+0000","last_active":"2024-06-04T04:57:10.466420+0000","last_peered":"2024-06-04T04:57:10.466420+0000","last_clean":"2024-06-04T04:57:10.466420+0000","last_became_active":"2024-06-04T04:53:36.378274+0000","last_became_peered":"2024-06-04T04:53:36.378274+0000","last_unstale":"2024-06-04T04:57:10.466420+0000","last_undegraded":"2024-06-04T04:57:10.466420+0000","last_fullsized":"2024-06-04T04:57:10.466420+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:30:19.685994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466173+0000","last_change":"2024-06-04T04:53:36.372436+0000","last_active":"2024-06-04T04:57:10.466173+0000","last_peered":"2024-06-04T04:57:10.466173+0000","last_clean":"2024-06-04T04:57:10.466173+0000","last_became_active":"2024-06-04T04:53:36.372100+0000","last_became_peered":"2024-06-04T04:53:36.372100+0000","last_unstale":"2024-06-04T04:57:10.466173+0000","last_undegraded":"2024-06-04T04:57:10.466173+0000","last_fullsized":"2024-06-04T04:57:10.466173+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:42:31.646754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466063+0000","last_change":"2024-06-04T04:53:36.366915+0000","last_active":"2024-06-04T04:57:10.466063+0000","last_peered":"2024-06-04T04:57:10.466063+0000","last_clean":"2024-06-04T04:57:10.466063+0000","last_became_active":"2024-06-04T04:53:36.366611+0000","last_became_peered":"2024-06-04T04:53:36.366611+0000","last_unstale":"2024-06-04T04:57:10.466063+0000","last_undegraded":"2024-06-04T04:57:10.466063+0000","last_fullsized":"2024-06-04T04:57:10.466063+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:17:48.775938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466271+0000","last_change":"2024-06-04T04:53:36.374486+0000","last_active":"2024-06-04T04:57:10.466271+0000","last_peered":"2024-06-04T04:57:10.466271+0000","last_clean":"2024-06-04T04:57:10.466271+0000","last_became_active":"2024-06-04T04:53:36.374178+0000","last_became_peered":"2024-06-04T04:53:36.374178+0000","last_unstale":"2024-06-04T04:57:10.466271+0000","last_undegraded":"2024-06-04T04:57:10.466271+0000","last_fullsized":"2024-06-04T04:57:10.466271+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:30:28.287070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965508+0000","last_change":"2024-06-04T04:53:36.376183+0000","last_active":"2024-06-04T04:57:09.965508+0000","last_peered":"2024-06-04T04:57:09.965508+0000","last_clean":"2024-06-04T04:57:09.965508+0000","last_became_active":"2024-06-04T04:53:36.375790+0000","last_became_peered":"2024-06-04T04:53:36.375790+0000","last_unstale":"2024-06-04T04:57:09.965508+0000","last_undegraded":"2024-06-04T04:57:09.965508+0000","last_fullsized":"2024-06-04T04:57:09.965508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:58.697440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946349+0000","last_change":"2024-06-04T04:53:36.370249+0000","last_active":"2024-06-04T04:57:09.946349+0000","last_peered":"2024-06-04T04:57:09.946349+0000","last_clean":"2024-06-04T04:57:09.946349+0000","last_became_active":"2024-06-04T04:53:36.369916+0000","last_became_peered":"2024-06-04T04:53:36.369916+0000","last_unstale":"2024-06-04T04:57:09.946349+0000","last_undegraded":"2024-06-04T04:57:09.946349+0000","last_fullsized":"2024-06-04T04:57:09.946349+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:02:41.739806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213216+0000","last_change":"2024-06-04T04:53:36.384329+0000","last_active":"2024-06-04T04:57:10.213216+0000","last_peered":"2024-06-04T04:57:10.213216+0000","last_clean":"2024-06-04T04:57:10.213216+0000","last_became_active":"2024-06-04T04:53:36.382504+0000","last_became_peered":"2024-06-04T04:53:36.382504+0000","last_unstale":"2024-06-04T04:57:10.213216+0000","last_undegraded":"2024-06-04T04:57:10.213216+0000","last_fullsized":"2024-06-04T04:57:10.213216+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:51:07.604691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942376+0000","last_change":"2024-06-04T04:53:36.372319+0000","last_active":"2024-06-04T04:57:09.942376+0000","last_peered":"2024-06-04T04:57:09.942376+0000","last_clean":"2024-06-04T04:57:09.942376+0000","last_became_active":"2024-06-04T04:53:36.372000+0000","last_became_peered":"2024-06-04T04:53:36.372000+0000","last_unstale":"2024-06-04T04:57:09.942376+0000","last_undegraded":"2024-06-04T04:57:09.942376+0000","last_fullsized":"2024-06-04T04:57:09.942376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:15:56.906996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215880+0000","last_change":"2024-06-04T04:53:36.376051+0000","last_active":"2024-06-04T04:57:10.215880+0000","last_peered":"2024-06-04T04:57:10.215880+0000","last_clean":"2024-06-04T04:57:10.215880+0000","last_became_active":"2024-06-04T04:53:36.375726+0000","last_became_peered":"2024-06-04T04:53:36.375726+0000","last_unstale":"2024-06-04T04:57:10.215880+0000","last_undegraded":"2024-06-04T04:57:10.215880+0000","last_fullsized":"2024-06-04T04:57:10.215880+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:09:08.542764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946736+0000","last_change":"2024-06-04T04:53:36.373266+0000","last_active":"2024-06-04T04:57:09.946736+0000","last_peered":"2024-06-04T04:57:09.946736+0000","last_clean":"2024-06-04T04:57:09.946736+0000","last_became_active":"2024-06-04T04:53:36.372898+0000","last_became_peered":"2024-06-04T04:53:36.372898+0000","last_unstale":"2024-06-04T04:57:09.946736+0000","last_undegraded":"2024-06-04T04:57:09.946736+0000","last_fullsized":"2024-06-04T04:57:09.946736+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:41:43.436781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213610+0000","last_change":"2024-06-04T04:53:36.381344+0000","last_active":"2024-06-04T04:57:10.213610+0000","last_peered":"2024-06-04T04:57:10.213610+0000","last_clean":"2024-06-04T04:57:10.213610+0000","last_became_active":"2024-06-04T04:53:36.380736+0000","last_became_peered":"2024-06-04T04:53:36.380736+0000","last_unstale":"2024-06-04T04:57:10.213610+0000","last_undegraded":"2024-06-04T04:57:10.213610+0000","last_fullsized":"2024-06-04T04:57:10.213610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:23:01.607272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466124+0000","last_change":"2024-06-04T04:53:36.374035+0000","last_active":"2024-06-04T04:57:10.466124+0000","last_peered":"2024-06-04T04:57:10.466124+0000","last_clean":"2024-06-04T04:57:10.466124+0000","last_became_active":"2024-06-04T04:53:36.373667+0000","last_became_peered":"2024-06-04T04:53:36.373667+0000","last_unstale":"2024-06-04T04:57:10.466124+0000","last_undegraded":"2024-06-04T04:57:10.466124+0000","last_fullsized":"2024-06-04T04:57:10.466124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:23:07.880369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215527+0000","last_change":"2024-06-04T04:53:36.369613+0000","last_active":"2024-06-04T04:57:10.215527+0000","last_peered":"2024-06-04T04:57:10.215527+0000","last_clean":"2024-06-04T04:57:10.215527+0000","last_became_active":"2024-06-04T04:53:36.369277+0000","last_became_peered":"2024-06-04T04:53:36.369277+0000","last_unstale":"2024-06-04T04:57:10.215527+0000","last_undegraded":"2024-06-04T04:57:10.215527+0000","last_fullsized":"2024-06-04T04:57:10.215527+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:13:33.756609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.465960+0000","last_change":"2024-06-04T04:53:36.368094+0000","last_active":"2024-06-04T04:57:10.465960+0000","last_peered":"2024-06-04T04:57:10.465960+0000","last_clean":"2024-06-04T04:57:10.465960+0000","last_became_active":"2024-06-04T04:53:36.367717+0000","last_became_peered":"2024-06-04T04:53:36.367717+0000","last_unstale":"2024-06-04T04:57:10.465960+0000","last_undegraded":"2024-06-04T04:57:10.465960+0000","last_fullsized":"2024-06-04T04:57:10.465960+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:39:56.305078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.465864+0000","last_change":"2024-06-04T04:53:36.366765+0000","last_active":"2024-06-04T04:57:10.465864+0000","last_peered":"2024-06-04T04:57:10.465864+0000","last_clean":"2024-06-04T04:57:10.465864+0000","last_became_active":"2024-06-04T04:53:36.366438+0000","last_became_peered":"2024-06-04T04:53:36.366438+0000","last_unstale":"2024-06-04T04:57:10.465864+0000","last_undegraded":"2024-06-04T04:57:10.465864+0000","last_fullsized":"2024-06-04T04:57:10.465864+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:38:37.313926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215176+0000","last_change":"2024-06-04T04:53:36.380088+0000","last_active":"2024-06-04T04:57:10.215176+0000","last_peered":"2024-06-04T04:57:10.215176+0000","last_clean":"2024-06-04T04:57:10.215176+0000","last_became_active":"2024-06-04T04:53:36.379467+0000","last_became_peered":"2024-06-04T04:53:36.379467+0000","last_unstale":"2024-06-04T04:57:10.215176+0000","last_undegraded":"2024-06-04T04:57:10.215176+0000","last_fullsized":"2024-06-04T04:57:10.215176+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:46:53.236419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466550+0000","last_change":"2024-06-04T04:53:36.381126+0000","last_active":"2024-06-04T04:57:10.466550+0000","last_peered":"2024-06-04T04:57:10.466550+0000","last_clean":"2024-06-04T04:57:10.466550+0000","last_became_active":"2024-06-04T04:53:36.378833+0000","last_became_peered":"2024-06-04T04:53:36.378833+0000","last_unstale":"2024-06-04T04:57:10.466550+0000","last_undegraded":"2024-06-04T04:57:10.466550+0000","last_fullsized":"2024-06-04T04:57:10.466550+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:20:28.639273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.048240+0000","last_change":"2024-06-04T04:53:36.369692+0000","last_active":"2024-06-04T04:57:10.048240+0000","last_peered":"2024-06-04T04:57:10.048240+0000","last_clean":"2024-06-04T04:57:10.048240+0000","last_became_active":"2024-06-04T04:53:36.369370+0000","last_became_peered":"2024-06-04T04:53:36.369370+0000","last_unstale":"2024-06-04T04:57:10.048240+0000","last_undegraded":"2024-06-04T04:57:10.048240+0000","last_fullsized":"2024-06-04T04:57:10.048240+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:55.275465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219675+0000","last_change":"2024-06-04T04:53:36.383639+0000","last_active":"2024-06-04T04:57:10.219675+0000","last_peered":"2024-06-04T04:57:10.219675+0000","last_clean":"2024-06-04T04:57:10.219675+0000","last_became_active":"2024-06-04T04:53:36.383303+0000","last_became_peered":"2024-06-04T04:53:36.383303+0000","last_unstale":"2024-06-04T04:57:10.219675+0000","last_undegraded":"2024-06-04T04:57:10.219675+0000","last_fullsized":"2024-06-04T04:57:10.219675+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:26:31.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'15","reported_seq":262,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.048139+0000","last_change":"2024-06-04T04:54:19.641520+0000","last_active":"2024-06-04T04:57:10.048139+0000","last_peered":"2024-06-04T04:57:10.048139+0000","last_clean":"2024-06-04T04:57:10.048139+0000","last_became_active":"2024-06-04T04:54:19.632433+0000","last_became_peered":"2024-06-04T04:54:19.632433+0000","last_unstale":"2024-06-04T04:57:10.048139+0000","last_undegraded":"2024-06-04T04:57:10.048139+0000","last_fullsized":"2024-06-04T04:57:10.048139+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:26:05.993919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":192,"num_read_kb":192,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942727+0000","last_change":"2024-06-04T04:53:36.371827+0000","last_active":"2024-06-04T04:57:09.942727+0000","last_peered":"2024-06-04T04:57:09.942727+0000","last_clean":"2024-06-04T04:57:09.942727+0000","last_became_active":"2024-06-04T04:53:36.371509+0000","last_became_peered":"2024-06-04T04:53:36.371509+0000","last_unstale":"2024-06-04T04:57:09.942727+0000","last_undegraded":"2024-06-04T04:57:09.942727+0000","last_fullsized":"2024-06-04T04:57:09.942727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:54:51.589008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.943009+0000","last_change":"2024-06-04T04:53:33.353976+0000","last_active":"2024-06-04T04:57:09.943009+0000","last_peered":"2024-06-04T04:57:09.943009+0000","last_clean":"2024-06-04T04:57:09.943009+0000","last_became_active":"2024-06-04T04:53:33.353586+0000","last_became_peered":"2024-06-04T04:53:33.353586+0000","last_unstale":"2024-06-04T04:57:09.943009+0000","last_undegraded":"2024-06-04T04:57:09.943009+0000","last_fullsized":"2024-06-04T04:57:09.943009+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:17:10.378008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466390+0000","last_change":"2024-06-04T04:53:36.367806+0000","last_active":"2024-06-04T04:57:10.466390+0000","last_peered":"2024-06-04T04:57:10.466390+0000","last_clean":"2024-06-04T04:57:10.466390+0000","last_became_active":"2024-06-04T04:53:36.367484+0000","last_became_peered":"2024-06-04T04:53:36.367484+0000","last_unstale":"2024-06-04T04:57:10.466390+0000","last_undegraded":"2024-06-04T04:57:10.466390+0000","last_fullsized":"2024-06-04T04:57:10.466390+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:25:36.194746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"82'13","reported_seq":43,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965867+0000","last_change":"2024-06-04T04:53:33.348902+0000","last_active":"2024-06-04T04:57:09.965867+0000","last_peered":"2024-06-04T04:57:09.965867+0000","last_clean":"2024-06-04T04:57:09.965867+0000","last_became_active":"2024-06-04T04:53:33.348441+0000","last_became_peered":"2024-06-04T04:53:33.348441+0000","last_unstale":"2024-06-04T04:57:09.965867+0000","last_undegraded":"2024-06-04T04:57:09.965867+0000","last_fullsized":"2024-06-04T04:57:09.965867+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:22:28.428303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215265+0000","last_change":"2024-06-04T04:53:36.376867+0000","last_active":"2024-06-04T04:57:10.215265+0000","last_peered":"2024-06-04T04:57:10.215265+0000","last_clean":"2024-06-04T04:57:10.215265+0000","last_became_active":"2024-06-04T04:53:36.376517+0000","last_became_peered":"2024-06-04T04:53:36.376517+0000","last_unstale":"2024-06-04T04:57:10.215265+0000","last_undegraded":"2024-06-04T04:57:10.215265+0000","last_fullsized":"2024-06-04T04:57:10.215265+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:21:59.494321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965814+0000","last_change":"2024-06-04T04:53:33.347993+0000","last_active":"2024-06-04T04:57:09.965814+0000","last_peered":"2024-06-04T04:57:09.965814+0000","last_clean":"2024-06-04T04:57:09.965814+0000","last_became_active":"2024-06-04T04:53:33.347656+0000","last_became_peered":"2024-06-04T04:53:33.347656+0000","last_unstale":"2024-06-04T04:57:09.965814+0000","last_undegraded":"2024-06-04T04:57:09.965814+0000","last_fullsized":"2024-06-04T04:57:09.965814+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:21:26.806557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":44,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946660+0000","last_change":"2024-06-04T04:54:19.632790+0000","last_active":"2024-06-04T04:57:09.946660+0000","last_peered":"2024-06-04T04:57:09.946660+0000","last_clean":"2024-06-04T04:57:09.946660+0000","last_became_active":"2024-06-04T04:54:19.632338+0000","last_became_peered":"2024-06-04T04:54:19.632338+0000","last_unstale":"2024-06-04T04:57:09.946660+0000","last_undegraded":"2024-06-04T04:57:09.946660+0000","last_fullsized":"2024-06-04T04:57:09.946660+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:44:00.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"81'6","reported_seq":235,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946559+0000","last_change":"2024-06-04T04:53:33.346111+0000","last_active":"2024-06-04T04:57:09.946559+0000","last_peered":"2024-06-04T04:57:09.946559+0000","last_clean":"2024-06-04T04:57:09.946559+0000","last_became_active":"2024-06-04T04:53:33.345773+0000","last_became_peered":"2024-06-04T04:53:33.345773+0000","last_unstale":"2024-06-04T04:57:09.946559+0000","last_undegraded":"2024-06-04T04:57:09.946559+0000","last_fullsized":"2024-06-04T04:57:09.946559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:18.565583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1115,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":200,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965482+0000","last_change":"2024-06-04T04:53:36.373369+0000","last_active":"2024-06-04T04:57:09.965482+0000","last_peered":"2024-06-04T04:57:09.965482+0000","last_clean":"2024-06-04T04:57:09.965482+0000","last_became_active":"2024-06-04T04:53:36.373001+0000","last_became_peered":"2024-06-04T04:53:36.373001+0000","last_unstale":"2024-06-04T04:57:09.965482+0000","last_undegraded":"2024-06-04T04:57:09.965482+0000","last_fullsized":"2024-06-04T04:57:09.965482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:02:22.012643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946291+0000","last_change":"2024-06-04T04:53:33.344358+0000","last_active":"2024-06-04T04:57:09.946291+0000","last_peered":"2024-06-04T04:57:09.946291+0000","last_clean":"2024-06-04T04:57:09.946291+0000","last_became_active":"2024-06-04T04:53:33.343980+0000","last_became_peered":"2024-06-04T04:53:33.343980+0000","last_unstale":"2024-06-04T04:57:09.946291+0000","last_undegraded":"2024-06-04T04:57:09.946291+0000","last_fullsized":"2024-06-04T04:57:09.946291+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:14.493138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946579+0000","last_change":"2024-06-04T04:53:36.371239+0000","last_active":"2024-06-04T04:57:09.946579+0000","last_peered":"2024-06-04T04:57:09.946579+0000","last_clean":"2024-06-04T04:57:09.946579+0000","last_became_active":"2024-06-04T04:53:36.370885+0000","last_became_peered":"2024-06-04T04:53:36.370885+0000","last_unstale":"2024-06-04T04:57:09.946579+0000","last_undegraded":"2024-06-04T04:57:09.946579+0000","last_fullsized":"2024-06-04T04:57:09.946579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T04:54:29.753796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215252+0000","last_change":"2024-06-04T04:53:33.338512+0000","last_active":"2024-06-04T04:57:10.215252+0000","last_peered":"2024-06-04T04:57:10.215252+0000","last_clean":"2024-06-04T04:57:10.215252+0000","last_became_active":"2024-06-04T04:53:33.338180+0000","last_became_peered":"2024-06-04T04:53:33.338180+0000","last_unstale":"2024-06-04T04:57:10.215252+0000","last_undegraded":"2024-06-04T04:57:10.215252+0000","last_fullsized":"2024-06-04T04:57:10.215252+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:57:13.406479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942451+0000","last_change":"2024-06-04T04:53:36.377316+0000","last_active":"2024-06-04T04:57:09.942451+0000","last_peered":"2024-06-04T04:57:09.942451+0000","last_clean":"2024-06-04T04:57:09.942451+0000","last_became_active":"2024-06-04T04:53:36.376992+0000","last_became_peered":"2024-06-04T04:53:36.376992+0000","last_unstale":"2024-06-04T04:57:09.942451+0000","last_undegraded":"2024-06-04T04:57:09.942451+0000","last_fullsized":"2024-06-04T04:57:09.942451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:33.145631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946281+0000","last_change":"2024-06-04T04:53:33.348305+0000","last_active":"2024-06-04T04:57:09.946281+0000","last_peered":"2024-06-04T04:57:09.946281+0000","last_clean":"2024-06-04T04:57:09.946281+0000","last_became_active":"2024-06-04T04:53:33.347948+0000","last_became_peered":"2024-06-04T04:53:33.347948+0000","last_unstale":"2024-06-04T04:57:09.946281+0000","last_undegraded":"2024-06-04T04:57:09.946281+0000","last_fullsized":"2024-06-04T04:57:09.946281+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:04:13.669274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.047728+0000","last_change":"2024-06-04T04:53:36.367955+0000","last_active":"2024-06-04T04:57:10.047728+0000","last_peered":"2024-06-04T04:57:10.047728+0000","last_clean":"2024-06-04T04:57:10.047728+0000","last_became_active":"2024-06-04T04:53:36.367628+0000","last_became_peered":"2024-06-04T04:53:36.367628+0000","last_unstale":"2024-06-04T04:57:10.047728+0000","last_undegraded":"2024-06-04T04:57:10.047728+0000","last_fullsized":"2024-06-04T04:57:10.047728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:33:26.185498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":428,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.732694+0000","last_change":"2024-06-04T04:53:33.349471+0000","last_active":"2024-06-04T04:57:10.732694+0000","last_peered":"2024-06-04T04:57:10.732694+0000","last_clean":"2024-06-04T04:57:10.732694+0000","last_became_active":"2024-06-04T04:53:33.349105+0000","last_became_peered":"2024-06-04T04:53:33.349105+0000","last_unstale":"2024-06-04T04:57:10.732694+0000","last_undegraded":"2024-06-04T04:57:10.732694+0000","last_fullsized":"2024-06-04T04:57:10.732694+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:31:35.584275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":394,"num_read_kb":394,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215768+0000","last_change":"2024-06-04T04:53:36.367249+0000","last_active":"2024-06-04T04:57:10.215768+0000","last_peered":"2024-06-04T04:57:10.215768+0000","last_clean":"2024-06-04T04:57:10.215768+0000","last_became_active":"2024-06-04T04:53:36.366863+0000","last_became_peered":"2024-06-04T04:53:36.366863+0000","last_unstale":"2024-06-04T04:57:10.215768+0000","last_undegraded":"2024-06-04T04:57:10.215768+0000","last_fullsized":"2024-06-04T04:57:10.215768+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:13:17.308321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213781+0000","last_change":"2024-06-04T04:53:33.346778+0000","last_active":"2024-06-04T04:57:10.213781+0000","last_peered":"2024-06-04T04:57:10.213781+0000","last_clean":"2024-06-04T04:57:10.213781+0000","last_became_active":"2024-06-04T04:53:33.346376+0000","last_became_peered":"2024-06-04T04:53:33.346376+0000","last_unstale":"2024-06-04T04:57:10.213781+0000","last_undegraded":"2024-06-04T04:57:10.213781+0000","last_fullsized":"2024-06-04T04:57:10.213781+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:03:16.973822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466320+0000","last_change":"2024-06-04T04:54:19.514491+0000","last_active":"2024-06-04T04:57:10.466320+0000","last_peered":"2024-06-04T04:57:10.466320+0000","last_clean":"2024-06-04T04:57:10.466320+0000","last_became_active":"2024-06-04T04:54:19.514148+0000","last_became_peered":"2024-06-04T04:54:19.514148+0000","last_unstale":"2024-06-04T04:57:10.466320+0000","last_undegraded":"2024-06-04T04:57:10.466320+0000","last_fullsized":"2024-06-04T04:57:10.466320+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:09:59.641041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946792+0000","last_change":"2024-06-04T04:53:33.354989+0000","last_active":"2024-06-04T04:57:09.946792+0000","last_peered":"2024-06-04T04:57:09.946792+0000","last_clean":"2024-06-04T04:57:09.946792+0000","last_became_active":"2024-06-04T04:53:33.354613+0000","last_became_peered":"2024-06-04T04:53:33.354613+0000","last_unstale":"2024-06-04T04:57:09.946792+0000","last_undegraded":"2024-06-04T04:57:09.946792+0000","last_fullsized":"2024-06-04T04:57:09.946792+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:19:59.398763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215492+0000","last_change":"2024-06-04T04:53:36.372216+0000","last_active":"2024-06-04T04:57:10.215492+0000","last_peered":"2024-06-04T04:57:10.215492+0000","last_clean":"2024-06-04T04:57:10.215492+0000","last_became_active":"2024-06-04T04:53:36.371820+0000","last_became_peered":"2024-06-04T04:53:36.371820+0000","last_unstale":"2024-06-04T04:57:10.215492+0000","last_undegraded":"2024-06-04T04:57:10.215492+0000","last_fullsized":"2024-06-04T04:57:10.215492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:55:59.887963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'40","reported_seq":474,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.753604+0000","last_change":"2024-06-04T04:53:33.345493+0000","last_active":"2024-06-04T04:57:10.753604+0000","last_peered":"2024-06-04T04:57:10.753604+0000","last_clean":"2024-06-04T04:57:10.753604+0000","last_became_active":"2024-06-04T04:53:33.345102+0000","last_became_peered":"2024-06-04T04:53:33.345102+0000","last_unstale":"2024-06-04T04:57:10.753604+0000","last_undegraded":"2024-06-04T04:57:10.753604+0000","last_fullsized":"2024-06-04T04:57:10.753604+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:50:27.853391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71650,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":406,"num_read_kb":273,"num_write":42,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965646+0000","last_change":"2024-06-04T04:53:36.382070+0000","last_active":"2024-06-04T04:57:09.965646+0000","last_peered":"2024-06-04T04:57:09.965646+0000","last_clean":"2024-06-04T04:57:09.965646+0000","last_became_active":"2024-06-04T04:53:36.379144+0000","last_became_peered":"2024-06-04T04:53:36.379144+0000","last_unstale":"2024-06-04T04:57:09.965646+0000","last_undegraded":"2024-06-04T04:57:09.965646+0000","last_fullsized":"2024-06-04T04:57:09.965646+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:16.702709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219432+0000","last_change":"2024-06-04T04:53:33.344726+0000","last_active":"2024-06-04T04:57:10.219432+0000","last_peered":"2024-06-04T04:57:10.219432+0000","last_clean":"2024-06-04T04:57:10.219432+0000","last_became_active":"2024-06-04T04:53:33.344360+0000","last_became_peered":"2024-06-04T04:53:33.344360+0000","last_unstale":"2024-06-04T04:57:10.219432+0000","last_undegraded":"2024-06-04T04:57:10.219432+0000","last_fullsized":"2024-06-04T04:57:10.219432+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:44:56.784808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215391+0000","last_change":"2024-06-04T04:53:36.379036+0000","last_active":"2024-06-04T04:57:10.215391+0000","last_peered":"2024-06-04T04:57:10.215391+0000","last_clean":"2024-06-04T04:57:10.215391+0000","last_became_active":"2024-06-04T04:53:36.378681+0000","last_became_peered":"2024-06-04T04:53:36.378681+0000","last_unstale":"2024-06-04T04:57:10.215391+0000","last_undegraded":"2024-06-04T04:57:10.215391+0000","last_fullsized":"2024-06-04T04:57:10.215391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:07:49.704731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946436+0000","last_change":"2024-06-04T04:53:33.345915+0000","last_active":"2024-06-04T04:57:09.946436+0000","last_peered":"2024-06-04T04:57:09.946436+0000","last_clean":"2024-06-04T04:57:09.946436+0000","last_became_active":"2024-06-04T04:53:33.345537+0000","last_became_peered":"2024-06-04T04:53:33.345537+0000","last_unstale":"2024-06-04T04:57:09.946436+0000","last_undegraded":"2024-06-04T04:57:09.946436+0000","last_fullsized":"2024-06-04T04:57:09.946436+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:06:44.086968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213503+0000","last_change":"2024-06-04T04:53:36.379094+0000","last_active":"2024-06-04T04:57:10.213503+0000","last_peered":"2024-06-04T04:57:10.213503+0000","last_clean":"2024-06-04T04:57:10.213503+0000","last_became_active":"2024-06-04T04:53:36.378763+0000","last_became_peered":"2024-06-04T04:53:36.378763+0000","last_unstale":"2024-06-04T04:57:10.213503+0000","last_undegraded":"2024-06-04T04:57:10.213503+0000","last_fullsized":"2024-06-04T04:57:10.213503+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:16.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'6","reported_seq":235,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.465965+0000","last_change":"2024-06-04T04:53:33.341342+0000","last_active":"2024-06-04T04:57:10.465965+0000","last_peered":"2024-06-04T04:57:10.465965+0000","last_clean":"2024-06-04T04:57:10.465965+0000","last_became_active":"2024-06-04T04:53:33.340862+0000","last_became_peered":"2024-06-04T04:53:33.340862+0000","last_unstale":"2024-06-04T04:57:10.465965+0000","last_undegraded":"2024-06-04T04:57:10.465965+0000","last_fullsized":"2024-06-04T04:57:10.465965+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:26.818063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":201,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213135+0000","last_change":"2024-06-04T04:53:36.383124+0000","last_active":"2024-06-04T04:57:10.213135+0000","last_peered":"2024-06-04T04:57:10.213135+0000","last_clean":"2024-06-04T04:57:10.213135+0000","last_became_active":"2024-06-04T04:53:36.382449+0000","last_became_peered":"2024-06-04T04:53:36.382449+0000","last_unstale":"2024-06-04T04:57:10.213135+0000","last_undegraded":"2024-06-04T04:57:10.213135+0000","last_fullsized":"2024-06-04T04:57:10.213135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:52:40.776003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":47,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965391+0000","last_change":"2024-06-04T04:53:33.354556+0000","last_active":"2024-06-04T04:57:09.965391+0000","last_peered":"2024-06-04T04:57:09.965391+0000","last_clean":"2024-06-04T04:57:09.965391+0000","last_became_active":"2024-06-04T04:53:33.354238+0000","last_became_peered":"2024-06-04T04:53:33.354238+0000","last_unstale":"2024-06-04T04:57:09.965391+0000","last_undegraded":"2024-06-04T04:57:09.965391+0000","last_fullsized":"2024-06-04T04:57:09.965391+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:54.697374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946345+0000","last_change":"2024-06-04T04:53:36.371387+0000","last_active":"2024-06-04T04:57:09.946345+0000","last_peered":"2024-06-04T04:57:09.946345+0000","last_clean":"2024-06-04T04:57:09.946345+0000","last_became_active":"2024-06-04T04:53:36.370852+0000","last_became_peered":"2024-06-04T04:53:36.370852+0000","last_unstale":"2024-06-04T04:57:09.946345+0000","last_undegraded":"2024-06-04T04:57:09.946345+0000","last_fullsized":"2024-06-04T04:57:09.946345+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:44:06.956210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'3","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965694+0000","last_change":"2024-06-04T04:53:33.349982+0000","last_active":"2024-06-04T04:57:09.965694+0000","last_peered":"2024-06-04T04:57:09.965694+0000","last_clean":"2024-06-04T04:57:09.965694+0000","last_became_active":"2024-06-04T04:53:33.349633+0000","last_became_peered":"2024-06-04T04:53:33.349633+0000","last_unstale":"2024-06-04T04:57:09.965694+0000","last_undegraded":"2024-06-04T04:57:09.965694+0000","last_fullsized":"2024-06-04T04:57:09.965694+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:40.107426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466355+0000","last_change":"2024-06-04T04:53:36.382313+0000","last_active":"2024-06-04T04:57:10.466355+0000","last_peered":"2024-06-04T04:57:10.466355+0000","last_clean":"2024-06-04T04:57:10.466355+0000","last_became_active":"2024-06-04T04:53:36.379638+0000","last_became_peered":"2024-06-04T04:53:36.379638+0000","last_unstale":"2024-06-04T04:57:10.466355+0000","last_undegraded":"2024-06-04T04:57:10.466355+0000","last_fullsized":"2024-06-04T04:57:10.466355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:43:42.527262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.048030+0000","last_change":"2024-06-04T04:54:19.513581+0000","last_active":"2024-06-04T04:57:10.048030+0000","last_peered":"2024-06-04T04:57:10.048030+0000","last_clean":"2024-06-04T04:57:10.048030+0000","last_became_active":"2024-06-04T04:54:19.513111+0000","last_became_peered":"2024-06-04T04:54:19.513111+0000","last_unstale":"2024-06-04T04:57:10.048030+0000","last_undegraded":"2024-06-04T04:57:10.048030+0000","last_fullsized":"2024-06-04T04:57:10.048030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:57.028083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946526+0000","last_change":"2024-06-04T04:53:36.369904+0000","last_active":"2024-06-04T04:57:09.946526+0000","last_peered":"2024-06-04T04:57:09.946526+0000","last_clean":"2024-06-04T04:57:09.946526+0000","last_became_active":"2024-06-04T04:53:36.367929+0000","last_became_peered":"2024-06-04T04:53:36.367929+0000","last_unstale":"2024-06-04T04:57:09.946526+0000","last_undegraded":"2024-06-04T04:57:09.946526+0000","last_fullsized":"2024-06-04T04:57:09.946526+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:37:08.721569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"82'10","reported_seq":241,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.900365+0000","last_change":"2024-06-04T04:53:33.342723+0000","last_active":"2024-06-04T04:57:10.900365+0000","last_peered":"2024-06-04T04:57:10.900365+0000","last_clean":"2024-06-04T04:57:10.900365+0000","last_became_active":"2024-06-04T04:53:33.342344+0000","last_became_peered":"2024-06-04T04:53:33.342344+0000","last_unstale":"2024-06-04T04:57:10.900365+0000","last_undegraded":"2024-06-04T04:57:10.900365+0000","last_fullsized":"2024-06-04T04:57:10.900365+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:59:50.147517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":202,"num_read_kb":5,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946315+0000","last_change":"2024-06-04T04:54:19.513534+0000","last_active":"2024-06-04T04:57:09.946315+0000","last_peered":"2024-06-04T04:57:09.946315+0000","last_clean":"2024-06-04T04:57:09.946315+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:57:09.946315+0000","last_undegraded":"2024-06-04T04:57:09.946315+0000","last_fullsized":"2024-06-04T04:57:09.946315+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:16:32.240932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946648+0000","last_change":"2024-06-04T04:53:36.371236+0000","last_active":"2024-06-04T04:57:09.946648+0000","last_peered":"2024-06-04T04:57:09.946648+0000","last_clean":"2024-06-04T04:57:09.946648+0000","last_became_active":"2024-06-04T04:53:36.370876+0000","last_became_peered":"2024-06-04T04:53:36.370876+0000","last_unstale":"2024-06-04T04:57:09.946648+0000","last_undegraded":"2024-06-04T04:57:09.946648+0000","last_fullsized":"2024-06-04T04:57:09.946648+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:15:56.751317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":232,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215195+0000","last_change":"2024-06-04T04:53:33.355642+0000","last_active":"2024-06-04T04:57:10.215195+0000","last_peered":"2024-06-04T04:57:10.215195+0000","last_clean":"2024-06-04T04:57:10.215195+0000","last_became_active":"2024-06-04T04:53:33.355187+0000","last_became_peered":"2024-06-04T04:53:33.355187+0000","last_unstale":"2024-06-04T04:57:10.215195+0000","last_undegraded":"2024-06-04T04:57:10.215195+0000","last_fullsized":"2024-06-04T04:57:10.215195+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:32:28.331518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":198,"num_read_kb":198,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.047902+0000","last_change":"2024-06-04T04:54:19.631750+0000","last_active":"2024-06-04T04:57:10.047902+0000","last_peered":"2024-06-04T04:57:10.047902+0000","last_clean":"2024-06-04T04:57:10.047902+0000","last_became_active":"2024-06-04T04:54:19.631485+0000","last_became_peered":"2024-06-04T04:54:19.631485+0000","last_unstale":"2024-06-04T04:57:10.047902+0000","last_undegraded":"2024-06-04T04:57:10.047902+0000","last_fullsized":"2024-06-04T04:57:10.047902+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:29:44.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466037+0000","last_change":"2024-06-04T04:53:33.340653+0000","last_active":"2024-06-04T04:57:10.466037+0000","last_peered":"2024-06-04T04:57:10.466037+0000","last_clean":"2024-06-04T04:57:10.466037+0000","last_became_active":"2024-06-04T04:53:33.340250+0000","last_became_peered":"2024-06-04T04:53:33.340250+0000","last_unstale":"2024-06-04T04:57:10.466037+0000","last_undegraded":"2024-06-04T04:57:10.466037+0000","last_fullsized":"2024-06-04T04:57:10.466037+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:46:20.329514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":103,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213299+0000","last_change":"2024-06-04T04:52:09.181155+0000","last_active":"2024-06-04T04:57:10.213299+0000","last_peered":"2024-06-04T04:57:10.213299+0000","last_clean":"2024-06-04T04:57:10.213299+0000","last_became_active":"2024-06-04T04:52:09.177002+0000","last_became_peered":"2024-06-04T04:52:09.177002+0000","last_unstale":"2024-06-04T04:57:10.213299+0000","last_undegraded":"2024-06-04T04:57:10.213299+0000","last_fullsized":"2024-06-04T04:57:10.213299+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_clean_scrub_stamp":"2024-06-04T04:48:44.631948+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:46:57.735302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215170+0000","last_change":"2024-06-04T04:53:36.372634+0000","last_active":"2024-06-04T04:57:10.215170+0000","last_peered":"2024-06-04T04:57:10.215170+0000","last_clean":"2024-06-04T04:57:10.215170+0000","last_became_active":"2024-06-04T04:53:36.372290+0000","last_became_peered":"2024-06-04T04:53:36.372290+0000","last_unstale":"2024-06-04T04:57:10.215170+0000","last_undegraded":"2024-06-04T04:57:10.215170+0000","last_fullsized":"2024-06-04T04:57:10.215170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:14:07.460382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213392+0000","last_change":"2024-06-04T04:53:33.348760+0000","last_active":"2024-06-04T04:57:10.213392+0000","last_peered":"2024-06-04T04:57:10.213392+0000","last_clean":"2024-06-04T04:57:10.213392+0000","last_became_active":"2024-06-04T04:53:33.348347+0000","last_became_peered":"2024-06-04T04:53:33.348347+0000","last_unstale":"2024-06-04T04:57:10.213392+0000","last_undegraded":"2024-06-04T04:57:10.213392+0000","last_fullsized":"2024-06-04T04:57:10.213392+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:36:20.641061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965676+0000","last_change":"2024-06-04T04:53:33.354840+0000","last_active":"2024-06-04T04:57:09.965676+0000","last_peered":"2024-06-04T04:57:09.965676+0000","last_clean":"2024-06-04T04:57:09.965676+0000","last_became_active":"2024-06-04T04:53:33.354287+0000","last_became_peered":"2024-06-04T04:53:33.354287+0000","last_unstale":"2024-06-04T04:57:09.965676+0000","last_undegraded":"2024-06-04T04:57:09.965676+0000","last_fullsized":"2024-06-04T04:57:09.965676+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:42.550098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215686+0000","last_change":"2024-06-04T04:53:36.372678+0000","last_active":"2024-06-04T04:57:10.215686+0000","last_peered":"2024-06-04T04:57:10.215686+0000","last_clean":"2024-06-04T04:57:10.215686+0000","last_became_active":"2024-06-04T04:53:36.372312+0000","last_became_peered":"2024-06-04T04:53:36.372312+0000","last_unstale":"2024-06-04T04:57:10.215686+0000","last_undegraded":"2024-06-04T04:57:10.215686+0000","last_fullsized":"2024-06-04T04:57:10.215686+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:34:41.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'51","reported_seq":103,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.173057+0000","last_change":"2024-06-04T04:53:33.348440+0000","last_active":"2024-06-04T04:57:10.173057+0000","last_peered":"2024-06-04T04:57:10.173057+0000","last_clean":"2024-06-04T04:57:10.173057+0000","last_became_active":"2024-06-04T04:53:33.348131+0000","last_became_peered":"2024-06-04T04:53:33.348131+0000","last_unstale":"2024-06-04T04:57:10.173057+0000","last_undegraded":"2024-06-04T04:57:10.173057+0000","last_fullsized":"2024-06-04T04:57:10.173057+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:14:11.552286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":91,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942741+0000","last_change":"2024-06-04T04:53:36.378669+0000","last_active":"2024-06-04T04:57:09.942741+0000","last_peered":"2024-06-04T04:57:09.942741+0000","last_clean":"2024-06-04T04:57:09.942741+0000","last_became_active":"2024-06-04T04:53:36.378233+0000","last_became_peered":"2024-06-04T04:53:36.378233+0000","last_unstale":"2024-06-04T04:57:09.942741+0000","last_undegraded":"2024-06-04T04:57:09.942741+0000","last_fullsized":"2024-06-04T04:57:09.942741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:36:45.870924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466139+0000","last_change":"2024-06-04T04:53:33.345078+0000","last_active":"2024-06-04T04:57:10.466139+0000","last_peered":"2024-06-04T04:57:10.466139+0000","last_clean":"2024-06-04T04:57:10.466139+0000","last_became_active":"2024-06-04T04:53:33.344708+0000","last_became_peered":"2024-06-04T04:53:33.344708+0000","last_unstale":"2024-06-04T04:57:10.466139+0000","last_undegraded":"2024-06-04T04:57:10.466139+0000","last_fullsized":"2024-06-04T04:57:10.466139+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:45:14.010758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466300+0000","last_change":"2024-06-04T04:53:36.364817+0000","last_active":"2024-06-04T04:57:10.466300+0000","last_peered":"2024-06-04T04:57:10.466300+0000","last_clean":"2024-06-04T04:57:10.466300+0000","last_became_active":"2024-06-04T04:53:36.364499+0000","last_became_peered":"2024-06-04T04:53:36.364499+0000","last_unstale":"2024-06-04T04:57:10.466300+0000","last_undegraded":"2024-06-04T04:57:10.466300+0000","last_fullsized":"2024-06-04T04:57:10.466300+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:39:30.518028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215210+0000","last_change":"2024-06-04T04:53:33.354527+0000","last_active":"2024-06-04T04:57:10.215210+0000","last_peered":"2024-06-04T04:57:10.215210+0000","last_clean":"2024-06-04T04:57:10.215210+0000","last_became_active":"2024-06-04T04:53:33.354175+0000","last_became_peered":"2024-06-04T04:53:33.354175+0000","last_unstale":"2024-06-04T04:57:10.215210+0000","last_undegraded":"2024-06-04T04:57:10.215210+0000","last_fullsized":"2024-06-04T04:57:10.215210+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:33:08.125428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965391+0000","last_change":"2024-06-04T04:53:36.373248+0000","last_active":"2024-06-04T04:57:09.965391+0000","last_peered":"2024-06-04T04:57:09.965391+0000","last_clean":"2024-06-04T04:57:09.965391+0000","last_became_active":"2024-06-04T04:53:36.372915+0000","last_became_peered":"2024-06-04T04:53:36.372915+0000","last_unstale":"2024-06-04T04:57:09.965391+0000","last_undegraded":"2024-06-04T04:57:09.965391+0000","last_fullsized":"2024-06-04T04:57:09.965391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:23:11.689349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942495+0000","last_change":"2024-06-04T04:53:33.340427+0000","last_active":"2024-06-04T04:57:09.942495+0000","last_peered":"2024-06-04T04:57:09.942495+0000","last_clean":"2024-06-04T04:57:09.942495+0000","last_became_active":"2024-06-04T04:53:33.340022+0000","last_became_peered":"2024-06-04T04:53:33.340022+0000","last_unstale":"2024-06-04T04:57:09.942495+0000","last_undegraded":"2024-06-04T04:57:09.942495+0000","last_fullsized":"2024-06-04T04:57:09.942495+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:56:16.496471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466096+0000","last_change":"2024-06-04T04:53:36.380114+0000","last_active":"2024-06-04T04:57:10.466096+0000","last_peered":"2024-06-04T04:57:10.466096+0000","last_clean":"2024-06-04T04:57:10.466096+0000","last_became_active":"2024-06-04T04:53:36.379392+0000","last_became_peered":"2024-06-04T04:53:36.379392+0000","last_unstale":"2024-06-04T04:57:10.466096+0000","last_undegraded":"2024-06-04T04:57:10.466096+0000","last_fullsized":"2024-06-04T04:57:10.466096+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:41:48.776439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"81'5","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466122+0000","last_change":"2024-06-04T04:54:19.519755+0000","last_active":"2024-06-04T04:57:10.466122+0000","last_peered":"2024-06-04T04:57:10.466122+0000","last_clean":"2024-06-04T04:57:10.466122+0000","last_became_active":"2024-06-04T04:54:19.513992+0000","last_became_peered":"2024-06-04T04:54:19.513992+0000","last_unstale":"2024-06-04T04:57:10.466122+0000","last_undegraded":"2024-06-04T04:57:10.466122+0000","last_fullsized":"2024-06-04T04:57:10.466122+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:01.951935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942779+0000","last_change":"2024-06-04T04:53:36.377420+0000","last_active":"2024-06-04T04:57:09.942779+0000","last_peered":"2024-06-04T04:57:09.942779+0000","last_clean":"2024-06-04T04:57:09.942779+0000","last_became_active":"2024-06-04T04:53:36.377109+0000","last_became_peered":"2024-06-04T04:53:36.377109+0000","last_unstale":"2024-06-04T04:57:09.942779+0000","last_undegraded":"2024-06-04T04:57:09.942779+0000","last_fullsized":"2024-06-04T04:57:09.942779+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:47:12.102396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965575+0000","last_change":"2024-06-04T04:54:19.645328+0000","last_active":"2024-06-04T04:57:09.965575+0000","last_peered":"2024-06-04T04:57:09.965575+0000","last_clean":"2024-06-04T04:57:09.965575+0000","last_became_active":"2024-06-04T04:54:19.633416+0000","last_became_peered":"2024-06-04T04:54:19.633416+0000","last_unstale":"2024-06-04T04:57:09.965575+0000","last_undegraded":"2024-06-04T04:57:09.965575+0000","last_fullsized":"2024-06-04T04:57:09.965575+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:04:20.494641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213349+0000","last_change":"2024-06-04T04:53:36.381187+0000","last_active":"2024-06-04T04:57:10.213349+0000","last_peered":"2024-06-04T04:57:10.213349+0000","last_clean":"2024-06-04T04:57:10.213349+0000","last_became_active":"2024-06-04T04:53:36.380591+0000","last_became_peered":"2024-06-04T04:53:36.380591+0000","last_unstale":"2024-06-04T04:57:10.213349+0000","last_undegraded":"2024-06-04T04:57:10.213349+0000","last_fullsized":"2024-06-04T04:57:10.213349+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:26:43.122339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'6","reported_seq":234,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219295+0000","last_change":"2024-06-04T04:53:33.355908+0000","last_active":"2024-06-04T04:57:10.219295+0000","last_peered":"2024-06-04T04:57:10.219295+0000","last_clean":"2024-06-04T04:57:10.219295+0000","last_became_active":"2024-06-04T04:53:33.355522+0000","last_became_peered":"2024-06-04T04:53:33.355522+0000","last_unstale":"2024-06-04T04:57:10.219295+0000","last_undegraded":"2024-06-04T04:57:10.219295+0000","last_fullsized":"2024-06-04T04:57:10.219295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:16:44.639431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":200,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213745+0000","last_change":"2024-06-04T04:53:36.382737+0000","last_active":"2024-06-04T04:57:10.213745+0000","last_peered":"2024-06-04T04:57:10.213745+0000","last_clean":"2024-06-04T04:57:10.213745+0000","last_became_active":"2024-06-04T04:53:36.382353+0000","last_became_peered":"2024-06-04T04:53:36.382353+0000","last_unstale":"2024-06-04T04:57:10.213745+0000","last_undegraded":"2024-06-04T04:57:10.213745+0000","last_fullsized":"2024-06-04T04:57:10.213745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:18:01.476058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'41","reported_seq":78,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.170281+0000","last_change":"2024-06-04T04:53:33.347273+0000","last_active":"2024-06-04T04:57:10.170281+0000","last_peered":"2024-06-04T04:57:10.170281+0000","last_clean":"2024-06-04T04:57:10.170281+0000","last_became_active":"2024-06-04T04:53:33.346915+0000","last_became_peered":"2024-06-04T04:53:33.346915+0000","last_unstale":"2024-06-04T04:57:10.170281+0000","last_undegraded":"2024-06-04T04:57:10.170281+0000","last_fullsized":"2024-06-04T04:57:10.170281+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:17.235484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213559+0000","last_change":"2024-06-04T04:53:36.383454+0000","last_active":"2024-06-04T04:57:10.213559+0000","last_peered":"2024-06-04T04:57:10.213559+0000","last_clean":"2024-06-04T04:57:10.213559+0000","last_became_active":"2024-06-04T04:53:36.383105+0000","last_became_peered":"2024-06-04T04:53:36.383105+0000","last_unstale":"2024-06-04T04:57:10.213559+0000","last_undegraded":"2024-06-04T04:57:10.213559+0000","last_fullsized":"2024-06-04T04:57:10.213559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:04:02.147688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215701+0000","last_change":"2024-06-04T04:53:33.345435+0000","last_active":"2024-06-04T04:57:10.215701+0000","last_peered":"2024-06-04T04:57:10.215701+0000","last_clean":"2024-06-04T04:57:10.215701+0000","last_became_active":"2024-06-04T04:53:33.345077+0000","last_became_peered":"2024-06-04T04:53:33.345077+0000","last_unstale":"2024-06-04T04:57:10.215701+0000","last_undegraded":"2024-06-04T04:57:10.215701+0000","last_fullsized":"2024-06-04T04:57:10.215701+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:20:13.992259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213743+0000","last_change":"2024-06-04T04:53:36.385238+0000","last_active":"2024-06-04T04:57:10.213743+0000","last_peered":"2024-06-04T04:57:10.213743+0000","last_clean":"2024-06-04T04:57:10.213743+0000","last_became_active":"2024-06-04T04:53:36.384899+0000","last_became_peered":"2024-06-04T04:53:36.384899+0000","last_unstale":"2024-06-04T04:57:10.213743+0000","last_undegraded":"2024-06-04T04:57:10.213743+0000","last_fullsized":"2024-06-04T04:57:10.213743+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:09:42.629340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466183+0000","last_change":"2024-06-04T04:53:36.383450+0000","last_active":"2024-06-04T04:57:10.466183+0000","last_peered":"2024-06-04T04:57:10.466183+0000","last_clean":"2024-06-04T04:57:10.466183+0000","last_became_active":"2024-06-04T04:53:36.382678+0000","last_became_peered":"2024-06-04T04:53:36.382678+0000","last_unstale":"2024-06-04T04:57:10.466183+0000","last_undegraded":"2024-06-04T04:57:10.466183+0000","last_fullsized":"2024-06-04T04:57:10.466183+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:36:28.932636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219063+0000","last_change":"2024-06-04T04:54:19.632563+0000","last_active":"2024-06-04T04:57:10.219063+0000","last_peered":"2024-06-04T04:57:10.219063+0000","last_clean":"2024-06-04T04:57:10.219063+0000","last_became_active":"2024-06-04T04:54:19.632070+0000","last_became_peered":"2024-06-04T04:54:19.632070+0000","last_unstale":"2024-06-04T04:57:10.219063+0000","last_undegraded":"2024-06-04T04:57:10.219063+0000","last_fullsized":"2024-06-04T04:57:10.219063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:07:16.552958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219185+0000","last_change":"2024-06-04T04:53:36.381408+0000","last_active":"2024-06-04T04:57:10.219185+0000","last_peered":"2024-06-04T04:57:10.219185+0000","last_clean":"2024-06-04T04:57:10.219185+0000","last_became_active":"2024-06-04T04:53:36.381078+0000","last_became_peered":"2024-06-04T04:53:36.381078+0000","last_unstale":"2024-06-04T04:57:10.219185+0000","last_undegraded":"2024-06-04T04:57:10.219185+0000","last_fullsized":"2024-06-04T04:57:10.219185+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:28:48.988591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466464+0000","last_change":"2024-06-04T04:53:36.360168+0000","last_active":"2024-06-04T04:57:10.466464+0000","last_peered":"2024-06-04T04:57:10.466464+0000","last_clean":"2024-06-04T04:57:10.466464+0000","last_became_active":"2024-06-04T04:53:36.359691+0000","last_became_peered":"2024-06-04T04:53:36.359691+0000","last_unstale":"2024-06-04T04:57:10.466464+0000","last_undegraded":"2024-06-04T04:57:10.466464+0000","last_fullsized":"2024-06-04T04:57:10.466464+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:55:17.418339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215688+0000","last_change":"2024-06-04T04:53:36.363018+0000","last_active":"2024-06-04T04:57:10.215688+0000","last_peered":"2024-06-04T04:57:10.215688+0000","last_clean":"2024-06-04T04:57:10.215688+0000","last_became_active":"2024-06-04T04:53:36.361595+0000","last_became_peered":"2024-06-04T04:53:36.361595+0000","last_unstale":"2024-06-04T04:57:10.215688+0000","last_undegraded":"2024-06-04T04:57:10.215688+0000","last_fullsized":"2024-06-04T04:57:10.215688+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:00:21.377322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466289+0000","last_change":"2024-06-04T04:53:36.375377+0000","last_active":"2024-06-04T04:57:10.466289+0000","last_peered":"2024-06-04T04:57:10.466289+0000","last_clean":"2024-06-04T04:57:10.466289+0000","last_became_active":"2024-06-04T04:53:36.373459+0000","last_became_peered":"2024-06-04T04:53:36.373459+0000","last_unstale":"2024-06-04T04:57:10.466289+0000","last_undegraded":"2024-06-04T04:57:10.466289+0000","last_fullsized":"2024-06-04T04:57:10.466289+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:51:31.686969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219589+0000","last_change":"2024-06-04T04:53:36.373850+0000","last_active":"2024-06-04T04:57:10.219589+0000","last_peered":"2024-06-04T04:57:10.219589+0000","last_clean":"2024-06-04T04:57:10.219589+0000","last_became_active":"2024-06-04T04:53:36.371951+0000","last_became_peered":"2024-06-04T04:53:36.371951+0000","last_unstale":"2024-06-04T04:57:10.219589+0000","last_undegraded":"2024-06-04T04:57:10.219589+0000","last_fullsized":"2024-06-04T04:57:10.219589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:06.452703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942719+0000","last_change":"2024-06-04T04:53:36.374871+0000","last_active":"2024-06-04T04:57:09.942719+0000","last_peered":"2024-06-04T04:57:09.942719+0000","last_clean":"2024-06-04T04:57:09.942719+0000","last_became_active":"2024-06-04T04:53:36.374486+0000","last_became_peered":"2024-06-04T04:53:36.374486+0000","last_unstale":"2024-06-04T04:57:09.942719+0000","last_undegraded":"2024-06-04T04:57:09.942719+0000","last_fullsized":"2024-06-04T04:57:09.942719+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:15:13.337374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213230+0000","last_change":"2024-06-04T04:54:19.513540+0000","last_active":"2024-06-04T04:57:10.213230+0000","last_peered":"2024-06-04T04:57:10.213230+0000","last_clean":"2024-06-04T04:57:10.213230+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:57:10.213230+0000","last_undegraded":"2024-06-04T04:57:10.213230+0000","last_fullsized":"2024-06-04T04:57:10.213230+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:09:27.993585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942434+0000","last_change":"2024-06-04T04:53:36.373055+0000","last_active":"2024-06-04T04:57:09.942434+0000","last_peered":"2024-06-04T04:57:09.942434+0000","last_clean":"2024-06-04T04:57:09.942434+0000","last_became_active":"2024-06-04T04:53:36.372754+0000","last_became_peered":"2024-06-04T04:53:36.372754+0000","last_unstale":"2024-06-04T04:57:09.942434+0000","last_undegraded":"2024-06-04T04:57:09.942434+0000","last_fullsized":"2024-06-04T04:57:09.942434+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:22:27.554200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215415+0000","last_change":"2024-06-04T04:53:36.378897+0000","last_active":"2024-06-04T04:57:10.215415+0000","last_peered":"2024-06-04T04:57:10.215415+0000","last_clean":"2024-06-04T04:57:10.215415+0000","last_became_active":"2024-06-04T04:53:36.378607+0000","last_became_peered":"2024-06-04T04:53:36.378607+0000","last_unstale":"2024-06-04T04:57:10.215415+0000","last_undegraded":"2024-06-04T04:57:10.215415+0000","last_fullsized":"2024-06-04T04:57:10.215415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:03.395491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215502+0000","last_change":"2024-06-04T04:53:36.378240+0000","last_active":"2024-06-04T04:57:10.215502+0000","last_peered":"2024-06-04T04:57:10.215502+0000","last_clean":"2024-06-04T04:57:10.215502+0000","last_became_active":"2024-06-04T04:53:36.377909+0000","last_became_peered":"2024-06-04T04:53:36.377909+0000","last_unstale":"2024-06-04T04:57:10.215502+0000","last_undegraded":"2024-06-04T04:57:10.215502+0000","last_fullsized":"2024-06-04T04:57:10.215502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:15:25.288340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466255+0000","last_change":"2024-06-04T04:53:36.379792+0000","last_active":"2024-06-04T04:57:10.466255+0000","last_peered":"2024-06-04T04:57:10.466255+0000","last_clean":"2024-06-04T04:57:10.466255+0000","last_became_active":"2024-06-04T04:53:36.379438+0000","last_became_peered":"2024-06-04T04:53:36.379438+0000","last_unstale":"2024-06-04T04:57:10.466255+0000","last_undegraded":"2024-06-04T04:57:10.466255+0000","last_fullsized":"2024-06-04T04:57:10.466255+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:23:57.617877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108276,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":606174,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2105,"num_read_kb":1171,"num_write":385,"num_write_kb":1476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":724992,"data_stored":1818522,"data_compressed":81405,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":308,"ondisk_log_size":308,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678074,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":880640,"data_stored":430053,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874302,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1052672,"data_stored":1097012,"data_compressed":46576,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037828,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":429490,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234058,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":831397,"data_compressed":27797,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430285,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":1024006,"data_compressed":41851,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593812,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":901120,"data_stored":439999,"data_compressed":8651,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":498762,"data_compressed":7592,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738482,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":909312,"data_stored":431039,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411431,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":511961,"data_compressed":10484,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":962833,"data_compressed":33953,"data_compressed_allocated":45056,"data_compressed_original":540672,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953566,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32044,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713108,"statfs":{"total":95995035648,"available":95962222592,"internally_reserved":0,"allocated":888832,"data_stored":429561,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790041,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":999424,"data_stored":1162751,"data_compressed":50899,"data_compressed_allocated":65536,"data_compressed_original":737280,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":75881,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":74608,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":134913,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":394044,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":205479,"data_compressed":7989,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-04T04:57:14.498 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-06-04T04:57:14.498 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-04T04:57:14.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:14 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:14 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:14 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:14 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:14 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:14 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:14 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi098 to 176.1M 2024-06-04T04:57:14.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:14 smithi098 ceph-mon[35569]: Unable to set osd_memory_target on smithi098 to 184732979: error parsing value: Value '184732979' is below minimum 939524096 2024-06-04T04:57:14.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:14 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:14 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:14 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:14 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:14 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:14 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:14 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:14 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.639+0000 7fb6a6786640 1 -- 172.21.15.70:0/2543181610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6a0101990 msgr2=0x7fb6a0101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:14.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.639+0000 7fb6a6786640 1 --2- 172.21.15.70:0/2543181610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6a0101990 0x7fb6a0101db0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb690009900 tx=0x7fb69002a410 comp rx=0 tx=0).stop 2024-06-04T04:57:14.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.639+0000 7fb6a6786640 1 -- 172.21.15.70:0/2543181610 shutdown_connections 2024-06-04T04:57:14.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.639+0000 7fb6a6786640 1 --2- 172.21.15.70:0/2543181610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6a0103540 0x7fb6a0109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.639+0000 7fb6a6786640 1 --2- 172.21.15.70:0/2543181610 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6a0102b80 0x7fb6a0103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.639+0000 7fb6a6786640 1 --2- 172.21.15.70:0/2543181610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6a0101990 0x7fb6a0101db0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.639+0000 7fb6a6786640 1 -- 172.21.15.70:0/2543181610 >> 172.21.15.70:0/2543181610 conn(0x7fb6a00fcf00 msgr2=0x7fb6a00ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:14.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.639+0000 7fb6a6786640 1 -- 172.21.15.70:0/2543181610 shutdown_connections 2024-06-04T04:57:14.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.639+0000 7fb6a6786640 1 -- 172.21.15.70:0/2543181610 wait complete. 2024-06-04T04:57:14.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.639+0000 7fb6a6786640 1 Processor -- start 2024-06-04T04:57:14.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.639+0000 7fb6a6786640 1 -- start start 2024-06-04T04:57:14.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb6a6786640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6a0101990 0x7fb6a006a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:14.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb6a6786640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6a0102b80 0x7fb6a019d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:14.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb6a6786640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6a0103540 0x7fb6a019ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:14.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb6a6786640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a010d360 con 0x7fb6a0103540 2024-06-04T04:57:14.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb6a6786640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a010d1e0 con 0x7fb6a0101990 2024-06-04T04:57:14.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb6a6786640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a010d4e0 con 0x7fb6a0102b80 2024-06-04T04:57:14.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb6a4d23640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6a0103540 0x7fb6a019ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:14.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb6a4d23640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6a0103540 0x7fb6a019ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33636/0 (socket says 172.21.15.70:33636) 2024-06-04T04:57:14.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb6a4d23640 1 -- 172.21.15.70:0/576556661 learned_addr learned my addr 172.21.15.70:0/576556661 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:14.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb69ffff640 1 --2- 172.21.15.70:0/576556661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6a0101990 0x7fb6a006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:14.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb69f7fe640 1 --2- 172.21.15.70:0/576556661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6a0102b80 0x7fb6a019d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:14.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb6a4d23640 1 -- 172.21.15.70:0/576556661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6a0102b80 msgr2=0x7fb6a019d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:14.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb6a4d23640 1 --2- 172.21.15.70:0/576556661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6a0102b80 0x7fb6a019d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb6a4d23640 1 -- 172.21.15.70:0/576556661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6a0101990 msgr2=0x7fb6a006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:14.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb6a4d23640 1 --2- 172.21.15.70:0/576556661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6a0101990 0x7fb6a006a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb6a4d23640 1 -- 172.21.15.70:0/576556661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a01a08d0 con 0x7fb6a0103540 2024-06-04T04:57:14.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb69ffff640 1 --2- 172.21.15.70:0/576556661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6a0101990 0x7fb6a006a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:14.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb6a4d23640 1 --2- 172.21.15.70:0/576556661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6a0103540 0x7fb6a019ffc0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb69800eb10 tx=0x7fb69800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:14.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.640+0000 7fb69d7fa640 1 -- 172.21.15.70:0/576556661 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb698004b20 con 0x7fb6a0103540 2024-06-04T04:57:14.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.641+0000 7fb6a6786640 1 -- 172.21.15.70:0/576556661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a01a0b60 con 0x7fb6a0103540 2024-06-04T04:57:14.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.641+0000 7fb69d7fa640 1 -- 172.21.15.70:0/576556661 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb698004cc0 con 0x7fb6a0103540 2024-06-04T04:57:14.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.641+0000 7fb69d7fa640 1 -- 172.21.15.70:0/576556661 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb698010690 con 0x7fb6a0103540 2024-06-04T04:57:14.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.641+0000 7fb6a6786640 1 -- 172.21.15.70:0/576556661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a01a10b0 con 0x7fb6a0103540 2024-06-04T04:57:14.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.642+0000 7fb6a6786640 1 -- 172.21.15.70:0/576556661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6a0108df0 con 0x7fb6a0103540 2024-06-04T04:57:14.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.643+0000 7fb69d7fa640 1 -- 172.21.15.70:0/576556661 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 22) ==== 149772+0+0 (secure 0 0 0) 0x7fb698005000 con 0x7fb6a0103540 2024-06-04T04:57:14.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.644+0000 7fb69d7fa640 1 --2- 172.21.15.70:0/576556661 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb6880b2960 0x7fb6880b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:14.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.644+0000 7fb69ffff640 1 --2- 172.21.15.70:0/576556661 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb6880b2960 0x7fb6880b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:14.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.644+0000 7fb69d7fa640 1 -- 172.21.15.70:0/576556661 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb6980e2ec0 con 0x7fb6a0103540 2024-06-04T04:57:14.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.644+0000 7fb69ffff640 1 --2- 172.21.15.70:0/576556661 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb6880b2960 0x7fb6880b4e20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb690006010 tx=0x7fb690031400 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:14.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.647+0000 7fb69d7fa640 1 -- 172.21.15.70:0/576556661 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb698054df0 con 0x7fb6a0103540 2024-06-04T04:57:14.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:14 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:14 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:14 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:14 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:14 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:14 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:14 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi098 to 176.1M 2024-06-04T04:57:14.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:14 smithi112 ceph-mon[36453]: Unable to set osd_memory_target on smithi098 to 184732979: error parsing value: Value '184732979' is below minimum 939524096 2024-06-04T04:57:14.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:14 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:14 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:14 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:14 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:14 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:14 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:14.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:14 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:14 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:14.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.707+0000 7fb69d7fa640 1 -- 172.21.15.70:0/576556661 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7fb6980e0890 con 0x7fb6a0103540 2024-06-04T04:57:14.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.848+0000 7fb6a6786640 1 -- 172.21.15.70:0/576556661 --> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb6a0062e90 con 0x7fb6880b2960 2024-06-04T04:57:14.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.856+0000 7fb69d7fa640 1 -- 172.21.15.70:0/576556661 <== mgr.24806 v2:172.21.15.70:6800/542036627 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258068 (secure 0 0 0) 0x7fb6a0062e90 con 0x7fb6880b2960 2024-06-04T04:57:14.861 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:57:14.863 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-06-04T04:57:14.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.860+0000 7fb6a6786640 1 -- 172.21.15.70:0/576556661 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb6880b2960 msgr2=0x7fb6880b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:14.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.860+0000 7fb6a6786640 1 --2- 172.21.15.70:0/576556661 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb6880b2960 0x7fb6880b4e20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb690006010 tx=0x7fb690031400 comp rx=0 tx=0).stop 2024-06-04T04:57:14.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.860+0000 7fb6a6786640 1 -- 172.21.15.70:0/576556661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6a0103540 msgr2=0x7fb6a019ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:14.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.860+0000 7fb6a6786640 1 --2- 172.21.15.70:0/576556661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6a0103540 0x7fb6a019ffc0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb69800eb10 tx=0x7fb69800c2d0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.860+0000 7fb6a6786640 1 -- 172.21.15.70:0/576556661 shutdown_connections 2024-06-04T04:57:14.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.860+0000 7fb6a6786640 1 --2- 172.21.15.70:0/576556661 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb6880b2960 0x7fb6880b4e20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.860+0000 7fb6a6786640 1 --2- 172.21.15.70:0/576556661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6a0103540 0x7fb6a019ffc0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.860+0000 7fb6a6786640 1 --2- 172.21.15.70:0/576556661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6a0102b80 0x7fb6a019d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.860+0000 7fb6a6786640 1 --2- 172.21.15.70:0/576556661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6a0101990 0x7fb6a006a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:14.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.860+0000 7fb6a6786640 1 -- 172.21.15.70:0/576556661 >> 172.21.15.70:0/576556661 conn(0x7fb6a00fcf00 msgr2=0x7fb6a0105fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:14.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.860+0000 7fb6a6786640 1 -- 172.21.15.70:0/576556661 shutdown_connections 2024-06-04T04:57:14.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:14.860+0000 7fb6a6786640 1 -- 172.21.15.70:0/576556661 wait complete. 2024-06-04T04:57:14.873 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-06-04T04:57:13.959890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1196776,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2186,"num_read_kb":1240,"num_write":510,"num_write_kb":3628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2952512,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":902,"ondisk_log_size":902,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356092,"kb_used_data":11072,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585732,"statfs":{"total":1151940427776,"available":1151575789568,"internally_reserved":0,"allocated":11337728,"data_stored":8248864,"data_compressed":250839,"data_compressed_allocated":335872,"data_compressed_original":3072000,"omap_allocated":19138,"internal_metadata":353154366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.993238"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466420+0000","last_change":"2024-06-04T04:53:36.383755+0000","last_active":"2024-06-04T04:57:10.466420+0000","last_peered":"2024-06-04T04:57:10.466420+0000","last_clean":"2024-06-04T04:57:10.466420+0000","last_became_active":"2024-06-04T04:53:36.378274+0000","last_became_peered":"2024-06-04T04:53:36.378274+0000","last_unstale":"2024-06-04T04:57:10.466420+0000","last_undegraded":"2024-06-04T04:57:10.466420+0000","last_fullsized":"2024-06-04T04:57:10.466420+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:30:19.685994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466173+0000","last_change":"2024-06-04T04:53:36.372436+0000","last_active":"2024-06-04T04:57:10.466173+0000","last_peered":"2024-06-04T04:57:10.466173+0000","last_clean":"2024-06-04T04:57:10.466173+0000","last_became_active":"2024-06-04T04:53:36.372100+0000","last_became_peered":"2024-06-04T04:53:36.372100+0000","last_unstale":"2024-06-04T04:57:10.466173+0000","last_undegraded":"2024-06-04T04:57:10.466173+0000","last_fullsized":"2024-06-04T04:57:10.466173+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:42:31.646754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466063+0000","last_change":"2024-06-04T04:53:36.366915+0000","last_active":"2024-06-04T04:57:10.466063+0000","last_peered":"2024-06-04T04:57:10.466063+0000","last_clean":"2024-06-04T04:57:10.466063+0000","last_became_active":"2024-06-04T04:53:36.366611+0000","last_became_peered":"2024-06-04T04:53:36.366611+0000","last_unstale":"2024-06-04T04:57:10.466063+0000","last_undegraded":"2024-06-04T04:57:10.466063+0000","last_fullsized":"2024-06-04T04:57:10.466063+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:17:48.775938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466271+0000","last_change":"2024-06-04T04:53:36.374486+0000","last_active":"2024-06-04T04:57:10.466271+0000","last_peered":"2024-06-04T04:57:10.466271+0000","last_clean":"2024-06-04T04:57:10.466271+0000","last_became_active":"2024-06-04T04:53:36.374178+0000","last_became_peered":"2024-06-04T04:53:36.374178+0000","last_unstale":"2024-06-04T04:57:10.466271+0000","last_undegraded":"2024-06-04T04:57:10.466271+0000","last_fullsized":"2024-06-04T04:57:10.466271+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:30:28.287070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965508+0000","last_change":"2024-06-04T04:53:36.376183+0000","last_active":"2024-06-04T04:57:09.965508+0000","last_peered":"2024-06-04T04:57:09.965508+0000","last_clean":"2024-06-04T04:57:09.965508+0000","last_became_active":"2024-06-04T04:53:36.375790+0000","last_became_peered":"2024-06-04T04:53:36.375790+0000","last_unstale":"2024-06-04T04:57:09.965508+0000","last_undegraded":"2024-06-04T04:57:09.965508+0000","last_fullsized":"2024-06-04T04:57:09.965508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:58.697440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946349+0000","last_change":"2024-06-04T04:53:36.370249+0000","last_active":"2024-06-04T04:57:09.946349+0000","last_peered":"2024-06-04T04:57:09.946349+0000","last_clean":"2024-06-04T04:57:09.946349+0000","last_became_active":"2024-06-04T04:53:36.369916+0000","last_became_peered":"2024-06-04T04:53:36.369916+0000","last_unstale":"2024-06-04T04:57:09.946349+0000","last_undegraded":"2024-06-04T04:57:09.946349+0000","last_fullsized":"2024-06-04T04:57:09.946349+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:02:41.739806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213216+0000","last_change":"2024-06-04T04:53:36.384329+0000","last_active":"2024-06-04T04:57:10.213216+0000","last_peered":"2024-06-04T04:57:10.213216+0000","last_clean":"2024-06-04T04:57:10.213216+0000","last_became_active":"2024-06-04T04:53:36.382504+0000","last_became_peered":"2024-06-04T04:53:36.382504+0000","last_unstale":"2024-06-04T04:57:10.213216+0000","last_undegraded":"2024-06-04T04:57:10.213216+0000","last_fullsized":"2024-06-04T04:57:10.213216+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:51:07.604691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942376+0000","last_change":"2024-06-04T04:53:36.372319+0000","last_active":"2024-06-04T04:57:09.942376+0000","last_peered":"2024-06-04T04:57:09.942376+0000","last_clean":"2024-06-04T04:57:09.942376+0000","last_became_active":"2024-06-04T04:53:36.372000+0000","last_became_peered":"2024-06-04T04:53:36.372000+0000","last_unstale":"2024-06-04T04:57:09.942376+0000","last_undegraded":"2024-06-04T04:57:09.942376+0000","last_fullsized":"2024-06-04T04:57:09.942376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:15:56.906996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215880+0000","last_change":"2024-06-04T04:53:36.376051+0000","last_active":"2024-06-04T04:57:10.215880+0000","last_peered":"2024-06-04T04:57:10.215880+0000","last_clean":"2024-06-04T04:57:10.215880+0000","last_became_active":"2024-06-04T04:53:36.375726+0000","last_became_peered":"2024-06-04T04:53:36.375726+0000","last_unstale":"2024-06-04T04:57:10.215880+0000","last_undegraded":"2024-06-04T04:57:10.215880+0000","last_fullsized":"2024-06-04T04:57:10.215880+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:09:08.542764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946736+0000","last_change":"2024-06-04T04:53:36.373266+0000","last_active":"2024-06-04T04:57:09.946736+0000","last_peered":"2024-06-04T04:57:09.946736+0000","last_clean":"2024-06-04T04:57:09.946736+0000","last_became_active":"2024-06-04T04:53:36.372898+0000","last_became_peered":"2024-06-04T04:53:36.372898+0000","last_unstale":"2024-06-04T04:57:09.946736+0000","last_undegraded":"2024-06-04T04:57:09.946736+0000","last_fullsized":"2024-06-04T04:57:09.946736+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:41:43.436781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213610+0000","last_change":"2024-06-04T04:53:36.381344+0000","last_active":"2024-06-04T04:57:10.213610+0000","last_peered":"2024-06-04T04:57:10.213610+0000","last_clean":"2024-06-04T04:57:10.213610+0000","last_became_active":"2024-06-04T04:53:36.380736+0000","last_became_peered":"2024-06-04T04:53:36.380736+0000","last_unstale":"2024-06-04T04:57:10.213610+0000","last_undegraded":"2024-06-04T04:57:10.213610+0000","last_fullsized":"2024-06-04T04:57:10.213610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:23:01.607272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466124+0000","last_change":"2024-06-04T04:53:36.374035+0000","last_active":"2024-06-04T04:57:10.466124+0000","last_peered":"2024-06-04T04:57:10.466124+0000","last_clean":"2024-06-04T04:57:10.466124+0000","last_became_active":"2024-06-04T04:53:36.373667+0000","last_became_peered":"2024-06-04T04:53:36.373667+0000","last_unstale":"2024-06-04T04:57:10.466124+0000","last_undegraded":"2024-06-04T04:57:10.466124+0000","last_fullsized":"2024-06-04T04:57:10.466124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:23:07.880369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215527+0000","last_change":"2024-06-04T04:53:36.369613+0000","last_active":"2024-06-04T04:57:10.215527+0000","last_peered":"2024-06-04T04:57:10.215527+0000","last_clean":"2024-06-04T04:57:10.215527+0000","last_became_active":"2024-06-04T04:53:36.369277+0000","last_became_peered":"2024-06-04T04:53:36.369277+0000","last_unstale":"2024-06-04T04:57:10.215527+0000","last_undegraded":"2024-06-04T04:57:10.215527+0000","last_fullsized":"2024-06-04T04:57:10.215527+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:13:33.756609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.465960+0000","last_change":"2024-06-04T04:53:36.368094+0000","last_active":"2024-06-04T04:57:10.465960+0000","last_peered":"2024-06-04T04:57:10.465960+0000","last_clean":"2024-06-04T04:57:10.465960+0000","last_became_active":"2024-06-04T04:53:36.367717+0000","last_became_peered":"2024-06-04T04:53:36.367717+0000","last_unstale":"2024-06-04T04:57:10.465960+0000","last_undegraded":"2024-06-04T04:57:10.465960+0000","last_fullsized":"2024-06-04T04:57:10.465960+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:39:56.305078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.465864+0000","last_change":"2024-06-04T04:53:36.366765+0000","last_active":"2024-06-04T04:57:10.465864+0000","last_peered":"2024-06-04T04:57:10.465864+0000","last_clean":"2024-06-04T04:57:10.465864+0000","last_became_active":"2024-06-04T04:53:36.366438+0000","last_became_peered":"2024-06-04T04:53:36.366438+0000","last_unstale":"2024-06-04T04:57:10.465864+0000","last_undegraded":"2024-06-04T04:57:10.465864+0000","last_fullsized":"2024-06-04T04:57:10.465864+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:38:37.313926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215176+0000","last_change":"2024-06-04T04:53:36.380088+0000","last_active":"2024-06-04T04:57:10.215176+0000","last_peered":"2024-06-04T04:57:10.215176+0000","last_clean":"2024-06-04T04:57:10.215176+0000","last_became_active":"2024-06-04T04:53:36.379467+0000","last_became_peered":"2024-06-04T04:53:36.379467+0000","last_unstale":"2024-06-04T04:57:10.215176+0000","last_undegraded":"2024-06-04T04:57:10.215176+0000","last_fullsized":"2024-06-04T04:57:10.215176+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:46:53.236419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466550+0000","last_change":"2024-06-04T04:53:36.381126+0000","last_active":"2024-06-04T04:57:10.466550+0000","last_peered":"2024-06-04T04:57:10.466550+0000","last_clean":"2024-06-04T04:57:10.466550+0000","last_became_active":"2024-06-04T04:53:36.378833+0000","last_became_peered":"2024-06-04T04:53:36.378833+0000","last_unstale":"2024-06-04T04:57:10.466550+0000","last_undegraded":"2024-06-04T04:57:10.466550+0000","last_fullsized":"2024-06-04T04:57:10.466550+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:20:28.639273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.048240+0000","last_change":"2024-06-04T04:53:36.369692+0000","last_active":"2024-06-04T04:57:10.048240+0000","last_peered":"2024-06-04T04:57:10.048240+0000","last_clean":"2024-06-04T04:57:10.048240+0000","last_became_active":"2024-06-04T04:53:36.369370+0000","last_became_peered":"2024-06-04T04:53:36.369370+0000","last_unstale":"2024-06-04T04:57:10.048240+0000","last_undegraded":"2024-06-04T04:57:10.048240+0000","last_fullsized":"2024-06-04T04:57:10.048240+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:55.275465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219675+0000","last_change":"2024-06-04T04:53:36.383639+0000","last_active":"2024-06-04T04:57:10.219675+0000","last_peered":"2024-06-04T04:57:10.219675+0000","last_clean":"2024-06-04T04:57:10.219675+0000","last_became_active":"2024-06-04T04:53:36.383303+0000","last_became_peered":"2024-06-04T04:53:36.383303+0000","last_unstale":"2024-06-04T04:57:10.219675+0000","last_undegraded":"2024-06-04T04:57:10.219675+0000","last_fullsized":"2024-06-04T04:57:10.219675+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:26:31.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'15","reported_seq":262,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.048139+0000","last_change":"2024-06-04T04:54:19.641520+0000","last_active":"2024-06-04T04:57:10.048139+0000","last_peered":"2024-06-04T04:57:10.048139+0000","last_clean":"2024-06-04T04:57:10.048139+0000","last_became_active":"2024-06-04T04:54:19.632433+0000","last_became_peered":"2024-06-04T04:54:19.632433+0000","last_unstale":"2024-06-04T04:57:10.048139+0000","last_undegraded":"2024-06-04T04:57:10.048139+0000","last_fullsized":"2024-06-04T04:57:10.048139+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:26:05.993919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":192,"num_read_kb":192,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942727+0000","last_change":"2024-06-04T04:53:36.371827+0000","last_active":"2024-06-04T04:57:09.942727+0000","last_peered":"2024-06-04T04:57:09.942727+0000","last_clean":"2024-06-04T04:57:09.942727+0000","last_became_active":"2024-06-04T04:53:36.371509+0000","last_became_peered":"2024-06-04T04:53:36.371509+0000","last_unstale":"2024-06-04T04:57:09.942727+0000","last_undegraded":"2024-06-04T04:57:09.942727+0000","last_fullsized":"2024-06-04T04:57:09.942727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:54:51.589008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.943009+0000","last_change":"2024-06-04T04:53:33.353976+0000","last_active":"2024-06-04T04:57:09.943009+0000","last_peered":"2024-06-04T04:57:09.943009+0000","last_clean":"2024-06-04T04:57:09.943009+0000","last_became_active":"2024-06-04T04:53:33.353586+0000","last_became_peered":"2024-06-04T04:53:33.353586+0000","last_unstale":"2024-06-04T04:57:09.943009+0000","last_undegraded":"2024-06-04T04:57:09.943009+0000","last_fullsized":"2024-06-04T04:57:09.943009+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:17:10.378008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466390+0000","last_change":"2024-06-04T04:53:36.367806+0000","last_active":"2024-06-04T04:57:10.466390+0000","last_peered":"2024-06-04T04:57:10.466390+0000","last_clean":"2024-06-04T04:57:10.466390+0000","last_became_active":"2024-06-04T04:53:36.367484+0000","last_became_peered":"2024-06-04T04:53:36.367484+0000","last_unstale":"2024-06-04T04:57:10.466390+0000","last_undegraded":"2024-06-04T04:57:10.466390+0000","last_fullsized":"2024-06-04T04:57:10.466390+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:25:36.194746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"82'13","reported_seq":43,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965867+0000","last_change":"2024-06-04T04:53:33.348902+0000","last_active":"2024-06-04T04:57:09.965867+0000","last_peered":"2024-06-04T04:57:09.965867+0000","last_clean":"2024-06-04T04:57:09.965867+0000","last_became_active":"2024-06-04T04:53:33.348441+0000","last_became_peered":"2024-06-04T04:53:33.348441+0000","last_unstale":"2024-06-04T04:57:09.965867+0000","last_undegraded":"2024-06-04T04:57:09.965867+0000","last_fullsized":"2024-06-04T04:57:09.965867+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:22:28.428303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215265+0000","last_change":"2024-06-04T04:53:36.376867+0000","last_active":"2024-06-04T04:57:10.215265+0000","last_peered":"2024-06-04T04:57:10.215265+0000","last_clean":"2024-06-04T04:57:10.215265+0000","last_became_active":"2024-06-04T04:53:36.376517+0000","last_became_peered":"2024-06-04T04:53:36.376517+0000","last_unstale":"2024-06-04T04:57:10.215265+0000","last_undegraded":"2024-06-04T04:57:10.215265+0000","last_fullsized":"2024-06-04T04:57:10.215265+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:21:59.494321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965814+0000","last_change":"2024-06-04T04:53:33.347993+0000","last_active":"2024-06-04T04:57:09.965814+0000","last_peered":"2024-06-04T04:57:09.965814+0000","last_clean":"2024-06-04T04:57:09.965814+0000","last_became_active":"2024-06-04T04:53:33.347656+0000","last_became_peered":"2024-06-04T04:53:33.347656+0000","last_unstale":"2024-06-04T04:57:09.965814+0000","last_undegraded":"2024-06-04T04:57:09.965814+0000","last_fullsized":"2024-06-04T04:57:09.965814+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:21:26.806557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":44,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946660+0000","last_change":"2024-06-04T04:54:19.632790+0000","last_active":"2024-06-04T04:57:09.946660+0000","last_peered":"2024-06-04T04:57:09.946660+0000","last_clean":"2024-06-04T04:57:09.946660+0000","last_became_active":"2024-06-04T04:54:19.632338+0000","last_became_peered":"2024-06-04T04:54:19.632338+0000","last_unstale":"2024-06-04T04:57:09.946660+0000","last_undegraded":"2024-06-04T04:57:09.946660+0000","last_fullsized":"2024-06-04T04:57:09.946660+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:44:00.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"81'6","reported_seq":235,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946559+0000","last_change":"2024-06-04T04:53:33.346111+0000","last_active":"2024-06-04T04:57:09.946559+0000","last_peered":"2024-06-04T04:57:09.946559+0000","last_clean":"2024-06-04T04:57:09.946559+0000","last_became_active":"2024-06-04T04:53:33.345773+0000","last_became_peered":"2024-06-04T04:53:33.345773+0000","last_unstale":"2024-06-04T04:57:09.946559+0000","last_undegraded":"2024-06-04T04:57:09.946559+0000","last_fullsized":"2024-06-04T04:57:09.946559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:18.565583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1115,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":200,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965482+0000","last_change":"2024-06-04T04:53:36.373369+0000","last_active":"2024-06-04T04:57:09.965482+0000","last_peered":"2024-06-04T04:57:09.965482+0000","last_clean":"2024-06-04T04:57:09.965482+0000","last_became_active":"2024-06-04T04:53:36.373001+0000","last_became_peered":"2024-06-04T04:53:36.373001+0000","last_unstale":"2024-06-04T04:57:09.965482+0000","last_undegraded":"2024-06-04T04:57:09.965482+0000","last_fullsized":"2024-06-04T04:57:09.965482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:02:22.012643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946291+0000","last_change":"2024-06-04T04:53:33.344358+0000","last_active":"2024-06-04T04:57:09.946291+0000","last_peered":"2024-06-04T04:57:09.946291+0000","last_clean":"2024-06-04T04:57:09.946291+0000","last_became_active":"2024-06-04T04:53:33.343980+0000","last_became_peered":"2024-06-04T04:53:33.343980+0000","last_unstale":"2024-06-04T04:57:09.946291+0000","last_undegraded":"2024-06-04T04:57:09.946291+0000","last_fullsized":"2024-06-04T04:57:09.946291+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:14.493138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946579+0000","last_change":"2024-06-04T04:53:36.371239+0000","last_active":"2024-06-04T04:57:09.946579+0000","last_peered":"2024-06-04T04:57:09.946579+0000","last_clean":"2024-06-04T04:57:09.946579+0000","last_became_active":"2024-06-04T04:53:36.370885+0000","last_became_peered":"2024-06-04T04:53:36.370885+0000","last_unstale":"2024-06-04T04:57:09.946579+0000","last_undegraded":"2024-06-04T04:57:09.946579+0000","last_fullsized":"2024-06-04T04:57:09.946579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T04:54:29.753796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215252+0000","last_change":"2024-06-04T04:53:33.338512+0000","last_active":"2024-06-04T04:57:10.215252+0000","last_peered":"2024-06-04T04:57:10.215252+0000","last_clean":"2024-06-04T04:57:10.215252+0000","last_became_active":"2024-06-04T04:53:33.338180+0000","last_became_peered":"2024-06-04T04:53:33.338180+0000","last_unstale":"2024-06-04T04:57:10.215252+0000","last_undegraded":"2024-06-04T04:57:10.215252+0000","last_fullsized":"2024-06-04T04:57:10.215252+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:57:13.406479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942451+0000","last_change":"2024-06-04T04:53:36.377316+0000","last_active":"2024-06-04T04:57:09.942451+0000","last_peered":"2024-06-04T04:57:09.942451+0000","last_clean":"2024-06-04T04:57:09.942451+0000","last_became_active":"2024-06-04T04:53:36.376992+0000","last_became_peered":"2024-06-04T04:53:36.376992+0000","last_unstale":"2024-06-04T04:57:09.942451+0000","last_undegraded":"2024-06-04T04:57:09.942451+0000","last_fullsized":"2024-06-04T04:57:09.942451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:33.145631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946281+0000","last_change":"2024-06-04T04:53:33.348305+0000","last_active":"2024-06-04T04:57:09.946281+0000","last_peered":"2024-06-04T04:57:09.946281+0000","last_clean":"2024-06-04T04:57:09.946281+0000","last_became_active":"2024-06-04T04:53:33.347948+0000","last_became_peered":"2024-06-04T04:53:33.347948+0000","last_unstale":"2024-06-04T04:57:09.946281+0000","last_undegraded":"2024-06-04T04:57:09.946281+0000","last_fullsized":"2024-06-04T04:57:09.946281+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:04:13.669274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.047728+0000","last_change":"2024-06-04T04:53:36.367955+0000","last_active":"2024-06-04T04:57:10.047728+0000","last_peered":"2024-06-04T04:57:10.047728+0000","last_clean":"2024-06-04T04:57:10.047728+0000","last_became_active":"2024-06-04T04:53:36.367628+0000","last_became_peered":"2024-06-04T04:53:36.367628+0000","last_unstale":"2024-06-04T04:57:10.047728+0000","last_undegraded":"2024-06-04T04:57:10.047728+0000","last_fullsized":"2024-06-04T04:57:10.047728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:33:26.185498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":428,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.732694+0000","last_change":"2024-06-04T04:53:33.349471+0000","last_active":"2024-06-04T04:57:10.732694+0000","last_peered":"2024-06-04T04:57:10.732694+0000","last_clean":"2024-06-04T04:57:10.732694+0000","last_became_active":"2024-06-04T04:53:33.349105+0000","last_became_peered":"2024-06-04T04:53:33.349105+0000","last_unstale":"2024-06-04T04:57:10.732694+0000","last_undegraded":"2024-06-04T04:57:10.732694+0000","last_fullsized":"2024-06-04T04:57:10.732694+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:31:35.584275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":394,"num_read_kb":394,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215768+0000","last_change":"2024-06-04T04:53:36.367249+0000","last_active":"2024-06-04T04:57:10.215768+0000","last_peered":"2024-06-04T04:57:10.215768+0000","last_clean":"2024-06-04T04:57:10.215768+0000","last_became_active":"2024-06-04T04:53:36.366863+0000","last_became_peered":"2024-06-04T04:53:36.366863+0000","last_unstale":"2024-06-04T04:57:10.215768+0000","last_undegraded":"2024-06-04T04:57:10.215768+0000","last_fullsized":"2024-06-04T04:57:10.215768+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:13:17.308321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213781+0000","last_change":"2024-06-04T04:53:33.346778+0000","last_active":"2024-06-04T04:57:10.213781+0000","last_peered":"2024-06-04T04:57:10.213781+0000","last_clean":"2024-06-04T04:57:10.213781+0000","last_became_active":"2024-06-04T04:53:33.346376+0000","last_became_peered":"2024-06-04T04:53:33.346376+0000","last_unstale":"2024-06-04T04:57:10.213781+0000","last_undegraded":"2024-06-04T04:57:10.213781+0000","last_fullsized":"2024-06-04T04:57:10.213781+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:03:16.973822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466320+0000","last_change":"2024-06-04T04:54:19.514491+0000","last_active":"2024-06-04T04:57:10.466320+0000","last_peered":"2024-06-04T04:57:10.466320+0000","last_clean":"2024-06-04T04:57:10.466320+0000","last_became_active":"2024-06-04T04:54:19.514148+0000","last_became_peered":"2024-06-04T04:54:19.514148+0000","last_unstale":"2024-06-04T04:57:10.466320+0000","last_undegraded":"2024-06-04T04:57:10.466320+0000","last_fullsized":"2024-06-04T04:57:10.466320+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:09:59.641041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946792+0000","last_change":"2024-06-04T04:53:33.354989+0000","last_active":"2024-06-04T04:57:09.946792+0000","last_peered":"2024-06-04T04:57:09.946792+0000","last_clean":"2024-06-04T04:57:09.946792+0000","last_became_active":"2024-06-04T04:53:33.354613+0000","last_became_peered":"2024-06-04T04:53:33.354613+0000","last_unstale":"2024-06-04T04:57:09.946792+0000","last_undegraded":"2024-06-04T04:57:09.946792+0000","last_fullsized":"2024-06-04T04:57:09.946792+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:19:59.398763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215492+0000","last_change":"2024-06-04T04:53:36.372216+0000","last_active":"2024-06-04T04:57:10.215492+0000","last_peered":"2024-06-04T04:57:10.215492+0000","last_clean":"2024-06-04T04:57:10.215492+0000","last_became_active":"2024-06-04T04:53:36.371820+0000","last_became_peered":"2024-06-04T04:53:36.371820+0000","last_unstale":"2024-06-04T04:57:10.215492+0000","last_undegraded":"2024-06-04T04:57:10.215492+0000","last_fullsized":"2024-06-04T04:57:10.215492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:55:59.887963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'40","reported_seq":474,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.753604+0000","last_change":"2024-06-04T04:53:33.345493+0000","last_active":"2024-06-04T04:57:10.753604+0000","last_peered":"2024-06-04T04:57:10.753604+0000","last_clean":"2024-06-04T04:57:10.753604+0000","last_became_active":"2024-06-04T04:53:33.345102+0000","last_became_peered":"2024-06-04T04:53:33.345102+0000","last_unstale":"2024-06-04T04:57:10.753604+0000","last_undegraded":"2024-06-04T04:57:10.753604+0000","last_fullsized":"2024-06-04T04:57:10.753604+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:50:27.853391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71650,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":406,"num_read_kb":273,"num_write":42,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965646+0000","last_change":"2024-06-04T04:53:36.382070+0000","last_active":"2024-06-04T04:57:09.965646+0000","last_peered":"2024-06-04T04:57:09.965646+0000","last_clean":"2024-06-04T04:57:09.965646+0000","last_became_active":"2024-06-04T04:53:36.379144+0000","last_became_peered":"2024-06-04T04:53:36.379144+0000","last_unstale":"2024-06-04T04:57:09.965646+0000","last_undegraded":"2024-06-04T04:57:09.965646+0000","last_fullsized":"2024-06-04T04:57:09.965646+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:16.702709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219432+0000","last_change":"2024-06-04T04:53:33.344726+0000","last_active":"2024-06-04T04:57:10.219432+0000","last_peered":"2024-06-04T04:57:10.219432+0000","last_clean":"2024-06-04T04:57:10.219432+0000","last_became_active":"2024-06-04T04:53:33.344360+0000","last_became_peered":"2024-06-04T04:53:33.344360+0000","last_unstale":"2024-06-04T04:57:10.219432+0000","last_undegraded":"2024-06-04T04:57:10.219432+0000","last_fullsized":"2024-06-04T04:57:10.219432+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:44:56.784808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215391+0000","last_change":"2024-06-04T04:53:36.379036+0000","last_active":"2024-06-04T04:57:10.215391+0000","last_peered":"2024-06-04T04:57:10.215391+0000","last_clean":"2024-06-04T04:57:10.215391+0000","last_became_active":"2024-06-04T04:53:36.378681+0000","last_became_peered":"2024-06-04T04:53:36.378681+0000","last_unstale":"2024-06-04T04:57:10.215391+0000","last_undegraded":"2024-06-04T04:57:10.215391+0000","last_fullsized":"2024-06-04T04:57:10.215391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:07:49.704731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946436+0000","last_change":"2024-06-04T04:53:33.345915+0000","last_active":"2024-06-04T04:57:09.946436+0000","last_peered":"2024-06-04T04:57:09.946436+0000","last_clean":"2024-06-04T04:57:09.946436+0000","last_became_active":"2024-06-04T04:53:33.345537+0000","last_became_peered":"2024-06-04T04:53:33.345537+0000","last_unstale":"2024-06-04T04:57:09.946436+0000","last_undegraded":"2024-06-04T04:57:09.946436+0000","last_fullsized":"2024-06-04T04:57:09.946436+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:06:44.086968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213503+0000","last_change":"2024-06-04T04:53:36.379094+0000","last_active":"2024-06-04T04:57:10.213503+0000","last_peered":"2024-06-04T04:57:10.213503+0000","last_clean":"2024-06-04T04:57:10.213503+0000","last_became_active":"2024-06-04T04:53:36.378763+0000","last_became_peered":"2024-06-04T04:53:36.378763+0000","last_unstale":"2024-06-04T04:57:10.213503+0000","last_undegraded":"2024-06-04T04:57:10.213503+0000","last_fullsized":"2024-06-04T04:57:10.213503+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:16.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'6","reported_seq":235,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.465965+0000","last_change":"2024-06-04T04:53:33.341342+0000","last_active":"2024-06-04T04:57:10.465965+0000","last_peered":"2024-06-04T04:57:10.465965+0000","last_clean":"2024-06-04T04:57:10.465965+0000","last_became_active":"2024-06-04T04:53:33.340862+0000","last_became_peered":"2024-06-04T04:53:33.340862+0000","last_unstale":"2024-06-04T04:57:10.465965+0000","last_undegraded":"2024-06-04T04:57:10.465965+0000","last_fullsized":"2024-06-04T04:57:10.465965+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:26.818063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":201,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213135+0000","last_change":"2024-06-04T04:53:36.383124+0000","last_active":"2024-06-04T04:57:10.213135+0000","last_peered":"2024-06-04T04:57:10.213135+0000","last_clean":"2024-06-04T04:57:10.213135+0000","last_became_active":"2024-06-04T04:53:36.382449+0000","last_became_peered":"2024-06-04T04:53:36.382449+0000","last_unstale":"2024-06-04T04:57:10.213135+0000","last_undegraded":"2024-06-04T04:57:10.213135+0000","last_fullsized":"2024-06-04T04:57:10.213135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:52:40.776003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":47,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965391+0000","last_change":"2024-06-04T04:53:33.354556+0000","last_active":"2024-06-04T04:57:09.965391+0000","last_peered":"2024-06-04T04:57:09.965391+0000","last_clean":"2024-06-04T04:57:09.965391+0000","last_became_active":"2024-06-04T04:53:33.354238+0000","last_became_peered":"2024-06-04T04:53:33.354238+0000","last_unstale":"2024-06-04T04:57:09.965391+0000","last_undegraded":"2024-06-04T04:57:09.965391+0000","last_fullsized":"2024-06-04T04:57:09.965391+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:54.697374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946345+0000","last_change":"2024-06-04T04:53:36.371387+0000","last_active":"2024-06-04T04:57:09.946345+0000","last_peered":"2024-06-04T04:57:09.946345+0000","last_clean":"2024-06-04T04:57:09.946345+0000","last_became_active":"2024-06-04T04:53:36.370852+0000","last_became_peered":"2024-06-04T04:53:36.370852+0000","last_unstale":"2024-06-04T04:57:09.946345+0000","last_undegraded":"2024-06-04T04:57:09.946345+0000","last_fullsized":"2024-06-04T04:57:09.946345+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:44:06.956210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'3","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965694+0000","last_change":"2024-06-04T04:53:33.349982+0000","last_active":"2024-06-04T04:57:09.965694+0000","last_peered":"2024-06-04T04:57:09.965694+0000","last_clean":"2024-06-04T04:57:09.965694+0000","last_became_active":"2024-06-04T04:53:33.349633+0000","last_became_peered":"2024-06-04T04:53:33.349633+0000","last_unstale":"2024-06-04T04:57:09.965694+0000","last_undegraded":"2024-06-04T04:57:09.965694+0000","last_fullsized":"2024-06-04T04:57:09.965694+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:40.107426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466355+0000","last_change":"2024-06-04T04:53:36.382313+0000","last_active":"2024-06-04T04:57:10.466355+0000","last_peered":"2024-06-04T04:57:10.466355+0000","last_clean":"2024-06-04T04:57:10.466355+0000","last_became_active":"2024-06-04T04:53:36.379638+0000","last_became_peered":"2024-06-04T04:53:36.379638+0000","last_unstale":"2024-06-04T04:57:10.466355+0000","last_undegraded":"2024-06-04T04:57:10.466355+0000","last_fullsized":"2024-06-04T04:57:10.466355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:43:42.527262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.048030+0000","last_change":"2024-06-04T04:54:19.513581+0000","last_active":"2024-06-04T04:57:10.048030+0000","last_peered":"2024-06-04T04:57:10.048030+0000","last_clean":"2024-06-04T04:57:10.048030+0000","last_became_active":"2024-06-04T04:54:19.513111+0000","last_became_peered":"2024-06-04T04:54:19.513111+0000","last_unstale":"2024-06-04T04:57:10.048030+0000","last_undegraded":"2024-06-04T04:57:10.048030+0000","last_fullsized":"2024-06-04T04:57:10.048030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:57.028083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946526+0000","last_change":"2024-06-04T04:53:36.369904+0000","last_active":"2024-06-04T04:57:09.946526+0000","last_peered":"2024-06-04T04:57:09.946526+0000","last_clean":"2024-06-04T04:57:09.946526+0000","last_became_active":"2024-06-04T04:53:36.367929+0000","last_became_peered":"2024-06-04T04:53:36.367929+0000","last_unstale":"2024-06-04T04:57:09.946526+0000","last_undegraded":"2024-06-04T04:57:09.946526+0000","last_fullsized":"2024-06-04T04:57:09.946526+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:37:08.721569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"82'10","reported_seq":241,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.900365+0000","last_change":"2024-06-04T04:53:33.342723+0000","last_active":"2024-06-04T04:57:10.900365+0000","last_peered":"2024-06-04T04:57:10.900365+0000","last_clean":"2024-06-04T04:57:10.900365+0000","last_became_active":"2024-06-04T04:53:33.342344+0000","last_became_peered":"2024-06-04T04:53:33.342344+0000","last_unstale":"2024-06-04T04:57:10.900365+0000","last_undegraded":"2024-06-04T04:57:10.900365+0000","last_fullsized":"2024-06-04T04:57:10.900365+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:59:50.147517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":202,"num_read_kb":5,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946315+0000","last_change":"2024-06-04T04:54:19.513534+0000","last_active":"2024-06-04T04:57:09.946315+0000","last_peered":"2024-06-04T04:57:09.946315+0000","last_clean":"2024-06-04T04:57:09.946315+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:57:09.946315+0000","last_undegraded":"2024-06-04T04:57:09.946315+0000","last_fullsized":"2024-06-04T04:57:09.946315+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:16:32.240932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946648+0000","last_change":"2024-06-04T04:53:36.371236+0000","last_active":"2024-06-04T04:57:09.946648+0000","last_peered":"2024-06-04T04:57:09.946648+0000","last_clean":"2024-06-04T04:57:09.946648+0000","last_became_active":"2024-06-04T04:53:36.370876+0000","last_became_peered":"2024-06-04T04:53:36.370876+0000","last_unstale":"2024-06-04T04:57:09.946648+0000","last_undegraded":"2024-06-04T04:57:09.946648+0000","last_fullsized":"2024-06-04T04:57:09.946648+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:15:56.751317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":232,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215195+0000","last_change":"2024-06-04T04:53:33.355642+0000","last_active":"2024-06-04T04:57:10.215195+0000","last_peered":"2024-06-04T04:57:10.215195+0000","last_clean":"2024-06-04T04:57:10.215195+0000","last_became_active":"2024-06-04T04:53:33.355187+0000","last_became_peered":"2024-06-04T04:53:33.355187+0000","last_unstale":"2024-06-04T04:57:10.215195+0000","last_undegraded":"2024-06-04T04:57:10.215195+0000","last_fullsized":"2024-06-04T04:57:10.215195+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:32:28.331518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":198,"num_read_kb":198,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.047902+0000","last_change":"2024-06-04T04:54:19.631750+0000","last_active":"2024-06-04T04:57:10.047902+0000","last_peered":"2024-06-04T04:57:10.047902+0000","last_clean":"2024-06-04T04:57:10.047902+0000","last_became_active":"2024-06-04T04:54:19.631485+0000","last_became_peered":"2024-06-04T04:54:19.631485+0000","last_unstale":"2024-06-04T04:57:10.047902+0000","last_undegraded":"2024-06-04T04:57:10.047902+0000","last_fullsized":"2024-06-04T04:57:10.047902+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:29:44.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466037+0000","last_change":"2024-06-04T04:53:33.340653+0000","last_active":"2024-06-04T04:57:10.466037+0000","last_peered":"2024-06-04T04:57:10.466037+0000","last_clean":"2024-06-04T04:57:10.466037+0000","last_became_active":"2024-06-04T04:53:33.340250+0000","last_became_peered":"2024-06-04T04:53:33.340250+0000","last_unstale":"2024-06-04T04:57:10.466037+0000","last_undegraded":"2024-06-04T04:57:10.466037+0000","last_fullsized":"2024-06-04T04:57:10.466037+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:46:20.329514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":103,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213299+0000","last_change":"2024-06-04T04:52:09.181155+0000","last_active":"2024-06-04T04:57:10.213299+0000","last_peered":"2024-06-04T04:57:10.213299+0000","last_clean":"2024-06-04T04:57:10.213299+0000","last_became_active":"2024-06-04T04:52:09.177002+0000","last_became_peered":"2024-06-04T04:52:09.177002+0000","last_unstale":"2024-06-04T04:57:10.213299+0000","last_undegraded":"2024-06-04T04:57:10.213299+0000","last_fullsized":"2024-06-04T04:57:10.213299+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_clean_scrub_stamp":"2024-06-04T04:48:44.631948+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:46:57.735302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215170+0000","last_change":"2024-06-04T04:53:36.372634+0000","last_active":"2024-06-04T04:57:10.215170+0000","last_peered":"2024-06-04T04:57:10.215170+0000","last_clean":"2024-06-04T04:57:10.215170+0000","last_became_active":"2024-06-04T04:53:36.372290+0000","last_became_peered":"2024-06-04T04:53:36.372290+0000","last_unstale":"2024-06-04T04:57:10.215170+0000","last_undegraded":"2024-06-04T04:57:10.215170+0000","last_fullsized":"2024-06-04T04:57:10.215170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:14:07.460382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213392+0000","last_change":"2024-06-04T04:53:33.348760+0000","last_active":"2024-06-04T04:57:10.213392+0000","last_peered":"2024-06-04T04:57:10.213392+0000","last_clean":"2024-06-04T04:57:10.213392+0000","last_became_active":"2024-06-04T04:53:33.348347+0000","last_became_peered":"2024-06-04T04:53:33.348347+0000","last_unstale":"2024-06-04T04:57:10.213392+0000","last_undegraded":"2024-06-04T04:57:10.213392+0000","last_fullsized":"2024-06-04T04:57:10.213392+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:36:20.641061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965676+0000","last_change":"2024-06-04T04:53:33.354840+0000","last_active":"2024-06-04T04:57:09.965676+0000","last_peered":"2024-06-04T04:57:09.965676+0000","last_clean":"2024-06-04T04:57:09.965676+0000","last_became_active":"2024-06-04T04:53:33.354287+0000","last_became_peered":"2024-06-04T04:53:33.354287+0000","last_unstale":"2024-06-04T04:57:09.965676+0000","last_undegraded":"2024-06-04T04:57:09.965676+0000","last_fullsized":"2024-06-04T04:57:09.965676+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:42.550098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215686+0000","last_change":"2024-06-04T04:53:36.372678+0000","last_active":"2024-06-04T04:57:10.215686+0000","last_peered":"2024-06-04T04:57:10.215686+0000","last_clean":"2024-06-04T04:57:10.215686+0000","last_became_active":"2024-06-04T04:53:36.372312+0000","last_became_peered":"2024-06-04T04:53:36.372312+0000","last_unstale":"2024-06-04T04:57:10.215686+0000","last_undegraded":"2024-06-04T04:57:10.215686+0000","last_fullsized":"2024-06-04T04:57:10.215686+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:34:41.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'51","reported_seq":103,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.173057+0000","last_change":"2024-06-04T04:53:33.348440+0000","last_active":"2024-06-04T04:57:10.173057+0000","last_peered":"2024-06-04T04:57:10.173057+0000","last_clean":"2024-06-04T04:57:10.173057+0000","last_became_active":"2024-06-04T04:53:33.348131+0000","last_became_peered":"2024-06-04T04:53:33.348131+0000","last_unstale":"2024-06-04T04:57:10.173057+0000","last_undegraded":"2024-06-04T04:57:10.173057+0000","last_fullsized":"2024-06-04T04:57:10.173057+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:14:11.552286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":91,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942741+0000","last_change":"2024-06-04T04:53:36.378669+0000","last_active":"2024-06-04T04:57:09.942741+0000","last_peered":"2024-06-04T04:57:09.942741+0000","last_clean":"2024-06-04T04:57:09.942741+0000","last_became_active":"2024-06-04T04:53:36.378233+0000","last_became_peered":"2024-06-04T04:53:36.378233+0000","last_unstale":"2024-06-04T04:57:09.942741+0000","last_undegraded":"2024-06-04T04:57:09.942741+0000","last_fullsized":"2024-06-04T04:57:09.942741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:36:45.870924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466139+0000","last_change":"2024-06-04T04:53:33.345078+0000","last_active":"2024-06-04T04:57:10.466139+0000","last_peered":"2024-06-04T04:57:10.466139+0000","last_clean":"2024-06-04T04:57:10.466139+0000","last_became_active":"2024-06-04T04:53:33.344708+0000","last_became_peered":"2024-06-04T04:53:33.344708+0000","last_unstale":"2024-06-04T04:57:10.466139+0000","last_undegraded":"2024-06-04T04:57:10.466139+0000","last_fullsized":"2024-06-04T04:57:10.466139+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:45:14.010758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466300+0000","last_change":"2024-06-04T04:53:36.364817+0000","last_active":"2024-06-04T04:57:10.466300+0000","last_peered":"2024-06-04T04:57:10.466300+0000","last_clean":"2024-06-04T04:57:10.466300+0000","last_became_active":"2024-06-04T04:53:36.364499+0000","last_became_peered":"2024-06-04T04:53:36.364499+0000","last_unstale":"2024-06-04T04:57:10.466300+0000","last_undegraded":"2024-06-04T04:57:10.466300+0000","last_fullsized":"2024-06-04T04:57:10.466300+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:39:30.518028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215210+0000","last_change":"2024-06-04T04:53:33.354527+0000","last_active":"2024-06-04T04:57:10.215210+0000","last_peered":"2024-06-04T04:57:10.215210+0000","last_clean":"2024-06-04T04:57:10.215210+0000","last_became_active":"2024-06-04T04:53:33.354175+0000","last_became_peered":"2024-06-04T04:53:33.354175+0000","last_unstale":"2024-06-04T04:57:10.215210+0000","last_undegraded":"2024-06-04T04:57:10.215210+0000","last_fullsized":"2024-06-04T04:57:10.215210+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:33:08.125428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965391+0000","last_change":"2024-06-04T04:53:36.373248+0000","last_active":"2024-06-04T04:57:09.965391+0000","last_peered":"2024-06-04T04:57:09.965391+0000","last_clean":"2024-06-04T04:57:09.965391+0000","last_became_active":"2024-06-04T04:53:36.372915+0000","last_became_peered":"2024-06-04T04:53:36.372915+0000","last_unstale":"2024-06-04T04:57:09.965391+0000","last_undegraded":"2024-06-04T04:57:09.965391+0000","last_fullsized":"2024-06-04T04:57:09.965391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:23:11.689349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942495+0000","last_change":"2024-06-04T04:53:33.340427+0000","last_active":"2024-06-04T04:57:09.942495+0000","last_peered":"2024-06-04T04:57:09.942495+0000","last_clean":"2024-06-04T04:57:09.942495+0000","last_became_active":"2024-06-04T04:53:33.340022+0000","last_became_peered":"2024-06-04T04:53:33.340022+0000","last_unstale":"2024-06-04T04:57:09.942495+0000","last_undegraded":"2024-06-04T04:57:09.942495+0000","last_fullsized":"2024-06-04T04:57:09.942495+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:56:16.496471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466096+0000","last_change":"2024-06-04T04:53:36.380114+0000","last_active":"2024-06-04T04:57:10.466096+0000","last_peered":"2024-06-04T04:57:10.466096+0000","last_clean":"2024-06-04T04:57:10.466096+0000","last_became_active":"2024-06-04T04:53:36.379392+0000","last_became_peered":"2024-06-04T04:53:36.379392+0000","last_unstale":"2024-06-04T04:57:10.466096+0000","last_undegraded":"2024-06-04T04:57:10.466096+0000","last_fullsized":"2024-06-04T04:57:10.466096+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:41:48.776439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"81'5","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466122+0000","last_change":"2024-06-04T04:54:19.519755+0000","last_active":"2024-06-04T04:57:10.466122+0000","last_peered":"2024-06-04T04:57:10.466122+0000","last_clean":"2024-06-04T04:57:10.466122+0000","last_became_active":"2024-06-04T04:54:19.513992+0000","last_became_peered":"2024-06-04T04:54:19.513992+0000","last_unstale":"2024-06-04T04:57:10.466122+0000","last_undegraded":"2024-06-04T04:57:10.466122+0000","last_fullsized":"2024-06-04T04:57:10.466122+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:01.951935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942779+0000","last_change":"2024-06-04T04:53:36.377420+0000","last_active":"2024-06-04T04:57:09.942779+0000","last_peered":"2024-06-04T04:57:09.942779+0000","last_clean":"2024-06-04T04:57:09.942779+0000","last_became_active":"2024-06-04T04:53:36.377109+0000","last_became_peered":"2024-06-04T04:53:36.377109+0000","last_unstale":"2024-06-04T04:57:09.942779+0000","last_undegraded":"2024-06-04T04:57:09.942779+0000","last_fullsized":"2024-06-04T04:57:09.942779+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:47:12.102396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965575+0000","last_change":"2024-06-04T04:54:19.645328+0000","last_active":"2024-06-04T04:57:09.965575+0000","last_peered":"2024-06-04T04:57:09.965575+0000","last_clean":"2024-06-04T04:57:09.965575+0000","last_became_active":"2024-06-04T04:54:19.633416+0000","last_became_peered":"2024-06-04T04:54:19.633416+0000","last_unstale":"2024-06-04T04:57:09.965575+0000","last_undegraded":"2024-06-04T04:57:09.965575+0000","last_fullsized":"2024-06-04T04:57:09.965575+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:04:20.494641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213349+0000","last_change":"2024-06-04T04:53:36.381187+0000","last_active":"2024-06-04T04:57:10.213349+0000","last_peered":"2024-06-04T04:57:10.213349+0000","last_clean":"2024-06-04T04:57:10.213349+0000","last_became_active":"2024-06-04T04:53:36.380591+0000","last_became_peered":"2024-06-04T04:53:36.380591+0000","last_unstale":"2024-06-04T04:57:10.213349+0000","last_undegraded":"2024-06-04T04:57:10.213349+0000","last_fullsized":"2024-06-04T04:57:10.213349+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:26:43.122339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'6","reported_seq":234,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219295+0000","last_change":"2024-06-04T04:53:33.355908+0000","last_active":"2024-06-04T04:57:10.219295+0000","last_peered":"2024-06-04T04:57:10.219295+0000","last_clean":"2024-06-04T04:57:10.219295+0000","last_became_active":"2024-06-04T04:53:33.355522+0000","last_became_peered":"2024-06-04T04:53:33.355522+0000","last_unstale":"2024-06-04T04:57:10.219295+0000","last_undegraded":"2024-06-04T04:57:10.219295+0000","last_fullsized":"2024-06-04T04:57:10.219295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:16:44.639431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":200,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213745+0000","last_change":"2024-06-04T04:53:36.382737+0000","last_active":"2024-06-04T04:57:10.213745+0000","last_peered":"2024-06-04T04:57:10.213745+0000","last_clean":"2024-06-04T04:57:10.213745+0000","last_became_active":"2024-06-04T04:53:36.382353+0000","last_became_peered":"2024-06-04T04:53:36.382353+0000","last_unstale":"2024-06-04T04:57:10.213745+0000","last_undegraded":"2024-06-04T04:57:10.213745+0000","last_fullsized":"2024-06-04T04:57:10.213745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:18:01.476058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'41","reported_seq":78,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.170281+0000","last_change":"2024-06-04T04:53:33.347273+0000","last_active":"2024-06-04T04:57:10.170281+0000","last_peered":"2024-06-04T04:57:10.170281+0000","last_clean":"2024-06-04T04:57:10.170281+0000","last_became_active":"2024-06-04T04:53:33.346915+0000","last_became_peered":"2024-06-04T04:53:33.346915+0000","last_unstale":"2024-06-04T04:57:10.170281+0000","last_undegraded":"2024-06-04T04:57:10.170281+0000","last_fullsized":"2024-06-04T04:57:10.170281+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:17.235484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213559+0000","last_change":"2024-06-04T04:53:36.383454+0000","last_active":"2024-06-04T04:57:10.213559+0000","last_peered":"2024-06-04T04:57:10.213559+0000","last_clean":"2024-06-04T04:57:10.213559+0000","last_became_active":"2024-06-04T04:53:36.383105+0000","last_became_peered":"2024-06-04T04:53:36.383105+0000","last_unstale":"2024-06-04T04:57:10.213559+0000","last_undegraded":"2024-06-04T04:57:10.213559+0000","last_fullsized":"2024-06-04T04:57:10.213559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:04:02.147688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215701+0000","last_change":"2024-06-04T04:53:33.345435+0000","last_active":"2024-06-04T04:57:10.215701+0000","last_peered":"2024-06-04T04:57:10.215701+0000","last_clean":"2024-06-04T04:57:10.215701+0000","last_became_active":"2024-06-04T04:53:33.345077+0000","last_became_peered":"2024-06-04T04:53:33.345077+0000","last_unstale":"2024-06-04T04:57:10.215701+0000","last_undegraded":"2024-06-04T04:57:10.215701+0000","last_fullsized":"2024-06-04T04:57:10.215701+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:20:13.992259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213743+0000","last_change":"2024-06-04T04:53:36.385238+0000","last_active":"2024-06-04T04:57:10.213743+0000","last_peered":"2024-06-04T04:57:10.213743+0000","last_clean":"2024-06-04T04:57:10.213743+0000","last_became_active":"2024-06-04T04:53:36.384899+0000","last_became_peered":"2024-06-04T04:53:36.384899+0000","last_unstale":"2024-06-04T04:57:10.213743+0000","last_undegraded":"2024-06-04T04:57:10.213743+0000","last_fullsized":"2024-06-04T04:57:10.213743+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:09:42.629340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466183+0000","last_change":"2024-06-04T04:53:36.383450+0000","last_active":"2024-06-04T04:57:10.466183+0000","last_peered":"2024-06-04T04:57:10.466183+0000","last_clean":"2024-06-04T04:57:10.466183+0000","last_became_active":"2024-06-04T04:53:36.382678+0000","last_became_peered":"2024-06-04T04:53:36.382678+0000","last_unstale":"2024-06-04T04:57:10.466183+0000","last_undegraded":"2024-06-04T04:57:10.466183+0000","last_fullsized":"2024-06-04T04:57:10.466183+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:36:28.932636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219063+0000","last_change":"2024-06-04T04:54:19.632563+0000","last_active":"2024-06-04T04:57:10.219063+0000","last_peered":"2024-06-04T04:57:10.219063+0000","last_clean":"2024-06-04T04:57:10.219063+0000","last_became_active":"2024-06-04T04:54:19.632070+0000","last_became_peered":"2024-06-04T04:54:19.632070+0000","last_unstale":"2024-06-04T04:57:10.219063+0000","last_undegraded":"2024-06-04T04:57:10.219063+0000","last_fullsized":"2024-06-04T04:57:10.219063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:07:16.552958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219185+0000","last_change":"2024-06-04T04:53:36.381408+0000","last_active":"2024-06-04T04:57:10.219185+0000","last_peered":"2024-06-04T04:57:10.219185+0000","last_clean":"2024-06-04T04:57:10.219185+0000","last_became_active":"2024-06-04T04:53:36.381078+0000","last_became_peered":"2024-06-04T04:53:36.381078+0000","last_unstale":"2024-06-04T04:57:10.219185+0000","last_undegraded":"2024-06-04T04:57:10.219185+0000","last_fullsized":"2024-06-04T04:57:10.219185+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:28:48.988591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466464+0000","last_change":"2024-06-04T04:53:36.360168+0000","last_active":"2024-06-04T04:57:10.466464+0000","last_peered":"2024-06-04T04:57:10.466464+0000","last_clean":"2024-06-04T04:57:10.466464+0000","last_became_active":"2024-06-04T04:53:36.359691+0000","last_became_peered":"2024-06-04T04:53:36.359691+0000","last_unstale":"2024-06-04T04:57:10.466464+0000","last_undegraded":"2024-06-04T04:57:10.466464+0000","last_fullsized":"2024-06-04T04:57:10.466464+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:55:17.418339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215688+0000","last_change":"2024-06-04T04:53:36.363018+0000","last_active":"2024-06-04T04:57:10.215688+0000","last_peered":"2024-06-04T04:57:10.215688+0000","last_clean":"2024-06-04T04:57:10.215688+0000","last_became_active":"2024-06-04T04:53:36.361595+0000","last_became_peered":"2024-06-04T04:53:36.361595+0000","last_unstale":"2024-06-04T04:57:10.215688+0000","last_undegraded":"2024-06-04T04:57:10.215688+0000","last_fullsized":"2024-06-04T04:57:10.215688+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:00:21.377322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466289+0000","last_change":"2024-06-04T04:53:36.375377+0000","last_active":"2024-06-04T04:57:10.466289+0000","last_peered":"2024-06-04T04:57:10.466289+0000","last_clean":"2024-06-04T04:57:10.466289+0000","last_became_active":"2024-06-04T04:53:36.373459+0000","last_became_peered":"2024-06-04T04:53:36.373459+0000","last_unstale":"2024-06-04T04:57:10.466289+0000","last_undegraded":"2024-06-04T04:57:10.466289+0000","last_fullsized":"2024-06-04T04:57:10.466289+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:51:31.686969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219589+0000","last_change":"2024-06-04T04:53:36.373850+0000","last_active":"2024-06-04T04:57:10.219589+0000","last_peered":"2024-06-04T04:57:10.219589+0000","last_clean":"2024-06-04T04:57:10.219589+0000","last_became_active":"2024-06-04T04:53:36.371951+0000","last_became_peered":"2024-06-04T04:53:36.371951+0000","last_unstale":"2024-06-04T04:57:10.219589+0000","last_undegraded":"2024-06-04T04:57:10.219589+0000","last_fullsized":"2024-06-04T04:57:10.219589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:06.452703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942719+0000","last_change":"2024-06-04T04:53:36.374871+0000","last_active":"2024-06-04T04:57:09.942719+0000","last_peered":"2024-06-04T04:57:09.942719+0000","last_clean":"2024-06-04T04:57:09.942719+0000","last_became_active":"2024-06-04T04:53:36.374486+0000","last_became_peered":"2024-06-04T04:53:36.374486+0000","last_unstale":"2024-06-04T04:57:09.942719+0000","last_undegraded":"2024-06-04T04:57:09.942719+0000","last_fullsized":"2024-06-04T04:57:09.942719+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:15:13.337374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213230+0000","last_change":"2024-06-04T04:54:19.513540+0000","last_active":"2024-06-04T04:57:10.213230+0000","last_peered":"2024-06-04T04:57:10.213230+0000","last_clean":"2024-06-04T04:57:10.213230+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:57:10.213230+0000","last_undegraded":"2024-06-04T04:57:10.213230+0000","last_fullsized":"2024-06-04T04:57:10.213230+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:09:27.993585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942434+0000","last_change":"2024-06-04T04:53:36.373055+0000","last_active":"2024-06-04T04:57:09.942434+0000","last_peered":"2024-06-04T04:57:09.942434+0000","last_clean":"2024-06-04T04:57:09.942434+0000","last_became_active":"2024-06-04T04:53:36.372754+0000","last_became_peered":"2024-06-04T04:53:36.372754+0000","last_unstale":"2024-06-04T04:57:09.942434+0000","last_undegraded":"2024-06-04T04:57:09.942434+0000","last_fullsized":"2024-06-04T04:57:09.942434+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:22:27.554200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215415+0000","last_change":"2024-06-04T04:53:36.378897+0000","last_active":"2024-06-04T04:57:10.215415+0000","last_peered":"2024-06-04T04:57:10.215415+0000","last_clean":"2024-06-04T04:57:10.215415+0000","last_became_active":"2024-06-04T04:53:36.378607+0000","last_became_peered":"2024-06-04T04:53:36.378607+0000","last_unstale":"2024-06-04T04:57:10.215415+0000","last_undegraded":"2024-06-04T04:57:10.215415+0000","last_fullsized":"2024-06-04T04:57:10.215415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:03.395491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215502+0000","last_change":"2024-06-04T04:53:36.378240+0000","last_active":"2024-06-04T04:57:10.215502+0000","last_peered":"2024-06-04T04:57:10.215502+0000","last_clean":"2024-06-04T04:57:10.215502+0000","last_became_active":"2024-06-04T04:53:36.377909+0000","last_became_peered":"2024-06-04T04:53:36.377909+0000","last_unstale":"2024-06-04T04:57:10.215502+0000","last_undegraded":"2024-06-04T04:57:10.215502+0000","last_fullsized":"2024-06-04T04:57:10.215502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:15:25.288340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466255+0000","last_change":"2024-06-04T04:53:36.379792+0000","last_active":"2024-06-04T04:57:10.466255+0000","last_peered":"2024-06-04T04:57:10.466255+0000","last_clean":"2024-06-04T04:57:10.466255+0000","last_became_active":"2024-06-04T04:53:36.379438+0000","last_became_peered":"2024-06-04T04:53:36.379438+0000","last_unstale":"2024-06-04T04:57:10.466255+0000","last_undegraded":"2024-06-04T04:57:10.466255+0000","last_fullsized":"2024-06-04T04:57:10.466255+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:23:57.617877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108276,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":606174,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2105,"num_read_kb":1171,"num_write":385,"num_write_kb":1476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":724992,"data_stored":1818522,"data_compressed":81405,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":308,"ondisk_log_size":308,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678074,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":880640,"data_stored":430053,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874302,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1052672,"data_stored":1097012,"data_compressed":46576,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037828,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":429490,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234058,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":831397,"data_compressed":27797,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430285,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":1024006,"data_compressed":41851,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593812,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":901120,"data_stored":439999,"data_compressed":8651,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":498762,"data_compressed":7592,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738482,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":909312,"data_stored":431039,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411431,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":511961,"data_compressed":10484,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":962833,"data_compressed":33953,"data_compressed_allocated":45056,"data_compressed_original":540672,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953566,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32044,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713108,"statfs":{"total":95995035648,"available":95962222592,"internally_reserved":0,"allocated":888832,"data_stored":429561,"data_compressed":5759,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790041,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":999424,"data_stored":1162751,"data_compressed":50899,"data_compressed_allocated":65536,"data_compressed_original":737280,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":75881,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":74608,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":134913,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":394044,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":205479,"data_compressed":7989,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-04T04:57:14.879 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:57:15.158 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:15.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:15 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi112 to 176.1M 2024-06-04T04:57:15.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:15 smithi098 ceph-mon[35569]: Unable to set osd_memory_target on smithi112 to 184710758: error parsing value: Value '184710758' is below minimum 939524096 2024-06-04T04:57:15.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:15 smithi098 ceph-mon[35569]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:57:15.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:15 smithi098 ceph-mon[35569]: from='client.24985 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:15.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:15 smithi098 ceph-mon[35569]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:15.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:15 smithi098 ceph-mon[35569]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-06-04T04:57:15.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:15 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi112 to 176.1M 2024-06-04T04:57:15.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:15 smithi070 ceph-mon[32631]: Unable to set osd_memory_target on smithi112 to 184710758: error parsing value: Value '184710758' is below minimum 939524096 2024-06-04T04:57:15.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:15 smithi070 ceph-mon[32631]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:57:15.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:15 smithi070 ceph-mon[32631]: from='client.24985 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:15.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:15 smithi070 ceph-mon[32631]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:15.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:15 smithi070 ceph-mon[32631]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-06-04T04:57:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:15 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi112 to 176.1M 2024-06-04T04:57:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:15 smithi112 ceph-mon[36453]: Unable to set osd_memory_target on smithi112 to 184710758: error parsing value: Value '184710758' is below minimum 939524096 2024-06-04T04:57:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:15 smithi112 ceph-mon[36453]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T04:57:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:15 smithi112 ceph-mon[36453]: from='client.24985 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:15 smithi112 ceph-mon[36453]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:15 smithi112 ceph-mon[36453]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-06-04T04:57:16.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:16 smithi070 ceph-mon[32631]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:16.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:16 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:16.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:16 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:16.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:16 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:16.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:16 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:16.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:16 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:16.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:16 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:16.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:16 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi070 to 176.1M 2024-06-04T04:57:16.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:16 smithi070 ceph-mon[32631]: Unable to set osd_memory_target on smithi070 to 184717209: error parsing value: Value '184717209' is below minimum 939524096 2024-06-04T04:57:16.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:16 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:57:16.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:16 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:57:16.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:16 smithi070 ceph-mon[32631]: Updating smithi070:/etc/ceph/ceph.conf 2024-06-04T04:57:16.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:16 smithi070 ceph-mon[32631]: Updating smithi098:/etc/ceph/ceph.conf 2024-06-04T04:57:16.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:16 smithi070 ceph-mon[32631]: Updating smithi112:/etc/ceph/ceph.conf 2024-06-04T04:57:16.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:16 smithi112 ceph-mon[36453]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:16.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:16 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:16.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:16 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:16.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:16 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:16.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:16 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:16.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:16 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:16.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:16 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:16.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:16 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi070 to 176.1M 2024-06-04T04:57:16.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:16 smithi112 ceph-mon[36453]: Unable to set osd_memory_target on smithi070 to 184717209: error parsing value: Value '184717209' is below minimum 939524096 2024-06-04T04:57:16.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:16 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:57:16.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:16 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:57:16.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:16 smithi112 ceph-mon[36453]: Updating smithi070:/etc/ceph/ceph.conf 2024-06-04T04:57:16.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:16 smithi112 ceph-mon[36453]: Updating smithi098:/etc/ceph/ceph.conf 2024-06-04T04:57:16.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:16 smithi112 ceph-mon[36453]: Updating smithi112:/etc/ceph/ceph.conf 2024-06-04T04:57:16.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:16 smithi098 ceph-mon[35569]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:16.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:16 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:16.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:16 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:16.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:16 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:16.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:16 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:16.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:16 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:16.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:16 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-04T04:57:16.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:16 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi070 to 176.1M 2024-06-04T04:57:16.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:16 smithi098 ceph-mon[35569]: Unable to set osd_memory_target on smithi070 to 184717209: error parsing value: Value '184717209' is below minimum 939524096 2024-06-04T04:57:16.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:16 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:57:16.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:16 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:57:16.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:16 smithi098 ceph-mon[35569]: Updating smithi070:/etc/ceph/ceph.conf 2024-06-04T04:57:16.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:16 smithi098 ceph-mon[35569]: Updating smithi098:/etc/ceph/ceph.conf 2024-06-04T04:57:16.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:16 smithi098 ceph-mon[35569]: Updating smithi112:/etc/ceph/ceph.conf 2024-06-04T04:57:17.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.010+0000 7fd8850e3700 1 -- 172.21.15.70:0/932342820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8801088f0 msgr2=0x7fd880110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:17.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.010+0000 7fd8850e3700 1 --2- 172.21.15.70:0/932342820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8801088f0 0x7fd880110470 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd87800b0d0 tx=0x7fd87800b3e0 comp rx=0 tx=0).stop 2024-06-04T04:57:17.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.010+0000 7fd8850e3700 1 -- 172.21.15.70:0/932342820 shutdown_connections 2024-06-04T04:57:17.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.010+0000 7fd8850e3700 1 --2- 172.21.15.70:0/932342820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8801088f0 0x7fd880110470 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:17.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.010+0000 7fd8850e3700 1 --2- 172.21.15.70:0/932342820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8801037f0 0x7fd880103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:17.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.010+0000 7fd8850e3700 1 --2- 172.21.15.70:0/932342820 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8801041a0 0x7fd8801081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:17.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.010+0000 7fd8850e3700 1 -- 172.21.15.70:0/932342820 >> 172.21.15.70:0/932342820 conn(0x7fd88007a6a0 msgr2=0x7fd88007aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:17.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.010+0000 7fd8850e3700 1 -- 172.21.15.70:0/932342820 shutdown_connections 2024-06-04T04:57:17.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.010+0000 7fd8850e3700 1 -- 172.21.15.70:0/932342820 wait complete. 2024-06-04T04:57:17.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.011+0000 7fd8850e3700 1 Processor -- start 2024-06-04T04:57:17.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.011+0000 7fd8850e3700 1 -- start start 2024-06-04T04:57:17.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.011+0000 7fd8850e3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8801037f0 0x7fd880075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:17.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.011+0000 7fd8850e3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8801041a0 0x7fd880075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:17.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.011+0000 7fd8850e3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8801088f0 0x7fd8800762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:17.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.011+0000 7fd8850e3700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd880113570 con 0x7fd8801088f0 2024-06-04T04:57:17.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.011+0000 7fd8850e3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8801133f0 con 0x7fd8801037f0 2024-06-04T04:57:17.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.011+0000 7fd8850e3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd8801136f0 con 0x7fd8801041a0 2024-06-04T04:57:17.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.012+0000 7fd87f59e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8801088f0 0x7fd8800762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:17.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.012+0000 7fd87f59e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8801088f0 0x7fd8800762c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45282/0 (socket says 172.21.15.70:45282) 2024-06-04T04:57:17.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.012+0000 7fd87e59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8801041a0 0x7fd880075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:17.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.012+0000 7fd87f59e700 1 -- 172.21.15.70:0/3449926626 learned_addr learned my addr 172.21.15.70:0/3449926626 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:17.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.012+0000 7fd87ed9d700 1 --2- 172.21.15.70:0/3449926626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8801037f0 0x7fd880075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:17.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.012+0000 7fd87f59e700 1 -- 172.21.15.70:0/3449926626 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8801041a0 msgr2=0x7fd880075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:17.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.012+0000 7fd87f59e700 1 --2- 172.21.15.70:0/3449926626 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8801041a0 0x7fd880075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:17.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.012+0000 7fd87f59e700 1 -- 172.21.15.70:0/3449926626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8801037f0 msgr2=0x7fd880075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:17.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.012+0000 7fd87f59e700 1 --2- 172.21.15.70:0/3449926626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8801037f0 0x7fd880075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:17.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.012+0000 7fd87f59e700 1 -- 172.21.15.70:0/3449926626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8801a7f70 con 0x7fd8801088f0 2024-06-04T04:57:17.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.012+0000 7fd87ed9d700 1 --2- 172.21.15.70:0/3449926626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8801037f0 0x7fd880075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:17.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.012+0000 7fd87f59e700 1 --2- 172.21.15.70:0/3449926626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8801088f0 0x7fd8800762c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd878009850 tx=0x7fd878007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:17.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.012+0000 7fd877fff700 1 -- 172.21.15.70:0/3449926626 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87801f070 con 0x7fd8801088f0 2024-06-04T04:57:17.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.012+0000 7fd877fff700 1 -- 172.21.15.70:0/3449926626 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd878004c50 con 0x7fd8801088f0 2024-06-04T04:57:17.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.012+0000 7fd8850e3700 1 -- 172.21.15.70:0/3449926626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8801a8200 con 0x7fd8801088f0 2024-06-04T04:57:17.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.012+0000 7fd877fff700 1 -- 172.21.15.70:0/3449926626 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87801b690 con 0x7fd8801088f0 2024-06-04T04:57:17.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.012+0000 7fd8850e3700 1 -- 172.21.15.70:0/3449926626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8801a8600 con 0x7fd8801088f0 2024-06-04T04:57:17.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.013+0000 7fd8850e3700 1 -- 172.21.15.70:0/3449926626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd88006a230 con 0x7fd8801088f0 2024-06-04T04:57:17.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.016+0000 7fd877fff700 1 -- 172.21.15.70:0/3449926626 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7fd878022070 con 0x7fd8801088f0 2024-06-04T04:57:17.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.016+0000 7fd877fff700 1 --2- 172.21.15.70:0/3449926626 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd86c0b2a80 0x7fd86c0b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:17.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.016+0000 7fd87ed9d700 1 --2- 172.21.15.70:0/3449926626 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd86c0b2a80 0x7fd86c0b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:17.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.016+0000 7fd877fff700 1 -- 172.21.15.70:0/3449926626 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fd8780e3790 con 0x7fd8801088f0 2024-06-04T04:57:17.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.017+0000 7fd87ed9d700 1 --2- 172.21.15.70:0/3449926626 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd86c0b2a80 0x7fd86c0b4f40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd870009cb0 tx=0x7fd870009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:17.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.020+0000 7fd877fff700 1 -- 172.21.15.70:0/3449926626 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd8780ab090 con 0x7fd8801088f0 2024-06-04T04:57:17.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.266+0000 7fd8850e3700 1 -- 172.21.15.70:0/3449926626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd88002e720 con 0x7fd8801088f0 2024-06-04T04:57:17.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.267+0000 7fd877fff700 1 -- 172.21.15.70:0/3449926626 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7fd878009cb0 con 0x7fd8801088f0 2024-06-04T04:57:17.269 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:57:17.270 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:57:09.013178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T04:57:17.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.269+0000 7fd8850e3700 1 -- 172.21.15.70:0/3449926626 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd86c0b2a80 msgr2=0x7fd86c0b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:17.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.269+0000 7fd8850e3700 1 --2- 172.21.15.70:0/3449926626 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd86c0b2a80 0x7fd86c0b4f40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd870009cb0 tx=0x7fd870009260 comp rx=0 tx=0).stop 2024-06-04T04:57:17.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.270+0000 7fd8850e3700 1 -- 172.21.15.70:0/3449926626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8801088f0 msgr2=0x7fd8800762c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:17.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.270+0000 7fd8850e3700 1 --2- 172.21.15.70:0/3449926626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8801088f0 0x7fd8800762c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd878009850 tx=0x7fd878007a20 comp rx=0 tx=0).stop 2024-06-04T04:57:17.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.270+0000 7fd8850e3700 1 -- 172.21.15.70:0/3449926626 shutdown_connections 2024-06-04T04:57:17.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.270+0000 7fd8850e3700 1 --2- 172.21.15.70:0/3449926626 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd86c0b2a80 0x7fd86c0b4f40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:17.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.270+0000 7fd8850e3700 1 --2- 172.21.15.70:0/3449926626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8801088f0 0x7fd8800762c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:17.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.270+0000 7fd8850e3700 1 --2- 172.21.15.70:0/3449926626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8801037f0 0x7fd880075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:17.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.270+0000 7fd8850e3700 1 --2- 172.21.15.70:0/3449926626 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8801041a0 0x7fd880075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:17.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.270+0000 7fd8850e3700 1 -- 172.21.15.70:0/3449926626 >> 172.21.15.70:0/3449926626 conn(0x7fd88007a6a0 msgr2=0x7fd880101920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:17.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.270+0000 7fd8850e3700 1 -- 172.21.15.70:0/3449926626 shutdown_connections 2024-06-04T04:57:17.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:17.270+0000 7fd8850e3700 1 -- 172.21.15.70:0/3449926626 wait complete. 2024-06-04T04:57:17.273 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-06-04T04:57:17.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: Updating smithi070:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:57:17.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: Updating smithi098:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:57:17.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: Updating smithi112:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:57:17.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-06-04T04:57:17.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:57:17.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:57:17.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:57:17.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: Updating smithi070:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-04T04:57:17.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: Updating smithi112:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-04T04:57:17.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: Updating smithi098:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-04T04:57:17.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:57:17.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:57:17.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:57:17.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:17 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3449926626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:57:17.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: Updating smithi070:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:57:17.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: Updating smithi098:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:57:17.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: Updating smithi112:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:57:17.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-06-04T04:57:17.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:57:17.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:57:17.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:57:17.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: Updating smithi070:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-04T04:57:17.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: Updating smithi112:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-04T04:57:17.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: Updating smithi098:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-04T04:57:17.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:57:17.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:57:17.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:57:17.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:17 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3449926626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:57:17.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: Updating smithi070:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:57:17.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: Updating smithi098:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:57:17.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: Updating smithi112:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-04T04:57:17.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-06-04T04:57:17.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:57:17.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:57:17.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-06-04T04:57:17.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: Updating smithi070:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-04T04:57:17.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: Updating smithi112:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-04T04:57:17.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: Updating smithi098:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-04T04:57:17.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:57:17.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:57:17.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:57:17.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:17.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:17 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3449926626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:57:18.019 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-04T04:57:18.259 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:19.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:19 smithi070 ceph-mon[32631]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-04T04:57:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.801+0000 7f3febfff700 1 -- 172.21.15.70:0/4150382236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fec1025d0 msgr2=0x7f3fec1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.801+0000 7f3febfff700 1 --2- 172.21.15.70:0/4150382236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fec1025d0 0x7f3fec1029b0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc0099c0 tx=0x7f3fdc009cd0 comp rx=0 tx=0).stop 2024-06-04T04:57:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.802+0000 7f3febfff700 1 -- 172.21.15.70:0/4150382236 shutdown_connections 2024-06-04T04:57:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.802+0000 7f3febfff700 1 --2- 172.21.15.70:0/4150382236 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fec1076b0 0x7f3fec10f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.802+0000 7f3febfff700 1 --2- 172.21.15.70:0/4150382236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fec1025d0 0x7f3fec1029b0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.802+0000 7f3febfff700 1 --2- 172.21.15.70:0/4150382236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fec102f80 0x7f3fec106fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.802+0000 7f3febfff700 1 -- 172.21.15.70:0/4150382236 >> 172.21.15.70:0/4150382236 conn(0x7f3fec0fde60 msgr2=0x7f3fec100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:19.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.802+0000 7f3febfff700 1 -- 172.21.15.70:0/4150382236 shutdown_connections 2024-06-04T04:57:19.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.802+0000 7f3febfff700 1 -- 172.21.15.70:0/4150382236 wait complete. 2024-06-04T04:57:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.802+0000 7f3febfff700 1 Processor -- start 2024-06-04T04:57:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3febfff700 1 -- start start 2024-06-04T04:57:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3febfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fec1025d0 0x7f3fec19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3febfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fec102f80 0x7f3fec19fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3febfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fec1076b0 0x7f3fec199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3febfff700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec1124c0 con 0x7f3fec1076b0 2024-06-04T04:57:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3febfff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec112340 con 0x7f3fec102f80 2024-06-04T04:57:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3febfff700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec112640 con 0x7f3fec1025d0 2024-06-04T04:57:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3feb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fec1076b0 0x7f3fec199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3feb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fec1076b0 0x7f3fec199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45302/0 (socket says 172.21.15.70:45302) 2024-06-04T04:57:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3feb7fe700 1 -- 172.21.15.70:0/711338077 learned_addr learned my addr 172.21.15.70:0/711338077 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3fea7fc700 1 --2- 172.21.15.70:0/711338077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fec102f80 0x7f3fec19fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3feaffd700 1 --2- 172.21.15.70:0/711338077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fec1025d0 0x7f3fec19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3feb7fe700 1 -- 172.21.15.70:0/711338077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fec1025d0 msgr2=0x7f3fec19f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3feb7fe700 1 --2- 172.21.15.70:0/711338077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fec1025d0 0x7f3fec19f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3feb7fe700 1 -- 172.21.15.70:0/711338077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fec102f80 msgr2=0x7f3fec19fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3feb7fe700 1 --2- 172.21.15.70:0/711338077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fec102f80 0x7f3fec19fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3feb7fe700 1 -- 172.21.15.70:0/711338077 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fec19a0d0 con 0x7f3fec1076b0 2024-06-04T04:57:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.804+0000 7f3feaffd700 1 --2- 172.21.15.70:0/711338077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fec1025d0 0x7f3fec19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.803+0000 7f3fea7fc700 1 --2- 172.21.15.70:0/711338077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fec102f80 0x7f3fec19fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.804+0000 7f3feb7fe700 1 --2- 172.21.15.70:0/711338077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fec1076b0 0x7f3fec199970 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f3fe400b970 tx=0x7f3fe400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.804+0000 7f3ff09ff700 1 -- 172.21.15.70:0/711338077 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4019070 con 0x7f3fec1076b0 2024-06-04T04:57:19.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.804+0000 7f3febfff700 1 -- 172.21.15.70:0/711338077 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fec19a3c0 con 0x7f3fec1076b0 2024-06-04T04:57:19.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.804+0000 7f3ff09ff700 1 -- 172.21.15.70:0/711338077 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3fe400ce40 con 0x7f3fec1076b0 2024-06-04T04:57:19.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.804+0000 7f3ff09ff700 1 -- 172.21.15.70:0/711338077 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4007370 con 0x7f3fec1076b0 2024-06-04T04:57:19.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.805+0000 7f3febfff700 1 -- 172.21.15.70:0/711338077 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fec103fa0 con 0x7f3fec1076b0 2024-06-04T04:57:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.805+0000 7f3febfff700 1 -- 172.21.15.70:0/711338077 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fec06a230 con 0x7f3fec1076b0 2024-06-04T04:57:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.806+0000 7f3ff09ff700 1 -- 172.21.15.70:0/711338077 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7f3fe401c070 con 0x7f3fec1076b0 2024-06-04T04:57:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.807+0000 7f3ff09ff700 1 --2- 172.21.15.70:0/711338077 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3fd40b2b00 0x7f3fd40b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.807+0000 7f3feaffd700 1 --2- 172.21.15.70:0/711338077 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3fd40b2b00 0x7f3fd40b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.807+0000 7f3ff09ff700 1 -- 172.21.15.70:0/711338077 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f3fe40e1ca0 con 0x7f3fec1076b0 2024-06-04T04:57:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.807+0000 7f3feaffd700 1 --2- 172.21.15.70:0/711338077 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3fd40b2b00 0x7f3fd40b4fc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc009fd0 tx=0x7f3fdc005e80 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:19.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:19.812+0000 7f3ff09ff700 1 -- 172.21.15.70:0/711338077 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3fe4051df0 con 0x7f3fec1076b0 2024-06-04T04:57:19.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:19 smithi112 ceph-mon[36453]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-04T04:57:19.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:19 smithi098 ceph-mon[35569]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-04T04:57:20.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.018+0000 7f3febfff700 1 -- 172.21.15.70:0/711338077 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3fec02e560 con 0x7f3fec1076b0 2024-06-04T04:57:20.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.019+0000 7f3ff09ff700 1 -- 172.21.15.70:0/711338077 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+23085 (secure 0 0 0) 0x7f3fe40a9620 con 0x7f3fec1076b0 2024-06-04T04:57:20.022 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:57:20.022 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":82,"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","created":"2024-06-04T04:45:22.297536+0000","modified":"2024-06-04T04:57:09.937740+0000","last_up_change":"2024-06-04T04:52:34.388708+0000","last_in_change":"2024-06-04T04:52:14.197780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-04T04:48:43.724356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-04T04:53:31.398455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-04T04:53:35.298617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ff491ea2-1906-48e4-883e-bc78e778260e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6803","nonce":824119100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6805","nonce":824119100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6809","nonce":824119100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6807","nonce":824119100}]},"public_addr":"172.21.15.70:6803/824119100","cluster_addr":"172.21.15.70:6805/824119100","heartbeat_back_addr":"172.21.15.70:6809/824119100","heartbeat_front_addr":"172.21.15.70:6807/824119100","state":["exists","up"]},{"osd":1,"uuid":"bd5e4806-535b-49b5-ba6b-93228da35a84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6801","nonce":2744598742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6803","nonce":2744598742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6807","nonce":2744598742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6805","nonce":2744598742}]},"public_addr":"172.21.15.98:6801/2744598742","cluster_addr":"172.21.15.98:6803/2744598742","heartbeat_back_addr":"172.21.15.98:6807/2744598742","heartbeat_front_addr":"172.21.15.98:6805/2744598742","state":["exists","up"]},{"osd":2,"uuid":"8456fb55-910a-4a59-b84b-2c0f68346ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6801","nonce":1328098370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6803","nonce":1328098370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6807","nonce":1328098370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6805","nonce":1328098370}]},"public_addr":"172.21.15.112:6801/1328098370","cluster_addr":"172.21.15.112:6803/1328098370","heartbeat_back_addr":"172.21.15.112:6807/1328098370","heartbeat_front_addr":"172.21.15.112:6805/1328098370","state":["exists","up"]},{"osd":3,"uuid":"c8fbc440-bf70-42d3-b01e-777ccfa7900b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6811","nonce":4041566346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6813","nonce":4041566346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6817","nonce":4041566346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6815","nonce":4041566346}]},"public_addr":"172.21.15.70:6811/4041566346","cluster_addr":"172.21.15.70:6813/4041566346","heartbeat_back_addr":"172.21.15.70:6817/4041566346","heartbeat_front_addr":"172.21.15.70:6815/4041566346","state":["exists","up"]},{"osd":4,"uuid":"549c8c70-8be8-46ae-bcb6-0fde32b40cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6809","nonce":3652775814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6811","nonce":3652775814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6815","nonce":3652775814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6813","nonce":3652775814}]},"public_addr":"172.21.15.98:6809/3652775814","cluster_addr":"172.21.15.98:6811/3652775814","heartbeat_back_addr":"172.21.15.98:6815/3652775814","heartbeat_front_addr":"172.21.15.98:6813/3652775814","state":["exists","up"]},{"osd":5,"uuid":"51d86af1-0d6c-4ae0-b090-00414fc1bf7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6809","nonce":1857257899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6811","nonce":1857257899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6815","nonce":1857257899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6813","nonce":1857257899}]},"public_addr":"172.21.15.112:6809/1857257899","cluster_addr":"172.21.15.112:6811/1857257899","heartbeat_back_addr":"172.21.15.112:6815/1857257899","heartbeat_front_addr":"172.21.15.112:6813/1857257899","state":["exists","up"]},{"osd":6,"uuid":"526a8138-bfa8-419e-a5b5-e6b80f0ae004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6819","nonce":1264676496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6821","nonce":1264676496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6825","nonce":1264676496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6823","nonce":1264676496}]},"public_addr":"172.21.15.70:6819/1264676496","cluster_addr":"172.21.15.70:6821/1264676496","heartbeat_back_addr":"172.21.15.70:6825/1264676496","heartbeat_front_addr":"172.21.15.70:6823/1264676496","state":["exists","up"]},{"osd":7,"uuid":"1173f785-6ed1-47ef-9265-2e233ea9ee02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6817","nonce":709593020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6819","nonce":709593020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6823","nonce":709593020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6821","nonce":709593020}]},"public_addr":"172.21.15.98:6817/709593020","cluster_addr":"172.21.15.98:6819/709593020","heartbeat_back_addr":"172.21.15.98:6823/709593020","heartbeat_front_addr":"172.21.15.98:6821/709593020","state":["exists","up"]},{"osd":8,"uuid":"6424cc6e-24a7-4b81-9ce6-39203fecc7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6817","nonce":2250225413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6819","nonce":2250225413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6823","nonce":2250225413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6821","nonce":2250225413}]},"public_addr":"172.21.15.112:6817/2250225413","cluster_addr":"172.21.15.112:6819/2250225413","heartbeat_back_addr":"172.21.15.112:6823/2250225413","heartbeat_front_addr":"172.21.15.112:6821/2250225413","state":["exists","up"]},{"osd":9,"uuid":"8d666db8-8831-4d63-9e3c-a4bf2504149e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6827","nonce":3557688707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6829","nonce":3557688707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6833","nonce":3557688707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6831","nonce":3557688707}]},"public_addr":"172.21.15.70:6827/3557688707","cluster_addr":"172.21.15.70:6829/3557688707","heartbeat_back_addr":"172.21.15.70:6833/3557688707","heartbeat_front_addr":"172.21.15.70:6831/3557688707","state":["exists","up"]},{"osd":10,"uuid":"ecb925d2-0263-4dfc-a2fd-9e27856022d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6825","nonce":1896493718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6827","nonce":1896493718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6831","nonce":1896493718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6829","nonce":1896493718}]},"public_addr":"172.21.15.98:6825/1896493718","cluster_addr":"172.21.15.98:6827/1896493718","heartbeat_back_addr":"172.21.15.98:6831/1896493718","heartbeat_front_addr":"172.21.15.98:6829/1896493718","state":["exists","up"]},{"osd":11,"uuid":"c8e3230c-bc3a-40bd-8dd0-5fef56e4df97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6825","nonce":1400793137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6827","nonce":1400793137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6831","nonce":1400793137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6829","nonce":1400793137}]},"public_addr":"172.21.15.112:6825/1400793137","cluster_addr":"172.21.15.112:6827/1400793137","heartbeat_back_addr":"172.21.15.112:6831/1400793137","heartbeat_front_addr":"172.21.15.112:6829/1400793137","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:47:53.626608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:17.784505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:41.971969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:07.130699+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:31.615496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:56.898607+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:22.739991+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:48.319727+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:13.902063+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:40.389470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:06.142092+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:32.949383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/4291129442":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/3584175673":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/3720789940":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/145533746":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/3442074771":"2024-06-05T04:46:17.636907+0000","172.21.15.70:6801/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/1442095493":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/2453733268":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6801/2752966840":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/1369781447":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6801/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/2752966840":"2024-06-05T04:57:09.937704+0000","172.21.15.70:6800/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/915186456":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/860033006":"2024-06-05T04:45:49.501864+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-04T04:57:20.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.022+0000 7f3febfff700 1 -- 172.21.15.70:0/711338077 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3fd40b2b00 msgr2=0x7f3fd40b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:20.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.022+0000 7f3febfff700 1 --2- 172.21.15.70:0/711338077 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3fd40b2b00 0x7f3fd40b4fc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc009fd0 tx=0x7f3fdc005e80 comp rx=0 tx=0).stop 2024-06-04T04:57:20.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.022+0000 7f3febfff700 1 -- 172.21.15.70:0/711338077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fec1076b0 msgr2=0x7f3fec199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:20.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.022+0000 7f3febfff700 1 --2- 172.21.15.70:0/711338077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fec1076b0 0x7f3fec199970 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f3fe400b970 tx=0x7f3fe400bd30 comp rx=0 tx=0).stop 2024-06-04T04:57:20.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.022+0000 7f3febfff700 1 -- 172.21.15.70:0/711338077 shutdown_connections 2024-06-04T04:57:20.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.022+0000 7f3febfff700 1 --2- 172.21.15.70:0/711338077 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3fd40b2b00 0x7f3fd40b4fc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:20.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.022+0000 7f3febfff700 1 --2- 172.21.15.70:0/711338077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fec1076b0 0x7f3fec199970 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:20.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.022+0000 7f3febfff700 1 --2- 172.21.15.70:0/711338077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fec1025d0 0x7f3fec19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:20.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.022+0000 7f3febfff700 1 --2- 172.21.15.70:0/711338077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fec102f80 0x7f3fec19fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:20.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.022+0000 7f3febfff700 1 -- 172.21.15.70:0/711338077 >> 172.21.15.70:0/711338077 conn(0x7f3fec0fde60 msgr2=0x7f3fec06b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:20.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.022+0000 7f3febfff700 1 -- 172.21.15.70:0/711338077 shutdown_connections 2024-06-04T04:57:20.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.022+0000 7f3febfff700 1 -- 172.21.15.70:0/711338077 wait complete. 2024-06-04T04:57:20.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:20 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/711338077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:57:20.836 DEBUG:tasks.fwd_scrub:Yielding 2024-06-04T04:57:20.836 INFO:teuthology.run_tasks:Running task exec... 2024-06-04T04:57:20.846 INFO:teuthology.task.exec:Executing custom commands... 2024-06-04T04:57:20.846 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi070.front.sepia.ceph.com 2024-06-04T04:57:20.846 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/log_level debug' 2024-06-04T04:57:20.847 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-06-04T04:57:20.847 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T04:57:20.847 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T04:57:20.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:20 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/711338077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:57:20.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.952+0000 7f6516d0a640 1 -- 172.21.15.70:0/3870616981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f651010d060 msgr2=0x7f651010f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.952+0000 7f6516d0a640 1 --2- 172.21.15.70:0/3870616981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f651010d060 0x7f651010f4b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6508009ca0 tx=0x7f650802a950 comp rx=0 tx=0).stop 2024-06-04T04:57:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.952+0000 7f6516d0a640 1 -- 172.21.15.70:0/3870616981 shutdown_connections 2024-06-04T04:57:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.952+0000 7f6516d0a640 1 --2- 172.21.15.70:0/3870616981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6510104640 0x7f651010cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.952+0000 7f6516d0a640 1 --2- 172.21.15.70:0/3870616981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f651010d060 0x7f651010f4b0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.952+0000 7f6516d0a640 1 --2- 172.21.15.70:0/3870616981 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6510103d20 0x7f6510104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.952+0000 7f6516d0a640 1 -- 172.21.15.70:0/3870616981 >> 172.21.15.70:0/3870616981 conn(0x7f6510078ad0 msgr2=0x7f6510078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.952+0000 7f6516d0a640 1 -- 172.21.15.70:0/3870616981 shutdown_connections 2024-06-04T04:57:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.952+0000 7f6516d0a640 1 -- 172.21.15.70:0/3870616981 wait complete. 2024-06-04T04:57:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f6516d0a640 1 Processor -- start 2024-06-04T04:57:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f6516d0a640 1 -- start start 2024-06-04T04:57:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f6516d0a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6510103d20 0x7f6510101970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f6516d0a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6510104640 0x7f6510101eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f6516d0a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f651010d060 0x7f65101023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f6516d0a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6510112bc0 con 0x7f6510104640 2024-06-04T04:57:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f6516d0a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6510112a40 con 0x7f6510103d20 2024-06-04T04:57:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f6516d0a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6510112d40 con 0x7f651010d060 2024-06-04T04:57:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f650ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6510104640 0x7f6510101eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f650ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6510104640 0x7f6510101eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45308/0 (socket says 172.21.15.70:45308) 2024-06-04T04:57:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f650ffff640 1 -- 172.21.15.70:0/402784336 learned_addr learned my addr 172.21.15.70:0/402784336 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f6514aa6640 1 --2- 172.21.15.70:0/402784336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6510103d20 0x7f6510101970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f65152a7640 1 --2- 172.21.15.70:0/402784336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f651010d060 0x7f65101023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f650ffff640 1 -- 172.21.15.70:0/402784336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f651010d060 msgr2=0x7f65101023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f650ffff640 1 --2- 172.21.15.70:0/402784336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f651010d060 0x7f65101023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f650ffff640 1 -- 172.21.15.70:0/402784336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6510103d20 msgr2=0x7f6510101970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f650ffff640 1 --2- 172.21.15.70:0/402784336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6510103d20 0x7f6510101970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:20.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f650ffff640 1 -- 172.21.15.70:0/402784336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65101a7580 con 0x7f6510104640 2024-06-04T04:57:20.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.953+0000 7f650ffff640 1 --2- 172.21.15.70:0/402784336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6510104640 0x7f6510101eb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00d950 tx=0x7f64fc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:20.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.954+0000 7f650dffb640 1 -- 172.21.15.70:0/402784336 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc017070 con 0x7f6510104640 2024-06-04T04:57:20.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.954+0000 7f650dffb640 1 -- 172.21.15.70:0/402784336 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f64fc009c30 con 0x7f6510104640 2024-06-04T04:57:20.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.954+0000 7f6516d0a640 1 -- 172.21.15.70:0/402784336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65101a7870 con 0x7f6510104640 2024-06-04T04:57:20.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.954+0000 7f650dffb640 1 -- 172.21.15.70:0/402784336 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc00f620 con 0x7f6510104640 2024-06-04T04:57:20.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.954+0000 7f6516d0a640 1 -- 172.21.15.70:0/402784336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65101a7e10 con 0x7f6510104640 2024-06-04T04:57:20.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.955+0000 7f6516d0a640 1 -- 172.21.15.70:0/402784336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65100507e0 con 0x7f6510104640 2024-06-04T04:57:20.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.956+0000 7f650dffb640 1 -- 172.21.15.70:0/402784336 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7f64fc010460 con 0x7f6510104640 2024-06-04T04:57:20.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.958+0000 7f650dffb640 1 --2- 172.21.15.70:0/402784336 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65000b2b10 0x7f65000b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:20.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.958+0000 7f6514aa6640 1 --2- 172.21.15.70:0/402784336 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65000b2b10 0x7f65000b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:20.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.958+0000 7f650dffb640 1 -- 172.21.15.70:0/402784336 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f64fc0e2250 con 0x7f6510104640 2024-06-04T04:57:20.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.958+0000 7f6514aa6640 1 --2- 172.21.15.70:0/402784336 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65000b2b10 0x7f65000b4fd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f650400bbe0 tx=0x7f650400b390 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:20.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:20.963+0000 7f650dffb640 1 -- 172.21.15.70:0/402784336 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f64fc009de0 con 0x7f6510104640 2024-06-04T04:57:20.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:20 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/711338077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:57:21.120 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:21.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.192+0000 7f6516d0a640 1 -- 172.21.15.70:0/402784336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7f65101081e0 con 0x7f6510104640 2024-06-04T04:57:21.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.196+0000 7f650dffb640 1 -- 172.21.15.70:0/402784336 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v27)=0 v27) ==== 129+0+0 (secure 0 0 0) 0x7f64fc0a9470 con 0x7f6510104640 2024-06-04T04:57:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.199+0000 7f6516d0a640 1 -- 172.21.15.70:0/402784336 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65000b2b10 msgr2=0x7f65000b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.199+0000 7f6516d0a640 1 --2- 172.21.15.70:0/402784336 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65000b2b10 0x7f65000b4fd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f650400bbe0 tx=0x7f650400b390 comp rx=0 tx=0).stop 2024-06-04T04:57:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.199+0000 7f6516d0a640 1 -- 172.21.15.70:0/402784336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6510104640 msgr2=0x7f6510101eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.199+0000 7f6516d0a640 1 --2- 172.21.15.70:0/402784336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6510104640 0x7f6510101eb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00d950 tx=0x7f64fc00de10 comp rx=0 tx=0).stop 2024-06-04T04:57:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.199+0000 7f6516d0a640 1 -- 172.21.15.70:0/402784336 shutdown_connections 2024-06-04T04:57:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.199+0000 7f6516d0a640 1 --2- 172.21.15.70:0/402784336 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65000b2b10 0x7f65000b4fd0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.199+0000 7f6516d0a640 1 --2- 172.21.15.70:0/402784336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f651010d060 0x7f65101023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.199+0000 7f6516d0a640 1 --2- 172.21.15.70:0/402784336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6510104640 0x7f6510101eb0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.199+0000 7f6516d0a640 1 --2- 172.21.15.70:0/402784336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6510103d20 0x7f6510101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.199+0000 7f6516d0a640 1 -- 172.21.15.70:0/402784336 >> 172.21.15.70:0/402784336 conn(0x7f6510078ad0 msgr2=0x7f65100fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.199+0000 7f6516d0a640 1 -- 172.21.15.70:0/402784336 shutdown_connections 2024-06-04T04:57:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.199+0000 7f6516d0a640 1 -- 172.21.15.70:0/402784336 wait complete. 2024-06-04T04:57:21.213 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-06-04T04:57:21.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.331+0000 7f89eba62640 1 -- 172.21.15.70:0/107229844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89e41134d0 msgr2=0x7f89e41138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:21.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.331+0000 7f89eba62640 1 --2- 172.21.15.70:0/107229844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89e41134d0 0x7f89e41138b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f89e0009930 tx=0x7f89e002a420 comp rx=0 tx=0).stop 2024-06-04T04:57:21.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.331+0000 7f89eba62640 1 -- 172.21.15.70:0/107229844 shutdown_connections 2024-06-04T04:57:21.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.331+0000 7f89eba62640 1 --2- 172.21.15.70:0/107229844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e410ddf0 0x7f89e411a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.331+0000 7f89eba62640 1 --2- 172.21.15.70:0/107229844 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f89e410d430 0x7f89e410d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.331+0000 7f89eba62640 1 --2- 172.21.15.70:0/107229844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89e41134d0 0x7f89e41138b0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.331+0000 7f89eba62640 1 -- 172.21.15.70:0/107229844 >> 172.21.15.70:0/107229844 conn(0x7f89e406dcd0 msgr2=0x7f89e410ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:21.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.331+0000 7f89eba62640 1 -- 172.21.15.70:0/107229844 shutdown_connections 2024-06-04T04:57:21.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.331+0000 7f89eba62640 1 -- 172.21.15.70:0/107229844 wait complete. 2024-06-04T04:57:21.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.332+0000 7f89eba62640 1 Processor -- start 2024-06-04T04:57:21.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.332+0000 7f89eba62640 1 -- start start 2024-06-04T04:57:21.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.332+0000 7f89eba62640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89e410d430 0x7f89e41a97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:21.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.332+0000 7f89eba62640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f89e410ddf0 0x7f89e41a9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:21.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.332+0000 7f89eba62640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e41134d0 0x7f89e41a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:21.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.332+0000 7f89eba62640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f89e411d5b0 con 0x7f89e410d430 2024-06-04T04:57:21.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.332+0000 7f89eba62640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89e411d430 con 0x7f89e41134d0 2024-06-04T04:57:21.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.332+0000 7f89eba62640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f89e411d730 con 0x7f89e410ddf0 2024-06-04T04:57:21.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.332+0000 7f89e97fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89e410d430 0x7f89e41a97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:21.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.332+0000 7f89e97fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89e410d430 0x7f89e41a97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45314/0 (socket says 172.21.15.70:45314) 2024-06-04T04:57:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.332+0000 7f89e97fe640 1 -- 172.21.15.70:0/51744166 learned_addr learned my addr 172.21.15.70:0/51744166 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.332+0000 7f89e9fff640 1 --2- 172.21.15.70:0/51744166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e41134d0 0x7f89e41a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.332+0000 7f89e97fe640 1 -- 172.21.15.70:0/51744166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f89e410ddf0 msgr2=0x7f89e41a9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.332+0000 7f89e97fe640 1 --2- 172.21.15.70:0/51744166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f89e410ddf0 0x7f89e41a9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.332+0000 7f89e97fe640 1 -- 172.21.15.70:0/51744166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e41134d0 msgr2=0x7f89e41a3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.332+0000 7f89e97fe640 1 --2- 172.21.15.70:0/51744166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e41134d0 0x7f89e41a3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.332+0000 7f89e97fe640 1 -- 172.21.15.70:0/51744166 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e41a40d0 con 0x7f89e410d430 2024-06-04T04:57:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.333+0000 7f89e97fe640 1 --2- 172.21.15.70:0/51744166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89e410d430 0x7f89e41a97a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f89e0005940 tx=0x7f89e00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.333+0000 7f89da7fc640 1 -- 172.21.15.70:0/51744166 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e003f030 con 0x7f89e410d430 2024-06-04T04:57:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.333+0000 7f89da7fc640 1 -- 172.21.15.70:0/51744166 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f89e0003c90 con 0x7f89e410d430 2024-06-04T04:57:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.333+0000 7f89eba62640 1 -- 172.21.15.70:0/51744166 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89e41a4360 con 0x7f89e410d430 2024-06-04T04:57:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.333+0000 7f89da7fc640 1 -- 172.21.15.70:0/51744166 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e0035070 con 0x7f89e410d430 2024-06-04T04:57:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.333+0000 7f89eba62640 1 -- 172.21.15.70:0/51744166 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89e41b0660 con 0x7f89e410d430 2024-06-04T04:57:21.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.334+0000 7f89eba62640 1 -- 172.21.15.70:0/51744166 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89e40507e0 con 0x7f89e410d430 2024-06-04T04:57:21.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.335+0000 7f89da7fc640 1 -- 172.21.15.70:0/51744166 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7f89e0042030 con 0x7f89e410d430 2024-06-04T04:57:21.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.336+0000 7f89da7fc640 1 --2- 172.21.15.70:0/51744166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f89d00b2a90 0x7f89d00b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:21.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.336+0000 7f89e8ffd640 1 --2- 172.21.15.70:0/51744166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f89d00b2a90 0x7f89d00b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:21.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.336+0000 7f89da7fc640 1 -- 172.21.15.70:0/51744166 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f89e01030e0 con 0x7f89e410d430 2024-06-04T04:57:21.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.336+0000 7f89e8ffd640 1 --2- 172.21.15.70:0/51744166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f89d00b2a90 0x7f89d00b4f50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f89e410cd80 tx=0x7f89d4006c90 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:21.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.339+0000 7f89da7fc640 1 -- 172.21.15.70:0/51744166 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f89e00ca9f0 con 0x7f89e410d430 2024-06-04T04:57:21.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.539+0000 7f89eba62640 1 -- 172.21.15.70:0/51744166 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f89e41159a0 con 0x7f89e410d430 2024-06-04T04:57:21.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.543+0000 7f89da7fc640 1 -- 172.21.15.70:0/51744166 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v28)=0 v28) ==== 138+0+0 (secure 0 0 0) 0x7f89e00ca2f0 con 0x7f89e410d430 2024-06-04T04:57:21.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.545+0000 7f89eba62640 1 -- 172.21.15.70:0/51744166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f89d00b2a90 msgr2=0x7f89d00b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:21.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.545+0000 7f89eba62640 1 --2- 172.21.15.70:0/51744166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f89d00b2a90 0x7f89d00b4f50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f89e410cd80 tx=0x7f89d4006c90 comp rx=0 tx=0).stop 2024-06-04T04:57:21.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.546+0000 7f89eba62640 1 -- 172.21.15.70:0/51744166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89e410d430 msgr2=0x7f89e41a97a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:21.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.546+0000 7f89eba62640 1 --2- 172.21.15.70:0/51744166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89e410d430 0x7f89e41a97a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f89e0005940 tx=0x7f89e00042f0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.546+0000 7f89eba62640 1 -- 172.21.15.70:0/51744166 shutdown_connections 2024-06-04T04:57:21.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.546+0000 7f89eba62640 1 --2- 172.21.15.70:0/51744166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f89d00b2a90 0x7f89d00b4f50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.546+0000 7f89eba62640 1 --2- 172.21.15.70:0/51744166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e41134d0 0x7f89e41a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.546+0000 7f89eba62640 1 --2- 172.21.15.70:0/51744166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f89e410ddf0 0x7f89e41a9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.546+0000 7f89eba62640 1 --2- 172.21.15.70:0/51744166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89e410d430 0x7f89e41a97a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.546+0000 7f89eba62640 1 -- 172.21.15.70:0/51744166 >> 172.21.15.70:0/51744166 conn(0x7f89e406dcd0 msgr2=0x7f89e41104c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:21.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.546+0000 7f89eba62640 1 -- 172.21.15.70:0/51744166 shutdown_connections 2024-06-04T04:57:21.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.546+0000 7f89eba62640 1 -- 172.21.15.70:0/51744166 wait complete. 2024-06-04T04:57:21.560 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-06-04T04:57:21.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.662+0000 7ff9c10bd640 1 -- 172.21.15.70:0/1530506546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc1045c0 msgr2=0x7ff9bc10caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:21.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.662+0000 7ff9c10bd640 1 --2- 172.21.15.70:0/1530506546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc1045c0 0x7ff9bc10caa0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0009900 tx=0x7ff9b002a480 comp rx=0 tx=0).stop 2024-06-04T04:57:21.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.662+0000 7ff9c10bd640 1 -- 172.21.15.70:0/1530506546 shutdown_connections 2024-06-04T04:57:21.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.662+0000 7ff9c10bd640 1 --2- 172.21.15.70:0/1530506546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9bc10d070 0x7ff9bc10f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.662+0000 7ff9c10bd640 1 --2- 172.21.15.70:0/1530506546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc1045c0 0x7ff9bc10caa0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.662+0000 7ff9c10bd640 1 --2- 172.21.15.70:0/1530506546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9bc103ca0 0x7ff9bc104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.662+0000 7ff9c10bd640 1 -- 172.21.15.70:0/1530506546 >> 172.21.15.70:0/1530506546 conn(0x7ff9bc078b60 msgr2=0x7ff9bc078f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:21.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.662+0000 7ff9c10bd640 1 -- 172.21.15.70:0/1530506546 shutdown_connections 2024-06-04T04:57:21.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.662+0000 7ff9c10bd640 1 -- 172.21.15.70:0/1530506546 wait complete. 2024-06-04T04:57:21.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.663+0000 7ff9c10bd640 1 Processor -- start 2024-06-04T04:57:21.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.663+0000 7ff9c10bd640 1 -- start start 2024-06-04T04:57:21.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.663+0000 7ff9c10bd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9bc103ca0 0x7ff9bc199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.663+0000 7ff9c10bd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9bc1045c0 0x7ff9bc1995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.663+0000 7ff9c10bd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc10d070 0x7ff9bc19d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9c10bd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc112ba0 con 0x7ff9bc10d070 2024-06-04T04:57:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9c10bd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc112a20 con 0x7ff9bc1045c0 2024-06-04T04:57:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9c10bd640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc112d20 con 0x7ff9bc103ca0 2024-06-04T04:57:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9bb59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc10d070 0x7ff9bc19d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9bb59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc10d070 0x7ff9bc19d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45344/0 (socket says 172.21.15.70:45344) 2024-06-04T04:57:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9bb59e640 1 -- 172.21.15.70:0/501357835 learned_addr learned my addr 172.21.15.70:0/501357835 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9ba59c640 1 --2- 172.21.15.70:0/501357835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9bc1045c0 0x7ff9bc1995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9bad9d640 1 --2- 172.21.15.70:0/501357835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9bc103ca0 0x7ff9bc199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9bb59e640 1 -- 172.21.15.70:0/501357835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9bc103ca0 msgr2=0x7ff9bc199080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9bb59e640 1 --2- 172.21.15.70:0/501357835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9bc103ca0 0x7ff9bc199080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9bb59e640 1 -- 172.21.15.70:0/501357835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9bc1045c0 msgr2=0x7ff9bc1995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9bb59e640 1 --2- 172.21.15.70:0/501357835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9bc1045c0 0x7ff9bc1995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9bb59e640 1 -- 172.21.15.70:0/501357835 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9bc19e070 con 0x7ff9bc10d070 2024-06-04T04:57:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9ba59c640 1 --2- 172.21.15.70:0/501357835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9bc1045c0 0x7ff9bc1995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:57:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9bad9d640 1 --2- 172.21.15.70:0/501357835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9bc103ca0 0x7ff9bc199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9bb59e640 1 --2- 172.21.15.70:0/501357835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc10d070 0x7ff9bc19d940 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400ea60 tx=0x7ff9b400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9a3fff640 1 -- 172.21.15.70:0/501357835 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b400ce40 con 0x7ff9bc10d070 2024-06-04T04:57:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9c10bd640 1 -- 172.21.15.70:0/501357835 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9bc19e360 con 0x7ff9bc10d070 2024-06-04T04:57:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9a3fff640 1 -- 172.21.15.70:0/501357835 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff9b4005020 con 0x7ff9bc10d070 2024-06-04T04:57:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9a3fff640 1 -- 172.21.15.70:0/501357835 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4010690 con 0x7ff9bc10d070 2024-06-04T04:57:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.664+0000 7ff9c10bd640 1 -- 172.21.15.70:0/501357835 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9bc1a5bf0 con 0x7ff9bc10d070 2024-06-04T04:57:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.665+0000 7ff9c10bd640 1 -- 172.21.15.70:0/501357835 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9bc0507e0 con 0x7ff9bc10d070 2024-06-04T04:57:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.667+0000 7ff9a3fff640 1 -- 172.21.15.70:0/501357835 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7ff9b4004a40 con 0x7ff9bc10d070 2024-06-04T04:57:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.667+0000 7ff9a3fff640 1 --2- 172.21.15.70:0/501357835 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9a40b2a90 0x7ff9a40b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.667+0000 7ff9a3fff640 1 -- 172.21.15.70:0/501357835 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ff9b40e2ee0 con 0x7ff9bc10d070 2024-06-04T04:57:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.667+0000 7ff9bad9d640 1 --2- 172.21.15.70:0/501357835 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9a40b2a90 0x7ff9a40b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.668+0000 7ff9bad9d640 1 --2- 172.21.15.70:0/501357835 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9a40b2a90 0x7ff9a40b4f50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0058c0 tx=0x7ff9ac005850 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:21.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.671+0000 7ff9a3fff640 1 -- 172.21.15.70:0/501357835 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff9b4054df0 con 0x7ff9bc10d070 2024-06-04T04:57:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:21 smithi070 ceph-mon[32631]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-06-04T04:57:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:21 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/402784336' entity='client.admin' 2024-06-04T04:57:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:21 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:57:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:21 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:57:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:21 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:57:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:21 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:21.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.859+0000 7ff9c10bd640 1 -- 172.21.15.70:0/501357835 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7ff9bc19a3d0 con 0x7ff9bc10d070 2024-06-04T04:57:21.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.863+0000 7ff9a3fff640 1 -- 172.21.15.70:0/501357835 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v29)=0 v29) ==== 133+0+0 (secure 0 0 0) 0x7ff9b40aa750 con 0x7ff9bc10d070 2024-06-04T04:57:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.865+0000 7ff9c10bd640 1 -- 172.21.15.70:0/501357835 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9a40b2a90 msgr2=0x7ff9a40b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.865+0000 7ff9c10bd640 1 --2- 172.21.15.70:0/501357835 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9a40b2a90 0x7ff9a40b4f50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0058c0 tx=0x7ff9ac005850 comp rx=0 tx=0).stop 2024-06-04T04:57:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.865+0000 7ff9c10bd640 1 -- 172.21.15.70:0/501357835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc10d070 msgr2=0x7ff9bc19d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.865+0000 7ff9c10bd640 1 --2- 172.21.15.70:0/501357835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc10d070 0x7ff9bc19d940 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400ea60 tx=0x7ff9b400ef20 comp rx=0 tx=0).stop 2024-06-04T04:57:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.866+0000 7ff9c10bd640 1 -- 172.21.15.70:0/501357835 shutdown_connections 2024-06-04T04:57:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.866+0000 7ff9c10bd640 1 --2- 172.21.15.70:0/501357835 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9a40b2a90 0x7ff9a40b4f50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.866+0000 7ff9c10bd640 1 --2- 172.21.15.70:0/501357835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9bc1045c0 0x7ff9bc1995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.866+0000 7ff9c10bd640 1 --2- 172.21.15.70:0/501357835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc10d070 0x7ff9bc19d940 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.866+0000 7ff9c10bd640 1 --2- 172.21.15.70:0/501357835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9bc103ca0 0x7ff9bc199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.866+0000 7ff9c10bd640 1 -- 172.21.15.70:0/501357835 >> 172.21.15.70:0/501357835 conn(0x7ff9bc078b60 msgr2=0x7ff9bc0fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.866+0000 7ff9c10bd640 1 -- 172.21.15.70:0/501357835 shutdown_connections 2024-06-04T04:57:21.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.866+0000 7ff9c10bd640 1 -- 172.21.15.70:0/501357835 wait complete. 2024-06-04T04:57:21.879 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-06-04T04:57:21.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:21 smithi112 ceph-mon[36453]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-06-04T04:57:21.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:21 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/402784336' entity='client.admin' 2024-06-04T04:57:21.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:21 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:57:21.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:21 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:57:21.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:21 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:57:21.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:21 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:21.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.983+0000 7f68c119e640 1 -- 172.21.15.70:0/697433230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc107750 msgr2=0x7f68bc107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:21.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.983+0000 7f68c119e640 1 --2- 172.21.15.70:0/697433230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc107750 0x7f68bc107b30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009930 tx=0x7f68ac02a420 comp rx=0 tx=0).stop 2024-06-04T04:57:21.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.983+0000 7f68c119e640 1 -- 172.21.15.70:0/697433230 shutdown_connections 2024-06-04T04:57:21.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.983+0000 7f68c119e640 1 --2- 172.21.15.70:0/697433230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc102100 0x7f68bc10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.983+0000 7f68c119e640 1 --2- 172.21.15.70:0/697433230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68bc101740 0x7f68bc101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.983+0000 7f68c119e640 1 --2- 172.21.15.70:0/697433230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc107750 0x7f68bc107b30 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.983+0000 7f68c119e640 1 -- 172.21.15.70:0/697433230 >> 172.21.15.70:0/697433230 conn(0x7f68bc0fd270 msgr2=0x7f68bc0ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.984+0000 7f68c119e640 1 -- 172.21.15.70:0/697433230 shutdown_connections 2024-06-04T04:57:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.984+0000 7f68c119e640 1 -- 172.21.15.70:0/697433230 wait complete. 2024-06-04T04:57:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.984+0000 7f68c119e640 1 Processor -- start 2024-06-04T04:57:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.984+0000 7f68c119e640 1 -- start start 2024-06-04T04:57:21.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68c119e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68bc101740 0x7f68bc198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:21.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68c119e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc102100 0x7f68bc199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:21.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68c119e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc107750 0x7f68bc19d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:21.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68c119e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc111880 con 0x7f68bc102100 2024-06-04T04:57:21.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68c119e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc111700 con 0x7f68bc107750 2024-06-04T04:57:21.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68c119e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc111a00 con 0x7f68bc101740 2024-06-04T04:57:21.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68bb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc102100 0x7f68bc199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:21.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68bb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc102100 0x7f68bc199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45362/0 (socket says 172.21.15.70:45362) 2024-06-04T04:57:21.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68bb7fe640 1 -- 172.21.15.70:0/1520563147 learned_addr learned my addr 172.21.15.70:0/1520563147 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:21.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68c099d640 1 --2- 172.21.15.70:0/1520563147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc107750 0x7f68bc19d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:21.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68bbfff640 1 --2- 172.21.15.70:0/1520563147 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68bc101740 0x7f68bc198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:21.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68bb7fe640 1 -- 172.21.15.70:0/1520563147 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68bc101740 msgr2=0x7f68bc198f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:21.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68bb7fe640 1 --2- 172.21.15.70:0/1520563147 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68bc101740 0x7f68bc198f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68bb7fe640 1 -- 172.21.15.70:0/1520563147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc107750 msgr2=0x7f68bc19d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68bb7fe640 1 --2- 172.21.15.70:0/1520563147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc107750 0x7f68bc19d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68bb7fe640 1 -- 172.21.15.70:0/1520563147 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68bc19df00 con 0x7f68bc102100 2024-06-04T04:57:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68bbfff640 1 --2- 172.21.15.70:0/1520563147 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68bc101740 0x7f68bc198f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68bb7fe640 1 --2- 172.21.15.70:0/1520563147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc102100 0x7f68bc199450 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f68b000d950 tx=0x7f68b000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68b97fa640 1 -- 172.21.15.70:0/1520563147 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b0017070 con 0x7f68bc102100 2024-06-04T04:57:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68b97fa640 1 -- 172.21.15.70:0/1520563147 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68b0009c30 con 0x7f68bc102100 2024-06-04T04:57:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68c119e640 1 -- 172.21.15.70:0/1520563147 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68bc19e1f0 con 0x7f68bc102100 2024-06-04T04:57:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.985+0000 7f68b97fa640 1 -- 172.21.15.70:0/1520563147 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b000f620 con 0x7f68bc102100 2024-06-04T04:57:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.986+0000 7f68c119e640 1 -- 172.21.15.70:0/1520563147 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68bc1a5a80 con 0x7f68bc102100 2024-06-04T04:57:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.986+0000 7f68c119e640 1 -- 172.21.15.70:0/1520563147 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68bc051030 con 0x7f68bc102100 2024-06-04T04:57:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.988+0000 7f68b97fa640 1 -- 172.21.15.70:0/1520563147 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7f68b0005030 con 0x7f68bc102100 2024-06-04T04:57:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.988+0000 7f68b97fa640 1 --2- 172.21.15.70:0/1520563147 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68a40b29e0 0x7f68a40b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.988+0000 7f68bbfff640 1 --2- 172.21.15.70:0/1520563147 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68a40b29e0 0x7f68a40b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.989+0000 7f68b97fa640 1 -- 172.21.15.70:0/1520563147 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f68b00e23b0 con 0x7f68bc102100 2024-06-04T04:57:21.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.989+0000 7f68bbfff640 1 --2- 172.21.15.70:0/1520563147 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68a40b29e0 0x7f68a40b4ea0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f68ac006010 tx=0x7f68ac000bc0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:21.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:21.992+0000 7f68b97fa640 1 -- 172.21.15.70:0/1520563147 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f68b00a9c20 con 0x7f68bc102100 2024-06-04T04:57:21.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:21 smithi098 ceph-mon[35569]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-06-04T04:57:21.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:21 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/402784336' entity='client.admin' 2024-06-04T04:57:21.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:21 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:57:21.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:21 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:57:21.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:21 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:57:21.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:21 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:22.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.204+0000 7f68c119e640 1 -- 172.21.15.70:0/1520563147 --> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f68bc062e90 con 0x7f68a40b29e0 2024-06-04T04:57:22.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.269+0000 7f68b97fa640 1 -- 172.21.15.70:0/1520563147 <== mgr.24806 v2:172.21.15.70:6800/542036627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f68bc062e90 con 0x7f68a40b29e0 2024-06-04T04:57:22.272 INFO:teuthology.orchestra.run.smithi070.stdout:Schedule set for path / 2024-06-04T04:57:22.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.272+0000 7f68c119e640 1 -- 172.21.15.70:0/1520563147 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68a40b29e0 msgr2=0x7f68a40b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:22.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.272+0000 7f68c119e640 1 --2- 172.21.15.70:0/1520563147 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68a40b29e0 0x7f68a40b4ea0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f68ac006010 tx=0x7f68ac000bc0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.272+0000 7f68c119e640 1 -- 172.21.15.70:0/1520563147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc102100 msgr2=0x7f68bc199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:22.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.272+0000 7f68c119e640 1 --2- 172.21.15.70:0/1520563147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc102100 0x7f68bc199450 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f68b000d950 tx=0x7f68b000de10 comp rx=0 tx=0).stop 2024-06-04T04:57:22.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.272+0000 7f68c119e640 1 -- 172.21.15.70:0/1520563147 shutdown_connections 2024-06-04T04:57:22.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.272+0000 7f68c119e640 1 --2- 172.21.15.70:0/1520563147 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68a40b29e0 0x7f68a40b4ea0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.272+0000 7f68c119e640 1 --2- 172.21.15.70:0/1520563147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc107750 0x7f68bc19d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.272+0000 7f68c119e640 1 --2- 172.21.15.70:0/1520563147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc102100 0x7f68bc199450 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.272+0000 7f68c119e640 1 --2- 172.21.15.70:0/1520563147 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68bc101740 0x7f68bc198f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.272+0000 7f68c119e640 1 -- 172.21.15.70:0/1520563147 >> 172.21.15.70:0/1520563147 conn(0x7f68bc0fd270 msgr2=0x7f68bc06a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:22.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.272+0000 7f68c119e640 1 -- 172.21.15.70:0/1520563147 shutdown_connections 2024-06-04T04:57:22.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.272+0000 7f68c119e640 1 -- 172.21.15.70:0/1520563147 wait complete. 2024-06-04T04:57:22.285 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-06-04T04:57:22.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.386+0000 7f3f991d4640 1 -- 172.21.15.70:0/3676040095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f94103400 msgr2=0x7f3f9410f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:22.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.386+0000 7f3f991d4640 1 --2- 172.21.15.70:0/3676040095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f94103400 0x7f3f9410f910 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c009cd0 tx=0x7f3f8c02a8e0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.386+0000 7f3f991d4640 1 -- 172.21.15.70:0/3676040095 shutdown_connections 2024-06-04T04:57:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.386+0000 7f3f991d4640 1 --2- 172.21.15.70:0/3676040095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f94103400 0x7f3f9410f910 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.386+0000 7f3f991d4640 1 --2- 172.21.15.70:0/3676040095 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f94102a40 0x7f3f94102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.386+0000 7f3f991d4640 1 --2- 172.21.15.70:0/3676040095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f94108a50 0x7f3f94108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.386+0000 7f3f991d4640 1 -- 172.21.15.70:0/3676040095 >> 172.21.15.70:0/3676040095 conn(0x7f3f940fe510 msgr2=0x7f3f94100930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.386+0000 7f3f991d4640 1 -- 172.21.15.70:0/3676040095 shutdown_connections 2024-06-04T04:57:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.386+0000 7f3f991d4640 1 -- 172.21.15.70:0/3676040095 wait complete. 2024-06-04T04:57:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.387+0000 7f3f991d4640 1 Processor -- start 2024-06-04T04:57:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.387+0000 7f3f991d4640 1 -- start start 2024-06-04T04:57:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.387+0000 7f3f991d4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f94102a40 0x7f3f941990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.387+0000 7f3f991d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f94103400 0x7f3f94199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.387+0000 7f3f991d4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f94108a50 0x7f3f9419d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.387+0000 7f3f991d4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3f94112bc0 con 0x7f3f94102a40 2024-06-04T04:57:22.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.387+0000 7f3f991d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3f94112a40 con 0x7f3f94103400 2024-06-04T04:57:22.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.387+0000 7f3f991d4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3f94112d40 con 0x7f3f94108a50 2024-06-04T04:57:22.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.387+0000 7f3f92d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f94102a40 0x7f3f941990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.387+0000 7f3f92d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f94102a40 0x7f3f941990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45382/0 (socket says 172.21.15.70:45382) 2024-06-04T04:57:22.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.387+0000 7f3f92d9d640 1 -- 172.21.15.70:0/2828389920 learned_addr learned my addr 172.21.15.70:0/2828389920 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:22.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.387+0000 7f3f9359e640 1 --2- 172.21.15.70:0/2828389920 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f94108a50 0x7f3f9419d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.387+0000 7f3f92d9d640 1 -- 172.21.15.70:0/2828389920 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f94108a50 msgr2=0x7f3f9419d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:22.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.387+0000 7f3f92d9d640 1 --2- 172.21.15.70:0/2828389920 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f94108a50 0x7f3f9419d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.387+0000 7f3f92d9d640 1 -- 172.21.15.70:0/2828389920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f94103400 msgr2=0x7f3f94199600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:22.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.387+0000 7f3f92d9d640 1 --2- 172.21.15.70:0/2828389920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f94103400 0x7f3f94199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.387+0000 7f3f92d9d640 1 -- 172.21.15.70:0/2828389920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f9419e0b0 con 0x7f3f94102a40 2024-06-04T04:57:22.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.388+0000 7f3f92d9d640 1 --2- 172.21.15.70:0/2828389920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f94102a40 0x7f3f941990c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400d950 tx=0x7f3f8400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:22.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.388+0000 7f3f7bfff640 1 -- 172.21.15.70:0/2828389920 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84017070 con 0x7f3f94102a40 2024-06-04T04:57:22.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.388+0000 7f3f7bfff640 1 -- 172.21.15.70:0/2828389920 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3f84009c30 con 0x7f3f94102a40 2024-06-04T04:57:22.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.388+0000 7f3f991d4640 1 -- 172.21.15.70:0/2828389920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f9419e3a0 con 0x7f3f94102a40 2024-06-04T04:57:22.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.388+0000 7f3f7bfff640 1 -- 172.21.15.70:0/2828389920 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8400f620 con 0x7f3f94102a40 2024-06-04T04:57:22.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.388+0000 7f3f991d4640 1 -- 172.21.15.70:0/2828389920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f941a5c30 con 0x7f3f94102a40 2024-06-04T04:57:22.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.389+0000 7f3f991d4640 1 -- 172.21.15.70:0/2828389920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f940507e0 con 0x7f3f94102a40 2024-06-04T04:57:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.390+0000 7f3f7bfff640 1 -- 172.21.15.70:0/2828389920 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7f3f84010460 con 0x7f3f94102a40 2024-06-04T04:57:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.391+0000 7f3f7bfff640 1 --2- 172.21.15.70:0/2828389920 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3f7c0b2a90 0x7f3f7c0b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.391+0000 7f3f9259c640 1 --2- 172.21.15.70:0/2828389920 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3f7c0b2a90 0x7f3f7c0b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.391+0000 7f3f7bfff640 1 -- 172.21.15.70:0/2828389920 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f3f840e2410 con 0x7f3f94102a40 2024-06-04T04:57:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.391+0000 7f3f9259c640 1 --2- 172.21.15.70:0/2828389920 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3f7c0b2a90 0x7f3f7c0b4f50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3f94100b90 tx=0x7f3f88005bf0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:22.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.395+0000 7f3f7bfff640 1 -- 172.21.15.70:0/2828389920 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3f840a9c80 con 0x7f3f94102a40 2024-06-04T04:57:22.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.612+0000 7f3f991d4640 1 -- 172.21.15.70:0/2828389920 --> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f3f94062e90 con 0x7f3f7c0b2a90 2024-06-04T04:57:22.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.674+0000 7f3f7bfff640 1 -- 172.21.15.70:0/2828389920 <== mgr.24806 v2:172.21.15.70:6800/542036627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f3f94062e90 con 0x7f3f7c0b2a90 2024-06-04T04:57:22.676 INFO:teuthology.orchestra.run.smithi070.stdout:Retention added to path / 2024-06-04T04:57:22.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.676+0000 7f3f991d4640 1 -- 172.21.15.70:0/2828389920 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3f7c0b2a90 msgr2=0x7f3f7c0b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:22.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.676+0000 7f3f991d4640 1 --2- 172.21.15.70:0/2828389920 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3f7c0b2a90 0x7f3f7c0b4f50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3f94100b90 tx=0x7f3f88005bf0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.676+0000 7f3f991d4640 1 -- 172.21.15.70:0/2828389920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f94102a40 msgr2=0x7f3f941990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:22.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.676+0000 7f3f991d4640 1 --2- 172.21.15.70:0/2828389920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f94102a40 0x7f3f941990c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400d950 tx=0x7f3f8400de10 comp rx=0 tx=0).stop 2024-06-04T04:57:22.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.676+0000 7f3f991d4640 1 -- 172.21.15.70:0/2828389920 shutdown_connections 2024-06-04T04:57:22.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.676+0000 7f3f991d4640 1 --2- 172.21.15.70:0/2828389920 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3f7c0b2a90 0x7f3f7c0b4f50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.676+0000 7f3f991d4640 1 --2- 172.21.15.70:0/2828389920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f94103400 0x7f3f94199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.676+0000 7f3f991d4640 1 --2- 172.21.15.70:0/2828389920 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f94108a50 0x7f3f9419d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.676+0000 7f3f991d4640 1 --2- 172.21.15.70:0/2828389920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f94102a40 0x7f3f941990c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.676+0000 7f3f991d4640 1 -- 172.21.15.70:0/2828389920 >> 172.21.15.70:0/2828389920 conn(0x7f3f940fe510 msgr2=0x7f3f940fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:22.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.676+0000 7f3f991d4640 1 -- 172.21.15.70:0/2828389920 shutdown_connections 2024-06-04T04:57:22.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.676+0000 7f3f991d4640 1 -- 172.21.15.70:0/2828389920 wait complete. 2024-06-04T04:57:22.689 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-06-04T04:57:22.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.805+0000 7f9ff09ff640 1 -- 172.21.15.70:0/2569803846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fec101790 msgr2=0x7f9fec101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:22.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.805+0000 7f9ff09ff640 1 --2- 172.21.15.70:0/2569803846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fec101790 0x7f9fec101c10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0009930 tx=0x7f9fe002a420 comp rx=0 tx=0).stop 2024-06-04T04:57:22.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.805+0000 7f9ff09ff640 1 -- 172.21.15.70:0/2569803846 shutdown_connections 2024-06-04T04:57:22.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.805+0000 7f9ff09ff640 1 --2- 172.21.15.70:0/2569803846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fec102150 0x7f9fec10e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.805+0000 7f9ff09ff640 1 --2- 172.21.15.70:0/2569803846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fec101790 0x7f9fec101c10 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.805+0000 7f9ff09ff640 1 --2- 172.21.15.70:0/2569803846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fec1077a0 0x7f9fec107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.805+0000 7f9ff09ff640 1 -- 172.21.15.70:0/2569803846 >> 172.21.15.70:0/2569803846 conn(0x7f9fec0fd2a0 msgr2=0x7f9fec0ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:22.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.805+0000 7f9ff09ff640 1 -- 172.21.15.70:0/2569803846 shutdown_connections 2024-06-04T04:57:22.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.805+0000 7f9ff09ff640 1 -- 172.21.15.70:0/2569803846 wait complete. 2024-06-04T04:57:22.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9ff09ff640 1 Processor -- start 2024-06-04T04:57:22.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9ff09ff640 1 -- start start 2024-06-04T04:57:22.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9ff09ff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fec101790 0x7f9fec199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9ff09ff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fec102150 0x7f9fec199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9ff09ff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fec1077a0 0x7f9fec19d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9ff09ff640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec111910 con 0x7f9fec102150 2024-06-04T04:57:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9ff09ff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec111790 con 0x7f9fec1077a0 2024-06-04T04:57:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9ff09ff640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec111a90 con 0x7f9fec101790 2024-06-04T04:57:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9fea59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fec102150 0x7f9fec199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9fea59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fec102150 0x7f9fec199540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45400/0 (socket says 172.21.15.70:45400) 2024-06-04T04:57:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9fea59c640 1 -- 172.21.15.70:0/1076391008 learned_addr learned my addr 172.21.15.70:0/1076391008 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9feb59e640 1 --2- 172.21.15.70:0/1076391008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fec1077a0 0x7f9fec19d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9fead9d640 1 --2- 172.21.15.70:0/1076391008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fec101790 0x7f9fec199000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9fea59c640 1 -- 172.21.15.70:0/1076391008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fec101790 msgr2=0x7f9fec199000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9fea59c640 1 --2- 172.21.15.70:0/1076391008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fec101790 0x7f9fec199000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9fea59c640 1 -- 172.21.15.70:0/1076391008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fec1077a0 msgr2=0x7f9fec19d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9fea59c640 1 --2- 172.21.15.70:0/1076391008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fec1077a0 0x7f9fec19d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.806+0000 7f9fea59c640 1 -- 172.21.15.70:0/1076391008 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fec19dff0 con 0x7f9fec102150 2024-06-04T04:57:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.807+0000 7f9fea59c640 1 --2- 172.21.15.70:0/1076391008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fec102150 0x7f9fec199540 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0005940 tx=0x7f9fe0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.807+0000 7f9fd3fff640 1 -- 172.21.15.70:0/1076391008 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe003f030 con 0x7f9fec102150 2024-06-04T04:57:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.807+0000 7f9ff09ff640 1 -- 172.21.15.70:0/1076391008 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fec19e280 con 0x7f9fec102150 2024-06-04T04:57:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.807+0000 7f9fd3fff640 1 -- 172.21.15.70:0/1076391008 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9fe0004440 con 0x7f9fec102150 2024-06-04T04:57:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.807+0000 7f9fd3fff640 1 -- 172.21.15.70:0/1076391008 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0035070 con 0x7f9fec102150 2024-06-04T04:57:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.807+0000 7f9ff09ff640 1 -- 172.21.15.70:0/1076391008 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fec1a5b70 con 0x7f9fec102150 2024-06-04T04:57:22.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.808+0000 7f9ff09ff640 1 -- 172.21.15.70:0/1076391008 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fec051030 con 0x7f9fec102150 2024-06-04T04:57:22.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.809+0000 7f9fd3fff640 1 -- 172.21.15.70:0/1076391008 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7f9fe0042030 con 0x7f9fec102150 2024-06-04T04:57:22.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.810+0000 7f9fd3fff640 1 --2- 172.21.15.70:0/1076391008 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9fd40b29e0 0x7f9fd40b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.810+0000 7f9fead9d640 1 --2- 172.21.15.70:0/1076391008 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9fd40b29e0 0x7f9fd40b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.810+0000 7f9fd3fff640 1 -- 172.21.15.70:0/1076391008 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f9fe0102fa0 con 0x7f9fec102150 2024-06-04T04:57:22.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.810+0000 7f9fead9d640 1 --2- 172.21.15.70:0/1076391008 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9fd40b29e0 0x7f9fd40b4ea0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc0099e0 tx=0x7f9fdc005ed0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.813+0000 7f9fd3fff640 1 -- 172.21.15.70:0/1076391008 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9fe00ca810 con 0x7f9fec102150 2024-06-04T04:57:22.856 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:22 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/51744166' entity='client.admin' 2024-06-04T04:57:22.856 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:22 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:57:22.856 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:22 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:57:22.856 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:22 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:57:22.856 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:22 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:22.857 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:22 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/501357835' entity='client.admin' 2024-06-04T04:57:22.857 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:22 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:57:22.857 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:22 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:57:22.857 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:22 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:57:22.857 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:22 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:22.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:22 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/51744166' entity='client.admin' 2024-06-04T04:57:22.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:22 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:57:22.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:22 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:57:22.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:22 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:57:22.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:22 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:22.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:22 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/501357835' entity='client.admin' 2024-06-04T04:57:22.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:22 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:57:22.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:22 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:57:22.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:22 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:57:22.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:22 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:22.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.962+0000 7f7e01586700 1 -- 172.21.15.70:0/2317279085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7dfc1095a0 msgr2=0x7f7dfc109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:22.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.962+0000 7f7e01586700 1 --2- 172.21.15.70:0/2317279085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7dfc1095a0 0x7f7dfc109980 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7df0009990 tx=0x7f7df0009ca0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.962+0000 7f7e01586700 1 -- 172.21.15.70:0/2317279085 shutdown_connections 2024-06-04T04:57:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.962+0000 7f7e01586700 1 --2- 172.21.15.70:0/2317279085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dfc06c470 0x7f7dfc1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.962+0000 7f7e01586700 1 --2- 172.21.15.70:0/2317279085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7dfc1095a0 0x7f7dfc109980 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.962+0000 7f7e01586700 1 --2- 172.21.15.70:0/2317279085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7dfc06bab0 0x7f7dfc06bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.962+0000 7f7e01586700 1 -- 172.21.15.70:0/2317279085 >> 172.21.15.70:0/2317279085 conn(0x7f7dfc0793b0 msgr2=0x7f7dfc0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.962+0000 7f7e01586700 1 -- 172.21.15.70:0/2317279085 shutdown_connections 2024-06-04T04:57:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.962+0000 7f7e01586700 1 -- 172.21.15.70:0/2317279085 wait complete. 2024-06-04T04:57:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.963+0000 7f7e01586700 1 Processor -- start 2024-06-04T04:57:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.963+0000 7f7e01586700 1 -- start start 2024-06-04T04:57:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.963+0000 7f7e01586700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dfc06bab0 0x7f7dfc102500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.963+0000 7f7e01586700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7dfc06c470 0x7f7dfc102a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.963+0000 7f7e01586700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7dfc1095a0 0x7f7dfc102f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.963+0000 7f7e01586700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7dfc113580 con 0x7f7dfc06c470 2024-06-04T04:57:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.963+0000 7f7e01586700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7dfc113400 con 0x7f7dfc06bab0 2024-06-04T04:57:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.963+0000 7f7e01586700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7dfc113700 con 0x7f7dfc1095a0 2024-06-04T04:57:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.964+0000 7f7dfa7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7dfc06c470 0x7f7dfc102a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.964+0000 7f7dfa7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7dfc06c470 0x7f7dfc102a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45414/0 (socket says 172.21.15.70:45414) 2024-06-04T04:57:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.964+0000 7f7dfa7fc700 1 -- 172.21.15.70:0/3648743517 learned_addr learned my addr 172.21.15.70:0/3648743517 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.964+0000 7f7dfaffd700 1 --2- 172.21.15.70:0/3648743517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dfc06bab0 0x7f7dfc102500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.964+0000 7f7dfb7fe700 1 --2- 172.21.15.70:0/3648743517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7dfc1095a0 0x7f7dfc102f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.964+0000 7f7dfa7fc700 1 -- 172.21.15.70:0/3648743517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7dfc1095a0 msgr2=0x7f7dfc102f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.964+0000 7f7dfa7fc700 1 --2- 172.21.15.70:0/3648743517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7dfc1095a0 0x7f7dfc102f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.964+0000 7f7dfa7fc700 1 -- 172.21.15.70:0/3648743517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dfc06bab0 msgr2=0x7f7dfc102500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.964+0000 7f7dfa7fc700 1 --2- 172.21.15.70:0/3648743517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dfc06bab0 0x7f7dfc102500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.964+0000 7f7dfa7fc700 1 -- 172.21.15.70:0/3648743517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dfc1a8080 con 0x7f7dfc06c470 2024-06-04T04:57:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.964+0000 7f7dfb7fe700 1 --2- 172.21.15.70:0/3648743517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7dfc1095a0 0x7f7dfc102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.964+0000 7f7dfa7fc700 1 --2- 172.21.15.70:0/3648743517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7dfc06c470 0x7f7dfc102a40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00d980 tx=0x7f7dec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.964+0000 7f7de3fff700 1 -- 172.21.15.70:0/3648743517 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec017070 con 0x7f7dfc06c470 2024-06-04T04:57:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.964+0000 7f7de3fff700 1 -- 172.21.15.70:0/3648743517 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7dec005020 con 0x7f7dfc06c470 2024-06-04T04:57:22.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.964+0000 7f7e01586700 1 -- 172.21.15.70:0/3648743517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dfc1a8370 con 0x7f7dfc06c470 2024-06-04T04:57:22.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.964+0000 7f7de3fff700 1 -- 172.21.15.70:0/3648743517 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec009ab0 con 0x7f7dfc06c470 2024-06-04T04:57:22.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.964+0000 7f7e01586700 1 -- 172.21.15.70:0/3648743517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dfc1a8860 con 0x7f7dfc06c470 2024-06-04T04:57:22.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.967+0000 7f7de3fff700 1 -- 172.21.15.70:0/3648743517 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7f7dec0051d0 con 0x7f7dfc06c470 2024-06-04T04:57:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.968+0000 7f7de3fff700 1 --2- 172.21.15.70:0/3648743517 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7de40b2a80 0x7f7de40b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.968+0000 7f7dfaffd700 1 --2- 172.21.15.70:0/3648743517 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7de40b2a80 0x7f7de40b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.968+0000 7f7de3fff700 1 -- 172.21.15.70:0/3648743517 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f7dec0e26e0 con 0x7f7dfc06c470 2024-06-04T04:57:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.968+0000 7f7e0be1bb80 1 Processor -- start 2024-06-04T04:57:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.968+0000 7f7dfaffd700 1 --2- 172.21.15.70:0/3648743517 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7de40b2a80 0x7f7de40b4f40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7df000b5c0 tx=0x7f7df0005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.968+0000 7f7e0be1bb80 1 -- start start 2024-06-04T04:57:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.968+0000 7f7e0be1bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074db3a0 0x560e074db780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.968+0000 7f7e0be1bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074d5920 0x560e074d5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.968+0000 7f7e0be1bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074d62e0 0x560e074e2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.969+0000 7f7e0be1bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560e07433df0 con 0x560e074d62e0 2024-06-04T04:57:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.969+0000 7f7e0be1bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560e07423990 con 0x560e074db3a0 2024-06-04T04:57:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.969+0000 7f7e0be1bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560e074279d0 con 0x560e074d5920 2024-06-04T04:57:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.969+0000 7f7dfb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074d62e0 0x560e074e2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.969+0000 7f7dfb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074d62e0 0x560e074e2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45418/0 (socket says 172.21.15.70:45418) 2024-06-04T04:57:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.969+0000 7f7dfb7fe700 1 -- 172.21.15.70:0/3532785443 learned_addr learned my addr 172.21.15.70:0/3532785443 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.969+0000 7f7dfa7fc700 1 --2- 172.21.15.70:0/3532785443 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074d5920 0x560e074d5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.969+0000 7f7dfaffd700 1 --2- 172.21.15.70:0/3532785443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074db3a0 0x560e074db780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.969+0000 7f7dfb7fe700 1 -- 172.21.15.70:0/3532785443 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074d5920 msgr2=0x560e074d5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.969+0000 7f7dfb7fe700 1 --2- 172.21.15.70:0/3532785443 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074d5920 0x560e074d5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.969+0000 7f7dfb7fe700 1 -- 172.21.15.70:0/3532785443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074db3a0 msgr2=0x560e074db780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.969+0000 7f7dfb7fe700 1 --2- 172.21.15.70:0/3532785443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074db3a0 0x560e074db780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.969+0000 7f7dfb7fe700 1 -- 172.21.15.70:0/3532785443 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e073cd030 con 0x560e074d62e0 2024-06-04T04:57:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.969+0000 7f7dfb7fe700 1 --2- 172.21.15.70:0/3532785443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074d62e0 0x560e074e2100 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009dd0 tx=0x7f7df40077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e09be215b1c4133e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.969+0000 7f7de17fa700 1 -- 172.21.15.70:0/3532785443 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df4004bb0 con 0x560e074d62e0 2024-06-04T04:57:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.969+0000 7f7de17fa700 1 -- 172.21.15.70:0/3532785443 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7df4004d50 con 0x560e074d62e0 2024-06-04T04:57:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.969+0000 7f7de17fa700 1 -- 172.21.15.70:0/3532785443 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df4014410 con 0x560e074d62e0 2024-06-04T04:57:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.970+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/3532785443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074d62e0 msgr2=0x560e074e2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.970+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/3532785443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074d62e0 0x560e074e2100 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009dd0 tx=0x7f7df40077e0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.970+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/3532785443 shutdown_connections 2024-06-04T04:57:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.970+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/3532785443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074d62e0 0x560e074e2100 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.970+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/3532785443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074db3a0 0x560e074db780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.970+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/3532785443 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074d5920 0x560e074d5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.970+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/3532785443 >> 172.21.15.70:0/3532785443 conn(0x560e074012c0 msgr2=0x560e074d0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.970+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/3532785443 shutdown_connections 2024-06-04T04:57:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.970+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/3532785443 wait complete. 2024-06-04T04:57:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7e0be1bb80 1 -- start start 2024-06-04T04:57:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7e0be1bb80 1 Processor -- start 2024-06-04T04:57:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7e0be1bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074d5920 0x560e074e1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7e0be1bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074d62e0 0x560e074df140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7e0be1bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074db3a0 0x560e074df680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7e0be1bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560e0732a020 con 0x560e074db3a0 2024-06-04T04:57:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7e0be1bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560e0738be20 con 0x560e074d62e0 2024-06-04T04:57:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7e0be1bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560e074e2760 con 0x560e074d5920 2024-06-04T04:57:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7e0be1bb80 10 client.0 _ll_register_callbacks cb 0x560e073be9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7dfb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074db3a0 0x560e074df680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7dfb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074db3a0 0x560e074df680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45424/0 (socket says 172.21.15.70:45424) 2024-06-04T04:57:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7dfb7fe700 1 -- 172.21.15.70:0/1368209575 learned_addr learned my addr 172.21.15.70:0/1368209575 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7dfa7fc700 1 --2- 172.21.15.70:0/1368209575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074d62e0 0x560e074df140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7dfaffd700 1 --2- 172.21.15.70:0/1368209575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074d5920 0x560e074e1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7dfb7fe700 1 -- 172.21.15.70:0/1368209575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074d5920 msgr2=0x560e074e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7dfb7fe700 1 --2- 172.21.15.70:0/1368209575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074d5920 0x560e074e1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7dfb7fe700 1 -- 172.21.15.70:0/1368209575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074d62e0 msgr2=0x560e074df140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7dfb7fe700 1 --2- 172.21.15.70:0/1368209575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074d62e0 0x560e074df140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.971+0000 7f7dfb7fe700 1 -- 172.21.15.70:0/1368209575 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e07373950 con 0x560e074db3a0 2024-06-04T04:57:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.972+0000 7f7dfb7fe700 1 --2- 172.21.15.70:0/1368209575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074db3a0 0x560e074df680 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7df4000f80 tx=0x7f7df4006080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea23c3bdfb84913a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.972+0000 7f7ddeff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:57:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.972+0000 7f7ddeff5700 1 -- 172.21.15.70:0/1368209575 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df4022070 con 0x560e074db3a0 2024-06-04T04:57:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.972+0000 7f7ddeff5700 1 -- 172.21.15.70:0/1368209575 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f7df4005290 con 0x560e074db3a0 2024-06-04T04:57:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.972+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/1368209575 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7df4004d50 con 0x560e074db3a0 2024-06-04T04:57:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.972+0000 7f7ddeff5700 1 -- 172.21.15.70:0/1368209575 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7df402e050 con 0x560e074db3a0 2024-06-04T04:57:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.972+0000 7f7e0be1bb80 10 client.15387 fetch_fsmap learned FSMap version 29 2024-06-04T04:57:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.972+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/1368209575 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560e073dd7e0 con 0x560e074db3a0 2024-06-04T04:57:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.972+0000 7f7ddeff5700 1 -- 172.21.15.70:0/1368209575 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7df4004d50 con 0x560e074db3a0 2024-06-04T04:57:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.972+0000 7f7ddeff5700 1 -- 172.21.15.70:0/1368209575 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f7df4014410 con 0x560e074db3a0 2024-06-04T04:57:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.973+0000 7f7e0be1bb80 10 client.15387 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:57:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.973+0000 7f7e0be1bb80 10 client.15387 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:57:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.973+0000 7f7e0be1bb80 20 client.15387 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.973+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/1368209575 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560e074ef180 0x560e074f1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.973+0000 7f7e0be1bb80 4 client.15387 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:57:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.973+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/1368209575 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e07435d40 con 0x560e074ef180 2024-06-04T04:57:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.973+0000 7f7dfaffd700 1 --2- 172.21.15.70:0/1368209575 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560e074ef180 0x560e074f1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.973+0000 7f7ddeff5700 10 client.15387 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.973+0000 7f7dfaffd700 1 --2- 172.21.15.70:0/1368209575 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560e074ef180 0x560e074f1570 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cc2b276919bf18f server_cookie=42b05103806d8084 in_seq=0 out_seq=0 2024-06-04T04:57:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.974+0000 7f7ddeff5700 1 -- 172.21.15.70:0/1368209575 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560e07435d40 con 0x560e074ef180 2024-06-04T04:57:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.974+0000 7f7ddeff5700 10 client.15387 handle_command_reply: tid=0 2024-06-04T04:57:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.974+0000 7f7e0be1bb80 1 client.15387 shutdown 2024-06-04T04:57:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.974+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/1368209575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074db3a0 msgr2=0x560e074df680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.974+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/1368209575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074db3a0 0x560e074df680 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7df4000f80 tx=0x7f7df4006080 comp rx=0 tx=0).stop 2024-06-04T04:57:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.974+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/1368209575 shutdown_connections 2024-06-04T04:57:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.974+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/1368209575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074db3a0 0x560e074df680 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.974+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/1368209575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074d5920 0x560e074e1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.974+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/1368209575 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560e074ef180 0x560e074f1570 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.974+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/1368209575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074d62e0 0x560e074df140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.974+0000 7f7ddeff5700 0 client.15387 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.974+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/1368209575 >> 172.21.15.70:0/1368209575 conn(0x560e074012c0 msgr2=0x560e074cfb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.974+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/1368209575 shutdown_connections 2024-06-04T04:57:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.974+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/1368209575 wait complete. 2024-06-04T04:57:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:22.974+0000 7f7e0be1bb80 20 client.15387 trim_cache size 0 max 16384 2024-06-04T04:57:22.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:22 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/51744166' entity='client.admin' 2024-06-04T04:57:22.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:22 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:57:22.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:22 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:57:22.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:22 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:57:22.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:22 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:22.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:22 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/501357835' entity='client.admin' 2024-06-04T04:57:22.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:22 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:57:22.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:22 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:57:22.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:22 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:57:22.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:22 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:57:23.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7e0be1bb80 1 Processor -- start 2024-06-04T04:57:23.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7e0be1bb80 1 -- start start 2024-06-04T04:57:23.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7e0be1bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074d5920 0x560e074d9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:23.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7e0be1bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074d62e0 0x560e074d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:23.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7e0be1bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074db3a0 0x560e074de140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:23.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7e0be1bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7dfc1b0090 con 0x560e074d5920 2024-06-04T04:57:23.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7e0be1bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7dfc1b1bd0 con 0x560e074d62e0 2024-06-04T04:57:23.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7e0be1bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7dfc1b0760 con 0x560e074db3a0 2024-06-04T04:57:23.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7dfaffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074d5920 0x560e074d9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:23.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7dfaffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074d5920 0x560e074d9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45438/0 (socket says 172.21.15.70:45438) 2024-06-04T04:57:23.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7dfaffd700 1 -- 172.21.15.70:0/889325912 learned_addr learned my addr 172.21.15.70:0/889325912 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:23.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7dfa7fc700 1 --2- 172.21.15.70:0/889325912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074d62e0 0x560e074d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:23.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7dfaffd700 1 -- 172.21.15.70:0/889325912 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074db3a0 msgr2=0x560e074de140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:23.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7dfb7fe700 1 --2- 172.21.15.70:0/889325912 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074db3a0 0x560e074de140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:23.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7dfaffd700 1 --2- 172.21.15.70:0/889325912 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074db3a0 0x560e074de140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7dfaffd700 1 -- 172.21.15.70:0/889325912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074d62e0 msgr2=0x560e074d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:23.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7dfaffd700 1 --2- 172.21.15.70:0/889325912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074d62e0 0x560e074d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.002+0000 7f7dfaffd700 1 -- 172.21.15.70:0/889325912 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e074e1930 con 0x560e074d5920 2024-06-04T04:57:23.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.003+0000 7f7dfaffd700 1 --2- 172.21.15.70:0/889325912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074d5920 0x560e074d9660 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7df0000c00 tx=0x7f7df0021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6bce2921a8d908c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:23.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.003+0000 7f7de0ff9700 1 -- 172.21.15.70:0/889325912 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df000b7d0 con 0x560e074d5920 2024-06-04T04:57:23.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.003+0000 7f7de0ff9700 1 -- 172.21.15.70:0/889325912 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7df000be00 con 0x560e074d5920 2024-06-04T04:57:23.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.003+0000 7f7de0ff9700 1 -- 172.21.15.70:0/889325912 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df000fc20 con 0x560e074d5920 2024-06-04T04:57:23.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.004+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/889325912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074d5920 msgr2=0x560e074d9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:23.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.004+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/889325912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074d5920 0x560e074d9660 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7df0000c00 tx=0x7f7df0021000 comp rx=0 tx=0).stop 2024-06-04T04:57:23.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.004+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/889325912 shutdown_connections 2024-06-04T04:57:23.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.004+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/889325912 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074db3a0 0x560e074de140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.004+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/889325912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074d5920 0x560e074d9660 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.004+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/889325912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074d62e0 0x560e074d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.004+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/889325912 >> 172.21.15.70:0/889325912 conn(0x560e074012c0 msgr2=0x560e074dda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:23.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.004+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/889325912 shutdown_connections 2024-06-04T04:57:23.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.004+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/889325912 wait complete. 2024-06-04T04:57:23.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.004+0000 7f7e0be1bb80 1 -- start start 2024-06-04T04:57:23.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.004+0000 7f7e0be1bb80 1 Processor -- start 2024-06-04T04:57:23.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.004+0000 7f7e0be1bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074d5920 0x560e074e7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:23.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.004+0000 7f7e0be1bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074d62e0 0x560e074e80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:23.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.004+0000 7f7e0be1bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074db3a0 0x560e074e4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:23.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.004+0000 7f7e0be1bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7dfc1b1500 con 0x560e074db3a0 2024-06-04T04:57:23.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.004+0000 7f7e0be1bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7dfc1b0e30 con 0x560e074d5920 2024-06-04T04:57:23.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.004+0000 7f7e0be1bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7dfc1ae030 con 0x560e074d62e0 2024-06-04T04:57:23.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7e0be1bb80 10 client.0 _ll_register_callbacks cb 0x560e073be9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:23.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7dfb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074db3a0 0x560e074e4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:23.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7dfb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074db3a0 0x560e074e4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45444/0 (socket says 172.21.15.70:45444) 2024-06-04T04:57:23.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7dfb7fe700 1 -- 172.21.15.70:0/4260423986 learned_addr learned my addr 172.21.15.70:0/4260423986 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:23.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7dfaffd700 1 --2- 172.21.15.70:0/4260423986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074d5920 0x560e074e7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:23.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7dfa7fc700 1 --2- 172.21.15.70:0/4260423986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074d62e0 0x560e074e80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:23.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7dfb7fe700 1 -- 172.21.15.70:0/4260423986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074d62e0 msgr2=0x560e074e80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:23.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7dfb7fe700 1 --2- 172.21.15.70:0/4260423986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074d62e0 0x560e074e80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7dfb7fe700 1 -- 172.21.15.70:0/4260423986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074d5920 msgr2=0x560e074e7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:23.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7dfb7fe700 1 --2- 172.21.15.70:0/4260423986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074d5920 0x560e074e7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7dfb7fe700 1 -- 172.21.15.70:0/4260423986 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e074e5030 con 0x560e074db3a0 2024-06-04T04:57:23.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7dfa7fc700 1 --2- 172.21.15.70:0/4260423986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074d62e0 0x560e074e80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:23.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7dfb7fe700 1 --2- 172.21.15.70:0/4260423986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074db3a0 0x560e074e4980 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009310 tx=0x7f7df4005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c7ec86a7d36c9f1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:23.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7ddeff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:57:23.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7ddeff5700 1 -- 172.21.15.70:0/4260423986 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df402b020 con 0x560e074db3a0 2024-06-04T04:57:23.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/4260423986 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7df000be00 con 0x560e074db3a0 2024-06-04T04:57:23.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7ddeff5700 1 -- 172.21.15.70:0/4260423986 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f7df4019040 con 0x560e074db3a0 2024-06-04T04:57:23.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7ddeff5700 1 -- 172.21.15.70:0/4260423986 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7df4022070 con 0x560e074db3a0 2024-06-04T04:57:23.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.005+0000 7f7e0be1bb80 10 client.15393 fetch_fsmap learned FSMap version 29 2024-06-04T04:57:23.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.006+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/4260423986 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560e0755a450 con 0x560e074db3a0 2024-06-04T04:57:23.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.006+0000 7f7ddeff5700 1 -- 172.21.15.70:0/4260423986 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7df000be00 con 0x560e074db3a0 2024-06-04T04:57:23.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.006+0000 7f7ddeff5700 1 -- 172.21.15.70:0/4260423986 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f7df40278d0 con 0x560e074db3a0 2024-06-04T04:57:23.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.006+0000 7f7e0be1bb80 10 client.15393 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:57:23.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.006+0000 7f7e0be1bb80 10 client.15393 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:57:23.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.006+0000 7f7e0be1bb80 20 client.15393 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:23.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.006+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/4260423986 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560e0755aaf0 0x560e0755cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:23.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.006+0000 7f7e0be1bb80 4 client.15393 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T04:57:23.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.006+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/4260423986 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560e0742f520 con 0x560e0755aaf0 2024-06-04T04:57:23.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.006+0000 7f7dfaffd700 1 --2- 172.21.15.70:0/4260423986 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560e0755aaf0 0x560e0755cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:23.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.007+0000 7f7ddeff5700 10 client.15393 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:23.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.007+0000 7f7dfaffd700 1 --2- 172.21.15.70:0/4260423986 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560e0755aaf0 0x560e0755cf10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e755e59bd366972 server_cookie=6b36ce8c898b5055 in_seq=0 out_seq=0 2024-06-04T04:57:23.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.008+0000 7f7ddeff5700 1 -- 172.21.15.70:0/4260423986 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560e0742f520 con 0x560e0755aaf0 2024-06-04T04:57:23.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.008+0000 7f7ddeff5700 10 client.15393 handle_command_reply: tid=0 2024-06-04T04:57:23.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.008+0000 7f7e0be1bb80 1 client.15393 shutdown 2024-06-04T04:57:23.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.008+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/4260423986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074db3a0 msgr2=0x560e074e4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:23.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.008+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/4260423986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074db3a0 0x560e074e4980 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009310 tx=0x7f7df4005750 comp rx=0 tx=0).stop 2024-06-04T04:57:23.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.008+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/4260423986 shutdown_connections 2024-06-04T04:57:23.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.008+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/4260423986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e074db3a0 0x560e074e4980 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.008+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/4260423986 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560e0755aaf0 0x560e0755cf10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.008+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/4260423986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e074d5920 0x560e074e7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.008+0000 7f7e0be1bb80 1 --2- 172.21.15.70:0/4260423986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e074d62e0 0x560e074e80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.008+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/4260423986 >> 172.21.15.70:0/4260423986 conn(0x560e074012c0 msgr2=0x560e074de2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:23.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.008+0000 7f7ddeff5700 0 client.15393 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:23.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.009+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/4260423986 shutdown_connections 2024-06-04T04:57:23.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.009+0000 7f7e0be1bb80 1 -- 172.21.15.70:0/4260423986 wait complete. 2024-06-04T04:57:23.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.009+0000 7f7e0be1bb80 20 client.15393 trim_cache size 0 max 16384 2024-06-04T04:57:23.016 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:57:23.016 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T04:57:23.016 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "6b7fb4d9-6a0c-4504-b8bf-cc7a477e26af", 2024-06-04T04:57:23.016 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T04:57:23.016 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:57:23.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.009+0000 7f7e01586700 1 -- 172.21.15.70:0/3648743517 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7de40b2a80 msgr2=0x7f7de40b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:23.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.009+0000 7f7e01586700 1 --2- 172.21.15.70:0/3648743517 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7de40b2a80 0x7f7de40b4f40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7df000b5c0 tx=0x7f7df0005960 comp rx=0 tx=0).stop 2024-06-04T04:57:23.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.009+0000 7f7e01586700 1 -- 172.21.15.70:0/3648743517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7dfc06c470 msgr2=0x7f7dfc102a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:23.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.009+0000 7f7e01586700 1 --2- 172.21.15.70:0/3648743517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7dfc06c470 0x7f7dfc102a40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00d980 tx=0x7f7dec00dc90 comp rx=0 tx=0).stop 2024-06-04T04:57:23.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.009+0000 7f7e01586700 1 -- 172.21.15.70:0/3648743517 shutdown_connections 2024-06-04T04:57:23.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.009+0000 7f7e01586700 1 --2- 172.21.15.70:0/3648743517 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7de40b2a80 0x7f7de40b4f40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.009+0000 7f7e01586700 1 --2- 172.21.15.70:0/3648743517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7dfc1095a0 0x7f7dfc102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.009+0000 7f7e01586700 1 --2- 172.21.15.70:0/3648743517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dfc06bab0 0x7f7dfc102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.009+0000 7f7e01586700 1 --2- 172.21.15.70:0/3648743517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7dfc06c470 0x7f7dfc102a40 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.009+0000 7f7e01586700 1 -- 172.21.15.70:0/3648743517 >> 172.21.15.70:0/3648743517 conn(0x7f7dfc0793b0 msgr2=0x7f7dfc101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:23.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.009+0000 7f7e01586700 1 -- 172.21.15.70:0/3648743517 shutdown_connections 2024-06-04T04:57:23.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.009+0000 7f7e01586700 1 -- 172.21.15.70:0/3648743517 wait complete. 2024-06-04T04:57:23.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.039+0000 7f9ff09ff640 1 -- 172.21.15.70:0/1076391008 --> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f9fec06ab30 con 0x7f9fd40b29e0 2024-06-04T04:57:23.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.047+0000 7f9fd3fff640 1 -- 172.21.15.70:0/1076391008 <== mgr.24806 v2:172.21.15.70:6800/542036627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f9fec06ab30 con 0x7f9fd40b29e0 2024-06-04T04:57:23.049 INFO:teuthology.orchestra.run.smithi070.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-06-04T00:00:00", "created": "2024-06-04T04:57:22", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-06-04T04:57:23.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.049+0000 7f9ff09ff640 1 -- 172.21.15.70:0/1076391008 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9fd40b29e0 msgr2=0x7f9fd40b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.049+0000 7f9ff09ff640 1 --2- 172.21.15.70:0/1076391008 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9fd40b29e0 0x7f9fd40b4ea0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc0099e0 tx=0x7f9fdc005ed0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.049+0000 7f9ff09ff640 1 -- 172.21.15.70:0/1076391008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fec102150 msgr2=0x7f9fec199540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.049+0000 7f9ff09ff640 1 --2- 172.21.15.70:0/1076391008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fec102150 0x7f9fec199540 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0005940 tx=0x7f9fe0004040 comp rx=0 tx=0).stop 2024-06-04T04:57:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.049+0000 7f9ff09ff640 1 -- 172.21.15.70:0/1076391008 shutdown_connections 2024-06-04T04:57:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.049+0000 7f9ff09ff640 1 --2- 172.21.15.70:0/1076391008 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9fd40b29e0 0x7f9fd40b4ea0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.049+0000 7f9ff09ff640 1 --2- 172.21.15.70:0/1076391008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fec1077a0 0x7f9fec19d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.049+0000 7f9ff09ff640 1 --2- 172.21.15.70:0/1076391008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fec102150 0x7f9fec199540 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.049+0000 7f9ff09ff640 1 --2- 172.21.15.70:0/1076391008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fec101790 0x7f9fec199000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.049+0000 7f9ff09ff640 1 -- 172.21.15.70:0/1076391008 >> 172.21.15.70:0/1076391008 conn(0x7f9fec0fd2a0 msgr2=0x7f9fec1056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.049+0000 7f9ff09ff640 1 -- 172.21.15.70:0/1076391008 shutdown_connections 2024-06-04T04:57:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.049+0000 7f9ff09ff640 1 -- 172.21.15.70:0/1076391008 wait complete. 2024-06-04T04:57:23.062 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-06-04T04:57:23.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.165+0000 7f85b2637640 1 -- 172.21.15.70:0/252531025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85ac107940 msgr2=0x7f85ac107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:23.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.165+0000 7f85b2637640 1 --2- 172.21.15.70:0/252531025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85ac107940 0x7f85ac107d20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f859c009930 tx=0x7f859c02a420 comp rx=0 tx=0).stop 2024-06-04T04:57:23.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.165+0000 7f85b2637640 1 -- 172.21.15.70:0/252531025 shutdown_connections 2024-06-04T04:57:23.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.165+0000 7f85b2637640 1 --2- 172.21.15.70:0/252531025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85ac102340 0x7f85ac10e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.165+0000 7f85b2637640 1 --2- 172.21.15.70:0/252531025 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85ac101980 0x7f85ac101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.165+0000 7f85b2637640 1 --2- 172.21.15.70:0/252531025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85ac107940 0x7f85ac107d20 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.165+0000 7f85b2637640 1 -- 172.21.15.70:0/252531025 >> 172.21.15.70:0/252531025 conn(0x7f85ac078ad0 msgr2=0x7f85ac078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:23.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.165+0000 7f85b2637640 1 -- 172.21.15.70:0/252531025 shutdown_connections 2024-06-04T04:57:23.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.165+0000 7f85b2637640 1 -- 172.21.15.70:0/252531025 wait complete. 2024-06-04T04:57:23.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.165+0000 7f85b2637640 1 Processor -- start 2024-06-04T04:57:23.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.165+0000 7f85b2637640 1 -- start start 2024-06-04T04:57:23.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85b2637640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85ac101980 0x7f85ac1990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:23.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85b2637640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85ac102340 0x7f85ac199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:23.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85b2637640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85ac107940 0x7f85ac19d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:23.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85b2637640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac112bb0 con 0x7f85ac101980 2024-06-04T04:57:23.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85b2637640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac112a30 con 0x7f85ac107940 2024-06-04T04:57:23.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85b2637640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac112d30 con 0x7f85ac102340 2024-06-04T04:57:23.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85abfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85ac101980 0x7f85ac1990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:23.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85abfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85ac101980 0x7f85ac1990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45456/0 (socket says 172.21.15.70:45456) 2024-06-04T04:57:23.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85abfff640 1 -- 172.21.15.70:0/37250702 learned_addr learned my addr 172.21.15.70:0/37250702 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:23.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85ab7fe640 1 --2- 172.21.15.70:0/37250702 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85ac102340 0x7f85ac199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:23.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85b0bd4640 1 --2- 172.21.15.70:0/37250702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85ac107940 0x7f85ac19d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:23.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85abfff640 1 -- 172.21.15.70:0/37250702 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85ac102340 msgr2=0x7f85ac199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:23.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85abfff640 1 --2- 172.21.15.70:0/37250702 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85ac102340 0x7f85ac199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85abfff640 1 -- 172.21.15.70:0/37250702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85ac107940 msgr2=0x7f85ac19d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:23.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85abfff640 1 --2- 172.21.15.70:0/37250702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85ac107940 0x7f85ac19d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85abfff640 1 -- 172.21.15.70:0/37250702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ac19e0e0 con 0x7f85ac101980 2024-06-04T04:57:23.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85b0bd4640 1 --2- 172.21.15.70:0/37250702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85ac107940 0x7f85ac19d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:23.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85ab7fe640 1 --2- 172.21.15.70:0/37250702 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85ac102340 0x7f85ac199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:23.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85abfff640 1 --2- 172.21.15.70:0/37250702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85ac101980 0x7f85ac1990f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f859c005940 tx=0x7f859c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:23.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85a97fa640 1 -- 172.21.15.70:0/37250702 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c03f030 con 0x7f85ac101980 2024-06-04T04:57:23.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.166+0000 7f85a97fa640 1 -- 172.21.15.70:0/37250702 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f859c003c90 con 0x7f85ac101980 2024-06-04T04:57:23.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.167+0000 7f85b2637640 1 -- 172.21.15.70:0/37250702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85ac19e370 con 0x7f85ac101980 2024-06-04T04:57:23.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.167+0000 7f85a97fa640 1 -- 172.21.15.70:0/37250702 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c035070 con 0x7f85ac101980 2024-06-04T04:57:23.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.167+0000 7f85b2637640 1 -- 172.21.15.70:0/37250702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85ac1a5c60 con 0x7f85ac101980 2024-06-04T04:57:23.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.168+0000 7f85b2637640 1 -- 172.21.15.70:0/37250702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85ac0507e0 con 0x7f85ac101980 2024-06-04T04:57:23.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.169+0000 7f85a97fa640 1 -- 172.21.15.70:0/37250702 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7f859c042030 con 0x7f85ac101980 2024-06-04T04:57:23.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.169+0000 7f85a97fa640 1 --2- 172.21.15.70:0/37250702 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f85980b2a90 0x7f85980b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:23.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.170+0000 7f85ab7fe640 1 --2- 172.21.15.70:0/37250702 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f85980b2a90 0x7f85980b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:23.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.170+0000 7f85a97fa640 1 -- 172.21.15.70:0/37250702 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f859c1030e0 con 0x7f85ac101980 2024-06-04T04:57:23.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.170+0000 7f85ab7fe640 1 --2- 172.21.15.70:0/37250702 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f85980b2a90 0x7f85980b4f50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f85940058c0 tx=0x7f8594005850 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:23.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.173+0000 7f85a97fa640 1 -- 172.21.15.70:0/37250702 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f859c0ca9f0 con 0x7f85ac101980 2024-06-04T04:57:23.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.391+0000 7f85b2637640 1 -- 172.21.15.70:0/37250702 --> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f85ac062e90 con 0x7f85980b2a90 2024-06-04T04:57:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.398+0000 7f85a97fa640 1 -- 172.21.15.70:0/37250702 <== mgr.24806 v2:172.21.15.70:6800/542036627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f85ac062e90 con 0x7f85980b2a90 2024-06-04T04:57:23.400 INFO:teuthology.orchestra.run.smithi070.stdout:/ 1m 6m3h 2024-06-04T04:57:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.400+0000 7f85b2637640 1 -- 172.21.15.70:0/37250702 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f85980b2a90 msgr2=0x7f85980b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.400+0000 7f85b2637640 1 --2- 172.21.15.70:0/37250702 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f85980b2a90 0x7f85980b4f50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f85940058c0 tx=0x7f8594005850 comp rx=0 tx=0).stop 2024-06-04T04:57:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.400+0000 7f85b2637640 1 -- 172.21.15.70:0/37250702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85ac101980 msgr2=0x7f85ac1990f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.400+0000 7f85b2637640 1 --2- 172.21.15.70:0/37250702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85ac101980 0x7f85ac1990f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f859c005940 tx=0x7f859c0042f0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.400+0000 7f85b2637640 1 -- 172.21.15.70:0/37250702 shutdown_connections 2024-06-04T04:57:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.400+0000 7f85b2637640 1 --2- 172.21.15.70:0/37250702 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f85980b2a90 0x7f85980b4f50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.400+0000 7f85b2637640 1 --2- 172.21.15.70:0/37250702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85ac107940 0x7f85ac19d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.400+0000 7f85b2637640 1 --2- 172.21.15.70:0/37250702 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85ac102340 0x7f85ac199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.400+0000 7f85b2637640 1 --2- 172.21.15.70:0/37250702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85ac101980 0x7f85ac1990f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.400+0000 7f85b2637640 1 -- 172.21.15.70:0/37250702 >> 172.21.15.70:0/37250702 conn(0x7f85ac078ad0 msgr2=0x7f85ac1045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.400+0000 7f85b2637640 1 -- 172.21.15.70:0/37250702 shutdown_connections 2024-06-04T04:57:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:23.400+0000 7f85b2637640 1 -- 172.21.15.70:0/37250702 wait complete. 2024-06-04T04:57:23.413 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-06-04T04:57:23.438 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-06-04T04:57:23.462 INFO:teuthology.run_tasks:Running task background_exec... 2024-06-04T04:57:23.488 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi070.front.sepia.ceph.com 2024-06-04T04:57:23.488 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-06-04T04:57:23.490 INFO:teuthology.run_tasks:Running task quiescer... 2024-06-04T04:57:23.544 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-04T04:57:23.544 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-04T04:57:23.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:23 smithi070 ceph-mon[32631]: pgmap v9: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 0 B/s wr, 15 op/s 2024-06-04T04:57:23.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:23 smithi070 ceph-mon[32631]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:57:23.734 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T04:57:23.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:23 smithi098 ceph-mon[35569]: pgmap v9: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 0 B/s wr, 15 op/s 2024-06-04T04:57:23.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:23 smithi098 ceph-mon[35569]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:57:23.788 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:23.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:23 smithi112 ceph-mon[36453]: pgmap v9: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 0 B/s wr, 15 op/s 2024-06-04T04:57:23.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:23 smithi112 ceph-mon[36453]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:57:24.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:24 smithi070 ceph-mon[32631]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:57:24.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:24 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T04:57:24.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:24 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T04:57:24.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:24 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T04:57:24.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:24 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T04:57:24.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:24 smithi070 ceph-mon[32631]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:57:24.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:24 smithi070 ceph-mon[32631]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:57:24.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:24 smithi112 ceph-mon[36453]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:57:24.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:24 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T04:57:24.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:24 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T04:57:24.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:24 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T04:57:24.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:24 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T04:57:24.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:24 smithi112 ceph-mon[36453]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:57:24.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:24 smithi112 ceph-mon[36453]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:57:24.968 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:24.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:24 smithi098 ceph-mon[35569]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:57:24.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:24 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T04:57:24.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:24 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T04:57:24.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:24 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T04:57:24.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:24 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T04:57:24.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:24 smithi098 ceph-mon[35569]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:57:24.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:24 smithi098 ceph-mon[35569]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T04:57:25.805 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:25 smithi070 ceph-mon[32631]: pgmap v10: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 0 B/s wr, 14 op/s 2024-06-04T04:57:25.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.794+0000 7f29157ea700 1 -- 172.21.15.70:0/3428225739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2910109520 msgr2=0x7f2910109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.794+0000 7f29157ea700 1 --2- 172.21.15.70:0/3428225739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2910109520 0x7f2910109900 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f29000099c0 tx=0x7f2900009cd0 comp rx=0 tx=0).stop 2024-06-04T04:57:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.794+0000 7f29157ea700 1 -- 172.21.15.70:0/3428225739 shutdown_connections 2024-06-04T04:57:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.794+0000 7f29157ea700 1 --2- 172.21.15.70:0/3428225739 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29101024f0 0x7f29101103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.794+0000 7f29157ea700 1 --2- 172.21.15.70:0/3428225739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2910109520 0x7f2910109900 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.794+0000 7f29157ea700 1 --2- 172.21.15.70:0/3428225739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2910101b30 0x7f2910101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.794+0000 7f29157ea700 1 -- 172.21.15.70:0/3428225739 >> 172.21.15.70:0/3428225739 conn(0x7f2910078ab0 msgr2=0x7f29100ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.794+0000 7f29157ea700 1 -- 172.21.15.70:0/3428225739 shutdown_connections 2024-06-04T04:57:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.794+0000 7f29157ea700 1 -- 172.21.15.70:0/3428225739 wait complete. 2024-06-04T04:57:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f29157ea700 1 Processor -- start 2024-06-04T04:57:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f29157ea700 1 -- start start 2024-06-04T04:57:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f29157ea700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2910101b30 0x7f291019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f29157ea700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29101024f0 0x7f291019fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f29157ea700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2910109520 0x7f29101998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f29157ea700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f29101135b0 con 0x7f29101024f0 2024-06-04T04:57:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f29157ea700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2910113430 con 0x7f2910109520 2024-06-04T04:57:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f29157ea700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2910113730 con 0x7f2910101b30 2024-06-04T04:57:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f290e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29101024f0 0x7f291019fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f290e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29101024f0 0x7f291019fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45476/0 (socket says 172.21.15.70:45476) 2024-06-04T04:57:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f290e7fc700 1 -- 172.21.15.70:0/2210959641 learned_addr learned my addr 172.21.15.70:0/2210959641 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f290effd700 1 --2- 172.21.15.70:0/2210959641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2910101b30 0x7f291019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f290f7fe700 1 --2- 172.21.15.70:0/2210959641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2910109520 0x7f29101998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f290e7fc700 1 -- 172.21.15.70:0/2210959641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2910101b30 msgr2=0x7f291019f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f290e7fc700 1 --2- 172.21.15.70:0/2210959641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2910101b30 0x7f291019f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f290e7fc700 1 -- 172.21.15.70:0/2210959641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2910109520 msgr2=0x7f29101998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f290e7fc700 1 --2- 172.21.15.70:0/2210959641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2910109520 0x7f29101998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.795+0000 7f290e7fc700 1 -- 172.21.15.70:0/2210959641 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291019a050 con 0x7f29101024f0 2024-06-04T04:57:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.796+0000 7f290e7fc700 1 --2- 172.21.15.70:0/2210959641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29101024f0 0x7f291019fc70 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f290400d950 tx=0x7f290400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.796+0000 7f28f7fff700 1 -- 172.21.15.70:0/2210959641 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2904016070 con 0x7f29101024f0 2024-06-04T04:57:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.796+0000 7f28f7fff700 1 -- 172.21.15.70:0/2210959641 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2904009dd0 con 0x7f29101024f0 2024-06-04T04:57:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.796+0000 7f29157ea700 1 -- 172.21.15.70:0/2210959641 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f291019a340 con 0x7f29101024f0 2024-06-04T04:57:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.796+0000 7f28f7fff700 1 -- 172.21.15.70:0/2210959641 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290400f3c0 con 0x7f29101024f0 2024-06-04T04:57:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.796+0000 7f29157ea700 1 -- 172.21.15.70:0/2210959641 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29101a6600 con 0x7f29101024f0 2024-06-04T04:57:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.797+0000 7f29157ea700 1 -- 172.21.15.70:0/2210959641 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2910104e50 con 0x7f29101024f0 2024-06-04T04:57:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.798+0000 7f28f7fff700 1 -- 172.21.15.70:0/2210959641 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7f2904019070 con 0x7f29101024f0 2024-06-04T04:57:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.799+0000 7f28f7fff700 1 --2- 172.21.15.70:0/2210959641 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f28f80b2a80 0x7f28f80b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.799+0000 7f290effd700 1 --2- 172.21.15.70:0/2210959641 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f28f80b2a80 0x7f28f80b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.799+0000 7f28f7fff700 1 -- 172.21.15.70:0/2210959641 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f29040e23f0 con 0x7f29101024f0 2024-06-04T04:57:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.799+0000 7f290effd700 1 --2- 172.21.15.70:0/2210959641 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f28f80b2a80 0x7f28f80b4f40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2900009990 tx=0x7f290001a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:25.802+0000 7f28f7fff700 1 -- 172.21.15.70:0/2210959641 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f29040a9c60 con 0x7f29101024f0 2024-06-04T04:57:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:25 smithi112 ceph-mon[36453]: pgmap v10: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 0 B/s wr, 14 op/s 2024-06-04T04:57:25.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:25 smithi098 ceph-mon[35569]: pgmap v10: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 0 B/s wr, 14 op/s 2024-06-04T04:57:26.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:26.004+0000 7f29157ea700 1 -- 172.21.15.70:0/2210959641 --> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2910063980 con 0x7f28f80b2a80 2024-06-04T04:57:26.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:26.012+0000 7f28f7fff700 1 -- 172.21.15.70:0/2210959641 <== mgr.24806 v2:172.21.15.70:6800/542036627 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258098 (secure 0 0 0) 0x7f2910063980 con 0x7f28f80b2a80 2024-06-04T04:57:26.018 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:57:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:26.015+0000 7f29157ea700 1 -- 172.21.15.70:0/2210959641 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f28f80b2a80 msgr2=0x7f28f80b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:26.015+0000 7f29157ea700 1 --2- 172.21.15.70:0/2210959641 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f28f80b2a80 0x7f28f80b4f40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2900009990 tx=0x7f290001a040 comp rx=0 tx=0).stop 2024-06-04T04:57:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:26.015+0000 7f29157ea700 1 -- 172.21.15.70:0/2210959641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29101024f0 msgr2=0x7f291019fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:26.015+0000 7f29157ea700 1 --2- 172.21.15.70:0/2210959641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29101024f0 0x7f291019fc70 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f290400d950 tx=0x7f290400dc60 comp rx=0 tx=0).stop 2024-06-04T04:57:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:26.015+0000 7f29157ea700 1 -- 172.21.15.70:0/2210959641 shutdown_connections 2024-06-04T04:57:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:26.015+0000 7f29157ea700 1 --2- 172.21.15.70:0/2210959641 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f28f80b2a80 0x7f28f80b4f40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:26.015+0000 7f29157ea700 1 --2- 172.21.15.70:0/2210959641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2910109520 0x7f29101998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:26.015+0000 7f29157ea700 1 --2- 172.21.15.70:0/2210959641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2910101b30 0x7f291019f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:26.015+0000 7f29157ea700 1 --2- 172.21.15.70:0/2210959641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29101024f0 0x7f291019fc70 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:26.015+0000 7f29157ea700 1 -- 172.21.15.70:0/2210959641 >> 172.21.15.70:0/2210959641 conn(0x7f2910078ab0 msgr2=0x7f2910100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:26.015+0000 7f29157ea700 1 -- 172.21.15.70:0/2210959641 shutdown_connections 2024-06-04T04:57:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:26.015+0000 7f29157ea700 1 -- 172.21.15.70:0/2210959641 wait complete. 2024-06-04T04:57:26.021 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-06-04T04:57:26.772 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-06-04T04:57:25.962526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1196776,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2621,"num_read_kb":1703,"num_write":621,"num_write_kb":4076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2952512,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":962,"ondisk_log_size":962,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356280,"kb_used_data":11244,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585544,"statfs":{"total":1151940427776,"available":1151575597056,"internally_reserved":0,"allocated":11513856,"data_stored":9231828,"data_compressed":312014,"data_compressed_allocated":417792,"data_compressed_original":4005888,"omap_allocated":19138,"internal_metadata":353154366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":435,"num_read_kb":463,"num_write":111,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002635"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466420+0000","last_change":"2024-06-04T04:53:36.383755+0000","last_active":"2024-06-04T04:57:10.466420+0000","last_peered":"2024-06-04T04:57:10.466420+0000","last_clean":"2024-06-04T04:57:10.466420+0000","last_became_active":"2024-06-04T04:53:36.378274+0000","last_became_peered":"2024-06-04T04:53:36.378274+0000","last_unstale":"2024-06-04T04:57:10.466420+0000","last_undegraded":"2024-06-04T04:57:10.466420+0000","last_fullsized":"2024-06-04T04:57:10.466420+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:30:19.685994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466173+0000","last_change":"2024-06-04T04:53:36.372436+0000","last_active":"2024-06-04T04:57:10.466173+0000","last_peered":"2024-06-04T04:57:10.466173+0000","last_clean":"2024-06-04T04:57:10.466173+0000","last_became_active":"2024-06-04T04:53:36.372100+0000","last_became_peered":"2024-06-04T04:53:36.372100+0000","last_unstale":"2024-06-04T04:57:10.466173+0000","last_undegraded":"2024-06-04T04:57:10.466173+0000","last_fullsized":"2024-06-04T04:57:10.466173+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:42:31.646754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466063+0000","last_change":"2024-06-04T04:53:36.366915+0000","last_active":"2024-06-04T04:57:10.466063+0000","last_peered":"2024-06-04T04:57:10.466063+0000","last_clean":"2024-06-04T04:57:10.466063+0000","last_became_active":"2024-06-04T04:53:36.366611+0000","last_became_peered":"2024-06-04T04:53:36.366611+0000","last_unstale":"2024-06-04T04:57:10.466063+0000","last_undegraded":"2024-06-04T04:57:10.466063+0000","last_fullsized":"2024-06-04T04:57:10.466063+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:17:48.775938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466271+0000","last_change":"2024-06-04T04:53:36.374486+0000","last_active":"2024-06-04T04:57:10.466271+0000","last_peered":"2024-06-04T04:57:10.466271+0000","last_clean":"2024-06-04T04:57:10.466271+0000","last_became_active":"2024-06-04T04:53:36.374178+0000","last_became_peered":"2024-06-04T04:53:36.374178+0000","last_unstale":"2024-06-04T04:57:10.466271+0000","last_undegraded":"2024-06-04T04:57:10.466271+0000","last_fullsized":"2024-06-04T04:57:10.466271+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:30:28.287070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965508+0000","last_change":"2024-06-04T04:53:36.376183+0000","last_active":"2024-06-04T04:57:09.965508+0000","last_peered":"2024-06-04T04:57:09.965508+0000","last_clean":"2024-06-04T04:57:09.965508+0000","last_became_active":"2024-06-04T04:53:36.375790+0000","last_became_peered":"2024-06-04T04:53:36.375790+0000","last_unstale":"2024-06-04T04:57:09.965508+0000","last_undegraded":"2024-06-04T04:57:09.965508+0000","last_fullsized":"2024-06-04T04:57:09.965508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:58.697440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946349+0000","last_change":"2024-06-04T04:53:36.370249+0000","last_active":"2024-06-04T04:57:09.946349+0000","last_peered":"2024-06-04T04:57:09.946349+0000","last_clean":"2024-06-04T04:57:09.946349+0000","last_became_active":"2024-06-04T04:53:36.369916+0000","last_became_peered":"2024-06-04T04:53:36.369916+0000","last_unstale":"2024-06-04T04:57:09.946349+0000","last_undegraded":"2024-06-04T04:57:09.946349+0000","last_fullsized":"2024-06-04T04:57:09.946349+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:02:41.739806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213216+0000","last_change":"2024-06-04T04:53:36.384329+0000","last_active":"2024-06-04T04:57:10.213216+0000","last_peered":"2024-06-04T04:57:10.213216+0000","last_clean":"2024-06-04T04:57:10.213216+0000","last_became_active":"2024-06-04T04:53:36.382504+0000","last_became_peered":"2024-06-04T04:53:36.382504+0000","last_unstale":"2024-06-04T04:57:10.213216+0000","last_undegraded":"2024-06-04T04:57:10.213216+0000","last_fullsized":"2024-06-04T04:57:10.213216+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:51:07.604691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942376+0000","last_change":"2024-06-04T04:53:36.372319+0000","last_active":"2024-06-04T04:57:09.942376+0000","last_peered":"2024-06-04T04:57:09.942376+0000","last_clean":"2024-06-04T04:57:09.942376+0000","last_became_active":"2024-06-04T04:53:36.372000+0000","last_became_peered":"2024-06-04T04:53:36.372000+0000","last_unstale":"2024-06-04T04:57:09.942376+0000","last_undegraded":"2024-06-04T04:57:09.942376+0000","last_fullsized":"2024-06-04T04:57:09.942376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:15:56.906996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215880+0000","last_change":"2024-06-04T04:53:36.376051+0000","last_active":"2024-06-04T04:57:10.215880+0000","last_peered":"2024-06-04T04:57:10.215880+0000","last_clean":"2024-06-04T04:57:10.215880+0000","last_became_active":"2024-06-04T04:53:36.375726+0000","last_became_peered":"2024-06-04T04:53:36.375726+0000","last_unstale":"2024-06-04T04:57:10.215880+0000","last_undegraded":"2024-06-04T04:57:10.215880+0000","last_fullsized":"2024-06-04T04:57:10.215880+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:09:08.542764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946736+0000","last_change":"2024-06-04T04:53:36.373266+0000","last_active":"2024-06-04T04:57:09.946736+0000","last_peered":"2024-06-04T04:57:09.946736+0000","last_clean":"2024-06-04T04:57:09.946736+0000","last_became_active":"2024-06-04T04:53:36.372898+0000","last_became_peered":"2024-06-04T04:53:36.372898+0000","last_unstale":"2024-06-04T04:57:09.946736+0000","last_undegraded":"2024-06-04T04:57:09.946736+0000","last_fullsized":"2024-06-04T04:57:09.946736+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:41:43.436781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213610+0000","last_change":"2024-06-04T04:53:36.381344+0000","last_active":"2024-06-04T04:57:10.213610+0000","last_peered":"2024-06-04T04:57:10.213610+0000","last_clean":"2024-06-04T04:57:10.213610+0000","last_became_active":"2024-06-04T04:53:36.380736+0000","last_became_peered":"2024-06-04T04:53:36.380736+0000","last_unstale":"2024-06-04T04:57:10.213610+0000","last_undegraded":"2024-06-04T04:57:10.213610+0000","last_fullsized":"2024-06-04T04:57:10.213610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:23:01.607272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466124+0000","last_change":"2024-06-04T04:53:36.374035+0000","last_active":"2024-06-04T04:57:10.466124+0000","last_peered":"2024-06-04T04:57:10.466124+0000","last_clean":"2024-06-04T04:57:10.466124+0000","last_became_active":"2024-06-04T04:53:36.373667+0000","last_became_peered":"2024-06-04T04:53:36.373667+0000","last_unstale":"2024-06-04T04:57:10.466124+0000","last_undegraded":"2024-06-04T04:57:10.466124+0000","last_fullsized":"2024-06-04T04:57:10.466124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:23:07.880369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215527+0000","last_change":"2024-06-04T04:53:36.369613+0000","last_active":"2024-06-04T04:57:10.215527+0000","last_peered":"2024-06-04T04:57:10.215527+0000","last_clean":"2024-06-04T04:57:10.215527+0000","last_became_active":"2024-06-04T04:53:36.369277+0000","last_became_peered":"2024-06-04T04:53:36.369277+0000","last_unstale":"2024-06-04T04:57:10.215527+0000","last_undegraded":"2024-06-04T04:57:10.215527+0000","last_fullsized":"2024-06-04T04:57:10.215527+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:13:33.756609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.465960+0000","last_change":"2024-06-04T04:53:36.368094+0000","last_active":"2024-06-04T04:57:10.465960+0000","last_peered":"2024-06-04T04:57:10.465960+0000","last_clean":"2024-06-04T04:57:10.465960+0000","last_became_active":"2024-06-04T04:53:36.367717+0000","last_became_peered":"2024-06-04T04:53:36.367717+0000","last_unstale":"2024-06-04T04:57:10.465960+0000","last_undegraded":"2024-06-04T04:57:10.465960+0000","last_fullsized":"2024-06-04T04:57:10.465960+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:39:56.305078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.465864+0000","last_change":"2024-06-04T04:53:36.366765+0000","last_active":"2024-06-04T04:57:10.465864+0000","last_peered":"2024-06-04T04:57:10.465864+0000","last_clean":"2024-06-04T04:57:10.465864+0000","last_became_active":"2024-06-04T04:53:36.366438+0000","last_became_peered":"2024-06-04T04:53:36.366438+0000","last_unstale":"2024-06-04T04:57:10.465864+0000","last_undegraded":"2024-06-04T04:57:10.465864+0000","last_fullsized":"2024-06-04T04:57:10.465864+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:38:37.313926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215176+0000","last_change":"2024-06-04T04:53:36.380088+0000","last_active":"2024-06-04T04:57:10.215176+0000","last_peered":"2024-06-04T04:57:10.215176+0000","last_clean":"2024-06-04T04:57:10.215176+0000","last_became_active":"2024-06-04T04:53:36.379467+0000","last_became_peered":"2024-06-04T04:53:36.379467+0000","last_unstale":"2024-06-04T04:57:10.215176+0000","last_undegraded":"2024-06-04T04:57:10.215176+0000","last_fullsized":"2024-06-04T04:57:10.215176+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:46:53.236419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466550+0000","last_change":"2024-06-04T04:53:36.381126+0000","last_active":"2024-06-04T04:57:10.466550+0000","last_peered":"2024-06-04T04:57:10.466550+0000","last_clean":"2024-06-04T04:57:10.466550+0000","last_became_active":"2024-06-04T04:53:36.378833+0000","last_became_peered":"2024-06-04T04:53:36.378833+0000","last_unstale":"2024-06-04T04:57:10.466550+0000","last_undegraded":"2024-06-04T04:57:10.466550+0000","last_fullsized":"2024-06-04T04:57:10.466550+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:20:28.639273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.048240+0000","last_change":"2024-06-04T04:53:36.369692+0000","last_active":"2024-06-04T04:57:10.048240+0000","last_peered":"2024-06-04T04:57:10.048240+0000","last_clean":"2024-06-04T04:57:10.048240+0000","last_became_active":"2024-06-04T04:53:36.369370+0000","last_became_peered":"2024-06-04T04:53:36.369370+0000","last_unstale":"2024-06-04T04:57:10.048240+0000","last_undegraded":"2024-06-04T04:57:10.048240+0000","last_fullsized":"2024-06-04T04:57:10.048240+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:55.275465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219675+0000","last_change":"2024-06-04T04:53:36.383639+0000","last_active":"2024-06-04T04:57:10.219675+0000","last_peered":"2024-06-04T04:57:10.219675+0000","last_clean":"2024-06-04T04:57:10.219675+0000","last_became_active":"2024-06-04T04:53:36.383303+0000","last_became_peered":"2024-06-04T04:53:36.383303+0000","last_unstale":"2024-06-04T04:57:10.219675+0000","last_undegraded":"2024-06-04T04:57:10.219675+0000","last_fullsized":"2024-06-04T04:57:10.219675+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:26:31.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'15","reported_seq":278,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:24.918272+0000","last_change":"2024-06-04T04:54:19.641520+0000","last_active":"2024-06-04T04:57:24.918272+0000","last_peered":"2024-06-04T04:57:24.918272+0000","last_clean":"2024-06-04T04:57:24.918272+0000","last_became_active":"2024-06-04T04:54:19.632433+0000","last_became_peered":"2024-06-04T04:54:19.632433+0000","last_unstale":"2024-06-04T04:57:24.918272+0000","last_undegraded":"2024-06-04T04:57:24.918272+0000","last_fullsized":"2024-06-04T04:57:24.918272+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:26:05.993919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":208,"num_read_kb":208,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942727+0000","last_change":"2024-06-04T04:53:36.371827+0000","last_active":"2024-06-04T04:57:09.942727+0000","last_peered":"2024-06-04T04:57:09.942727+0000","last_clean":"2024-06-04T04:57:09.942727+0000","last_became_active":"2024-06-04T04:53:36.371509+0000","last_became_peered":"2024-06-04T04:53:36.371509+0000","last_unstale":"2024-06-04T04:57:09.942727+0000","last_undegraded":"2024-06-04T04:57:09.942727+0000","last_fullsized":"2024-06-04T04:57:09.942727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:54:51.589008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.943009+0000","last_change":"2024-06-04T04:53:33.353976+0000","last_active":"2024-06-04T04:57:09.943009+0000","last_peered":"2024-06-04T04:57:09.943009+0000","last_clean":"2024-06-04T04:57:09.943009+0000","last_became_active":"2024-06-04T04:53:33.353586+0000","last_became_peered":"2024-06-04T04:53:33.353586+0000","last_unstale":"2024-06-04T04:57:09.943009+0000","last_undegraded":"2024-06-04T04:57:09.943009+0000","last_fullsized":"2024-06-04T04:57:09.943009+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:17:10.378008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466390+0000","last_change":"2024-06-04T04:53:36.367806+0000","last_active":"2024-06-04T04:57:10.466390+0000","last_peered":"2024-06-04T04:57:10.466390+0000","last_clean":"2024-06-04T04:57:10.466390+0000","last_became_active":"2024-06-04T04:53:36.367484+0000","last_became_peered":"2024-06-04T04:53:36.367484+0000","last_unstale":"2024-06-04T04:57:10.466390+0000","last_undegraded":"2024-06-04T04:57:10.466390+0000","last_fullsized":"2024-06-04T04:57:10.466390+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:25:36.194746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"82'13","reported_seq":43,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965867+0000","last_change":"2024-06-04T04:53:33.348902+0000","last_active":"2024-06-04T04:57:09.965867+0000","last_peered":"2024-06-04T04:57:09.965867+0000","last_clean":"2024-06-04T04:57:09.965867+0000","last_became_active":"2024-06-04T04:53:33.348441+0000","last_became_peered":"2024-06-04T04:53:33.348441+0000","last_unstale":"2024-06-04T04:57:09.965867+0000","last_undegraded":"2024-06-04T04:57:09.965867+0000","last_fullsized":"2024-06-04T04:57:09.965867+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:22:28.428303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215265+0000","last_change":"2024-06-04T04:53:36.376867+0000","last_active":"2024-06-04T04:57:10.215265+0000","last_peered":"2024-06-04T04:57:10.215265+0000","last_clean":"2024-06-04T04:57:10.215265+0000","last_became_active":"2024-06-04T04:53:36.376517+0000","last_became_peered":"2024-06-04T04:53:36.376517+0000","last_unstale":"2024-06-04T04:57:10.215265+0000","last_undegraded":"2024-06-04T04:57:10.215265+0000","last_fullsized":"2024-06-04T04:57:10.215265+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:21:59.494321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965814+0000","last_change":"2024-06-04T04:53:33.347993+0000","last_active":"2024-06-04T04:57:09.965814+0000","last_peered":"2024-06-04T04:57:09.965814+0000","last_clean":"2024-06-04T04:57:09.965814+0000","last_became_active":"2024-06-04T04:53:33.347656+0000","last_became_peered":"2024-06-04T04:53:33.347656+0000","last_unstale":"2024-06-04T04:57:09.965814+0000","last_undegraded":"2024-06-04T04:57:09.965814+0000","last_fullsized":"2024-06-04T04:57:09.965814+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:21:26.806557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":44,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946660+0000","last_change":"2024-06-04T04:54:19.632790+0000","last_active":"2024-06-04T04:57:09.946660+0000","last_peered":"2024-06-04T04:57:09.946660+0000","last_clean":"2024-06-04T04:57:09.946660+0000","last_became_active":"2024-06-04T04:54:19.632338+0000","last_became_peered":"2024-06-04T04:54:19.632338+0000","last_unstale":"2024-06-04T04:57:09.946660+0000","last_undegraded":"2024-06-04T04:57:09.946660+0000","last_fullsized":"2024-06-04T04:57:09.946660+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:44:00.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"81'6","reported_seq":245,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:19.745245+0000","last_change":"2024-06-04T04:53:33.346111+0000","last_active":"2024-06-04T04:57:19.745245+0000","last_peered":"2024-06-04T04:57:19.745245+0000","last_clean":"2024-06-04T04:57:19.745245+0000","last_became_active":"2024-06-04T04:53:33.345773+0000","last_became_peered":"2024-06-04T04:53:33.345773+0000","last_unstale":"2024-06-04T04:57:19.745245+0000","last_undegraded":"2024-06-04T04:57:19.745245+0000","last_fullsized":"2024-06-04T04:57:19.745245+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:18.565583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1115,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":210,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965482+0000","last_change":"2024-06-04T04:53:36.373369+0000","last_active":"2024-06-04T04:57:09.965482+0000","last_peered":"2024-06-04T04:57:09.965482+0000","last_clean":"2024-06-04T04:57:09.965482+0000","last_became_active":"2024-06-04T04:53:36.373001+0000","last_became_peered":"2024-06-04T04:53:36.373001+0000","last_unstale":"2024-06-04T04:57:09.965482+0000","last_undegraded":"2024-06-04T04:57:09.965482+0000","last_fullsized":"2024-06-04T04:57:09.965482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:02:22.012643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946291+0000","last_change":"2024-06-04T04:53:33.344358+0000","last_active":"2024-06-04T04:57:09.946291+0000","last_peered":"2024-06-04T04:57:09.946291+0000","last_clean":"2024-06-04T04:57:09.946291+0000","last_became_active":"2024-06-04T04:53:33.343980+0000","last_became_peered":"2024-06-04T04:53:33.343980+0000","last_unstale":"2024-06-04T04:57:09.946291+0000","last_undegraded":"2024-06-04T04:57:09.946291+0000","last_fullsized":"2024-06-04T04:57:09.946291+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:14.493138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946579+0000","last_change":"2024-06-04T04:53:36.371239+0000","last_active":"2024-06-04T04:57:09.946579+0000","last_peered":"2024-06-04T04:57:09.946579+0000","last_clean":"2024-06-04T04:57:09.946579+0000","last_became_active":"2024-06-04T04:53:36.370885+0000","last_became_peered":"2024-06-04T04:53:36.370885+0000","last_unstale":"2024-06-04T04:57:09.946579+0000","last_undegraded":"2024-06-04T04:57:09.946579+0000","last_fullsized":"2024-06-04T04:57:09.946579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T04:54:29.753796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215252+0000","last_change":"2024-06-04T04:53:33.338512+0000","last_active":"2024-06-04T04:57:10.215252+0000","last_peered":"2024-06-04T04:57:10.215252+0000","last_clean":"2024-06-04T04:57:10.215252+0000","last_became_active":"2024-06-04T04:53:33.338180+0000","last_became_peered":"2024-06-04T04:53:33.338180+0000","last_unstale":"2024-06-04T04:57:10.215252+0000","last_undegraded":"2024-06-04T04:57:10.215252+0000","last_fullsized":"2024-06-04T04:57:10.215252+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:57:13.406479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942451+0000","last_change":"2024-06-04T04:53:36.377316+0000","last_active":"2024-06-04T04:57:09.942451+0000","last_peered":"2024-06-04T04:57:09.942451+0000","last_clean":"2024-06-04T04:57:09.942451+0000","last_became_active":"2024-06-04T04:53:36.376992+0000","last_became_peered":"2024-06-04T04:53:36.376992+0000","last_unstale":"2024-06-04T04:57:09.942451+0000","last_undegraded":"2024-06-04T04:57:09.942451+0000","last_fullsized":"2024-06-04T04:57:09.942451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:33.145631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946281+0000","last_change":"2024-06-04T04:53:33.348305+0000","last_active":"2024-06-04T04:57:09.946281+0000","last_peered":"2024-06-04T04:57:09.946281+0000","last_clean":"2024-06-04T04:57:09.946281+0000","last_became_active":"2024-06-04T04:53:33.347948+0000","last_became_peered":"2024-06-04T04:53:33.347948+0000","last_unstale":"2024-06-04T04:57:09.946281+0000","last_undegraded":"2024-06-04T04:57:09.946281+0000","last_fullsized":"2024-06-04T04:57:09.946281+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:04:13.669274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.047728+0000","last_change":"2024-06-04T04:53:36.367955+0000","last_active":"2024-06-04T04:57:10.047728+0000","last_peered":"2024-06-04T04:57:10.047728+0000","last_clean":"2024-06-04T04:57:10.047728+0000","last_became_active":"2024-06-04T04:53:36.367628+0000","last_became_peered":"2024-06-04T04:53:36.367628+0000","last_unstale":"2024-06-04T04:57:10.047728+0000","last_undegraded":"2024-06-04T04:57:10.047728+0000","last_fullsized":"2024-06-04T04:57:10.047728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:33:26.185498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":458,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:25.753443+0000","last_change":"2024-06-04T04:53:33.349471+0000","last_active":"2024-06-04T04:57:25.753443+0000","last_peered":"2024-06-04T04:57:25.753443+0000","last_clean":"2024-06-04T04:57:25.753443+0000","last_became_active":"2024-06-04T04:53:33.349105+0000","last_became_peered":"2024-06-04T04:53:33.349105+0000","last_unstale":"2024-06-04T04:57:25.753443+0000","last_undegraded":"2024-06-04T04:57:25.753443+0000","last_fullsized":"2024-06-04T04:57:25.753443+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:31:35.584275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":424,"num_read_kb":424,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215768+0000","last_change":"2024-06-04T04:53:36.367249+0000","last_active":"2024-06-04T04:57:10.215768+0000","last_peered":"2024-06-04T04:57:10.215768+0000","last_clean":"2024-06-04T04:57:10.215768+0000","last_became_active":"2024-06-04T04:53:36.366863+0000","last_became_peered":"2024-06-04T04:53:36.366863+0000","last_unstale":"2024-06-04T04:57:10.215768+0000","last_undegraded":"2024-06-04T04:57:10.215768+0000","last_fullsized":"2024-06-04T04:57:10.215768+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:13:17.308321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213781+0000","last_change":"2024-06-04T04:53:33.346778+0000","last_active":"2024-06-04T04:57:10.213781+0000","last_peered":"2024-06-04T04:57:10.213781+0000","last_clean":"2024-06-04T04:57:10.213781+0000","last_became_active":"2024-06-04T04:53:33.346376+0000","last_became_peered":"2024-06-04T04:53:33.346376+0000","last_unstale":"2024-06-04T04:57:10.213781+0000","last_undegraded":"2024-06-04T04:57:10.213781+0000","last_fullsized":"2024-06-04T04:57:10.213781+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:03:16.973822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466320+0000","last_change":"2024-06-04T04:54:19.514491+0000","last_active":"2024-06-04T04:57:10.466320+0000","last_peered":"2024-06-04T04:57:10.466320+0000","last_clean":"2024-06-04T04:57:10.466320+0000","last_became_active":"2024-06-04T04:54:19.514148+0000","last_became_peered":"2024-06-04T04:54:19.514148+0000","last_unstale":"2024-06-04T04:57:10.466320+0000","last_undegraded":"2024-06-04T04:57:10.466320+0000","last_fullsized":"2024-06-04T04:57:10.466320+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:09:59.641041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946792+0000","last_change":"2024-06-04T04:53:33.354989+0000","last_active":"2024-06-04T04:57:09.946792+0000","last_peered":"2024-06-04T04:57:09.946792+0000","last_clean":"2024-06-04T04:57:09.946792+0000","last_became_active":"2024-06-04T04:53:33.354613+0000","last_became_peered":"2024-06-04T04:53:33.354613+0000","last_unstale":"2024-06-04T04:57:09.946792+0000","last_undegraded":"2024-06-04T04:57:09.946792+0000","last_fullsized":"2024-06-04T04:57:09.946792+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:19:59.398763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215492+0000","last_change":"2024-06-04T04:53:36.372216+0000","last_active":"2024-06-04T04:57:10.215492+0000","last_peered":"2024-06-04T04:57:10.215492+0000","last_clean":"2024-06-04T04:57:10.215492+0000","last_became_active":"2024-06-04T04:53:36.371820+0000","last_became_peered":"2024-06-04T04:53:36.371820+0000","last_unstale":"2024-06-04T04:57:10.215492+0000","last_undegraded":"2024-06-04T04:57:10.215492+0000","last_fullsized":"2024-06-04T04:57:10.215492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:55:59.887963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'40","reported_seq":505,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:25.774009+0000","last_change":"2024-06-04T04:53:33.345493+0000","last_active":"2024-06-04T04:57:25.774009+0000","last_peered":"2024-06-04T04:57:25.774009+0000","last_clean":"2024-06-04T04:57:25.774009+0000","last_became_active":"2024-06-04T04:53:33.345102+0000","last_became_peered":"2024-06-04T04:53:33.345102+0000","last_unstale":"2024-06-04T04:57:25.774009+0000","last_undegraded":"2024-06-04T04:57:25.774009+0000","last_fullsized":"2024-06-04T04:57:25.774009+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:50:27.853391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71650,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":437,"num_read_kb":289,"num_write":42,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965646+0000","last_change":"2024-06-04T04:53:36.382070+0000","last_active":"2024-06-04T04:57:09.965646+0000","last_peered":"2024-06-04T04:57:09.965646+0000","last_clean":"2024-06-04T04:57:09.965646+0000","last_became_active":"2024-06-04T04:53:36.379144+0000","last_became_peered":"2024-06-04T04:53:36.379144+0000","last_unstale":"2024-06-04T04:57:09.965646+0000","last_undegraded":"2024-06-04T04:57:09.965646+0000","last_fullsized":"2024-06-04T04:57:09.965646+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:16.702709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219432+0000","last_change":"2024-06-04T04:53:33.344726+0000","last_active":"2024-06-04T04:57:10.219432+0000","last_peered":"2024-06-04T04:57:10.219432+0000","last_clean":"2024-06-04T04:57:10.219432+0000","last_became_active":"2024-06-04T04:53:33.344360+0000","last_became_peered":"2024-06-04T04:53:33.344360+0000","last_unstale":"2024-06-04T04:57:10.219432+0000","last_undegraded":"2024-06-04T04:57:10.219432+0000","last_fullsized":"2024-06-04T04:57:10.219432+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:44:56.784808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215391+0000","last_change":"2024-06-04T04:53:36.379036+0000","last_active":"2024-06-04T04:57:10.215391+0000","last_peered":"2024-06-04T04:57:10.215391+0000","last_clean":"2024-06-04T04:57:10.215391+0000","last_became_active":"2024-06-04T04:53:36.378681+0000","last_became_peered":"2024-06-04T04:53:36.378681+0000","last_unstale":"2024-06-04T04:57:10.215391+0000","last_undegraded":"2024-06-04T04:57:10.215391+0000","last_fullsized":"2024-06-04T04:57:10.215391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:07:49.704731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946436+0000","last_change":"2024-06-04T04:53:33.345915+0000","last_active":"2024-06-04T04:57:09.946436+0000","last_peered":"2024-06-04T04:57:09.946436+0000","last_clean":"2024-06-04T04:57:09.946436+0000","last_became_active":"2024-06-04T04:53:33.345537+0000","last_became_peered":"2024-06-04T04:53:33.345537+0000","last_unstale":"2024-06-04T04:57:09.946436+0000","last_undegraded":"2024-06-04T04:57:09.946436+0000","last_fullsized":"2024-06-04T04:57:09.946436+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:06:44.086968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213503+0000","last_change":"2024-06-04T04:53:36.379094+0000","last_active":"2024-06-04T04:57:10.213503+0000","last_peered":"2024-06-04T04:57:10.213503+0000","last_clean":"2024-06-04T04:57:10.213503+0000","last_became_active":"2024-06-04T04:53:36.378763+0000","last_became_peered":"2024-06-04T04:53:36.378763+0000","last_unstale":"2024-06-04T04:57:10.213503+0000","last_undegraded":"2024-06-04T04:57:10.213503+0000","last_fullsized":"2024-06-04T04:57:10.213503+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:16.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'6","reported_seq":250,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:24.768904+0000","last_change":"2024-06-04T04:53:33.341342+0000","last_active":"2024-06-04T04:57:24.768904+0000","last_peered":"2024-06-04T04:57:24.768904+0000","last_clean":"2024-06-04T04:57:24.768904+0000","last_became_active":"2024-06-04T04:53:33.340862+0000","last_became_peered":"2024-06-04T04:53:33.340862+0000","last_unstale":"2024-06-04T04:57:24.768904+0000","last_undegraded":"2024-06-04T04:57:24.768904+0000","last_fullsized":"2024-06-04T04:57:24.768904+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:26.818063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":216,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213135+0000","last_change":"2024-06-04T04:53:36.383124+0000","last_active":"2024-06-04T04:57:10.213135+0000","last_peered":"2024-06-04T04:57:10.213135+0000","last_clean":"2024-06-04T04:57:10.213135+0000","last_became_active":"2024-06-04T04:53:36.382449+0000","last_became_peered":"2024-06-04T04:53:36.382449+0000","last_unstale":"2024-06-04T04:57:10.213135+0000","last_undegraded":"2024-06-04T04:57:10.213135+0000","last_fullsized":"2024-06-04T04:57:10.213135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:52:40.776003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":47,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965391+0000","last_change":"2024-06-04T04:53:33.354556+0000","last_active":"2024-06-04T04:57:09.965391+0000","last_peered":"2024-06-04T04:57:09.965391+0000","last_clean":"2024-06-04T04:57:09.965391+0000","last_became_active":"2024-06-04T04:53:33.354238+0000","last_became_peered":"2024-06-04T04:53:33.354238+0000","last_unstale":"2024-06-04T04:57:09.965391+0000","last_undegraded":"2024-06-04T04:57:09.965391+0000","last_fullsized":"2024-06-04T04:57:09.965391+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:54.697374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946345+0000","last_change":"2024-06-04T04:53:36.371387+0000","last_active":"2024-06-04T04:57:09.946345+0000","last_peered":"2024-06-04T04:57:09.946345+0000","last_clean":"2024-06-04T04:57:09.946345+0000","last_became_active":"2024-06-04T04:53:36.370852+0000","last_became_peered":"2024-06-04T04:53:36.370852+0000","last_unstale":"2024-06-04T04:57:09.946345+0000","last_undegraded":"2024-06-04T04:57:09.946345+0000","last_fullsized":"2024-06-04T04:57:09.946345+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:44:06.956210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'3","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965694+0000","last_change":"2024-06-04T04:53:33.349982+0000","last_active":"2024-06-04T04:57:09.965694+0000","last_peered":"2024-06-04T04:57:09.965694+0000","last_clean":"2024-06-04T04:57:09.965694+0000","last_became_active":"2024-06-04T04:53:33.349633+0000","last_became_peered":"2024-06-04T04:53:33.349633+0000","last_unstale":"2024-06-04T04:57:09.965694+0000","last_undegraded":"2024-06-04T04:57:09.965694+0000","last_fullsized":"2024-06-04T04:57:09.965694+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:40.107426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466355+0000","last_change":"2024-06-04T04:53:36.382313+0000","last_active":"2024-06-04T04:57:10.466355+0000","last_peered":"2024-06-04T04:57:10.466355+0000","last_clean":"2024-06-04T04:57:10.466355+0000","last_became_active":"2024-06-04T04:53:36.379638+0000","last_became_peered":"2024-06-04T04:53:36.379638+0000","last_unstale":"2024-06-04T04:57:10.466355+0000","last_undegraded":"2024-06-04T04:57:10.466355+0000","last_fullsized":"2024-06-04T04:57:10.466355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:43:42.527262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.048030+0000","last_change":"2024-06-04T04:54:19.513581+0000","last_active":"2024-06-04T04:57:10.048030+0000","last_peered":"2024-06-04T04:57:10.048030+0000","last_clean":"2024-06-04T04:57:10.048030+0000","last_became_active":"2024-06-04T04:54:19.513111+0000","last_became_peered":"2024-06-04T04:54:19.513111+0000","last_unstale":"2024-06-04T04:57:10.048030+0000","last_undegraded":"2024-06-04T04:57:10.048030+0000","last_fullsized":"2024-06-04T04:57:10.048030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:57.028083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946526+0000","last_change":"2024-06-04T04:53:36.369904+0000","last_active":"2024-06-04T04:57:09.946526+0000","last_peered":"2024-06-04T04:57:09.946526+0000","last_clean":"2024-06-04T04:57:09.946526+0000","last_became_active":"2024-06-04T04:53:36.367929+0000","last_became_peered":"2024-06-04T04:53:36.367929+0000","last_unstale":"2024-06-04T04:57:09.946526+0000","last_undegraded":"2024-06-04T04:57:09.946526+0000","last_fullsized":"2024-06-04T04:57:09.946526+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:37:08.721569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"82'10","reported_seq":256,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:24.918778+0000","last_change":"2024-06-04T04:53:33.342723+0000","last_active":"2024-06-04T04:57:24.918778+0000","last_peered":"2024-06-04T04:57:24.918778+0000","last_clean":"2024-06-04T04:57:24.918778+0000","last_became_active":"2024-06-04T04:53:33.342344+0000","last_became_peered":"2024-06-04T04:53:33.342344+0000","last_unstale":"2024-06-04T04:57:24.918778+0000","last_undegraded":"2024-06-04T04:57:24.918778+0000","last_fullsized":"2024-06-04T04:57:24.918778+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:59:50.147517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":217,"num_read_kb":5,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946315+0000","last_change":"2024-06-04T04:54:19.513534+0000","last_active":"2024-06-04T04:57:09.946315+0000","last_peered":"2024-06-04T04:57:09.946315+0000","last_clean":"2024-06-04T04:57:09.946315+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:57:09.946315+0000","last_undegraded":"2024-06-04T04:57:09.946315+0000","last_fullsized":"2024-06-04T04:57:09.946315+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:16:32.240932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946648+0000","last_change":"2024-06-04T04:53:36.371236+0000","last_active":"2024-06-04T04:57:09.946648+0000","last_peered":"2024-06-04T04:57:09.946648+0000","last_clean":"2024-06-04T04:57:09.946648+0000","last_became_active":"2024-06-04T04:53:36.370876+0000","last_became_peered":"2024-06-04T04:53:36.370876+0000","last_unstale":"2024-06-04T04:57:09.946648+0000","last_undegraded":"2024-06-04T04:57:09.946648+0000","last_fullsized":"2024-06-04T04:57:09.946648+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:15:56.751317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":247,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:24.768230+0000","last_change":"2024-06-04T04:53:33.355642+0000","last_active":"2024-06-04T04:57:24.768230+0000","last_peered":"2024-06-04T04:57:24.768230+0000","last_clean":"2024-06-04T04:57:24.768230+0000","last_became_active":"2024-06-04T04:53:33.355187+0000","last_became_peered":"2024-06-04T04:53:33.355187+0000","last_unstale":"2024-06-04T04:57:24.768230+0000","last_undegraded":"2024-06-04T04:57:24.768230+0000","last_fullsized":"2024-06-04T04:57:24.768230+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:32:28.331518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":213,"num_read_kb":213,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.047902+0000","last_change":"2024-06-04T04:54:19.631750+0000","last_active":"2024-06-04T04:57:10.047902+0000","last_peered":"2024-06-04T04:57:10.047902+0000","last_clean":"2024-06-04T04:57:10.047902+0000","last_became_active":"2024-06-04T04:54:19.631485+0000","last_became_peered":"2024-06-04T04:54:19.631485+0000","last_unstale":"2024-06-04T04:57:10.047902+0000","last_undegraded":"2024-06-04T04:57:10.047902+0000","last_fullsized":"2024-06-04T04:57:10.047902+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:29:44.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466037+0000","last_change":"2024-06-04T04:53:33.340653+0000","last_active":"2024-06-04T04:57:10.466037+0000","last_peered":"2024-06-04T04:57:10.466037+0000","last_clean":"2024-06-04T04:57:10.466037+0000","last_became_active":"2024-06-04T04:53:33.340250+0000","last_became_peered":"2024-06-04T04:53:33.340250+0000","last_unstale":"2024-06-04T04:57:10.466037+0000","last_undegraded":"2024-06-04T04:57:10.466037+0000","last_fullsized":"2024-06-04T04:57:10.466037+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:46:20.329514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"82'99","reported_seq":131,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:12.052884+0000","last_change":"2024-06-04T04:52:09.181155+0000","last_active":"2024-06-04T04:57:12.052884+0000","last_peered":"2024-06-04T04:57:12.052884+0000","last_clean":"2024-06-04T04:57:12.052884+0000","last_became_active":"2024-06-04T04:52:09.177002+0000","last_became_peered":"2024-06-04T04:52:09.177002+0000","last_unstale":"2024-06-04T04:57:12.052884+0000","last_undegraded":"2024-06-04T04:57:12.052884+0000","last_fullsized":"2024-06-04T04:57:12.052884+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_clean_scrub_stamp":"2024-06-04T04:48:44.631948+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:46:57.735302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215170+0000","last_change":"2024-06-04T04:53:36.372634+0000","last_active":"2024-06-04T04:57:10.215170+0000","last_peered":"2024-06-04T04:57:10.215170+0000","last_clean":"2024-06-04T04:57:10.215170+0000","last_became_active":"2024-06-04T04:53:36.372290+0000","last_became_peered":"2024-06-04T04:53:36.372290+0000","last_unstale":"2024-06-04T04:57:10.215170+0000","last_undegraded":"2024-06-04T04:57:10.215170+0000","last_fullsized":"2024-06-04T04:57:10.215170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:14:07.460382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213392+0000","last_change":"2024-06-04T04:53:33.348760+0000","last_active":"2024-06-04T04:57:10.213392+0000","last_peered":"2024-06-04T04:57:10.213392+0000","last_clean":"2024-06-04T04:57:10.213392+0000","last_became_active":"2024-06-04T04:53:33.348347+0000","last_became_peered":"2024-06-04T04:53:33.348347+0000","last_unstale":"2024-06-04T04:57:10.213392+0000","last_undegraded":"2024-06-04T04:57:10.213392+0000","last_fullsized":"2024-06-04T04:57:10.213392+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:36:20.641061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965676+0000","last_change":"2024-06-04T04:53:33.354840+0000","last_active":"2024-06-04T04:57:09.965676+0000","last_peered":"2024-06-04T04:57:09.965676+0000","last_clean":"2024-06-04T04:57:09.965676+0000","last_became_active":"2024-06-04T04:53:33.354287+0000","last_became_peered":"2024-06-04T04:53:33.354287+0000","last_unstale":"2024-06-04T04:57:09.965676+0000","last_undegraded":"2024-06-04T04:57:09.965676+0000","last_fullsized":"2024-06-04T04:57:09.965676+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:42.550098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215686+0000","last_change":"2024-06-04T04:53:36.372678+0000","last_active":"2024-06-04T04:57:10.215686+0000","last_peered":"2024-06-04T04:57:10.215686+0000","last_clean":"2024-06-04T04:57:10.215686+0000","last_became_active":"2024-06-04T04:53:36.372312+0000","last_became_peered":"2024-06-04T04:53:36.372312+0000","last_unstale":"2024-06-04T04:57:10.215686+0000","last_undegraded":"2024-06-04T04:57:10.215686+0000","last_fullsized":"2024-06-04T04:57:10.215686+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:34:41.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'104","reported_seq":225,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:23.396837+0000","last_change":"2024-06-04T04:53:33.348440+0000","last_active":"2024-06-04T04:57:23.396837+0000","last_peered":"2024-06-04T04:57:23.396837+0000","last_clean":"2024-06-04T04:57:23.396837+0000","last_became_active":"2024-06-04T04:53:33.348131+0000","last_became_peered":"2024-06-04T04:53:33.348131+0000","last_unstale":"2024-06-04T04:57:23.396837+0000","last_undegraded":"2024-06-04T04:57:23.396837+0000","last_fullsized":"2024-06-04T04:57:23.396837+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:14:11.552286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":190,"num_write_kb":1286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942741+0000","last_change":"2024-06-04T04:53:36.378669+0000","last_active":"2024-06-04T04:57:09.942741+0000","last_peered":"2024-06-04T04:57:09.942741+0000","last_clean":"2024-06-04T04:57:09.942741+0000","last_became_active":"2024-06-04T04:53:36.378233+0000","last_became_peered":"2024-06-04T04:53:36.378233+0000","last_unstale":"2024-06-04T04:57:09.942741+0000","last_undegraded":"2024-06-04T04:57:09.942741+0000","last_fullsized":"2024-06-04T04:57:09.942741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:36:45.870924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466139+0000","last_change":"2024-06-04T04:53:33.345078+0000","last_active":"2024-06-04T04:57:10.466139+0000","last_peered":"2024-06-04T04:57:10.466139+0000","last_clean":"2024-06-04T04:57:10.466139+0000","last_became_active":"2024-06-04T04:53:33.344708+0000","last_became_peered":"2024-06-04T04:53:33.344708+0000","last_unstale":"2024-06-04T04:57:10.466139+0000","last_undegraded":"2024-06-04T04:57:10.466139+0000","last_fullsized":"2024-06-04T04:57:10.466139+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:45:14.010758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466300+0000","last_change":"2024-06-04T04:53:36.364817+0000","last_active":"2024-06-04T04:57:10.466300+0000","last_peered":"2024-06-04T04:57:10.466300+0000","last_clean":"2024-06-04T04:57:10.466300+0000","last_became_active":"2024-06-04T04:53:36.364499+0000","last_became_peered":"2024-06-04T04:53:36.364499+0000","last_unstale":"2024-06-04T04:57:10.466300+0000","last_undegraded":"2024-06-04T04:57:10.466300+0000","last_fullsized":"2024-06-04T04:57:10.466300+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:39:30.518028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215210+0000","last_change":"2024-06-04T04:53:33.354527+0000","last_active":"2024-06-04T04:57:10.215210+0000","last_peered":"2024-06-04T04:57:10.215210+0000","last_clean":"2024-06-04T04:57:10.215210+0000","last_became_active":"2024-06-04T04:53:33.354175+0000","last_became_peered":"2024-06-04T04:53:33.354175+0000","last_unstale":"2024-06-04T04:57:10.215210+0000","last_undegraded":"2024-06-04T04:57:10.215210+0000","last_fullsized":"2024-06-04T04:57:10.215210+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:33:08.125428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965391+0000","last_change":"2024-06-04T04:53:36.373248+0000","last_active":"2024-06-04T04:57:09.965391+0000","last_peered":"2024-06-04T04:57:09.965391+0000","last_clean":"2024-06-04T04:57:09.965391+0000","last_became_active":"2024-06-04T04:53:36.372915+0000","last_became_peered":"2024-06-04T04:53:36.372915+0000","last_unstale":"2024-06-04T04:57:09.965391+0000","last_undegraded":"2024-06-04T04:57:09.965391+0000","last_fullsized":"2024-06-04T04:57:09.965391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:23:11.689349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942495+0000","last_change":"2024-06-04T04:53:33.340427+0000","last_active":"2024-06-04T04:57:09.942495+0000","last_peered":"2024-06-04T04:57:09.942495+0000","last_clean":"2024-06-04T04:57:09.942495+0000","last_became_active":"2024-06-04T04:53:33.340022+0000","last_became_peered":"2024-06-04T04:53:33.340022+0000","last_unstale":"2024-06-04T04:57:09.942495+0000","last_undegraded":"2024-06-04T04:57:09.942495+0000","last_fullsized":"2024-06-04T04:57:09.942495+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:56:16.496471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466096+0000","last_change":"2024-06-04T04:53:36.380114+0000","last_active":"2024-06-04T04:57:10.466096+0000","last_peered":"2024-06-04T04:57:10.466096+0000","last_clean":"2024-06-04T04:57:10.466096+0000","last_became_active":"2024-06-04T04:53:36.379392+0000","last_became_peered":"2024-06-04T04:53:36.379392+0000","last_unstale":"2024-06-04T04:57:10.466096+0000","last_undegraded":"2024-06-04T04:57:10.466096+0000","last_fullsized":"2024-06-04T04:57:10.466096+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:41:48.776439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"81'5","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466122+0000","last_change":"2024-06-04T04:54:19.519755+0000","last_active":"2024-06-04T04:57:10.466122+0000","last_peered":"2024-06-04T04:57:10.466122+0000","last_clean":"2024-06-04T04:57:10.466122+0000","last_became_active":"2024-06-04T04:54:19.513992+0000","last_became_peered":"2024-06-04T04:54:19.513992+0000","last_unstale":"2024-06-04T04:57:10.466122+0000","last_undegraded":"2024-06-04T04:57:10.466122+0000","last_fullsized":"2024-06-04T04:57:10.466122+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:01.951935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942779+0000","last_change":"2024-06-04T04:53:36.377420+0000","last_active":"2024-06-04T04:57:09.942779+0000","last_peered":"2024-06-04T04:57:09.942779+0000","last_clean":"2024-06-04T04:57:09.942779+0000","last_became_active":"2024-06-04T04:53:36.377109+0000","last_became_peered":"2024-06-04T04:53:36.377109+0000","last_unstale":"2024-06-04T04:57:09.942779+0000","last_undegraded":"2024-06-04T04:57:09.942779+0000","last_fullsized":"2024-06-04T04:57:09.942779+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:47:12.102396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965575+0000","last_change":"2024-06-04T04:54:19.645328+0000","last_active":"2024-06-04T04:57:09.965575+0000","last_peered":"2024-06-04T04:57:09.965575+0000","last_clean":"2024-06-04T04:57:09.965575+0000","last_became_active":"2024-06-04T04:54:19.633416+0000","last_became_peered":"2024-06-04T04:54:19.633416+0000","last_unstale":"2024-06-04T04:57:09.965575+0000","last_undegraded":"2024-06-04T04:57:09.965575+0000","last_fullsized":"2024-06-04T04:57:09.965575+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:04:20.494641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213349+0000","last_change":"2024-06-04T04:53:36.381187+0000","last_active":"2024-06-04T04:57:10.213349+0000","last_peered":"2024-06-04T04:57:10.213349+0000","last_clean":"2024-06-04T04:57:10.213349+0000","last_became_active":"2024-06-04T04:53:36.380591+0000","last_became_peered":"2024-06-04T04:53:36.380591+0000","last_unstale":"2024-06-04T04:57:10.213349+0000","last_undegraded":"2024-06-04T04:57:10.213349+0000","last_fullsized":"2024-06-04T04:57:10.213349+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:26:43.122339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'6","reported_seq":244,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:19.799724+0000","last_change":"2024-06-04T04:53:33.355908+0000","last_active":"2024-06-04T04:57:19.799724+0000","last_peered":"2024-06-04T04:57:19.799724+0000","last_clean":"2024-06-04T04:57:19.799724+0000","last_became_active":"2024-06-04T04:53:33.355522+0000","last_became_peered":"2024-06-04T04:53:33.355522+0000","last_unstale":"2024-06-04T04:57:19.799724+0000","last_undegraded":"2024-06-04T04:57:19.799724+0000","last_fullsized":"2024-06-04T04:57:19.799724+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:16:44.639431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":210,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213745+0000","last_change":"2024-06-04T04:53:36.382737+0000","last_active":"2024-06-04T04:57:10.213745+0000","last_peered":"2024-06-04T04:57:10.213745+0000","last_clean":"2024-06-04T04:57:10.213745+0000","last_became_active":"2024-06-04T04:53:36.382353+0000","last_became_peered":"2024-06-04T04:53:36.382353+0000","last_unstale":"2024-06-04T04:57:10.213745+0000","last_undegraded":"2024-06-04T04:57:10.213745+0000","last_fullsized":"2024-06-04T04:57:10.213745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:18:01.476058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'41","reported_seq":78,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.170281+0000","last_change":"2024-06-04T04:53:33.347273+0000","last_active":"2024-06-04T04:57:10.170281+0000","last_peered":"2024-06-04T04:57:10.170281+0000","last_clean":"2024-06-04T04:57:10.170281+0000","last_became_active":"2024-06-04T04:53:33.346915+0000","last_became_peered":"2024-06-04T04:53:33.346915+0000","last_unstale":"2024-06-04T04:57:10.170281+0000","last_undegraded":"2024-06-04T04:57:10.170281+0000","last_fullsized":"2024-06-04T04:57:10.170281+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:17.235484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213559+0000","last_change":"2024-06-04T04:53:36.383454+0000","last_active":"2024-06-04T04:57:10.213559+0000","last_peered":"2024-06-04T04:57:10.213559+0000","last_clean":"2024-06-04T04:57:10.213559+0000","last_became_active":"2024-06-04T04:53:36.383105+0000","last_became_peered":"2024-06-04T04:53:36.383105+0000","last_unstale":"2024-06-04T04:57:10.213559+0000","last_undegraded":"2024-06-04T04:57:10.213559+0000","last_fullsized":"2024-06-04T04:57:10.213559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:04:02.147688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215701+0000","last_change":"2024-06-04T04:53:33.345435+0000","last_active":"2024-06-04T04:57:10.215701+0000","last_peered":"2024-06-04T04:57:10.215701+0000","last_clean":"2024-06-04T04:57:10.215701+0000","last_became_active":"2024-06-04T04:53:33.345077+0000","last_became_peered":"2024-06-04T04:53:33.345077+0000","last_unstale":"2024-06-04T04:57:10.215701+0000","last_undegraded":"2024-06-04T04:57:10.215701+0000","last_fullsized":"2024-06-04T04:57:10.215701+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:20:13.992259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213743+0000","last_change":"2024-06-04T04:53:36.385238+0000","last_active":"2024-06-04T04:57:10.213743+0000","last_peered":"2024-06-04T04:57:10.213743+0000","last_clean":"2024-06-04T04:57:10.213743+0000","last_became_active":"2024-06-04T04:53:36.384899+0000","last_became_peered":"2024-06-04T04:53:36.384899+0000","last_unstale":"2024-06-04T04:57:10.213743+0000","last_undegraded":"2024-06-04T04:57:10.213743+0000","last_fullsized":"2024-06-04T04:57:10.213743+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:09:42.629340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466183+0000","last_change":"2024-06-04T04:53:36.383450+0000","last_active":"2024-06-04T04:57:10.466183+0000","last_peered":"2024-06-04T04:57:10.466183+0000","last_clean":"2024-06-04T04:57:10.466183+0000","last_became_active":"2024-06-04T04:53:36.382678+0000","last_became_peered":"2024-06-04T04:53:36.382678+0000","last_unstale":"2024-06-04T04:57:10.466183+0000","last_undegraded":"2024-06-04T04:57:10.466183+0000","last_fullsized":"2024-06-04T04:57:10.466183+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:36:28.932636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219063+0000","last_change":"2024-06-04T04:54:19.632563+0000","last_active":"2024-06-04T04:57:10.219063+0000","last_peered":"2024-06-04T04:57:10.219063+0000","last_clean":"2024-06-04T04:57:10.219063+0000","last_became_active":"2024-06-04T04:54:19.632070+0000","last_became_peered":"2024-06-04T04:54:19.632070+0000","last_unstale":"2024-06-04T04:57:10.219063+0000","last_undegraded":"2024-06-04T04:57:10.219063+0000","last_fullsized":"2024-06-04T04:57:10.219063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:07:16.552958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219185+0000","last_change":"2024-06-04T04:53:36.381408+0000","last_active":"2024-06-04T04:57:10.219185+0000","last_peered":"2024-06-04T04:57:10.219185+0000","last_clean":"2024-06-04T04:57:10.219185+0000","last_became_active":"2024-06-04T04:53:36.381078+0000","last_became_peered":"2024-06-04T04:53:36.381078+0000","last_unstale":"2024-06-04T04:57:10.219185+0000","last_undegraded":"2024-06-04T04:57:10.219185+0000","last_fullsized":"2024-06-04T04:57:10.219185+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:28:48.988591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466464+0000","last_change":"2024-06-04T04:53:36.360168+0000","last_active":"2024-06-04T04:57:10.466464+0000","last_peered":"2024-06-04T04:57:10.466464+0000","last_clean":"2024-06-04T04:57:10.466464+0000","last_became_active":"2024-06-04T04:53:36.359691+0000","last_became_peered":"2024-06-04T04:53:36.359691+0000","last_unstale":"2024-06-04T04:57:10.466464+0000","last_undegraded":"2024-06-04T04:57:10.466464+0000","last_fullsized":"2024-06-04T04:57:10.466464+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:55:17.418339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215688+0000","last_change":"2024-06-04T04:53:36.363018+0000","last_active":"2024-06-04T04:57:10.215688+0000","last_peered":"2024-06-04T04:57:10.215688+0000","last_clean":"2024-06-04T04:57:10.215688+0000","last_became_active":"2024-06-04T04:53:36.361595+0000","last_became_peered":"2024-06-04T04:53:36.361595+0000","last_unstale":"2024-06-04T04:57:10.215688+0000","last_undegraded":"2024-06-04T04:57:10.215688+0000","last_fullsized":"2024-06-04T04:57:10.215688+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:00:21.377322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466289+0000","last_change":"2024-06-04T04:53:36.375377+0000","last_active":"2024-06-04T04:57:10.466289+0000","last_peered":"2024-06-04T04:57:10.466289+0000","last_clean":"2024-06-04T04:57:10.466289+0000","last_became_active":"2024-06-04T04:53:36.373459+0000","last_became_peered":"2024-06-04T04:53:36.373459+0000","last_unstale":"2024-06-04T04:57:10.466289+0000","last_undegraded":"2024-06-04T04:57:10.466289+0000","last_fullsized":"2024-06-04T04:57:10.466289+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:51:31.686969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219589+0000","last_change":"2024-06-04T04:53:36.373850+0000","last_active":"2024-06-04T04:57:10.219589+0000","last_peered":"2024-06-04T04:57:10.219589+0000","last_clean":"2024-06-04T04:57:10.219589+0000","last_became_active":"2024-06-04T04:53:36.371951+0000","last_became_peered":"2024-06-04T04:53:36.371951+0000","last_unstale":"2024-06-04T04:57:10.219589+0000","last_undegraded":"2024-06-04T04:57:10.219589+0000","last_fullsized":"2024-06-04T04:57:10.219589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:06.452703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942719+0000","last_change":"2024-06-04T04:53:36.374871+0000","last_active":"2024-06-04T04:57:09.942719+0000","last_peered":"2024-06-04T04:57:09.942719+0000","last_clean":"2024-06-04T04:57:09.942719+0000","last_became_active":"2024-06-04T04:53:36.374486+0000","last_became_peered":"2024-06-04T04:53:36.374486+0000","last_unstale":"2024-06-04T04:57:09.942719+0000","last_undegraded":"2024-06-04T04:57:09.942719+0000","last_fullsized":"2024-06-04T04:57:09.942719+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:15:13.337374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213230+0000","last_change":"2024-06-04T04:54:19.513540+0000","last_active":"2024-06-04T04:57:10.213230+0000","last_peered":"2024-06-04T04:57:10.213230+0000","last_clean":"2024-06-04T04:57:10.213230+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:57:10.213230+0000","last_undegraded":"2024-06-04T04:57:10.213230+0000","last_fullsized":"2024-06-04T04:57:10.213230+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:09:27.993585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942434+0000","last_change":"2024-06-04T04:53:36.373055+0000","last_active":"2024-06-04T04:57:09.942434+0000","last_peered":"2024-06-04T04:57:09.942434+0000","last_clean":"2024-06-04T04:57:09.942434+0000","last_became_active":"2024-06-04T04:53:36.372754+0000","last_became_peered":"2024-06-04T04:53:36.372754+0000","last_unstale":"2024-06-04T04:57:09.942434+0000","last_undegraded":"2024-06-04T04:57:09.942434+0000","last_fullsized":"2024-06-04T04:57:09.942434+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:22:27.554200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215415+0000","last_change":"2024-06-04T04:53:36.378897+0000","last_active":"2024-06-04T04:57:10.215415+0000","last_peered":"2024-06-04T04:57:10.215415+0000","last_clean":"2024-06-04T04:57:10.215415+0000","last_became_active":"2024-06-04T04:53:36.378607+0000","last_became_peered":"2024-06-04T04:53:36.378607+0000","last_unstale":"2024-06-04T04:57:10.215415+0000","last_undegraded":"2024-06-04T04:57:10.215415+0000","last_fullsized":"2024-06-04T04:57:10.215415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:03.395491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215502+0000","last_change":"2024-06-04T04:53:36.378240+0000","last_active":"2024-06-04T04:57:10.215502+0000","last_peered":"2024-06-04T04:57:10.215502+0000","last_clean":"2024-06-04T04:57:10.215502+0000","last_became_active":"2024-06-04T04:53:36.377909+0000","last_became_peered":"2024-06-04T04:53:36.377909+0000","last_unstale":"2024-06-04T04:57:10.215502+0000","last_undegraded":"2024-06-04T04:57:10.215502+0000","last_fullsized":"2024-06-04T04:57:10.215502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:15:25.288340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466255+0000","last_change":"2024-06-04T04:53:36.379792+0000","last_active":"2024-06-04T04:57:10.466255+0000","last_peered":"2024-06-04T04:57:10.466255+0000","last_clean":"2024-06-04T04:57:10.466255+0000","last_became_active":"2024-06-04T04:53:36.379438+0000","last_became_peered":"2024-06-04T04:53:36.379438+0000","last_unstale":"2024-06-04T04:57:10.466255+0000","last_undegraded":"2024-06-04T04:57:10.466255+0000","last_fullsized":"2024-06-04T04:57:10.466255+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:23:57.617877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108276,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":606174,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2477,"num_read_kb":1455,"num_write":484,"num_write_kb":1924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":774144,"data_stored":2211786,"data_compressed":99926,"data_compressed_allocated":135168,"data_compressed_original":1941504,"omap_allocated":0,"internal_metadata":0},"log_size":361,"ondisk_log_size":361,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678076,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":892928,"data_stored":439038,"data_compressed":8651,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874305,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1101824,"data_stored":1429949,"data_compressed":61778,"data_compressed_allocated":81920,"data_compressed_original":929792,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037831,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":831691,"data_compressed":27940,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234061,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":831397,"data_compressed":27940,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430288,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":1033773,"data_compressed":44743,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593815,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":901120,"data_stored":439999,"data_compressed":8651,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":978944,"data_stored":513234,"data_compressed":10484,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":921600,"data_stored":440024,"data_compressed":8651,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411433,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":511961,"data_compressed":10484,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":987136,"data_stored":1159465,"data_compressed":43142,"data_compressed_allocated":57344,"data_compressed_original":724992,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953569,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":901120,"data_stored":438546,"data_compressed":8651,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790043,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":1162751,"data_compressed":50899,"data_compressed_allocated":65536,"data_compressed_original":737280,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":75881,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":74608,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":721878,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":134913,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":394044,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":402111,"data_compressed":17035,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-04T04:57:26.777 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-04T04:57:27.755 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:27.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.766+0000 7f57fd4ed700 1 -- 172.21.15.70:0/4195521691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f81035c0 msgr2=0x7f57f8103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:27.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.766+0000 7f57fd4ed700 1 --2- 172.21.15.70:0/4195521691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f81035c0 0x7f57f8103a40 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f57ec009990 tx=0x7f57ec009ca0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.766+0000 7f57fd4ed700 1 -- 172.21.15.70:0/4195521691 shutdown_connections 2024-06-04T04:57:27.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.766+0000 7f57fd4ed700 1 --2- 172.21.15.70:0/4195521691 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57f8103f80 0x7f57f8110460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.766+0000 7f57fd4ed700 1 --2- 172.21.15.70:0/4195521691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f81096d0 0x7f57f8109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.766+0000 7f57fd4ed700 1 --2- 172.21.15.70:0/4195521691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f81035c0 0x7f57f8103a40 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.766+0000 7f57fd4ed700 1 -- 172.21.15.70:0/4195521691 >> 172.21.15.70:0/4195521691 conn(0x7f57f807a6f0 msgr2=0x7f57f807ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:27.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.766+0000 7f57fd4ed700 1 -- 172.21.15.70:0/4195521691 shutdown_connections 2024-06-04T04:57:27.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.766+0000 7f57fd4ed700 1 -- 172.21.15.70:0/4195521691 wait complete. 2024-06-04T04:57:27.769 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:27 smithi070 ceph-mon[32631]: pgmap v11: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 45 op/s 2024-06-04T04:57:27.769 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:27 smithi070 ceph-mon[32631]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:27.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.767+0000 7f57fd4ed700 1 Processor -- start 2024-06-04T04:57:27.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.767+0000 7f57fd4ed700 1 -- start start 2024-06-04T04:57:27.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.768+0000 7f57fd4ed700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f81035c0 0x7f57f8075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.768+0000 7f57fd4ed700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f8103f80 0x7f57f8075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.768+0000 7f57fd4ed700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57f81096d0 0x7f57f80762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.768+0000 7f57fd4ed700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57f8113650 con 0x7f57f8103f80 2024-06-04T04:57:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.768+0000 7f57fd4ed700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81134d0 con 0x7f57f81035c0 2024-06-04T04:57:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.768+0000 7f57fd4ed700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81137d0 con 0x7f57f81096d0 2024-06-04T04:57:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.768+0000 7f57f6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f81035c0 0x7f57f8075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.768+0000 7f57f6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f81035c0 0x7f57f8075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:60398/0 (socket says 172.21.15.70:60398) 2024-06-04T04:57:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.768+0000 7f57f6ffd700 1 -- 172.21.15.70:0/2323328948 learned_addr learned my addr 172.21.15.70:0/2323328948 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.768+0000 7f57f67fc700 1 --2- 172.21.15.70:0/2323328948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f8103f80 0x7f57f8075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.768+0000 7f57f77fe700 1 --2- 172.21.15.70:0/2323328948 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57f81096d0 0x7f57f80762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.768+0000 7f57f6ffd700 1 -- 172.21.15.70:0/2323328948 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57f81096d0 msgr2=0x7f57f80762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.768+0000 7f57f6ffd700 1 --2- 172.21.15.70:0/2323328948 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57f81096d0 0x7f57f80762c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.768+0000 7f57f6ffd700 1 -- 172.21.15.70:0/2323328948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f8103f80 msgr2=0x7f57f8075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.768+0000 7f57f6ffd700 1 --2- 172.21.15.70:0/2323328948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f8103f80 0x7f57f8075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.768+0000 7f57f6ffd700 1 -- 172.21.15.70:0/2323328948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f81a7f30 con 0x7f57f81035c0 2024-06-04T04:57:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.769+0000 7f57f6ffd700 1 --2- 172.21.15.70:0/2323328948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f81035c0 0x7f57f8075840 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57e800c9b0 tx=0x7f57e800ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.769+0000 7f57dffff700 1 -- 172.21.15.70:0/2323328948 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8007b00 con 0x7f57f81035c0 2024-06-04T04:57:27.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.769+0000 7f57fd4ed700 1 -- 172.21.15.70:0/2323328948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f81a8220 con 0x7f57f81035c0 2024-06-04T04:57:27.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.769+0000 7f57dffff700 1 -- 172.21.15.70:0/2323328948 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f57e800ce50 con 0x7f57f81035c0 2024-06-04T04:57:27.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.769+0000 7f57dffff700 1 -- 172.21.15.70:0/2323328948 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8018660 con 0x7f57f81035c0 2024-06-04T04:57:27.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.769+0000 7f57fd4ed700 1 -- 172.21.15.70:0/2323328948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f81a8730 con 0x7f57f81035c0 2024-06-04T04:57:27.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.772+0000 7f57dffff700 1 -- 172.21.15.70:0/2323328948 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7f57e800f450 con 0x7f57f81035c0 2024-06-04T04:57:27.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.772+0000 7f57dffff700 1 --2- 172.21.15.70:0/2323328948 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f57e00b2c80 0x7f57e00b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.773+0000 7f57dffff700 1 -- 172.21.15.70:0/2323328948 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f57e8018400 con 0x7f57f81035c0 2024-06-04T04:57:27.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.773+0000 7f57f67fc700 1 --2- 172.21.15.70:0/2323328948 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f57e00b2c80 0x7f57e00b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.773+0000 7f5807d82b80 1 Processor -- start 2024-06-04T04:57:27.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.773+0000 7f5807d82b80 1 -- start start 2024-06-04T04:57:27.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.773+0000 7f5807d82b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad81860 0x56311ad81c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.773+0000 7f5807d82b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ad85940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.773+0000 7f5807d82b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad85f60 0x56311ad863a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.773+0000 7f57f67fc700 1 --2- 172.21.15.70:0/2323328948 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f57e00b2c80 0x7f57e00b5140 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00b5c0 tx=0x7f57ec005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.773+0000 7f5807d82b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56311ace0c90 con 0x56311ad82210 2024-06-04T04:57:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.773+0000 7f5807d82b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56311acd0830 con 0x56311ad81860 2024-06-04T04:57:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.773+0000 7f5807d82b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56311acd4870 con 0x56311ad85f60 2024-06-04T04:57:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.773+0000 7f57f67fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ad85940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.773+0000 7f57f67fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ad85940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35736/0 (socket says 172.21.15.70:35736) 2024-06-04T04:57:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.774+0000 7f57f67fc700 1 -- 172.21.15.70:0/3866218161 learned_addr learned my addr 172.21.15.70:0/3866218161 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.774+0000 7f57f6ffd700 1 --2- 172.21.15.70:0/3866218161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad81860 0x56311ad81c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.774+0000 7f57f77fe700 1 --2- 172.21.15.70:0/3866218161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad85f60 0x56311ad863a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.774+0000 7f57f67fc700 1 -- 172.21.15.70:0/3866218161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad85f60 msgr2=0x56311ad863a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.774+0000 7f57f67fc700 1 --2- 172.21.15.70:0/3866218161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad85f60 0x56311ad863a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.774+0000 7f57f67fc700 1 -- 172.21.15.70:0/3866218161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad81860 msgr2=0x56311ad81c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.774+0000 7f57f67fc700 1 --2- 172.21.15.70:0/3866218161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad81860 0x56311ad81c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.774+0000 7f57f67fc700 1 -- 172.21.15.70:0/3866218161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56311ac208d0 con 0x56311ad82210 2024-06-04T04:57:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.774+0000 7f57f67fc700 1 --2- 172.21.15.70:0/3866218161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ad85940 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f57ec015000 tx=0x7f57ec01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303baf847c1386e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.774+0000 7f57dd7fa700 1 -- 172.21.15.70:0/3866218161 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ec034070 con 0x56311ad82210 2024-06-04T04:57:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.774+0000 7f57dd7fa700 1 -- 172.21.15.70:0/3866218161 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f57ec02f900 con 0x56311ad82210 2024-06-04T04:57:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.774+0000 7f57dd7fa700 1 -- 172.21.15.70:0/3866218161 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ec00f820 con 0x56311ad82210 2024-06-04T04:57:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.775+0000 7f5807d82b80 1 -- 172.21.15.70:0/3866218161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 msgr2=0x56311ad85940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.775+0000 7f5807d82b80 1 --2- 172.21.15.70:0/3866218161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ad85940 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f57ec015000 tx=0x7f57ec01f000 comp rx=0 tx=0).stop 2024-06-04T04:57:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.775+0000 7f5807d82b80 1 -- 172.21.15.70:0/3866218161 shutdown_connections 2024-06-04T04:57:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.775+0000 7f5807d82b80 1 --2- 172.21.15.70:0/3866218161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad85f60 0x56311ad863a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.775+0000 7f5807d82b80 1 --2- 172.21.15.70:0/3866218161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad81860 0x56311ad81c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.775+0000 7f5807d82b80 1 --2- 172.21.15.70:0/3866218161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ad85940 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.775+0000 7f5807d82b80 1 -- 172.21.15.70:0/3866218161 >> 172.21.15.70:0/3866218161 conn(0x56311acaf080 msgr2=0x56311ad7be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:27.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.775+0000 7f5807d82b80 1 -- 172.21.15.70:0/3866218161 shutdown_connections 2024-06-04T04:57:27.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.775+0000 7f5807d82b80 1 -- 172.21.15.70:0/3866218161 wait complete. 2024-06-04T04:57:27.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.776+0000 7f5807d82b80 1 -- start start 2024-06-04T04:57:27.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.776+0000 7f5807d82b80 1 Processor -- start 2024-06-04T04:57:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.776+0000 7f5807d82b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad81860 0x56311ad8d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.776+0000 7f5807d82b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ad8a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.776+0000 7f5807d82b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad85f60 0x56311ad8ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.776+0000 7f5807d82b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56311abd6fa0 con 0x56311ad82210 2024-06-04T04:57:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.776+0000 7f5807d82b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56311ac727c0 con 0x56311ad85f60 2024-06-04T04:57:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.776+0000 7f5807d82b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56311ad8e0c0 con 0x56311ad81860 2024-06-04T04:57:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.776+0000 7f5807d82b80 10 client.0 _ll_register_callbacks cb 0x56311ac6b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.776+0000 7f57f67fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ad8a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.776+0000 7f57f77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad85f60 0x56311ad8ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.776+0000 7f57f67fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ad8a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35748/0 (socket says 172.21.15.70:35748) 2024-06-04T04:57:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.776+0000 7f57f67fc700 1 -- 172.21.15.70:0/2424527718 learned_addr learned my addr 172.21.15.70:0/2424527718 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.776+0000 7f57f6ffd700 1 --2- 172.21.15.70:0/2424527718 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad81860 0x56311ad8d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.777+0000 7f57f67fc700 1 -- 172.21.15.70:0/2424527718 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad81860 msgr2=0x56311ad8d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.777+0000 7f57f67fc700 1 --2- 172.21.15.70:0/2424527718 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad81860 0x56311ad8d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.777+0000 7f57f67fc700 1 -- 172.21.15.70:0/2424527718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad85f60 msgr2=0x56311ad8ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.777+0000 7f57f67fc700 1 --2- 172.21.15.70:0/2424527718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad85f60 0x56311ad8ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.777+0000 7f57f67fc700 1 -- 172.21.15.70:0/2424527718 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56311ac6f690 con 0x56311ad82210 2024-06-04T04:57:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.777+0000 7f57f77fe700 1 --2- 172.21.15.70:0/2424527718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad85f60 0x56311ad8ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:57:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.777+0000 7f57f67fc700 1 --2- 172.21.15.70:0/2424527718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ad8a6f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0098e0 tx=0x7f57ec02f750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9ca593ee03fe60 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.777+0000 7f57daff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:57:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.777+0000 7f57daff5700 1 -- 172.21.15.70:0/2424527718 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ec03f060 con 0x56311ad82210 2024-06-04T04:57:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.777+0000 7f57daff5700 1 -- 172.21.15.70:0/2424527718 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f57ec02fbf0 con 0x56311ad82210 2024-06-04T04:57:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.777+0000 7f5807d82b80 1 -- 172.21.15.70:0/2424527718 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57ec02f900 con 0x56311ad82210 2024-06-04T04:57:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.777+0000 7f57daff5700 1 -- 172.21.15.70:0/2424527718 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f57ec034040 con 0x56311ad82210 2024-06-04T04:57:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.777+0000 7f5807d82b80 10 client.15417 fetch_fsmap learned FSMap version 29 2024-06-04T04:57:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.777+0000 7f5807d82b80 1 -- 172.21.15.70:0/2424527718 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56311ad98840 con 0x56311ad82210 2024-06-04T04:57:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.778+0000 7f57daff5700 1 -- 172.21.15.70:0/2424527718 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f57ec02f900 con 0x56311ad82210 2024-06-04T04:57:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.778+0000 7f57daff5700 1 -- 172.21.15.70:0/2424527718 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f57ec024420 con 0x56311ad82210 2024-06-04T04:57:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.778+0000 7f5807d82b80 10 client.15417 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:57:27.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.778+0000 7f5807d82b80 10 client.15417 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:57:27.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.778+0000 7f5807d82b80 20 client.15417 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:27.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.778+0000 7f5807d82b80 1 --2- 172.21.15.70:0/2424527718 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56311ad9ad10 0x56311ad9d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.778+0000 7f5807d82b80 4 client.15417 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:57:27.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.778+0000 7f5807d82b80 1 -- 172.21.15.70:0/2424527718 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56311ad4d9e0 con 0x56311ad9ad10 2024-06-04T04:57:27.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.778+0000 7f57f6ffd700 1 --2- 172.21.15.70:0/2424527718 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56311ad9ad10 0x56311ad9d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.779+0000 7f57daff5700 10 client.15417 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:27.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.779+0000 7f57f6ffd700 1 --2- 172.21.15.70:0/2424527718 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56311ad9ad10 0x56311ad9d100 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d56c39e092ea07cd server_cookie=d791f44e003a86b in_seq=0 out_seq=0 2024-06-04T04:57:27.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.779+0000 7f57daff5700 1 -- 172.21.15.70:0/2424527718 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56311ad4d9e0 con 0x56311ad9ad10 2024-06-04T04:57:27.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.779+0000 7f57daff5700 10 client.15417 handle_command_reply: tid=0 2024-06-04T04:57:27.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.779+0000 7f5807d82b80 1 client.15417 shutdown 2024-06-04T04:57:27.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.779+0000 7f5807d82b80 1 -- 172.21.15.70:0/2424527718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 msgr2=0x56311ad8a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:27.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.779+0000 7f5807d82b80 1 --2- 172.21.15.70:0/2424527718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ad8a6f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0098e0 tx=0x7f57ec02f750 comp rx=0 tx=0).stop 2024-06-04T04:57:27.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.780+0000 7f5807d82b80 1 -- 172.21.15.70:0/2424527718 shutdown_connections 2024-06-04T04:57:27.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.780+0000 7f5807d82b80 1 --2- 172.21.15.70:0/2424527718 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56311ad9ad10 0x56311ad9d100 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.780+0000 7f5807d82b80 1 --2- 172.21.15.70:0/2424527718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad85f60 0x56311ad8ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.780+0000 7f5807d82b80 1 --2- 172.21.15.70:0/2424527718 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad81860 0x56311ad8d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.780+0000 7f5807d82b80 1 --2- 172.21.15.70:0/2424527718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ad8a6f0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.780+0000 7f5807d82b80 1 -- 172.21.15.70:0/2424527718 >> 172.21.15.70:0/2424527718 conn(0x56311acaf080 msgr2=0x56311ad87350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:27.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.780+0000 7f57daff5700 0 client.15417 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:27.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.780+0000 7f5807d82b80 1 -- 172.21.15.70:0/2424527718 shutdown_connections 2024-06-04T04:57:27.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.780+0000 7f5807d82b80 1 -- 172.21.15.70:0/2424527718 wait complete. 2024-06-04T04:57:27.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.780+0000 7f5807d82b80 20 client.15417 trim_cache size 0 max 16384 2024-06-04T04:57:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f5807d82b80 1 Processor -- start 2024-06-04T04:57:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f5807d82b80 1 -- start start 2024-06-04T04:57:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f5807d82b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad81860 0x56311ad89350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f5807d82b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad82210 0x56311ad8c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f5807d82b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad85f60 0x56311ad9fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f5807d82b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81afe30 con 0x56311ad85f60 2024-06-04T04:57:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f5807d82b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81b1970 con 0x56311ad82210 2024-06-04T04:57:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f5807d82b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81b0500 con 0x56311ad81860 2024-06-04T04:57:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f57f77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad85f60 0x56311ad9fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f57f77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad85f60 0x56311ad9fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35764/0 (socket says 172.21.15.70:35764) 2024-06-04T04:57:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f57f77fe700 1 -- 172.21.15.70:0/421119036 learned_addr learned my addr 172.21.15.70:0/421119036 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f57f67fc700 1 --2- 172.21.15.70:0/421119036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad82210 0x56311ad8c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f57f6ffd700 1 --2- 172.21.15.70:0/421119036 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad81860 0x56311ad89350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f57f77fe700 1 -- 172.21.15.70:0/421119036 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad81860 msgr2=0x56311ad89350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f57f77fe700 1 --2- 172.21.15.70:0/421119036 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad81860 0x56311ad89350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f57f77fe700 1 -- 172.21.15.70:0/421119036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad82210 msgr2=0x56311ad8c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f57f77fe700 1 --2- 172.21.15.70:0/421119036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad82210 0x56311ad8c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f57f77fe700 1 -- 172.21.15.70:0/421119036 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56311ad8d760 con 0x56311ad85f60 2024-06-04T04:57:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.801+0000 7f57f6ffd700 1 --2- 172.21.15.70:0/421119036 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad81860 0x56311ad89350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.802+0000 7f57f77fe700 1 --2- 172.21.15.70:0/421119036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad85f60 0x56311ad9fd40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f57f0003f80 tx=0x7f57f0004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8794ccd65602651 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.802+0000 7f57dcff9700 1 -- 172.21.15.70:0/421119036 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f00127e0 con 0x56311ad85f60 2024-06-04T04:57:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.802+0000 7f57dcff9700 1 -- 172.21.15.70:0/421119036 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f57f001a070 con 0x56311ad85f60 2024-06-04T04:57:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.802+0000 7f57dcff9700 1 -- 172.21.15.70:0/421119036 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f001fbd0 con 0x56311ad85f60 2024-06-04T04:57:27.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.803+0000 7f5807d82b80 1 -- 172.21.15.70:0/421119036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad85f60 msgr2=0x56311ad9fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.803+0000 7f5807d82b80 1 --2- 172.21.15.70:0/421119036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad85f60 0x56311ad9fd40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f57f0003f80 tx=0x7f57f0004320 comp rx=0 tx=0).stop 2024-06-04T04:57:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.803+0000 7f5807d82b80 1 -- 172.21.15.70:0/421119036 shutdown_connections 2024-06-04T04:57:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.803+0000 7f5807d82b80 1 --2- 172.21.15.70:0/421119036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad85f60 0x56311ad9fd40 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.803+0000 7f5807d82b80 1 --2- 172.21.15.70:0/421119036 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad81860 0x56311ad89350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.803+0000 7f5807d82b80 1 --2- 172.21.15.70:0/421119036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad82210 0x56311ad8c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.803+0000 7f5807d82b80 1 -- 172.21.15.70:0/421119036 >> 172.21.15.70:0/421119036 conn(0x56311acaf080 msgr2=0x56311ad8bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.803+0000 7f5807d82b80 1 -- 172.21.15.70:0/421119036 shutdown_connections 2024-06-04T04:57:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.803+0000 7f5807d82b80 1 -- 172.21.15.70:0/421119036 wait complete. 2024-06-04T04:57:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.803+0000 7f5807d82b80 1 -- start start 2024-06-04T04:57:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.803+0000 7f5807d82b80 1 Processor -- start 2024-06-04T04:57:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.803+0000 7f5807d82b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad81860 0x56311ad9fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.803+0000 7f5807d82b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ada0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.803+0000 7f5807d82b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad85f60 0x56311ad9d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.803+0000 7f5807d82b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81b12a0 con 0x56311ad82210 2024-06-04T04:57:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.803+0000 7f5807d82b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81b0bd0 con 0x56311ad81860 2024-06-04T04:57:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.803+0000 7f5807d82b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81addd0 con 0x56311ad85f60 2024-06-04T04:57:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f5807d82b80 10 client.0 _ll_register_callbacks cb 0x56311ac6b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f57f67fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ada0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f57f67fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ada0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35780/0 (socket says 172.21.15.70:35780) 2024-06-04T04:57:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f57f67fc700 1 -- 172.21.15.70:0/2316692267 learned_addr learned my addr 172.21.15.70:0/2316692267 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f57f6ffd700 1 --2- 172.21.15.70:0/2316692267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad81860 0x56311ad9fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f57f77fe700 1 --2- 172.21.15.70:0/2316692267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad85f60 0x56311ad9d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f57f67fc700 1 -- 172.21.15.70:0/2316692267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad85f60 msgr2=0x56311ad9d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f57f67fc700 1 --2- 172.21.15.70:0/2316692267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad85f60 0x56311ad9d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f57f67fc700 1 -- 172.21.15.70:0/2316692267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad81860 msgr2=0x56311ad9fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f57f67fc700 1 --2- 172.21.15.70:0/2316692267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad81860 0x56311ad9fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f57f67fc700 1 -- 172.21.15.70:0/2316692267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56311ad9da30 con 0x56311ad82210 2024-06-04T04:57:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f57f6ffd700 1 --2- 172.21.15.70:0/2316692267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad81860 0x56311ad9fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f57f67fc700 1 --2- 172.21.15.70:0/2316692267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ada0240 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f57ec009ce0 tx=0x7f57ec02f7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=274b878f023407f1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f57daff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:57:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f57daff5700 1 -- 172.21.15.70:0/2316692267 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ec03f050 con 0x56311ad82210 2024-06-04T04:57:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f57daff5700 1 -- 172.21.15.70:0/2316692267 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f57ec024420 con 0x56311ad82210 2024-06-04T04:57:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f5807d82b80 1 -- 172.21.15.70:0/2316692267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57f001a070 con 0x56311ad82210 2024-06-04T04:57:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f57daff5700 1 -- 172.21.15.70:0/2316692267 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f57ec054050 con 0x56311ad82210 2024-06-04T04:57:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f5807d82b80 10 client.15423 fetch_fsmap learned FSMap version 29 2024-06-04T04:57:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.804+0000 7f5807d82b80 1 -- 172.21.15.70:0/2316692267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56311ae04ed0 con 0x56311ad82210 2024-06-04T04:57:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.805+0000 7f57daff5700 1 -- 172.21.15.70:0/2316692267 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f57f001a070 con 0x56311ad82210 2024-06-04T04:57:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.805+0000 7f57daff5700 1 -- 172.21.15.70:0/2316692267 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f57ec04b070 con 0x56311ad82210 2024-06-04T04:57:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.805+0000 7f5807d82b80 10 client.15423 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:57:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.805+0000 7f5807d82b80 10 client.15423 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:57:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.805+0000 7f5807d82b80 20 client.15423 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.805+0000 7f5807d82b80 1 --2- 172.21.15.70:0/2316692267 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56311ae05580 0x56311ae079a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.805+0000 7f5807d82b80 4 client.15423 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T04:57:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.805+0000 7f5807d82b80 1 -- 172.21.15.70:0/2316692267 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56311ace2be0 con 0x56311ae05580 2024-06-04T04:57:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.805+0000 7f57f6ffd700 1 --2- 172.21.15.70:0/2316692267 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56311ae05580 0x56311ae079a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.805+0000 7f57daff5700 10 client.15423 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.805+0000 7f57f6ffd700 1 --2- 172.21.15.70:0/2316692267 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56311ae05580 0x56311ae079a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33f531e37d688edb server_cookie=818ae0b583242563 in_seq=0 out_seq=0 2024-06-04T04:57:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.806+0000 7f57daff5700 1 -- 172.21.15.70:0/2316692267 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56311ace2be0 con 0x56311ae05580 2024-06-04T04:57:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.806+0000 7f57daff5700 10 client.15423 handle_command_reply: tid=0 2024-06-04T04:57:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.806+0000 7f5807d82b80 1 client.15423 shutdown 2024-06-04T04:57:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.806+0000 7f5807d82b80 1 -- 172.21.15.70:0/2316692267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 msgr2=0x56311ada0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.806+0000 7f5807d82b80 1 --2- 172.21.15.70:0/2316692267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ada0240 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f57ec009ce0 tx=0x7f57ec02f7c0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.806+0000 7f5807d82b80 1 -- 172.21.15.70:0/2316692267 shutdown_connections 2024-06-04T04:57:27.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.806+0000 7f5807d82b80 1 --2- 172.21.15.70:0/2316692267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56311ad85f60 0x56311ad9d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.806+0000 7f5807d82b80 1 --2- 172.21.15.70:0/2316692267 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56311ae05580 0x56311ae079a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.806+0000 7f5807d82b80 1 --2- 172.21.15.70:0/2316692267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56311ad81860 0x56311ad9fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.806+0000 7f5807d82b80 1 --2- 172.21.15.70:0/2316692267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56311ad82210 0x56311ada0240 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.806+0000 7f5807d82b80 1 -- 172.21.15.70:0/2316692267 >> 172.21.15.70:0/2316692267 conn(0x56311acaf080 msgr2=0x56311ad80f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:27.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.806+0000 7f57daff5700 0 client.15423 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:27.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.806+0000 7f5807d82b80 1 -- 172.21.15.70:0/2316692267 shutdown_connections 2024-06-04T04:57:27.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.806+0000 7f5807d82b80 1 -- 172.21.15.70:0/2316692267 wait complete. 2024-06-04T04:57:27.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.806+0000 7f5807d82b80 20 client.15423 trim_cache size 0 max 16384 2024-06-04T04:57:27.814 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:57:27.815 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T04:57:27.815 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T04:57:27.815 INFO:teuthology.orchestra.run.smithi070.stdout: "6b7fb4d9-6a0c-4504-b8bf-cc7a477e26af": { 2024-06-04T04:57:27.815 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T04:57:27.815 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "6b7fb4d9-6a0c-4504-b8bf-cc7a477e26af", 2024-06-04T04:57:27.815 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T04:57:27.815 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T04:57:27.815 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T04:57:27.815 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:57:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.807+0000 7f57fd4ed700 1 -- 172.21.15.70:0/2323328948 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f57e00b2c80 msgr2=0x7f57e00b5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.807+0000 7f57fd4ed700 1 --2- 172.21.15.70:0/2323328948 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f57e00b2c80 0x7f57e00b5140 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00b5c0 tx=0x7f57ec005960 comp rx=0 tx=0).stop 2024-06-04T04:57:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.807+0000 7f57fd4ed700 1 -- 172.21.15.70:0/2323328948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f81035c0 msgr2=0x7f57f8075840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.807+0000 7f57fd4ed700 1 --2- 172.21.15.70:0/2323328948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f81035c0 0x7f57f8075840 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57e800c9b0 tx=0x7f57e800ccc0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.807+0000 7f57fd4ed700 1 -- 172.21.15.70:0/2323328948 shutdown_connections 2024-06-04T04:57:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.807+0000 7f57fd4ed700 1 --2- 172.21.15.70:0/2323328948 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f57e00b2c80 0x7f57e00b5140 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.807+0000 7f57fd4ed700 1 --2- 172.21.15.70:0/2323328948 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57f81096d0 0x7f57f80762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.807+0000 7f57fd4ed700 1 --2- 172.21.15.70:0/2323328948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f81035c0 0x7f57f8075840 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.807+0000 7f57fd4ed700 1 --2- 172.21.15.70:0/2323328948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f8103f80 0x7f57f8075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.807+0000 7f57fd4ed700 1 -- 172.21.15.70:0/2323328948 >> 172.21.15.70:0/2323328948 conn(0x7f57f807a6f0 msgr2=0x7f57f8102150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:27.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.807+0000 7f57fd4ed700 1 -- 172.21.15.70:0/2323328948 shutdown_connections 2024-06-04T04:57:27.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:27.807+0000 7f57fd4ed700 1 -- 172.21.15.70:0/2323328948 wait complete. 2024-06-04T04:57:27.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:27 smithi112 ceph-mon[36453]: pgmap v11: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 45 op/s 2024-06-04T04:57:27.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:27 smithi112 ceph-mon[36453]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:27.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:27 smithi098 ceph-mon[35569]: pgmap v11: 97 pgs: 97 active+clean; 1.1 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 45 op/s 2024-06-04T04:57:27.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:27 smithi098 ceph-mon[35569]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:29.608 INFO:tasks.cephfs.filesystem:scrub status for tag:6b7fb4d9-6a0c-4504-b8bf-cc7a477e26af - {'path': '/', 'tag': '6b7fb4d9-6a0c-4504-b8bf-cc7a477e26af', 'options': 'recursive,force'} 2024-06-04T04:57:29.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:29 smithi070 ceph-mon[32631]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 49 op/s 2024-06-04T04:57:29.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:29 smithi112 ceph-mon[36453]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 49 op/s 2024-06-04T04:57:29.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:29 smithi098 ceph-mon[35569]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 49 op/s 2024-06-04T04:57:30.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.148+0000 7ff41dd27700 1 -- 172.21.15.70:0/3552664095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4181088f0 msgr2=0x7ff418110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:30.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.148+0000 7ff41dd27700 1 --2- 172.21.15.70:0/3552664095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4181088f0 0x7ff418110470 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff41000b0d0 tx=0x7ff41000b3e0 comp rx=0 tx=0).stop 2024-06-04T04:57:30.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.148+0000 7ff41dd27700 1 -- 172.21.15.70:0/3552664095 shutdown_connections 2024-06-04T04:57:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.148+0000 7ff41dd27700 1 --2- 172.21.15.70:0/3552664095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4181088f0 0x7ff418110470 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.148+0000 7ff41dd27700 1 --2- 172.21.15.70:0/3552664095 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4181037f0 0x7ff418103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.148+0000 7ff41dd27700 1 --2- 172.21.15.70:0/3552664095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4181041a0 0x7ff4181081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.148+0000 7ff41dd27700 1 -- 172.21.15.70:0/3552664095 >> 172.21.15.70:0/3552664095 conn(0x7ff41807a6a0 msgr2=0x7ff41807aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.148+0000 7ff41dd27700 1 -- 172.21.15.70:0/3552664095 shutdown_connections 2024-06-04T04:57:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.148+0000 7ff41dd27700 1 -- 172.21.15.70:0/3552664095 wait complete. 2024-06-04T04:57:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.148+0000 7ff41dd27700 1 Processor -- start 2024-06-04T04:57:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.149+0000 7ff41dd27700 1 -- start start 2024-06-04T04:57:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.149+0000 7ff41dd27700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4181037f0 0x7ff41819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.149+0000 7ff41dd27700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4181041a0 0x7ff41819fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.149+0000 7ff41dd27700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4181088f0 0x7ff4181998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.149+0000 7ff41dd27700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff418113590 con 0x7ff4181037f0 2024-06-04T04:57:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.149+0000 7ff41dd27700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff418113410 con 0x7ff4181088f0 2024-06-04T04:57:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.149+0000 7ff41dd27700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff418113710 con 0x7ff4181041a0 2024-06-04T04:57:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.149+0000 7ff4177fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4181037f0 0x7ff41819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.149+0000 7ff4177fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4181037f0 0x7ff41819f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35804/0 (socket says 172.21.15.70:35804) 2024-06-04T04:57:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.149+0000 7ff4177fe700 1 -- 172.21.15.70:0/3263504499 learned_addr learned my addr 172.21.15.70:0/3263504499 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.149+0000 7ff417fff700 1 --2- 172.21.15.70:0/3263504499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4181088f0 0x7ff4181998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.149+0000 7ff416ffd700 1 --2- 172.21.15.70:0/3263504499 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4181041a0 0x7ff41819fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.150+0000 7ff4177fe700 1 -- 172.21.15.70:0/3263504499 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4181041a0 msgr2=0x7ff41819fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.150+0000 7ff4177fe700 1 --2- 172.21.15.70:0/3263504499 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4181041a0 0x7ff41819fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.150+0000 7ff4177fe700 1 -- 172.21.15.70:0/3263504499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4181088f0 msgr2=0x7ff4181998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.150+0000 7ff4177fe700 1 --2- 172.21.15.70:0/3263504499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4181088f0 0x7ff4181998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.150+0000 7ff4177fe700 1 -- 172.21.15.70:0/3263504499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff41819a030 con 0x7ff4181037f0 2024-06-04T04:57:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.150+0000 7ff4177fe700 1 --2- 172.21.15.70:0/3263504499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4181037f0 0x7ff41819f6c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff40800d980 tx=0x7ff40800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.150+0000 7ff414ff9700 1 -- 172.21.15.70:0/3263504499 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff408017070 con 0x7ff4181037f0 2024-06-04T04:57:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.150+0000 7ff41dd27700 1 -- 172.21.15.70:0/3263504499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff41819a320 con 0x7ff4181037f0 2024-06-04T04:57:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.150+0000 7ff414ff9700 1 -- 172.21.15.70:0/3263504499 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff408005020 con 0x7ff4181037f0 2024-06-04T04:57:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.150+0000 7ff414ff9700 1 -- 172.21.15.70:0/3263504499 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff40800f980 con 0x7ff4181037f0 2024-06-04T04:57:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.150+0000 7ff41dd27700 1 -- 172.21.15.70:0/3263504499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4181a6590 con 0x7ff4181037f0 2024-06-04T04:57:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.151+0000 7ff41dd27700 1 -- 172.21.15.70:0/3263504499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff41806a230 con 0x7ff4181037f0 2024-06-04T04:57:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.152+0000 7ff414ff9700 1 -- 172.21.15.70:0/3263504499 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7ff408004a30 con 0x7ff4181037f0 2024-06-04T04:57:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.153+0000 7ff414ff9700 1 --2- 172.21.15.70:0/3263504499 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4000b2b00 0x7ff4000b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:30.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.153+0000 7ff416ffd700 1 --2- 172.21.15.70:0/3263504499 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4000b2b00 0x7ff4000b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:30.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.153+0000 7ff414ff9700 1 -- 172.21.15.70:0/3263504499 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ff4080e29b0 con 0x7ff4181037f0 2024-06-04T04:57:30.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.153+0000 7ff416ffd700 1 --2- 172.21.15.70:0/3263504499 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4000b2b00 0x7ff4000b4fc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff4181030c0 tx=0x7ff40c009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:30.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.157+0000 7ff414ff9700 1 -- 172.21.15.70:0/3263504499 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff408020020 con 0x7ff4181037f0 2024-06-04T04:57:30.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.356+0000 7ff41dd27700 1 -- 172.21.15.70:0/3263504499 --> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff418063980 con 0x7ff4000b2b00 2024-06-04T04:57:30.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.363+0000 7ff414ff9700 1 -- 172.21.15.70:0/3263504499 <== mgr.24806 v2:172.21.15.70:6800/542036627 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258109 (secure 0 0 0) 0x7ff418063980 con 0x7ff4000b2b00 2024-06-04T04:57:30.369 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:57:30.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.366+0000 7ff41dd27700 1 -- 172.21.15.70:0/3263504499 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4000b2b00 msgr2=0x7ff4000b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:30.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.366+0000 7ff41dd27700 1 --2- 172.21.15.70:0/3263504499 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4000b2b00 0x7ff4000b4fc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff4181030c0 tx=0x7ff40c009260 comp rx=0 tx=0).stop 2024-06-04T04:57:30.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.366+0000 7ff41dd27700 1 -- 172.21.15.70:0/3263504499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4181037f0 msgr2=0x7ff41819f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:30.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.366+0000 7ff41dd27700 1 --2- 172.21.15.70:0/3263504499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4181037f0 0x7ff41819f6c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff40800d980 tx=0x7ff40800dc90 comp rx=0 tx=0).stop 2024-06-04T04:57:30.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.366+0000 7ff41dd27700 1 -- 172.21.15.70:0/3263504499 shutdown_connections 2024-06-04T04:57:30.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.366+0000 7ff41dd27700 1 --2- 172.21.15.70:0/3263504499 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4000b2b00 0x7ff4000b4fc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:30.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.366+0000 7ff41dd27700 1 --2- 172.21.15.70:0/3263504499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4181088f0 0x7ff4181998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:30.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.366+0000 7ff41dd27700 1 --2- 172.21.15.70:0/3263504499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4181037f0 0x7ff41819f6c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:30.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.366+0000 7ff41dd27700 1 --2- 172.21.15.70:0/3263504499 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4181041a0 0x7ff41819fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:30.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.366+0000 7ff41dd27700 1 -- 172.21.15.70:0/3263504499 >> 172.21.15.70:0/3263504499 conn(0x7ff41807a6a0 msgr2=0x7ff418101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:30.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.366+0000 7ff41dd27700 1 -- 172.21.15.70:0/3263504499 shutdown_connections 2024-06-04T04:57:30.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:30.366+0000 7ff41dd27700 1 -- 172.21.15.70:0/3263504499 wait complete. 2024-06-04T04:57:30.372 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-06-04T04:57:31.034 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-06-04T04:57:29.963536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1393408,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2654,"num_read_kb":1726,"num_write":648,"num_write_kb":4542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2952512,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":988,"ondisk_log_size":988,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356312,"kb_used_data":11268,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585512,"statfs":{"total":1151940427776,"available":1151575564288,"internally_reserved":0,"allocated":11538432,"data_stored":9428460,"data_compressed":321060,"data_compressed_allocated":430080,"data_compressed_original":4190208,"omap_allocated":19138,"internal_metadata":353154366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":355,"num_read_kb":282,"num_write":126,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002778"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466420+0000","last_change":"2024-06-04T04:53:36.383755+0000","last_active":"2024-06-04T04:57:10.466420+0000","last_peered":"2024-06-04T04:57:10.466420+0000","last_clean":"2024-06-04T04:57:10.466420+0000","last_became_active":"2024-06-04T04:53:36.378274+0000","last_became_peered":"2024-06-04T04:53:36.378274+0000","last_unstale":"2024-06-04T04:57:10.466420+0000","last_undegraded":"2024-06-04T04:57:10.466420+0000","last_fullsized":"2024-06-04T04:57:10.466420+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:30:19.685994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466173+0000","last_change":"2024-06-04T04:53:36.372436+0000","last_active":"2024-06-04T04:57:10.466173+0000","last_peered":"2024-06-04T04:57:10.466173+0000","last_clean":"2024-06-04T04:57:10.466173+0000","last_became_active":"2024-06-04T04:53:36.372100+0000","last_became_peered":"2024-06-04T04:53:36.372100+0000","last_unstale":"2024-06-04T04:57:10.466173+0000","last_undegraded":"2024-06-04T04:57:10.466173+0000","last_fullsized":"2024-06-04T04:57:10.466173+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:42:31.646754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466063+0000","last_change":"2024-06-04T04:53:36.366915+0000","last_active":"2024-06-04T04:57:10.466063+0000","last_peered":"2024-06-04T04:57:10.466063+0000","last_clean":"2024-06-04T04:57:10.466063+0000","last_became_active":"2024-06-04T04:53:36.366611+0000","last_became_peered":"2024-06-04T04:53:36.366611+0000","last_unstale":"2024-06-04T04:57:10.466063+0000","last_undegraded":"2024-06-04T04:57:10.466063+0000","last_fullsized":"2024-06-04T04:57:10.466063+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:17:48.775938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466271+0000","last_change":"2024-06-04T04:53:36.374486+0000","last_active":"2024-06-04T04:57:10.466271+0000","last_peered":"2024-06-04T04:57:10.466271+0000","last_clean":"2024-06-04T04:57:10.466271+0000","last_became_active":"2024-06-04T04:53:36.374178+0000","last_became_peered":"2024-06-04T04:53:36.374178+0000","last_unstale":"2024-06-04T04:57:10.466271+0000","last_undegraded":"2024-06-04T04:57:10.466271+0000","last_fullsized":"2024-06-04T04:57:10.466271+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:30:28.287070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965508+0000","last_change":"2024-06-04T04:53:36.376183+0000","last_active":"2024-06-04T04:57:09.965508+0000","last_peered":"2024-06-04T04:57:09.965508+0000","last_clean":"2024-06-04T04:57:09.965508+0000","last_became_active":"2024-06-04T04:53:36.375790+0000","last_became_peered":"2024-06-04T04:53:36.375790+0000","last_unstale":"2024-06-04T04:57:09.965508+0000","last_undegraded":"2024-06-04T04:57:09.965508+0000","last_fullsized":"2024-06-04T04:57:09.965508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:58.697440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946349+0000","last_change":"2024-06-04T04:53:36.370249+0000","last_active":"2024-06-04T04:57:09.946349+0000","last_peered":"2024-06-04T04:57:09.946349+0000","last_clean":"2024-06-04T04:57:09.946349+0000","last_became_active":"2024-06-04T04:53:36.369916+0000","last_became_peered":"2024-06-04T04:53:36.369916+0000","last_unstale":"2024-06-04T04:57:09.946349+0000","last_undegraded":"2024-06-04T04:57:09.946349+0000","last_fullsized":"2024-06-04T04:57:09.946349+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:02:41.739806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213216+0000","last_change":"2024-06-04T04:53:36.384329+0000","last_active":"2024-06-04T04:57:10.213216+0000","last_peered":"2024-06-04T04:57:10.213216+0000","last_clean":"2024-06-04T04:57:10.213216+0000","last_became_active":"2024-06-04T04:53:36.382504+0000","last_became_peered":"2024-06-04T04:53:36.382504+0000","last_unstale":"2024-06-04T04:57:10.213216+0000","last_undegraded":"2024-06-04T04:57:10.213216+0000","last_fullsized":"2024-06-04T04:57:10.213216+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:51:07.604691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942376+0000","last_change":"2024-06-04T04:53:36.372319+0000","last_active":"2024-06-04T04:57:09.942376+0000","last_peered":"2024-06-04T04:57:09.942376+0000","last_clean":"2024-06-04T04:57:09.942376+0000","last_became_active":"2024-06-04T04:53:36.372000+0000","last_became_peered":"2024-06-04T04:53:36.372000+0000","last_unstale":"2024-06-04T04:57:09.942376+0000","last_undegraded":"2024-06-04T04:57:09.942376+0000","last_fullsized":"2024-06-04T04:57:09.942376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:15:56.906996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215880+0000","last_change":"2024-06-04T04:53:36.376051+0000","last_active":"2024-06-04T04:57:10.215880+0000","last_peered":"2024-06-04T04:57:10.215880+0000","last_clean":"2024-06-04T04:57:10.215880+0000","last_became_active":"2024-06-04T04:53:36.375726+0000","last_became_peered":"2024-06-04T04:53:36.375726+0000","last_unstale":"2024-06-04T04:57:10.215880+0000","last_undegraded":"2024-06-04T04:57:10.215880+0000","last_fullsized":"2024-06-04T04:57:10.215880+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:09:08.542764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946736+0000","last_change":"2024-06-04T04:53:36.373266+0000","last_active":"2024-06-04T04:57:09.946736+0000","last_peered":"2024-06-04T04:57:09.946736+0000","last_clean":"2024-06-04T04:57:09.946736+0000","last_became_active":"2024-06-04T04:53:36.372898+0000","last_became_peered":"2024-06-04T04:53:36.372898+0000","last_unstale":"2024-06-04T04:57:09.946736+0000","last_undegraded":"2024-06-04T04:57:09.946736+0000","last_fullsized":"2024-06-04T04:57:09.946736+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:41:43.436781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213610+0000","last_change":"2024-06-04T04:53:36.381344+0000","last_active":"2024-06-04T04:57:10.213610+0000","last_peered":"2024-06-04T04:57:10.213610+0000","last_clean":"2024-06-04T04:57:10.213610+0000","last_became_active":"2024-06-04T04:53:36.380736+0000","last_became_peered":"2024-06-04T04:53:36.380736+0000","last_unstale":"2024-06-04T04:57:10.213610+0000","last_undegraded":"2024-06-04T04:57:10.213610+0000","last_fullsized":"2024-06-04T04:57:10.213610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:23:01.607272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466124+0000","last_change":"2024-06-04T04:53:36.374035+0000","last_active":"2024-06-04T04:57:10.466124+0000","last_peered":"2024-06-04T04:57:10.466124+0000","last_clean":"2024-06-04T04:57:10.466124+0000","last_became_active":"2024-06-04T04:53:36.373667+0000","last_became_peered":"2024-06-04T04:53:36.373667+0000","last_unstale":"2024-06-04T04:57:10.466124+0000","last_undegraded":"2024-06-04T04:57:10.466124+0000","last_fullsized":"2024-06-04T04:57:10.466124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:23:07.880369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215527+0000","last_change":"2024-06-04T04:53:36.369613+0000","last_active":"2024-06-04T04:57:10.215527+0000","last_peered":"2024-06-04T04:57:10.215527+0000","last_clean":"2024-06-04T04:57:10.215527+0000","last_became_active":"2024-06-04T04:53:36.369277+0000","last_became_peered":"2024-06-04T04:53:36.369277+0000","last_unstale":"2024-06-04T04:57:10.215527+0000","last_undegraded":"2024-06-04T04:57:10.215527+0000","last_fullsized":"2024-06-04T04:57:10.215527+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:13:33.756609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.465960+0000","last_change":"2024-06-04T04:53:36.368094+0000","last_active":"2024-06-04T04:57:10.465960+0000","last_peered":"2024-06-04T04:57:10.465960+0000","last_clean":"2024-06-04T04:57:10.465960+0000","last_became_active":"2024-06-04T04:53:36.367717+0000","last_became_peered":"2024-06-04T04:53:36.367717+0000","last_unstale":"2024-06-04T04:57:10.465960+0000","last_undegraded":"2024-06-04T04:57:10.465960+0000","last_fullsized":"2024-06-04T04:57:10.465960+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:39:56.305078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.465864+0000","last_change":"2024-06-04T04:53:36.366765+0000","last_active":"2024-06-04T04:57:10.465864+0000","last_peered":"2024-06-04T04:57:10.465864+0000","last_clean":"2024-06-04T04:57:10.465864+0000","last_became_active":"2024-06-04T04:53:36.366438+0000","last_became_peered":"2024-06-04T04:53:36.366438+0000","last_unstale":"2024-06-04T04:57:10.465864+0000","last_undegraded":"2024-06-04T04:57:10.465864+0000","last_fullsized":"2024-06-04T04:57:10.465864+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:38:37.313926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215176+0000","last_change":"2024-06-04T04:53:36.380088+0000","last_active":"2024-06-04T04:57:10.215176+0000","last_peered":"2024-06-04T04:57:10.215176+0000","last_clean":"2024-06-04T04:57:10.215176+0000","last_became_active":"2024-06-04T04:53:36.379467+0000","last_became_peered":"2024-06-04T04:53:36.379467+0000","last_unstale":"2024-06-04T04:57:10.215176+0000","last_undegraded":"2024-06-04T04:57:10.215176+0000","last_fullsized":"2024-06-04T04:57:10.215176+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:46:53.236419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466550+0000","last_change":"2024-06-04T04:53:36.381126+0000","last_active":"2024-06-04T04:57:10.466550+0000","last_peered":"2024-06-04T04:57:10.466550+0000","last_clean":"2024-06-04T04:57:10.466550+0000","last_became_active":"2024-06-04T04:53:36.378833+0000","last_became_peered":"2024-06-04T04:53:36.378833+0000","last_unstale":"2024-06-04T04:57:10.466550+0000","last_undegraded":"2024-06-04T04:57:10.466550+0000","last_fullsized":"2024-06-04T04:57:10.466550+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:20:28.639273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.048240+0000","last_change":"2024-06-04T04:53:36.369692+0000","last_active":"2024-06-04T04:57:10.048240+0000","last_peered":"2024-06-04T04:57:10.048240+0000","last_clean":"2024-06-04T04:57:10.048240+0000","last_became_active":"2024-06-04T04:53:36.369370+0000","last_became_peered":"2024-06-04T04:53:36.369370+0000","last_unstale":"2024-06-04T04:57:10.048240+0000","last_undegraded":"2024-06-04T04:57:10.048240+0000","last_fullsized":"2024-06-04T04:57:10.048240+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:55.275465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219675+0000","last_change":"2024-06-04T04:53:36.383639+0000","last_active":"2024-06-04T04:57:10.219675+0000","last_peered":"2024-06-04T04:57:10.219675+0000","last_clean":"2024-06-04T04:57:10.219675+0000","last_became_active":"2024-06-04T04:53:36.383303+0000","last_became_peered":"2024-06-04T04:53:36.383303+0000","last_unstale":"2024-06-04T04:57:10.219675+0000","last_undegraded":"2024-06-04T04:57:10.219675+0000","last_fullsized":"2024-06-04T04:57:10.219675+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:26:31.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'15","reported_seq":278,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:24.918272+0000","last_change":"2024-06-04T04:54:19.641520+0000","last_active":"2024-06-04T04:57:24.918272+0000","last_peered":"2024-06-04T04:57:24.918272+0000","last_clean":"2024-06-04T04:57:24.918272+0000","last_became_active":"2024-06-04T04:54:19.632433+0000","last_became_peered":"2024-06-04T04:54:19.632433+0000","last_unstale":"2024-06-04T04:57:24.918272+0000","last_undegraded":"2024-06-04T04:57:24.918272+0000","last_fullsized":"2024-06-04T04:57:24.918272+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:26:05.993919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":208,"num_read_kb":208,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942727+0000","last_change":"2024-06-04T04:53:36.371827+0000","last_active":"2024-06-04T04:57:09.942727+0000","last_peered":"2024-06-04T04:57:09.942727+0000","last_clean":"2024-06-04T04:57:09.942727+0000","last_became_active":"2024-06-04T04:53:36.371509+0000","last_became_peered":"2024-06-04T04:53:36.371509+0000","last_unstale":"2024-06-04T04:57:09.942727+0000","last_undegraded":"2024-06-04T04:57:09.942727+0000","last_fullsized":"2024-06-04T04:57:09.942727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:54:51.589008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.943009+0000","last_change":"2024-06-04T04:53:33.353976+0000","last_active":"2024-06-04T04:57:09.943009+0000","last_peered":"2024-06-04T04:57:09.943009+0000","last_clean":"2024-06-04T04:57:09.943009+0000","last_became_active":"2024-06-04T04:53:33.353586+0000","last_became_peered":"2024-06-04T04:53:33.353586+0000","last_unstale":"2024-06-04T04:57:09.943009+0000","last_undegraded":"2024-06-04T04:57:09.943009+0000","last_fullsized":"2024-06-04T04:57:09.943009+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:17:10.378008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466390+0000","last_change":"2024-06-04T04:53:36.367806+0000","last_active":"2024-06-04T04:57:10.466390+0000","last_peered":"2024-06-04T04:57:10.466390+0000","last_clean":"2024-06-04T04:57:10.466390+0000","last_became_active":"2024-06-04T04:53:36.367484+0000","last_became_peered":"2024-06-04T04:53:36.367484+0000","last_unstale":"2024-06-04T04:57:10.466390+0000","last_undegraded":"2024-06-04T04:57:10.466390+0000","last_fullsized":"2024-06-04T04:57:10.466390+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:25:36.194746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"82'13","reported_seq":43,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965867+0000","last_change":"2024-06-04T04:53:33.348902+0000","last_active":"2024-06-04T04:57:09.965867+0000","last_peered":"2024-06-04T04:57:09.965867+0000","last_clean":"2024-06-04T04:57:09.965867+0000","last_became_active":"2024-06-04T04:53:33.348441+0000","last_became_peered":"2024-06-04T04:53:33.348441+0000","last_unstale":"2024-06-04T04:57:09.965867+0000","last_undegraded":"2024-06-04T04:57:09.965867+0000","last_fullsized":"2024-06-04T04:57:09.965867+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:22:28.428303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215265+0000","last_change":"2024-06-04T04:53:36.376867+0000","last_active":"2024-06-04T04:57:10.215265+0000","last_peered":"2024-06-04T04:57:10.215265+0000","last_clean":"2024-06-04T04:57:10.215265+0000","last_became_active":"2024-06-04T04:53:36.376517+0000","last_became_peered":"2024-06-04T04:53:36.376517+0000","last_unstale":"2024-06-04T04:57:10.215265+0000","last_undegraded":"2024-06-04T04:57:10.215265+0000","last_fullsized":"2024-06-04T04:57:10.215265+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:21:59.494321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965814+0000","last_change":"2024-06-04T04:53:33.347993+0000","last_active":"2024-06-04T04:57:09.965814+0000","last_peered":"2024-06-04T04:57:09.965814+0000","last_clean":"2024-06-04T04:57:09.965814+0000","last_became_active":"2024-06-04T04:53:33.347656+0000","last_became_peered":"2024-06-04T04:53:33.347656+0000","last_unstale":"2024-06-04T04:57:09.965814+0000","last_undegraded":"2024-06-04T04:57:09.965814+0000","last_fullsized":"2024-06-04T04:57:09.965814+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:21:26.806557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":44,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946660+0000","last_change":"2024-06-04T04:54:19.632790+0000","last_active":"2024-06-04T04:57:09.946660+0000","last_peered":"2024-06-04T04:57:09.946660+0000","last_clean":"2024-06-04T04:57:09.946660+0000","last_became_active":"2024-06-04T04:54:19.632338+0000","last_became_peered":"2024-06-04T04:54:19.632338+0000","last_unstale":"2024-06-04T04:57:09.946660+0000","last_undegraded":"2024-06-04T04:57:09.946660+0000","last_fullsized":"2024-06-04T04:57:09.946660+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:44:00.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"81'6","reported_seq":250,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:24.752835+0000","last_change":"2024-06-04T04:53:33.346111+0000","last_active":"2024-06-04T04:57:24.752835+0000","last_peered":"2024-06-04T04:57:24.752835+0000","last_clean":"2024-06-04T04:57:24.752835+0000","last_became_active":"2024-06-04T04:53:33.345773+0000","last_became_peered":"2024-06-04T04:53:33.345773+0000","last_unstale":"2024-06-04T04:57:24.752835+0000","last_undegraded":"2024-06-04T04:57:24.752835+0000","last_fullsized":"2024-06-04T04:57:24.752835+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:18.565583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1115,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":215,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965482+0000","last_change":"2024-06-04T04:53:36.373369+0000","last_active":"2024-06-04T04:57:09.965482+0000","last_peered":"2024-06-04T04:57:09.965482+0000","last_clean":"2024-06-04T04:57:09.965482+0000","last_became_active":"2024-06-04T04:53:36.373001+0000","last_became_peered":"2024-06-04T04:53:36.373001+0000","last_unstale":"2024-06-04T04:57:09.965482+0000","last_undegraded":"2024-06-04T04:57:09.965482+0000","last_fullsized":"2024-06-04T04:57:09.965482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:02:22.012643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946291+0000","last_change":"2024-06-04T04:53:33.344358+0000","last_active":"2024-06-04T04:57:09.946291+0000","last_peered":"2024-06-04T04:57:09.946291+0000","last_clean":"2024-06-04T04:57:09.946291+0000","last_became_active":"2024-06-04T04:53:33.343980+0000","last_became_peered":"2024-06-04T04:53:33.343980+0000","last_unstale":"2024-06-04T04:57:09.946291+0000","last_undegraded":"2024-06-04T04:57:09.946291+0000","last_fullsized":"2024-06-04T04:57:09.946291+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:14.493138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946579+0000","last_change":"2024-06-04T04:53:36.371239+0000","last_active":"2024-06-04T04:57:09.946579+0000","last_peered":"2024-06-04T04:57:09.946579+0000","last_clean":"2024-06-04T04:57:09.946579+0000","last_became_active":"2024-06-04T04:53:36.370885+0000","last_became_peered":"2024-06-04T04:53:36.370885+0000","last_unstale":"2024-06-04T04:57:09.946579+0000","last_undegraded":"2024-06-04T04:57:09.946579+0000","last_fullsized":"2024-06-04T04:57:09.946579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T04:54:29.753796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215252+0000","last_change":"2024-06-04T04:53:33.338512+0000","last_active":"2024-06-04T04:57:10.215252+0000","last_peered":"2024-06-04T04:57:10.215252+0000","last_clean":"2024-06-04T04:57:10.215252+0000","last_became_active":"2024-06-04T04:53:33.338180+0000","last_became_peered":"2024-06-04T04:53:33.338180+0000","last_unstale":"2024-06-04T04:57:10.215252+0000","last_undegraded":"2024-06-04T04:57:10.215252+0000","last_fullsized":"2024-06-04T04:57:10.215252+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:57:13.406479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942451+0000","last_change":"2024-06-04T04:53:36.377316+0000","last_active":"2024-06-04T04:57:09.942451+0000","last_peered":"2024-06-04T04:57:09.942451+0000","last_clean":"2024-06-04T04:57:09.942451+0000","last_became_active":"2024-06-04T04:53:36.376992+0000","last_became_peered":"2024-06-04T04:53:36.376992+0000","last_unstale":"2024-06-04T04:57:09.942451+0000","last_undegraded":"2024-06-04T04:57:09.942451+0000","last_fullsized":"2024-06-04T04:57:09.942451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:33.145631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946281+0000","last_change":"2024-06-04T04:53:33.348305+0000","last_active":"2024-06-04T04:57:09.946281+0000","last_peered":"2024-06-04T04:57:09.946281+0000","last_clean":"2024-06-04T04:57:09.946281+0000","last_became_active":"2024-06-04T04:53:33.347948+0000","last_became_peered":"2024-06-04T04:53:33.347948+0000","last_unstale":"2024-06-04T04:57:09.946281+0000","last_undegraded":"2024-06-04T04:57:09.946281+0000","last_fullsized":"2024-06-04T04:57:09.946281+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:04:13.669274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.047728+0000","last_change":"2024-06-04T04:53:36.367955+0000","last_active":"2024-06-04T04:57:10.047728+0000","last_peered":"2024-06-04T04:57:10.047728+0000","last_clean":"2024-06-04T04:57:10.047728+0000","last_became_active":"2024-06-04T04:53:36.367628+0000","last_became_peered":"2024-06-04T04:53:36.367628+0000","last_unstale":"2024-06-04T04:57:10.047728+0000","last_undegraded":"2024-06-04T04:57:10.047728+0000","last_fullsized":"2024-06-04T04:57:10.047728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:33:26.185498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":458,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:25.753443+0000","last_change":"2024-06-04T04:53:33.349471+0000","last_active":"2024-06-04T04:57:25.753443+0000","last_peered":"2024-06-04T04:57:25.753443+0000","last_clean":"2024-06-04T04:57:25.753443+0000","last_became_active":"2024-06-04T04:53:33.349105+0000","last_became_peered":"2024-06-04T04:53:33.349105+0000","last_unstale":"2024-06-04T04:57:25.753443+0000","last_undegraded":"2024-06-04T04:57:25.753443+0000","last_fullsized":"2024-06-04T04:57:25.753443+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:31:35.584275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":424,"num_read_kb":424,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215768+0000","last_change":"2024-06-04T04:53:36.367249+0000","last_active":"2024-06-04T04:57:10.215768+0000","last_peered":"2024-06-04T04:57:10.215768+0000","last_clean":"2024-06-04T04:57:10.215768+0000","last_became_active":"2024-06-04T04:53:36.366863+0000","last_became_peered":"2024-06-04T04:53:36.366863+0000","last_unstale":"2024-06-04T04:57:10.215768+0000","last_undegraded":"2024-06-04T04:57:10.215768+0000","last_fullsized":"2024-06-04T04:57:10.215768+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:13:17.308321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213781+0000","last_change":"2024-06-04T04:53:33.346778+0000","last_active":"2024-06-04T04:57:10.213781+0000","last_peered":"2024-06-04T04:57:10.213781+0000","last_clean":"2024-06-04T04:57:10.213781+0000","last_became_active":"2024-06-04T04:53:33.346376+0000","last_became_peered":"2024-06-04T04:53:33.346376+0000","last_unstale":"2024-06-04T04:57:10.213781+0000","last_undegraded":"2024-06-04T04:57:10.213781+0000","last_fullsized":"2024-06-04T04:57:10.213781+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:03:16.973822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466320+0000","last_change":"2024-06-04T04:54:19.514491+0000","last_active":"2024-06-04T04:57:10.466320+0000","last_peered":"2024-06-04T04:57:10.466320+0000","last_clean":"2024-06-04T04:57:10.466320+0000","last_became_active":"2024-06-04T04:54:19.514148+0000","last_became_peered":"2024-06-04T04:54:19.514148+0000","last_unstale":"2024-06-04T04:57:10.466320+0000","last_undegraded":"2024-06-04T04:57:10.466320+0000","last_fullsized":"2024-06-04T04:57:10.466320+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:09:59.641041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946792+0000","last_change":"2024-06-04T04:53:33.354989+0000","last_active":"2024-06-04T04:57:09.946792+0000","last_peered":"2024-06-04T04:57:09.946792+0000","last_clean":"2024-06-04T04:57:09.946792+0000","last_became_active":"2024-06-04T04:53:33.354613+0000","last_became_peered":"2024-06-04T04:53:33.354613+0000","last_unstale":"2024-06-04T04:57:09.946792+0000","last_undegraded":"2024-06-04T04:57:09.946792+0000","last_fullsized":"2024-06-04T04:57:09.946792+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:19:59.398763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215492+0000","last_change":"2024-06-04T04:53:36.372216+0000","last_active":"2024-06-04T04:57:10.215492+0000","last_peered":"2024-06-04T04:57:10.215492+0000","last_clean":"2024-06-04T04:57:10.215492+0000","last_became_active":"2024-06-04T04:53:36.371820+0000","last_became_peered":"2024-06-04T04:53:36.371820+0000","last_unstale":"2024-06-04T04:57:10.215492+0000","last_undegraded":"2024-06-04T04:57:10.215492+0000","last_fullsized":"2024-06-04T04:57:10.215492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:55:59.887963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'40","reported_seq":505,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:25.774009+0000","last_change":"2024-06-04T04:53:33.345493+0000","last_active":"2024-06-04T04:57:25.774009+0000","last_peered":"2024-06-04T04:57:25.774009+0000","last_clean":"2024-06-04T04:57:25.774009+0000","last_became_active":"2024-06-04T04:53:33.345102+0000","last_became_peered":"2024-06-04T04:53:33.345102+0000","last_unstale":"2024-06-04T04:57:25.774009+0000","last_undegraded":"2024-06-04T04:57:25.774009+0000","last_fullsized":"2024-06-04T04:57:25.774009+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:50:27.853391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71650,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":437,"num_read_kb":289,"num_write":42,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965646+0000","last_change":"2024-06-04T04:53:36.382070+0000","last_active":"2024-06-04T04:57:09.965646+0000","last_peered":"2024-06-04T04:57:09.965646+0000","last_clean":"2024-06-04T04:57:09.965646+0000","last_became_active":"2024-06-04T04:53:36.379144+0000","last_became_peered":"2024-06-04T04:53:36.379144+0000","last_unstale":"2024-06-04T04:57:09.965646+0000","last_undegraded":"2024-06-04T04:57:09.965646+0000","last_fullsized":"2024-06-04T04:57:09.965646+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:16.702709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219432+0000","last_change":"2024-06-04T04:53:33.344726+0000","last_active":"2024-06-04T04:57:10.219432+0000","last_peered":"2024-06-04T04:57:10.219432+0000","last_clean":"2024-06-04T04:57:10.219432+0000","last_became_active":"2024-06-04T04:53:33.344360+0000","last_became_peered":"2024-06-04T04:53:33.344360+0000","last_unstale":"2024-06-04T04:57:10.219432+0000","last_undegraded":"2024-06-04T04:57:10.219432+0000","last_fullsized":"2024-06-04T04:57:10.219432+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:44:56.784808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215391+0000","last_change":"2024-06-04T04:53:36.379036+0000","last_active":"2024-06-04T04:57:10.215391+0000","last_peered":"2024-06-04T04:57:10.215391+0000","last_clean":"2024-06-04T04:57:10.215391+0000","last_became_active":"2024-06-04T04:53:36.378681+0000","last_became_peered":"2024-06-04T04:53:36.378681+0000","last_unstale":"2024-06-04T04:57:10.215391+0000","last_undegraded":"2024-06-04T04:57:10.215391+0000","last_fullsized":"2024-06-04T04:57:10.215391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:07:49.704731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946436+0000","last_change":"2024-06-04T04:53:33.345915+0000","last_active":"2024-06-04T04:57:09.946436+0000","last_peered":"2024-06-04T04:57:09.946436+0000","last_clean":"2024-06-04T04:57:09.946436+0000","last_became_active":"2024-06-04T04:53:33.345537+0000","last_became_peered":"2024-06-04T04:53:33.345537+0000","last_unstale":"2024-06-04T04:57:09.946436+0000","last_undegraded":"2024-06-04T04:57:09.946436+0000","last_fullsized":"2024-06-04T04:57:09.946436+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:06:44.086968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213503+0000","last_change":"2024-06-04T04:53:36.379094+0000","last_active":"2024-06-04T04:57:10.213503+0000","last_peered":"2024-06-04T04:57:10.213503+0000","last_clean":"2024-06-04T04:57:10.213503+0000","last_became_active":"2024-06-04T04:53:36.378763+0000","last_became_peered":"2024-06-04T04:53:36.378763+0000","last_unstale":"2024-06-04T04:57:10.213503+0000","last_undegraded":"2024-06-04T04:57:10.213503+0000","last_fullsized":"2024-06-04T04:57:10.213503+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:16.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'6","reported_seq":250,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:24.768904+0000","last_change":"2024-06-04T04:53:33.341342+0000","last_active":"2024-06-04T04:57:24.768904+0000","last_peered":"2024-06-04T04:57:24.768904+0000","last_clean":"2024-06-04T04:57:24.768904+0000","last_became_active":"2024-06-04T04:53:33.340862+0000","last_became_peered":"2024-06-04T04:53:33.340862+0000","last_unstale":"2024-06-04T04:57:24.768904+0000","last_undegraded":"2024-06-04T04:57:24.768904+0000","last_fullsized":"2024-06-04T04:57:24.768904+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:26.818063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":216,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213135+0000","last_change":"2024-06-04T04:53:36.383124+0000","last_active":"2024-06-04T04:57:10.213135+0000","last_peered":"2024-06-04T04:57:10.213135+0000","last_clean":"2024-06-04T04:57:10.213135+0000","last_became_active":"2024-06-04T04:53:36.382449+0000","last_became_peered":"2024-06-04T04:53:36.382449+0000","last_unstale":"2024-06-04T04:57:10.213135+0000","last_undegraded":"2024-06-04T04:57:10.213135+0000","last_fullsized":"2024-06-04T04:57:10.213135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:52:40.776003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":47,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965391+0000","last_change":"2024-06-04T04:53:33.354556+0000","last_active":"2024-06-04T04:57:09.965391+0000","last_peered":"2024-06-04T04:57:09.965391+0000","last_clean":"2024-06-04T04:57:09.965391+0000","last_became_active":"2024-06-04T04:53:33.354238+0000","last_became_peered":"2024-06-04T04:53:33.354238+0000","last_unstale":"2024-06-04T04:57:09.965391+0000","last_undegraded":"2024-06-04T04:57:09.965391+0000","last_fullsized":"2024-06-04T04:57:09.965391+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:54.697374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946345+0000","last_change":"2024-06-04T04:53:36.371387+0000","last_active":"2024-06-04T04:57:09.946345+0000","last_peered":"2024-06-04T04:57:09.946345+0000","last_clean":"2024-06-04T04:57:09.946345+0000","last_became_active":"2024-06-04T04:53:36.370852+0000","last_became_peered":"2024-06-04T04:53:36.370852+0000","last_unstale":"2024-06-04T04:57:09.946345+0000","last_undegraded":"2024-06-04T04:57:09.946345+0000","last_fullsized":"2024-06-04T04:57:09.946345+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:44:06.956210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'3","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965694+0000","last_change":"2024-06-04T04:53:33.349982+0000","last_active":"2024-06-04T04:57:09.965694+0000","last_peered":"2024-06-04T04:57:09.965694+0000","last_clean":"2024-06-04T04:57:09.965694+0000","last_became_active":"2024-06-04T04:53:33.349633+0000","last_became_peered":"2024-06-04T04:53:33.349633+0000","last_unstale":"2024-06-04T04:57:09.965694+0000","last_undegraded":"2024-06-04T04:57:09.965694+0000","last_fullsized":"2024-06-04T04:57:09.965694+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:40.107426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466355+0000","last_change":"2024-06-04T04:53:36.382313+0000","last_active":"2024-06-04T04:57:10.466355+0000","last_peered":"2024-06-04T04:57:10.466355+0000","last_clean":"2024-06-04T04:57:10.466355+0000","last_became_active":"2024-06-04T04:53:36.379638+0000","last_became_peered":"2024-06-04T04:53:36.379638+0000","last_unstale":"2024-06-04T04:57:10.466355+0000","last_undegraded":"2024-06-04T04:57:10.466355+0000","last_fullsized":"2024-06-04T04:57:10.466355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:43:42.527262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.048030+0000","last_change":"2024-06-04T04:54:19.513581+0000","last_active":"2024-06-04T04:57:10.048030+0000","last_peered":"2024-06-04T04:57:10.048030+0000","last_clean":"2024-06-04T04:57:10.048030+0000","last_became_active":"2024-06-04T04:54:19.513111+0000","last_became_peered":"2024-06-04T04:54:19.513111+0000","last_unstale":"2024-06-04T04:57:10.048030+0000","last_undegraded":"2024-06-04T04:57:10.048030+0000","last_fullsized":"2024-06-04T04:57:10.048030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:57.028083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946526+0000","last_change":"2024-06-04T04:53:36.369904+0000","last_active":"2024-06-04T04:57:09.946526+0000","last_peered":"2024-06-04T04:57:09.946526+0000","last_clean":"2024-06-04T04:57:09.946526+0000","last_became_active":"2024-06-04T04:53:36.367929+0000","last_became_peered":"2024-06-04T04:53:36.367929+0000","last_unstale":"2024-06-04T04:57:09.946526+0000","last_undegraded":"2024-06-04T04:57:09.946526+0000","last_fullsized":"2024-06-04T04:57:09.946526+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:37:08.721569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"82'10","reported_seq":256,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:24.918778+0000","last_change":"2024-06-04T04:53:33.342723+0000","last_active":"2024-06-04T04:57:24.918778+0000","last_peered":"2024-06-04T04:57:24.918778+0000","last_clean":"2024-06-04T04:57:24.918778+0000","last_became_active":"2024-06-04T04:53:33.342344+0000","last_became_peered":"2024-06-04T04:53:33.342344+0000","last_unstale":"2024-06-04T04:57:24.918778+0000","last_undegraded":"2024-06-04T04:57:24.918778+0000","last_fullsized":"2024-06-04T04:57:24.918778+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:59:50.147517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":217,"num_read_kb":5,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946315+0000","last_change":"2024-06-04T04:54:19.513534+0000","last_active":"2024-06-04T04:57:09.946315+0000","last_peered":"2024-06-04T04:57:09.946315+0000","last_clean":"2024-06-04T04:57:09.946315+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:57:09.946315+0000","last_undegraded":"2024-06-04T04:57:09.946315+0000","last_fullsized":"2024-06-04T04:57:09.946315+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:16:32.240932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946648+0000","last_change":"2024-06-04T04:53:36.371236+0000","last_active":"2024-06-04T04:57:09.946648+0000","last_peered":"2024-06-04T04:57:09.946648+0000","last_clean":"2024-06-04T04:57:09.946648+0000","last_became_active":"2024-06-04T04:53:36.370876+0000","last_became_peered":"2024-06-04T04:53:36.370876+0000","last_unstale":"2024-06-04T04:57:09.946648+0000","last_undegraded":"2024-06-04T04:57:09.946648+0000","last_fullsized":"2024-06-04T04:57:09.946648+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:15:56.751317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":247,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:24.768230+0000","last_change":"2024-06-04T04:53:33.355642+0000","last_active":"2024-06-04T04:57:24.768230+0000","last_peered":"2024-06-04T04:57:24.768230+0000","last_clean":"2024-06-04T04:57:24.768230+0000","last_became_active":"2024-06-04T04:53:33.355187+0000","last_became_peered":"2024-06-04T04:53:33.355187+0000","last_unstale":"2024-06-04T04:57:24.768230+0000","last_undegraded":"2024-06-04T04:57:24.768230+0000","last_fullsized":"2024-06-04T04:57:24.768230+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:32:28.331518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":213,"num_read_kb":213,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.047902+0000","last_change":"2024-06-04T04:54:19.631750+0000","last_active":"2024-06-04T04:57:10.047902+0000","last_peered":"2024-06-04T04:57:10.047902+0000","last_clean":"2024-06-04T04:57:10.047902+0000","last_became_active":"2024-06-04T04:54:19.631485+0000","last_became_peered":"2024-06-04T04:54:19.631485+0000","last_unstale":"2024-06-04T04:57:10.047902+0000","last_undegraded":"2024-06-04T04:57:10.047902+0000","last_fullsized":"2024-06-04T04:57:10.047902+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:29:44.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466037+0000","last_change":"2024-06-04T04:53:33.340653+0000","last_active":"2024-06-04T04:57:10.466037+0000","last_peered":"2024-06-04T04:57:10.466037+0000","last_clean":"2024-06-04T04:57:10.466037+0000","last_became_active":"2024-06-04T04:53:33.340250+0000","last_became_peered":"2024-06-04T04:53:33.340250+0000","last_unstale":"2024-06-04T04:57:10.466037+0000","last_undegraded":"2024-06-04T04:57:10.466037+0000","last_fullsized":"2024-06-04T04:57:10.466037+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:46:20.329514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"82'99","reported_seq":131,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:12.052884+0000","last_change":"2024-06-04T04:52:09.181155+0000","last_active":"2024-06-04T04:57:12.052884+0000","last_peered":"2024-06-04T04:57:12.052884+0000","last_clean":"2024-06-04T04:57:12.052884+0000","last_became_active":"2024-06-04T04:52:09.177002+0000","last_became_peered":"2024-06-04T04:52:09.177002+0000","last_unstale":"2024-06-04T04:57:12.052884+0000","last_undegraded":"2024-06-04T04:57:12.052884+0000","last_fullsized":"2024-06-04T04:57:12.052884+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_clean_scrub_stamp":"2024-06-04T04:48:44.631948+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:46:57.735302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215170+0000","last_change":"2024-06-04T04:53:36.372634+0000","last_active":"2024-06-04T04:57:10.215170+0000","last_peered":"2024-06-04T04:57:10.215170+0000","last_clean":"2024-06-04T04:57:10.215170+0000","last_became_active":"2024-06-04T04:53:36.372290+0000","last_became_peered":"2024-06-04T04:53:36.372290+0000","last_unstale":"2024-06-04T04:57:10.215170+0000","last_undegraded":"2024-06-04T04:57:10.215170+0000","last_fullsized":"2024-06-04T04:57:10.215170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:14:07.460382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213392+0000","last_change":"2024-06-04T04:53:33.348760+0000","last_active":"2024-06-04T04:57:10.213392+0000","last_peered":"2024-06-04T04:57:10.213392+0000","last_clean":"2024-06-04T04:57:10.213392+0000","last_became_active":"2024-06-04T04:53:33.348347+0000","last_became_peered":"2024-06-04T04:53:33.348347+0000","last_unstale":"2024-06-04T04:57:10.213392+0000","last_undegraded":"2024-06-04T04:57:10.213392+0000","last_fullsized":"2024-06-04T04:57:10.213392+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:36:20.641061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965676+0000","last_change":"2024-06-04T04:53:33.354840+0000","last_active":"2024-06-04T04:57:09.965676+0000","last_peered":"2024-06-04T04:57:09.965676+0000","last_clean":"2024-06-04T04:57:09.965676+0000","last_became_active":"2024-06-04T04:53:33.354287+0000","last_became_peered":"2024-06-04T04:53:33.354287+0000","last_unstale":"2024-06-04T04:57:09.965676+0000","last_undegraded":"2024-06-04T04:57:09.965676+0000","last_fullsized":"2024-06-04T04:57:09.965676+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:42.550098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215686+0000","last_change":"2024-06-04T04:53:36.372678+0000","last_active":"2024-06-04T04:57:10.215686+0000","last_peered":"2024-06-04T04:57:10.215686+0000","last_clean":"2024-06-04T04:57:10.215686+0000","last_became_active":"2024-06-04T04:53:36.372312+0000","last_became_peered":"2024-06-04T04:53:36.372312+0000","last_unstale":"2024-06-04T04:57:10.215686+0000","last_undegraded":"2024-06-04T04:57:10.215686+0000","last_fullsized":"2024-06-04T04:57:10.215686+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:34:41.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'104","reported_seq":225,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:23.396837+0000","last_change":"2024-06-04T04:53:33.348440+0000","last_active":"2024-06-04T04:57:23.396837+0000","last_peered":"2024-06-04T04:57:23.396837+0000","last_clean":"2024-06-04T04:57:23.396837+0000","last_became_active":"2024-06-04T04:53:33.348131+0000","last_became_peered":"2024-06-04T04:53:33.348131+0000","last_unstale":"2024-06-04T04:57:23.396837+0000","last_undegraded":"2024-06-04T04:57:23.396837+0000","last_fullsized":"2024-06-04T04:57:23.396837+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:14:11.552286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":190,"num_write_kb":1286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942741+0000","last_change":"2024-06-04T04:53:36.378669+0000","last_active":"2024-06-04T04:57:09.942741+0000","last_peered":"2024-06-04T04:57:09.942741+0000","last_clean":"2024-06-04T04:57:09.942741+0000","last_became_active":"2024-06-04T04:53:36.378233+0000","last_became_peered":"2024-06-04T04:53:36.378233+0000","last_unstale":"2024-06-04T04:57:09.942741+0000","last_undegraded":"2024-06-04T04:57:09.942741+0000","last_fullsized":"2024-06-04T04:57:09.942741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:36:45.870924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466139+0000","last_change":"2024-06-04T04:53:33.345078+0000","last_active":"2024-06-04T04:57:10.466139+0000","last_peered":"2024-06-04T04:57:10.466139+0000","last_clean":"2024-06-04T04:57:10.466139+0000","last_became_active":"2024-06-04T04:53:33.344708+0000","last_became_peered":"2024-06-04T04:53:33.344708+0000","last_unstale":"2024-06-04T04:57:10.466139+0000","last_undegraded":"2024-06-04T04:57:10.466139+0000","last_fullsized":"2024-06-04T04:57:10.466139+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:45:14.010758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466300+0000","last_change":"2024-06-04T04:53:36.364817+0000","last_active":"2024-06-04T04:57:10.466300+0000","last_peered":"2024-06-04T04:57:10.466300+0000","last_clean":"2024-06-04T04:57:10.466300+0000","last_became_active":"2024-06-04T04:53:36.364499+0000","last_became_peered":"2024-06-04T04:53:36.364499+0000","last_unstale":"2024-06-04T04:57:10.466300+0000","last_undegraded":"2024-06-04T04:57:10.466300+0000","last_fullsized":"2024-06-04T04:57:10.466300+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:39:30.518028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215210+0000","last_change":"2024-06-04T04:53:33.354527+0000","last_active":"2024-06-04T04:57:10.215210+0000","last_peered":"2024-06-04T04:57:10.215210+0000","last_clean":"2024-06-04T04:57:10.215210+0000","last_became_active":"2024-06-04T04:53:33.354175+0000","last_became_peered":"2024-06-04T04:53:33.354175+0000","last_unstale":"2024-06-04T04:57:10.215210+0000","last_undegraded":"2024-06-04T04:57:10.215210+0000","last_fullsized":"2024-06-04T04:57:10.215210+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:33:08.125428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965391+0000","last_change":"2024-06-04T04:53:36.373248+0000","last_active":"2024-06-04T04:57:09.965391+0000","last_peered":"2024-06-04T04:57:09.965391+0000","last_clean":"2024-06-04T04:57:09.965391+0000","last_became_active":"2024-06-04T04:53:36.372915+0000","last_became_peered":"2024-06-04T04:53:36.372915+0000","last_unstale":"2024-06-04T04:57:09.965391+0000","last_undegraded":"2024-06-04T04:57:09.965391+0000","last_fullsized":"2024-06-04T04:57:09.965391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:23:11.689349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942495+0000","last_change":"2024-06-04T04:53:33.340427+0000","last_active":"2024-06-04T04:57:09.942495+0000","last_peered":"2024-06-04T04:57:09.942495+0000","last_clean":"2024-06-04T04:57:09.942495+0000","last_became_active":"2024-06-04T04:53:33.340022+0000","last_became_peered":"2024-06-04T04:53:33.340022+0000","last_unstale":"2024-06-04T04:57:09.942495+0000","last_undegraded":"2024-06-04T04:57:09.942495+0000","last_fullsized":"2024-06-04T04:57:09.942495+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:56:16.496471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466096+0000","last_change":"2024-06-04T04:53:36.380114+0000","last_active":"2024-06-04T04:57:10.466096+0000","last_peered":"2024-06-04T04:57:10.466096+0000","last_clean":"2024-06-04T04:57:10.466096+0000","last_became_active":"2024-06-04T04:53:36.379392+0000","last_became_peered":"2024-06-04T04:53:36.379392+0000","last_unstale":"2024-06-04T04:57:10.466096+0000","last_undegraded":"2024-06-04T04:57:10.466096+0000","last_fullsized":"2024-06-04T04:57:10.466096+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:41:48.776439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"81'5","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466122+0000","last_change":"2024-06-04T04:54:19.519755+0000","last_active":"2024-06-04T04:57:10.466122+0000","last_peered":"2024-06-04T04:57:10.466122+0000","last_clean":"2024-06-04T04:57:10.466122+0000","last_became_active":"2024-06-04T04:54:19.513992+0000","last_became_peered":"2024-06-04T04:54:19.513992+0000","last_unstale":"2024-06-04T04:57:10.466122+0000","last_undegraded":"2024-06-04T04:57:10.466122+0000","last_fullsized":"2024-06-04T04:57:10.466122+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:01.951935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942779+0000","last_change":"2024-06-04T04:53:36.377420+0000","last_active":"2024-06-04T04:57:09.942779+0000","last_peered":"2024-06-04T04:57:09.942779+0000","last_clean":"2024-06-04T04:57:09.942779+0000","last_became_active":"2024-06-04T04:53:36.377109+0000","last_became_peered":"2024-06-04T04:53:36.377109+0000","last_unstale":"2024-06-04T04:57:09.942779+0000","last_undegraded":"2024-06-04T04:57:09.942779+0000","last_fullsized":"2024-06-04T04:57:09.942779+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:47:12.102396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965575+0000","last_change":"2024-06-04T04:54:19.645328+0000","last_active":"2024-06-04T04:57:09.965575+0000","last_peered":"2024-06-04T04:57:09.965575+0000","last_clean":"2024-06-04T04:57:09.965575+0000","last_became_active":"2024-06-04T04:54:19.633416+0000","last_became_peered":"2024-06-04T04:54:19.633416+0000","last_unstale":"2024-06-04T04:57:09.965575+0000","last_undegraded":"2024-06-04T04:57:09.965575+0000","last_fullsized":"2024-06-04T04:57:09.965575+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:04:20.494641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213349+0000","last_change":"2024-06-04T04:53:36.381187+0000","last_active":"2024-06-04T04:57:10.213349+0000","last_peered":"2024-06-04T04:57:10.213349+0000","last_clean":"2024-06-04T04:57:10.213349+0000","last_became_active":"2024-06-04T04:53:36.380591+0000","last_became_peered":"2024-06-04T04:53:36.380591+0000","last_unstale":"2024-06-04T04:57:10.213349+0000","last_undegraded":"2024-06-04T04:57:10.213349+0000","last_fullsized":"2024-06-04T04:57:10.213349+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:26:43.122339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'6","reported_seq":249,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:24.807178+0000","last_change":"2024-06-04T04:53:33.355908+0000","last_active":"2024-06-04T04:57:24.807178+0000","last_peered":"2024-06-04T04:57:24.807178+0000","last_clean":"2024-06-04T04:57:24.807178+0000","last_became_active":"2024-06-04T04:53:33.355522+0000","last_became_peered":"2024-06-04T04:53:33.355522+0000","last_unstale":"2024-06-04T04:57:24.807178+0000","last_undegraded":"2024-06-04T04:57:24.807178+0000","last_fullsized":"2024-06-04T04:57:24.807178+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:16:44.639431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":215,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213745+0000","last_change":"2024-06-04T04:53:36.382737+0000","last_active":"2024-06-04T04:57:10.213745+0000","last_peered":"2024-06-04T04:57:10.213745+0000","last_clean":"2024-06-04T04:57:10.213745+0000","last_became_active":"2024-06-04T04:53:36.382353+0000","last_became_peered":"2024-06-04T04:53:36.382353+0000","last_unstale":"2024-06-04T04:57:10.213745+0000","last_undegraded":"2024-06-04T04:57:10.213745+0000","last_fullsized":"2024-06-04T04:57:10.213745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:18:01.476058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'67","reported_seq":127,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:23.044873+0000","last_change":"2024-06-04T04:53:33.347273+0000","last_active":"2024-06-04T04:57:23.044873+0000","last_peered":"2024-06-04T04:57:23.044873+0000","last_clean":"2024-06-04T04:57:23.044873+0000","last_became_active":"2024-06-04T04:53:33.346915+0000","last_became_peered":"2024-06-04T04:53:33.346915+0000","last_unstale":"2024-06-04T04:57:23.044873+0000","last_undegraded":"2024-06-04T04:57:23.044873+0000","last_fullsized":"2024-06-04T04:57:23.044873+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:17.235484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213559+0000","last_change":"2024-06-04T04:53:36.383454+0000","last_active":"2024-06-04T04:57:10.213559+0000","last_peered":"2024-06-04T04:57:10.213559+0000","last_clean":"2024-06-04T04:57:10.213559+0000","last_became_active":"2024-06-04T04:53:36.383105+0000","last_became_peered":"2024-06-04T04:53:36.383105+0000","last_unstale":"2024-06-04T04:57:10.213559+0000","last_undegraded":"2024-06-04T04:57:10.213559+0000","last_fullsized":"2024-06-04T04:57:10.213559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:04:02.147688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215701+0000","last_change":"2024-06-04T04:53:33.345435+0000","last_active":"2024-06-04T04:57:10.215701+0000","last_peered":"2024-06-04T04:57:10.215701+0000","last_clean":"2024-06-04T04:57:10.215701+0000","last_became_active":"2024-06-04T04:53:33.345077+0000","last_became_peered":"2024-06-04T04:53:33.345077+0000","last_unstale":"2024-06-04T04:57:10.215701+0000","last_undegraded":"2024-06-04T04:57:10.215701+0000","last_fullsized":"2024-06-04T04:57:10.215701+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:20:13.992259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213743+0000","last_change":"2024-06-04T04:53:36.385238+0000","last_active":"2024-06-04T04:57:10.213743+0000","last_peered":"2024-06-04T04:57:10.213743+0000","last_clean":"2024-06-04T04:57:10.213743+0000","last_became_active":"2024-06-04T04:53:36.384899+0000","last_became_peered":"2024-06-04T04:53:36.384899+0000","last_unstale":"2024-06-04T04:57:10.213743+0000","last_undegraded":"2024-06-04T04:57:10.213743+0000","last_fullsized":"2024-06-04T04:57:10.213743+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:09:42.629340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466183+0000","last_change":"2024-06-04T04:53:36.383450+0000","last_active":"2024-06-04T04:57:10.466183+0000","last_peered":"2024-06-04T04:57:10.466183+0000","last_clean":"2024-06-04T04:57:10.466183+0000","last_became_active":"2024-06-04T04:53:36.382678+0000","last_became_peered":"2024-06-04T04:53:36.382678+0000","last_unstale":"2024-06-04T04:57:10.466183+0000","last_undegraded":"2024-06-04T04:57:10.466183+0000","last_fullsized":"2024-06-04T04:57:10.466183+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:36:28.932636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219063+0000","last_change":"2024-06-04T04:54:19.632563+0000","last_active":"2024-06-04T04:57:10.219063+0000","last_peered":"2024-06-04T04:57:10.219063+0000","last_clean":"2024-06-04T04:57:10.219063+0000","last_became_active":"2024-06-04T04:54:19.632070+0000","last_became_peered":"2024-06-04T04:54:19.632070+0000","last_unstale":"2024-06-04T04:57:10.219063+0000","last_undegraded":"2024-06-04T04:57:10.219063+0000","last_fullsized":"2024-06-04T04:57:10.219063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:07:16.552958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219185+0000","last_change":"2024-06-04T04:53:36.381408+0000","last_active":"2024-06-04T04:57:10.219185+0000","last_peered":"2024-06-04T04:57:10.219185+0000","last_clean":"2024-06-04T04:57:10.219185+0000","last_became_active":"2024-06-04T04:53:36.381078+0000","last_became_peered":"2024-06-04T04:53:36.381078+0000","last_unstale":"2024-06-04T04:57:10.219185+0000","last_undegraded":"2024-06-04T04:57:10.219185+0000","last_fullsized":"2024-06-04T04:57:10.219185+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:28:48.988591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466464+0000","last_change":"2024-06-04T04:53:36.360168+0000","last_active":"2024-06-04T04:57:10.466464+0000","last_peered":"2024-06-04T04:57:10.466464+0000","last_clean":"2024-06-04T04:57:10.466464+0000","last_became_active":"2024-06-04T04:53:36.359691+0000","last_became_peered":"2024-06-04T04:53:36.359691+0000","last_unstale":"2024-06-04T04:57:10.466464+0000","last_undegraded":"2024-06-04T04:57:10.466464+0000","last_fullsized":"2024-06-04T04:57:10.466464+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:55:17.418339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215688+0000","last_change":"2024-06-04T04:53:36.363018+0000","last_active":"2024-06-04T04:57:10.215688+0000","last_peered":"2024-06-04T04:57:10.215688+0000","last_clean":"2024-06-04T04:57:10.215688+0000","last_became_active":"2024-06-04T04:53:36.361595+0000","last_became_peered":"2024-06-04T04:53:36.361595+0000","last_unstale":"2024-06-04T04:57:10.215688+0000","last_undegraded":"2024-06-04T04:57:10.215688+0000","last_fullsized":"2024-06-04T04:57:10.215688+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:00:21.377322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466289+0000","last_change":"2024-06-04T04:53:36.375377+0000","last_active":"2024-06-04T04:57:10.466289+0000","last_peered":"2024-06-04T04:57:10.466289+0000","last_clean":"2024-06-04T04:57:10.466289+0000","last_became_active":"2024-06-04T04:53:36.373459+0000","last_became_peered":"2024-06-04T04:53:36.373459+0000","last_unstale":"2024-06-04T04:57:10.466289+0000","last_undegraded":"2024-06-04T04:57:10.466289+0000","last_fullsized":"2024-06-04T04:57:10.466289+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:51:31.686969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219589+0000","last_change":"2024-06-04T04:53:36.373850+0000","last_active":"2024-06-04T04:57:10.219589+0000","last_peered":"2024-06-04T04:57:10.219589+0000","last_clean":"2024-06-04T04:57:10.219589+0000","last_became_active":"2024-06-04T04:53:36.371951+0000","last_became_peered":"2024-06-04T04:53:36.371951+0000","last_unstale":"2024-06-04T04:57:10.219589+0000","last_undegraded":"2024-06-04T04:57:10.219589+0000","last_fullsized":"2024-06-04T04:57:10.219589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:06.452703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942719+0000","last_change":"2024-06-04T04:53:36.374871+0000","last_active":"2024-06-04T04:57:09.942719+0000","last_peered":"2024-06-04T04:57:09.942719+0000","last_clean":"2024-06-04T04:57:09.942719+0000","last_became_active":"2024-06-04T04:53:36.374486+0000","last_became_peered":"2024-06-04T04:53:36.374486+0000","last_unstale":"2024-06-04T04:57:09.942719+0000","last_undegraded":"2024-06-04T04:57:09.942719+0000","last_fullsized":"2024-06-04T04:57:09.942719+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:15:13.337374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213230+0000","last_change":"2024-06-04T04:54:19.513540+0000","last_active":"2024-06-04T04:57:10.213230+0000","last_peered":"2024-06-04T04:57:10.213230+0000","last_clean":"2024-06-04T04:57:10.213230+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:57:10.213230+0000","last_undegraded":"2024-06-04T04:57:10.213230+0000","last_fullsized":"2024-06-04T04:57:10.213230+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:09:27.993585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942434+0000","last_change":"2024-06-04T04:53:36.373055+0000","last_active":"2024-06-04T04:57:09.942434+0000","last_peered":"2024-06-04T04:57:09.942434+0000","last_clean":"2024-06-04T04:57:09.942434+0000","last_became_active":"2024-06-04T04:53:36.372754+0000","last_became_peered":"2024-06-04T04:53:36.372754+0000","last_unstale":"2024-06-04T04:57:09.942434+0000","last_undegraded":"2024-06-04T04:57:09.942434+0000","last_fullsized":"2024-06-04T04:57:09.942434+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:22:27.554200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215415+0000","last_change":"2024-06-04T04:53:36.378897+0000","last_active":"2024-06-04T04:57:10.215415+0000","last_peered":"2024-06-04T04:57:10.215415+0000","last_clean":"2024-06-04T04:57:10.215415+0000","last_became_active":"2024-06-04T04:53:36.378607+0000","last_became_peered":"2024-06-04T04:53:36.378607+0000","last_unstale":"2024-06-04T04:57:10.215415+0000","last_undegraded":"2024-06-04T04:57:10.215415+0000","last_fullsized":"2024-06-04T04:57:10.215415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:03.395491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215502+0000","last_change":"2024-06-04T04:53:36.378240+0000","last_active":"2024-06-04T04:57:10.215502+0000","last_peered":"2024-06-04T04:57:10.215502+0000","last_clean":"2024-06-04T04:57:10.215502+0000","last_became_active":"2024-06-04T04:53:36.377909+0000","last_became_peered":"2024-06-04T04:53:36.377909+0000","last_unstale":"2024-06-04T04:57:10.215502+0000","last_undegraded":"2024-06-04T04:57:10.215502+0000","last_fullsized":"2024-06-04T04:57:10.215502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:15:25.288340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466255+0000","last_change":"2024-06-04T04:53:36.379792+0000","last_active":"2024-06-04T04:57:10.466255+0000","last_peered":"2024-06-04T04:57:10.466255+0000","last_clean":"2024-06-04T04:57:10.466255+0000","last_became_active":"2024-06-04T04:53:36.379438+0000","last_became_peered":"2024-06-04T04:53:36.379438+0000","last_unstale":"2024-06-04T04:57:10.466255+0000","last_undegraded":"2024-06-04T04:57:10.466255+0000","last_fullsized":"2024-06-04T04:57:10.466255+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:23:57.617877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108276,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":802806,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2510,"num_read_kb":1478,"num_write":511,"num_write_kb":2390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":798720,"data_stored":2408418,"data_compressed":108972,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":387,"ondisk_log_size":387,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678077,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":892928,"data_stored":439038,"data_compressed":8651,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874305,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1101824,"data_stored":1429949,"data_compressed":61778,"data_compressed_allocated":81920,"data_compressed_original":929792,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037831,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":831691,"data_compressed":27940,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234061,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":831397,"data_compressed":27940,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430288,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":1033773,"data_compressed":44743,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593815,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":901120,"data_stored":439999,"data_compressed":8651,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":978944,"data_stored":513234,"data_compressed":10484,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738485,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":921600,"data_stored":440024,"data_compressed":8651,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411434,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":511961,"data_compressed":10484,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":987136,"data_stored":1159465,"data_compressed":43142,"data_compressed_allocated":57344,"data_compressed_original":724992,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953569,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":901120,"data_stored":438546,"data_compressed":8651,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790044,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":1359383,"data_compressed":59945,"data_compressed_allocated":77824,"data_compressed_original":921600,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":75881,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":74608,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":721878,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":331545,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":394044,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":402111,"data_compressed":17035,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-04T04:57:31.038 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-04T04:57:31.038 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-04T04:57:31.277 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:31.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:31 smithi070 ceph-mon[32631]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-06-04T04:57:31.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:31 smithi070 ceph-mon[32631]: from='client.15429 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:31.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:31 smithi112 ceph-mon[36453]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-06-04T04:57:31.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:31 smithi112 ceph-mon[36453]: from='client.15429 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:31.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:31 smithi098 ceph-mon[35569]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-06-04T04:57:31.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:31 smithi098 ceph-mon[35569]: from='client.15429 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:32.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.937+0000 7fdf6911f700 1 -- 172.21.15.70:0/1231008866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64103850 msgr2=0x7fdf64103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:32.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.937+0000 7fdf6911f700 1 --2- 172.21.15.70:0/1231008866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64103850 0x7fdf64103c30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fdf580099c0 tx=0x7fdf58009cd0 comp rx=0 tx=0).stop 2024-06-04T04:57:32.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.937+0000 7fdf6911f700 1 -- 172.21.15.70:0/1231008866 shutdown_connections 2024-06-04T04:57:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.937+0000 7fdf6911f700 1 --2- 172.21.15.70:0/1231008866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf64108920 0x7fdf641104a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.937+0000 7fdf6911f700 1 --2- 172.21.15.70:0/1231008866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64103850 0x7fdf64103c30 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.937+0000 7fdf6911f700 1 --2- 172.21.15.70:0/1231008866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf64104200 0x7fdf641081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.937+0000 7fdf6911f700 1 -- 172.21.15.70:0/1231008866 >> 172.21.15.70:0/1231008866 conn(0x7fdf6407a620 msgr2=0x7fdf6407aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.937+0000 7fdf6911f700 1 -- 172.21.15.70:0/1231008866 shutdown_connections 2024-06-04T04:57:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.937+0000 7fdf6911f700 1 -- 172.21.15.70:0/1231008866 wait complete. 2024-06-04T04:57:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf6911f700 1 Processor -- start 2024-06-04T04:57:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf6911f700 1 -- start start 2024-06-04T04:57:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf6911f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64103850 0x7fdf6419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf6911f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf64104200 0x7fdf6419fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf6911f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf64108920 0x7fdf641998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf6911f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdf641135b0 con 0x7fdf64103850 2024-06-04T04:57:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf6911f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf64113430 con 0x7fdf64108920 2024-06-04T04:57:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf6911f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdf64113730 con 0x7fdf64104200 2024-06-04T04:57:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf62d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64103850 0x7fdf6419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf62d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64103850 0x7fdf6419f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35814/0 (socket says 172.21.15.70:35814) 2024-06-04T04:57:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf62d9d700 1 -- 172.21.15.70:0/1950489814 learned_addr learned my addr 172.21.15.70:0/1950489814 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf6259c700 1 --2- 172.21.15.70:0/1950489814 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf64104200 0x7fdf6419fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf6359e700 1 --2- 172.21.15.70:0/1950489814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf64108920 0x7fdf641998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf62d9d700 1 -- 172.21.15.70:0/1950489814 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf64104200 msgr2=0x7fdf6419fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf62d9d700 1 --2- 172.21.15.70:0/1950489814 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf64104200 0x7fdf6419fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf62d9d700 1 -- 172.21.15.70:0/1950489814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf64108920 msgr2=0x7fdf641998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf62d9d700 1 --2- 172.21.15.70:0/1950489814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf64108920 0x7fdf641998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf62d9d700 1 -- 172.21.15.70:0/1950489814 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6419a050 con 0x7fdf64103850 2024-06-04T04:57:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.938+0000 7fdf6359e700 1 --2- 172.21.15.70:0/1950489814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf64108920 0x7fdf641998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.939+0000 7fdf62d9d700 1 --2- 172.21.15.70:0/1950489814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64103850 0x7fdf6419f6e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fdf58000c00 tx=0x7fdf580047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.939+0000 7fdf4bfff700 1 -- 172.21.15.70:0/1950489814 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5800f8a0 con 0x7fdf64103850 2024-06-04T04:57:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.939+0000 7fdf6911f700 1 -- 172.21.15.70:0/1950489814 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf6419a2e0 con 0x7fdf64103850 2024-06-04T04:57:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.939+0000 7fdf4bfff700 1 -- 172.21.15.70:0/1950489814 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdf58004830 con 0x7fdf64103850 2024-06-04T04:57:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.939+0000 7fdf4bfff700 1 -- 172.21.15.70:0/1950489814 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf58005410 con 0x7fdf64103850 2024-06-04T04:57:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.939+0000 7fdf6911f700 1 -- 172.21.15.70:0/1950489814 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf641a65b0 con 0x7fdf64103850 2024-06-04T04:57:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.940+0000 7fdf6911f700 1 -- 172.21.15.70:0/1950489814 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf6406a230 con 0x7fdf64103850 2024-06-04T04:57:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.941+0000 7fdf4bfff700 1 -- 172.21.15.70:0/1950489814 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7fdf58029060 con 0x7fdf64103850 2024-06-04T04:57:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.942+0000 7fdf4bfff700 1 --2- 172.21.15.70:0/1950489814 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdf4c0b2a80 0x7fdf4c0b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.942+0000 7fdf6259c700 1 --2- 172.21.15.70:0/1950489814 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdf4c0b2a80 0x7fdf4c0b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.942+0000 7fdf4bfff700 1 -- 172.21.15.70:0/1950489814 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fdf580e32d0 con 0x7fdf64103850 2024-06-04T04:57:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.942+0000 7fdf6259c700 1 --2- 172.21.15.70:0/1950489814 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdf4c0b2a80 0x7fdf4c0b4f40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdf64103180 tx=0x7fdf540073d0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:32.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:32.946+0000 7fdf4bfff700 1 -- 172.21.15.70:0/1950489814 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdf58013070 con 0x7fdf64103850 2024-06-04T04:57:33.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:33.152+0000 7fdf6911f700 1 -- 172.21.15.70:0/1950489814 --> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdf64063980 con 0x7fdf4c0b2a80 2024-06-04T04:57:33.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:33.159+0000 7fdf4bfff700 1 -- 172.21.15.70:0/1950489814 <== mgr.24806 v2:172.21.15.70:6800/542036627 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258109 (secure 0 0 0) 0x7fdf64063980 con 0x7fdf4c0b2a80 2024-06-04T04:57:33.165 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:57:33.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:33.162+0000 7fdf6911f700 1 -- 172.21.15.70:0/1950489814 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdf4c0b2a80 msgr2=0x7fdf4c0b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:33.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:33.162+0000 7fdf6911f700 1 --2- 172.21.15.70:0/1950489814 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdf4c0b2a80 0x7fdf4c0b4f40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdf64103180 tx=0x7fdf540073d0 comp rx=0 tx=0).stop 2024-06-04T04:57:33.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:33.162+0000 7fdf6911f700 1 -- 172.21.15.70:0/1950489814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64103850 msgr2=0x7fdf6419f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:33.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:33.162+0000 7fdf6911f700 1 --2- 172.21.15.70:0/1950489814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64103850 0x7fdf6419f6e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fdf58000c00 tx=0x7fdf580047b0 comp rx=0 tx=0).stop 2024-06-04T04:57:33.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:33.162+0000 7fdf6911f700 1 -- 172.21.15.70:0/1950489814 shutdown_connections 2024-06-04T04:57:33.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:33.162+0000 7fdf6911f700 1 --2- 172.21.15.70:0/1950489814 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdf4c0b2a80 0x7fdf4c0b4f40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:33.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:33.162+0000 7fdf6911f700 1 --2- 172.21.15.70:0/1950489814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf64108920 0x7fdf641998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:33.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:33.162+0000 7fdf6911f700 1 --2- 172.21.15.70:0/1950489814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64103850 0x7fdf6419f6e0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:33.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:33.162+0000 7fdf6911f700 1 --2- 172.21.15.70:0/1950489814 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf64104200 0x7fdf6419fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:33.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:33.162+0000 7fdf6911f700 1 -- 172.21.15.70:0/1950489814 >> 172.21.15.70:0/1950489814 conn(0x7fdf6407a620 msgr2=0x7fdf641018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:33.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:33.162+0000 7fdf6911f700 1 -- 172.21.15.70:0/1950489814 shutdown_connections 2024-06-04T04:57:33.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:33.162+0000 7fdf6911f700 1 -- 172.21.15.70:0/1950489814 wait complete. 2024-06-04T04:57:33.168 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-06-04T04:57:33.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:33 smithi070 ceph-mon[32631]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 44 op/s 2024-06-04T04:57:33.928 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-06-04T04:57:31.963935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1393408,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2704,"num_read_kb":1751,"num_write":648,"num_write_kb":4542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2952512,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":988,"ondisk_log_size":988,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356312,"kb_used_data":11268,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585512,"statfs":{"total":1151940427776,"available":1151575564288,"internally_reserved":0,"allocated":11538432,"data_stored":9428460,"data_compressed":321060,"data_compressed_allocated":430080,"data_compressed_original":4190208,"omap_allocated":19138,"internal_metadata":353154366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":405,"num_read_kb":307,"num_write":126,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002575"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466420+0000","last_change":"2024-06-04T04:53:36.383755+0000","last_active":"2024-06-04T04:57:10.466420+0000","last_peered":"2024-06-04T04:57:10.466420+0000","last_clean":"2024-06-04T04:57:10.466420+0000","last_became_active":"2024-06-04T04:53:36.378274+0000","last_became_peered":"2024-06-04T04:53:36.378274+0000","last_unstale":"2024-06-04T04:57:10.466420+0000","last_undegraded":"2024-06-04T04:57:10.466420+0000","last_fullsized":"2024-06-04T04:57:10.466420+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:30:19.685994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466173+0000","last_change":"2024-06-04T04:53:36.372436+0000","last_active":"2024-06-04T04:57:10.466173+0000","last_peered":"2024-06-04T04:57:10.466173+0000","last_clean":"2024-06-04T04:57:10.466173+0000","last_became_active":"2024-06-04T04:53:36.372100+0000","last_became_peered":"2024-06-04T04:53:36.372100+0000","last_unstale":"2024-06-04T04:57:10.466173+0000","last_undegraded":"2024-06-04T04:57:10.466173+0000","last_fullsized":"2024-06-04T04:57:10.466173+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:42:31.646754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466063+0000","last_change":"2024-06-04T04:53:36.366915+0000","last_active":"2024-06-04T04:57:10.466063+0000","last_peered":"2024-06-04T04:57:10.466063+0000","last_clean":"2024-06-04T04:57:10.466063+0000","last_became_active":"2024-06-04T04:53:36.366611+0000","last_became_peered":"2024-06-04T04:53:36.366611+0000","last_unstale":"2024-06-04T04:57:10.466063+0000","last_undegraded":"2024-06-04T04:57:10.466063+0000","last_fullsized":"2024-06-04T04:57:10.466063+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:17:48.775938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466271+0000","last_change":"2024-06-04T04:53:36.374486+0000","last_active":"2024-06-04T04:57:10.466271+0000","last_peered":"2024-06-04T04:57:10.466271+0000","last_clean":"2024-06-04T04:57:10.466271+0000","last_became_active":"2024-06-04T04:53:36.374178+0000","last_became_peered":"2024-06-04T04:53:36.374178+0000","last_unstale":"2024-06-04T04:57:10.466271+0000","last_undegraded":"2024-06-04T04:57:10.466271+0000","last_fullsized":"2024-06-04T04:57:10.466271+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:30:28.287070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965508+0000","last_change":"2024-06-04T04:53:36.376183+0000","last_active":"2024-06-04T04:57:09.965508+0000","last_peered":"2024-06-04T04:57:09.965508+0000","last_clean":"2024-06-04T04:57:09.965508+0000","last_became_active":"2024-06-04T04:53:36.375790+0000","last_became_peered":"2024-06-04T04:53:36.375790+0000","last_unstale":"2024-06-04T04:57:09.965508+0000","last_undegraded":"2024-06-04T04:57:09.965508+0000","last_fullsized":"2024-06-04T04:57:09.965508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:58.697440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946349+0000","last_change":"2024-06-04T04:53:36.370249+0000","last_active":"2024-06-04T04:57:09.946349+0000","last_peered":"2024-06-04T04:57:09.946349+0000","last_clean":"2024-06-04T04:57:09.946349+0000","last_became_active":"2024-06-04T04:53:36.369916+0000","last_became_peered":"2024-06-04T04:53:36.369916+0000","last_unstale":"2024-06-04T04:57:09.946349+0000","last_undegraded":"2024-06-04T04:57:09.946349+0000","last_fullsized":"2024-06-04T04:57:09.946349+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:02:41.739806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213216+0000","last_change":"2024-06-04T04:53:36.384329+0000","last_active":"2024-06-04T04:57:10.213216+0000","last_peered":"2024-06-04T04:57:10.213216+0000","last_clean":"2024-06-04T04:57:10.213216+0000","last_became_active":"2024-06-04T04:53:36.382504+0000","last_became_peered":"2024-06-04T04:53:36.382504+0000","last_unstale":"2024-06-04T04:57:10.213216+0000","last_undegraded":"2024-06-04T04:57:10.213216+0000","last_fullsized":"2024-06-04T04:57:10.213216+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:51:07.604691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942376+0000","last_change":"2024-06-04T04:53:36.372319+0000","last_active":"2024-06-04T04:57:09.942376+0000","last_peered":"2024-06-04T04:57:09.942376+0000","last_clean":"2024-06-04T04:57:09.942376+0000","last_became_active":"2024-06-04T04:53:36.372000+0000","last_became_peered":"2024-06-04T04:53:36.372000+0000","last_unstale":"2024-06-04T04:57:09.942376+0000","last_undegraded":"2024-06-04T04:57:09.942376+0000","last_fullsized":"2024-06-04T04:57:09.942376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:15:56.906996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215880+0000","last_change":"2024-06-04T04:53:36.376051+0000","last_active":"2024-06-04T04:57:10.215880+0000","last_peered":"2024-06-04T04:57:10.215880+0000","last_clean":"2024-06-04T04:57:10.215880+0000","last_became_active":"2024-06-04T04:53:36.375726+0000","last_became_peered":"2024-06-04T04:53:36.375726+0000","last_unstale":"2024-06-04T04:57:10.215880+0000","last_undegraded":"2024-06-04T04:57:10.215880+0000","last_fullsized":"2024-06-04T04:57:10.215880+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:09:08.542764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946736+0000","last_change":"2024-06-04T04:53:36.373266+0000","last_active":"2024-06-04T04:57:09.946736+0000","last_peered":"2024-06-04T04:57:09.946736+0000","last_clean":"2024-06-04T04:57:09.946736+0000","last_became_active":"2024-06-04T04:53:36.372898+0000","last_became_peered":"2024-06-04T04:53:36.372898+0000","last_unstale":"2024-06-04T04:57:09.946736+0000","last_undegraded":"2024-06-04T04:57:09.946736+0000","last_fullsized":"2024-06-04T04:57:09.946736+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:41:43.436781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213610+0000","last_change":"2024-06-04T04:53:36.381344+0000","last_active":"2024-06-04T04:57:10.213610+0000","last_peered":"2024-06-04T04:57:10.213610+0000","last_clean":"2024-06-04T04:57:10.213610+0000","last_became_active":"2024-06-04T04:53:36.380736+0000","last_became_peered":"2024-06-04T04:53:36.380736+0000","last_unstale":"2024-06-04T04:57:10.213610+0000","last_undegraded":"2024-06-04T04:57:10.213610+0000","last_fullsized":"2024-06-04T04:57:10.213610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:23:01.607272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466124+0000","last_change":"2024-06-04T04:53:36.374035+0000","last_active":"2024-06-04T04:57:10.466124+0000","last_peered":"2024-06-04T04:57:10.466124+0000","last_clean":"2024-06-04T04:57:10.466124+0000","last_became_active":"2024-06-04T04:53:36.373667+0000","last_became_peered":"2024-06-04T04:53:36.373667+0000","last_unstale":"2024-06-04T04:57:10.466124+0000","last_undegraded":"2024-06-04T04:57:10.466124+0000","last_fullsized":"2024-06-04T04:57:10.466124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:23:07.880369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215527+0000","last_change":"2024-06-04T04:53:36.369613+0000","last_active":"2024-06-04T04:57:10.215527+0000","last_peered":"2024-06-04T04:57:10.215527+0000","last_clean":"2024-06-04T04:57:10.215527+0000","last_became_active":"2024-06-04T04:53:36.369277+0000","last_became_peered":"2024-06-04T04:53:36.369277+0000","last_unstale":"2024-06-04T04:57:10.215527+0000","last_undegraded":"2024-06-04T04:57:10.215527+0000","last_fullsized":"2024-06-04T04:57:10.215527+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:13:33.756609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.465960+0000","last_change":"2024-06-04T04:53:36.368094+0000","last_active":"2024-06-04T04:57:10.465960+0000","last_peered":"2024-06-04T04:57:10.465960+0000","last_clean":"2024-06-04T04:57:10.465960+0000","last_became_active":"2024-06-04T04:53:36.367717+0000","last_became_peered":"2024-06-04T04:53:36.367717+0000","last_unstale":"2024-06-04T04:57:10.465960+0000","last_undegraded":"2024-06-04T04:57:10.465960+0000","last_fullsized":"2024-06-04T04:57:10.465960+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:39:56.305078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.465864+0000","last_change":"2024-06-04T04:53:36.366765+0000","last_active":"2024-06-04T04:57:10.465864+0000","last_peered":"2024-06-04T04:57:10.465864+0000","last_clean":"2024-06-04T04:57:10.465864+0000","last_became_active":"2024-06-04T04:53:36.366438+0000","last_became_peered":"2024-06-04T04:53:36.366438+0000","last_unstale":"2024-06-04T04:57:10.465864+0000","last_undegraded":"2024-06-04T04:57:10.465864+0000","last_fullsized":"2024-06-04T04:57:10.465864+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:38:37.313926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215176+0000","last_change":"2024-06-04T04:53:36.380088+0000","last_active":"2024-06-04T04:57:10.215176+0000","last_peered":"2024-06-04T04:57:10.215176+0000","last_clean":"2024-06-04T04:57:10.215176+0000","last_became_active":"2024-06-04T04:53:36.379467+0000","last_became_peered":"2024-06-04T04:53:36.379467+0000","last_unstale":"2024-06-04T04:57:10.215176+0000","last_undegraded":"2024-06-04T04:57:10.215176+0000","last_fullsized":"2024-06-04T04:57:10.215176+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:46:53.236419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466550+0000","last_change":"2024-06-04T04:53:36.381126+0000","last_active":"2024-06-04T04:57:10.466550+0000","last_peered":"2024-06-04T04:57:10.466550+0000","last_clean":"2024-06-04T04:57:10.466550+0000","last_became_active":"2024-06-04T04:53:36.378833+0000","last_became_peered":"2024-06-04T04:53:36.378833+0000","last_unstale":"2024-06-04T04:57:10.466550+0000","last_undegraded":"2024-06-04T04:57:10.466550+0000","last_fullsized":"2024-06-04T04:57:10.466550+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:20:28.639273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.048240+0000","last_change":"2024-06-04T04:53:36.369692+0000","last_active":"2024-06-04T04:57:10.048240+0000","last_peered":"2024-06-04T04:57:10.048240+0000","last_clean":"2024-06-04T04:57:10.048240+0000","last_became_active":"2024-06-04T04:53:36.369370+0000","last_became_peered":"2024-06-04T04:53:36.369370+0000","last_unstale":"2024-06-04T04:57:10.048240+0000","last_undegraded":"2024-06-04T04:57:10.048240+0000","last_fullsized":"2024-06-04T04:57:10.048240+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:55.275465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219675+0000","last_change":"2024-06-04T04:53:36.383639+0000","last_active":"2024-06-04T04:57:10.219675+0000","last_peered":"2024-06-04T04:57:10.219675+0000","last_clean":"2024-06-04T04:57:10.219675+0000","last_became_active":"2024-06-04T04:53:36.383303+0000","last_became_peered":"2024-06-04T04:53:36.383303+0000","last_unstale":"2024-06-04T04:57:10.219675+0000","last_undegraded":"2024-06-04T04:57:10.219675+0000","last_fullsized":"2024-06-04T04:57:10.219675+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:26:31.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'15","reported_seq":283,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:29.925085+0000","last_change":"2024-06-04T04:54:19.641520+0000","last_active":"2024-06-04T04:57:29.925085+0000","last_peered":"2024-06-04T04:57:29.925085+0000","last_clean":"2024-06-04T04:57:29.925085+0000","last_became_active":"2024-06-04T04:54:19.632433+0000","last_became_peered":"2024-06-04T04:54:19.632433+0000","last_unstale":"2024-06-04T04:57:29.925085+0000","last_undegraded":"2024-06-04T04:57:29.925085+0000","last_fullsized":"2024-06-04T04:57:29.925085+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:26:05.993919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":213,"num_read_kb":213,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942727+0000","last_change":"2024-06-04T04:53:36.371827+0000","last_active":"2024-06-04T04:57:09.942727+0000","last_peered":"2024-06-04T04:57:09.942727+0000","last_clean":"2024-06-04T04:57:09.942727+0000","last_became_active":"2024-06-04T04:53:36.371509+0000","last_became_peered":"2024-06-04T04:53:36.371509+0000","last_unstale":"2024-06-04T04:57:09.942727+0000","last_undegraded":"2024-06-04T04:57:09.942727+0000","last_fullsized":"2024-06-04T04:57:09.942727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:54:51.589008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.943009+0000","last_change":"2024-06-04T04:53:33.353976+0000","last_active":"2024-06-04T04:57:09.943009+0000","last_peered":"2024-06-04T04:57:09.943009+0000","last_clean":"2024-06-04T04:57:09.943009+0000","last_became_active":"2024-06-04T04:53:33.353586+0000","last_became_peered":"2024-06-04T04:53:33.353586+0000","last_unstale":"2024-06-04T04:57:09.943009+0000","last_undegraded":"2024-06-04T04:57:09.943009+0000","last_fullsized":"2024-06-04T04:57:09.943009+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:17:10.378008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466390+0000","last_change":"2024-06-04T04:53:36.367806+0000","last_active":"2024-06-04T04:57:10.466390+0000","last_peered":"2024-06-04T04:57:10.466390+0000","last_clean":"2024-06-04T04:57:10.466390+0000","last_became_active":"2024-06-04T04:53:36.367484+0000","last_became_peered":"2024-06-04T04:53:36.367484+0000","last_unstale":"2024-06-04T04:57:10.466390+0000","last_undegraded":"2024-06-04T04:57:10.466390+0000","last_fullsized":"2024-06-04T04:57:10.466390+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:25:36.194746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"82'13","reported_seq":43,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965867+0000","last_change":"2024-06-04T04:53:33.348902+0000","last_active":"2024-06-04T04:57:09.965867+0000","last_peered":"2024-06-04T04:57:09.965867+0000","last_clean":"2024-06-04T04:57:09.965867+0000","last_became_active":"2024-06-04T04:53:33.348441+0000","last_became_peered":"2024-06-04T04:53:33.348441+0000","last_unstale":"2024-06-04T04:57:09.965867+0000","last_undegraded":"2024-06-04T04:57:09.965867+0000","last_fullsized":"2024-06-04T04:57:09.965867+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:22:28.428303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215265+0000","last_change":"2024-06-04T04:53:36.376867+0000","last_active":"2024-06-04T04:57:10.215265+0000","last_peered":"2024-06-04T04:57:10.215265+0000","last_clean":"2024-06-04T04:57:10.215265+0000","last_became_active":"2024-06-04T04:53:36.376517+0000","last_became_peered":"2024-06-04T04:53:36.376517+0000","last_unstale":"2024-06-04T04:57:10.215265+0000","last_undegraded":"2024-06-04T04:57:10.215265+0000","last_fullsized":"2024-06-04T04:57:10.215265+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:21:59.494321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965814+0000","last_change":"2024-06-04T04:53:33.347993+0000","last_active":"2024-06-04T04:57:09.965814+0000","last_peered":"2024-06-04T04:57:09.965814+0000","last_clean":"2024-06-04T04:57:09.965814+0000","last_became_active":"2024-06-04T04:53:33.347656+0000","last_became_peered":"2024-06-04T04:53:33.347656+0000","last_unstale":"2024-06-04T04:57:09.965814+0000","last_undegraded":"2024-06-04T04:57:09.965814+0000","last_fullsized":"2024-06-04T04:57:09.965814+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:21:26.806557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":44,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946660+0000","last_change":"2024-06-04T04:54:19.632790+0000","last_active":"2024-06-04T04:57:09.946660+0000","last_peered":"2024-06-04T04:57:09.946660+0000","last_clean":"2024-06-04T04:57:09.946660+0000","last_became_active":"2024-06-04T04:54:19.632338+0000","last_became_peered":"2024-06-04T04:54:19.632338+0000","last_unstale":"2024-06-04T04:57:09.946660+0000","last_undegraded":"2024-06-04T04:57:09.946660+0000","last_fullsized":"2024-06-04T04:57:09.946660+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:44:00.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"81'6","reported_seq":255,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:29.760927+0000","last_change":"2024-06-04T04:53:33.346111+0000","last_active":"2024-06-04T04:57:29.760927+0000","last_peered":"2024-06-04T04:57:29.760927+0000","last_clean":"2024-06-04T04:57:29.760927+0000","last_became_active":"2024-06-04T04:53:33.345773+0000","last_became_peered":"2024-06-04T04:53:33.345773+0000","last_unstale":"2024-06-04T04:57:29.760927+0000","last_undegraded":"2024-06-04T04:57:29.760927+0000","last_fullsized":"2024-06-04T04:57:29.760927+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:18.565583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1115,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":220,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965482+0000","last_change":"2024-06-04T04:53:36.373369+0000","last_active":"2024-06-04T04:57:09.965482+0000","last_peered":"2024-06-04T04:57:09.965482+0000","last_clean":"2024-06-04T04:57:09.965482+0000","last_became_active":"2024-06-04T04:53:36.373001+0000","last_became_peered":"2024-06-04T04:53:36.373001+0000","last_unstale":"2024-06-04T04:57:09.965482+0000","last_undegraded":"2024-06-04T04:57:09.965482+0000","last_fullsized":"2024-06-04T04:57:09.965482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:02:22.012643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946291+0000","last_change":"2024-06-04T04:53:33.344358+0000","last_active":"2024-06-04T04:57:09.946291+0000","last_peered":"2024-06-04T04:57:09.946291+0000","last_clean":"2024-06-04T04:57:09.946291+0000","last_became_active":"2024-06-04T04:53:33.343980+0000","last_became_peered":"2024-06-04T04:53:33.343980+0000","last_unstale":"2024-06-04T04:57:09.946291+0000","last_undegraded":"2024-06-04T04:57:09.946291+0000","last_fullsized":"2024-06-04T04:57:09.946291+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:14.493138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946579+0000","last_change":"2024-06-04T04:53:36.371239+0000","last_active":"2024-06-04T04:57:09.946579+0000","last_peered":"2024-06-04T04:57:09.946579+0000","last_clean":"2024-06-04T04:57:09.946579+0000","last_became_active":"2024-06-04T04:53:36.370885+0000","last_became_peered":"2024-06-04T04:53:36.370885+0000","last_unstale":"2024-06-04T04:57:09.946579+0000","last_undegraded":"2024-06-04T04:57:09.946579+0000","last_fullsized":"2024-06-04T04:57:09.946579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T04:54:29.753796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215252+0000","last_change":"2024-06-04T04:53:33.338512+0000","last_active":"2024-06-04T04:57:10.215252+0000","last_peered":"2024-06-04T04:57:10.215252+0000","last_clean":"2024-06-04T04:57:10.215252+0000","last_became_active":"2024-06-04T04:53:33.338180+0000","last_became_peered":"2024-06-04T04:53:33.338180+0000","last_unstale":"2024-06-04T04:57:10.215252+0000","last_undegraded":"2024-06-04T04:57:10.215252+0000","last_fullsized":"2024-06-04T04:57:10.215252+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:57:13.406479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942451+0000","last_change":"2024-06-04T04:53:36.377316+0000","last_active":"2024-06-04T04:57:09.942451+0000","last_peered":"2024-06-04T04:57:09.942451+0000","last_clean":"2024-06-04T04:57:09.942451+0000","last_became_active":"2024-06-04T04:53:36.376992+0000","last_became_peered":"2024-06-04T04:53:36.376992+0000","last_unstale":"2024-06-04T04:57:09.942451+0000","last_undegraded":"2024-06-04T04:57:09.942451+0000","last_fullsized":"2024-06-04T04:57:09.942451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:43:33.145631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946281+0000","last_change":"2024-06-04T04:53:33.348305+0000","last_active":"2024-06-04T04:57:09.946281+0000","last_peered":"2024-06-04T04:57:09.946281+0000","last_clean":"2024-06-04T04:57:09.946281+0000","last_became_active":"2024-06-04T04:53:33.347948+0000","last_became_peered":"2024-06-04T04:53:33.347948+0000","last_unstale":"2024-06-04T04:57:09.946281+0000","last_undegraded":"2024-06-04T04:57:09.946281+0000","last_fullsized":"2024-06-04T04:57:09.946281+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:04:13.669274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.047728+0000","last_change":"2024-06-04T04:53:36.367955+0000","last_active":"2024-06-04T04:57:10.047728+0000","last_peered":"2024-06-04T04:57:10.047728+0000","last_clean":"2024-06-04T04:57:10.047728+0000","last_became_active":"2024-06-04T04:53:36.367628+0000","last_became_peered":"2024-06-04T04:53:36.367628+0000","last_unstale":"2024-06-04T04:57:10.047728+0000","last_undegraded":"2024-06-04T04:57:10.047728+0000","last_fullsized":"2024-06-04T04:57:10.047728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:33:26.185498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":468,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:30.761626+0000","last_change":"2024-06-04T04:53:33.349471+0000","last_active":"2024-06-04T04:57:30.761626+0000","last_peered":"2024-06-04T04:57:30.761626+0000","last_clean":"2024-06-04T04:57:30.761626+0000","last_became_active":"2024-06-04T04:53:33.349105+0000","last_became_peered":"2024-06-04T04:53:33.349105+0000","last_unstale":"2024-06-04T04:57:30.761626+0000","last_undegraded":"2024-06-04T04:57:30.761626+0000","last_fullsized":"2024-06-04T04:57:30.761626+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:31:35.584275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":434,"num_read_kb":434,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215768+0000","last_change":"2024-06-04T04:53:36.367249+0000","last_active":"2024-06-04T04:57:10.215768+0000","last_peered":"2024-06-04T04:57:10.215768+0000","last_clean":"2024-06-04T04:57:10.215768+0000","last_became_active":"2024-06-04T04:53:36.366863+0000","last_became_peered":"2024-06-04T04:53:36.366863+0000","last_unstale":"2024-06-04T04:57:10.215768+0000","last_undegraded":"2024-06-04T04:57:10.215768+0000","last_fullsized":"2024-06-04T04:57:10.215768+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:13:17.308321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213781+0000","last_change":"2024-06-04T04:53:33.346778+0000","last_active":"2024-06-04T04:57:10.213781+0000","last_peered":"2024-06-04T04:57:10.213781+0000","last_clean":"2024-06-04T04:57:10.213781+0000","last_became_active":"2024-06-04T04:53:33.346376+0000","last_became_peered":"2024-06-04T04:53:33.346376+0000","last_unstale":"2024-06-04T04:57:10.213781+0000","last_undegraded":"2024-06-04T04:57:10.213781+0000","last_fullsized":"2024-06-04T04:57:10.213781+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:03:16.973822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466320+0000","last_change":"2024-06-04T04:54:19.514491+0000","last_active":"2024-06-04T04:57:10.466320+0000","last_peered":"2024-06-04T04:57:10.466320+0000","last_clean":"2024-06-04T04:57:10.466320+0000","last_became_active":"2024-06-04T04:54:19.514148+0000","last_became_peered":"2024-06-04T04:54:19.514148+0000","last_unstale":"2024-06-04T04:57:10.466320+0000","last_undegraded":"2024-06-04T04:57:10.466320+0000","last_fullsized":"2024-06-04T04:57:10.466320+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:09:59.641041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946792+0000","last_change":"2024-06-04T04:53:33.354989+0000","last_active":"2024-06-04T04:57:09.946792+0000","last_peered":"2024-06-04T04:57:09.946792+0000","last_clean":"2024-06-04T04:57:09.946792+0000","last_became_active":"2024-06-04T04:53:33.354613+0000","last_became_peered":"2024-06-04T04:53:33.354613+0000","last_unstale":"2024-06-04T04:57:09.946792+0000","last_undegraded":"2024-06-04T04:57:09.946792+0000","last_fullsized":"2024-06-04T04:57:09.946792+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:19:59.398763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215492+0000","last_change":"2024-06-04T04:53:36.372216+0000","last_active":"2024-06-04T04:57:10.215492+0000","last_peered":"2024-06-04T04:57:10.215492+0000","last_clean":"2024-06-04T04:57:10.215492+0000","last_became_active":"2024-06-04T04:53:36.371820+0000","last_became_peered":"2024-06-04T04:53:36.371820+0000","last_unstale":"2024-06-04T04:57:10.215492+0000","last_undegraded":"2024-06-04T04:57:10.215492+0000","last_fullsized":"2024-06-04T04:57:10.215492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:55:59.887963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'40","reported_seq":515,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:30.781408+0000","last_change":"2024-06-04T04:53:33.345493+0000","last_active":"2024-06-04T04:57:30.781408+0000","last_peered":"2024-06-04T04:57:30.781408+0000","last_clean":"2024-06-04T04:57:30.781408+0000","last_became_active":"2024-06-04T04:53:33.345102+0000","last_became_peered":"2024-06-04T04:53:33.345102+0000","last_unstale":"2024-06-04T04:57:30.781408+0000","last_undegraded":"2024-06-04T04:57:30.781408+0000","last_fullsized":"2024-06-04T04:57:30.781408+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:50:27.853391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71650,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":447,"num_read_kb":294,"num_write":42,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965646+0000","last_change":"2024-06-04T04:53:36.382070+0000","last_active":"2024-06-04T04:57:09.965646+0000","last_peered":"2024-06-04T04:57:09.965646+0000","last_clean":"2024-06-04T04:57:09.965646+0000","last_became_active":"2024-06-04T04:53:36.379144+0000","last_became_peered":"2024-06-04T04:53:36.379144+0000","last_unstale":"2024-06-04T04:57:09.965646+0000","last_undegraded":"2024-06-04T04:57:09.965646+0000","last_fullsized":"2024-06-04T04:57:09.965646+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:16.702709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219432+0000","last_change":"2024-06-04T04:53:33.344726+0000","last_active":"2024-06-04T04:57:10.219432+0000","last_peered":"2024-06-04T04:57:10.219432+0000","last_clean":"2024-06-04T04:57:10.219432+0000","last_became_active":"2024-06-04T04:53:33.344360+0000","last_became_peered":"2024-06-04T04:53:33.344360+0000","last_unstale":"2024-06-04T04:57:10.219432+0000","last_undegraded":"2024-06-04T04:57:10.219432+0000","last_fullsized":"2024-06-04T04:57:10.219432+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:44:56.784808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215391+0000","last_change":"2024-06-04T04:53:36.379036+0000","last_active":"2024-06-04T04:57:10.215391+0000","last_peered":"2024-06-04T04:57:10.215391+0000","last_clean":"2024-06-04T04:57:10.215391+0000","last_became_active":"2024-06-04T04:53:36.378681+0000","last_became_peered":"2024-06-04T04:53:36.378681+0000","last_unstale":"2024-06-04T04:57:10.215391+0000","last_undegraded":"2024-06-04T04:57:10.215391+0000","last_fullsized":"2024-06-04T04:57:10.215391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:07:49.704731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946436+0000","last_change":"2024-06-04T04:53:33.345915+0000","last_active":"2024-06-04T04:57:09.946436+0000","last_peered":"2024-06-04T04:57:09.946436+0000","last_clean":"2024-06-04T04:57:09.946436+0000","last_became_active":"2024-06-04T04:53:33.345537+0000","last_became_peered":"2024-06-04T04:53:33.345537+0000","last_unstale":"2024-06-04T04:57:09.946436+0000","last_undegraded":"2024-06-04T04:57:09.946436+0000","last_fullsized":"2024-06-04T04:57:09.946436+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:06:44.086968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213503+0000","last_change":"2024-06-04T04:53:36.379094+0000","last_active":"2024-06-04T04:57:10.213503+0000","last_peered":"2024-06-04T04:57:10.213503+0000","last_clean":"2024-06-04T04:57:10.213503+0000","last_became_active":"2024-06-04T04:53:36.378763+0000","last_became_peered":"2024-06-04T04:53:36.378763+0000","last_unstale":"2024-06-04T04:57:10.213503+0000","last_undegraded":"2024-06-04T04:57:10.213503+0000","last_fullsized":"2024-06-04T04:57:10.213503+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:16.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'6","reported_seq":255,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:29.776513+0000","last_change":"2024-06-04T04:53:33.341342+0000","last_active":"2024-06-04T04:57:29.776513+0000","last_peered":"2024-06-04T04:57:29.776513+0000","last_clean":"2024-06-04T04:57:29.776513+0000","last_became_active":"2024-06-04T04:53:33.340862+0000","last_became_peered":"2024-06-04T04:53:33.340862+0000","last_unstale":"2024-06-04T04:57:29.776513+0000","last_undegraded":"2024-06-04T04:57:29.776513+0000","last_fullsized":"2024-06-04T04:57:29.776513+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:11:26.818063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":221,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213135+0000","last_change":"2024-06-04T04:53:36.383124+0000","last_active":"2024-06-04T04:57:10.213135+0000","last_peered":"2024-06-04T04:57:10.213135+0000","last_clean":"2024-06-04T04:57:10.213135+0000","last_became_active":"2024-06-04T04:53:36.382449+0000","last_became_peered":"2024-06-04T04:53:36.382449+0000","last_unstale":"2024-06-04T04:57:10.213135+0000","last_undegraded":"2024-06-04T04:57:10.213135+0000","last_fullsized":"2024-06-04T04:57:10.213135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:52:40.776003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":47,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965391+0000","last_change":"2024-06-04T04:53:33.354556+0000","last_active":"2024-06-04T04:57:09.965391+0000","last_peered":"2024-06-04T04:57:09.965391+0000","last_clean":"2024-06-04T04:57:09.965391+0000","last_became_active":"2024-06-04T04:53:33.354238+0000","last_became_peered":"2024-06-04T04:53:33.354238+0000","last_unstale":"2024-06-04T04:57:09.965391+0000","last_undegraded":"2024-06-04T04:57:09.965391+0000","last_fullsized":"2024-06-04T04:57:09.965391+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:54.697374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946345+0000","last_change":"2024-06-04T04:53:36.371387+0000","last_active":"2024-06-04T04:57:09.946345+0000","last_peered":"2024-06-04T04:57:09.946345+0000","last_clean":"2024-06-04T04:57:09.946345+0000","last_became_active":"2024-06-04T04:53:36.370852+0000","last_became_peered":"2024-06-04T04:53:36.370852+0000","last_unstale":"2024-06-04T04:57:09.946345+0000","last_undegraded":"2024-06-04T04:57:09.946345+0000","last_fullsized":"2024-06-04T04:57:09.946345+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:44:06.956210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'3","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965694+0000","last_change":"2024-06-04T04:53:33.349982+0000","last_active":"2024-06-04T04:57:09.965694+0000","last_peered":"2024-06-04T04:57:09.965694+0000","last_clean":"2024-06-04T04:57:09.965694+0000","last_became_active":"2024-06-04T04:53:33.349633+0000","last_became_peered":"2024-06-04T04:53:33.349633+0000","last_unstale":"2024-06-04T04:57:09.965694+0000","last_undegraded":"2024-06-04T04:57:09.965694+0000","last_fullsized":"2024-06-04T04:57:09.965694+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:28:40.107426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466355+0000","last_change":"2024-06-04T04:53:36.382313+0000","last_active":"2024-06-04T04:57:10.466355+0000","last_peered":"2024-06-04T04:57:10.466355+0000","last_clean":"2024-06-04T04:57:10.466355+0000","last_became_active":"2024-06-04T04:53:36.379638+0000","last_became_peered":"2024-06-04T04:53:36.379638+0000","last_unstale":"2024-06-04T04:57:10.466355+0000","last_undegraded":"2024-06-04T04:57:10.466355+0000","last_fullsized":"2024-06-04T04:57:10.466355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:43:42.527262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.048030+0000","last_change":"2024-06-04T04:54:19.513581+0000","last_active":"2024-06-04T04:57:10.048030+0000","last_peered":"2024-06-04T04:57:10.048030+0000","last_clean":"2024-06-04T04:57:10.048030+0000","last_became_active":"2024-06-04T04:54:19.513111+0000","last_became_peered":"2024-06-04T04:54:19.513111+0000","last_unstale":"2024-06-04T04:57:10.048030+0000","last_undegraded":"2024-06-04T04:57:10.048030+0000","last_fullsized":"2024-06-04T04:57:10.048030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:28:57.028083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946526+0000","last_change":"2024-06-04T04:53:36.369904+0000","last_active":"2024-06-04T04:57:09.946526+0000","last_peered":"2024-06-04T04:57:09.946526+0000","last_clean":"2024-06-04T04:57:09.946526+0000","last_became_active":"2024-06-04T04:53:36.367929+0000","last_became_peered":"2024-06-04T04:53:36.367929+0000","last_unstale":"2024-06-04T04:57:09.946526+0000","last_undegraded":"2024-06-04T04:57:09.946526+0000","last_fullsized":"2024-06-04T04:57:09.946526+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:37:08.721569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"82'10","reported_seq":261,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:29.925821+0000","last_change":"2024-06-04T04:53:33.342723+0000","last_active":"2024-06-04T04:57:29.925821+0000","last_peered":"2024-06-04T04:57:29.925821+0000","last_clean":"2024-06-04T04:57:29.925821+0000","last_became_active":"2024-06-04T04:53:33.342344+0000","last_became_peered":"2024-06-04T04:53:33.342344+0000","last_unstale":"2024-06-04T04:57:29.925821+0000","last_undegraded":"2024-06-04T04:57:29.925821+0000","last_fullsized":"2024-06-04T04:57:29.925821+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:59:50.147517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":222,"num_read_kb":5,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946315+0000","last_change":"2024-06-04T04:54:19.513534+0000","last_active":"2024-06-04T04:57:09.946315+0000","last_peered":"2024-06-04T04:57:09.946315+0000","last_clean":"2024-06-04T04:57:09.946315+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:57:09.946315+0000","last_undegraded":"2024-06-04T04:57:09.946315+0000","last_fullsized":"2024-06-04T04:57:09.946315+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:16:32.240932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.946648+0000","last_change":"2024-06-04T04:53:36.371236+0000","last_active":"2024-06-04T04:57:09.946648+0000","last_peered":"2024-06-04T04:57:09.946648+0000","last_clean":"2024-06-04T04:57:09.946648+0000","last_became_active":"2024-06-04T04:53:36.370876+0000","last_became_peered":"2024-06-04T04:53:36.370876+0000","last_unstale":"2024-06-04T04:57:09.946648+0000","last_undegraded":"2024-06-04T04:57:09.946648+0000","last_fullsized":"2024-06-04T04:57:09.946648+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:15:56.751317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":252,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:29.775931+0000","last_change":"2024-06-04T04:53:33.355642+0000","last_active":"2024-06-04T04:57:29.775931+0000","last_peered":"2024-06-04T04:57:29.775931+0000","last_clean":"2024-06-04T04:57:29.775931+0000","last_became_active":"2024-06-04T04:53:33.355187+0000","last_became_peered":"2024-06-04T04:53:33.355187+0000","last_unstale":"2024-06-04T04:57:29.775931+0000","last_undegraded":"2024-06-04T04:57:29.775931+0000","last_fullsized":"2024-06-04T04:57:29.775931+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:32:28.331518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":218,"num_read_kb":218,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.047902+0000","last_change":"2024-06-04T04:54:19.631750+0000","last_active":"2024-06-04T04:57:10.047902+0000","last_peered":"2024-06-04T04:57:10.047902+0000","last_clean":"2024-06-04T04:57:10.047902+0000","last_became_active":"2024-06-04T04:54:19.631485+0000","last_became_peered":"2024-06-04T04:54:19.631485+0000","last_unstale":"2024-06-04T04:57:10.047902+0000","last_undegraded":"2024-06-04T04:57:10.047902+0000","last_fullsized":"2024-06-04T04:57:10.047902+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:29:44.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466037+0000","last_change":"2024-06-04T04:53:33.340653+0000","last_active":"2024-06-04T04:57:10.466037+0000","last_peered":"2024-06-04T04:57:10.466037+0000","last_clean":"2024-06-04T04:57:10.466037+0000","last_became_active":"2024-06-04T04:53:33.340250+0000","last_became_peered":"2024-06-04T04:53:33.340250+0000","last_unstale":"2024-06-04T04:57:10.466037+0000","last_undegraded":"2024-06-04T04:57:10.466037+0000","last_fullsized":"2024-06-04T04:57:10.466037+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:46:20.329514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"82'99","reported_seq":131,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:12.052884+0000","last_change":"2024-06-04T04:52:09.181155+0000","last_active":"2024-06-04T04:57:12.052884+0000","last_peered":"2024-06-04T04:57:12.052884+0000","last_clean":"2024-06-04T04:57:12.052884+0000","last_became_active":"2024-06-04T04:52:09.177002+0000","last_became_peered":"2024-06-04T04:52:09.177002+0000","last_unstale":"2024-06-04T04:57:12.052884+0000","last_undegraded":"2024-06-04T04:57:12.052884+0000","last_fullsized":"2024-06-04T04:57:12.052884+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:48:44.631948+0000","last_clean_scrub_stamp":"2024-06-04T04:48:44.631948+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:46:57.735302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215170+0000","last_change":"2024-06-04T04:53:36.372634+0000","last_active":"2024-06-04T04:57:10.215170+0000","last_peered":"2024-06-04T04:57:10.215170+0000","last_clean":"2024-06-04T04:57:10.215170+0000","last_became_active":"2024-06-04T04:53:36.372290+0000","last_became_peered":"2024-06-04T04:53:36.372290+0000","last_unstale":"2024-06-04T04:57:10.215170+0000","last_undegraded":"2024-06-04T04:57:10.215170+0000","last_fullsized":"2024-06-04T04:57:10.215170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:14:07.460382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213392+0000","last_change":"2024-06-04T04:53:33.348760+0000","last_active":"2024-06-04T04:57:10.213392+0000","last_peered":"2024-06-04T04:57:10.213392+0000","last_clean":"2024-06-04T04:57:10.213392+0000","last_became_active":"2024-06-04T04:53:33.348347+0000","last_became_peered":"2024-06-04T04:53:33.348347+0000","last_unstale":"2024-06-04T04:57:10.213392+0000","last_undegraded":"2024-06-04T04:57:10.213392+0000","last_fullsized":"2024-06-04T04:57:10.213392+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:36:20.641061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965676+0000","last_change":"2024-06-04T04:53:33.354840+0000","last_active":"2024-06-04T04:57:09.965676+0000","last_peered":"2024-06-04T04:57:09.965676+0000","last_clean":"2024-06-04T04:57:09.965676+0000","last_became_active":"2024-06-04T04:53:33.354287+0000","last_became_peered":"2024-06-04T04:53:33.354287+0000","last_unstale":"2024-06-04T04:57:09.965676+0000","last_undegraded":"2024-06-04T04:57:09.965676+0000","last_fullsized":"2024-06-04T04:57:09.965676+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:59:42.550098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215686+0000","last_change":"2024-06-04T04:53:36.372678+0000","last_active":"2024-06-04T04:57:10.215686+0000","last_peered":"2024-06-04T04:57:10.215686+0000","last_clean":"2024-06-04T04:57:10.215686+0000","last_became_active":"2024-06-04T04:53:36.372312+0000","last_became_peered":"2024-06-04T04:53:36.372312+0000","last_unstale":"2024-06-04T04:57:10.215686+0000","last_undegraded":"2024-06-04T04:57:10.215686+0000","last_fullsized":"2024-06-04T04:57:10.215686+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:34:41.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'104","reported_seq":225,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:23.396837+0000","last_change":"2024-06-04T04:53:33.348440+0000","last_active":"2024-06-04T04:57:23.396837+0000","last_peered":"2024-06-04T04:57:23.396837+0000","last_clean":"2024-06-04T04:57:23.396837+0000","last_became_active":"2024-06-04T04:53:33.348131+0000","last_became_peered":"2024-06-04T04:53:33.348131+0000","last_unstale":"2024-06-04T04:57:23.396837+0000","last_undegraded":"2024-06-04T04:57:23.396837+0000","last_fullsized":"2024-06-04T04:57:23.396837+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T14:14:11.552286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":190,"num_write_kb":1286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942741+0000","last_change":"2024-06-04T04:53:36.378669+0000","last_active":"2024-06-04T04:57:09.942741+0000","last_peered":"2024-06-04T04:57:09.942741+0000","last_clean":"2024-06-04T04:57:09.942741+0000","last_became_active":"2024-06-04T04:53:36.378233+0000","last_became_peered":"2024-06-04T04:53:36.378233+0000","last_unstale":"2024-06-04T04:57:09.942741+0000","last_undegraded":"2024-06-04T04:57:09.942741+0000","last_fullsized":"2024-06-04T04:57:09.942741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:36:45.870924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466139+0000","last_change":"2024-06-04T04:53:33.345078+0000","last_active":"2024-06-04T04:57:10.466139+0000","last_peered":"2024-06-04T04:57:10.466139+0000","last_clean":"2024-06-04T04:57:10.466139+0000","last_became_active":"2024-06-04T04:53:33.344708+0000","last_became_peered":"2024-06-04T04:53:33.344708+0000","last_unstale":"2024-06-04T04:57:10.466139+0000","last_undegraded":"2024-06-04T04:57:10.466139+0000","last_fullsized":"2024-06-04T04:57:10.466139+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:45:14.010758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466300+0000","last_change":"2024-06-04T04:53:36.364817+0000","last_active":"2024-06-04T04:57:10.466300+0000","last_peered":"2024-06-04T04:57:10.466300+0000","last_clean":"2024-06-04T04:57:10.466300+0000","last_became_active":"2024-06-04T04:53:36.364499+0000","last_became_peered":"2024-06-04T04:53:36.364499+0000","last_unstale":"2024-06-04T04:57:10.466300+0000","last_undegraded":"2024-06-04T04:57:10.466300+0000","last_fullsized":"2024-06-04T04:57:10.466300+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:39:30.518028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215210+0000","last_change":"2024-06-04T04:53:33.354527+0000","last_active":"2024-06-04T04:57:10.215210+0000","last_peered":"2024-06-04T04:57:10.215210+0000","last_clean":"2024-06-04T04:57:10.215210+0000","last_became_active":"2024-06-04T04:53:33.354175+0000","last_became_peered":"2024-06-04T04:53:33.354175+0000","last_unstale":"2024-06-04T04:57:10.215210+0000","last_undegraded":"2024-06-04T04:57:10.215210+0000","last_fullsized":"2024-06-04T04:57:10.215210+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:33:08.125428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965391+0000","last_change":"2024-06-04T04:53:36.373248+0000","last_active":"2024-06-04T04:57:09.965391+0000","last_peered":"2024-06-04T04:57:09.965391+0000","last_clean":"2024-06-04T04:57:09.965391+0000","last_became_active":"2024-06-04T04:53:36.372915+0000","last_became_peered":"2024-06-04T04:53:36.372915+0000","last_unstale":"2024-06-04T04:57:09.965391+0000","last_undegraded":"2024-06-04T04:57:09.965391+0000","last_fullsized":"2024-06-04T04:57:09.965391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:23:11.689349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942495+0000","last_change":"2024-06-04T04:53:33.340427+0000","last_active":"2024-06-04T04:57:09.942495+0000","last_peered":"2024-06-04T04:57:09.942495+0000","last_clean":"2024-06-04T04:57:09.942495+0000","last_became_active":"2024-06-04T04:53:33.340022+0000","last_became_peered":"2024-06-04T04:53:33.340022+0000","last_unstale":"2024-06-04T04:57:09.942495+0000","last_undegraded":"2024-06-04T04:57:09.942495+0000","last_fullsized":"2024-06-04T04:57:09.942495+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:56:16.496471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466096+0000","last_change":"2024-06-04T04:53:36.380114+0000","last_active":"2024-06-04T04:57:10.466096+0000","last_peered":"2024-06-04T04:57:10.466096+0000","last_clean":"2024-06-04T04:57:10.466096+0000","last_became_active":"2024-06-04T04:53:36.379392+0000","last_became_peered":"2024-06-04T04:53:36.379392+0000","last_unstale":"2024-06-04T04:57:10.466096+0000","last_undegraded":"2024-06-04T04:57:10.466096+0000","last_fullsized":"2024-06-04T04:57:10.466096+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:41:48.776439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"81'5","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466122+0000","last_change":"2024-06-04T04:54:19.519755+0000","last_active":"2024-06-04T04:57:10.466122+0000","last_peered":"2024-06-04T04:57:10.466122+0000","last_clean":"2024-06-04T04:57:10.466122+0000","last_became_active":"2024-06-04T04:54:19.513992+0000","last_became_peered":"2024-06-04T04:54:19.513992+0000","last_unstale":"2024-06-04T04:57:10.466122+0000","last_undegraded":"2024-06-04T04:57:10.466122+0000","last_fullsized":"2024-06-04T04:57:10.466122+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:01:01.951935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942779+0000","last_change":"2024-06-04T04:53:36.377420+0000","last_active":"2024-06-04T04:57:09.942779+0000","last_peered":"2024-06-04T04:57:09.942779+0000","last_clean":"2024-06-04T04:57:09.942779+0000","last_became_active":"2024-06-04T04:53:36.377109+0000","last_became_peered":"2024-06-04T04:53:36.377109+0000","last_unstale":"2024-06-04T04:57:09.942779+0000","last_undegraded":"2024-06-04T04:57:09.942779+0000","last_fullsized":"2024-06-04T04:57:09.942779+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:47:12.102396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.965575+0000","last_change":"2024-06-04T04:54:19.645328+0000","last_active":"2024-06-04T04:57:09.965575+0000","last_peered":"2024-06-04T04:57:09.965575+0000","last_clean":"2024-06-04T04:57:09.965575+0000","last_became_active":"2024-06-04T04:54:19.633416+0000","last_became_peered":"2024-06-04T04:54:19.633416+0000","last_unstale":"2024-06-04T04:57:09.965575+0000","last_undegraded":"2024-06-04T04:57:09.965575+0000","last_fullsized":"2024-06-04T04:57:09.965575+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:04:20.494641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213349+0000","last_change":"2024-06-04T04:53:36.381187+0000","last_active":"2024-06-04T04:57:10.213349+0000","last_peered":"2024-06-04T04:57:10.213349+0000","last_clean":"2024-06-04T04:57:10.213349+0000","last_became_active":"2024-06-04T04:53:36.380591+0000","last_became_peered":"2024-06-04T04:53:36.380591+0000","last_unstale":"2024-06-04T04:57:10.213349+0000","last_undegraded":"2024-06-04T04:57:10.213349+0000","last_fullsized":"2024-06-04T04:57:10.213349+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:26:43.122339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'6","reported_seq":254,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:29.814917+0000","last_change":"2024-06-04T04:53:33.355908+0000","last_active":"2024-06-04T04:57:29.814917+0000","last_peered":"2024-06-04T04:57:29.814917+0000","last_clean":"2024-06-04T04:57:29.814917+0000","last_became_active":"2024-06-04T04:53:33.355522+0000","last_became_peered":"2024-06-04T04:53:33.355522+0000","last_unstale":"2024-06-04T04:57:29.814917+0000","last_undegraded":"2024-06-04T04:57:29.814917+0000","last_fullsized":"2024-06-04T04:57:29.814917+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T05:16:44.639431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":220,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213745+0000","last_change":"2024-06-04T04:53:36.382737+0000","last_active":"2024-06-04T04:57:10.213745+0000","last_peered":"2024-06-04T04:57:10.213745+0000","last_clean":"2024-06-04T04:57:10.213745+0000","last_became_active":"2024-06-04T04:53:36.382353+0000","last_became_peered":"2024-06-04T04:53:36.382353+0000","last_unstale":"2024-06-04T04:57:10.213745+0000","last_undegraded":"2024-06-04T04:57:10.213745+0000","last_fullsized":"2024-06-04T04:57:10.213745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:18:01.476058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'67","reported_seq":127,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:23.044873+0000","last_change":"2024-06-04T04:53:33.347273+0000","last_active":"2024-06-04T04:57:23.044873+0000","last_peered":"2024-06-04T04:57:23.044873+0000","last_clean":"2024-06-04T04:57:23.044873+0000","last_became_active":"2024-06-04T04:53:33.346915+0000","last_became_peered":"2024-06-04T04:53:33.346915+0000","last_unstale":"2024-06-04T04:57:23.044873+0000","last_undegraded":"2024-06-04T04:57:23.044873+0000","last_fullsized":"2024-06-04T04:57:23.044873+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:17.235484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213559+0000","last_change":"2024-06-04T04:53:36.383454+0000","last_active":"2024-06-04T04:57:10.213559+0000","last_peered":"2024-06-04T04:57:10.213559+0000","last_clean":"2024-06-04T04:57:10.213559+0000","last_became_active":"2024-06-04T04:53:36.383105+0000","last_became_peered":"2024-06-04T04:53:36.383105+0000","last_unstale":"2024-06-04T04:57:10.213559+0000","last_undegraded":"2024-06-04T04:57:10.213559+0000","last_fullsized":"2024-06-04T04:57:10.213559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:04:02.147688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215701+0000","last_change":"2024-06-04T04:53:33.345435+0000","last_active":"2024-06-04T04:57:10.215701+0000","last_peered":"2024-06-04T04:57:10.215701+0000","last_clean":"2024-06-04T04:57:10.215701+0000","last_became_active":"2024-06-04T04:53:33.345077+0000","last_became_peered":"2024-06-04T04:53:33.345077+0000","last_unstale":"2024-06-04T04:57:10.215701+0000","last_undegraded":"2024-06-04T04:57:10.215701+0000","last_fullsized":"2024-06-04T04:57:10.215701+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:32.320368+0000","last_clean_scrub_stamp":"2024-06-04T04:53:32.320368+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T11:20:13.992259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213743+0000","last_change":"2024-06-04T04:53:36.385238+0000","last_active":"2024-06-04T04:57:10.213743+0000","last_peered":"2024-06-04T04:57:10.213743+0000","last_clean":"2024-06-04T04:57:10.213743+0000","last_became_active":"2024-06-04T04:53:36.384899+0000","last_became_peered":"2024-06-04T04:53:36.384899+0000","last_unstale":"2024-06-04T04:57:10.213743+0000","last_undegraded":"2024-06-04T04:57:10.213743+0000","last_fullsized":"2024-06-04T04:57:10.213743+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:09:42.629340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466183+0000","last_change":"2024-06-04T04:53:36.383450+0000","last_active":"2024-06-04T04:57:10.466183+0000","last_peered":"2024-06-04T04:57:10.466183+0000","last_clean":"2024-06-04T04:57:10.466183+0000","last_became_active":"2024-06-04T04:53:36.382678+0000","last_became_peered":"2024-06-04T04:53:36.382678+0000","last_unstale":"2024-06-04T04:57:10.466183+0000","last_undegraded":"2024-06-04T04:57:10.466183+0000","last_fullsized":"2024-06-04T04:57:10.466183+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T13:36:28.932636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219063+0000","last_change":"2024-06-04T04:54:19.632563+0000","last_active":"2024-06-04T04:57:10.219063+0000","last_peered":"2024-06-04T04:57:10.219063+0000","last_clean":"2024-06-04T04:57:10.219063+0000","last_became_active":"2024-06-04T04:54:19.632070+0000","last_became_peered":"2024-06-04T04:54:19.632070+0000","last_unstale":"2024-06-04T04:57:10.219063+0000","last_undegraded":"2024-06-04T04:57:10.219063+0000","last_fullsized":"2024-06-04T04:57:10.219063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:07:16.552958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219185+0000","last_change":"2024-06-04T04:53:36.381408+0000","last_active":"2024-06-04T04:57:10.219185+0000","last_peered":"2024-06-04T04:57:10.219185+0000","last_clean":"2024-06-04T04:57:10.219185+0000","last_became_active":"2024-06-04T04:53:36.381078+0000","last_became_peered":"2024-06-04T04:53:36.381078+0000","last_unstale":"2024-06-04T04:57:10.219185+0000","last_undegraded":"2024-06-04T04:57:10.219185+0000","last_fullsized":"2024-06-04T04:57:10.219185+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:28:48.988591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466464+0000","last_change":"2024-06-04T04:53:36.360168+0000","last_active":"2024-06-04T04:57:10.466464+0000","last_peered":"2024-06-04T04:57:10.466464+0000","last_clean":"2024-06-04T04:57:10.466464+0000","last_became_active":"2024-06-04T04:53:36.359691+0000","last_became_peered":"2024-06-04T04:53:36.359691+0000","last_unstale":"2024-06-04T04:57:10.466464+0000","last_undegraded":"2024-06-04T04:57:10.466464+0000","last_fullsized":"2024-06-04T04:57:10.466464+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T07:55:17.418339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215688+0000","last_change":"2024-06-04T04:53:36.363018+0000","last_active":"2024-06-04T04:57:10.215688+0000","last_peered":"2024-06-04T04:57:10.215688+0000","last_clean":"2024-06-04T04:57:10.215688+0000","last_became_active":"2024-06-04T04:53:36.361595+0000","last_became_peered":"2024-06-04T04:53:36.361595+0000","last_unstale":"2024-06-04T04:57:10.215688+0000","last_undegraded":"2024-06-04T04:57:10.215688+0000","last_fullsized":"2024-06-04T04:57:10.215688+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T08:00:21.377322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466289+0000","last_change":"2024-06-04T04:53:36.375377+0000","last_active":"2024-06-04T04:57:10.466289+0000","last_peered":"2024-06-04T04:57:10.466289+0000","last_clean":"2024-06-04T04:57:10.466289+0000","last_became_active":"2024-06-04T04:53:36.373459+0000","last_became_peered":"2024-06-04T04:53:36.373459+0000","last_unstale":"2024-06-04T04:57:10.466289+0000","last_undegraded":"2024-06-04T04:57:10.466289+0000","last_fullsized":"2024-06-04T04:57:10.466289+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T16:51:31.686969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.219589+0000","last_change":"2024-06-04T04:53:36.373850+0000","last_active":"2024-06-04T04:57:10.219589+0000","last_peered":"2024-06-04T04:57:10.219589+0000","last_clean":"2024-06-04T04:57:10.219589+0000","last_became_active":"2024-06-04T04:53:36.371951+0000","last_became_peered":"2024-06-04T04:53:36.371951+0000","last_unstale":"2024-06-04T04:57:10.219589+0000","last_undegraded":"2024-06-04T04:57:10.219589+0000","last_fullsized":"2024-06-04T04:57:10.219589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:51:06.452703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942719+0000","last_change":"2024-06-04T04:53:36.374871+0000","last_active":"2024-06-04T04:57:09.942719+0000","last_peered":"2024-06-04T04:57:09.942719+0000","last_clean":"2024-06-04T04:57:09.942719+0000","last_became_active":"2024-06-04T04:53:36.374486+0000","last_became_peered":"2024-06-04T04:53:36.374486+0000","last_unstale":"2024-06-04T04:57:09.942719+0000","last_undegraded":"2024-06-04T04:57:09.942719+0000","last_fullsized":"2024-06-04T04:57:09.942719+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T12:15:13.337374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.213230+0000","last_change":"2024-06-04T04:54:19.513540+0000","last_active":"2024-06-04T04:57:10.213230+0000","last_peered":"2024-06-04T04:57:10.213230+0000","last_clean":"2024-06-04T04:57:10.213230+0000","last_became_active":"2024-06-04T04:54:19.513109+0000","last_became_peered":"2024-06-04T04:54:19.513109+0000","last_unstale":"2024-06-04T04:57:10.213230+0000","last_undegraded":"2024-06-04T04:57:10.213230+0000","last_fullsized":"2024-06-04T04:57:10.213230+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:09:27.993585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:09.942434+0000","last_change":"2024-06-04T04:53:36.373055+0000","last_active":"2024-06-04T04:57:09.942434+0000","last_peered":"2024-06-04T04:57:09.942434+0000","last_clean":"2024-06-04T04:57:09.942434+0000","last_became_active":"2024-06-04T04:53:36.372754+0000","last_became_peered":"2024-06-04T04:53:36.372754+0000","last_unstale":"2024-06-04T04:57:09.942434+0000","last_undegraded":"2024-06-04T04:57:09.942434+0000","last_fullsized":"2024-06-04T04:57:09.942434+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T06:22:27.554200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215415+0000","last_change":"2024-06-04T04:53:36.378897+0000","last_active":"2024-06-04T04:57:10.215415+0000","last_peered":"2024-06-04T04:57:10.215415+0000","last_clean":"2024-06-04T04:57:10.215415+0000","last_became_active":"2024-06-04T04:53:36.378607+0000","last_became_peered":"2024-06-04T04:53:36.378607+0000","last_unstale":"2024-06-04T04:57:10.215415+0000","last_undegraded":"2024-06-04T04:57:10.215415+0000","last_fullsized":"2024-06-04T04:57:10.215415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T15:09:03.395491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.215502+0000","last_change":"2024-06-04T04:53:36.378240+0000","last_active":"2024-06-04T04:57:10.215502+0000","last_peered":"2024-06-04T04:57:10.215502+0000","last_clean":"2024-06-04T04:57:10.215502+0000","last_became_active":"2024-06-04T04:53:36.377909+0000","last_became_peered":"2024-06-04T04:53:36.377909+0000","last_unstale":"2024-06-04T04:57:10.215502+0000","last_undegraded":"2024-06-04T04:57:10.215502+0000","last_fullsized":"2024-06-04T04:57:10.215502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T09:15:25.288340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-06-04T04:57:10.466255+0000","last_change":"2024-06-04T04:53:36.379792+0000","last_active":"2024-06-04T04:57:10.466255+0000","last_peered":"2024-06-04T04:57:10.466255+0000","last_clean":"2024-06-04T04:57:10.466255+0000","last_became_active":"2024-06-04T04:53:36.379438+0000","last_became_peered":"2024-06-04T04:53:36.379438+0000","last_unstale":"2024-06-04T04:57:10.466255+0000","last_undegraded":"2024-06-04T04:57:10.466255+0000","last_fullsized":"2024-06-04T04:57:10.466255+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-04T04:53:35.343027+0000","last_clean_scrub_stamp":"2024-06-04T04:53:35.343027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-05T10:23:57.617877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108276,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":802806,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2560,"num_read_kb":1503,"num_write":511,"num_write_kb":2390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":798720,"data_stored":2408418,"data_compressed":108972,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":387,"ondisk_log_size":387,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678078,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":892928,"data_stored":439038,"data_compressed":8651,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874306,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1101824,"data_stored":1429949,"data_compressed":61778,"data_compressed_allocated":81920,"data_compressed_original":929792,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037832,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":831691,"data_compressed":27940,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234062,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":831397,"data_compressed":27940,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430289,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":1033773,"data_compressed":44743,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593816,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":901120,"data_stored":439999,"data_compressed":8651,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574961,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":978944,"data_stored":513234,"data_compressed":10484,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738486,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":921600,"data_stored":440024,"data_compressed":8651,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411435,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":511961,"data_compressed":10484,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149799,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":987136,"data_stored":1159465,"data_compressed":43142,"data_compressed_allocated":57344,"data_compressed_original":724992,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953570,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":901120,"data_stored":438546,"data_compressed":8651,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790045,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":1359383,"data_compressed":59945,"data_compressed_allocated":77824,"data_compressed_original":921600,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":75881,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":74608,"data_compressed":1833,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":721878,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":331545,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":394044,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":402111,"data_compressed":17035,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-04T04:57:33.932 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:57:33.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:33 smithi112 ceph-mon[36453]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 44 op/s 2024-06-04T04:57:33.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:33 smithi098 ceph-mon[35569]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 44 op/s 2024-06-04T04:57:34.169 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:34.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:34 smithi070 ceph-mon[32631]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:34.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:34 smithi112 ceph-mon[36453]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:34.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:34 smithi098 ceph-mon[35569]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-04T04:57:35.587 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:35 smithi070 ceph-mon[32631]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-06-04T04:57:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:35 smithi112 ceph-mon[36453]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-06-04T04:57:35.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.963+0000 7f848c996700 1 -- 172.21.15.70:0/3684558417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8488109670 msgr2=0x7f8488109a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:35.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.963+0000 7f848c996700 1 --2- 172.21.15.70:0/3684558417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8488109670 0x7f8488109a50 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f84780099c0 tx=0x7f8478009cd0 comp rx=0 tx=0).stop 2024-06-04T04:57:35.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.963+0000 7f848c996700 1 -- 172.21.15.70:0/3684558417 shutdown_connections 2024-06-04T04:57:35.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.963+0000 7f848c996700 1 --2- 172.21.15.70:0/3684558417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8488103f90 0x7f8488110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:35.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.963+0000 7f848c996700 1 --2- 172.21.15.70:0/3684558417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8488109670 0x7f8488109a50 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:35.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.963+0000 7f848c996700 1 --2- 172.21.15.70:0/3684558417 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84881035d0 0x7f8488103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:35.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.963+0000 7f848c996700 1 -- 172.21.15.70:0/3684558417 >> 172.21.15.70:0/3684558417 conn(0x7f848807a730 msgr2=0x7f848807ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:35.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.963+0000 7f848c996700 1 -- 172.21.15.70:0/3684558417 shutdown_connections 2024-06-04T04:57:35.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.963+0000 7f848c996700 1 -- 172.21.15.70:0/3684558417 wait complete. 2024-06-04T04:57:35.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.964+0000 7f848c996700 1 Processor -- start 2024-06-04T04:57:35.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.964+0000 7f848c996700 1 -- start start 2024-06-04T04:57:35.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.964+0000 7f848c996700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84881035d0 0x7f848819f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:35.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.964+0000 7f848c996700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8488103f90 0x7f848819fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:35.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.964+0000 7f848c996700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8488109670 0x7f8488199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:35.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.964+0000 7f848c996700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8488113680 con 0x7f84881035d0 2024-06-04T04:57:35.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.964+0000 7f848c996700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8488113500 con 0x7f8488109670 2024-06-04T04:57:35.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.964+0000 7f848c996700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8488113800 con 0x7f8488103f90 2024-06-04T04:57:35.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.964+0000 7f848659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84881035d0 0x7f848819f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:35.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.964+0000 7f848659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84881035d0 0x7f848819f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35822/0 (socket says 172.21.15.70:35822) 2024-06-04T04:57:35.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.964+0000 7f848659c700 1 -- 172.21.15.70:0/3295247815 learned_addr learned my addr 172.21.15.70:0/3295247815 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:35.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.965+0000 7f8485d9b700 1 --2- 172.21.15.70:0/3295247815 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8488103f90 0x7f848819fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:35.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.965+0000 7f8486d9d700 1 --2- 172.21.15.70:0/3295247815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8488109670 0x7f8488199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:35.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.965+0000 7f848659c700 1 -- 172.21.15.70:0/3295247815 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8488103f90 msgr2=0x7f848819fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:35.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.965+0000 7f848659c700 1 --2- 172.21.15.70:0/3295247815 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8488103f90 0x7f848819fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:35.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.965+0000 7f848659c700 1 -- 172.21.15.70:0/3295247815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8488109670 msgr2=0x7f8488199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:35.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.965+0000 7f848659c700 1 --2- 172.21.15.70:0/3295247815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8488109670 0x7f8488199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:35.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.965+0000 7f848659c700 1 -- 172.21.15.70:0/3295247815 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f848819a0e0 con 0x7f84881035d0 2024-06-04T04:57:35.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.965+0000 7f8486d9d700 1 --2- 172.21.15.70:0/3295247815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8488109670 0x7f8488199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:35.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.965+0000 7f848659c700 1 --2- 172.21.15.70:0/3295247815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84881035d0 0x7f848819f7c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8478000c00 tx=0x7f847800f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:35.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.965+0000 7f84777fe700 1 -- 172.21.15.70:0/3295247815 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8478004a30 con 0x7f84881035d0 2024-06-04T04:57:35.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.965+0000 7f84777fe700 1 -- 172.21.15.70:0/3295247815 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f84780054b0 con 0x7f84881035d0 2024-06-04T04:57:35.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.965+0000 7f848c996700 1 -- 172.21.15.70:0/3295247815 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f848819a370 con 0x7f84881035d0 2024-06-04T04:57:35.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.965+0000 7f84777fe700 1 -- 172.21.15.70:0/3295247815 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f847800fc20 con 0x7f84881035d0 2024-06-04T04:57:35.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.965+0000 7f848c996700 1 -- 172.21.15.70:0/3295247815 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8488075770 con 0x7f84881035d0 2024-06-04T04:57:35.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.966+0000 7f848c996700 1 -- 172.21.15.70:0/3295247815 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f848810bb00 con 0x7f84881035d0 2024-06-04T04:57:35.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.968+0000 7f84777fe700 1 -- 172.21.15.70:0/3295247815 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7f8478004cf0 con 0x7f84881035d0 2024-06-04T04:57:35.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.968+0000 7f84777fe700 1 --2- 172.21.15.70:0/3295247815 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f84700b2a80 0x7f84700b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:35.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.969+0000 7f8485d9b700 1 --2- 172.21.15.70:0/3295247815 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f84700b2a80 0x7f84700b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:35.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.969+0000 7f84777fe700 1 -- 172.21.15.70:0/3295247815 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f84780e7ba0 con 0x7f84881035d0 2024-06-04T04:57:35.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.969+0000 7f8485d9b700 1 --2- 172.21.15.70:0/3295247815 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f84700b2a80 0x7f84700b4f40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f847c006fd0 tx=0x7f847c008040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:35.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:35.972+0000 7f84777fe700 1 -- 172.21.15.70:0/3295247815 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8478064df0 con 0x7f84881035d0 2024-06-04T04:57:35.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:35 smithi098 ceph-mon[35569]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-06-04T04:57:36.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:36.222+0000 7f848c996700 1 -- 172.21.15.70:0/3295247815 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f848819aeb0 con 0x7f84881035d0 2024-06-04T04:57:36.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:36.223+0000 7f84777fe700 1 -- 172.21.15.70:0/3295247815 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f8478013070 con 0x7f84881035d0 2024-06-04T04:57:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:57:36.226 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:57:09.013178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T04:57:36.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:36.225+0000 7f848c996700 1 -- 172.21.15.70:0/3295247815 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f84700b2a80 msgr2=0x7f84700b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:36.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:36.225+0000 7f848c996700 1 --2- 172.21.15.70:0/3295247815 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f84700b2a80 0x7f84700b4f40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f847c006fd0 tx=0x7f847c008040 comp rx=0 tx=0).stop 2024-06-04T04:57:36.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:36.225+0000 7f848c996700 1 -- 172.21.15.70:0/3295247815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84881035d0 msgr2=0x7f848819f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:36.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:36.225+0000 7f848c996700 1 --2- 172.21.15.70:0/3295247815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84881035d0 0x7f848819f7c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8478000c00 tx=0x7f847800f460 comp rx=0 tx=0).stop 2024-06-04T04:57:36.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:36.225+0000 7f848c996700 1 -- 172.21.15.70:0/3295247815 shutdown_connections 2024-06-04T04:57:36.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:36.225+0000 7f848c996700 1 --2- 172.21.15.70:0/3295247815 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f84700b2a80 0x7f84700b4f40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:36.225+0000 7f848c996700 1 --2- 172.21.15.70:0/3295247815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8488109670 0x7f8488199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:36.225+0000 7f848c996700 1 --2- 172.21.15.70:0/3295247815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84881035d0 0x7f848819f7c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:36.225+0000 7f848c996700 1 --2- 172.21.15.70:0/3295247815 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8488103f90 0x7f848819fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:36.225+0000 7f848c996700 1 -- 172.21.15.70:0/3295247815 >> 172.21.15.70:0/3295247815 conn(0x7f848807a730 msgr2=0x7f8488102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:36.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:36.225+0000 7f848c996700 1 -- 172.21.15.70:0/3295247815 shutdown_connections 2024-06-04T04:57:36.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:36.225+0000 7f848c996700 1 -- 172.21.15.70:0/3295247815 wait complete. 2024-06-04T04:57:36.230 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-06-04T04:57:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.619+0000 7fc96de2a640 1 -- 172.21.15.70:0/3299758232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc968103ce0 msgr2=0x7fc9681040c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.619+0000 7fc96de2a640 1 --2- 172.21.15.70:0/3299758232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc968103ce0 0x7fc9681040c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc958009ca0 tx=0x7fc95802a9e0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.620+0000 7fc96de2a640 1 -- 172.21.15.70:0/3299758232 shutdown_connections 2024-06-04T04:57:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.620+0000 7fc96de2a640 1 --2- 172.21.15.70:0/3299758232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc96810d020 0x7fc96810f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.620+0000 7fc96de2a640 1 --2- 172.21.15.70:0/3299758232 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc968104600 0x7fc96810cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.620+0000 7fc96de2a640 1 --2- 172.21.15.70:0/3299758232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc968103ce0 0x7fc9681040c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.620+0000 7fc96de2a640 1 -- 172.21.15.70:0/3299758232 >> 172.21.15.70:0/3299758232 conn(0x7fc968078ad0 msgr2=0x7fc968078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.620+0000 7fc96de2a640 1 -- 172.21.15.70:0/3299758232 shutdown_connections 2024-06-04T04:57:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.620+0000 7fc96de2a640 1 -- 172.21.15.70:0/3299758232 wait complete. 2024-06-04T04:57:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.620+0000 7fc96de2a640 1 Processor -- start 2024-06-04T04:57:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.620+0000 7fc96de2a640 1 -- start start 2024-06-04T04:57:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc96de2a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc968103ce0 0x7fc968199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc96de2a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc968104600 0x7fc968199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc96de2a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc96810d020 0x7fc96819da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc96de2a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc968112b80 con 0x7fc968104600 2024-06-04T04:57:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc96de2a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc968112a00 con 0x7fc96810d020 2024-06-04T04:57:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc96de2a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc968112d00 con 0x7fc968103ce0 2024-06-04T04:57:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc966ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc968104600 0x7fc968199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc966ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc968104600 0x7fc968199680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52448/0 (socket says 172.21.15.70:52448) 2024-06-04T04:57:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc966ffd640 1 -- 172.21.15.70:0/1694917006 learned_addr learned my addr 172.21.15.70:0/1694917006 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc9677fe640 1 --2- 172.21.15.70:0/1694917006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc968103ce0 0x7fc968199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc967fff640 1 --2- 172.21.15.70:0/1694917006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc96810d020 0x7fc96819da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc966ffd640 1 -- 172.21.15.70:0/1694917006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc968103ce0 msgr2=0x7fc968199140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc966ffd640 1 --2- 172.21.15.70:0/1694917006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc968103ce0 0x7fc968199140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc966ffd640 1 -- 172.21.15.70:0/1694917006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc96810d020 msgr2=0x7fc96819da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc966ffd640 1 --2- 172.21.15.70:0/1694917006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc96810d020 0x7fc96819da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc966ffd640 1 -- 172.21.15.70:0/1694917006 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc96819e130 con 0x7fc968104600 2024-06-04T04:57:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc966ffd640 1 --2- 172.21.15.70:0/1694917006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc968104600 0x7fc968199680 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc95000c890 tx=0x7fc95000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc964ff9640 1 -- 172.21.15.70:0/1694917006 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc950004220 con 0x7fc968104600 2024-06-04T04:57:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc964ff9640 1 -- 172.21.15.70:0/1694917006 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc950007e40 con 0x7fc968104600 2024-06-04T04:57:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc96de2a640 1 -- 172.21.15.70:0/1694917006 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc96819e420 con 0x7fc968104600 2024-06-04T04:57:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc964ff9640 1 -- 172.21.15.70:0/1694917006 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc950004a00 con 0x7fc968104600 2024-06-04T04:57:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.621+0000 7fc96de2a640 1 -- 172.21.15.70:0/1694917006 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9681a5cb0 con 0x7fc968104600 2024-06-04T04:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.624+0000 7fc964ff9640 1 -- 172.21.15.70:0/1694917006 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7fc9500079b0 con 0x7fc968104600 2024-06-04T04:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.624+0000 7fc970c23740 1 Processor -- start 2024-06-04T04:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.624+0000 7fc970c23740 1 -- start start 2024-06-04T04:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.624+0000 7fc970c23740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983a820 0x564b0983ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.624+0000 7fc970c23740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b0983b140 0x564b098454c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.624+0000 7fc970c23740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b09847df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.624+0000 7fc970c23740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564b097acc30 con 0x564b09845a00 2024-06-04T04:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.624+0000 7fc970c23740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564b0979c7d0 con 0x564b0983b140 2024-06-04T04:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.624+0000 7fc970c23740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564b097a0810 con 0x564b0983a820 2024-06-04T04:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.624+0000 7fc966ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b09847df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.624+0000 7fc966ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b09847df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52450/0 (socket says 172.21.15.70:52450) 2024-06-04T04:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.624+0000 7fc966ffd640 1 -- 172.21.15.70:0/1688369475 learned_addr learned my addr 172.21.15.70:0/1688369475 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.624+0000 7fc9677fe640 1 --2- 172.21.15.70:0/1688369475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b0983b140 0x564b098454c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.624+0000 7fc967fff640 1 --2- 172.21.15.70:0/1688369475 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983a820 0x564b0983ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.624+0000 7fc966ffd640 1 -- 172.21.15.70:0/1688369475 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983a820 msgr2=0x564b0983ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.624+0000 7fc966ffd640 1 --2- 172.21.15.70:0/1688369475 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983a820 0x564b0983ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.625+0000 7fc966ffd640 1 -- 172.21.15.70:0/1688369475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b0983b140 msgr2=0x564b098454c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.625+0000 7fc966ffd640 1 --2- 172.21.15.70:0/1688369475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b0983b140 0x564b098454c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.625+0000 7fc966ffd640 1 -- 172.21.15.70:0/1688369475 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b094b1210 con 0x564b09845a00 2024-06-04T04:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.625+0000 7fc966ffd640 1 --2- 172.21.15.70:0/1688369475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b09847df0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc9500dfab0 tx=0x7fc9500e8a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac4c979c7fbba6aa server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.625+0000 7fc95dffb640 1 -- 172.21.15.70:0/1688369475 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9500f5070 con 0x564b09845a00 2024-06-04T04:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.625+0000 7fc964ff9640 1 --2- 172.21.15.70:0/1694917006 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9540b2b10 0x7fc9540b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.625+0000 7fc95dffb640 1 -- 172.21.15.70:0/1688369475 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9500a9100 con 0x564b09845a00 2024-06-04T04:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.625+0000 7fc95dffb640 1 -- 172.21.15.70:0/1688369475 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9500bd040 con 0x564b09845a00 2024-06-04T04:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.625+0000 7fc967fff640 1 --2- 172.21.15.70:0/1694917006 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9540b2b10 0x7fc9540b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.625+0000 7fc964ff9640 1 -- 172.21.15.70:0/1694917006 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc9500e2480 con 0x7fc968104600 2024-06-04T04:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc970c23740 1 -- 172.21.15.70:0/1688369475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 msgr2=0x564b09847df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc967fff640 1 --2- 172.21.15.70:0/1694917006 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9540b2b10 0x7fc9540b4fd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc960005fd0 tx=0x7fc96000d5b0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc970c23740 1 --2- 172.21.15.70:0/1688369475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b09847df0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc9500dfab0 tx=0x7fc9500e8a50 comp rx=0 tx=0).stop 2024-06-04T04:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc970c23740 1 -- 172.21.15.70:0/1688369475 shutdown_connections 2024-06-04T04:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc970c23740 1 --2- 172.21.15.70:0/1688369475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b0983b140 0x564b098454c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc970c23740 1 --2- 172.21.15.70:0/1688369475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b09847df0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc970c23740 1 --2- 172.21.15.70:0/1688369475 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983a820 0x564b0983ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc970c23740 1 -- 172.21.15.70:0/1688369475 >> 172.21.15.70:0/1688369475 conn(0x564b09836d30 msgr2=0x564b09837110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc970c23740 1 -- 172.21.15.70:0/1688369475 shutdown_connections 2024-06-04T04:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc970c23740 1 -- 172.21.15.70:0/1688369475 wait complete. 2024-06-04T04:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc970c23740 1 -- start start 2024-06-04T04:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc970c23740 1 Processor -- start 2024-06-04T04:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc970c23740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983a820 0x564b098429c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc970c23740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b0983b140 0x564b09842f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc970c23740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b09843440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc970c23740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564b09482790 con 0x564b09845a00 2024-06-04T04:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc970c23740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564b09758890 con 0x564b0983b140 2024-06-04T04:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.626+0000 7fc970c23740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564b09848630 con 0x564b0983a820 2024-06-04T04:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc970c23740 10 client.0 _ll_register_callbacks cb 0x564b094b0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc967fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b09843440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc967fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b09843440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52464/0 (socket says 172.21.15.70:52464) 2024-06-04T04:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc967fff640 1 -- 172.21.15.70:0/1257525363 learned_addr learned my addr 172.21.15.70:0/1257525363 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc9677fe640 1 --2- 172.21.15.70:0/1257525363 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983a820 0x564b098429c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc966ffd640 1 --2- 172.21.15.70:0/1257525363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b0983b140 0x564b09842f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc967fff640 1 -- 172.21.15.70:0/1257525363 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983a820 msgr2=0x564b098429c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc967fff640 1 --2- 172.21.15.70:0/1257525363 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983a820 0x564b098429c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc967fff640 1 -- 172.21.15.70:0/1257525363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b0983b140 msgr2=0x564b09842f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc967fff640 1 --2- 172.21.15.70:0/1257525363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b0983b140 0x564b09842f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc967fff640 1 -- 172.21.15.70:0/1257525363 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b0957d620 con 0x564b09845a00 2024-06-04T04:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc967fff640 1 --2- 172.21.15.70:0/1257525363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b09843440 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc9600126d0 tx=0x7fc9600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7014b25afd745f5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc946ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc946ffd640 1 -- 172.21.15.70:0/1257525363 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc960005220 con 0x564b09845a00 2024-06-04T04:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc946ffd640 1 -- 172.21.15.70:0/1257525363 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc9600053c0 con 0x564b09845a00 2024-06-04T04:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc970c23740 1 -- 172.21.15.70:0/1257525363 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9500a9100 con 0x564b09845a00 2024-06-04T04:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc946ffd640 1 -- 172.21.15.70:0/1257525363 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc9600055c0 con 0x564b09845a00 2024-06-04T04:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc970c23740 10 client.15453 fetch_fsmap learned FSMap version 29 2024-06-04T04:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.627+0000 7fc970c23740 1 -- 172.21.15.70:0/1257525363 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564b09854410 con 0x564b09845a00 2024-06-04T04:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.628+0000 7fc946ffd640 1 -- 172.21.15.70:0/1257525363 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9500a9100 con 0x564b09845a00 2024-06-04T04:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.628+0000 7fc946ffd640 1 -- 172.21.15.70:0/1257525363 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc96001fa80 con 0x564b09845a00 2024-06-04T04:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.628+0000 7fc970c23740 10 client.15453 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.628+0000 7fc970c23740 10 client.15453 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:57:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.628+0000 7fc970c23740 20 client.15453 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.628+0000 7fc970c23740 1 --2- 172.21.15.70:0/1257525363 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564b098558f0 0x564b09857ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.628+0000 7fc970c23740 4 client.15453 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:57:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.628+0000 7fc970c23740 1 -- 172.21.15.70:0/1257525363 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b097aeb80 con 0x564b098558f0 2024-06-04T04:57:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.628+0000 7fc9677fe640 1 --2- 172.21.15.70:0/1257525363 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564b098558f0 0x564b09857ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.628+0000 7fc946ffd640 10 client.15453 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.628+0000 7fc9677fe640 1 --2- 172.21.15.70:0/1257525363 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564b098558f0 0x564b09857ce0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd8aa4717c3d110b server_cookie=83d5d4d387ba1269 in_seq=0 out_seq=0 2024-06-04T04:57:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.629+0000 7fc946ffd640 1 -- 172.21.15.70:0/1257525363 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564b097aeb80 con 0x564b098558f0 2024-06-04T04:57:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.629+0000 7fc946ffd640 10 client.15453 handle_command_reply: tid=0 2024-06-04T04:57:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.629+0000 7fc970c23740 1 client.15453 shutdown 2024-06-04T04:57:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.629+0000 7fc970c23740 1 -- 172.21.15.70:0/1257525363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 msgr2=0x564b09843440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.629+0000 7fc970c23740 1 --2- 172.21.15.70:0/1257525363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b09843440 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc9600126d0 tx=0x7fc9600046c0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.629+0000 7fc970c23740 1 -- 172.21.15.70:0/1257525363 shutdown_connections 2024-06-04T04:57:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.629+0000 7fc970c23740 1 --2- 172.21.15.70:0/1257525363 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564b098558f0 0x564b09857ce0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.629+0000 7fc970c23740 1 --2- 172.21.15.70:0/1257525363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b0983b140 0x564b09842f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.629+0000 7fc970c23740 1 --2- 172.21.15.70:0/1257525363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b09843440 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.629+0000 7fc946ffd640 0 client.15453 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.629+0000 7fc970c23740 1 --2- 172.21.15.70:0/1257525363 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983a820 0x564b098429c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.629+0000 7fc970c23740 1 -- 172.21.15.70:0/1257525363 >> 172.21.15.70:0/1257525363 conn(0x564b09836d30 msgr2=0x564b09835fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.629+0000 7fc970c23740 1 -- 172.21.15.70:0/1257525363 shutdown_connections 2024-06-04T04:57:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.630+0000 7fc970c23740 1 -- 172.21.15.70:0/1257525363 wait complete. 2024-06-04T04:57:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.630+0000 7fc970c23740 20 client.15453 trim_cache size 0 max 16384 2024-06-04T04:57:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.644+0000 7fc970c23740 1 Processor -- start 2024-06-04T04:57:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc970c23740 1 -- start start 2024-06-04T04:57:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc970c23740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b0983a820 0x564b0983d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc970c23740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983b140 0x564b0985d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc970c23740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b09845a00 0x564b0985db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc970c23740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc9681ae670 con 0x564b0983a820 2024-06-04T04:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc970c23740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9681b01b0 con 0x564b09845a00 2024-06-04T04:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc970c23740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc9681aed40 con 0x564b0983b140 2024-06-04T04:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc9677fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b0983a820 0x564b0983d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc9677fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b0983a820 0x564b0983d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52478/0 (socket says 172.21.15.70:52478) 2024-06-04T04:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc9677fe640 1 -- 172.21.15.70:0/596126905 learned_addr learned my addr 172.21.15.70:0/596126905 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc9677fe640 1 -- 172.21.15.70:0/596126905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983b140 msgr2=0x564b0985d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc9677fe640 1 --2- 172.21.15.70:0/596126905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983b140 0x564b0985d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc967fff640 1 --2- 172.21.15.70:0/596126905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b09845a00 0x564b0985db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc9677fe640 1 -- 172.21.15.70:0/596126905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b09845a00 msgr2=0x564b0985db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc9677fe640 1 --2- 172.21.15.70:0/596126905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b09845a00 0x564b0985db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc9677fe640 1 -- 172.21.15.70:0/596126905 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b09836b60 con 0x564b0983a820 2024-06-04T04:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc9677fe640 1 --2- 172.21.15.70:0/596126905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b0983a820 0x564b0983d1e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc958006010 tx=0x7fc95802e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2bd69e417f30c70 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc95d7fa640 1 -- 172.21.15.70:0/596126905 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc958033040 con 0x564b0983a820 2024-06-04T04:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc95d7fa640 1 -- 172.21.15.70:0/596126905 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc958003e00 con 0x564b0983a820 2024-06-04T04:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.645+0000 7fc95d7fa640 1 -- 172.21.15.70:0/596126905 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95802f7e0 con 0x564b0983a820 2024-06-04T04:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.646+0000 7fc970c23740 1 -- 172.21.15.70:0/596126905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b0983a820 msgr2=0x564b0983d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.646+0000 7fc970c23740 1 --2- 172.21.15.70:0/596126905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b0983a820 0x564b0983d1e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc958006010 tx=0x7fc95802e910 comp rx=0 tx=0).stop 2024-06-04T04:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.646+0000 7fc970c23740 1 -- 172.21.15.70:0/596126905 shutdown_connections 2024-06-04T04:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.646+0000 7fc970c23740 1 --2- 172.21.15.70:0/596126905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b09845a00 0x564b0985db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.646+0000 7fc970c23740 1 --2- 172.21.15.70:0/596126905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983b140 0x564b0985d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.646+0000 7fc970c23740 1 --2- 172.21.15.70:0/596126905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b0983a820 0x564b0983d1e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.646+0000 7fc970c23740 1 -- 172.21.15.70:0/596126905 >> 172.21.15.70:0/596126905 conn(0x564b09836d30 msgr2=0x564b09843210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.646+0000 7fc970c23740 1 -- 172.21.15.70:0/596126905 shutdown_connections 2024-06-04T04:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.646+0000 7fc970c23740 1 -- 172.21.15.70:0/596126905 wait complete. 2024-06-04T04:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.646+0000 7fc970c23740 1 -- start start 2024-06-04T04:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc970c23740 1 Processor -- start 2024-06-04T04:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc970c23740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b0983a820 0x564b09849610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc970c23740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983b140 0x564b09849b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc970c23740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b0984a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc970c23740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc9681afae0 con 0x564b09845a00 2024-06-04T04:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc970c23740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9681af410 con 0x564b0983a820 2024-06-04T04:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc970c23740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc9681ac650 con 0x564b0983b140 2024-06-04T04:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc970c23740 10 client.0 _ll_register_callbacks cb 0x7fc96000d860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc967fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b0984a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc967fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b0984a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52492/0 (socket says 172.21.15.70:52492) 2024-06-04T04:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc967fff640 1 -- 172.21.15.70:0/2789116933 learned_addr learned my addr 172.21.15.70:0/2789116933 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc9677fe640 1 --2- 172.21.15.70:0/2789116933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b0983a820 0x564b09849610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc966ffd640 1 --2- 172.21.15.70:0/2789116933 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983b140 0x564b09849b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc967fff640 1 -- 172.21.15.70:0/2789116933 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983b140 msgr2=0x564b09849b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc967fff640 1 --2- 172.21.15.70:0/2789116933 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983b140 0x564b09849b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc967fff640 1 -- 172.21.15.70:0/2789116933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b0983a820 msgr2=0x564b09849610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc967fff640 1 --2- 172.21.15.70:0/2789116933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b0983a820 0x564b09849610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc967fff640 1 -- 172.21.15.70:0/2789116933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b098bf510 con 0x564b09845a00 2024-06-04T04:57:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc967fff640 1 --2- 172.21.15.70:0/2789116933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b0984a090 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc960012fd0 tx=0x7fc960030850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0a8901cb29b812 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc946ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:57:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.647+0000 7fc946ffd640 1 -- 172.21.15.70:0/2789116933 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc960030e00 con 0x564b09845a00 2024-06-04T04:57:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.648+0000 7fc946ffd640 1 -- 172.21.15.70:0/2789116933 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc960004030 con 0x564b09845a00 2024-06-04T04:57:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.648+0000 7fc970c23740 1 -- 172.21.15.70:0/2789116933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc958003e00 con 0x564b09845a00 2024-06-04T04:57:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.648+0000 7fc946ffd640 1 -- 172.21.15.70:0/2789116933 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc960004230 con 0x564b09845a00 2024-06-04T04:57:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.648+0000 7fc970c23740 10 client.15459 fetch_fsmap learned FSMap version 29 2024-06-04T04:57:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.648+0000 7fc970c23740 1 -- 172.21.15.70:0/2789116933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564b098c0800 con 0x564b09845a00 2024-06-04T04:57:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.648+0000 7fc946ffd640 1 -- 172.21.15.70:0/2789116933 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc958003e00 con 0x564b09845a00 2024-06-04T04:57:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.648+0000 7fc946ffd640 1 -- 172.21.15.70:0/2789116933 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc960003820 con 0x564b09845a00 2024-06-04T04:57:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.648+0000 7fc970c23740 10 client.15459 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:57:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.648+0000 7fc970c23740 10 client.15459 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:57:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.648+0000 7fc970c23740 20 client.15459 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.648+0000 7fc970c23740 1 --2- 172.21.15.70:0/2789116933 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564b098c0fa0 0x564b098c33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.648+0000 7fc970c23740 4 client.15459 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T04:57:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.648+0000 7fc970c23740 1 -- 172.21.15.70:0/2789116933 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564b097a8360 con 0x564b098c0fa0 2024-06-04T04:57:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.648+0000 7fc9677fe640 1 --2- 172.21.15.70:0/2789116933 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564b098c0fa0 0x564b098c33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.649+0000 7fc946ffd640 10 client.15459 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.649+0000 7fc9677fe640 1 --2- 172.21.15.70:0/2789116933 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564b098c0fa0 0x564b098c33c0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=419d313fe7a504c1 server_cookie=6d1f941ab3ec1756 in_seq=0 out_seq=0 2024-06-04T04:57:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.660+0000 7fc946ffd640 1 -- 172.21.15.70:0/2789116933 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x564b097a8360 con 0x564b098c0fa0 2024-06-04T04:57:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.660+0000 7fc946ffd640 10 client.15459 handle_command_reply: tid=0 2024-06-04T04:57:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.660+0000 7fc970c23740 1 client.15459 shutdown 2024-06-04T04:57:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc970c23740 1 -- 172.21.15.70:0/2789116933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 msgr2=0x564b0984a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc970c23740 1 --2- 172.21.15.70:0/2789116933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b0984a090 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc960012fd0 tx=0x7fc960030850 comp rx=0 tx=0).stop 2024-06-04T04:57:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc970c23740 1 -- 172.21.15.70:0/2789116933 shutdown_connections 2024-06-04T04:57:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc970c23740 1 --2- 172.21.15.70:0/2789116933 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564b098c0fa0 0x564b098c33c0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc970c23740 1 --2- 172.21.15.70:0/2789116933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b09845a00 0x564b0984a090 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc970c23740 1 --2- 172.21.15.70:0/2789116933 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564b0983b140 0x564b09849b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc946ffd640 0 client.15459 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc970c23740 1 --2- 172.21.15.70:0/2789116933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b0983a820 0x564b09849610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc970c23740 1 -- 172.21.15.70:0/2789116933 >> 172.21.15.70:0/2789116933 conn(0x564b09836d30 msgr2=0x564b09842960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc970c23740 1 -- 172.21.15.70:0/2789116933 shutdown_connections 2024-06-04T04:57:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc970c23740 1 -- 172.21.15.70:0/2789116933 wait complete. 2024-06-04T04:57:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc970c23740 20 client.15459 trim_cache size 0 max 16384 2024-06-04T04:57:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T04:57:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T04:57:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T04:57:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc96de2a640 1 -- 172.21.15.70:0/1694917006 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9540b2b10 msgr2=0x7fc9540b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc96de2a640 1 --2- 172.21.15.70:0/1694917006 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9540b2b10 0x7fc9540b4fd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc960005fd0 tx=0x7fc96000d5b0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc96de2a640 1 -- 172.21.15.70:0/1694917006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc968104600 msgr2=0x7fc968199680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc96de2a640 1 --2- 172.21.15.70:0/1694917006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc968104600 0x7fc968199680 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc95000c890 tx=0x7fc95000cd50 comp rx=0 tx=0).stop 2024-06-04T04:57:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc96de2a640 1 -- 172.21.15.70:0/1694917006 shutdown_connections 2024-06-04T04:57:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc96de2a640 1 --2- 172.21.15.70:0/1694917006 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9540b2b10 0x7fc9540b4fd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc96de2a640 1 --2- 172.21.15.70:0/1694917006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc96810d020 0x7fc96819da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc96de2a640 1 --2- 172.21.15.70:0/1694917006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc968104600 0x7fc968199680 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc96de2a640 1 --2- 172.21.15.70:0/1694917006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc968103ce0 0x7fc968199140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc96de2a640 1 -- 172.21.15.70:0/1694917006 >> 172.21.15.70:0/1694917006 conn(0x7fc968078ad0 msgr2=0x7fc9680feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.661+0000 7fc96de2a640 1 -- 172.21.15.70:0/1694917006 shutdown_connections 2024-06-04T04:57:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:36.662+0000 7fc96de2a640 1 -- 172.21.15.70:0/1694917006 wait complete. 2024-06-04T04:57:36.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:36 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3295247815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:57:36.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:36 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3295247815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:57:36.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:36 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3295247815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:57:37.013 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:57:37.253 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:37.698 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:37 smithi070 ceph-mon[32631]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 43 op/s 2024-06-04T04:57:37.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:37 smithi112 ceph-mon[36453]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 43 op/s 2024-06-04T04:57:37.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:37 smithi098 ceph-mon[35569]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 43 op/s 2024-06-04T04:57:38.784 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:38 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T04:57:38.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:38 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T04:57:38.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:38 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T04:57:39.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.017+0000 7f0f71156700 1 -- 172.21.15.70:0/2259308448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f6c102470 msgr2=0x7f0f6c1028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:39.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.017+0000 7f0f71156700 1 --2- 172.21.15.70:0/2259308448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f6c102470 0x7f0f6c1028f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0f60009990 tx=0x7f0f60009ca0 comp rx=0 tx=0).stop 2024-06-04T04:57:39.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.017+0000 7f0f71156700 1 -- 172.21.15.70:0/2259308448 shutdown_connections 2024-06-04T04:57:39.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.017+0000 7f0f71156700 1 --2- 172.21.15.70:0/2259308448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c102e30 0x7f0f6c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:39.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.017+0000 7f0f71156700 1 --2- 172.21.15.70:0/2259308448 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f6c1084b0 0x7f0f6c108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:39.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.017+0000 7f0f71156700 1 --2- 172.21.15.70:0/2259308448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f6c102470 0x7f0f6c1028f0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:39.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.017+0000 7f0f71156700 1 -- 172.21.15.70:0/2259308448 >> 172.21.15.70:0/2259308448 conn(0x7f0f6c0793f0 msgr2=0x7f0f6c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:39.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.017+0000 7f0f71156700 1 -- 172.21.15.70:0/2259308448 shutdown_connections 2024-06-04T04:57:39.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.017+0000 7f0f71156700 1 -- 172.21.15.70:0/2259308448 wait complete. 2024-06-04T04:57:39.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.018+0000 7f0f71156700 1 Processor -- start 2024-06-04T04:57:39.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.018+0000 7f0f71156700 1 -- start start 2024-06-04T04:57:39.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.018+0000 7f0f71156700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f6c102470 0x7f0f6c19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:39.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.018+0000 7f0f71156700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f6c102e30 0x7f0f6c19af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:39.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.018+0000 7f0f71156700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c1084b0 0x7f0f6c1a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:39.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.018+0000 7f0f71156700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6c1124f0 con 0x7f0f6c102470 2024-06-04T04:57:39.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.018+0000 7f0f71156700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6c112370 con 0x7f0f6c1084b0 2024-06-04T04:57:39.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.018+0000 7f0f71156700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6c112670 con 0x7f0f6c102e30 2024-06-04T04:57:39.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.019+0000 7f0f6ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f6c102470 0x7f0f6c19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:39.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.019+0000 7f0f6ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f6c102470 0x7f0f6c19a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52514/0 (socket says 172.21.15.70:52514) 2024-06-04T04:57:39.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.019+0000 7f0f6ad9d700 1 -- 172.21.15.70:0/1121382705 learned_addr learned my addr 172.21.15.70:0/1121382705 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:39.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.019+0000 7f0f6a59c700 1 --2- 172.21.15.70:0/1121382705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f6c102e30 0x7f0f6c19af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:39.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.019+0000 7f0f6b59e700 1 --2- 172.21.15.70:0/1121382705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c1084b0 0x7f0f6c1a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:39.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.019+0000 7f0f6ad9d700 1 -- 172.21.15.70:0/1121382705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f6c102e30 msgr2=0x7f0f6c19af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:39.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.019+0000 7f0f6ad9d700 1 --2- 172.21.15.70:0/1121382705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f6c102e30 0x7f0f6c19af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:39.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.019+0000 7f0f6ad9d700 1 -- 172.21.15.70:0/1121382705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c1084b0 msgr2=0x7f0f6c1a5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:39.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.019+0000 7f0f6ad9d700 1 --2- 172.21.15.70:0/1121382705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c1084b0 0x7f0f6c1a5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:39.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.019+0000 7f0f6ad9d700 1 -- 172.21.15.70:0/1121382705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f6c1a5fb0 con 0x7f0f6c102470 2024-06-04T04:57:39.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.019+0000 7f0f6b59e700 1 --2- 172.21.15.70:0/1121382705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c1084b0 0x7f0f6c1a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:39.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.019+0000 7f0f6ad9d700 1 --2- 172.21.15.70:0/1121382705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f6c102470 0x7f0f6c19a9f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00b530 tx=0x7f0f5c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:39.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.019+0000 7f0f53fff700 1 -- 172.21.15.70:0/1121382705 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c015020 con 0x7f0f6c102470 2024-06-04T04:57:39.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.019+0000 7f0f53fff700 1 -- 172.21.15.70:0/1121382705 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0f5c011cc0 con 0x7f0f6c102470 2024-06-04T04:57:39.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.019+0000 7f0f71156700 1 -- 172.21.15.70:0/1121382705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f6c1a61e0 con 0x7f0f6c102470 2024-06-04T04:57:39.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.019+0000 7f0f53fff700 1 -- 172.21.15.70:0/1121382705 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c010dc0 con 0x7f0f6c102470 2024-06-04T04:57:39.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.019+0000 7f0f71156700 1 -- 172.21.15.70:0/1121382705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f6c1a6740 con 0x7f0f6c102470 2024-06-04T04:57:39.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.020+0000 7f0f71156700 1 -- 172.21.15.70:0/1121382705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f6c06a230 con 0x7f0f6c102470 2024-06-04T04:57:39.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.022+0000 7f0f53fff700 1 -- 172.21.15.70:0/1121382705 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7f0f5c018020 con 0x7f0f6c102470 2024-06-04T04:57:39.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.023+0000 7f0f53fff700 1 --2- 172.21.15.70:0/1121382705 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0f540b2b00 0x7f0f540b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:39.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.023+0000 7f0f6a59c700 1 --2- 172.21.15.70:0/1121382705 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0f540b2b00 0x7f0f540b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:39.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.023+0000 7f0f53fff700 1 -- 172.21.15.70:0/1121382705 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f0f5c0e2440 con 0x7f0f6c102470 2024-06-04T04:57:39.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.023+0000 7f0f6a59c700 1 --2- 172.21.15.70:0/1121382705 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0f540b2b00 0x7f0f540b4fc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000b5c0 tx=0x7f0f60005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:39.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.026+0000 7f0f53fff700 1 -- 172.21.15.70:0/1121382705 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0f5c01f020 con 0x7f0f6c102470 2024-06-04T04:57:39.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.275+0000 7f0f71156700 1 -- 172.21.15.70:0/1121382705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f6c02e560 con 0x7f0f6c102470 2024-06-04T04:57:39.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.276+0000 7f0f53fff700 1 -- 172.21.15.70:0/1121382705 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f0f5c0a9dc0 con 0x7f0f6c102470 2024-06-04T04:57:39.278 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:57:39.278 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:57:09.013178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T04:57:39.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.278+0000 7f0f71156700 1 -- 172.21.15.70:0/1121382705 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0f540b2b00 msgr2=0x7f0f540b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:39.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.278+0000 7f0f71156700 1 --2- 172.21.15.70:0/1121382705 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0f540b2b00 0x7f0f540b4fc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000b5c0 tx=0x7f0f60005960 comp rx=0 tx=0).stop 2024-06-04T04:57:39.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.278+0000 7f0f71156700 1 -- 172.21.15.70:0/1121382705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f6c102470 msgr2=0x7f0f6c19a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:39.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.278+0000 7f0f71156700 1 --2- 172.21.15.70:0/1121382705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f6c102470 0x7f0f6c19a9f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00b530 tx=0x7f0f5c00b840 comp rx=0 tx=0).stop 2024-06-04T04:57:39.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.278+0000 7f0f71156700 1 -- 172.21.15.70:0/1121382705 shutdown_connections 2024-06-04T04:57:39.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.278+0000 7f0f71156700 1 --2- 172.21.15.70:0/1121382705 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0f540b2b00 0x7f0f540b4fc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:39.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.278+0000 7f0f71156700 1 --2- 172.21.15.70:0/1121382705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c1084b0 0x7f0f6c1a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:39.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.278+0000 7f0f71156700 1 --2- 172.21.15.70:0/1121382705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f6c102470 0x7f0f6c19a9f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:39.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.278+0000 7f0f71156700 1 --2- 172.21.15.70:0/1121382705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0f6c102e30 0x7f0f6c19af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:39.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.278+0000 7f0f71156700 1 -- 172.21.15.70:0/1121382705 >> 172.21.15.70:0/1121382705 conn(0x7f0f6c0793f0 msgr2=0x7f0f6c100560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:39.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.278+0000 7f0f71156700 1 -- 172.21.15.70:0/1121382705 shutdown_connections 2024-06-04T04:57:39.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:39.278+0000 7f0f71156700 1 -- 172.21.15.70:0/1121382705 wait complete. 2024-06-04T04:57:39.281 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-06-04T04:57:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:39 smithi070 ceph-mon[32631]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 39 KiB/s wr, 13 op/s 2024-06-04T04:57:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:39 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1121382705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:57:39.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:39 smithi112 ceph-mon[36453]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 39 KiB/s wr, 13 op/s 2024-06-04T04:57:39.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:39 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1121382705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:57:39.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:39 smithi098 ceph-mon[35569]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 39 KiB/s wr, 13 op/s 2024-06-04T04:57:39.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:39 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1121382705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:57:40.067 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-04T04:57:40.315 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:40.717 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:40 smithi070 ceph-mon[32631]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-04T04:57:40.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:40 smithi112 ceph-mon[36453]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-04T04:57:40.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:40 smithi098 ceph-mon[35569]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-04T04:57:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.151+0000 7fc2f736a700 1 -- 172.21.15.70:0/3581408347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f006c230 msgr2=0x7fc2f006c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.151+0000 7fc2f736a700 1 --2- 172.21.15.70:0/3581408347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f006c230 0x7fc2f006c6b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009990 tx=0x7fc2e0009ca0 comp rx=0 tx=0).stop 2024-06-04T04:57:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.151+0000 7fc2f736a700 1 -- 172.21.15.70:0/3581408347 shutdown_connections 2024-06-04T04:57:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.151+0000 7fc2f736a700 1 --2- 172.21.15.70:0/3581408347 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2f0105fb0 0x7fc2f0106410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.151+0000 7fc2f736a700 1 --2- 172.21.15.70:0/3581408347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f01084a0 0x7fc2f006bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.151+0000 7fc2f736a700 1 --2- 172.21.15.70:0/3581408347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f006c230 0x7fc2f006c6b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.151+0000 7fc2f736a700 1 -- 172.21.15.70:0/3581408347 >> 172.21.15.70:0/3581408347 conn(0x7fc2f00794e0 msgr2=0x7fc2f00798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.151+0000 7fc2f736a700 1 -- 172.21.15.70:0/3581408347 shutdown_connections 2024-06-04T04:57:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.151+0000 7fc2f736a700 1 -- 172.21.15.70:0/3581408347 wait complete. 2024-06-04T04:57:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.151+0000 7fc2f736a700 1 Processor -- start 2024-06-04T04:57:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.151+0000 7fc2f736a700 1 -- start start 2024-06-04T04:57:42.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f736a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2f006c230 0x7fc2f01a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:42.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f736a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f0105fb0 0x7fc2f01a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:42.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f736a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f01084a0 0x7fc2f01a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:42.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f736a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f0119bf0 con 0x7fc2f01084a0 2024-06-04T04:57:42.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f736a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f0119a70 con 0x7fc2f0105fb0 2024-06-04T04:57:42.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f736a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f0119d70 con 0x7fc2f006c230 2024-06-04T04:57:42.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f5907700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f01084a0 0x7fc2f01a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:42.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f5907700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f01084a0 0x7fc2f01a4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52520/0 (socket says 172.21.15.70:52520) 2024-06-04T04:57:42.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f5907700 1 -- 172.21.15.70:0/3988373661 learned_addr learned my addr 172.21.15.70:0/3988373661 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:42.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f4905700 1 --2- 172.21.15.70:0/3988373661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f0105fb0 0x7fc2f01a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:42.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f5106700 1 --2- 172.21.15.70:0/3988373661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2f006c230 0x7fc2f01a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:42.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f5907700 1 -- 172.21.15.70:0/3988373661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2f006c230 msgr2=0x7fc2f01a0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:42.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f5907700 1 --2- 172.21.15.70:0/3988373661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2f006c230 0x7fc2f01a0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:42.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f5907700 1 -- 172.21.15.70:0/3988373661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f0105fb0 msgr2=0x7fc2f01a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:42.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f5907700 1 --2- 172.21.15.70:0/3988373661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f0105fb0 0x7fc2f01a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:42.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f5907700 1 -- 172.21.15.70:0/3988373661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f01a5150 con 0x7fc2f01084a0 2024-06-04T04:57:42.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f5907700 1 --2- 172.21.15.70:0/3988373661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f01084a0 0x7fc2f01a4a10 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800ada0 tx=0x7fc2e800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:42.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2e67fc700 1 -- 172.21.15.70:0/3988373661 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8017020 con 0x7fc2f01084a0 2024-06-04T04:57:42.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2e67fc700 1 -- 172.21.15.70:0/3988373661 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc2e800cd60 con 0x7fc2f01084a0 2024-06-04T04:57:42.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f736a700 1 -- 172.21.15.70:0/3988373661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f01a5440 con 0x7fc2f01084a0 2024-06-04T04:57:42.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2e67fc700 1 -- 172.21.15.70:0/3988373661 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8005280 con 0x7fc2f01084a0 2024-06-04T04:57:42.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.152+0000 7fc2f736a700 1 -- 172.21.15.70:0/3988373661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f01accc0 con 0x7fc2f01084a0 2024-06-04T04:57:42.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.153+0000 7fc2f736a700 1 -- 172.21.15.70:0/3988373661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f006a230 con 0x7fc2f01084a0 2024-06-04T04:57:42.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.156+0000 7fc2e67fc700 1 -- 172.21.15.70:0/3988373661 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7fc2e800c8d0 con 0x7fc2f01084a0 2024-06-04T04:57:42.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.156+0000 7fc2e67fc700 1 --2- 172.21.15.70:0/3988373661 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc2dc0b2b00 0x7fc2dc0b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:42.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.156+0000 7fc2f5106700 1 --2- 172.21.15.70:0/3988373661 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc2dc0b2b00 0x7fc2dc0b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:42.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.156+0000 7fc2e67fc700 1 -- 172.21.15.70:0/3988373661 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc2e80e24a0 con 0x7fc2f01084a0 2024-06-04T04:57:42.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.157+0000 7fc2f5106700 1 --2- 172.21.15.70:0/3988373661 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc2dc0b2b00 0x7fc2dc0b4fc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec007b60 tx=0x7fc2ec0073d0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:42.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.160+0000 7fc2e67fc700 1 -- 172.21.15.70:0/3988373661 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc2e8021020 con 0x7fc2f01084a0 2024-06-04T04:57:42.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.363+0000 7fc2f736a700 1 -- 172.21.15.70:0/3988373661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2f01acef0 con 0x7fc2f01084a0 2024-06-04T04:57:42.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.364+0000 7fc2e67fc700 1 -- 172.21.15.70:0/3988373661 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+23085 (secure 0 0 0) 0x7fc2e80a9d40 con 0x7fc2f01084a0 2024-06-04T04:57:42.367 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:57:42.367 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":82,"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","created":"2024-06-04T04:45:22.297536+0000","modified":"2024-06-04T04:57:09.937740+0000","last_up_change":"2024-06-04T04:52:34.388708+0000","last_in_change":"2024-06-04T04:52:14.197780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-04T04:48:43.724356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-04T04:53:31.398455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-04T04:53:35.298617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ff491ea2-1906-48e4-883e-bc78e778260e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6803","nonce":824119100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6805","nonce":824119100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6809","nonce":824119100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6807","nonce":824119100}]},"public_addr":"172.21.15.70:6803/824119100","cluster_addr":"172.21.15.70:6805/824119100","heartbeat_back_addr":"172.21.15.70:6809/824119100","heartbeat_front_addr":"172.21.15.70:6807/824119100","state":["exists","up"]},{"osd":1,"uuid":"bd5e4806-535b-49b5-ba6b-93228da35a84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6801","nonce":2744598742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6803","nonce":2744598742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6807","nonce":2744598742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6805","nonce":2744598742}]},"public_addr":"172.21.15.98:6801/2744598742","cluster_addr":"172.21.15.98:6803/2744598742","heartbeat_back_addr":"172.21.15.98:6807/2744598742","heartbeat_front_addr":"172.21.15.98:6805/2744598742","state":["exists","up"]},{"osd":2,"uuid":"8456fb55-910a-4a59-b84b-2c0f68346ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6801","nonce":1328098370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6803","nonce":1328098370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6807","nonce":1328098370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6805","nonce":1328098370}]},"public_addr":"172.21.15.112:6801/1328098370","cluster_addr":"172.21.15.112:6803/1328098370","heartbeat_back_addr":"172.21.15.112:6807/1328098370","heartbeat_front_addr":"172.21.15.112:6805/1328098370","state":["exists","up"]},{"osd":3,"uuid":"c8fbc440-bf70-42d3-b01e-777ccfa7900b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6811","nonce":4041566346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6813","nonce":4041566346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6817","nonce":4041566346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6815","nonce":4041566346}]},"public_addr":"172.21.15.70:6811/4041566346","cluster_addr":"172.21.15.70:6813/4041566346","heartbeat_back_addr":"172.21.15.70:6817/4041566346","heartbeat_front_addr":"172.21.15.70:6815/4041566346","state":["exists","up"]},{"osd":4,"uuid":"549c8c70-8be8-46ae-bcb6-0fde32b40cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6809","nonce":3652775814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6811","nonce":3652775814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6815","nonce":3652775814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6813","nonce":3652775814}]},"public_addr":"172.21.15.98:6809/3652775814","cluster_addr":"172.21.15.98:6811/3652775814","heartbeat_back_addr":"172.21.15.98:6815/3652775814","heartbeat_front_addr":"172.21.15.98:6813/3652775814","state":["exists","up"]},{"osd":5,"uuid":"51d86af1-0d6c-4ae0-b090-00414fc1bf7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6809","nonce":1857257899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6811","nonce":1857257899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6815","nonce":1857257899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6813","nonce":1857257899}]},"public_addr":"172.21.15.112:6809/1857257899","cluster_addr":"172.21.15.112:6811/1857257899","heartbeat_back_addr":"172.21.15.112:6815/1857257899","heartbeat_front_addr":"172.21.15.112:6813/1857257899","state":["exists","up"]},{"osd":6,"uuid":"526a8138-bfa8-419e-a5b5-e6b80f0ae004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6819","nonce":1264676496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6821","nonce":1264676496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6825","nonce":1264676496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6823","nonce":1264676496}]},"public_addr":"172.21.15.70:6819/1264676496","cluster_addr":"172.21.15.70:6821/1264676496","heartbeat_back_addr":"172.21.15.70:6825/1264676496","heartbeat_front_addr":"172.21.15.70:6823/1264676496","state":["exists","up"]},{"osd":7,"uuid":"1173f785-6ed1-47ef-9265-2e233ea9ee02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6817","nonce":709593020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6819","nonce":709593020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6823","nonce":709593020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6821","nonce":709593020}]},"public_addr":"172.21.15.98:6817/709593020","cluster_addr":"172.21.15.98:6819/709593020","heartbeat_back_addr":"172.21.15.98:6823/709593020","heartbeat_front_addr":"172.21.15.98:6821/709593020","state":["exists","up"]},{"osd":8,"uuid":"6424cc6e-24a7-4b81-9ce6-39203fecc7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6817","nonce":2250225413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6819","nonce":2250225413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6823","nonce":2250225413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6821","nonce":2250225413}]},"public_addr":"172.21.15.112:6817/2250225413","cluster_addr":"172.21.15.112:6819/2250225413","heartbeat_back_addr":"172.21.15.112:6823/2250225413","heartbeat_front_addr":"172.21.15.112:6821/2250225413","state":["exists","up"]},{"osd":9,"uuid":"8d666db8-8831-4d63-9e3c-a4bf2504149e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6827","nonce":3557688707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6829","nonce":3557688707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6833","nonce":3557688707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6831","nonce":3557688707}]},"public_addr":"172.21.15.70:6827/3557688707","cluster_addr":"172.21.15.70:6829/3557688707","heartbeat_back_addr":"172.21.15.70:6833/3557688707","heartbeat_front_addr":"172.21.15.70:6831/3557688707","state":["exists","up"]},{"osd":10,"uuid":"ecb925d2-0263-4dfc-a2fd-9e27856022d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6825","nonce":1896493718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6827","nonce":1896493718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6831","nonce":1896493718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6829","nonce":1896493718}]},"public_addr":"172.21.15.98:6825/1896493718","cluster_addr":"172.21.15.98:6827/1896493718","heartbeat_back_addr":"172.21.15.98:6831/1896493718","heartbeat_front_addr":"172.21.15.98:6829/1896493718","state":["exists","up"]},{"osd":11,"uuid":"c8e3230c-bc3a-40bd-8dd0-5fef56e4df97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6825","nonce":1400793137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6827","nonce":1400793137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6831","nonce":1400793137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6829","nonce":1400793137}]},"public_addr":"172.21.15.112:6825/1400793137","cluster_addr":"172.21.15.112:6827/1400793137","heartbeat_back_addr":"172.21.15.112:6831/1400793137","heartbeat_front_addr":"172.21.15.112:6829/1400793137","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:47:53.626608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:17.784505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:41.971969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:07.130699+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:31.615496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:56.898607+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:22.739991+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:48.319727+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:13.902063+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:40.389470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:06.142092+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:32.949383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/4291129442":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/3584175673":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/3720789940":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/145533746":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/3442074771":"2024-06-05T04:46:17.636907+0000","172.21.15.70:6801/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/1442095493":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/2453733268":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6801/2752966840":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/1369781447":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6801/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/2752966840":"2024-06-05T04:57:09.937704+0000","172.21.15.70:6800/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/915186456":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/860033006":"2024-06-05T04:45:49.501864+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-04T04:57:42.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.367+0000 7fc2f736a700 1 -- 172.21.15.70:0/3988373661 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc2dc0b2b00 msgr2=0x7fc2dc0b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:42.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.367+0000 7fc2f736a700 1 --2- 172.21.15.70:0/3988373661 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc2dc0b2b00 0x7fc2dc0b4fc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec007b60 tx=0x7fc2ec0073d0 comp rx=0 tx=0).stop 2024-06-04T04:57:42.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.367+0000 7fc2f736a700 1 -- 172.21.15.70:0/3988373661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f01084a0 msgr2=0x7fc2f01a4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:42.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.367+0000 7fc2f736a700 1 --2- 172.21.15.70:0/3988373661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f01084a0 0x7fc2f01a4a10 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800ada0 tx=0x7fc2e800c330 comp rx=0 tx=0).stop 2024-06-04T04:57:42.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.367+0000 7fc2f736a700 1 -- 172.21.15.70:0/3988373661 shutdown_connections 2024-06-04T04:57:42.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.367+0000 7fc2f736a700 1 --2- 172.21.15.70:0/3988373661 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc2dc0b2b00 0x7fc2dc0b4fc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:42.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.367+0000 7fc2f736a700 1 --2- 172.21.15.70:0/3988373661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f01084a0 0x7fc2f01a4a10 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:42.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.367+0000 7fc2f736a700 1 --2- 172.21.15.70:0/3988373661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2f006c230 0x7fc2f01a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:42.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.367+0000 7fc2f736a700 1 --2- 172.21.15.70:0/3988373661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f0105fb0 0x7fc2f01a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:42.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.367+0000 7fc2f736a700 1 -- 172.21.15.70:0/3988373661 >> 172.21.15.70:0/3988373661 conn(0x7fc2f00794e0 msgr2=0x7fc2f0104f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:42.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.367+0000 7fc2f736a700 1 -- 172.21.15.70:0/3988373661 shutdown_connections 2024-06-04T04:57:42.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:42.367+0000 7fc2f736a700 1 -- 172.21.15.70:0/3988373661 wait complete. 2024-06-04T04:57:43.047 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-06-04T04:57:43.288 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:43.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:43 smithi070 ceph-mon[32631]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-06-04T04:57:43.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:43 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3988373661' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:57:43.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:43 smithi112 ceph-mon[36453]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-06-04T04:57:43.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:43 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3988373661' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:57:43.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:43 smithi098 ceph-mon[35569]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-06-04T04:57:43.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:43 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3988373661' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T04:57:45.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:45 smithi070 ceph-mon[32631]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-06-04T04:57:45.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:45 smithi112 ceph-mon[36453]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-06-04T04:57:45.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:45 smithi098 ceph-mon[35569]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-06-04T04:57:45.770 INFO:teuthology.orchestra.run.smithi070.stdout:{"run_dir":"/var/run/ceph"} 2024-06-04T04:57:45.770 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-06-04T04:57:45.770 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 6827 2024-06-04T04:57:45.770 DEBUG:tasks.quiescer:Yielding 2024-06-04T04:57:45.771 INFO:teuthology.run_tasks:Running task workunit... 2024-06-04T04:57:45.784 INFO:tasks.workunit:Pulling workunits from ref 7aec00165b4b073a25396fd0279151d2b2fa23fe 2024-06-04T04:57:45.785 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-06-04T04:57:45.785 INFO:tasks.workunit:timeout=3h 2024-06-04T04:57:45.785 INFO:tasks.workunit:cleanup=True 2024-06-04T04:57:45.785 DEBUG:teuthology.orchestra.run.smithi070:> stat -- /home/ubuntu/cephtest/mnt.0 2024-06-04T04:57:45.785 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:57:45.802 INFO:teuthology.orchestra.run.smithi070.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-06-04T04:57:45.802 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-06-04T04:57:45.802 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 77h/119d Inode: 1 Links: 2 2024-06-04T04:57:45.803 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-04T04:57:45.803 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:fusefs_t:s0 2024-06-04T04:57:45.803 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-06-04 04:54:02.391417809 +0000 2024-06-04T04:57:45.803 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-06-04 04:54:02.391417809 +0000 2024-06-04T04:57:45.803 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-06-04 04:57:08.616850484 +0000 2024-06-04T04:57:45.803 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: - 2024-06-04T04:57:45.803 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-06-04T04:57:45.803 DEBUG:teuthology.orchestra.run.smithi070:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-06-04T04:57:45.874 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7aec00165b4b073a25396fd0279151d2b2fa23fe 2024-06-04T04:57:45.929 INFO:tasks.workunit.client.0.smithi070.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-06-04T04:57:46.032 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:47.251 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:47 smithi070 ceph-mon[32631]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.1 KiB/s wr, 16 op/s 2024-06-04T04:57:47.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:47 smithi112 ceph-mon[36453]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.1 KiB/s wr, 16 op/s 2024-06-04T04:57:47.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:47 smithi098 ceph-mon[35569]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.1 KiB/s wr, 16 op/s 2024-06-04T04:57:47.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.789+0000 7f48bce6f700 1 -- 172.21.15.70:0/1824094077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48b8102e00 msgr2=0x7f48b810f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.789+0000 7f48bce6f700 1 --2- 172.21.15.70:0/1824094077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48b8102e00 0x7f48b810f2f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f48a8009990 tx=0x7f48a8009ca0 comp rx=0 tx=0).stop 2024-06-04T04:57:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.789+0000 7f48bce6f700 1 -- 172.21.15.70:0/1824094077 shutdown_connections 2024-06-04T04:57:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.789+0000 7f48bce6f700 1 --2- 172.21.15.70:0/1824094077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48b8102e00 0x7f48b810f2f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.789+0000 7f48bce6f700 1 --2- 172.21.15.70:0/1824094077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48b8108520 0x7f48b8108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.789+0000 7f48bce6f700 1 --2- 172.21.15.70:0/1824094077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b8102440 0x7f48b81028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.789+0000 7f48bce6f700 1 -- 172.21.15.70:0/1824094077 >> 172.21.15.70:0/1824094077 conn(0x7f48b80793f0 msgr2=0x7f48b8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.789+0000 7f48bce6f700 1 -- 172.21.15.70:0/1824094077 shutdown_connections 2024-06-04T04:57:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.789+0000 7f48bce6f700 1 -- 172.21.15.70:0/1824094077 wait complete. 2024-06-04T04:57:47.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48bce6f700 1 Processor -- start 2024-06-04T04:57:47.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48bce6f700 1 -- start start 2024-06-04T04:57:47.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48bce6f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48b8102440 0x7f48b8104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:47.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48bce6f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b8102e00 0x7f48b81045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:47.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48bce6f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48b8108520 0x7f48b8104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:47.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48bce6f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f48b8112580 con 0x7f48b8102440 2024-06-04T04:57:47.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48bce6f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48b8112400 con 0x7f48b8102e00 2024-06-04T04:57:47.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48bce6f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f48b8112700 con 0x7f48b8108520 2024-06-04T04:57:47.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48b659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48b8102440 0x7f48b8104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48b659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48b8102440 0x7f48b8104080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54152/0 (socket says 172.21.15.70:54152) 2024-06-04T04:57:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48b659c700 1 -- 172.21.15.70:0/3218693131 learned_addr learned my addr 172.21.15.70:0/3218693131 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48b6d9d700 1 --2- 172.21.15.70:0/3218693131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48b8108520 0x7f48b8104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48b5d9b700 1 --2- 172.21.15.70:0/3218693131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b8102e00 0x7f48b81045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48b659c700 1 -- 172.21.15.70:0/3218693131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48b8108520 msgr2=0x7f48b8104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48b659c700 1 --2- 172.21.15.70:0/3218693131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48b8108520 0x7f48b8104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48b659c700 1 -- 172.21.15.70:0/3218693131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b8102e00 msgr2=0x7f48b81045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48b659c700 1 --2- 172.21.15.70:0/3218693131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b8102e00 0x7f48b81045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.790+0000 7f48b659c700 1 -- 172.21.15.70:0/3218693131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48b81a8230 con 0x7f48b8102440 2024-06-04T04:57:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.791+0000 7f48b659c700 1 --2- 172.21.15.70:0/3218693131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48b8102440 0x7f48b8104080 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f48b000ba50 tx=0x7f48b000bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.791+0000 7f48af7fe700 1 -- 172.21.15.70:0/3218693131 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0019070 con 0x7f48b8102440 2024-06-04T04:57:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.791+0000 7f48af7fe700 1 -- 172.21.15.70:0/3218693131 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f48b000cdc0 con 0x7f48b8102440 2024-06-04T04:57:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.791+0000 7f48bce6f700 1 -- 172.21.15.70:0/3218693131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48b81a8520 con 0x7f48b8102440 2024-06-04T04:57:47.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.791+0000 7f48af7fe700 1 -- 172.21.15.70:0/3218693131 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0014e50 con 0x7f48b8102440 2024-06-04T04:57:47.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.791+0000 7f48bce6f700 1 -- 172.21.15.70:0/3218693131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48b81a8a10 con 0x7f48b8102440 2024-06-04T04:57:47.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.792+0000 7f48bce6f700 1 -- 172.21.15.70:0/3218693131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48b806a230 con 0x7f48b8102440 2024-06-04T04:57:47.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.793+0000 7f48af7fe700 1 -- 172.21.15.70:0/3218693131 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7f48b001c070 con 0x7f48b8102440 2024-06-04T04:57:47.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.794+0000 7f48af7fe700 1 --2- 172.21.15.70:0/3218693131 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48a40b2c20 0x7f48a40b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:47.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.794+0000 7f48b5d9b700 1 --2- 172.21.15.70:0/3218693131 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48a40b2c20 0x7f48a40b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:47.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.794+0000 7f48af7fe700 1 -- 172.21.15.70:0/3218693131 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f48b00e1eb0 con 0x7f48b8102440 2024-06-04T04:57:47.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.794+0000 7f48b5d9b700 1 --2- 172.21.15.70:0/3218693131 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48a40b2c20 0x7f48a40b50e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f48b8079cf0 tx=0x7f48a0009190 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:47.798+0000 7f48af7fe700 1 -- 172.21.15.70:0/3218693131 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f48b00a9830 con 0x7f48b8102440 2024-06-04T04:57:48.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:48.056+0000 7f48bce6f700 1 -- 172.21.15.70:0/3218693131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48b802e560 con 0x7f48b8102440 2024-06-04T04:57:48.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:48.056+0000 7f48af7fe700 1 -- 172.21.15.70:0/3218693131 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f48b00a9130 con 0x7f48b8102440 2024-06-04T04:57:48.059 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:57:48.059 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:57:09.013178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T04:57:48.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:48.059+0000 7f48bce6f700 1 -- 172.21.15.70:0/3218693131 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48a40b2c20 msgr2=0x7f48a40b50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:48.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:48.059+0000 7f48bce6f700 1 --2- 172.21.15.70:0/3218693131 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48a40b2c20 0x7f48a40b50e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f48b8079cf0 tx=0x7f48a0009190 comp rx=0 tx=0).stop 2024-06-04T04:57:48.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:48.059+0000 7f48bce6f700 1 -- 172.21.15.70:0/3218693131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48b8102440 msgr2=0x7f48b8104080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:48.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:48.059+0000 7f48bce6f700 1 --2- 172.21.15.70:0/3218693131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48b8102440 0x7f48b8104080 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f48b000ba50 tx=0x7f48b000bd60 comp rx=0 tx=0).stop 2024-06-04T04:57:48.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:48.059+0000 7f48bce6f700 1 -- 172.21.15.70:0/3218693131 shutdown_connections 2024-06-04T04:57:48.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:48.059+0000 7f48bce6f700 1 --2- 172.21.15.70:0/3218693131 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48a40b2c20 0x7f48a40b50e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:48.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:48.059+0000 7f48bce6f700 1 --2- 172.21.15.70:0/3218693131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48b8108520 0x7f48b8104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:48.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:48.059+0000 7f48bce6f700 1 --2- 172.21.15.70:0/3218693131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48b8102440 0x7f48b8104080 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:48.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:48.059+0000 7f48bce6f700 1 --2- 172.21.15.70:0/3218693131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b8102e00 0x7f48b81045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:48.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:48.059+0000 7f48bce6f700 1 -- 172.21.15.70:0/3218693131 >> 172.21.15.70:0/3218693131 conn(0x7f48b80793f0 msgr2=0x7f48b806b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:48.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:48.059+0000 7f48bce6f700 1 -- 172.21.15.70:0/3218693131 shutdown_connections 2024-06-04T04:57:48.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:48.059+0000 7f48bce6f700 1 -- 172.21.15.70:0/3218693131 wait complete. 2024-06-04T04:57:48.062 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-06-04T04:57:48.753 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-06-04T04:53:38.237993+0000', 'modified': '2024-06-04T04:57:09.013178+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24448, 'mds_1': 24442, 'mds_2': 24440, 'mds_3': 24436, 'mds_4': 24410}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14580': {'gid': 14580, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6839/3817259285', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6838', 'nonce': 3817259285}, {'type': 'v1', 'addr': '172.21.15.70:6839', 'nonce': 3817259285}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14592': {'gid': 14592, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6841/2748767095', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6840', 'nonce': 2748767095}, {'type': 'v1', 'addr': '172.21.15.70:6841', 'nonce': 2748767095}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24371': {'gid': 24371, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6833/1896647585', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6832', 'nonce': 1896647585}, {'type': 'v1', 'addr': '172.21.15.98:6833', 'nonce': 1896647585}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24386': {'gid': 24386, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6835/3148471566', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 3148471566}, {'type': 'v1', 'addr': '172.21.15.98:6835', 'nonce': 3148471566}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24400': {'gid': 24400, 'name': 'l', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6833/1359739019', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6832', 'nonce': 1359739019}, {'type': 'v1', 'addr': '172.21.15.112:6833', 'nonce': 1359739019}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24410': {'gid': 24410, 'name': 'h', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.98:6837/3160044154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6836', 'nonce': 3160044154}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3160044154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24436': {'gid': 24436, 'name': 'f', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6835/242926435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6834', 'nonce': 242926435}, {'type': 'v1', 'addr': '172.21.15.112:6835', 'nonce': 242926435}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'b', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.98:6839/1361276220', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6838', 'nonce': 1361276220}, {'type': 'v1', 'addr': '172.21.15.98:6839', 'nonce': 1361276220}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'c', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.112:6837/3596492666', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6836', 'nonce': 3596492666}, {'type': 'v1', 'addr': '172.21.15.112:6837', 'nonce': 3596492666}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24448': {'gid': 24448, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.112:6839/2996116087', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6838', 'nonce': 2996116087}, {'type': 'v1', 'addr': '172.21.15.112:6839', 'nonce': 2996116087}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24448, 'qdb_cluster': [24448, 24442, 24440, 24410, 24436]} 2024-06-04T04:57:48.753 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-06-04T04:57:48.754 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.h status 2024-06-04T04:57:49.038 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:49.280 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:49 smithi070 ceph-mon[32631]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-06-04T04:57:49.280 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:49 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3218693131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:57:49.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:49 smithi112 ceph-mon[36453]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-06-04T04:57:49.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:49 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3218693131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:57:49.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:49 smithi098 ceph-mon[35569]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-06-04T04:57:49.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:49 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3218693131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:57:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.751+0000 7fb14eb6a640 1 -- 172.21.15.70:0/2216902412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb148106cd0 msgr2=0x7fb14810e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.751+0000 7fb14eb6a640 1 --2- 172.21.15.70:0/2216902412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb148106cd0 0x7fb14810e8e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb140009ca0 tx=0x7fb14002a8c0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.751+0000 7fb14eb6a640 1 -- 172.21.15.70:0/2216902412 shutdown_connections 2024-06-04T04:57:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.751+0000 7fb14eb6a640 1 --2- 172.21.15.70:0/2216902412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb148102ce0 0x7fb1481065f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.751+0000 7fb14eb6a640 1 --2- 172.21.15.70:0/2216902412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb148106cd0 0x7fb14810e8e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.751+0000 7fb14eb6a640 1 --2- 172.21.15.70:0/2216902412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb148102330 0x7fb148102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.751+0000 7fb14eb6a640 1 -- 172.21.15.70:0/2216902412 >> 172.21.15.70:0/2216902412 conn(0x7fb148077e70 msgr2=0x7fb1480ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.751+0000 7fb14eb6a640 1 -- 172.21.15.70:0/2216902412 shutdown_connections 2024-06-04T04:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.751+0000 7fb14eb6a640 1 -- 172.21.15.70:0/2216902412 wait complete. 2024-06-04T04:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.751+0000 7fb14eb6a640 1 Processor -- start 2024-06-04T04:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.751+0000 7fb14eb6a640 1 -- start start 2024-06-04T04:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb14eb6a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb148102330 0x7fb1481980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb14eb6a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb148106cd0 0x7fb1481985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb14eb6a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb14819c920 0x7fb14819cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb14eb6a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb148111b70 con 0x7fb14819c920 2024-06-04T04:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb14eb6a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481119f0 con 0x7fb148102330 2024-06-04T04:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb14eb6a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb148111cf0 con 0x7fb148106cd0 2024-06-04T04:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb14d107640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb14819c920 0x7fb14819cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb14d107640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb14819c920 0x7fb14819cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54166/0 (socket says 172.21.15.70:54166) 2024-06-04T04:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb14d107640 1 -- 172.21.15.70:0/2565970795 learned_addr learned my addr 172.21.15.70:0/2565970795 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb14c906640 1 --2- 172.21.15.70:0/2565970795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb148102330 0x7fb1481980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb147fff640 1 --2- 172.21.15.70:0/2565970795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb148106cd0 0x7fb1481985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb14d107640 1 -- 172.21.15.70:0/2565970795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb148106cd0 msgr2=0x7fb1481985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb14d107640 1 --2- 172.21.15.70:0/2565970795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb148106cd0 0x7fb1481985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb14d107640 1 -- 172.21.15.70:0/2565970795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb148102330 msgr2=0x7fb1481980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb14d107640 1 --2- 172.21.15.70:0/2565970795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb148102330 0x7fb1481980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb14d107640 1 -- 172.21.15.70:0/2565970795 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb14819d410 con 0x7fb14819c920 2024-06-04T04:57:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb14d107640 1 --2- 172.21.15.70:0/2565970795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb14819c920 0x7fb14819cdd0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb140005cc0 tx=0x7fb140032a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb145ffb640 1 -- 172.21.15.70:0/2565970795 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb140032e00 con 0x7fb14819c920 2024-06-04T04:57:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb145ffb640 1 -- 172.21.15.70:0/2565970795 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb1400044d0 con 0x7fb14819c920 2024-06-04T04:57:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb145ffb640 1 -- 172.21.15.70:0/2565970795 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb140030b70 con 0x7fb14819c920 2024-06-04T04:57:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.752+0000 7fb14eb6a640 1 -- 172.21.15.70:0/2565970795 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb14819d6a0 con 0x7fb14819c920 2024-06-04T04:57:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.753+0000 7fb14eb6a640 1 -- 172.21.15.70:0/2565970795 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1481a4ef0 con 0x7fb14819c920 2024-06-04T04:57:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.755+0000 7fb145ffb640 1 -- 172.21.15.70:0/2565970795 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7fb140004680 con 0x7fb14819c920 2024-06-04T04:57:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.755+0000 7fb151522740 1 Processor -- start 2024-06-04T04:57:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.755+0000 7fb151522740 1 -- start start 2024-06-04T04:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb151522740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310226820 0x564310226c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb151522740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310227140 0x5643102314c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb151522740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310231a00 0x564310233df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb151522740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564310198c30 con 0x564310226820 2024-06-04T04:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb151522740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5643101887d0 con 0x564310231a00 2024-06-04T04:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb151522740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56431018c810 con 0x564310227140 2024-06-04T04:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb147fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310226820 0x564310226c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb147fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310226820 0x564310226c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54170/0 (socket says 172.21.15.70:54170) 2024-06-04T04:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb147fff640 1 -- 172.21.15.70:0/84242915 learned_addr learned my addr 172.21.15.70:0/84242915 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb14c906640 1 --2- 172.21.15.70:0/84242915 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310227140 0x5643102314c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb147fff640 1 --2- 172.21.15.70:0/84242915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310231a00 0x564310233df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb145ffb640 1 --2- 172.21.15.70:0/2565970795 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1380b2b10 0x7fb1380b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb147fff640 1 -- 172.21.15.70:0/84242915 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310227140 msgr2=0x5643102314c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb147fff640 1 --2- 172.21.15.70:0/84242915 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310227140 0x5643102314c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb147fff640 1 -- 172.21.15.70:0/84242915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310231a00 msgr2=0x564310233df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb147fff640 1 --2- 172.21.15.70:0/84242915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310231a00 0x564310233df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb147fff640 1 -- 172.21.15.70:0/84242915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56430fe9d210 con 0x564310226820 2024-06-04T04:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb14c906640 1 --2- 172.21.15.70:0/84242915 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310227140 0x5643102314c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb14d107640 1 --2- 172.21.15.70:0/2565970795 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1380b2b10 0x7fb1380b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb147fff640 1 --2- 172.21.15.70:0/84242915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310226820 0x564310226c00 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb13400ff10 tx=0x7fb13400f7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77feb59a06900fd8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb145ffb640 1 -- 172.21.15.70:0/2565970795 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb140103b40 con 0x7fb14819c920 2024-06-04T04:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb132ffd640 1 -- 172.21.15.70:0/84242915 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13400f230 con 0x564310226820 2024-06-04T04:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb132ffd640 1 -- 172.21.15.70:0/84242915 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb134013070 con 0x564310226820 2024-06-04T04:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.756+0000 7fb132ffd640 1 -- 172.21.15.70:0/84242915 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13400a670 con 0x564310226820 2024-06-04T04:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.757+0000 7fb151522740 1 -- 172.21.15.70:0/84242915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310226820 msgr2=0x564310226c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.757+0000 7fb14d107640 1 --2- 172.21.15.70:0/2565970795 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1380b2b10 0x7fb1380b4fd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb1401012e0 tx=0x7fb140097690 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.757+0000 7fb151522740 1 --2- 172.21.15.70:0/84242915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310226820 0x564310226c00 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb13400ff10 tx=0x7fb13400f7e0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.757+0000 7fb151522740 1 -- 172.21.15.70:0/84242915 shutdown_connections 2024-06-04T04:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.757+0000 7fb151522740 1 --2- 172.21.15.70:0/84242915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310231a00 0x564310233df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.757+0000 7fb151522740 1 --2- 172.21.15.70:0/84242915 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310227140 0x5643102314c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.757+0000 7fb151522740 1 --2- 172.21.15.70:0/84242915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310226820 0x564310226c00 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.757+0000 7fb151522740 1 -- 172.21.15.70:0/84242915 >> 172.21.15.70:0/84242915 conn(0x564310222d30 msgr2=0x564310223110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.757+0000 7fb151522740 1 -- 172.21.15.70:0/84242915 shutdown_connections 2024-06-04T04:57:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.757+0000 7fb151522740 1 -- 172.21.15.70:0/84242915 wait complete. 2024-06-04T04:57:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.757+0000 7fb151522740 1 -- start start 2024-06-04T04:57:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb151522740 1 Processor -- start 2024-06-04T04:57:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb151522740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310226820 0x56431022e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb151522740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310227140 0x56431022eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb151522740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310231a00 0x56431022f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb151522740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56430fe6e790 con 0x564310231a00 2024-06-04T04:57:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb151522740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564310144890 con 0x564310227140 2024-06-04T04:57:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb151522740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564310234630 con 0x564310226820 2024-06-04T04:57:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb151522740 10 client.0 _ll_register_callbacks cb 0x56430fe9ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb147fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310231a00 0x56431022f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb147fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310231a00 0x56431022f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54184/0 (socket says 172.21.15.70:54184) 2024-06-04T04:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb147fff640 1 -- 172.21.15.70:0/232689752 learned_addr learned my addr 172.21.15.70:0/232689752 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb14c906640 1 --2- 172.21.15.70:0/232689752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310227140 0x56431022eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb147fff640 1 -- 172.21.15.70:0/232689752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310226820 msgr2=0x56431022e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb147fff640 1 --2- 172.21.15.70:0/232689752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310226820 0x56431022e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb147fff640 1 -- 172.21.15.70:0/232689752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310227140 msgr2=0x56431022eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb147fff640 1 --2- 172.21.15.70:0/232689752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310227140 0x56431022eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb147fff640 1 -- 172.21.15.70:0/232689752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56430ff69620 con 0x564310231a00 2024-06-04T04:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.758+0000 7fb14c906640 1 --2- 172.21.15.70:0/232689752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310227140 0x56431022eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.759+0000 7fb147fff640 1 --2- 172.21.15.70:0/232689752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310231a00 0x56431022f420 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb134005780 tx=0x7fb13400f0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f964c050a5a32bd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.759+0000 7fb127fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:57:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.759+0000 7fb127fff640 1 -- 172.21.15.70:0/232689752 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13400e7a0 con 0x564310231a00 2024-06-04T04:57:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.759+0000 7fb127fff640 1 -- 172.21.15.70:0/232689752 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb13400e940 con 0x564310231a00 2024-06-04T04:57:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.759+0000 7fb151522740 1 -- 172.21.15.70:0/232689752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb134013070 con 0x564310231a00 2024-06-04T04:57:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.759+0000 7fb127fff640 1 -- 172.21.15.70:0/232689752 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb13400eb40 con 0x564310231a00 2024-06-04T04:57:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.759+0000 7fb151522740 10 client.15489 fetch_fsmap learned FSMap version 29 2024-06-04T04:57:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.759+0000 7fb151522740 1 -- 172.21.15.70:0/232689752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564310240410 con 0x564310231a00 2024-06-04T04:57:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.759+0000 7fb127fff640 1 -- 172.21.15.70:0/232689752 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb134013070 con 0x564310231a00 2024-06-04T04:57:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.759+0000 7fb127fff640 1 -- 172.21.15.70:0/232689752 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fb13402c4e0 con 0x564310231a00 2024-06-04T04:57:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.759+0000 7fb151522740 10 client.15489 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:57:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.759+0000 7fb151522740 10 client.15489 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:57:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.759+0000 7fb151522740 20 client.15489 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.759+0000 7fb151522740 1 --2- 172.21.15.70:0/232689752 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564310241820 0x564310243c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.759+0000 7fb151522740 4 client.15489 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:57:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.759+0000 7fb151522740 1 -- 172.21.15.70:0/232689752 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56431019ab80 con 0x564310241820 2024-06-04T04:57:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.760+0000 7fb14c906640 1 --2- 172.21.15.70:0/232689752 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564310241820 0x564310243c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.760+0000 7fb14c906640 1 --2- 172.21.15.70:0/232689752 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564310241820 0x564310243c10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8373e5024db3d34 server_cookie=4a774e2a137d691f in_seq=0 out_seq=0 2024-06-04T04:57:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.760+0000 7fb127fff640 10 client.15489 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.761+0000 7fb127fff640 1 -- 172.21.15.70:0/232689752 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56431019ab80 con 0x564310241820 2024-06-04T04:57:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.761+0000 7fb127fff640 10 client.15489 handle_command_reply: tid=0 2024-06-04T04:57:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.761+0000 7fb151522740 1 client.15489 shutdown 2024-06-04T04:57:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.761+0000 7fb151522740 1 -- 172.21.15.70:0/232689752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310231a00 msgr2=0x56431022f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.761+0000 7fb151522740 1 --2- 172.21.15.70:0/232689752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310231a00 0x56431022f420 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb134005780 tx=0x7fb13400f0b0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.761+0000 7fb151522740 1 -- 172.21.15.70:0/232689752 shutdown_connections 2024-06-04T04:57:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.761+0000 7fb151522740 1 --2- 172.21.15.70:0/232689752 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564310241820 0x564310243c10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.761+0000 7fb151522740 1 --2- 172.21.15.70:0/232689752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310227140 0x56431022eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.761+0000 7fb127fff640 0 client.15489 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.761+0000 7fb151522740 1 --2- 172.21.15.70:0/232689752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310231a00 0x56431022f420 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.761+0000 7fb151522740 1 --2- 172.21.15.70:0/232689752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310226820 0x56431022e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.761+0000 7fb151522740 1 -- 172.21.15.70:0/232689752 >> 172.21.15.70:0/232689752 conn(0x564310222d30 msgr2=0x564310228ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.761+0000 7fb151522740 1 -- 172.21.15.70:0/232689752 shutdown_connections 2024-06-04T04:57:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.761+0000 7fb151522740 1 -- 172.21.15.70:0/232689752 wait complete. 2024-06-04T04:57:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.761+0000 7fb151522740 20 client.15489 trim_cache size 0 max 16384 2024-06-04T04:57:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.776+0000 7fb151522740 1 Processor -- start 2024-06-04T04:57:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.776+0000 7fb151522740 1 -- start start 2024-06-04T04:57:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.776+0000 7fb151522740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310226820 0x564310239c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.776+0000 7fb151522740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310227140 0x56431022e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.776+0000 7fb151522740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310231a00 0x56431022eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.776+0000 7fb151522740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481ac290 con 0x564310226820 2024-06-04T04:57:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.776+0000 7fb151522740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481addd0 con 0x564310231a00 2024-06-04T04:57:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.776+0000 7fb151522740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481ac960 con 0x564310227140 2024-06-04T04:57:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.776+0000 7fb147fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310226820 0x564310239c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.776+0000 7fb147fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310226820 0x564310239c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54196/0 (socket says 172.21.15.70:54196) 2024-06-04T04:57:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.776+0000 7fb147fff640 1 -- 172.21.15.70:0/592355567 learned_addr learned my addr 172.21.15.70:0/592355567 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb147fff640 1 -- 172.21.15.70:0/592355567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310227140 msgr2=0x56431022e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb147fff640 1 --2- 172.21.15.70:0/592355567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310227140 0x56431022e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb147fff640 1 -- 172.21.15.70:0/592355567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310231a00 msgr2=0x56431022eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb147fff640 1 --2- 172.21.15.70:0/592355567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310231a00 0x56431022eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb147fff640 1 -- 172.21.15.70:0/592355567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564310243f80 con 0x564310226820 2024-06-04T04:57:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb147fff640 1 --2- 172.21.15.70:0/592355567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310226820 0x564310239c90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb13400c010 tx=0x7fb13402cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8653859791bd6e3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb1327fc640 1 -- 172.21.15.70:0/592355567 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13400ece0 con 0x564310226820 2024-06-04T04:57:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb1327fc640 1 -- 172.21.15.70:0/592355567 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb134038020 con 0x564310226820 2024-06-04T04:57:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb1327fc640 1 -- 172.21.15.70:0/592355567 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb134004030 con 0x564310226820 2024-06-04T04:57:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb151522740 1 -- 172.21.15.70:0/592355567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310226820 msgr2=0x564310239c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb151522740 1 --2- 172.21.15.70:0/592355567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310226820 0x564310239c90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb13400c010 tx=0x7fb13402cb30 comp rx=0 tx=0).stop 2024-06-04T04:57:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb151522740 1 -- 172.21.15.70:0/592355567 shutdown_connections 2024-06-04T04:57:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb151522740 1 --2- 172.21.15.70:0/592355567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310231a00 0x56431022eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb151522740 1 --2- 172.21.15.70:0/592355567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310227140 0x56431022e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb151522740 1 --2- 172.21.15.70:0/592355567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310226820 0x564310239c90 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb151522740 1 -- 172.21.15.70:0/592355567 >> 172.21.15.70:0/592355567 conn(0x564310222d30 msgr2=0x56431021de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb151522740 1 -- 172.21.15.70:0/592355567 shutdown_connections 2024-06-04T04:57:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.777+0000 7fb151522740 1 -- 172.21.15.70:0/592355567 wait complete. 2024-06-04T04:57:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.778+0000 7fb151522740 1 -- start start 2024-06-04T04:57:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.778+0000 7fb151522740 1 Processor -- start 2024-06-04T04:57:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.778+0000 7fb151522740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310226820 0x5643102a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.778+0000 7fb151522740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310227140 0x5643102a8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.778+0000 7fb151522740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310231a00 0x564310248db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.778+0000 7fb151522740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481ad700 con 0x564310227140 2024-06-04T04:57:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.778+0000 7fb151522740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481ad030 con 0x564310231a00 2024-06-04T04:57:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.778+0000 7fb151522740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481aa270 con 0x564310226820 2024-06-04T04:57:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.778+0000 7fb151522740 10 client.0 _ll_register_callbacks cb 0x7fb13400b570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.778+0000 7fb14c906640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310227140 0x5643102a8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.778+0000 7fb14c906640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310227140 0x5643102a8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54210/0 (socket says 172.21.15.70:54210) 2024-06-04T04:57:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.778+0000 7fb14c906640 1 -- 172.21.15.70:0/1172195252 learned_addr learned my addr 172.21.15.70:0/1172195252 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb147fff640 1 --2- 172.21.15.70:0/1172195252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310226820 0x5643102a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb14c906640 1 -- 172.21.15.70:0/1172195252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310226820 msgr2=0x5643102a8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb14c906640 1 --2- 172.21.15.70:0/1172195252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310226820 0x5643102a8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb14c906640 1 -- 172.21.15.70:0/1172195252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310231a00 msgr2=0x564310248db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb14c906640 1 --2- 172.21.15.70:0/1172195252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310231a00 0x564310248db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb14c906640 1 -- 172.21.15.70:0/1172195252 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643102494f0 con 0x564310227140 2024-06-04T04:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb14c906640 1 --2- 172.21.15.70:0/1172195252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310227140 0x5643102a8f70 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x56431022e200 tx=0x7fb13c00bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e696f9af696cb9e5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb127fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb127fff640 1 -- 172.21.15.70:0/1172195252 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13c01b530 con 0x564310227140 2024-06-04T04:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb127fff640 1 -- 172.21.15.70:0/1172195252 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb13c01b6d0 con 0x564310227140 2024-06-04T04:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb151522740 1 -- 172.21.15.70:0/1172195252 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb134038020 con 0x564310227140 2024-06-04T04:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb127fff640 1 -- 172.21.15.70:0/1172195252 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb13c02b050 con 0x564310227140 2024-06-04T04:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb151522740 10 client.15495 fetch_fsmap learned FSMap version 29 2024-06-04T04:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb151522740 1 -- 172.21.15.70:0/1172195252 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5643102b02f0 con 0x564310227140 2024-06-04T04:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb127fff640 1 -- 172.21.15.70:0/1172195252 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb134038020 con 0x564310227140 2024-06-04T04:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb127fff640 1 -- 172.21.15.70:0/1172195252 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fb13c012b00 con 0x564310227140 2024-06-04T04:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb151522740 10 client.15495 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb151522740 10 client.15495 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb151522740 20 client.15495 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb151522740 1 --2- 172.21.15.70:0/1172195252 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5643102b0a50 0x5643102b2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb151522740 4 client.15495 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T04:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.779+0000 7fb151522740 1 -- 172.21.15.70:0/1172195252 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564310194360 con 0x5643102b0a50 2024-06-04T04:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.780+0000 7fb147fff640 1 --2- 172.21.15.70:0/1172195252 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5643102b0a50 0x5643102b2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.780+0000 7fb127fff640 10 client.15495 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.780+0000 7fb147fff640 1 --2- 172.21.15.70:0/1172195252 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5643102b0a50 0x5643102b2e70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b2f01536946875c server_cookie=215f290cd783e8a4 in_seq=0 out_seq=0 2024-06-04T04:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.788+0000 7fb127fff640 1 -- 172.21.15.70:0/1172195252 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x564310194360 con 0x5643102b0a50 2024-06-04T04:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.788+0000 7fb127fff640 10 client.15495 handle_command_reply: tid=0 2024-06-04T04:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.788+0000 7fb151522740 1 client.15495 shutdown 2024-06-04T04:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.788+0000 7fb151522740 1 -- 172.21.15.70:0/1172195252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310227140 msgr2=0x5643102a8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.788+0000 7fb151522740 1 --2- 172.21.15.70:0/1172195252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310227140 0x5643102a8f70 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x56431022e200 tx=0x7fb13c00bec0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.788+0000 7fb151522740 1 -- 172.21.15.70:0/1172195252 shutdown_connections 2024-06-04T04:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.788+0000 7fb151522740 1 --2- 172.21.15.70:0/1172195252 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5643102b0a50 0x5643102b2e70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.788+0000 7fb151522740 1 --2- 172.21.15.70:0/1172195252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310231a00 0x564310248db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.788+0000 7fb151522740 1 --2- 172.21.15.70:0/1172195252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310227140 0x5643102a8f70 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.788+0000 7fb151522740 1 --2- 172.21.15.70:0/1172195252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310226820 0x5643102a8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.788+0000 7fb151522740 1 -- 172.21.15.70:0/1172195252 >> 172.21.15.70:0/1172195252 conn(0x564310222d30 msgr2=0x56431022e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.788+0000 7fb127fff640 0 client.15495 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.788+0000 7fb151522740 1 -- 172.21.15.70:0/1172195252 shutdown_connections 2024-06-04T04:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.788+0000 7fb151522740 1 -- 172.21.15.70:0/1172195252 wait complete. 2024-06-04T04:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.788+0000 7fb151522740 20 client.15495 trim_cache size 0 max 16384 2024-06-04T04:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T04:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T04:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T04:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.789+0000 7fb14eb6a640 1 -- 172.21.15.70:0/2565970795 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1380b2b10 msgr2=0x7fb1380b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.789+0000 7fb14eb6a640 1 --2- 172.21.15.70:0/2565970795 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1380b2b10 0x7fb1380b4fd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb1401012e0 tx=0x7fb140097690 comp rx=0 tx=0).stop 2024-06-04T04:57:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.789+0000 7fb14eb6a640 1 -- 172.21.15.70:0/2565970795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb14819c920 msgr2=0x7fb14819cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.789+0000 7fb14eb6a640 1 --2- 172.21.15.70:0/2565970795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb14819c920 0x7fb14819cdd0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb140005cc0 tx=0x7fb140032a10 comp rx=0 tx=0).stop 2024-06-04T04:57:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.789+0000 7fb14eb6a640 1 -- 172.21.15.70:0/2565970795 shutdown_connections 2024-06-04T04:57:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.789+0000 7fb14eb6a640 1 --2- 172.21.15.70:0/2565970795 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1380b2b10 0x7fb1380b4fd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.789+0000 7fb14eb6a640 1 --2- 172.21.15.70:0/2565970795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb14819c920 0x7fb14819cdd0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.789+0000 7fb14eb6a640 1 --2- 172.21.15.70:0/2565970795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb148106cd0 0x7fb1481985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.789+0000 7fb14eb6a640 1 --2- 172.21.15.70:0/2565970795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb148102330 0x7fb1481980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.789+0000 7fb14eb6a640 1 -- 172.21.15.70:0/2565970795 >> 172.21.15.70:0/2565970795 conn(0x7fb148077e70 msgr2=0x7fb148105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.789+0000 7fb14eb6a640 1 -- 172.21.15.70:0/2565970795 shutdown_connections 2024-06-04T04:57:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:57:49.789+0000 7fb14eb6a640 1 -- 172.21.15.70:0/2565970795 wait complete. 2024-06-04T04:57:50.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.732+0000 7f77af96e700 1 -- 172.21.15.70:0/477084995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a81084b0 msgr2=0x7f77a8108890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:50.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.732+0000 7f77af96e700 1 --2- 172.21.15.70:0/477084995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a81084b0 0x7f77a8108890 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f77a4009990 tx=0x7f77a4009ca0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.732+0000 7f77af96e700 1 -- 172.21.15.70:0/477084995 shutdown_connections 2024-06-04T04:57:50.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.732+0000 7f77af96e700 1 --2- 172.21.15.70:0/477084995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77a8102e30 0x7f77a810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.732+0000 7f77af96e700 1 --2- 172.21.15.70:0/477084995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a81084b0 0x7f77a8108890 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.732+0000 7f77af96e700 1 --2- 172.21.15.70:0/477084995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77a8102470 0x7f77a81028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.732+0000 7f77af96e700 1 -- 172.21.15.70:0/477084995 >> 172.21.15.70:0/477084995 conn(0x7f77a8079410 msgr2=0x7f77a8079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:50.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.732+0000 7f77af96e700 1 -- 172.21.15.70:0/477084995 shutdown_connections 2024-06-04T04:57:50.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.732+0000 7f77af96e700 1 -- 172.21.15.70:0/477084995 wait complete. 2024-06-04T04:57:50.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.732+0000 7f77af96e700 1 Processor -- start 2024-06-04T04:57:50.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.732+0000 7f77af96e700 1 -- start start 2024-06-04T04:57:50.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77af96e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a8102470 0x7f77a819a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77af96e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77a8102e30 0x7f77a819ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77af96e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77a81084b0 0x7f77a81a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77af96e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f77a8112470 con 0x7f77a8102470 2024-06-04T04:57:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77af96e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f77a81122f0 con 0x7f77a8102e30 2024-06-04T04:57:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77af96e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f77a81125f0 con 0x7f77a81084b0 2024-06-04T04:57:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77ad70a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a8102470 0x7f77a819a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77ad70a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a8102470 0x7f77a819a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54230/0 (socket says 172.21.15.70:54230) 2024-06-04T04:57:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77ad70a700 1 -- 172.21.15.70:0/1681299965 learned_addr learned my addr 172.21.15.70:0/1681299965 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77acf09700 1 --2- 172.21.15.70:0/1681299965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77a8102e30 0x7f77a819ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77adf0b700 1 --2- 172.21.15.70:0/1681299965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77a81084b0 0x7f77a81a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77ad70a700 1 -- 172.21.15.70:0/1681299965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77a81084b0 msgr2=0x7f77a81a5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77ad70a700 1 --2- 172.21.15.70:0/1681299965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77a81084b0 0x7f77a81a5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77ad70a700 1 -- 172.21.15.70:0/1681299965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77a8102e30 msgr2=0x7f77a819ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:50.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77ad70a700 1 --2- 172.21.15.70:0/1681299965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77a8102e30 0x7f77a819ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77ad70a700 1 -- 172.21.15.70:0/1681299965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77a81a5ed0 con 0x7f77a8102470 2024-06-04T04:57:50.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77adf0b700 1 --2- 172.21.15.70:0/1681299965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77a81084b0 0x7f77a81a5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:50.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77ad70a700 1 --2- 172.21.15.70:0/1681299965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a8102470 0x7f77a819a910 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f77a4006010 tx=0x7f77a400bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:50.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f779e7fc700 1 -- 172.21.15.70:0/1681299965 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a4027070 con 0x7f77a8102470 2024-06-04T04:57:50.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f779e7fc700 1 -- 172.21.15.70:0/1681299965 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f77a400fb90 con 0x7f77a8102470 2024-06-04T04:57:50.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f77af96e700 1 -- 172.21.15.70:0/1681299965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77a81a6100 con 0x7f77a8102470 2024-06-04T04:57:50.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.733+0000 7f779e7fc700 1 -- 172.21.15.70:0/1681299965 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a40173f0 con 0x7f77a8102470 2024-06-04T04:57:50.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.734+0000 7f77af96e700 1 -- 172.21.15.70:0/1681299965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77a81a6550 con 0x7f77a8102470 2024-06-04T04:57:50.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.737+0000 7f779e7fc700 1 -- 172.21.15.70:0/1681299965 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7f77a402a070 con 0x7f77a8102470 2024-06-04T04:57:50.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.737+0000 7f779e7fc700 1 --2- 172.21.15.70:0/1681299965 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f77940b2a80 0x7f77940b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.738+0000 7f77acf09700 1 --2- 172.21.15.70:0/1681299965 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f77940b2a80 0x7f77940b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:50.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.738+0000 7f779e7fc700 1 -- 172.21.15.70:0/1681299965 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f77a40e3730 con 0x7f77a8102470 2024-06-04T04:57:50.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.738+0000 7f77ba203b80 1 Processor -- start 2024-06-04T04:57:50.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.738+0000 7f77ba203b80 1 -- start start 2024-06-04T04:57:50.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.738+0000 7f77acf09700 1 --2- 172.21.15.70:0/1681299965 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f77940b2a80 0x7f77940b4f40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7798005fd0 tx=0x7f7798005e30 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:50.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.738+0000 7f77ba203b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6e02d0 0x56057f6e06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.738+0000 7f77ba203b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6da850 0x56057f6dacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.738+0000 7f77ba203b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6db210 0x56057f6e7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.738+0000 7f77ba203b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56057f638d20 con 0x56057f6db210 2024-06-04T04:57:50.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.738+0000 7f77ba203b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56057f6288c0 con 0x56057f6e02d0 2024-06-04T04:57:50.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.738+0000 7f77ba203b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56057f62c900 con 0x56057f6da850 2024-06-04T04:57:50.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.738+0000 7f77adf0b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6db210 0x56057f6e7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:50.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.738+0000 7f77adf0b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6db210 0x56057f6e7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54240/0 (socket says 172.21.15.70:54240) 2024-06-04T04:57:50.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.738+0000 7f77adf0b700 1 -- 172.21.15.70:0/1746350182 learned_addr learned my addr 172.21.15.70:0/1746350182 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:50.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.739+0000 7f77ad70a700 1 --2- 172.21.15.70:0/1746350182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6e02d0 0x56057f6e06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:50.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.739+0000 7f77acf09700 1 --2- 172.21.15.70:0/1746350182 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6da850 0x56057f6dacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:50.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.739+0000 7f77adf0b700 1 -- 172.21.15.70:0/1746350182 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6da850 msgr2=0x56057f6dacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:50.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.739+0000 7f77adf0b700 1 --2- 172.21.15.70:0/1746350182 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6da850 0x56057f6dacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.739+0000 7f77adf0b700 1 -- 172.21.15.70:0/1746350182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6e02d0 msgr2=0x56057f6e06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:50.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.739+0000 7f77adf0b700 1 --2- 172.21.15.70:0/1746350182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6e02d0 0x56057f6e06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.739+0000 7f77adf0b700 1 -- 172.21.15.70:0/1746350182 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56057f5d1f60 con 0x56057f6db210 2024-06-04T04:57:50.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.739+0000 7f77ad70a700 1 --2- 172.21.15.70:0/1746350182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6e02d0 0x56057f6e06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:50.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.739+0000 7f77acf09700 1 --2- 172.21.15.70:0/1746350182 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6da850 0x56057f6dacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:50.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.739+0000 7f77adf0b700 1 --2- 172.21.15.70:0/1746350182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6db210 0x56057f6e7030 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f77a0009400 tx=0x7f77a0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1cb6460592bdd7e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:50.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.739+0000 7f77937fe700 1 -- 172.21.15.70:0/1746350182 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a001a070 con 0x56057f6db210 2024-06-04T04:57:50.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.739+0000 7f77937fe700 1 -- 172.21.15.70:0/1746350182 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f77a0015a80 con 0x56057f6db210 2024-06-04T04:57:50.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.739+0000 7f77937fe700 1 -- 172.21.15.70:0/1746350182 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a00148f0 con 0x56057f6db210 2024-06-04T04:57:50.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.740+0000 7f77ba203b80 1 -- 172.21.15.70:0/1746350182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6db210 msgr2=0x56057f6e7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:50.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.740+0000 7f77ba203b80 1 --2- 172.21.15.70:0/1746350182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6db210 0x56057f6e7030 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f77a0009400 tx=0x7f77a0009920 comp rx=0 tx=0).stop 2024-06-04T04:57:50.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.740+0000 7f77ba203b80 1 -- 172.21.15.70:0/1746350182 shutdown_connections 2024-06-04T04:57:50.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.740+0000 7f77ba203b80 1 --2- 172.21.15.70:0/1746350182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6db210 0x56057f6e7030 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.740+0000 7f77ba203b80 1 --2- 172.21.15.70:0/1746350182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6e02d0 0x56057f6e06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.740+0000 7f77ba203b80 1 --2- 172.21.15.70:0/1746350182 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6da850 0x56057f6dacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.740+0000 7f77ba203b80 1 -- 172.21.15.70:0/1746350182 >> 172.21.15.70:0/1746350182 conn(0x56057f6061f0 msgr2=0x56057f6d5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:50.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.740+0000 7f77ba203b80 1 -- 172.21.15.70:0/1746350182 shutdown_connections 2024-06-04T04:57:50.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.740+0000 7f77ba203b80 1 -- 172.21.15.70:0/1746350182 wait complete. 2024-06-04T04:57:50.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.740+0000 7f77ba203b80 1 -- start start 2024-06-04T04:57:50.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.740+0000 7f77ba203b80 1 Processor -- start 2024-06-04T04:57:50.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77ba203b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6da850 0x56057f6e6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77ba203b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6db210 0x56057f6e3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77ba203b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6e02d0 0x56057f6e4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77ba203b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56057f52ef50 con 0x56057f6da850 2024-06-04T04:57:50.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77ba203b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56057f590d50 con 0x56057f6db210 2024-06-04T04:57:50.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77ad70a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6da850 0x56057f6e6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:50.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77ba203b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56057f6e7690 con 0x56057f6e02d0 2024-06-04T04:57:50.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77ad70a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6da850 0x56057f6e6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54246/0 (socket says 172.21.15.70:54246) 2024-06-04T04:57:50.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77ad70a700 1 -- 172.21.15.70:0/911079116 learned_addr learned my addr 172.21.15.70:0/911079116 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:50.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77ba203b80 10 client.0 _ll_register_callbacks cb 0x56057f5c3900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:50.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77acf09700 1 --2- 172.21.15.70:0/911079116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6db210 0x56057f6e3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:50.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77adf0b700 1 --2- 172.21.15.70:0/911079116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6e02d0 0x56057f6e4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:50.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77ad70a700 1 -- 172.21.15.70:0/911079116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6e02d0 msgr2=0x56057f6e4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:50.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77ad70a700 1 --2- 172.21.15.70:0/911079116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6e02d0 0x56057f6e4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77ad70a700 1 -- 172.21.15.70:0/911079116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6db210 msgr2=0x56057f6e3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:50.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77ad70a700 1 --2- 172.21.15.70:0/911079116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6db210 0x56057f6e3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77ad70a700 1 -- 172.21.15.70:0/911079116 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56057f578880 con 0x56057f6da850 2024-06-04T04:57:50.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77acf09700 1 --2- 172.21.15.70:0/911079116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6db210 0x56057f6e3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:50.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.741+0000 7f77adf0b700 1 --2- 172.21.15.70:0/911079116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6e02d0 0x56057f6e4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:50.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.742+0000 7f77ad70a700 1 --2- 172.21.15.70:0/911079116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6da850 0x56057f6e6c50 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f77a40aa8c0 tx=0x7f77a40e1350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291f160b4b44ada8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:50.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.742+0000 7f7790ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:57:50.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.742+0000 7f7790ff9700 1 -- 172.21.15.70:0/911079116 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a40aa440 con 0x56057f6da850 2024-06-04T04:57:50.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.742+0000 7f7790ff9700 1 -- 172.21.15.70:0/911079116 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f77a40b3270 con 0x56057f6da850 2024-06-04T04:57:50.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.742+0000 7f77ba203b80 1 -- 172.21.15.70:0/911079116 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77a0015a80 con 0x56057f6da850 2024-06-04T04:57:50.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.742+0000 7f7790ff9700 1 -- 172.21.15.70:0/911079116 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f77a40ab030 con 0x56057f6da850 2024-06-04T04:57:50.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.742+0000 7f77ba203b80 10 client.15507 fetch_fsmap learned FSMap version 29 2024-06-04T04:57:50.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.742+0000 7f77ba203b80 1 -- 172.21.15.70:0/911079116 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56057f5e2710 con 0x56057f6da850 2024-06-04T04:57:50.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.742+0000 7f7790ff9700 1 -- 172.21.15.70:0/911079116 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f77a0015a80 con 0x56057f6da850 2024-06-04T04:57:50.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.742+0000 7f7790ff9700 1 -- 172.21.15.70:0/911079116 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f77a4005110 con 0x56057f6da850 2024-06-04T04:57:50.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.743+0000 7f77ba203b80 10 client.15507 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:57:50.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.743+0000 7f77ba203b80 10 client.15507 resolve_mds: resolved name 'h' to daemon mds.h 2024-06-04T04:57:50.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.743+0000 7f77ba203b80 20 client.15507 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:50.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.743+0000 7f77ba203b80 1 --2- 172.21.15.70:0/911079116 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56057f6fc930 0x56057f6fdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.743+0000 7f77ba203b80 4 client.15507 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:57:50.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.743+0000 7f77ba203b80 1 -- 172.21.15.70:0/911079116 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56057f63ac70 con 0x56057f6fc930 2024-06-04T04:57:50.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.743+0000 7f77acf09700 1 --2- 172.21.15.70:0/911079116 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56057f6fc930 0x56057f6fdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:50.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.743+0000 7f77acf09700 1 --2- 172.21.15.70:0/911079116 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56057f6fc930 0x56057f6fdda0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1687a3cf6c08e13e server_cookie=96f614858fa4a57b in_seq=0 out_seq=0 2024-06-04T04:57:50.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.743+0000 7f7790ff9700 10 client.15507 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T04:57:50.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.744+0000 7f7790ff9700 1 -- 172.21.15.70:0/911079116 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56057f63ac70 con 0x56057f6fc930 2024-06-04T04:57:50.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.744+0000 7f7790ff9700 10 client.15507 handle_command_reply: tid=0 2024-06-04T04:57:50.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.744+0000 7f77ba203b80 1 client.15507 shutdown 2024-06-04T04:57:50.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.744+0000 7f77ba203b80 1 -- 172.21.15.70:0/911079116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6da850 msgr2=0x56057f6e6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:50.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.744+0000 7f77ba203b80 1 --2- 172.21.15.70:0/911079116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6da850 0x56057f6e6c50 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f77a40aa8c0 tx=0x7f77a40e1350 comp rx=0 tx=0).stop 2024-06-04T04:57:50.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.744+0000 7f77ba203b80 1 -- 172.21.15.70:0/911079116 shutdown_connections 2024-06-04T04:57:50.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.744+0000 7f77ba203b80 1 --2- 172.21.15.70:0/911079116 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56057f6fc930 0x56057f6fdda0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.744+0000 7f77ba203b80 1 --2- 172.21.15.70:0/911079116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6e02d0 0x56057f6e4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.744+0000 7f77ba203b80 1 --2- 172.21.15.70:0/911079116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6da850 0x56057f6e6c50 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.744+0000 7f77ba203b80 1 --2- 172.21.15.70:0/911079116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6db210 0x56057f6e3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.744+0000 7f77ba203b80 1 -- 172.21.15.70:0/911079116 >> 172.21.15.70:0/911079116 conn(0x56057f6061f0 msgr2=0x56057f6d4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:50.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.744+0000 7f7790ff9700 0 client.15507 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T04:57:50.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.745+0000 7f77ba203b80 1 -- 172.21.15.70:0/911079116 shutdown_connections 2024-06-04T04:57:50.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.745+0000 7f77ba203b80 1 -- 172.21.15.70:0/911079116 wait complete. 2024-06-04T04:57:50.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.745+0000 7f77ba203b80 20 client.15507 trim_cache size 0 max 16384 2024-06-04T04:57:50.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.765+0000 7f77ba203b80 1 Processor -- start 2024-06-04T04:57:50.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.765+0000 7f77ba203b80 1 -- start start 2024-06-04T04:57:50.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f77ba203b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6da850 0x56057f6fc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f77ba203b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6db210 0x56057f6df790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f77ba203b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6e02d0 0x56057f6e0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f77ba203b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56057f6fc0e0 con 0x56057f6db210 2024-06-04T04:57:50.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f77ba203b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56057f6fc7b0 con 0x56057f6e02d0 2024-06-04T04:57:50.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f77ba203b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56057f6fce80 con 0x56057f6da850 2024-06-04T04:57:50.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f77acf09700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6db210 0x56057f6df790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:50.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f77acf09700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6db210 0x56057f6df790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54254/0 (socket says 172.21.15.70:54254) 2024-06-04T04:57:50.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f77acf09700 1 -- 172.21.15.70:0/3128269760 learned_addr learned my addr 172.21.15.70:0/3128269760 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:50.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f77acf09700 1 -- 172.21.15.70:0/3128269760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6da850 msgr2=0x56057f6fc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:50.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f77acf09700 1 --2- 172.21.15.70:0/3128269760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6da850 0x56057f6fc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f77acf09700 1 -- 172.21.15.70:0/3128269760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6e02d0 msgr2=0x56057f6e0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:50.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f77adf0b700 1 --2- 172.21.15.70:0/3128269760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6e02d0 0x56057f6e0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:50.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f77acf09700 1 --2- 172.21.15.70:0/3128269760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6e02d0 0x56057f6e0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f77acf09700 1 -- 172.21.15.70:0/3128269760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56057f6e24c0 con 0x56057f6db210 2024-06-04T04:57:50.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f77adf0b700 1 --2- 172.21.15.70:0/3128269760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6e02d0 0x56057f6e0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:50.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f77acf09700 1 --2- 172.21.15.70:0/3128269760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6db210 0x56057f6df790 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f77a8079b10 tx=0x7f7798014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7745fb2fc9c5ee77 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:50.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f7792ffd700 1 -- 172.21.15.70:0/3128269760 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779801eda0 con 0x56057f6db210 2024-06-04T04:57:50.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f7792ffd700 1 -- 172.21.15.70:0/3128269760 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7798004cc0 con 0x56057f6db210 2024-06-04T04:57:50.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.766+0000 7f7792ffd700 1 -- 172.21.15.70:0/3128269760 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77980273a0 con 0x56057f6db210 2024-06-04T04:57:50.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.767+0000 7f77ba203b80 1 -- 172.21.15.70:0/3128269760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6db210 msgr2=0x56057f6df790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:50.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.767+0000 7f77ba203b80 1 --2- 172.21.15.70:0/3128269760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6db210 0x56057f6df790 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f77a8079b10 tx=0x7f7798014000 comp rx=0 tx=0).stop 2024-06-04T04:57:50.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.767+0000 7f77ba203b80 1 -- 172.21.15.70:0/3128269760 shutdown_connections 2024-06-04T04:57:50.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.767+0000 7f77ba203b80 1 --2- 172.21.15.70:0/3128269760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6e02d0 0x56057f6e0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.767+0000 7f77ba203b80 1 --2- 172.21.15.70:0/3128269760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6da850 0x56057f6fc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.767+0000 7f77ba203b80 1 --2- 172.21.15.70:0/3128269760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6db210 0x56057f6df790 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.767+0000 7f77ba203b80 1 -- 172.21.15.70:0/3128269760 >> 172.21.15.70:0/3128269760 conn(0x56057f6061f0 msgr2=0x56057f6df080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:50.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.767+0000 7f77ba203b80 1 -- 172.21.15.70:0/3128269760 shutdown_connections 2024-06-04T04:57:50.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.767+0000 7f77ba203b80 1 -- 172.21.15.70:0/3128269760 wait complete. 2024-06-04T04:57:50.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.767+0000 7f77ba203b80 1 -- start start 2024-06-04T04:57:50.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77ba203b80 1 Processor -- start 2024-06-04T04:57:50.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77ba203b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6da850 0x56057f707220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77ba203b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6db210 0x56057f707760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77ba203b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6e02d0 0x56057f7001c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77ba203b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56057f6fdc20 con 0x56057f6e02d0 2024-06-04T04:57:50.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77ba203b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56057f6fd550 con 0x56057f6da850 2024-06-04T04:57:50.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77ba203b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56057f6fe2f0 con 0x56057f6db210 2024-06-04T04:57:50.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77ba203b80 10 client.0 _ll_register_callbacks cb 0x56057f5c3900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:50.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77adf0b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6e02d0 0x56057f7001c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:50.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77ad70a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6da850 0x56057f707220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:50.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77adf0b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6e02d0 0x56057f7001c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54268/0 (socket says 172.21.15.70:54268) 2024-06-04T04:57:50.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77ad70a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6da850 0x56057f707220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:35018/0 (socket says 172.21.15.70:35018) 2024-06-04T04:57:50.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77adf0b700 1 -- 172.21.15.70:0/2265144320 learned_addr learned my addr 172.21.15.70:0/2265144320 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:50.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77acf09700 1 --2- 172.21.15.70:0/2265144320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6db210 0x56057f707760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:50.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77adf0b700 1 -- 172.21.15.70:0/2265144320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6db210 msgr2=0x56057f707760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:50.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77adf0b700 1 --2- 172.21.15.70:0/2265144320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6db210 0x56057f707760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77adf0b700 1 -- 172.21.15.70:0/2265144320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6da850 msgr2=0x56057f707220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:50.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77adf0b700 1 --2- 172.21.15.70:0/2265144320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6da850 0x56057f707220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77adf0b700 1 -- 172.21.15.70:0/2265144320 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56057f700700 con 0x56057f6e02d0 2024-06-04T04:57:50.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.768+0000 7f77ad70a700 1 --2- 172.21.15.70:0/2265144320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6da850 0x56057f707220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:57:50.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.769+0000 7f77adf0b700 1 --2- 172.21.15.70:0/2265144320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6e02d0 0x56057f7001c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f77a00093a0 tx=0x7f77a00147b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea513407491c6182 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:50.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.769+0000 7f7790ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:57:50.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.769+0000 7f7790ff9700 1 -- 172.21.15.70:0/2265144320 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a0022020 con 0x56057f6e02d0 2024-06-04T04:57:50.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.769+0000 7f7790ff9700 1 -- 172.21.15.70:0/2265144320 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f77a0014ba0 con 0x56057f6e02d0 2024-06-04T04:57:50.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.769+0000 7f77ba203b80 1 -- 172.21.15.70:0/2265144320 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7798004cc0 con 0x56057f6e02d0 2024-06-04T04:57:50.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.769+0000 7f7790ff9700 1 -- 172.21.15.70:0/2265144320 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f77a002d050 con 0x56057f6e02d0 2024-06-04T04:57:50.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.769+0000 7f77ba203b80 10 client.15513 fetch_fsmap learned FSMap version 29 2024-06-04T04:57:50.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.769+0000 7f77ba203b80 1 -- 172.21.15.70:0/2265144320 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56057f7687d0 con 0x56057f6e02d0 2024-06-04T04:57:50.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.769+0000 7f7790ff9700 1 -- 172.21.15.70:0/2265144320 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7798004cc0 con 0x56057f6e02d0 2024-06-04T04:57:50.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.769+0000 7f7790ff9700 1 -- 172.21.15.70:0/2265144320 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f77a001d030 con 0x56057f6e02d0 2024-06-04T04:57:50.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.769+0000 7f77ba203b80 10 client.15513 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:57:50.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.769+0000 7f77ba203b80 10 client.15513 resolve_mds: resolved name 'h' to daemon mds.h 2024-06-04T04:57:50.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.769+0000 7f77ba203b80 20 client.15513 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:50.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.769+0000 7f77ba203b80 1 --2- 172.21.15.70:0/2265144320 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56057f76d030 0x56057f76bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:50.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.769+0000 7f77ba203b80 4 client.15513 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-04T04:57:50.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.769+0000 7f77ba203b80 1 -- 172.21.15.70:0/2265144320 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "status"}) -- 0x56057f634450 con 0x56057f76d030 2024-06-04T04:57:50.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.770+0000 7f77ad70a700 1 --2- 172.21.15.70:0/2265144320 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56057f76d030 0x56057f76bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:50.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.770+0000 7f7790ff9700 10 client.15513 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T04:57:50.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.770+0000 7f77ad70a700 1 --2- 172.21.15.70:0/2265144320 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56057f76d030 0x56057f76bd40 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c144c3850df0ea2e server_cookie=cc63a32e11511770 in_seq=0 out_seq=0 2024-06-04T04:57:50.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.770+0000 7f7790ff9700 1 -- 172.21.15.70:0/2265144320 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56057f634450 con 0x56057f76d030 2024-06-04T04:57:50.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.770+0000 7f7790ff9700 10 client.15513 handle_command_reply: tid=0 2024-06-04T04:57:50.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.770+0000 7f77ba203b80 1 client.15513 shutdown 2024-06-04T04:57:50.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.770+0000 7f77ba203b80 1 -- 172.21.15.70:0/2265144320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6e02d0 msgr2=0x56057f7001c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:50.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.770+0000 7f77ba203b80 1 --2- 172.21.15.70:0/2265144320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6e02d0 0x56057f7001c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f77a00093a0 tx=0x7f77a00147b0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.771+0000 7f77ba203b80 1 -- 172.21.15.70:0/2265144320 shutdown_connections 2024-06-04T04:57:50.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.771+0000 7f77ba203b80 1 --2- 172.21.15.70:0/2265144320 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56057f76d030 0x56057f76bd40 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.771+0000 7f77ba203b80 1 --2- 172.21.15.70:0/2265144320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56057f6e02d0 0x56057f7001c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.771+0000 7f77ba203b80 1 --2- 172.21.15.70:0/2265144320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56057f6da850 0x56057f707220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.771+0000 7f77ba203b80 1 --2- 172.21.15.70:0/2265144320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56057f6db210 0x56057f707760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.771+0000 7f77ba203b80 1 -- 172.21.15.70:0/2265144320 >> 172.21.15.70:0/2265144320 conn(0x56057f6061f0 msgr2=0x56057f6ddc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:50.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.771+0000 7f7790ff9700 0 client.15513 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T04:57:50.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.771+0000 7f77ba203b80 1 -- 172.21.15.70:0/2265144320 shutdown_connections 2024-06-04T04:57:50.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.771+0000 7f77ba203b80 1 -- 172.21.15.70:0/2265144320 wait complete. 2024-06-04T04:57:50.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.771+0000 7f77ba203b80 20 client.15513 trim_cache size 0 max 16384 2024-06-04T04:57:50.779 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:57:50.779 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:57:50.779 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 4, 2024-06-04T04:57:50.779 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24410, 2024-06-04T04:57:50.779 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-06-04T04:57:50.779 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-06-04T04:57:50.779 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-06-04T04:57:50.779 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 237.33477698799999, 2024-06-04T04:57:50.779 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 29, 2024-06-04T04:57:50.780 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 82, 2024-06-04T04:57:50.780 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 81, 2024-06-04T04:57:50.780 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 280.79616650899999 2024-06-04T04:57:50.780 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:57:50.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.771+0000 7f77af96e700 1 -- 172.21.15.70:0/1681299965 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f77940b2a80 msgr2=0x7f77940b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:50.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.771+0000 7f77af96e700 1 --2- 172.21.15.70:0/1681299965 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f77940b2a80 0x7f77940b4f40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7798005fd0 tx=0x7f7798005e30 comp rx=0 tx=0).stop 2024-06-04T04:57:50.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.771+0000 7f77af96e700 1 -- 172.21.15.70:0/1681299965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a8102470 msgr2=0x7f77a819a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:50.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.771+0000 7f77af96e700 1 --2- 172.21.15.70:0/1681299965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a8102470 0x7f77a819a910 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f77a4006010 tx=0x7f77a400bfa0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.772+0000 7f77af96e700 1 -- 172.21.15.70:0/1681299965 shutdown_connections 2024-06-04T04:57:50.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.772+0000 7f77af96e700 1 --2- 172.21.15.70:0/1681299965 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f77940b2a80 0x7f77940b4f40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.772+0000 7f77af96e700 1 --2- 172.21.15.70:0/1681299965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77a81084b0 0x7f77a81a5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.772+0000 7f77af96e700 1 --2- 172.21.15.70:0/1681299965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a8102470 0x7f77a819a910 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.772+0000 7f77af96e700 1 --2- 172.21.15.70:0/1681299965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77a8102e30 0x7f77a819ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:50.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.772+0000 7f77af96e700 1 -- 172.21.15.70:0/1681299965 >> 172.21.15.70:0/1681299965 conn(0x7f77a8079410 msgr2=0x7f77a8100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:50.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.772+0000 7f77af96e700 1 -- 172.21.15.70:0/1681299965 shutdown_connections 2024-06-04T04:57:50.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:50.772+0000 7f77af96e700 1 -- 172.21.15.70:0/1681299965 wait complete. 2024-06-04T04:57:51.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:51 smithi070 ceph-mon[32631]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-06-04T04:57:51.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:51 smithi112 ceph-mon[36453]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-06-04T04:57:51.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:51 smithi098 ceph-mon[35569]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-06-04T04:57:51.499 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.f status 2024-06-04T04:57:51.744 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:53.267 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:53 smithi070 ceph-mon[32631]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.1 KiB/s wr, 19 op/s 2024-06-04T04:57:53.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:53 smithi112 ceph-mon[36453]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.1 KiB/s wr, 19 op/s 2024-06-04T04:57:53.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:53 smithi098 ceph-mon[35569]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.1 KiB/s wr, 19 op/s 2024-06-04T04:57:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.494+0000 7facd526a700 1 -- 172.21.15.70:0/1217824248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd01095a0 msgr2=0x7facd0109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.494+0000 7facd526a700 1 --2- 172.21.15.70:0/1217824248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd01095a0 0x7facd0109980 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7facc0009990 tx=0x7facc0009ca0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.494+0000 7facd526a700 1 -- 172.21.15.70:0/1217824248 shutdown_connections 2024-06-04T04:57:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.494+0000 7facd526a700 1 --2- 172.21.15.70:0/1217824248 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7facd006c470 0x7facd01103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.494+0000 7facd526a700 1 --2- 172.21.15.70:0/1217824248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd01095a0 0x7facd0109980 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.494+0000 7facd526a700 1 --2- 172.21.15.70:0/1217824248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd006bab0 0x7facd006bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.494+0000 7facd526a700 1 -- 172.21.15.70:0/1217824248 >> 172.21.15.70:0/1217824248 conn(0x7facd00793b0 msgr2=0x7facd00797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.494+0000 7facd526a700 1 -- 172.21.15.70:0/1217824248 shutdown_connections 2024-06-04T04:57:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.494+0000 7facd526a700 1 -- 172.21.15.70:0/1217824248 wait complete. 2024-06-04T04:57:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.495+0000 7facd526a700 1 Processor -- start 2024-06-04T04:57:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.495+0000 7facd526a700 1 -- start start 2024-06-04T04:57:53.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.495+0000 7facd526a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd006bab0 0x7facd0102500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.495+0000 7facd526a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd006c470 0x7facd0102a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.495+0000 7facd526a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7facd01095a0 0x7facd0102f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.495+0000 7facd526a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7facd0113660 con 0x7facd006c470 2024-06-04T04:57:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.495+0000 7facd526a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7facd01134e0 con 0x7facd006bab0 2024-06-04T04:57:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.495+0000 7facd526a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7facd01137e0 con 0x7facd01095a0 2024-06-04T04:57:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.495+0000 7facc67fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd006c470 0x7facd0102a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.496+0000 7facc67fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd006c470 0x7facd0102a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54280/0 (socket says 172.21.15.70:54280) 2024-06-04T04:57:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.496+0000 7facc67fc700 1 -- 172.21.15.70:0/1092934828 learned_addr learned my addr 172.21.15.70:0/1092934828 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.496+0000 7facceffd700 1 --2- 172.21.15.70:0/1092934828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd006bab0 0x7facd0102500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.496+0000 7faccf7fe700 1 --2- 172.21.15.70:0/1092934828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7facd01095a0 0x7facd0102f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.496+0000 7facc67fc700 1 -- 172.21.15.70:0/1092934828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7facd01095a0 msgr2=0x7facd0102f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.496+0000 7facc67fc700 1 --2- 172.21.15.70:0/1092934828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7facd01095a0 0x7facd0102f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.496+0000 7facc67fc700 1 -- 172.21.15.70:0/1092934828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd006bab0 msgr2=0x7facd0102500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.496+0000 7facc67fc700 1 --2- 172.21.15.70:0/1092934828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd006bab0 0x7facd0102500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.496+0000 7facc67fc700 1 -- 172.21.15.70:0/1092934828 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd01a81c0 con 0x7facd006c470 2024-06-04T04:57:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.496+0000 7facceffd700 1 --2- 172.21.15.70:0/1092934828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd006bab0 0x7facd0102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.496+0000 7facc67fc700 1 --2- 172.21.15.70:0/1092934828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd006c470 0x7facd0102a40 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7facb800b530 tx=0x7facb800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.496+0000 7facccff9700 1 -- 172.21.15.70:0/1092934828 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb8015020 con 0x7facd006c470 2024-06-04T04:57:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.496+0000 7facccff9700 1 -- 172.21.15.70:0/1092934828 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7facb8011cc0 con 0x7facd006c470 2024-06-04T04:57:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.496+0000 7facd526a700 1 -- 172.21.15.70:0/1092934828 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facd01a84b0 con 0x7facd006c470 2024-06-04T04:57:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.496+0000 7facccff9700 1 -- 172.21.15.70:0/1092934828 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb8010d30 con 0x7facd006c470 2024-06-04T04:57:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.496+0000 7facd526a700 1 -- 172.21.15.70:0/1092934828 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facd01a89f0 con 0x7facd006c470 2024-06-04T04:57:53.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.499+0000 7facccff9700 1 -- 172.21.15.70:0/1092934828 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7facb8018020 con 0x7facd006c470 2024-06-04T04:57:53.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.499+0000 7facccff9700 1 --2- 172.21.15.70:0/1092934828 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7facbc0b2a80 0x7facbc0b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.499+0000 7facceffd700 1 --2- 172.21.15.70:0/1092934828 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7facbc0b2a80 0x7facbc0b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.499+0000 7facccff9700 1 -- 172.21.15.70:0/1092934828 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7facb80e1ef0 con 0x7facd006c470 2024-06-04T04:57:53.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.500+0000 7facdfaffb80 1 Processor -- start 2024-06-04T04:57:53.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.500+0000 7facceffd700 1 --2- 172.21.15.70:0/1092934828 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7facbc0b2a80 0x7facbc0b4f40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7facc000b5c0 tx=0x7facc001a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:53.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.500+0000 7facdfaffb80 1 -- start start 2024-06-04T04:57:53.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.500+0000 7facdfaffb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded436b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.500+0000 7facdfaffb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3d850 0x55fcded3dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.500+0000 7facdfaffb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded3e210 0x55fcded4a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.500+0000 7facdfaffb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fcdec9bd20 con 0x55fcded432d0 2024-06-04T04:57:53.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.500+0000 7facdfaffb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fcdec8b8c0 con 0x55fcded3e210 2024-06-04T04:57:53.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.500+0000 7facdfaffb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fcdec8f900 con 0x55fcded3d850 2024-06-04T04:57:53.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.500+0000 7facceffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.500+0000 7facceffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded436b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54284/0 (socket says 172.21.15.70:54284) 2024-06-04T04:57:53.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.500+0000 7facceffd700 1 -- 172.21.15.70:0/2890196888 learned_addr learned my addr 172.21.15.70:0/2890196888 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:53.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.500+0000 7faccf7fe700 1 --2- 172.21.15.70:0/2890196888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded3e210 0x55fcded4a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.501+0000 7facc67fc700 1 --2- 172.21.15.70:0/2890196888 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3d850 0x55fcded3dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.501+0000 7facceffd700 1 -- 172.21.15.70:0/2890196888 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3d850 msgr2=0x55fcded3dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:53.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.501+0000 7facceffd700 1 --2- 172.21.15.70:0/2890196888 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3d850 0x55fcded3dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.501+0000 7facceffd700 1 -- 172.21.15.70:0/2890196888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded3e210 msgr2=0x55fcded4a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.501+0000 7facceffd700 1 --2- 172.21.15.70:0/2890196888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded3e210 0x55fcded4a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.501+0000 7facceffd700 1 -- 172.21.15.70:0/2890196888 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcdec34f60 con 0x55fcded432d0 2024-06-04T04:57:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.501+0000 7facc67fc700 1 --2- 172.21.15.70:0/2890196888 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3d850 0x55fcded3dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.501+0000 7facceffd700 1 --2- 172.21.15.70:0/2890196888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded436b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7facc0009990 tx=0x7facc001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41df60cdd91c6e11 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.501+0000 7facc57fa700 1 -- 172.21.15.70:0/2890196888 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc000fb20 con 0x55fcded432d0 2024-06-04T04:57:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.501+0000 7facc57fa700 1 -- 172.21.15.70:0/2890196888 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7facc0032070 con 0x55fcded432d0 2024-06-04T04:57:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.501+0000 7facc57fa700 1 -- 172.21.15.70:0/2890196888 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc002d3f0 con 0x55fcded432d0 2024-06-04T04:57:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.502+0000 7facdfaffb80 1 -- 172.21.15.70:0/2890196888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 msgr2=0x55fcded436b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.502+0000 7facdfaffb80 1 --2- 172.21.15.70:0/2890196888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded436b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7facc0009990 tx=0x7facc001e000 comp rx=0 tx=0).stop 2024-06-04T04:57:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.502+0000 7facdfaffb80 1 -- 172.21.15.70:0/2890196888 shutdown_connections 2024-06-04T04:57:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.502+0000 7facdfaffb80 1 --2- 172.21.15.70:0/2890196888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded3e210 0x55fcded4a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.502+0000 7facdfaffb80 1 --2- 172.21.15.70:0/2890196888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded436b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.502+0000 7facdfaffb80 1 --2- 172.21.15.70:0/2890196888 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3d850 0x55fcded3dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.502+0000 7facdfaffb80 1 -- 172.21.15.70:0/2890196888 >> 172.21.15.70:0/2890196888 conn(0x55fcdec691f0 msgr2=0x55fcded38390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:53.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.502+0000 7facdfaffb80 1 -- 172.21.15.70:0/2890196888 shutdown_connections 2024-06-04T04:57:53.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.502+0000 7facdfaffb80 1 -- 172.21.15.70:0/2890196888 wait complete. 2024-06-04T04:57:53.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.502+0000 7facdfaffb80 1 -- start start 2024-06-04T04:57:53.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.502+0000 7facdfaffb80 1 Processor -- start 2024-06-04T04:57:53.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7facdfaffb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded3d850 0x55fcded49d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7facdfaffb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3e210 0x55fcded47010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7facdfaffb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded47550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7facdfaffb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fcdeb91f50 con 0x55fcded432d0 2024-06-04T04:57:53.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7facdfaffb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fcdebf3d50 con 0x55fcded3d850 2024-06-04T04:57:53.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7facdfaffb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fcded4a630 con 0x55fcded3e210 2024-06-04T04:57:53.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7facdfaffb80 10 client.0 _ll_register_callbacks cb 0x55fcdec26900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:53.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7faccf7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded47550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7faccf7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded47550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54286/0 (socket says 172.21.15.70:54286) 2024-06-04T04:57:53.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7faccf7fe700 1 -- 172.21.15.70:0/2642887581 learned_addr learned my addr 172.21.15.70:0/2642887581 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:53.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7facceffd700 1 --2- 172.21.15.70:0/2642887581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded3d850 0x55fcded49d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7facc67fc700 1 --2- 172.21.15.70:0/2642887581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3e210 0x55fcded47010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7faccf7fe700 1 -- 172.21.15.70:0/2642887581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3e210 msgr2=0x55fcded47010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:53.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7faccf7fe700 1 --2- 172.21.15.70:0/2642887581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3e210 0x55fcded47010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7faccf7fe700 1 -- 172.21.15.70:0/2642887581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded3d850 msgr2=0x55fcded49d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:53.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7faccf7fe700 1 --2- 172.21.15.70:0/2642887581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded3d850 0x55fcded49d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7faccf7fe700 1 -- 172.21.15.70:0/2642887581 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcdebdb880 con 0x55fcded432d0 2024-06-04T04:57:53.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7faccf7fe700 1 --2- 172.21.15.70:0/2642887581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded47550 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7facc800be50 tx=0x7facc800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e9d3024a9408e2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:53.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7facb67fc700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:57:53.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7facb67fc700 1 -- 172.21.15.70:0/2642887581 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc8007ad0 con 0x55fcded432d0 2024-06-04T04:57:53.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7facb67fc700 1 -- 172.21.15.70:0/2642887581 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7facc8007c70 con 0x55fcded432d0 2024-06-04T04:57:53.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.503+0000 7facdfaffb80 1 -- 172.21.15.70:0/2642887581 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facc0032070 con 0x55fcded432d0 2024-06-04T04:57:53.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.504+0000 7facb67fc700 1 -- 172.21.15.70:0/2642887581 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7facc8025050 con 0x55fcded432d0 2024-06-04T04:57:53.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.504+0000 7facdfaffb80 10 client.15525 fetch_fsmap learned FSMap version 29 2024-06-04T04:57:53.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.504+0000 7facdfaffb80 1 -- 172.21.15.70:0/2642887581 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fcdec45710 con 0x55fcded432d0 2024-06-04T04:57:53.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.504+0000 7facb67fc700 1 -- 172.21.15.70:0/2642887581 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7facc0032070 con 0x55fcded432d0 2024-06-04T04:57:53.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.504+0000 7facb67fc700 1 -- 172.21.15.70:0/2642887581 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7facc8020030 con 0x55fcded432d0 2024-06-04T04:57:53.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.504+0000 7facdfaffb80 10 client.15525 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:57:53.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.504+0000 7facdfaffb80 10 client.15525 resolve_mds: resolved name 'f' to daemon mds.f 2024-06-04T04:57:53.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.504+0000 7facdfaffb80 20 client.15525 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:53.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.504+0000 7facdfaffb80 1 --2- 172.21.15.70:0/2642887581 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55fcded571a0 0x55fcded58610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.504+0000 7facdfaffb80 4 client.15525 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:57:53.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.504+0000 7facdfaffb80 1 -- 172.21.15.70:0/2642887581 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcdec9dc70 con 0x55fcded571a0 2024-06-04T04:57:53.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.504+0000 7facceffd700 1 --2- 172.21.15.70:0/2642887581 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55fcded571a0 0x55fcded58610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.505+0000 7facceffd700 1 --2- 172.21.15.70:0/2642887581 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55fcded571a0 0x55fcded58610 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f43db951afb0545f server_cookie=a2f446c79ad74082 in_seq=0 out_seq=0 2024-06-04T04:57:53.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.505+0000 7facb67fc700 10 client.15525 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T04:57:53.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.506+0000 7facb67fc700 1 -- 172.21.15.70:0/2642887581 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fcdec9dc70 con 0x55fcded571a0 2024-06-04T04:57:53.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.506+0000 7facb67fc700 10 client.15525 handle_command_reply: tid=0 2024-06-04T04:57:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.506+0000 7facdfaffb80 1 client.15525 shutdown 2024-06-04T04:57:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.506+0000 7facdfaffb80 1 -- 172.21.15.70:0/2642887581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 msgr2=0x55fcded47550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.506+0000 7facdfaffb80 1 --2- 172.21.15.70:0/2642887581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded47550 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7facc800be50 tx=0x7facc800be80 comp rx=0 tx=0).stop 2024-06-04T04:57:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.506+0000 7facdfaffb80 1 -- 172.21.15.70:0/2642887581 shutdown_connections 2024-06-04T04:57:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.506+0000 7facdfaffb80 1 --2- 172.21.15.70:0/2642887581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded47550 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.506+0000 7facdfaffb80 1 --2- 172.21.15.70:0/2642887581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded3d850 0x55fcded49d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.506+0000 7facdfaffb80 1 --2- 172.21.15.70:0/2642887581 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55fcded571a0 0x55fcded58610 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.506+0000 7facdfaffb80 1 --2- 172.21.15.70:0/2642887581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3e210 0x55fcded47010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.506+0000 7facb67fc700 0 client.15525 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T04:57:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.506+0000 7facdfaffb80 1 -- 172.21.15.70:0/2642887581 >> 172.21.15.70:0/2642887581 conn(0x55fcdec691f0 msgr2=0x55fcded37aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.506+0000 7facdfaffb80 1 -- 172.21.15.70:0/2642887581 shutdown_connections 2024-06-04T04:57:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.506+0000 7facdfaffb80 1 -- 172.21.15.70:0/2642887581 wait complete. 2024-06-04T04:57:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.506+0000 7facdfaffb80 20 client.15525 trim_cache size 0 max 16384 2024-06-04T04:57:53.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.527+0000 7facdfaffb80 1 Processor -- start 2024-06-04T04:57:53.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.527+0000 7facdfaffb80 1 -- start start 2024-06-04T04:57:53.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.527+0000 7facdfaffb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded3d850 0x55fcded47580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.527+0000 7facdfaffb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3e210 0x55fcded42850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.527+0000 7facdfaffb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded38220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.527+0000 7facdfaffb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fcded56950 con 0x55fcded432d0 2024-06-04T04:57:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.527+0000 7facdfaffb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fcded57020 con 0x55fcded3d850 2024-06-04T04:57:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.527+0000 7facdfaffb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fcded576f0 con 0x55fcded3e210 2024-06-04T04:57:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.527+0000 7faccf7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded38220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.527+0000 7faccf7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded38220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54298/0 (socket says 172.21.15.70:54298) 2024-06-04T04:57:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.527+0000 7faccf7fe700 1 -- 172.21.15.70:0/3458959987 learned_addr learned my addr 172.21.15.70:0/3458959987 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.527+0000 7facceffd700 1 --2- 172.21.15.70:0/3458959987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded3d850 0x55fcded47580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.527+0000 7facc67fc700 1 --2- 172.21.15.70:0/3458959987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3e210 0x55fcded42850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.528+0000 7faccf7fe700 1 -- 172.21.15.70:0/3458959987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3e210 msgr2=0x55fcded42850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.528+0000 7faccf7fe700 1 --2- 172.21.15.70:0/3458959987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3e210 0x55fcded42850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.528+0000 7faccf7fe700 1 -- 172.21.15.70:0/3458959987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded3d850 msgr2=0x55fcded47580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.528+0000 7faccf7fe700 1 --2- 172.21.15.70:0/3458959987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded3d850 0x55fcded47580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.528+0000 7faccf7fe700 1 -- 172.21.15.70:0/3458959987 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcded49950 con 0x55fcded432d0 2024-06-04T04:57:53.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.528+0000 7faccf7fe700 1 --2- 172.21.15.70:0/3458959987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded38220 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7facc80093a0 tx=0x7facc8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c54c4de313f32f1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:53.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.528+0000 7facc4ff9700 1 -- 172.21.15.70:0/3458959987 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc802f020 con 0x55fcded432d0 2024-06-04T04:57:53.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.528+0000 7facc4ff9700 1 -- 172.21.15.70:0/3458959987 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7facc8017da0 con 0x55fcded432d0 2024-06-04T04:57:53.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.528+0000 7facc4ff9700 1 -- 172.21.15.70:0/3458959987 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc8018400 con 0x55fcded432d0 2024-06-04T04:57:53.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.529+0000 7facdfaffb80 1 -- 172.21.15.70:0/3458959987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 msgr2=0x55fcded38220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.529+0000 7facdfaffb80 1 --2- 172.21.15.70:0/3458959987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded38220 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7facc80093a0 tx=0x7facc8004ad0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.529+0000 7facdfaffb80 1 -- 172.21.15.70:0/3458959987 shutdown_connections 2024-06-04T04:57:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.529+0000 7facdfaffb80 1 --2- 172.21.15.70:0/3458959987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded432d0 0x55fcded38220 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.529+0000 7facdfaffb80 1 --2- 172.21.15.70:0/3458959987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded3d850 0x55fcded47580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.529+0000 7facdfaffb80 1 --2- 172.21.15.70:0/3458959987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3e210 0x55fcded42850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.529+0000 7facdfaffb80 1 -- 172.21.15.70:0/3458959987 >> 172.21.15.70:0/3458959987 conn(0x55fcdec691f0 msgr2=0x55fcded42140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.529+0000 7facdfaffb80 1 -- 172.21.15.70:0/3458959987 shutdown_connections 2024-06-04T04:57:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.529+0000 7facdfaffb80 1 -- 172.21.15.70:0/3458959987 wait complete. 2024-06-04T04:57:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.529+0000 7facdfaffb80 1 -- start start 2024-06-04T04:57:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.529+0000 7facdfaffb80 1 Processor -- start 2024-06-04T04:57:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facdfaffb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3d850 0x55fcded378d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facdfaffb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded3e210 0x55fcded37e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facdfaffb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded432d0 0x55fcded38350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facdfaffb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fcded58490 con 0x55fcded3e210 2024-06-04T04:57:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facdfaffb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fcded57dc0 con 0x55fcded432d0 2024-06-04T04:57:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facdfaffb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fcded58b60 con 0x55fcded3d850 2024-06-04T04:57:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facdfaffb80 10 client.0 _ll_register_callbacks cb 0x55fcdec26900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facc67fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded3e210 0x55fcded37e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facc67fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded3e210 0x55fcded37e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54308/0 (socket says 172.21.15.70:54308) 2024-06-04T04:57:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facceffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3d850 0x55fcded378d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facc67fc700 1 -- 172.21.15.70:0/2368208710 learned_addr learned my addr 172.21.15.70:0/2368208710 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7faccf7fe700 1 --2- 172.21.15.70:0/2368208710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded432d0 0x55fcded38350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facc67fc700 1 -- 172.21.15.70:0/2368208710 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3d850 msgr2=0x55fcded378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facc67fc700 1 --2- 172.21.15.70:0/2368208710 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3d850 0x55fcded378d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facc67fc700 1 -- 172.21.15.70:0/2368208710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded432d0 msgr2=0x55fcded38350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:53.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facc67fc700 1 --2- 172.21.15.70:0/2368208710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded432d0 0x55fcded38350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facc67fc700 1 -- 172.21.15.70:0/2368208710 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcded60110 con 0x55fcded3e210 2024-06-04T04:57:53.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facc67fc700 1 --2- 172.21.15.70:0/2368208710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded3e210 0x55fcded37e10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7facb80a9100 tx=0x7facb80106d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=789e7be5773804cf server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:53.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facb67fc700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:57:53.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.530+0000 7facb67fc700 1 -- 172.21.15.70:0/2368208710 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb80a8c80 con 0x55fcded3e210 2024-06-04T04:57:53.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.531+0000 7facb67fc700 1 -- 172.21.15.70:0/2368208710 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7facb80df830 con 0x55fcded3e210 2024-06-04T04:57:53.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.531+0000 7facdfaffb80 1 -- 172.21.15.70:0/2368208710 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facc8017da0 con 0x55fcded3e210 2024-06-04T04:57:53.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.531+0000 7facb67fc700 1 -- 172.21.15.70:0/2368208710 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7facb80dfa30 con 0x55fcded3e210 2024-06-04T04:57:53.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.531+0000 7facdfaffb80 10 client.15531 fetch_fsmap learned FSMap version 29 2024-06-04T04:57:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.531+0000 7facdfaffb80 1 -- 172.21.15.70:0/2368208710 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fcdedc2da0 con 0x55fcded3e210 2024-06-04T04:57:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.531+0000 7facb67fc700 1 -- 172.21.15.70:0/2368208710 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7facc8017da0 con 0x55fcded3e210 2024-06-04T04:57:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.531+0000 7facb67fc700 1 -- 172.21.15.70:0/2368208710 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7facb80a9460 con 0x55fcded3e210 2024-06-04T04:57:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.531+0000 7facdfaffb80 10 client.15531 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:57:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.531+0000 7facdfaffb80 10 client.15531 resolve_mds: resolved name 'f' to daemon mds.f 2024-06-04T04:57:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.531+0000 7facdfaffb80 20 client.15531 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.531+0000 7facdfaffb80 1 --2- 172.21.15.70:0/2368208710 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55fcded571a0 0x55fcdedc66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.531+0000 7facdfaffb80 4 client.15531 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-04T04:57:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.531+0000 7facdfaffb80 1 -- 172.21.15.70:0/2368208710 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "status"}) -- 0x55fcdec97450 con 0x55fcded571a0 2024-06-04T04:57:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.531+0000 7facceffd700 1 --2- 172.21.15.70:0/2368208710 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55fcded571a0 0x55fcdedc66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.532+0000 7facb67fc700 10 client.15531 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T04:57:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.532+0000 7facceffd700 1 --2- 172.21.15.70:0/2368208710 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55fcded571a0 0x55fcdedc66a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7a2692478c02a4b5 server_cookie=906bfc9101779499 in_seq=0 out_seq=0 2024-06-04T04:57:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.532+0000 7facb67fc700 1 -- 172.21.15.70:0/2368208710 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55fcdec97450 con 0x55fcded571a0 2024-06-04T04:57:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.532+0000 7facb67fc700 10 client.15531 handle_command_reply: tid=0 2024-06-04T04:57:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.532+0000 7facdfaffb80 1 client.15531 shutdown 2024-06-04T04:57:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.532+0000 7facdfaffb80 1 -- 172.21.15.70:0/2368208710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded3e210 msgr2=0x55fcded37e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.532+0000 7facdfaffb80 1 --2- 172.21.15.70:0/2368208710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded3e210 0x55fcded37e10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7facb80a9100 tx=0x7facb80106d0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.532+0000 7facdfaffb80 1 -- 172.21.15.70:0/2368208710 shutdown_connections 2024-06-04T04:57:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.532+0000 7facdfaffb80 1 --2- 172.21.15.70:0/2368208710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcded432d0 0x55fcded38350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.532+0000 7facdfaffb80 1 --2- 172.21.15.70:0/2368208710 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55fcded571a0 0x55fcdedc66a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.532+0000 7facdfaffb80 1 --2- 172.21.15.70:0/2368208710 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcded3d850 0x55fcded378d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.532+0000 7facb67fc700 0 client.15531 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T04:57:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.532+0000 7facdfaffb80 1 --2- 172.21.15.70:0/2368208710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fcded3e210 0x55fcded37e10 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.532+0000 7facdfaffb80 1 -- 172.21.15.70:0/2368208710 >> 172.21.15.70:0/2368208710 conn(0x55fcdec691f0 msgr2=0x55fcded40d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.532+0000 7facdfaffb80 1 -- 172.21.15.70:0/2368208710 shutdown_connections 2024-06-04T04:57:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.532+0000 7facdfaffb80 1 -- 172.21.15.70:0/2368208710 wait complete. 2024-06-04T04:57:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.533+0000 7facdfaffb80 20 client.15531 trim_cache size 0 max 16384 2024-06-04T04:57:53.541 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:57:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:57:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 3, 2024-06-04T04:57:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24436, 2024-06-04T04:57:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-06-04T04:57:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-06-04T04:57:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-06-04T04:57:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 241.10149931199999, 2024-06-04T04:57:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 29, 2024-06-04T04:57:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 82, 2024-06-04T04:57:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 81, 2024-06-04T04:57:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 289.248640534 2024-06-04T04:57:53.542 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:57:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.533+0000 7facd526a700 1 -- 172.21.15.70:0/1092934828 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7facbc0b2a80 msgr2=0x7facbc0b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.533+0000 7facd526a700 1 --2- 172.21.15.70:0/1092934828 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7facbc0b2a80 0x7facbc0b4f40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7facc000b5c0 tx=0x7facc001a040 comp rx=0 tx=0).stop 2024-06-04T04:57:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.533+0000 7facd526a700 1 -- 172.21.15.70:0/1092934828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd006c470 msgr2=0x7facd0102a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.533+0000 7facd526a700 1 --2- 172.21.15.70:0/1092934828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd006c470 0x7facd0102a40 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7facb800b530 tx=0x7facb800b8f0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.533+0000 7facd526a700 1 -- 172.21.15.70:0/1092934828 shutdown_connections 2024-06-04T04:57:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.533+0000 7facd526a700 1 --2- 172.21.15.70:0/1092934828 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7facbc0b2a80 0x7facbc0b4f40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.533+0000 7facd526a700 1 --2- 172.21.15.70:0/1092934828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7facd01095a0 0x7facd0102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.533+0000 7facd526a700 1 --2- 172.21.15.70:0/1092934828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd006bab0 0x7facd0102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.533+0000 7facd526a700 1 --2- 172.21.15.70:0/1092934828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd006c470 0x7facd0102a40 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.533+0000 7facd526a700 1 -- 172.21.15.70:0/1092934828 >> 172.21.15.70:0/1092934828 conn(0x7facd00793b0 msgr2=0x7facd01013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:53.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.533+0000 7facd526a700 1 -- 172.21.15.70:0/1092934828 shutdown_connections 2024-06-04T04:57:53.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:53.533+0000 7facd526a700 1 -- 172.21.15.70:0/1092934828 wait complete. 2024-06-04T04:57:54.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-06-04T04:57:54.608 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:55.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:55 smithi070 ceph-mon[32631]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-06-04T04:57:55.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:55 smithi112 ceph-mon[36453]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-06-04T04:57:55.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:55 smithi098 ceph-mon[35569]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-06-04T04:57:56.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.425+0000 7fe5fe031700 1 -- 172.21.15.70:0/1856293007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f81095b0 msgr2=0x7fe5f8109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:56.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.425+0000 7fe5fe031700 1 --2- 172.21.15.70:0/1856293007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f81095b0 0x7fe5f8109990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800b0a0 tx=0x7fe5e800b3b0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.425+0000 7fe5fe031700 1 -- 172.21.15.70:0/1856293007 shutdown_connections 2024-06-04T04:57:56.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.425+0000 7fe5fe031700 1 --2- 172.21.15.70:0/1856293007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f806bbe0 0x7fe5f81103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.425+0000 7fe5fe031700 1 --2- 172.21.15.70:0/1856293007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f81095b0 0x7fe5f8109990 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.425+0000 7fe5fe031700 1 --2- 172.21.15.70:0/1856293007 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe5f806b220 0x7fe5f806b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.425+0000 7fe5fe031700 1 -- 172.21.15.70:0/1856293007 >> 172.21.15.70:0/1856293007 conn(0x7fe5f80793f0 msgr2=0x7fe5f8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:56.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.425+0000 7fe5fe031700 1 -- 172.21.15.70:0/1856293007 shutdown_connections 2024-06-04T04:57:56.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.425+0000 7fe5fe031700 1 -- 172.21.15.70:0/1856293007 wait complete. 2024-06-04T04:57:56.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.426+0000 7fe5fe031700 1 Processor -- start 2024-06-04T04:57:56.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.426+0000 7fe5fe031700 1 -- start start 2024-06-04T04:57:56.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.426+0000 7fe5fe031700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f806b220 0x7fe5f8102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.426+0000 7fe5fe031700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f806bbe0 0x7fe5f8102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.426+0000 7fe5fe031700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe5f81095b0 0x7fe5f8102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.426+0000 7fe5fe031700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f81135f0 con 0x7fe5f806bbe0 2024-06-04T04:57:56.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.426+0000 7fe5fe031700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f8113470 con 0x7fe5f806b220 2024-06-04T04:57:56.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.426+0000 7fe5fe031700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f8113770 con 0x7fe5f81095b0 2024-06-04T04:57:56.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.426+0000 7fe5f6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f806bbe0 0x7fe5f8102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.426+0000 7fe5f6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f806bbe0 0x7fe5f8102a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50304/0 (socket says 172.21.15.70:50304) 2024-06-04T04:57:56.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.426+0000 7fe5f6ffd700 1 -- 172.21.15.70:0/662589423 learned_addr learned my addr 172.21.15.70:0/662589423 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:56.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.426+0000 7fe5f7fff700 1 --2- 172.21.15.70:0/662589423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe5f81095b0 0x7fe5f8102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.427+0000 7fe5f77fe700 1 --2- 172.21.15.70:0/662589423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f806b220 0x7fe5f8102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.427+0000 7fe5f6ffd700 1 -- 172.21.15.70:0/662589423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe5f81095b0 msgr2=0x7fe5f8102f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:56.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.427+0000 7fe5f6ffd700 1 --2- 172.21.15.70:0/662589423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe5f81095b0 0x7fe5f8102f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.427+0000 7fe5f6ffd700 1 -- 172.21.15.70:0/662589423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f806b220 msgr2=0x7fe5f8102510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:56.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.427+0000 7fe5f6ffd700 1 --2- 172.21.15.70:0/662589423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f806b220 0x7fe5f8102510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.427+0000 7fe5f6ffd700 1 -- 172.21.15.70:0/662589423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f81a80d0 con 0x7fe5f806bbe0 2024-06-04T04:57:56.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.427+0000 7fe5f7fff700 1 --2- 172.21.15.70:0/662589423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe5f81095b0 0x7fe5f8102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:56.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.427+0000 7fe5f77fe700 1 --2- 172.21.15.70:0/662589423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f806b220 0x7fe5f8102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:56.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.427+0000 7fe5f6ffd700 1 --2- 172.21.15.70:0/662589423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f806bbe0 0x7fe5f8102a50 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000d980 tx=0x7fe5e000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:56.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.427+0000 7fe5f4ff9700 1 -- 172.21.15.70:0/662589423 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0017070 con 0x7fe5f806bbe0 2024-06-04T04:57:56.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.427+0000 7fe5fe031700 1 -- 172.21.15.70:0/662589423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5f81a8360 con 0x7fe5f806bbe0 2024-06-04T04:57:56.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.427+0000 7fe5f4ff9700 1 -- 172.21.15.70:0/662589423 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5e0004830 con 0x7fe5f806bbe0 2024-06-04T04:57:56.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.427+0000 7fe5f4ff9700 1 -- 172.21.15.70:0/662589423 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00053a0 con 0x7fe5f806bbe0 2024-06-04T04:57:56.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.427+0000 7fe5fe031700 1 -- 172.21.15.70:0/662589423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5f81a88a0 con 0x7fe5f806bbe0 2024-06-04T04:57:56.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.429+0000 7fe5f4ff9700 1 -- 172.21.15.70:0/662589423 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7fe5e00049e0 con 0x7fe5f806bbe0 2024-06-04T04:57:56.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.430+0000 7fe5f4ff9700 1 --2- 172.21.15.70:0/662589423 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5e40b2a80 0x7fe5e40b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.430+0000 7fe5f77fe700 1 --2- 172.21.15.70:0/662589423 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5e40b2a80 0x7fe5e40b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.430+0000 7fe5f4ff9700 1 -- 172.21.15.70:0/662589423 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fe5e00e2760 con 0x7fe5f806bbe0 2024-06-04T04:57:56.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.430+0000 7fe5f77fe700 1 --2- 172.21.15.70:0/662589423 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5e40b2a80 0x7fe5e40b4f40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009e60 tx=0x7fe5e800b7f0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:56.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe6088c6b80 1 Processor -- start 2024-06-04T04:57:56.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe6088c6b80 1 -- start start 2024-06-04T04:57:56.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe6088c6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a220bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe6088c6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a2211c0 0x56036a2248f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe6088c6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a224f10 0x56036a225350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe6088c6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56036a17fc40 con 0x56036a220810 2024-06-04T04:57:56.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe6088c6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56036a16f7e0 con 0x56036a224f10 2024-06-04T04:57:56.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe6088c6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56036a173820 con 0x56036a2211c0 2024-06-04T04:57:56.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe5f77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a220bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe5f77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a220bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50310/0 (socket says 172.21.15.70:50310) 2024-06-04T04:57:56.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe5f77fe700 1 -- 172.21.15.70:0/1028264522 learned_addr learned my addr 172.21.15.70:0/1028264522 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:56.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe5f6ffd700 1 --2- 172.21.15.70:0/1028264522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a2211c0 0x56036a2248f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe5f7fff700 1 --2- 172.21.15.70:0/1028264522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a224f10 0x56036a225350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe5f77fe700 1 -- 172.21.15.70:0/1028264522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a2211c0 msgr2=0x56036a2248f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:56.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe5f77fe700 1 --2- 172.21.15.70:0/1028264522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a2211c0 0x56036a2248f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe5f77fe700 1 -- 172.21.15.70:0/1028264522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a224f10 msgr2=0x56036a225350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:56.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe5f77fe700 1 --2- 172.21.15.70:0/1028264522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a224f10 0x56036a225350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe5f77fe700 1 -- 172.21.15.70:0/1028264522 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56036a0bf880 con 0x56036a220810 2024-06-04T04:57:56.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.431+0000 7fe5f6ffd700 1 --2- 172.21.15.70:0/1028264522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a2211c0 0x56036a2248f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:56.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.432+0000 7fe5f77fe700 1 --2- 172.21.15.70:0/1028264522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a220bf0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8004a50 tx=0x7fe5e801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61317390e6700535 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:56.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.432+0000 7fe5edffb700 1 -- 172.21.15.70:0/1028264522 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e802c070 con 0x56036a220810 2024-06-04T04:57:56.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.432+0000 7fe5edffb700 1 -- 172.21.15.70:0/1028264522 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5e8022070 con 0x56036a220810 2024-06-04T04:57:56.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.432+0000 7fe5edffb700 1 -- 172.21.15.70:0/1028264522 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8005680 con 0x56036a220810 2024-06-04T04:57:56.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.432+0000 7fe6088c6b80 1 -- 172.21.15.70:0/1028264522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 msgr2=0x56036a220bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:56.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.432+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/1028264522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a220bf0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8004a50 tx=0x7fe5e801b000 comp rx=0 tx=0).stop 2024-06-04T04:57:56.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.433+0000 7fe6088c6b80 1 -- 172.21.15.70:0/1028264522 shutdown_connections 2024-06-04T04:57:56.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.433+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/1028264522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a224f10 0x56036a225350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.433+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/1028264522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a220bf0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.433+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/1028264522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a2211c0 0x56036a2248f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.433+0000 7fe6088c6b80 1 -- 172.21.15.70:0/1028264522 >> 172.21.15.70:0/1028264522 conn(0x56036a14e030 msgr2=0x56036a21ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:56.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.433+0000 7fe6088c6b80 1 -- 172.21.15.70:0/1028264522 shutdown_connections 2024-06-04T04:57:56.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.433+0000 7fe6088c6b80 1 -- 172.21.15.70:0/1028264522 wait complete. 2024-06-04T04:57:56.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.433+0000 7fe6088c6b80 1 -- start start 2024-06-04T04:57:56.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.433+0000 7fe6088c6b80 1 Processor -- start 2024-06-04T04:57:56.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe6088c6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a22c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe6088c6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a2211c0 0x56036a229690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe6088c6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a224f10 0x56036a229bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe6088c6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56036a075f50 con 0x56036a220810 2024-06-04T04:57:56.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe6088c6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56036a111770 con 0x56036a224f10 2024-06-04T04:57:56.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe6088c6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56036a22d070 con 0x56036a2211c0 2024-06-04T04:57:56.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe5f77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a22c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe5f77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a22c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50318/0 (socket says 172.21.15.70:50318) 2024-06-04T04:57:56.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe6088c6b80 10 client.0 _ll_register_callbacks cb 0x56036a10a8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:56.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe5f77fe700 1 -- 172.21.15.70:0/3537085824 learned_addr learned my addr 172.21.15.70:0/3537085824 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:56.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe5f6ffd700 1 --2- 172.21.15.70:0/3537085824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a2211c0 0x56036a229690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe5f7fff700 1 --2- 172.21.15.70:0/3537085824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a224f10 0x56036a229bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe5f77fe700 1 -- 172.21.15.70:0/3537085824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a2211c0 msgr2=0x56036a229690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:56.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe5f77fe700 1 --2- 172.21.15.70:0/3537085824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a2211c0 0x56036a229690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe5f77fe700 1 -- 172.21.15.70:0/3537085824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a224f10 msgr2=0x56036a229bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:56.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe5f77fe700 1 --2- 172.21.15.70:0/3537085824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a224f10 0x56036a229bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe5f77fe700 1 -- 172.21.15.70:0/3537085824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56036a10e640 con 0x56036a220810 2024-06-04T04:57:56.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe5f77fe700 1 --2- 172.21.15.70:0/3537085824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a22c390 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800b0a0 tx=0x7fe5e8005a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c77e8ea96f3570 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:56.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe5d6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:57:56.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe5d6ffd700 1 -- 172.21.15.70:0/3537085824 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e802c040 con 0x56036a220810 2024-06-04T04:57:56.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe5d6ffd700 1 -- 172.21.15.70:0/3537085824 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fe5e80078b0 con 0x56036a220810 2024-06-04T04:57:56.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe6088c6b80 1 -- 172.21.15.70:0/3537085824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5e8022070 con 0x56036a220810 2024-06-04T04:57:56.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe5d6ffd700 1 -- 172.21.15.70:0/3537085824 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe5e8007ab0 con 0x56036a220810 2024-06-04T04:57:56.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe6088c6b80 10 client.15543 fetch_fsmap learned FSMap version 29 2024-06-04T04:57:56.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.434+0000 7fe6088c6b80 1 -- 172.21.15.70:0/3537085824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56036a2377a0 con 0x56036a220810 2024-06-04T04:57:56.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.435+0000 7fe5d6ffd700 1 -- 172.21.15.70:0/3537085824 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5e8022070 con 0x56036a220810 2024-06-04T04:57:56.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.435+0000 7fe5d6ffd700 1 -- 172.21.15.70:0/3537085824 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe5e8027070 con 0x56036a220810 2024-06-04T04:57:56.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.435+0000 7fe6088c6b80 10 client.15543 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:57:56.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.435+0000 7fe6088c6b80 10 client.15543 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-04T04:57:56.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.435+0000 7fe6088c6b80 20 client.15543 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:56.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.435+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/3537085824 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56036a239940 0x56036a23adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.435+0000 7fe6088c6b80 4 client.15543 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:57:56.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.435+0000 7fe6088c6b80 1 -- 172.21.15.70:0/3537085824 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56036a1ec990 con 0x56036a239940 2024-06-04T04:57:56.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.435+0000 7fe5f6ffd700 1 --2- 172.21.15.70:0/3537085824 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56036a239940 0x56036a23adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.436+0000 7fe5f6ffd700 1 --2- 172.21.15.70:0/3537085824 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56036a239940 0x56036a23adb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e862b925b14b75d8 server_cookie=ded9948ec1ed936a in_seq=0 out_seq=0 2024-06-04T04:57:56.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.436+0000 7fe5d6ffd700 10 client.15543 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T04:57:56.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.436+0000 7fe5d6ffd700 1 -- 172.21.15.70:0/3537085824 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56036a1ec990 con 0x56036a239940 2024-06-04T04:57:56.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.436+0000 7fe5d6ffd700 10 client.15543 handle_command_reply: tid=0 2024-06-04T04:57:56.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.436+0000 7fe6088c6b80 1 client.15543 shutdown 2024-06-04T04:57:56.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.437+0000 7fe6088c6b80 1 -- 172.21.15.70:0/3537085824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 msgr2=0x56036a22c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:56.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.437+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/3537085824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a22c390 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800b0a0 tx=0x7fe5e8005a30 comp rx=0 tx=0).stop 2024-06-04T04:57:56.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.437+0000 7fe6088c6b80 1 -- 172.21.15.70:0/3537085824 shutdown_connections 2024-06-04T04:57:56.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.437+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/3537085824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a224f10 0x56036a229bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.437+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/3537085824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a22c390 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.437+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/3537085824 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56036a239940 0x56036a23adb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.437+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/3537085824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a2211c0 0x56036a229690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.437+0000 7fe5d6ffd700 0 client.15543 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T04:57:56.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.437+0000 7fe6088c6b80 1 -- 172.21.15.70:0/3537085824 >> 172.21.15.70:0/3537085824 conn(0x56036a14e030 msgr2=0x56036a21a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:56.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.437+0000 7fe6088c6b80 1 -- 172.21.15.70:0/3537085824 shutdown_connections 2024-06-04T04:57:56.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.437+0000 7fe6088c6b80 1 -- 172.21.15.70:0/3537085824 wait complete. 2024-06-04T04:57:56.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.437+0000 7fe6088c6b80 20 client.15543 trim_cache size 0 max 16384 2024-06-04T04:57:56.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe6088c6b80 1 Processor -- start 2024-06-04T04:57:56.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe6088c6b80 1 -- start start 2024-06-04T04:57:56.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe6088c6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a220810 0x56036a21fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe6088c6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a2211c0 0x56036a21ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe6088c6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a224f10 0x56036a227c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe6088c6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56036a2390f0 con 0x56036a2211c0 2024-06-04T04:57:56.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe6088c6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56036a2397c0 con 0x56036a224f10 2024-06-04T04:57:56.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe6088c6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56036a239e90 con 0x56036a220810 2024-06-04T04:57:56.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe5f6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a2211c0 0x56036a21ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe5f6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a2211c0 0x56036a21ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50334/0 (socket says 172.21.15.70:50334) 2024-06-04T04:57:56.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe5f6ffd700 1 -- 172.21.15.70:0/207296775 learned_addr learned my addr 172.21.15.70:0/207296775 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:56.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe5f6ffd700 1 -- 172.21.15.70:0/207296775 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a220810 msgr2=0x56036a21fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:56.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe5f77fe700 1 --2- 172.21.15.70:0/207296775 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a220810 0x56036a21fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe5f6ffd700 1 --2- 172.21.15.70:0/207296775 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a220810 0x56036a21fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe5f7fff700 1 --2- 172.21.15.70:0/207296775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a224f10 0x56036a227c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe5f6ffd700 1 -- 172.21.15.70:0/207296775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a224f10 msgr2=0x56036a227c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:56.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe5f6ffd700 1 --2- 172.21.15.70:0/207296775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a224f10 0x56036a227c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe5f6ffd700 1 -- 172.21.15.70:0/207296775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56036a220fb0 con 0x56036a2211c0 2024-06-04T04:57:56.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.458+0000 7fe5f7fff700 1 --2- 172.21.15.70:0/207296775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a224f10 0x56036a227c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:56.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.459+0000 7fe5f6ffd700 1 --2- 172.21.15.70:0/207296775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a2211c0 0x56036a21ac40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe5e00a95d0 tx=0x7fe5e00b2040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3daf62d59446d66f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:56.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.459+0000 7fe5ed7fa700 1 -- 172.21.15.70:0/207296775 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00adc80 con 0x56036a2211c0 2024-06-04T04:57:56.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.459+0000 7fe5ed7fa700 1 -- 172.21.15.70:0/207296775 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5e00b2730 con 0x56036a2211c0 2024-06-04T04:57:56.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.459+0000 7fe5ed7fa700 1 -- 172.21.15.70:0/207296775 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0005740 con 0x56036a2211c0 2024-06-04T04:57:56.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.459+0000 7fe6088c6b80 1 -- 172.21.15.70:0/207296775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a2211c0 msgr2=0x56036a21ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:56.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.459+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/207296775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a2211c0 0x56036a21ac40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe5e00a95d0 tx=0x7fe5e00b2040 comp rx=0 tx=0).stop 2024-06-04T04:57:56.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.459+0000 7fe6088c6b80 1 -- 172.21.15.70:0/207296775 shutdown_connections 2024-06-04T04:57:56.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.459+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/207296775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a224f10 0x56036a227c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.459+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/207296775 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a220810 0x56036a21fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.459+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/207296775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a2211c0 0x56036a21ac40 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.459+0000 7fe6088c6b80 1 -- 172.21.15.70:0/207296775 >> 172.21.15.70:0/207296775 conn(0x56036a14e030 msgr2=0x56036a21f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:56.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.459+0000 7fe6088c6b80 1 -- 172.21.15.70:0/207296775 shutdown_connections 2024-06-04T04:57:56.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.459+0000 7fe6088c6b80 1 -- 172.21.15.70:0/207296775 wait complete. 2024-06-04T04:57:56.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.460+0000 7fe6088c6b80 1 -- start start 2024-06-04T04:57:56.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.460+0000 7fe6088c6b80 1 Processor -- start 2024-06-04T04:57:56.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.460+0000 7fe6088c6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a244a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.460+0000 7fe6088c6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a2211c0 0x56036a240e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.460+0000 7fe6088c6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a224f10 0x56036a241560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.460+0000 7fe6088c6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56036a23ac30 con 0x56036a220810 2024-06-04T04:57:56.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.460+0000 7fe6088c6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56036a23a560 con 0x56036a2211c0 2024-06-04T04:57:56.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.460+0000 7fe6088c6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56036a23b300 con 0x56036a224f10 2024-06-04T04:57:56.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.460+0000 7fe6088c6b80 10 client.0 _ll_register_callbacks cb 0x56036a10a8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:56.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.460+0000 7fe5f77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a244a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.460+0000 7fe5f77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a244a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50346/0 (socket says 172.21.15.70:50346) 2024-06-04T04:57:56.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.460+0000 7fe5f77fe700 1 -- 172.21.15.70:0/1997663120 learned_addr learned my addr 172.21.15.70:0/1997663120 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:56.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.460+0000 7fe5f6ffd700 1 --2- 172.21.15.70:0/1997663120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a2211c0 0x56036a240e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe5f7fff700 1 --2- 172.21.15.70:0/1997663120 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a224f10 0x56036a241560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe5f77fe700 1 -- 172.21.15.70:0/1997663120 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a224f10 msgr2=0x56036a241560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:56.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe5f77fe700 1 --2- 172.21.15.70:0/1997663120 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a224f10 0x56036a241560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe5f77fe700 1 -- 172.21.15.70:0/1997663120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a2211c0 msgr2=0x56036a240e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:56.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe5f77fe700 1 --2- 172.21.15.70:0/1997663120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a2211c0 0x56036a240e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe5f77fe700 1 -- 172.21.15.70:0/1997663120 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56036a2a50e0 con 0x56036a220810 2024-06-04T04:57:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe5f6ffd700 1 --2- 172.21.15.70:0/1997663120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a2211c0 0x56036a240e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe5f77fe700 1 --2- 172.21.15.70:0/1997663120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a244a90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009610 tx=0x7fe5e8007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3031a91d57e5afc3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe5d6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:57:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe5d6ffd700 1 -- 172.21.15.70:0/1997663120 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8049070 con 0x56036a220810 2024-06-04T04:57:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe5d6ffd700 1 -- 172.21.15.70:0/1997663120 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fe5e8005e70 con 0x56036a220810 2024-06-04T04:57:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe6088c6b80 1 -- 172.21.15.70:0/1997663120 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5e00b2730 con 0x56036a220810 2024-06-04T04:57:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe5d6ffd700 1 -- 172.21.15.70:0/1997663120 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe5e8055050 con 0x56036a220810 2024-06-04T04:57:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe6088c6b80 10 client.15549 fetch_fsmap learned FSMap version 29 2024-06-04T04:57:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe6088c6b80 1 -- 172.21.15.70:0/1997663120 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56036a2a6260 con 0x56036a220810 2024-06-04T04:57:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe5d6ffd700 1 -- 172.21.15.70:0/1997663120 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5e00b2730 con 0x56036a220810 2024-06-04T04:57:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe5d6ffd700 1 -- 172.21.15.70:0/1997663120 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe5e8027050 con 0x56036a220810 2024-06-04T04:57:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.461+0000 7fe6088c6b80 10 client.15549 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:57:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.462+0000 7fe6088c6b80 10 client.15549 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-04T04:57:56.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.462+0000 7fe6088c6b80 20 client.15549 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:56.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.462+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/1997663120 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56036a2aad10 0x56036a2a9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:56.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.462+0000 7fe6088c6b80 4 client.15549 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-04T04:57:56.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.462+0000 7fe6088c6b80 1 -- 172.21.15.70:0/1997663120 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "status"}) -- 0x56036a181b90 con 0x56036a2aad10 2024-06-04T04:57:56.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.462+0000 7fe5f6ffd700 1 --2- 172.21.15.70:0/1997663120 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56036a2aad10 0x56036a2a9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:56.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.462+0000 7fe5d6ffd700 10 client.15549 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T04:57:56.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.462+0000 7fe5f6ffd700 1 --2- 172.21.15.70:0/1997663120 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56036a2aad10 0x56036a2a9a20 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=96604a673deade36 server_cookie=f47158dda940d566 in_seq=0 out_seq=0 2024-06-04T04:57:56.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.462+0000 7fe5d6ffd700 1 -- 172.21.15.70:0/1997663120 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56036a181b90 con 0x56036a2aad10 2024-06-04T04:57:56.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.462+0000 7fe5d6ffd700 10 client.15549 handle_command_reply: tid=0 2024-06-04T04:57:56.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.462+0000 7fe6088c6b80 1 client.15549 shutdown 2024-06-04T04:57:56.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe6088c6b80 1 -- 172.21.15.70:0/1997663120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 msgr2=0x56036a244a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:56.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/1997663120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a244a90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009610 tx=0x7fe5e8007a10 comp rx=0 tx=0).stop 2024-06-04T04:57:56.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe6088c6b80 1 -- 172.21.15.70:0/1997663120 shutdown_connections 2024-06-04T04:57:56.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/1997663120 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56036a2aad10 0x56036a2a9a20 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/1997663120 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56036a224f10 0x56036a241560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/1997663120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56036a220810 0x56036a244a90 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe6088c6b80 1 --2- 172.21.15.70:0/1997663120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56036a2211c0 0x56036a240e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe6088c6b80 1 -- 172.21.15.70:0/1997663120 >> 172.21.15.70:0/1997663120 conn(0x56036a14e030 msgr2=0x56036a21feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:56.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe5d6ffd700 0 client.15549 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T04:57:56.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe6088c6b80 1 -- 172.21.15.70:0/1997663120 shutdown_connections 2024-06-04T04:57:56.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe6088c6b80 1 -- 172.21.15.70:0/1997663120 wait complete. 2024-06-04T04:57:56.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe6088c6b80 20 client.15549 trim_cache size 0 max 16384 2024-06-04T04:57:56.472 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:57:56.472 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:57:56.473 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 2, 2024-06-04T04:57:56.473 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24440, 2024-06-04T04:57:56.473 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-06-04T04:57:56.473 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-06-04T04:57:56.473 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-06-04T04:57:56.473 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 245.04264050099999, 2024-06-04T04:57:56.473 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 29, 2024-06-04T04:57:56.473 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 82, 2024-06-04T04:57:56.473 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 81, 2024-06-04T04:57:56.473 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 272.82868717100001 2024-06-04T04:57:56.473 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:57:56.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe5fe031700 1 -- 172.21.15.70:0/662589423 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5e40b2a80 msgr2=0x7fe5e40b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:56.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe5fe031700 1 --2- 172.21.15.70:0/662589423 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5e40b2a80 0x7fe5e40b4f40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009e60 tx=0x7fe5e800b7f0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe5fe031700 1 -- 172.21.15.70:0/662589423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f806bbe0 msgr2=0x7fe5f8102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:56.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe5fe031700 1 --2- 172.21.15.70:0/662589423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f806bbe0 0x7fe5f8102a50 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000d980 tx=0x7fe5e000dd40 comp rx=0 tx=0).stop 2024-06-04T04:57:56.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe5fe031700 1 -- 172.21.15.70:0/662589423 shutdown_connections 2024-06-04T04:57:56.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe5fe031700 1 --2- 172.21.15.70:0/662589423 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5e40b2a80 0x7fe5e40b4f40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe5fe031700 1 --2- 172.21.15.70:0/662589423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe5f81095b0 0x7fe5f8102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe5fe031700 1 --2- 172.21.15.70:0/662589423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f806b220 0x7fe5f8102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.463+0000 7fe5fe031700 1 --2- 172.21.15.70:0/662589423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f806bbe0 0x7fe5f8102a50 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:56.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.464+0000 7fe5fe031700 1 -- 172.21.15.70:0/662589423 >> 172.21.15.70:0/662589423 conn(0x7fe5f80793f0 msgr2=0x7fe5f8101120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:56.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.464+0000 7fe5fe031700 1 -- 172.21.15.70:0/662589423 shutdown_connections 2024-06-04T04:57:56.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:56.464+0000 7fe5fe031700 1 -- 172.21.15.70:0/662589423 wait complete. 2024-06-04T04:57:57.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:57 smithi070 ceph-mon[32631]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.4 KiB/s wr, 15 op/s 2024-06-04T04:57:57.164 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.c status 2024-06-04T04:57:57.403 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:57:57.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:57 smithi112 ceph-mon[36453]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.4 KiB/s wr, 15 op/s 2024-06-04T04:57:57.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:57 smithi098 ceph-mon[35569]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.4 KiB/s wr, 15 op/s 2024-06-04T04:57:59.172 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:57:59 smithi070 ceph-mon[32631]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-06-04T04:57:59.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.174+0000 7f526a86d700 1 -- 172.21.15.70:0/3389162256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52641024f0 msgr2=0x7f52641103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:59.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.174+0000 7f526a86d700 1 --2- 172.21.15.70:0/3389162256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52641024f0 0x7f52641103d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f525c00b0d0 tx=0x7f525c00b3e0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.174+0000 7f526a86d700 1 -- 172.21.15.70:0/3389162256 shutdown_connections 2024-06-04T04:57:59.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.174+0000 7f526a86d700 1 --2- 172.21.15.70:0/3389162256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52641024f0 0x7f52641103d0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.174+0000 7f526a86d700 1 --2- 172.21.15.70:0/3389162256 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5264109520 0x7f5264109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.174+0000 7f526a86d700 1 --2- 172.21.15.70:0/3389162256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5264101b30 0x7f5264101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.174+0000 7f526a86d700 1 -- 172.21.15.70:0/3389162256 >> 172.21.15.70:0/3389162256 conn(0x7f5264078ab0 msgr2=0x7f52640ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.174+0000 7f526a86d700 1 -- 172.21.15.70:0/3389162256 shutdown_connections 2024-06-04T04:57:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.174+0000 7f526a86d700 1 -- 172.21.15.70:0/3389162256 wait complete. 2024-06-04T04:57:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.174+0000 7f526a86d700 1 Processor -- start 2024-06-04T04:57:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.174+0000 7f526a86d700 1 -- start start 2024-06-04T04:57:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f526a86d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5264101b30 0x7f526419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f526a86d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52641024f0 0x7f526419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f526a86d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5264109520 0x7f5264199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f526a86d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5264113610 con 0x7f52641024f0 2024-06-04T04:57:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f526a86d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5264113490 con 0x7f5264109520 2024-06-04T04:57:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f526a86d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5264113790 con 0x7f5264101b30 2024-06-04T04:57:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f52637fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52641024f0 0x7f526419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f52637fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52641024f0 0x7f526419fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50368/0 (socket says 172.21.15.70:50368) 2024-06-04T04:57:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f52637fe700 1 -- 172.21.15.70:0/1846991558 learned_addr learned my addr 172.21.15.70:0/1846991558 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f5268e0a700 1 --2- 172.21.15.70:0/1846991558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5264109520 0x7f5264199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f5263fff700 1 --2- 172.21.15.70:0/1846991558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5264101b30 0x7f526419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f52637fe700 1 -- 172.21.15.70:0/1846991558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5264101b30 msgr2=0x7f526419f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f52637fe700 1 --2- 172.21.15.70:0/1846991558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5264101b30 0x7f526419f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f52637fe700 1 -- 172.21.15.70:0/1846991558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5264109520 msgr2=0x7f5264199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f52637fe700 1 --2- 172.21.15.70:0/1846991558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5264109520 0x7f5264199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f52637fe700 1 -- 172.21.15.70:0/1846991558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5264199fd0 con 0x7f52641024f0 2024-06-04T04:57:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f5263fff700 1 --2- 172.21.15.70:0/1846991558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5264101b30 0x7f526419f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f52637fe700 1 --2- 172.21.15.70:0/1846991558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52641024f0 0x7f526419fbf0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f525800c9b0 tx=0x7f525800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f52617fa700 1 -- 172.21.15.70:0/1846991558 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5258007b00 con 0x7f52641024f0 2024-06-04T04:57:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.175+0000 7f52617fa700 1 -- 172.21.15.70:0/1846991558 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5258007ca0 con 0x7f52641024f0 2024-06-04T04:57:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.176+0000 7f526a86d700 1 -- 172.21.15.70:0/1846991558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f526419a2c0 con 0x7f52641024f0 2024-06-04T04:57:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.176+0000 7f52617fa700 1 -- 172.21.15.70:0/1846991558 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5258018660 con 0x7f52641024f0 2024-06-04T04:57:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.176+0000 7f526a86d700 1 -- 172.21.15.70:0/1846991558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52641a6580 con 0x7f52641024f0 2024-06-04T04:57:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.178+0000 7f52617fa700 1 -- 172.21.15.70:0/1846991558 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7f525800f450 con 0x7f52641024f0 2024-06-04T04:57:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.178+0000 7f52617fa700 1 --2- 172.21.15.70:0/1846991558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f524c0b2b00 0x7f524c0b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.179+0000 7f5263fff700 1 --2- 172.21.15.70:0/1846991558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f524c0b2b00 0x7f524c0b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.179+0000 7f52617fa700 1 -- 172.21.15.70:0/1846991558 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f52580e2750 con 0x7f52641024f0 2024-06-04T04:57:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.179+0000 7f5263fff700 1 --2- 172.21.15.70:0/1846991558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f524c0b2b00 0x7f524c0b4fc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5254005e10 tx=0x7f5254005da0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.179+0000 7f5275102b80 1 Processor -- start 2024-06-04T04:57:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.179+0000 7f5275102b80 1 -- start start 2024-06-04T04:57:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.179+0000 7f5275102b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08bc20 0x56385e08c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.179+0000 7f5275102b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e08c5d0 0x56385e08fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.179+0000 7f5275102b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e090320 0x56385e090760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.179+0000 7f5275102b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56385dfe6c40 con 0x56385e08bc20 2024-06-04T04:57:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.179+0000 7f5275102b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56385dfd67e0 con 0x56385e090320 2024-06-04T04:57:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.179+0000 7f5275102b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56385dfda820 con 0x56385e08c5d0 2024-06-04T04:57:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.180+0000 7f5263fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08bc20 0x56385e08c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.180+0000 7f5263fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08bc20 0x56385e08c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50378/0 (socket says 172.21.15.70:50378) 2024-06-04T04:57:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.180+0000 7f5263fff700 1 -- 172.21.15.70:0/2615738904 learned_addr learned my addr 172.21.15.70:0/2615738904 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.180+0000 7f52637fe700 1 --2- 172.21.15.70:0/2615738904 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e08c5d0 0x56385e08fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.180+0000 7f5268e0a700 1 --2- 172.21.15.70:0/2615738904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e090320 0x56385e090760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.180+0000 7f5263fff700 1 -- 172.21.15.70:0/2615738904 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e08c5d0 msgr2=0x56385e08fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.180+0000 7f5263fff700 1 --2- 172.21.15.70:0/2615738904 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e08c5d0 0x56385e08fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.180+0000 7f5263fff700 1 -- 172.21.15.70:0/2615738904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e090320 msgr2=0x56385e090760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.180+0000 7f5263fff700 1 --2- 172.21.15.70:0/2615738904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e090320 0x56385e090760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.180+0000 7f5263fff700 1 -- 172.21.15.70:0/2615738904 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56385df26880 con 0x56385e08bc20 2024-06-04T04:57:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.180+0000 7f5268e0a700 1 --2- 172.21.15.70:0/2615738904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e090320 0x56385e090760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.180+0000 7f52637fe700 1 --2- 172.21.15.70:0/2615738904 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e08c5d0 0x56385e08fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:57:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.180+0000 7f5263fff700 1 --2- 172.21.15.70:0/2615738904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08bc20 0x56385e08c000 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f525400b5d0 tx=0x7f5254012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a912d081802cb46 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.180+0000 7f524a7fc700 1 -- 172.21.15.70:0/2615738904 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254022070 con 0x56385e08bc20 2024-06-04T04:57:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.180+0000 7f524a7fc700 1 -- 172.21.15.70:0/2615738904 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5254011c60 con 0x56385e08bc20 2024-06-04T04:57:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.180+0000 7f524a7fc700 1 -- 172.21.15.70:0/2615738904 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525401d610 con 0x56385e08bc20 2024-06-04T04:57:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.181+0000 7f5275102b80 1 -- 172.21.15.70:0/2615738904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08bc20 msgr2=0x56385e08c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.181+0000 7f5275102b80 1 --2- 172.21.15.70:0/2615738904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08bc20 0x56385e08c000 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f525400b5d0 tx=0x7f5254012000 comp rx=0 tx=0).stop 2024-06-04T04:57:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.181+0000 7f5275102b80 1 -- 172.21.15.70:0/2615738904 shutdown_connections 2024-06-04T04:57:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.181+0000 7f5275102b80 1 --2- 172.21.15.70:0/2615738904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e090320 0x56385e090760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.181+0000 7f5275102b80 1 --2- 172.21.15.70:0/2615738904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08bc20 0x56385e08c000 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.181+0000 7f5275102b80 1 --2- 172.21.15.70:0/2615738904 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e08c5d0 0x56385e08fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.181+0000 7f5275102b80 1 -- 172.21.15.70:0/2615738904 >> 172.21.15.70:0/2615738904 conn(0x56385dfb5030 msgr2=0x56385e081e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.181+0000 7f5275102b80 1 -- 172.21.15.70:0/2615738904 shutdown_connections 2024-06-04T04:57:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.181+0000 7f5275102b80 1 -- 172.21.15.70:0/2615738904 wait complete. 2024-06-04T04:57:59.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.181+0000 7f5275102b80 1 -- start start 2024-06-04T04:57:59.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f5275102b80 1 Processor -- start 2024-06-04T04:57:59.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f5275102b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e08bc20 0x56385e0976d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f5275102b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08c5d0 0x56385e0949d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f5275102b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e090320 0x56385e094f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f5275102b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56385dedcf50 con 0x56385e08c5d0 2024-06-04T04:57:59.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f5275102b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56385df78770 con 0x56385e08bc20 2024-06-04T04:57:59.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f5275102b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56385e0984e0 con 0x56385e090320 2024-06-04T04:57:59.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f5275102b80 10 client.0 _ll_register_callbacks cb 0x56385df718d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:59.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f52637fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08c5d0 0x56385e0949d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f52637fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08c5d0 0x56385e0949d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50386/0 (socket says 172.21.15.70:50386) 2024-06-04T04:57:59.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f52637fe700 1 -- 172.21.15.70:0/1769080811 learned_addr learned my addr 172.21.15.70:0/1769080811 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:59.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f5268e0a700 1 --2- 172.21.15.70:0/1769080811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e090320 0x56385e094f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f5263fff700 1 --2- 172.21.15.70:0/1769080811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e08bc20 0x56385e0976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f52637fe700 1 -- 172.21.15.70:0/1769080811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e090320 msgr2=0x56385e094f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:59.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f52637fe700 1 --2- 172.21.15.70:0/1769080811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e090320 0x56385e094f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f52637fe700 1 -- 172.21.15.70:0/1769080811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e08bc20 msgr2=0x56385e0976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:59.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f52637fe700 1 --2- 172.21.15.70:0/1769080811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e08bc20 0x56385e0976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f52637fe700 1 -- 172.21.15.70:0/1769080811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56385df75640 con 0x56385e08c5d0 2024-06-04T04:57:59.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f5263fff700 1 --2- 172.21.15.70:0/1769080811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e08bc20 0x56385e0976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:59.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f52637fe700 1 --2- 172.21.15.70:0/1769080811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08c5d0 0x56385e0949d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f525800afd0 tx=0x7f52580cc350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437d8519bd65d1b2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:59.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.182+0000 7f52437fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:57:59.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.183+0000 7f52437fe700 1 -- 172.21.15.70:0/1769080811 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525800f950 con 0x56385e08c5d0 2024-06-04T04:57:59.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.183+0000 7f52437fe700 1 -- 172.21.15.70:0/1769080811 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f52580a9b20 con 0x56385e08c5d0 2024-06-04T04:57:59.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.183+0000 7f5275102b80 1 -- 172.21.15.70:0/1769080811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5254011c60 con 0x56385e08c5d0 2024-06-04T04:57:59.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.183+0000 7f52437fe700 1 -- 172.21.15.70:0/1769080811 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f52580a9fc0 con 0x56385e08c5d0 2024-06-04T04:57:59.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.183+0000 7f5275102b80 10 client.15561 fetch_fsmap learned FSMap version 29 2024-06-04T04:57:59.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.183+0000 7f5275102b80 1 -- 172.21.15.70:0/1769080811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56385e0a2ad0 con 0x56385e08c5d0 2024-06-04T04:57:59.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.183+0000 7f52437fe700 1 -- 172.21.15.70:0/1769080811 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5254011c60 con 0x56385e08c5d0 2024-06-04T04:57:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.183+0000 7f52437fe700 1 -- 172.21.15.70:0/1769080811 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f52580b2290 con 0x56385e08c5d0 2024-06-04T04:57:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.183+0000 7f5275102b80 10 client.15561 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:57:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.183+0000 7f5275102b80 10 client.15561 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-04T04:57:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.183+0000 7f5275102b80 20 client.15561 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.183+0000 7f5275102b80 1 --2- 172.21.15.70:0/1769080811 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x56385e0a5080 0x56385e0a64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.183+0000 7f5275102b80 4 client.15561 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:57:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.183+0000 7f5275102b80 1 -- 172.21.15.70:0/1769080811 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56385e053990 con 0x56385e0a5080 2024-06-04T04:57:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.183+0000 7f5263fff700 1 --2- 172.21.15.70:0/1769080811 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x56385e0a5080 0x56385e0a64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.184+0000 7f5263fff700 1 --2- 172.21.15.70:0/1769080811 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x56385e0a5080 0x56385e0a64f0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d0ea2f6b39f8dcb6 server_cookie=898858c506d8d090 in_seq=0 out_seq=0 2024-06-04T04:57:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.184+0000 7f52437fe700 10 client.15561 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T04:57:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.184+0000 7f52437fe700 1 -- 172.21.15.70:0/1769080811 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56385e053990 con 0x56385e0a5080 2024-06-04T04:57:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.184+0000 7f52437fe700 10 client.15561 handle_command_reply: tid=0 2024-06-04T04:57:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.184+0000 7f5275102b80 1 client.15561 shutdown 2024-06-04T04:57:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.185+0000 7f5275102b80 1 -- 172.21.15.70:0/1769080811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08c5d0 msgr2=0x56385e0949d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.185+0000 7f5275102b80 1 --2- 172.21.15.70:0/1769080811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08c5d0 0x56385e0949d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f525800afd0 tx=0x7f52580cc350 comp rx=0 tx=0).stop 2024-06-04T04:57:59.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.185+0000 7f5275102b80 1 -- 172.21.15.70:0/1769080811 shutdown_connections 2024-06-04T04:57:59.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.185+0000 7f5275102b80 1 --2- 172.21.15.70:0/1769080811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e090320 0x56385e094f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.185+0000 7f5275102b80 1 --2- 172.21.15.70:0/1769080811 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x56385e0a5080 0x56385e0a64f0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.185+0000 7f5275102b80 1 --2- 172.21.15.70:0/1769080811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e08bc20 0x56385e0976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.185+0000 7f5275102b80 1 --2- 172.21.15.70:0/1769080811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08c5d0 0x56385e0949d0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.185+0000 7f52437fe700 0 client.15561 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T04:57:59.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.185+0000 7f5275102b80 1 -- 172.21.15.70:0/1769080811 >> 172.21.15.70:0/1769080811 conn(0x56385dfb5030 msgr2=0x56385e0814c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:59.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.185+0000 7f5275102b80 1 -- 172.21.15.70:0/1769080811 shutdown_connections 2024-06-04T04:57:59.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.185+0000 7f5275102b80 1 -- 172.21.15.70:0/1769080811 wait complete. 2024-06-04T04:57:59.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.185+0000 7f5275102b80 20 client.15561 trim_cache size 0 max 16384 2024-06-04T04:57:59.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.206+0000 7f5275102b80 1 Processor -- start 2024-06-04T04:57:59.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.206+0000 7f5275102b80 1 -- start start 2024-06-04T04:57:59.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.206+0000 7f5275102b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e08bc20 0x56385e0955a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.206+0000 7f5275102b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e08c5d0 0x56385e081c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.206+0000 7f5275102b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e090320 0x56385e082180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.206+0000 7f5275102b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56385e0a4830 con 0x56385e090320 2024-06-04T04:57:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.206+0000 7f5275102b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56385e0a4f00 con 0x56385e08c5d0 2024-06-04T04:57:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.206+0000 7f5275102b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56385e0a55d0 con 0x56385e08bc20 2024-06-04T04:57:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.206+0000 7f5268e0a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e090320 0x56385e082180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.207+0000 7f5268e0a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e090320 0x56385e082180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50392/0 (socket says 172.21.15.70:50392) 2024-06-04T04:57:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.207+0000 7f5268e0a700 1 -- 172.21.15.70:0/3497504535 learned_addr learned my addr 172.21.15.70:0/3497504535 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.207+0000 7f52637fe700 1 --2- 172.21.15.70:0/3497504535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e08c5d0 0x56385e081c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.207+0000 7f5263fff700 1 --2- 172.21.15.70:0/3497504535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e08bc20 0x56385e0955a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.207+0000 7f5268e0a700 1 -- 172.21.15.70:0/3497504535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e08bc20 msgr2=0x56385e0955a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.207+0000 7f5268e0a700 1 --2- 172.21.15.70:0/3497504535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e08bc20 0x56385e0955a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.207+0000 7f5268e0a700 1 -- 172.21.15.70:0/3497504535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e08c5d0 msgr2=0x56385e081c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:59.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.207+0000 7f5268e0a700 1 --2- 172.21.15.70:0/3497504535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e08c5d0 0x56385e081c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.207+0000 7f5268e0a700 1 -- 172.21.15.70:0/3497504535 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56385e082cf0 con 0x56385e090320 2024-06-04T04:57:59.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.207+0000 7f52637fe700 1 --2- 172.21.15.70:0/3497504535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e08c5d0 0x56385e081c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:57:59.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.207+0000 7f5268e0a700 1 --2- 172.21.15.70:0/3497504535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e090320 0x56385e082180 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f525c009340 tx=0x7f525c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e949cc9eb30e02bf server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:59.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.207+0000 7f5249ffb700 1 -- 172.21.15.70:0/3497504535 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525c003a00 con 0x56385e090320 2024-06-04T04:57:59.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.207+0000 7f5249ffb700 1 -- 172.21.15.70:0/3497504535 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f525c003ba0 con 0x56385e090320 2024-06-04T04:57:59.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.207+0000 7f5249ffb700 1 -- 172.21.15.70:0/3497504535 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525c005310 con 0x56385e090320 2024-06-04T04:57:59.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.208+0000 7f5275102b80 1 -- 172.21.15.70:0/3497504535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e090320 msgr2=0x56385e082180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.208+0000 7f5275102b80 1 --2- 172.21.15.70:0/3497504535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e090320 0x56385e082180 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f525c009340 tx=0x7f525c00b040 comp rx=0 tx=0).stop 2024-06-04T04:57:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.208+0000 7f5275102b80 1 -- 172.21.15.70:0/3497504535 shutdown_connections 2024-06-04T04:57:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.208+0000 7f5275102b80 1 --2- 172.21.15.70:0/3497504535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e090320 0x56385e082180 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.208+0000 7f5275102b80 1 --2- 172.21.15.70:0/3497504535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e08bc20 0x56385e0955a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.208+0000 7f5275102b80 1 --2- 172.21.15.70:0/3497504535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e08c5d0 0x56385e081c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.208+0000 7f5275102b80 1 -- 172.21.15.70:0/3497504535 >> 172.21.15.70:0/3497504535 conn(0x56385dfb5030 msgr2=0x56385e093300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.208+0000 7f5275102b80 1 -- 172.21.15.70:0/3497504535 shutdown_connections 2024-06-04T04:57:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.208+0000 7f5275102b80 1 -- 172.21.15.70:0/3497504535 wait complete. 2024-06-04T04:57:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.208+0000 7f5275102b80 1 -- start start 2024-06-04T04:57:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.208+0000 7f5275102b80 1 Processor -- start 2024-06-04T04:57:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5275102b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08bc20 0x56385e0ab680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5275102b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e08c5d0 0x56385e0abbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5275102b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e090320 0x56385e0ac100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5275102b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56385e0a6370 con 0x56385e08bc20 2024-06-04T04:57:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5275102b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56385e0a5ca0 con 0x56385e08c5d0 2024-06-04T04:57:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5275102b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56385e0a6a40 con 0x56385e090320 2024-06-04T04:57:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5275102b80 10 client.0 _ll_register_callbacks cb 0x56385df718d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:57:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5263fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08bc20 0x56385e0ab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5263fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08bc20 0x56385e0ab680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50404/0 (socket says 172.21.15.70:50404) 2024-06-04T04:57:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5263fff700 1 -- 172.21.15.70:0/1579702558 learned_addr learned my addr 172.21.15.70:0/1579702558 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:57:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5268e0a700 1 --2- 172.21.15.70:0/1579702558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e090320 0x56385e0ac100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f52637fe700 1 --2- 172.21.15.70:0/1579702558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e08c5d0 0x56385e0abbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5263fff700 1 -- 172.21.15.70:0/1579702558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e090320 msgr2=0x56385e0ac100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5263fff700 1 --2- 172.21.15.70:0/1579702558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e090320 0x56385e0ac100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5263fff700 1 -- 172.21.15.70:0/1579702558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e08c5d0 msgr2=0x56385e0abbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:57:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5263fff700 1 --2- 172.21.15.70:0/1579702558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e08c5d0 0x56385e0abbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5263fff700 1 -- 172.21.15.70:0/1579702558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56385e10f860 con 0x56385e08bc20 2024-06-04T04:57:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5263fff700 1 --2- 172.21.15.70:0/1579702558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08bc20 0x56385e0ab680 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f525401ecd0 tx=0x7f525401e680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21092c473e4fc6b5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:57:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f52437fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:57:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f52437fe700 1 -- 172.21.15.70:0/1579702558 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254022040 con 0x56385e08bc20 2024-06-04T04:57:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f52437fe700 1 -- 172.21.15.70:0/1579702558 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f5254005050 con 0x56385e08bc20 2024-06-04T04:57:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.209+0000 7f5275102b80 1 -- 172.21.15.70:0/1579702558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f525c003ba0 con 0x56385e08bc20 2024-06-04T04:57:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.210+0000 7f52437fe700 1 -- 172.21.15.70:0/1579702558 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5254042050 con 0x56385e08bc20 2024-06-04T04:57:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.210+0000 7f5275102b80 10 client.15567 fetch_fsmap learned FSMap version 29 2024-06-04T04:57:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.210+0000 7f5275102b80 1 -- 172.21.15.70:0/1579702558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56385e110b40 con 0x56385e08bc20 2024-06-04T04:57:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.210+0000 7f52437fe700 1 -- 172.21.15.70:0/1579702558 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f525c003ba0 con 0x56385e08bc20 2024-06-04T04:57:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.210+0000 7f52437fe700 1 -- 172.21.15.70:0/1579702558 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f5254004830 con 0x56385e08bc20 2024-06-04T04:57:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.210+0000 7f5275102b80 10 client.15567 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:57:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.210+0000 7f5275102b80 10 client.15567 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-04T04:57:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.210+0000 7f5275102b80 20 client.15567 populate_metadata read hostname 'smithi070' 2024-06-04T04:57:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.210+0000 7f5275102b80 1 --2- 172.21.15.70:0/1579702558 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x56385e115420 0x56385e114130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:57:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.210+0000 7f5275102b80 4 client.15567 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-04T04:57:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.210+0000 7f5275102b80 1 -- 172.21.15.70:0/1579702558 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "status"}) -- 0x56385dfe8b90 con 0x56385e115420 2024-06-04T04:57:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.210+0000 7f52637fe700 1 --2- 172.21.15.70:0/1579702558 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x56385e115420 0x56385e114130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:57:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.211+0000 7f52437fe700 10 client.15567 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T04:57:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.211+0000 7f52637fe700 1 --2- 172.21.15.70:0/1579702558 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x56385e115420 0x56385e114130 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5ca2a209ffef5a82 server_cookie=ff79152b4ed93943 in_seq=0 out_seq=0 2024-06-04T04:57:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.211+0000 7f52437fe700 1 -- 172.21.15.70:0/1579702558 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56385dfe8b90 con 0x56385e115420 2024-06-04T04:57:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.211+0000 7f52437fe700 10 client.15567 handle_command_reply: tid=0 2024-06-04T04:57:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.211+0000 7f5275102b80 1 client.15567 shutdown 2024-06-04T04:57:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.211+0000 7f5275102b80 1 -- 172.21.15.70:0/1579702558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08bc20 msgr2=0x56385e0ab680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.211+0000 7f5275102b80 1 --2- 172.21.15.70:0/1579702558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08bc20 0x56385e0ab680 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f525401ecd0 tx=0x7f525401e680 comp rx=0 tx=0).stop 2024-06-04T04:57:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.211+0000 7f5275102b80 1 -- 172.21.15.70:0/1579702558 shutdown_connections 2024-06-04T04:57:59.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.211+0000 7f5275102b80 1 --2- 172.21.15.70:0/1579702558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385e090320 0x56385e0ac100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.211+0000 7f5275102b80 1 --2- 172.21.15.70:0/1579702558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385e08bc20 0x56385e0ab680 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.211+0000 7f5275102b80 1 --2- 172.21.15.70:0/1579702558 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x56385e115420 0x56385e114130 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.211+0000 7f5275102b80 1 --2- 172.21.15.70:0/1579702558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385e08c5d0 0x56385e0abbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.211+0000 7f5275102b80 1 -- 172.21.15.70:0/1579702558 >> 172.21.15.70:0/1579702558 conn(0x56385dfb5030 msgr2=0x56385e08ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:59.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.211+0000 7f52437fe700 0 client.15567 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T04:57:59.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.211+0000 7f5275102b80 1 -- 172.21.15.70:0/1579702558 shutdown_connections 2024-06-04T04:57:59.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.211+0000 7f5275102b80 1 -- 172.21.15.70:0/1579702558 wait complete. 2024-06-04T04:57:59.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.211+0000 7f5275102b80 20 client.15567 trim_cache size 0 max 16384 2024-06-04T04:57:59.219 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:57:59.219 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:57:59.220 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 1, 2024-06-04T04:57:59.220 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24442, 2024-06-04T04:57:59.220 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-06-04T04:57:59.220 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-06-04T04:57:59.220 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-06-04T04:57:59.220 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 248.79972657799999, 2024-06-04T04:57:59.220 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 29, 2024-06-04T04:57:59.220 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 82, 2024-06-04T04:57:59.220 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 81, 2024-06-04T04:57:59.220 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 292.20010275499999 2024-06-04T04:57:59.220 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:57:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.212+0000 7f526a86d700 1 -- 172.21.15.70:0/1846991558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f524c0b2b00 msgr2=0x7f524c0b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.212+0000 7f526a86d700 1 --2- 172.21.15.70:0/1846991558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f524c0b2b00 0x7f524c0b4fc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5254005e10 tx=0x7f5254005da0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.212+0000 7f526a86d700 1 -- 172.21.15.70:0/1846991558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52641024f0 msgr2=0x7f526419fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:57:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.212+0000 7f526a86d700 1 --2- 172.21.15.70:0/1846991558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52641024f0 0x7f526419fbf0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f525800c9b0 tx=0x7f525800ccc0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.212+0000 7f526a86d700 1 -- 172.21.15.70:0/1846991558 shutdown_connections 2024-06-04T04:57:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.212+0000 7f526a86d700 1 --2- 172.21.15.70:0/1846991558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f524c0b2b00 0x7f524c0b4fc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.212+0000 7f526a86d700 1 --2- 172.21.15.70:0/1846991558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5264109520 0x7f5264199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.212+0000 7f526a86d700 1 --2- 172.21.15.70:0/1846991558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5264101b30 0x7f526419f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.212+0000 7f526a86d700 1 --2- 172.21.15.70:0/1846991558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52641024f0 0x7f526419fbf0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:57:59.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.212+0000 7f526a86d700 1 -- 172.21.15.70:0/1846991558 >> 172.21.15.70:0/1846991558 conn(0x7f5264078ab0 msgr2=0x7f5264100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:57:59.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.212+0000 7f526a86d700 1 -- 172.21.15.70:0/1846991558 shutdown_connections 2024-06-04T04:57:59.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:57:59.212+0000 7f526a86d700 1 -- 172.21.15.70:0/1846991558 wait complete. 2024-06-04T04:57:59.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:57:59 smithi112 ceph-mon[36453]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-06-04T04:57:59.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:57:59 smithi098 ceph-mon[35569]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-06-04T04:57:59.609 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T04:57:59.943 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-06-04T04:57:59.984 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:01.044 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:58:00 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:58:00.684+0000 7fb9731f2700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:58:01.045 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:58:00 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:58:00.684+0000 7fb9731f2700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:58:01.045 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:58:00 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:58:00.684+0000 7fb9731f2700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:58:01.045 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:58:00 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:58:00.684+0000 7fb9731f2700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:58:01.045 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 04:58:00 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T04:58:00.684+0000 7fb9731f2700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-04T04:58:01.329 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:01 smithi070 ceph-mon[32631]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-06-04T04:58:01.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:01 smithi112 ceph-mon[36453]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-06-04T04:58:01.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:01 smithi098 ceph-mon[35569]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-06-04T04:58:01.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.675+0000 7f10aaef3700 1 -- 172.21.15.70:0/1468928145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a4102470 msgr2=0x7f10a41028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:01.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.675+0000 7f10aaef3700 1 --2- 172.21.15.70:0/1468928145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a4102470 0x7f10a41028f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1090009990 tx=0x7f1090009ca0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.676+0000 7f10aaef3700 1 -- 172.21.15.70:0/1468928145 shutdown_connections 2024-06-04T04:58:01.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.676+0000 7f10aaef3700 1 --2- 172.21.15.70:0/1468928145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10a4102e30 0x7f10a410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.676+0000 7f10aaef3700 1 --2- 172.21.15.70:0/1468928145 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10a41084b0 0x7f10a4108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.676+0000 7f10aaef3700 1 --2- 172.21.15.70:0/1468928145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a4102470 0x7f10a41028f0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.676+0000 7f10aaef3700 1 -- 172.21.15.70:0/1468928145 >> 172.21.15.70:0/1468928145 conn(0x7f10a4079410 msgr2=0x7f10a4079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:01.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.676+0000 7f10aaef3700 1 -- 172.21.15.70:0/1468928145 shutdown_connections 2024-06-04T04:58:01.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.676+0000 7f10aaef3700 1 -- 172.21.15.70:0/1468928145 wait complete. 2024-06-04T04:58:01.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.676+0000 7f10aaef3700 1 Processor -- start 2024-06-04T04:58:01.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.676+0000 7f10aaef3700 1 -- start start 2024-06-04T04:58:01.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.676+0000 7f10aaef3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10a4102470 0x7f10a419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.676+0000 7f10aaef3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10a4102e30 0x7f10a419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.676+0000 7f10aaef3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a41084b0 0x7f10a4199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.676+0000 7f10aaef3700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f10a4112470 con 0x7f10a41084b0 2024-06-04T04:58:01.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.676+0000 7f10aaef3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10a41122f0 con 0x7f10a4102e30 2024-06-04T04:58:01.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.676+0000 7f10aaef3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f10a41125f0 con 0x7f10a4102470 2024-06-04T04:58:01.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.677+0000 7f10a9490700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a41084b0 0x7f10a4199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.677+0000 7f10a9490700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a41084b0 0x7f10a4199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50442/0 (socket says 172.21.15.70:50442) 2024-06-04T04:58:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.677+0000 7f10a9490700 1 -- 172.21.15.70:0/3710903854 learned_addr learned my addr 172.21.15.70:0/3710903854 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.677+0000 7f10a3fff700 1 --2- 172.21.15.70:0/3710903854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10a4102e30 0x7f10a419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.677+0000 7f10a8c8f700 1 --2- 172.21.15.70:0/3710903854 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10a4102470 0x7f10a419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.677+0000 7f10a9490700 1 -- 172.21.15.70:0/3710903854 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10a4102470 msgr2=0x7f10a419f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.677+0000 7f10a9490700 1 --2- 172.21.15.70:0/3710903854 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10a4102470 0x7f10a419f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.677+0000 7f10a9490700 1 -- 172.21.15.70:0/3710903854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10a4102e30 msgr2=0x7f10a419fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.677+0000 7f10a9490700 1 --2- 172.21.15.70:0/3710903854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10a4102e30 0x7f10a419fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.677+0000 7f10a9490700 1 -- 172.21.15.70:0/3710903854 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10a419a0c0 con 0x7f10a41084b0 2024-06-04T04:58:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.677+0000 7f10a9490700 1 --2- 172.21.15.70:0/3710903854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a41084b0 0x7f10a4199960 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f109c00ef30 tx=0x7f109c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.677+0000 7f10a1ffb700 1 -- 172.21.15.70:0/3710903854 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109c0092e0 con 0x7f10a41084b0 2024-06-04T04:58:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.677+0000 7f10a1ffb700 1 -- 172.21.15.70:0/3710903854 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f109c004bb0 con 0x7f10a41084b0 2024-06-04T04:58:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.677+0000 7f10aaef3700 1 -- 172.21.15.70:0/3710903854 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10a419a350 con 0x7f10a41084b0 2024-06-04T04:58:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.677+0000 7f10a1ffb700 1 -- 172.21.15.70:0/3710903854 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109c004020 con 0x7f10a41084b0 2024-06-04T04:58:01.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.677+0000 7f10aaef3700 1 -- 172.21.15.70:0/3710903854 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10a41a6670 con 0x7f10a41084b0 2024-06-04T04:58:01.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.680+0000 7f10a1ffb700 1 -- 172.21.15.70:0/3710903854 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 149828+0+0 (secure 0 0 0) 0x7f109c01b030 con 0x7f10a41084b0 2024-06-04T04:58:01.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.680+0000 7f10a1ffb700 1 --2- 172.21.15.70:0/3710903854 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10940b2b00 0x7f10940b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.680+0000 7f10a8c8f700 1 --2- 172.21.15.70:0/3710903854 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10940b2b00 0x7f10940b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.681+0000 7f10a1ffb700 1 -- 172.21.15.70:0/3710903854 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f109c0e21c0 con 0x7f10a41084b0 2024-06-04T04:58:01.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.681+0000 7f10a8c8f700 1 --2- 172.21.15.70:0/3710903854 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10940b2b00 0x7f10940b4fc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1098005950 tx=0x7f109800a300 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:01.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.681+0000 7f10b5788b80 1 Processor -- start 2024-06-04T04:58:01.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.681+0000 7f10b5788b80 1 -- start start 2024-06-04T04:58:01.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.681+0000 7f10b5788b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c85c70 0x561599c86050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.681+0000 7f10b5788b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c86620 0x561599c89d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.681+0000 7f10b5788b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c8a370 0x561599c8a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.681+0000 7f10b5788b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561599be0c90 con 0x561599c85c70 2024-06-04T04:58:01.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.681+0000 7f10b5788b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561599bd0830 con 0x561599c8a370 2024-06-04T04:58:01.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.681+0000 7f10b5788b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561599bd4870 con 0x561599c86620 2024-06-04T04:58:01.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.682+0000 7f10a8c8f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c85c70 0x561599c86050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.682+0000 7f10a8c8f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c85c70 0x561599c86050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50458/0 (socket says 172.21.15.70:50458) 2024-06-04T04:58:01.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.682+0000 7f10a8c8f700 1 -- 172.21.15.70:0/1591676855 learned_addr learned my addr 172.21.15.70:0/1591676855 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:01.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.682+0000 7f10a3fff700 1 --2- 172.21.15.70:0/1591676855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c86620 0x561599c89d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.682+0000 7f10a9490700 1 --2- 172.21.15.70:0/1591676855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c8a370 0x561599c8a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.682+0000 7f10a8c8f700 1 -- 172.21.15.70:0/1591676855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c86620 msgr2=0x561599c89d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:01.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.682+0000 7f10a8c8f700 1 --2- 172.21.15.70:0/1591676855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c86620 0x561599c89d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.682+0000 7f10a8c8f700 1 -- 172.21.15.70:0/1591676855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c8a370 msgr2=0x561599c8a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:01.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.682+0000 7f10a8c8f700 1 --2- 172.21.15.70:0/1591676855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c8a370 0x561599c8a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.682+0000 7f10a8c8f700 1 -- 172.21.15.70:0/1591676855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561599b208d0 con 0x561599c85c70 2024-06-04T04:58:01.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.682+0000 7f10a3fff700 1 --2- 172.21.15.70:0/1591676855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c86620 0x561599c89d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:01.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.682+0000 7f10a9490700 1 --2- 172.21.15.70:0/1591676855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c8a370 0x561599c8a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:01.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.682+0000 7f10a8c8f700 1 --2- 172.21.15.70:0/1591676855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c85c70 0x561599c86050 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f109800fd30 tx=0x7f1098012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1447e79aa84d589d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:01.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.682+0000 7f108effd700 1 -- 172.21.15.70:0/1591676855 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1098016070 con 0x561599c85c70 2024-06-04T04:58:01.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.682+0000 7f108effd700 1 -- 172.21.15.70:0/1591676855 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1098010c70 con 0x561599c85c70 2024-06-04T04:58:01.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.682+0000 7f108effd700 1 -- 172.21.15.70:0/1591676855 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1098025d30 con 0x561599c85c70 2024-06-04T04:58:01.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.683+0000 7f10b5788b80 1 -- 172.21.15.70:0/1591676855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c85c70 msgr2=0x561599c86050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:01.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.683+0000 7f10b5788b80 1 --2- 172.21.15.70:0/1591676855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c85c70 0x561599c86050 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f109800fd30 tx=0x7f1098012000 comp rx=0 tx=0).stop 2024-06-04T04:58:01.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.683+0000 7f10b5788b80 1 -- 172.21.15.70:0/1591676855 shutdown_connections 2024-06-04T04:58:01.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.683+0000 7f10b5788b80 1 --2- 172.21.15.70:0/1591676855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c8a370 0x561599c8a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.683+0000 7f10b5788b80 1 --2- 172.21.15.70:0/1591676855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c85c70 0x561599c86050 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.683+0000 7f10b5788b80 1 --2- 172.21.15.70:0/1591676855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c86620 0x561599c89d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.683+0000 7f10b5788b80 1 -- 172.21.15.70:0/1591676855 >> 172.21.15.70:0/1591676855 conn(0x561599baf080 msgr2=0x561599c7be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:01.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.683+0000 7f10b5788b80 1 -- 172.21.15.70:0/1591676855 shutdown_connections 2024-06-04T04:58:01.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.683+0000 7f10b5788b80 1 -- 172.21.15.70:0/1591676855 wait complete. 2024-06-04T04:58:01.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.684+0000 7f10b5788b80 1 -- start start 2024-06-04T04:58:01.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.684+0000 7f10b5788b80 1 Processor -- start 2024-06-04T04:58:01.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.684+0000 7f10b5788b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c85c70 0x561599c91910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.684+0000 7f10b5788b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c86620 0x561599c8ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.684+0000 7f10b5788b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c8a370 0x561599c8f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.684+0000 7f10b5788b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561599ad6fa0 con 0x561599c86620 2024-06-04T04:58:01.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.684+0000 7f10b5788b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561599b727c0 con 0x561599c8a370 2024-06-04T04:58:01.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.684+0000 7f10b5788b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561599c92530 con 0x561599c85c70 2024-06-04T04:58:01.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.684+0000 7f10b5788b80 10 client.0 _ll_register_callbacks cb 0x561599b6b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:01.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.684+0000 7f10a3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c86620 0x561599c8ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.684+0000 7f10a3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c86620 0x561599c8ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50474/0 (socket says 172.21.15.70:50474) 2024-06-04T04:58:01.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.684+0000 7f10a3fff700 1 -- 172.21.15.70:0/2880723785 learned_addr learned my addr 172.21.15.70:0/2880723785 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:01.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.685+0000 7f10a9490700 1 --2- 172.21.15.70:0/2880723785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c8a370 0x561599c8f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.685+0000 7f10a8c8f700 1 --2- 172.21.15.70:0/2880723785 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c85c70 0x561599c91910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.685+0000 7f10a3fff700 1 -- 172.21.15.70:0/2880723785 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c85c70 msgr2=0x561599c91910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:01.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.685+0000 7f10a3fff700 1 --2- 172.21.15.70:0/2880723785 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c85c70 0x561599c91910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.685+0000 7f10a3fff700 1 -- 172.21.15.70:0/2880723785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c8a370 msgr2=0x561599c8f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:01.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.685+0000 7f10a3fff700 1 --2- 172.21.15.70:0/2880723785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c8a370 0x561599c8f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.685+0000 7f10a3fff700 1 -- 172.21.15.70:0/2880723785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561599b6f690 con 0x561599c86620 2024-06-04T04:58:01.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.685+0000 7f10a8c8f700 1 --2- 172.21.15.70:0/2880723785 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c85c70 0x561599c91910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:01.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.685+0000 7f10a9490700 1 --2- 172.21.15.70:0/2880723785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c8a370 0x561599c8f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:01.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.685+0000 7f10a3fff700 1 --2- 172.21.15.70:0/2880723785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c86620 0x561599c8ec10 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1090009990 tx=0x7f1090003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678e25092c0c0211 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:01.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.685+0000 7f1083fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:01.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.685+0000 7f1083fff700 1 -- 172.21.15.70:0/2880723785 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109000b8b0 con 0x561599c86620 2024-06-04T04:58:01.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.685+0000 7f1083fff700 1 -- 172.21.15.70:0/2880723785 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f109000ba50 con 0x561599c86620 2024-06-04T04:58:01.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.685+0000 7f10b5788b80 1 -- 172.21.15.70:0/2880723785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1098010c70 con 0x561599c86620 2024-06-04T04:58:01.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.685+0000 7f1083fff700 1 -- 172.21.15.70:0/2880723785 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f109000bc50 con 0x561599c86620 2024-06-04T04:58:01.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.686+0000 7f10b5788b80 10 client.15585 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:01.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.686+0000 7f10b5788b80 1 -- 172.21.15.70:0/2880723785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561599c9cd90 con 0x561599c86620 2024-06-04T04:58:01.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.686+0000 7f1083fff700 1 -- 172.21.15.70:0/2880723785 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1098010c70 con 0x561599c86620 2024-06-04T04:58:01.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.686+0000 7f1083fff700 1 -- 172.21.15.70:0/2880723785 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f109002d460 con 0x561599c86620 2024-06-04T04:58:01.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.686+0000 7f10b5788b80 10 client.15585 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:01.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.686+0000 7f10b5788b80 10 client.15585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:01.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.686+0000 7f10b5788b80 20 client.15585 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:01.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.686+0000 7f10b5788b80 1 --2- 172.21.15.70:0/2880723785 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561599c9f420 0x561599ca1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.686+0000 7f10b5788b80 4 client.15585 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:01.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.686+0000 7f10b5788b80 1 -- 172.21.15.70:0/2880723785 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561599c4d9e0 con 0x561599c9f420 2024-06-04T04:58:01.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.686+0000 7f10a8c8f700 1 --2- 172.21.15.70:0/2880723785 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561599c9f420 0x561599ca1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.687+0000 7f10a8c8f700 1 --2- 172.21.15.70:0/2880723785 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561599c9f420 0x561599ca1840 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffaf7e3fb65916e8 server_cookie=982941eb18aab68d in_seq=0 out_seq=0 2024-06-04T04:58:01.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.687+0000 7f1083fff700 10 client.15585 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:01.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.688+0000 7f1083fff700 1 -- 172.21.15.70:0/2880723785 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561599c4d9e0 con 0x561599c9f420 2024-06-04T04:58:01.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.688+0000 7f1083fff700 10 client.15585 handle_command_reply: tid=0 2024-06-04T04:58:01.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.688+0000 7f10b5788b80 1 client.15585 shutdown 2024-06-04T04:58:01.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.688+0000 7f10b5788b80 1 -- 172.21.15.70:0/2880723785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c86620 msgr2=0x561599c8ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:01.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.688+0000 7f10b5788b80 1 --2- 172.21.15.70:0/2880723785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c86620 0x561599c8ec10 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1090009990 tx=0x7f1090003730 comp rx=0 tx=0).stop 2024-06-04T04:58:01.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.688+0000 7f10b5788b80 1 -- 172.21.15.70:0/2880723785 shutdown_connections 2024-06-04T04:58:01.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.688+0000 7f10b5788b80 1 --2- 172.21.15.70:0/2880723785 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561599c9f420 0x561599ca1840 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.688+0000 7f10b5788b80 1 --2- 172.21.15.70:0/2880723785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c8a370 0x561599c8f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.688+0000 7f1083fff700 0 client.15585 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:01.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.688+0000 7f10b5788b80 1 --2- 172.21.15.70:0/2880723785 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c85c70 0x561599c91910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.688+0000 7f10b5788b80 1 --2- 172.21.15.70:0/2880723785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c86620 0x561599c8ec10 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.688+0000 7f10b5788b80 1 -- 172.21.15.70:0/2880723785 >> 172.21.15.70:0/2880723785 conn(0x561599baf080 msgr2=0x561599c7b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:01.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.688+0000 7f10b5788b80 1 -- 172.21.15.70:0/2880723785 shutdown_connections 2024-06-04T04:58:01.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.688+0000 7f10b5788b80 1 -- 172.21.15.70:0/2880723785 wait complete. 2024-06-04T04:58:01.697 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:01.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.688+0000 7f10b5788b80 20 client.15585 trim_cache size 0 max 16384 2024-06-04T04:58:01.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.710+0000 7f10b5788b80 1 Processor -- start 2024-06-04T04:58:01.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.710+0000 7f10b5788b80 1 -- start start 2024-06-04T04:58:01.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.710+0000 7f10b5788b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c85c70 0x561599c80ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.710+0000 7f10b5788b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c86620 0x561599c7bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.710+0000 7f10b5788b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c8a370 0x561599c7c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.710+0000 7f10b5788b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f10a41abaf0 con 0x561599c8a370 2024-06-04T04:58:01.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.710+0000 7f10b5788b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10a41ad630 con 0x561599c85c70 2024-06-04T04:58:01.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.710+0000 7f10b5788b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f10a41ac1c0 con 0x561599c86620 2024-06-04T04:58:01.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.711+0000 7f10a9490700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c8a370 0x561599c7c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.711+0000 7f10a9490700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c8a370 0x561599c7c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50490/0 (socket says 172.21.15.70:50490) 2024-06-04T04:58:01.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.711+0000 7f10a9490700 1 -- 172.21.15.70:0/338662307 learned_addr learned my addr 172.21.15.70:0/338662307 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:01.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.711+0000 7f10a3fff700 1 --2- 172.21.15.70:0/338662307 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c86620 0x561599c7bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.711+0000 7f10a8c8f700 1 --2- 172.21.15.70:0/338662307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c85c70 0x561599c80ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.711+0000 7f10a9490700 1 -- 172.21.15.70:0/338662307 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c86620 msgr2=0x561599c7bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:01.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.711+0000 7f10a9490700 1 --2- 172.21.15.70:0/338662307 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c86620 0x561599c7bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.711+0000 7f10a9490700 1 -- 172.21.15.70:0/338662307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c85c70 msgr2=0x561599c80ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:01.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.711+0000 7f10a9490700 1 --2- 172.21.15.70:0/338662307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c85c70 0x561599c80ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.711+0000 7f10a9490700 1 -- 172.21.15.70:0/338662307 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561599ca1e20 con 0x561599c8a370 2024-06-04T04:58:01.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.711+0000 7f10a9490700 1 --2- 172.21.15.70:0/338662307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c8a370 0x561599c7c1d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f109c0df8e0 tx=0x7f109c0a8de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c1fabfff62d6f27 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:01.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.711+0000 7f108e7fc700 1 -- 172.21.15.70:0/338662307 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109c0a9ac0 con 0x561599c8a370 2024-06-04T04:58:01.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.711+0000 7f108e7fc700 1 -- 172.21.15.70:0/338662307 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f109c0a93c0 con 0x561599c8a370 2024-06-04T04:58:01.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.711+0000 7f108e7fc700 1 -- 172.21.15.70:0/338662307 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109c0a9880 con 0x561599c8a370 2024-06-04T04:58:01.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.712+0000 7f10b5788b80 1 -- 172.21.15.70:0/338662307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c8a370 msgr2=0x561599c7c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:01.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.712+0000 7f10b5788b80 1 --2- 172.21.15.70:0/338662307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c8a370 0x561599c7c1d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f109c0df8e0 tx=0x7f109c0a8de0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.712+0000 7f10b5788b80 1 -- 172.21.15.70:0/338662307 shutdown_connections 2024-06-04T04:58:01.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.712+0000 7f10b5788b80 1 --2- 172.21.15.70:0/338662307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c8a370 0x561599c7c1d0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.712+0000 7f10b5788b80 1 --2- 172.21.15.70:0/338662307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c85c70 0x561599c80ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.712+0000 7f10b5788b80 1 --2- 172.21.15.70:0/338662307 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c86620 0x561599c7bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.712+0000 7f10b5788b80 1 -- 172.21.15.70:0/338662307 >> 172.21.15.70:0/338662307 conn(0x561599baf080 msgr2=0x561599c90cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:01.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.712+0000 7f10b5788b80 1 -- 172.21.15.70:0/338662307 shutdown_connections 2024-06-04T04:58:01.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.712+0000 7f10b5788b80 1 -- 172.21.15.70:0/338662307 wait complete. 2024-06-04T04:58:01.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.713+0000 7f10b5788b80 1 -- start start 2024-06-04T04:58:01.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.713+0000 7f10b5788b80 1 Processor -- start 2024-06-04T04:58:01.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.713+0000 7f10b5788b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c85c70 0x561599c95920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.713+0000 7f10b5788b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c86620 0x561599c95e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.713+0000 7f10b5788b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c8a370 0x561599d08fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.713+0000 7f10b5788b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f10a41acf60 con 0x561599c8a370 2024-06-04T04:58:01.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f10b5788b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10a41ac890 con 0x561599c86620 2024-06-04T04:58:01.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f10b5788b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f10a41a9b00 con 0x561599c85c70 2024-06-04T04:58:01.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f10a8c8f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c85c70 0x561599c95920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f10a8c8f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c85c70 0x561599c95920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:56656/0 (socket says 172.21.15.70:56656) 2024-06-04T04:58:01.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f10a8c8f700 1 -- 172.21.15.70:0/3904171765 learned_addr learned my addr 172.21.15.70:0/3904171765 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:01.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f10b5788b80 10 client.0 _ll_register_callbacks cb 0x561599b6b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:01.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f10a9490700 1 --2- 172.21.15.70:0/3904171765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c8a370 0x561599d08fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f10a3fff700 1 --2- 172.21.15.70:0/3904171765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c86620 0x561599c95e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f10a8c8f700 1 -- 172.21.15.70:0/3904171765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c86620 msgr2=0x561599c95e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:01.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f10a8c8f700 1 --2- 172.21.15.70:0/3904171765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c86620 0x561599c95e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f10a8c8f700 1 -- 172.21.15.70:0/3904171765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c8a370 msgr2=0x561599d08fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:01.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f10a8c8f700 1 --2- 172.21.15.70:0/3904171765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c8a370 0x561599d08fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f10a8c8f700 1 -- 172.21.15.70:0/3904171765 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561599d09510 con 0x561599c85c70 2024-06-04T04:58:01.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f10a3fff700 1 --2- 172.21.15.70:0/3904171765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c86620 0x561599c95e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:01.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f10a8c8f700 1 --2- 172.21.15.70:0/3904171765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c85c70 0x561599c95920 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f109800a690 tx=0x7f1098004780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c4d9be7a53b630 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:01.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f1083fff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:58:01.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f1083fff700 1 -- 172.21.15.70:0/3904171765 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1098016040 con 0x561599c85c70 2024-06-04T04:58:01.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f10b5788b80 1 -- 172.21.15.70:0/3904171765 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f109c0a93c0 con 0x561599c85c70 2024-06-04T04:58:01.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f1083fff700 1 -- 172.21.15.70:0/3904171765 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f1098026c50 con 0x561599c85c70 2024-06-04T04:58:01.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f1083fff700 1 -- 172.21.15.70:0/3904171765 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1098041050 con 0x561599c85c70 2024-06-04T04:58:01.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.714+0000 7f10b5788b80 10 client.25159 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:01.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.715+0000 7f10b5788b80 1 -- 172.21.15.70:0/3904171765 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561599d0a700 con 0x561599c85c70 2024-06-04T04:58:01.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.715+0000 7f1083fff700 1 -- 172.21.15.70:0/3904171765 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f109c0a93c0 con 0x561599c85c70 2024-06-04T04:58:01.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.715+0000 7f1083fff700 1 -- 172.21.15.70:0/3904171765 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f1098031610 con 0x561599c85c70 2024-06-04T04:58:01.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.715+0000 7f10b5788b80 10 client.25159 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:01.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.715+0000 7f10b5788b80 10 client.25159 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:01.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.715+0000 7f10b5788b80 20 client.25159 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:01.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.715+0000 7f10b5788b80 1 --2- 172.21.15.70:0/3904171765 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561599c9f420 0x561599d0cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:01.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.715+0000 7f10b5788b80 4 client.25159 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T04:58:01.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.715+0000 7f10b5788b80 1 -- 172.21.15.70:0/3904171765 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561599be2be0 con 0x561599c9f420 2024-06-04T04:58:01.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.715+0000 7f10a3fff700 1 --2- 172.21.15.70:0/3904171765 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561599c9f420 0x561599d0cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:01.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.716+0000 7f1083fff700 10 client.25159 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:01.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.716+0000 7f10a3fff700 1 --2- 172.21.15.70:0/3904171765 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561599c9f420 0x561599d0cb40 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=698a5fbd50ff1822 server_cookie=f7bbeb0c459d275d in_seq=0 out_seq=0 2024-06-04T04:58:01.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.716+0000 7f1083fff700 1 -- 172.21.15.70:0/3904171765 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561599be2be0 con 0x561599c9f420 2024-06-04T04:58:01.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.716+0000 7f1083fff700 10 client.25159 handle_command_reply: tid=0 2024-06-04T04:58:01.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.716+0000 7f10b5788b80 1 client.25159 shutdown 2024-06-04T04:58:01.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.716+0000 7f10b5788b80 1 -- 172.21.15.70:0/3904171765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c85c70 msgr2=0x561599c95920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:01.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.716+0000 7f10b5788b80 1 --2- 172.21.15.70:0/3904171765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c85c70 0x561599c95920 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f109800a690 tx=0x7f1098004780 comp rx=0 tx=0).stop 2024-06-04T04:58:01.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.716+0000 7f10b5788b80 1 -- 172.21.15.70:0/3904171765 shutdown_connections 2024-06-04T04:58:01.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.716+0000 7f10b5788b80 1 --2- 172.21.15.70:0/3904171765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561599c8a370 0x561599d08fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.716+0000 7f10b5788b80 1 --2- 172.21.15.70:0/3904171765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561599c85c70 0x561599c95920 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.716+0000 7f10b5788b80 1 --2- 172.21.15.70:0/3904171765 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561599c9f420 0x561599d0cb40 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.716+0000 7f10b5788b80 1 --2- 172.21.15.70:0/3904171765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561599c86620 0x561599c95e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.716+0000 7f1083fff700 0 client.25159 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:01.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.716+0000 7f10b5788b80 1 -- 172.21.15.70:0/3904171765 >> 172.21.15.70:0/3904171765 conn(0x561599baf080 msgr2=0x561599c88cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:01.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.716+0000 7f10b5788b80 1 -- 172.21.15.70:0/3904171765 shutdown_connections 2024-06-04T04:58:01.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.716+0000 7f10b5788b80 1 -- 172.21.15.70:0/3904171765 wait complete. 2024-06-04T04:58:01.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.716+0000 7f10b5788b80 20 client.25159 trim_cache size 0 max 16384 2024-06-04T04:58:01.724 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:58:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T04:58:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T04:58:01.724 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:58:01.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.717+0000 7f10aaef3700 1 -- 172.21.15.70:0/3710903854 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10940b2b00 msgr2=0x7f10940b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:01.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.717+0000 7f10aaef3700 1 --2- 172.21.15.70:0/3710903854 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10940b2b00 0x7f10940b4fc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1098005950 tx=0x7f109800a300 comp rx=0 tx=0).stop 2024-06-04T04:58:01.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.717+0000 7f10aaef3700 1 -- 172.21.15.70:0/3710903854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a41084b0 msgr2=0x7f10a4199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:01.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.717+0000 7f10aaef3700 1 --2- 172.21.15.70:0/3710903854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a41084b0 0x7f10a4199960 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f109c00ef30 tx=0x7f109c00c3a0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.717+0000 7f10aaef3700 1 -- 172.21.15.70:0/3710903854 shutdown_connections 2024-06-04T04:58:01.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.717+0000 7f10aaef3700 1 --2- 172.21.15.70:0/3710903854 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10940b2b00 0x7f10940b4fc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.717+0000 7f10aaef3700 1 --2- 172.21.15.70:0/3710903854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a41084b0 0x7f10a4199960 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.717+0000 7f10aaef3700 1 --2- 172.21.15.70:0/3710903854 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10a4102470 0x7f10a419f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.717+0000 7f10aaef3700 1 --2- 172.21.15.70:0/3710903854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10a4102e30 0x7f10a419fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:01.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.717+0000 7f10aaef3700 1 -- 172.21.15.70:0/3710903854 >> 172.21.15.70:0/3710903854 conn(0x7f10a4079410 msgr2=0x7f10a406c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:01.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.718+0000 7f10aaef3700 1 -- 172.21.15.70:0/3710903854 shutdown_connections 2024-06-04T04:58:01.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:01.718+0000 7f10aaef3700 1 -- 172.21.15.70:0/3710903854 wait complete. 2024-06-04T04:58:02.837 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T04:58:02.838 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:58:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.884+0000 7fc1c2237640 1 -- 172.21.15.70:0/1033068057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1bc104620 msgr2=0x7fc1bc10cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.884+0000 7fc1c2237640 1 --2- 172.21.15.70:0/1033068057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1bc104620 0x7fc1bc10cb00 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0009900 tx=0x7fc1b002a480 comp rx=0 tx=0).stop 2024-06-04T04:58:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.884+0000 7fc1c2237640 1 -- 172.21.15.70:0/1033068057 shutdown_connections 2024-06-04T04:58:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.884+0000 7fc1c2237640 1 --2- 172.21.15.70:0/1033068057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1bc10d040 0x7fc1bc10f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.884+0000 7fc1c2237640 1 --2- 172.21.15.70:0/1033068057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1bc104620 0x7fc1bc10cb00 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.884+0000 7fc1c2237640 1 --2- 172.21.15.70:0/1033068057 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1bc103d00 0x7fc1bc1040e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.884+0000 7fc1c2237640 1 -- 172.21.15.70:0/1033068057 >> 172.21.15.70:0/1033068057 conn(0x7fc1bc078ad0 msgr2=0x7fc1bc078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.884+0000 7fc1c2237640 1 -- 172.21.15.70:0/1033068057 shutdown_connections 2024-06-04T04:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.884+0000 7fc1c2237640 1 -- 172.21.15.70:0/1033068057 wait complete. 2024-06-04T04:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.884+0000 7fc1c2237640 1 Processor -- start 2024-06-04T04:58:02.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.884+0000 7fc1c2237640 1 -- start start 2024-06-04T04:58:02.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.885+0000 7fc1c2237640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1bc103d00 0x7fc1bc199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.885+0000 7fc1c2237640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1bc104620 0x7fc1bc199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.885+0000 7fc1c2237640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1bc10d040 0x7fc1bc19d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.885+0000 7fc1c2237640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc1bc112bc0 con 0x7fc1bc104620 2024-06-04T04:58:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.885+0000 7fc1c2237640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc1bc112a40 con 0x7fc1bc10d040 2024-06-04T04:58:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.885+0000 7fc1c2237640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc1bc112d40 con 0x7fc1bc103d00 2024-06-04T04:58:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.885+0000 7fc1baffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1bc104620 0x7fc1bc199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.885+0000 7fc1baffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1bc104620 0x7fc1bc199640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50508/0 (socket says 172.21.15.70:50508) 2024-06-04T04:58:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.885+0000 7fc1baffd640 1 -- 172.21.15.70:0/1556500710 learned_addr learned my addr 172.21.15.70:0/1556500710 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.885+0000 7fc1bbfff640 1 --2- 172.21.15.70:0/1556500710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1bc10d040 0x7fc1bc19d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.885+0000 7fc1bb7fe640 1 --2- 172.21.15.70:0/1556500710 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1bc103d00 0x7fc1bc199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.885+0000 7fc1baffd640 1 -- 172.21.15.70:0/1556500710 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1bc103d00 msgr2=0x7fc1bc199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.885+0000 7fc1baffd640 1 --2- 172.21.15.70:0/1556500710 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1bc103d00 0x7fc1bc199100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.885+0000 7fc1baffd640 1 -- 172.21.15.70:0/1556500710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1bc10d040 msgr2=0x7fc1bc19d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:02.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.885+0000 7fc1baffd640 1 --2- 172.21.15.70:0/1556500710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1bc10d040 0x7fc1bc19d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.885+0000 7fc1baffd640 1 -- 172.21.15.70:0/1556500710 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1bc19e0f0 con 0x7fc1bc104620 2024-06-04T04:58:02.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.885+0000 7fc1baffd640 1 --2- 172.21.15.70:0/1556500710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1bc104620 0x7fc1bc199640 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc1b002a9a0 tx=0x7fc1b0003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:02.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.886+0000 7fc1b8ff9640 1 -- 172.21.15.70:0/1556500710 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b003f070 con 0x7fc1bc104620 2024-06-04T04:58:02.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.886+0000 7fc1c2237640 1 -- 172.21.15.70:0/1556500710 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1bc19e380 con 0x7fc1bc104620 2024-06-04T04:58:02.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.886+0000 7fc1b8ff9640 1 -- 172.21.15.70:0/1556500710 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc1b0003e40 con 0x7fc1bc104620 2024-06-04T04:58:02.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.886+0000 7fc1b8ff9640 1 -- 172.21.15.70:0/1556500710 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b0035030 con 0x7fc1bc104620 2024-06-04T04:58:02.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.886+0000 7fc1c2237640 1 -- 172.21.15.70:0/1556500710 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1bc1a5c70 con 0x7fc1bc104620 2024-06-04T04:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.888+0000 7fc1b8ff9640 1 -- 172.21.15.70:0/1556500710 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 149885+0+0 (secure 0 0 0) 0x7fc1b0042070 con 0x7fc1bc104620 2024-06-04T04:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.889+0000 7fc1b8ff9640 1 --2- 172.21.15.70:0/1556500710 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc1a40b2b30 0x7fc1a40b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.889+0000 7fc1bb7fe640 1 --2- 172.21.15.70:0/1556500710 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc1a40b2b30 0x7fc1a40b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.889+0000 7fc1b8ff9640 1 -- 172.21.15.70:0/1556500710 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc1b0102f60 con 0x7fc1bc104620 2024-06-04T04:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.889+0000 7fc1c502d740 1 Processor -- start 2024-06-04T04:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.889+0000 7fc1c502d740 1 -- start start 2024-06-04T04:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.889+0000 7fc1bb7fe640 1 --2- 172.21.15.70:0/1556500710 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc1a40b2b30 0x7fc1a40b4ff0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac005d00 tx=0x7fc1ac00b410 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.889+0000 7fc1c502d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c076820 0x56080c076c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.889+0000 7fc1c502d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c077140 0x56080c0814c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.889+0000 7fc1c502d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c081a00 0x56080c083df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.889+0000 7fc1c502d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56080bfe8c30 con 0x56080c076820 2024-06-04T04:58:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.889+0000 7fc1c502d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56080bfd87d0 con 0x56080c077140 2024-06-04T04:58:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.889+0000 7fc1c502d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56080bfdc810 con 0x56080c081a00 2024-06-04T04:58:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.889+0000 7fc1bb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c076820 0x56080c076c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.889+0000 7fc1bb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c076820 0x56080c076c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50518/0 (socket says 172.21.15.70:50518) 2024-06-04T04:58:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.889+0000 7fc1bb7fe640 1 -- 172.21.15.70:0/2942156464 learned_addr learned my addr 172.21.15.70:0/2942156464 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.889+0000 7fc1bbfff640 1 --2- 172.21.15.70:0/2942156464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c081a00 0x56080c083df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.890+0000 7fc1baffd640 1 --2- 172.21.15.70:0/2942156464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c077140 0x56080c0814c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.890+0000 7fc1bb7fe640 1 -- 172.21.15.70:0/2942156464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c081a00 msgr2=0x56080c083df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.890+0000 7fc1bb7fe640 1 --2- 172.21.15.70:0/2942156464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c081a00 0x56080c083df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.890+0000 7fc1bb7fe640 1 -- 172.21.15.70:0/2942156464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c077140 msgr2=0x56080c0814c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.890+0000 7fc1bb7fe640 1 --2- 172.21.15.70:0/2942156464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c077140 0x56080c0814c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.890+0000 7fc1bb7fe640 1 -- 172.21.15.70:0/2942156464 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56080bced210 con 0x56080c076820 2024-06-04T04:58:02.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.890+0000 7fc1bb7fe640 1 --2- 172.21.15.70:0/2942156464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c076820 0x56080c076c00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac011010 tx=0x7fc1ac013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94086a41190a936 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:02.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.890+0000 7fc1a1ffb640 1 -- 172.21.15.70:0/2942156464 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac01e620 con 0x56080c076820 2024-06-04T04:58:02.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.890+0000 7fc1a1ffb640 1 -- 172.21.15.70:0/2942156464 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc1ac01ec50 con 0x56080c076820 2024-06-04T04:58:02.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.890+0000 7fc1a1ffb640 1 -- 172.21.15.70:0/2942156464 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac026b20 con 0x56080c076820 2024-06-04T04:58:02.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.891+0000 7fc1c502d740 1 -- 172.21.15.70:0/2942156464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c076820 msgr2=0x56080c076c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:02.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.891+0000 7fc1c502d740 1 --2- 172.21.15.70:0/2942156464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c076820 0x56080c076c00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac011010 tx=0x7fc1ac013000 comp rx=0 tx=0).stop 2024-06-04T04:58:02.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.891+0000 7fc1c502d740 1 -- 172.21.15.70:0/2942156464 shutdown_connections 2024-06-04T04:58:02.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.891+0000 7fc1c502d740 1 --2- 172.21.15.70:0/2942156464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c077140 0x56080c0814c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.891+0000 7fc1c502d740 1 --2- 172.21.15.70:0/2942156464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c081a00 0x56080c083df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.891+0000 7fc1c502d740 1 --2- 172.21.15.70:0/2942156464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c076820 0x56080c076c00 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.891+0000 7fc1c502d740 1 -- 172.21.15.70:0/2942156464 >> 172.21.15.70:0/2942156464 conn(0x56080c072d30 msgr2=0x56080c073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:02.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.891+0000 7fc1c502d740 1 -- 172.21.15.70:0/2942156464 shutdown_connections 2024-06-04T04:58:02.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.891+0000 7fc1c502d740 1 -- 172.21.15.70:0/2942156464 wait complete. 2024-06-04T04:58:02.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.891+0000 7fc1c502d740 1 -- start start 2024-06-04T04:58:02.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.891+0000 7fc1c502d740 1 Processor -- start 2024-06-04T04:58:02.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1c502d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c076820 0x56080c07ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1c502d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c077140 0x56080c07ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1c502d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c081a00 0x56080c07f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1c502d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56080bcbe790 con 0x56080c077140 2024-06-04T04:58:02.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1c502d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56080bf94890 con 0x56080c076820 2024-06-04T04:58:02.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1c502d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56080c084630 con 0x56080c081a00 2024-06-04T04:58:02.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1c502d740 10 client.0 _ll_register_callbacks cb 0x56080bceca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:02.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1baffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c077140 0x56080c07ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1bbfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c081a00 0x56080c07f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1baffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c077140 0x56080c07ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50530/0 (socket says 172.21.15.70:50530) 2024-06-04T04:58:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1bb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c076820 0x56080c07ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1baffd640 1 -- 172.21.15.70:0/2638328315 learned_addr learned my addr 172.21.15.70:0/2638328315 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1bbfff640 1 -- 172.21.15.70:0/2638328315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c076820 msgr2=0x56080c07ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1bbfff640 1 --2- 172.21.15.70:0/2638328315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c076820 0x56080c07ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1bbfff640 1 -- 172.21.15.70:0/2638328315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c077140 msgr2=0x56080c07ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1bbfff640 1 --2- 172.21.15.70:0/2638328315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c077140 0x56080c07ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1bbfff640 1 -- 172.21.15.70:0/2638328315 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56080bdb9620 con 0x56080c081a00 2024-06-04T04:58:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1baffd640 1 --2- 172.21.15.70:0/2638328315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c077140 0x56080c07ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc1bbfff640 1 --2- 172.21.15.70:0/2638328315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c081a00 0x56080c07f4a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x56080c073ec0 tx=0x7fc1b4013dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a1201123fc1233e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc19affd640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:58:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.892+0000 7fc19affd640 1 -- 172.21.15.70:0/2638328315 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4003970 con 0x56080c081a00 2024-06-04T04:58:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.893+0000 7fc19affd640 1 -- 172.21.15.70:0/2638328315 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc1b4003b10 con 0x56080c081a00 2024-06-04T04:58:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.893+0000 7fc1c502d740 1 -- 172.21.15.70:0/2638328315 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1ac01ec50 con 0x56080c081a00 2024-06-04T04:58:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.893+0000 7fc19affd640 1 -- 172.21.15.70:0/2638328315 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc1b4003d10 con 0x56080c081a00 2024-06-04T04:58:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.893+0000 7fc1c502d740 10 client.25171 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.893+0000 7fc1c502d740 1 -- 172.21.15.70:0/2638328315 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56080c090470 con 0x56080c081a00 2024-06-04T04:58:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.893+0000 7fc19affd640 1 -- 172.21.15.70:0/2638328315 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc1ac01ec50 con 0x56080c081a00 2024-06-04T04:58:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.893+0000 7fc19affd640 1 -- 172.21.15.70:0/2638328315 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc1b401a020 con 0x56080c081a00 2024-06-04T04:58:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.893+0000 7fc1c502d740 10 client.25171 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.893+0000 7fc1c502d740 10 client.25171 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.893+0000 7fc1c502d740 20 client.25171 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.893+0000 7fc1c502d740 1 --2- 172.21.15.70:0/2638328315 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56080c091930 0x56080c093d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.893+0000 7fc1c502d740 4 client.25171 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.893+0000 7fc1c502d740 1 -- 172.21.15.70:0/2638328315 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56080bfeab80 con 0x56080c091930 2024-06-04T04:58:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.894+0000 7fc1bb7fe640 1 --2- 172.21.15.70:0/2638328315 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56080c091930 0x56080c093d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.894+0000 7fc19affd640 10 client.25171 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.894+0000 7fc1bb7fe640 1 --2- 172.21.15.70:0/2638328315 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56080c091930 0x56080c093d20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15a201eea7d1f248 server_cookie=548c7c34542ce70 in_seq=0 out_seq=0 2024-06-04T04:58:02.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.895+0000 7fc19affd640 1 -- 172.21.15.70:0/2638328315 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56080bfeab80 con 0x56080c091930 2024-06-04T04:58:02.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.895+0000 7fc19affd640 10 client.25171 handle_command_reply: tid=0 2024-06-04T04:58:02.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.895+0000 7fc1c502d740 1 client.25171 shutdown 2024-06-04T04:58:02.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.895+0000 7fc1c502d740 1 -- 172.21.15.70:0/2638328315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c081a00 msgr2=0x56080c07f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:02.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.895+0000 7fc1c502d740 1 --2- 172.21.15.70:0/2638328315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c081a00 0x56080c07f4a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x56080c073ec0 tx=0x7fc1b4013dd0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.895+0000 7fc1c502d740 1 -- 172.21.15.70:0/2638328315 shutdown_connections 2024-06-04T04:58:02.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.895+0000 7fc1c502d740 1 --2- 172.21.15.70:0/2638328315 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56080c091930 0x56080c093d20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.895+0000 7fc1c502d740 1 --2- 172.21.15.70:0/2638328315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c081a00 0x56080c07f4a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.895+0000 7fc1c502d740 1 --2- 172.21.15.70:0/2638328315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c077140 0x56080c07ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.895+0000 7fc1c502d740 1 --2- 172.21.15.70:0/2638328315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c076820 0x56080c07ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.895+0000 7fc19affd640 0 client.25171 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:02.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.895+0000 7fc1c502d740 1 -- 172.21.15.70:0/2638328315 >> 172.21.15.70:0/2638328315 conn(0x56080c072d30 msgr2=0x56080c071fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:02.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.895+0000 7fc1c502d740 1 -- 172.21.15.70:0/2638328315 shutdown_connections 2024-06-04T04:58:02.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.895+0000 7fc1c502d740 1 -- 172.21.15.70:0/2638328315 wait complete. 2024-06-04T04:58:02.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.895+0000 7fc1c502d740 20 client.25171 trim_cache size 0 max 16384 2024-06-04T04:58:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.910+0000 7fc1c502d740 1 Processor -- start 2024-06-04T04:58:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.910+0000 7fc1c502d740 1 -- start start 2024-06-04T04:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.910+0000 7fc1c502d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c076820 0x56080c0995e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.910+0000 7fc1c502d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c077140 0x56080c099b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.910+0000 7fc1c502d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c081a00 0x56080c07e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.910+0000 7fc1c502d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc1bc1ae640 con 0x56080c076820 2024-06-04T04:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.910+0000 7fc1c502d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc1bc1b0180 con 0x56080c077140 2024-06-04T04:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.910+0000 7fc1c502d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc1bc1aed10 con 0x56080c081a00 2024-06-04T04:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.911+0000 7fc1bb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c076820 0x56080c0995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.911+0000 7fc1bb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c076820 0x56080c0995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50534/0 (socket says 172.21.15.70:50534) 2024-06-04T04:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.911+0000 7fc1bb7fe640 1 -- 172.21.15.70:0/2577400369 learned_addr learned my addr 172.21.15.70:0/2577400369 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.911+0000 7fc1bbfff640 1 --2- 172.21.15.70:0/2577400369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c081a00 0x56080c07e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.911+0000 7fc1baffd640 1 --2- 172.21.15.70:0/2577400369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c077140 0x56080c099b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.911+0000 7fc1bb7fe640 1 -- 172.21.15.70:0/2577400369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c081a00 msgr2=0x56080c07e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.911+0000 7fc1bb7fe640 1 --2- 172.21.15.70:0/2577400369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c081a00 0x56080c07e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.911+0000 7fc1bb7fe640 1 -- 172.21.15.70:0/2577400369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c077140 msgr2=0x56080c099b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.911+0000 7fc1bb7fe640 1 --2- 172.21.15.70:0/2577400369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c077140 0x56080c099b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.911+0000 7fc1bb7fe640 1 -- 172.21.15.70:0/2577400369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56080c06e6f0 con 0x56080c076820 2024-06-04T04:58:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.911+0000 7fc1bb7fe640 1 --2- 172.21.15.70:0/2577400369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c076820 0x56080c0995e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac00bcb0 tx=0x7fc1ac00bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d08da1af42b0f1a5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.911+0000 7fc1a17fa640 1 -- 172.21.15.70:0/2577400369 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac026890 con 0x56080c076820 2024-06-04T04:58:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.911+0000 7fc1a17fa640 1 -- 172.21.15.70:0/2577400369 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc1ac01e420 con 0x56080c076820 2024-06-04T04:58:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.911+0000 7fc1a17fa640 1 -- 172.21.15.70:0/2577400369 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac033650 con 0x56080c076820 2024-06-04T04:58:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.912+0000 7fc1c502d740 1 -- 172.21.15.70:0/2577400369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c076820 msgr2=0x56080c0995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.912+0000 7fc1c502d740 1 --2- 172.21.15.70:0/2577400369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c076820 0x56080c0995e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac00bcb0 tx=0x7fc1ac00bce0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.912+0000 7fc1c502d740 1 -- 172.21.15.70:0/2577400369 shutdown_connections 2024-06-04T04:58:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.912+0000 7fc1c502d740 1 --2- 172.21.15.70:0/2577400369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c077140 0x56080c099b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.912+0000 7fc1c502d740 1 --2- 172.21.15.70:0/2577400369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c081a00 0x56080c07e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.912+0000 7fc1c502d740 1 --2- 172.21.15.70:0/2577400369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c076820 0x56080c0995e0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.912+0000 7fc1c502d740 1 -- 172.21.15.70:0/2577400369 >> 172.21.15.70:0/2577400369 conn(0x56080c072d30 msgr2=0x56080c06de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.912+0000 7fc1c502d740 1 -- 172.21.15.70:0/2577400369 shutdown_connections 2024-06-04T04:58:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.912+0000 7fc1c502d740 1 -- 172.21.15.70:0/2577400369 wait complete. 2024-06-04T04:58:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.912+0000 7fc1c502d740 1 -- start start 2024-06-04T04:58:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1c502d740 1 Processor -- start 2024-06-04T04:58:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1c502d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c076820 0x56080c089f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1c502d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c077140 0x56080c07b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1c502d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c081a00 0x56080c07bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1c502d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc1bc1afab0 con 0x56080c077140 2024-06-04T04:58:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1c502d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc1bc1af3e0 con 0x56080c081a00 2024-06-04T04:58:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1c502d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc1bc1ac620 con 0x56080c076820 2024-06-04T04:58:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1c502d740 10 client.0 _ll_register_callbacks cb 0x7fc1b4009b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1baffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c077140 0x56080c07b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1baffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c077140 0x56080c07b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50548/0 (socket says 172.21.15.70:50548) 2024-06-04T04:58:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1baffd640 1 -- 172.21.15.70:0/3672649304 learned_addr learned my addr 172.21.15.70:0/3672649304 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1bb7fe640 1 --2- 172.21.15.70:0/3672649304 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c076820 0x56080c089f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1bbfff640 1 --2- 172.21.15.70:0/3672649304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c081a00 0x56080c07bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1baffd640 1 -- 172.21.15.70:0/3672649304 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c076820 msgr2=0x56080c089f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1baffd640 1 --2- 172.21.15.70:0/3672649304 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c076820 0x56080c089f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1baffd640 1 -- 172.21.15.70:0/3672649304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c081a00 msgr2=0x56080c07bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1baffd640 1 --2- 172.21.15.70:0/3672649304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c081a00 0x56080c07bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1baffd640 1 -- 172.21.15.70:0/3672649304 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56080c07c3d0 con 0x56080c077140 2024-06-04T04:58:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1bb7fe640 1 --2- 172.21.15.70:0/3672649304 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c076820 0x56080c089f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc1baffd640 1 --2- 172.21.15.70:0/3672649304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c077140 0x56080c07b660 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc1b00c9e70 tx=0x7fc1b0031f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57a2985008f2579d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.913+0000 7fc19affd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.914+0000 7fc19affd640 1 -- 172.21.15.70:0/3672649304 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b00e4060 con 0x56080c077140 2024-06-04T04:58:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.914+0000 7fc19affd640 1 -- 172.21.15.70:0/3672649304 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc1b00c6760 con 0x56080c077140 2024-06-04T04:58:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.914+0000 7fc1c502d740 1 -- 172.21.15.70:0/3672649304 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1ac01e420 con 0x56080c077140 2024-06-04T04:58:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.914+0000 7fc19affd640 1 -- 172.21.15.70:0/3672649304 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc1b00eb050 con 0x56080c077140 2024-06-04T04:58:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.914+0000 7fc1c502d740 10 client.15609 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.914+0000 7fc1c502d740 1 -- 172.21.15.70:0/3672649304 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56080c1007f0 con 0x56080c077140 2024-06-04T04:58:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.914+0000 7fc19affd640 1 -- 172.21.15.70:0/3672649304 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc1ac01e420 con 0x56080c077140 2024-06-04T04:58:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.914+0000 7fc19affd640 1 -- 172.21.15.70:0/3672649304 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc1b00d2a90 con 0x56080c077140 2024-06-04T04:58:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.914+0000 7fc1c502d740 10 client.15609 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.914+0000 7fc1c502d740 10 client.15609 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.914+0000 7fc1c502d740 20 client.15609 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.914+0000 7fc1c502d740 1 --2- 172.21.15.70:0/3672649304 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56080c100f20 0x56080c103340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.914+0000 7fc1c502d740 4 client.15609 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T04:58:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.914+0000 7fc1c502d740 1 -- 172.21.15.70:0/3672649304 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56080bfe4360 con 0x56080c100f20 2024-06-04T04:58:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.914+0000 7fc1bb7fe640 1 --2- 172.21.15.70:0/3672649304 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56080c100f20 0x56080c103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.915+0000 7fc19affd640 10 client.15609 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.915+0000 7fc1bb7fe640 1 --2- 172.21.15.70:0/3672649304 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56080c100f20 0x56080c103340 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33315e198f6cf7ae server_cookie=c1a759b41d4a6129 in_seq=0 out_seq=0 2024-06-04T04:58:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.922+0000 7fc19affd640 1 -- 172.21.15.70:0/3672649304 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56080bfe4360 con 0x56080c100f20 2024-06-04T04:58:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.922+0000 7fc19affd640 10 client.15609 handle_command_reply: tid=0 2024-06-04T04:58:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.922+0000 7fc1c502d740 1 client.15609 shutdown 2024-06-04T04:58:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.922+0000 7fc1c502d740 1 -- 172.21.15.70:0/3672649304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c077140 msgr2=0x56080c07b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.922+0000 7fc1c502d740 1 --2- 172.21.15.70:0/3672649304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c077140 0x56080c07b660 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc1b00c9e70 tx=0x7fc1b0031f70 comp rx=0 tx=0).stop 2024-06-04T04:58:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.922+0000 7fc1c502d740 1 -- 172.21.15.70:0/3672649304 shutdown_connections 2024-06-04T04:58:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.922+0000 7fc1c502d740 1 --2- 172.21.15.70:0/3672649304 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56080c100f20 0x56080c103340 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.922+0000 7fc1c502d740 1 --2- 172.21.15.70:0/3672649304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56080c081a00 0x56080c07bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.922+0000 7fc1c502d740 1 --2- 172.21.15.70:0/3672649304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56080c077140 0x56080c07b660 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.922+0000 7fc19affd640 0 client.15609 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.922+0000 7fc1c502d740 1 --2- 172.21.15.70:0/3672649304 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56080c076820 0x56080c089f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.922+0000 7fc1c502d740 1 -- 172.21.15.70:0/3672649304 >> 172.21.15.70:0/3672649304 conn(0x56080c072d30 msgr2=0x56080c078220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.922+0000 7fc1c502d740 1 -- 172.21.15.70:0/3672649304 shutdown_connections 2024-06-04T04:58:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.922+0000 7fc1c502d740 1 -- 172.21.15.70:0/3672649304 wait complete. 2024-06-04T04:58:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.922+0000 7fc1c502d740 20 client.15609 trim_cache size 0 max 16384 2024-06-04T04:58:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T04:58:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T04:58:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T04:58:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.923+0000 7fc1c2237640 1 -- 172.21.15.70:0/1556500710 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc1a40b2b30 msgr2=0x7fc1a40b4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.923+0000 7fc1c2237640 1 --2- 172.21.15.70:0/1556500710 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc1a40b2b30 0x7fc1a40b4ff0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac005d00 tx=0x7fc1ac00b410 comp rx=0 tx=0).stop 2024-06-04T04:58:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.923+0000 7fc1c2237640 1 -- 172.21.15.70:0/1556500710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1bc104620 msgr2=0x7fc1bc199640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.923+0000 7fc1c2237640 1 --2- 172.21.15.70:0/1556500710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1bc104620 0x7fc1bc199640 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc1b002a9a0 tx=0x7fc1b0003c20 comp rx=0 tx=0).stop 2024-06-04T04:58:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.923+0000 7fc1c2237640 1 -- 172.21.15.70:0/1556500710 shutdown_connections 2024-06-04T04:58:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.923+0000 7fc1c2237640 1 --2- 172.21.15.70:0/1556500710 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc1a40b2b30 0x7fc1a40b4ff0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.923+0000 7fc1c2237640 1 --2- 172.21.15.70:0/1556500710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1bc10d040 0x7fc1bc19d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.923+0000 7fc1c2237640 1 --2- 172.21.15.70:0/1556500710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1bc104620 0x7fc1bc199640 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.923+0000 7fc1c2237640 1 --2- 172.21.15.70:0/1556500710 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1bc103d00 0x7fc1bc199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.923+0000 7fc1c2237640 1 -- 172.21.15.70:0/1556500710 >> 172.21.15.70:0/1556500710 conn(0x7fc1bc078ad0 msgr2=0x7fc1bc0feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.923+0000 7fc1c2237640 1 -- 172.21.15.70:0/1556500710 shutdown_connections 2024-06-04T04:58:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:02.923+0000 7fc1c2237640 1 -- 172.21.15.70:0/1556500710 wait complete. 2024-06-04T04:58:03.093 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:03 smithi070 ceph-mon[32631]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-06-04T04:58:03.094 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:03 smithi070 ceph-mon[32631]: mgrmap e24: x(active, since 52s), standbys: z, y 2024-06-04T04:58:03.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:03 smithi112 ceph-mon[36453]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-06-04T04:58:03.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:03 smithi112 ceph-mon[36453]: mgrmap e24: x(active, since 52s), standbys: z, y 2024-06-04T04:58:03.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:03 smithi098 ceph-mon[35569]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-06-04T04:58:03.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:03 smithi098 ceph-mon[35569]: mgrmap e24: x(active, since 52s), standbys: z, y 2024-06-04T04:58:04.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.363+0000 7f7f05700700 1 -- 172.21.15.70:0/721577825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f0006bab0 msgr2=0x7f7f0006bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:04.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.363+0000 7f7f05700700 1 --2- 172.21.15.70:0/721577825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f0006bab0 0x7f7f0006bf30 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009990 tx=0x7f7ef4009ca0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.363+0000 7f7f05700700 1 -- 172.21.15.70:0/721577825 shutdown_connections 2024-06-04T04:58:04.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.363+0000 7f7f05700700 1 --2- 172.21.15.70:0/721577825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f0006c470 0x7f7f001103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.363+0000 7f7f05700700 1 --2- 172.21.15.70:0/721577825 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f001095a0 0x7f7f00109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.363+0000 7f7f05700700 1 --2- 172.21.15.70:0/721577825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f0006bab0 0x7f7f0006bf30 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.363+0000 7f7f05700700 1 -- 172.21.15.70:0/721577825 >> 172.21.15.70:0/721577825 conn(0x7f7f000793b0 msgr2=0x7f7f000797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:04.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.363+0000 7f7f05700700 1 -- 172.21.15.70:0/721577825 shutdown_connections 2024-06-04T04:58:04.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.363+0000 7f7f05700700 1 -- 172.21.15.70:0/721577825 wait complete. 2024-06-04T04:58:04.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.364+0000 7f7f05700700 1 Processor -- start 2024-06-04T04:58:04.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.364+0000 7f7f05700700 1 -- start start 2024-06-04T04:58:04.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.364+0000 7f7f05700700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f0006bab0 0x7f7f0019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.364+0000 7f7f05700700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f0006c470 0x7f7f0019fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.364+0000 7f7f05700700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f001095a0 0x7f7f00199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7f05700700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7f00113560 con 0x7f7f001095a0 2024-06-04T04:58:04.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7f05700700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001133e0 con 0x7f7f0006bab0 2024-06-04T04:58:04.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7f05700700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001136e0 con 0x7f7f0006c470 2024-06-04T04:58:04.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7eff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f001095a0 0x7f7f00199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7eff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f001095a0 0x7f7f00199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50570/0 (socket says 172.21.15.70:50570) 2024-06-04T04:58:04.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7eff7fe700 1 -- 172.21.15.70:0/3037152561 learned_addr learned my addr 172.21.15.70:0/3037152561 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:04.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7efe7fc700 1 --2- 172.21.15.70:0/3037152561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f0006c470 0x7f7f0019fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7efeffd700 1 --2- 172.21.15.70:0/3037152561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f0006bab0 0x7f7f0019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7eff7fe700 1 -- 172.21.15.70:0/3037152561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f0006c470 msgr2=0x7f7f0019fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:04.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7eff7fe700 1 --2- 172.21.15.70:0/3037152561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f0006c470 0x7f7f0019fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7eff7fe700 1 -- 172.21.15.70:0/3037152561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f0006bab0 msgr2=0x7f7f0019f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:04.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7eff7fe700 1 --2- 172.21.15.70:0/3037152561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f0006bab0 0x7f7f0019f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7eff7fe700 1 -- 172.21.15.70:0/3037152561 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f0019a0d0 con 0x7f7f001095a0 2024-06-04T04:58:04.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7efe7fc700 1 --2- 172.21.15.70:0/3037152561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f0006c470 0x7f7f0019fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:04.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7eff7fe700 1 --2- 172.21.15.70:0/3037152561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f001095a0 0x7f7f00199970 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8007ee0 tx=0x7f7ef800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:04.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7ee7fff700 1 -- 172.21.15.70:0/3037152561 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef800dd20 con 0x7f7f001095a0 2024-06-04T04:58:04.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7f05700700 1 -- 172.21.15.70:0/3037152561 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f0019a3c0 con 0x7f7f001095a0 2024-06-04T04:58:04.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7ee7fff700 1 -- 172.21.15.70:0/3037152561 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7ef8004020 con 0x7f7f001095a0 2024-06-04T04:58:04.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7ee7fff700 1 -- 172.21.15.70:0/3037152561 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8015660 con 0x7f7f001095a0 2024-06-04T04:58:04.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.365+0000 7f7f05700700 1 -- 172.21.15.70:0/3037152561 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f001a6630 con 0x7f7f001095a0 2024-06-04T04:58:04.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.368+0000 7f7ee7fff700 1 -- 172.21.15.70:0/3037152561 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 149885+0+0 (secure 0 0 0) 0x7f7ef8015800 con 0x7f7f001095a0 2024-06-04T04:58:04.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.369+0000 7f7ee7fff700 1 --2- 172.21.15.70:0/3037152561 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ee80b2b10 0x7f7ee80b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.369+0000 7f7efeffd700 1 --2- 172.21.15.70:0/3037152561 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ee80b2b10 0x7f7ee80b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.369+0000 7f7ee7fff700 1 -- 172.21.15.70:0/3037152561 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f7ef80e2260 con 0x7f7f001095a0 2024-06-04T04:58:04.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.369+0000 7f7f0ff95b80 1 Processor -- start 2024-06-04T04:58:04.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.369+0000 7f7efeffd700 1 --2- 172.21.15.70:0/3037152561 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ee80b2b10 0x7f7ee80b4fd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009680 tx=0x7f7ef0009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:04.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7f0ff95b80 1 -- start start 2024-06-04T04:58:04.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7f0ff95b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551447670 0x560551447a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7f0ff95b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551441c60 0x5605514420e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7f0ff95b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551442620 0x56055144e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7f0ff95b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56055139bd20 con 0x560551442620 2024-06-04T04:58:04.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7f0ff95b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56055138b8c0 con 0x560551447670 2024-06-04T04:58:04.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7f0ff95b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56055138f900 con 0x560551441c60 2024-06-04T04:58:04.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7eff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551442620 0x56055144e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7efeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551447670 0x560551447a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7eff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551442620 0x56055144e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50578/0 (socket says 172.21.15.70:50578) 2024-06-04T04:58:04.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7efeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551447670 0x560551447a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:42776/0 (socket says 172.21.15.70:42776) 2024-06-04T04:58:04.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7eff7fe700 1 -- 172.21.15.70:0/2171331640 learned_addr learned my addr 172.21.15.70:0/2171331640 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:04.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7efe7fc700 1 --2- 172.21.15.70:0/2171331640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551441c60 0x5605514420e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7eff7fe700 1 -- 172.21.15.70:0/2171331640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551441c60 msgr2=0x5605514420e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:04.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7eff7fe700 1 --2- 172.21.15.70:0/2171331640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551441c60 0x5605514420e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7eff7fe700 1 -- 172.21.15.70:0/2171331640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551447670 msgr2=0x560551447a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:04.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7eff7fe700 1 --2- 172.21.15.70:0/2171331640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551447670 0x560551447a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.370+0000 7f7eff7fe700 1 -- 172.21.15.70:0/2171331640 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560551334f60 con 0x560551442620 2024-06-04T04:58:04.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.371+0000 7f7eff7fe700 1 --2- 172.21.15.70:0/2171331640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551442620 0x56055144e410 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7ef80a9280 tx=0x7f7ef80a91f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d856fc8f8c65de61 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:04.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.371+0000 7f7ee57fa700 1 -- 172.21.15.70:0/2171331640 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef80e2040 con 0x560551442620 2024-06-04T04:58:04.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.371+0000 7f7ee57fa700 1 -- 172.21.15.70:0/2171331640 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7ef80ad6a0 con 0x560551442620 2024-06-04T04:58:04.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.371+0000 7f7ee57fa700 1 -- 172.21.15.70:0/2171331640 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8004bd0 con 0x560551442620 2024-06-04T04:58:04.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.372+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/2171331640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551442620 msgr2=0x56055144e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:04.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.372+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/2171331640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551442620 0x56055144e410 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7ef80a9280 tx=0x7f7ef80a91f0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.372+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/2171331640 shutdown_connections 2024-06-04T04:58:04.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.372+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/2171331640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551442620 0x56055144e410 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.372+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/2171331640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551447670 0x560551447a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.372+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/2171331640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551441c60 0x5605514420e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.372+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/2171331640 >> 172.21.15.70:0/2171331640 conn(0x5605513691f0 msgr2=0x560551438390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:04.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.372+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/2171331640 shutdown_connections 2024-06-04T04:58:04.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.372+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/2171331640 wait complete. 2024-06-04T04:58:04.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.372+0000 7f7f0ff95b80 1 -- start start 2024-06-04T04:58:04.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.372+0000 7f7f0ff95b80 1 Processor -- start 2024-06-04T04:58:04.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7f0ff95b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551441c60 0x56055144e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7f0ff95b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551442620 0x56055144b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7f0ff95b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551447670 0x56055144b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7f0ff95b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560551291f50 con 0x560551441c60 2024-06-04T04:58:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7f0ff95b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5605512f3d50 con 0x560551447670 2024-06-04T04:58:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7f0ff95b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56055144eb50 con 0x560551442620 2024-06-04T04:58:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7f0ff95b80 10 client.0 _ll_register_callbacks cb 0x560551326900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7efe7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551442620 0x56055144b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7efe7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551442620 0x56055144b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:56742/0 (socket says 172.21.15.70:56742) 2024-06-04T04:58:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7efe7fc700 1 -- 172.21.15.70:0/2507517724 learned_addr learned my addr 172.21.15.70:0/2507517724 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7efeffd700 1 --2- 172.21.15.70:0/2507517724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551441c60 0x56055144e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7eff7fe700 1 --2- 172.21.15.70:0/2507517724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551447670 0x56055144b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7efe7fc700 1 -- 172.21.15.70:0/2507517724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551447670 msgr2=0x56055144b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7efe7fc700 1 --2- 172.21.15.70:0/2507517724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551447670 0x56055144b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7efe7fc700 1 -- 172.21.15.70:0/2507517724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551441c60 msgr2=0x56055144e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7efe7fc700 1 --2- 172.21.15.70:0/2507517724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551441c60 0x56055144e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7efe7fc700 1 -- 172.21.15.70:0/2507517724 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605512db880 con 0x560551442620 2024-06-04T04:58:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7eff7fe700 1 --2- 172.21.15.70:0/2507517724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551447670 0x56055144b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7efeffd700 1 --2- 172.21.15.70:0/2507517724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551441c60 0x56055144e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.373+0000 7f7efe7fc700 1 --2- 172.21.15.70:0/2507517724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551442620 0x56055144b370 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009990 tx=0x7f7ef40050f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=942285527c3624d7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.374+0000 7f7ee2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:58:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.374+0000 7f7ee2ff5700 1 -- 172.21.15.70:0/2507517724 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4029060 con 0x560551442620 2024-06-04T04:58:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.374+0000 7f7ee2ff5700 1 -- 172.21.15.70:0/2507517724 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f7ef4003880 con 0x560551442620 2024-06-04T04:58:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.374+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/2507517724 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ef80ad6a0 con 0x560551442620 2024-06-04T04:58:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.374+0000 7f7ee2ff5700 1 -- 172.21.15.70:0/2507517724 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7ef4003a80 con 0x560551442620 2024-06-04T04:58:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.374+0000 7f7f0ff95b80 10 client.25189 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.374+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/2507517724 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560551345710 con 0x560551442620 2024-06-04T04:58:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.374+0000 7f7ee2ff5700 1 -- 172.21.15.70:0/2507517724 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7ef80ad6a0 con 0x560551442620 2024-06-04T04:58:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.374+0000 7f7ee2ff5700 1 -- 172.21.15.70:0/2507517724 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f7ef400f460 con 0x560551442620 2024-06-04T04:58:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.374+0000 7f7f0ff95b80 10 client.25189 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.374+0000 7f7f0ff95b80 10 client.25189 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-04T04:58:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.374+0000 7f7f0ff95b80 20 client.25189 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.374+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/2507517724 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56055145bb80 0x56055145cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.374+0000 7f7f0ff95b80 4 client.25189 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.374+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/2507517724 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56055139dc70 con 0x56055145bb80 2024-06-04T04:58:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.375+0000 7f7efeffd700 1 --2- 172.21.15.70:0/2507517724 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56055145bb80 0x56055145cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.375+0000 7f7ee2ff5700 10 client.25189 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:04.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.375+0000 7f7efeffd700 1 --2- 172.21.15.70:0/2507517724 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56055145bb80 0x56055145cff0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a880535c5fd872 server_cookie=acb5bfdaedb944a7 in_seq=0 out_seq=0 2024-06-04T04:58:04.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.376+0000 7f7ee2ff5700 1 -- 172.21.15.70:0/2507517724 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56055139dc70 con 0x56055145bb80 2024-06-04T04:58:04.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.376+0000 7f7ee2ff5700 10 client.25189 handle_command_reply: tid=0 2024-06-04T04:58:04.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.376+0000 7f7f0ff95b80 1 client.25189 shutdown 2024-06-04T04:58:04.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.376+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/2507517724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551442620 msgr2=0x56055144b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:04.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.376+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/2507517724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551442620 0x56055144b370 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009990 tx=0x7f7ef40050f0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.376+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/2507517724 shutdown_connections 2024-06-04T04:58:04.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.376+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/2507517724 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56055145bb80 0x56055145cff0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.376+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/2507517724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551447670 0x56055144b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.376+0000 7f7ee2ff5700 0 client.25189 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:04.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.376+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/2507517724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551441c60 0x56055144e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.376+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/2507517724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551442620 0x56055144b370 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.376+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/2507517724 >> 172.21.15.70:0/2507517724 conn(0x5605513691f0 msgr2=0x560551437aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:04.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.376+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/2507517724 shutdown_connections 2024-06-04T04:58:04.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.376+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/2507517724 wait complete. 2024-06-04T04:58:04.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.376+0000 7f7f0ff95b80 20 client.25189 trim_cache size 0 max 16384 2024-06-04T04:58:04.392 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:04.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.397+0000 7f7f0ff95b80 1 Processor -- start 2024-06-04T04:58:04.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.397+0000 7f7f0ff95b80 1 -- start start 2024-06-04T04:58:04.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.398+0000 7f7f0ff95b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551441c60 0x56055145b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.398+0000 7f7f0ff95b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551442620 0x56055145ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.398+0000 7f7f0ff95b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551447670 0x56055143c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.398+0000 7f7f0ff95b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56055145b330 con 0x560551447670 2024-06-04T04:58:04.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.398+0000 7f7f0ff95b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56055145ba00 con 0x560551442620 2024-06-04T04:58:04.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.398+0000 7f7f0ff95b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56055145c0d0 con 0x560551441c60 2024-06-04T04:58:04.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.398+0000 7f7eff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551447670 0x56055143c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.398+0000 7f7eff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551447670 0x56055143c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50602/0 (socket says 172.21.15.70:50602) 2024-06-04T04:58:04.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.398+0000 7f7eff7fe700 1 -- 172.21.15.70:0/3191075747 learned_addr learned my addr 172.21.15.70:0/3191075747 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:04.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.398+0000 7f7efeffd700 1 --2- 172.21.15.70:0/3191075747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551441c60 0x56055145b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.398+0000 7f7efe7fc700 1 --2- 172.21.15.70:0/3191075747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551442620 0x56055145ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.398+0000 7f7efeffd700 1 -- 172.21.15.70:0/3191075747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551442620 msgr2=0x56055145ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:04.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.398+0000 7f7efeffd700 1 --2- 172.21.15.70:0/3191075747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551442620 0x56055145ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.398+0000 7f7efeffd700 1 -- 172.21.15.70:0/3191075747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551447670 msgr2=0x56055143c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:04.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.398+0000 7f7efeffd700 1 --2- 172.21.15.70:0/3191075747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551447670 0x56055143c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.398+0000 7f7efeffd700 1 -- 172.21.15.70:0/3191075747 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560551439c40 con 0x560551441c60 2024-06-04T04:58:04.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.398+0000 7f7eff7fe700 1 --2- 172.21.15.70:0/3191075747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551447670 0x56055143c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:04.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.399+0000 7f7efeffd700 1 --2- 172.21.15.70:0/3191075747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551441c60 0x56055145b4b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x56055145db00 tx=0x7f7ef0014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20f7ce7f564d52d3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:04.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.399+0000 7f7ee4ff9700 1 -- 172.21.15.70:0/3191075747 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef001d310 con 0x560551441c60 2024-06-04T04:58:04.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.399+0000 7f7ee4ff9700 1 -- 172.21.15.70:0/3191075747 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7ef00044e0 con 0x560551441c60 2024-06-04T04:58:04.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.399+0000 7f7ee4ff9700 1 -- 172.21.15.70:0/3191075747 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef001bdb0 con 0x560551441c60 2024-06-04T04:58:04.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.399+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/3191075747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551441c60 msgr2=0x56055145b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:04.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.399+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/3191075747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551441c60 0x56055145b4b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x56055145db00 tx=0x7f7ef0014000 comp rx=0 tx=0).stop 2024-06-04T04:58:04.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/3191075747 shutdown_connections 2024-06-04T04:58:04.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/3191075747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551447670 0x56055143c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/3191075747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551441c60 0x56055145b4b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/3191075747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551442620 0x56055145ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/3191075747 >> 172.21.15.70:0/3191075747 conn(0x5605513691f0 msgr2=0x56055145a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:04.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/3191075747 shutdown_connections 2024-06-04T04:58:04.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/3191075747 wait complete. 2024-06-04T04:58:04.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7f0ff95b80 1 -- start start 2024-06-04T04:58:04.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7f0ff95b80 1 Processor -- start 2024-06-04T04:58:04.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7f0ff95b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551441c60 0x560551466a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7f0ff95b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551442620 0x560551437410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7f0ff95b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551447670 0x560551437950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7f0ff95b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56055145ce70 con 0x560551442620 2024-06-04T04:58:04.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7f0ff95b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56055145c7a0 con 0x560551441c60 2024-06-04T04:58:04.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7f0ff95b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56055145d540 con 0x560551447670 2024-06-04T04:58:04.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7f0ff95b80 10 client.0 _ll_register_callbacks cb 0x560551326900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:04.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7efe7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551442620 0x560551437410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7efe7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551442620 0x560551437410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50612/0 (socket says 172.21.15.70:50612) 2024-06-04T04:58:04.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7eff7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551447670 0x560551437950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7efe7fc700 1 -- 172.21.15.70:0/1918620893 learned_addr learned my addr 172.21.15.70:0/1918620893 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:04.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.400+0000 7f7efeffd700 1 --2- 172.21.15.70:0/1918620893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551441c60 0x560551466a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.401+0000 7f7eff7fe700 1 -- 172.21.15.70:0/1918620893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551441c60 msgr2=0x560551466a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:04.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.401+0000 7f7eff7fe700 1 --2- 172.21.15.70:0/1918620893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551441c60 0x560551466a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.401+0000 7f7eff7fe700 1 -- 172.21.15.70:0/1918620893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551442620 msgr2=0x560551437410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:04.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.401+0000 7f7eff7fe700 1 --2- 172.21.15.70:0/1918620893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551442620 0x560551437410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.401+0000 7f7eff7fe700 1 -- 172.21.15.70:0/1918620893 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560551438060 con 0x560551447670 2024-06-04T04:58:04.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.401+0000 7f7efe7fc700 1 --2- 172.21.15.70:0/1918620893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551442620 0x560551437410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:04.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.401+0000 7f7eff7fe700 1 --2- 172.21.15.70:0/1918620893 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551447670 0x560551437950 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8015000 tx=0x7f7ef80a9780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e0e551265af6c1b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:04.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.401+0000 7f7ee2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:58:04.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.401+0000 7f7ee2ff5700 1 -- 172.21.15.70:0/1918620893 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef80c2070 con 0x560551447670 2024-06-04T04:58:04.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.401+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/1918620893 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ef00044e0 con 0x560551447670 2024-06-04T04:58:04.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.401+0000 7f7ee2ff5700 1 -- 172.21.15.70:0/1918620893 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f7ef80b1cc0 con 0x560551447670 2024-06-04T04:58:04.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.401+0000 7f7ee2ff5700 1 -- 172.21.15.70:0/1918620893 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7ef80e2040 con 0x560551447670 2024-06-04T04:58:04.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.401+0000 7f7f0ff95b80 10 client.25195 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:04.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.401+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/1918620893 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5605514c7310 con 0x560551447670 2024-06-04T04:58:04.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.402+0000 7f7ee2ff5700 1 -- 172.21.15.70:0/1918620893 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7ef00044e0 con 0x560551447670 2024-06-04T04:58:04.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.402+0000 7f7ee2ff5700 1 -- 172.21.15.70:0/1918620893 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f7ef80c3be0 con 0x560551447670 2024-06-04T04:58:04.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.402+0000 7f7f0ff95b80 10 client.25195 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:04.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.402+0000 7f7f0ff95b80 10 client.25195 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-04T04:58:04.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.402+0000 7f7f0ff95b80 20 client.25195 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:04.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.402+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/1918620893 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5605514cb930 0x5605514ca640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:04.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.402+0000 7f7f0ff95b80 4 client.25195 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-04T04:58:04.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.402+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/1918620893 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "status"}) -- 0x560551397450 con 0x5605514cb930 2024-06-04T04:58:04.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.402+0000 7f7efeffd700 1 --2- 172.21.15.70:0/1918620893 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5605514cb930 0x5605514ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:04.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.403+0000 7f7ee2ff5700 10 client.25195 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:04.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.403+0000 7f7efeffd700 1 --2- 172.21.15.70:0/1918620893 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5605514cb930 0x5605514ca640 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b02dfa50747db1e server_cookie=cde29cf9f9d8b1f1 in_seq=0 out_seq=0 2024-06-04T04:58:04.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.403+0000 7f7ee2ff5700 1 -- 172.21.15.70:0/1918620893 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x560551397450 con 0x5605514cb930 2024-06-04T04:58:04.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.403+0000 7f7ee2ff5700 10 client.25195 handle_command_reply: tid=0 2024-06-04T04:58:04.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.403+0000 7f7f0ff95b80 1 client.25195 shutdown 2024-06-04T04:58:04.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.403+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/1918620893 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551447670 msgr2=0x560551437950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:04.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.403+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/1918620893 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551447670 0x560551437950 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8015000 tx=0x7f7ef80a9780 comp rx=0 tx=0).stop 2024-06-04T04:58:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.403+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/1918620893 shutdown_connections 2024-06-04T04:58:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.403+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/1918620893 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560551447670 0x560551437950 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.403+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/1918620893 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5605514cb930 0x5605514ca640 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.403+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/1918620893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560551441c60 0x560551466a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.403+0000 7f7f0ff95b80 1 --2- 172.21.15.70:0/1918620893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560551442620 0x560551437410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.403+0000 7f7ee2ff5700 0 client.25195 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.403+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/1918620893 >> 172.21.15.70:0/1918620893 conn(0x5605513691f0 msgr2=0x56055143c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.403+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/1918620893 shutdown_connections 2024-06-04T04:58:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.403+0000 7f7f0ff95b80 1 -- 172.21.15.70:0/1918620893 wait complete. 2024-06-04T04:58:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.403+0000 7f7f0ff95b80 20 client.25195 trim_cache size 0 max 16384 2024-06-04T04:58:04.411 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:58:04.411 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "164dbaba-222d-11ef-bc9b-c7b262605968", 2024-06-04T04:58:04.411 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-06-04T04:58:04.411 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24448, 2024-06-04T04:58:04.411 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-06-04T04:58:04.411 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-06-04T04:58:04.411 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-06-04T04:58:04.411 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 266.03575806800001, 2024-06-04T04:58:04.411 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 29, 2024-06-04T04:58:04.411 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 82, 2024-06-04T04:58:04.411 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 81, 2024-06-04T04:58:04.411 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 291.61299467200001 2024-06-04T04:58:04.411 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:58:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.404+0000 7f7f05700700 1 -- 172.21.15.70:0/3037152561 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ee80b2b10 msgr2=0x7f7ee80b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.404+0000 7f7f05700700 1 --2- 172.21.15.70:0/3037152561 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ee80b2b10 0x7f7ee80b4fd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009680 tx=0x7f7ef0009260 comp rx=0 tx=0).stop 2024-06-04T04:58:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.404+0000 7f7f05700700 1 -- 172.21.15.70:0/3037152561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f001095a0 msgr2=0x7f7f00199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.404+0000 7f7f05700700 1 --2- 172.21.15.70:0/3037152561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f001095a0 0x7f7f00199970 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8007ee0 tx=0x7f7ef800d3b0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.404+0000 7f7f05700700 1 -- 172.21.15.70:0/3037152561 shutdown_connections 2024-06-04T04:58:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.404+0000 7f7f05700700 1 --2- 172.21.15.70:0/3037152561 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ee80b2b10 0x7f7ee80b4fd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.404+0000 7f7f05700700 1 --2- 172.21.15.70:0/3037152561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f001095a0 0x7f7f00199970 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.404+0000 7f7f05700700 1 --2- 172.21.15.70:0/3037152561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f0006bab0 0x7f7f0019f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.404+0000 7f7f05700700 1 --2- 172.21.15.70:0/3037152561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f0006c470 0x7f7f0019fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.404+0000 7f7f05700700 1 -- 172.21.15.70:0/3037152561 >> 172.21.15.70:0/3037152561 conn(0x7f7f000793b0 msgr2=0x7f7f00101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.404+0000 7f7f05700700 1 -- 172.21.15.70:0/3037152561 shutdown_connections 2024-06-04T04:58:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:04.404+0000 7f7f05700700 1 -- 172.21.15.70:0/3037152561 wait complete. 2024-06-04T04:58:05.120 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-06-04T04:58:05.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:05 smithi070 ceph-mon[32631]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-06-04T04:58:05.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:05 smithi112 ceph-mon[36453]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-06-04T04:58:05.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:05 smithi098 ceph-mon[35569]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-06-04T04:58:06.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.769+0000 7f22db16d700 1 -- 172.21.15.70:0/2311933330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f22d4105fd0 msgr2=0x7f22d4106450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:06.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.769+0000 7f22db16d700 1 --2- 172.21.15.70:0/2311933330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f22d4105fd0 0x7f22d4106450 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f22cc00b0a0 tx=0x7f22cc00b3b0 comp rx=0 tx=0).stop 2024-06-04T04:58:06.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.769+0000 7f22db16d700 1 -- 172.21.15.70:0/2311933330 shutdown_connections 2024-06-04T04:58:06.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.769+0000 7f22db16d700 1 --2- 172.21.15.70:0/2311933330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f22d4105fd0 0x7f22d4106450 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:06.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.769+0000 7f22db16d700 1 --2- 172.21.15.70:0/2311933330 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22d4108560 0x7f22d406bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:06.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.769+0000 7f22db16d700 1 --2- 172.21.15.70:0/2311933330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22d406c1f0 0x7f22d406c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:06.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.769+0000 7f22db16d700 1 -- 172.21.15.70:0/2311933330 >> 172.21.15.70:0/2311933330 conn(0x7f22d4078490 msgr2=0x7f22d40788a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:06.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.769+0000 7f22db16d700 1 -- 172.21.15.70:0/2311933330 shutdown_connections 2024-06-04T04:58:06.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.769+0000 7f22db16d700 1 -- 172.21.15.70:0/2311933330 wait complete. 2024-06-04T04:58:06.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22db16d700 1 Processor -- start 2024-06-04T04:58:06.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22db16d700 1 -- start start 2024-06-04T04:58:06.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22db16d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22d406c1f0 0x7f22d41a0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:06.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22db16d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22d4105fd0 0x7f22d41a0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:06.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22db16d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f22d4108560 0x7f22d41a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:06.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22db16d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f22d4119b70 con 0x7f22d4108560 2024-06-04T04:58:06.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22db16d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f22d41199f0 con 0x7f22d406c1f0 2024-06-04T04:58:06.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22db16d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f22d4119cf0 con 0x7f22d4105fd0 2024-06-04T04:58:06.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22d970a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f22d4108560 0x7f22d41a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:06.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22d970a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f22d4108560 0x7f22d41a48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55770/0 (socket says 172.21.15.70:55770) 2024-06-04T04:58:06.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22d970a700 1 -- 172.21.15.70:0/899471013 learned_addr learned my addr 172.21.15.70:0/899471013 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:06.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22d8f09700 1 --2- 172.21.15.70:0/899471013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22d406c1f0 0x7f22d41a0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:06.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22d3fff700 1 --2- 172.21.15.70:0/899471013 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22d4105fd0 0x7f22d41a0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:06.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22d970a700 1 -- 172.21.15.70:0/899471013 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22d4105fd0 msgr2=0x7f22d41a0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:06.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22d970a700 1 --2- 172.21.15.70:0/899471013 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22d4105fd0 0x7f22d41a0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:06.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22d970a700 1 -- 172.21.15.70:0/899471013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22d406c1f0 msgr2=0x7f22d41a0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:06.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22d970a700 1 --2- 172.21.15.70:0/899471013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22d406c1f0 0x7f22d41a0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:06.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22d970a700 1 -- 172.21.15.70:0/899471013 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d41a5010 con 0x7f22d4108560 2024-06-04T04:58:06.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22d8f09700 1 --2- 172.21.15.70:0/899471013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22d406c1f0 0x7f22d41a0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:06.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.770+0000 7f22d970a700 1 --2- 172.21.15.70:0/899471013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f22d4108560 0x7f22d41a48d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f22cc004cd0 tx=0x7f22cc004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:06.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.771+0000 7f22d1ffb700 1 -- 172.21.15.70:0/899471013 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc021070 con 0x7f22d4108560 2024-06-04T04:58:06.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.771+0000 7f22db16d700 1 -- 172.21.15.70:0/899471013 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d41a52a0 con 0x7f22d4108560 2024-06-04T04:58:06.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.771+0000 7f22d1ffb700 1 -- 172.21.15.70:0/899471013 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f22cc007c80 con 0x7f22d4108560 2024-06-04T04:58:06.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.771+0000 7f22d1ffb700 1 -- 172.21.15.70:0/899471013 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc005330 con 0x7f22d4108560 2024-06-04T04:58:06.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.771+0000 7f22db16d700 1 -- 172.21.15.70:0/899471013 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d41acb80 con 0x7f22d4108560 2024-06-04T04:58:06.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.772+0000 7f22db16d700 1 -- 172.21.15.70:0/899471013 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22d4102ab0 con 0x7f22d4108560 2024-06-04T04:58:06.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.773+0000 7f22d1ffb700 1 -- 172.21.15.70:0/899471013 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 149885+0+0 (secure 0 0 0) 0x7f22cc024070 con 0x7f22d4108560 2024-06-04T04:58:06.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.774+0000 7f22d1ffb700 1 --2- 172.21.15.70:0/899471013 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f22c40b2b10 0x7f22c40b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:06.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.774+0000 7f22d8f09700 1 --2- 172.21.15.70:0/899471013 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f22c40b2b10 0x7f22c40b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:06.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.774+0000 7f22d1ffb700 1 -- 172.21.15.70:0/899471013 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f22cc00f020 con 0x7f22d4108560 2024-06-04T04:58:06.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.774+0000 7f22d8f09700 1 --2- 172.21.15.70:0/899471013 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f22c40b2b10 0x7f22c40b4fd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f22c800bc10 tx=0x7f22c800b5f0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:06.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:06.778+0000 7f22d1ffb700 1 -- 172.21.15.70:0/899471013 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f22cc02b020 con 0x7f22d4108560 2024-06-04T04:58:07.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:07.027+0000 7f22db16d700 1 -- 172.21.15.70:0/899471013 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22d406a230 con 0x7f22d4108560 2024-06-04T04:58:07.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:07.028+0000 7f22d1ffb700 1 -- 172.21.15.70:0/899471013 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f22d406a230 con 0x7f22d4108560 2024-06-04T04:58:07.031 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:58:07.032 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:57:09.013178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T04:58:07.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:07.031+0000 7f22db16d700 1 -- 172.21.15.70:0/899471013 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f22c40b2b10 msgr2=0x7f22c40b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:07.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:07.031+0000 7f22db16d700 1 --2- 172.21.15.70:0/899471013 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f22c40b2b10 0x7f22c40b4fd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f22c800bc10 tx=0x7f22c800b5f0 comp rx=0 tx=0).stop 2024-06-04T04:58:07.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:07.031+0000 7f22db16d700 1 -- 172.21.15.70:0/899471013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f22d4108560 msgr2=0x7f22d41a48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:07.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:07.031+0000 7f22db16d700 1 --2- 172.21.15.70:0/899471013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f22d4108560 0x7f22d41a48d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f22cc004cd0 tx=0x7f22cc004db0 comp rx=0 tx=0).stop 2024-06-04T04:58:07.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:07.031+0000 7f22db16d700 1 -- 172.21.15.70:0/899471013 shutdown_connections 2024-06-04T04:58:07.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:07.031+0000 7f22db16d700 1 --2- 172.21.15.70:0/899471013 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f22c40b2b10 0x7f22c40b4fd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:07.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:07.031+0000 7f22db16d700 1 --2- 172.21.15.70:0/899471013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f22d4108560 0x7f22d41a48d0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:07.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:07.031+0000 7f22db16d700 1 --2- 172.21.15.70:0/899471013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22d406c1f0 0x7f22d41a0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:07.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:07.031+0000 7f22db16d700 1 --2- 172.21.15.70:0/899471013 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22d4105fd0 0x7f22d41a0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:07.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:07.031+0000 7f22db16d700 1 -- 172.21.15.70:0/899471013 >> 172.21.15.70:0/899471013 conn(0x7f22d4078490 msgr2=0x7f22d4103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:07.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:07.031+0000 7f22db16d700 1 -- 172.21.15.70:0/899471013 shutdown_connections 2024-06-04T04:58:07.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:07.031+0000 7f22db16d700 1 -- 172.21.15.70:0/899471013 wait complete. 2024-06-04T04:58:07.035 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-06-04T04:58:07.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:07 smithi070 ceph-mon[32631]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-06-04T04:58:07.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:07 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/899471013' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:58:07.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:07 smithi112 ceph-mon[36453]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-06-04T04:58:07.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:07 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/899471013' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:58:07.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:07 smithi098 ceph-mon[35569]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-06-04T04:58:07.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:07 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/899471013' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:58:07.792 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T04:58:08.035 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:09.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:09 smithi070 ceph-mon[32631]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-06-04T04:58:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:09 smithi112 ceph-mon[36453]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-06-04T04:58:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:09 smithi098 ceph-mon[35569]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-06-04T04:58:09.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.845+0000 7f0ddabc4700 1 -- 172.21.15.70:0/4180388685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dd41095a0 msgr2=0x7f0dd4109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:09.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.845+0000 7f0ddabc4700 1 --2- 172.21.15.70:0/4180388685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dd41095a0 0x7f0dd4109980 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8009990 tx=0x7f0dc8009ca0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.845+0000 7f0ddabc4700 1 -- 172.21.15.70:0/4180388685 shutdown_connections 2024-06-04T04:58:09.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.845+0000 7f0ddabc4700 1 --2- 172.21.15.70:0/4180388685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0dd406c470 0x7f0dd41103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.845+0000 7f0ddabc4700 1 --2- 172.21.15.70:0/4180388685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dd41095a0 0x7f0dd4109980 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.845+0000 7f0ddabc4700 1 --2- 172.21.15.70:0/4180388685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dd406bab0 0x7f0dd406bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.845+0000 7f0ddabc4700 1 -- 172.21.15.70:0/4180388685 >> 172.21.15.70:0/4180388685 conn(0x7f0dd40793b0 msgr2=0x7f0dd40797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:09.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.845+0000 7f0ddabc4700 1 -- 172.21.15.70:0/4180388685 shutdown_connections 2024-06-04T04:58:09.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.845+0000 7f0ddabc4700 1 -- 172.21.15.70:0/4180388685 wait complete. 2024-06-04T04:58:09.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.845+0000 7f0ddabc4700 1 Processor -- start 2024-06-04T04:58:09.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.846+0000 7f0ddabc4700 1 -- start start 2024-06-04T04:58:09.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.846+0000 7f0ddabc4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dd406bab0 0x7f0dd4102500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.846+0000 7f0ddabc4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0dd406c470 0x7f0dd4102a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.846+0000 7f0ddabc4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dd41095a0 0x7f0dd4102f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.846+0000 7f0ddabc4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd4113560 con 0x7f0dd406bab0 2024-06-04T04:58:09.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.846+0000 7f0ddabc4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd41133e0 con 0x7f0dd41095a0 2024-06-04T04:58:09.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.846+0000 7f0ddabc4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd41136e0 con 0x7f0dd406c470 2024-06-04T04:58:09.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.846+0000 7f0dd8960700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dd406bab0 0x7f0dd4102500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.846+0000 7f0dd8960700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dd406bab0 0x7f0dd4102500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55798/0 (socket says 172.21.15.70:55798) 2024-06-04T04:58:09.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.846+0000 7f0dd8960700 1 -- 172.21.15.70:0/2496734077 learned_addr learned my addr 172.21.15.70:0/2496734077 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:09.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.846+0000 7f0dd9161700 1 --2- 172.21.15.70:0/2496734077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dd41095a0 0x7f0dd4102f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.846+0000 7f0dd3fff700 1 --2- 172.21.15.70:0/2496734077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0dd406c470 0x7f0dd4102a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.847+0000 7f0dd8960700 1 -- 172.21.15.70:0/2496734077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0dd406c470 msgr2=0x7f0dd4102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:09.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.847+0000 7f0dd8960700 1 --2- 172.21.15.70:0/2496734077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0dd406c470 0x7f0dd4102a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.847+0000 7f0dd8960700 1 -- 172.21.15.70:0/2496734077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dd41095a0 msgr2=0x7f0dd4102f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:09.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.847+0000 7f0dd8960700 1 --2- 172.21.15.70:0/2496734077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dd41095a0 0x7f0dd4102f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.847+0000 7f0dd8960700 1 -- 172.21.15.70:0/2496734077 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd41a8000 con 0x7f0dd406bab0 2024-06-04T04:58:09.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.847+0000 7f0dd3fff700 1 --2- 172.21.15.70:0/2496734077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0dd406c470 0x7f0dd4102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:09.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.847+0000 7f0dd9161700 1 --2- 172.21.15.70:0/2496734077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dd41095a0 0x7f0dd4102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:09.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.847+0000 7f0dd8960700 1 --2- 172.21.15.70:0/2496734077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dd406bab0 0x7f0dd4102500 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8006010 tx=0x7f0dc800bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:09.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.847+0000 7f0dd1ffb700 1 -- 172.21.15.70:0/2496734077 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8027070 con 0x7f0dd406bab0 2024-06-04T04:58:09.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.847+0000 7f0ddabc4700 1 -- 172.21.15.70:0/2496734077 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dd41a8290 con 0x7f0dd406bab0 2024-06-04T04:58:09.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.847+0000 7f0dd1ffb700 1 -- 172.21.15.70:0/2496734077 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0dc8022470 con 0x7f0dd406bab0 2024-06-04T04:58:09.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.847+0000 7f0dd1ffb700 1 -- 172.21.15.70:0/2496734077 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8017650 con 0x7f0dd406bab0 2024-06-04T04:58:09.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.847+0000 7f0ddabc4700 1 -- 172.21.15.70:0/2496734077 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dd41a8740 con 0x7f0dd406bab0 2024-06-04T04:58:09.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.850+0000 7f0dd1ffb700 1 -- 172.21.15.70:0/2496734077 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 149885+0+0 (secure 0 0 0) 0x7f0dc8009d10 con 0x7f0dd406bab0 2024-06-04T04:58:09.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.851+0000 7f0dd1ffb700 1 --2- 172.21.15.70:0/2496734077 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0dc40b2b10 0x7f0dc40b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.851+0000 7f0dd1ffb700 1 -- 172.21.15.70:0/2496734077 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f0dc80e3560 con 0x7f0dd406bab0 2024-06-04T04:58:09.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.851+0000 7f0dd3fff700 1 --2- 172.21.15.70:0/2496734077 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0dc40b2b10 0x7f0dc40b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.851+0000 7f0de5459b80 1 Processor -- start 2024-06-04T04:58:09.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0de5459b80 1 -- start start 2024-06-04T04:58:09.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0de5459b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bc1320 0x55e781bc1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0de5459b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bbb8a0 0x55e781bbbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0de5459b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbc260 0x55e781bc8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0de5459b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e781b19d70 con 0x55e781bbb8a0 2024-06-04T04:58:09.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0de5459b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e781b09910 con 0x55e781bc1320 2024-06-04T04:58:09.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0de5459b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e781b0d950 con 0x55e781bbc260 2024-06-04T04:58:09.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0dd3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bbb8a0 0x55e781bbbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0dd3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bbb8a0 0x55e781bbbd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55800/0 (socket says 172.21.15.70:55800) 2024-06-04T04:58:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0dd3fff700 1 -- 172.21.15.70:0/954874433 learned_addr learned my addr 172.21.15.70:0/954874433 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0dd9161700 1 --2- 172.21.15.70:0/954874433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbc260 0x55e781bc8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0dd3fff700 1 --2- 172.21.15.70:0/2496734077 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0dc40b2b10 0x7f0dc40b4fd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4079a20 tx=0x7f0dc0009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0dd8960700 1 --2- 172.21.15.70:0/954874433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bc1320 0x55e781bc1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0dd3fff700 1 -- 172.21.15.70:0/954874433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbc260 msgr2=0x55e781bc8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0dd3fff700 1 --2- 172.21.15.70:0/954874433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbc260 0x55e781bc8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0dd3fff700 1 -- 172.21.15.70:0/954874433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bc1320 msgr2=0x55e781bc1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0dd3fff700 1 --2- 172.21.15.70:0/954874433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bc1320 0x55e781bc1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.852+0000 7f0dd3fff700 1 -- 172.21.15.70:0/954874433 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e781ab2fb0 con 0x55e781bbb8a0 2024-06-04T04:58:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.853+0000 7f0dd3fff700 1 --2- 172.21.15.70:0/954874433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bbb8a0 0x55e781bbbd20 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0010c90 tx=0x7f0dc0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107daf72951ca12a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.853+0000 7f0dbeffd700 1 -- 172.21.15.70:0/954874433 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0023020 con 0x55e781bbb8a0 2024-06-04T04:58:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.853+0000 7f0dbeffd700 1 -- 172.21.15.70:0/954874433 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0dc00128e0 con 0x55e781bbb8a0 2024-06-04T04:58:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.853+0000 7f0dbeffd700 1 -- 172.21.15.70:0/954874433 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0003c00 con 0x55e781bbb8a0 2024-06-04T04:58:09.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.854+0000 7f0de5459b80 1 -- 172.21.15.70:0/954874433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bbb8a0 msgr2=0x55e781bbbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:09.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.854+0000 7f0de5459b80 1 --2- 172.21.15.70:0/954874433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bbb8a0 0x55e781bbbd20 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0010c90 tx=0x7f0dc0013000 comp rx=0 tx=0).stop 2024-06-04T04:58:09.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.854+0000 7f0de5459b80 1 -- 172.21.15.70:0/954874433 shutdown_connections 2024-06-04T04:58:09.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.854+0000 7f0de5459b80 1 --2- 172.21.15.70:0/954874433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbc260 0x55e781bc8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.854+0000 7f0de5459b80 1 --2- 172.21.15.70:0/954874433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bc1320 0x55e781bc1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.854+0000 7f0de5459b80 1 --2- 172.21.15.70:0/954874433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bbb8a0 0x55e781bbbd20 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.854+0000 7f0de5459b80 1 -- 172.21.15.70:0/954874433 >> 172.21.15.70:0/954874433 conn(0x55e781ae7240 msgr2=0x55e781bb63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:09.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.854+0000 7f0de5459b80 1 -- 172.21.15.70:0/954874433 shutdown_connections 2024-06-04T04:58:09.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.854+0000 7f0de5459b80 1 -- 172.21.15.70:0/954874433 wait complete. 2024-06-04T04:58:09.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.854+0000 7f0de5459b80 1 -- start start 2024-06-04T04:58:09.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.854+0000 7f0de5459b80 1 Processor -- start 2024-06-04T04:58:09.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0de5459b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 0x55e781bc7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0de5459b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bbc260 0x55e781bc4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0de5459b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bc1320 0x55e781bc5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0de5459b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e781a0ffa0 con 0x55e781bc1320 2024-06-04T04:58:09.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0dd9161700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bc1320 0x55e781bc5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0dd8960700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 0x55e781bc7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0dd8960700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 0x55e781bc7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59122/0 (socket says 172.21.15.70:59122) 2024-06-04T04:58:09.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0dd8960700 1 -- 172.21.15.70:0/3822296439 learned_addr learned my addr 172.21.15.70:0/3822296439 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:09.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0de5459b80 1 -- 172.21.15.70:0/3822296439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e781a71da0 con 0x55e781bbc260 2024-06-04T04:58:09.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0de5459b80 1 -- 172.21.15.70:0/3822296439 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e781bc86e0 con 0x55e781bbb8a0 2024-06-04T04:58:09.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0dd3fff700 1 --2- 172.21.15.70:0/3822296439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bbc260 0x55e781bc4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0de5459b80 10 client.0 _ll_register_callbacks cb 0x55e781aa4950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:09.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0dd8960700 1 -- 172.21.15.70:0/3822296439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bbc260 msgr2=0x55e781bc4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:09.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0dd8960700 1 --2- 172.21.15.70:0/3822296439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bbc260 0x55e781bc4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0dd8960700 1 -- 172.21.15.70:0/3822296439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bc1320 msgr2=0x55e781bc5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:09.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0dd8960700 1 --2- 172.21.15.70:0/3822296439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bc1320 0x55e781bc5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0dd8960700 1 -- 172.21.15.70:0/3822296439 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e781a598d0 con 0x55e781bbb8a0 2024-06-04T04:58:09.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0dd9161700 1 --2- 172.21.15.70:0/3822296439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bc1320 0x55e781bc5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:09.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0de5459b80 1 -- 172.21.15.70:0/3822296439 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0dc00128e0 con 0x55e781bbb8a0 2024-06-04T04:58:09.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0dd8960700 1 --2- 172.21.15.70:0/3822296439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 0x55e781bc7cf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0dc80a6700 tx=0x7f0dc8009710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a85f2a95de4f899c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:09.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.855+0000 7f0db3fff700 10 client.25207 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:58:09.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.856+0000 7f0db3fff700 1 -- 172.21.15.70:0/3822296439 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc80aad80 con 0x55e781bbb8a0 2024-06-04T04:58:09.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.856+0000 7f0db3fff700 1 -- 172.21.15.70:0/3822296439 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f0dc80b2fc0 con 0x55e781bbb8a0 2024-06-04T04:58:09.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.856+0000 7f0db3fff700 1 -- 172.21.15.70:0/3822296439 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0dc80b31c0 con 0x55e781bbb8a0 2024-06-04T04:58:09.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.856+0000 7f0de5459b80 10 client.25207 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:09.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.856+0000 7f0de5459b80 1 -- 172.21.15.70:0/3822296439 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55e781ac3760 con 0x55e781bbb8a0 2024-06-04T04:58:09.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.856+0000 7f0db3fff700 1 -- 172.21.15.70:0/3822296439 <== mon.1 v2:172.21.15.112:3300/0 4 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f0dc80afbc0 con 0x55e781bbb8a0 2024-06-04T04:58:09.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.856+0000 7f0de5459b80 10 client.25207 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:09.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.856+0000 7f0de5459b80 10 client.25207 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T04:58:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.856+0000 7f0de5459b80 20 client.25207 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.856+0000 7f0de5459b80 1 --2- 172.21.15.70:0/3822296439 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55e781bd4f60 0x55e781bd73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.856+0000 7f0de5459b80 4 client.25207 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.856+0000 7f0de5459b80 1 -- 172.21.15.70:0/3822296439 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e781b1bcc0 con 0x55e781bd4f60 2024-06-04T04:58:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.857+0000 7f0dd3fff700 1 --2- 172.21.15.70:0/3822296439 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55e781bd4f60 0x55e781bd73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.857+0000 7f0dd3fff700 1 --2- 172.21.15.70:0/3822296439 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55e781bd4f60 0x55e781bd73d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9de760480b295bf9 server_cookie=39e9a6c2cde0bc12 in_seq=0 out_seq=0 2024-06-04T04:58:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.857+0000 7f0db3fff700 10 client.25207 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T04:58:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.858+0000 7f0db3fff700 1 -- 172.21.15.70:0/3822296439 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e781b1bcc0 con 0x55e781bd4f60 2024-06-04T04:58:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.858+0000 7f0db3fff700 10 client.25207 handle_command_reply: tid=0 2024-06-04T04:58:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.858+0000 7f0de5459b80 1 client.25207 shutdown 2024-06-04T04:58:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.858+0000 7f0de5459b80 1 -- 172.21.15.70:0/3822296439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 msgr2=0x55e781bc7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.858+0000 7f0de5459b80 1 --2- 172.21.15.70:0/3822296439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 0x55e781bc7cf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0dc80a6700 tx=0x7f0dc8009710 comp rx=0 tx=0).stop 2024-06-04T04:58:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.858+0000 7f0de5459b80 1 -- 172.21.15.70:0/3822296439 shutdown_connections 2024-06-04T04:58:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.858+0000 7f0de5459b80 1 --2- 172.21.15.70:0/3822296439 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55e781bd4f60 0x55e781bd73d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.858+0000 7f0de5459b80 1 --2- 172.21.15.70:0/3822296439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bc1320 0x55e781bc5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.858+0000 7f0de5459b80 1 --2- 172.21.15.70:0/3822296439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 0x55e781bc7cf0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.858+0000 7f0de5459b80 1 --2- 172.21.15.70:0/3822296439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bbc260 0x55e781bc4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.858+0000 7f0de5459b80 1 -- 172.21.15.70:0/3822296439 >> 172.21.15.70:0/3822296439 conn(0x55e781ae7240 msgr2=0x55e781bb5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.858+0000 7f0db3fff700 0 client.25207 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T04:58:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.858+0000 7f0de5459b80 1 -- 172.21.15.70:0/3822296439 shutdown_connections 2024-06-04T04:58:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.858+0000 7f0de5459b80 1 -- 172.21.15.70:0/3822296439 wait complete. 2024-06-04T04:58:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.858+0000 7f0de5459b80 20 client.25207 trim_cache size 0 max 16384 2024-06-04T04:58:09.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0de5459b80 1 Processor -- start 2024-06-04T04:58:09.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0de5459b80 1 -- start start 2024-06-04T04:58:09.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0de5459b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 0x55e781bc41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0de5459b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bbc260 0x55e781bb6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0de5459b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bc1320 0x55e781bc6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0de5459b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd41afe30 con 0x55e781bbc260 2024-06-04T04:58:09.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0de5459b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd41b1970 con 0x55e781bc1320 2024-06-04T04:58:09.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0de5459b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd41b0500 con 0x55e781bbb8a0 2024-06-04T04:58:09.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0dd3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bbc260 0x55e781bb6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0dd3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bbc260 0x55e781bb6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55822/0 (socket says 172.21.15.70:55822) 2024-06-04T04:58:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0dd3fff700 1 -- 172.21.15.70:0/2252081579 learned_addr learned my addr 172.21.15.70:0/2252081579 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0dd8960700 1 --2- 172.21.15.70:0/2252081579 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 0x55e781bc41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0dd9161700 1 --2- 172.21.15.70:0/2252081579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bc1320 0x55e781bc6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0dd3fff700 1 -- 172.21.15.70:0/2252081579 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 msgr2=0x55e781bc41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0dd3fff700 1 --2- 172.21.15.70:0/2252081579 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 0x55e781bc41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0dd3fff700 1 -- 172.21.15.70:0/2252081579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bc1320 msgr2=0x55e781bc6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0dd3fff700 1 --2- 172.21.15.70:0/2252081579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bc1320 0x55e781bc6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.873+0000 7f0dd3fff700 1 -- 172.21.15.70:0/2252081579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e781bc84f0 con 0x55e781bbc260 2024-06-04T04:58:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.874+0000 7f0dd9161700 1 --2- 172.21.15.70:0/2252081579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bc1320 0x55e781bc6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.874+0000 7f0dd3fff700 1 --2- 172.21.15.70:0/2252081579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bbc260 0x55e781bb6270 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0010c60 tx=0x7f0dc00092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191f052d229e2313 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.874+0000 7f0dbe7fc700 1 -- 172.21.15.70:0/2252081579 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0016070 con 0x55e781bbc260 2024-06-04T04:58:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.874+0000 7f0dbe7fc700 1 -- 172.21.15.70:0/2252081579 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0dc00036a0 con 0x55e781bbc260 2024-06-04T04:58:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.874+0000 7f0dbe7fc700 1 -- 172.21.15.70:0/2252081579 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0023040 con 0x55e781bbc260 2024-06-04T04:58:09.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.875+0000 7f0de5459b80 1 -- 172.21.15.70:0/2252081579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bbc260 msgr2=0x55e781bb6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:09.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.875+0000 7f0de5459b80 1 --2- 172.21.15.70:0/2252081579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bbc260 0x55e781bb6270 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0010c60 tx=0x7f0dc00092d0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.875+0000 7f0de5459b80 1 -- 172.21.15.70:0/2252081579 shutdown_connections 2024-06-04T04:58:09.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.875+0000 7f0de5459b80 1 --2- 172.21.15.70:0/2252081579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bc1320 0x55e781bc6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.875+0000 7f0de5459b80 1 --2- 172.21.15.70:0/2252081579 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 0x55e781bc41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.875+0000 7f0de5459b80 1 --2- 172.21.15.70:0/2252081579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bbc260 0x55e781bb6270 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.875+0000 7f0de5459b80 1 -- 172.21.15.70:0/2252081579 >> 172.21.15.70:0/2252081579 conn(0x55e781ae7240 msgr2=0x55e781bc6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:09.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.875+0000 7f0de5459b80 1 -- 172.21.15.70:0/2252081579 shutdown_connections 2024-06-04T04:58:09.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.875+0000 7f0de5459b80 1 -- 172.21.15.70:0/2252081579 wait complete. 2024-06-04T04:58:09.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.875+0000 7f0de5459b80 1 -- start start 2024-06-04T04:58:09.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.875+0000 7f0de5459b80 1 Processor -- start 2024-06-04T04:58:09.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0de5459b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 0x55e781bdbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0de5459b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bbc260 0x55e781bdc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0de5459b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bc1320 0x55e781bcc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0de5459b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd41b12a0 con 0x55e781bc1320 2024-06-04T04:58:09.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0de5459b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd41b0bd0 con 0x55e781bbc260 2024-06-04T04:58:09.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0de5459b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd41addd0 con 0x55e781bbb8a0 2024-06-04T04:58:09.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0dd9161700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bc1320 0x55e781bcc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0dd8960700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 0x55e781bdbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0dd8960700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 0x55e781bdbd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59142/0 (socket says 172.21.15.70:59142) 2024-06-04T04:58:09.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0dd8960700 1 -- 172.21.15.70:0/1141000294 learned_addr learned my addr 172.21.15.70:0/1141000294 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:09.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0de5459b80 10 client.0 _ll_register_callbacks cb 0x55e781aa4950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:09.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0dd3fff700 1 --2- 172.21.15.70:0/1141000294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bbc260 0x55e781bdc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0dd9161700 1 -- 172.21.15.70:0/1141000294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 msgr2=0x55e781bdbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:09.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0dd9161700 1 --2- 172.21.15.70:0/1141000294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 0x55e781bdbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0dd9161700 1 -- 172.21.15.70:0/1141000294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bbc260 msgr2=0x55e781bdc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:09.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0dd9161700 1 --2- 172.21.15.70:0/1141000294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bbc260 0x55e781bdc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0dd9161700 1 -- 172.21.15.70:0/1141000294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e781bdcb60 con 0x55e781bc1320 2024-06-04T04:58:09.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.877+0000 7f0dd3fff700 1 --2- 172.21.15.70:0/1141000294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bbc260 0x55e781bdc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:09.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.877+0000 7f0dd9161700 1 --2- 172.21.15.70:0/1141000294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bc1320 0x55e781bcc930 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x55e781bb0ef0 tx=0x7f0dcc004910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff40be846a38b7c3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:09.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.877+0000 7f0db3fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:09.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.876+0000 7f0dd8960700 1 --2- 172.21.15.70:0/1141000294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 0x55e781bdbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:09.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.877+0000 7f0db3fff700 1 -- 172.21.15.70:0/1141000294 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc011c40 con 0x55e781bc1320 2024-06-04T04:58:09.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.877+0000 7f0db3fff700 1 -- 172.21.15.70:0/1141000294 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f0dcc011de0 con 0x55e781bc1320 2024-06-04T04:58:09.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.877+0000 7f0de5459b80 1 -- 172.21.15.70:0/1141000294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0dc00036a0 con 0x55e781bc1320 2024-06-04T04:58:09.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.877+0000 7f0db3fff700 1 -- 172.21.15.70:0/1141000294 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0dcc017430 con 0x55e781bc1320 2024-06-04T04:58:09.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.877+0000 7f0de5459b80 10 client.15651 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:09.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.877+0000 7f0de5459b80 1 -- 172.21.15.70:0/1141000294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e781bcdf20 con 0x55e781bc1320 2024-06-04T04:58:09.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.877+0000 7f0db3fff700 1 -- 172.21.15.70:0/1141000294 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0dc00036a0 con 0x55e781bc1320 2024-06-04T04:58:09.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.878+0000 7f0db3fff700 1 -- 172.21.15.70:0/1141000294 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f0dcc01f590 con 0x55e781bc1320 2024-06-04T04:58:09.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.878+0000 7f0de5459b80 10 client.15651 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:09.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.878+0000 7f0de5459b80 10 client.15651 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T04:58:09.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.878+0000 7f0de5459b80 20 client.15651 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:09.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.878+0000 7f0de5459b80 1 --2- 172.21.15.70:0/1141000294 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55e781c3fbc0 0x55e781c41fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.878+0000 7f0de5459b80 4 client.15651 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T04:58:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.878+0000 7f0de5459b80 1 -- 172.21.15.70:0/1141000294 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e781b154a0 con 0x55e781c3fbc0 2024-06-04T04:58:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.878+0000 7f0dd8960700 1 --2- 172.21.15.70:0/1141000294 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55e781c3fbc0 0x55e781c41fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.878+0000 7f0dd8960700 1 --2- 172.21.15.70:0/1141000294 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55e781c3fbc0 0x55e781c41fb0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4316d47a93556653 server_cookie=b4f863726ab31d6b in_seq=0 out_seq=0 2024-06-04T04:58:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.878+0000 7f0db3fff700 10 client.15651 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T04:58:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.879+0000 7f0db3fff700 1 -- 172.21.15.70:0/1141000294 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e781b154a0 con 0x55e781c3fbc0 2024-06-04T04:58:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.879+0000 7f0db3fff700 10 client.15651 handle_command_reply: tid=0 2024-06-04T04:58:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.879+0000 7f0de5459b80 1 client.15651 shutdown 2024-06-04T04:58:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.879+0000 7f0de5459b80 1 -- 172.21.15.70:0/1141000294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bc1320 msgr2=0x55e781bcc930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.879+0000 7f0de5459b80 1 --2- 172.21.15.70:0/1141000294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bc1320 0x55e781bcc930 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x55e781bb0ef0 tx=0x7f0dcc004910 comp rx=0 tx=0).stop 2024-06-04T04:58:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.879+0000 7f0de5459b80 1 -- 172.21.15.70:0/1141000294 shutdown_connections 2024-06-04T04:58:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.879+0000 7f0de5459b80 1 --2- 172.21.15.70:0/1141000294 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55e781c3fbc0 0x55e781c41fb0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.879+0000 7f0de5459b80 1 --2- 172.21.15.70:0/1141000294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e781bc1320 0x55e781bcc930 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.879+0000 7f0de5459b80 1 --2- 172.21.15.70:0/1141000294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e781bbb8a0 0x55e781bdbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.879+0000 7f0de5459b80 1 --2- 172.21.15.70:0/1141000294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e781bbc260 0x55e781bdc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.879+0000 7f0de5459b80 1 -- 172.21.15.70:0/1141000294 >> 172.21.15.70:0/1141000294 conn(0x55e781ae7240 msgr2=0x55e781bc4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:09.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.879+0000 7f0db3fff700 0 client.15651 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T04:58:09.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.879+0000 7f0de5459b80 1 -- 172.21.15.70:0/1141000294 shutdown_connections 2024-06-04T04:58:09.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.879+0000 7f0de5459b80 1 -- 172.21.15.70:0/1141000294 wait complete. 2024-06-04T04:58:09.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.879+0000 7f0de5459b80 20 client.15651 trim_cache size 0 max 16384 2024-06-04T04:58:09.887 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T04:58:09.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.880+0000 7f0ddabc4700 1 -- 172.21.15.70:0/2496734077 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0dc40b2b10 msgr2=0x7f0dc40b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:09.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.880+0000 7f0ddabc4700 1 --2- 172.21.15.70:0/2496734077 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0dc40b2b10 0x7f0dc40b4fd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4079a20 tx=0x7f0dc0009260 comp rx=0 tx=0).stop 2024-06-04T04:58:09.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.880+0000 7f0ddabc4700 1 -- 172.21.15.70:0/2496734077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dd406bab0 msgr2=0x7f0dd4102500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:09.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.880+0000 7f0ddabc4700 1 --2- 172.21.15.70:0/2496734077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dd406bab0 0x7f0dd4102500 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8006010 tx=0x7f0dc800bfc0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.880+0000 7f0ddabc4700 1 -- 172.21.15.70:0/2496734077 shutdown_connections 2024-06-04T04:58:09.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.880+0000 7f0ddabc4700 1 --2- 172.21.15.70:0/2496734077 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0dc40b2b10 0x7f0dc40b4fd0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.880+0000 7f0ddabc4700 1 --2- 172.21.15.70:0/2496734077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dd41095a0 0x7f0dd4102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.880+0000 7f0ddabc4700 1 --2- 172.21.15.70:0/2496734077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dd406bab0 0x7f0dd4102500 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.880+0000 7f0ddabc4700 1 --2- 172.21.15.70:0/2496734077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0dd406c470 0x7f0dd4102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:09.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.880+0000 7f0ddabc4700 1 -- 172.21.15.70:0/2496734077 >> 172.21.15.70:0/2496734077 conn(0x7f0dd40793b0 msgr2=0x7f0dd4101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:09.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.880+0000 7f0ddabc4700 1 -- 172.21.15.70:0/2496734077 shutdown_connections 2024-06-04T04:58:09.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:09.880+0000 7f0ddabc4700 1 -- 172.21.15.70:0/2496734077 wait complete. 2024-06-04T04:58:10.614 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T04:58:10.869 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:11.316 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:11 smithi070 ceph-mon[32631]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-06-04T04:58:11.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:11 smithi112 ceph-mon[36453]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-06-04T04:58:11.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:11 smithi098 ceph-mon[35569]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-06-04T04:58:12.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.661+0000 7fb4bc93b700 1 -- 172.21.15.70:0/3339151131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4b8108710 msgr2=0x7fb4b8108af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:12.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.661+0000 7fb4bc93b700 1 --2- 172.21.15.70:0/3339151131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4b8108710 0x7fb4b8108af0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8009990 tx=0x7fb4a8009ca0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.661+0000 7fb4bc93b700 1 -- 172.21.15.70:0/3339151131 shutdown_connections 2024-06-04T04:58:12.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.661+0000 7fb4bc93b700 1 --2- 172.21.15.70:0/3339151131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4b8103560 0x7fb4b810f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.662+0000 7fb4bc93b700 1 --2- 172.21.15.70:0/3339151131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4b8108710 0x7fb4b8108af0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.662+0000 7fb4bc93b700 1 --2- 172.21.15.70:0/3339151131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4b8102ba0 0x7fb4b8103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.662+0000 7fb4bc93b700 1 -- 172.21.15.70:0/3339151131 >> 172.21.15.70:0/3339151131 conn(0x7fb4b8078a40 msgr2=0x7fb4b80ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:12.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.662+0000 7fb4bc93b700 1 -- 172.21.15.70:0/3339151131 shutdown_connections 2024-06-04T04:58:12.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.662+0000 7fb4bc93b700 1 -- 172.21.15.70:0/3339151131 wait complete. 2024-06-04T04:58:12.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.662+0000 7fb4bc93b700 1 Processor -- start 2024-06-04T04:58:12.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.662+0000 7fb4bc93b700 1 -- start start 2024-06-04T04:58:12.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4bc93b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4b8102ba0 0x7fb4b8199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4bc93b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4b8103560 0x7fb4b819a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4bc93b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4b8108710 0x7fb4b819e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4bc93b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b81127d0 con 0x7fb4b8102ba0 2024-06-04T04:58:12.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4bc93b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b8112650 con 0x7fb4b8108710 2024-06-04T04:58:12.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4bc93b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b8112950 con 0x7fb4b8103560 2024-06-04T04:58:12.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4b659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4b8102ba0 0x7fb4b8199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:12.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4b659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4b8102ba0 0x7fb4b8199e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55860/0 (socket says 172.21.15.70:55860) 2024-06-04T04:58:12.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4b659c700 1 -- 172.21.15.70:0/2648967084 learned_addr learned my addr 172.21.15.70:0/2648967084 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:12.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4b6d9d700 1 --2- 172.21.15.70:0/2648967084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4b8108710 0x7fb4b819e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:12.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4b659c700 1 -- 172.21.15.70:0/2648967084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4b8103560 msgr2=0x7fb4b819a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:12.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4b659c700 1 --2- 172.21.15.70:0/2648967084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4b8103560 0x7fb4b819a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4b659c700 1 -- 172.21.15.70:0/2648967084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4b8108710 msgr2=0x7fb4b819e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:12.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4b659c700 1 --2- 172.21.15.70:0/2648967084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4b8108710 0x7fb4b819e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4b659c700 1 -- 172.21.15.70:0/2648967084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b819ee60 con 0x7fb4b8102ba0 2024-06-04T04:58:12.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4b659c700 1 --2- 172.21.15.70:0/2648967084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4b8102ba0 0x7fb4b8199e50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8000c00 tx=0x7fb4a80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:12.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4a77fe700 1 -- 172.21.15.70:0/2648967084 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a8027070 con 0x7fb4b8102ba0 2024-06-04T04:58:12.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4a77fe700 1 -- 172.21.15.70:0/2648967084 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb4a800fdb0 con 0x7fb4b8102ba0 2024-06-04T04:58:12.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4a77fe700 1 -- 172.21.15.70:0/2648967084 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a80173f0 con 0x7fb4b8102ba0 2024-06-04T04:58:12.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.663+0000 7fb4bc93b700 1 -- 172.21.15.70:0/2648967084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4b819f0f0 con 0x7fb4b8102ba0 2024-06-04T04:58:12.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.664+0000 7fb4bc93b700 1 -- 172.21.15.70:0/2648967084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4b81a69d0 con 0x7fb4b8102ba0 2024-06-04T04:58:12.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.666+0000 7fb4a77fe700 1 -- 172.21.15.70:0/2648967084 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 149885+0+0 (secure 0 0 0) 0x7fb4a802a070 con 0x7fb4b8102ba0 2024-06-04T04:58:12.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.667+0000 7fb4a77fe700 1 --2- 172.21.15.70:0/2648967084 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb4a00b2b90 0x7fb4a00b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.667+0000 7fb4b5d9b700 1 --2- 172.21.15.70:0/2648967084 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb4a00b2b90 0x7fb4a00b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:12.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.667+0000 7fb4a77fe700 1 -- 172.21.15.70:0/2648967084 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb4a80e3930 con 0x7fb4b8102ba0 2024-06-04T04:58:12.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.668+0000 7fb4c71d0b80 1 Processor -- start 2024-06-04T04:58:12.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.668+0000 7fb4b5d9b700 1 --2- 172.21.15.70:0/2648967084 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb4a00b2b90 0x7fb4a00b5050 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8101010 tx=0x7fb4ac006d20 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:12.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.668+0000 7fb4c71d0b80 1 -- start start 2024-06-04T04:58:12.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.668+0000 7fb4c71d0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057ea0320 0x558057ea0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.668+0000 7fb4c71d0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057e9a8a0 0x558057e9ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.668+0000 7fb4c71d0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057ea7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.668+0000 7fb4c71d0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558057df8d70 con 0x558057e9b260 2024-06-04T04:58:12.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.668+0000 7fb4c71d0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558057de8910 con 0x558057ea0320 2024-06-04T04:58:12.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.668+0000 7fb4c71d0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558057dec950 con 0x558057e9a8a0 2024-06-04T04:58:12.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.668+0000 7fb4b6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057ea7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:12.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.668+0000 7fb4b6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057ea7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55874/0 (socket says 172.21.15.70:55874) 2024-06-04T04:58:12.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.668+0000 7fb4b6d9d700 1 -- 172.21.15.70:0/539580416 learned_addr learned my addr 172.21.15.70:0/539580416 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:12.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.668+0000 7fb4b659c700 1 --2- 172.21.15.70:0/539580416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057ea0320 0x558057ea0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:12.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.668+0000 7fb4b5d9b700 1 --2- 172.21.15.70:0/539580416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057e9a8a0 0x558057e9ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:12.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.669+0000 7fb4b6d9d700 1 -- 172.21.15.70:0/539580416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057e9a8a0 msgr2=0x558057e9ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:12.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.669+0000 7fb4b6d9d700 1 --2- 172.21.15.70:0/539580416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057e9a8a0 0x558057e9ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.669+0000 7fb4b6d9d700 1 -- 172.21.15.70:0/539580416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057ea0320 msgr2=0x558057ea0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:12.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.669+0000 7fb4b6d9d700 1 --2- 172.21.15.70:0/539580416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057ea0320 0x558057ea0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.669+0000 7fb4b6d9d700 1 -- 172.21.15.70:0/539580416 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558057d91fb0 con 0x558057e9b260 2024-06-04T04:58:12.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.669+0000 7fb4b6d9d700 1 --2- 172.21.15.70:0/539580416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057ea7080 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0009f10 tx=0x7fb4b0009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1f708135a93bec server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:12.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.669+0000 7fb4a4ff9700 1 -- 172.21.15.70:0/539580416 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b001a070 con 0x558057e9b260 2024-06-04T04:58:12.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.669+0000 7fb4a4ff9700 1 -- 172.21.15.70:0/539580416 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb4b0015410 con 0x558057e9b260 2024-06-04T04:58:12.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.669+0000 7fb4a4ff9700 1 -- 172.21.15.70:0/539580416 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b000a660 con 0x558057e9b260 2024-06-04T04:58:12.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.670+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/539580416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 msgr2=0x558057ea7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:12.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.670+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/539580416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057ea7080 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0009f10 tx=0x7fb4b0009f40 comp rx=0 tx=0).stop 2024-06-04T04:58:12.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.670+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/539580416 shutdown_connections 2024-06-04T04:58:12.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.670+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/539580416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057ea7080 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.670+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/539580416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057ea0320 0x558057ea0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.670+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/539580416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057e9a8a0 0x558057e9ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.670+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/539580416 >> 172.21.15.70:0/539580416 conn(0x558057dc6240 msgr2=0x558057e953e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:12.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.670+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/539580416 shutdown_connections 2024-06-04T04:58:12.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.670+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/539580416 wait complete. 2024-06-04T04:58:12.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4c71d0b80 1 -- start start 2024-06-04T04:58:12.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4c71d0b80 1 Processor -- start 2024-06-04T04:58:12.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4c71d0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057e9a8a0 0x558057ea6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4c71d0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057e9b260 0x558057ea4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4c71d0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057ea0320 0x558057ea4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4c71d0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558057ceefa0 con 0x558057ea0320 2024-06-04T04:58:12.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4c71d0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558057d50da0 con 0x558057e9b260 2024-06-04T04:58:12.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4c71d0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558057eabaf0 con 0x558057e9a8a0 2024-06-04T04:58:12.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4c71d0b80 10 client.0 _ll_register_callbacks cb 0x558057d83950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:12.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4b6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057ea0320 0x558057ea4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:12.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4b6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057ea0320 0x558057ea4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55886/0 (socket says 172.21.15.70:55886) 2024-06-04T04:58:12.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4b6d9d700 1 -- 172.21.15.70:0/3849520792 learned_addr learned my addr 172.21.15.70:0/3849520792 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:12.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4b659c700 1 --2- 172.21.15.70:0/3849520792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057e9a8a0 0x558057ea6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:12.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4b5d9b700 1 --2- 172.21.15.70:0/3849520792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057e9b260 0x558057ea4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:12.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4b6d9d700 1 -- 172.21.15.70:0/3849520792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057e9a8a0 msgr2=0x558057ea6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:12.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4b6d9d700 1 --2- 172.21.15.70:0/3849520792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057e9a8a0 0x558057ea6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4b6d9d700 1 -- 172.21.15.70:0/3849520792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057e9b260 msgr2=0x558057ea4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:12.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4b6d9d700 1 --2- 172.21.15.70:0/3849520792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057e9b260 0x558057ea4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4b6d9d700 1 -- 172.21.15.70:0/3849520792 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558057d388d0 con 0x558057ea0320 2024-06-04T04:58:12.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.671+0000 7fb4b659c700 1 --2- 172.21.15.70:0/3849520792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057e9a8a0 0x558057ea6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:12.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.672+0000 7fb4b6d9d700 1 --2- 172.21.15.70:0/3849520792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057ea0320 0x558057ea4570 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00070b0 tx=0x7fb4b0015a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bbc002d78711cd1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:12.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.672+0000 7fb49dffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:12.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.672+0000 7fb49dffb700 1 -- 172.21.15.70:0/3849520792 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b001a040 con 0x558057ea0320 2024-06-04T04:58:12.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.672+0000 7fb49dffb700 1 -- 172.21.15.70:0/3849520792 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb4b000ca30 con 0x558057ea0320 2024-06-04T04:58:12.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.672+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/3849520792 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4b0015410 con 0x558057ea0320 2024-06-04T04:58:12.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.672+0000 7fb49dffb700 1 -- 172.21.15.70:0/3849520792 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb4b002e050 con 0x558057ea0320 2024-06-04T04:58:12.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.672+0000 7fb4c71d0b80 10 client.15663 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:12.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.672+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/3849520792 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558057da2760 con 0x558057ea0320 2024-06-04T04:58:12.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.672+0000 7fb49dffb700 1 -- 172.21.15.70:0/3849520792 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb4b0015410 con 0x558057ea0320 2024-06-04T04:58:12.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.673+0000 7fb49dffb700 1 -- 172.21.15.70:0/3849520792 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fb4b0020a50 con 0x558057ea0320 2024-06-04T04:58:12.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.673+0000 7fb4c71d0b80 10 client.15663 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:12.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.673+0000 7fb4c71d0b80 10 client.15663 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T04:58:12.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.673+0000 7fb4c71d0b80 20 client.15663 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:12.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.673+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/3849520792 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x558057eb84e0 0x558057eba8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.673+0000 7fb4c71d0b80 4 client.15663 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:12.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.673+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/3849520792 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558057dfacc0 con 0x558057eb84e0 2024-06-04T04:58:12.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.673+0000 7fb4b659c700 1 --2- 172.21.15.70:0/3849520792 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x558057eb84e0 0x558057eba8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:12.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.673+0000 7fb4b659c700 1 --2- 172.21.15.70:0/3849520792 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x558057eb84e0 0x558057eba8d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=435683edbc85f6d2 server_cookie=e4849ab2e981cba2 in_seq=0 out_seq=0 2024-06-04T04:58:12.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.673+0000 7fb49dffb700 10 client.15663 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T04:58:12.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.674+0000 7fb49dffb700 1 -- 172.21.15.70:0/3849520792 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558057dfacc0 con 0x558057eb84e0 2024-06-04T04:58:12.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.674+0000 7fb49dffb700 10 client.15663 handle_command_reply: tid=0 2024-06-04T04:58:12.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.674+0000 7fb4c71d0b80 1 client.15663 shutdown 2024-06-04T04:58:12.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.674+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/3849520792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057ea0320 msgr2=0x558057ea4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:12.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.674+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/3849520792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057ea0320 0x558057ea4570 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00070b0 tx=0x7fb4b0015a50 comp rx=0 tx=0).stop 2024-06-04T04:58:12.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.674+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/3849520792 shutdown_connections 2024-06-04T04:58:12.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.674+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/3849520792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057ea0320 0x558057ea4570 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.674+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/3849520792 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x558057eb84e0 0x558057eba8d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.674+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/3849520792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057e9a8a0 0x558057ea6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.674+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/3849520792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057e9b260 0x558057ea4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.674+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/3849520792 >> 172.21.15.70:0/3849520792 conn(0x558057dc6240 msgr2=0x558057e94af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:12.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.674+0000 7fb49dffb700 0 client.15663 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T04:58:12.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.674+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/3849520792 shutdown_connections 2024-06-04T04:58:12.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.675+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/3849520792 wait complete. 2024-06-04T04:58:12.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.675+0000 7fb4c71d0b80 20 client.15663 trim_cache size 0 max 16384 2024-06-04T04:58:12.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4c71d0b80 1 Processor -- start 2024-06-04T04:58:12.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4c71d0b80 1 -- start start 2024-06-04T04:58:12.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4c71d0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057e9a8a0 0x558057ea4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4c71d0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057e95270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4c71d0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057ea0320 0x558057e99fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4c71d0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b81abb20 con 0x558057e9b260 2024-06-04T04:58:12.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4c71d0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b81ad660 con 0x558057e9a8a0 2024-06-04T04:58:12.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4c71d0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b81ac1f0 con 0x558057ea0320 2024-06-04T04:58:12.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4b5d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057e95270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:12.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4b5d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057e95270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55894/0 (socket says 172.21.15.70:55894) 2024-06-04T04:58:12.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4b5d9b700 1 -- 172.21.15.70:0/672630965 learned_addr learned my addr 172.21.15.70:0/672630965 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:12.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4b659c700 1 --2- 172.21.15.70:0/672630965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057e9a8a0 0x558057ea4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:12.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4b5d9b700 1 -- 172.21.15.70:0/672630965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057ea0320 msgr2=0x558057e99fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:12.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4b5d9b700 1 --2- 172.21.15.70:0/672630965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057ea0320 0x558057e99fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4b5d9b700 1 -- 172.21.15.70:0/672630965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057e9a8a0 msgr2=0x558057ea4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:12.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4b5d9b700 1 --2- 172.21.15.70:0/672630965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057e9a8a0 0x558057ea4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4b5d9b700 1 -- 172.21.15.70:0/672630965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558057e96df0 con 0x558057e9b260 2024-06-04T04:58:12.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.690+0000 7fb4b5d9b700 1 --2- 172.21.15.70:0/672630965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057e95270 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac015b30 tx=0x7fb4ac006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fc2a10486a8df26 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:12.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.691+0000 7fb49ffff700 1 -- 172.21.15.70:0/672630965 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac0129c0 con 0x558057e9b260 2024-06-04T04:58:12.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.691+0000 7fb49ffff700 1 -- 172.21.15.70:0/672630965 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb4ac025070 con 0x558057e9b260 2024-06-04T04:58:12.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.691+0000 7fb49ffff700 1 -- 172.21.15.70:0/672630965 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac020c20 con 0x558057e9b260 2024-06-04T04:58:12.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.691+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/672630965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 msgr2=0x558057e95270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:12.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.691+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/672630965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057e95270 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac015b30 tx=0x7fb4ac006f90 comp rx=0 tx=0).stop 2024-06-04T04:58:12.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.691+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/672630965 shutdown_connections 2024-06-04T04:58:12.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.691+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/672630965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057ea0320 0x558057e99fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.691+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/672630965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057e9a8a0 0x558057ea4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.691+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/672630965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057e95270 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.691+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/672630965 >> 172.21.15.70:0/672630965 conn(0x558057dc6240 msgr2=0x558057e99890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:12.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.691+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/672630965 shutdown_connections 2024-06-04T04:58:12.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.691+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/672630965 wait complete. 2024-06-04T04:58:12.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4c71d0b80 1 -- start start 2024-06-04T04:58:12.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4c71d0b80 1 Processor -- start 2024-06-04T04:58:12.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4c71d0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057e9a8a0 0x558057ebef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4c71d0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057ebf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4c71d0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057ea0320 0x558057ebf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4c71d0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b81acf90 con 0x558057e9b260 2024-06-04T04:58:12.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4c71d0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b81ac8c0 con 0x558057ea0320 2024-06-04T04:58:12.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4c71d0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b81a9ac0 con 0x558057e9a8a0 2024-06-04T04:58:12.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4c71d0b80 10 client.0 _ll_register_callbacks cb 0x558057d83950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:12.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4b5d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057ebf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:12.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4b5d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057ebf460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55896/0 (socket says 172.21.15.70:55896) 2024-06-04T04:58:12.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4b5d9b700 1 -- 172.21.15.70:0/2834972104 learned_addr learned my addr 172.21.15.70:0/2834972104 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:12.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4b659c700 1 --2- 172.21.15.70:0/2834972104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057e9a8a0 0x558057ebef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:12.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4b6d9d700 1 --2- 172.21.15.70:0/2834972104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057ea0320 0x558057ebf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:12.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4b5d9b700 1 -- 172.21.15.70:0/2834972104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057e9a8a0 msgr2=0x558057ebef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:12.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4b5d9b700 1 --2- 172.21.15.70:0/2834972104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057e9a8a0 0x558057ebef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4b5d9b700 1 -- 172.21.15.70:0/2834972104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057ea0320 msgr2=0x558057ebf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:12.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4b5d9b700 1 --2- 172.21.15.70:0/2834972104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057ea0320 0x558057ebf9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4b5d9b700 1 -- 172.21.15.70:0/2834972104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558057eafd70 con 0x558057e9b260 2024-06-04T04:58:12.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb4b5d9b700 1 --2- 172.21.15.70:0/2834972104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057ebf460 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac020840 tx=0x7fb4ac020710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4946e601763292b2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:12.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.692+0000 7fb49dffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:12.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.693+0000 7fb49dffb700 1 -- 172.21.15.70:0/2834972104 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac02d030 con 0x558057e9b260 2024-06-04T04:58:12.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.693+0000 7fb49dffb700 1 -- 172.21.15.70:0/2834972104 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb4ac0129d0 con 0x558057e9b260 2024-06-04T04:58:12.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.693+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/2834972104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4ac025070 con 0x558057e9b260 2024-06-04T04:58:12.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.693+0000 7fb49dffb700 1 -- 172.21.15.70:0/2834972104 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb4ac012bd0 con 0x558057e9b260 2024-06-04T04:58:12.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.693+0000 7fb4c71d0b80 10 client.15669 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:12.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.693+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/2834972104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558057eb1080 con 0x558057e9b260 2024-06-04T04:58:12.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.693+0000 7fb49dffb700 1 -- 172.21.15.70:0/2834972104 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb4ac025070 con 0x558057e9b260 2024-06-04T04:58:12.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.693+0000 7fb49dffb700 1 -- 172.21.15.70:0/2834972104 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fb4ac03ac30 con 0x558057e9b260 2024-06-04T04:58:12.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.693+0000 7fb4c71d0b80 10 client.15669 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.693+0000 7fb4c71d0b80 10 client.15669 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T04:58:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.693+0000 7fb4c71d0b80 20 client.15669 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.693+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/2834972104 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x558057f22e00 0x558057f251f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.693+0000 7fb4c71d0b80 4 client.15669 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T04:58:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.693+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/2834972104 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558057df44a0 con 0x558057f22e00 2024-06-04T04:58:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.693+0000 7fb4b659c700 1 --2- 172.21.15.70:0/2834972104 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x558057f22e00 0x558057f251f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.694+0000 7fb4b659c700 1 --2- 172.21.15.70:0/2834972104 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x558057f22e00 0x558057f251f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=20f20fee642009e3 server_cookie=928e2429999a125e in_seq=0 out_seq=0 2024-06-04T04:58:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.694+0000 7fb49dffb700 10 client.15669 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T04:58:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.694+0000 7fb49dffb700 1 -- 172.21.15.70:0/2834972104 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558057df44a0 con 0x558057f22e00 2024-06-04T04:58:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.694+0000 7fb49dffb700 10 client.15669 handle_command_reply: tid=0 2024-06-04T04:58:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.694+0000 7fb4c71d0b80 1 client.15669 shutdown 2024-06-04T04:58:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.694+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/2834972104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 msgr2=0x558057ebf460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.694+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/2834972104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057ebf460 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac020840 tx=0x7fb4ac020710 comp rx=0 tx=0).stop 2024-06-04T04:58:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.694+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/2834972104 shutdown_connections 2024-06-04T04:58:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.694+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/2834972104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558057ea0320 0x558057ebf9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.694+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/2834972104 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x558057f22e00 0x558057f251f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.694+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/2834972104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558057e9a8a0 0x558057ebef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.694+0000 7fb4c71d0b80 1 --2- 172.21.15.70:0/2834972104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558057e9b260 0x558057ebf460 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.694+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/2834972104 >> 172.21.15.70:0/2834972104 conn(0x558057dc6240 msgr2=0x558057e9a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.694+0000 7fb49dffb700 0 client.15669 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T04:58:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.694+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/2834972104 shutdown_connections 2024-06-04T04:58:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.695+0000 7fb4c71d0b80 1 -- 172.21.15.70:0/2834972104 wait complete. 2024-06-04T04:58:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.695+0000 7fb4c71d0b80 20 client.15669 trim_cache size 0 max 16384 2024-06-04T04:58:12.703 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T04:58:12.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.695+0000 7fb4bc93b700 1 -- 172.21.15.70:0/2648967084 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb4a00b2b90 msgr2=0x7fb4a00b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:12.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.695+0000 7fb4bc93b700 1 --2- 172.21.15.70:0/2648967084 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb4a00b2b90 0x7fb4a00b5050 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8101010 tx=0x7fb4ac006d20 comp rx=0 tx=0).stop 2024-06-04T04:58:12.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.695+0000 7fb4bc93b700 1 -- 172.21.15.70:0/2648967084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4b8102ba0 msgr2=0x7fb4b8199e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:12.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.695+0000 7fb4bc93b700 1 --2- 172.21.15.70:0/2648967084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4b8102ba0 0x7fb4b8199e50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8000c00 tx=0x7fb4a80047b0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.695+0000 7fb4bc93b700 1 -- 172.21.15.70:0/2648967084 shutdown_connections 2024-06-04T04:58:12.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.695+0000 7fb4bc93b700 1 --2- 172.21.15.70:0/2648967084 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb4a00b2b90 0x7fb4a00b5050 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.695+0000 7fb4bc93b700 1 --2- 172.21.15.70:0/2648967084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4b8108710 0x7fb4b819e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.695+0000 7fb4bc93b700 1 --2- 172.21.15.70:0/2648967084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4b8102ba0 0x7fb4b8199e50 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.695+0000 7fb4bc93b700 1 --2- 172.21.15.70:0/2648967084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4b8103560 0x7fb4b819a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:12.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.695+0000 7fb4bc93b700 1 -- 172.21.15.70:0/2648967084 >> 172.21.15.70:0/2648967084 conn(0x7fb4b8078a40 msgr2=0x7fb4b8101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:12.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.695+0000 7fb4bc93b700 1 -- 172.21.15.70:0/2648967084 shutdown_connections 2024-06-04T04:58:12.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:12.695+0000 7fb4bc93b700 1 -- 172.21.15.70:0/2648967084 wait complete. 2024-06-04T04:58:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:13 smithi070 ceph-mon[32631]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-06-04T04:58:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:13 smithi112 ceph-mon[36453]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-06-04T04:58:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:13 smithi098 ceph-mon[35569]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-06-04T04:58:13.578 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T04:58:13.819 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:15.307 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:15 smithi070 ceph-mon[32631]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 10 op/s 2024-06-04T04:58:15.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:15 smithi112 ceph-mon[36453]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 10 op/s 2024-06-04T04:58:15.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:15 smithi098 ceph-mon[35569]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 10 op/s 2024-06-04T04:58:15.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.547+0000 7f6f25217700 1 -- 172.21.15.70:0/602089184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f20102e90 msgr2=0x7f6f2010f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:15.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.547+0000 7f6f25217700 1 --2- 172.21.15.70:0/602089184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f20102e90 0x7f6f2010f310 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800b0a0 tx=0x7f6f1800b3b0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.547+0000 7f6f25217700 1 -- 172.21.15.70:0/602089184 shutdown_connections 2024-06-04T04:58:15.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.547+0000 7f6f25217700 1 --2- 172.21.15.70:0/602089184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f20102e90 0x7f6f2010f310 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.547+0000 7f6f25217700 1 --2- 172.21.15.70:0/602089184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f201084d0 0x7f6f201088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.547+0000 7f6f25217700 1 --2- 172.21.15.70:0/602089184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f201024d0 0x7f6f20102950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.547+0000 7f6f25217700 1 -- 172.21.15.70:0/602089184 >> 172.21.15.70:0/602089184 conn(0x7f6f20078490 msgr2=0x7f6f200788a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:15.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.548+0000 7f6f25217700 1 -- 172.21.15.70:0/602089184 shutdown_connections 2024-06-04T04:58:15.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.548+0000 7f6f25217700 1 -- 172.21.15.70:0/602089184 wait complete. 2024-06-04T04:58:15.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.548+0000 7f6f25217700 1 Processor -- start 2024-06-04T04:58:15.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f25217700 1 -- start start 2024-06-04T04:58:15.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f25217700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f201024d0 0x7f6f2019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f25217700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f20102e90 0x7f6f2019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f25217700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f201084d0 0x7f6f20198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f25217700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6f20112490 con 0x7f6f20102e90 2024-06-04T04:58:15.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f25217700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6f20112310 con 0x7f6f201024d0 2024-06-04T04:58:15.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f25217700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6f20112610 con 0x7f6f201084d0 2024-06-04T04:58:15.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f1e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f20102e90 0x7f6f2019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:15.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f1e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f20102e90 0x7f6f2019ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55906/0 (socket says 172.21.15.70:55906) 2024-06-04T04:58:15.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f1ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f201024d0 0x7f6f2019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:15.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f1e59c700 1 -- 172.21.15.70:0/2241104190 learned_addr learned my addr 172.21.15.70:0/2241104190 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:15.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f1f59e700 1 --2- 172.21.15.70:0/2241104190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f201084d0 0x7f6f20198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:15.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f1e59c700 1 -- 172.21.15.70:0/2241104190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f201084d0 msgr2=0x7f6f20198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:15.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f1e59c700 1 --2- 172.21.15.70:0/2241104190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f201084d0 0x7f6f20198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f1e59c700 1 -- 172.21.15.70:0/2241104190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f201024d0 msgr2=0x7f6f2019e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:15.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f1e59c700 1 --2- 172.21.15.70:0/2241104190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f201024d0 0x7f6f2019e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f1e59c700 1 -- 172.21.15.70:0/2241104190 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f20198ff0 con 0x7f6f20102e90 2024-06-04T04:58:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.549+0000 7f6f1f59e700 1 --2- 172.21.15.70:0/2241104190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f201084d0 0x7f6f20198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.550+0000 7f6f1e59c700 1 --2- 172.21.15.70:0/2241104190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f20102e90 0x7f6f2019ebb0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400c920 tx=0x7f6f1400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.550+0000 7f6f07fff700 1 -- 172.21.15.70:0/2241104190 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f14007b00 con 0x7f6f20102e90 2024-06-04T04:58:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.550+0000 7f6f07fff700 1 -- 172.21.15.70:0/2241104190 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f14004cc0 con 0x7f6f20102e90 2024-06-04T04:58:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.550+0000 7f6f25217700 1 -- 172.21.15.70:0/2241104190 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f201992e0 con 0x7f6f20102e90 2024-06-04T04:58:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.550+0000 7f6f07fff700 1 -- 172.21.15.70:0/2241104190 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f14005440 con 0x7f6f20102e90 2024-06-04T04:58:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.550+0000 7f6f25217700 1 -- 172.21.15.70:0/2241104190 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f2006be80 con 0x7f6f20102e90 2024-06-04T04:58:15.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.552+0000 7f6f07fff700 1 -- 172.21.15.70:0/2241104190 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 149885+0+0 (secure 0 0 0) 0x7f6f14004750 con 0x7f6f20102e90 2024-06-04T04:58:15.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.553+0000 7f6f07fff700 1 --2- 172.21.15.70:0/2241104190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6f080b2b10 0x7f6f080b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.553+0000 7f6f1ed9d700 1 --2- 172.21.15.70:0/2241104190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6f080b2b10 0x7f6f080b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:15.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.553+0000 7f6f07fff700 1 -- 172.21.15.70:0/2241104190 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f6f140e2460 con 0x7f6f20102e90 2024-06-04T04:58:15.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.553+0000 7f6f2faacb80 1 Processor -- start 2024-06-04T04:58:15.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.553+0000 7f6f2faacb80 1 -- start start 2024-06-04T04:58:15.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.553+0000 7f6f1ed9d700 1 --2- 172.21.15.70:0/2241104190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6f080b2b10 0x7f6f080b4fd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6f1000abd0 tx=0x7f6f1000a380 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:15.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.553+0000 7f6f2faacb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573df3320 0x556573df3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.553+0000 7f6f2faacb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573ded8a0 0x556573dedd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f2faacb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573dfa080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f2faacb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556573d4bd70 con 0x556573dee260 2024-06-04T04:58:15.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f2faacb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556573d3b910 con 0x556573df3320 2024-06-04T04:58:15.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f2faacb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556573d3f950 con 0x556573ded8a0 2024-06-04T04:58:15.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f1f59e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573dfa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:15.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f1f59e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573dfa080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55922/0 (socket says 172.21.15.70:55922) 2024-06-04T04:58:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f1f59e700 1 -- 172.21.15.70:0/23116403 learned_addr learned my addr 172.21.15.70:0/23116403 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f1e59c700 1 --2- 172.21.15.70:0/23116403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573ded8a0 0x556573dedd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f1ed9d700 1 --2- 172.21.15.70:0/23116403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573df3320 0x556573df3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f1f59e700 1 -- 172.21.15.70:0/23116403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573ded8a0 msgr2=0x556573dedd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f1f59e700 1 --2- 172.21.15.70:0/23116403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573ded8a0 0x556573dedd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f1f59e700 1 -- 172.21.15.70:0/23116403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573df3320 msgr2=0x556573df3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f1f59e700 1 --2- 172.21.15.70:0/23116403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573df3320 0x556573df3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f1f59e700 1 -- 172.21.15.70:0/23116403 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556573ce4fb0 con 0x556573dee260 2024-06-04T04:58:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f1f59e700 1 --2- 172.21.15.70:0/23116403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573dfa080 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800b830 tx=0x7f6f18009300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c38e11ed3a40ce96 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f057fa700 1 -- 172.21.15.70:0/23116403 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18022070 con 0x556573dee260 2024-06-04T04:58:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f057fa700 1 -- 172.21.15.70:0/23116403 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f18004c50 con 0x556573dee260 2024-06-04T04:58:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.554+0000 7f6f057fa700 1 -- 172.21.15.70:0/23116403 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1801c6a0 con 0x556573dee260 2024-06-04T04:58:15.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.555+0000 7f6f2faacb80 1 -- 172.21.15.70:0/23116403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 msgr2=0x556573dfa080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:15.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.555+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/23116403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573dfa080 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800b830 tx=0x7f6f18009300 comp rx=0 tx=0).stop 2024-06-04T04:58:15.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.555+0000 7f6f2faacb80 1 -- 172.21.15.70:0/23116403 shutdown_connections 2024-06-04T04:58:15.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.555+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/23116403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573dfa080 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.555+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/23116403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573df3320 0x556573df3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.555+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/23116403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573ded8a0 0x556573dedd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.555+0000 7f6f2faacb80 1 -- 172.21.15.70:0/23116403 >> 172.21.15.70:0/23116403 conn(0x556573d19240 msgr2=0x556573de83e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:15.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.555+0000 7f6f2faacb80 1 -- 172.21.15.70:0/23116403 shutdown_connections 2024-06-04T04:58:15.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.555+0000 7f6f2faacb80 1 -- 172.21.15.70:0/23116403 wait complete. 2024-06-04T04:58:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f2faacb80 1 -- start start 2024-06-04T04:58:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f2faacb80 1 Processor -- start 2024-06-04T04:58:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f2faacb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573ded8a0 0x556573df9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f2faacb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573dee260 0x556573df6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f2faacb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573df3320 0x556573df74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f2faacb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556573c41fa0 con 0x556573df3320 2024-06-04T04:58:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f2faacb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556573ca3da0 con 0x556573dee260 2024-06-04T04:58:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f2faacb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556573dfa680 con 0x556573ded8a0 2024-06-04T04:58:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f2faacb80 10 client.0 _ll_register_callbacks cb 0x556573cd6950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f1f59e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573df3320 0x556573df74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f1f59e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573df3320 0x556573df74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55936/0 (socket says 172.21.15.70:55936) 2024-06-04T04:58:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f1f59e700 1 -- 172.21.15.70:0/2201442827 learned_addr learned my addr 172.21.15.70:0/2201442827 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f1e59c700 1 --2- 172.21.15.70:0/2201442827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573dee260 0x556573df6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f1ed9d700 1 --2- 172.21.15.70:0/2201442827 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573ded8a0 0x556573df9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f1f59e700 1 -- 172.21.15.70:0/2201442827 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573ded8a0 msgr2=0x556573df9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f1f59e700 1 --2- 172.21.15.70:0/2201442827 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573ded8a0 0x556573df9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f1f59e700 1 -- 172.21.15.70:0/2201442827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573dee260 msgr2=0x556573df6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f1f59e700 1 --2- 172.21.15.70:0/2201442827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573dee260 0x556573df6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f1f59e700 1 -- 172.21.15.70:0/2201442827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556573c8b8d0 con 0x556573df3320 2024-06-04T04:58:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.556+0000 7f6f1ed9d700 1 --2- 172.21.15.70:0/2201442827 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573ded8a0 0x556573df9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.557+0000 7f6f1f59e700 1 --2- 172.21.15.70:0/2201442827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573df3320 0x556573df74f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6f180095d0 tx=0x7f6f1801c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d2af96aac37a3c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.557+0000 7f6f02ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.557+0000 7f6f02ff5700 1 -- 172.21.15.70:0/2201442827 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1802a030 con 0x556573df3320 2024-06-04T04:58:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.557+0000 7f6f02ff5700 1 -- 172.21.15.70:0/2201442827 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f6f18004a50 con 0x556573df3320 2024-06-04T04:58:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.557+0000 7f6f2faacb80 1 -- 172.21.15.70:0/2201442827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f18004c50 con 0x556573df3320 2024-06-04T04:58:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.557+0000 7f6f02ff5700 1 -- 172.21.15.70:0/2201442827 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6f18036050 con 0x556573df3320 2024-06-04T04:58:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.557+0000 7f6f2faacb80 10 client.15681 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.557+0000 7f6f2faacb80 1 -- 172.21.15.70:0/2201442827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556573cf5760 con 0x556573df3320 2024-06-04T04:58:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.557+0000 7f6f02ff5700 1 -- 172.21.15.70:0/2201442827 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f18004c50 con 0x556573df3320 2024-06-04T04:58:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.557+0000 7f6f02ff5700 1 -- 172.21.15.70:0/2201442827 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f6f18011070 con 0x556573df3320 2024-06-04T04:58:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.557+0000 7f6f2faacb80 10 client.15681 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.557+0000 7f6f2faacb80 10 client.15681 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T04:58:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.557+0000 7f6f2faacb80 20 client.15681 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.557+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/2201442827 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556573e06fe0 0x556573e093d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.557+0000 7f6f2faacb80 4 client.15681 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.557+0000 7f6f2faacb80 1 -- 172.21.15.70:0/2201442827 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556573d4dcc0 con 0x556573e06fe0 2024-06-04T04:58:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.558+0000 7f6f1ed9d700 1 --2- 172.21.15.70:0/2201442827 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556573e06fe0 0x556573e093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.558+0000 7f6f02ff5700 10 client.15681 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T04:58:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.558+0000 7f6f1ed9d700 1 --2- 172.21.15.70:0/2201442827 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556573e06fe0 0x556573e093d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=aac32613b8e7b87c server_cookie=fec5a332c7a81d36 in_seq=0 out_seq=0 2024-06-04T04:58:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.559+0000 7f6f02ff5700 1 -- 172.21.15.70:0/2201442827 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556573d4dcc0 con 0x556573e06fe0 2024-06-04T04:58:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.559+0000 7f6f02ff5700 10 client.15681 handle_command_reply: tid=0 2024-06-04T04:58:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.559+0000 7f6f2faacb80 1 client.15681 shutdown 2024-06-04T04:58:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.559+0000 7f6f2faacb80 1 -- 172.21.15.70:0/2201442827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573df3320 msgr2=0x556573df74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.559+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/2201442827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573df3320 0x556573df74f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6f180095d0 tx=0x7f6f1801c6f0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.559+0000 7f6f2faacb80 1 -- 172.21.15.70:0/2201442827 shutdown_connections 2024-06-04T04:58:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.559+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/2201442827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573df3320 0x556573df74f0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.559+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/2201442827 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556573e06fe0 0x556573e093d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.559+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/2201442827 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573ded8a0 0x556573df9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.559+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/2201442827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573dee260 0x556573df6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.559+0000 7f6f2faacb80 1 -- 172.21.15.70:0/2201442827 >> 172.21.15.70:0/2201442827 conn(0x556573d19240 msgr2=0x556573de7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.559+0000 7f6f02ff5700 0 client.15681 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T04:58:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.559+0000 7f6f2faacb80 1 -- 172.21.15.70:0/2201442827 shutdown_connections 2024-06-04T04:58:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.559+0000 7f6f2faacb80 1 -- 172.21.15.70:0/2201442827 wait complete. 2024-06-04T04:58:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.559+0000 7f6f2faacb80 20 client.15681 trim_cache size 0 max 16384 2024-06-04T04:58:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f2faacb80 1 Processor -- start 2024-06-04T04:58:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f2faacb80 1 -- start start 2024-06-04T04:58:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f2faacb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573ded8a0 0x556573df2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f2faacb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573dedc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f2faacb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573df3320 0x556573de8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f2faacb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6f201afe50 con 0x556573dee260 2024-06-04T04:58:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f2faacb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6f201b1990 con 0x556573df3320 2024-06-04T04:58:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f2faacb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6f201b0520 con 0x556573ded8a0 2024-06-04T04:58:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f1e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573dedc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f1e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573dedc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55940/0 (socket says 172.21.15.70:55940) 2024-06-04T04:58:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f1e59c700 1 -- 172.21.15.70:0/119502878 learned_addr learned my addr 172.21.15.70:0/119502878 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f1e59c700 1 -- 172.21.15.70:0/119502878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573ded8a0 msgr2=0x556573df2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f1f59e700 1 --2- 172.21.15.70:0/119502878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573df3320 0x556573de8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f1e59c700 1 --2- 172.21.15.70:0/119502878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573ded8a0 0x556573df2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f1e59c700 1 -- 172.21.15.70:0/119502878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573df3320 msgr2=0x556573de8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f1e59c700 1 --2- 172.21.15.70:0/119502878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573df3320 0x556573de8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f1e59c700 1 -- 172.21.15.70:0/119502878 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556573dfa020 con 0x556573dee260 2024-06-04T04:58:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.578+0000 7f6f1e59c700 1 --2- 172.21.15.70:0/119502878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573dedc80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6f140a5570 tx=0x7f6f140a93f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2157606759d3bd5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.579+0000 7f6f04ff9700 1 -- 172.21.15.70:0/119502878 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1401d040 con 0x556573dee260 2024-06-04T04:58:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.579+0000 7f6f04ff9700 1 -- 172.21.15.70:0/119502878 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f140a9bf0 con 0x556573dee260 2024-06-04T04:58:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.579+0000 7f6f04ff9700 1 -- 172.21.15.70:0/119502878 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f140a9750 con 0x556573dee260 2024-06-04T04:58:15.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.579+0000 7f6f2faacb80 1 -- 172.21.15.70:0/119502878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 msgr2=0x556573dedc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:15.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.579+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/119502878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573dedc80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6f140a5570 tx=0x7f6f140a93f0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.579+0000 7f6f2faacb80 1 -- 172.21.15.70:0/119502878 shutdown_connections 2024-06-04T04:58:15.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.579+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/119502878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573df3320 0x556573de8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.579+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/119502878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573ded8a0 0x556573df2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.579+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/119502878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573dedc80 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.579+0000 7f6f2faacb80 1 -- 172.21.15.70:0/119502878 >> 172.21.15.70:0/119502878 conn(0x556573d19240 msgr2=0x556573de4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:15.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.579+0000 7f6f2faacb80 1 -- 172.21.15.70:0/119502878 shutdown_connections 2024-06-04T04:58:15.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.579+0000 7f6f2faacb80 1 -- 172.21.15.70:0/119502878 wait complete. 2024-06-04T04:58:15.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f2faacb80 1 -- start start 2024-06-04T04:58:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f2faacb80 1 Processor -- start 2024-06-04T04:58:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f2faacb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573ded8a0 0x556573e68b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f2faacb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573e690b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f2faacb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573df3320 0x556573e09970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f2faacb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6f201b12c0 con 0x556573dee260 2024-06-04T04:58:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f2faacb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6f201b0bf0 con 0x556573ded8a0 2024-06-04T04:58:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f2faacb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6f201addf0 con 0x556573df3320 2024-06-04T04:58:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f2faacb80 10 client.0 _ll_register_callbacks cb 0x556573cd6950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f1e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573e690b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f1e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573e690b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55942/0 (socket says 172.21.15.70:55942) 2024-06-04T04:58:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f1e59c700 1 -- 172.21.15.70:0/3938433155 learned_addr learned my addr 172.21.15.70:0/3938433155 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f1ed9d700 1 --2- 172.21.15.70:0/3938433155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573ded8a0 0x556573e68b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f1e59c700 1 -- 172.21.15.70:0/3938433155 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573df3320 msgr2=0x556573e09970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f1e59c700 1 --2- 172.21.15.70:0/3938433155 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573df3320 0x556573e09970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f1e59c700 1 -- 172.21.15.70:0/3938433155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573ded8a0 msgr2=0x556573e68b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f1e59c700 1 --2- 172.21.15.70:0/3938433155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573ded8a0 0x556573e68b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f1e59c700 1 -- 172.21.15.70:0/3938433155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556573dea380 con 0x556573dee260 2024-06-04T04:58:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f1ed9d700 1 --2- 172.21.15.70:0/3938433155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573ded8a0 0x556573e68b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f1e59c700 1 --2- 172.21.15.70:0/3938433155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573e690b0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f6f140a91f0 tx=0x7f6f140b2400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709419f7d41b2569 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.580+0000 7f6f02ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.581+0000 7f6f02ff5700 1 -- 172.21.15.70:0/3938433155 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1401d040 con 0x556573dee260 2024-06-04T04:58:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.581+0000 7f6f02ff5700 1 -- 172.21.15.70:0/3938433155 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f6f140b1ec0 con 0x556573dee260 2024-06-04T04:58:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.581+0000 7f6f2faacb80 1 -- 172.21.15.70:0/3938433155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f140a9bf0 con 0x556573dee260 2024-06-04T04:58:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.581+0000 7f6f02ff5700 1 -- 172.21.15.70:0/3938433155 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6f140b20c0 con 0x556573dee260 2024-06-04T04:58:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.581+0000 7f6f2faacb80 10 client.15687 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.581+0000 7f6f2faacb80 1 -- 172.21.15.70:0/3938433155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556573e71190 con 0x556573dee260 2024-06-04T04:58:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.581+0000 7f6f02ff5700 1 -- 172.21.15.70:0/3938433155 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f140a9bf0 con 0x556573dee260 2024-06-04T04:58:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.581+0000 7f6f02ff5700 1 -- 172.21.15.70:0/3938433155 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f6f140dfd10 con 0x556573dee260 2024-06-04T04:58:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.581+0000 7f6f2faacb80 10 client.15687 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.581+0000 7f6f2faacb80 10 client.15687 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T04:58:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.581+0000 7f6f2faacb80 20 client.15687 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.581+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/3938433155 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556573e717e0 0x556573e73bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.581+0000 7f6f2faacb80 4 client.15687 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T04:58:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.581+0000 7f6f2faacb80 1 -- 172.21.15.70:0/3938433155 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556573d474a0 con 0x556573e717e0 2024-06-04T04:58:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.581+0000 7f6f1ed9d700 1 --2- 172.21.15.70:0/3938433155 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556573e717e0 0x556573e73bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.582+0000 7f6f1ed9d700 1 --2- 172.21.15.70:0/3938433155 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556573e717e0 0x556573e73bd0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a8d6fa11501db64a server_cookie=6661e2592f20a9fd in_seq=0 out_seq=0 2024-06-04T04:58:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.582+0000 7f6f02ff5700 10 client.15687 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T04:58:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.582+0000 7f6f02ff5700 1 -- 172.21.15.70:0/3938433155 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556573d474a0 con 0x556573e717e0 2024-06-04T04:58:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.582+0000 7f6f02ff5700 10 client.15687 handle_command_reply: tid=0 2024-06-04T04:58:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.582+0000 7f6f2faacb80 1 client.15687 shutdown 2024-06-04T04:58:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.582+0000 7f6f2faacb80 1 -- 172.21.15.70:0/3938433155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 msgr2=0x556573e690b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.582+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/3938433155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573e690b0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f6f140a91f0 tx=0x7f6f140b2400 comp rx=0 tx=0).stop 2024-06-04T04:58:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.582+0000 7f6f2faacb80 1 -- 172.21.15.70:0/3938433155 shutdown_connections 2024-06-04T04:58:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.582+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/3938433155 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556573e717e0 0x556573e73bd0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.582+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/3938433155 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556573df3320 0x556573e09970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.582+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/3938433155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573ded8a0 0x556573e68b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.582+0000 7f6f02ff5700 0 client.15687 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T04:58:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.582+0000 7f6f2faacb80 1 --2- 172.21.15.70:0/3938433155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556573dee260 0x556573e690b0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.582+0000 7f6f2faacb80 1 -- 172.21.15.70:0/3938433155 >> 172.21.15.70:0/3938433155 conn(0x556573d19240 msgr2=0x556573df0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:15.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.582+0000 7f6f2faacb80 1 -- 172.21.15.70:0/3938433155 shutdown_connections 2024-06-04T04:58:15.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.582+0000 7f6f2faacb80 1 -- 172.21.15.70:0/3938433155 wait complete. 2024-06-04T04:58:15.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.582+0000 7f6f2faacb80 20 client.15687 trim_cache size 0 max 16384 2024-06-04T04:58:15.591 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T04:58:15.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.583+0000 7f6f25217700 1 -- 172.21.15.70:0/2241104190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6f080b2b10 msgr2=0x7f6f080b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:15.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.583+0000 7f6f25217700 1 --2- 172.21.15.70:0/2241104190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6f080b2b10 0x7f6f080b4fd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6f1000abd0 tx=0x7f6f1000a380 comp rx=0 tx=0).stop 2024-06-04T04:58:15.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.583+0000 7f6f25217700 1 -- 172.21.15.70:0/2241104190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f20102e90 msgr2=0x7f6f2019ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:15.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.583+0000 7f6f25217700 1 --2- 172.21.15.70:0/2241104190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f20102e90 0x7f6f2019ebb0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400c920 tx=0x7f6f1400cc30 comp rx=0 tx=0).stop 2024-06-04T04:58:15.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.583+0000 7f6f25217700 1 -- 172.21.15.70:0/2241104190 shutdown_connections 2024-06-04T04:58:15.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.583+0000 7f6f25217700 1 --2- 172.21.15.70:0/2241104190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6f080b2b10 0x7f6f080b4fd0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.583+0000 7f6f25217700 1 --2- 172.21.15.70:0/2241104190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f201084d0 0x7f6f20198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.583+0000 7f6f25217700 1 --2- 172.21.15.70:0/2241104190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f201024d0 0x7f6f2019e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.583+0000 7f6f25217700 1 --2- 172.21.15.70:0/2241104190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f20102e90 0x7f6f2019ebb0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:15.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.583+0000 7f6f25217700 1 -- 172.21.15.70:0/2241104190 >> 172.21.15.70:0/2241104190 conn(0x7f6f20078490 msgr2=0x7f6f20100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:15.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.583+0000 7f6f25217700 1 -- 172.21.15.70:0/2241104190 shutdown_connections 2024-06-04T04:58:15.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:15.583+0000 7f6f25217700 1 -- 172.21.15.70:0/2241104190 wait complete. 2024-06-04T04:58:16.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.015+0000 7fe10a10a640 1 -- 172.21.15.70:0/136958154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe104101980 msgr2=0x7fe104101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:16.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.015+0000 7fe10a10a640 1 --2- 172.21.15.70:0/136958154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe104101980 0x7fe104101e00 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009930 tx=0x7fe0ec02a420 comp rx=0 tx=0).stop 2024-06-04T04:58:16.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.015+0000 7fe10a10a640 1 -- 172.21.15.70:0/136958154 shutdown_connections 2024-06-04T04:58:16.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.015+0000 7fe10a10a640 1 --2- 172.21.15.70:0/136958154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe104102340 0x7fe10410e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.015+0000 7fe10a10a640 1 --2- 172.21.15.70:0/136958154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe104101980 0x7fe104101e00 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.015+0000 7fe10a10a640 1 --2- 172.21.15.70:0/136958154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe104107940 0x7fe104107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.015+0000 7fe10a10a640 1 -- 172.21.15.70:0/136958154 >> 172.21.15.70:0/136958154 conn(0x7fe104078ad0 msgr2=0x7fe104078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.015+0000 7fe10a10a640 1 -- 172.21.15.70:0/136958154 shutdown_connections 2024-06-04T04:58:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.015+0000 7fe10a10a640 1 -- 172.21.15.70:0/136958154 wait complete. 2024-06-04T04:58:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe10a10a640 1 Processor -- start 2024-06-04T04:58:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe10a10a640 1 -- start start 2024-06-04T04:58:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe10a10a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe104101980 0x7fe104198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe10a10a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe104102340 0x7fe104198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe10a10a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe104107940 0x7fe10419c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe10a10a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe104111a80 con 0x7fe104101980 2024-06-04T04:58:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe10a10a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe104111900 con 0x7fe104102340 2024-06-04T04:58:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe10a10a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe104111c00 con 0x7fe104107940 2024-06-04T04:58:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe1037fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe104101980 0x7fe104198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe1037fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe104101980 0x7fe104198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45938/0 (socket says 172.21.15.70:45938) 2024-06-04T04:58:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe1037fe640 1 -- 172.21.15.70:0/2844989909 learned_addr learned my addr 172.21.15.70:0/2844989909 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe0fbfff640 1 --2- 172.21.15.70:0/2844989909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe104102340 0x7fe104198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe1037fe640 1 -- 172.21.15.70:0/2844989909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe104107940 msgr2=0x7fe10419c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe1037fe640 1 --2- 172.21.15.70:0/2844989909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe104107940 0x7fe10419c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe1037fe640 1 -- 172.21.15.70:0/2844989909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe104102340 msgr2=0x7fe104198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe1037fe640 1 --2- 172.21.15.70:0/2844989909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe104102340 0x7fe104198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe1037fe640 1 -- 172.21.15.70:0/2844989909 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe10419cff0 con 0x7fe104101980 2024-06-04T04:58:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe0fbfff640 1 --2- 172.21.15.70:0/2844989909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe104102340 0x7fe104198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe1037fe640 1 --2- 172.21.15.70:0/2844989909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe104101980 0x7fe104198000 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fe0f400d950 tx=0x7fe0f400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.016+0000 7fe1017fa640 1 -- 172.21.15.70:0/2844989909 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f4017030 con 0x7fe104101980 2024-06-04T04:58:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.017+0000 7fe10a10a640 1 -- 172.21.15.70:0/2844989909 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe10419d2e0 con 0x7fe104101980 2024-06-04T04:58:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.017+0000 7fe1017fa640 1 -- 172.21.15.70:0/2844989909 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0f4004020 con 0x7fe104101980 2024-06-04T04:58:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.017+0000 7fe1017fa640 1 -- 172.21.15.70:0/2844989909 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f40055d0 con 0x7fe104101980 2024-06-04T04:58:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.017+0000 7fe10a10a640 1 -- 172.21.15.70:0/2844989909 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1041a4b70 con 0x7fe104101980 2024-06-04T04:58:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe10ce24740 1 Processor -- start 2024-06-04T04:58:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe1017fa640 1 -- 172.21.15.70:0/2844989909 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 149885+0+0 (secure 0 0 0) 0x7fe0f4009a30 con 0x7fe104101980 2024-06-04T04:58:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe10ce24740 1 -- start start 2024-06-04T04:58:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe10ce24740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912e3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe10ce24740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912e4140 0x5628912ee4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe10ce24740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912eea00 0x5628912f0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe10ce24740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562891255c30 con 0x5628912e3820 2024-06-04T04:58:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe10ce24740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5628912457d0 con 0x5628912e4140 2024-06-04T04:58:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe10ce24740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562891249810 con 0x5628912eea00 2024-06-04T04:58:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe103fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912e3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe103fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912e3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45946/0 (socket says 172.21.15.70:45946) 2024-06-04T04:58:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe103fff640 1 -- 172.21.15.70:0/3371871170 learned_addr learned my addr 172.21.15.70:0/3371871170 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe1037fe640 1 --2- 172.21.15.70:0/3371871170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912e4140 0x5628912ee4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe0fbfff640 1 --2- 172.21.15.70:0/3371871170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912eea00 0x5628912f0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe103fff640 1 -- 172.21.15.70:0/3371871170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912eea00 msgr2=0x5628912f0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe103fff640 1 --2- 172.21.15.70:0/3371871170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912eea00 0x5628912f0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe103fff640 1 -- 172.21.15.70:0/3371871170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912e4140 msgr2=0x5628912ee4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe103fff640 1 --2- 172.21.15.70:0/3371871170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912e4140 0x5628912ee4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe103fff640 1 -- 172.21.15.70:0/3371871170 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562890f5a210 con 0x5628912e3820 2024-06-04T04:58:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe0fbfff640 1 --2- 172.21.15.70:0/3371871170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912eea00 0x5628912f0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.020+0000 7fe1017fa640 1 --2- 172.21.15.70:0/2844989909 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe0f00b2b30 0x7fe0f00b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.021+0000 7fe103fff640 1 --2- 172.21.15.70:0/3371871170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912e3c00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009ea0 tx=0x7fe0fc009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5575eb738b938825 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.021+0000 7fe1037fe640 1 --2- 172.21.15.70:0/2844989909 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe0f00b2b30 0x7fe0f00b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.021+0000 7fe0f9ffb640 1 -- 172.21.15.70:0/3371871170 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc004d20 con 0x5628912e3820 2024-06-04T04:58:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.021+0000 7fe1017fa640 1 -- 172.21.15.70:0/2844989909 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fe0f40e22f0 con 0x7fe104101980 2024-06-04T04:58:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.021+0000 7fe0f9ffb640 1 -- 172.21.15.70:0/3371871170 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0fc00ee60 con 0x5628912e3820 2024-06-04T04:58:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.021+0000 7fe0f9ffb640 1 -- 172.21.15.70:0/3371871170 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc010670 con 0x5628912e3820 2024-06-04T04:58:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.021+0000 7fe10ce24740 1 -- 172.21.15.70:0/3371871170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 msgr2=0x5628912e3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.021+0000 7fe1037fe640 1 --2- 172.21.15.70:0/2844989909 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe0f00b2b30 0x7fe0f00b4ff0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe0f40df920 tx=0x7fe0f40e4040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.021+0000 7fe10ce24740 1 --2- 172.21.15.70:0/3371871170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912e3c00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009ea0 tx=0x7fe0fc009f90 comp rx=0 tx=0).stop 2024-06-04T04:58:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.021+0000 7fe10ce24740 1 -- 172.21.15.70:0/3371871170 shutdown_connections 2024-06-04T04:58:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.021+0000 7fe10ce24740 1 --2- 172.21.15.70:0/3371871170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912e4140 0x5628912ee4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.021+0000 7fe10ce24740 1 --2- 172.21.15.70:0/3371871170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912eea00 0x5628912f0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.021+0000 7fe10ce24740 1 --2- 172.21.15.70:0/3371871170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912e3c00 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.021+0000 7fe10ce24740 1 -- 172.21.15.70:0/3371871170 >> 172.21.15.70:0/3371871170 conn(0x5628912dfd30 msgr2=0x5628912e0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.022+0000 7fe10ce24740 1 -- 172.21.15.70:0/3371871170 shutdown_connections 2024-06-04T04:58:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.022+0000 7fe10ce24740 1 -- 172.21.15.70:0/3371871170 wait complete. 2024-06-04T04:58:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.022+0000 7fe10ce24740 1 -- start start 2024-06-04T04:58:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.022+0000 7fe10ce24740 1 Processor -- start 2024-06-04T04:58:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.022+0000 7fe10ce24740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912e3820 0x5628912eb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.022+0000 7fe10ce24740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912e4140 0x5628912ebee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.022+0000 7fe10ce24740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912eea00 0x5628912ec420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.022+0000 7fe10ce24740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562890f2b790 con 0x5628912eea00 2024-06-04T04:58:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.022+0000 7fe10ce24740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562891201890 con 0x5628912e4140 2024-06-04T04:58:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.022+0000 7fe10ce24740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5628912f1630 con 0x5628912e3820 2024-06-04T04:58:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.022+0000 7fe10ce24740 10 client.0 _ll_register_callbacks cb 0x562890f59a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.022+0000 7fe103fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912eea00 0x5628912ec420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.022+0000 7fe103fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912eea00 0x5628912ec420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45960/0 (socket says 172.21.15.70:45960) 2024-06-04T04:58:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.022+0000 7fe103fff640 1 -- 172.21.15.70:0/1186134878 learned_addr learned my addr 172.21.15.70:0/1186134878 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.022+0000 7fe0fbfff640 1 --2- 172.21.15.70:0/1186134878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912e4140 0x5628912ebee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe103fff640 1 --2- 172.21.15.70:0/1186134878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912e3820 0x5628912eb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe103fff640 1 -- 172.21.15.70:0/1186134878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912e3820 msgr2=0x5628912eb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe103fff640 1 --2- 172.21.15.70:0/1186134878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912e3820 0x5628912eb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe103fff640 1 -- 172.21.15.70:0/1186134878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912e4140 msgr2=0x5628912ebee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe103fff640 1 --2- 172.21.15.70:0/1186134878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912e4140 0x5628912ebee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe103fff640 1 -- 172.21.15.70:0/1186134878 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562891026620 con 0x5628912eea00 2024-06-04T04:58:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe103fff640 1 --2- 172.21.15.70:0/1186134878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912eea00 0x5628912ec420 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009560 tx=0x7fe0fc009ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01961e2446fa2bd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe0e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe0e2ffd640 1 -- 172.21.15.70:0/1186134878 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc01d040 con 0x5628912eea00 2024-06-04T04:58:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe0e2ffd640 1 -- 172.21.15.70:0/1186134878 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fe0fc010dd0 con 0x5628912eea00 2024-06-04T04:58:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe10ce24740 1 -- 172.21.15.70:0/1186134878 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0fc00ee60 con 0x5628912eea00 2024-06-04T04:58:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe0e2ffd640 1 -- 172.21.15.70:0/1186134878 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe0fc005000 con 0x5628912eea00 2024-06-04T04:58:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe10ce24740 10 client.15699 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe10ce24740 1 -- 172.21.15.70:0/1186134878 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5628912fd320 con 0x5628912eea00 2024-06-04T04:58:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe0e2ffd640 1 -- 172.21.15.70:0/1186134878 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0fc00ee60 con 0x5628912eea00 2024-06-04T04:58:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe0e2ffd640 1 -- 172.21.15.70:0/1186134878 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe0fc0288e0 con 0x5628912eea00 2024-06-04T04:58:16.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe10ce24740 10 client.15699 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:16.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.023+0000 7fe10ce24740 10 client.15699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:16.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.024+0000 7fe10ce24740 20 client.15699 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:16.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.024+0000 7fe10ce24740 1 --2- 172.21.15.70:0/1186134878 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5628912fe720 0x562891300b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.024+0000 7fe10ce24740 4 client.15699 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:16.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.024+0000 7fe10ce24740 1 -- 172.21.15.70:0/1186134878 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562891257b80 con 0x5628912fe720 2024-06-04T04:58:16.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.024+0000 7fe0fbfff640 1 --2- 172.21.15.70:0/1186134878 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5628912fe720 0x562891300b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:16.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.024+0000 7fe0e2ffd640 10 client.15699 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:16.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.024+0000 7fe0fbfff640 1 --2- 172.21.15.70:0/1186134878 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5628912fe720 0x562891300b10 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=986ba196a2cf628a server_cookie=fc6ccde1a4d1fe64 in_seq=0 out_seq=0 2024-06-04T04:58:16.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.025+0000 7fe0e2ffd640 1 -- 172.21.15.70:0/1186134878 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562891257b80 con 0x5628912fe720 2024-06-04T04:58:16.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.025+0000 7fe0e2ffd640 10 client.15699 handle_command_reply: tid=0 2024-06-04T04:58:16.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.025+0000 7fe10ce24740 1 client.15699 shutdown 2024-06-04T04:58:16.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.025+0000 7fe10ce24740 1 -- 172.21.15.70:0/1186134878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912eea00 msgr2=0x5628912ec420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:16.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.025+0000 7fe10ce24740 1 --2- 172.21.15.70:0/1186134878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912eea00 0x5628912ec420 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009560 tx=0x7fe0fc009ae0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.025+0000 7fe10ce24740 1 -- 172.21.15.70:0/1186134878 shutdown_connections 2024-06-04T04:58:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.025+0000 7fe10ce24740 1 --2- 172.21.15.70:0/1186134878 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5628912fe720 0x562891300b10 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.025+0000 7fe10ce24740 1 --2- 172.21.15.70:0/1186134878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912e4140 0x5628912ebee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.025+0000 7fe0e2ffd640 0 client.15699 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.025+0000 7fe10ce24740 1 --2- 172.21.15.70:0/1186134878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912eea00 0x5628912ec420 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.025+0000 7fe10ce24740 1 --2- 172.21.15.70:0/1186134878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912e3820 0x5628912eb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.025+0000 7fe10ce24740 1 -- 172.21.15.70:0/1186134878 >> 172.21.15.70:0/1186134878 conn(0x5628912dfd30 msgr2=0x5628912e5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.025+0000 7fe10ce24740 1 -- 172.21.15.70:0/1186134878 shutdown_connections 2024-06-04T04:58:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.025+0000 7fe10ce24740 1 -- 172.21.15.70:0/1186134878 wait complete. 2024-06-04T04:58:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.025+0000 7fe10ce24740 20 client.15699 trim_cache size 0 max 16384 2024-06-04T04:58:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.040+0000 7fe10ce24740 1 Processor -- start 2024-06-04T04:58:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.040+0000 7fe10ce24740 1 -- start start 2024-06-04T04:58:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.040+0000 7fe10ce24740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912ee0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.040+0000 7fe10ce24740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912e4140 0x5628912ea140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.040+0000 7fe10ce24740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912eea00 0x5628912f6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.040+0000 7fe10ce24740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe1041ad610 con 0x5628912e3820 2024-06-04T04:58:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.040+0000 7fe10ce24740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1041af150 con 0x5628912e4140 2024-06-04T04:58:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.040+0000 7fe10ce24740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe1041adce0 con 0x5628912eea00 2024-06-04T04:58:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.041+0000 7fe103fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912ee0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.041+0000 7fe103fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912ee0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45968/0 (socket says 172.21.15.70:45968) 2024-06-04T04:58:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.041+0000 7fe103fff640 1 -- 172.21.15.70:0/1382292632 learned_addr learned my addr 172.21.15.70:0/1382292632 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.041+0000 7fe103fff640 1 --2- 172.21.15.70:0/1382292632 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912eea00 0x5628912f6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.041+0000 7fe0fbfff640 1 --2- 172.21.15.70:0/1382292632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912e4140 0x5628912ea140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.041+0000 7fe103fff640 1 -- 172.21.15.70:0/1382292632 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912eea00 msgr2=0x5628912f6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.041+0000 7fe103fff640 1 --2- 172.21.15.70:0/1382292632 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912eea00 0x5628912f6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.041+0000 7fe103fff640 1 -- 172.21.15.70:0/1382292632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912e4140 msgr2=0x5628912ea140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.041+0000 7fe103fff640 1 --2- 172.21.15.70:0/1382292632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912e4140 0x5628912ea140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.041+0000 7fe103fff640 1 -- 172.21.15.70:0/1382292632 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628912e9ef0 con 0x5628912e3820 2024-06-04T04:58:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.041+0000 7fe0fbfff640 1 --2- 172.21.15.70:0/1382292632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912e4140 0x5628912ea140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.041+0000 7fe103fff640 1 --2- 172.21.15.70:0/1382292632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912ee0d0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009690 tx=0x7fe0fc009c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87a477ec5eeac4e5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.041+0000 7fe0f97fa640 1 -- 172.21.15.70:0/1382292632 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc010ce0 con 0x5628912e3820 2024-06-04T04:58:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.041+0000 7fe0f97fa640 1 -- 172.21.15.70:0/1382292632 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0fc00ee60 con 0x5628912e3820 2024-06-04T04:58:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.041+0000 7fe0f97fa640 1 -- 172.21.15.70:0/1382292632 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc005400 con 0x5628912e3820 2024-06-04T04:58:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.042+0000 7fe10ce24740 1 -- 172.21.15.70:0/1382292632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 msgr2=0x5628912ee0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.042+0000 7fe10ce24740 1 --2- 172.21.15.70:0/1382292632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912ee0d0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009690 tx=0x7fe0fc009c60 comp rx=0 tx=0).stop 2024-06-04T04:58:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.042+0000 7fe10ce24740 1 -- 172.21.15.70:0/1382292632 shutdown_connections 2024-06-04T04:58:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.042+0000 7fe10ce24740 1 --2- 172.21.15.70:0/1382292632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912e4140 0x5628912ea140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.042+0000 7fe10ce24740 1 --2- 172.21.15.70:0/1382292632 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912eea00 0x5628912f6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.042+0000 7fe10ce24740 1 --2- 172.21.15.70:0/1382292632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912ee0d0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.042+0000 7fe10ce24740 1 -- 172.21.15.70:0/1382292632 >> 172.21.15.70:0/1382292632 conn(0x5628912dfd30 msgr2=0x5628912defb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.042+0000 7fe10ce24740 1 -- 172.21.15.70:0/1382292632 shutdown_connections 2024-06-04T04:58:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.042+0000 7fe10ce24740 1 -- 172.21.15.70:0/1382292632 wait complete. 2024-06-04T04:58:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.042+0000 7fe10ce24740 1 -- start start 2024-06-04T04:58:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.042+0000 7fe10ce24740 1 Processor -- start 2024-06-04T04:58:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe10ce24740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912e7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe10ce24740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912e4140 0x5628912e8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe10ce24740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912eea00 0x5628912e8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe10ce24740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe1041aea80 con 0x5628912e3820 2024-06-04T04:58:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe10ce24740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1041ae3b0 con 0x5628912eea00 2024-06-04T04:58:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe10ce24740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe1041ab5f0 con 0x5628912e4140 2024-06-04T04:58:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe10ce24740 10 client.0 _ll_register_callbacks cb 0x7fe0fc009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe103fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912e7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe103fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912e7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45972/0 (socket says 172.21.15.70:45972) 2024-06-04T04:58:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe103fff640 1 -- 172.21.15.70:0/1625584772 learned_addr learned my addr 172.21.15.70:0/1625584772 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe0fbfff640 1 --2- 172.21.15.70:0/1625584772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912e4140 0x5628912e8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe103fff640 1 -- 172.21.15.70:0/1625584772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912e4140 msgr2=0x5628912e8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe103fff640 1 --2- 172.21.15.70:0/1625584772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912e4140 0x5628912e8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe103fff640 1 -- 172.21.15.70:0/1625584772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912eea00 msgr2=0x5628912e8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe103fff640 1 --2- 172.21.15.70:0/1625584772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912eea00 0x5628912e8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe103fff640 1 -- 172.21.15.70:0/1625584772 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56289136a460 con 0x5628912e3820 2024-06-04T04:58:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe103fff640 1 --2- 172.21.15.70:0/1625584772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912e7bf0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009b50 tx=0x7fe0fc009fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65efe3e338f6b96c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe0e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe0e2ffd640 1 -- 172.21.15.70:0/1625584772 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc047060 con 0x5628912e3820 2024-06-04T04:58:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe0e2ffd640 1 -- 172.21.15.70:0/1625584772 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fe0fc028700 con 0x5628912e3820 2024-06-04T04:58:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe10ce24740 1 -- 172.21.15.70:0/1625584772 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0fc00ee60 con 0x5628912e3820 2024-06-04T04:58:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe0e2ffd640 1 -- 172.21.15.70:0/1625584772 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe0fc028900 con 0x5628912e3820 2024-06-04T04:58:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe10ce24740 10 client.15705 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe10ce24740 1 -- 172.21.15.70:0/1625584772 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56289136b6e0 con 0x5628912e3820 2024-06-04T04:58:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.043+0000 7fe0e2ffd640 1 -- 172.21.15.70:0/1625584772 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0fc00ee60 con 0x5628912e3820 2024-06-04T04:58:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.044+0000 7fe0e2ffd640 1 -- 172.21.15.70:0/1625584772 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe0fc039070 con 0x5628912e3820 2024-06-04T04:58:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.044+0000 7fe10ce24740 10 client.15705 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.044+0000 7fe10ce24740 10 client.15705 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.044+0000 7fe10ce24740 20 client.15705 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.044+0000 7fe10ce24740 1 --2- 172.21.15.70:0/1625584772 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5628912fe720 0x56289136ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.044+0000 7fe10ce24740 4 client.15705 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T04:58:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.044+0000 7fe10ce24740 1 -- 172.21.15.70:0/1625584772 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562891251360 con 0x5628912fe720 2024-06-04T04:58:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.044+0000 7fe0fbfff640 1 --2- 172.21.15.70:0/1625584772 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5628912fe720 0x56289136ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.044+0000 7fe0e2ffd640 10 client.15705 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.044+0000 7fe0fbfff640 1 --2- 172.21.15.70:0/1625584772 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5628912fe720 0x56289136ddb0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec6cbad7fb0e4f16 server_cookie=11ca6616b6bf59e7 in_seq=0 out_seq=0 2024-06-04T04:58:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.048+0000 7fe0e2ffd640 1 -- 172.21.15.70:0/1625584772 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x562891251360 con 0x5628912fe720 2024-06-04T04:58:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.048+0000 7fe0e2ffd640 10 client.15705 handle_command_reply: tid=0 2024-06-04T04:58:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.048+0000 7fe10ce24740 1 client.15705 shutdown 2024-06-04T04:58:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.048+0000 7fe10ce24740 1 -- 172.21.15.70:0/1625584772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 msgr2=0x5628912e7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.048+0000 7fe10ce24740 1 --2- 172.21.15.70:0/1625584772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912e7bf0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009b50 tx=0x7fe0fc009fb0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.048+0000 7fe10ce24740 1 -- 172.21.15.70:0/1625584772 shutdown_connections 2024-06-04T04:58:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.048+0000 7fe10ce24740 1 --2- 172.21.15.70:0/1625584772 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5628912fe720 0x56289136ddb0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.048+0000 7fe10ce24740 1 --2- 172.21.15.70:0/1625584772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628912eea00 0x5628912e8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.048+0000 7fe10ce24740 1 --2- 172.21.15.70:0/1625584772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628912e4140 0x5628912e8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.048+0000 7fe0e2ffd640 0 client.15705 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.048+0000 7fe10ce24740 1 --2- 172.21.15.70:0/1625584772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628912e3820 0x5628912e7bf0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.048+0000 7fe10ce24740 1 -- 172.21.15.70:0/1625584772 >> 172.21.15.70:0/1625584772 conn(0x5628912dfd30 msgr2=0x5628912e51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.048+0000 7fe10ce24740 1 -- 172.21.15.70:0/1625584772 shutdown_connections 2024-06-04T04:58:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.048+0000 7fe10ce24740 1 -- 172.21.15.70:0/1625584772 wait complete. 2024-06-04T04:58:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.048+0000 7fe10ce24740 20 client.15705 trim_cache size 0 max 16384 2024-06-04T04:58:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T04:58:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T04:58:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T04:58:16.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.049+0000 7fe10a10a640 1 -- 172.21.15.70:0/2844989909 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe0f00b2b30 msgr2=0x7fe0f00b4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:16.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.049+0000 7fe10a10a640 1 --2- 172.21.15.70:0/2844989909 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe0f00b2b30 0x7fe0f00b4ff0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe0f40df920 tx=0x7fe0f40e4040 comp rx=0 tx=0).stop 2024-06-04T04:58:16.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.049+0000 7fe10a10a640 1 -- 172.21.15.70:0/2844989909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe104101980 msgr2=0x7fe104198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:16.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.049+0000 7fe10a10a640 1 --2- 172.21.15.70:0/2844989909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe104101980 0x7fe104198000 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fe0f400d950 tx=0x7fe0f400de10 comp rx=0 tx=0).stop 2024-06-04T04:58:16.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.049+0000 7fe10a10a640 1 -- 172.21.15.70:0/2844989909 shutdown_connections 2024-06-04T04:58:16.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.049+0000 7fe10a10a640 1 --2- 172.21.15.70:0/2844989909 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe0f00b2b30 0x7fe0f00b4ff0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.049+0000 7fe10a10a640 1 --2- 172.21.15.70:0/2844989909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe104102340 0x7fe104198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.049+0000 7fe10a10a640 1 --2- 172.21.15.70:0/2844989909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe104107940 0x7fe10419c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.049+0000 7fe10a10a640 1 --2- 172.21.15.70:0/2844989909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe104101980 0x7fe104198000 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:16.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.049+0000 7fe10a10a640 1 -- 172.21.15.70:0/2844989909 >> 172.21.15.70:0/2844989909 conn(0x7fe104078ad0 msgr2=0x7fe104104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:16.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.049+0000 7fe10a10a640 1 -- 172.21.15.70:0/2844989909 shutdown_connections 2024-06-04T04:58:16.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:16.049+0000 7fe10a10a640 1 -- 172.21.15.70:0/2844989909 wait complete. 2024-06-04T04:58:16.396 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T04:58:16.633 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:17.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:17 smithi070 ceph-mon[32631]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1023 B/s wr, 14 op/s 2024-06-04T04:58:17.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:17 smithi112 ceph-mon[36453]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1023 B/s wr, 14 op/s 2024-06-04T04:58:17.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:17 smithi098 ceph-mon[35569]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1023 B/s wr, 14 op/s 2024-06-04T04:58:18.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.315+0000 7f6a99745700 1 -- 172.21.15.70:0/3923539041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a9406bbe0 msgr2=0x7f6a941103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:18.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.315+0000 7f6a99745700 1 --2- 172.21.15.70:0/3923539041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a9406bbe0 0x7f6a941103d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00b0a0 tx=0x7f6a8c00b3b0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.316+0000 7f6a99745700 1 -- 172.21.15.70:0/3923539041 shutdown_connections 2024-06-04T04:58:18.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.316+0000 7f6a99745700 1 --2- 172.21.15.70:0/3923539041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a9406bbe0 0x7f6a941103d0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.316+0000 7f6a99745700 1 --2- 172.21.15.70:0/3923539041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a941095b0 0x7f6a94109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.316+0000 7f6a99745700 1 --2- 172.21.15.70:0/3923539041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a9406b220 0x7f6a9406b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.316+0000 7f6a99745700 1 -- 172.21.15.70:0/3923539041 >> 172.21.15.70:0/3923539041 conn(0x7f6a940793f0 msgr2=0x7f6a94079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:18.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.316+0000 7f6a99745700 1 -- 172.21.15.70:0/3923539041 shutdown_connections 2024-06-04T04:58:18.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.316+0000 7f6a99745700 1 -- 172.21.15.70:0/3923539041 wait complete. 2024-06-04T04:58:18.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.316+0000 7f6a99745700 1 Processor -- start 2024-06-04T04:58:18.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.316+0000 7f6a99745700 1 -- start start 2024-06-04T04:58:18.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a99745700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a9406b220 0x7f6a9419f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a99745700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a9406bbe0 0x7f6a9419fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a99745700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a941095b0 0x7f6a94199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a99745700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6a941135f0 con 0x7f6a9406bbe0 2024-06-04T04:58:18.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a99745700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a94113470 con 0x7f6a941095b0 2024-06-04T04:58:18.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a99745700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6a94113770 con 0x7f6a9406b220 2024-06-04T04:58:18.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a927fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a9406bbe0 0x7f6a9419fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:18.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a927fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a9406bbe0 0x7f6a9419fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45992/0 (socket says 172.21.15.70:45992) 2024-06-04T04:58:18.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a927fc700 1 -- 172.21.15.70:0/278982891 learned_addr learned my addr 172.21.15.70:0/278982891 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:18.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a937fe700 1 --2- 172.21.15.70:0/278982891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a941095b0 0x7f6a94199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:18.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a92ffd700 1 --2- 172.21.15.70:0/278982891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a9406b220 0x7f6a9419f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:18.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a927fc700 1 -- 172.21.15.70:0/278982891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a9406b220 msgr2=0x7f6a9419f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:18.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a927fc700 1 --2- 172.21.15.70:0/278982891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a9406b220 0x7f6a9419f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a927fc700 1 -- 172.21.15.70:0/278982891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a941095b0 msgr2=0x7f6a94199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:18.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a927fc700 1 --2- 172.21.15.70:0/278982891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a941095b0 0x7f6a94199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a927fc700 1 -- 172.21.15.70:0/278982891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a9419a0b0 con 0x7f6a9406bbe0 2024-06-04T04:58:18.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a927fc700 1 --2- 172.21.15.70:0/278982891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a9406bbe0 0x7f6a9419fcd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6a8800c920 tx=0x7f6a8800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:18.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a7bfff700 1 -- 172.21.15.70:0/278982891 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a88004a30 con 0x7f6a9406bbe0 2024-06-04T04:58:18.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a7bfff700 1 -- 172.21.15.70:0/278982891 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6a880054b0 con 0x7f6a9406bbe0 2024-06-04T04:58:18.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a99745700 1 -- 172.21.15.70:0/278982891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a9419a340 con 0x7f6a9406bbe0 2024-06-04T04:58:18.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.317+0000 7f6a7bfff700 1 -- 172.21.15.70:0/278982891 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a88007b70 con 0x7f6a9406bbe0 2024-06-04T04:58:18.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.318+0000 7f6a99745700 1 -- 172.21.15.70:0/278982891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a941a6660 con 0x7f6a9406bbe0 2024-06-04T04:58:18.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.320+0000 7f6a7bfff700 1 -- 172.21.15.70:0/278982891 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 149885+0+0 (secure 0 0 0) 0x7f6a88004bd0 con 0x7f6a9406bbe0 2024-06-04T04:58:18.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.321+0000 7f6a7bfff700 1 --2- 172.21.15.70:0/278982891 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6a7c0b2b10 0x7f6a7c0b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.321+0000 7f6a92ffd700 1 --2- 172.21.15.70:0/278982891 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6a7c0b2b10 0x7f6a7c0b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:18.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.321+0000 7f6a7bfff700 1 -- 172.21.15.70:0/278982891 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f6a880e2420 con 0x7f6a9406bbe0 2024-06-04T04:58:18.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.321+0000 7f6a92ffd700 1 --2- 172.21.15.70:0/278982891 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6a7c0b2b10 0x7f6a7c0b4fd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a9419aa90 tx=0x7f6a84009190 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:18.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.321+0000 7f6aa3fdab80 1 Processor -- start 2024-06-04T04:58:18.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6aa3fdab80 1 -- start start 2024-06-04T04:58:18.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6aa3fdab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231e860 0x56392231ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6aa3fdab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392231f210 0x563922322940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6aa3fdab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563922322f60 0x5639223233a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6aa3fdab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56392227dc90 con 0x56392231e860 2024-06-04T04:58:18.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6aa3fdab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56392226d830 con 0x563922322f60 2024-06-04T04:58:18.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6aa3fdab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563922271870 con 0x56392231f210 2024-06-04T04:58:18.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6a92ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231e860 0x56392231ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:18.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6a92ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231e860 0x56392231ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46004/0 (socket says 172.21.15.70:46004) 2024-06-04T04:58:18.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6a92ffd700 1 -- 172.21.15.70:0/3561298754 learned_addr learned my addr 172.21.15.70:0/3561298754 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:18.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6a937fe700 1 --2- 172.21.15.70:0/3561298754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563922322f60 0x5639223233a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:18.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6a927fc700 1 --2- 172.21.15.70:0/3561298754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392231f210 0x563922322940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:18.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6a92ffd700 1 -- 172.21.15.70:0/3561298754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392231f210 msgr2=0x563922322940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:18.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6a92ffd700 1 --2- 172.21.15.70:0/3561298754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392231f210 0x563922322940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6a92ffd700 1 -- 172.21.15.70:0/3561298754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563922322f60 msgr2=0x5639223233a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:18.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6a92ffd700 1 --2- 172.21.15.70:0/3561298754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563922322f60 0x5639223233a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6a92ffd700 1 -- 172.21.15.70:0/3561298754 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639221bd8d0 con 0x56392231e860 2024-06-04T04:58:18.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6a92ffd700 1 --2- 172.21.15.70:0/3561298754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231e860 0x56392231ec40 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6a84009fd0 tx=0x7f6a84011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8407ab2896cf862e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:18.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.322+0000 7f6a797fa700 1 -- 172.21.15.70:0/3561298754 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8401b040 con 0x56392231e860 2024-06-04T04:58:18.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.323+0000 7f6a797fa700 1 -- 172.21.15.70:0/3561298754 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6a84024070 con 0x56392231e860 2024-06-04T04:58:18.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.323+0000 7f6a797fa700 1 -- 172.21.15.70:0/3561298754 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a840136b0 con 0x56392231e860 2024-06-04T04:58:18.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.323+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/3561298754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231e860 msgr2=0x56392231ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:18.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.323+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/3561298754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231e860 0x56392231ec40 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6a84009fd0 tx=0x7f6a84011000 comp rx=0 tx=0).stop 2024-06-04T04:58:18.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.323+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/3561298754 shutdown_connections 2024-06-04T04:58:18.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.323+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/3561298754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563922322f60 0x5639223233a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.323+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/3561298754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231e860 0x56392231ec40 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.323+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/3561298754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392231f210 0x563922322940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.323+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/3561298754 >> 172.21.15.70:0/3561298754 conn(0x56392224c080 msgr2=0x563922318e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:18.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.323+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/3561298754 shutdown_connections 2024-06-04T04:58:18.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.323+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/3561298754 wait complete. 2024-06-04T04:58:18.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6aa3fdab80 1 -- start start 2024-06-04T04:58:18.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6aa3fdab80 1 Processor -- start 2024-06-04T04:58:18.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6aa3fdab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392231e860 0x56392232a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6aa3fdab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231f210 0x5639223276d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6aa3fdab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563922322f60 0x563922327c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6aa3fdab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563922173fa0 con 0x56392231f210 2024-06-04T04:58:18.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6aa3fdab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56392220f7c0 con 0x56392231e860 2024-06-04T04:58:18.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6aa3fdab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56392232b120 con 0x563922322f60 2024-06-04T04:58:18.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6aa3fdab80 10 client.0 _ll_register_callbacks cb 0x563922208920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:18.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6a927fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231f210 0x5639223276d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:18.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6a927fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231f210 0x5639223276d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46016/0 (socket says 172.21.15.70:46016) 2024-06-04T04:58:18.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6a927fc700 1 -- 172.21.15.70:0/2394144802 learned_addr learned my addr 172.21.15.70:0/2394144802 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:18.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6a92ffd700 1 --2- 172.21.15.70:0/2394144802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392231e860 0x56392232a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:18.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6a927fc700 1 -- 172.21.15.70:0/2394144802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563922322f60 msgr2=0x563922327c10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T04:58:18.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6a927fc700 1 --2- 172.21.15.70:0/2394144802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563922322f60 0x563922327c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6a927fc700 1 -- 172.21.15.70:0/2394144802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392231e860 msgr2=0x56392232a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:18.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6a927fc700 1 --2- 172.21.15.70:0/2394144802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392231e860 0x56392232a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6a927fc700 1 -- 172.21.15.70:0/2394144802 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56392220c690 con 0x56392231f210 2024-06-04T04:58:18.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.324+0000 7f6a92ffd700 1 --2- 172.21.15.70:0/2394144802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392231e860 0x56392232a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:18.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.325+0000 7f6a927fc700 1 --2- 172.21.15.70:0/2394144802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231f210 0x5639223276d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6a88000c00 tx=0x7f6a88007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7da0ecf229c65eb5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:18.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.325+0000 7f6a76ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:18.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.325+0000 7f6a76ff5700 1 -- 172.21.15.70:0/2394144802 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a880c3060 con 0x56392231f210 2024-06-04T04:58:18.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.325+0000 7f6a76ff5700 1 -- 172.21.15.70:0/2394144802 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f6a880b2030 con 0x56392231f210 2024-06-04T04:58:18.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.325+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/2394144802 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a84024070 con 0x56392231f210 2024-06-04T04:58:18.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.325+0000 7f6a76ff5700 1 -- 172.21.15.70:0/2394144802 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6a8801d040 con 0x56392231f210 2024-06-04T04:58:18.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.325+0000 7f6aa3fdab80 10 client.15717 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:18.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.325+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/2394144802 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5639223358b0 con 0x56392231f210 2024-06-04T04:58:18.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.325+0000 7f6a76ff5700 1 -- 172.21.15.70:0/2394144802 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6a84024070 con 0x56392231f210 2024-06-04T04:58:18.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.325+0000 7f6a76ff5700 1 -- 172.21.15.70:0/2394144802 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f6a880dfc70 con 0x56392231f210 2024-06-04T04:58:18.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.325+0000 7f6aa3fdab80 10 client.15717 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:18.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.325+0000 7f6aa3fdab80 10 client.15717 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T04:58:18.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.325+0000 7f6aa3fdab80 20 client.15717 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:18.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.325+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/2394144802 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x563922337e40 0x56392233a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.325+0000 7f6aa3fdab80 4 client.15717 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:18.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.325+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/2394144802 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639222ea9e0 con 0x563922337e40 2024-06-04T04:58:18.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.326+0000 7f6a92ffd700 1 --2- 172.21.15.70:0/2394144802 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x563922337e40 0x56392233a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:18.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.326+0000 7f6a76ff5700 10 client.15717 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T04:58:18.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.326+0000 7f6a92ffd700 1 --2- 172.21.15.70:0/2394144802 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x563922337e40 0x56392233a230 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d1b8dde8390144ac server_cookie=f5c2a58e24ff5345 in_seq=0 out_seq=0 2024-06-04T04:58:18.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.327+0000 7f6a76ff5700 1 -- 172.21.15.70:0/2394144802 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5639222ea9e0 con 0x563922337e40 2024-06-04T04:58:18.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.327+0000 7f6a76ff5700 10 client.15717 handle_command_reply: tid=0 2024-06-04T04:58:18.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.327+0000 7f6aa3fdab80 1 client.15717 shutdown 2024-06-04T04:58:18.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.327+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/2394144802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231f210 msgr2=0x5639223276d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:18.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.327+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/2394144802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231f210 0x5639223276d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6a88000c00 tx=0x7f6a88007f90 comp rx=0 tx=0).stop 2024-06-04T04:58:18.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.327+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/2394144802 shutdown_connections 2024-06-04T04:58:18.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.327+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/2394144802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563922322f60 0x563922327c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.327+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/2394144802 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x563922337e40 0x56392233a230 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.327+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/2394144802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392231e860 0x56392232a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.327+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/2394144802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231f210 0x5639223276d0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.327+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/2394144802 >> 172.21.15.70:0/2394144802 conn(0x56392224c080 msgr2=0x563922318510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:18.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.327+0000 7f6a76ff5700 0 client.15717 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T04:58:18.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.327+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/2394144802 shutdown_connections 2024-06-04T04:58:18.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.327+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/2394144802 wait complete. 2024-06-04T04:58:18.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.327+0000 7f6aa3fdab80 20 client.15717 trim_cache size 0 max 16384 2024-06-04T04:58:18.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.342+0000 7f6aa3fdab80 1 Processor -- start 2024-06-04T04:58:18.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.342+0000 7f6aa3fdab80 1 -- start start 2024-06-04T04:58:18.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.342+0000 7f6aa3fdab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392231e860 0x563922329610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.342+0000 7f6aa3fdab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231f210 0x56392233fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.342+0000 7f6aa3fdab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563922322f60 0x56392233fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.342+0000 7f6aa3fdab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6a941ade00 con 0x56392231f210 2024-06-04T04:58:18.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.342+0000 7f6aa3fdab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a941af940 con 0x563922322f60 2024-06-04T04:58:18.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.342+0000 7f6aa3fdab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6a941ae4d0 con 0x56392231e860 2024-06-04T04:58:18.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.343+0000 7f6a927fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231f210 0x56392233fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:18.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.343+0000 7f6a927fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231f210 0x56392233fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46026/0 (socket says 172.21.15.70:46026) 2024-06-04T04:58:18.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.343+0000 7f6a927fc700 1 -- 172.21.15.70:0/2496866788 learned_addr learned my addr 172.21.15.70:0/2496866788 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:18.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.343+0000 7f6a92ffd700 1 --2- 172.21.15.70:0/2496866788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392231e860 0x563922329610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:18.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.343+0000 7f6a937fe700 1 --2- 172.21.15.70:0/2496866788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563922322f60 0x56392233fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:18.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.343+0000 7f6a927fc700 1 -- 172.21.15.70:0/2496866788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392231e860 msgr2=0x563922329610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:18.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.343+0000 7f6a927fc700 1 --2- 172.21.15.70:0/2496866788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392231e860 0x563922329610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.343+0000 7f6a927fc700 1 -- 172.21.15.70:0/2496866788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563922322f60 msgr2=0x56392233fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:18.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.343+0000 7f6a927fc700 1 --2- 172.21.15.70:0/2496866788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563922322f60 0x56392233fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.343+0000 7f6a927fc700 1 -- 172.21.15.70:0/2496866788 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563922325c80 con 0x56392231f210 2024-06-04T04:58:18.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.343+0000 7f6a92ffd700 1 --2- 172.21.15.70:0/2496866788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392231e860 0x563922329610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:18.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.343+0000 7f6a937fe700 1 --2- 172.21.15.70:0/2496866788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563922322f60 0x56392233fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.343+0000 7f6a927fc700 1 --2- 172.21.15.70:0/2496866788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231f210 0x56392233fab0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6a8800c8f0 tx=0x7f6a880a90c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24c1af823cccea94 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.343+0000 7f6a78ff9700 1 -- 172.21.15.70:0/2496866788 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a880a9b50 con 0x56392231f210 2024-06-04T04:58:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.343+0000 7f6a78ff9700 1 -- 172.21.15.70:0/2496866788 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6a880a9450 con 0x56392231f210 2024-06-04T04:58:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.343+0000 7f6a78ff9700 1 -- 172.21.15.70:0/2496866788 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a880dfc70 con 0x56392231f210 2024-06-04T04:58:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.344+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/2496866788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231f210 msgr2=0x56392233fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.344+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/2496866788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231f210 0x56392233fab0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6a8800c8f0 tx=0x7f6a880a90c0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.344+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/2496866788 shutdown_connections 2024-06-04T04:58:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.344+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/2496866788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563922322f60 0x56392233fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.344+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/2496866788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392231e860 0x563922329610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.344+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/2496866788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392231f210 0x56392233fab0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.344+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/2496866788 >> 172.21.15.70:0/2496866788 conn(0x56392224c080 msgr2=0x563922328f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.344+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/2496866788 shutdown_connections 2024-06-04T04:58:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.344+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/2496866788 wait complete. 2024-06-04T04:58:18.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.344+0000 7f6aa3fdab80 1 -- start start 2024-06-04T04:58:18.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.344+0000 7f6aa3fdab80 1 Processor -- start 2024-06-04T04:58:18.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6aa3fdab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392231e860 0x563922330bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6aa3fdab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563922322f60 0x56392239b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6aa3fdab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392239b940 0x56392239bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6aa3fdab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6a941af270 con 0x563922322f60 2024-06-04T04:58:18.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6aa3fdab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a941aeba0 con 0x56392231e860 2024-06-04T04:58:18.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6aa3fdab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6a941abda0 con 0x56392239b940 2024-06-04T04:58:18.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6aa3fdab80 10 client.0 _ll_register_callbacks cb 0x563922208920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:18.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6a927fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563922322f60 0x56392239b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:18.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6a937fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392239b940 0x56392239bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:18.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6a927fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563922322f60 0x56392239b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46040/0 (socket says 172.21.15.70:46040) 2024-06-04T04:58:18.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6a937fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392239b940 0x56392239bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:55614/0 (socket says 172.21.15.70:55614) 2024-06-04T04:58:18.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6a927fc700 1 -- 172.21.15.70:0/3249484608 learned_addr learned my addr 172.21.15.70:0/3249484608 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:18.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6a92ffd700 1 --2- 172.21.15.70:0/3249484608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392231e860 0x563922330bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:18.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6a927fc700 1 -- 172.21.15.70:0/3249484608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392239b940 msgr2=0x56392239bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:18.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6a927fc700 1 --2- 172.21.15.70:0/3249484608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392239b940 0x56392239bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6a927fc700 1 -- 172.21.15.70:0/3249484608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392231e860 msgr2=0x563922330bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6a927fc700 1 --2- 172.21.15.70:0/3249484608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392231e860 0x563922330bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6a927fc700 1 -- 172.21.15.70:0/3249484608 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56392232e8b0 con 0x563922322f60 2024-06-04T04:58:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6a937fe700 1 --2- 172.21.15.70:0/3249484608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392239b940 0x56392239bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6a927fc700 1 --2- 172.21.15.70:0/3249484608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563922322f60 0x56392239b3b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6a880050a0 tx=0x7f6a88005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe502a1688cd025 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.345+0000 7f6a76ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.346+0000 7f6a76ff5700 1 -- 172.21.15.70:0/3249484608 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a880ad800 con 0x563922322f60 2024-06-04T04:58:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.346+0000 7f6a76ff5700 1 -- 172.21.15.70:0/3249484608 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f6a880a3b70 con 0x563922322f60 2024-06-04T04:58:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.346+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/3249484608 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a880a9450 con 0x563922322f60 2024-06-04T04:58:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.346+0000 7f6a76ff5700 1 -- 172.21.15.70:0/3249484608 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6a880c3020 con 0x563922322f60 2024-06-04T04:58:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.346+0000 7f6aa3fdab80 10 client.15723 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.346+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/3249484608 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5639223a7210 con 0x563922322f60 2024-06-04T04:58:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.346+0000 7f6a76ff5700 1 -- 172.21.15.70:0/3249484608 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6a880a9450 con 0x563922322f60 2024-06-04T04:58:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.346+0000 7f6a76ff5700 1 -- 172.21.15.70:0/3249484608 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f6a88083b10 con 0x563922322f60 2024-06-04T04:58:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.346+0000 7f6aa3fdab80 10 client.15723 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.346+0000 7f6aa3fdab80 10 client.15723 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T04:58:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.346+0000 7f6aa3fdab80 20 client.15723 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.346+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/3249484608 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5639223a7620 0x5639223a9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.346+0000 7f6aa3fdab80 4 client.15723 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T04:58:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.346+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/3249484608 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56392227fbe0 con 0x5639223a7620 2024-06-04T04:58:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.347+0000 7f6a92ffd700 1 --2- 172.21.15.70:0/3249484608 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5639223a7620 0x5639223a9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:18.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.347+0000 7f6a92ffd700 1 --2- 172.21.15.70:0/3249484608 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5639223a7620 0x5639223a9a10 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=12e2ded48d9b7bf3 server_cookie=935ee233272fde73 in_seq=0 out_seq=0 2024-06-04T04:58:18.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.347+0000 7f6a76ff5700 10 client.15723 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T04:58:18.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.347+0000 7f6a76ff5700 1 -- 172.21.15.70:0/3249484608 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56392227fbe0 con 0x5639223a7620 2024-06-04T04:58:18.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.347+0000 7f6a76ff5700 10 client.15723 handle_command_reply: tid=0 2024-06-04T04:58:18.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.347+0000 7f6aa3fdab80 1 client.15723 shutdown 2024-06-04T04:58:18.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.347+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/3249484608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563922322f60 msgr2=0x56392239b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:18.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.347+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/3249484608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563922322f60 0x56392239b3b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6a880050a0 tx=0x7f6a88005300 comp rx=0 tx=0).stop 2024-06-04T04:58:18.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.347+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/3249484608 shutdown_connections 2024-06-04T04:58:18.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.347+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/3249484608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392239b940 0x56392239bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.347+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/3249484608 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5639223a7620 0x5639223a9a10 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.347+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/3249484608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392231e860 0x563922330bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.347+0000 7f6aa3fdab80 1 --2- 172.21.15.70:0/3249484608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563922322f60 0x56392239b3b0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.347+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/3249484608 >> 172.21.15.70:0/3249484608 conn(0x56392224c080 msgr2=0x56392231df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:18.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.347+0000 7f6a76ff5700 0 client.15723 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T04:58:18.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.347+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/3249484608 shutdown_connections 2024-06-04T04:58:18.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.347+0000 7f6aa3fdab80 1 -- 172.21.15.70:0/3249484608 wait complete. 2024-06-04T04:58:18.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.348+0000 7f6aa3fdab80 20 client.15723 trim_cache size 0 max 16384 2024-06-04T04:58:18.359 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T04:58:18.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.348+0000 7f6a99745700 1 -- 172.21.15.70:0/278982891 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6a7c0b2b10 msgr2=0x7f6a7c0b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:18.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.348+0000 7f6a99745700 1 --2- 172.21.15.70:0/278982891 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6a7c0b2b10 0x7f6a7c0b4fd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a9419aa90 tx=0x7f6a84009190 comp rx=0 tx=0).stop 2024-06-04T04:58:18.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.348+0000 7f6a99745700 1 -- 172.21.15.70:0/278982891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a9406bbe0 msgr2=0x7f6a9419fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:18.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.348+0000 7f6a99745700 1 --2- 172.21.15.70:0/278982891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a9406bbe0 0x7f6a9419fcd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6a8800c920 tx=0x7f6a8800cc30 comp rx=0 tx=0).stop 2024-06-04T04:58:18.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.348+0000 7f6a99745700 1 -- 172.21.15.70:0/278982891 shutdown_connections 2024-06-04T04:58:18.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.348+0000 7f6a99745700 1 --2- 172.21.15.70:0/278982891 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6a7c0b2b10 0x7f6a7c0b4fd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.348+0000 7f6a99745700 1 --2- 172.21.15.70:0/278982891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a941095b0 0x7f6a94199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.348+0000 7f6a99745700 1 --2- 172.21.15.70:0/278982891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a9406b220 0x7f6a9419f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.348+0000 7f6a99745700 1 --2- 172.21.15.70:0/278982891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a9406bbe0 0x7f6a9419fcd0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:18.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.348+0000 7f6a99745700 1 -- 172.21.15.70:0/278982891 >> 172.21.15.70:0/278982891 conn(0x7f6a940793f0 msgr2=0x7f6a941012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:18.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.348+0000 7f6a99745700 1 -- 172.21.15.70:0/278982891 shutdown_connections 2024-06-04T04:58:18.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:18.348+0000 7f6a99745700 1 -- 172.21.15.70:0/278982891 wait complete. 2024-06-04T04:58:19.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:19 smithi070 ceph-mon[32631]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-04T04:58:19.288 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T04:58:19.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:19 smithi112 ceph-mon[36453]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-04T04:58:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:19 smithi098 ceph-mon[35569]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-04T04:58:19.537 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:20.121 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 34.9 2024-06-04T04:58:20.121 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:58:21.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.274+0000 7fa80e442700 1 -- 172.21.15.70:0/3899736194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa80806c470 msgr2=0x7fa8081103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:21.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.274+0000 7fa80e442700 1 --2- 172.21.15.70:0/3899736194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa80806c470 0x7fa8081103c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa80000b0a0 tx=0x7fa80000b3b0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.274+0000 7fa80e442700 1 -- 172.21.15.70:0/3899736194 shutdown_connections 2024-06-04T04:58:21.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.274+0000 7fa80e442700 1 --2- 172.21.15.70:0/3899736194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa80806c470 0x7fa8081103c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.274+0000 7fa80e442700 1 --2- 172.21.15.70:0/3899736194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8081095a0 0x7fa808109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.274+0000 7fa80e442700 1 --2- 172.21.15.70:0/3899736194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa80806bab0 0x7fa80806bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.274+0000 7fa80e442700 1 -- 172.21.15.70:0/3899736194 >> 172.21.15.70:0/3899736194 conn(0x7fa8080793b0 msgr2=0x7fa8080797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:21.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.274+0000 7fa80e442700 1 -- 172.21.15.70:0/3899736194 shutdown_connections 2024-06-04T04:58:21.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.274+0000 7fa80e442700 1 -- 172.21.15.70:0/3899736194 wait complete. 2024-06-04T04:58:21.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.275+0000 7fa80e442700 1 Processor -- start 2024-06-04T04:58:21.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.275+0000 7fa80e442700 1 -- start start 2024-06-04T04:58:21.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.275+0000 7fa80e442700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa80806bab0 0x7fa8081a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.275+0000 7fa80e442700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa80806c470 0x7fa8081a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.275+0000 7fa80e442700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8081095a0 0x7fa80819baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.275+0000 7fa80e442700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa808113590 con 0x7fa80806c470 2024-06-04T04:58:21.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.275+0000 7fa80e442700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa808113410 con 0x7fa80806bab0 2024-06-04T04:58:21.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.275+0000 7fa80e442700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa808113710 con 0x7fa8081095a0 2024-06-04T04:58:21.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.276+0000 7fa8077fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa80806c470 0x7fa8081a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.276+0000 7fa8077fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa80806c470 0x7fa8081a1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46056/0 (socket says 172.21.15.70:46056) 2024-06-04T04:58:21.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.276+0000 7fa8077fe700 1 -- 172.21.15.70:0/19161512 learned_addr learned my addr 172.21.15.70:0/19161512 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:21.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.276+0000 7fa807fff700 1 --2- 172.21.15.70:0/19161512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa80806bab0 0x7fa8081a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.276+0000 7fa80c9df700 1 --2- 172.21.15.70:0/19161512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8081095a0 0x7fa80819baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.276+0000 7fa8077fe700 1 -- 172.21.15.70:0/19161512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8081095a0 msgr2=0x7fa80819baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:21.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.276+0000 7fa8077fe700 1 --2- 172.21.15.70:0/19161512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8081095a0 0x7fa80819baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.276+0000 7fa8077fe700 1 -- 172.21.15.70:0/19161512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa80806bab0 msgr2=0x7fa8081a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:21.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.276+0000 7fa8077fe700 1 --2- 172.21.15.70:0/19161512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa80806bab0 0x7fa8081a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.276+0000 7fa8077fe700 1 -- 172.21.15.70:0/19161512 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa80819c250 con 0x7fa80806c470 2024-06-04T04:58:21.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.276+0000 7fa807fff700 1 --2- 172.21.15.70:0/19161512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa80806bab0 0x7fa8081a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:21.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.276+0000 7fa8077fe700 1 --2- 172.21.15.70:0/19161512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa80806c470 0x7fa8081a1e70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00e9c0 tx=0x7fa7fc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:21.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.276+0000 7fa8057fa700 1 -- 172.21.15.70:0/19161512 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc00cbd0 con 0x7fa80806c470 2024-06-04T04:58:21.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.276+0000 7fa8057fa700 1 -- 172.21.15.70:0/19161512 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa7fc00cd70 con 0x7fa80806c470 2024-06-04T04:58:21.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.276+0000 7fa80e442700 1 -- 172.21.15.70:0/19161512 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa80819c4e0 con 0x7fa80806c470 2024-06-04T04:58:21.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.277+0000 7fa8057fa700 1 -- 172.21.15.70:0/19161512 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc005540 con 0x7fa80806c470 2024-06-04T04:58:21.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.277+0000 7fa80e442700 1 -- 172.21.15.70:0/19161512 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8081a8800 con 0x7fa80806c470 2024-06-04T04:58:21.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.280+0000 7fa8057fa700 1 -- 172.21.15.70:0/19161512 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 149885+0+0 (secure 0 0 0) 0x7fa7fc004750 con 0x7fa80806c470 2024-06-04T04:58:21.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.281+0000 7fa818cd7b80 1 Processor -- start 2024-06-04T04:58:21.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.281+0000 7fa818cd7b80 1 -- start start 2024-06-04T04:58:21.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.281+0000 7fa818cd7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67a070 0x55862a67a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.281+0000 7fa818cd7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67aa20 0x55862a67e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.281+0000 7fa818cd7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a67e770 0x55862a67ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.281+0000 7fa818cd7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55862a5d0c90 con 0x55862a67aa20 2024-06-04T04:58:21.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.281+0000 7fa818cd7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55862a5c0830 con 0x55862a67e770 2024-06-04T04:58:21.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.281+0000 7fa818cd7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55862a5c4870 con 0x55862a67a070 2024-06-04T04:58:21.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.281+0000 7fa807fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67aa20 0x55862a67e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.281+0000 7fa807fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67aa20 0x55862a67e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46058/0 (socket says 172.21.15.70:46058) 2024-06-04T04:58:21.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.281+0000 7fa807fff700 1 -- 172.21.15.70:0/157219644 learned_addr learned my addr 172.21.15.70:0/157219644 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:21.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.281+0000 7fa8077fe700 1 --2- 172.21.15.70:0/157219644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a67e770 0x55862a67ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:21 smithi070 ceph-mon[32631]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-04T04:58:21.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.282+0000 7fa80c9df700 1 --2- 172.21.15.70:0/157219644 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67a070 0x55862a67a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.282+0000 7fa807fff700 1 -- 172.21.15.70:0/157219644 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67a070 msgr2=0x55862a67a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:21.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.282+0000 7fa807fff700 1 --2- 172.21.15.70:0/157219644 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67a070 0x55862a67a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.282+0000 7fa807fff700 1 -- 172.21.15.70:0/157219644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a67e770 msgr2=0x55862a67ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:21.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.282+0000 7fa807fff700 1 --2- 172.21.15.70:0/157219644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a67e770 0x55862a67ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.282+0000 7fa807fff700 1 -- 172.21.15.70:0/157219644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55862a5108d0 con 0x55862a67aa20 2024-06-04T04:58:21.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.282+0000 7fa807fff700 1 --2- 172.21.15.70:0/157219644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67aa20 0x55862a67e150 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8009ca0 tx=0x7fa7f800f360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef535e0cd73d9e62 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:21.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.282+0000 7fa7ee7fc700 1 -- 172.21.15.70:0/157219644 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f800fc90 con 0x55862a67aa20 2024-06-04T04:58:21.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.282+0000 7fa7ee7fc700 1 -- 172.21.15.70:0/157219644 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa7f800fe30 con 0x55862a67aa20 2024-06-04T04:58:21.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.282+0000 7fa7ee7fc700 1 -- 172.21.15.70:0/157219644 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f8011440 con 0x55862a67aa20 2024-06-04T04:58:21.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.282+0000 7fa8057fa700 1 --2- 172.21.15.70:0/19161512 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa7f00b2b10 0x7fa7f00b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.283+0000 7fa8057fa700 1 -- 172.21.15.70:0/19161512 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fa7fc0e2b80 con 0x7fa80806c470 2024-06-04T04:58:21.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.283+0000 7fa818cd7b80 1 -- 172.21.15.70:0/157219644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67aa20 msgr2=0x55862a67e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:21.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.283+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/157219644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67aa20 0x55862a67e150 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8009ca0 tx=0x7fa7f800f360 comp rx=0 tx=0).stop 2024-06-04T04:58:21.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.283+0000 7fa818cd7b80 1 -- 172.21.15.70:0/157219644 shutdown_connections 2024-06-04T04:58:21.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.283+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/157219644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a67e770 0x55862a67ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.283+0000 7fa80c9df700 1 --2- 172.21.15.70:0/19161512 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa7f00b2b10 0x7fa7f00b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.283+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/157219644 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67a070 0x55862a67a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.284+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/157219644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67aa20 0x55862a67e150 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.284+0000 7fa818cd7b80 1 -- 172.21.15.70:0/157219644 >> 172.21.15.70:0/157219644 conn(0x55862a59f080 msgr2=0x55862a66be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:21.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.284+0000 7fa818cd7b80 1 -- 172.21.15.70:0/157219644 shutdown_connections 2024-06-04T04:58:21.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.284+0000 7fa818cd7b80 1 -- 172.21.15.70:0/157219644 wait complete. 2024-06-04T04:58:21.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.284+0000 7fa818cd7b80 1 -- start start 2024-06-04T04:58:21.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.284+0000 7fa818cd7b80 1 Processor -- start 2024-06-04T04:58:21.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa80c9df700 1 --2- 172.21.15.70:0/19161512 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa7f00b2b10 0x7fa7f00b4fd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa800009fd0 tx=0x7fa80000b720 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:21.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa818cd7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67a070 0x55862a685b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa818cd7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67aa20 0x55862a682e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa818cd7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a67e770 0x55862a6833c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa818cd7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55862a4c6fa0 con 0x55862a67a070 2024-06-04T04:58:21.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa818cd7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55862a5627c0 con 0x55862a67e770 2024-06-04T04:58:21.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa818cd7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55862a686930 con 0x55862a67aa20 2024-06-04T04:58:21.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa818cd7b80 10 client.0 _ll_register_callbacks cb 0x55862a55b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:21.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa807fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67a070 0x55862a685b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa807fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67a070 0x55862a685b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46072/0 (socket says 172.21.15.70:46072) 2024-06-04T04:58:21.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa807fff700 1 -- 172.21.15.70:0/4256273533 learned_addr learned my addr 172.21.15.70:0/4256273533 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:21.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa80c9df700 1 --2- 172.21.15.70:0/4256273533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a67e770 0x55862a6833c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa807fff700 1 -- 172.21.15.70:0/4256273533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67aa20 msgr2=0x55862a682e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:21.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa8077fe700 1 --2- 172.21.15.70:0/4256273533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67aa20 0x55862a682e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa807fff700 1 --2- 172.21.15.70:0/4256273533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67aa20 0x55862a682e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa807fff700 1 -- 172.21.15.70:0/4256273533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a67e770 msgr2=0x55862a6833c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:21.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa807fff700 1 --2- 172.21.15.70:0/4256273533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a67e770 0x55862a6833c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa807fff700 1 -- 172.21.15.70:0/4256273533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55862a55f690 con 0x55862a67a070 2024-06-04T04:58:21.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa80c9df700 1 --2- 172.21.15.70:0/4256273533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a67e770 0x55862a6833c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:21.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.285+0000 7fa8077fe700 1 --2- 172.21.15.70:0/4256273533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67aa20 0x55862a682e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:21.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.286+0000 7fa807fff700 1 --2- 172.21.15.70:0/4256273533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67a070 0x55862a685b80 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8009630 tx=0x7fa7f80038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26134cffe1fbaab5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:21.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.286+0000 7fa7e77fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:21.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.286+0000 7fa7e77fe700 1 -- 172.21.15.70:0/4256273533 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f802a060 con 0x55862a67a070 2024-06-04T04:58:21.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.286+0000 7fa7e77fe700 1 -- 172.21.15.70:0/4256273533 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fa7f8005040 con 0x55862a67a070 2024-06-04T04:58:21.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.286+0000 7fa818cd7b80 1 -- 172.21.15.70:0/4256273533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7f800fe30 con 0x55862a67a070 2024-06-04T04:58:21.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.286+0000 7fa7e77fe700 1 -- 172.21.15.70:0/4256273533 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa7f8005240 con 0x55862a67a070 2024-06-04T04:58:21.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.286+0000 7fa818cd7b80 10 client.15735 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:21.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.286+0000 7fa818cd7b80 1 -- 172.21.15.70:0/4256273533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55862a691020 con 0x55862a67a070 2024-06-04T04:58:21.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.287+0000 7fa7e77fe700 1 -- 172.21.15.70:0/4256273533 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa7f8004be0 con 0x55862a67a070 2024-06-04T04:58:21.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.287+0000 7fa7e77fe700 1 -- 172.21.15.70:0/4256273533 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fa7f8003cb0 con 0x55862a67a070 2024-06-04T04:58:21.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.287+0000 7fa818cd7b80 10 client.15735 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:21.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.287+0000 7fa818cd7b80 10 client.15735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:21.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.287+0000 7fa818cd7b80 20 client.15735 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:21.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.287+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/4256273533 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55862a6935a0 0x55862a695990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.287+0000 7fa818cd7b80 4 client.15735 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:21.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.287+0000 7fa818cd7b80 1 -- 172.21.15.70:0/4256273533 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa7f800fe30 con 0x55862a6935a0 2024-06-04T04:58:21.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.287+0000 7fa8077fe700 1 --2- 172.21.15.70:0/4256273533 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55862a6935a0 0x55862a695990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.288+0000 7fa7e77fe700 10 client.15735 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:21.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.288+0000 7fa8077fe700 1 --2- 172.21.15.70:0/4256273533 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55862a6935a0 0x55862a695990 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35edfc1cdece7200 server_cookie=b90a516ec4aa2215 in_seq=0 out_seq=0 2024-06-04T04:58:21.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.289+0000 7fa7e77fe700 1 -- 172.21.15.70:0/4256273533 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fa7f800fe30 con 0x55862a6935a0 2024-06-04T04:58:21.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.289+0000 7fa7e77fe700 10 client.15735 handle_command_reply: tid=0 2024-06-04T04:58:21.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.289+0000 7fa818cd7b80 1 client.15735 shutdown 2024-06-04T04:58:21.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.289+0000 7fa818cd7b80 1 -- 172.21.15.70:0/4256273533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67a070 msgr2=0x55862a685b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:21.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.289+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/4256273533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67a070 0x55862a685b80 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8009630 tx=0x7fa7f80038b0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.289+0000 7fa818cd7b80 1 -- 172.21.15.70:0/4256273533 shutdown_connections 2024-06-04T04:58:21.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.289+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/4256273533 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55862a6935a0 0x55862a695990 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.289+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/4256273533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a67e770 0x55862a6833c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.289+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/4256273533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67a070 0x55862a685b80 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.289+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/4256273533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67aa20 0x55862a682e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.289+0000 7fa818cd7b80 1 -- 172.21.15.70:0/4256273533 >> 172.21.15.70:0/4256273533 conn(0x55862a59f080 msgr2=0x55862a66b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:21.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.289+0000 7fa7e77fe700 0 client.15735 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:21.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.289+0000 7fa818cd7b80 1 -- 172.21.15.70:0/4256273533 shutdown_connections 2024-06-04T04:58:21.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.289+0000 7fa818cd7b80 1 -- 172.21.15.70:0/4256273533 wait complete. 2024-06-04T04:58:21.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.289+0000 7fa818cd7b80 20 client.15735 trim_cache size 0 max 16384 2024-06-04T04:58:21.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.306+0000 7fa818cd7b80 1 Processor -- start 2024-06-04T04:58:21.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.306+0000 7fa818cd7b80 1 -- start start 2024-06-04T04:58:21.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.306+0000 7fa818cd7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67aa20 0x55862a67d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.306+0000 7fa818cd7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a67e770 0x55862a66bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.306+0000 7fa818cd7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a66c1d0 0x55862a682760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.306+0000 7fa818cd7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8081abc10 con 0x55862a66c1d0 2024-06-04T04:58:21.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.306+0000 7fa818cd7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8081ad750 con 0x55862a67e770 2024-06-04T04:58:21.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.306+0000 7fa818cd7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa8081ac2e0 con 0x55862a67aa20 2024-06-04T04:58:21.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.306+0000 7fa80c9df700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a66c1d0 0x55862a682760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.306+0000 7fa80c9df700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a66c1d0 0x55862a682760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46074/0 (socket says 172.21.15.70:46074) 2024-06-04T04:58:21.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.306+0000 7fa80c9df700 1 -- 172.21.15.70:0/2641911918 learned_addr learned my addr 172.21.15.70:0/2641911918 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:21.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.306+0000 7fa8077fe700 1 --2- 172.21.15.70:0/2641911918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a67e770 0x55862a66bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.306+0000 7fa807fff700 1 --2- 172.21.15.70:0/2641911918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67aa20 0x55862a67d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.307+0000 7fa80c9df700 1 -- 172.21.15.70:0/2641911918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67aa20 msgr2=0x55862a67d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:21.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.307+0000 7fa80c9df700 1 --2- 172.21.15.70:0/2641911918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67aa20 0x55862a67d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.307+0000 7fa80c9df700 1 -- 172.21.15.70:0/2641911918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a67e770 msgr2=0x55862a66bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:21.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.307+0000 7fa80c9df700 1 --2- 172.21.15.70:0/2641911918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a67e770 0x55862a66bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.307+0000 7fa80c9df700 1 -- 172.21.15.70:0/2641911918 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55862a681430 con 0x55862a66c1d0 2024-06-04T04:58:21.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.307+0000 7fa80c9df700 1 --2- 172.21.15.70:0/2641911918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a66c1d0 0x55862a682760 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa800015000 tx=0x7fa80001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2596440a424172be server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:21.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.307+0000 7fa7edffb700 1 -- 172.21.15.70:0/2641911918 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8000074e0 con 0x55862a66c1d0 2024-06-04T04:58:21.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.307+0000 7fa7edffb700 1 -- 172.21.15.70:0/2641911918 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa800007b10 con 0x55862a66c1d0 2024-06-04T04:58:21.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.307+0000 7fa7edffb700 1 -- 172.21.15.70:0/2641911918 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80002ab70 con 0x55862a66c1d0 2024-06-04T04:58:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.308+0000 7fa818cd7b80 1 -- 172.21.15.70:0/2641911918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a66c1d0 msgr2=0x55862a682760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.308+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/2641911918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a66c1d0 0x55862a682760 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa800015000 tx=0x7fa80001f000 comp rx=0 tx=0).stop 2024-06-04T04:58:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.308+0000 7fa818cd7b80 1 -- 172.21.15.70:0/2641911918 shutdown_connections 2024-06-04T04:58:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.308+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/2641911918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a66c1d0 0x55862a682760 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.308+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/2641911918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67aa20 0x55862a67d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.308+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/2641911918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a67e770 0x55862a66bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.308+0000 7fa818cd7b80 1 -- 172.21.15.70:0/2641911918 >> 172.21.15.70:0/2641911918 conn(0x55862a59f080 msgr2=0x55862a684f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.308+0000 7fa818cd7b80 1 -- 172.21.15.70:0/2641911918 shutdown_connections 2024-06-04T04:58:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.308+0000 7fa818cd7b80 1 -- 172.21.15.70:0/2641911918 wait complete. 2024-06-04T04:58:21.313 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:21.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.308+0000 7fa818cd7b80 1 -- start start 2024-06-04T04:58:21.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.309+0000 7fa818cd7b80 1 Processor -- start 2024-06-04T04:58:21.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.309+0000 7fa818cd7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a66c1d0 0x55862a6f1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.309+0000 7fa818cd7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67aa20 0x55862a6f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.309+0000 7fa818cd7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67e770 0x55862a6fcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.309+0000 7fa818cd7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8081ad080 con 0x55862a67aa20 2024-06-04T04:58:21.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.309+0000 7fa818cd7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8081ac9b0 con 0x55862a66c1d0 2024-06-04T04:58:21.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.309+0000 7fa818cd7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa8081a9be0 con 0x55862a67e770 2024-06-04T04:58:21.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.309+0000 7fa818cd7b80 10 client.0 _ll_register_callbacks cb 0x55862a55b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:21.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.309+0000 7fa8077fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67aa20 0x55862a6f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.309+0000 7fa8077fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67aa20 0x55862a6f22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46080/0 (socket says 172.21.15.70:46080) 2024-06-04T04:58:21.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.309+0000 7fa8077fe700 1 -- 172.21.15.70:0/1502443262 learned_addr learned my addr 172.21.15.70:0/1502443262 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:21.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.309+0000 7fa807fff700 1 --2- 172.21.15.70:0/1502443262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a66c1d0 0x55862a6f1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.309+0000 7fa8077fe700 1 -- 172.21.15.70:0/1502443262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67e770 msgr2=0x55862a6fcdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:21.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.309+0000 7fa80c9df700 1 --2- 172.21.15.70:0/1502443262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67e770 0x55862a6fcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.309+0000 7fa8077fe700 1 --2- 172.21.15.70:0/1502443262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67e770 0x55862a6fcdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.310+0000 7fa8077fe700 1 -- 172.21.15.70:0/1502443262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a66c1d0 msgr2=0x55862a6f1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:21.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.310+0000 7fa8077fe700 1 --2- 172.21.15.70:0/1502443262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a66c1d0 0x55862a6f1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.310+0000 7fa8077fe700 1 -- 172.21.15.70:0/1502443262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55862a6fd330 con 0x55862a67aa20 2024-06-04T04:58:21.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.310+0000 7fa807fff700 1 --2- 172.21.15.70:0/1502443262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a66c1d0 0x55862a6f1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:21.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.310+0000 7fa8077fe700 1 --2- 172.21.15.70:0/1502443262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67aa20 0x55862a6f22b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc0e0260 tx=0x7fa7fc00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472175e9af03e18e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:21.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.310+0000 7fa7e77fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:21.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.310+0000 7fa7e77fe700 1 -- 172.21.15.70:0/1502443262 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc0a53a0 con 0x55862a67aa20 2024-06-04T04:58:21.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.310+0000 7fa7e77fe700 1 -- 172.21.15.70:0/1502443262 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fa7fc0aa080 con 0x55862a67aa20 2024-06-04T04:58:21.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.310+0000 7fa818cd7b80 1 -- 172.21.15.70:0/1502443262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa800007b10 con 0x55862a67aa20 2024-06-04T04:58:21.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.310+0000 7fa7e77fe700 1 -- 172.21.15.70:0/1502443262 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa7fc0b23b0 con 0x55862a67aa20 2024-06-04T04:58:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.310+0000 7fa818cd7b80 10 client.15741 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.310+0000 7fa818cd7b80 1 -- 172.21.15.70:0/1502443262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55862a6fe5c0 con 0x55862a67aa20 2024-06-04T04:58:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.310+0000 7fa7e77fe700 1 -- 172.21.15.70:0/1502443262 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa7fc0c42e0 con 0x55862a67aa20 2024-06-04T04:58:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.311+0000 7fa7e77fe700 1 -- 172.21.15.70:0/1502443262 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fa7fc0e9070 con 0x55862a67aa20 2024-06-04T04:58:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.311+0000 7fa818cd7b80 10 client.15741 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.311+0000 7fa818cd7b80 10 client.15741 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.311+0000 7fa818cd7b80 20 client.15741 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.311+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/1502443262 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55862a6fed00 0x55862a701120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.311+0000 7fa818cd7b80 4 client.15741 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T04:58:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.311+0000 7fa818cd7b80 1 -- 172.21.15.70:0/1502443262 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fa800007b10 con 0x55862a6fed00 2024-06-04T04:58:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.311+0000 7fa807fff700 1 --2- 172.21.15.70:0/1502443262 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55862a6fed00 0x55862a701120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.312+0000 7fa807fff700 1 --2- 172.21.15.70:0/1502443262 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55862a6fed00 0x55862a701120 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e870d0c6eb99944 server_cookie=ed7e922aeb66072d in_seq=0 out_seq=0 2024-06-04T04:58:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.312+0000 7fa7e77fe700 10 client.15741 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.312+0000 7fa7e77fe700 1 -- 172.21.15.70:0/1502443262 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fa800007b10 con 0x55862a6fed00 2024-06-04T04:58:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.312+0000 7fa7e77fe700 10 client.15741 handle_command_reply: tid=0 2024-06-04T04:58:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.312+0000 7fa818cd7b80 1 client.15741 shutdown 2024-06-04T04:58:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.312+0000 7fa818cd7b80 1 -- 172.21.15.70:0/1502443262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67aa20 msgr2=0x55862a6f22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.312+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/1502443262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67aa20 0x55862a6f22b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc0e0260 tx=0x7fa7fc00cf20 comp rx=0 tx=0).stop 2024-06-04T04:58:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.312+0000 7fa818cd7b80 1 -- 172.21.15.70:0/1502443262 shutdown_connections 2024-06-04T04:58:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.312+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/1502443262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55862a67e770 0x55862a6fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.312+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/1502443262 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55862a6fed00 0x55862a701120 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.312+0000 7fa7e77fe700 0 client.15741 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.312+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/1502443262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55862a66c1d0 0x55862a6f1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.312+0000 7fa818cd7b80 1 --2- 172.21.15.70:0/1502443262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55862a67aa20 0x55862a6f22b0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.312+0000 7fa818cd7b80 1 -- 172.21.15.70:0/1502443262 >> 172.21.15.70:0/1502443262 conn(0x55862a59f080 msgr2=0x55862a6846b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.312+0000 7fa818cd7b80 1 -- 172.21.15.70:0/1502443262 shutdown_connections 2024-06-04T04:58:21.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.312+0000 7fa818cd7b80 1 -- 172.21.15.70:0/1502443262 wait complete. 2024-06-04T04:58:21.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.312+0000 7fa818cd7b80 20 client.15741 trim_cache size 0 max 16384 2024-06-04T04:58:21.320 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T04:58:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.313+0000 7fa80e442700 1 -- 172.21.15.70:0/19161512 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa7f00b2b10 msgr2=0x7fa7f00b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.313+0000 7fa80e442700 1 --2- 172.21.15.70:0/19161512 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa7f00b2b10 0x7fa7f00b4fd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa800009fd0 tx=0x7fa80000b720 comp rx=0 tx=0).stop 2024-06-04T04:58:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.313+0000 7fa80e442700 1 -- 172.21.15.70:0/19161512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa80806c470 msgr2=0x7fa8081a1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.313+0000 7fa80e442700 1 --2- 172.21.15.70:0/19161512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa80806c470 0x7fa8081a1e70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00e9c0 tx=0x7fa7fc00ecd0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.313+0000 7fa80e442700 1 -- 172.21.15.70:0/19161512 shutdown_connections 2024-06-04T04:58:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.313+0000 7fa80e442700 1 --2- 172.21.15.70:0/19161512 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa7f00b2b10 0x7fa7f00b4fd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.313+0000 7fa80e442700 1 --2- 172.21.15.70:0/19161512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8081095a0 0x7fa80819baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.313+0000 7fa80e442700 1 --2- 172.21.15.70:0/19161512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa80806bab0 0x7fa8081a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.313+0000 7fa80e442700 1 --2- 172.21.15.70:0/19161512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa80806c470 0x7fa8081a1e70 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.313+0000 7fa80e442700 1 -- 172.21.15.70:0/19161512 >> 172.21.15.70:0/19161512 conn(0x7fa8080793b0 msgr2=0x7fa80810e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.313+0000 7fa80e442700 1 -- 172.21.15.70:0/19161512 shutdown_connections 2024-06-04T04:58:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:21.313+0000 7fa80e442700 1 -- 172.21.15.70:0/19161512 wait complete. 2024-06-04T04:58:21.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:21 smithi112 ceph-mon[36453]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-04T04:58:21.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:21 smithi098 ceph-mon[35569]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-04T04:58:23.029 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T04:58:23.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T04:58:23.333 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:23 smithi070 ceph-mon[32631]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-06-04T04:58:23.333 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:23 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:58:23.333 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:23 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:58:23.333 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:23 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:58:23.333 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:23 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:58:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:23.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:23 smithi112 ceph-mon[36453]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-06-04T04:58:23.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:23 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:58:23.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:23 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:58:23.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:23 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:58:23.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:23 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:58:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:23 smithi098 ceph-mon[35569]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-06-04T04:58:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:23 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:58:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:23 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:58:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:23 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:58:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:23 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:58:24.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.013+0000 7f2a89840700 1 -- 172.21.15.70:0/3016950752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a84103f50 msgr2=0x7f2a84110430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:24.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.013+0000 7f2a89840700 1 --2- 172.21.15.70:0/3016950752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a84103f50 0x7f2a84110430 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009990 tx=0x7f2a7c009ca0 comp rx=0 tx=0).stop 2024-06-04T04:58:24.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.013+0000 7f2a89840700 1 -- 172.21.15.70:0/3016950752 shutdown_connections 2024-06-04T04:58:24.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.013+0000 7f2a89840700 1 --2- 172.21.15.70:0/3016950752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a84103f50 0x7f2a84110430 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:24.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.013+0000 7f2a89840700 1 --2- 172.21.15.70:0/3016950752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a841096a0 0x7f2a84109a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:24.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.013+0000 7f2a89840700 1 --2- 172.21.15.70:0/3016950752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a84103590 0x7f2a84103a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:24.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.013+0000 7f2a89840700 1 -- 172.21.15.70:0/3016950752 >> 172.21.15.70:0/3016950752 conn(0x7f2a8407a6f0 msgr2=0x7f2a8407ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:24.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.013+0000 7f2a89840700 1 -- 172.21.15.70:0/3016950752 shutdown_connections 2024-06-04T04:58:24.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.013+0000 7f2a89840700 1 -- 172.21.15.70:0/3016950752 wait complete. 2024-06-04T04:58:24.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.014+0000 7f2a89840700 1 Processor -- start 2024-06-04T04:58:24.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.014+0000 7f2a89840700 1 -- start start 2024-06-04T04:58:24.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a89840700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a84103590 0x7f2a8419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:24.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a89840700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a84103f50 0x7f2a8419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:24.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a89840700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a841096a0 0x7f2a84199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:24.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a89840700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2a841135e0 con 0x7f2a84103590 2024-06-04T04:58:24.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a89840700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a84113460 con 0x7f2a84103f50 2024-06-04T04:58:24.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a89840700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2a84113760 con 0x7f2a841096a0 2024-06-04T04:58:24.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a82ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a84103590 0x7f2a8419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:24.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a82ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a84103590 0x7f2a8419f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46096/0 (socket says 172.21.15.70:46096) 2024-06-04T04:58:24.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a82ffd700 1 -- 172.21.15.70:0/1454765452 learned_addr learned my addr 172.21.15.70:0/1454765452 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:24.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a837fe700 1 --2- 172.21.15.70:0/1454765452 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a841096a0 0x7f2a84199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:24.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a827fc700 1 --2- 172.21.15.70:0/1454765452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a84103f50 0x7f2a8419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:24.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a82ffd700 1 -- 172.21.15.70:0/1454765452 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a841096a0 msgr2=0x7f2a84199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:24.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a82ffd700 1 --2- 172.21.15.70:0/1454765452 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a841096a0 0x7f2a84199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:24.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a82ffd700 1 -- 172.21.15.70:0/1454765452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a84103f50 msgr2=0x7f2a8419fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:24.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a82ffd700 1 --2- 172.21.15.70:0/1454765452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a84103f50 0x7f2a8419fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:24.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a82ffd700 1 -- 172.21.15.70:0/1454765452 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a8419a060 con 0x7f2a84103590 2024-06-04T04:58:24.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a827fc700 1 --2- 172.21.15.70:0/1454765452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a84103f50 0x7f2a8419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:24.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a82ffd700 1 --2- 172.21.15.70:0/1454765452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a84103590 0x7f2a8419f6b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f2a740060b0 tx=0x7f2a7400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:24.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.015+0000 7f2a8883e700 1 -- 172.21.15.70:0/1454765452 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7401a070 con 0x7f2a84103590 2024-06-04T04:58:24.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.016+0000 7f2a8883e700 1 -- 172.21.15.70:0/1454765452 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2a7400ddd0 con 0x7f2a84103590 2024-06-04T04:58:24.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.016+0000 7f2a8883e700 1 -- 172.21.15.70:0/1454765452 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a74014690 con 0x7f2a84103590 2024-06-04T04:58:24.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.016+0000 7f2a89840700 1 -- 172.21.15.70:0/1454765452 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a8419a350 con 0x7f2a84103590 2024-06-04T04:58:24.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.016+0000 7f2a89840700 1 -- 172.21.15.70:0/1454765452 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a841a6510 con 0x7f2a84103590 2024-06-04T04:58:24.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.017+0000 7f2a89840700 1 -- 172.21.15.70:0/1454765452 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a84104cb0 con 0x7f2a84103590 2024-06-04T04:58:24.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.018+0000 7f2a8883e700 1 -- 172.21.15.70:0/1454765452 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 149885+0+0 (secure 0 0 0) 0x7f2a7401d070 con 0x7f2a84103590 2024-06-04T04:58:24.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.019+0000 7f2a8883e700 1 --2- 172.21.15.70:0/1454765452 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2a700b2b10 0x7f2a700b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:24.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.019+0000 7f2a827fc700 1 --2- 172.21.15.70:0/1454765452 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2a700b2b10 0x7f2a700b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:24.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.019+0000 7f2a8883e700 1 -- 172.21.15.70:0/1454765452 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f2a740e2600 con 0x7f2a84103590 2024-06-04T04:58:24.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.019+0000 7f2a827fc700 1 --2- 172.21.15.70:0/1454765452 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2a700b2b10 0x7f2a700b4fd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2a8407ac80 tx=0x7f2a6c00b2c0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:24.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.022+0000 7f2a8883e700 1 -- 172.21.15.70:0/1454765452 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2a74010070 con 0x7f2a84103590 2024-06-04T04:58:24.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.314+0000 7f2a89840700 1 -- 172.21.15.70:0/1454765452 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a8402e560 con 0x7f2a84103590 2024-06-04T04:58:24.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.315+0000 7f2a8883e700 1 -- 172.21.15.70:0/1454765452 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f2a740a9d90 con 0x7f2a84103590 2024-06-04T04:58:24.318 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:58:24.318 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:57:09.013178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T04:58:24.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.318+0000 7f2a89840700 1 -- 172.21.15.70:0/1454765452 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2a700b2b10 msgr2=0x7f2a700b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:24.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.318+0000 7f2a89840700 1 --2- 172.21.15.70:0/1454765452 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2a700b2b10 0x7f2a700b4fd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2a8407ac80 tx=0x7f2a6c00b2c0 comp rx=0 tx=0).stop 2024-06-04T04:58:24.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.318+0000 7f2a89840700 1 -- 172.21.15.70:0/1454765452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a84103590 msgr2=0x7f2a8419f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:24.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.318+0000 7f2a89840700 1 --2- 172.21.15.70:0/1454765452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a84103590 0x7f2a8419f6b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f2a740060b0 tx=0x7f2a7400d3b0 comp rx=0 tx=0).stop 2024-06-04T04:58:24.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.318+0000 7f2a89840700 1 -- 172.21.15.70:0/1454765452 shutdown_connections 2024-06-04T04:58:24.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.318+0000 7f2a89840700 1 --2- 172.21.15.70:0/1454765452 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2a700b2b10 0x7f2a700b4fd0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:24.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.318+0000 7f2a89840700 1 --2- 172.21.15.70:0/1454765452 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a841096a0 0x7f2a84199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:24.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.318+0000 7f2a89840700 1 --2- 172.21.15.70:0/1454765452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a84103590 0x7f2a8419f6b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:24.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.318+0000 7f2a89840700 1 --2- 172.21.15.70:0/1454765452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a84103f50 0x7f2a8419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:24.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.318+0000 7f2a89840700 1 -- 172.21.15.70:0/1454765452 >> 172.21.15.70:0/1454765452 conn(0x7f2a8407a6f0 msgr2=0x7f2a84102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:24.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.318+0000 7f2a89840700 1 -- 172.21.15.70:0/1454765452 shutdown_connections 2024-06-04T04:58:24.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:24.318+0000 7f2a89840700 1 -- 172.21.15.70:0/1454765452 wait complete. 2024-06-04T04:58:24.321 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-06-04T04:58:25.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:25 smithi070 ceph-mon[32631]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:58:25.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:25 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1454765452' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:58:25.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:25 smithi112 ceph-mon[36453]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:58:25.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:25 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1454765452' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:58:25.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:25 smithi098 ceph-mon[35569]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:58:25.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:25 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1454765452' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:58:25.823 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 154.9 --await-for 45.0' 2024-06-04T04:58:25.823 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 154.9 --await-for 45.0 2024-06-04T04:58:26.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.266+0000 7f3e48e9a700 1 -- 172.21.15.70:0/2435703208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e44102c40 msgr2=0x7f3e441030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:26.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.266+0000 7f3e48e9a700 1 --2- 172.21.15.70:0/2435703208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e44102c40 0x7f3e441030c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3e38009990 tx=0x7f3e38009ca0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.266+0000 7f3e48e9a700 1 -- 172.21.15.70:0/2435703208 shutdown_connections 2024-06-04T04:58:26.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.266+0000 7f3e48e9a700 1 --2- 172.21.15.70:0/2435703208 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e44103600 0x7f3e4410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.266+0000 7f3e48e9a700 1 --2- 172.21.15.70:0/2435703208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e441084c0 0x7f3e441088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.266+0000 7f3e48e9a700 1 --2- 172.21.15.70:0/2435703208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e44102c40 0x7f3e441030c0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.266+0000 7f3e48e9a700 1 -- 172.21.15.70:0/2435703208 >> 172.21.15.70:0/2435703208 conn(0x7f3e44078ab0 msgr2=0x7f3e440ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:26.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.266+0000 7f3e48e9a700 1 -- 172.21.15.70:0/2435703208 shutdown_connections 2024-06-04T04:58:26.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.266+0000 7f3e48e9a700 1 -- 172.21.15.70:0/2435703208 wait complete. 2024-06-04T04:58:26.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.267+0000 7f3e48e9a700 1 Processor -- start 2024-06-04T04:58:26.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.267+0000 7f3e48e9a700 1 -- start start 2024-06-04T04:58:26.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.268+0000 7f3e48e9a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e44102c40 0x7f3e4419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.268+0000 7f3e48e9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e44103600 0x7f3e4419fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.268+0000 7f3e48e9a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e441084c0 0x7f3e44199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.268+0000 7f3e48e9a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e44113620 con 0x7f3e441084c0 2024-06-04T04:58:26.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.268+0000 7f3e48e9a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e441134a0 con 0x7f3e44103600 2024-06-04T04:58:26.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.268+0000 7f3e48e9a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3e441137a0 con 0x7f3e44102c40 2024-06-04T04:58:26.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.268+0000 7f3e42d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e441084c0 0x7f3e44199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.268+0000 7f3e42d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e441084c0 0x7f3e44199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35738/0 (socket says 172.21.15.70:35738) 2024-06-04T04:58:26.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.268+0000 7f3e4259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e44102c40 0x7f3e4419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.268+0000 7f3e42d9d700 1 -- 172.21.15.70:0/1543362362 learned_addr learned my addr 172.21.15.70:0/1543362362 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:26.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.268+0000 7f3e41d9b700 1 --2- 172.21.15.70:0/1543362362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e44103600 0x7f3e4419fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.268+0000 7f3e41d9b700 1 -- 172.21.15.70:0/1543362362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e44102c40 msgr2=0x7f3e4419f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:26.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.268+0000 7f3e41d9b700 1 --2- 172.21.15.70:0/1543362362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e44102c40 0x7f3e4419f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.268+0000 7f3e41d9b700 1 -- 172.21.15.70:0/1543362362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e441084c0 msgr2=0x7f3e44199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:26.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.268+0000 7f3e41d9b700 1 --2- 172.21.15.70:0/1543362362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e441084c0 0x7f3e44199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.269+0000 7f3e41d9b700 1 -- 172.21.15.70:0/1543362362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e4419a160 con 0x7f3e44103600 2024-06-04T04:58:26.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.269+0000 7f3e42d9d700 1 --2- 172.21.15.70:0/1543362362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e441084c0 0x7f3e44199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:26.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.269+0000 7f3e4259c700 1 --2- 172.21.15.70:0/1543362362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e44102c40 0x7f3e4419f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:26.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.269+0000 7f3e41d9b700 1 --2- 172.21.15.70:0/1543362362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e44103600 0x7f3e4419fd30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3e380098e0 tx=0x7f3e380047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:26.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.269+0000 7f3e337fe700 1 -- 172.21.15.70:0/1543362362 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e38026070 con 0x7f3e44103600 2024-06-04T04:58:26.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.269+0000 7f3e337fe700 1 -- 172.21.15.70:0/1543362362 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e38021410 con 0x7f3e44103600 2024-06-04T04:58:26.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.269+0000 7f3e48e9a700 1 -- 172.21.15.70:0/1543362362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e4419a3f0 con 0x7f3e44103600 2024-06-04T04:58:26.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.269+0000 7f3e337fe700 1 -- 172.21.15.70:0/1543362362 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e38017650 con 0x7f3e44103600 2024-06-04T04:58:26.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.269+0000 7f3e48e9a700 1 -- 172.21.15.70:0/1543362362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e441a66c0 con 0x7f3e44103600 2024-06-04T04:58:26.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.272+0000 7f3e337fe700 1 -- 172.21.15.70:0/1543362362 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 149885+0+0 (secure 0 0 0) 0x7f3e38029070 con 0x7f3e44103600 2024-06-04T04:58:26.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.273+0000 7f3e337fe700 1 --2- 172.21.15.70:0/1543362362 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e2c0b2b10 0x7f3e2c0b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.273+0000 7f3e4259c700 1 --2- 172.21.15.70:0/1543362362 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e2c0b2b10 0x7f3e2c0b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.273+0000 7f3e337fe700 1 -- 172.21.15.70:0/1543362362 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f3e380e36c0 con 0x7f3e44103600 2024-06-04T04:58:26.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.274+0000 7f3e4259c700 1 --2- 172.21.15.70:0/1543362362 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e2c0b2b10 0x7f3e2c0b4fd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3e441076f0 tx=0x7f3e34009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:26.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.274+0000 7f3e5372fb80 1 Processor -- start 2024-06-04T04:58:26.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.275+0000 7f3e5372fb80 1 -- start start 2024-06-04T04:58:26.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.275+0000 7f3e5372fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed38e0 0x558895ed3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.275+0000 7f3e5372fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895ed4290 0x558895ed79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.275+0000 7f3e5372fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895ed7fe0 0x558895ed8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.275+0000 7f3e5372fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558895e32d10 con 0x558895ed38e0 2024-06-04T04:58:26.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.275+0000 7f3e4259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed38e0 0x558895ed3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.275+0000 7f3e5372fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558895e228b0 con 0x558895ed4290 2024-06-04T04:58:26.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.275+0000 7f3e4259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed38e0 0x558895ed3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35752/0 (socket says 172.21.15.70:35752) 2024-06-04T04:58:26.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.275+0000 7f3e4259c700 1 -- 172.21.15.70:0/82540655 learned_addr learned my addr 172.21.15.70:0/82540655 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:26.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.275+0000 7f3e5372fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558895e268f0 con 0x558895ed7fe0 2024-06-04T04:58:26.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.275+0000 7f3e41d9b700 1 --2- 172.21.15.70:0/82540655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895ed4290 0x558895ed79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.275+0000 7f3e42d9d700 1 --2- 172.21.15.70:0/82540655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895ed7fe0 0x558895ed8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.276+0000 7f3e4259c700 1 -- 172.21.15.70:0/82540655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895ed7fe0 msgr2=0x558895ed8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:26.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.276+0000 7f3e4259c700 1 --2- 172.21.15.70:0/82540655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895ed7fe0 0x558895ed8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.276+0000 7f3e4259c700 1 -- 172.21.15.70:0/82540655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895ed4290 msgr2=0x558895ed79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:26.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.276+0000 7f3e4259c700 1 --2- 172.21.15.70:0/82540655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895ed4290 0x558895ed79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.276+0000 7f3e4259c700 1 -- 172.21.15.70:0/82540655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558895d72950 con 0x558895ed38e0 2024-06-04T04:58:26.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.276+0000 7f3e4259c700 1 --2- 172.21.15.70:0/82540655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed38e0 0x558895ed3cc0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3e340043a0 tx=0x7f3e34014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c874e66443b32303 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:26.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.276+0000 7f3e30ff9700 1 -- 172.21.15.70:0/82540655 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e34003dc0 con 0x558895ed38e0 2024-06-04T04:58:26.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.276+0000 7f3e30ff9700 1 -- 172.21.15.70:0/82540655 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e34026890 con 0x558895ed38e0 2024-06-04T04:58:26.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.276+0000 7f3e30ff9700 1 -- 172.21.15.70:0/82540655 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3401e870 con 0x558895ed38e0 2024-06-04T04:58:26.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.277+0000 7f3e5372fb80 1 -- 172.21.15.70:0/82540655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed38e0 msgr2=0x558895ed3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:26.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.277+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/82540655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed38e0 0x558895ed3cc0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3e340043a0 tx=0x7f3e34014000 comp rx=0 tx=0).stop 2024-06-04T04:58:26.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.277+0000 7f3e5372fb80 1 -- 172.21.15.70:0/82540655 shutdown_connections 2024-06-04T04:58:26.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.277+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/82540655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895ed7fe0 0x558895ed8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.277+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/82540655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed38e0 0x558895ed3cc0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.277+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/82540655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895ed4290 0x558895ed79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.277+0000 7f3e5372fb80 1 -- 172.21.15.70:0/82540655 >> 172.21.15.70:0/82540655 conn(0x558895e01100 msgr2=0x558895ecdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:26.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.277+0000 7f3e5372fb80 1 -- 172.21.15.70:0/82540655 shutdown_connections 2024-06-04T04:58:26.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.277+0000 7f3e5372fb80 1 -- 172.21.15.70:0/82540655 wait complete. 2024-06-04T04:58:26.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.278+0000 7f3e5372fb80 1 -- start start 2024-06-04T04:58:26.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.278+0000 7f3e5372fb80 1 Processor -- start 2024-06-04T04:58:26.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.278+0000 7f3e5372fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed4290 0x558895edf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.278+0000 7f3e5372fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895ed7fe0 0x558895edc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.278+0000 7f3e5372fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895edcec0 0x558895edd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.278+0000 7f3e5372fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558895d29020 con 0x558895ed4290 2024-06-04T04:58:26.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.278+0000 7f3e5372fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558895dc4840 con 0x558895edcec0 2024-06-04T04:58:26.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.278+0000 7f3e5372fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558895ee01a0 con 0x558895ed7fe0 2024-06-04T04:58:26.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.279+0000 7f3e5372fb80 10 client.0 _ll_register_callbacks cb 0x558895dbd9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:26.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.279+0000 7f3e4259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed4290 0x558895edf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.279+0000 7f3e4259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed4290 0x558895edf680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35766/0 (socket says 172.21.15.70:35766) 2024-06-04T04:58:26.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.279+0000 7f3e4259c700 1 -- 172.21.15.70:0/3086873470 learned_addr learned my addr 172.21.15.70:0/3086873470 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:26.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.279+0000 7f3e41d9b700 1 --2- 172.21.15.70:0/3086873470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895ed7fe0 0x558895edc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.279+0000 7f3e42d9d700 1 --2- 172.21.15.70:0/3086873470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895edcec0 0x558895edd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.279+0000 7f3e41d9b700 1 -- 172.21.15.70:0/3086873470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895edcec0 msgr2=0x558895edd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:26.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.279+0000 7f3e41d9b700 1 --2- 172.21.15.70:0/3086873470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895edcec0 0x558895edd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.279+0000 7f3e41d9b700 1 -- 172.21.15.70:0/3086873470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed4290 msgr2=0x558895edf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:26.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.279+0000 7f3e41d9b700 1 --2- 172.21.15.70:0/3086873470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed4290 0x558895edf680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.279+0000 7f3e41d9b700 1 -- 172.21.15.70:0/3086873470 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558895dc1710 con 0x558895ed7fe0 2024-06-04T04:58:26.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.279+0000 7f3e4259c700 1 --2- 172.21.15.70:0/3086873470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed4290 0x558895edf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:26.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.279+0000 7f3e41d9b700 1 --2- 172.21.15.70:0/3086873470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895ed7fe0 0x558895edc980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3e380e0c90 tx=0x7f3e38021be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c8b6357542559a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:26.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.279+0000 7f3e29ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:58:26.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.280+0000 7f3e29ffb700 1 -- 172.21.15.70:0/3086873470 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e380a9f50 con 0x558895ed7fe0 2024-06-04T04:58:26.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.280+0000 7f3e29ffb700 1 -- 172.21.15.70:0/3086873470 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f3e380b3330 con 0x558895ed7fe0 2024-06-04T04:58:26.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.280+0000 7f3e5372fb80 1 -- 172.21.15.70:0/3086873470 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e34026890 con 0x558895ed7fe0 2024-06-04T04:58:26.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.280+0000 7f3e29ffb700 1 -- 172.21.15.70:0/3086873470 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3e380aa2f0 con 0x558895ed7fe0 2024-06-04T04:58:26.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.280+0000 7f3e5372fb80 10 client.25282 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:26.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.280+0000 7f3e5372fb80 1 -- 172.21.15.70:0/3086873470 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558895eeaf10 con 0x558895ed7fe0 2024-06-04T04:58:26.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.280+0000 7f3e29ffb700 1 -- 172.21.15.70:0/3086873470 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e380aaee0 con 0x558895ed7fe0 2024-06-04T04:58:26.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.280+0000 7f3e29ffb700 1 -- 172.21.15.70:0/3086873470 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f3e38092110 con 0x558895ed7fe0 2024-06-04T04:58:26.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.281+0000 7f3e5372fb80 10 client.25282 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:26.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.281+0000 7f3e5372fb80 10 client.25282 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:26.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.281+0000 7f3e5372fb80 20 client.25282 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:26.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.281+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/3086873470 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558895eed570 0x558895eef960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.281+0000 7f3e5372fb80 4 client.25282 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:26.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.281+0000 7f3e5372fb80 1 -- 172.21.15.70:0/3086873470 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3e34026890 con 0x558895eed570 2024-06-04T04:58:26.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.281+0000 7f3e4259c700 1 --2- 172.21.15.70:0/3086873470 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558895eed570 0x558895eef960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.282+0000 7f3e4259c700 1 --2- 172.21.15.70:0/3086873470 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558895eed570 0x558895eef960 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4647c4930a8a20a server_cookie=bea1ba665d2525dc in_seq=0 out_seq=0 2024-06-04T04:58:26.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.282+0000 7f3e29ffb700 10 client.25282 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:26.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.282+0000 7f3e29ffb700 1 -- 172.21.15.70:0/3086873470 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f3e34026890 con 0x558895eed570 2024-06-04T04:58:26.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.282+0000 7f3e29ffb700 10 client.25282 handle_command_reply: tid=0 2024-06-04T04:58:26.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.282+0000 7f3e5372fb80 1 client.25282 shutdown 2024-06-04T04:58:26.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.282+0000 7f3e5372fb80 1 -- 172.21.15.70:0/3086873470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895ed7fe0 msgr2=0x558895edc980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:26.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.282+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/3086873470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895ed7fe0 0x558895edc980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3e380e0c90 tx=0x7f3e38021be0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.283+0000 7f3e5372fb80 1 -- 172.21.15.70:0/3086873470 shutdown_connections 2024-06-04T04:58:26.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.283+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/3086873470 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558895eed570 0x558895eef960 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.283+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/3086873470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895edcec0 0x558895edd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.283+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/3086873470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed4290 0x558895edf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.283+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/3086873470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895ed7fe0 0x558895edc980 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.283+0000 7f3e29ffb700 0 client.25282 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:26.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.283+0000 7f3e5372fb80 1 -- 172.21.15.70:0/3086873470 >> 172.21.15.70:0/3086873470 conn(0x558895e01100 msgr2=0x558895ecd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:26.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.283+0000 7f3e5372fb80 1 -- 172.21.15.70:0/3086873470 shutdown_connections 2024-06-04T04:58:26.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.283+0000 7f3e5372fb80 1 -- 172.21.15.70:0/3086873470 wait complete. 2024-06-04T04:58:26.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.284+0000 7f3e5372fb80 20 client.25282 trim_cache size 0 max 16384 2024-06-04T04:58:26.308 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.325+0000 7f3e5372fb80 1 Processor -- start 2024-06-04T04:58:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.325+0000 7f3e5372fb80 1 -- start start 2024-06-04T04:58:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.326+0000 7f3e5372fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed4290 0x558895ecdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.326+0000 7f3e5372fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895ed7fe0 0x558895ed9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.326+0000 7f3e5372fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895edcec0 0x558895eda310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.326+0000 7f3e5372fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e441adbd0 con 0x558895ed4290 2024-06-04T04:58:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.326+0000 7f3e5372fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e441af710 con 0x558895ed7fe0 2024-06-04T04:58:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.326+0000 7f3e4259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed4290 0x558895ecdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.326+0000 7f3e4259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed4290 0x558895ecdd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35782/0 (socket says 172.21.15.70:35782) 2024-06-04T04:58:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.326+0000 7f3e4259c700 1 -- 172.21.15.70:0/3356499108 learned_addr learned my addr 172.21.15.70:0/3356499108 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.326+0000 7f3e5372fb80 1 -- 172.21.15.70:0/3356499108 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3e441ae2a0 con 0x558895edcec0 2024-06-04T04:58:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.326+0000 7f3e41d9b700 1 --2- 172.21.15.70:0/3356499108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895ed7fe0 0x558895ed9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.326+0000 7f3e42d9d700 1 --2- 172.21.15.70:0/3356499108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895edcec0 0x558895eda310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.326+0000 7f3e41d9b700 1 -- 172.21.15.70:0/3356499108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895edcec0 msgr2=0x558895eda310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.326+0000 7f3e41d9b700 1 --2- 172.21.15.70:0/3356499108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895edcec0 0x558895eda310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.326+0000 7f3e41d9b700 1 -- 172.21.15.70:0/3356499108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed4290 msgr2=0x558895ecdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.326+0000 7f3e41d9b700 1 --2- 172.21.15.70:0/3356499108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed4290 0x558895ecdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.326+0000 7f3e41d9b700 1 -- 172.21.15.70:0/3356499108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558895ed7d30 con 0x558895ed7fe0 2024-06-04T04:58:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.326+0000 7f3e4259c700 1 --2- 172.21.15.70:0/3356499108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed4290 0x558895ecdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.327+0000 7f3e41d9b700 1 --2- 172.21.15.70:0/3356499108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895ed7fe0 0x558895ed9dd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3e380a6860 tx=0x7f3e380a66a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1616390fc1a006ef server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.327+0000 7f3e2bfff700 1 -- 172.21.15.70:0/3356499108 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e380aeb90 con 0x558895ed7fe0 2024-06-04T04:58:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.327+0000 7f3e2bfff700 1 -- 172.21.15.70:0/3356499108 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e380aaee0 con 0x558895ed7fe0 2024-06-04T04:58:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.327+0000 7f3e2bfff700 1 -- 172.21.15.70:0/3356499108 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e380b35b0 con 0x558895ed7fe0 2024-06-04T04:58:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.328+0000 7f3e5372fb80 1 -- 172.21.15.70:0/3356499108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895ed7fe0 msgr2=0x558895ed9dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.328+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/3356499108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895ed7fe0 0x558895ed9dd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3e380a6860 tx=0x7f3e380a66a0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.328+0000 7f3e5372fb80 1 -- 172.21.15.70:0/3356499108 shutdown_connections 2024-06-04T04:58:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.328+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/3356499108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895edcec0 0x558895eda310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.328+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/3356499108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed4290 0x558895ecdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.328+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/3356499108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895ed7fe0 0x558895ed9dd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.328+0000 7f3e5372fb80 1 -- 172.21.15.70:0/3356499108 >> 172.21.15.70:0/3356499108 conn(0x558895e01100 msgr2=0x558895ed2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.328+0000 7f3e5372fb80 1 -- 172.21.15.70:0/3356499108 shutdown_connections 2024-06-04T04:58:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.328+0000 7f3e5372fb80 1 -- 172.21.15.70:0/3356499108 wait complete. 2024-06-04T04:58:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.329+0000 7f3e5372fb80 1 -- start start 2024-06-04T04:58:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.329+0000 7f3e5372fb80 1 Processor -- start 2024-06-04T04:58:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.329+0000 7f3e5372fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895ed4290 0x558895ee5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.329+0000 7f3e5372fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed7fe0 0x558895ee5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e5372fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895edcec0 0x558895ee2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e5372fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e441af040 con 0x558895ed7fe0 2024-06-04T04:58:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e5372fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e441ae970 con 0x558895ed4290 2024-06-04T04:58:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e5372fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3e441abb70 con 0x558895edcec0 2024-06-04T04:58:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e41d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed7fe0 0x558895ee5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e4259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895ed4290 0x558895ee5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e41d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed7fe0 0x558895ee5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35786/0 (socket says 172.21.15.70:35786) 2024-06-04T04:58:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e41d9b700 1 -- 172.21.15.70:0/1615688040 learned_addr learned my addr 172.21.15.70:0/1615688040 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e5372fb80 10 client.0 _ll_register_callbacks cb 0x558895dbd9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e42d9d700 1 --2- 172.21.15.70:0/1615688040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895edcec0 0x558895ee2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e4259c700 1 -- 172.21.15.70:0/1615688040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895edcec0 msgr2=0x558895ee2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e4259c700 1 --2- 172.21.15.70:0/1615688040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895edcec0 0x558895ee2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e4259c700 1 -- 172.21.15.70:0/1615688040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed7fe0 msgr2=0x558895ee5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e4259c700 1 --2- 172.21.15.70:0/1615688040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed7fe0 0x558895ee5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e4259c700 1 -- 172.21.15.70:0/1615688040 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558895ee2c90 con 0x558895ed4290 2024-06-04T04:58:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e42d9d700 1 --2- 172.21.15.70:0/1615688040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895edcec0 0x558895ee2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e41d9b700 1 --2- 172.21.15.70:0/1615688040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed7fe0 0x558895ee5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:26.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e4259c700 1 --2- 172.21.15.70:0/1615688040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895ed4290 0x558895ee5640 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3e340043a0 tx=0x7f3e34009440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d355f7f214be6e9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:26.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.330+0000 7f3e29ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T04:58:26.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.331+0000 7f3e29ffb700 1 -- 172.21.15.70:0/1615688040 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e34005450 con 0x558895ed4290 2024-06-04T04:58:26.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.331+0000 7f3e5372fb80 1 -- 172.21.15.70:0/1615688040 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e380aaee0 con 0x558895ed4290 2024-06-04T04:58:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.331+0000 7f3e29ffb700 1 -- 172.21.15.70:0/1615688040 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f3e340055f0 con 0x558895ed4290 2024-06-04T04:58:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.331+0000 7f3e29ffb700 1 -- 172.21.15.70:0/1615688040 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3e34042050 con 0x558895ed4290 2024-06-04T04:58:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.331+0000 7f3e5372fb80 10 client.25232 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.331+0000 7f3e5372fb80 1 -- 172.21.15.70:0/1615688040 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558895f58ce0 con 0x558895ed4290 2024-06-04T04:58:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.331+0000 7f3e29ffb700 1 -- 172.21.15.70:0/1615688040 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e380aaee0 con 0x558895ed4290 2024-06-04T04:58:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.331+0000 7f3e29ffb700 1 -- 172.21.15.70:0/1615688040 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f3e34026630 con 0x558895ed4290 2024-06-04T04:58:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.331+0000 7f3e5372fb80 10 client.25232 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.331+0000 7f3e5372fb80 10 client.25232 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.331+0000 7f3e5372fb80 20 client.25232 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.331+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/1615688040 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558895f59060 0x558895f5b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.331+0000 7f3e5372fb80 4 client.25232 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T04:58:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.331+0000 7f3e5372fb80 1 -- 172.21.15.70:0/1615688040 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558895e9fa60 con 0x558895f59060 2024-06-04T04:58:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.332+0000 7f3e41d9b700 1 --2- 172.21.15.70:0/1615688040 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558895f59060 0x558895f5b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.332+0000 7f3e29ffb700 10 client.25232 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.332+0000 7f3e41d9b700 1 --2- 172.21.15.70:0/1615688040 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558895f59060 0x558895f5b450 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c4a62369dc4a2b3 server_cookie=c282eab637f6529e in_seq=0 out_seq=0 2024-06-04T04:58:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.333+0000 7f3e29ffb700 1 -- 172.21.15.70:0/1615688040 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558895e9fa60 con 0x558895f59060 2024-06-04T04:58:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.333+0000 7f3e29ffb700 10 client.25232 handle_command_reply: tid=0 2024-06-04T04:58:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.333+0000 7f3e5372fb80 1 client.25232 shutdown 2024-06-04T04:58:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.333+0000 7f3e5372fb80 1 -- 172.21.15.70:0/1615688040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895ed4290 msgr2=0x558895ee5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.333+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/1615688040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895ed4290 0x558895ee5640 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3e340043a0 tx=0x7f3e34009440 comp rx=0 tx=0).stop 2024-06-04T04:58:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.333+0000 7f3e5372fb80 1 -- 172.21.15.70:0/1615688040 shutdown_connections 2024-06-04T04:58:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.333+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/1615688040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558895edcec0 0x558895ee2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.333+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/1615688040 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558895f59060 0x558895f5b450 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.333+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/1615688040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558895ed4290 0x558895ee5640 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.333+0000 7f3e5372fb80 1 --2- 172.21.15.70:0/1615688040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558895ed7fe0 0x558895ee5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.333+0000 7f3e5372fb80 1 -- 172.21.15.70:0/1615688040 >> 172.21.15.70:0/1615688040 conn(0x558895e01100 msgr2=0x558895ed6930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.334+0000 7f3e29ffb700 0 client.25232 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:26.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.334+0000 7f3e5372fb80 1 -- 172.21.15.70:0/1615688040 shutdown_connections 2024-06-04T04:58:26.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.334+0000 7f3e5372fb80 1 -- 172.21.15.70:0/1615688040 wait complete. 2024-06-04T04:58:26.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.334+0000 7f3e5372fb80 20 client.25232 trim_cache size 0 max 16384 2024-06-04T04:58:26.339 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:58:26.339 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T04:58:26.339 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "965e5a3b-2721-402d-986e-a73a5ed413a3", 2024-06-04T04:58:26.339 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T04:58:26.339 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:58:26.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.334+0000 7f3e48e9a700 1 -- 172.21.15.70:0/1543362362 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e2c0b2b10 msgr2=0x7f3e2c0b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:26.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.334+0000 7f3e48e9a700 1 --2- 172.21.15.70:0/1543362362 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e2c0b2b10 0x7f3e2c0b4fd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3e441076f0 tx=0x7f3e34009260 comp rx=0 tx=0).stop 2024-06-04T04:58:26.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.334+0000 7f3e48e9a700 1 -- 172.21.15.70:0/1543362362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e44103600 msgr2=0x7f3e4419fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:26.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.334+0000 7f3e48e9a700 1 --2- 172.21.15.70:0/1543362362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e44103600 0x7f3e4419fd30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3e380098e0 tx=0x7f3e380047b0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.335+0000 7f3e48e9a700 1 -- 172.21.15.70:0/1543362362 shutdown_connections 2024-06-04T04:58:26.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.335+0000 7f3e48e9a700 1 --2- 172.21.15.70:0/1543362362 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e2c0b2b10 0x7f3e2c0b4fd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.335+0000 7f3e48e9a700 1 --2- 172.21.15.70:0/1543362362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e441084c0 0x7f3e44199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.335+0000 7f3e48e9a700 1 --2- 172.21.15.70:0/1543362362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e44102c40 0x7f3e4419f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.335+0000 7f3e48e9a700 1 --2- 172.21.15.70:0/1543362362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e44103600 0x7f3e4419fd30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:26.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.335+0000 7f3e48e9a700 1 -- 172.21.15.70:0/1543362362 >> 172.21.15.70:0/1543362362 conn(0x7f3e44078ab0 msgr2=0x7f3e44100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:26.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.335+0000 7f3e48e9a700 1 -- 172.21.15.70:0/1543362362 shutdown_connections 2024-06-04T04:58:26.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:26.335+0000 7f3e48e9a700 1 -- 172.21.15.70:0/1543362362 wait complete. 2024-06-04T04:58:27.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:27 smithi070 ceph-mon[32631]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-06-04T04:58:27.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:27 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T04:58:27.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:27 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T04:58:27.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:27 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T04:58:27.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:27 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T04:58:27.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:27 smithi112 ceph-mon[36453]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-06-04T04:58:27.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:27 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T04:58:27.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:27 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T04:58:27.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:27 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T04:58:27.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:27 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T04:58:27.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:27 smithi098 ceph-mon[35569]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-06-04T04:58:27.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:27 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T04:58:27.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:27 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T04:58:27.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:27 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T04:58:27.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:27 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T04:58:28.216 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:28 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T04:58:28.244 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T04:58:28.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:28 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T04:58:28.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:28 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T04:58:28.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.724+0000 7fd2abe41700 1 -- 172.21.15.70:0/2348889334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2a41084c0 msgr2=0x7fd2a41088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:28.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.724+0000 7fd2abe41700 1 --2- 172.21.15.70:0/2348889334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2a41084c0 0x7fd2a41088a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0009990 tx=0x7fd2a0009ca0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.724+0000 7fd2abe41700 1 -- 172.21.15.70:0/2348889334 shutdown_connections 2024-06-04T04:58:28.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.724+0000 7fd2abe41700 1 --2- 172.21.15.70:0/2348889334 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2a4102e00 0x7fd2a410f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.724+0000 7fd2abe41700 1 --2- 172.21.15.70:0/2348889334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2a41084c0 0x7fd2a41088a0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.724+0000 7fd2abe41700 1 --2- 172.21.15.70:0/2348889334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a4102440 0x7fd2a41028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.724+0000 7fd2abe41700 1 -- 172.21.15.70:0/2348889334 >> 172.21.15.70:0/2348889334 conn(0x7fd2a40793e0 msgr2=0x7fd2a40797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:28.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.724+0000 7fd2abe41700 1 -- 172.21.15.70:0/2348889334 shutdown_connections 2024-06-04T04:58:28.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.724+0000 7fd2abe41700 1 -- 172.21.15.70:0/2348889334 wait complete. 2024-06-04T04:58:28.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.725+0000 7fd2abe41700 1 Processor -- start 2024-06-04T04:58:28.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.725+0000 7fd2abe41700 1 -- start start 2024-06-04T04:58:28.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.726+0000 7fd2abe41700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a4102440 0x7fd2a419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.726+0000 7fd2abe41700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2a4102e00 0x7fd2a4197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.726+0000 7fd2abe41700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2a4197eb0 0x7fd2a4198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.726+0000 7fd2abe41700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a41124a0 con 0x7fd2a4197eb0 2024-06-04T04:58:28.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.726+0000 7fd2abe41700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a4112320 con 0x7fd2a4102440 2024-06-04T04:58:28.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.726+0000 7fd2abe41700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a4112620 con 0x7fd2a4102e00 2024-06-04T04:58:28.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.726+0000 7fd2a9bdd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a4102440 0x7fd2a419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.726+0000 7fd2a9bdd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a4102440 0x7fd2a419f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54954/0 (socket says 172.21.15.70:54954) 2024-06-04T04:58:28.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.726+0000 7fd2a9bdd700 1 -- 172.21.15.70:0/1087349083 learned_addr learned my addr 172.21.15.70:0/1087349083 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:28.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.726+0000 7fd2aa3de700 1 --2- 172.21.15.70:0/1087349083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2a4197eb0 0x7fd2a4198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.726+0000 7fd2a93dc700 1 --2- 172.21.15.70:0/1087349083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2a4102e00 0x7fd2a4197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.727+0000 7fd2aa3de700 1 -- 172.21.15.70:0/1087349083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2a4102e00 msgr2=0x7fd2a4197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:28.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.727+0000 7fd2aa3de700 1 --2- 172.21.15.70:0/1087349083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2a4102e00 0x7fd2a4197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.727+0000 7fd2aa3de700 1 -- 172.21.15.70:0/1087349083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a4102440 msgr2=0x7fd2a419f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:28.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.727+0000 7fd2aa3de700 1 --2- 172.21.15.70:0/1087349083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a4102440 0x7fd2a419f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.727+0000 7fd2aa3de700 1 -- 172.21.15.70:0/1087349083 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a41b51b0 con 0x7fd2a4197eb0 2024-06-04T04:58:28.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.727+0000 7fd2aa3de700 1 --2- 172.21.15.70:0/1087349083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2a4197eb0 0x7fd2a4198360 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd29c007f10 tx=0x7fd29c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:28.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.727+0000 7fd29affd700 1 -- 172.21.15.70:0/1087349083 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29c004db0 con 0x7fd2a4197eb0 2024-06-04T04:58:28.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.727+0000 7fd29affd700 1 -- 172.21.15.70:0/1087349083 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd29c009e20 con 0x7fd2a4197eb0 2024-06-04T04:58:28.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.727+0000 7fd29affd700 1 -- 172.21.15.70:0/1087349083 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29c015400 con 0x7fd2a4197eb0 2024-06-04T04:58:28.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.728+0000 7fd2abe41700 1 -- 172.21.15.70:0/1087349083 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2a41b5440 con 0x7fd2a4197eb0 2024-06-04T04:58:28.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.728+0000 7fd2abe41700 1 -- 172.21.15.70:0/1087349083 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a41b5930 con 0x7fd2a4197eb0 2024-06-04T04:58:28.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.731+0000 7fd29affd700 1 -- 172.21.15.70:0/1087349083 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 149885+0+0 (secure 0 0 0) 0x7fd29c005240 con 0x7fd2a4197eb0 2024-06-04T04:58:28.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.732+0000 7fd29affd700 1 --2- 172.21.15.70:0/1087349083 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2900b2c40 0x7fd2900b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.732+0000 7fd2a9bdd700 1 --2- 172.21.15.70:0/1087349083 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2900b2c40 0x7fd2900b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.732+0000 7fd2b66d6b80 1 Processor -- start 2024-06-04T04:58:28.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.732+0000 7fd29affd700 1 -- 172.21.15.70:0/1087349083 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fd29c0e20e0 con 0x7fd2a4197eb0 2024-06-04T04:58:28.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.733+0000 7fd2b66d6b80 1 -- start start 2024-06-04T04:58:28.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.733+0000 7fd2a9bdd700 1 --2- 172.21.15.70:0/1087349083 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2900b2c40 0x7fd2900b5100 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0003760 tx=0x7fd2a0005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:28.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.733+0000 7fd2b66d6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b975340 0x55713b975720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.733+0000 7fd2b66d6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b96f970 0x55713b96fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.733+0000 7fd2b66d6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b970330 0x55713b97c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.733+0000 7fd2b66d6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55713b8cee20 con 0x55713b975340 2024-06-04T04:58:28.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.733+0000 7fd2b66d6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55713b8be9c0 con 0x55713b970330 2024-06-04T04:58:28.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.733+0000 7fd2b66d6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55713b8c2a00 con 0x55713b96f970 2024-06-04T04:58:28.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.733+0000 7fd2a9bdd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b975340 0x55713b975720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.733+0000 7fd2a9bdd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b975340 0x55713b975720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35824/0 (socket says 172.21.15.70:35824) 2024-06-04T04:58:28.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.733+0000 7fd2a9bdd700 1 -- 172.21.15.70:0/3835129128 learned_addr learned my addr 172.21.15.70:0/3835129128 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:28.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.733+0000 7fd2a93dc700 1 --2- 172.21.15.70:0/3835129128 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b96f970 0x55713b96fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.733+0000 7fd2aa3de700 1 --2- 172.21.15.70:0/3835129128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b970330 0x55713b97c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.734+0000 7fd2a9bdd700 1 -- 172.21.15.70:0/3835129128 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b96f970 msgr2=0x55713b96fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:28.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.734+0000 7fd2a9bdd700 1 --2- 172.21.15.70:0/3835129128 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b96f970 0x55713b96fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.734+0000 7fd2a9bdd700 1 -- 172.21.15.70:0/3835129128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b970330 msgr2=0x55713b97c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:28.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.734+0000 7fd2a9bdd700 1 --2- 172.21.15.70:0/3835129128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b970330 0x55713b97c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.734+0000 7fd2a9bdd700 1 -- 172.21.15.70:0/3835129128 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55713b80ea60 con 0x55713b975340 2024-06-04T04:58:28.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.734+0000 7fd2a9bdd700 1 --2- 172.21.15.70:0/3835129128 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b975340 0x55713b975720 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0015000 tx=0x7fd2a001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82680c32f053af67 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:28.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.734+0000 7fd28ffff700 1 -- 172.21.15.70:0/3835129128 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a00055e0 con 0x55713b975340 2024-06-04T04:58:28.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.734+0000 7fd28ffff700 1 -- 172.21.15.70:0/3835129128 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd2a000bc60 con 0x55713b975340 2024-06-04T04:58:28.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.734+0000 7fd28ffff700 1 -- 172.21.15.70:0/3835129128 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a000f800 con 0x55713b975340 2024-06-04T04:58:28.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.735+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/3835129128 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b975340 msgr2=0x55713b975720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:28.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.735+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/3835129128 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b975340 0x55713b975720 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0015000 tx=0x7fd2a001f000 comp rx=0 tx=0).stop 2024-06-04T04:58:28.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.735+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/3835129128 shutdown_connections 2024-06-04T04:58:28.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.735+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/3835129128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b970330 0x55713b97c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.735+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/3835129128 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b975340 0x55713b975720 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.735+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/3835129128 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b96f970 0x55713b96fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.735+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/3835129128 >> 172.21.15.70:0/3835129128 conn(0x55713b89d210 msgr2=0x55713b96a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:28.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.735+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/3835129128 shutdown_connections 2024-06-04T04:58:28.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.735+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/3835129128 wait complete. 2024-06-04T04:58:28.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.736+0000 7fd2b66d6b80 1 -- start start 2024-06-04T04:58:28.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.736+0000 7fd2b66d6b80 1 Processor -- start 2024-06-04T04:58:28.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2b66d6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b96f970 0x55713b97bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2b66d6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b970330 0x55713b9791f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2b66d6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b975340 0x55713b979730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2b66d6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55713b7c5130 con 0x55713b975340 2024-06-04T04:58:28.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2b66d6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55713b860950 con 0x55713b970330 2024-06-04T04:58:28.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2aa3de700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b975340 0x55713b979730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2b66d6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55713b97c6b0 con 0x55713b96f970 2024-06-04T04:58:28.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2aa3de700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b975340 0x55713b979730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35836/0 (socket says 172.21.15.70:35836) 2024-06-04T04:58:28.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2aa3de700 1 -- 172.21.15.70:0/188800316 learned_addr learned my addr 172.21.15.70:0/188800316 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:28.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2b66d6b80 10 client.0 _ll_register_callbacks cb 0x55713b859ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:28.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2a93dc700 1 --2- 172.21.15.70:0/188800316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b970330 0x55713b9791f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2a9bdd700 1 --2- 172.21.15.70:0/188800316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b96f970 0x55713b97bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2aa3de700 1 -- 172.21.15.70:0/188800316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b96f970 msgr2=0x55713b97bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:28.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2aa3de700 1 --2- 172.21.15.70:0/188800316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b96f970 0x55713b97bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2aa3de700 1 -- 172.21.15.70:0/188800316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b970330 msgr2=0x55713b9791f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:28.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2aa3de700 1 --2- 172.21.15.70:0/188800316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b970330 0x55713b9791f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2aa3de700 1 -- 172.21.15.70:0/188800316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55713b85d820 con 0x55713b975340 2024-06-04T04:58:28.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd2aa3de700 1 --2- 172.21.15.70:0/188800316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b975340 0x55713b979730 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd29c01b000 tx=0x7fd29c0a96f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74ded6a25040f299 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:28.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.737+0000 7fd28d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:28.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.738+0000 7fd28d7fa700 1 -- 172.21.15.70:0/188800316 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29c0c3060 con 0x55713b975340 2024-06-04T04:58:28.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.738+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/188800316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2a000bc60 con 0x55713b975340 2024-06-04T04:58:28.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.738+0000 7fd28d7fa700 1 -- 172.21.15.70:0/188800316 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fd29c0dfc40 con 0x55713b975340 2024-06-04T04:58:28.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.738+0000 7fd28d7fa700 1 -- 172.21.15.70:0/188800316 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd29c0a9210 con 0x55713b975340 2024-06-04T04:58:28.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.738+0000 7fd2b66d6b80 10 client.15777 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:28.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.738+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/188800316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55713b986f70 con 0x55713b975340 2024-06-04T04:58:28.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.738+0000 7fd28d7fa700 1 -- 172.21.15.70:0/188800316 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd2a000bc60 con 0x55713b975340 2024-06-04T04:58:28.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.738+0000 7fd28d7fa700 1 -- 172.21.15.70:0/188800316 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fd29c0ad200 con 0x55713b975340 2024-06-04T04:58:28.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.738+0000 7fd2b66d6b80 10 client.15777 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:28.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.738+0000 7fd2b66d6b80 10 client.15777 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T04:58:28.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.738+0000 7fd2b66d6b80 20 client.15777 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:28.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.738+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/188800316 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55713b989500 0x55713b98b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.738+0000 7fd2b66d6b80 4 client.15777 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:28.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.738+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/188800316 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55713b93bb70 con 0x55713b989500 2024-06-04T04:58:28.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.739+0000 7fd2a9bdd700 1 --2- 172.21.15.70:0/188800316 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55713b989500 0x55713b98b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.740+0000 7fd28d7fa700 10 client.15777 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:28.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.740+0000 7fd2a9bdd700 1 --2- 172.21.15.70:0/188800316 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55713b989500 0x55713b98b8f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d60baae5d7f9e81 server_cookie=1554ea651e7dc7ba in_seq=0 out_seq=0 2024-06-04T04:58:28.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.741+0000 7fd28d7fa700 1 -- 172.21.15.70:0/188800316 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55713b93bb70 con 0x55713b989500 2024-06-04T04:58:28.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.741+0000 7fd28d7fa700 10 client.15777 handle_command_reply: tid=0 2024-06-04T04:58:28.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.741+0000 7fd2b66d6b80 1 client.15777 shutdown 2024-06-04T04:58:28.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.741+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/188800316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b975340 msgr2=0x55713b979730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:28.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.741+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/188800316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b975340 0x55713b979730 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd29c01b000 tx=0x7fd29c0a96f0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.741+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/188800316 shutdown_connections 2024-06-04T04:58:28.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.741+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/188800316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b975340 0x55713b979730 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.741+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/188800316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b96f970 0x55713b97bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.741+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/188800316 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55713b989500 0x55713b98b8f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.741+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/188800316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b970330 0x55713b9791f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.741+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/188800316 >> 172.21.15.70:0/188800316 conn(0x55713b89d210 msgr2=0x55713b964d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:28.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.741+0000 7fd28d7fa700 0 client.15777 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:28.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.741+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/188800316 shutdown_connections 2024-06-04T04:58:28.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.741+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/188800316 wait complete. 2024-06-04T04:58:28.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.741+0000 7fd2b66d6b80 20 client.15777 trim_cache size 0 max 16384 2024-06-04T04:58:28.749 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:28.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.764+0000 7fd2b66d6b80 1 Processor -- start 2024-06-04T04:58:28.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.764+0000 7fd2b66d6b80 1 -- start start 2024-06-04T04:58:28.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.764+0000 7fd2b66d6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b96f970 0x55713b985750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.764+0000 7fd2b66d6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b970330 0x55713b973530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.764+0000 7fd2b66d6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b975340 0x55713b97ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.764+0000 7fd2b66d6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a41bd070 con 0x55713b970330 2024-06-04T04:58:28.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.764+0000 7fd2b66d6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a41bebb0 con 0x55713b975340 2024-06-04T04:58:28.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.764+0000 7fd2b66d6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a41bd740 con 0x55713b96f970 2024-06-04T04:58:28.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.764+0000 7fd2a93dc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b970330 0x55713b973530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.764+0000 7fd2a93dc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b970330 0x55713b973530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35850/0 (socket says 172.21.15.70:35850) 2024-06-04T04:58:28.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.764+0000 7fd2a93dc700 1 -- 172.21.15.70:0/98034491 learned_addr learned my addr 172.21.15.70:0/98034491 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:28.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.764+0000 7fd2a9bdd700 1 --2- 172.21.15.70:0/98034491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b96f970 0x55713b985750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.764+0000 7fd2aa3de700 1 --2- 172.21.15.70:0/98034491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b975340 0x55713b97ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.765+0000 7fd2a93dc700 1 -- 172.21.15.70:0/98034491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b96f970 msgr2=0x55713b985750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:28.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.765+0000 7fd2a93dc700 1 --2- 172.21.15.70:0/98034491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b96f970 0x55713b985750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.765+0000 7fd2a93dc700 1 -- 172.21.15.70:0/98034491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b975340 msgr2=0x55713b97ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:28.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.765+0000 7fd2a93dc700 1 --2- 172.21.15.70:0/98034491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b975340 0x55713b97ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.765+0000 7fd2a93dc700 1 -- 172.21.15.70:0/98034491 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55713b974d90 con 0x55713b970330 2024-06-04T04:58:28.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.765+0000 7fd2aa3de700 1 --2- 172.21.15.70:0/98034491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b975340 0x55713b97ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:28.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.765+0000 7fd2a93dc700 1 --2- 172.21.15.70:0/98034491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b970330 0x55713b973530 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd294004640 tx=0x7fd29400bba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e807e3719bae9373 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:28.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.765+0000 7fd28f7fe700 1 -- 172.21.15.70:0/98034491 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd294003a20 con 0x55713b970330 2024-06-04T04:58:28.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.765+0000 7fd28f7fe700 1 -- 172.21.15.70:0/98034491 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd294020070 con 0x55713b970330 2024-06-04T04:58:28.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.765+0000 7fd28f7fe700 1 -- 172.21.15.70:0/98034491 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29401b400 con 0x55713b970330 2024-06-04T04:58:28.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.766+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/98034491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b970330 msgr2=0x55713b973530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:28.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.766+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/98034491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b970330 0x55713b973530 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd294004640 tx=0x7fd29400bba0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.766+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/98034491 shutdown_connections 2024-06-04T04:58:28.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.766+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/98034491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b975340 0x55713b97ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.766+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/98034491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b96f970 0x55713b985750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.766+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/98034491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b970330 0x55713b973530 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.766+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/98034491 >> 172.21.15.70:0/98034491 conn(0x55713b89d210 msgr2=0x55713b972e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:28.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.766+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/98034491 shutdown_connections 2024-06-04T04:58:28.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.766+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/98034491 wait complete. 2024-06-04T04:58:28.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.767+0000 7fd2b66d6b80 1 -- start start 2024-06-04T04:58:28.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.767+0000 7fd2b66d6b80 1 Processor -- start 2024-06-04T04:58:28.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.767+0000 7fd2b66d6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b96f970 0x55713b97e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.767+0000 7fd2b66d6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b975340 0x55713b97ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.767+0000 7fd2b66d6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b97f390 0x55713b9f4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.767+0000 7fd2b66d6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a41be4e0 con 0x55713b96f970 2024-06-04T04:58:28.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.767+0000 7fd2b66d6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a41bde10 con 0x55713b97f390 2024-06-04T04:58:28.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.767+0000 7fd2b66d6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a41bb010 con 0x55713b975340 2024-06-04T04:58:28.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.767+0000 7fd2b66d6b80 10 client.0 _ll_register_callbacks cb 0x55713b859ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:28.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.767+0000 7fd2a9bdd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b96f970 0x55713b97e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.768+0000 7fd2a9bdd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b96f970 0x55713b97e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35866/0 (socket says 172.21.15.70:35866) 2024-06-04T04:58:28.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.768+0000 7fd2a9bdd700 1 -- 172.21.15.70:0/816590275 learned_addr learned my addr 172.21.15.70:0/816590275 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:28.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.768+0000 7fd2a93dc700 1 --2- 172.21.15.70:0/816590275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b975340 0x55713b97ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.768+0000 7fd2aa3de700 1 --2- 172.21.15.70:0/816590275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b97f390 0x55713b9f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.768+0000 7fd2a93dc700 1 -- 172.21.15.70:0/816590275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b97f390 msgr2=0x55713b9f4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:28.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.768+0000 7fd2a93dc700 1 --2- 172.21.15.70:0/816590275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b97f390 0x55713b9f4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.768+0000 7fd2a93dc700 1 -- 172.21.15.70:0/816590275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b96f970 msgr2=0x55713b97e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:28.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.768+0000 7fd2a93dc700 1 --2- 172.21.15.70:0/816590275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b96f970 0x55713b97e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.768+0000 7fd2a93dc700 1 -- 172.21.15.70:0/816590275 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55713b9f45d0 con 0x55713b975340 2024-06-04T04:58:28.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.769+0000 7fd2a93dc700 1 --2- 172.21.15.70:0/816590275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b975340 0x55713b97ee50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x55713b977dc0 tx=0x7fd294011e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=497531aeb44df9e3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:28.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.769+0000 7fd28d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:58:28.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.769+0000 7fd28d7fa700 1 -- 172.21.15.70:0/816590275 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2940116c0 con 0x55713b975340 2024-06-04T04:58:28.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.769+0000 7fd28d7fa700 1 -- 172.21.15.70:0/816590275 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fd294011860 con 0x55713b975340 2024-06-04T04:58:28.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.769+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/816590275 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd294020070 con 0x55713b975340 2024-06-04T04:58:28.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.769+0000 7fd28d7fa700 1 -- 172.21.15.70:0/816590275 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd294003820 con 0x55713b975340 2024-06-04T04:58:28.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.769+0000 7fd2b66d6b80 10 client.25306 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:28.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.769+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/816590275 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55713b9f5750 con 0x55713b975340 2024-06-04T04:58:28.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.769+0000 7fd28d7fa700 1 -- 172.21.15.70:0/816590275 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd294020070 con 0x55713b975340 2024-06-04T04:58:28.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.770+0000 7fd28d7fa700 1 -- 172.21.15.70:0/816590275 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fd29402ea10 con 0x55713b975340 2024-06-04T04:58:28.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.770+0000 7fd2b66d6b80 10 client.25306 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:28.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.770+0000 7fd2b66d6b80 10 client.25306 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T04:58:28.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.770+0000 7fd2b66d6b80 20 client.25306 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:28.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.770+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/816590275 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55713b9f5ec0 0x55713b9f82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:28.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.770+0000 7fd2b66d6b80 4 client.25306 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 154.9, "await_for": 45.0}] 2024-06-04T04:58:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.770+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/816590275 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 154.9, "await_for": 45.0}) -- 0x55713b8d0d70 con 0x55713b9f5ec0 2024-06-04T04:58:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.770+0000 7fd2a9bdd700 1 --2- 172.21.15.70:0/816590275 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55713b9f5ec0 0x55713b9f82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.771+0000 7fd2a9bdd700 1 --2- 172.21.15.70:0/816590275 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55713b9f5ec0 0x55713b9f82b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19bd7622e4386b2c server_cookie=9123ae06b66001cc in_seq=0 out_seq=0 2024-06-04T04:58:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.771+0000 7fd28d7fa700 10 client.25306 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:28.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.782+0000 7fd28d7fa700 1 -- 172.21.15.70:0/816590275 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55713b8d0d70 con 0x55713b9f5ec0 2024-06-04T04:58:28.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.782+0000 7fd28d7fa700 10 client.25306 handle_command_reply: tid=0 2024-06-04T04:58:28.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.782+0000 7fd2b66d6b80 1 client.25306 shutdown 2024-06-04T04:58:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.782+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/816590275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b975340 msgr2=0x55713b97ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.782+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/816590275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b975340 0x55713b97ee50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x55713b977dc0 tx=0x7fd294011e70 comp rx=0 tx=0).stop 2024-06-04T04:58:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.783+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/816590275 shutdown_connections 2024-06-04T04:58:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.783+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/816590275 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55713b9f5ec0 0x55713b9f82b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.783+0000 7fd28d7fa700 0 client.25306 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.783+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/816590275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55713b97f390 0x55713b9f4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.783+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/816590275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713b96f970 0x55713b97e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.783+0000 7fd2b66d6b80 1 --2- 172.21.15.70:0/816590275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55713b975340 0x55713b97ee50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.783+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/816590275 >> 172.21.15.70:0/816590275 conn(0x55713b89d210 msgr2=0x55713b9736b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:28.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.783+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/816590275 shutdown_connections 2024-06-04T04:58:28.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.783+0000 7fd2b66d6b80 1 -- 172.21.15.70:0/816590275 wait complete. 2024-06-04T04:58:28.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.783+0000 7fd2b66d6b80 20 client.25306 trim_cache size 0 max 16384 2024-06-04T04:58:28.786 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:58:28.786 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 29, 2024-06-04T04:58:28.786 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T04:58:28.787 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 2, 2024-06-04T04:58:28.787 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T04:58:28.787 INFO:teuthology.orchestra.run.smithi070.stdout: "f277a116": { 2024-06-04T04:58:28.787 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 2, 2024-06-04T04:58:28.787 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T04:58:28.787 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T04:58:28.787 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T04:58:28.787 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T04:58:28.787 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T04:58:28.787 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T04:58:28.787 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 154.9, 2024-06-04T04:58:28.787 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T04:58:28.787 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T04:58:28.788 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T04:58:28.788 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T04:58:28.788 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T04:58:28.788 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T04:58:28.788 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T04:58:28.788 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T04:58:28.788 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T04:58:28.788 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T04:58:28.788 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T04:58:28.788 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:58:28.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.784+0000 7fd2abe41700 1 -- 172.21.15.70:0/1087349083 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2900b2c40 msgr2=0x7fd2900b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:28.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.784+0000 7fd2abe41700 1 --2- 172.21.15.70:0/1087349083 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2900b2c40 0x7fd2900b5100 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0003760 tx=0x7fd2a0005960 comp rx=0 tx=0).stop 2024-06-04T04:58:28.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.784+0000 7fd2abe41700 1 -- 172.21.15.70:0/1087349083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2a4197eb0 msgr2=0x7fd2a4198360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:28.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.784+0000 7fd2abe41700 1 --2- 172.21.15.70:0/1087349083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2a4197eb0 0x7fd2a4198360 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd29c007f10 tx=0x7fd29c00d3b0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.784+0000 7fd2abe41700 1 -- 172.21.15.70:0/1087349083 shutdown_connections 2024-06-04T04:58:28.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.784+0000 7fd2abe41700 1 --2- 172.21.15.70:0/1087349083 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2900b2c40 0x7fd2900b5100 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.784+0000 7fd2abe41700 1 --2- 172.21.15.70:0/1087349083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2a4197eb0 0x7fd2a4198360 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.784+0000 7fd2abe41700 1 --2- 172.21.15.70:0/1087349083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a4102440 0x7fd2a419f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.784+0000 7fd2abe41700 1 --2- 172.21.15.70:0/1087349083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2a4102e00 0x7fd2a4197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:28.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.784+0000 7fd2abe41700 1 -- 172.21.15.70:0/1087349083 >> 172.21.15.70:0/1087349083 conn(0x7fd2a40793e0 msgr2=0x7fd2a4105fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:28.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.784+0000 7fd2abe41700 1 -- 172.21.15.70:0/1087349083 shutdown_connections 2024-06-04T04:58:28.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:28.784+0000 7fd2abe41700 1 -- 172.21.15.70:0/1087349083 wait complete. 2024-06-04T04:58:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.139+0000 7f6eae778640 1 -- 172.21.15.70:0/3952136127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ea8108a80 msgr2=0x7f6ea8108e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.139+0000 7f6eae778640 1 --2- 172.21.15.70:0/3952136127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ea8108a80 0x7f6ea8108e60 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009930 tx=0x7f6e9802a420 comp rx=0 tx=0).stop 2024-06-04T04:58:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.139+0000 7f6eae778640 1 -- 172.21.15.70:0/3952136127 shutdown_connections 2024-06-04T04:58:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.139+0000 7f6eae778640 1 --2- 172.21.15.70:0/3952136127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea8103430 0x7f6ea810f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.139+0000 7f6eae778640 1 --2- 172.21.15.70:0/3952136127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ea8102a70 0x7f6ea8102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.139+0000 7f6eae778640 1 --2- 172.21.15.70:0/3952136127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ea8108a80 0x7f6ea8108e60 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.139+0000 7f6eae778640 1 -- 172.21.15.70:0/3952136127 >> 172.21.15.70:0/3952136127 conn(0x7f6ea80fe500 msgr2=0x7f6ea8100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.139+0000 7f6eae778640 1 -- 172.21.15.70:0/3952136127 shutdown_connections 2024-06-04T04:58:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.139+0000 7f6eae778640 1 -- 172.21.15.70:0/3952136127 wait complete. 2024-06-04T04:58:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.140+0000 7f6eae778640 1 Processor -- start 2024-06-04T04:58:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.140+0000 7f6eae778640 1 -- start start 2024-06-04T04:58:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.140+0000 7f6eae778640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ea8102a70 0x7f6ea819b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.140+0000 7f6eae778640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ea8103430 0x7f6ea819b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.140+0000 7f6eae778640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea8108a80 0x7f6ea819fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.140+0000 7f6eae778640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea8112bf0 con 0x7f6ea8103430 2024-06-04T04:58:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.141+0000 7f6eae778640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea8112a70 con 0x7f6ea8108a80 2024-06-04T04:58:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.141+0000 7f6eae778640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea8112d70 con 0x7f6ea8102a70 2024-06-04T04:58:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.141+0000 7f6ea77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ea8103430 0x7f6ea819b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.141+0000 7f6ea77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ea8103430 0x7f6ea819b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35884/0 (socket says 172.21.15.70:35884) 2024-06-04T04:58:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.141+0000 7f6ea77fe640 1 -- 172.21.15.70:0/2242858855 learned_addr learned my addr 172.21.15.70:0/2242858855 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.141+0000 7f6ea7fff640 1 --2- 172.21.15.70:0/2242858855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ea8102a70 0x7f6ea819b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.141+0000 7f6eacd15640 1 --2- 172.21.15.70:0/2242858855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea8108a80 0x7f6ea819fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.141+0000 7f6ea77fe640 1 -- 172.21.15.70:0/2242858855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ea8102a70 msgr2=0x7f6ea819b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.141+0000 7f6ea77fe640 1 --2- 172.21.15.70:0/2242858855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ea8102a70 0x7f6ea819b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.141+0000 7f6ea77fe640 1 -- 172.21.15.70:0/2242858855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea8108a80 msgr2=0x7f6ea819fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.141+0000 7f6ea77fe640 1 --2- 172.21.15.70:0/2242858855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea8108a80 0x7f6ea819fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.141+0000 7f6ea77fe640 1 -- 172.21.15.70:0/2242858855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea81a02d0 con 0x7f6ea8103430 2024-06-04T04:58:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.141+0000 7f6ea77fe640 1 --2- 172.21.15.70:0/2242858855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ea8103430 0x7f6ea819b820 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00d770 tx=0x7f6e9c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.141+0000 7f6ea57fa640 1 -- 172.21.15.70:0/2242858855 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c017070 con 0x7f6ea8103430 2024-06-04T04:58:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.141+0000 7f6eae778640 1 -- 172.21.15.70:0/2242858855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea81a05c0 con 0x7f6ea8103430 2024-06-04T04:58:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.141+0000 7f6ea57fa640 1 -- 172.21.15.70:0/2242858855 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e9c0054b0 con 0x7f6ea8103430 2024-06-04T04:58:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.141+0000 7f6ea57fa640 1 -- 172.21.15.70:0/2242858855 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c004c60 con 0x7f6ea8103430 2024-06-04T04:58:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.142+0000 7f6eae778640 1 -- 172.21.15.70:0/2242858855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea81a7e50 con 0x7f6ea8103430 2024-06-04T04:58:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.144+0000 7f6eb156a740 1 Processor -- start 2024-06-04T04:58:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.145+0000 7f6eb156a740 1 -- start start 2024-06-04T04:58:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.145+0000 7f6eb156a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x5628355efc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.145+0000 7f6eb156a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355f0140 0x5628355fa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.145+0000 7f6eb156a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355faa00 0x5628355fcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.145+0000 7f6eb156a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562835561c30 con 0x5628355ef820 2024-06-04T04:58:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.145+0000 7f6eb156a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5628355517d0 con 0x5628355faa00 2024-06-04T04:58:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.145+0000 7f6eb156a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562835555810 con 0x5628355f0140 2024-06-04T04:58:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.145+0000 7f6eacd15640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x5628355efc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.145+0000 7f6ea57fa640 1 -- 172.21.15.70:0/2242858855 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 149885+0+0 (secure 0 0 0) 0x7f6e9c004e00 con 0x7f6ea8103430 2024-06-04T04:58:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.145+0000 7f6eacd15640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x5628355efc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35886/0 (socket says 172.21.15.70:35886) 2024-06-04T04:58:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.145+0000 7f6ea7fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355f0140 0x5628355fa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.145+0000 7f6eacd15640 1 -- 172.21.15.70:0/1176886237 learned_addr learned my addr 172.21.15.70:0/1176886237 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.145+0000 7f6eacd15640 1 -- 172.21.15.70:0/1176886237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355f0140 msgr2=0x5628355fa4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.145+0000 7f6eacd15640 1 --2- 172.21.15.70:0/1176886237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355f0140 0x5628355fa4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.146+0000 7f6eacd15640 1 -- 172.21.15.70:0/1176886237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355faa00 msgr2=0x5628355fcdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.146+0000 7f6eacd15640 1 --2- 172.21.15.70:0/1176886237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355faa00 0x5628355fcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.146+0000 7f6eacd15640 1 -- 172.21.15.70:0/1176886237 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562835266210 con 0x5628355ef820 2024-06-04T04:58:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.146+0000 7f6eacd15640 1 --2- 172.21.15.70:0/1176886237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x5628355efc00 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0007dd0 tx=0x7f6ea000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e782eb40ea3e0366 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.146+0000 7f6e8e7fc640 1 -- 172.21.15.70:0/1176886237 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0014070 con 0x5628355ef820 2024-06-04T04:58:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.146+0000 7f6e8e7fc640 1 -- 172.21.15.70:0/1176886237 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ea0004d20 con 0x5628355ef820 2024-06-04T04:58:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.146+0000 7f6e8e7fc640 1 -- 172.21.15.70:0/1176886237 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea00107f0 con 0x5628355ef820 2024-06-04T04:58:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.147+0000 7f6ea57fa640 1 --2- 172.21.15.70:0/2242858855 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6e900b2b30 0x7f6e900b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.147+0000 7f6eb156a740 1 -- 172.21.15.70:0/1176886237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 msgr2=0x5628355efc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.147+0000 7f6eb156a740 1 --2- 172.21.15.70:0/1176886237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x5628355efc00 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0007dd0 tx=0x7f6ea000e720 comp rx=0 tx=0).stop 2024-06-04T04:58:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.147+0000 7f6ea7fff640 1 --2- 172.21.15.70:0/2242858855 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6e900b2b30 0x7f6e900b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.147+0000 7f6eb156a740 1 -- 172.21.15.70:0/1176886237 shutdown_connections 2024-06-04T04:58:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.147+0000 7f6eb156a740 1 --2- 172.21.15.70:0/1176886237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355faa00 0x5628355fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.147+0000 7f6eb156a740 1 --2- 172.21.15.70:0/1176886237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355f0140 0x5628355fa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.147+0000 7f6eb156a740 1 --2- 172.21.15.70:0/1176886237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x5628355efc00 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.147+0000 7f6eb156a740 1 -- 172.21.15.70:0/1176886237 >> 172.21.15.70:0/1176886237 conn(0x5628355ebd30 msgr2=0x5628355ec110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.147+0000 7f6ea57fa640 1 -- 172.21.15.70:0/2242858855 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f6e9c0e2320 con 0x7f6ea8103430 2024-06-04T04:58:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.147+0000 7f6eb156a740 1 -- 172.21.15.70:0/1176886237 shutdown_connections 2024-06-04T04:58:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.148+0000 7f6eb156a740 1 -- 172.21.15.70:0/1176886237 wait complete. 2024-06-04T04:58:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.148+0000 7f6ea7fff640 1 --2- 172.21.15.70:0/2242858855 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6e900b2b30 0x7f6e900b4ff0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6e98004770 tx=0x7f6e980058d0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.148+0000 7f6eb156a740 1 -- start start 2024-06-04T04:58:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.148+0000 7f6eb156a740 1 Processor -- start 2024-06-04T04:58:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6eb156a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x5628355f7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6eb156a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355f0140 0x5628355f8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6eb156a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355faa00 0x5628355f8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6eb156a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562835237790 con 0x5628355ef820 2024-06-04T04:58:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6eb156a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56283550d890 con 0x5628355faa00 2024-06-04T04:58:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6eb156a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5628355fd630 con 0x5628355f0140 2024-06-04T04:58:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6ea7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x5628355f7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6ea7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x5628355f7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35894/0 (socket says 172.21.15.70:35894) 2024-06-04T04:58:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6ea7fff640 1 -- 172.21.15.70:0/2208620950 learned_addr learned my addr 172.21.15.70:0/2208620950 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6eb156a740 10 client.0 _ll_register_callbacks cb 0x562835265a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6eacd15640 1 --2- 172.21.15.70:0/2208620950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355faa00 0x5628355f8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6ea77fe640 1 --2- 172.21.15.70:0/2208620950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355f0140 0x5628355f8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6ea7fff640 1 -- 172.21.15.70:0/2208620950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355f0140 msgr2=0x5628355f8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6ea7fff640 1 --2- 172.21.15.70:0/2208620950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355f0140 0x5628355f8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6ea7fff640 1 -- 172.21.15.70:0/2208620950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355faa00 msgr2=0x5628355f8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6ea7fff640 1 --2- 172.21.15.70:0/2208620950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355faa00 0x5628355f8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6ea7fff640 1 -- 172.21.15.70:0/2208620950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562835332620 con 0x5628355ef820 2024-06-04T04:58:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6ea7fff640 1 --2- 172.21.15.70:0/2208620950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x5628355f7ae0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6e980057c0 tx=0x7f6e98009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f25360e0b6d7a7b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.149+0000 7f6e877fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.150+0000 7f6e877fe640 1 -- 172.21.15.70:0/2208620950 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e98035040 con 0x5628355ef820 2024-06-04T04:58:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.150+0000 7f6e877fe640 1 -- 172.21.15.70:0/2208620950 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f6e980037f0 con 0x5628355ef820 2024-06-04T04:58:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.150+0000 7f6eb156a740 1 -- 172.21.15.70:0/2208620950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ea0004d20 con 0x5628355ef820 2024-06-04T04:58:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.150+0000 7f6e877fe640 1 -- 172.21.15.70:0/2208620950 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6e980039f0 con 0x5628355ef820 2024-06-04T04:58:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.150+0000 7f6eb156a740 10 client.15795 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.150+0000 7f6eb156a740 1 -- 172.21.15.70:0/2208620950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5628356094e0 con 0x5628355ef820 2024-06-04T04:58:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.150+0000 7f6e877fe640 1 -- 172.21.15.70:0/2208620950 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ea0004d20 con 0x5628355ef820 2024-06-04T04:58:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.151+0000 7f6e877fe640 1 -- 172.21.15.70:0/2208620950 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f6e98044070 con 0x5628355ef820 2024-06-04T04:58:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.151+0000 7f6eb156a740 10 client.15795 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.151+0000 7f6eb156a740 10 client.15795 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.151+0000 7f6eb156a740 20 client.15795 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.151+0000 7f6eb156a740 1 --2- 172.21.15.70:0/2208620950 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56283560a9a0 0x56283560cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.151+0000 7f6eb156a740 4 client.15795 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.151+0000 7f6eb156a740 1 -- 172.21.15.70:0/2208620950 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562835563b80 con 0x56283560a9a0 2024-06-04T04:58:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.151+0000 7f6ea77fe640 1 --2- 172.21.15.70:0/2208620950 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56283560a9a0 0x56283560cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.151+0000 7f6e877fe640 10 client.15795 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.151+0000 7f6ea77fe640 1 --2- 172.21.15.70:0/2208620950 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56283560a9a0 0x56283560cd90 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4079c796f456fdbc server_cookie=d6fe45fc4353dd82 in_seq=0 out_seq=0 2024-06-04T04:58:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.152+0000 7f6e877fe640 1 -- 172.21.15.70:0/2208620950 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562835563b80 con 0x56283560a9a0 2024-06-04T04:58:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.152+0000 7f6e877fe640 10 client.15795 handle_command_reply: tid=0 2024-06-04T04:58:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.152+0000 7f6eb156a740 1 client.15795 shutdown 2024-06-04T04:58:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.152+0000 7f6eb156a740 1 -- 172.21.15.70:0/2208620950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 msgr2=0x5628355f7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.152+0000 7f6eb156a740 1 --2- 172.21.15.70:0/2208620950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x5628355f7ae0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6e980057c0 tx=0x7f6e98009260 comp rx=0 tx=0).stop 2024-06-04T04:58:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.152+0000 7f6eb156a740 1 -- 172.21.15.70:0/2208620950 shutdown_connections 2024-06-04T04:58:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.152+0000 7f6eb156a740 1 --2- 172.21.15.70:0/2208620950 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56283560a9a0 0x56283560cd90 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.152+0000 7f6eb156a740 1 --2- 172.21.15.70:0/2208620950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355faa00 0x5628355f8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.152+0000 7f6e877fe640 0 client.15795 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.152+0000 7f6eb156a740 1 --2- 172.21.15.70:0/2208620950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355f0140 0x5628355f8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.152+0000 7f6eb156a740 1 --2- 172.21.15.70:0/2208620950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x5628355f7ae0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.152+0000 7f6eb156a740 1 -- 172.21.15.70:0/2208620950 >> 172.21.15.70:0/2208620950 conn(0x5628355ebd30 msgr2=0x5628355eafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.153+0000 7f6eb156a740 1 -- 172.21.15.70:0/2208620950 shutdown_connections 2024-06-04T04:58:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.153+0000 7f6eb156a740 1 -- 172.21.15.70:0/2208620950 wait complete. 2024-06-04T04:58:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.153+0000 7f6eb156a740 20 client.15795 trim_cache size 0 max 16384 2024-06-04T04:58:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.173+0000 7f6eb156a740 1 Processor -- start 2024-06-04T04:58:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.173+0000 7f6eb156a740 1 -- start start 2024-06-04T04:58:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6eb156a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355ef820 0x5628355f8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6eb156a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355f0140 0x562835605d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6eb156a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355faa00 0x5628356062d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6eb156a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea81b07f0 con 0x5628355faa00 2024-06-04T04:58:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6eb156a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea81b2330 con 0x5628355f0140 2024-06-04T04:58:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6eb156a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea81b0ec0 con 0x5628355ef820 2024-06-04T04:58:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6eacd15640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355faa00 0x5628356062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6eacd15640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355faa00 0x5628356062d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35896/0 (socket says 172.21.15.70:35896) 2024-06-04T04:58:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6eacd15640 1 -- 172.21.15.70:0/423850017 learned_addr learned my addr 172.21.15.70:0/423850017 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6ea77fe640 1 --2- 172.21.15.70:0/423850017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355f0140 0x562835605d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6ea7fff640 1 --2- 172.21.15.70:0/423850017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355ef820 0x5628355f8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6eacd15640 1 -- 172.21.15.70:0/423850017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355ef820 msgr2=0x5628355f8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6eacd15640 1 --2- 172.21.15.70:0/423850017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355ef820 0x5628355f8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6eacd15640 1 -- 172.21.15.70:0/423850017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355f0140 msgr2=0x562835605d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6eacd15640 1 --2- 172.21.15.70:0/423850017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355f0140 0x562835605d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6eacd15640 1 -- 172.21.15.70:0/423850017 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628355e76f0 con 0x5628355faa00 2024-06-04T04:58:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6ea77fe640 1 --2- 172.21.15.70:0/423850017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355f0140 0x562835605d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.174+0000 7f6eacd15640 1 --2- 172.21.15.70:0/423850017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355faa00 0x5628356062d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6ea00075e0 tx=0x7f6ea0007450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a4d30b87a52cbb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.175+0000 7f6e8dffb640 1 -- 172.21.15.70:0/423850017 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0014070 con 0x5628355faa00 2024-06-04T04:58:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.175+0000 7f6e8dffb640 1 -- 172.21.15.70:0/423850017 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ea0004d00 con 0x5628355faa00 2024-06-04T04:58:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.175+0000 7f6e8dffb640 1 -- 172.21.15.70:0/423850017 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0010700 con 0x5628355faa00 2024-06-04T04:58:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.175+0000 7f6eb156a740 1 -- 172.21.15.70:0/423850017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355faa00 msgr2=0x5628356062d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.175+0000 7f6eb156a740 1 --2- 172.21.15.70:0/423850017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355faa00 0x5628356062d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6ea00075e0 tx=0x7f6ea0007450 comp rx=0 tx=0).stop 2024-06-04T04:58:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.175+0000 7f6eb156a740 1 -- 172.21.15.70:0/423850017 shutdown_connections 2024-06-04T04:58:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.175+0000 7f6eb156a740 1 --2- 172.21.15.70:0/423850017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355f0140 0x562835605d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.175+0000 7f6eb156a740 1 --2- 172.21.15.70:0/423850017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355faa00 0x5628356062d0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.175+0000 7f6eb156a740 1 --2- 172.21.15.70:0/423850017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355ef820 0x5628355f8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.175+0000 7f6eb156a740 1 -- 172.21.15.70:0/423850017 >> 172.21.15.70:0/423850017 conn(0x5628355ebd30 msgr2=0x562835605680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.175+0000 7f6eb156a740 1 -- 172.21.15.70:0/423850017 shutdown_connections 2024-06-04T04:58:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.175+0000 7f6eb156a740 1 -- 172.21.15.70:0/423850017 wait complete. 2024-06-04T04:58:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.176+0000 7f6eb156a740 1 -- start start 2024-06-04T04:58:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.176+0000 7f6eb156a740 1 Processor -- start 2024-06-04T04:58:29.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.176+0000 7f6eb156a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x562835602c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.176+0000 7f6eb156a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355f0140 0x5628355fe020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.176+0000 7f6eb156a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355faa00 0x5628355fe560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.176+0000 7f6eb156a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea81b1c60 con 0x5628355ef820 2024-06-04T04:58:29.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.176+0000 7f6eb156a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea81b1590 con 0x5628355faa00 2024-06-04T04:58:29.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.176+0000 7f6eb156a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea81ae7d0 con 0x5628355f0140 2024-06-04T04:58:29.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.176+0000 7f6eb156a740 10 client.0 _ll_register_callbacks cb 0x7f6e98031b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:29.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.176+0000 7f6ea7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x562835602c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:29.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.176+0000 7f6ea7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x562835602c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35902/0 (socket says 172.21.15.70:35902) 2024-06-04T04:58:29.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.176+0000 7f6ea7fff640 1 -- 172.21.15.70:0/1792689548 learned_addr learned my addr 172.21.15.70:0/1792689548 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.176+0000 7f6ea77fe640 1 --2- 172.21.15.70:0/1792689548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355f0140 0x5628355fe020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.177+0000 7f6eacd15640 1 --2- 172.21.15.70:0/1792689548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355faa00 0x5628355fe560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.177+0000 7f6ea7fff640 1 -- 172.21.15.70:0/1792689548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355f0140 msgr2=0x5628355fe020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.177+0000 7f6ea7fff640 1 --2- 172.21.15.70:0/1792689548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355f0140 0x5628355fe020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.177+0000 7f6ea7fff640 1 -- 172.21.15.70:0/1792689548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355faa00 msgr2=0x5628355fe560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.177+0000 7f6ea7fff640 1 --2- 172.21.15.70:0/1792689548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355faa00 0x5628355fe560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.177+0000 7f6ea7fff640 1 -- 172.21.15.70:0/1792689548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628355feaa0 con 0x5628355ef820 2024-06-04T04:58:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.177+0000 7f6eacd15640 1 --2- 172.21.15.70:0/1792689548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355faa00 0x5628355fe560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.177+0000 7f6ea7fff640 1 --2- 172.21.15.70:0/1792689548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x562835602c60 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6e980057c0 tx=0x7f6e98005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dbd071f734dfa4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.177+0000 7f6e877fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.177+0000 7f6e877fe640 1 -- 172.21.15.70:0/1792689548 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e98035070 con 0x5628355ef820 2024-06-04T04:58:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.177+0000 7f6e877fe640 1 -- 172.21.15.70:0/1792689548 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f6e98003d90 con 0x5628355ef820 2024-06-04T04:58:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.177+0000 7f6eb156a740 1 -- 172.21.15.70:0/1792689548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ea0004d00 con 0x5628355ef820 2024-06-04T04:58:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.177+0000 7f6e877fe640 1 -- 172.21.15.70:0/1792689548 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6e98003800 con 0x5628355ef820 2024-06-04T04:58:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.177+0000 7f6eb156a740 10 client.15801 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.177+0000 7f6eb156a740 1 -- 172.21.15.70:0/1792689548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5628356751e0 con 0x5628355ef820 2024-06-04T04:58:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.177+0000 7f6e877fe640 1 -- 172.21.15.70:0/1792689548 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ea0004d00 con 0x5628355ef820 2024-06-04T04:58:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.178+0000 7f6e877fe640 1 -- 172.21.15.70:0/1792689548 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f6e9805f020 con 0x5628355ef820 2024-06-04T04:58:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.178+0000 7f6eb156a740 10 client.15801 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.178+0000 7f6eb156a740 10 client.15801 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.178+0000 7f6eb156a740 20 client.15801 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.178+0000 7f6eb156a740 1 --2- 172.21.15.70:0/1792689548 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56283560a9a0 0x562835677990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.178+0000 7f6eb156a740 4 client.15801 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T04:58:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.178+0000 7f6eb156a740 1 -- 172.21.15.70:0/1792689548 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56283555d360 con 0x56283560a9a0 2024-06-04T04:58:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.178+0000 7f6ea77fe640 1 --2- 172.21.15.70:0/1792689548 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56283560a9a0 0x562835677990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.178+0000 7f6ea77fe640 1 --2- 172.21.15.70:0/1792689548 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56283560a9a0 0x562835677990 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7260ebf8a8c8ad7 server_cookie=effeb5fe36c55dbb in_seq=0 out_seq=0 2024-06-04T04:58:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.178+0000 7f6e877fe640 10 client.15801 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.182+0000 7f6e877fe640 1 -- 172.21.15.70:0/1792689548 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56283555d360 con 0x56283560a9a0 2024-06-04T04:58:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.182+0000 7f6e877fe640 10 client.15801 handle_command_reply: tid=0 2024-06-04T04:58:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.182+0000 7f6eb156a740 1 client.15801 shutdown 2024-06-04T04:58:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.182+0000 7f6eb156a740 1 -- 172.21.15.70:0/1792689548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 msgr2=0x562835602c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.182+0000 7f6eb156a740 1 --2- 172.21.15.70:0/1792689548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x562835602c60 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6e980057c0 tx=0x7f6e98005c90 comp rx=0 tx=0).stop 2024-06-04T04:58:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.182+0000 7f6eb156a740 1 -- 172.21.15.70:0/1792689548 shutdown_connections 2024-06-04T04:58:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.182+0000 7f6eb156a740 1 --2- 172.21.15.70:0/1792689548 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56283560a9a0 0x562835677990 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.182+0000 7f6eb156a740 1 --2- 172.21.15.70:0/1792689548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5628355faa00 0x5628355fe560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.182+0000 7f6e877fe640 0 client.15801 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.182+0000 7f6eb156a740 1 --2- 172.21.15.70:0/1792689548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5628355f0140 0x5628355fe020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.182+0000 7f6eb156a740 1 --2- 172.21.15.70:0/1792689548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5628355ef820 0x562835602c60 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.182+0000 7f6eb156a740 1 -- 172.21.15.70:0/1792689548 >> 172.21.15.70:0/1792689548 conn(0x5628355ebd30 msgr2=0x562835605680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.182+0000 7f6eb156a740 1 -- 172.21.15.70:0/1792689548 shutdown_connections 2024-06-04T04:58:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.182+0000 7f6eb156a740 1 -- 172.21.15.70:0/1792689548 wait complete. 2024-06-04T04:58:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.182+0000 7f6eb156a740 20 client.15801 trim_cache size 0 max 16384 2024-06-04T04:58:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T04:58:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T04:58:29.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T04:58:29.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.183+0000 7f6eae778640 1 -- 172.21.15.70:0/2242858855 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6e900b2b30 msgr2=0x7f6e900b4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:29.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.183+0000 7f6eae778640 1 --2- 172.21.15.70:0/2242858855 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6e900b2b30 0x7f6e900b4ff0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6e98004770 tx=0x7f6e980058d0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.183+0000 7f6eae778640 1 -- 172.21.15.70:0/2242858855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ea8103430 msgr2=0x7f6ea819b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.183+0000 7f6eae778640 1 --2- 172.21.15.70:0/2242858855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ea8103430 0x7f6ea819b820 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00d770 tx=0x7f6e9c00dc30 comp rx=0 tx=0).stop 2024-06-04T04:58:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.183+0000 7f6eae778640 1 -- 172.21.15.70:0/2242858855 shutdown_connections 2024-06-04T04:58:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.183+0000 7f6eae778640 1 --2- 172.21.15.70:0/2242858855 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6e900b2b30 0x7f6e900b4ff0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.183+0000 7f6eae778640 1 --2- 172.21.15.70:0/2242858855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea8108a80 0x7f6ea819fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.183+0000 7f6eae778640 1 --2- 172.21.15.70:0/2242858855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ea8103430 0x7f6ea819b820 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.183+0000 7f6eae778640 1 --2- 172.21.15.70:0/2242858855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ea8102a70 0x7f6ea819b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.183+0000 7f6eae778640 1 -- 172.21.15.70:0/2242858855 >> 172.21.15.70:0/2242858855 conn(0x7f6ea80fe500 msgr2=0x7f6ea80feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.183+0000 7f6eae778640 1 -- 172.21.15.70:0/2242858855 shutdown_connections 2024-06-04T04:58:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:29.183+0000 7f6eae778640 1 -- 172.21.15.70:0/2242858855 wait complete. 2024-06-04T04:58:29.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:29 smithi070 ceph-mon[32631]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:58:29.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:29 smithi112 ceph-mon[36453]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:58:29.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:29 smithi098 ceph-mon[35569]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:58:31.004 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'f277a116', quiesce took 10.9 seconds. Will release after: 24.0 2024-06-04T04:58:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.249+0000 7fc184472700 1 -- 172.21.15.70:0/553959617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc17c102440 msgr2=0x7fc17c1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.249+0000 7fc184472700 1 --2- 172.21.15.70:0/553959617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc17c102440 0x7fc17c1028c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc16c009990 tx=0x7fc16c009ca0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.249+0000 7fc184472700 1 -- 172.21.15.70:0/553959617 shutdown_connections 2024-06-04T04:58:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.249+0000 7fc184472700 1 --2- 172.21.15.70:0/553959617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc17c102e00 0x7fc17c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.249+0000 7fc184472700 1 --2- 172.21.15.70:0/553959617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc17c108520 0x7fc17c108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.249+0000 7fc184472700 1 --2- 172.21.15.70:0/553959617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc17c102440 0x7fc17c1028c0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.249+0000 7fc184472700 1 -- 172.21.15.70:0/553959617 >> 172.21.15.70:0/553959617 conn(0x7fc17c0793f0 msgr2=0x7fc17c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:31.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.250+0000 7fc184472700 1 -- 172.21.15.70:0/553959617 shutdown_connections 2024-06-04T04:58:31.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.250+0000 7fc184472700 1 -- 172.21.15.70:0/553959617 wait complete. 2024-06-04T04:58:31.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.250+0000 7fc184472700 1 Processor -- start 2024-06-04T04:58:31.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.250+0000 7fc184472700 1 -- start start 2024-06-04T04:58:31.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.250+0000 7fc184472700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc17c102440 0x7fc17c19f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.250+0000 7fc184472700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc17c102e00 0x7fc17c19fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.250+0000 7fc184472700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc17c108520 0x7fc17c199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.250+0000 7fc184472700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c1124a0 con 0x7fc17c102e00 2024-06-04T04:58:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.250+0000 7fc184472700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c112320 con 0x7fc17c102440 2024-06-04T04:58:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.250+0000 7fc184472700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c112620 con 0x7fc17c108520 2024-06-04T04:58:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.251+0000 7fc181a0d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc17c102e00 0x7fc17c19fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.251+0000 7fc181a0d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc17c102e00 0x7fc17c19fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35912/0 (socket says 172.21.15.70:35912) 2024-06-04T04:58:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.251+0000 7fc181a0d700 1 -- 172.21.15.70:0/2904880247 learned_addr learned my addr 172.21.15.70:0/2904880247 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.251+0000 7fc18220e700 1 --2- 172.21.15.70:0/2904880247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc17c102440 0x7fc17c19f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.251+0000 7fc182a0f700 1 --2- 172.21.15.70:0/2904880247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc17c108520 0x7fc17c199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.251+0000 7fc181a0d700 1 -- 172.21.15.70:0/2904880247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc17c108520 msgr2=0x7fc17c199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.251+0000 7fc181a0d700 1 --2- 172.21.15.70:0/2904880247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc17c108520 0x7fc17c199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.251+0000 7fc181a0d700 1 -- 172.21.15.70:0/2904880247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc17c102440 msgr2=0x7fc17c19f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:31.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.251+0000 7fc181a0d700 1 --2- 172.21.15.70:0/2904880247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc17c102440 0x7fc17c19f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.251+0000 7fc181a0d700 1 -- 172.21.15.70:0/2904880247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17c19a0a0 con 0x7fc17c102e00 2024-06-04T04:58:31.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.251+0000 7fc182a0f700 1 --2- 172.21.15.70:0/2904880247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc17c108520 0x7fc17c199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:31.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.251+0000 7fc18220e700 1 --2- 172.21.15.70:0/2904880247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc17c102440 0x7fc17c19f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:31.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.251+0000 7fc181a0d700 1 --2- 172.21.15.70:0/2904880247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc17c102e00 0x7fc17c19fcc0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc16c000c00 tx=0x7fc16c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:31.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.251+0000 7fc1737fe700 1 -- 172.21.15.70:0/2904880247 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c027070 con 0x7fc17c102e00 2024-06-04T04:58:31.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.251+0000 7fc1737fe700 1 -- 172.21.15.70:0/2904880247 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc16c00fc70 con 0x7fc17c102e00 2024-06-04T04:58:31.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.251+0000 7fc184472700 1 -- 172.21.15.70:0/2904880247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc17c19a330 con 0x7fc17c102e00 2024-06-04T04:58:31.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.251+0000 7fc1737fe700 1 -- 172.21.15.70:0/2904880247 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c022d40 con 0x7fc17c102e00 2024-06-04T04:58:31.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.252+0000 7fc184472700 1 -- 172.21.15.70:0/2904880247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc17c1a6650 con 0x7fc17c102e00 2024-06-04T04:58:31.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.254+0000 7fc1737fe700 1 -- 172.21.15.70:0/2904880247 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 149885+0+0 (secure 0 0 0) 0x7fc16c02a070 con 0x7fc17c102e00 2024-06-04T04:58:31.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.255+0000 7fc1737fe700 1 --2- 172.21.15.70:0/2904880247 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc1680b2b90 0x7fc1680b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.255+0000 7fc18220e700 1 --2- 172.21.15.70:0/2904880247 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc1680b2b90 0x7fc1680b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.255+0000 7fc1737fe700 1 -- 172.21.15.70:0/2904880247 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc16c0e3660 con 0x7fc17c102e00 2024-06-04T04:58:31.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.255+0000 7fc18ed07b80 1 Processor -- start 2024-06-04T04:58:31.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.255+0000 7fc18ed07b80 1 -- start start 2024-06-04T04:58:31.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc18220e700 1 --2- 172.21.15.70:0/2904880247 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc1680b2b90 0x7fc1680b5050 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc178005e20 tx=0x7fc178005db0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:31.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc18ed07b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680ca320 0x55b9680ca700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc18ed07b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680c48a0 0x55b9680c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc18ed07b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c5260 0x55b9680d1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc18ed07b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b968022d70 con 0x55b9680ca320 2024-06-04T04:58:31.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc18ed07b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b968012910 con 0x55b9680c5260 2024-06-04T04:58:31.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc18ed07b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b968016950 con 0x55b9680c48a0 2024-06-04T04:58:31.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc18220e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680ca320 0x55b9680ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc18220e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680ca320 0x55b9680ca700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35926/0 (socket says 172.21.15.70:35926) 2024-06-04T04:58:31.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc18220e700 1 -- 172.21.15.70:0/3087331492 learned_addr learned my addr 172.21.15.70:0/3087331492 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:31.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc181a0d700 1 --2- 172.21.15.70:0/3087331492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680c48a0 0x55b9680c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc182a0f700 1 --2- 172.21.15.70:0/3087331492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c5260 0x55b9680d1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc18220e700 1 -- 172.21.15.70:0/3087331492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680c48a0 msgr2=0x55b9680c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:31.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc18220e700 1 --2- 172.21.15.70:0/3087331492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680c48a0 0x55b9680c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc18220e700 1 -- 172.21.15.70:0/3087331492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c5260 msgr2=0x55b9680d1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:31.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc18220e700 1 --2- 172.21.15.70:0/3087331492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c5260 0x55b9680d1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc18220e700 1 -- 172.21.15.70:0/3087331492 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b967fbbfb0 con 0x55b9680ca320 2024-06-04T04:58:31.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.256+0000 7fc18220e700 1 --2- 172.21.15.70:0/3087331492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680ca320 0x55b9680ca700 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc17800ec50 tx=0x7fc178011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f64390a9c2befc server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:31.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.257+0000 7fc170d70700 1 -- 172.21.15.70:0/3087331492 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178021070 con 0x55b9680ca320 2024-06-04T04:58:31.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.257+0000 7fc170d70700 1 -- 172.21.15.70:0/3087331492 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc178010cd0 con 0x55b9680ca320 2024-06-04T04:58:31.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.257+0000 7fc170d70700 1 -- 172.21.15.70:0/3087331492 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17800f5e0 con 0x55b9680ca320 2024-06-04T04:58:31.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.257+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3087331492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680ca320 msgr2=0x55b9680ca700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:31.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.257+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/3087331492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680ca320 0x55b9680ca700 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc17800ec50 tx=0x7fc178011000 comp rx=0 tx=0).stop 2024-06-04T04:58:31.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.257+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3087331492 shutdown_connections 2024-06-04T04:58:31.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.257+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/3087331492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c5260 0x55b9680d1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.257+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/3087331492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680ca320 0x55b9680ca700 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.257+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/3087331492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680c48a0 0x55b9680c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.257+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3087331492 >> 172.21.15.70:0/3087331492 conn(0x55b967ff0240 msgr2=0x55b9680bf3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:31.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.258+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3087331492 shutdown_connections 2024-06-04T04:58:31.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.258+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3087331492 wait complete. 2024-06-04T04:58:31.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.258+0000 7fc18ed07b80 1 -- start start 2024-06-04T04:58:31.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.258+0000 7fc18ed07b80 1 Processor -- start 2024-06-04T04:58:31.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.258+0000 7fc18ed07b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680c48a0 0x55b9680d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.258+0000 7fc18ed07b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c5260 0x55b9680cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.258+0000 7fc18ed07b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680ca320 0x55b9680ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc18ed07b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b967f18fa0 con 0x55b9680c48a0 2024-06-04T04:58:31.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc18ed07b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b967f7ada0 con 0x55b9680c5260 2024-06-04T04:58:31.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc18ed07b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b9680d5af0 con 0x55b9680ca320 2024-06-04T04:58:31.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc18ed07b80 10 client.0 _ll_register_callbacks cb 0x55b967fad950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:31.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc18220e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680c48a0 0x55b9680d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc18220e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680c48a0 0x55b9680d0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35934/0 (socket says 172.21.15.70:35934) 2024-06-04T04:58:31.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc18220e700 1 -- 172.21.15.70:0/3630729283 learned_addr learned my addr 172.21.15.70:0/3630729283 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:31.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc182a0f700 1 --2- 172.21.15.70:0/3630729283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680ca320 0x55b9680ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc181a0d700 1 --2- 172.21.15.70:0/3630729283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c5260 0x55b9680cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc18220e700 1 -- 172.21.15.70:0/3630729283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680ca320 msgr2=0x55b9680ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:31.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc18220e700 1 --2- 172.21.15.70:0/3630729283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680ca320 0x55b9680ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc18220e700 1 -- 172.21.15.70:0/3630729283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c5260 msgr2=0x55b9680cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:31.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc18220e700 1 --2- 172.21.15.70:0/3630729283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c5260 0x55b9680cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc18220e700 1 -- 172.21.15.70:0/3630729283 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b967f628d0 con 0x55b9680c48a0 2024-06-04T04:58:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc18220e700 1 --2- 172.21.15.70:0/3630729283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680c48a0 0x55b9680d0cb0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc17801dba0 tx=0x7fc17801dbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32586051c6ed9a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc165ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc165ffb700 1 -- 172.21.15.70:0/3630729283 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178035060 con 0x55b9680c48a0 2024-06-04T04:58:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc165ffb700 1 -- 172.21.15.70:0/3630729283 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc178010ad0 con 0x55b9680c48a0 2024-06-04T04:58:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.259+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3630729283 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc178010cd0 con 0x55b9680c48a0 2024-06-04T04:58:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.260+0000 7fc165ffb700 1 -- 172.21.15.70:0/3630729283 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc178021050 con 0x55b9680c48a0 2024-06-04T04:58:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.260+0000 7fc18ed07b80 10 client.15813 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.260+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3630729283 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b967fcc760 con 0x55b9680c48a0 2024-06-04T04:58:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.260+0000 7fc165ffb700 1 -- 172.21.15.70:0/3630729283 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc178010cd0 con 0x55b9680c48a0 2024-06-04T04:58:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.260+0000 7fc165ffb700 1 -- 172.21.15.70:0/3630729283 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc17800f4e0 con 0x55b9680c48a0 2024-06-04T04:58:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.260+0000 7fc18ed07b80 10 client.15813 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.260+0000 7fc18ed07b80 10 client.15813 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.260+0000 7fc18ed07b80 20 client.15813 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:31.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.260+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/3630729283 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b9680e2530 0x55b9680e4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.260+0000 7fc18ed07b80 4 client.15813 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:31.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.260+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3630729283 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b968024cc0 con 0x55b9680e2530 2024-06-04T04:58:31.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.260+0000 7fc181a0d700 1 --2- 172.21.15.70:0/3630729283 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b9680e2530 0x55b9680e4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.261+0000 7fc165ffb700 10 client.15813 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:31.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.261+0000 7fc181a0d700 1 --2- 172.21.15.70:0/3630729283 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b9680e2530 0x55b9680e4920 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43dda9148b99a9a3 server_cookie=b30a8db3a792953a in_seq=0 out_seq=0 2024-06-04T04:58:31.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.261+0000 7fc165ffb700 1 -- 172.21.15.70:0/3630729283 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b968024cc0 con 0x55b9680e2530 2024-06-04T04:58:31.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.261+0000 7fc165ffb700 10 client.15813 handle_command_reply: tid=0 2024-06-04T04:58:31.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.262+0000 7fc18ed07b80 1 client.15813 shutdown 2024-06-04T04:58:31.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.262+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3630729283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680c48a0 msgr2=0x55b9680d0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:31.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.262+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/3630729283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680c48a0 0x55b9680d0cb0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc17801dba0 tx=0x7fc17801dbd0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.262+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3630729283 shutdown_connections 2024-06-04T04:58:31.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.262+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/3630729283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680ca320 0x55b9680ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.262+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/3630729283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680c48a0 0x55b9680d0cb0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.262+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/3630729283 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b9680e2530 0x55b9680e4920 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.262+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/3630729283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c5260 0x55b9680cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.262+0000 7fc165ffb700 0 client.15813 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:31.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.262+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3630729283 >> 172.21.15.70:0/3630729283 conn(0x55b967ff0240 msgr2=0x55b9680beaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:31.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.262+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3630729283 shutdown_connections 2024-06-04T04:58:31.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.262+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3630729283 wait complete. 2024-06-04T04:58:31.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.262+0000 7fc18ed07b80 20 client.15813 trim_cache size 0 max 16384 2024-06-04T04:58:31.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.298+0000 7fc18ed07b80 1 Processor -- start 2024-06-04T04:58:31.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.298+0000 7fc18ed07b80 1 -- start start 2024-06-04T04:58:31.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.298+0000 7fc18ed07b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c48a0 0x55b9680ca700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.298+0000 7fc18ed07b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680c5260 0x55b9680bf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.298+0000 7fc18ed07b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680ca320 0x55b9680cb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.298+0000 7fc18ed07b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c1abb10 con 0x55b9680ca320 2024-06-04T04:58:31.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.298+0000 7fc18ed07b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c1ad650 con 0x55b9680c48a0 2024-06-04T04:58:31.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.298+0000 7fc18ed07b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c1ac1e0 con 0x55b9680c5260 2024-06-04T04:58:31.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.299+0000 7fc182a0f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680ca320 0x55b9680cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.299+0000 7fc182a0f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680ca320 0x55b9680cb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35946/0 (socket says 172.21.15.70:35946) 2024-06-04T04:58:31.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.299+0000 7fc182a0f700 1 -- 172.21.15.70:0/3568550224 learned_addr learned my addr 172.21.15.70:0/3568550224 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:31.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.299+0000 7fc18220e700 1 --2- 172.21.15.70:0/3568550224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c48a0 0x55b9680ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.299+0000 7fc181a0d700 1 --2- 172.21.15.70:0/3568550224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680c5260 0x55b9680bf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.299+0000 7fc182a0f700 1 -- 172.21.15.70:0/3568550224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680c5260 msgr2=0x55b9680bf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:31.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.299+0000 7fc182a0f700 1 --2- 172.21.15.70:0/3568550224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680c5260 0x55b9680bf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.299+0000 7fc182a0f700 1 -- 172.21.15.70:0/3568550224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c48a0 msgr2=0x55b9680ca700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:31.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.299+0000 7fc182a0f700 1 --2- 172.21.15.70:0/3568550224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c48a0 0x55b9680ca700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.299+0000 7fc182a0f700 1 -- 172.21.15.70:0/3568550224 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9680c9f40 con 0x55b9680ca320 2024-06-04T04:58:31.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.299+0000 7fc181a0d700 1 --2- 172.21.15.70:0/3568550224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680c5260 0x55b9680bf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:31.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.299+0000 7fc182a0f700 1 --2- 172.21.15.70:0/3568550224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680ca320 0x55b9680cb010 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc174009f50 tx=0x7fc1740077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d928153c49a143 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:31.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.299+0000 7fc167fff700 1 -- 172.21.15.70:0/3568550224 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17401b070 con 0x55b9680ca320 2024-06-04T04:58:31.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.299+0000 7fc167fff700 1 -- 172.21.15.70:0/3568550224 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc1740178d0 con 0x55b9680ca320 2024-06-04T04:58:31.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.299+0000 7fc167fff700 1 -- 172.21.15.70:0/3568550224 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc174016880 con 0x55b9680ca320 2024-06-04T04:58:31.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.300+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3568550224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680ca320 msgr2=0x55b9680cb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:31.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.300+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/3568550224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680ca320 0x55b9680cb010 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc174009f50 tx=0x7fc1740077e0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.300+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3568550224 shutdown_connections 2024-06-04T04:58:31.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.300+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/3568550224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680ca320 0x55b9680cb010 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.300+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/3568550224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c48a0 0x55b9680ca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.300+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/3568550224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680c5260 0x55b9680bf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.300+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3568550224 >> 172.21.15.70:0/3568550224 conn(0x55b967ff0240 msgr2=0x55b9680c4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:31.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.300+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3568550224 shutdown_connections 2024-06-04T04:58:31.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.300+0000 7fc18ed07b80 1 -- 172.21.15.70:0/3568550224 wait complete. 2024-06-04T04:58:31.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc18ed07b80 1 -- start start 2024-06-04T04:58:31.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc18ed07b80 1 Processor -- start 2024-06-04T04:58:31.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc18ed07b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c48a0 0x55b9680e3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc18ed07b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680c5260 0x55b9680e3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc18ed07b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680ca320 0x55b96814bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc18ed07b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c1acf80 con 0x55b9680c5260 2024-06-04T04:58:31.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc18ed07b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c1ac8b0 con 0x55b9680c48a0 2024-06-04T04:58:31.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc18ed07b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c1a9ab0 con 0x55b9680ca320 2024-06-04T04:58:31.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc18ed07b80 10 client.0 _ll_register_callbacks cb 0x55b967fad950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:31.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc181a0d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680c5260 0x55b9680e3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc181a0d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680c5260 0x55b9680e3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35948/0 (socket says 172.21.15.70:35948) 2024-06-04T04:58:31.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc181a0d700 1 -- 172.21.15.70:0/1519160285 learned_addr learned my addr 172.21.15.70:0/1519160285 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:31.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc18220e700 1 --2- 172.21.15.70:0/1519160285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c48a0 0x55b9680e3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc182a0f700 1 --2- 172.21.15.70:0/1519160285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680ca320 0x55b96814bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc181a0d700 1 -- 172.21.15.70:0/1519160285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680ca320 msgr2=0x55b96814bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:31.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc181a0d700 1 --2- 172.21.15.70:0/1519160285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680ca320 0x55b96814bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc181a0d700 1 -- 172.21.15.70:0/1519160285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c48a0 msgr2=0x55b9680e3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:31.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc181a0d700 1 --2- 172.21.15.70:0/1519160285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c48a0 0x55b9680e3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.301+0000 7fc181a0d700 1 -- 172.21.15.70:0/1519160285 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b96814c2a0 con 0x55b9680c5260 2024-06-04T04:58:31.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.302+0000 7fc182a0f700 1 --2- 172.21.15.70:0/1519160285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680ca320 0x55b96814bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:31.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.302+0000 7fc18220e700 1 --2- 172.21.15.70:0/1519160285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c48a0 0x55b9680e3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:31.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.302+0000 7fc181a0d700 1 --2- 172.21.15.70:0/1519160285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680c5260 0x55b9680e3f40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc16c0e13b0 tx=0x7fc16c0cd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828c61f012b470d4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:31.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.302+0000 7fc165ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:31.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.302+0000 7fc165ffb700 1 -- 172.21.15.70:0/1519160285 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c0aa780 con 0x55b9680c5260 2024-06-04T04:58:31.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.302+0000 7fc165ffb700 1 -- 172.21.15.70:0/1519160285 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc16c0b32d0 con 0x55b9680c5260 2024-06-04T04:58:31.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.302+0000 7fc18ed07b80 1 -- 172.21.15.70:0/1519160285 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1740178d0 con 0x55b9680c5260 2024-06-04T04:58:31.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.302+0000 7fc165ffb700 1 -- 172.21.15.70:0/1519160285 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc16c0aeb30 con 0x55b9680c5260 2024-06-04T04:58:31.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.302+0000 7fc18ed07b80 10 client.15819 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:31.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.302+0000 7fc18ed07b80 1 -- 172.21.15.70:0/1519160285 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b96814d450 con 0x55b9680c5260 2024-06-04T04:58:31.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.302+0000 7fc165ffb700 1 -- 172.21.15.70:0/1519160285 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc1740178d0 con 0x55b9680c5260 2024-06-04T04:58:31.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.302+0000 7fc165ffb700 1 -- 172.21.15.70:0/1519160285 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc16c0bfb40 con 0x55b9680c5260 2024-06-04T04:58:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.303+0000 7fc18ed07b80 10 client.15819 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.303+0000 7fc18ed07b80 10 client.15819 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.303+0000 7fc18ed07b80 20 client.15819 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.303+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/1519160285 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b96814db60 0x55b96814ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.303+0000 7fc18ed07b80 4 client.15819 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T04:58:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.303+0000 7fc18ed07b80 1 -- 172.21.15.70:0/1519160285 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b9680c9d20 con 0x55b96814db60 2024-06-04T04:58:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.303+0000 7fc18220e700 1 --2- 172.21.15.70:0/1519160285 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b96814db60 0x55b96814ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.303+0000 7fc18220e700 1 --2- 172.21.15.70:0/1519160285 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b96814db60 0x55b96814ff80 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da228bc1bb841c52 server_cookie=c58520a9b8358897 in_seq=0 out_seq=0 2024-06-04T04:58:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.303+0000 7fc165ffb700 10 client.15819 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.304+0000 7fc165ffb700 1 -- 172.21.15.70:0/1519160285 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b9680c9d20 con 0x55b96814db60 2024-06-04T04:58:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.304+0000 7fc165ffb700 10 client.15819 handle_command_reply: tid=0 2024-06-04T04:58:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.304+0000 7fc18ed07b80 1 client.15819 shutdown 2024-06-04T04:58:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.304+0000 7fc18ed07b80 1 -- 172.21.15.70:0/1519160285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680c5260 msgr2=0x55b9680e3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:31.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.304+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/1519160285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680c5260 0x55b9680e3f40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc16c0e13b0 tx=0x7fc16c0cd040 comp rx=0 tx=0).stop 2024-06-04T04:58:31.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.304+0000 7fc18ed07b80 1 -- 172.21.15.70:0/1519160285 shutdown_connections 2024-06-04T04:58:31.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.304+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/1519160285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9680ca320 0x55b96814bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.304+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/1519160285 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b96814db60 0x55b96814ff80 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.304+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/1519160285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9680c48a0 0x55b9680e3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.304+0000 7fc18ed07b80 1 --2- 172.21.15.70:0/1519160285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9680c5260 0x55b9680e3f40 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.304+0000 7fc18ed07b80 1 -- 172.21.15.70:0/1519160285 >> 172.21.15.70:0/1519160285 conn(0x55b967ff0240 msgr2=0x55b9680c7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:31.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.304+0000 7fc165ffb700 0 client.15819 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:31.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.304+0000 7fc18ed07b80 1 -- 172.21.15.70:0/1519160285 shutdown_connections 2024-06-04T04:58:31.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.304+0000 7fc18ed07b80 1 -- 172.21.15.70:0/1519160285 wait complete. 2024-06-04T04:58:31.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.304+0000 7fc18ed07b80 20 client.15819 trim_cache size 0 max 16384 2024-06-04T04:58:31.312 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:58:31.312 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T04:58:31.312 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T04:58:31.312 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:58:31.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.305+0000 7fc184472700 1 -- 172.21.15.70:0/2904880247 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc1680b2b90 msgr2=0x7fc1680b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:31.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.305+0000 7fc184472700 1 --2- 172.21.15.70:0/2904880247 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc1680b2b90 0x7fc1680b5050 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc178005e20 tx=0x7fc178005db0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.305+0000 7fc184472700 1 -- 172.21.15.70:0/2904880247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc17c102e00 msgr2=0x7fc17c19fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:31.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.305+0000 7fc184472700 1 --2- 172.21.15.70:0/2904880247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc17c102e00 0x7fc17c19fcc0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc16c000c00 tx=0x7fc16c009f90 comp rx=0 tx=0).stop 2024-06-04T04:58:31.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.305+0000 7fc184472700 1 -- 172.21.15.70:0/2904880247 shutdown_connections 2024-06-04T04:58:31.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.305+0000 7fc184472700 1 --2- 172.21.15.70:0/2904880247 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc1680b2b90 0x7fc1680b5050 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.305+0000 7fc184472700 1 --2- 172.21.15.70:0/2904880247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc17c108520 0x7fc17c199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.305+0000 7fc184472700 1 --2- 172.21.15.70:0/2904880247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc17c102440 0x7fc17c19f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.305+0000 7fc184472700 1 --2- 172.21.15.70:0/2904880247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc17c102e00 0x7fc17c19fcc0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:31.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.305+0000 7fc184472700 1 -- 172.21.15.70:0/2904880247 >> 172.21.15.70:0/2904880247 conn(0x7fc17c0793f0 msgr2=0x7fc17c06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:31.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.305+0000 7fc184472700 1 -- 172.21.15.70:0/2904880247 shutdown_connections 2024-06-04T04:58:31.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:31.305+0000 7fc184472700 1 -- 172.21.15.70:0/2904880247 wait complete. 2024-06-04T04:58:31.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:31 smithi070 ceph-mon[32631]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:58:31.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:31 smithi112 ceph-mon[36453]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:58:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:31 smithi098 ceph-mon[35569]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:58:32.135 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T04:58:32.135 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:58:32.405 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:33.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:33 smithi070 ceph-mon[32631]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 511 B/s wr, 14 op/s 2024-06-04T04:58:33.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:33 smithi112 ceph-mon[36453]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 511 B/s wr, 14 op/s 2024-06-04T04:58:33.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:33 smithi098 ceph-mon[35569]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 511 B/s wr, 14 op/s 2024-06-04T04:58:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.225+0000 7f3428a4c700 1 -- 172.21.15.70:0/1570006260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3424101e60 msgr2=0x7f3424102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.225+0000 7f3428a4c700 1 --2- 172.21.15.70:0/1570006260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3424101e60 0x7f3424102240 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f341400aad0 tx=0x7f341400ade0 comp rx=0 tx=0).stop 2024-06-04T04:58:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.225+0000 7f3428a4c700 1 -- 172.21.15.70:0/1570006260 shutdown_connections 2024-06-04T04:58:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.225+0000 7f3428a4c700 1 --2- 172.21.15.70:0/1570006260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3424106ef0 0x7f342410eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.225+0000 7f3428a4c700 1 --2- 172.21.15.70:0/1570006260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3424101e60 0x7f3424102240 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.225+0000 7f3428a4c700 1 --2- 172.21.15.70:0/1570006260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3424102810 0x7f3424106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.225+0000 7f3428a4c700 1 -- 172.21.15.70:0/1570006260 >> 172.21.15.70:0/1570006260 conn(0x7f3424078ab0 msgr2=0x7f34240ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.225+0000 7f3428a4c700 1 -- 172.21.15.70:0/1570006260 shutdown_connections 2024-06-04T04:58:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.225+0000 7f3428a4c700 1 -- 172.21.15.70:0/1570006260 wait complete. 2024-06-04T04:58:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.226+0000 7f3428a4c700 1 Processor -- start 2024-06-04T04:58:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.226+0000 7f3428a4c700 1 -- start start 2024-06-04T04:58:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.226+0000 7f3428a4c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3424101e60 0x7f342419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.227+0000 7f3428a4c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3424102810 0x7f342419db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.227+0000 7f3428a4c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3424106ef0 0x7f3424197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.227+0000 7f3428a4c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3424111420 con 0x7f3424106ef0 2024-06-04T04:58:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.227+0000 7f3428a4c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f34241112a0 con 0x7f3424101e60 2024-06-04T04:58:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.227+0000 7f3428a4c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f34241115a0 con 0x7f3424102810 2024-06-04T04:58:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.227+0000 7f3422d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3424106ef0 0x7f3424197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.227+0000 7f3422d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3424106ef0 0x7f3424197780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35962/0 (socket says 172.21.15.70:35962) 2024-06-04T04:58:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.227+0000 7f3422d9d700 1 -- 172.21.15.70:0/814896794 learned_addr learned my addr 172.21.15.70:0/814896794 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.227+0000 7f3421d9b700 1 --2- 172.21.15.70:0/814896794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3424102810 0x7f342419db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:34.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.227+0000 7f342259c700 1 --2- 172.21.15.70:0/814896794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3424101e60 0x7f342419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:34.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.227+0000 7f3421d9b700 1 -- 172.21.15.70:0/814896794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3424101e60 msgr2=0x7f342419d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:34.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.227+0000 7f3421d9b700 1 --2- 172.21.15.70:0/814896794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3424101e60 0x7f342419d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:34.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.227+0000 7f3421d9b700 1 -- 172.21.15.70:0/814896794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3424106ef0 msgr2=0x7f3424197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:34.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.227+0000 7f3421d9b700 1 --2- 172.21.15.70:0/814896794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3424106ef0 0x7f3424197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:34.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.227+0000 7f3421d9b700 1 -- 172.21.15.70:0/814896794 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3424197f60 con 0x7f3424102810 2024-06-04T04:58:34.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.227+0000 7f3421d9b700 1 --2- 172.21.15.70:0/814896794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3424102810 0x7f342419db40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f341800e9c0 tx=0x7f341800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:34.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.228+0000 7f34137fe700 1 -- 172.21.15.70:0/814896794 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341800cbd0 con 0x7f3424102810 2024-06-04T04:58:34.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.228+0000 7f3428a4c700 1 -- 172.21.15.70:0/814896794 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3424198250 con 0x7f3424102810 2024-06-04T04:58:34.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.228+0000 7f3428a4c700 1 -- 172.21.15.70:0/814896794 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34241a4460 con 0x7f3424102810 2024-06-04T04:58:34.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.228+0000 7f34137fe700 1 -- 172.21.15.70:0/814896794 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f341800cd70 con 0x7f3424102810 2024-06-04T04:58:34.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.228+0000 7f34137fe700 1 -- 172.21.15.70:0/814896794 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3418018680 con 0x7f3424102810 2024-06-04T04:58:34.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.229+0000 7f3428a4c700 1 -- 172.21.15.70:0/814896794 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3404005330 con 0x7f3424102810 2024-06-04T04:58:34.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.231+0000 7f34137fe700 1 -- 172.21.15.70:0/814896794 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 24) ==== 149885+0+0 (secure 0 0 0) 0x7f341804ab90 con 0x7f3424102810 2024-06-04T04:58:34.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.232+0000 7f34137fe700 1 --2- 172.21.15.70:0/814896794 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f340c0b2b10 0x7f340c0b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:34.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.233+0000 7f342259c700 1 --2- 172.21.15.70:0/814896794 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f340c0b2b10 0x7f340c0b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:34.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.233+0000 7f34137fe700 1 -- 172.21.15.70:0/814896794 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f34180e2950 con 0x7f3424102810 2024-06-04T04:58:34.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.233+0000 7f342259c700 1 --2- 172.21.15.70:0/814896794 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f340c0b2b10 0x7f340c0b4fd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f341400afd0 tx=0x7f3414005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:34.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.235+0000 7f34137fe700 1 -- 172.21.15.70:0/814896794 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f34180aa0e0 con 0x7f3424102810 2024-06-04T04:58:34.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.567+0000 7f3428a4c700 1 -- 172.21.15.70:0/814896794 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3404005620 con 0x7f3424102810 2024-06-04T04:58:34.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.568+0000 7f34137fe700 1 -- 172.21.15.70:0/814896794 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f34180b2160 con 0x7f3424102810 2024-06-04T04:58:34.570 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:58:34.571 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:57:09.013178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T04:58:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.571+0000 7f3428a4c700 1 -- 172.21.15.70:0/814896794 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f340c0b2b10 msgr2=0x7f340c0b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.571+0000 7f3428a4c700 1 --2- 172.21.15.70:0/814896794 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f340c0b2b10 0x7f340c0b4fd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f341400afd0 tx=0x7f3414005fb0 comp rx=0 tx=0).stop 2024-06-04T04:58:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.571+0000 7f3428a4c700 1 -- 172.21.15.70:0/814896794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3424102810 msgr2=0x7f342419db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.571+0000 7f3428a4c700 1 --2- 172.21.15.70:0/814896794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3424102810 0x7f342419db40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f341800e9c0 tx=0x7f341800ecd0 comp rx=0 tx=0).stop 2024-06-04T04:58:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.571+0000 7f3428a4c700 1 -- 172.21.15.70:0/814896794 shutdown_connections 2024-06-04T04:58:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.571+0000 7f3428a4c700 1 --2- 172.21.15.70:0/814896794 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f340c0b2b10 0x7f340c0b4fd0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.571+0000 7f3428a4c700 1 --2- 172.21.15.70:0/814896794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3424106ef0 0x7f3424197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.571+0000 7f3428a4c700 1 --2- 172.21.15.70:0/814896794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3424101e60 0x7f342419d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.571+0000 7f3428a4c700 1 --2- 172.21.15.70:0/814896794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3424102810 0x7f342419db40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.571+0000 7f3428a4c700 1 -- 172.21.15.70:0/814896794 >> 172.21.15.70:0/814896794 conn(0x7f3424078ab0 msgr2=0x7f34240ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.571+0000 7f3428a4c700 1 -- 172.21.15.70:0/814896794 shutdown_connections 2024-06-04T04:58:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:34.571+0000 7f3428a4c700 1 -- 172.21.15.70:0/814896794 wait complete. 2024-06-04T04:58:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-06-04T04:58:35.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:35 smithi070 ceph-mon[32631]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-06-04T04:58:35.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:35 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/814896794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:58:35.403 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T04:58:35.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:35 smithi112 ceph-mon[36453]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-06-04T04:58:35.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:35 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/814896794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:58:35.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:35 smithi098 ceph-mon[35569]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-06-04T04:58:35.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:35 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/814896794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:58:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:37.163 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:37 smithi070 ceph-mon[32631]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 14 op/s 2024-06-04T04:58:37.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:37 smithi112 ceph-mon[36453]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 14 op/s 2024-06-04T04:58:37.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:37 smithi098 ceph-mon[35569]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 14 op/s 2024-06-04T04:58:37.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.580+0000 7f5357c4a700 1 -- 172.21.15.70:0/2888904014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f535006bb70 msgr2=0x7f535010fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:37.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.580+0000 7f5357c4a700 1 --2- 172.21.15.70:0/2888904014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f535006bb70 0x7f535010fab0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5340009990 tx=0x7f5340009ca0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.580+0000 7f5357c4a700 1 -- 172.21.15.70:0/2888904014 shutdown_connections 2024-06-04T04:58:37.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.580+0000 7f5357c4a700 1 --2- 172.21.15.70:0/2888904014 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f535010fff0 0x7f53501123e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.580+0000 7f5357c4a700 1 --2- 172.21.15.70:0/2888904014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f535006b250 0x7f535006b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.580+0000 7f5357c4a700 1 --2- 172.21.15.70:0/2888904014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f535006bb70 0x7f535010fab0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.580+0000 7f5357c4a700 1 -- 172.21.15.70:0/2888904014 >> 172.21.15.70:0/2888904014 conn(0x7f53500793f0 msgr2=0x7f5350079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:37.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.580+0000 7f5357c4a700 1 -- 172.21.15.70:0/2888904014 shutdown_connections 2024-06-04T04:58:37.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.580+0000 7f5357c4a700 1 -- 172.21.15.70:0/2888904014 wait complete. 2024-06-04T04:58:37.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.581+0000 7f5357c4a700 1 Processor -- start 2024-06-04T04:58:37.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.581+0000 7f5357c4a700 1 -- start start 2024-06-04T04:58:37.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.581+0000 7f5357c4a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f535006b250 0x7f535019bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.581+0000 7f5357c4a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f535006bb70 0x7f535019c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.581+0000 7f5357c4a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f535010fff0 0x7f53501a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.581+0000 7f5357c4a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5350115830 con 0x7f535006b250 2024-06-04T04:58:37.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.581+0000 7f5357c4a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53501156b0 con 0x7f535006bb70 2024-06-04T04:58:37.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.581+0000 7f5357c4a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f53501159b0 con 0x7f535010fff0 2024-06-04T04:58:37.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.582+0000 7f53559e6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f535006b250 0x7f535019bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:37.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.582+0000 7f53559e6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f535006b250 0x7f535019bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36938/0 (socket says 172.21.15.70:36938) 2024-06-04T04:58:37.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.582+0000 7f53559e6700 1 -- 172.21.15.70:0/4198147885 learned_addr learned my addr 172.21.15.70:0/4198147885 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:37.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.582+0000 7f53551e5700 1 --2- 172.21.15.70:0/4198147885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f535006bb70 0x7f535019c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:37.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.582+0000 7f53559e6700 1 -- 172.21.15.70:0/4198147885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f535010fff0 msgr2=0x7f53501a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:37.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.582+0000 7f53559e6700 1 --2- 172.21.15.70:0/4198147885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f535010fff0 0x7f53501a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.582+0000 7f53559e6700 1 -- 172.21.15.70:0/4198147885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f535006bb70 msgr2=0x7f535019c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:37.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.582+0000 7f53559e6700 1 --2- 172.21.15.70:0/4198147885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f535006bb70 0x7f535019c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.582+0000 7f53559e6700 1 -- 172.21.15.70:0/4198147885 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53501a0d90 con 0x7f535006b250 2024-06-04T04:58:37.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.582+0000 7f53559e6700 1 --2- 172.21.15.70:0/4198147885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f535006b250 0x7f535019bd80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f534c00d980 tx=0x7f534c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:37.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.582+0000 7f5346ffd700 1 -- 172.21.15.70:0/4198147885 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534c016070 con 0x7f535006b250 2024-06-04T04:58:37.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.583+0000 7f5357c4a700 1 -- 172.21.15.70:0/4198147885 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53501a1080 con 0x7f535006b250 2024-06-04T04:58:37.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.583+0000 7f5346ffd700 1 -- 172.21.15.70:0/4198147885 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f534c004830 con 0x7f535006b250 2024-06-04T04:58:37.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.583+0000 7f5346ffd700 1 -- 172.21.15.70:0/4198147885 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534c005440 con 0x7f535006b250 2024-06-04T04:58:37.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.583+0000 7f5357c4a700 1 -- 172.21.15.70:0/4198147885 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53501a8900 con 0x7f535006b250 2024-06-04T04:58:37.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.586+0000 7f5346ffd700 1 -- 172.21.15.70:0/4198147885 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 149885+0+0 (secure 0 0 0) 0x7f534c0055e0 con 0x7f535006b250 2024-06-04T04:58:37.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.587+0000 7f5346ffd700 1 --2- 172.21.15.70:0/4198147885 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f533c0b2b10 0x7f533c0b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.587+0000 7f53551e5700 1 --2- 172.21.15.70:0/4198147885 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f533c0b2b10 0x7f533c0b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:37.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.587+0000 7f5346ffd700 1 -- 172.21.15.70:0/4198147885 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f534c0e26f0 con 0x7f535006b250 2024-06-04T04:58:37.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.587+0000 7f53624dfb80 1 Processor -- start 2024-06-04T04:58:37.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.587+0000 7f53624dfb80 1 -- start start 2024-06-04T04:58:37.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.588+0000 7f53624dfb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f302e070 0x55f3f302e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.588+0000 7f53624dfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302ea20 0x55f3f3032150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.588+0000 7f53624dfb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f3032770 0x55f3f3032bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.588+0000 7f53624dfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f3f2f84c90 con 0x55f3f302ea20 2024-06-04T04:58:37.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.588+0000 7f53551e5700 1 --2- 172.21.15.70:0/4198147885 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f533c0b2b10 0x7f533c0b4fd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5340003760 tx=0x7f5340005fd0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:37.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.588+0000 7f53624dfb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f3f2f74830 con 0x55f3f302e070 2024-06-04T04:58:37.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.588+0000 7f53624dfb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f3f2f78870 con 0x55f3f3032770 2024-06-04T04:58:37.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.588+0000 7f53551e5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302ea20 0x55f3f3032150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:37.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.588+0000 7f53551e5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302ea20 0x55f3f3032150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36950/0 (socket says 172.21.15.70:36950) 2024-06-04T04:58:37.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.588+0000 7f53551e5700 1 -- 172.21.15.70:0/4019219525 learned_addr learned my addr 172.21.15.70:0/4019219525 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:37.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.588+0000 7f53559e6700 1 --2- 172.21.15.70:0/4019219525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f302e070 0x55f3f302e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:37.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.588+0000 7f53561e7700 1 --2- 172.21.15.70:0/4019219525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f3032770 0x55f3f3032bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:37.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.588+0000 7f53551e5700 1 -- 172.21.15.70:0/4019219525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f3032770 msgr2=0x55f3f3032bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:37.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.588+0000 7f53551e5700 1 --2- 172.21.15.70:0/4019219525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f3032770 0x55f3f3032bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.588+0000 7f53551e5700 1 -- 172.21.15.70:0/4019219525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f302e070 msgr2=0x55f3f302e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:37.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.588+0000 7f53551e5700 1 --2- 172.21.15.70:0/4019219525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f302e070 0x55f3f302e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.588+0000 7f53551e5700 1 -- 172.21.15.70:0/4019219525 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3f2ec48d0 con 0x55f3f302ea20 2024-06-04T04:58:37.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.589+0000 7f53551e5700 1 --2- 172.21.15.70:0/4019219525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302ea20 0x55f3f3032150 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5340009750 tx=0x7f534001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df0ef53121d596a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:37.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.589+0000 7f533bfff700 1 -- 172.21.15.70:0/4019219525 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53400055e0 con 0x55f3f302ea20 2024-06-04T04:58:37.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.589+0000 7f533bfff700 1 -- 172.21.15.70:0/4019219525 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f534000bc60 con 0x55f3f302ea20 2024-06-04T04:58:37.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.589+0000 7f533bfff700 1 -- 172.21.15.70:0/4019219525 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534000f750 con 0x55f3f302ea20 2024-06-04T04:58:37.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.590+0000 7f53624dfb80 1 -- 172.21.15.70:0/4019219525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302ea20 msgr2=0x55f3f3032150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:37.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.590+0000 7f53624dfb80 1 --2- 172.21.15.70:0/4019219525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302ea20 0x55f3f3032150 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5340009750 tx=0x7f534001f000 comp rx=0 tx=0).stop 2024-06-04T04:58:37.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.590+0000 7f53624dfb80 1 -- 172.21.15.70:0/4019219525 shutdown_connections 2024-06-04T04:58:37.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.590+0000 7f53624dfb80 1 --2- 172.21.15.70:0/4019219525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f3032770 0x55f3f3032bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.590+0000 7f53624dfb80 1 --2- 172.21.15.70:0/4019219525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f302e070 0x55f3f302e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.590+0000 7f53624dfb80 1 --2- 172.21.15.70:0/4019219525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302ea20 0x55f3f3032150 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.590+0000 7f53624dfb80 1 -- 172.21.15.70:0/4019219525 >> 172.21.15.70:0/4019219525 conn(0x55f3f2f53080 msgr2=0x55f3f301fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:37.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.590+0000 7f53624dfb80 1 -- 172.21.15.70:0/4019219525 shutdown_connections 2024-06-04T04:58:37.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.590+0000 7f53624dfb80 1 -- 172.21.15.70:0/4019219525 wait complete. 2024-06-04T04:58:37.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.590+0000 7f53624dfb80 1 -- start start 2024-06-04T04:58:37.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.590+0000 7f53624dfb80 1 Processor -- start 2024-06-04T04:58:37.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.591+0000 7f53624dfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f3039c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.591+0000 7f53624dfb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f302ea20 0x55f3f3036f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.591+0000 7f53624dfb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f3032770 0x55f3f3037470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.591+0000 7f53624dfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f3f2e7afa0 con 0x55f3f302e070 2024-06-04T04:58:37.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.591+0000 7f53624dfb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f3f2f167c0 con 0x55f3f3032770 2024-06-04T04:58:37.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.591+0000 7f53624dfb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f3f303a870 con 0x55f3f302ea20 2024-06-04T04:58:37.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.591+0000 7f53624dfb80 10 client.0 _ll_register_callbacks cb 0x55f3f2f0f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:37.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.591+0000 7f53559e6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f3039c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:37.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.591+0000 7f53559e6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f3039c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36956/0 (socket says 172.21.15.70:36956) 2024-06-04T04:58:37.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.591+0000 7f53559e6700 1 -- 172.21.15.70:0/1918513445 learned_addr learned my addr 172.21.15.70:0/1918513445 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:37.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.591+0000 7f53561e7700 1 --2- 172.21.15.70:0/1918513445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f3032770 0x55f3f3037470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:37.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.591+0000 7f53559e6700 1 -- 172.21.15.70:0/1918513445 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f302ea20 msgr2=0x55f3f3036f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-04T04:58:37.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.592+0000 7f53559e6700 1 --2- 172.21.15.70:0/1918513445 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f302ea20 0x55f3f3036f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.592+0000 7f53559e6700 1 -- 172.21.15.70:0/1918513445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f3032770 msgr2=0x55f3f3037470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:37.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.592+0000 7f53559e6700 1 --2- 172.21.15.70:0/1918513445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f3032770 0x55f3f3037470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.592+0000 7f53559e6700 1 -- 172.21.15.70:0/1918513445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3f2f13690 con 0x55f3f302e070 2024-06-04T04:58:37.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.592+0000 7f53561e7700 1 --2- 172.21.15.70:0/1918513445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f3032770 0x55f3f3037470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:37.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.592+0000 7f53559e6700 1 --2- 172.21.15.70:0/1918513445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f3039c30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f534c0a5800 tx=0x7f534c0a9780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e611b43f9071510 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:37.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.592+0000 7f53397fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:37.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.592+0000 7f53397fa700 1 -- 172.21.15.70:0/1918513445 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534c0c3060 con 0x55f3f302e070 2024-06-04T04:58:37.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.592+0000 7f53397fa700 1 -- 172.21.15.70:0/1918513445 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f534c00fe70 con 0x55f3f302e070 2024-06-04T04:58:37.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.592+0000 7f53624dfb80 1 -- 172.21.15.70:0/1918513445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f534000bc60 con 0x55f3f302e070 2024-06-04T04:58:37.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.592+0000 7f53397fa700 1 -- 172.21.15.70:0/1918513445 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f534c0b25e0 con 0x55f3f302e070 2024-06-04T04:58:37.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.592+0000 7f53624dfb80 10 client.15837 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:37.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.592+0000 7f53624dfb80 1 -- 172.21.15.70:0/1918513445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f3f3045080 con 0x55f3f302e070 2024-06-04T04:58:37.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.593+0000 7f53397fa700 1 -- 172.21.15.70:0/1918513445 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f534000bc60 con 0x55f3f302e070 2024-06-04T04:58:37.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.593+0000 7f53397fa700 1 -- 172.21.15.70:0/1918513445 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f534c0b2150 con 0x55f3f302e070 2024-06-04T04:58:37.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.593+0000 7f53624dfb80 10 client.15837 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:37.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.593+0000 7f53624dfb80 10 client.15837 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T04:58:37.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.593+0000 7f53624dfb80 20 client.15837 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:37.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.593+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1918513445 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55f3f30476c0 0x55f3f3049ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.593+0000 7f53624dfb80 4 client.15837 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:37.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.593+0000 7f53624dfb80 1 -- 172.21.15.70:0/1918513445 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3f2ff19e0 con 0x55f3f30476c0 2024-06-04T04:58:37.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.593+0000 7f53551e5700 1 --2- 172.21.15.70:0/1918513445 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55f3f30476c0 0x55f3f3049ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:37.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.594+0000 7f53551e5700 1 --2- 172.21.15.70:0/1918513445 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55f3f30476c0 0x55f3f3049ab0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c9c9ee2ca942fdd6 server_cookie=1bd3678a39182bbb in_seq=0 out_seq=0 2024-06-04T04:58:37.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.594+0000 7f53397fa700 10 client.15837 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T04:58:37.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.594+0000 7f53397fa700 1 -- 172.21.15.70:0/1918513445 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f3f2ff19e0 con 0x55f3f30476c0 2024-06-04T04:58:37.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.594+0000 7f53397fa700 10 client.15837 handle_command_reply: tid=0 2024-06-04T04:58:37.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.594+0000 7f53624dfb80 1 client.15837 shutdown 2024-06-04T04:58:37.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.595+0000 7f53624dfb80 1 -- 172.21.15.70:0/1918513445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 msgr2=0x55f3f3039c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:37.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.595+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1918513445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f3039c30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f534c0a5800 tx=0x7f534c0a9780 comp rx=0 tx=0).stop 2024-06-04T04:58:37.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.595+0000 7f53624dfb80 1 -- 172.21.15.70:0/1918513445 shutdown_connections 2024-06-04T04:58:37.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.595+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1918513445 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55f3f30476c0 0x55f3f3049ab0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.595+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1918513445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f3032770 0x55f3f3037470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.595+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1918513445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f3039c30 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.595+0000 7f53397fa700 0 client.15837 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T04:58:37.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.595+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1918513445 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f302ea20 0x55f3f3036f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.595+0000 7f53624dfb80 1 -- 172.21.15.70:0/1918513445 >> 172.21.15.70:0/1918513445 conn(0x55f3f2f53080 msgr2=0x55f3f3033b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:37.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.595+0000 7f53624dfb80 1 -- 172.21.15.70:0/1918513445 shutdown_connections 2024-06-04T04:58:37.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.595+0000 7f53624dfb80 1 -- 172.21.15.70:0/1918513445 wait complete. 2024-06-04T04:58:37.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.595+0000 7f53624dfb80 20 client.15837 trim_cache size 0 max 16384 2024-06-04T04:58:37.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.615+0000 7f53624dfb80 1 Processor -- start 2024-06-04T04:58:37.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.615+0000 7f53624dfb80 1 -- start start 2024-06-04T04:58:37.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.616+0000 7f53624dfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f302d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.616+0000 7f53624dfb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f302ea20 0x55f3f3039770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.616+0000 7f53624dfb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f3032770 0x55f3f304f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.616+0000 7f53624dfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f53501abc70 con 0x55f3f302e070 2024-06-04T04:58:37.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.616+0000 7f53624dfb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53501ad7b0 con 0x55f3f302ea20 2024-06-04T04:58:37.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.616+0000 7f53624dfb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f53501ac340 con 0x55f3f3032770 2024-06-04T04:58:37.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.616+0000 7f53559e6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f302d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:37.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.616+0000 7f53559e6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f302d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36958/0 (socket says 172.21.15.70:36958) 2024-06-04T04:58:37.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.616+0000 7f53559e6700 1 -- 172.21.15.70:0/1791323984 learned_addr learned my addr 172.21.15.70:0/1791323984 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:37.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.616+0000 7f53551e5700 1 --2- 172.21.15.70:0/1791323984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f302ea20 0x55f3f3039770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:37.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.616+0000 7f53559e6700 1 -- 172.21.15.70:0/1791323984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f3032770 msgr2=0x55f3f304f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:37.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.616+0000 7f53559e6700 1 --2- 172.21.15.70:0/1791323984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f3032770 0x55f3f304f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.616+0000 7f53559e6700 1 -- 172.21.15.70:0/1791323984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f302ea20 msgr2=0x55f3f3039770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:37.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.616+0000 7f53559e6700 1 --2- 172.21.15.70:0/1791323984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f302ea20 0x55f3f3039770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.616+0000 7f53559e6700 1 -- 172.21.15.70:0/1791323984 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3f303a500 con 0x55f3f302e070 2024-06-04T04:58:37.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.617+0000 7f53559e6700 1 --2- 172.21.15.70:0/1791323984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f302d590 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f534c00d950 tx=0x7f534c091090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d6536812a2820de server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:37.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.617+0000 7f533b7fe700 1 -- 172.21.15.70:0/1791323984 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534c0c3030 con 0x55f3f302e070 2024-06-04T04:58:37.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.617+0000 7f533b7fe700 1 -- 172.21.15.70:0/1791323984 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f534c0049e0 con 0x55f3f302e070 2024-06-04T04:58:37.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.617+0000 7f533b7fe700 1 -- 172.21.15.70:0/1791323984 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534c0bebd0 con 0x55f3f302e070 2024-06-04T04:58:37.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.618+0000 7f53624dfb80 1 -- 172.21.15.70:0/1791323984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 msgr2=0x55f3f302d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:37.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.618+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1791323984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f302d590 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f534c00d950 tx=0x7f534c091090 comp rx=0 tx=0).stop 2024-06-04T04:58:37.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.618+0000 7f53624dfb80 1 -- 172.21.15.70:0/1791323984 shutdown_connections 2024-06-04T04:58:37.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.618+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1791323984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f3032770 0x55f3f304f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.618+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1791323984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f302d590 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.618+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1791323984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f302ea20 0x55f3f3039770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.618+0000 7f53624dfb80 1 -- 172.21.15.70:0/1791323984 >> 172.21.15.70:0/1791323984 conn(0x55f3f2f53080 msgr2=0x55f3f302ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:37.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.618+0000 7f53624dfb80 1 -- 172.21.15.70:0/1791323984 shutdown_connections 2024-06-04T04:58:37.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.618+0000 7f53624dfb80 1 -- 172.21.15.70:0/1791323984 wait complete. 2024-06-04T04:58:37.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.619+0000 7f53624dfb80 1 -- start start 2024-06-04T04:58:37.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.619+0000 7f53624dfb80 1 Processor -- start 2024-06-04T04:58:37.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.619+0000 7f53624dfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f303fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.619+0000 7f53559e6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f303fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:37.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.619+0000 7f53559e6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f303fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36962/0 (socket says 172.21.15.70:36962) 2024-06-04T04:58:37.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.620+0000 7f53559e6700 1 -- 172.21.15.70:0/1234373403 learned_addr learned my addr 172.21.15.70:0/1234373403 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.620+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1234373403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f302ea20 0x55f3f30403a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.620+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1234373403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f3032770 0x55f3f303d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.620+0000 7f53624dfb80 1 -- 172.21.15.70:0/1234373403 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f53501ad0e0 con 0x55f3f302e070 2024-06-04T04:58:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.620+0000 7f53624dfb80 1 -- 172.21.15.70:0/1234373403 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53501aca10 con 0x55f3f302ea20 2024-06-04T04:58:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.620+0000 7f53624dfb80 1 -- 172.21.15.70:0/1234373403 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5350108870 con 0x55f3f3032770 2024-06-04T04:58:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.620+0000 7f53561e7700 1 --2- 172.21.15.70:0/1234373403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f3032770 0x55f3f303d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.620+0000 7f53551e5700 1 --2- 172.21.15.70:0/1234373403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f302ea20 0x55f3f30403a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.620+0000 7f53551e5700 1 -- 172.21.15.70:0/1234373403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f3032770 msgr2=0x55f3f303d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.620+0000 7f53551e5700 1 --2- 172.21.15.70:0/1234373403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f3032770 0x55f3f303d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.620+0000 7f53551e5700 1 -- 172.21.15.70:0/1234373403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 msgr2=0x55f3f303fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.620+0000 7f53551e5700 1 --2- 172.21.15.70:0/1234373403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f303fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.620+0000 7f53551e5700 1 -- 172.21.15.70:0/1234373403 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3f303dc00 con 0x55f3f302ea20 2024-06-04T04:58:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.620+0000 7f53559e6700 1 --2- 172.21.15.70:0/1234373403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f303fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.620+0000 7f53624dfb80 10 client.0 _ll_register_callbacks cb 0x55f3f301fc50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.621+0000 7f53624dfb80 1 -- 172.21.15.70:0/1234373403 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f534c0049e0 con 0x55f3f302ea20 2024-06-04T04:58:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.621+0000 7f53551e5700 1 --2- 172.21.15.70:0/1234373403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f302ea20 0x55f3f30403a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5340005040 tx=0x7f5340009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bfe51a79b4eac0f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.621+0000 7f53397fa700 10 client.25298 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T04:58:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.621+0000 7f53397fa700 1 -- 172.21.15.70:0/1234373403 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534003e020 con 0x55f3f302ea20 2024-06-04T04:58:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.621+0000 7f53397fa700 1 -- 172.21.15.70:0/1234373403 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f534002a040 con 0x55f3f302ea20 2024-06-04T04:58:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.621+0000 7f53397fa700 1 -- 172.21.15.70:0/1234373403 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f534000b9e0 con 0x55f3f302ea20 2024-06-04T04:58:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.621+0000 7f53624dfb80 10 client.25298 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.621+0000 7f53624dfb80 1 -- 172.21.15.70:0/1234373403 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55f3f30b1bf0 con 0x55f3f302ea20 2024-06-04T04:58:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.622+0000 7f53397fa700 1 -- 172.21.15.70:0/1234373403 <== mon.2 v2:172.21.15.98:3300/0 4 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f5340041020 con 0x55f3f302ea20 2024-06-04T04:58:37.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.622+0000 7f53624dfb80 10 client.25298 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:37.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.622+0000 7f53624dfb80 10 client.25298 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T04:58:37.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.622+0000 7f53624dfb80 20 client.25298 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:37.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.622+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1234373403 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55f3f30b1e20 0x55f3f30b4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:37.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.622+0000 7f53624dfb80 4 client.25298 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T04:58:37.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.622+0000 7f53624dfb80 1 -- 172.21.15.70:0/1234373403 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f3f2f86be0 con 0x55f3f30b1e20 2024-06-04T04:58:37.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.622+0000 7f53559e6700 1 --2- 172.21.15.70:0/1234373403 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55f3f30b1e20 0x55f3f30b4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:37.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.622+0000 7f53559e6700 1 --2- 172.21.15.70:0/1234373403 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55f3f30b1e20 0x55f3f30b4210 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=80da42b1ef3a6b85 server_cookie=5927a44a0e88bd70 in_seq=0 out_seq=0 2024-06-04T04:58:37.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.622+0000 7f53397fa700 10 client.25298 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T04:58:37.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.622+0000 7f53397fa700 1 -- 172.21.15.70:0/1234373403 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f3f2f86be0 con 0x55f3f30b1e20 2024-06-04T04:58:37.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.622+0000 7f53397fa700 10 client.25298 handle_command_reply: tid=0 2024-06-04T04:58:37.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.623+0000 7f53624dfb80 1 client.25298 shutdown 2024-06-04T04:58:37.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.623+0000 7f53624dfb80 1 -- 172.21.15.70:0/1234373403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f302ea20 msgr2=0x55f3f30403a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:37.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.623+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1234373403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f302ea20 0x55f3f30403a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5340005040 tx=0x7f5340009310 comp rx=0 tx=0).stop 2024-06-04T04:58:37.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.623+0000 7f53624dfb80 1 -- 172.21.15.70:0/1234373403 shutdown_connections 2024-06-04T04:58:37.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.623+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1234373403 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55f3f30b1e20 0x55f3f30b4210 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.623+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1234373403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f3f3032770 0x55f3f303d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.623+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1234373403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f3f302e070 0x55f3f303fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.623+0000 7f53624dfb80 1 --2- 172.21.15.70:0/1234373403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3f302ea20 0x55f3f30403a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.623+0000 7f53624dfb80 1 -- 172.21.15.70:0/1234373403 >> 172.21.15.70:0/1234373403 conn(0x55f3f2f53080 msgr2=0x55f3f3031530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:37.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.623+0000 7f53397fa700 0 client.25298 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T04:58:37.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.623+0000 7f53624dfb80 1 -- 172.21.15.70:0/1234373403 shutdown_connections 2024-06-04T04:58:37.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.623+0000 7f53624dfb80 1 -- 172.21.15.70:0/1234373403 wait complete. 2024-06-04T04:58:37.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.623+0000 7f53624dfb80 20 client.25298 trim_cache size 0 max 16384 2024-06-04T04:58:37.630 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T04:58:37.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.624+0000 7f5357c4a700 1 -- 172.21.15.70:0/4198147885 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f533c0b2b10 msgr2=0x7f533c0b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:37.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.624+0000 7f5357c4a700 1 --2- 172.21.15.70:0/4198147885 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f533c0b2b10 0x7f533c0b4fd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5340003760 tx=0x7f5340005fd0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.624+0000 7f5357c4a700 1 -- 172.21.15.70:0/4198147885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f535006b250 msgr2=0x7f535019bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:37.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.624+0000 7f5357c4a700 1 --2- 172.21.15.70:0/4198147885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f535006b250 0x7f535019bd80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f534c00d980 tx=0x7f534c00dc90 comp rx=0 tx=0).stop 2024-06-04T04:58:37.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.624+0000 7f5357c4a700 1 -- 172.21.15.70:0/4198147885 shutdown_connections 2024-06-04T04:58:37.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.624+0000 7f5357c4a700 1 --2- 172.21.15.70:0/4198147885 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f533c0b2b10 0x7f533c0b4fd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.624+0000 7f5357c4a700 1 --2- 172.21.15.70:0/4198147885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f535010fff0 0x7f53501a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.624+0000 7f5357c4a700 1 --2- 172.21.15.70:0/4198147885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f535006b250 0x7f535019bd80 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.624+0000 7f5357c4a700 1 --2- 172.21.15.70:0/4198147885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f535006bb70 0x7f535019c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:37.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.624+0000 7f5357c4a700 1 -- 172.21.15.70:0/4198147885 >> 172.21.15.70:0/4198147885 conn(0x7f53500793f0 msgr2=0x7f53501115f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:37.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.624+0000 7f5357c4a700 1 -- 172.21.15.70:0/4198147885 shutdown_connections 2024-06-04T04:58:37.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:37.624+0000 7f5357c4a700 1 -- 172.21.15.70:0/4198147885 wait complete. 2024-06-04T04:58:38.390 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T04:58:38.675 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:39.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:39 smithi070 ceph-mon[32631]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-06-04T04:58:39.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:39 smithi112 ceph-mon[36453]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-06-04T04:58:39.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:39 smithi098 ceph-mon[35569]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-06-04T04:58:40.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.571+0000 7f9a3d889700 1 -- 172.21.15.70:0/1725228837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a38102430 msgr2=0x7f9a381028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:40.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.571+0000 7f9a3d889700 1 --2- 172.21.15.70:0/1725228837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a38102430 0x7f9a381028b0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c009990 tx=0x7f9a2c009ca0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.571+0000 7f9a3d889700 1 -- 172.21.15.70:0/1725228837 shutdown_connections 2024-06-04T04:58:40.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.571+0000 7f9a3d889700 1 --2- 172.21.15.70:0/1725228837 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a38102df0 0x7f9a3810f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.571+0000 7f9a3d889700 1 --2- 172.21.15.70:0/1725228837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a38108510 0x7f9a381088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.571+0000 7f9a3d889700 1 --2- 172.21.15.70:0/1725228837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a38102430 0x7f9a381028b0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.571+0000 7f9a3d889700 1 -- 172.21.15.70:0/1725228837 >> 172.21.15.70:0/1725228837 conn(0x7f9a380793b0 msgr2=0x7f9a380797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:40.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.572+0000 7f9a3d889700 1 -- 172.21.15.70:0/1725228837 shutdown_connections 2024-06-04T04:58:40.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.573+0000 7f9a3d889700 1 -- 172.21.15.70:0/1725228837 wait complete. 2024-06-04T04:58:40.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.574+0000 7f9a3d889700 1 Processor -- start 2024-06-04T04:58:40.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.574+0000 7f9a3d889700 1 -- start start 2024-06-04T04:58:40.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.575+0000 7f9a3d889700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a38102430 0x7f9a3819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.575+0000 7f9a3d889700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a38102df0 0x7f9a3819fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.575+0000 7f9a3d889700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a38108510 0x7f9a38199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.575+0000 7f9a367fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a38102df0 0x7f9a3819fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:40.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.575+0000 7f9a3d889700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9a38113640 con 0x7f9a38102df0 2024-06-04T04:58:40.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.575+0000 7f9a3d889700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381134c0 con 0x7f9a38108510 2024-06-04T04:58:40.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.575+0000 7f9a3d889700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381137c0 con 0x7f9a38102430 2024-06-04T04:58:40.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.575+0000 7f9a367fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a38102df0 0x7f9a3819fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36982/0 (socket says 172.21.15.70:36982) 2024-06-04T04:58:40.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.575+0000 7f9a367fc700 1 -- 172.21.15.70:0/3715870031 learned_addr learned my addr 172.21.15.70:0/3715870031 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:40.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.575+0000 7f9a36ffd700 1 --2- 172.21.15.70:0/3715870031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a38102430 0x7f9a3819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:40.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.576+0000 7f9a377fe700 1 --2- 172.21.15.70:0/3715870031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a38108510 0x7f9a38199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:40.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.576+0000 7f9a367fc700 1 -- 172.21.15.70:0/3715870031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a38102430 msgr2=0x7f9a3819f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:40.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.576+0000 7f9a367fc700 1 --2- 172.21.15.70:0/3715870031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a38102430 0x7f9a3819f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.576+0000 7f9a367fc700 1 -- 172.21.15.70:0/3715870031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a38108510 msgr2=0x7f9a38199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:40.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.576+0000 7f9a367fc700 1 --2- 172.21.15.70:0/3715870031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a38108510 0x7f9a38199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.576+0000 7f9a367fc700 1 -- 172.21.15.70:0/3715870031 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a3819a020 con 0x7f9a38102df0 2024-06-04T04:58:40.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.576+0000 7f9a377fe700 1 --2- 172.21.15.70:0/3715870031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a38108510 0x7f9a38199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:40.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.576+0000 7f9a367fc700 1 --2- 172.21.15.70:0/3715870031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a38102df0 0x7f9a3819fc00 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c005910 tx=0x7f9a2c00f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:40.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.577+0000 7f9a3c887700 1 -- 172.21.15.70:0/3715870031 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c00be40 con 0x7f9a38102df0 2024-06-04T04:58:40.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.577+0000 7f9a3c887700 1 -- 172.21.15.70:0/3715870031 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9a2c004cc0 con 0x7f9a38102df0 2024-06-04T04:58:40.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.577+0000 7f9a3d889700 1 -- 172.21.15.70:0/3715870031 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a3819a2b0 con 0x7f9a38102df0 2024-06-04T04:58:40.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.577+0000 7f9a3c887700 1 -- 172.21.15.70:0/3715870031 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c021410 con 0x7f9a38102df0 2024-06-04T04:58:40.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.577+0000 7f9a3d889700 1 -- 172.21.15.70:0/3715870031 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a3810fb20 con 0x7f9a38102df0 2024-06-04T04:58:40.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.581+0000 7f9a3c887700 1 -- 172.21.15.70:0/3715870031 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 149885+0+0 (secure 0 0 0) 0x7f9a2c009d10 con 0x7f9a38102df0 2024-06-04T04:58:40.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.582+0000 7f9a4811eb80 1 Processor -- start 2024-06-04T04:58:40.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.582+0000 7f9a3c887700 1 --2- 172.21.15.70:0/3715870031 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9a200b2c10 0x7f9a200b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.582+0000 7f9a4811eb80 1 -- start start 2024-06-04T04:58:40.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.582+0000 7f9a3c887700 1 -- 172.21.15.70:0/3715870031 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f9a2c0e3920 con 0x7f9a38102df0 2024-06-04T04:58:40.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.583+0000 7f9a377fe700 1 --2- 172.21.15.70:0/3715870031 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9a200b2c10 0x7f9a200b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:40.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.583+0000 7f9a4811eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668f070 0x55856668f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.583+0000 7f9a4811eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55856668fa20 0x558566693150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.583+0000 7f9a36ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668f070 0x55856668f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:40.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.583+0000 7f9a36ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668f070 0x55856668f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:47974/0 (socket says 172.21.15.70:47974) 2024-06-04T04:58:40.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.583+0000 7f9a4811eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558566693770 0x558566693bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.583+0000 7f9a377fe700 1 --2- 172.21.15.70:0/3715870031 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9a200b2c10 0x7f9a200b50d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9a300093a0 tx=0x7f9a3000b040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:40.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.583+0000 7f9a4811eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5585665e5c90 con 0x558566693770 2024-06-04T04:58:40.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.583+0000 7f9a4811eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5585665d5830 con 0x55856668fa20 2024-06-04T04:58:40.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.583+0000 7f9a4811eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5585665d9870 con 0x55856668f070 2024-06-04T04:58:40.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.584+0000 7f9a367fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55856668fa20 0x558566693150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:40.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.584+0000 7f9a367fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55856668fa20 0x558566693150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46604/0 (socket says 172.21.15.70:46604) 2024-06-04T04:58:40.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.584+0000 7f9a367fc700 1 -- 172.21.15.70:0/3250133306 learned_addr learned my addr 172.21.15.70:0/3250133306 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:40.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.584+0000 7f9a377fe700 1 --2- 172.21.15.70:0/3250133306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558566693770 0x558566693bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:40.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.584+0000 7f9a36ffd700 1 -- 172.21.15.70:0/3250133306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55856668fa20 msgr2=0x558566693150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:40.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.584+0000 7f9a36ffd700 1 --2- 172.21.15.70:0/3250133306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55856668fa20 0x558566693150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.584+0000 7f9a36ffd700 1 -- 172.21.15.70:0/3250133306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558566693770 msgr2=0x558566693bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:40.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.584+0000 7f9a36ffd700 1 --2- 172.21.15.70:0/3250133306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558566693770 0x558566693bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.584+0000 7f9a36ffd700 1 -- 172.21.15.70:0/3250133306 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585665258d0 con 0x55856668f070 2024-06-04T04:58:40.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.584+0000 7f9a36ffd700 1 --2- 172.21.15.70:0/3250133306 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668f070 0x55856668f450 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9a2800ddb0 tx=0x7f9a2800d5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efc1bcc5f19dab4d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:40.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.584+0000 7f9a1dffb700 1 -- 172.21.15.70:0/3250133306 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2801b070 con 0x55856668f070 2024-06-04T04:58:40.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.585+0000 7f9a1dffb700 1 -- 172.21.15.70:0/3250133306 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9a2800bd40 con 0x55856668f070 2024-06-04T04:58:40.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.585+0000 7f9a1dffb700 1 -- 172.21.15.70:0/3250133306 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a28016b90 con 0x55856668f070 2024-06-04T04:58:40.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.585+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3250133306 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668f070 msgr2=0x55856668f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.585+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/3250133306 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668f070 0x55856668f450 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9a2800ddb0 tx=0x7f9a2800d5a0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.586+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3250133306 shutdown_connections 2024-06-04T04:58:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.586+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/3250133306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558566693770 0x558566693bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.586+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/3250133306 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668f070 0x55856668f450 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.586+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/3250133306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55856668fa20 0x558566693150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.586+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3250133306 >> 172.21.15.70:0/3250133306 conn(0x5585665b4080 msgr2=0x558566680e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.586+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3250133306 shutdown_connections 2024-06-04T04:58:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.586+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3250133306 wait complete. 2024-06-04T04:58:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.586+0000 7f9a4811eb80 1 -- start start 2024-06-04T04:58:40.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.587+0000 7f9a4811eb80 1 Processor -- start 2024-06-04T04:58:40.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.587+0000 7f9a4811eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668f070 0x55856669ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.587+0000 7f9a4811eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668fa20 0x558566697e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.587+0000 7f9a4811eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558566693770 0x5585666983d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.587+0000 7f9a36ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668f070 0x55856669ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:40.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.587+0000 7f9a4811eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5585664dbfa0 con 0x55856668f070 2024-06-04T04:58:40.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.588+0000 7f9a4811eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5585665777c0 con 0x558566693770 2024-06-04T04:58:40.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.588+0000 7f9a4811eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55856669b930 con 0x55856668fa20 2024-06-04T04:58:40.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.588+0000 7f9a36ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668f070 0x55856669ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37000/0 (socket says 172.21.15.70:37000) 2024-06-04T04:58:40.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.588+0000 7f9a36ffd700 1 -- 172.21.15.70:0/3270314129 learned_addr learned my addr 172.21.15.70:0/3270314129 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:40.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.588+0000 7f9a4811eb80 10 client.0 _ll_register_callbacks cb 0x558566570920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:40.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.588+0000 7f9a367fc700 1 --2- 172.21.15.70:0/3270314129 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668fa20 0x558566697e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:40.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.588+0000 7f9a377fe700 1 --2- 172.21.15.70:0/3270314129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558566693770 0x5585666983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:40.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.588+0000 7f9a36ffd700 1 -- 172.21.15.70:0/3270314129 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668fa20 msgr2=0x558566697e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:40.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.588+0000 7f9a36ffd700 1 --2- 172.21.15.70:0/3270314129 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668fa20 0x558566697e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.588+0000 7f9a36ffd700 1 -- 172.21.15.70:0/3270314129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558566693770 msgr2=0x5585666983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:40.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.588+0000 7f9a36ffd700 1 --2- 172.21.15.70:0/3270314129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558566693770 0x5585666983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.588+0000 7f9a36ffd700 1 -- 172.21.15.70:0/3270314129 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558566574690 con 0x55856668f070 2024-06-04T04:58:40.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.588+0000 7f9a36ffd700 1 --2- 172.21.15.70:0/3270314129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668f070 0x55856669ab90 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9a28006c20 tx=0x7f9a2800d3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed650222cf4e4cc3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:40.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.588+0000 7f9a16ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:40.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.589+0000 7f9a16ffd700 1 -- 172.21.15.70:0/3270314129 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2801b040 con 0x55856668f070 2024-06-04T04:58:40.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.589+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3270314129 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a2800bd40 con 0x55856668f070 2024-06-04T04:58:40.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.589+0000 7f9a16ffd700 1 -- 172.21.15.70:0/3270314129 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f9a28016c40 con 0x55856668f070 2024-06-04T04:58:40.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.589+0000 7f9a16ffd700 1 -- 172.21.15.70:0/3270314129 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9a2802f050 con 0x55856668f070 2024-06-04T04:58:40.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.589+0000 7f9a4811eb80 10 client.15855 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:40.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.589+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3270314129 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5585666a5e10 con 0x55856668f070 2024-06-04T04:58:40.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.589+0000 7f9a16ffd700 1 -- 172.21.15.70:0/3270314129 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9a2800bd40 con 0x55856668f070 2024-06-04T04:58:40.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.590+0000 7f9a16ffd700 1 -- 172.21.15.70:0/3270314129 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f9a28020bf0 con 0x55856668f070 2024-06-04T04:58:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.590+0000 7f9a4811eb80 10 client.15855 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.590+0000 7f9a4811eb80 10 client.15855 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T04:58:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.590+0000 7f9a4811eb80 20 client.15855 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.590+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/3270314129 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5585666a8690 0x5585666aaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.590+0000 7f9a4811eb80 4 client.15855 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.590+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3270314129 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585666529e0 con 0x5585666a8690 2024-06-04T04:58:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.591+0000 7f9a367fc700 1 --2- 172.21.15.70:0/3270314129 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5585666a8690 0x5585666aaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.591+0000 7f9a16ffd700 10 client.15855 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T04:58:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.591+0000 7f9a367fc700 1 --2- 172.21.15.70:0/3270314129 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5585666a8690 0x5585666aaaf0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4f260f4e5ad6effa server_cookie=65b2ce63e54e43d2 in_seq=0 out_seq=0 2024-06-04T04:58:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.592+0000 7f9a16ffd700 1 -- 172.21.15.70:0/3270314129 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5585666529e0 con 0x5585666a8690 2024-06-04T04:58:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.592+0000 7f9a16ffd700 10 client.15855 handle_command_reply: tid=0 2024-06-04T04:58:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.592+0000 7f9a4811eb80 1 client.15855 shutdown 2024-06-04T04:58:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.592+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3270314129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668f070 msgr2=0x55856669ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:40.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.592+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/3270314129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668f070 0x55856669ab90 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9a28006c20 tx=0x7f9a2800d3f0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.592+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3270314129 shutdown_connections 2024-06-04T04:58:40.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.592+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/3270314129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558566693770 0x5585666983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.592+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/3270314129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668f070 0x55856669ab90 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.592+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/3270314129 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5585666a8690 0x5585666aaaf0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.592+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/3270314129 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668fa20 0x558566697e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.592+0000 7f9a16ffd700 0 client.15855 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T04:58:40.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.593+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3270314129 >> 172.21.15.70:0/3270314129 conn(0x5585665b4080 msgr2=0x558566680510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:40.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.594+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3270314129 shutdown_connections 2024-06-04T04:58:40.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.594+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3270314129 wait complete. 2024-06-04T04:58:40.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.594+0000 7f9a4811eb80 20 client.15855 trim_cache size 0 max 16384 2024-06-04T04:58:40.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.609+0000 7f9a4811eb80 1 Processor -- start 2024-06-04T04:58:40.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.609+0000 7f9a4811eb80 1 -- start start 2024-06-04T04:58:40.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.610+0000 7f9a4811eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668f070 0x558566699dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.610+0000 7f9a4811eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668fa20 0x558566680c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.610+0000 7f9a4811eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558566693770 0x5585666811d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.610+0000 7f9a4811eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381ab910 con 0x55856668fa20 2024-06-04T04:58:40.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.610+0000 7f9a4811eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381ad450 con 0x558566693770 2024-06-04T04:58:40.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.610+0000 7f9a4811eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381abfe0 con 0x55856668f070 2024-06-04T04:58:40.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.610+0000 7f9a367fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668fa20 0x558566680c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:40.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.610+0000 7f9a377fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558566693770 0x5585666811d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:40.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.610+0000 7f9a367fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668fa20 0x558566680c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37002/0 (socket says 172.21.15.70:37002) 2024-06-04T04:58:40.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.610+0000 7f9a367fc700 1 -- 172.21.15.70:0/3984008433 learned_addr learned my addr 172.21.15.70:0/3984008433 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:40.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.610+0000 7f9a36ffd700 1 --2- 172.21.15.70:0/3984008433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668f070 0x558566699dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:40.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.611+0000 7f9a367fc700 1 -- 172.21.15.70:0/3984008433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668f070 msgr2=0x558566699dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:40.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.611+0000 7f9a367fc700 1 --2- 172.21.15.70:0/3984008433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668f070 0x558566699dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.611+0000 7f9a367fc700 1 -- 172.21.15.70:0/3984008433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558566693770 msgr2=0x5585666811d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.611+0000 7f9a367fc700 1 --2- 172.21.15.70:0/3984008433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558566693770 0x5585666811d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.611+0000 7f9a367fc700 1 -- 172.21.15.70:0/3984008433 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585666934c0 con 0x55856668fa20 2024-06-04T04:58:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.611+0000 7f9a377fe700 1 --2- 172.21.15.70:0/3984008433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558566693770 0x5585666811d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.611+0000 7f9a36ffd700 1 --2- 172.21.15.70:0/3984008433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668f070 0x558566699dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.611+0000 7f9a367fc700 1 --2- 172.21.15.70:0/3984008433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668fa20 0x558566680c90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c00b2e0 tx=0x7f9a2c092040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9fe144947d645b8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.611+0000 7f9a1d7fa700 1 -- 172.21.15.70:0/3984008433 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c0aa150 con 0x55856668fa20 2024-06-04T04:58:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.611+0000 7f9a1d7fa700 1 -- 172.21.15.70:0/3984008433 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9a2c0b30f0 con 0x55856668fa20 2024-06-04T04:58:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.611+0000 7f9a1d7fa700 1 -- 172.21.15.70:0/3984008433 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c0a6170 con 0x55856668fa20 2024-06-04T04:58:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.612+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3984008433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668fa20 msgr2=0x558566680c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.612+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/3984008433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668fa20 0x558566680c90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c00b2e0 tx=0x7f9a2c092040 comp rx=0 tx=0).stop 2024-06-04T04:58:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.612+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3984008433 shutdown_connections 2024-06-04T04:58:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.612+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/3984008433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558566693770 0x5585666811d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.612+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/3984008433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668f070 0x558566699dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.612+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/3984008433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668fa20 0x558566680c90 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.612+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3984008433 >> 172.21.15.70:0/3984008433 conn(0x5585665b4080 msgr2=0x5585666996c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:40.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.612+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3984008433 shutdown_connections 2024-06-04T04:58:40.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.612+0000 7f9a4811eb80 1 -- 172.21.15.70:0/3984008433 wait complete. 2024-06-04T04:58:40.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.613+0000 7f9a4811eb80 1 -- start start 2024-06-04T04:58:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.613+0000 7f9a4811eb80 1 Processor -- start 2024-06-04T04:58:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.613+0000 7f9a4811eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668f070 0x55856669dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.613+0000 7f9a4811eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668fa20 0x55856669e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.613+0000 7f9a4811eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558566693770 0x55856669e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.614+0000 7f9a4811eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381acd80 con 0x55856668f070 2024-06-04T04:58:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.614+0000 7f9a4811eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381ac6b0 con 0x558566693770 2024-06-04T04:58:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.614+0000 7f9a4811eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381a98d0 con 0x55856668fa20 2024-06-04T04:58:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.614+0000 7f9a4811eb80 10 client.0 _ll_register_callbacks cb 0x558566570920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.614+0000 7f9a36ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668f070 0x55856669dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.614+0000 7f9a36ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668f070 0x55856669dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37004/0 (socket says 172.21.15.70:37004) 2024-06-04T04:58:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.614+0000 7f9a36ffd700 1 -- 172.21.15.70:0/142206234 learned_addr learned my addr 172.21.15.70:0/142206234 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.614+0000 7f9a367fc700 1 --2- 172.21.15.70:0/142206234 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668fa20 0x55856669e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.614+0000 7f9a36ffd700 1 -- 172.21.15.70:0/142206234 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668fa20 msgr2=0x55856669e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:40.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.614+0000 7f9a36ffd700 1 --2- 172.21.15.70:0/142206234 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668fa20 0x55856669e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.614+0000 7f9a36ffd700 1 -- 172.21.15.70:0/142206234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558566693770 msgr2=0x55856669e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:40.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.614+0000 7f9a36ffd700 1 --2- 172.21.15.70:0/142206234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558566693770 0x55856669e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.614+0000 7f9a36ffd700 1 -- 172.21.15.70:0/142206234 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558566711400 con 0x55856668f070 2024-06-04T04:58:40.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.614+0000 7f9a36ffd700 1 --2- 172.21.15.70:0/142206234 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668f070 0x55856669dbc0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9a2800b6a0 tx=0x7f9a280046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358962f5c75b3753 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:40.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.614+0000 7f9a16ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:40.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.615+0000 7f9a16ffd700 1 -- 172.21.15.70:0/142206234 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a28038060 con 0x55856668f070 2024-06-04T04:58:40.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.615+0000 7f9a16ffd700 1 -- 172.21.15.70:0/142206234 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f9a28016420 con 0x55856668f070 2024-06-04T04:58:40.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.615+0000 7f9a4811eb80 1 -- 172.21.15.70:0/142206234 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5585665e7be0 con 0x55856668f070 2024-06-04T04:58:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.615+0000 7f9a16ffd700 1 -- 172.21.15.70:0/142206234 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9a2801b030 con 0x55856668f070 2024-06-04T04:58:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.615+0000 7f9a4811eb80 10 client.15861 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.615+0000 7f9a4811eb80 1 -- 172.21.15.70:0/142206234 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5585667126e0 con 0x55856668f070 2024-06-04T04:58:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.615+0000 7f9a16ffd700 1 -- 172.21.15.70:0/142206234 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x5585665e7be0 con 0x55856668f070 2024-06-04T04:58:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.615+0000 7f9a16ffd700 1 -- 172.21.15.70:0/142206234 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f9a2801fb30 con 0x55856668f070 2024-06-04T04:58:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.615+0000 7f9a4811eb80 10 client.15861 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.615+0000 7f9a4811eb80 10 client.15861 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T04:58:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.615+0000 7f9a4811eb80 20 client.15861 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.615+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/142206234 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5585666a8690 0x558566714dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.615+0000 7f9a4811eb80 4 client.15861 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T04:58:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.615+0000 7f9a4811eb80 1 -- 172.21.15.70:0/142206234 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5585665e13c0 con 0x5585666a8690 2024-06-04T04:58:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.616+0000 7f9a367fc700 1 --2- 172.21.15.70:0/142206234 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5585666a8690 0x558566714dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.616+0000 7f9a16ffd700 10 client.15861 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T04:58:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.616+0000 7f9a367fc700 1 --2- 172.21.15.70:0/142206234 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5585666a8690 0x558566714dd0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c85465c7187d9e80 server_cookie=60c37b20008774f in_seq=0 out_seq=0 2024-06-04T04:58:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.617+0000 7f9a16ffd700 1 -- 172.21.15.70:0/142206234 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5585665e13c0 con 0x5585666a8690 2024-06-04T04:58:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.617+0000 7f9a16ffd700 10 client.15861 handle_command_reply: tid=0 2024-06-04T04:58:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.617+0000 7f9a4811eb80 1 client.15861 shutdown 2024-06-04T04:58:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.617+0000 7f9a4811eb80 1 -- 172.21.15.70:0/142206234 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668f070 msgr2=0x55856669dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.617+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/142206234 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668f070 0x55856669dbc0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9a2800b6a0 tx=0x7f9a280046d0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.617+0000 7f9a4811eb80 1 -- 172.21.15.70:0/142206234 shutdown_connections 2024-06-04T04:58:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.617+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/142206234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558566693770 0x55856669e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.617+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/142206234 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55856668f070 0x55856669dbc0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.617+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/142206234 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5585666a8690 0x558566714dd0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.617+0000 7f9a4811eb80 1 --2- 172.21.15.70:0/142206234 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55856668fa20 0x55856669e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.617+0000 7f9a4811eb80 1 -- 172.21.15.70:0/142206234 >> 172.21.15.70:0/142206234 conn(0x5585665b4080 msgr2=0x558566685f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.617+0000 7f9a16ffd700 0 client.15861 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T04:58:40.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.617+0000 7f9a4811eb80 1 -- 172.21.15.70:0/142206234 shutdown_connections 2024-06-04T04:58:40.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.617+0000 7f9a4811eb80 1 -- 172.21.15.70:0/142206234 wait complete. 2024-06-04T04:58:40.624 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T04:58:40.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.617+0000 7f9a4811eb80 20 client.15861 trim_cache size 0 max 16384 2024-06-04T04:58:40.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.618+0000 7f9a3d889700 1 -- 172.21.15.70:0/3715870031 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9a200b2c10 msgr2=0x7f9a200b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:40.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.618+0000 7f9a3d889700 1 --2- 172.21.15.70:0/3715870031 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9a200b2c10 0x7f9a200b50d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9a300093a0 tx=0x7f9a3000b040 comp rx=0 tx=0).stop 2024-06-04T04:58:40.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.618+0000 7f9a3d889700 1 -- 172.21.15.70:0/3715870031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a38102df0 msgr2=0x7f9a3819fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:40.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.618+0000 7f9a3d889700 1 --2- 172.21.15.70:0/3715870031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a38102df0 0x7f9a3819fc00 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c005910 tx=0x7f9a2c00f670 comp rx=0 tx=0).stop 2024-06-04T04:58:40.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.618+0000 7f9a3d889700 1 -- 172.21.15.70:0/3715870031 shutdown_connections 2024-06-04T04:58:40.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.618+0000 7f9a3d889700 1 --2- 172.21.15.70:0/3715870031 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9a200b2c10 0x7f9a200b50d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.618+0000 7f9a3d889700 1 --2- 172.21.15.70:0/3715870031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a38108510 0x7f9a38199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.618+0000 7f9a3d889700 1 --2- 172.21.15.70:0/3715870031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a38102430 0x7f9a3819f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.618+0000 7f9a3d889700 1 --2- 172.21.15.70:0/3715870031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a38102df0 0x7f9a3819fc00 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:40.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.618+0000 7f9a3d889700 1 -- 172.21.15.70:0/3715870031 >> 172.21.15.70:0/3715870031 conn(0x7f9a380793b0 msgr2=0x7f9a38100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:40.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.618+0000 7f9a3d889700 1 -- 172.21.15.70:0/3715870031 shutdown_connections 2024-06-04T04:58:40.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:40.618+0000 7f9a3d889700 1 -- 172.21.15.70:0/3715870031 wait complete. 2024-06-04T04:58:41.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:41 smithi070 ceph-mon[32631]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-06-04T04:58:41.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:41 smithi112 ceph-mon[36453]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-06-04T04:58:41.482 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T04:58:41.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:41 smithi098 ceph-mon[35569]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-06-04T04:58:41.755 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.274+0000 7f875d52b640 1 -- 172.21.15.70:0/3538303852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f875810d020 msgr2=0x7f875810f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.274+0000 7f875d52b640 1 --2- 172.21.15.70:0/3538303852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f875810d020 0x7f875810f4c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f8750009f00 tx=0x7f875002ac40 comp rx=0 tx=0).stop 2024-06-04T04:58:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.275+0000 7f875d52b640 1 -- 172.21.15.70:0/3538303852 shutdown_connections 2024-06-04T04:58:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.275+0000 7f875d52b640 1 --2- 172.21.15.70:0/3538303852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8758104600 0x7f875810cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.275+0000 7f875d52b640 1 --2- 172.21.15.70:0/3538303852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f875810d020 0x7f875810f4c0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.275+0000 7f875d52b640 1 --2- 172.21.15.70:0/3538303852 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8758103ce0 0x7f87581040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.275+0000 7f875d52b640 1 -- 172.21.15.70:0/3538303852 >> 172.21.15.70:0/3538303852 conn(0x7f8758078ad0 msgr2=0x7f8758078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:42.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.275+0000 7f875d52b640 1 -- 172.21.15.70:0/3538303852 shutdown_connections 2024-06-04T04:58:42.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.275+0000 7f875d52b640 1 -- 172.21.15.70:0/3538303852 wait complete. 2024-06-04T04:58:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.275+0000 7f875d52b640 1 Processor -- start 2024-06-04T04:58:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.275+0000 7f875d52b640 1 -- start start 2024-06-04T04:58:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.276+0000 7f875d52b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8758103ce0 0x7f87581990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.276+0000 7f875d52b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8758104600 0x7f8758199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.276+0000 7f875d52b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f875810d020 0x7f875819d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.276+0000 7f875d52b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8758112b30 con 0x7f8758104600 2024-06-04T04:58:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.276+0000 7f875d52b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f87581129b0 con 0x7f8758103ce0 2024-06-04T04:58:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.276+0000 7f875d52b640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8758112cb0 con 0x7f875810d020 2024-06-04T04:58:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.276+0000 7f87567fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8758104600 0x7f8758199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.276+0000 7f87567fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8758104600 0x7f8758199610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37020/0 (socket says 172.21.15.70:37020) 2024-06-04T04:58:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.276+0000 7f87567fc640 1 -- 172.21.15.70:0/2200530820 learned_addr learned my addr 172.21.15.70:0/2200530820 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.276+0000 7f8756ffd640 1 --2- 172.21.15.70:0/2200530820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8758103ce0 0x7f87581990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.276+0000 7f87567fc640 1 -- 172.21.15.70:0/2200530820 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f875810d020 msgr2=0x7f875819d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.276+0000 7f87567fc640 1 --2- 172.21.15.70:0/2200530820 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f875810d020 0x7f875819d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.276+0000 7f87567fc640 1 -- 172.21.15.70:0/2200530820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8758103ce0 msgr2=0x7f87581990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.276+0000 7f87567fc640 1 --2- 172.21.15.70:0/2200530820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8758103ce0 0x7f87581990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.276+0000 7f87567fc640 1 -- 172.21.15.70:0/2200530820 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f875819e0c0 con 0x7f8758104600 2024-06-04T04:58:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.276+0000 7f87567fc640 1 --2- 172.21.15.70:0/2200530820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8758104600 0x7f8758199610 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f874c00d950 tx=0x7f874c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.277+0000 7f873ffff640 1 -- 172.21.15.70:0/2200530820 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c017070 con 0x7f8758104600 2024-06-04T04:58:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.277+0000 7f873ffff640 1 -- 172.21.15.70:0/2200530820 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f874c009c30 con 0x7f8758104600 2024-06-04T04:58:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.277+0000 7f875d52b640 1 -- 172.21.15.70:0/2200530820 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f875819e3b0 con 0x7f8758104600 2024-06-04T04:58:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.277+0000 7f873ffff640 1 -- 172.21.15.70:0/2200530820 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c00f620 con 0x7f8758104600 2024-06-04T04:58:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.277+0000 7f875d52b640 1 -- 172.21.15.70:0/2200530820 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87581a5c40 con 0x7f8758104600 2024-06-04T04:58:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f876024a740 1 Processor -- start 2024-06-04T04:58:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f873ffff640 1 -- 172.21.15.70:0/2200530820 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f874c010460 con 0x7f8758104600 2024-06-04T04:58:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f876024a740 1 -- start start 2024-06-04T04:58:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f876024a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baecd60 0x55f56baeb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f876024a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baebc20 0x55f56baec0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f876024a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 0x55f56bb01490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f876024a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f56ba5dc30 con 0x55f56baecd60 2024-06-04T04:58:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f876024a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f56ba4d7d0 con 0x55f56baff0a0 2024-06-04T04:58:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f876024a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f56ba51810 con 0x55f56baebc20 2024-06-04T04:58:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f87577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baecd60 0x55f56baeb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f87577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baecd60 0x55f56baeb6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37022/0 (socket says 172.21.15.70:37022) 2024-06-04T04:58:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f87577fe640 1 -- 172.21.15.70:0/693864603 learned_addr learned my addr 172.21.15.70:0/693864603 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f87567fc640 1 --2- 172.21.15.70:0/693864603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 0x55f56bb01490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f8756ffd640 1 --2- 172.21.15.70:0/693864603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baebc20 0x55f56baec0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f87577fe640 1 -- 172.21.15.70:0/693864603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baebc20 msgr2=0x55f56baec0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f87577fe640 1 --2- 172.21.15.70:0/693864603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baebc20 0x55f56baec0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f87577fe640 1 -- 172.21.15.70:0/693864603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 msgr2=0x55f56bb01490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f87577fe640 1 --2- 172.21.15.70:0/693864603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 0x55f56bb01490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.281+0000 7f87577fe640 1 -- 172.21.15.70:0/693864603 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f56b762210 con 0x55f56baecd60 2024-06-04T04:58:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.282+0000 7f87567fc640 1 --2- 172.21.15.70:0/693864603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 0x55f56bb01490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.282+0000 7f87577fe640 1 --2- 172.21.15.70:0/693864603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baecd60 0x55f56baeb6e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f8750005e30 tx=0x7f8750005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e58fdd4e79d8e1cd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.282+0000 7f873ffff640 1 --2- 172.21.15.70:0/2200530820 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f87400b2a90 0x7f87400b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.282+0000 7f873d7fa640 1 -- 172.21.15.70:0/693864603 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8750038070 con 0x55f56baecd60 2024-06-04T04:58:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.282+0000 7f873d7fa640 1 -- 172.21.15.70:0/693864603 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8750039040 con 0x55f56baecd60 2024-06-04T04:58:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.282+0000 7f873d7fa640 1 -- 172.21.15.70:0/693864603 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875002b9e0 con 0x55f56baecd60 2024-06-04T04:58:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.282+0000 7f873ffff640 1 -- 172.21.15.70:0/2200530820 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f874c0e21c0 con 0x7f8758104600 2024-06-04T04:58:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.282+0000 7f8756ffd640 1 --2- 172.21.15.70:0/2200530820 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f87400b2a90 0x7f87400b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.283+0000 7f876024a740 1 -- 172.21.15.70:0/693864603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baecd60 msgr2=0x55f56baeb6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.283+0000 7f8756ffd640 1 --2- 172.21.15.70:0/2200530820 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f87400b2a90 0x7f87400b4f50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x55f56baee410 tx=0x7f8748005f90 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.283+0000 7f876024a740 1 --2- 172.21.15.70:0/693864603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baecd60 0x55f56baeb6e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f8750005e30 tx=0x7f8750005fa0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.283+0000 7f876024a740 1 -- 172.21.15.70:0/693864603 shutdown_connections 2024-06-04T04:58:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.283+0000 7f876024a740 1 --2- 172.21.15.70:0/693864603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 0x55f56bb01490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.283+0000 7f876024a740 1 --2- 172.21.15.70:0/693864603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baebc20 0x55f56baec0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.283+0000 7f876024a740 1 --2- 172.21.15.70:0/693864603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baecd60 0x55f56baeb6e0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.283+0000 7f876024a740 1 -- 172.21.15.70:0/693864603 >> 172.21.15.70:0/693864603 conn(0x55f56bae7d30 msgr2=0x55f56bae8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.283+0000 7f876024a740 1 -- 172.21.15.70:0/693864603 shutdown_connections 2024-06-04T04:58:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.283+0000 7f876024a740 1 -- 172.21.15.70:0/693864603 wait complete. 2024-06-04T04:58:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.284+0000 7f876024a740 1 -- start start 2024-06-04T04:58:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.284+0000 7f876024a740 1 Processor -- start 2024-06-04T04:58:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.284+0000 7f876024a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baebc20 0x55f56baef540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.284+0000 7f876024a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baecd60 0x55f56baefa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.284+0000 7f876024a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 0x55f56baeffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.284+0000 7f876024a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f56b733790 con 0x55f56baebc20 2024-06-04T04:58:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.284+0000 7f876024a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f56ba09890 con 0x55f56baff0a0 2024-06-04T04:58:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.284+0000 7f876024a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f56bb01da0 con 0x55f56baecd60 2024-06-04T04:58:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.284+0000 7f8756ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baebc20 0x55f56baef540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.284+0000 7f8756ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baebc20 0x55f56baef540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37032/0 (socket says 172.21.15.70:37032) 2024-06-04T04:58:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.284+0000 7f8756ffd640 1 -- 172.21.15.70:0/2333616005 learned_addr learned my addr 172.21.15.70:0/2333616005 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.284+0000 7f876024a740 10 client.0 _ll_register_callbacks cb 0x55f56b761a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.285+0000 7f87567fc640 1 --2- 172.21.15.70:0/2333616005 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baecd60 0x55f56baefa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.285+0000 7f87577fe640 1 --2- 172.21.15.70:0/2333616005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 0x55f56baeffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.285+0000 7f8756ffd640 1 -- 172.21.15.70:0/2333616005 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baecd60 msgr2=0x55f56baefa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.285+0000 7f8756ffd640 1 --2- 172.21.15.70:0/2333616005 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baecd60 0x55f56baefa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.285+0000 7f8756ffd640 1 -- 172.21.15.70:0/2333616005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 msgr2=0x55f56baeffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.285+0000 7f8756ffd640 1 --2- 172.21.15.70:0/2333616005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 0x55f56baeffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.285+0000 7f8756ffd640 1 -- 172.21.15.70:0/2333616005 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f56b82e620 con 0x55f56baebc20 2024-06-04T04:58:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.285+0000 7f87567fc640 1 --2- 172.21.15.70:0/2333616005 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baecd60 0x55f56baefa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.285+0000 7f87577fe640 1 --2- 172.21.15.70:0/2333616005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 0x55f56baeffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.285+0000 7f8756ffd640 1 --2- 172.21.15.70:0/2333616005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baebc20 0x55f56baef540 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8748009560 tx=0x7f8748004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b8b16d202e46962 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.285+0000 7f873aff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.285+0000 7f873aff5640 1 -- 172.21.15.70:0/2333616005 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874801e470 con 0x55f56baebc20 2024-06-04T04:58:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.285+0000 7f873aff5640 1 -- 172.21.15.70:0/2333616005 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f874801e610 con 0x55f56baebc20 2024-06-04T04:58:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.285+0000 7f876024a740 1 -- 172.21.15.70:0/2333616005 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8750039040 con 0x55f56baebc20 2024-06-04T04:58:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.285+0000 7f873aff5640 1 -- 172.21.15.70:0/2333616005 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f874801e810 con 0x55f56baebc20 2024-06-04T04:58:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.285+0000 7f876024a740 10 client.15873 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.285+0000 7f876024a740 1 -- 172.21.15.70:0/2333616005 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f56bb0dc60 con 0x55f56baebc20 2024-06-04T04:58:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.286+0000 7f873aff5640 1 -- 172.21.15.70:0/2333616005 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8750039040 con 0x55f56baebc20 2024-06-04T04:58:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.286+0000 7f873aff5640 1 -- 172.21.15.70:0/2333616005 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f8748027070 con 0x55f56baebc20 2024-06-04T04:58:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.286+0000 7f876024a740 10 client.15873 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.286+0000 7f876024a740 10 client.15873 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.286+0000 7f876024a740 20 client.15873 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.286+0000 7f876024a740 1 --2- 172.21.15.70:0/2333616005 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f56bb0f190 0x55f56bb11580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.286+0000 7f876024a740 4 client.15873 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.286+0000 7f876024a740 1 -- 172.21.15.70:0/2333616005 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f56ba5fb80 con 0x55f56bb0f190 2024-06-04T04:58:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.286+0000 7f87567fc640 1 --2- 172.21.15.70:0/2333616005 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f56bb0f190 0x55f56bb11580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.287+0000 7f87567fc640 1 --2- 172.21.15.70:0/2333616005 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f56bb0f190 0x55f56bb11580 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82ccf410b0254ffb server_cookie=a098bc0c94637309 in_seq=0 out_seq=0 2024-06-04T04:58:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.287+0000 7f873aff5640 10 client.15873 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.287+0000 7f873aff5640 1 -- 172.21.15.70:0/2333616005 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f56ba5fb80 con 0x55f56bb0f190 2024-06-04T04:58:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.287+0000 7f873aff5640 10 client.15873 handle_command_reply: tid=0 2024-06-04T04:58:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.287+0000 7f876024a740 1 client.15873 shutdown 2024-06-04T04:58:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.288+0000 7f876024a740 1 -- 172.21.15.70:0/2333616005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baebc20 msgr2=0x55f56baef540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.288+0000 7f876024a740 1 --2- 172.21.15.70:0/2333616005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baebc20 0x55f56baef540 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8748009560 tx=0x7f8748004730 comp rx=0 tx=0).stop 2024-06-04T04:58:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.288+0000 7f876024a740 1 -- 172.21.15.70:0/2333616005 shutdown_connections 2024-06-04T04:58:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.288+0000 7f876024a740 1 --2- 172.21.15.70:0/2333616005 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f56bb0f190 0x55f56bb11580 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.288+0000 7f876024a740 1 --2- 172.21.15.70:0/2333616005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 0x55f56baeffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.288+0000 7f876024a740 1 --2- 172.21.15.70:0/2333616005 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baecd60 0x55f56baefa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.288+0000 7f876024a740 1 --2- 172.21.15.70:0/2333616005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baebc20 0x55f56baef540 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.288+0000 7f876024a740 1 -- 172.21.15.70:0/2333616005 >> 172.21.15.70:0/2333616005 conn(0x55f56bae7d30 msgr2=0x55f56baedb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.288+0000 7f873aff5640 0 client.15873 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.288+0000 7f876024a740 1 -- 172.21.15.70:0/2333616005 shutdown_connections 2024-06-04T04:58:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.288+0000 7f876024a740 1 -- 172.21.15.70:0/2333616005 wait complete. 2024-06-04T04:58:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.288+0000 7f876024a740 20 client.15873 trim_cache size 0 max 16384 2024-06-04T04:58:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.303+0000 7f876024a740 1 Processor -- start 2024-06-04T04:58:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.303+0000 7f876024a740 1 -- start start 2024-06-04T04:58:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.303+0000 7f876024a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baecd60 0x55f56bae7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.303+0000 7f876024a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baff0a0 0x55f56bb16ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.303+0000 7f876024a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56bafbaf0 0x55f56bb17400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.303+0000 7f876024a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f87581ac4b0 con 0x55f56baecd60 2024-06-04T04:58:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.303+0000 7f876024a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f87581adff0 con 0x55f56bafbaf0 2024-06-04T04:58:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.303+0000 7f876024a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f87581acb80 con 0x55f56baff0a0 2024-06-04T04:58:42.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.303+0000 7f8756ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baecd60 0x55f56bae7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:42.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.303+0000 7f8756ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baecd60 0x55f56bae7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37042/0 (socket says 172.21.15.70:37042) 2024-06-04T04:58:42.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.303+0000 7f8756ffd640 1 -- 172.21.15.70:0/103667673 learned_addr learned my addr 172.21.15.70:0/103667673 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:42.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.303+0000 7f87567fc640 1 --2- 172.21.15.70:0/103667673 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baff0a0 0x55f56bb16ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:42.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.303+0000 7f87577fe640 1 --2- 172.21.15.70:0/103667673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56bafbaf0 0x55f56bb17400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:42.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.304+0000 7f87567fc640 1 -- 172.21.15.70:0/103667673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56bafbaf0 msgr2=0x55f56bb17400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:42.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.304+0000 7f87567fc640 1 --2- 172.21.15.70:0/103667673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56bafbaf0 0x55f56bb17400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.304+0000 7f87567fc640 1 -- 172.21.15.70:0/103667673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baecd60 msgr2=0x55f56bae7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:42.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.304+0000 7f87567fc640 1 --2- 172.21.15.70:0/103667673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baecd60 0x55f56bae7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.304+0000 7f87567fc640 1 -- 172.21.15.70:0/103667673 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f56bb17940 con 0x55f56baff0a0 2024-06-04T04:58:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.304+0000 7f8756ffd640 1 --2- 172.21.15.70:0/103667673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baecd60 0x55f56bae7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.304+0000 7f87567fc640 1 --2- 172.21.15.70:0/103667673 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baff0a0 0x55f56bb16ec0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f874c0df880 tx=0x7f874c005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=629ea0b5ae19025e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.304+0000 7f873cff9640 1 -- 172.21.15.70:0/103667673 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c020040 con 0x55f56baff0a0 2024-06-04T04:58:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.304+0000 7f873cff9640 1 -- 172.21.15.70:0/103667673 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f874c0a9b40 con 0x55f56baff0a0 2024-06-04T04:58:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.304+0000 7f873cff9640 1 -- 172.21.15.70:0/103667673 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c0dfb30 con 0x55f56baff0a0 2024-06-04T04:58:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.305+0000 7f876024a740 1 -- 172.21.15.70:0/103667673 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baff0a0 msgr2=0x55f56bb16ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.305+0000 7f876024a740 1 --2- 172.21.15.70:0/103667673 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baff0a0 0x55f56bb16ec0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f874c0df880 tx=0x7f874c005880 comp rx=0 tx=0).stop 2024-06-04T04:58:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.305+0000 7f876024a740 1 -- 172.21.15.70:0/103667673 shutdown_connections 2024-06-04T04:58:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.305+0000 7f876024a740 1 --2- 172.21.15.70:0/103667673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56bafbaf0 0x55f56bb17400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.305+0000 7f876024a740 1 --2- 172.21.15.70:0/103667673 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baff0a0 0x55f56bb16ec0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.305+0000 7f876024a740 1 --2- 172.21.15.70:0/103667673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56baecd60 0x55f56bae7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.305+0000 7f876024a740 1 -- 172.21.15.70:0/103667673 >> 172.21.15.70:0/103667673 conn(0x55f56bae7d30 msgr2=0x55f56baedb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.305+0000 7f876024a740 1 -- 172.21.15.70:0/103667673 shutdown_connections 2024-06-04T04:58:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.305+0000 7f876024a740 1 -- 172.21.15.70:0/103667673 wait complete. 2024-06-04T04:58:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.305+0000 7f876024a740 1 -- start start 2024-06-04T04:58:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.305+0000 7f876024a740 1 Processor -- start 2024-06-04T04:58:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.305+0000 7f876024a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baecd60 0x55f56bb07740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.305+0000 7f876024a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56bafbaf0 0x55f56bafce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.305+0000 7f876024a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 0x55f56bafd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.305+0000 7f876024a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f87581ad920 con 0x55f56bafbaf0 2024-06-04T04:58:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.305+0000 7f876024a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f87581ad250 con 0x55f56baff0a0 2024-06-04T04:58:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.305+0000 7f876024a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f87581aa490 con 0x55f56baecd60 2024-06-04T04:58:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.306+0000 7f876024a740 10 client.0 _ll_register_callbacks cb 0x7f8748009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.306+0000 7f87567fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56bafbaf0 0x55f56bafce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.306+0000 7f87567fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56bafbaf0 0x55f56bafce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37048/0 (socket says 172.21.15.70:37048) 2024-06-04T04:58:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.306+0000 7f87567fc640 1 -- 172.21.15.70:0/266920441 learned_addr learned my addr 172.21.15.70:0/266920441 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.306+0000 7f8756ffd640 1 --2- 172.21.15.70:0/266920441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baecd60 0x55f56bb07740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.306+0000 7f87577fe640 1 --2- 172.21.15.70:0/266920441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 0x55f56bafd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.306+0000 7f87567fc640 1 -- 172.21.15.70:0/266920441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baecd60 msgr2=0x55f56bb07740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.306+0000 7f87567fc640 1 --2- 172.21.15.70:0/266920441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baecd60 0x55f56bb07740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.306+0000 7f87567fc640 1 -- 172.21.15.70:0/266920441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 msgr2=0x55f56bafd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.306+0000 7f87567fc640 1 --2- 172.21.15.70:0/266920441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 0x55f56bafd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.306+0000 7f87567fc640 1 -- 172.21.15.70:0/266920441 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f56bafdb50 con 0x55f56bafbaf0 2024-06-04T04:58:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.306+0000 7f87577fe640 1 --2- 172.21.15.70:0/266920441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 0x55f56bafd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.306+0000 7f87567fc640 1 --2- 172.21.15.70:0/266920441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56bafbaf0 0x55f56bafce30 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f874c0df8e0 tx=0x7f874c0097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab23ae56ba9da78 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.306+0000 7f873aff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.306+0000 7f873aff5640 1 -- 172.21.15.70:0/266920441 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c020050 con 0x55f56bafbaf0 2024-06-04T04:58:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.306+0000 7f873aff5640 1 -- 172.21.15.70:0/266920441 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f874c0dfb30 con 0x55f56bafbaf0 2024-06-04T04:58:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.306+0000 7f876024a740 1 -- 172.21.15.70:0/266920441 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f874c0a9b40 con 0x55f56bafbaf0 2024-06-04T04:58:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.307+0000 7f873aff5640 1 -- 172.21.15.70:0/266920441 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f874c0a9440 con 0x55f56bafbaf0 2024-06-04T04:58:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.307+0000 7f876024a740 10 client.15879 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.307+0000 7f876024a740 1 -- 172.21.15.70:0/266920441 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f56bb79d80 con 0x55f56bafbaf0 2024-06-04T04:58:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.307+0000 7f873aff5640 1 -- 172.21.15.70:0/266920441 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f874c0a9b40 con 0x55f56bafbaf0 2024-06-04T04:58:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.307+0000 7f873aff5640 1 -- 172.21.15.70:0/266920441 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f874c0ca300 con 0x55f56bafbaf0 2024-06-04T04:58:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.307+0000 7f876024a740 10 client.15879 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.307+0000 7f876024a740 10 client.15879 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.307+0000 7f876024a740 20 client.15879 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.307+0000 7f876024a740 1 --2- 172.21.15.70:0/266920441 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f56bb7a4e0 0x55f56bb7c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.307+0000 7f876024a740 4 client.15879 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T04:58:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.307+0000 7f876024a740 1 -- 172.21.15.70:0/266920441 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f56ba59360 con 0x55f56bb7a4e0 2024-06-04T04:58:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.307+0000 7f8756ffd640 1 --2- 172.21.15.70:0/266920441 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f56bb7a4e0 0x55f56bb7c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.308+0000 7f873aff5640 10 client.15879 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.308+0000 7f8756ffd640 1 --2- 172.21.15.70:0/266920441 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f56bb7a4e0 0x55f56bb7c900 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=930b3d47419daf8b server_cookie=98f7003f1c88f94e in_seq=0 out_seq=0 2024-06-04T04:58:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.313+0000 7f873aff5640 1 -- 172.21.15.70:0/266920441 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f56ba59360 con 0x55f56bb7a4e0 2024-06-04T04:58:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.313+0000 7f873aff5640 10 client.15879 handle_command_reply: tid=0 2024-06-04T04:58:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.313+0000 7f876024a740 1 client.15879 shutdown 2024-06-04T04:58:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.313+0000 7f876024a740 1 -- 172.21.15.70:0/266920441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56bafbaf0 msgr2=0x55f56bafce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.313+0000 7f876024a740 1 --2- 172.21.15.70:0/266920441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56bafbaf0 0x55f56bafce30 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f874c0df8e0 tx=0x7f874c0097b0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.313+0000 7f876024a740 1 -- 172.21.15.70:0/266920441 shutdown_connections 2024-06-04T04:58:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.313+0000 7f876024a740 1 --2- 172.21.15.70:0/266920441 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f56bb7a4e0 0x55f56bb7c900 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.313+0000 7f876024a740 1 --2- 172.21.15.70:0/266920441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56baff0a0 0x55f56bafd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.313+0000 7f873aff5640 0 client.15879 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.313+0000 7f876024a740 1 --2- 172.21.15.70:0/266920441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f56bafbaf0 0x55f56bafce30 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.313+0000 7f876024a740 1 --2- 172.21.15.70:0/266920441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f56baecd60 0x55f56bb07740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.313+0000 7f876024a740 1 -- 172.21.15.70:0/266920441 >> 172.21.15.70:0/266920441 conn(0x55f56bae7d30 msgr2=0x55f56baed2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.313+0000 7f876024a740 1 -- 172.21.15.70:0/266920441 shutdown_connections 2024-06-04T04:58:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.313+0000 7f876024a740 1 -- 172.21.15.70:0/266920441 wait complete. 2024-06-04T04:58:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.314+0000 7f876024a740 20 client.15879 trim_cache size 0 max 16384 2024-06-04T04:58:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T04:58:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T04:58:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T04:58:42.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.314+0000 7f875d52b640 1 -- 172.21.15.70:0/2200530820 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f87400b2a90 msgr2=0x7f87400b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:42.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.314+0000 7f875d52b640 1 --2- 172.21.15.70:0/2200530820 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f87400b2a90 0x7f87400b4f50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x55f56baee410 tx=0x7f8748005f90 comp rx=0 tx=0).stop 2024-06-04T04:58:42.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.314+0000 7f875d52b640 1 -- 172.21.15.70:0/2200530820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8758104600 msgr2=0x7f8758199610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:42.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.314+0000 7f875d52b640 1 --2- 172.21.15.70:0/2200530820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8758104600 0x7f8758199610 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f874c00d950 tx=0x7f874c00de10 comp rx=0 tx=0).stop 2024-06-04T04:58:42.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.314+0000 7f875d52b640 1 -- 172.21.15.70:0/2200530820 shutdown_connections 2024-06-04T04:58:42.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.314+0000 7f875d52b640 1 --2- 172.21.15.70:0/2200530820 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f87400b2a90 0x7f87400b4f50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.314+0000 7f875d52b640 1 --2- 172.21.15.70:0/2200530820 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f875810d020 0x7f875819d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.314+0000 7f875d52b640 1 --2- 172.21.15.70:0/2200530820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8758104600 0x7f8758199610 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.314+0000 7f875d52b640 1 --2- 172.21.15.70:0/2200530820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8758103ce0 0x7f87581990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.314+0000 7f875d52b640 1 -- 172.21.15.70:0/2200530820 >> 172.21.15.70:0/2200530820 conn(0x7f8758078ad0 msgr2=0x7f87580feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.314+0000 7f875d52b640 1 -- 172.21.15.70:0/2200530820 shutdown_connections 2024-06-04T04:58:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:42.314+0000 7f875d52b640 1 -- 172.21.15.70:0/2200530820 wait complete. 2024-06-04T04:58:43.321 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:43 smithi070 ceph-mon[32631]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 597 B/s wr, 14 op/s 2024-06-04T04:58:43.321 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:43 smithi070 ceph-mon[32631]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-06-04T04:58:43.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:43 smithi112 ceph-mon[36453]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 597 B/s wr, 14 op/s 2024-06-04T04:58:43.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:43 smithi112 ceph-mon[36453]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-06-04T04:58:43.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:43 smithi098 ceph-mon[35569]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 597 B/s wr, 14 op/s 2024-06-04T04:58:43.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:43 smithi098 ceph-mon[35569]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-06-04T04:58:43.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.774+0000 7ffb927c8700 1 -- 172.21.15.70:0/3963795075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb8c108510 msgr2=0x7ffb8c1088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:43.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.774+0000 7ffb927c8700 1 --2- 172.21.15.70:0/3963795075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb8c108510 0x7ffb8c1088f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009990 tx=0x7ffb7c009ca0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.774+0000 7ffb927c8700 1 -- 172.21.15.70:0/3963795075 shutdown_connections 2024-06-04T04:58:43.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.774+0000 7ffb927c8700 1 --2- 172.21.15.70:0/3963795075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8c102df0 0x7ffb8c10f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.774+0000 7ffb927c8700 1 --2- 172.21.15.70:0/3963795075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb8c108510 0x7ffb8c1088f0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.774+0000 7ffb927c8700 1 --2- 172.21.15.70:0/3963795075 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb8c102430 0x7ffb8c1028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.774+0000 7ffb927c8700 1 -- 172.21.15.70:0/3963795075 >> 172.21.15.70:0/3963795075 conn(0x7ffb8c0793b0 msgr2=0x7ffb8c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:43.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.774+0000 7ffb927c8700 1 -- 172.21.15.70:0/3963795075 shutdown_connections 2024-06-04T04:58:43.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.774+0000 7ffb927c8700 1 -- 172.21.15.70:0/3963795075 wait complete. 2024-06-04T04:58:43.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.775+0000 7ffb927c8700 1 Processor -- start 2024-06-04T04:58:43.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.775+0000 7ffb927c8700 1 -- start start 2024-06-04T04:58:43.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.775+0000 7ffb927c8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8c102430 0x7ffb8c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.775+0000 7ffb927c8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb8c102df0 0x7ffb8c198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.775+0000 7ffb927c8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb8c108510 0x7ffb8c1991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.775+0000 7ffb927c8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c112490 con 0x7ffb8c108510 2024-06-04T04:58:43.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.775+0000 7ffb927c8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c112310 con 0x7ffb8c102430 2024-06-04T04:58:43.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.775+0000 7ffb927c8700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c112610 con 0x7ffb8c102df0 2024-06-04T04:58:43.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.775+0000 7ffb90d65700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb8c108510 0x7ffb8c1991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.775+0000 7ffb90d65700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb8c108510 0x7ffb8c1991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37068/0 (socket says 172.21.15.70:37068) 2024-06-04T04:58:43.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.775+0000 7ffb8b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb8c102df0 0x7ffb8c198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.775+0000 7ffb90d65700 1 -- 172.21.15.70:0/3111279360 learned_addr learned my addr 172.21.15.70:0/3111279360 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.776+0000 7ffb8bfff700 1 --2- 172.21.15.70:0/3111279360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8c102430 0x7ffb8c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.776+0000 7ffb90d65700 1 -- 172.21.15.70:0/3111279360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb8c102df0 msgr2=0x7ffb8c198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.776+0000 7ffb90d65700 1 --2- 172.21.15.70:0/3111279360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb8c102df0 0x7ffb8c198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.776+0000 7ffb90d65700 1 -- 172.21.15.70:0/3111279360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8c102430 msgr2=0x7ffb8c198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.776+0000 7ffb90d65700 1 --2- 172.21.15.70:0/3111279360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8c102430 0x7ffb8c198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.776+0000 7ffb90d65700 1 -- 172.21.15.70:0/3111279360 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb8c1a8020 con 0x7ffb8c108510 2024-06-04T04:58:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.776+0000 7ffb8bfff700 1 --2- 172.21.15.70:0/3111279360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8c102430 0x7ffb8c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.776+0000 7ffb90d65700 1 --2- 172.21.15.70:0/3111279360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb8c108510 0x7ffb8c1991a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ffb84007ee0 tx=0x7ffb8400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.777+0000 7ffb897fa700 1 -- 172.21.15.70:0/3111279360 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb8400dbd0 con 0x7ffb8c108510 2024-06-04T04:58:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.777+0000 7ffb897fa700 1 -- 172.21.15.70:0/3111279360 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb8400dd70 con 0x7ffb8c108510 2024-06-04T04:58:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.777+0000 7ffb927c8700 1 -- 172.21.15.70:0/3111279360 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb8c1a8310 con 0x7ffb8c108510 2024-06-04T04:58:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.777+0000 7ffb897fa700 1 -- 172.21.15.70:0/3111279360 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb840058d0 con 0x7ffb8c108510 2024-06-04T04:58:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.777+0000 7ffb927c8700 1 -- 172.21.15.70:0/3111279360 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb8c1a8800 con 0x7ffb8c108510 2024-06-04T04:58:43.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.781+0000 7ffb9d05db80 1 Processor -- start 2024-06-04T04:58:43.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.781+0000 7ffb897fa700 1 -- 172.21.15.70:0/3111279360 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ffb8401a020 con 0x7ffb8c108510 2024-06-04T04:58:43.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.781+0000 7ffb9d05db80 1 -- start start 2024-06-04T04:58:43.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.781+0000 7ffb9d05db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b673860 0x556c5b673c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.781+0000 7ffb9d05db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b674210 0x556c5b677940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.781+0000 7ffb9d05db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 0x556c5b6783a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.781+0000 7ffb9d05db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556c5b5d2c90 con 0x556c5b677f60 2024-06-04T04:58:43.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.781+0000 7ffb9d05db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556c5b5c2830 con 0x556c5b674210 2024-06-04T04:58:43.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.781+0000 7ffb9d05db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556c5b5c6870 con 0x556c5b673860 2024-06-04T04:58:43.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.781+0000 7ffb8b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 0x556c5b6783a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.781+0000 7ffb8b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 0x556c5b6783a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37070/0 (socket says 172.21.15.70:37070) 2024-06-04T04:58:43.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.781+0000 7ffb8b7fe700 1 -- 172.21.15.70:0/1782982947 learned_addr learned my addr 172.21.15.70:0/1782982947 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:43.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.781+0000 7ffb8bfff700 1 --2- 172.21.15.70:0/1782982947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b674210 0x556c5b677940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.782+0000 7ffb8b7fe700 1 --2- 172.21.15.70:0/1782982947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b673860 0x556c5b673c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.782+0000 7ffb8bfff700 1 -- 172.21.15.70:0/1782982947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b673860 msgr2=0x556c5b673c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:43.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.782+0000 7ffb8bfff700 1 --2- 172.21.15.70:0/1782982947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b673860 0x556c5b673c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.782+0000 7ffb8bfff700 1 -- 172.21.15.70:0/1782982947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 msgr2=0x556c5b6783a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:43.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.782+0000 7ffb8bfff700 1 --2- 172.21.15.70:0/1782982947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 0x556c5b6783a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.782+0000 7ffb8bfff700 1 -- 172.21.15.70:0/1782982947 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c5b5128d0 con 0x556c5b674210 2024-06-04T04:58:43.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.782+0000 7ffb8b7fe700 1 --2- 172.21.15.70:0/1782982947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b673860 0x556c5b673c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:43.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.782+0000 7ffb8bfff700 1 --2- 172.21.15.70:0/1782982947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b674210 0x556c5b677940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c0098e0 tx=0x7ffb7c005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ece73b944e57fb91 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:43.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.782+0000 7ffb897fa700 1 --2- 172.21.15.70:0/3111279360 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ffb740b2a70 0x7ffb740b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.782+0000 7ffb727fc700 1 -- 172.21.15.70:0/1782982947 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c027070 con 0x556c5b674210 2024-06-04T04:58:43.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.783+0000 7ffb8b7fe700 1 --2- 172.21.15.70:0/3111279360 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ffb740b2a70 0x7ffb740b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.783+0000 7ffb897fa700 1 -- 172.21.15.70:0/3111279360 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ffb840e1dd0 con 0x7ffb8c108510 2024-06-04T04:58:43.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.783+0000 7ffb727fc700 1 -- 172.21.15.70:0/1782982947 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb7c0037c0 con 0x556c5b674210 2024-06-04T04:58:43.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.783+0000 7ffb727fc700 1 -- 172.21.15.70:0/1782982947 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c00ba30 con 0x556c5b674210 2024-06-04T04:58:43.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.784+0000 7ffb9d05db80 1 -- 172.21.15.70:0/1782982947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b674210 msgr2=0x556c5b677940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:43.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.784+0000 7ffb8b7fe700 1 --2- 172.21.15.70:0/3111279360 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ffb740b2a70 0x7ffb740b4f30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ffb80004100 tx=0x7ffb8000b8e0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:43.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.784+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/1782982947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b674210 0x556c5b677940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c0098e0 tx=0x7ffb7c005c00 comp rx=0 tx=0).stop 2024-06-04T04:58:43.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.784+0000 7ffb9d05db80 1 -- 172.21.15.70:0/1782982947 shutdown_connections 2024-06-04T04:58:43.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.784+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/1782982947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 0x556c5b6783a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.784+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/1782982947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b673860 0x556c5b673c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.784+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/1782982947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b674210 0x556c5b677940 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.784+0000 7ffb9d05db80 1 -- 172.21.15.70:0/1782982947 >> 172.21.15.70:0/1782982947 conn(0x556c5b5a1080 msgr2=0x556c5b66de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:43.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.784+0000 7ffb9d05db80 1 -- 172.21.15.70:0/1782982947 shutdown_connections 2024-06-04T04:58:43.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.784+0000 7ffb9d05db80 1 -- 172.21.15.70:0/1782982947 wait complete. 2024-06-04T04:58:43.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.784+0000 7ffb9d05db80 1 -- start start 2024-06-04T04:58:43.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.785+0000 7ffb9d05db80 1 Processor -- start 2024-06-04T04:58:43.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.785+0000 7ffb9d05db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b673860 0x556c5b67f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.785+0000 7ffb9d05db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b674210 0x556c5b67c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.785+0000 7ffb9d05db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 0x556c5b67cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.785+0000 7ffb9d05db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556c5b4c8fa0 con 0x556c5b677f60 2024-06-04T04:58:43.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.785+0000 7ffb9d05db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556c5b5647c0 con 0x556c5b674210 2024-06-04T04:58:43.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.785+0000 7ffb9d05db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556c5b6800c0 con 0x556c5b673860 2024-06-04T04:58:43.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.785+0000 7ffb9d05db80 10 client.0 _ll_register_callbacks cb 0x556c5b55d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:43.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.785+0000 7ffb90d65700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 0x556c5b67cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.785+0000 7ffb8b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b674210 0x556c5b67c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.785+0000 7ffb8b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b674210 0x556c5b67c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46728/0 (socket says 172.21.15.70:46728) 2024-06-04T04:58:43.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.785+0000 7ffb8b7fe700 1 -- 172.21.15.70:0/3966289769 learned_addr learned my addr 172.21.15.70:0/3966289769 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:43.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.786+0000 7ffb8bfff700 1 --2- 172.21.15.70:0/3966289769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b673860 0x556c5b67f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.786+0000 7ffb8b7fe700 1 -- 172.21.15.70:0/3966289769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b673860 msgr2=0x556c5b67f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:43.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.786+0000 7ffb8b7fe700 1 --2- 172.21.15.70:0/3966289769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b673860 0x556c5b67f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.786+0000 7ffb8b7fe700 1 -- 172.21.15.70:0/3966289769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 msgr2=0x556c5b67cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:43.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.786+0000 7ffb8b7fe700 1 --2- 172.21.15.70:0/3966289769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 0x556c5b67cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.786+0000 7ffb8b7fe700 1 -- 172.21.15.70:0/3966289769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c5b561690 con 0x556c5b674210 2024-06-04T04:58:43.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.786+0000 7ffb90d65700 1 --2- 172.21.15.70:0/3966289769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 0x556c5b67cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:43.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.786+0000 7ffb8b7fe700 1 --2- 172.21.15.70:0/3966289769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b674210 0x556c5b67c600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffb80003dc0 tx=0x7ffb8000b3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3907a8376780d9a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:43.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.786+0000 7ffb6b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T04:58:43.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.786+0000 7ffb6b7fe700 1 -- 172.21.15.70:0/3966289769 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb80021070 con 0x556c5b674210 2024-06-04T04:58:43.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.786+0000 7ffb9d05db80 1 -- 172.21.15.70:0/3966289769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb7c0037c0 con 0x556c5b674210 2024-06-04T04:58:43.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.786+0000 7ffb6b7fe700 1 -- 172.21.15.70:0/3966289769 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ffb80010cd0 con 0x556c5b674210 2024-06-04T04:58:43.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.787+0000 7ffb6b7fe700 1 -- 172.21.15.70:0/3966289769 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffb80003ea0 con 0x556c5b674210 2024-06-04T04:58:43.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.787+0000 7ffb9d05db80 10 client.25337 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:43.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.787+0000 7ffb9d05db80 1 -- 172.21.15.70:0/3966289769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556c5b68a750 con 0x556c5b674210 2024-06-04T04:58:43.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.787+0000 7ffb6b7fe700 1 -- 172.21.15.70:0/3966289769 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb8002c070 con 0x556c5b674210 2024-06-04T04:58:43.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.787+0000 7ffb6b7fe700 1 -- 172.21.15.70:0/3966289769 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7ffb80031490 con 0x556c5b674210 2024-06-04T04:58:43.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.787+0000 7ffb9d05db80 10 client.25337 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:43.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.787+0000 7ffb9d05db80 10 client.25337 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T04:58:43.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.787+0000 7ffb9d05db80 20 client.25337 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:43.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.787+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/3966289769 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556c5b68cd30 0x556c5b68f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.787+0000 7ffb9d05db80 4 client.25337 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:43.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.787+0000 7ffb9d05db80 1 -- 172.21.15.70:0/3966289769 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ffb7c0037c0 con 0x556c5b68cd30 2024-06-04T04:58:43.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.787+0000 7ffb8bfff700 1 --2- 172.21.15.70:0/3966289769 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556c5b68cd30 0x556c5b68f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.788+0000 7ffb6b7fe700 10 client.25337 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T04:58:43.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.788+0000 7ffb8bfff700 1 --2- 172.21.15.70:0/3966289769 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556c5b68cd30 0x556c5b68f150 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8091def2cf42d905 server_cookie=79d32d83c757795d in_seq=0 out_seq=0 2024-06-04T04:58:43.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.789+0000 7ffb6b7fe700 1 -- 172.21.15.70:0/3966289769 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7ffb7c0037c0 con 0x556c5b68cd30 2024-06-04T04:58:43.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.789+0000 7ffb6b7fe700 10 client.25337 handle_command_reply: tid=0 2024-06-04T04:58:43.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.789+0000 7ffb9d05db80 1 client.25337 shutdown 2024-06-04T04:58:43.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.789+0000 7ffb9d05db80 1 -- 172.21.15.70:0/3966289769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b674210 msgr2=0x556c5b67c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:43.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.789+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/3966289769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b674210 0x556c5b67c600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffb80003dc0 tx=0x7ffb8000b3c0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.789+0000 7ffb9d05db80 1 -- 172.21.15.70:0/3966289769 shutdown_connections 2024-06-04T04:58:43.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.789+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/3966289769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 0x556c5b67cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.789+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/3966289769 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556c5b68cd30 0x556c5b68f150 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.789+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/3966289769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b673860 0x556c5b67f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.789+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/3966289769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b674210 0x556c5b67c600 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.789+0000 7ffb9d05db80 1 -- 172.21.15.70:0/3966289769 >> 172.21.15.70:0/3966289769 conn(0x556c5b5a1080 msgr2=0x556c5b66d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:43.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.789+0000 7ffb6b7fe700 0 client.25337 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T04:58:43.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.789+0000 7ffb9d05db80 1 -- 172.21.15.70:0/3966289769 shutdown_connections 2024-06-04T04:58:43.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.789+0000 7ffb9d05db80 1 -- 172.21.15.70:0/3966289769 wait complete. 2024-06-04T04:58:43.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.789+0000 7ffb9d05db80 20 client.25337 trim_cache size 0 max 16384 2024-06-04T04:58:43.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb9d05db80 1 Processor -- start 2024-06-04T04:58:43.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb9d05db80 1 -- start start 2024-06-04T04:58:43.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb9d05db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b673860 0x556c5b673050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb9d05db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b674210 0x556c5b6758b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb9d05db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b677f60 0x556c5b675df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb9d05db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c1aba00 con 0x556c5b674210 2024-06-04T04:58:43.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb9d05db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c1ad540 con 0x556c5b673860 2024-06-04T04:58:43.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb9d05db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c1ac0d0 con 0x556c5b677f60 2024-06-04T04:58:43.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b673860 0x556c5b673050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b673860 0x556c5b673050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46732/0 (socket says 172.21.15.70:46732) 2024-06-04T04:58:43.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb8bfff700 1 -- 172.21.15.70:0/1673461719 learned_addr learned my addr 172.21.15.70:0/1673461719 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:43.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb90d65700 1 --2- 172.21.15.70:0/1673461719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b677f60 0x556c5b675df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb8b7fe700 1 --2- 172.21.15.70:0/1673461719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b674210 0x556c5b6758b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb8bfff700 1 -- 172.21.15.70:0/1673461719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b677f60 msgr2=0x556c5b675df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:43.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb8bfff700 1 --2- 172.21.15.70:0/1673461719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b677f60 0x556c5b675df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb8bfff700 1 -- 172.21.15.70:0/1673461719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b674210 msgr2=0x556c5b6758b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:43.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb8bfff700 1 --2- 172.21.15.70:0/1673461719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b674210 0x556c5b6758b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb8bfff700 1 -- 172.21.15.70:0/1673461719 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c5b67ab90 con 0x556c5b673860 2024-06-04T04:58:43.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb90d65700 1 --2- 172.21.15.70:0/1673461719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b677f60 0x556c5b675df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:43.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb8b7fe700 1 --2- 172.21.15.70:0/1673461719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b674210 0x556c5b6758b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:43.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.805+0000 7ffb8bfff700 1 --2- 172.21.15.70:0/1673461719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b673860 0x556c5b673050 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009990 tx=0x7ffb7c01a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7be7788743c2036 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:43.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.806+0000 7ffb71ffb700 1 -- 172.21.15.70:0/1673461719 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c027040 con 0x556c5b673860 2024-06-04T04:58:43.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.806+0000 7ffb71ffb700 1 -- 172.21.15.70:0/1673461719 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb7c034040 con 0x556c5b673860 2024-06-04T04:58:43.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.806+0000 7ffb71ffb700 1 -- 172.21.15.70:0/1673461719 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c022480 con 0x556c5b673860 2024-06-04T04:58:43.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.806+0000 7ffb9d05db80 1 -- 172.21.15.70:0/1673461719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b673860 msgr2=0x556c5b673050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:43.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.806+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/1673461719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b673860 0x556c5b673050 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009990 tx=0x7ffb7c01a040 comp rx=0 tx=0).stop 2024-06-04T04:58:43.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.807+0000 7ffb9d05db80 1 -- 172.21.15.70:0/1673461719 shutdown_connections 2024-06-04T04:58:43.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.807+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/1673461719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b677f60 0x556c5b675df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.807+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/1673461719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b673860 0x556c5b673050 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.807+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/1673461719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b674210 0x556c5b6758b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.807+0000 7ffb9d05db80 1 -- 172.21.15.70:0/1673461719 >> 172.21.15.70:0/1673461719 conn(0x556c5b5a1080 msgr2=0x556c5b67e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:43.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.807+0000 7ffb9d05db80 1 -- 172.21.15.70:0/1673461719 shutdown_connections 2024-06-04T04:58:43.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.807+0000 7ffb9d05db80 1 -- 172.21.15.70:0/1673461719 wait complete. 2024-06-04T04:58:43.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.807+0000 7ffb9d05db80 1 -- start start 2024-06-04T04:58:43.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.807+0000 7ffb9d05db80 1 Processor -- start 2024-06-04T04:58:43.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.808+0000 7ffb9d05db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b673860 0x556c5b66d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.808+0000 7ffb9d05db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b674210 0x556c5b66dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.808+0000 7ffb9d05db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 0x556c5b66e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.808+0000 7ffb9d05db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c1ace70 con 0x556c5b677f60 2024-06-04T04:58:43.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.808+0000 7ffb9d05db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c1ac7a0 con 0x556c5b673860 2024-06-04T04:58:43.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.808+0000 7ffb9d05db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c19ea20 con 0x556c5b674210 2024-06-04T04:58:43.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.808+0000 7ffb9d05db80 10 client.0 _ll_register_callbacks cb 0x556c5b55d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:43.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.808+0000 7ffb90d65700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 0x556c5b66e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.808+0000 7ffb8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b673860 0x556c5b66d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.808+0000 7ffb90d65700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 0x556c5b66e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37092/0 (socket says 172.21.15.70:37092) 2024-06-04T04:58:43.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.808+0000 7ffb8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b673860 0x556c5b66d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46734/0 (socket says 172.21.15.70:46734) 2024-06-04T04:58:43.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.808+0000 7ffb90d65700 1 -- 172.21.15.70:0/3886936680 learned_addr learned my addr 172.21.15.70:0/3886936680 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:43.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.808+0000 7ffb8b7fe700 1 --2- 172.21.15.70:0/3886936680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b674210 0x556c5b66dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.809+0000 7ffb90d65700 1 -- 172.21.15.70:0/3886936680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b674210 msgr2=0x556c5b66dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:43.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.809+0000 7ffb90d65700 1 --2- 172.21.15.70:0/3886936680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b674210 0x556c5b66dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.809+0000 7ffb90d65700 1 -- 172.21.15.70:0/3886936680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b673860 msgr2=0x556c5b66d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:43.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.809+0000 7ffb90d65700 1 --2- 172.21.15.70:0/3886936680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b673860 0x556c5b66d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.809+0000 7ffb90d65700 1 -- 172.21.15.70:0/3886936680 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c5b6f68b0 con 0x556c5b677f60 2024-06-04T04:58:43.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.809+0000 7ffb8bfff700 1 --2- 172.21.15.70:0/3886936680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b673860 0x556c5b66d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:43.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.809+0000 7ffb90d65700 1 --2- 172.21.15.70:0/3886936680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 0x556c5b66e160 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ffb840a62c0 tx=0x7ffb840a8ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f26b69c30a4fb79 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:43.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.809+0000 7ffb6b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:43.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.809+0000 7ffb6b7fe700 1 -- 172.21.15.70:0/3886936680 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb840a9050 con 0x556c5b677f60 2024-06-04T04:58:43.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.809+0000 7ffb6b7fe700 1 -- 172.21.15.70:0/3886936680 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ffb840a92b0 con 0x556c5b677f60 2024-06-04T04:58:43.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.809+0000 7ffb9d05db80 1 -- 172.21.15.70:0/3886936680 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb7c034040 con 0x556c5b677f60 2024-06-04T04:58:43.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.809+0000 7ffb6b7fe700 1 -- 172.21.15.70:0/3886936680 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffb840ad400 con 0x556c5b677f60 2024-06-04T04:58:43.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.809+0000 7ffb9d05db80 10 client.15894 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:43.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.809+0000 7ffb9d05db80 1 -- 172.21.15.70:0/3886936680 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556c5b6f79a0 con 0x556c5b677f60 2024-06-04T04:58:43.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.810+0000 7ffb6b7fe700 1 -- 172.21.15.70:0/3886936680 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb7c034040 con 0x556c5b677f60 2024-06-04T04:58:43.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.810+0000 7ffb6b7fe700 1 -- 172.21.15.70:0/3886936680 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7ffb84004fb0 con 0x556c5b677f60 2024-06-04T04:58:43.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.810+0000 7ffb9d05db80 10 client.15894 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:43.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.810+0000 7ffb9d05db80 10 client.15894 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T04:58:43.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.810+0000 7ffb9d05db80 20 client.15894 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:43.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.810+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/3886936680 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556c5b6f8050 0x556c5b6fa470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:43.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.810+0000 7ffb9d05db80 4 client.15894 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T04:58:43.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.810+0000 7ffb9d05db80 1 -- 172.21.15.70:0/3886936680 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556c5b63f9e0 con 0x556c5b6f8050 2024-06-04T04:58:43.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.810+0000 7ffb8bfff700 1 --2- 172.21.15.70:0/3886936680 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556c5b6f8050 0x556c5b6fa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:43.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.811+0000 7ffb6b7fe700 10 client.15894 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T04:58:43.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.811+0000 7ffb8bfff700 1 --2- 172.21.15.70:0/3886936680 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556c5b6f8050 0x556c5b6fa470 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8ea0b08a74cc4561 server_cookie=7dcaa7919055ef7b in_seq=0 out_seq=0 2024-06-04T04:58:43.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.811+0000 7ffb6b7fe700 1 -- 172.21.15.70:0/3886936680 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556c5b63f9e0 con 0x556c5b6f8050 2024-06-04T04:58:43.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.811+0000 7ffb6b7fe700 10 client.15894 handle_command_reply: tid=0 2024-06-04T04:58:43.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.811+0000 7ffb9d05db80 1 client.15894 shutdown 2024-06-04T04:58:43.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.811+0000 7ffb9d05db80 1 -- 172.21.15.70:0/3886936680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 msgr2=0x556c5b66e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:43.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.811+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/3886936680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 0x556c5b66e160 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ffb840a62c0 tx=0x7ffb840a8ab0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.811+0000 7ffb9d05db80 1 -- 172.21.15.70:0/3886936680 shutdown_connections 2024-06-04T04:58:43.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.811+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/3886936680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c5b677f60 0x556c5b66e160 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.811+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/3886936680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c5b673860 0x556c5b66d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.811+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/3886936680 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556c5b6f8050 0x556c5b6fa470 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.811+0000 7ffb9d05db80 1 --2- 172.21.15.70:0/3886936680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c5b674210 0x556c5b66dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.811+0000 7ffb6b7fe700 0 client.15894 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T04:58:43.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.811+0000 7ffb9d05db80 1 -- 172.21.15.70:0/3886936680 >> 172.21.15.70:0/3886936680 conn(0x556c5b5a1080 msgr2=0x556c5b6763c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:43.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.812+0000 7ffb9d05db80 1 -- 172.21.15.70:0/3886936680 shutdown_connections 2024-06-04T04:58:43.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.812+0000 7ffb9d05db80 1 -- 172.21.15.70:0/3886936680 wait complete. 2024-06-04T04:58:43.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.812+0000 7ffb9d05db80 20 client.15894 trim_cache size 0 max 16384 2024-06-04T04:58:43.820 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T04:58:43.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.812+0000 7ffb927c8700 1 -- 172.21.15.70:0/3111279360 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ffb740b2a70 msgr2=0x7ffb740b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:43.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.812+0000 7ffb927c8700 1 --2- 172.21.15.70:0/3111279360 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ffb740b2a70 0x7ffb740b4f30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ffb80004100 tx=0x7ffb8000b8e0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.812+0000 7ffb927c8700 1 -- 172.21.15.70:0/3111279360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb8c108510 msgr2=0x7ffb8c1991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:43.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.812+0000 7ffb927c8700 1 --2- 172.21.15.70:0/3111279360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb8c108510 0x7ffb8c1991a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ffb84007ee0 tx=0x7ffb8400d3b0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.813+0000 7ffb927c8700 1 -- 172.21.15.70:0/3111279360 shutdown_connections 2024-06-04T04:58:43.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.813+0000 7ffb927c8700 1 --2- 172.21.15.70:0/3111279360 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ffb740b2a70 0x7ffb740b4f30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.813+0000 7ffb927c8700 1 --2- 172.21.15.70:0/3111279360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb8c108510 0x7ffb8c1991a0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.813+0000 7ffb927c8700 1 --2- 172.21.15.70:0/3111279360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8c102430 0x7ffb8c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.813+0000 7ffb927c8700 1 --2- 172.21.15.70:0/3111279360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb8c102df0 0x7ffb8c198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:43.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.813+0000 7ffb927c8700 1 -- 172.21.15.70:0/3111279360 >> 172.21.15.70:0/3111279360 conn(0x7ffb8c0793b0 msgr2=0x7ffb8c101330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:43.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.813+0000 7ffb927c8700 1 -- 172.21.15.70:0/3111279360 shutdown_connections 2024-06-04T04:58:43.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:43.813+0000 7ffb927c8700 1 -- 172.21.15.70:0/3111279360 wait complete. 2024-06-04T04:58:44.628 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T04:58:44.996 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:45.394 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:45 smithi070 ceph-mon[32631]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-06-04T04:58:45.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:45 smithi112 ceph-mon[36453]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-06-04T04:58:45.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:45 smithi098 ceph-mon[35569]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-06-04T04:58:46.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.916+0000 7fd8986ca700 1 -- 172.21.15.70:0/912192866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd890102c40 msgr2=0x7fd8901030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:46.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.916+0000 7fd8986ca700 1 --2- 172.21.15.70:0/912192866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd890102c40 0x7fd8901030c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd880009990 tx=0x7fd880009ca0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.916+0000 7fd8986ca700 1 -- 172.21.15.70:0/912192866 shutdown_connections 2024-06-04T04:58:46.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.916+0000 7fd8986ca700 1 --2- 172.21.15.70:0/912192866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd890103600 0x7fd89010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.916+0000 7fd8986ca700 1 --2- 172.21.15.70:0/912192866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8901084c0 0x7fd8901088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.916+0000 7fd8986ca700 1 --2- 172.21.15.70:0/912192866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd890102c40 0x7fd8901030c0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.916+0000 7fd8986ca700 1 -- 172.21.15.70:0/912192866 >> 172.21.15.70:0/912192866 conn(0x7fd890078ab0 msgr2=0x7fd8900ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:46.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.916+0000 7fd8986ca700 1 -- 172.21.15.70:0/912192866 shutdown_connections 2024-06-04T04:58:46.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.916+0000 7fd8986ca700 1 -- 172.21.15.70:0/912192866 wait complete. 2024-06-04T04:58:46.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.917+0000 7fd8986ca700 1 Processor -- start 2024-06-04T04:58:46.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.917+0000 7fd8986ca700 1 -- start start 2024-06-04T04:58:46.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.918+0000 7fd8986ca700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd890102c40 0x7fd89019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.918+0000 7fd8986ca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd890103600 0x7fd89019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.918+0000 7fd8986ca700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8901084c0 0x7fd890199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.918+0000 7fd8986ca700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd890112480 con 0x7fd8901084c0 2024-06-04T04:58:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.918+0000 7fd8986ca700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd890112300 con 0x7fd890103600 2024-06-04T04:58:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.918+0000 7fd8986ca700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd890112600 con 0x7fd890102c40 2024-06-04T04:58:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.918+0000 7fd895c65700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd890103600 0x7fd89019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.918+0000 7fd895c65700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd890103600 0x7fd89019ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:53990/0 (socket says 172.21.15.70:53990) 2024-06-04T04:58:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.918+0000 7fd895c65700 1 -- 172.21.15.70:0/4046522430 learned_addr learned my addr 172.21.15.70:0/4046522430 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.918+0000 7fd896c67700 1 --2- 172.21.15.70:0/4046522430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8901084c0 0x7fd890199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.918+0000 7fd895c65700 1 -- 172.21.15.70:0/4046522430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd890102c40 msgr2=0x7fd89019e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.918+0000 7fd895c65700 1 --2- 172.21.15.70:0/4046522430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd890102c40 0x7fd89019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.918+0000 7fd895c65700 1 -- 172.21.15.70:0/4046522430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8901084c0 msgr2=0x7fd890199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.918+0000 7fd895c65700 1 --2- 172.21.15.70:0/4046522430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8901084c0 0x7fd890199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.918+0000 7fd895c65700 1 -- 172.21.15.70:0/4046522430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89019a400 con 0x7fd890103600 2024-06-04T04:58:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.918+0000 7fd896c67700 1 --2- 172.21.15.70:0/4046522430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8901084c0 0x7fd890199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.919+0000 7fd895c65700 1 --2- 172.21.15.70:0/4046522430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd890103600 0x7fd89019ec20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd880006010 tx=0x7fd88000bfc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.919+0000 7fd8877fe700 1 -- 172.21.15.70:0/4046522430 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd880027070 con 0x7fd890103600 2024-06-04T04:58:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.919+0000 7fd8986ca700 1 -- 172.21.15.70:0/4046522430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd89019a690 con 0x7fd890103600 2024-06-04T04:58:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.919+0000 7fd8986ca700 1 -- 172.21.15.70:0/4046522430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8901a67a0 con 0x7fd890103600 2024-06-04T04:58:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.919+0000 7fd8877fe700 1 -- 172.21.15.70:0/4046522430 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd880022470 con 0x7fd890103600 2024-06-04T04:58:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.919+0000 7fd8877fe700 1 -- 172.21.15.70:0/4046522430 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd880017650 con 0x7fd890103600 2024-06-04T04:58:46.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.923+0000 7fd8877fe700 1 -- 172.21.15.70:0/4046522430 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fd88002a070 con 0x7fd890103600 2024-06-04T04:58:46.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.924+0000 7fd8877fe700 1 --2- 172.21.15.70:0/4046522430 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd87c0b2a70 0x7fd87c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.924+0000 7fd896466700 1 --2- 172.21.15.70:0/4046522430 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd87c0b2a70 0x7fd87c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.924+0000 7fd8877fe700 1 -- 172.21.15.70:0/4046522430 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fd8800e35d0 con 0x7fd890103600 2024-06-04T04:58:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.924+0000 7fd8a2f5fb80 1 Processor -- start 2024-06-04T04:58:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.925+0000 7fd896466700 1 --2- 172.21.15.70:0/4046522430 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd87c0b2a70 0x7fd87c0b4f30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd88c005fd0 tx=0x7fd88c005f00 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.925+0000 7fd8a2f5fb80 1 -- start start 2024-06-04T04:58:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.925+0000 7fd8a2f5fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 0x55c553f7cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.925+0000 7fd8a2f5fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f7d210 0x55c553f8a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.925+0000 7fd8a2f5fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f8a9c0 0x55c553f8ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.925+0000 7fd8a2f5fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c553edbc90 con 0x55c553f7d210 2024-06-04T04:58:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.926+0000 7fd8a2f5fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c553ecb830 con 0x55c553f7c860 2024-06-04T04:58:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.926+0000 7fd8a2f5fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c553ecf870 con 0x55c553f8a9c0 2024-06-04T04:58:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.926+0000 7fd895c65700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f7d210 0x55c553f8a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.926+0000 7fd895c65700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f7d210 0x55c553f8a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42482/0 (socket says 172.21.15.70:42482) 2024-06-04T04:58:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.926+0000 7fd895c65700 1 -- 172.21.15.70:0/1268634682 learned_addr learned my addr 172.21.15.70:0/1268634682 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.926+0000 7fd896466700 1 --2- 172.21.15.70:0/1268634682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 0x55c553f7cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.926+0000 7fd895c65700 1 -- 172.21.15.70:0/1268634682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f8a9c0 msgr2=0x55c553f8ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.926+0000 7fd895c65700 1 --2- 172.21.15.70:0/1268634682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f8a9c0 0x55c553f8ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.926+0000 7fd895c65700 1 -- 172.21.15.70:0/1268634682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 msgr2=0x55c553f7cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.926+0000 7fd895c65700 1 --2- 172.21.15.70:0/1268634682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 0x55c553f7cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.926+0000 7fd895c65700 1 -- 172.21.15.70:0/1268634682 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c553e1b8d0 con 0x55c553f7d210 2024-06-04T04:58:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.926+0000 7fd896466700 1 --2- 172.21.15.70:0/1268634682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 0x55c553f7cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.926+0000 7fd895c65700 1 --2- 172.21.15.70:0/1268634682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f7d210 0x55c553f8a3a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd8800aa870 tx=0x7fd8800aa050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81231f504d2f4d21 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.926+0000 7fd884ff9700 1 -- 172.21.15.70:0/1268634682 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8800a6070 con 0x55c553f7d210 2024-06-04T04:58:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.926+0000 7fd884ff9700 1 -- 172.21.15.70:0/1268634682 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8800aafe0 con 0x55c553f7d210 2024-06-04T04:58:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.927+0000 7fd884ff9700 1 -- 172.21.15.70:0/1268634682 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8800b10b0 con 0x55c553f7d210 2024-06-04T04:58:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.927+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/1268634682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f7d210 msgr2=0x55c553f8a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.928+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/1268634682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f7d210 0x55c553f8a3a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd8800aa870 tx=0x7fd8800aa050 comp rx=0 tx=0).stop 2024-06-04T04:58:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.928+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/1268634682 shutdown_connections 2024-06-04T04:58:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.928+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/1268634682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f8a9c0 0x55c553f8ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.928+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/1268634682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 0x55c553f7cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.928+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/1268634682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f7d210 0x55c553f8a3a0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.928+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/1268634682 >> 172.21.15.70:0/1268634682 conn(0x55c553eaa080 msgr2=0x55c553f76e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.928+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/1268634682 shutdown_connections 2024-06-04T04:58:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.928+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/1268634682 wait complete. 2024-06-04T04:58:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.929+0000 7fd8a2f5fb80 1 -- start start 2024-06-04T04:58:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.929+0000 7fd8a2f5fb80 1 Processor -- start 2024-06-04T04:58:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.929+0000 7fd8a2f5fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 0x55c553f90ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.929+0000 7fd8a2f5fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f7d210 0x55c553f8dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.929+0000 7fd8a2f5fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f8a9c0 0x55c553f8e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.930+0000 7fd896466700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 0x55c553f90ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.930+0000 7fd896466700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 0x55c553f90ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54010/0 (socket says 172.21.15.70:54010) 2024-06-04T04:58:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.930+0000 7fd895c65700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f7d210 0x55c553f8dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.930+0000 7fd896c67700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f8a9c0 0x55c553f8e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.930+0000 7fd895c65700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f7d210 0x55c553f8dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:50668/0 (socket says 172.21.15.70:50668) 2024-06-04T04:58:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.930+0000 7fd896c67700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f8a9c0 0x55c553f8e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42484/0 (socket says 172.21.15.70:42484) 2024-06-04T04:58:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.930+0000 7fd8a2f5fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c553dd1fa0 con 0x55c553f8a9c0 2024-06-04T04:58:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.930+0000 7fd896466700 1 -- 172.21.15.70:0/2328624544 learned_addr learned my addr 172.21.15.70:0/2328624544 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.930+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2328624544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c553e6d7c0 con 0x55c553f7c860 2024-06-04T04:58:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.930+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2328624544 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c553f91710 con 0x55c553f7d210 2024-06-04T04:58:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.930+0000 7fd8a2f5fb80 10 client.0 _ll_register_callbacks cb 0x55c553e66920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:46.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.931+0000 7fd896c67700 1 -- 172.21.15.70:0/2328624544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f7d210 msgr2=0x55c553f8dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:46.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.931+0000 7fd896c67700 1 --2- 172.21.15.70:0/2328624544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f7d210 0x55c553f8dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.931+0000 7fd896c67700 1 -- 172.21.15.70:0/2328624544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 msgr2=0x55c553f90ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:46.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.931+0000 7fd896c67700 1 --2- 172.21.15.70:0/2328624544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 0x55c553f90ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.931+0000 7fd896c67700 1 -- 172.21.15.70:0/2328624544 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c553e6a690 con 0x55c553f8a9c0 2024-06-04T04:58:46.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.931+0000 7fd896466700 1 --2- 172.21.15.70:0/2328624544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 0x55c553f90ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:46.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.931+0000 7fd896c67700 1 --2- 172.21.15.70:0/2328624544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f8a9c0 0x55c553f8e3e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x55c553f8b380 tx=0x7fd888007d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3604564c556ad252 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:46.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.931+0000 7fd879ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:46.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.931+0000 7fd879ffb700 1 -- 172.21.15.70:0/2328624544 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd888004bb0 con 0x55c553f8a9c0 2024-06-04T04:58:46.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.931+0000 7fd879ffb700 1 -- 172.21.15.70:0/2328624544 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fd888004d50 con 0x55c553f8a9c0 2024-06-04T04:58:46.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.931+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2328624544 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8800aafe0 con 0x55c553f8a9c0 2024-06-04T04:58:46.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.931+0000 7fd879ffb700 1 -- 172.21.15.70:0/2328624544 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd888023050 con 0x55c553f8a9c0 2024-06-04T04:58:46.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.932+0000 7fd8a2f5fb80 10 client.15903 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:46.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.932+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2328624544 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c553f9bc50 con 0x55c553f8a9c0 2024-06-04T04:58:46.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.932+0000 7fd879ffb700 1 -- 172.21.15.70:0/2328624544 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd88800fc60 con 0x55c553f8a9c0 2024-06-04T04:58:46.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.932+0000 7fd879ffb700 1 -- 172.21.15.70:0/2328624544 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fd888016b00 con 0x55c553f8a9c0 2024-06-04T04:58:46.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.932+0000 7fd8a2f5fb80 10 client.15903 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:46.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.932+0000 7fd8a2f5fb80 10 client.15903 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T04:58:46.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.932+0000 7fd8a2f5fb80 20 client.15903 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:46.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.932+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/2328624544 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55c553f9e330 0x55c553fa0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.932+0000 7fd8a2f5fb80 4 client.15903 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.932+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2328624544 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd8800aafe0 con 0x55c553f9e330 2024-06-04T04:58:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.933+0000 7fd896466700 1 --2- 172.21.15.70:0/2328624544 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55c553f9e330 0x55c553fa0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.933+0000 7fd879ffb700 10 client.15903 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T04:58:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.933+0000 7fd896466700 1 --2- 172.21.15.70:0/2328624544 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55c553f9e330 0x55c553fa0720 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=54b77f69633f2e7e server_cookie=7fba80815bad3532 in_seq=0 out_seq=0 2024-06-04T04:58:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.934+0000 7fd879ffb700 1 -- 172.21.15.70:0/2328624544 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fd8800aafe0 con 0x55c553f9e330 2024-06-04T04:58:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.934+0000 7fd879ffb700 10 client.15903 handle_command_reply: tid=0 2024-06-04T04:58:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.934+0000 7fd8a2f5fb80 1 client.15903 shutdown 2024-06-04T04:58:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.934+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2328624544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f8a9c0 msgr2=0x55c553f8e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.935+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/2328624544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f8a9c0 0x55c553f8e3e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x55c553f8b380 tx=0x7fd888007d60 comp rx=0 tx=0).stop 2024-06-04T04:58:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.935+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2328624544 shutdown_connections 2024-06-04T04:58:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.935+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/2328624544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f8a9c0 0x55c553f8e3e0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.935+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/2328624544 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55c553f9e330 0x55c553fa0720 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.935+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/2328624544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 0x55c553f90ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.935+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/2328624544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f7d210 0x55c553f8dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.935+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2328624544 >> 172.21.15.70:0/2328624544 conn(0x55c553eaa080 msgr2=0x55c553f76c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:46.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.935+0000 7fd879ffb700 0 client.15903 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T04:58:46.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.935+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2328624544 shutdown_connections 2024-06-04T04:58:46.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.935+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2328624544 wait complete. 2024-06-04T04:58:46.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.935+0000 7fd8a2f5fb80 20 client.15903 trim_cache size 0 max 16384 2024-06-04T04:58:46.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.965+0000 7fd8a2f5fb80 1 Processor -- start 2024-06-04T04:58:46.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.965+0000 7fd8a2f5fb80 1 -- start start 2024-06-04T04:58:46.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.965+0000 7fd8a2f5fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 0x55c553f889a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.965+0000 7fd8a2f5fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f7d210 0x55c553f906e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.965+0000 7fd8a2f5fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f8a9c0 0x55c553f76c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.965+0000 7fd896466700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 0x55c553f889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:46.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.965+0000 7fd896466700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 0x55c553f889a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54018/0 (socket says 172.21.15.70:54018) 2024-06-04T04:58:46.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.965+0000 7fd896c67700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f8a9c0 0x55c553f76c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:46.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.965+0000 7fd896c67700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f8a9c0 0x55c553f76c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42494/0 (socket says 172.21.15.70:42494) 2024-06-04T04:58:46.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.965+0000 7fd8a2f5fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd87400d600 con 0x55c553f8a9c0 2024-06-04T04:58:46.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.965+0000 7fd8a2f5fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd87400f140 con 0x55c553f7c860 2024-06-04T04:58:46.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.965+0000 7fd8a2f5fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd87400dcd0 con 0x55c553f7d210 2024-06-04T04:58:46.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.965+0000 7fd896466700 1 -- 172.21.15.70:0/894668918 learned_addr learned my addr 172.21.15.70:0/894668918 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:46.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.966+0000 7fd896c67700 1 -- 172.21.15.70:0/894668918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f7d210 msgr2=0x55c553f906e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:46.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.966+0000 7fd895c65700 1 --2- 172.21.15.70:0/894668918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f7d210 0x55c553f906e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:46.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.966+0000 7fd896c67700 1 --2- 172.21.15.70:0/894668918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f7d210 0x55c553f906e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.966+0000 7fd896c67700 1 -- 172.21.15.70:0/894668918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 msgr2=0x55c553f889a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:46.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.966+0000 7fd896c67700 1 --2- 172.21.15.70:0/894668918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 0x55c553f889a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.966+0000 7fd896c67700 1 -- 172.21.15.70:0/894668918 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c553f8d1e0 con 0x55c553f8a9c0 2024-06-04T04:58:46.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.966+0000 7fd896466700 1 --2- 172.21.15.70:0/894668918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 0x55c553f889a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:46.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.966+0000 7fd896c67700 1 --2- 172.21.15.70:0/894668918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f8a9c0 0x55c553f76c90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd888014770 tx=0x7fd888004ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134a6aeaa8393cb9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:46.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.966+0000 7fd87bfff700 1 -- 172.21.15.70:0/894668918 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd888014aa0 con 0x55c553f8a9c0 2024-06-04T04:58:46.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.966+0000 7fd87bfff700 1 -- 172.21.15.70:0/894668918 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd88800fc60 con 0x55c553f8a9c0 2024-06-04T04:58:46.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.966+0000 7fd87bfff700 1 -- 172.21.15.70:0/894668918 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd888014ce0 con 0x55c553f8a9c0 2024-06-04T04:58:46.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.967+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/894668918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f8a9c0 msgr2=0x55c553f76c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:46.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.967+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/894668918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f8a9c0 0x55c553f76c90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd888014770 tx=0x7fd888004ca0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.967+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/894668918 shutdown_connections 2024-06-04T04:58:46.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.967+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/894668918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f8a9c0 0x55c553f76c90 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.967+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/894668918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f7c860 0x55c553f889a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.967+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/894668918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f7d210 0x55c553f906e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.967+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/894668918 >> 172.21.15.70:0/894668918 conn(0x55c553eaa080 msgr2=0x55c553f88290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:46.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.967+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/894668918 shutdown_connections 2024-06-04T04:58:46.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.968+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/894668918 wait complete. 2024-06-04T04:58:46.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.968+0000 7fd8a2f5fb80 1 -- start start 2024-06-04T04:58:46.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.968+0000 7fd8a2f5fb80 1 Processor -- start 2024-06-04T04:58:46.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.968+0000 7fd8a2f5fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f7c860 0x55c553f96d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.968+0000 7fd8a2f5fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f7d210 0x55c553f972b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.968+0000 7fd8a2f5fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f8a9c0 0x55c553fa57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.968+0000 7fd8a2f5fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd87400ea70 con 0x55c553f7c860 2024-06-04T04:58:46.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.968+0000 7fd8a2f5fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd87400e3a0 con 0x55c553f8a9c0 2024-06-04T04:58:46.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.968+0000 7fd8a2f5fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd87400b5a0 con 0x55c553f7d210 2024-06-04T04:58:46.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.968+0000 7fd8a2f5fb80 10 client.0 _ll_register_callbacks cb 0x55c553e66920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:46.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.968+0000 7fd896466700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f7c860 0x55c553f96d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:46.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.969+0000 7fd896466700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f7c860 0x55c553f96d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42508/0 (socket says 172.21.15.70:42508) 2024-06-04T04:58:46.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.969+0000 7fd896466700 1 -- 172.21.15.70:0/2765242626 learned_addr learned my addr 172.21.15.70:0/2765242626 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:46.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.969+0000 7fd895c65700 1 --2- 172.21.15.70:0/2765242626 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f7d210 0x55c553f972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:46.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.969+0000 7fd896466700 1 -- 172.21.15.70:0/2765242626 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f7d210 msgr2=0x55c553f972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:46.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.969+0000 7fd896466700 1 --2- 172.21.15.70:0/2765242626 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f7d210 0x55c553f972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.969+0000 7fd896466700 1 -- 172.21.15.70:0/2765242626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f8a9c0 msgr2=0x55c553fa57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:46.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.969+0000 7fd896466700 1 --2- 172.21.15.70:0/2765242626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f8a9c0 0x55c553fa57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.969+0000 7fd896466700 1 -- 172.21.15.70:0/2765242626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c553fa5f90 con 0x55c553f7c860 2024-06-04T04:58:46.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.969+0000 7fd896466700 1 --2- 172.21.15.70:0/2765242626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f7c860 0x55c553f96d70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00dbf0 tx=0x7fd88c0058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42eec4400772c5c1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:46.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.969+0000 7fd879ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:46.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.969+0000 7fd879ffb700 1 -- 172.21.15.70:0/2765242626 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88c028300 con 0x55c553f7c860 2024-06-04T04:58:46.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.969+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2765242626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd88800fc60 con 0x55c553f7c860 2024-06-04T04:58:46.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.969+0000 7fd879ffb700 1 -- 172.21.15.70:0/2765242626 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fd88c0284a0 con 0x55c553f7c860 2024-06-04T04:58:46.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.970+0000 7fd879ffb700 1 -- 172.21.15.70:0/2765242626 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd88c0286a0 con 0x55c553f7c860 2024-06-04T04:58:46.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.970+0000 7fd8a2f5fb80 10 client.15909 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:46.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.970+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2765242626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c554008b10 con 0x55c553f7c860 2024-06-04T04:58:46.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.970+0000 7fd879ffb700 1 -- 172.21.15.70:0/2765242626 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd88800fc60 con 0x55c553f7c860 2024-06-04T04:58:46.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.970+0000 7fd879ffb700 1 -- 172.21.15.70:0/2765242626 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fd88c030a20 con 0x55c553f7c860 2024-06-04T04:58:46.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.970+0000 7fd8a2f5fb80 10 client.15909 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:46.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.970+0000 7fd8a2f5fb80 10 client.15909 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T04:58:46.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.970+0000 7fd8a2f5fb80 20 client.15909 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:46.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.970+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/2765242626 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55c554009230 0x55c55400b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:46.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.970+0000 7fd8a2f5fb80 4 client.15909 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T04:58:46.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.970+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2765242626 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c553f489e0 con 0x55c554009230 2024-06-04T04:58:46.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.970+0000 7fd895c65700 1 --2- 172.21.15.70:0/2765242626 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55c554009230 0x55c55400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:46.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.971+0000 7fd895c65700 1 --2- 172.21.15.70:0/2765242626 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55c554009230 0x55c55400b670 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=770d7cd350c81200 server_cookie=dbe44fcb64d7a55e in_seq=0 out_seq=0 2024-06-04T04:58:46.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.971+0000 7fd879ffb700 10 client.15909 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T04:58:46.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.971+0000 7fd879ffb700 1 -- 172.21.15.70:0/2765242626 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c553f489e0 con 0x55c554009230 2024-06-04T04:58:46.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.971+0000 7fd879ffb700 10 client.15909 handle_command_reply: tid=0 2024-06-04T04:58:46.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.971+0000 7fd8a2f5fb80 1 client.15909 shutdown 2024-06-04T04:58:46.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.971+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2765242626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f7c860 msgr2=0x55c553f96d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:46.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.971+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/2765242626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f7c860 0x55c553f96d70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00dbf0 tx=0x7fd88c0058e0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.971+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2765242626 shutdown_connections 2024-06-04T04:58:46.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.971+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/2765242626 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55c554009230 0x55c55400b670 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.971+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/2765242626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c553f8a9c0 0x55c553fa57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.971+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/2765242626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c553f7c860 0x55c553f96d70 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.971+0000 7fd8a2f5fb80 1 --2- 172.21.15.70:0/2765242626 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c553f7d210 0x55c553f972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.971+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2765242626 >> 172.21.15.70:0/2765242626 conn(0x55c553eaa080 msgr2=0x55c553f7b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:46.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.972+0000 7fd879ffb700 0 client.15909 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T04:58:46.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.972+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2765242626 shutdown_connections 2024-06-04T04:58:46.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.972+0000 7fd8a2f5fb80 1 -- 172.21.15.70:0/2765242626 wait complete. 2024-06-04T04:58:46.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.972+0000 7fd8a2f5fb80 20 client.15909 trim_cache size 0 max 16384 2024-06-04T04:58:46.979 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T04:58:46.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.972+0000 7fd8986ca700 1 -- 172.21.15.70:0/4046522430 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd87c0b2a70 msgr2=0x7fd87c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:46.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.972+0000 7fd8986ca700 1 --2- 172.21.15.70:0/4046522430 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd87c0b2a70 0x7fd87c0b4f30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd88c005fd0 tx=0x7fd88c005f00 comp rx=0 tx=0).stop 2024-06-04T04:58:46.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.972+0000 7fd8986ca700 1 -- 172.21.15.70:0/4046522430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd890103600 msgr2=0x7fd89019ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:46.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.972+0000 7fd8986ca700 1 --2- 172.21.15.70:0/4046522430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd890103600 0x7fd89019ec20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd880006010 tx=0x7fd88000bfc0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.973+0000 7fd8986ca700 1 -- 172.21.15.70:0/4046522430 shutdown_connections 2024-06-04T04:58:46.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.973+0000 7fd8986ca700 1 --2- 172.21.15.70:0/4046522430 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd87c0b2a70 0x7fd87c0b4f30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.973+0000 7fd8986ca700 1 --2- 172.21.15.70:0/4046522430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8901084c0 0x7fd890199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.973+0000 7fd8986ca700 1 --2- 172.21.15.70:0/4046522430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd890102c40 0x7fd89019e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.973+0000 7fd8986ca700 1 --2- 172.21.15.70:0/4046522430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd890103600 0x7fd89019ec20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:46.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.973+0000 7fd8986ca700 1 -- 172.21.15.70:0/4046522430 >> 172.21.15.70:0/4046522430 conn(0x7fd890078ab0 msgr2=0x7fd8900ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:46.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.973+0000 7fd8986ca700 1 -- 172.21.15.70:0/4046522430 shutdown_connections 2024-06-04T04:58:46.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:46.973+0000 7fd8986ca700 1 -- 172.21.15.70:0/4046522430 wait complete. 2024-06-04T04:58:47.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:47 smithi112 ceph-mon[36453]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 13 op/s 2024-06-04T04:58:47.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:47 smithi098 ceph-mon[35569]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 13 op/s 2024-06-04T04:58:47.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:47 smithi070 ceph-mon[32631]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 13 op/s 2024-06-04T04:58:47.727 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T04:58:48.071 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:49.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:49 smithi112 ceph-mon[36453]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-06-04T04:58:49.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:49 smithi098 ceph-mon[35569]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-06-04T04:58:49.547 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:49 smithi070 ceph-mon[32631]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-06-04T04:58:49.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.964+0000 7fcf06338700 1 -- 172.21.15.70:0/1530437600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf00102c40 msgr2=0x7fcf001030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:49.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.964+0000 7fcf06338700 1 --2- 172.21.15.70:0/1530437600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf00102c40 0x7fcf001030c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fcef4009990 tx=0x7fcef4009ca0 comp rx=0 tx=0).stop 2024-06-04T04:58:49.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.964+0000 7fcf06338700 1 -- 172.21.15.70:0/1530437600 shutdown_connections 2024-06-04T04:58:49.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.964+0000 7fcf06338700 1 --2- 172.21.15.70:0/1530437600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf00103600 0x7fcf0010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:49.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.964+0000 7fcf06338700 1 --2- 172.21.15.70:0/1530437600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf001084c0 0x7fcf001088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:49.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.964+0000 7fcf06338700 1 --2- 172.21.15.70:0/1530437600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf00102c40 0x7fcf001030c0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:49.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.964+0000 7fcf06338700 1 -- 172.21.15.70:0/1530437600 >> 172.21.15.70:0/1530437600 conn(0x7fcf00078ab0 msgr2=0x7fcf000ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:49.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.964+0000 7fcf06338700 1 -- 172.21.15.70:0/1530437600 shutdown_connections 2024-06-04T04:58:49.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.964+0000 7fcf06338700 1 -- 172.21.15.70:0/1530437600 wait complete. 2024-06-04T04:58:49.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.965+0000 7fcf06338700 1 Processor -- start 2024-06-04T04:58:49.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.965+0000 7fcf06338700 1 -- start start 2024-06-04T04:58:49.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.965+0000 7fcf06338700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf00102c40 0x7fcf0019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:49.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.965+0000 7fcf06338700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf00103600 0x7fcf0019fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:49.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.965+0000 7fcf06338700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf001084c0 0x7fcf00199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:49.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.965+0000 7fcf06338700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcf001124a0 con 0x7fcf00102c40 2024-06-04T04:58:49.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.965+0000 7fcf06338700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcf00112320 con 0x7fcf00103600 2024-06-04T04:58:49.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.965+0000 7fcf06338700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcf00112620 con 0x7fcf001084c0 2024-06-04T04:58:49.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.965+0000 7fcefffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf00102c40 0x7fcf0019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:49.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.965+0000 7fcefffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf00102c40 0x7fcf0019f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42524/0 (socket says 172.21.15.70:42524) 2024-06-04T04:58:49.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.965+0000 7fcefffff700 1 -- 172.21.15.70:0/1838327833 learned_addr learned my addr 172.21.15.70:0/1838327833 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:49.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.966+0000 7fcf048d5700 1 --2- 172.21.15.70:0/1838327833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf001084c0 0x7fcf00199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:49.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.966+0000 7fceff7fe700 1 --2- 172.21.15.70:0/1838327833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf00103600 0x7fcf0019fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:49.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.966+0000 7fcefffff700 1 -- 172.21.15.70:0/1838327833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf001084c0 msgr2=0x7fcf00199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:49.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.966+0000 7fcefffff700 1 --2- 172.21.15.70:0/1838327833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf001084c0 0x7fcf00199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:49.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.966+0000 7fcefffff700 1 -- 172.21.15.70:0/1838327833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf00103600 msgr2=0x7fcf0019fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:49.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.966+0000 7fcefffff700 1 --2- 172.21.15.70:0/1838327833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf00103600 0x7fcf0019fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:49.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.966+0000 7fcefffff700 1 -- 172.21.15.70:0/1838327833 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0019a310 con 0x7fcf00102c40 2024-06-04T04:58:49.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.966+0000 7fcefffff700 1 --2- 172.21.15.70:0/1838327833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf00102c40 0x7fcf0019f720 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fcef000d980 tx=0x7fcef000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:49.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.967+0000 7fcefd7fa700 1 -- 172.21.15.70:0/1838327833 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef0016070 con 0x7fcf00102c40 2024-06-04T04:58:49.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.967+0000 7fcefd7fa700 1 -- 172.21.15.70:0/1838327833 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcef000fd20 con 0x7fcf00102c40 2024-06-04T04:58:49.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.967+0000 7fcefd7fa700 1 -- 172.21.15.70:0/1838327833 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef0010e30 con 0x7fcf00102c40 2024-06-04T04:58:49.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.967+0000 7fcf06338700 1 -- 172.21.15.70:0/1838327833 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf0019a600 con 0x7fcf00102c40 2024-06-04T04:58:49.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.967+0000 7fcf06338700 1 -- 172.21.15.70:0/1838327833 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf001a65f0 con 0x7fcf00102c40 2024-06-04T04:58:49.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.969+0000 7fcefd7fa700 1 -- 172.21.15.70:0/1838327833 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fcef0019070 con 0x7fcf00102c40 2024-06-04T04:58:49.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.970+0000 7fcefd7fa700 1 --2- 172.21.15.70:0/1838327833 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcee80b2a70 0x7fcee80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:49.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.970+0000 7fcefd7fa700 1 -- 172.21.15.70:0/1838327833 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fcef00e23b0 con 0x7fcf00102c40 2024-06-04T04:58:49.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.970+0000 7fceff7fe700 1 --2- 172.21.15.70:0/1838327833 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcee80b2a70 0x7fcee80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:49.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.971+0000 7fceff7fe700 1 --2- 172.21.15.70:0/1838327833 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcee80b2a70 0x7fcee80b4f30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcef400b5c0 tx=0x7fcef4005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:49.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.971+0000 7fcf10bcdb80 1 Processor -- start 2024-06-04T04:58:49.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.972+0000 7fcf10bcdb80 1 -- start start 2024-06-04T04:58:49.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.972+0000 7fcf10bcdb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf36860 0x557acdf36c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:49.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.972+0000 7fcf10bcdb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf3a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:49.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.972+0000 7fcf10bcdb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf3af60 0x557acdf3b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:49.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.972+0000 7fcf10bcdb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557acde95c90 con 0x557acdf37210 2024-06-04T04:58:49.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.972+0000 7fcf10bcdb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557acde85830 con 0x557acdf3af60 2024-06-04T04:58:49.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.972+0000 7fcf10bcdb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557acde89870 con 0x557acdf36860 2024-06-04T04:58:49.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.972+0000 7fceff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf3a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:49.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.972+0000 7fceff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf3a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42540/0 (socket says 172.21.15.70:42540) 2024-06-04T04:58:49.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.972+0000 7fceff7fe700 1 -- 172.21.15.70:0/3350928908 learned_addr learned my addr 172.21.15.70:0/3350928908 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:49.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.972+0000 7fcf048d5700 1 --2- 172.21.15.70:0/3350928908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf3af60 0x557acdf3b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:49.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.972+0000 7fceff7fe700 1 -- 172.21.15.70:0/3350928908 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf36860 msgr2=0x557acdf36c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:49.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.972+0000 7fceff7fe700 1 --2- 172.21.15.70:0/3350928908 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf36860 0x557acdf36c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:49.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.972+0000 7fceff7fe700 1 -- 172.21.15.70:0/3350928908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf3af60 msgr2=0x557acdf3b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:49.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.972+0000 7fceff7fe700 1 --2- 172.21.15.70:0/3350928908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf3af60 0x557acdf3b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:49.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.972+0000 7fceff7fe700 1 -- 172.21.15.70:0/3350928908 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557acddd58d0 con 0x557acdf37210 2024-06-04T04:58:49.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.973+0000 7fceff7fe700 1 --2- 172.21.15.70:0/3350928908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf3a940 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fcef4015000 tx=0x7fcef401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c7649fc42100486 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:49.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.973+0000 7fcee67fc700 1 -- 172.21.15.70:0/3350928908 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef40056e0 con 0x557acdf37210 2024-06-04T04:58:49.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.973+0000 7fcee67fc700 1 -- 172.21.15.70:0/3350928908 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcef400bc60 con 0x557acdf37210 2024-06-04T04:58:49.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.973+0000 7fcee67fc700 1 -- 172.21.15.70:0/3350928908 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef400f8c0 con 0x557acdf37210 2024-06-04T04:58:49.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.974+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/3350928908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 msgr2=0x557acdf3a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:49.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.974+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/3350928908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf3a940 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fcef4015000 tx=0x7fcef401f000 comp rx=0 tx=0).stop 2024-06-04T04:58:49.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.974+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/3350928908 shutdown_connections 2024-06-04T04:58:49.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.974+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/3350928908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf3af60 0x557acdf3b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:49.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.974+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/3350928908 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf36860 0x557acdf36c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:49.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.974+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/3350928908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf3a940 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:49.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.974+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/3350928908 >> 172.21.15.70:0/3350928908 conn(0x557acde64080 msgr2=0x557acdf30e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:49.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.974+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/3350928908 shutdown_connections 2024-06-04T04:58:49.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.975+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/3350928908 wait complete. 2024-06-04T04:58:49.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.975+0000 7fcf10bcdb80 1 -- start start 2024-06-04T04:58:49.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.976+0000 7fcf10bcdb80 1 Processor -- start 2024-06-04T04:58:49.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.976+0000 7fcf10bcdb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf36860 0x557acdf42560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:49.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.976+0000 7fcf10bcdb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf3f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:49.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.976+0000 7fcf10bcdb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf3af60 0x557acdf3fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.976+0000 7fceff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf3f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.976+0000 7fcf10bcdb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557acdd8bfa0 con 0x557acdf37210 2024-06-04T04:58:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.976+0000 7fcf10bcdb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557acde277c0 con 0x557acdf3af60 2024-06-04T04:58:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.976+0000 7fcf10bcdb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557acdf53f80 con 0x557acdf36860 2024-06-04T04:58:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.976+0000 7fceff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf3f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42548/0 (socket says 172.21.15.70:42548) 2024-06-04T04:58:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.976+0000 7fceff7fe700 1 -- 172.21.15.70:0/3443525130 learned_addr learned my addr 172.21.15.70:0/3443525130 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.976+0000 7fcf10bcdb80 10 client.0 _ll_register_callbacks cb 0x557acde20920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.977+0000 7fceff7fe700 1 -- 172.21.15.70:0/3443525130 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf36860 msgr2=0x557acdf42560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.977+0000 7fceff7fe700 1 --2- 172.21.15.70:0/3443525130 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf36860 0x557acdf42560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.977+0000 7fceff7fe700 1 -- 172.21.15.70:0/3443525130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf3af60 msgr2=0x557acdf3fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.977+0000 7fcf048d5700 1 --2- 172.21.15.70:0/3443525130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf3af60 0x557acdf3fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.977+0000 7fceff7fe700 1 --2- 172.21.15.70:0/3443525130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf3af60 0x557acdf3fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.977+0000 7fceff7fe700 1 -- 172.21.15.70:0/3443525130 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557acde24690 con 0x557acdf37210 2024-06-04T04:58:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.977+0000 7fcf048d5700 1 --2- 172.21.15.70:0/3443525130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf3af60 0x557acdf3fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:49.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.977+0000 7fceff7fe700 1 --2- 172.21.15.70:0/3443525130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf3f880 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fcef4006010 tx=0x7fcef400bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b01128ddad48c0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:49.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.977+0000 7fcedf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:49.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.977+0000 7fcedf7fe700 1 -- 172.21.15.70:0/3443525130 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef403a060 con 0x557acdf37210 2024-06-04T04:58:49.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.978+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/3443525130 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcef400bc60 con 0x557acdf37210 2024-06-04T04:58:49.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.978+0000 7fcedf7fe700 1 -- 172.21.15.70:0/3443525130 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fcef400be10 con 0x557acdf37210 2024-06-04T04:58:49.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.978+0000 7fcedf7fe700 1 -- 172.21.15.70:0/3443525130 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcef40056e0 con 0x557acdf37210 2024-06-04T04:58:49.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.978+0000 7fcf10bcdb80 10 client.15921 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:49.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.978+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/3443525130 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557acdf5e660 con 0x557acdf37210 2024-06-04T04:58:49.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.979+0000 7fcedf7fe700 1 -- 172.21.15.70:0/3443525130 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcef400f670 con 0x557acdf37210 2024-06-04T04:58:49.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.979+0000 7fcedf7fe700 1 -- 172.21.15.70:0/3443525130 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fcef4022410 con 0x557acdf37210 2024-06-04T04:58:49.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.979+0000 7fcf10bcdb80 10 client.15921 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:49.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.979+0000 7fcf10bcdb80 10 client.15921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.979+0000 7fcf10bcdb80 20 client.15921 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.979+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/3443525130 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557acdf60b30 0x557acdf62f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.979+0000 7fcf10bcdb80 4 client.15921 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.979+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/3443525130 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fcef400bc60 con 0x557acdf60b30 2024-06-04T04:58:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.980+0000 7fcefffff700 1 --2- 172.21.15.70:0/3443525130 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557acdf60b30 0x557acdf62f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.980+0000 7fcefffff700 1 --2- 172.21.15.70:0/3443525130 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557acdf60b30 0x557acdf62f20 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de6457a9a1e0c9c0 server_cookie=e105fca89c08fb29 in_seq=0 out_seq=0 2024-06-04T04:58:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.980+0000 7fcedf7fe700 10 client.15921 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.981+0000 7fcedf7fe700 1 -- 172.21.15.70:0/3443525130 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fcef400bc60 con 0x557acdf60b30 2024-06-04T04:58:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.981+0000 7fcedf7fe700 10 client.15921 handle_command_reply: tid=0 2024-06-04T04:58:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.981+0000 7fcf10bcdb80 1 client.15921 shutdown 2024-06-04T04:58:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.981+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/3443525130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 msgr2=0x557acdf3f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.981+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/3443525130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf3f880 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fcef4006010 tx=0x7fcef400bb30 comp rx=0 tx=0).stop 2024-06-04T04:58:49.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.982+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/3443525130 shutdown_connections 2024-06-04T04:58:49.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.982+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/3443525130 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557acdf60b30 0x557acdf62f20 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:49.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.982+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/3443525130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf3af60 0x557acdf3fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:49.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.982+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/3443525130 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf36860 0x557acdf42560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:49.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.982+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/3443525130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf3f880 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:49.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.982+0000 7fcedf7fe700 0 client.15921 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:49.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.982+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/3443525130 >> 172.21.15.70:0/3443525130 conn(0x557acde64080 msgr2=0x557acdf3c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:49.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.982+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/3443525130 shutdown_connections 2024-06-04T04:58:49.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.982+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/3443525130 wait complete. 2024-06-04T04:58:49.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:49.982+0000 7fcf10bcdb80 20 client.15921 trim_cache size 0 max 16384 2024-06-04T04:58:50.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.011+0000 7fcf10bcdb80 1 Processor -- start 2024-06-04T04:58:50.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.011+0000 7fcf10bcdb80 1 -- start start 2024-06-04T04:58:50.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.011+0000 7fcf10bcdb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf36860 0x557acdf3a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:50.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.011+0000 7fcf10bcdb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf37210 0x557acdf30c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:50.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.012+0000 7fcf10bcdb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf3af60 0x557acdf40f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:50.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.012+0000 7fcf10bcdb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcf00074420 con 0x557acdf36860 2024-06-04T04:58:50.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.012+0000 7fcf10bcdb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcf001a7070 con 0x557acdf37210 2024-06-04T04:58:50.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.012+0000 7fcf10bcdb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcf00074af0 con 0x557acdf3af60 2024-06-04T04:58:50.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.012+0000 7fcf048d5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf3af60 0x557acdf40f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:50.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.012+0000 7fcf048d5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf3af60 0x557acdf40f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:50750/0 (socket says 172.21.15.70:50750) 2024-06-04T04:58:50.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.012+0000 7fceff7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf37210 0x557acdf30c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:50.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.012+0000 7fcf048d5700 1 -- 172.21.15.70:0/157899570 learned_addr learned my addr 172.21.15.70:0/157899570 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:50.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.012+0000 7fceff7fe700 1 -- 172.21.15.70:0/157899570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf3af60 msgr2=0x557acdf40f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:50.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.012+0000 7fceff7fe700 1 --2- 172.21.15.70:0/157899570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf3af60 0x557acdf40f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:50.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.012+0000 7fceff7fe700 1 -- 172.21.15.70:0/157899570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf36860 msgr2=0x557acdf3a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:50.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.012+0000 7fceff7fe700 1 --2- 172.21.15.70:0/157899570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf36860 0x557acdf3a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:50.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.012+0000 7fceff7fe700 1 -- 172.21.15.70:0/157899570 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557acdf403c0 con 0x557acdf37210 2024-06-04T04:58:50.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.012+0000 7fcf048d5700 1 --2- 172.21.15.70:0/157899570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf3af60 0x557acdf40f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:50.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.012+0000 7fceff7fe700 1 --2- 172.21.15.70:0/157899570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf37210 0x557acdf30c90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcef40098e0 tx=0x7fcef4010f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2416e45825dcbc server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:50.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.013+0000 7fcee5ffb700 1 -- 172.21.15.70:0/157899570 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef403a070 con 0x557acdf37210 2024-06-04T04:58:50.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.013+0000 7fcee5ffb700 1 -- 172.21.15.70:0/157899570 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcef400be70 con 0x557acdf37210 2024-06-04T04:58:50.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.013+0000 7fcee5ffb700 1 -- 172.21.15.70:0/157899570 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4022660 con 0x557acdf37210 2024-06-04T04:58:50.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.014+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/157899570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf37210 msgr2=0x557acdf30c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:50.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.014+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/157899570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf37210 0x557acdf30c90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcef40098e0 tx=0x7fcef4010f00 comp rx=0 tx=0).stop 2024-06-04T04:58:50.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.014+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/157899570 shutdown_connections 2024-06-04T04:58:50.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.014+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/157899570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf3af60 0x557acdf40f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:50.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.014+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/157899570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf36860 0x557acdf3a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:50.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.014+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/157899570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf37210 0x557acdf30c90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:50.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.014+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/157899570 >> 172.21.15.70:0/157899570 conn(0x557acde64080 msgr2=0x557acdf35f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:50.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.014+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/157899570 shutdown_connections 2024-06-04T04:58:50.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.014+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/157899570 wait complete. 2024-06-04T04:58:50.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.015+0000 7fcf10bcdb80 1 -- start start 2024-06-04T04:58:50.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.015+0000 7fcf10bcdb80 1 Processor -- start 2024-06-04T04:58:50.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.016+0000 7fcf10bcdb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf36860 0x557acdf61310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:50.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.016+0000 7fcf10bcdb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf61850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:50.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.016+0000 7fcf10bcdb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf3af60 0x557acdfcbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:50.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.016+0000 7fcf10bcdb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcf001a69a0 con 0x557acdf37210 2024-06-04T04:58:50.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.016+0000 7fcf10bcdb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcf000751c0 con 0x557acdf3af60 2024-06-04T04:58:50.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.016+0000 7fceff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf61850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:50.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.016+0000 7fceff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf61850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42578/0 (socket says 172.21.15.70:42578) 2024-06-04T04:58:50.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.016+0000 7fceff7fe700 1 -- 172.21.15.70:0/4097944225 learned_addr learned my addr 172.21.15.70:0/4097944225 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:50.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.016+0000 7fcefffff700 1 --2- 172.21.15.70:0/4097944225 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf36860 0x557acdf61310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:50.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.016+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/4097944225 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcf000723c0 con 0x557acdf36860 2024-06-04T04:58:50.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.016+0000 7fcf10bcdb80 10 client.0 _ll_register_callbacks cb 0x557acde20920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:50.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.017+0000 7fcf048d5700 1 --2- 172.21.15.70:0/4097944225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf3af60 0x557acdfcbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:50.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.017+0000 7fcefffff700 1 -- 172.21.15.70:0/4097944225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf3af60 msgr2=0x557acdfcbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:50.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.017+0000 7fcefffff700 1 --2- 172.21.15.70:0/4097944225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf3af60 0x557acdfcbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:50.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.017+0000 7fcefffff700 1 -- 172.21.15.70:0/4097944225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 msgr2=0x557acdf61850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:50.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.017+0000 7fcefffff700 1 --2- 172.21.15.70:0/4097944225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf61850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:50.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.017+0000 7fcefffff700 1 -- 172.21.15.70:0/4097944225 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557acdfcc600 con 0x557acdf36860 2024-06-04T04:58:50.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.017+0000 7fceff7fe700 1 --2- 172.21.15.70:0/4097944225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf61850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:50.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.017+0000 7fcefffff700 1 --2- 172.21.15.70:0/4097944225 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf36860 0x557acdf61310 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcef00dfaa0 tx=0x7fcef0010790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93db62b480ef48e2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:50.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.017+0000 7fcedf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:58:50.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.017+0000 7fcedf7fe700 1 -- 172.21.15.70:0/4097944225 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef0020040 con 0x557acdf36860 2024-06-04T04:58:50.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.017+0000 7fcf048d5700 1 --2- 172.21.15.70:0/4097944225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf3af60 0x557acdfcbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:50.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.017+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/4097944225 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcef400be70 con 0x557acdf36860 2024-06-04T04:58:50.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.017+0000 7fcedf7fe700 1 -- 172.21.15.70:0/4097944225 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fcef00dfd20 con 0x557acdf36860 2024-06-04T04:58:50.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.017+0000 7fcedf7fe700 1 -- 172.21.15.70:0/4097944225 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcef00a9630 con 0x557acdf36860 2024-06-04T04:58:50.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.018+0000 7fcf10bcdb80 10 client.25414 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:50.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.018+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/4097944225 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557acdfd39c0 con 0x557acdf36860 2024-06-04T04:58:50.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.018+0000 7fcedf7fe700 1 -- 172.21.15.70:0/4097944225 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcef400be70 con 0x557acdf36860 2024-06-04T04:58:50.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.018+0000 7fcedf7fe700 1 -- 172.21.15.70:0/4097944225 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fcef00b2000 con 0x557acdf36860 2024-06-04T04:58:50.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.018+0000 7fcf10bcdb80 10 client.25414 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:50.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.018+0000 7fcf10bcdb80 10 client.25414 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:50.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.018+0000 7fcf10bcdb80 20 client.25414 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:50.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.018+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/4097944225 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557acdfd4030 0x557acdfd6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:50.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.018+0000 7fcf10bcdb80 4 client.25414 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T04:58:50.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.018+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/4097944225 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557acdf029e0 con 0x557acdfd4030 2024-06-04T04:58:50.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.018+0000 7fceff7fe700 1 --2- 172.21.15.70:0/4097944225 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557acdfd4030 0x557acdfd6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:50.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.019+0000 7fcedf7fe700 10 client.25414 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:50.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.019+0000 7fceff7fe700 1 --2- 172.21.15.70:0/4097944225 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557acdfd4030 0x557acdfd6450 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdbd1b519da7ab85 server_cookie=ce417b682117481e in_seq=0 out_seq=0 2024-06-04T04:58:50.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.019+0000 7fcedf7fe700 1 -- 172.21.15.70:0/4097944225 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557acdf029e0 con 0x557acdfd4030 2024-06-04T04:58:50.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.019+0000 7fcedf7fe700 10 client.25414 handle_command_reply: tid=0 2024-06-04T04:58:50.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.019+0000 7fcf10bcdb80 1 client.25414 shutdown 2024-06-04T04:58:50.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.019+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/4097944225 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf36860 msgr2=0x557acdf61310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:50.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.019+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/4097944225 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf36860 0x557acdf61310 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcef00dfaa0 tx=0x7fcef0010790 comp rx=0 tx=0).stop 2024-06-04T04:58:50.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.019+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/4097944225 shutdown_connections 2024-06-04T04:58:50.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.019+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/4097944225 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557acdfd4030 0x557acdfd6450 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:50.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.019+0000 7fcedf7fe700 0 client.25414 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:50.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.019+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/4097944225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557acdf3af60 0x557acdfcbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:50.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.019+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/4097944225 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557acdf36860 0x557acdf61310 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:50.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.019+0000 7fcf10bcdb80 1 --2- 172.21.15.70:0/4097944225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557acdf37210 0x557acdf61850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:50.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.019+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/4097944225 >> 172.21.15.70:0/4097944225 conn(0x557acde64080 msgr2=0x557acdf35f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:50.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.019+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/4097944225 shutdown_connections 2024-06-04T04:58:50.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.019+0000 7fcf10bcdb80 1 -- 172.21.15.70:0/4097944225 wait complete. 2024-06-04T04:58:50.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.020+0000 7fcf10bcdb80 20 client.25414 trim_cache size 0 max 16384 2024-06-04T04:58:50.025 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T04:58:50.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.020+0000 7fcf06338700 1 -- 172.21.15.70:0/1838327833 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcee80b2a70 msgr2=0x7fcee80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:50.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.020+0000 7fcf06338700 1 --2- 172.21.15.70:0/1838327833 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcee80b2a70 0x7fcee80b4f30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcef400b5c0 tx=0x7fcef4005960 comp rx=0 tx=0).stop 2024-06-04T04:58:50.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.020+0000 7fcf06338700 1 -- 172.21.15.70:0/1838327833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf00102c40 msgr2=0x7fcf0019f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:50.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.020+0000 7fcf06338700 1 --2- 172.21.15.70:0/1838327833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf00102c40 0x7fcf0019f720 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fcef000d980 tx=0x7fcef000dc90 comp rx=0 tx=0).stop 2024-06-04T04:58:50.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.021+0000 7fcf06338700 1 -- 172.21.15.70:0/1838327833 shutdown_connections 2024-06-04T04:58:50.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.021+0000 7fcf06338700 1 --2- 172.21.15.70:0/1838327833 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcee80b2a70 0x7fcee80b4f30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:50.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.021+0000 7fcf06338700 1 --2- 172.21.15.70:0/1838327833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf001084c0 0x7fcf00199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:50.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.021+0000 7fcf06338700 1 --2- 172.21.15.70:0/1838327833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf00102c40 0x7fcf0019f720 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:50.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.021+0000 7fcf06338700 1 --2- 172.21.15.70:0/1838327833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf00103600 0x7fcf0019fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:50.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.021+0000 7fcf06338700 1 -- 172.21.15.70:0/1838327833 >> 172.21.15.70:0/1838327833 conn(0x7fcf00078ab0 msgr2=0x7fcf000ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:50.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.021+0000 7fcf06338700 1 -- 172.21.15.70:0/1838327833 shutdown_connections 2024-06-04T04:58:50.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:50.021+0000 7fcf06338700 1 -- 172.21.15.70:0/1838327833 wait complete. 2024-06-04T04:58:51.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:51 smithi112 ceph-mon[36453]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-06-04T04:58:51.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:51 smithi098 ceph-mon[35569]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-06-04T04:58:51.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:51 smithi070 ceph-mon[32631]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-06-04T04:58:51.881 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T04:58:51.881 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T04:58:52.157 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:53.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:53 smithi112 ceph-mon[36453]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 13 op/s 2024-06-04T04:58:53.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:53 smithi098 ceph-mon[35569]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 13 op/s 2024-06-04T04:58:53.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:53 smithi070 ceph-mon[32631]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 13 op/s 2024-06-04T04:58:55.005 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:58:55.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:55 smithi070 ceph-mon[32631]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:58:55.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:55 smithi112 ceph-mon[36453]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:58:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.434+0000 7fb202c54640 1 -- 172.21.15.70:0/1039398745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1fc10d020 msgr2=0x7fb1fc10f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.434+0000 7fb202c54640 1 --2- 172.21.15.70:0/1039398745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1fc10d020 0x7fb1fc10f4c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4009ca0 tx=0x7fb1f402a9e0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.434+0000 7fb202c54640 1 -- 172.21.15.70:0/1039398745 shutdown_connections 2024-06-04T04:58:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.434+0000 7fb202c54640 1 --2- 172.21.15.70:0/1039398745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1fc104600 0x7fb1fc10cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.435+0000 7fb202c54640 1 --2- 172.21.15.70:0/1039398745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1fc10d020 0x7fb1fc10f4c0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.435+0000 7fb202c54640 1 --2- 172.21.15.70:0/1039398745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc103ce0 0x7fb1fc1040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.435+0000 7fb202c54640 1 -- 172.21.15.70:0/1039398745 >> 172.21.15.70:0/1039398745 conn(0x7fb1fc078ad0 msgr2=0x7fb1fc078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.435+0000 7fb202c54640 1 -- 172.21.15.70:0/1039398745 shutdown_connections 2024-06-04T04:58:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.435+0000 7fb202c54640 1 -- 172.21.15.70:0/1039398745 wait complete. 2024-06-04T04:58:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.435+0000 7fb202c54640 1 Processor -- start 2024-06-04T04:58:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.435+0000 7fb202c54640 1 -- start start 2024-06-04T04:58:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.436+0000 7fb202c54640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc103ce0 0x7fb1fc101930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.436+0000 7fb202c54640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1fc104600 0x7fb1fc101e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.436+0000 7fb202c54640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1fc10d020 0x7fb1fc1023b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.436+0000 7fb202c54640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc112bd0 con 0x7fb1fc10d020 2024-06-04T04:58:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.436+0000 7fb202c54640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc112a50 con 0x7fb1fc104600 2024-06-04T04:58:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.436+0000 7fb202c54640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc112d50 con 0x7fb1fc103ce0 2024-06-04T04:58:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.436+0000 7fb2011f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1fc10d020 0x7fb1fc1023b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.436+0000 7fb2011f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1fc10d020 0x7fb1fc1023b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42606/0 (socket says 172.21.15.70:42606) 2024-06-04T04:58:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.436+0000 7fb2011f1640 1 -- 172.21.15.70:0/581449136 learned_addr learned my addr 172.21.15.70:0/581449136 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.436+0000 7fb2009f0640 1 --2- 172.21.15.70:0/581449136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc103ce0 0x7fb1fc101930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.436+0000 7fb1fbfff640 1 --2- 172.21.15.70:0/581449136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1fc104600 0x7fb1fc101e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.436+0000 7fb2011f1640 1 -- 172.21.15.70:0/581449136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc103ce0 msgr2=0x7fb1fc101930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.436+0000 7fb2011f1640 1 --2- 172.21.15.70:0/581449136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc103ce0 0x7fb1fc101930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.436+0000 7fb2011f1640 1 -- 172.21.15.70:0/581449136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1fc104600 msgr2=0x7fb1fc101e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.436+0000 7fb2011f1640 1 --2- 172.21.15.70:0/581449136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1fc104600 0x7fb1fc101e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.436+0000 7fb2011f1640 1 -- 172.21.15.70:0/581449136 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1fc1a7590 con 0x7fb1fc10d020 2024-06-04T04:58:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.437+0000 7fb2011f1640 1 --2- 172.21.15.70:0/581449136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1fc10d020 0x7fb1fc1023b0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4005e30 tx=0x7fb1f4003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.437+0000 7fb1f9ffb640 1 -- 172.21.15.70:0/581449136 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4047070 con 0x7fb1fc10d020 2024-06-04T04:58:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.437+0000 7fb1f9ffb640 1 -- 172.21.15.70:0/581449136 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb1f402ee40 con 0x7fb1fc10d020 2024-06-04T04:58:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.437+0000 7fb202c54640 1 -- 172.21.15.70:0/581449136 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1fc1a7820 con 0x7fb1fc10d020 2024-06-04T04:58:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.437+0000 7fb202c54640 1 -- 172.21.15.70:0/581449136 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1fc1a7ca0 con 0x7fb1fc10d020 2024-06-04T04:58:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.438+0000 7fb1f9ffb640 1 -- 172.21.15.70:0/581449136 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4033050 con 0x7fb1fc10d020 2024-06-04T04:58:55.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.441+0000 7fb1f9ffb640 1 -- 172.21.15.70:0/581449136 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fb1f404a070 con 0x7fb1fc10d020 2024-06-04T04:58:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.441+0000 7fb20596c740 1 Processor -- start 2024-06-04T04:58:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.441+0000 7fb20596c740 1 -- start start 2024-06-04T04:58:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.441+0000 7fb20596c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca20a820 0x5646ca20ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.441+0000 7fb20596c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca20b140 0x5646ca2154c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.441+0000 7fb20596c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca215a00 0x5646ca217df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.441+0000 7fb20596c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5646ca17cc30 con 0x5646ca20a820 2024-06-04T04:58:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.441+0000 7fb20596c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5646ca16c7d0 con 0x5646ca20b140 2024-06-04T04:58:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.441+0000 7fb20596c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5646ca170810 con 0x5646ca215a00 2024-06-04T04:58:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.441+0000 7fb1fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca20a820 0x5646ca20ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.442+0000 7fb1fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca20a820 0x5646ca20ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42610/0 (socket says 172.21.15.70:42610) 2024-06-04T04:58:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.442+0000 7fb1fbfff640 1 -- 172.21.15.70:0/697430607 learned_addr learned my addr 172.21.15.70:0/697430607 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.442+0000 7fb1fbfff640 1 --2- 172.21.15.70:0/697430607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca215a00 0x5646ca217df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.442+0000 7fb2009f0640 1 --2- 172.21.15.70:0/697430607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca20b140 0x5646ca2154c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.442+0000 7fb1fbfff640 1 -- 172.21.15.70:0/697430607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca215a00 msgr2=0x5646ca217df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:55.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.442+0000 7fb1fbfff640 1 --2- 172.21.15.70:0/697430607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca215a00 0x5646ca217df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.442+0000 7fb1fbfff640 1 -- 172.21.15.70:0/697430607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca20b140 msgr2=0x5646ca2154c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.442+0000 7fb1fbfff640 1 --2- 172.21.15.70:0/697430607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca20b140 0x5646ca2154c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.442+0000 7fb1fbfff640 1 -- 172.21.15.70:0/697430607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646c9e81210 con 0x5646ca20a820 2024-06-04T04:58:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.442+0000 7fb2009f0640 1 --2- 172.21.15.70:0/697430607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca20b140 0x5646ca2154c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.442+0000 7fb1f9ffb640 1 --2- 172.21.15.70:0/581449136 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1ec0b2a90 0x7fb1ec0b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.442+0000 7fb1fbfff640 1 --2- 172.21.15.70:0/697430607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca20a820 0x5646ca20ac00 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80058c0 tx=0x7fb1e8005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c71d548310bcd399 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.442+0000 7fb2009f0640 1 --2- 172.21.15.70:0/581449136 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1ec0b2a90 0x7fb1ec0b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.442+0000 7fb1f9ffb640 1 -- 172.21.15.70:0/581449136 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb1f41030c0 con 0x7fb1fc10d020 2024-06-04T04:58:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.443+0000 7fb1e6ffd640 1 -- 172.21.15.70:0/697430607 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8004050 con 0x5646ca20a820 2024-06-04T04:58:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.443+0000 7fb1e6ffd640 1 -- 172.21.15.70:0/697430607 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb1e80035f0 con 0x5646ca20a820 2024-06-04T04:58:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.443+0000 7fb1e6ffd640 1 -- 172.21.15.70:0/697430607 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e800f740 con 0x5646ca20a820 2024-06-04T04:58:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.444+0000 7fb20596c740 1 -- 172.21.15.70:0/697430607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca20a820 msgr2=0x5646ca20ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.444+0000 7fb2009f0640 1 --2- 172.21.15.70:0/581449136 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1ec0b2a90 0x7fb1ec0b4f50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x5646ca20bdc0 tx=0x7fb1f000b410 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.444+0000 7fb20596c740 1 --2- 172.21.15.70:0/697430607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca20a820 0x5646ca20ac00 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80058c0 tx=0x7fb1e8005850 comp rx=0 tx=0).stop 2024-06-04T04:58:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.444+0000 7fb20596c740 1 -- 172.21.15.70:0/697430607 shutdown_connections 2024-06-04T04:58:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.444+0000 7fb20596c740 1 --2- 172.21.15.70:0/697430607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca20b140 0x5646ca2154c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.444+0000 7fb20596c740 1 --2- 172.21.15.70:0/697430607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca215a00 0x5646ca217df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.444+0000 7fb20596c740 1 --2- 172.21.15.70:0/697430607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca20a820 0x5646ca20ac00 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.444+0000 7fb20596c740 1 -- 172.21.15.70:0/697430607 >> 172.21.15.70:0/697430607 conn(0x5646ca206d30 msgr2=0x5646ca207110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.444+0000 7fb20596c740 1 -- 172.21.15.70:0/697430607 shutdown_connections 2024-06-04T04:58:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.444+0000 7fb20596c740 1 -- 172.21.15.70:0/697430607 wait complete. 2024-06-04T04:58:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.444+0000 7fb20596c740 1 -- start start 2024-06-04T04:58:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.445+0000 7fb20596c740 1 Processor -- start 2024-06-04T04:58:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.445+0000 7fb20596c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca20b140 0x5646ca212be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.445+0000 7fb20596c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca215a00 0x5646ca213120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.445+0000 7fb20596c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca213660 0x5646ca221950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.445+0000 7fb20596c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5646c9e52790 con 0x5646ca20b140 2024-06-04T04:58:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.445+0000 7fb20596c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5646ca128890 con 0x5646ca215a00 2024-06-04T04:58:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.445+0000 7fb20596c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5646ca2185d0 con 0x5646ca213660 2024-06-04T04:58:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.445+0000 7fb20596c740 10 client.0 _ll_register_callbacks cb 0x5646c9e80a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.445+0000 7fb2009f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca20b140 0x5646ca212be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.445+0000 7fb2009f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca20b140 0x5646ca212be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42620/0 (socket says 172.21.15.70:42620) 2024-06-04T04:58:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.445+0000 7fb2009f0640 1 -- 172.21.15.70:0/4149879194 learned_addr learned my addr 172.21.15.70:0/4149879194 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.445+0000 7fb1fbfff640 1 --2- 172.21.15.70:0/4149879194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca215a00 0x5646ca213120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.446+0000 7fb2011f1640 1 --2- 172.21.15.70:0/4149879194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca213660 0x5646ca221950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.446+0000 7fb2009f0640 1 -- 172.21.15.70:0/4149879194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca213660 msgr2=0x5646ca221950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.446+0000 7fb2009f0640 1 --2- 172.21.15.70:0/4149879194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca213660 0x5646ca221950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.446+0000 7fb2009f0640 1 -- 172.21.15.70:0/4149879194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca215a00 msgr2=0x5646ca213120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.446+0000 7fb2009f0640 1 --2- 172.21.15.70:0/4149879194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca215a00 0x5646ca213120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.446+0000 7fb2009f0640 1 -- 172.21.15.70:0/4149879194 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646c9f4d620 con 0x5646ca20b140 2024-06-04T04:58:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.446+0000 7fb1fbfff640 1 --2- 172.21.15.70:0/4149879194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca215a00 0x5646ca213120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:58:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.446+0000 7fb2009f0640 1 --2- 172.21.15.70:0/4149879194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca20b140 0x5646ca212be0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0010880 tx=0x7fb1f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e01c8dd564d8b696 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.446+0000 7fb1dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.446+0000 7fb1dbfff640 1 -- 172.21.15.70:0/4149879194 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0023070 con 0x5646ca20b140 2024-06-04T04:58:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.446+0000 7fb1dbfff640 1 -- 172.21.15.70:0/4149879194 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb1f000fac0 con 0x5646ca20b140 2024-06-04T04:58:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.446+0000 7fb20596c740 1 -- 172.21.15.70:0/4149879194 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1e80035f0 con 0x5646ca20b140 2024-06-04T04:58:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.447+0000 7fb1dbfff640 1 -- 172.21.15.70:0/4149879194 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb1f002f050 con 0x5646ca20b140 2024-06-04T04:58:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.447+0000 7fb20596c740 10 client.15936 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.447+0000 7fb20596c740 1 -- 172.21.15.70:0/4149879194 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5646ca224aa0 con 0x5646ca20b140 2024-06-04T04:58:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.447+0000 7fb1dbfff640 1 -- 172.21.15.70:0/4149879194 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb1e80035f0 con 0x5646ca20b140 2024-06-04T04:58:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.447+0000 7fb1dbfff640 1 -- 172.21.15.70:0/4149879194 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fb1f001fcc0 con 0x5646ca20b140 2024-06-04T04:58:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.447+0000 7fb20596c740 10 client.15936 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.447+0000 7fb20596c740 10 client.15936 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.447+0000 7fb20596c740 20 client.15936 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.447+0000 7fb20596c740 1 --2- 172.21.15.70:0/4149879194 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5646ca226430 0x5646ca2288d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.447+0000 7fb20596c740 4 client.15936 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.447+0000 7fb20596c740 1 -- 172.21.15.70:0/4149879194 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646ca17eb80 con 0x5646ca226430 2024-06-04T04:58:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.448+0000 7fb1fbfff640 1 --2- 172.21.15.70:0/4149879194 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5646ca226430 0x5646ca2288d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.448+0000 7fb1dbfff640 10 client.15936 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.448+0000 7fb1fbfff640 1 --2- 172.21.15.70:0/4149879194 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5646ca226430 0x5646ca2288d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8053bf7af3335815 server_cookie=8b32f017cd97c79e in_seq=0 out_seq=0 2024-06-04T04:58:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.449+0000 7fb1dbfff640 1 -- 172.21.15.70:0/4149879194 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5646ca17eb80 con 0x5646ca226430 2024-06-04T04:58:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.449+0000 7fb1dbfff640 10 client.15936 handle_command_reply: tid=0 2024-06-04T04:58:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.449+0000 7fb20596c740 1 client.15936 shutdown 2024-06-04T04:58:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.449+0000 7fb20596c740 1 -- 172.21.15.70:0/4149879194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca20b140 msgr2=0x5646ca212be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.449+0000 7fb20596c740 1 --2- 172.21.15.70:0/4149879194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca20b140 0x5646ca212be0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0010880 tx=0x7fb1f00046c0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.449+0000 7fb20596c740 1 -- 172.21.15.70:0/4149879194 shutdown_connections 2024-06-04T04:58:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.449+0000 7fb20596c740 1 --2- 172.21.15.70:0/4149879194 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5646ca226430 0x5646ca2288d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.449+0000 7fb20596c740 1 --2- 172.21.15.70:0/4149879194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca215a00 0x5646ca213120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.449+0000 7fb20596c740 1 --2- 172.21.15.70:0/4149879194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca213660 0x5646ca221950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.449+0000 7fb1dbfff640 0 client.15936 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.449+0000 7fb20596c740 1 --2- 172.21.15.70:0/4149879194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca20b140 0x5646ca212be0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.449+0000 7fb20596c740 1 -- 172.21.15.70:0/4149879194 >> 172.21.15.70:0/4149879194 conn(0x5646ca206d30 msgr2=0x5646ca20cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.449+0000 7fb20596c740 1 -- 172.21.15.70:0/4149879194 shutdown_connections 2024-06-04T04:58:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.449+0000 7fb20596c740 1 -- 172.21.15.70:0/4149879194 wait complete. 2024-06-04T04:58:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.450+0000 7fb20596c740 20 client.15936 trim_cache size 0 max 16384 2024-06-04T04:58:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.479+0000 7fb20596c740 1 Processor -- start 2024-06-04T04:58:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.479+0000 7fb20596c740 1 -- start start 2024-06-04T04:58:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.479+0000 7fb20596c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca20b140 0x5646ca20d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.479+0000 7fb20596c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca213660 0x5646ca2216c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.479+0000 7fb20596c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca215a00 0x5646ca22e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.479+0000 7fb20596c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e000e7d0 con 0x5646ca215a00 2024-06-04T04:58:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.480+0000 7fb20596c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e0010310 con 0x5646ca20b140 2024-06-04T04:58:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.480+0000 7fb20596c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e000eea0 con 0x5646ca213660 2024-06-04T04:58:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.480+0000 7fb2011f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca215a00 0x5646ca22e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.480+0000 7fb2011f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca215a00 0x5646ca22e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42634/0 (socket says 172.21.15.70:42634) 2024-06-04T04:58:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.480+0000 7fb2011f1640 1 -- 172.21.15.70:0/2498806713 learned_addr learned my addr 172.21.15.70:0/2498806713 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.480+0000 7fb2009f0640 1 --2- 172.21.15.70:0/2498806713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca20b140 0x5646ca20d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.480+0000 7fb1fbfff640 1 --2- 172.21.15.70:0/2498806713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca213660 0x5646ca2216c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.480+0000 7fb2011f1640 1 -- 172.21.15.70:0/2498806713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca213660 msgr2=0x5646ca2216c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:55.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.480+0000 7fb2011f1640 1 --2- 172.21.15.70:0/2498806713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca213660 0x5646ca2216c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.480+0000 7fb2011f1640 1 -- 172.21.15.70:0/2498806713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca20b140 msgr2=0x5646ca20d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:55.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.480+0000 7fb2011f1640 1 --2- 172.21.15.70:0/2498806713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca20b140 0x5646ca20d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.480+0000 7fb2011f1640 1 -- 172.21.15.70:0/2498806713 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646ca213480 con 0x5646ca215a00 2024-06-04T04:58:55.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.480+0000 7fb1fbfff640 1 --2- 172.21.15.70:0/2498806713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca213660 0x5646ca2216c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:58:55.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.480+0000 7fb2011f1640 1 --2- 172.21.15.70:0/2498806713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca215a00 0x5646ca22e1a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4100870 tx=0x7fb1f40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17a572937f74f30b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:55.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.480+0000 7fb1e67fc640 1 -- 172.21.15.70:0/2498806713 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f40e4070 con 0x5646ca215a00 2024-06-04T04:58:55.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.480+0000 7fb1e67fc640 1 -- 172.21.15.70:0/2498806713 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb1f40ca450 con 0x5646ca215a00 2024-06-04T04:58:55.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.480+0000 7fb1e67fc640 1 -- 172.21.15.70:0/2498806713 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f40c6870 con 0x5646ca215a00 2024-06-04T04:58:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.481+0000 7fb20596c740 1 -- 172.21.15.70:0/2498806713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca215a00 msgr2=0x5646ca22e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.481+0000 7fb20596c740 1 --2- 172.21.15.70:0/2498806713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca215a00 0x5646ca22e1a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4100870 tx=0x7fb1f40095c0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.482+0000 7fb20596c740 1 -- 172.21.15.70:0/2498806713 shutdown_connections 2024-06-04T04:58:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.482+0000 7fb20596c740 1 --2- 172.21.15.70:0/2498806713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca215a00 0x5646ca22e1a0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.482+0000 7fb20596c740 1 --2- 172.21.15.70:0/2498806713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca213660 0x5646ca2216c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.482+0000 7fb20596c740 1 --2- 172.21.15.70:0/2498806713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca20b140 0x5646ca20d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.482+0000 7fb20596c740 1 -- 172.21.15.70:0/2498806713 >> 172.21.15.70:0/2498806713 conn(0x5646ca206d30 msgr2=0x5646ca205fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.482+0000 7fb20596c740 1 -- 172.21.15.70:0/2498806713 shutdown_connections 2024-06-04T04:58:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.482+0000 7fb20596c740 1 -- 172.21.15.70:0/2498806713 wait complete. 2024-06-04T04:58:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.482+0000 7fb20596c740 1 -- start start 2024-06-04T04:58:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.482+0000 7fb20596c740 1 Processor -- start 2024-06-04T04:58:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb20596c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca20b140 0x5646ca219580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb20596c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca213660 0x5646ca219ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb20596c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca215a00 0x5646ca21a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb20596c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e000fc40 con 0x5646ca215a00 2024-06-04T04:58:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb20596c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e000f570 con 0x5646ca213660 2024-06-04T04:58:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb20596c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e000c7b0 con 0x5646ca20b140 2024-06-04T04:58:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb2011f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca215a00 0x5646ca21a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb2011f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca215a00 0x5646ca21a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42640/0 (socket says 172.21.15.70:42640) 2024-06-04T04:58:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb2011f1640 1 -- 172.21.15.70:0/2459721523 learned_addr learned my addr 172.21.15.70:0/2459721523 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb1fbfff640 1 --2- 172.21.15.70:0/2459721523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca213660 0x5646ca219ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb20596c740 10 client.0 _ll_register_callbacks cb 0x7fb1f0005850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb2009f0640 1 --2- 172.21.15.70:0/2459721523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca20b140 0x5646ca219580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb2011f1640 1 -- 172.21.15.70:0/2459721523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca20b140 msgr2=0x5646ca219580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb2011f1640 1 --2- 172.21.15.70:0/2459721523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca20b140 0x5646ca219580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb2011f1640 1 -- 172.21.15.70:0/2459721523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca213660 msgr2=0x5646ca219ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb2011f1640 1 --2- 172.21.15.70:0/2459721523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca213660 0x5646ca219ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb2011f1640 1 -- 172.21.15.70:0/2459721523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646ca290070 con 0x5646ca215a00 2024-06-04T04:58:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb2011f1640 1 --2- 172.21.15.70:0/2459721523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca215a00 0x5646ca21a000 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb1f402e950 tx=0x7fb1f402fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=417900da586b16c5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.483+0000 7fb1dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.484+0000 7fb1dbfff640 1 -- 172.21.15.70:0/2459721523 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f40e4040 con 0x5646ca215a00 2024-06-04T04:58:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.484+0000 7fb1dbfff640 1 -- 172.21.15.70:0/2459721523 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb1f40c4d80 con 0x5646ca215a00 2024-06-04T04:58:55.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.484+0000 7fb20596c740 1 -- 172.21.15.70:0/2459721523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1f40ca450 con 0x5646ca215a00 2024-06-04T04:58:55.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.484+0000 7fb1dbfff640 1 -- 172.21.15.70:0/2459721523 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb1f40ce750 con 0x5646ca215a00 2024-06-04T04:58:55.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.484+0000 7fb20596c740 10 client.15942 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:55.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.484+0000 7fb20596c740 1 -- 172.21.15.70:0/2459721523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5646ca2912e0 con 0x5646ca215a00 2024-06-04T04:58:55.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.485+0000 7fb1dbfff640 1 -- 172.21.15.70:0/2459721523 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb1f40ca450 con 0x5646ca215a00 2024-06-04T04:58:55.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.485+0000 7fb1dbfff640 1 -- 172.21.15.70:0/2459721523 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fb1f40c96b0 con 0x5646ca215a00 2024-06-04T04:58:55.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.485+0000 7fb20596c740 10 client.15942 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:55.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.485+0000 7fb20596c740 10 client.15942 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:55.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.485+0000 7fb20596c740 20 client.15942 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:55.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.485+0000 7fb20596c740 1 --2- 172.21.15.70:0/2459721523 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5646ca2918f0 0x5646ca293ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:55.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.485+0000 7fb20596c740 4 client.15942 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T04:58:55.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.485+0000 7fb20596c740 1 -- 172.21.15.70:0/2459721523 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5646ca178360 con 0x5646ca2918f0 2024-06-04T04:58:55.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.485+0000 7fb2009f0640 1 --2- 172.21.15.70:0/2459721523 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5646ca2918f0 0x5646ca293ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:55.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.486+0000 7fb2009f0640 1 --2- 172.21.15.70:0/2459721523 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5646ca2918f0 0x5646ca293ce0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b28b5f40ee0c1a1 server_cookie=63fa9dc0a52cc9f1 in_seq=0 out_seq=0 2024-06-04T04:58:55.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.486+0000 7fb1dbfff640 10 client.15942 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.489+0000 7fb1dbfff640 1 -- 172.21.15.70:0/2459721523 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5646ca178360 con 0x5646ca2918f0 2024-06-04T04:58:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.489+0000 7fb1dbfff640 10 client.15942 handle_command_reply: tid=0 2024-06-04T04:58:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.489+0000 7fb20596c740 1 client.15942 shutdown 2024-06-04T04:58:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.489+0000 7fb20596c740 1 -- 172.21.15.70:0/2459721523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca215a00 msgr2=0x5646ca21a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.489+0000 7fb20596c740 1 --2- 172.21.15.70:0/2459721523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca215a00 0x5646ca21a000 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb1f402e950 tx=0x7fb1f402fa80 comp rx=0 tx=0).stop 2024-06-04T04:58:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.490+0000 7fb20596c740 1 -- 172.21.15.70:0/2459721523 shutdown_connections 2024-06-04T04:58:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.490+0000 7fb20596c740 1 --2- 172.21.15.70:0/2459721523 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5646ca2918f0 0x5646ca293ce0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.490+0000 7fb20596c740 1 --2- 172.21.15.70:0/2459721523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ca213660 0x5646ca219ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.490+0000 7fb20596c740 1 --2- 172.21.15.70:0/2459721523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ca215a00 0x5646ca21a000 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.490+0000 7fb20596c740 1 --2- 172.21.15.70:0/2459721523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ca20b140 0x5646ca219580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.490+0000 7fb20596c740 1 -- 172.21.15.70:0/2459721523 >> 172.21.15.70:0/2459721523 conn(0x5646ca206d30 msgr2=0x5646ca205720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.490+0000 7fb1dbfff640 0 client.15942 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.490+0000 7fb20596c740 1 -- 172.21.15.70:0/2459721523 shutdown_connections 2024-06-04T04:58:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.490+0000 7fb20596c740 1 -- 172.21.15.70:0/2459721523 wait complete. 2024-06-04T04:58:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.490+0000 7fb20596c740 20 client.15942 trim_cache size 0 max 16384 2024-06-04T04:58:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T04:58:55.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T04:58:55.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T04:58:55.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.491+0000 7fb202c54640 1 -- 172.21.15.70:0/581449136 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1ec0b2a90 msgr2=0x7fb1ec0b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:55.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.491+0000 7fb202c54640 1 --2- 172.21.15.70:0/581449136 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1ec0b2a90 0x7fb1ec0b4f50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x5646ca20bdc0 tx=0x7fb1f000b410 comp rx=0 tx=0).stop 2024-06-04T04:58:55.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.491+0000 7fb202c54640 1 -- 172.21.15.70:0/581449136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1fc10d020 msgr2=0x7fb1fc1023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.491+0000 7fb202c54640 1 --2- 172.21.15.70:0/581449136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1fc10d020 0x7fb1fc1023b0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4005e30 tx=0x7fb1f4003ea0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.491+0000 7fb202c54640 1 -- 172.21.15.70:0/581449136 shutdown_connections 2024-06-04T04:58:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.491+0000 7fb202c54640 1 --2- 172.21.15.70:0/581449136 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1ec0b2a90 0x7fb1ec0b4f50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.491+0000 7fb202c54640 1 --2- 172.21.15.70:0/581449136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1fc104600 0x7fb1fc101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.491+0000 7fb202c54640 1 --2- 172.21.15.70:0/581449136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1fc10d020 0x7fb1fc1023b0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.491+0000 7fb202c54640 1 --2- 172.21.15.70:0/581449136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc103ce0 0x7fb1fc101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.491+0000 7fb202c54640 1 -- 172.21.15.70:0/581449136 >> 172.21.15.70:0/581449136 conn(0x7fb1fc078ad0 msgr2=0x7fb1fc0fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.491+0000 7fb202c54640 1 -- 172.21.15.70:0/581449136 shutdown_connections 2024-06-04T04:58:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:58:55.491+0000 7fb202c54640 1 -- 172.21.15.70:0/581449136 wait complete. 2024-06-04T04:58:55.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:55 smithi098 ceph-mon[35569]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:58:57.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:57 smithi070 ceph-mon[32631]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 511 B/s wr, 13 op/s 2024-06-04T04:58:57.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:57 smithi112 ceph-mon[36453]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 511 B/s wr, 13 op/s 2024-06-04T04:58:57.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:57 smithi098 ceph-mon[35569]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 511 B/s wr, 13 op/s 2024-06-04T04:58:57.819 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:58:57.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.832+0000 7f183afa6700 1 -- 172.21.15.70:0/2700652777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1834102810 msgr2=0x7f1834106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:57.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.832+0000 7f183afa6700 1 --2- 172.21.15.70:0/2700652777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1834102810 0x7f1834106810 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f182000aad0 tx=0x7f182000ade0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.832+0000 7f183afa6700 1 -- 172.21.15.70:0/2700652777 shutdown_connections 2024-06-04T04:58:57.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.832+0000 7f183afa6700 1 --2- 172.21.15.70:0/2700652777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1834106ef0 0x7f183410eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.832+0000 7f183afa6700 1 --2- 172.21.15.70:0/2700652777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1834101e60 0x7f1834102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.832+0000 7f183afa6700 1 --2- 172.21.15.70:0/2700652777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1834102810 0x7f1834106810 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.832+0000 7f183afa6700 1 -- 172.21.15.70:0/2700652777 >> 172.21.15.70:0/2700652777 conn(0x7f1834078ab0 msgr2=0x7f18340ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:57.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.832+0000 7f183afa6700 1 -- 172.21.15.70:0/2700652777 shutdown_connections 2024-06-04T04:58:57.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.832+0000 7f183afa6700 1 -- 172.21.15.70:0/2700652777 wait complete. 2024-06-04T04:58:57.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.833+0000 7f183afa6700 1 Processor -- start 2024-06-04T04:58:57.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.833+0000 7f183afa6700 1 -- start start 2024-06-04T04:58:57.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.833+0000 7f183afa6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1834101e60 0x7f18341a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.833+0000 7f183afa6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1834102810 0x7f18341a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.833+0000 7f183afa6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1834106ef0 0x7f183419bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.834+0000 7f183afa6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f18341157a0 con 0x7f1834106ef0 2024-06-04T04:58:57.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.834+0000 7f183afa6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1834115620 con 0x7f1834102810 2024-06-04T04:58:57.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.834+0000 7f183afa6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1834115920 con 0x7f1834101e60 2024-06-04T04:58:57.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.834+0000 7f1838d42700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1834101e60 0x7f18341a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.834+0000 7f1839543700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1834106ef0 0x7f183419bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.834+0000 7f1838d42700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1834101e60 0x7f18341a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:44794/0 (socket says 172.21.15.70:44794) 2024-06-04T04:58:57.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.834+0000 7f1838d42700 1 -- 172.21.15.70:0/1459376063 learned_addr learned my addr 172.21.15.70:0/1459376063 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:57.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.834+0000 7f1833fff700 1 --2- 172.21.15.70:0/1459376063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1834102810 0x7f18341a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.834+0000 7f1838d42700 1 -- 172.21.15.70:0/1459376063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1834102810 msgr2=0x7f18341a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:57.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.834+0000 7f1838d42700 1 --2- 172.21.15.70:0/1459376063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1834102810 0x7f18341a1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.834+0000 7f1838d42700 1 -- 172.21.15.70:0/1459376063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1834106ef0 msgr2=0x7f183419bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:57.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.834+0000 7f1838d42700 1 --2- 172.21.15.70:0/1459376063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1834106ef0 0x7f183419bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.834+0000 7f1838d42700 1 -- 172.21.15.70:0/1459376063 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f183419c220 con 0x7f1834101e60 2024-06-04T04:58:57.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.834+0000 7f1839543700 1 --2- 172.21.15.70:0/1459376063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1834106ef0 0x7f183419bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:57.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.834+0000 7f1838d42700 1 --2- 172.21.15.70:0/1459376063 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1834101e60 0x7f18341a1900 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f182800c9b0 tx=0x7f182800ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:57.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.835+0000 7f1831ffb700 1 -- 172.21.15.70:0/1459376063 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1828007b00 con 0x7f1834101e60 2024-06-04T04:58:57.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.835+0000 7f183afa6700 1 -- 172.21.15.70:0/1459376063 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f183419c510 con 0x7f1834101e60 2024-06-04T04:58:57.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.835+0000 7f1831ffb700 1 -- 172.21.15.70:0/1459376063 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f182800ce50 con 0x7f1834101e60 2024-06-04T04:58:57.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.835+0000 7f1831ffb700 1 -- 172.21.15.70:0/1459376063 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1828007b00 con 0x7f1834101e60 2024-06-04T04:58:57.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.836+0000 7f183afa6700 1 -- 172.21.15.70:0/1459376063 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18341a87d0 con 0x7f1834101e60 2024-06-04T04:58:57.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.840+0000 7f1831ffb700 1 -- 172.21.15.70:0/1459376063 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f182800f450 con 0x7f1834101e60 2024-06-04T04:58:57.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.841+0000 7f1831ffb700 1 --2- 172.21.15.70:0/1459376063 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f18240b2bf0 0x7f18240b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.841+0000 7f1831ffb700 1 -- 172.21.15.70:0/1459376063 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f18280e1b50 con 0x7f1834101e60 2024-06-04T04:58:57.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.841+0000 7f1833fff700 1 --2- 172.21.15.70:0/1459376063 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f18240b2bf0 0x7f18240b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.841+0000 7f184583bb80 1 Processor -- start 2024-06-04T04:58:57.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.841+0000 7f184583bb80 1 -- start start 2024-06-04T04:58:57.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.842+0000 7f1833fff700 1 --2- 172.21.15.70:0/1459376063 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f18240b2bf0 0x7f18240b50b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1820006010 tx=0x7f182001b040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:57.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.842+0000 7f184583bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 0x564f2fb51cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.842+0000 7f184583bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb52290 0x564f2fb559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.842+0000 7f184583bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb55fe0 0x564f2fb56420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.842+0000 7f1838d42700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 0x564f2fb51cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.842+0000 7f1838d42700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 0x564f2fb51cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:42440/0 (socket says 172.21.15.70:42440) 2024-06-04T04:58:57.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.842+0000 7f184583bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564f2fab0d10 con 0x564f2fb55fe0 2024-06-04T04:58:57.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.842+0000 7f1839543700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb55fe0 0x564f2fb56420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.842+0000 7f1838d42700 1 -- 172.21.15.70:0/859845396 learned_addr learned my addr 172.21.15.70:0/859845396 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:57.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.842+0000 7f1833fff700 1 --2- 172.21.15.70:0/859845396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb52290 0x564f2fb559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.842+0000 7f184583bb80 1 -- 172.21.15.70:0/859845396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564f2faa08b0 con 0x564f2fb518e0 2024-06-04T04:58:57.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.842+0000 7f184583bb80 1 -- 172.21.15.70:0/859845396 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564f2faa48f0 con 0x564f2fb52290 2024-06-04T04:58:57.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.843+0000 7f1839543700 1 -- 172.21.15.70:0/859845396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb52290 msgr2=0x564f2fb559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:57.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.843+0000 7f1839543700 1 --2- 172.21.15.70:0/859845396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb52290 0x564f2fb559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.843+0000 7f1839543700 1 -- 172.21.15.70:0/859845396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 msgr2=0x564f2fb51cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:57.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.843+0000 7f1839543700 1 --2- 172.21.15.70:0/859845396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 0x564f2fb51cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.843+0000 7f1839543700 1 -- 172.21.15.70:0/859845396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f2f9f0950 con 0x564f2fb55fe0 2024-06-04T04:58:57.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.843+0000 7f1839543700 1 --2- 172.21.15.70:0/859845396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb55fe0 0x564f2fb56420 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f182c007600 tx=0x7f182c007910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81fa177a0c508822 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:57.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.843+0000 7f181effd700 1 -- 172.21.15.70:0/859845396 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c019020 con 0x564f2fb55fe0 2024-06-04T04:58:57.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.843+0000 7f181effd700 1 -- 172.21.15.70:0/859845396 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f182c015da0 con 0x564f2fb55fe0 2024-06-04T04:58:57.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.843+0000 7f181effd700 1 -- 172.21.15.70:0/859845396 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c014cf0 con 0x564f2fb55fe0 2024-06-04T04:58:57.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.844+0000 7f184583bb80 1 -- 172.21.15.70:0/859845396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb55fe0 msgr2=0x564f2fb56420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:57.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.844+0000 7f184583bb80 1 --2- 172.21.15.70:0/859845396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb55fe0 0x564f2fb56420 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f182c007600 tx=0x7f182c007910 comp rx=0 tx=0).stop 2024-06-04T04:58:57.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.845+0000 7f184583bb80 1 -- 172.21.15.70:0/859845396 shutdown_connections 2024-06-04T04:58:57.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.845+0000 7f184583bb80 1 --2- 172.21.15.70:0/859845396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb55fe0 0x564f2fb56420 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.845+0000 7f184583bb80 1 --2- 172.21.15.70:0/859845396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 0x564f2fb51cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.845+0000 7f184583bb80 1 --2- 172.21.15.70:0/859845396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb52290 0x564f2fb559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.845+0000 7f184583bb80 1 -- 172.21.15.70:0/859845396 >> 172.21.15.70:0/859845396 conn(0x564f2fa7f100 msgr2=0x564f2fb4bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:57.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.846+0000 7f184583bb80 1 -- 172.21.15.70:0/859845396 shutdown_connections 2024-06-04T04:58:57.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.846+0000 7f184583bb80 1 -- 172.21.15.70:0/859845396 wait complete. 2024-06-04T04:58:57.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.847+0000 7f184583bb80 1 -- start start 2024-06-04T04:58:57.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.847+0000 7f184583bb80 1 Processor -- start 2024-06-04T04:58:57.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.848+0000 7f184583bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 0x564f2fb5d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.848+0000 7f184583bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb52290 0x564f2fb5a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.848+0000 7f1838d42700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 0x564f2fb5d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.848+0000 7f1838d42700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 0x564f2fb5d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:42446/0 (socket says 172.21.15.70:42446) 2024-06-04T04:58:57.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.848+0000 7f1838d42700 1 -- 172.21.15.70:0/3460816430 learned_addr learned my addr 172.21.15.70:0/3460816430 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:57.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.849+0000 7f1833fff700 1 --2- 172.21.15.70:0/3460816430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb52290 0x564f2fb5a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.849+0000 7f184583bb80 1 --2- 172.21.15.70:0/3460816430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb55fe0 0x564f2fb5ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.849+0000 7f184583bb80 1 -- 172.21.15.70:0/3460816430 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564f2f9a7020 con 0x564f2fb52290 2024-06-04T04:58:57.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.849+0000 7f184583bb80 1 -- 172.21.15.70:0/3460816430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564f2fa42840 con 0x564f2fb518e0 2024-06-04T04:58:57.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.849+0000 7f1839543700 1 --2- 172.21.15.70:0/3460816430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb55fe0 0x564f2fb5ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.849+0000 7f184583bb80 1 -- 172.21.15.70:0/3460816430 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564f2fb5e1a0 con 0x564f2fb55fe0 2024-06-04T04:58:57.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.850+0000 7f1833fff700 1 -- 172.21.15.70:0/3460816430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb55fe0 msgr2=0x564f2fb5ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:57.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.850+0000 7f1833fff700 1 --2- 172.21.15.70:0/3460816430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb55fe0 0x564f2fb5ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.850+0000 7f1833fff700 1 -- 172.21.15.70:0/3460816430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 msgr2=0x564f2fb5d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:57.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.850+0000 7f1833fff700 1 --2- 172.21.15.70:0/3460816430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 0x564f2fb5d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.850+0000 7f1833fff700 1 -- 172.21.15.70:0/3460816430 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f2fa3f710 con 0x564f2fb52290 2024-06-04T04:58:57.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.850+0000 7f1839543700 1 --2- 172.21.15.70:0/3460816430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb55fe0 0x564f2fb5ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:57.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.850+0000 7f1838d42700 1 --2- 172.21.15.70:0/3460816430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 0x564f2fb5d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:58:57.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.850+0000 7f184583bb80 10 client.0 _ll_register_callbacks cb 0x564f2fa3b9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:57.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.850+0000 7f1833fff700 1 --2- 172.21.15.70:0/3460816430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb52290 0x564f2fb5a7a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1820000c00 tx=0x7f182000a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a56edc770cdf5648 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:57.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.850+0000 7f184583bb80 1 -- 172.21.15.70:0/3460816430 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f182c015da0 con 0x564f2fb52290 2024-06-04T04:58:57.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.850+0000 7f1813fff700 10 client.15954 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:57.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.851+0000 7f1813fff700 1 -- 172.21.15.70:0/3460816430 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1820044050 con 0x564f2fb52290 2024-06-04T04:58:57.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.851+0000 7f1813fff700 1 -- 172.21.15.70:0/3460816430 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f1820035a50 con 0x564f2fb52290 2024-06-04T04:58:57.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.851+0000 7f1813fff700 1 -- 172.21.15.70:0/3460816430 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f18200039f0 con 0x564f2fb52290 2024-06-04T04:58:57.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.851+0000 7f184583bb80 10 client.15954 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:57.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.851+0000 7f184583bb80 1 -- 172.21.15.70:0/3460816430 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({fsmap=29}) -- 0x564f2fb68740 con 0x564f2fb52290 2024-06-04T04:58:57.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.852+0000 7f1813fff700 1 -- 172.21.15.70:0/3460816430 <== mon.0 v2:172.21.15.70:3300/0 4 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f182000b040 con 0x564f2fb52290 2024-06-04T04:58:57.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.852+0000 7f184583bb80 10 client.15954 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:57.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.852+0000 7f184583bb80 10 client.15954 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:57.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.852+0000 7f184583bb80 20 client.15954 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:57.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.852+0000 7f184583bb80 1 --2- 172.21.15.70:0/3460816430 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564f2fb6ac70 0x564f2fb6d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.852+0000 7f184583bb80 4 client.15954 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:58:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.852+0000 7f184583bb80 1 -- 172.21.15.70:0/3460816430 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f2fb1da60 con 0x564f2fb6ac70 2024-06-04T04:58:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.852+0000 7f1838d42700 1 --2- 172.21.15.70:0/3460816430 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564f2fb6ac70 0x564f2fb6d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.853+0000 7f1813fff700 10 client.15954 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.853+0000 7f1838d42700 1 --2- 172.21.15.70:0/3460816430 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564f2fb6ac70 0x564f2fb6d060 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19a83814ff53510a server_cookie=135f2becccfd4f08 in_seq=0 out_seq=0 2024-06-04T04:58:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.853+0000 7f1813fff700 1 -- 172.21.15.70:0/3460816430 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564f2fb1da60 con 0x564f2fb6ac70 2024-06-04T04:58:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.853+0000 7f1813fff700 10 client.15954 handle_command_reply: tid=0 2024-06-04T04:58:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.853+0000 7f184583bb80 1 client.15954 shutdown 2024-06-04T04:58:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.854+0000 7f184583bb80 1 -- 172.21.15.70:0/3460816430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb52290 msgr2=0x564f2fb5a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.854+0000 7f184583bb80 1 --2- 172.21.15.70:0/3460816430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb52290 0x564f2fb5a7a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1820000c00 tx=0x7f182000a420 comp rx=0 tx=0).stop 2024-06-04T04:58:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.854+0000 7f184583bb80 1 -- 172.21.15.70:0/3460816430 shutdown_connections 2024-06-04T04:58:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.854+0000 7f184583bb80 1 --2- 172.21.15.70:0/3460816430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb55fe0 0x564f2fb5ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.854+0000 7f184583bb80 1 --2- 172.21.15.70:0/3460816430 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564f2fb6ac70 0x564f2fb6d060 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.854+0000 7f1813fff700 0 client.15954 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:57.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.854+0000 7f184583bb80 1 --2- 172.21.15.70:0/3460816430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 0x564f2fb5d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.854+0000 7f184583bb80 1 --2- 172.21.15.70:0/3460816430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb52290 0x564f2fb5a7a0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.854+0000 7f184583bb80 1 -- 172.21.15.70:0/3460816430 >> 172.21.15.70:0/3460816430 conn(0x564f2fa7f100 msgr2=0x564f2fb573d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:57.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.854+0000 7f184583bb80 1 -- 172.21.15.70:0/3460816430 shutdown_connections 2024-06-04T04:58:57.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.854+0000 7f184583bb80 1 -- 172.21.15.70:0/3460816430 wait complete. 2024-06-04T04:58:57.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.854+0000 7f184583bb80 20 client.15954 trim_cache size 0 max 16384 2024-06-04T04:58:57.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.875+0000 7f184583bb80 1 Processor -- start 2024-06-04T04:58:57.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.875+0000 7f184583bb80 1 -- start start 2024-06-04T04:58:57.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.875+0000 7f184583bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fa7f100 0x564f2fb50e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.875+0000 7f184583bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb518e0 0x564f2fb5cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.875+0000 7f184583bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb52290 0x564f2fb72900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.876+0000 7f184583bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f18341abba0 con 0x564f2fb52290 2024-06-04T04:58:57.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.876+0000 7f184583bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f18341ad6e0 con 0x564f2fa7f100 2024-06-04T04:58:57.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.876+0000 7f184583bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f18341ac270 con 0x564f2fb518e0 2024-06-04T04:58:57.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.876+0000 7f1839543700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb52290 0x564f2fb72900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.876+0000 7f1839543700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb52290 0x564f2fb72900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56236/0 (socket says 172.21.15.70:56236) 2024-06-04T04:58:57.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.876+0000 7f1839543700 1 -- 172.21.15.70:0/2821653551 learned_addr learned my addr 172.21.15.70:0/2821653551 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:57.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.876+0000 7f1833fff700 1 --2- 172.21.15.70:0/2821653551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb518e0 0x564f2fb5cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.876+0000 7f1838d42700 1 --2- 172.21.15.70:0/2821653551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fa7f100 0x564f2fb50e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.876+0000 7f1839543700 1 -- 172.21.15.70:0/2821653551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb518e0 msgr2=0x564f2fb5cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:57.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.876+0000 7f1839543700 1 --2- 172.21.15.70:0/2821653551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb518e0 0x564f2fb5cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.876+0000 7f1839543700 1 -- 172.21.15.70:0/2821653551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fa7f100 msgr2=0x564f2fb50e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:57.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.876+0000 7f1839543700 1 --2- 172.21.15.70:0/2821653551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fa7f100 0x564f2fb50e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.876+0000 7f1839543700 1 -- 172.21.15.70:0/2821653551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f2fb5b680 con 0x564f2fb52290 2024-06-04T04:58:57.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.876+0000 7f1839543700 1 --2- 172.21.15.70:0/2821653551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb52290 0x564f2fb72900 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f182c0093f0 tx=0x7f182c006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2fa8e64935872a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:57.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.876+0000 7f181e7fc700 1 -- 172.21.15.70:0/2821653551 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c019070 con 0x564f2fb52290 2024-06-04T04:58:57.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.876+0000 7f181e7fc700 1 -- 172.21.15.70:0/2821653551 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f182c045070 con 0x564f2fb52290 2024-06-04T04:58:57.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.876+0000 7f181e7fc700 1 -- 172.21.15.70:0/2821653551 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c014910 con 0x564f2fb52290 2024-06-04T04:58:57.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.877+0000 7f184583bb80 1 -- 172.21.15.70:0/2821653551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb52290 msgr2=0x564f2fb72900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:57.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.877+0000 7f184583bb80 1 --2- 172.21.15.70:0/2821653551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb52290 0x564f2fb72900 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f182c0093f0 tx=0x7f182c006000 comp rx=0 tx=0).stop 2024-06-04T04:58:57.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.877+0000 7f184583bb80 1 -- 172.21.15.70:0/2821653551 shutdown_connections 2024-06-04T04:58:57.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.877+0000 7f184583bb80 1 --2- 172.21.15.70:0/2821653551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fb52290 0x564f2fb72900 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.877+0000 7f184583bb80 1 --2- 172.21.15.70:0/2821653551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fa7f100 0x564f2fb50e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.877+0000 7f184583bb80 1 --2- 172.21.15.70:0/2821653551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb518e0 0x564f2fb5cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.877+0000 7f184583bb80 1 -- 172.21.15.70:0/2821653551 >> 172.21.15.70:0/2821653551 conn(0x564f2fb4ba30 msgr2=0x564f2fb506f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:57.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.878+0000 7f184583bb80 1 -- 172.21.15.70:0/2821653551 shutdown_connections 2024-06-04T04:58:57.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.878+0000 7f184583bb80 1 -- 172.21.15.70:0/2821653551 wait complete. 2024-06-04T04:58:57.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.878+0000 7f184583bb80 1 -- start start 2024-06-04T04:58:57.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.878+0000 7f184583bb80 1 Processor -- start 2024-06-04T04:58:57.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f184583bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fa7f100 0x564f2fb635d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f184583bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 0x564f2fb63b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f184583bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb52290 0x564f2fb61ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f184583bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f18341ad010 con 0x564f2fa7f100 2024-06-04T04:58:57.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f1838d42700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fa7f100 0x564f2fb635d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f184583bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f18341ac940 con 0x564f2fb518e0 2024-06-04T04:58:57.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f1838d42700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fa7f100 0x564f2fb635d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56244/0 (socket says 172.21.15.70:56244) 2024-06-04T04:58:57.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f1838d42700 1 -- 172.21.15.70:0/556839620 learned_addr learned my addr 172.21.15.70:0/556839620 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:58:57.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f184583bb80 1 -- 172.21.15.70:0/556839620 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f18341a9bb0 con 0x564f2fb52290 2024-06-04T04:58:57.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f184583bb80 10 client.0 _ll_register_callbacks cb 0x564f2fa3b9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:58:57.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f1833fff700 1 --2- 172.21.15.70:0/556839620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 0x564f2fb63b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f1839543700 1 --2- 172.21.15.70:0/556839620 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb52290 0x564f2fb61ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f1838d42700 1 -- 172.21.15.70:0/556839620 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb52290 msgr2=0x564f2fb61ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:57.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f1838d42700 1 --2- 172.21.15.70:0/556839620 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb52290 0x564f2fb61ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f1838d42700 1 -- 172.21.15.70:0/556839620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 msgr2=0x564f2fb63b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:58:57.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f1838d42700 1 --2- 172.21.15.70:0/556839620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 0x564f2fb63b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f1838d42700 1 -- 172.21.15.70:0/556839620 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f2fb622c0 con 0x564f2fa7f100 2024-06-04T04:58:57.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.879+0000 7f1838d42700 1 --2- 172.21.15.70:0/556839620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fa7f100 0x564f2fb635d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f18280a4eb0 tx=0x7f18280a8d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4d4bcc32224ebe server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:58:57.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.880+0000 7f1813fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:58:57.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.880+0000 7f1813fff700 1 -- 172.21.15.70:0/556839620 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1828088060 con 0x564f2fa7f100 2024-06-04T04:58:57.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.880+0000 7f1813fff700 1 -- 172.21.15.70:0/556839620 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f18280a5420 con 0x564f2fa7f100 2024-06-04T04:58:57.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.880+0000 7f184583bb80 1 -- 172.21.15.70:0/556839620 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f182c045070 con 0x564f2fa7f100 2024-06-04T04:58:57.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.880+0000 7f1813fff700 1 -- 172.21.15.70:0/556839620 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f18280ad1e0 con 0x564f2fa7f100 2024-06-04T04:58:57.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.880+0000 7f184583bb80 10 client.15960 fetch_fsmap learned FSMap version 29 2024-06-04T04:58:57.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.880+0000 7f184583bb80 1 -- 172.21.15.70:0/556839620 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564f2fbd63c0 con 0x564f2fa7f100 2024-06-04T04:58:57.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.880+0000 7f1813fff700 1 -- 172.21.15.70:0/556839620 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f18280c7070 con 0x564f2fa7f100 2024-06-04T04:58:57.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.881+0000 7f1813fff700 1 -- 172.21.15.70:0/556839620 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f18280a9090 con 0x564f2fa7f100 2024-06-04T04:58:57.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.881+0000 7f184583bb80 10 client.15960 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:58:57.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.881+0000 7f184583bb80 10 client.15960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:58:57.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.881+0000 7f184583bb80 20 client.15960 populate_metadata read hostname 'smithi070' 2024-06-04T04:58:57.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.881+0000 7f184583bb80 1 --2- 172.21.15.70:0/556839620 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564f2fb55fe0 0x564f2fbd89e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:58:57.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.881+0000 7f184583bb80 4 client.15960 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T04:58:57.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.881+0000 7f184583bb80 1 -- 172.21.15.70:0/556839620 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f182c045070 con 0x564f2fb55fe0 2024-06-04T04:58:57.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.881+0000 7f1833fff700 1 --2- 172.21.15.70:0/556839620 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564f2fb55fe0 0x564f2fbd89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:58:57.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.881+0000 7f1833fff700 1 --2- 172.21.15.70:0/556839620 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564f2fb55fe0 0x564f2fbd89e0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bf95b4dafd30f9 server_cookie=a2997b780a8eccaa in_seq=0 out_seq=0 2024-06-04T04:58:57.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.881+0000 7f1813fff700 10 client.15960 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:57.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.883+0000 7f1813fff700 1 -- 172.21.15.70:0/556839620 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f182c045070 con 0x564f2fb55fe0 2024-06-04T04:58:57.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.883+0000 7f1813fff700 10 client.15960 handle_command_reply: tid=0 2024-06-04T04:58:57.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.883+0000 7f184583bb80 1 client.15960 shutdown 2024-06-04T04:58:57.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.883+0000 7f184583bb80 1 -- 172.21.15.70:0/556839620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fa7f100 msgr2=0x564f2fb635d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:57.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.883+0000 7f184583bb80 1 --2- 172.21.15.70:0/556839620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fa7f100 0x564f2fb635d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f18280a4eb0 tx=0x7f18280a8d10 comp rx=0 tx=0).stop 2024-06-04T04:58:57.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.883+0000 7f184583bb80 1 -- 172.21.15.70:0/556839620 shutdown_connections 2024-06-04T04:58:57.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.883+0000 7f184583bb80 1 --2- 172.21.15.70:0/556839620 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f2fb52290 0x564f2fb61ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.883+0000 7f184583bb80 1 --2- 172.21.15.70:0/556839620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f2fa7f100 0x564f2fb635d0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.883+0000 7f184583bb80 1 --2- 172.21.15.70:0/556839620 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564f2fb55fe0 0x564f2fbd89e0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.883+0000 7f184583bb80 1 --2- 172.21.15.70:0/556839620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f2fb518e0 0x564f2fb63b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.883+0000 7f184583bb80 1 -- 172.21.15.70:0/556839620 >> 172.21.15.70:0/556839620 conn(0x564f2fb4ba30 msgr2=0x564f2fb50f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:57.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.883+0000 7f1813fff700 0 client.15960 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:58:57.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.884+0000 7f184583bb80 1 -- 172.21.15.70:0/556839620 shutdown_connections 2024-06-04T04:58:57.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.884+0000 7f184583bb80 1 -- 172.21.15.70:0/556839620 wait complete. 2024-06-04T04:58:57.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.884+0000 7f184583bb80 20 client.15960 trim_cache size 0 max 16384 2024-06-04T04:58:57.890 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:58:57.890 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T04:58:57.890 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "a3f72ca7-35a6-4890-af23-231180cad789", 2024-06-04T04:58:57.890 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T04:58:57.890 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:58:57.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.884+0000 7f183afa6700 1 -- 172.21.15.70:0/1459376063 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f18240b2bf0 msgr2=0x7f18240b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:57.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.884+0000 7f183afa6700 1 --2- 172.21.15.70:0/1459376063 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f18240b2bf0 0x7f18240b50b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1820006010 tx=0x7f182001b040 comp rx=0 tx=0).stop 2024-06-04T04:58:57.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.884+0000 7f183afa6700 1 -- 172.21.15.70:0/1459376063 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1834101e60 msgr2=0x7f18341a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:58:57.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.884+0000 7f183afa6700 1 --2- 172.21.15.70:0/1459376063 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1834101e60 0x7f18341a1900 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f182800c9b0 tx=0x7f182800ccc0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.884+0000 7f183afa6700 1 -- 172.21.15.70:0/1459376063 shutdown_connections 2024-06-04T04:58:57.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.885+0000 7f183afa6700 1 --2- 172.21.15.70:0/1459376063 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f18240b2bf0 0x7f18240b50b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.885+0000 7f183afa6700 1 --2- 172.21.15.70:0/1459376063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1834106ef0 0x7f183419bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.885+0000 7f183afa6700 1 --2- 172.21.15.70:0/1459376063 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1834101e60 0x7f18341a1900 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f182800c9b0 tx=0x7f182800ccc0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.885+0000 7f183afa6700 1 --2- 172.21.15.70:0/1459376063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1834102810 0x7f18341a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:58:57.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.885+0000 7f183afa6700 1 -- 172.21.15.70:0/1459376063 >> 172.21.15.70:0/1459376063 conn(0x7f1834078ab0 msgr2=0x7f18340ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:58:57.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.885+0000 7f183afa6700 1 -- 172.21.15.70:0/1459376063 shutdown_connections 2024-06-04T04:58:57.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:58:57.885+0000 7f183afa6700 1 -- 172.21.15.70:0/1459376063 wait complete. 2024-06-04T04:58:59.345 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:59 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T04:58:59.345 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:59 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T04:58:59.345 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:59 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T04:58:59.346 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:59 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T04:58:59.346 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:58:59 smithi070 ceph-mon[32631]: pgmap v57: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:58:59.427 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T04:58:59.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:59 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T04:58:59.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:59 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T04:58:59.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:59 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T04:58:59.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:59 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T04:58:59.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:58:59 smithi112 ceph-mon[36453]: pgmap v57: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:58:59.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:59 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T04:58:59.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:59 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T04:58:59.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:59 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T04:58:59.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:59 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T04:58:59.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:58:59 smithi098 ceph-mon[35569]: pgmap v57: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:58:59.891 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:59:01.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.330+0000 7f562adbd700 1 -- 172.21.15.70:0/2137979104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5624108510 msgr2=0x7f56241088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:01.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.330+0000 7f562adbd700 1 --2- 172.21.15.70:0/2137979104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5624108510 0x7f56241088f0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f5618009990 tx=0x7f5618009ca0 comp rx=0 tx=0).stop 2024-06-04T04:59:01.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.330+0000 7f562adbd700 1 -- 172.21.15.70:0/2137979104 shutdown_connections 2024-06-04T04:59:01.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.330+0000 7f562adbd700 1 --2- 172.21.15.70:0/2137979104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5624102df0 0x7f562410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.330+0000 7f562adbd700 1 --2- 172.21.15.70:0/2137979104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5624108510 0x7f56241088f0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.330+0000 7f562adbd700 1 --2- 172.21.15.70:0/2137979104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5624102430 0x7f56241028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.330+0000 7f562adbd700 1 -- 172.21.15.70:0/2137979104 >> 172.21.15.70:0/2137979104 conn(0x7f56240793b0 msgr2=0x7f56240797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.330+0000 7f562adbd700 1 -- 172.21.15.70:0/2137979104 shutdown_connections 2024-06-04T04:59:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.330+0000 7f562adbd700 1 -- 172.21.15.70:0/2137979104 wait complete. 2024-06-04T04:59:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.330+0000 7f562adbd700 1 Processor -- start 2024-06-04T04:59:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f562adbd700 1 -- start start 2024-06-04T04:59:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f562adbd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5624102430 0x7f562419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f562adbd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5624102df0 0x7f562419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f562adbd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5624108510 0x7f56241987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f562adbd700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5624112490 con 0x7f5624102430 2024-06-04T04:59:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f562adbd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5624112310 con 0x7f5624108510 2024-06-04T04:59:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f562adbd700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5624112610 con 0x7f5624102df0 2024-06-04T04:59:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f5628b59700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5624102430 0x7f562419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f5628b59700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5624102430 0x7f562419e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56270/0 (socket says 172.21.15.70:56270) 2024-06-04T04:59:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f5628b59700 1 -- 172.21.15.70:0/457323416 learned_addr learned my addr 172.21.15.70:0/457323416 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f562935a700 1 --2- 172.21.15.70:0/457323416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5624108510 0x7f56241987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f5623fff700 1 --2- 172.21.15.70:0/457323416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5624102df0 0x7f562419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f5628b59700 1 -- 172.21.15.70:0/457323416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5624102df0 msgr2=0x7f562419eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f5628b59700 1 --2- 172.21.15.70:0/457323416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5624102df0 0x7f562419eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f5628b59700 1 -- 172.21.15.70:0/457323416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5624108510 msgr2=0x7f56241987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f5628b59700 1 --2- 172.21.15.70:0/457323416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5624108510 0x7f56241987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f5628b59700 1 -- 172.21.15.70:0/457323416 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5624198fb0 con 0x7f5624102430 2024-06-04T04:59:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f5623fff700 1 --2- 172.21.15.70:0/457323416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5624102df0 0x7f562419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:59:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.331+0000 7f562935a700 1 --2- 172.21.15.70:0/457323416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5624108510 0x7f56241987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:59:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.332+0000 7f5628b59700 1 --2- 172.21.15.70:0/457323416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5624102430 0x7f562419e650 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5618000c00 tx=0x7f56180047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:01.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.332+0000 7f5621ffb700 1 -- 172.21.15.70:0/457323416 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5618027070 con 0x7f5624102430 2024-06-04T04:59:01.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.332+0000 7f562adbd700 1 -- 172.21.15.70:0/457323416 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5624199240 con 0x7f5624102430 2024-06-04T04:59:01.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.332+0000 7f5621ffb700 1 -- 172.21.15.70:0/457323416 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f561800fdb0 con 0x7f5624102430 2024-06-04T04:59:01.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.332+0000 7f5621ffb700 1 -- 172.21.15.70:0/457323416 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56180173f0 con 0x7f5624102430 2024-06-04T04:59:01.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.332+0000 7f562adbd700 1 -- 172.21.15.70:0/457323416 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56241a54b0 con 0x7f5624102430 2024-06-04T04:59:01.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.333+0000 7f562adbd700 1 -- 172.21.15.70:0/457323416 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f562410a9a0 con 0x7f5624102430 2024-06-04T04:59:01.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.334+0000 7f5621ffb700 1 -- 172.21.15.70:0/457323416 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f561802a070 con 0x7f5624102430 2024-06-04T04:59:01.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.335+0000 7f5621ffb700 1 --2- 172.21.15.70:0/457323416 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56140b29f0 0x7f56140b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:01.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.335+0000 7f5623fff700 1 --2- 172.21.15.70:0/457323416 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56140b29f0 0x7f56140b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:01.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.335+0000 7f5621ffb700 1 -- 172.21.15.70:0/457323416 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f56180e3c60 con 0x7f5624102430 2024-06-04T04:59:01.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.336+0000 7f5623fff700 1 --2- 172.21.15.70:0/457323416 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56140b29f0 0x7f56140b4eb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5610005fd0 tx=0x7f5610005dc0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:01.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.339+0000 7f5621ffb700 1 -- 172.21.15.70:0/457323416 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5618031070 con 0x7f5624102430 2024-06-04T04:59:01.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:01 smithi112 ceph-mon[36453]: pgmap v58: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:59:01.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:01 smithi098 ceph-mon[35569]: pgmap v58: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:59:01.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.600+0000 7f562adbd700 1 -- 172.21.15.70:0/457323416 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5624103b50 con 0x7f5624102430 2024-06-04T04:59:01.603 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:01 smithi070 ceph-mon[32631]: pgmap v58: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-04T04:59:01.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.601+0000 7f5621ffb700 1 -- 172.21.15.70:0/457323416 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f5624103b50 con 0x7f5624102430 2024-06-04T04:59:01.604 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:59:01.604 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:57:09.013178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T04:59:01.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.604+0000 7f562adbd700 1 -- 172.21.15.70:0/457323416 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56140b29f0 msgr2=0x7f56140b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:01.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.604+0000 7f562adbd700 1 --2- 172.21.15.70:0/457323416 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56140b29f0 0x7f56140b4eb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5610005fd0 tx=0x7f5610005dc0 comp rx=0 tx=0).stop 2024-06-04T04:59:01.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.604+0000 7f562adbd700 1 -- 172.21.15.70:0/457323416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5624102430 msgr2=0x7f562419e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:01.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.604+0000 7f562adbd700 1 --2- 172.21.15.70:0/457323416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5624102430 0x7f562419e650 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5618000c00 tx=0x7f56180047b0 comp rx=0 tx=0).stop 2024-06-04T04:59:01.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.604+0000 7f562adbd700 1 -- 172.21.15.70:0/457323416 shutdown_connections 2024-06-04T04:59:01.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.604+0000 7f562adbd700 1 --2- 172.21.15.70:0/457323416 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56140b29f0 0x7f56140b4eb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:01.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.604+0000 7f562adbd700 1 --2- 172.21.15.70:0/457323416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5624108510 0x7f56241987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:01.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.604+0000 7f562adbd700 1 --2- 172.21.15.70:0/457323416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5624102430 0x7f562419e650 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:01.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.604+0000 7f562adbd700 1 --2- 172.21.15.70:0/457323416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5624102df0 0x7f562419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:01.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.604+0000 7f562adbd700 1 -- 172.21.15.70:0/457323416 >> 172.21.15.70:0/457323416 conn(0x7f56240793b0 msgr2=0x7f56241012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:01.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.604+0000 7f562adbd700 1 -- 172.21.15.70:0/457323416 shutdown_connections 2024-06-04T04:59:01.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:01.604+0000 7f562adbd700 1 -- 172.21.15.70:0/457323416 wait complete. 2024-06-04T04:59:01.607 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-06-04T04:59:02.293 INFO:tasks.workunit.client.0.smithi070.stderr:Updating files: 97% (12256/12566) Updating files: 98% (12315/12566) Updating files: 99% (12441/12566) Updating files: 100% (12566/12566) Updating files: 100% (12566/12566), done. 2024-06-04T04:59:02.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:02 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/457323416' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:59:02.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:02 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/457323416' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:59:02.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:02 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/457323416' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:59:02.642 INFO:tasks.workunit.client.0.smithi070.stderr:Note: switching to '7aec00165b4b073a25396fd0279151d2b2fa23fe'. 2024-06-04T04:59:02.642 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-06-04T04:59:02.642 INFO:tasks.workunit.client.0.smithi070.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-06-04T04:59:02.642 INFO:tasks.workunit.client.0.smithi070.stderr:changes and commit them, and you can discard any commits you make in this 2024-06-04T04:59:02.642 INFO:tasks.workunit.client.0.smithi070.stderr:state without impacting any branches by switching back to a branch. 2024-06-04T04:59:02.642 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-06-04T04:59:02.642 INFO:tasks.workunit.client.0.smithi070.stderr:If you want to create a new branch to retain commits you create, you may 2024-06-04T04:59:02.642 INFO:tasks.workunit.client.0.smithi070.stderr:do so (now or later) by using -c with the switch command. Example: 2024-06-04T04:59:02.642 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-06-04T04:59:02.643 INFO:tasks.workunit.client.0.smithi070.stderr: git switch -c 2024-06-04T04:59:02.643 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-06-04T04:59:02.643 INFO:tasks.workunit.client.0.smithi070.stderr:Or undo this operation with: 2024-06-04T04:59:02.643 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-06-04T04:59:02.643 INFO:tasks.workunit.client.0.smithi070.stderr: git switch - 2024-06-04T04:59:02.643 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-06-04T04:59:02.643 INFO:tasks.workunit.client.0.smithi070.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-06-04T04:59:02.643 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-06-04T04:59:02.643 INFO:tasks.workunit.client.0.smithi070.stderr:HEAD is now at 7aec00165b4 Merge pull request #57616 from Matan-B/wip-matanb-fix-python-tests 2024-06-04T04:59:02.644 DEBUG:teuthology.orchestra.run.smithi070:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-06-04T04:59:02.696 INFO:tasks.workunit.client.0.smithi070.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-06-04T04:59:02.698 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-06-04T04:59:02.698 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-06-04T04:59:02.865 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id f277a116 --release --await' 2024-06-04T04:59:02.865 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id f277a116 --release --await 2024-06-04T04:59:03.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:03 smithi112 ceph-mon[36453]: pgmap v59: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-06-04T04:59:03.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:03 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T04:59:03.474 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:03 smithi070 ceph-mon[32631]: pgmap v59: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-06-04T04:59:03.474 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:03 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T04:59:03.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:03 smithi098 ceph-mon[35569]: pgmap v59: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-06-04T04:59:03.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:03 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T04:59:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.472+0000 7fa56dbe6700 1 -- 172.21.15.70:0/1297071159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568102e00 msgr2=0x7fa56810f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.472+0000 7fa56dbe6700 1 --2- 172.21.15.70:0/1297071159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568102e00 0x7fa56810f2f0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa56000b1a0 tx=0x7fa56000b4b0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.472+0000 7fa56dbe6700 1 -- 172.21.15.70:0/1297071159 shutdown_connections 2024-06-04T04:59:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.472+0000 7fa56dbe6700 1 --2- 172.21.15.70:0/1297071159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568102e00 0x7fa56810f2f0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.472+0000 7fa56dbe6700 1 --2- 172.21.15.70:0/1297071159 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa568108520 0x7fa568108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.472+0000 7fa56dbe6700 1 --2- 172.21.15.70:0/1297071159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa568102440 0x7fa5681028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.472+0000 7fa56dbe6700 1 -- 172.21.15.70:0/1297071159 >> 172.21.15.70:0/1297071159 conn(0x7fa5680793f0 msgr2=0x7fa568079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.472+0000 7fa56dbe6700 1 -- 172.21.15.70:0/1297071159 shutdown_connections 2024-06-04T04:59:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.472+0000 7fa56dbe6700 1 -- 172.21.15.70:0/1297071159 wait complete. 2024-06-04T04:59:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.472+0000 7fa56dbe6700 1 Processor -- start 2024-06-04T04:59:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.473+0000 7fa56dbe6700 1 -- start start 2024-06-04T04:59:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.473+0000 7fa56dbe6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa568102440 0x7fa56819d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.473+0000 7fa56dbe6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568102e00 0x7fa56819db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.473+0000 7fa56dbe6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa568108520 0x7fa568197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.473+0000 7fa56dbe6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa568111410 con 0x7fa568102e00 2024-06-04T04:59:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.473+0000 7fa56dbe6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa568111290 con 0x7fa568108520 2024-06-04T04:59:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.473+0000 7fa56dbe6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa568111590 con 0x7fa568102440 2024-06-04T04:59:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.473+0000 7fa566ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568102e00 0x7fa56819db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.473+0000 7fa566ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568102e00 0x7fa56819db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56280/0 (socket says 172.21.15.70:56280) 2024-06-04T04:59:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.473+0000 7fa566ffd700 1 -- 172.21.15.70:0/3113909173 learned_addr learned my addr 172.21.15.70:0/3113909173 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.473+0000 7fa567fff700 1 --2- 172.21.15.70:0/3113909173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa568108520 0x7fa568197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.473+0000 7fa5677fe700 1 --2- 172.21.15.70:0/3113909173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa568102440 0x7fa56819d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.473+0000 7fa566ffd700 1 -- 172.21.15.70:0/3113909173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa568102440 msgr2=0x7fa56819d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.473+0000 7fa566ffd700 1 --2- 172.21.15.70:0/3113909173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa568102440 0x7fa56819d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.473+0000 7fa566ffd700 1 -- 172.21.15.70:0/3113909173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa568108520 msgr2=0x7fa568197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.473+0000 7fa566ffd700 1 --2- 172.21.15.70:0/3113909173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa568108520 0x7fa568197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.473+0000 7fa566ffd700 1 -- 172.21.15.70:0/3113909173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa568197f70 con 0x7fa568102e00 2024-06-04T04:59:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.474+0000 7fa566ffd700 1 --2- 172.21.15.70:0/3113909173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568102e00 0x7fa56819db00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00c920 tx=0x7fa55c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.474+0000 7fa564ff9700 1 -- 172.21.15.70:0/3113909173 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c004a30 con 0x7fa568102e00 2024-06-04T04:59:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.474+0000 7fa56dbe6700 1 -- 172.21.15.70:0/3113909173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa568198260 con 0x7fa568102e00 2024-06-04T04:59:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.474+0000 7fa564ff9700 1 -- 172.21.15.70:0/3113909173 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa55c0054b0 con 0x7fa568102e00 2024-06-04T04:59:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.474+0000 7fa564ff9700 1 -- 172.21.15.70:0/3113909173 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c007b70 con 0x7fa568102e00 2024-06-04T04:59:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.474+0000 7fa56dbe6700 1 -- 172.21.15.70:0/3113909173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5681a4540 con 0x7fa568102e00 2024-06-04T04:59:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.477+0000 7fa564ff9700 1 -- 172.21.15.70:0/3113909173 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fa55c007d10 con 0x7fa568102e00 2024-06-04T04:59:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.477+0000 7fa564ff9700 1 --2- 172.21.15.70:0/3113909173 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa5500b29f0 0x7fa5500b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.477+0000 7fa5677fe700 1 --2- 172.21.15.70:0/3113909173 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa5500b29f0 0x7fa5500b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.477+0000 7fa564ff9700 1 -- 172.21.15.70:0/3113909173 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fa55c0e2140 con 0x7fa568102e00 2024-06-04T04:59:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa57847bb80 1 Processor -- start 2024-06-04T04:59:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa57847bb80 1 -- start start 2024-06-04T04:59:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa5677fe700 1 --2- 172.21.15.70:0/3113909173 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa5500b29f0 0x7fa5500b4eb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa558005950 tx=0x7fa55800b410 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa57847bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd1c70 0x563c03fd2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa57847bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd2620 0x563c03fd5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa57847bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd6370 0x563c03fd67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa57847bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563c03f2cc90 con 0x563c03fd6370 2024-06-04T04:59:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa57847bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563c03f1c830 con 0x563c03fd2620 2024-06-04T04:59:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa57847bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563c03f20870 con 0x563c03fd1c70 2024-06-04T04:59:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa567fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd6370 0x563c03fd67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa567fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd6370 0x563c03fd67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56288/0 (socket says 172.21.15.70:56288) 2024-06-04T04:59:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa567fff700 1 -- 172.21.15.70:0/1213803312 learned_addr learned my addr 172.21.15.70:0/1213803312 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa5677fe700 1 --2- 172.21.15.70:0/1213803312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd1c70 0x563c03fd2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa566ffd700 1 --2- 172.21.15.70:0/1213803312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd2620 0x563c03fd5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa567fff700 1 -- 172.21.15.70:0/1213803312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd1c70 msgr2=0x563c03fd2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa567fff700 1 --2- 172.21.15.70:0/1213803312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd1c70 0x563c03fd2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa567fff700 1 -- 172.21.15.70:0/1213803312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd2620 msgr2=0x563c03fd5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa567fff700 1 --2- 172.21.15.70:0/1213803312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd2620 0x563c03fd5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa567fff700 1 -- 172.21.15.70:0/1213803312 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c03e6c8d0 con 0x563c03fd6370 2024-06-04T04:59:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa566ffd700 1 --2- 172.21.15.70:0/1213803312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd2620 0x563c03fd5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:59:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.478+0000 7fa567fff700 1 --2- 172.21.15.70:0/1213803312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd6370 0x563c03fd67b0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa560007590 tx=0x7fa5600075c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc8fad3ea74c586 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.479+0000 7fa54dffb700 1 -- 172.21.15.70:0/1213803312 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5600092f0 con 0x563c03fd6370 2024-06-04T04:59:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.479+0000 7fa54dffb700 1 -- 172.21.15.70:0/1213803312 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa560009490 con 0x563c03fd6370 2024-06-04T04:59:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.479+0000 7fa54dffb700 1 -- 172.21.15.70:0/1213803312 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5600116c0 con 0x563c03fd6370 2024-06-04T04:59:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.479+0000 7fa57847bb80 1 -- 172.21.15.70:0/1213803312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd6370 msgr2=0x563c03fd67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.479+0000 7fa57847bb80 1 --2- 172.21.15.70:0/1213803312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd6370 0x563c03fd67b0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa560007590 tx=0x7fa5600075c0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.479+0000 7fa57847bb80 1 -- 172.21.15.70:0/1213803312 shutdown_connections 2024-06-04T04:59:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.479+0000 7fa57847bb80 1 --2- 172.21.15.70:0/1213803312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd6370 0x563c03fd67b0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.479+0000 7fa57847bb80 1 --2- 172.21.15.70:0/1213803312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd1c70 0x563c03fd2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.479+0000 7fa57847bb80 1 --2- 172.21.15.70:0/1213803312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd2620 0x563c03fd5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.479+0000 7fa57847bb80 1 -- 172.21.15.70:0/1213803312 >> 172.21.15.70:0/1213803312 conn(0x563c03efb080 msgr2=0x563c03fc7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.479+0000 7fa57847bb80 1 -- 172.21.15.70:0/1213803312 shutdown_connections 2024-06-04T04:59:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.479+0000 7fa57847bb80 1 -- 172.21.15.70:0/1213803312 wait complete. 2024-06-04T04:59:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.480+0000 7fa57847bb80 1 -- start start 2024-06-04T04:59:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.480+0000 7fa57847bb80 1 Processor -- start 2024-06-04T04:59:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.480+0000 7fa57847bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd1c70 0x563c03fdd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.480+0000 7fa57847bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd2620 0x563c03fdaa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.480+0000 7fa57847bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd6370 0x563c03fdaf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.480+0000 7fa57847bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563c03e22fa0 con 0x563c03fd2620 2024-06-04T04:59:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.480+0000 7fa57847bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563c03ebe7c0 con 0x563c03fd1c70 2024-06-04T04:59:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.480+0000 7fa57847bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563c03fde4d0 con 0x563c03fd6370 2024-06-04T04:59:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.480+0000 7fa57847bb80 10 client.0 _ll_register_callbacks cb 0x563c03eb7920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa566ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd2620 0x563c03fdaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa566ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd2620 0x563c03fdaa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56298/0 (socket says 172.21.15.70:56298) 2024-06-04T04:59:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa566ffd700 1 -- 172.21.15.70:0/2642364463 learned_addr learned my addr 172.21.15.70:0/2642364463 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa567fff700 1 --2- 172.21.15.70:0/2642364463 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd6370 0x563c03fdaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa566ffd700 1 -- 172.21.15.70:0/2642364463 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd6370 msgr2=0x563c03fdaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa566ffd700 1 --2- 172.21.15.70:0/2642364463 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd6370 0x563c03fdaf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa5677fe700 1 --2- 172.21.15.70:0/2642364463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd1c70 0x563c03fdd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa566ffd700 1 -- 172.21.15.70:0/2642364463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd1c70 msgr2=0x563c03fdd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa566ffd700 1 --2- 172.21.15.70:0/2642364463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd1c70 0x563c03fdd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa566ffd700 1 -- 172.21.15.70:0/2642364463 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c03ebb690 con 0x563c03fd2620 2024-06-04T04:59:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa567fff700 1 --2- 172.21.15.70:0/2642364463 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd6370 0x563c03fdaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:59:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa5677fe700 1 --2- 172.21.15.70:0/2642364463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd1c70 0x563c03fdd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:59:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa566ffd700 1 --2- 172.21.15.70:0/2642364463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd2620 0x563c03fdaa40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa55c0a93f0 tx=0x7fa55c0a8f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a387092f61acc1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa546ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:59:03.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa546ffd700 1 -- 172.21.15.70:0/2642364463 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c0c3060 con 0x563c03fd2620 2024-06-04T04:59:03.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa546ffd700 1 -- 172.21.15.70:0/2642364463 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fa55c0a98d0 con 0x563c03fd2620 2024-06-04T04:59:03.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa57847bb80 1 -- 172.21.15.70:0/2642364463 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa560009490 con 0x563c03fd2620 2024-06-04T04:59:03.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa546ffd700 1 -- 172.21.15.70:0/2642364463 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa55c01d030 con 0x563c03fd2620 2024-06-04T04:59:03.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa57847bb80 10 client.15978 fetch_fsmap learned FSMap version 29 2024-06-04T04:59:03.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa57847bb80 1 -- 172.21.15.70:0/2642364463 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563c03fe8be0 con 0x563c03fd2620 2024-06-04T04:59:03.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.481+0000 7fa546ffd700 1 -- 172.21.15.70:0/2642364463 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa560009490 con 0x563c03fd2620 2024-06-04T04:59:03.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.482+0000 7fa546ffd700 1 -- 172.21.15.70:0/2642364463 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fa55c0a3b80 con 0x563c03fd2620 2024-06-04T04:59:03.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.482+0000 7fa57847bb80 10 client.15978 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:59:03.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.482+0000 7fa57847bb80 10 client.15978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:59:03.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.482+0000 7fa57847bb80 20 client.15978 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:03.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.482+0000 7fa57847bb80 1 --2- 172.21.15.70:0/2642364463 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563c03feb170 0x563c03fed560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.482+0000 7fa57847bb80 4 client.15978 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:59:03.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.482+0000 7fa57847bb80 1 -- 172.21.15.70:0/2642364463 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c03f999e0 con 0x563c03feb170 2024-06-04T04:59:03.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.482+0000 7fa5677fe700 1 --2- 172.21.15.70:0/2642364463 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563c03feb170 0x563c03fed560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:03.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.482+0000 7fa546ffd700 10 client.15978 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:03.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.482+0000 7fa5677fe700 1 --2- 172.21.15.70:0/2642364463 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563c03feb170 0x563c03fed560 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f55499bf352df5b server_cookie=f3916bb2d22986b8 in_seq=0 out_seq=0 2024-06-04T04:59:03.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.483+0000 7fa546ffd700 1 -- 172.21.15.70:0/2642364463 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563c03f999e0 con 0x563c03feb170 2024-06-04T04:59:03.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.483+0000 7fa546ffd700 10 client.15978 handle_command_reply: tid=0 2024-06-04T04:59:03.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.483+0000 7fa57847bb80 1 client.15978 shutdown 2024-06-04T04:59:03.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.483+0000 7fa57847bb80 1 -- 172.21.15.70:0/2642364463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd2620 msgr2=0x563c03fdaa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:03.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.483+0000 7fa57847bb80 1 --2- 172.21.15.70:0/2642364463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd2620 0x563c03fdaa40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa55c0a93f0 tx=0x7fa55c0a8f70 comp rx=0 tx=0).stop 2024-06-04T04:59:03.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.483+0000 7fa57847bb80 1 -- 172.21.15.70:0/2642364463 shutdown_connections 2024-06-04T04:59:03.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.483+0000 7fa57847bb80 1 --2- 172.21.15.70:0/2642364463 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd6370 0x563c03fdaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.483+0000 7fa57847bb80 1 --2- 172.21.15.70:0/2642364463 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563c03feb170 0x563c03fed560 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.483+0000 7fa57847bb80 1 --2- 172.21.15.70:0/2642364463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd1c70 0x563c03fdd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.483+0000 7fa546ffd700 0 client.15978 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:03.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.483+0000 7fa57847bb80 1 --2- 172.21.15.70:0/2642364463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd2620 0x563c03fdaa40 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.483+0000 7fa57847bb80 1 -- 172.21.15.70:0/2642364463 >> 172.21.15.70:0/2642364463 conn(0x563c03efb080 msgr2=0x563c03fc7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:03.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.483+0000 7fa57847bb80 1 -- 172.21.15.70:0/2642364463 shutdown_connections 2024-06-04T04:59:03.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.483+0000 7fa57847bb80 1 -- 172.21.15.70:0/2642364463 wait complete. 2024-06-04T04:59:03.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.483+0000 7fa57847bb80 20 client.15978 trim_cache size 0 max 16384 2024-06-04T04:59:03.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.504+0000 7fa57847bb80 1 Processor -- start 2024-06-04T04:59:03.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.504+0000 7fa57847bb80 1 -- start start 2024-06-04T04:59:03.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.504+0000 7fa57847bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd1c70 0x563c03fd5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.504+0000 7fa57847bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd2620 0x563c03fc7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.504+0000 7fa57847bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd6370 0x563c03ff2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.504+0000 7fa57847bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa5681aba80 con 0x563c03fd1c70 2024-06-04T04:59:03.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.504+0000 7fa57847bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa5681ad5c0 con 0x563c03fd6370 2024-06-04T04:59:03.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.504+0000 7fa57847bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa5681ac150 con 0x563c03fd2620 2024-06-04T04:59:03.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.504+0000 7fa5677fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd1c70 0x563c03fd5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:03.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.504+0000 7fa5677fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd1c70 0x563c03fd5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56308/0 (socket says 172.21.15.70:56308) 2024-06-04T04:59:03.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.504+0000 7fa5677fe700 1 -- 172.21.15.70:0/809554911 learned_addr learned my addr 172.21.15.70:0/809554911 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:03.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.504+0000 7fa566ffd700 1 --2- 172.21.15.70:0/809554911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd2620 0x563c03fc7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:03.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.504+0000 7fa5677fe700 1 -- 172.21.15.70:0/809554911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd2620 msgr2=0x563c03fc7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:03.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.504+0000 7fa5677fe700 1 --2- 172.21.15.70:0/809554911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd2620 0x563c03fc7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.504+0000 7fa5677fe700 1 -- 172.21.15.70:0/809554911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd6370 msgr2=0x563c03ff2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:03.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.504+0000 7fa5677fe700 1 --2- 172.21.15.70:0/809554911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd6370 0x563c03ff2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.504+0000 7fa5677fe700 1 -- 172.21.15.70:0/809554911 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c03fc8940 con 0x563c03fd1c70 2024-06-04T04:59:03.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.505+0000 7fa5677fe700 1 --2- 172.21.15.70:0/809554911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd1c70 0x563c03fd5030 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x563c03feddd0 tx=0x7fa558013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4edd40ff8438dfd9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:03.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.505+0000 7fa54d7fa700 1 -- 172.21.15.70:0/809554911 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558020110 con 0x563c03fd1c70 2024-06-04T04:59:03.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.505+0000 7fa54d7fa700 1 -- 172.21.15.70:0/809554911 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa558020d60 con 0x563c03fd1c70 2024-06-04T04:59:03.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.505+0000 7fa54d7fa700 1 -- 172.21.15.70:0/809554911 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558005020 con 0x563c03fd1c70 2024-06-04T04:59:03.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.505+0000 7fa57847bb80 1 -- 172.21.15.70:0/809554911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd1c70 msgr2=0x563c03fd5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:03.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.505+0000 7fa57847bb80 1 --2- 172.21.15.70:0/809554911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd1c70 0x563c03fd5030 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x563c03feddd0 tx=0x7fa558013000 comp rx=0 tx=0).stop 2024-06-04T04:59:03.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.506+0000 7fa57847bb80 1 -- 172.21.15.70:0/809554911 shutdown_connections 2024-06-04T04:59:03.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.506+0000 7fa57847bb80 1 --2- 172.21.15.70:0/809554911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd6370 0x563c03ff2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.506+0000 7fa57847bb80 1 --2- 172.21.15.70:0/809554911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd1c70 0x563c03fd5030 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.506+0000 7fa57847bb80 1 --2- 172.21.15.70:0/809554911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd2620 0x563c03fc7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.506+0000 7fa57847bb80 1 -- 172.21.15.70:0/809554911 >> 172.21.15.70:0/809554911 conn(0x563c03efb080 msgr2=0x563c03fccf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:03.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.506+0000 7fa57847bb80 1 -- 172.21.15.70:0/809554911 shutdown_connections 2024-06-04T04:59:03.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.506+0000 7fa57847bb80 1 -- 172.21.15.70:0/809554911 wait complete. 2024-06-04T04:59:03.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.506+0000 7fa57847bb80 1 -- start start 2024-06-04T04:59:03.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.506+0000 7fa57847bb80 1 Processor -- start 2024-06-04T04:59:03.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.506+0000 7fa57847bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd1c70 0x563c03fe2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.506+0000 7fa57847bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd2620 0x563c03fe2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.506+0000 7fa57847bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd6370 0x563c04055d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.506+0000 7fa57847bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa5681acef0 con 0x563c03fd1c70 2024-06-04T04:59:03.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.506+0000 7fa57847bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa5681ac820 con 0x563c03fd2620 2024-06-04T04:59:03.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.506+0000 7fa57847bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa5681a9a20 con 0x563c03fd6370 2024-06-04T04:59:03.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa57847bb80 10 client.0 _ll_register_callbacks cb 0x563c03eb7920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:03.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa5677fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd1c70 0x563c03fe2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:03.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa5677fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd1c70 0x563c03fe2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56314/0 (socket says 172.21.15.70:56314) 2024-06-04T04:59:03.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa5677fe700 1 -- 172.21.15.70:0/2252856181 learned_addr learned my addr 172.21.15.70:0/2252856181 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:03.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa567fff700 1 --2- 172.21.15.70:0/2252856181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd6370 0x563c04055d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:03.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa566ffd700 1 --2- 172.21.15.70:0/2252856181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd2620 0x563c03fe2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:03.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa5677fe700 1 -- 172.21.15.70:0/2252856181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd6370 msgr2=0x563c04055d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:03.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa5677fe700 1 --2- 172.21.15.70:0/2252856181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd6370 0x563c04055d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa5677fe700 1 -- 172.21.15.70:0/2252856181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd2620 msgr2=0x563c03fe2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:03.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa5677fe700 1 --2- 172.21.15.70:0/2252856181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd2620 0x563c03fe2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa5677fe700 1 -- 172.21.15.70:0/2252856181 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c03fe3670 con 0x563c03fd1c70 2024-06-04T04:59:03.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa567fff700 1 --2- 172.21.15.70:0/2252856181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd6370 0x563c04055d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:59:03.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa5677fe700 1 --2- 172.21.15.70:0/2252856181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd1c70 0x563c03fe2850 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa558004ea0 tx=0x7fa5580203d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b0d2a5cce942d7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:03.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa546ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:59:03.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa546ffd700 1 -- 172.21.15.70:0/2252856181 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558004960 con 0x563c03fd1c70 2024-06-04T04:59:03.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa546ffd700 1 -- 172.21.15.70:0/2252856181 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fa558005000 con 0x563c03fd1c70 2024-06-04T04:59:03.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa57847bb80 1 -- 172.21.15.70:0/2252856181 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa558020d60 con 0x563c03fd1c70 2024-06-04T04:59:03.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa546ffd700 1 -- 172.21.15.70:0/2252856181 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa558005200 con 0x563c03fd1c70 2024-06-04T04:59:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa57847bb80 10 client.15984 fetch_fsmap learned FSMap version 29 2024-06-04T04:59:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.507+0000 7fa57847bb80 1 -- 172.21.15.70:0/2252856181 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563c04057380 con 0x563c03fd1c70 2024-06-04T04:59:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.508+0000 7fa546ffd700 1 -- 172.21.15.70:0/2252856181 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa558020d60 con 0x563c03fd1c70 2024-06-04T04:59:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.508+0000 7fa546ffd700 1 -- 172.21.15.70:0/2252856181 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fa5580309b0 con 0x563c03fd1c70 2024-06-04T04:59:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.508+0000 7fa57847bb80 10 client.15984 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:59:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.508+0000 7fa57847bb80 10 client.15984 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:59:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.508+0000 7fa57847bb80 20 client.15984 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.508+0000 7fa57847bb80 1 --2- 172.21.15.70:0/2252856181 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563c040579f0 0x563c04059e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.508+0000 7fa57847bb80 4 client.15984 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T04:59:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.508+0000 7fa57847bb80 1 -- 172.21.15.70:0/2252856181 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563c03f2ebe0 con 0x563c040579f0 2024-06-04T04:59:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.508+0000 7fa566ffd700 1 --2- 172.21.15.70:0/2252856181 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563c040579f0 0x563c04059e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.508+0000 7fa566ffd700 1 --2- 172.21.15.70:0/2252856181 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563c040579f0 0x563c04059e10 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba4056c9c30540a1 server_cookie=d39258fa0bd90fc6 in_seq=0 out_seq=0 2024-06-04T04:59:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.508+0000 7fa546ffd700 10 client.15984 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.509+0000 7fa546ffd700 1 -- 172.21.15.70:0/2252856181 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563c03f2ebe0 con 0x563c040579f0 2024-06-04T04:59:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.509+0000 7fa546ffd700 10 client.15984 handle_command_reply: tid=0 2024-06-04T04:59:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.509+0000 7fa57847bb80 1 client.15984 shutdown 2024-06-04T04:59:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.509+0000 7fa57847bb80 1 -- 172.21.15.70:0/2252856181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd1c70 msgr2=0x563c03fe2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.509+0000 7fa57847bb80 1 --2- 172.21.15.70:0/2252856181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd1c70 0x563c03fe2850 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa558004ea0 tx=0x7fa5580203d0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.509+0000 7fa57847bb80 1 -- 172.21.15.70:0/2252856181 shutdown_connections 2024-06-04T04:59:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.509+0000 7fa57847bb80 1 --2- 172.21.15.70:0/2252856181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563c03fd6370 0x563c04055d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.509+0000 7fa57847bb80 1 --2- 172.21.15.70:0/2252856181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c03fd1c70 0x563c03fe2850 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.509+0000 7fa57847bb80 1 --2- 172.21.15.70:0/2252856181 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563c040579f0 0x563c04059e10 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.509+0000 7fa57847bb80 1 --2- 172.21.15.70:0/2252856181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c03fd2620 0x563c03fe2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.509+0000 7fa57847bb80 1 -- 172.21.15.70:0/2252856181 >> 172.21.15.70:0/2252856181 conn(0x563c03efb080 msgr2=0x563c03fdc270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.509+0000 7fa546ffd700 0 client.15984 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.509+0000 7fa57847bb80 1 -- 172.21.15.70:0/2252856181 shutdown_connections 2024-06-04T04:59:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.509+0000 7fa57847bb80 1 -- 172.21.15.70:0/2252856181 wait complete. 2024-06-04T04:59:03.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.509+0000 7fa57847bb80 20 client.15984 trim_cache size 0 max 16384 2024-06-04T04:59:03.522 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:59:03.522 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T04:59:03.522 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T04:59:03.522 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:59:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.510+0000 7fa56dbe6700 1 -- 172.21.15.70:0/3113909173 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa5500b29f0 msgr2=0x7fa5500b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.510+0000 7fa56dbe6700 1 --2- 172.21.15.70:0/3113909173 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa5500b29f0 0x7fa5500b4eb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa558005950 tx=0x7fa55800b410 comp rx=0 tx=0).stop 2024-06-04T04:59:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.510+0000 7fa56dbe6700 1 -- 172.21.15.70:0/3113909173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568102e00 msgr2=0x7fa56819db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.510+0000 7fa56dbe6700 1 --2- 172.21.15.70:0/3113909173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568102e00 0x7fa56819db00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00c920 tx=0x7fa55c00cc30 comp rx=0 tx=0).stop 2024-06-04T04:59:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.510+0000 7fa56dbe6700 1 -- 172.21.15.70:0/3113909173 shutdown_connections 2024-06-04T04:59:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.510+0000 7fa56dbe6700 1 --2- 172.21.15.70:0/3113909173 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa5500b29f0 0x7fa5500b4eb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.510+0000 7fa56dbe6700 1 --2- 172.21.15.70:0/3113909173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa568108520 0x7fa568197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.510+0000 7fa56dbe6700 1 --2- 172.21.15.70:0/3113909173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa568102440 0x7fa56819d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.510+0000 7fa56dbe6700 1 --2- 172.21.15.70:0/3113909173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568102e00 0x7fa56819db00 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.510+0000 7fa56dbe6700 1 -- 172.21.15.70:0/3113909173 >> 172.21.15.70:0/3113909173 conn(0x7fa5680793f0 msgr2=0x7fa5681005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.510+0000 7fa56dbe6700 1 -- 172.21.15.70:0/3113909173 shutdown_connections 2024-06-04T04:59:03.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:03.510+0000 7fa56dbe6700 1 -- 172.21.15.70:0/3113909173 wait complete. 2024-06-04T04:59:04.412 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T04:59:04.413 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T04:59:04.443 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:59:04.762 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-06-04T04:59:04.851 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-06-04T04:59:04.891 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-06-04T04:59:04.893 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-06-04T04:59:04.893 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-06-04T04:59:04.932 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-06-04T04:59:04.936 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T04:59:04.936 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-06-04T04:59:04.992 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-06-04T04:59:04.993 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-06-04T04:59:04.994 DEBUG:teuthology.orchestra.run.smithi070:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7aec00165b4b073a25396fd0279151d2b2fa23fe TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-06-04T04:59:05.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:05 smithi112 ceph-mon[36453]: pgmap v60: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-06-04T04:59:05.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:05 smithi098 ceph-mon[35569]: pgmap v60: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-06-04T04:59:05.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:05 smithi070 ceph-mon[32631]: pgmap v60: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-06-04T04:59:05.824 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:59:07.403 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:07 smithi070 ceph-mon[32631]: pgmap v61: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-06-04T04:59:07.446 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:07 smithi112 ceph-mon[36453]: pgmap v61: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-06-04T04:59:07.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:07 smithi098 ceph-mon[35569]: pgmap v61: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-06-04T04:59:07.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.510+0000 7fb9aa4a6700 1 -- 172.21.15.70:0/2837090668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9a4102e10 msgr2=0x7fb9a410f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:07.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.510+0000 7fb9aa4a6700 1 --2- 172.21.15.70:0/2837090668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9a4102e10 0x7fb9a410f2f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb99c00b0a0 tx=0x7fb99c00b3b0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.510+0000 7fb9aa4a6700 1 -- 172.21.15.70:0/2837090668 shutdown_connections 2024-06-04T04:59:07.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.510+0000 7fb9aa4a6700 1 --2- 172.21.15.70:0/2837090668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9a4102e10 0x7fb9a410f2f0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.510+0000 7fb9aa4a6700 1 --2- 172.21.15.70:0/2837090668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9a41084b0 0x7fb9a4108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.510+0000 7fb9aa4a6700 1 --2- 172.21.15.70:0/2837090668 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9a4102470 0x7fb9a41028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.510+0000 7fb9aa4a6700 1 -- 172.21.15.70:0/2837090668 >> 172.21.15.70:0/2837090668 conn(0x7fb9a40793e0 msgr2=0x7fb9a40797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:07.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.510+0000 7fb9aa4a6700 1 -- 172.21.15.70:0/2837090668 shutdown_connections 2024-06-04T04:59:07.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.510+0000 7fb9aa4a6700 1 -- 172.21.15.70:0/2837090668 wait complete. 2024-06-04T04:59:07.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb9aa4a6700 1 Processor -- start 2024-06-04T04:59:07.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb9aa4a6700 1 -- start start 2024-06-04T04:59:07.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb9aa4a6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9a4102470 0x7fb9a419f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb9aa4a6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9a4102e10 0x7fb9a419fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb9aa4a6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9a41084b0 0x7fb9a4199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb9aa4a6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a4112550 con 0x7fb9a4102e10 2024-06-04T04:59:07.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb9aa4a6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41123d0 con 0x7fb9a41084b0 2024-06-04T04:59:07.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb9aa4a6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41126d0 con 0x7fb9a4102470 2024-06-04T04:59:07.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb99bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9a4102e10 0x7fb9a419fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb99bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9a4102e10 0x7fb9a419fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44196/0 (socket says 172.21.15.70:44196) 2024-06-04T04:59:07.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb99bfff700 1 -- 172.21.15.70:0/4201728315 learned_addr learned my addr 172.21.15.70:0/4201728315 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:07.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb9a3fff700 1 --2- 172.21.15.70:0/4201728315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9a4102470 0x7fb9a419f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb9a8a43700 1 --2- 172.21.15.70:0/4201728315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9a41084b0 0x7fb9a4199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb99bfff700 1 -- 172.21.15.70:0/4201728315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9a4102470 msgr2=0x7fb9a419f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:07.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb99bfff700 1 --2- 172.21.15.70:0/4201728315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9a4102470 0x7fb9a419f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb99bfff700 1 -- 172.21.15.70:0/4201728315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9a41084b0 msgr2=0x7fb9a4199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:07.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb99bfff700 1 --2- 172.21.15.70:0/4201728315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9a41084b0 0x7fb9a4199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb99bfff700 1 -- 172.21.15.70:0/4201728315 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a419a200 con 0x7fb9a4102e10 2024-06-04T04:59:07.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb9a8a43700 1 --2- 172.21.15.70:0/4201728315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9a41084b0 0x7fb9a4199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:59:07.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.511+0000 7fb99bfff700 1 --2- 172.21.15.70:0/4201728315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9a4102e10 0x7fb9a419fe20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb98c00b530 tx=0x7fb98c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:07.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.512+0000 7fb9a1ffb700 1 -- 172.21.15.70:0/4201728315 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c015020 con 0x7fb9a4102e10 2024-06-04T04:59:07.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.512+0000 7fb9aa4a6700 1 -- 172.21.15.70:0/4201728315 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a419a490 con 0x7fb9a4102e10 2024-06-04T04:59:07.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.512+0000 7fb9a1ffb700 1 -- 172.21.15.70:0/4201728315 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb98c011cc0 con 0x7fb9a4102e10 2024-06-04T04:59:07.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.512+0000 7fb9a1ffb700 1 -- 172.21.15.70:0/4201728315 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c010d30 con 0x7fb9a4102e10 2024-06-04T04:59:07.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.512+0000 7fb9aa4a6700 1 -- 172.21.15.70:0/4201728315 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a41a67b0 con 0x7fb9a4102e10 2024-06-04T04:59:07.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.515+0000 7fb9a1ffb700 1 -- 172.21.15.70:0/4201728315 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fb98c018020 con 0x7fb9a4102e10 2024-06-04T04:59:07.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.515+0000 7fb9a1ffb700 1 --2- 172.21.15.70:0/4201728315 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9900b29f0 0x7fb9900b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.515+0000 7fb9a3fff700 1 --2- 172.21.15.70:0/4201728315 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9900b29f0 0x7fb9900b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.516+0000 7fb9a1ffb700 1 -- 172.21.15.70:0/4201728315 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb98c0e1d90 con 0x7fb9a4102e10 2024-06-04T04:59:07.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.516+0000 7fb9b4d3bb80 1 Processor -- start 2024-06-04T04:59:07.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.516+0000 7fb9b4d3bb80 1 -- start start 2024-06-04T04:59:07.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.516+0000 7fb9a3fff700 1 --2- 172.21.15.70:0/4201728315 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9900b29f0 0x7fb9900b4eb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb994006fd0 tx=0x7fb994008040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:07.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.516+0000 7fb9b4d3bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb392b0 0x5597bbb39690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.516+0000 7fb9b4d3bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb338e0 0x5597bbb33d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.516+0000 7fb9b4d3bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb342a0 0x5597bbb400e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.516+0000 7fb9b4d3bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5597bba92d90 con 0x5597bbb342a0 2024-06-04T04:59:07.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.516+0000 7fb9b4d3bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5597bba82930 con 0x5597bbb338e0 2024-06-04T04:59:07.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.516+0000 7fb9b4d3bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5597bba86970 con 0x5597bbb392b0 2024-06-04T04:59:07.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.516+0000 7fb9a8a43700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb342a0 0x5597bbb400e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.516+0000 7fb9a8a43700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb342a0 0x5597bbb400e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44204/0 (socket says 172.21.15.70:44204) 2024-06-04T04:59:07.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.516+0000 7fb9a8a43700 1 -- 172.21.15.70:0/486457314 learned_addr learned my addr 172.21.15.70:0/486457314 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:07.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.516+0000 7fb9a3fff700 1 --2- 172.21.15.70:0/486457314 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb392b0 0x5597bbb39690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.516+0000 7fb99bfff700 1 --2- 172.21.15.70:0/486457314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb338e0 0x5597bbb33d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.517+0000 7fb9a8a43700 1 -- 172.21.15.70:0/486457314 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb392b0 msgr2=0x5597bbb39690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:07.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.517+0000 7fb9a8a43700 1 --2- 172.21.15.70:0/486457314 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb392b0 0x5597bbb39690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.517+0000 7fb9a8a43700 1 -- 172.21.15.70:0/486457314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb338e0 msgr2=0x5597bbb33d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:07.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.517+0000 7fb9a8a43700 1 --2- 172.21.15.70:0/486457314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb338e0 0x5597bbb33d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.517+0000 7fb9a8a43700 1 -- 172.21.15.70:0/486457314 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597bb9d29d0 con 0x5597bbb342a0 2024-06-04T04:59:07.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.517+0000 7fb9a8a43700 1 --2- 172.21.15.70:0/486457314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb342a0 0x5597bbb400e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fb99c006130 tx=0x7fb99c009150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788a554ca1a4d326 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:07.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.517+0000 7fb99a7fc700 1 -- 172.21.15.70:0/486457314 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c004a50 con 0x5597bbb342a0 2024-06-04T04:59:07.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.517+0000 7fb99a7fc700 1 -- 172.21.15.70:0/486457314 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb99c004bf0 con 0x5597bbb342a0 2024-06-04T04:59:07.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.517+0000 7fb99a7fc700 1 -- 172.21.15.70:0/486457314 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c0078e0 con 0x5597bbb342a0 2024-06-04T04:59:07.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.518+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/486457314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb342a0 msgr2=0x5597bbb400e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:07.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.518+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/486457314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb342a0 0x5597bbb400e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fb99c006130 tx=0x7fb99c009150 comp rx=0 tx=0).stop 2024-06-04T04:59:07.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.518+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/486457314 shutdown_connections 2024-06-04T04:59:07.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.518+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/486457314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb342a0 0x5597bbb400e0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.518+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/486457314 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb392b0 0x5597bbb39690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.518+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/486457314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb338e0 0x5597bbb33d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.518+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/486457314 >> 172.21.15.70:0/486457314 conn(0x5597bba61180 msgr2=0x5597bbb2e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:07.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.518+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/486457314 shutdown_connections 2024-06-04T04:59:07.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.518+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/486457314 wait complete. 2024-06-04T04:59:07.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.518+0000 7fb9b4d3bb80 1 -- start start 2024-06-04T04:59:07.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.518+0000 7fb9b4d3bb80 1 Processor -- start 2024-06-04T04:59:07.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.518+0000 7fb9b4d3bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb338e0 0x5597bbb3fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.518+0000 7fb9b4d3bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb342a0 0x5597bbb3d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.518+0000 7fb9b4d3bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb392b0 0x5597bbb3d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9b4d3bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5597bb9890a0 con 0x5597bbb392b0 2024-06-04T04:59:07.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9b4d3bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5597bba248c0 con 0x5597bbb342a0 2024-06-04T04:59:07.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9b4d3bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5597bbb40680 con 0x5597bbb338e0 2024-06-04T04:59:07.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9b4d3bb80 10 client.0 _ll_register_callbacks cb 0x5597bba1da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:07.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9a8a43700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb392b0 0x5597bbb3d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9a8a43700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb392b0 0x5597bbb3d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44208/0 (socket says 172.21.15.70:44208) 2024-06-04T04:59:07.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9a8a43700 1 -- 172.21.15.70:0/1363411519 learned_addr learned my addr 172.21.15.70:0/1363411519 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:07.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb99bfff700 1 --2- 172.21.15.70:0/1363411519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb342a0 0x5597bbb3d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9a3fff700 1 --2- 172.21.15.70:0/1363411519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb338e0 0x5597bbb3fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9a8a43700 1 -- 172.21.15.70:0/1363411519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb338e0 msgr2=0x5597bbb3fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:07.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9a8a43700 1 --2- 172.21.15.70:0/1363411519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb338e0 0x5597bbb3fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9a8a43700 1 -- 172.21.15.70:0/1363411519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb342a0 msgr2=0x5597bbb3d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:07.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9a8a43700 1 --2- 172.21.15.70:0/1363411519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb342a0 0x5597bbb3d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9a8a43700 1 -- 172.21.15.70:0/1363411519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597bba21790 con 0x5597bbb392b0 2024-06-04T04:59:07.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9a3fff700 1 --2- 172.21.15.70:0/1363411519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb338e0 0x5597bbb3fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:59:07.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9a8a43700 1 --2- 172.21.15.70:0/1363411519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb392b0 0x5597bbb3d650 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb99c009a20 tx=0x7fb99c01de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa1049669e56db server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:07.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9837fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:59:07.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9837fe700 1 -- 172.21.15.70:0/1363411519 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c02d070 con 0x5597bbb392b0 2024-06-04T04:59:07.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9837fe700 1 -- 172.21.15.70:0/1363411519 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb99c0078b0 con 0x5597bbb392b0 2024-06-04T04:59:07.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/1363411519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb99c004bf0 con 0x5597bbb392b0 2024-06-04T04:59:07.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9837fe700 1 -- 172.21.15.70:0/1363411519 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb99c007ab0 con 0x5597bbb392b0 2024-06-04T04:59:07.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9b4d3bb80 10 client.15996 fetch_fsmap learned FSMap version 29 2024-06-04T04:59:07.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.519+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/1363411519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5597bbb4add0 con 0x5597bbb392b0 2024-06-04T04:59:07.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.520+0000 7fb9837fe700 1 -- 172.21.15.70:0/1363411519 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb99c004bf0 con 0x5597bbb392b0 2024-06-04T04:59:07.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.520+0000 7fb9837fe700 1 -- 172.21.15.70:0/1363411519 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fb99c011070 con 0x5597bbb392b0 2024-06-04T04:59:07.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.520+0000 7fb9b4d3bb80 10 client.15996 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:59:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.520+0000 7fb9b4d3bb80 10 client.15996 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T04:59:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.520+0000 7fb9b4d3bb80 20 client.15996 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.520+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/1363411519 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5597bbb4d2a0 0x5597bbb4f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.520+0000 7fb9b4d3bb80 4 client.15996 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:59:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.520+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/1363411519 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597bbaffae0 con 0x5597bbb4d2a0 2024-06-04T04:59:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.520+0000 7fb9a3fff700 1 --2- 172.21.15.70:0/1363411519 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5597bbb4d2a0 0x5597bbb4f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.521+0000 7fb9a3fff700 1 --2- 172.21.15.70:0/1363411519 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5597bbb4d2a0 0x5597bbb4f690 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1855e9e9ff88b641 server_cookie=8ff1fd5a709bf18 in_seq=0 out_seq=0 2024-06-04T04:59:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.521+0000 7fb9837fe700 10 client.15996 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.521+0000 7fb9837fe700 1 -- 172.21.15.70:0/1363411519 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5597bbaffae0 con 0x5597bbb4d2a0 2024-06-04T04:59:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.521+0000 7fb9837fe700 10 client.15996 handle_command_reply: tid=0 2024-06-04T04:59:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.521+0000 7fb9b4d3bb80 1 client.15996 shutdown 2024-06-04T04:59:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.522+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/1363411519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb392b0 msgr2=0x5597bbb3d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.522+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/1363411519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb392b0 0x5597bbb3d650 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb99c009a20 tx=0x7fb99c01de40 comp rx=0 tx=0).stop 2024-06-04T04:59:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.522+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/1363411519 shutdown_connections 2024-06-04T04:59:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.522+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/1363411519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb392b0 0x5597bbb3d650 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.522+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/1363411519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb338e0 0x5597bbb3fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.522+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/1363411519 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5597bbb4d2a0 0x5597bbb4f690 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.522+0000 7fb9837fe700 0 client.15996 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.522+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/1363411519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb342a0 0x5597bbb3d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.522+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/1363411519 >> 172.21.15.70:0/1363411519 conn(0x5597bba61180 msgr2=0x5597bbb28ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.522+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/1363411519 shutdown_connections 2024-06-04T04:59:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.522+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/1363411519 wait complete. 2024-06-04T04:59:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.522+0000 7fb9b4d3bb80 20 client.15996 trim_cache size 0 max 16384 2024-06-04T04:59:07.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.542+0000 7fb9b4d3bb80 1 Processor -- start 2024-06-04T04:59:07.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.542+0000 7fb9b4d3bb80 1 -- start start 2024-06-04T04:59:07.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.542+0000 7fb9b4d3bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb338e0 0x5597bbb3ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.542+0000 7fb9b4d3bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb342a0 0x5597bbb3f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.542+0000 7fb9b4d3bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb392b0 0x5597bbb48910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb9b4d3bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41abaf0 con 0x5597bbb338e0 2024-06-04T04:59:07.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb9b4d3bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41ad630 con 0x5597bbb392b0 2024-06-04T04:59:07.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb9b4d3bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41ac1c0 con 0x5597bbb342a0 2024-06-04T04:59:07.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb9a3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb338e0 0x5597bbb3ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb9a3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb338e0 0x5597bbb3ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44218/0 (socket says 172.21.15.70:44218) 2024-06-04T04:59:07.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb9a3fff700 1 -- 172.21.15.70:0/736177471 learned_addr learned my addr 172.21.15.70:0/736177471 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:07.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb99bfff700 1 --2- 172.21.15.70:0/736177471 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb342a0 0x5597bbb3f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb9a8a43700 1 --2- 172.21.15.70:0/736177471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb392b0 0x5597bbb48910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb9a3fff700 1 -- 172.21.15.70:0/736177471 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb342a0 msgr2=0x5597bbb3f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:07.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb9a3fff700 1 --2- 172.21.15.70:0/736177471 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb342a0 0x5597bbb3f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb9a3fff700 1 -- 172.21.15.70:0/736177471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb392b0 msgr2=0x5597bbb48910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:07.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb9a3fff700 1 --2- 172.21.15.70:0/736177471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb392b0 0x5597bbb48910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb9a3fff700 1 -- 172.21.15.70:0/736177471 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597bbb2a9f0 con 0x5597bbb338e0 2024-06-04T04:59:07.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb9a8a43700 1 --2- 172.21.15.70:0/736177471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb392b0 0x5597bbb48910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:59:07.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb9a3fff700 1 --2- 172.21.15.70:0/736177471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb338e0 0x5597bbb3ca10 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x5597bbb40790 tx=0x7fb994013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932388f9b75fe8bd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:07.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb999ffb700 1 -- 172.21.15.70:0/736177471 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb994031070 con 0x5597bbb338e0 2024-06-04T04:59:07.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb999ffb700 1 -- 172.21.15.70:0/736177471 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb9940248c0 con 0x5597bbb338e0 2024-06-04T04:59:07.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.543+0000 7fb999ffb700 1 -- 172.21.15.70:0/736177471 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99401d040 con 0x5597bbb338e0 2024-06-04T04:59:07.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.544+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/736177471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb338e0 msgr2=0x5597bbb3ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:07.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.544+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/736177471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb338e0 0x5597bbb3ca10 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x5597bbb40790 tx=0x7fb994013000 comp rx=0 tx=0).stop 2024-06-04T04:59:07.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.544+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/736177471 shutdown_connections 2024-06-04T04:59:07.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.544+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/736177471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb392b0 0x5597bbb48910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.544+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/736177471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb338e0 0x5597bbb3ca10 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.544+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/736177471 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb342a0 0x5597bbb3f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.544+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/736177471 >> 172.21.15.70:0/736177471 conn(0x5597bba61180 msgr2=0x5597bbb3e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:07.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.544+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/736177471 shutdown_connections 2024-06-04T04:59:07.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.544+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/736177471 wait complete. 2024-06-04T04:59:07.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.545+0000 7fb9b4d3bb80 1 -- start start 2024-06-04T04:59:07.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.545+0000 7fb9b4d3bb80 1 Processor -- start 2024-06-04T04:59:07.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.545+0000 7fb9b4d3bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb338e0 0x5597bbb51c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.545+0000 7fb9b4d3bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb342a0 0x5597bbb52140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.545+0000 7fb9b4d3bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb392b0 0x5597bbbc0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.545+0000 7fb9b4d3bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41acf60 con 0x5597bbb338e0 2024-06-04T04:59:07.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.545+0000 7fb9b4d3bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41ac890 con 0x5597bbb392b0 2024-06-04T04:59:07.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.545+0000 7fb9b4d3bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41a9ad0 con 0x5597bbb342a0 2024-06-04T04:59:07.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.545+0000 7fb9a3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb338e0 0x5597bbb51c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.545+0000 7fb9b4d3bb80 10 client.0 _ll_register_callbacks cb 0x5597bba1da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:07.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.545+0000 7fb9a3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb338e0 0x5597bbb51c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44230/0 (socket says 172.21.15.70:44230) 2024-06-04T04:59:07.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.545+0000 7fb9a3fff700 1 -- 172.21.15.70:0/3460941586 learned_addr learned my addr 172.21.15.70:0/3460941586 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:07.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.546+0000 7fb9a8a43700 1 --2- 172.21.15.70:0/3460941586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb392b0 0x5597bbbc0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.546+0000 7fb99bfff700 1 --2- 172.21.15.70:0/3460941586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb342a0 0x5597bbb52140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.546+0000 7fb9a3fff700 1 -- 172.21.15.70:0/3460941586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb342a0 msgr2=0x5597bbb52140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:07.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.546+0000 7fb9a3fff700 1 --2- 172.21.15.70:0/3460941586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb342a0 0x5597bbb52140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.546+0000 7fb9a3fff700 1 -- 172.21.15.70:0/3460941586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb392b0 msgr2=0x5597bbbc0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:07.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.546+0000 7fb9a3fff700 1 --2- 172.21.15.70:0/3460941586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb392b0 0x5597bbbc0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.546+0000 7fb9a3fff700 1 -- 172.21.15.70:0/3460941586 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597bbbc0940 con 0x5597bbb338e0 2024-06-04T04:59:07.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.546+0000 7fb9a3fff700 1 --2- 172.21.15.70:0/3460941586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb338e0 0x5597bbb51c00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb994008010 tx=0x7fb994024560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b97e1ee928a8df server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:07.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.546+0000 7fb9837fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:59:07.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.546+0000 7fb9837fe700 1 -- 172.21.15.70:0/3460941586 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb994031030 con 0x5597bbb338e0 2024-06-04T04:59:07.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.546+0000 7fb9837fe700 1 -- 172.21.15.70:0/3460941586 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb994029070 con 0x5597bbb338e0 2024-06-04T04:59:07.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.546+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/3460941586 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9940248c0 con 0x5597bbb338e0 2024-06-04T04:59:07.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.546+0000 7fb9837fe700 1 -- 172.21.15.70:0/3460941586 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb994024c80 con 0x5597bbb338e0 2024-06-04T04:59:07.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.546+0000 7fb9b4d3bb80 10 client.16002 fetch_fsmap learned FSMap version 29 2024-06-04T04:59:07.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.546+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/3460941586 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5597bbbc1bd0 con 0x5597bbb338e0 2024-06-04T04:59:07.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.547+0000 7fb9837fe700 1 -- 172.21.15.70:0/3460941586 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb9940248c0 con 0x5597bbb338e0 2024-06-04T04:59:07.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.547+0000 7fb9837fe700 1 -- 172.21.15.70:0/3460941586 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fb994013070 con 0x5597bbb338e0 2024-06-04T04:59:07.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.547+0000 7fb9b4d3bb80 10 client.16002 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:59:07.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.547+0000 7fb9b4d3bb80 10 client.16002 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T04:59:07.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.547+0000 7fb9b4d3bb80 20 client.16002 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:07.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.547+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/3460941586 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5597bbbc2250 0x5597bbbc4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:07.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.547+0000 7fb9b4d3bb80 4 client.16002 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "f277a116", "release": true, "await": true}] 2024-06-04T04:59:07.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.547+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/3460941586 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "f277a116", "release": true, "await": true}) -- 0x5597bba94ce0 con 0x5597bbbc2250 2024-06-04T04:59:07.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.547+0000 7fb99bfff700 1 --2- 172.21.15.70:0/3460941586 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5597bbbc2250 0x5597bbbc4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:07.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.548+0000 7fb99bfff700 1 --2- 172.21.15.70:0/3460941586 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5597bbbc2250 0x5597bbbc4670 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea3eb9244ee2b2d1 server_cookie=126c052b99df159f in_seq=0 out_seq=0 2024-06-04T04:59:07.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.548+0000 7fb9837fe700 10 client.16002 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:07.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.552+0000 7fb9837fe700 1 -- 172.21.15.70:0/3460941586 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x5597bba94ce0 con 0x5597bbbc2250 2024-06-04T04:59:07.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.552+0000 7fb9837fe700 10 client.16002 handle_command_reply: tid=0 2024-06-04T04:59:07.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.552+0000 7fb9b4d3bb80 1 client.16002 shutdown 2024-06-04T04:59:07.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/3460941586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb338e0 msgr2=0x5597bbb51c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:07.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/3460941586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb338e0 0x5597bbb51c00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb994008010 tx=0x7fb994024560 comp rx=0 tx=0).stop 2024-06-04T04:59:07.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/3460941586 shutdown_connections 2024-06-04T04:59:07.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/3460941586 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5597bbbc2250 0x5597bbbc4670 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/3460941586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597bbb392b0 0x5597bbbc0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/3460941586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597bbb338e0 0x5597bbb51c00 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9b4d3bb80 1 --2- 172.21.15.70:0/3460941586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5597bbb342a0 0x5597bbb52140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/3460941586 >> 172.21.15.70:0/3460941586 conn(0x5597bba61180 msgr2=0x5597bbb3f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:07.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9837fe700 0 client.16002 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:07.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/3460941586 shutdown_connections 2024-06-04T04:59:07.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9b4d3bb80 1 -- 172.21.15.70:0/3460941586 wait complete. 2024-06-04T04:59:07.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9b4d3bb80 20 client.16002 trim_cache size 0 max 16384 2024-06-04T04:59:07.563 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:59:07.563 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 29, 2024-06-04T04:59:07.563 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T04:59:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 4, 2024-06-04T04:59:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T04:59:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "f277a116": { 2024-06-04T04:59:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 4, 2024-06-04T04:59:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T04:59:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T04:59:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T04:59:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T04:59:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T04:59:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T04:59:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 154.9, 2024-06-04T04:59:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T04:59:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T04:59:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T04:59:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T04:59:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T04:59:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T04:59:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T04:59:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T04:59:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T04:59:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T04:59:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T04:59:07.566 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:59:07.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9aa4a6700 1 -- 172.21.15.70:0/4201728315 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9900b29f0 msgr2=0x7fb9900b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:07.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9aa4a6700 1 --2- 172.21.15.70:0/4201728315 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9900b29f0 0x7fb9900b4eb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb994006fd0 tx=0x7fb994008040 comp rx=0 tx=0).stop 2024-06-04T04:59:07.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9aa4a6700 1 -- 172.21.15.70:0/4201728315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9a4102e10 msgr2=0x7fb9a419fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:07.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9aa4a6700 1 --2- 172.21.15.70:0/4201728315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9a4102e10 0x7fb9a419fe20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb98c00b530 tx=0x7fb98c00b8f0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9aa4a6700 1 -- 172.21.15.70:0/4201728315 shutdown_connections 2024-06-04T04:59:07.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9aa4a6700 1 --2- 172.21.15.70:0/4201728315 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9900b29f0 0x7fb9900b4eb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9aa4a6700 1 --2- 172.21.15.70:0/4201728315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9a41084b0 0x7fb9a4199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9aa4a6700 1 --2- 172.21.15.70:0/4201728315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9a4102470 0x7fb9a419f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9aa4a6700 1 --2- 172.21.15.70:0/4201728315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9a4102e10 0x7fb9a419fe20 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:07.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.553+0000 7fb9aa4a6700 1 -- 172.21.15.70:0/4201728315 >> 172.21.15.70:0/4201728315 conn(0x7fb9a40793e0 msgr2=0x7fb9a406c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:07.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.554+0000 7fb9aa4a6700 1 -- 172.21.15.70:0/4201728315 shutdown_connections 2024-06-04T04:59:07.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:07.554+0000 7fb9aa4a6700 1 -- 172.21.15.70:0/4201728315 wait complete. 2024-06-04T04:59:07.568 INFO:tasks.workunit.client.0.smithi070.stderr:+ dbench 1 2024-06-04T04:59:07.595 INFO:tasks.workunit.client.0.smithi070.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-06-04T04:59:07.595 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-06-04T04:59:07.596 INFO:tasks.workunit.client.0.smithi070.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-06-04T04:59:07.612 INFO:tasks.workunit.client.0.smithi070.stdout:0 of 1 processes prepared for launch 0 sec 2024-06-04T04:59:07.712 INFO:tasks.workunit.client.0.smithi070.stdout:1 of 1 processes prepared for launch 0 sec 2024-06-04T04:59:07.712 INFO:tasks.workunit.client.0.smithi070.stdout:releasing clients 2024-06-04T04:59:08.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.355+0000 7f7a1faea700 1 -- 172.21.15.70:0/1101512681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a181035e0 msgr2=0x7f7a18103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:08.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.355+0000 7f7a1faea700 1 --2- 172.21.15.70:0/1101512681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a181035e0 0x7f7a18103a60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f7a080099c0 tx=0x7f7a08009cd0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.356+0000 7f7a1faea700 1 -- 172.21.15.70:0/1101512681 shutdown_connections 2024-06-04T04:59:08.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.356+0000 7f7a1faea700 1 --2- 172.21.15.70:0/1101512681 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a18103fa0 0x7f7a18110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.356+0000 7f7a1faea700 1 --2- 172.21.15.70:0/1101512681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a18109680 0x7f7a18109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.356+0000 7f7a1faea700 1 --2- 172.21.15.70:0/1101512681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a181035e0 0x7f7a18103a60 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.356+0000 7f7a1faea700 1 -- 172.21.15.70:0/1101512681 >> 172.21.15.70:0/1101512681 conn(0x7f7a1807a6f0 msgr2=0x7f7a1807ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.356+0000 7f7a1faea700 1 -- 172.21.15.70:0/1101512681 shutdown_connections 2024-06-04T04:59:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.356+0000 7f7a1faea700 1 -- 172.21.15.70:0/1101512681 wait complete. 2024-06-04T04:59:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.356+0000 7f7a1faea700 1 Processor -- start 2024-06-04T04:59:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1faea700 1 -- start start 2024-06-04T04:59:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1faea700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a181035e0 0x7f7a1819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1faea700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a18103fa0 0x7f7a1819fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1faea700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a18109680 0x7f7a18199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1faea700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7a18113650 con 0x7f7a181035e0 2024-06-04T04:59:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1faea700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a181134d0 con 0x7f7a18103fa0 2024-06-04T04:59:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1faea700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7a181137d0 con 0x7f7a18109680 2024-06-04T04:59:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1d886700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a181035e0 0x7f7a1819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1d085700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a18103fa0 0x7f7a1819fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1d886700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a181035e0 0x7f7a1819f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44244/0 (socket says 172.21.15.70:44244) 2024-06-04T04:59:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1d886700 1 -- 172.21.15.70:0/1444970101 learned_addr learned my addr 172.21.15.70:0/1444970101 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1d085700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a18103fa0 0x7f7a1819fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59740/0 (socket says 172.21.15.70:59740) 2024-06-04T04:59:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1e087700 1 --2- 172.21.15.70:0/1444970101 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a18109680 0x7f7a18199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1d085700 1 -- 172.21.15.70:0/1444970101 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a18109680 msgr2=0x7f7a18199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1d085700 1 --2- 172.21.15.70:0/1444970101 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a18109680 0x7f7a18199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1d085700 1 -- 172.21.15.70:0/1444970101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a181035e0 msgr2=0x7f7a1819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1d085700 1 --2- 172.21.15.70:0/1444970101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a181035e0 0x7f7a1819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1d085700 1 -- 172.21.15.70:0/1444970101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a1819a0d0 con 0x7f7a18103fa0 2024-06-04T04:59:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.357+0000 7f7a1e087700 1 --2- 172.21.15.70:0/1444970101 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a18109680 0x7f7a18199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:59:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.358+0000 7f7a1d085700 1 --2- 172.21.15.70:0/1444970101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a18103fa0 0x7f7a1819fca0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7a080098e0 tx=0x7f7a080045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.358+0000 7f7a0effd700 1 -- 172.21.15.70:0/1444970101 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a08026070 con 0x7f7a18103fa0 2024-06-04T04:59:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.358+0000 7f7a0effd700 1 -- 172.21.15.70:0/1444970101 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7a0800f460 con 0x7f7a18103fa0 2024-06-04T04:59:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.358+0000 7f7a1faea700 1 -- 172.21.15.70:0/1444970101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a1819a360 con 0x7f7a18103fa0 2024-06-04T04:59:08.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.358+0000 7f7a0effd700 1 -- 172.21.15.70:0/1444970101 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a08009e60 con 0x7f7a18103fa0 2024-06-04T04:59:08.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.358+0000 7f7a1faea700 1 -- 172.21.15.70:0/1444970101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a181a6630 con 0x7f7a18103fa0 2024-06-04T04:59:08.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.359+0000 7f7a1faea700 1 -- 172.21.15.70:0/1444970101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a18104cf0 con 0x7f7a18103fa0 2024-06-04T04:59:08.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.361+0000 7f7a0effd700 1 -- 172.21.15.70:0/1444970101 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7a08029070 con 0x7f7a18103fa0 2024-06-04T04:59:08.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.361+0000 7f7a0effd700 1 --2- 172.21.15.70:0/1444970101 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7a040b29f0 0x7f7a040b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.362+0000 7f7a1d886700 1 --2- 172.21.15.70:0/1444970101 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7a040b29f0 0x7f7a040b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.362+0000 7f7a0effd700 1 -- 172.21.15.70:0/1444970101 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f7a080e2d50 con 0x7f7a18103fa0 2024-06-04T04:59:08.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.362+0000 7f7a1d886700 1 --2- 172.21.15.70:0/1444970101 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7a040b29f0 0x7f7a040b4eb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7a181090d0 tx=0x7f7a14007480 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:08.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.366+0000 7f7a0effd700 1 -- 172.21.15.70:0/1444970101 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7a080e6030 con 0x7f7a18103fa0 2024-06-04T04:59:08.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.594+0000 7f103aa93640 1 -- 172.21.15.70:0/3439815521 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1034106920 msgr2=0x7f1034106d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.594+0000 7f103aa93640 1 --2- 172.21.15.70:0/3439815521 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1034106920 0x7f1034106d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1028009930 tx=0x7f102802a420 comp rx=0 tx=0).stop 2024-06-04T04:59:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.594+0000 7f103aa93640 1 -- 172.21.15.70:0/3439815521 shutdown_connections 2024-06-04T04:59:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.594+0000 7f103aa93640 1 --2- 172.21.15.70:0/3439815521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10341020c0 0x7f1034102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.594+0000 7f103aa93640 1 --2- 172.21.15.70:0/3439815521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1034102a60 0x7f103410e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.594+0000 7f103aa93640 1 --2- 172.21.15.70:0/3439815521 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1034106920 0x7f1034106d00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.594+0000 7f103aa93640 1 -- 172.21.15.70:0/3439815521 >> 172.21.15.70:0/3439815521 conn(0x7f1034077ee0 msgr2=0x7f10340ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.594+0000 7f103aa93640 1 -- 172.21.15.70:0/3439815521 shutdown_connections 2024-06-04T04:59:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.594+0000 7f103aa93640 1 -- 172.21.15.70:0/3439815521 wait complete. 2024-06-04T04:59:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.594+0000 7f103aa93640 1 Processor -- start 2024-06-04T04:59:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.594+0000 7f103aa93640 1 -- start start 2024-06-04T04:59:08.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.595+0000 7f103aa93640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10341020c0 0x7f10341980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.595+0000 7f103aa93640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1034102a60 0x7f10341985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.595+0000 7f103aa93640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1034106920 0x7f103419c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.595+0000 7f103aa93640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1034111bd0 con 0x7f10341020c0 2024-06-04T04:59:08.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.595+0000 7f103aa93640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1034111a50 con 0x7f1034106920 2024-06-04T04:59:08.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.595+0000 7f103aa93640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1034111d50 con 0x7f1034102a60 2024-06-04T04:59:08.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.595+0000 7f103882f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10341020c0 0x7f10341980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.595+0000 7f103882f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10341020c0 0x7f10341980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44254/0 (socket says 172.21.15.70:44254) 2024-06-04T04:59:08.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.595+0000 7f103882f640 1 -- 172.21.15.70:0/3656308347 learned_addr learned my addr 172.21.15.70:0/3656308347 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:08.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.595+0000 7f1033fff640 1 --2- 172.21.15.70:0/3656308347 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1034102a60 0x7f10341985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.595+0000 7f1039030640 1 --2- 172.21.15.70:0/3656308347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1034106920 0x7f103419c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.595+0000 7f103882f640 1 -- 172.21.15.70:0/3656308347 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1034102a60 msgr2=0x7f10341985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:08.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.595+0000 7f103882f640 1 --2- 172.21.15.70:0/3656308347 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1034102a60 0x7f10341985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.595+0000 7f103882f640 1 -- 172.21.15.70:0/3656308347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1034106920 msgr2=0x7f103419c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:08.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.595+0000 7f103882f640 1 --2- 172.21.15.70:0/3656308347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1034106920 0x7f103419c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.595+0000 7f103882f640 1 -- 172.21.15.70:0/3656308347 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f103419d0a0 con 0x7f10341020c0 2024-06-04T04:59:08.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.595+0000 7f103882f640 1 --2- 172.21.15.70:0/3656308347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10341020c0 0x7f10341980b0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f102802a940 tx=0x7f1028003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:08.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.596+0000 7f1031ffb640 1 -- 172.21.15.70:0/3656308347 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102803f070 con 0x7f10341020c0 2024-06-04T04:59:08.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.596+0000 7f103aa93640 1 -- 172.21.15.70:0/3656308347 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f103419d330 con 0x7f10341020c0 2024-06-04T04:59:08.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.596+0000 7f1031ffb640 1 -- 172.21.15.70:0/3656308347 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f102803baf0 con 0x7f10341020c0 2024-06-04T04:59:08.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.596+0000 7f1031ffb640 1 -- 172.21.15.70:0/3656308347 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1028035030 con 0x7f10341020c0 2024-06-04T04:59:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.596+0000 7f103aa93640 1 -- 172.21.15.70:0/3656308347 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1034100fb0 con 0x7f10341020c0 2024-06-04T04:59:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.599+0000 7f1031ffb640 1 -- 172.21.15.70:0/3656308347 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f1028042070 con 0x7f10341020c0 2024-06-04T04:59:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.599+0000 7f103d7a9740 1 Processor -- start 2024-06-04T04:59:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.599+0000 7f103d7a9740 1 -- start start 2024-06-04T04:59:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.599+0000 7f103d7a9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c81820 0x556825c81c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.599+0000 7f103d7a9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c82140 0x556825c8c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.599+0000 7f103d7a9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c8ca00 0x556825c8edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.599+0000 7f103d7a9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556825bf3c30 con 0x556825c82140 2024-06-04T04:59:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.599+0000 7f103d7a9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556825be37d0 con 0x556825c81820 2024-06-04T04:59:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.599+0000 7f103d7a9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556825be7810 con 0x556825c8ca00 2024-06-04T04:59:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.599+0000 7f103882f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c82140 0x556825c8c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.599+0000 7f1039030640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c81820 0x556825c81c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.599+0000 7f103882f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c82140 0x556825c8c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44260/0 (socket says 172.21.15.70:44260) 2024-06-04T04:59:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.599+0000 7f1039030640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c81820 0x556825c81c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59756/0 (socket says 172.21.15.70:59756) 2024-06-04T04:59:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.599+0000 7f103882f640 1 -- 172.21.15.70:0/801953546 learned_addr learned my addr 172.21.15.70:0/801953546 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.599+0000 7f1033fff640 1 --2- 172.21.15.70:0/801953546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c8ca00 0x556825c8edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.600+0000 7f1039030640 1 -- 172.21.15.70:0/801953546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c8ca00 msgr2=0x556825c8edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.600+0000 7f1039030640 1 --2- 172.21.15.70:0/801953546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c8ca00 0x556825c8edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.600+0000 7f1039030640 1 -- 172.21.15.70:0/801953546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c82140 msgr2=0x556825c8c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.600+0000 7f1039030640 1 --2- 172.21.15.70:0/801953546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c82140 0x556825c8c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.600+0000 7f1039030640 1 -- 172.21.15.70:0/801953546 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568258f8210 con 0x556825c81820 2024-06-04T04:59:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.600+0000 7f1033fff640 1 --2- 172.21.15.70:0/801953546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c8ca00 0x556825c8edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.600+0000 7f103882f640 1 --2- 172.21.15.70:0/801953546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c82140 0x556825c8c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.600+0000 7f1031ffb640 1 --2- 172.21.15.70:0/3656308347 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10240b2a10 0x7f10240b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.600+0000 7f103882f640 1 --2- 172.21.15.70:0/3656308347 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10240b2a10 0x7f10240b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.600+0000 7f1039030640 1 --2- 172.21.15.70:0/801953546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c81820 0x556825c81c00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f102c009f00 tx=0x7f102c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a53ceecdf1d3283b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.600+0000 7f1031ffb640 1 -- 172.21.15.70:0/3656308347 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f1028102c80 con 0x7f10341020c0 2024-06-04T04:59:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.600+0000 7f101effd640 1 -- 172.21.15.70:0/801953546 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c004d20 con 0x556825c81820 2024-06-04T04:59:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.600+0000 7f101effd640 1 -- 172.21.15.70:0/801953546 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f102c004050 con 0x556825c81820 2024-06-04T04:59:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.600+0000 7f101effd640 1 -- 172.21.15.70:0/801953546 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c005260 con 0x556825c81820 2024-06-04T04:59:08.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.601+0000 7f103d7a9740 1 -- 172.21.15.70:0/801953546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c81820 msgr2=0x556825c81c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.601+0000 7f103d7a9740 1 --2- 172.21.15.70:0/801953546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c81820 0x556825c81c00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f102c009f00 tx=0x7f102c011810 comp rx=0 tx=0).stop 2024-06-04T04:59:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.601+0000 7f103882f640 1 --2- 172.21.15.70:0/3656308347 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10240b2a10 0x7f10240b4ed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f10280c9fa0 tx=0x7f10281067d0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.601+0000 7f103d7a9740 1 -- 172.21.15.70:0/801953546 shutdown_connections 2024-06-04T04:59:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.601+0000 7f103d7a9740 1 --2- 172.21.15.70:0/801953546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c8ca00 0x556825c8edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.601+0000 7f103d7a9740 1 --2- 172.21.15.70:0/801953546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c82140 0x556825c8c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.601+0000 7f103d7a9740 1 --2- 172.21.15.70:0/801953546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c81820 0x556825c81c00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.601+0000 7f103d7a9740 1 -- 172.21.15.70:0/801953546 >> 172.21.15.70:0/801953546 conn(0x556825c7dd30 msgr2=0x556825c7e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.601+0000 7f103d7a9740 1 -- 172.21.15.70:0/801953546 shutdown_connections 2024-06-04T04:59:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.601+0000 7f103d7a9740 1 -- 172.21.15.70:0/801953546 wait complete. 2024-06-04T04:59:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.602+0000 7f103d7a9740 1 -- start start 2024-06-04T04:59:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.602+0000 7f103d7a9740 1 Processor -- start 2024-06-04T04:59:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.602+0000 7f103d7a9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c81820 0x556825c89a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.602+0000 7f103d7a9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c82140 0x556825c89f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.602+0000 7f103d7a9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c8ca00 0x556825c8a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.602+0000 7f103d7a9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5568258c9790 con 0x556825c82140 2024-06-04T04:59:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.602+0000 7f103d7a9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556825b9f890 con 0x556825c8ca00 2024-06-04T04:59:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.602+0000 7f103d7a9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556825c8f630 con 0x556825c81820 2024-06-04T04:59:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.602+0000 7f103d7a9740 10 client.0 _ll_register_callbacks cb 0x5568258f7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.602+0000 7f1033fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c82140 0x556825c89f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.602+0000 7f1033fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c82140 0x556825c89f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44270/0 (socket says 172.21.15.70:44270) 2024-06-04T04:59:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.602+0000 7f1033fff640 1 -- 172.21.15.70:0/2575092439 learned_addr learned my addr 172.21.15.70:0/2575092439 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.602+0000 7f1039030640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c81820 0x556825c89a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.602+0000 7f1039030640 1 --2- 172.21.15.70:0/2575092439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c8ca00 0x556825c8a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.603+0000 7f1033fff640 1 -- 172.21.15.70:0/2575092439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c81820 msgr2=0x556825c89a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:08.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.603+0000 7f1033fff640 1 --2- 172.21.15.70:0/2575092439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c81820 0x556825c89a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.603+0000 7f1033fff640 1 -- 172.21.15.70:0/2575092439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c8ca00 msgr2=0x556825c8a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:08.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.603+0000 7f1033fff640 1 --2- 172.21.15.70:0/2575092439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c8ca00 0x556825c8a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.603+0000 7f1033fff640 1 -- 172.21.15.70:0/2575092439 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568259c4620 con 0x556825c82140 2024-06-04T04:59:08.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.603+0000 7f1039030640 1 --2- 172.21.15.70:0/2575092439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c8ca00 0x556825c8a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:59:08.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.603+0000 7f1033fff640 1 --2- 172.21.15.70:0/2575092439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c82140 0x556825c89f50 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f102000bf50 tx=0x7f1020012bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f04713a9d39fb77 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:08.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.603+0000 7f1013fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:59:08.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.603+0000 7f1013fff640 1 -- 172.21.15.70:0/2575092439 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1020004020 con 0x556825c82140 2024-06-04T04:59:08.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.603+0000 7f1013fff640 1 -- 172.21.15.70:0/2575092439 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f10200041c0 con 0x556825c82140 2024-06-04T04:59:08.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.603+0000 7f103d7a9740 1 -- 172.21.15.70:0/2575092439 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f102c004050 con 0x556825c82140 2024-06-04T04:59:08.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.603+0000 7f1013fff640 1 -- 172.21.15.70:0/2575092439 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f10200043c0 con 0x556825c82140 2024-06-04T04:59:08.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.603+0000 7f103d7a9740 10 client.16020 fetch_fsmap learned FSMap version 29 2024-06-04T04:59:08.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.603+0000 7f103d7a9740 1 -- 172.21.15.70:0/2575092439 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556825c9b4b0 con 0x556825c82140 2024-06-04T04:59:08.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.603+0000 7f1013fff640 1 -- 172.21.15.70:0/2575092439 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f102c004050 con 0x556825c82140 2024-06-04T04:59:08.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.604+0000 7f1013fff640 1 -- 172.21.15.70:0/2575092439 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f1020003a60 con 0x556825c82140 2024-06-04T04:59:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.604+0000 7f103d7a9740 10 client.16020 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:59:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.604+0000 7f103d7a9740 10 client.16020 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:59:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.604+0000 7f103d7a9740 20 client.16020 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.604+0000 7f103d7a9740 1 --2- 172.21.15.70:0/2575092439 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556825c9ca30 0x556825c9ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.604+0000 7f103d7a9740 4 client.16020 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:59:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.604+0000 7f103d7a9740 1 -- 172.21.15.70:0/2575092439 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556825bf5b80 con 0x556825c9ca30 2024-06-04T04:59:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.604+0000 7f1039030640 1 --2- 172.21.15.70:0/2575092439 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556825c9ca30 0x556825c9ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.604+0000 7f1013fff640 10 client.16020 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.604+0000 7f1039030640 1 --2- 172.21.15.70:0/2575092439 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556825c9ca30 0x556825c9ee20 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74e0f2b7e53c1152 server_cookie=b1696577b7663e36 in_seq=0 out_seq=0 2024-06-04T04:59:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.605+0000 7f1013fff640 1 -- 172.21.15.70:0/2575092439 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556825bf5b80 con 0x556825c9ca30 2024-06-04T04:59:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.605+0000 7f1013fff640 10 client.16020 handle_command_reply: tid=0 2024-06-04T04:59:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.605+0000 7f103d7a9740 1 client.16020 shutdown 2024-06-04T04:59:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.605+0000 7f103d7a9740 1 -- 172.21.15.70:0/2575092439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c82140 msgr2=0x556825c89f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.605+0000 7f103d7a9740 1 --2- 172.21.15.70:0/2575092439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c82140 0x556825c89f50 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f102000bf50 tx=0x7f1020012bf0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.605+0000 7f103d7a9740 1 -- 172.21.15.70:0/2575092439 shutdown_connections 2024-06-04T04:59:08.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.605+0000 7f103d7a9740 1 --2- 172.21.15.70:0/2575092439 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556825c9ca30 0x556825c9ee20 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.605+0000 7f103d7a9740 1 --2- 172.21.15.70:0/2575092439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c8ca00 0x556825c8a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.605+0000 7f1013fff640 0 client.16020 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:08.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.605+0000 7f103d7a9740 1 --2- 172.21.15.70:0/2575092439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c82140 0x556825c89f50 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.605+0000 7f103d7a9740 1 --2- 172.21.15.70:0/2575092439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c81820 0x556825c89a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.606+0000 7f103d7a9740 1 -- 172.21.15.70:0/2575092439 >> 172.21.15.70:0/2575092439 conn(0x556825c7dd30 msgr2=0x556825c7cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:08.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.606+0000 7f103d7a9740 1 -- 172.21.15.70:0/2575092439 shutdown_connections 2024-06-04T04:59:08.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.606+0000 7f103d7a9740 1 -- 172.21.15.70:0/2575092439 wait complete. 2024-06-04T04:59:08.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.606+0000 7f103d7a9740 20 client.16020 trim_cache size 0 max 16384 2024-06-04T04:59:08.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.623+0000 7f103d7a9740 1 Processor -- start 2024-06-04T04:59:08.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.623+0000 7f103d7a9740 1 -- start start 2024-06-04T04:59:08.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.623+0000 7f103d7a9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c81820 0x556825c7d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.623+0000 7f103d7a9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c82140 0x556825ca46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.623+0000 7f103d7a9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c8ca00 0x556825ca4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.623+0000 7f103d7a9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f10341ac380 con 0x556825c81820 2024-06-04T04:59:08.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.623+0000 7f103d7a9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10341adec0 con 0x556825c82140 2024-06-04T04:59:08.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.623+0000 7f103d7a9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f10341aca50 con 0x556825c8ca00 2024-06-04T04:59:08.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.623+0000 7f1039030640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c81820 0x556825c7d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.623+0000 7f1039030640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c81820 0x556825c7d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44280/0 (socket says 172.21.15.70:44280) 2024-06-04T04:59:08.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.623+0000 7f1039030640 1 -- 172.21.15.70:0/3213785999 learned_addr learned my addr 172.21.15.70:0/3213785999 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:08.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.623+0000 7f1039030640 1 --2- 172.21.15.70:0/3213785999 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c8ca00 0x556825ca4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.623+0000 7f1033fff640 1 --2- 172.21.15.70:0/3213785999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c82140 0x556825ca46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.624+0000 7f1039030640 1 -- 172.21.15.70:0/3213785999 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c8ca00 msgr2=0x556825ca4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:08.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.624+0000 7f1039030640 1 --2- 172.21.15.70:0/3213785999 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c8ca00 0x556825ca4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.624+0000 7f1039030640 1 -- 172.21.15.70:0/3213785999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c82140 msgr2=0x556825ca46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:08.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.624+0000 7f1039030640 1 --2- 172.21.15.70:0/3213785999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c82140 0x556825ca46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.624+0000 7f1039030640 1 -- 172.21.15.70:0/3213785999 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556825c9f190 con 0x556825c81820 2024-06-04T04:59:08.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.624+0000 7f1033fff640 1 --2- 172.21.15.70:0/3213785999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c82140 0x556825ca46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.624+0000 7f1039030640 1 --2- 172.21.15.70:0/3213785999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c81820 0x556825c7d730 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f102c003af0 tx=0x7f102c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=310899a6131d1194 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.624+0000 7f101e7fc640 1 -- 172.21.15.70:0/3213785999 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c003c20 con 0x556825c81820 2024-06-04T04:59:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.624+0000 7f101e7fc640 1 -- 172.21.15.70:0/3213785999 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f102c003dc0 con 0x556825c81820 2024-06-04T04:59:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.624+0000 7f101e7fc640 1 -- 172.21.15.70:0/3213785999 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c0054d0 con 0x556825c81820 2024-06-04T04:59:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.625+0000 7f103d7a9740 1 -- 172.21.15.70:0/3213785999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c81820 msgr2=0x556825c7d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.625+0000 7f103d7a9740 1 --2- 172.21.15.70:0/3213785999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c81820 0x556825c7d730 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f102c003af0 tx=0x7f102c011810 comp rx=0 tx=0).stop 2024-06-04T04:59:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.625+0000 7f103d7a9740 1 -- 172.21.15.70:0/3213785999 shutdown_connections 2024-06-04T04:59:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.625+0000 7f103d7a9740 1 --2- 172.21.15.70:0/3213785999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c82140 0x556825ca46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.625+0000 7f103d7a9740 1 --2- 172.21.15.70:0/3213785999 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c8ca00 0x556825ca4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.625+0000 7f103d7a9740 1 --2- 172.21.15.70:0/3213785999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c81820 0x556825c7d730 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.625+0000 7f103d7a9740 1 -- 172.21.15.70:0/3213785999 >> 172.21.15.70:0/3213785999 conn(0x556825c7dd30 msgr2=0x556825c83a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.625+0000 7f103d7a9740 1 -- 172.21.15.70:0/3213785999 shutdown_connections 2024-06-04T04:59:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.625+0000 7f103d7a9740 1 -- 172.21.15.70:0/3213785999 wait complete. 2024-06-04T04:59:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.625+0000 7f103d7a9740 1 -- start start 2024-06-04T04:59:08.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.625+0000 7f103d7a9740 1 Processor -- start 2024-06-04T04:59:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f103d7a9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c81820 0x556825c90c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f103d7a9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c82140 0x556825c911d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f103d7a9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c8ca00 0x556825d06580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f103d7a9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f10341ad7f0 con 0x556825c81820 2024-06-04T04:59:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f103d7a9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10341ad120 con 0x556825c8ca00 2024-06-04T04:59:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f103d7a9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f10341aa360 con 0x556825c82140 2024-06-04T04:59:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f103d7a9740 10 client.0 _ll_register_callbacks cb 0x7f102000bb90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f1039030640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c81820 0x556825c90c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f1039030640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c81820 0x556825c90c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44292/0 (socket says 172.21.15.70:44292) 2024-06-04T04:59:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f1039030640 1 -- 172.21.15.70:0/3699763688 learned_addr learned my addr 172.21.15.70:0/3699763688 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f1033fff640 1 --2- 172.21.15.70:0/3699763688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c82140 0x556825c911d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f1039030640 1 --2- 172.21.15.70:0/3699763688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c8ca00 0x556825d06580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f1039030640 1 -- 172.21.15.70:0/3699763688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c82140 msgr2=0x556825c911d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f1039030640 1 --2- 172.21.15.70:0/3699763688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c82140 0x556825c911d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f1039030640 1 -- 172.21.15.70:0/3699763688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c8ca00 msgr2=0x556825d06580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f1039030640 1 --2- 172.21.15.70:0/3699763688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c8ca00 0x556825d06580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f1039030640 1 -- 172.21.15.70:0/3699763688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556825d06ac0 con 0x556825c81820 2024-06-04T04:59:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f1039030640 1 --2- 172.21.15.70:0/3699763688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c81820 0x556825c90c90 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f102c003fd0 tx=0x7f102c009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca36c24479409536 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f1013fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:59:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f1013fff640 1 -- 172.21.15.70:0/3699763688 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c005390 con 0x556825c81820 2024-06-04T04:59:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f1013fff640 1 -- 172.21.15.70:0/3699763688 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f102c005530 con 0x556825c81820 2024-06-04T04:59:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.626+0000 7f103d7a9740 1 -- 172.21.15.70:0/3699763688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f102c003dc0 con 0x556825c81820 2024-06-04T04:59:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.627+0000 7f1013fff640 1 -- 172.21.15.70:0/3699763688 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f102c005730 con 0x556825c81820 2024-06-04T04:59:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.627+0000 7f103d7a9740 10 client.16026 fetch_fsmap learned FSMap version 29 2024-06-04T04:59:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.627+0000 7f103d7a9740 1 -- 172.21.15.70:0/3699763688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556825d07cd0 con 0x556825c81820 2024-06-04T04:59:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.627+0000 7f1013fff640 1 -- 172.21.15.70:0/3699763688 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f102c003dc0 con 0x556825c81820 2024-06-04T04:59:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.627+0000 7f1013fff640 1 -- 172.21.15.70:0/3699763688 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f102c025040 con 0x556825c81820 2024-06-04T04:59:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.627+0000 7f103d7a9740 10 client.16026 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:59:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.627+0000 7f103d7a9740 10 client.16026 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:59:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.627+0000 7f103d7a9740 20 client.16026 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.627+0000 7f103d7a9740 1 --2- 172.21.15.70:0/3699763688 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556825d08000 0x556825d0a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.627+0000 7f103d7a9740 4 client.16026 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T04:59:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.627+0000 7f103d7a9740 1 -- 172.21.15.70:0/3699763688 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556825bef360 con 0x556825d08000 2024-06-04T04:59:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.627+0000 7f1033fff640 1 --2- 172.21.15.70:0/3699763688 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556825d08000 0x556825d0a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.628+0000 7f1013fff640 10 client.16026 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.628+0000 7f1033fff640 1 --2- 172.21.15.70:0/3699763688 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556825d08000 0x556825d0a3f0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a34dab6af32e02a1 server_cookie=13dc200de208c412 in_seq=0 out_seq=0 2024-06-04T04:59:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.661+0000 7f1013fff640 1 -- 172.21.15.70:0/3699763688 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x556825bef360 con 0x556825d08000 2024-06-04T04:59:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.661+0000 7f1013fff640 10 client.16026 handle_command_reply: tid=0 2024-06-04T04:59:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.661+0000 7f103d7a9740 1 client.16026 shutdown 2024-06-04T04:59:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.661+0000 7f103d7a9740 1 -- 172.21.15.70:0/3699763688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c81820 msgr2=0x556825c90c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.662+0000 7f103d7a9740 1 --2- 172.21.15.70:0/3699763688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c81820 0x556825c90c90 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f102c003fd0 tx=0x7f102c009230 comp rx=0 tx=0).stop 2024-06-04T04:59:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.662+0000 7f103d7a9740 1 -- 172.21.15.70:0/3699763688 shutdown_connections 2024-06-04T04:59:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.662+0000 7f103d7a9740 1 --2- 172.21.15.70:0/3699763688 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556825d08000 0x556825d0a3f0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.662+0000 7f103d7a9740 1 --2- 172.21.15.70:0/3699763688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556825c8ca00 0x556825d06580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.662+0000 7f103d7a9740 1 --2- 172.21.15.70:0/3699763688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556825c82140 0x556825c911d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.662+0000 7f103d7a9740 1 --2- 172.21.15.70:0/3699763688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556825c81820 0x556825c90c90 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.662+0000 7f103d7a9740 1 -- 172.21.15.70:0/3699763688 >> 172.21.15.70:0/3699763688 conn(0x556825c7dd30 msgr2=0x556825c97680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.662+0000 7f1013fff640 0 client.16026 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.662+0000 7f103d7a9740 1 -- 172.21.15.70:0/3699763688 shutdown_connections 2024-06-04T04:59:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.662+0000 7f103d7a9740 1 -- 172.21.15.70:0/3699763688 wait complete. 2024-06-04T04:59:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.662+0000 7f103d7a9740 20 client.16026 trim_cache size 0 max 16384 2024-06-04T04:59:08.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T04:59:08.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T04:59:08.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T04:59:08.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.662+0000 7f103aa93640 1 -- 172.21.15.70:0/3656308347 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10240b2a10 msgr2=0x7f10240b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:08.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.662+0000 7f103aa93640 1 --2- 172.21.15.70:0/3656308347 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10240b2a10 0x7f10240b4ed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f10280c9fa0 tx=0x7f10281067d0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.662+0000 7f103aa93640 1 -- 172.21.15.70:0/3656308347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10341020c0 msgr2=0x7f10341980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:08.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.662+0000 7f103aa93640 1 --2- 172.21.15.70:0/3656308347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10341020c0 0x7f10341980b0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f102802a940 tx=0x7f1028003b90 comp rx=0 tx=0).stop 2024-06-04T04:59:08.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.663+0000 7f103aa93640 1 -- 172.21.15.70:0/3656308347 shutdown_connections 2024-06-04T04:59:08.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.663+0000 7f103aa93640 1 --2- 172.21.15.70:0/3656308347 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10240b2a10 0x7f10240b4ed0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.663+0000 7f103aa93640 1 --2- 172.21.15.70:0/3656308347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1034106920 0x7f103419c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.663+0000 7f103aa93640 1 --2- 172.21.15.70:0/3656308347 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1034102a60 0x7f10341985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.663+0000 7f103aa93640 1 --2- 172.21.15.70:0/3656308347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10341020c0 0x7f10341980b0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.663+0000 7f103aa93640 1 -- 172.21.15.70:0/3656308347 >> 172.21.15.70:0/3656308347 conn(0x7f1034077ee0 msgr2=0x7f1034104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:08.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.663+0000 7f103aa93640 1 -- 172.21.15.70:0/3656308347 shutdown_connections 2024-06-04T04:59:08.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:08.663+0000 7f103aa93640 1 -- 172.21.15.70:0/3656308347 wait complete. 2024-06-04T04:59:08.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.685+0000 7f7a1faea700 1 -- 172.21.15.70:0/1444970101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a1806a230 con 0x7f7a18103fa0 2024-06-04T04:59:08.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.686+0000 7f7a0effd700 1 -- 172.21.15.70:0/1444970101 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f7a08013070 con 0x7f7a18103fa0 2024-06-04T04:59:08.688 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T04:59:08.689 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:57:09.013178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T04:59:08.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.689+0000 7f7a1faea700 1 -- 172.21.15.70:0/1444970101 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7a040b29f0 msgr2=0x7f7a040b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:08.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.689+0000 7f7a1faea700 1 --2- 172.21.15.70:0/1444970101 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7a040b29f0 0x7f7a040b4eb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7a181090d0 tx=0x7f7a14007480 comp rx=0 tx=0).stop 2024-06-04T04:59:08.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.689+0000 7f7a1faea700 1 -- 172.21.15.70:0/1444970101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a18103fa0 msgr2=0x7f7a1819fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:08.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.689+0000 7f7a1faea700 1 --2- 172.21.15.70:0/1444970101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a18103fa0 0x7f7a1819fca0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7a080098e0 tx=0x7f7a080045e0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.689+0000 7f7a1faea700 1 -- 172.21.15.70:0/1444970101 shutdown_connections 2024-06-04T04:59:08.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.689+0000 7f7a1faea700 1 --2- 172.21.15.70:0/1444970101 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7a040b29f0 0x7f7a040b4eb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.689+0000 7f7a1faea700 1 --2- 172.21.15.70:0/1444970101 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a18109680 0x7f7a18199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.689+0000 7f7a1faea700 1 --2- 172.21.15.70:0/1444970101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a181035e0 0x7f7a1819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.689+0000 7f7a1faea700 1 --2- 172.21.15.70:0/1444970101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a18103fa0 0x7f7a1819fca0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:08.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.689+0000 7f7a1faea700 1 -- 172.21.15.70:0/1444970101 >> 172.21.15.70:0/1444970101 conn(0x7f7a1807a6f0 msgr2=0x7f7a181022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:08.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.689+0000 7f7a1faea700 1 -- 172.21.15.70:0/1444970101 shutdown_connections 2024-06-04T04:59:08.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:08.689+0000 7f7a1faea700 1 -- 172.21.15.70:0/1444970101 wait complete. 2024-06-04T04:59:08.692 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-06-04T04:59:08.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 930 21.78 MB/sec warmup 1 sec latency 18.945 ms 2024-06-04T04:59:08.997 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'f277a116', total seconds elapsed: 48.9 2024-06-04T04:59:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:09 smithi070 ceph-mon[32631]: pgmap v62: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-06-04T04:59:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:09 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1444970101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:59:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:09 smithi112 ceph-mon[36453]: pgmap v62: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-06-04T04:59:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:09 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1444970101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:59:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:09 smithi098 ceph-mon[35569]: pgmap v62: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-06-04T04:59:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:09 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1444970101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T04:59:09.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 1698 13.11 MB/sec warmup 2 sec latency 16.403 ms 2024-06-04T04:59:09.848 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T04:59:10.142 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:59:10.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 2428 9.37 MB/sec warmup 3 sec latency 17.110 ms 2024-06-04T04:59:11.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:11 smithi112 ceph-mon[36453]: pgmap v63: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-06-04T04:59:11.443 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:11 smithi070 ceph-mon[32631]: pgmap v63: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-06-04T04:59:11.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:11 smithi098 ceph-mon[35569]: pgmap v63: 97 pgs: 97 active+clean; 1.4 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-06-04T04:59:11.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 3311 8.84 MB/sec warmup 4 sec latency 15.757 ms 2024-06-04T04:59:12.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.062+0000 7f3e0c2ae700 1 -- 172.21.15.70:0/3839427568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e04103f50 msgr2=0x7f3e04110430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:12.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.062+0000 7f3e0c2ae700 1 --2- 172.21.15.70:0/3839427568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e04103f50 0x7f3e04110430 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc00b0a0 tx=0x7f3dfc00b3b0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.062+0000 7f3e0c2ae700 1 -- 172.21.15.70:0/3839427568 shutdown_connections 2024-06-04T04:59:12.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.062+0000 7f3e0c2ae700 1 --2- 172.21.15.70:0/3839427568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e04103f50 0x7f3e04110430 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.062+0000 7f3e0c2ae700 1 --2- 172.21.15.70:0/3839427568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e041096a0 0x7f3e04109a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.062+0000 7f3e0c2ae700 1 --2- 172.21.15.70:0/3839427568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e04103590 0x7f3e04103a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.062+0000 7f3e0c2ae700 1 -- 172.21.15.70:0/3839427568 >> 172.21.15.70:0/3839427568 conn(0x7f3e0407a6f0 msgr2=0x7f3e0407ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:12.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.062+0000 7f3e0c2ae700 1 -- 172.21.15.70:0/3839427568 shutdown_connections 2024-06-04T04:59:12.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.062+0000 7f3e0c2ae700 1 -- 172.21.15.70:0/3839427568 wait complete. 2024-06-04T04:59:12.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0c2ae700 1 Processor -- start 2024-06-04T04:59:12.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0c2ae700 1 -- start start 2024-06-04T04:59:12.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0c2ae700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e04103590 0x7f3e04075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0c2ae700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e04103f50 0x7f3e04075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0c2ae700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e041096a0 0x7f3e040762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0c2ae700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e04113620 con 0x7f3e04103590 2024-06-04T04:59:12.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0c2ae700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e041134a0 con 0x7f3e041096a0 2024-06-04T04:59:12.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0c2ae700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3e041137a0 con 0x7f3e04103f50 2024-06-04T04:59:12.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0a04a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e04103590 0x7f3e04075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0a04a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e04103590 0x7f3e04075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44306/0 (socket says 172.21.15.70:44306) 2024-06-04T04:59:12.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0a04a700 1 -- 172.21.15.70:0/1016721174 learned_addr learned my addr 172.21.15.70:0/1016721174 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:12.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e09849700 1 --2- 172.21.15.70:0/1016721174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e04103f50 0x7f3e04075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0a04a700 1 -- 172.21.15.70:0/1016721174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e04103f50 msgr2=0x7f3e04075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:12.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0a04a700 1 --2- 172.21.15.70:0/1016721174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e04103f50 0x7f3e04075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0a04a700 1 -- 172.21.15.70:0/1016721174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e041096a0 msgr2=0x7f3e040762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:12.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0a84b700 1 --2- 172.21.15.70:0/1016721174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e041096a0 0x7f3e040762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0a04a700 1 --2- 172.21.15.70:0/1016721174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e041096a0 0x7f3e040762c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0a04a700 1 -- 172.21.15.70:0/1016721174 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e041a8010 con 0x7f3e04103590 2024-06-04T04:59:12.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e09849700 1 --2- 172.21.15.70:0/1016721174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e04103f50 0x7f3e04075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:12.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.063+0000 7f3e0a84b700 1 --2- 172.21.15.70:0/1016721174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e041096a0 0x7f3e040762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:59:12.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.064+0000 7f3e0a04a700 1 --2- 172.21.15.70:0/1016721174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e04103590 0x7f3e04075840 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3e0000e9c0 tx=0x7f3e0000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:12.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.064+0000 7f3dfb7fe700 1 -- 172.21.15.70:0/1016721174 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0000cbd0 con 0x7f3e04103590 2024-06-04T04:59:12.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.064+0000 7f3dfb7fe700 1 -- 172.21.15.70:0/1016721174 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e0000cd70 con 0x7f3e04103590 2024-06-04T04:59:12.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.064+0000 7f3dfb7fe700 1 -- 172.21.15.70:0/1016721174 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e00005540 con 0x7f3e04103590 2024-06-04T04:59:12.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.064+0000 7f3e0c2ae700 1 -- 172.21.15.70:0/1016721174 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e041a82a0 con 0x7f3e04103590 2024-06-04T04:59:12.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.064+0000 7f3e0c2ae700 1 -- 172.21.15.70:0/1016721174 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e041a8790 con 0x7f3e04103590 2024-06-04T04:59:12.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.067+0000 7f3dfb7fe700 1 -- 172.21.15.70:0/1016721174 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3e00004750 con 0x7f3e04103590 2024-06-04T04:59:12.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.068+0000 7f3dfb7fe700 1 --2- 172.21.15.70:0/1016721174 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3df00b2a70 0x7f3df00b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.068+0000 7f3e09849700 1 --2- 172.21.15.70:0/1016721174 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3df00b2a70 0x7f3df00b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.068+0000 7f3dfb7fe700 1 -- 172.21.15.70:0/1016721174 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f3e000e27e0 con 0x7f3e04103590 2024-06-04T04:59:12.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.068+0000 7f3e16b43b80 1 Processor -- start 2024-06-04T04:59:12.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.068+0000 7f3e16b43b80 1 -- start start 2024-06-04T04:59:12.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.068+0000 7f3e09849700 1 --2- 172.21.15.70:0/1016721174 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3df00b2a70 0x7f3df00b4f30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3e0407ad30 tx=0x7f3df400a300 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e16b43b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787e860 0x560bb787ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e16b43b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787f210 0x560bb7882940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e16b43b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb7882f60 0x560bb78833a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e16b43b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560bb77ddc90 con 0x560bb787f210 2024-06-04T04:59:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e16b43b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560bb77cd830 con 0x560bb7882f60 2024-06-04T04:59:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e16b43b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560bb77d1870 con 0x560bb787e860 2024-06-04T04:59:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e09849700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787f210 0x560bb7882940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e0a04a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787e860 0x560bb787ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e09849700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787f210 0x560bb7882940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44316/0 (socket says 172.21.15.70:44316) 2024-06-04T04:59:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e09849700 1 -- 172.21.15.70:0/1777745891 learned_addr learned my addr 172.21.15.70:0/1777745891 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e0a04a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787e860 0x560bb787ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46160/0 (socket says 172.21.15.70:46160) 2024-06-04T04:59:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e0a84b700 1 --2- 172.21.15.70:0/1777745891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb7882f60 0x560bb78833a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e0a04a700 1 -- 172.21.15.70:0/1777745891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb7882f60 msgr2=0x560bb78833a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:12.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e0a04a700 1 --2- 172.21.15.70:0/1777745891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb7882f60 0x560bb78833a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e0a04a700 1 -- 172.21.15.70:0/1777745891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787f210 msgr2=0x560bb7882940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:12.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e0a04a700 1 --2- 172.21.15.70:0/1777745891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787f210 0x560bb7882940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e0a04a700 1 -- 172.21.15.70:0/1777745891 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bb771d8d0 con 0x560bb787e860 2024-06-04T04:59:12.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e09849700 1 --2- 172.21.15.70:0/1777745891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787f210 0x560bb7882940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:12.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3e0a04a700 1 --2- 172.21.15.70:0/1777745891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787e860 0x560bb787ec40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3e000dff30 tx=0x7f3e000a94e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d116f0d53195e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:12.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.069+0000 7f3df8d70700 1 -- 172.21.15.70:0/1777745891 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e000b2270 con 0x560bb787e860 2024-06-04T04:59:12.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.070+0000 7f3df8d70700 1 -- 172.21.15.70:0/1777745891 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e000a95d0 con 0x560bb787e860 2024-06-04T04:59:12.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.070+0000 7f3df8d70700 1 -- 172.21.15.70:0/1777745891 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e000a9d20 con 0x560bb787e860 2024-06-04T04:59:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.070+0000 7f3e16b43b80 1 -- 172.21.15.70:0/1777745891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787e860 msgr2=0x560bb787ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.070+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/1777745891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787e860 0x560bb787ec40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3e000dff30 tx=0x7f3e000a94e0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.070+0000 7f3e16b43b80 1 -- 172.21.15.70:0/1777745891 shutdown_connections 2024-06-04T04:59:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.070+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/1777745891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb7882f60 0x560bb78833a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.070+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/1777745891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787e860 0x560bb787ec40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.070+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/1777745891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787f210 0x560bb7882940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.070+0000 7f3e16b43b80 1 -- 172.21.15.70:0/1777745891 >> 172.21.15.70:0/1777745891 conn(0x560bb77ac080 msgr2=0x560bb7878e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.071+0000 7f3e16b43b80 1 -- 172.21.15.70:0/1777745891 shutdown_connections 2024-06-04T04:59:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.071+0000 7f3e16b43b80 1 -- 172.21.15.70:0/1777745891 wait complete. 2024-06-04T04:59:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.071+0000 7f3e16b43b80 1 -- start start 2024-06-04T04:59:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.071+0000 7f3e16b43b80 1 Processor -- start 2024-06-04T04:59:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.071+0000 7f3e16b43b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787e860 0x560bb788a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.071+0000 7f3e16b43b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787f210 0x560bb7887710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.071+0000 7f3e16b43b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb7882f60 0x560bb7887c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.071+0000 7f3e16b43b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560bb76d3fa0 con 0x560bb787e860 2024-06-04T04:59:12.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.071+0000 7f3e16b43b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560bb776f7c0 con 0x560bb7882f60 2024-06-04T04:59:12.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.071+0000 7f3e16b43b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560bb788b120 con 0x560bb787f210 2024-06-04T04:59:12.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.071+0000 7f3e16b43b80 10 client.0 _ll_register_callbacks cb 0x560bb7768920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:12.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3e09849700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787f210 0x560bb7887710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3e09849700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787f210 0x560bb7887710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46174/0 (socket says 172.21.15.70:46174) 2024-06-04T04:59:12.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3e0a04a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787e860 0x560bb788a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3e09849700 1 -- 172.21.15.70:0/2563876859 learned_addr learned my addr 172.21.15.70:0/2563876859 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:12.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3e0a84b700 1 --2- 172.21.15.70:0/2563876859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb7882f60 0x560bb7887c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3e09849700 1 -- 172.21.15.70:0/2563876859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb7882f60 msgr2=0x560bb7887c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3e09849700 1 --2- 172.21.15.70:0/2563876859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb7882f60 0x560bb7887c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3e09849700 1 -- 172.21.15.70:0/2563876859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787e860 msgr2=0x560bb788a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3e09849700 1 --2- 172.21.15.70:0/2563876859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787e860 0x560bb788a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3e09849700 1 -- 172.21.15.70:0/2563876859 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bb776c690 con 0x560bb787f210 2024-06-04T04:59:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3e0a04a700 1 --2- 172.21.15.70:0/2563876859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787e860 0x560bb788a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3e09849700 1 --2- 172.21.15.70:0/2563876859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787f210 0x560bb7887710 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3df4005950 tx=0x7f3df40047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d142db2a337d3f47 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3dedffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:59:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3dedffb700 1 -- 172.21.15.70:0/2563876859 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df402a060 con 0x560bb787f210 2024-06-04T04:59:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3dedffb700 1 -- 172.21.15.70:0/2563876859 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f3df401dd00 con 0x560bb787f210 2024-06-04T04:59:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3e16b43b80 1 -- 172.21.15.70:0/2563876859 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e000a95d0 con 0x560bb787f210 2024-06-04T04:59:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3dedffb700 1 -- 172.21.15.70:0/2563876859 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3df4016070 con 0x560bb787f210 2024-06-04T04:59:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3e16b43b80 10 client.25507 fetch_fsmap learned FSMap version 29 2024-06-04T04:59:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.072+0000 7f3e16b43b80 1 -- 172.21.15.70:0/2563876859 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560bb78958e0 con 0x560bb787f210 2024-06-04T04:59:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.073+0000 7f3dedffb700 1 -- 172.21.15.70:0/2563876859 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e000a95d0 con 0x560bb787f210 2024-06-04T04:59:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.073+0000 7f3dedffb700 1 -- 172.21.15.70:0/2563876859 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f3df4024030 con 0x560bb787f210 2024-06-04T04:59:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.073+0000 7f3e16b43b80 10 client.25507 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:59:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.073+0000 7f3e16b43b80 10 client.25507 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T04:59:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.073+0000 7f3e16b43b80 20 client.25507 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.073+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/2563876859 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x560bb7897f20 0x560bb789a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.073+0000 7f3e16b43b80 4 client.25507 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:59:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.073+0000 7f3e16b43b80 1 -- 172.21.15.70:0/2563876859 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bb784a9e0 con 0x560bb7897f20 2024-06-04T04:59:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.073+0000 7f3e0a04a700 1 --2- 172.21.15.70:0/2563876859 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x560bb7897f20 0x560bb789a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.073+0000 7f3e0a04a700 1 --2- 172.21.15.70:0/2563876859 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x560bb7897f20 0x560bb789a310 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6f9c58e1c4af120f server_cookie=391c99db250089d5 in_seq=0 out_seq=0 2024-06-04T04:59:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.073+0000 7f3dedffb700 10 client.25507 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T04:59:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.074+0000 7f3dedffb700 1 -- 172.21.15.70:0/2563876859 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f3e0008ae00 con 0x560bb7897f20 2024-06-04T04:59:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.074+0000 7f3dedffb700 10 client.25507 handle_command_reply: tid=0 2024-06-04T04:59:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.074+0000 7f3e16b43b80 1 client.25507 shutdown 2024-06-04T04:59:12.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.074+0000 7f3e16b43b80 1 -- 172.21.15.70:0/2563876859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787f210 msgr2=0x560bb7887710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:12.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.074+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/2563876859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787f210 0x560bb7887710 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3df4005950 tx=0x7f3df40047c0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.074+0000 7f3e16b43b80 1 -- 172.21.15.70:0/2563876859 shutdown_connections 2024-06-04T04:59:12.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.074+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/2563876859 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x560bb7897f20 0x560bb789a310 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.074+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/2563876859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb7882f60 0x560bb7887c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.074+0000 7f3dedffb700 0 client.25507 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T04:59:12.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.074+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/2563876859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787e860 0x560bb788a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.074+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/2563876859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787f210 0x560bb7887710 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.074+0000 7f3e16b43b80 1 -- 172.21.15.70:0/2563876859 >> 172.21.15.70:0/2563876859 conn(0x560bb77ac080 msgr2=0x560bb7878510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:12.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.074+0000 7f3e16b43b80 1 -- 172.21.15.70:0/2563876859 shutdown_connections 2024-06-04T04:59:12.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.075+0000 7f3e16b43b80 1 -- 172.21.15.70:0/2563876859 wait complete. 2024-06-04T04:59:12.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.075+0000 7f3e16b43b80 20 client.25507 trim_cache size 0 max 16384 2024-06-04T04:59:12.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.092+0000 7f3e16b43b80 1 Processor -- start 2024-06-04T04:59:12.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.092+0000 7f3e16b43b80 1 -- start start 2024-06-04T04:59:12.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.092+0000 7f3e16b43b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787e860 0x560bb7889650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.092+0000 7f3e16b43b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb787f210 0x560bb7878c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.092+0000 7f3e16b43b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb7882f60 0x560bb789fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.092+0000 7f3e16b43b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e041b0020 con 0x560bb787e860 2024-06-04T04:59:12.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.092+0000 7f3e16b43b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e041b1b60 con 0x560bb787f210 2024-06-04T04:59:12.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.092+0000 7f3e16b43b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3e041b06f0 con 0x560bb7882f60 2024-06-04T04:59:12.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.092+0000 7f3e0a84b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb7882f60 0x560bb789fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.092+0000 7f3e0a04a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787e860 0x560bb7889650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.092+0000 7f3e0a84b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb7882f60 0x560bb789fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46180/0 (socket says 172.21.15.70:46180) 2024-06-04T04:59:12.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.092+0000 7f3e0a84b700 1 -- 172.21.15.70:0/2510161941 learned_addr learned my addr 172.21.15.70:0/2510161941 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:12.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.092+0000 7f3e09849700 1 --2- 172.21.15.70:0/2510161941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb787f210 0x560bb7878c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.093+0000 7f3e0a84b700 1 -- 172.21.15.70:0/2510161941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb787f210 msgr2=0x560bb7878c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:12.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.093+0000 7f3e0a84b700 1 --2- 172.21.15.70:0/2510161941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb787f210 0x560bb7878c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.093+0000 7f3e0a84b700 1 -- 172.21.15.70:0/2510161941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787e860 msgr2=0x560bb7889650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:12.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.093+0000 7f3e0a84b700 1 --2- 172.21.15.70:0/2510161941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787e860 0x560bb7889650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.093+0000 7f3e0a84b700 1 -- 172.21.15.70:0/2510161941 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bb7886670 con 0x560bb7882f60 2024-06-04T04:59:12.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.093+0000 7f3e0a04a700 1 --2- 172.21.15.70:0/2510161941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787e860 0x560bb7889650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:12.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.093+0000 7f3e0a84b700 1 --2- 172.21.15.70:0/2510161941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb7882f60 0x560bb789fc20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc015000 tx=0x7f3dfc009470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a33b73759b473517 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:12.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.093+0000 7f3deffff700 1 -- 172.21.15.70:0/2510161941 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc00bcb0 con 0x560bb7882f60 2024-06-04T04:59:12.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.093+0000 7f3deffff700 1 -- 172.21.15.70:0/2510161941 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3dfc00be50 con 0x560bb7882f60 2024-06-04T04:59:12.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.093+0000 7f3deffff700 1 -- 172.21.15.70:0/2510161941 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc005040 con 0x560bb7882f60 2024-06-04T04:59:12.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.094+0000 7f3e16b43b80 1 -- 172.21.15.70:0/2510161941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb7882f60 msgr2=0x560bb789fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:12.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.094+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/2510161941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb7882f60 0x560bb789fc20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc015000 tx=0x7f3dfc009470 comp rx=0 tx=0).stop 2024-06-04T04:59:12.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.094+0000 7f3e16b43b80 1 -- 172.21.15.70:0/2510161941 shutdown_connections 2024-06-04T04:59:12.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.094+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/2510161941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb7882f60 0x560bb789fc20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.094+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/2510161941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787e860 0x560bb7889650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.094+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/2510161941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb787f210 0x560bb7878c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.094+0000 7f3e16b43b80 1 -- 172.21.15.70:0/2510161941 >> 172.21.15.70:0/2510161941 conn(0x560bb77ac080 msgr2=0x560bb7888f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:12.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.094+0000 7f3e16b43b80 1 -- 172.21.15.70:0/2510161941 shutdown_connections 2024-06-04T04:59:12.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.094+0000 7f3e16b43b80 1 -- 172.21.15.70:0/2510161941 wait complete. 2024-06-04T04:59:12.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.094+0000 7f3e16b43b80 1 -- start start 2024-06-04T04:59:12.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.094+0000 7f3e16b43b80 1 Processor -- start 2024-06-04T04:59:12.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e16b43b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787e860 0x560bb7890b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e16b43b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787f210 0x560bb788df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e16b43b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb7882f60 0x560bb788e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e16b43b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e041b1490 con 0x560bb787f210 2024-06-04T04:59:12.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e16b43b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e041b0dc0 con 0x560bb7882f60 2024-06-04T04:59:12.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e16b43b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3e041adfc0 con 0x560bb787e860 2024-06-04T04:59:12.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e16b43b80 10 client.0 _ll_register_callbacks cb 0x560bb7768920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:12.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e0a04a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787e860 0x560bb7890b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e0a04a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787e860 0x560bb7890b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46182/0 (socket says 172.21.15.70:46182) 2024-06-04T04:59:12.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e0a04a700 1 -- 172.21.15.70:0/3462904315 learned_addr learned my addr 172.21.15.70:0/3462904315 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:12.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e0a84b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb7882f60 0x560bb788e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e09849700 1 --2- 172.21.15.70:0/3462904315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787f210 0x560bb788df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e0a04a700 1 -- 172.21.15.70:0/3462904315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb7882f60 msgr2=0x560bb788e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:12.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e0a04a700 1 --2- 172.21.15.70:0/3462904315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb7882f60 0x560bb788e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e0a04a700 1 -- 172.21.15.70:0/3462904315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787f210 msgr2=0x560bb788df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:12.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e0a04a700 1 --2- 172.21.15.70:0/3462904315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787f210 0x560bb788df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e0a04a700 1 -- 172.21.15.70:0/3462904315 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bb788ed30 con 0x560bb787e860 2024-06-04T04:59:12.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e09849700 1 --2- 172.21.15.70:0/3462904315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787f210 0x560bb788df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:12.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3e0a04a700 1 --2- 172.21.15.70:0/3462904315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787e860 0x560bb7890b40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3e000a97c0 tx=0x7f3e0000cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59f3cf0b436e1eb1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:12.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.095+0000 7f3dedffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:59:12.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.096+0000 7f3dedffb700 1 -- 172.21.15.70:0/3462904315 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e000c9070 con 0x560bb787e860 2024-06-04T04:59:12.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.096+0000 7f3dedffb700 1 -- 172.21.15.70:0/3462904315 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f3e000a9d20 con 0x560bb787e860 2024-06-04T04:59:12.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.096+0000 7f3e16b43b80 1 -- 172.21.15.70:0/3462904315 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3dfc00be50 con 0x560bb787e860 2024-06-04T04:59:12.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.096+0000 7f3dedffb700 1 -- 172.21.15.70:0/3462904315 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3e000a9ac0 con 0x560bb787e860 2024-06-04T04:59:12.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.096+0000 7f3e16b43b80 10 client.25513 fetch_fsmap learned FSMap version 29 2024-06-04T04:59:12.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.096+0000 7f3e16b43b80 1 -- 172.21.15.70:0/3462904315 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560bb7902c00 con 0x560bb787e860 2024-06-04T04:59:12.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.096+0000 7f3dedffb700 1 -- 172.21.15.70:0/3462904315 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3dfc00be50 con 0x560bb787e860 2024-06-04T04:59:12.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.096+0000 7f3dedffb700 1 -- 172.21.15.70:0/3462904315 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f3e000b2490 con 0x560bb787e860 2024-06-04T04:59:12.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.096+0000 7f3e16b43b80 10 client.25513 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:59:12.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.096+0000 7f3e16b43b80 10 client.25513 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T04:59:12.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.096+0000 7f3e16b43b80 20 client.25513 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:12.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.096+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/3462904315 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x560bb7902e90 0x560bb7905280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:12.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.096+0000 7f3e16b43b80 4 client.25513 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T04:59:12.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.096+0000 7f3e16b43b80 1 -- 172.21.15.70:0/3462904315 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560bb77dfbe0 con 0x560bb7902e90 2024-06-04T04:59:12.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3e09849700 1 --2- 172.21.15.70:0/3462904315 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x560bb7902e90 0x560bb7905280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:12.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3dedffb700 10 client.25513 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T04:59:12.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3e09849700 1 --2- 172.21.15.70:0/3462904315 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x560bb7902e90 0x560bb7905280 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b0807bbc4477fc3a server_cookie=f8657949ba69be52 in_seq=0 out_seq=0 2024-06-04T04:59:12.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3dedffb700 1 -- 172.21.15.70:0/3462904315 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560bb77dfbe0 con 0x560bb7902e90 2024-06-04T04:59:12.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3dedffb700 10 client.25513 handle_command_reply: tid=0 2024-06-04T04:59:12.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3e16b43b80 1 client.25513 shutdown 2024-06-04T04:59:12.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3e16b43b80 1 -- 172.21.15.70:0/3462904315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787e860 msgr2=0x560bb7890b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:12.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/3462904315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787e860 0x560bb7890b40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3e000a97c0 tx=0x7f3e0000cf20 comp rx=0 tx=0).stop 2024-06-04T04:59:12.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3e16b43b80 1 -- 172.21.15.70:0/3462904315 shutdown_connections 2024-06-04T04:59:12.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/3462904315 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x560bb7902e90 0x560bb7905280 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/3462904315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bb7882f60 0x560bb788e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/3462904315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bb787e860 0x560bb7890b40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3dedffb700 0 client.25513 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T04:59:12.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3e16b43b80 1 --2- 172.21.15.70:0/3462904315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bb787f210 0x560bb788df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3e16b43b80 1 -- 172.21.15.70:0/3462904315 >> 172.21.15.70:0/3462904315 conn(0x560bb77ac080 msgr2=0x560bb7882140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:12.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3e16b43b80 1 -- 172.21.15.70:0/3462904315 shutdown_connections 2024-06-04T04:59:12.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3e16b43b80 1 -- 172.21.15.70:0/3462904315 wait complete. 2024-06-04T04:59:12.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.097+0000 7f3e16b43b80 20 client.25513 trim_cache size 0 max 16384 2024-06-04T04:59:12.106 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T04:59:12.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.098+0000 7f3e0c2ae700 1 -- 172.21.15.70:0/1016721174 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3df00b2a70 msgr2=0x7f3df00b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:12.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.098+0000 7f3e0c2ae700 1 --2- 172.21.15.70:0/1016721174 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3df00b2a70 0x7f3df00b4f30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3e0407ad30 tx=0x7f3df400a300 comp rx=0 tx=0).stop 2024-06-04T04:59:12.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.098+0000 7f3e0c2ae700 1 -- 172.21.15.70:0/1016721174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e04103590 msgr2=0x7f3e04075840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:12.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.098+0000 7f3e0c2ae700 1 --2- 172.21.15.70:0/1016721174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e04103590 0x7f3e04075840 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3e0000e9c0 tx=0x7f3e0000ecd0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.098+0000 7f3e0c2ae700 1 -- 172.21.15.70:0/1016721174 shutdown_connections 2024-06-04T04:59:12.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.098+0000 7f3e0c2ae700 1 --2- 172.21.15.70:0/1016721174 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3df00b2a70 0x7f3df00b4f30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.098+0000 7f3e0c2ae700 1 --2- 172.21.15.70:0/1016721174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e041096a0 0x7f3e040762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.098+0000 7f3e0c2ae700 1 --2- 172.21.15.70:0/1016721174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e04103590 0x7f3e04075840 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.098+0000 7f3e0c2ae700 1 --2- 172.21.15.70:0/1016721174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e04103f50 0x7f3e04075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:12.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.098+0000 7f3e0c2ae700 1 -- 172.21.15.70:0/1016721174 >> 172.21.15.70:0/1016721174 conn(0x7f3e0407a6f0 msgr2=0x7f3e041021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:12.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.098+0000 7f3e0c2ae700 1 -- 172.21.15.70:0/1016721174 shutdown_connections 2024-06-04T04:59:12.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:12.098+0000 7f3e0c2ae700 1 -- 172.21.15.70:0/1016721174 wait complete. 2024-06-04T04:59:12.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 3890 7.97 MB/sec warmup 5 sec latency 14.576 ms 2024-06-04T04:59:12.926 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T04:59:13.190 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:59:13.426 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:13 smithi070 ceph-mon[32631]: pgmap v64: 97 pgs: 97 active+clean; 23 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 2.1 MiB/s wr, 76 op/s 2024-06-04T04:59:13.435 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:13 smithi112 ceph-mon[36453]: pgmap v64: 97 pgs: 97 active+clean; 23 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 2.1 MiB/s wr, 76 op/s 2024-06-04T04:59:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:13 smithi098 ceph-mon[35569]: pgmap v64: 97 pgs: 97 active+clean; 23 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 2.1 MiB/s wr, 76 op/s 2024-06-04T04:59:13.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 4388 6.77 MB/sec warmup 6 sec latency 15.950 ms 2024-06-04T04:59:14.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 5151 6.47 MB/sec warmup 7 sec latency 13.029 ms 2024-06-04T04:59:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.921+0000 7f48305a7700 1 -- 172.21.15.70:0/87925611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48281041a0 msgr2=0x7f48281081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.921+0000 7f48305a7700 1 --2- 172.21.15.70:0/87925611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48281041a0 0x7f48281081c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f48180099c0 tx=0x7f4818009cd0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.921+0000 7f48305a7700 1 -- 172.21.15.70:0/87925611 shutdown_connections 2024-06-04T04:59:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.921+0000 7f48305a7700 1 --2- 172.21.15.70:0/87925611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48281088f0 0x7f4828110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.921+0000 7f48305a7700 1 --2- 172.21.15.70:0/87925611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48281037f0 0x7f4828103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.921+0000 7f48305a7700 1 --2- 172.21.15.70:0/87925611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48281041a0 0x7f48281081c0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.921+0000 7f48305a7700 1 -- 172.21.15.70:0/87925611 >> 172.21.15.70:0/87925611 conn(0x7f482807a6a0 msgr2=0x7f482807aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.921+0000 7f48305a7700 1 -- 172.21.15.70:0/87925611 shutdown_connections 2024-06-04T04:59:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.921+0000 7f48305a7700 1 -- 172.21.15.70:0/87925611 wait complete. 2024-06-04T04:59:14.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.922+0000 7f48305a7700 1 Processor -- start 2024-06-04T04:59:14.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.922+0000 7f48305a7700 1 -- start start 2024-06-04T04:59:14.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.922+0000 7f48305a7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48281037f0 0x7f4828075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.922+0000 7f48305a7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48281041a0 0x7f4828075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.922+0000 7f48305a7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48281088f0 0x7f48280762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.922+0000 7f48305a7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4828113580 con 0x7f48281037f0 2024-06-04T04:59:14.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.922+0000 7f48305a7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4828113400 con 0x7f48281088f0 2024-06-04T04:59:14.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.922+0000 7f48305a7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4828113700 con 0x7f48281041a0 2024-06-04T04:59:14.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.922+0000 7f482e343700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48281037f0 0x7f4828075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.923+0000 7f482db42700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48281041a0 0x7f4828075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.923+0000 7f482db42700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48281041a0 0x7f4828075d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46200/0 (socket says 172.21.15.70:46200) 2024-06-04T04:59:14.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.923+0000 7f482db42700 1 -- 172.21.15.70:0/3926463959 learned_addr learned my addr 172.21.15.70:0/3926463959 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:14.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.923+0000 7f482eb44700 1 --2- 172.21.15.70:0/3926463959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48281088f0 0x7f48280762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.923+0000 7f482e343700 1 -- 172.21.15.70:0/3926463959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48281041a0 msgr2=0x7f4828075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:14.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.923+0000 7f482e343700 1 --2- 172.21.15.70:0/3926463959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48281041a0 0x7f4828075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.923+0000 7f482e343700 1 -- 172.21.15.70:0/3926463959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48281088f0 msgr2=0x7f48280762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:14.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.923+0000 7f482e343700 1 --2- 172.21.15.70:0/3926463959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48281088f0 0x7f48280762c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.923+0000 7f482e343700 1 -- 172.21.15.70:0/3926463959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48281a7fb0 con 0x7f48281037f0 2024-06-04T04:59:14.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.923+0000 7f482db42700 1 --2- 172.21.15.70:0/3926463959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48281041a0 0x7f4828075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:14.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.923+0000 7f482e343700 1 --2- 172.21.15.70:0/3926463959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48281037f0 0x7f4828075840 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f482400c970 tx=0x7f482400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:14.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.923+0000 7f481f7fe700 1 -- 172.21.15.70:0/3926463959 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4824004a30 con 0x7f48281037f0 2024-06-04T04:59:14.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.923+0000 7f481f7fe700 1 -- 172.21.15.70:0/3926463959 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f48240054b0 con 0x7f48281037f0 2024-06-04T04:59:14.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.923+0000 7f48305a7700 1 -- 172.21.15.70:0/3926463959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48281a82a0 con 0x7f48281037f0 2024-06-04T04:59:14.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.923+0000 7f481f7fe700 1 -- 172.21.15.70:0/3926463959 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482400eae0 con 0x7f48281037f0 2024-06-04T04:59:14.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.923+0000 7f48305a7700 1 -- 172.21.15.70:0/3926463959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48281a87b0 con 0x7f48281037f0 2024-06-04T04:59:14.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.926+0000 7f481f7fe700 1 -- 172.21.15.70:0/3926463959 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f4824004bd0 con 0x7f48281037f0 2024-06-04T04:59:14.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.927+0000 7f481f7fe700 1 --2- 172.21.15.70:0/3926463959 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48140b29f0 0x7f48140b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.928+0000 7f482db42700 1 --2- 172.21.15.70:0/3926463959 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48140b29f0 0x7f48140b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.928+0000 7f481f7fe700 1 -- 172.21.15.70:0/3926463959 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f48240e1ff0 con 0x7f48281037f0 2024-06-04T04:59:14.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.928+0000 7f482db42700 1 --2- 172.21.15.70:0/3926463959 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48140b29f0 0x7f48140b4eb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4828077280 tx=0x7f4818005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:14.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.928+0000 7f483ae3cb80 1 Processor -- start 2024-06-04T04:59:14.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.928+0000 7f483ae3cb80 1 -- start start 2024-06-04T04:59:14.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.929+0000 7f483ae3cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c933320 0x55d53c933700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.929+0000 7f483ae3cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92d8a0 0x55d53c92dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.929+0000 7f483ae3cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c92e260 0x55d53c93a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.929+0000 7f483ae3cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d53c88bd70 con 0x55d53c92d8a0 2024-06-04T04:59:14.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.929+0000 7f483ae3cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d53c87b910 con 0x55d53c92e260 2024-06-04T04:59:14.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.929+0000 7f483ae3cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d53c87f950 con 0x55d53c933320 2024-06-04T04:59:14.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.929+0000 7f482db42700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92d8a0 0x55d53c92dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.929+0000 7f482e343700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c933320 0x55d53c933700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.929+0000 7f482db42700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92d8a0 0x55d53c92dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44364/0 (socket says 172.21.15.70:44364) 2024-06-04T04:59:14.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.929+0000 7f482db42700 1 -- 172.21.15.70:0/739050068 learned_addr learned my addr 172.21.15.70:0/739050068 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:14.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.929+0000 7f482eb44700 1 --2- 172.21.15.70:0/739050068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c92e260 0x55d53c93a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.929+0000 7f482eb44700 1 -- 172.21.15.70:0/739050068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c933320 msgr2=0x55d53c933700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:14.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.929+0000 7f482eb44700 1 --2- 172.21.15.70:0/739050068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c933320 0x55d53c933700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.929+0000 7f482eb44700 1 -- 172.21.15.70:0/739050068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92d8a0 msgr2=0x55d53c92dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:14.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.929+0000 7f482eb44700 1 --2- 172.21.15.70:0/739050068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92d8a0 0x55d53c92dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.929+0000 7f482eb44700 1 -- 172.21.15.70:0/739050068 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d53c824fb0 con 0x55d53c92e260 2024-06-04T04:59:14.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.929+0000 7f482e343700 1 --2- 172.21.15.70:0/739050068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c933320 0x55d53c933700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:14.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.930+0000 7f482eb44700 1 --2- 172.21.15.70:0/739050068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c92e260 0x55d53c93a080 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4820011580 tx=0x7f4820009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12b85b494001549a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:14.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.930+0000 7f481cff9700 1 -- 172.21.15.70:0/739050068 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4820004db0 con 0x55d53c92e260 2024-06-04T04:59:14.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.930+0000 7f481cff9700 1 -- 172.21.15.70:0/739050068 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4820003a00 con 0x55d53c92e260 2024-06-04T04:59:14.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.930+0000 7f481cff9700 1 -- 172.21.15.70:0/739050068 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4820004580 con 0x55d53c92e260 2024-06-04T04:59:14.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.931+0000 7f483ae3cb80 1 -- 172.21.15.70:0/739050068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c92e260 msgr2=0x55d53c93a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:14.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.931+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/739050068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c92e260 0x55d53c93a080 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4820011580 tx=0x7f4820009920 comp rx=0 tx=0).stop 2024-06-04T04:59:14.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.931+0000 7f483ae3cb80 1 -- 172.21.15.70:0/739050068 shutdown_connections 2024-06-04T04:59:14.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.931+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/739050068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c92e260 0x55d53c93a080 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.931+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/739050068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c933320 0x55d53c933700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.931+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/739050068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92d8a0 0x55d53c92dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.931+0000 7f483ae3cb80 1 -- 172.21.15.70:0/739050068 >> 172.21.15.70:0/739050068 conn(0x55d53c859240 msgr2=0x55d53c9283e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:14.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.931+0000 7f483ae3cb80 1 -- 172.21.15.70:0/739050068 shutdown_connections 2024-06-04T04:59:14.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.931+0000 7f483ae3cb80 1 -- 172.21.15.70:0/739050068 wait complete. 2024-06-04T04:59:14.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.931+0000 7f483ae3cb80 1 -- start start 2024-06-04T04:59:14.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.931+0000 7f483ae3cb80 1 Processor -- start 2024-06-04T04:59:14.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f483ae3cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c92d8a0 0x55d53c939cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f483ae3cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92e260 0x55d53c936ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f483ae3cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c933320 0x55d53c937530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f483ae3cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d53c781fa0 con 0x55d53c92e260 2024-06-04T04:59:14.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f483ae3cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d53c7e3da0 con 0x55d53c933320 2024-06-04T04:59:14.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f483ae3cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d53c93a6e0 con 0x55d53c92d8a0 2024-06-04T04:59:14.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f483ae3cb80 10 client.0 _ll_register_callbacks cb 0x55d53c816950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:14.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f482db42700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92e260 0x55d53c936ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f482eb44700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c933320 0x55d53c937530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f482db42700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92e260 0x55d53c936ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44372/0 (socket says 172.21.15.70:44372) 2024-06-04T04:59:14.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f482db42700 1 -- 172.21.15.70:0/511510358 learned_addr learned my addr 172.21.15.70:0/511510358 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:14.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f482e343700 1 --2- 172.21.15.70:0/511510358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c92d8a0 0x55d53c939cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f482eb44700 1 -- 172.21.15.70:0/511510358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c92d8a0 msgr2=0x55d53c939cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:14.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f482eb44700 1 --2- 172.21.15.70:0/511510358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c92d8a0 0x55d53c939cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f482eb44700 1 -- 172.21.15.70:0/511510358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92e260 msgr2=0x55d53c936ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:14.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f482eb44700 1 --2- 172.21.15.70:0/511510358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92e260 0x55d53c936ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f482eb44700 1 -- 172.21.15.70:0/511510358 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d53c7cb8d0 con 0x55d53c933320 2024-06-04T04:59:14.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f482e343700 1 --2- 172.21.15.70:0/511510358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c92d8a0 0x55d53c939cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:14.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f482eb44700 1 --2- 172.21.15.70:0/511510358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c933320 0x55d53c937530 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4820012010 tx=0x7f4820004e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60078a081dd1a90f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:14.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.932+0000 7f4811ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T04:59:14.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.933+0000 7f4811ffb700 1 -- 172.21.15.70:0/511510358 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4820004200 con 0x55d53c933320 2024-06-04T04:59:14.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.933+0000 7f483ae3cb80 1 -- 172.21.15.70:0/511510358 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4820003a00 con 0x55d53c933320 2024-06-04T04:59:14.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.933+0000 7f4811ffb700 1 -- 172.21.15.70:0/511510358 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f48200043a0 con 0x55d53c933320 2024-06-04T04:59:14.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.933+0000 7f4811ffb700 1 -- 172.21.15.70:0/511510358 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f48200045a0 con 0x55d53c933320 2024-06-04T04:59:14.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.933+0000 7f483ae3cb80 10 client.25463 fetch_fsmap learned FSMap version 29 2024-06-04T04:59:14.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.933+0000 7f483ae3cb80 1 -- 172.21.15.70:0/511510358 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d53c835760 con 0x55d53c933320 2024-06-04T04:59:14.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.933+0000 7f4811ffb700 1 -- 172.21.15.70:0/511510358 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4820003a00 con 0x55d53c933320 2024-06-04T04:59:14.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.934+0000 7f4811ffb700 1 -- 172.21.15.70:0/511510358 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f4820025070 con 0x55d53c933320 2024-06-04T04:59:14.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.934+0000 7f483ae3cb80 10 client.25463 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:59:14.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.934+0000 7f483ae3cb80 10 client.25463 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T04:59:14.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.934+0000 7f483ae3cb80 20 client.25463 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:14.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.934+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/511510358 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d53c94f950 0x55d53c951d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.934+0000 7f483ae3cb80 4 client.25463 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:59:14.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.934+0000 7f483ae3cb80 1 -- 172.21.15.70:0/511510358 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d53c88dcc0 con 0x55d53c94f950 2024-06-04T04:59:14.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.934+0000 7f482e343700 1 --2- 172.21.15.70:0/511510358 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d53c94f950 0x55d53c951d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.934+0000 7f4811ffb700 10 client.25463 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T04:59:14.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.934+0000 7f482e343700 1 --2- 172.21.15.70:0/511510358 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d53c94f950 0x55d53c951d40 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2d6ac619c43d01c1 server_cookie=daf27bafdf6641f3 in_seq=0 out_seq=0 2024-06-04T04:59:14.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.935+0000 7f4811ffb700 1 -- 172.21.15.70:0/511510358 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d53c88dcc0 con 0x55d53c94f950 2024-06-04T04:59:14.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.935+0000 7f4811ffb700 10 client.25463 handle_command_reply: tid=0 2024-06-04T04:59:14.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.935+0000 7f483ae3cb80 1 client.25463 shutdown 2024-06-04T04:59:14.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.935+0000 7f483ae3cb80 1 -- 172.21.15.70:0/511510358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c933320 msgr2=0x55d53c937530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:14.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.935+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/511510358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c933320 0x55d53c937530 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4820012010 tx=0x7f4820004e40 comp rx=0 tx=0).stop 2024-06-04T04:59:14.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.935+0000 7f483ae3cb80 1 -- 172.21.15.70:0/511510358 shutdown_connections 2024-06-04T04:59:14.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.935+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/511510358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c933320 0x55d53c937530 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.935+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/511510358 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d53c94f950 0x55d53c951d40 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.935+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/511510358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c92d8a0 0x55d53c939cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.935+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/511510358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92e260 0x55d53c936ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.935+0000 7f483ae3cb80 1 -- 172.21.15.70:0/511510358 >> 172.21.15.70:0/511510358 conn(0x55d53c859240 msgr2=0x55d53c927af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:14.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.935+0000 7f4811ffb700 0 client.25463 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T04:59:14.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.935+0000 7f483ae3cb80 1 -- 172.21.15.70:0/511510358 shutdown_connections 2024-06-04T04:59:14.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.935+0000 7f483ae3cb80 1 -- 172.21.15.70:0/511510358 wait complete. 2024-06-04T04:59:14.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.935+0000 7f483ae3cb80 20 client.25463 trim_cache size 0 max 16384 2024-06-04T04:59:14.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.951+0000 7f483ae3cb80 1 Processor -- start 2024-06-04T04:59:14.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.952+0000 7f483ae3cb80 1 -- start start 2024-06-04T04:59:14.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.952+0000 7f483ae3cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92d8a0 0x55d53c933700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.952+0000 7f483ae3cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c92e260 0x55d53c932830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.952+0000 7f483ae3cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c933320 0x55d53c92c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.952+0000 7f483ae3cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f48281abc20 con 0x55d53c92d8a0 2024-06-04T04:59:14.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.952+0000 7f483ae3cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48281ad760 con 0x55d53c933320 2024-06-04T04:59:14.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.952+0000 7f483ae3cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f48281ac2f0 con 0x55d53c92e260 2024-06-04T04:59:14.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.952+0000 7f482e343700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92d8a0 0x55d53c933700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.952+0000 7f482e343700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92d8a0 0x55d53c933700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44380/0 (socket says 172.21.15.70:44380) 2024-06-04T04:59:14.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.952+0000 7f482e343700 1 -- 172.21.15.70:0/1732165551 learned_addr learned my addr 172.21.15.70:0/1732165551 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:14.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.952+0000 7f482db42700 1 --2- 172.21.15.70:0/1732165551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c92e260 0x55d53c932830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.952+0000 7f482eb44700 1 --2- 172.21.15.70:0/1732165551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c933320 0x55d53c92c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.952+0000 7f482e343700 1 -- 172.21.15.70:0/1732165551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c92e260 msgr2=0x55d53c932830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:14.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.952+0000 7f482e343700 1 --2- 172.21.15.70:0/1732165551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c92e260 0x55d53c932830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.952+0000 7f482e343700 1 -- 172.21.15.70:0/1732165551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c933320 msgr2=0x55d53c92c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:14.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.952+0000 7f482e343700 1 --2- 172.21.15.70:0/1732165551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c933320 0x55d53c92c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.952+0000 7f482e343700 1 -- 172.21.15.70:0/1732165551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d53c935020 con 0x55d53c92d8a0 2024-06-04T04:59:14.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.953+0000 7f482e343700 1 --2- 172.21.15.70:0/1732165551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92d8a0 0x55d53c933700 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f48240a9260 tx=0x7f482401a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c70fe766ae0802b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:14.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.953+0000 7f4813fff700 1 -- 172.21.15.70:0/1732165551 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482401d070 con 0x55d53c92d8a0 2024-06-04T04:59:14.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.953+0000 7f4813fff700 1 -- 172.21.15.70:0/1732165551 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f48240b2130 con 0x55d53c92d8a0 2024-06-04T04:59:14.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.953+0000 7f4813fff700 1 -- 172.21.15.70:0/1732165551 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48240b1f00 con 0x55d53c92d8a0 2024-06-04T04:59:14.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.953+0000 7f483ae3cb80 1 -- 172.21.15.70:0/1732165551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92d8a0 msgr2=0x55d53c933700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:14.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.953+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/1732165551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92d8a0 0x55d53c933700 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f48240a9260 tx=0x7f482401a040 comp rx=0 tx=0).stop 2024-06-04T04:59:14.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.954+0000 7f483ae3cb80 1 -- 172.21.15.70:0/1732165551 shutdown_connections 2024-06-04T04:59:14.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.954+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/1732165551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c933320 0x55d53c92c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.954+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/1732165551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92d8a0 0x55d53c933700 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.954+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/1732165551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c92e260 0x55d53c932830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.954+0000 7f483ae3cb80 1 -- 172.21.15.70:0/1732165551 >> 172.21.15.70:0/1732165551 conn(0x55d53c859240 msgr2=0x55d53c932120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:14.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.954+0000 7f483ae3cb80 1 -- 172.21.15.70:0/1732165551 shutdown_connections 2024-06-04T04:59:14.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.954+0000 7f483ae3cb80 1 -- 172.21.15.70:0/1732165551 wait complete. 2024-06-04T04:59:14.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.954+0000 7f483ae3cb80 1 -- start start 2024-06-04T04:59:14.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.954+0000 7f483ae3cb80 1 Processor -- start 2024-06-04T04:59:14.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.954+0000 7f483ae3cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c92d8a0 0x55d53c956510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.954+0000 7f483ae3cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92e260 0x55d53c956a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.954+0000 7f483ae3cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c933320 0x55d53c956f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.954+0000 7f483ae3cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f48281ad090 con 0x55d53c92e260 2024-06-04T04:59:14.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.954+0000 7f483ae3cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48281ac9c0 con 0x55d53c92d8a0 2024-06-04T04:59:14.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.954+0000 7f483ae3cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f48281a9c80 con 0x55d53c933320 2024-06-04T04:59:14.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.954+0000 7f483ae3cb80 10 client.0 _ll_register_callbacks cb 0x55d53c816950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:14.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.955+0000 7f482db42700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92e260 0x55d53c956a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.955+0000 7f482db42700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92e260 0x55d53c956a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44392/0 (socket says 172.21.15.70:44392) 2024-06-04T04:59:14.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.955+0000 7f482db42700 1 -- 172.21.15.70:0/3607810646 learned_addr learned my addr 172.21.15.70:0/3607810646 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:14.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.955+0000 7f482eb44700 1 --2- 172.21.15.70:0/3607810646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c933320 0x55d53c956f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.955+0000 7f482e343700 1 --2- 172.21.15.70:0/3607810646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c92d8a0 0x55d53c956510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.955+0000 7f482db42700 1 -- 172.21.15.70:0/3607810646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c933320 msgr2=0x55d53c956f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:14.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.955+0000 7f482db42700 1 --2- 172.21.15.70:0/3607810646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c933320 0x55d53c956f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.955+0000 7f482db42700 1 -- 172.21.15.70:0/3607810646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c92d8a0 msgr2=0x55d53c956510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:14.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.955+0000 7f482db42700 1 --2- 172.21.15.70:0/3607810646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c92d8a0 0x55d53c956510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.955+0000 7f482db42700 1 -- 172.21.15.70:0/3607810646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d53c9471c0 con 0x55d53c92e260 2024-06-04T04:59:14.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.955+0000 7f482e343700 1 --2- 172.21.15.70:0/3607810646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c92d8a0 0x55d53c956510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:59:14.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.955+0000 7f482eb44700 1 --2- 172.21.15.70:0/3607810646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c933320 0x55d53c956f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:14.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.955+0000 7f482db42700 1 --2- 172.21.15.70:0/3607810646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92e260 0x55d53c956a50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f481800bda0 tx=0x7f481801a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba083dfefe1e34b9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:14.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.955+0000 7f4811ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:59:14.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.955+0000 7f4811ffb700 1 -- 172.21.15.70:0/3607810646 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48180054e0 con 0x55d53c92e260 2024-06-04T04:59:14.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.955+0000 7f483ae3cb80 1 -- 172.21.15.70:0/3607810646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48240b2130 con 0x55d53c92e260 2024-06-04T04:59:14.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.955+0000 7f4811ffb700 1 -- 172.21.15.70:0/3607810646 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f4818005680 con 0x55d53c92e260 2024-06-04T04:59:14.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.956+0000 7f4811ffb700 1 -- 172.21.15.70:0/3607810646 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4818040050 con 0x55d53c92e260 2024-06-04T04:59:14.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.956+0000 7f483ae3cb80 10 client.16062 fetch_fsmap learned FSMap version 29 2024-06-04T04:59:14.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.956+0000 7f483ae3cb80 1 -- 172.21.15.70:0/3607810646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d53c948450 con 0x55d53c92e260 2024-06-04T04:59:14.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.956+0000 7f4811ffb700 1 -- 172.21.15.70:0/3607810646 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f48240b2130 con 0x55d53c92e260 2024-06-04T04:59:14.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.956+0000 7f4811ffb700 1 -- 172.21.15.70:0/3607810646 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f4818003ac0 con 0x55d53c92e260 2024-06-04T04:59:14.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.956+0000 7f483ae3cb80 10 client.16062 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:59:14.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.956+0000 7f483ae3cb80 10 client.16062 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T04:59:14.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.956+0000 7f483ae3cb80 20 client.16062 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:14.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.956+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/3607810646 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d53c9ba3f0 0x55d53c9bc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:14.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.956+0000 7f483ae3cb80 4 client.16062 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T04:59:14.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.956+0000 7f483ae3cb80 1 -- 172.21.15.70:0/3607810646 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d53c8874a0 con 0x55d53c9ba3f0 2024-06-04T04:59:14.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.956+0000 7f482e343700 1 --2- 172.21.15.70:0/3607810646 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d53c9ba3f0 0x55d53c9bc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:14.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.957+0000 7f482e343700 1 --2- 172.21.15.70:0/3607810646 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d53c9ba3f0 0x55d53c9bc7e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ef658c9aaa517bdb server_cookie=b67b83867404a0c0 in_seq=0 out_seq=0 2024-06-04T04:59:14.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.957+0000 7f4811ffb700 10 client.16062 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T04:59:14.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.957+0000 7f4811ffb700 1 -- 172.21.15.70:0/3607810646 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d53c8874a0 con 0x55d53c9ba3f0 2024-06-04T04:59:14.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.957+0000 7f4811ffb700 10 client.16062 handle_command_reply: tid=0 2024-06-04T04:59:14.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.957+0000 7f483ae3cb80 1 client.16062 shutdown 2024-06-04T04:59:14.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.957+0000 7f483ae3cb80 1 -- 172.21.15.70:0/3607810646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92e260 msgr2=0x55d53c956a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:14.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.957+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/3607810646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92e260 0x55d53c956a50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f481800bda0 tx=0x7f481801a050 comp rx=0 tx=0).stop 2024-06-04T04:59:14.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.957+0000 7f483ae3cb80 1 -- 172.21.15.70:0/3607810646 shutdown_connections 2024-06-04T04:59:14.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.957+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/3607810646 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d53c9ba3f0 0x55d53c9bc7e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.957+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/3607810646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d53c933320 0x55d53c956f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.957+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/3607810646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d53c92d8a0 0x55d53c956510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.957+0000 7f4811ffb700 0 client.16062 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T04:59:14.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.957+0000 7f483ae3cb80 1 --2- 172.21.15.70:0/3607810646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d53c92e260 0x55d53c956a50 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.957+0000 7f483ae3cb80 1 -- 172.21.15.70:0/3607810646 >> 172.21.15.70:0/3607810646 conn(0x55d53c859240 msgr2=0x55d53c930ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:14.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.957+0000 7f483ae3cb80 1 -- 172.21.15.70:0/3607810646 shutdown_connections 2024-06-04T04:59:14.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.957+0000 7f483ae3cb80 1 -- 172.21.15.70:0/3607810646 wait complete. 2024-06-04T04:59:14.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.957+0000 7f483ae3cb80 20 client.16062 trim_cache size 0 max 16384 2024-06-04T04:59:14.965 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T04:59:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.958+0000 7f48305a7700 1 -- 172.21.15.70:0/3926463959 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48140b29f0 msgr2=0x7f48140b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.958+0000 7f48305a7700 1 --2- 172.21.15.70:0/3926463959 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48140b29f0 0x7f48140b4eb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4828077280 tx=0x7f4818005960 comp rx=0 tx=0).stop 2024-06-04T04:59:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.958+0000 7f48305a7700 1 -- 172.21.15.70:0/3926463959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48281037f0 msgr2=0x7f4828075840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.958+0000 7f48305a7700 1 --2- 172.21.15.70:0/3926463959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48281037f0 0x7f4828075840 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f482400c970 tx=0x7f482400cc80 comp rx=0 tx=0).stop 2024-06-04T04:59:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.958+0000 7f48305a7700 1 -- 172.21.15.70:0/3926463959 shutdown_connections 2024-06-04T04:59:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.958+0000 7f48305a7700 1 --2- 172.21.15.70:0/3926463959 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48140b29f0 0x7f48140b4eb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.958+0000 7f48305a7700 1 --2- 172.21.15.70:0/3926463959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48281088f0 0x7f48280762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.958+0000 7f48305a7700 1 --2- 172.21.15.70:0/3926463959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48281037f0 0x7f4828075840 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.958+0000 7f48305a7700 1 --2- 172.21.15.70:0/3926463959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48281041a0 0x7f4828075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.958+0000 7f48305a7700 1 -- 172.21.15.70:0/3926463959 >> 172.21.15.70:0/3926463959 conn(0x7f482807a6a0 msgr2=0x7f4828101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.958+0000 7f48305a7700 1 -- 172.21.15.70:0/3926463959 shutdown_connections 2024-06-04T04:59:14.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:14.958+0000 7f48305a7700 1 -- 172.21.15.70:0/3926463959 wait complete. 2024-06-04T04:59:15.320 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:15 smithi070 ceph-mon[32631]: pgmap v65: 97 pgs: 97 active+clean; 23 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 2.0 MiB/s wr, 56 op/s 2024-06-04T04:59:15.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:15 smithi112 ceph-mon[36453]: pgmap v65: 97 pgs: 97 active+clean; 23 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 2.0 MiB/s wr, 56 op/s 2024-06-04T04:59:15.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:15 smithi098 ceph-mon[35569]: pgmap v65: 97 pgs: 97 active+clean; 23 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 2.0 MiB/s wr, 56 op/s 2024-06-04T04:59:15.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 5918 5.88 MB/sec warmup 8 sec latency 14.822 ms 2024-06-04T04:59:15.756 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T04:59:16.020 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:59:16.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 6843 6.05 MB/sec warmup 9 sec latency 12.444 ms 2024-06-04T04:59:17.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:17 smithi112 ceph-mon[36453]: pgmap v66: 97 pgs: 97 active+clean; 25 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 2.8 MiB/s wr, 97 op/s 2024-06-04T04:59:17.469 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:17 smithi070 ceph-mon[32631]: pgmap v66: 97 pgs: 97 active+clean; 25 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 2.8 MiB/s wr, 97 op/s 2024-06-04T04:59:17.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:17 smithi098 ceph-mon[35569]: pgmap v66: 97 pgs: 97 active+clean; 25 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 2.8 MiB/s wr, 97 op/s 2024-06-04T04:59:17.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 7373 5.89 MB/sec warmup 10 sec latency 25.565 ms 2024-06-04T04:59:17.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.844+0000 7f8a11a21700 1 -- 172.21.15.70:0/2884391017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a0c108520 msgr2=0x7f8a0c108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:17.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.844+0000 7f8a11a21700 1 --2- 172.21.15.70:0/2884391017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a0c108520 0x7f8a0c108900 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009990 tx=0x7f89fc009ca0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.844+0000 7f8a11a21700 1 -- 172.21.15.70:0/2884391017 shutdown_connections 2024-06-04T04:59:17.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.844+0000 7f8a11a21700 1 --2- 172.21.15.70:0/2884391017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a0c102e00 0x7f8a0c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.844+0000 7f8a11a21700 1 --2- 172.21.15.70:0/2884391017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a0c108520 0x7f8a0c108900 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.844+0000 7f8a11a21700 1 --2- 172.21.15.70:0/2884391017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a0c102440 0x7f8a0c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.844+0000 7f8a11a21700 1 -- 172.21.15.70:0/2884391017 >> 172.21.15.70:0/2884391017 conn(0x7f8a0c0793f0 msgr2=0x7f8a0c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:17.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.844+0000 7f8a11a21700 1 -- 172.21.15.70:0/2884391017 shutdown_connections 2024-06-04T04:59:17.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.844+0000 7f8a11a21700 1 -- 172.21.15.70:0/2884391017 wait complete. 2024-06-04T04:59:17.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.844+0000 7f8a11a21700 1 Processor -- start 2024-06-04T04:59:17.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a11a21700 1 -- start start 2024-06-04T04:59:17.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a11a21700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a0c102440 0x7f8a0c104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a11a21700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a0c102e00 0x7f8a0c1045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a11a21700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a0c108520 0x7f8a0c104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a11a21700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c112480 con 0x7f8a0c108520 2024-06-04T04:59:17.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a11a21700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c112300 con 0x7f8a0c102e00 2024-06-04T04:59:17.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a11a21700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c112600 con 0x7f8a0c102440 2024-06-04T04:59:17.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a0affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a0c102440 0x7f8a0c104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a0b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a0c108520 0x7f8a0c104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a0affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a0c102440 0x7f8a0c104080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:45180/0 (socket says 172.21.15.70:45180) 2024-06-04T04:59:17.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a0affd700 1 -- 172.21.15.70:0/3156647630 learned_addr learned my addr 172.21.15.70:0/3156647630 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:17.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a0a7fc700 1 --2- 172.21.15.70:0/3156647630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a0c102e00 0x7f8a0c1045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a0affd700 1 -- 172.21.15.70:0/3156647630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a0c102e00 msgr2=0x7f8a0c1045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:17.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a0affd700 1 --2- 172.21.15.70:0/3156647630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a0c102e00 0x7f8a0c1045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a0affd700 1 -- 172.21.15.70:0/3156647630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a0c108520 msgr2=0x7f8a0c104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:17.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a0affd700 1 --2- 172.21.15.70:0/3156647630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a0c108520 0x7f8a0c104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a0affd700 1 -- 172.21.15.70:0/3156647630 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0c1a8090 con 0x7f8a0c102440 2024-06-04T04:59:17.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.845+0000 7f8a0affd700 1 --2- 172.21.15.70:0/3156647630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a0c102440 0x7f8a0c104080 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f89fc000c00 tx=0x7f89fc0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:17.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.846+0000 7f8a10a1f700 1 -- 172.21.15.70:0/3156647630 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc027070 con 0x7f8a0c102440 2024-06-04T04:59:17.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.846+0000 7f8a10a1f700 1 -- 172.21.15.70:0/3156647630 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f89fc00fb00 con 0x7f8a0c102440 2024-06-04T04:59:17.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.846+0000 7f8a11a21700 1 -- 172.21.15.70:0/3156647630 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a0c1a8320 con 0x7f8a0c102440 2024-06-04T04:59:17.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.846+0000 7f8a10a1f700 1 -- 172.21.15.70:0/3156647630 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc022c40 con 0x7f8a0c102440 2024-06-04T04:59:17.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.846+0000 7f8a11a21700 1 -- 172.21.15.70:0/3156647630 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a0c1a8770 con 0x7f8a0c102440 2024-06-04T04:59:17.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.848+0000 7f8a10a1f700 1 -- 172.21.15.70:0/3156647630 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f89fc02a070 con 0x7f8a0c102440 2024-06-04T04:59:17.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.849+0000 7f8a10a1f700 1 --2- 172.21.15.70:0/3156647630 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f89f40b29f0 0x7f89f40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.849+0000 7f8a10a1f700 1 -- 172.21.15.70:0/3156647630 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f89fc0e34e0 con 0x7f8a0c102440 2024-06-04T04:59:17.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.849+0000 7f8a0a7fc700 1 --2- 172.21.15.70:0/3156647630 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f89f40b29f0 0x7f89f40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a1c2b6b80 1 Processor -- start 2024-06-04T04:59:17.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a1c2b6b80 1 -- start start 2024-06-04T04:59:17.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a0a7fc700 1 --2- 172.21.15.70:0/3156647630 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f89f40b29f0 0x7f89f40b4eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c079ad0 tx=0x7f8a0000b410 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:17.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a1c2b6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a6a320 0x559955a6a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a1c2b6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a648a0 0x559955a64d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a1c2b6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a71080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a1c2b6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5599559c2d70 con 0x559955a65260 2024-06-04T04:59:17.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a1c2b6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5599559b2910 con 0x559955a6a320 2024-06-04T04:59:17.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a1c2b6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5599559b6950 con 0x559955a648a0 2024-06-04T04:59:17.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a0b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a71080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a0b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a71080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40028/0 (socket says 172.21.15.70:40028) 2024-06-04T04:59:17.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a0a7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a648a0 0x559955a64d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a0b7fe700 1 -- 172.21.15.70:0/2954309522 learned_addr learned my addr 172.21.15.70:0/2954309522 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:17.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a0affd700 1 --2- 172.21.15.70:0/2954309522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a6a320 0x559955a6a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a0b7fe700 1 -- 172.21.15.70:0/2954309522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a648a0 msgr2=0x559955a64d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:17.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a0b7fe700 1 --2- 172.21.15.70:0/2954309522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a648a0 0x559955a64d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a0b7fe700 1 -- 172.21.15.70:0/2954309522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a6a320 msgr2=0x559955a6a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:17.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a0b7fe700 1 --2- 172.21.15.70:0/2954309522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a6a320 0x559955a6a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.850+0000 7f8a0b7fe700 1 -- 172.21.15.70:0/2954309522 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55995595bfb0 con 0x559955a65260 2024-06-04T04:59:17.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.851+0000 7f8a0b7fe700 1 --2- 172.21.15.70:0/2954309522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a71080 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8a04003fa0 tx=0x7f8a04009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8cca90fa5aed960 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:17.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.851+0000 7f89f1ffb700 1 -- 172.21.15.70:0/2954309522 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a04004bb0 con 0x559955a65260 2024-06-04T04:59:17.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.851+0000 7f89f1ffb700 1 -- 172.21.15.70:0/2954309522 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a040048e0 con 0x559955a65260 2024-06-04T04:59:17.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.851+0000 7f89f1ffb700 1 -- 172.21.15.70:0/2954309522 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a04007c80 con 0x559955a65260 2024-06-04T04:59:17.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.851+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/2954309522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 msgr2=0x559955a71080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:17.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.852+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/2954309522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a71080 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8a04003fa0 tx=0x7f8a04009ba0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.852+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/2954309522 shutdown_connections 2024-06-04T04:59:17.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.852+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/2954309522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a71080 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.852+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/2954309522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a6a320 0x559955a6a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.852+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/2954309522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a648a0 0x559955a64d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.852+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/2954309522 >> 172.21.15.70:0/2954309522 conn(0x559955990240 msgr2=0x559955a5f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:17.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.852+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/2954309522 shutdown_connections 2024-06-04T04:59:17.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.852+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/2954309522 wait complete. 2024-06-04T04:59:17.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.852+0000 7f8a1c2b6b80 1 -- start start 2024-06-04T04:59:17.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.852+0000 7f8a1c2b6b80 1 Processor -- start 2024-06-04T04:59:17.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a1c2b6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a648a0 0x559955a70cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a1c2b6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a6dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a1c2b6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a6a320 0x559955a6e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a1c2b6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5599558b8fa0 con 0x559955a65260 2024-06-04T04:59:17.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a1c2b6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55995591ada0 con 0x559955a648a0 2024-06-04T04:59:17.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a1c2b6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559955a716e0 con 0x559955a6a320 2024-06-04T04:59:17.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a1c2b6b80 10 client.0 _ll_register_callbacks cb 0x55995594d950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:17.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a0a7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a6dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a0a7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a6dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40038/0 (socket says 172.21.15.70:40038) 2024-06-04T04:59:17.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a0a7fc700 1 -- 172.21.15.70:0/4225922776 learned_addr learned my addr 172.21.15.70:0/4225922776 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:17.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a0b7fe700 1 --2- 172.21.15.70:0/4225922776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a6a320 0x559955a6e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a0affd700 1 --2- 172.21.15.70:0/4225922776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a648a0 0x559955a70cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a0a7fc700 1 -- 172.21.15.70:0/4225922776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a6a320 msgr2=0x559955a6e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:17.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a0a7fc700 1 --2- 172.21.15.70:0/4225922776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a6a320 0x559955a6e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a0a7fc700 1 -- 172.21.15.70:0/4225922776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a648a0 msgr2=0x559955a70cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:17.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a0a7fc700 1 --2- 172.21.15.70:0/4225922776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a648a0 0x559955a70cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a0a7fc700 1 -- 172.21.15.70:0/4225922776 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599559028d0 con 0x559955a65260 2024-06-04T04:59:17.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a0a7fc700 1 --2- 172.21.15.70:0/4225922776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a6dfb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8a000116f0 tx=0x7f8a000047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9779bef716200e0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:17.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f89eaffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:59:17.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f89eaffd700 1 -- 172.21.15.70:0/4225922776 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a00004830 con 0x559955a65260 2024-06-04T04:59:17.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f89eaffd700 1 -- 172.21.15.70:0/4225922776 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f8a000049d0 con 0x559955a65260 2024-06-04T04:59:17.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.853+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/4225922776 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a040048e0 con 0x559955a65260 2024-06-04T04:59:17.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.854+0000 7f89eaffd700 1 -- 172.21.15.70:0/4225922776 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8a00004bd0 con 0x559955a65260 2024-06-04T04:59:17.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.854+0000 7f8a1c2b6b80 10 client.16074 fetch_fsmap learned FSMap version 29 2024-06-04T04:59:17.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.854+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/4225922776 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55995596c760 con 0x559955a65260 2024-06-04T04:59:17.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.854+0000 7f89eaffd700 1 -- 172.21.15.70:0/4225922776 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a040048e0 con 0x559955a65260 2024-06-04T04:59:17.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.854+0000 7f89eaffd700 1 -- 172.21.15.70:0/4225922776 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f8a00030510 con 0x559955a65260 2024-06-04T04:59:17.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.854+0000 7f8a1c2b6b80 10 client.16074 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:59:17.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.854+0000 7f8a1c2b6b80 10 client.16074 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T04:59:17.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.854+0000 7f8a1c2b6b80 20 client.16074 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:17.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.854+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/4225922776 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x559955a7e1b0 0x559955a805a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.854+0000 7f8a1c2b6b80 4 client.16074 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:59:17.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.854+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/4225922776 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599559c4cc0 con 0x559955a7e1b0 2024-06-04T04:59:17.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.854+0000 7f8a0affd700 1 --2- 172.21.15.70:0/4225922776 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x559955a7e1b0 0x559955a805a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.855+0000 7f89eaffd700 10 client.16074 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T04:59:17.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.855+0000 7f8a0affd700 1 --2- 172.21.15.70:0/4225922776 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x559955a7e1b0 0x559955a805a0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4b2b9fb30d561672 server_cookie=2393d228ed4620ff in_seq=0 out_seq=0 2024-06-04T04:59:17.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.855+0000 7f89eaffd700 1 -- 172.21.15.70:0/4225922776 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5599559c4cc0 con 0x559955a7e1b0 2024-06-04T04:59:17.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.855+0000 7f89eaffd700 10 client.16074 handle_command_reply: tid=0 2024-06-04T04:59:17.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.855+0000 7f8a1c2b6b80 1 client.16074 shutdown 2024-06-04T04:59:17.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.856+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/4225922776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 msgr2=0x559955a6dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:17.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.856+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/4225922776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a6dfb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8a000116f0 tx=0x7f8a000047c0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.856+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/4225922776 shutdown_connections 2024-06-04T04:59:17.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.856+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/4225922776 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x559955a7e1b0 0x559955a805a0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.856+0000 7f89eaffd700 0 client.16074 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T04:59:17.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.856+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/4225922776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a6a320 0x559955a6e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.856+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/4225922776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a648a0 0x559955a70cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.856+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/4225922776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a6dfb0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.856+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/4225922776 >> 172.21.15.70:0/4225922776 conn(0x559955990240 msgr2=0x559955a5eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:17.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.856+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/4225922776 shutdown_connections 2024-06-04T04:59:17.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.856+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/4225922776 wait complete. 2024-06-04T04:59:17.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.856+0000 7f8a1c2b6b80 20 client.16074 trim_cache size 0 max 16384 2024-06-04T04:59:17.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.874+0000 7f8a1c2b6b80 1 Processor -- start 2024-06-04T04:59:17.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.874+0000 7f8a1c2b6b80 1 -- start start 2024-06-04T04:59:17.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.874+0000 7f8a1c2b6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a648a0 0x559955a805a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.874+0000 7f8a1c2b6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a65260 0x559955a5f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.874+0000 7f8a1c2b6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a6a320 0x559955a85e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.874+0000 7f8a1c2b6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c1afe70 con 0x559955a6a320 2024-06-04T04:59:17.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.874+0000 7f8a1c2b6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c1b19b0 con 0x559955a65260 2024-06-04T04:59:17.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.874+0000 7f8a1c2b6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c1b0540 con 0x559955a648a0 2024-06-04T04:59:17.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.874+0000 7f8a0b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a6a320 0x559955a85e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.875+0000 7f8a0affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a648a0 0x559955a805a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.875+0000 7f8a0b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a6a320 0x559955a85e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40040/0 (socket says 172.21.15.70:40040) 2024-06-04T04:59:17.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.875+0000 7f8a0b7fe700 1 -- 172.21.15.70:0/3095752246 learned_addr learned my addr 172.21.15.70:0/3095752246 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:17.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.875+0000 7f8a0affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a648a0 0x559955a805a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:45206/0 (socket says 172.21.15.70:45206) 2024-06-04T04:59:17.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.875+0000 7f8a0a7fc700 1 --2- 172.21.15.70:0/3095752246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a65260 0x559955a5f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.875+0000 7f8a0b7fe700 1 -- 172.21.15.70:0/3095752246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a648a0 msgr2=0x559955a805a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:17.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.875+0000 7f8a0b7fe700 1 --2- 172.21.15.70:0/3095752246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a648a0 0x559955a805a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.875+0000 7f8a0b7fe700 1 -- 172.21.15.70:0/3095752246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a65260 msgr2=0x559955a5f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:17.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.875+0000 7f8a0b7fe700 1 --2- 172.21.15.70:0/3095752246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a65260 0x559955a5f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.875+0000 7f8a0b7fe700 1 -- 172.21.15.70:0/3095752246 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559955a69f40 con 0x559955a6a320 2024-06-04T04:59:17.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.875+0000 7f8a0affd700 1 --2- 172.21.15.70:0/3095752246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a648a0 0x559955a805a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:17.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.875+0000 7f8a0b7fe700 1 --2- 172.21.15.70:0/3095752246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a6a320 0x559955a85e70 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8a04003fa0 tx=0x7f8a04003a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607f3c1280868013 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:17.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.875+0000 7f89f17fa700 1 -- 172.21.15.70:0/3095752246 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a04007580 con 0x559955a6a320 2024-06-04T04:59:17.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.875+0000 7f89f17fa700 1 -- 172.21.15.70:0/3095752246 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a04007d60 con 0x559955a6a320 2024-06-04T04:59:17.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.875+0000 7f89f17fa700 1 -- 172.21.15.70:0/3095752246 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a040057a0 con 0x559955a6a320 2024-06-04T04:59:17.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.876+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/3095752246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a6a320 msgr2=0x559955a85e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:17.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.876+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/3095752246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a6a320 0x559955a85e70 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8a04003fa0 tx=0x7f8a04003a20 comp rx=0 tx=0).stop 2024-06-04T04:59:17.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.876+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/3095752246 shutdown_connections 2024-06-04T04:59:17.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.876+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/3095752246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a6a320 0x559955a85e70 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.876+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/3095752246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a648a0 0x559955a805a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.876+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/3095752246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a65260 0x559955a5f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.876+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/3095752246 >> 172.21.15.70:0/3095752246 conn(0x559955990240 msgr2=0x559955a6c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:17.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.876+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/3095752246 shutdown_connections 2024-06-04T04:59:17.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.876+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/3095752246 wait complete. 2024-06-04T04:59:17.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.876+0000 7f8a1c2b6b80 1 -- start start 2024-06-04T04:59:17.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a1c2b6b80 1 Processor -- start 2024-06-04T04:59:17.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a1c2b6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a648a0 0x559955a76930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a1c2b6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a76e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a1c2b6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a6a320 0x559955a84ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a1c2b6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c1b12e0 con 0x559955a65260 2024-06-04T04:59:17.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a1c2b6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c1b0c10 con 0x559955a648a0 2024-06-04T04:59:17.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a1c2b6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c1ade10 con 0x559955a6a320 2024-06-04T04:59:17.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a1c2b6b80 10 client.0 _ll_register_callbacks cb 0x55995594d950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:17.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a0a7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a76e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a0a7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a76e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40044/0 (socket says 172.21.15.70:40044) 2024-06-04T04:59:17.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a0a7fc700 1 -- 172.21.15.70:0/2116539512 learned_addr learned my addr 172.21.15.70:0/2116539512 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:17.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a0affd700 1 --2- 172.21.15.70:0/2116539512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a648a0 0x559955a76930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a0b7fe700 1 --2- 172.21.15.70:0/2116539512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a6a320 0x559955a84ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a0a7fc700 1 -- 172.21.15.70:0/2116539512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a6a320 msgr2=0x559955a84ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:17.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a0a7fc700 1 --2- 172.21.15.70:0/2116539512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a6a320 0x559955a84ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a0a7fc700 1 -- 172.21.15.70:0/2116539512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a648a0 msgr2=0x559955a76930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:17.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a0a7fc700 1 --2- 172.21.15.70:0/2116539512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a648a0 0x559955a76930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a0a7fc700 1 -- 172.21.15.70:0/2116539512 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559955a85410 con 0x559955a65260 2024-06-04T04:59:17.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.877+0000 7f8a0affd700 1 --2- 172.21.15.70:0/2116539512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a648a0 0x559955a76930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:17.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.878+0000 7f8a0a7fc700 1 --2- 172.21.15.70:0/2116539512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a76e70 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8a00005950 tx=0x7f8a0000ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e054baf434d82e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:17.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.878+0000 7f89eaffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:59:17.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.878+0000 7f89eaffd700 1 -- 172.21.15.70:0/2116539512 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a00036040 con 0x559955a65260 2024-06-04T04:59:17.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.878+0000 7f89eaffd700 1 -- 172.21.15.70:0/2116539512 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f8a00030680 con 0x559955a65260 2024-06-04T04:59:17.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.878+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/2116539512 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a04007d60 con 0x559955a65260 2024-06-04T04:59:17.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.878+0000 7f89eaffd700 1 -- 172.21.15.70:0/2116539512 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8a00048050 con 0x559955a65260 2024-06-04T04:59:17.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.878+0000 7f8a1c2b6b80 10 client.16080 fetch_fsmap learned FSMap version 29 2024-06-04T04:59:17.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.878+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/2116539512 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559955ae85b0 con 0x559955a65260 2024-06-04T04:59:17.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.878+0000 7f89eaffd700 1 -- 172.21.15.70:0/2116539512 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a04007d60 con 0x559955a65260 2024-06-04T04:59:17.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.878+0000 7f89eaffd700 1 -- 172.21.15.70:0/2116539512 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f8a00039070 con 0x559955a65260 2024-06-04T04:59:17.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.878+0000 7f8a1c2b6b80 10 client.16080 fetch_fsmap finished waiting for FSMap version 29 2024-06-04T04:59:17.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.878+0000 7f8a1c2b6b80 10 client.16080 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T04:59:17.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.878+0000 7f8a1c2b6b80 20 client.16080 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.878+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/2116539512 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x559955a7e1b0 0x559955aead20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.878+0000 7f8a1c2b6b80 4 client.16080 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T04:59:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.879+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/2116539512 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5599559be4a0 con 0x559955a7e1b0 2024-06-04T04:59:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.879+0000 7f8a0affd700 1 --2- 172.21.15.70:0/2116539512 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x559955a7e1b0 0x559955aead20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.879+0000 7f8a0affd700 1 --2- 172.21.15.70:0/2116539512 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x559955a7e1b0 0x559955aead20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=41c43b7a6d2bf69 server_cookie=200d722d2550ac2f in_seq=0 out_seq=0 2024-06-04T04:59:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.879+0000 7f89eaffd700 10 client.16080 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T04:59:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.879+0000 7f89eaffd700 1 -- 172.21.15.70:0/2116539512 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5599559be4a0 con 0x559955a7e1b0 2024-06-04T04:59:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.879+0000 7f89eaffd700 10 client.16080 handle_command_reply: tid=0 2024-06-04T04:59:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.879+0000 7f8a1c2b6b80 1 client.16080 shutdown 2024-06-04T04:59:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.880+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/2116539512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 msgr2=0x559955a76e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.880+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/2116539512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a76e70 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8a00005950 tx=0x7f8a0000ba40 comp rx=0 tx=0).stop 2024-06-04T04:59:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.880+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/2116539512 shutdown_connections 2024-06-04T04:59:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.880+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/2116539512 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x559955a7e1b0 0x559955aead20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.880+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/2116539512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559955a6a320 0x559955a84ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.880+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/2116539512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559955a648a0 0x559955a76930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.880+0000 7f89eaffd700 0 client.16080 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T04:59:17.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.880+0000 7f8a1c2b6b80 1 --2- 172.21.15.70:0/2116539512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559955a65260 0x559955a76e70 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.880+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/2116539512 >> 172.21.15.70:0/2116539512 conn(0x559955990240 msgr2=0x559955a690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:17.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.880+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/2116539512 shutdown_connections 2024-06-04T04:59:17.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.880+0000 7f8a1c2b6b80 1 -- 172.21.15.70:0/2116539512 wait complete. 2024-06-04T04:59:17.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.880+0000 7f8a1c2b6b80 20 client.16080 trim_cache size 0 max 16384 2024-06-04T04:59:17.888 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T04:59:17.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.880+0000 7f8a11a21700 1 -- 172.21.15.70:0/3156647630 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f89f40b29f0 msgr2=0x7f89f40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:17.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.880+0000 7f8a11a21700 1 --2- 172.21.15.70:0/3156647630 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f89f40b29f0 0x7f89f40b4eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c079ad0 tx=0x7f8a0000b410 comp rx=0 tx=0).stop 2024-06-04T04:59:17.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.880+0000 7f8a11a21700 1 -- 172.21.15.70:0/3156647630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a0c102440 msgr2=0x7f8a0c104080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:17.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.880+0000 7f8a11a21700 1 --2- 172.21.15.70:0/3156647630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a0c102440 0x7f8a0c104080 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f89fc000c00 tx=0x7f89fc0047b0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.881+0000 7f8a11a21700 1 -- 172.21.15.70:0/3156647630 shutdown_connections 2024-06-04T04:59:17.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.881+0000 7f8a11a21700 1 --2- 172.21.15.70:0/3156647630 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f89f40b29f0 0x7f89f40b4eb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.881+0000 7f8a11a21700 1 --2- 172.21.15.70:0/3156647630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a0c108520 0x7f8a0c104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.881+0000 7f8a11a21700 1 --2- 172.21.15.70:0/3156647630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a0c102440 0x7f8a0c104080 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.881+0000 7f8a11a21700 1 --2- 172.21.15.70:0/3156647630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a0c102e00 0x7f8a0c1045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:17.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.881+0000 7f8a11a21700 1 -- 172.21.15.70:0/3156647630 >> 172.21.15.70:0/3156647630 conn(0x7f8a0c0793f0 msgr2=0x7f8a0c06b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:17.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.881+0000 7f8a11a21700 1 -- 172.21.15.70:0/3156647630 shutdown_connections 2024-06-04T04:59:17.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:17.881+0000 7f8a11a21700 1 -- 172.21.15.70:0/3156647630 wait complete. 2024-06-04T04:59:18.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 7794 5.40 MB/sec warmup 11 sec latency 11.772 ms 2024-06-04T04:59:18.763 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T04:59:19.032 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:59:19.430 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:19 smithi070 ceph-mon[32631]: pgmap v67: 97 pgs: 97 active+clean; 25 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 2.8 MiB/s wr, 93 op/s 2024-06-04T04:59:19.431 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:19 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T04:59:19.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:19 smithi112 ceph-mon[36453]: pgmap v67: 97 pgs: 97 active+clean; 25 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 2.8 MiB/s wr, 93 op/s 2024-06-04T04:59:19.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:19 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T04:59:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:19 smithi098 ceph-mon[35569]: pgmap v67: 97 pgs: 97 active+clean; 25 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 2.8 MiB/s wr, 93 op/s 2024-06-04T04:59:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:19 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T04:59:19.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 8530 5.35 MB/sec warmup 12 sec latency 14.990 ms 2024-06-04T04:59:20.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 9165 4.99 MB/sec warmup 13 sec latency 45.488 ms 2024-06-04T04:59:20.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.886+0000 7f1cbaf3e700 1 -- 172.21.15.70:0/840241073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cb406bcb0 msgr2=0x7f1cb406c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:20.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.886+0000 7f1cbaf3e700 1 --2- 172.21.15.70:0/840241073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cb406bcb0 0x7f1cb406c130 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4009990 tx=0x7f1ca4009ca0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.886+0000 7f1cbaf3e700 1 -- 172.21.15.70:0/840241073 shutdown_connections 2024-06-04T04:59:20.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.886+0000 7f1cbaf3e700 1 --2- 172.21.15.70:0/840241073 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1cb406c670 0x7f1cb4110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.886+0000 7f1cbaf3e700 1 --2- 172.21.15.70:0/840241073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cb4109620 0x7f1cb4109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.886+0000 7f1cbaf3e700 1 --2- 172.21.15.70:0/840241073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cb406bcb0 0x7f1cb406c130 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.886+0000 7f1cbaf3e700 1 -- 172.21.15.70:0/840241073 >> 172.21.15.70:0/840241073 conn(0x7f1cb4078490 msgr2=0x7f1cb40788a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:20.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.886+0000 7f1cbaf3e700 1 -- 172.21.15.70:0/840241073 shutdown_connections 2024-06-04T04:59:20.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.886+0000 7f1cbaf3e700 1 -- 172.21.15.70:0/840241073 wait complete. 2024-06-04T04:59:20.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cbaf3e700 1 Processor -- start 2024-06-04T04:59:20.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cbaf3e700 1 -- start start 2024-06-04T04:59:20.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cbaf3e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1cb406bcb0 0x7f1cb419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cbaf3e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cb406c670 0x7f1cb419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cbaf3e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cb4109620 0x7f1cb4199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cbaf3e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb4113640 con 0x7f1cb406c670 2024-06-04T04:59:20.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cbaf3e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41134c0 con 0x7f1cb4109620 2024-06-04T04:59:20.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cbaf3e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41137c0 con 0x7f1cb406bcb0 2024-06-04T04:59:20.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cb3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cb406c670 0x7f1cb419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cb3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cb406c670 0x7f1cb419fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40056/0 (socket says 172.21.15.70:40056) 2024-06-04T04:59:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cb3fff700 1 -- 172.21.15.70:0/2377674412 learned_addr learned my addr 172.21.15.70:0/2377674412 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cb8cda700 1 --2- 172.21.15.70:0/2377674412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1cb406bcb0 0x7f1cb419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cb94db700 1 --2- 172.21.15.70:0/2377674412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cb4109620 0x7f1cb4199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cb3fff700 1 -- 172.21.15.70:0/2377674412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1cb406bcb0 msgr2=0x7f1cb419f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cb3fff700 1 --2- 172.21.15.70:0/2377674412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1cb406bcb0 0x7f1cb419f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cb3fff700 1 -- 172.21.15.70:0/2377674412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cb4109620 msgr2=0x7f1cb4199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cb3fff700 1 --2- 172.21.15.70:0/2377674412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cb4109620 0x7f1cb4199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cb3fff700 1 -- 172.21.15.70:0/2377674412 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb419a0c0 con 0x7f1cb406c670 2024-06-04T04:59:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.887+0000 7f1cb8cda700 1 --2- 172.21.15.70:0/2377674412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1cb406bcb0 0x7f1cb419f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.888+0000 7f1cb3fff700 1 --2- 172.21.15.70:0/2377674412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cb406c670 0x7f1cb419fce0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400b650 tx=0x7f1ca400bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.888+0000 7f1cb1ffb700 1 -- 172.21.15.70:0/2377674412 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4027070 con 0x7f1cb406c670 2024-06-04T04:59:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.888+0000 7f1cb1ffb700 1 -- 172.21.15.70:0/2377674412 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1ca4022470 con 0x7f1cb406c670 2024-06-04T04:59:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.888+0000 7f1cbaf3e700 1 -- 172.21.15.70:0/2377674412 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cb419a350 con 0x7f1cb406c670 2024-06-04T04:59:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.888+0000 7f1cb1ffb700 1 -- 172.21.15.70:0/2377674412 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4017650 con 0x7f1cb406c670 2024-06-04T04:59:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.888+0000 7f1cbaf3e700 1 -- 172.21.15.70:0/2377674412 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cb41a6670 con 0x7f1cb406c670 2024-06-04T04:59:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.891+0000 7f1cb1ffb700 1 -- 172.21.15.70:0/2377674412 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f1ca402a070 con 0x7f1cb406c670 2024-06-04T04:59:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.892+0000 7f1cb1ffb700 1 --2- 172.21.15.70:0/2377674412 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1c9c0b2a70 0x7f1c9c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.892+0000 7f1cb8cda700 1 --2- 172.21.15.70:0/2377674412 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1c9c0b2a70 0x7f1c9c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.892+0000 7f1cb1ffb700 1 -- 172.21.15.70:0/2377674412 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f1ca40e3550 con 0x7f1cb406c670 2024-06-04T04:59:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.892+0000 7f1cc57d3b80 1 Processor -- start 2024-06-04T04:59:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.892+0000 7f1cb8cda700 1 --2- 172.21.15.70:0/2377674412 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1c9c0b2a70 0x7f1c9c0b4f30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1cb419aaa0 tx=0x7f1ca80073d0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.892+0000 7f1cc57d3b80 1 -- start start 2024-06-04T04:59:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.893+0000 7f1cc57d3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 0x5565310c9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.893+0000 7f1cc57d3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 0x5565310c3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.893+0000 7f1cc57d3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c4260 0x5565310d0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.893+0000 7f1cc57d3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556531021d70 con 0x5565310c38a0 2024-06-04T04:59:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.893+0000 7f1cc57d3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556531011910 con 0x5565310c4260 2024-06-04T04:59:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.893+0000 7f1cc57d3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556531015950 con 0x5565310c9320 2024-06-04T04:59:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.893+0000 7f1cb8cda700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 0x5565310c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.893+0000 7f1cb3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 0x5565310c3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.893+0000 7f1cb8cda700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 0x5565310c9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:45250/0 (socket says 172.21.15.70:45250) 2024-06-04T04:59:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.893+0000 7f1cb8cda700 1 -- 172.21.15.70:0/4206832389 learned_addr learned my addr 172.21.15.70:0/4206832389 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.893+0000 7f1cb94db700 1 --2- 172.21.15.70:0/4206832389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c4260 0x5565310d0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.893+0000 7f1cb8cda700 1 -- 172.21.15.70:0/4206832389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c4260 msgr2=0x5565310d0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.893+0000 7f1cb8cda700 1 --2- 172.21.15.70:0/4206832389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c4260 0x5565310d0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.893+0000 7f1cb8cda700 1 -- 172.21.15.70:0/4206832389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 msgr2=0x5565310c3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.893+0000 7f1cb8cda700 1 --2- 172.21.15.70:0/4206832389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 0x5565310c3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.893+0000 7f1cb8cda700 1 -- 172.21.15.70:0/4206832389 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556530fbafb0 con 0x5565310c9320 2024-06-04T04:59:20.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.893+0000 7f1cb8cda700 1 --2- 172.21.15.70:0/4206832389 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 0x5565310c9700 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80115d0 tx=0x7f1ca8013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9953cae2b30d3770 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:20.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.894+0000 7f1c9affd700 1 -- 172.21.15.70:0/4206832389 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca801e620 con 0x5565310c9320 2024-06-04T04:59:20.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.894+0000 7f1c9affd700 1 -- 172.21.15.70:0/4206832389 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1ca801ec50 con 0x5565310c9320 2024-06-04T04:59:20.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.894+0000 7f1c9affd700 1 -- 172.21.15.70:0/4206832389 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca801cb70 con 0x5565310c9320 2024-06-04T04:59:20.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.895+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/4206832389 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 msgr2=0x5565310c9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:20.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.895+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/4206832389 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 0x5565310c9700 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80115d0 tx=0x7f1ca8013000 comp rx=0 tx=0).stop 2024-06-04T04:59:20.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.895+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/4206832389 shutdown_connections 2024-06-04T04:59:20.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.895+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/4206832389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c4260 0x5565310d0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.895+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/4206832389 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 0x5565310c9700 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.895+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/4206832389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 0x5565310c3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.895+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/4206832389 >> 172.21.15.70:0/4206832389 conn(0x556530fef240 msgr2=0x5565310be3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:20.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.895+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/4206832389 shutdown_connections 2024-06-04T04:59:20.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.895+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/4206832389 wait complete. 2024-06-04T04:59:20.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.895+0000 7f1cc57d3b80 1 -- start start 2024-06-04T04:59:20.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.895+0000 7f1cc57d3b80 1 Processor -- start 2024-06-04T04:59:20.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cc57d3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 0x5565310cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cc57d3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c4260 0x5565310ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cc57d3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 0x5565310cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cc57d3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556530f17fa0 con 0x5565310c38a0 2024-06-04T04:59:20.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cc57d3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556530f79da0 con 0x5565310c4260 2024-06-04T04:59:20.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cc57d3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5565310d06e0 con 0x5565310c9320 2024-06-04T04:59:20.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cb8cda700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 0x5565310cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cc57d3b80 10 client.0 _ll_register_callbacks cb 0x556530fac950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:20.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cb8cda700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 0x5565310cfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40078/0 (socket says 172.21.15.70:40078) 2024-06-04T04:59:20.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cb8cda700 1 -- 172.21.15.70:0/1430232228 learned_addr learned my addr 172.21.15.70:0/1430232228 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:20.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cb94db700 1 --2- 172.21.15.70:0/1430232228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 0x5565310cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cb3fff700 1 --2- 172.21.15.70:0/1430232228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c4260 0x5565310ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cb94db700 1 -- 172.21.15.70:0/1430232228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c4260 msgr2=0x5565310ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:20.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cb94db700 1 --2- 172.21.15.70:0/1430232228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c4260 0x5565310ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cb94db700 1 -- 172.21.15.70:0/1430232228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 msgr2=0x5565310cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:20.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cb94db700 1 --2- 172.21.15.70:0/1430232228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 0x5565310cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cb94db700 1 -- 172.21.15.70:0/1430232228 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556530f618d0 con 0x5565310c9320 2024-06-04T04:59:20.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cb3fff700 1 --2- 172.21.15.70:0/1430232228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c4260 0x5565310ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:20.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.896+0000 7f1cb8cda700 1 --2- 172.21.15.70:0/1430232228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 0x5565310cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:20.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.897+0000 7f1cb94db700 1 --2- 172.21.15.70:0/1430232228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 0x5565310cd4e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1cac011e90 tx=0x7f1cac014600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fb4da6d760df0cc server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:20.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.897+0000 7f1c93fff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:59:20.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.897+0000 7f1c93fff700 1 -- 172.21.15.70:0/1430232228 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac004db0 con 0x5565310c9320 2024-06-04T04:59:20.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.897+0000 7f1c93fff700 1 -- 172.21.15.70:0/1430232228 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f1cac014a80 con 0x5565310c9320 2024-06-04T04:59:20.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.897+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/1430232228 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ca801ec50 con 0x5565310c9320 2024-06-04T04:59:20.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.897+0000 7f1c93fff700 1 -- 172.21.15.70:0/1430232228 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1cac0093a0 con 0x5565310c9320 2024-06-04T04:59:20.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.897+0000 7f1cc57d3b80 10 client.25561 fetch_fsmap learned FSMap version 30 2024-06-04T04:59:20.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.897+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/1430232228 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556530fcb760 con 0x5565310c9320 2024-06-04T04:59:20.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.897+0000 7f1c93fff700 1 -- 172.21.15.70:0/1430232228 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1cac014c80 con 0x5565310c9320 2024-06-04T04:59:20.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.897+0000 7f1c93fff700 1 -- 172.21.15.70:0/1430232228 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f1cac004390 con 0x5565310c9320 2024-06-04T04:59:20.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.897+0000 7f1cc57d3b80 10 client.25561 fetch_fsmap finished waiting for FSMap version 30 2024-06-04T04:59:20.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.897+0000 7f1cc57d3b80 10 client.25561 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T04:59:20.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.897+0000 7f1cc57d3b80 20 client.25561 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:20.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.898+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/1430232228 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5565310e1590 0x5565310e39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.898+0000 7f1cc57d3b80 4 client.25561 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:59:20.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.898+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/1430232228 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1ca801ec50 con 0x5565310e1590 2024-06-04T04:59:20.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.898+0000 7f1cb8cda700 1 --2- 172.21.15.70:0/1430232228 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5565310e1590 0x5565310e39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.898+0000 7f1c93fff700 10 client.25561 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T04:59:20.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.898+0000 7f1cb8cda700 1 --2- 172.21.15.70:0/1430232228 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5565310e1590 0x5565310e39c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bd385771070ae8f2 server_cookie=5439bfbb65cfbbb1 in_seq=0 out_seq=0 2024-06-04T04:59:20.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.899+0000 7f1c93fff700 1 -- 172.21.15.70:0/1430232228 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f1ca801ec50 con 0x5565310e1590 2024-06-04T04:59:20.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.899+0000 7f1c93fff700 10 client.25561 handle_command_reply: tid=0 2024-06-04T04:59:20.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.899+0000 7f1cc57d3b80 1 client.25561 shutdown 2024-06-04T04:59:20.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.899+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/1430232228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 msgr2=0x5565310cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:20.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.899+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/1430232228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 0x5565310cd4e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1cac011e90 tx=0x7f1cac014600 comp rx=0 tx=0).stop 2024-06-04T04:59:20.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.899+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/1430232228 shutdown_connections 2024-06-04T04:59:20.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.899+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/1430232228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 0x5565310cd4e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.899+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/1430232228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 0x5565310cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.899+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/1430232228 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5565310e1590 0x5565310e39c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.899+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/1430232228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c4260 0x5565310ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.899+0000 7f1c93fff700 0 client.25561 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T04:59:20.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.899+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/1430232228 >> 172.21.15.70:0/1430232228 conn(0x556530fef240 msgr2=0x5565310bdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:20.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.900+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/1430232228 shutdown_connections 2024-06-04T04:59:20.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.900+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/1430232228 wait complete. 2024-06-04T04:59:20.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.900+0000 7f1cc57d3b80 20 client.25561 trim_cache size 0 max 16384 2024-06-04T04:59:20.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.917+0000 7f1cc57d3b80 1 Processor -- start 2024-06-04T04:59:20.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cc57d3b80 1 -- start start 2024-06-04T04:59:20.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cc57d3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 0x5565310c7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cc57d3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c4260 0x5565310be270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cc57d3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 0x5565310e9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cc57d3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41abb00 con 0x5565310c38a0 2024-06-04T04:59:20.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cc57d3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41ad670 con 0x5565310c4260 2024-06-04T04:59:20.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cc57d3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41ac1d0 con 0x5565310c9320 2024-06-04T04:59:20.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cb8cda700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 0x5565310c7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cb8cda700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 0x5565310c7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40094/0 (socket says 172.21.15.70:40094) 2024-06-04T04:59:20.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cb8cda700 1 -- 172.21.15.70:0/2496284326 learned_addr learned my addr 172.21.15.70:0/2496284326 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:20.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cb94db700 1 --2- 172.21.15.70:0/2496284326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 0x5565310e9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cb3fff700 1 --2- 172.21.15.70:0/2496284326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c4260 0x5565310be270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cb8cda700 1 -- 172.21.15.70:0/2496284326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 msgr2=0x5565310e9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:20.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cb8cda700 1 --2- 172.21.15.70:0/2496284326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 0x5565310e9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cb8cda700 1 -- 172.21.15.70:0/2496284326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c4260 msgr2=0x5565310be270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:20.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cb8cda700 1 --2- 172.21.15.70:0/2496284326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c4260 0x5565310be270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cb8cda700 1 -- 172.21.15.70:0/2496284326 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565310cb510 con 0x5565310c38a0 2024-06-04T04:59:20.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.918+0000 7f1cb8cda700 1 --2- 172.21.15.70:0/2496284326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 0x5565310c7570 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8007fd0 tx=0x7f1ca8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50aa6a9705cf6dfd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:20.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.919+0000 7f1c9a7fc700 1 -- 172.21.15.70:0/2496284326 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca8005040 con 0x5565310c38a0 2024-06-04T04:59:20.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.919+0000 7f1c9a7fc700 1 -- 172.21.15.70:0/2496284326 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1ca8005670 con 0x5565310c38a0 2024-06-04T04:59:20.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.919+0000 7f1c9a7fc700 1 -- 172.21.15.70:0/2496284326 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca8004710 con 0x5565310c38a0 2024-06-04T04:59:20.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.919+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/2496284326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 msgr2=0x5565310c7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:20.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.919+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/2496284326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 0x5565310c7570 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8007fd0 tx=0x7f1ca8016000 comp rx=0 tx=0).stop 2024-06-04T04:59:20.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.919+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/2496284326 shutdown_connections 2024-06-04T04:59:20.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.919+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/2496284326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c9320 0x5565310e9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.919+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/2496284326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c38a0 0x5565310c7570 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.919+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/2496284326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c4260 0x5565310be270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.919+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/2496284326 >> 172.21.15.70:0/2496284326 conn(0x556530fef240 msgr2=0x5565310c86f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:20.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.919+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/2496284326 shutdown_connections 2024-06-04T04:59:20.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.920+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/2496284326 wait complete. 2024-06-04T04:59:20.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.920+0000 7f1cc57d3b80 1 -- start start 2024-06-04T04:59:20.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.920+0000 7f1cc57d3b80 1 Processor -- start 2024-06-04T04:59:20.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.920+0000 7f1cc57d3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c38a0 0x5565310e8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.920+0000 7f1cc57d3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c4260 0x5565310e9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.920+0000 7f1cc57d3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c9320 0x5565310e9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.920+0000 7f1cc57d3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41acf70 con 0x5565310c9320 2024-06-04T04:59:20.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.920+0000 7f1cc57d3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41ac8a0 con 0x5565310c38a0 2024-06-04T04:59:20.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.920+0000 7f1cc57d3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41a9aa0 con 0x5565310c4260 2024-06-04T04:59:20.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.920+0000 7f1cc57d3b80 10 client.0 _ll_register_callbacks cb 0x556530fac950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:20.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.921+0000 7f1cb94db700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c9320 0x5565310e9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.921+0000 7f1cb94db700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c9320 0x5565310e9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40108/0 (socket says 172.21.15.70:40108) 2024-06-04T04:59:20.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.921+0000 7f1cb3fff700 1 --2- 172.21.15.70:0/3790549123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c4260 0x5565310e9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.921+0000 7f1cb94db700 1 -- 172.21.15.70:0/3790549123 learned_addr learned my addr 172.21.15.70:0/3790549123 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:20.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.921+0000 7f1cb8cda700 1 --2- 172.21.15.70:0/3790549123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c38a0 0x5565310e8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.921+0000 7f1cb94db700 1 -- 172.21.15.70:0/3790549123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c4260 msgr2=0x5565310e9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:20.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.921+0000 7f1cb94db700 1 --2- 172.21.15.70:0/3790549123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c4260 0x5565310e9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.921+0000 7f1cb94db700 1 -- 172.21.15.70:0/3790549123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c38a0 msgr2=0x5565310e8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:20.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.921+0000 7f1cb94db700 1 --2- 172.21.15.70:0/3790549123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c38a0 0x5565310e8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.921+0000 7f1cb94db700 1 -- 172.21.15.70:0/3790549123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55653114b260 con 0x5565310c9320 2024-06-04T04:59:20.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.921+0000 7f1cb94db700 1 --2- 172.21.15.70:0/3790549123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c9320 0x5565310e9940 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f1cac011e60 tx=0x7f1cac014b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3980c84105b3e412 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:20.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.921+0000 7f1c93fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:59:20.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.921+0000 7f1c93fff700 1 -- 172.21.15.70:0/3790549123 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac036060 con 0x5565310c9320 2024-06-04T04:59:20.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.921+0000 7f1c93fff700 1 -- 172.21.15.70:0/3790549123 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f1cac007700 con 0x5565310c9320 2024-06-04T04:59:20.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.921+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/3790549123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ca8005670 con 0x5565310c9320 2024-06-04T04:59:20.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.922+0000 7f1c93fff700 1 -- 172.21.15.70:0/3790549123 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1cac02d070 con 0x5565310c9320 2024-06-04T04:59:20.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.922+0000 7f1cc57d3b80 10 client.16098 fetch_fsmap learned FSMap version 30 2024-06-04T04:59:20.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.922+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/3790549123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55653114c560 con 0x5565310c9320 2024-06-04T04:59:20.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.922+0000 7f1c93fff700 1 -- 172.21.15.70:0/3790549123 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1ca8005670 con 0x5565310c9320 2024-06-04T04:59:20.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.922+0000 7f1c93fff700 1 -- 172.21.15.70:0/3790549123 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f1cac007ac0 con 0x5565310c9320 2024-06-04T04:59:20.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.922+0000 7f1cc57d3b80 10 client.16098 fetch_fsmap finished waiting for FSMap version 30 2024-06-04T04:59:20.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.922+0000 7f1cc57d3b80 10 client.16098 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T04:59:20.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.922+0000 7f1cc57d3b80 20 client.16098 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:20.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.922+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/3790549123 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55653114cbe0 0x55653114f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:20.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.922+0000 7f1cc57d3b80 4 client.16098 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T04:59:20.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.922+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/3790549123 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556531023cc0 con 0x55653114cbe0 2024-06-04T04:59:20.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.922+0000 7f1cb8cda700 1 --2- 172.21.15.70:0/3790549123 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55653114cbe0 0x55653114f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:20.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.923+0000 7f1c93fff700 10 client.16098 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T04:59:20.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.923+0000 7f1cb8cda700 1 --2- 172.21.15.70:0/3790549123 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55653114cbe0 0x55653114f000 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3e74fb9a6a918d86 server_cookie=11e185bc12a03056 in_seq=0 out_seq=0 2024-06-04T04:59:20.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.923+0000 7f1c93fff700 1 -- 172.21.15.70:0/3790549123 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556531023cc0 con 0x55653114cbe0 2024-06-04T04:59:20.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.923+0000 7f1c93fff700 10 client.16098 handle_command_reply: tid=0 2024-06-04T04:59:20.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.923+0000 7f1cc57d3b80 1 client.16098 shutdown 2024-06-04T04:59:20.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.923+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/3790549123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c9320 msgr2=0x5565310e9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:20.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.923+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/3790549123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c9320 0x5565310e9940 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f1cac011e60 tx=0x7f1cac014b80 comp rx=0 tx=0).stop 2024-06-04T04:59:20.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.923+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/3790549123 shutdown_connections 2024-06-04T04:59:20.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.923+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/3790549123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565310c9320 0x5565310e9940 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.923+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/3790549123 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55653114cbe0 0x55653114f000 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.923+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/3790549123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565310c38a0 0x5565310e8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.923+0000 7f1cc57d3b80 1 --2- 172.21.15.70:0/3790549123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565310c4260 0x5565310e9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.923+0000 7f1c93fff700 0 client.16098 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T04:59:20.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.923+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/3790549123 >> 172.21.15.70:0/3790549123 conn(0x556530fef240 msgr2=0x5565310ca010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:20.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.923+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/3790549123 shutdown_connections 2024-06-04T04:59:20.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.923+0000 7f1cc57d3b80 1 -- 172.21.15.70:0/3790549123 wait complete. 2024-06-04T04:59:20.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.923+0000 7f1cc57d3b80 20 client.16098 trim_cache size 0 max 16384 2024-06-04T04:59:20.931 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T04:59:20.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.924+0000 7f1cbaf3e700 1 -- 172.21.15.70:0/2377674412 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1c9c0b2a70 msgr2=0x7f1c9c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:20.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.924+0000 7f1cbaf3e700 1 --2- 172.21.15.70:0/2377674412 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1c9c0b2a70 0x7f1c9c0b4f30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1cb419aaa0 tx=0x7f1ca80073d0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.924+0000 7f1cbaf3e700 1 -- 172.21.15.70:0/2377674412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cb406c670 msgr2=0x7f1cb419fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:20.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.924+0000 7f1cbaf3e700 1 --2- 172.21.15.70:0/2377674412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cb406c670 0x7f1cb419fce0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400b650 tx=0x7f1ca400bfc0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.924+0000 7f1cbaf3e700 1 -- 172.21.15.70:0/2377674412 shutdown_connections 2024-06-04T04:59:20.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.924+0000 7f1cbaf3e700 1 --2- 172.21.15.70:0/2377674412 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1c9c0b2a70 0x7f1c9c0b4f30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.924+0000 7f1cbaf3e700 1 --2- 172.21.15.70:0/2377674412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cb4109620 0x7f1cb4199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.924+0000 7f1cbaf3e700 1 --2- 172.21.15.70:0/2377674412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1cb406bcb0 0x7f1cb419f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.924+0000 7f1cbaf3e700 1 --2- 172.21.15.70:0/2377674412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cb406c670 0x7f1cb419fce0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:20.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.924+0000 7f1cbaf3e700 1 -- 172.21.15.70:0/2377674412 >> 172.21.15.70:0/2377674412 conn(0x7f1cb4078490 msgr2=0x7f1cb41012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:20.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.924+0000 7f1cbaf3e700 1 -- 172.21.15.70:0/2377674412 shutdown_connections 2024-06-04T04:59:20.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:20.924+0000 7f1cbaf3e700 1 -- 172.21.15.70:0/2377674412 wait complete. 2024-06-04T04:59:21.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:21 smithi112 ceph-mon[36453]: pgmap v68: 97 pgs: 97 active+clean; 25 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 2.8 MiB/s wr, 93 op/s 2024-06-04T04:59:21.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:21 smithi098 ceph-mon[35569]: pgmap v68: 97 pgs: 97 active+clean; 25 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 2.8 MiB/s wr, 93 op/s 2024-06-04T04:59:21.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:21 smithi070 ceph-mon[32631]: pgmap v68: 97 pgs: 97 active+clean; 25 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 2.8 MiB/s wr, 93 op/s 2024-06-04T04:59:21.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 9854 5.01 MB/sec warmup 14 sec latency 15.614 ms 2024-06-04T04:59:21.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.755+0000 7fc26c158640 1 -- 172.21.15.70:0/2568291885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc26410d020 msgr2=0x7fc26410f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:21.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.755+0000 7fc26c158640 1 --2- 172.21.15.70:0/2568291885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc26410d020 0x7fc26410f4c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009ca0 tx=0x7fc25c02a9e0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.755+0000 7fc26c158640 1 -- 172.21.15.70:0/2568291885 shutdown_connections 2024-06-04T04:59:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.755+0000 7fc26c158640 1 --2- 172.21.15.70:0/2568291885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc264104600 0x7fc26410cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.755+0000 7fc26c158640 1 --2- 172.21.15.70:0/2568291885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc26410d020 0x7fc26410f4c0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.755+0000 7fc26c158640 1 --2- 172.21.15.70:0/2568291885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc264103ce0 0x7fc2641040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.755+0000 7fc26c158640 1 -- 172.21.15.70:0/2568291885 >> 172.21.15.70:0/2568291885 conn(0x7fc264078ad0 msgr2=0x7fc264078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.755+0000 7fc26c158640 1 -- 172.21.15.70:0/2568291885 shutdown_connections 2024-06-04T04:59:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.755+0000 7fc26c158640 1 -- 172.21.15.70:0/2568291885 wait complete. 2024-06-04T04:59:21.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.756+0000 7fc26c158640 1 Processor -- start 2024-06-04T04:59:21.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.756+0000 7fc26c158640 1 -- start start 2024-06-04T04:59:21.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.756+0000 7fc26c158640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc264103ce0 0x7fc2641990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.756+0000 7fc26c158640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc264104600 0x7fc2641995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.756+0000 7fc26c158640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc26410d020 0x7fc26419d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.756+0000 7fc26c158640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc264112b80 con 0x7fc26410d020 2024-06-04T04:59:21.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.756+0000 7fc26c158640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc264112a00 con 0x7fc264103ce0 2024-06-04T04:59:21.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.756+0000 7fc26c158640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc264112d00 con 0x7fc264104600 2024-06-04T04:59:21.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.756+0000 7fc26a6f5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc26410d020 0x7fc26419d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.756+0000 7fc2696f3640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc264104600 0x7fc2641995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.756+0000 7fc2696f3640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc264104600 0x7fc2641995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:45300/0 (socket says 172.21.15.70:45300) 2024-06-04T04:59:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.756+0000 7fc26a6f5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc26410d020 0x7fc26419d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40124/0 (socket says 172.21.15.70:40124) 2024-06-04T04:59:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.756+0000 7fc2696f3640 1 -- 172.21.15.70:0/3651282702 learned_addr learned my addr 172.21.15.70:0/3651282702 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.757+0000 7fc269ef4640 1 --2- 172.21.15.70:0/3651282702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc264103ce0 0x7fc2641990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.757+0000 7fc2696f3640 1 -- 172.21.15.70:0/3651282702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc264103ce0 msgr2=0x7fc2641990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.757+0000 7fc2696f3640 1 --2- 172.21.15.70:0/3651282702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc264103ce0 0x7fc2641990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.757+0000 7fc2696f3640 1 -- 172.21.15.70:0/3651282702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc26410d020 msgr2=0x7fc26419d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.757+0000 7fc2696f3640 1 --2- 172.21.15.70:0/3651282702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc26410d020 0x7fc26419d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.757+0000 7fc2696f3640 1 -- 172.21.15.70:0/3651282702 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc26419e090 con 0x7fc264104600 2024-06-04T04:59:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.757+0000 7fc26a6f5640 1 --2- 172.21.15.70:0/3651282702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc26410d020 0x7fc26419d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.757+0000 7fc2696f3640 1 --2- 172.21.15.70:0/3651282702 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc264104600 0x7fc2641995e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc25400e9f0 tx=0x7fc25400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.757+0000 7fc25affd640 1 -- 172.21.15.70:0/3651282702 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25400cd80 con 0x7fc264104600 2024-06-04T04:59:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.757+0000 7fc26c158640 1 -- 172.21.15.70:0/3651282702 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc26419e380 con 0x7fc264104600 2024-06-04T04:59:21.762 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.757+0000 7fc26c158640 1 -- 172.21.15.70:0/3651282702 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2641a5c10 con 0x7fc264104600 2024-06-04T04:59:21.762 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.757+0000 7fc25affd640 1 -- 172.21.15.70:0/3651282702 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc2540108c0 con 0x7fc264104600 2024-06-04T04:59:21.762 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.757+0000 7fc25affd640 1 -- 172.21.15.70:0/3651282702 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc254018840 con 0x7fc264104600 2024-06-04T04:59:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.760+0000 7fc26ee75740 1 Processor -- start 2024-06-04T04:59:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.760+0000 7fc26ee75740 1 -- start start 2024-06-04T04:59:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.760+0000 7fc25affd640 1 -- 172.21.15.70:0/3651282702 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fc2540189e0 con 0x7fc264104600 2024-06-04T04:59:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.760+0000 7fc26ee75740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2e9820 0x561f2c2e9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.760+0000 7fc26ee75740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2ea140 0x561f2c2f44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.760+0000 7fc26ee75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 0x561f2c2f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.760+0000 7fc26ee75740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561f2c25bc30 con 0x561f2c2ea140 2024-06-04T04:59:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.760+0000 7fc26ee75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561f2c24b7d0 con 0x561f2c2f4a00 2024-06-04T04:59:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.760+0000 7fc26ee75740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561f2c24f810 con 0x561f2c2e9820 2024-06-04T04:59:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.760+0000 7fc269ef4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2ea140 0x561f2c2f44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.760+0000 7fc269ef4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2ea140 0x561f2c2f44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40132/0 (socket says 172.21.15.70:40132) 2024-06-04T04:59:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.760+0000 7fc269ef4640 1 -- 172.21.15.70:0/3444455794 learned_addr learned my addr 172.21.15.70:0/3444455794 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.760+0000 7fc26a6f5640 1 --2- 172.21.15.70:0/3444455794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2e9820 0x561f2c2e9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.760+0000 7fc2696f3640 1 --2- 172.21.15.70:0/3444455794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 0x561f2c2f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.761+0000 7fc26a6f5640 1 -- 172.21.15.70:0/3444455794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 msgr2=0x561f2c2f6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.761+0000 7fc26a6f5640 1 --2- 172.21.15.70:0/3444455794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 0x561f2c2f6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.761+0000 7fc26a6f5640 1 -- 172.21.15.70:0/3444455794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2ea140 msgr2=0x561f2c2f44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.761+0000 7fc26a6f5640 1 --2- 172.21.15.70:0/3444455794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2ea140 0x561f2c2f44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.761+0000 7fc26a6f5640 1 -- 172.21.15.70:0/3444455794 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f2bf60210 con 0x561f2c2e9820 2024-06-04T04:59:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.761+0000 7fc26a6f5640 1 --2- 172.21.15.70:0/3444455794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2e9820 0x561f2c2e9c00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc25c005cc0 tx=0x7fc25c005c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af995ecabdd9c758 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.761+0000 7fc25affd640 1 --2- 172.21.15.70:0/3651282702 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc2500b2a10 0x7fc2500b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.761+0000 7fc24ffff640 1 -- 172.21.15.70:0/3444455794 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25c033050 con 0x561f2c2e9820 2024-06-04T04:59:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.761+0000 7fc24ffff640 1 -- 172.21.15.70:0/3444455794 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc25c047070 con 0x561f2c2e9820 2024-06-04T04:59:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.761+0000 7fc24ffff640 1 -- 172.21.15.70:0/3444455794 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25c02f8a0 con 0x561f2c2e9820 2024-06-04T04:59:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.761+0000 7fc269ef4640 1 --2- 172.21.15.70:0/3651282702 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc2500b2a10 0x7fc2500b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:21.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.761+0000 7fc25affd640 1 -- 172.21.15.70:0/3651282702 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc2540e2850 con 0x7fc264104600 2024-06-04T04:59:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.762+0000 7fc26ee75740 1 -- 172.21.15.70:0/3444455794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2e9820 msgr2=0x561f2c2e9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.762+0000 7fc269ef4640 1 --2- 172.21.15.70:0/3651282702 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc2500b2a10 0x7fc2500b4ed0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x561f2c2eadc0 tx=0x7fc260005f90 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.762+0000 7fc26ee75740 1 --2- 172.21.15.70:0/3444455794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2e9820 0x561f2c2e9c00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc25c005cc0 tx=0x7fc25c005c50 comp rx=0 tx=0).stop 2024-06-04T04:59:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.762+0000 7fc26ee75740 1 -- 172.21.15.70:0/3444455794 shutdown_connections 2024-06-04T04:59:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.762+0000 7fc26ee75740 1 --2- 172.21.15.70:0/3444455794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 0x561f2c2f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.762+0000 7fc26ee75740 1 --2- 172.21.15.70:0/3444455794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2ea140 0x561f2c2f44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.762+0000 7fc26ee75740 1 --2- 172.21.15.70:0/3444455794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2e9820 0x561f2c2e9c00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.762+0000 7fc26ee75740 1 -- 172.21.15.70:0/3444455794 >> 172.21.15.70:0/3444455794 conn(0x561f2c2e5d30 msgr2=0x561f2c2e6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.762+0000 7fc26ee75740 1 -- 172.21.15.70:0/3444455794 shutdown_connections 2024-06-04T04:59:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.762+0000 7fc26ee75740 1 -- 172.21.15.70:0/3444455794 wait complete. 2024-06-04T04:59:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.763+0000 7fc26ee75740 1 -- start start 2024-06-04T04:59:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.763+0000 7fc26ee75740 1 Processor -- start 2024-06-04T04:59:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.763+0000 7fc26ee75740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2e9820 0x561f2c2f1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.763+0000 7fc26ee75740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2ea140 0x561f2c2f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.763+0000 7fc26ee75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 0x561f2c2f23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.763+0000 7fc26ee75740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561f2bf31790 con 0x561f2c2e9820 2024-06-04T04:59:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.763+0000 7fc26ee75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561f2c207890 con 0x561f2c2f4a00 2024-06-04T04:59:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.763+0000 7fc26ee75740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561f2c2f75d0 con 0x561f2c2ea140 2024-06-04T04:59:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.763+0000 7fc26ee75740 10 client.0 _ll_register_callbacks cb 0x561f2bf5fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.763+0000 7fc269ef4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2e9820 0x561f2c2f1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.763+0000 7fc2696f3640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2ea140 0x561f2c2f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.763+0000 7fc269ef4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2e9820 0x561f2c2f1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40134/0 (socket says 172.21.15.70:40134) 2024-06-04T04:59:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.763+0000 7fc269ef4640 1 -- 172.21.15.70:0/3078298598 learned_addr learned my addr 172.21.15.70:0/3078298598 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.764+0000 7fc2696f3640 1 -- 172.21.15.70:0/3078298598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 msgr2=0x561f2c2f23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.764+0000 7fc2696f3640 1 --2- 172.21.15.70:0/3078298598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 0x561f2c2f23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.764+0000 7fc2696f3640 1 -- 172.21.15.70:0/3078298598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2e9820 msgr2=0x561f2c2f1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.764+0000 7fc2696f3640 1 --2- 172.21.15.70:0/3078298598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2e9820 0x561f2c2f1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.764+0000 7fc2696f3640 1 -- 172.21.15.70:0/3078298598 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f2c02c620 con 0x561f2c2ea140 2024-06-04T04:59:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.764+0000 7fc2696f3640 1 --2- 172.21.15.70:0/3078298598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2ea140 0x561f2c2f1ea0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc2540dffa0 tx=0x7fc25400cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbf7cdfc640d329c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.764+0000 7fc24d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:59:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.764+0000 7fc24d7fa640 1 -- 172.21.15.70:0/3078298598 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2540a9900 con 0x561f2c2ea140 2024-06-04T04:59:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.764+0000 7fc24d7fa640 1 -- 172.21.15.70:0/3078298598 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc2540a8fa0 con 0x561f2c2ea140 2024-06-04T04:59:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.764+0000 7fc26ee75740 1 -- 172.21.15.70:0/3078298598 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc25c047070 con 0x561f2c2ea140 2024-06-04T04:59:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.764+0000 7fc24d7fa640 1 -- 172.21.15.70:0/3078298598 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc2540aae60 con 0x561f2c2ea140 2024-06-04T04:59:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.764+0000 7fc26ee75740 10 client.25579 fetch_fsmap learned FSMap version 30 2024-06-04T04:59:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.764+0000 7fc26ee75740 1 -- 172.21.15.70:0/3078298598 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561f2c303380 con 0x561f2c2ea140 2024-06-04T04:59:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.765+0000 7fc24d7fa640 1 -- 172.21.15.70:0/3078298598 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc25c047070 con 0x561f2c2ea140 2024-06-04T04:59:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.765+0000 7fc24d7fa640 1 -- 172.21.15.70:0/3078298598 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fc2540a8800 con 0x561f2c2ea140 2024-06-04T04:59:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.765+0000 7fc26ee75740 10 client.25579 fetch_fsmap finished waiting for FSMap version 30 2024-06-04T04:59:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.765+0000 7fc26ee75740 10 client.25579 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:59:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.765+0000 7fc26ee75740 20 client.25579 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.765+0000 7fc26ee75740 1 --2- 172.21.15.70:0/3078298598 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561f2c3048c0 0x561f2c306d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.765+0000 7fc26ee75740 4 client.25579 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:59:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.765+0000 7fc26ee75740 1 -- 172.21.15.70:0/3078298598 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f2c25db80 con 0x561f2c3048c0 2024-06-04T04:59:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.765+0000 7fc269ef4640 1 --2- 172.21.15.70:0/3078298598 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561f2c3048c0 0x561f2c306d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.766+0000 7fc24d7fa640 10 client.25579 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.766+0000 7fc269ef4640 1 --2- 172.21.15.70:0/3078298598 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561f2c3048c0 0x561f2c306d80 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e32702de9fc4783 server_cookie=248001db8df484b0 in_seq=0 out_seq=0 2024-06-04T04:59:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.766+0000 7fc24d7fa640 1 -- 172.21.15.70:0/3078298598 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561f2c25db80 con 0x561f2c3048c0 2024-06-04T04:59:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.766+0000 7fc24d7fa640 10 client.25579 handle_command_reply: tid=0 2024-06-04T04:59:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.766+0000 7fc26ee75740 1 client.25579 shutdown 2024-06-04T04:59:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.767+0000 7fc26ee75740 1 -- 172.21.15.70:0/3078298598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2ea140 msgr2=0x561f2c2f1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.767+0000 7fc26ee75740 1 --2- 172.21.15.70:0/3078298598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2ea140 0x561f2c2f1ea0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc2540dffa0 tx=0x7fc25400cf20 comp rx=0 tx=0).stop 2024-06-04T04:59:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.767+0000 7fc26ee75740 1 -- 172.21.15.70:0/3078298598 shutdown_connections 2024-06-04T04:59:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.767+0000 7fc26ee75740 1 --2- 172.21.15.70:0/3078298598 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561f2c3048c0 0x561f2c306d80 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.767+0000 7fc26ee75740 1 --2- 172.21.15.70:0/3078298598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 0x561f2c2f23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.767+0000 7fc26ee75740 1 --2- 172.21.15.70:0/3078298598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2ea140 0x561f2c2f1ea0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.767+0000 7fc24d7fa640 0 client.25579 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.767+0000 7fc26ee75740 1 --2- 172.21.15.70:0/3078298598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2e9820 0x561f2c2f1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.767+0000 7fc26ee75740 1 -- 172.21.15.70:0/3078298598 >> 172.21.15.70:0/3078298598 conn(0x561f2c2e5d30 msgr2=0x561f2c2ebab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.767+0000 7fc26ee75740 1 -- 172.21.15.70:0/3078298598 shutdown_connections 2024-06-04T04:59:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.767+0000 7fc26ee75740 1 -- 172.21.15.70:0/3078298598 wait complete. 2024-06-04T04:59:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.767+0000 7fc26ee75740 20 client.25579 trim_cache size 0 max 16384 2024-06-04T04:59:21.790 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T04:59:21.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.792+0000 7fc26ee75740 1 Processor -- start 2024-06-04T04:59:21.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.792+0000 7fc26ee75740 1 -- start start 2024-06-04T04:59:21.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.792+0000 7fc26ee75740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2e9820 0x561f2c2e4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.792+0000 7fc26ee75740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2ea140 0x561f2c30c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.792+0000 7fc26ee75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 0x561f2c30cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.792+0000 7fc26ee75740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc24800e7d0 con 0x561f2c2ea140 2024-06-04T04:59:21.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.792+0000 7fc26ee75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc248010340 con 0x561f2c2f4a00 2024-06-04T04:59:21.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.792+0000 7fc26ee75740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc24800eea0 con 0x561f2c2e9820 2024-06-04T04:59:21.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.792+0000 7fc2696f3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2ea140 0x561f2c30c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:21.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.792+0000 7fc2696f3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2ea140 0x561f2c30c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40140/0 (socket says 172.21.15.70:40140) 2024-06-04T04:59:21.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.792+0000 7fc2696f3640 1 -- 172.21.15.70:0/2001213764 learned_addr learned my addr 172.21.15.70:0/2001213764 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:21.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.792+0000 7fc26a6f5640 1 --2- 172.21.15.70:0/2001213764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 0x561f2c30cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:21.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.792+0000 7fc269ef4640 1 --2- 172.21.15.70:0/2001213764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2e9820 0x561f2c2e4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:21.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.793+0000 7fc2696f3640 1 -- 172.21.15.70:0/2001213764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2e9820 msgr2=0x561f2c2e4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:21.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.793+0000 7fc2696f3640 1 --2- 172.21.15.70:0/2001213764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2e9820 0x561f2c2e4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.793+0000 7fc2696f3640 1 -- 172.21.15.70:0/2001213764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 msgr2=0x561f2c30cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:21.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.793+0000 7fc2696f3640 1 --2- 172.21.15.70:0/2001213764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 0x561f2c30cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.793+0000 7fc2696f3640 1 -- 172.21.15.70:0/2001213764 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f2c2e14d0 con 0x561f2c2ea140 2024-06-04T04:59:21.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.793+0000 7fc2696f3640 1 --2- 172.21.15.70:0/2001213764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2ea140 0x561f2c30c650 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc254023040 tx=0x7fc2540e2be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d3934cf9891cee2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:21.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.793+0000 7fc24f7fe640 1 -- 172.21.15.70:0/2001213764 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2540a92a0 con 0x561f2c2ea140 2024-06-04T04:59:21.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.793+0000 7fc24f7fe640 1 -- 172.21.15.70:0/2001213764 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc2540c2020 con 0x561f2c2ea140 2024-06-04T04:59:21.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.793+0000 7fc24f7fe640 1 -- 172.21.15.70:0/2001213764 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2540a9480 con 0x561f2c2ea140 2024-06-04T04:59:21.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.794+0000 7fc26ee75740 1 -- 172.21.15.70:0/2001213764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2ea140 msgr2=0x561f2c30c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:21.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.794+0000 7fc26ee75740 1 --2- 172.21.15.70:0/2001213764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2ea140 0x561f2c30c650 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc254023040 tx=0x7fc2540e2be0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.794+0000 7fc26ee75740 1 -- 172.21.15.70:0/2001213764 shutdown_connections 2024-06-04T04:59:21.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.794+0000 7fc26ee75740 1 --2- 172.21.15.70:0/2001213764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 0x561f2c30cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.794+0000 7fc26ee75740 1 --2- 172.21.15.70:0/2001213764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2ea140 0x561f2c30c650 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.794+0000 7fc26ee75740 1 --2- 172.21.15.70:0/2001213764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2e9820 0x561f2c2e4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.794+0000 7fc26ee75740 1 -- 172.21.15.70:0/2001213764 >> 172.21.15.70:0/2001213764 conn(0x561f2c2e5d30 msgr2=0x561f2c2e4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:21.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.794+0000 7fc26ee75740 1 -- 172.21.15.70:0/2001213764 shutdown_connections 2024-06-04T04:59:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.794+0000 7fc26ee75740 1 -- 172.21.15.70:0/2001213764 wait complete. 2024-06-04T04:59:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.794+0000 7fc26ee75740 1 -- start start 2024-06-04T04:59:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.794+0000 7fc26ee75740 1 Processor -- start 2024-06-04T04:59:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc26ee75740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2e9820 0x561f2c2fcc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc26ee75740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2ea140 0x561f2c2f8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc26ee75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 0x561f2c2f8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc26ee75740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc24800fc40 con 0x561f2c2e9820 2024-06-04T04:59:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc26ee75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc24800f570 con 0x561f2c2f4a00 2024-06-04T04:59:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc26ee75740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc24800c7b0 con 0x561f2c2ea140 2024-06-04T04:59:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc26ee75740 10 client.0 _ll_register_callbacks cb 0x7fc25400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc269ef4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2e9820 0x561f2c2fcc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc269ef4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2e9820 0x561f2c2fcc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40146/0 (socket says 172.21.15.70:40146) 2024-06-04T04:59:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc26a6f5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 0x561f2c2f8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc269ef4640 1 -- 172.21.15.70:0/3043294745 learned_addr learned my addr 172.21.15.70:0/3043294745 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc2696f3640 1 --2- 172.21.15.70:0/3043294745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2ea140 0x561f2c2f8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc2696f3640 1 -- 172.21.15.70:0/3043294745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 msgr2=0x561f2c2f8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc2696f3640 1 --2- 172.21.15.70:0/3043294745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 0x561f2c2f8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc2696f3640 1 -- 172.21.15.70:0/3043294745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2e9820 msgr2=0x561f2c2fcc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc2696f3640 1 --2- 172.21.15.70:0/3043294745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2e9820 0x561f2c2fcc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc2696f3640 1 -- 172.21.15.70:0/3043294745 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f2c2f8ab0 con 0x561f2c2ea140 2024-06-04T04:59:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.795+0000 7fc269ef4640 1 --2- 172.21.15.70:0/3043294745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2e9820 0x561f2c2fcc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.796+0000 7fc2696f3640 1 --2- 172.21.15.70:0/3043294745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2ea140 0x561f2c2f8030 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x561f2c2e5d00 tx=0x7fc2540e2c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b8b06735a7ebedb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.796+0000 7fc24d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:59:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.796+0000 7fc24d7fa640 1 -- 172.21.15.70:0/3043294745 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2540a9480 con 0x561f2c2ea140 2024-06-04T04:59:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.796+0000 7fc26ee75740 1 -- 172.21.15.70:0/3043294745 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2540c2020 con 0x561f2c2ea140 2024-06-04T04:59:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.796+0000 7fc24d7fa640 1 -- 172.21.15.70:0/3043294745 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc2540a6120 con 0x561f2c2ea140 2024-06-04T04:59:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.796+0000 7fc24d7fa640 1 -- 172.21.15.70:0/3043294745 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc2540a9900 con 0x561f2c2ea140 2024-06-04T04:59:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.796+0000 7fc26ee75740 10 client.25585 fetch_fsmap learned FSMap version 30 2024-06-04T04:59:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.796+0000 7fc26ee75740 1 -- 172.21.15.70:0/3043294745 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561f2c36f240 con 0x561f2c2ea140 2024-06-04T04:59:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.796+0000 7fc24d7fa640 1 -- 172.21.15.70:0/3043294745 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc2540c2020 con 0x561f2c2ea140 2024-06-04T04:59:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.796+0000 7fc24d7fa640 1 -- 172.21.15.70:0/3043294745 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fc2540cc0b0 con 0x561f2c2ea140 2024-06-04T04:59:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.797+0000 7fc26ee75740 10 client.25585 fetch_fsmap finished waiting for FSMap version 30 2024-06-04T04:59:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.797+0000 7fc26ee75740 10 client.25585 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:59:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.797+0000 7fc26ee75740 20 client.25585 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.797+0000 7fc26ee75740 1 --2- 172.21.15.70:0/3043294745 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561f2c36f710 0x561f2c371b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.797+0000 7fc26ee75740 4 client.25585 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T04:59:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.797+0000 7fc26ee75740 1 -- 172.21.15.70:0/3043294745 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f2c257360 con 0x561f2c36f710 2024-06-04T04:59:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.797+0000 7fc269ef4640 1 --2- 172.21.15.70:0/3043294745 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561f2c36f710 0x561f2c371b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.797+0000 7fc24d7fa640 10 client.25585 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.797+0000 7fc269ef4640 1 --2- 172.21.15.70:0/3043294745 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561f2c36f710 0x561f2c371b30 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d293604049015e8 server_cookie=4afc3fc8c18f5cf8 in_seq=0 out_seq=0 2024-06-04T04:59:21.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.826+0000 7fc24d7fa640 1 -- 172.21.15.70:0/3043294745 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x561f2c257360 con 0x561f2c36f710 2024-06-04T04:59:21.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.826+0000 7fc24d7fa640 10 client.25585 handle_command_reply: tid=0 2024-06-04T04:59:21.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.826+0000 7fc26ee75740 1 client.25585 shutdown 2024-06-04T04:59:21.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.826+0000 7fc26ee75740 1 -- 172.21.15.70:0/3043294745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2ea140 msgr2=0x561f2c2f8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:21.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.826+0000 7fc26ee75740 1 --2- 172.21.15.70:0/3043294745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2ea140 0x561f2c2f8030 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x561f2c2e5d00 tx=0x7fc2540e2c50 comp rx=0 tx=0).stop 2024-06-04T04:59:21.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.826+0000 7fc26ee75740 1 -- 172.21.15.70:0/3043294745 shutdown_connections 2024-06-04T04:59:21.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.826+0000 7fc26ee75740 1 --2- 172.21.15.70:0/3043294745 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561f2c36f710 0x561f2c371b30 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.826+0000 7fc26ee75740 1 --2- 172.21.15.70:0/3043294745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f2c2f4a00 0x561f2c2f8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc24d7fa640 0 client.25585 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:21.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26ee75740 1 --2- 172.21.15.70:0/3043294745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f2c2ea140 0x561f2c2f8030 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26ee75740 1 --2- 172.21.15.70:0/3043294745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f2c2e9820 0x561f2c2fcc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26ee75740 1 -- 172.21.15.70:0/3043294745 >> 172.21.15.70:0/3043294745 conn(0x561f2c2e5d30 msgr2=0x561f2c2e4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:21.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26ee75740 1 -- 172.21.15.70:0/3043294745 shutdown_connections 2024-06-04T04:59:21.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26ee75740 1 -- 172.21.15.70:0/3043294745 wait complete. 2024-06-04T04:59:21.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26ee75740 20 client.25585 trim_cache size 0 max 16384 2024-06-04T04:59:21.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T04:59:21.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T04:59:21.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T04:59:21.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26c158640 1 -- 172.21.15.70:0/3651282702 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc2500b2a10 msgr2=0x7fc2500b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:21.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26c158640 1 --2- 172.21.15.70:0/3651282702 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc2500b2a10 0x7fc2500b4ed0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x561f2c2eadc0 tx=0x7fc260005f90 comp rx=0 tx=0).stop 2024-06-04T04:59:21.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26c158640 1 -- 172.21.15.70:0/3651282702 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc264104600 msgr2=0x7fc2641995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:21.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26c158640 1 --2- 172.21.15.70:0/3651282702 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc264104600 0x7fc2641995e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc25400e9f0 tx=0x7fc25400eeb0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26c158640 1 -- 172.21.15.70:0/3651282702 shutdown_connections 2024-06-04T04:59:21.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26c158640 1 --2- 172.21.15.70:0/3651282702 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc2500b2a10 0x7fc2500b4ed0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26c158640 1 --2- 172.21.15.70:0/3651282702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc26410d020 0x7fc26419d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26c158640 1 --2- 172.21.15.70:0/3651282702 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc264104600 0x7fc2641995e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26c158640 1 --2- 172.21.15.70:0/3651282702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc264103ce0 0x7fc2641990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:21.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26c158640 1 -- 172.21.15.70:0/3651282702 >> 172.21.15.70:0/3651282702 conn(0x7fc264078ad0 msgr2=0x7fc2640feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:21.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26c158640 1 -- 172.21.15.70:0/3651282702 shutdown_connections 2024-06-04T04:59:21.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:21.827+0000 7fc26c158640 1 -- 172.21.15.70:0/3651282702 wait complete. 2024-06-04T04:59:22.055 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:59:22.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 10423 4.91 MB/sec warmup 15 sec latency 72.934 ms 2024-06-04T04:59:23.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:23 smithi070 ceph-mon[32631]: pgmap v69: 97 pgs: 97 active+clean; 25 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 3.4 MiB/s wr, 139 op/s 2024-06-04T04:59:23.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:23 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:59:23.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:23 smithi112 ceph-mon[36453]: pgmap v69: 97 pgs: 97 active+clean; 25 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 3.4 MiB/s wr, 139 op/s 2024-06-04T04:59:23.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:23 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:59:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:23 smithi098 ceph-mon[35569]: pgmap v69: 97 pgs: 97 active+clean; 25 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 3.4 MiB/s wr, 139 op/s 2024-06-04T04:59:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:23 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T04:59:23.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 10880 4.88 MB/sec warmup 16 sec latency 78.689 ms 2024-06-04T04:59:24.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:24 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T04:59:24.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:24 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T04:59:24.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:24 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T04:59:24.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 11125 4.61 MB/sec warmup 17 sec latency 36.324 ms 2024-06-04T04:59:25.394 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:25 smithi070 ceph-mon[32631]: pgmap v70: 97 pgs: 97 active+clean; 25 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.3 MiB/s wr, 87 op/s 2024-06-04T04:59:25.394 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:25 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:59:25.394 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:25 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:59:25.394 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:25 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:59:25.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:25 smithi098 ceph-mon[35569]: pgmap v70: 97 pgs: 97 active+clean; 25 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.3 MiB/s wr, 87 op/s 2024-06-04T04:59:25.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:25 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:59:25.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:25 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:59:25.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:25 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:59:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:25 smithi112 ceph-mon[36453]: pgmap v70: 97 pgs: 97 active+clean; 25 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.3 MiB/s wr, 87 op/s 2024-06-04T04:59:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:25 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T04:59:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:25 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T04:59:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:25 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T04:59:25.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 11593 4.40 MB/sec warmup 18 sec latency 15.958 ms 2024-06-04T04:59:25.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.787+0000 7f513b5a2700 1 -- 172.21.15.70:0/3521543817 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f513406bbe0 msgr2=0x7f51341103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:25.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.787+0000 7f513b5a2700 1 --2- 172.21.15.70:0/3521543817 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f513406bbe0 0x7f51341103d0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f512c00b0a0 tx=0x7f512c00b3b0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.787+0000 7f513b5a2700 1 -- 172.21.15.70:0/3521543817 shutdown_connections 2024-06-04T04:59:25.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.787+0000 7f513b5a2700 1 --2- 172.21.15.70:0/3521543817 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f513406bbe0 0x7f51341103d0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.787+0000 7f513b5a2700 1 --2- 172.21.15.70:0/3521543817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51341095b0 0x7f5134109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.787+0000 7f513b5a2700 1 --2- 172.21.15.70:0/3521543817 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f513406b220 0x7f513406b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.787+0000 7f513b5a2700 1 -- 172.21.15.70:0/3521543817 >> 172.21.15.70:0/3521543817 conn(0x7f51340793f0 msgr2=0x7f5134079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:25.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.787+0000 7f513b5a2700 1 -- 172.21.15.70:0/3521543817 shutdown_connections 2024-06-04T04:59:25.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.787+0000 7f513b5a2700 1 -- 172.21.15.70:0/3521543817 wait complete. 2024-06-04T04:59:25.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.788+0000 7f513b5a2700 1 Processor -- start 2024-06-04T04:59:25.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.788+0000 7f513b5a2700 1 -- start start 2024-06-04T04:59:25.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.788+0000 7f513b5a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f513406b220 0x7f5134102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.788+0000 7f513b5a2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513406bbe0 0x7f5134102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.788+0000 7f513b5a2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51341095b0 0x7f5134102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.788+0000 7f513b5a2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5134113570 con 0x7f513406b220 2024-06-04T04:59:25.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.788+0000 7f513b5a2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51341133f0 con 0x7f513406bbe0 2024-06-04T04:59:25.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.788+0000 7f513b5a2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f51341136f0 con 0x7f51341095b0 2024-06-04T04:59:25.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.788+0000 7f513933e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f513406b220 0x7f5134102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.788+0000 7f5139b3f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51341095b0 0x7f5134102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.788+0000 7f5138b3d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513406bbe0 0x7f5134102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.788+0000 7f5139b3f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51341095b0 0x7f5134102f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:45356/0 (socket says 172.21.15.70:45356) 2024-06-04T04:59:25.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.788+0000 7f5139b3f700 1 -- 172.21.15.70:0/861029847 learned_addr learned my addr 172.21.15.70:0/861029847 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:25.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.789+0000 7f5138b3d700 1 -- 172.21.15.70:0/861029847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51341095b0 msgr2=0x7f5134102f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:25.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.789+0000 7f5138b3d700 1 --2- 172.21.15.70:0/861029847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51341095b0 0x7f5134102f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.789+0000 7f5138b3d700 1 -- 172.21.15.70:0/861029847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f513406b220 msgr2=0x7f5134102510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:25.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.789+0000 7f5138b3d700 1 --2- 172.21.15.70:0/861029847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f513406b220 0x7f5134102510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.789+0000 7f5138b3d700 1 -- 172.21.15.70:0/861029847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51341a7ff0 con 0x7f513406bbe0 2024-06-04T04:59:25.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.789+0000 7f513933e700 1 --2- 172.21.15.70:0/861029847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f513406b220 0x7f5134102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:25.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.789+0000 7f5139b3f700 1 --2- 172.21.15.70:0/861029847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51341095b0 0x7f5134102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:25.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.789+0000 7f5138b3d700 1 --2- 172.21.15.70:0/861029847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513406bbe0 0x7f5134102a50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f512400c970 tx=0x7f512400cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:25.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.789+0000 7f512a7fc700 1 -- 172.21.15.70:0/861029847 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5124004a30 con 0x7f513406bbe0 2024-06-04T04:59:25.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.789+0000 7f513b5a2700 1 -- 172.21.15.70:0/861029847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51341a8280 con 0x7f513406bbe0 2024-06-04T04:59:25.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.789+0000 7f512a7fc700 1 -- 172.21.15.70:0/861029847 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51240054b0 con 0x7f513406bbe0 2024-06-04T04:59:25.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.789+0000 7f512a7fc700 1 -- 172.21.15.70:0/861029847 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512400eae0 con 0x7f513406bbe0 2024-06-04T04:59:25.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.789+0000 7f513b5a2700 1 -- 172.21.15.70:0/861029847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51341a87c0 con 0x7f513406bbe0 2024-06-04T04:59:25.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.792+0000 7f512a7fc700 1 -- 172.21.15.70:0/861029847 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f512400ec80 con 0x7f513406bbe0 2024-06-04T04:59:25.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.792+0000 7f512a7fc700 1 --2- 172.21.15.70:0/861029847 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f51200b29f0 0x7f51200b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.792+0000 7f513933e700 1 --2- 172.21.15.70:0/861029847 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f51200b29f0 0x7f51200b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.793+0000 7f512a7fc700 1 -- 172.21.15.70:0/861029847 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f51240e2090 con 0x7f513406bbe0 2024-06-04T04:59:25.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.793+0000 7f513933e700 1 --2- 172.21.15.70:0/861029847 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f51200b29f0 0x7f51200b4eb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5130009680 tx=0x7f5130009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:25.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.793+0000 7f5145e37b80 1 Processor -- start 2024-06-04T04:59:25.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.793+0000 7f5145e37b80 1 -- start start 2024-06-04T04:59:25.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.794+0000 7f5145e37b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 0x564464b66c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.794+0000 7f5145e37b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b67210 0x564464b6a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.794+0000 7f5145e37b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b6af60 0x564464b6b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.794+0000 7f5145e37b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564464ac5c90 con 0x564464b6af60 2024-06-04T04:59:25.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.794+0000 7f5145e37b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564464ab5830 con 0x564464b67210 2024-06-04T04:59:25.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.794+0000 7f5145e37b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564464ab9870 con 0x564464b66860 2024-06-04T04:59:25.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.794+0000 7f513933e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 0x564464b66c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.794+0000 7f5138b3d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b67210 0x564464b6a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.794+0000 7f5139b3f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b6af60 0x564464b6b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.794+0000 7f513933e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 0x564464b66c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:45372/0 (socket says 172.21.15.70:45372) 2024-06-04T04:59:25.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.794+0000 7f513933e700 1 -- 172.21.15.70:0/165980349 learned_addr learned my addr 172.21.15.70:0/165980349 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:25.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.794+0000 7f513933e700 1 -- 172.21.15.70:0/165980349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b67210 msgr2=0x564464b6a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:25.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.794+0000 7f513933e700 1 --2- 172.21.15.70:0/165980349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b67210 0x564464b6a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.794+0000 7f513933e700 1 -- 172.21.15.70:0/165980349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b6af60 msgr2=0x564464b6b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:25.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.794+0000 7f513933e700 1 --2- 172.21.15.70:0/165980349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b6af60 0x564464b6b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.794+0000 7f513933e700 1 -- 172.21.15.70:0/165980349 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564464a058d0 con 0x564464b66860 2024-06-04T04:59:25.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.794+0000 7f513933e700 1 --2- 172.21.15.70:0/165980349 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 0x564464b66c40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f513000e950 tx=0x7f5130014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a26884d0ee2fc98c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:25.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.795+0000 7f511f7fe700 1 -- 172.21.15.70:0/165980349 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5130029070 con 0x564464b66860 2024-06-04T04:59:25.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.795+0000 7f511f7fe700 1 -- 172.21.15.70:0/165980349 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f513001c990 con 0x564464b66860 2024-06-04T04:59:25.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.795+0000 7f511f7fe700 1 -- 172.21.15.70:0/165980349 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513001aac0 con 0x564464b66860 2024-06-04T04:59:25.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.796+0000 7f5145e37b80 1 -- 172.21.15.70:0/165980349 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 msgr2=0x564464b66c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:25.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.796+0000 7f5145e37b80 1 --2- 172.21.15.70:0/165980349 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 0x564464b66c40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f513000e950 tx=0x7f5130014000 comp rx=0 tx=0).stop 2024-06-04T04:59:25.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.796+0000 7f5145e37b80 1 -- 172.21.15.70:0/165980349 shutdown_connections 2024-06-04T04:59:25.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.796+0000 7f5145e37b80 1 --2- 172.21.15.70:0/165980349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b6af60 0x564464b6b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.796+0000 7f5145e37b80 1 --2- 172.21.15.70:0/165980349 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 0x564464b66c40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.796+0000 7f5145e37b80 1 --2- 172.21.15.70:0/165980349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b67210 0x564464b6a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.796+0000 7f5145e37b80 1 -- 172.21.15.70:0/165980349 >> 172.21.15.70:0/165980349 conn(0x564464a94080 msgr2=0x564464b60e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:25.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.796+0000 7f5145e37b80 1 -- 172.21.15.70:0/165980349 shutdown_connections 2024-06-04T04:59:25.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.796+0000 7f5145e37b80 1 -- 172.21.15.70:0/165980349 wait complete. 2024-06-04T04:59:25.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.796+0000 7f5145e37b80 1 -- start start 2024-06-04T04:59:25.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.796+0000 7f5145e37b80 1 Processor -- start 2024-06-04T04:59:25.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f5145e37b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b66860 0x564464b723b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f5145e37b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b67210 0x564464b6f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f5145e37b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b6af60 0x564464b6fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f5145e37b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5644649bbfa0 con 0x564464b6af60 2024-06-04T04:59:25.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f5145e37b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564464a577c0 con 0x564464b66860 2024-06-04T04:59:25.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f5145e37b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564464b73120 con 0x564464b67210 2024-06-04T04:59:25.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f5145e37b80 10 client.0 _ll_register_callbacks cb 0x564464a50920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:25.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f5139b3f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b6af60 0x564464b6fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f513933e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b66860 0x564464b723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f5139b3f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b6af60 0x564464b6fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40176/0 (socket says 172.21.15.70:40176) 2024-06-04T04:59:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f5138b3d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b67210 0x564464b6f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f5139b3f700 1 -- 172.21.15.70:0/3594383690 learned_addr learned my addr 172.21.15.70:0/3594383690 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f513933e700 1 -- 172.21.15.70:0/3594383690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b67210 msgr2=0x564464b6f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f513933e700 1 --2- 172.21.15.70:0/3594383690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b67210 0x564464b6f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f513933e700 1 -- 172.21.15.70:0/3594383690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b6af60 msgr2=0x564464b6fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f513933e700 1 --2- 172.21.15.70:0/3594383690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b6af60 0x564464b6fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f513933e700 1 -- 172.21.15.70:0/3594383690 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564464a54690 con 0x564464b66860 2024-06-04T04:59:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f5138b3d700 1 --2- 172.21.15.70:0/3594383690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b67210 0x564464b6f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f513933e700 1 --2- 172.21.15.70:0/3594383690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b66860 0x564464b723b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f513000e920 tx=0x7f513001c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3b6983c62b1bc85 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.797+0000 7f511cff9700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T04:59:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.798+0000 7f511cff9700 1 -- 172.21.15.70:0/3594383690 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5130032060 con 0x564464b66860 2024-06-04T04:59:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.798+0000 7f511cff9700 1 -- 172.21.15.70:0/3594383690 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f5130005250 con 0x564464b66860 2024-06-04T04:59:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.798+0000 7f5145e37b80 1 -- 172.21.15.70:0/3594383690 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f513001c990 con 0x564464b66860 2024-06-04T04:59:25.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.798+0000 7f511cff9700 1 -- 172.21.15.70:0/3594383690 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5130029070 con 0x564464b66860 2024-06-04T04:59:25.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.798+0000 7f5145e37b80 10 client.25532 fetch_fsmap learned FSMap version 31 2024-06-04T04:59:25.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.798+0000 7f5145e37b80 1 -- 172.21.15.70:0/3594383690 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564464b7d820 con 0x564464b66860 2024-06-04T04:59:25.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.798+0000 7f511cff9700 1 -- 172.21.15.70:0/3594383690 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f513001c990 con 0x564464b66860 2024-06-04T04:59:25.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.798+0000 7f511cff9700 1 -- 172.21.15.70:0/3594383690 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5130018040 con 0x564464b66860 2024-06-04T04:59:25.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.798+0000 7f5145e37b80 10 client.25532 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T04:59:25.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.798+0000 7f5145e37b80 10 client.25532 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:59:25.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.798+0000 7f5145e37b80 20 client.25532 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:25.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.798+0000 7f5145e37b80 1 --2- 172.21.15.70:0/3594383690 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564464b7fcf0 0x564464b820e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.798+0000 7f5145e37b80 4 client.25532 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:59:25.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.798+0000 7f5145e37b80 1 -- 172.21.15.70:0/3594383690 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564464b329e0 con 0x564464b7fcf0 2024-06-04T04:59:25.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.799+0000 7f5138b3d700 1 --2- 172.21.15.70:0/3594383690 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564464b7fcf0 0x564464b820e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.799+0000 7f511cff9700 10 client.25532 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:25.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.799+0000 7f5138b3d700 1 --2- 172.21.15.70:0/3594383690 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564464b7fcf0 0x564464b820e0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c1c11331cebc8e1 server_cookie=d186a63c77ffa58 in_seq=0 out_seq=0 2024-06-04T04:59:25.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.800+0000 7f511cff9700 1 -- 172.21.15.70:0/3594383690 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564464b329e0 con 0x564464b7fcf0 2024-06-04T04:59:25.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.800+0000 7f511cff9700 10 client.25532 handle_command_reply: tid=0 2024-06-04T04:59:25.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.800+0000 7f5145e37b80 1 client.25532 shutdown 2024-06-04T04:59:25.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.800+0000 7f5145e37b80 1 -- 172.21.15.70:0/3594383690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b66860 msgr2=0x564464b723b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:25.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.800+0000 7f5145e37b80 1 --2- 172.21.15.70:0/3594383690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b66860 0x564464b723b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f513000e920 tx=0x7f513001c8e0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.800+0000 7f5145e37b80 1 -- 172.21.15.70:0/3594383690 shutdown_connections 2024-06-04T04:59:25.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.800+0000 7f5145e37b80 1 --2- 172.21.15.70:0/3594383690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b6af60 0x564464b6fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.800+0000 7f5145e37b80 1 --2- 172.21.15.70:0/3594383690 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564464b7fcf0 0x564464b820e0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.800+0000 7f5145e37b80 1 --2- 172.21.15.70:0/3594383690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b66860 0x564464b723b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.800+0000 7f5145e37b80 1 --2- 172.21.15.70:0/3594383690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b67210 0x564464b6f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.800+0000 7f511cff9700 0 client.25532 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:25.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.800+0000 7f5145e37b80 1 -- 172.21.15.70:0/3594383690 >> 172.21.15.70:0/3594383690 conn(0x564464a94080 msgr2=0x564464b60510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:25.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.800+0000 7f5145e37b80 1 -- 172.21.15.70:0/3594383690 shutdown_connections 2024-06-04T04:59:25.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.800+0000 7f5145e37b80 1 -- 172.21.15.70:0/3594383690 wait complete. 2024-06-04T04:59:25.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.800+0000 7f5145e37b80 20 client.25532 trim_cache size 0 max 16384 2024-06-04T04:59:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5145e37b80 1 Processor -- start 2024-06-04T04:59:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5145e37b80 1 -- start start 2024-06-04T04:59:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5145e37b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 0x564464b65d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5145e37b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b67210 0x564464b87740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5145e37b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b6af60 0x564464b87c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5145e37b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f51341affe0 con 0x564464b67210 2024-06-04T04:59:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5145e37b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51341b1b80 con 0x564464b6af60 2024-06-04T04:59:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5145e37b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f51341b06b0 con 0x564464b66860 2024-06-04T04:59:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5138b3d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b67210 0x564464b87740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5138b3d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b67210 0x564464b87740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40184/0 (socket says 172.21.15.70:40184) 2024-06-04T04:59:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5138b3d700 1 -- 172.21.15.70:0/3953394140 learned_addr learned my addr 172.21.15.70:0/3953394140 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f513933e700 1 --2- 172.21.15.70:0/3953394140 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 0x564464b65d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5138b3d700 1 -- 172.21.15.70:0/3953394140 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 msgr2=0x564464b65d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:25.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5138b3d700 1 --2- 172.21.15.70:0/3953394140 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 0x564464b65d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5138b3d700 1 -- 172.21.15.70:0/3953394140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b6af60 msgr2=0x564464b87c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:25.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5139b3f700 1 --2- 172.21.15.70:0/3953394140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b6af60 0x564464b87c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5138b3d700 1 --2- 172.21.15.70:0/3953394140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b6af60 0x564464b87c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5138b3d700 1 -- 172.21.15.70:0/3953394140 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564464b881c0 con 0x564464b67210 2024-06-04T04:59:25.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.817+0000 7f5138b3d700 1 --2- 172.21.15.70:0/3953394140 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b67210 0x564464b87740 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f51240a53f0 tx=0x7f5124012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f581eea1f4afaecb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:25.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.818+0000 7f511effd700 1 -- 172.21.15.70:0/3953394140 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512401d040 con 0x564464b67210 2024-06-04T04:59:25.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.818+0000 7f511effd700 1 -- 172.21.15.70:0/3953394140 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51240ad720 con 0x564464b67210 2024-06-04T04:59:25.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.818+0000 7f511effd700 1 -- 172.21.15.70:0/3953394140 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51240b1d40 con 0x564464b67210 2024-06-04T04:59:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.818+0000 7f5145e37b80 1 -- 172.21.15.70:0/3953394140 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b67210 msgr2=0x564464b87740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.818+0000 7f5145e37b80 1 --2- 172.21.15.70:0/3953394140 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b67210 0x564464b87740 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f51240a53f0 tx=0x7f5124012000 comp rx=0 tx=0).stop 2024-06-04T04:59:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.818+0000 7f5145e37b80 1 -- 172.21.15.70:0/3953394140 shutdown_connections 2024-06-04T04:59:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.818+0000 7f5145e37b80 1 --2- 172.21.15.70:0/3953394140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b6af60 0x564464b87c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.818+0000 7f5145e37b80 1 --2- 172.21.15.70:0/3953394140 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 0x564464b65d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.819+0000 7f5145e37b80 1 --2- 172.21.15.70:0/3953394140 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b67210 0x564464b87740 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.819+0000 7f5145e37b80 1 -- 172.21.15.70:0/3953394140 >> 172.21.15.70:0/3953394140 conn(0x564464a94080 msgr2=0x564464b65670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.819+0000 7f5145e37b80 1 -- 172.21.15.70:0/3953394140 shutdown_connections 2024-06-04T04:59:25.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.819+0000 7f5145e37b80 1 -- 172.21.15.70:0/3953394140 wait complete. 2024-06-04T04:59:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.819+0000 7f5145e37b80 1 -- start start 2024-06-04T04:59:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.819+0000 7f5145e37b80 1 Processor -- start 2024-06-04T04:59:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.819+0000 7f5145e37b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 0x564464be29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.819+0000 7f5145e37b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b67210 0x564464be2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.819+0000 7f5145e37b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b6af60 0x564464b759e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.819+0000 7f5145e37b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f51341b1450 con 0x564464b6af60 2024-06-04T04:59:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.819+0000 7f5145e37b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51341b0d80 con 0x564464b67210 2024-06-04T04:59:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.819+0000 7f5145e37b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f51341adf80 con 0x564464b66860 2024-06-04T04:59:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.819+0000 7f5145e37b80 10 client.0 _ll_register_callbacks cb 0x564464a50920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f513933e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 0x564464be29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f5138b3d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b67210 0x564464be2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f513933e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 0x564464be29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:45386/0 (socket says 172.21.15.70:45386) 2024-06-04T04:59:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f513933e700 1 -- 172.21.15.70:0/3927685623 learned_addr learned my addr 172.21.15.70:0/3927685623 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f5139b3f700 1 --2- 172.21.15.70:0/3927685623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b6af60 0x564464b759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f513933e700 1 -- 172.21.15.70:0/3927685623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b67210 msgr2=0x564464be2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f513933e700 1 --2- 172.21.15.70:0/3927685623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b67210 0x564464be2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f513933e700 1 -- 172.21.15.70:0/3927685623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b6af60 msgr2=0x564464b759e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f513933e700 1 --2- 172.21.15.70:0/3927685623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b6af60 0x564464b759e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f513933e700 1 -- 172.21.15.70:0/3927685623 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564464b760e0 con 0x564464b66860 2024-06-04T04:59:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f5138b3d700 1 --2- 172.21.15.70:0/3927685623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b67210 0x564464be2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f513933e700 1 --2- 172.21.15.70:0/3927685623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 0x564464be29f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f513001a4d0 tx=0x7f513000ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=790b4daa46047731 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f511cff9700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:59:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f511cff9700 1 -- 172.21.15.70:0/3927685623 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5130032040 con 0x564464b66860 2024-06-04T04:59:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f5145e37b80 1 -- 172.21.15.70:0/3927685623 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51240ad720 con 0x564464b66860 2024-06-04T04:59:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f511cff9700 1 -- 172.21.15.70:0/3927685623 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f5130005450 con 0x564464b66860 2024-06-04T04:59:25.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f511cff9700 1 -- 172.21.15.70:0/3927685623 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5130004ad0 con 0x564464b66860 2024-06-04T04:59:25.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f5145e37b80 10 client.25600 fetch_fsmap learned FSMap version 31 2024-06-04T04:59:25.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.820+0000 7f5145e37b80 1 -- 172.21.15.70:0/3927685623 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564464bea220 con 0x564464b66860 2024-06-04T04:59:25.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.821+0000 7f511cff9700 1 -- 172.21.15.70:0/3927685623 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51240ad720 con 0x564464b66860 2024-06-04T04:59:25.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.821+0000 7f511cff9700 1 -- 172.21.15.70:0/3927685623 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5130018040 con 0x564464b66860 2024-06-04T04:59:25.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.821+0000 7f5145e37b80 10 client.25600 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T04:59:25.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.821+0000 7f5145e37b80 10 client.25600 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:59:25.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.821+0000 7f5145e37b80 20 client.25600 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:25.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.821+0000 7f5145e37b80 1 --2- 172.21.15.70:0/3927685623 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564464bea8b0 0x564464beccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:25.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.821+0000 7f5145e37b80 4 client.25600 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T04:59:25.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.821+0000 7f5145e37b80 1 -- 172.21.15.70:0/3927685623 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564464ac7be0 con 0x564464bea8b0 2024-06-04T04:59:25.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.821+0000 7f5138b3d700 1 --2- 172.21.15.70:0/3927685623 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564464bea8b0 0x564464beccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:25.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.821+0000 7f511cff9700 10 client.25600 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:25.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.821+0000 7f5138b3d700 1 --2- 172.21.15.70:0/3927685623 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564464bea8b0 0x564464beccd0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fcec7a561f1c565 server_cookie=588e7923812d9032 in_seq=0 out_seq=0 2024-06-04T04:59:25.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.823+0000 7f511cff9700 1 -- 172.21.15.70:0/3927685623 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564464ac7be0 con 0x564464bea8b0 2024-06-04T04:59:25.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.823+0000 7f511cff9700 10 client.25600 handle_command_reply: tid=0 2024-06-04T04:59:25.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.823+0000 7f5145e37b80 1 client.25600 shutdown 2024-06-04T04:59:25.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.823+0000 7f5145e37b80 1 -- 172.21.15.70:0/3927685623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 msgr2=0x564464be29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:25.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.823+0000 7f5145e37b80 1 --2- 172.21.15.70:0/3927685623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 0x564464be29f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f513001a4d0 tx=0x7f513000ef60 comp rx=0 tx=0).stop 2024-06-04T04:59:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.823+0000 7f5145e37b80 1 -- 172.21.15.70:0/3927685623 shutdown_connections 2024-06-04T04:59:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.823+0000 7f5145e37b80 1 --2- 172.21.15.70:0/3927685623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564464b6af60 0x564464b759e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.823+0000 7f5145e37b80 1 --2- 172.21.15.70:0/3927685623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564464b66860 0x564464be29f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.823+0000 7f5145e37b80 1 --2- 172.21.15.70:0/3927685623 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564464bea8b0 0x564464beccd0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.823+0000 7f5145e37b80 1 --2- 172.21.15.70:0/3927685623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564464b67210 0x564464be2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.823+0000 7f511cff9700 0 client.25600 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.823+0000 7f5145e37b80 1 -- 172.21.15.70:0/3927685623 >> 172.21.15.70:0/3927685623 conn(0x564464a94080 msgr2=0x564464b70ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.823+0000 7f5145e37b80 1 -- 172.21.15.70:0/3927685623 shutdown_connections 2024-06-04T04:59:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.823+0000 7f5145e37b80 1 -- 172.21.15.70:0/3927685623 wait complete. 2024-06-04T04:59:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.823+0000 7f5145e37b80 20 client.25600 trim_cache size 0 max 16384 2024-06-04T04:59:25.829 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T04:59:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.824+0000 7f513b5a2700 1 -- 172.21.15.70:0/861029847 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f51200b29f0 msgr2=0x7f51200b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.824+0000 7f513b5a2700 1 --2- 172.21.15.70:0/861029847 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f51200b29f0 0x7f51200b4eb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5130009680 tx=0x7f5130009260 comp rx=0 tx=0).stop 2024-06-04T04:59:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.824+0000 7f513b5a2700 1 -- 172.21.15.70:0/861029847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513406bbe0 msgr2=0x7f5134102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.824+0000 7f513b5a2700 1 --2- 172.21.15.70:0/861029847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513406bbe0 0x7f5134102a50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f512400c970 tx=0x7f512400cc80 comp rx=0 tx=0).stop 2024-06-04T04:59:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.824+0000 7f513b5a2700 1 -- 172.21.15.70:0/861029847 shutdown_connections 2024-06-04T04:59:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.824+0000 7f513b5a2700 1 --2- 172.21.15.70:0/861029847 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f51200b29f0 0x7f51200b4eb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.824+0000 7f513b5a2700 1 --2- 172.21.15.70:0/861029847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51341095b0 0x7f5134102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.824+0000 7f513b5a2700 1 --2- 172.21.15.70:0/861029847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f513406b220 0x7f5134102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.824+0000 7f513b5a2700 1 --2- 172.21.15.70:0/861029847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513406bbe0 0x7f5134102a50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.824+0000 7f513b5a2700 1 -- 172.21.15.70:0/861029847 >> 172.21.15.70:0/861029847 conn(0x7f51340793f0 msgr2=0x7f51341011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.824+0000 7f513b5a2700 1 -- 172.21.15.70:0/861029847 shutdown_connections 2024-06-04T04:59:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:25.824+0000 7f513b5a2700 1 -- 172.21.15.70:0/861029847 wait complete. 2024-06-04T04:59:26.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 12210 4.39 MB/sec warmup 19 sec latency 15.871 ms 2024-06-04T04:59:27.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:27 smithi098 ceph-mon[35569]: pgmap v71: 97 pgs: 97 active+clean; 30 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 2.1 MiB/s wr, 160 op/s 2024-06-04T04:59:27.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:27 smithi070 ceph-mon[32631]: pgmap v71: 97 pgs: 97 active+clean; 30 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 2.1 MiB/s wr, 160 op/s 2024-06-04T04:59:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:27 smithi112 ceph-mon[36453]: pgmap v71: 97 pgs: 97 active+clean; 30 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 2.1 MiB/s wr, 160 op/s 2024-06-04T04:59:27.697 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T04:59:27.697 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T04:59:27.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 12951 4.26 MB/sec warmup 20 sec latency 15.075 ms 2024-06-04T04:59:27.961 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:59:28.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 13568 4.33 MB/sec warmup 21 sec latency 21.806 ms 2024-06-04T04:59:29.372 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:29 smithi070 ceph-mon[32631]: pgmap v72: 97 pgs: 97 active+clean; 30 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-06-04T04:59:29.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:29 smithi098 ceph-mon[35569]: pgmap v72: 97 pgs: 97 active+clean; 30 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-06-04T04:59:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:29 smithi112 ceph-mon[36453]: pgmap v72: 97 pgs: 97 active+clean; 30 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-06-04T04:59:29.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 14040 4.22 MB/sec warmup 22 sec latency 67.304 ms 2024-06-04T04:59:29.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.735+0000 7fdc190a7700 1 -- 172.21.15.70:0/271322503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc1406b220 msgr2=0x7fdc1406b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:29.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.735+0000 7fdc190a7700 1 --2- 172.21.15.70:0/271322503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc1406b220 0x7fdc1406b6a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fdc08009990 tx=0x7fdc08009ca0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.735+0000 7fdc190a7700 1 -- 172.21.15.70:0/271322503 shutdown_connections 2024-06-04T04:59:29.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.735+0000 7fdc190a7700 1 --2- 172.21.15.70:0/271322503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1406bbe0 0x7fdc141103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.735+0000 7fdc190a7700 1 --2- 172.21.15.70:0/271322503 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc141095b0 0x7fdc14109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.735+0000 7fdc190a7700 1 --2- 172.21.15.70:0/271322503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc1406b220 0x7fdc1406b6a0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.735+0000 7fdc190a7700 1 -- 172.21.15.70:0/271322503 >> 172.21.15.70:0/271322503 conn(0x7fdc140793f0 msgr2=0x7fdc14079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:29.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.736+0000 7fdc190a7700 1 -- 172.21.15.70:0/271322503 shutdown_connections 2024-06-04T04:59:29.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.736+0000 7fdc190a7700 1 -- 172.21.15.70:0/271322503 wait complete. 2024-06-04T04:59:29.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.736+0000 7fdc190a7700 1 Processor -- start 2024-06-04T04:59:29.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.736+0000 7fdc190a7700 1 -- start start 2024-06-04T04:59:29.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc190a7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1406b220 0x7fdc1419f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc190a7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc1406bbe0 0x7fdc1419fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc190a7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc141095b0 0x7fdc14199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc190a7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc14113570 con 0x7fdc1406bbe0 2024-06-04T04:59:29.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc190a7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141133f0 con 0x7fdc1406b220 2024-06-04T04:59:29.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc190a7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141136f0 con 0x7fdc141095b0 2024-06-04T04:59:29.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc1259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc1406bbe0 0x7fdc1419fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc12d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1406b220 0x7fdc1419f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc12d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1406b220 0x7fdc1419f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46352/0 (socket says 172.21.15.70:46352) 2024-06-04T04:59:29.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc12d9d700 1 -- 172.21.15.70:0/1593485802 learned_addr learned my addr 172.21.15.70:0/1593485802 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:29.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc1359e700 1 --2- 172.21.15.70:0/1593485802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc141095b0 0x7fdc14199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc12d9d700 1 -- 172.21.15.70:0/1593485802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc141095b0 msgr2=0x7fdc14199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:29.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc12d9d700 1 --2- 172.21.15.70:0/1593485802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc141095b0 0x7fdc14199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc12d9d700 1 -- 172.21.15.70:0/1593485802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc1406bbe0 msgr2=0x7fdc1419fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:29.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc12d9d700 1 --2- 172.21.15.70:0/1593485802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc1406bbe0 0x7fdc1419fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc12d9d700 1 -- 172.21.15.70:0/1593485802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1419a0b0 con 0x7fdc1406b220 2024-06-04T04:59:29.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc1259c700 1 --2- 172.21.15.70:0/1593485802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc1406bbe0 0x7fdc1419fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:29.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc1359e700 1 --2- 172.21.15.70:0/1593485802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc141095b0 0x7fdc14199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:29.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.737+0000 7fdc12d9d700 1 --2- 172.21.15.70:0/1593485802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1406b220 0x7fdc1419f790 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400e9a0 tx=0x7fdc0400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:29.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.738+0000 7fdbfbfff700 1 -- 172.21.15.70:0/1593485802 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0400ca10 con 0x7fdc1406b220 2024-06-04T04:59:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.738+0000 7fdbfbfff700 1 -- 172.21.15.70:0/1593485802 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc0400cbb0 con 0x7fdc1406b220 2024-06-04T04:59:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.738+0000 7fdc190a7700 1 -- 172.21.15.70:0/1593485802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc1419a3a0 con 0x7fdc1406b220 2024-06-04T04:59:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.738+0000 7fdbfbfff700 1 -- 172.21.15.70:0/1593485802 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04018680 con 0x7fdc1406b220 2024-06-04T04:59:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.738+0000 7fdc190a7700 1 -- 172.21.15.70:0/1593485802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc141a6660 con 0x7fdc1406b220 2024-06-04T04:59:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.740+0000 7fdbfbfff700 1 -- 172.21.15.70:0/1593485802 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fdc04010430 con 0x7fdc1406b220 2024-06-04T04:59:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.741+0000 7fdbfbfff700 1 --2- 172.21.15.70:0/1593485802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdbfc0b29f0 0x7fdbfc0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.741+0000 7fdc1259c700 1 --2- 172.21.15.70:0/1593485802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdbfc0b29f0 0x7fdbfc0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.741+0000 7fdbfbfff700 1 -- 172.21.15.70:0/1593485802 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fdc040e2760 con 0x7fdc1406b220 2024-06-04T04:59:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.741+0000 7fdc2393cb80 1 Processor -- start 2024-06-04T04:59:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.741+0000 7fdc1259c700 1 --2- 172.21.15.70:0/1593485802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdbfc0b29f0 0x7fdbfc0b4eb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdc14079b30 tx=0x7fdc0801a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:29.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.741+0000 7fdc2393cb80 1 -- start start 2024-06-04T04:59:29.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.742+0000 7fdc2393cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583098323a0 0x558309832780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.742+0000 7fdc2393cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 0x55830982cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.742+0000 7fdc2393cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x558309839100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.742+0000 7fdc2393cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55830978adf0 con 0x55830982c920 2024-06-04T04:59:29.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.742+0000 7fdc2393cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55830977a990 con 0x5583098323a0 2024-06-04T04:59:29.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.742+0000 7fdc2393cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55830977e9d0 con 0x55830982d2e0 2024-06-04T04:59:29.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.742+0000 7fdc1259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 0x55830982cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.742+0000 7fdc1259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 0x55830982cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54316/0 (socket says 172.21.15.70:54316) 2024-06-04T04:59:29.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.742+0000 7fdc1259c700 1 -- 172.21.15.70:0/947486678 learned_addr learned my addr 172.21.15.70:0/947486678 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:29.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.742+0000 7fdc1359e700 1 --2- 172.21.15.70:0/947486678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x558309839100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.742+0000 7fdc12d9d700 1 --2- 172.21.15.70:0/947486678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583098323a0 0x558309832780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.742+0000 7fdc1359e700 1 -- 172.21.15.70:0/947486678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583098323a0 msgr2=0x558309832780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:29.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.742+0000 7fdc1359e700 1 --2- 172.21.15.70:0/947486678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583098323a0 0x558309832780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.742+0000 7fdc1359e700 1 -- 172.21.15.70:0/947486678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 msgr2=0x55830982cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:29.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.742+0000 7fdc1359e700 1 --2- 172.21.15.70:0/947486678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 0x55830982cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.742+0000 7fdc1359e700 1 -- 172.21.15.70:0/947486678 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558309724030 con 0x55830982d2e0 2024-06-04T04:59:29.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.742+0000 7fdc1359e700 1 --2- 172.21.15.70:0/947486678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x558309839100 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0076b0 tx=0x7fdc0c0079c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf5147df6548f68a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:29.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.743+0000 7fdbf97fa700 1 -- 172.21.15.70:0/947486678 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c019030 con 0x55830982d2e0 2024-06-04T04:59:29.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.743+0000 7fdbf97fa700 1 -- 172.21.15.70:0/947486678 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc0c004bb0 con 0x55830982d2e0 2024-06-04T04:59:29.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.743+0000 7fdbf97fa700 1 -- 172.21.15.70:0/947486678 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c005260 con 0x55830982d2e0 2024-06-04T04:59:29.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.743+0000 7fdc2393cb80 1 -- 172.21.15.70:0/947486678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 msgr2=0x558309839100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:29.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.743+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/947486678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x558309839100 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0076b0 tx=0x7fdc0c0079c0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.743+0000 7fdc2393cb80 1 -- 172.21.15.70:0/947486678 shutdown_connections 2024-06-04T04:59:29.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.743+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/947486678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x558309839100 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.743+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/947486678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583098323a0 0x558309832780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.743+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/947486678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 0x55830982cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.743+0000 7fdc2393cb80 1 -- 172.21.15.70:0/947486678 >> 172.21.15.70:0/947486678 conn(0x5583097582c0 msgr2=0x558309827460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.743+0000 7fdc2393cb80 1 -- 172.21.15.70:0/947486678 shutdown_connections 2024-06-04T04:59:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.743+0000 7fdc2393cb80 1 -- 172.21.15.70:0/947486678 wait complete. 2024-06-04T04:59:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.744+0000 7fdc2393cb80 1 -- start start 2024-06-04T04:59:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.744+0000 7fdc2393cb80 1 Processor -- start 2024-06-04T04:59:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.744+0000 7fdc2393cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55830982c920 0x558309838d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.744+0000 7fdc2393cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x558309836020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.744+0000 7fdc2393cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5583098323a0 0x558309836560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.744+0000 7fdc2393cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558309681020 con 0x5583098323a0 2024-06-04T04:59:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.744+0000 7fdc2393cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5583096e2e20 con 0x55830982c920 2024-06-04T04:59:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.744+0000 7fdc2393cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55830983db70 con 0x55830982d2e0 2024-06-04T04:59:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.744+0000 7fdc2393cb80 10 client.0 _ll_register_callbacks cb 0x5583097159d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.744+0000 7fdc1359e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5583098323a0 0x558309836560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.744+0000 7fdc1359e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5583098323a0 0x558309836560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54326/0 (socket says 172.21.15.70:54326) 2024-06-04T04:59:29.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.744+0000 7fdc1359e700 1 -- 172.21.15.70:0/679580752 learned_addr learned my addr 172.21.15.70:0/679580752 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:29.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.744+0000 7fdc1259c700 1 --2- 172.21.15.70:0/679580752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x558309836020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.744+0000 7fdc12d9d700 1 --2- 172.21.15.70:0/679580752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55830982c920 0x558309838d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.745+0000 7fdc1359e700 1 -- 172.21.15.70:0/679580752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 msgr2=0x558309836020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:29.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.745+0000 7fdc1359e700 1 --2- 172.21.15.70:0/679580752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x558309836020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.745+0000 7fdc1359e700 1 -- 172.21.15.70:0/679580752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55830982c920 msgr2=0x558309838d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:29.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.745+0000 7fdc1359e700 1 --2- 172.21.15.70:0/679580752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55830982c920 0x558309838d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.745+0000 7fdc1359e700 1 -- 172.21.15.70:0/679580752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583096ca950 con 0x5583098323a0 2024-06-04T04:59:29.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.745+0000 7fdc1359e700 1 --2- 172.21.15.70:0/679580752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5583098323a0 0x558309836560 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c000f80 tx=0x7fdc0c015b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f0eb43680940ed server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:29.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.745+0000 7fdbf6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:59:29.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.745+0000 7fdbf6ff5700 1 -- 172.21.15.70:0/679580752 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c01dd10 con 0x5583098323a0 2024-06-04T04:59:29.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.745+0000 7fdbf6ff5700 1 -- 172.21.15.70:0/679580752 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fdc0c01f450 con 0x5583098323a0 2024-06-04T04:59:29.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.745+0000 7fdc2393cb80 1 -- 172.21.15.70:0/679580752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc0c004bb0 con 0x5583098323a0 2024-06-04T04:59:29.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.745+0000 7fdbf6ff5700 1 -- 172.21.15.70:0/679580752 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdc0c015c00 con 0x5583098323a0 2024-06-04T04:59:29.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.745+0000 7fdc2393cb80 10 client.16146 fetch_fsmap learned FSMap version 31 2024-06-04T04:59:29.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.745+0000 7fdc2393cb80 1 -- 172.21.15.70:0/679580752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5583097347e0 con 0x5583098323a0 2024-06-04T04:59:29.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.745+0000 7fdbf6ff5700 1 -- 172.21.15.70:0/679580752 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc0c004bb0 con 0x5583098323a0 2024-06-04T04:59:29.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.746+0000 7fdbf6ff5700 1 -- 172.21.15.70:0/679580752 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fdc0c0304a0 con 0x5583098323a0 2024-06-04T04:59:29.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.746+0000 7fdc2393cb80 10 client.16146 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T04:59:29.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.746+0000 7fdc2393cb80 10 client.16146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:59:29.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.746+0000 7fdc2393cb80 20 client.16146 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:29.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.746+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/679580752 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55830984a500 0x55830984c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.746+0000 7fdc2393cb80 4 client.16146 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:59:29.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.746+0000 7fdc2393cb80 1 -- 172.21.15.70:0/679580752 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55830978cd40 con 0x55830984a500 2024-06-04T04:59:29.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.746+0000 7fdc12d9d700 1 --2- 172.21.15.70:0/679580752 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55830984a500 0x55830984c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.746+0000 7fdbf6ff5700 10 client.16146 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:29.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.746+0000 7fdc12d9d700 1 --2- 172.21.15.70:0/679580752 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55830984a500 0x55830984c8f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ac1332d790a66ee server_cookie=76e250e92cd4a230 in_seq=0 out_seq=0 2024-06-04T04:59:29.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.748+0000 7fdbf6ff5700 1 -- 172.21.15.70:0/679580752 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55830978cd40 con 0x55830984a500 2024-06-04T04:59:29.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.748+0000 7fdbf6ff5700 10 client.16146 handle_command_reply: tid=0 2024-06-04T04:59:29.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.748+0000 7fdc2393cb80 1 client.16146 shutdown 2024-06-04T04:59:29.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.748+0000 7fdc2393cb80 1 -- 172.21.15.70:0/679580752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5583098323a0 msgr2=0x558309836560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:29.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.748+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/679580752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5583098323a0 0x558309836560 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c000f80 tx=0x7fdc0c015b50 comp rx=0 tx=0).stop 2024-06-04T04:59:29.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.748+0000 7fdc2393cb80 1 -- 172.21.15.70:0/679580752 shutdown_connections 2024-06-04T04:59:29.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.748+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/679580752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5583098323a0 0x558309836560 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.748+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/679580752 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55830984a500 0x55830984c8f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.748+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/679580752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55830982c920 0x558309838d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.748+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/679580752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x558309836020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.748+0000 7fdc2393cb80 1 -- 172.21.15.70:0/679580752 >> 172.21.15.70:0/679580752 conn(0x5583097582c0 msgr2=0x558309826b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:29.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.748+0000 7fdbf6ff5700 0 client.16146 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:29.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.748+0000 7fdc2393cb80 1 -- 172.21.15.70:0/679580752 shutdown_connections 2024-06-04T04:59:29.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.748+0000 7fdc2393cb80 1 -- 172.21.15.70:0/679580752 wait complete. 2024-06-04T04:59:29.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.748+0000 7fdc2393cb80 20 client.16146 trim_cache size 0 max 16384 2024-06-04T04:59:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.769+0000 7fdc2393cb80 1 Processor -- start 2024-06-04T04:59:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.769+0000 7fdc2393cb80 1 -- start start 2024-06-04T04:59:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.769+0000 7fdc2393cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 0x5583098272f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.769+0000 7fdc2393cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x558309837f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.769+0000 7fdc2393cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583098323a0 0x55830982edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.769+0000 7fdc2393cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141abc50 con 0x55830982c920 2024-06-04T04:59:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.769+0000 7fdc2393cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141ad7f0 con 0x5583098323a0 2024-06-04T04:59:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.769+0000 7fdc2393cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141ac320 con 0x55830982d2e0 2024-06-04T04:59:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.769+0000 7fdc12d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 0x5583098272f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.770+0000 7fdc12d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 0x5583098272f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54338/0 (socket says 172.21.15.70:54338) 2024-06-04T04:59:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.770+0000 7fdc12d9d700 1 -- 172.21.15.70:0/1157216751 learned_addr learned my addr 172.21.15.70:0/1157216751 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.770+0000 7fdc1259c700 1 --2- 172.21.15.70:0/1157216751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x558309837f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.770+0000 7fdc1359e700 1 --2- 172.21.15.70:0/1157216751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583098323a0 0x55830982edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.770+0000 7fdc12d9d700 1 -- 172.21.15.70:0/1157216751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 msgr2=0x558309837f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.770+0000 7fdc12d9d700 1 --2- 172.21.15.70:0/1157216751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x558309837f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.770+0000 7fdc12d9d700 1 -- 172.21.15.70:0/1157216751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583098323a0 msgr2=0x55830982edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.770+0000 7fdc12d9d700 1 --2- 172.21.15.70:0/1157216751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583098323a0 0x55830982edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.770+0000 7fdc12d9d700 1 -- 172.21.15.70:0/1157216751 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558309838960 con 0x55830982c920 2024-06-04T04:59:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.770+0000 7fdc1359e700 1 --2- 172.21.15.70:0/1157216751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583098323a0 0x55830982edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:59:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.770+0000 7fdc12d9d700 1 --2- 172.21.15.70:0/1157216751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 0x5583098272f0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fdc040dff10 tx=0x7fdc040a9940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a201b41db3abdc65 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.770+0000 7fdbf8ff9700 1 -- 172.21.15.70:0/1157216751 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc040b2040 con 0x55830982c920 2024-06-04T04:59:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.770+0000 7fdbf8ff9700 1 -- 172.21.15.70:0/1157216751 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc04010e50 con 0x55830982c920 2024-06-04T04:59:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.770+0000 7fdbf8ff9700 1 -- 172.21.15.70:0/1157216751 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc040a9f20 con 0x55830982c920 2024-06-04T04:59:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.771+0000 7fdc2393cb80 1 -- 172.21.15.70:0/1157216751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 msgr2=0x5583098272f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.771+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/1157216751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 0x5583098272f0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fdc040dff10 tx=0x7fdc040a9940 comp rx=0 tx=0).stop 2024-06-04T04:59:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.771+0000 7fdc2393cb80 1 -- 172.21.15.70:0/1157216751 shutdown_connections 2024-06-04T04:59:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.771+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/1157216751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583098323a0 0x55830982edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.771+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/1157216751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 0x5583098272f0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.771+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/1157216751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x558309837f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.771+0000 7fdc2393cb80 1 -- 172.21.15.70:0/1157216751 >> 172.21.15.70:0/1157216751 conn(0x5583097582c0 msgr2=0x558309837850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.771+0000 7fdc2393cb80 1 -- 172.21.15.70:0/1157216751 shutdown_connections 2024-06-04T04:59:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.771+0000 7fdc2393cb80 1 -- 172.21.15.70:0/1157216751 wait complete. 2024-06-04T04:59:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.771+0000 7fdc2393cb80 1 -- start start 2024-06-04T04:59:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc2393cb80 1 Processor -- start 2024-06-04T04:59:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc2393cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 0x55830983ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc2393cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x55830983e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc2393cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583098323a0 0x55830983e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc2393cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141ad0c0 con 0x55830982c920 2024-06-04T04:59:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc2393cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141ac9f0 con 0x5583098323a0 2024-06-04T04:59:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc2393cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141a9c10 con 0x55830982d2e0 2024-06-04T04:59:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc2393cb80 10 client.0 _ll_register_callbacks cb 0x5583097159d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc12d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 0x55830983ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc12d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 0x55830983ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54354/0 (socket says 172.21.15.70:54354) 2024-06-04T04:59:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc12d9d700 1 -- 172.21.15.70:0/404578708 learned_addr learned my addr 172.21.15.70:0/404578708 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc1359e700 1 --2- 172.21.15.70:0/404578708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583098323a0 0x55830983e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc1259c700 1 --2- 172.21.15.70:0/404578708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x55830983e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc12d9d700 1 -- 172.21.15.70:0/404578708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 msgr2=0x55830983e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc12d9d700 1 --2- 172.21.15.70:0/404578708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x55830983e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc12d9d700 1 -- 172.21.15.70:0/404578708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583098323a0 msgr2=0x55830983e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc12d9d700 1 --2- 172.21.15.70:0/404578708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583098323a0 0x55830983e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc12d9d700 1 -- 172.21.15.70:0/404578708 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583098b4dc0 con 0x55830982c920 2024-06-04T04:59:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.772+0000 7fdc1259c700 1 --2- 172.21.15.70:0/404578708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x55830983e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T04:59:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.773+0000 7fdc12d9d700 1 --2- 172.21.15.70:0/404578708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 0x55830983ddb0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fdc040e0430 tx=0x7fdc040a9810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90b6e1276a16abe7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.773+0000 7fdbf6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:59:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.773+0000 7fdbf6ff5700 1 -- 172.21.15.70:0/404578708 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0408e020 con 0x55830982c920 2024-06-04T04:59:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.773+0000 7fdbf6ff5700 1 -- 172.21.15.70:0/404578708 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fdc040a9eb0 con 0x55830982c920 2024-06-04T04:59:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.773+0000 7fdc2393cb80 1 -- 172.21.15.70:0/404578708 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc04010e50 con 0x55830982c920 2024-06-04T04:59:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.773+0000 7fdbf6ff5700 1 -- 172.21.15.70:0/404578708 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdc040b28a0 con 0x55830982c920 2024-06-04T04:59:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.773+0000 7fdc2393cb80 10 client.16152 fetch_fsmap learned FSMap version 31 2024-06-04T04:59:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.773+0000 7fdc2393cb80 1 -- 172.21.15.70:0/404578708 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5583098b6080 con 0x55830982c920 2024-06-04T04:59:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.773+0000 7fdbf6ff5700 1 -- 172.21.15.70:0/404578708 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc04010e50 con 0x55830982c920 2024-06-04T04:59:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.773+0000 7fdbf6ff5700 1 -- 172.21.15.70:0/404578708 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fdc04010820 con 0x55830982c920 2024-06-04T04:59:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.773+0000 7fdc2393cb80 10 client.16152 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T04:59:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.773+0000 7fdc2393cb80 10 client.16152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:59:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.773+0000 7fdc2393cb80 20 client.16152 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.773+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/404578708 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5583098b66c0 0x5583098b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.773+0000 7fdc2393cb80 4 client.16152 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T04:59:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.773+0000 7fdc2393cb80 1 -- 172.21.15.70:0/404578708 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558309786520 con 0x5583098b66c0 2024-06-04T04:59:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.774+0000 7fdc1259c700 1 --2- 172.21.15.70:0/404578708 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5583098b66c0 0x5583098b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.774+0000 7fdbf6ff5700 10 client.16152 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.774+0000 7fdc1259c700 1 --2- 172.21.15.70:0/404578708 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5583098b66c0 0x5583098b8ae0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1cfc3e3174e7bc9 server_cookie=86bd7cd269a8221a in_seq=0 out_seq=0 2024-06-04T04:59:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.775+0000 7fdbf6ff5700 1 -- 172.21.15.70:0/404578708 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558309786520 con 0x5583098b66c0 2024-06-04T04:59:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.775+0000 7fdbf6ff5700 10 client.16152 handle_command_reply: tid=0 2024-06-04T04:59:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.775+0000 7fdc2393cb80 1 client.16152 shutdown 2024-06-04T04:59:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.775+0000 7fdc2393cb80 1 -- 172.21.15.70:0/404578708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 msgr2=0x55830983ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.775+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/404578708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 0x55830983ddb0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fdc040e0430 tx=0x7fdc040a9810 comp rx=0 tx=0).stop 2024-06-04T04:59:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.775+0000 7fdc2393cb80 1 -- 172.21.15.70:0/404578708 shutdown_connections 2024-06-04T04:59:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.775+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/404578708 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5583098b66c0 0x5583098b8ae0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.775+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/404578708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583098323a0 0x55830983e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.775+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/404578708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55830982c920 0x55830983ddb0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.775+0000 7fdc2393cb80 1 --2- 172.21.15.70:0/404578708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55830982d2e0 0x55830983e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.775+0000 7fdc2393cb80 1 -- 172.21.15.70:0/404578708 >> 172.21.15.70:0/404578708 conn(0x5583097582c0 msgr2=0x55830982c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.775+0000 7fdbf6ff5700 0 client.16152 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.775+0000 7fdc2393cb80 1 -- 172.21.15.70:0/404578708 shutdown_connections 2024-06-04T04:59:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.775+0000 7fdc2393cb80 1 -- 172.21.15.70:0/404578708 wait complete. 2024-06-04T04:59:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.775+0000 7fdc2393cb80 20 client.16152 trim_cache size 0 max 16384 2024-06-04T04:59:29.782 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:59:29.783 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T04:59:29.783 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "b225b7fc-3902-4f20-9556-29160bb46ab4", 2024-06-04T04:59:29.783 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T04:59:29.783 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:59:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.776+0000 7fdc190a7700 1 -- 172.21.15.70:0/1593485802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdbfc0b29f0 msgr2=0x7fdbfc0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.776+0000 7fdc190a7700 1 --2- 172.21.15.70:0/1593485802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdbfc0b29f0 0x7fdbfc0b4eb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdc14079b30 tx=0x7fdc0801a040 comp rx=0 tx=0).stop 2024-06-04T04:59:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.776+0000 7fdc190a7700 1 -- 172.21.15.70:0/1593485802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1406b220 msgr2=0x7fdc1419f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.776+0000 7fdc190a7700 1 --2- 172.21.15.70:0/1593485802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1406b220 0x7fdc1419f790 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400e9a0 tx=0x7fdc0400ecb0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.776+0000 7fdc190a7700 1 -- 172.21.15.70:0/1593485802 shutdown_connections 2024-06-04T04:59:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.776+0000 7fdc190a7700 1 --2- 172.21.15.70:0/1593485802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdbfc0b29f0 0x7fdbfc0b4eb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.776+0000 7fdc190a7700 1 --2- 172.21.15.70:0/1593485802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc141095b0 0x7fdc14199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.776+0000 7fdc190a7700 1 --2- 172.21.15.70:0/1593485802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1406b220 0x7fdc1419f790 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.776+0000 7fdc190a7700 1 --2- 172.21.15.70:0/1593485802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc1406bbe0 0x7fdc1419fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.776+0000 7fdc190a7700 1 -- 172.21.15.70:0/1593485802 >> 172.21.15.70:0/1593485802 conn(0x7fdc140793f0 msgr2=0x7fdc141012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.776+0000 7fdc190a7700 1 -- 172.21.15.70:0/1593485802 shutdown_connections 2024-06-04T04:59:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:29.776+0000 7fdc190a7700 1 -- 172.21.15.70:0/1593485802 wait complete. 2024-06-04T04:59:30.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:30 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T04:59:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T04:59:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T04:59:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T04:59:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:30.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:30 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T04:59:30.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:30.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T04:59:30.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T04:59:30.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:30.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T04:59:30.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:30.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:30 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T04:59:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T04:59:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T04:59:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T04:59:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:30.702 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T04:59:30.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 14518 4.22 MB/sec warmup 23 sec latency 15.991 ms 2024-06-04T04:59:30.979 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T04:59:31.355 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:31 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T04:59:31.355 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:31 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:31.355 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:31 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T04:59:31.355 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:31 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:31.355 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:31 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T04:59:31.355 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:31 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:31.355 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:31 smithi070 ceph-mon[32631]: pgmap v73: 97 pgs: 97 active+clean; 30 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-06-04T04:59:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:31 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T04:59:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:31 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:31 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T04:59:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:31 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:31 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T04:59:31.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:31 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:31.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:31 smithi098 ceph-mon[35569]: pgmap v73: 97 pgs: 97 active+clean; 30 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-06-04T04:59:31.587 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:31 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T04:59:31.587 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:31 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:31.587 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:31 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T04:59:31.587 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:31 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:31.587 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:31 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T04:59:31.587 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:31 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T04:59:31.588 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:31 smithi112 ceph-mon[36453]: pgmap v73: 97 pgs: 97 active+clean; 30 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-06-04T04:59:31.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 14805 4.06 MB/sec warmup 24 sec latency 12.142 ms 2024-06-04T04:59:32.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 15321 3.98 MB/sec warmup 25 sec latency 43.119 ms 2024-06-04T04:59:32.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.856+0000 7f3402a64700 1 -- 172.21.15.70:0/2863390408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc1041a0 msgr2=0x7f33fc1081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:32.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.856+0000 7f3402a64700 1 --2- 172.21.15.70:0/2863390408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc1041a0 0x7f33fc1081c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f33e80099c0 tx=0x7f33e8009cd0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.856+0000 7f3402a64700 1 -- 172.21.15.70:0/2863390408 shutdown_connections 2024-06-04T04:59:32.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.856+0000 7f3402a64700 1 --2- 172.21.15.70:0/2863390408 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33fc1088f0 0x7f33fc110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.856+0000 7f3402a64700 1 --2- 172.21.15.70:0/2863390408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc1037f0 0x7f33fc103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.856+0000 7f3402a64700 1 --2- 172.21.15.70:0/2863390408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc1041a0 0x7f33fc1081c0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.856+0000 7f3402a64700 1 -- 172.21.15.70:0/2863390408 >> 172.21.15.70:0/2863390408 conn(0x7f33fc07a6a0 msgr2=0x7f33fc07aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:32.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.856+0000 7f3402a64700 1 -- 172.21.15.70:0/2863390408 shutdown_connections 2024-06-04T04:59:32.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.856+0000 7f3402a64700 1 -- 172.21.15.70:0/2863390408 wait complete. 2024-06-04T04:59:32.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.857+0000 7f3402a64700 1 Processor -- start 2024-06-04T04:59:32.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.857+0000 7f3402a64700 1 -- start start 2024-06-04T04:59:32.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.857+0000 7f3402a64700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc1037f0 0x7f33fc075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.857+0000 7f3402a64700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33fc1041a0 0x7f33fc075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.857+0000 7f3402a64700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc1088f0 0x7f33fc0762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.857+0000 7f3402a64700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc113580 con 0x7f33fc1037f0 2024-06-04T04:59:32.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.857+0000 7f3402a64700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc113400 con 0x7f33fc1088f0 2024-06-04T04:59:32.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.857+0000 7f3402a64700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc113700 con 0x7f33fc1041a0 2024-06-04T04:59:32.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.857+0000 7f33fbfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33fc1041a0 0x7f33fc075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.857+0000 7f3400800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc1037f0 0x7f33fc075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.857+0000 7f3401001700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc1088f0 0x7f33fc0762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.857+0000 7f33fbfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33fc1041a0 0x7f33fc075d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46768/0 (socket says 172.21.15.70:46768) 2024-06-04T04:59:32.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.857+0000 7f33fbfff700 1 -- 172.21.15.70:0/1160052812 learned_addr learned my addr 172.21.15.70:0/1160052812 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:32.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.857+0000 7f3401001700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc1088f0 0x7f33fc0762c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46404/0 (socket says 172.21.15.70:46404) 2024-06-04T04:59:32.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.858+0000 7f33fbfff700 1 -- 172.21.15.70:0/1160052812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc1088f0 msgr2=0x7f33fc0762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:32.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.858+0000 7f33fbfff700 1 --2- 172.21.15.70:0/1160052812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc1088f0 0x7f33fc0762c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.858+0000 7f33fbfff700 1 -- 172.21.15.70:0/1160052812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc1037f0 msgr2=0x7f33fc075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:32.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.858+0000 7f33fbfff700 1 --2- 172.21.15.70:0/1160052812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc1037f0 0x7f33fc075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.858+0000 7f33fbfff700 1 -- 172.21.15.70:0/1160052812 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33fc1a8000 con 0x7f33fc1041a0 2024-06-04T04:59:32.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.858+0000 7f3401001700 1 --2- 172.21.15.70:0/1160052812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc1088f0 0x7f33fc0762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:32.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.858+0000 7f3400800700 1 --2- 172.21.15.70:0/1160052812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc1037f0 0x7f33fc075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:32.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.858+0000 7f33fbfff700 1 --2- 172.21.15.70:0/1160052812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33fc1041a0 0x7f33fc075d80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f33e8006010 tx=0x7f33e80045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:32.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.858+0000 7f33f9ffb700 1 -- 172.21.15.70:0/1160052812 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e8026070 con 0x7f33fc1041a0 2024-06-04T04:59:32.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.858+0000 7f3402a64700 1 -- 172.21.15.70:0/1160052812 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33fc1a8290 con 0x7f33fc1041a0 2024-06-04T04:59:32.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.858+0000 7f33f9ffb700 1 -- 172.21.15.70:0/1160052812 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f33e800f460 con 0x7f33fc1041a0 2024-06-04T04:59:32.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.858+0000 7f33f9ffb700 1 -- 172.21.15.70:0/1160052812 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e8009e60 con 0x7f33fc1041a0 2024-06-04T04:59:32.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.858+0000 7f3402a64700 1 -- 172.21.15.70:0/1160052812 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33fc1a8740 con 0x7f33fc1041a0 2024-06-04T04:59:32.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.861+0000 7f33f9ffb700 1 -- 172.21.15.70:0/1160052812 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f33e8029070 con 0x7f33fc1041a0 2024-06-04T04:59:32.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.862+0000 7f33f9ffb700 1 --2- 172.21.15.70:0/1160052812 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33ec0b2a70 0x7f33ec0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.862+0000 7f340d2f9b80 1 Processor -- start 2024-06-04T04:59:32.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.862+0000 7f340d2f9b80 1 -- start start 2024-06-04T04:59:32.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.862+0000 7f3400800700 1 --2- 172.21.15.70:0/1160052812 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33ec0b2a70 0x7f33ec0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.862+0000 7f33f9ffb700 1 -- 172.21.15.70:0/1160052812 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f33e80e2dd0 con 0x7f33fc1041a0 2024-06-04T04:59:32.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.862+0000 7f340d2f9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e47860 0x564f41e47c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.862+0000 7f340d2f9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e4b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.862+0000 7f340d2f9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e4bf60 0x564f41e4c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.862+0000 7f340d2f9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564f41da6c90 con 0x564f41e48210 2024-06-04T04:59:32.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.862+0000 7f340d2f9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564f41d96830 con 0x564f41e47860 2024-06-04T04:59:32.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.862+0000 7f340d2f9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564f41d9a870 con 0x564f41e4bf60 2024-06-04T04:59:32.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.863+0000 7f3401001700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e4bf60 0x564f41e4c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.863+0000 7f3400800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e47860 0x564f41e47c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.863+0000 7f33fbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e4b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.863+0000 7f3401001700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e4bf60 0x564f41e4c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46784/0 (socket says 172.21.15.70:46784) 2024-06-04T04:59:32.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.863+0000 7f3400800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e47860 0x564f41e47c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46408/0 (socket says 172.21.15.70:46408) 2024-06-04T04:59:32.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.863+0000 7f3401001700 1 -- 172.21.15.70:0/3023007132 learned_addr learned my addr 172.21.15.70:0/3023007132 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:32.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.863+0000 7f3400800700 1 --2- 172.21.15.70:0/1160052812 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33ec0b2a70 0x7f33ec0b4f30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f33f0011550 tx=0x7f33f0009f90 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:32.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.863+0000 7f3401001700 1 -- 172.21.15.70:0/3023007132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e47860 msgr2=0x564f41e47c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:32.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.863+0000 7f3401001700 1 --2- 172.21.15.70:0/3023007132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e47860 0x564f41e47c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.863+0000 7f3401001700 1 -- 172.21.15.70:0/3023007132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 msgr2=0x564f41e4b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:32.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.863+0000 7f3401001700 1 --2- 172.21.15.70:0/3023007132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e4b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.863+0000 7f3401001700 1 -- 172.21.15.70:0/3023007132 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f41ce68d0 con 0x564f41e4bf60 2024-06-04T04:59:32.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.863+0000 7f33fbfff700 1 --2- 172.21.15.70:0/3023007132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e4b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:32.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.863+0000 7f3401001700 1 --2- 172.21.15.70:0/3023007132 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e4bf60 0x564f41e4c3a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f33f4007740 tx=0x7f33f4009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34349bff03094d8f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:32.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.863+0000 7f33e6ffd700 1 -- 172.21.15.70:0/3023007132 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f4019020 con 0x564f41e4bf60 2024-06-04T04:59:32.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.863+0000 7f33e6ffd700 1 -- 172.21.15.70:0/3023007132 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f33f4015ce0 con 0x564f41e4bf60 2024-06-04T04:59:32.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.863+0000 7f33e6ffd700 1 -- 172.21.15.70:0/3023007132 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f4014cc0 con 0x564f41e4bf60 2024-06-04T04:59:32.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.864+0000 7f340d2f9b80 1 -- 172.21.15.70:0/3023007132 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e4bf60 msgr2=0x564f41e4c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:32.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.864+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/3023007132 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e4bf60 0x564f41e4c3a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f33f4007740 tx=0x7f33f4009f90 comp rx=0 tx=0).stop 2024-06-04T04:59:32.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.864+0000 7f340d2f9b80 1 -- 172.21.15.70:0/3023007132 shutdown_connections 2024-06-04T04:59:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.864+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/3023007132 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e4bf60 0x564f41e4c3a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.864+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/3023007132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e47860 0x564f41e47c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.864+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/3023007132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e4b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.864+0000 7f340d2f9b80 1 -- 172.21.15.70:0/3023007132 >> 172.21.15.70:0/3023007132 conn(0x564f41d75080 msgr2=0x564f41e41e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.864+0000 7f340d2f9b80 1 -- 172.21.15.70:0/3023007132 shutdown_connections 2024-06-04T04:59:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.864+0000 7f340d2f9b80 1 -- 172.21.15.70:0/3023007132 wait complete. 2024-06-04T04:59:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.865+0000 7f340d2f9b80 1 -- start start 2024-06-04T04:59:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.865+0000 7f340d2f9b80 1 Processor -- start 2024-06-04T04:59:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.865+0000 7f340d2f9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e47860 0x564f41e53430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.865+0000 7f340d2f9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e50730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.865+0000 7f340d2f9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e4bf60 0x564f41e50c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.865+0000 7f340d2f9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564f41c9cfa0 con 0x564f41e48210 2024-06-04T04:59:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.865+0000 7f340d2f9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564f41d387c0 con 0x564f41e4bf60 2024-06-04T04:59:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.865+0000 7f340d2f9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564f41e54120 con 0x564f41e47860 2024-06-04T04:59:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.865+0000 7f340d2f9b80 10 client.0 _ll_register_callbacks cb 0x564f41d31920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.865+0000 7f3400800700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e47860 0x564f41e53430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.865+0000 7f3400800700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e47860 0x564f41e53430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46800/0 (socket says 172.21.15.70:46800) 2024-06-04T04:59:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.865+0000 7f3400800700 1 -- 172.21.15.70:0/2709814767 learned_addr learned my addr 172.21.15.70:0/2709814767 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.866+0000 7f3401001700 1 --2- 172.21.15.70:0/2709814767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e4bf60 0x564f41e50c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.866+0000 7f33fbfff700 1 --2- 172.21.15.70:0/2709814767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e50730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.866+0000 7f3400800700 1 -- 172.21.15.70:0/2709814767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e4bf60 msgr2=0x564f41e50c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.866+0000 7f3400800700 1 --2- 172.21.15.70:0/2709814767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e4bf60 0x564f41e50c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.866+0000 7f3400800700 1 -- 172.21.15.70:0/2709814767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 msgr2=0x564f41e50730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.866+0000 7f3400800700 1 --2- 172.21.15.70:0/2709814767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e50730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.866+0000 7f3400800700 1 -- 172.21.15.70:0/2709814767 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f41d35690 con 0x564f41e47860 2024-06-04T04:59:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.866+0000 7f3400800700 1 --2- 172.21.15.70:0/2709814767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e47860 0x564f41e53430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f33f000fb50 tx=0x7f33f00047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5cdc7105a5cca8a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:32.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.866+0000 7f33dbfff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:59:32.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.866+0000 7f33dbfff700 1 -- 172.21.15.70:0/2709814767 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f0017040 con 0x564f41e47860 2024-06-04T04:59:32.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.866+0000 7f33dbfff700 1 -- 172.21.15.70:0/2709814767 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f33f001eb10 con 0x564f41e47860 2024-06-04T04:59:32.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.866+0000 7f340d2f9b80 1 -- 172.21.15.70:0/2709814767 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33f4015ce0 con 0x564f41e47860 2024-06-04T04:59:32.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.866+0000 7f33dbfff700 1 -- 172.21.15.70:0/2709814767 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f33f000fe60 con 0x564f41e47860 2024-06-04T04:59:32.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.866+0000 7f340d2f9b80 10 client.25627 fetch_fsmap learned FSMap version 31 2024-06-04T04:59:32.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.866+0000 7f340d2f9b80 1 -- 172.21.15.70:0/2709814767 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564f41e5e900 con 0x564f41e47860 2024-06-04T04:59:32.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.867+0000 7f33dbfff700 1 -- 172.21.15.70:0/2709814767 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f33f4015ce0 con 0x564f41e47860 2024-06-04T04:59:32.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.867+0000 7f33dbfff700 1 -- 172.21.15.70:0/2709814767 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f33f0003ad0 con 0x564f41e47860 2024-06-04T04:59:32.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.867+0000 7f340d2f9b80 10 client.25627 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T04:59:32.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.867+0000 7f340d2f9b80 10 client.25627 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:59:32.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.867+0000 7f340d2f9b80 20 client.25627 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:32.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.867+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/2709814767 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564f41e60f90 0x564f41e633d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.867+0000 7f340d2f9b80 4 client.25627 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:59:32.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.867+0000 7f340d2f9b80 1 -- 172.21.15.70:0/2709814767 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f41e139e0 con 0x564f41e60f90 2024-06-04T04:59:32.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.867+0000 7f33fbfff700 1 --2- 172.21.15.70:0/2709814767 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564f41e60f90 0x564f41e633d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.867+0000 7f33fbfff700 1 --2- 172.21.15.70:0/2709814767 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564f41e60f90 0x564f41e633d0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dafdf99344241042 server_cookie=69480fdfcef5a957 in_seq=0 out_seq=0 2024-06-04T04:59:32.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.867+0000 7f33dbfff700 10 client.25627 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:32.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.868+0000 7f33dbfff700 1 -- 172.21.15.70:0/2709814767 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564f41e139e0 con 0x564f41e60f90 2024-06-04T04:59:32.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.868+0000 7f33dbfff700 10 client.25627 handle_command_reply: tid=0 2024-06-04T04:59:32.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.868+0000 7f340d2f9b80 1 client.25627 shutdown 2024-06-04T04:59:32.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.868+0000 7f340d2f9b80 1 -- 172.21.15.70:0/2709814767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e47860 msgr2=0x564f41e53430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:32.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.868+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/2709814767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e47860 0x564f41e53430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f33f000fb50 tx=0x7f33f00047c0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.868+0000 7f340d2f9b80 1 -- 172.21.15.70:0/2709814767 shutdown_connections 2024-06-04T04:59:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.868+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/2709814767 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564f41e60f90 0x564f41e633d0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.868+0000 7f33dbfff700 0 client.25627 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.868+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/2709814767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e4bf60 0x564f41e50c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.868+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/2709814767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e47860 0x564f41e53430 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.868+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/2709814767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e50730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.868+0000 7f340d2f9b80 1 -- 172.21.15.70:0/2709814767 >> 172.21.15.70:0/2709814767 conn(0x564f41d75080 msgr2=0x564f41e4d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.869+0000 7f340d2f9b80 1 -- 172.21.15.70:0/2709814767 shutdown_connections 2024-06-04T04:59:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.869+0000 7f340d2f9b80 1 -- 172.21.15.70:0/2709814767 wait complete. 2024-06-04T04:59:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.869+0000 7f340d2f9b80 20 client.25627 trim_cache size 0 max 16384 2024-06-04T04:59:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.889+0000 7f340d2f9b80 1 Processor -- start 2024-06-04T04:59:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.889+0000 7f340d2f9b80 1 -- start start 2024-06-04T04:59:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f340d2f9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e40ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f340d2f9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e4bf60 0x564f41e52670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f340d2f9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e418f0 0x564f41e68ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f340d2f9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc1abb00 con 0x564f41e48210 2024-06-04T04:59:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f340d2f9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc1ad6a0 con 0x564f41e418f0 2024-06-04T04:59:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f340d2f9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc1ac1d0 con 0x564f41e4bf60 2024-06-04T04:59:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f3400800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e40ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f33fbfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e4bf60 0x564f41e52670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f3400800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e40ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54386/0 (socket says 172.21.15.70:54386) 2024-06-04T04:59:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f3400800700 1 -- 172.21.15.70:0/87252327 learned_addr learned my addr 172.21.15.70:0/87252327 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f3401001700 1 --2- 172.21.15.70:0/87252327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e418f0 0x564f41e68ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f33fbfff700 1 -- 172.21.15.70:0/87252327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e418f0 msgr2=0x564f41e68ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f33fbfff700 1 --2- 172.21.15.70:0/87252327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e418f0 0x564f41e68ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f33fbfff700 1 -- 172.21.15.70:0/87252327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 msgr2=0x564f41e40ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:32.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f33fbfff700 1 --2- 172.21.15.70:0/87252327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e40ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f33fbfff700 1 -- 172.21.15.70:0/87252327 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f41e63a50 con 0x564f41e4bf60 2024-06-04T04:59:32.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f33fbfff700 1 --2- 172.21.15.70:0/87252327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e4bf60 0x564f41e52670 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f33e80e04c0 tx=0x7f33e8003700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec2d2fa7e47682f5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:32.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f33e67fc700 1 -- 172.21.15.70:0/87252327 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e8090070 con 0x564f41e4bf60 2024-06-04T04:59:32.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f33e67fc700 1 -- 172.21.15.70:0/87252327 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f33e80aa7e0 con 0x564f41e4bf60 2024-06-04T04:59:32.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.890+0000 7f33e67fc700 1 -- 172.21.15.70:0/87252327 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e80e0740 con 0x564f41e4bf60 2024-06-04T04:59:32.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.891+0000 7f340d2f9b80 1 -- 172.21.15.70:0/87252327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e4bf60 msgr2=0x564f41e52670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:32.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.891+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/87252327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e4bf60 0x564f41e52670 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f33e80e04c0 tx=0x7f33e8003700 comp rx=0 tx=0).stop 2024-06-04T04:59:32.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.891+0000 7f340d2f9b80 1 -- 172.21.15.70:0/87252327 shutdown_connections 2024-06-04T04:59:32.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.891+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/87252327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e418f0 0x564f41e68ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.891+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/87252327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e40ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.891+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/87252327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e4bf60 0x564f41e52670 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.891+0000 7f340d2f9b80 1 -- 172.21.15.70:0/87252327 >> 172.21.15.70:0/87252327 conn(0x564f41d75080 msgr2=0x564f41e51f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:32.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.891+0000 7f340d2f9b80 1 -- 172.21.15.70:0/87252327 shutdown_connections 2024-06-04T04:59:32.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.891+0000 7f340d2f9b80 1 -- 172.21.15.70:0/87252327 wait complete. 2024-06-04T04:59:32.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.892+0000 7f340d2f9b80 1 -- start start 2024-06-04T04:59:32.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.892+0000 7f340d2f9b80 1 Processor -- start 2024-06-04T04:59:32.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.892+0000 7f340d2f9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e418f0 0x564f41e572a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.892+0000 7f340d2f9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e577e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.892+0000 7f340d2f9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e4bf60 0x564f41e57d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.892+0000 7f340d2f9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc1acf70 con 0x564f41e48210 2024-06-04T04:59:32.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.892+0000 7f340d2f9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc1ac8a0 con 0x564f41e4bf60 2024-06-04T04:59:32.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.892+0000 7f340d2f9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc19da50 con 0x564f41e418f0 2024-06-04T04:59:32.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.892+0000 7f340d2f9b80 10 client.0 _ll_register_callbacks cb 0x564f41d31920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:32.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.892+0000 7f33fbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.892+0000 7f3400800700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e418f0 0x564f41e572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.892+0000 7f3400800700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e418f0 0x564f41e572a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46822/0 (socket says 172.21.15.70:46822) 2024-06-04T04:59:32.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.892+0000 7f3400800700 1 -- 172.21.15.70:0/1835354423 learned_addr learned my addr 172.21.15.70:0/1835354423 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:32.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.892+0000 7f3401001700 1 --2- 172.21.15.70:0/1835354423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e4bf60 0x564f41e57d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.893+0000 7f3400800700 1 -- 172.21.15.70:0/1835354423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e4bf60 msgr2=0x564f41e57d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:32.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.893+0000 7f3400800700 1 --2- 172.21.15.70:0/1835354423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e4bf60 0x564f41e57d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.893+0000 7f3400800700 1 -- 172.21.15.70:0/1835354423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 msgr2=0x564f41e577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:32.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.893+0000 7f3400800700 1 --2- 172.21.15.70:0/1835354423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.893+0000 7f3400800700 1 -- 172.21.15.70:0/1835354423 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f41ecad40 con 0x564f41e418f0 2024-06-04T04:59:32.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.893+0000 7f33fbfff700 1 --2- 172.21.15.70:0/1835354423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:32.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.893+0000 7f3400800700 1 --2- 172.21.15.70:0/1835354423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e418f0 0x564f41e572a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f33f000fb50 tx=0x7f33f000f430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=508b6fab227421a5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:32.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.893+0000 7f33dbfff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:59:32.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.893+0000 7f33dbfff700 1 -- 172.21.15.70:0/1835354423 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f0034070 con 0x564f41e418f0 2024-06-04T04:59:32.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.893+0000 7f340d2f9b80 1 -- 172.21.15.70:0/1835354423 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33e80aa7e0 con 0x564f41e418f0 2024-06-04T04:59:32.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.893+0000 7f33dbfff700 1 -- 172.21.15.70:0/1835354423 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f33f0003680 con 0x564f41e418f0 2024-06-04T04:59:32.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.893+0000 7f33dbfff700 1 -- 172.21.15.70:0/1835354423 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f33f0017040 con 0x564f41e418f0 2024-06-04T04:59:32.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.893+0000 7f340d2f9b80 10 client.25633 fetch_fsmap learned FSMap version 31 2024-06-04T04:59:32.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.893+0000 7f340d2f9b80 1 -- 172.21.15.70:0/1835354423 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564f41ecc030 con 0x564f41e418f0 2024-06-04T04:59:32.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.894+0000 7f33dbfff700 1 -- 172.21.15.70:0/1835354423 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f33e80aa7e0 con 0x564f41e418f0 2024-06-04T04:59:32.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.894+0000 7f33dbfff700 1 -- 172.21.15.70:0/1835354423 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f33f0004000 con 0x564f41e418f0 2024-06-04T04:59:32.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.894+0000 7f340d2f9b80 10 client.25633 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T04:59:32.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.894+0000 7f340d2f9b80 10 client.25633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T04:59:32.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.894+0000 7f340d2f9b80 20 client.25633 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:32.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.894+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/1835354423 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564f41ecc7f0 0x564f41ecec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.894+0000 7f340d2f9b80 4 client.25633 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T04:59:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.894+0000 7f340d2f9b80 1 -- 172.21.15.70:0/1835354423 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564f41da8be0 con 0x564f41ecc7f0 2024-06-04T04:59:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.894+0000 7f33fbfff700 1 --2- 172.21.15.70:0/1835354423 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564f41ecc7f0 0x564f41ecec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.894+0000 7f33fbfff700 1 --2- 172.21.15.70:0/1835354423 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564f41ecc7f0 0x564f41ecec10 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9ae86c01b6a92dd server_cookie=92dc6da93fd7d1af in_seq=0 out_seq=0 2024-06-04T04:59:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.894+0000 7f33dbfff700 10 client.25633 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.895+0000 7f33dbfff700 1 -- 172.21.15.70:0/1835354423 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564f41da8be0 con 0x564f41ecc7f0 2024-06-04T04:59:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.895+0000 7f33dbfff700 10 client.25633 handle_command_reply: tid=0 2024-06-04T04:59:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.895+0000 7f340d2f9b80 1 client.25633 shutdown 2024-06-04T04:59:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.895+0000 7f340d2f9b80 1 -- 172.21.15.70:0/1835354423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e418f0 msgr2=0x564f41e572a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.895+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/1835354423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e418f0 0x564f41e572a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f33f000fb50 tx=0x7f33f000f430 comp rx=0 tx=0).stop 2024-06-04T04:59:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.895+0000 7f340d2f9b80 1 -- 172.21.15.70:0/1835354423 shutdown_connections 2024-06-04T04:59:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.895+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/1835354423 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564f41ecc7f0 0x564f41ecec10 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.895+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/1835354423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f41e4bf60 0x564f41e57d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.895+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/1835354423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f41e418f0 0x564f41e572a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.895+0000 7f33dbfff700 0 client.25633 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:32.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.895+0000 7f340d2f9b80 1 --2- 172.21.15.70:0/1835354423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564f41e48210 0x564f41e577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.895+0000 7f340d2f9b80 1 -- 172.21.15.70:0/1835354423 >> 172.21.15.70:0/1835354423 conn(0x564f41d75080 msgr2=0x564f41e46f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:32.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.895+0000 7f340d2f9b80 1 -- 172.21.15.70:0/1835354423 shutdown_connections 2024-06-04T04:59:32.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.895+0000 7f340d2f9b80 1 -- 172.21.15.70:0/1835354423 wait complete. 2024-06-04T04:59:32.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.895+0000 7f340d2f9b80 20 client.25633 trim_cache size 0 max 16384 2024-06-04T04:59:32.903 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T04:59:32.903 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T04:59:32.903 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T04:59:32.904 INFO:teuthology.orchestra.run.smithi070.stdout: "b225b7fc-3902-4f20-9556-29160bb46ab4": { 2024-06-04T04:59:32.904 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T04:59:32.904 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "b225b7fc-3902-4f20-9556-29160bb46ab4", 2024-06-04T04:59:32.904 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T04:59:32.904 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T04:59:32.904 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T04:59:32.904 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T04:59:32.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.896+0000 7f3402a64700 1 -- 172.21.15.70:0/1160052812 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33ec0b2a70 msgr2=0x7f33ec0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:32.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.896+0000 7f3402a64700 1 --2- 172.21.15.70:0/1160052812 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33ec0b2a70 0x7f33ec0b4f30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f33f0011550 tx=0x7f33f0009f90 comp rx=0 tx=0).stop 2024-06-04T04:59:32.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.896+0000 7f3402a64700 1 -- 172.21.15.70:0/1160052812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33fc1041a0 msgr2=0x7f33fc075d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:32.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.896+0000 7f3402a64700 1 --2- 172.21.15.70:0/1160052812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33fc1041a0 0x7f33fc075d80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f33e8006010 tx=0x7f33e80045e0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.896+0000 7f3402a64700 1 -- 172.21.15.70:0/1160052812 shutdown_connections 2024-06-04T04:59:32.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.896+0000 7f3402a64700 1 --2- 172.21.15.70:0/1160052812 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33ec0b2a70 0x7f33ec0b4f30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.896+0000 7f3402a64700 1 --2- 172.21.15.70:0/1160052812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc1088f0 0x7f33fc0762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.896+0000 7f3402a64700 1 --2- 172.21.15.70:0/1160052812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc1037f0 0x7f33fc075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.896+0000 7f3402a64700 1 --2- 172.21.15.70:0/1160052812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33fc1041a0 0x7f33fc075d80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:32.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.896+0000 7f3402a64700 1 -- 172.21.15.70:0/1160052812 >> 172.21.15.70:0/1160052812 conn(0x7f33fc07a6a0 msgr2=0x7f33fc1019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:32.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.896+0000 7f3402a64700 1 -- 172.21.15.70:0/1160052812 shutdown_connections 2024-06-04T04:59:32.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T04:59:32.896+0000 7f3402a64700 1 -- 172.21.15.70:0/1160052812 wait complete. 2024-06-04T04:59:33.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:33 smithi098 ceph-mon[35569]: pgmap v74: 97 pgs: 97 active+clean; 31 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 1.9 MiB/s wr, 193 op/s 2024-06-04T04:59:33.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:33 smithi070 ceph-mon[32631]: pgmap v74: 97 pgs: 97 active+clean; 31 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 1.9 MiB/s wr, 193 op/s 2024-06-04T04:59:33.643 INFO:tasks.cephfs.filesystem:scrub status for tag:b225b7fc-3902-4f20-9556-29160bb46ab4 - {'path': '/', 'tag': 'b225b7fc-3902-4f20-9556-29160bb46ab4', 'options': 'recursive,force'} 2024-06-04T04:59:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:33 smithi112 ceph-mon[36453]: pgmap v74: 97 pgs: 97 active+clean; 31 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 1.9 MiB/s wr, 193 op/s 2024-06-04T04:59:33.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 15959 3.95 MB/sec warmup 26 sec latency 15.196 ms 2024-06-04T04:59:34.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 16596 3.87 MB/sec warmup 27 sec latency 15.054 ms 2024-06-04T04:59:34.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.927+0000 7f612a30a640 1 -- 172.21.15.70:0/3638587502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6124107940 msgr2=0x7f6124107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:34.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.927+0000 7f612a30a640 1 --2- 172.21.15.70:0/3638587502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6124107940 0x7f6124107d20 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6114009930 tx=0x7f611402a420 comp rx=0 tx=0).stop 2024-06-04T04:59:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.927+0000 7f612a30a640 1 -- 172.21.15.70:0/3638587502 shutdown_connections 2024-06-04T04:59:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.927+0000 7f612a30a640 1 --2- 172.21.15.70:0/3638587502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6124102340 0x7f612410e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.927+0000 7f612a30a640 1 --2- 172.21.15.70:0/3638587502 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6124101980 0x7f6124101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.927+0000 7f612a30a640 1 --2- 172.21.15.70:0/3638587502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6124107940 0x7f6124107d20 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.927+0000 7f612a30a640 1 -- 172.21.15.70:0/3638587502 >> 172.21.15.70:0/3638587502 conn(0x7f6124078ad0 msgr2=0x7f6124078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.928+0000 7f612a30a640 1 -- 172.21.15.70:0/3638587502 shutdown_connections 2024-06-04T04:59:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.928+0000 7f612a30a640 1 -- 172.21.15.70:0/3638587502 wait complete. 2024-06-04T04:59:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.928+0000 7f612a30a640 1 Processor -- start 2024-06-04T04:59:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.928+0000 7f612a30a640 1 -- start start 2024-06-04T04:59:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f612a30a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6124101980 0x7f6124199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f612a30a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6124102340 0x7f612419a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f612a30a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6124107940 0x7f61241a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f612a30a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6124111ab0 con 0x7f6124107940 2024-06-04T04:59:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f612a30a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6124111930 con 0x7f6124101980 2024-06-04T04:59:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f612a30a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6124111c30 con 0x7f6124102340 2024-06-04T04:59:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f61237fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6124102340 0x7f612419a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f61288a7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6124107940 0x7f61241a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f61288a7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6124107940 0x7f61241a4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54406/0 (socket says 172.21.15.70:54406) 2024-06-04T04:59:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f6123fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6124101980 0x7f6124199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f61237fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6124102340 0x7f612419a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46840/0 (socket says 172.21.15.70:46840) 2024-06-04T04:59:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f61288a7640 1 -- 172.21.15.70:0/1061487965 learned_addr learned my addr 172.21.15.70:0/1061487965 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f61288a7640 1 -- 172.21.15.70:0/1061487965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6124102340 msgr2=0x7f612419a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f61288a7640 1 --2- 172.21.15.70:0/1061487965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6124102340 0x7f612419a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f61288a7640 1 -- 172.21.15.70:0/1061487965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6124101980 msgr2=0x7f6124199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f61288a7640 1 --2- 172.21.15.70:0/1061487965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6124101980 0x7f6124199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f61288a7640 1 -- 172.21.15.70:0/1061487965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61241a54f0 con 0x7f6124107940 2024-06-04T04:59:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f6123fff640 1 --2- 172.21.15.70:0/1061487965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6124101980 0x7f6124199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f61237fe640 1 --2- 172.21.15.70:0/1061487965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6124102340 0x7f612419a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f61288a7640 1 --2- 172.21.15.70:0/1061487965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6124107940 0x7f61241a4fb0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f611c00eb10 tx=0x7f611c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:34.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.929+0000 7f61217fa640 1 -- 172.21.15.70:0/1061487965 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c004b20 con 0x7f6124107940 2024-06-04T04:59:34.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.930+0000 7f612a30a640 1 -- 172.21.15.70:0/1061487965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6124197d20 con 0x7f6124107940 2024-06-04T04:59:34.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.930+0000 7f61217fa640 1 -- 172.21.15.70:0/1061487965 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f611c004cc0 con 0x7f6124107940 2024-06-04T04:59:34.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.930+0000 7f612a30a640 1 -- 172.21.15.70:0/1061487965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6124198220 con 0x7f6124107940 2024-06-04T04:59:34.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.930+0000 7f61217fa640 1 -- 172.21.15.70:0/1061487965 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c018630 con 0x7f6124107940 2024-06-04T04:59:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.932+0000 7f612d025740 1 Processor -- start 2024-06-04T04:59:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.932+0000 7f61217fa640 1 -- 172.21.15.70:0/1061487965 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f611c005000 con 0x7f6124107940 2024-06-04T04:59:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.932+0000 7f612d025740 1 -- start start 2024-06-04T04:59:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f612d025740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f318820 0x55dd1f318c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f612d025740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f319140 0x55dd1f3234c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f612d025740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 0x55dd1f325df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f612d025740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dd1f28ac30 con 0x55dd1f319140 2024-06-04T04:59:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f612d025740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dd1f27a7d0 con 0x55dd1f318820 2024-06-04T04:59:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f612d025740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55dd1f27e810 con 0x55dd1f323a00 2024-06-04T04:59:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f6123fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f319140 0x55dd1f3234c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f61237fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 0x55dd1f325df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f6123fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f319140 0x55dd1f3234c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54414/0 (socket says 172.21.15.70:54414) 2024-06-04T04:59:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f6123fff640 1 -- 172.21.15.70:0/3686585270 learned_addr learned my addr 172.21.15.70:0/3686585270 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f61237fe640 1 --2- 172.21.15.70:0/3686585270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f318820 0x55dd1f318c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f61237fe640 1 -- 172.21.15.70:0/3686585270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f318820 msgr2=0x55dd1f318c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f61237fe640 1 --2- 172.21.15.70:0/3686585270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f318820 0x55dd1f318c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f61237fe640 1 -- 172.21.15.70:0/3686585270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f319140 msgr2=0x55dd1f3234c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f61237fe640 1 --2- 172.21.15.70:0/3686585270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f319140 0x55dd1f3234c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f61237fe640 1 -- 172.21.15.70:0/3686585270 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd1ef8f210 con 0x55dd1f323a00 2024-06-04T04:59:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f6123fff640 1 --2- 172.21.15.70:0/3686585270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f319140 0x55dd1f3234c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f61217fa640 1 --2- 172.21.15.70:0/1061487965 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f610c0b2a10 0x7f610c0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f61237fe640 1 --2- 172.21.15.70:0/3686585270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 0x55dd1f325df0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f612406ac50 tx=0x7f6118005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6894cad7ded0ddf5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.933+0000 7f6123fff640 1 --2- 172.21.15.70:0/1061487965 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f610c0b2a10 0x7f610c0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.934+0000 7f61217fa640 1 -- 172.21.15.70:0/1061487965 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f611c0e2c60 con 0x7f6124107940 2024-06-04T04:59:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.934+0000 7f610a7fc640 1 -- 172.21.15.70:0/3686585270 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6118011050 con 0x55dd1f323a00 2024-06-04T04:59:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.934+0000 7f610a7fc640 1 -- 172.21.15.70:0/3686585270 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f611800fd10 con 0x55dd1f323a00 2024-06-04T04:59:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.934+0000 7f610a7fc640 1 -- 172.21.15.70:0/3686585270 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61180055a0 con 0x55dd1f323a00 2024-06-04T04:59:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.934+0000 7f612d025740 1 -- 172.21.15.70:0/3686585270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 msgr2=0x55dd1f325df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.934+0000 7f6123fff640 1 --2- 172.21.15.70:0/1061487965 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f610c0b2a10 0x7f610c0b4ed0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6114000c00 tx=0x7f61140058d0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.934+0000 7f612d025740 1 --2- 172.21.15.70:0/3686585270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 0x55dd1f325df0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f612406ac50 tx=0x7f6118005850 comp rx=0 tx=0).stop 2024-06-04T04:59:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.934+0000 7f612d025740 1 -- 172.21.15.70:0/3686585270 shutdown_connections 2024-06-04T04:59:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.934+0000 7f612d025740 1 --2- 172.21.15.70:0/3686585270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 0x55dd1f325df0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.934+0000 7f612d025740 1 --2- 172.21.15.70:0/3686585270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f319140 0x55dd1f3234c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.934+0000 7f612d025740 1 --2- 172.21.15.70:0/3686585270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f318820 0x55dd1f318c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.935+0000 7f612d025740 1 -- 172.21.15.70:0/3686585270 >> 172.21.15.70:0/3686585270 conn(0x55dd1f314d30 msgr2=0x55dd1f315110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.935+0000 7f612d025740 1 -- 172.21.15.70:0/3686585270 shutdown_connections 2024-06-04T04:59:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.935+0000 7f612d025740 1 -- 172.21.15.70:0/3686585270 wait complete. 2024-06-04T04:59:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.935+0000 7f612d025740 1 -- start start 2024-06-04T04:59:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.935+0000 7f612d025740 1 Processor -- start 2024-06-04T04:59:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.935+0000 7f612d025740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 0x55dd1f320a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.935+0000 7f612d025740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f319140 0x55dd1f320f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.935+0000 7f612d025740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 0x55dd1f3214a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.935+0000 7f612d025740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dd1ef60790 con 0x55dd1f318820 2024-06-04T04:59:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.935+0000 7f612d025740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dd1f236890 con 0x55dd1f319140 2024-06-04T04:59:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.935+0000 7f612d025740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55dd1f326630 con 0x55dd1f323a00 2024-06-04T04:59:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.935+0000 7f612d025740 10 client.0 _ll_register_callbacks cb 0x55dd1ef8ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f6123fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 0x55dd1f320a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f61288a7640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 0x55dd1f3214a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f6123fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 0x55dd1f320a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54424/0 (socket says 172.21.15.70:54424) 2024-06-04T04:59:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f61288a7640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 0x55dd1f3214a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46864/0 (socket says 172.21.15.70:46864) 2024-06-04T04:59:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f6123fff640 1 -- 172.21.15.70:0/3158943289 learned_addr learned my addr 172.21.15.70:0/3158943289 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f61237fe640 1 --2- 172.21.15.70:0/3158943289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f319140 0x55dd1f320f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f61288a7640 1 -- 172.21.15.70:0/3158943289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f319140 msgr2=0x55dd1f320f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f61288a7640 1 --2- 172.21.15.70:0/3158943289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f319140 0x55dd1f320f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f61288a7640 1 -- 172.21.15.70:0/3158943289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 msgr2=0x55dd1f320a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f61288a7640 1 --2- 172.21.15.70:0/3158943289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 0x55dd1f320a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f61288a7640 1 -- 172.21.15.70:0/3158943289 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd1f05b620 con 0x55dd1f323a00 2024-06-04T04:59:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f6123fff640 1 --2- 172.21.15.70:0/3158943289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 0x55dd1f320a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f61288a7640 1 --2- 172.21.15.70:0/3158943289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 0x55dd1f3214a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f611c0e03b0 tx=0x7f611c0a9950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10e4f92a8db118c4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f61037fe640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T04:59:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f61037fe640 1 -- 172.21.15.70:0/3158943289 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c0aa640 con 0x55dd1f323a00 2024-06-04T04:59:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f612d025740 1 -- 172.21.15.70:0/3158943289 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f611800fd10 con 0x55dd1f323a00 2024-06-04T04:59:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f61037fe640 1 -- 172.21.15.70:0/3158943289 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f611c0b2b20 con 0x55dd1f323a00 2024-06-04T04:59:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f61037fe640 1 -- 172.21.15.70:0/3158943289 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f611c0a9f40 con 0x55dd1f323a00 2024-06-04T04:59:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f612d025740 10 client.25645 fetch_fsmap learned FSMap version 31 2024-06-04T04:59:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.936+0000 7f612d025740 1 -- 172.21.15.70:0/3158943289 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55dd1f332470 con 0x55dd1f323a00 2024-06-04T04:59:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.937+0000 7f61037fe640 1 -- 172.21.15.70:0/3158943289 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f611800fd10 con 0x55dd1f323a00 2024-06-04T04:59:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.937+0000 7f61037fe640 1 -- 172.21.15.70:0/3158943289 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f611c0a58b0 con 0x55dd1f323a00 2024-06-04T04:59:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.937+0000 7f612d025740 10 client.25645 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T04:59:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.937+0000 7f612d025740 10 client.25645 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:59:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.937+0000 7f612d025740 20 client.25645 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.937+0000 7f612d025740 1 --2- 172.21.15.70:0/3158943289 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dd1f333980 0x55dd1f335d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.937+0000 7f612d025740 4 client.25645 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:59:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.937+0000 7f612d025740 1 -- 172.21.15.70:0/3158943289 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd1f28cb80 con 0x55dd1f333980 2024-06-04T04:59:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.937+0000 7f6123fff640 1 --2- 172.21.15.70:0/3158943289 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dd1f333980 0x55dd1f335d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.937+0000 7f61037fe640 10 client.25645 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.937+0000 7f6123fff640 1 --2- 172.21.15.70:0/3158943289 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dd1f333980 0x55dd1f335d70 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a74cd35cac704ef server_cookie=c2c435d2b445e05 in_seq=0 out_seq=0 2024-06-04T04:59:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.939+0000 7f61037fe640 1 -- 172.21.15.70:0/3158943289 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55dd1f28cb80 con 0x55dd1f333980 2024-06-04T04:59:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.939+0000 7f61037fe640 10 client.25645 handle_command_reply: tid=0 2024-06-04T04:59:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.939+0000 7f612d025740 1 client.25645 shutdown 2024-06-04T04:59:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.939+0000 7f612d025740 1 -- 172.21.15.70:0/3158943289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 msgr2=0x55dd1f3214a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.939+0000 7f612d025740 1 --2- 172.21.15.70:0/3158943289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 0x55dd1f3214a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f611c0e03b0 tx=0x7f611c0a9950 comp rx=0 tx=0).stop 2024-06-04T04:59:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.939+0000 7f612d025740 1 -- 172.21.15.70:0/3158943289 shutdown_connections 2024-06-04T04:59:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.939+0000 7f612d025740 1 --2- 172.21.15.70:0/3158943289 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dd1f333980 0x55dd1f335d70 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.939+0000 7f612d025740 1 --2- 172.21.15.70:0/3158943289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f319140 0x55dd1f320f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.939+0000 7f612d025740 1 --2- 172.21.15.70:0/3158943289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 0x55dd1f3214a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.939+0000 7f61037fe640 0 client.25645 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.939+0000 7f612d025740 1 --2- 172.21.15.70:0/3158943289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 0x55dd1f320a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.939+0000 7f612d025740 1 -- 172.21.15.70:0/3158943289 >> 172.21.15.70:0/3158943289 conn(0x55dd1f314d30 msgr2=0x55dd1f313fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.939+0000 7f612d025740 1 -- 172.21.15.70:0/3158943289 shutdown_connections 2024-06-04T04:59:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.939+0000 7f612d025740 1 -- 172.21.15.70:0/3158943289 wait complete. 2024-06-04T04:59:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.939+0000 7f612d025740 20 client.25645 trim_cache size 0 max 16384 2024-06-04T04:59:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.957+0000 7f612d025740 1 Processor -- start 2024-06-04T04:59:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.957+0000 7f612d025740 1 -- start start 2024-06-04T04:59:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.957+0000 7f612d025740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 0x55dd1f31b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.957+0000 7f612d025740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f319140 0x55dd1f33b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.957+0000 7f612d025740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f323a00 0x55dd1f33bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.957+0000 7f612d025740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61241ac3f0 con 0x55dd1f318820 2024-06-04T04:59:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.957+0000 7f612d025740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61241adf90 con 0x55dd1f323a00 2024-06-04T04:59:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.957+0000 7f612d025740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f61241acac0 con 0x55dd1f319140 2024-06-04T04:59:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.957+0000 7f6123fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 0x55dd1f31b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.957+0000 7f61288a7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f323a00 0x55dd1f33bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.957+0000 7f61237fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f319140 0x55dd1f33b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.957+0000 7f61288a7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f323a00 0x55dd1f33bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46474/0 (socket says 172.21.15.70:46474) 2024-06-04T04:59:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.957+0000 7f61288a7640 1 -- 172.21.15.70:0/3205916122 learned_addr learned my addr 172.21.15.70:0/3205916122 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.958+0000 7f6123fff640 1 -- 172.21.15.70:0/3205916122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f319140 msgr2=0x55dd1f33b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.958+0000 7f6123fff640 1 --2- 172.21.15.70:0/3205916122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f319140 0x55dd1f33b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.958+0000 7f6123fff640 1 -- 172.21.15.70:0/3205916122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f323a00 msgr2=0x55dd1f33bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.958+0000 7f6123fff640 1 --2- 172.21.15.70:0/3205916122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f323a00 0x55dd1f33bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.958+0000 7f6123fff640 1 -- 172.21.15.70:0/3205916122 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd1f31ef90 con 0x55dd1f318820 2024-06-04T04:59:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.958+0000 7f61288a7640 1 --2- 172.21.15.70:0/3205916122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f323a00 0x55dd1f33bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.958+0000 7f6123fff640 1 --2- 172.21.15.70:0/3205916122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 0x55dd1f31b230 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f611402a940 tx=0x7f6114046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e54bc5328dd3956f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.958+0000 7f6109ffb640 1 -- 172.21.15.70:0/3205916122 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611405a070 con 0x55dd1f318820 2024-06-04T04:59:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.958+0000 7f6109ffb640 1 -- 172.21.15.70:0/3205916122 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f611403adc0 con 0x55dd1f318820 2024-06-04T04:59:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.958+0000 7f6109ffb640 1 -- 172.21.15.70:0/3205916122 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6114035040 con 0x55dd1f318820 2024-06-04T04:59:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.959+0000 7f612d025740 1 -- 172.21.15.70:0/3205916122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 msgr2=0x55dd1f31b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.959+0000 7f612d025740 1 --2- 172.21.15.70:0/3205916122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 0x55dd1f31b230 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f611402a940 tx=0x7f6114046000 comp rx=0 tx=0).stop 2024-06-04T04:59:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.959+0000 7f612d025740 1 -- 172.21.15.70:0/3205916122 shutdown_connections 2024-06-04T04:59:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.959+0000 7f612d025740 1 --2- 172.21.15.70:0/3205916122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f323a00 0x55dd1f33bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.959+0000 7f612d025740 1 --2- 172.21.15.70:0/3205916122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f319140 0x55dd1f33b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.959+0000 7f612d025740 1 --2- 172.21.15.70:0/3205916122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 0x55dd1f31b230 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.959+0000 7f612d025740 1 -- 172.21.15.70:0/3205916122 >> 172.21.15.70:0/3205916122 conn(0x55dd1f314d30 msgr2=0x55dd1f32ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.959+0000 7f612d025740 1 -- 172.21.15.70:0/3205916122 shutdown_connections 2024-06-04T04:59:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.959+0000 7f612d025740 1 -- 172.21.15.70:0/3205916122 wait complete. 2024-06-04T04:59:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.959+0000 7f612d025740 1 -- start start 2024-06-04T04:59:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.959+0000 7f612d025740 1 Processor -- start 2024-06-04T04:59:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f612d025740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 0x55dd1f32bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f612d025740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f319140 0x55dd1f328850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f612d025740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 0x55dd1f328d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f612d025740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61241ad860 con 0x55dd1f318820 2024-06-04T04:59:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f612d025740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61241ad190 con 0x55dd1f319140 2024-06-04T04:59:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f612d025740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f61241aa3d0 con 0x55dd1f323a00 2024-06-04T04:59:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f612d025740 10 client.0 _ll_register_callbacks cb 0x7f611c0a9d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f6123fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 0x55dd1f32bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f6123fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 0x55dd1f32bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54438/0 (socket says 172.21.15.70:54438) 2024-06-04T04:59:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f6123fff640 1 -- 172.21.15.70:0/784695883 learned_addr learned my addr 172.21.15.70:0/784695883 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f61288a7640 1 --2- 172.21.15.70:0/784695883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 0x55dd1f328d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f61237fe640 1 --2- 172.21.15.70:0/784695883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f319140 0x55dd1f328850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f6123fff640 1 -- 172.21.15.70:0/784695883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 msgr2=0x55dd1f328d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f6123fff640 1 --2- 172.21.15.70:0/784695883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 0x55dd1f328d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f6123fff640 1 -- 172.21.15.70:0/784695883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f319140 msgr2=0x55dd1f328850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f6123fff640 1 --2- 172.21.15.70:0/784695883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f319140 0x55dd1f328850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f6123fff640 1 -- 172.21.15.70:0/784695883 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd1f3292d0 con 0x55dd1f318820 2024-06-04T04:59:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f6123fff640 1 --2- 172.21.15.70:0/784695883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 0x55dd1f32bcb0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f6114005940 tx=0x7f6114003880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d1173fbd565da50 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f61037fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:59:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.960+0000 7f61037fe640 1 -- 172.21.15.70:0/784695883 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6114062020 con 0x55dd1f318820 2024-06-04T04:59:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.961+0000 7f61037fe640 1 -- 172.21.15.70:0/784695883 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f6114003cb0 con 0x55dd1f318820 2024-06-04T04:59:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.961+0000 7f612d025740 1 -- 172.21.15.70:0/784695883 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f611403adc0 con 0x55dd1f318820 2024-06-04T04:59:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.961+0000 7f61037fe640 1 -- 172.21.15.70:0/784695883 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f611405a050 con 0x55dd1f318820 2024-06-04T04:59:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.961+0000 7f612d025740 10 client.16188 fetch_fsmap learned FSMap version 31 2024-06-04T04:59:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.961+0000 7f612d025740 1 -- 172.21.15.70:0/784695883 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55dd1f39f190 con 0x55dd1f318820 2024-06-04T04:59:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.961+0000 7f61037fe640 1 -- 172.21.15.70:0/784695883 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f611403adc0 con 0x55dd1f318820 2024-06-04T04:59:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.961+0000 7f61037fe640 1 -- 172.21.15.70:0/784695883 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6114047c70 con 0x55dd1f318820 2024-06-04T04:59:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.961+0000 7f612d025740 10 client.16188 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T04:59:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.961+0000 7f612d025740 10 client.16188 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:59:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.961+0000 7f612d025740 20 client.16188 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.961+0000 7f612d025740 1 --2- 172.21.15.70:0/784695883 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dd1f39f8a0 0x55dd1f3a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.961+0000 7f612d025740 4 client.16188 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T04:59:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.961+0000 7f612d025740 1 -- 172.21.15.70:0/784695883 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dd1f286360 con 0x55dd1f39f8a0 2024-06-04T04:59:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.961+0000 7f61237fe640 1 --2- 172.21.15.70:0/784695883 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dd1f39f8a0 0x55dd1f3a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.962+0000 7f61237fe640 1 --2- 172.21.15.70:0/784695883 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dd1f39f8a0 0x55dd1f3a1cc0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aba15e0c2fc810b5 server_cookie=cfd1167990f41147 in_seq=0 out_seq=0 2024-06-04T04:59:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.962+0000 7f61037fe640 10 client.16188 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.982+0000 7f61037fe640 1 -- 172.21.15.70:0/784695883 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55dd1f286360 con 0x55dd1f39f8a0 2024-06-04T04:59:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.982+0000 7f61037fe640 10 client.16188 handle_command_reply: tid=0 2024-06-04T04:59:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.982+0000 7f612d025740 1 client.16188 shutdown 2024-06-04T04:59:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.982+0000 7f612d025740 1 -- 172.21.15.70:0/784695883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 msgr2=0x55dd1f32bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.982+0000 7f612d025740 1 --2- 172.21.15.70:0/784695883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 0x55dd1f32bcb0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f6114005940 tx=0x7f6114003880 comp rx=0 tx=0).stop 2024-06-04T04:59:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.982+0000 7f612d025740 1 -- 172.21.15.70:0/784695883 shutdown_connections 2024-06-04T04:59:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.982+0000 7f612d025740 1 --2- 172.21.15.70:0/784695883 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dd1f39f8a0 0x55dd1f3a1cc0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.982+0000 7f612d025740 1 --2- 172.21.15.70:0/784695883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd1f319140 0x55dd1f328850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.982+0000 7f612d025740 1 --2- 172.21.15.70:0/784695883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dd1f323a00 0x55dd1f328d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.982+0000 7f612d025740 1 --2- 172.21.15.70:0/784695883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd1f318820 0x55dd1f32bcb0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.982+0000 7f61037fe640 0 client.16188 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.982+0000 7f612d025740 1 -- 172.21.15.70:0/784695883 >> 172.21.15.70:0/784695883 conn(0x55dd1f314d30 msgr2=0x55dd1f322250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.982+0000 7f612d025740 1 -- 172.21.15.70:0/784695883 shutdown_connections 2024-06-04T04:59:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.982+0000 7f612d025740 1 -- 172.21.15.70:0/784695883 wait complete. 2024-06-04T04:59:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.983+0000 7f612d025740 20 client.16188 trim_cache size 0 max 16384 2024-06-04T04:59:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T04:59:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T04:59:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T04:59:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.983+0000 7f612a30a640 1 -- 172.21.15.70:0/1061487965 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f610c0b2a10 msgr2=0x7f610c0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.983+0000 7f612a30a640 1 --2- 172.21.15.70:0/1061487965 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f610c0b2a10 0x7f610c0b4ed0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6114000c00 tx=0x7f61140058d0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.983+0000 7f612a30a640 1 -- 172.21.15.70:0/1061487965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6124107940 msgr2=0x7f61241a4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.983+0000 7f612a30a640 1 --2- 172.21.15.70:0/1061487965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6124107940 0x7f61241a4fb0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f611c00eb10 tx=0x7f611c00c2d0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.983+0000 7f612a30a640 1 -- 172.21.15.70:0/1061487965 shutdown_connections 2024-06-04T04:59:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.983+0000 7f612a30a640 1 --2- 172.21.15.70:0/1061487965 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f610c0b2a10 0x7f610c0b4ed0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.983+0000 7f612a30a640 1 --2- 172.21.15.70:0/1061487965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6124107940 0x7f61241a4fb0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.983+0000 7f612a30a640 1 --2- 172.21.15.70:0/1061487965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6124102340 0x7f612419a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.983+0000 7f612a30a640 1 --2- 172.21.15.70:0/1061487965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6124101980 0x7f6124199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.983+0000 7f612a30a640 1 -- 172.21.15.70:0/1061487965 >> 172.21.15.70:0/1061487965 conn(0x7f6124078ad0 msgr2=0x7f61241045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.983+0000 7f612a30a640 1 -- 172.21.15.70:0/1061487965 shutdown_connections 2024-06-04T04:59:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:34.983+0000 7f612a30a640 1 -- 172.21.15.70:0/1061487965 wait complete. 2024-06-04T04:59:35.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:35 smithi098 ceph-mon[35569]: pgmap v75: 97 pgs: 97 active+clean; 31 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-06-04T04:59:35.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:35 smithi070 ceph-mon[32631]: pgmap v75: 97 pgs: 97 active+clean; 31 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-06-04T04:59:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:35 smithi112 ceph-mon[36453]: pgmap v75: 97 pgs: 97 active+clean; 31 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-06-04T04:59:35.712 INFO:tasks.workunit.client.0.smithi070.stdout: 1 17368 3.95 MB/sec warmup 28 sec latency 17.157 ms 2024-06-04T04:59:36.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 17852 3.97 MB/sec warmup 29 sec latency 11.592 ms 2024-06-04T04:59:37.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:37 smithi098 ceph-mon[35569]: pgmap v76: 97 pgs: 97 active+clean; 28 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 1.9 MiB/s wr, 217 op/s 2024-06-04T04:59:37.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:37 smithi070 ceph-mon[32631]: pgmap v76: 97 pgs: 97 active+clean; 28 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 1.9 MiB/s wr, 217 op/s 2024-06-04T04:59:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:37 smithi112 ceph-mon[36453]: pgmap v76: 97 pgs: 97 active+clean; 28 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 1.9 MiB/s wr, 217 op/s 2024-06-04T04:59:37.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 18170 3.88 MB/sec warmup 30 sec latency 11.710 ms 2024-06-04T04:59:38.509 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:38 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T04:59:38.509 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:38 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T04:59:38.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:38 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T04:59:38.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:38 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T04:59:38.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 18566 3.78 MB/sec warmup 31 sec latency 16.073 ms 2024-06-04T04:59:38.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:38 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T04:59:38.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:38 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T04:59:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:39 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T04:59:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:39 smithi070 ceph-mon[32631]: pgmap v77: 97 pgs: 97 active+clean; 28 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-06-04T04:59:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:39 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T04:59:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:39 smithi112 ceph-mon[36453]: pgmap v77: 97 pgs: 97 active+clean; 28 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-06-04T04:59:39.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 19156 3.80 MB/sec warmup 32 sec latency 15.455 ms 2024-06-04T04:59:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:39 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T04:59:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:39 smithi098 ceph-mon[35569]: pgmap v77: 97 pgs: 97 active+clean; 28 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-06-04T04:59:40.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 19811 3.70 MB/sec warmup 33 sec latency 15.674 ms 2024-06-04T04:59:41.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:41 smithi070 ceph-mon[32631]: pgmap v78: 97 pgs: 97 active+clean; 28 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-06-04T04:59:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:41 smithi112 ceph-mon[36453]: pgmap v78: 97 pgs: 97 active+clean; 28 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-06-04T04:59:41.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 20391 3.67 MB/sec warmup 34 sec latency 14.566 ms 2024-06-04T04:59:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:41 smithi098 ceph-mon[35569]: pgmap v78: 97 pgs: 97 active+clean; 28 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-06-04T04:59:42.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 21097 3.74 MB/sec warmup 35 sec latency 16.148 ms 2024-06-04T04:59:43.547 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:43 smithi098 ceph-mon[35569]: pgmap v79: 97 pgs: 97 active+clean; 27 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.6 MiB/s wr, 210 op/s 2024-06-04T04:59:43.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:43 smithi070 ceph-mon[32631]: pgmap v79: 97 pgs: 97 active+clean; 27 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.6 MiB/s wr, 210 op/s 2024-06-04T04:59:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:43 smithi112 ceph-mon[36453]: pgmap v79: 97 pgs: 97 active+clean; 27 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.6 MiB/s wr, 210 op/s 2024-06-04T04:59:43.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 21577 3.76 MB/sec warmup 36 sec latency 19.948 ms 2024-06-04T04:59:44.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 21857 3.67 MB/sec warmup 37 sec latency 44.356 ms 2024-06-04T04:59:45.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:45 smithi070 ceph-mon[32631]: pgmap v80: 97 pgs: 97 active+clean; 27 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 1.0 MiB/s wr, 136 op/s 2024-06-04T04:59:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:45 smithi112 ceph-mon[36453]: pgmap v80: 97 pgs: 97 active+clean; 27 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 1.0 MiB/s wr, 136 op/s 2024-06-04T04:59:45.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 22349 3.61 MB/sec warmup 38 sec latency 12.500 ms 2024-06-04T04:59:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:45 smithi098 ceph-mon[35569]: pgmap v80: 97 pgs: 97 active+clean; 27 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 1.0 MiB/s wr, 136 op/s 2024-06-04T04:59:46.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 22947 3.61 MB/sec warmup 39 sec latency 11.189 ms 2024-06-04T04:59:47.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:47 smithi070 ceph-mon[32631]: pgmap v81: 97 pgs: 97 active+clean; 31 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.7 MiB/s wr, 203 op/s 2024-06-04T04:59:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:47 smithi112 ceph-mon[36453]: pgmap v81: 97 pgs: 97 active+clean; 31 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.7 MiB/s wr, 203 op/s 2024-06-04T04:59:47.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 23661 3.57 MB/sec warmup 40 sec latency 14.947 ms 2024-06-04T04:59:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:47 smithi098 ceph-mon[35569]: pgmap v81: 97 pgs: 97 active+clean; 31 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.7 MiB/s wr, 203 op/s 2024-06-04T04:59:48.081 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.076+0000 7f1211571640 1 -- 172.21.15.70:0/3813653656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f120c102340 msgr2=0x7f120c10e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:48.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.076+0000 7f1211571640 1 --2- 172.21.15.70:0/3813653656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f120c102340 0x7f120c10e800 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f1204009cd0 tx=0x7f120402a8e0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.076+0000 7f1211571640 1 -- 172.21.15.70:0/3813653656 shutdown_connections 2024-06-04T04:59:48.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.076+0000 7f1211571640 1 --2- 172.21.15.70:0/3813653656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f120c101980 0x7f120c101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.076+0000 7f1211571640 1 --2- 172.21.15.70:0/3813653656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f120c102340 0x7f120c10e800 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.076+0000 7f1211571640 1 --2- 172.21.15.70:0/3813653656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f120c107940 0x7f120c107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.076+0000 7f1211571640 1 -- 172.21.15.70:0/3813653656 >> 172.21.15.70:0/3813653656 conn(0x7f120c078ad0 msgr2=0x7f120c078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:48.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.076+0000 7f1211571640 1 -- 172.21.15.70:0/3813653656 shutdown_connections 2024-06-04T04:59:48.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.076+0000 7f1211571640 1 -- 172.21.15.70:0/3813653656 wait complete. 2024-06-04T04:59:48.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.076+0000 7f1211571640 1 Processor -- start 2024-06-04T04:59:48.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1211571640 1 -- start start 2024-06-04T04:59:48.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1211571640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f120c101980 0x7f120c199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1211571640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f120c102340 0x7f120c1995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1211571640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f120c107940 0x7f120c19d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1211571640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f120c111ab0 con 0x7f120c102340 2024-06-04T04:59:48.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1211571640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f120c111930 con 0x7f120c101980 2024-06-04T04:59:48.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1211571640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f120c111c30 con 0x7f120c107940 2024-06-04T04:59:48.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1203fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f120c102340 0x7f120c1995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1203fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f120c102340 0x7f120c1995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57676/0 (socket says 172.21.15.70:57676) 2024-06-04T04:59:48.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1203fff640 1 -- 172.21.15.70:0/462418624 learned_addr learned my addr 172.21.15.70:0/462418624 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:48.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f120affd640 1 --2- 172.21.15.70:0/462418624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f120c101980 0x7f120c199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f120b7fe640 1 --2- 172.21.15.70:0/462418624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f120c107940 0x7f120c19d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1203fff640 1 -- 172.21.15.70:0/462418624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f120c107940 msgr2=0x7f120c19d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:48.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1203fff640 1 --2- 172.21.15.70:0/462418624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f120c107940 0x7f120c19d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1203fff640 1 -- 172.21.15.70:0/462418624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f120c101980 msgr2=0x7f120c199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:48.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1203fff640 1 --2- 172.21.15.70:0/462418624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f120c101980 0x7f120c199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1203fff640 1 -- 172.21.15.70:0/462418624 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120c19e050 con 0x7f120c102340 2024-06-04T04:59:48.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1203fff640 1 --2- 172.21.15.70:0/462418624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f120c102340 0x7f120c1995a0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f11f400e9f0 tx=0x7f11f400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:48.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1208ff9640 1 -- 172.21.15.70:0/462418624 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f400cd80 con 0x7f120c102340 2024-06-04T04:59:48.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1211571640 1 -- 172.21.15.70:0/462418624 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f120c19e340 con 0x7f120c102340 2024-06-04T04:59:48.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1208ff9640 1 -- 172.21.15.70:0/462418624 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11f40108c0 con 0x7f120c102340 2024-06-04T04:59:48.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.077+0000 7f1208ff9640 1 -- 172.21.15.70:0/462418624 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f40187c0 con 0x7f120c102340 2024-06-04T04:59:48.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.078+0000 7f1211571640 1 -- 172.21.15.70:0/462418624 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f120c1a5bd0 con 0x7f120c102340 2024-06-04T04:59:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.080+0000 7f1208ff9640 1 -- 172.21.15.70:0/462418624 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f11f4010430 con 0x7f120c102340 2024-06-04T04:59:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.080+0000 7f1214366740 1 Processor -- start 2024-06-04T04:59:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.080+0000 7f1214366740 1 -- start start 2024-06-04T04:59:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.080+0000 7f1214366740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c331740 0x55eb6c331b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f1214366740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c332060 0x55eb6c3408d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f1214366740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c340e10 0x55eb6c343200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f1214366740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55eb6c2a3c30 con 0x55eb6c340e10 2024-06-04T04:59:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f1214366740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eb6c2937d0 con 0x55eb6c331740 2024-06-04T04:59:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f1214366740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55eb6c297810 con 0x55eb6c332060 2024-06-04T04:59:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f120affd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c332060 0x55eb6c3408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f1203fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c340e10 0x55eb6c343200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f120affd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c332060 0x55eb6c3408d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48340/0 (socket says 172.21.15.70:48340) 2024-06-04T04:59:48.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f120affd640 1 -- 172.21.15.70:0/1705690457 learned_addr learned my addr 172.21.15.70:0/1705690457 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:48.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f1208ff9640 1 --2- 172.21.15.70:0/462418624 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f11f80b2a10 0x7f11f80b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f120b7fe640 1 --2- 172.21.15.70:0/1705690457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c331740 0x55eb6c331b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f120affd640 1 -- 172.21.15.70:0/1705690457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c331740 msgr2=0x55eb6c331b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:48.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f120affd640 1 --2- 172.21.15.70:0/1705690457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c331740 0x55eb6c331b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f120b7fe640 1 --2- 172.21.15.70:0/462418624 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f11f80b2a10 0x7f11f80b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f120affd640 1 -- 172.21.15.70:0/1705690457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c340e10 msgr2=0x55eb6c343200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:48.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f120affd640 1 --2- 172.21.15.70:0/1705690457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c340e10 0x55eb6c343200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f120affd640 1 -- 172.21.15.70:0/1705690457 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb6bfa8210 con 0x55eb6c332060 2024-06-04T04:59:48.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f1208ff9640 1 -- 172.21.15.70:0/462418624 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f11f40e2a30 con 0x7f120c102340 2024-06-04T04:59:48.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f120affd640 1 --2- 172.21.15.70:0/1705690457 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c332060 0x55eb6c3408d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009ca0 tx=0x7f11fc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9301af005b3541c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:48.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f12017fa640 1 -- 172.21.15.70:0/1705690457 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11fc01e070 con 0x55eb6c332060 2024-06-04T04:59:48.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f12017fa640 1 -- 172.21.15.70:0/1705690457 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11fc019ac0 con 0x55eb6c332060 2024-06-04T04:59:48.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.081+0000 7f12017fa640 1 -- 172.21.15.70:0/1705690457 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11fc00e890 con 0x55eb6c332060 2024-06-04T04:59:48.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.082+0000 7f1214366740 1 -- 172.21.15.70:0/1705690457 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c332060 msgr2=0x55eb6c3408d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:48.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.082+0000 7f120b7fe640 1 --2- 172.21.15.70:0/462418624 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f11f80b2a10 0x7f11f80b4ed0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1204005e30 tx=0x7f12040095c0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:48.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.082+0000 7f1214366740 1 --2- 172.21.15.70:0/1705690457 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c332060 0x55eb6c3408d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009ca0 tx=0x7f11fc011810 comp rx=0 tx=0).stop 2024-06-04T04:59:48.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.082+0000 7f1214366740 1 -- 172.21.15.70:0/1705690457 shutdown_connections 2024-06-04T04:59:48.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.082+0000 7f1214366740 1 --2- 172.21.15.70:0/1705690457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c340e10 0x55eb6c343200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.082+0000 7f1214366740 1 --2- 172.21.15.70:0/1705690457 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c332060 0x55eb6c3408d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.082+0000 7f1214366740 1 --2- 172.21.15.70:0/1705690457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c331740 0x55eb6c331b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.082+0000 7f1214366740 1 -- 172.21.15.70:0/1705690457 >> 172.21.15.70:0/1705690457 conn(0x55eb6c32dd30 msgr2=0x55eb6c32e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:48.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.082+0000 7f1214366740 1 -- 172.21.15.70:0/1705690457 shutdown_connections 2024-06-04T04:59:48.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.082+0000 7f1214366740 1 -- 172.21.15.70:0/1705690457 wait complete. 2024-06-04T04:59:48.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f1214366740 1 -- start start 2024-06-04T04:59:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f1214366740 1 Processor -- start 2024-06-04T04:59:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f1214366740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c331740 0x55eb6c33dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f1214366740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c332060 0x55eb6c33e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f1214366740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c340e10 0x55eb6c33e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f1214366740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55eb6bf79790 con 0x55eb6c332060 2024-06-04T04:59:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f1214366740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eb6c24f890 con 0x55eb6c340e10 2024-06-04T04:59:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f1214366740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55eb6c343a40 con 0x55eb6c331740 2024-06-04T04:59:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f1214366740 10 client.0 _ll_register_callbacks cb 0x55eb6bfa7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f1203fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c332060 0x55eb6c33e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f1203fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c332060 0x55eb6c33e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57688/0 (socket says 172.21.15.70:57688) 2024-06-04T04:59:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f120affd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c331740 0x55eb6c33dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f1203fff640 1 -- 172.21.15.70:0/3898257978 learned_addr learned my addr 172.21.15.70:0/3898257978 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f120b7fe640 1 --2- 172.21.15.70:0/3898257978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c340e10 0x55eb6c33e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f1203fff640 1 -- 172.21.15.70:0/3898257978 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c331740 msgr2=0x55eb6c33dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f1203fff640 1 --2- 172.21.15.70:0/3898257978 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c331740 0x55eb6c33dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f1203fff640 1 -- 172.21.15.70:0/3898257978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c340e10 msgr2=0x55eb6c33e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f1203fff640 1 --2- 172.21.15.70:0/3898257978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c340e10 0x55eb6c33e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f1203fff640 1 -- 172.21.15.70:0/3898257978 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb6c074620 con 0x55eb6c332060 2024-06-04T04:59:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.083+0000 7f120affd640 1 --2- 172.21.15.70:0/3898257978 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c331740 0x55eb6c33dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.084+0000 7f1203fff640 1 --2- 172.21.15.70:0/3898257978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c332060 0x55eb6c33e320 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f11f40e01e0 tx=0x7f11f40a9740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39fe90db87b6f0bc server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.084+0000 7f11f27fc640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:59:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.084+0000 7f11f27fc640 1 -- 172.21.15.70:0/3898257978 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f40aa410 con 0x55eb6c332060 2024-06-04T04:59:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.084+0000 7f11f27fc640 1 -- 172.21.15.70:0/3898257978 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f11f40a8f70 con 0x55eb6c332060 2024-06-04T04:59:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.084+0000 7f1214366740 1 -- 172.21.15.70:0/3898257978 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11fc019ac0 con 0x55eb6c332060 2024-06-04T04:59:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.084+0000 7f11f27fc640 1 -- 172.21.15.70:0/3898257978 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f11f40b2b70 con 0x55eb6c332060 2024-06-04T04:59:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.084+0000 7f1214366740 10 client.16200 fetch_fsmap learned FSMap version 31 2024-06-04T04:59:48.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.084+0000 7f1214366740 1 -- 172.21.15.70:0/3898257978 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55eb6c34f7e0 con 0x55eb6c332060 2024-06-04T04:59:48.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.084+0000 7f11f27fc640 1 -- 172.21.15.70:0/3898257978 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11fc019ac0 con 0x55eb6c332060 2024-06-04T04:59:48.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.084+0000 7f11f27fc640 1 -- 172.21.15.70:0/3898257978 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f11f40a89e0 con 0x55eb6c332060 2024-06-04T04:59:48.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.084+0000 7f1214366740 10 client.16200 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T04:59:48.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.084+0000 7f1214366740 10 client.16200 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:59:48.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.084+0000 7f1214366740 20 client.16200 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:48.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.084+0000 7f1214366740 1 --2- 172.21.15.70:0/3898257978 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55eb6c350ca0 0x55eb6c353130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.084+0000 7f1214366740 4 client.16200 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T04:59:48.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.084+0000 7f1214366740 1 -- 172.21.15.70:0/3898257978 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb6c2a5b80 con 0x55eb6c350ca0 2024-06-04T04:59:48.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.085+0000 7f120affd640 1 --2- 172.21.15.70:0/3898257978 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55eb6c350ca0 0x55eb6c353130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.085+0000 7f11f27fc640 10 client.16200 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:48.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.085+0000 7f120affd640 1 --2- 172.21.15.70:0/3898257978 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55eb6c350ca0 0x55eb6c353130 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b4c6bf07f55afdc server_cookie=44cd95a797034617 in_seq=0 out_seq=0 2024-06-04T04:59:48.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.086+0000 7f11f27fc640 1 -- 172.21.15.70:0/3898257978 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55eb6c2a5b80 con 0x55eb6c350ca0 2024-06-04T04:59:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.086+0000 7f11f27fc640 10 client.16200 handle_command_reply: tid=0 2024-06-04T04:59:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.086+0000 7f1214366740 1 client.16200 shutdown 2024-06-04T04:59:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.086+0000 7f1214366740 1 -- 172.21.15.70:0/3898257978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c332060 msgr2=0x55eb6c33e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.086+0000 7f1214366740 1 --2- 172.21.15.70:0/3898257978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c332060 0x55eb6c33e320 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f11f40e01e0 tx=0x7f11f40a9740 comp rx=0 tx=0).stop 2024-06-04T04:59:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.086+0000 7f1214366740 1 -- 172.21.15.70:0/3898257978 shutdown_connections 2024-06-04T04:59:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.086+0000 7f1214366740 1 --2- 172.21.15.70:0/3898257978 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55eb6c350ca0 0x55eb6c353130 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.086+0000 7f1214366740 1 --2- 172.21.15.70:0/3898257978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c340e10 0x55eb6c33e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.086+0000 7f11f27fc640 0 client.16200 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.086+0000 7f1214366740 1 --2- 172.21.15.70:0/3898257978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c332060 0x55eb6c33e320 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.086+0000 7f1214366740 1 --2- 172.21.15.70:0/3898257978 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c331740 0x55eb6c33dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.086+0000 7f1214366740 1 -- 172.21.15.70:0/3898257978 >> 172.21.15.70:0/3898257978 conn(0x55eb6c32dd30 msgr2=0x55eb6c333ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.086+0000 7f1214366740 1 -- 172.21.15.70:0/3898257978 shutdown_connections 2024-06-04T04:59:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.086+0000 7f1214366740 1 -- 172.21.15.70:0/3898257978 wait complete. 2024-06-04T04:59:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.086+0000 7f1214366740 20 client.16200 trim_cache size 0 max 16384 2024-06-04T04:59:48.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.104+0000 7f1214366740 1 Processor -- start 2024-06-04T04:59:48.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.104+0000 7f1214366740 1 -- start start 2024-06-04T04:59:48.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.104+0000 7f1214366740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c331740 0x55eb6c333950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.104+0000 7f1214366740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c332060 0x55eb6c358a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.104+0000 7f1214366740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c340e10 0x55eb6c358f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.104+0000 7f1214366740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f120c1ac340 con 0x55eb6c332060 2024-06-04T04:59:48.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.104+0000 7f1214366740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f120c1adee0 con 0x55eb6c340e10 2024-06-04T04:59:48.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.104+0000 7f1214366740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f120c1aca10 con 0x55eb6c331740 2024-06-04T04:59:48.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.104+0000 7f1203fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c332060 0x55eb6c358a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.104+0000 7f120affd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c331740 0x55eb6c333950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.104+0000 7f1203fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c332060 0x55eb6c358a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57690/0 (socket says 172.21.15.70:57690) 2024-06-04T04:59:48.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.104+0000 7f1203fff640 1 -- 172.21.15.70:0/2479034092 learned_addr learned my addr 172.21.15.70:0/2479034092 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:48.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.104+0000 7f120b7fe640 1 --2- 172.21.15.70:0/2479034092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c340e10 0x55eb6c358f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.105+0000 7f1203fff640 1 -- 172.21.15.70:0/2479034092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c331740 msgr2=0x55eb6c333950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:48.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.105+0000 7f1203fff640 1 --2- 172.21.15.70:0/2479034092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c331740 0x55eb6c333950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.105+0000 7f1203fff640 1 -- 172.21.15.70:0/2479034092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c340e10 msgr2=0x55eb6c358f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.105+0000 7f1203fff640 1 --2- 172.21.15.70:0/2479034092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c340e10 0x55eb6c358f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.105+0000 7f1203fff640 1 -- 172.21.15.70:0/2479034092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb6c32db60 con 0x55eb6c332060 2024-06-04T04:59:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.105+0000 7f120affd640 1 --2- 172.21.15.70:0/2479034092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c331740 0x55eb6c333950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.105+0000 7f1203fff640 1 --2- 172.21.15.70:0/2479034092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c332060 0x55eb6c358a50 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f11f40a6f80 tx=0x7f11f400c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661abc0031f008fa server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.105+0000 7f1200ff9640 1 -- 172.21.15.70:0/2479034092 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f40a6360 con 0x55eb6c332060 2024-06-04T04:59:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.105+0000 7f1200ff9640 1 -- 172.21.15.70:0/2479034092 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11f40aa410 con 0x55eb6c332060 2024-06-04T04:59:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.105+0000 7f1200ff9640 1 -- 172.21.15.70:0/2479034092 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f4010720 con 0x55eb6c332060 2024-06-04T04:59:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.106+0000 7f1214366740 1 -- 172.21.15.70:0/2479034092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c332060 msgr2=0x55eb6c358a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.106+0000 7f1214366740 1 --2- 172.21.15.70:0/2479034092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c332060 0x55eb6c358a50 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f11f40a6f80 tx=0x7f11f400c7d0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.106+0000 7f1214366740 1 -- 172.21.15.70:0/2479034092 shutdown_connections 2024-06-04T04:59:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.106+0000 7f1214366740 1 --2- 172.21.15.70:0/2479034092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c340e10 0x55eb6c358f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.106+0000 7f1214366740 1 --2- 172.21.15.70:0/2479034092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c332060 0x55eb6c358a50 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.106+0000 7f1214366740 1 --2- 172.21.15.70:0/2479034092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c331740 0x55eb6c333950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.106+0000 7f1214366740 1 -- 172.21.15.70:0/2479034092 >> 172.21.15.70:0/2479034092 conn(0x55eb6c32dd30 msgr2=0x55eb6c333240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.106+0000 7f1214366740 1 -- 172.21.15.70:0/2479034092 shutdown_connections 2024-06-04T04:59:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.106+0000 7f1214366740 1 -- 172.21.15.70:0/2479034092 wait complete. 2024-06-04T04:59:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.106+0000 7f1214366740 1 -- start start 2024-06-04T04:59:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.106+0000 7f1214366740 1 Processor -- start 2024-06-04T04:59:48.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f1214366740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c331740 0x55eb6c349300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f1214366740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c332060 0x55eb6c344670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f1214366740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c340e10 0x55eb6c344bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f1214366740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f120c1ad7b0 con 0x55eb6c340e10 2024-06-04T04:59:48.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f1214366740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f120c1ad0e0 con 0x55eb6c331740 2024-06-04T04:59:48.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f1214366740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f120c1aa320 con 0x55eb6c332060 2024-06-04T04:59:48.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f1214366740 10 client.0 _ll_register_callbacks cb 0x7f11f400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T04:59:48.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f120b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c340e10 0x55eb6c344bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f120b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c340e10 0x55eb6c344bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57696/0 (socket says 172.21.15.70:57696) 2024-06-04T04:59:48.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f1203fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c332060 0x55eb6c344670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f120b7fe640 1 -- 172.21.15.70:0/3331425049 learned_addr learned my addr 172.21.15.70:0/3331425049 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T04:59:48.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f120affd640 1 --2- 172.21.15.70:0/3331425049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c331740 0x55eb6c349300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f120b7fe640 1 -- 172.21.15.70:0/3331425049 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c332060 msgr2=0x55eb6c344670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:48.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f120b7fe640 1 --2- 172.21.15.70:0/3331425049 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c332060 0x55eb6c344670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f120b7fe640 1 -- 172.21.15.70:0/3331425049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c331740 msgr2=0x55eb6c349300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T04:59:48.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f120b7fe640 1 --2- 172.21.15.70:0/3331425049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c331740 0x55eb6c349300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f120b7fe640 1 -- 172.21.15.70:0/3331425049 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb6c3453b0 con 0x55eb6c340e10 2024-06-04T04:59:48.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f1203fff640 1 --2- 172.21.15.70:0/3331425049 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c332060 0x55eb6c344670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T04:59:48.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f120affd640 1 --2- 172.21.15.70:0/3331425049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c331740 0x55eb6c349300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T04:59:48.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f120b7fe640 1 --2- 172.21.15.70:0/3331425049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c340e10 0x55eb6c344bb0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1204035000 tx=0x7f120402a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d248a149f6bcf1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T04:59:48.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f11f27fc640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T04:59:48.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f11f27fc640 1 -- 172.21.15.70:0/3331425049 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1204058060 con 0x55eb6c340e10 2024-06-04T04:59:48.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f11f27fc640 1 -- 172.21.15.70:0/3331425049 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f1204004540 con 0x55eb6c340e10 2024-06-04T04:59:48.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.107+0000 7f1214366740 1 -- 172.21.15.70:0/3331425049 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11f40aa410 con 0x55eb6c340e10 2024-06-04T04:59:48.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.108+0000 7f11f27fc640 1 -- 172.21.15.70:0/3331425049 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1204004740 con 0x55eb6c340e10 2024-06-04T04:59:48.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.108+0000 7f1214366740 10 client.16206 fetch_fsmap learned FSMap version 31 2024-06-04T04:59:48.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.108+0000 7f1214366740 1 -- 172.21.15.70:0/3331425049 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55eb6c3bb8c0 con 0x55eb6c340e10 2024-06-04T04:59:48.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.108+0000 7f11f27fc640 1 -- 172.21.15.70:0/3331425049 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11f40aa410 con 0x55eb6c340e10 2024-06-04T04:59:48.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.108+0000 7f11f27fc640 1 -- 172.21.15.70:0/3331425049 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f1204030070 con 0x55eb6c340e10 2024-06-04T04:59:48.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.108+0000 7f1214366740 10 client.16206 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T04:59:48.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.108+0000 7f1214366740 10 client.16206 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T04:59:48.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.108+0000 7f1214366740 20 client.16206 populate_metadata read hostname 'smithi070' 2024-06-04T04:59:48.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.108+0000 7f1214366740 1 --2- 172.21.15.70:0/3331425049 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55eb6c3bc0d0 0x55eb6c3be4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T04:59:48.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.108+0000 7f1214366740 4 client.16206 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T04:59:48.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.108+0000 7f1214366740 1 -- 172.21.15.70:0/3331425049 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eb6c29f360 con 0x55eb6c3bc0d0 2024-06-04T04:59:48.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.108+0000 7f120affd640 1 --2- 172.21.15.70:0/3331425049 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55eb6c3bc0d0 0x55eb6c3be4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T04:59:48.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.109+0000 7f11f27fc640 10 client.16206 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:48.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.109+0000 7f120affd640 1 --2- 172.21.15.70:0/3331425049 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55eb6c3bc0d0 0x55eb6c3be4f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15b27430970e569 server_cookie=72f929be3c644abf in_seq=0 out_seq=0 2024-06-04T04:59:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.131+0000 7f11f27fc640 1 -- 172.21.15.70:0/3331425049 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55eb6c29f360 con 0x55eb6c3bc0d0 2024-06-04T04:59:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.131+0000 7f11f27fc640 10 client.16206 handle_command_reply: tid=0 2024-06-04T04:59:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.131+0000 7f1214366740 1 client.16206 shutdown 2024-06-04T04:59:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.131+0000 7f1214366740 1 -- 172.21.15.70:0/3331425049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c340e10 msgr2=0x55eb6c344bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.131+0000 7f1214366740 1 --2- 172.21.15.70:0/3331425049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c340e10 0x55eb6c344bb0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1204035000 tx=0x7f120402a8e0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.131+0000 7f1214366740 1 -- 172.21.15.70:0/3331425049 shutdown_connections 2024-06-04T04:59:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.131+0000 7f1214366740 1 --2- 172.21.15.70:0/3331425049 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55eb6c3bc0d0 0x55eb6c3be4f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.131+0000 7f1214366740 1 --2- 172.21.15.70:0/3331425049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c340e10 0x55eb6c344bb0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.131+0000 7f1214366740 1 --2- 172.21.15.70:0/3331425049 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb6c332060 0x55eb6c344670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.131+0000 7f11f27fc640 0 client.16206 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T04:59:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.131+0000 7f1214366740 1 --2- 172.21.15.70:0/3331425049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb6c331740 0x55eb6c349300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.131+0000 7f1214366740 1 -- 172.21.15.70:0/3331425049 >> 172.21.15.70:0/3331425049 conn(0x55eb6c32dd30 msgr2=0x55eb6c33e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.131+0000 7f1214366740 1 -- 172.21.15.70:0/3331425049 shutdown_connections 2024-06-04T04:59:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.131+0000 7f1214366740 1 -- 172.21.15.70:0/3331425049 wait complete. 2024-06-04T04:59:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.131+0000 7f1214366740 20 client.16206 trim_cache size 0 max 16384 2024-06-04T04:59:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T04:59:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T04:59:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T04:59:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.132+0000 7f1211571640 1 -- 172.21.15.70:0/462418624 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f11f80b2a10 msgr2=0x7f11f80b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.132+0000 7f1211571640 1 --2- 172.21.15.70:0/462418624 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f11f80b2a10 0x7f11f80b4ed0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1204005e30 tx=0x7f12040095c0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.132+0000 7f1211571640 1 -- 172.21.15.70:0/462418624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f120c102340 msgr2=0x7f120c1995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T04:59:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.132+0000 7f1211571640 1 --2- 172.21.15.70:0/462418624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f120c102340 0x7f120c1995a0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f11f400e9f0 tx=0x7f11f400eeb0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.132+0000 7f1211571640 1 -- 172.21.15.70:0/462418624 shutdown_connections 2024-06-04T04:59:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.132+0000 7f1211571640 1 --2- 172.21.15.70:0/462418624 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f11f80b2a10 0x7f11f80b4ed0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.132+0000 7f1211571640 1 --2- 172.21.15.70:0/462418624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f120c107940 0x7f120c19d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.132+0000 7f1211571640 1 --2- 172.21.15.70:0/462418624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f120c102340 0x7f120c1995a0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.132+0000 7f1211571640 1 --2- 172.21.15.70:0/462418624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f120c101980 0x7f120c199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T04:59:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.132+0000 7f1211571640 1 -- 172.21.15.70:0/462418624 >> 172.21.15.70:0/462418624 conn(0x7f120c078ad0 msgr2=0x7f120c06a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T04:59:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.132+0000 7f1211571640 1 -- 172.21.15.70:0/462418624 shutdown_connections 2024-06-04T04:59:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T04:59:48.132+0000 7f1211571640 1 -- 172.21.15.70:0/462418624 wait complete. 2024-06-04T04:59:48.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 24236 3.61 MB/sec warmup 41 sec latency 24.232 ms 2024-06-04T04:59:49.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:49 smithi070 ceph-mon[32631]: pgmap v82: 97 pgs: 97 active+clean; 31 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-06-04T04:59:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:49 smithi112 ceph-mon[36453]: pgmap v82: 97 pgs: 97 active+clean; 31 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-06-04T04:59:49.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 24716 3.58 MB/sec warmup 42 sec latency 14.937 ms 2024-06-04T04:59:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:49 smithi098 ceph-mon[35569]: pgmap v82: 97 pgs: 97 active+clean; 31 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-06-04T04:59:50.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 25193 3.59 MB/sec warmup 43 sec latency 15.716 ms 2024-06-04T04:59:51.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:51 smithi070 ceph-mon[32631]: pgmap v83: 97 pgs: 97 active+clean; 31 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-06-04T04:59:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:51 smithi112 ceph-mon[36453]: pgmap v83: 97 pgs: 97 active+clean; 31 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-06-04T04:59:51.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 25484 3.52 MB/sec warmup 44 sec latency 15.112 ms 2024-06-04T04:59:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:51 smithi098 ceph-mon[35569]: pgmap v83: 97 pgs: 97 active+clean; 31 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-06-04T04:59:52.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 26038 3.49 MB/sec warmup 45 sec latency 15.162 ms 2024-06-04T04:59:53.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:53 smithi070 ceph-mon[32631]: pgmap v84: 97 pgs: 97 active+clean; 32 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 1.8 MiB/s wr, 200 op/s 2024-06-04T04:59:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:53 smithi112 ceph-mon[36453]: pgmap v84: 97 pgs: 97 active+clean; 32 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 1.8 MiB/s wr, 200 op/s 2024-06-04T04:59:53.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 26687 3.48 MB/sec warmup 46 sec latency 14.881 ms 2024-06-04T04:59:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:53 smithi098 ceph-mon[35569]: pgmap v84: 97 pgs: 97 active+clean; 32 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 1.8 MiB/s wr, 200 op/s 2024-06-04T04:59:54.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 27304 3.45 MB/sec warmup 47 sec latency 15.648 ms 2024-06-04T04:59:55.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:55 smithi070 ceph-mon[32631]: pgmap v85: 97 pgs: 97 active+clean; 32 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-06-04T04:59:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:55 smithi112 ceph-mon[36453]: pgmap v85: 97 pgs: 97 active+clean; 32 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-06-04T04:59:55.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 28074 3.50 MB/sec warmup 48 sec latency 15.152 ms 2024-06-04T04:59:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:55 smithi098 ceph-mon[35569]: pgmap v85: 97 pgs: 97 active+clean; 32 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-06-04T04:59:56.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 28527 3.52 MB/sec warmup 49 sec latency 15.170 ms 2024-06-04T04:59:57.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:57 smithi070 ceph-mon[32631]: pgmap v86: 97 pgs: 97 active+clean; 33 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.9 MiB/s wr, 198 op/s 2024-06-04T04:59:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:57 smithi112 ceph-mon[36453]: pgmap v86: 97 pgs: 97 active+clean; 33 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.9 MiB/s wr, 198 op/s 2024-06-04T04:59:57.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 28830 3.48 MB/sec warmup 50 sec latency 13.178 ms 2024-06-04T04:59:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:57 smithi098 ceph-mon[35569]: pgmap v86: 97 pgs: 97 active+clean; 33 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.9 MiB/s wr, 198 op/s 2024-06-04T04:59:58.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 29202 3.42 MB/sec warmup 51 sec latency 24.656 ms 2024-06-04T04:59:59.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 04:59:59 smithi070 ceph-mon[32631]: pgmap v87: 97 pgs: 97 active+clean; 33 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-06-04T04:59:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 04:59:59 smithi112 ceph-mon[36453]: pgmap v87: 97 pgs: 97 active+clean; 33 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-06-04T04:59:59.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 29680 3.39 MB/sec warmup 52 sec latency 56.781 ms 2024-06-04T04:59:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 04:59:59 smithi098 ceph-mon[35569]: pgmap v87: 97 pgs: 97 active+clean; 33 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-06-04T05:00:00.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:00 smithi070 ceph-mon[32631]: overall HEALTH_OK 2024-06-04T05:00:00.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:00 smithi112 ceph-mon[36453]: overall HEALTH_OK 2024-06-04T05:00:00.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 30343 3.39 MB/sec warmup 53 sec latency 15.481 ms 2024-06-04T05:00:00.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:00 smithi098 ceph-mon[35569]: overall HEALTH_OK 2024-06-04T05:00:01.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.230+0000 7ff85eb12640 1 -- 172.21.15.70:0/1209403350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff858107940 msgr2=0x7ff858107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:01.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.230+0000 7ff85eb12640 1 --2- 172.21.15.70:0/1209403350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff858107940 0x7ff858107d20 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff84c009930 tx=0x7ff84c02a420 comp rx=0 tx=0).stop 2024-06-04T05:00:01.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.230+0000 7ff85eb12640 1 -- 172.21.15.70:0/1209403350 shutdown_connections 2024-06-04T05:00:01.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.230+0000 7ff85eb12640 1 --2- 172.21.15.70:0/1209403350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff858101980 0x7ff858101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.230+0000 7ff85eb12640 1 --2- 172.21.15.70:0/1209403350 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff858102340 0x7ff85810e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.230+0000 7ff85eb12640 1 --2- 172.21.15.70:0/1209403350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff858107940 0x7ff858107d20 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.230+0000 7ff85eb12640 1 -- 172.21.15.70:0/1209403350 >> 172.21.15.70:0/1209403350 conn(0x7ff858078ad0 msgr2=0x7ff858078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.230+0000 7ff85eb12640 1 -- 172.21.15.70:0/1209403350 shutdown_connections 2024-06-04T05:00:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.230+0000 7ff85eb12640 1 -- 172.21.15.70:0/1209403350 wait complete. 2024-06-04T05:00:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85eb12640 1 Processor -- start 2024-06-04T05:00:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85eb12640 1 -- start start 2024-06-04T05:00:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85eb12640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff858101980 0x7ff858199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85eb12640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff858102340 0x7ff85819a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85eb12640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff858107940 0x7ff8581a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85eb12640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff858111a80 con 0x7ff858101980 2024-06-04T05:00:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85eb12640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff858111900 con 0x7ff858102340 2024-06-04T05:00:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85eb12640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff858111c00 con 0x7ff858107940 2024-06-04T05:00:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85c8ae640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff858101980 0x7ff858199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85c8ae640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff858101980 0x7ff858199fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35666/0 (socket says 172.21.15.70:35666) 2024-06-04T05:00:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85c8ae640 1 -- 172.21.15.70:0/3069380979 learned_addr learned my addr 172.21.15.70:0/3069380979 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85d0af640 1 --2- 172.21.15.70:0/3069380979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff858107940 0x7ff8581a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff857fff640 1 --2- 172.21.15.70:0/3069380979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff858102340 0x7ff85819a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85c8ae640 1 -- 172.21.15.70:0/3069380979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff858107940 msgr2=0x7ff8581a5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85c8ae640 1 --2- 172.21.15.70:0/3069380979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff858107940 0x7ff8581a5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85c8ae640 1 -- 172.21.15.70:0/3069380979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff858102340 msgr2=0x7ff85819a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85c8ae640 1 --2- 172.21.15.70:0/3069380979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff858102340 0x7ff85819a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85c8ae640 1 -- 172.21.15.70:0/3069380979 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8581a5590 con 0x7ff858101980 2024-06-04T05:00:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.231+0000 7ff85c8ae640 1 --2- 172.21.15.70:0/3069380979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff858101980 0x7ff858199fd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff84c005940 tx=0x7ff84c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.232+0000 7ff855ffb640 1 -- 172.21.15.70:0/3069380979 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c03f030 con 0x7ff858101980 2024-06-04T05:00:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.232+0000 7ff855ffb640 1 -- 172.21.15.70:0/3069380979 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff84c004440 con 0x7ff858101980 2024-06-04T05:00:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.232+0000 7ff85eb12640 1 -- 172.21.15.70:0/3069380979 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8581a5760 con 0x7ff858101980 2024-06-04T05:00:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.232+0000 7ff855ffb640 1 -- 172.21.15.70:0/3069380979 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c035070 con 0x7ff858101980 2024-06-04T05:00:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.232+0000 7ff85eb12640 1 -- 172.21.15.70:0/3069380979 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8581a5bc0 con 0x7ff858101980 2024-06-04T05:00:01.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.234+0000 7ff855ffb640 1 -- 172.21.15.70:0/3069380979 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff84c042030 con 0x7ff858101980 2024-06-04T05:00:01.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.234+0000 7ff86181f740 1 Processor -- start 2024-06-04T05:00:01.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff86181f740 1 -- start start 2024-06-04T05:00:01.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff86181f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468e820 0x564fd468ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff86181f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd468f140 0x564fd46994c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff86181f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd4699a00 0x564fd469bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff86181f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564fd4600c30 con 0x564fd468e820 2024-06-04T05:00:01.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff86181f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564fd45f07d0 con 0x564fd4699a00 2024-06-04T05:00:01.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff86181f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564fd45f4810 con 0x564fd468f140 2024-06-04T05:00:01.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff85d0af640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468e820 0x564fd468ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff85d0af640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468e820 0x564fd468ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35676/0 (socket says 172.21.15.70:35676) 2024-06-04T05:00:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff85d0af640 1 -- 172.21.15.70:0/4145384963 learned_addr learned my addr 172.21.15.70:0/4145384963 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff857fff640 1 --2- 172.21.15.70:0/4145384963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd4699a00 0x564fd469bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff85c8ae640 1 --2- 172.21.15.70:0/4145384963 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd468f140 0x564fd46994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff85d0af640 1 -- 172.21.15.70:0/4145384963 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd468f140 msgr2=0x564fd46994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff85d0af640 1 --2- 172.21.15.70:0/4145384963 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd468f140 0x564fd46994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff85d0af640 1 -- 172.21.15.70:0/4145384963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd4699a00 msgr2=0x564fd469bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff85d0af640 1 --2- 172.21.15.70:0/4145384963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd4699a00 0x564fd469bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff85d0af640 1 -- 172.21.15.70:0/4145384963 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fd4305210 con 0x564fd468e820 2024-06-04T05:00:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff855ffb640 1 --2- 172.21.15.70:0/3069380979 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8480b2a10 0x7ff8480b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.235+0000 7ff85d0af640 1 --2- 172.21.15.70:0/4145384963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468e820 0x564fd468ec00 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff850009f50 tx=0x7ff8500118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd76ccd10f62269 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.236+0000 7ff85c8ae640 1 --2- 172.21.15.70:0/3069380979 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8480b2a10 0x7ff8480b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.236+0000 7ff842ffd640 1 -- 172.21.15.70:0/4145384963 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff850004d20 con 0x564fd468e820 2024-06-04T05:00:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.236+0000 7ff842ffd640 1 -- 172.21.15.70:0/4145384963 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff850004050 con 0x564fd468e820 2024-06-04T05:00:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.236+0000 7ff842ffd640 1 -- 172.21.15.70:0/4145384963 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff850005220 con 0x564fd468e820 2024-06-04T05:00:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.236+0000 7ff855ffb640 1 -- 172.21.15.70:0/3069380979 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ff84c102a90 con 0x7ff858101980 2024-06-04T05:00:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.236+0000 7ff86181f740 1 -- 172.21.15.70:0/4145384963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468e820 msgr2=0x564fd468ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.236+0000 7ff85c8ae640 1 --2- 172.21.15.70:0/3069380979 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8480b2a10 0x7ff8480b4ed0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff84c100290 tx=0x7ff84c1065e0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.236+0000 7ff86181f740 1 --2- 172.21.15.70:0/4145384963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468e820 0x564fd468ec00 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff850009f50 tx=0x7ff8500118d0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.236+0000 7ff86181f740 1 -- 172.21.15.70:0/4145384963 shutdown_connections 2024-06-04T05:00:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.236+0000 7ff86181f740 1 --2- 172.21.15.70:0/4145384963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd4699a00 0x564fd469bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.236+0000 7ff86181f740 1 --2- 172.21.15.70:0/4145384963 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd468f140 0x564fd46994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.236+0000 7ff86181f740 1 --2- 172.21.15.70:0/4145384963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468e820 0x564fd468ec00 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.236+0000 7ff86181f740 1 -- 172.21.15.70:0/4145384963 >> 172.21.15.70:0/4145384963 conn(0x564fd468ad30 msgr2=0x564fd468b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.236+0000 7ff86181f740 1 -- 172.21.15.70:0/4145384963 shutdown_connections 2024-06-04T05:00:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.236+0000 7ff86181f740 1 -- 172.21.15.70:0/4145384963 wait complete. 2024-06-04T05:00:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.237+0000 7ff86181f740 1 -- start start 2024-06-04T05:00:01.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.237+0000 7ff86181f740 1 Processor -- start 2024-06-04T05:00:01.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.237+0000 7ff86181f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468e820 0x564fd4696960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.237+0000 7ff86181f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd468f140 0x564fd4696ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.237+0000 7ff86181f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd4699a00 0x564fd46973e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.237+0000 7ff86181f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564fd42d6790 con 0x564fd468e820 2024-06-04T05:00:01.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.237+0000 7ff86181f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564fd45ac890 con 0x564fd4699a00 2024-06-04T05:00:01.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.237+0000 7ff86181f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564fd469c630 con 0x564fd468f140 2024-06-04T05:00:01.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.237+0000 7ff86181f740 10 client.0 _ll_register_callbacks cb 0x564fd4304a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:01.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.237+0000 7ff85d0af640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468e820 0x564fd4696960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.237+0000 7ff857fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd468f140 0x564fd4696ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.237+0000 7ff85d0af640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468e820 0x564fd4696960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35684/0 (socket says 172.21.15.70:35684) 2024-06-04T05:00:01.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.237+0000 7ff857fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd468f140 0x564fd4696ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41650/0 (socket says 172.21.15.70:41650) 2024-06-04T05:00:01.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.237+0000 7ff85d0af640 1 -- 172.21.15.70:0/646013849 learned_addr learned my addr 172.21.15.70:0/646013849 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.238+0000 7ff85d0af640 1 --2- 172.21.15.70:0/646013849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd4699a00 0x564fd46973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.238+0000 7ff857fff640 1 -- 172.21.15.70:0/646013849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd4699a00 msgr2=0x564fd46973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.238+0000 7ff857fff640 1 --2- 172.21.15.70:0/646013849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd4699a00 0x564fd46973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.238+0000 7ff857fff640 1 -- 172.21.15.70:0/646013849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468e820 msgr2=0x564fd4696960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.238+0000 7ff857fff640 1 --2- 172.21.15.70:0/646013849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468e820 0x564fd4696960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.238+0000 7ff857fff640 1 -- 172.21.15.70:0/646013849 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fd43d1620 con 0x564fd468f140 2024-06-04T05:00:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.238+0000 7ff85d0af640 1 --2- 172.21.15.70:0/646013849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468e820 0x564fd4696960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.238+0000 7ff857fff640 1 --2- 172.21.15.70:0/646013849 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd468f140 0x564fd4696ea0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x564fd468bec0 tx=0x7ff84400de70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d32bbc6cec13a79b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.238+0000 7ff837fff640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:00:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.238+0000 7ff837fff640 1 -- 172.21.15.70:0/646013849 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff844005020 con 0x564fd468f140 2024-06-04T05:00:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.238+0000 7ff837fff640 1 -- 172.21.15.70:0/646013849 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ff8440051c0 con 0x564fd468f140 2024-06-04T05:00:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.238+0000 7ff86181f740 1 -- 172.21.15.70:0/646013849 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff850004050 con 0x564fd468f140 2024-06-04T05:00:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.238+0000 7ff837fff640 1 -- 172.21.15.70:0/646013849 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff8440053c0 con 0x564fd468f140 2024-06-04T05:00:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.238+0000 7ff86181f740 10 client.25678 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.238+0000 7ff86181f740 1 -- 172.21.15.70:0/646013849 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564fd46a83a0 con 0x564fd468f140 2024-06-04T05:00:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.238+0000 7ff837fff640 1 -- 172.21.15.70:0/646013849 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff850004050 con 0x564fd468f140 2024-06-04T05:00:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.239+0000 7ff837fff640 1 -- 172.21.15.70:0/646013849 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff844019030 con 0x564fd468f140 2024-06-04T05:00:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.239+0000 7ff86181f740 10 client.25678 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.239+0000 7ff86181f740 10 client.25678 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.239+0000 7ff86181f740 20 client.25678 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.239+0000 7ff86181f740 1 --2- 172.21.15.70:0/646013849 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564fd46a9860 0x564fd46abcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.239+0000 7ff86181f740 4 client.25678 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:00:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.239+0000 7ff86181f740 1 -- 172.21.15.70:0/646013849 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564fd4602b80 con 0x564fd46a9860 2024-06-04T05:00:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.239+0000 7ff85d0af640 1 --2- 172.21.15.70:0/646013849 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564fd46a9860 0x564fd46abcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.239+0000 7ff837fff640 10 client.25678 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.239+0000 7ff85d0af640 1 --2- 172.21.15.70:0/646013849 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564fd46a9860 0x564fd46abcc0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a6b3d4c835e3a5d server_cookie=d391f4d0587adf47 in_seq=0 out_seq=0 2024-06-04T05:00:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.240+0000 7ff837fff640 1 -- 172.21.15.70:0/646013849 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564fd4602b80 con 0x564fd46a9860 2024-06-04T05:00:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.240+0000 7ff837fff640 10 client.25678 handle_command_reply: tid=0 2024-06-04T05:00:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.240+0000 7ff86181f740 1 client.25678 shutdown 2024-06-04T05:00:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.240+0000 7ff86181f740 1 -- 172.21.15.70:0/646013849 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd468f140 msgr2=0x564fd4696ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.240+0000 7ff86181f740 1 --2- 172.21.15.70:0/646013849 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd468f140 0x564fd4696ea0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x564fd468bec0 tx=0x7ff84400de70 comp rx=0 tx=0).stop 2024-06-04T05:00:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.240+0000 7ff86181f740 1 -- 172.21.15.70:0/646013849 shutdown_connections 2024-06-04T05:00:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.240+0000 7ff86181f740 1 --2- 172.21.15.70:0/646013849 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564fd46a9860 0x564fd46abcc0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.240+0000 7ff86181f740 1 --2- 172.21.15.70:0/646013849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd4699a00 0x564fd46973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.240+0000 7ff86181f740 1 --2- 172.21.15.70:0/646013849 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd468f140 0x564fd4696ea0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.240+0000 7ff837fff640 0 client.25678 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.240+0000 7ff86181f740 1 --2- 172.21.15.70:0/646013849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468e820 0x564fd4696960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.240+0000 7ff86181f740 1 -- 172.21.15.70:0/646013849 >> 172.21.15.70:0/646013849 conn(0x564fd468ad30 msgr2=0x564fd4689fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.240+0000 7ff86181f740 1 -- 172.21.15.70:0/646013849 shutdown_connections 2024-06-04T05:00:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.240+0000 7ff86181f740 1 -- 172.21.15.70:0/646013849 wait complete. 2024-06-04T05:00:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.240+0000 7ff86181f740 20 client.25678 trim_cache size 0 max 16384 2024-06-04T05:00:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.258+0000 7ff86181f740 1 Processor -- start 2024-06-04T05:00:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.258+0000 7ff86181f740 1 -- start start 2024-06-04T05:00:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.258+0000 7ff86181f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd468e820 0x564fd46b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.258+0000 7ff86181f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468f140 0x564fd46b1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.258+0000 7ff86181f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd4699a00 0x564fd4693bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.258+0000 7ff86181f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff8581ae660 con 0x564fd468f140 2024-06-04T05:00:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.258+0000 7ff86181f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8581b0200 con 0x564fd468e820 2024-06-04T05:00:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.258+0000 7ff86181f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff8581aed30 con 0x564fd4699a00 2024-06-04T05:00:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.258+0000 7ff857fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468f140 0x564fd46b1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.258+0000 7ff857fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468f140 0x564fd46b1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35692/0 (socket says 172.21.15.70:35692) 2024-06-04T05:00:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.258+0000 7ff85d0af640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd468e820 0x564fd46b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.258+0000 7ff857fff640 1 -- 172.21.15.70:0/4257314630 learned_addr learned my addr 172.21.15.70:0/4257314630 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.259+0000 7ff85d0af640 1 --2- 172.21.15.70:0/4257314630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd4699a00 0x564fd4693bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.259+0000 7ff85d0af640 1 -- 172.21.15.70:0/4257314630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd4699a00 msgr2=0x564fd4693bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.259+0000 7ff85d0af640 1 --2- 172.21.15.70:0/4257314630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd4699a00 0x564fd4693bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.259+0000 7ff85d0af640 1 -- 172.21.15.70:0/4257314630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468f140 msgr2=0x564fd46b1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.259+0000 7ff85d0af640 1 --2- 172.21.15.70:0/4257314630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468f140 0x564fd46b1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.259+0000 7ff85d0af640 1 -- 172.21.15.70:0/4257314630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fd468f8b0 con 0x564fd468e820 2024-06-04T05:00:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.259+0000 7ff857fff640 1 --2- 172.21.15.70:0/4257314630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468f140 0x564fd46b1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.259+0000 7ff85d0af640 1 --2- 172.21.15.70:0/4257314630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd468e820 0x564fd46b15d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff850009d60 tx=0x7ff8500118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27d1d51b62b55898 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.259+0000 7ff8427fc640 1 -- 172.21.15.70:0/4257314630 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff850003a30 con 0x564fd468e820 2024-06-04T05:00:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.259+0000 7ff8427fc640 1 -- 172.21.15.70:0/4257314630 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff850003bd0 con 0x564fd468e820 2024-06-04T05:00:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.259+0000 7ff8427fc640 1 -- 172.21.15.70:0/4257314630 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85000f480 con 0x564fd468e820 2024-06-04T05:00:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.260+0000 7ff86181f740 1 -- 172.21.15.70:0/4257314630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd468e820 msgr2=0x564fd46b15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.260+0000 7ff86181f740 1 --2- 172.21.15.70:0/4257314630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd468e820 0x564fd46b15d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff850009d60 tx=0x7ff8500118d0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.260+0000 7ff86181f740 1 -- 172.21.15.70:0/4257314630 shutdown_connections 2024-06-04T05:00:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.260+0000 7ff86181f740 1 --2- 172.21.15.70:0/4257314630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd4699a00 0x564fd4693bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.260+0000 7ff86181f740 1 --2- 172.21.15.70:0/4257314630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd468f140 0x564fd46b1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.260+0000 7ff86181f740 1 --2- 172.21.15.70:0/4257314630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd468e820 0x564fd46b15d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.260+0000 7ff86181f740 1 -- 172.21.15.70:0/4257314630 >> 172.21.15.70:0/4257314630 conn(0x564fd468ad30 msgr2=0x564fd46971b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.260+0000 7ff86181f740 1 -- 172.21.15.70:0/4257314630 shutdown_connections 2024-06-04T05:00:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.260+0000 7ff86181f740 1 -- 172.21.15.70:0/4257314630 wait complete. 2024-06-04T05:00:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.260+0000 7ff86181f740 1 -- start start 2024-06-04T05:00:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.260+0000 7ff86181f740 1 Processor -- start 2024-06-04T05:00:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff86181f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd468e820 0x564fd46a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff86181f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd468f140 0x564fd469d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff86181f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd4699a00 0x564fd469d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff86181f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff8581afad0 con 0x564fd4699a00 2024-06-04T05:00:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff86181f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8581af400 con 0x564fd468f140 2024-06-04T05:00:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff86181f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff8581ac640 con 0x564fd468e820 2024-06-04T05:00:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff86181f740 10 client.0 _ll_register_callbacks cb 0x7ff844004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff85d0af640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd4699a00 0x564fd469d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff85d0af640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd4699a00 0x564fd469d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35708/0 (socket says 172.21.15.70:35708) 2024-06-04T05:00:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff85d0af640 1 -- 172.21.15.70:0/901863056 learned_addr learned my addr 172.21.15.70:0/901863056 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff857fff640 1 --2- 172.21.15.70:0/901863056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd468f140 0x564fd469d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff85d0af640 1 --2- 172.21.15.70:0/901863056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd468e820 0x564fd46a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff857fff640 1 -- 172.21.15.70:0/901863056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd468e820 msgr2=0x564fd46a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff857fff640 1 --2- 172.21.15.70:0/901863056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd468e820 0x564fd46a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff857fff640 1 -- 172.21.15.70:0/901863056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd4699a00 msgr2=0x564fd469d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff857fff640 1 --2- 172.21.15.70:0/901863056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd4699a00 0x564fd469d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff857fff640 1 -- 172.21.15.70:0/901863056 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fd469df70 con 0x564fd468f140 2024-06-04T05:00:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff857fff640 1 --2- 172.21.15.70:0/901863056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd468f140 0x564fd469d230 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff8440115e0 tx=0x7ff844003a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eea3991032a1358b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.261+0000 7ff837fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:00:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.262+0000 7ff837fff640 1 -- 172.21.15.70:0/901863056 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff844031060 con 0x564fd468f140 2024-06-04T05:00:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.262+0000 7ff86181f740 1 -- 172.21.15.70:0/901863056 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff850003bd0 con 0x564fd468f140 2024-06-04T05:00:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.262+0000 7ff837fff640 1 -- 172.21.15.70:0/901863056 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ff844023430 con 0x564fd468f140 2024-06-04T05:00:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.262+0000 7ff837fff640 1 -- 172.21.15.70:0/901863056 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff844027070 con 0x564fd468f140 2024-06-04T05:00:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.262+0000 7ff86181f740 10 client.25625 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.262+0000 7ff86181f740 1 -- 172.21.15.70:0/901863056 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564fd4714420 con 0x564fd468f140 2024-06-04T05:00:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.262+0000 7ff837fff640 1 -- 172.21.15.70:0/901863056 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff850003bd0 con 0x564fd468f140 2024-06-04T05:00:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.262+0000 7ff837fff640 1 -- 172.21.15.70:0/901863056 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff844019070 con 0x564fd468f140 2024-06-04T05:00:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.262+0000 7ff86181f740 10 client.25625 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.262+0000 7ff86181f740 10 client.25625 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.262+0000 7ff86181f740 20 client.25625 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.262+0000 7ff86181f740 1 --2- 172.21.15.70:0/901863056 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564fd4714c30 0x564fd4717050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:01.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.262+0000 7ff86181f740 4 client.25625 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:00:01.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.262+0000 7ff86181f740 1 -- 172.21.15.70:0/901863056 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564fd45fc360 con 0x564fd4714c30 2024-06-04T05:00:01.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.262+0000 7ff85d0af640 1 --2- 172.21.15.70:0/901863056 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564fd4714c30 0x564fd4717050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:01.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.263+0000 7ff85d0af640 1 --2- 172.21.15.70:0/901863056 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564fd4714c30 0x564fd4717050 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf57d0035ed750af server_cookie=9bb38af4e203f513 in_seq=0 out_seq=0 2024-06-04T05:00:01.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.263+0000 7ff837fff640 10 client.25625 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.286+0000 7ff837fff640 1 -- 172.21.15.70:0/901863056 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x564fd45fc360 con 0x564fd4714c30 2024-06-04T05:00:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.286+0000 7ff837fff640 10 client.25625 handle_command_reply: tid=0 2024-06-04T05:00:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.286+0000 7ff86181f740 1 client.25625 shutdown 2024-06-04T05:00:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.286+0000 7ff86181f740 1 -- 172.21.15.70:0/901863056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd468f140 msgr2=0x564fd469d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.286+0000 7ff86181f740 1 --2- 172.21.15.70:0/901863056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd468f140 0x564fd469d230 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff8440115e0 tx=0x7ff844003a30 comp rx=0 tx=0).stop 2024-06-04T05:00:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.286+0000 7ff86181f740 1 -- 172.21.15.70:0/901863056 shutdown_connections 2024-06-04T05:00:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.286+0000 7ff86181f740 1 --2- 172.21.15.70:0/901863056 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564fd4714c30 0x564fd4717050 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.286+0000 7ff86181f740 1 --2- 172.21.15.70:0/901863056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564fd468f140 0x564fd469d230 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.286+0000 7ff86181f740 1 --2- 172.21.15.70:0/901863056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fd4699a00 0x564fd469d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.286+0000 7ff837fff640 0 client.25625 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.286+0000 7ff86181f740 1 --2- 172.21.15.70:0/901863056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564fd468e820 0x564fd46a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.286+0000 7ff86181f740 1 -- 172.21.15.70:0/901863056 >> 172.21.15.70:0/901863056 conn(0x564fd468ad30 msgr2=0x564fd46901d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.286+0000 7ff86181f740 1 -- 172.21.15.70:0/901863056 shutdown_connections 2024-06-04T05:00:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.286+0000 7ff86181f740 1 -- 172.21.15.70:0/901863056 wait complete. 2024-06-04T05:00:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.286+0000 7ff86181f740 20 client.25625 trim_cache size 0 max 16384 2024-06-04T05:00:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:00:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:00:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:00:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.287+0000 7ff85eb12640 1 -- 172.21.15.70:0/3069380979 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8480b2a10 msgr2=0x7ff8480b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.287+0000 7ff85eb12640 1 --2- 172.21.15.70:0/3069380979 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8480b2a10 0x7ff8480b4ed0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff84c100290 tx=0x7ff84c1065e0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.287+0000 7ff85eb12640 1 -- 172.21.15.70:0/3069380979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff858101980 msgr2=0x7ff858199fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.287+0000 7ff85eb12640 1 --2- 172.21.15.70:0/3069380979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff858101980 0x7ff858199fd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff84c005940 tx=0x7ff84c004040 comp rx=0 tx=0).stop 2024-06-04T05:00:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.287+0000 7ff85eb12640 1 -- 172.21.15.70:0/3069380979 shutdown_connections 2024-06-04T05:00:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.287+0000 7ff85eb12640 1 --2- 172.21.15.70:0/3069380979 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8480b2a10 0x7ff8480b4ed0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.287+0000 7ff85eb12640 1 --2- 172.21.15.70:0/3069380979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff858102340 0x7ff85819a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.287+0000 7ff85eb12640 1 --2- 172.21.15.70:0/3069380979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff858107940 0x7ff8581a5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.287+0000 7ff85eb12640 1 --2- 172.21.15.70:0/3069380979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff858101980 0x7ff858199fd0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.287+0000 7ff85eb12640 1 -- 172.21.15.70:0/3069380979 >> 172.21.15.70:0/3069380979 conn(0x7ff858078ad0 msgr2=0x7ff858104620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.287+0000 7ff85eb12640 1 -- 172.21.15.70:0/3069380979 shutdown_connections 2024-06-04T05:00:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:01.287+0000 7ff85eb12640 1 -- 172.21.15.70:0/3069380979 wait complete. 2024-06-04T05:00:01.587 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:01 smithi112 ceph-mon[36453]: pgmap v88: 97 pgs: 97 active+clean; 33 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-06-04T05:00:01.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:01 smithi070 ceph-mon[32631]: pgmap v88: 97 pgs: 97 active+clean; 33 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-06-04T05:00:01.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 30915 3.37 MB/sec warmup 54 sec latency 19.401 ms 2024-06-04T05:00:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:01 smithi098 ceph-mon[35569]: pgmap v88: 97 pgs: 97 active+clean; 33 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-06-04T05:00:02.715 INFO:tasks.workunit.client.0.smithi070.stdout: 1 31709 3.42 MB/sec warmup 55 sec latency 10.861 ms 2024-06-04T05:00:03.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:03 smithi070 ceph-mon[32631]: pgmap v89: 97 pgs: 97 active+clean; 32 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 1.6 MiB/s wr, 213 op/s 2024-06-04T05:00:03.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:00:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:03 smithi112 ceph-mon[36453]: pgmap v89: 97 pgs: 97 active+clean; 32 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 1.6 MiB/s wr, 213 op/s 2024-06-04T05:00:03.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 32181 3.44 MB/sec warmup 56 sec latency 14.357 ms 2024-06-04T05:00:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:03 smithi098 ceph-mon[35569]: pgmap v89: 97 pgs: 97 active+clean; 32 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 1.6 MiB/s wr, 213 op/s 2024-06-04T05:00:03.910 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:00:04.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 32434 3.39 MB/sec warmup 57 sec latency 12.578 ms 2024-06-04T05:00:05.318 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:05 smithi070 ceph-mon[32631]: pgmap v90: 97 pgs: 97 active+clean; 32 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 1.0 MiB/s wr, 146 op/s 2024-06-04T05:00:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:05 smithi112 ceph-mon[36453]: pgmap v90: 97 pgs: 97 active+clean; 32 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 1.0 MiB/s wr, 146 op/s 2024-06-04T05:00:05.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 32878 3.34 MB/sec warmup 58 sec latency 12.350 ms 2024-06-04T05:00:05.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.734+0000 7f1466dd0700 1 -- 172.21.15.70:0/3748253755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1460102440 msgr2=0x7f14601028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:05.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.735+0000 7f1466dd0700 1 --2- 172.21.15.70:0/3748253755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1460102440 0x7f14601028c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f144c009990 tx=0x7f144c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.735+0000 7f1466dd0700 1 -- 172.21.15.70:0/3748253755 shutdown_connections 2024-06-04T05:00:05.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.735+0000 7f1466dd0700 1 --2- 172.21.15.70:0/3748253755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1460102e00 0x7f146010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.735+0000 7f1466dd0700 1 --2- 172.21.15.70:0/3748253755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1460108520 0x7f1460108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.735+0000 7f1466dd0700 1 --2- 172.21.15.70:0/3748253755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1460102440 0x7f14601028c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.735+0000 7f1466dd0700 1 -- 172.21.15.70:0/3748253755 >> 172.21.15.70:0/3748253755 conn(0x7f14600793f0 msgr2=0x7f1460079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:05.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.735+0000 7f1466dd0700 1 -- 172.21.15.70:0/3748253755 shutdown_connections 2024-06-04T05:00:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.735+0000 7f1466dd0700 1 -- 172.21.15.70:0/3748253755 wait complete. 2024-06-04T05:00:05.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.735+0000 7f1466dd0700 1 Processor -- start 2024-06-04T05:00:05.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f1466dd0700 1 -- start start 2024-06-04T05:00:05.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f1466dd0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1460102440 0x7f1460104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f1466dd0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1460102e00 0x7f14601045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f1466dd0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1460108520 0x7f1460104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f1466dd0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f14601124f0 con 0x7f1460108520 2024-06-04T05:00:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f1466dd0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1460112370 con 0x7f1460102e00 2024-06-04T05:00:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f1466dd0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1460112670 con 0x7f1460102440 2024-06-04T05:00:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f146536d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1460108520 0x7f1460104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f146536d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1460108520 0x7f1460104b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35734/0 (socket says 172.21.15.70:35734) 2024-06-04T05:00:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f146536d700 1 -- 172.21.15.70:0/1230160351 learned_addr learned my addr 172.21.15.70:0/1230160351 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f145ffff700 1 --2- 172.21.15.70:0/1230160351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1460102e00 0x7f14601045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f1464b6c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1460102440 0x7f1460104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f145ffff700 1 -- 172.21.15.70:0/1230160351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1460102440 msgr2=0x7f1460104080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f145ffff700 1 --2- 172.21.15.70:0/1230160351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1460102440 0x7f1460104080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f145ffff700 1 -- 172.21.15.70:0/1230160351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1460108520 msgr2=0x7f1460104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f145ffff700 1 --2- 172.21.15.70:0/1230160351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1460108520 0x7f1460104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f145ffff700 1 -- 172.21.15.70:0/1230160351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14601a8160 con 0x7f1460102e00 2024-06-04T05:00:05.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f1464b6c700 1 --2- 172.21.15.70:0/1230160351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1460102440 0x7f1460104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:05.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.736+0000 7f146536d700 1 --2- 172.21.15.70:0/1230160351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1460108520 0x7f1460104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:05.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.737+0000 7f145ffff700 1 --2- 172.21.15.70:0/1230160351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1460102e00 0x7f14601045c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f144c0098e0 tx=0x7f144c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:05.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.737+0000 7f145dffb700 1 -- 172.21.15.70:0/1230160351 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c026070 con 0x7f1460102e00 2024-06-04T05:00:05.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.737+0000 7f145dffb700 1 -- 172.21.15.70:0/1230160351 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f144c021410 con 0x7f1460102e00 2024-06-04T05:00:05.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.737+0000 7f145dffb700 1 -- 172.21.15.70:0/1230160351 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c017650 con 0x7f1460102e00 2024-06-04T05:00:05.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.737+0000 7f1466dd0700 1 -- 172.21.15.70:0/1230160351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14601a83f0 con 0x7f1460102e00 2024-06-04T05:00:05.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.737+0000 7f1466dd0700 1 -- 172.21.15.70:0/1230160351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14601a87f0 con 0x7f1460102e00 2024-06-04T05:00:05.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.740+0000 7f145dffb700 1 -- 172.21.15.70:0/1230160351 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f144c029070 con 0x7f1460102e00 2024-06-04T05:00:05.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.741+0000 7f145dffb700 1 --2- 172.21.15.70:0/1230160351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f14500b2a70 0x7f14500b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.741+0000 7f1464b6c700 1 --2- 172.21.15.70:0/1230160351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f14500b2a70 0x7f14500b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.741+0000 7f145dffb700 1 -- 172.21.15.70:0/1230160351 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f144c0e3320 con 0x7f1460102e00 2024-06-04T05:00:05.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.741+0000 7f1471665b80 1 Processor -- start 2024-06-04T05:00:05.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.741+0000 7f1471665b80 1 -- start start 2024-06-04T05:00:05.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.741+0000 7f1471665b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969b4320 0x55c4969b4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.741+0000 7f1471665b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969ae8a0 0x55c4969aed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.742+0000 7f1471665b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969af260 0x55c4969bb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.742+0000 7f1471665b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c49690cd70 con 0x55c4969b4320 2024-06-04T05:00:05.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.742+0000 7f1471665b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c4968fc910 con 0x55c4969af260 2024-06-04T05:00:05.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.742+0000 7f1471665b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c496900950 con 0x55c4969ae8a0 2024-06-04T05:00:05.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.742+0000 7f145ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969ae8a0 0x55c4969aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.742+0000 7f145ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969ae8a0 0x55c4969aed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41688/0 (socket says 172.21.15.70:41688) 2024-06-04T05:00:05.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.742+0000 7f145ffff700 1 -- 172.21.15.70:0/2363661969 learned_addr learned my addr 172.21.15.70:0/2363661969 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:05.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.742+0000 7f1464b6c700 1 --2- 172.21.15.70:0/2363661969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969b4320 0x55c4969b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.742+0000 7f146536d700 1 --2- 172.21.15.70:0/2363661969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969af260 0x55c4969bb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.742+0000 7f1464b6c700 1 --2- 172.21.15.70:0/1230160351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f14500b2a70 0x7f14500b4f30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1454009fd0 tx=0x7f1454009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:05.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.742+0000 7f145ffff700 1 -- 172.21.15.70:0/2363661969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969af260 msgr2=0x55c4969bb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:05.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.742+0000 7f145ffff700 1 --2- 172.21.15.70:0/2363661969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969af260 0x55c4969bb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.742+0000 7f145ffff700 1 -- 172.21.15.70:0/2363661969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969b4320 msgr2=0x55c4969b4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:05.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.742+0000 7f145ffff700 1 --2- 172.21.15.70:0/2363661969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969b4320 0x55c4969b4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.742+0000 7f145ffff700 1 -- 172.21.15.70:0/2363661969 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4968a5fb0 con 0x55c4969ae8a0 2024-06-04T05:00:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:05 smithi098 ceph-mon[35569]: pgmap v90: 97 pgs: 97 active+clean; 32 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 1.0 MiB/s wr, 146 op/s 2024-06-04T05:00:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.742+0000 7f145ffff700 1 --2- 172.21.15.70:0/2363661969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969ae8a0 0x55c4969aed20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f144c009350 tx=0x7f144c0a9e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40f94dfcdafa09bb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.743+0000 7f144affd700 1 -- 172.21.15.70:0/2363661969 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c0aa630 con 0x55c4969ae8a0 2024-06-04T05:00:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.743+0000 7f144affd700 1 -- 172.21.15.70:0/2363661969 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f144c0ae9e0 con 0x55c4969ae8a0 2024-06-04T05:00:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.743+0000 7f144affd700 1 -- 172.21.15.70:0/2363661969 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c0aaaf0 con 0x55c4969ae8a0 2024-06-04T05:00:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.744+0000 7f1471665b80 1 -- 172.21.15.70:0/2363661969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969ae8a0 msgr2=0x55c4969aed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.744+0000 7f1471665b80 1 --2- 172.21.15.70:0/2363661969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969ae8a0 0x55c4969aed20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f144c009350 tx=0x7f144c0a9e70 comp rx=0 tx=0).stop 2024-06-04T05:00:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.744+0000 7f1471665b80 1 -- 172.21.15.70:0/2363661969 shutdown_connections 2024-06-04T05:00:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.744+0000 7f1471665b80 1 --2- 172.21.15.70:0/2363661969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969af260 0x55c4969bb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.744+0000 7f1471665b80 1 --2- 172.21.15.70:0/2363661969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969b4320 0x55c4969b4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.744+0000 7f1471665b80 1 --2- 172.21.15.70:0/2363661969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969ae8a0 0x55c4969aed20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.744+0000 7f1471665b80 1 -- 172.21.15.70:0/2363661969 >> 172.21.15.70:0/2363661969 conn(0x55c4968da240 msgr2=0x55c4969a93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.744+0000 7f1471665b80 1 -- 172.21.15.70:0/2363661969 shutdown_connections 2024-06-04T05:00:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.744+0000 7f1471665b80 1 -- 172.21.15.70:0/2363661969 wait complete. 2024-06-04T05:00:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.744+0000 7f1471665b80 1 -- start start 2024-06-04T05:00:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.744+0000 7f1471665b80 1 Processor -- start 2024-06-04T05:00:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f1471665b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969ae8a0 0x55c4969bad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f1471665b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969af260 0x55c4969b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f1471665b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969b4320 0x55c4969b8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f1471665b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c496802fa0 con 0x55c4969ae8a0 2024-06-04T05:00:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f1471665b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c496864da0 con 0x55c4969b4320 2024-06-04T05:00:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f1471665b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c496900950 con 0x55c4969af260 2024-06-04T05:00:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f1471665b80 10 client.0 _ll_register_callbacks cb 0x55c496897950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f1464b6c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969ae8a0 0x55c4969bad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f1464b6c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969ae8a0 0x55c4969bad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35760/0 (socket says 172.21.15.70:35760) 2024-06-04T05:00:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f1464b6c700 1 -- 172.21.15.70:0/718833855 learned_addr learned my addr 172.21.15.70:0/718833855 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f146536d700 1 --2- 172.21.15.70:0/718833855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969b4320 0x55c4969b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f145ffff700 1 --2- 172.21.15.70:0/718833855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969af260 0x55c4969b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f1464b6c700 1 -- 172.21.15.70:0/718833855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969af260 msgr2=0x55c4969b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f1464b6c700 1 --2- 172.21.15.70:0/718833855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969af260 0x55c4969b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f1464b6c700 1 -- 172.21.15.70:0/718833855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969b4320 msgr2=0x55c4969b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f1464b6c700 1 --2- 172.21.15.70:0/718833855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969b4320 0x55c4969b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f1464b6c700 1 -- 172.21.15.70:0/718833855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c49684c8d0 con 0x55c4969ae8a0 2024-06-04T05:00:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f145ffff700 1 --2- 172.21.15.70:0/718833855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969af260 0x55c4969b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f1464b6c700 1 --2- 172.21.15.70:0/718833855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969ae8a0 0x55c4969bad70 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1454003ad0 tx=0x7f14540047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db931a9bbf47469 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.745+0000 7f143ffff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.746+0000 7f143ffff700 1 -- 172.21.15.70:0/718833855 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145401f890 con 0x55c4969ae8a0 2024-06-04T05:00:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.746+0000 7f143ffff700 1 -- 172.21.15.70:0/718833855 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f145401fa30 con 0x55c4969ae8a0 2024-06-04T05:00:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.746+0000 7f1471665b80 1 -- 172.21.15.70:0/718833855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f144c0ae9e0 con 0x55c4969ae8a0 2024-06-04T05:00:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.746+0000 7f143ffff700 1 -- 172.21.15.70:0/718833855 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f145401fc30 con 0x55c4969ae8a0 2024-06-04T05:00:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.746+0000 7f1471665b80 10 client.16236 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.746+0000 7f1471665b80 1 -- 172.21.15.70:0/718833855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c4968b6760 con 0x55c4969ae8a0 2024-06-04T05:00:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.746+0000 7f143ffff700 1 -- 172.21.15.70:0/718833855 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f144c0ae9e0 con 0x55c4969ae8a0 2024-06-04T05:00:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.746+0000 7f143ffff700 1 -- 172.21.15.70:0/718833855 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f1454026020 con 0x55c4969ae8a0 2024-06-04T05:00:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.746+0000 7f1471665b80 10 client.16236 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.746+0000 7f1471665b80 10 client.16236 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.746+0000 7f1471665b80 20 client.16236 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.746+0000 7f1471665b80 1 --2- 172.21.15.70:0/718833855 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c4969c7f90 0x55c4969ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.746+0000 7f1471665b80 4 client.16236 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:00:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.746+0000 7f1471665b80 1 -- 172.21.15.70:0/718833855 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c49690ecc0 con 0x55c4969c7f90 2024-06-04T05:00:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.746+0000 7f145ffff700 1 --2- 172.21.15.70:0/718833855 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c4969c7f90 0x55c4969ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.747+0000 7f143ffff700 10 client.16236 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.747+0000 7f145ffff700 1 --2- 172.21.15.70:0/718833855 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c4969c7f90 0x55c4969ca3c0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8760ab9d19c548ca server_cookie=19d957d0b0c3b195 in_seq=0 out_seq=0 2024-06-04T05:00:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.747+0000 7f143ffff700 1 -- 172.21.15.70:0/718833855 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c49690ecc0 con 0x55c4969c7f90 2024-06-04T05:00:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.747+0000 7f143ffff700 10 client.16236 handle_command_reply: tid=0 2024-06-04T05:00:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.747+0000 7f1471665b80 1 client.16236 shutdown 2024-06-04T05:00:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.748+0000 7f1471665b80 1 -- 172.21.15.70:0/718833855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969ae8a0 msgr2=0x55c4969bad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.748+0000 7f1471665b80 1 --2- 172.21.15.70:0/718833855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969ae8a0 0x55c4969bad70 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1454003ad0 tx=0x7f14540047c0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.748+0000 7f1471665b80 1 -- 172.21.15.70:0/718833855 shutdown_connections 2024-06-04T05:00:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.748+0000 7f1471665b80 1 --2- 172.21.15.70:0/718833855 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c4969c7f90 0x55c4969ca3c0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.748+0000 7f1471665b80 1 --2- 172.21.15.70:0/718833855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969b4320 0x55c4969b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.748+0000 7f143ffff700 0 client.16236 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.748+0000 7f1471665b80 1 --2- 172.21.15.70:0/718833855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969ae8a0 0x55c4969bad70 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.748+0000 7f1471665b80 1 --2- 172.21.15.70:0/718833855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969af260 0x55c4969b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.748+0000 7f1471665b80 1 -- 172.21.15.70:0/718833855 >> 172.21.15.70:0/718833855 conn(0x55c4968da240 msgr2=0x55c4969a8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.748+0000 7f1471665b80 1 -- 172.21.15.70:0/718833855 shutdown_connections 2024-06-04T05:00:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.748+0000 7f1471665b80 1 -- 172.21.15.70:0/718833855 wait complete. 2024-06-04T05:00:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.748+0000 7f1471665b80 20 client.16236 trim_cache size 0 max 16384 2024-06-04T05:00:05.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.772+0000 7f1471665b80 1 Processor -- start 2024-06-04T05:00:05.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.772+0000 7f1471665b80 1 -- start start 2024-06-04T05:00:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f1471665b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969af260 0x55c4969a9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f1471665b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969b4320 0x55c4969b9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f1471665b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969c44a0 0x55c4969b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f1471665b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f14601afea0 con 0x55c4969af260 2024-06-04T05:00:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f1471665b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f14601b1a40 con 0x55c4969c44a0 2024-06-04T05:00:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f1471665b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f14601b0570 con 0x55c4969b4320 2024-06-04T05:00:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f1464b6c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969af260 0x55c4969a9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f145ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969b4320 0x55c4969b9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f1464b6c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969af260 0x55c4969a9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35768/0 (socket says 172.21.15.70:35768) 2024-06-04T05:00:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f1464b6c700 1 -- 172.21.15.70:0/2315719373 learned_addr learned my addr 172.21.15.70:0/2315719373 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f146536d700 1 --2- 172.21.15.70:0/2315719373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969c44a0 0x55c4969b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f145ffff700 1 -- 172.21.15.70:0/2315719373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969c44a0 msgr2=0x55c4969b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f145ffff700 1 --2- 172.21.15.70:0/2315719373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969c44a0 0x55c4969b0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f145ffff700 1 -- 172.21.15.70:0/2315719373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969af260 msgr2=0x55c4969a9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:05.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f145ffff700 1 --2- 172.21.15.70:0/2315719373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969af260 0x55c4969a9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f145ffff700 1 -- 172.21.15.70:0/2315719373 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4969ab380 con 0x55c4969b4320 2024-06-04T05:00:05.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f1464b6c700 1 --2- 172.21.15.70:0/2315719373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969af260 0x55c4969a9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:05.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f145ffff700 1 --2- 172.21.15.70:0/2315719373 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969b4320 0x55c4969b9fb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f144c0920b0 tx=0x7f144c0b1050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=871427d75c38df3f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:05.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f144a7fc700 1 -- 172.21.15.70:0/2315719373 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c0a9da0 con 0x55c4969b4320 2024-06-04T05:00:05.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f144a7fc700 1 -- 172.21.15.70:0/2315719373 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f144c0ae9e0 con 0x55c4969b4320 2024-06-04T05:00:05.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.773+0000 7f144a7fc700 1 -- 172.21.15.70:0/2315719373 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c0aaaf0 con 0x55c4969b4320 2024-06-04T05:00:05.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.774+0000 7f1471665b80 1 -- 172.21.15.70:0/2315719373 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969b4320 msgr2=0x55c4969b9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:05.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.774+0000 7f1471665b80 1 --2- 172.21.15.70:0/2315719373 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969b4320 0x55c4969b9fb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f144c0920b0 tx=0x7f144c0b1050 comp rx=0 tx=0).stop 2024-06-04T05:00:05.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.774+0000 7f1471665b80 1 -- 172.21.15.70:0/2315719373 shutdown_connections 2024-06-04T05:00:05.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.774+0000 7f1471665b80 1 --2- 172.21.15.70:0/2315719373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969c44a0 0x55c4969b0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.774+0000 7f1471665b80 1 --2- 172.21.15.70:0/2315719373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969af260 0x55c4969a9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.774+0000 7f1471665b80 1 --2- 172.21.15.70:0/2315719373 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969b4320 0x55c4969b9fb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.774+0000 7f1471665b80 1 -- 172.21.15.70:0/2315719373 >> 172.21.15.70:0/2315719373 conn(0x55c4968da240 msgr2=0x55c4969b98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:05.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.774+0000 7f1471665b80 1 -- 172.21.15.70:0/2315719373 shutdown_connections 2024-06-04T05:00:05.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.774+0000 7f1471665b80 1 -- 172.21.15.70:0/2315719373 wait complete. 2024-06-04T05:00:05.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.775+0000 7f1471665b80 1 -- start start 2024-06-04T05:00:05.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.775+0000 7f1471665b80 1 Processor -- start 2024-06-04T05:00:05.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.775+0000 7f1471665b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969af260 0x55c4969c0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.775+0000 7f1471665b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969b4320 0x55c4969bb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.775+0000 7f1471665b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969c44a0 0x55c4969bbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.775+0000 7f1471665b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f14601b1310 con 0x55c4969b4320 2024-06-04T05:00:05.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.775+0000 7f1471665b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f14601b0c40 con 0x55c4969c44a0 2024-06-04T05:00:05.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.775+0000 7f1471665b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f14601ade40 con 0x55c4969af260 2024-06-04T05:00:05.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.775+0000 7f1471665b80 10 client.0 _ll_register_callbacks cb 0x55c496897950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:05.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.775+0000 7f145ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969b4320 0x55c4969bb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.775+0000 7f145ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969b4320 0x55c4969bb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35778/0 (socket says 172.21.15.70:35778) 2024-06-04T05:00:05.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.775+0000 7f145ffff700 1 -- 172.21.15.70:0/3612345309 learned_addr learned my addr 172.21.15.70:0/3612345309 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:05.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.775+0000 7f1464b6c700 1 --2- 172.21.15.70:0/3612345309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969af260 0x55c4969c0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.776+0000 7f146536d700 1 --2- 172.21.15.70:0/3612345309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969c44a0 0x55c4969bbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.776+0000 7f145ffff700 1 -- 172.21.15.70:0/3612345309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969af260 msgr2=0x55c4969c0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:05.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.776+0000 7f145ffff700 1 --2- 172.21.15.70:0/3612345309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969af260 0x55c4969c0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.776+0000 7f145ffff700 1 -- 172.21.15.70:0/3612345309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969c44a0 msgr2=0x55c4969bbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:05.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.776+0000 7f145ffff700 1 --2- 172.21.15.70:0/3612345309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969c44a0 0x55c4969bbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.776+0000 7f145ffff700 1 -- 172.21.15.70:0/3612345309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4969bc250 con 0x55c4969b4320 2024-06-04T05:00:05.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.776+0000 7f146536d700 1 --2- 172.21.15.70:0/3612345309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969c44a0 0x55c4969bbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:05.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.776+0000 7f1464b6c700 1 --2- 172.21.15.70:0/3612345309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969af260 0x55c4969c0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:05.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.776+0000 7f145ffff700 1 --2- 172.21.15.70:0/3612345309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969b4320 0x55c4969bb650 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f144c000c00 tx=0x7f144c0e0f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=648df8aa374a8e87 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:05.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.776+0000 7f143ffff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:05.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.776+0000 7f143ffff700 1 -- 172.21.15.70:0/3612345309 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c0c9060 con 0x55c4969b4320 2024-06-04T05:00:05.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.776+0000 7f143ffff700 1 -- 172.21.15.70:0/3612345309 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f144c0aa890 con 0x55c4969b4320 2024-06-04T05:00:05.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.776+0000 7f1471665b80 1 -- 172.21.15.70:0/3612345309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f144c0ae9e0 con 0x55c4969b4320 2024-06-04T05:00:05.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.776+0000 7f143ffff700 1 -- 172.21.15.70:0/3612345309 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f144c089050 con 0x55c4969b4320 2024-06-04T05:00:05.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.776+0000 7f1471665b80 10 client.16242 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:05.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.776+0000 7f1471665b80 1 -- 172.21.15.70:0/3612345309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c496a33a80 con 0x55c4969b4320 2024-06-04T05:00:05.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.777+0000 7f143ffff700 1 -- 172.21.15.70:0/3612345309 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f144c0ae9e0 con 0x55c4969b4320 2024-06-04T05:00:05.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.777+0000 7f143ffff700 1 -- 172.21.15.70:0/3612345309 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f144c090b50 con 0x55c4969b4320 2024-06-04T05:00:05.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.777+0000 7f1471665b80 10 client.16242 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:05.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.777+0000 7f1471665b80 10 client.16242 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:05.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.777+0000 7f1471665b80 20 client.16242 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:05.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.777+0000 7f1471665b80 1 --2- 172.21.15.70:0/3612345309 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c496a33d50 0x55c496a36170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:05.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.777+0000 7f1471665b80 4 client.16242 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:00:05.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.777+0000 7f1471665b80 1 -- 172.21.15.70:0/3612345309 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c4969084a0 con 0x55c496a33d50 2024-06-04T05:00:05.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.777+0000 7f1464b6c700 1 --2- 172.21.15.70:0/3612345309 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c496a33d50 0x55c496a36170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:05.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.777+0000 7f143ffff700 10 client.16242 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:05.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.777+0000 7f1464b6c700 1 --2- 172.21.15.70:0/3612345309 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c496a33d50 0x55c496a36170 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d13728c9b3f0c4d server_cookie=5a86cc1325665b98 in_seq=0 out_seq=0 2024-06-04T05:00:05.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.778+0000 7f143ffff700 1 -- 172.21.15.70:0/3612345309 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c4969084a0 con 0x55c496a33d50 2024-06-04T05:00:05.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.778+0000 7f143ffff700 10 client.16242 handle_command_reply: tid=0 2024-06-04T05:00:05.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.778+0000 7f1471665b80 1 client.16242 shutdown 2024-06-04T05:00:05.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.778+0000 7f1471665b80 1 -- 172.21.15.70:0/3612345309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969b4320 msgr2=0x55c4969bb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:05.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.778+0000 7f1471665b80 1 --2- 172.21.15.70:0/3612345309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969b4320 0x55c4969bb650 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f144c000c00 tx=0x7f144c0e0f70 comp rx=0 tx=0).stop 2024-06-04T05:00:05.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.778+0000 7f1471665b80 1 -- 172.21.15.70:0/3612345309 shutdown_connections 2024-06-04T05:00:05.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.778+0000 7f1471665b80 1 --2- 172.21.15.70:0/3612345309 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c496a33d50 0x55c496a36170 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.778+0000 7f1471665b80 1 --2- 172.21.15.70:0/3612345309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4969c44a0 0x55c4969bbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.778+0000 7f1471665b80 1 --2- 172.21.15.70:0/3612345309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4969af260 0x55c4969c0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.778+0000 7f1471665b80 1 --2- 172.21.15.70:0/3612345309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4969b4320 0x55c4969bb650 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.778+0000 7f143ffff700 0 client.16242 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:05.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.778+0000 7f1471665b80 1 -- 172.21.15.70:0/3612345309 >> 172.21.15.70:0/3612345309 conn(0x55c4968da240 msgr2=0x55c4969c3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:05.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.778+0000 7f1471665b80 1 -- 172.21.15.70:0/3612345309 shutdown_connections 2024-06-04T05:00:05.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.779+0000 7f1471665b80 1 -- 172.21.15.70:0/3612345309 wait complete. 2024-06-04T05:00:05.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.779+0000 7f1471665b80 20 client.16242 trim_cache size 0 max 16384 2024-06-04T05:00:05.787 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:00:05.788 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:00:05.788 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:00:05.788 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:00:05.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.779+0000 7f1466dd0700 1 -- 172.21.15.70:0/1230160351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f14500b2a70 msgr2=0x7f14500b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:05.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.779+0000 7f1466dd0700 1 --2- 172.21.15.70:0/1230160351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f14500b2a70 0x7f14500b4f30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1454009fd0 tx=0x7f1454009260 comp rx=0 tx=0).stop 2024-06-04T05:00:05.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.779+0000 7f1466dd0700 1 -- 172.21.15.70:0/1230160351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1460102e00 msgr2=0x7f14601045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:05.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.779+0000 7f1466dd0700 1 --2- 172.21.15.70:0/1230160351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1460102e00 0x7f14601045c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f144c0098e0 tx=0x7f144c0047b0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.779+0000 7f1466dd0700 1 -- 172.21.15.70:0/1230160351 shutdown_connections 2024-06-04T05:00:05.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.779+0000 7f1466dd0700 1 --2- 172.21.15.70:0/1230160351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f14500b2a70 0x7f14500b4f30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.779+0000 7f1466dd0700 1 --2- 172.21.15.70:0/1230160351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1460108520 0x7f1460104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.779+0000 7f1466dd0700 1 --2- 172.21.15.70:0/1230160351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1460102440 0x7f1460104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.779+0000 7f1466dd0700 1 --2- 172.21.15.70:0/1230160351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1460102e00 0x7f14601045c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:05.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.779+0000 7f1466dd0700 1 -- 172.21.15.70:0/1230160351 >> 172.21.15.70:0/1230160351 conn(0x7f14600793f0 msgr2=0x7f146006c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:05.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.779+0000 7f1466dd0700 1 -- 172.21.15.70:0/1230160351 shutdown_connections 2024-06-04T05:00:05.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:05.779+0000 7f1466dd0700 1 -- 172.21.15.70:0/1230160351 wait complete. 2024-06-04T05:00:06.680 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:00:06.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:00:06.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 33458 3.36 MB/sec warmup 59 sec latency 15.541 ms 2024-06-04T05:00:06.946 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:00:07.319 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:07 smithi070 ceph-mon[32631]: pgmap v91: 97 pgs: 97 active+clean; 38 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 1.7 MiB/s wr, 216 op/s 2024-06-04T05:00:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:07 smithi112 ceph-mon[36453]: pgmap v91: 97 pgs: 97 active+clean; 38 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 1.7 MiB/s wr, 216 op/s 2024-06-04T05:00:07.713 INFO:tasks.workunit.client.0.smithi070.stdout: 1 34133 3.31 MB/sec warmup 60 sec latency 15.224 ms 2024-06-04T05:00:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:07 smithi098 ceph-mon[35569]: pgmap v91: 97 pgs: 97 active+clean; 38 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 1.7 MiB/s wr, 216 op/s 2024-06-04T05:00:08.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 34724 3.35 MB/sec warmup 61 sec latency 16.715 ms 2024-06-04T05:00:08.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.908+0000 7f8e4de9e700 1 -- 172.21.15.70:0/3284632506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e48102e00 msgr2=0x7f8e4810f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:08.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.908+0000 7f8e4de9e700 1 --2- 172.21.15.70:0/3284632506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e48102e00 0x7f8e4810f2f0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f8e4000b0a0 tx=0x7f8e4000b3b0 comp rx=0 tx=0).stop 2024-06-04T05:00:08.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.908+0000 7f8e4de9e700 1 -- 172.21.15.70:0/3284632506 shutdown_connections 2024-06-04T05:00:08.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.908+0000 7f8e4de9e700 1 --2- 172.21.15.70:0/3284632506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e48102e00 0x7f8e4810f2f0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:08.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.908+0000 7f8e4de9e700 1 --2- 172.21.15.70:0/3284632506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e48108520 0x7f8e48108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:08.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.908+0000 7f8e4de9e700 1 --2- 172.21.15.70:0/3284632506 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e48102440 0x7f8e481028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:08.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.908+0000 7f8e4de9e700 1 -- 172.21.15.70:0/3284632506 >> 172.21.15.70:0/3284632506 conn(0x7f8e480793f0 msgr2=0x7f8e48079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:08.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.908+0000 7f8e4de9e700 1 -- 172.21.15.70:0/3284632506 shutdown_connections 2024-06-04T05:00:08.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.908+0000 7f8e4de9e700 1 -- 172.21.15.70:0/3284632506 wait complete. 2024-06-04T05:00:08.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.909+0000 7f8e4de9e700 1 Processor -- start 2024-06-04T05:00:08.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.909+0000 7f8e4de9e700 1 -- start start 2024-06-04T05:00:08.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.909+0000 7f8e4de9e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e48102440 0x7f8e48104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:08.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.909+0000 7f8e4de9e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e48102e00 0x7f8e481045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:08.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.909+0000 7f8e4de9e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e48108520 0x7f8e48104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:08.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.909+0000 7f8e4de9e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e48112480 con 0x7f8e48108520 2024-06-04T05:00:08.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.909+0000 7f8e4de9e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8e48112300 con 0x7f8e48102440 2024-06-04T05:00:08.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.909+0000 7f8e4de9e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8e48112600 con 0x7f8e48102e00 2024-06-04T05:00:08.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.909+0000 7f8e46ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e48102e00 0x7f8e481045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:08.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.909+0000 7f8e47fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e48108520 0x7f8e48104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:08.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.910+0000 7f8e46ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e48102e00 0x7f8e481045c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59662/0 (socket says 172.21.15.70:59662) 2024-06-04T05:00:08.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.910+0000 7f8e46ffd700 1 -- 172.21.15.70:0/3020349177 learned_addr learned my addr 172.21.15.70:0/3020349177 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:08.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.910+0000 7f8e477fe700 1 --2- 172.21.15.70:0/3020349177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e48102440 0x7f8e48104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:08.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.910+0000 7f8e46ffd700 1 -- 172.21.15.70:0/3020349177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e48102440 msgr2=0x7f8e48104080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:08.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.910+0000 7f8e46ffd700 1 --2- 172.21.15.70:0/3020349177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e48102440 0x7f8e48104080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:08.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.910+0000 7f8e46ffd700 1 -- 172.21.15.70:0/3020349177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e48108520 msgr2=0x7f8e48104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:08.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.910+0000 7f8e46ffd700 1 --2- 172.21.15.70:0/3020349177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e48108520 0x7f8e48104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:08.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.910+0000 7f8e46ffd700 1 -- 172.21.15.70:0/3020349177 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e481a8000 con 0x7f8e48102e00 2024-06-04T05:00:08.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.910+0000 7f8e47fff700 1 --2- 172.21.15.70:0/3020349177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e48108520 0x7f8e48104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:08.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.910+0000 7f8e46ffd700 1 --2- 172.21.15.70:0/3020349177 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e48102e00 0x7f8e481045c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00d980 tx=0x7f8e3c00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:08.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.910+0000 7f8e44ff9700 1 -- 172.21.15.70:0/3020349177 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c016070 con 0x7f8e48102e00 2024-06-04T05:00:08.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.910+0000 7f8e44ff9700 1 -- 172.21.15.70:0/3020349177 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e3c004830 con 0x7f8e48102e00 2024-06-04T05:00:08.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.910+0000 7f8e4de9e700 1 -- 172.21.15.70:0/3020349177 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e481a8290 con 0x7f8e48102e00 2024-06-04T05:00:08.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.910+0000 7f8e44ff9700 1 -- 172.21.15.70:0/3020349177 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c0054c0 con 0x7f8e48102e00 2024-06-04T05:00:08.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.911+0000 7f8e4de9e700 1 -- 172.21.15.70:0/3020349177 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e481a8760 con 0x7f8e48102e00 2024-06-04T05:00:08.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.912+0000 7f8e4de9e700 1 -- 172.21.15.70:0/3020349177 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e4806a230 con 0x7f8e48102e00 2024-06-04T05:00:08.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.913+0000 7f8e44ff9700 1 -- 172.21.15.70:0/3020349177 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f8e3c047d10 con 0x7f8e48102e00 2024-06-04T05:00:08.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.914+0000 7f8e44ff9700 1 --2- 172.21.15.70:0/3020349177 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8e300b2a70 0x7f8e300b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:08.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.914+0000 7f8e477fe700 1 --2- 172.21.15.70:0/3020349177 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8e300b2a70 0x7f8e300b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:08.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.914+0000 7f8e44ff9700 1 -- 172.21.15.70:0/3020349177 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f8e3c0e25a0 con 0x7f8e48102e00 2024-06-04T05:00:08.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.914+0000 7f8e477fe700 1 --2- 172.21.15.70:0/3020349177 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8e300b2a70 0x7f8e300b4f30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8e38005950 tx=0x7f8e3800b410 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:08.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:08.918+0000 7f8e44ff9700 1 -- 172.21.15.70:0/3020349177 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8e3c01f020 con 0x7f8e48102e00 2024-06-04T05:00:09.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:09.202+0000 7f8e4de9e700 1 -- 172.21.15.70:0/3020349177 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e4802e720 con 0x7f8e48102e00 2024-06-04T05:00:09.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:09.203+0000 7f8e44ff9700 1 -- 172.21.15.70:0/3020349177 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11932 (secure 0 0 0) 0x7f8e3c0a9f20 con 0x7f8e48102e00 2024-06-04T05:00:09.206 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:00:09.206 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:59:22.801688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:00:09.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:09.206+0000 7f8e4de9e700 1 -- 172.21.15.70:0/3020349177 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8e300b2a70 msgr2=0x7f8e300b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:09.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:09.206+0000 7f8e4de9e700 1 --2- 172.21.15.70:0/3020349177 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8e300b2a70 0x7f8e300b4f30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8e38005950 tx=0x7f8e3800b410 comp rx=0 tx=0).stop 2024-06-04T05:00:09.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:09.206+0000 7f8e4de9e700 1 -- 172.21.15.70:0/3020349177 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e48102e00 msgr2=0x7f8e481045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:09.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:09.206+0000 7f8e4de9e700 1 --2- 172.21.15.70:0/3020349177 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e48102e00 0x7f8e481045c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00d980 tx=0x7f8e3c00dc90 comp rx=0 tx=0).stop 2024-06-04T05:00:09.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:09.206+0000 7f8e4de9e700 1 -- 172.21.15.70:0/3020349177 shutdown_connections 2024-06-04T05:00:09.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:09.206+0000 7f8e4de9e700 1 --2- 172.21.15.70:0/3020349177 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8e300b2a70 0x7f8e300b4f30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:09.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:09.206+0000 7f8e4de9e700 1 --2- 172.21.15.70:0/3020349177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e48108520 0x7f8e48104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:09.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:09.206+0000 7f8e4de9e700 1 --2- 172.21.15.70:0/3020349177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e48102440 0x7f8e48104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:09.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:09.206+0000 7f8e4de9e700 1 --2- 172.21.15.70:0/3020349177 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e48102e00 0x7f8e481045c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:09.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:09.206+0000 7f8e4de9e700 1 -- 172.21.15.70:0/3020349177 >> 172.21.15.70:0/3020349177 conn(0x7f8e480793f0 msgr2=0x7f8e4806b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:09.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:09.206+0000 7f8e4de9e700 1 -- 172.21.15.70:0/3020349177 shutdown_connections 2024-06-04T05:00:09.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:09.206+0000 7f8e4de9e700 1 -- 172.21.15.70:0/3020349177 wait complete. 2024-06-04T05:00:09.210 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-06-04T05:00:09.517 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:09 smithi070 ceph-mon[32631]: pgmap v92: 97 pgs: 97 active+clean; 38 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-06-04T05:00:09.517 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:09 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3020349177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:00:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:09 smithi112 ceph-mon[36453]: pgmap v92: 97 pgs: 97 active+clean; 38 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-06-04T05:00:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:09 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3020349177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:00:09.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 35291 3.35 MB/sec warmup 62 sec latency 16.036 ms 2024-06-04T05:00:09.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:09 smithi098 ceph-mon[35569]: pgmap v92: 97 pgs: 97 active+clean; 38 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-06-04T05:00:09.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:09 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3020349177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:00:10.053 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:00:10.367 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:00:10.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 35756 3.36 MB/sec warmup 63 sec latency 15.574 ms 2024-06-04T05:00:11.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:11 smithi070 ceph-mon[32631]: pgmap v93: 97 pgs: 97 active+clean; 38 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-06-04T05:00:11.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:11 smithi112 ceph-mon[36453]: pgmap v93: 97 pgs: 97 active+clean; 38 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-06-04T05:00:11.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 36033 3.32 MB/sec warmup 64 sec latency 12.414 ms 2024-06-04T05:00:11.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:11 smithi098 ceph-mon[35569]: pgmap v93: 97 pgs: 97 active+clean; 38 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-06-04T05:00:12.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.277+0000 7f264040c700 1 -- 172.21.15.70:0/1396427949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2638108510 msgr2=0x7f26381088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:12.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.277+0000 7f264040c700 1 --2- 172.21.15.70:0/1396427949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2638108510 0x7f26381088f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f2634009990 tx=0x7f2634009ca0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.277+0000 7f264040c700 1 -- 172.21.15.70:0/1396427949 shutdown_connections 2024-06-04T05:00:12.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.277+0000 7f264040c700 1 --2- 172.21.15.70:0/1396427949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2638102df0 0x7f263810f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.277+0000 7f264040c700 1 --2- 172.21.15.70:0/1396427949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2638108510 0x7f26381088f0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.277+0000 7f264040c700 1 --2- 172.21.15.70:0/1396427949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2638102430 0x7f26381028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.277+0000 7f264040c700 1 -- 172.21.15.70:0/1396427949 >> 172.21.15.70:0/1396427949 conn(0x7f26380793b0 msgr2=0x7f26380797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:12.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.277+0000 7f264040c700 1 -- 172.21.15.70:0/1396427949 shutdown_connections 2024-06-04T05:00:12.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.277+0000 7f264040c700 1 -- 172.21.15.70:0/1396427949 wait complete. 2024-06-04T05:00:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.278+0000 7f264040c700 1 Processor -- start 2024-06-04T05:00:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.278+0000 7f264040c700 1 -- start start 2024-06-04T05:00:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f264040c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2638102430 0x7f2638199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f264040c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2638102df0 0x7f263819a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f264040c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2638108510 0x7f263819e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f264040c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f26381135c0 con 0x7f2638102430 2024-06-04T05:00:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f264040c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2638113440 con 0x7f2638108510 2024-06-04T05:00:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f264040c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2638113740 con 0x7f2638102df0 2024-06-04T05:00:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f263d9a7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2638102df0 0x7f263819a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f263e1a8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2638102430 0x7f2638199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f263e1a8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2638102430 0x7f2638199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50110/0 (socket says 172.21.15.70:50110) 2024-06-04T05:00:12.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f263e1a8700 1 -- 172.21.15.70:0/3228581123 learned_addr learned my addr 172.21.15.70:0/3228581123 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:12.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f263d9a7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2638102df0 0x7f263819a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59672/0 (socket says 172.21.15.70:59672) 2024-06-04T05:00:12.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f263e9a9700 1 --2- 172.21.15.70:0/3228581123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2638108510 0x7f263819e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:12.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f263e1a8700 1 -- 172.21.15.70:0/3228581123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2638102df0 msgr2=0x7f263819a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:12.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f263e1a8700 1 --2- 172.21.15.70:0/3228581123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2638102df0 0x7f263819a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f263e1a8700 1 -- 172.21.15.70:0/3228581123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2638108510 msgr2=0x7f263819e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:12.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f263e1a8700 1 --2- 172.21.15.70:0/3228581123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2638108510 0x7f263819e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f263e1a8700 1 -- 172.21.15.70:0/3228581123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263819eb10 con 0x7f2638102430 2024-06-04T05:00:12.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f263d9a7700 1 --2- 172.21.15.70:0/3228581123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2638102df0 0x7f263819a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:12.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f263e9a9700 1 --2- 172.21.15.70:0/3228581123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2638108510 0x7f263819e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:12.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f263e1a8700 1 --2- 172.21.15.70:0/3228581123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2638102430 0x7f2638199b10 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f2634006010 tx=0x7f26340057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:12.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.279+0000 7f262f7fe700 1 -- 172.21.15.70:0/3228581123 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2634027070 con 0x7f2638102430 2024-06-04T05:00:12.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.280+0000 7f262f7fe700 1 -- 172.21.15.70:0/3228581123 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2634022470 con 0x7f2638102430 2024-06-04T05:00:12.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.280+0000 7f264040c700 1 -- 172.21.15.70:0/3228581123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f263819eda0 con 0x7f2638102430 2024-06-04T05:00:12.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.280+0000 7f262f7fe700 1 -- 172.21.15.70:0/3228581123 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2634017650 con 0x7f2638102430 2024-06-04T05:00:12.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.280+0000 7f264040c700 1 -- 172.21.15.70:0/3228581123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f263819f190 con 0x7f2638102430 2024-06-04T05:00:12.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.283+0000 7f262f7fe700 1 -- 172.21.15.70:0/3228581123 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f263402a070 con 0x7f2638102430 2024-06-04T05:00:12.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.284+0000 7f264aca1b80 1 Processor -- start 2024-06-04T05:00:12.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.285+0000 7f264aca1b80 1 -- start start 2024-06-04T05:00:12.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.285+0000 7f262f7fe700 1 --2- 172.21.15.70:0/3228581123 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f26240b29f0 0x7f26240b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.285+0000 7f264aca1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 0x555786b50700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.285+0000 7f264aca1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4a8a0 0x555786b4ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.285+0000 7f264aca1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4b260 0x555786b57080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.285+0000 7f264aca1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555786aa8d70 con 0x555786b4a8a0 2024-06-04T05:00:12.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.285+0000 7f264aca1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555786a98910 con 0x555786b4b260 2024-06-04T05:00:12.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.285+0000 7f264aca1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555786a9c950 con 0x555786b50320 2024-06-04T05:00:12.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.285+0000 7f263e9a9700 1 --2- 172.21.15.70:0/3228581123 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f26240b29f0 0x7f26240b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:12.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.285+0000 7f263e1a8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 0x555786b50700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:12.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.285+0000 7f262f7fe700 1 -- 172.21.15.70:0/3228581123 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f26340e3450 con 0x7f2638102430 2024-06-04T05:00:12.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.285+0000 7f263e1a8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 0x555786b50700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59686/0 (socket says 172.21.15.70:59686) 2024-06-04T05:00:12.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.285+0000 7f263e1a8700 1 -- 172.21.15.70:0/124034726 learned_addr learned my addr 172.21.15.70:0/124034726 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:12.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.285+0000 7f263e9a9700 1 --2- 172.21.15.70:0/124034726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4b260 0x555786b57080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:12.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.286+0000 7f263e1a8700 1 -- 172.21.15.70:0/124034726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4b260 msgr2=0x555786b57080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:12.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.286+0000 7f263e1a8700 1 --2- 172.21.15.70:0/124034726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4b260 0x555786b57080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.286+0000 7f263e1a8700 1 -- 172.21.15.70:0/124034726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4a8a0 msgr2=0x555786b4ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:12.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.286+0000 7f263e1a8700 1 --2- 172.21.15.70:0/124034726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4a8a0 0x555786b4ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.286+0000 7f263e1a8700 1 -- 172.21.15.70:0/124034726 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555786a41fb0 con 0x555786b50320 2024-06-04T05:00:12.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.286+0000 7f263e9a9700 1 --2- 172.21.15.70:0/3228581123 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f26240b29f0 0x7f26240b4eb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2630004230 tx=0x7f26300045f0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:12.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.286+0000 7f263e1a8700 1 --2- 172.21.15.70:0/124034726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 0x555786b50700 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f26340e0bf0 tx=0x7f26340a6100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8b0b0898d6cd301 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:12.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.286+0000 7f262cd70700 1 -- 172.21.15.70:0/124034726 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26340aa1c0 con 0x555786b50320 2024-06-04T05:00:12.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.286+0000 7f262cd70700 1 -- 172.21.15.70:0/124034726 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2634022920 con 0x555786b50320 2024-06-04T05:00:12.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.286+0000 7f262cd70700 1 -- 172.21.15.70:0/124034726 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26340a9fe0 con 0x555786b50320 2024-06-04T05:00:12.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.287+0000 7f264aca1b80 1 -- 172.21.15.70:0/124034726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 msgr2=0x555786b50700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:12.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.287+0000 7f264aca1b80 1 --2- 172.21.15.70:0/124034726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 0x555786b50700 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f26340e0bf0 tx=0x7f26340a6100 comp rx=0 tx=0).stop 2024-06-04T05:00:12.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.287+0000 7f264aca1b80 1 -- 172.21.15.70:0/124034726 shutdown_connections 2024-06-04T05:00:12.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.287+0000 7f264aca1b80 1 --2- 172.21.15.70:0/124034726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4b260 0x555786b57080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.287+0000 7f264aca1b80 1 --2- 172.21.15.70:0/124034726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 0x555786b50700 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.287+0000 7f264aca1b80 1 --2- 172.21.15.70:0/124034726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4a8a0 0x555786b4ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.287+0000 7f264aca1b80 1 -- 172.21.15.70:0/124034726 >> 172.21.15.70:0/124034726 conn(0x555786a76240 msgr2=0x555786b453e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:12.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.287+0000 7f264aca1b80 1 -- 172.21.15.70:0/124034726 shutdown_connections 2024-06-04T05:00:12.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.287+0000 7f264aca1b80 1 -- 172.21.15.70:0/124034726 wait complete. 2024-06-04T05:00:12.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f264aca1b80 1 -- start start 2024-06-04T05:00:12.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f264aca1b80 1 Processor -- start 2024-06-04T05:00:12.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f264aca1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4a8a0 0x555786b56cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f264aca1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b4b260 0x555786b53fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f264aca1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b50320 0x555786b544f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f264aca1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55578699efa0 con 0x555786b50320 2024-06-04T05:00:12.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f264aca1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555786a00da0 con 0x555786b4a8a0 2024-06-04T05:00:12.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f264aca1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555786a9c950 con 0x555786b4b260 2024-06-04T05:00:12.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f264aca1b80 10 client.0 _ll_register_callbacks cb 0x555786a33950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:12.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f263e1a8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4a8a0 0x555786b56cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:12.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f263e1a8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4a8a0 0x555786b56cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:60552/0 (socket says 172.21.15.70:60552) 2024-06-04T05:00:12.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f263e1a8700 1 -- 172.21.15.70:0/2844505417 learned_addr learned my addr 172.21.15.70:0/2844505417 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:12.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f263d9a7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b4b260 0x555786b53fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:12.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f263e9a9700 1 --2- 172.21.15.70:0/2844505417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b50320 0x555786b544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:12.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f263e1a8700 1 -- 172.21.15.70:0/2844505417 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b4b260 msgr2=0x555786b53fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:12.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f263e1a8700 1 --2- 172.21.15.70:0/2844505417 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b4b260 0x555786b53fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f263e1a8700 1 -- 172.21.15.70:0/2844505417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b50320 msgr2=0x555786b544f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:12.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f263e1a8700 1 --2- 172.21.15.70:0/2844505417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b50320 0x555786b544f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.288+0000 7f263e1a8700 1 -- 172.21.15.70:0/2844505417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557869e88d0 con 0x555786b4a8a0 2024-06-04T05:00:12.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.289+0000 7f263e1a8700 1 --2- 172.21.15.70:0/2844505417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4a8a0 0x555786b56cb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f263400b2e0 tx=0x7f26340aa450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e92c8994314a4218 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:12.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.289+0000 7f2621ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:00:12.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.289+0000 7f2621ffb700 1 -- 172.21.15.70:0/2844505417 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26340aaf70 con 0x555786b4a8a0 2024-06-04T05:00:12.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.289+0000 7f264aca1b80 1 -- 172.21.15.70:0/2844505417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2634022920 con 0x555786b4a8a0 2024-06-04T05:00:12.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.289+0000 7f2621ffb700 1 -- 172.21.15.70:0/2844505417 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f26340b31b0 con 0x555786b4a8a0 2024-06-04T05:00:12.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.289+0000 7f2621ffb700 1 -- 172.21.15.70:0/2844505417 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f26340aec20 con 0x555786b4a8a0 2024-06-04T05:00:12.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.289+0000 7f264aca1b80 10 client.25661 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:12.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.289+0000 7f264aca1b80 1 -- 172.21.15.70:0/2844505417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555786a52760 con 0x555786b4a8a0 2024-06-04T05:00:12.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.289+0000 7f2621ffb700 1 -- 172.21.15.70:0/2844505417 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2634022920 con 0x555786b4a8a0 2024-06-04T05:00:12.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.289+0000 7f2621ffb700 1 -- 172.21.15.70:0/2844505417 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f26340a9bd0 con 0x555786b4a8a0 2024-06-04T05:00:12.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.290+0000 7f264aca1b80 10 client.25661 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:12.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.290+0000 7f264aca1b80 10 client.25661 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:00:12.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.290+0000 7f264aca1b80 20 client.25661 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:12.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.290+0000 7f264aca1b80 1 --2- 172.21.15.70:0/2844505417 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555786b63da0 0x555786b66190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.290+0000 7f264aca1b80 4 client.25661 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:00:12.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.290+0000 7f264aca1b80 1 -- 172.21.15.70:0/2844505417 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555786aaacc0 con 0x555786b63da0 2024-06-04T05:00:12.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.290+0000 7f263d9a7700 1 --2- 172.21.15.70:0/2844505417 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555786b63da0 0x555786b66190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:12.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.290+0000 7f2621ffb700 10 client.25661 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:00:12.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.290+0000 7f263d9a7700 1 --2- 172.21.15.70:0/2844505417 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555786b63da0 0x555786b66190 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=44180455792a4da9 server_cookie=7f0a49ddfcc7beba in_seq=0 out_seq=0 2024-06-04T05:00:12.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.291+0000 7f2621ffb700 1 -- 172.21.15.70:0/2844505417 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555786aaacc0 con 0x555786b63da0 2024-06-04T05:00:12.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.291+0000 7f2621ffb700 10 client.25661 handle_command_reply: tid=0 2024-06-04T05:00:12.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.291+0000 7f264aca1b80 1 client.25661 shutdown 2024-06-04T05:00:12.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.291+0000 7f264aca1b80 1 -- 172.21.15.70:0/2844505417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4a8a0 msgr2=0x555786b56cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:12.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.291+0000 7f264aca1b80 1 --2- 172.21.15.70:0/2844505417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4a8a0 0x555786b56cb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f263400b2e0 tx=0x7f26340aa450 comp rx=0 tx=0).stop 2024-06-04T05:00:12.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.291+0000 7f264aca1b80 1 -- 172.21.15.70:0/2844505417 shutdown_connections 2024-06-04T05:00:12.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.291+0000 7f264aca1b80 1 --2- 172.21.15.70:0/2844505417 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555786b63da0 0x555786b66190 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.291+0000 7f264aca1b80 1 --2- 172.21.15.70:0/2844505417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b50320 0x555786b544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.291+0000 7f264aca1b80 1 --2- 172.21.15.70:0/2844505417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4a8a0 0x555786b56cb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.291+0000 7f264aca1b80 1 --2- 172.21.15.70:0/2844505417 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b4b260 0x555786b53fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.291+0000 7f264aca1b80 1 -- 172.21.15.70:0/2844505417 >> 172.21.15.70:0/2844505417 conn(0x555786a76240 msgr2=0x555786b44af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:12.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.291+0000 7f2621ffb700 0 client.25661 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:00:12.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.292+0000 7f264aca1b80 1 -- 172.21.15.70:0/2844505417 shutdown_connections 2024-06-04T05:00:12.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.292+0000 7f264aca1b80 1 -- 172.21.15.70:0/2844505417 wait complete. 2024-06-04T05:00:12.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.292+0000 7f264aca1b80 20 client.25661 trim_cache size 0 max 16384 2024-06-04T05:00:12.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f264aca1b80 1 Processor -- start 2024-06-04T05:00:12.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f264aca1b80 1 -- start start 2024-06-04T05:00:12.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f264aca1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4a8a0 0x555786b45270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f264aca1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4b260 0x555786b49fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f264aca1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 0x555786b51010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f264aca1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f26381afe10 con 0x555786b4b260 2024-06-04T05:00:12.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f264aca1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f26381b19b0 con 0x555786b4a8a0 2024-06-04T05:00:12.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f264aca1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f26381b04e0 con 0x555786b50320 2024-06-04T05:00:12.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f263d9a7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4b260 0x555786b49fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:12.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f263d9a7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4b260 0x555786b49fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50132/0 (socket says 172.21.15.70:50132) 2024-06-04T05:00:12.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f263d9a7700 1 -- 172.21.15.70:0/1259696098 learned_addr learned my addr 172.21.15.70:0/1259696098 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:12.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f263e1a8700 1 --2- 172.21.15.70:0/1259696098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4a8a0 0x555786b45270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:12.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f263e9a9700 1 --2- 172.21.15.70:0/1259696098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 0x555786b51010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:12.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f263d9a7700 1 -- 172.21.15.70:0/1259696098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 msgr2=0x555786b51010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:12.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f263d9a7700 1 --2- 172.21.15.70:0/1259696098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 0x555786b51010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f263d9a7700 1 -- 172.21.15.70:0/1259696098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4a8a0 msgr2=0x555786b45270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:12.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f263d9a7700 1 --2- 172.21.15.70:0/1259696098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4a8a0 0x555786b45270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f263d9a7700 1 -- 172.21.15.70:0/1259696098 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555786b50700 con 0x555786b4b260 2024-06-04T05:00:12.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.308+0000 7f263d9a7700 1 --2- 172.21.15.70:0/1259696098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4b260 0x555786b49fa0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f2638079910 tx=0x7f2628013880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7add7de4def418 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:12.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.309+0000 7f2623fff700 1 -- 172.21.15.70:0/1259696098 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628012370 con 0x555786b4b260 2024-06-04T05:00:12.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.309+0000 7f2623fff700 1 -- 172.21.15.70:0/1259696098 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f262801f850 con 0x555786b4b260 2024-06-04T05:00:12.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.309+0000 7f2623fff700 1 -- 172.21.15.70:0/1259696098 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262801ea70 con 0x555786b4b260 2024-06-04T05:00:12.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.309+0000 7f264aca1b80 1 -- 172.21.15.70:0/1259696098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4b260 msgr2=0x555786b49fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:12.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.309+0000 7f264aca1b80 1 --2- 172.21.15.70:0/1259696098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4b260 0x555786b49fa0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f2638079910 tx=0x7f2628013880 comp rx=0 tx=0).stop 2024-06-04T05:00:12.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.309+0000 7f264aca1b80 1 -- 172.21.15.70:0/1259696098 shutdown_connections 2024-06-04T05:00:12.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.310+0000 7f264aca1b80 1 --2- 172.21.15.70:0/1259696098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 0x555786b51010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.310+0000 7f264aca1b80 1 --2- 172.21.15.70:0/1259696098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4a8a0 0x555786b45270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.310+0000 7f264aca1b80 1 --2- 172.21.15.70:0/1259696098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4b260 0x555786b49fa0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.310+0000 7f264aca1b80 1 -- 172.21.15.70:0/1259696098 >> 172.21.15.70:0/1259696098 conn(0x555786a76240 msgr2=0x555786b49890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:12.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.310+0000 7f264aca1b80 1 -- 172.21.15.70:0/1259696098 shutdown_connections 2024-06-04T05:00:12.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.310+0000 7f264aca1b80 1 -- 172.21.15.70:0/1259696098 wait complete. 2024-06-04T05:00:12.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.310+0000 7f264aca1b80 1 -- start start 2024-06-04T05:00:12.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.310+0000 7f264aca1b80 1 Processor -- start 2024-06-04T05:00:12.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.310+0000 7f264aca1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4a8a0 0x555786b57bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.310+0000 7f264aca1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4b260 0x555786b58130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.310+0000 7f264aca1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 0x555786bcf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.310+0000 7f264aca1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f26381b1280 con 0x555786b4a8a0 2024-06-04T05:00:12.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.310+0000 7f264aca1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f26381b0bb0 con 0x555786b4b260 2024-06-04T05:00:12.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.310+0000 7f264aca1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f26381addb0 con 0x555786b50320 2024-06-04T05:00:12.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.310+0000 7f264aca1b80 10 client.0 _ll_register_callbacks cb 0x555786a33950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:12.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.310+0000 7f263e1a8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4a8a0 0x555786b57bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:12.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.311+0000 7f263e1a8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4a8a0 0x555786b57bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50146/0 (socket says 172.21.15.70:50146) 2024-06-04T05:00:12.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.311+0000 7f263e9a9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 0x555786bcf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:12.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.311+0000 7f263e1a8700 1 -- 172.21.15.70:0/374192513 learned_addr learned my addr 172.21.15.70:0/374192513 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:12.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.311+0000 7f263d9a7700 1 --2- 172.21.15.70:0/374192513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4b260 0x555786b58130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:12.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.311+0000 7f263e9a9700 1 -- 172.21.15.70:0/374192513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4b260 msgr2=0x555786b58130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:12.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.311+0000 7f263e9a9700 1 --2- 172.21.15.70:0/374192513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4b260 0x555786b58130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.311+0000 7f263e9a9700 1 -- 172.21.15.70:0/374192513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4a8a0 msgr2=0x555786b57bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:12.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.311+0000 7f263e9a9700 1 --2- 172.21.15.70:0/374192513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4a8a0 0x555786b57bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.311+0000 7f263e9a9700 1 -- 172.21.15.70:0/374192513 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555786bcf6a0 con 0x555786b50320 2024-06-04T05:00:12.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.311+0000 7f263e1a8700 1 --2- 172.21.15.70:0/374192513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4a8a0 0x555786b57bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:12.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.311+0000 7f263d9a7700 1 --2- 172.21.15.70:0/374192513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4b260 0x555786b58130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:12.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.311+0000 7f263e9a9700 1 --2- 172.21.15.70:0/374192513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 0x555786bcf0b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2630009fd0 tx=0x7f2630004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3bf99a3a1046d0a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:12.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.311+0000 7f2621ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:00:12.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.311+0000 7f2621ffb700 1 -- 172.21.15.70:0/374192513 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2630029070 con 0x555786b50320 2024-06-04T05:00:12.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.311+0000 7f2621ffb700 1 -- 172.21.15.70:0/374192513 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f2630009220 con 0x555786b50320 2024-06-04T05:00:12.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.311+0000 7f264aca1b80 1 -- 172.21.15.70:0/374192513 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f262801f850 con 0x555786b50320 2024-06-04T05:00:12.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.312+0000 7f2621ffb700 1 -- 172.21.15.70:0/374192513 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f263001aab0 con 0x555786b50320 2024-06-04T05:00:12.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.312+0000 7f264aca1b80 10 client.25726 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:12.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.312+0000 7f264aca1b80 1 -- 172.21.15.70:0/374192513 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555786bd6a40 con 0x555786b50320 2024-06-04T05:00:12.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.312+0000 7f2621ffb700 1 -- 172.21.15.70:0/374192513 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f262801f850 con 0x555786b50320 2024-06-04T05:00:12.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.312+0000 7f2621ffb700 1 -- 172.21.15.70:0/374192513 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f2630008ba0 con 0x555786b50320 2024-06-04T05:00:12.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.312+0000 7f264aca1b80 10 client.25726 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:12.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.312+0000 7f264aca1b80 10 client.25726 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:00:12.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.312+0000 7f264aca1b80 20 client.25726 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:12.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.312+0000 7f264aca1b80 1 --2- 172.21.15.70:0/374192513 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555786bd71b0 0x555786bd95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:12.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.312+0000 7f264aca1b80 4 client.25726 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:00:12.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.312+0000 7f264aca1b80 1 -- 172.21.15.70:0/374192513 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555786aa44a0 con 0x555786bd71b0 2024-06-04T05:00:12.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.312+0000 7f263e1a8700 1 --2- 172.21.15.70:0/374192513 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555786bd71b0 0x555786bd95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:12.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.313+0000 7f2621ffb700 10 client.25726 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:00:12.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.313+0000 7f263e1a8700 1 --2- 172.21.15.70:0/374192513 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555786bd71b0 0x555786bd95d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c0049ec05e9578f3 server_cookie=38d4f8f6e8f719fe in_seq=0 out_seq=0 2024-06-04T05:00:12.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.313+0000 7f2621ffb700 1 -- 172.21.15.70:0/374192513 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555786aa44a0 con 0x555786bd71b0 2024-06-04T05:00:12.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.313+0000 7f2621ffb700 10 client.25726 handle_command_reply: tid=0 2024-06-04T05:00:12.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.313+0000 7f264aca1b80 1 client.25726 shutdown 2024-06-04T05:00:12.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.313+0000 7f264aca1b80 1 -- 172.21.15.70:0/374192513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 msgr2=0x555786bcf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:12.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.313+0000 7f264aca1b80 1 --2- 172.21.15.70:0/374192513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 0x555786bcf0b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2630009fd0 tx=0x7f2630004b40 comp rx=0 tx=0).stop 2024-06-04T05:00:12.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.313+0000 7f264aca1b80 1 -- 172.21.15.70:0/374192513 shutdown_connections 2024-06-04T05:00:12.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.313+0000 7f264aca1b80 1 --2- 172.21.15.70:0/374192513 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555786bd71b0 0x555786bd95d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.313+0000 7f2621ffb700 0 client.25726 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:00:12.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.313+0000 7f264aca1b80 1 --2- 172.21.15.70:0/374192513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555786b50320 0x555786bcf0b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.313+0000 7f264aca1b80 1 --2- 172.21.15.70:0/374192513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555786b4a8a0 0x555786b57bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.313+0000 7f264aca1b80 1 --2- 172.21.15.70:0/374192513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555786b4b260 0x555786b58130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.313+0000 7f264aca1b80 1 -- 172.21.15.70:0/374192513 >> 172.21.15.70:0/374192513 conn(0x555786a76240 msgr2=0x555786b51010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:12.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.313+0000 7f264aca1b80 1 -- 172.21.15.70:0/374192513 shutdown_connections 2024-06-04T05:00:12.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.313+0000 7f264aca1b80 1 -- 172.21.15.70:0/374192513 wait complete. 2024-06-04T05:00:12.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.313+0000 7f264aca1b80 20 client.25726 trim_cache size 0 max 16384 2024-06-04T05:00:12.321 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:00:12.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.314+0000 7f264040c700 1 -- 172.21.15.70:0/3228581123 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f26240b29f0 msgr2=0x7f26240b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:12.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.314+0000 7f264040c700 1 --2- 172.21.15.70:0/3228581123 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f26240b29f0 0x7f26240b4eb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2630004230 tx=0x7f26300045f0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.314+0000 7f264040c700 1 -- 172.21.15.70:0/3228581123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2638102430 msgr2=0x7f2638199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:12.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.314+0000 7f264040c700 1 --2- 172.21.15.70:0/3228581123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2638102430 0x7f2638199b10 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f2634006010 tx=0x7f26340057c0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.314+0000 7f264040c700 1 -- 172.21.15.70:0/3228581123 shutdown_connections 2024-06-04T05:00:12.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.314+0000 7f264040c700 1 --2- 172.21.15.70:0/3228581123 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f26240b29f0 0x7f26240b4eb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.314+0000 7f264040c700 1 --2- 172.21.15.70:0/3228581123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2638108510 0x7f263819e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.314+0000 7f264040c700 1 --2- 172.21.15.70:0/3228581123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2638102430 0x7f2638199b10 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.314+0000 7f264040c700 1 --2- 172.21.15.70:0/3228581123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2638102df0 0x7f263819a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:12.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.314+0000 7f264040c700 1 -- 172.21.15.70:0/3228581123 >> 172.21.15.70:0/3228581123 conn(0x7f26380793b0 msgr2=0x7f2638101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:12.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.314+0000 7f264040c700 1 -- 172.21.15.70:0/3228581123 shutdown_connections 2024-06-04T05:00:12.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:12.314+0000 7f264040c700 1 -- 172.21.15.70:0/3228581123 wait complete. 2024-06-04T05:00:12.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 36482 3.28 MB/sec warmup 65 sec latency 43.140 ms 2024-06-04T05:00:13.114 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:00:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:13 smithi070 ceph-mon[32631]: pgmap v94: 97 pgs: 97 active+clean; 39 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.8 MiB/s wr, 214 op/s 2024-06-04T05:00:13.377 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:00:13.547 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:13 smithi098 ceph-mon[35569]: pgmap v94: 97 pgs: 97 active+clean; 39 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.8 MiB/s wr, 214 op/s 2024-06-04T05:00:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:13 smithi112 ceph-mon[36453]: pgmap v94: 97 pgs: 97 active+clean; 39 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.8 MiB/s wr, 214 op/s 2024-06-04T05:00:13.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 37070 3.29 MB/sec warmup 66 sec latency 15.287 ms 2024-06-04T05:00:13.812 INFO:tasks.quiescer.fs.[cephfs]:Total duration (58.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-06-04T05:00:13.813 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.05 2024-06-04T05:00:13.813 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:00:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.385+0000 7f42d6bdd640 1 -- 172.21.15.70:0/1162579516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42d0107940 msgr2=0x7f42d0107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.385+0000 7f42d6bdd640 1 --2- 172.21.15.70:0/1162579516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42d0107940 0x7f42d0107d20 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f42c4009930 tx=0x7f42c402a420 comp rx=0 tx=0).stop 2024-06-04T05:00:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.385+0000 7f42d6bdd640 1 -- 172.21.15.70:0/1162579516 shutdown_connections 2024-06-04T05:00:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.385+0000 7f42d6bdd640 1 --2- 172.21.15.70:0/1162579516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d0101980 0x7f42d0101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.385+0000 7f42d6bdd640 1 --2- 172.21.15.70:0/1162579516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42d0102340 0x7f42d010e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.385+0000 7f42d6bdd640 1 --2- 172.21.15.70:0/1162579516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42d0107940 0x7f42d0107d20 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.385+0000 7f42d6bdd640 1 -- 172.21.15.70:0/1162579516 >> 172.21.15.70:0/1162579516 conn(0x7f42d0078ad0 msgr2=0x7f42d0078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.385+0000 7f42d6bdd640 1 -- 172.21.15.70:0/1162579516 shutdown_connections 2024-06-04T05:00:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.385+0000 7f42d6bdd640 1 -- 172.21.15.70:0/1162579516 wait complete. 2024-06-04T05:00:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.385+0000 7f42d6bdd640 1 Processor -- start 2024-06-04T05:00:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42d6bdd640 1 -- start start 2024-06-04T05:00:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42d6bdd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d0101980 0x7f42d0197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42d6bdd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42d0102340 0x7f42d0198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42d6bdd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42d0107940 0x7f42d01986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42d6bdd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f42d0111ab0 con 0x7f42d0107940 2024-06-04T05:00:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42d6bdd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f42d0111930 con 0x7f42d0101980 2024-06-04T05:00:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42d6bdd640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f42d0111c30 con 0x7f42d0102340 2024-06-04T05:00:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42d4979640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d0101980 0x7f42d0197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42d517a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42d0107940 0x7f42d01986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42d4979640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d0101980 0x7f42d0197c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:60592/0 (socket says 172.21.15.70:60592) 2024-06-04T05:00:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42d4979640 1 -- 172.21.15.70:0/4217825155 learned_addr learned my addr 172.21.15.70:0/4217825155 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42cffff640 1 --2- 172.21.15.70:0/4217825155 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42d0102340 0x7f42d0198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42d4979640 1 -- 172.21.15.70:0/4217825155 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42d0102340 msgr2=0x7f42d0198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42d4979640 1 --2- 172.21.15.70:0/4217825155 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42d0102340 0x7f42d0198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42d4979640 1 -- 172.21.15.70:0/4217825155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42d0107940 msgr2=0x7f42d01986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42d4979640 1 --2- 172.21.15.70:0/4217825155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42d0107940 0x7f42d01986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42d4979640 1 -- 172.21.15.70:0/4217825155 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d01a74b0 con 0x7f42d0101980 2024-06-04T05:00:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.386+0000 7f42d517a640 1 --2- 172.21.15.70:0/4217825155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42d0107940 0x7f42d01986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.387+0000 7f42d4979640 1 --2- 172.21.15.70:0/4217825155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d0101980 0x7f42d0197c40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f42c4005940 tx=0x7f42c40042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.387+0000 7f42cdffb640 1 -- 172.21.15.70:0/4217825155 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c403f030 con 0x7f42d0101980 2024-06-04T05:00:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.387+0000 7f42d6bdd640 1 -- 172.21.15.70:0/4217825155 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42d01a7740 con 0x7f42d0101980 2024-06-04T05:00:14.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.387+0000 7f42cdffb640 1 -- 172.21.15.70:0/4217825155 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f42c4003c90 con 0x7f42d0101980 2024-06-04T05:00:14.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.387+0000 7f42cdffb640 1 -- 172.21.15.70:0/4217825155 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c4035070 con 0x7f42d0101980 2024-06-04T05:00:14.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.387+0000 7f42d6bdd640 1 -- 172.21.15.70:0/4217825155 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42d01a7c80 con 0x7f42d0101980 2024-06-04T05:00:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.389+0000 7f42cdffb640 1 -- 172.21.15.70:0/4217825155 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f42c4042030 con 0x7f42d0101980 2024-06-04T05:00:14.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d99d2740 1 Processor -- start 2024-06-04T05:00:14.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d99d2740 1 -- start start 2024-06-04T05:00:14.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d99d2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad517820 0x55acad517c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d99d2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad518140 0x55acad5224c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d99d2740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad522a00 0x55acad524df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d99d2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55acad489c30 con 0x55acad518140 2024-06-04T05:00:14.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d99d2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55acad4797d0 con 0x55acad517820 2024-06-04T05:00:14.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d99d2740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55acad47d810 con 0x55acad522a00 2024-06-04T05:00:14.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d4979640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad518140 0x55acad5224c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d4979640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad518140 0x55acad5224c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50166/0 (socket says 172.21.15.70:50166) 2024-06-04T05:00:14.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d4979640 1 -- 172.21.15.70:0/1401475855 learned_addr learned my addr 172.21.15.70:0/1401475855 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d517a640 1 --2- 172.21.15.70:0/1401475855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad517820 0x55acad517c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42cffff640 1 --2- 172.21.15.70:0/1401475855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad522a00 0x55acad524df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42cdffb640 1 --2- 172.21.15.70:0/4217825155 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f42c00b2a10 0x7f42c00b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d4979640 1 -- 172.21.15.70:0/1401475855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad522a00 msgr2=0x55acad524df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d4979640 1 --2- 172.21.15.70:0/1401475855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad522a00 0x55acad524df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d4979640 1 -- 172.21.15.70:0/1401475855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad517820 msgr2=0x55acad517c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d4979640 1 --2- 172.21.15.70:0/1401475855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad517820 0x55acad517c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d4979640 1 -- 172.21.15.70:0/1401475855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acad18e210 con 0x55acad518140 2024-06-04T05:00:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.390+0000 7f42d517a640 1 --2- 172.21.15.70:0/4217825155 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f42c00b2a10 0x7f42c00b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.391+0000 7f42cdffb640 1 -- 172.21.15.70:0/4217825155 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f42c4102a90 con 0x7f42d0101980 2024-06-04T05:00:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.391+0000 7f42d4979640 1 --2- 172.21.15.70:0/1401475855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad518140 0x55acad5224c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f42c4100150 tx=0x7f42c40f9380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d20379bc7c701c0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.391+0000 7f42baffd640 1 -- 172.21.15.70:0/1401475855 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c4049070 con 0x55acad518140 2024-06-04T05:00:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.391+0000 7f42baffd640 1 -- 172.21.15.70:0/1401475855 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f42c410a040 con 0x55acad518140 2024-06-04T05:00:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.391+0000 7f42baffd640 1 -- 172.21.15.70:0/1401475855 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c40c9b40 con 0x55acad518140 2024-06-04T05:00:14.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d99d2740 1 -- 172.21.15.70:0/1401475855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad518140 msgr2=0x55acad5224c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:14.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d517a640 1 --2- 172.21.15.70:0/4217825155 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f42c00b2a10 0x7f42c00b4ed0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f42d006b1e0 tx=0x7f42c8004a40 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:14.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d99d2740 1 --2- 172.21.15.70:0/1401475855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad518140 0x55acad5224c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f42c4100150 tx=0x7f42c40f9380 comp rx=0 tx=0).stop 2024-06-04T05:00:14.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d99d2740 1 -- 172.21.15.70:0/1401475855 shutdown_connections 2024-06-04T05:00:14.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d99d2740 1 --2- 172.21.15.70:0/1401475855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad522a00 0x55acad524df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d99d2740 1 --2- 172.21.15.70:0/1401475855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad518140 0x55acad5224c0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d99d2740 1 --2- 172.21.15.70:0/1401475855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad517820 0x55acad517c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d99d2740 1 -- 172.21.15.70:0/1401475855 >> 172.21.15.70:0/1401475855 conn(0x55acad513d30 msgr2=0x55acad514110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:14.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d99d2740 1 -- 172.21.15.70:0/1401475855 shutdown_connections 2024-06-04T05:00:14.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d99d2740 1 -- 172.21.15.70:0/1401475855 wait complete. 2024-06-04T05:00:14.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d99d2740 1 -- start start 2024-06-04T05:00:14.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d99d2740 1 Processor -- start 2024-06-04T05:00:14.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d99d2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad517820 0x55acad51fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d99d2740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad518140 0x55acad51ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d99d2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad522a00 0x55acad520490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d99d2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55acad15f790 con 0x55acad522a00 2024-06-04T05:00:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d99d2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55acad435890 con 0x55acad517820 2024-06-04T05:00:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.392+0000 7f42d99d2740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55acad489c30 con 0x55acad518140 2024-06-04T05:00:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42d99d2740 10 client.0 _ll_register_callbacks cb 0x55acad18da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42d517a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad522a00 0x55acad520490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42cffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad518140 0x55acad51ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42d4979640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad517820 0x55acad51fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42d517a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad522a00 0x55acad520490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50172/0 (socket says 172.21.15.70:50172) 2024-06-04T05:00:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42d517a640 1 -- 172.21.15.70:0/1452981891 learned_addr learned my addr 172.21.15.70:0/1452981891 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42d4979640 1 -- 172.21.15.70:0/1452981891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad518140 msgr2=0x55acad51ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42d4979640 1 --2- 172.21.15.70:0/1452981891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad518140 0x55acad51ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42d4979640 1 -- 172.21.15.70:0/1452981891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad522a00 msgr2=0x55acad520490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42d4979640 1 --2- 172.21.15.70:0/1452981891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad522a00 0x55acad520490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42d4979640 1 -- 172.21.15.70:0/1452981891 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acad25a620 con 0x55acad517820 2024-06-04T05:00:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42d4979640 1 --2- 172.21.15.70:0/1452981891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad517820 0x55acad51fa10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f42c4000c00 tx=0x7f42c4100150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=812975188ce9b29d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42affff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:00:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42affff640 1 -- 172.21.15.70:0/1452981891 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c4049040 con 0x55acad517820 2024-06-04T05:00:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42affff640 1 -- 172.21.15.70:0/1452981891 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f42c4100460 con 0x55acad517820 2024-06-04T05:00:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42d99d2740 1 -- 172.21.15.70:0/1452981891 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42c410a040 con 0x55acad517820 2024-06-04T05:00:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42affff640 1 -- 172.21.15.70:0/1452981891 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f42c40d2e20 con 0x55acad517820 2024-06-04T05:00:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.393+0000 7f42d99d2740 10 client.25676 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.394+0000 7f42d99d2740 1 -- 172.21.15.70:0/1452981891 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55acad531050 con 0x55acad517820 2024-06-04T05:00:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.394+0000 7f42affff640 1 -- 172.21.15.70:0/1452981891 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f42c410a040 con 0x55acad517820 2024-06-04T05:00:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.394+0000 7f42affff640 1 -- 172.21.15.70:0/1452981891 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f42c40c6000 con 0x55acad517820 2024-06-04T05:00:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.394+0000 7f42d99d2740 10 client.25676 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.394+0000 7f42d99d2740 10 client.25676 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.394+0000 7f42d99d2740 20 client.25676 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.394+0000 7f42d99d2740 1 --2- 172.21.15.70:0/1452981891 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55acad5324b0 0x55acad5348a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.394+0000 7f42d99d2740 4 client.25676 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:00:14.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.394+0000 7f42d99d2740 1 -- 172.21.15.70:0/1452981891 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55acad48bb80 con 0x55acad5324b0 2024-06-04T05:00:14.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.394+0000 7f42cffff640 1 --2- 172.21.15.70:0/1452981891 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55acad5324b0 0x55acad5348a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.394+0000 7f42affff640 10 client.25676 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:14.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.394+0000 7f42cffff640 1 --2- 172.21.15.70:0/1452981891 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55acad5324b0 0x55acad5348a0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd22f5580faa0448 server_cookie=526f0406a32796f in_seq=0 out_seq=0 2024-06-04T05:00:14.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.395+0000 7f42affff640 1 -- 172.21.15.70:0/1452981891 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55acad48bb80 con 0x55acad5324b0 2024-06-04T05:00:14.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.395+0000 7f42affff640 10 client.25676 handle_command_reply: tid=0 2024-06-04T05:00:14.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.395+0000 7f42d99d2740 1 client.25676 shutdown 2024-06-04T05:00:14.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.395+0000 7f42d99d2740 1 -- 172.21.15.70:0/1452981891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad517820 msgr2=0x55acad51fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:14.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.395+0000 7f42d99d2740 1 --2- 172.21.15.70:0/1452981891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad517820 0x55acad51fa10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f42c4000c00 tx=0x7f42c4100150 comp rx=0 tx=0).stop 2024-06-04T05:00:14.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.395+0000 7f42d99d2740 1 -- 172.21.15.70:0/1452981891 shutdown_connections 2024-06-04T05:00:14.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.395+0000 7f42d99d2740 1 --2- 172.21.15.70:0/1452981891 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55acad5324b0 0x55acad5348a0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.395+0000 7f42d99d2740 1 --2- 172.21.15.70:0/1452981891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad522a00 0x55acad520490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.395+0000 7f42affff640 0 client.25676 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:14.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.395+0000 7f42d99d2740 1 --2- 172.21.15.70:0/1452981891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad518140 0x55acad51ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.395+0000 7f42d99d2740 1 --2- 172.21.15.70:0/1452981891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad517820 0x55acad51fa10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.395+0000 7f42d99d2740 1 -- 172.21.15.70:0/1452981891 >> 172.21.15.70:0/1452981891 conn(0x55acad513d30 msgr2=0x55acad512fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:14.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.395+0000 7f42d99d2740 1 -- 172.21.15.70:0/1452981891 shutdown_connections 2024-06-04T05:00:14.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.395+0000 7f42d99d2740 1 -- 172.21.15.70:0/1452981891 wait complete. 2024-06-04T05:00:14.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.396+0000 7f42d99d2740 20 client.25676 trim_cache size 0 max 16384 2024-06-04T05:00:14.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42d99d2740 1 Processor -- start 2024-06-04T05:00:14.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42d99d2740 1 -- start start 2024-06-04T05:00:14.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42d99d2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad517820 0x55acad5202b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42d99d2740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad518140 0x55acad53a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42d99d2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad522a00 0x55acad53a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42d99d2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f42d01b0770 con 0x55acad517820 2024-06-04T05:00:14.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42d99d2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f42d01b2310 con 0x55acad522a00 2024-06-04T05:00:14.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42d99d2740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f42d01b0e40 con 0x55acad518140 2024-06-04T05:00:14.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42d4979640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad517820 0x55acad5202b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42d4979640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad517820 0x55acad5202b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50186/0 (socket says 172.21.15.70:50186) 2024-06-04T05:00:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42d4979640 1 -- 172.21.15.70:0/722736548 learned_addr learned my addr 172.21.15.70:0/722736548 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42cffff640 1 --2- 172.21.15.70:0/722736548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad518140 0x55acad53a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42d517a640 1 --2- 172.21.15.70:0/722736548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad522a00 0x55acad53a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42d4979640 1 -- 172.21.15.70:0/722736548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad518140 msgr2=0x55acad53a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42d4979640 1 --2- 172.21.15.70:0/722736548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad518140 0x55acad53a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42d4979640 1 -- 172.21.15.70:0/722736548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad522a00 msgr2=0x55acad53a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42d4979640 1 --2- 172.21.15.70:0/722736548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad522a00 0x55acad53a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.413+0000 7f42d4979640 1 -- 172.21.15.70:0/722736548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acad517f70 con 0x55acad517820 2024-06-04T05:00:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.414+0000 7f42d4979640 1 --2- 172.21.15.70:0/722736548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad517820 0x55acad5202b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f42c41007e0 tx=0x7f42c41001c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610d52315eea80f5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.414+0000 7f42ba7fc640 1 -- 172.21.15.70:0/722736548 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c408d030 con 0x55acad517820 2024-06-04T05:00:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.414+0000 7f42ba7fc640 1 -- 172.21.15.70:0/722736548 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f42c40c9880 con 0x55acad517820 2024-06-04T05:00:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.414+0000 7f42ba7fc640 1 -- 172.21.15.70:0/722736548 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c4049070 con 0x55acad517820 2024-06-04T05:00:14.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.414+0000 7f42d99d2740 1 -- 172.21.15.70:0/722736548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad517820 msgr2=0x55acad5202b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:14.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.414+0000 7f42d99d2740 1 --2- 172.21.15.70:0/722736548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad517820 0x55acad5202b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f42c41007e0 tx=0x7f42c41001c0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.414+0000 7f42d99d2740 1 -- 172.21.15.70:0/722736548 shutdown_connections 2024-06-04T05:00:14.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.414+0000 7f42d99d2740 1 --2- 172.21.15.70:0/722736548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad522a00 0x55acad53a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.414+0000 7f42d99d2740 1 --2- 172.21.15.70:0/722736548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad518140 0x55acad53a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.414+0000 7f42d99d2740 1 --2- 172.21.15.70:0/722736548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad517820 0x55acad5202b0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.415+0000 7f42d99d2740 1 -- 172.21.15.70:0/722736548 >> 172.21.15.70:0/722736548 conn(0x55acad513d30 msgr2=0x55acad50eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:14.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.415+0000 7f42d99d2740 1 -- 172.21.15.70:0/722736548 shutdown_connections 2024-06-04T05:00:14.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.415+0000 7f42d99d2740 1 -- 172.21.15.70:0/722736548 wait complete. 2024-06-04T05:00:14.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.415+0000 7f42d99d2740 1 -- start start 2024-06-04T05:00:14.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.415+0000 7f42d99d2740 1 Processor -- start 2024-06-04T05:00:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.415+0000 7f42d99d2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad517820 0x55acad525a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.415+0000 7f42d99d2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad518140 0x55acad525fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.415+0000 7f42d99d2740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad522a00 0x55acad59b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.415+0000 7f42d99d2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f42d01b1be0 con 0x55acad518140 2024-06-04T05:00:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.415+0000 7f42d99d2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f42d01b1510 con 0x55acad517820 2024-06-04T05:00:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.415+0000 7f42d99d2740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f42d01ae750 con 0x55acad522a00 2024-06-04T05:00:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42d99d2740 10 client.0 _ll_register_callbacks cb 0x7f42c4106050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42cffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad518140 0x55acad525fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42cffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad518140 0x55acad525fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50194/0 (socket says 172.21.15.70:50194) 2024-06-04T05:00:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42cffff640 1 -- 172.21.15.70:0/3034862973 learned_addr learned my addr 172.21.15.70:0/3034862973 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42d517a640 1 --2- 172.21.15.70:0/3034862973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad522a00 0x55acad59b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42d4979640 1 --2- 172.21.15.70:0/3034862973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad517820 0x55acad525a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42cffff640 1 -- 172.21.15.70:0/3034862973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad522a00 msgr2=0x55acad59b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42cffff640 1 --2- 172.21.15.70:0/3034862973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad522a00 0x55acad59b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42cffff640 1 -- 172.21.15.70:0/3034862973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad517820 msgr2=0x55acad525a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42cffff640 1 --2- 172.21.15.70:0/3034862973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad517820 0x55acad525a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42cffff640 1 -- 172.21.15.70:0/3034862973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acad59b620 con 0x55acad518140 2024-06-04T05:00:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42d517a640 1 --2- 172.21.15.70:0/3034862973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad522a00 0x55acad59b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42cffff640 1 --2- 172.21.15.70:0/3034862973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad518140 0x55acad525fa0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00ac30 tx=0x7f42bc010b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7123a9927416b825 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42affff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42affff640 1 -- 172.21.15.70:0/3034862973 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc01eab0 con 0x55acad518140 2024-06-04T05:00:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42affff640 1 -- 172.21.15.70:0/3034862973 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f42bc01ec50 con 0x55acad518140 2024-06-04T05:00:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42d99d2740 1 -- 172.21.15.70:0/3034862973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42c40c9880 con 0x55acad518140 2024-06-04T05:00:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42affff640 1 -- 172.21.15.70:0/3034862973 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f42bc010e50 con 0x55acad518140 2024-06-04T05:00:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42d99d2740 10 client.16281 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.416+0000 7f42d99d2740 1 -- 172.21.15.70:0/3034862973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55acad59c9c0 con 0x55acad518140 2024-06-04T05:00:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.417+0000 7f42affff640 1 -- 172.21.15.70:0/3034862973 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f42c40c9880 con 0x55acad518140 2024-06-04T05:00:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.417+0000 7f42affff640 1 -- 172.21.15.70:0/3034862973 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f42bc004af0 con 0x55acad518140 2024-06-04T05:00:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.417+0000 7f42d99d2740 10 client.16281 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.417+0000 7f42d99d2740 10 client.16281 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.417+0000 7f42d99d2740 20 client.16281 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.417+0000 7f42d99d2740 1 --2- 172.21.15.70:0/3034862973 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55acad5324b0 0x55acad59f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.417+0000 7f42d99d2740 4 client.16281 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:00:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.417+0000 7f42d99d2740 1 -- 172.21.15.70:0/3034862973 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55acad485360 con 0x55acad5324b0 2024-06-04T05:00:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.417+0000 7f42d4979640 1 --2- 172.21.15.70:0/3034862973 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55acad5324b0 0x55acad59f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.417+0000 7f42affff640 10 client.16281 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.417+0000 7f42d4979640 1 --2- 172.21.15.70:0/3034862973 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55acad5324b0 0x55acad59f210 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93ab0963b54145ec server_cookie=396d157beb87680c in_seq=0 out_seq=0 2024-06-04T05:00:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.439+0000 7f42affff640 1 -- 172.21.15.70:0/3034862973 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55acad485360 con 0x55acad5324b0 2024-06-04T05:00:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.439+0000 7f42affff640 10 client.16281 handle_command_reply: tid=0 2024-06-04T05:00:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.439+0000 7f42d99d2740 1 client.16281 shutdown 2024-06-04T05:00:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.439+0000 7f42d99d2740 1 -- 172.21.15.70:0/3034862973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad518140 msgr2=0x55acad525fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.439+0000 7f42d99d2740 1 --2- 172.21.15.70:0/3034862973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad518140 0x55acad525fa0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00ac30 tx=0x7f42bc010b40 comp rx=0 tx=0).stop 2024-06-04T05:00:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.439+0000 7f42d99d2740 1 -- 172.21.15.70:0/3034862973 shutdown_connections 2024-06-04T05:00:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.439+0000 7f42d99d2740 1 --2- 172.21.15.70:0/3034862973 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55acad5324b0 0x55acad59f210 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.439+0000 7f42d99d2740 1 --2- 172.21.15.70:0/3034862973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55acad522a00 0x55acad59b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.439+0000 7f42affff640 0 client.16281 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.439+0000 7f42d99d2740 1 --2- 172.21.15.70:0/3034862973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acad518140 0x55acad525fa0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.439+0000 7f42d99d2740 1 --2- 172.21.15.70:0/3034862973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55acad517820 0x55acad525a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.439+0000 7f42d99d2740 1 -- 172.21.15.70:0/3034862973 >> 172.21.15.70:0/3034862973 conn(0x55acad513d30 msgr2=0x55acad512720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.439+0000 7f42d99d2740 1 -- 172.21.15.70:0/3034862973 shutdown_connections 2024-06-04T05:00:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.439+0000 7f42d99d2740 1 -- 172.21.15.70:0/3034862973 wait complete. 2024-06-04T05:00:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.439+0000 7f42d99d2740 20 client.16281 trim_cache size 0 max 16384 2024-06-04T05:00:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:00:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:00:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:00:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.440+0000 7f42d6bdd640 1 -- 172.21.15.70:0/4217825155 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f42c00b2a10 msgr2=0x7f42c00b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.440+0000 7f42d6bdd640 1 --2- 172.21.15.70:0/4217825155 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f42c00b2a10 0x7f42c00b4ed0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f42d006b1e0 tx=0x7f42c8004a40 comp rx=0 tx=0).stop 2024-06-04T05:00:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.440+0000 7f42d6bdd640 1 -- 172.21.15.70:0/4217825155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d0101980 msgr2=0x7f42d0197c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.440+0000 7f42d6bdd640 1 --2- 172.21.15.70:0/4217825155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d0101980 0x7f42d0197c40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f42c4005940 tx=0x7f42c40042f0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.440+0000 7f42d6bdd640 1 -- 172.21.15.70:0/4217825155 shutdown_connections 2024-06-04T05:00:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.440+0000 7f42d6bdd640 1 --2- 172.21.15.70:0/4217825155 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f42c00b2a10 0x7f42c00b4ed0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.440+0000 7f42d6bdd640 1 --2- 172.21.15.70:0/4217825155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42d0107940 0x7f42d01986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.440+0000 7f42d6bdd640 1 --2- 172.21.15.70:0/4217825155 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42d0102340 0x7f42d0198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.440+0000 7f42d6bdd640 1 --2- 172.21.15.70:0/4217825155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d0101980 0x7f42d0197c40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.440+0000 7f42d6bdd640 1 -- 172.21.15.70:0/4217825155 >> 172.21.15.70:0/4217825155 conn(0x7f42d0078ad0 msgr2=0x7f42d0104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.440+0000 7f42d6bdd640 1 -- 172.21.15.70:0/4217825155 shutdown_connections 2024-06-04T05:00:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:14.440+0000 7f42d6bdd640 1 -- 172.21.15.70:0/4217825155 wait complete. 2024-06-04T05:00:14.659 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:00:14.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 37720 3.25 MB/sec warmup 67 sec latency 16.359 ms 2024-06-04T05:00:15.575 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:15 smithi070 ceph-mon[32631]: pgmap v95: 97 pgs: 97 active+clean; 39 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-06-04T05:00:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:15 smithi112 ceph-mon[36453]: pgmap v95: 97 pgs: 97 active+clean; 39 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-06-04T05:00:15.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 38273 3.28 MB/sec warmup 68 sec latency 45.755 ms 2024-06-04T05:00:15.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:15 smithi098 ceph-mon[35569]: pgmap v95: 97 pgs: 97 active+clean; 39 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-06-04T05:00:15.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.827+0000 7fa6e0c07700 1 -- 172.21.15.70:0/1882039688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6dc1024f0 msgr2=0x7fa6dc1103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:15.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.827+0000 7fa6e0c07700 1 --2- 172.21.15.70:0/1882039688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6dc1024f0 0x7fa6dc1103d0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400b0d0 tx=0x7fa6d400b3e0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.827+0000 7fa6e0c07700 1 -- 172.21.15.70:0/1882039688 shutdown_connections 2024-06-04T05:00:15.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.827+0000 7fa6e0c07700 1 --2- 172.21.15.70:0/1882039688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6dc1024f0 0x7fa6dc1103d0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.827+0000 7fa6e0c07700 1 --2- 172.21.15.70:0/1882039688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6dc109520 0x7fa6dc109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.827+0000 7fa6e0c07700 1 --2- 172.21.15.70:0/1882039688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6dc101b30 0x7fa6dc101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.827+0000 7fa6e0c07700 1 -- 172.21.15.70:0/1882039688 >> 172.21.15.70:0/1882039688 conn(0x7fa6dc078ab0 msgr2=0x7fa6dc0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:15.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.828+0000 7fa6e0c07700 1 -- 172.21.15.70:0/1882039688 shutdown_connections 2024-06-04T05:00:15.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.828+0000 7fa6e0c07700 1 -- 172.21.15.70:0/1882039688 wait complete. 2024-06-04T05:00:15.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.828+0000 7fa6e0c07700 1 Processor -- start 2024-06-04T05:00:15.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.828+0000 7fa6e0c07700 1 -- start start 2024-06-04T05:00:15.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6e0c07700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6dc101b30 0x7fa6dc19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6e0c07700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6dc1024f0 0x7fa6dc19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6e0c07700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6dc109520 0x7fa6dc1998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6e0c07700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc113610 con 0x7fa6dc109520 2024-06-04T05:00:15.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6e0c07700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc113490 con 0x7fa6dc1024f0 2024-06-04T05:00:15.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6e0c07700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc113790 con 0x7fa6dc101b30 2024-06-04T05:00:15.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6dad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6dc109520 0x7fa6dc1998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6da59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6dc101b30 0x7fa6dc19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6d9d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6dc1024f0 0x7fa6dc19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6da59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6dc101b30 0x7fa6dc19f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59790/0 (socket says 172.21.15.70:59790) 2024-06-04T05:00:15.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6dad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6dc109520 0x7fa6dc1998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50212/0 (socket says 172.21.15.70:50212) 2024-06-04T05:00:15.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6d9d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6dc1024f0 0x7fa6dc19fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:60654/0 (socket says 172.21.15.70:60654) 2024-06-04T05:00:15.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6da59c700 1 -- 172.21.15.70:0/1220337326 learned_addr learned my addr 172.21.15.70:0/1220337326 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:15.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6d9d9b700 1 -- 172.21.15.70:0/1220337326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6dc101b30 msgr2=0x7fa6dc19f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:15.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6d9d9b700 1 --2- 172.21.15.70:0/1220337326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6dc101b30 0x7fa6dc19f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6d9d9b700 1 -- 172.21.15.70:0/1220337326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6dc109520 msgr2=0x7fa6dc1998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:15.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6d9d9b700 1 --2- 172.21.15.70:0/1220337326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6dc109520 0x7fa6dc1998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6d9d9b700 1 -- 172.21.15.70:0/1220337326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6dc19a050 con 0x7fa6dc1024f0 2024-06-04T05:00:15.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6dad9d700 1 --2- 172.21.15.70:0/1220337326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6dc109520 0x7fa6dc1998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:15.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6d9d9b700 1 --2- 172.21.15.70:0/1220337326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6dc1024f0 0x7fa6dc19fc70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0009c50 tx=0x7fa6d0009f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:15.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6cb7fe700 1 -- 172.21.15.70:0/1220337326 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d001e070 con 0x7fa6dc1024f0 2024-06-04T05:00:15.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6e0c07700 1 -- 172.21.15.70:0/1220337326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6dc19a340 con 0x7fa6dc1024f0 2024-06-04T05:00:15.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6cb7fe700 1 -- 172.21.15.70:0/1220337326 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa6d0019410 con 0x7fa6dc1024f0 2024-06-04T05:00:15.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.829+0000 7fa6cb7fe700 1 -- 172.21.15.70:0/1220337326 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0018610 con 0x7fa6dc1024f0 2024-06-04T05:00:15.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.830+0000 7fa6e0c07700 1 -- 172.21.15.70:0/1220337326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6dc102c40 con 0x7fa6dc1024f0 2024-06-04T05:00:15.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.832+0000 7fa6cb7fe700 1 -- 172.21.15.70:0/1220337326 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fa6d0021070 con 0x7fa6dc1024f0 2024-06-04T05:00:15.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.833+0000 7fa6cb7fe700 1 --2- 172.21.15.70:0/1220337326 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa6c40b29f0 0x7fa6c40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.833+0000 7fa6da59c700 1 --2- 172.21.15.70:0/1220337326 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa6c40b29f0 0x7fa6c40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.833+0000 7fa6cb7fe700 1 -- 172.21.15.70:0/1220337326 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fa6d00e5b50 con 0x7fa6dc1024f0 2024-06-04T05:00:15.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.833+0000 7fa6da59c700 1 --2- 172.21.15.70:0/1220337326 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa6c40b29f0 0x7fa6c40b4eb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc19aa30 tx=0x7fa6cc009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:15.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.833+0000 7fa6eb49cb80 1 Processor -- start 2024-06-04T05:00:15.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.834+0000 7fa6eb49cb80 1 -- start start 2024-06-04T05:00:15.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.834+0000 7fa6eb49cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a5320 0x55c6db2a5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.834+0000 7fa6eb49cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 0x55c6db29fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.834+0000 7fa6eb49cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db2a0260 0x55c6db2ac080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.834+0000 7fa6eb49cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c6db1fdd70 con 0x55c6db29f8a0 2024-06-04T05:00:15.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.834+0000 7fa6eb49cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c6db1ed910 con 0x55c6db2a5320 2024-06-04T05:00:15.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.834+0000 7fa6d9d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 0x55c6db29fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.834+0000 7fa6eb49cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c6db1f1950 con 0x55c6db2a0260 2024-06-04T05:00:15.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.834+0000 7fa6d9d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 0x55c6db29fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50218/0 (socket says 172.21.15.70:50218) 2024-06-04T05:00:15.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.834+0000 7fa6d9d9b700 1 -- 172.21.15.70:0/36221683 learned_addr learned my addr 172.21.15.70:0/36221683 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:15.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.834+0000 7fa6da59c700 1 --2- 172.21.15.70:0/36221683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a5320 0x55c6db2a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.834+0000 7fa6dad9d700 1 --2- 172.21.15.70:0/36221683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db2a0260 0x55c6db2ac080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.834+0000 7fa6dad9d700 1 -- 172.21.15.70:0/36221683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a5320 msgr2=0x55c6db2a5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:15.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.834+0000 7fa6dad9d700 1 --2- 172.21.15.70:0/36221683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a5320 0x55c6db2a5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.834+0000 7fa6dad9d700 1 -- 172.21.15.70:0/36221683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 msgr2=0x55c6db29fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:15.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.834+0000 7fa6dad9d700 1 --2- 172.21.15.70:0/36221683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 0x55c6db29fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.834+0000 7fa6dad9d700 1 -- 172.21.15.70:0/36221683 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6db196fb0 con 0x55c6db2a0260 2024-06-04T05:00:15.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.835+0000 7fa6dad9d700 1 --2- 172.21.15.70:0/36221683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db2a0260 0x55c6db2ac080 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400b770 tx=0x7fa6d400bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb8c99e9096a7c1a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:15.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.835+0000 7fa6c8ff9700 1 -- 172.21.15.70:0/36221683 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d40092e0 con 0x55c6db2a0260 2024-06-04T05:00:15.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.835+0000 7fa6c8ff9700 1 -- 172.21.15.70:0/36221683 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa6d4009480 con 0x55c6db2a0260 2024-06-04T05:00:15.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.835+0000 7fa6c8ff9700 1 -- 172.21.15.70:0/36221683 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d40074e0 con 0x55c6db2a0260 2024-06-04T05:00:15.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/36221683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db2a0260 msgr2=0x55c6db2ac080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:15.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/36221683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db2a0260 0x55c6db2ac080 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400b770 tx=0x7fa6d400bf30 comp rx=0 tx=0).stop 2024-06-04T05:00:15.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/36221683 shutdown_connections 2024-06-04T05:00:15.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/36221683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db2a0260 0x55c6db2ac080 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/36221683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a5320 0x55c6db2a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/36221683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 0x55c6db29fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/36221683 >> 172.21.15.70:0/36221683 conn(0x55c6db1cb240 msgr2=0x55c6db29a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:15.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/36221683 shutdown_connections 2024-06-04T05:00:15.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/36221683 wait complete. 2024-06-04T05:00:15.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 1 -- start start 2024-06-04T05:00:15.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 1 Processor -- start 2024-06-04T05:00:15.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db29f8a0 0x55c6db2abcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db2a0260 0x55c6db2a8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a5320 0x55c6db2a9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c6db0f3fa0 con 0x55c6db2a0260 2024-06-04T05:00:15.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c6db155da0 con 0x55c6db2a5320 2024-06-04T05:00:15.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c6db2ac680 con 0x55c6db29f8a0 2024-06-04T05:00:15.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.836+0000 7fa6eb49cb80 10 client.0 _ll_register_callbacks cb 0x55c6db188950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:15.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6d9d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db2a0260 0x55c6db2a8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6da59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db29f8a0 0x55c6db2abcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6d9d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db2a0260 0x55c6db2a8fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50226/0 (socket says 172.21.15.70:50226) 2024-06-04T05:00:15.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6d9d9b700 1 -- 172.21.15.70:0/4123231806 learned_addr learned my addr 172.21.15.70:0/4123231806 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:15.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6dad9d700 1 --2- 172.21.15.70:0/4123231806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a5320 0x55c6db2a9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6da59c700 1 -- 172.21.15.70:0/4123231806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a5320 msgr2=0x55c6db2a9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:15.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6da59c700 1 --2- 172.21.15.70:0/4123231806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a5320 0x55c6db2a9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6da59c700 1 -- 172.21.15.70:0/4123231806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db2a0260 msgr2=0x55c6db2a8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6da59c700 1 --2- 172.21.15.70:0/4123231806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db2a0260 0x55c6db2a8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6da59c700 1 -- 172.21.15.70:0/4123231806 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6db13d8d0 con 0x55c6db29f8a0 2024-06-04T05:00:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6da59c700 1 --2- 172.21.15.70:0/4123231806 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db29f8a0 0x55c6db2abcd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc009880 tx=0x7fa6cc0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53008af94549834 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6c1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:00:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6c1ffb700 1 -- 172.21.15.70:0/4123231806 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc01f990 con 0x55c6db29f8a0 2024-06-04T05:00:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/4123231806 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6d4009480 con 0x55c6db29f8a0 2024-06-04T05:00:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6c1ffb700 1 -- 172.21.15.70:0/4123231806 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fa6cc01fb30 con 0x55c6db29f8a0 2024-06-04T05:00:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6c1ffb700 1 -- 172.21.15.70:0/4123231806 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa6cc01fd30 con 0x55c6db29f8a0 2024-06-04T05:00:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6eb49cb80 10 client.25756 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.837+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/4123231806 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c6db1a7760 con 0x55c6db29f8a0 2024-06-04T05:00:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.838+0000 7fa6c1ffb700 1 -- 172.21.15.70:0/4123231806 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa6d4009480 con 0x55c6db29f8a0 2024-06-04T05:00:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.838+0000 7fa6c1ffb700 1 -- 172.21.15.70:0/4123231806 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fa6cc026070 con 0x55c6db29f8a0 2024-06-04T05:00:15.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.838+0000 7fa6eb49cb80 10 client.25756 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:15.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.838+0000 7fa6eb49cb80 10 client.25756 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:00:15.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.838+0000 7fa6eb49cb80 20 client.25756 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:15.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.838+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/4123231806 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55c6db2b91d0 0x55c6db2bb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.838+0000 7fa6eb49cb80 4 client.25756 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:00:15.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.838+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/4123231806 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6db1ffcc0 con 0x55c6db2b91d0 2024-06-04T05:00:15.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.838+0000 7fa6d9d9b700 1 --2- 172.21.15.70:0/4123231806 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55c6db2b91d0 0x55c6db2bb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.838+0000 7fa6c1ffb700 10 client.25756 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:00:15.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.838+0000 7fa6d9d9b700 1 --2- 172.21.15.70:0/4123231806 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55c6db2b91d0 0x55c6db2bb5e0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=109b2574b103c3dc server_cookie=ce7bcc97097dccf0 in_seq=0 out_seq=0 2024-06-04T05:00:15.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.839+0000 7fa6c1ffb700 1 -- 172.21.15.70:0/4123231806 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c6db1ffcc0 con 0x55c6db2b91d0 2024-06-04T05:00:15.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.839+0000 7fa6c1ffb700 10 client.25756 handle_command_reply: tid=0 2024-06-04T05:00:15.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.839+0000 7fa6eb49cb80 1 client.25756 shutdown 2024-06-04T05:00:15.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.839+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/4123231806 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db29f8a0 msgr2=0x55c6db2abcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:15.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.839+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/4123231806 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db29f8a0 0x55c6db2abcd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc009880 tx=0x7fa6cc0047c0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.839+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/4123231806 shutdown_connections 2024-06-04T05:00:15.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.839+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/4123231806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a5320 0x55c6db2a9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.839+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/4123231806 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55c6db2b91d0 0x55c6db2bb5e0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.839+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/4123231806 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db29f8a0 0x55c6db2abcd0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.839+0000 7fa6c1ffb700 0 client.25756 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:00:15.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.839+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/4123231806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db2a0260 0x55c6db2a8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.839+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/4123231806 >> 172.21.15.70:0/4123231806 conn(0x55c6db1cb240 msgr2=0x55c6db299af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:15.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.839+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/4123231806 shutdown_connections 2024-06-04T05:00:15.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.840+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/4123231806 wait complete. 2024-06-04T05:00:15.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.840+0000 7fa6eb49cb80 20 client.25756 trim_cache size 0 max 16384 2024-06-04T05:00:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.857+0000 7fa6eb49cb80 1 Processor -- start 2024-06-04T05:00:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.857+0000 7fa6eb49cb80 1 -- start start 2024-06-04T05:00:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.857+0000 7fa6eb49cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 0x55c6db2a3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.857+0000 7fa6eb49cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a0260 0x55c6db29a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.857+0000 7fa6eb49cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db2a5320 0x55c6db2a6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.857+0000 7fa6eb49cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc1abab0 con 0x55c6db29f8a0 2024-06-04T05:00:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.857+0000 7fa6eb49cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc1ad650 con 0x55c6db2a0260 2024-06-04T05:00:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.857+0000 7fa6eb49cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc1ac180 con 0x55c6db2a5320 2024-06-04T05:00:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.858+0000 7fa6da59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 0x55c6db2a3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.858+0000 7fa6da59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 0x55c6db2a3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50242/0 (socket says 172.21.15.70:50242) 2024-06-04T05:00:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.858+0000 7fa6da59c700 1 -- 172.21.15.70:0/1982898162 learned_addr learned my addr 172.21.15.70:0/1982898162 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.858+0000 7fa6d9d9b700 1 --2- 172.21.15.70:0/1982898162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a0260 0x55c6db29a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.858+0000 7fa6dad9d700 1 --2- 172.21.15.70:0/1982898162 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db2a5320 0x55c6db2a6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.858+0000 7fa6da59c700 1 -- 172.21.15.70:0/1982898162 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db2a5320 msgr2=0x55c6db2a6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.858+0000 7fa6da59c700 1 --2- 172.21.15.70:0/1982898162 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db2a5320 0x55c6db2a6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.858+0000 7fa6da59c700 1 -- 172.21.15.70:0/1982898162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a0260 msgr2=0x55c6db29a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.858+0000 7fa6da59c700 1 --2- 172.21.15.70:0/1982898162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a0260 0x55c6db29a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.858+0000 7fa6da59c700 1 -- 172.21.15.70:0/1982898162 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6db2a9890 con 0x55c6db29f8a0 2024-06-04T05:00:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.858+0000 7fa6da59c700 1 --2- 172.21.15.70:0/1982898162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 0x55c6db2a3460 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc005c50 tx=0x7fa6cc01f480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa2f727f7951131d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.858+0000 7fa6c3fff700 1 -- 172.21.15.70:0/1982898162 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc01f990 con 0x55c6db29f8a0 2024-06-04T05:00:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.858+0000 7fa6c3fff700 1 -- 172.21.15.70:0/1982898162 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa6cc01fb30 con 0x55c6db29f8a0 2024-06-04T05:00:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.858+0000 7fa6c3fff700 1 -- 172.21.15.70:0/1982898162 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc028aa0 con 0x55c6db29f8a0 2024-06-04T05:00:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.859+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/1982898162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 msgr2=0x55c6db2a3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.859+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/1982898162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 0x55c6db2a3460 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc005c50 tx=0x7fa6cc01f480 comp rx=0 tx=0).stop 2024-06-04T05:00:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.859+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/1982898162 shutdown_connections 2024-06-04T05:00:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.859+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/1982898162 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db2a5320 0x55c6db2a6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.859+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/1982898162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 0x55c6db2a3460 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.859+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/1982898162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a0260 0x55c6db29a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.859+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/1982898162 >> 172.21.15.70:0/1982898162 conn(0x55c6db1cb240 msgr2=0x55c6db2a2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.859+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/1982898162 shutdown_connections 2024-06-04T05:00:15.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.860+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/1982898162 wait complete. 2024-06-04T05:00:15.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.860+0000 7fa6eb49cb80 1 -- start start 2024-06-04T05:00:15.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.860+0000 7fa6eb49cb80 1 Processor -- start 2024-06-04T05:00:15.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.860+0000 7fa6eb49cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 0x55c6db2b1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.860+0000 7fa6eb49cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a0260 0x55c6db2c0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.860+0000 7fa6eb49cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db2a5320 0x55c6db2c0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.860+0000 7fa6eb49cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc1acf20 con 0x55c6db29f8a0 2024-06-04T05:00:15.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.860+0000 7fa6eb49cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc1ac850 con 0x55c6db2a0260 2024-06-04T05:00:15.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.860+0000 7fa6eb49cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc1a9a50 con 0x55c6db2a5320 2024-06-04T05:00:15.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.860+0000 7fa6eb49cb80 10 client.0 _ll_register_callbacks cb 0x55c6db188950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:15.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.860+0000 7fa6da59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 0x55c6db2b1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.860+0000 7fa6da59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 0x55c6db2b1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50258/0 (socket says 172.21.15.70:50258) 2024-06-04T05:00:15.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.860+0000 7fa6da59c700 1 -- 172.21.15.70:0/4293839184 learned_addr learned my addr 172.21.15.70:0/4293839184 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:15.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.860+0000 7fa6d9d9b700 1 --2- 172.21.15.70:0/4293839184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a0260 0x55c6db2c0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.860+0000 7fa6dad9d700 1 --2- 172.21.15.70:0/4293839184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db2a5320 0x55c6db2c0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.861+0000 7fa6da59c700 1 -- 172.21.15.70:0/4293839184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db2a5320 msgr2=0x55c6db2c0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:15.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.861+0000 7fa6da59c700 1 --2- 172.21.15.70:0/4293839184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db2a5320 0x55c6db2c0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.861+0000 7fa6da59c700 1 -- 172.21.15.70:0/4293839184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a0260 msgr2=0x55c6db2c0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:15.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.861+0000 7fa6da59c700 1 --2- 172.21.15.70:0/4293839184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a0260 0x55c6db2c0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.861+0000 7fa6da59c700 1 -- 172.21.15.70:0/4293839184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6db2c1230 con 0x55c6db29f8a0 2024-06-04T05:00:15.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.861+0000 7fa6d9d9b700 1 --2- 172.21.15.70:0/4293839184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a0260 0x55c6db2c0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:15.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.861+0000 7fa6da59c700 1 --2- 172.21.15.70:0/4293839184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 0x55c6db2b1eb0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc010010 tx=0x7fa6cc005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49dfae8550f7d3d1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:15.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.861+0000 7fa6c1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:15.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.861+0000 7fa6c1ffb700 1 -- 172.21.15.70:0/4293839184 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc0049c0 con 0x55c6db29f8a0 2024-06-04T05:00:15.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.861+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/4293839184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6cc01fb30 con 0x55c6db29f8a0 2024-06-04T05:00:15.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.861+0000 7fa6c1ffb700 1 -- 172.21.15.70:0/4293839184 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fa6cc004b60 con 0x55c6db29f8a0 2024-06-04T05:00:15.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.861+0000 7fa6c1ffb700 1 -- 172.21.15.70:0/4293839184 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa6cc004d60 con 0x55c6db29f8a0 2024-06-04T05:00:15.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.861+0000 7fa6eb49cb80 10 client.16299 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:15.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.861+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/4293839184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c6db323ca0 con 0x55c6db29f8a0 2024-06-04T05:00:15.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.862+0000 7fa6c1ffb700 1 -- 172.21.15.70:0/4293839184 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa6cc01fb30 con 0x55c6db29f8a0 2024-06-04T05:00:15.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.862+0000 7fa6c1ffb700 1 -- 172.21.15.70:0/4293839184 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fa6cc037070 con 0x55c6db29f8a0 2024-06-04T05:00:15.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.862+0000 7fa6eb49cb80 10 client.16299 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:15.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.862+0000 7fa6eb49cb80 10 client.16299 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:00:15.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.862+0000 7fa6eb49cb80 20 client.16299 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:15.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.862+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/4293839184 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55c6db324290 0x55c6db3266b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:15.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.862+0000 7fa6eb49cb80 4 client.16299 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:00:15.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.862+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/4293839184 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c6db1f94a0 con 0x55c6db324290 2024-06-04T05:00:15.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.862+0000 7fa6d9d9b700 1 --2- 172.21.15.70:0/4293839184 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55c6db324290 0x55c6db3266b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:15.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.863+0000 7fa6d9d9b700 1 --2- 172.21.15.70:0/4293839184 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55c6db324290 0x55c6db3266b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3bd21272ca2aeee7 server_cookie=53f265824e047bee in_seq=0 out_seq=0 2024-06-04T05:00:15.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.863+0000 7fa6c1ffb700 10 client.16299 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:00:15.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.863+0000 7fa6c1ffb700 1 -- 172.21.15.70:0/4293839184 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c6db1f94a0 con 0x55c6db324290 2024-06-04T05:00:15.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.863+0000 7fa6c1ffb700 10 client.16299 handle_command_reply: tid=0 2024-06-04T05:00:15.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.863+0000 7fa6eb49cb80 1 client.16299 shutdown 2024-06-04T05:00:15.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.863+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/4293839184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 msgr2=0x55c6db2b1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:15.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.863+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/4293839184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 0x55c6db2b1eb0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc010010 tx=0x7fa6cc005a00 comp rx=0 tx=0).stop 2024-06-04T05:00:15.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.863+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/4293839184 shutdown_connections 2024-06-04T05:00:15.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.863+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/4293839184 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55c6db324290 0x55c6db3266b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.863+0000 7fa6c1ffb700 0 client.16299 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:00:15.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.863+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/4293839184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c6db2a5320 0x55c6db2c0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.863+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/4293839184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6db29f8a0 0x55c6db2b1eb0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.863+0000 7fa6eb49cb80 1 --2- 172.21.15.70:0/4293839184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6db2a0260 0x55c6db2c0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.863+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/4293839184 >> 172.21.15.70:0/4293839184 conn(0x55c6db1cb240 msgr2=0x55c6db2a5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:15.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.863+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/4293839184 shutdown_connections 2024-06-04T05:00:15.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.863+0000 7fa6eb49cb80 1 -- 172.21.15.70:0/4293839184 wait complete. 2024-06-04T05:00:15.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.863+0000 7fa6eb49cb80 20 client.16299 trim_cache size 0 max 16384 2024-06-04T05:00:15.872 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:00:15.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.864+0000 7fa6e0c07700 1 -- 172.21.15.70:0/1220337326 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa6c40b29f0 msgr2=0x7fa6c40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:15.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.864+0000 7fa6e0c07700 1 --2- 172.21.15.70:0/1220337326 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa6c40b29f0 0x7fa6c40b4eb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc19aa30 tx=0x7fa6cc009260 comp rx=0 tx=0).stop 2024-06-04T05:00:15.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.864+0000 7fa6e0c07700 1 -- 172.21.15.70:0/1220337326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6dc1024f0 msgr2=0x7fa6dc19fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:15.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.864+0000 7fa6e0c07700 1 --2- 172.21.15.70:0/1220337326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6dc1024f0 0x7fa6dc19fc70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0009c50 tx=0x7fa6d0009f60 comp rx=0 tx=0).stop 2024-06-04T05:00:15.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.864+0000 7fa6e0c07700 1 -- 172.21.15.70:0/1220337326 shutdown_connections 2024-06-04T05:00:15.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.864+0000 7fa6e0c07700 1 --2- 172.21.15.70:0/1220337326 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa6c40b29f0 0x7fa6c40b4eb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.864+0000 7fa6e0c07700 1 --2- 172.21.15.70:0/1220337326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6dc109520 0x7fa6dc1998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.864+0000 7fa6e0c07700 1 --2- 172.21.15.70:0/1220337326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6dc101b30 0x7fa6dc19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.864+0000 7fa6e0c07700 1 --2- 172.21.15.70:0/1220337326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6dc1024f0 0x7fa6dc19fc70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:15.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.864+0000 7fa6e0c07700 1 -- 172.21.15.70:0/1220337326 >> 172.21.15.70:0/1220337326 conn(0x7fa6dc078ab0 msgr2=0x7fa6dc1003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:15.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.864+0000 7fa6e0c07700 1 -- 172.21.15.70:0/1220337326 shutdown_connections 2024-06-04T05:00:15.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:15.864+0000 7fa6e0c07700 1 -- 172.21.15.70:0/1220337326 wait complete. 2024-06-04T05:00:16.678 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:00:16.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 38838 3.28 MB/sec warmup 69 sec latency 16.713 ms 2024-06-04T05:00:17.471 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:17 smithi070 ceph-mon[32631]: pgmap v96: 97 pgs: 97 active+clean; 36 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 1.9 MiB/s wr, 201 op/s 2024-06-04T05:00:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:17 smithi112 ceph-mon[36453]: pgmap v96: 97 pgs: 97 active+clean; 36 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 1.9 MiB/s wr, 201 op/s 2024-06-04T05:00:17.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 39329 3.30 MB/sec warmup 70 sec latency 15.051 ms 2024-06-04T05:00:17.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.720+0000 7f21f827c700 1 -- 172.21.15.70:0/1195568467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21f0102430 msgr2=0x7f21f01028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:17.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.720+0000 7f21f827c700 1 --2- 172.21.15.70:0/1195568467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21f0102430 0x7f21f01028b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f21e0009990 tx=0x7f21e0009ca0 comp rx=0 tx=0).stop 2024-06-04T05:00:17.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.720+0000 7f21f827c700 1 -- 172.21.15.70:0/1195568467 shutdown_connections 2024-06-04T05:00:17.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.720+0000 7f21f827c700 1 --2- 172.21.15.70:0/1195568467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f0102df0 0x7f21f010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:17.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.720+0000 7f21f827c700 1 --2- 172.21.15.70:0/1195568467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21f0108510 0x7f21f01088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:17.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.720+0000 7f21f827c700 1 --2- 172.21.15.70:0/1195568467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21f0102430 0x7f21f01028b0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:17.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.720+0000 7f21f827c700 1 -- 172.21.15.70:0/1195568467 >> 172.21.15.70:0/1195568467 conn(0x7f21f00793b0 msgr2=0x7f21f00797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:17.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.720+0000 7f21f827c700 1 -- 172.21.15.70:0/1195568467 shutdown_connections 2024-06-04T05:00:17.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.720+0000 7f21f827c700 1 -- 172.21.15.70:0/1195568467 wait complete. 2024-06-04T05:00:17.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.721+0000 7f21f827c700 1 Processor -- start 2024-06-04T05:00:17.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.721+0000 7f21f827c700 1 -- start start 2024-06-04T05:00:17.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.721+0000 7f21f827c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21f0102430 0x7f21f019a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:17.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.721+0000 7f21f827c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f0102df0 0x7f21f019aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:17.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.721+0000 7f21f827c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21f0108510 0x7f21f01a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:17.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.722+0000 7f21f6018700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21f0102430 0x7f21f019a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:17.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.722+0000 7f21f6018700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21f0102430 0x7f21f019a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51900/0 (socket says 172.21.15.70:51900) 2024-06-04T05:00:17.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.722+0000 7f21f6819700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21f0108510 0x7f21f01a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:17.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.722+0000 7f21f5817700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f0102df0 0x7f21f019aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:17.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.722+0000 7f21f6819700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21f0108510 0x7f21f01a5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39886/0 (socket says 172.21.15.70:39886) 2024-06-04T05:00:17.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.722+0000 7f21f6819700 1 -- 172.21.15.70:0/2438000478 learned_addr learned my addr 172.21.15.70:0/2438000478 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:17.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.722+0000 7f21f827c700 1 -- 172.21.15.70:0/2438000478 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f21f0112490 con 0x7f21f0108510 2024-06-04T05:00:17.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.722+0000 7f21f827c700 1 -- 172.21.15.70:0/2438000478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21f0112310 con 0x7f21f0102df0 2024-06-04T05:00:17.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.722+0000 7f21f827c700 1 -- 172.21.15.70:0/2438000478 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f21f0112610 con 0x7f21f0102430 2024-06-04T05:00:17.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.722+0000 7f21f6819700 1 -- 172.21.15.70:0/2438000478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21f0102430 msgr2=0x7f21f019a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:17.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.722+0000 7f21f6819700 1 --2- 172.21.15.70:0/2438000478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21f0102430 0x7f21f019a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:17.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.722+0000 7f21f6819700 1 -- 172.21.15.70:0/2438000478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f0102df0 msgr2=0x7f21f019aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:17.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.722+0000 7f21f6819700 1 --2- 172.21.15.70:0/2438000478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f0102df0 0x7f21f019aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:17.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.722+0000 7f21f6819700 1 -- 172.21.15.70:0/2438000478 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f01a5f50 con 0x7f21f0108510 2024-06-04T05:00:17.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.722+0000 7f21f6819700 1 --2- 172.21.15.70:0/2438000478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21f0108510 0x7f21f01a5a10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f21e800ef30 tx=0x7f21e800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:17.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.723+0000 7f21e77fe700 1 -- 172.21.15.70:0/2438000478 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e80092e0 con 0x7f21f0108510 2024-06-04T05:00:17.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.723+0000 7f21f827c700 1 -- 172.21.15.70:0/2438000478 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21f01a6120 con 0x7f21f0108510 2024-06-04T05:00:17.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.723+0000 7f21e77fe700 1 -- 172.21.15.70:0/2438000478 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21e8004bb0 con 0x7f21f0108510 2024-06-04T05:00:17.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.723+0000 7f21e77fe700 1 -- 172.21.15.70:0/2438000478 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e8004020 con 0x7f21f0108510 2024-06-04T05:00:17.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.723+0000 7f21f827c700 1 -- 172.21.15.70:0/2438000478 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21f01a6630 con 0x7f21f0108510 2024-06-04T05:00:17.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.724+0000 7f21f827c700 1 -- 172.21.15.70:0/2438000478 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21f010a9a0 con 0x7f21f0108510 2024-06-04T05:00:17.727 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:00:17.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.725+0000 7f21e77fe700 1 -- 172.21.15.70:0/2438000478 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f21e801b030 con 0x7f21f0108510 2024-06-04T05:00:17.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.726+0000 7f21e77fe700 1 --2- 172.21.15.70:0/2438000478 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f21dc0b2a70 0x7f21dc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:17.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.727+0000 7f21f6018700 1 --2- 172.21.15.70:0/2438000478 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f21dc0b2a70 0x7f21dc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:17.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.727+0000 7f21e77fe700 1 -- 172.21.15.70:0/2438000478 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f21e80e2a50 con 0x7f21f0108510 2024-06-04T05:00:17.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.727+0000 7f21f6018700 1 --2- 172.21.15.70:0/2438000478 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f21dc0b2a70 0x7f21dc0b4f30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f21ec0045b0 tx=0x7f21ec00b3c0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:17.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:17.731+0000 7f21e77fe700 1 -- 172.21.15.70:0/2438000478 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f21e8022020 con 0x7f21f0108510 2024-06-04T05:00:17.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:17 smithi098 ceph-mon[35569]: pgmap v96: 97 pgs: 97 active+clean; 36 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 1.9 MiB/s wr, 201 op/s 2024-06-04T05:00:18.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:18.039+0000 7f21f827c700 1 -- 172.21.15.70:0/2438000478 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21f0103b50 con 0x7f21f0108510 2024-06-04T05:00:18.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:18.040+0000 7f21e77fe700 1 -- 172.21.15.70:0/2438000478 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11932 (secure 0 0 0) 0x7f21f0103b50 con 0x7f21f0108510 2024-06-04T05:00:18.042 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:00:18.042 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:59:22.801688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:00:18.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:18.042+0000 7f21f827c700 1 -- 172.21.15.70:0/2438000478 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f21dc0b2a70 msgr2=0x7f21dc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:18.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:18.042+0000 7f21f827c700 1 --2- 172.21.15.70:0/2438000478 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f21dc0b2a70 0x7f21dc0b4f30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f21ec0045b0 tx=0x7f21ec00b3c0 comp rx=0 tx=0).stop 2024-06-04T05:00:18.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:18.042+0000 7f21f827c700 1 -- 172.21.15.70:0/2438000478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21f0108510 msgr2=0x7f21f01a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:18.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:18.042+0000 7f21f827c700 1 --2- 172.21.15.70:0/2438000478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21f0108510 0x7f21f01a5a10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f21e800ef30 tx=0x7f21e800c3a0 comp rx=0 tx=0).stop 2024-06-04T05:00:18.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:18.042+0000 7f21f827c700 1 -- 172.21.15.70:0/2438000478 shutdown_connections 2024-06-04T05:00:18.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:18.042+0000 7f21f827c700 1 --2- 172.21.15.70:0/2438000478 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f21dc0b2a70 0x7f21dc0b4f30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:18.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:18.042+0000 7f21f827c700 1 --2- 172.21.15.70:0/2438000478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21f0108510 0x7f21f01a5a10 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:18.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:18.042+0000 7f21f827c700 1 --2- 172.21.15.70:0/2438000478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21f0102430 0x7f21f019a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:18.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:18.042+0000 7f21f827c700 1 --2- 172.21.15.70:0/2438000478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f0102df0 0x7f21f019aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:18.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:18.042+0000 7f21f827c700 1 -- 172.21.15.70:0/2438000478 >> 172.21.15.70:0/2438000478 conn(0x7f21f00793b0 msgr2=0x7f21f0100640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:18.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:18.042+0000 7f21f827c700 1 -- 172.21.15.70:0/2438000478 shutdown_connections 2024-06-04T05:00:18.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:18.042+0000 7f21f827c700 1 -- 172.21.15.70:0/2438000478 wait complete. 2024-06-04T05:00:18.045 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-06-04T05:00:18.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:18 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2438000478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:00:18.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:18 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2438000478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:00:18.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 39613 3.26 MB/sec warmup 71 sec latency 12.823 ms 2024-06-04T05:00:18.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:18 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2438000478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:00:18.866 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 149.05 --await-for 45.0' 2024-06-04T05:00:18.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 149.05 --await-for 45.0 2024-06-04T05:00:19.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:19 smithi070 ceph-mon[32631]: pgmap v97: 97 pgs: 97 active+clean; 36 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-06-04T05:00:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:19 smithi112 ceph-mon[36453]: pgmap v97: 97 pgs: 97 active+clean; 36 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-06-04T05:00:19.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 40115 3.24 MB/sec warmup 72 sec latency 15.382 ms 2024-06-04T05:00:19.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:19 smithi098 ceph-mon[35569]: pgmap v97: 97 pgs: 97 active+clean; 36 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-06-04T05:00:20.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.348+0000 7f9b264c3700 1 -- 172.21.15.70:0/56008012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b20102440 msgr2=0x7f9b201028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:20.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.348+0000 7f9b264c3700 1 --2- 172.21.15.70:0/56008012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b20102440 0x7f9b201028c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9b140098f0 tx=0x7f9b14009c00 comp rx=0 tx=0).stop 2024-06-04T05:00:20.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.348+0000 7f9b264c3700 1 -- 172.21.15.70:0/56008012 shutdown_connections 2024-06-04T05:00:20.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.348+0000 7f9b264c3700 1 --2- 172.21.15.70:0/56008012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b20102e00 0x7f9b2010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.348+0000 7f9b264c3700 1 --2- 172.21.15.70:0/56008012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b20108520 0x7f9b20108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.348+0000 7f9b264c3700 1 --2- 172.21.15.70:0/56008012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b20102440 0x7f9b201028c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.348+0000 7f9b264c3700 1 -- 172.21.15.70:0/56008012 >> 172.21.15.70:0/56008012 conn(0x7f9b200793f0 msgr2=0x7f9b20079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:20.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.348+0000 7f9b264c3700 1 -- 172.21.15.70:0/56008012 shutdown_connections 2024-06-04T05:00:20.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.348+0000 7f9b264c3700 1 -- 172.21.15.70:0/56008012 wait complete. 2024-06-04T05:00:20.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.349+0000 7f9b264c3700 1 Processor -- start 2024-06-04T05:00:20.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.349+0000 7f9b264c3700 1 -- start start 2024-06-04T05:00:20.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.349+0000 7f9b264c3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b20102440 0x7f9b20198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.349+0000 7f9b264c3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b20102e00 0x7f9b20198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.349+0000 7f9b264c3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b20108520 0x7f9b201992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.349+0000 7f9b264c3700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9b20112480 con 0x7f9b20108520 2024-06-04T05:00:20.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.349+0000 7f9b264c3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b20112300 con 0x7f9b20102440 2024-06-04T05:00:20.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.349+0000 7f9b264c3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9b20112600 con 0x7f9b20102e00 2024-06-04T05:00:20.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b1ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b20102440 0x7f9b20198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b1f7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b20102e00 0x7f9b20198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b24a60700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b20108520 0x7f9b201992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b1ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b20102440 0x7f9b20198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:51410/0 (socket says 172.21.15.70:51410) 2024-06-04T05:00:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b1f7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b20102e00 0x7f9b20198d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51930/0 (socket says 172.21.15.70:51930) 2024-06-04T05:00:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b1ffff700 1 -- 172.21.15.70:0/278945467 learned_addr learned my addr 172.21.15.70:0/278945467 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b24a60700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b20108520 0x7f9b201992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39904/0 (socket says 172.21.15.70:39904) 2024-06-04T05:00:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b1ffff700 1 -- 172.21.15.70:0/278945467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b20102e00 msgr2=0x7f9b20198d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b1ffff700 1 --2- 172.21.15.70:0/278945467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b20102e00 0x7f9b20198d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b1ffff700 1 -- 172.21.15.70:0/278945467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b20108520 msgr2=0x7f9b201992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b1ffff700 1 --2- 172.21.15.70:0/278945467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b20108520 0x7f9b201992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b1ffff700 1 -- 172.21.15.70:0/278945467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b201a8040 con 0x7f9b20102440 2024-06-04T05:00:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b1f7fe700 1 --2- 172.21.15.70:0/278945467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b20102e00 0x7f9b20198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b24a60700 1 --2- 172.21.15.70:0/278945467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b20108520 0x7f9b201992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b1ffff700 1 --2- 172.21.15.70:0/278945467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b20102440 0x7f9b20198830 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9b1000ec40 tx=0x7f9b1000ef50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b1d7fa700 1 -- 172.21.15.70:0/278945467 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b1000ccf0 con 0x7f9b20102440 2024-06-04T05:00:20.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b1d7fa700 1 -- 172.21.15.70:0/278945467 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b1000ce90 con 0x7f9b20102440 2024-06-04T05:00:20.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b264c3700 1 -- 172.21.15.70:0/278945467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b201a8330 con 0x7f9b20102440 2024-06-04T05:00:20.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.350+0000 7f9b1d7fa700 1 -- 172.21.15.70:0/278945467 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b10010430 con 0x7f9b20102440 2024-06-04T05:00:20.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.351+0000 7f9b264c3700 1 -- 172.21.15.70:0/278945467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b201a8840 con 0x7f9b20102440 2024-06-04T05:00:20.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.354+0000 7f9b1d7fa700 1 -- 172.21.15.70:0/278945467 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f9b10003ee0 con 0x7f9b20102440 2024-06-04T05:00:20.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.354+0000 7f9b30d58b80 1 Processor -- start 2024-06-04T05:00:20.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.354+0000 7f9b30d58b80 1 -- start start 2024-06-04T05:00:20.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.355+0000 7f9b30d58b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6c860 0x56473ea6cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.355+0000 7f9b30d58b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea6d210 0x56473ea70940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.355+0000 7f9b30d58b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea70f60 0x56473ea713a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.355+0000 7f9b30d58b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56473e9cbc90 con 0x56473ea6d210 2024-06-04T05:00:20.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.355+0000 7f9b30d58b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56473e9bb830 con 0x56473ea6c860 2024-06-04T05:00:20.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.355+0000 7f9b30d58b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56473e9bf870 con 0x56473ea70f60 2024-06-04T05:00:20.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.355+0000 7f9b24a60700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6c860 0x56473ea6cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.355+0000 7f9b24a60700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6c860 0x56473ea6cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:51416/0 (socket says 172.21.15.70:51416) 2024-06-04T05:00:20.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.355+0000 7f9b24a60700 1 -- 172.21.15.70:0/2097371525 learned_addr learned my addr 172.21.15.70:0/2097371525 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:20.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.355+0000 7f9b1f7fe700 1 --2- 172.21.15.70:0/2097371525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea70f60 0x56473ea713a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.355+0000 7f9b1ffff700 1 --2- 172.21.15.70:0/2097371525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea6d210 0x56473ea70940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.355+0000 7f9b1f7fe700 1 -- 172.21.15.70:0/2097371525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6c860 msgr2=0x56473ea6cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:20.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.355+0000 7f9b1f7fe700 1 --2- 172.21.15.70:0/2097371525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6c860 0x56473ea6cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.355+0000 7f9b1f7fe700 1 -- 172.21.15.70:0/2097371525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea6d210 msgr2=0x56473ea70940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:20.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.355+0000 7f9b1f7fe700 1 --2- 172.21.15.70:0/2097371525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea6d210 0x56473ea70940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.356+0000 7f9b1f7fe700 1 -- 172.21.15.70:0/2097371525 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56473e90b8d0 con 0x56473ea70f60 2024-06-04T05:00:20.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.356+0000 7f9b1ffff700 1 --2- 172.21.15.70:0/2097371525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea6d210 0x56473ea70940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:20.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.356+0000 7f9b1f7fe700 1 --2- 172.21.15.70:0/2097371525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea70f60 0x56473ea713a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9b14005e90 tx=0x7f9b14005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e468cbb7a6a5a0f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:20.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.356+0000 7f9b1d7fa700 1 --2- 172.21.15.70:0/278945467 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9b080b29f0 0x7f9b080b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.356+0000 7f9b067fc700 1 -- 172.21.15.70:0/2097371525 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b1400f660 con 0x56473ea70f60 2024-06-04T05:00:20.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.356+0000 7f9b1d7fa700 1 -- 172.21.15.70:0/278945467 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f9b100e2940 con 0x7f9b20102440 2024-06-04T05:00:20.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.356+0000 7f9b067fc700 1 -- 172.21.15.70:0/2097371525 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b1400f800 con 0x56473ea70f60 2024-06-04T05:00:20.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.356+0000 7f9b067fc700 1 -- 172.21.15.70:0/2097371525 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b14004ca0 con 0x56473ea70f60 2024-06-04T05:00:20.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.356+0000 7f9b24a60700 1 --2- 172.21.15.70:0/278945467 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9b080b29f0 0x7f9b080b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.357+0000 7f9b30d58b80 1 -- 172.21.15.70:0/2097371525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea70f60 msgr2=0x56473ea713a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:20.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.357+0000 7f9b24a60700 1 --2- 172.21.15.70:0/278945467 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9b080b29f0 0x7f9b080b4eb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9b18003e50 tx=0x7f9b18009100 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:20.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.357+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/2097371525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea70f60 0x56473ea713a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9b14005e90 tx=0x7f9b14005dc0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.357+0000 7f9b30d58b80 1 -- 172.21.15.70:0/2097371525 shutdown_connections 2024-06-04T05:00:20.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.357+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/2097371525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea70f60 0x56473ea713a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.357+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/2097371525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6c860 0x56473ea6cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.357+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/2097371525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea6d210 0x56473ea70940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.357+0000 7f9b30d58b80 1 -- 172.21.15.70:0/2097371525 >> 172.21.15.70:0/2097371525 conn(0x56473e99a080 msgr2=0x56473ea66e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:20.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.357+0000 7f9b30d58b80 1 -- 172.21.15.70:0/2097371525 shutdown_connections 2024-06-04T05:00:20.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.357+0000 7f9b30d58b80 1 -- 172.21.15.70:0/2097371525 wait complete. 2024-06-04T05:00:20.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.358+0000 7f9b30d58b80 1 -- start start 2024-06-04T05:00:20.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.358+0000 7f9b30d58b80 1 Processor -- start 2024-06-04T05:00:20.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b30d58b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6c860 0x56473ea783b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b30d58b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea6d210 0x56473ea756b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b30d58b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea70f60 0x56473ea75bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b30d58b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56473e8c1fa0 con 0x56473ea6d210 2024-06-04T05:00:20.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b30d58b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56473e95d7c0 con 0x56473ea6c860 2024-06-04T05:00:20.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b30d58b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56473ea790c0 con 0x56473ea70f60 2024-06-04T05:00:20.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b1f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea6d210 0x56473ea756b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b30d58b80 10 client.0 _ll_register_callbacks cb 0x56473e956920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:20.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b1f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea6d210 0x56473ea756b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39928/0 (socket says 172.21.15.70:39928) 2024-06-04T05:00:20.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b1f7fe700 1 -- 172.21.15.70:0/1333750422 learned_addr learned my addr 172.21.15.70:0/1333750422 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:20.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b1ffff700 1 --2- 172.21.15.70:0/1333750422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6c860 0x56473ea783b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b24a60700 1 --2- 172.21.15.70:0/1333750422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea70f60 0x56473ea75bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b1ffff700 1 -- 172.21.15.70:0/1333750422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea70f60 msgr2=0x56473ea75bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:20.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b1ffff700 1 --2- 172.21.15.70:0/1333750422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea70f60 0x56473ea75bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b1ffff700 1 -- 172.21.15.70:0/1333750422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea6d210 msgr2=0x56473ea756b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:20.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b1ffff700 1 --2- 172.21.15.70:0/1333750422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea6d210 0x56473ea756b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b1ffff700 1 -- 172.21.15.70:0/1333750422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56473e95a690 con 0x56473ea6c860 2024-06-04T05:00:20.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.359+0000 7f9b1f7fe700 1 --2- 172.21.15.70:0/1333750422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea6d210 0x56473ea756b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:20.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.360+0000 7f9aff7fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:00:20.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.360+0000 7f9b1ffff700 1 --2- 172.21.15.70:0/1333750422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6c860 0x56473ea783b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9b100e0090 tx=0x7f9b100a99f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a28cf0669bc4a50e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:20.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.360+0000 7f9aff7fe700 1 -- 172.21.15.70:0/1333750422 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b100a9730 con 0x56473ea6c860 2024-06-04T05:00:20.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.360+0000 7f9aff7fe700 1 -- 172.21.15.70:0/1333750422 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f9b10003680 con 0x56473ea6c860 2024-06-04T05:00:20.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.360+0000 7f9b30d58b80 1 -- 172.21.15.70:0/1333750422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b1400f800 con 0x56473ea6c860 2024-06-04T05:00:20.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.361+0000 7f9aff7fe700 1 -- 172.21.15.70:0/1333750422 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9b100aa320 con 0x56473ea6c860 2024-06-04T05:00:20.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.361+0000 7f9b30d58b80 10 client.25718 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:20.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.361+0000 7f9b30d58b80 1 -- 172.21.15.70:0/1333750422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56473ea837c0 con 0x56473ea6c860 2024-06-04T05:00:20.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.361+0000 7f9aff7fe700 1 -- 172.21.15.70:0/1333750422 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b1400f800 con 0x56473ea6c860 2024-06-04T05:00:20.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.361+0000 7f9aff7fe700 1 -- 172.21.15.70:0/1333750422 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f9b100c4020 con 0x56473ea6c860 2024-06-04T05:00:20.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.362+0000 7f9b30d58b80 10 client.25718 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:20.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.362+0000 7f9b30d58b80 10 client.25718 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:00:20.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.362+0000 7f9b30d58b80 20 client.25718 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:20.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.362+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/1333750422 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56473ea85d50 0x56473ea88140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.362+0000 7f9b30d58b80 4 client.25718 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:00:20.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.362+0000 7f9b30d58b80 1 -- 172.21.15.70:0/1333750422 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56473ea389e0 con 0x56473ea85d50 2024-06-04T05:00:20.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.362+0000 7f9b1f7fe700 1 --2- 172.21.15.70:0/1333750422 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56473ea85d50 0x56473ea88140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.362+0000 7f9b1f7fe700 1 --2- 172.21.15.70:0/1333750422 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56473ea85d50 0x56473ea88140 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=67911701aeb81d69 server_cookie=7a4025a029cdc78a in_seq=0 out_seq=0 2024-06-04T05:00:20.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.362+0000 7f9aff7fe700 10 client.25718 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:00:20.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.363+0000 7f9aff7fe700 1 -- 172.21.15.70:0/1333750422 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56473ea389e0 con 0x56473ea85d50 2024-06-04T05:00:20.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.363+0000 7f9aff7fe700 10 client.25718 handle_command_reply: tid=0 2024-06-04T05:00:20.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.363+0000 7f9b30d58b80 1 client.25718 shutdown 2024-06-04T05:00:20.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.364+0000 7f9b30d58b80 1 -- 172.21.15.70:0/1333750422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6c860 msgr2=0x56473ea783b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:20.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.364+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/1333750422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6c860 0x56473ea783b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9b100e0090 tx=0x7f9b100a99f0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.364+0000 7f9b30d58b80 1 -- 172.21.15.70:0/1333750422 shutdown_connections 2024-06-04T05:00:20.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.364+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/1333750422 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56473ea85d50 0x56473ea88140 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.364+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/1333750422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea70f60 0x56473ea75bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.364+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/1333750422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6c860 0x56473ea783b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.364+0000 7f9aff7fe700 0 client.25718 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:00:20.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.364+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/1333750422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea6d210 0x56473ea756b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.364+0000 7f9b30d58b80 1 -- 172.21.15.70:0/1333750422 >> 172.21.15.70:0/1333750422 conn(0x56473e99a080 msgr2=0x56473ea66510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:20.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.364+0000 7f9b30d58b80 1 -- 172.21.15.70:0/1333750422 shutdown_connections 2024-06-04T05:00:20.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.364+0000 7f9b30d58b80 1 -- 172.21.15.70:0/1333750422 wait complete. 2024-06-04T05:00:20.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.365+0000 7f9b30d58b80 20 client.25718 trim_cache size 0 max 16384 2024-06-04T05:00:20.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.386+0000 7f9b30d58b80 1 Processor -- start 2024-06-04T05:00:20.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.386+0000 7f9b30d58b80 1 -- start start 2024-06-04T05:00:20.388 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:00:20.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.386+0000 7f9b30d58b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea6c860 0x56473ea66c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.386+0000 7f9b30d58b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6d210 0x56473ea6e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.386+0000 7f9b30d58b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea70f60 0x56473ea6edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.386+0000 7f9b30d58b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9b201b0080 con 0x56473ea70f60 2024-06-04T05:00:20.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.386+0000 7f9b30d58b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b201b1c20 con 0x56473ea6d210 2024-06-04T05:00:20.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.386+0000 7f9b30d58b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9b201b0750 con 0x56473ea6c860 2024-06-04T05:00:20.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.386+0000 7f9b24a60700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea70f60 0x56473ea6edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.386+0000 7f9b1ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea6c860 0x56473ea66c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.386+0000 7f9b1ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea6c860 0x56473ea66c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51956/0 (socket says 172.21.15.70:51956) 2024-06-04T05:00:20.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.386+0000 7f9b24a60700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea70f60 0x56473ea6edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39940/0 (socket says 172.21.15.70:39940) 2024-06-04T05:00:20.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.386+0000 7f9b1ffff700 1 -- 172.21.15.70:0/19309539 learned_addr learned my addr 172.21.15.70:0/19309539 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:20.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.386+0000 7f9b1f7fe700 1 --2- 172.21.15.70:0/19309539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6d210 0x56473ea6e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.387+0000 7f9b1ffff700 1 -- 172.21.15.70:0/19309539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6d210 msgr2=0x56473ea6e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:20.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.387+0000 7f9b1ffff700 1 --2- 172.21.15.70:0/19309539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6d210 0x56473ea6e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.387+0000 7f9b1ffff700 1 -- 172.21.15.70:0/19309539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea70f60 msgr2=0x56473ea6edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:20.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.387+0000 7f9b1ffff700 1 --2- 172.21.15.70:0/19309539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea70f60 0x56473ea6edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.387+0000 7f9b1ffff700 1 -- 172.21.15.70:0/19309539 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56473ea887c0 con 0x56473ea6c860 2024-06-04T05:00:20.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.387+0000 7f9b1ffff700 1 --2- 172.21.15.70:0/19309539 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea6c860 0x56473ea66c90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9b100a6e90 tx=0x7f9b10016540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eeed0d355e4bf92e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:20.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.387+0000 7f9b05ffb700 1 -- 172.21.15.70:0/19309539 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b100a9730 con 0x56473ea6c860 2024-06-04T05:00:20.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.387+0000 7f9b05ffb700 1 -- 172.21.15.70:0/19309539 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b1006e020 con 0x56473ea6c860 2024-06-04T05:00:20.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.387+0000 7f9b05ffb700 1 -- 172.21.15.70:0/19309539 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b100a6210 con 0x56473ea6c860 2024-06-04T05:00:20.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.388+0000 7f9b30d58b80 1 -- 172.21.15.70:0/19309539 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea6c860 msgr2=0x56473ea66c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:20.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.388+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/19309539 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea6c860 0x56473ea66c90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9b100a6e90 tx=0x7f9b10016540 comp rx=0 tx=0).stop 2024-06-04T05:00:20.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.388+0000 7f9b30d58b80 1 -- 172.21.15.70:0/19309539 shutdown_connections 2024-06-04T05:00:20.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.388+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/19309539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea70f60 0x56473ea6edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.388+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/19309539 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea6c860 0x56473ea66c90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.388+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/19309539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6d210 0x56473ea6e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.388+0000 7f9b30d58b80 1 -- 172.21.15.70:0/19309539 >> 172.21.15.70:0/19309539 conn(0x56473e99a080 msgr2=0x56473ea77770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:20.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.388+0000 7f9b30d58b80 1 -- 172.21.15.70:0/19309539 shutdown_connections 2024-06-04T05:00:20.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.388+0000 7f9b30d58b80 1 -- 172.21.15.70:0/19309539 wait complete. 2024-06-04T05:00:20.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.389+0000 7f9b30d58b80 1 -- start start 2024-06-04T05:00:20.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.389+0000 7f9b30d58b80 1 Processor -- start 2024-06-04T05:00:20.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.389+0000 7f9b30d58b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea6c860 0x56473ea7e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.389+0000 7f9b30d58b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6d210 0x56473ea7eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.389+0000 7f9b30d58b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea70f60 0x56473ea79bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.389+0000 7f9b30d58b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9b201b14f0 con 0x56473ea70f60 2024-06-04T05:00:20.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.389+0000 7f9b30d58b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b201b0e20 con 0x56473ea6d210 2024-06-04T05:00:20.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.389+0000 7f9b30d58b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9b201ae020 con 0x56473ea6c860 2024-06-04T05:00:20.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.389+0000 7f9b30d58b80 10 client.0 _ll_register_callbacks cb 0x56473e956920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:20.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.389+0000 7f9b1ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea6c860 0x56473ea7e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.389+0000 7f9b1ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea6c860 0x56473ea7e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51972/0 (socket says 172.21.15.70:51972) 2024-06-04T05:00:20.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.389+0000 7f9b1ffff700 1 -- 172.21.15.70:0/4040446122 learned_addr learned my addr 172.21.15.70:0/4040446122 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:20.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.389+0000 7f9b1f7fe700 1 --2- 172.21.15.70:0/4040446122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6d210 0x56473ea7eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.389+0000 7f9b24a60700 1 --2- 172.21.15.70:0/4040446122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea70f60 0x56473ea79bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.390+0000 7f9b1ffff700 1 -- 172.21.15.70:0/4040446122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6d210 msgr2=0x56473ea7eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:20.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.390+0000 7f9b1ffff700 1 --2- 172.21.15.70:0/4040446122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6d210 0x56473ea7eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.390+0000 7f9b1ffff700 1 -- 172.21.15.70:0/4040446122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea70f60 msgr2=0x56473ea79bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:20.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.390+0000 7f9b1ffff700 1 --2- 172.21.15.70:0/4040446122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea70f60 0x56473ea79bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.390+0000 7f9b1ffff700 1 -- 172.21.15.70:0/4040446122 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56473ea7a410 con 0x56473ea6c860 2024-06-04T05:00:20.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.390+0000 7f9b1ffff700 1 --2- 172.21.15.70:0/4040446122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea6c860 0x56473ea7e650 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9b100164c0 tx=0x7f9b100b2820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ea3ff2de6fd0070 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:20.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.390+0000 7f9aff7fe700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:00:20.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.390+0000 7f9aff7fe700 1 -- 172.21.15.70:0/4040446122 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b100c2070 con 0x56473ea6c860 2024-06-04T05:00:20.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.390+0000 7f9aff7fe700 1 -- 172.21.15.70:0/4040446122 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f9b100a6210 con 0x56473ea6c860 2024-06-04T05:00:20.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.390+0000 7f9b30d58b80 1 -- 172.21.15.70:0/4040446122 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b1006e020 con 0x56473ea6c860 2024-06-04T05:00:20.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.390+0000 7f9aff7fe700 1 -- 172.21.15.70:0/4040446122 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9b100a9730 con 0x56473ea6c860 2024-06-04T05:00:20.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.390+0000 7f9b30d58b80 10 client.25786 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:20.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.390+0000 7f9b30d58b80 1 -- 172.21.15.70:0/4040446122 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56473eaf0540 con 0x56473ea6c860 2024-06-04T05:00:20.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.391+0000 7f9aff7fe700 1 -- 172.21.15.70:0/4040446122 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b100b2a80 con 0x56473ea6c860 2024-06-04T05:00:20.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.391+0000 7f9aff7fe700 1 -- 172.21.15.70:0/4040446122 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f9b100c4050 con 0x56473ea6c860 2024-06-04T05:00:20.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.391+0000 7f9b30d58b80 10 client.25786 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:20.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.391+0000 7f9b30d58b80 10 client.25786 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:00:20.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.391+0000 7f9b30d58b80 20 client.25786 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:20.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.391+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/4040446122 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56473eaf08c0 0x56473eaf2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:20.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.391+0000 7f9b30d58b80 4 client.25786 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:00:20.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.391+0000 7f9b30d58b80 1 -- 172.21.15.70:0/4040446122 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f9b1006e020 con 0x56473eaf08c0 2024-06-04T05:00:20.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.391+0000 7f9b1f7fe700 1 --2- 172.21.15.70:0/4040446122 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56473eaf08c0 0x56473eaf2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:20.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.391+0000 7f9aff7fe700 10 client.25786 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:00:20.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.391+0000 7f9b1f7fe700 1 --2- 172.21.15.70:0/4040446122 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56473eaf08c0 0x56473eaf2ce0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=89dfdd92691a5cd1 server_cookie=5d8df42e2efdc781 in_seq=0 out_seq=0 2024-06-04T05:00:20.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.392+0000 7f9aff7fe700 1 -- 172.21.15.70:0/4040446122 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f9b1006e020 con 0x56473eaf08c0 2024-06-04T05:00:20.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.392+0000 7f9aff7fe700 10 client.25786 handle_command_reply: tid=0 2024-06-04T05:00:20.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.392+0000 7f9b30d58b80 1 client.25786 shutdown 2024-06-04T05:00:20.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.392+0000 7f9b30d58b80 1 -- 172.21.15.70:0/4040446122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea6c860 msgr2=0x56473ea7e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:20.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.392+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/4040446122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea6c860 0x56473ea7e650 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9b100164c0 tx=0x7f9b100b2820 comp rx=0 tx=0).stop 2024-06-04T05:00:20.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.392+0000 7f9b30d58b80 1 -- 172.21.15.70:0/4040446122 shutdown_connections 2024-06-04T05:00:20.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.392+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/4040446122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56473ea70f60 0x56473ea79bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.392+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/4040446122 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56473eaf08c0 0x56473eaf2ce0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.392+0000 7f9aff7fe700 0 client.25786 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:00:20.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.392+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/4040446122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56473ea6c860 0x56473ea7e650 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9b100164c0 tx=0x7f9b100b2820 comp rx=0 tx=0).stop 2024-06-04T05:00:20.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.392+0000 7f9b30d58b80 1 --2- 172.21.15.70:0/4040446122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56473ea6d210 0x56473ea7eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.392+0000 7f9b30d58b80 1 -- 172.21.15.70:0/4040446122 >> 172.21.15.70:0/4040446122 conn(0x56473e99a080 msgr2=0x56473ea6f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:20.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.392+0000 7f9b30d58b80 1 -- 172.21.15.70:0/4040446122 shutdown_connections 2024-06-04T05:00:20.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.392+0000 7f9b30d58b80 1 -- 172.21.15.70:0/4040446122 wait complete. 2024-06-04T05:00:20.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.392+0000 7f9b30d58b80 20 client.25786 trim_cache size 0 max 16384 2024-06-04T05:00:20.399 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:00:20.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.393+0000 7f9b264c3700 1 -- 172.21.15.70:0/278945467 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9b080b29f0 msgr2=0x7f9b080b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:20.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.393+0000 7f9b264c3700 1 --2- 172.21.15.70:0/278945467 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9b080b29f0 0x7f9b080b4eb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9b18003e50 tx=0x7f9b18009100 comp rx=0 tx=0).stop 2024-06-04T05:00:20.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.393+0000 7f9b264c3700 1 -- 172.21.15.70:0/278945467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b20102440 msgr2=0x7f9b20198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:20.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.393+0000 7f9b264c3700 1 --2- 172.21.15.70:0/278945467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b20102440 0x7f9b20198830 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9b1000ec40 tx=0x7f9b1000ef50 comp rx=0 tx=0).stop 2024-06-04T05:00:20.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.394+0000 7f9b264c3700 1 -- 172.21.15.70:0/278945467 shutdown_connections 2024-06-04T05:00:20.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.394+0000 7f9b264c3700 1 --2- 172.21.15.70:0/278945467 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9b080b29f0 0x7f9b080b4eb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.394+0000 7f9b264c3700 1 --2- 172.21.15.70:0/278945467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b20108520 0x7f9b201992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.394+0000 7f9b264c3700 1 --2- 172.21.15.70:0/278945467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b20102440 0x7f9b20198830 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.394+0000 7f9b264c3700 1 --2- 172.21.15.70:0/278945467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b20102e00 0x7f9b20198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:20.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.394+0000 7f9b264c3700 1 -- 172.21.15.70:0/278945467 >> 172.21.15.70:0/278945467 conn(0x7f9b200793f0 msgr2=0x7f9b20100690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:20.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.394+0000 7f9b264c3700 1 -- 172.21.15.70:0/278945467 shutdown_connections 2024-06-04T05:00:20.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:20.394+0000 7f9b264c3700 1 -- 172.21.15.70:0/278945467 wait complete. 2024-06-04T05:00:20.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 40679 3.24 MB/sec warmup 73 sec latency 15.219 ms 2024-06-04T05:00:21.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:21 smithi070 ceph-mon[32631]: pgmap v98: 97 pgs: 97 active+clean; 36 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-06-04T05:00:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:21 smithi112 ceph-mon[36453]: pgmap v98: 97 pgs: 97 active+clean; 36 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-06-04T05:00:21.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 41393 3.22 MB/sec warmup 74 sec latency 15.685 ms 2024-06-04T05:00:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:21 smithi098 ceph-mon[35569]: pgmap v98: 97 pgs: 97 active+clean; 36 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-06-04T05:00:21.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:00:22.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 41985 3.25 MB/sec warmup 75 sec latency 15.300 ms 2024-06-04T05:00:23.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.035+0000 7fd1e29a9700 1 -- 172.21.15.70:0/1304774114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1dc06bb10 msgr2=0x7fd1dc06bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:23.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.035+0000 7fd1e29a9700 1 --2- 172.21.15.70:0/1304774114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1dc06bb10 0x7fd1dc06bf90 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40098f0 tx=0x7fd1c4009c00 comp rx=0 tx=0).stop 2024-06-04T05:00:23.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.035+0000 7fd1e29a9700 1 -- 172.21.15.70:0/1304774114 shutdown_connections 2024-06-04T05:00:23.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.035+0000 7fd1e29a9700 1 --2- 172.21.15.70:0/1304774114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1dc06c4d0 0x7fd1dc1103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.035+0000 7fd1e29a9700 1 --2- 172.21.15.70:0/1304774114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1dc1095d0 0x7fd1dc1099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.035+0000 7fd1e29a9700 1 --2- 172.21.15.70:0/1304774114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1dc06bb10 0x7fd1dc06bf90 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.035+0000 7fd1e29a9700 1 -- 172.21.15.70:0/1304774114 >> 172.21.15.70:0/1304774114 conn(0x7fd1dc079480 msgr2=0x7fd1dc079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:23.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.035+0000 7fd1e29a9700 1 -- 172.21.15.70:0/1304774114 shutdown_connections 2024-06-04T05:00:23.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.035+0000 7fd1e29a9700 1 -- 172.21.15.70:0/1304774114 wait complete. 2024-06-04T05:00:23.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.036+0000 7fd1e29a9700 1 Processor -- start 2024-06-04T05:00:23.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.036+0000 7fd1e29a9700 1 -- start start 2024-06-04T05:00:23.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.036+0000 7fd1e29a9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1dc06bb10 0x7fd1dc1a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.036+0000 7fd1e29a9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1dc06c4d0 0x7fd1dc1a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.036+0000 7fd1e29a9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1dc1095d0 0x7fd1dc19bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.036+0000 7fd1e29a9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc115860 con 0x7fd1dc1095d0 2024-06-04T05:00:23.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.036+0000 7fd1e29a9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc1156e0 con 0x7fd1dc06bb10 2024-06-04T05:00:23.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.036+0000 7fd1e29a9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc1159e0 con 0x7fd1dc06c4d0 2024-06-04T05:00:23.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.037+0000 7fd1d37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1dc06c4d0 0x7fd1dc1a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:23.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.037+0000 7fd1e0f46700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1dc1095d0 0x7fd1dc19bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:23.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.037+0000 7fd1d37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1dc06c4d0 0x7fd1dc1a1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51982/0 (socket says 172.21.15.70:51982) 2024-06-04T05:00:23.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.037+0000 7fd1d37fe700 1 -- 172.21.15.70:0/2624728619 learned_addr learned my addr 172.21.15.70:0/2624728619 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:23.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.037+0000 7fd1dbfff700 1 --2- 172.21.15.70:0/2624728619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1dc06bb10 0x7fd1dc1a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:23.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.037+0000 7fd1d37fe700 1 -- 172.21.15.70:0/2624728619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1dc06bb10 msgr2=0x7fd1dc1a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:23.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.037+0000 7fd1d37fe700 1 --2- 172.21.15.70:0/2624728619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1dc06bb10 0x7fd1dc1a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.037+0000 7fd1d37fe700 1 -- 172.21.15.70:0/2624728619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1dc1095d0 msgr2=0x7fd1dc19bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:23.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.037+0000 7fd1d37fe700 1 --2- 172.21.15.70:0/2624728619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1dc1095d0 0x7fd1dc19bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.037+0000 7fd1d37fe700 1 -- 172.21.15.70:0/2624728619 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1dc19c260 con 0x7fd1dc06c4d0 2024-06-04T05:00:23.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.037+0000 7fd1d37fe700 1 --2- 172.21.15.70:0/2624728619 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1dc06c4d0 0x7fd1dc1a1e80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400fa00 tx=0x7fd1c400fa30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:23.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.038+0000 7fd1d9ffb700 1 -- 172.21.15.70:0/2624728619 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4027070 con 0x7fd1dc06c4d0 2024-06-04T05:00:23.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.038+0000 7fd1d9ffb700 1 -- 172.21.15.70:0/2624728619 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd1c400b9f0 con 0x7fd1dc06c4d0 2024-06-04T05:00:23.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.038+0000 7fd1e29a9700 1 -- 172.21.15.70:0/2624728619 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1dc19c4f0 con 0x7fd1dc06c4d0 2024-06-04T05:00:23.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.038+0000 7fd1d9ffb700 1 -- 172.21.15.70:0/2624728619 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c40046d0 con 0x7fd1dc06c4d0 2024-06-04T05:00:23.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.038+0000 7fd1e29a9700 1 -- 172.21.15.70:0/2624728619 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1dc1a8810 con 0x7fd1dc06c4d0 2024-06-04T05:00:23.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.041+0000 7fd1d9ffb700 1 -- 172.21.15.70:0/2624728619 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fd1c402a070 con 0x7fd1dc06c4d0 2024-06-04T05:00:23.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.042+0000 7fd1d9ffb700 1 --2- 172.21.15.70:0/2624728619 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd1bc0b2ba0 0x7fd1bc0b5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.042+0000 7fd1ed23eb80 1 Processor -- start 2024-06-04T05:00:23.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.042+0000 7fd1d9ffb700 1 -- 172.21.15.70:0/2624728619 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fd1c40e2f40 con 0x7fd1dc06c4d0 2024-06-04T05:00:23.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.042+0000 7fd1dbfff700 1 --2- 172.21.15.70:0/2624728619 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd1bc0b2ba0 0x7fd1bc0b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:23.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.042+0000 7fd1ed23eb80 1 -- start start 2024-06-04T05:00:23.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.042+0000 7fd1dbfff700 1 --2- 172.21.15.70:0/2624728619 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd1bc0b2ba0 0x7fd1bc0b5060 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc009750 tx=0x7fd1cc009310 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:23.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.043+0000 7fd1ed23eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d947132fd0 0x55d9471333b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.043+0000 7fd1ed23eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712e4a0 0x55d94712e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.043+0000 7fd1ed23eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d94712ee60 0x55d94713aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.043+0000 7fd1ed23eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d947083f10 con 0x55d94712e4a0 2024-06-04T05:00:23.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.043+0000 7fd1ed23eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d947073ab0 con 0x55d947132fd0 2024-06-04T05:00:23.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.043+0000 7fd1ed23eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d947077af0 con 0x55d94712ee60 2024-06-04T05:00:23.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.043+0000 7fd1d37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712e4a0 0x55d94712e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:23.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.043+0000 7fd1d37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712e4a0 0x55d94712e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39984/0 (socket says 172.21.15.70:39984) 2024-06-04T05:00:23.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.043+0000 7fd1d37fe700 1 -- 172.21.15.70:0/3724224204 learned_addr learned my addr 172.21.15.70:0/3724224204 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:23.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.043+0000 7fd1dbfff700 1 --2- 172.21.15.70:0/3724224204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d947132fd0 0x55d9471333b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:23.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.043+0000 7fd1dbfff700 1 -- 172.21.15.70:0/3724224204 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d94712ee60 msgr2=0x55d94713aed0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:00:23.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.043+0000 7fd1dbfff700 1 --2- 172.21.15.70:0/3724224204 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d94712ee60 0x55d94713aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.043+0000 7fd1dbfff700 1 -- 172.21.15.70:0/3724224204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712e4a0 msgr2=0x55d94712e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:23.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.043+0000 7fd1dbfff700 1 --2- 172.21.15.70:0/3724224204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712e4a0 0x55d94712e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.043+0000 7fd1dbfff700 1 -- 172.21.15.70:0/3724224204 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d94701d150 con 0x55d947132fd0 2024-06-04T05:00:23.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.043+0000 7fd1d37fe700 1 --2- 172.21.15.70:0/3724224204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712e4a0 0x55d94712e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:23.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.044+0000 7fd1dbfff700 1 --2- 172.21.15.70:0/3724224204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d947132fd0 0x55d9471333b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc005fd0 tx=0x7fd1cc014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db4f419d73800179 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:23.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.044+0000 7fd1d27fc700 1 -- 172.21.15.70:0/3724224204 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc00ee20 con 0x55d947132fd0 2024-06-04T05:00:23.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.044+0000 7fd1d27fc700 1 -- 172.21.15.70:0/3724224204 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd1cc01c8a0 con 0x55d947132fd0 2024-06-04T05:00:23.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.044+0000 7fd1d27fc700 1 -- 172.21.15.70:0/3724224204 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc01d8d0 con 0x55d947132fd0 2024-06-04T05:00:23.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.045+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/3724224204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d947132fd0 msgr2=0x55d9471333b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:23.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.045+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/3724224204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d947132fd0 0x55d9471333b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc005fd0 tx=0x7fd1cc014000 comp rx=0 tx=0).stop 2024-06-04T05:00:23.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.045+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/3724224204 shutdown_connections 2024-06-04T05:00:23.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.045+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/3724224204 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d94712ee60 0x55d94713aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.045+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/3724224204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d947132fd0 0x55d9471333b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.045+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/3724224204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712e4a0 0x55d94712e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.045+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/3724224204 >> 172.21.15.70:0/3724224204 conn(0x55d9470513e0 msgr2=0x55d947120980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.045+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/3724224204 shutdown_connections 2024-06-04T05:00:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.045+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/3724224204 wait complete. 2024-06-04T05:00:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.046+0000 7fd1ed23eb80 1 -- start start 2024-06-04T05:00:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.046+0000 7fd1ed23eb80 1 Processor -- start 2024-06-04T05:00:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.046+0000 7fd1ed23eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712e4a0 0x55d94713abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.046+0000 7fd1ed23eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d94712ee60 0x55d947137ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.046+0000 7fd1ed23eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d947132fd0 0x55d947138430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.046+0000 7fd1ed23eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d946f7a140 con 0x55d94712e4a0 2024-06-04T05:00:23.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.046+0000 7fd1ed23eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d946fdbf40 con 0x55d94712ee60 2024-06-04T05:00:23.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.046+0000 7fd1ed23eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d94713b580 con 0x55d947132fd0 2024-06-04T05:00:23.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.046+0000 7fd1ed23eb80 10 client.0 _ll_register_callbacks cb 0x55d94700eaf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:23.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.046+0000 7fd1e0f46700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d947132fd0 0x55d947138430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:23.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.047+0000 7fd1e0f46700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d947132fd0 0x55d947138430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51996/0 (socket says 172.21.15.70:51996) 2024-06-04T05:00:23.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.047+0000 7fd1e0f46700 1 -- 172.21.15.70:0/549500965 learned_addr learned my addr 172.21.15.70:0/549500965 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:23.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.047+0000 7fd1dbfff700 1 --2- 172.21.15.70:0/549500965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712e4a0 0x55d94713abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:23.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.047+0000 7fd1d37fe700 1 --2- 172.21.15.70:0/549500965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d94712ee60 0x55d947137ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:23.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.047+0000 7fd1e0f46700 1 -- 172.21.15.70:0/549500965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d94712ee60 msgr2=0x55d947137ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:23.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.047+0000 7fd1e0f46700 1 --2- 172.21.15.70:0/549500965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d94712ee60 0x55d947137ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.047+0000 7fd1e0f46700 1 -- 172.21.15.70:0/549500965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712e4a0 msgr2=0x55d94713abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:23.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.047+0000 7fd1e0f46700 1 --2- 172.21.15.70:0/549500965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712e4a0 0x55d94713abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.047+0000 7fd1e0f46700 1 -- 172.21.15.70:0/549500965 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d946fc3a70 con 0x55d947132fd0 2024-06-04T05:00:23.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.047+0000 7fd1e0f46700 1 --2- 172.21.15.70:0/549500965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d947132fd0 0x55d947138430 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4011d80 tx=0x7fd1d400f490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2860b6632984863 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:23.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.047+0000 7fd1cb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:00:23.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.047+0000 7fd1cb7fe700 1 -- 172.21.15.70:0/549500965 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4019030 con 0x55d947132fd0 2024-06-04T05:00:23.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.047+0000 7fd1cb7fe700 1 -- 172.21.15.70:0/549500965 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fd1d40092e0 con 0x55d947132fd0 2024-06-04T05:00:23.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.047+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/549500965 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1cc01c8a0 con 0x55d947132fd0 2024-06-04T05:00:23.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.047+0000 7fd1cb7fe700 1 -- 172.21.15.70:0/549500965 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd1d4023050 con 0x55d947132fd0 2024-06-04T05:00:23.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.047+0000 7fd1ed23eb80 10 client.25795 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:23.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.048+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/549500965 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d94702d900 con 0x55d947132fd0 2024-06-04T05:00:23.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.048+0000 7fd1cb7fe700 1 -- 172.21.15.70:0/549500965 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd1d400fa20 con 0x55d947132fd0 2024-06-04T05:00:23.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.048+0000 7fd1cb7fe700 1 -- 172.21.15.70:0/549500965 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd1d40041b0 con 0x55d947132fd0 2024-06-04T05:00:23.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.048+0000 7fd1ed23eb80 10 client.25795 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:23.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.048+0000 7fd1ed23eb80 10 client.25795 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:00:23.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.048+0000 7fd1ed23eb80 20 client.25795 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:23.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.048+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/549500965 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d947148430 0x55d94714a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.048+0000 7fd1ed23eb80 4 client.25795 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:00:23.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.048+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/549500965 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd1cc01c8a0 con 0x55d947148430 2024-06-04T05:00:23.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.048+0000 7fd1dbfff700 1 --2- 172.21.15.70:0/549500965 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d947148430 0x55d94714a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:23.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.049+0000 7fd1dbfff700 1 --2- 172.21.15.70:0/549500965 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d947148430 0x55d94714a850 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=843002bb4748ce30 server_cookie=5fa66a90cea469a0 in_seq=0 out_seq=0 2024-06-04T05:00:23.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.049+0000 7fd1cb7fe700 10 client.25795 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:23.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.049+0000 7fd1cb7fe700 1 -- 172.21.15.70:0/549500965 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fd1cc01c8a0 con 0x55d947148430 2024-06-04T05:00:23.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.049+0000 7fd1cb7fe700 10 client.25795 handle_command_reply: tid=0 2024-06-04T05:00:23.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.049+0000 7fd1ed23eb80 1 client.25795 shutdown 2024-06-04T05:00:23.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.050+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/549500965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d947132fd0 msgr2=0x55d947138430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:23.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.050+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/549500965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d947132fd0 0x55d947138430 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4011d80 tx=0x7fd1d400f490 comp rx=0 tx=0).stop 2024-06-04T05:00:23.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.050+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/549500965 shutdown_connections 2024-06-04T05:00:23.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.050+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/549500965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d947132fd0 0x55d947138430 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.050+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/549500965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712e4a0 0x55d94713abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.050+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/549500965 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d947148430 0x55d94714a850 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.050+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/549500965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d94712ee60 0x55d947137ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.050+0000 7fd1cb7fe700 0 client.25795 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:23.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.050+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/549500965 >> 172.21.15.70:0/549500965 conn(0x55d9470513e0 msgr2=0x55d94712d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:23.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.050+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/549500965 shutdown_connections 2024-06-04T05:00:23.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.050+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/549500965 wait complete. 2024-06-04T05:00:23.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.050+0000 7fd1ed23eb80 20 client.25795 trim_cache size 0 max 16384 2024-06-04T05:00:23.060 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:00:23.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.073+0000 7fd1ed23eb80 1 Processor -- start 2024-06-04T05:00:23.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.073+0000 7fd1ed23eb80 1 -- start start 2024-06-04T05:00:23.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.073+0000 7fd1ed23eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d94712e4a0 0x55d94711af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.073+0000 7fd1ed23eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712ee60 0x55d947138520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.073+0000 7fd1ed23eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d947132fd0 0x55d94712dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1ed23eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc1abaa0 con 0x55d94712ee60 2024-06-04T05:00:23.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1ed23eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc1ad640 con 0x55d94712e4a0 2024-06-04T05:00:23.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1ed23eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc1ac170 con 0x55d947132fd0 2024-06-04T05:00:23.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1d37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712ee60 0x55d947138520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:23.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1e0f46700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d947132fd0 0x55d94712dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:23.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1e0f46700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d947132fd0 0x55d94712dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:52002/0 (socket says 172.21.15.70:52002) 2024-06-04T05:00:23.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1d37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712ee60 0x55d947138520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40012/0 (socket says 172.21.15.70:40012) 2024-06-04T05:00:23.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1d37fe700 1 -- 172.21.15.70:0/2419853660 learned_addr learned my addr 172.21.15.70:0/2419853660 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:23.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1dbfff700 1 --2- 172.21.15.70:0/2419853660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d94712e4a0 0x55d94711af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:23.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1dbfff700 1 -- 172.21.15.70:0/2419853660 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d947132fd0 msgr2=0x55d94712dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:23.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1dbfff700 1 --2- 172.21.15.70:0/2419853660 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d947132fd0 0x55d94712dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1dbfff700 1 -- 172.21.15.70:0/2419853660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712ee60 msgr2=0x55d947138520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:23.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1dbfff700 1 --2- 172.21.15.70:0/2419853660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712ee60 0x55d947138520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1dbfff700 1 -- 172.21.15.70:0/2419853660 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d947136ec0 con 0x55d94712e4a0 2024-06-04T05:00:23.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1d37fe700 1 --2- 172.21.15.70:0/2419853660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712ee60 0x55d947138520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:23.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1dbfff700 1 --2- 172.21.15.70:0/2419853660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d94712e4a0 0x55d94711af10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x55d94714b0c0 tx=0x7fd1cc009380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b40f29b3ced2810 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:23.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1d1ffb700 1 -- 172.21.15.70:0/2419853660 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc00ec20 con 0x55d94712e4a0 2024-06-04T05:00:23.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1d1ffb700 1 -- 172.21.15.70:0/2419853660 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd1cc00edc0 con 0x55d94712e4a0 2024-06-04T05:00:23.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.074+0000 7fd1d1ffb700 1 -- 172.21.15.70:0/2419853660 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc003bf0 con 0x55d94712e4a0 2024-06-04T05:00:23.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.075+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/2419853660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d94712e4a0 msgr2=0x55d94711af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:23.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.075+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/2419853660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d94712e4a0 0x55d94711af10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x55d94714b0c0 tx=0x7fd1cc009380 comp rx=0 tx=0).stop 2024-06-04T05:00:23.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.075+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/2419853660 shutdown_connections 2024-06-04T05:00:23.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.075+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/2419853660 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d947132fd0 0x55d94712dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.075+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/2419853660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d94712e4a0 0x55d94711af10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.075+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/2419853660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712ee60 0x55d947138520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.075+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/2419853660 >> 172.21.15.70:0/2419853660 conn(0x55d9470513e0 msgr2=0x55d94712d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:23.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.075+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/2419853660 shutdown_connections 2024-06-04T05:00:23.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.075+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/2419853660 wait complete. 2024-06-04T05:00:23.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.076+0000 7fd1ed23eb80 1 -- start start 2024-06-04T05:00:23.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.076+0000 7fd1ed23eb80 1 Processor -- start 2024-06-04T05:00:23.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.076+0000 7fd1ed23eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d94712e4a0 0x55d94714fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.076+0000 7fd1ed23eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712ee60 0x55d9471503b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.076+0000 7fd1ed23eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d947132fd0 0x55d9471508f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.076+0000 7fd1ed23eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc1acf10 con 0x55d94712ee60 2024-06-04T05:00:23.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.076+0000 7fd1ed23eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc1ac840 con 0x55d947132fd0 2024-06-04T05:00:23.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.076+0000 7fd1ed23eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc114e90 con 0x55d94712e4a0 2024-06-04T05:00:23.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.076+0000 7fd1dbfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d94712e4a0 0x55d94714fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:23.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.076+0000 7fd1ed23eb80 10 client.0 _ll_register_callbacks cb 0x55d94700eaf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:23.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.076+0000 7fd1d37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712ee60 0x55d9471503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:23.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.076+0000 7fd1dbfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d94712e4a0 0x55d94714fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:52006/0 (socket says 172.21.15.70:52006) 2024-06-04T05:00:23.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.076+0000 7fd1dbfff700 1 -- 172.21.15.70:0/1115589863 learned_addr learned my addr 172.21.15.70:0/1115589863 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:23.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.076+0000 7fd1e0f46700 1 --2- 172.21.15.70:0/1115589863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d947132fd0 0x55d9471508f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:23.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.077+0000 7fd1e0f46700 1 -- 172.21.15.70:0/1115589863 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d94712e4a0 msgr2=0x55d94714fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:23.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.077+0000 7fd1e0f46700 1 --2- 172.21.15.70:0/1115589863 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d94712e4a0 0x55d94714fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.077+0000 7fd1e0f46700 1 -- 172.21.15.70:0/1115589863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712ee60 msgr2=0x55d9471503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:23.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.077+0000 7fd1e0f46700 1 --2- 172.21.15.70:0/1115589863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712ee60 0x55d9471503b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.077+0000 7fd1e0f46700 1 -- 172.21.15.70:0/1115589863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9471b21e0 con 0x55d947132fd0 2024-06-04T05:00:23.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.077+0000 7fd1e0f46700 1 --2- 172.21.15.70:0/1115589863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d947132fd0 0x55d9471508f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd1d40045f0 tx=0x7fd1d40092e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0b0a7c298ec5bdd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:23.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.077+0000 7fd1cb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:00:23.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.077+0000 7fd1cb7fe700 1 -- 172.21.15.70:0/1115589863 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4019040 con 0x55d947132fd0 2024-06-04T05:00:23.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.077+0000 7fd1cb7fe700 1 -- 172.21.15.70:0/1115589863 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fd1d400fa20 con 0x55d947132fd0 2024-06-04T05:00:23.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.077+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/1115589863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1cc00edc0 con 0x55d947132fd0 2024-06-04T05:00:23.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.077+0000 7fd1cb7fe700 1 -- 172.21.15.70:0/1115589863 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd1d4035050 con 0x55d947132fd0 2024-06-04T05:00:23.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.077+0000 7fd1ed23eb80 10 client.25742 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:23.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.077+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/1115589863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d9471b3480 con 0x55d947132fd0 2024-06-04T05:00:23.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.078+0000 7fd1cb7fe700 1 -- 172.21.15.70:0/1115589863 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd1d4004c70 con 0x55d947132fd0 2024-06-04T05:00:23.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.078+0000 7fd1cb7fe700 1 -- 172.21.15.70:0/1115589863 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd1d4007880 con 0x55d947132fd0 2024-06-04T05:00:23.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.078+0000 7fd1ed23eb80 10 client.25742 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:23.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.078+0000 7fd1ed23eb80 10 client.25742 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:00:23.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.078+0000 7fd1ed23eb80 20 client.25742 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:23.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.078+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/1115589863 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d9471b3c60 0x55d9471b6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:23.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.078+0000 7fd1ed23eb80 4 client.25742 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.05, "await_for": 45.0}] 2024-06-04T05:00:23.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.078+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/1115589863 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.05, "await_for": 45.0}) -- 0x7fd1cc00edc0 con 0x55d9471b3c60 2024-06-04T05:00:23.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.078+0000 7fd1dbfff700 1 --2- 172.21.15.70:0/1115589863 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d9471b3c60 0x55d9471b6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:23.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.079+0000 7fd1cb7fe700 10 client.25742 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:23.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.079+0000 7fd1dbfff700 1 --2- 172.21.15.70:0/1115589863 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d9471b3c60 0x55d9471b6080 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cb0972328654708 server_cookie=7b062b9c381b772d in_seq=0 out_seq=0 2024-06-04T05:00:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.175+0000 7fd1cb7fe700 1 -- 172.21.15.70:0/1115589863 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x7fd1cc00edc0 con 0x55d9471b3c60 2024-06-04T05:00:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.175+0000 7fd1cb7fe700 10 client.25742 handle_command_reply: tid=0 2024-06-04T05:00:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.175+0000 7fd1ed23eb80 1 client.25742 shutdown 2024-06-04T05:00:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.175+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/1115589863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d947132fd0 msgr2=0x55d9471508f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.175+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/1115589863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d947132fd0 0x55d9471508f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd1d40045f0 tx=0x7fd1d40092e0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.175+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/1115589863 shutdown_connections 2024-06-04T05:00:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.175+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/1115589863 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d9471b3c60 0x55d9471b6080 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.175+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/1115589863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d947132fd0 0x55d9471508f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.175+0000 7fd1cb7fe700 0 client.25742 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.175+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/1115589863 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d94712e4a0 0x55d94714fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.175+0000 7fd1ed23eb80 1 --2- 172.21.15.70:0/1115589863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d94712ee60 0x55d9471503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.175+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/1115589863 >> 172.21.15.70:0/1115589863 conn(0x55d9470513e0 msgr2=0x55d947133c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.175+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/1115589863 shutdown_connections 2024-06-04T05:00:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.175+0000 7fd1ed23eb80 1 -- 172.21.15.70:0/1115589863 wait complete. 2024-06-04T05:00:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.175+0000 7fd1ed23eb80 20 client.25742 trim_cache size 0 max 16384 2024-06-04T05:00:23.180 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:00:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 31, 2024-06-04T05:00:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:00:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 6, 2024-06-04T05:00:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:00:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "ae3a3592": { 2024-06-04T05:00:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 6, 2024-06-04T05:00:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:00:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:00:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:00:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:00:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:00:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:00:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 149.1, 2024-06-04T05:00:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:00:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:00:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:00:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:00:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:00:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:00:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:00:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:00:23.182 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:00:23.182 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:00:23.182 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:00:23.182 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:00:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.176+0000 7fd1e29a9700 1 -- 172.21.15.70:0/2624728619 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd1bc0b2ba0 msgr2=0x7fd1bc0b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.176+0000 7fd1e29a9700 1 --2- 172.21.15.70:0/2624728619 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd1bc0b2ba0 0x7fd1bc0b5060 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc009750 tx=0x7fd1cc009310 comp rx=0 tx=0).stop 2024-06-04T05:00:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.176+0000 7fd1e29a9700 1 -- 172.21.15.70:0/2624728619 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1dc06c4d0 msgr2=0x7fd1dc1a1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.176+0000 7fd1e29a9700 1 --2- 172.21.15.70:0/2624728619 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1dc06c4d0 0x7fd1dc1a1e80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400fa00 tx=0x7fd1c400fa30 comp rx=0 tx=0).stop 2024-06-04T05:00:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.176+0000 7fd1e29a9700 1 -- 172.21.15.70:0/2624728619 shutdown_connections 2024-06-04T05:00:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.176+0000 7fd1e29a9700 1 --2- 172.21.15.70:0/2624728619 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd1bc0b2ba0 0x7fd1bc0b5060 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.176+0000 7fd1e29a9700 1 --2- 172.21.15.70:0/2624728619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1dc1095d0 0x7fd1dc19bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.176+0000 7fd1e29a9700 1 --2- 172.21.15.70:0/2624728619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1dc06bb10 0x7fd1dc1a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.176+0000 7fd1e29a9700 1 --2- 172.21.15.70:0/2624728619 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1dc06c4d0 0x7fd1dc1a1e80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.176+0000 7fd1e29a9700 1 -- 172.21.15.70:0/2624728619 >> 172.21.15.70:0/2624728619 conn(0x7fd1dc079480 msgr2=0x7fd1dc101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.176+0000 7fd1e29a9700 1 -- 172.21.15.70:0/2624728619 shutdown_connections 2024-06-04T05:00:23.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:23.176+0000 7fd1e29a9700 1 -- 172.21.15.70:0/2624728619 wait complete. 2024-06-04T05:00:23.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:23 smithi070 ceph-mon[32631]: pgmap v99: 97 pgs: 97 active+clean; 36 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 1.7 MiB/s wr, 199 op/s 2024-06-04T05:00:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:23 smithi112 ceph-mon[36453]: pgmap v99: 97 pgs: 97 active+clean; 36 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 1.7 MiB/s wr, 199 op/s 2024-06-04T05:00:23.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 3.22 MB/sec warmup 76 sec latency 626.675 ms 2024-06-04T05:00:23.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:23 smithi098 ceph-mon[35569]: pgmap v99: 97 pgs: 97 active+clean; 36 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 1.7 MiB/s wr, 199 op/s 2024-06-04T05:00:24.203 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ae3a3592', quiesce took 10.4 seconds. Will release after: 18.65 2024-06-04T05:00:24.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 3.18 MB/sec warmup 77 sec latency 1626.711 ms 2024-06-04T05:00:25.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:25 smithi070 ceph-mon[32631]: pgmap v100: 97 pgs: 97 active+clean; 36 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 1.1 MiB/s wr, 136 op/s 2024-06-04T05:00:25.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:25 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:00:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:25 smithi112 ceph-mon[36453]: pgmap v100: 97 pgs: 97 active+clean; 36 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 1.1 MiB/s wr, 136 op/s 2024-06-04T05:00:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:25 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:00:25.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 3.14 MB/sec warmup 78 sec latency 2626.733 ms 2024-06-04T05:00:25.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:25 smithi098 ceph-mon[35569]: pgmap v100: 97 pgs: 97 active+clean; 36 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 1.1 MiB/s wr, 136 op/s 2024-06-04T05:00:25.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:25 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:00:25.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.932+0000 7fca8ae26700 1 -- 172.21.15.70:0/884744642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca84102e00 msgr2=0x7fca8410f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:25.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.932+0000 7fca8ae26700 1 --2- 172.21.15.70:0/884744642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca84102e00 0x7fca8410f2f0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00b0a0 tx=0x7fca7c00b3b0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.932+0000 7fca8ae26700 1 -- 172.21.15.70:0/884744642 shutdown_connections 2024-06-04T05:00:25.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.932+0000 7fca8ae26700 1 --2- 172.21.15.70:0/884744642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca84102e00 0x7fca8410f2f0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.932+0000 7fca8ae26700 1 --2- 172.21.15.70:0/884744642 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca84108520 0x7fca84108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.932+0000 7fca8ae26700 1 --2- 172.21.15.70:0/884744642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca84102440 0x7fca841028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.932+0000 7fca8ae26700 1 -- 172.21.15.70:0/884744642 >> 172.21.15.70:0/884744642 conn(0x7fca840793f0 msgr2=0x7fca84079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:25.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.932+0000 7fca8ae26700 1 -- 172.21.15.70:0/884744642 shutdown_connections 2024-06-04T05:00:25.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.932+0000 7fca8ae26700 1 -- 172.21.15.70:0/884744642 wait complete. 2024-06-04T05:00:25.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.933+0000 7fca8ae26700 1 Processor -- start 2024-06-04T05:00:25.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.933+0000 7fca8ae26700 1 -- start start 2024-06-04T05:00:25.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.933+0000 7fca8ae26700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca84102440 0x7fca8419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.933+0000 7fca8ae26700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca84102e00 0x7fca8419ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.933+0000 7fca8ae26700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca84108520 0x7fca841a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca8ae26700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fca841124a0 con 0x7fca84102e00 2024-06-04T05:00:25.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca8ae26700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fca84112320 con 0x7fca84102440 2024-06-04T05:00:25.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca8ae26700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fca84112620 con 0x7fca84108520 2024-06-04T05:00:25.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca83fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca84102e00 0x7fca8419ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca83fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca84102e00 0x7fca8419ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40046/0 (socket says 172.21.15.70:40046) 2024-06-04T05:00:25.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca83fff700 1 -- 172.21.15.70:0/3019117824 learned_addr learned my addr 172.21.15.70:0/3019117824 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:25.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca88bc2700 1 --2- 172.21.15.70:0/3019117824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca84102440 0x7fca8419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca893c3700 1 --2- 172.21.15.70:0/3019117824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca84108520 0x7fca841a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca83fff700 1 -- 172.21.15.70:0/3019117824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca84108520 msgr2=0x7fca841a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:25.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca83fff700 1 --2- 172.21.15.70:0/3019117824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca84108520 0x7fca841a5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca83fff700 1 -- 172.21.15.70:0/3019117824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca84102440 msgr2=0x7fca8419a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:25.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca83fff700 1 --2- 172.21.15.70:0/3019117824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca84102440 0x7fca8419a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca83fff700 1 -- 172.21.15.70:0/3019117824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca841a5ec0 con 0x7fca84102e00 2024-06-04T05:00:25.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca83fff700 1 --2- 172.21.15.70:0/3019117824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca84102e00 0x7fca8419ae40 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fca7000d950 tx=0x7fca7000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:25.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca81ffb700 1 -- 172.21.15.70:0/3019117824 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca70016070 con 0x7fca84102e00 2024-06-04T05:00:25.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca81ffb700 1 -- 172.21.15.70:0/3019117824 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca7000fd20 con 0x7fca84102e00 2024-06-04T05:00:25.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca81ffb700 1 -- 172.21.15.70:0/3019117824 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7000e3b0 con 0x7fca84102e00 2024-06-04T05:00:25.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.934+0000 7fca8ae26700 1 -- 172.21.15.70:0/3019117824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca841a60f0 con 0x7fca84102e00 2024-06-04T05:00:25.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.935+0000 7fca8ae26700 1 -- 172.21.15.70:0/3019117824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca841a6650 con 0x7fca84102e00 2024-06-04T05:00:25.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.937+0000 7fca81ffb700 1 -- 172.21.15.70:0/3019117824 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fca70019070 con 0x7fca84102e00 2024-06-04T05:00:25.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.937+0000 7fca81ffb700 1 --2- 172.21.15.70:0/3019117824 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fca740b2a70 0x7fca740b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.938+0000 7fca88bc2700 1 --2- 172.21.15.70:0/3019117824 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fca740b2a70 0x7fca740b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.938+0000 7fca81ffb700 1 -- 172.21.15.70:0/3019117824 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fca700e2080 con 0x7fca84102e00 2024-06-04T05:00:25.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.938+0000 7fca88bc2700 1 --2- 172.21.15.70:0/3019117824 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fca740b2a70 0x7fca740b4f30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fca780045b0 tx=0x7fca7800b3c0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:25.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.938+0000 7fca956bbb80 1 Processor -- start 2024-06-04T05:00:25.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.938+0000 7fca956bbb80 1 -- start start 2024-06-04T05:00:25.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.938+0000 7fca956bbb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a5b320 0x559ea1a5b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.938+0000 7fca956bbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a558a0 0x559ea1a55d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.938+0000 7fca956bbb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a56260 0x559ea1a62080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.938+0000 7fca956bbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559ea19b3d70 con 0x559ea1a558a0 2024-06-04T05:00:25.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.938+0000 7fca956bbb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559ea19a3910 con 0x559ea1a56260 2024-06-04T05:00:25.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.938+0000 7fca956bbb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559ea19a7950 con 0x559ea1a5b320 2024-06-04T05:00:25.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.938+0000 7fca83fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a558a0 0x559ea1a55d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.938+0000 7fca83fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a558a0 0x559ea1a55d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40062/0 (socket says 172.21.15.70:40062) 2024-06-04T05:00:25.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.938+0000 7fca83fff700 1 -- 172.21.15.70:0/57520229 learned_addr learned my addr 172.21.15.70:0/57520229 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:25.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.939+0000 7fca893c3700 1 --2- 172.21.15.70:0/57520229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a56260 0x559ea1a62080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.939+0000 7fca88bc2700 1 --2- 172.21.15.70:0/57520229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a5b320 0x559ea1a5b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.939+0000 7fca83fff700 1 -- 172.21.15.70:0/57520229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a5b320 msgr2=0x559ea1a5b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:25.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.939+0000 7fca83fff700 1 --2- 172.21.15.70:0/57520229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a5b320 0x559ea1a5b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.939+0000 7fca83fff700 1 -- 172.21.15.70:0/57520229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a56260 msgr2=0x559ea1a62080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:25.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.939+0000 7fca83fff700 1 --2- 172.21.15.70:0/57520229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a56260 0x559ea1a62080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.939+0000 7fca83fff700 1 -- 172.21.15.70:0/57520229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ea194cfb0 con 0x559ea1a558a0 2024-06-04T05:00:25.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.939+0000 7fca893c3700 1 --2- 172.21.15.70:0/57520229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a56260 0x559ea1a62080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:25.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.939+0000 7fca83fff700 1 --2- 172.21.15.70:0/57520229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a558a0 0x559ea1a55d20 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fca700a53e0 tx=0x7fca700a8ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03bc6c287ffbdaf server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:25.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.939+0000 7fca6effd700 1 -- 172.21.15.70:0/57520229 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca70020070 con 0x559ea1a558a0 2024-06-04T05:00:25.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.939+0000 7fca6effd700 1 -- 172.21.15.70:0/57520229 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca700ad710 con 0x559ea1a558a0 2024-06-04T05:00:25.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.939+0000 7fca6effd700 1 -- 172.21.15.70:0/57520229 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca700a9820 con 0x559ea1a558a0 2024-06-04T05:00:25.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.940+0000 7fca956bbb80 1 -- 172.21.15.70:0/57520229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a558a0 msgr2=0x559ea1a55d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:25.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.940+0000 7fca956bbb80 1 --2- 172.21.15.70:0/57520229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a558a0 0x559ea1a55d20 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fca700a53e0 tx=0x7fca700a8ef0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.940+0000 7fca956bbb80 1 -- 172.21.15.70:0/57520229 shutdown_connections 2024-06-04T05:00:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.940+0000 7fca956bbb80 1 --2- 172.21.15.70:0/57520229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a56260 0x559ea1a62080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.940+0000 7fca956bbb80 1 --2- 172.21.15.70:0/57520229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a5b320 0x559ea1a5b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.940+0000 7fca956bbb80 1 --2- 172.21.15.70:0/57520229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a558a0 0x559ea1a55d20 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.940+0000 7fca956bbb80 1 -- 172.21.15.70:0/57520229 >> 172.21.15.70:0/57520229 conn(0x559ea1981240 msgr2=0x559ea1a503e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.940+0000 7fca956bbb80 1 -- 172.21.15.70:0/57520229 shutdown_connections 2024-06-04T05:00:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.940+0000 7fca956bbb80 1 -- 172.21.15.70:0/57520229 wait complete. 2024-06-04T05:00:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.940+0000 7fca956bbb80 1 -- start start 2024-06-04T05:00:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.940+0000 7fca956bbb80 1 Processor -- start 2024-06-04T05:00:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca956bbb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a558a0 0x559ea1a61dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca956bbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a56260 0x559ea1a5f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca956bbb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a5b320 0x559ea1a5f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca956bbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559ea18a9fa0 con 0x559ea1a56260 2024-06-04T05:00:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca956bbb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559ea190bda0 con 0x559ea1a558a0 2024-06-04T05:00:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca956bbb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559ea1a62620 con 0x559ea1a5b320 2024-06-04T05:00:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca956bbb80 10 client.0 _ll_register_callbacks cb 0x559ea193e950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca83fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a56260 0x559ea1a5f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca83fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a56260 0x559ea1a5f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40074/0 (socket says 172.21.15.70:40074) 2024-06-04T05:00:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca83fff700 1 -- 172.21.15.70:0/3029394577 learned_addr learned my addr 172.21.15.70:0/3029394577 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca88bc2700 1 --2- 172.21.15.70:0/3029394577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a558a0 0x559ea1a61dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca893c3700 1 --2- 172.21.15.70:0/3029394577 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a5b320 0x559ea1a5f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca83fff700 1 -- 172.21.15.70:0/3029394577 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a5b320 msgr2=0x559ea1a5f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca83fff700 1 --2- 172.21.15.70:0/3029394577 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a5b320 0x559ea1a5f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca83fff700 1 -- 172.21.15.70:0/3029394577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a558a0 msgr2=0x559ea1a61dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca83fff700 1 --2- 172.21.15.70:0/3029394577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a558a0 0x559ea1a61dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca83fff700 1 -- 172.21.15.70:0/3029394577 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ea18f38d0 con 0x559ea1a56260 2024-06-04T05:00:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca893c3700 1 --2- 172.21.15.70:0/3029394577 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a5b320 0x559ea1a5f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:00:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca83fff700 1 --2- 172.21.15.70:0/3029394577 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a56260 0x559ea1a5f0c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fca7000ffa0 tx=0x7fca700109e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b35d7916ae22f4f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:25.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.941+0000 7fca63fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:25.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.942+0000 7fca63fff700 1 -- 172.21.15.70:0/3029394577 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca700c2070 con 0x559ea1a56260 2024-06-04T05:00:25.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.942+0000 7fca63fff700 1 -- 172.21.15.70:0/3029394577 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fca700a95c0 con 0x559ea1a56260 2024-06-04T05:00:25.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.942+0000 7fca956bbb80 1 -- 172.21.15.70:0/3029394577 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca700ad710 con 0x559ea1a56260 2024-06-04T05:00:25.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.942+0000 7fca63fff700 1 -- 172.21.15.70:0/3029394577 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fca70020070 con 0x559ea1a56260 2024-06-04T05:00:25.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.942+0000 7fca956bbb80 10 client.16353 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:25.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.942+0000 7fca956bbb80 1 -- 172.21.15.70:0/3029394577 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559ea195d760 con 0x559ea1a56260 2024-06-04T05:00:25.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.942+0000 7fca63fff700 1 -- 172.21.15.70:0/3029394577 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca700ad710 con 0x559ea1a56260 2024-06-04T05:00:25.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.942+0000 7fca63fff700 1 -- 172.21.15.70:0/3029394577 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fca700a4cd0 con 0x559ea1a56260 2024-06-04T05:00:25.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.942+0000 7fca956bbb80 10 client.16353 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:25.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.942+0000 7fca956bbb80 10 client.16353 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:00:25.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.942+0000 7fca956bbb80 20 client.16353 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:25.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.942+0000 7fca956bbb80 1 --2- 172.21.15.70:0/3029394577 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x559ea1a6f090 0x559ea1a71480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.942+0000 7fca956bbb80 4 client.16353 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:00:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.942+0000 7fca956bbb80 1 -- 172.21.15.70:0/3029394577 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ea19b5cc0 con 0x559ea1a6f090 2024-06-04T05:00:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.943+0000 7fca88bc2700 1 --2- 172.21.15.70:0/3029394577 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x559ea1a6f090 0x559ea1a71480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.943+0000 7fca88bc2700 1 --2- 172.21.15.70:0/3029394577 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x559ea1a6f090 0x559ea1a71480 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d2244d5b521796fe server_cookie=85ba733d10dfdb59 in_seq=0 out_seq=0 2024-06-04T05:00:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.943+0000 7fca63fff700 10 client.16353 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:00:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.944+0000 7fca63fff700 1 -- 172.21.15.70:0/3029394577 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559ea19b5cc0 con 0x559ea1a6f090 2024-06-04T05:00:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.944+0000 7fca63fff700 10 client.16353 handle_command_reply: tid=0 2024-06-04T05:00:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.944+0000 7fca956bbb80 1 client.16353 shutdown 2024-06-04T05:00:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.944+0000 7fca956bbb80 1 -- 172.21.15.70:0/3029394577 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a56260 msgr2=0x559ea1a5f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.944+0000 7fca956bbb80 1 --2- 172.21.15.70:0/3029394577 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a56260 0x559ea1a5f0c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fca7000ffa0 tx=0x7fca700109e0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.944+0000 7fca956bbb80 1 -- 172.21.15.70:0/3029394577 shutdown_connections 2024-06-04T05:00:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.944+0000 7fca956bbb80 1 --2- 172.21.15.70:0/3029394577 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a5b320 0x559ea1a5f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.944+0000 7fca956bbb80 1 --2- 172.21.15.70:0/3029394577 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x559ea1a6f090 0x559ea1a71480 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.944+0000 7fca956bbb80 1 --2- 172.21.15.70:0/3029394577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a558a0 0x559ea1a61dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.944+0000 7fca956bbb80 1 --2- 172.21.15.70:0/3029394577 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a56260 0x559ea1a5f0c0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.944+0000 7fca956bbb80 1 -- 172.21.15.70:0/3029394577 >> 172.21.15.70:0/3029394577 conn(0x559ea1981240 msgr2=0x559ea1a4faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:25.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.944+0000 7fca63fff700 0 client.16353 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:00:25.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.944+0000 7fca956bbb80 1 -- 172.21.15.70:0/3029394577 shutdown_connections 2024-06-04T05:00:25.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.944+0000 7fca956bbb80 1 -- 172.21.15.70:0/3029394577 wait complete. 2024-06-04T05:00:25.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.944+0000 7fca956bbb80 20 client.16353 trim_cache size 0 max 16384 2024-06-04T05:00:25.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.959+0000 7fca956bbb80 1 Processor -- start 2024-06-04T05:00:25.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.959+0000 7fca956bbb80 1 -- start start 2024-06-04T05:00:25.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.959+0000 7fca956bbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a558a0 0x559ea1a5d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.959+0000 7fca956bbb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a56260 0x559ea1a5ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.959+0000 7fca956bbb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a5b320 0x559ea1a5e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.959+0000 7fca956bbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fca841ade10 con 0x559ea1a558a0 2024-06-04T05:00:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.959+0000 7fca956bbb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fca841af9b0 con 0x559ea1a5b320 2024-06-04T05:00:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.959+0000 7fca956bbb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fca841ae4e0 con 0x559ea1a56260 2024-06-04T05:00:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.959+0000 7fca88bc2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a558a0 0x559ea1a5d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.959+0000 7fca88bc2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a558a0 0x559ea1a5d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40084/0 (socket says 172.21.15.70:40084) 2024-06-04T05:00:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.959+0000 7fca88bc2700 1 -- 172.21.15.70:0/2835293123 learned_addr learned my addr 172.21.15.70:0/2835293123 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.960+0000 7fca893c3700 1 --2- 172.21.15.70:0/2835293123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a5b320 0x559ea1a5e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.960+0000 7fca83fff700 1 --2- 172.21.15.70:0/2835293123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a56260 0x559ea1a5ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.960+0000 7fca88bc2700 1 -- 172.21.15.70:0/2835293123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a56260 msgr2=0x559ea1a5ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.960+0000 7fca88bc2700 1 --2- 172.21.15.70:0/2835293123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a56260 0x559ea1a5ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.960+0000 7fca88bc2700 1 -- 172.21.15.70:0/2835293123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a5b320 msgr2=0x559ea1a5e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.960+0000 7fca88bc2700 1 --2- 172.21.15.70:0/2835293123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a5b320 0x559ea1a5e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.960+0000 7fca88bc2700 1 -- 172.21.15.70:0/2835293123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ea1a51e10 con 0x559ea1a558a0 2024-06-04T05:00:25.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.960+0000 7fca83fff700 1 --2- 172.21.15.70:0/2835293123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a56260 0x559ea1a5ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:25.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.960+0000 7fca893c3700 1 --2- 172.21.15.70:0/2835293123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a5b320 0x559ea1a5e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:25.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.960+0000 7fca88bc2700 1 --2- 172.21.15.70:0/2835293123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a558a0 0x559ea1a5d880 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fca780046e0 tx=0x7fca78013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2286d231f888ff2b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:25.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.960+0000 7fca6e7fc700 1 -- 172.21.15.70:0/2835293123 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca78011850 con 0x559ea1a558a0 2024-06-04T05:00:25.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.960+0000 7fca6e7fc700 1 -- 172.21.15.70:0/2835293123 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca78027280 con 0x559ea1a558a0 2024-06-04T05:00:25.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.960+0000 7fca6e7fc700 1 -- 172.21.15.70:0/2835293123 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7801e680 con 0x559ea1a558a0 2024-06-04T05:00:25.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.961+0000 7fca956bbb80 1 -- 172.21.15.70:0/2835293123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a558a0 msgr2=0x559ea1a5d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:25.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.961+0000 7fca956bbb80 1 --2- 172.21.15.70:0/2835293123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a558a0 0x559ea1a5d880 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fca780046e0 tx=0x7fca78013000 comp rx=0 tx=0).stop 2024-06-04T05:00:25.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.961+0000 7fca956bbb80 1 -- 172.21.15.70:0/2835293123 shutdown_connections 2024-06-04T05:00:25.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.961+0000 7fca956bbb80 1 --2- 172.21.15.70:0/2835293123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a5b320 0x559ea1a5e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.961+0000 7fca956bbb80 1 --2- 172.21.15.70:0/2835293123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a558a0 0x559ea1a5d880 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.961+0000 7fca956bbb80 1 --2- 172.21.15.70:0/2835293123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a56260 0x559ea1a5ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.961+0000 7fca956bbb80 1 -- 172.21.15.70:0/2835293123 >> 172.21.15.70:0/2835293123 conn(0x559ea1981240 msgr2=0x559ea1a61180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.961+0000 7fca956bbb80 1 -- 172.21.15.70:0/2835293123 shutdown_connections 2024-06-04T05:00:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.961+0000 7fca956bbb80 1 -- 172.21.15.70:0/2835293123 wait complete. 2024-06-04T05:00:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.961+0000 7fca956bbb80 1 -- start start 2024-06-04T05:00:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.961+0000 7fca956bbb80 1 Processor -- start 2024-06-04T05:00:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca956bbb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a558a0 0x559ea1a740f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca956bbb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a56260 0x559ea1a74630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca956bbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a5b320 0x559ea1a6f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca956bbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fca841af280 con 0x559ea1a5b320 2024-06-04T05:00:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca956bbb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fca841aebb0 con 0x559ea1a56260 2024-06-04T05:00:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca956bbb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fca841abdb0 con 0x559ea1a558a0 2024-06-04T05:00:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca956bbb80 10 client.0 _ll_register_callbacks cb 0x559ea193e950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca893c3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a5b320 0x559ea1a6f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca893c3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a5b320 0x559ea1a6f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40088/0 (socket says 172.21.15.70:40088) 2024-06-04T05:00:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca83fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a56260 0x559ea1a74630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca893c3700 1 -- 172.21.15.70:0/307739706 learned_addr learned my addr 172.21.15.70:0/307739706 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:25.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca88bc2700 1 --2- 172.21.15.70:0/307739706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a558a0 0x559ea1a740f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca893c3700 1 -- 172.21.15.70:0/307739706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a558a0 msgr2=0x559ea1a740f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:25.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca893c3700 1 --2- 172.21.15.70:0/307739706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a558a0 0x559ea1a740f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca893c3700 1 -- 172.21.15.70:0/307739706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a56260 msgr2=0x559ea1a74630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:25.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca893c3700 1 --2- 172.21.15.70:0/307739706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a56260 0x559ea1a74630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca893c3700 1 -- 172.21.15.70:0/307739706 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ea1a6fe40 con 0x559ea1a5b320 2024-06-04T05:00:25.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca83fff700 1 --2- 172.21.15.70:0/307739706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a56260 0x559ea1a74630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:25.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca893c3700 1 --2- 172.21.15.70:0/307739706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a5b320 0x559ea1a6f670 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fca7c015000 tx=0x7fca7c003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7aa057db7f82c12 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:25.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca63fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:25.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca63fff700 1 -- 172.21.15.70:0/307739706 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7c0076f0 con 0x559ea1a5b320 2024-06-04T05:00:25.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca63fff700 1 -- 172.21.15.70:0/307739706 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fca7c007890 con 0x559ea1a5b320 2024-06-04T05:00:25.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.962+0000 7fca956bbb80 1 -- 172.21.15.70:0/307739706 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca78027280 con 0x559ea1a5b320 2024-06-04T05:00:25.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.963+0000 7fca63fff700 1 -- 172.21.15.70:0/307739706 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fca7c007a90 con 0x559ea1a5b320 2024-06-04T05:00:25.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.963+0000 7fca956bbb80 10 client.16359 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:25.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.963+0000 7fca956bbb80 1 -- 172.21.15.70:0/307739706 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559ea1ad9320 con 0x559ea1a5b320 2024-06-04T05:00:25.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.963+0000 7fca63fff700 1 -- 172.21.15.70:0/307739706 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca78027280 con 0x559ea1a5b320 2024-06-04T05:00:25.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.963+0000 7fca63fff700 1 -- 172.21.15.70:0/307739706 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fca7c011070 con 0x559ea1a5b320 2024-06-04T05:00:25.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.963+0000 7fca956bbb80 10 client.16359 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:25.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.963+0000 7fca956bbb80 10 client.16359 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:00:25.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.963+0000 7fca956bbb80 20 client.16359 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:25.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.963+0000 7fca956bbb80 1 --2- 172.21.15.70:0/307739706 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x559ea1ad9ae0 0x559ea1adbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:25.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.963+0000 7fca956bbb80 4 client.16359 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:00:25.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.963+0000 7fca956bbb80 1 -- 172.21.15.70:0/307739706 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559ea19af4a0 con 0x559ea1ad9ae0 2024-06-04T05:00:25.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.963+0000 7fca88bc2700 1 --2- 172.21.15.70:0/307739706 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x559ea1ad9ae0 0x559ea1adbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:25.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.964+0000 7fca88bc2700 1 --2- 172.21.15.70:0/307739706 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x559ea1ad9ae0 0x559ea1adbf00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=78f876aa9c8d92c3 server_cookie=58a24340a753154e in_seq=0 out_seq=0 2024-06-04T05:00:25.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.964+0000 7fca63fff700 10 client.16359 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:00:25.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.964+0000 7fca63fff700 1 -- 172.21.15.70:0/307739706 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559ea19af4a0 con 0x559ea1ad9ae0 2024-06-04T05:00:25.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.964+0000 7fca63fff700 10 client.16359 handle_command_reply: tid=0 2024-06-04T05:00:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.964+0000 7fca956bbb80 1 client.16359 shutdown 2024-06-04T05:00:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.964+0000 7fca956bbb80 1 -- 172.21.15.70:0/307739706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a5b320 msgr2=0x559ea1a6f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.964+0000 7fca956bbb80 1 --2- 172.21.15.70:0/307739706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a5b320 0x559ea1a6f670 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fca7c015000 tx=0x7fca7c003c30 comp rx=0 tx=0).stop 2024-06-04T05:00:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.964+0000 7fca956bbb80 1 -- 172.21.15.70:0/307739706 shutdown_connections 2024-06-04T05:00:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.964+0000 7fca956bbb80 1 --2- 172.21.15.70:0/307739706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ea1a5b320 0x559ea1a6f670 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.964+0000 7fca956bbb80 1 --2- 172.21.15.70:0/307739706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559ea1a558a0 0x559ea1a740f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.964+0000 7fca956bbb80 1 --2- 172.21.15.70:0/307739706 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x559ea1ad9ae0 0x559ea1adbf00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.964+0000 7fca956bbb80 1 --2- 172.21.15.70:0/307739706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ea1a56260 0x559ea1a74630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.964+0000 7fca956bbb80 1 -- 172.21.15.70:0/307739706 >> 172.21.15.70:0/307739706 conn(0x559ea1981240 msgr2=0x559ea1a54890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.964+0000 7fca63fff700 0 client.16359 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:00:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.964+0000 7fca956bbb80 1 -- 172.21.15.70:0/307739706 shutdown_connections 2024-06-04T05:00:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.964+0000 7fca956bbb80 1 -- 172.21.15.70:0/307739706 wait complete. 2024-06-04T05:00:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.964+0000 7fca956bbb80 20 client.16359 trim_cache size 0 max 16384 2024-06-04T05:00:25.972 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:00:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.965+0000 7fca8ae26700 1 -- 172.21.15.70:0/3019117824 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fca740b2a70 msgr2=0x7fca740b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.965+0000 7fca8ae26700 1 --2- 172.21.15.70:0/3019117824 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fca740b2a70 0x7fca740b4f30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fca780045b0 tx=0x7fca7800b3c0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.965+0000 7fca8ae26700 1 -- 172.21.15.70:0/3019117824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca84102e00 msgr2=0x7fca8419ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.965+0000 7fca8ae26700 1 --2- 172.21.15.70:0/3019117824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca84102e00 0x7fca8419ae40 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fca7000d950 tx=0x7fca7000dc60 comp rx=0 tx=0).stop 2024-06-04T05:00:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.965+0000 7fca8ae26700 1 -- 172.21.15.70:0/3019117824 shutdown_connections 2024-06-04T05:00:25.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.965+0000 7fca8ae26700 1 --2- 172.21.15.70:0/3019117824 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fca740b2a70 0x7fca740b4f30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.965+0000 7fca8ae26700 1 --2- 172.21.15.70:0/3019117824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca84108520 0x7fca841a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.965+0000 7fca8ae26700 1 --2- 172.21.15.70:0/3019117824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca84102440 0x7fca8419a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.965+0000 7fca8ae26700 1 --2- 172.21.15.70:0/3019117824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca84102e00 0x7fca8419ae40 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:25.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.965+0000 7fca8ae26700 1 -- 172.21.15.70:0/3019117824 >> 172.21.15.70:0/3019117824 conn(0x7fca840793f0 msgr2=0x7fca84100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:25.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.965+0000 7fca8ae26700 1 -- 172.21.15.70:0/3019117824 shutdown_connections 2024-06-04T05:00:25.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:25.965+0000 7fca8ae26700 1 -- 172.21.15.70:0/3019117824 wait complete. 2024-06-04T05:00:26.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:26 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:00:26.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:26 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:00:26.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:26 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:00:26.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:26 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:00:26.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:26 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:00:26.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:26 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:00:26.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 3.10 MB/sec warmup 79 sec latency 3626.808 ms 2024-06-04T05:00:26.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:26 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:00:26.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:26 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:00:26.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:26 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:00:26.781 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:00:27.020 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:00:27.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.529+0000 7fb323cc5640 1 -- 172.21.15.70:0/2840493959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb31c1011c0 msgr2=0x7fb31c1015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:27.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.529+0000 7fb323cc5640 1 --2- 172.21.15.70:0/2840493959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb31c1011c0 0x7fb31c1015a0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb318009900 tx=0x7fb31802a410 comp rx=0 tx=0).stop 2024-06-04T05:00:27.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.529+0000 7fb323cc5640 1 -- 172.21.15.70:0/2840493959 shutdown_connections 2024-06-04T05:00:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.529+0000 7fb323cc5640 1 --2- 172.21.15.70:0/2840493959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb31c101b70 0x7fb31c107700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.529+0000 7fb323cc5640 1 --2- 172.21.15.70:0/2840493959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb31c107e30 0x7fb31c10f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.529+0000 7fb323cc5640 1 --2- 172.21.15.70:0/2840493959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb31c1011c0 0x7fb31c1015a0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.529+0000 7fb323cc5640 1 -- 172.21.15.70:0/2840493959 >> 172.21.15.70:0/2840493959 conn(0x7fb31c077ee0 msgr2=0x7fb31c0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.529+0000 7fb323cc5640 1 -- 172.21.15.70:0/2840493959 shutdown_connections 2024-06-04T05:00:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.529+0000 7fb323cc5640 1 -- 172.21.15.70:0/2840493959 wait complete. 2024-06-04T05:00:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.530+0000 7fb323cc5640 1 Processor -- start 2024-06-04T05:00:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.530+0000 7fb323cc5640 1 -- start start 2024-06-04T05:00:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.530+0000 7fb323cc5640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb31c1011c0 0x7fb31c19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.530+0000 7fb323cc5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb31c107e30 0x7fb31c19b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.530+0000 7fb323cc5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb31c19fc50 0x7fb31c1a0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.530+0000 7fb323cc5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb31c112bf0 con 0x7fb31c107e30 2024-06-04T05:00:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.530+0000 7fb323cc5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb31c112a70 con 0x7fb31c19fc50 2024-06-04T05:00:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.530+0000 7fb323cc5640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb31c112d70 con 0x7fb31c1011c0 2024-06-04T05:00:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.530+0000 7fb321260640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb31c107e30 0x7fb31c19b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.530+0000 7fb321260640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb31c107e30 0x7fb31c19b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59832/0 (socket says 172.21.15.70:59832) 2024-06-04T05:00:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.530+0000 7fb321260640 1 -- 172.21.15.70:0/3804443286 learned_addr learned my addr 172.21.15.70:0/3804443286 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.530+0000 7fb321a61640 1 --2- 172.21.15.70:0/3804443286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb31c1011c0 0x7fb31c19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.530+0000 7fb321260640 1 -- 172.21.15.70:0/3804443286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb31c1011c0 msgr2=0x7fb31c19b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.530+0000 7fb321260640 1 --2- 172.21.15.70:0/3804443286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb31c1011c0 0x7fb31c19b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.530+0000 7fb321260640 1 -- 172.21.15.70:0/3804443286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb31c19fc50 msgr2=0x7fb31c1a0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.530+0000 7fb321260640 1 --2- 172.21.15.70:0/3804443286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb31c19fc50 0x7fb31c1a0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.530+0000 7fb321260640 1 -- 172.21.15.70:0/3804443286 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb31c1a0740 con 0x7fb31c107e30 2024-06-04T05:00:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.531+0000 7fb321260640 1 --2- 172.21.15.70:0/3804443286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb31c107e30 0x7fb31c19b8d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00e9f0 tx=0x7fb30c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:27.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.531+0000 7fb312ffd640 1 -- 172.21.15.70:0/3804443286 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c00ccc0 con 0x7fb31c107e30 2024-06-04T05:00:27.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.531+0000 7fb323cc5640 1 -- 172.21.15.70:0/3804443286 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb31c1a0a30 con 0x7fb31c107e30 2024-06-04T05:00:27.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.531+0000 7fb312ffd640 1 -- 172.21.15.70:0/3804443286 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb30c0108c0 con 0x7fb31c107e30 2024-06-04T05:00:27.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.531+0000 7fb312ffd640 1 -- 172.21.15.70:0/3804443286 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c018660 con 0x7fb31c107e30 2024-06-04T05:00:27.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.531+0000 7fb323cc5640 1 -- 172.21.15.70:0/3804443286 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb31c1a81c0 con 0x7fb31c107e30 2024-06-04T05:00:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.533+0000 7fb3269d6740 1 Processor -- start 2024-06-04T05:00:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb3269d6740 1 -- start start 2024-06-04T05:00:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb312ffd640 1 -- 172.21.15.70:0/3804443286 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fb30c010430 con 0x7fb31c107e30 2024-06-04T05:00:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb3269d6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x560050770c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb3269d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560050771140 0x56005077b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb3269d6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56005077ba00 0x56005077ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb3269d6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5600506e2c30 con 0x560050770820 2024-06-04T05:00:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb3269d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5600506d27d0 con 0x560050771140 2024-06-04T05:00:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb3269d6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5600506d6810 con 0x56005077ba00 2024-06-04T05:00:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb322262640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x560050770c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb322262640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x560050770c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59842/0 (socket says 172.21.15.70:59842) 2024-06-04T05:00:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb322262640 1 -- 172.21.15.70:0/4084812772 learned_addr learned my addr 172.21.15.70:0/4084812772 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb321a61640 1 --2- 172.21.15.70:0/4084812772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560050771140 0x56005077b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb321260640 1 --2- 172.21.15.70:0/4084812772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56005077ba00 0x56005077ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb322262640 1 -- 172.21.15.70:0/4084812772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56005077ba00 msgr2=0x56005077ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb322262640 1 --2- 172.21.15.70:0/4084812772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56005077ba00 0x56005077ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb322262640 1 -- 172.21.15.70:0/4084812772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560050771140 msgr2=0x56005077b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb322262640 1 --2- 172.21.15.70:0/4084812772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560050771140 0x56005077b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb322262640 1 -- 172.21.15.70:0/4084812772 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600503e7210 con 0x560050770820 2024-06-04T05:00:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb321a61640 1 --2- 172.21.15.70:0/4084812772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560050771140 0x56005077b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.534+0000 7fb322262640 1 --2- 172.21.15.70:0/4084812772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x560050770c00 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb314007bb0 tx=0x7fb31400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ccc92bc4246327 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.535+0000 7fb312ffd640 1 --2- 172.21.15.70:0/3804443286 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3080b2a10 0x7fb3080b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.535+0000 7fb307fff640 1 -- 172.21.15.70:0/4084812772 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb314014070 con 0x560050770820 2024-06-04T05:00:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.535+0000 7fb321a61640 1 --2- 172.21.15.70:0/3804443286 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3080b2a10 0x7fb3080b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.535+0000 7fb307fff640 1 -- 172.21.15.70:0/4084812772 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb31400fae0 con 0x560050770820 2024-06-04T05:00:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.535+0000 7fb307fff640 1 -- 172.21.15.70:0/4084812772 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31400e940 con 0x560050770820 2024-06-04T05:00:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.535+0000 7fb312ffd640 1 -- 172.21.15.70:0/3804443286 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb30c0e2a70 con 0x7fb31c107e30 2024-06-04T05:00:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.535+0000 7fb321a61640 1 --2- 172.21.15.70:0/3804443286 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3080b2a10 0x7fb3080b4ed0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb3180057c0 tx=0x7fb3180058d0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.535+0000 7fb3269d6740 1 -- 172.21.15.70:0/4084812772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 msgr2=0x560050770c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.535+0000 7fb3269d6740 1 --2- 172.21.15.70:0/4084812772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x560050770c00 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb314007bb0 tx=0x7fb31400c6a0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.536+0000 7fb3269d6740 1 -- 172.21.15.70:0/4084812772 shutdown_connections 2024-06-04T05:00:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.536+0000 7fb3269d6740 1 --2- 172.21.15.70:0/4084812772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560050771140 0x56005077b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.536+0000 7fb3269d6740 1 --2- 172.21.15.70:0/4084812772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56005077ba00 0x56005077ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.536+0000 7fb3269d6740 1 --2- 172.21.15.70:0/4084812772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x560050770c00 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.536+0000 7fb3269d6740 1 -- 172.21.15.70:0/4084812772 >> 172.21.15.70:0/4084812772 conn(0x56005076cd30 msgr2=0x56005076d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.536+0000 7fb3269d6740 1 -- 172.21.15.70:0/4084812772 shutdown_connections 2024-06-04T05:00:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.536+0000 7fb3269d6740 1 -- 172.21.15.70:0/4084812772 wait complete. 2024-06-04T05:00:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.536+0000 7fb3269d6740 1 -- start start 2024-06-04T05:00:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.536+0000 7fb3269d6740 1 Processor -- start 2024-06-04T05:00:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.536+0000 7fb3269d6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x5600507789d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.536+0000 7fb3269d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560050771140 0x560050778f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.536+0000 7fb3269d6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56005077ba00 0x560050779450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.536+0000 7fb3269d6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5600503b8790 con 0x560050770820 2024-06-04T05:00:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.536+0000 7fb3269d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56005068e890 con 0x560050771140 2024-06-04T05:00:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.536+0000 7fb3269d6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56005077e630 con 0x56005077ba00 2024-06-04T05:00:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb321a61640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x5600507789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb321a61640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x5600507789d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59852/0 (socket says 172.21.15.70:59852) 2024-06-04T05:00:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb321a61640 1 -- 172.21.15.70:0/3063666394 learned_addr learned my addr 172.21.15.70:0/3063666394 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb3269d6740 10 client.0 _ll_register_callbacks cb 0x5600503e6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb322262640 1 --2- 172.21.15.70:0/3063666394 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56005077ba00 0x560050779450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb321260640 1 --2- 172.21.15.70:0/3063666394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560050771140 0x560050778f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb321a61640 1 -- 172.21.15.70:0/3063666394 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56005077ba00 msgr2=0x560050779450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb321a61640 1 --2- 172.21.15.70:0/3063666394 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56005077ba00 0x560050779450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb321a61640 1 -- 172.21.15.70:0/3063666394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560050771140 msgr2=0x560050778f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb321a61640 1 --2- 172.21.15.70:0/3063666394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560050771140 0x560050778f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb321a61640 1 -- 172.21.15.70:0/3063666394 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600504b3620 con 0x560050770820 2024-06-04T05:00:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb321a61640 1 --2- 172.21.15.70:0/3063666394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x5600507789d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb318033000 tx=0x7fb318009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5f303f5338367a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb3057fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb3057fa640 1 -- 172.21.15.70:0/3063666394 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb318035040 con 0x560050770820 2024-06-04T05:00:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb3057fa640 1 -- 172.21.15.70:0/3063666394 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb3180043c0 con 0x560050770820 2024-06-04T05:00:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb3269d6740 1 -- 172.21.15.70:0/3063666394 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb31400fae0 con 0x560050770820 2024-06-04T05:00:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb3057fa640 1 -- 172.21.15.70:0/3063666394 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb3180035f0 con 0x560050770820 2024-06-04T05:00:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb3269d6740 10 client.16371 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.537+0000 7fb3269d6740 1 -- 172.21.15.70:0/3063666394 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56005078a420 con 0x560050770820 2024-06-04T05:00:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.538+0000 7fb3057fa640 1 -- 172.21.15.70:0/3063666394 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb31400fae0 con 0x560050770820 2024-06-04T05:00:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.538+0000 7fb3057fa640 1 -- 172.21.15.70:0/3063666394 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb318044070 con 0x560050770820 2024-06-04T05:00:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.538+0000 7fb3269d6740 10 client.16371 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.538+0000 7fb3269d6740 10 client.16371 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.538+0000 7fb3269d6740 20 client.16371 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.538+0000 7fb3269d6740 1 --2- 172.21.15.70:0/3063666394 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56005078b980 0x56005078ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.538+0000 7fb3269d6740 4 client.16371 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:00:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.538+0000 7fb3269d6740 1 -- 172.21.15.70:0/3063666394 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600506e4b80 con 0x56005078b980 2024-06-04T05:00:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.538+0000 7fb321260640 1 --2- 172.21.15.70:0/3063666394 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56005078b980 0x56005078ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.538+0000 7fb3057fa640 10 client.16371 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.538+0000 7fb321260640 1 --2- 172.21.15.70:0/3063666394 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56005078b980 0x56005078ddf0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=380cc6843d47ec98 server_cookie=d927f76b392cf6d0 in_seq=0 out_seq=0 2024-06-04T05:00:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.539+0000 7fb3057fa640 1 -- 172.21.15.70:0/3063666394 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5600506e4b80 con 0x56005078b980 2024-06-04T05:00:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.539+0000 7fb3057fa640 10 client.16371 handle_command_reply: tid=0 2024-06-04T05:00:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.539+0000 7fb3269d6740 1 client.16371 shutdown 2024-06-04T05:00:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.539+0000 7fb3269d6740 1 -- 172.21.15.70:0/3063666394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 msgr2=0x5600507789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.539+0000 7fb3269d6740 1 --2- 172.21.15.70:0/3063666394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x5600507789d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb318033000 tx=0x7fb318009260 comp rx=0 tx=0).stop 2024-06-04T05:00:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.539+0000 7fb3269d6740 1 -- 172.21.15.70:0/3063666394 shutdown_connections 2024-06-04T05:00:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.539+0000 7fb3269d6740 1 --2- 172.21.15.70:0/3063666394 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56005078b980 0x56005078ddf0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.539+0000 7fb3269d6740 1 --2- 172.21.15.70:0/3063666394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560050771140 0x560050778f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.539+0000 7fb3269d6740 1 --2- 172.21.15.70:0/3063666394 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56005077ba00 0x560050779450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.539+0000 7fb3057fa640 0 client.16371 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.539+0000 7fb3269d6740 1 --2- 172.21.15.70:0/3063666394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x5600507789d0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.539+0000 7fb3269d6740 1 -- 172.21.15.70:0/3063666394 >> 172.21.15.70:0/3063666394 conn(0x56005076cd30 msgr2=0x56005076bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.539+0000 7fb3269d6740 1 -- 172.21.15.70:0/3063666394 shutdown_connections 2024-06-04T05:00:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.539+0000 7fb3269d6740 1 -- 172.21.15.70:0/3063666394 wait complete. 2024-06-04T05:00:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.539+0000 7fb3269d6740 20 client.16371 trim_cache size 0 max 16384 2024-06-04T05:00:27.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.554+0000 7fb3269d6740 1 Processor -- start 2024-06-04T05:00:27.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.554+0000 7fb3269d6740 1 -- start start 2024-06-04T05:00:27.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb3269d6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x5600507731e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb3269d6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560050771140 0x560050779270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb3269d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005077ba00 0x560050793700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb3269d6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb31c1b0830 con 0x560050770820 2024-06-04T05:00:27.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb3269d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb31c1b23d0 con 0x56005077ba00 2024-06-04T05:00:27.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb3269d6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb31c1b0f00 con 0x560050771140 2024-06-04T05:00:27.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb321a61640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x5600507731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:27.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb321a61640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x5600507731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59856/0 (socket says 172.21.15.70:59856) 2024-06-04T05:00:27.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb321a61640 1 -- 172.21.15.70:0/435582013 learned_addr learned my addr 172.21.15.70:0/435582013 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:27.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb321260640 1 --2- 172.21.15.70:0/435582013 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560050771140 0x560050779270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:27.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb322262640 1 --2- 172.21.15.70:0/435582013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005077ba00 0x560050793700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:27.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb321a61640 1 -- 172.21.15.70:0/435582013 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560050771140 msgr2=0x560050779270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb321a61640 1 --2- 172.21.15.70:0/435582013 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560050771140 0x560050779270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb321a61640 1 -- 172.21.15.70:0/435582013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005077ba00 msgr2=0x560050793700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb321a61640 1 --2- 172.21.15.70:0/435582013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005077ba00 0x560050793700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb321a61640 1 -- 172.21.15.70:0/435582013 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560050776f20 con 0x560050770820 2024-06-04T05:00:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb322262640 1 --2- 172.21.15.70:0/435582013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005077ba00 0x560050793700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:00:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb321260640 1 --2- 172.21.15.70:0/435582013 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560050771140 0x560050779270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:00:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb321a61640 1 --2- 172.21.15.70:0/435582013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x5600507731e0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb318005c40 tx=0x7fb3180314d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31000a1d2f587a2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb3077fe640 1 -- 172.21.15.70:0/435582013 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb318035070 con 0x560050770820 2024-06-04T05:00:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb3077fe640 1 -- 172.21.15.70:0/435582013 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb31803da20 con 0x560050770820 2024-06-04T05:00:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.555+0000 7fb3077fe640 1 -- 172.21.15.70:0/435582013 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb318050430 con 0x560050770820 2024-06-04T05:00:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.556+0000 7fb3269d6740 1 -- 172.21.15.70:0/435582013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 msgr2=0x5600507731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.556+0000 7fb3269d6740 1 --2- 172.21.15.70:0/435582013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x5600507731e0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb318005c40 tx=0x7fb3180314d0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.556+0000 7fb3269d6740 1 -- 172.21.15.70:0/435582013 shutdown_connections 2024-06-04T05:00:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.556+0000 7fb3269d6740 1 --2- 172.21.15.70:0/435582013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005077ba00 0x560050793700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.556+0000 7fb3269d6740 1 --2- 172.21.15.70:0/435582013 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560050771140 0x560050779270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.556+0000 7fb3269d6740 1 --2- 172.21.15.70:0/435582013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560050770820 0x5600507731e0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.556+0000 7fb3269d6740 1 -- 172.21.15.70:0/435582013 >> 172.21.15.70:0/435582013 conn(0x56005076cd30 msgr2=0x56005076bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.556+0000 7fb3269d6740 1 -- 172.21.15.70:0/435582013 shutdown_connections 2024-06-04T05:00:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.556+0000 7fb3269d6740 1 -- 172.21.15.70:0/435582013 wait complete. 2024-06-04T05:00:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.556+0000 7fb3269d6740 1 -- start start 2024-06-04T05:00:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.556+0000 7fb3269d6740 1 Processor -- start 2024-06-04T05:00:27.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb3269d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560050770820 0x560050783c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb3269d6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560050771140 0x56005077efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb3269d6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56005077ba00 0x56005077f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb3269d6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb31c1b1ca0 con 0x56005077ba00 2024-06-04T05:00:27.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb3269d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb31c1b15d0 con 0x560050770820 2024-06-04T05:00:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb3269d6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb31c1ae810 con 0x560050771140 2024-06-04T05:00:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb3269d6740 10 client.0 _ll_register_callbacks cb 0x7fb318009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb322262640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56005077ba00 0x56005077f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb322262640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56005077ba00 0x56005077f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59870/0 (socket says 172.21.15.70:59870) 2024-06-04T05:00:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb322262640 1 -- 172.21.15.70:0/3215211542 learned_addr learned my addr 172.21.15.70:0/3215211542 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb321a61640 1 --2- 172.21.15.70:0/3215211542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560050770820 0x560050783c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb322262640 1 -- 172.21.15.70:0/3215211542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560050771140 msgr2=0x56005077efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb322262640 1 --2- 172.21.15.70:0/3215211542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560050771140 0x56005077efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb322262640 1 -- 172.21.15.70:0/3215211542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560050770820 msgr2=0x560050783c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb322262640 1 --2- 172.21.15.70:0/3215211542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560050770820 0x560050783c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb322262640 1 -- 172.21.15.70:0/3215211542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56005077fa40 con 0x56005077ba00 2024-06-04T05:00:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb322262640 1 --2- 172.21.15.70:0/3215211542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56005077ba00 0x56005077f500 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb314000f80 tx=0x7fb3140078f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b30f4c6e96dd37 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb3057fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.557+0000 7fb3057fa640 1 -- 172.21.15.70:0/3215211542 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb314028060 con 0x56005077ba00 2024-06-04T05:00:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.558+0000 7fb3057fa640 1 -- 172.21.15.70:0/3215211542 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb314004450 con 0x56005077ba00 2024-06-04T05:00:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.558+0000 7fb3269d6740 1 -- 172.21.15.70:0/3215211542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb31803da20 con 0x56005077ba00 2024-06-04T05:00:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.558+0000 7fb3057fa640 1 -- 172.21.15.70:0/3215211542 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb314014040 con 0x56005077ba00 2024-06-04T05:00:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.558+0000 7fb3269d6740 10 client.16377 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.558+0000 7fb3269d6740 1 -- 172.21.15.70:0/3215211542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5600507f6200 con 0x56005077ba00 2024-06-04T05:00:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.558+0000 7fb3057fa640 1 -- 172.21.15.70:0/3215211542 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb31803da20 con 0x56005077ba00 2024-06-04T05:00:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.558+0000 7fb3057fa640 1 -- 172.21.15.70:0/3215211542 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb314022070 con 0x56005077ba00 2024-06-04T05:00:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.558+0000 7fb3269d6740 10 client.16377 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.558+0000 7fb3269d6740 10 client.16377 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.558+0000 7fb3269d6740 20 client.16377 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.558+0000 7fb3269d6740 1 --2- 172.21.15.70:0/3215211542 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5600507f66e0 0x5600507f8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.558+0000 7fb3269d6740 4 client.16377 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:00:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.558+0000 7fb3269d6740 1 -- 172.21.15.70:0/3215211542 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5600506de360 con 0x5600507f66e0 2024-06-04T05:00:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.558+0000 7fb321a61640 1 --2- 172.21.15.70:0/3215211542 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5600507f66e0 0x5600507f8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.559+0000 7fb3057fa640 10 client.16377 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.559+0000 7fb321a61640 1 --2- 172.21.15.70:0/3215211542 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5600507f66e0 0x5600507f8b20 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba843bd414d993ca server_cookie=4a6726b3caa05d4d in_seq=0 out_seq=0 2024-06-04T05:00:27.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.578+0000 7fb3057fa640 1 -- 172.21.15.70:0/3215211542 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5600506de360 con 0x5600507f66e0 2024-06-04T05:00:27.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.578+0000 7fb3057fa640 10 client.16377 handle_command_reply: tid=0 2024-06-04T05:00:27.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.578+0000 7fb3269d6740 1 client.16377 shutdown 2024-06-04T05:00:27.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.578+0000 7fb3269d6740 1 -- 172.21.15.70:0/3215211542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56005077ba00 msgr2=0x56005077f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:27.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.578+0000 7fb3269d6740 1 --2- 172.21.15.70:0/3215211542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56005077ba00 0x56005077f500 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb314000f80 tx=0x7fb3140078f0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.578+0000 7fb3269d6740 1 -- 172.21.15.70:0/3215211542 shutdown_connections 2024-06-04T05:00:27.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.578+0000 7fb3269d6740 1 --2- 172.21.15.70:0/3215211542 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5600507f66e0 0x5600507f8b20 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.578+0000 7fb3269d6740 1 --2- 172.21.15.70:0/3215211542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56005077ba00 0x56005077f500 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.578+0000 7fb3269d6740 1 --2- 172.21.15.70:0/3215211542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560050771140 0x56005077efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.578+0000 7fb3057fa640 0 client.16377 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:27.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.578+0000 7fb3269d6740 1 --2- 172.21.15.70:0/3215211542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560050770820 0x560050783c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.578+0000 7fb3269d6740 1 -- 172.21.15.70:0/3215211542 >> 172.21.15.70:0/3215211542 conn(0x56005076cd30 msgr2=0x560050786680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:27.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.578+0000 7fb3269d6740 1 -- 172.21.15.70:0/3215211542 shutdown_connections 2024-06-04T05:00:27.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.578+0000 7fb3269d6740 1 -- 172.21.15.70:0/3215211542 wait complete. 2024-06-04T05:00:27.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.578+0000 7fb3269d6740 20 client.16377 trim_cache size 0 max 16384 2024-06-04T05:00:27.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:00:27.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:00:27.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:00:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.579+0000 7fb323cc5640 1 -- 172.21.15.70:0/3804443286 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3080b2a10 msgr2=0x7fb3080b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.579+0000 7fb323cc5640 1 --2- 172.21.15.70:0/3804443286 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3080b2a10 0x7fb3080b4ed0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb3180057c0 tx=0x7fb3180058d0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.579+0000 7fb323cc5640 1 -- 172.21.15.70:0/3804443286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb31c107e30 msgr2=0x7fb31c19b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.579+0000 7fb323cc5640 1 --2- 172.21.15.70:0/3804443286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb31c107e30 0x7fb31c19b8d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00e9f0 tx=0x7fb30c00eeb0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.579+0000 7fb323cc5640 1 -- 172.21.15.70:0/3804443286 shutdown_connections 2024-06-04T05:00:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.579+0000 7fb323cc5640 1 --2- 172.21.15.70:0/3804443286 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3080b2a10 0x7fb3080b4ed0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.579+0000 7fb323cc5640 1 --2- 172.21.15.70:0/3804443286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb31c19fc50 0x7fb31c1a0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.579+0000 7fb323cc5640 1 --2- 172.21.15.70:0/3804443286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb31c107e30 0x7fb31c19b8d0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.579+0000 7fb323cc5640 1 --2- 172.21.15.70:0/3804443286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb31c1011c0 0x7fb31c19b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.579+0000 7fb323cc5640 1 -- 172.21.15.70:0/3804443286 >> 172.21.15.70:0/3804443286 conn(0x7fb31c077ee0 msgr2=0x7fb31c106230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.579+0000 7fb323cc5640 1 -- 172.21.15.70:0/3804443286 shutdown_connections 2024-06-04T05:00:27.584 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:27.579+0000 7fb323cc5640 1 -- 172.21.15.70:0/3804443286 wait complete. 2024-06-04T05:00:27.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 3.06 MB/sec warmup 80 sec latency 4626.858 ms 2024-06-04T05:00:27.847 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:27 smithi070 ceph-mon[32631]: pgmap v101: 97 pgs: 97 active+clean; 38 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-06-04T05:00:27.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:27 smithi112 ceph-mon[36453]: pgmap v101: 97 pgs: 97 active+clean; 38 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-06-04T05:00:27.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:27 smithi098 ceph-mon[35569]: pgmap v101: 97 pgs: 97 active+clean; 38 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-06-04T05:00:28.714 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 3.02 MB/sec warmup 81 sec latency 5626.903 ms 2024-06-04T05:00:28.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.744+0000 7f5a72ece700 1 -- 172.21.15.70:0/2080489974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a6c1028c0 msgr2=0x7f5a6c110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:28.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.744+0000 7f5a72ece700 1 --2- 172.21.15.70:0/2080489974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a6c1028c0 0x7f5a6c110330 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f5a64009990 tx=0x7f5a64009ca0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.744+0000 7f5a72ece700 1 -- 172.21.15.70:0/2080489974 shutdown_connections 2024-06-04T05:00:28.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.744+0000 7f5a72ece700 1 --2- 172.21.15.70:0/2080489974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a6c1028c0 0x7f5a6c110330 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.744+0000 7f5a72ece700 1 --2- 172.21.15.70:0/2080489974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a6c108620 0x7f5a6c108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.744+0000 7f5a72ece700 1 --2- 172.21.15.70:0/2080489974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a6c101f00 0x7f5a6c102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.744+0000 7f5a72ece700 1 -- 172.21.15.70:0/2080489974 >> 172.21.15.70:0/2080489974 conn(0x7f5a6c077780 msgr2=0x7f5a6c077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:28.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.744+0000 7f5a72ece700 1 -- 172.21.15.70:0/2080489974 shutdown_connections 2024-06-04T05:00:28.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.744+0000 7f5a72ece700 1 -- 172.21.15.70:0/2080489974 wait complete. 2024-06-04T05:00:28.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.745+0000 7f5a72ece700 1 Processor -- start 2024-06-04T05:00:28.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.745+0000 7f5a72ece700 1 -- start start 2024-06-04T05:00:28.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.745+0000 7f5a72ece700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a6c101f00 0x7f5a6c19f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.745+0000 7f5a72ece700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a6c1028c0 0x7f5a6c19fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.745+0000 7f5a72ece700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a6c108620 0x7f5a6c199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.745+0000 7f5a72ece700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6c113730 con 0x7f5a6c1028c0 2024-06-04T05:00:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.745+0000 7f5a72ece700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6c1135b0 con 0x7f5a6c108620 2024-06-04T05:00:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.745+0000 7f5a72ece700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6c1138b0 con 0x7f5a6c101f00 2024-06-04T05:00:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.745+0000 7f5a63fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a6c1028c0 0x7f5a6c19fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.745+0000 7f5a63fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a6c1028c0 0x7f5a6c19fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59894/0 (socket says 172.21.15.70:59894) 2024-06-04T05:00:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.745+0000 7f5a63fff700 1 -- 172.21.15.70:0/1848962667 learned_addr learned my addr 172.21.15.70:0/1848962667 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.745+0000 7f5a7146b700 1 --2- 172.21.15.70:0/1848962667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a6c108620 0x7f5a6c199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.745+0000 7f5a70c6a700 1 --2- 172.21.15.70:0/1848962667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a6c101f00 0x7f5a6c19f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.746+0000 7f5a63fff700 1 -- 172.21.15.70:0/1848962667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a6c101f00 msgr2=0x7f5a6c19f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.746+0000 7f5a63fff700 1 --2- 172.21.15.70:0/1848962667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a6c101f00 0x7f5a6c19f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.746+0000 7f5a63fff700 1 -- 172.21.15.70:0/1848962667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a6c108620 msgr2=0x7f5a6c199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.746+0000 7f5a63fff700 1 --2- 172.21.15.70:0/1848962667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a6c108620 0x7f5a6c199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.746+0000 7f5a63fff700 1 -- 172.21.15.70:0/1848962667 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6c19a1f0 con 0x7f5a6c1028c0 2024-06-04T05:00:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.746+0000 7f5a63fff700 1 --2- 172.21.15.70:0/1848962667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a6c1028c0 0x7f5a6c19fe10 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c00b530 tx=0x7f5a5c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.746+0000 7f5a61ffb700 1 -- 172.21.15.70:0/1848962667 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c015020 con 0x7f5a6c1028c0 2024-06-04T05:00:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.746+0000 7f5a61ffb700 1 -- 172.21.15.70:0/1848962667 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a5c011cc0 con 0x7f5a6c1028c0 2024-06-04T05:00:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.746+0000 7f5a72ece700 1 -- 172.21.15.70:0/1848962667 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a6c19a4e0 con 0x7f5a6c1028c0 2024-06-04T05:00:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.746+0000 7f5a61ffb700 1 -- 172.21.15.70:0/1848962667 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c010dc0 con 0x7f5a6c1028c0 2024-06-04T05:00:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.746+0000 7f5a72ece700 1 -- 172.21.15.70:0/1848962667 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a6c1a6820 con 0x7f5a6c1028c0 2024-06-04T05:00:28.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.749+0000 7f5a61ffb700 1 -- 172.21.15.70:0/1848962667 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f5a5c018020 con 0x7f5a6c1028c0 2024-06-04T05:00:28.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.749+0000 7f5a61ffb700 1 --2- 172.21.15.70:0/1848962667 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5a540b2a70 0x7f5a540b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.749+0000 7f5a70c6a700 1 --2- 172.21.15.70:0/1848962667 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5a540b2a70 0x7f5a540b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.749+0000 7f5a61ffb700 1 -- 172.21.15.70:0/1848962667 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f5a5c0e1cc0 con 0x7f5a6c1028c0 2024-06-04T05:00:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.750+0000 7f5a70c6a700 1 --2- 172.21.15.70:0/1848962667 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5a540b2a70 0x7f5a540b4f30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c19abd0 tx=0x7f5a680094f0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.750+0000 7f5a7d763b80 1 Processor -- start 2024-06-04T05:00:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.750+0000 7f5a7d763b80 1 -- start start 2024-06-04T05:00:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.750+0000 7f5a7d763b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4f6860 0x55b2fb4f6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.750+0000 7f5a7d763b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f7210 0x55b2fb4fa940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.750+0000 7f5a7d763b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb4fb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.750+0000 7f5a7d763b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b2fb455c90 con 0x55b2fb4faf60 2024-06-04T05:00:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.750+0000 7f5a7d763b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b2fb445830 con 0x55b2fb4f6860 2024-06-04T05:00:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.750+0000 7f5a7d763b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b2fb449870 con 0x55b2fb4f7210 2024-06-04T05:00:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.750+0000 7f5a7146b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb4fb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.750+0000 7f5a7146b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb4fb3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59906/0 (socket says 172.21.15.70:59906) 2024-06-04T05:00:28.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.750+0000 7f5a7146b700 1 -- 172.21.15.70:0/2375288122 learned_addr learned my addr 172.21.15.70:0/2375288122 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:28.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.750+0000 7f5a63fff700 1 --2- 172.21.15.70:0/2375288122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f7210 0x55b2fb4fa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.750+0000 7f5a70c6a700 1 --2- 172.21.15.70:0/2375288122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4f6860 0x55b2fb4f6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.751+0000 7f5a63fff700 1 -- 172.21.15.70:0/2375288122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4f6860 msgr2=0x55b2fb4f6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:28.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.751+0000 7f5a63fff700 1 --2- 172.21.15.70:0/2375288122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4f6860 0x55b2fb4f6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.751+0000 7f5a63fff700 1 -- 172.21.15.70:0/2375288122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 msgr2=0x55b2fb4fb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:28.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.751+0000 7f5a63fff700 1 --2- 172.21.15.70:0/2375288122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb4fb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.751+0000 7f5a63fff700 1 -- 172.21.15.70:0/2375288122 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2fb3958d0 con 0x55b2fb4f7210 2024-06-04T05:00:28.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.751+0000 7f5a7146b700 1 --2- 172.21.15.70:0/2375288122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb4fb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:28.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.751+0000 7f5a63fff700 1 --2- 172.21.15.70:0/2375288122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f7210 0x55b2fb4fa940 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0df520 tx=0x7f5a5c0a88a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccbbf9bf5f285df server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:28.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.751+0000 7f5a52ffd700 1 -- 172.21.15.70:0/2375288122 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c0a90b0 con 0x55b2fb4f7210 2024-06-04T05:00:28.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.751+0000 7f5a52ffd700 1 -- 172.21.15.70:0/2375288122 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a5c0ad460 con 0x55b2fb4f7210 2024-06-04T05:00:28.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.751+0000 7f5a52ffd700 1 -- 172.21.15.70:0/2375288122 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c0a9570 con 0x55b2fb4f7210 2024-06-04T05:00:28.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.752+0000 7f5a7d763b80 1 -- 172.21.15.70:0/2375288122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f7210 msgr2=0x55b2fb4fa940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:28.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.752+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/2375288122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f7210 0x55b2fb4fa940 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0df520 tx=0x7f5a5c0a88a0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.752+0000 7f5a7d763b80 1 -- 172.21.15.70:0/2375288122 shutdown_connections 2024-06-04T05:00:28.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.752+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/2375288122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb4fb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.752+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/2375288122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4f6860 0x55b2fb4f6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.752+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/2375288122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f7210 0x55b2fb4fa940 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.752+0000 7f5a7d763b80 1 -- 172.21.15.70:0/2375288122 >> 172.21.15.70:0/2375288122 conn(0x55b2fb424080 msgr2=0x55b2fb4f0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:28.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.752+0000 7f5a7d763b80 1 -- 172.21.15.70:0/2375288122 shutdown_connections 2024-06-04T05:00:28.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.752+0000 7f5a7d763b80 1 -- 172.21.15.70:0/2375288122 wait complete. 2024-06-04T05:00:28.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.752+0000 7f5a7d763b80 1 -- start start 2024-06-04T05:00:28.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.752+0000 7f5a7d763b80 1 Processor -- start 2024-06-04T05:00:28.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a7d763b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f6860 0x55b2fb502440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a7d763b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4f7210 0x55b2fb4ff740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a7d763b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4faf60 0x55b2fb4ffc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a7d763b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b2fb34bfa0 con 0x55b2fb4f7210 2024-06-04T05:00:28.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a7d763b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b2fb3e77c0 con 0x55b2fb4faf60 2024-06-04T05:00:28.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a7d763b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b2fb503120 con 0x55b2fb4f6860 2024-06-04T05:00:28.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a7d763b80 10 client.0 _ll_register_callbacks cb 0x55b2fb3e0920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:28.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a63fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4f7210 0x55b2fb4ff740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a63fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4f7210 0x55b2fb4ff740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59920/0 (socket says 172.21.15.70:59920) 2024-06-04T05:00:28.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a63fff700 1 -- 172.21.15.70:0/3666665343 learned_addr learned my addr 172.21.15.70:0/3666665343 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:28.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a70c6a700 1 --2- 172.21.15.70:0/3666665343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f6860 0x55b2fb502440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a7146b700 1 --2- 172.21.15.70:0/3666665343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4faf60 0x55b2fb4ffc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a63fff700 1 -- 172.21.15.70:0/3666665343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f6860 msgr2=0x55b2fb502440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:28.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a63fff700 1 --2- 172.21.15.70:0/3666665343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f6860 0x55b2fb502440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a63fff700 1 -- 172.21.15.70:0/3666665343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4faf60 msgr2=0x55b2fb4ffc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:28.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a63fff700 1 --2- 172.21.15.70:0/3666665343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4faf60 0x55b2fb4ffc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a63fff700 1 -- 172.21.15.70:0/3666665343 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2fb3e4690 con 0x55b2fb4f7210 2024-06-04T05:00:28.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a63fff700 1 --2- 172.21.15.70:0/3666665343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4f7210 0x55b2fb4ff740 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0dfa10 tx=0x7f5a5c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ded1389378714a2e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:28.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.753+0000 7f5a4bfff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:28.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.754+0000 7f5a4bfff700 1 -- 172.21.15.70:0/3666665343 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c0c9070 con 0x55b2fb4f7210 2024-06-04T05:00:28.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.754+0000 7f5a4bfff700 1 -- 172.21.15.70:0/3666665343 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f5a5c0a56a0 con 0x55b2fb4f7210 2024-06-04T05:00:28.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.754+0000 7f5a7d763b80 1 -- 172.21.15.70:0/3666665343 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a5c0ad460 con 0x55b2fb4f7210 2024-06-04T05:00:28.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.754+0000 7f5a4bfff700 1 -- 172.21.15.70:0/3666665343 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5a5c08f1a0 con 0x55b2fb4f7210 2024-06-04T05:00:28.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.754+0000 7f5a7d763b80 10 client.16389 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:28.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.754+0000 7f5a7d763b80 1 -- 172.21.15.70:0/3666665343 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b2fb50d8b0 con 0x55b2fb4f7210 2024-06-04T05:00:28.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.754+0000 7f5a4bfff700 1 -- 172.21.15.70:0/3666665343 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a5c0ad460 con 0x55b2fb4f7210 2024-06-04T05:00:28.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.754+0000 7f5a4bfff700 1 -- 172.21.15.70:0/3666665343 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5a5c0b1a80 con 0x55b2fb4f7210 2024-06-04T05:00:28.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.754+0000 7f5a7d763b80 10 client.16389 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:28.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.754+0000 7f5a7d763b80 10 client.16389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:28.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.754+0000 7f5a7d763b80 20 client.16389 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:28.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.754+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/3666665343 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b2fb50fd80 0x55b2fb512170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.754+0000 7f5a7d763b80 4 client.16389 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:00:28.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.754+0000 7f5a7d763b80 1 -- 172.21.15.70:0/3666665343 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2fb4c29e0 con 0x55b2fb50fd80 2024-06-04T05:00:28.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.755+0000 7f5a70c6a700 1 --2- 172.21.15.70:0/3666665343 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b2fb50fd80 0x55b2fb512170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.755+0000 7f5a70c6a700 1 --2- 172.21.15.70:0/3666665343 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b2fb50fd80 0x55b2fb512170 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=209ce16fb4b5dd7b server_cookie=d36862cd86bf213e in_seq=0 out_seq=0 2024-06-04T05:00:28.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.755+0000 7f5a4bfff700 10 client.16389 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:28.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.756+0000 7f5a4bfff700 1 -- 172.21.15.70:0/3666665343 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b2fb4c29e0 con 0x55b2fb50fd80 2024-06-04T05:00:28.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.756+0000 7f5a4bfff700 10 client.16389 handle_command_reply: tid=0 2024-06-04T05:00:28.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.756+0000 7f5a7d763b80 1 client.16389 shutdown 2024-06-04T05:00:28.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.756+0000 7f5a7d763b80 1 -- 172.21.15.70:0/3666665343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4f7210 msgr2=0x55b2fb4ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:28.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.756+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/3666665343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4f7210 0x55b2fb4ff740 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0dfa10 tx=0x7f5a5c000bc0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.756+0000 7f5a7d763b80 1 -- 172.21.15.70:0/3666665343 shutdown_connections 2024-06-04T05:00:28.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.756+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/3666665343 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b2fb50fd80 0x55b2fb512170 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.756+0000 7f5a4bfff700 0 client.16389 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:28.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.756+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/3666665343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4faf60 0x55b2fb4ffc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.756+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/3666665343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f6860 0x55b2fb502440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.756+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/3666665343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4f7210 0x55b2fb4ff740 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.756+0000 7f5a7d763b80 1 -- 172.21.15.70:0/3666665343 >> 172.21.15.70:0/3666665343 conn(0x55b2fb424080 msgr2=0x55b2fb4f0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:28.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.756+0000 7f5a7d763b80 1 -- 172.21.15.70:0/3666665343 shutdown_connections 2024-06-04T05:00:28.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.756+0000 7f5a7d763b80 1 -- 172.21.15.70:0/3666665343 wait complete. 2024-06-04T05:00:28.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.756+0000 7f5a7d763b80 20 client.16389 trim_cache size 0 max 16384 2024-06-04T05:00:28.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.771+0000 7f5a7d763b80 1 Processor -- start 2024-06-04T05:00:28.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.771+0000 7f5a7d763b80 1 -- start start 2024-06-04T05:00:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.771+0000 7f5a7d763b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4f6860 0x55b2fb4fe610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.771+0000 7f5a7d763b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f7210 0x55b2fb4f0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.771+0000 7f5a7d763b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb4feff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.771+0000 7f5a7d763b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6c1abd70 con 0x55b2fb4faf60 2024-06-04T05:00:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.771+0000 7f5a7d763b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6c1ad910 con 0x55b2fb4f6860 2024-06-04T05:00:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.771+0000 7f5a7d763b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6c1ac440 con 0x55b2fb4f7210 2024-06-04T05:00:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.772+0000 7f5a7146b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb4feff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.772+0000 7f5a7146b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb4feff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59934/0 (socket says 172.21.15.70:59934) 2024-06-04T05:00:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.772+0000 7f5a7146b700 1 -- 172.21.15.70:0/1314391119 learned_addr learned my addr 172.21.15.70:0/1314391119 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.772+0000 7f5a70c6a700 1 --2- 172.21.15.70:0/1314391119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4f6860 0x55b2fb4fe610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.772+0000 7f5a63fff700 1 --2- 172.21.15.70:0/1314391119 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f7210 0x55b2fb4f0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.772+0000 7f5a7146b700 1 -- 172.21.15.70:0/1314391119 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f7210 msgr2=0x55b2fb4f0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.772+0000 7f5a7146b700 1 --2- 172.21.15.70:0/1314391119 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f7210 0x55b2fb4f0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.772+0000 7f5a7146b700 1 -- 172.21.15.70:0/1314391119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4f6860 msgr2=0x55b2fb4fe610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:28.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.772+0000 7f5a7146b700 1 --2- 172.21.15.70:0/1314391119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4f6860 0x55b2fb4fe610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.772+0000 7f5a7146b700 1 -- 172.21.15.70:0/1314391119 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2fb500680 con 0x55b2fb4faf60 2024-06-04T05:00:28.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.772+0000 7f5a63fff700 1 --2- 172.21.15.70:0/1314391119 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f7210 0x55b2fb4f0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:28.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.772+0000 7f5a7146b700 1 --2- 172.21.15.70:0/1314391119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb4feff0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5a640052f0 tx=0x7f5a64005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=747d00c9a38b167a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:28.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.772+0000 7f5a527fc700 1 -- 172.21.15.70:0/1314391119 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a64029070 con 0x55b2fb4faf60 2024-06-04T05:00:28.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.772+0000 7f5a527fc700 1 -- 172.21.15.70:0/1314391119 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a64005650 con 0x55b2fb4faf60 2024-06-04T05:00:28.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.772+0000 7f5a527fc700 1 -- 172.21.15.70:0/1314391119 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6400bd20 con 0x55b2fb4faf60 2024-06-04T05:00:28.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.773+0000 7f5a7d763b80 1 -- 172.21.15.70:0/1314391119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 msgr2=0x55b2fb4feff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:28.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.773+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/1314391119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb4feff0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5a640052f0 tx=0x7f5a64005040 comp rx=0 tx=0).stop 2024-06-04T05:00:28.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.773+0000 7f5a7d763b80 1 -- 172.21.15.70:0/1314391119 shutdown_connections 2024-06-04T05:00:28.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.773+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/1314391119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb4feff0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.773+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/1314391119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4f6860 0x55b2fb4fe610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.773+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/1314391119 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f7210 0x55b2fb4f0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.773+0000 7f5a7d763b80 1 -- 172.21.15.70:0/1314391119 >> 172.21.15.70:0/1314391119 conn(0x55b2fb424080 msgr2=0x55b2fb4fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:28.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.773+0000 7f5a7d763b80 1 -- 172.21.15.70:0/1314391119 shutdown_connections 2024-06-04T05:00:28.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.773+0000 7f5a7d763b80 1 -- 172.21.15.70:0/1314391119 wait complete. 2024-06-04T05:00:28.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.773+0000 7f5a7d763b80 1 -- start start 2024-06-04T05:00:28.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.773+0000 7f5a7d763b80 1 Processor -- start 2024-06-04T05:00:28.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a7d763b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f6860 0x55b2fb572d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a7d763b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4f7210 0x55b2fb5732b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a7d763b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb514c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a7d763b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6c1ad1e0 con 0x55b2fb4faf60 2024-06-04T05:00:28.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a7d763b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6c1acb10 con 0x55b2fb4f7210 2024-06-04T05:00:28.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a7d763b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6c1a9d10 con 0x55b2fb4f6860 2024-06-04T05:00:28.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a7d763b80 10 client.0 _ll_register_callbacks cb 0x55b2fb3e0920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:28.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a7146b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb514c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a63fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4f7210 0x55b2fb5732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a63fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4f7210 0x55b2fb5732b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45428/0 (socket says 172.21.15.70:45428) 2024-06-04T05:00:28.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a7146b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb514c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59936/0 (socket says 172.21.15.70:59936) 2024-06-04T05:00:28.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a63fff700 1 -- 172.21.15.70:0/2971710956 learned_addr learned my addr 172.21.15.70:0/2971710956 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:28.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a70c6a700 1 --2- 172.21.15.70:0/2971710956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f6860 0x55b2fb572d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a63fff700 1 -- 172.21.15.70:0/2971710956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f6860 msgr2=0x55b2fb572d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:28.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a63fff700 1 --2- 172.21.15.70:0/2971710956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f6860 0x55b2fb572d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a63fff700 1 -- 172.21.15.70:0/2971710956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 msgr2=0x55b2fb514c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:28.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a63fff700 1 --2- 172.21.15.70:0/2971710956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb514c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a63fff700 1 -- 172.21.15.70:0/2971710956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2fb515330 con 0x55b2fb4f7210 2024-06-04T05:00:28.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a70c6a700 1 --2- 172.21.15.70:0/2971710956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f6860 0x55b2fb572d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:28.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a7146b700 1 --2- 172.21.15.70:0/2971710956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb514c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:28.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a63fff700 1 --2- 172.21.15.70:0/2971710956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4f7210 0x55b2fb5732b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0a5130 tx=0x7f5a5c0dfe60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3002c72b4cb9899f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:28.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.774+0000 7f5a4bfff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:00:28.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.775+0000 7f5a4bfff700 1 -- 172.21.15.70:0/2971710956 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c0ad460 con 0x55b2fb4f7210 2024-06-04T05:00:28.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.775+0000 7f5a7d763b80 1 -- 172.21.15.70:0/2971710956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a64005650 con 0x55b2fb4f7210 2024-06-04T05:00:28.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.775+0000 7f5a4bfff700 1 -- 172.21.15.70:0/2971710956 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f5a5c0a56a0 con 0x55b2fb4f7210 2024-06-04T05:00:28.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.775+0000 7f5a4bfff700 1 -- 172.21.15.70:0/2971710956 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5a5c0a9520 con 0x55b2fb4f7210 2024-06-04T05:00:28.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.775+0000 7f5a7d763b80 10 client.25784 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:28.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.775+0000 7f5a7d763b80 1 -- 172.21.15.70:0/2971710956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b2fb57a5c0 con 0x55b2fb4f7210 2024-06-04T05:00:28.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.775+0000 7f5a4bfff700 1 -- 172.21.15.70:0/2971710956 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a64005650 con 0x55b2fb4f7210 2024-06-04T05:00:28.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.775+0000 7f5a4bfff700 1 -- 172.21.15.70:0/2971710956 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5a5c0a4a20 con 0x55b2fb4f7210 2024-06-04T05:00:28.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.775+0000 7f5a7d763b80 10 client.25784 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:28.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.775+0000 7f5a7d763b80 10 client.25784 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:28.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.775+0000 7f5a7d763b80 20 client.25784 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:28.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.775+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/2971710956 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b2fb50fd80 0x55b2fb57cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:28.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.775+0000 7f5a7d763b80 4 client.25784 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:00:28.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.775+0000 7f5a7d763b80 1 -- 172.21.15.70:0/2971710956 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b2fb457be0 con 0x55b2fb50fd80 2024-06-04T05:00:28.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.776+0000 7f5a70c6a700 1 --2- 172.21.15.70:0/2971710956 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b2fb50fd80 0x55b2fb57cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:28.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.776+0000 7f5a4bfff700 10 client.25784 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:28.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.776+0000 7f5a70c6a700 1 --2- 172.21.15.70:0/2971710956 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b2fb50fd80 0x55b2fb57cad0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a5ae4cac15b775f server_cookie=a185944e0b868f4 in_seq=0 out_seq=0 2024-06-04T05:00:28.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.776+0000 7f5a4bfff700 1 -- 172.21.15.70:0/2971710956 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b2fb457be0 con 0x55b2fb50fd80 2024-06-04T05:00:28.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.776+0000 7f5a4bfff700 10 client.25784 handle_command_reply: tid=0 2024-06-04T05:00:28.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.776+0000 7f5a7d763b80 1 client.25784 shutdown 2024-06-04T05:00:28.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.776+0000 7f5a7d763b80 1 -- 172.21.15.70:0/2971710956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4f7210 msgr2=0x55b2fb5732b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:28.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.776+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/2971710956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4f7210 0x55b2fb5732b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0a5130 tx=0x7f5a5c0dfe60 comp rx=0 tx=0).stop 2024-06-04T05:00:28.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.776+0000 7f5a7d763b80 1 -- 172.21.15.70:0/2971710956 shutdown_connections 2024-06-04T05:00:28.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.776+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/2971710956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2fb4faf60 0x55b2fb514c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.776+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/2971710956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2fb4f6860 0x55b2fb572d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.776+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/2971710956 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b2fb50fd80 0x55b2fb57cad0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.776+0000 7f5a7d763b80 1 --2- 172.21.15.70:0/2971710956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2fb4f7210 0x55b2fb5732b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.776+0000 7f5a4bfff700 0 client.25784 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:28.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.776+0000 7f5a7d763b80 1 -- 172.21.15.70:0/2971710956 >> 172.21.15.70:0/2971710956 conn(0x55b2fb424080 msgr2=0x55b2fb4fa140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:28.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.777+0000 7f5a7d763b80 1 -- 172.21.15.70:0/2971710956 shutdown_connections 2024-06-04T05:00:28.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.777+0000 7f5a7d763b80 1 -- 172.21.15.70:0/2971710956 wait complete. 2024-06-04T05:00:28.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.777+0000 7f5a7d763b80 20 client.25784 trim_cache size 0 max 16384 2024-06-04T05:00:28.784 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:00:28.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.777+0000 7f5a72ece700 1 -- 172.21.15.70:0/1848962667 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5a540b2a70 msgr2=0x7f5a540b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:28.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.777+0000 7f5a72ece700 1 --2- 172.21.15.70:0/1848962667 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5a540b2a70 0x7f5a540b4f30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c19abd0 tx=0x7f5a680094f0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.777+0000 7f5a72ece700 1 -- 172.21.15.70:0/1848962667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a6c1028c0 msgr2=0x7f5a6c19fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.777+0000 7f5a72ece700 1 --2- 172.21.15.70:0/1848962667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a6c1028c0 0x7f5a6c19fe10 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c00b530 tx=0x7f5a5c00b840 comp rx=0 tx=0).stop 2024-06-04T05:00:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.777+0000 7f5a72ece700 1 -- 172.21.15.70:0/1848962667 shutdown_connections 2024-06-04T05:00:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.777+0000 7f5a72ece700 1 --2- 172.21.15.70:0/1848962667 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5a540b2a70 0x7f5a540b4f30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.777+0000 7f5a72ece700 1 --2- 172.21.15.70:0/1848962667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a6c108620 0x7f5a6c199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.777+0000 7f5a72ece700 1 --2- 172.21.15.70:0/1848962667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a6c101f00 0x7f5a6c19f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.777+0000 7f5a72ece700 1 --2- 172.21.15.70:0/1848962667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a6c1028c0 0x7f5a6c19fe10 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.777+0000 7f5a72ece700 1 -- 172.21.15.70:0/1848962667 >> 172.21.15.70:0/1848962667 conn(0x7f5a6c077780 msgr2=0x7f5a6c100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.777+0000 7f5a72ece700 1 -- 172.21.15.70:0/1848962667 shutdown_connections 2024-06-04T05:00:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:28.777+0000 7f5a72ece700 1 -- 172.21.15.70:0/1848962667 wait complete. 2024-06-04T05:00:29.715 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.98 MB/sec warmup 82 sec latency 6627.253 ms 2024-06-04T05:00:29.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:29 smithi070 ceph-mon[32631]: pgmap v102: 97 pgs: 97 active+clean; 38 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 873 KiB/s wr, 103 op/s 2024-06-04T05:00:29.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:29 smithi112 ceph-mon[36453]: pgmap v102: 97 pgs: 97 active+clean; 38 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 873 KiB/s wr, 103 op/s 2024-06-04T05:00:29.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:29 smithi098 ceph-mon[35569]: pgmap v102: 97 pgs: 97 active+clean; 38 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 873 KiB/s wr, 103 op/s 2024-06-04T05:00:30.561 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:00:30.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:00:30.715 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.95 MB/sec warmup 83 sec latency 7627.318 ms 2024-06-04T05:00:30.818 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:00:31.715 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.91 MB/sec warmup 84 sec latency 8627.356 ms 2024-06-04T05:00:31.868 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:31 smithi070 ceph-mon[32631]: pgmap v103: 97 pgs: 97 active+clean; 38 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 873 KiB/s wr, 103 op/s 2024-06-04T05:00:31.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:31 smithi112 ceph-mon[36453]: pgmap v103: 97 pgs: 97 active+clean; 38 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 873 KiB/s wr, 103 op/s 2024-06-04T05:00:31.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:31 smithi098 ceph-mon[35569]: pgmap v103: 97 pgs: 97 active+clean; 38 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 873 KiB/s wr, 103 op/s 2024-06-04T05:00:32.715 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.88 MB/sec warmup 85 sec latency 9627.425 ms 2024-06-04T05:00:33.715 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.84 MB/sec warmup 86 sec latency 10627.514 ms 2024-06-04T05:00:33.926 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:33 smithi070 ceph-mon[32631]: pgmap v104: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 886 KiB/s wr, 112 op/s 2024-06-04T05:00:33.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:33 smithi112 ceph-mon[36453]: pgmap v104: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 886 KiB/s wr, 112 op/s 2024-06-04T05:00:33.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:33 smithi098 ceph-mon[35569]: pgmap v104: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 886 KiB/s wr, 112 op/s 2024-06-04T05:00:34.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.310+0000 7f1e012a8700 1 -- 172.21.15.70:0/835506087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dfc102a80 msgr2=0x7f1dfc102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:34.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.310+0000 7f1e012a8700 1 --2- 172.21.15.70:0/835506087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dfc102a80 0x7f1dfc102e60 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1dec0099c0 tx=0x7f1dec009cd0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.311+0000 7f1e012a8700 1 -- 172.21.15.70:0/835506087 shutdown_connections 2024-06-04T05:00:34.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.311+0000 7f1e012a8700 1 --2- 172.21.15.70:0/835506087 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1dfc1077d0 0x7f1dfc10f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.311+0000 7f1e012a8700 1 --2- 172.21.15.70:0/835506087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dfc102a80 0x7f1dfc102e60 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.311+0000 7f1e012a8700 1 --2- 172.21.15.70:0/835506087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dfc103430 0x7f1dfc1070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.311+0000 7f1e012a8700 1 -- 172.21.15.70:0/835506087 >> 172.21.15.70:0/835506087 conn(0x7f1dfc0793f0 msgr2=0x7f1dfc079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:34.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.311+0000 7f1e012a8700 1 -- 172.21.15.70:0/835506087 shutdown_connections 2024-06-04T05:00:34.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.311+0000 7f1e012a8700 1 -- 172.21.15.70:0/835506087 wait complete. 2024-06-04T05:00:34.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.311+0000 7f1e012a8700 1 Processor -- start 2024-06-04T05:00:34.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.311+0000 7f1e012a8700 1 -- start start 2024-06-04T05:00:34.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1e012a8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dfc102a80 0x7f1dfc104140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1e012a8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dfc103430 0x7f1dfc104680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1e012a8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1dfc1077d0 0x7f1dfc104bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1e012a8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc112560 con 0x7f1dfc102a80 2024-06-04T05:00:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1e012a8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1123e0 con 0x7f1dfc103430 2024-06-04T05:00:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1e012a8700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1126e0 con 0x7f1dfc1077d0 2024-06-04T05:00:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1dfaffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dfc102a80 0x7f1dfc104140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1dfaffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dfc102a80 0x7f1dfc104140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59952/0 (socket says 172.21.15.70:59952) 2024-06-04T05:00:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1dfaffd700 1 -- 172.21.15.70:0/1710965716 learned_addr learned my addr 172.21.15.70:0/1710965716 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1dfb7fe700 1 --2- 172.21.15.70:0/1710965716 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1dfc1077d0 0x7f1dfc104bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1dfa7fc700 1 --2- 172.21.15.70:0/1710965716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dfc103430 0x7f1dfc104680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1dfaffd700 1 -- 172.21.15.70:0/1710965716 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1dfc1077d0 msgr2=0x7f1dfc104bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1dfaffd700 1 --2- 172.21.15.70:0/1710965716 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1dfc1077d0 0x7f1dfc104bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1dfaffd700 1 -- 172.21.15.70:0/1710965716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dfc103430 msgr2=0x7f1dfc104680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1dfaffd700 1 --2- 172.21.15.70:0/1710965716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dfc103430 0x7f1dfc104680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1dfaffd700 1 -- 172.21.15.70:0/1710965716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dfc1a80f0 con 0x7f1dfc102a80 2024-06-04T05:00:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1dfa7fc700 1 --2- 172.21.15.70:0/1710965716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dfc103430 0x7f1dfc104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1dfaffd700 1 --2- 172.21.15.70:0/1710965716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dfc102a80 0x7f1dfc104140 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00b580 tx=0x7f1dec0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1df3fff700 1 -- 172.21.15.70:0/1710965716 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec027070 con 0x7f1dfc102a80 2024-06-04T05:00:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1e012a8700 1 -- 172.21.15.70:0/1710965716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dfc1a8380 con 0x7f1dfc102a80 2024-06-04T05:00:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1df3fff700 1 -- 172.21.15.70:0/1710965716 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1dec00fd30 con 0x7f1dfc102a80 2024-06-04T05:00:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1df3fff700 1 -- 172.21.15.70:0/1710965716 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec022e50 con 0x7f1dfc102a80 2024-06-04T05:00:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.312+0000 7f1e012a8700 1 -- 172.21.15.70:0/1710965716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dfc1a87d0 con 0x7f1dfc102a80 2024-06-04T05:00:34.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.315+0000 7f1df3fff700 1 -- 172.21.15.70:0/1710965716 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f1dec02a070 con 0x7f1dfc102a80 2024-06-04T05:00:34.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.315+0000 7f1df3fff700 1 --2- 172.21.15.70:0/1710965716 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1de80b2b90 0x7f1de80b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.315+0000 7f1dfa7fc700 1 --2- 172.21.15.70:0/1710965716 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1de80b2b90 0x7f1de80b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.316+0000 7f1df3fff700 1 -- 172.21.15.70:0/1710965716 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f1dec0e3270 con 0x7f1dfc102a80 2024-06-04T05:00:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.316+0000 7f1dfa7fc700 1 --2- 172.21.15.70:0/1710965716 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1de80b2b90 0x7f1de80b5050 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc102780 tx=0x7f1de4005f50 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.316+0000 7f1e0bb3db80 1 Processor -- start 2024-06-04T05:00:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.316+0000 7f1e0bb3db80 1 -- start start 2024-06-04T05:00:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.316+0000 7f1e0bb3db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd178e0 0x55ebefd17cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.316+0000 7f1e0bb3db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd18290 0x55ebefd1b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.316+0000 7f1e0bb3db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd1bfe0 0x55ebefd1c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.316+0000 7f1e0bb3db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ebefc76d10 con 0x55ebefd178e0 2024-06-04T05:00:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.316+0000 7f1e0bb3db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ebefc668b0 con 0x55ebefd18290 2024-06-04T05:00:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.316+0000 7f1e0bb3db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ebefc6a8f0 con 0x55ebefd1bfe0 2024-06-04T05:00:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.316+0000 7f1dfaffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd178e0 0x55ebefd17cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.317+0000 7f1dfaffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd178e0 0x55ebefd17cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59958/0 (socket says 172.21.15.70:59958) 2024-06-04T05:00:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.317+0000 7f1dfaffd700 1 -- 172.21.15.70:0/472289224 learned_addr learned my addr 172.21.15.70:0/472289224 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.317+0000 7f1dfa7fc700 1 --2- 172.21.15.70:0/472289224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd18290 0x55ebefd1b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.317+0000 7f1dfb7fe700 1 --2- 172.21.15.70:0/472289224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd1bfe0 0x55ebefd1c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.317+0000 7f1dfaffd700 1 -- 172.21.15.70:0/472289224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd1bfe0 msgr2=0x55ebefd1c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:34.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.317+0000 7f1dfaffd700 1 --2- 172.21.15.70:0/472289224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd1bfe0 0x55ebefd1c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.317+0000 7f1dfaffd700 1 -- 172.21.15.70:0/472289224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd18290 msgr2=0x55ebefd1b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:34.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.317+0000 7f1dfaffd700 1 --2- 172.21.15.70:0/472289224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd18290 0x55ebefd1b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.317+0000 7f1dfaffd700 1 -- 172.21.15.70:0/472289224 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebefbb6950 con 0x55ebefd178e0 2024-06-04T05:00:34.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.317+0000 7f1dfaffd700 1 --2- 172.21.15.70:0/472289224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd178e0 0x55ebefd17cc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f1dec0aa570 tx=0x7f1dec0aa020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd58f821c2cbc89 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:34.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.317+0000 7f1df17fa700 1 -- 172.21.15.70:0/472289224 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec0aace0 con 0x55ebefd178e0 2024-06-04T05:00:34.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.317+0000 7f1df17fa700 1 -- 172.21.15.70:0/472289224 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1dec0aa5e0 con 0x55ebefd178e0 2024-06-04T05:00:34.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.317+0000 7f1df17fa700 1 -- 172.21.15.70:0/472289224 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec0b2f20 con 0x55ebefd178e0 2024-06-04T05:00:34.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.318+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/472289224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd178e0 msgr2=0x55ebefd17cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:34.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.318+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/472289224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd178e0 0x55ebefd17cc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f1dec0aa570 tx=0x7f1dec0aa020 comp rx=0 tx=0).stop 2024-06-04T05:00:34.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.318+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/472289224 shutdown_connections 2024-06-04T05:00:34.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.318+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/472289224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd1bfe0 0x55ebefd1c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.318+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/472289224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd178e0 0x55ebefd17cc0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.318+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/472289224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd18290 0x55ebefd1b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.318+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/472289224 >> 172.21.15.70:0/472289224 conn(0x55ebefc45100 msgr2=0x55ebefd11f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:34.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.318+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/472289224 shutdown_connections 2024-06-04T05:00:34.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.318+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/472289224 wait complete. 2024-06-04T05:00:34.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.318+0000 7f1e0bb3db80 1 -- start start 2024-06-04T05:00:34.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.318+0000 7f1e0bb3db80 1 Processor -- start 2024-06-04T05:00:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1e0bb3db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd178e0 0x55ebefd23530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1e0bb3db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd18290 0x55ebefd20830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1e0bb3db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd1bfe0 0x55ebefd20d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1e0bb3db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ebefb6d020 con 0x55ebefd18290 2024-06-04T05:00:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1e0bb3db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ebefc08840 con 0x55ebefd1bfe0 2024-06-04T05:00:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1e0bb3db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ebefd23f60 con 0x55ebefd178e0 2024-06-04T05:00:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1e0bb3db80 10 client.0 _ll_register_callbacks cb 0x55ebefc019a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1dfa7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd18290 0x55ebefd20830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1dfa7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd18290 0x55ebefd20830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59972/0 (socket says 172.21.15.70:59972) 2024-06-04T05:00:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1dfa7fc700 1 -- 172.21.15.70:0/1628977106 learned_addr learned my addr 172.21.15.70:0/1628977106 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1dfaffd700 1 --2- 172.21.15.70:0/1628977106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd178e0 0x55ebefd23530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1dfb7fe700 1 --2- 172.21.15.70:0/1628977106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd1bfe0 0x55ebefd20d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1dfa7fc700 1 -- 172.21.15.70:0/1628977106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd178e0 msgr2=0x55ebefd23530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1dfa7fc700 1 --2- 172.21.15.70:0/1628977106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd178e0 0x55ebefd23530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1dfa7fc700 1 -- 172.21.15.70:0/1628977106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd1bfe0 msgr2=0x55ebefd20d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1dfa7fc700 1 --2- 172.21.15.70:0/1628977106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd1bfe0 0x55ebefd20d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1dfa7fc700 1 -- 172.21.15.70:0/1628977106 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebefc05710 con 0x55ebefd18290 2024-06-04T05:00:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1dfa7fc700 1 --2- 172.21.15.70:0/1628977106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd18290 0x55ebefd20830 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc06ba80 tx=0x7f1de4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9585081f4b9edc85 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1de27fc700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1de27fc700 1 -- 172.21.15.70:0/1628977106 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de4004a30 con 0x55ebefd18290 2024-06-04T05:00:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.319+0000 7f1de27fc700 1 -- 172.21.15.70:0/1628977106 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f1de4004bd0 con 0x55ebefd18290 2024-06-04T05:00:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.320+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/1628977106 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dec0aa5e0 con 0x55ebefd18290 2024-06-04T05:00:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.320+0000 7f1de27fc700 1 -- 172.21.15.70:0/1628977106 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1de4004dd0 con 0x55ebefd18290 2024-06-04T05:00:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.320+0000 7f1e0bb3db80 10 client.16407 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.320+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/1628977106 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ebefd2e730 con 0x55ebefd18290 2024-06-04T05:00:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.320+0000 7f1de27fc700 1 -- 172.21.15.70:0/1628977106 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1dec0aa5e0 con 0x55ebefd18290 2024-06-04T05:00:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.320+0000 7f1de27fc700 1 -- 172.21.15.70:0/1628977106 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f1de401dad0 con 0x55ebefd18290 2024-06-04T05:00:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.320+0000 7f1e0bb3db80 10 client.16407 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:34.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.320+0000 7f1e0bb3db80 10 client.16407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:34.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.320+0000 7f1e0bb3db80 20 client.16407 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:34.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.320+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/1628977106 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ebefd30e10 0x55ebefd33200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.320+0000 7f1e0bb3db80 4 client.16407 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:00:34.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.320+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/1628977106 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebefce3a60 con 0x55ebefd30e10 2024-06-04T05:00:34.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.321+0000 7f1dfaffd700 1 --2- 172.21.15.70:0/1628977106 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ebefd30e10 0x55ebefd33200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.321+0000 7f1de27fc700 10 client.16407 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:34.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.321+0000 7f1dfaffd700 1 --2- 172.21.15.70:0/1628977106 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ebefd30e10 0x55ebefd33200 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a6415a8f8046ca2 server_cookie=d9dc5efc3e521a73 in_seq=0 out_seq=0 2024-06-04T05:00:34.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.322+0000 7f1de27fc700 1 -- 172.21.15.70:0/1628977106 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ebefce3a60 con 0x55ebefd30e10 2024-06-04T05:00:34.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.322+0000 7f1de27fc700 10 client.16407 handle_command_reply: tid=0 2024-06-04T05:00:34.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.322+0000 7f1e0bb3db80 1 client.16407 shutdown 2024-06-04T05:00:34.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.322+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/1628977106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd18290 msgr2=0x55ebefd20830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:34.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.322+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/1628977106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd18290 0x55ebefd20830 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc06ba80 tx=0x7f1de4004750 comp rx=0 tx=0).stop 2024-06-04T05:00:34.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.322+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/1628977106 shutdown_connections 2024-06-04T05:00:34.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.322+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/1628977106 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ebefd30e10 0x55ebefd33200 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.322+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/1628977106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd1bfe0 0x55ebefd20d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.322+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/1628977106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd178e0 0x55ebefd23530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.322+0000 7f1de27fc700 0 client.16407 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:34.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.322+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/1628977106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd18290 0x55ebefd20830 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.322+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/1628977106 >> 172.21.15.70:0/1628977106 conn(0x55ebefc45100 msgr2=0x55ebefd11590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:34.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.322+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/1628977106 shutdown_connections 2024-06-04T05:00:34.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.322+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/1628977106 wait complete. 2024-06-04T05:00:34.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.322+0000 7f1e0bb3db80 20 client.16407 trim_cache size 0 max 16384 2024-06-04T05:00:34.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.342+0000 7f1e0bb3db80 1 Processor -- start 2024-06-04T05:00:34.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.342+0000 7f1e0bb3db80 1 -- start start 2024-06-04T05:00:34.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.342+0000 7f1e0bb3db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd178e0 0x55ebefd2cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.342+0000 7f1e0bb3db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd18290 0x55ebefd23070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.342+0000 7f1e0bb3db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd1bfe0 0x55ebefd38ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1e0bb3db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1aff10 con 0x55ebefd1bfe0 2024-06-04T05:00:34.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1e0bb3db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1b1ab0 con 0x55ebefd18290 2024-06-04T05:00:34.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1e0bb3db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1b05e0 con 0x55ebefd178e0 2024-06-04T05:00:34.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1dfb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd1bfe0 0x55ebefd38ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1dfb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd1bfe0 0x55ebefd38ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59974/0 (socket says 172.21.15.70:59974) 2024-06-04T05:00:34.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1dfb7fe700 1 -- 172.21.15.70:0/1633476625 learned_addr learned my addr 172.21.15.70:0/1633476625 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:34.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1dfb7fe700 1 -- 172.21.15.70:0/1633476625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd178e0 msgr2=0x55ebefd2cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:34.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1dfaffd700 1 --2- 172.21.15.70:0/1633476625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd178e0 0x55ebefd2cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1dfb7fe700 1 --2- 172.21.15.70:0/1633476625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd178e0 0x55ebefd2cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1dfa7fc700 1 --2- 172.21.15.70:0/1633476625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd18290 0x55ebefd23070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1dfb7fe700 1 -- 172.21.15.70:0/1633476625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd18290 msgr2=0x55ebefd23070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:34.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1dfb7fe700 1 --2- 172.21.15.70:0/1633476625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd18290 0x55ebefd23070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1dfb7fe700 1 -- 172.21.15.70:0/1633476625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebefd33860 con 0x55ebefd1bfe0 2024-06-04T05:00:34.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1dfaffd700 1 --2- 172.21.15.70:0/1633476625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd178e0 0x55ebefd2cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:00:34.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1dfb7fe700 1 --2- 172.21.15.70:0/1633476625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd1bfe0 0x55ebefd38ab0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f1df40099f0 tx=0x7f1df4009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25cf9b022a557ea9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:34.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1df0ff9700 1 -- 172.21.15.70:0/1633476625 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df401a070 con 0x55ebefd1bfe0 2024-06-04T05:00:34.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1df0ff9700 1 -- 172.21.15.70:0/1633476625 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1df4007500 con 0x55ebefd1bfe0 2024-06-04T05:00:34.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.343+0000 7f1df0ff9700 1 -- 172.21.15.70:0/1633476625 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4016690 con 0x55ebefd1bfe0 2024-06-04T05:00:34.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.344+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/1633476625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd1bfe0 msgr2=0x55ebefd38ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:34.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.344+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/1633476625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd1bfe0 0x55ebefd38ab0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f1df40099f0 tx=0x7f1df4009a20 comp rx=0 tx=0).stop 2024-06-04T05:00:34.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.344+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/1633476625 shutdown_connections 2024-06-04T05:00:34.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.344+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/1633476625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd1bfe0 0x55ebefd38ab0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.344+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/1633476625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd178e0 0x55ebefd2cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.344+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/1633476625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd18290 0x55ebefd23070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.344+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/1633476625 >> 172.21.15.70:0/1633476625 conn(0x55ebefc45100 msgr2=0x55ebefd2c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:34.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.344+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/1633476625 shutdown_connections 2024-06-04T05:00:34.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.344+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/1633476625 wait complete. 2024-06-04T05:00:34.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.344+0000 7f1e0bb3db80 1 -- start start 2024-06-04T05:00:34.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.344+0000 7f1e0bb3db80 1 Processor -- start 2024-06-04T05:00:34.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1e0bb3db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd178e0 0x55ebefd280f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1e0bb3db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd18290 0x55ebefd28630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1e0bb3db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd1bfe0 0x55ebefd28b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1e0bb3db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1b1380 con 0x55ebefd18290 2024-06-04T05:00:34.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1e0bb3db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1b0cb0 con 0x55ebefd1bfe0 2024-06-04T05:00:34.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1e0bb3db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1adeb0 con 0x55ebefd178e0 2024-06-04T05:00:34.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1e0bb3db80 10 client.0 _ll_register_callbacks cb 0x55ebefc019a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:34.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1dfa7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd18290 0x55ebefd28630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1dfa7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd18290 0x55ebefd28630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59978/0 (socket says 172.21.15.70:59978) 2024-06-04T05:00:34.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1dfa7fc700 1 -- 172.21.15.70:0/2190368523 learned_addr learned my addr 172.21.15.70:0/2190368523 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:34.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1dfb7fe700 1 --2- 172.21.15.70:0/2190368523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd1bfe0 0x55ebefd28b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1dfaffd700 1 --2- 172.21.15.70:0/2190368523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd178e0 0x55ebefd280f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1dfa7fc700 1 -- 172.21.15.70:0/2190368523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd178e0 msgr2=0x55ebefd280f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:34.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1dfa7fc700 1 --2- 172.21.15.70:0/2190368523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd178e0 0x55ebefd280f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1dfa7fc700 1 -- 172.21.15.70:0/2190368523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd1bfe0 msgr2=0x55ebefd28b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:34.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1dfa7fc700 1 --2- 172.21.15.70:0/2190368523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd1bfe0 0x55ebefd28b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1dfa7fc700 1 -- 172.21.15.70:0/2190368523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebefd9bb30 con 0x55ebefd18290 2024-06-04T05:00:34.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1dfaffd700 1 --2- 172.21.15.70:0/2190368523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd178e0 0x55ebefd280f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:34.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1dfa7fc700 1 --2- 172.21.15.70:0/2190368523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd18290 0x55ebefd28630 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1de401df40 tx=0x7f1de401e7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d56713d16f0488f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:34.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1de27fc700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:34.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1de27fc700 1 -- 172.21.15.70:0/2190368523 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de403d060 con 0x55ebefd18290 2024-06-04T05:00:34.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1de27fc700 1 -- 172.21.15.70:0/2190368523 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f1de401ece0 con 0x55ebefd18290 2024-06-04T05:00:34.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.345+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/2190368523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1df4007500 con 0x55ebefd18290 2024-06-04T05:00:34.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.346+0000 7f1de27fc700 1 -- 172.21.15.70:0/2190368523 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1de401d6a0 con 0x55ebefd18290 2024-06-04T05:00:34.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.346+0000 7f1e0bb3db80 10 client.16413 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:34.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.346+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/2190368523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ebefd9cde0 con 0x55ebefd18290 2024-06-04T05:00:34.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.346+0000 7f1de27fc700 1 -- 172.21.15.70:0/2190368523 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1df4007500 con 0x55ebefd18290 2024-06-04T05:00:34.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.346+0000 7f1de27fc700 1 -- 172.21.15.70:0/2190368523 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f1de4005200 con 0x55ebefd18290 2024-06-04T05:00:34.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.346+0000 7f1e0bb3db80 10 client.16413 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.346+0000 7f1e0bb3db80 10 client.16413 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.346+0000 7f1e0bb3db80 20 client.16413 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.346+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/2190368523 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ebefd9d430 0x55ebefd9f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.346+0000 7f1e0bb3db80 4 client.16413 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:00:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.346+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/2190368523 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ebefc78c60 con 0x55ebefd9d430 2024-06-04T05:00:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.346+0000 7f1dfaffd700 1 --2- 172.21.15.70:0/2190368523 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ebefd9d430 0x55ebefd9f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.347+0000 7f1dfaffd700 1 --2- 172.21.15.70:0/2190368523 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ebefd9d430 0x55ebefd9f850 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49f279ac745bc3d0 server_cookie=9a51f0c489e69023 in_seq=0 out_seq=0 2024-06-04T05:00:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.347+0000 7f1de27fc700 10 client.16413 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.348+0000 7f1de27fc700 1 -- 172.21.15.70:0/2190368523 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ebefc78c60 con 0x55ebefd9d430 2024-06-04T05:00:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.348+0000 7f1de27fc700 10 client.16413 handle_command_reply: tid=0 2024-06-04T05:00:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.348+0000 7f1e0bb3db80 1 client.16413 shutdown 2024-06-04T05:00:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.348+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/2190368523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd18290 msgr2=0x55ebefd28630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.348+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/2190368523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd18290 0x55ebefd28630 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1de401df40 tx=0x7f1de401e7e0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.348+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/2190368523 shutdown_connections 2024-06-04T05:00:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.348+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/2190368523 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ebefd9d430 0x55ebefd9f850 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.348+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/2190368523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebefd1bfe0 0x55ebefd28b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.348+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/2190368523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebefd178e0 0x55ebefd280f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.348+0000 7f1e0bb3db80 1 --2- 172.21.15.70:0/2190368523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebefd18290 0x55ebefd28630 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.348+0000 7f1de27fc700 0 client.16413 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.348+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/2190368523 >> 172.21.15.70:0/2190368523 conn(0x55ebefc45100 msgr2=0x55ebefd2cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.348+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/2190368523 shutdown_connections 2024-06-04T05:00:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.348+0000 7f1e0bb3db80 1 -- 172.21.15.70:0/2190368523 wait complete. 2024-06-04T05:00:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.348+0000 7f1e0bb3db80 20 client.16413 trim_cache size 0 max 16384 2024-06-04T05:00:34.355 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:00:34.356 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:00:34.356 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "e855c38a-3834-4eb6-85c7-9525e06df599", 2024-06-04T05:00:34.356 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:00:34.356 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:00:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.349+0000 7f1e012a8700 1 -- 172.21.15.70:0/1710965716 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1de80b2b90 msgr2=0x7f1de80b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.349+0000 7f1e012a8700 1 --2- 172.21.15.70:0/1710965716 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1de80b2b90 0x7f1de80b5050 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc102780 tx=0x7f1de4005f50 comp rx=0 tx=0).stop 2024-06-04T05:00:34.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.349+0000 7f1e012a8700 1 -- 172.21.15.70:0/1710965716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dfc102a80 msgr2=0x7f1dfc104140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:34.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.349+0000 7f1e012a8700 1 --2- 172.21.15.70:0/1710965716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dfc102a80 0x7f1dfc104140 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00b580 tx=0x7f1dec0047b0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.349+0000 7f1e012a8700 1 -- 172.21.15.70:0/1710965716 shutdown_connections 2024-06-04T05:00:34.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.349+0000 7f1e012a8700 1 --2- 172.21.15.70:0/1710965716 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1de80b2b90 0x7f1de80b5050 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.349+0000 7f1e012a8700 1 --2- 172.21.15.70:0/1710965716 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1dfc1077d0 0x7f1dfc104bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.349+0000 7f1e012a8700 1 --2- 172.21.15.70:0/1710965716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dfc102a80 0x7f1dfc104140 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.349+0000 7f1e012a8700 1 --2- 172.21.15.70:0/1710965716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dfc103430 0x7f1dfc104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:34.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.349+0000 7f1e012a8700 1 -- 172.21.15.70:0/1710965716 >> 172.21.15.70:0/1710965716 conn(0x7f1dfc0793f0 msgr2=0x7f1dfc06c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:34.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.349+0000 7f1e012a8700 1 -- 172.21.15.70:0/1710965716 shutdown_connections 2024-06-04T05:00:34.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:34.349+0000 7f1e012a8700 1 -- 172.21.15.70:0/1710965716 wait complete. 2024-06-04T05:00:34.620 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:00:34 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:00:34.357+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000215 [...5,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL/ auth{1=1} v7518 ap=4 f(v238 m2024-06-04T05:00:18.749143+0000 11=11+0) n(v8 rc2024-06-04T05:00:18.749143+0000 b2861056 12=11+1) old_inodes=1 (inest sync->mix g=1) (ipolicy sync r=1) (iquiesce lock x=1 by request(mds.0:61 nref=3)) caps={15231=pAsLsXs/-@6102} | dirtyscattered=0 request=1 lock=2 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55d6deff1b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000215:[<0x10000000009/COREL v50>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x10000000215:[<0x10000000009/COREL v7518>,<0x10000000008/~dmtmp v5276>,<0x10000000007/client0 v217>,<0x10000000006/clients v200>,<0x10000000005/tmp v188>,<0x10000000003/client.0 v179>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v193>,<0x10000000001/sv_0 v204>,<0x10000000000/qa v202>,<0x1/volumes v197>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:00:18.738354+0000 12=12+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:00:18.749143+0000 b2861056 12=11+1)","memory_value.dirstat":"f(v238 m2024-06-04T05:00:18.749143+0000 11=11+0)","memory_value.rstat":"n(v8 rc2024-06-04T05:00:18.749143+0000 b2861056 12=11+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:00:34.620 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:00:34 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:00:34.360+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000221 [...5,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/WORDPRO/ auth{1=1} v7520 ap=4 f(v349 m2024-06-04T05:00:23.074491+0000 9=9+0) n(v8 rc2024-06-04T05:00:23.074491+0000 b2295936 10=9+1) old_inodes=1 (inest sync->mix g=1) (ipolicy sync r=1) (iquiesce lock x=1 by request(mds.0:65 nref=3)) caps={15231=pAsLsXs/-@6529} | dirtyscattered=0 request=1 lock=2 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55d6dea56100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000221:[<0x10000000009/WORDPRO v58>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x10000000221:[<0x10000000009/WORDPRO v7520>,<0x10000000008/~dmtmp v5276>,<0x10000000007/client0 v217>,<0x10000000006/clients v200>,<0x10000000005/tmp v188>,<0x10000000003/client.0 v179>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v193>,<0x10000000001/sv_0 v204>,<0x10000000000/qa v202>,<0x1/volumes v197>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:00:23.060748+0000 10=10+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:00:23.074491+0000 b2295936 10=9+1)","memory_value.dirstat":"f(v349 m2024-06-04T05:00:23.074491+0000 9=9+0)","memory_value.rstat":"n(v8 rc2024-06-04T05:00:23.074491+0000 b2295936 10=9+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:00:34.620 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:00:34 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:00:34.363+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000229 [...5,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PARADOX/ auth{1=1,2=1,3=1,4=1} v7522 ap=7 f(v350 m2024-06-04T05:00:21.597108+0000 24=24+0) n(v8 rc2024-06-04T05:00:21.597108+0000 b5230239 25=24+1) old_inodes=1 (inest sync->mix g=1,2,3,4) (ipolicy sync r=1) (iquiesce lock x=1 by request(mds.0:66 nref=3)) caps={15231=pAsLsXs/-@7298} | dirtyscattered=0 request=1 lock=2 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55d6df003080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000229:[<0x10000000009/PARADOX v108>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x10000000229:[<0x10000000009/PARADOX v7522>,<0x10000000008/~dmtmp v5276>,<0x10000000007/client0 v217>,<0x10000000006/clients v200>,<0x10000000005/tmp v188>,<0x10000000003/client.0 v179>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v193>,<0x10000000001/sv_0 v204>,<0x10000000000/qa v202>,<0x1/volumes v197>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:00:21.582874+0000 25=25+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:00:21.597108+0000 b5230239 25=24+1)","memory_value.dirstat":"f(v350 m2024-06-04T05:00:21.597108+0000 24=24+0)","memory_value.rstat":"n(v8 rc2024-06-04T05:00:21.597108+0000 b5230239 25=24+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:00:34.715 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.81 MB/sec warmup 87 sec latency 11627.612 ms 2024-06-04T05:00:34.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:34 smithi112 ceph-mon[36453]: pgmap v105: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 373 KiB/s wr, 45 op/s 2024-06-04T05:00:34.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:34 smithi098 ceph-mon[35569]: pgmap v105: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 373 KiB/s wr, 45 op/s 2024-06-04T05:00:35.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:34 smithi070 ceph-mon[32631]: pgmap v105: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 373 KiB/s wr, 45 op/s 2024-06-04T05:00:35.304 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:00:35.548 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:00:35.715 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.78 MB/sec warmup 88 sec latency 12627.701 ms 2024-06-04T05:00:35.926 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:35 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:00:35.926 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:35 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:35.926 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:35 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:00:35.926 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:35 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:00:35.926 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:35 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:35.926 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:35 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:00:35.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:35 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:00:35.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:35 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:35.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:35 smithi070 ceph-mon[32631]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.i log and `damage ls` output for details 2024-06-04T05:00:35.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:35 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:00:35.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:35 smithi070 ceph-mon[32631]: Scrub error on inode 0x10000000221 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/WORDPRO) see mds.i log and `damage ls` output for details 2024-06-04T05:00:35.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:35 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:35.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:35 smithi070 ceph-mon[32631]: Scrub error on inode 0x10000000229 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PARADOX) see mds.i log and `damage ls` output for details 2024-06-04T05:00:35.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:35 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:35.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:35 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:35 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:00:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:35 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:35 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:00:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:35 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:00:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:35 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:35 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:00:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:35 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:00:35.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:35 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:35.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:35 smithi112 ceph-mon[36453]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.i log and `damage ls` output for details 2024-06-04T05:00:35.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:35 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:00:35.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:35 smithi112 ceph-mon[36453]: Scrub error on inode 0x10000000221 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/WORDPRO) see mds.i log and `damage ls` output for details 2024-06-04T05:00:35.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:35 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:35.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:35 smithi112 ceph-mon[36453]: Scrub error on inode 0x10000000229 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PARADOX) see mds.i log and `damage ls` output for details 2024-06-04T05:00:35.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:35 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:35.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:35 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:35.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:35 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:00:35.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:35 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:35.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:35 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:00:35.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:35 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:00:35.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:35 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:35.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:35 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:00:35.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:35 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:00:35.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:35 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:35.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:35 smithi098 ceph-mon[35569]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.i log and `damage ls` output for details 2024-06-04T05:00:35.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:35 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:00:35.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:35 smithi098 ceph-mon[35569]: Scrub error on inode 0x10000000221 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/WORDPRO) see mds.i log and `damage ls` output for details 2024-06-04T05:00:35.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:35 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:35.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:35 smithi098 ceph-mon[35569]: Scrub error on inode 0x10000000229 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PARADOX) see mds.i log and `damage ls` output for details 2024-06-04T05:00:35.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:35 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:35.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:35 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:00:36.715 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.75 MB/sec warmup 89 sec latency 13627.810 ms 2024-06-04T05:00:36.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:36 smithi112 ceph-mon[36453]: pgmap v106: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 376 KiB/s wr, 57 op/s 2024-06-04T05:00:36.949 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:36 smithi070 ceph-mon[32631]: pgmap v106: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 376 KiB/s wr, 57 op/s 2024-06-04T05:00:36.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:36 smithi098 ceph-mon[35569]: pgmap v106: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 376 KiB/s wr, 57 op/s 2024-06-04T05:00:37.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.383+0000 7effc17fd700 1 -- 172.21.15.70:0/4082170179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effbc101b30 msgr2=0x7effbc101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:37.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.383+0000 7effc17fd700 1 --2- 172.21.15.70:0/4082170179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effbc101b30 0x7effbc101fb0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7effb00099c0 tx=0x7effb0009cd0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.383+0000 7effc17fd700 1 -- 172.21.15.70:0/4082170179 shutdown_connections 2024-06-04T05:00:37.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.383+0000 7effc17fd700 1 --2- 172.21.15.70:0/4082170179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effbc1024f0 0x7effbc1103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.383+0000 7effc17fd700 1 --2- 172.21.15.70:0/4082170179 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7effbc109520 0x7effbc109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.383+0000 7effc17fd700 1 --2- 172.21.15.70:0/4082170179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effbc101b30 0x7effbc101fb0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.383+0000 7effc17fd700 1 -- 172.21.15.70:0/4082170179 >> 172.21.15.70:0/4082170179 conn(0x7effbc078ab0 msgr2=0x7effbc0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:37.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.383+0000 7effc17fd700 1 -- 172.21.15.70:0/4082170179 shutdown_connections 2024-06-04T05:00:37.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.383+0000 7effc17fd700 1 -- 172.21.15.70:0/4082170179 wait complete. 2024-06-04T05:00:37.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.384+0000 7effc17fd700 1 Processor -- start 2024-06-04T05:00:37.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.384+0000 7effc17fd700 1 -- start start 2024-06-04T05:00:37.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.384+0000 7effc17fd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effbc101b30 0x7effbc102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.384+0000 7effc17fd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effbc1024f0 0x7effbc103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.384+0000 7effc17fd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7effbc109520 0x7effbc103790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.384+0000 7effc17fd700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7effbc113690 con 0x7effbc101b30 2024-06-04T05:00:37.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.384+0000 7effc17fd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7effbc113510 con 0x7effbc1024f0 2024-06-04T05:00:37.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.384+0000 7effc17fd700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7effbc113810 con 0x7effbc109520 2024-06-04T05:00:37.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.384+0000 7effbaffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effbc101b30 0x7effbc102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:37.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.384+0000 7effbaffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effbc101b30 0x7effbc102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56486/0 (socket says 172.21.15.70:56486) 2024-06-04T05:00:37.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.384+0000 7effbaffd700 1 -- 172.21.15.70:0/2247541187 learned_addr learned my addr 172.21.15.70:0/2247541187 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:37.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.384+0000 7effba7fc700 1 --2- 172.21.15.70:0/2247541187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effbc1024f0 0x7effbc103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:37.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.384+0000 7effbb7fe700 1 --2- 172.21.15.70:0/2247541187 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7effbc109520 0x7effbc103790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:37.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.385+0000 7effbaffd700 1 -- 172.21.15.70:0/2247541187 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7effbc109520 msgr2=0x7effbc103790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:37.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.385+0000 7effbaffd700 1 --2- 172.21.15.70:0/2247541187 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7effbc109520 0x7effbc103790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.385+0000 7effbaffd700 1 -- 172.21.15.70:0/2247541187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effbc1024f0 msgr2=0x7effbc103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:37.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.385+0000 7effbaffd700 1 --2- 172.21.15.70:0/2247541187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effbc1024f0 0x7effbc103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.385+0000 7effbaffd700 1 -- 172.21.15.70:0/2247541187 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effbc1a8150 con 0x7effbc101b30 2024-06-04T05:00:37.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.385+0000 7effbaffd700 1 --2- 172.21.15.70:0/2247541187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effbc101b30 0x7effbc102d10 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7effac00b530 tx=0x7effac00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:37.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.385+0000 7effa3fff700 1 -- 172.21.15.70:0/2247541187 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac015020 con 0x7effbc101b30 2024-06-04T05:00:37.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.385+0000 7effc17fd700 1 -- 172.21.15.70:0/2247541187 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effbc1a8440 con 0x7effbc101b30 2024-06-04T05:00:37.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.385+0000 7effa3fff700 1 -- 172.21.15.70:0/2247541187 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7effac011cc0 con 0x7effbc101b30 2024-06-04T05:00:37.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.385+0000 7effa3fff700 1 -- 172.21.15.70:0/2247541187 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac010c80 con 0x7effbc101b30 2024-06-04T05:00:37.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.385+0000 7effc17fd700 1 -- 172.21.15.70:0/2247541187 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effbc1a89a0 con 0x7effbc101b30 2024-06-04T05:00:37.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.387+0000 7effa3fff700 1 -- 172.21.15.70:0/2247541187 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7effac018020 con 0x7effbc101b30 2024-06-04T05:00:37.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.388+0000 7effa3fff700 1 --2- 172.21.15.70:0/2247541187 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7effa40b29f0 0x7effa40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.388+0000 7effba7fc700 1 --2- 172.21.15.70:0/2247541187 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7effa40b29f0 0x7effa40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:37.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.388+0000 7effa3fff700 1 -- 172.21.15.70:0/2247541187 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7effac0e1bf0 con 0x7effbc101b30 2024-06-04T05:00:37.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effcc092b80 1 Processor -- start 2024-06-04T05:00:37.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effba7fc700 1 --2- 172.21.15.70:0/2247541187 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7effa40b29f0 0x7effa40b4eb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7effb0000c00 tx=0x7effb0005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:37.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effcc092b80 1 -- start start 2024-06-04T05:00:37.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effcc092b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577963860 0x55a577963c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effcc092b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577964210 0x55a577967940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effcc092b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577967f60 0x55a5779683a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effcc092b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a5778c2c90 con 0x55a577967f60 2024-06-04T05:00:37.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effcc092b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a5778b2830 con 0x55a577964210 2024-06-04T05:00:37.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effcc092b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a5778b6870 con 0x55a577963860 2024-06-04T05:00:37.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effbb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577967f60 0x55a5779683a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:37.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effbb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577967f60 0x55a5779683a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56488/0 (socket says 172.21.15.70:56488) 2024-06-04T05:00:37.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effbb7fe700 1 -- 172.21.15.70:0/1946821907 learned_addr learned my addr 172.21.15.70:0/1946821907 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:37.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effbaffd700 1 --2- 172.21.15.70:0/1946821907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577963860 0x55a577963c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:37.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effba7fc700 1 --2- 172.21.15.70:0/1946821907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577964210 0x55a577967940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:37.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effbb7fe700 1 -- 172.21.15.70:0/1946821907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577963860 msgr2=0x55a577963c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:37.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effbb7fe700 1 --2- 172.21.15.70:0/1946821907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577963860 0x55a577963c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effbb7fe700 1 -- 172.21.15.70:0/1946821907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577964210 msgr2=0x55a577967940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:37.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effbb7fe700 1 --2- 172.21.15.70:0/1946821907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577964210 0x55a577967940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effbb7fe700 1 -- 172.21.15.70:0/1946821907 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5778028d0 con 0x55a577967f60 2024-06-04T05:00:37.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.389+0000 7effba7fc700 1 --2- 172.21.15.70:0/1946821907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577964210 0x55a577967940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:37.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.390+0000 7effbb7fe700 1 --2- 172.21.15.70:0/1946821907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577967f60 0x55a5779683a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7effb400a430 tx=0x7effb4009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afeb05431da39001 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:37.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.390+0000 7effa17fa700 1 -- 172.21.15.70:0/1946821907 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb4014620 con 0x55a577967f60 2024-06-04T05:00:37.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.390+0000 7effa17fa700 1 -- 172.21.15.70:0/1946821907 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7effb4019070 con 0x55a577967f60 2024-06-04T05:00:37.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.390+0000 7effa17fa700 1 -- 172.21.15.70:0/1946821907 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb40139e0 con 0x55a577967f60 2024-06-04T05:00:37.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.390+0000 7effcc092b80 1 -- 172.21.15.70:0/1946821907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577967f60 msgr2=0x55a5779683a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:37.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.390+0000 7effcc092b80 1 --2- 172.21.15.70:0/1946821907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577967f60 0x55a5779683a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7effb400a430 tx=0x7effb4009650 comp rx=0 tx=0).stop 2024-06-04T05:00:37.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.390+0000 7effcc092b80 1 -- 172.21.15.70:0/1946821907 shutdown_connections 2024-06-04T05:00:37.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.390+0000 7effcc092b80 1 --2- 172.21.15.70:0/1946821907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577967f60 0x55a5779683a0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.390+0000 7effcc092b80 1 --2- 172.21.15.70:0/1946821907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577963860 0x55a577963c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.390+0000 7effcc092b80 1 --2- 172.21.15.70:0/1946821907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577964210 0x55a577967940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.390+0000 7effcc092b80 1 -- 172.21.15.70:0/1946821907 >> 172.21.15.70:0/1946821907 conn(0x55a577891080 msgr2=0x55a57795de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:37.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.391+0000 7effcc092b80 1 -- 172.21.15.70:0/1946821907 shutdown_connections 2024-06-04T05:00:37.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.391+0000 7effcc092b80 1 -- 172.21.15.70:0/1946821907 wait complete. 2024-06-04T05:00:37.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.391+0000 7effcc092b80 1 -- start start 2024-06-04T05:00:37.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.391+0000 7effcc092b80 1 Processor -- start 2024-06-04T05:00:37.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.391+0000 7effcc092b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577963860 0x55a57796f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.391+0000 7effcc092b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577964210 0x55a57796c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.391+0000 7effcc092b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577967f60 0x55a57796cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.391+0000 7effcc092b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a5777b8fa0 con 0x55a577963860 2024-06-04T05:00:37.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.391+0000 7effcc092b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a5778547c0 con 0x55a577967f60 2024-06-04T05:00:37.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.391+0000 7effcc092b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a577970120 con 0x55a577964210 2024-06-04T05:00:37.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.391+0000 7effcc092b80 10 client.0 _ll_register_callbacks cb 0x55a57784d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:37.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.391+0000 7effbaffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577963860 0x55a57796f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:37.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.391+0000 7effbaffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577963860 0x55a57796f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56504/0 (socket says 172.21.15.70:56504) 2024-06-04T05:00:37.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.391+0000 7effbaffd700 1 -- 172.21.15.70:0/12555822 learned_addr learned my addr 172.21.15.70:0/12555822 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:37.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.391+0000 7effba7fc700 1 --2- 172.21.15.70:0/12555822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577964210 0x55a57796c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:37.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.391+0000 7effbb7fe700 1 --2- 172.21.15.70:0/12555822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577967f60 0x55a57796cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:37.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.392+0000 7effbaffd700 1 -- 172.21.15.70:0/12555822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577964210 msgr2=0x55a57796c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:37.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.392+0000 7effbaffd700 1 --2- 172.21.15.70:0/12555822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577964210 0x55a57796c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.392+0000 7effbaffd700 1 -- 172.21.15.70:0/12555822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577967f60 msgr2=0x55a57796cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:37.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.392+0000 7effbaffd700 1 --2- 172.21.15.70:0/12555822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577967f60 0x55a57796cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.392+0000 7effbaffd700 1 -- 172.21.15.70:0/12555822 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a577851690 con 0x55a577963860 2024-06-04T05:00:37.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.392+0000 7effbb7fe700 1 --2- 172.21.15.70:0/12555822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577967f60 0x55a57796cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:37.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.392+0000 7effbaffd700 1 --2- 172.21.15.70:0/12555822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577963860 0x55a57796f330 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7effac0df500 tx=0x7effac0a8af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af333201d788152 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:37.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.392+0000 7eff9eff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:37.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.392+0000 7eff9eff5700 1 -- 172.21.15.70:0/12555822 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac0c2020 con 0x55a577963860 2024-06-04T05:00:37.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.392+0000 7eff9eff5700 1 -- 172.21.15.70:0/12555822 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7effac0a9450 con 0x55a577963860 2024-06-04T05:00:37.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.392+0000 7effcc092b80 1 -- 172.21.15.70:0/12555822 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effb4019070 con 0x55a577963860 2024-06-04T05:00:37.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.392+0000 7eff9eff5700 1 -- 172.21.15.70:0/12555822 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7effac0a8fe0 con 0x55a577963860 2024-06-04T05:00:37.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.392+0000 7effcc092b80 10 client.16425 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:37.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.392+0000 7effcc092b80 1 -- 172.21.15.70:0/12555822 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a57797a830 con 0x55a577963860 2024-06-04T05:00:37.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.392+0000 7eff9eff5700 1 -- 172.21.15.70:0/12555822 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7effb4019070 con 0x55a577963860 2024-06-04T05:00:37.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.392+0000 7eff9eff5700 1 -- 172.21.15.70:0/12555822 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7effac0a8240 con 0x55a577963860 2024-06-04T05:00:37.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.393+0000 7effcc092b80 10 client.16425 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:37.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.393+0000 7effcc092b80 10 client.16425 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:37.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.393+0000 7effcc092b80 20 client.16425 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:37.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.393+0000 7effcc092b80 1 --2- 172.21.15.70:0/12555822 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a57797cec0 0x55a57797f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.393+0000 7effcc092b80 4 client.16425 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:00:37.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.393+0000 7effcc092b80 1 -- 172.21.15.70:0/12555822 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a57792f9e0 con 0x55a57797cec0 2024-06-04T05:00:37.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.393+0000 7effba7fc700 1 --2- 172.21.15.70:0/12555822 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a57797cec0 0x55a57797f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:37.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.393+0000 7effba7fc700 1 --2- 172.21.15.70:0/12555822 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a57797cec0 0x55a57797f2b0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d860d3f49876d09d server_cookie=f0ce5fa1b69b1253 in_seq=0 out_seq=0 2024-06-04T05:00:37.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.393+0000 7eff9eff5700 10 client.16425 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:37.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.394+0000 7eff9eff5700 1 -- 172.21.15.70:0/12555822 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a57792f9e0 con 0x55a57797cec0 2024-06-04T05:00:37.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.394+0000 7eff9eff5700 10 client.16425 handle_command_reply: tid=0 2024-06-04T05:00:37.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.394+0000 7effcc092b80 1 client.16425 shutdown 2024-06-04T05:00:37.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.394+0000 7effcc092b80 1 -- 172.21.15.70:0/12555822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577963860 msgr2=0x55a57796f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:37.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.394+0000 7effcc092b80 1 --2- 172.21.15.70:0/12555822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577963860 0x55a57796f330 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7effac0df500 tx=0x7effac0a8af0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.394+0000 7effcc092b80 1 -- 172.21.15.70:0/12555822 shutdown_connections 2024-06-04T05:00:37.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.394+0000 7effcc092b80 1 --2- 172.21.15.70:0/12555822 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a57797cec0 0x55a57797f2b0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.394+0000 7effcc092b80 1 --2- 172.21.15.70:0/12555822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577967f60 0x55a57796cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.394+0000 7effcc092b80 1 --2- 172.21.15.70:0/12555822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577963860 0x55a57796f330 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.394+0000 7effcc092b80 1 --2- 172.21.15.70:0/12555822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577964210 0x55a57796c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.394+0000 7effcc092b80 1 -- 172.21.15.70:0/12555822 >> 172.21.15.70:0/12555822 conn(0x55a577891080 msgr2=0x55a57795d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:37.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.394+0000 7eff9eff5700 0 client.16425 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:37.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.394+0000 7effcc092b80 1 -- 172.21.15.70:0/12555822 shutdown_connections 2024-06-04T05:00:37.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.394+0000 7effcc092b80 1 -- 172.21.15.70:0/12555822 wait complete. 2024-06-04T05:00:37.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.394+0000 7effcc092b80 20 client.16425 trim_cache size 0 max 16384 2024-06-04T05:00:37.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.414+0000 7effcc092b80 1 Processor -- start 2024-06-04T05:00:37.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.414+0000 7effcc092b80 1 -- start start 2024-06-04T05:00:37.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.414+0000 7effcc092b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577963860 0x55a577966f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.414+0000 7effcc092b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577964210 0x55a57795dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.414+0000 7effcc092b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577967f60 0x55a577984b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.414+0000 7effcc092b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7effbc1b0040 con 0x55a577964210 2024-06-04T05:00:37.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.414+0000 7effcc092b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7effbc1b1be0 con 0x55a577967f60 2024-06-04T05:00:37.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.414+0000 7effcc092b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7effbc1b0710 con 0x55a577963860 2024-06-04T05:00:37.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.415+0000 7effba7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577964210 0x55a57795dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:37.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.415+0000 7effba7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577964210 0x55a57795dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56514/0 (socket says 172.21.15.70:56514) 2024-06-04T05:00:37.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.415+0000 7effba7fc700 1 -- 172.21.15.70:0/321493077 learned_addr learned my addr 172.21.15.70:0/321493077 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:37.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.415+0000 7effbb7fe700 1 --2- 172.21.15.70:0/321493077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577967f60 0x55a577984b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:37.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.415+0000 7effba7fc700 1 -- 172.21.15.70:0/321493077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577963860 msgr2=0x55a577966f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:37.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.415+0000 7effba7fc700 1 --2- 172.21.15.70:0/321493077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577963860 0x55a577966f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.415+0000 7effba7fc700 1 -- 172.21.15.70:0/321493077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577967f60 msgr2=0x55a577984b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:37.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.415+0000 7effba7fc700 1 --2- 172.21.15.70:0/321493077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577967f60 0x55a577984b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.415+0000 7effba7fc700 1 -- 172.21.15.70:0/321493077 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a577967c80 con 0x55a577964210 2024-06-04T05:00:37.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.415+0000 7effbb7fe700 1 --2- 172.21.15.70:0/321493077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577967f60 0x55a577984b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:00:37.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.415+0000 7effba7fc700 1 --2- 172.21.15.70:0/321493077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577964210 0x55a57795dc90 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7effb0009750 tx=0x7effb0005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b161d6a6572202c5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:37.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.415+0000 7effa0ff9700 1 -- 172.21.15.70:0/321493077 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb0003880 con 0x55a577964210 2024-06-04T05:00:37.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.415+0000 7effa0ff9700 1 -- 172.21.15.70:0/321493077 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7effb000f460 con 0x55a577964210 2024-06-04T05:00:37.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.415+0000 7effa0ff9700 1 -- 172.21.15.70:0/321493077 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb0031660 con 0x55a577964210 2024-06-04T05:00:37.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 1 -- 172.21.15.70:0/321493077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577964210 msgr2=0x55a57795dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:37.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 1 --2- 172.21.15.70:0/321493077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577964210 0x55a57795dc90 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7effb0009750 tx=0x7effb0005000 comp rx=0 tx=0).stop 2024-06-04T05:00:37.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 1 -- 172.21.15.70:0/321493077 shutdown_connections 2024-06-04T05:00:37.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 1 --2- 172.21.15.70:0/321493077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577967f60 0x55a577984b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 1 --2- 172.21.15.70:0/321493077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577963860 0x55a577966f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 1 --2- 172.21.15.70:0/321493077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577964210 0x55a57795dc90 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 1 -- 172.21.15.70:0/321493077 >> 172.21.15.70:0/321493077 conn(0x55a577891080 msgr2=0x55a577962f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:37.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 1 -- 172.21.15.70:0/321493077 shutdown_connections 2024-06-04T05:00:37.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 1 -- 172.21.15.70:0/321493077 wait complete. 2024-06-04T05:00:37.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 1 -- start start 2024-06-04T05:00:37.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 1 Processor -- start 2024-06-04T05:00:37.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577963860 0x55a577973770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577964210 0x55a577973cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577967f60 0x55a5779e6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7effbc1b14b0 con 0x55a577964210 2024-06-04T05:00:37.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7effbc1b0de0 con 0x55a577967f60 2024-06-04T05:00:37.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7effbc1adfe0 con 0x55a577963860 2024-06-04T05:00:37.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effcc092b80 10 client.0 _ll_register_callbacks cb 0x55a57784d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:37.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.416+0000 7effba7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577964210 0x55a577973cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:37.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7effba7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577964210 0x55a577973cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56524/0 (socket says 172.21.15.70:56524) 2024-06-04T05:00:37.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7effba7fc700 1 -- 172.21.15.70:0/1017916713 learned_addr learned my addr 172.21.15.70:0/1017916713 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:37.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7effbaffd700 1 --2- 172.21.15.70:0/1017916713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577963860 0x55a577973770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:37.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7effba7fc700 1 -- 172.21.15.70:0/1017916713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577963860 msgr2=0x55a577973770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:37.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7effba7fc700 1 --2- 172.21.15.70:0/1017916713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577963860 0x55a577973770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7effba7fc700 1 -- 172.21.15.70:0/1017916713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577967f60 msgr2=0x55a5779e6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:37.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7effba7fc700 1 --2- 172.21.15.70:0/1017916713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577967f60 0x55a5779e6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7effba7fc700 1 -- 172.21.15.70:0/1017916713 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5779e6f70 con 0x55a577964210 2024-06-04T05:00:37.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7effbaffd700 1 --2- 172.21.15.70:0/1017916713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577963860 0x55a577973770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:37.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7effba7fc700 1 --2- 172.21.15.70:0/1017916713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577964210 0x55a577973cb0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7effb0039000 tx=0x7effb000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a12ccede84dc5aa server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:37.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7eff9eff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:37.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7eff9eff5700 1 -- 172.21.15.70:0/1017916713 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb000f810 con 0x55a577964210 2024-06-04T05:00:37.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7eff9eff5700 1 -- 172.21.15.70:0/1017916713 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7effb000f9b0 con 0x55a577964210 2024-06-04T05:00:37.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7effcc092b80 1 -- 172.21.15.70:0/1017916713 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effb000f460 con 0x55a577964210 2024-06-04T05:00:37.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7eff9eff5700 1 -- 172.21.15.70:0/1017916713 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7effb0003680 con 0x55a577964210 2024-06-04T05:00:37.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7effcc092b80 10 client.16431 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:37.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7effcc092b80 1 -- 172.21.15.70:0/1017916713 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a5779e8120 con 0x55a577964210 2024-06-04T05:00:37.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.417+0000 7eff9eff5700 1 -- 172.21.15.70:0/1017916713 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7effb000f460 con 0x55a577964210 2024-06-04T05:00:37.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.418+0000 7eff9eff5700 1 -- 172.21.15.70:0/1017916713 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7effb003a070 con 0x55a577964210 2024-06-04T05:00:37.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.418+0000 7effcc092b80 10 client.16431 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:37.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.418+0000 7effcc092b80 10 client.16431 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:37.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.418+0000 7effcc092b80 20 client.16431 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:37.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.418+0000 7effcc092b80 1 --2- 172.21.15.70:0/1017916713 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a5779e87c0 0x55a5779eabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:37.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.418+0000 7effcc092b80 4 client.16431 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:00:37.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.418+0000 7effcc092b80 1 -- 172.21.15.70:0/1017916713 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a5778c4be0 con 0x55a5779e87c0 2024-06-04T05:00:37.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.418+0000 7effbaffd700 1 --2- 172.21.15.70:0/1017916713 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a5779e87c0 0x55a5779eabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:37.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.418+0000 7effbaffd700 1 --2- 172.21.15.70:0/1017916713 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a5779e87c0 0x55a5779eabe0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5e07cc0f2bea122 server_cookie=7a6f1ed840e979ea in_seq=0 out_seq=0 2024-06-04T05:00:37.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.418+0000 7eff9eff5700 10 client.16431 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:37.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7eff9eff5700 1 -- 172.21.15.70:0/1017916713 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a5778c4be0 con 0x55a5779e87c0 2024-06-04T05:00:37.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7eff9eff5700 10 client.16431 handle_command_reply: tid=0 2024-06-04T05:00:37.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effcc092b80 1 client.16431 shutdown 2024-06-04T05:00:37.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effcc092b80 1 -- 172.21.15.70:0/1017916713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577964210 msgr2=0x55a577973cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:37.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effcc092b80 1 --2- 172.21.15.70:0/1017916713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577964210 0x55a577973cb0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7effb0039000 tx=0x7effb000bfd0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effcc092b80 1 -- 172.21.15.70:0/1017916713 shutdown_connections 2024-06-04T05:00:37.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effcc092b80 1 --2- 172.21.15.70:0/1017916713 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a5779e87c0 0x55a5779eabe0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effcc092b80 1 --2- 172.21.15.70:0/1017916713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a577967f60 0x55a5779e6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effcc092b80 1 --2- 172.21.15.70:0/1017916713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a577963860 0x55a577973770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effcc092b80 1 --2- 172.21.15.70:0/1017916713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a577964210 0x55a577973cb0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effcc092b80 1 -- 172.21.15.70:0/1017916713 >> 172.21.15.70:0/1017916713 conn(0x55a577891080 msgr2=0x55a577962f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:37.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7eff9eff5700 0 client.16431 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:37.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effcc092b80 1 -- 172.21.15.70:0/1017916713 shutdown_connections 2024-06-04T05:00:37.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effcc092b80 1 -- 172.21.15.70:0/1017916713 wait complete. 2024-06-04T05:00:37.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effcc092b80 20 client.16431 trim_cache size 0 max 16384 2024-06-04T05:00:37.427 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:00:37.427 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:00:37.427 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:00:37.427 INFO:teuthology.orchestra.run.smithi070.stdout: "e855c38a-3834-4eb6-85c7-9525e06df599": { 2024-06-04T05:00:37.428 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:00:37.428 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "e855c38a-3834-4eb6-85c7-9525e06df599", 2024-06-04T05:00:37.428 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:00:37.428 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:00:37.428 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:00:37.428 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:00:37.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effc17fd700 1 -- 172.21.15.70:0/2247541187 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7effa40b29f0 msgr2=0x7effa40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:37.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effc17fd700 1 --2- 172.21.15.70:0/2247541187 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7effa40b29f0 0x7effa40b4eb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7effb0000c00 tx=0x7effb0005fb0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effc17fd700 1 -- 172.21.15.70:0/2247541187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effbc101b30 msgr2=0x7effbc102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:37.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effc17fd700 1 --2- 172.21.15.70:0/2247541187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effbc101b30 0x7effbc102d10 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7effac00b530 tx=0x7effac00b840 comp rx=0 tx=0).stop 2024-06-04T05:00:37.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effc17fd700 1 -- 172.21.15.70:0/2247541187 shutdown_connections 2024-06-04T05:00:37.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effc17fd700 1 --2- 172.21.15.70:0/2247541187 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7effa40b29f0 0x7effa40b4eb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effc17fd700 1 --2- 172.21.15.70:0/2247541187 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7effbc109520 0x7effbc103790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effc17fd700 1 --2- 172.21.15.70:0/2247541187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effbc101b30 0x7effbc102d10 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effc17fd700 1 --2- 172.21.15.70:0/2247541187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effbc1024f0 0x7effbc103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:37.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.419+0000 7effc17fd700 1 -- 172.21.15.70:0/2247541187 >> 172.21.15.70:0/2247541187 conn(0x7effbc078ab0 msgr2=0x7effbc100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:37.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.420+0000 7effc17fd700 1 -- 172.21.15.70:0/2247541187 shutdown_connections 2024-06-04T05:00:37.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:37.420+0000 7effc17fd700 1 -- 172.21.15.70:0/2247541187 wait complete. 2024-06-04T05:00:37.715 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.72 MB/sec warmup 90 sec latency 14627.853 ms 2024-06-04T05:00:38.162 INFO:tasks.cephfs.filesystem:scrub status for tag:e855c38a-3834-4eb6-85c7-9525e06df599 - {'path': '/', 'tag': 'e855c38a-3834-4eb6-85c7-9525e06df599', 'options': 'recursive,force'} 2024-06-04T05:00:38.715 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.69 MB/sec warmup 91 sec latency 15627.895 ms 2024-06-04T05:00:39.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:39 smithi070 ceph-mon[32631]: pgmap v107: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 15 KiB/s wr, 21 op/s 2024-06-04T05:00:39.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:39 smithi112 ceph-mon[36453]: pgmap v107: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 15 KiB/s wr, 21 op/s 2024-06-04T05:00:39.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:39 smithi098 ceph-mon[35569]: pgmap v107: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 15 KiB/s wr, 21 op/s 2024-06-04T05:00:39.716 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.66 MB/sec warmup 92 sec latency 16627.988 ms 2024-06-04T05:00:40.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.676+0000 7fead5828640 1 -- 172.21.15.70:0/3716994884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fead0107960 msgr2=0x7fead0107d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.676+0000 7fead5828640 1 --2- 172.21.15.70:0/3716994884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fead0107960 0x7fead0107d40 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7feac0009930 tx=0x7feac002a420 comp rx=0 tx=0).stop 2024-06-04T05:00:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.676+0000 7fead5828640 1 -- 172.21.15.70:0/3716994884 shutdown_connections 2024-06-04T05:00:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.676+0000 7fead5828640 1 --2- 172.21.15.70:0/3716994884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead0101950 0x7fead0101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.676+0000 7fead5828640 1 --2- 172.21.15.70:0/3716994884 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fead0102310 0x7fead010e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.676+0000 7fead5828640 1 --2- 172.21.15.70:0/3716994884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fead0107960 0x7fead0107d40 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.676+0000 7fead5828640 1 -- 172.21.15.70:0/3716994884 >> 172.21.15.70:0/3716994884 conn(0x7fead0078ad0 msgr2=0x7fead0078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.676+0000 7fead5828640 1 -- 172.21.15.70:0/3716994884 shutdown_connections 2024-06-04T05:00:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.676+0000 7fead5828640 1 -- 172.21.15.70:0/3716994884 wait complete. 2024-06-04T05:00:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7fead5828640 1 Processor -- start 2024-06-04T05:00:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7fead5828640 1 -- start start 2024-06-04T05:00:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7fead5828640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fead0101950 0x7fead0199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7fead5828640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fead0102310 0x7fead01995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7fead5828640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead0107960 0x7fead019d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7fead5828640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fead0112bd0 con 0x7fead0102310 2024-06-04T05:00:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7fead5828640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fead0112a50 con 0x7fead0107960 2024-06-04T05:00:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7fead5828640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fead0112d50 con 0x7fead0101950 2024-06-04T05:00:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7feace7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fead0102310 0x7fead01995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7feace7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fead0102310 0x7fead01995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56550/0 (socket says 172.21.15.70:56550) 2024-06-04T05:00:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7feace7fc640 1 -- 172.21.15.70:0/3458418503 learned_addr learned my addr 172.21.15.70:0/3458418503 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7feaceffd640 1 --2- 172.21.15.70:0/3458418503 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fead0101950 0x7fead0199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7feacf7fe640 1 --2- 172.21.15.70:0/3458418503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead0107960 0x7fead019d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7feace7fc640 1 -- 172.21.15.70:0/3458418503 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fead0101950 msgr2=0x7fead0199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7feace7fc640 1 --2- 172.21.15.70:0/3458418503 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fead0101950 0x7fead0199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7feace7fc640 1 -- 172.21.15.70:0/3458418503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead0107960 msgr2=0x7fead019d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7feace7fc640 1 --2- 172.21.15.70:0/3458418503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead0107960 0x7fead019d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.677+0000 7feace7fc640 1 -- 172.21.15.70:0/3458418503 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead019e080 con 0x7fead0102310 2024-06-04T05:00:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.678+0000 7feace7fc640 1 --2- 172.21.15.70:0/3458418503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fead0102310 0x7fead01995d0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7feab800d950 tx=0x7feab800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.678+0000 7fead4826640 1 -- 172.21.15.70:0/3458418503 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab8017070 con 0x7fead0102310 2024-06-04T05:00:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.678+0000 7fead4826640 1 -- 172.21.15.70:0/3458418503 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feab8009c30 con 0x7fead0102310 2024-06-04T05:00:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.678+0000 7fead5828640 1 -- 172.21.15.70:0/3458418503 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead019e370 con 0x7fead0102310 2024-06-04T05:00:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.678+0000 7fead4826640 1 -- 172.21.15.70:0/3458418503 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab800f620 con 0x7fead0102310 2024-06-04T05:00:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.678+0000 7fead5828640 1 -- 172.21.15.70:0/3458418503 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead01a5c00 con 0x7fead0102310 2024-06-04T05:00:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.680+0000 7fead8617740 1 Processor -- start 2024-06-04T05:00:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.680+0000 7fead8617740 1 -- start start 2024-06-04T05:00:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.680+0000 7fead4826640 1 -- 172.21.15.70:0/3458418503 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7feab8010460 con 0x7fead0102310 2024-06-04T05:00:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7fead8617740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875b820 0x55af9875bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7fead8617740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af987664c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7fead8617740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 0x55af98768df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7fead8617740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55af986cdc30 con 0x55af9875c140 2024-06-04T05:00:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7fead8617740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55af986bd7d0 con 0x55af9875b820 2024-06-04T05:00:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7fead8617740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55af986c1810 con 0x55af98766a00 2024-06-04T05:00:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7feaceffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af987664c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7feaceffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af987664c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56560/0 (socket says 172.21.15.70:56560) 2024-06-04T05:00:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7feaceffd640 1 -- 172.21.15.70:0/2875497641 learned_addr learned my addr 172.21.15.70:0/2875497641 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7feacf7fe640 1 --2- 172.21.15.70:0/2875497641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875b820 0x55af9875bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7feace7fc640 1 --2- 172.21.15.70:0/2875497641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 0x55af98768df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7feaceffd640 1 -- 172.21.15.70:0/2875497641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 msgr2=0x55af98768df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7feaceffd640 1 --2- 172.21.15.70:0/2875497641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 0x55af98768df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7feaceffd640 1 -- 172.21.15.70:0/2875497641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875b820 msgr2=0x55af9875bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7feaceffd640 1 --2- 172.21.15.70:0/2875497641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875b820 0x55af9875bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7feaceffd640 1 -- 172.21.15.70:0/2875497641 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af983d2210 con 0x55af9875c140 2024-06-04T05:00:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7fead4826640 1 --2- 172.21.15.70:0/3458418503 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feabc0b2a10 0x7feabc0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7feaceffd640 1 --2- 172.21.15.70:0/2875497641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af987664c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7feac0000c00 tx=0x7feac0032dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9403c78b38367432 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.681+0000 7feacf7fe640 1 --2- 172.21.15.70:0/3458418503 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feabc0b2a10 0x7feabc0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.682+0000 7feac5ffb640 1 -- 172.21.15.70:0/2875497641 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac003f070 con 0x55af9875c140 2024-06-04T05:00:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.682+0000 7fead4826640 1 -- 172.21.15.70:0/3458418503 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7feab80e22e0 con 0x7fead0102310 2024-06-04T05:00:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.682+0000 7feac5ffb640 1 -- 172.21.15.70:0/2875497641 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feac00044c0 con 0x55af9875c140 2024-06-04T05:00:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.682+0000 7feac5ffb640 1 -- 172.21.15.70:0/2875497641 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0035070 con 0x55af9875c140 2024-06-04T05:00:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.682+0000 7fead8617740 1 -- 172.21.15.70:0/2875497641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 msgr2=0x55af987664c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.682+0000 7feacf7fe640 1 --2- 172.21.15.70:0/3458418503 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feabc0b2a10 0x7feabc0b4ed0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7feac8007e50 tx=0x7feac8007600 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.682+0000 7fead8617740 1 --2- 172.21.15.70:0/2875497641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af987664c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7feac0000c00 tx=0x7feac0032dd0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.682+0000 7fead8617740 1 -- 172.21.15.70:0/2875497641 shutdown_connections 2024-06-04T05:00:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.682+0000 7fead8617740 1 --2- 172.21.15.70:0/2875497641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 0x55af98768df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.682+0000 7fead8617740 1 --2- 172.21.15.70:0/2875497641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af987664c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.682+0000 7fead8617740 1 --2- 172.21.15.70:0/2875497641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875b820 0x55af9875bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.682+0000 7fead8617740 1 -- 172.21.15.70:0/2875497641 >> 172.21.15.70:0/2875497641 conn(0x55af98757d30 msgr2=0x55af98758110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.682+0000 7fead8617740 1 -- 172.21.15.70:0/2875497641 shutdown_connections 2024-06-04T05:00:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.682+0000 7fead8617740 1 -- 172.21.15.70:0/2875497641 wait complete. 2024-06-04T05:00:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.683+0000 7fead8617740 1 -- start start 2024-06-04T05:00:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.683+0000 7fead8617740 1 Processor -- start 2024-06-04T05:00:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.683+0000 7fead8617740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875b820 0x55af987639a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.683+0000 7fead8617740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875c140 0x55af98763ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.683+0000 7fead8617740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 0x55af98764420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.683+0000 7fead8617740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55af983a3790 con 0x55af9875b820 2024-06-04T05:00:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.683+0000 7fead8617740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55af98679890 con 0x55af9875c140 2024-06-04T05:00:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.683+0000 7fead8617740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55af987695d0 con 0x55af98766a00 2024-06-04T05:00:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.683+0000 7fead8617740 10 client.0 _ll_register_callbacks cb 0x55af983d1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.683+0000 7feaceffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875b820 0x55af987639a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.683+0000 7feaceffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875b820 0x55af987639a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56564/0 (socket says 172.21.15.70:56564) 2024-06-04T05:00:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.683+0000 7feaceffd640 1 -- 172.21.15.70:0/2597651491 learned_addr learned my addr 172.21.15.70:0/2597651491 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.684+0000 7feacf7fe640 1 --2- 172.21.15.70:0/2597651491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 0x55af98764420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.684+0000 7feace7fc640 1 --2- 172.21.15.70:0/2597651491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875c140 0x55af98763ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.684+0000 7feaceffd640 1 -- 172.21.15.70:0/2597651491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 msgr2=0x55af98764420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.684+0000 7feaceffd640 1 --2- 172.21.15.70:0/2597651491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 0x55af98764420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.684+0000 7feaceffd640 1 -- 172.21.15.70:0/2597651491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875c140 msgr2=0x55af98763ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.684+0000 7feaceffd640 1 --2- 172.21.15.70:0/2597651491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875c140 0x55af98763ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.684+0000 7feaceffd640 1 -- 172.21.15.70:0/2597651491 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af9849e620 con 0x55af9875b820 2024-06-04T05:00:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.684+0000 7feacf7fe640 1 --2- 172.21.15.70:0/2597651491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 0x55af98764420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.684+0000 7feaceffd640 1 --2- 172.21.15.70:0/2597651491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875b820 0x55af987639a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7feac0009900 tx=0x7feac0031d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca66e85d2ace08d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.684+0000 7feaaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.684+0000 7feaaeffd640 1 -- 172.21.15.70:0/2597651491 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0035070 con 0x55af9875b820 2024-06-04T05:00:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.684+0000 7feaaeffd640 1 -- 172.21.15.70:0/2597651491 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7feac00046c0 con 0x55af9875b820 2024-06-04T05:00:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.684+0000 7fead8617740 1 -- 172.21.15.70:0/2597651491 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55af986cfb80 con 0x55af9875b820 2024-06-04T05:00:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.684+0000 7feaaeffd640 1 -- 172.21.15.70:0/2597651491 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7feac0055050 con 0x55af9875b820 2024-06-04T05:00:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.684+0000 7fead8617740 10 client.16443 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.684+0000 7fead8617740 1 -- 172.21.15.70:0/2597651491 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55af98775340 con 0x55af9875b820 2024-06-04T05:00:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.684+0000 7feaaeffd640 1 -- 172.21.15.70:0/2597651491 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55af986cfb80 con 0x55af9875b820 2024-06-04T05:00:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.685+0000 7feaaeffd640 1 -- 172.21.15.70:0/2597651491 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7feac003b550 con 0x55af9875b820 2024-06-04T05:00:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.685+0000 7fead8617740 10 client.16443 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.685+0000 7fead8617740 10 client.16443 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.685+0000 7fead8617740 20 client.16443 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.685+0000 7fead8617740 1 --2- 172.21.15.70:0/2597651491 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55af98776690 0x55af98778a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.685+0000 7fead8617740 4 client.16443 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:00:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.685+0000 7fead8617740 1 -- 172.21.15.70:0/2597651491 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af986c9360 con 0x55af98776690 2024-06-04T05:00:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.685+0000 7feace7fc640 1 --2- 172.21.15.70:0/2597651491 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55af98776690 0x55af98778a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.685+0000 7feace7fc640 1 --2- 172.21.15.70:0/2597651491 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55af98776690 0x55af98778a80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6ac0eb2b9444f40 server_cookie=f0a1cea45b87b204 in_seq=0 out_seq=0 2024-06-04T05:00:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.685+0000 7feaaeffd640 10 client.16443 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.686+0000 7feaaeffd640 1 -- 172.21.15.70:0/2597651491 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55af986c9360 con 0x55af98776690 2024-06-04T05:00:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.686+0000 7feaaeffd640 10 client.16443 handle_command_reply: tid=0 2024-06-04T05:00:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.686+0000 7fead8617740 1 client.16443 shutdown 2024-06-04T05:00:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.686+0000 7fead8617740 1 -- 172.21.15.70:0/2597651491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875b820 msgr2=0x55af987639a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.686+0000 7fead8617740 1 --2- 172.21.15.70:0/2597651491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875b820 0x55af987639a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7feac0009900 tx=0x7feac0031d20 comp rx=0 tx=0).stop 2024-06-04T05:00:40.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.686+0000 7fead8617740 1 -- 172.21.15.70:0/2597651491 shutdown_connections 2024-06-04T05:00:40.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.686+0000 7fead8617740 1 --2- 172.21.15.70:0/2597651491 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55af98776690 0x55af98778a80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.686+0000 7fead8617740 1 --2- 172.21.15.70:0/2597651491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875c140 0x55af98763ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.686+0000 7fead8617740 1 --2- 172.21.15.70:0/2597651491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 0x55af98764420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.686+0000 7fead8617740 1 --2- 172.21.15.70:0/2597651491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875b820 0x55af987639a0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.686+0000 7feaaeffd640 0 client.16443 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:40.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.686+0000 7fead8617740 1 -- 172.21.15.70:0/2597651491 >> 172.21.15.70:0/2597651491 conn(0x55af98757d30 msgr2=0x55af98756fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:40.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.686+0000 7fead8617740 1 -- 172.21.15.70:0/2597651491 shutdown_connections 2024-06-04T05:00:40.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.686+0000 7fead8617740 1 -- 172.21.15.70:0/2597651491 wait complete. 2024-06-04T05:00:40.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.686+0000 7fead8617740 20 client.16443 trim_cache size 0 max 16384 2024-06-04T05:00:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7fead8617740 1 Processor -- start 2024-06-04T05:00:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7fead8617740 1 -- start start 2024-06-04T05:00:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7fead8617740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875b820 0x55af9875e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7fead8617740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af98764240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7fead8617740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 0x55af9876ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7fead8617740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fead01ae470 con 0x55af9875c140 2024-06-04T05:00:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7fead8617740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fead01b0010 con 0x55af9875b820 2024-06-04T05:00:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7fead8617740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fead01aeb40 con 0x55af98766a00 2024-06-04T05:00:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7feace7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af98764240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7feace7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af98764240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56566/0 (socket says 172.21.15.70:56566) 2024-06-04T05:00:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7feace7fc640 1 -- 172.21.15.70:0/2493080424 learned_addr learned my addr 172.21.15.70:0/2493080424 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7feaceffd640 1 --2- 172.21.15.70:0/2493080424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875b820 0x55af9875e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7feace7fc640 1 -- 172.21.15.70:0/2493080424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 msgr2=0x55af9876ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7feace7fc640 1 --2- 172.21.15.70:0/2493080424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 0x55af9876ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7feace7fc640 1 -- 172.21.15.70:0/2493080424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875b820 msgr2=0x55af9875e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7feace7fc640 1 --2- 172.21.15.70:0/2493080424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875b820 0x55af9875e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7feace7fc640 1 -- 172.21.15.70:0/2493080424 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af98762950 con 0x55af9875c140 2024-06-04T05:00:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7feaceffd640 1 --2- 172.21.15.70:0/2493080424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875b820 0x55af9875e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7feace7fc640 1 --2- 172.21.15.70:0/2493080424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af98764240 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7feab80a5640 tx=0x7feab8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db877bd1c9905d4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.702+0000 7feac57fa640 1 -- 172.21.15.70:0/2493080424 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab8020040 con 0x55af9875c140 2024-06-04T05:00:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.703+0000 7feac57fa640 1 -- 172.21.15.70:0/2493080424 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feab80e1070 con 0x55af9875c140 2024-06-04T05:00:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.703+0000 7feac57fa640 1 -- 172.21.15.70:0/2493080424 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab80b2420 con 0x55af9875c140 2024-06-04T05:00:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.703+0000 7fead8617740 1 -- 172.21.15.70:0/2493080424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 msgr2=0x55af98764240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.703+0000 7fead8617740 1 --2- 172.21.15.70:0/2493080424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af98764240 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7feab80a5640 tx=0x7feab8005880 comp rx=0 tx=0).stop 2024-06-04T05:00:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.703+0000 7fead8617740 1 -- 172.21.15.70:0/2493080424 shutdown_connections 2024-06-04T05:00:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.703+0000 7fead8617740 1 --2- 172.21.15.70:0/2493080424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 0x55af9876ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.703+0000 7fead8617740 1 --2- 172.21.15.70:0/2493080424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af98764240 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.703+0000 7fead8617740 1 --2- 172.21.15.70:0/2493080424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875b820 0x55af9875e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.703+0000 7fead8617740 1 -- 172.21.15.70:0/2493080424 >> 172.21.15.70:0/2493080424 conn(0x55af98757d30 msgr2=0x55af98756fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.703+0000 7fead8617740 1 -- 172.21.15.70:0/2493080424 shutdown_connections 2024-06-04T05:00:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.703+0000 7fead8617740 1 -- 172.21.15.70:0/2493080424 wait complete. 2024-06-04T05:00:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.703+0000 7fead8617740 1 -- start start 2024-06-04T05:00:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7fead8617740 1 Processor -- start 2024-06-04T05:00:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7fead8617740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875b820 0x55af987dd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7fead8617740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af987dddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7fead8617740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 0x55af9877db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7fead8617740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fead01af8e0 con 0x55af9875c140 2024-06-04T05:00:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7fead8617740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fead01af210 con 0x55af9875b820 2024-06-04T05:00:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7fead8617740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fead01ac450 con 0x55af98766a00 2024-06-04T05:00:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7fead8617740 10 client.0 _ll_register_callbacks cb 0x7feac80074a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7feace7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af987dddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7feace7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af987dddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56576/0 (socket says 172.21.15.70:56576) 2024-06-04T05:00:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7feace7fc640 1 -- 172.21.15.70:0/3127113169 learned_addr learned my addr 172.21.15.70:0/3127113169 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:40.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7feaceffd640 1 --2- 172.21.15.70:0/3127113169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875b820 0x55af987dd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:40.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7feace7fc640 1 -- 172.21.15.70:0/3127113169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 msgr2=0x55af9877db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:40.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7feace7fc640 1 --2- 172.21.15.70:0/3127113169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 0x55af9877db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7feace7fc640 1 -- 172.21.15.70:0/3127113169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875b820 msgr2=0x55af987dd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:40.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7feace7fc640 1 --2- 172.21.15.70:0/3127113169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875b820 0x55af987dd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7feace7fc640 1 -- 172.21.15.70:0/3127113169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af9877e210 con 0x55af9875c140 2024-06-04T05:00:40.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7feace7fc640 1 --2- 172.21.15.70:0/3127113169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af987dddf0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7feab80a92c0 tx=0x7feab80dfee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8c2569aba986a2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:40.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.704+0000 7feaaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:40.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.705+0000 7feaaeffd640 1 -- 172.21.15.70:0/3127113169 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab8020070 con 0x55af9875c140 2024-06-04T05:00:40.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.705+0000 7fead8617740 1 -- 172.21.15.70:0/3127113169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feab80e1070 con 0x55af9875c140 2024-06-04T05:00:40.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.705+0000 7feaaeffd640 1 -- 172.21.15.70:0/3127113169 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7feab80a5850 con 0x55af9875c140 2024-06-04T05:00:40.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.705+0000 7feaaeffd640 1 -- 172.21.15.70:0/3127113169 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7feab8009de0 con 0x55af9875c140 2024-06-04T05:00:40.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.705+0000 7fead8617740 10 client.16449 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:40.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.705+0000 7fead8617740 1 -- 172.21.15.70:0/3127113169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55af987e4fd0 con 0x55af9875c140 2024-06-04T05:00:40.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.705+0000 7feaaeffd640 1 -- 172.21.15.70:0/3127113169 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feab80e1070 con 0x55af9875c140 2024-06-04T05:00:40.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.705+0000 7feaaeffd640 1 -- 172.21.15.70:0/3127113169 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7feab80a3ce0 con 0x55af9875c140 2024-06-04T05:00:40.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.705+0000 7fead8617740 10 client.16449 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:40.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.705+0000 7fead8617740 10 client.16449 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:40.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.705+0000 7fead8617740 20 client.16449 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:40.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.705+0000 7fead8617740 1 --2- 172.21.15.70:0/3127113169 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55af987e56b0 0x55af987e7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:40.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.705+0000 7fead8617740 4 client.16449 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:00:40.716 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.705+0000 7fead8617740 1 -- 172.21.15.70:0/3127113169 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55af986435f0 con 0x55af987e56b0 2024-06-04T05:00:40.716 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.705+0000 7feaceffd640 1 --2- 172.21.15.70:0/3127113169 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55af987e56b0 0x55af987e7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:40.716 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.706+0000 7feaaeffd640 10 client.16449 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:40.716 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.706+0000 7feaceffd640 1 --2- 172.21.15.70:0/3127113169 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55af987e56b0 0x55af987e7ad0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9e93b4f36c7f16f server_cookie=e15b0d5abc2ae4f4 in_seq=0 out_seq=0 2024-06-04T05:00:40.716 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.63 MB/sec warmup 93 sec latency 17628.031 ms 2024-06-04T05:00:40.723 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.721+0000 7feaaeffd640 1 -- 172.21.15.70:0/3127113169 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55af986435f0 con 0x55af987e56b0 2024-06-04T05:00:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.721+0000 7feaaeffd640 10 client.16449 handle_command_reply: tid=0 2024-06-04T05:00:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.721+0000 7fead8617740 1 client.16449 shutdown 2024-06-04T05:00:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.721+0000 7fead8617740 1 -- 172.21.15.70:0/3127113169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 msgr2=0x55af987dddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.721+0000 7fead8617740 1 --2- 172.21.15.70:0/3127113169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af987dddf0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7feab80a92c0 tx=0x7feab80dfee0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.721+0000 7fead8617740 1 -- 172.21.15.70:0/3127113169 shutdown_connections 2024-06-04T05:00:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.721+0000 7fead8617740 1 --2- 172.21.15.70:0/3127113169 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55af987e56b0 0x55af987e7ad0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.721+0000 7fead8617740 1 --2- 172.21.15.70:0/3127113169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55af98766a00 0x55af9877db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.721+0000 7fead8617740 1 --2- 172.21.15.70:0/3127113169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af9875c140 0x55af987dddf0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.721+0000 7feaaeffd640 0 client.16449 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.721+0000 7fead8617740 1 --2- 172.21.15.70:0/3127113169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9875b820 0x55af987dd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.721+0000 7fead8617740 1 -- 172.21.15.70:0/3127113169 >> 172.21.15.70:0/3127113169 conn(0x55af98757d30 msgr2=0x55af98771620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.721+0000 7fead8617740 1 -- 172.21.15.70:0/3127113169 shutdown_connections 2024-06-04T05:00:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.721+0000 7fead8617740 1 -- 172.21.15.70:0/3127113169 wait complete. 2024-06-04T05:00:40.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.721+0000 7fead8617740 20 client.16449 trim_cache size 0 max 16384 2024-06-04T05:00:40.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:00:40.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:00:40.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:00:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.722+0000 7fead5828640 1 -- 172.21.15.70:0/3458418503 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feabc0b2a10 msgr2=0x7feabc0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.722+0000 7fead5828640 1 --2- 172.21.15.70:0/3458418503 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feabc0b2a10 0x7feabc0b4ed0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7feac8007e50 tx=0x7feac8007600 comp rx=0 tx=0).stop 2024-06-04T05:00:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.722+0000 7fead5828640 1 -- 172.21.15.70:0/3458418503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fead0102310 msgr2=0x7fead01995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.722+0000 7fead5828640 1 --2- 172.21.15.70:0/3458418503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fead0102310 0x7fead01995d0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7feab800d950 tx=0x7feab800de10 comp rx=0 tx=0).stop 2024-06-04T05:00:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.722+0000 7fead5828640 1 -- 172.21.15.70:0/3458418503 shutdown_connections 2024-06-04T05:00:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.722+0000 7fead5828640 1 --2- 172.21.15.70:0/3458418503 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feabc0b2a10 0x7feabc0b4ed0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.722+0000 7fead5828640 1 --2- 172.21.15.70:0/3458418503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead0107960 0x7fead019d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.722+0000 7fead5828640 1 --2- 172.21.15.70:0/3458418503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fead0102310 0x7fead01995d0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.722+0000 7fead5828640 1 --2- 172.21.15.70:0/3458418503 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fead0101950 0x7fead0199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.722+0000 7fead5828640 1 -- 172.21.15.70:0/3458418503 >> 172.21.15.70:0/3458418503 conn(0x7fead0078ad0 msgr2=0x7fead0104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.722+0000 7fead5828640 1 -- 172.21.15.70:0/3458418503 shutdown_connections 2024-06-04T05:00:40.727 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:40.722+0000 7fead5828640 1 -- 172.21.15.70:0/3458418503 wait complete. 2024-06-04T05:00:41.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:41 smithi070 ceph-mon[32631]: pgmap v108: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 15 KiB/s wr, 21 op/s 2024-06-04T05:00:41.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:41 smithi112 ceph-mon[36453]: pgmap v108: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 15 KiB/s wr, 21 op/s 2024-06-04T05:00:41.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:41 smithi098 ceph-mon[35569]: pgmap v108: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 15 KiB/s wr, 21 op/s 2024-06-04T05:00:41.715 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.60 MB/sec warmup 94 sec latency 18628.089 ms 2024-06-04T05:00:42.716 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.58 MB/sec warmup 95 sec latency 19628.197 ms 2024-06-04T05:00:42.854 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:00:43.111 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:00:43.233 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:43 smithi070 ceph-mon[32631]: pgmap v109: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-06-04T05:00:43.233 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:43 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:00:43.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:43 smithi112 ceph-mon[36453]: pgmap v109: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-06-04T05:00:43.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:43 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:00:43.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:43 smithi098 ceph-mon[35569]: pgmap v109: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-06-04T05:00:43.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:43 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:00:43.716 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.55 MB/sec warmup 96 sec latency 20628.321 ms 2024-06-04T05:00:44.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:44 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:00:44.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:44 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:00:44.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:44 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:00:44.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:44 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:00:44.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:44 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:00:44.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:44 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:00:44.716 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.52 MB/sec warmup 97 sec latency 21628.368 ms 2024-06-04T05:00:44.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.949+0000 7f2e0b01a700 1 -- 172.21.15.70:0/4132264955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e041037f0 msgr2=0x7f2e04103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:44.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.949+0000 7f2e0b01a700 1 --2- 172.21.15.70:0/4132264955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e041037f0 0x7f2e04103bd0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2df80099c0 tx=0x7f2df8009cd0 comp rx=0 tx=0).stop 2024-06-04T05:00:44.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.949+0000 7f2e0b01a700 1 -- 172.21.15.70:0/4132264955 shutdown_connections 2024-06-04T05:00:44.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.949+0000 7f2e0b01a700 1 --2- 172.21.15.70:0/4132264955 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e041088f0 0x7f2e04110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:44.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.949+0000 7f2e0b01a700 1 --2- 172.21.15.70:0/4132264955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e041037f0 0x7f2e04103bd0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:44.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.949+0000 7f2e0b01a700 1 --2- 172.21.15.70:0/4132264955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e041041a0 0x7f2e041081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:44.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.949+0000 7f2e0b01a700 1 -- 172.21.15.70:0/4132264955 >> 172.21.15.70:0/4132264955 conn(0x7f2e0407a6a0 msgr2=0x7f2e0407aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:44.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.949+0000 7f2e0b01a700 1 -- 172.21.15.70:0/4132264955 shutdown_connections 2024-06-04T05:00:44.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.949+0000 7f2e0b01a700 1 -- 172.21.15.70:0/4132264955 wait complete. 2024-06-04T05:00:44.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.949+0000 7f2e0b01a700 1 Processor -- start 2024-06-04T05:00:44.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.949+0000 7f2e0b01a700 1 -- start start 2024-06-04T05:00:44.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e0b01a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e041037f0 0x7f2e04077430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:44.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e0b01a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e041041a0 0x7f2e04075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:44.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e0b01a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e041088f0 0x7f2e04075ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:44.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e0b01a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2e04113570 con 0x7f2e041088f0 2024-06-04T05:00:44.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e0b01a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e041133f0 con 0x7f2e041041a0 2024-06-04T05:00:44.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e0b01a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2e041136f0 con 0x7f2e041037f0 2024-06-04T05:00:44.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e095b7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e041088f0 0x7f2e04075ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:44.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e08db6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e041037f0 0x7f2e04077430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:44.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e095b7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e041088f0 0x7f2e04075ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56582/0 (socket says 172.21.15.70:56582) 2024-06-04T05:00:44.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e08db6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e041037f0 0x7f2e04077430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:33922/0 (socket says 172.21.15.70:33922) 2024-06-04T05:00:44.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e095b7700 1 -- 172.21.15.70:0/3593120051 learned_addr learned my addr 172.21.15.70:0/3593120051 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:44.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e03fff700 1 --2- 172.21.15.70:0/3593120051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e041041a0 0x7f2e04075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:44.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e095b7700 1 -- 172.21.15.70:0/3593120051 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e041037f0 msgr2=0x7f2e04077430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:44.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e095b7700 1 --2- 172.21.15.70:0/3593120051 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e041037f0 0x7f2e04077430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:44.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e095b7700 1 -- 172.21.15.70:0/3593120051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e041041a0 msgr2=0x7f2e04075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:44.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e095b7700 1 --2- 172.21.15.70:0/3593120051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e041041a0 0x7f2e04075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:44.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e095b7700 1 -- 172.21.15.70:0/3593120051 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e041ac4f0 con 0x7f2e041088f0 2024-06-04T05:00:44.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.950+0000 7f2e095b7700 1 --2- 172.21.15.70:0/3593120051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e041088f0 0x7f2e04075ea0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc007ee0 tx=0x7f2dfc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:44.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.951+0000 7f2e01ffb700 1 -- 172.21.15.70:0/3593120051 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc004bb0 con 0x7f2e041088f0 2024-06-04T05:00:44.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.951+0000 7f2e0b01a700 1 -- 172.21.15.70:0/3593120051 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e041ac780 con 0x7f2e041088f0 2024-06-04T05:00:44.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.951+0000 7f2e01ffb700 1 -- 172.21.15.70:0/3593120051 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2dfc004d50 con 0x7f2e041088f0 2024-06-04T05:00:44.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.951+0000 7f2e01ffb700 1 -- 172.21.15.70:0/3593120051 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc0058b0 con 0x7f2e041088f0 2024-06-04T05:00:44.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.951+0000 7f2e0b01a700 1 -- 172.21.15.70:0/3593120051 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e041acc90 con 0x7f2e041088f0 2024-06-04T05:00:44.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.952+0000 7f2e0b01a700 1 -- 172.21.15.70:0/3593120051 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e0410bb30 con 0x7f2e041088f0 2024-06-04T05:00:44.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.953+0000 7f2e01ffb700 1 -- 172.21.15.70:0/3593120051 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f2dfc01a020 con 0x7f2e041088f0 2024-06-04T05:00:44.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.954+0000 7f2e01ffb700 1 --2- 172.21.15.70:0/3593120051 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2df40b29f0 0x7f2df40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:44.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.954+0000 7f2e08db6700 1 --2- 172.21.15.70:0/3593120051 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2df40b29f0 0x7f2df40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:44.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.954+0000 7f2e01ffb700 1 -- 172.21.15.70:0/3593120051 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f2dfc0e1c50 con 0x7f2e041088f0 2024-06-04T05:00:44.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.954+0000 7f2e08db6700 1 --2- 172.21.15.70:0/3593120051 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2df40b29f0 0x7f2df40b4eb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2df8000c00 tx=0x7f2df8005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:44.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:44.958+0000 7f2e01ffb700 1 -- 172.21.15.70:0/3593120051 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2dfc0e7020 con 0x7f2e041088f0 2024-06-04T05:00:45.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:45.205+0000 7f2e0b01a700 1 -- 172.21.15.70:0/3593120051 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e0410b510 con 0x7f2e041088f0 2024-06-04T05:00:45.208 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:45 smithi070 ceph-mon[32631]: pgmap v110: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.7 KiB/s wr, 19 op/s 2024-06-04T05:00:45.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:45.206+0000 7f2e01ffb700 1 -- 172.21.15.70:0/3593120051 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11932 (secure 0 0 0) 0x7f2dfc010070 con 0x7f2e041088f0 2024-06-04T05:00:45.208 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:00:45.208 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:59:22.801688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:00:45.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:45.208+0000 7f2e0b01a700 1 -- 172.21.15.70:0/3593120051 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2df40b29f0 msgr2=0x7f2df40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:45.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:45.208+0000 7f2e0b01a700 1 --2- 172.21.15.70:0/3593120051 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2df40b29f0 0x7f2df40b4eb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2df8000c00 tx=0x7f2df8005fb0 comp rx=0 tx=0).stop 2024-06-04T05:00:45.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:45.208+0000 7f2e0b01a700 1 -- 172.21.15.70:0/3593120051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e041088f0 msgr2=0x7f2e04075ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:45.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:45.208+0000 7f2e0b01a700 1 --2- 172.21.15.70:0/3593120051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e041088f0 0x7f2e04075ea0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc007ee0 tx=0x7f2dfc00d3b0 comp rx=0 tx=0).stop 2024-06-04T05:00:45.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:45.208+0000 7f2e0b01a700 1 -- 172.21.15.70:0/3593120051 shutdown_connections 2024-06-04T05:00:45.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:45.208+0000 7f2e0b01a700 1 --2- 172.21.15.70:0/3593120051 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2df40b29f0 0x7f2df40b4eb0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:45.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:45.208+0000 7f2e0b01a700 1 --2- 172.21.15.70:0/3593120051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e041088f0 0x7f2e04075ea0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:45.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:45.208+0000 7f2e0b01a700 1 --2- 172.21.15.70:0/3593120051 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e041037f0 0x7f2e04077430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:45.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:45.208+0000 7f2e0b01a700 1 --2- 172.21.15.70:0/3593120051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e041041a0 0x7f2e04075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:45.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:45.208+0000 7f2e0b01a700 1 -- 172.21.15.70:0/3593120051 >> 172.21.15.70:0/3593120051 conn(0x7f2e0407a6a0 msgr2=0x7f2e04101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:45.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:45.208+0000 7f2e0b01a700 1 -- 172.21.15.70:0/3593120051 shutdown_connections 2024-06-04T05:00:45.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:45.208+0000 7f2e0b01a700 1 -- 172.21.15.70:0/3593120051 wait complete. 2024-06-04T05:00:45.211 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-06-04T05:00:45.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:45 smithi112 ceph-mon[36453]: pgmap v110: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.7 KiB/s wr, 19 op/s 2024-06-04T05:00:45.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:45 smithi098 ceph-mon[35569]: pgmap v110: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.7 KiB/s wr, 19 op/s 2024-06-04T05:00:45.716 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.50 MB/sec warmup 98 sec latency 22628.455 ms 2024-06-04T05:00:45.974 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id ae3a3592 --release --await' 2024-06-04T05:00:45.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id ae3a3592 --release --await 2024-06-04T05:00:46.219 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:00:46.313 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:46 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3593120051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:00:46.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:46 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3593120051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:00:46.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:46 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3593120051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:00:46.716 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.47 MB/sec warmup 99 sec latency 23628.504 ms 2024-06-04T05:00:47.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:47 smithi070 ceph-mon[32631]: pgmap v111: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 9.7 KiB/s wr, 23 op/s 2024-06-04T05:00:47.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:47 smithi112 ceph-mon[36453]: pgmap v111: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 9.7 KiB/s wr, 23 op/s 2024-06-04T05:00:47.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:47 smithi098 ceph-mon[35569]: pgmap v111: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 9.7 KiB/s wr, 23 op/s 2024-06-04T05:00:47.716 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42263 2.45 MB/sec warmup 100 sec latency 24628.531 ms 2024-06-04T05:00:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.009+0000 7f417c474700 1 -- 172.21.15.70:0/1485769292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41741084d0 msgr2=0x7f41741088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.009+0000 7f417c474700 1 --2- 172.21.15.70:0/1485769292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41741084d0 0x7f41741088b0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f417000b0a0 tx=0x7f417000b3b0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.009+0000 7f417c474700 1 -- 172.21.15.70:0/1485769292 shutdown_connections 2024-06-04T05:00:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.009+0000 7f417c474700 1 --2- 172.21.15.70:0/1485769292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4174102e90 0x7f417410f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.009+0000 7f417c474700 1 --2- 172.21.15.70:0/1485769292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41741084d0 0x7f41741088b0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.009+0000 7f417c474700 1 --2- 172.21.15.70:0/1485769292 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41741024d0 0x7f4174102950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.009+0000 7f417c474700 1 -- 172.21.15.70:0/1485769292 >> 172.21.15.70:0/1485769292 conn(0x7f4174078490 msgr2=0x7f41740788a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.009+0000 7f417c474700 1 -- 172.21.15.70:0/1485769292 shutdown_connections 2024-06-04T05:00:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.009+0000 7f417c474700 1 -- 172.21.15.70:0/1485769292 wait complete. 2024-06-04T05:00:48.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f417c474700 1 Processor -- start 2024-06-04T05:00:48.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f417c474700 1 -- start start 2024-06-04T05:00:48.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f417c474700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41741024d0 0x7f417419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f417c474700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4174102e90 0x7f417419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f417c474700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41741084d0 0x7f41741999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f417c474700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4174112510 con 0x7f41741024d0 2024-06-04T05:00:48.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f417c474700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4174112390 con 0x7f4174102e90 2024-06-04T05:00:48.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f417c474700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4174112690 con 0x7f41741084d0 2024-06-04T05:00:48.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f417a210700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41741024d0 0x7f417419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f417a210700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41741024d0 0x7f417419f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56688/0 (socket says 172.21.15.70:56688) 2024-06-04T05:00:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f417a210700 1 -- 172.21.15.70:0/2839682997 learned_addr learned my addr 172.21.15.70:0/2839682997 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f4179a0f700 1 --2- 172.21.15.70:0/2839682997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4174102e90 0x7f417419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f417aa11700 1 --2- 172.21.15.70:0/2839682997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41741084d0 0x7f41741999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f417a210700 1 -- 172.21.15.70:0/2839682997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41741084d0 msgr2=0x7f41741999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f417a210700 1 --2- 172.21.15.70:0/2839682997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41741084d0 0x7f41741999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f417a210700 1 -- 172.21.15.70:0/2839682997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4174102e90 msgr2=0x7f417419fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f417a210700 1 --2- 172.21.15.70:0/2839682997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4174102e90 0x7f417419fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.010+0000 7f417a210700 1 -- 172.21.15.70:0/2839682997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f417419a110 con 0x7f41741024d0 2024-06-04T05:00:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.011+0000 7f417a210700 1 --2- 172.21.15.70:0/2839682997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41741024d0 0x7f417419f7a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f41700062a0 tx=0x7f4170004dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.011+0000 7f416b7fe700 1 -- 172.21.15.70:0/2839682997 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170021070 con 0x7f41741024d0 2024-06-04T05:00:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.011+0000 7f417c474700 1 -- 172.21.15.70:0/2839682997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f417419a3a0 con 0x7f41741024d0 2024-06-04T05:00:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.011+0000 7f416b7fe700 1 -- 172.21.15.70:0/2839682997 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4170007c40 con 0x7f41741024d0 2024-06-04T05:00:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.011+0000 7f416b7fe700 1 -- 172.21.15.70:0/2839682997 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170005040 con 0x7f41741024d0 2024-06-04T05:00:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.011+0000 7f417c474700 1 -- 172.21.15.70:0/2839682997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41741a6670 con 0x7f41741024d0 2024-06-04T05:00:48.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.014+0000 7f416b7fe700 1 -- 172.21.15.70:0/2839682997 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f4170024070 con 0x7f41741024d0 2024-06-04T05:00:48.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.015+0000 7f416b7fe700 1 --2- 172.21.15.70:0/2839682997 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f41600b29f0 0x7f41600b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.015+0000 7f4179a0f700 1 --2- 172.21.15.70:0/2839682997 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f41600b29f0 0x7f41600b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.015+0000 7f416b7fe700 1 -- 172.21.15.70:0/2839682997 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f417000f020 con 0x7f41741024d0 2024-06-04T05:00:48.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.015+0000 7f4186d09b80 1 Processor -- start 2024-06-04T05:00:48.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.015+0000 7f4179a0f700 1 --2- 172.21.15.70:0/2839682997 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f41600b29f0 0x7f41600b4eb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4174078c40 tx=0x7f4164009190 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:48.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.015+0000 7f4186d09b80 1 -- start start 2024-06-04T05:00:48.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.015+0000 7f4186d09b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499af720 0x55ba499afb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f4186d09b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499aabf0 0x55ba499ab070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f4186d09b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499ab5b0 0x55ba499b7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f4186d09b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ba49908e70 con 0x55ba499af720 2024-06-04T05:00:48.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f4186d09b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ba498f8a10 con 0x55ba499ab5b0 2024-06-04T05:00:48.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f4186d09b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ba498fca50 con 0x55ba499aabf0 2024-06-04T05:00:48.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f417a210700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499af720 0x55ba499afb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f417a210700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499af720 0x55ba499afb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56704/0 (socket says 172.21.15.70:56704) 2024-06-04T05:00:48.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f417a210700 1 -- 172.21.15.70:0/2629401960 learned_addr learned my addr 172.21.15.70:0/2629401960 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:48.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f4179a0f700 1 --2- 172.21.15.70:0/2629401960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499aabf0 0x55ba499ab070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f417aa11700 1 --2- 172.21.15.70:0/2629401960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499ab5b0 0x55ba499b7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f417a210700 1 -- 172.21.15.70:0/2629401960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499aabf0 msgr2=0x55ba499ab070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:48.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f417a210700 1 --2- 172.21.15.70:0/2629401960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499aabf0 0x55ba499ab070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f417a210700 1 -- 172.21.15.70:0/2629401960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499ab5b0 msgr2=0x55ba499b7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:48.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f417a210700 1 --2- 172.21.15.70:0/2629401960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499ab5b0 0x55ba499b7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f417a210700 1 -- 172.21.15.70:0/2629401960 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba498a20b0 con 0x55ba499af720 2024-06-04T05:00:48.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f417a210700 1 --2- 172.21.15.70:0/2629401960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499af720 0x55ba499afb00 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f41700aaa30 tx=0x7f417000cfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac204aee47d4176 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:48.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f4168d70700 1 -- 172.21.15.70:0/2629401960 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41700aad30 con 0x55ba499af720 2024-06-04T05:00:48.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f4168d70700 1 -- 172.21.15.70:0/2629401960 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f41700af0e0 con 0x55ba499af720 2024-06-04T05:00:48.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.016+0000 7f4168d70700 1 -- 172.21.15.70:0/2629401960 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41700aaf90 con 0x55ba499af720 2024-06-04T05:00:48.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.017+0000 7f4186d09b80 1 -- 172.21.15.70:0/2629401960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499af720 msgr2=0x55ba499afb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:48.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.017+0000 7f4186d09b80 1 --2- 172.21.15.70:0/2629401960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499af720 0x55ba499afb00 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f41700aaa30 tx=0x7f417000cfb0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.017+0000 7f4186d09b80 1 -- 172.21.15.70:0/2629401960 shutdown_connections 2024-06-04T05:00:48.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.017+0000 7f4186d09b80 1 --2- 172.21.15.70:0/2629401960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499ab5b0 0x55ba499b7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.017+0000 7f4186d09b80 1 --2- 172.21.15.70:0/2629401960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499af720 0x55ba499afb00 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.017+0000 7f4186d09b80 1 --2- 172.21.15.70:0/2629401960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499aabf0 0x55ba499ab070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.017+0000 7f4186d09b80 1 -- 172.21.15.70:0/2629401960 >> 172.21.15.70:0/2629401960 conn(0x55ba498d6340 msgr2=0x55ba499a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:48.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.017+0000 7f4186d09b80 1 -- 172.21.15.70:0/2629401960 shutdown_connections 2024-06-04T05:00:48.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.017+0000 7f4186d09b80 1 -- 172.21.15.70:0/2629401960 wait complete. 2024-06-04T05:00:48.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.018+0000 7f4186d09b80 1 -- start start 2024-06-04T05:00:48.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.018+0000 7f4186d09b80 1 Processor -- start 2024-06-04T05:00:48.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.018+0000 7f4186d09b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499aabf0 0x55ba499b7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.018+0000 7f4186d09b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499ab5b0 0x55ba499b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.018+0000 7f4186d09b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499af720 0x55ba499b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.018+0000 7f4186d09b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ba497ff0a0 con 0x55ba499aabf0 2024-06-04T05:00:48.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.018+0000 7f4186d09b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ba49860ea0 con 0x55ba499af720 2024-06-04T05:00:48.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.018+0000 7f4186d09b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ba49908e70 con 0x55ba499ab5b0 2024-06-04T05:00:48.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.018+0000 7f4186d09b80 10 client.0 _ll_register_callbacks cb 0x55ba49893a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:48.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.018+0000 7f417a210700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499aabf0 0x55ba499b7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.018+0000 7f417a210700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499aabf0 0x55ba499b7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56712/0 (socket says 172.21.15.70:56712) 2024-06-04T05:00:48.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.018+0000 7f417a210700 1 -- 172.21.15.70:0/3982279348 learned_addr learned my addr 172.21.15.70:0/3982279348 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:48.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.018+0000 7f4179a0f700 1 --2- 172.21.15.70:0/3982279348 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499ab5b0 0x55ba499b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.018+0000 7f417aa11700 1 --2- 172.21.15.70:0/3982279348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499af720 0x55ba499b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f417a210700 1 -- 172.21.15.70:0/3982279348 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499ab5b0 msgr2=0x55ba499b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:48.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f417a210700 1 --2- 172.21.15.70:0/3982279348 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499ab5b0 0x55ba499b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f417a210700 1 -- 172.21.15.70:0/3982279348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499af720 msgr2=0x55ba499b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:48.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f417a210700 1 --2- 172.21.15.70:0/3982279348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499af720 0x55ba499b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f417a210700 1 -- 172.21.15.70:0/3982279348 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba498489d0 con 0x55ba499aabf0 2024-06-04T05:00:48.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f417a210700 1 --2- 172.21.15.70:0/3982279348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499aabf0 0x55ba499b7270 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f41700e11c0 tx=0x7f417000b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f0e1d07b2f1eb02 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:48.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f415dffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:48.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f415dffb700 1 -- 172.21.15.70:0/3982279348 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41700c5390 con 0x55ba499aabf0 2024-06-04T05:00:48.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f415dffb700 1 -- 172.21.15.70:0/3982279348 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f41700aaf90 con 0x55ba499aabf0 2024-06-04T05:00:48.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f4186d09b80 1 -- 172.21.15.70:0/3982279348 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41700af0e0 con 0x55ba499aabf0 2024-06-04T05:00:48.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f415dffb700 1 -- 172.21.15.70:0/3982279348 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f41700255c0 con 0x55ba499aabf0 2024-06-04T05:00:48.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f4186d09b80 10 client.16467 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:48.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f4186d09b80 1 -- 172.21.15.70:0/3982279348 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ba498b2860 con 0x55ba499aabf0 2024-06-04T05:00:48.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f415dffb700 1 -- 172.21.15.70:0/3982279348 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f41700af0e0 con 0x55ba499aabf0 2024-06-04T05:00:48.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f415dffb700 1 -- 172.21.15.70:0/3982279348 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f4170011070 con 0x55ba499aabf0 2024-06-04T05:00:48.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f4186d09b80 10 client.16467 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:48.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f4186d09b80 10 client.16467 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:00:48.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.019+0000 7f4186d09b80 20 client.16467 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:48.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.020+0000 7f4186d09b80 1 --2- 172.21.15.70:0/3982279348 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ba499c43b0 0x55ba499c67a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.020+0000 7f4186d09b80 4 client.16467 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:00:48.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.020+0000 7f4186d09b80 1 -- 172.21.15.70:0/3982279348 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba4990adc0 con 0x55ba499c43b0 2024-06-04T05:00:48.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.020+0000 7f4179a0f700 1 --2- 172.21.15.70:0/3982279348 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ba499c43b0 0x55ba499c67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.020+0000 7f4179a0f700 1 --2- 172.21.15.70:0/3982279348 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ba499c43b0 0x55ba499c67a0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3bfb44f0eb257ff server_cookie=6d1280e37f4f6463 in_seq=0 out_seq=0 2024-06-04T05:00:48.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.020+0000 7f415dffb700 10 client.16467 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:48.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.021+0000 7f415dffb700 1 -- 172.21.15.70:0/3982279348 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ba4990adc0 con 0x55ba499c43b0 2024-06-04T05:00:48.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.021+0000 7f415dffb700 10 client.16467 handle_command_reply: tid=0 2024-06-04T05:00:48.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.021+0000 7f4186d09b80 1 client.16467 shutdown 2024-06-04T05:00:48.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.021+0000 7f4186d09b80 1 -- 172.21.15.70:0/3982279348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499aabf0 msgr2=0x55ba499b7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:48.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.021+0000 7f4186d09b80 1 --2- 172.21.15.70:0/3982279348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499aabf0 0x55ba499b7270 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f41700e11c0 tx=0x7f417000b3b0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.021+0000 7f4186d09b80 1 -- 172.21.15.70:0/3982279348 shutdown_connections 2024-06-04T05:00:48.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.021+0000 7f4186d09b80 1 --2- 172.21.15.70:0/3982279348 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ba499c43b0 0x55ba499c67a0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.021+0000 7f4186d09b80 1 --2- 172.21.15.70:0/3982279348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499af720 0x55ba499b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.021+0000 7f4186d09b80 1 --2- 172.21.15.70:0/3982279348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499aabf0 0x55ba499b7270 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.021+0000 7f4186d09b80 1 --2- 172.21.15.70:0/3982279348 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499ab5b0 0x55ba499b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.021+0000 7f415dffb700 0 client.16467 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:48.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.021+0000 7f4186d09b80 1 -- 172.21.15.70:0/3982279348 >> 172.21.15.70:0/3982279348 conn(0x55ba498d6340 msgr2=0x55ba499a9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:48.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.021+0000 7f4186d09b80 1 -- 172.21.15.70:0/3982279348 shutdown_connections 2024-06-04T05:00:48.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.021+0000 7f4186d09b80 1 -- 172.21.15.70:0/3982279348 wait complete. 2024-06-04T05:00:48.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.021+0000 7f4186d09b80 20 client.16467 trim_cache size 0 max 16384 2024-06-04T05:00:48.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.041+0000 7f4186d09b80 1 Processor -- start 2024-06-04T05:00:48.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.041+0000 7f4186d09b80 1 -- start start 2024-06-04T05:00:48.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.041+0000 7f4186d09b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499aabf0 0x55ba499c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.041+0000 7f4186d09b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499ab5b0 0x55ba4999fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.041+0000 7f4186d09b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499af720 0x55ba499b0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.041+0000 7f4186d09b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f41741adc90 con 0x55ba499af720 2024-06-04T05:00:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.041+0000 7f4186d09b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f41741af830 con 0x55ba499ab5b0 2024-06-04T05:00:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.042+0000 7f4186d09b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f41741ae360 con 0x55ba499aabf0 2024-06-04T05:00:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.042+0000 7f417aa11700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499af720 0x55ba499b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.042+0000 7f417aa11700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499af720 0x55ba499b0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56718/0 (socket says 172.21.15.70:56718) 2024-06-04T05:00:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.042+0000 7f417aa11700 1 -- 172.21.15.70:0/606739126 learned_addr learned my addr 172.21.15.70:0/606739126 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.042+0000 7f4179a0f700 1 --2- 172.21.15.70:0/606739126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499ab5b0 0x55ba4999fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.042+0000 7f417a210700 1 --2- 172.21.15.70:0/606739126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499aabf0 0x55ba499c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.042+0000 7f417aa11700 1 -- 172.21.15.70:0/606739126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499aabf0 msgr2=0x55ba499c0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.042+0000 7f417aa11700 1 --2- 172.21.15.70:0/606739126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499aabf0 0x55ba499c0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.042+0000 7f417aa11700 1 -- 172.21.15.70:0/606739126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499ab5b0 msgr2=0x55ba4999fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.042+0000 7f417aa11700 1 --2- 172.21.15.70:0/606739126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499ab5b0 0x55ba4999fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.042+0000 7f417aa11700 1 -- 172.21.15.70:0/606739126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba499af2a0 con 0x55ba499af720 2024-06-04T05:00:48.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.042+0000 7f417aa11700 1 --2- 172.21.15.70:0/606739126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499af720 0x55ba499b0480 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f416c00b830 tx=0x7f416c00bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=535ae7c39ce1651c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:48.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.042+0000 7f415ffff700 1 -- 172.21.15.70:0/606739126 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c003800 con 0x55ba499af720 2024-06-04T05:00:48.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.042+0000 7f415ffff700 1 -- 172.21.15.70:0/606739126 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f416c003e30 con 0x55ba499af720 2024-06-04T05:00:48.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.042+0000 7f415ffff700 1 -- 172.21.15.70:0/606739126 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c011e30 con 0x55ba499af720 2024-06-04T05:00:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.043+0000 7f4186d09b80 1 -- 172.21.15.70:0/606739126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499af720 msgr2=0x55ba499b0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.043+0000 7f4186d09b80 1 --2- 172.21.15.70:0/606739126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499af720 0x55ba499b0480 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f416c00b830 tx=0x7f416c00bb40 comp rx=0 tx=0).stop 2024-06-04T05:00:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.043+0000 7f4186d09b80 1 -- 172.21.15.70:0/606739126 shutdown_connections 2024-06-04T05:00:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.043+0000 7f4186d09b80 1 --2- 172.21.15.70:0/606739126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499af720 0x55ba499b0480 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.043+0000 7f4186d09b80 1 --2- 172.21.15.70:0/606739126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499aabf0 0x55ba499c0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.043+0000 7f4186d09b80 1 --2- 172.21.15.70:0/606739126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499ab5b0 0x55ba4999fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.043+0000 7f4186d09b80 1 -- 172.21.15.70:0/606739126 >> 172.21.15.70:0/606739126 conn(0x55ba498d6340 msgr2=0x55ba499a5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.043+0000 7f4186d09b80 1 -- 172.21.15.70:0/606739126 shutdown_connections 2024-06-04T05:00:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.043+0000 7f4186d09b80 1 -- 172.21.15.70:0/606739126 wait complete. 2024-06-04T05:00:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.043+0000 7f4186d09b80 1 -- start start 2024-06-04T05:00:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.043+0000 7f4186d09b80 1 Processor -- start 2024-06-04T05:00:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f4186d09b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499aabf0 0x55ba499bce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f4186d09b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499ab5b0 0x55ba499bd350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f4186d09b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499af720 0x55ba499ac280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f4186d09b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f41741af100 con 0x55ba499ab5b0 2024-06-04T05:00:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f4186d09b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f41741aea30 con 0x55ba499aabf0 2024-06-04T05:00:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f4186d09b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f41741abc30 con 0x55ba499af720 2024-06-04T05:00:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f4186d09b80 10 client.0 _ll_register_callbacks cb 0x55ba49893a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f4179a0f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499ab5b0 0x55ba499bd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f4179a0f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499ab5b0 0x55ba499bd350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56728/0 (socket says 172.21.15.70:56728) 2024-06-04T05:00:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f4179a0f700 1 -- 172.21.15.70:0/2141869044 learned_addr learned my addr 172.21.15.70:0/2141869044 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f417a210700 1 --2- 172.21.15.70:0/2141869044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499aabf0 0x55ba499bce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f417aa11700 1 --2- 172.21.15.70:0/2141869044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499af720 0x55ba499ac280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f4179a0f700 1 -- 172.21.15.70:0/2141869044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499af720 msgr2=0x55ba499ac280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:48.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f4179a0f700 1 --2- 172.21.15.70:0/2141869044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499af720 0x55ba499ac280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f4179a0f700 1 -- 172.21.15.70:0/2141869044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499aabf0 msgr2=0x55ba499bce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:48.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f4179a0f700 1 --2- 172.21.15.70:0/2141869044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499aabf0 0x55ba499bce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f4179a0f700 1 -- 172.21.15.70:0/2141869044 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba499acac0 con 0x55ba499ab5b0 2024-06-04T05:00:48.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f417aa11700 1 --2- 172.21.15.70:0/2141869044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499af720 0x55ba499ac280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:00:48.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f4179a0f700 1 --2- 172.21.15.70:0/2141869044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499ab5b0 0x55ba499bd350 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f41640097f0 tx=0x7f4164004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aaed411ad0ee03a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:48.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f415dffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:48.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f415dffb700 1 -- 172.21.15.70:0/2141869044 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164031060 con 0x55ba499ab5b0 2024-06-04T05:00:48.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f415dffb700 1 -- 172.21.15.70:0/2141869044 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f4164015a70 con 0x55ba499ab5b0 2024-06-04T05:00:48.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.044+0000 7f4186d09b80 1 -- 172.21.15.70:0/2141869044 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f416c003e30 con 0x55ba499ab5b0 2024-06-04T05:00:48.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.045+0000 7f415dffb700 1 -- 172.21.15.70:0/2141869044 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f416401b040 con 0x55ba499ab5b0 2024-06-04T05:00:48.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.045+0000 7f4186d09b80 10 client.16473 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:48.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.045+0000 7f4186d09b80 1 -- 172.21.15.70:0/2141869044 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ba499ade60 con 0x55ba499ab5b0 2024-06-04T05:00:48.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.045+0000 7f415dffb700 1 -- 172.21.15.70:0/2141869044 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f416c003e30 con 0x55ba499ab5b0 2024-06-04T05:00:48.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.045+0000 7f415dffb700 1 -- 172.21.15.70:0/2141869044 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f4164022430 con 0x55ba499ab5b0 2024-06-04T05:00:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.045+0000 7f4186d09b80 10 client.16473 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.045+0000 7f4186d09b80 10 client.16473 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:00:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.045+0000 7f4186d09b80 20 client.16473 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.045+0000 7f4186d09b80 1 --2- 172.21.15.70:0/2141869044 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ba49a302f0 0x55ba49a32710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.045+0000 7f4186d09b80 4 client.16473 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ae3a3592", "release": true, "await": true}] 2024-06-04T05:00:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.045+0000 7f4186d09b80 1 -- 172.21.15.70:0/2141869044 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ae3a3592", "release": true, "await": true}) -- 0x55ba499045a0 con 0x55ba49a302f0 2024-06-04T05:00:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.045+0000 7f417a210700 1 --2- 172.21.15.70:0/2141869044 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ba49a302f0 0x55ba49a32710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.046+0000 7f415dffb700 10 client.16473 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.046+0000 7f417a210700 1 --2- 172.21.15.70:0/2141869044 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ba49a302f0 0x55ba49a32710 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19e1ab7e75fdaa44 server_cookie=5a1e2f5e7ef86fda in_seq=0 out_seq=0 2024-06-04T05:00:48.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.051+0000 7f415dffb700 1 -- 172.21.15.70:0/2141869044 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55ba499045a0 con 0x55ba49a302f0 2024-06-04T05:00:48.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.051+0000 7f415dffb700 10 client.16473 handle_command_reply: tid=0 2024-06-04T05:00:48.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.052+0000 7f4186d09b80 1 client.16473 shutdown 2024-06-04T05:00:48.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.052+0000 7f4186d09b80 1 -- 172.21.15.70:0/2141869044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499ab5b0 msgr2=0x55ba499bd350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.052+0000 7f4186d09b80 1 --2- 172.21.15.70:0/2141869044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499ab5b0 0x55ba499bd350 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f41640097f0 tx=0x7f4164004750 comp rx=0 tx=0).stop 2024-06-04T05:00:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.052+0000 7f4186d09b80 1 -- 172.21.15.70:0/2141869044 shutdown_connections 2024-06-04T05:00:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.052+0000 7f4186d09b80 1 --2- 172.21.15.70:0/2141869044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba499af720 0x55ba499ac280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.052+0000 7f4186d09b80 1 --2- 172.21.15.70:0/2141869044 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ba49a302f0 0x55ba49a32710 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.052+0000 7f4186d09b80 1 --2- 172.21.15.70:0/2141869044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba499aabf0 0x55ba499bce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.052+0000 7f4186d09b80 1 --2- 172.21.15.70:0/2141869044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba499ab5b0 0x55ba499bd350 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.052+0000 7f4186d09b80 1 -- 172.21.15.70:0/2141869044 >> 172.21.15.70:0/2141869044 conn(0x55ba498d6340 msgr2=0x55ba499b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.052+0000 7f415dffb700 0 client.16473 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.052+0000 7f4186d09b80 1 -- 172.21.15.70:0/2141869044 shutdown_connections 2024-06-04T05:00:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.052+0000 7f4186d09b80 1 -- 172.21.15.70:0/2141869044 wait complete. 2024-06-04T05:00:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.052+0000 7f4186d09b80 20 client.16473 trim_cache size 0 max 16384 2024-06-04T05:00:48.056 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:00:48.056 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 31, 2024-06-04T05:00:48.056 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:00:48.056 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 8, 2024-06-04T05:00:48.056 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:00:48.056 INFO:teuthology.orchestra.run.smithi070.stdout: "ae3a3592": { 2024-06-04T05:00:48.056 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 8, 2024-06-04T05:00:48.056 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:00:48.056 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:00:48.056 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:00:48.057 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:00:48.057 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:00:48.057 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:00:48.057 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 149.1, 2024-06-04T05:00:48.057 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:00:48.057 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:00:48.057 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:00:48.057 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:00:48.057 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:00:48.057 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:00:48.057 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:00:48.057 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:00:48.057 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:00:48.058 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:00:48.058 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:00:48.058 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:00:48.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.053+0000 7f417c474700 1 -- 172.21.15.70:0/2839682997 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f41600b29f0 msgr2=0x7f41600b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:48.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.053+0000 7f417c474700 1 --2- 172.21.15.70:0/2839682997 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f41600b29f0 0x7f41600b4eb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4174078c40 tx=0x7f4164009190 comp rx=0 tx=0).stop 2024-06-04T05:00:48.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.053+0000 7f417c474700 1 -- 172.21.15.70:0/2839682997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41741024d0 msgr2=0x7f417419f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:48.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.053+0000 7f417c474700 1 --2- 172.21.15.70:0/2839682997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41741024d0 0x7f417419f7a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f41700062a0 tx=0x7f4170004dd0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.053+0000 7f417c474700 1 -- 172.21.15.70:0/2839682997 shutdown_connections 2024-06-04T05:00:48.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.053+0000 7f417c474700 1 --2- 172.21.15.70:0/2839682997 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f41600b29f0 0x7f41600b4eb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.053+0000 7f417c474700 1 --2- 172.21.15.70:0/2839682997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41741084d0 0x7f41741999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.053+0000 7f417c474700 1 --2- 172.21.15.70:0/2839682997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41741024d0 0x7f417419f7a0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.053+0000 7f417c474700 1 --2- 172.21.15.70:0/2839682997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4174102e90 0x7f417419fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:48.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.053+0000 7f417c474700 1 -- 172.21.15.70:0/2839682997 >> 172.21.15.70:0/2839682997 conn(0x7f4174078490 msgr2=0x7f417406c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:48.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.053+0000 7f417c474700 1 -- 172.21.15.70:0/2839682997 shutdown_connections 2024-06-04T05:00:48.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:48.053+0000 7f417c474700 1 -- 172.21.15.70:0/2839682997 wait complete. 2024-06-04T05:00:48.716 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42446 2.43 MB/sec warmup 101 sec latency 24972.224 ms 2024-06-04T05:00:48.954 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ae3a3592', total seconds elapsed: 35.1 2024-06-04T05:00:48.954 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.05 2024-06-04T05:00:48.954 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:00:49.217 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:49 smithi070 ceph-mon[32631]: pgmap v112: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.2 KiB/s wr, 11 op/s 2024-06-04T05:00:49.222 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:00:49.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:49 smithi112 ceph-mon[36453]: pgmap v112: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.2 KiB/s wr, 11 op/s 2024-06-04T05:00:49.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:49 smithi098 ceph-mon[35569]: pgmap v112: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.2 KiB/s wr, 11 op/s 2024-06-04T05:00:49.716 INFO:tasks.workunit.client.0.smithi070.stdout: 1 42920 2.45 MB/sec warmup 102 sec latency 12.977 ms 2024-06-04T05:00:50.716 INFO:tasks.workunit.client.0.smithi070.stdout: 1 43199 2.43 MB/sec warmup 103 sec latency 15.613 ms 2024-06-04T05:00:51.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.095+0000 7f9d19b43700 1 -- 172.21.15.70:0/528644218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d1406b220 msgr2=0x7f9d1406b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:51.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.095+0000 7f9d19b43700 1 --2- 172.21.15.70:0/528644218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d1406b220 0x7f9d1406b6a0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f9d08009990 tx=0x7f9d08009ca0 comp rx=0 tx=0).stop 2024-06-04T05:00:51.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.095+0000 7f9d19b43700 1 -- 172.21.15.70:0/528644218 shutdown_connections 2024-06-04T05:00:51.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.095+0000 7f9d19b43700 1 --2- 172.21.15.70:0/528644218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d1406bbe0 0x7f9d141103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:51.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.095+0000 7f9d19b43700 1 --2- 172.21.15.70:0/528644218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d141095b0 0x7f9d14109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:51.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.095+0000 7f9d19b43700 1 --2- 172.21.15.70:0/528644218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d1406b220 0x7f9d1406b6a0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:51.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.095+0000 7f9d19b43700 1 -- 172.21.15.70:0/528644218 >> 172.21.15.70:0/528644218 conn(0x7f9d140793f0 msgr2=0x7f9d14079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:51.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.095+0000 7f9d19b43700 1 -- 172.21.15.70:0/528644218 shutdown_connections 2024-06-04T05:00:51.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.095+0000 7f9d19b43700 1 -- 172.21.15.70:0/528644218 wait complete. 2024-06-04T05:00:51.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.095+0000 7f9d19b43700 1 Processor -- start 2024-06-04T05:00:51.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d19b43700 1 -- start start 2024-06-04T05:00:51.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d19b43700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d1406b220 0x7f9d1419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:51.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d19b43700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d1406bbe0 0x7f9d1419fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:51.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d19b43700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d141095b0 0x7f9d141999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:51.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d19b43700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d14113590 con 0x7f9d1406bbe0 2024-06-04T05:00:51.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d19b43700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9d14113410 con 0x7f9d141095b0 2024-06-04T05:00:51.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d19b43700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9d14113710 con 0x7f9d1406b220 2024-06-04T05:00:51.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d137fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d1406b220 0x7f9d1419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:51.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d137fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d1406b220 0x7f9d1419f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41496/0 (socket says 172.21.15.70:41496) 2024-06-04T05:00:51.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d137fe700 1 -- 172.21.15.70:0/241330271 learned_addr learned my addr 172.21.15.70:0/241330271 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:51.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d12ffd700 1 --2- 172.21.15.70:0/241330271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d1406bbe0 0x7f9d1419fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:51.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d13fff700 1 --2- 172.21.15.70:0/241330271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d141095b0 0x7f9d141999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:51.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d137fe700 1 -- 172.21.15.70:0/241330271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d141095b0 msgr2=0x7f9d141999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:51.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d137fe700 1 --2- 172.21.15.70:0/241330271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d141095b0 0x7f9d141999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:51.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d137fe700 1 -- 172.21.15.70:0/241330271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d1406bbe0 msgr2=0x7f9d1419fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:51.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d137fe700 1 --2- 172.21.15.70:0/241330271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d1406bbe0 0x7f9d1419fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:51.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d137fe700 1 -- 172.21.15.70:0/241330271 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1419a120 con 0x7f9d1406b220 2024-06-04T05:00:51.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.096+0000 7f9d12ffd700 1 --2- 172.21.15.70:0/241330271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d1406bbe0 0x7f9d1419fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:51.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.097+0000 7f9d137fe700 1 --2- 172.21.15.70:0/241330271 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d1406b220 0x7f9d1419f800 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400eb30 tx=0x7f9d0400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:51.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.097+0000 7f9d10ff9700 1 -- 172.21.15.70:0/241330271 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0400cc90 con 0x7f9d1406b220 2024-06-04T05:00:51.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.097+0000 7f9d10ff9700 1 -- 172.21.15.70:0/241330271 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d0400ce30 con 0x7f9d1406b220 2024-06-04T05:00:51.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.097+0000 7f9d10ff9700 1 -- 172.21.15.70:0/241330271 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d04003e10 con 0x7f9d1406b220 2024-06-04T05:00:51.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.097+0000 7f9d19b43700 1 -- 172.21.15.70:0/241330271 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d1419a410 con 0x7f9d1406b220 2024-06-04T05:00:51.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.097+0000 7f9d19b43700 1 -- 172.21.15.70:0/241330271 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d141a66d0 con 0x7f9d1406b220 2024-06-04T05:00:51.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.098+0000 7f9d19b43700 1 -- 172.21.15.70:0/241330271 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d1406a230 con 0x7f9d1406b220 2024-06-04T05:00:51.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.099+0000 7f9d10ff9700 1 -- 172.21.15.70:0/241330271 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f9d04004000 con 0x7f9d1406b220 2024-06-04T05:00:51.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.100+0000 7f9d10ff9700 1 --2- 172.21.15.70:0/241330271 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9cfc0b2a70 0x7f9cfc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:51.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.100+0000 7f9d12ffd700 1 --2- 172.21.15.70:0/241330271 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9cfc0b2a70 0x7f9cfc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:51.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.100+0000 7f9d10ff9700 1 -- 172.21.15.70:0/241330271 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f9d040e27f0 con 0x7f9d1406b220 2024-06-04T05:00:51.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.101+0000 7f9d12ffd700 1 --2- 172.21.15.70:0/241330271 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9cfc0b2a70 0x7f9cfc0b4f30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9d141085d0 tx=0x7f9d0801a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:51.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.104+0000 7f9d10ff9700 1 -- 172.21.15.70:0/241330271 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9d040aa1d0 con 0x7f9d1406b220 2024-06-04T05:00:51.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:51 smithi070 ceph-mon[32631]: pgmap v113: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.2 KiB/s wr, 11 op/s 2024-06-04T05:00:51.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.402+0000 7f9d19b43700 1 -- 172.21.15.70:0/241330271 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d1419aef0 con 0x7f9d1406b220 2024-06-04T05:00:51.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.402+0000 7f9d10ff9700 1 -- 172.21.15.70:0/241330271 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11932 (secure 0 0 0) 0x7f9d040a9ad0 con 0x7f9d1406b220 2024-06-04T05:00:51.405 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:00:51.405 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:59:22.801688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:00:51.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.405+0000 7f9d19b43700 1 -- 172.21.15.70:0/241330271 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9cfc0b2a70 msgr2=0x7f9cfc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:51.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.405+0000 7f9d19b43700 1 --2- 172.21.15.70:0/241330271 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9cfc0b2a70 0x7f9cfc0b4f30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9d141085d0 tx=0x7f9d0801a040 comp rx=0 tx=0).stop 2024-06-04T05:00:51.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.405+0000 7f9d19b43700 1 -- 172.21.15.70:0/241330271 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d1406b220 msgr2=0x7f9d1419f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:51.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.405+0000 7f9d19b43700 1 --2- 172.21.15.70:0/241330271 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d1406b220 0x7f9d1419f800 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400eb30 tx=0x7f9d0400ee40 comp rx=0 tx=0).stop 2024-06-04T05:00:51.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.405+0000 7f9d19b43700 1 -- 172.21.15.70:0/241330271 shutdown_connections 2024-06-04T05:00:51.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.405+0000 7f9d19b43700 1 --2- 172.21.15.70:0/241330271 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9cfc0b2a70 0x7f9cfc0b4f30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:51.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.405+0000 7f9d19b43700 1 --2- 172.21.15.70:0/241330271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d141095b0 0x7f9d141999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:51.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.405+0000 7f9d19b43700 1 --2- 172.21.15.70:0/241330271 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d1406b220 0x7f9d1419f800 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:51.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.405+0000 7f9d19b43700 1 --2- 172.21.15.70:0/241330271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d1406bbe0 0x7f9d1419fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:51.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.405+0000 7f9d19b43700 1 -- 172.21.15.70:0/241330271 >> 172.21.15.70:0/241330271 conn(0x7f9d140793f0 msgr2=0x7f9d141012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:51.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.405+0000 7f9d19b43700 1 -- 172.21.15.70:0/241330271 shutdown_connections 2024-06-04T05:00:51.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:51.405+0000 7f9d19b43700 1 -- 172.21.15.70:0/241330271 wait complete. 2024-06-04T05:00:51.408 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-06-04T05:00:51.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:51 smithi112 ceph-mon[36453]: pgmap v113: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.2 KiB/s wr, 11 op/s 2024-06-04T05:00:51.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:51 smithi098 ceph-mon[35569]: pgmap v113: 97 pgs: 97 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.2 KiB/s wr, 11 op/s 2024-06-04T05:00:51.716 INFO:tasks.workunit.client.0.smithi070.stdout: 1 43719 2.43 MB/sec warmup 104 sec latency 15.440 ms 2024-06-04T05:00:52.250 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 149.05 --await-for 45.0' 2024-06-04T05:00:52.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 149.05 --await-for 45.0 2024-06-04T05:00:52.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:52 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/241330271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:00:52.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:52 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/241330271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:00:52.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:52 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/241330271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:00:52.522 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:00:52.716 INFO:tasks.workunit.client.0.smithi070.stdout: 1 44304 2.44 MB/sec warmup 105 sec latency 14.766 ms 2024-06-04T05:00:53.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:53 smithi070 ceph-mon[32631]: pgmap v114: 97 pgs: 97 active+clean; 40 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 307 KiB/s wr, 47 op/s 2024-06-04T05:00:53.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:53 smithi112 ceph-mon[36453]: pgmap v114: 97 pgs: 97 active+clean; 40 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 307 KiB/s wr, 47 op/s 2024-06-04T05:00:53.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:53 smithi098 ceph-mon[35569]: pgmap v114: 97 pgs: 97 active+clean; 40 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 307 KiB/s wr, 47 op/s 2024-06-04T05:00:53.716 INFO:tasks.workunit.client.0.smithi070.stdout: 1 44990 2.43 MB/sec warmup 106 sec latency 15.717 ms 2024-06-04T05:00:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.817+0000 7f4206864640 1 -- 172.21.15.70:0/1817042002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4200101980 msgr2=0x7f4200101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.817+0000 7f4206864640 1 --2- 172.21.15.70:0/1817042002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4200101980 0x7f4200101e00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f41f4009900 tx=0x7f41f402a3f0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.817+0000 7f4206864640 1 -- 172.21.15.70:0/1817042002 shutdown_connections 2024-06-04T05:00:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.817+0000 7f4206864640 1 --2- 172.21.15.70:0/1817042002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4200101980 0x7f4200101e00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.817+0000 7f4206864640 1 --2- 172.21.15.70:0/1817042002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4200102340 0x7f420010e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.817+0000 7f4206864640 1 --2- 172.21.15.70:0/1817042002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4200107940 0x7f4200107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.817+0000 7f4206864640 1 -- 172.21.15.70:0/1817042002 >> 172.21.15.70:0/1817042002 conn(0x7f4200078ad0 msgr2=0x7f4200078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.817+0000 7f4206864640 1 -- 172.21.15.70:0/1817042002 shutdown_connections 2024-06-04T05:00:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.817+0000 7f4206864640 1 -- 172.21.15.70:0/1817042002 wait complete. 2024-06-04T05:00:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.818+0000 7f4206864640 1 Processor -- start 2024-06-04T05:00:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.818+0000 7f4206864640 1 -- start start 2024-06-04T05:00:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.818+0000 7f4206864640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4200101980 0x7f4200198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.818+0000 7f4206864640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4200102340 0x7f4200198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.818+0000 7f4206864640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4200107940 0x7f420019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.818+0000 7f4206864640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4200111ab0 con 0x7f4200107940 2024-06-04T05:00:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.818+0000 7f4206864640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4200111930 con 0x7f4200101980 2024-06-04T05:00:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.818+0000 7f4206864640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4200111c30 con 0x7f4200102340 2024-06-04T05:00:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f4204e01640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4200107940 0x7f420019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f4204e01640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4200107940 0x7f420019c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56764/0 (socket says 172.21.15.70:56764) 2024-06-04T05:00:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f4204e01640 1 -- 172.21.15.70:0/2354291150 learned_addr learned my addr 172.21.15.70:0/2354291150 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f41ff7fe640 1 --2- 172.21.15.70:0/2354291150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4200102340 0x7f4200198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f41fffff640 1 --2- 172.21.15.70:0/2354291150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4200101980 0x7f4200198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f4204e01640 1 -- 172.21.15.70:0/2354291150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4200102340 msgr2=0x7f4200198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f4204e01640 1 --2- 172.21.15.70:0/2354291150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4200102340 0x7f4200198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f4204e01640 1 -- 172.21.15.70:0/2354291150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4200101980 msgr2=0x7f4200198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f4204e01640 1 --2- 172.21.15.70:0/2354291150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4200101980 0x7f4200198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f4204e01640 1 -- 172.21.15.70:0/2354291150 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420019d000 con 0x7f4200107940 2024-06-04T05:00:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f41ff7fe640 1 --2- 172.21.15.70:0/2354291150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4200102340 0x7f4200198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f41fffff640 1 --2- 172.21.15.70:0/2354291150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4200101980 0x7f4200198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f4204e01640 1 --2- 172.21.15.70:0/2354291150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4200107940 0x7f420019c8d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f41f8009fd0 tx=0x7f41f8007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f41fd7fa640 1 -- 172.21.15.70:0/2354291150 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f800de10 con 0x7f4200107940 2024-06-04T05:00:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f4206864640 1 -- 172.21.15.70:0/2354291150 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f420019d2f0 con 0x7f4200107940 2024-06-04T05:00:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f41fd7fa640 1 -- 172.21.15.70:0/2354291150 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f41f8003e00 con 0x7f4200107940 2024-06-04T05:00:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f41fd7fa640 1 -- 172.21.15.70:0/2354291150 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f80047a0 con 0x7f4200107940 2024-06-04T05:00:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.819+0000 7f4206864640 1 -- 172.21.15.70:0/2354291150 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f420006a100 con 0x7f4200107940 2024-06-04T05:00:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.822+0000 7f420957d740 1 Processor -- start 2024-06-04T05:00:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.822+0000 7f41fd7fa640 1 -- 172.21.15.70:0/2354291150 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f41f8003970 con 0x7f4200107940 2024-06-04T05:00:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.822+0000 7f420957d740 1 -- start start 2024-06-04T05:00:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.822+0000 7f420957d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060a820 0x56061060ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.822+0000 7f420957d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060b140 0x5606106154c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.822+0000 7f420957d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 0x560610617df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.822+0000 7f420957d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56061057cc30 con 0x560610615a00 2024-06-04T05:00:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.822+0000 7f420957d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56061056c7d0 con 0x56061060b140 2024-06-04T05:00:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.822+0000 7f420957d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560610570810 con 0x56061060a820 2024-06-04T05:00:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.822+0000 7f41ff7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 0x560610617df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.822+0000 7f41ff7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 0x560610617df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56776/0 (socket says 172.21.15.70:56776) 2024-06-04T05:00:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.822+0000 7f41ff7fe640 1 -- 172.21.15.70:0/1188201210 learned_addr learned my addr 172.21.15.70:0/1188201210 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.822+0000 7f41fffff640 1 --2- 172.21.15.70:0/1188201210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060b140 0x5606106154c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.823+0000 7f41ff7fe640 1 --2- 172.21.15.70:0/1188201210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060a820 0x56061060ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.823+0000 7f41ff7fe640 1 -- 172.21.15.70:0/1188201210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060a820 msgr2=0x56061060ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.823+0000 7f41ff7fe640 1 --2- 172.21.15.70:0/1188201210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060a820 0x56061060ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.823+0000 7f41ff7fe640 1 -- 172.21.15.70:0/1188201210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060b140 msgr2=0x5606106154c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.823+0000 7f41ff7fe640 1 --2- 172.21.15.70:0/1188201210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060b140 0x5606106154c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.823+0000 7f41ff7fe640 1 -- 172.21.15.70:0/1188201210 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560610281210 con 0x560610615a00 2024-06-04T05:00:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.823+0000 7f41fffff640 1 --2- 172.21.15.70:0/1188201210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060b140 0x5606106154c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.823+0000 7f41fd7fa640 1 --2- 172.21.15.70:0/2354291150 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f41e80b2a10 0x7f41e80b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.823+0000 7f41ff7fe640 1 --2- 172.21.15.70:0/1188201210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 0x560610617df0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f41f4005de0 tx=0x7f41f4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2eebec8c4a77a5e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.823+0000 7f41fffff640 1 --2- 172.21.15.70:0/2354291150 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f41e80b2a10 0x7f41e80b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.823+0000 7f41fd7fa640 1 -- 172.21.15.70:0/2354291150 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f41f80e3cf0 con 0x7f4200107940 2024-06-04T05:00:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.823+0000 7f41e67fc640 1 -- 172.21.15.70:0/1188201210 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4035030 con 0x560610615a00 2024-06-04T05:00:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.823+0000 7f41e67fc640 1 -- 172.21.15.70:0/1188201210 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f41f40042f0 con 0x560610615a00 2024-06-04T05:00:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.823+0000 7f41e67fc640 1 -- 172.21.15.70:0/1188201210 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4003ae0 con 0x560610615a00 2024-06-04T05:00:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.824+0000 7f420957d740 1 -- 172.21.15.70:0/1188201210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 msgr2=0x560610617df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.824+0000 7f420957d740 1 --2- 172.21.15.70:0/1188201210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 0x560610617df0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f41f4005de0 tx=0x7f41f4005d50 comp rx=0 tx=0).stop 2024-06-04T05:00:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.824+0000 7f41fffff640 1 --2- 172.21.15.70:0/2354291150 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f41e80b2a10 0x7f41e80b4ed0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f41f0009fd0 tx=0x7f41f0005f90 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.824+0000 7f420957d740 1 -- 172.21.15.70:0/1188201210 shutdown_connections 2024-06-04T05:00:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.824+0000 7f420957d740 1 --2- 172.21.15.70:0/1188201210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060b140 0x5606106154c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.824+0000 7f420957d740 1 --2- 172.21.15.70:0/1188201210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 0x560610617df0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.824+0000 7f420957d740 1 --2- 172.21.15.70:0/1188201210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060a820 0x56061060ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.824+0000 7f420957d740 1 -- 172.21.15.70:0/1188201210 >> 172.21.15.70:0/1188201210 conn(0x560610606d30 msgr2=0x560610607110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.824+0000 7f420957d740 1 -- 172.21.15.70:0/1188201210 shutdown_connections 2024-06-04T05:00:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.824+0000 7f420957d740 1 -- 172.21.15.70:0/1188201210 wait complete. 2024-06-04T05:00:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.824+0000 7f420957d740 1 -- start start 2024-06-04T05:00:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f420957d740 1 Processor -- start 2024-06-04T05:00:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f420957d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060a820 0x560610612ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f420957d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56061060b140 0x560610612ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f420957d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560610615a00 0x560610613530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f420957d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560610252790 con 0x56061060b140 2024-06-04T05:00:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f420957d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560610528890 con 0x560610615a00 2024-06-04T05:00:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f420957d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5606106185d0 con 0x56061060a820 2024-06-04T05:00:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f420957d740 10 client.0 _ll_register_callbacks cb 0x560610280a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f41ff7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56061060b140 0x560610612ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f41fffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060a820 0x560610612ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f41ff7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56061060b140 0x560610612ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56788/0 (socket says 172.21.15.70:56788) 2024-06-04T05:00:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f41ff7fe640 1 -- 172.21.15.70:0/3823591428 learned_addr learned my addr 172.21.15.70:0/3823591428 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f4204e01640 1 --2- 172.21.15.70:0/3823591428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560610615a00 0x560610613530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f41ff7fe640 1 -- 172.21.15.70:0/3823591428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060a820 msgr2=0x560610612ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f41ff7fe640 1 --2- 172.21.15.70:0/3823591428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060a820 0x560610612ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f41ff7fe640 1 -- 172.21.15.70:0/3823591428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560610615a00 msgr2=0x560610613530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f41ff7fe640 1 --2- 172.21.15.70:0/3823591428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560610615a00 0x560610613530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f41ff7fe640 1 -- 172.21.15.70:0/3823591428 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56061034d620 con 0x56061060b140 2024-06-04T05:00:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.825+0000 7f41fffff640 1 --2- 172.21.15.70:0/3823591428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060a820 0x560610612ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.826+0000 7f41ff7fe640 1 --2- 172.21.15.70:0/3823591428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56061060b140 0x560610612ff0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f41f4006010 tx=0x7f41f4005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8edf5adb1810b266 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.826+0000 7f41df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:00:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.826+0000 7f41df7fe640 1 -- 172.21.15.70:0/3823591428 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f404e070 con 0x56061060b140 2024-06-04T05:00:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.826+0000 7f41df7fe640 1 -- 172.21.15.70:0/3823591428 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f41f40035f0 con 0x56061060b140 2024-06-04T05:00:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.826+0000 7f420957d740 1 -- 172.21.15.70:0/3823591428 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41f40042f0 con 0x56061060b140 2024-06-04T05:00:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.826+0000 7f41df7fe640 1 -- 172.21.15.70:0/3823591428 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f41f405a050 con 0x56061060b140 2024-06-04T05:00:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.826+0000 7f420957d740 10 client.16491 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.826+0000 7f420957d740 1 -- 172.21.15.70:0/3823591428 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560610624450 con 0x56061060b140 2024-06-04T05:00:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.826+0000 7f41df7fe640 1 -- 172.21.15.70:0/3823591428 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f41f40042f0 con 0x56061060b140 2024-06-04T05:00:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.826+0000 7f41df7fe640 1 -- 172.21.15.70:0/3823591428 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f41f403db40 con 0x56061060b140 2024-06-04T05:00:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.826+0000 7f420957d740 10 client.16491 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.826+0000 7f420957d740 10 client.16491 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.826+0000 7f420957d740 20 client.16491 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.826+0000 7f420957d740 1 --2- 172.21.15.70:0/3823591428 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5606106257a0 0x560610627b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.826+0000 7f420957d740 4 client.16491 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:00:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.826+0000 7f420957d740 1 -- 172.21.15.70:0/3823591428 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56061057eb80 con 0x5606106257a0 2024-06-04T05:00:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.827+0000 7f41fffff640 1 --2- 172.21.15.70:0/3823591428 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5606106257a0 0x560610627b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.827+0000 7f41fffff640 1 --2- 172.21.15.70:0/3823591428 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5606106257a0 0x560610627b90 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f55a0cc80095d79e server_cookie=bd7e7328fb7e146a in_seq=0 out_seq=0 2024-06-04T05:00:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.827+0000 7f41df7fe640 10 client.16491 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.828+0000 7f41df7fe640 1 -- 172.21.15.70:0/3823591428 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56061057eb80 con 0x5606106257a0 2024-06-04T05:00:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.828+0000 7f41df7fe640 10 client.16491 handle_command_reply: tid=0 2024-06-04T05:00:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.828+0000 7f420957d740 1 client.16491 shutdown 2024-06-04T05:00:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.828+0000 7f420957d740 1 -- 172.21.15.70:0/3823591428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56061060b140 msgr2=0x560610612ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.828+0000 7f420957d740 1 --2- 172.21.15.70:0/3823591428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56061060b140 0x560610612ff0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f41f4006010 tx=0x7f41f4005890 comp rx=0 tx=0).stop 2024-06-04T05:00:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.828+0000 7f420957d740 1 -- 172.21.15.70:0/3823591428 shutdown_connections 2024-06-04T05:00:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.828+0000 7f420957d740 1 --2- 172.21.15.70:0/3823591428 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5606106257a0 0x560610627b90 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.828+0000 7f420957d740 1 --2- 172.21.15.70:0/3823591428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560610615a00 0x560610613530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.828+0000 7f420957d740 1 --2- 172.21.15.70:0/3823591428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56061060b140 0x560610612ff0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.828+0000 7f41df7fe640 0 client.16491 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.828+0000 7f420957d740 1 --2- 172.21.15.70:0/3823591428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060a820 0x560610612ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.828+0000 7f420957d740 1 -- 172.21.15.70:0/3823591428 >> 172.21.15.70:0/3823591428 conn(0x560610606d30 msgr2=0x560610605fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.828+0000 7f420957d740 1 -- 172.21.15.70:0/3823591428 shutdown_connections 2024-06-04T05:00:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.828+0000 7f420957d740 1 -- 172.21.15.70:0/3823591428 wait complete. 2024-06-04T05:00:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.829+0000 7f420957d740 20 client.16491 trim_cache size 0 max 16384 2024-06-04T05:00:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f420957d740 1 Processor -- start 2024-06-04T05:00:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f420957d740 1 -- start start 2024-06-04T05:00:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f420957d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060a820 0x560610606730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f420957d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060b140 0x56061060d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f420957d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 0x56061062abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f420957d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f42001ac380 con 0x560610615a00 2024-06-04T05:00:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f420957d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f42001adf20 con 0x56061060a820 2024-06-04T05:00:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f420957d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f42001aca50 con 0x56061060b140 2024-06-04T05:00:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f4204e01640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 0x56061062abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f41fffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060a820 0x560610606730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f4204e01640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 0x56061062abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56802/0 (socket says 172.21.15.70:56802) 2024-06-04T05:00:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f4204e01640 1 -- 172.21.15.70:0/3975560496 learned_addr learned my addr 172.21.15.70:0/3975560496 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f41ff7fe640 1 --2- 172.21.15.70:0/3975560496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060b140 0x56061060d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f41ff7fe640 1 -- 172.21.15.70:0/3975560496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060a820 msgr2=0x560610606730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f41ff7fe640 1 --2- 172.21.15.70:0/3975560496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060a820 0x560610606730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f41ff7fe640 1 -- 172.21.15.70:0/3975560496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 msgr2=0x56061062abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f41ff7fe640 1 --2- 172.21.15.70:0/3975560496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 0x56061062abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f41ff7fe640 1 -- 172.21.15.70:0/3975560496 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606106024d0 con 0x56061060b140 2024-06-04T05:00:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f41fffff640 1 --2- 172.21.15.70:0/3975560496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060a820 0x560610606730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f4204e01640 1 --2- 172.21.15.70:0/3975560496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 0x56061062abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.847+0000 7f41ff7fe640 1 --2- 172.21.15.70:0/3975560496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060b140 0x56061060d1e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f41f404a4c0 tx=0x7f41f402a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=285ff95fc85aa0d1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.848+0000 7f41e5ffb640 1 -- 172.21.15.70:0/3975560496 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f404e070 con 0x56061060b140 2024-06-04T05:00:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.848+0000 7f41e5ffb640 1 -- 172.21.15.70:0/3975560496 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f41f404ae50 con 0x56061060b140 2024-06-04T05:00:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.848+0000 7f41e5ffb640 1 -- 172.21.15.70:0/3975560496 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4035040 con 0x56061060b140 2024-06-04T05:00:53.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.848+0000 7f420957d740 1 -- 172.21.15.70:0/3975560496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060b140 msgr2=0x56061060d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:53.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.848+0000 7f420957d740 1 --2- 172.21.15.70:0/3975560496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060b140 0x56061060d1e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f41f404a4c0 tx=0x7f41f402a3f0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.848+0000 7f420957d740 1 -- 172.21.15.70:0/3975560496 shutdown_connections 2024-06-04T05:00:53.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.848+0000 7f420957d740 1 --2- 172.21.15.70:0/3975560496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 0x56061062abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.848+0000 7f420957d740 1 --2- 172.21.15.70:0/3975560496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060b140 0x56061060d1e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.848+0000 7f420957d740 1 --2- 172.21.15.70:0/3975560496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060a820 0x560610606730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.848+0000 7f420957d740 1 -- 172.21.15.70:0/3975560496 >> 172.21.15.70:0/3975560496 conn(0x560610606d30 msgr2=0x5606106210b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:53.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.849+0000 7f420957d740 1 -- 172.21.15.70:0/3975560496 shutdown_connections 2024-06-04T05:00:53.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.849+0000 7f420957d740 1 -- 172.21.15.70:0/3975560496 wait complete. 2024-06-04T05:00:53.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.849+0000 7f420957d740 1 -- start start 2024-06-04T05:00:53.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.849+0000 7f420957d740 1 Processor -- start 2024-06-04T05:00:53.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.849+0000 7f420957d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060a820 0x56061062b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.849+0000 7f420957d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060b140 0x560610628480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.849+0000 7f420957d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 0x5606106289c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.849+0000 7f420957d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f42001ad7f0 con 0x560610615a00 2024-06-04T05:00:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.849+0000 7f420957d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f42001ad120 con 0x56061060a820 2024-06-04T05:00:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.849+0000 7f420957d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f42001aa360 con 0x56061060b140 2024-06-04T05:00:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.849+0000 7f420957d740 10 client.0 _ll_register_callbacks cb 0x7f41f0009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f41ff7fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060b140 0x560610628480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f4204e01640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 0x5606106289c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f41ff7fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060b140 0x560610628480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41550/0 (socket says 172.21.15.70:41550) 2024-06-04T05:00:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f41ff7fe640 1 -- 172.21.15.70:0/374236751 learned_addr learned my addr 172.21.15.70:0/374236751 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f41fffff640 1 --2- 172.21.15.70:0/374236751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060a820 0x56061062b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f41ff7fe640 1 -- 172.21.15.70:0/374236751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060a820 msgr2=0x56061062b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f41ff7fe640 1 --2- 172.21.15.70:0/374236751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060a820 0x56061062b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f41ff7fe640 1 -- 172.21.15.70:0/374236751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 msgr2=0x5606106289c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f41ff7fe640 1 --2- 172.21.15.70:0/374236751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 0x5606106289c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f41ff7fe640 1 -- 172.21.15.70:0/374236751 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560610629000 con 0x56061060b140 2024-06-04T05:00:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f4204e01640 1 --2- 172.21.15.70:0/374236751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 0x5606106289c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f41fffff640 1 --2- 172.21.15.70:0/374236751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060a820 0x56061062b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f41ff7fe640 1 --2- 172.21.15.70:0/374236751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060b140 0x560610628480 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f41f404a490 tx=0x7f41f4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=747128701068c00f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f41df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:00:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f41df7fe640 1 -- 172.21.15.70:0/374236751 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4063070 con 0x56061060b140 2024-06-04T05:00:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f41df7fe640 1 -- 172.21.15.70:0/374236751 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f41f4031990 con 0x56061060b140 2024-06-04T05:00:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f420957d740 1 -- 172.21.15.70:0/374236751 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41f404ae50 con 0x56061060b140 2024-06-04T05:00:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f41df7fe640 1 -- 172.21.15.70:0/374236751 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f41f406c050 con 0x56061060b140 2024-06-04T05:00:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f420957d740 10 client.25909 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.850+0000 7f420957d740 1 -- 172.21.15.70:0/374236751 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56061068fe40 con 0x56061060b140 2024-06-04T05:00:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.851+0000 7f41df7fe640 1 -- 172.21.15.70:0/374236751 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f41f404ae50 con 0x56061060b140 2024-06-04T05:00:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.851+0000 7f41df7fe640 1 -- 172.21.15.70:0/374236751 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f41f4066020 con 0x56061060b140 2024-06-04T05:00:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.851+0000 7f420957d740 10 client.25909 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.851+0000 7f420957d740 10 client.25909 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:00:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.851+0000 7f420957d740 20 client.25909 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.851+0000 7f420957d740 1 --2- 172.21.15.70:0/374236751 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560610690670 0x560610692a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.851+0000 7f420957d740 4 client.25909 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:00:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.851+0000 7f420957d740 1 -- 172.21.15.70:0/374236751 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560610578360 con 0x560610690670 2024-06-04T05:00:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.851+0000 7f41fffff640 1 --2- 172.21.15.70:0/374236751 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560610690670 0x560610692a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.851+0000 7f41fffff640 1 --2- 172.21.15.70:0/374236751 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560610690670 0x560610692a90 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19cad5edc01afce8 server_cookie=1d5f4bc5c1529904 in_seq=0 out_seq=0 2024-06-04T05:00:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.851+0000 7f41df7fe640 10 client.25909 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:53.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.866+0000 7f41df7fe640 1 -- 172.21.15.70:0/374236751 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x560610578360 con 0x560610690670 2024-06-04T05:00:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.866+0000 7f41df7fe640 10 client.25909 handle_command_reply: tid=0 2024-06-04T05:00:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.866+0000 7f420957d740 1 client.25909 shutdown 2024-06-04T05:00:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.866+0000 7f420957d740 1 -- 172.21.15.70:0/374236751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060b140 msgr2=0x560610628480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.866+0000 7f420957d740 1 --2- 172.21.15.70:0/374236751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060b140 0x560610628480 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f41f404a490 tx=0x7f41f4009780 comp rx=0 tx=0).stop 2024-06-04T05:00:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.866+0000 7f420957d740 1 -- 172.21.15.70:0/374236751 shutdown_connections 2024-06-04T05:00:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.866+0000 7f420957d740 1 --2- 172.21.15.70:0/374236751 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560610690670 0x560610692a90 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.866+0000 7f420957d740 1 --2- 172.21.15.70:0/374236751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560610615a00 0x5606106289c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.866+0000 7f41df7fe640 0 client.25909 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.866+0000 7f420957d740 1 --2- 172.21.15.70:0/374236751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56061060b140 0x560610628480 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.866+0000 7f420957d740 1 --2- 172.21.15.70:0/374236751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56061060a820 0x56061062b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.866+0000 7f420957d740 1 -- 172.21.15.70:0/374236751 >> 172.21.15.70:0/374236751 conn(0x560610606d30 msgr2=0x560610612a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.866+0000 7f420957d740 1 -- 172.21.15.70:0/374236751 shutdown_connections 2024-06-04T05:00:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.866+0000 7f420957d740 1 -- 172.21.15.70:0/374236751 wait complete. 2024-06-04T05:00:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.866+0000 7f420957d740 20 client.25909 trim_cache size 0 max 16384 2024-06-04T05:00:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:00:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:00:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:00:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.867+0000 7f4206864640 1 -- 172.21.15.70:0/2354291150 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f41e80b2a10 msgr2=0x7f41e80b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.867+0000 7f4206864640 1 --2- 172.21.15.70:0/2354291150 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f41e80b2a10 0x7f41e80b4ed0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f41f0009fd0 tx=0x7f41f0005f90 comp rx=0 tx=0).stop 2024-06-04T05:00:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.867+0000 7f4206864640 1 -- 172.21.15.70:0/2354291150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4200107940 msgr2=0x7f420019c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.867+0000 7f4206864640 1 --2- 172.21.15.70:0/2354291150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4200107940 0x7f420019c8d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f41f8009fd0 tx=0x7f41f8007590 comp rx=0 tx=0).stop 2024-06-04T05:00:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.867+0000 7f4206864640 1 -- 172.21.15.70:0/2354291150 shutdown_connections 2024-06-04T05:00:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.867+0000 7f4206864640 1 --2- 172.21.15.70:0/2354291150 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f41e80b2a10 0x7f41e80b4ed0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.867+0000 7f4206864640 1 --2- 172.21.15.70:0/2354291150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4200107940 0x7f420019c8d0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.867+0000 7f4206864640 1 --2- 172.21.15.70:0/2354291150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4200102340 0x7f4200198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.867+0000 7f4206864640 1 --2- 172.21.15.70:0/2354291150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4200101980 0x7f4200198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:53.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.867+0000 7f4206864640 1 -- 172.21.15.70:0/2354291150 >> 172.21.15.70:0/2354291150 conn(0x7f4200078ad0 msgr2=0x7f4200104540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:53.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.867+0000 7f4206864640 1 -- 172.21.15.70:0/2354291150 shutdown_connections 2024-06-04T05:00:53.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:00:53.867+0000 7f4206864640 1 -- 172.21.15.70:0/2354291150 wait complete. 2024-06-04T05:00:54.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.426+0000 7fd60b350700 1 -- 172.21.15.70:0/2492516074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd604102f70 msgr2=0x7fd604103350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:54.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.426+0000 7fd60b350700 1 --2- 172.21.15.70:0/2492516074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd604102f70 0x7fd604103350 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd600009990 tx=0x7fd600009ca0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.426+0000 7fd60b350700 1 -- 172.21.15.70:0/2492516074 shutdown_connections 2024-06-04T05:00:54.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.426+0000 7fd60b350700 1 --2- 172.21.15.70:0/2492516074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd60410fff0 0x7fd6041123e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.426+0000 7fd60b350700 1 --2- 172.21.15.70:0/2492516074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd604102f70 0x7fd604103350 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.426+0000 7fd60b350700 1 --2- 172.21.15.70:0/2492516074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd604103890 0x7fd60410fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.426+0000 7fd60b350700 1 -- 172.21.15.70:0/2492516074 >> 172.21.15.70:0/2492516074 conn(0x7fd6040793e0 msgr2=0x7fd6040797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:54.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.426+0000 7fd60b350700 1 -- 172.21.15.70:0/2492516074 shutdown_connections 2024-06-04T05:00:54.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.426+0000 7fd60b350700 1 -- 172.21.15.70:0/2492516074 wait complete. 2024-06-04T05:00:54.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.427+0000 7fd60b350700 1 Processor -- start 2024-06-04T05:00:54.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.427+0000 7fd60b350700 1 -- start start 2024-06-04T05:00:54.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.427+0000 7fd60b350700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd604102f70 0x7fd6041a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.427+0000 7fd60b350700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd604103890 0x7fd6041a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.427+0000 7fd60b350700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd60410fff0 0x7fd60419bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.427+0000 7fd60b350700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6041157b0 con 0x7fd604103890 2024-06-04T05:00:54.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.427+0000 7fd60b350700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd604115630 con 0x7fd604102f70 2024-06-04T05:00:54.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.427+0000 7fd60b350700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd604115930 con 0x7fd60410fff0 2024-06-04T05:00:54.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.427+0000 7fd6088eb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd604103890 0x7fd6041a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.428+0000 7fd6088eb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd604103890 0x7fd6041a1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56832/0 (socket says 172.21.15.70:56832) 2024-06-04T05:00:54.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.428+0000 7fd6088eb700 1 -- 172.21.15.70:0/1272587583 learned_addr learned my addr 172.21.15.70:0/1272587583 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:54.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.428+0000 7fd6098ed700 1 --2- 172.21.15.70:0/1272587583 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd60410fff0 0x7fd60419bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.428+0000 7fd6090ec700 1 --2- 172.21.15.70:0/1272587583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd604102f70 0x7fd6041a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.428+0000 7fd6088eb700 1 -- 172.21.15.70:0/1272587583 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd60410fff0 msgr2=0x7fd60419bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:54.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.428+0000 7fd6088eb700 1 --2- 172.21.15.70:0/1272587583 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd60410fff0 0x7fd60419bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.428+0000 7fd6088eb700 1 -- 172.21.15.70:0/1272587583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd604102f70 msgr2=0x7fd6041a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:54.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.428+0000 7fd6088eb700 1 --2- 172.21.15.70:0/1272587583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd604102f70 0x7fd6041a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.428+0000 7fd6088eb700 1 -- 172.21.15.70:0/1272587583 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd60419c290 con 0x7fd604103890 2024-06-04T05:00:54.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.428+0000 7fd6090ec700 1 --2- 172.21.15.70:0/1272587583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd604102f70 0x7fd6041a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:54.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.428+0000 7fd6088eb700 1 --2- 172.21.15.70:0/1272587583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd604103890 0x7fd6041a1e60 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400e970 tx=0x7fd5f400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:54.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.428+0000 7fd5fa7fc700 1 -- 172.21.15.70:0/1272587583 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f400cbd0 con 0x7fd604103890 2024-06-04T05:00:54.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.428+0000 7fd60b350700 1 -- 172.21.15.70:0/1272587583 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd60419c580 con 0x7fd604103890 2024-06-04T05:00:54.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.428+0000 7fd5fa7fc700 1 -- 172.21.15.70:0/1272587583 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd5f40044b0 con 0x7fd604103890 2024-06-04T05:00:54.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.428+0000 7fd5fa7fc700 1 -- 172.21.15.70:0/1272587583 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f4003c90 con 0x7fd604103890 2024-06-04T05:00:54.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.428+0000 7fd60b350700 1 -- 172.21.15.70:0/1272587583 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6041a87f0 con 0x7fd604103890 2024-06-04T05:00:54.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.431+0000 7fd5fa7fc700 1 -- 172.21.15.70:0/1272587583 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fd5f4003f10 con 0x7fd604103890 2024-06-04T05:00:54.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.432+0000 7fd5fa7fc700 1 --2- 172.21.15.70:0/1272587583 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd5f00b2b20 0x7fd5f00b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.432+0000 7fd6090ec700 1 --2- 172.21.15.70:0/1272587583 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd5f00b2b20 0x7fd5f00b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.432+0000 7fd5fa7fc700 1 -- 172.21.15.70:0/1272587583 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fd5f40e2940 con 0x7fd604103890 2024-06-04T05:00:54.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.432+0000 7fd6090ec700 1 --2- 172.21.15.70:0/1272587583 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd5f00b2b20 0x7fd5f00b4fe0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd60000b2e0 tx=0x7fd60001a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:54.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.433+0000 7fd615be5b80 1 Processor -- start 2024-06-04T05:00:54.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.433+0000 7fd615be5b80 1 -- start start 2024-06-04T05:00:54.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.433+0000 7fd615be5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4e9cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.433+0000 7fd615be5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e97c90 0x55c0a4e98110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.433+0000 7fd615be5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e98650 0x55c0a4ea46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.433+0000 7fd615be5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c0a4df5f10 con 0x55c0a4e98650 2024-06-04T05:00:54.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.433+0000 7fd615be5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c0a4de5ab0 con 0x55c0a4e97c90 2024-06-04T05:00:54.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.433+0000 7fd615be5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c0a4de9af0 con 0x55c0a4e9c7c0 2024-06-04T05:00:54.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.433+0000 7fd6090ec700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4e9cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.433+0000 7fd6088eb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e97c90 0x55c0a4e98110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.433+0000 7fd6098ed700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e98650 0x55c0a4ea46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.433+0000 7fd6088eb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e97c90 0x55c0a4e98110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:44772/0 (socket says 172.21.15.70:44772) 2024-06-04T05:00:54.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.433+0000 7fd6088eb700 1 -- 172.21.15.70:0/1721871813 learned_addr learned my addr 172.21.15.70:0/1721871813 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:54.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.433+0000 7fd6090ec700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4e9cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41576/0 (socket says 172.21.15.70:41576) 2024-06-04T05:00:54.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.434+0000 7fd6098ed700 1 -- 172.21.15.70:0/1721871813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e9c7c0 msgr2=0x55c0a4e9cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:54.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.434+0000 7fd6098ed700 1 --2- 172.21.15.70:0/1721871813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4e9cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.434+0000 7fd6098ed700 1 -- 172.21.15.70:0/1721871813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e97c90 msgr2=0x55c0a4e98110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:54.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.434+0000 7fd6098ed700 1 --2- 172.21.15.70:0/1721871813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e97c90 0x55c0a4e98110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.434+0000 7fd6098ed700 1 -- 172.21.15.70:0/1721871813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0a4d8f150 con 0x55c0a4e98650 2024-06-04T05:00:54.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.434+0000 7fd6088eb700 1 --2- 172.21.15.70:0/1721871813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e97c90 0x55c0a4e98110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:54.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.434+0000 7fd6098ed700 1 --2- 172.21.15.70:0/1721871813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e98650 0x55c0a4ea46c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00a510 tx=0x7fd5fc0078d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6072f616352a0f89 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:54.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.434+0000 7fd5ef7fe700 1 -- 172.21.15.70:0/1721871813 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc00a0d0 con 0x55c0a4e98650 2024-06-04T05:00:54.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.434+0000 7fd5ef7fe700 1 -- 172.21.15.70:0/1721871813 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd5fc004020 con 0x55c0a4e98650 2024-06-04T05:00:54.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.434+0000 7fd5ef7fe700 1 -- 172.21.15.70:0/1721871813 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc00a270 con 0x55c0a4e98650 2024-06-04T05:00:54.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.435+0000 7fd615be5b80 1 -- 172.21.15.70:0/1721871813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e98650 msgr2=0x55c0a4ea46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:54.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.435+0000 7fd615be5b80 1 --2- 172.21.15.70:0/1721871813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e98650 0x55c0a4ea46c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00a510 tx=0x7fd5fc0078d0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.435+0000 7fd615be5b80 1 -- 172.21.15.70:0/1721871813 shutdown_connections 2024-06-04T05:00:54.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.435+0000 7fd615be5b80 1 --2- 172.21.15.70:0/1721871813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e98650 0x55c0a4ea46c0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.435+0000 7fd615be5b80 1 --2- 172.21.15.70:0/1721871813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4e9cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.435+0000 7fd615be5b80 1 --2- 172.21.15.70:0/1721871813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e97c90 0x55c0a4e98110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.435+0000 7fd615be5b80 1 -- 172.21.15.70:0/1721871813 >> 172.21.15.70:0/1721871813 conn(0x55c0a4dc33e0 msgr2=0x55c0a4e92980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:54.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.435+0000 7fd615be5b80 1 -- 172.21.15.70:0/1721871813 shutdown_connections 2024-06-04T05:00:54.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.435+0000 7fd615be5b80 1 -- 172.21.15.70:0/1721871813 wait complete. 2024-06-04T05:00:54.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.435+0000 7fd615be5b80 1 -- start start 2024-06-04T05:00:54.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.435+0000 7fd615be5b80 1 Processor -- start 2024-06-04T05:00:54.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd615be5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e97c90 0x55c0a4ea4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd615be5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e98650 0x55c0a4ea1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd615be5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4ea1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd615be5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c0a4cec140 con 0x55c0a4e9c7c0 2024-06-04T05:00:54.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd615be5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c0a4d4df40 con 0x55c0a4e98650 2024-06-04T05:00:54.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd615be5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c0a4ea4cc0 con 0x55c0a4e97c90 2024-06-04T05:00:54.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd615be5b80 10 client.0 _ll_register_callbacks cb 0x55c0a4d80af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:54.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd6090ec700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e97c90 0x55c0a4ea4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd6098ed700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4ea1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd6090ec700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e97c90 0x55c0a4ea4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41592/0 (socket says 172.21.15.70:41592) 2024-06-04T05:00:54.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd6090ec700 1 -- 172.21.15.70:0/981670634 learned_addr learned my addr 172.21.15.70:0/981670634 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:54.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd6088eb700 1 --2- 172.21.15.70:0/981670634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e98650 0x55c0a4ea1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd6090ec700 1 -- 172.21.15.70:0/981670634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e98650 msgr2=0x55c0a4ea1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:54.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd6090ec700 1 --2- 172.21.15.70:0/981670634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e98650 0x55c0a4ea1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd6090ec700 1 -- 172.21.15.70:0/981670634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e9c7c0 msgr2=0x55c0a4ea1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:54.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd6090ec700 1 --2- 172.21.15.70:0/981670634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4ea1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd6090ec700 1 -- 172.21.15.70:0/981670634 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0a4d35a70 con 0x55c0a4e97c90 2024-06-04T05:00:54.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.436+0000 7fd6088eb700 1 --2- 172.21.15.70:0/981670634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e98650 0x55c0a4ea1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:00:54.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.437+0000 7fd6090ec700 1 --2- 172.21.15.70:0/981670634 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e97c90 0x55c0a4ea4340 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd600015000 tx=0x7fd600009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3fb53a10275cee server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:54.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.437+0000 7fd5ecff9700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:00:54.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.437+0000 7fd5ecff9700 1 -- 172.21.15.70:0/981670634 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd600005440 con 0x55c0a4e97c90 2024-06-04T05:00:54.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.437+0000 7fd615be5b80 1 -- 172.21.15.70:0/981670634 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5fc004020 con 0x55c0a4e97c90 2024-06-04T05:00:54.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.437+0000 7fd5ecff9700 1 -- 172.21.15.70:0/981670634 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fd6000055e0 con 0x55c0a4e97c90 2024-06-04T05:00:54.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.437+0000 7fd5ecff9700 1 -- 172.21.15.70:0/981670634 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd6000047b0 con 0x55c0a4e97c90 2024-06-04T05:00:54.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.437+0000 7fd615be5b80 10 client.25921 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:54.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.437+0000 7fd615be5b80 1 -- 172.21.15.70:0/981670634 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c0a4d9f900 con 0x55c0a4e97c90 2024-06-04T05:00:54.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.437+0000 7fd5ecff9700 1 -- 172.21.15.70:0/981670634 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd5fc004020 con 0x55c0a4e97c90 2024-06-04T05:00:54.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.437+0000 7fd5ecff9700 1 -- 172.21.15.70:0/981670634 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd60002e070 con 0x55c0a4e97c90 2024-06-04T05:00:54.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.437+0000 7fd615be5b80 10 client.25921 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:54.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.437+0000 7fd615be5b80 10 client.25921 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:00:54.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.438+0000 7fd615be5b80 20 client.25921 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:54.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.438+0000 7fd615be5b80 1 --2- 172.21.15.70:0/981670634 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c0a4eb1860 0x55c0a4eb3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.438+0000 7fd615be5b80 4 client.25921 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:00:54.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.438+0000 7fd615be5b80 1 -- 172.21.15.70:0/981670634 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0a4df7e60 con 0x55c0a4eb1860 2024-06-04T05:00:54.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.438+0000 7fd6088eb700 1 --2- 172.21.15.70:0/981670634 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c0a4eb1860 0x55c0a4eb3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.438+0000 7fd5ecff9700 10 client.25921 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:54.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.438+0000 7fd6088eb700 1 --2- 172.21.15.70:0/981670634 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c0a4eb1860 0x55c0a4eb3c90 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d0c64bc7d235c80 server_cookie=1dc7b7fecee30ce9 in_seq=0 out_seq=0 2024-06-04T05:00:54.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.439+0000 7fd5ecff9700 1 -- 172.21.15.70:0/981670634 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c0a4df7e60 con 0x55c0a4eb1860 2024-06-04T05:00:54.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.439+0000 7fd5ecff9700 10 client.25921 handle_command_reply: tid=0 2024-06-04T05:00:54.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.439+0000 7fd615be5b80 1 client.25921 shutdown 2024-06-04T05:00:54.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.439+0000 7fd615be5b80 1 -- 172.21.15.70:0/981670634 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e97c90 msgr2=0x55c0a4ea4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:54.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.439+0000 7fd615be5b80 1 --2- 172.21.15.70:0/981670634 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e97c90 0x55c0a4ea4340 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd600015000 tx=0x7fd600009310 comp rx=0 tx=0).stop 2024-06-04T05:00:54.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.440+0000 7fd615be5b80 1 -- 172.21.15.70:0/981670634 shutdown_connections 2024-06-04T05:00:54.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.440+0000 7fd615be5b80 1 --2- 172.21.15.70:0/981670634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4ea1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.440+0000 7fd615be5b80 1 --2- 172.21.15.70:0/981670634 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e97c90 0x55c0a4ea4340 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.440+0000 7fd615be5b80 1 --2- 172.21.15.70:0/981670634 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c0a4eb1860 0x55c0a4eb3c90 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.440+0000 7fd615be5b80 1 --2- 172.21.15.70:0/981670634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e98650 0x55c0a4ea1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.440+0000 7fd615be5b80 1 -- 172.21.15.70:0/981670634 >> 172.21.15.70:0/981670634 conn(0x55c0a4dc33e0 msgr2=0x55c0a4e96c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:54.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.440+0000 7fd5ecff9700 0 client.25921 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:54.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.440+0000 7fd615be5b80 1 -- 172.21.15.70:0/981670634 shutdown_connections 2024-06-04T05:00:54.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.440+0000 7fd615be5b80 1 -- 172.21.15.70:0/981670634 wait complete. 2024-06-04T05:00:54.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.440+0000 7fd615be5b80 20 client.25921 trim_cache size 0 max 16384 2024-06-04T05:00:54.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.460+0000 7fd615be5b80 1 Processor -- start 2024-06-04T05:00:54.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd615be5b80 1 -- start start 2024-06-04T05:00:54.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd615be5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e97c90 0x55c0a4ea1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd615be5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e98650 0x55c0a4ea3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd615be5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4ea3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd615be5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6041aff30 con 0x55c0a4e9c7c0 2024-06-04T05:00:54.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd615be5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd6041b1ad0 con 0x55c0a4e97c90 2024-06-04T05:00:54.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd615be5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd6041b0600 con 0x55c0a4e98650 2024-06-04T05:00:54.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd6098ed700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4ea3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd6088eb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e98650 0x55c0a4ea3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd6090ec700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e97c90 0x55c0a4ea1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd6088eb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e98650 0x55c0a4ea3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41596/0 (socket says 172.21.15.70:41596) 2024-06-04T05:00:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd6090ec700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e97c90 0x55c0a4ea1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:44800/0 (socket says 172.21.15.70:44800) 2024-06-04T05:00:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd6088eb700 1 -- 172.21.15.70:0/852895272 learned_addr learned my addr 172.21.15.70:0/852895272 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd6088eb700 1 -- 172.21.15.70:0/852895272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e97c90 msgr2=0x55c0a4ea1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd6088eb700 1 --2- 172.21.15.70:0/852895272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e97c90 0x55c0a4ea1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd6088eb700 1 -- 172.21.15.70:0/852895272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e9c7c0 msgr2=0x55c0a4ea3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd6088eb700 1 --2- 172.21.15.70:0/852895272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4ea3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd6088eb700 1 -- 172.21.15.70:0/852895272 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0a4e9fbb0 con 0x55c0a4e98650 2024-06-04T05:00:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd6098ed700 1 --2- 172.21.15.70:0/852895272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4ea3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.461+0000 7fd6088eb700 1 --2- 172.21.15.70:0/852895272 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e98650 0x55c0a4ea3580 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd5f40e0690 tx=0x7fd5f4005910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f95c21d43a1a444 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.462+0000 7fd5eeffd700 1 -- 172.21.15.70:0/852895272 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f40c2070 con 0x55c0a4e98650 2024-06-04T05:00:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.462+0000 7fd5eeffd700 1 -- 172.21.15.70:0/852895272 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd5f40a9730 con 0x55c0a4e98650 2024-06-04T05:00:54.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.462+0000 7fd5eeffd700 1 -- 172.21.15.70:0/852895272 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f40b2a80 con 0x55c0a4e98650 2024-06-04T05:00:54.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.462+0000 7fd615be5b80 1 -- 172.21.15.70:0/852895272 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e98650 msgr2=0x55c0a4ea3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:54.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.462+0000 7fd615be5b80 1 --2- 172.21.15.70:0/852895272 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e98650 0x55c0a4ea3580 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd5f40e0690 tx=0x7fd5f4005910 comp rx=0 tx=0).stop 2024-06-04T05:00:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.462+0000 7fd615be5b80 1 -- 172.21.15.70:0/852895272 shutdown_connections 2024-06-04T05:00:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.462+0000 7fd615be5b80 1 --2- 172.21.15.70:0/852895272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4ea3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.462+0000 7fd615be5b80 1 --2- 172.21.15.70:0/852895272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e97c90 0x55c0a4ea1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.462+0000 7fd615be5b80 1 --2- 172.21.15.70:0/852895272 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e98650 0x55c0a4ea3580 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.462+0000 7fd615be5b80 1 -- 172.21.15.70:0/852895272 >> 172.21.15.70:0/852895272 conn(0x55c0a4dc33e0 msgr2=0x55c0a4ea2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.463+0000 7fd615be5b80 1 -- 172.21.15.70:0/852895272 shutdown_connections 2024-06-04T05:00:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.463+0000 7fd615be5b80 1 -- 172.21.15.70:0/852895272 wait complete. 2024-06-04T05:00:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.463+0000 7fd615be5b80 1 -- start start 2024-06-04T05:00:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.463+0000 7fd615be5b80 1 Processor -- start 2024-06-04T05:00:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.463+0000 7fd615be5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e97c90 0x55c0a4ea8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.463+0000 7fd615be5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e98650 0x55c0a4ea8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.463+0000 7fd615be5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4f15450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.463+0000 7fd615be5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6041b13a0 con 0x55c0a4e98650 2024-06-04T05:00:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.463+0000 7fd615be5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd6041b0cd0 con 0x55c0a4e9c7c0 2024-06-04T05:00:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.463+0000 7fd615be5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd6041aded0 con 0x55c0a4e97c90 2024-06-04T05:00:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.463+0000 7fd615be5b80 10 client.0 _ll_register_callbacks cb 0x55c0a4d80af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:00:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd6088eb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e98650 0x55c0a4ea8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd6090ec700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e97c90 0x55c0a4ea8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd6090ec700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e97c90 0x55c0a4ea8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41612/0 (socket says 172.21.15.70:41612) 2024-06-04T05:00:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd6090ec700 1 -- 172.21.15.70:0/3020649041 learned_addr learned my addr 172.21.15.70:0/3020649041 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:00:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd6088eb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e98650 0x55c0a4ea8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56866/0 (socket says 172.21.15.70:56866) 2024-06-04T05:00:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd6098ed700 1 --2- 172.21.15.70:0/3020649041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4f15450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd6090ec700 1 -- 172.21.15.70:0/3020649041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e9c7c0 msgr2=0x55c0a4f15450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd6090ec700 1 --2- 172.21.15.70:0/3020649041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4f15450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd6090ec700 1 -- 172.21.15.70:0/3020649041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e98650 msgr2=0x55c0a4ea8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:00:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd6090ec700 1 --2- 172.21.15.70:0/3020649041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e98650 0x55c0a4ea8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd6090ec700 1 -- 172.21.15.70:0/3020649041 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0a4f15b30 con 0x55c0a4e97c90 2024-06-04T05:00:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd6088eb700 1 --2- 172.21.15.70:0/3020649041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e98650 0x55c0a4ea8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:00:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd6090ec700 1 --2- 172.21.15.70:0/3020649041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e97c90 0x55c0a4ea8660 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd600006010 tx=0x7fd600009900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2e38940250537c7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:00:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd5ecff9700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:00:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd5ecff9700 1 -- 172.21.15.70:0/3020649041 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd600045060 con 0x55c0a4e97c90 2024-06-04T05:00:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd5ecff9700 1 -- 172.21.15.70:0/3020649041 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fd600030640 con 0x55c0a4e97c90 2024-06-04T05:00:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd615be5b80 1 -- 172.21.15.70:0/3020649041 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5f40a9730 con 0x55c0a4e97c90 2024-06-04T05:00:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.464+0000 7fd5ecff9700 1 -- 172.21.15.70:0/3020649041 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd60003c040 con 0x55c0a4e97c90 2024-06-04T05:00:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.465+0000 7fd615be5b80 10 client.25927 fetch_fsmap learned FSMap version 31 2024-06-04T05:00:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.465+0000 7fd615be5b80 1 -- 172.21.15.70:0/3020649041 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c0a4f1cec0 con 0x55c0a4e97c90 2024-06-04T05:00:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.465+0000 7fd5ecff9700 1 -- 172.21.15.70:0/3020649041 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd5f40a9730 con 0x55c0a4e97c90 2024-06-04T05:00:54.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.465+0000 7fd5ecff9700 1 -- 172.21.15.70:0/3020649041 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd60002e040 con 0x55c0a4e97c90 2024-06-04T05:00:54.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.465+0000 7fd615be5b80 10 client.25927 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:00:54.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.465+0000 7fd615be5b80 10 client.25927 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:00:54.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.465+0000 7fd615be5b80 20 client.25927 populate_metadata read hostname 'smithi070' 2024-06-04T05:00:54.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.465+0000 7fd615be5b80 1 --2- 172.21.15.70:0/3020649041 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c0a4f1d580 0x55c0a4f1f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:00:54.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.465+0000 7fd615be5b80 4 client.25927 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.05, "await_for": 45.0}] 2024-06-04T05:00:54.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.465+0000 7fd615be5b80 1 -- 172.21.15.70:0/3020649041 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.05, "await_for": 45.0}) -- 0x55c0a4df1640 con 0x55c0a4f1d580 2024-06-04T05:00:54.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.465+0000 7fd6088eb700 1 --2- 172.21.15.70:0/3020649041 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c0a4f1d580 0x55c0a4f1f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:00:54.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.465+0000 7fd5ecff9700 10 client.25927 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:54.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.465+0000 7fd6088eb700 1 --2- 172.21.15.70:0/3020649041 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c0a4f1d580 0x55c0a4f1f9a0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40071206a045d1a7 server_cookie=5c0fa7501f25f3cd in_seq=0 out_seq=0 2024-06-04T05:00:54.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.558+0000 7fd5ecff9700 1 -- 172.21.15.70:0/3020649041 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55c0a4df1640 con 0x55c0a4f1d580 2024-06-04T05:00:54.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.558+0000 7fd5ecff9700 10 client.25927 handle_command_reply: tid=0 2024-06-04T05:00:54.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.558+0000 7fd615be5b80 1 client.25927 shutdown 2024-06-04T05:00:54.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.558+0000 7fd615be5b80 1 -- 172.21.15.70:0/3020649041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e97c90 msgr2=0x55c0a4ea8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:54.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.558+0000 7fd615be5b80 1 --2- 172.21.15.70:0/3020649041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e97c90 0x55c0a4ea8660 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd600006010 tx=0x7fd600009900 comp rx=0 tx=0).stop 2024-06-04T05:00:54.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.558+0000 7fd615be5b80 1 -- 172.21.15.70:0/3020649041 shutdown_connections 2024-06-04T05:00:54.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.558+0000 7fd615be5b80 1 --2- 172.21.15.70:0/3020649041 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c0a4f1d580 0x55c0a4f1f9a0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.558+0000 7fd615be5b80 1 --2- 172.21.15.70:0/3020649041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0a4e9c7c0 0x55c0a4f15450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.558+0000 7fd615be5b80 1 --2- 172.21.15.70:0/3020649041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0a4e97c90 0x55c0a4ea8660 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.558+0000 7fd615be5b80 1 --2- 172.21.15.70:0/3020649041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0a4e98650 0x55c0a4ea8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.558+0000 7fd5ecff9700 0 client.25927 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:00:54.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.558+0000 7fd615be5b80 1 -- 172.21.15.70:0/3020649041 >> 172.21.15.70:0/3020649041 conn(0x55c0a4dc33e0 msgr2=0x55c0a4ea3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:54.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.558+0000 7fd615be5b80 1 -- 172.21.15.70:0/3020649041 shutdown_connections 2024-06-04T05:00:54.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.558+0000 7fd615be5b80 1 -- 172.21.15.70:0/3020649041 wait complete. 2024-06-04T05:00:54.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.558+0000 7fd615be5b80 20 client.25927 trim_cache size 0 max 16384 2024-06-04T05:00:54.563 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:00:54.563 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 31, 2024-06-04T05:00:54.563 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:00:54.563 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 10, 2024-06-04T05:00:54.563 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:00:54.563 INFO:teuthology.orchestra.run.smithi070.stdout: "3ea23d63": { 2024-06-04T05:00:54.563 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 10, 2024-06-04T05:00:54.563 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:00:54.563 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:00:54.563 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:00:54.563 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:00:54.564 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:00:54.564 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:00:54.564 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 149.1, 2024-06-04T05:00:54.564 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:00:54.564 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:00:54.564 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:00:54.564 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:00:54.564 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:00:54.564 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:00:54.564 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:00:54.565 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:00:54.565 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:00:54.565 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:00:54.565 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:00:54.565 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:00:54.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.559+0000 7fd60b350700 1 -- 172.21.15.70:0/1272587583 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd5f00b2b20 msgr2=0x7fd5f00b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:54.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.559+0000 7fd60b350700 1 --2- 172.21.15.70:0/1272587583 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd5f00b2b20 0x7fd5f00b4fe0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd60000b2e0 tx=0x7fd60001a040 comp rx=0 tx=0).stop 2024-06-04T05:00:54.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.559+0000 7fd60b350700 1 -- 172.21.15.70:0/1272587583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd604103890 msgr2=0x7fd6041a1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:00:54.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.559+0000 7fd60b350700 1 --2- 172.21.15.70:0/1272587583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd604103890 0x7fd6041a1e60 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400e970 tx=0x7fd5f400ec80 comp rx=0 tx=0).stop 2024-06-04T05:00:54.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.559+0000 7fd60b350700 1 -- 172.21.15.70:0/1272587583 shutdown_connections 2024-06-04T05:00:54.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.559+0000 7fd60b350700 1 --2- 172.21.15.70:0/1272587583 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd5f00b2b20 0x7fd5f00b4fe0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.559+0000 7fd60b350700 1 --2- 172.21.15.70:0/1272587583 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd60410fff0 0x7fd60419bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.559+0000 7fd60b350700 1 --2- 172.21.15.70:0/1272587583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd604102f70 0x7fd6041a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.559+0000 7fd60b350700 1 --2- 172.21.15.70:0/1272587583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd604103890 0x7fd6041a1e60 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:00:54.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.559+0000 7fd60b350700 1 -- 172.21.15.70:0/1272587583 >> 172.21.15.70:0/1272587583 conn(0x7fd6040793e0 msgr2=0x7fd604111690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:00:54.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.559+0000 7fd60b350700 1 -- 172.21.15.70:0/1272587583 shutdown_connections 2024-06-04T05:00:54.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:00:54.559+0000 7fd60b350700 1 -- 172.21.15.70:0/1272587583 wait complete. 2024-06-04T05:00:54.716 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 2.45 MB/sec warmup 107 sec latency 158.883 ms 2024-06-04T05:00:55.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:55 smithi070 ceph-mon[32631]: pgmap v115: 97 pgs: 97 active+clean; 40 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 300 KiB/s wr, 39 op/s 2024-06-04T05:00:55.393 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '3ea23d63', quiesce took 6.4 seconds. Will release after: 22.65 2024-06-04T05:00:55.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:55 smithi112 ceph-mon[36453]: pgmap v115: 97 pgs: 97 active+clean; 40 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 300 KiB/s wr, 39 op/s 2024-06-04T05:00:55.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:55 smithi098 ceph-mon[35569]: pgmap v115: 97 pgs: 97 active+clean; 40 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 300 KiB/s wr, 39 op/s 2024-06-04T05:00:55.716 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 2.43 MB/sec warmup 108 sec latency 1158.995 ms 2024-06-04T05:00:56.717 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 2.41 MB/sec warmup 109 sec latency 2159.148 ms 2024-06-04T05:00:57.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:57 smithi070 ceph-mon[32631]: pgmap v116: 97 pgs: 97 active+clean; 40 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 749 KiB/s wr, 97 op/s 2024-06-04T05:00:57.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:57 smithi112 ceph-mon[36453]: pgmap v116: 97 pgs: 97 active+clean; 40 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 749 KiB/s wr, 97 op/s 2024-06-04T05:00:57.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:57 smithi098 ceph-mon[35569]: pgmap v116: 97 pgs: 97 active+clean; 40 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 749 KiB/s wr, 97 op/s 2024-06-04T05:00:57.717 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 2.38 MB/sec warmup 110 sec latency 3159.230 ms 2024-06-04T05:00:58.717 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 2.36 MB/sec warmup 111 sec latency 4159.347 ms 2024-06-04T05:00:59.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:00:59 smithi070 ceph-mon[32631]: pgmap v117: 97 pgs: 97 active+clean; 40 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 749 KiB/s wr, 93 op/s 2024-06-04T05:00:59.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:00:59 smithi112 ceph-mon[36453]: pgmap v117: 97 pgs: 97 active+clean; 40 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 749 KiB/s wr, 93 op/s 2024-06-04T05:00:59.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:00:59 smithi098 ceph-mon[35569]: pgmap v117: 97 pgs: 97 active+clean; 40 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 749 KiB/s wr, 93 op/s 2024-06-04T05:00:59.717 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 2.34 MB/sec warmup 112 sec latency 5159.469 ms 2024-06-04T05:01:00.717 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 2.32 MB/sec warmup 113 sec latency 6159.543 ms 2024-06-04T05:01:01.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:01 smithi070 ceph-mon[32631]: pgmap v118: 97 pgs: 97 active+clean; 40 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 749 KiB/s wr, 93 op/s 2024-06-04T05:01:01.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:01 smithi112 ceph-mon[36453]: pgmap v118: 97 pgs: 97 active+clean; 40 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 749 KiB/s wr, 93 op/s 2024-06-04T05:01:01.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:01 smithi098 ceph-mon[35569]: pgmap v118: 97 pgs: 97 active+clean; 40 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 749 KiB/s wr, 93 op/s 2024-06-04T05:01:01.717 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 2.30 MB/sec warmup 114 sec latency 7159.619 ms 2024-06-04T05:01:02.717 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 2.28 MB/sec warmup 115 sec latency 8159.718 ms 2024-06-04T05:01:03.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:03 smithi070 ceph-mon[32631]: pgmap v119: 97 pgs: 97 active+clean; 40 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 777 KiB/s wr, 114 op/s 2024-06-04T05:01:03.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:03 smithi112 ceph-mon[36453]: pgmap v119: 97 pgs: 97 active+clean; 40 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 777 KiB/s wr, 114 op/s 2024-06-04T05:01:03.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:03 smithi098 ceph-mon[35569]: pgmap v119: 97 pgs: 97 active+clean; 40 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 777 KiB/s wr, 114 op/s 2024-06-04T05:01:03.717 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 2.26 MB/sec warmup 116 sec latency 9159.821 ms 2024-06-04T05:01:04.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 2.24 MB/sec warmup 117 sec latency 10159.872 ms 2024-06-04T05:01:05.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:05 smithi070 ceph-mon[32631]: pgmap v120: 97 pgs: 97 active+clean; 40 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 477 KiB/s wr, 78 op/s 2024-06-04T05:01:05.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:05 smithi112 ceph-mon[36453]: pgmap v120: 97 pgs: 97 active+clean; 40 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 477 KiB/s wr, 78 op/s 2024-06-04T05:01:05.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:05 smithi098 ceph-mon[35569]: pgmap v120: 97 pgs: 97 active+clean; 40 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 477 KiB/s wr, 78 op/s 2024-06-04T05:01:05.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 2.22 MB/sec warmup 118 sec latency 11159.965 ms 2024-06-04T05:01:06.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 2.20 MB/sec warmup 119 sec latency 12160.064 ms 2024-06-04T05:01:06.970 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.968+0000 7ff746245640 1 -- 172.21.15.70:0/1458574386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff740103410 msgr2=0x7ff74010f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.968+0000 7ff746245640 1 --2- 172.21.15.70:0/1458574386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff740103410 0x7ff74010f920 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ff738009cd0 tx=0x7ff73802a8e0 comp rx=0 tx=0).stop 2024-06-04T05:01:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.968+0000 7ff746245640 1 -- 172.21.15.70:0/1458574386 shutdown_connections 2024-06-04T05:01:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.968+0000 7ff746245640 1 --2- 172.21.15.70:0/1458574386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff740103410 0x7ff74010f920 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.968+0000 7ff746245640 1 --2- 172.21.15.70:0/1458574386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff740102a50 0x7ff740102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.968+0000 7ff746245640 1 --2- 172.21.15.70:0/1458574386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff740108a60 0x7ff740108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.968+0000 7ff746245640 1 -- 172.21.15.70:0/1458574386 >> 172.21.15.70:0/1458574386 conn(0x7ff7400fe520 msgr2=0x7ff740100940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.968+0000 7ff746245640 1 -- 172.21.15.70:0/1458574386 shutdown_connections 2024-06-04T05:01:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.968+0000 7ff746245640 1 -- 172.21.15.70:0/1458574386 wait complete. 2024-06-04T05:01:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.968+0000 7ff746245640 1 Processor -- start 2024-06-04T05:01:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff746245640 1 -- start start 2024-06-04T05:01:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff746245640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff740102a50 0x7ff740199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff746245640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff740103410 0x7ff7401995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff746245640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff740108a60 0x7ff74019d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff746245640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff740112bd0 con 0x7ff740108a60 2024-06-04T05:01:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff746245640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff740112a50 con 0x7ff740103410 2024-06-04T05:01:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff746245640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff740112d50 con 0x7ff740102a50 2024-06-04T05:01:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff73ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff740108a60 0x7ff74019d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff73ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff740108a60 0x7ff74019d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45202/0 (socket says 172.21.15.70:45202) 2024-06-04T05:01:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff73ffff640 1 -- 172.21.15.70:0/2840905144 learned_addr learned my addr 172.21.15.70:0/2840905144 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff73f7fe640 1 --2- 172.21.15.70:0/2840905144 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff740102a50 0x7ff740199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff73effd640 1 --2- 172.21.15.70:0/2840905144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff740103410 0x7ff7401995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff73ffff640 1 -- 172.21.15.70:0/2840905144 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff740102a50 msgr2=0x7ff740199070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff73ffff640 1 --2- 172.21.15.70:0/2840905144 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff740102a50 0x7ff740199070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff73ffff640 1 -- 172.21.15.70:0/2840905144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff740103410 msgr2=0x7ff7401995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff73ffff640 1 --2- 172.21.15.70:0/2840905144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff740103410 0x7ff7401995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff73ffff640 1 -- 172.21.15.70:0/2840905144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff74019e060 con 0x7ff740108a60 2024-06-04T05:01:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.969+0000 7ff73ffff640 1 --2- 172.21.15.70:0/2840905144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff740108a60 0x7ff74019d930 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff738005cc0 tx=0x7ff738003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.970+0000 7ff73cff9640 1 -- 172.21.15.70:0/2840905144 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff738004040 con 0x7ff740108a60 2024-06-04T05:01:06.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.970+0000 7ff746245640 1 -- 172.21.15.70:0/2840905144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff74019e2f0 con 0x7ff740108a60 2024-06-04T05:01:06.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.970+0000 7ff73cff9640 1 -- 172.21.15.70:0/2840905144 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff738004670 con 0x7ff740108a60 2024-06-04T05:01:06.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.970+0000 7ff73cff9640 1 -- 172.21.15.70:0/2840905144 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff738030ce0 con 0x7ff740108a60 2024-06-04T05:01:06.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.970+0000 7ff746245640 1 -- 172.21.15.70:0/2840905144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7401a5be0 con 0x7ff740108a60 2024-06-04T05:01:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.973+0000 7ff748bfd740 1 Processor -- start 2024-06-04T05:01:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.973+0000 7ff73cff9640 1 -- 172.21.15.70:0/2840905144 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff7380041e0 con 0x7ff740108a60 2024-06-04T05:01:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.973+0000 7ff748bfd740 1 -- start start 2024-06-04T05:01:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.973+0000 7ff748bfd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a96360820 0x558a96360c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.973+0000 7ff748bfd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a96361140 0x558a9636b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.973+0000 7ff748bfd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a9636ba00 0x558a9636ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.973+0000 7ff748bfd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558a962d2c30 con 0x558a96361140 2024-06-04T05:01:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.973+0000 7ff748bfd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558a962c27d0 con 0x558a96360820 2024-06-04T05:01:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.973+0000 7ff748bfd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558a962c6810 con 0x558a9636ba00 2024-06-04T05:01:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.973+0000 7ff73f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a96361140 0x558a9636b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.973+0000 7ff73f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a96361140 0x558a9636b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45212/0 (socket says 172.21.15.70:45212) 2024-06-04T05:01:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.973+0000 7ff73f7fe640 1 -- 172.21.15.70:0/1279432586 learned_addr learned my addr 172.21.15.70:0/1279432586 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.974+0000 7ff73effd640 1 --2- 172.21.15.70:0/1279432586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a96360820 0x558a96360c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.974+0000 7ff73effd640 1 --2- 172.21.15.70:0/1279432586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a9636ba00 0x558a9636ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.974+0000 7ff73f7fe640 1 -- 172.21.15.70:0/1279432586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a9636ba00 msgr2=0x558a9636ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.974+0000 7ff73f7fe640 1 --2- 172.21.15.70:0/1279432586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a9636ba00 0x558a9636ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.974+0000 7ff73f7fe640 1 -- 172.21.15.70:0/1279432586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a96360820 msgr2=0x558a96360c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.974+0000 7ff73f7fe640 1 --2- 172.21.15.70:0/1279432586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a96360820 0x558a96360c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.974+0000 7ff73f7fe640 1 -- 172.21.15.70:0/1279432586 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a95fd7210 con 0x558a96361140 2024-06-04T05:01:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.974+0000 7ff73effd640 1 --2- 172.21.15.70:0/1279432586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a96360820 0x558a96360c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:01:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.974+0000 7ff73cff9640 1 --2- 172.21.15.70:0/2840905144 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff7280b2a10 0x7ff7280b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.974+0000 7ff73effd640 1 --2- 172.21.15.70:0/2840905144 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff7280b2a10 0x7ff7280b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.974+0000 7ff73f7fe640 1 --2- 172.21.15.70:0/1279432586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a96361140 0x558a9636b4c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff73000cbe0 tx=0x7ff73000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc66bc4f882c141 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.974+0000 7ff73cff9640 1 -- 172.21.15.70:0/2840905144 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ff73802e030 con 0x7ff740108a60 2024-06-04T05:01:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.974+0000 7ff725ffb640 1 -- 172.21.15.70:0/1279432586 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff730005220 con 0x558a96361140 2024-06-04T05:01:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.974+0000 7ff725ffb640 1 -- 172.21.15.70:0/1279432586 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff730004c30 con 0x558a96361140 2024-06-04T05:01:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.974+0000 7ff725ffb640 1 -- 172.21.15.70:0/1279432586 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff730010720 con 0x558a96361140 2024-06-04T05:01:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.975+0000 7ff748bfd740 1 -- 172.21.15.70:0/1279432586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a96361140 msgr2=0x558a9636b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.975+0000 7ff73effd640 1 --2- 172.21.15.70:0/2840905144 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff7280b2a10 0x7ff7280b4ed0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x558a9635e150 tx=0x7ff7340095c0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.975+0000 7ff748bfd740 1 --2- 172.21.15.70:0/1279432586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a96361140 0x558a9636b4c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff73000cbe0 tx=0x7ff73000e720 comp rx=0 tx=0).stop 2024-06-04T05:01:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.975+0000 7ff748bfd740 1 -- 172.21.15.70:0/1279432586 shutdown_connections 2024-06-04T05:01:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.975+0000 7ff748bfd740 1 --2- 172.21.15.70:0/1279432586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a9636ba00 0x558a9636ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.975+0000 7ff748bfd740 1 --2- 172.21.15.70:0/1279432586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a96361140 0x558a9636b4c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.975+0000 7ff748bfd740 1 --2- 172.21.15.70:0/1279432586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a96360820 0x558a96360c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.975+0000 7ff748bfd740 1 -- 172.21.15.70:0/1279432586 >> 172.21.15.70:0/1279432586 conn(0x558a9635cd30 msgr2=0x558a9635d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.975+0000 7ff748bfd740 1 -- 172.21.15.70:0/1279432586 shutdown_connections 2024-06-04T05:01:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.975+0000 7ff748bfd740 1 -- 172.21.15.70:0/1279432586 wait complete. 2024-06-04T05:01:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.975+0000 7ff748bfd740 1 -- start start 2024-06-04T05:01:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.976+0000 7ff748bfd740 1 Processor -- start 2024-06-04T05:01:06.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.976+0000 7ff748bfd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a96360820 0x558a963689f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:06.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.976+0000 7ff748bfd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a96361140 0x558a96368f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:06.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.976+0000 7ff748bfd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a9636ba00 0x558a96369470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:06.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.976+0000 7ff748bfd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558a95fa8790 con 0x558a9636ba00 2024-06-04T05:01:06.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.976+0000 7ff748bfd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558a9627e890 con 0x558a96361140 2024-06-04T05:01:06.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.976+0000 7ff748bfd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558a9636e630 con 0x558a96360820 2024-06-04T05:01:06.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.976+0000 7ff748bfd740 10 client.0 _ll_register_callbacks cb 0x558a95fd6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:06.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.976+0000 7ff73ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a9636ba00 0x558a96369470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.976+0000 7ff73ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a9636ba00 0x558a96369470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45222/0 (socket says 172.21.15.70:45222) 2024-06-04T05:01:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.976+0000 7ff73ffff640 1 -- 172.21.15.70:0/3946479301 learned_addr learned my addr 172.21.15.70:0/3946479301 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.976+0000 7ff73effd640 1 --2- 172.21.15.70:0/3946479301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a96361140 0x558a96368f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.976+0000 7ff73f7fe640 1 --2- 172.21.15.70:0/3946479301 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a96360820 0x558a963689f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.976+0000 7ff73ffff640 1 -- 172.21.15.70:0/3946479301 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a96360820 msgr2=0x558a963689f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff73ffff640 1 --2- 172.21.15.70:0/3946479301 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a96360820 0x558a963689f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff73ffff640 1 -- 172.21.15.70:0/3946479301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a96361140 msgr2=0x558a96368f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff73ffff640 1 --2- 172.21.15.70:0/3946479301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a96361140 0x558a96368f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff73ffff640 1 -- 172.21.15.70:0/3946479301 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a960a3620 con 0x558a9636ba00 2024-06-04T05:01:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff73ffff640 1 --2- 172.21.15.70:0/3946479301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a9636ba00 0x558a96369470 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff738005e30 tx=0x7ff7380ca590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ce94bb97d4515d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff71effd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff71effd640 1 -- 172.21.15.70:0/3946479301 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff738004820 con 0x558a9636ba00 2024-06-04T05:01:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff71effd640 1 -- 172.21.15.70:0/3946479301 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ff7380ca040 con 0x558a9636ba00 2024-06-04T05:01:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff748bfd740 1 -- 172.21.15.70:0/3946479301 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff730004c30 con 0x558a9636ba00 2024-06-04T05:01:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff71effd640 1 -- 172.21.15.70:0/3946479301 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff7380d3370 con 0x558a9636ba00 2024-06-04T05:01:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff748bfd740 10 client.16527 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff748bfd740 1 -- 172.21.15.70:0/3946479301 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558a9637a420 con 0x558a9636ba00 2024-06-04T05:01:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff71effd640 1 -- 172.21.15.70:0/3946479301 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff730004c30 con 0x558a9636ba00 2024-06-04T05:01:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff71effd640 1 -- 172.21.15.70:0/3946479301 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff7380c6050 con 0x558a9636ba00 2024-06-04T05:01:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff748bfd740 10 client.16527 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff748bfd740 10 client.16527 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff748bfd740 20 client.16527 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff748bfd740 1 --2- 172.21.15.70:0/3946479301 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558a9637b920 0x558a9637dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff748bfd740 4 client.16527 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:01:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.977+0000 7ff748bfd740 1 -- 172.21.15.70:0/3946479301 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a962d4b80 con 0x558a9637b920 2024-06-04T05:01:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.978+0000 7ff73f7fe640 1 --2- 172.21.15.70:0/3946479301 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558a9637b920 0x558a9637dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.978+0000 7ff73f7fe640 1 --2- 172.21.15.70:0/3946479301 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558a9637b920 0x558a9637dda0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bc411265b7613f3 server_cookie=1f399b1e8107d3ff in_seq=0 out_seq=0 2024-06-04T05:01:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.978+0000 7ff71effd640 10 client.16527 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.979+0000 7ff71effd640 1 -- 172.21.15.70:0/3946479301 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558a962d4b80 con 0x558a9637b920 2024-06-04T05:01:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.979+0000 7ff71effd640 10 client.16527 handle_command_reply: tid=0 2024-06-04T05:01:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.979+0000 7ff748bfd740 1 client.16527 shutdown 2024-06-04T05:01:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.979+0000 7ff748bfd740 1 -- 172.21.15.70:0/3946479301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a9636ba00 msgr2=0x558a96369470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.979+0000 7ff748bfd740 1 --2- 172.21.15.70:0/3946479301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a9636ba00 0x558a96369470 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff738005e30 tx=0x7ff7380ca590 comp rx=0 tx=0).stop 2024-06-04T05:01:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.979+0000 7ff748bfd740 1 -- 172.21.15.70:0/3946479301 shutdown_connections 2024-06-04T05:01:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.979+0000 7ff748bfd740 1 --2- 172.21.15.70:0/3946479301 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558a9637b920 0x558a9637dda0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.979+0000 7ff748bfd740 1 --2- 172.21.15.70:0/3946479301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a96361140 0x558a96368f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.979+0000 7ff71effd640 0 client.16527 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.979+0000 7ff748bfd740 1 --2- 172.21.15.70:0/3946479301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a9636ba00 0x558a96369470 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.979+0000 7ff748bfd740 1 --2- 172.21.15.70:0/3946479301 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a96360820 0x558a963689f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.979+0000 7ff748bfd740 1 -- 172.21.15.70:0/3946479301 >> 172.21.15.70:0/3946479301 conn(0x558a9635cd30 msgr2=0x558a9635bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.979+0000 7ff748bfd740 1 -- 172.21.15.70:0/3946479301 shutdown_connections 2024-06-04T05:01:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.979+0000 7ff748bfd740 1 -- 172.21.15.70:0/3946479301 wait complete. 2024-06-04T05:01:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.979+0000 7ff748bfd740 20 client.16527 trim_cache size 0 max 16384 2024-06-04T05:01:07.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff748bfd740 1 Processor -- start 2024-06-04T05:01:07.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff748bfd740 1 -- start start 2024-06-04T05:01:07.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff748bfd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a96360820 0x558a963690c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:07.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff748bfd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a96361140 0x558a96383660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:07.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff748bfd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a9636ba00 0x558a96383ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:07.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff748bfd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff7401ae4f0 con 0x558a96360820 2024-06-04T05:01:07.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff748bfd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7401b0090 con 0x558a9636ba00 2024-06-04T05:01:07.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff748bfd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff7401aebc0 con 0x558a96361140 2024-06-04T05:01:07.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff73f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a96360820 0x558a963690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:07.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff73f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a96360820 0x558a963690c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45232/0 (socket says 172.21.15.70:45232) 2024-06-04T05:01:07.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff73f7fe640 1 -- 172.21.15.70:0/1456958909 learned_addr learned my addr 172.21.15.70:0/1456958909 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:07.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff73ffff640 1 --2- 172.21.15.70:0/1456958909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a9636ba00 0x558a96383ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:07.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff73effd640 1 --2- 172.21.15.70:0/1456958909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a96361140 0x558a96383660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:07.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff73f7fe640 1 -- 172.21.15.70:0/1456958909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a96361140 msgr2=0x558a96383660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:07.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff73f7fe640 1 --2- 172.21.15.70:0/1456958909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a96361140 0x558a96383660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:07.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff73f7fe640 1 -- 172.21.15.70:0/1456958909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a9636ba00 msgr2=0x558a96383ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:07.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff73f7fe640 1 --2- 172.21.15.70:0/1456958909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a9636ba00 0x558a96383ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:07.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff73f7fe640 1 -- 172.21.15.70:0/1456958909 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a963586f0 con 0x558a96360820 2024-06-04T05:01:07.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff73f7fe640 1 --2- 172.21.15.70:0/1456958909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a96360820 0x558a963690c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff730027fd0 tx=0x7ff73000c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e267feeca9d77a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:07.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.995+0000 7ff7257fa640 1 -- 172.21.15.70:0/1456958909 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff730010630 con 0x558a96360820 2024-06-04T05:01:07.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.996+0000 7ff7257fa640 1 -- 172.21.15.70:0/1456958909 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff730010c60 con 0x558a96360820 2024-06-04T05:01:07.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.996+0000 7ff7257fa640 1 -- 172.21.15.70:0/1456958909 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff730025b40 con 0x558a96360820 2024-06-04T05:01:07.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.996+0000 7ff748bfd740 1 -- 172.21.15.70:0/1456958909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a96360820 msgr2=0x558a963690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:07.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.996+0000 7ff748bfd740 1 --2- 172.21.15.70:0/1456958909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a96360820 0x558a963690c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff730027fd0 tx=0x7ff73000c370 comp rx=0 tx=0).stop 2024-06-04T05:01:07.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.996+0000 7ff748bfd740 1 -- 172.21.15.70:0/1456958909 shutdown_connections 2024-06-04T05:01:07.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.996+0000 7ff748bfd740 1 --2- 172.21.15.70:0/1456958909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a9636ba00 0x558a96383ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:07.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.996+0000 7ff748bfd740 1 --2- 172.21.15.70:0/1456958909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a96361140 0x558a96383660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:07.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.996+0000 7ff748bfd740 1 --2- 172.21.15.70:0/1456958909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a96360820 0x558a963690c0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:07.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.996+0000 7ff748bfd740 1 -- 172.21.15.70:0/1456958909 >> 172.21.15.70:0/1456958909 conn(0x558a9635cd30 msgr2=0x558a963689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:07.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.996+0000 7ff748bfd740 1 -- 172.21.15.70:0/1456958909 shutdown_connections 2024-06-04T05:01:07.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.996+0000 7ff748bfd740 1 -- 172.21.15.70:0/1456958909 wait complete. 2024-06-04T05:01:07.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff748bfd740 1 -- start start 2024-06-04T05:01:07.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff748bfd740 1 Processor -- start 2024-06-04T05:01:07.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff748bfd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a96360820 0x558a9636fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:07.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff748bfd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a96361140 0x558a96370220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:07.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff748bfd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a9636ba00 0x558a963e5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff748bfd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff7401af960 con 0x558a9636ba00 2024-06-04T05:01:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff748bfd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7401af290 con 0x558a96361140 2024-06-04T05:01:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff748bfd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff7401ac4d0 con 0x558a96360820 2024-06-04T05:01:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff748bfd740 10 client.0 _ll_register_callbacks cb 0x7ff7380305a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff73ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a9636ba00 0x558a963e5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff73ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a9636ba00 0x558a963e5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45234/0 (socket says 172.21.15.70:45234) 2024-06-04T05:01:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff73ffff640 1 -- 172.21.15.70:0/223904758 learned_addr learned my addr 172.21.15.70:0/223904758 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff73f7fe640 1 --2- 172.21.15.70:0/223904758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a96360820 0x558a9636fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff73effd640 1 --2- 172.21.15.70:0/223904758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a96361140 0x558a96370220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff73ffff640 1 -- 172.21.15.70:0/223904758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a96360820 msgr2=0x558a9636fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff73ffff640 1 --2- 172.21.15.70:0/223904758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a96360820 0x558a9636fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff73ffff640 1 -- 172.21.15.70:0/223904758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a96361140 msgr2=0x558a96370220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.997+0000 7ff73ffff640 1 --2- 172.21.15.70:0/223904758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a96361140 0x558a96370220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff73ffff640 1 -- 172.21.15.70:0/223904758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a963e5a70 con 0x558a9636ba00 2024-06-04T05:01:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff73effd640 1 --2- 172.21.15.70:0/223904758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a96361140 0x558a96370220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:01:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff73ffff640 1 --2- 172.21.15.70:0/223904758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a9636ba00 0x558a963e5530 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff7380c7170 tx=0x7ff7380095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db87758dc655a74 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff71effd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff71effd640 1 -- 172.21.15.70:0/223904758 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7380ea060 con 0x558a9636ba00 2024-06-04T05:01:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff71effd640 1 -- 172.21.15.70:0/223904758 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ff7380c6750 con 0x558a9636ba00 2024-06-04T05:01:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff748bfd740 1 -- 172.21.15.70:0/223904758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff730010c60 con 0x558a9636ba00 2024-06-04T05:01:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff71effd640 1 -- 172.21.15.70:0/223904758 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff7380ca340 con 0x558a9636ba00 2024-06-04T05:01:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff748bfd740 10 client.16533 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff748bfd740 1 -- 172.21.15.70:0/223904758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558a963e6d20 con 0x558a9636ba00 2024-06-04T05:01:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff71effd640 1 -- 172.21.15.70:0/223904758 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff730010c60 con 0x558a9636ba00 2024-06-04T05:01:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff71effd640 1 -- 172.21.15.70:0/223904758 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff7380c6040 con 0x558a9636ba00 2024-06-04T05:01:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff748bfd740 10 client.16533 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff748bfd740 10 client.16533 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff748bfd740 20 client.16533 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff748bfd740 1 --2- 172.21.15.70:0/223904758 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558a963e7310 0x558a963e9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff748bfd740 4 client.16533 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:01:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.998+0000 7ff748bfd740 1 -- 172.21.15.70:0/223904758 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558a962ce360 con 0x558a963e7310 2024-06-04T05:01:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.999+0000 7ff73f7fe640 1 --2- 172.21.15.70:0/223904758 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558a963e7310 0x558a963e9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.999+0000 7ff73f7fe640 1 --2- 172.21.15.70:0/223904758 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558a963e7310 0x558a963e9730 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5424df4030e7e70e server_cookie=46ce54220cb3070e in_seq=0 out_seq=0 2024-06-04T05:01:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:06.999+0000 7ff71effd640 10 client.16533 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:07.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.019+0000 7ff71effd640 1 -- 172.21.15.70:0/223904758 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x558a962ce360 con 0x558a963e7310 2024-06-04T05:01:07.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.019+0000 7ff71effd640 10 client.16533 handle_command_reply: tid=0 2024-06-04T05:01:07.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.019+0000 7ff748bfd740 1 client.16533 shutdown 2024-06-04T05:01:07.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.019+0000 7ff748bfd740 1 -- 172.21.15.70:0/223904758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a9636ba00 msgr2=0x558a963e5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:07.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.019+0000 7ff748bfd740 1 --2- 172.21.15.70:0/223904758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a9636ba00 0x558a963e5530 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff7380c7170 tx=0x7ff7380095c0 comp rx=0 tx=0).stop 2024-06-04T05:01:07.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.019+0000 7ff748bfd740 1 -- 172.21.15.70:0/223904758 shutdown_connections 2024-06-04T05:01:07.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.019+0000 7ff748bfd740 1 --2- 172.21.15.70:0/223904758 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558a963e7310 0x558a963e9730 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:07.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.019+0000 7ff748bfd740 1 --2- 172.21.15.70:0/223904758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a96361140 0x558a96370220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:07.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.019+0000 7ff748bfd740 1 --2- 172.21.15.70:0/223904758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a9636ba00 0x558a963e5530 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.019+0000 7ff748bfd740 1 --2- 172.21.15.70:0/223904758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a96360820 0x558a9636fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.019+0000 7ff71effd640 0 client.16533 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.019+0000 7ff748bfd740 1 -- 172.21.15.70:0/223904758 >> 172.21.15.70:0/223904758 conn(0x558a9635cd30 msgr2=0x558a96368990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.019+0000 7ff748bfd740 1 -- 172.21.15.70:0/223904758 shutdown_connections 2024-06-04T05:01:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.019+0000 7ff748bfd740 1 -- 172.21.15.70:0/223904758 wait complete. 2024-06-04T05:01:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.019+0000 7ff748bfd740 20 client.16533 trim_cache size 0 max 16384 2024-06-04T05:01:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:01:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:01:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:01:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.020+0000 7ff746245640 1 -- 172.21.15.70:0/2840905144 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff7280b2a10 msgr2=0x7ff7280b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.020+0000 7ff746245640 1 --2- 172.21.15.70:0/2840905144 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff7280b2a10 0x7ff7280b4ed0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x558a9635e150 tx=0x7ff7340095c0 comp rx=0 tx=0).stop 2024-06-04T05:01:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.020+0000 7ff746245640 1 -- 172.21.15.70:0/2840905144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff740108a60 msgr2=0x7ff74019d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.020+0000 7ff746245640 1 --2- 172.21.15.70:0/2840905144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff740108a60 0x7ff74019d930 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff738005cc0 tx=0x7ff738003c00 comp rx=0 tx=0).stop 2024-06-04T05:01:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.020+0000 7ff746245640 1 -- 172.21.15.70:0/2840905144 shutdown_connections 2024-06-04T05:01:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.020+0000 7ff746245640 1 --2- 172.21.15.70:0/2840905144 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff7280b2a10 0x7ff7280b4ed0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.020+0000 7ff746245640 1 --2- 172.21.15.70:0/2840905144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff740103410 0x7ff7401995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.020+0000 7ff746245640 1 --2- 172.21.15.70:0/2840905144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff740108a60 0x7ff74019d930 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.020+0000 7ff746245640 1 --2- 172.21.15.70:0/2840905144 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff740102a50 0x7ff740199070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.020+0000 7ff746245640 1 -- 172.21.15.70:0/2840905144 >> 172.21.15.70:0/2840905144 conn(0x7ff7400fe520 msgr2=0x7ff7400fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.020+0000 7ff746245640 1 -- 172.21.15.70:0/2840905144 shutdown_connections 2024-06-04T05:01:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:07.020+0000 7ff746245640 1 -- 172.21.15.70:0/2840905144 wait complete. 2024-06-04T05:01:07.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:07 smithi070 ceph-mon[32631]: pgmap v121: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 481 KiB/s wr, 84 op/s 2024-06-04T05:01:07.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:07 smithi112 ceph-mon[36453]: pgmap v121: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 481 KiB/s wr, 84 op/s 2024-06-04T05:01:07.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:07 smithi098 ceph-mon[35569]: pgmap v121: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 481 KiB/s wr, 84 op/s 2024-06-04T05:01:08.163 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:01:08.435 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:01:08.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 1 sec latency 14160.180 ms 2024-06-04T05:01:09.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:09 smithi070 ceph-mon[32631]: pgmap v122: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 33 KiB/s wr, 27 op/s 2024-06-04T05:01:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:09 smithi112 ceph-mon[36453]: pgmap v122: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 33 KiB/s wr, 27 op/s 2024-06-04T05:01:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:09 smithi098 ceph-mon[35569]: pgmap v122: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 33 KiB/s wr, 27 op/s 2024-06-04T05:01:09.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 2 sec latency 15160.207 ms 2024-06-04T05:01:10.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.240+0000 7f545357e700 1 -- 172.21.15.70:0/1076395026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c102e00 msgr2=0x7f544c10f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:10.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.240+0000 7f545357e700 1 --2- 172.21.15.70:0/1076395026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c102e00 0x7f544c10f2f0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f5444009990 tx=0x7f5444009ca0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.240+0000 7f545357e700 1 -- 172.21.15.70:0/1076395026 shutdown_connections 2024-06-04T05:01:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.240+0000 7f545357e700 1 --2- 172.21.15.70:0/1076395026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c102e00 0x7f544c10f2f0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.240+0000 7f545357e700 1 --2- 172.21.15.70:0/1076395026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f544c108520 0x7f544c108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.240+0000 7f545357e700 1 --2- 172.21.15.70:0/1076395026 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f544c102440 0x7f544c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.240+0000 7f545357e700 1 -- 172.21.15.70:0/1076395026 >> 172.21.15.70:0/1076395026 conn(0x7f544c0793f0 msgr2=0x7f544c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.240+0000 7f545357e700 1 -- 172.21.15.70:0/1076395026 shutdown_connections 2024-06-04T05:01:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.240+0000 7f545357e700 1 -- 172.21.15.70:0/1076395026 wait complete. 2024-06-04T05:01:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.241+0000 7f545357e700 1 Processor -- start 2024-06-04T05:01:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.241+0000 7f545357e700 1 -- start start 2024-06-04T05:01:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.241+0000 7f545357e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f544c102440 0x7f544c19bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.241+0000 7f545357e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c102e00 0x7f544c19c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.241+0000 7f545357e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f544c108520 0x7f544c1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.241+0000 7f545357e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f544c112500 con 0x7f544c102e00 2024-06-04T05:01:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.241+0000 7f545357e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f544c112380 con 0x7f544c102440 2024-06-04T05:01:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.241+0000 7f545357e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f544c112680 con 0x7f544c108520 2024-06-04T05:01:10.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.241+0000 7f5450b19700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c102e00 0x7f544c19c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.241+0000 7f5450b19700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c102e00 0x7f544c19c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45260/0 (socket says 172.21.15.70:45260) 2024-06-04T05:01:10.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.241+0000 7f5450b19700 1 -- 172.21.15.70:0/1924637627 learned_addr learned my addr 172.21.15.70:0/1924637627 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:10.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.241+0000 7f545131a700 1 --2- 172.21.15.70:0/1924637627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f544c102440 0x7f544c19bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.242+0000 7f5451b1b700 1 --2- 172.21.15.70:0/1924637627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f544c108520 0x7f544c1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.242+0000 7f5450b19700 1 -- 172.21.15.70:0/1924637627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f544c108520 msgr2=0x7f544c1a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:10.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.242+0000 7f5450b19700 1 --2- 172.21.15.70:0/1924637627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f544c108520 0x7f544c1a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.242+0000 7f5450b19700 1 -- 172.21.15.70:0/1924637627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f544c102440 msgr2=0x7f544c19bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:10.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.242+0000 7f5450b19700 1 --2- 172.21.15.70:0/1924637627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f544c102440 0x7f544c19bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.242+0000 7f5450b19700 1 -- 172.21.15.70:0/1924637627 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f544c1a0da0 con 0x7f544c102e00 2024-06-04T05:01:10.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.242+0000 7f5450b19700 1 --2- 172.21.15.70:0/1924637627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c102e00 0x7f544c19c2d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f543c00b530 tx=0x7f543c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:10.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.242+0000 7f54427fc700 1 -- 172.21.15.70:0/1924637627 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543c015020 con 0x7f544c102e00 2024-06-04T05:01:10.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.242+0000 7f545357e700 1 -- 172.21.15.70:0/1924637627 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f544c1a1090 con 0x7f544c102e00 2024-06-04T05:01:10.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.242+0000 7f54427fc700 1 -- 172.21.15.70:0/1924637627 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f543c011cc0 con 0x7f544c102e00 2024-06-04T05:01:10.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.242+0000 7f54427fc700 1 -- 172.21.15.70:0/1924637627 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543c010dc0 con 0x7f544c102e00 2024-06-04T05:01:10.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.242+0000 7f545357e700 1 -- 172.21.15.70:0/1924637627 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f544c1a8910 con 0x7f544c102e00 2024-06-04T05:01:10.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.244+0000 7f54427fc700 1 -- 172.21.15.70:0/1924637627 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f543c018020 con 0x7f544c102e00 2024-06-04T05:01:10.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.245+0000 7f54427fc700 1 --2- 172.21.15.70:0/1924637627 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f54380b29f0 0x7f54380b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.245+0000 7f545131a700 1 --2- 172.21.15.70:0/1924637627 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f54380b29f0 0x7f54380b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.245+0000 7f54427fc700 1 -- 172.21.15.70:0/1924637627 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f543c0e1df0 con 0x7f544c102e00 2024-06-04T05:01:10.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.245+0000 7f545131a700 1 --2- 172.21.15.70:0/1924637627 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f54380b29f0 0x7f54380b4eb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5448009550 tx=0x7f54480091b0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:10.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.246+0000 7f545de13b80 1 Processor -- start 2024-06-04T05:01:10.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.246+0000 7f545de13b80 1 -- start start 2024-06-04T05:01:10.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.246+0000 7f545de13b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4acf320 0x5605b4acf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.246+0000 7f545de13b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4ac98a0 0x5605b4ac9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.246+0000 7f545de13b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4aca260 0x5605b4ad6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.246+0000 7f545de13b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5605b4a27d70 con 0x5605b4acf320 2024-06-04T05:01:10.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.246+0000 7f545de13b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5605b4a17910 con 0x5605b4aca260 2024-06-04T05:01:10.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.246+0000 7f545de13b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5605b4a1b950 con 0x5605b4ac98a0 2024-06-04T05:01:10.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.246+0000 7f545131a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4acf320 0x5605b4acf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.246+0000 7f545131a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4acf320 0x5605b4acf700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45270/0 (socket says 172.21.15.70:45270) 2024-06-04T05:01:10.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.246+0000 7f545131a700 1 -- 172.21.15.70:0/1792772567 learned_addr learned my addr 172.21.15.70:0/1792772567 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:10.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.246+0000 7f5451b1b700 1 --2- 172.21.15.70:0/1792772567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4aca260 0x5605b4ad6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.246+0000 7f5450b19700 1 --2- 172.21.15.70:0/1792772567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4ac98a0 0x5605b4ac9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.247+0000 7f545131a700 1 -- 172.21.15.70:0/1792772567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4ac98a0 msgr2=0x5605b4ac9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:10.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.247+0000 7f545131a700 1 --2- 172.21.15.70:0/1792772567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4ac98a0 0x5605b4ac9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.247+0000 7f545131a700 1 -- 172.21.15.70:0/1792772567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4aca260 msgr2=0x5605b4ad6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:10.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.247+0000 7f545131a700 1 --2- 172.21.15.70:0/1792772567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4aca260 0x5605b4ad6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.247+0000 7f545131a700 1 -- 172.21.15.70:0/1792772567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605b49c0fb0 con 0x5605b4acf320 2024-06-04T05:01:10.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.247+0000 7f5451b1b700 1 --2- 172.21.15.70:0/1792772567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4aca260 0x5605b4ad6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:01:10.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.247+0000 7f545131a700 1 --2- 172.21.15.70:0/1792772567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4acf320 0x5605b4acf700 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f54480046e0 tx=0x7f5448013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e5e8acd8872acb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:10.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.247+0000 7f54377fe700 1 -- 172.21.15.70:0/1792772567 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f544801c050 con 0x5605b4acf320 2024-06-04T05:01:10.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.247+0000 7f54377fe700 1 -- 172.21.15.70:0/1792772567 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5448026070 con 0x5605b4acf320 2024-06-04T05:01:10.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.247+0000 7f54377fe700 1 -- 172.21.15.70:0/1792772567 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5448015640 con 0x5605b4acf320 2024-06-04T05:01:10.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.248+0000 7f545de13b80 1 -- 172.21.15.70:0/1792772567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4acf320 msgr2=0x5605b4acf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:10.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.248+0000 7f545de13b80 1 --2- 172.21.15.70:0/1792772567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4acf320 0x5605b4acf700 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f54480046e0 tx=0x7f5448013000 comp rx=0 tx=0).stop 2024-06-04T05:01:10.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.248+0000 7f545de13b80 1 -- 172.21.15.70:0/1792772567 shutdown_connections 2024-06-04T05:01:10.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.248+0000 7f545de13b80 1 --2- 172.21.15.70:0/1792772567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4aca260 0x5605b4ad6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.248+0000 7f545de13b80 1 --2- 172.21.15.70:0/1792772567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4acf320 0x5605b4acf700 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.248+0000 7f545de13b80 1 --2- 172.21.15.70:0/1792772567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4ac98a0 0x5605b4ac9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.248+0000 7f545de13b80 1 -- 172.21.15.70:0/1792772567 >> 172.21.15.70:0/1792772567 conn(0x5605b49f5240 msgr2=0x5605b4ac43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:10.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.248+0000 7f545de13b80 1 -- 172.21.15.70:0/1792772567 shutdown_connections 2024-06-04T05:01:10.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.248+0000 7f545de13b80 1 -- 172.21.15.70:0/1792772567 wait complete. 2024-06-04T05:01:10.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.248+0000 7f545de13b80 1 -- start start 2024-06-04T05:01:10.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.248+0000 7f545de13b80 1 Processor -- start 2024-06-04T05:01:10.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f545de13b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4ac98a0 0x5605b4ad5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f545de13b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4aca260 0x5605b4ad3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f545de13b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4acf320 0x5605b4ad3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f545de13b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5605b491dfa0 con 0x5605b4ac98a0 2024-06-04T05:01:10.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f545de13b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5605b497fda0 con 0x5605b4aca260 2024-06-04T05:01:10.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f545de13b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5605b4ad66e0 con 0x5605b4acf320 2024-06-04T05:01:10.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f545de13b80 10 client.0 _ll_register_callbacks cb 0x5605b49b2950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:10.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f545131a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4ac98a0 0x5605b4ad5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f545131a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4ac98a0 0x5605b4ad5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45280/0 (socket says 172.21.15.70:45280) 2024-06-04T05:01:10.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f545131a700 1 -- 172.21.15.70:0/1667246098 learned_addr learned my addr 172.21.15.70:0/1667246098 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:10.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f5450b19700 1 --2- 172.21.15.70:0/1667246098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4aca260 0x5605b4ad3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f5451b1b700 1 --2- 172.21.15.70:0/1667246098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4acf320 0x5605b4ad3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f545131a700 1 -- 172.21.15.70:0/1667246098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4acf320 msgr2=0x5605b4ad3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:10.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f545131a700 1 --2- 172.21.15.70:0/1667246098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4acf320 0x5605b4ad3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f545131a700 1 -- 172.21.15.70:0/1667246098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4aca260 msgr2=0x5605b4ad3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:10.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f545131a700 1 --2- 172.21.15.70:0/1667246098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4aca260 0x5605b4ad3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f545131a700 1 -- 172.21.15.70:0/1667246098 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605b49678d0 con 0x5605b4ac98a0 2024-06-04T05:01:10.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f545131a700 1 --2- 172.21.15.70:0/1667246098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4ac98a0 0x5605b4ad5ce0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5448021720 tx=0x7f5448021750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e753dc50c7fea030 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:10.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f5434ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:10.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f5434ff9700 1 -- 172.21.15.70:0/1667246098 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5448004bf0 con 0x5605b4ac98a0 2024-06-04T05:01:10.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f5434ff9700 1 -- 172.21.15.70:0/1667246098 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f5448004d90 con 0x5605b4ac98a0 2024-06-04T05:01:10.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.249+0000 7f545de13b80 1 -- 172.21.15.70:0/1667246098 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5448026070 con 0x5605b4ac98a0 2024-06-04T05:01:10.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.250+0000 7f5434ff9700 1 -- 172.21.15.70:0/1667246098 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f544801c020 con 0x5605b4ac98a0 2024-06-04T05:01:10.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.250+0000 7f545de13b80 10 client.16545 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:10.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.250+0000 7f545de13b80 1 -- 172.21.15.70:0/1667246098 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5605b49d1760 con 0x5605b4ac98a0 2024-06-04T05:01:10.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.250+0000 7f5434ff9700 1 -- 172.21.15.70:0/1667246098 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5448026070 con 0x5605b4ac98a0 2024-06-04T05:01:10.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.250+0000 7f5434ff9700 1 -- 172.21.15.70:0/1667246098 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f54480050e0 con 0x5605b4ac98a0 2024-06-04T05:01:10.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.250+0000 7f545de13b80 10 client.16545 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:10.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.250+0000 7f545de13b80 10 client.16545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:10.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.250+0000 7f545de13b80 20 client.16545 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:10.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.250+0000 7f545de13b80 1 --2- 172.21.15.70:0/1667246098 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5605b4ae30e0 0x5605b4ae54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.250+0000 7f545de13b80 4 client.16545 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:01:10.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.250+0000 7f545de13b80 1 -- 172.21.15.70:0/1667246098 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605b4a29cc0 con 0x5605b4ae30e0 2024-06-04T05:01:10.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.250+0000 7f5450b19700 1 --2- 172.21.15.70:0/1667246098 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5605b4ae30e0 0x5605b4ae54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.251+0000 7f5434ff9700 10 client.16545 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:10.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.251+0000 7f5450b19700 1 --2- 172.21.15.70:0/1667246098 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5605b4ae30e0 0x5605b4ae54d0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92787a0a7d490852 server_cookie=a9dc389ccad42a in_seq=0 out_seq=0 2024-06-04T05:01:10.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.251+0000 7f5434ff9700 1 -- 172.21.15.70:0/1667246098 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5605b4a29cc0 con 0x5605b4ae30e0 2024-06-04T05:01:10.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.251+0000 7f5434ff9700 10 client.16545 handle_command_reply: tid=0 2024-06-04T05:01:10.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.251+0000 7f545de13b80 1 client.16545 shutdown 2024-06-04T05:01:10.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.252+0000 7f545de13b80 1 -- 172.21.15.70:0/1667246098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4ac98a0 msgr2=0x5605b4ad5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:10.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.252+0000 7f545de13b80 1 --2- 172.21.15.70:0/1667246098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4ac98a0 0x5605b4ad5ce0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5448021720 tx=0x7f5448021750 comp rx=0 tx=0).stop 2024-06-04T05:01:10.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.252+0000 7f545de13b80 1 -- 172.21.15.70:0/1667246098 shutdown_connections 2024-06-04T05:01:10.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.252+0000 7f545de13b80 1 --2- 172.21.15.70:0/1667246098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4acf320 0x5605b4ad3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.252+0000 7f545de13b80 1 --2- 172.21.15.70:0/1667246098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4ac98a0 0x5605b4ad5ce0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.252+0000 7f545de13b80 1 --2- 172.21.15.70:0/1667246098 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5605b4ae30e0 0x5605b4ae54d0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.252+0000 7f545de13b80 1 --2- 172.21.15.70:0/1667246098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4aca260 0x5605b4ad3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.252+0000 7f5434ff9700 0 client.16545 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:10.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.252+0000 7f545de13b80 1 -- 172.21.15.70:0/1667246098 >> 172.21.15.70:0/1667246098 conn(0x5605b49f5240 msgr2=0x5605b4ac3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:10.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.252+0000 7f545de13b80 1 -- 172.21.15.70:0/1667246098 shutdown_connections 2024-06-04T05:01:10.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.252+0000 7f545de13b80 1 -- 172.21.15.70:0/1667246098 wait complete. 2024-06-04T05:01:10.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.252+0000 7f545de13b80 20 client.16545 trim_cache size 0 max 16384 2024-06-04T05:01:10.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.272+0000 7f545de13b80 1 Processor -- start 2024-06-04T05:01:10.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.272+0000 7f545de13b80 1 -- start start 2024-06-04T05:01:10.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.272+0000 7f545de13b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4ac98a0 0x5605b4ad1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.272+0000 7f545de13b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4aca260 0x5605b4ac4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.272+0000 7f545de13b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4acf320 0x5605b4ad0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.272+0000 7f545de13b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f544c1affe0 con 0x5605b4aca260 2024-06-04T05:01:10.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.272+0000 7f545de13b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f544c1b1b80 con 0x5605b4acf320 2024-06-04T05:01:10.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.272+0000 7f545de13b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f544c1b06b0 con 0x5605b4ac98a0 2024-06-04T05:01:10.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.272+0000 7f5450b19700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4aca260 0x5605b4ac4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.272+0000 7f5450b19700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4aca260 0x5605b4ac4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45290/0 (socket says 172.21.15.70:45290) 2024-06-04T05:01:10.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.272+0000 7f5450b19700 1 -- 172.21.15.70:0/552448857 learned_addr learned my addr 172.21.15.70:0/552448857 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:10.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.273+0000 7f5451b1b700 1 --2- 172.21.15.70:0/552448857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4acf320 0x5605b4ad0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.273+0000 7f545131a700 1 --2- 172.21.15.70:0/552448857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4ac98a0 0x5605b4ad1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.273+0000 7f5450b19700 1 -- 172.21.15.70:0/552448857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4ac98a0 msgr2=0x5605b4ad1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:10.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.273+0000 7f5450b19700 1 --2- 172.21.15.70:0/552448857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4ac98a0 0x5605b4ad1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.273+0000 7f5450b19700 1 -- 172.21.15.70:0/552448857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4acf320 msgr2=0x5605b4ad0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:10.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.273+0000 7f5450b19700 1 --2- 172.21.15.70:0/552448857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4acf320 0x5605b4ad0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.273+0000 7f5450b19700 1 -- 172.21.15.70:0/552448857 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605b4acef40 con 0x5605b4aca260 2024-06-04T05:01:10.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.273+0000 7f545131a700 1 --2- 172.21.15.70:0/552448857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4ac98a0 0x5605b4ad1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:01:10.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.273+0000 7f5451b1b700 1 --2- 172.21.15.70:0/552448857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4acf320 0x5605b4ad0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:01:10.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.273+0000 7f5450b19700 1 --2- 172.21.15.70:0/552448857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4aca260 0x5605b4ac4270 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f543c0a8ee0 tx=0x7f543c011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a75799ea3339ce2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:10.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.273+0000 7f5436ffd700 1 -- 172.21.15.70:0/552448857 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543c0ad590 con 0x5605b4aca260 2024-06-04T05:01:10.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.273+0000 7f5436ffd700 1 -- 172.21.15.70:0/552448857 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f543c0b2040 con 0x5605b4aca260 2024-06-04T05:01:10.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.273+0000 7f5436ffd700 1 -- 172.21.15.70:0/552448857 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543c0a9440 con 0x5605b4aca260 2024-06-04T05:01:10.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.274+0000 7f545de13b80 1 -- 172.21.15.70:0/552448857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4aca260 msgr2=0x5605b4ac4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:10.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.274+0000 7f545de13b80 1 --2- 172.21.15.70:0/552448857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4aca260 0x5605b4ac4270 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f543c0a8ee0 tx=0x7f543c011ef0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.274+0000 7f545de13b80 1 -- 172.21.15.70:0/552448857 shutdown_connections 2024-06-04T05:01:10.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.274+0000 7f545de13b80 1 --2- 172.21.15.70:0/552448857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4acf320 0x5605b4ad0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.274+0000 7f545de13b80 1 --2- 172.21.15.70:0/552448857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4ac98a0 0x5605b4ad1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.274+0000 7f545de13b80 1 --2- 172.21.15.70:0/552448857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4aca260 0x5605b4ac4270 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.274+0000 7f545de13b80 1 -- 172.21.15.70:0/552448857 >> 172.21.15.70:0/552448857 conn(0x5605b49f5240 msgr2=0x5605b4ac9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.274+0000 7f545de13b80 1 -- 172.21.15.70:0/552448857 shutdown_connections 2024-06-04T05:01:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.274+0000 7f545de13b80 1 -- 172.21.15.70:0/552448857 wait complete. 2024-06-04T05:01:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.274+0000 7f545de13b80 1 -- start start 2024-06-04T05:01:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.274+0000 7f545de13b80 1 Processor -- start 2024-06-04T05:01:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f545de13b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4ac98a0 0x5605b4adb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f545de13b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4aca260 0x5605b4adbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f545de13b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4acf320 0x5605b4ad68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f545de13b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f544c1b1450 con 0x5605b4aca260 2024-06-04T05:01:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f545de13b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f544c1b0d80 con 0x5605b4acf320 2024-06-04T05:01:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f545de13b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f544c1adf80 con 0x5605b4ac98a0 2024-06-04T05:01:10.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f545de13b80 10 client.0 _ll_register_callbacks cb 0x5605b49b2950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f5450b19700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4aca260 0x5605b4adbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f5450b19700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4aca260 0x5605b4adbe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45304/0 (socket says 172.21.15.70:45304) 2024-06-04T05:01:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f5450b19700 1 -- 172.21.15.70:0/2268023856 learned_addr learned my addr 172.21.15.70:0/2268023856 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f545131a700 1 --2- 172.21.15.70:0/2268023856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4ac98a0 0x5605b4adb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f5451b1b700 1 --2- 172.21.15.70:0/2268023856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4acf320 0x5605b4ad68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f5450b19700 1 -- 172.21.15.70:0/2268023856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4ac98a0 msgr2=0x5605b4adb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f5450b19700 1 --2- 172.21.15.70:0/2268023856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4ac98a0 0x5605b4adb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f5450b19700 1 -- 172.21.15.70:0/2268023856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4acf320 msgr2=0x5605b4ad68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f5450b19700 1 --2- 172.21.15.70:0/2268023856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4acf320 0x5605b4ad68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f5450b19700 1 -- 172.21.15.70:0/2268023856 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605b4ad6e60 con 0x5605b4aca260 2024-06-04T05:01:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f5451b1b700 1 --2- 172.21.15.70:0/2268023856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4acf320 0x5605b4ad68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:01:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f5450b19700 1 --2- 172.21.15.70:0/2268023856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4aca260 0x5605b4adbe80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f543c0df700 tx=0x7f543c0a8cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5781750c8a7a5b0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f5434ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:10.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f5434ff9700 1 -- 172.21.15.70:0/2268023856 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543c0ad150 con 0x5605b4aca260 2024-06-04T05:01:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f5434ff9700 1 -- 172.21.15.70:0/2268023856 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f543c0ad2f0 con 0x5605b4aca260 2024-06-04T05:01:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.275+0000 7f545de13b80 1 -- 172.21.15.70:0/2268023856 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f543c0b2040 con 0x5605b4aca260 2024-06-04T05:01:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.276+0000 7f5434ff9700 1 -- 172.21.15.70:0/2268023856 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f543c0a9440 con 0x5605b4aca260 2024-06-04T05:01:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.276+0000 7f545de13b80 10 client.16551 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.276+0000 7f545de13b80 1 -- 172.21.15.70:0/2268023856 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5605b4b4e0d0 con 0x5605b4aca260 2024-06-04T05:01:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.276+0000 7f5434ff9700 1 -- 172.21.15.70:0/2268023856 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f543c0b2040 con 0x5605b4aca260 2024-06-04T05:01:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.276+0000 7f5434ff9700 1 -- 172.21.15.70:0/2268023856 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f543c0a8440 con 0x5605b4aca260 2024-06-04T05:01:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.276+0000 7f545de13b80 10 client.16551 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.276+0000 7f545de13b80 10 client.16551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.276+0000 7f545de13b80 20 client.16551 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.276+0000 7f545de13b80 1 --2- 172.21.15.70:0/2268023856 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5605b4b4e840 0x5605b4b50c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.276+0000 7f545de13b80 4 client.16551 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:01:10.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.276+0000 7f545de13b80 1 -- 172.21.15.70:0/2268023856 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5605b4a234a0 con 0x5605b4b4e840 2024-06-04T05:01:10.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.276+0000 7f545131a700 1 --2- 172.21.15.70:0/2268023856 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5605b4b4e840 0x5605b4b50c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:10.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.277+0000 7f5434ff9700 10 client.16551 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:10.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.277+0000 7f545131a700 1 --2- 172.21.15.70:0/2268023856 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5605b4b4e840 0x5605b4b50c60 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51ebe97366a8f084 server_cookie=3d3151386cd225e6 in_seq=0 out_seq=0 2024-06-04T05:01:10.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.277+0000 7f5434ff9700 1 -- 172.21.15.70:0/2268023856 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5605b4a234a0 con 0x5605b4b4e840 2024-06-04T05:01:10.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.277+0000 7f5434ff9700 10 client.16551 handle_command_reply: tid=0 2024-06-04T05:01:10.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.277+0000 7f545de13b80 1 client.16551 shutdown 2024-06-04T05:01:10.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.277+0000 7f545de13b80 1 -- 172.21.15.70:0/2268023856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4aca260 msgr2=0x5605b4adbe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:10.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.277+0000 7f545de13b80 1 --2- 172.21.15.70:0/2268023856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4aca260 0x5605b4adbe80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f543c0df700 tx=0x7f543c0a8cf0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.277+0000 7f545de13b80 1 -- 172.21.15.70:0/2268023856 shutdown_connections 2024-06-04T05:01:10.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.277+0000 7f545de13b80 1 --2- 172.21.15.70:0/2268023856 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5605b4b4e840 0x5605b4b50c60 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.277+0000 7f545de13b80 1 --2- 172.21.15.70:0/2268023856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b4acf320 0x5605b4ad68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.277+0000 7f545de13b80 1 --2- 172.21.15.70:0/2268023856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5605b4ac98a0 0x5605b4adb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.277+0000 7f545de13b80 1 --2- 172.21.15.70:0/2268023856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605b4aca260 0x5605b4adbe80 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.277+0000 7f545de13b80 1 -- 172.21.15.70:0/2268023856 >> 172.21.15.70:0/2268023856 conn(0x5605b49f5240 msgr2=0x5605b4accce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:10.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.277+0000 7f5434ff9700 0 client.16551 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:10.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.277+0000 7f545de13b80 1 -- 172.21.15.70:0/2268023856 shutdown_connections 2024-06-04T05:01:10.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.277+0000 7f545de13b80 1 -- 172.21.15.70:0/2268023856 wait complete. 2024-06-04T05:01:10.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.277+0000 7f545de13b80 20 client.16551 trim_cache size 0 max 16384 2024-06-04T05:01:10.285 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:01:10.285 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:01:10.285 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:01:10.285 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:01:10.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.278+0000 7f545357e700 1 -- 172.21.15.70:0/1924637627 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f54380b29f0 msgr2=0x7f54380b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:10.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.278+0000 7f545357e700 1 --2- 172.21.15.70:0/1924637627 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f54380b29f0 0x7f54380b4eb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5448009550 tx=0x7f54480091b0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.278+0000 7f545357e700 1 -- 172.21.15.70:0/1924637627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c102e00 msgr2=0x7f544c19c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:10.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.278+0000 7f545357e700 1 --2- 172.21.15.70:0/1924637627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c102e00 0x7f544c19c2d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f543c00b530 tx=0x7f543c00b840 comp rx=0 tx=0).stop 2024-06-04T05:01:10.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.278+0000 7f545357e700 1 -- 172.21.15.70:0/1924637627 shutdown_connections 2024-06-04T05:01:10.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.278+0000 7f545357e700 1 --2- 172.21.15.70:0/1924637627 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f54380b29f0 0x7f54380b4eb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.278+0000 7f545357e700 1 --2- 172.21.15.70:0/1924637627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f544c108520 0x7f544c1a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.278+0000 7f545357e700 1 --2- 172.21.15.70:0/1924637627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f544c102440 0x7f544c19bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.278+0000 7f545357e700 1 --2- 172.21.15.70:0/1924637627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c102e00 0x7f544c19c2d0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:10.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.278+0000 7f545357e700 1 -- 172.21.15.70:0/1924637627 >> 172.21.15.70:0/1924637627 conn(0x7f544c0793f0 msgr2=0x7f544c06b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:10.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.278+0000 7f545357e700 1 -- 172.21.15.70:0/1924637627 shutdown_connections 2024-06-04T05:01:10.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:10.278+0000 7f545357e700 1 -- 172.21.15.70:0/1924637627 wait complete. 2024-06-04T05:01:10.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 3 sec latency 16160.276 ms 2024-06-04T05:01:11.055 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:01:11.056 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:01:11.297 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:01:11.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:11 smithi070 ceph-mon[32631]: pgmap v123: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 33 KiB/s wr, 27 op/s 2024-06-04T05:01:11.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:11 smithi112 ceph-mon[36453]: pgmap v123: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 33 KiB/s wr, 27 op/s 2024-06-04T05:01:11.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:11 smithi098 ceph-mon[35569]: pgmap v123: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 33 KiB/s wr, 27 op/s 2024-06-04T05:01:11.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 4 sec latency 17160.342 ms 2024-06-04T05:01:12.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 5 sec latency 18160.380 ms 2024-06-04T05:01:13.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.038+0000 7f3b29471700 1 -- 172.21.15.70:0/830892512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b24102440 msgr2=0x7f3b241028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:13.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.038+0000 7f3b29471700 1 --2- 172.21.15.70:0/830892512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b24102440 0x7f3b241028c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009990 tx=0x7f3b18009ca0 comp rx=0 tx=0).stop 2024-06-04T05:01:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.039+0000 7f3b29471700 1 -- 172.21.15.70:0/830892512 shutdown_connections 2024-06-04T05:01:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.039+0000 7f3b29471700 1 --2- 172.21.15.70:0/830892512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b24102e00 0x7f3b2410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.039+0000 7f3b29471700 1 --2- 172.21.15.70:0/830892512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b24108520 0x7f3b24108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.039+0000 7f3b29471700 1 --2- 172.21.15.70:0/830892512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b24102440 0x7f3b241028c0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.039+0000 7f3b29471700 1 -- 172.21.15.70:0/830892512 >> 172.21.15.70:0/830892512 conn(0x7f3b240793f0 msgr2=0x7f3b24079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.039+0000 7f3b29471700 1 -- 172.21.15.70:0/830892512 shutdown_connections 2024-06-04T05:01:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.039+0000 7f3b29471700 1 -- 172.21.15.70:0/830892512 wait complete. 2024-06-04T05:01:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.039+0000 7f3b29471700 1 Processor -- start 2024-06-04T05:01:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.039+0000 7f3b29471700 1 -- start start 2024-06-04T05:01:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b29471700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b24102440 0x7f3b2419e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b29471700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b24102e00 0x7f3b2419ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b29471700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b24108520 0x7f3b24198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b29471700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3b24112550 con 0x7f3b24102440 2024-06-04T05:01:13.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b29471700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b241123d0 con 0x7f3b24102e00 2024-06-04T05:01:13.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b29471700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3b241126d0 con 0x7f3b24108520 2024-06-04T05:01:13.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b22ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b24102440 0x7f3b2419e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:13.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b22ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b24102440 0x7f3b2419e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45334/0 (socket says 172.21.15.70:45334) 2024-06-04T05:01:13.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b22ffd700 1 -- 172.21.15.70:0/1515197000 learned_addr learned my addr 172.21.15.70:0/1515197000 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:13.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b237fe700 1 --2- 172.21.15.70:0/1515197000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b24108520 0x7f3b24198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:13.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b227fc700 1 --2- 172.21.15.70:0/1515197000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b24102e00 0x7f3b2419ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:13.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b22ffd700 1 -- 172.21.15.70:0/1515197000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b24108520 msgr2=0x7f3b24198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:13.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b22ffd700 1 --2- 172.21.15.70:0/1515197000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b24108520 0x7f3b24198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:13.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b22ffd700 1 -- 172.21.15.70:0/1515197000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b24102e00 msgr2=0x7f3b2419ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:13.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b22ffd700 1 --2- 172.21.15.70:0/1515197000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b24102e00 0x7f3b2419ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:13.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b22ffd700 1 -- 172.21.15.70:0/1515197000 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b24199090 con 0x7f3b24102440 2024-06-04T05:01:13.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b227fc700 1 --2- 172.21.15.70:0/1515197000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b24102e00 0x7f3b2419ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:01:13.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b22ffd700 1 --2- 172.21.15.70:0/1515197000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b24102440 0x7f3b2419e770 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400d950 tx=0x7f3b1400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:13.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.040+0000 7f3b0bfff700 1 -- 172.21.15.70:0/1515197000 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b14016070 con 0x7f3b24102440 2024-06-04T05:01:13.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.041+0000 7f3b0bfff700 1 -- 172.21.15.70:0/1515197000 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b14009dd0 con 0x7f3b24102440 2024-06-04T05:01:13.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.041+0000 7f3b29471700 1 -- 172.21.15.70:0/1515197000 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b24199380 con 0x7f3b24102440 2024-06-04T05:01:13.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.041+0000 7f3b0bfff700 1 -- 172.21.15.70:0/1515197000 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1400f3c0 con 0x7f3b24102440 2024-06-04T05:01:13.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.041+0000 7f3b29471700 1 -- 172.21.15.70:0/1515197000 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b241a5640 con 0x7f3b24102440 2024-06-04T05:01:13.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.042+0000 7f3b29471700 1 -- 172.21.15.70:0/1515197000 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b2406a230 con 0x7f3b24102440 2024-06-04T05:01:13.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.043+0000 7f3b0bfff700 1 -- 172.21.15.70:0/1515197000 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3b14019070 con 0x7f3b24102440 2024-06-04T05:01:13.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.043+0000 7f3b0bfff700 1 --2- 172.21.15.70:0/1515197000 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3b0c0b29f0 0x7f3b0c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:13.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.044+0000 7f3b227fc700 1 --2- 172.21.15.70:0/1515197000 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3b0c0b29f0 0x7f3b0c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:13.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.044+0000 7f3b0bfff700 1 -- 172.21.15.70:0/1515197000 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f3b140e2290 con 0x7f3b24102440 2024-06-04T05:01:13.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.044+0000 7f3b227fc700 1 --2- 172.21.15.70:0/1515197000 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3b0c0b29f0 0x7f3b0c0b4eb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800b5c0 tx=0x7f3b1801a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:13.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.048+0000 7f3b0bfff700 1 -- 172.21.15.70:0/1515197000 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3b140a9c70 con 0x7f3b24102440 2024-06-04T05:01:13.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.299+0000 7f3b29471700 1 -- 172.21.15.70:0/1515197000 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b2402e560 con 0x7f3b24102440 2024-06-04T05:01:13.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:13 smithi070 ceph-mon[32631]: pgmap v124: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 41 KiB/s wr, 35 op/s 2024-06-04T05:01:13.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.300+0000 7f3b0bfff700 1 -- 172.21.15.70:0/1515197000 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11932 (secure 0 0 0) 0x7f3b140a9570 con 0x7f3b24102440 2024-06-04T05:01:13.303 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:01:13.303 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:59:22.801688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:01:13.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.302+0000 7f3b29471700 1 -- 172.21.15.70:0/1515197000 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3b0c0b29f0 msgr2=0x7f3b0c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.302+0000 7f3b29471700 1 --2- 172.21.15.70:0/1515197000 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3b0c0b29f0 0x7f3b0c0b4eb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800b5c0 tx=0x7f3b1801a040 comp rx=0 tx=0).stop 2024-06-04T05:01:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.302+0000 7f3b29471700 1 -- 172.21.15.70:0/1515197000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b24102440 msgr2=0x7f3b2419e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.302+0000 7f3b29471700 1 --2- 172.21.15.70:0/1515197000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b24102440 0x7f3b2419e770 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400d950 tx=0x7f3b1400dc60 comp rx=0 tx=0).stop 2024-06-04T05:01:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.302+0000 7f3b29471700 1 -- 172.21.15.70:0/1515197000 shutdown_connections 2024-06-04T05:01:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.302+0000 7f3b29471700 1 --2- 172.21.15.70:0/1515197000 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3b0c0b29f0 0x7f3b0c0b4eb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.302+0000 7f3b29471700 1 --2- 172.21.15.70:0/1515197000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b24108520 0x7f3b24198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.302+0000 7f3b29471700 1 --2- 172.21.15.70:0/1515197000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b24102440 0x7f3b2419e770 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.302+0000 7f3b29471700 1 --2- 172.21.15.70:0/1515197000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b24102e00 0x7f3b2419ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.302+0000 7f3b29471700 1 -- 172.21.15.70:0/1515197000 >> 172.21.15.70:0/1515197000 conn(0x7f3b240793f0 msgr2=0x7f3b24100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.302+0000 7f3b29471700 1 -- 172.21.15.70:0/1515197000 shutdown_connections 2024-06-04T05:01:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:13.302+0000 7f3b29471700 1 -- 172.21.15.70:0/1515197000 wait complete. 2024-06-04T05:01:13.306 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-06-04T05:01:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:13 smithi112 ceph-mon[36453]: pgmap v124: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 41 KiB/s wr, 35 op/s 2024-06-04T05:01:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:13 smithi098 ceph-mon[35569]: pgmap v124: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 41 KiB/s wr, 35 op/s 2024-06-04T05:01:13.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 6 sec latency 19160.472 ms 2024-06-04T05:01:14.108 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:01:14.348 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:01:14.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:14 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1515197000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:01:14.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:14 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1515197000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:01:14.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:14 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1515197000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:01:14.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 7 sec latency 20160.509 ms 2024-06-04T05:01:15.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:15 smithi112 ceph-mon[36453]: pgmap v125: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-06-04T05:01:15.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:15 smithi070 ceph-mon[32631]: pgmap v125: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-06-04T05:01:15.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:15 smithi098 ceph-mon[35569]: pgmap v125: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-06-04T05:01:15.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 8 sec latency 21160.556 ms 2024-06-04T05:01:16.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.108+0000 7fc615d47700 1 -- 172.21.15.70:0/4024313742 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6101027f0 msgr2=0x7fc610106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:16.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.108+0000 7fc615d47700 1 --2- 172.21.15.70:0/4024313742 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6101027f0 0x7fc610106840 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc6040099c0 tx=0x7fc604009cd0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.108+0000 7fc615d47700 1 -- 172.21.15.70:0/4024313742 shutdown_connections 2024-06-04T05:01:16.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.108+0000 7fc615d47700 1 --2- 172.21.15.70:0/4024313742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc610106f20 0x7fc61010eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.108+0000 7fc615d47700 1 --2- 172.21.15.70:0/4024313742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc610101e40 0x7fc610102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.108+0000 7fc615d47700 1 --2- 172.21.15.70:0/4024313742 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6101027f0 0x7fc610106840 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.108+0000 7fc615d47700 1 -- 172.21.15.70:0/4024313742 >> 172.21.15.70:0/4024313742 conn(0x7fc610078a40 msgr2=0x7fc6100ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.108+0000 7fc615d47700 1 -- 172.21.15.70:0/4024313742 shutdown_connections 2024-06-04T05:01:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.108+0000 7fc615d47700 1 -- 172.21.15.70:0/4024313742 wait complete. 2024-06-04T05:01:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.109+0000 7fc615d47700 1 Processor -- start 2024-06-04T05:01:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.109+0000 7fc615d47700 1 -- start start 2024-06-04T05:01:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.109+0000 7fc615d47700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc610101e40 0x7fc61019f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.109+0000 7fc615d47700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6101027f0 0x7fc61019fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.109+0000 7fc615d47700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc610106f20 0x7fc610199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc615d47700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc6101125c0 con 0x7fc610101e40 2024-06-04T05:01:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc615d47700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc610112440 con 0x7fc6101027f0 2024-06-04T05:01:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc615d47700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc610112740 con 0x7fc610106f20 2024-06-04T05:01:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc60f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc610101e40 0x7fc61019f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc60f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc610101e40 0x7fc61019f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42714/0 (socket says 172.21.15.70:42714) 2024-06-04T05:01:16.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc60f7fe700 1 -- 172.21.15.70:0/1321536814 learned_addr learned my addr 172.21.15.70:0/1321536814 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:16.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc60ffff700 1 --2- 172.21.15.70:0/1321536814 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc610106f20 0x7fc610199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc60effd700 1 --2- 172.21.15.70:0/1321536814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6101027f0 0x7fc61019fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc60f7fe700 1 -- 172.21.15.70:0/1321536814 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc610106f20 msgr2=0x7fc610199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:16.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc60f7fe700 1 --2- 172.21.15.70:0/1321536814 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc610106f20 0x7fc610199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc60f7fe700 1 -- 172.21.15.70:0/1321536814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6101027f0 msgr2=0x7fc61019fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:16.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc60f7fe700 1 --2- 172.21.15.70:0/1321536814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6101027f0 0x7fc61019fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc60f7fe700 1 -- 172.21.15.70:0/1321536814 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61019a1d0 con 0x7fc610101e40 2024-06-04T05:01:16.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc60ffff700 1 --2- 172.21.15.70:0/1321536814 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc610106f20 0x7fc610199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:01:16.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc60f7fe700 1 --2- 172.21.15.70:0/1321536814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc610101e40 0x7fc61019f8b0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc60000b530 tx=0x7fc60000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:16.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc60cff9700 1 -- 172.21.15.70:0/1321536814 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc600015020 con 0x7fc610101e40 2024-06-04T05:01:16.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc60cff9700 1 -- 172.21.15.70:0/1321536814 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc600011cc0 con 0x7fc610101e40 2024-06-04T05:01:16.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc615d47700 1 -- 172.21.15.70:0/1321536814 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61019a4c0 con 0x7fc610101e40 2024-06-04T05:01:16.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc60cff9700 1 -- 172.21.15.70:0/1321536814 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc600010d30 con 0x7fc610101e40 2024-06-04T05:01:16.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.110+0000 7fc615d47700 1 -- 172.21.15.70:0/1321536814 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6101a6780 con 0x7fc610101e40 2024-06-04T05:01:16.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.113+0000 7fc60cff9700 1 -- 172.21.15.70:0/1321536814 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fc600018020 con 0x7fc610101e40 2024-06-04T05:01:16.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.113+0000 7fc60cff9700 1 --2- 172.21.15.70:0/1321536814 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc5f80b29f0 0x7fc5f80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.113+0000 7fc60effd700 1 --2- 172.21.15.70:0/1321536814 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc5f80b29f0 0x7fc5f80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.113+0000 7fc60cff9700 1 -- 172.21.15.70:0/1321536814 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc6000e1d50 con 0x7fc610101e40 2024-06-04T05:01:16.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.114+0000 7fc60effd700 1 --2- 172.21.15.70:0/1321536814 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc5f80b29f0 0x7fc5f80b4eb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc604006010 tx=0x7fc604009f10 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:16.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.114+0000 7fc6205dcb80 1 Processor -- start 2024-06-04T05:01:16.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.114+0000 7fc6205dcb80 1 -- start start 2024-06-04T05:01:16.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.114+0000 7fc6205dcb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b66c0 0x5646ff1b6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.114+0000 7fc6205dcb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b0cb0 0x5646ff1b1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.114+0000 7fc6205dcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b1670 0x5646ff1bd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.114+0000 7fc6205dcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5646ff10ad70 con 0x5646ff1b1670 2024-06-04T05:01:16.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.114+0000 7fc6205dcb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5646ff0fa910 con 0x5646ff1b66c0 2024-06-04T05:01:16.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.114+0000 7fc6205dcb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5646ff0fe950 con 0x5646ff1b0cb0 2024-06-04T05:01:16.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.114+0000 7fc60ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b1670 0x5646ff1bd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.114+0000 7fc60ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b1670 0x5646ff1bd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42716/0 (socket says 172.21.15.70:42716) 2024-06-04T05:01:16.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.114+0000 7fc60ffff700 1 -- 172.21.15.70:0/209237165 learned_addr learned my addr 172.21.15.70:0/209237165 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:16.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.115+0000 7fc60f7fe700 1 --2- 172.21.15.70:0/209237165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b66c0 0x5646ff1b6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.115+0000 7fc60effd700 1 --2- 172.21.15.70:0/209237165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b0cb0 0x5646ff1b1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.115+0000 7fc60ffff700 1 -- 172.21.15.70:0/209237165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b0cb0 msgr2=0x5646ff1b1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:16.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.115+0000 7fc60ffff700 1 --2- 172.21.15.70:0/209237165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b0cb0 0x5646ff1b1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.115+0000 7fc60ffff700 1 -- 172.21.15.70:0/209237165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b66c0 msgr2=0x5646ff1b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:16.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.115+0000 7fc60ffff700 1 --2- 172.21.15.70:0/209237165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b66c0 0x5646ff1b6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.115+0000 7fc60ffff700 1 -- 172.21.15.70:0/209237165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646ff0a3fb0 con 0x5646ff1b1670 2024-06-04T05:01:16.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.115+0000 7fc60f7fe700 1 --2- 172.21.15.70:0/209237165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b66c0 0x5646ff1b6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:01:16.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.115+0000 7fc60ffff700 1 --2- 172.21.15.70:0/209237165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b1670 0x5646ff1bd460 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fc60800c420 tx=0x7fc60800c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6109ddb41980df5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:16.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.115+0000 7fc5f5ffb700 1 -- 172.21.15.70:0/209237165 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60801a070 con 0x5646ff1b1670 2024-06-04T05:01:16.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.115+0000 7fc5f5ffb700 1 -- 172.21.15.70:0/209237165 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc60800cb80 con 0x5646ff1b1670 2024-06-04T05:01:16.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.115+0000 7fc5f5ffb700 1 -- 172.21.15.70:0/209237165 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60800a400 con 0x5646ff1b1670 2024-06-04T05:01:16.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.116+0000 7fc6205dcb80 1 -- 172.21.15.70:0/209237165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b1670 msgr2=0x5646ff1bd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:16.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.116+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/209237165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b1670 0x5646ff1bd460 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fc60800c420 tx=0x7fc60800c450 comp rx=0 tx=0).stop 2024-06-04T05:01:16.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.116+0000 7fc6205dcb80 1 -- 172.21.15.70:0/209237165 shutdown_connections 2024-06-04T05:01:16.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.116+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/209237165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b1670 0x5646ff1bd460 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.116+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/209237165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b66c0 0x5646ff1b6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.116+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/209237165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b0cb0 0x5646ff1b1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.116+0000 7fc6205dcb80 1 -- 172.21.15.70:0/209237165 >> 172.21.15.70:0/209237165 conn(0x5646ff0d8240 msgr2=0x5646ff1a73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:16.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.116+0000 7fc6205dcb80 1 -- 172.21.15.70:0/209237165 shutdown_connections 2024-06-04T05:01:16.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.116+0000 7fc6205dcb80 1 -- 172.21.15.70:0/209237165 wait complete. 2024-06-04T05:01:16.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.116+0000 7fc6205dcb80 1 -- start start 2024-06-04T05:01:16.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc6205dcb80 1 Processor -- start 2024-06-04T05:01:16.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc6205dcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b0cb0 0x5646ff1bd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc6205dcb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b1670 0x5646ff1ba3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc6205dcb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b66c0 0x5646ff1ba900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc6205dcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5646ff000fa0 con 0x5646ff1b0cb0 2024-06-04T05:01:16.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc6205dcb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5646ff062da0 con 0x5646ff1b1670 2024-06-04T05:01:16.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc6205dcb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5646ff1bdb50 con 0x5646ff1b66c0 2024-06-04T05:01:16.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc6205dcb80 10 client.0 _ll_register_callbacks cb 0x5646ff095950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:16.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc60f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b0cb0 0x5646ff1bd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc60f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b0cb0 0x5646ff1bd0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42730/0 (socket says 172.21.15.70:42730) 2024-06-04T05:01:16.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc60ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b66c0 0x5646ff1ba900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc60f7fe700 1 -- 172.21.15.70:0/1925274546 learned_addr learned my addr 172.21.15.70:0/1925274546 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:16.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc60effd700 1 --2- 172.21.15.70:0/1925274546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b1670 0x5646ff1ba3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc60f7fe700 1 -- 172.21.15.70:0/1925274546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b66c0 msgr2=0x5646ff1ba900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:16.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc60f7fe700 1 --2- 172.21.15.70:0/1925274546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b66c0 0x5646ff1ba900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc60f7fe700 1 -- 172.21.15.70:0/1925274546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b1670 msgr2=0x5646ff1ba3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:16.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc60f7fe700 1 --2- 172.21.15.70:0/1925274546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b1670 0x5646ff1ba3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.117+0000 7fc60f7fe700 1 -- 172.21.15.70:0/1925274546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646ff04a8d0 con 0x5646ff1b0cb0 2024-06-04T05:01:16.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.118+0000 7fc60f7fe700 1 --2- 172.21.15.70:0/1925274546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b0cb0 0x5646ff1bd0c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc6000df610 tx=0x7fc6000a8930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bc663e38c76a867 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:16.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.118+0000 7fc5eeffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:16.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.118+0000 7fc5eeffd700 1 -- 172.21.15.70:0/1925274546 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6000c3060 con 0x5646ff1b0cb0 2024-06-04T05:01:16.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.118+0000 7fc5eeffd700 1 -- 172.21.15.70:0/1925274546 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc6000a53d0 con 0x5646ff1b0cb0 2024-06-04T05:01:16.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.118+0000 7fc6205dcb80 1 -- 172.21.15.70:0/1925274546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc60800cb80 con 0x5646ff1b0cb0 2024-06-04T05:01:16.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.118+0000 7fc5eeffd700 1 -- 172.21.15.70:0/1925274546 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc6000a9840 con 0x5646ff1b0cb0 2024-06-04T05:01:16.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.118+0000 7fc6205dcb80 10 client.16569 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:16.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.118+0000 7fc6205dcb80 1 -- 172.21.15.70:0/1925274546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5646ff0b4760 con 0x5646ff1b0cb0 2024-06-04T05:01:16.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.118+0000 7fc5eeffd700 1 -- 172.21.15.70:0/1925274546 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc60800cb80 con 0x5646ff1b0cb0 2024-06-04T05:01:16.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.118+0000 7fc5eeffd700 1 -- 172.21.15.70:0/1925274546 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc6000a9430 con 0x5646ff1b0cb0 2024-06-04T05:01:16.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.118+0000 7fc6205dcb80 10 client.16569 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:16.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.118+0000 7fc6205dcb80 10 client.16569 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:01:16.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.118+0000 7fc6205dcb80 20 client.16569 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:16.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.118+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/1925274546 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5646ff1ca670 0x5646ff1cca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.118+0000 7fc6205dcb80 4 client.16569 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:01:16.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.118+0000 7fc6205dcb80 1 -- 172.21.15.70:0/1925274546 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646ff10ccc0 con 0x5646ff1ca670 2024-06-04T05:01:16.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.119+0000 7fc60effd700 1 --2- 172.21.15.70:0/1925274546 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5646ff1ca670 0x5646ff1cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.119+0000 7fc60effd700 1 --2- 172.21.15.70:0/1925274546 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5646ff1ca670 0x5646ff1cca60 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f642e58c6ad10eda server_cookie=e806f2259feeef7a in_seq=0 out_seq=0 2024-06-04T05:01:16.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.119+0000 7fc5eeffd700 10 client.16569 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:01:16.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.120+0000 7fc5eeffd700 1 -- 172.21.15.70:0/1925274546 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5646ff10ccc0 con 0x5646ff1ca670 2024-06-04T05:01:16.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.120+0000 7fc5eeffd700 10 client.16569 handle_command_reply: tid=0 2024-06-04T05:01:16.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.120+0000 7fc6205dcb80 1 client.16569 shutdown 2024-06-04T05:01:16.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.120+0000 7fc6205dcb80 1 -- 172.21.15.70:0/1925274546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b0cb0 msgr2=0x5646ff1bd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:16.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.120+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/1925274546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b0cb0 0x5646ff1bd0c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc6000df610 tx=0x7fc6000a8930 comp rx=0 tx=0).stop 2024-06-04T05:01:16.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.120+0000 7fc6205dcb80 1 -- 172.21.15.70:0/1925274546 shutdown_connections 2024-06-04T05:01:16.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.120+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/1925274546 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5646ff1ca670 0x5646ff1cca60 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.120+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/1925274546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b66c0 0x5646ff1ba900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.120+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/1925274546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b0cb0 0x5646ff1bd0c0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.120+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/1925274546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b1670 0x5646ff1ba3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.120+0000 7fc6205dcb80 1 -- 172.21.15.70:0/1925274546 >> 172.21.15.70:0/1925274546 conn(0x5646ff0d8240 msgr2=0x5646ff1a6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:16.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.120+0000 7fc5eeffd700 0 client.16569 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:01:16.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.120+0000 7fc6205dcb80 1 -- 172.21.15.70:0/1925274546 shutdown_connections 2024-06-04T05:01:16.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.120+0000 7fc6205dcb80 1 -- 172.21.15.70:0/1925274546 wait complete. 2024-06-04T05:01:16.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.120+0000 7fc6205dcb80 20 client.16569 trim_cache size 0 max 16384 2024-06-04T05:01:16.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.135+0000 7fc6205dcb80 1 Processor -- start 2024-06-04T05:01:16.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.135+0000 7fc6205dcb80 1 -- start start 2024-06-04T05:01:16.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.135+0000 7fc6205dcb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b0cb0 0x5646ff1b9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.135+0000 7fc6205dcb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b1670 0x5646ff1a7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.135+0000 7fc6205dcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b66c0 0x5646ff1bc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.135+0000 7fc6205dcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc6101abc50 con 0x5646ff1b66c0 2024-06-04T05:01:16.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.135+0000 7fc6205dcb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc6101ad7f0 con 0x5646ff1b1670 2024-06-04T05:01:16.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.135+0000 7fc6205dcb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc6101ac320 con 0x5646ff1b0cb0 2024-06-04T05:01:16.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.136+0000 7fc60ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b66c0 0x5646ff1bc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.136+0000 7fc60ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b66c0 0x5646ff1bc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42738/0 (socket says 172.21.15.70:42738) 2024-06-04T05:01:16.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.136+0000 7fc60ffff700 1 -- 172.21.15.70:0/714183871 learned_addr learned my addr 172.21.15.70:0/714183871 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:16.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.136+0000 7fc60f7fe700 1 --2- 172.21.15.70:0/714183871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b0cb0 0x5646ff1b9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.136+0000 7fc60effd700 1 --2- 172.21.15.70:0/714183871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b1670 0x5646ff1a7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.136+0000 7fc60ffff700 1 -- 172.21.15.70:0/714183871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b0cb0 msgr2=0x5646ff1b9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:16.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.136+0000 7fc60ffff700 1 --2- 172.21.15.70:0/714183871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b0cb0 0x5646ff1b9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.136+0000 7fc60ffff700 1 -- 172.21.15.70:0/714183871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b1670 msgr2=0x5646ff1a7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:16.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.136+0000 7fc60ffff700 1 --2- 172.21.15.70:0/714183871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b1670 0x5646ff1a7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.136+0000 7fc60ffff700 1 -- 172.21.15.70:0/714183871 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646ff1bcd00 con 0x5646ff1b66c0 2024-06-04T05:01:16.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.136+0000 7fc60ffff700 1 --2- 172.21.15.70:0/714183871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b66c0 0x5646ff1bc300 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc608018040 tx=0x7fc60800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8580e85c8e2cd4b9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:16.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.136+0000 7fc5f57fa700 1 -- 172.21.15.70:0/714183871 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc608024020 con 0x5646ff1b66c0 2024-06-04T05:01:16.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.136+0000 7fc5f57fa700 1 -- 172.21.15.70:0/714183871 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc60800cdf0 con 0x5646ff1b66c0 2024-06-04T05:01:16.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.136+0000 7fc5f57fa700 1 -- 172.21.15.70:0/714183871 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60801a040 con 0x5646ff1b66c0 2024-06-04T05:01:16.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.137+0000 7fc6205dcb80 1 -- 172.21.15.70:0/714183871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b66c0 msgr2=0x5646ff1bc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:16.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.137+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/714183871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b66c0 0x5646ff1bc300 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc608018040 tx=0x7fc60800cd80 comp rx=0 tx=0).stop 2024-06-04T05:01:16.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.137+0000 7fc6205dcb80 1 -- 172.21.15.70:0/714183871 shutdown_connections 2024-06-04T05:01:16.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.137+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/714183871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b66c0 0x5646ff1bc300 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.137+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/714183871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b0cb0 0x5646ff1b9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.137+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/714183871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b1670 0x5646ff1a7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.137+0000 7fc6205dcb80 1 -- 172.21.15.70:0/714183871 >> 172.21.15.70:0/714183871 conn(0x5646ff0d8240 msgr2=0x5646ff1bbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:16.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.137+0000 7fc6205dcb80 1 -- 172.21.15.70:0/714183871 shutdown_connections 2024-06-04T05:01:16.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.137+0000 7fc6205dcb80 1 -- 172.21.15.70:0/714183871 wait complete. 2024-06-04T05:01:16.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.137+0000 7fc6205dcb80 1 -- start start 2024-06-04T05:01:16.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc6205dcb80 1 Processor -- start 2024-06-04T05:01:16.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc6205dcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b0cb0 0x5646ff1c3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc6205dcb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b1670 0x5646ff1c3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc6205dcb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b66c0 0x5646ff1d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc6205dcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc6101ad0c0 con 0x5646ff1b0cb0 2024-06-04T05:01:16.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc6205dcb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc6101ac9f0 con 0x5646ff1b66c0 2024-06-04T05:01:16.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc6205dcb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc6101a9bf0 con 0x5646ff1b1670 2024-06-04T05:01:16.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc6205dcb80 10 client.0 _ll_register_callbacks cb 0x5646ff095950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:16.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc60f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b0cb0 0x5646ff1c3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc60f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b0cb0 0x5646ff1c3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42746/0 (socket says 172.21.15.70:42746) 2024-06-04T05:01:16.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc60f7fe700 1 -- 172.21.15.70:0/3525722555 learned_addr learned my addr 172.21.15.70:0/3525722555 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:16.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc60effd700 1 --2- 172.21.15.70:0/3525722555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b1670 0x5646ff1c3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc60ffff700 1 --2- 172.21.15.70:0/3525722555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b66c0 0x5646ff1d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc60f7fe700 1 -- 172.21.15.70:0/3525722555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b1670 msgr2=0x5646ff1c3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:16.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc60f7fe700 1 --2- 172.21.15.70:0/3525722555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b1670 0x5646ff1c3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc60f7fe700 1 -- 172.21.15.70:0/3525722555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b66c0 msgr2=0x5646ff1d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:16.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc60f7fe700 1 --2- 172.21.15.70:0/3525722555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b66c0 0x5646ff1d1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc60f7fe700 1 -- 172.21.15.70:0/3525722555 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646ff1d1ec0 con 0x5646ff1b0cb0 2024-06-04T05:01:16.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc60effd700 1 --2- 172.21.15.70:0/3525722555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b1670 0x5646ff1c3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:01:16.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc60f7fe700 1 --2- 172.21.15.70:0/3525722555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b0cb0 0x5646ff1c3030 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fc60000b500 tx=0x7fc600011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d8d10a8da4f0478 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:16.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.138+0000 7fc5eeffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:16.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.139+0000 7fc5eeffd700 1 -- 172.21.15.70:0/3525722555 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6000ad4f0 con 0x5646ff1b0cb0 2024-06-04T05:01:16.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.139+0000 7fc5eeffd700 1 -- 172.21.15.70:0/3525722555 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc6000a93a0 con 0x5646ff1b0cb0 2024-06-04T05:01:16.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.139+0000 7fc6205dcb80 1 -- 172.21.15.70:0/3525722555 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc60800cdf0 con 0x5646ff1b0cb0 2024-06-04T05:01:16.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.139+0000 7fc5eeffd700 1 -- 172.21.15.70:0/3525722555 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc6000b1fa0 con 0x5646ff1b0cb0 2024-06-04T05:01:16.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.139+0000 7fc6205dcb80 10 client.16575 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:16.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.139+0000 7fc6205dcb80 1 -- 172.21.15.70:0/3525722555 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5646ff234cb0 con 0x5646ff1b0cb0 2024-06-04T05:01:16.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.139+0000 7fc5eeffd700 1 -- 172.21.15.70:0/3525722555 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc60800cdf0 con 0x5646ff1b0cb0 2024-06-04T05:01:16.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.139+0000 7fc5eeffd700 1 -- 172.21.15.70:0/3525722555 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc6000a84c0 con 0x5646ff1b0cb0 2024-06-04T05:01:16.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.139+0000 7fc6205dcb80 10 client.16575 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:16.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.139+0000 7fc6205dcb80 10 client.16575 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:01:16.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.139+0000 7fc6205dcb80 20 client.16575 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:16.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.139+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/3525722555 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5646ff235360 0x5646ff2377a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:16.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.139+0000 7fc6205dcb80 4 client.16575 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:01:16.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.139+0000 7fc6205dcb80 1 -- 172.21.15.70:0/3525722555 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5646ff1064a0 con 0x5646ff235360 2024-06-04T05:01:16.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.140+0000 7fc60effd700 1 --2- 172.21.15.70:0/3525722555 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5646ff235360 0x5646ff2377a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:16.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.140+0000 7fc60effd700 1 --2- 172.21.15.70:0/3525722555 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5646ff235360 0x5646ff2377a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=86643d1c2441537d server_cookie=c7c86cb54f967c9 in_seq=0 out_seq=0 2024-06-04T05:01:16.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.140+0000 7fc5eeffd700 10 client.16575 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:01:16.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.140+0000 7fc5eeffd700 1 -- 172.21.15.70:0/3525722555 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5646ff1064a0 con 0x5646ff235360 2024-06-04T05:01:16.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.140+0000 7fc5eeffd700 10 client.16575 handle_command_reply: tid=0 2024-06-04T05:01:16.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.140+0000 7fc6205dcb80 1 client.16575 shutdown 2024-06-04T05:01:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.140+0000 7fc6205dcb80 1 -- 172.21.15.70:0/3525722555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b0cb0 msgr2=0x5646ff1c3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.140+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/3525722555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b0cb0 0x5646ff1c3030 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fc60000b500 tx=0x7fc600011000 comp rx=0 tx=0).stop 2024-06-04T05:01:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.140+0000 7fc6205dcb80 1 -- 172.21.15.70:0/3525722555 shutdown_connections 2024-06-04T05:01:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.140+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/3525722555 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5646ff235360 0x5646ff2377a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.140+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/3525722555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646ff1b66c0 0x5646ff1d1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.140+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/3525722555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646ff1b0cb0 0x5646ff1c3030 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.140+0000 7fc5eeffd700 0 client.16575 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:01:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.140+0000 7fc6205dcb80 1 --2- 172.21.15.70:0/3525722555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646ff1b1670 0x5646ff1c3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.140+0000 7fc6205dcb80 1 -- 172.21.15.70:0/3525722555 >> 172.21.15.70:0/3525722555 conn(0x5646ff0d8240 msgr2=0x5646ff1ac120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.141+0000 7fc6205dcb80 1 -- 172.21.15.70:0/3525722555 shutdown_connections 2024-06-04T05:01:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.141+0000 7fc6205dcb80 1 -- 172.21.15.70:0/3525722555 wait complete. 2024-06-04T05:01:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.141+0000 7fc6205dcb80 20 client.16575 trim_cache size 0 max 16384 2024-06-04T05:01:16.149 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:01:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.141+0000 7fc615d47700 1 -- 172.21.15.70:0/1321536814 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc5f80b29f0 msgr2=0x7fc5f80b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.141+0000 7fc615d47700 1 --2- 172.21.15.70:0/1321536814 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc5f80b29f0 0x7fc5f80b4eb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc604006010 tx=0x7fc604009f10 comp rx=0 tx=0).stop 2024-06-04T05:01:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.141+0000 7fc615d47700 1 -- 172.21.15.70:0/1321536814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc610101e40 msgr2=0x7fc61019f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.141+0000 7fc615d47700 1 --2- 172.21.15.70:0/1321536814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc610101e40 0x7fc61019f8b0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc60000b530 tx=0x7fc60000b8f0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.141+0000 7fc615d47700 1 -- 172.21.15.70:0/1321536814 shutdown_connections 2024-06-04T05:01:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.141+0000 7fc615d47700 1 --2- 172.21.15.70:0/1321536814 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc5f80b29f0 0x7fc5f80b4eb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.141+0000 7fc615d47700 1 --2- 172.21.15.70:0/1321536814 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc610106f20 0x7fc610199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.141+0000 7fc615d47700 1 --2- 172.21.15.70:0/1321536814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc610101e40 0x7fc61019f8b0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.141+0000 7fc615d47700 1 --2- 172.21.15.70:0/1321536814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6101027f0 0x7fc61019fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.141+0000 7fc615d47700 1 -- 172.21.15.70:0/1321536814 >> 172.21.15.70:0/1321536814 conn(0x7fc610078a40 msgr2=0x7fc61010f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.141+0000 7fc615d47700 1 -- 172.21.15.70:0/1321536814 shutdown_connections 2024-06-04T05:01:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:16.142+0000 7fc615d47700 1 -- 172.21.15.70:0/1321536814 wait complete. 2024-06-04T05:01:16.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 9 sec latency 22160.636 ms 2024-06-04T05:01:16.996 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:01:17.240 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:01:17.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:17 smithi070 ceph-mon[32631]: pgmap v126: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 18 op/s 2024-06-04T05:01:17.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:17 smithi112 ceph-mon[36453]: pgmap v126: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 18 op/s 2024-06-04T05:01:17.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:17 smithi098 ceph-mon[35569]: pgmap v126: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 18 op/s 2024-06-04T05:01:17.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 10 sec latency 23160.660 ms 2024-06-04T05:01:18.044 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:01:18.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 11 sec latency 24160.686 ms 2024-06-04T05:01:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.052+0000 7fd61572a700 1 -- 172.21.15.70:0/1745165079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd610102470 msgr2=0x7fd6101028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.052+0000 7fd61572a700 1 --2- 172.21.15.70:0/1745165079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd610102470 0x7fd6101028f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd604009990 tx=0x7fd604009ca0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.052+0000 7fd61572a700 1 -- 172.21.15.70:0/1745165079 shutdown_connections 2024-06-04T05:01:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.052+0000 7fd61572a700 1 --2- 172.21.15.70:0/1745165079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd610102e30 0x7fd61010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.052+0000 7fd61572a700 1 --2- 172.21.15.70:0/1745165079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6101084b0 0x7fd610108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.052+0000 7fd61572a700 1 --2- 172.21.15.70:0/1745165079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd610102470 0x7fd6101028f0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.052+0000 7fd61572a700 1 -- 172.21.15.70:0/1745165079 >> 172.21.15.70:0/1745165079 conn(0x7fd610079410 msgr2=0x7fd610079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.053+0000 7fd61572a700 1 -- 172.21.15.70:0/1745165079 shutdown_connections 2024-06-04T05:01:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.053+0000 7fd61572a700 1 -- 172.21.15.70:0/1745165079 wait complete. 2024-06-04T05:01:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.053+0000 7fd61572a700 1 Processor -- start 2024-06-04T05:01:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.053+0000 7fd61572a700 1 -- start start 2024-06-04T05:01:19.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.053+0000 7fd61572a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd610102470 0x7fd61019f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.053+0000 7fd61572a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd610102e30 0x7fd61019fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.053+0000 7fd61572a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6101084b0 0x7fd610199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.053+0000 7fd61572a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6101124f0 con 0x7fd610102470 2024-06-04T05:01:19.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd61572a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd610112370 con 0x7fd610102e30 2024-06-04T05:01:19.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd61572a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd610112670 con 0x7fd6101084b0 2024-06-04T05:01:19.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd60effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd610102470 0x7fd61019f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:19.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd60effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd610102470 0x7fd61019f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42768/0 (socket says 172.21.15.70:42768) 2024-06-04T05:01:19.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd60effd700 1 -- 172.21.15.70:0/622995695 learned_addr learned my addr 172.21.15.70:0/622995695 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:19.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd60f7fe700 1 --2- 172.21.15.70:0/622995695 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6101084b0 0x7fd610199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:19.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd60e7fc700 1 --2- 172.21.15.70:0/622995695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd610102e30 0x7fd61019fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:19.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd60effd700 1 -- 172.21.15.70:0/622995695 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6101084b0 msgr2=0x7fd610199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:19.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd60effd700 1 --2- 172.21.15.70:0/622995695 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6101084b0 0x7fd610199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd60effd700 1 -- 172.21.15.70:0/622995695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd610102e30 msgr2=0x7fd61019fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:19.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd60effd700 1 --2- 172.21.15.70:0/622995695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd610102e30 0x7fd61019fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd60effd700 1 -- 172.21.15.70:0/622995695 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd61019a0d0 con 0x7fd610102470 2024-06-04T05:01:19.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd60e7fc700 1 --2- 172.21.15.70:0/622995695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd610102e30 0x7fd61019fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:01:19.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd60effd700 1 --2- 172.21.15.70:0/622995695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd610102470 0x7fd61019f7b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fd60000b530 tx=0x7fd60000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:19.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd5f7fff700 1 -- 172.21.15.70:0/622995695 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd600015020 con 0x7fd610102470 2024-06-04T05:01:19.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd5f7fff700 1 -- 172.21.15.70:0/622995695 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd600011cc0 con 0x7fd610102470 2024-06-04T05:01:19.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd61572a700 1 -- 172.21.15.70:0/622995695 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd61019a3c0 con 0x7fd610102470 2024-06-04T05:01:19.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd5f7fff700 1 -- 172.21.15.70:0/622995695 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd600010dc0 con 0x7fd610102470 2024-06-04T05:01:19.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.054+0000 7fd61572a700 1 -- 172.21.15.70:0/622995695 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6101a6680 con 0x7fd610102470 2024-06-04T05:01:19.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.057+0000 7fd5f7fff700 1 -- 172.21.15.70:0/622995695 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fd600018020 con 0x7fd610102470 2024-06-04T05:01:19.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.057+0000 7fd5f7fff700 1 --2- 172.21.15.70:0/622995695 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd5f80b29f0 0x7fd5f80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.057+0000 7fd60e7fc700 1 --2- 172.21.15.70:0/622995695 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd5f80b29f0 0x7fd5f80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:19.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.057+0000 7fd5f7fff700 1 -- 172.21.15.70:0/622995695 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fd6000e1d80 con 0x7fd610102470 2024-06-04T05:01:19.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.058+0000 7fd60e7fc700 1 --2- 172.21.15.70:0/622995695 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd5f80b29f0 0x7fd5f80b4eb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd604006010 tx=0x7fd60401a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:19.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.058+0000 7fd61ffbfb80 1 Processor -- start 2024-06-04T05:01:19.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.058+0000 7fd61ffbfb80 1 -- start start 2024-06-04T05:01:19.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.058+0000 7fd61ffbfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12abc70 0x5560e12ac050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.058+0000 7fd61ffbfb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12ac620 0x5560e12afd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.058+0000 7fd61ffbfb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12b0370 0x5560e12b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.058+0000 7fd61ffbfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5560e1206c90 con 0x5560e12abc70 2024-06-04T05:01:19.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.058+0000 7fd61ffbfb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5560e11f6830 con 0x5560e12ac620 2024-06-04T05:01:19.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.058+0000 7fd61ffbfb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5560e11fa870 con 0x5560e12b0370 2024-06-04T05:01:19.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.059+0000 7fd60effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12abc70 0x5560e12ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:19.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.059+0000 7fd60effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12abc70 0x5560e12ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42780/0 (socket says 172.21.15.70:42780) 2024-06-04T05:01:19.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.059+0000 7fd60effd700 1 -- 172.21.15.70:0/3004119433 learned_addr learned my addr 172.21.15.70:0/3004119433 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:19.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.059+0000 7fd60f7fe700 1 --2- 172.21.15.70:0/3004119433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12b0370 0x5560e12b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.059+0000 7fd60e7fc700 1 --2- 172.21.15.70:0/3004119433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12ac620 0x5560e12afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.059+0000 7fd60effd700 1 -- 172.21.15.70:0/3004119433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12b0370 msgr2=0x5560e12b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.059+0000 7fd60effd700 1 --2- 172.21.15.70:0/3004119433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12b0370 0x5560e12b07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.059+0000 7fd60effd700 1 -- 172.21.15.70:0/3004119433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12ac620 msgr2=0x5560e12afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.059+0000 7fd60effd700 1 --2- 172.21.15.70:0/3004119433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12ac620 0x5560e12afd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.059+0000 7fd60effd700 1 -- 172.21.15.70:0/3004119433 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560e11468d0 con 0x5560e12abc70 2024-06-04T05:01:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.059+0000 7fd60e7fc700 1 --2- 172.21.15.70:0/3004119433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12ac620 0x5560e12afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:01:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.059+0000 7fd60effd700 1 --2- 172.21.15.70:0/3004119433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12abc70 0x5560e12ac050 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd6000a8e70 tx=0x7fd6000a8960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd85a90aae325b4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.059+0000 7fd5f57fa700 1 -- 172.21.15.70:0/3004119433 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6000c3070 con 0x5560e12abc70 2024-06-04T05:01:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.059+0000 7fd5f57fa700 1 -- 172.21.15.70:0/3004119433 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd6000a9170 con 0x5560e12abc70 2024-06-04T05:01:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.059+0000 7fd5f57fa700 1 -- 172.21.15.70:0/3004119433 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6000a95f0 con 0x5560e12abc70 2024-06-04T05:01:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.060+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/3004119433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12abc70 msgr2=0x5560e12ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.060+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/3004119433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12abc70 0x5560e12ac050 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd6000a8e70 tx=0x7fd6000a8960 comp rx=0 tx=0).stop 2024-06-04T05:01:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.060+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/3004119433 shutdown_connections 2024-06-04T05:01:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.060+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/3004119433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12b0370 0x5560e12b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.060+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/3004119433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12abc70 0x5560e12ac050 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.060+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/3004119433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12ac620 0x5560e12afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.060+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/3004119433 >> 172.21.15.70:0/3004119433 conn(0x5560e11d5080 msgr2=0x5560e12a1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.060+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/3004119433 shutdown_connections 2024-06-04T05:01:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.060+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/3004119433 wait complete. 2024-06-04T05:01:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.060+0000 7fd61ffbfb80 1 -- start start 2024-06-04T05:01:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.060+0000 7fd61ffbfb80 1 Processor -- start 2024-06-04T05:01:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd61ffbfb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12abc70 0x5560e12b78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd61ffbfb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12ac620 0x5560e12b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd61ffbfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12b0370 0x5560e12b5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd61ffbfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5560e10fcfa0 con 0x5560e12b0370 2024-06-04T05:01:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd61ffbfb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5560e11987c0 con 0x5560e12ac620 2024-06-04T05:01:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd61ffbfb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5560e12b8530 con 0x5560e12abc70 2024-06-04T05:01:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd61ffbfb80 10 client.0 _ll_register_callbacks cb 0x5560e1191920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd60f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12b0370 0x5560e12b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd60effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12abc70 0x5560e12b78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd60effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12abc70 0x5560e12b78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:60270/0 (socket says 172.21.15.70:60270) 2024-06-04T05:01:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd60f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12b0370 0x5560e12b5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42790/0 (socket says 172.21.15.70:42790) 2024-06-04T05:01:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd60effd700 1 -- 172.21.15.70:0/3686973588 learned_addr learned my addr 172.21.15.70:0/3686973588 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd60e7fc700 1 --2- 172.21.15.70:0/3686973588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12ac620 0x5560e12b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd60f7fe700 1 -- 172.21.15.70:0/3686973588 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12abc70 msgr2=0x5560e12b78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd60f7fe700 1 --2- 172.21.15.70:0/3686973588 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12abc70 0x5560e12b78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd60f7fe700 1 -- 172.21.15.70:0/3686973588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12ac620 msgr2=0x5560e12b4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd60f7fe700 1 --2- 172.21.15.70:0/3686973588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12ac620 0x5560e12b4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd60f7fe700 1 -- 172.21.15.70:0/3686973588 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560e1195690 con 0x5560e12b0370 2024-06-04T05:01:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd60f7fe700 1 --2- 172.21.15.70:0/3686973588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12b0370 0x5560e12b5100 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd608011cc0 tx=0x7fd608007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba30c56cc722217 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd5f2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd5f2ff5700 1 -- 172.21.15.70:0/3686973588 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60801b030 con 0x5560e12b0370 2024-06-04T05:01:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd5f2ff5700 1 -- 172.21.15.70:0/3686973588 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fd608010040 con 0x5560e12b0370 2024-06-04T05:01:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.061+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/3686973588 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6000a9170 con 0x5560e12b0370 2024-06-04T05:01:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.062+0000 7fd5f2ff5700 1 -- 172.21.15.70:0/3686973588 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd608024050 con 0x5560e12b0370 2024-06-04T05:01:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.062+0000 7fd61ffbfb80 10 client.16587 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.062+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/3686973588 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5560e12c2d30 con 0x5560e12b0370 2024-06-04T05:01:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.062+0000 7fd5f2ff5700 1 -- 172.21.15.70:0/3686973588 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd6000a9170 con 0x5560e12b0370 2024-06-04T05:01:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.062+0000 7fd5f2ff5700 1 -- 172.21.15.70:0/3686973588 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd60800fc60 con 0x5560e12b0370 2024-06-04T05:01:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.062+0000 7fd61ffbfb80 10 client.16587 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.062+0000 7fd61ffbfb80 10 client.16587 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:01:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.062+0000 7fd61ffbfb80 20 client.16587 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:19.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.062+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/3686973588 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5560e12c5370 0x5560e12c7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.062+0000 7fd61ffbfb80 4 client.16587 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:01:19.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.062+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/3686973588 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560e12739e0 con 0x5560e12c5370 2024-06-04T05:01:19.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.062+0000 7fd60effd700 1 --2- 172.21.15.70:0/3686973588 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5560e12c5370 0x5560e12c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:19.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.063+0000 7fd5f2ff5700 10 client.16587 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:01:19.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.063+0000 7fd60effd700 1 --2- 172.21.15.70:0/3686973588 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5560e12c5370 0x5560e12c7760 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6fca99c0fd3fefb4 server_cookie=85d2d55912208b8a in_seq=0 out_seq=0 2024-06-04T05:01:19.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.063+0000 7fd5f2ff5700 1 -- 172.21.15.70:0/3686973588 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5560e12739e0 con 0x5560e12c5370 2024-06-04T05:01:19.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.063+0000 7fd5f2ff5700 10 client.16587 handle_command_reply: tid=0 2024-06-04T05:01:19.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.063+0000 7fd61ffbfb80 1 client.16587 shutdown 2024-06-04T05:01:19.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.064+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/3686973588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12b0370 msgr2=0x5560e12b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:19.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.064+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/3686973588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12b0370 0x5560e12b5100 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd608011cc0 tx=0x7fd608007e50 comp rx=0 tx=0).stop 2024-06-04T05:01:19.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.064+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/3686973588 shutdown_connections 2024-06-04T05:01:19.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.064+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/3686973588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12b0370 0x5560e12b5100 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.064+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/3686973588 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5560e12c5370 0x5560e12c7760 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.064+0000 7fd5f2ff5700 0 client.16587 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:01:19.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.064+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/3686973588 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12abc70 0x5560e12b78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.064+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/3686973588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12ac620 0x5560e12b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.064+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/3686973588 >> 172.21.15.70:0/3686973588 conn(0x5560e11d5080 msgr2=0x5560e12a1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:19.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.064+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/3686973588 shutdown_connections 2024-06-04T05:01:19.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.064+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/3686973588 wait complete. 2024-06-04T05:01:19.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.064+0000 7fd61ffbfb80 20 client.16587 trim_cache size 0 max 16384 2024-06-04T05:01:19.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.079+0000 7fd61ffbfb80 1 Processor -- start 2024-06-04T05:01:19.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.079+0000 7fd61ffbfb80 1 -- start start 2024-06-04T05:01:19.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.079+0000 7fd61ffbfb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12abc70 0x5560e12c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.079+0000 7fd61ffbfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12ac620 0x5560e12af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.079+0000 7fd61ffbfb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12b0370 0x5560e12a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.079+0000 7fd61ffbfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6101abc00 con 0x5560e12ac620 2024-06-04T05:01:19.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.079+0000 7fd61ffbfb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd6101ad7a0 con 0x5560e12b0370 2024-06-04T05:01:19.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.079+0000 7fd61ffbfb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd6101ac2d0 con 0x5560e12abc70 2024-06-04T05:01:19.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.079+0000 7fd60e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12ac620 0x5560e12af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:19.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.079+0000 7fd60e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12ac620 0x5560e12af3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42794/0 (socket says 172.21.15.70:42794) 2024-06-04T05:01:19.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.079+0000 7fd60e7fc700 1 -- 172.21.15.70:0/1067699969 learned_addr learned my addr 172.21.15.70:0/1067699969 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:19.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.079+0000 7fd60e7fc700 1 -- 172.21.15.70:0/1067699969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12abc70 msgr2=0x5560e12c15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:19.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.079+0000 7fd60effd700 1 --2- 172.21.15.70:0/1067699969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12abc70 0x5560e12c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:19.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.080+0000 7fd60e7fc700 1 --2- 172.21.15.70:0/1067699969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12abc70 0x5560e12c15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.080+0000 7fd60f7fe700 1 --2- 172.21.15.70:0/1067699969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12b0370 0x5560e12a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:19.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.080+0000 7fd60e7fc700 1 -- 172.21.15.70:0/1067699969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12b0370 msgr2=0x5560e12a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:19.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.080+0000 7fd60e7fc700 1 --2- 172.21.15.70:0/1067699969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12b0370 0x5560e12a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.080+0000 7fd60e7fc700 1 -- 172.21.15.70:0/1067699969 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560e12b4990 con 0x5560e12ac620 2024-06-04T05:01:19.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.080+0000 7fd60f7fe700 1 --2- 172.21.15.70:0/1067699969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12b0370 0x5560e12a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:01:19.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.080+0000 7fd60effd700 1 --2- 172.21.15.70:0/1067699969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12abc70 0x5560e12c15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:01:19.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.080+0000 7fd60e7fc700 1 --2- 172.21.15.70:0/1067699969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12ac620 0x5560e12af3d0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd604000c00 tx=0x7fd60401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b53027cf844d7786 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:19.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.080+0000 7fd5f4ff9700 1 -- 172.21.15.70:0/1067699969 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60400fb80 con 0x5560e12ac620 2024-06-04T05:01:19.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.080+0000 7fd5f4ff9700 1 -- 172.21.15.70:0/1067699969 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd604033070 con 0x5560e12ac620 2024-06-04T05:01:19.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.080+0000 7fd5f4ff9700 1 -- 172.21.15.70:0/1067699969 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd604023410 con 0x5560e12ac620 2024-06-04T05:01:19.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/1067699969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12ac620 msgr2=0x5560e12af3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:19.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/1067699969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12ac620 0x5560e12af3d0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd604000c00 tx=0x7fd60401e000 comp rx=0 tx=0).stop 2024-06-04T05:01:19.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/1067699969 shutdown_connections 2024-06-04T05:01:19.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/1067699969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12b0370 0x5560e12a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/1067699969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12abc70 0x5560e12c15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/1067699969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12ac620 0x5560e12af3d0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/1067699969 >> 172.21.15.70:0/1067699969 conn(0x5560e11d5080 msgr2=0x5560e12aecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:19.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/1067699969 shutdown_connections 2024-06-04T05:01:19.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/1067699969 wait complete. 2024-06-04T05:01:19.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 1 -- start start 2024-06-04T05:01:19.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 1 Processor -- start 2024-06-04T05:01:19.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12abc70 0x5560e12bdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12ac620 0x5560e12bb060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12b0370 0x5560e12bb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6101ad070 con 0x5560e12abc70 2024-06-04T05:01:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd6101ac9a0 con 0x5560e12b0370 2024-06-04T05:01:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd6101a9ba0 con 0x5560e12ac620 2024-06-04T05:01:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd60effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12abc70 0x5560e12bdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.081+0000 7fd61ffbfb80 10 client.0 _ll_register_callbacks cb 0x5560e1191920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd60effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12abc70 0x5560e12bdcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42798/0 (socket says 172.21.15.70:42798) 2024-06-04T05:01:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd60effd700 1 -- 172.21.15.70:0/1658315745 learned_addr learned my addr 172.21.15.70:0/1658315745 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd60e7fc700 1 --2- 172.21.15.70:0/1658315745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12ac620 0x5560e12bb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd60effd700 1 -- 172.21.15.70:0/1658315745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12ac620 msgr2=0x5560e12bb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd60effd700 1 --2- 172.21.15.70:0/1658315745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12ac620 0x5560e12bb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd60effd700 1 -- 172.21.15.70:0/1658315745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12b0370 msgr2=0x5560e12bb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd60effd700 1 --2- 172.21.15.70:0/1658315745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12b0370 0x5560e12bb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd60effd700 1 -- 172.21.15.70:0/1658315745 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560e12bbbe0 con 0x5560e12abc70 2024-06-04T05:01:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd60e7fc700 1 --2- 172.21.15.70:0/1658315745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12ac620 0x5560e12bb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:01:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd60effd700 1 --2- 172.21.15.70:0/1658315745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12abc70 0x5560e12bdcc0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fd6000a51f0 tx=0x7fd600000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3217f786bfebe0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:19.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd5f2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:19.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd5f2ff5700 1 -- 172.21.15.70:0/1658315745 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60006c030 con 0x5560e12abc70 2024-06-04T05:01:19.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd5f2ff5700 1 -- 172.21.15.70:0/1658315745 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fd6000a5760 con 0x5560e12abc70 2024-06-04T05:01:19.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/1658315745 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd604033070 con 0x5560e12abc70 2024-06-04T05:01:19.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd5f2ff5700 1 -- 172.21.15.70:0/1658315745 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd6000c3040 con 0x5560e12abc70 2024-06-04T05:01:19.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd61ffbfb80 10 client.16593 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:19.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/1658315745 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5560e132fd10 con 0x5560e12abc70 2024-06-04T05:01:19.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd5f2ff5700 1 -- 172.21.15.70:0/1658315745 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd604033070 con 0x5560e12abc70 2024-06-04T05:01:19.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.082+0000 7fd5f2ff5700 1 -- 172.21.15.70:0/1658315745 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd6000b1b40 con 0x5560e12abc70 2024-06-04T05:01:19.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd61ffbfb80 10 client.16593 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:19.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd61ffbfb80 10 client.16593 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:01:19.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd61ffbfb80 20 client.16593 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:19.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/1658315745 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5560e12c5370 0x5560e1331fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd61ffbfb80 4 client.16593 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:01:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/1658315745 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5560e1208be0 con 0x5560e12c5370 2024-06-04T05:01:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd60e7fc700 1 --2- 172.21.15.70:0/1658315745 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5560e12c5370 0x5560e1331fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd5f2ff5700 10 client.16593 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:01:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd60e7fc700 1 --2- 172.21.15.70:0/1658315745 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5560e12c5370 0x5560e1331fd0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=41ac79a9be1e3175 server_cookie=9b95f45eb567ab99 in_seq=0 out_seq=0 2024-06-04T05:01:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd5f2ff5700 1 -- 172.21.15.70:0/1658315745 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5560e1208be0 con 0x5560e12c5370 2024-06-04T05:01:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd5f2ff5700 10 client.16593 handle_command_reply: tid=0 2024-06-04T05:01:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd61ffbfb80 1 client.16593 shutdown 2024-06-04T05:01:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/1658315745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12abc70 msgr2=0x5560e12bdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/1658315745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12abc70 0x5560e12bdcc0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fd6000a51f0 tx=0x7fd600000bc0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/1658315745 shutdown_connections 2024-06-04T05:01:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/1658315745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560e12b0370 0x5560e12bb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/1658315745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5560e12abc70 0x5560e12bdcc0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/1658315745 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5560e12c5370 0x5560e1331fd0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.083+0000 7fd61ffbfb80 1 --2- 172.21.15.70:0/1658315745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5560e12ac620 0x5560e12bb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.084+0000 7fd5f2ff5700 0 client.16593 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:01:19.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.084+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/1658315745 >> 172.21.15.70:0/1658315745 conn(0x5560e11d5080 msgr2=0x5560e12af550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:19.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.084+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/1658315745 shutdown_connections 2024-06-04T05:01:19.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.084+0000 7fd61ffbfb80 1 -- 172.21.15.70:0/1658315745 wait complete. 2024-06-04T05:01:19.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.084+0000 7fd61ffbfb80 20 client.16593 trim_cache size 0 max 16384 2024-06-04T05:01:19.092 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:01:19.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.084+0000 7fd61572a700 1 -- 172.21.15.70:0/622995695 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd5f80b29f0 msgr2=0x7fd5f80b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:19.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.084+0000 7fd61572a700 1 --2- 172.21.15.70:0/622995695 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd5f80b29f0 0x7fd5f80b4eb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd604006010 tx=0x7fd60401a040 comp rx=0 tx=0).stop 2024-06-04T05:01:19.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.084+0000 7fd61572a700 1 -- 172.21.15.70:0/622995695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd610102470 msgr2=0x7fd61019f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:19.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.084+0000 7fd61572a700 1 --2- 172.21.15.70:0/622995695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd610102470 0x7fd61019f7b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fd60000b530 tx=0x7fd60000b840 comp rx=0 tx=0).stop 2024-06-04T05:01:19.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.084+0000 7fd61572a700 1 -- 172.21.15.70:0/622995695 shutdown_connections 2024-06-04T05:01:19.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.084+0000 7fd61572a700 1 --2- 172.21.15.70:0/622995695 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd5f80b29f0 0x7fd5f80b4eb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.084+0000 7fd61572a700 1 --2- 172.21.15.70:0/622995695 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6101084b0 0x7fd610199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.084+0000 7fd61572a700 1 --2- 172.21.15.70:0/622995695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd610102470 0x7fd61019f7b0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.084+0000 7fd61572a700 1 --2- 172.21.15.70:0/622995695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd610102e30 0x7fd61019fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:19.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.084+0000 7fd61572a700 1 -- 172.21.15.70:0/622995695 >> 172.21.15.70:0/622995695 conn(0x7fd610079410 msgr2=0x7fd61006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:19.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.084+0000 7fd61572a700 1 -- 172.21.15.70:0/622995695 shutdown_connections 2024-06-04T05:01:19.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:19.084+0000 7fd61572a700 1 -- 172.21.15.70:0/622995695 wait complete. 2024-06-04T05:01:19.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:19 smithi070 ceph-mon[32631]: pgmap v127: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 12 op/s 2024-06-04T05:01:19.414 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:01:19.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:19 smithi112 ceph-mon[36453]: pgmap v127: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 12 op/s 2024-06-04T05:01:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:19 smithi098 ceph-mon[35569]: pgmap v127: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 12 op/s 2024-06-04T05:01:19.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 12 sec latency 25160.723 ms 2024-06-04T05:01:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.108+0000 7f8bc10e1640 1 -- 172.21.15.70:0/263999715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bbc103ce0 msgr2=0x7f8bbc1040c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.108+0000 7f8bc10e1640 1 --2- 172.21.15.70:0/263999715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bbc103ce0 0x7f8bbc1040c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009900 tx=0x7f8bac02a510 comp rx=0 tx=0).stop 2024-06-04T05:01:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.108+0000 7f8bc10e1640 1 -- 172.21.15.70:0/263999715 shutdown_connections 2024-06-04T05:01:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.108+0000 7f8bc10e1640 1 --2- 172.21.15.70:0/263999715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bbc10d020 0x7f8bbc10f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.108+0000 7f8bc10e1640 1 --2- 172.21.15.70:0/263999715 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bbc104600 0x7f8bbc10cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.108+0000 7f8bc10e1640 1 --2- 172.21.15.70:0/263999715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bbc103ce0 0x7f8bbc1040c0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.108+0000 7f8bc10e1640 1 -- 172.21.15.70:0/263999715 >> 172.21.15.70:0/263999715 conn(0x7f8bbc078ad0 msgr2=0x7f8bbc078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.108+0000 7f8bc10e1640 1 -- 172.21.15.70:0/263999715 shutdown_connections 2024-06-04T05:01:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.108+0000 7f8bc10e1640 1 -- 172.21.15.70:0/263999715 wait complete. 2024-06-04T05:01:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.109+0000 7f8bc10e1640 1 Processor -- start 2024-06-04T05:01:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.109+0000 7f8bc10e1640 1 -- start start 2024-06-04T05:01:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.109+0000 7f8bc10e1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bbc103ce0 0x7f8bbc199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bc10e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bbc104600 0x7f8bbc1996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bc10e1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bbc10d020 0x7f8bbc19d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bc10e1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc112bc0 con 0x7f8bbc10d020 2024-06-04T05:01:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bc10e1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc112a40 con 0x7f8bbc104600 2024-06-04T05:01:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bc10e1640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc112d40 con 0x7f8bbc103ce0 2024-06-04T05:01:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bbb59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bbc10d020 0x7f8bbc19d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bbb59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bbc10d020 0x7f8bbc19d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42822/0 (socket says 172.21.15.70:42822) 2024-06-04T05:01:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bbb59e640 1 -- 172.21.15.70:0/4153115994 learned_addr learned my addr 172.21.15.70:0/4153115994 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bbad9d640 1 --2- 172.21.15.70:0/4153115994 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bbc103ce0 0x7f8bbc199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bba59c640 1 --2- 172.21.15.70:0/4153115994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bbc104600 0x7f8bbc1996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bbb59e640 1 -- 172.21.15.70:0/4153115994 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bbc103ce0 msgr2=0x7f8bbc199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bbb59e640 1 --2- 172.21.15.70:0/4153115994 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bbc103ce0 0x7f8bbc199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bbb59e640 1 -- 172.21.15.70:0/4153115994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bbc104600 msgr2=0x7f8bbc1996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bbb59e640 1 --2- 172.21.15.70:0/4153115994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bbc104600 0x7f8bbc1996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bbb59e640 1 -- 172.21.15.70:0/4153115994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bbc19e120 con 0x7f8bbc10d020 2024-06-04T05:01:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bbb59e640 1 --2- 172.21.15.70:0/4153115994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bbc10d020 0x7f8bbc19d9f0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400cce0 tx=0x7f8bb400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8ba3fff640 1 -- 172.21.15.70:0/4153115994 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb400f450 con 0x7f8bbc10d020 2024-06-04T05:01:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8ba3fff640 1 -- 172.21.15.70:0/4153115994 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8bb400fa80 con 0x7f8bbc10d020 2024-06-04T05:01:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bc10e1640 1 -- 172.21.15.70:0/4153115994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bbc19e410 con 0x7f8bbc10d020 2024-06-04T05:01:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8ba3fff640 1 -- 172.21.15.70:0/4153115994 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4018ad0 con 0x7f8bbc10d020 2024-06-04T05:01:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.110+0000 7f8bc10e1640 1 -- 172.21.15.70:0/4153115994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bbc1a5ca0 con 0x7f8bbc10d020 2024-06-04T05:01:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.113+0000 7f8ba3fff640 1 -- 172.21.15.70:0/4153115994 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f8bb400f5f0 con 0x7f8bbc10d020 2024-06-04T05:01:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.113+0000 7f8bc3e04740 1 Processor -- start 2024-06-04T05:01:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.113+0000 7f8bc3e04740 1 -- start start 2024-06-04T05:01:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.113+0000 7f8bc3e04740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d3d820 0x55de69d3dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.113+0000 7f8bc3e04740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3e140 0x55de69d484c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.113+0000 7f8bc3e04740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d48a00 0x55de69d4adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.113+0000 7f8bc3e04740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55de69cafc30 con 0x55de69d48a00 2024-06-04T05:01:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.113+0000 7f8bc3e04740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55de69c9f7d0 con 0x55de69d3e140 2024-06-04T05:01:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.113+0000 7f8bc3e04740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55de69ca3810 con 0x55de69d3d820 2024-06-04T05:01:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.113+0000 7f8bba59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d48a00 0x55de69d4adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.113+0000 7f8bba59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d48a00 0x55de69d4adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42830/0 (socket says 172.21.15.70:42830) 2024-06-04T05:01:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.113+0000 7f8bba59c640 1 -- 172.21.15.70:0/3187839693 learned_addr learned my addr 172.21.15.70:0/3187839693 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.113+0000 7f8bbad9d640 1 --2- 172.21.15.70:0/3187839693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3e140 0x55de69d484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.113+0000 7f8bba59c640 1 --2- 172.21.15.70:0/3187839693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d3d820 0x55de69d3dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.114+0000 7f8ba3fff640 1 --2- 172.21.15.70:0/4153115994 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8ba40b2a10 0x7f8ba40b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.114+0000 7f8bba59c640 1 -- 172.21.15.70:0/3187839693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d3d820 msgr2=0x55de69d3dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.114+0000 7f8bba59c640 1 --2- 172.21.15.70:0/3187839693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d3d820 0x55de69d3dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.114+0000 7f8bba59c640 1 -- 172.21.15.70:0/3187839693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3e140 msgr2=0x55de69d484c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.114+0000 7f8bba59c640 1 --2- 172.21.15.70:0/3187839693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3e140 0x55de69d484c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.114+0000 7f8bba59c640 1 -- 172.21.15.70:0/3187839693 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de699b4210 con 0x55de69d48a00 2024-06-04T05:01:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.114+0000 7f8bbb59e640 1 --2- 172.21.15.70:0/4153115994 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8ba40b2a10 0x7f8ba40b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.114+0000 7f8bba59c640 1 --2- 172.21.15.70:0/3187839693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d48a00 0x55de69d4adf0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0009e90 tx=0x7f8bb0009420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192ed619bf198d00 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.114+0000 7f8ba3fff640 1 -- 172.21.15.70:0/4153115994 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f8bb40e2df0 con 0x7f8bbc10d020 2024-06-04T05:01:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.114+0000 7f8ba17fa640 1 -- 172.21.15.70:0/3187839693 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb001c070 con 0x55de69d48a00 2024-06-04T05:01:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.114+0000 7f8ba17fa640 1 -- 172.21.15.70:0/3187839693 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8bb0010040 con 0x55de69d48a00 2024-06-04T05:01:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.114+0000 7f8ba17fa640 1 -- 172.21.15.70:0/3187839693 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb0014610 con 0x55de69d48a00 2024-06-04T05:01:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 1 -- 172.21.15.70:0/3187839693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d48a00 msgr2=0x55de69d4adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bbb59e640 1 --2- 172.21.15.70:0/4153115994 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8ba40b2a10 0x7f8ba40b4ed0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8bb40aa060 tx=0x7f8bb4076a00 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/3187839693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d48a00 0x55de69d4adf0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0009e90 tx=0x7f8bb0009420 comp rx=0 tx=0).stop 2024-06-04T05:01:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 1 -- 172.21.15.70:0/3187839693 shutdown_connections 2024-06-04T05:01:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/3187839693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3e140 0x55de69d484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/3187839693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d48a00 0x55de69d4adf0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/3187839693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d3d820 0x55de69d3dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 1 -- 172.21.15.70:0/3187839693 >> 172.21.15.70:0/3187839693 conn(0x55de69d39d30 msgr2=0x55de69d3a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 1 -- 172.21.15.70:0/3187839693 shutdown_connections 2024-06-04T05:01:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 1 -- 172.21.15.70:0/3187839693 wait complete. 2024-06-04T05:01:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 1 -- start start 2024-06-04T05:01:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 1 Processor -- start 2024-06-04T05:01:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3d820 0x55de69d45a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3e140 0x55de69d45f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d48a00 0x55de69d464c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55de69985790 con 0x55de69d3d820 2024-06-04T05:01:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55de69c5b890 con 0x55de69d3e140 2024-06-04T05:01:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55de69d4b630 con 0x55de69d48a00 2024-06-04T05:01:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bc3e04740 10 client.0 _ll_register_callbacks cb 0x55de699b3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bba59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3d820 0x55de69d45a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.115+0000 7f8bba59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3d820 0x55de69d45a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42844/0 (socket says 172.21.15.70:42844) 2024-06-04T05:01:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.116+0000 7f8bba59c640 1 -- 172.21.15.70:0/2675111053 learned_addr learned my addr 172.21.15.70:0/2675111053 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.116+0000 7f8bba59c640 1 --2- 172.21.15.70:0/2675111053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d48a00 0x55de69d464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.116+0000 7f8bba59c640 1 -- 172.21.15.70:0/2675111053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d48a00 msgr2=0x55de69d464c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.116+0000 7f8bba59c640 1 --2- 172.21.15.70:0/2675111053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d48a00 0x55de69d464c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.116+0000 7f8bba59c640 1 -- 172.21.15.70:0/2675111053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3e140 msgr2=0x55de69d45f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.116+0000 7f8bba59c640 1 --2- 172.21.15.70:0/2675111053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3e140 0x55de69d45f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.116+0000 7f8bba59c640 1 -- 172.21.15.70:0/2675111053 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de69a80620 con 0x55de69d3d820 2024-06-04T05:01:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.116+0000 7f8bba59c640 1 --2- 172.21.15.70:0/2675111053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3d820 0x55de69d45a40 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0017f00 tx=0x7f8bb0017e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fcc64664875db66 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.116+0000 7f8b9eff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.116+0000 7f8b9eff5640 1 -- 172.21.15.70:0/2675111053 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb001c040 con 0x55de69d3d820 2024-06-04T05:01:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.116+0000 7f8b9eff5640 1 -- 172.21.15.70:0/2675111053 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f8bb0024420 con 0x55de69d3d820 2024-06-04T05:01:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.116+0000 7f8bc3e04740 1 -- 172.21.15.70:0/2675111053 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bb0010040 con 0x55de69d3d820 2024-06-04T05:01:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.116+0000 7f8b9eff5640 1 -- 172.21.15.70:0/2675111053 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8bb0024620 con 0x55de69d3d820 2024-06-04T05:01:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.116+0000 7f8bc3e04740 10 client.16605 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.116+0000 7f8bc3e04740 1 -- 172.21.15.70:0/2675111053 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55de69d57430 con 0x55de69d3d820 2024-06-04T05:01:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.116+0000 7f8b9eff5640 1 -- 172.21.15.70:0/2675111053 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8bb0010040 con 0x55de69d3d820 2024-06-04T05:01:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.117+0000 7f8b9eff5640 1 -- 172.21.15.70:0/2675111053 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8bb0027560 con 0x55de69d3d820 2024-06-04T05:01:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.117+0000 7f8bc3e04740 10 client.16605 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.117+0000 7f8bc3e04740 10 client.16605 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.117+0000 7f8bc3e04740 20 client.16605 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.117+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/2675111053 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55de69d58780 0x55de69d5ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.117+0000 7f8bc3e04740 4 client.16605 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:01:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.117+0000 7f8bc3e04740 1 -- 172.21.15.70:0/2675111053 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de69cb1b80 con 0x55de69d58780 2024-06-04T05:01:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.117+0000 7f8bbad9d640 1 --2- 172.21.15.70:0/2675111053 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55de69d58780 0x55de69d5ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.117+0000 7f8b9eff5640 10 client.16605 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.117+0000 7f8bbad9d640 1 --2- 172.21.15.70:0/2675111053 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55de69d58780 0x55de69d5ab70 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=718bbfeb559cb94a server_cookie=4fb7c7ca965db867 in_seq=0 out_seq=0 2024-06-04T05:01:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.118+0000 7f8b9eff5640 1 -- 172.21.15.70:0/2675111053 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55de69cb1b80 con 0x55de69d58780 2024-06-04T05:01:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.118+0000 7f8b9eff5640 10 client.16605 handle_command_reply: tid=0 2024-06-04T05:01:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.118+0000 7f8bc3e04740 1 client.16605 shutdown 2024-06-04T05:01:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.118+0000 7f8bc3e04740 1 -- 172.21.15.70:0/2675111053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3d820 msgr2=0x55de69d45a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.118+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/2675111053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3d820 0x55de69d45a40 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0017f00 tx=0x7f8bb0017e00 comp rx=0 tx=0).stop 2024-06-04T05:01:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.118+0000 7f8bc3e04740 1 -- 172.21.15.70:0/2675111053 shutdown_connections 2024-06-04T05:01:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.118+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/2675111053 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55de69d58780 0x55de69d5ab70 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.118+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/2675111053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3e140 0x55de69d45f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.118+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/2675111053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d48a00 0x55de69d464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.118+0000 7f8b9eff5640 0 client.16605 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.118+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/2675111053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3d820 0x55de69d45a40 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.118+0000 7f8bc3e04740 1 -- 172.21.15.70:0/2675111053 >> 172.21.15.70:0/2675111053 conn(0x55de69d39d30 msgr2=0x55de69d38fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.118+0000 7f8bc3e04740 1 -- 172.21.15.70:0/2675111053 shutdown_connections 2024-06-04T05:01:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.118+0000 7f8bc3e04740 1 -- 172.21.15.70:0/2675111053 wait complete. 2024-06-04T05:01:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.118+0000 7f8bc3e04740 20 client.16605 trim_cache size 0 max 16384 2024-06-04T05:01:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bc3e04740 1 Processor -- start 2024-06-04T05:01:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bc3e04740 1 -- start start 2024-06-04T05:01:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bc3e04740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3d820 0x55de69d39730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bc3e04740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3e140 0x55de69d401e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bc3e04740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d48a00 0x55de69d60330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bc3e04740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc1ae600 con 0x55de69d3e140 2024-06-04T05:01:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bc3e04740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc1b01a0 con 0x55de69d3d820 2024-06-04T05:01:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bc3e04740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc1aecd0 con 0x55de69d48a00 2024-06-04T05:01:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bbad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3e140 0x55de69d401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bbad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3e140 0x55de69d401e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42858/0 (socket says 172.21.15.70:42858) 2024-06-04T05:01:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bbad9d640 1 -- 172.21.15.70:0/3909112492 learned_addr learned my addr 172.21.15.70:0/3909112492 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bba59c640 1 --2- 172.21.15.70:0/3909112492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3d820 0x55de69d39730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bbad9d640 1 -- 172.21.15.70:0/3909112492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d48a00 msgr2=0x55de69d60330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bbad9d640 1 --2- 172.21.15.70:0/3909112492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d48a00 0x55de69d60330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bbad9d640 1 -- 172.21.15.70:0/3909112492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3d820 msgr2=0x55de69d39730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bbad9d640 1 --2- 172.21.15.70:0/3909112492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3d820 0x55de69d39730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bbad9d640 1 -- 172.21.15.70:0/3909112492 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de69d469a0 con 0x55de69d3e140 2024-06-04T05:01:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.134+0000 7f8bbad9d640 1 --2- 172.21.15.70:0/3909112492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3e140 0x55de69d401e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f8bac02aa30 tx=0x7f8bac005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170f448bfa7dcdc6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.135+0000 7f8ba0ff9640 1 -- 172.21.15.70:0/3909112492 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac035040 con 0x55de69d3e140 2024-06-04T05:01:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.135+0000 7f8ba0ff9640 1 -- 172.21.15.70:0/3909112492 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8bac031890 con 0x55de69d3e140 2024-06-04T05:01:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.135+0000 7f8ba0ff9640 1 -- 172.21.15.70:0/3909112492 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac003d80 con 0x55de69d3e140 2024-06-04T05:01:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.135+0000 7f8bc3e04740 1 -- 172.21.15.70:0/3909112492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3e140 msgr2=0x55de69d401e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.135+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/3909112492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3e140 0x55de69d401e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f8bac02aa30 tx=0x7f8bac005e00 comp rx=0 tx=0).stop 2024-06-04T05:01:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.135+0000 7f8bc3e04740 1 -- 172.21.15.70:0/3909112492 shutdown_connections 2024-06-04T05:01:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.135+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/3909112492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d48a00 0x55de69d60330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.135+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/3909112492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3e140 0x55de69d401e0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.135+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/3909112492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3d820 0x55de69d39730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.135+0000 7f8bc3e04740 1 -- 172.21.15.70:0/3909112492 >> 172.21.15.70:0/3909112492 conn(0x55de69d39d30 msgr2=0x55de69d46290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.135+0000 7f8bc3e04740 1 -- 172.21.15.70:0/3909112492 shutdown_connections 2024-06-04T05:01:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bc3e04740 1 -- 172.21.15.70:0/3909112492 wait complete. 2024-06-04T05:01:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bc3e04740 1 -- start start 2024-06-04T05:01:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bc3e04740 1 Processor -- start 2024-06-04T05:01:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bc3e04740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3d820 0x55de69d51010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bc3e04740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3e140 0x55de69d4c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bc3e04740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d48a00 0x55de69d4c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bc3e04740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc1afa70 con 0x55de69d3d820 2024-06-04T05:01:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bc3e04740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc1af3a0 con 0x55de69d3e140 2024-06-04T05:01:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bc3e04740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc1ac5e0 con 0x55de69d48a00 2024-06-04T05:01:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bc3e04740 10 client.0 _ll_register_callbacks cb 0x7f8bb001a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bba59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3d820 0x55de69d51010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bba59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3d820 0x55de69d51010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42874/0 (socket says 172.21.15.70:42874) 2024-06-04T05:01:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bba59c640 1 -- 172.21.15.70:0/2081870876 learned_addr learned my addr 172.21.15.70:0/2081870876 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bbad9d640 1 --2- 172.21.15.70:0/2081870876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3e140 0x55de69d4c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bba59c640 1 -- 172.21.15.70:0/2081870876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d48a00 msgr2=0x55de69d4c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bba59c640 1 --2- 172.21.15.70:0/2081870876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d48a00 0x55de69d4c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bba59c640 1 -- 172.21.15.70:0/2081870876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3e140 msgr2=0x55de69d4c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.136+0000 7f8bba59c640 1 --2- 172.21.15.70:0/2081870876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3e140 0x55de69d4c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.137+0000 7f8bba59c640 1 -- 172.21.15.70:0/2081870876 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de69d4d150 con 0x55de69d3d820 2024-06-04T05:01:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.137+0000 7f8bba59c640 1 --2- 172.21.15.70:0/2081870876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3d820 0x55de69d51010 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f8bb001a040 tx=0x7f8bb00093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a8e905881155cc0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.137+0000 7f8b9eff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.137+0000 7f8b9eff5640 1 -- 172.21.15.70:0/2081870876 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb001c040 con 0x55de69d3d820 2024-06-04T05:01:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.137+0000 7f8b9eff5640 1 -- 172.21.15.70:0/2081870876 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f8bb0027650 con 0x55de69d3d820 2024-06-04T05:01:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.137+0000 7f8bc3e04740 1 -- 172.21.15.70:0/2081870876 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bac031890 con 0x55de69d3d820 2024-06-04T05:01:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.137+0000 7f8b9eff5640 1 -- 172.21.15.70:0/2081870876 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8bb0009ce0 con 0x55de69d3d820 2024-06-04T05:01:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.137+0000 7f8bc3e04740 10 client.16611 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.137+0000 7f8bc3e04740 1 -- 172.21.15.70:0/2081870876 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55de69dc32e0 con 0x55de69d3d820 2024-06-04T05:01:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.137+0000 7f8b9eff5640 1 -- 172.21.15.70:0/2081870876 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8bac031890 con 0x55de69d3d820 2024-06-04T05:01:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.137+0000 7f8b9eff5640 1 -- 172.21.15.70:0/2081870876 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8bb00247c0 con 0x55de69d3d820 2024-06-04T05:01:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.137+0000 7f8bc3e04740 10 client.16611 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.137+0000 7f8bc3e04740 10 client.16611 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.137+0000 7f8bc3e04740 20 client.16611 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.137+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/2081870876 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55de69d58780 0x55de69dc5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.137+0000 7f8bc3e04740 4 client.16611 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:01:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.137+0000 7f8bc3e04740 1 -- 172.21.15.70:0/2081870876 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55de69cab360 con 0x55de69d58780 2024-06-04T05:01:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.138+0000 7f8bbad9d640 1 --2- 172.21.15.70:0/2081870876 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55de69d58780 0x55de69dc5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:20.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.138+0000 7f8bbad9d640 1 --2- 172.21.15.70:0/2081870876 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55de69d58780 0x55de69dc5990 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82a4c5df94e4e2a8 server_cookie=c717c97ab8fd8b5 in_seq=0 out_seq=0 2024-06-04T05:01:20.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.138+0000 7f8b9eff5640 10 client.16611 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8b9eff5640 1 -- 172.21.15.70:0/2081870876 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55de69cab360 con 0x55de69d58780 2024-06-04T05:01:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8b9eff5640 10 client.16611 handle_command_reply: tid=0 2024-06-04T05:01:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc3e04740 1 client.16611 shutdown 2024-06-04T05:01:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc3e04740 1 -- 172.21.15.70:0/2081870876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3d820 msgr2=0x55de69d51010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/2081870876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3d820 0x55de69d51010 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f8bb001a040 tx=0x7f8bb00093f0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc3e04740 1 -- 172.21.15.70:0/2081870876 shutdown_connections 2024-06-04T05:01:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/2081870876 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55de69d58780 0x55de69dc5990 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/2081870876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de69d3e140 0x55de69d4c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/2081870876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55de69d48a00 0x55de69d4c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc3e04740 1 --2- 172.21.15.70:0/2081870876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55de69d3d820 0x55de69d51010 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8b9eff5640 0 client.16611 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc3e04740 1 -- 172.21.15.70:0/2081870876 >> 172.21.15.70:0/2081870876 conn(0x55de69d39d30 msgr2=0x55de69d53680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc3e04740 1 -- 172.21.15.70:0/2081870876 shutdown_connections 2024-06-04T05:01:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc3e04740 1 -- 172.21.15.70:0/2081870876 wait complete. 2024-06-04T05:01:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc3e04740 20 client.16611 trim_cache size 0 max 16384 2024-06-04T05:01:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:01:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:01:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:01:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc10e1640 1 -- 172.21.15.70:0/4153115994 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8ba40b2a10 msgr2=0x7f8ba40b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc10e1640 1 --2- 172.21.15.70:0/4153115994 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8ba40b2a10 0x7f8ba40b4ed0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8bb40aa060 tx=0x7f8bb4076a00 comp rx=0 tx=0).stop 2024-06-04T05:01:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc10e1640 1 -- 172.21.15.70:0/4153115994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bbc10d020 msgr2=0x7f8bbc19d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc10e1640 1 --2- 172.21.15.70:0/4153115994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bbc10d020 0x7f8bbc19d9f0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400cce0 tx=0x7f8bb400a620 comp rx=0 tx=0).stop 2024-06-04T05:01:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc10e1640 1 -- 172.21.15.70:0/4153115994 shutdown_connections 2024-06-04T05:01:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc10e1640 1 --2- 172.21.15.70:0/4153115994 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8ba40b2a10 0x7f8ba40b4ed0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc10e1640 1 --2- 172.21.15.70:0/4153115994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bbc104600 0x7f8bbc1996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc10e1640 1 --2- 172.21.15.70:0/4153115994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bbc10d020 0x7f8bbc19d9f0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc10e1640 1 --2- 172.21.15.70:0/4153115994 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bbc103ce0 0x7f8bbc199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.154+0000 7f8bc10e1640 1 -- 172.21.15.70:0/4153115994 >> 172.21.15.70:0/4153115994 conn(0x7f8bbc078ad0 msgr2=0x7f8bbc0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.155+0000 7f8bc10e1640 1 -- 172.21.15.70:0/4153115994 shutdown_connections 2024-06-04T05:01:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:20.155+0000 7f8bc10e1640 1 -- 172.21.15.70:0/4153115994 wait complete. 2024-06-04T05:01:20.252 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:01:20.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 13 sec latency 26160.833 ms 2024-06-04T05:01:21.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:21 smithi112 ceph-mon[36453]: pgmap v128: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 12 op/s 2024-06-04T05:01:21.445 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:21 smithi070 ceph-mon[32631]: pgmap v128: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 12 op/s 2024-06-04T05:01:21.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:21 smithi098 ceph-mon[35569]: pgmap v128: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 12 op/s 2024-06-04T05:01:21.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 14 sec latency 27160.915 ms 2024-06-04T05:01:21.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.845+0000 7fbf422de700 1 -- 172.21.15.70:0/2065994691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf3c1084f0 msgr2=0x7fbf3c1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:21.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.845+0000 7fbf422de700 1 --2- 172.21.15.70:0/2065994691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf3c1084f0 0x7fbf3c1088d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c009990 tx=0x7fbf2c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:01:21.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.845+0000 7fbf422de700 1 -- 172.21.15.70:0/2065994691 shutdown_connections 2024-06-04T05:01:21.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.845+0000 7fbf422de700 1 --2- 172.21.15.70:0/2065994691 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf3c102e20 0x7fbf3c10f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:21.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.845+0000 7fbf422de700 1 --2- 172.21.15.70:0/2065994691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf3c1084f0 0x7fbf3c1088d0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:21.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.845+0000 7fbf422de700 1 --2- 172.21.15.70:0/2065994691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf3c102460 0x7fbf3c1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:21.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.845+0000 7fbf422de700 1 -- 172.21.15.70:0/2065994691 >> 172.21.15.70:0/2065994691 conn(0x7fbf3c0794e0 msgr2=0x7fbf3c0798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:21.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.845+0000 7fbf422de700 1 -- 172.21.15.70:0/2065994691 shutdown_connections 2024-06-04T05:01:21.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.845+0000 7fbf422de700 1 -- 172.21.15.70:0/2065994691 wait complete. 2024-06-04T05:01:21.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.845+0000 7fbf422de700 1 Processor -- start 2024-06-04T05:01:21.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.845+0000 7fbf422de700 1 -- start start 2024-06-04T05:01:21.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf422de700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf3c102460 0x7fbf3c19aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:21.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf422de700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf3c102e20 0x7fbf3c19af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:21.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf422de700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf3c1084f0 0x7fbf3c1a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:21.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf422de700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3c112530 con 0x7fbf3c1084f0 2024-06-04T05:01:21.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf422de700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3c1123b0 con 0x7fbf3c102e20 2024-06-04T05:01:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf422de700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3c1126b0 con 0x7fbf3c102460 2024-06-04T05:01:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf4087b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf3c1084f0 0x7fbf3c1a5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf4087b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf3c1084f0 0x7fbf3c1a5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42896/0 (socket says 172.21.15.70:42896) 2024-06-04T05:01:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf4087b700 1 -- 172.21.15.70:0/425474509 learned_addr learned my addr 172.21.15.70:0/425474509 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf3bfff700 1 --2- 172.21.15.70:0/425474509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf3c102460 0x7fbf3c19aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf3b7fe700 1 --2- 172.21.15.70:0/425474509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf3c102e20 0x7fbf3c19af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf4087b700 1 -- 172.21.15.70:0/425474509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf3c102460 msgr2=0x7fbf3c19aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf4087b700 1 --2- 172.21.15.70:0/425474509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf3c102460 0x7fbf3c19aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf4087b700 1 -- 172.21.15.70:0/425474509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf3c102e20 msgr2=0x7fbf3c19af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf4087b700 1 --2- 172.21.15.70:0/425474509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf3c102e20 0x7fbf3c19af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf4087b700 1 -- 172.21.15.70:0/425474509 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf3c1a5fd0 con 0x7fbf3c1084f0 2024-06-04T05:01:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf3b7fe700 1 --2- 172.21.15.70:0/425474509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf3c102e20 0x7fbf3c19af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:01:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.846+0000 7fbf4087b700 1 --2- 172.21.15.70:0/425474509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf3c1084f0 0x7fbf3c1a5a90 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fbf3400b970 tx=0x7fbf3400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:21.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.847+0000 7fbf397fa700 1 -- 172.21.15.70:0/425474509 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf34019070 con 0x7fbf3c1084f0 2024-06-04T05:01:21.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.847+0000 7fbf397fa700 1 -- 172.21.15.70:0/425474509 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf3400cdc0 con 0x7fbf3c1084f0 2024-06-04T05:01:21.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.847+0000 7fbf422de700 1 -- 172.21.15.70:0/425474509 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf3c1a6200 con 0x7fbf3c1084f0 2024-06-04T05:01:21.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.847+0000 7fbf397fa700 1 -- 172.21.15.70:0/425474509 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf34014e30 con 0x7fbf3c1084f0 2024-06-04T05:01:21.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.847+0000 7fbf422de700 1 -- 172.21.15.70:0/425474509 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf3c1a6710 con 0x7fbf3c1084f0 2024-06-04T05:01:21.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.848+0000 7fbf422de700 1 -- 172.21.15.70:0/425474509 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf3c10aa60 con 0x7fbf3c1084f0 2024-06-04T05:01:21.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.849+0000 7fbf397fa700 1 -- 172.21.15.70:0/425474509 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fbf3401c070 con 0x7fbf3c1084f0 2024-06-04T05:01:21.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.849+0000 7fbf397fa700 1 --2- 172.21.15.70:0/425474509 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf280b29f0 0x7fbf280b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:21.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.850+0000 7fbf3bfff700 1 --2- 172.21.15.70:0/425474509 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf280b29f0 0x7fbf280b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:21.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.850+0000 7fbf397fa700 1 -- 172.21.15.70:0/425474509 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fbf340e1850 con 0x7fbf3c1084f0 2024-06-04T05:01:21.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.850+0000 7fbf3bfff700 1 --2- 172.21.15.70:0/425474509 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf280b29f0 0x7fbf280b4eb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c00b5c0 tx=0x7fbf2c005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:21.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:21.854+0000 7fbf397fa700 1 -- 172.21.15.70:0/425474509 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbf340a9340 con 0x7fbf3c1084f0 2024-06-04T05:01:21.933 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:01:22.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:22.131+0000 7fbf422de700 1 -- 172.21.15.70:0/425474509 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf3c10a440 con 0x7fbf3c1084f0 2024-06-04T05:01:22.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:22.132+0000 7fbf397fa700 1 -- 172.21.15.70:0/425474509 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11932 (secure 0 0 0) 0x7fbf340b13c0 con 0x7fbf3c1084f0 2024-06-04T05:01:22.134 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:01:22.134 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T04:59:22.801688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:01:22.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:22.134+0000 7fbf422de700 1 -- 172.21.15.70:0/425474509 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf280b29f0 msgr2=0x7fbf280b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:22.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:22.134+0000 7fbf422de700 1 --2- 172.21.15.70:0/425474509 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf280b29f0 0x7fbf280b4eb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c00b5c0 tx=0x7fbf2c005960 comp rx=0 tx=0).stop 2024-06-04T05:01:22.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:22.134+0000 7fbf422de700 1 -- 172.21.15.70:0/425474509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf3c1084f0 msgr2=0x7fbf3c1a5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:22.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:22.134+0000 7fbf422de700 1 --2- 172.21.15.70:0/425474509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf3c1084f0 0x7fbf3c1a5a90 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fbf3400b970 tx=0x7fbf3400bc80 comp rx=0 tx=0).stop 2024-06-04T05:01:22.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:22.134+0000 7fbf422de700 1 -- 172.21.15.70:0/425474509 shutdown_connections 2024-06-04T05:01:22.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:22.134+0000 7fbf422de700 1 --2- 172.21.15.70:0/425474509 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf280b29f0 0x7fbf280b4eb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:22.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:22.134+0000 7fbf422de700 1 --2- 172.21.15.70:0/425474509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf3c1084f0 0x7fbf3c1a5a90 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:22.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:22.134+0000 7fbf422de700 1 --2- 172.21.15.70:0/425474509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf3c102460 0x7fbf3c19aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:22.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:22.134+0000 7fbf422de700 1 --2- 172.21.15.70:0/425474509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf3c102e20 0x7fbf3c19af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:22.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:22.134+0000 7fbf422de700 1 -- 172.21.15.70:0/425474509 >> 172.21.15.70:0/425474509 conn(0x7fbf3c0794e0 msgr2=0x7fbf3c1051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:22.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:22.134+0000 7fbf422de700 1 -- 172.21.15.70:0/425474509 shutdown_connections 2024-06-04T05:01:22.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:22.134+0000 7fbf422de700 1 -- 172.21.15.70:0/425474509 wait complete. 2024-06-04T05:01:22.137 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-06-04T05:01:22.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 15 sec latency 28161.023 ms 2024-06-04T05:01:23.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:23 smithi070 ceph-mon[32631]: pgmap v129: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 19 op/s 2024-06-04T05:01:23.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:23 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/425474509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:01:23.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:23 smithi112 ceph-mon[36453]: pgmap v129: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 19 op/s 2024-06-04T05:01:23.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:23 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/425474509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:01:23.437 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id 3ea23d63 --release --await' 2024-06-04T05:01:23.437 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id 3ea23d63 --release --await 2024-06-04T05:01:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:23 smithi098 ceph-mon[35569]: pgmap v129: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 19 op/s 2024-06-04T05:01:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:23 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/425474509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:01:23.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 16 sec latency 29161.062 ms 2024-06-04T05:01:24.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.431+0000 7fc620b8f700 1 -- 172.21.15.70:0/175433989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c102e00 msgr2=0x7fc61c10f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:24.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.431+0000 7fc620b8f700 1 --2- 172.21.15.70:0/175433989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c102e00 0x7fc61c10f2f0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc61400b0a0 tx=0x7fc61400b3b0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.431+0000 7fc620b8f700 1 -- 172.21.15.70:0/175433989 shutdown_connections 2024-06-04T05:01:24.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.431+0000 7fc620b8f700 1 --2- 172.21.15.70:0/175433989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c102e00 0x7fc61c10f2f0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.431+0000 7fc620b8f700 1 --2- 172.21.15.70:0/175433989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc61c108520 0x7fc61c108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.432+0000 7fc620b8f700 1 --2- 172.21.15.70:0/175433989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc61c102440 0x7fc61c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.432+0000 7fc620b8f700 1 -- 172.21.15.70:0/175433989 >> 172.21.15.70:0/175433989 conn(0x7fc61c0793f0 msgr2=0x7fc61c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:24.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.432+0000 7fc620b8f700 1 -- 172.21.15.70:0/175433989 shutdown_connections 2024-06-04T05:01:24.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.432+0000 7fc620b8f700 1 -- 172.21.15.70:0/175433989 wait complete. 2024-06-04T05:01:24.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.432+0000 7fc620b8f700 1 Processor -- start 2024-06-04T05:01:24.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.432+0000 7fc620b8f700 1 -- start start 2024-06-04T05:01:24.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.432+0000 7fc620b8f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c102440 0x7fc61c19e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc620b8f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc61c102e00 0x7fc61c19eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc620b8f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc61c108520 0x7fc61c1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc620b8f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c112480 con 0x7fc61c102440 2024-06-04T05:01:24.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc620b8f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c112300 con 0x7fc61c102e00 2024-06-04T05:01:24.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc620b8f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c112600 con 0x7fc61c108520 2024-06-04T05:01:24.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc61a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c102440 0x7fc61c19e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc61a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c102440 0x7fc61c19e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42914/0 (socket says 172.21.15.70:42914) 2024-06-04T05:01:24.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc61a59c700 1 -- 172.21.15.70:0/2729654176 learned_addr learned my addr 172.21.15.70:0/2729654176 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:24.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc619d9b700 1 --2- 172.21.15.70:0/2729654176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc61c102e00 0x7fc61c19eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc61ad9d700 1 --2- 172.21.15.70:0/2729654176 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc61c108520 0x7fc61c1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc61a59c700 1 -- 172.21.15.70:0/2729654176 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc61c108520 msgr2=0x7fc61c1986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:24.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc61a59c700 1 --2- 172.21.15.70:0/2729654176 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc61c108520 0x7fc61c1986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc61a59c700 1 -- 172.21.15.70:0/2729654176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc61c102e00 msgr2=0x7fc61c19eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:24.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc61a59c700 1 --2- 172.21.15.70:0/2729654176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc61c102e00 0x7fc61c19eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc61a59c700 1 -- 172.21.15.70:0/2729654176 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c198f80 con 0x7fc61c102440 2024-06-04T05:01:24.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc61a59c700 1 --2- 172.21.15.70:0/2729654176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c102440 0x7fc61c19e5d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fc60c00c9b0 tx=0x7fc60c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:24.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc60b7fe700 1 -- 172.21.15.70:0/2729654176 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60c007b00 con 0x7fc61c102440 2024-06-04T05:01:24.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc620b8f700 1 -- 172.21.15.70:0/2729654176 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61c199210 con 0x7fc61c102440 2024-06-04T05:01:24.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc60b7fe700 1 -- 172.21.15.70:0/2729654176 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc60c00ce50 con 0x7fc61c102440 2024-06-04T05:01:24.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc60b7fe700 1 -- 172.21.15.70:0/2729654176 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60c018660 con 0x7fc61c102440 2024-06-04T05:01:24.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.433+0000 7fc620b8f700 1 -- 172.21.15.70:0/2729654176 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc61c1a5550 con 0x7fc61c102440 2024-06-04T05:01:24.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.436+0000 7fc60b7fe700 1 -- 172.21.15.70:0/2729654176 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fc60c00f450 con 0x7fc61c102440 2024-06-04T05:01:24.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.437+0000 7fc60b7fe700 1 --2- 172.21.15.70:0/2729654176 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc6040b29f0 0x7fc6040b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.437+0000 7fc619d9b700 1 --2- 172.21.15.70:0/2729654176 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc6040b29f0 0x7fc6040b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.437+0000 7fc60b7fe700 1 -- 172.21.15.70:0/2729654176 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc60c0e2520 con 0x7fc61c102440 2024-06-04T05:01:24.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.437+0000 7fc619d9b700 1 --2- 172.21.15.70:0/2729654176 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc6040b29f0 0x7fc6040b4eb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc61c199e60 tx=0x7fc61000a380 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:24.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.438+0000 7fc62b424b80 1 Processor -- start 2024-06-04T05:01:24.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.438+0000 7fc62b424b80 1 -- start start 2024-06-04T05:01:24.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.438+0000 7fc62b424b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5087d70 0x5603c5088150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.438+0000 7fc62b424b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5086680 0x5603c5086b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.438+0000 7fc62b424b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087040 0x5603c5096d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.438+0000 7fc62b424b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5603c4fe5c90 con 0x5603c5087040 2024-06-04T05:01:24.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.438+0000 7fc62b424b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5603c4fd5830 con 0x5603c5087d70 2024-06-04T05:01:24.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.438+0000 7fc62b424b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5603c4fd9870 con 0x5603c5086680 2024-06-04T05:01:24.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.438+0000 7fc61ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087040 0x5603c5096d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.438+0000 7fc61ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087040 0x5603c5096d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42918/0 (socket says 172.21.15.70:42918) 2024-06-04T05:01:24.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.438+0000 7fc61ad9d700 1 -- 172.21.15.70:0/742840009 learned_addr learned my addr 172.21.15.70:0/742840009 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:24.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.438+0000 7fc61a59c700 1 --2- 172.21.15.70:0/742840009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5087d70 0x5603c5088150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.438+0000 7fc619d9b700 1 --2- 172.21.15.70:0/742840009 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5086680 0x5603c5086b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.439+0000 7fc61ad9d700 1 -- 172.21.15.70:0/742840009 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5086680 msgr2=0x5603c5086b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:24.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.439+0000 7fc61ad9d700 1 --2- 172.21.15.70:0/742840009 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5086680 0x5603c5086b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.439+0000 7fc61ad9d700 1 -- 172.21.15.70:0/742840009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5087d70 msgr2=0x5603c5088150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:24.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.439+0000 7fc61ad9d700 1 --2- 172.21.15.70:0/742840009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5087d70 0x5603c5088150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.439+0000 7fc61ad9d700 1 -- 172.21.15.70:0/742840009 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603c4f258d0 con 0x5603c5087040 2024-06-04T05:01:24.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.439+0000 7fc61ad9d700 1 --2- 172.21.15.70:0/742840009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087040 0x5603c5096d10 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fc6140093f0 tx=0x7fc614009420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9d654c6655bb2e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:24.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.439+0000 7fc608ff9700 1 -- 172.21.15.70:0/742840009 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614020070 con 0x5603c5087040 2024-06-04T05:01:24.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.439+0000 7fc608ff9700 1 -- 172.21.15.70:0/742840009 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc6140074e0 con 0x5603c5087040 2024-06-04T05:01:24.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.439+0000 7fc608ff9700 1 -- 172.21.15.70:0/742840009 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614005260 con 0x5603c5087040 2024-06-04T05:01:24.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.440+0000 7fc62b424b80 1 -- 172.21.15.70:0/742840009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087040 msgr2=0x5603c5096d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:24.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.440+0000 7fc62b424b80 1 --2- 172.21.15.70:0/742840009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087040 0x5603c5096d10 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fc6140093f0 tx=0x7fc614009420 comp rx=0 tx=0).stop 2024-06-04T05:01:24.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.440+0000 7fc62b424b80 1 -- 172.21.15.70:0/742840009 shutdown_connections 2024-06-04T05:01:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.440+0000 7fc62b424b80 1 --2- 172.21.15.70:0/742840009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087040 0x5603c5096d10 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.440+0000 7fc62b424b80 1 --2- 172.21.15.70:0/742840009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5087d70 0x5603c5088150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.440+0000 7fc62b424b80 1 --2- 172.21.15.70:0/742840009 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5086680 0x5603c5086b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.440+0000 7fc62b424b80 1 -- 172.21.15.70:0/742840009 >> 172.21.15.70:0/742840009 conn(0x5603c4fb4080 msgr2=0x5603c5080e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.440+0000 7fc62b424b80 1 -- 172.21.15.70:0/742840009 shutdown_connections 2024-06-04T05:01:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.440+0000 7fc62b424b80 1 -- 172.21.15.70:0/742840009 wait complete. 2024-06-04T05:01:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.440+0000 7fc62b424b80 1 -- start start 2024-06-04T05:01:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.440+0000 7fc62b424b80 1 Processor -- start 2024-06-04T05:01:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.441+0000 7fc62b424b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5086680 0x5603c5096a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.441+0000 7fc62b424b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5087040 0x5603c5093d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.441+0000 7fc62b424b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087d70 0x5603c5094270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.441+0000 7fc62b424b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5603c4edbfa0 con 0x5603c5087d70 2024-06-04T05:01:24.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.441+0000 7fc62b424b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5603c4f777c0 con 0x5603c5087040 2024-06-04T05:01:24.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.441+0000 7fc62b424b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5603c5097570 con 0x5603c5086680 2024-06-04T05:01:24.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.441+0000 7fc61ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087d70 0x5603c5094270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.441+0000 7fc61ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087d70 0x5603c5094270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42930/0 (socket says 172.21.15.70:42930) 2024-06-04T05:01:24.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.441+0000 7fc61ad9d700 1 -- 172.21.15.70:0/2256080031 learned_addr learned my addr 172.21.15.70:0/2256080031 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:24.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.441+0000 7fc62b424b80 10 client.0 _ll_register_callbacks cb 0x5603c4f70920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:24.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.441+0000 7fc61a59c700 1 --2- 172.21.15.70:0/2256080031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5086680 0x5603c5096a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.441+0000 7fc619d9b700 1 --2- 172.21.15.70:0/2256080031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5087040 0x5603c5093d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.441+0000 7fc61ad9d700 1 -- 172.21.15.70:0/2256080031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5086680 msgr2=0x5603c5096a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:24.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.441+0000 7fc61ad9d700 1 --2- 172.21.15.70:0/2256080031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5086680 0x5603c5096a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.441+0000 7fc61ad9d700 1 -- 172.21.15.70:0/2256080031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5087040 msgr2=0x5603c5093d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:24.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.441+0000 7fc61ad9d700 1 --2- 172.21.15.70:0/2256080031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5087040 0x5603c5093d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.441+0000 7fc61ad9d700 1 -- 172.21.15.70:0/2256080031 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603c4f74690 con 0x5603c5087d70 2024-06-04T05:01:24.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.442+0000 7fc61ad9d700 1 --2- 172.21.15.70:0/2256080031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087d70 0x5603c5094270 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc61400b3f0 tx=0x7fc614007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2993c38aa1c7137b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:24.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.442+0000 7fc601ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:24.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.442+0000 7fc601ffb700 1 -- 172.21.15.70:0/2256080031 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614020040 con 0x5603c5087d70 2024-06-04T05:01:24.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.442+0000 7fc601ffb700 1 -- 172.21.15.70:0/2256080031 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc61402f480 con 0x5603c5087d70 2024-06-04T05:01:24.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.442+0000 7fc62b424b80 1 -- 172.21.15.70:0/2256080031 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6140074e0 con 0x5603c5087d70 2024-06-04T05:01:24.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.442+0000 7fc601ffb700 1 -- 172.21.15.70:0/2256080031 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc61402f680 con 0x5603c5087d70 2024-06-04T05:01:24.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.442+0000 7fc62b424b80 10 client.16629 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:24.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.442+0000 7fc62b424b80 1 -- 172.21.15.70:0/2256080031 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5603c50a1d30 con 0x5603c5087d70 2024-06-04T05:01:24.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.442+0000 7fc601ffb700 1 -- 172.21.15.70:0/2256080031 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc6140074e0 con 0x5603c5087d70 2024-06-04T05:01:24.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.442+0000 7fc601ffb700 1 -- 172.21.15.70:0/2256080031 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc614011070 con 0x5603c5087d70 2024-06-04T05:01:24.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.442+0000 7fc62b424b80 10 client.16629 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:24.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.442+0000 7fc62b424b80 10 client.16629 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:01:24.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.442+0000 7fc62b424b80 20 client.16629 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:24.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.443+0000 7fc62b424b80 1 --2- 172.21.15.70:0/2256080031 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5603c50a4200 0x5603c50a65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.443+0000 7fc62b424b80 4 client.16629 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:01:24.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.443+0000 7fc62b424b80 1 -- 172.21.15.70:0/2256080031 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603c50529e0 con 0x5603c50a4200 2024-06-04T05:01:24.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.443+0000 7fc61a59c700 1 --2- 172.21.15.70:0/2256080031 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5603c50a4200 0x5603c50a65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.443+0000 7fc61a59c700 1 --2- 172.21.15.70:0/2256080031 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5603c50a4200 0x5603c50a65f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1289160a81e81333 server_cookie=f2c0c98252dbd25d in_seq=0 out_seq=0 2024-06-04T05:01:24.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.443+0000 7fc601ffb700 10 client.16629 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:01:24.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.444+0000 7fc601ffb700 1 -- 172.21.15.70:0/2256080031 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5603c50529e0 con 0x5603c50a4200 2024-06-04T05:01:24.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.444+0000 7fc601ffb700 10 client.16629 handle_command_reply: tid=0 2024-06-04T05:01:24.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.444+0000 7fc62b424b80 1 client.16629 shutdown 2024-06-04T05:01:24.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.444+0000 7fc62b424b80 1 -- 172.21.15.70:0/2256080031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087d70 msgr2=0x5603c5094270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:24.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.444+0000 7fc62b424b80 1 --2- 172.21.15.70:0/2256080031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087d70 0x5603c5094270 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc61400b3f0 tx=0x7fc614007ea0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.444+0000 7fc62b424b80 1 -- 172.21.15.70:0/2256080031 shutdown_connections 2024-06-04T05:01:24.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.444+0000 7fc62b424b80 1 --2- 172.21.15.70:0/2256080031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087d70 0x5603c5094270 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.444+0000 7fc62b424b80 1 --2- 172.21.15.70:0/2256080031 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5603c50a4200 0x5603c50a65f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.444+0000 7fc62b424b80 1 --2- 172.21.15.70:0/2256080031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5086680 0x5603c5096a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.444+0000 7fc62b424b80 1 --2- 172.21.15.70:0/2256080031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5087040 0x5603c5093d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.444+0000 7fc62b424b80 1 -- 172.21.15.70:0/2256080031 >> 172.21.15.70:0/2256080031 conn(0x5603c4fb4080 msgr2=0x5603c5080510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:24.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.444+0000 7fc601ffb700 0 client.16629 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:01:24.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.444+0000 7fc62b424b80 1 -- 172.21.15.70:0/2256080031 shutdown_connections 2024-06-04T05:01:24.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.444+0000 7fc62b424b80 1 -- 172.21.15.70:0/2256080031 wait complete. 2024-06-04T05:01:24.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.444+0000 7fc62b424b80 20 client.16629 trim_cache size 0 max 16384 2024-06-04T05:01:24.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.459+0000 7fc62b424b80 1 Processor -- start 2024-06-04T05:01:24.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.459+0000 7fc62b424b80 1 -- start start 2024-06-04T05:01:24.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.460+0000 7fc62b424b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5086680 0x5603c5093cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.460+0000 7fc62b424b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5087040 0x5603c5094230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.460+0000 7fc62b424b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087d70 0x5603c5080c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.460+0000 7fc62b424b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c1ab950 con 0x5603c5087d70 2024-06-04T05:01:24.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.460+0000 7fc62b424b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c1ad4f0 con 0x5603c5086680 2024-06-04T05:01:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.460+0000 7fc62b424b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c1ac020 con 0x5603c5087040 2024-06-04T05:01:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.460+0000 7fc61ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087d70 0x5603c5080c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.460+0000 7fc61ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087d70 0x5603c5080c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42938/0 (socket says 172.21.15.70:42938) 2024-06-04T05:01:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.460+0000 7fc61ad9d700 1 -- 172.21.15.70:0/3027687833 learned_addr learned my addr 172.21.15.70:0/3027687833 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.460+0000 7fc619d9b700 1 --2- 172.21.15.70:0/3027687833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5087040 0x5603c5094230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.460+0000 7fc61a59c700 1 --2- 172.21.15.70:0/3027687833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5086680 0x5603c5093cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.460+0000 7fc61ad9d700 1 -- 172.21.15.70:0/3027687833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5087040 msgr2=0x5603c5094230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.460+0000 7fc61ad9d700 1 --2- 172.21.15.70:0/3027687833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5087040 0x5603c5094230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.460+0000 7fc61ad9d700 1 -- 172.21.15.70:0/3027687833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5086680 msgr2=0x5603c5093cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.460+0000 7fc61ad9d700 1 --2- 172.21.15.70:0/3027687833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5086680 0x5603c5093cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.460+0000 7fc61ad9d700 1 -- 172.21.15.70:0/3027687833 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603c5094c70 con 0x5603c5087d70 2024-06-04T05:01:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.460+0000 7fc61ad9d700 1 --2- 172.21.15.70:0/3027687833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087d70 0x5603c5080c90 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fc614009a20 tx=0x7fc6140021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70551cbd0650c11a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.461+0000 7fc603fff700 1 -- 172.21.15.70:0/3027687833 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614020070 con 0x5603c5087d70 2024-06-04T05:01:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.461+0000 7fc603fff700 1 -- 172.21.15.70:0/3027687833 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc614007b50 con 0x5603c5087d70 2024-06-04T05:01:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.461+0000 7fc603fff700 1 -- 172.21.15.70:0/3027687833 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614005c60 con 0x5603c5087d70 2024-06-04T05:01:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.461+0000 7fc62b424b80 1 -- 172.21.15.70:0/3027687833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087d70 msgr2=0x5603c5080c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.461+0000 7fc62b424b80 1 --2- 172.21.15.70:0/3027687833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087d70 0x5603c5080c90 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fc614009a20 tx=0x7fc6140021d0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc62b424b80 1 -- 172.21.15.70:0/3027687833 shutdown_connections 2024-06-04T05:01:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc62b424b80 1 --2- 172.21.15.70:0/3027687833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087d70 0x5603c5080c90 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc62b424b80 1 --2- 172.21.15.70:0/3027687833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5086680 0x5603c5093cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc62b424b80 1 --2- 172.21.15.70:0/3027687833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5087040 0x5603c5094230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc62b424b80 1 -- 172.21.15.70:0/3027687833 >> 172.21.15.70:0/3027687833 conn(0x5603c4fb4080 msgr2=0x5603c50935e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc62b424b80 1 -- 172.21.15.70:0/3027687833 shutdown_connections 2024-06-04T05:01:24.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc62b424b80 1 -- 172.21.15.70:0/3027687833 wait complete. 2024-06-04T05:01:24.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc62b424b80 1 -- start start 2024-06-04T05:01:24.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc62b424b80 1 Processor -- start 2024-06-04T05:01:24.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc62b424b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5086680 0x5603c50a9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc62b424b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087040 0x5603c50a6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc62b424b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5087d70 0x5603c50a70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc62b424b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c1acdc0 con 0x5603c5087040 2024-06-04T05:01:24.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc62b424b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c1ac6f0 con 0x5603c5086680 2024-06-04T05:01:24.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc62b424b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c1a98f0 con 0x5603c5087d70 2024-06-04T05:01:24.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc62b424b80 10 client.0 _ll_register_callbacks cb 0x5603c4f70920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:24.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc619d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087040 0x5603c50a6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc619d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087040 0x5603c50a6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42954/0 (socket says 172.21.15.70:42954) 2024-06-04T05:01:24.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.462+0000 7fc619d9b700 1 -- 172.21.15.70:0/1009879535 learned_addr learned my addr 172.21.15.70:0/1009879535 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:24.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.463+0000 7fc61ad9d700 1 --2- 172.21.15.70:0/1009879535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5087d70 0x5603c50a70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.463+0000 7fc61a59c700 1 --2- 172.21.15.70:0/1009879535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5086680 0x5603c50a9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.463+0000 7fc619d9b700 1 -- 172.21.15.70:0/1009879535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5087d70 msgr2=0x5603c50a70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:24.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.463+0000 7fc619d9b700 1 --2- 172.21.15.70:0/1009879535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5087d70 0x5603c50a70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.463+0000 7fc619d9b700 1 -- 172.21.15.70:0/1009879535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5086680 msgr2=0x5603c50a9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:24.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.463+0000 7fc619d9b700 1 --2- 172.21.15.70:0/1009879535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5086680 0x5603c50a9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.463+0000 7fc619d9b700 1 -- 172.21.15.70:0/1009879535 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603c50a7800 con 0x5603c5087040 2024-06-04T05:01:24.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.463+0000 7fc61ad9d700 1 --2- 172.21.15.70:0/1009879535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5087d70 0x5603c50a70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:24.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.463+0000 7fc61a59c700 1 --2- 172.21.15.70:0/1009879535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5086680 0x5603c50a9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:01:24.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.463+0000 7fc619d9b700 1 --2- 172.21.15.70:0/1009879535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087040 0x5603c50a6bb0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc61c199ef0 tx=0x7fc610012070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2583d73b4f4011d5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:24.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.463+0000 7fc601ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:24.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.463+0000 7fc601ffb700 1 -- 172.21.15.70:0/1009879535 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61002d060 con 0x5603c5087040 2024-06-04T05:01:24.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.463+0000 7fc62b424b80 1 -- 172.21.15.70:0/1009879535 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc614007b50 con 0x5603c5087040 2024-06-04T05:01:24.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.463+0000 7fc601ffb700 1 -- 172.21.15.70:0/1009879535 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc610010900 con 0x5603c5087040 2024-06-04T05:01:24.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.463+0000 7fc601ffb700 1 -- 172.21.15.70:0/1009879535 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc610016070 con 0x5603c5087040 2024-06-04T05:01:24.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.463+0000 7fc62b424b80 10 client.16635 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:24.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.463+0000 7fc62b424b80 1 -- 172.21.15.70:0/1009879535 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5603c510e9f0 con 0x5603c5087040 2024-06-04T05:01:24.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.464+0000 7fc601ffb700 1 -- 172.21.15.70:0/1009879535 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc614007b50 con 0x5603c5087040 2024-06-04T05:01:24.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.464+0000 7fc601ffb700 1 -- 172.21.15.70:0/1009879535 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc610028ba0 con 0x5603c5087040 2024-06-04T05:01:24.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.464+0000 7fc62b424b80 10 client.16635 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:24.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.464+0000 7fc62b424b80 10 client.16635 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:01:24.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.464+0000 7fc62b424b80 20 client.16635 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:24.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.464+0000 7fc62b424b80 1 --2- 172.21.15.70:0/1009879535 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5603c510f160 0x5603c5111580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:24.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.464+0000 7fc62b424b80 4 client.16635 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:01:24.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.464+0000 7fc62b424b80 1 -- 172.21.15.70:0/1009879535 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5603c4fe7be0 con 0x5603c510f160 2024-06-04T05:01:24.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.464+0000 7fc61a59c700 1 --2- 172.21.15.70:0/1009879535 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5603c510f160 0x5603c5111580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:24.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.465+0000 7fc601ffb700 10 client.16635 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:01:24.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.465+0000 7fc61a59c700 1 --2- 172.21.15.70:0/1009879535 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5603c510f160 0x5603c5111580 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=742f91826d3796a5 server_cookie=147722dd6147c32c in_seq=0 out_seq=0 2024-06-04T05:01:24.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.465+0000 7fc601ffb700 1 -- 172.21.15.70:0/1009879535 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5603c4fe7be0 con 0x5603c510f160 2024-06-04T05:01:24.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.465+0000 7fc601ffb700 10 client.16635 handle_command_reply: tid=0 2024-06-04T05:01:24.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.465+0000 7fc62b424b80 1 client.16635 shutdown 2024-06-04T05:01:24.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.465+0000 7fc62b424b80 1 -- 172.21.15.70:0/1009879535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087040 msgr2=0x5603c50a6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:24.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.465+0000 7fc62b424b80 1 --2- 172.21.15.70:0/1009879535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087040 0x5603c50a6bb0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc61c199ef0 tx=0x7fc610012070 comp rx=0 tx=0).stop 2024-06-04T05:01:24.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.465+0000 7fc62b424b80 1 -- 172.21.15.70:0/1009879535 shutdown_connections 2024-06-04T05:01:24.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.465+0000 7fc62b424b80 1 --2- 172.21.15.70:0/1009879535 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5603c510f160 0x5603c5111580 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.465+0000 7fc62b424b80 1 --2- 172.21.15.70:0/1009879535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603c5087d70 0x5603c50a70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.465+0000 7fc62b424b80 1 --2- 172.21.15.70:0/1009879535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603c5086680 0x5603c50a9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.465+0000 7fc62b424b80 1 --2- 172.21.15.70:0/1009879535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603c5087040 0x5603c50a6bb0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.465+0000 7fc62b424b80 1 -- 172.21.15.70:0/1009879535 >> 172.21.15.70:0/1009879535 conn(0x5603c4fb4080 msgr2=0x5603c5093d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:24.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.465+0000 7fc601ffb700 0 client.16635 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:01:24.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.465+0000 7fc62b424b80 1 -- 172.21.15.70:0/1009879535 shutdown_connections 2024-06-04T05:01:24.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.465+0000 7fc62b424b80 1 -- 172.21.15.70:0/1009879535 wait complete. 2024-06-04T05:01:24.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.465+0000 7fc62b424b80 20 client.16635 trim_cache size 0 max 16384 2024-06-04T05:01:24.475 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:01:24.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.466+0000 7fc620b8f700 1 -- 172.21.15.70:0/2729654176 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc6040b29f0 msgr2=0x7fc6040b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:24.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.466+0000 7fc620b8f700 1 --2- 172.21.15.70:0/2729654176 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc6040b29f0 0x7fc6040b4eb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc61c199e60 tx=0x7fc61000a380 comp rx=0 tx=0).stop 2024-06-04T05:01:24.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.466+0000 7fc620b8f700 1 -- 172.21.15.70:0/2729654176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c102440 msgr2=0x7fc61c19e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.466+0000 7fc620b8f700 1 --2- 172.21.15.70:0/2729654176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c102440 0x7fc61c19e5d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fc60c00c9b0 tx=0x7fc60c00ccc0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.466+0000 7fc620b8f700 1 -- 172.21.15.70:0/2729654176 shutdown_connections 2024-06-04T05:01:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.466+0000 7fc620b8f700 1 --2- 172.21.15.70:0/2729654176 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc6040b29f0 0x7fc6040b4eb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.466+0000 7fc620b8f700 1 --2- 172.21.15.70:0/2729654176 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc61c108520 0x7fc61c1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.466+0000 7fc620b8f700 1 --2- 172.21.15.70:0/2729654176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c102440 0x7fc61c19e5d0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.466+0000 7fc620b8f700 1 --2- 172.21.15.70:0/2729654176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc61c102e00 0x7fc61c19eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.466+0000 7fc620b8f700 1 -- 172.21.15.70:0/2729654176 >> 172.21.15.70:0/2729654176 conn(0x7fc61c0793f0 msgr2=0x7fc61c1003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.466+0000 7fc620b8f700 1 -- 172.21.15.70:0/2729654176 shutdown_connections 2024-06-04T05:01:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:24.466+0000 7fc620b8f700 1 -- 172.21.15.70:0/2729654176 wait complete. 2024-06-04T05:01:24.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 17 sec latency 30161.130 ms 2024-06-04T05:01:25.177 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:01:25.202 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:01:25.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:25 smithi070 ceph-mon[32631]: pgmap v130: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 11 op/s 2024-06-04T05:01:25.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:25 smithi112 ceph-mon[36453]: pgmap v130: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 11 op/s 2024-06-04T05:01:25.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:25 smithi098 ceph-mon[35569]: pgmap v130: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 11 op/s 2024-06-04T05:01:25.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 18 sec latency 31161.177 ms 2024-06-04T05:01:26.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:26 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:01:26.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:26 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:01:26.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:26 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:01:26.682 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:01:26.718 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45493 0.00 MB/sec execute 19 sec latency 32161.201 ms 2024-06-04T05:01:26.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.939+0000 7f312c954700 1 -- 172.21.15.70:0/3059839963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3128108860 msgr2=0x7f3128110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:26.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.939+0000 7f312c954700 1 --2- 172.21.15.70:0/3059839963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3128108860 0x7f3128110470 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f312000b0d0 tx=0x7f312000b3e0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.939+0000 7f312c954700 1 -- 172.21.15.70:0/3059839963 shutdown_connections 2024-06-04T05:01:26.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.939+0000 7f312c954700 1 --2- 172.21.15.70:0/3059839963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3128108860 0x7f3128110470 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.939+0000 7f312c954700 1 --2- 172.21.15.70:0/3059839963 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f312806bf10 0x7f312806c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.939+0000 7f312c954700 1 --2- 172.21.15.70:0/3059839963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f312806c8c0 0x7f3128108180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.939+0000 7f312c954700 1 -- 172.21.15.70:0/3059839963 >> 172.21.15.70:0/3059839963 conn(0x7f3128078490 msgr2=0x7f31280788a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:26.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.939+0000 7f312c954700 1 -- 172.21.15.70:0/3059839963 shutdown_connections 2024-06-04T05:01:26.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.939+0000 7f312c954700 1 -- 172.21.15.70:0/3059839963 wait complete. 2024-06-04T05:01:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.940+0000 7f312c954700 1 Processor -- start 2024-06-04T05:01:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.940+0000 7f312c954700 1 -- start start 2024-06-04T05:01:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.940+0000 7f312c954700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f312806bf10 0x7f312819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.940+0000 7f312c954700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f312806c8c0 0x7f312819fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.940+0000 7f312c954700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3128108860 0x7f3128199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.940+0000 7f312c954700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3128113640 con 0x7f312806bf10 2024-06-04T05:01:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.940+0000 7f312c954700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f31281134c0 con 0x7f3128108860 2024-06-04T05:01:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.940+0000 7f312c954700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f31281137c0 con 0x7f312806c8c0 2024-06-04T05:01:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.940+0000 7f312659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f312806bf10 0x7f312819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.940+0000 7f312659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f312806bf10 0x7f312819f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59648/0 (socket says 172.21.15.70:59648) 2024-06-04T05:01:26.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.940+0000 7f312659c700 1 -- 172.21.15.70:0/1806114456 learned_addr learned my addr 172.21.15.70:0/1806114456 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:26.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.940+0000 7f311dd9b700 1 --2- 172.21.15.70:0/1806114456 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f312806c8c0 0x7f312819fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.941+0000 7f3126d9d700 1 --2- 172.21.15.70:0/1806114456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3128108860 0x7f3128199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.941+0000 7f312659c700 1 -- 172.21.15.70:0/1806114456 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f312806c8c0 msgr2=0x7f312819fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:26.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.941+0000 7f312659c700 1 --2- 172.21.15.70:0/1806114456 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f312806c8c0 0x7f312819fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.941+0000 7f312659c700 1 -- 172.21.15.70:0/1806114456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3128108860 msgr2=0x7f3128199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:26.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.941+0000 7f312659c700 1 --2- 172.21.15.70:0/1806114456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3128108860 0x7f3128199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.941+0000 7f312659c700 1 -- 172.21.15.70:0/1806114456 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f312819a080 con 0x7f312806bf10 2024-06-04T05:01:26.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.941+0000 7f312659c700 1 --2- 172.21.15.70:0/1806114456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f312806bf10 0x7f312819f760 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f311800d980 tx=0x7f311800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:26.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.941+0000 7f311ffff700 1 -- 172.21.15.70:0/1806114456 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3118017070 con 0x7f312806bf10 2024-06-04T05:01:26.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.941+0000 7f312c954700 1 -- 172.21.15.70:0/1806114456 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f312819a370 con 0x7f312806bf10 2024-06-04T05:01:26.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.941+0000 7f311ffff700 1 -- 172.21.15.70:0/1806114456 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3118005020 con 0x7f312806bf10 2024-06-04T05:01:26.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.941+0000 7f311ffff700 1 -- 172.21.15.70:0/1806114456 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311800f980 con 0x7f312806bf10 2024-06-04T05:01:26.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.941+0000 7f312c954700 1 -- 172.21.15.70:0/1806114456 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31281a6630 con 0x7f312806bf10 2024-06-04T05:01:26.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.944+0000 7f311ffff700 1 -- 172.21.15.70:0/1806114456 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3118004a30 con 0x7f312806bf10 2024-06-04T05:01:26.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.945+0000 7f311ffff700 1 --2- 172.21.15.70:0/1806114456 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f31140b29f0 0x7f31140b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.945+0000 7f311dd9b700 1 --2- 172.21.15.70:0/1806114456 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f31140b29f0 0x7f31140b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.945+0000 7f311ffff700 1 -- 172.21.15.70:0/1806114456 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f31180e2490 con 0x7f312806bf10 2024-06-04T05:01:26.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.945+0000 7f31371e9b80 1 Processor -- start 2024-06-04T05:01:26.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f31371e9b80 1 -- start start 2024-06-04T05:01:26.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f311dd9b700 1 --2- 172.21.15.70:0/1806114456 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f31140b29f0 0x7f31140b4eb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3128101010 tx=0x7f311000b410 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:26.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f31371e9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2bd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f31371e9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2b9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f31371e9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f31371e9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ba8d213d90 con 0x55ba8d2b96b0 2024-06-04T05:01:26.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f31371e9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ba8d203930 con 0x55ba8d2bd5c0 2024-06-04T05:01:26.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f31371e9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ba8d207970 con 0x55ba8d2b8cf0 2024-06-04T05:01:26.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f3126d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f3126d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2c54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59662/0 (socket says 172.21.15.70:59662) 2024-06-04T05:01:26.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f3126d9d700 1 -- 172.21.15.70:0/2753725613 learned_addr learned my addr 172.21.15.70:0/2753725613 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:26.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f312659c700 1 --2- 172.21.15.70:0/2753725613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2bd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f311dd9b700 1 --2- 172.21.15.70:0/2753725613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2b9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f312659c700 1 -- 172.21.15.70:0/2753725613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b8cf0 msgr2=0x55ba8d2b9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f312659c700 1 --2- 172.21.15.70:0/2753725613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2b9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f312659c700 1 -- 172.21.15.70:0/2753725613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2b96b0 msgr2=0x55ba8d2c54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f312659c700 1 --2- 172.21.15.70:0/2753725613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2c54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f312659c700 1 -- 172.21.15.70:0/2753725613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba8d1539d0 con 0x55ba8d2bd5c0 2024-06-04T05:01:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.946+0000 7f3126d9d700 1 --2- 172.21.15.70:0/2753725613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.947+0000 7f312659c700 1 --2- 172.21.15.70:0/2753725613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2bd9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f31180dfbe0 tx=0x7f3118004da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=354ce4964133d0fa server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.947+0000 7f311cd99700 1 -- 172.21.15.70:0/2753725613 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31180a9e70 con 0x55ba8d2bd5c0 2024-06-04T05:01:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.947+0000 7f311cd99700 1 -- 172.21.15.70:0/2753725613 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f31180a9770 con 0x55ba8d2bd5c0 2024-06-04T05:01:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.947+0000 7f311cd99700 1 -- 172.21.15.70:0/2753725613 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31180a9c30 con 0x55ba8d2bd5c0 2024-06-04T05:01:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.947+0000 7f31371e9b80 1 -- 172.21.15.70:0/2753725613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2bd5c0 msgr2=0x55ba8d2bd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.947+0000 7f31371e9b80 1 --2- 172.21.15.70:0/2753725613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2bd9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f31180dfbe0 tx=0x7f3118004da0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.947+0000 7f31371e9b80 1 -- 172.21.15.70:0/2753725613 shutdown_connections 2024-06-04T05:01:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.947+0000 7f31371e9b80 1 --2- 172.21.15.70:0/2753725613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.947+0000 7f31371e9b80 1 --2- 172.21.15.70:0/2753725613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2bd9a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.948+0000 7f31371e9b80 1 --2- 172.21.15.70:0/2753725613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2b9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.948+0000 7f31371e9b80 1 -- 172.21.15.70:0/2753725613 >> 172.21.15.70:0/2753725613 conn(0x55ba8d1e2180 msgr2=0x55ba8d2af1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:26.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.948+0000 7f31371e9b80 1 -- 172.21.15.70:0/2753725613 shutdown_connections 2024-06-04T05:01:26.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.948+0000 7f31371e9b80 1 -- 172.21.15.70:0/2753725613 wait complete. 2024-06-04T05:01:26.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.948+0000 7f31371e9b80 1 -- start start 2024-06-04T05:01:26.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.948+0000 7f31371e9b80 1 Processor -- start 2024-06-04T05:01:26.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f31371e9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2c5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f31371e9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2c2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f31371e9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2c2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f31371e9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ba8d10a0a0 con 0x55ba8d2bd5c0 2024-06-04T05:01:26.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f31371e9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ba8d1a58c0 con 0x55ba8d2b8cf0 2024-06-04T05:01:26.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f31371e9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ba8d2c5b40 con 0x55ba8d2b96b0 2024-06-04T05:01:26.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f31371e9b80 10 client.0 _ll_register_callbacks cb 0x55ba8d19ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:26.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f312659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f312659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2c5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:41316/0 (socket says 172.21.15.70:41316) 2024-06-04T05:01:26.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f312659c700 1 -- 172.21.15.70:0/1628529467 learned_addr learned my addr 172.21.15.70:0/1628529467 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:26.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f3126d9d700 1 --2- 172.21.15.70:0/1628529467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2c2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f311dd9b700 1 --2- 172.21.15.70:0/1628529467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2c2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f312659c700 1 -- 172.21.15.70:0/1628529467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b96b0 msgr2=0x55ba8d2c2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:26.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f312659c700 1 --2- 172.21.15.70:0/1628529467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2c2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f312659c700 1 -- 172.21.15.70:0/1628529467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2bd5c0 msgr2=0x55ba8d2c2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:26.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f312659c700 1 --2- 172.21.15.70:0/1628529467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2c2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f312659c700 1 -- 172.21.15.70:0/1628529467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba8d1a2790 con 0x55ba8d2b8cf0 2024-06-04T05:01:26.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f312659c700 1 --2- 172.21.15.70:0/1628529467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2c5250 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f311800d950 tx=0x7f31180057a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97f4460d6c778fd0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:26.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f310dffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:01:26.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.949+0000 7f310dffb700 1 -- 172.21.15.70:0/1628529467 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31180b25d0 con 0x55ba8d2b8cf0 2024-06-04T05:01:26.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.950+0000 7f31371e9b80 1 -- 172.21.15.70:0/1628529467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31180a9770 con 0x55ba8d2b8cf0 2024-06-04T05:01:26.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.950+0000 7f310dffb700 1 -- 172.21.15.70:0/1628529467 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f31180a9be0 con 0x55ba8d2b8cf0 2024-06-04T05:01:26.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.950+0000 7f310dffb700 1 -- 172.21.15.70:0/1628529467 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f31180a9e70 con 0x55ba8d2b8cf0 2024-06-04T05:01:26.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.950+0000 7f31371e9b80 10 client.25964 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:26.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.950+0000 7f31371e9b80 1 -- 172.21.15.70:0/1628529467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ba8d2d0340 con 0x55ba8d2b8cf0 2024-06-04T05:01:26.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.950+0000 7f310dffb700 1 -- 172.21.15.70:0/1628529467 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f31180a9770 con 0x55ba8d2b8cf0 2024-06-04T05:01:26.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.950+0000 7f310dffb700 1 -- 172.21.15.70:0/1628529467 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f31180a52a0 con 0x55ba8d2b8cf0 2024-06-04T05:01:26.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.950+0000 7f31371e9b80 10 client.25964 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:26.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.950+0000 7f31371e9b80 10 client.25964 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:01:26.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.950+0000 7f31371e9b80 20 client.25964 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.950+0000 7f31371e9b80 1 --2- 172.21.15.70:0/1628529467 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ba8d2d2810 0x55ba8d2d4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.950+0000 7f31371e9b80 4 client.25964 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:01:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.950+0000 7f31371e9b80 1 -- 172.21.15.70:0/1628529467 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba8d280ae0 con 0x55ba8d2d2810 2024-06-04T05:01:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.950+0000 7f311dd9b700 1 --2- 172.21.15.70:0/1628529467 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ba8d2d2810 0x55ba8d2d4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.951+0000 7f311dd9b700 1 --2- 172.21.15.70:0/1628529467 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ba8d2d2810 0x55ba8d2d4c00 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1855117e908b24ad server_cookie=65f7b554bb6658c3 in_seq=0 out_seq=0 2024-06-04T05:01:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.951+0000 7f310dffb700 10 client.25964 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.951+0000 7f310dffb700 1 -- 172.21.15.70:0/1628529467 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ba8d280ae0 con 0x55ba8d2d2810 2024-06-04T05:01:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.951+0000 7f310dffb700 10 client.25964 handle_command_reply: tid=0 2024-06-04T05:01:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.951+0000 7f31371e9b80 1 client.25964 shutdown 2024-06-04T05:01:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.952+0000 7f31371e9b80 1 -- 172.21.15.70:0/1628529467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2b8cf0 msgr2=0x55ba8d2c5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.952+0000 7f31371e9b80 1 --2- 172.21.15.70:0/1628529467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2c5250 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f311800d950 tx=0x7f31180057a0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.952+0000 7f31371e9b80 1 -- 172.21.15.70:0/1628529467 shutdown_connections 2024-06-04T05:01:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.952+0000 7f31371e9b80 1 --2- 172.21.15.70:0/1628529467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2c2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.952+0000 7f31371e9b80 1 --2- 172.21.15.70:0/1628529467 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ba8d2d2810 0x55ba8d2d4c00 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.952+0000 7f31371e9b80 1 --2- 172.21.15.70:0/1628529467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2c5250 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.952+0000 7f31371e9b80 1 --2- 172.21.15.70:0/1628529467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2c2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.952+0000 7f31371e9b80 1 -- 172.21.15.70:0/1628529467 >> 172.21.15.70:0/1628529467 conn(0x55ba8d1e2180 msgr2=0x55ba8d2bec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.952+0000 7f310dffb700 0 client.25964 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.952+0000 7f31371e9b80 1 -- 172.21.15.70:0/1628529467 shutdown_connections 2024-06-04T05:01:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.952+0000 7f31371e9b80 1 -- 172.21.15.70:0/1628529467 wait complete. 2024-06-04T05:01:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.952+0000 7f31371e9b80 20 client.25964 trim_cache size 0 max 16384 2024-06-04T05:01:26.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.971+0000 7f31371e9b80 1 Processor -- start 2024-06-04T05:01:26.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.971+0000 7f31371e9b80 1 -- start start 2024-06-04T05:01:26.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f31371e9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f31371e9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f31371e9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2d7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f31371e9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f31281abb20 con 0x55ba8d2b96b0 2024-06-04T05:01:26.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f31371e9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f31281ad6c0 con 0x55ba8d2bd5c0 2024-06-04T05:01:26.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f31371e9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f31281ac1f0 con 0x55ba8d2b8cf0 2024-06-04T05:01:26.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f311dd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f311dd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2c4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59670/0 (socket says 172.21.15.70:59670) 2024-06-04T05:01:26.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f311dd9b700 1 -- 172.21.15.70:0/130110696 learned_addr learned my addr 172.21.15.70:0/130110696 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:26.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f3126d9d700 1 --2- 172.21.15.70:0/130110696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2d7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f312659c700 1 --2- 172.21.15.70:0/130110696 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f311dd9b700 1 -- 172.21.15.70:0/130110696 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b8cf0 msgr2=0x55ba8d2b43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:26.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f311dd9b700 1 --2- 172.21.15.70:0/130110696 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2b43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f311dd9b700 1 -- 172.21.15.70:0/130110696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2bd5c0 msgr2=0x55ba8d2d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:26.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f311dd9b700 1 --2- 172.21.15.70:0/130110696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2d7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f311dd9b700 1 -- 172.21.15.70:0/130110696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba8d2c04a0 con 0x55ba8d2b96b0 2024-06-04T05:01:26.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f311dd9b700 1 --2- 172.21.15.70:0/130110696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2c4d90 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3110005950 tx=0x7f3110013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45a9ba9cfd8f9670 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:26.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f310ffff700 1 -- 172.21.15.70:0/130110696 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110011be0 con 0x55ba8d2b96b0 2024-06-04T05:01:26.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f310ffff700 1 -- 172.21.15.70:0/130110696 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f311002f070 con 0x55ba8d2b96b0 2024-06-04T05:01:26.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.972+0000 7f310ffff700 1 -- 172.21.15.70:0/130110696 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311001f690 con 0x55ba8d2b96b0 2024-06-04T05:01:26.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.973+0000 7f31371e9b80 1 -- 172.21.15.70:0/130110696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2b96b0 msgr2=0x55ba8d2c4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:26.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.973+0000 7f31371e9b80 1 --2- 172.21.15.70:0/130110696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2c4d90 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3110005950 tx=0x7f3110013000 comp rx=0 tx=0).stop 2024-06-04T05:01:26.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.973+0000 7f31371e9b80 1 -- 172.21.15.70:0/130110696 shutdown_connections 2024-06-04T05:01:26.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.973+0000 7f31371e9b80 1 --2- 172.21.15.70:0/130110696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2d7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.973+0000 7f31371e9b80 1 --2- 172.21.15.70:0/130110696 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2b43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.973+0000 7f31371e9b80 1 --2- 172.21.15.70:0/130110696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2c4d90 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.973+0000 7f31371e9b80 1 -- 172.21.15.70:0/130110696 >> 172.21.15.70:0/130110696 conn(0x55ba8d1e2180 msgr2=0x55ba8d2c15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:26.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.973+0000 7f31371e9b80 1 -- 172.21.15.70:0/130110696 shutdown_connections 2024-06-04T05:01:26.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.973+0000 7f31371e9b80 1 -- 172.21.15.70:0/130110696 wait complete. 2024-06-04T05:01:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.973+0000 7f31371e9b80 1 -- start start 2024-06-04T05:01:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f31371e9b80 1 Processor -- start 2024-06-04T05:01:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f31371e9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2d8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f31371e9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2d6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f31371e9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2d6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f31371e9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f31281acf90 con 0x55ba8d2bd5c0 2024-06-04T05:01:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f31371e9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f31281ac8c0 con 0x55ba8d2b96b0 2024-06-04T05:01:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f31371e9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f31281a9ac0 con 0x55ba8d2b8cf0 2024-06-04T05:01:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f31371e9b80 10 client.0 _ll_register_callbacks cb 0x55ba8d19ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f3126d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2d6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f311dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2d6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f311dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2d6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:41324/0 (socket says 172.21.15.70:41324) 2024-06-04T05:01:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f3126d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2d6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59682/0 (socket says 172.21.15.70:59682) 2024-06-04T05:01:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f311dd9b700 1 -- 172.21.15.70:0/1752447162 learned_addr learned my addr 172.21.15.70:0/1752447162 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f312659c700 1 --2- 172.21.15.70:0/1752447162 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2d8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f311dd9b700 1 -- 172.21.15.70:0/1752447162 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b8cf0 msgr2=0x55ba8d2d8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f311dd9b700 1 --2- 172.21.15.70:0/1752447162 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2d8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f311dd9b700 1 -- 172.21.15.70:0/1752447162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2bd5c0 msgr2=0x55ba8d2d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f311dd9b700 1 --2- 172.21.15.70:0/1752447162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2d6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f311dd9b700 1 -- 172.21.15.70:0/1752447162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba8d2d73b0 con 0x55ba8d2b96b0 2024-06-04T05:01:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f3126d9d700 1 --2- 172.21.15.70:0/1752447162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2d6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.974+0000 7f312659c700 1 --2- 172.21.15.70:0/1752447162 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2d8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.975+0000 7f311dd9b700 1 --2- 172.21.15.70:0/1752447162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2d6630 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f311002e000 tx=0x7f3110012a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48800e0efc370e4d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.975+0000 7f310dffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:01:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.975+0000 7f310dffb700 1 -- 172.21.15.70:0/1752447162 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110038070 con 0x55ba8d2b96b0 2024-06-04T05:01:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.975+0000 7f31371e9b80 1 -- 172.21.15.70:0/1752447162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f311002f070 con 0x55ba8d2b96b0 2024-06-04T05:01:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.975+0000 7f310dffb700 1 -- 172.21.15.70:0/1752447162 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f3110005240 con 0x55ba8d2b96b0 2024-06-04T05:01:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.975+0000 7f310dffb700 1 -- 172.21.15.70:0/1752447162 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3110005480 con 0x55ba8d2b96b0 2024-06-04T05:01:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.975+0000 7f31371e9b80 10 client.25970 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.975+0000 7f31371e9b80 1 -- 172.21.15.70:0/1752447162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ba8d33e030 con 0x55ba8d2b96b0 2024-06-04T05:01:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.975+0000 7f310dffb700 1 -- 172.21.15.70:0/1752447162 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f311002f070 con 0x55ba8d2b96b0 2024-06-04T05:01:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.975+0000 7f310dffb700 1 -- 172.21.15.70:0/1752447162 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f311002a560 con 0x55ba8d2b96b0 2024-06-04T05:01:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.975+0000 7f31371e9b80 10 client.25970 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.975+0000 7f31371e9b80 10 client.25970 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:01:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.975+0000 7f31371e9b80 20 client.25970 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.975+0000 7f31371e9b80 1 --2- 172.21.15.70:0/1752447162 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ba8d33e770 0x55ba8d340b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.975+0000 7f31371e9b80 4 client.25970 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "3ea23d63", "release": true, "await": true}] 2024-06-04T05:01:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.975+0000 7f31371e9b80 1 -- 172.21.15.70:0/1752447162 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "3ea23d63", "release": true, "await": true}) -- 0x55ba8d215ce0 con 0x55ba8d33e770 2024-06-04T05:01:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.976+0000 7f312659c700 1 --2- 172.21.15.70:0/1752447162 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ba8d33e770 0x55ba8d340b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.976+0000 7f310dffb700 10 client.25970 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.976+0000 7f312659c700 1 --2- 172.21.15.70:0/1752447162 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ba8d33e770 0x55ba8d340b90 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd274659c65d333b server_cookie=eb0c4490b8a58681 in_seq=0 out_seq=0 2024-06-04T05:01:26.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f310dffb700 1 -- 172.21.15.70:0/1752447162 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55ba8d215ce0 con 0x55ba8d33e770 2024-06-04T05:01:26.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f310dffb700 10 client.25970 handle_command_reply: tid=0 2024-06-04T05:01:26.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f31371e9b80 1 client.25970 shutdown 2024-06-04T05:01:26.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f31371e9b80 1 -- 172.21.15.70:0/1752447162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2b96b0 msgr2=0x55ba8d2d6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:26.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f31371e9b80 1 --2- 172.21.15.70:0/1752447162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2d6630 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f311002e000 tx=0x7f3110012a70 comp rx=0 tx=0).stop 2024-06-04T05:01:26.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f31371e9b80 1 -- 172.21.15.70:0/1752447162 shutdown_connections 2024-06-04T05:01:26.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f31371e9b80 1 --2- 172.21.15.70:0/1752447162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ba8d2bd5c0 0x55ba8d2d6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f31371e9b80 1 --2- 172.21.15.70:0/1752447162 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ba8d2b8cf0 0x55ba8d2d8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f31371e9b80 1 --2- 172.21.15.70:0/1752447162 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ba8d33e770 0x55ba8d340b90 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f31371e9b80 1 --2- 172.21.15.70:0/1752447162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba8d2b96b0 0x55ba8d2d6630 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f31371e9b80 1 -- 172.21.15.70:0/1752447162 >> 172.21.15.70:0/1752447162 conn(0x55ba8d1e2180 msgr2=0x55ba8d2c1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:26.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f310dffb700 0 client.25970 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:26.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f31371e9b80 1 -- 172.21.15.70:0/1752447162 shutdown_connections 2024-06-04T05:01:26.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f31371e9b80 1 -- 172.21.15.70:0/1752447162 wait complete. 2024-06-04T05:01:26.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f31371e9b80 20 client.25970 trim_cache size 0 max 16384 2024-06-04T05:01:26.986 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:01:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 31, 2024-06-04T05:01:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:01:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 12, 2024-06-04T05:01:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:01:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: "3ea23d63": { 2024-06-04T05:01:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 12, 2024-06-04T05:01:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:01:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:01:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:01:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:01:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:01:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:01:26.987 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 149.1, 2024-06-04T05:01:26.987 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:01:26.987 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:01:26.987 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:01:26.987 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:01:26.987 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:01:26.987 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:01:26.987 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:01:26.987 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:01:26.987 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:01:26.987 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:01:26.987 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:01:26.987 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:01:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f312c954700 1 -- 172.21.15.70:0/1806114456 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f31140b29f0 msgr2=0x7f31140b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f312c954700 1 --2- 172.21.15.70:0/1806114456 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f31140b29f0 0x7f31140b4eb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3128101010 tx=0x7f311000b410 comp rx=0 tx=0).stop 2024-06-04T05:01:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f312c954700 1 -- 172.21.15.70:0/1806114456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f312806bf10 msgr2=0x7f312819f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f312c954700 1 --2- 172.21.15.70:0/1806114456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f312806bf10 0x7f312819f760 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f311800d980 tx=0x7f311800dc90 comp rx=0 tx=0).stop 2024-06-04T05:01:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f312c954700 1 -- 172.21.15.70:0/1806114456 shutdown_connections 2024-06-04T05:01:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f312c954700 1 --2- 172.21.15.70:0/1806114456 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f31140b29f0 0x7f31140b4eb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f312c954700 1 --2- 172.21.15.70:0/1806114456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3128108860 0x7f3128199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.982+0000 7f312c954700 1 --2- 172.21.15.70:0/1806114456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f312806bf10 0x7f312819f760 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.983+0000 7f312c954700 1 --2- 172.21.15.70:0/1806114456 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f312806c8c0 0x7f312819fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.983+0000 7f312c954700 1 -- 172.21.15.70:0/1806114456 >> 172.21.15.70:0/1806114456 conn(0x7f3128078490 msgr2=0x7f3128109a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.983+0000 7f312c954700 1 -- 172.21.15.70:0/1806114456 shutdown_connections 2024-06-04T05:01:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:26.983+0000 7f312c954700 1 -- 172.21.15.70:0/1806114456 wait complete. 2024-06-04T05:01:27.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:27 smithi070 ceph-mon[32631]: pgmap v131: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.5 KiB/s wr, 15 op/s 2024-06-04T05:01:27.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:27 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:01:27.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:27 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:01:27.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:27 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:01:27.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:27 smithi112 ceph-mon[36453]: pgmap v131: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.5 KiB/s wr, 15 op/s 2024-06-04T05:01:27.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:27 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:01:27.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:27 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:01:27.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:27 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:01:27.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:27 smithi098 ceph-mon[35569]: pgmap v131: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.5 KiB/s wr, 15 op/s 2024-06-04T05:01:27.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:27 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:01:27.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:27 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:01:27.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:27 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:01:27.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 45913 0.13 MB/sec execute 20 sec latency 32430.776 ms 2024-06-04T05:01:27.861 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '3ea23d63', total seconds elapsed: 38.9 2024-06-04T05:01:28.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 46389 0.33 MB/sec execute 21 sec latency 14.758 ms 2024-06-04T05:01:29.265 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:29 smithi070 ceph-mon[32631]: pgmap v132: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 11 op/s 2024-06-04T05:01:29.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:29 smithi112 ceph-mon[36453]: pgmap v132: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 11 op/s 2024-06-04T05:01:29.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:29 smithi098 ceph-mon[35569]: pgmap v132: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 11 op/s 2024-06-04T05:01:29.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.531+0000 7f7d1440e700 1 -- 172.21.15.70:0/1816482198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d0c108520 msgr2=0x7f7d0c108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:29.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.531+0000 7f7d1440e700 1 --2- 172.21.15.70:0/1816482198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d0c108520 0x7f7d0c108900 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f7d08009990 tx=0x7f7d08009ca0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.531+0000 7f7d1440e700 1 -- 172.21.15.70:0/1816482198 shutdown_connections 2024-06-04T05:01:29.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.531+0000 7f7d1440e700 1 --2- 172.21.15.70:0/1816482198 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d0c102e00 0x7f7d0c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.531+0000 7f7d1440e700 1 --2- 172.21.15.70:0/1816482198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d0c108520 0x7f7d0c108900 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.531+0000 7f7d1440e700 1 --2- 172.21.15.70:0/1816482198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d0c102440 0x7f7d0c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.531+0000 7f7d1440e700 1 -- 172.21.15.70:0/1816482198 >> 172.21.15.70:0/1816482198 conn(0x7f7d0c0793f0 msgr2=0x7f7d0c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:29.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.531+0000 7f7d1440e700 1 -- 172.21.15.70:0/1816482198 shutdown_connections 2024-06-04T05:01:29.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.531+0000 7f7d1440e700 1 -- 172.21.15.70:0/1816482198 wait complete. 2024-06-04T05:01:29.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.532+0000 7f7d1440e700 1 Processor -- start 2024-06-04T05:01:29.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.532+0000 7f7d1440e700 1 -- start start 2024-06-04T05:01:29.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.532+0000 7f7d1440e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d0c102440 0x7f7d0c19a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.532+0000 7f7d1440e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d0c102e00 0x7f7d0c19ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.532+0000 7f7d1440e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d0c108520 0x7f7d0c1a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.532+0000 7f7d1440e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c1124a0 con 0x7f7d0c102440 2024-06-04T05:01:29.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.532+0000 7f7d1440e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c112320 con 0x7f7d0c108520 2024-06-04T05:01:29.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.532+0000 7f7d1440e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c112620 con 0x7f7d0c102e00 2024-06-04T05:01:29.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.532+0000 7f7d121aa700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d0c102440 0x7f7d0c19a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.532+0000 7f7d121aa700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d0c102440 0x7f7d0c19a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59706/0 (socket says 172.21.15.70:59706) 2024-06-04T05:01:29.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.532+0000 7f7d121aa700 1 -- 172.21.15.70:0/2944866884 learned_addr learned my addr 172.21.15.70:0/2944866884 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:29.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.532+0000 7f7d119a9700 1 --2- 172.21.15.70:0/2944866884 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d0c102e00 0x7f7d0c19ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.532+0000 7f7d129ab700 1 --2- 172.21.15.70:0/2944866884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d0c108520 0x7f7d0c1a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.533+0000 7f7d121aa700 1 -- 172.21.15.70:0/2944866884 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d0c102e00 msgr2=0x7f7d0c19ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:29.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.533+0000 7f7d121aa700 1 --2- 172.21.15.70:0/2944866884 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d0c102e00 0x7f7d0c19ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.533+0000 7f7d121aa700 1 -- 172.21.15.70:0/2944866884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d0c108520 msgr2=0x7f7d0c1a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:29.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.533+0000 7f7d121aa700 1 --2- 172.21.15.70:0/2944866884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d0c108520 0x7f7d0c1a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.533+0000 7f7d121aa700 1 -- 172.21.15.70:0/2944866884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d0c1a5ee0 con 0x7f7d0c102440 2024-06-04T05:01:29.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.533+0000 7f7d121aa700 1 --2- 172.21.15.70:0/2944866884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d0c102440 0x7f7d0c19a920 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f7d08006010 tx=0x7f7d080047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:29.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.533+0000 7f7d037fe700 1 -- 172.21.15.70:0/2944866884 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08027070 con 0x7f7d0c102440 2024-06-04T05:01:29.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.533+0000 7f7d037fe700 1 -- 172.21.15.70:0/2944866884 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d08022470 con 0x7f7d0c102440 2024-06-04T05:01:29.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.533+0000 7f7d037fe700 1 -- 172.21.15.70:0/2944866884 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08017650 con 0x7f7d0c102440 2024-06-04T05:01:29.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.533+0000 7f7d1440e700 1 -- 172.21.15.70:0/2944866884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d0c1a60b0 con 0x7f7d0c102440 2024-06-04T05:01:29.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.533+0000 7f7d1440e700 1 -- 172.21.15.70:0/2944866884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d0c1a64b0 con 0x7f7d0c102440 2024-06-04T05:01:29.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.536+0000 7f7d037fe700 1 -- 172.21.15.70:0/2944866884 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7d08009d10 con 0x7f7d0c102440 2024-06-04T05:01:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.536+0000 7f7d037fe700 1 --2- 172.21.15.70:0/2944866884 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7cf80b2a70 0x7f7cf80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.536+0000 7f7d119a9700 1 --2- 172.21.15.70:0/2944866884 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7cf80b2a70 0x7f7cf80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.537+0000 7f7d037fe700 1 -- 172.21.15.70:0/2944866884 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f7d080e2fd0 con 0x7f7d0c102440 2024-06-04T05:01:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.537+0000 7f7d119a9700 1 --2- 172.21.15.70:0/2944866884 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7cf80b2a70 0x7f7cf80b4f30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c079b60 tx=0x7f7cfc009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.537+0000 7f7d1eca3b80 1 Processor -- start 2024-06-04T05:01:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d1eca3b80 1 -- start start 2024-06-04T05:01:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d1eca3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b54c70 0x55a1c7b55050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d1eca3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b55620 0x55a1c7b58d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d1eca3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b59370 0x55a1c7b597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d1eca3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a1c7aafc90 con 0x55a1c7b59370 2024-06-04T05:01:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d1eca3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a1c7a9f830 con 0x55a1c7b55620 2024-06-04T05:01:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d1eca3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a1c7aa3870 con 0x55a1c7b54c70 2024-06-04T05:01:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d119a9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b55620 0x55a1c7b58d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d121aa700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b54c70 0x55a1c7b55050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d129ab700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b59370 0x55a1c7b597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d119a9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b55620 0x55a1c7b58d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:41350/0 (socket says 172.21.15.70:41350) 2024-06-04T05:01:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d121aa700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b54c70 0x55a1c7b55050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:58168/0 (socket says 172.21.15.70:58168) 2024-06-04T05:01:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d119a9700 1 -- 172.21.15.70:0/3482103404 learned_addr learned my addr 172.21.15.70:0/3482103404 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d119a9700 1 -- 172.21.15.70:0/3482103404 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b54c70 msgr2=0x55a1c7b55050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d119a9700 1 --2- 172.21.15.70:0/3482103404 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b54c70 0x55a1c7b55050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d119a9700 1 -- 172.21.15.70:0/3482103404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b59370 msgr2=0x55a1c7b597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d119a9700 1 --2- 172.21.15.70:0/3482103404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b59370 0x55a1c7b597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d119a9700 1 -- 172.21.15.70:0/3482103404 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1c79ef8d0 con 0x55a1c7b55620 2024-06-04T05:01:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d121aa700 1 --2- 172.21.15.70:0/3482103404 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b54c70 0x55a1c7b55050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.538+0000 7f7d129ab700 1 --2- 172.21.15.70:0/3482103404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b59370 0x55a1c7b597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.539+0000 7f7d119a9700 1 --2- 172.21.15.70:0/3482103404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b55620 0x55a1c7b58d50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc011d20 tx=0x7f7cfc014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=119206914d92ae3c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.539+0000 7f7d00d70700 1 -- 172.21.15.70:0/3482103404 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc00eb20 con 0x55a1c7b55620 2024-06-04T05:01:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.539+0000 7f7d00d70700 1 -- 172.21.15.70:0/3482103404 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cfc00ecc0 con 0x55a1c7b55620 2024-06-04T05:01:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.539+0000 7f7d00d70700 1 -- 172.21.15.70:0/3482103404 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc01d670 con 0x55a1c7b55620 2024-06-04T05:01:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.540+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/3482103404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b55620 msgr2=0x55a1c7b58d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.540+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/3482103404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b55620 0x55a1c7b58d50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc011d20 tx=0x7f7cfc014000 comp rx=0 tx=0).stop 2024-06-04T05:01:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.540+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/3482103404 shutdown_connections 2024-06-04T05:01:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.540+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/3482103404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b59370 0x55a1c7b597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.540+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/3482103404 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b54c70 0x55a1c7b55050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.540+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/3482103404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b55620 0x55a1c7b58d50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.540+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/3482103404 >> 172.21.15.70:0/3482103404 conn(0x55a1c7a7e080 msgr2=0x55a1c7b4ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:29.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.540+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/3482103404 shutdown_connections 2024-06-04T05:01:29.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.540+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/3482103404 wait complete. 2024-06-04T05:01:29.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.540+0000 7f7d1eca3b80 1 -- start start 2024-06-04T05:01:29.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.540+0000 7f7d1eca3b80 1 Processor -- start 2024-06-04T05:01:29.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d1eca3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b54c70 0x55a1c7b60790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d1eca3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b55620 0x55a1c7b5da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d1eca3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b59370 0x55a1c7b5dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d1eca3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a1c79a5fa0 con 0x55a1c7b54c70 2024-06-04T05:01:29.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d1eca3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a1c7a417c0 con 0x55a1c7b59370 2024-06-04T05:01:29.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d1eca3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a1c7b61530 con 0x55a1c7b55620 2024-06-04T05:01:29.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d1eca3b80 10 client.0 _ll_register_callbacks cb 0x55a1c7a3a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:29.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d119a9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b55620 0x55a1c7b5da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d119a9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b55620 0x55a1c7b5da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:58182/0 (socket says 172.21.15.70:58182) 2024-06-04T05:01:29.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d119a9700 1 -- 172.21.15.70:0/2020958287 learned_addr learned my addr 172.21.15.70:0/2020958287 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:29.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d121aa700 1 --2- 172.21.15.70:0/2020958287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b54c70 0x55a1c7b60790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d129ab700 1 --2- 172.21.15.70:0/2020958287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b59370 0x55a1c7b5dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d119a9700 1 -- 172.21.15.70:0/2020958287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b59370 msgr2=0x55a1c7b5dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:29.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d119a9700 1 --2- 172.21.15.70:0/2020958287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b59370 0x55a1c7b5dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d119a9700 1 -- 172.21.15.70:0/2020958287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b54c70 msgr2=0x55a1c7b60790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:29.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d119a9700 1 --2- 172.21.15.70:0/2020958287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b54c70 0x55a1c7b60790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d119a9700 1 -- 172.21.15.70:0/2020958287 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1c7a3e690 con 0x55a1c7b55620 2024-06-04T05:01:29.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d129ab700 1 --2- 172.21.15.70:0/2020958287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b59370 0x55a1c7b5dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:01:29.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d121aa700 1 --2- 172.21.15.70:0/2020958287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b54c70 0x55a1c7b60790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:29.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7d119a9700 1 --2- 172.21.15.70:0/2020958287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b55620 0x55a1c7b5da90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc011cf0 tx=0x7f7cfc00eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6f2585ae546e427 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:29.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.541+0000 7f7cf5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:01:29.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.542+0000 7f7cf5ffb700 1 -- 172.21.15.70:0/2020958287 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc01c620 con 0x55a1c7b55620 2024-06-04T05:01:29.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.542+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/2020958287 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cfc00ecc0 con 0x55a1c7b55620 2024-06-04T05:01:29.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.542+0000 7f7cf5ffb700 1 -- 172.21.15.70:0/2020958287 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f7cfc01c7c0 con 0x55a1c7b55620 2024-06-04T05:01:29.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.542+0000 7f7cf5ffb700 1 -- 172.21.15.70:0/2020958287 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7cfc03b050 con 0x55a1c7b55620 2024-06-04T05:01:29.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.542+0000 7f7d1eca3b80 10 client.26026 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:29.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.542+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/2020958287 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a1c7b6bc70 con 0x55a1c7b55620 2024-06-04T05:01:29.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.542+0000 7f7cf5ffb700 1 -- 172.21.15.70:0/2020958287 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cfc00ecc0 con 0x55a1c7b55620 2024-06-04T05:01:29.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.542+0000 7f7cf5ffb700 1 -- 172.21.15.70:0/2020958287 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f7cfc036030 con 0x55a1c7b55620 2024-06-04T05:01:29.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.542+0000 7f7d1eca3b80 10 client.26026 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:29.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.542+0000 7f7d1eca3b80 10 client.26026 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:01:29.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.542+0000 7f7d1eca3b80 20 client.26026 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:29.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.542+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/2020958287 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55a1c7b6e1a0 0x55a1c7b70590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.542+0000 7f7d1eca3b80 4 client.26026 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:01:29.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.542+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/2020958287 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1c7b1c9e0 con 0x55a1c7b6e1a0 2024-06-04T05:01:29.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.543+0000 7f7d121aa700 1 --2- 172.21.15.70:0/2020958287 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55a1c7b6e1a0 0x55a1c7b70590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.543+0000 7f7d121aa700 1 --2- 172.21.15.70:0/2020958287 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55a1c7b6e1a0 0x55a1c7b70590 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c409776282d7438c server_cookie=de99e9e31b307b16 in_seq=0 out_seq=0 2024-06-04T05:01:29.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.543+0000 7f7cf5ffb700 10 client.26026 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:01:29.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.543+0000 7f7cf5ffb700 1 -- 172.21.15.70:0/2020958287 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a1c7b1c9e0 con 0x55a1c7b6e1a0 2024-06-04T05:01:29.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.544+0000 7f7cf5ffb700 10 client.26026 handle_command_reply: tid=0 2024-06-04T05:01:29.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.544+0000 7f7d1eca3b80 1 client.26026 shutdown 2024-06-04T05:01:29.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.544+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/2020958287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b55620 msgr2=0x55a1c7b5da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:29.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.544+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/2020958287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b55620 0x55a1c7b5da90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc011cf0 tx=0x7f7cfc00eef0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.544+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/2020958287 shutdown_connections 2024-06-04T05:01:29.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.544+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/2020958287 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55a1c7b6e1a0 0x55a1c7b70590 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.544+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/2020958287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b59370 0x55a1c7b5dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.544+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/2020958287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b54c70 0x55a1c7b60790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.544+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/2020958287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b55620 0x55a1c7b5da90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.544+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/2020958287 >> 172.21.15.70:0/2020958287 conn(0x55a1c7a7e080 msgr2=0x55a1c7b5a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:29.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.544+0000 7f7cf5ffb700 0 client.26026 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:01:29.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.544+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/2020958287 shutdown_connections 2024-06-04T05:01:29.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.544+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/2020958287 wait complete. 2024-06-04T05:01:29.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.544+0000 7f7d1eca3b80 20 client.26026 trim_cache size 0 max 16384 2024-06-04T05:01:29.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.560+0000 7f7d1eca3b80 1 Processor -- start 2024-06-04T05:01:29.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.560+0000 7f7d1eca3b80 1 -- start start 2024-06-04T05:01:29.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.560+0000 7f7d1eca3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b54c70 0x55a1c7b5da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.560+0000 7f7d1eca3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b55620 0x55a1c7b5df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.560+0000 7f7d1eca3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b59370 0x55a1c7b731d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.560+0000 7f7d1eca3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c1aba80 con 0x55a1c7b59370 2024-06-04T05:01:29.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.560+0000 7f7d1eca3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c1ad620 con 0x55a1c7b55620 2024-06-04T05:01:29.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.560+0000 7f7d1eca3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c1ac150 con 0x55a1c7b54c70 2024-06-04T05:01:29.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.560+0000 7f7d129ab700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b59370 0x55a1c7b731d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.560+0000 7f7d129ab700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b59370 0x55a1c7b731d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59738/0 (socket says 172.21.15.70:59738) 2024-06-04T05:01:29.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.560+0000 7f7d129ab700 1 -- 172.21.15.70:0/2748132438 learned_addr learned my addr 172.21.15.70:0/2748132438 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:29.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.560+0000 7f7d121aa700 1 --2- 172.21.15.70:0/2748132438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b54c70 0x55a1c7b5da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.560+0000 7f7d119a9700 1 --2- 172.21.15.70:0/2748132438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b55620 0x55a1c7b5df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.561+0000 7f7d129ab700 1 -- 172.21.15.70:0/2748132438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b54c70 msgr2=0x55a1c7b5da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:29.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.561+0000 7f7d129ab700 1 --2- 172.21.15.70:0/2748132438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b54c70 0x55a1c7b5da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.561+0000 7f7d129ab700 1 -- 172.21.15.70:0/2748132438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b55620 msgr2=0x55a1c7b5df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:29.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.561+0000 7f7d129ab700 1 --2- 172.21.15.70:0/2748132438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b55620 0x55a1c7b5df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.561+0000 7f7d129ab700 1 -- 172.21.15.70:0/2748132438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1c7b5d860 con 0x55a1c7b59370 2024-06-04T05:01:29.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.561+0000 7f7d119a9700 1 --2- 172.21.15.70:0/2748132438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b55620 0x55a1c7b5df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:01:29.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.561+0000 7f7d129ab700 1 --2- 172.21.15.70:0/2748132438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b59370 0x55a1c7b731d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f7d04007630 tx=0x7f7d0400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16173bc2464c2609 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:29.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.561+0000 7f7cf7fff700 1 -- 172.21.15.70:0/2748132438 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d04020670 con 0x55a1c7b59370 2024-06-04T05:01:29.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.561+0000 7f7cf7fff700 1 -- 172.21.15.70:0/2748132438 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d04020ca0 con 0x55a1c7b59370 2024-06-04T05:01:29.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.561+0000 7f7cf7fff700 1 -- 172.21.15.70:0/2748132438 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d04017dc0 con 0x55a1c7b59370 2024-06-04T05:01:29.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.562+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/2748132438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b59370 msgr2=0x55a1c7b731d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:29.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.562+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/2748132438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b59370 0x55a1c7b731d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f7d04007630 tx=0x7f7d0400b040 comp rx=0 tx=0).stop 2024-06-04T05:01:29.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.562+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/2748132438 shutdown_connections 2024-06-04T05:01:29.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.562+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/2748132438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b59370 0x55a1c7b731d0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.562+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/2748132438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b54c70 0x55a1c7b5da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.562+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/2748132438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b55620 0x55a1c7b5df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.562+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/2748132438 >> 172.21.15.70:0/2748132438 conn(0x55a1c7a7e080 msgr2=0x55a1c7b4ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:29.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.562+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/2748132438 shutdown_connections 2024-06-04T05:01:29.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.562+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/2748132438 wait complete. 2024-06-04T05:01:29.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.562+0000 7f7d1eca3b80 1 -- start start 2024-06-04T05:01:29.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.562+0000 7f7d1eca3b80 1 Processor -- start 2024-06-04T05:01:29.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d1eca3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b54c70 0x55a1c7b70290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d1eca3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b55620 0x55a1c7b707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d1eca3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b59370 0x55a1c7bd69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d1eca3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c1acef0 con 0x55a1c7b54c70 2024-06-04T05:01:29.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d1eca3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c1ac820 con 0x55a1c7b59370 2024-06-04T05:01:29.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d1eca3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c1a9a40 con 0x55a1c7b55620 2024-06-04T05:01:29.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d1eca3b80 10 client.0 _ll_register_callbacks cb 0x55a1c7a3a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:29.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d121aa700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b54c70 0x55a1c7b70290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d129ab700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b59370 0x55a1c7bd69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d121aa700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b54c70 0x55a1c7b70290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59742/0 (socket says 172.21.15.70:59742) 2024-06-04T05:01:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d121aa700 1 -- 172.21.15.70:0/1537555872 learned_addr learned my addr 172.21.15.70:0/1537555872 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d119a9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b55620 0x55a1c7b707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d121aa700 1 -- 172.21.15.70:0/1537555872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b55620 msgr2=0x55a1c7b707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d121aa700 1 --2- 172.21.15.70:0/1537555872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b55620 0x55a1c7b707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d121aa700 1 -- 172.21.15.70:0/1537555872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b59370 msgr2=0x55a1c7bd69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d121aa700 1 --2- 172.21.15.70:0/1537555872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b59370 0x55a1c7bd69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d121aa700 1 -- 172.21.15.70:0/1537555872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1c7bd6f10 con 0x55a1c7b54c70 2024-06-04T05:01:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d119a9700 1 --2- 172.21.15.70:0/1537555872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b55620 0x55a1c7b707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d129ab700 1 --2- 172.21.15.70:0/1537555872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b59370 0x55a1c7bd69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:29.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d121aa700 1 --2- 172.21.15.70:0/1537555872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b54c70 0x55a1c7b70290 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f7d080aa040 tx=0x7f7d080a9bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eccf8f24aa42d318 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:29.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7cf5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:29.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7cf5ffb700 1 -- 172.21.15.70:0/1537555872 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d080ae6f0 con 0x55a1c7b54c70 2024-06-04T05:01:29.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7cf5ffb700 1 -- 172.21.15.70:0/1537555872 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f7d080aa7b0 con 0x55a1c7b54c70 2024-06-04T05:01:29.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.563+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/1537555872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d04020ca0 con 0x55a1c7b54c70 2024-06-04T05:01:29.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.564+0000 7f7cf5ffb700 1 -- 172.21.15.70:0/1537555872 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7d080b3110 con 0x55a1c7b54c70 2024-06-04T05:01:29.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.564+0000 7f7d1eca3b80 10 client.16671 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:29.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.564+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/1537555872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a1c7bd80b0 con 0x55a1c7b54c70 2024-06-04T05:01:29.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.564+0000 7f7cf5ffb700 1 -- 172.21.15.70:0/1537555872 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d04020ca0 con 0x55a1c7b54c70 2024-06-04T05:01:29.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.564+0000 7f7cf5ffb700 1 -- 172.21.15.70:0/1537555872 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f7d0800fc10 con 0x55a1c7b54c70 2024-06-04T05:01:29.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.564+0000 7f7d1eca3b80 10 client.16671 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:29.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.564+0000 7f7d1eca3b80 10 client.16671 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:01:29.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.564+0000 7f7d1eca3b80 20 client.16671 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:29.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.564+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/1537555872 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55a1c7bd8710 0x55a1c7bdab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:29.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.564+0000 7f7d1eca3b80 4 client.16671 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:01:29.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.564+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/1537555872 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a1c7ab1be0 con 0x55a1c7bd8710 2024-06-04T05:01:29.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.564+0000 7f7d119a9700 1 --2- 172.21.15.70:0/1537555872 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55a1c7bd8710 0x55a1c7bdab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:29.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.565+0000 7f7cf5ffb700 10 client.16671 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:01:29.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.565+0000 7f7d119a9700 1 --2- 172.21.15.70:0/1537555872 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55a1c7bd8710 0x55a1c7bdab00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eadebf946070217b server_cookie=a705d823c54d5d46 in_seq=0 out_seq=0 2024-06-04T05:01:29.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.565+0000 7f7cf5ffb700 1 -- 172.21.15.70:0/1537555872 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a1c7ab1be0 con 0x55a1c7bd8710 2024-06-04T05:01:29.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.565+0000 7f7cf5ffb700 10 client.16671 handle_command_reply: tid=0 2024-06-04T05:01:29.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.565+0000 7f7d1eca3b80 1 client.16671 shutdown 2024-06-04T05:01:29.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.565+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/1537555872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b54c70 msgr2=0x55a1c7b70290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:29.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.565+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/1537555872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b54c70 0x55a1c7b70290 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f7d080aa040 tx=0x7f7d080a9bd0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.565+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/1537555872 shutdown_connections 2024-06-04T05:01:29.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.565+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/1537555872 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55a1c7bd8710 0x55a1c7bdab00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.565+0000 7f7cf5ffb700 0 client.16671 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:01:29.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.565+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/1537555872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1c7b59370 0x55a1c7bd69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.565+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/1537555872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1c7b54c70 0x55a1c7b70290 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.565+0000 7f7d1eca3b80 1 --2- 172.21.15.70:0/1537555872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a1c7b55620 0x55a1c7b707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.565+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/1537555872 >> 172.21.15.70:0/1537555872 conn(0x55a1c7a7e080 msgr2=0x55a1c7b4ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:29.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.565+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/1537555872 shutdown_connections 2024-06-04T05:01:29.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.565+0000 7f7d1eca3b80 1 -- 172.21.15.70:0/1537555872 wait complete. 2024-06-04T05:01:29.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.565+0000 7f7d1eca3b80 20 client.16671 trim_cache size 0 max 16384 2024-06-04T05:01:29.578 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:01:29.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.566+0000 7f7d1440e700 1 -- 172.21.15.70:0/2944866884 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7cf80b2a70 msgr2=0x7f7cf80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:29.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.566+0000 7f7d1440e700 1 --2- 172.21.15.70:0/2944866884 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7cf80b2a70 0x7f7cf80b4f30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c079b60 tx=0x7f7cfc009260 comp rx=0 tx=0).stop 2024-06-04T05:01:29.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.566+0000 7f7d1440e700 1 -- 172.21.15.70:0/2944866884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d0c102440 msgr2=0x7f7d0c19a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:29.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.566+0000 7f7d1440e700 1 --2- 172.21.15.70:0/2944866884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d0c102440 0x7f7d0c19a920 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f7d08006010 tx=0x7f7d080047b0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.566+0000 7f7d1440e700 1 -- 172.21.15.70:0/2944866884 shutdown_connections 2024-06-04T05:01:29.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.566+0000 7f7d1440e700 1 --2- 172.21.15.70:0/2944866884 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7cf80b2a70 0x7f7cf80b4f30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.566+0000 7f7d1440e700 1 --2- 172.21.15.70:0/2944866884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d0c108520 0x7f7d0c1a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.566+0000 7f7d1440e700 1 --2- 172.21.15.70:0/2944866884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d0c102440 0x7f7d0c19a920 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.566+0000 7f7d1440e700 1 --2- 172.21.15.70:0/2944866884 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d0c102e00 0x7f7d0c19ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:29.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.566+0000 7f7d1440e700 1 -- 172.21.15.70:0/2944866884 >> 172.21.15.70:0/2944866884 conn(0x7f7d0c0793f0 msgr2=0x7f7d0c101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:29.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.566+0000 7f7d1440e700 1 -- 172.21.15.70:0/2944866884 shutdown_connections 2024-06-04T05:01:29.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:29.566+0000 7f7d1440e700 1 -- 172.21.15.70:0/2944866884 wait complete. 2024-06-04T05:01:29.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 46655 0.33 MB/sec execute 22 sec latency 13.694 ms 2024-06-04T05:01:30.466 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:01:30.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 47118 0.35 MB/sec execute 23 sec latency 13.973 ms 2024-06-04T05:01:30.734 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:01:31.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:31 smithi112 ceph-mon[36453]: pgmap v133: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 11 op/s 2024-06-04T05:01:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:31 smithi098 ceph-mon[35569]: pgmap v133: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 11 op/s 2024-06-04T05:01:31.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:31 smithi070 ceph-mon[32631]: pgmap v133: 97 pgs: 97 active+clean; 40 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 11 op/s 2024-06-04T05:01:31.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 47706 0.52 MB/sec execute 24 sec latency 15.352 ms 2024-06-04T05:01:32.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.672+0000 7f39c8a03700 1 -- 172.21.15.70:0/1653906093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39c4102f80 msgr2=0x7f39c4106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:32.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.672+0000 7f39c8a03700 1 --2- 172.21.15.70:0/1653906093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39c4102f80 0x7f39c4106fd0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f39b80099c0 tx=0x7f39b8009cd0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.672+0000 7f39c8a03700 1 -- 172.21.15.70:0/1653906093 shutdown_connections 2024-06-04T05:01:32.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.672+0000 7f39c8a03700 1 --2- 172.21.15.70:0/1653906093 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c41076b0 0x7f39c410f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.672+0000 7f39c8a03700 1 --2- 172.21.15.70:0/1653906093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c41025d0 0x7f39c41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.672+0000 7f39c8a03700 1 --2- 172.21.15.70:0/1653906093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39c4102f80 0x7f39c4106fd0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.672+0000 7f39c8a03700 1 -- 172.21.15.70:0/1653906093 >> 172.21.15.70:0/1653906093 conn(0x7f39c40fde60 msgr2=0x7f39c4100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:32.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.672+0000 7f39c8a03700 1 -- 172.21.15.70:0/1653906093 shutdown_connections 2024-06-04T05:01:32.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.672+0000 7f39c8a03700 1 -- 172.21.15.70:0/1653906093 wait complete. 2024-06-04T05:01:32.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.673+0000 7f39c8a03700 1 Processor -- start 2024-06-04T05:01:32.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.673+0000 7f39c8a03700 1 -- start start 2024-06-04T05:01:32.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.674+0000 7f39c8a03700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c41025d0 0x7f39c419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.674+0000 7f39c8a03700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39c4102f80 0x7f39c419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.674+0000 7f39c8a03700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c41076b0 0x7f39c41998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.674+0000 7f39c8a03700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39c41123f0 con 0x7f39c4102f80 2024-06-04T05:01:32.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.674+0000 7f39c8a03700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39c4112270 con 0x7f39c41025d0 2024-06-04T05:01:32.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.674+0000 7f39c8a03700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f39c4112570 con 0x7f39c41076b0 2024-06-04T05:01:32.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.674+0000 7f39c259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39c4102f80 0x7f39c419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:32.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.674+0000 7f39c259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39c4102f80 0x7f39c419fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59768/0 (socket says 172.21.15.70:59768) 2024-06-04T05:01:32.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.674+0000 7f39c259c700 1 -- 172.21.15.70:0/3801868434 learned_addr learned my addr 172.21.15.70:0/3801868434 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:32.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.674+0000 7f39c359e700 1 --2- 172.21.15.70:0/3801868434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c41076b0 0x7f39c41998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:32.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.674+0000 7f39c2d9d700 1 --2- 172.21.15.70:0/3801868434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c41025d0 0x7f39c419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:32.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.674+0000 7f39c259c700 1 -- 172.21.15.70:0/3801868434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c41076b0 msgr2=0x7f39c41998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:32.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.674+0000 7f39c259c700 1 --2- 172.21.15.70:0/3801868434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c41076b0 0x7f39c41998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.674+0000 7f39c259c700 1 -- 172.21.15.70:0/3801868434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c41025d0 msgr2=0x7f39c419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:32.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.674+0000 7f39c259c700 1 --2- 172.21.15.70:0/3801868434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c41025d0 0x7f39c419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.674+0000 7f39c259c700 1 -- 172.21.15.70:0/3801868434 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c419a050 con 0x7f39c4102f80 2024-06-04T05:01:32.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.674+0000 7f39c2d9d700 1 --2- 172.21.15.70:0/3801868434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c41025d0 0x7f39c419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:01:32.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.675+0000 7f39c259c700 1 --2- 172.21.15.70:0/3801868434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39c4102f80 0x7f39c419fc70 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f39b8005f50 tx=0x7f39b8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:32.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.675+0000 7f39abfff700 1 -- 172.21.15.70:0/3801868434 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b8026070 con 0x7f39c4102f80 2024-06-04T05:01:32.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.675+0000 7f39abfff700 1 -- 172.21.15.70:0/3801868434 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39b8003c50 con 0x7f39c4102f80 2024-06-04T05:01:32.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.675+0000 7f39abfff700 1 -- 172.21.15.70:0/3801868434 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b800fa30 con 0x7f39c4102f80 2024-06-04T05:01:32.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.675+0000 7f39c8a03700 1 -- 172.21.15.70:0/3801868434 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39c419a2e0 con 0x7f39c4102f80 2024-06-04T05:01:32.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.675+0000 7f39c8a03700 1 -- 172.21.15.70:0/3801868434 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c41a6600 con 0x7f39c4102f80 2024-06-04T05:01:32.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.678+0000 7f39abfff700 1 -- 172.21.15.70:0/3801868434 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f39b8029070 con 0x7f39c4102f80 2024-06-04T05:01:32.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.678+0000 7f39abfff700 1 --2- 172.21.15.70:0/3801868434 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39ac0b2a70 0x7f39ac0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.679+0000 7f39c2d9d700 1 --2- 172.21.15.70:0/3801868434 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39ac0b2a70 0x7f39ac0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:32.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.679+0000 7f39abfff700 1 -- 172.21.15.70:0/3801868434 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f39b80e2e30 con 0x7f39c4102f80 2024-06-04T05:01:32.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.679+0000 7f39c2d9d700 1 --2- 172.21.15.70:0/3801868434 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39ac0b2a70 0x7f39ac0b4f30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f39c419aa30 tx=0x7f39b4009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:32.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.679+0000 7f39d3a99b80 1 Processor -- start 2024-06-04T05:01:32.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.679+0000 7f39d3a99b80 1 -- start start 2024-06-04T05:01:32.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.680+0000 7f39d3a99b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cbcde0 0x560e96cbd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.680+0000 7f39d3a99b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cb8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.680+0000 7f39d3a99b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cb9040 0x560e96cc4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.680+0000 7f39d3a99b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560e96c17c90 con 0x560e96cb8680 2024-06-04T05:01:32.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.680+0000 7f39d3a99b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560e96c07830 con 0x560e96cbcde0 2024-06-04T05:01:32.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.680+0000 7f39d3a99b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560e96c0b870 con 0x560e96cb9040 2024-06-04T05:01:32.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.680+0000 7f39c259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cb8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:32.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.680+0000 7f39c2d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cbcde0 0x560e96cbd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:32.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.680+0000 7f39c259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cb8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59784/0 (socket says 172.21.15.70:59784) 2024-06-04T05:01:32.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.680+0000 7f39c259c700 1 -- 172.21.15.70:0/516794679 learned_addr learned my addr 172.21.15.70:0/516794679 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:32.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.680+0000 7f39c259c700 1 -- 172.21.15.70:0/516794679 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cb9040 msgr2=0x560e96cc4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:32.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.680+0000 7f39c259c700 1 --2- 172.21.15.70:0/516794679 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cb9040 0x560e96cc4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.680+0000 7f39c259c700 1 -- 172.21.15.70:0/516794679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cbcde0 msgr2=0x560e96cbd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:32.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.680+0000 7f39c259c700 1 --2- 172.21.15.70:0/516794679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cbcde0 0x560e96cbd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.680+0000 7f39c259c700 1 -- 172.21.15.70:0/516794679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e96b578d0 con 0x560e96cb8680 2024-06-04T05:01:32.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.680+0000 7f39c2d9d700 1 --2- 172.21.15.70:0/516794679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cbcde0 0x560e96cbd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:32.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.680+0000 7f39c259c700 1 --2- 172.21.15.70:0/516794679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cb8b00 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f39b80a6220 tx=0x7f39b80aa070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1fecd5fb6c2f60e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:32.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.681+0000 7f39a97fa700 1 -- 172.21.15.70:0/516794679 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b8003e00 con 0x560e96cb8680 2024-06-04T05:01:32.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.681+0000 7f39a97fa700 1 -- 172.21.15.70:0/516794679 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39b80a9cb0 con 0x560e96cb8680 2024-06-04T05:01:32.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.681+0000 7f39a97fa700 1 -- 172.21.15.70:0/516794679 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b80aa400 con 0x560e96cb8680 2024-06-04T05:01:32.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.681+0000 7f39d3a99b80 1 -- 172.21.15.70:0/516794679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 msgr2=0x560e96cb8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:32.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.681+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/516794679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cb8b00 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f39b80a6220 tx=0x7f39b80aa070 comp rx=0 tx=0).stop 2024-06-04T05:01:32.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.681+0000 7f39d3a99b80 1 -- 172.21.15.70:0/516794679 shutdown_connections 2024-06-04T05:01:32.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.681+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/516794679 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cb9040 0x560e96cc4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.681+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/516794679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cbcde0 0x560e96cbd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.681+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/516794679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cb8b00 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.681+0000 7f39d3a99b80 1 -- 172.21.15.70:0/516794679 >> 172.21.15.70:0/516794679 conn(0x560e96be6080 msgr2=0x560e96cb2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:32.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.682+0000 7f39d3a99b80 1 -- 172.21.15.70:0/516794679 shutdown_connections 2024-06-04T05:01:32.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.682+0000 7f39d3a99b80 1 -- 172.21.15.70:0/516794679 wait complete. 2024-06-04T05:01:32.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.682+0000 7f39d3a99b80 1 -- start start 2024-06-04T05:01:32.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.682+0000 7f39d3a99b80 1 Processor -- start 2024-06-04T05:01:32.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.682+0000 7f39d3a99b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cc4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.682+0000 7f39d3a99b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cb9040 0x560e96cc1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.682+0000 7f39d3a99b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cbcde0 0x560e96cc22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.682+0000 7f39d3a99b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560e96b0dfa0 con 0x560e96cb8680 2024-06-04T05:01:32.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.682+0000 7f39d3a99b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560e96ba97c0 con 0x560e96cb9040 2024-06-04T05:01:32.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.682+0000 7f39d3a99b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560e96c17c90 con 0x560e96cbcde0 2024-06-04T05:01:32.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.682+0000 7f39d3a99b80 10 client.0 _ll_register_callbacks cb 0x560e96ba2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:32.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.682+0000 7f39c2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cc4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:32.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.682+0000 7f39c2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cc4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59800/0 (socket says 172.21.15.70:59800) 2024-06-04T05:01:32.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.682+0000 7f39c2d9d700 1 -- 172.21.15.70:0/908950655 learned_addr learned my addr 172.21.15.70:0/908950655 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:32.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.683+0000 7f39c259c700 1 --2- 172.21.15.70:0/908950655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cb9040 0x560e96cc1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:32.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.683+0000 7f39c2d9d700 1 -- 172.21.15.70:0/908950655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cbcde0 msgr2=0x560e96cc22f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:01:32.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.683+0000 7f39c2d9d700 1 --2- 172.21.15.70:0/908950655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cbcde0 0x560e96cc22f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.683+0000 7f39c2d9d700 1 -- 172.21.15.70:0/908950655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cb9040 msgr2=0x560e96cc1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:32.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.683+0000 7f39c2d9d700 1 --2- 172.21.15.70:0/908950655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cb9040 0x560e96cc1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.683+0000 7f39c2d9d700 1 -- 172.21.15.70:0/908950655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e96ba6690 con 0x560e96cb8680 2024-06-04T05:01:32.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.683+0000 7f39c2d9d700 1 --2- 172.21.15.70:0/908950655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cc4ab0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f39b40126d0 tx=0x7f39b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7053df2b6374873 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:32.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.683+0000 7f39a6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:32.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.683+0000 7f39a6ff5700 1 -- 172.21.15.70:0/908950655 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b4020840 con 0x560e96cb8680 2024-06-04T05:01:32.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.683+0000 7f39a6ff5700 1 -- 172.21.15.70:0/908950655 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f39b40209e0 con 0x560e96cb8680 2024-06-04T05:01:32.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.683+0000 7f39d3a99b80 1 -- 172.21.15.70:0/908950655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39b80a9cb0 con 0x560e96cb8680 2024-06-04T05:01:32.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.684+0000 7f39a6ff5700 1 -- 172.21.15.70:0/908950655 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f39b4020be0 con 0x560e96cb8680 2024-06-04T05:01:32.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.684+0000 7f39d3a99b80 10 client.16683 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:32.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.684+0000 7f39d3a99b80 1 -- 172.21.15.70:0/908950655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560e96ccf8f0 con 0x560e96cb8680 2024-06-04T05:01:32.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.684+0000 7f39a6ff5700 1 -- 172.21.15.70:0/908950655 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39b80a9cb0 con 0x560e96cb8680 2024-06-04T05:01:32.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.684+0000 7f39a6ff5700 1 -- 172.21.15.70:0/908950655 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f39b400f040 con 0x560e96cb8680 2024-06-04T05:01:32.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.684+0000 7f39d3a99b80 10 client.16683 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:32.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.684+0000 7f39d3a99b80 10 client.16683 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:32.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.684+0000 7f39d3a99b80 20 client.16683 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:32.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.684+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/908950655 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560e96cd1e80 0x560e96cd4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.684+0000 7f39d3a99b80 4 client.16683 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:01:32.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.684+0000 7f39d3a99b80 1 -- 172.21.15.70:0/908950655 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e96c849e0 con 0x560e96cd1e80 2024-06-04T05:01:32.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.684+0000 7f39c259c700 1 --2- 172.21.15.70:0/908950655 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560e96cd1e80 0x560e96cd4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:32.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.685+0000 7f39a6ff5700 10 client.16683 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:32.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.685+0000 7f39c259c700 1 --2- 172.21.15.70:0/908950655 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560e96cd1e80 0x560e96cd4320 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17d2500e81c2e9c7 server_cookie=b44f678340eefca8 in_seq=0 out_seq=0 2024-06-04T05:01:32.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.685+0000 7f39a6ff5700 1 -- 172.21.15.70:0/908950655 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560e96c849e0 con 0x560e96cd1e80 2024-06-04T05:01:32.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.685+0000 7f39a6ff5700 10 client.16683 handle_command_reply: tid=0 2024-06-04T05:01:32.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.685+0000 7f39d3a99b80 1 client.16683 shutdown 2024-06-04T05:01:32.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.686+0000 7f39d3a99b80 1 -- 172.21.15.70:0/908950655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 msgr2=0x560e96cc4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:32.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.686+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/908950655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cc4ab0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f39b40126d0 tx=0x7f39b4004750 comp rx=0 tx=0).stop 2024-06-04T05:01:32.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.686+0000 7f39d3a99b80 1 -- 172.21.15.70:0/908950655 shutdown_connections 2024-06-04T05:01:32.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.686+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/908950655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cbcde0 0x560e96cc22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.686+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/908950655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cc4ab0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.686+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/908950655 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560e96cd1e80 0x560e96cd4320 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.686+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/908950655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cb9040 0x560e96cc1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.686+0000 7f39d3a99b80 1 -- 172.21.15.70:0/908950655 >> 172.21.15.70:0/908950655 conn(0x560e96be6080 msgr2=0x560e96cbe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:32.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.686+0000 7f39a6ff5700 0 client.16683 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:32.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.686+0000 7f39d3a99b80 1 -- 172.21.15.70:0/908950655 shutdown_connections 2024-06-04T05:01:32.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.686+0000 7f39d3a99b80 1 -- 172.21.15.70:0/908950655 wait complete. 2024-06-04T05:01:32.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.686+0000 7f39d3a99b80 20 client.16683 trim_cache size 0 max 16384 2024-06-04T05:01:32.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.702+0000 7f39d3a99b80 1 Processor -- start 2024-06-04T05:01:32.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.702+0000 7f39d3a99b80 1 -- start start 2024-06-04T05:01:32.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39d3a99b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cce140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39d3a99b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cb9040 0x560e96cc0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39d3a99b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cbcde0 0x560e96cc45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39d3a99b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39c41adbb0 con 0x560e96cb8680 2024-06-04T05:01:32.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39d3a99b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39c41af750 con 0x560e96cbcde0 2024-06-04T05:01:32.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39d3a99b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f39c41ae280 con 0x560e96cb9040 2024-06-04T05:01:32.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39c2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:32.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39c259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cb9040 0x560e96cc0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:32.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39c2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cce140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59810/0 (socket says 172.21.15.70:59810) 2024-06-04T05:01:32.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39c2d9d700 1 -- 172.21.15.70:0/2092955332 learned_addr learned my addr 172.21.15.70:0/2092955332 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:32.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39c359e700 1 --2- 172.21.15.70:0/2092955332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cbcde0 0x560e96cc45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:32.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39c2d9d700 1 -- 172.21.15.70:0/2092955332 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cb9040 msgr2=0x560e96cc0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:32.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39c2d9d700 1 --2- 172.21.15.70:0/2092955332 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cb9040 0x560e96cc0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39c2d9d700 1 -- 172.21.15.70:0/2092955332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cbcde0 msgr2=0x560e96cc45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:32.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39c2d9d700 1 --2- 172.21.15.70:0/2092955332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cbcde0 0x560e96cc45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39c2d9d700 1 -- 172.21.15.70:0/2092955332 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e96cc0340 con 0x560e96cb8680 2024-06-04T05:01:32.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39c2d9d700 1 --2- 172.21.15.70:0/2092955332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cce140 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f39b4000c00 tx=0x7f39b40107c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc74ca1cdd36c00 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:32.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39a8ff9700 1 -- 172.21.15.70:0/2092955332 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b4020640 con 0x560e96cb8680 2024-06-04T05:01:32.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.703+0000 7f39a8ff9700 1 -- 172.21.15.70:0/2092955332 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39b4024070 con 0x560e96cb8680 2024-06-04T05:01:32.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.704+0000 7f39a8ff9700 1 -- 172.21.15.70:0/2092955332 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b4032850 con 0x560e96cb8680 2024-06-04T05:01:32.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.704+0000 7f39d3a99b80 1 -- 172.21.15.70:0/2092955332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 msgr2=0x560e96cce140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:32.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.704+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/2092955332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cce140 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f39b4000c00 tx=0x7f39b40107c0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.704+0000 7f39d3a99b80 1 -- 172.21.15.70:0/2092955332 shutdown_connections 2024-06-04T05:01:32.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.704+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/2092955332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cbcde0 0x560e96cc45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.704+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/2092955332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cce140 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.704+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/2092955332 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cb9040 0x560e96cc0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.704+0000 7f39d3a99b80 1 -- 172.21.15.70:0/2092955332 >> 172.21.15.70:0/2092955332 conn(0x560e96be6080 msgr2=0x560e96cc0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:32.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.704+0000 7f39d3a99b80 1 -- 172.21.15.70:0/2092955332 shutdown_connections 2024-06-04T05:01:32.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.704+0000 7f39d3a99b80 1 -- 172.21.15.70:0/2092955332 wait complete. 2024-06-04T05:01:32.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.705+0000 7f39d3a99b80 1 -- start start 2024-06-04T05:01:32.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.705+0000 7f39d3a99b80 1 Processor -- start 2024-06-04T05:01:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.705+0000 7f39d3a99b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.705+0000 7f39d3a99b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cb9040 0x560e96cd9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.705+0000 7f39d3a99b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cbcde0 0x560e96cd9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.705+0000 7f39d3a99b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39c41af020 con 0x560e96cb8680 2024-06-04T05:01:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.705+0000 7f39d3a99b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39c41ae950 con 0x560e96cb9040 2024-06-04T05:01:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.705+0000 7f39d3a99b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f39c41abb50 con 0x560e96cbcde0 2024-06-04T05:01:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.705+0000 7f39d3a99b80 10 client.0 _ll_register_callbacks cb 0x560e96ba2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.705+0000 7f39c2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.705+0000 7f39c2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cca710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59816/0 (socket says 172.21.15.70:59816) 2024-06-04T05:01:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.705+0000 7f39c2d9d700 1 -- 172.21.15.70:0/3525293918 learned_addr learned my addr 172.21.15.70:0/3525293918 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.705+0000 7f39c259c700 1 --2- 172.21.15.70:0/3525293918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cb9040 0x560e96cd9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.705+0000 7f39c359e700 1 --2- 172.21.15.70:0/3525293918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cbcde0 0x560e96cd9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.706+0000 7f39c2d9d700 1 -- 172.21.15.70:0/3525293918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cbcde0 msgr2=0x560e96cd9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.706+0000 7f39c2d9d700 1 --2- 172.21.15.70:0/3525293918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cbcde0 0x560e96cd9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.706+0000 7f39c2d9d700 1 -- 172.21.15.70:0/3525293918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cb9040 msgr2=0x560e96cd9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.706+0000 7f39c2d9d700 1 --2- 172.21.15.70:0/3525293918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cb9040 0x560e96cd9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.706+0000 7f39c2d9d700 1 -- 172.21.15.70:0/3525293918 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e96cd9c90 con 0x560e96cb8680 2024-06-04T05:01:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.706+0000 7f39c2d9d700 1 --2- 172.21.15.70:0/3525293918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cca710 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f39b4020bb0 tx=0x7f39b4005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8755a601f2b758f1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.706+0000 7f39a6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.706+0000 7f39a6ff5700 1 -- 172.21.15.70:0/3525293918 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b40328e0 con 0x560e96cb8680 2024-06-04T05:01:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.706+0000 7f39a6ff5700 1 -- 172.21.15.70:0/3525293918 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f39b4005090 con 0x560e96cb8680 2024-06-04T05:01:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.706+0000 7f39d3a99b80 1 -- 172.21.15.70:0/3525293918 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39b4024070 con 0x560e96cb8680 2024-06-04T05:01:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.706+0000 7f39a6ff5700 1 -- 172.21.15.70:0/3525293918 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f39b4050050 con 0x560e96cb8680 2024-06-04T05:01:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.706+0000 7f39d3a99b80 10 client.16689 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.706+0000 7f39d3a99b80 1 -- 172.21.15.70:0/3525293918 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560e96d3c860 con 0x560e96cb8680 2024-06-04T05:01:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.706+0000 7f39a6ff5700 1 -- 172.21.15.70:0/3525293918 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39b4024070 con 0x560e96cb8680 2024-06-04T05:01:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.706+0000 7f39a6ff5700 1 -- 172.21.15.70:0/3525293918 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f39b404b030 con 0x560e96cb8680 2024-06-04T05:01:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39d3a99b80 10 client.16689 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39d3a99b80 10 client.16689 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39d3a99b80 20 client.16689 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/3525293918 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560e96d3cfc0 0x560e96d3f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39d3a99b80 4 client.16689 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:01:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39d3a99b80 1 -- 172.21.15.70:0/3525293918 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560e96c19be0 con 0x560e96d3cfc0 2024-06-04T05:01:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39c259c700 1 --2- 172.21.15.70:0/3525293918 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560e96d3cfc0 0x560e96d3f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39a6ff5700 10 client.16689 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39c259c700 1 --2- 172.21.15.70:0/3525293918 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560e96d3cfc0 0x560e96d3f3e0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd528191322b211 server_cookie=4af89681e0e15b14 in_seq=0 out_seq=0 2024-06-04T05:01:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39a6ff5700 1 -- 172.21.15.70:0/3525293918 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560e96c19be0 con 0x560e96d3cfc0 2024-06-04T05:01:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39a6ff5700 10 client.16689 handle_command_reply: tid=0 2024-06-04T05:01:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39d3a99b80 1 client.16689 shutdown 2024-06-04T05:01:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39d3a99b80 1 -- 172.21.15.70:0/3525293918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 msgr2=0x560e96cca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/3525293918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cca710 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f39b4020bb0 tx=0x7f39b4005a00 comp rx=0 tx=0).stop 2024-06-04T05:01:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39d3a99b80 1 -- 172.21.15.70:0/3525293918 shutdown_connections 2024-06-04T05:01:32.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/3525293918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560e96cbcde0 0x560e96cd9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/3525293918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e96cb8680 0x560e96cca710 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.707+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/3525293918 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560e96d3cfc0 0x560e96d3f3e0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.708+0000 7f39d3a99b80 1 --2- 172.21.15.70:0/3525293918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e96cb9040 0x560e96cd9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.708+0000 7f39d3a99b80 1 -- 172.21.15.70:0/3525293918 >> 172.21.15.70:0/3525293918 conn(0x560e96be6080 msgr2=0x560e96cbe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:32.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.708+0000 7f39a6ff5700 0 client.16689 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:32.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.708+0000 7f39d3a99b80 1 -- 172.21.15.70:0/3525293918 shutdown_connections 2024-06-04T05:01:32.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.708+0000 7f39d3a99b80 1 -- 172.21.15.70:0/3525293918 wait complete. 2024-06-04T05:01:32.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.708+0000 7f39d3a99b80 20 client.16689 trim_cache size 0 max 16384 2024-06-04T05:01:32.716 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:01:32.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.708+0000 7f39c8a03700 1 -- 172.21.15.70:0/3801868434 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39ac0b2a70 msgr2=0x7f39ac0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:32.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.708+0000 7f39c8a03700 1 --2- 172.21.15.70:0/3801868434 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39ac0b2a70 0x7f39ac0b4f30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f39c419aa30 tx=0x7f39b4009260 comp rx=0 tx=0).stop 2024-06-04T05:01:32.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.708+0000 7f39c8a03700 1 -- 172.21.15.70:0/3801868434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39c4102f80 msgr2=0x7f39c419fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:32.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.708+0000 7f39c8a03700 1 --2- 172.21.15.70:0/3801868434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39c4102f80 0x7f39c419fc70 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f39b8005f50 tx=0x7f39b8004270 comp rx=0 tx=0).stop 2024-06-04T05:01:32.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.708+0000 7f39c8a03700 1 -- 172.21.15.70:0/3801868434 shutdown_connections 2024-06-04T05:01:32.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.708+0000 7f39c8a03700 1 --2- 172.21.15.70:0/3801868434 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39ac0b2a70 0x7f39ac0b4f30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.708+0000 7f39c8a03700 1 --2- 172.21.15.70:0/3801868434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c41076b0 0x7f39c41998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.708+0000 7f39c8a03700 1 --2- 172.21.15.70:0/3801868434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c41025d0 0x7f39c419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.708+0000 7f39c8a03700 1 --2- 172.21.15.70:0/3801868434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39c4102f80 0x7f39c419fc70 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:32.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.708+0000 7f39c8a03700 1 -- 172.21.15.70:0/3801868434 >> 172.21.15.70:0/3801868434 conn(0x7f39c40fde60 msgr2=0x7f39c406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:32.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.709+0000 7f39c8a03700 1 -- 172.21.15.70:0/3801868434 shutdown_connections 2024-06-04T05:01:32.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:32.709+0000 7f39c8a03700 1 -- 172.21.15.70:0/3801868434 wait complete. 2024-06-04T05:01:32.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 48364 0.52 MB/sec execute 25 sec latency 15.575 ms 2024-06-04T05:01:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.249+0000 7f4696609640 1 -- 172.21.15.70:0/76829897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4690102a40 msgr2=0x7f4690102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.249+0000 7f4696609640 1 --2- 172.21.15.70:0/76829897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4690102a40 0x7f4690102ec0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f4684009930 tx=0x7f468402a420 comp rx=0 tx=0).stop 2024-06-04T05:01:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.249+0000 7f4696609640 1 -- 172.21.15.70:0/76829897 shutdown_connections 2024-06-04T05:01:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.249+0000 7f4696609640 1 --2- 172.21.15.70:0/76829897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4690103400 0x7f469010f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.249+0000 7f4696609640 1 --2- 172.21.15.70:0/76829897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4690102a40 0x7f4690102ec0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.249+0000 7f4696609640 1 --2- 172.21.15.70:0/76829897 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4690108a50 0x7f4690108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.249+0000 7f4696609640 1 -- 172.21.15.70:0/76829897 >> 172.21.15.70:0/76829897 conn(0x7f46900fe510 msgr2=0x7f4690100930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.249+0000 7f4696609640 1 -- 172.21.15.70:0/76829897 shutdown_connections 2024-06-04T05:01:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.249+0000 7f4696609640 1 -- 172.21.15.70:0/76829897 wait complete. 2024-06-04T05:01:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.250+0000 7f4696609640 1 Processor -- start 2024-06-04T05:01:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.250+0000 7f4696609640 1 -- start start 2024-06-04T05:01:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.250+0000 7f4696609640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4690102a40 0x7f4690199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.250+0000 7f4696609640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4690103400 0x7f46901996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.250+0000 7f4696609640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4690108a50 0x7f469019da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.250+0000 7f4696609640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4690112bc0 con 0x7f4690102a40 2024-06-04T05:01:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.250+0000 7f4696609640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4690112a40 con 0x7f4690108a50 2024-06-04T05:01:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.250+0000 7f4696609640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4690112d40 con 0x7f4690103400 2024-06-04T05:01:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.250+0000 7f468ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4690102a40 0x7f4690199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.250+0000 7f468ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4690102a40 0x7f4690199190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59834/0 (socket says 172.21.15.70:59834) 2024-06-04T05:01:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.250+0000 7f468ffff640 1 -- 172.21.15.70:0/1672348777 learned_addr learned my addr 172.21.15.70:0/1672348777 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.250+0000 7f468f7fe640 1 --2- 172.21.15.70:0/1672348777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4690103400 0x7f46901996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.251+0000 7f4694ba6640 1 --2- 172.21.15.70:0/1672348777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4690108a50 0x7f469019da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.251+0000 7f468ffff640 1 -- 172.21.15.70:0/1672348777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4690103400 msgr2=0x7f46901996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.251+0000 7f468ffff640 1 --2- 172.21.15.70:0/1672348777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4690103400 0x7f46901996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.251+0000 7f468ffff640 1 -- 172.21.15.70:0/1672348777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4690108a50 msgr2=0x7f469019da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.251+0000 7f468ffff640 1 --2- 172.21.15.70:0/1672348777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4690108a50 0x7f469019da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.251+0000 7f468ffff640 1 -- 172.21.15.70:0/1672348777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469019e180 con 0x7f4690102a40 2024-06-04T05:01:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.251+0000 7f4694ba6640 1 --2- 172.21.15.70:0/1672348777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4690108a50 0x7f469019da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:01:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.251+0000 7f468ffff640 1 --2- 172.21.15.70:0/1672348777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4690102a40 0x7f4690199190 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f468000d950 tx=0x7f468000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.251+0000 7f468d7fa640 1 -- 172.21.15.70:0/1672348777 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4680017070 con 0x7f4690102a40 2024-06-04T05:01:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.251+0000 7f4696609640 1 -- 172.21.15.70:0/1672348777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f469019e470 con 0x7f4690102a40 2024-06-04T05:01:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.251+0000 7f468d7fa640 1 -- 172.21.15.70:0/1672348777 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4680009c30 con 0x7f4690102a40 2024-06-04T05:01:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.251+0000 7f468d7fa640 1 -- 172.21.15.70:0/1672348777 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468000f620 con 0x7f4690102a40 2024-06-04T05:01:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.251+0000 7f4696609640 1 -- 172.21.15.70:0/1672348777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46901a5d00 con 0x7f4690102a40 2024-06-04T05:01:33.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.254+0000 7f4699404740 1 Processor -- start 2024-06-04T05:01:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.254+0000 7f4699404740 1 -- start start 2024-06-04T05:01:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.254+0000 7f4699404740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e55820 0x55c723e55c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.254+0000 7f4699404740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e56140 0x55c723e604c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.254+0000 7f4699404740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e60a00 0x55c723e62df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.254+0000 7f4699404740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c723dc7c30 con 0x55c723e55820 2024-06-04T05:01:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.254+0000 7f4699404740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c723db77d0 con 0x55c723e60a00 2024-06-04T05:01:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.254+0000 7f4699404740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c723dbb810 con 0x55c723e56140 2024-06-04T05:01:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.255+0000 7f468d7fa640 1 -- 172.21.15.70:0/1672348777 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f4680010460 con 0x7f4690102a40 2024-06-04T05:01:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.255+0000 7f4694ba6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e55820 0x55c723e55c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.255+0000 7f4694ba6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e55820 0x55c723e55c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59844/0 (socket says 172.21.15.70:59844) 2024-06-04T05:01:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.255+0000 7f4694ba6640 1 -- 172.21.15.70:0/1618427296 learned_addr learned my addr 172.21.15.70:0/1618427296 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:33.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.255+0000 7f468ffff640 1 --2- 172.21.15.70:0/1618427296 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e56140 0x55c723e604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.255+0000 7f468f7fe640 1 --2- 172.21.15.70:0/1618427296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e60a00 0x55c723e62df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.255+0000 7f4694ba6640 1 -- 172.21.15.70:0/1618427296 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e56140 msgr2=0x55c723e604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:33.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.255+0000 7f4694ba6640 1 --2- 172.21.15.70:0/1618427296 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e56140 0x55c723e604c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.255+0000 7f4694ba6640 1 -- 172.21.15.70:0/1618427296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e60a00 msgr2=0x55c723e62df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:33.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.255+0000 7f4694ba6640 1 --2- 172.21.15.70:0/1618427296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e60a00 0x55c723e62df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.255+0000 7f4694ba6640 1 -- 172.21.15.70:0/1618427296 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c723acc210 con 0x55c723e55820 2024-06-04T05:01:33.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.255+0000 7f468ffff640 1 --2- 172.21.15.70:0/1618427296 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e56140 0x55c723e604c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:01:33.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.255+0000 7f4694ba6640 1 --2- 172.21.15.70:0/1618427296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e55820 0x55c723e55c00 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4688009f60 tx=0x7f4688009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2143990fb4ddd266 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:33.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.255+0000 7f46767fc640 1 -- 172.21.15.70:0/1618427296 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4688011640 con 0x55c723e55820 2024-06-04T05:01:33.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.255+0000 7f46767fc640 1 -- 172.21.15.70:0/1618427296 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4688011c70 con 0x55c723e55820 2024-06-04T05:01:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.255+0000 7f46767fc640 1 -- 172.21.15.70:0/1618427296 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4688010c10 con 0x55c723e55820 2024-06-04T05:01:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.255+0000 7f468d7fa640 1 --2- 172.21.15.70:0/1672348777 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f46780b2a10 0x7f46780b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.256+0000 7f468ffff640 1 --2- 172.21.15.70:0/1672348777 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f46780b2a10 0x7f46780b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.256+0000 7f468d7fa640 1 -- 172.21.15.70:0/1672348777 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f46800e21c0 con 0x7f4690102a40 2024-06-04T05:01:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.256+0000 7f4699404740 1 -- 172.21.15.70:0/1618427296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e55820 msgr2=0x55c723e55c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.256+0000 7f468ffff640 1 --2- 172.21.15.70:0/1672348777 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f46780b2a10 0x7f46780b4ed0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f46800df9c0 tx=0x7f468000bf60 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.256+0000 7f4699404740 1 --2- 172.21.15.70:0/1618427296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e55820 0x55c723e55c00 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4688009f60 tx=0x7f4688009f90 comp rx=0 tx=0).stop 2024-06-04T05:01:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.256+0000 7f4699404740 1 -- 172.21.15.70:0/1618427296 shutdown_connections 2024-06-04T05:01:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.256+0000 7f4699404740 1 --2- 172.21.15.70:0/1618427296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e60a00 0x55c723e62df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.256+0000 7f4699404740 1 --2- 172.21.15.70:0/1618427296 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e56140 0x55c723e604c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.256+0000 7f4699404740 1 --2- 172.21.15.70:0/1618427296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e55820 0x55c723e55c00 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.256+0000 7f4699404740 1 -- 172.21.15.70:0/1618427296 >> 172.21.15.70:0/1618427296 conn(0x55c723e51d30 msgr2=0x55c723e52110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.256+0000 7f4699404740 1 -- 172.21.15.70:0/1618427296 shutdown_connections 2024-06-04T05:01:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.256+0000 7f4699404740 1 -- 172.21.15.70:0/1618427296 wait complete. 2024-06-04T05:01:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.257+0000 7f4699404740 1 -- start start 2024-06-04T05:01:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.257+0000 7f4699404740 1 Processor -- start 2024-06-04T05:01:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.257+0000 7f4699404740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e55820 0x55c723e5d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.257+0000 7f4699404740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e56140 0x55c723e5df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.257+0000 7f4699404740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e60a00 0x55c723e5e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.257+0000 7f4699404740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c723a9d790 con 0x55c723e56140 2024-06-04T05:01:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.257+0000 7f4699404740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c723d73890 con 0x55c723e60a00 2024-06-04T05:01:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.257+0000 7f4699404740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c723e63630 con 0x55c723e55820 2024-06-04T05:01:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.257+0000 7f4699404740 10 client.0 _ll_register_callbacks cb 0x55c723acba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.257+0000 7f4694ba6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e55820 0x55c723e5d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.257+0000 7f468f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e56140 0x55c723e5df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.257+0000 7f4694ba6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e55820 0x55c723e5d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:58276/0 (socket says 172.21.15.70:58276) 2024-06-04T05:01:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.257+0000 7f468f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e56140 0x55c723e5df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59848/0 (socket says 172.21.15.70:59848) 2024-06-04T05:01:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.257+0000 7f4694ba6640 1 -- 172.21.15.70:0/330882467 learned_addr learned my addr 172.21.15.70:0/330882467 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.257+0000 7f4694ba6640 1 --2- 172.21.15.70:0/330882467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e60a00 0x55c723e5e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.258+0000 7f4694ba6640 1 -- 172.21.15.70:0/330882467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e60a00 msgr2=0x55c723e5e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.258+0000 7f4694ba6640 1 --2- 172.21.15.70:0/330882467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e60a00 0x55c723e5e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.258+0000 7f4694ba6640 1 -- 172.21.15.70:0/330882467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e56140 msgr2=0x55c723e5df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.258+0000 7f4694ba6640 1 --2- 172.21.15.70:0/330882467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e56140 0x55c723e5df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.258+0000 7f4694ba6640 1 -- 172.21.15.70:0/330882467 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c723b98620 con 0x55c723e55820 2024-06-04T05:01:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.258+0000 7f468f7fe640 1 --2- 172.21.15.70:0/330882467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e56140 0x55c723e5df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.258+0000 7f4694ba6640 1 --2- 172.21.15.70:0/330882467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e55820 0x55c723e5d9c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4688005c40 tx=0x7f46880107b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffb16da2b467f288 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.258+0000 7f466f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:01:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.258+0000 7f466f7fe640 1 -- 172.21.15.70:0/330882467 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4688024040 con 0x55c723e55820 2024-06-04T05:01:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.258+0000 7f466f7fe640 1 -- 172.21.15.70:0/330882467 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f4688010bb0 con 0x55c723e55820 2024-06-04T05:01:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.258+0000 7f4699404740 1 -- 172.21.15.70:0/330882467 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4688011c70 con 0x55c723e55820 2024-06-04T05:01:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.258+0000 7f466f7fe640 1 -- 172.21.15.70:0/330882467 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4688004a40 con 0x55c723e55820 2024-06-04T05:01:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.258+0000 7f4699404740 10 client.26056 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.258+0000 7f4699404740 1 -- 172.21.15.70:0/330882467 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c723e6f3b0 con 0x55c723e55820 2024-06-04T05:01:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.259+0000 7f466f7fe640 1 -- 172.21.15.70:0/330882467 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4688011c70 con 0x55c723e55820 2024-06-04T05:01:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.259+0000 7f466f7fe640 1 -- 172.21.15.70:0/330882467 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f4688011870 con 0x55c723e55820 2024-06-04T05:01:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.259+0000 7f4699404740 10 client.26056 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.259+0000 7f4699404740 10 client.26056 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.259+0000 7f4699404740 20 client.26056 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.259+0000 7f4699404740 1 --2- 172.21.15.70:0/330882467 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c723e707c0 0x55c723e72bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.259+0000 7f4699404740 4 client.26056 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:01:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.259+0000 7f4699404740 1 -- 172.21.15.70:0/330882467 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c723dc9b80 con 0x55c723e707c0 2024-06-04T05:01:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.259+0000 7f468f7fe640 1 --2- 172.21.15.70:0/330882467 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c723e707c0 0x55c723e72bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.259+0000 7f466f7fe640 10 client.26056 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.259+0000 7f468f7fe640 1 --2- 172.21.15.70:0/330882467 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c723e707c0 0x55c723e72bb0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cd60451ba144978 server_cookie=22da4b6ac7b1f355 in_seq=0 out_seq=0 2024-06-04T05:01:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.260+0000 7f466f7fe640 1 -- 172.21.15.70:0/330882467 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c723dc9b80 con 0x55c723e707c0 2024-06-04T05:01:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.260+0000 7f466f7fe640 10 client.26056 handle_command_reply: tid=0 2024-06-04T05:01:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.260+0000 7f4699404740 1 client.26056 shutdown 2024-06-04T05:01:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.260+0000 7f4699404740 1 -- 172.21.15.70:0/330882467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e55820 msgr2=0x55c723e5d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.260+0000 7f4699404740 1 --2- 172.21.15.70:0/330882467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e55820 0x55c723e5d9c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4688005c40 tx=0x7f46880107b0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.260+0000 7f4699404740 1 -- 172.21.15.70:0/330882467 shutdown_connections 2024-06-04T05:01:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.260+0000 7f4699404740 1 --2- 172.21.15.70:0/330882467 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c723e707c0 0x55c723e72bb0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.260+0000 7f4699404740 1 --2- 172.21.15.70:0/330882467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e60a00 0x55c723e5e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.260+0000 7f4699404740 1 --2- 172.21.15.70:0/330882467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e56140 0x55c723e5df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.260+0000 7f466f7fe640 0 client.26056 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.260+0000 7f4699404740 1 --2- 172.21.15.70:0/330882467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e55820 0x55c723e5d9c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.260+0000 7f4699404740 1 -- 172.21.15.70:0/330882467 >> 172.21.15.70:0/330882467 conn(0x55c723e51d30 msgr2=0x55c723e50fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.260+0000 7f4699404740 1 -- 172.21.15.70:0/330882467 shutdown_connections 2024-06-04T05:01:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.260+0000 7f4699404740 1 -- 172.21.15.70:0/330882467 wait complete. 2024-06-04T05:01:33.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.260+0000 7f4699404740 20 client.26056 trim_cache size 0 max 16384 2024-06-04T05:01:33.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.278+0000 7f4699404740 1 Processor -- start 2024-06-04T05:01:33.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.278+0000 7f4699404740 1 -- start start 2024-06-04T05:01:33.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.278+0000 7f4699404740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e55820 0x55c723e5e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.278+0000 7f4699404740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e56140 0x55c723e50e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.278+0000 7f4699404740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e60a00 0x55c723e51370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.278+0000 7f4699404740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f46901ae760 con 0x55c723e56140 2024-06-04T05:01:33.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.278+0000 7f4699404740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f46901b0300 con 0x55c723e55820 2024-06-04T05:01:33.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.278+0000 7f4699404740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f46901aee30 con 0x55c723e60a00 2024-06-04T05:01:33.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.278+0000 7f4694ba6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e55820 0x55c723e5e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.278+0000 7f468f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e56140 0x55c723e50e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.278+0000 7f4694ba6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e55820 0x55c723e5e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:41488/0 (socket says 172.21.15.70:41488) 2024-06-04T05:01:33.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.278+0000 7f4694ba6640 1 -- 172.21.15.70:0/684931356 learned_addr learned my addr 172.21.15.70:0/684931356 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:33.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.278+0000 7f4694ba6640 1 --2- 172.21.15.70:0/684931356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e60a00 0x55c723e51370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.279+0000 7f4694ba6640 1 -- 172.21.15.70:0/684931356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e60a00 msgr2=0x55c723e51370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:33.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.279+0000 7f4694ba6640 1 --2- 172.21.15.70:0/684931356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e60a00 0x55c723e51370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.279+0000 7f4694ba6640 1 -- 172.21.15.70:0/684931356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e56140 msgr2=0x55c723e50e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:33.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.279+0000 7f4694ba6640 1 --2- 172.21.15.70:0/684931356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e56140 0x55c723e50e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.279+0000 7f4694ba6640 1 -- 172.21.15.70:0/684931356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c723e52900 con 0x55c723e55820 2024-06-04T05:01:33.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.279+0000 7f468f7fe640 1 --2- 172.21.15.70:0/684931356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e56140 0x55c723e50e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:33.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.279+0000 7f4694ba6640 1 --2- 172.21.15.70:0/684931356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e55820 0x55c723e5e260 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4688006010 tx=0x7f4688005cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f97f3750b8184279 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:33.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.279+0000 7f4675ffb640 1 -- 172.21.15.70:0/684931356 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4688004680 con 0x55c723e55820 2024-06-04T05:01:33.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.279+0000 7f4675ffb640 1 -- 172.21.15.70:0/684931356 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4688004820 con 0x55c723e55820 2024-06-04T05:01:33.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.279+0000 7f4675ffb640 1 -- 172.21.15.70:0/684931356 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4688011440 con 0x55c723e55820 2024-06-04T05:01:33.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.280+0000 7f4699404740 1 -- 172.21.15.70:0/684931356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e55820 msgr2=0x55c723e5e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.280+0000 7f4699404740 1 --2- 172.21.15.70:0/684931356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e55820 0x55c723e5e260 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4688006010 tx=0x7f4688005cf0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.280+0000 7f4699404740 1 -- 172.21.15.70:0/684931356 shutdown_connections 2024-06-04T05:01:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.280+0000 7f4699404740 1 --2- 172.21.15.70:0/684931356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e60a00 0x55c723e51370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.280+0000 7f4699404740 1 --2- 172.21.15.70:0/684931356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e56140 0x55c723e50e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.280+0000 7f4699404740 1 --2- 172.21.15.70:0/684931356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e55820 0x55c723e5e260 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.280+0000 7f4699404740 1 -- 172.21.15.70:0/684931356 >> 172.21.15.70:0/684931356 conn(0x55c723e51d30 msgr2=0x55c723e50720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.280+0000 7f4699404740 1 -- 172.21.15.70:0/684931356 shutdown_connections 2024-06-04T05:01:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.280+0000 7f4699404740 1 -- 172.21.15.70:0/684931356 wait complete. 2024-06-04T05:01:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.280+0000 7f4699404740 1 -- start start 2024-06-04T05:01:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.280+0000 7f4699404740 1 Processor -- start 2024-06-04T05:01:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f4699404740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e55820 0x55c723e68b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f4699404740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e56140 0x55c723e69090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f4699404740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e60a00 0x55c723e5a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f4699404740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f46901afbd0 con 0x55c723e55820 2024-06-04T05:01:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f4699404740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f46901af500 con 0x55c723e56140 2024-06-04T05:01:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f4699404740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f46901ac740 con 0x55c723e60a00 2024-06-04T05:01:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f4699404740 10 client.0 _ll_register_callbacks cb 0x7f4688015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f4694ba6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e55820 0x55c723e68b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f4694ba6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e55820 0x55c723e68b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59864/0 (socket says 172.21.15.70:59864) 2024-06-04T05:01:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f468f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e56140 0x55c723e69090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f4694ba6640 1 -- 172.21.15.70:0/2299135357 learned_addr learned my addr 172.21.15.70:0/2299135357 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f4694ba6640 1 --2- 172.21.15.70:0/2299135357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e60a00 0x55c723e5a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f468f7fe640 1 -- 172.21.15.70:0/2299135357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e60a00 msgr2=0x55c723e5a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f468f7fe640 1 --2- 172.21.15.70:0/2299135357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e60a00 0x55c723e5a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f468f7fe640 1 -- 172.21.15.70:0/2299135357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e55820 msgr2=0x55c723e68b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f468f7fe640 1 --2- 172.21.15.70:0/2299135357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e55820 0x55c723e68b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f468f7fe640 1 -- 172.21.15.70:0/2299135357 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c723e5ab40 con 0x55c723e56140 2024-06-04T05:01:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f4694ba6640 1 --2- 172.21.15.70:0/2299135357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e55820 0x55c723e68b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f468f7fe640 1 --2- 172.21.15.70:0/2299135357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e56140 0x55c723e69090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4684009900 tx=0x7f4684004460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8342bbe99775202e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.281+0000 7f466f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:01:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.282+0000 7f466f7fe640 1 -- 172.21.15.70:0/2299135357 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684035040 con 0x55c723e56140 2024-06-04T05:01:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.282+0000 7f466f7fe640 1 -- 172.21.15.70:0/2299135357 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f4684031810 con 0x55c723e56140 2024-06-04T05:01:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.282+0000 7f4699404740 1 -- 172.21.15.70:0/2299135357 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4688004820 con 0x55c723e56140 2024-06-04T05:01:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.282+0000 7f466f7fe640 1 -- 172.21.15.70:0/2299135357 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f46840035f0 con 0x55c723e56140 2024-06-04T05:01:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.282+0000 7f4699404740 10 client.26018 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.282+0000 7f4699404740 1 -- 172.21.15.70:0/2299135357 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c723edad70 con 0x55c723e56140 2024-06-04T05:01:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.282+0000 7f466f7fe640 1 -- 172.21.15.70:0/2299135357 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4688004820 con 0x55c723e56140 2024-06-04T05:01:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.282+0000 7f466f7fe640 1 -- 172.21.15.70:0/2299135357 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f4684045530 con 0x55c723e56140 2024-06-04T05:01:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.282+0000 7f4699404740 10 client.26018 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.282+0000 7f4699404740 10 client.26018 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.282+0000 7f4699404740 20 client.26018 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.282+0000 7f4699404740 1 --2- 172.21.15.70:0/2299135357 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c723edb520 0x55c723edd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.282+0000 7f4699404740 4 client.26018 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:01:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.282+0000 7f4699404740 1 -- 172.21.15.70:0/2299135357 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c723dc3360 con 0x55c723edb520 2024-06-04T05:01:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.282+0000 7f4694ba6640 1 --2- 172.21.15.70:0/2299135357 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c723edb520 0x55c723edd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.283+0000 7f466f7fe640 10 client.26018 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.283+0000 7f4694ba6640 1 --2- 172.21.15.70:0/2299135357 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c723edb520 0x55c723edd940 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da3ef7d2cf080c59 server_cookie=49e463d63b1cb040 in_seq=0 out_seq=0 2024-06-04T05:01:33.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.297+0000 7f466f7fe640 1 -- 172.21.15.70:0/2299135357 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55c723dc3360 con 0x55c723edb520 2024-06-04T05:01:33.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.297+0000 7f466f7fe640 10 client.26018 handle_command_reply: tid=0 2024-06-04T05:01:33.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.297+0000 7f4699404740 1 client.26018 shutdown 2024-06-04T05:01:33.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.298+0000 7f4699404740 1 -- 172.21.15.70:0/2299135357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e56140 msgr2=0x55c723e69090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:33.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.298+0000 7f4699404740 1 --2- 172.21.15.70:0/2299135357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e56140 0x55c723e69090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4684009900 tx=0x7f4684004460 comp rx=0 tx=0).stop 2024-06-04T05:01:33.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.298+0000 7f4699404740 1 -- 172.21.15.70:0/2299135357 shutdown_connections 2024-06-04T05:01:33.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.298+0000 7f4699404740 1 --2- 172.21.15.70:0/2299135357 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c723edb520 0x55c723edd940 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.298+0000 7f4699404740 1 --2- 172.21.15.70:0/2299135357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c723e56140 0x55c723e69090 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.298+0000 7f4699404740 1 --2- 172.21.15.70:0/2299135357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c723e60a00 0x55c723e5a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.298+0000 7f466f7fe640 0 client.26018 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:33.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.298+0000 7f4699404740 1 --2- 172.21.15.70:0/2299135357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c723e55820 0x55c723e68b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.298+0000 7f4699404740 1 -- 172.21.15.70:0/2299135357 >> 172.21.15.70:0/2299135357 conn(0x55c723e51d30 msgr2=0x55c723e50720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:33.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.298+0000 7f4699404740 1 -- 172.21.15.70:0/2299135357 shutdown_connections 2024-06-04T05:01:33.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.298+0000 7f4699404740 1 -- 172.21.15.70:0/2299135357 wait complete. 2024-06-04T05:01:33.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.298+0000 7f4699404740 20 client.26018 trim_cache size 0 max 16384 2024-06-04T05:01:33.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:01:33.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:01:33.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:01:33.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.298+0000 7f4696609640 1 -- 172.21.15.70:0/1672348777 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f46780b2a10 msgr2=0x7f46780b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:33.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.298+0000 7f4696609640 1 --2- 172.21.15.70:0/1672348777 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f46780b2a10 0x7f46780b4ed0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f46800df9c0 tx=0x7f468000bf60 comp rx=0 tx=0).stop 2024-06-04T05:01:33.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.298+0000 7f4696609640 1 -- 172.21.15.70:0/1672348777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4690102a40 msgr2=0x7f4690199190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:33.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.298+0000 7f4696609640 1 --2- 172.21.15.70:0/1672348777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4690102a40 0x7f4690199190 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f468000d950 tx=0x7f468000de10 comp rx=0 tx=0).stop 2024-06-04T05:01:33.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.299+0000 7f4696609640 1 -- 172.21.15.70:0/1672348777 shutdown_connections 2024-06-04T05:01:33.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.299+0000 7f4696609640 1 --2- 172.21.15.70:0/1672348777 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f46780b2a10 0x7f46780b4ed0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.299+0000 7f4696609640 1 --2- 172.21.15.70:0/1672348777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4690108a50 0x7f469019da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.299+0000 7f4696609640 1 --2- 172.21.15.70:0/1672348777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4690103400 0x7f46901996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.299+0000 7f4696609640 1 --2- 172.21.15.70:0/1672348777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4690102a40 0x7f4690199190 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:33.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.299+0000 7f4696609640 1 -- 172.21.15.70:0/1672348777 >> 172.21.15.70:0/1672348777 conn(0x7f46900fe510 msgr2=0x7f46900fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:33.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.299+0000 7f4696609640 1 -- 172.21.15.70:0/1672348777 shutdown_connections 2024-06-04T05:01:33.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:33.299+0000 7f4696609640 1 -- 172.21.15.70:0/1672348777 wait complete. 2024-06-04T05:01:33.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:33 smithi112 ceph-mon[36453]: pgmap v134: 97 pgs: 97 active+clean; 44 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 544 KiB/s wr, 62 op/s 2024-06-04T05:01:33.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:33 smithi098 ceph-mon[35569]: pgmap v134: 97 pgs: 97 active+clean; 44 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 544 KiB/s wr, 62 op/s 2024-06-04T05:01:33.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:33 smithi070 ceph-mon[32631]: pgmap v134: 97 pgs: 97 active+clean; 44 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 544 KiB/s wr, 62 op/s 2024-06-04T05:01:33.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 48919 0.70 MB/sec execute 26 sec latency 16.516 ms 2024-06-04T05:01:34.658 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:01:34.658 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:01:34.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 49479 0.80 MB/sec execute 27 sec latency 15.187 ms 2024-06-04T05:01:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:01:35.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:35 smithi070 ceph-mon[32631]: pgmap v135: 97 pgs: 97 active+clean; 44 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 538 KiB/s wr, 55 op/s 2024-06-04T05:01:35.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:35 smithi112 ceph-mon[36453]: pgmap v135: 97 pgs: 97 active+clean; 44 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 538 KiB/s wr, 55 op/s 2024-06-04T05:01:35.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:35 smithi098 ceph-mon[35569]: pgmap v135: 97 pgs: 97 active+clean; 44 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 538 KiB/s wr, 55 op/s 2024-06-04T05:01:35.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 49963 0.93 MB/sec execute 28 sec latency 68.172 ms 2024-06-04T05:01:36.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 50250 0.91 MB/sec execute 29 sec latency 11.767 ms 2024-06-04T05:01:37.336 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:37 smithi070 ceph-mon[32631]: pgmap v136: 97 pgs: 97 active+clean; 43 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-06-04T05:01:37.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:37 smithi112 ceph-mon[36453]: pgmap v136: 97 pgs: 97 active+clean; 43 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-06-04T05:01:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.453+0000 7fc691322700 1 -- 172.21.15.70:0/1598181398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc68c103fa0 msgr2=0x7fc68c110450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.453+0000 7fc691322700 1 --2- 172.21.15.70:0/1598181398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc68c103fa0 0x7fc68c110450 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fc68400b0d0 tx=0x7fc68400b3e0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.453+0000 7fc691322700 1 -- 172.21.15.70:0/1598181398 shutdown_connections 2024-06-04T05:01:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.453+0000 7fc691322700 1 --2- 172.21.15.70:0/1598181398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc68c103fa0 0x7fc68c110450 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.453+0000 7fc691322700 1 --2- 172.21.15.70:0/1598181398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc68c109680 0x7fc68c109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.453+0000 7fc691322700 1 --2- 172.21.15.70:0/1598181398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc68c1035e0 0x7fc68c103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.453+0000 7fc691322700 1 -- 172.21.15.70:0/1598181398 >> 172.21.15.70:0/1598181398 conn(0x7fc68c07a6f0 msgr2=0x7fc68c07ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.453+0000 7fc691322700 1 -- 172.21.15.70:0/1598181398 shutdown_connections 2024-06-04T05:01:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.453+0000 7fc691322700 1 -- 172.21.15.70:0/1598181398 wait complete. 2024-06-04T05:01:37.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.453+0000 7fc691322700 1 Processor -- start 2024-06-04T05:01:37.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.454+0000 7fc691322700 1 -- start start 2024-06-04T05:01:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.454+0000 7fc691322700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc68c1035e0 0x7fc68c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.454+0000 7fc691322700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc68c103fa0 0x7fc68c19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.454+0000 7fc691322700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc68c109680 0x7fc68c199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.454+0000 7fc691322700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c113650 con 0x7fc68c1035e0 2024-06-04T05:01:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.454+0000 7fc691322700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c1134d0 con 0x7fc68c103fa0 2024-06-04T05:01:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.454+0000 7fc691322700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c1137d0 con 0x7fc68c109680 2024-06-04T05:01:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.454+0000 7fc68affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc68c1035e0 0x7fc68c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.454+0000 7fc68a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc68c103fa0 0x7fc68c19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.454+0000 7fc68affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc68c1035e0 0x7fc68c19f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52986/0 (socket says 172.21.15.70:52986) 2024-06-04T05:01:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.454+0000 7fc68affd700 1 -- 172.21.15.70:0/1825876687 learned_addr learned my addr 172.21.15.70:0/1825876687 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.454+0000 7fc68b7fe700 1 --2- 172.21.15.70:0/1825876687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc68c109680 0x7fc68c199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.454+0000 7fc68a7fc700 1 -- 172.21.15.70:0/1825876687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc68c109680 msgr2=0x7fc68c199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.454+0000 7fc68a7fc700 1 --2- 172.21.15.70:0/1825876687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc68c109680 0x7fc68c199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.454+0000 7fc68a7fc700 1 -- 172.21.15.70:0/1825876687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc68c1035e0 msgr2=0x7fc68c19f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.454+0000 7fc68a7fc700 1 --2- 172.21.15.70:0/1825876687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc68c1035e0 0x7fc68c19f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.454+0000 7fc68a7fc700 1 -- 172.21.15.70:0/1825876687 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc68c19a090 con 0x7fc68c103fa0 2024-06-04T05:01:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.455+0000 7fc68a7fc700 1 --2- 172.21.15.70:0/1825876687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc68c103fa0 0x7fc68c19fc60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc68000e9f0 tx=0x7fc68000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.455+0000 7fc673fff700 1 -- 172.21.15.70:0/1825876687 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc680004220 con 0x7fc68c103fa0 2024-06-04T05:01:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.455+0000 7fc673fff700 1 -- 172.21.15.70:0/1825876687 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc6800043c0 con 0x7fc68c103fa0 2024-06-04T05:01:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.455+0000 7fc691322700 1 -- 172.21.15.70:0/1825876687 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc68c19a380 con 0x7fc68c103fa0 2024-06-04T05:01:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.455+0000 7fc673fff700 1 -- 172.21.15.70:0/1825876687 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68000ce30 con 0x7fc68c103fa0 2024-06-04T05:01:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.455+0000 7fc691322700 1 -- 172.21.15.70:0/1825876687 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc68c1a65f0 con 0x7fc68c103fa0 2024-06-04T05:01:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.458+0000 7fc673fff700 1 -- 172.21.15.70:0/1825876687 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fc6800050a0 con 0x7fc68c103fa0 2024-06-04T05:01:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.458+0000 7fc673fff700 1 --2- 172.21.15.70:0/1825876687 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc6740b2a70 0x7fc6740b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc68affd700 1 --2- 172.21.15.70:0/1825876687 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc6740b2a70 0x7fc6740b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc69bbb7b80 1 Processor -- start 2024-06-04T05:01:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc673fff700 1 -- 172.21.15.70:0/1825876687 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc6800e25c0 con 0x7fc68c103fa0 2024-06-04T05:01:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc69bbb7b80 1 -- start start 2024-06-04T05:01:37.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc69bbb7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcecf0 0x564db2fcf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc69bbb7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fcf6a0 0x564db2fd2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc69bbb7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 0x564db2fd3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc69bbb7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564db2f29d10 con 0x564db2fd33f0 2024-06-04T05:01:37.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc69bbb7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564db2f198b0 con 0x564db2fcecf0 2024-06-04T05:01:37.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc69bbb7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564db2f1d8f0 con 0x564db2fcf6a0 2024-06-04T05:01:37.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc68affd700 1 --2- 172.21.15.70:0/1825876687 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc6740b2a70 0x7fc6740b4f30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc68c1090d0 tx=0x7fc67c009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:37.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc68affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcecf0 0x564db2fcf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc68affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcecf0 0x564db2fcf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:38374/0 (socket says 172.21.15.70:38374) 2024-06-04T05:01:37.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc68affd700 1 -- 172.21.15.70:0/546587429 learned_addr learned my addr 172.21.15.70:0/546587429 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:37.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc68a7fc700 1 --2- 172.21.15.70:0/546587429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fcf6a0 0x564db2fd2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc68b7fe700 1 --2- 172.21.15.70:0/546587429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 0x564db2fd3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc68affd700 1 -- 172.21.15.70:0/546587429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fcf6a0 msgr2=0x564db2fd2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:37.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc68affd700 1 --2- 172.21.15.70:0/546587429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fcf6a0 0x564db2fd2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc68affd700 1 -- 172.21.15.70:0/546587429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 msgr2=0x564db2fd3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:37.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.459+0000 7fc68affd700 1 --2- 172.21.15.70:0/546587429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 0x564db2fd3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.460+0000 7fc68affd700 1 -- 172.21.15.70:0/546587429 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564db2e69950 con 0x564db2fcecf0 2024-06-04T05:01:37.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.460+0000 7fc68affd700 1 --2- 172.21.15.70:0/546587429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcecf0 0x564db2fcf0d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00e540 tx=0x7fc67c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20daec7486489c2d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:37.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.460+0000 7fc6717fa700 1 -- 172.21.15.70:0/546587429 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c00ee60 con 0x564db2fcecf0 2024-06-04T05:01:37.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.460+0000 7fc6717fa700 1 -- 172.21.15.70:0/546587429 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc67c004cc0 con 0x564db2fcecf0 2024-06-04T05:01:37.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.460+0000 7fc6717fa700 1 -- 172.21.15.70:0/546587429 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c0266e0 con 0x564db2fcecf0 2024-06-04T05:01:37.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.461+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/546587429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcecf0 msgr2=0x564db2fcf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:37.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.461+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/546587429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcecf0 0x564db2fcf0d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00e540 tx=0x7fc67c014000 comp rx=0 tx=0).stop 2024-06-04T05:01:37.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.461+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/546587429 shutdown_connections 2024-06-04T05:01:37.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.461+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/546587429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 0x564db2fd3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.461+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/546587429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcecf0 0x564db2fcf0d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.461+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/546587429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fcf6a0 0x564db2fd2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.461+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/546587429 >> 172.21.15.70:0/546587429 conn(0x564db2ef8100 msgr2=0x564db2fc4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:37.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.461+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/546587429 shutdown_connections 2024-06-04T05:01:37.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.461+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/546587429 wait complete. 2024-06-04T05:01:37.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.461+0000 7fc69bbb7b80 1 -- start start 2024-06-04T05:01:37.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.461+0000 7fc69bbb7b80 1 Processor -- start 2024-06-04T05:01:37.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc69bbb7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fcecf0 0x564db2fda820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc69bbb7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcf6a0 0x564db2fd7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc69bbb7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 0x564db2fd8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc69bbb7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564db2e20020 con 0x564db2fd33f0 2024-06-04T05:01:37.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc69bbb7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564db2ebb840 con 0x564db2fcf6a0 2024-06-04T05:01:37.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc69bbb7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564db2fdb5b0 con 0x564db2fcecf0 2024-06-04T05:01:37.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc69bbb7b80 10 client.0 _ll_register_callbacks cb 0x564db2eb49a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:37.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc68b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 0x564db2fd8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc68a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcf6a0 0x564db2fd7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc68affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fcecf0 0x564db2fda820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc68b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 0x564db2fd8060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53014/0 (socket says 172.21.15.70:53014) 2024-06-04T05:01:37.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc68b7fe700 1 -- 172.21.15.70:0/1378028511 learned_addr learned my addr 172.21.15.70:0/1378028511 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:37.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc68a7fc700 1 --2- 172.21.15.70:0/1378028511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcf6a0 0x564db2fd7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:38382/0 (socket says 172.21.15.70:38382) 2024-06-04T05:01:37.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc68a7fc700 1 -- 172.21.15.70:0/1378028511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fcecf0 msgr2=0x564db2fda820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:37.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc68a7fc700 1 --2- 172.21.15.70:0/1378028511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fcecf0 0x564db2fda820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc68a7fc700 1 -- 172.21.15.70:0/1378028511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 msgr2=0x564db2fd8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:37.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc68a7fc700 1 --2- 172.21.15.70:0/1378028511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 0x564db2fd8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc68a7fc700 1 -- 172.21.15.70:0/1378028511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564db2eb8710 con 0x564db2fcf6a0 2024-06-04T05:01:37.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc68a7fc700 1 --2- 172.21.15.70:0/1378028511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcf6a0 0x564db2fd7b20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc6800dfcb0 tx=0x7fc6800cc2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f981646fd5d02a71 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:37.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc66eff5700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:01:37.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc66eff5700 1 -- 172.21.15.70:0/1378028511 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6800c3060 con 0x564db2fcf6a0 2024-06-04T05:01:37.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc66eff5700 1 -- 172.21.15.70:0/1378028511 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc6800a9cb0 con 0x564db2fcf6a0 2024-06-04T05:01:37.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.462+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/1378028511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc67c004cc0 con 0x564db2fcf6a0 2024-06-04T05:01:37.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.463+0000 7fc66eff5700 1 -- 172.21.15.70:0/1378028511 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc68001c040 con 0x564db2fcf6a0 2024-06-04T05:01:37.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.463+0000 7fc69bbb7b80 10 client.26030 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:37.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.463+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/1378028511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564db2fe5d20 con 0x564db2fcf6a0 2024-06-04T05:01:37.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.463+0000 7fc66eff5700 1 -- 172.21.15.70:0/1378028511 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc6800c6030 con 0x564db2fcf6a0 2024-06-04T05:01:37.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.463+0000 7fc66eff5700 1 -- 172.21.15.70:0/1378028511 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc6800b2210 con 0x564db2fcf6a0 2024-06-04T05:01:37.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.463+0000 7fc69bbb7b80 10 client.26030 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:37.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.463+0000 7fc69bbb7b80 10 client.26030 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:37.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.463+0000 7fc69bbb7b80 20 client.26030 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:37.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.463+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/1378028511 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564db2fe83b0 0x564db2fea7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.463+0000 7fc69bbb7b80 4 client.26030 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:01:37.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.463+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/1378028511 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564db2f96a60 con 0x564db2fe83b0 2024-06-04T05:01:37.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.463+0000 7fc68affd700 1 --2- 172.21.15.70:0/1378028511 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564db2fe83b0 0x564db2fea7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.463+0000 7fc68affd700 1 --2- 172.21.15.70:0/1378028511 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564db2fe83b0 0x564db2fea7d0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ff01f2e9b5e7300 server_cookie=6ceb154647d8b033 in_seq=0 out_seq=0 2024-06-04T05:01:37.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.463+0000 7fc66eff5700 10 client.26030 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:37.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.464+0000 7fc66eff5700 1 -- 172.21.15.70:0/1378028511 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564db2f96a60 con 0x564db2fe83b0 2024-06-04T05:01:37.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.464+0000 7fc66eff5700 10 client.26030 handle_command_reply: tid=0 2024-06-04T05:01:37.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.464+0000 7fc69bbb7b80 1 client.26030 shutdown 2024-06-04T05:01:37.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.464+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/1378028511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcf6a0 msgr2=0x564db2fd7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:37.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.464+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/1378028511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcf6a0 0x564db2fd7b20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc6800dfcb0 tx=0x7fc6800cc2d0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.464+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/1378028511 shutdown_connections 2024-06-04T05:01:37.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.464+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/1378028511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 0x564db2fd8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.464+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/1378028511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fcecf0 0x564db2fda820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.464+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/1378028511 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564db2fe83b0 0x564db2fea7d0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.464+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/1378028511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcf6a0 0x564db2fd7b20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.464+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/1378028511 >> 172.21.15.70:0/1378028511 conn(0x564db2ef8100 msgr2=0x564db2fc4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.464+0000 7fc66eff5700 0 client.26030 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.465+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/1378028511 shutdown_connections 2024-06-04T05:01:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.465+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/1378028511 wait complete. 2024-06-04T05:01:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.465+0000 7fc69bbb7b80 20 client.26030 trim_cache size 0 max 16384 2024-06-04T05:01:37.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.489+0000 7fc69bbb7b80 1 Processor -- start 2024-06-04T05:01:37.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.489+0000 7fc69bbb7b80 1 -- start start 2024-06-04T05:01:37.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.489+0000 7fc69bbb7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fcecf0 0x564db2fd2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.489+0000 7fc69bbb7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcf6a0 0x564db2fc4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.489+0000 7fc69bbb7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 0x564db2fc5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.489+0000 7fc69bbb7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c1abb70 con 0x564db2fd33f0 2024-06-04T05:01:37.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.489+0000 7fc69bbb7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c1ad710 con 0x564db2fcf6a0 2024-06-04T05:01:37.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.489+0000 7fc69bbb7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c1ac240 con 0x564db2fcecf0 2024-06-04T05:01:37.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.489+0000 7fc68b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 0x564db2fc5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.490+0000 7fc68b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 0x564db2fc5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53026/0 (socket says 172.21.15.70:53026) 2024-06-04T05:01:37.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.490+0000 7fc68b7fe700 1 -- 172.21.15.70:0/2066241367 learned_addr learned my addr 172.21.15.70:0/2066241367 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:37.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.490+0000 7fc68affd700 1 --2- 172.21.15.70:0/2066241367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fcecf0 0x564db2fd2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.490+0000 7fc68a7fc700 1 --2- 172.21.15.70:0/2066241367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcf6a0 0x564db2fc4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.490+0000 7fc68b7fe700 1 -- 172.21.15.70:0/2066241367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fcecf0 msgr2=0x564db2fd2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:37.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.490+0000 7fc68b7fe700 1 --2- 172.21.15.70:0/2066241367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fcecf0 0x564db2fd2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.490+0000 7fc68b7fe700 1 -- 172.21.15.70:0/2066241367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcf6a0 msgr2=0x564db2fc4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:37.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.490+0000 7fc68b7fe700 1 --2- 172.21.15.70:0/2066241367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcf6a0 0x564db2fc4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.490+0000 7fc68b7fe700 1 -- 172.21.15.70:0/2066241367 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564db2fc65a0 con 0x564db2fd33f0 2024-06-04T05:01:37.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.490+0000 7fc68b7fe700 1 --2- 172.21.15.70:0/2066241367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 0x564db2fc5250 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fc684009550 tx=0x7fc684009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8775e1df7aaaff99 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:37.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.490+0000 7fc670ff9700 1 -- 172.21.15.70:0/2066241367 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68400bd20 con 0x564db2fd33f0 2024-06-04T05:01:37.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.490+0000 7fc670ff9700 1 -- 172.21.15.70:0/2066241367 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc6840074e0 con 0x564db2fd33f0 2024-06-04T05:01:37.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.490+0000 7fc670ff9700 1 -- 172.21.15.70:0/2066241367 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc684005320 con 0x564db2fd33f0 2024-06-04T05:01:37.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:37 smithi098 ceph-mon[35569]: pgmap v136: 97 pgs: 97 active+clean; 43 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-06-04T05:01:37.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.491+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/2066241367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 msgr2=0x564db2fc5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:37.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.491+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/2066241367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 0x564db2fc5250 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fc684009550 tx=0x7fc684009580 comp rx=0 tx=0).stop 2024-06-04T05:01:37.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.491+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/2066241367 shutdown_connections 2024-06-04T05:01:37.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.491+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/2066241367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fd33f0 0x564db2fc5250 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.491+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/2066241367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fcecf0 0x564db2fd2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.491+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/2066241367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcf6a0 0x564db2fc4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.491+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/2066241367 >> 172.21.15.70:0/2066241367 conn(0x564db2ef8100 msgr2=0x564db2fd9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:37.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.491+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/2066241367 shutdown_connections 2024-06-04T05:01:37.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.491+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/2066241367 wait complete. 2024-06-04T05:01:37.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.491+0000 7fc69bbb7b80 1 -- start start 2024-06-04T05:01:37.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc69bbb7b80 1 Processor -- start 2024-06-04T05:01:37.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc69bbb7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fcecf0 0x564db2fdfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc69bbb7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcf6a0 0x564db2fe0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc69bbb7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fd33f0 0x564db304cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc69bbb7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c1acfe0 con 0x564db2fcecf0 2024-06-04T05:01:37.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc69bbb7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c1ac910 con 0x564db2fcf6a0 2024-06-04T05:01:37.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc69bbb7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c1a9b10 con 0x564db2fd33f0 2024-06-04T05:01:37.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc69bbb7b80 10 client.0 _ll_register_callbacks cb 0x564db2eb49a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:37.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc68affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fcecf0 0x564db2fdfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc68affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fcecf0 0x564db2fdfe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53030/0 (socket says 172.21.15.70:53030) 2024-06-04T05:01:37.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc68affd700 1 -- 172.21.15.70:0/2767435720 learned_addr learned my addr 172.21.15.70:0/2767435720 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:37.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc68a7fc700 1 --2- 172.21.15.70:0/2767435720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcf6a0 0x564db2fe0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc68b7fe700 1 --2- 172.21.15.70:0/2767435720 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fd33f0 0x564db304cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc68affd700 1 -- 172.21.15.70:0/2767435720 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fd33f0 msgr2=0x564db304cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:37.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc68affd700 1 --2- 172.21.15.70:0/2767435720 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fd33f0 0x564db304cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc68affd700 1 -- 172.21.15.70:0/2767435720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcf6a0 msgr2=0x564db2fe0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:37.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc68affd700 1 --2- 172.21.15.70:0/2767435720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcf6a0 0x564db2fe0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc68affd700 1 -- 172.21.15.70:0/2767435720 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564db304d580 con 0x564db2fcecf0 2024-06-04T05:01:37.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc68affd700 1 --2- 172.21.15.70:0/2767435720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fcecf0 0x564db2fdfe10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00e540 tx=0x7fc67c005b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=565a6485b792ffe9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:37.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.492+0000 7fc66eff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:37.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.493+0000 7fc66eff5700 1 -- 172.21.15.70:0/2767435720 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c005690 con 0x564db2fcecf0 2024-06-04T05:01:37.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.493+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/2767435720 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6840074e0 con 0x564db2fcecf0 2024-06-04T05:01:37.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.493+0000 7fc66eff5700 1 -- 172.21.15.70:0/2767435720 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fc67c004710 con 0x564db2fcecf0 2024-06-04T05:01:37.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.493+0000 7fc66eff5700 1 -- 172.21.15.70:0/2767435720 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc67c004910 con 0x564db2fcecf0 2024-06-04T05:01:37.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.493+0000 7fc69bbb7b80 10 client.16725 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:37.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.493+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/2767435720 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564db3054930 con 0x564db2fcecf0 2024-06-04T05:01:37.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.493+0000 7fc66eff5700 1 -- 172.21.15.70:0/2767435720 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc6840074e0 con 0x564db2fcecf0 2024-06-04T05:01:37.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.493+0000 7fc66eff5700 1 -- 172.21.15.70:0/2767435720 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc67c02e040 con 0x564db2fcecf0 2024-06-04T05:01:37.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.493+0000 7fc69bbb7b80 10 client.16725 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:37.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.493+0000 7fc69bbb7b80 10 client.16725 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:37.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.493+0000 7fc69bbb7b80 20 client.16725 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:37.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.493+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/2767435720 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564db3054d00 0x564db30570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:37.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.493+0000 7fc69bbb7b80 4 client.16725 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:01:37.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.493+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/2767435720 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564db2f2bc60 con 0x564db3054d00 2024-06-04T05:01:37.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.493+0000 7fc68a7fc700 1 --2- 172.21.15.70:0/2767435720 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564db3054d00 0x564db30570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:37.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.494+0000 7fc68a7fc700 1 --2- 172.21.15.70:0/2767435720 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564db3054d00 0x564db30570f0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c54548137a0cbc6 server_cookie=feb1db90a2c8824e in_seq=0 out_seq=0 2024-06-04T05:01:37.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.494+0000 7fc66eff5700 10 client.16725 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:37.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.495+0000 7fc66eff5700 1 -- 172.21.15.70:0/2767435720 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564db2f2bc60 con 0x564db3054d00 2024-06-04T05:01:37.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.495+0000 7fc66eff5700 10 client.16725 handle_command_reply: tid=0 2024-06-04T05:01:37.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.495+0000 7fc69bbb7b80 1 client.16725 shutdown 2024-06-04T05:01:37.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.495+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/2767435720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fcecf0 msgr2=0x564db2fdfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:37.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.495+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/2767435720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fcecf0 0x564db2fdfe10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00e540 tx=0x7fc67c005b40 comp rx=0 tx=0).stop 2024-06-04T05:01:37.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.495+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/2767435720 shutdown_connections 2024-06-04T05:01:37.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.495+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/2767435720 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564db2fd33f0 0x564db304cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.495+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/2767435720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564db2fcecf0 0x564db2fdfe10 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.495+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/2767435720 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564db3054d00 0x564db30570f0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.495+0000 7fc69bbb7b80 1 --2- 172.21.15.70:0/2767435720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564db2fcf6a0 0x564db2fe0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.495+0000 7fc66eff5700 0 client.16725 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:37.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.495+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/2767435720 >> 172.21.15.70:0/2767435720 conn(0x564db2ef8100 msgr2=0x564db2fe3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:37.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.495+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/2767435720 shutdown_connections 2024-06-04T05:01:37.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.495+0000 7fc69bbb7b80 1 -- 172.21.15.70:0/2767435720 wait complete. 2024-06-04T05:01:37.504 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:01:37.504 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:01:37.504 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "b53f3ea4-8906-4d26-a6a4-2a9b668c1c67", 2024-06-04T05:01:37.504 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:01:37.504 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:01:37.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.495+0000 7fc69bbb7b80 20 client.16725 trim_cache size 0 max 16384 2024-06-04T05:01:37.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.496+0000 7fc691322700 1 -- 172.21.15.70:0/1825876687 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc6740b2a70 msgr2=0x7fc6740b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:37.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.496+0000 7fc691322700 1 --2- 172.21.15.70:0/1825876687 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc6740b2a70 0x7fc6740b4f30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc68c1090d0 tx=0x7fc67c009260 comp rx=0 tx=0).stop 2024-06-04T05:01:37.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.496+0000 7fc691322700 1 -- 172.21.15.70:0/1825876687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc68c103fa0 msgr2=0x7fc68c19fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:37.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.496+0000 7fc691322700 1 --2- 172.21.15.70:0/1825876687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc68c103fa0 0x7fc68c19fc60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc68000e9f0 tx=0x7fc68000ed00 comp rx=0 tx=0).stop 2024-06-04T05:01:37.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.496+0000 7fc691322700 1 -- 172.21.15.70:0/1825876687 shutdown_connections 2024-06-04T05:01:37.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.496+0000 7fc691322700 1 --2- 172.21.15.70:0/1825876687 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc6740b2a70 0x7fc6740b4f30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.496+0000 7fc691322700 1 --2- 172.21.15.70:0/1825876687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc68c109680 0x7fc68c199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.496+0000 7fc691322700 1 --2- 172.21.15.70:0/1825876687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc68c1035e0 0x7fc68c19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.496+0000 7fc691322700 1 --2- 172.21.15.70:0/1825876687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc68c103fa0 0x7fc68c19fc60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:37.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.496+0000 7fc691322700 1 -- 172.21.15.70:0/1825876687 >> 172.21.15.70:0/1825876687 conn(0x7fc68c07a6f0 msgr2=0x7fc68c1022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:37.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.496+0000 7fc691322700 1 -- 172.21.15.70:0/1825876687 shutdown_connections 2024-06-04T05:01:37.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:37.496+0000 7fc691322700 1 -- 172.21.15.70:0/1825876687 wait complete. 2024-06-04T05:01:37.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 50729 0.94 MB/sec execute 30 sec latency 67.012 ms 2024-06-04T05:01:38.380 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:01:38.403 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:38 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:01:38.403 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:38 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.403 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:38 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:01:38.403 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:38 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:01:38.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:38 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:38 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:01:38.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:38 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:01:38.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:38 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:38 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:38 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:01:38.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:38 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:01:38.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:38 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:38 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:01:38.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:38 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:38 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:38 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:38 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:01:38.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:38 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:38 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:01:38.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:38 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:01:38.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:38 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:38 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:01:38.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:38 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:01:38.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:38 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:38 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:38 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:01:38.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:38 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:01:38.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:38 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:38 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:01:38.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:38 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:38 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:38 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:38 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:01:38.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:38 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:38 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:01:38.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:38 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:01:38.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:38 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:38 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:01:38.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:38 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:01:38.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:38 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:38 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:38 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:01:38.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:38 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:01:38.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:38 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:38 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:01:38.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:38 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:38 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:38 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:01:38.649 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:01:38.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 51283 1.02 MB/sec execute 31 sec latency 53.468 ms 2024-06-04T05:01:39.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:39 smithi070 ceph-mon[32631]: pgmap v137: 97 pgs: 97 active+clean; 43 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-06-04T05:01:39.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:39 smithi112 ceph-mon[36453]: pgmap v137: 97 pgs: 97 active+clean; 43 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-06-04T05:01:39.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:39 smithi098 ceph-mon[35569]: pgmap v137: 97 pgs: 97 active+clean; 43 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-06-04T05:01:39.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 51990 1.04 MB/sec execute 32 sec latency 15.163 ms 2024-06-04T05:01:40.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.468+0000 7f0c7a174700 1 -- 172.21.15.70:0/1132688367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c74102440 msgr2=0x7f0c741028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:40.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.468+0000 7f0c7a174700 1 --2- 172.21.15.70:0/1132688367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c74102440 0x7f0c741028c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f0c68009990 tx=0x7f0c68009ca0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.468+0000 7f0c7a174700 1 -- 172.21.15.70:0/1132688367 shutdown_connections 2024-06-04T05:01:40.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.468+0000 7f0c7a174700 1 --2- 172.21.15.70:0/1132688367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0c74102e00 0x7f0c7410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.468+0000 7f0c7a174700 1 --2- 172.21.15.70:0/1132688367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c74108520 0x7f0c74108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.468+0000 7f0c7a174700 1 --2- 172.21.15.70:0/1132688367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c74102440 0x7f0c741028c0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.468+0000 7f0c7a174700 1 -- 172.21.15.70:0/1132688367 >> 172.21.15.70:0/1132688367 conn(0x7f0c740793f0 msgr2=0x7f0c74079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:40.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.468+0000 7f0c7a174700 1 -- 172.21.15.70:0/1132688367 shutdown_connections 2024-06-04T05:01:40.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.468+0000 7f0c7a174700 1 -- 172.21.15.70:0/1132688367 wait complete. 2024-06-04T05:01:40.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.469+0000 7f0c7a174700 1 Processor -- start 2024-06-04T05:01:40.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.469+0000 7f0c7a174700 1 -- start start 2024-06-04T05:01:40.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.469+0000 7f0c7a174700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c74102440 0x7f0c7419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.469+0000 7f0c7a174700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0c74102e00 0x7f0c7419fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.469+0000 7f0c7a174700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c74108520 0x7f0c74199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.469+0000 7f0c7a174700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0c741124a0 con 0x7f0c74108520 2024-06-04T05:01:40.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.469+0000 7f0c7a174700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c74112320 con 0x7f0c74102440 2024-06-04T05:01:40.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.470+0000 7f0c7a174700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0c74112620 con 0x7f0c74102e00 2024-06-04T05:01:40.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.470+0000 7f0c737fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c74102440 0x7f0c7419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.470+0000 7f0c72ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0c74102e00 0x7f0c7419fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.470+0000 7f0c72ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0c74102e00 0x7f0c7419fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:52908/0 (socket says 172.21.15.70:52908) 2024-06-04T05:01:40.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.470+0000 7f0c737fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c74102440 0x7f0c7419f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:38428/0 (socket says 172.21.15.70:38428) 2024-06-04T05:01:40.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.470+0000 7f0c72ffd700 1 -- 172.21.15.70:0/1584833754 learned_addr learned my addr 172.21.15.70:0/1584833754 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:40.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.470+0000 7f0c73fff700 1 --2- 172.21.15.70:0/1584833754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c74108520 0x7f0c74199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.470+0000 7f0c72ffd700 1 -- 172.21.15.70:0/1584833754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c74102440 msgr2=0x7f0c7419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:40.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.470+0000 7f0c72ffd700 1 --2- 172.21.15.70:0/1584833754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c74102440 0x7f0c7419f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.470+0000 7f0c72ffd700 1 -- 172.21.15.70:0/1584833754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c74108520 msgr2=0x7f0c74199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:40.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.470+0000 7f0c72ffd700 1 --2- 172.21.15.70:0/1584833754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c74108520 0x7f0c74199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.470+0000 7f0c72ffd700 1 -- 172.21.15.70:0/1584833754 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7419a0a0 con 0x7f0c74102e00 2024-06-04T05:01:40.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.470+0000 7f0c737fe700 1 --2- 172.21.15.70:0/1584833754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c74102440 0x7f0c7419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:40.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.470+0000 7f0c72ffd700 1 --2- 172.21.15.70:0/1584833754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0c74102e00 0x7f0c7419fcc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0c68006010 tx=0x7f0c680047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:40.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.470+0000 7f0c70ff9700 1 -- 172.21.15.70:0/1584833754 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c68027070 con 0x7f0c74102e00 2024-06-04T05:01:40.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.470+0000 7f0c7a174700 1 -- 172.21.15.70:0/1584833754 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c7419a330 con 0x7f0c74102e00 2024-06-04T05:01:40.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.470+0000 7f0c7a174700 1 -- 172.21.15.70:0/1584833754 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c741a6650 con 0x7f0c74102e00 2024-06-04T05:01:40.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.471+0000 7f0c70ff9700 1 -- 172.21.15.70:0/1584833754 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c68022470 con 0x7f0c74102e00 2024-06-04T05:01:40.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.471+0000 7f0c70ff9700 1 -- 172.21.15.70:0/1584833754 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c68017650 con 0x7f0c74102e00 2024-06-04T05:01:40.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.474+0000 7f0c70ff9700 1 -- 172.21.15.70:0/1584833754 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f0c68009d10 con 0x7f0c74102e00 2024-06-04T05:01:40.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.475+0000 7f0c70ff9700 1 --2- 172.21.15.70:0/1584833754 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0c5c0b29f0 0x7f0c5c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.475+0000 7f0c84a09b80 1 Processor -- start 2024-06-04T05:01:40.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.475+0000 7f0c737fe700 1 --2- 172.21.15.70:0/1584833754 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0c5c0b29f0 0x7f0c5c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.475+0000 7f0c84a09b80 1 -- start start 2024-06-04T05:01:40.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.475+0000 7f0c70ff9700 1 -- 172.21.15.70:0/1584833754 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f0c680e3050 con 0x7f0c74102e00 2024-06-04T05:01:40.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.475+0000 7f0c84a09b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb6320 0x56187efb6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.475+0000 7f0c84a09b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb08a0 0x56187efb0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.475+0000 7f0c737fe700 1 --2- 172.21.15.70:0/1584833754 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0c5c0b29f0 0x7f0c5c0b4eb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0c7419aa80 tx=0x7f0c64009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:40.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.475+0000 7f0c737fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb6320 0x56187efb6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.475+0000 7f0c84a09b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187efb1260 0x56187efbd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.475+0000 7f0c737fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb6320 0x56187efb6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:52920/0 (socket says 172.21.15.70:52920) 2024-06-04T05:01:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.475+0000 7f0c84a09b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56187ef0ed70 con 0x56187efb08a0 2024-06-04T05:01:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.475+0000 7f0c84a09b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56187eefe910 con 0x56187efb1260 2024-06-04T05:01:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.475+0000 7f0c72ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb08a0 0x56187efb0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.475+0000 7f0c84a09b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56187ef02950 con 0x56187efb6320 2024-06-04T05:01:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.475+0000 7f0c72ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb08a0 0x56187efb0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53074/0 (socket says 172.21.15.70:53074) 2024-06-04T05:01:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.475+0000 7f0c72ffd700 1 -- 172.21.15.70:0/3521441468 learned_addr learned my addr 172.21.15.70:0/3521441468 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.476+0000 7f0c73fff700 1 --2- 172.21.15.70:0/3521441468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187efb1260 0x56187efbd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.476+0000 7f0c72ffd700 1 -- 172.21.15.70:0/3521441468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb6320 msgr2=0x56187efb6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.476+0000 7f0c72ffd700 1 --2- 172.21.15.70:0/3521441468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb6320 0x56187efb6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.476+0000 7f0c72ffd700 1 -- 172.21.15.70:0/3521441468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187efb1260 msgr2=0x56187efbd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.476+0000 7f0c72ffd700 1 --2- 172.21.15.70:0/3521441468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187efb1260 0x56187efbd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.476+0000 7f0c72ffd700 1 -- 172.21.15.70:0/3521441468 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56187eea7fb0 con 0x56187efb08a0 2024-06-04T05:01:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.476+0000 7f0c73fff700 1 --2- 172.21.15.70:0/3521441468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187efb1260 0x56187efbd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:01:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.476+0000 7f0c737fe700 1 --2- 172.21.15.70:0/3521441468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb6320 0x56187efb6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:40.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.476+0000 7f0c72ffd700 1 --2- 172.21.15.70:0/3521441468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb08a0 0x56187efb0d20 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f0c680aa350 tx=0x7f0c680a9d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65180d4687c099d2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:40.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.476+0000 7f0c59ffb700 1 -- 172.21.15.70:0/3521441468 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c680aaac0 con 0x56187efb08a0 2024-06-04T05:01:40.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.476+0000 7f0c59ffb700 1 -- 172.21.15.70:0/3521441468 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c680aa3c0 con 0x56187efb08a0 2024-06-04T05:01:40.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.476+0000 7f0c59ffb700 1 -- 172.21.15.70:0/3521441468 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c680aa880 con 0x56187efb08a0 2024-06-04T05:01:40.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.477+0000 7f0c84a09b80 1 -- 172.21.15.70:0/3521441468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb08a0 msgr2=0x56187efb0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:40.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.477+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/3521441468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb08a0 0x56187efb0d20 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f0c680aa350 tx=0x7f0c680a9d10 comp rx=0 tx=0).stop 2024-06-04T05:01:40.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.477+0000 7f0c84a09b80 1 -- 172.21.15.70:0/3521441468 shutdown_connections 2024-06-04T05:01:40.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.477+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/3521441468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187efb1260 0x56187efbd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.477+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/3521441468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb6320 0x56187efb6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.477+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/3521441468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb08a0 0x56187efb0d20 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.477+0000 7f0c84a09b80 1 -- 172.21.15.70:0/3521441468 >> 172.21.15.70:0/3521441468 conn(0x56187eedc240 msgr2=0x56187efab3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:40.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.477+0000 7f0c84a09b80 1 -- 172.21.15.70:0/3521441468 shutdown_connections 2024-06-04T05:01:40.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.477+0000 7f0c84a09b80 1 -- 172.21.15.70:0/3521441468 wait complete. 2024-06-04T05:01:40.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.477+0000 7f0c84a09b80 1 -- start start 2024-06-04T05:01:40.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c84a09b80 1 Processor -- start 2024-06-04T05:01:40.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c84a09b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 0x56187efbcda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c84a09b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 0x56187efba0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c84a09b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187efb6320 0x56187efba5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c84a09b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56187ee04fa0 con 0x56187efb1260 2024-06-04T05:01:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c84a09b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56187ee66da0 con 0x56187efb6320 2024-06-04T05:01:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c84a09b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56187ef0ed70 con 0x56187efb08a0 2024-06-04T05:01:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c84a09b80 10 client.0 _ll_register_callbacks cb 0x56187ee99950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c72ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 0x56187efba0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c72ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 0x56187efba0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53076/0 (socket says 172.21.15.70:53076) 2024-06-04T05:01:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c72ffd700 1 -- 172.21.15.70:0/4239665758 learned_addr learned my addr 172.21.15.70:0/4239665758 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c737fe700 1 --2- 172.21.15.70:0/4239665758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 0x56187efbcda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c73fff700 1 --2- 172.21.15.70:0/4239665758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187efb6320 0x56187efba5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c72ffd700 1 -- 172.21.15.70:0/4239665758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 msgr2=0x56187efbcda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c72ffd700 1 --2- 172.21.15.70:0/4239665758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 0x56187efbcda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c72ffd700 1 -- 172.21.15.70:0/4239665758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187efb6320 msgr2=0x56187efba5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:40.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c72ffd700 1 --2- 172.21.15.70:0/4239665758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187efb6320 0x56187efba5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.478+0000 7f0c72ffd700 1 -- 172.21.15.70:0/4239665758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56187ee4e8d0 con 0x56187efb1260 2024-06-04T05:01:40.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.479+0000 7f0c72ffd700 1 --2- 172.21.15.70:0/4239665758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 0x56187efba0a0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f0c680e0da0 tx=0x7f0c680aa850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7f337d4e1e39092 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:40.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.479+0000 7f0c52ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:40.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.479+0000 7f0c52ffd700 1 -- 172.21.15.70:0/4239665758 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c680ae770 con 0x56187efb1260 2024-06-04T05:01:40.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.479+0000 7f0c84a09b80 1 -- 172.21.15.70:0/4239665758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c680aa3c0 con 0x56187efb1260 2024-06-04T05:01:40.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.479+0000 7f0c52ffd700 1 -- 172.21.15.70:0/4239665758 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f0c680b2f10 con 0x56187efb1260 2024-06-04T05:01:40.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.479+0000 7f0c52ffd700 1 -- 172.21.15.70:0/4239665758 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0c680b3190 con 0x56187efb1260 2024-06-04T05:01:40.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.479+0000 7f0c84a09b80 10 client.16737 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:40.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.479+0000 7f0c84a09b80 1 -- 172.21.15.70:0/4239665758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56187eeb8760 con 0x56187efb1260 2024-06-04T05:01:40.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.479+0000 7f0c52ffd700 1 -- 172.21.15.70:0/4239665758 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c680aa3c0 con 0x56187efb1260 2024-06-04T05:01:40.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.480+0000 7f0c52ffd700 1 -- 172.21.15.70:0/4239665758 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f0c680a4ae0 con 0x56187efb1260 2024-06-04T05:01:40.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.480+0000 7f0c84a09b80 10 client.16737 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:40.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.480+0000 7f0c84a09b80 10 client.16737 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:40.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.480+0000 7f0c84a09b80 20 client.16737 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:40.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.480+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/4239665758 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56187efc9e90 0x56187efcc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.480+0000 7f0c84a09b80 4 client.16737 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:01:40.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.480+0000 7f0c84a09b80 1 -- 172.21.15.70:0/4239665758 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56187ef10cc0 con 0x56187efc9e90 2024-06-04T05:01:40.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.480+0000 7f0c737fe700 1 --2- 172.21.15.70:0/4239665758 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56187efc9e90 0x56187efcc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.480+0000 7f0c52ffd700 10 client.16737 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:40.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.481+0000 7f0c737fe700 1 --2- 172.21.15.70:0/4239665758 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56187efc9e90 0x56187efcc280 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ced740dbf07c9e8 server_cookie=cd549f75281259ed in_seq=0 out_seq=0 2024-06-04T05:01:40.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.481+0000 7f0c52ffd700 1 -- 172.21.15.70:0/4239665758 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56187ef10cc0 con 0x56187efc9e90 2024-06-04T05:01:40.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.481+0000 7f0c52ffd700 10 client.16737 handle_command_reply: tid=0 2024-06-04T05:01:40.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.482+0000 7f0c84a09b80 1 client.16737 shutdown 2024-06-04T05:01:40.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.482+0000 7f0c84a09b80 1 -- 172.21.15.70:0/4239665758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 msgr2=0x56187efba0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:40.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.482+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/4239665758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 0x56187efba0a0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f0c680e0da0 tx=0x7f0c680aa850 comp rx=0 tx=0).stop 2024-06-04T05:01:40.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.482+0000 7f0c84a09b80 1 -- 172.21.15.70:0/4239665758 shutdown_connections 2024-06-04T05:01:40.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.482+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/4239665758 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56187efc9e90 0x56187efcc280 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.482+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/4239665758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187efb6320 0x56187efba5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.482+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/4239665758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 0x56187efbcda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.482+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/4239665758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 0x56187efba0a0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.482+0000 7f0c84a09b80 1 -- 172.21.15.70:0/4239665758 >> 172.21.15.70:0/4239665758 conn(0x56187eedc240 msgr2=0x56187efaaaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:40.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.482+0000 7f0c52ffd700 0 client.16737 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:40.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.482+0000 7f0c84a09b80 1 -- 172.21.15.70:0/4239665758 shutdown_connections 2024-06-04T05:01:40.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.482+0000 7f0c84a09b80 1 -- 172.21.15.70:0/4239665758 wait complete. 2024-06-04T05:01:40.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.482+0000 7f0c84a09b80 20 client.16737 trim_cache size 0 max 16384 2024-06-04T05:01:40.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.505+0000 7f0c84a09b80 1 Processor -- start 2024-06-04T05:01:40.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.505+0000 7f0c84a09b80 1 -- start start 2024-06-04T05:01:40.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.505+0000 7f0c84a09b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187eedc240 0x56187efab270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.505+0000 7f0c84a09b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 0x56187efaffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.505+0000 7f0c84a09b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 0x56187efb2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.505+0000 7f0c84a09b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0c741abb40 con 0x56187efb1260 2024-06-04T05:01:40.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.505+0000 7f0c84a09b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c741ad6e0 con 0x56187eedc240 2024-06-04T05:01:40.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.505+0000 7f0c84a09b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0c741ac210 con 0x56187efb08a0 2024-06-04T05:01:40.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.505+0000 7f0c73fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 0x56187efb2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.505+0000 7f0c72ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 0x56187efaffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.505+0000 7f0c73fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 0x56187efb2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53080/0 (socket says 172.21.15.70:53080) 2024-06-04T05:01:40.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.505+0000 7f0c73fff700 1 -- 172.21.15.70:0/1682773593 learned_addr learned my addr 172.21.15.70:0/1682773593 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:40.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.506+0000 7f0c737fe700 1 --2- 172.21.15.70:0/1682773593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187eedc240 0x56187efab270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.506+0000 7f0c72ffd700 1 -- 172.21.15.70:0/1682773593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187eedc240 msgr2=0x56187efab270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:40.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.506+0000 7f0c72ffd700 1 --2- 172.21.15.70:0/1682773593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187eedc240 0x56187efab270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.506+0000 7f0c72ffd700 1 -- 172.21.15.70:0/1682773593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 msgr2=0x56187efb2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:40.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.506+0000 7f0c72ffd700 1 --2- 172.21.15.70:0/1682773593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 0x56187efb2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.506+0000 7f0c72ffd700 1 -- 172.21.15.70:0/1682773593 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56187efbafe0 con 0x56187efb08a0 2024-06-04T05:01:40.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.506+0000 7f0c73fff700 1 --2- 172.21.15.70:0/1682773593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 0x56187efb2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:40.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.506+0000 7f0c72ffd700 1 --2- 172.21.15.70:0/1682773593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 0x56187efaffa0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0c680059d0 tx=0x7f0c680a9d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69f80b070f5979f7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:40.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.506+0000 7f0c597fa700 1 -- 172.21.15.70:0/1682773593 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c680aaac0 con 0x56187efb08a0 2024-06-04T05:01:40.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.506+0000 7f0c597fa700 1 -- 172.21.15.70:0/1682773593 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c680ae770 con 0x56187efb08a0 2024-06-04T05:01:40.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.506+0000 7f0c597fa700 1 -- 172.21.15.70:0/1682773593 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6800fb10 con 0x56187efb08a0 2024-06-04T05:01:40.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.507+0000 7f0c84a09b80 1 -- 172.21.15.70:0/1682773593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 msgr2=0x56187efaffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:40.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.507+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/1682773593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 0x56187efaffa0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0c680059d0 tx=0x7f0c680a9d10 comp rx=0 tx=0).stop 2024-06-04T05:01:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.507+0000 7f0c84a09b80 1 -- 172.21.15.70:0/1682773593 shutdown_connections 2024-06-04T05:01:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.507+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/1682773593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 0x56187efb2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.507+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/1682773593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187eedc240 0x56187efab270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.507+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/1682773593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 0x56187efaffa0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.507+0000 7f0c84a09b80 1 -- 172.21.15.70:0/1682773593 >> 172.21.15.70:0/1682773593 conn(0x56187efa7ad0 msgr2=0x56187efaf890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.507+0000 7f0c84a09b80 1 -- 172.21.15.70:0/1682773593 shutdown_connections 2024-06-04T05:01:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.507+0000 7f0c84a09b80 1 -- 172.21.15.70:0/1682773593 wait complete. 2024-06-04T05:01:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.507+0000 7f0c84a09b80 1 -- start start 2024-06-04T05:01:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.508+0000 7f0c84a09b80 1 Processor -- start 2024-06-04T05:01:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.508+0000 7f0c84a09b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187eedc240 0x56187efd1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.508+0000 7f0c84a09b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 0x56187efd1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.508+0000 7f0c84a09b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 0x56187efd2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.508+0000 7f0c84a09b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0c741acfb0 con 0x56187efb1260 2024-06-04T05:01:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.508+0000 7f0c84a09b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c741ac8e0 con 0x56187eedc240 2024-06-04T05:01:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.508+0000 7f0c84a09b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0c741a9ae0 con 0x56187efb08a0 2024-06-04T05:01:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.508+0000 7f0c73fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 0x56187efd2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.508+0000 7f0c84a09b80 10 client.0 _ll_register_callbacks cb 0x56187ee99950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.508+0000 7f0c72ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 0x56187efd1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.508+0000 7f0c73fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 0x56187efd2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53086/0 (socket says 172.21.15.70:53086) 2024-06-04T05:01:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.508+0000 7f0c73fff700 1 -- 172.21.15.70:0/2452495274 learned_addr learned my addr 172.21.15.70:0/2452495274 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.508+0000 7f0c72ffd700 1 --2- 172.21.15.70:0/2452495274 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 0x56187efd1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:52948/0 (socket says 172.21.15.70:52948) 2024-06-04T05:01:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.508+0000 7f0c737fe700 1 --2- 172.21.15.70:0/2452495274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187eedc240 0x56187efd1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.508+0000 7f0c72ffd700 1 -- 172.21.15.70:0/2452495274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187eedc240 msgr2=0x56187efd1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.508+0000 7f0c72ffd700 1 --2- 172.21.15.70:0/2452495274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187eedc240 0x56187efd1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.509+0000 7f0c72ffd700 1 -- 172.21.15.70:0/2452495274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 msgr2=0x56187efd2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.509+0000 7f0c72ffd700 1 --2- 172.21.15.70:0/2452495274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 0x56187efd2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.509+0000 7f0c72ffd700 1 -- 172.21.15.70:0/2452495274 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56187efc0e70 con 0x56187efb08a0 2024-06-04T05:01:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.509+0000 7f0c72ffd700 1 --2- 172.21.15.70:0/2452495274 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 0x56187efd1b50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0c680059d0 tx=0x7f0c680a9e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9496a4118982c584 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.509+0000 7f0c52ffd700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:01:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.509+0000 7f0c52ffd700 1 -- 172.21.15.70:0/2452495274 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c680aa3c0 con 0x56187efb08a0 2024-06-04T05:01:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.509+0000 7f0c52ffd700 1 -- 172.21.15.70:0/2452495274 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f0c680a9620 con 0x56187efb08a0 2024-06-04T05:01:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.509+0000 7f0c84a09b80 1 -- 172.21.15.70:0/2452495274 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c680ae770 con 0x56187efb08a0 2024-06-04T05:01:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.509+0000 7f0c52ffd700 1 -- 172.21.15.70:0/2452495274 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0c680aaac0 con 0x56187efb08a0 2024-06-04T05:01:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.509+0000 7f0c84a09b80 10 client.26098 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.509+0000 7f0c84a09b80 1 -- 172.21.15.70:0/2452495274 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56187efc0e70 con 0x56187efb08a0 2024-06-04T05:01:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.509+0000 7f0c52ffd700 1 -- 172.21.15.70:0/2452495274 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c680ae770 con 0x56187efb08a0 2024-06-04T05:01:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.509+0000 7f0c52ffd700 1 -- 172.21.15.70:0/2452495274 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f0c680cb1c0 con 0x56187efb08a0 2024-06-04T05:01:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.509+0000 7f0c84a09b80 10 client.26098 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.509+0000 7f0c84a09b80 10 client.26098 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:40.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.509+0000 7f0c84a09b80 20 client.26098 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:40.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.510+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/2452495274 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56187efb6320 0x56187f036f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:40.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.510+0000 7f0c84a09b80 4 client.26098 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:01:40.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.510+0000 7f0c84a09b80 1 -- 172.21.15.70:0/2452495274 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56187ef0a4a0 con 0x56187efb6320 2024-06-04T05:01:40.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.510+0000 7f0c737fe700 1 --2- 172.21.15.70:0/2452495274 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56187efb6320 0x56187f036f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:40.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.510+0000 7f0c52ffd700 10 client.26098 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:40.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.510+0000 7f0c737fe700 1 --2- 172.21.15.70:0/2452495274 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56187efb6320 0x56187f036f50 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5892a34023ee9399 server_cookie=33d424848a045c9 in_seq=0 out_seq=0 2024-06-04T05:01:40.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.511+0000 7f0c52ffd700 1 -- 172.21.15.70:0/2452495274 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56187ef0a4a0 con 0x56187efb6320 2024-06-04T05:01:40.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.511+0000 7f0c52ffd700 10 client.26098 handle_command_reply: tid=0 2024-06-04T05:01:40.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.511+0000 7f0c84a09b80 1 client.26098 shutdown 2024-06-04T05:01:40.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.511+0000 7f0c84a09b80 1 -- 172.21.15.70:0/2452495274 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 msgr2=0x56187efd1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:40.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.511+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/2452495274 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 0x56187efd1b50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0c680059d0 tx=0x7f0c680a9e40 comp rx=0 tx=0).stop 2024-06-04T05:01:40.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.511+0000 7f0c84a09b80 1 -- 172.21.15.70:0/2452495274 shutdown_connections 2024-06-04T05:01:40.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.511+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/2452495274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56187efb1260 0x56187efd2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.511+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/2452495274 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56187efb6320 0x56187f036f50 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.511+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/2452495274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56187eedc240 0x56187efd1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.511+0000 7f0c84a09b80 1 --2- 172.21.15.70:0/2452495274 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56187efb08a0 0x56187efd1b50 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.511+0000 7f0c84a09b80 1 -- 172.21.15.70:0/2452495274 >> 172.21.15.70:0/2452495274 conn(0x56187efa7ad0 msgr2=0x56187efb0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:40.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.511+0000 7f0c52ffd700 0 client.26098 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:40.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.511+0000 7f0c84a09b80 1 -- 172.21.15.70:0/2452495274 shutdown_connections 2024-06-04T05:01:40.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.511+0000 7f0c84a09b80 1 -- 172.21.15.70:0/2452495274 wait complete. 2024-06-04T05:01:40.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.511+0000 7f0c84a09b80 20 client.26098 trim_cache size 0 max 16384 2024-06-04T05:01:40.519 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:01:40.519 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:01:40.519 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:01:40.519 INFO:teuthology.orchestra.run.smithi070.stdout: "b53f3ea4-8906-4d26-a6a4-2a9b668c1c67": { 2024-06-04T05:01:40.519 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:01:40.519 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "b53f3ea4-8906-4d26-a6a4-2a9b668c1c67", 2024-06-04T05:01:40.519 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:01:40.519 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:01:40.519 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:01:40.519 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:01:40.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.512+0000 7f0c7a174700 1 -- 172.21.15.70:0/1584833754 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0c5c0b29f0 msgr2=0x7f0c5c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:40.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.512+0000 7f0c7a174700 1 --2- 172.21.15.70:0/1584833754 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0c5c0b29f0 0x7f0c5c0b4eb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0c7419aa80 tx=0x7f0c64009260 comp rx=0 tx=0).stop 2024-06-04T05:01:40.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.512+0000 7f0c7a174700 1 -- 172.21.15.70:0/1584833754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0c74102e00 msgr2=0x7f0c7419fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:40.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.512+0000 7f0c7a174700 1 --2- 172.21.15.70:0/1584833754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0c74102e00 0x7f0c7419fcc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0c68006010 tx=0x7f0c680047b0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.512+0000 7f0c7a174700 1 -- 172.21.15.70:0/1584833754 shutdown_connections 2024-06-04T05:01:40.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.512+0000 7f0c7a174700 1 --2- 172.21.15.70:0/1584833754 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0c5c0b29f0 0x7f0c5c0b4eb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.512+0000 7f0c7a174700 1 --2- 172.21.15.70:0/1584833754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c74108520 0x7f0c74199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.512+0000 7f0c7a174700 1 --2- 172.21.15.70:0/1584833754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c74102440 0x7f0c7419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.512+0000 7f0c7a174700 1 --2- 172.21.15.70:0/1584833754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0c74102e00 0x7f0c7419fcc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:40.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.512+0000 7f0c7a174700 1 -- 172.21.15.70:0/1584833754 >> 172.21.15.70:0/1584833754 conn(0x7f0c740793f0 msgr2=0x7f0c7406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:40.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.512+0000 7f0c7a174700 1 -- 172.21.15.70:0/1584833754 shutdown_connections 2024-06-04T05:01:40.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:01:40.512+0000 7f0c7a174700 1 -- 172.21.15.70:0/1584833754 wait complete. 2024-06-04T05:01:40.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 52585 1.16 MB/sec execute 33 sec latency 47.472 ms 2024-06-04T05:01:41.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:41 smithi070 ceph-mon[32631]: pgmap v138: 97 pgs: 97 active+clean; 43 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-06-04T05:01:41.421 INFO:tasks.cephfs.filesystem:scrub status for tag:b53f3ea4-8906-4d26-a6a4-2a9b668c1c67 - {'path': '/', 'tag': 'b53f3ea4-8906-4d26-a6a4-2a9b668c1c67', 'options': 'recursive,force'} 2024-06-04T05:01:41.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:41 smithi112 ceph-mon[36453]: pgmap v138: 97 pgs: 97 active+clean; 43 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-06-04T05:01:41.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:41 smithi098 ceph-mon[35569]: pgmap v138: 97 pgs: 97 active+clean; 43 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-06-04T05:01:41.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 53078 1.20 MB/sec execute 34 sec latency 15.187 ms 2024-06-04T05:01:42.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 53563 1.29 MB/sec execute 35 sec latency 14.576 ms 2024-06-04T05:01:43.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:43 smithi112 ceph-mon[36453]: pgmap v139: 97 pgs: 97 active+clean; 44 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-06-04T05:01:43.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:43 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:01:43.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:43 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:01:43.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:43 smithi098 ceph-mon[35569]: pgmap v139: 97 pgs: 97 active+clean; 44 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-06-04T05:01:43.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:43 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:01:43.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:43 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:01:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:43 smithi070 ceph-mon[32631]: pgmap v139: 97 pgs: 97 active+clean; 44 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-06-04T05:01:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:43 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:01:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:43 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:01:43.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 53853 1.26 MB/sec execute 36 sec latency 14.295 ms 2024-06-04T05:01:44.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:44 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:01:44.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:44 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:01:44.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:44 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:01:44.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 54412 1.29 MB/sec execute 37 sec latency 15.286 ms 2024-06-04T05:01:45.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:45 smithi112 ceph-mon[36453]: pgmap v140: 97 pgs: 97 active+clean; 44 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-06-04T05:01:45.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:45 smithi098 ceph-mon[35569]: pgmap v140: 97 pgs: 97 active+clean; 44 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-06-04T05:01:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:45 smithi070 ceph-mon[32631]: pgmap v140: 97 pgs: 97 active+clean; 44 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-06-04T05:01:45.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 55015 1.34 MB/sec execute 38 sec latency 15.766 ms 2024-06-04T05:01:46.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.394+0000 7f65fa3c1640 1 -- 172.21.15.70:0/692325248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65f4102340 msgr2=0x7f65f410e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:46.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.394+0000 7f65fa3c1640 1 --2- 172.21.15.70:0/692325248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65f4102340 0x7f65f410e800 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f65ec009cd0 tx=0x7f65ec02a8e0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.394+0000 7f65fa3c1640 1 -- 172.21.15.70:0/692325248 shutdown_connections 2024-06-04T05:01:46.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.394+0000 7f65fa3c1640 1 --2- 172.21.15.70:0/692325248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65f4102340 0x7f65f410e800 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.394+0000 7f65fa3c1640 1 --2- 172.21.15.70:0/692325248 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65f4101980 0x7f65f4101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.394+0000 7f65fa3c1640 1 --2- 172.21.15.70:0/692325248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65f4107940 0x7f65f4107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.394+0000 7f65fa3c1640 1 -- 172.21.15.70:0/692325248 >> 172.21.15.70:0/692325248 conn(0x7f65f4078ad0 msgr2=0x7f65f4078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.394+0000 7f65fa3c1640 1 -- 172.21.15.70:0/692325248 shutdown_connections 2024-06-04T05:01:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.394+0000 7f65fa3c1640 1 -- 172.21.15.70:0/692325248 wait complete. 2024-06-04T05:01:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.395+0000 7f65fa3c1640 1 Processor -- start 2024-06-04T05:01:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.395+0000 7f65fa3c1640 1 -- start start 2024-06-04T05:01:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.395+0000 7f65fa3c1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65f4101980 0x7f65f41990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.395+0000 7f65fa3c1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65f4102340 0x7f65f4199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.395+0000 7f65fa3c1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65f4107940 0x7f65f419d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.395+0000 7f65fa3c1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65f4111ab0 con 0x7f65f4102340 2024-06-04T05:01:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.395+0000 7f65fa3c1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f65f4111930 con 0x7f65f4101980 2024-06-04T05:01:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.395+0000 7f65fa3c1640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f65f4111c30 con 0x7f65f4107940 2024-06-04T05:01:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.395+0000 7f65f37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65f4102340 0x7f65f4199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.395+0000 7f65f37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65f4102340 0x7f65f4199620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53550/0 (socket says 172.21.15.70:53550) 2024-06-04T05:01:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.395+0000 7f65f37fe640 1 -- 172.21.15.70:0/3655275857 learned_addr learned my addr 172.21.15.70:0/3655275857 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.396+0000 7f65f3fff640 1 --2- 172.21.15.70:0/3655275857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65f4101980 0x7f65f41990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.396+0000 7f65f895e640 1 --2- 172.21.15.70:0/3655275857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65f4107940 0x7f65f419d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.396+0000 7f65f37fe640 1 -- 172.21.15.70:0/3655275857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65f4107940 msgr2=0x7f65f419d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.396+0000 7f65f37fe640 1 --2- 172.21.15.70:0/3655275857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65f4107940 0x7f65f419d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.396+0000 7f65f37fe640 1 -- 172.21.15.70:0/3655275857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65f4101980 msgr2=0x7f65f41990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:46.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.396+0000 7f65f37fe640 1 --2- 172.21.15.70:0/3655275857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65f4101980 0x7f65f41990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.396+0000 7f65f37fe640 1 -- 172.21.15.70:0/3655275857 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f419e0d0 con 0x7f65f4102340 2024-06-04T05:01:46.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.396+0000 7f65f3fff640 1 --2- 172.21.15.70:0/3655275857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65f4101980 0x7f65f41990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:46.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.396+0000 7f65f37fe640 1 --2- 172.21.15.70:0/3655275857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65f4102340 0x7f65f4199620 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f65e800e9f0 tx=0x7f65e800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:46.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.396+0000 7f65f17fa640 1 -- 172.21.15.70:0/3655275857 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e800cd80 con 0x7f65f4102340 2024-06-04T05:01:46.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.396+0000 7f65fa3c1640 1 -- 172.21.15.70:0/3655275857 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f419e3c0 con 0x7f65f4102340 2024-06-04T05:01:46.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.396+0000 7f65f17fa640 1 -- 172.21.15.70:0/3655275857 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65e80054b0 con 0x7f65f4102340 2024-06-04T05:01:46.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.396+0000 7f65f17fa640 1 -- 172.21.15.70:0/3655275857 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e8010430 con 0x7f65f4102340 2024-06-04T05:01:46.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.396+0000 7f65fa3c1640 1 -- 172.21.15.70:0/3655275857 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f41a5c50 con 0x7f65f4102340 2024-06-04T05:01:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.399+0000 7f65fd1b9740 1 Processor -- start 2024-06-04T05:01:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.399+0000 7f65f17fa640 1 -- 172.21.15.70:0/3655275857 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f65e80046c0 con 0x7f65f4102340 2024-06-04T05:01:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.399+0000 7f65fd1b9740 1 -- start start 2024-06-04T05:01:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.399+0000 7f65fd1b9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077ea820 0x560f077eac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.399+0000 7f65fd1b9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077eb140 0x560f077f54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.399+0000 7f65fd1b9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077f5a00 0x560f077f7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.399+0000 7f65fd1b9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560f0775cc30 con 0x560f077f5a00 2024-06-04T05:01:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.399+0000 7f65fd1b9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560f0774c7d0 con 0x560f077ea820 2024-06-04T05:01:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.399+0000 7f65fd1b9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560f07750810 con 0x560f077eb140 2024-06-04T05:01:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.399+0000 7f65f37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077f5a00 0x560f077f7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.399+0000 7f65f37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077f5a00 0x560f077f7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53554/0 (socket says 172.21.15.70:53554) 2024-06-04T05:01:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.399+0000 7f65f37fe640 1 -- 172.21.15.70:0/2217147166 learned_addr learned my addr 172.21.15.70:0/2217147166 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.399+0000 7f65f895e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077ea820 0x560f077eac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.399+0000 7f65f3fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077eb140 0x560f077f54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.400+0000 7f65f3fff640 1 -- 172.21.15.70:0/2217147166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077ea820 msgr2=0x560f077eac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.400+0000 7f65f3fff640 1 --2- 172.21.15.70:0/2217147166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077ea820 0x560f077eac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.400+0000 7f65f3fff640 1 -- 172.21.15.70:0/2217147166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077f5a00 msgr2=0x560f077f7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.400+0000 7f65f3fff640 1 --2- 172.21.15.70:0/2217147166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077f5a00 0x560f077f7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.400+0000 7f65f3fff640 1 -- 172.21.15.70:0/2217147166 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f07461210 con 0x560f077eb140 2024-06-04T05:01:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.400+0000 7f65f17fa640 1 --2- 172.21.15.70:0/3655275857 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65dc0b2a10 0x7f65dc0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.400+0000 7f65f3fff640 1 --2- 172.21.15.70:0/2217147166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077eb140 0x560f077f54c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f65e4009f50 tx=0x7f65e40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1f14334252470de server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.400+0000 7f65f895e640 1 --2- 172.21.15.70:0/3655275857 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65dc0b2a10 0x7f65dc0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.400+0000 7f65da7fc640 1 -- 172.21.15.70:0/2217147166 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4004030 con 0x560f077eb140 2024-06-04T05:01:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.400+0000 7f65f17fa640 1 -- 172.21.15.70:0/3655275857 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f65e80e2990 con 0x7f65f4102340 2024-06-04T05:01:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.400+0000 7f65da7fc640 1 -- 172.21.15.70:0/2217147166 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65e4005020 con 0x560f077eb140 2024-06-04T05:01:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.400+0000 7f65da7fc640 1 -- 172.21.15.70:0/2217147166 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4004b60 con 0x560f077eb140 2024-06-04T05:01:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.401+0000 7f65fd1b9740 1 -- 172.21.15.70:0/2217147166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077eb140 msgr2=0x560f077f54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.401+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/2217147166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077eb140 0x560f077f54c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f65e4009f50 tx=0x7f65e40118d0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.401+0000 7f65f895e640 1 --2- 172.21.15.70:0/3655275857 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65dc0b2a10 0x7f65dc0b4ed0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f65ec004ab0 tx=0x7f65ec004a40 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.401+0000 7f65fd1b9740 1 -- 172.21.15.70:0/2217147166 shutdown_connections 2024-06-04T05:01:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.401+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/2217147166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077f5a00 0x560f077f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.401+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/2217147166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077eb140 0x560f077f54c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.401+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/2217147166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077ea820 0x560f077eac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.401+0000 7f65fd1b9740 1 -- 172.21.15.70:0/2217147166 >> 172.21.15.70:0/2217147166 conn(0x560f077e6d30 msgr2=0x560f077e7110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.401+0000 7f65fd1b9740 1 -- 172.21.15.70:0/2217147166 shutdown_connections 2024-06-04T05:01:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.401+0000 7f65fd1b9740 1 -- 172.21.15.70:0/2217147166 wait complete. 2024-06-04T05:01:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.401+0000 7f65fd1b9740 1 -- start start 2024-06-04T05:01:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65fd1b9740 1 Processor -- start 2024-06-04T05:01:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65fd1b9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077ea820 0x560f077f29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65fd1b9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077eb140 0x560f077f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65fd1b9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077f5a00 0x560f077f3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65fd1b9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560f07432790 con 0x560f077ea820 2024-06-04T05:01:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65fd1b9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560f07708890 con 0x560f077eb140 2024-06-04T05:01:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65fd1b9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560f077f8630 con 0x560f077f5a00 2024-06-04T05:01:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65fd1b9740 10 client.0 _ll_register_callbacks cb 0x560f07460a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65f3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077ea820 0x560f077f29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65f3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077ea820 0x560f077f29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53568/0 (socket says 172.21.15.70:53568) 2024-06-04T05:01:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65f3fff640 1 -- 172.21.15.70:0/827114041 learned_addr learned my addr 172.21.15.70:0/827114041 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65f37fe640 1 --2- 172.21.15.70:0/827114041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077eb140 0x560f077f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65f895e640 1 --2- 172.21.15.70:0/827114041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077f5a00 0x560f077f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65f3fff640 1 -- 172.21.15.70:0/827114041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077f5a00 msgr2=0x560f077f3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65f3fff640 1 --2- 172.21.15.70:0/827114041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077f5a00 0x560f077f3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65f3fff640 1 -- 172.21.15.70:0/827114041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077eb140 msgr2=0x560f077f2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65f3fff640 1 --2- 172.21.15.70:0/827114041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077eb140 0x560f077f2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65f3fff640 1 -- 172.21.15.70:0/827114041 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f0752d620 con 0x560f077ea820 2024-06-04T05:01:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65f37fe640 1 --2- 172.21.15.70:0/827114041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077eb140 0x560f077f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65f3fff640 1 --2- 172.21.15.70:0/827114041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077ea820 0x560f077f29f0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f65e4009700 tx=0x7f65e4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35091361798b3862 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.402+0000 7f65d37fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.403+0000 7f65d37fe640 1 -- 172.21.15.70:0/827114041 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4028020 con 0x560f077ea820 2024-06-04T05:01:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.403+0000 7f65fd1b9740 1 -- 172.21.15.70:0/827114041 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65e4005020 con 0x560f077ea820 2024-06-04T05:01:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.403+0000 7f65d37fe640 1 -- 172.21.15.70:0/827114041 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f65e4016040 con 0x560f077ea820 2024-06-04T05:01:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.403+0000 7f65d37fe640 1 -- 172.21.15.70:0/827114041 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f65e40048a0 con 0x560f077ea820 2024-06-04T05:01:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.403+0000 7f65fd1b9740 10 client.16755 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.403+0000 7f65fd1b9740 1 -- 172.21.15.70:0/827114041 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560f07804420 con 0x560f077ea820 2024-06-04T05:01:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.403+0000 7f65d37fe640 1 -- 172.21.15.70:0/827114041 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65e4005020 con 0x560f077ea820 2024-06-04T05:01:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.403+0000 7f65d37fe640 1 -- 172.21.15.70:0/827114041 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f65e4003b10 con 0x560f077ea820 2024-06-04T05:01:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.403+0000 7f65fd1b9740 10 client.16755 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.403+0000 7f65fd1b9740 10 client.16755 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.403+0000 7f65fd1b9740 20 client.16755 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.403+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/827114041 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560f07805890 0x560f07807c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.403+0000 7f65fd1b9740 4 client.16755 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:01:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.403+0000 7f65fd1b9740 1 -- 172.21.15.70:0/827114041 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f0775eb80 con 0x560f07805890 2024-06-04T05:01:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.403+0000 7f65f37fe640 1 --2- 172.21.15.70:0/827114041 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560f07805890 0x560f07807c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.404+0000 7f65d37fe640 10 client.16755 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.404+0000 7f65f37fe640 1 --2- 172.21.15.70:0/827114041 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560f07805890 0x560f07807c80 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bb52e24d8fcf5d4 server_cookie=bbbbfd19f6de4546 in_seq=0 out_seq=0 2024-06-04T05:01:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.405+0000 7f65d37fe640 1 -- 172.21.15.70:0/827114041 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560f0775eb80 con 0x560f07805890 2024-06-04T05:01:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.405+0000 7f65d37fe640 10 client.16755 handle_command_reply: tid=0 2024-06-04T05:01:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.405+0000 7f65fd1b9740 1 client.16755 shutdown 2024-06-04T05:01:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.405+0000 7f65fd1b9740 1 -- 172.21.15.70:0/827114041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077ea820 msgr2=0x560f077f29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.405+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/827114041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077ea820 0x560f077f29f0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f65e4009700 tx=0x7f65e4004270 comp rx=0 tx=0).stop 2024-06-04T05:01:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.405+0000 7f65fd1b9740 1 -- 172.21.15.70:0/827114041 shutdown_connections 2024-06-04T05:01:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.405+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/827114041 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560f07805890 0x560f07807c80 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.405+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/827114041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077eb140 0x560f077f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.405+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/827114041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077f5a00 0x560f077f3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.405+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/827114041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077ea820 0x560f077f29f0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.405+0000 7f65d37fe640 0 client.16755 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.405+0000 7f65fd1b9740 1 -- 172.21.15.70:0/827114041 >> 172.21.15.70:0/827114041 conn(0x560f077e6d30 msgr2=0x560f077e5fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.405+0000 7f65fd1b9740 1 -- 172.21.15.70:0/827114041 shutdown_connections 2024-06-04T05:01:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.405+0000 7f65fd1b9740 1 -- 172.21.15.70:0/827114041 wait complete. 2024-06-04T05:01:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.405+0000 7f65fd1b9740 20 client.16755 trim_cache size 0 max 16384 2024-06-04T05:01:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.423+0000 7f65fd1b9740 1 Processor -- start 2024-06-04T05:01:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65fd1b9740 1 -- start start 2024-06-04T05:01:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65fd1b9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077ea820 0x560f077e5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65fd1b9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077eb140 0x560f0780d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65fd1b9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077f5a00 0x560f0780da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65fd1b9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65f41ac4c0 con 0x560f077ea820 2024-06-04T05:01:46.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65fd1b9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f65f41ae060 con 0x560f077f5a00 2024-06-04T05:01:46.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65fd1b9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f65f41acb90 con 0x560f077eb140 2024-06-04T05:01:46.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65f3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077ea820 0x560f077e5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65f3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077ea820 0x560f077e5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53572/0 (socket says 172.21.15.70:53572) 2024-06-04T05:01:46.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65f3fff640 1 -- 172.21.15.70:0/327713882 learned_addr learned my addr 172.21.15.70:0/327713882 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:46.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65f37fe640 1 --2- 172.21.15.70:0/327713882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077eb140 0x560f0780d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65f895e640 1 --2- 172.21.15.70:0/327713882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077f5a00 0x560f0780da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65f3fff640 1 -- 172.21.15.70:0/327713882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077eb140 msgr2=0x560f0780d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:46.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65f3fff640 1 --2- 172.21.15.70:0/327713882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077eb140 0x560f0780d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65f3fff640 1 -- 172.21.15.70:0/327713882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077f5a00 msgr2=0x560f0780da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:46.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65f3fff640 1 --2- 172.21.15.70:0/327713882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077f5a00 0x560f0780da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65f3fff640 1 -- 172.21.15.70:0/327713882 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f078082e0 con 0x560f077ea820 2024-06-04T05:01:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65f3fff640 1 --2- 172.21.15.70:0/327713882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077ea820 0x560f077e5e30 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f65e4011910 tx=0x7f65e4004830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef84d93db7950c19 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.424+0000 7f65d9ffb640 1 -- 172.21.15.70:0/327713882 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4028040 con 0x560f077ea820 2024-06-04T05:01:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.425+0000 7f65d9ffb640 1 -- 172.21.15.70:0/327713882 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65e4005550 con 0x560f077ea820 2024-06-04T05:01:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.425+0000 7f65d9ffb640 1 -- 172.21.15.70:0/327713882 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4004fd0 con 0x560f077ea820 2024-06-04T05:01:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.425+0000 7f65fd1b9740 1 -- 172.21.15.70:0/327713882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077ea820 msgr2=0x560f077e5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.425+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/327713882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077ea820 0x560f077e5e30 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f65e4011910 tx=0x7f65e4004830 comp rx=0 tx=0).stop 2024-06-04T05:01:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.425+0000 7f65fd1b9740 1 -- 172.21.15.70:0/327713882 shutdown_connections 2024-06-04T05:01:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.425+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/327713882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077f5a00 0x560f0780da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.425+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/327713882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077eb140 0x560f0780d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.425+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/327713882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077ea820 0x560f077e5e30 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.425+0000 7f65fd1b9740 1 -- 172.21.15.70:0/327713882 >> 172.21.15.70:0/327713882 conn(0x560f077e6d30 msgr2=0x560f077e5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.425+0000 7f65fd1b9740 1 -- 172.21.15.70:0/327713882 shutdown_connections 2024-06-04T05:01:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.425+0000 7f65fd1b9740 1 -- 172.21.15.70:0/327713882 wait complete. 2024-06-04T05:01:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65fd1b9740 1 -- start start 2024-06-04T05:01:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65fd1b9740 1 Processor -- start 2024-06-04T05:01:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65fd1b9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077ea820 0x560f077fe070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65fd1b9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077eb140 0x560f077f8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65fd1b9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077f5a00 0x560f077f8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65fd1b9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65f41ad930 con 0x560f077eb140 2024-06-04T05:01:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65fd1b9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f65f41ad260 con 0x560f077ea820 2024-06-04T05:01:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65fd1b9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f65f41aa4a0 con 0x560f077f5a00 2024-06-04T05:01:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65fd1b9740 10 client.0 _ll_register_callbacks cb 0x7f65e80ea040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65f37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077eb140 0x560f077f8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65f37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077eb140 0x560f077f8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53586/0 (socket says 172.21.15.70:53586) 2024-06-04T05:01:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65f37fe640 1 -- 172.21.15.70:0/1900919154 learned_addr learned my addr 172.21.15.70:0/1900919154 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65f895e640 1 --2- 172.21.15.70:0/1900919154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077f5a00 0x560f077f8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65f3fff640 1 --2- 172.21.15.70:0/1900919154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077ea820 0x560f077fe070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65f37fe640 1 -- 172.21.15.70:0/1900919154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077f5a00 msgr2=0x560f077f8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65f37fe640 1 --2- 172.21.15.70:0/1900919154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077f5a00 0x560f077f8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65f37fe640 1 -- 172.21.15.70:0/1900919154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077ea820 msgr2=0x560f077fe070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65f37fe640 1 --2- 172.21.15.70:0/1900919154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077ea820 0x560f077fe070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.426+0000 7f65f37fe640 1 -- 172.21.15.70:0/1900919154 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f077f9680 con 0x560f077eb140 2024-06-04T05:01:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.427+0000 7f65f37fe640 1 --2- 172.21.15.70:0/1900919154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077eb140 0x560f077f8840 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f65e80e0080 tx=0x7f65e80e6430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950109751a25ff34 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.427+0000 7f65d37fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.427+0000 7f65d37fe640 1 -- 172.21.15.70:0/1900919154 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e80a9a40 con 0x560f077eb140 2024-06-04T05:01:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.427+0000 7f65fd1b9740 1 -- 172.21.15.70:0/1900919154 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65e4005550 con 0x560f077eb140 2024-06-04T05:01:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.427+0000 7f65d37fe640 1 -- 172.21.15.70:0/1900919154 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f65e80aa0e0 con 0x560f077eb140 2024-06-04T05:01:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.427+0000 7f65d37fe640 1 -- 172.21.15.70:0/1900919154 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f65e80a5400 con 0x560f077eb140 2024-06-04T05:01:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.427+0000 7f65fd1b9740 10 client.16761 fetch_fsmap learned FSMap version 31 2024-06-04T05:01:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.427+0000 7f65fd1b9740 1 -- 172.21.15.70:0/1900919154 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560f0786f510 con 0x560f077eb140 2024-06-04T05:01:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.427+0000 7f65d37fe640 1 -- 172.21.15.70:0/1900919154 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65e4005550 con 0x560f077eb140 2024-06-04T05:01:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.427+0000 7f65d37fe640 1 -- 172.21.15.70:0/1900919154 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f65e80b2640 con 0x560f077eb140 2024-06-04T05:01:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.427+0000 7f65fd1b9740 10 client.16761 fetch_fsmap finished waiting for FSMap version 31 2024-06-04T05:01:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.427+0000 7f65fd1b9740 10 client.16761 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.427+0000 7f65fd1b9740 20 client.16761 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.427+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/1900919154 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560f0786fd10 0x560f07872130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.427+0000 7f65fd1b9740 4 client.16761 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:01:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.427+0000 7f65fd1b9740 1 -- 172.21.15.70:0/1900919154 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560f07758360 con 0x560f0786fd10 2024-06-04T05:01:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.428+0000 7f65f3fff640 1 --2- 172.21.15.70:0/1900919154 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560f0786fd10 0x560f07872130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.428+0000 7f65f3fff640 1 --2- 172.21.15.70:0/1900919154 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560f0786fd10 0x560f07872130 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8efbc3cf29fbe50b server_cookie=aeb618872f32ec6c in_seq=0 out_seq=0 2024-06-04T05:01:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.428+0000 7f65d37fe640 10 client.16761 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.447+0000 7f65d37fe640 1 -- 172.21.15.70:0/1900919154 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x560f07758360 con 0x560f0786fd10 2024-06-04T05:01:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.447+0000 7f65d37fe640 10 client.16761 handle_command_reply: tid=0 2024-06-04T05:01:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.447+0000 7f65fd1b9740 1 client.16761 shutdown 2024-06-04T05:01:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.447+0000 7f65fd1b9740 1 -- 172.21.15.70:0/1900919154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077eb140 msgr2=0x560f077f8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.447+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/1900919154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077eb140 0x560f077f8840 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f65e80e0080 tx=0x7f65e80e6430 comp rx=0 tx=0).stop 2024-06-04T05:01:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.447+0000 7f65fd1b9740 1 -- 172.21.15.70:0/1900919154 shutdown_connections 2024-06-04T05:01:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.447+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/1900919154 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560f0786fd10 0x560f07872130 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.447+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/1900919154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560f077f5a00 0x560f077f8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.447+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/1900919154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f077eb140 0x560f077f8840 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.447+0000 7f65fd1b9740 1 --2- 172.21.15.70:0/1900919154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f077ea820 0x560f077fe070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.447+0000 7f65fd1b9740 1 -- 172.21.15.70:0/1900919154 >> 172.21.15.70:0/1900919154 conn(0x560f077e6d30 msgr2=0x560f077e5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.447+0000 7f65d37fe640 0 client.16761 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.448+0000 7f65fd1b9740 1 -- 172.21.15.70:0/1900919154 shutdown_connections 2024-06-04T05:01:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.448+0000 7f65fd1b9740 1 -- 172.21.15.70:0/1900919154 wait complete. 2024-06-04T05:01:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.448+0000 7f65fd1b9740 20 client.16761 trim_cache size 0 max 16384 2024-06-04T05:01:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:01:46.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:01:46.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:01:46.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.448+0000 7f65fa3c1640 1 -- 172.21.15.70:0/3655275857 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65dc0b2a10 msgr2=0x7f65dc0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:46.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.448+0000 7f65fa3c1640 1 --2- 172.21.15.70:0/3655275857 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65dc0b2a10 0x7f65dc0b4ed0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f65ec004ab0 tx=0x7f65ec004a40 comp rx=0 tx=0).stop 2024-06-04T05:01:46.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.448+0000 7f65fa3c1640 1 -- 172.21.15.70:0/3655275857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65f4102340 msgr2=0x7f65f4199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.448+0000 7f65fa3c1640 1 --2- 172.21.15.70:0/3655275857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65f4102340 0x7f65f4199620 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f65e800e9f0 tx=0x7f65e800edf0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.448+0000 7f65fa3c1640 1 -- 172.21.15.70:0/3655275857 shutdown_connections 2024-06-04T05:01:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.448+0000 7f65fa3c1640 1 --2- 172.21.15.70:0/3655275857 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65dc0b2a10 0x7f65dc0b4ed0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.448+0000 7f65fa3c1640 1 --2- 172.21.15.70:0/3655275857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65f4107940 0x7f65f419d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.448+0000 7f65fa3c1640 1 --2- 172.21.15.70:0/3655275857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65f4102340 0x7f65f4199620 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.448+0000 7f65fa3c1640 1 --2- 172.21.15.70:0/3655275857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65f4101980 0x7f65f41990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.448+0000 7f65fa3c1640 1 -- 172.21.15.70:0/3655275857 >> 172.21.15.70:0/3655275857 conn(0x7f65f4078ad0 msgr2=0x7f65f406a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.448+0000 7f65fa3c1640 1 -- 172.21.15.70:0/3655275857 shutdown_connections 2024-06-04T05:01:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:46.448+0000 7f65fa3c1640 1 -- 172.21.15.70:0/3655275857 wait complete. 2024-06-04T05:01:46.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 55644 1.35 MB/sec execute 39 sec latency 20.029 ms 2024-06-04T05:01:47.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:47 smithi112 ceph-mon[36453]: pgmap v141: 97 pgs: 97 active+clean; 44 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 1.7 MiB/s wr, 205 op/s 2024-06-04T05:01:47.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:47 smithi098 ceph-mon[35569]: pgmap v141: 97 pgs: 97 active+clean; 44 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 1.7 MiB/s wr, 205 op/s 2024-06-04T05:01:47.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:47 smithi070 ceph-mon[32631]: pgmap v141: 97 pgs: 97 active+clean; 44 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 1.7 MiB/s wr, 205 op/s 2024-06-04T05:01:47.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 56410 1.47 MB/sec execute 40 sec latency 12.114 ms 2024-06-04T05:01:48.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 56867 1.54 MB/sec execute 41 sec latency 15.037 ms 2024-06-04T05:01:49.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:49 smithi112 ceph-mon[36453]: pgmap v142: 97 pgs: 97 active+clean; 44 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-06-04T05:01:49.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:49 smithi098 ceph-mon[35569]: pgmap v142: 97 pgs: 97 active+clean; 44 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-06-04T05:01:49.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:49 smithi070 ceph-mon[32631]: pgmap v142: 97 pgs: 97 active+clean; 44 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-06-04T05:01:49.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 57191 1.54 MB/sec execute 42 sec latency 14.595 ms 2024-06-04T05:01:50.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 57565 1.51 MB/sec execute 43 sec latency 11.519 ms 2024-06-04T05:01:51.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:51 smithi112 ceph-mon[36453]: pgmap v143: 97 pgs: 97 active+clean; 44 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-06-04T05:01:51.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:51 smithi098 ceph-mon[35569]: pgmap v143: 97 pgs: 97 active+clean; 44 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-06-04T05:01:51.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:51 smithi070 ceph-mon[32631]: pgmap v143: 97 pgs: 97 active+clean; 44 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-06-04T05:01:51.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 58068 1.52 MB/sec execute 44 sec latency 15.980 ms 2024-06-04T05:01:52.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 58706 1.56 MB/sec execute 45 sec latency 15.471 ms 2024-06-04T05:01:53.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:53 smithi112 ceph-mon[36453]: pgmap v144: 97 pgs: 97 active+clean; 49 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 1.8 MiB/s wr, 209 op/s 2024-06-04T05:01:53.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:53 smithi098 ceph-mon[35569]: pgmap v144: 97 pgs: 97 active+clean; 49 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 1.8 MiB/s wr, 209 op/s 2024-06-04T05:01:53.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:53 smithi070 ceph-mon[32631]: pgmap v144: 97 pgs: 97 active+clean; 49 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 1.8 MiB/s wr, 209 op/s 2024-06-04T05:01:53.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 59275 1.57 MB/sec execute 46 sec latency 16.281 ms 2024-06-04T05:01:54.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:54 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:01:54.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:54 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:01:54.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:54 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:01:54.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 60069 1.68 MB/sec execute 47 sec latency 11.801 ms 2024-06-04T05:01:55.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:55 smithi112 ceph-mon[36453]: pgmap v145: 97 pgs: 97 active+clean; 49 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 1.2 MiB/s wr, 136 op/s 2024-06-04T05:01:55.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:55 smithi098 ceph-mon[35569]: pgmap v145: 97 pgs: 97 active+clean; 49 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 1.2 MiB/s wr, 136 op/s 2024-06-04T05:01:55.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:55 smithi070 ceph-mon[32631]: pgmap v145: 97 pgs: 97 active+clean; 49 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 1.2 MiB/s wr, 136 op/s 2024-06-04T05:01:55.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 60538 1.74 MB/sec execute 48 sec latency 14.373 ms 2024-06-04T05:01:56.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 60795 1.71 MB/sec execute 49 sec latency 15.299 ms 2024-06-04T05:01:57.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:57 smithi112 ceph-mon[36453]: pgmap v146: 97 pgs: 97 active+clean; 50 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.9 MiB/s wr, 197 op/s 2024-06-04T05:01:57.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:57 smithi098 ceph-mon[35569]: pgmap v146: 97 pgs: 97 active+clean; 50 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.9 MiB/s wr, 197 op/s 2024-06-04T05:01:57.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:57 smithi070 ceph-mon[32631]: pgmap v146: 97 pgs: 97 active+clean; 50 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.9 MiB/s wr, 197 op/s 2024-06-04T05:01:57.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 61249 1.69 MB/sec execute 50 sec latency 14.945 ms 2024-06-04T05:01:58.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 61833 1.74 MB/sec execute 51 sec latency 14.735 ms 2024-06-04T05:01:59.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:01:59 smithi112 ceph-mon[36453]: pgmap v147: 97 pgs: 97 active+clean; 50 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.4 MiB/s wr, 130 op/s 2024-06-04T05:01:59.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:01:59 smithi098 ceph-mon[35569]: pgmap v147: 97 pgs: 97 active+clean; 50 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.4 MiB/s wr, 130 op/s 2024-06-04T05:01:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.543+0000 7f02f46c5640 1 -- 172.21.15.70:0/1347830077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02ec101980 msgr2=0x7f02ec101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.543+0000 7f02f46c5640 1 --2- 172.21.15.70:0/1347830077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02ec101980 0x7f02ec101e00 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009930 tx=0x7f02dc02a420 comp rx=0 tx=0).stop 2024-06-04T05:01:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.543+0000 7f02f46c5640 1 -- 172.21.15.70:0/1347830077 shutdown_connections 2024-06-04T05:01:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.543+0000 7f02f46c5640 1 --2- 172.21.15.70:0/1347830077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02ec102340 0x7f02ec10e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.543+0000 7f02f46c5640 1 --2- 172.21.15.70:0/1347830077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02ec101980 0x7f02ec101e00 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.543+0000 7f02f46c5640 1 --2- 172.21.15.70:0/1347830077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02ec107940 0x7f02ec107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.543+0000 7f02f46c5640 1 -- 172.21.15.70:0/1347830077 >> 172.21.15.70:0/1347830077 conn(0x7f02ec078ad0 msgr2=0x7f02ec078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.543+0000 7f02f46c5640 1 -- 172.21.15.70:0/1347830077 shutdown_connections 2024-06-04T05:01:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.543+0000 7f02f46c5640 1 -- 172.21.15.70:0/1347830077 wait complete. 2024-06-04T05:01:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.543+0000 7f02f46c5640 1 Processor -- start 2024-06-04T05:01:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.543+0000 7f02f46c5640 1 -- start start 2024-06-04T05:01:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f46c5640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02ec101980 0x7f02ec198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f46c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02ec102340 0x7f02ec198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f46c5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02ec107940 0x7f02ec19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f46c5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec111ab0 con 0x7f02ec107940 2024-06-04T05:01:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f46c5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec111930 con 0x7f02ec102340 2024-06-04T05:01:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f46c5640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec111c30 con 0x7f02ec101980 2024-06-04T05:01:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f2c62640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02ec107940 0x7f02ec19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f2c62640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02ec107940 0x7f02ec19c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60772/0 (socket says 172.21.15.70:60772) 2024-06-04T05:01:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f2c62640 1 -- 172.21.15.70:0/1804178593 learned_addr learned my addr 172.21.15.70:0/1804178593 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f2461640 1 --2- 172.21.15.70:0/1804178593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02ec101980 0x7f02ec198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f1c60640 1 --2- 172.21.15.70:0/1804178593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02ec102340 0x7f02ec198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f2c62640 1 -- 172.21.15.70:0/1804178593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02ec101980 msgr2=0x7f02ec198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f2c62640 1 --2- 172.21.15.70:0/1804178593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02ec101980 0x7f02ec198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f2c62640 1 -- 172.21.15.70:0/1804178593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02ec102340 msgr2=0x7f02ec198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:59.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f2c62640 1 --2- 172.21.15.70:0/1804178593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02ec102340 0x7f02ec198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f2c62640 1 -- 172.21.15.70:0/1804178593 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02ec19d000 con 0x7f02ec107940 2024-06-04T05:01:59.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f2c62640 1 --2- 172.21.15.70:0/1804178593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02ec107940 0x7f02ec19c8d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f02e4010d20 tx=0x7f02e400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:59.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02e37fe640 1 -- 172.21.15.70:0/1804178593 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e4004d20 con 0x7f02ec107940 2024-06-04T05:01:59.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f46c5640 1 -- 172.21.15.70:0/1804178593 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02ec19d2f0 con 0x7f02ec107940 2024-06-04T05:01:59.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02e37fe640 1 -- 172.21.15.70:0/1804178593 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02e40054b0 con 0x7f02ec107940 2024-06-04T05:01:59.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02e37fe640 1 -- 172.21.15.70:0/1804178593 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e4018850 con 0x7f02ec107940 2024-06-04T05:01:59.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.544+0000 7f02f46c5640 1 -- 172.21.15.70:0/1804178593 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02ec06a100 con 0x7f02ec107940 2024-06-04T05:01:59.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.547+0000 7f02f73d7740 1 Processor -- start 2024-06-04T05:01:59.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.547+0000 7f02e37fe640 1 -- 172.21.15.70:0/1804178593 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f02e4005020 con 0x7f02ec107940 2024-06-04T05:01:59.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.547+0000 7f02f73d7740 1 -- start start 2024-06-04T05:01:59.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f73d7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560af820 0x55f0560afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f73d7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560b0140 0x55f0560ba4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f73d7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560baa00 0x55f0560bcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f73d7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f056021c30 con 0x55f0560b0140 2024-06-04T05:01:59.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f73d7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f0560117d0 con 0x55f0560baa00 2024-06-04T05:01:59.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f73d7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f056015810 con 0x55f0560af820 2024-06-04T05:01:59.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f2461640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560b0140 0x55f0560ba4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f2461640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560b0140 0x55f0560ba4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60784/0 (socket says 172.21.15.70:60784) 2024-06-04T05:01:59.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f2461640 1 -- 172.21.15.70:0/1458561058 learned_addr learned my addr 172.21.15.70:0/1458561058 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:59.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f1c60640 1 --2- 172.21.15.70:0/1458561058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560baa00 0x55f0560bcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f1c60640 1 --2- 172.21.15.70:0/1458561058 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560af820 0x55f0560afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f2461640 1 -- 172.21.15.70:0/1458561058 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560af820 msgr2=0x55f0560afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:59.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f2461640 1 --2- 172.21.15.70:0/1458561058 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560af820 0x55f0560afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02e37fe640 1 --2- 172.21.15.70:0/1804178593 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02d80b2a10 0x7f02d80b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f2461640 1 -- 172.21.15.70:0/1458561058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560baa00 msgr2=0x55f0560bcdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:59.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f2461640 1 --2- 172.21.15.70:0/1458561058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560baa00 0x55f0560bcdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f2461640 1 -- 172.21.15.70:0/1458561058 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f055d26210 con 0x55f0560b0140 2024-06-04T05:01:59.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f1c60640 1 --2- 172.21.15.70:0/1458561058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560baa00 0x55f0560bcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:01:59.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f2c62640 1 --2- 172.21.15.70:0/1804178593 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02d80b2a10 0x7f02d80b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02f2461640 1 --2- 172.21.15.70:0/1458561058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560b0140 0x55f0560ba4c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009ca0 tx=0x7f02e8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda914ed7898f899 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:59.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02e37fe640 1 -- 172.21.15.70:0/1804178593 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f02e40e4660 con 0x7f02ec107940 2024-06-04T05:01:59.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.548+0000 7f02e0ff9640 1 -- 172.21.15.70:0/1458561058 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e801e070 con 0x55f0560b0140 2024-06-04T05:01:59.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.549+0000 7f02e0ff9640 1 -- 172.21.15.70:0/1458561058 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02e8019ac0 con 0x55f0560b0140 2024-06-04T05:01:59.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.549+0000 7f02e0ff9640 1 -- 172.21.15.70:0/1458561058 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e800e8d0 con 0x55f0560b0140 2024-06-04T05:01:59.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.549+0000 7f02f73d7740 1 -- 172.21.15.70:0/1458561058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560b0140 msgr2=0x55f0560ba4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:59.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.549+0000 7f02f2c62640 1 --2- 172.21.15.70:0/1804178593 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02d80b2a10 0x7f02d80b4ed0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f02e40e23b0 tx=0x7f02e40ab7c0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:59.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.549+0000 7f02f73d7740 1 --2- 172.21.15.70:0/1458561058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560b0140 0x55f0560ba4c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009ca0 tx=0x7f02e8011810 comp rx=0 tx=0).stop 2024-06-04T05:01:59.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.549+0000 7f02f73d7740 1 -- 172.21.15.70:0/1458561058 shutdown_connections 2024-06-04T05:01:59.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.549+0000 7f02f73d7740 1 --2- 172.21.15.70:0/1458561058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560baa00 0x55f0560bcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.549+0000 7f02f73d7740 1 --2- 172.21.15.70:0/1458561058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560b0140 0x55f0560ba4c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.549+0000 7f02f73d7740 1 --2- 172.21.15.70:0/1458561058 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560af820 0x55f0560afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.549+0000 7f02f73d7740 1 -- 172.21.15.70:0/1458561058 >> 172.21.15.70:0/1458561058 conn(0x55f0560abd30 msgr2=0x55f0560ac110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:59.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.550+0000 7f02f73d7740 1 -- 172.21.15.70:0/1458561058 shutdown_connections 2024-06-04T05:01:59.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.550+0000 7f02f73d7740 1 -- 172.21.15.70:0/1458561058 wait complete. 2024-06-04T05:01:59.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.550+0000 7f02f73d7740 1 -- start start 2024-06-04T05:01:59.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.550+0000 7f02f73d7740 1 Processor -- start 2024-06-04T05:01:59.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.550+0000 7f02f73d7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560af820 0x55f0560b7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.550+0000 7f02f73d7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560b0140 0x55f0560b7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.550+0000 7f02f73d7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560baa00 0x55f0560b8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.550+0000 7f02f73d7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f055cf7790 con 0x55f0560b0140 2024-06-04T05:01:59.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.550+0000 7f02f73d7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f055fcd890 con 0x55f0560baa00 2024-06-04T05:01:59.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.550+0000 7f02f73d7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f0560bd630 con 0x55f0560af820 2024-06-04T05:01:59.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.550+0000 7f02f73d7740 10 client.0 _ll_register_callbacks cb 0x55f055d25a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:59.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.550+0000 7f02f2461640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560b0140 0x55f0560b7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.550+0000 7f02f1c60640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560af820 0x55f0560b7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.550+0000 7f02f2461640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560b0140 0x55f0560b7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60792/0 (socket says 172.21.15.70:60792) 2024-06-04T05:01:59.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.550+0000 7f02f1c60640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560af820 0x55f0560b7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:55258/0 (socket says 172.21.15.70:55258) 2024-06-04T05:01:59.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.550+0000 7f02f2461640 1 -- 172.21.15.70:0/1417356186 learned_addr learned my addr 172.21.15.70:0/1417356186 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:59.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.550+0000 7f02f1c60640 1 --2- 172.21.15.70:0/1417356186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560baa00 0x55f0560b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.551+0000 7f02f1c60640 1 -- 172.21.15.70:0/1417356186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560baa00 msgr2=0x55f0560b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:59.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.551+0000 7f02f1c60640 1 --2- 172.21.15.70:0/1417356186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560baa00 0x55f0560b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.551+0000 7f02f1c60640 1 -- 172.21.15.70:0/1417356186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560b0140 msgr2=0x55f0560b7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:59.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.551+0000 7f02f1c60640 1 --2- 172.21.15.70:0/1417356186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560b0140 0x55f0560b7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.551+0000 7f02f1c60640 1 -- 172.21.15.70:0/1417356186 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f055df2620 con 0x55f0560af820 2024-06-04T05:01:59.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.551+0000 7f02f2461640 1 --2- 172.21.15.70:0/1417356186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560b0140 0x55f0560b7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:01:59.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.551+0000 7f02f1c60640 1 --2- 172.21.15.70:0/1417356186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560af820 0x55f0560b7a10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009900 tx=0x7f02dc005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd52af3c89956c60 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:59.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.551+0000 7f02d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:01:59.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.551+0000 7f02d5ffb640 1 -- 172.21.15.70:0/1417356186 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc035040 con 0x55f0560af820 2024-06-04T05:01:59.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.551+0000 7f02f73d7740 1 -- 172.21.15.70:0/1417356186 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02e8019ac0 con 0x55f0560af820 2024-06-04T05:01:59.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.551+0000 7f02d5ffb640 1 -- 172.21.15.70:0/1417356186 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f02dc031810 con 0x55f0560af820 2024-06-04T05:01:59.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.551+0000 7f02d5ffb640 1 -- 172.21.15.70:0/1417356186 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f02dc004040 con 0x55f0560af820 2024-06-04T05:01:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.551+0000 7f02f73d7740 10 client.26128 fetch_fsmap learned FSMap version 32 2024-06-04T05:01:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.551+0000 7f02f73d7740 1 -- 172.21.15.70:0/1417356186 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f0560c9460 con 0x55f0560af820 2024-06-04T05:01:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.551+0000 7f02d5ffb640 1 -- 172.21.15.70:0/1417356186 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02e8019ac0 con 0x55f0560af820 2024-06-04T05:01:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.552+0000 7f02d5ffb640 1 -- 172.21.15.70:0/1417356186 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f02dc04d4b0 con 0x55f0560af820 2024-06-04T05:01:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.552+0000 7f02f73d7740 10 client.26128 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:01:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.552+0000 7f02f73d7740 10 client.26128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.552+0000 7f02f73d7740 20 client.26128 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.552+0000 7f02f73d7740 1 --2- 172.21.15.70:0/1417356186 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f0560ca8d0 0x55f0560cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.552+0000 7f02f73d7740 4 client.26128 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:01:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.552+0000 7f02f73d7740 1 -- 172.21.15.70:0/1417356186 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f056023b80 con 0x55f0560ca8d0 2024-06-04T05:01:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.552+0000 7f02f2461640 1 --2- 172.21.15.70:0/1417356186 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f0560ca8d0 0x55f0560cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.552+0000 7f02f2461640 1 --2- 172.21.15.70:0/1417356186 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f0560ca8d0 0x55f0560cccc0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8a5de7500cb9b89 server_cookie=2038787f35fc760f in_seq=0 out_seq=0 2024-06-04T05:01:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.552+0000 7f02d5ffb640 10 client.26128 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:59.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.553+0000 7f02d5ffb640 1 -- 172.21.15.70:0/1417356186 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f056023b80 con 0x55f0560ca8d0 2024-06-04T05:01:59.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.553+0000 7f02d5ffb640 10 client.26128 handle_command_reply: tid=0 2024-06-04T05:01:59.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.553+0000 7f02f73d7740 1 client.26128 shutdown 2024-06-04T05:01:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.553+0000 7f02f73d7740 1 -- 172.21.15.70:0/1417356186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560af820 msgr2=0x55f0560b7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.553+0000 7f02f73d7740 1 --2- 172.21.15.70:0/1417356186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560af820 0x55f0560b7a10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009900 tx=0x7f02dc005a20 comp rx=0 tx=0).stop 2024-06-04T05:01:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.553+0000 7f02f73d7740 1 -- 172.21.15.70:0/1417356186 shutdown_connections 2024-06-04T05:01:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.553+0000 7f02f73d7740 1 --2- 172.21.15.70:0/1417356186 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f0560ca8d0 0x55f0560cccc0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.553+0000 7f02f73d7740 1 --2- 172.21.15.70:0/1417356186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560baa00 0x55f0560b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.553+0000 7f02f73d7740 1 --2- 172.21.15.70:0/1417356186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560b0140 0x55f0560b7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.553+0000 7f02d5ffb640 0 client.26128 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.553+0000 7f02f73d7740 1 --2- 172.21.15.70:0/1417356186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560af820 0x55f0560b7a10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.553+0000 7f02f73d7740 1 -- 172.21.15.70:0/1417356186 >> 172.21.15.70:0/1417356186 conn(0x55f0560abd30 msgr2=0x55f0560aafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.553+0000 7f02f73d7740 1 -- 172.21.15.70:0/1417356186 shutdown_connections 2024-06-04T05:01:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.553+0000 7f02f73d7740 1 -- 172.21.15.70:0/1417356186 wait complete. 2024-06-04T05:01:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.553+0000 7f02f73d7740 20 client.26128 trim_cache size 0 max 16384 2024-06-04T05:01:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.570+0000 7f02f73d7740 1 Processor -- start 2024-06-04T05:01:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f73d7740 1 -- start start 2024-06-04T05:01:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f73d7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560af820 0x55f0560b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f73d7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560b0140 0x55f0560b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f73d7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560baa00 0x55f0560b84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f73d7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec1ac370 con 0x55f0560baa00 2024-06-04T05:01:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f73d7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec1adeb0 con 0x55f0560b0140 2024-06-04T05:01:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f73d7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec1aca40 con 0x55f0560af820 2024-06-04T05:01:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f1c60640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560baa00 0x55f0560b84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f1c60640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560baa00 0x55f0560b84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60804/0 (socket says 172.21.15.70:60804) 2024-06-04T05:01:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f2461640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560b0140 0x55f0560b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f1c60640 1 -- 172.21.15.70:0/679885396 learned_addr learned my addr 172.21.15.70:0/679885396 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:59.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f1c60640 1 --2- 172.21.15.70:0/679885396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560af820 0x55f0560b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f2461640 1 -- 172.21.15.70:0/679885396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560af820 msgr2=0x55f0560b21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:59.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f2461640 1 --2- 172.21.15.70:0/679885396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560af820 0x55f0560b21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f2461640 1 -- 172.21.15.70:0/679885396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560baa00 msgr2=0x55f0560b84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:59.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f2461640 1 --2- 172.21.15.70:0/679885396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560baa00 0x55f0560b84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f2461640 1 -- 172.21.15.70:0/679885396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0560bd510 con 0x55f0560b0140 2024-06-04T05:01:59.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02f2461640 1 --2- 172.21.15.70:0/679885396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560b0140 0x55f0560b7f60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f02e800e5b0 tx=0x7f02e8011e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b9a222fda0ccfd7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:59.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.571+0000 7f02d7fff640 1 -- 172.21.15.70:0/679885396 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e801e040 con 0x55f0560b0140 2024-06-04T05:01:59.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.572+0000 7f02d7fff640 1 -- 172.21.15.70:0/679885396 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02e801f040 con 0x55f0560b0140 2024-06-04T05:01:59.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.572+0000 7f02d7fff640 1 -- 172.21.15.70:0/679885396 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e80195d0 con 0x55f0560b0140 2024-06-04T05:01:59.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.572+0000 7f02f73d7740 1 -- 172.21.15.70:0/679885396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560b0140 msgr2=0x55f0560b7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.572+0000 7f02f73d7740 1 --2- 172.21.15.70:0/679885396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560b0140 0x55f0560b7f60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f02e800e5b0 tx=0x7f02e8011e40 comp rx=0 tx=0).stop 2024-06-04T05:01:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.572+0000 7f02f73d7740 1 -- 172.21.15.70:0/679885396 shutdown_connections 2024-06-04T05:01:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.572+0000 7f02f73d7740 1 --2- 172.21.15.70:0/679885396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560b0140 0x55f0560b7f60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.572+0000 7f02f73d7740 1 --2- 172.21.15.70:0/679885396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560baa00 0x55f0560b84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.572+0000 7f02f73d7740 1 --2- 172.21.15.70:0/679885396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560af820 0x55f0560b21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.572+0000 7f02f73d7740 1 -- 172.21.15.70:0/679885396 >> 172.21.15.70:0/679885396 conn(0x55f0560abd30 msgr2=0x55f0560c5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.572+0000 7f02f73d7740 1 -- 172.21.15.70:0/679885396 shutdown_connections 2024-06-04T05:01:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.572+0000 7f02f73d7740 1 -- 172.21.15.70:0/679885396 wait complete. 2024-06-04T05:01:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f73d7740 1 -- start start 2024-06-04T05:01:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f73d7740 1 Processor -- start 2024-06-04T05:01:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f73d7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560af820 0x55f0560c2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f73d7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560b0140 0x55f0560c2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f73d7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560baa00 0x55f0560bde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f73d7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec1ad7e0 con 0x55f0560af820 2024-06-04T05:01:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f73d7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec1ad110 con 0x55f0560b0140 2024-06-04T05:01:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f73d7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec1aa350 con 0x55f0560baa00 2024-06-04T05:01:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f73d7740 10 client.0 _ll_register_callbacks cb 0x7f02e8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:01:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f1c60640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560af820 0x55f0560c2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f1c60640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560af820 0x55f0560c2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60820/0 (socket says 172.21.15.70:60820) 2024-06-04T05:01:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f1c60640 1 -- 172.21.15.70:0/2409054125 learned_addr learned my addr 172.21.15.70:0/2409054125 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:01:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f2461640 1 --2- 172.21.15.70:0/2409054125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560b0140 0x55f0560c2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f1c60640 1 --2- 172.21.15.70:0/2409054125 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560baa00 0x55f0560bde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f1c60640 1 -- 172.21.15.70:0/2409054125 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560baa00 msgr2=0x55f0560bde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f1c60640 1 --2- 172.21.15.70:0/2409054125 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560baa00 0x55f0560bde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f1c60640 1 -- 172.21.15.70:0/2409054125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560b0140 msgr2=0x55f0560c2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:01:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f1c60640 1 --2- 172.21.15.70:0/2409054125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560b0140 0x55f0560c2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f1c60640 1 -- 172.21.15.70:0/2409054125 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0560be540 con 0x55f0560af820 2024-06-04T05:01:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02f1c60640 1 --2- 172.21.15.70:0/2409054125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560af820 0x55f0560c2800 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f02dc0057c0 tx=0x7f02dc031b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563c8fc9eb174d72 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:01:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.573+0000 7f02d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:01:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.574+0000 7f02d5ffb640 1 -- 172.21.15.70:0/2409054125 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc035030 con 0x55f0560af820 2024-06-04T05:01:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.574+0000 7f02f73d7740 1 -- 172.21.15.70:0/2409054125 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02e801f040 con 0x55f0560af820 2024-06-04T05:01:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.574+0000 7f02d5ffb640 1 -- 172.21.15.70:0/2409054125 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f02dc04d0b0 con 0x55f0560af820 2024-06-04T05:01:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.574+0000 7f02d5ffb640 1 -- 172.21.15.70:0/2409054125 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f02dc069050 con 0x55f0560af820 2024-06-04T05:01:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.574+0000 7f02f73d7740 10 client.16779 fetch_fsmap learned FSMap version 32 2024-06-04T05:01:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.574+0000 7f02f73d7740 1 -- 172.21.15.70:0/2409054125 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f056134d80 con 0x55f0560af820 2024-06-04T05:01:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.574+0000 7f02d5ffb640 1 -- 172.21.15.70:0/2409054125 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02e801f040 con 0x55f0560af820 2024-06-04T05:01:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.574+0000 7f02d5ffb640 1 -- 172.21.15.70:0/2409054125 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f02dc003b60 con 0x55f0560af820 2024-06-04T05:01:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.574+0000 7f02f73d7740 10 client.16779 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:01:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.574+0000 7f02f73d7740 10 client.16779 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:01:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.574+0000 7f02f73d7740 20 client.16779 populate_metadata read hostname 'smithi070' 2024-06-04T05:01:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.574+0000 7f02f73d7740 1 --2- 172.21.15.70:0/2409054125 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f0561354a0 0x55f056137890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:01:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.574+0000 7f02f73d7740 4 client.16779 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:01:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.574+0000 7f02f73d7740 1 -- 172.21.15.70:0/2409054125 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f05601d360 con 0x55f0561354a0 2024-06-04T05:01:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.574+0000 7f02f2461640 1 --2- 172.21.15.70:0/2409054125 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f0561354a0 0x55f056137890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:01:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.575+0000 7f02f2461640 1 --2- 172.21.15.70:0/2409054125 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f0561354a0 0x55f056137890 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2b634be65ac85aa server_cookie=bcacbb196daef098 in_seq=0 out_seq=0 2024-06-04T05:01:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.575+0000 7f02d5ffb640 10 client.16779 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:59.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.595+0000 7f02d5ffb640 1 -- 172.21.15.70:0/2409054125 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f05601d360 con 0x55f0561354a0 2024-06-04T05:01:59.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.595+0000 7f02d5ffb640 10 client.16779 handle_command_reply: tid=0 2024-06-04T05:01:59.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.595+0000 7f02f73d7740 1 client.16779 shutdown 2024-06-04T05:01:59.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.595+0000 7f02f73d7740 1 -- 172.21.15.70:0/2409054125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560af820 msgr2=0x55f0560c2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:59.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.595+0000 7f02f73d7740 1 --2- 172.21.15.70:0/2409054125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560af820 0x55f0560c2800 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f02dc0057c0 tx=0x7f02dc031b80 comp rx=0 tx=0).stop 2024-06-04T05:01:59.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.595+0000 7f02f73d7740 1 -- 172.21.15.70:0/2409054125 shutdown_connections 2024-06-04T05:01:59.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.595+0000 7f02f73d7740 1 --2- 172.21.15.70:0/2409054125 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f0561354a0 0x55f056137890 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.595+0000 7f02f73d7740 1 --2- 172.21.15.70:0/2409054125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0560b0140 0x55f0560c2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.595+0000 7f02d5ffb640 0 client.16779 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:01:59.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.595+0000 7f02f73d7740 1 --2- 172.21.15.70:0/2409054125 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0560baa00 0x55f0560bde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.595+0000 7f02f73d7740 1 --2- 172.21.15.70:0/2409054125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0560af820 0x55f0560c2800 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.595+0000 7f02f73d7740 1 -- 172.21.15.70:0/2409054125 >> 172.21.15.70:0/2409054125 conn(0x55f0560abd30 msgr2=0x55f0560aa720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:59.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.596+0000 7f02f73d7740 1 -- 172.21.15.70:0/2409054125 shutdown_connections 2024-06-04T05:01:59.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.596+0000 7f02f73d7740 1 -- 172.21.15.70:0/2409054125 wait complete. 2024-06-04T05:01:59.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.596+0000 7f02f73d7740 20 client.16779 trim_cache size 0 max 16384 2024-06-04T05:01:59.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:01:59.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:01:59.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:01:59.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.596+0000 7f02f46c5640 1 -- 172.21.15.70:0/1804178593 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02d80b2a10 msgr2=0x7f02d80b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:59.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.596+0000 7f02f46c5640 1 --2- 172.21.15.70:0/1804178593 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02d80b2a10 0x7f02d80b4ed0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f02e40e23b0 tx=0x7f02e40ab7c0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.596+0000 7f02f46c5640 1 -- 172.21.15.70:0/1804178593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02ec107940 msgr2=0x7f02ec19c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:01:59.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.596+0000 7f02f46c5640 1 --2- 172.21.15.70:0/1804178593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02ec107940 0x7f02ec19c8d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f02e4010d20 tx=0x7f02e400c6a0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.596+0000 7f02f46c5640 1 -- 172.21.15.70:0/1804178593 shutdown_connections 2024-06-04T05:01:59.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.596+0000 7f02f46c5640 1 --2- 172.21.15.70:0/1804178593 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02d80b2a10 0x7f02d80b4ed0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.596+0000 7f02f46c5640 1 --2- 172.21.15.70:0/1804178593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02ec102340 0x7f02ec198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.596+0000 7f02f46c5640 1 --2- 172.21.15.70:0/1804178593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02ec107940 0x7f02ec19c8d0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.596+0000 7f02f46c5640 1 --2- 172.21.15.70:0/1804178593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02ec101980 0x7f02ec198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:01:59.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.596+0000 7f02f46c5640 1 -- 172.21.15.70:0/1804178593 >> 172.21.15.70:0/1804178593 conn(0x7f02ec078ad0 msgr2=0x7f02ec104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:01:59.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.596+0000 7f02f46c5640 1 -- 172.21.15.70:0/1804178593 shutdown_connections 2024-06-04T05:01:59.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:01:59.596+0000 7f02f46c5640 1 -- 172.21.15.70:0/1804178593 wait complete. 2024-06-04T05:01:59.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:01:59 smithi070 ceph-mon[32631]: pgmap v147: 97 pgs: 97 active+clean; 50 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.4 MiB/s wr, 130 op/s 2024-06-04T05:01:59.719 INFO:tasks.workunit.client.0.smithi070.stdout: 1 62508 1.72 MB/sec execute 52 sec latency 19.503 ms 2024-06-04T05:02:00.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 63087 1.79 MB/sec execute 53 sec latency 15.892 ms 2024-06-04T05:02:01.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:01 smithi112 ceph-mon[36453]: pgmap v148: 97 pgs: 97 active+clean; 50 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.4 MiB/s wr, 130 op/s 2024-06-04T05:02:01.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:01 smithi098 ceph-mon[35569]: pgmap v148: 97 pgs: 97 active+clean; 50 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.4 MiB/s wr, 130 op/s 2024-06-04T05:02:01.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:01 smithi070 ceph-mon[32631]: pgmap v148: 97 pgs: 97 active+clean; 50 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.4 MiB/s wr, 130 op/s 2024-06-04T05:02:01.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 63647 1.82 MB/sec execute 54 sec latency 58.074 ms 2024-06-04T05:02:02.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 64106 1.86 MB/sec execute 55 sec latency 92.188 ms 2024-06-04T05:02:03.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:03 smithi112 ceph-mon[36453]: pgmap v149: 97 pgs: 97 active+clean; 47 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 2.0 MiB/s wr, 202 op/s 2024-06-04T05:02:03.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:03 smithi098 ceph-mon[35569]: pgmap v149: 97 pgs: 97 active+clean; 47 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 2.0 MiB/s wr, 202 op/s 2024-06-04T05:02:03.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:03 smithi070 ceph-mon[32631]: pgmap v149: 97 pgs: 97 active+clean; 47 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 2.0 MiB/s wr, 202 op/s 2024-06-04T05:02:03.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 64380 1.84 MB/sec execute 56 sec latency 15.192 ms 2024-06-04T05:02:04.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 64850 1.82 MB/sec execute 57 sec latency 12.693 ms 2024-06-04T05:02:05.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:05 smithi098 ceph-mon[35569]: pgmap v150: 97 pgs: 97 active+clean; 47 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 1.3 MiB/s wr, 133 op/s 2024-06-04T05:02:05.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:05 smithi070 ceph-mon[32631]: pgmap v150: 97 pgs: 97 active+clean; 47 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 1.3 MiB/s wr, 133 op/s 2024-06-04T05:02:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:05 smithi112 ceph-mon[36453]: pgmap v150: 97 pgs: 97 active+clean; 47 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 1.3 MiB/s wr, 133 op/s 2024-06-04T05:02:05.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 65444 1.86 MB/sec execute 58 sec latency 15.513 ms 2024-06-04T05:02:06.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 66172 1.86 MB/sec execute 59 sec latency 15.421 ms 2024-06-04T05:02:07.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:07 smithi098 ceph-mon[35569]: pgmap v151: 97 pgs: 97 active+clean; 51 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 1.9 MiB/s wr, 213 op/s 2024-06-04T05:02:07.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:07 smithi070 ceph-mon[32631]: pgmap v151: 97 pgs: 97 active+clean; 51 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 1.9 MiB/s wr, 213 op/s 2024-06-04T05:02:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:07 smithi112 ceph-mon[36453]: pgmap v151: 97 pgs: 97 active+clean; 51 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 1.9 MiB/s wr, 213 op/s 2024-06-04T05:02:07.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 66793 1.92 MB/sec execute 60 sec latency 14.429 ms 2024-06-04T05:02:08.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 67261 1.92 MB/sec execute 61 sec latency 15.297 ms 2024-06-04T05:02:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:09 smithi098 ceph-mon[35569]: pgmap v152: 97 pgs: 97 active+clean; 51 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 1.2 MiB/s wr, 152 op/s 2024-06-04T05:02:09.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:09 smithi070 ceph-mon[32631]: pgmap v152: 97 pgs: 97 active+clean; 51 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 1.2 MiB/s wr, 152 op/s 2024-06-04T05:02:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:09 smithi112 ceph-mon[36453]: pgmap v152: 97 pgs: 97 active+clean; 51 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 1.2 MiB/s wr, 152 op/s 2024-06-04T05:02:09.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 67757 1.97 MB/sec execute 62 sec latency 14.734 ms 2024-06-04T05:02:10.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 68078 1.94 MB/sec execute 63 sec latency 12.126 ms 2024-06-04T05:02:11.421 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:02:11.446 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:11 smithi070 ceph-mon[32631]: pgmap v153: 97 pgs: 97 active+clean; 51 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 1.2 MiB/s wr, 152 op/s 2024-06-04T05:02:11.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:11 smithi098 ceph-mon[35569]: pgmap v153: 97 pgs: 97 active+clean; 51 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 1.2 MiB/s wr, 152 op/s 2024-06-04T05:02:11.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:11 smithi112 ceph-mon[36453]: pgmap v153: 97 pgs: 97 active+clean; 51 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 1.2 MiB/s wr, 152 op/s 2024-06-04T05:02:11.686 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:02:11.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 68628 1.94 MB/sec execute 64 sec latency 11.335 ms 2024-06-04T05:02:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.697+0000 7f2f284c8640 1 -- 172.21.15.70:0/2084061937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f20104640 msgr2=0x7f2f2010cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.697+0000 7f2f284c8640 1 --2- 172.21.15.70:0/2084061937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f20104640 0x7f2f2010cb20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f2f10009900 tx=0x7f2f1002a480 comp rx=0 tx=0).stop 2024-06-04T05:02:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.697+0000 7f2f284c8640 1 -- 172.21.15.70:0/2084061937 shutdown_connections 2024-06-04T05:02:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.697+0000 7f2f284c8640 1 --2- 172.21.15.70:0/2084061937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f2010d060 0x7f2f2010f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.697+0000 7f2f284c8640 1 --2- 172.21.15.70:0/2084061937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f20104640 0x7f2f2010cb20 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.697+0000 7f2f284c8640 1 --2- 172.21.15.70:0/2084061937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f20103d20 0x7f2f20104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.697+0000 7f2f284c8640 1 -- 172.21.15.70:0/2084061937 >> 172.21.15.70:0/2084061937 conn(0x7f2f20078ad0 msgr2=0x7f2f20078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.697+0000 7f2f284c8640 1 -- 172.21.15.70:0/2084061937 shutdown_connections 2024-06-04T05:02:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.697+0000 7f2f284c8640 1 -- 172.21.15.70:0/2084061937 wait complete. 2024-06-04T05:02:12.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.698+0000 7f2f284c8640 1 Processor -- start 2024-06-04T05:02:12.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.698+0000 7f2f284c8640 1 -- start start 2024-06-04T05:02:12.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.698+0000 7f2f284c8640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f20103d20 0x7f2f201990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.698+0000 7f2f284c8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f20104640 0x7f2f20199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.698+0000 7f2f284c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f2010d060 0x7f2f2019d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.698+0000 7f2f284c8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2f20112b70 con 0x7f2f2010d060 2024-06-04T05:02:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.698+0000 7f2f284c8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2f201129f0 con 0x7f2f20104640 2024-06-04T05:02:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.698+0000 7f2f284c8640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2f20112cf0 con 0x7f2f20103d20 2024-06-04T05:02:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.698+0000 7f2f26a65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f2010d060 0x7f2f2019d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.698+0000 7f2f26a65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f2010d060 0x7f2f2019d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49644/0 (socket says 172.21.15.70:49644) 2024-06-04T05:02:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.698+0000 7f2f26264640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f20103d20 0x7f2f201990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.698+0000 7f2f26a65640 1 -- 172.21.15.70:0/671309682 learned_addr learned my addr 172.21.15.70:0/671309682 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.699+0000 7f2f25a63640 1 --2- 172.21.15.70:0/671309682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f20104640 0x7f2f20199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.699+0000 7f2f26a65640 1 -- 172.21.15.70:0/671309682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f20103d20 msgr2=0x7f2f201990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.699+0000 7f2f26a65640 1 --2- 172.21.15.70:0/671309682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f20103d20 0x7f2f201990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.699+0000 7f2f26a65640 1 -- 172.21.15.70:0/671309682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f20104640 msgr2=0x7f2f20199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.699+0000 7f2f26a65640 1 --2- 172.21.15.70:0/671309682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f20104640 0x7f2f20199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.699+0000 7f2f26a65640 1 -- 172.21.15.70:0/671309682 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2019e0e0 con 0x7f2f2010d060 2024-06-04T05:02:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.699+0000 7f2f26264640 1 --2- 172.21.15.70:0/671309682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f20103d20 0x7f2f201990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.699+0000 7f2f26a65640 1 --2- 172.21.15.70:0/671309682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f2010d060 0x7f2f2019d9b0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2f1800eb10 tx=0x7f2f1800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.699+0000 7f2f177fe640 1 -- 172.21.15.70:0/671309682 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f18004b20 con 0x7f2f2010d060 2024-06-04T05:02:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.699+0000 7f2f177fe640 1 -- 172.21.15.70:0/671309682 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2f18004cc0 con 0x7f2f2010d060 2024-06-04T05:02:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.699+0000 7f2f284c8640 1 -- 172.21.15.70:0/671309682 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f2019e3d0 con 0x7f2f2010d060 2024-06-04T05:02:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.699+0000 7f2f177fe640 1 -- 172.21.15.70:0/671309682 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f18010690 con 0x7f2f2010d060 2024-06-04T05:02:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.699+0000 7f2f284c8640 1 -- 172.21.15.70:0/671309682 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f201a5c60 con 0x7f2f2010d060 2024-06-04T05:02:12.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.702+0000 7f2f177fe640 1 -- 172.21.15.70:0/671309682 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f2f18005020 con 0x7f2f2010d060 2024-06-04T05:02:12.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.702+0000 7f2f2b1e3740 1 Processor -- start 2024-06-04T05:02:12.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.702+0000 7f2f2b1e3740 1 -- start start 2024-06-04T05:02:12.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.702+0000 7f2f2b1e3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5859820 0x563cc5859c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.702+0000 7f2f2b1e3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc585a140 0x563cc58644c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.702+0000 7f2f2b1e3740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5864a00 0x563cc5866df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.702+0000 7f2f2b1e3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563cc57cbc30 con 0x563cc5859820 2024-06-04T05:02:12.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.702+0000 7f2f2b1e3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563cc57bb7d0 con 0x563cc585a140 2024-06-04T05:02:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.702+0000 7f2f2b1e3740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563cc57bf810 con 0x563cc5864a00 2024-06-04T05:02:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.703+0000 7f2f177fe640 1 --2- 172.21.15.70:0/671309682 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2f0c0b2a10 0x7f2f0c0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.703+0000 7f2f25a63640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5859820 0x563cc5859c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.703+0000 7f2f25a63640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5859820 0x563cc5859c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49660/0 (socket says 172.21.15.70:49660) 2024-06-04T05:02:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.703+0000 7f2f25a63640 1 -- 172.21.15.70:0/4050628942 learned_addr learned my addr 172.21.15.70:0/4050628942 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.703+0000 7f2f25a63640 1 --2- 172.21.15.70:0/4050628942 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5864a00 0x563cc5866df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.703+0000 7f2f26264640 1 --2- 172.21.15.70:0/4050628942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc585a140 0x563cc58644c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.703+0000 7f2f26a65640 1 --2- 172.21.15.70:0/671309682 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2f0c0b2a10 0x7f2f0c0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.703+0000 7f2f25a63640 1 -- 172.21.15.70:0/4050628942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc585a140 msgr2=0x563cc58644c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.703+0000 7f2f25a63640 1 --2- 172.21.15.70:0/4050628942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc585a140 0x563cc58644c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.703+0000 7f2f25a63640 1 -- 172.21.15.70:0/4050628942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5859820 msgr2=0x563cc5859c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.703+0000 7f2f25a63640 1 --2- 172.21.15.70:0/4050628942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5859820 0x563cc5859c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.703+0000 7f2f177fe640 1 -- 172.21.15.70:0/671309682 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f2f180e2b40 con 0x7f2f2010d060 2024-06-04T05:02:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.703+0000 7f2f25a63640 1 -- 172.21.15.70:0/4050628942 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cc54d0210 con 0x563cc5864a00 2024-06-04T05:02:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.703+0000 7f2f25a63640 1 --2- 172.21.15.70:0/4050628942 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5864a00 0x563cc5866df0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2f10005c40 tx=0x7f2f10009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53a39325e1965760 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.703+0000 7f2f26a65640 1 --2- 172.21.15.70:0/671309682 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2f0c0b2a10 0x7f2f0c0b4ed0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2f180a9db0 tx=0x7f2f1806f7b0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.703+0000 7f2f14bfe640 1 -- 172.21.15.70:0/4050628942 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f10041020 con 0x563cc5864a00 2024-06-04T05:02:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.703+0000 7f2f14bfe640 1 -- 172.21.15.70:0/4050628942 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2f100044c0 con 0x563cc5864a00 2024-06-04T05:02:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.704+0000 7f2f14bfe640 1 -- 172.21.15.70:0/4050628942 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f10035040 con 0x563cc5864a00 2024-06-04T05:02:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.704+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/4050628942 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5864a00 msgr2=0x563cc5866df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.704+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/4050628942 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5864a00 0x563cc5866df0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2f10005c40 tx=0x7f2f10009260 comp rx=0 tx=0).stop 2024-06-04T05:02:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.704+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/4050628942 shutdown_connections 2024-06-04T05:02:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.704+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/4050628942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc585a140 0x563cc58644c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.704+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/4050628942 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5864a00 0x563cc5866df0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.704+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/4050628942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5859820 0x563cc5859c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.704+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/4050628942 >> 172.21.15.70:0/4050628942 conn(0x563cc5855d30 msgr2=0x563cc5856110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.704+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/4050628942 shutdown_connections 2024-06-04T05:02:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.704+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/4050628942 wait complete. 2024-06-04T05:02:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.705+0000 7f2f2b1e3740 1 -- start start 2024-06-04T05:02:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.705+0000 7f2f2b1e3740 1 Processor -- start 2024-06-04T05:02:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.705+0000 7f2f2b1e3740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5859820 0x563cc58619c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.705+0000 7f2f2b1e3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc585a140 0x563cc5861f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.705+0000 7f2f2b1e3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5864a00 0x563cc5862440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.705+0000 7f2f2b1e3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563cc54a1790 con 0x563cc5864a00 2024-06-04T05:02:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.705+0000 7f2f2b1e3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563cc5777890 con 0x563cc585a140 2024-06-04T05:02:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.705+0000 7f2f2b1e3740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563cc58675d0 con 0x563cc5859820 2024-06-04T05:02:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.705+0000 7f2f2b1e3740 10 client.0 _ll_register_callbacks cb 0x563cc54cfa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.705+0000 7f2f25a63640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5859820 0x563cc58619c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.705+0000 7f2f25a63640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5859820 0x563cc58619c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:44466/0 (socket says 172.21.15.70:44466) 2024-06-04T05:02:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.705+0000 7f2f25a63640 1 -- 172.21.15.70:0/3271045610 learned_addr learned my addr 172.21.15.70:0/3271045610 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.705+0000 7f2f26264640 1 --2- 172.21.15.70:0/3271045610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc585a140 0x563cc5861f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.705+0000 7f2f25a63640 1 --2- 172.21.15.70:0/3271045610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5864a00 0x563cc5862440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.706+0000 7f2f25a63640 1 -- 172.21.15.70:0/3271045610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc585a140 msgr2=0x563cc5861f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.706+0000 7f2f25a63640 1 --2- 172.21.15.70:0/3271045610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc585a140 0x563cc5861f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.706+0000 7f2f25a63640 1 -- 172.21.15.70:0/3271045610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5864a00 msgr2=0x563cc5862440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.706+0000 7f2f25a63640 1 --2- 172.21.15.70:0/3271045610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5864a00 0x563cc5862440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.706+0000 7f2f25a63640 1 -- 172.21.15.70:0/3271045610 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cc559c620 con 0x563cc5859820 2024-06-04T05:02:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.706+0000 7f2f25a63640 1 --2- 172.21.15.70:0/3271045610 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5859820 0x563cc58619c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2f10001eb0 tx=0x7f2f10032e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39ca9c236279d6f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.706+0000 7f2f09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:02:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.706+0000 7f2f09ffb640 1 -- 172.21.15.70:0/3271045610 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f10041030 con 0x563cc5859820 2024-06-04T05:02:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.706+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/3271045610 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563cc57cdb80 con 0x563cc5859820 2024-06-04T05:02:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.706+0000 7f2f09ffb640 1 -- 172.21.15.70:0/3271045610 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f2f10031a70 con 0x563cc5859820 2024-06-04T05:02:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.706+0000 7f2f09ffb640 1 -- 172.21.15.70:0/3271045610 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2f1005c050 con 0x563cc5859820 2024-06-04T05:02:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.706+0000 7f2f2b1e3740 10 client.26146 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.706+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/3271045610 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563cc5873390 con 0x563cc5859820 2024-06-04T05:02:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.706+0000 7f2f09ffb640 1 -- 172.21.15.70:0/3271045610 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x563cc57cdb80 con 0x563cc5859820 2024-06-04T05:02:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.707+0000 7f2f09ffb640 1 -- 172.21.15.70:0/3271045610 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f2f1003a040 con 0x563cc5859820 2024-06-04T05:02:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.707+0000 7f2f2b1e3740 10 client.26146 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.707+0000 7f2f2b1e3740 10 client.26146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:02:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.707+0000 7f2f2b1e3740 20 client.26146 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.707+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/3271045610 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563cc58746e0 0x563cc5876ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.707+0000 7f2f2b1e3740 4 client.26146 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:02:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.707+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/3271045610 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563cc57c7360 con 0x563cc58746e0 2024-06-04T05:02:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.707+0000 7f2f26264640 1 --2- 172.21.15.70:0/3271045610 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563cc58746e0 0x563cc5876ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.707+0000 7f2f09ffb640 10 client.26146 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.707+0000 7f2f26264640 1 --2- 172.21.15.70:0/3271045610 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563cc58746e0 0x563cc5876ad0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72e5be6a46db4863 server_cookie=9cc9380f49dc8f59 in_seq=0 out_seq=0 2024-06-04T05:02:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.708+0000 7f2f09ffb640 1 -- 172.21.15.70:0/3271045610 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563cc57c7360 con 0x563cc58746e0 2024-06-04T05:02:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.708+0000 7f2f09ffb640 10 client.26146 handle_command_reply: tid=0 2024-06-04T05:02:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.708+0000 7f2f2b1e3740 1 client.26146 shutdown 2024-06-04T05:02:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.708+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/3271045610 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5859820 msgr2=0x563cc58619c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.708+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/3271045610 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5859820 0x563cc58619c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2f10001eb0 tx=0x7f2f10032e10 comp rx=0 tx=0).stop 2024-06-04T05:02:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.708+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/3271045610 shutdown_connections 2024-06-04T05:02:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.708+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/3271045610 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563cc58746e0 0x563cc5876ad0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.708+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/3271045610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc585a140 0x563cc5861f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.716 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.708+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/3271045610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5864a00 0x563cc5862440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.716 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.708+0000 7f2f09ffb640 0 client.26146 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:12.716 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.708+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/3271045610 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5859820 0x563cc58619c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.716 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.708+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/3271045610 >> 172.21.15.70:0/3271045610 conn(0x563cc5855d30 msgr2=0x563cc585bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:12.716 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.708+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/3271045610 shutdown_connections 2024-06-04T05:02:12.716 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.708+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/3271045610 wait complete. 2024-06-04T05:02:12.716 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.708+0000 7f2f2b1e3740 20 client.26146 trim_cache size 0 max 16384 2024-06-04T05:02:12.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 69288 1.96 MB/sec execute 65 sec latency 15.397 ms 2024-06-04T05:02:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.727+0000 7f2f2b1e3740 1 Processor -- start 2024-06-04T05:02:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.727+0000 7f2f2b1e3740 1 -- start start 2024-06-04T05:02:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.727+0000 7f2f2b1e3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5859820 0x563cc5862260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.727+0000 7f2f2b1e3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc585a140 0x563cc5854e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.727+0000 7f2f2b1e3740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5864a00 0x563cc5855370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.727+0000 7f2f2b1e3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2f201ae5f0 con 0x563cc5859820 2024-06-04T05:02:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.727+0000 7f2f2b1e3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2f201b0130 con 0x563cc585a140 2024-06-04T05:02:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.727+0000 7f2f2b1e3740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2f201aecc0 con 0x563cc5864a00 2024-06-04T05:02:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.727+0000 7f2f25a63640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5859820 0x563cc5862260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.727+0000 7f2f25a63640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5859820 0x563cc5862260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49674/0 (socket says 172.21.15.70:49674) 2024-06-04T05:02:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.727+0000 7f2f25a63640 1 -- 172.21.15.70:0/1556107825 learned_addr learned my addr 172.21.15.70:0/1556107825 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.728+0000 7f2f25a63640 1 --2- 172.21.15.70:0/1556107825 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5864a00 0x563cc5855370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.728+0000 7f2f26264640 1 --2- 172.21.15.70:0/1556107825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc585a140 0x563cc5854e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.728+0000 7f2f25a63640 1 -- 172.21.15.70:0/1556107825 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5864a00 msgr2=0x563cc5855370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.728+0000 7f2f25a63640 1 --2- 172.21.15.70:0/1556107825 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5864a00 0x563cc5855370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.728+0000 7f2f25a63640 1 -- 172.21.15.70:0/1556107825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc585a140 msgr2=0x563cc5854e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:12.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.728+0000 7f2f25a63640 1 --2- 172.21.15.70:0/1556107825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc585a140 0x563cc5854e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.728+0000 7f2f25a63640 1 -- 172.21.15.70:0/1556107825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cc58514d0 con 0x563cc5859820 2024-06-04T05:02:12.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.728+0000 7f2f26264640 1 --2- 172.21.15.70:0/1556107825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc585a140 0x563cc5854e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:02:12.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.728+0000 7f2f25a63640 1 --2- 172.21.15.70:0/1556107825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5859820 0x563cc5862260 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f2f100092a0 tx=0x7f2f10009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bd87e17351b8c0b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:12.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.728+0000 7f2f0bfff640 1 -- 172.21.15.70:0/1556107825 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f10035070 con 0x563cc5859820 2024-06-04T05:02:12.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.728+0000 7f2f0bfff640 1 -- 172.21.15.70:0/1556107825 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2f10009780 con 0x563cc5859820 2024-06-04T05:02:12.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.728+0000 7f2f0bfff640 1 -- 172.21.15.70:0/1556107825 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f10041040 con 0x563cc5859820 2024-06-04T05:02:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.729+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/1556107825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5859820 msgr2=0x563cc5862260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.729+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/1556107825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5859820 0x563cc5862260 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f2f100092a0 tx=0x7f2f10009230 comp rx=0 tx=0).stop 2024-06-04T05:02:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.729+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/1556107825 shutdown_connections 2024-06-04T05:02:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.729+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/1556107825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc585a140 0x563cc5854e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.729+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/1556107825 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5864a00 0x563cc5855370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.729+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/1556107825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc5859820 0x563cc5862260 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.729+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/1556107825 >> 172.21.15.70:0/1556107825 conn(0x563cc5855d30 msgr2=0x563cc5854720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.729+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/1556107825 shutdown_connections 2024-06-04T05:02:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.729+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/1556107825 wait complete. 2024-06-04T05:02:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.729+0000 7f2f2b1e3740 1 -- start start 2024-06-04T05:02:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.729+0000 7f2f2b1e3740 1 Processor -- start 2024-06-04T05:02:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f2b1e3740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5859820 0x563cc587a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f2b1e3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc585a140 0x563cc5877520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f2b1e3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc5864a00 0x563cc5877a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f2b1e3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2f201afa60 con 0x563cc585a140 2024-06-04T05:02:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f2b1e3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2f201af390 con 0x563cc5864a00 2024-06-04T05:02:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f2b1e3740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2f201ac5d0 con 0x563cc5859820 2024-06-04T05:02:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f26264640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc585a140 0x563cc5877520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f2b1e3740 10 client.0 _ll_register_callbacks cb 0x7f2f180093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f26264640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc585a140 0x563cc5877520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49680/0 (socket says 172.21.15.70:49680) 2024-06-04T05:02:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f26264640 1 -- 172.21.15.70:0/4130221283 learned_addr learned my addr 172.21.15.70:0/4130221283 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f25a63640 1 --2- 172.21.15.70:0/4130221283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5859820 0x563cc587a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f25a63640 1 --2- 172.21.15.70:0/4130221283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc5864a00 0x563cc5877a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f26264640 1 -- 172.21.15.70:0/4130221283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5859820 msgr2=0x563cc587a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f26264640 1 --2- 172.21.15.70:0/4130221283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5859820 0x563cc587a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f26264640 1 -- 172.21.15.70:0/4130221283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc5864a00 msgr2=0x563cc5877a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f26264640 1 --2- 172.21.15.70:0/4130221283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc5864a00 0x563cc5877a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f26264640 1 -- 172.21.15.70:0/4130221283 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cc5878290 con 0x563cc585a140 2024-06-04T05:02:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f25a63640 1 --2- 172.21.15.70:0/4130221283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc5864a00 0x563cc5877a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:02:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f26264640 1 --2- 172.21.15.70:0/4130221283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc585a140 0x563cc5877520 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c0126a0 tx=0x7f2f1c012b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d725b90ffe9e1bcd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:02:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f09ffb640 1 -- 172.21.15.70:0/4130221283 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c0123f0 con 0x563cc585a140 2024-06-04T05:02:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/4130221283 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f10009780 con 0x563cc585a140 2024-06-04T05:02:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.730+0000 7f2f09ffb640 1 -- 172.21.15.70:0/4130221283 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f2f1c004250 con 0x563cc585a140 2024-06-04T05:02:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.731+0000 7f2f09ffb640 1 -- 172.21.15.70:0/4130221283 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2f1c02f050 con 0x563cc585a140 2024-06-04T05:02:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.731+0000 7f2f2b1e3740 10 client.16797 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.731+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/4130221283 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563cc58def00 con 0x563cc585a140 2024-06-04T05:02:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.731+0000 7f2f09ffb640 1 -- 172.21.15.70:0/4130221283 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2f10009780 con 0x563cc585a140 2024-06-04T05:02:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.731+0000 7f2f09ffb640 1 -- 172.21.15.70:0/4130221283 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f2f1c003960 con 0x563cc585a140 2024-06-04T05:02:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.731+0000 7f2f2b1e3740 10 client.16797 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.731+0000 7f2f2b1e3740 10 client.16797 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:02:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.731+0000 7f2f2b1e3740 20 client.16797 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.731+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/4130221283 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563cc58df660 0x563cc58e1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.731+0000 7f2f2b1e3740 4 client.16797 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:02:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.731+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/4130221283 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563cc57415f0 con 0x563cc58df660 2024-06-04T05:02:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.731+0000 7f2f25a63640 1 --2- 172.21.15.70:0/4130221283 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563cc58df660 0x563cc58e1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.732+0000 7f2f25a63640 1 --2- 172.21.15.70:0/4130221283 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563cc58df660 0x563cc58e1a80 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3545604ae654f09f server_cookie=444a318e34cf49b1 in_seq=0 out_seq=0 2024-06-04T05:02:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.732+0000 7f2f09ffb640 10 client.16797 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.750+0000 7f2f09ffb640 1 -- 172.21.15.70:0/4130221283 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x563cc57415f0 con 0x563cc58df660 2024-06-04T05:02:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.750+0000 7f2f09ffb640 10 client.16797 handle_command_reply: tid=0 2024-06-04T05:02:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.750+0000 7f2f2b1e3740 1 client.16797 shutdown 2024-06-04T05:02:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.750+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/4130221283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc585a140 msgr2=0x563cc5877520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.750+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/4130221283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc585a140 0x563cc5877520 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c0126a0 tx=0x7f2f1c012b80 comp rx=0 tx=0).stop 2024-06-04T05:02:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.750+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/4130221283 shutdown_connections 2024-06-04T05:02:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.750+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/4130221283 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563cc58df660 0x563cc58e1a80 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.750+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/4130221283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc5864a00 0x563cc5877a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.750+0000 7f2f09ffb640 0 client.16797 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.750+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/4130221283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc585a140 0x563cc5877520 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.750+0000 7f2f2b1e3740 1 --2- 172.21.15.70:0/4130221283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc5859820 0x563cc587a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.750+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/4130221283 >> 172.21.15.70:0/4130221283 conn(0x563cc5855d30 msgr2=0x563cc5854fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.750+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/4130221283 shutdown_connections 2024-06-04T05:02:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.750+0000 7f2f2b1e3740 1 -- 172.21.15.70:0/4130221283 wait complete. 2024-06-04T05:02:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.750+0000 7f2f2b1e3740 20 client.16797 trim_cache size 0 max 16384 2024-06-04T05:02:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:02:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:02:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:02:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.751+0000 7f2f284c8640 1 -- 172.21.15.70:0/671309682 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2f0c0b2a10 msgr2=0x7f2f0c0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.751+0000 7f2f284c8640 1 --2- 172.21.15.70:0/671309682 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2f0c0b2a10 0x7f2f0c0b4ed0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2f180a9db0 tx=0x7f2f1806f7b0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.751+0000 7f2f284c8640 1 -- 172.21.15.70:0/671309682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f2010d060 msgr2=0x7f2f2019d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.751+0000 7f2f284c8640 1 --2- 172.21.15.70:0/671309682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f2010d060 0x7f2f2019d9b0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2f1800eb10 tx=0x7f2f1800c2d0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.751+0000 7f2f284c8640 1 -- 172.21.15.70:0/671309682 shutdown_connections 2024-06-04T05:02:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.751+0000 7f2f284c8640 1 --2- 172.21.15.70:0/671309682 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2f0c0b2a10 0x7f2f0c0b4ed0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.751+0000 7f2f284c8640 1 --2- 172.21.15.70:0/671309682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f20104640 0x7f2f20199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.751+0000 7f2f284c8640 1 --2- 172.21.15.70:0/671309682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f2010d060 0x7f2f2019d9b0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.751+0000 7f2f284c8640 1 --2- 172.21.15.70:0/671309682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f20103d20 0x7f2f201990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.751+0000 7f2f284c8640 1 -- 172.21.15.70:0/671309682 >> 172.21.15.70:0/671309682 conn(0x7f2f20078ad0 msgr2=0x7f2f200feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.751+0000 7f2f284c8640 1 -- 172.21.15.70:0/671309682 shutdown_connections 2024-06-04T05:02:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:12.751+0000 7f2f284c8640 1 -- 172.21.15.70:0/671309682 wait complete. 2024-06-04T05:02:13.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.426+0000 7f81ad5a5700 1 -- 172.21.15.70:0/698974367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a8101b30 msgr2=0x7f81a8101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:13.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.426+0000 7f81ad5a5700 1 --2- 172.21.15.70:0/698974367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a8101b30 0x7f81a8101fb0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f819c0099c0 tx=0x7f819c009cd0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.429 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:13 smithi070 ceph-mon[32631]: pgmap v154: 97 pgs: 97 active+clean; 55 MiB data, 694 MiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 1.9 MiB/s wr, 218 op/s 2024-06-04T05:02:13.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.426+0000 7f81ad5a5700 1 -- 172.21.15.70:0/698974367 shutdown_connections 2024-06-04T05:02:13.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.426+0000 7f81ad5a5700 1 --2- 172.21.15.70:0/698974367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81a81024f0 0x7f81a81103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.426+0000 7f81ad5a5700 1 --2- 172.21.15.70:0/698974367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81a8109520 0x7f81a8109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.426+0000 7f81ad5a5700 1 --2- 172.21.15.70:0/698974367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a8101b30 0x7f81a8101fb0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.426+0000 7f81ad5a5700 1 -- 172.21.15.70:0/698974367 >> 172.21.15.70:0/698974367 conn(0x7f81a8078ab0 msgr2=0x7f81a80ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:13.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.426+0000 7f81ad5a5700 1 -- 172.21.15.70:0/698974367 shutdown_connections 2024-06-04T05:02:13.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.426+0000 7f81ad5a5700 1 -- 172.21.15.70:0/698974367 wait complete. 2024-06-04T05:02:13.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.426+0000 7f81ad5a5700 1 Processor -- start 2024-06-04T05:02:13.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81ad5a5700 1 -- start start 2024-06-04T05:02:13.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81ad5a5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81a8101b30 0x7f81a819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81ad5a5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81a81024f0 0x7f81a819fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81ad5a5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a8109520 0x7f81a8199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81ad5a5700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f81a81135d0 con 0x7f81a8109520 2024-06-04T05:02:13.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81ad5a5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f81a8113450 con 0x7f81a8101b30 2024-06-04T05:02:13.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81ad5a5700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f81a8113750 con 0x7f81a81024f0 2024-06-04T05:02:13.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81a6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81a8101b30 0x7f81a819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81a77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a8109520 0x7f81a8199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81a77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a8109520 0x7f81a8199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49690/0 (socket says 172.21.15.70:49690) 2024-06-04T05:02:13.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81a77fe700 1 -- 172.21.15.70:0/10904483 learned_addr learned my addr 172.21.15.70:0/10904483 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:13.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81a6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81a8101b30 0x7f81a819f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:36242/0 (socket says 172.21.15.70:36242) 2024-06-04T05:02:13.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81a67fc700 1 --2- 172.21.15.70:0/10904483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81a81024f0 0x7f81a819fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81a77fe700 1 -- 172.21.15.70:0/10904483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81a81024f0 msgr2=0x7f81a819fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:13.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81a77fe700 1 --2- 172.21.15.70:0/10904483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81a81024f0 0x7f81a819fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81a77fe700 1 -- 172.21.15.70:0/10904483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81a8101b30 msgr2=0x7f81a819f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:13.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81a77fe700 1 --2- 172.21.15.70:0/10904483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81a8101b30 0x7f81a819f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81a77fe700 1 -- 172.21.15.70:0/10904483 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a819a090 con 0x7f81a8109520 2024-06-04T05:02:13.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.427+0000 7f81a67fc700 1 --2- 172.21.15.70:0/10904483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81a81024f0 0x7f81a819fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:02:13.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.428+0000 7f81a77fe700 1 --2- 172.21.15.70:0/10904483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a8109520 0x7f81a8199930 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f81a000ef30 tx=0x7f81a000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:13.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.428+0000 7f818ffff700 1 -- 172.21.15.70:0/10904483 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a00092e0 con 0x7f81a8109520 2024-06-04T05:02:13.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.428+0000 7f818ffff700 1 -- 172.21.15.70:0/10904483 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f81a0004bb0 con 0x7f81a8109520 2024-06-04T05:02:13.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.428+0000 7f81ad5a5700 1 -- 172.21.15.70:0/10904483 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81a819a320 con 0x7f81a8109520 2024-06-04T05:02:13.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.428+0000 7f818ffff700 1 -- 172.21.15.70:0/10904483 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a0004020 con 0x7f81a8109520 2024-06-04T05:02:13.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.428+0000 7f81ad5a5700 1 -- 172.21.15.70:0/10904483 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81a8102c40 con 0x7f81a8109520 2024-06-04T05:02:13.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.431+0000 7f818ffff700 1 -- 172.21.15.70:0/10904483 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f81a001b030 con 0x7f81a8109520 2024-06-04T05:02:13.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.432+0000 7f818ffff700 1 --2- 172.21.15.70:0/10904483 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81900b2a70 0x7f81900b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.432+0000 7f81a6ffd700 1 --2- 172.21.15.70:0/10904483 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81900b2a70 0x7f81900b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.432+0000 7f818ffff700 1 -- 172.21.15.70:0/10904483 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f81a00e2120 con 0x7f81a8109520 2024-06-04T05:02:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.432+0000 7f81a6ffd700 1 --2- 172.21.15.70:0/10904483 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81900b2a70 0x7f81900b4f30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f81a819aa70 tx=0x7f8198009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.432+0000 7f81b7e3ab80 1 Processor -- start 2024-06-04T05:02:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81b7e3ab80 1 -- start start 2024-06-04T05:02:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81b7e3ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 0x55ad96627c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81b7e3ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad96628210 0x55ad9662b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81b7e3ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad9662bf60 0x55ad9662c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81b7e3ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ad96586c90 con 0x55ad9662bf60 2024-06-04T05:02:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81b7e3ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ad96576830 con 0x55ad96628210 2024-06-04T05:02:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81b7e3ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ad9657a870 con 0x55ad96627860 2024-06-04T05:02:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81a6ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 0x55ad96627c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81a6ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 0x55ad96627c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:44514/0 (socket says 172.21.15.70:44514) 2024-06-04T05:02:13.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81a6ffd700 1 -- 172.21.15.70:0/70213693 learned_addr learned my addr 172.21.15.70:0/70213693 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:13.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81a77fe700 1 --2- 172.21.15.70:0/70213693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad9662bf60 0x55ad9662c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81a67fc700 1 --2- 172.21.15.70:0/70213693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad96628210 0x55ad9662b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81a6ffd700 1 -- 172.21.15.70:0/70213693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad96628210 msgr2=0x55ad9662b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:13.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81a6ffd700 1 --2- 172.21.15.70:0/70213693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad96628210 0x55ad9662b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81a6ffd700 1 -- 172.21.15.70:0/70213693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad9662bf60 msgr2=0x55ad9662c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:13.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81a6ffd700 1 --2- 172.21.15.70:0/70213693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad9662bf60 0x55ad9662c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81a6ffd700 1 -- 172.21.15.70:0/70213693 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad964c68d0 con 0x55ad96627860 2024-06-04T05:02:13.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.433+0000 7f81a6ffd700 1 --2- 172.21.15.70:0/70213693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 0x55ad96627c40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8198011d60 tx=0x7f8198014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63e9e5da923cef64 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:13.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.434+0000 7f818d7fa700 1 -- 172.21.15.70:0/70213693 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8198011d90 con 0x55ad96627860 2024-06-04T05:02:13.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.434+0000 7f818d7fa700 1 -- 172.21.15.70:0/70213693 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8198004030 con 0x55ad96627860 2024-06-04T05:02:13.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.434+0000 7f818d7fa700 1 -- 172.21.15.70:0/70213693 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8198003be0 con 0x55ad96627860 2024-06-04T05:02:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.434+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/70213693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 msgr2=0x55ad96627c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.434+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/70213693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 0x55ad96627c40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8198011d60 tx=0x7f8198014000 comp rx=0 tx=0).stop 2024-06-04T05:02:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/70213693 shutdown_connections 2024-06-04T05:02:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/70213693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad9662bf60 0x55ad9662c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/70213693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 0x55ad96627c40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/70213693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad96628210 0x55ad9662b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/70213693 >> 172.21.15.70:0/70213693 conn(0x55ad96555080 msgr2=0x55ad96621e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/70213693 shutdown_connections 2024-06-04T05:02:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/70213693 wait complete. 2024-06-04T05:02:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81b7e3ab80 1 -- start start 2024-06-04T05:02:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81b7e3ab80 1 Processor -- start 2024-06-04T05:02:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81b7e3ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96627860 0x55ad96633440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81b7e3ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96628210 0x55ad96630740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81b7e3ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad9662bf60 0x55ad96630c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81b7e3ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ad9647cfa0 con 0x55ad96627860 2024-06-04T05:02:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81b7e3ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ad965187c0 con 0x55ad9662bf60 2024-06-04T05:02:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81b7e3ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ad96634120 con 0x55ad96628210 2024-06-04T05:02:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81b7e3ab80 10 client.0 _ll_register_callbacks cb 0x55ad96511920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81a6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96627860 0x55ad96633440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81a6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96627860 0x55ad96633440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49706/0 (socket says 172.21.15.70:49706) 2024-06-04T05:02:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.435+0000 7f81a6ffd700 1 -- 172.21.15.70:0/738179339 learned_addr learned my addr 172.21.15.70:0/738179339 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.436+0000 7f81a67fc700 1 --2- 172.21.15.70:0/738179339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96628210 0x55ad96630740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.436+0000 7f81a77fe700 1 --2- 172.21.15.70:0/738179339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad9662bf60 0x55ad96630c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.436+0000 7f81a6ffd700 1 -- 172.21.15.70:0/738179339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96628210 msgr2=0x55ad96630740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.436+0000 7f81a6ffd700 1 --2- 172.21.15.70:0/738179339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96628210 0x55ad96630740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.436+0000 7f81a6ffd700 1 -- 172.21.15.70:0/738179339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad9662bf60 msgr2=0x55ad96630c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.436+0000 7f81a6ffd700 1 --2- 172.21.15.70:0/738179339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad9662bf60 0x55ad96630c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.436+0000 7f81a6ffd700 1 -- 172.21.15.70:0/738179339 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad96515690 con 0x55ad96627860 2024-06-04T05:02:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.436+0000 7f81a67fc700 1 --2- 172.21.15.70:0/738179339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96628210 0x55ad96630740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.436+0000 7f81a77fe700 1 --2- 172.21.15.70:0/738179339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad9662bf60 0x55ad96630c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:02:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.436+0000 7f81a6ffd700 1 --2- 172.21.15.70:0/738179339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96627860 0x55ad96633440 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f8198009aa0 tx=0x7f81980054a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61884aaaef9c8112 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.436+0000 7f818aff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:02:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.436+0000 7f818aff5700 1 -- 172.21.15.70:0/738179339 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8198003a20 con 0x55ad96627860 2024-06-04T05:02:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.436+0000 7f818aff5700 1 -- 172.21.15.70:0/738179339 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f8198003bc0 con 0x55ad96627860 2024-06-04T05:02:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.436+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/738179339 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8198004030 con 0x55ad96627860 2024-06-04T05:02:13.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.436+0000 7f818aff5700 1 -- 172.21.15.70:0/738179339 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8198003dc0 con 0x55ad96627860 2024-06-04T05:02:13.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.436+0000 7f81b7e3ab80 10 client.16809 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:13.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.436+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/738179339 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ad9663e8b0 con 0x55ad96627860 2024-06-04T05:02:13.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.437+0000 7f818aff5700 1 -- 172.21.15.70:0/738179339 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8198004030 con 0x55ad96627860 2024-06-04T05:02:13.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.437+0000 7f818aff5700 1 -- 172.21.15.70:0/738179339 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f819802e070 con 0x55ad96627860 2024-06-04T05:02:13.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.437+0000 7f81b7e3ab80 10 client.16809 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:13.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.437+0000 7f81b7e3ab80 10 client.16809 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:02:13.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.437+0000 7f81b7e3ab80 20 client.16809 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:13.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.437+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/738179339 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ad96640d80 0x55ad96643170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.437+0000 7f81b7e3ab80 4 client.16809 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:02:13.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.437+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/738179339 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad965f39e0 con 0x55ad96640d80 2024-06-04T05:02:13.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.437+0000 7f81a67fc700 1 --2- 172.21.15.70:0/738179339 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ad96640d80 0x55ad96643170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.438+0000 7f81a67fc700 1 --2- 172.21.15.70:0/738179339 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ad96640d80 0x55ad96643170 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c739c528a69a140d server_cookie=5dba6683fa5894e5 in_seq=0 out_seq=0 2024-06-04T05:02:13.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.438+0000 7f818aff5700 10 client.16809 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:13.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.439+0000 7f818aff5700 1 -- 172.21.15.70:0/738179339 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ad965f39e0 con 0x55ad96640d80 2024-06-04T05:02:13.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.439+0000 7f818aff5700 10 client.16809 handle_command_reply: tid=0 2024-06-04T05:02:13.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.439+0000 7f81b7e3ab80 1 client.16809 shutdown 2024-06-04T05:02:13.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.439+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/738179339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96627860 msgr2=0x55ad96633440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:13.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.439+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/738179339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96627860 0x55ad96633440 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f8198009aa0 tx=0x7f81980054a0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.439+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/738179339 shutdown_connections 2024-06-04T05:02:13.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.439+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/738179339 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ad96640d80 0x55ad96643170 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.439+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/738179339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad9662bf60 0x55ad96630c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.439+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/738179339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96627860 0x55ad96633440 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.439+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/738179339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96628210 0x55ad96630740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.439+0000 7f818aff5700 0 client.16809 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:13.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.439+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/738179339 >> 172.21.15.70:0/738179339 conn(0x55ad96555080 msgr2=0x55ad96621510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:13.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.439+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/738179339 shutdown_connections 2024-06-04T05:02:13.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.439+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/738179339 wait complete. 2024-06-04T05:02:13.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.439+0000 7f81b7e3ab80 20 client.16809 trim_cache size 0 max 16384 2024-06-04T05:02:13.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.465+0000 7f81b7e3ab80 1 Processor -- start 2024-06-04T05:02:13.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.465+0000 7f81b7e3ab80 1 -- start start 2024-06-04T05:02:13.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.465+0000 7f81b7e3ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 0x55ad96648710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.465+0000 7f81b7e3ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96628210 0x55ad96648c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.465+0000 7f81b7e3ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad9662bf60 0x55ad96621c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.465+0000 7f81b7e3ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f81a81b0090 con 0x55ad96628210 2024-06-04T05:02:13.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.465+0000 7f81b7e3ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f81a81b1bd0 con 0x55ad9662bf60 2024-06-04T05:02:13.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.465+0000 7f81b7e3ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f81a81b0760 con 0x55ad96627860 2024-06-04T05:02:13.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.465+0000 7f81a67fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96628210 0x55ad96648c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.466+0000 7f81a67fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96628210 0x55ad96648c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49722/0 (socket says 172.21.15.70:49722) 2024-06-04T05:02:13.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.466+0000 7f81a67fc700 1 -- 172.21.15.70:0/2133684794 learned_addr learned my addr 172.21.15.70:0/2133684794 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:13.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.466+0000 7f81a6ffd700 1 --2- 172.21.15.70:0/2133684794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 0x55ad96648710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.466+0000 7f81a77fe700 1 --2- 172.21.15.70:0/2133684794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad9662bf60 0x55ad96621c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.466+0000 7f81a67fc700 1 -- 172.21.15.70:0/2133684794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 msgr2=0x55ad96648710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:13.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.466+0000 7f81a67fc700 1 --2- 172.21.15.70:0/2133684794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 0x55ad96648710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.466+0000 7f81a67fc700 1 -- 172.21.15.70:0/2133684794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad9662bf60 msgr2=0x55ad96621c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:13.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.466+0000 7f81a67fc700 1 --2- 172.21.15.70:0/2133684794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad9662bf60 0x55ad96621c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.466+0000 7f81a67fc700 1 -- 172.21.15.70:0/2133684794 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad96631680 con 0x55ad96628210 2024-06-04T05:02:13.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.466+0000 7f81a67fc700 1 --2- 172.21.15.70:0/2133684794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96628210 0x55ad96648c50 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f819c0099c0 tx=0x7f819c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65330a05dee36623 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:13.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.466+0000 7f818cff9700 1 -- 172.21.15.70:0/2133684794 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f819c030070 con 0x55ad96628210 2024-06-04T05:02:13.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.466+0000 7f818cff9700 1 -- 172.21.15.70:0/2133684794 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f819c0054e0 con 0x55ad96628210 2024-06-04T05:02:13.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.466+0000 7f818cff9700 1 -- 172.21.15.70:0/2133684794 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f819c00bc50 con 0x55ad96628210 2024-06-04T05:02:13.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.467+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/2133684794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96628210 msgr2=0x55ad96648c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:13.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.467+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/2133684794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96628210 0x55ad96648c50 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f819c0099c0 tx=0x7f819c009310 comp rx=0 tx=0).stop 2024-06-04T05:02:13.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.467+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/2133684794 shutdown_connections 2024-06-04T05:02:13.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.467+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/2133684794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad9662bf60 0x55ad96621c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.467+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/2133684794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 0x55ad96648710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.467+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/2133684794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96628210 0x55ad96648c50 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.467+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/2133684794 >> 172.21.15.70:0/2133684794 conn(0x55ad96555080 msgr2=0x55ad9662b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:13.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.467+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/2133684794 shutdown_connections 2024-06-04T05:02:13.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.467+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/2133684794 wait complete. 2024-06-04T05:02:13.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.467+0000 7f81b7e3ab80 1 -- start start 2024-06-04T05:02:13.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.467+0000 7f81b7e3ab80 1 Processor -- start 2024-06-04T05:02:13.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81b7e3ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 0x55ad96639920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81b7e3ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96628210 0x55ad96636cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81b7e3ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad9662bf60 0x55ad96637200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81b7e3ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f81a81b1500 con 0x55ad96628210 2024-06-04T05:02:13.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81b7e3ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f81a81b0e30 con 0x55ad9662bf60 2024-06-04T05:02:13.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81b7e3ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f81a81ae030 con 0x55ad96627860 2024-06-04T05:02:13.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81b7e3ab80 10 client.0 _ll_register_callbacks cb 0x55ad96511920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:13.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81a67fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96628210 0x55ad96636cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81a6ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 0x55ad96639920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81a67fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96628210 0x55ad96636cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49728/0 (socket says 172.21.15.70:49728) 2024-06-04T05:02:13.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81a77fe700 1 --2- 172.21.15.70:0/4077928142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad9662bf60 0x55ad96637200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81a67fc700 1 -- 172.21.15.70:0/4077928142 learned_addr learned my addr 172.21.15.70:0/4077928142 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:13.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81a6ffd700 1 -- 172.21.15.70:0/4077928142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad9662bf60 msgr2=0x55ad96637200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:13.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81a6ffd700 1 --2- 172.21.15.70:0/4077928142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad9662bf60 0x55ad96637200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81a6ffd700 1 -- 172.21.15.70:0/4077928142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96628210 msgr2=0x55ad96636cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:13.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81a6ffd700 1 --2- 172.21.15.70:0/4077928142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96628210 0x55ad96636cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81a6ffd700 1 -- 172.21.15.70:0/4077928142 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad966378b0 con 0x55ad96627860 2024-06-04T05:02:13.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81a6ffd700 1 --2- 172.21.15.70:0/4077928142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 0x55ad96639920 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f819801e5e0 tx=0x7f8198003cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=428872fe8e4ac3e9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:13.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f818aff5700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:02:13.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f818aff5700 1 -- 172.21.15.70:0/4077928142 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f819801ea70 con 0x55ad96627860 2024-06-04T05:02:13.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.468+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/4077928142 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f819c0054e0 con 0x55ad96627860 2024-06-04T05:02:13.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.469+0000 7f818aff5700 1 -- 172.21.15.70:0/4077928142 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f819801ec10 con 0x55ad96627860 2024-06-04T05:02:13.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.469+0000 7f818aff5700 1 -- 172.21.15.70:0/4077928142 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f819801ee10 con 0x55ad96627860 2024-06-04T05:02:13.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.469+0000 7f81b7e3ab80 10 client.26170 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:13.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.469+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/4077928142 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ad966ab4c0 con 0x55ad96627860 2024-06-04T05:02:13.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.469+0000 7f818aff5700 1 -- 172.21.15.70:0/4077928142 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f819c0054e0 con 0x55ad96627860 2024-06-04T05:02:13.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.469+0000 7f818aff5700 1 -- 172.21.15.70:0/4077928142 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f819802e040 con 0x55ad96627860 2024-06-04T05:02:13.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.469+0000 7f81b7e3ab80 10 client.26170 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:13.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.469+0000 7f81b7e3ab80 10 client.26170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:02:13.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.469+0000 7f81b7e3ab80 20 client.26170 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:13.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.469+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/4077928142 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ad966abb00 0x55ad966adef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:13.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.469+0000 7f81b7e3ab80 4 client.26170 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:02:13.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.469+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/4077928142 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ad96643500 con 0x55ad966abb00 2024-06-04T05:02:13.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.469+0000 7f81a67fc700 1 --2- 172.21.15.70:0/4077928142 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ad966abb00 0x55ad966adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:13.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.470+0000 7f81a67fc700 1 --2- 172.21.15.70:0/4077928142 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ad966abb00 0x55ad966adef0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8c33bc9d00a711a server_cookie=407f3b88c03f57b6 in_seq=0 out_seq=0 2024-06-04T05:02:13.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.470+0000 7f818aff5700 10 client.26170 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:13.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.470+0000 7f818aff5700 1 -- 172.21.15.70:0/4077928142 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ad96643500 con 0x55ad966abb00 2024-06-04T05:02:13.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.470+0000 7f818aff5700 10 client.26170 handle_command_reply: tid=0 2024-06-04T05:02:13.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.470+0000 7f81b7e3ab80 1 client.26170 shutdown 2024-06-04T05:02:13.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.470+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/4077928142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 msgr2=0x55ad96639920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:13.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.470+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/4077928142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 0x55ad96639920 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f819801e5e0 tx=0x7f8198003cb0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/4077928142 shutdown_connections 2024-06-04T05:02:13.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/4077928142 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ad966abb00 0x55ad966adef0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/4077928142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad9662bf60 0x55ad96637200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/4077928142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad96627860 0x55ad96639920 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f818aff5700 0 client.26170 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:13.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81b7e3ab80 1 --2- 172.21.15.70:0/4077928142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad96628210 0x55ad96636cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/4077928142 >> 172.21.15.70:0/4077928142 conn(0x55ad96555080 msgr2=0x55ad9662ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:13.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/4077928142 shutdown_connections 2024-06-04T05:02:13.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81b7e3ab80 1 -- 172.21.15.70:0/4077928142 wait complete. 2024-06-04T05:02:13.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81b7e3ab80 20 client.26170 trim_cache size 0 max 16384 2024-06-04T05:02:13.478 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:02:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:02:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:02:13.478 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:02:13.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81ad5a5700 1 -- 172.21.15.70:0/10904483 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81900b2a70 msgr2=0x7f81900b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:13.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81ad5a5700 1 --2- 172.21.15.70:0/10904483 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81900b2a70 0x7f81900b4f30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f81a819aa70 tx=0x7f8198009260 comp rx=0 tx=0).stop 2024-06-04T05:02:13.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81ad5a5700 1 -- 172.21.15.70:0/10904483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a8109520 msgr2=0x7f81a8199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:13.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81ad5a5700 1 --2- 172.21.15.70:0/10904483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a8109520 0x7f81a8199930 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f81a000ef30 tx=0x7f81a000c3a0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81ad5a5700 1 -- 172.21.15.70:0/10904483 shutdown_connections 2024-06-04T05:02:13.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81ad5a5700 1 --2- 172.21.15.70:0/10904483 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81900b2a70 0x7f81900b4f30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81ad5a5700 1 --2- 172.21.15.70:0/10904483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a8109520 0x7f81a8199930 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81ad5a5700 1 --2- 172.21.15.70:0/10904483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81a8101b30 0x7f81a819f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81ad5a5700 1 --2- 172.21.15.70:0/10904483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81a81024f0 0x7f81a819fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:13.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.471+0000 7f81ad5a5700 1 -- 172.21.15.70:0/10904483 >> 172.21.15.70:0/10904483 conn(0x7f81a8078ab0 msgr2=0x7f81a81003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:13.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.472+0000 7f81ad5a5700 1 -- 172.21.15.70:0/10904483 shutdown_connections 2024-06-04T05:02:13.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:13.472+0000 7f81ad5a5700 1 -- 172.21.15.70:0/10904483 wait complete. 2024-06-04T05:02:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:13 smithi098 ceph-mon[35569]: pgmap v154: 97 pgs: 97 active+clean; 55 MiB data, 694 MiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 1.9 MiB/s wr, 218 op/s 2024-06-04T05:02:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:13 smithi112 ceph-mon[36453]: pgmap v154: 97 pgs: 97 active+clean; 55 MiB data, 694 MiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 1.9 MiB/s wr, 218 op/s 2024-06-04T05:02:13.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 69873 1.96 MB/sec execute 66 sec latency 15.332 ms 2024-06-04T05:02:14.291 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:02:14.292 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:02:14.597 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:02:14.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 70697 2.04 MB/sec execute 67 sec latency 15.594 ms 2024-06-04T05:02:15.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:15 smithi070 ceph-mon[32631]: pgmap v155: 97 pgs: 97 active+clean; 55 MiB data, 694 MiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 1.3 MiB/s wr, 146 op/s 2024-06-04T05:02:15.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:15 smithi098 ceph-mon[35569]: pgmap v155: 97 pgs: 97 active+clean; 55 MiB data, 694 MiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 1.3 MiB/s wr, 146 op/s 2024-06-04T05:02:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:15 smithi112 ceph-mon[36453]: pgmap v155: 97 pgs: 97 active+clean; 55 MiB data, 694 MiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 1.3 MiB/s wr, 146 op/s 2024-06-04T05:02:15.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 71113 2.06 MB/sec execute 68 sec latency 15.398 ms 2024-06-04T05:02:16.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.497+0000 7f68fa73a700 1 -- 172.21.15.70:0/4119665719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68f4108510 msgr2=0x7f68f41088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:16.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.497+0000 7f68fa73a700 1 --2- 172.21.15.70:0/4119665719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68f4108510 0x7f68f41088f0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f68e4009990 tx=0x7f68e4009ca0 comp rx=0 tx=0).stop 2024-06-04T05:02:16.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.497+0000 7f68fa73a700 1 -- 172.21.15.70:0/4119665719 shutdown_connections 2024-06-04T05:02:16.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.497+0000 7f68fa73a700 1 --2- 172.21.15.70:0/4119665719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f4102df0 0x7f68f410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:16.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.497+0000 7f68fa73a700 1 --2- 172.21.15.70:0/4119665719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68f4108510 0x7f68f41088f0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:16.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.497+0000 7f68fa73a700 1 --2- 172.21.15.70:0/4119665719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f4102430 0x7f68f41028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:16.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.497+0000 7f68fa73a700 1 -- 172.21.15.70:0/4119665719 >> 172.21.15.70:0/4119665719 conn(0x7f68f40793b0 msgr2=0x7f68f40797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:16.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.497+0000 7f68fa73a700 1 -- 172.21.15.70:0/4119665719 shutdown_connections 2024-06-04T05:02:16.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.497+0000 7f68fa73a700 1 -- 172.21.15.70:0/4119665719 wait complete. 2024-06-04T05:02:16.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.498+0000 7f68fa73a700 1 Processor -- start 2024-06-04T05:02:16.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.498+0000 7f68fa73a700 1 -- start start 2024-06-04T05:02:16.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68fa73a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f4102430 0x7f68f419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:16.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68fa73a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f4102df0 0x7f68f419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:16.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68fa73a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68f4108510 0x7f68f41988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:16.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68fa73a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f68f41124e0 con 0x7f68f4108510 2024-06-04T05:02:16.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68fa73a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68f4112360 con 0x7f68f4102df0 2024-06-04T05:02:16.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68fa73a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f68f4112660 con 0x7f68f4102430 2024-06-04T05:02:16.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68f8cd7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68f4108510 0x7f68f41988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:16.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68f3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f4102430 0x7f68f419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:16.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68f3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f4102430 0x7f68f419e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:49184/0 (socket says 172.21.15.70:49184) 2024-06-04T05:02:16.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68f8cd7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68f4108510 0x7f68f41988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38982/0 (socket says 172.21.15.70:38982) 2024-06-04T05:02:16.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68f3fff700 1 -- 172.21.15.70:0/3822608870 learned_addr learned my addr 172.21.15.70:0/3822608870 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:16.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68f37fe700 1 --2- 172.21.15.70:0/3822608870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f4102df0 0x7f68f419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:16.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68f3fff700 1 -- 172.21.15.70:0/3822608870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f4102df0 msgr2=0x7f68f419ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:16.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68f3fff700 1 --2- 172.21.15.70:0/3822608870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f4102df0 0x7f68f419ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:16.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68f3fff700 1 -- 172.21.15.70:0/3822608870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68f4108510 msgr2=0x7f68f41988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:16.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68f3fff700 1 --2- 172.21.15.70:0/3822608870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68f4108510 0x7f68f41988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:16.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68f3fff700 1 -- 172.21.15.70:0/3822608870 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f4199020 con 0x7f68f4102430 2024-06-04T05:02:16.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68f37fe700 1 --2- 172.21.15.70:0/3822608870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f4102df0 0x7f68f419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:16.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68f8cd7700 1 --2- 172.21.15.70:0/3822608870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68f4108510 0x7f68f41988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:16.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.499+0000 7f68f3fff700 1 --2- 172.21.15.70:0/3822608870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f4102430 0x7f68f419e700 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f68e400b650 tx=0x7f68e40047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:16.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.500+0000 7f68f17fa700 1 -- 172.21.15.70:0/3822608870 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e4027070 con 0x7f68f4102430 2024-06-04T05:02:16.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.500+0000 7f68f17fa700 1 -- 172.21.15.70:0/3822608870 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68e4022470 con 0x7f68f4102430 2024-06-04T05:02:16.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.500+0000 7f68fa73a700 1 -- 172.21.15.70:0/3822608870 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f41992b0 con 0x7f68f4102430 2024-06-04T05:02:16.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.500+0000 7f68f17fa700 1 -- 172.21.15.70:0/3822608870 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e4017650 con 0x7f68f4102430 2024-06-04T05:02:16.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.500+0000 7f68fa73a700 1 -- 172.21.15.70:0/3822608870 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f41a55d0 con 0x7f68f4102430 2024-06-04T05:02:16.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.501+0000 7f68fa73a700 1 -- 172.21.15.70:0/3822608870 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f410a9a0 con 0x7f68f4102430 2024-06-04T05:02:16.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.503+0000 7f68f17fa700 1 -- 172.21.15.70:0/3822608870 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f68e402a070 con 0x7f68f4102430 2024-06-04T05:02:16.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.503+0000 7f68f17fa700 1 --2- 172.21.15.70:0/3822608870 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68dc0b2a70 0x7f68dc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:16.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.503+0000 7f68f37fe700 1 --2- 172.21.15.70:0/3822608870 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68dc0b2a70 0x7f68dc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:16.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.503+0000 7f68f17fa700 1 -- 172.21.15.70:0/3822608870 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f68e40e3b60 con 0x7f68f4102430 2024-06-04T05:02:16.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.504+0000 7f68f37fe700 1 --2- 172.21.15.70:0/3822608870 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68dc0b2a70 0x7f68dc0b4f30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f68f4107460 tx=0x7f68e8009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:16.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.507+0000 7f68f17fa700 1 -- 172.21.15.70:0/3822608870 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f68e4031070 con 0x7f68f4102430 2024-06-04T05:02:16.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 71417 2.04 MB/sec execute 69 sec latency 11.751 ms 2024-06-04T05:02:16.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.804+0000 7f68fa73a700 1 -- 172.21.15.70:0/3822608870 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68f406a230 con 0x7f68f4102430 2024-06-04T05:02:16.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.805+0000 7f68f17fa700 1 -- 172.21.15.70:0/3822608870 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11929 (secure 0 0 0) 0x7f68f406a230 con 0x7f68f4102430 2024-06-04T05:02:16.808 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:02:16.808 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:01:52.807483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:02:16.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.808+0000 7f68fa73a700 1 -- 172.21.15.70:0/3822608870 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68dc0b2a70 msgr2=0x7f68dc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:16.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.808+0000 7f68fa73a700 1 --2- 172.21.15.70:0/3822608870 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68dc0b2a70 0x7f68dc0b4f30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f68f4107460 tx=0x7f68e8009260 comp rx=0 tx=0).stop 2024-06-04T05:02:16.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.808+0000 7f68fa73a700 1 -- 172.21.15.70:0/3822608870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f4102430 msgr2=0x7f68f419e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:16.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.808+0000 7f68fa73a700 1 --2- 172.21.15.70:0/3822608870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f4102430 0x7f68f419e700 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f68e400b650 tx=0x7f68e40047b0 comp rx=0 tx=0).stop 2024-06-04T05:02:16.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.808+0000 7f68fa73a700 1 -- 172.21.15.70:0/3822608870 shutdown_connections 2024-06-04T05:02:16.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.808+0000 7f68fa73a700 1 --2- 172.21.15.70:0/3822608870 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68dc0b2a70 0x7f68dc0b4f30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:16.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.808+0000 7f68fa73a700 1 --2- 172.21.15.70:0/3822608870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68f4108510 0x7f68f41988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:16.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.808+0000 7f68fa73a700 1 --2- 172.21.15.70:0/3822608870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f4102430 0x7f68f419e700 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:16.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.808+0000 7f68fa73a700 1 --2- 172.21.15.70:0/3822608870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f4102df0 0x7f68f419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:16.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.808+0000 7f68fa73a700 1 -- 172.21.15.70:0/3822608870 >> 172.21.15.70:0/3822608870 conn(0x7f68f40793b0 msgr2=0x7f68f4101330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:16.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.808+0000 7f68fa73a700 1 -- 172.21.15.70:0/3822608870 shutdown_connections 2024-06-04T05:02:16.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:16.808+0000 7f68fa73a700 1 -- 172.21.15.70:0/3822608870 wait complete. 2024-06-04T05:02:16.811 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 32 2024-06-04T05:02:17.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:17 smithi098 ceph-mon[35569]: pgmap v156: 97 pgs: 97 active+clean; 57 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 2.0 MiB/s wr, 214 op/s 2024-06-04T05:02:17.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:17 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3822608870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:02:17.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:17 smithi070 ceph-mon[32631]: pgmap v156: 97 pgs: 97 active+clean; 57 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 2.0 MiB/s wr, 214 op/s 2024-06-04T05:02:17.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:17 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3822608870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:02:17.645 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:02:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:17 smithi112 ceph-mon[36453]: pgmap v156: 97 pgs: 97 active+clean; 57 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 2.0 MiB/s wr, 214 op/s 2024-06-04T05:02:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:17 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3822608870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:02:17.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 71836 2.02 MB/sec execute 70 sec latency 15.541 ms 2024-06-04T05:02:17.920 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:02:18.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 72418 2.06 MB/sec execute 71 sec latency 16.389 ms 2024-06-04T05:02:19.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:19 smithi070 ceph-mon[32631]: pgmap v157: 97 pgs: 97 active+clean; 57 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-06-04T05:02:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:19 smithi098 ceph-mon[35569]: pgmap v157: 97 pgs: 97 active+clean; 57 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-06-04T05:02:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:19 smithi112 ceph-mon[36453]: pgmap v157: 97 pgs: 97 active+clean; 57 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-06-04T05:02:19.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 73112 2.04 MB/sec execute 72 sec latency 17.000 ms 2024-06-04T05:02:19.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.740+0000 7f328072f700 1 -- 172.21.15.70:0/1268025652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3278103850 msgr2=0x7f3278103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:19.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.740+0000 7f328072f700 1 --2- 172.21.15.70:0/1268025652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3278103850 0x7f3278103c30 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f32740099c0 tx=0x7f3274009cd0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.741+0000 7f328072f700 1 -- 172.21.15.70:0/1268025652 shutdown_connections 2024-06-04T05:02:19.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.741+0000 7f328072f700 1 --2- 172.21.15.70:0/1268025652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3278108920 0x7f32781104a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.741+0000 7f328072f700 1 --2- 172.21.15.70:0/1268025652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3278103850 0x7f3278103c30 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.741+0000 7f328072f700 1 --2- 172.21.15.70:0/1268025652 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3278104200 0x7f32781081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.741+0000 7f328072f700 1 -- 172.21.15.70:0/1268025652 >> 172.21.15.70:0/1268025652 conn(0x7f327807a620 msgr2=0x7f327807aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:19.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.741+0000 7f328072f700 1 -- 172.21.15.70:0/1268025652 shutdown_connections 2024-06-04T05:02:19.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.741+0000 7f328072f700 1 -- 172.21.15.70:0/1268025652 wait complete. 2024-06-04T05:02:19.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.741+0000 7f328072f700 1 Processor -- start 2024-06-04T05:02:19.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.741+0000 7f328072f700 1 -- start start 2024-06-04T05:02:19.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f328072f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3278103850 0x7f3278075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f328072f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3278104200 0x7f3278075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f328072f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3278108920 0x7f32780762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f328072f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f32781135d0 con 0x7f3278108920 2024-06-04T05:02:19.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f328072f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3278113450 con 0x7f3278104200 2024-06-04T05:02:19.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f328072f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3278113750 con 0x7f3278103850 2024-06-04T05:02:19.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f327eccc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3278108920 0x7f32780762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f327e4cb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3278103850 0x7f3278075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f327dcca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3278104200 0x7f3278075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f327e4cb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3278103850 0x7f3278075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:49212/0 (socket says 172.21.15.70:49212) 2024-06-04T05:02:19.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f327eccc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3278108920 0x7f32780762c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39010/0 (socket says 172.21.15.70:39010) 2024-06-04T05:02:19.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f327e4cb700 1 -- 172.21.15.70:0/950893525 learned_addr learned my addr 172.21.15.70:0/950893525 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:19.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f327dcca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3278104200 0x7f3278075d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:43210/0 (socket says 172.21.15.70:43210) 2024-06-04T05:02:19.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f327e4cb700 1 -- 172.21.15.70:0/950893525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3278104200 msgr2=0x7f3278075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:19.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f327e4cb700 1 --2- 172.21.15.70:0/950893525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3278104200 0x7f3278075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f327e4cb700 1 -- 172.21.15.70:0/950893525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3278108920 msgr2=0x7f32780762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:19.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f327e4cb700 1 --2- 172.21.15.70:0/950893525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3278108920 0x7f32780762c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f327e4cb700 1 -- 172.21.15.70:0/950893525 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32781a7fd0 con 0x7f3278103850 2024-06-04T05:02:19.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.742+0000 7f327e4cb700 1 --2- 172.21.15.70:0/950893525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3278103850 0x7f3278075840 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3274006010 tx=0x7f32740045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:19.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.743+0000 7f326f7fe700 1 -- 172.21.15.70:0/950893525 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3274026070 con 0x7f3278103850 2024-06-04T05:02:19.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.743+0000 7f326f7fe700 1 -- 172.21.15.70:0/950893525 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f327400f460 con 0x7f3278103850 2024-06-04T05:02:19.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.743+0000 7f328072f700 1 -- 172.21.15.70:0/950893525 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32781a8260 con 0x7f3278103850 2024-06-04T05:02:19.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.743+0000 7f326f7fe700 1 -- 172.21.15.70:0/950893525 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3274009e60 con 0x7f3278103850 2024-06-04T05:02:19.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.743+0000 7f328072f700 1 -- 172.21.15.70:0/950893525 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32781a8710 con 0x7f3278103850 2024-06-04T05:02:19.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.746+0000 7f326f7fe700 1 -- 172.21.15.70:0/950893525 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3274029070 con 0x7f3278103850 2024-06-04T05:02:19.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.746+0000 7f326f7fe700 1 --2- 172.21.15.70:0/950893525 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f32640b2a70 0x7f32640b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.746+0000 7f328afc4b80 1 Processor -- start 2024-06-04T05:02:19.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.746+0000 7f327dcca700 1 --2- 172.21.15.70:0/950893525 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f32640b2a70 0x7f32640b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f326f7fe700 1 -- 172.21.15.70:0/950893525 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f32740e2d70 con 0x7f3278103850 2024-06-04T05:02:19.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f328afc4b80 1 -- start start 2024-06-04T05:02:19.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f328afc4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690de7860 0x555690de7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f328afc4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690de8210 0x555690deb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f328afc4b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690debf60 0x555690dec3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f328afc4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555690d46c90 con 0x555690de7860 2024-06-04T05:02:19.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f328afc4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555690d36830 con 0x555690de8210 2024-06-04T05:02:19.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f328afc4b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555690d3a870 con 0x555690debf60 2024-06-04T05:02:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f327e4cb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690de7860 0x555690de7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f327e4cb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690de7860 0x555690de7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39026/0 (socket says 172.21.15.70:39026) 2024-06-04T05:02:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f327e4cb700 1 -- 172.21.15.70:0/1375159425 learned_addr learned my addr 172.21.15.70:0/1375159425 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f327eccc700 1 --2- 172.21.15.70:0/1375159425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690debf60 0x555690dec3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f327dcca700 1 --2- 172.21.15.70:0/950893525 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f32640b2a70 0x7f32640b4f30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f32781031c0 tx=0x7f3268009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f327dcca700 1 --2- 172.21.15.70:0/1375159425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690de8210 0x555690deb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f327e4cb700 1 -- 172.21.15.70:0/1375159425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690debf60 msgr2=0x555690dec3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f327e4cb700 1 --2- 172.21.15.70:0/1375159425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690debf60 0x555690dec3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f327e4cb700 1 -- 172.21.15.70:0/1375159425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690de8210 msgr2=0x555690deb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f327e4cb700 1 --2- 172.21.15.70:0/1375159425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690de8210 0x555690deb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f327e4cb700 1 -- 172.21.15.70:0/1375159425 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555690c868d0 con 0x555690de7860 2024-06-04T05:02:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.747+0000 7f327dcca700 1 --2- 172.21.15.70:0/1375159425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690de8210 0x555690deb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:02:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.748+0000 7f327e4cb700 1 --2- 172.21.15.70:0/1375159425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690de7860 0x555690de7c40 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f3274009990 tx=0x7f32740a9a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82556bd762f5bf server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.748+0000 7f326cff9700 1 -- 172.21.15.70:0/1375159425 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32740aa7e0 con 0x555690de7860 2024-06-04T05:02:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.748+0000 7f326cff9700 1 -- 172.21.15.70:0/1375159425 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f32740aa0e0 con 0x555690de7860 2024-06-04T05:02:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.748+0000 7f326cff9700 1 -- 172.21.15.70:0/1375159425 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327400f910 con 0x555690de7860 2024-06-04T05:02:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.749+0000 7f328afc4b80 1 -- 172.21.15.70:0/1375159425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690de7860 msgr2=0x555690de7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.749+0000 7f328afc4b80 1 --2- 172.21.15.70:0/1375159425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690de7860 0x555690de7c40 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f3274009990 tx=0x7f32740a9a30 comp rx=0 tx=0).stop 2024-06-04T05:02:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.749+0000 7f328afc4b80 1 -- 172.21.15.70:0/1375159425 shutdown_connections 2024-06-04T05:02:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.749+0000 7f328afc4b80 1 --2- 172.21.15.70:0/1375159425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690debf60 0x555690dec3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.749+0000 7f328afc4b80 1 --2- 172.21.15.70:0/1375159425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690de7860 0x555690de7c40 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.749+0000 7f328afc4b80 1 --2- 172.21.15.70:0/1375159425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690de8210 0x555690deb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.749+0000 7f328afc4b80 1 -- 172.21.15.70:0/1375159425 >> 172.21.15.70:0/1375159425 conn(0x555690d15080 msgr2=0x555690de1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.749+0000 7f328afc4b80 1 -- 172.21.15.70:0/1375159425 shutdown_connections 2024-06-04T05:02:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.749+0000 7f328afc4b80 1 -- 172.21.15.70:0/1375159425 wait complete. 2024-06-04T05:02:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.749+0000 7f328afc4b80 1 -- start start 2024-06-04T05:02:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.749+0000 7f328afc4b80 1 Processor -- start 2024-06-04T05:02:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f328afc4b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de7860 0x555690df33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f328afc4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690de8210 0x555690df06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f328afc4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690debf60 0x555690df0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f328afc4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555690c3cfa0 con 0x555690debf60 2024-06-04T05:02:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f328afc4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555690cd87c0 con 0x555690de8210 2024-06-04T05:02:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f328afc4b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555690df3ee0 con 0x555690de7860 2024-06-04T05:02:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f328afc4b80 10 client.0 _ll_register_callbacks cb 0x555690cd1920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f327e4cb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de7860 0x555690df33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f327eccc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690debf60 0x555690df0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f327eccc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690debf60 0x555690df0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39038/0 (socket says 172.21.15.70:39038) 2024-06-04T05:02:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f327eccc700 1 -- 172.21.15.70:0/3427842899 learned_addr learned my addr 172.21.15.70:0/3427842899 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f327dcca700 1 --2- 172.21.15.70:0/3427842899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690de8210 0x555690df06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f327eccc700 1 -- 172.21.15.70:0/3427842899 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de7860 msgr2=0x555690df33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f327eccc700 1 --2- 172.21.15.70:0/3427842899 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de7860 0x555690df33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f327eccc700 1 -- 172.21.15.70:0/3427842899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690de8210 msgr2=0x555690df06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f327eccc700 1 --2- 172.21.15.70:0/3427842899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690de8210 0x555690df06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f327eccc700 1 -- 172.21.15.70:0/3427842899 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555690cd5690 con 0x555690debf60 2024-06-04T05:02:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f327eccc700 1 --2- 172.21.15.70:0/3427842899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690debf60 0x555690df0bf0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f3270003fd0 tx=0x7f3270004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7abcddb82039d8a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f3261ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:02:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.750+0000 7f3261ffb700 1 -- 172.21.15.70:0/3427842899 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327001a070 con 0x555690debf60 2024-06-04T05:02:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.751+0000 7f3261ffb700 1 -- 172.21.15.70:0/3427842899 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f3270004db0 con 0x555690debf60 2024-06-04T05:02:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.751+0000 7f328afc4b80 1 -- 172.21.15.70:0/3427842899 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32740aa0e0 con 0x555690debf60 2024-06-04T05:02:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.751+0000 7f3261ffb700 1 -- 172.21.15.70:0/3427842899 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3270025050 con 0x555690debf60 2024-06-04T05:02:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.751+0000 7f328afc4b80 10 client.16833 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.751+0000 7f328afc4b80 1 -- 172.21.15.70:0/3427842899 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555690dfe4f0 con 0x555690debf60 2024-06-04T05:02:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.751+0000 7f3261ffb700 1 -- 172.21.15.70:0/3427842899 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f32740aa0e0 con 0x555690debf60 2024-06-04T05:02:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.751+0000 7f3261ffb700 1 -- 172.21.15.70:0/3427842899 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f32700164c0 con 0x555690debf60 2024-06-04T05:02:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.751+0000 7f328afc4b80 10 client.16833 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.751+0000 7f328afc4b80 10 client.16833 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:02:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.751+0000 7f328afc4b80 20 client.16833 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.751+0000 7f328afc4b80 1 --2- 172.21.15.70:0/3427842899 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555690e00ad0 0x555690e02ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.751+0000 7f328afc4b80 4 client.16833 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:02:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.751+0000 7f328afc4b80 1 -- 172.21.15.70:0/3427842899 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555690db39e0 con 0x555690e00ad0 2024-06-04T05:02:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.752+0000 7f327e4cb700 1 --2- 172.21.15.70:0/3427842899 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555690e00ad0 0x555690e02ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.752+0000 7f327e4cb700 1 --2- 172.21.15.70:0/3427842899 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555690e00ad0 0x555690e02ef0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c84607bd3a37518d server_cookie=e562a2e347e0fc0b in_seq=0 out_seq=0 2024-06-04T05:02:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.752+0000 7f3261ffb700 10 client.16833 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:02:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.752+0000 7f3261ffb700 1 -- 172.21.15.70:0/3427842899 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555690db39e0 con 0x555690e00ad0 2024-06-04T05:02:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.752+0000 7f3261ffb700 10 client.16833 handle_command_reply: tid=0 2024-06-04T05:02:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.753+0000 7f328afc4b80 1 client.16833 shutdown 2024-06-04T05:02:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.753+0000 7f328afc4b80 1 -- 172.21.15.70:0/3427842899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690debf60 msgr2=0x555690df0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.753+0000 7f328afc4b80 1 --2- 172.21.15.70:0/3427842899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690debf60 0x555690df0bf0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f3270003fd0 tx=0x7f3270004040 comp rx=0 tx=0).stop 2024-06-04T05:02:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.753+0000 7f328afc4b80 1 -- 172.21.15.70:0/3427842899 shutdown_connections 2024-06-04T05:02:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.753+0000 7f328afc4b80 1 --2- 172.21.15.70:0/3427842899 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555690e00ad0 0x555690e02ef0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.753+0000 7f328afc4b80 1 --2- 172.21.15.70:0/3427842899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690debf60 0x555690df0bf0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.753+0000 7f328afc4b80 1 --2- 172.21.15.70:0/3427842899 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de7860 0x555690df33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.753+0000 7f3261ffb700 0 client.16833 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:02:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.753+0000 7f328afc4b80 1 --2- 172.21.15.70:0/3427842899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690de8210 0x555690df06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.753+0000 7f328afc4b80 1 -- 172.21.15.70:0/3427842899 >> 172.21.15.70:0/3427842899 conn(0x555690d15080 msgr2=0x555690de1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.753+0000 7f328afc4b80 1 -- 172.21.15.70:0/3427842899 shutdown_connections 2024-06-04T05:02:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.753+0000 7f328afc4b80 1 -- 172.21.15.70:0/3427842899 wait complete. 2024-06-04T05:02:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.753+0000 7f328afc4b80 20 client.16833 trim_cache size 0 max 16384 2024-06-04T05:02:19.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.774+0000 7f328afc4b80 1 Processor -- start 2024-06-04T05:02:19.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.774+0000 7f328afc4b80 1 -- start start 2024-06-04T05:02:19.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.774+0000 7f328afc4b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de7860 0x555690deafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.774+0000 7f328afc4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690de8210 0x555690de1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.774+0000 7f328afc4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690debf60 0x555690de21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.774+0000 7f328afc4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f32781afe60 con 0x555690debf60 2024-06-04T05:02:19.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.774+0000 7f328afc4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f32781b19a0 con 0x555690de8210 2024-06-04T05:02:19.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.774+0000 7f328afc4b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f32781b0530 con 0x555690de7860 2024-06-04T05:02:19.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.774+0000 7f327eccc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690debf60 0x555690de21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.774+0000 7f327e4cb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de7860 0x555690deafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.774+0000 7f327e4cb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de7860 0x555690deafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:49244/0 (socket says 172.21.15.70:49244) 2024-06-04T05:02:19.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.774+0000 7f327e4cb700 1 -- 172.21.15.70:0/1715349624 learned_addr learned my addr 172.21.15.70:0/1715349624 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:19.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.774+0000 7f327dcca700 1 --2- 172.21.15.70:0/1715349624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690de8210 0x555690de1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.775+0000 7f327eccc700 1 -- 172.21.15.70:0/1715349624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de7860 msgr2=0x555690deafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:19.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.775+0000 7f327eccc700 1 --2- 172.21.15.70:0/1715349624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de7860 0x555690deafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.775+0000 7f327eccc700 1 -- 172.21.15.70:0/1715349624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690de8210 msgr2=0x555690de1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:19.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.775+0000 7f327eccc700 1 --2- 172.21.15.70:0/1715349624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690de8210 0x555690de1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.775+0000 7f327eccc700 1 -- 172.21.15.70:0/1715349624 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555690deec20 con 0x555690debf60 2024-06-04T05:02:19.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.775+0000 7f327e4cb700 1 --2- 172.21.15.70:0/1715349624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de7860 0x555690deafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:19.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.775+0000 7f327eccc700 1 --2- 172.21.15.70:0/1715349624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690debf60 0x555690de21d0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f3270000f80 tx=0x7f3270004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f4985c0cace0db server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:19.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.775+0000 7f3263fff700 1 -- 172.21.15.70:0/1715349624 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327001a070 con 0x555690debf60 2024-06-04T05:02:19.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.775+0000 7f3263fff700 1 -- 172.21.15.70:0/1715349624 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f327002e070 con 0x555690debf60 2024-06-04T05:02:19.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.775+0000 7f3263fff700 1 -- 172.21.15.70:0/1715349624 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3270007500 con 0x555690debf60 2024-06-04T05:02:19.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.776+0000 7f328afc4b80 1 -- 172.21.15.70:0/1715349624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690debf60 msgr2=0x555690de21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:19.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.776+0000 7f328afc4b80 1 --2- 172.21.15.70:0/1715349624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690debf60 0x555690de21d0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f3270000f80 tx=0x7f3270004410 comp rx=0 tx=0).stop 2024-06-04T05:02:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.776+0000 7f328afc4b80 1 -- 172.21.15.70:0/1715349624 shutdown_connections 2024-06-04T05:02:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.776+0000 7f328afc4b80 1 --2- 172.21.15.70:0/1715349624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690debf60 0x555690de21d0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.776+0000 7f328afc4b80 1 --2- 172.21.15.70:0/1715349624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de7860 0x555690deafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.776+0000 7f328afc4b80 1 --2- 172.21.15.70:0/1715349624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690de8210 0x555690de1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.776+0000 7f328afc4b80 1 -- 172.21.15.70:0/1715349624 >> 172.21.15.70:0/1715349624 conn(0x555690d15080 msgr2=0x555690dea8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.776+0000 7f328afc4b80 1 -- 172.21.15.70:0/1715349624 shutdown_connections 2024-06-04T05:02:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.776+0000 7f328afc4b80 1 -- 172.21.15.70:0/1715349624 wait complete. 2024-06-04T05:02:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.776+0000 7f328afc4b80 1 -- start start 2024-06-04T05:02:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f328afc4b80 1 Processor -- start 2024-06-04T05:02:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f328afc4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690de7860 0x555690df7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f328afc4b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de8210 0x555690df7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f328afc4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690debf60 0x555690df7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f328afc4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f32781b12d0 con 0x555690de7860 2024-06-04T05:02:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f328afc4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f32781b0c00 con 0x555690debf60 2024-06-04T05:02:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f328afc4b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f32781ade00 con 0x555690de8210 2024-06-04T05:02:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f327e4cb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690de7860 0x555690df7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f328afc4b80 10 client.0 _ll_register_callbacks cb 0x555690cd1920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f327e4cb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690de7860 0x555690df7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39066/0 (socket says 172.21.15.70:39066) 2024-06-04T05:02:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f327e4cb700 1 -- 172.21.15.70:0/1966540727 learned_addr learned my addr 172.21.15.70:0/1966540727 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f327dcca700 1 --2- 172.21.15.70:0/1966540727 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de8210 0x555690df7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f327eccc700 1 --2- 172.21.15.70:0/1966540727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690debf60 0x555690df7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f327dcca700 1 -- 172.21.15.70:0/1966540727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690debf60 msgr2=0x555690df7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f327dcca700 1 --2- 172.21.15.70:0/1966540727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690debf60 0x555690df7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f327dcca700 1 -- 172.21.15.70:0/1966540727 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690de7860 msgr2=0x555690df7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f327dcca700 1 --2- 172.21.15.70:0/1966540727 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690de7860 0x555690df7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f327dcca700 1 -- 172.21.15.70:0/1966540727 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555690e6a7d0 con 0x555690de8210 2024-06-04T05:02:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.777+0000 7f327e4cb700 1 --2- 172.21.15.70:0/1966540727 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690de7860 0x555690df7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:19.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.778+0000 7f327dcca700 1 --2- 172.21.15.70:0/1966540727 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de8210 0x555690df7570 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x555690ddcf40 tx=0x7f32680047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=371c577feef886d4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:19.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.778+0000 7f3261ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:02:19.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.778+0000 7f3261ffb700 1 -- 172.21.15.70:0/1966540727 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326802d060 con 0x555690de8210 2024-06-04T05:02:19.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.778+0000 7f3261ffb700 1 -- 172.21.15.70:0/1966540727 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f3268003d70 con 0x555690de8210 2024-06-04T05:02:19.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.778+0000 7f328afc4b80 1 -- 172.21.15.70:0/1966540727 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f327002e070 con 0x555690de8210 2024-06-04T05:02:19.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.778+0000 7f3261ffb700 1 -- 172.21.15.70:0/1966540727 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f326801e420 con 0x555690de8210 2024-06-04T05:02:19.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.778+0000 7f328afc4b80 10 client.26194 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:19.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.778+0000 7f328afc4b80 1 -- 172.21.15.70:0/1966540727 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555690e6bb40 con 0x555690de8210 2024-06-04T05:02:19.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.778+0000 7f3261ffb700 1 -- 172.21.15.70:0/1966540727 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f327002e070 con 0x555690de8210 2024-06-04T05:02:19.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.778+0000 7f3261ffb700 1 -- 172.21.15.70:0/1966540727 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f3268027030 con 0x555690de8210 2024-06-04T05:02:19.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.778+0000 7f328afc4b80 10 client.26194 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:19.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.778+0000 7f328afc4b80 10 client.26194 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:02:19.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.778+0000 7f328afc4b80 20 client.26194 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:19.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.778+0000 7f328afc4b80 1 --2- 172.21.15.70:0/1966540727 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555690e00ad0 0x555690e6e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:19.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.778+0000 7f328afc4b80 4 client.26194 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:02:19.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.778+0000 7f328afc4b80 1 -- 172.21.15.70:0/1966540727 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555690d48be0 con 0x555690e00ad0 2024-06-04T05:02:19.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.779+0000 7f327e4cb700 1 --2- 172.21.15.70:0/1966540727 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555690e00ad0 0x555690e6e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:19.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.779+0000 7f3261ffb700 10 client.26194 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:02:19.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.779+0000 7f327e4cb700 1 --2- 172.21.15.70:0/1966540727 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555690e00ad0 0x555690e6e2c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=56f231e011896a88 server_cookie=8796f005ee76ca5c in_seq=0 out_seq=0 2024-06-04T05:02:19.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.779+0000 7f3261ffb700 1 -- 172.21.15.70:0/1966540727 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555690d48be0 con 0x555690e00ad0 2024-06-04T05:02:19.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.779+0000 7f3261ffb700 10 client.26194 handle_command_reply: tid=0 2024-06-04T05:02:19.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.779+0000 7f328afc4b80 1 client.26194 shutdown 2024-06-04T05:02:19.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.779+0000 7f328afc4b80 1 -- 172.21.15.70:0/1966540727 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de8210 msgr2=0x555690df7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:19.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.779+0000 7f328afc4b80 1 --2- 172.21.15.70:0/1966540727 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de8210 0x555690df7570 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x555690ddcf40 tx=0x7f32680047c0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.779+0000 7f328afc4b80 1 -- 172.21.15.70:0/1966540727 shutdown_connections 2024-06-04T05:02:19.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.779+0000 7f328afc4b80 1 --2- 172.21.15.70:0/1966540727 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555690e00ad0 0x555690e6e2c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.779+0000 7f328afc4b80 1 --2- 172.21.15.70:0/1966540727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555690debf60 0x555690df7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.779+0000 7f3261ffb700 0 client.26194 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:02:19.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.779+0000 7f328afc4b80 1 --2- 172.21.15.70:0/1966540727 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555690de7860 0x555690df7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.779+0000 7f328afc4b80 1 --2- 172.21.15.70:0/1966540727 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555690de8210 0x555690df7570 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.779+0000 7f328afc4b80 1 -- 172.21.15.70:0/1966540727 >> 172.21.15.70:0/1966540727 conn(0x555690d15080 msgr2=0x555690deb140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:19.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.779+0000 7f328afc4b80 1 -- 172.21.15.70:0/1966540727 shutdown_connections 2024-06-04T05:02:19.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.779+0000 7f328afc4b80 1 -- 172.21.15.70:0/1966540727 wait complete. 2024-06-04T05:02:19.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.780+0000 7f328afc4b80 20 client.26194 trim_cache size 0 max 16384 2024-06-04T05:02:19.787 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:02:19.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.780+0000 7f328072f700 1 -- 172.21.15.70:0/950893525 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f32640b2a70 msgr2=0x7f32640b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:19.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.780+0000 7f328072f700 1 --2- 172.21.15.70:0/950893525 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f32640b2a70 0x7f32640b4f30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f32781031c0 tx=0x7f3268009260 comp rx=0 tx=0).stop 2024-06-04T05:02:19.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.780+0000 7f328072f700 1 -- 172.21.15.70:0/950893525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3278103850 msgr2=0x7f3278075840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:19.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.780+0000 7f328072f700 1 --2- 172.21.15.70:0/950893525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3278103850 0x7f3278075840 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3274006010 tx=0x7f32740045e0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.780+0000 7f328072f700 1 -- 172.21.15.70:0/950893525 shutdown_connections 2024-06-04T05:02:19.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.780+0000 7f328072f700 1 --2- 172.21.15.70:0/950893525 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f32640b2a70 0x7f32640b4f30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.780+0000 7f328072f700 1 --2- 172.21.15.70:0/950893525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3278108920 0x7f32780762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.780+0000 7f328072f700 1 --2- 172.21.15.70:0/950893525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3278103850 0x7f3278075840 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.780+0000 7f328072f700 1 --2- 172.21.15.70:0/950893525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3278104200 0x7f3278075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:19.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.780+0000 7f328072f700 1 -- 172.21.15.70:0/950893525 >> 172.21.15.70:0/950893525 conn(0x7f327807a620 msgr2=0x7f3278101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:19.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.780+0000 7f328072f700 1 -- 172.21.15.70:0/950893525 shutdown_connections 2024-06-04T05:02:19.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:19.780+0000 7f328072f700 1 -- 172.21.15.70:0/950893525 wait complete. 2024-06-04T05:02:20.585 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:02:20.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 73712 2.08 MB/sec execute 73 sec latency 14.797 ms 2024-06-04T05:02:20.854 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:02:21.328 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:21 smithi070 ceph-mon[32631]: pgmap v158: 97 pgs: 97 active+clean; 57 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-06-04T05:02:21.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:21 smithi098 ceph-mon[35569]: pgmap v158: 97 pgs: 97 active+clean; 57 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-06-04T05:02:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:21 smithi112 ceph-mon[36453]: pgmap v158: 97 pgs: 97 active+clean; 57 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-06-04T05:02:21.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 74315 2.10 MB/sec execute 74 sec latency 16.275 ms 2024-06-04T05:02:22.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 74789 2.13 MB/sec execute 75 sec latency 39.650 ms 2024-06-04T05:02:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.771+0000 7ff88d367700 1 -- 172.21.15.70:0/1678561705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff888102c40 msgr2=0x7ff8881030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.771+0000 7ff88d367700 1 --2- 172.21.15.70:0/1678561705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff888102c40 0x7ff8881030c0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ff87c009990 tx=0x7ff87c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.771+0000 7ff88d367700 1 -- 172.21.15.70:0/1678561705 shutdown_connections 2024-06-04T05:02:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.771+0000 7ff88d367700 1 --2- 172.21.15.70:0/1678561705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff888103600 0x7ff88810f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.771+0000 7ff88d367700 1 --2- 172.21.15.70:0/1678561705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8881084c0 0x7ff8881088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.771+0000 7ff88d367700 1 --2- 172.21.15.70:0/1678561705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff888102c40 0x7ff8881030c0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.771+0000 7ff88d367700 1 -- 172.21.15.70:0/1678561705 >> 172.21.15.70:0/1678561705 conn(0x7ff888078ab0 msgr2=0x7ff8880ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.771+0000 7ff88d367700 1 -- 172.21.15.70:0/1678561705 shutdown_connections 2024-06-04T05:02:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.771+0000 7ff88d367700 1 -- 172.21.15.70:0/1678561705 wait complete. 2024-06-04T05:02:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.772+0000 7ff88d367700 1 Processor -- start 2024-06-04T05:02:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.772+0000 7ff88d367700 1 -- start start 2024-06-04T05:02:22.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.772+0000 7ff88d367700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff888102c40 0x7ff88819d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.772+0000 7ff88d367700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff888103600 0x7ff88819db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.772+0000 7ff88d367700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8881084c0 0x7ff8881977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.772+0000 7ff88d367700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff888111410 con 0x7ff888103600 2024-06-04T05:02:22.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.772+0000 7ff88d367700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff888111290 con 0x7ff888102c40 2024-06-04T05:02:22.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.772+0000 7ff88d367700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff888111590 con 0x7ff8881084c0 2024-06-04T05:02:22.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.772+0000 7ff8867fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff888103600 0x7ff88819db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.772+0000 7ff8867fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff888103600 0x7ff88819db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39080/0 (socket says 172.21.15.70:39080) 2024-06-04T05:02:22.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.772+0000 7ff8867fc700 1 -- 172.21.15.70:0/3428864557 learned_addr learned my addr 172.21.15.70:0/3428864557 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:22.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.772+0000 7ff886ffd700 1 --2- 172.21.15.70:0/3428864557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff888102c40 0x7ff88819d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.772+0000 7ff8877fe700 1 --2- 172.21.15.70:0/3428864557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8881084c0 0x7ff8881977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.773+0000 7ff8867fc700 1 -- 172.21.15.70:0/3428864557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8881084c0 msgr2=0x7ff8881977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:22.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.773+0000 7ff8867fc700 1 --2- 172.21.15.70:0/3428864557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8881084c0 0x7ff8881977d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.773+0000 7ff8867fc700 1 -- 172.21.15.70:0/3428864557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff888102c40 msgr2=0x7ff88819d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.773+0000 7ff8867fc700 1 --2- 172.21.15.70:0/3428864557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff888102c40 0x7ff88819d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.773+0000 7ff8867fc700 1 -- 172.21.15.70:0/3428864557 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff888197fb0 con 0x7ff888103600 2024-06-04T05:02:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.773+0000 7ff8867fc700 1 --2- 172.21.15.70:0/3428864557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff888103600 0x7ff88819db90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7ff87c0098e0 tx=0x7ff87c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.773+0000 7ff86ffff700 1 -- 172.21.15.70:0/3428864557 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c027070 con 0x7ff888103600 2024-06-04T05:02:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.773+0000 7ff86ffff700 1 -- 172.21.15.70:0/3428864557 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff87c022470 con 0x7ff888103600 2024-06-04T05:02:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.773+0000 7ff86ffff700 1 -- 172.21.15.70:0/3428864557 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c017650 con 0x7ff888103600 2024-06-04T05:02:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.773+0000 7ff88d367700 1 -- 172.21.15.70:0/3428864557 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff888198240 con 0x7ff888103600 2024-06-04T05:02:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.773+0000 7ff88d367700 1 -- 172.21.15.70:0/3428864557 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8881a44b0 con 0x7ff888103600 2024-06-04T05:02:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.775+0000 7ff86ffff700 1 -- 172.21.15.70:0/3428864557 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff87c02a070 con 0x7ff888103600 2024-06-04T05:02:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.776+0000 7ff86ffff700 1 --2- 172.21.15.70:0/3428864557 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8700b2a70 0x7ff8700b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.776+0000 7ff886ffd700 1 --2- 172.21.15.70:0/3428864557 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8700b2a70 0x7ff8700b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.776+0000 7ff86ffff700 1 -- 172.21.15.70:0/3428864557 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ff87c0e31f0 con 0x7ff888103600 2024-06-04T05:02:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff897bfcb80 1 Processor -- start 2024-06-04T05:02:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff897bfcb80 1 -- start start 2024-06-04T05:02:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff886ffd700 1 --2- 172.21.15.70:0/3428864557 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8700b2a70 0x7ff8700b4f30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff88806ae90 tx=0x7ff87800f040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:22.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff897bfcb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f4468d320 0x557f4468d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff897bfcb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 0x557f44687d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff897bfcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 0x557f44694080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff897bfcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557f445e5d70 con 0x557f44688260 2024-06-04T05:02:22.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff897bfcb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557f445d5910 con 0x557f4468d320 2024-06-04T05:02:22.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff897bfcb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557f445d9950 con 0x557f446878a0 2024-06-04T05:02:22.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff8877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 0x557f44694080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff8877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 0x557f44694080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39090/0 (socket says 172.21.15.70:39090) 2024-06-04T05:02:22.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff8877fe700 1 -- 172.21.15.70:0/3441263468 learned_addr learned my addr 172.21.15.70:0/3441263468 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:22.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff8867fc700 1 --2- 172.21.15.70:0/3441263468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 0x557f44687d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff886ffd700 1 --2- 172.21.15.70:0/3441263468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f4468d320 0x557f4468d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff8877fe700 1 -- 172.21.15.70:0/3441263468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 msgr2=0x557f44687d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:22.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff8877fe700 1 --2- 172.21.15.70:0/3441263468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 0x557f44687d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff8877fe700 1 -- 172.21.15.70:0/3441263468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f4468d320 msgr2=0x557f4468d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:22.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff8877fe700 1 --2- 172.21.15.70:0/3441263468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f4468d320 0x557f4468d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff8877fe700 1 -- 172.21.15.70:0/3441263468 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f4457efb0 con 0x557f44688260 2024-06-04T05:02:22.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.777+0000 7ff8877fe700 1 --2- 172.21.15.70:0/3441263468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 0x557f44694080 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7ff88000e8a0 tx=0x7ff880009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b5ff501c457cf0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:22.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.778+0000 7ff86d7fa700 1 -- 172.21.15.70:0/3441263468 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88001e030 con 0x557f44688260 2024-06-04T05:02:22.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.778+0000 7ff86d7fa700 1 -- 172.21.15.70:0/3441263468 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff8800092e0 con 0x557f44688260 2024-06-04T05:02:22.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.778+0000 7ff86d7fa700 1 -- 172.21.15.70:0/3441263468 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff880007500 con 0x557f44688260 2024-06-04T05:02:22.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.778+0000 7ff897bfcb80 1 -- 172.21.15.70:0/3441263468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 msgr2=0x557f44694080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:22.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.778+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/3441263468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 0x557f44694080 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7ff88000e8a0 tx=0x7ff880009920 comp rx=0 tx=0).stop 2024-06-04T05:02:22.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.778+0000 7ff897bfcb80 1 -- 172.21.15.70:0/3441263468 shutdown_connections 2024-06-04T05:02:22.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.778+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/3441263468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 0x557f44694080 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.778+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/3441263468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f4468d320 0x557f4468d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/3441263468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 0x557f44687d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff897bfcb80 1 -- 172.21.15.70:0/3441263468 >> 172.21.15.70:0/3441263468 conn(0x557f445b3240 msgr2=0x557f446823e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:22.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff897bfcb80 1 -- 172.21.15.70:0/3441263468 shutdown_connections 2024-06-04T05:02:22.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff897bfcb80 1 -- 172.21.15.70:0/3441263468 wait complete. 2024-06-04T05:02:22.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff897bfcb80 1 -- start start 2024-06-04T05:02:22.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff897bfcb80 1 Processor -- start 2024-06-04T05:02:22.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff897bfcb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 0x557f44693cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff897bfcb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f44688260 0x557f44690fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff897bfcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f4468d320 0x557f446914f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff897bfcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557f444dbfa0 con 0x557f4468d320 2024-06-04T05:02:22.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff897bfcb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557f4453dda0 con 0x557f44688260 2024-06-04T05:02:22.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff897bfcb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557f446946e0 con 0x557f446878a0 2024-06-04T05:02:22.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff897bfcb80 10 client.0 _ll_register_callbacks cb 0x557f44570950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:22.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff8877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f4468d320 0x557f446914f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff8867fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f44688260 0x557f44690fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff8877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f4468d320 0x557f446914f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39102/0 (socket says 172.21.15.70:39102) 2024-06-04T05:02:22.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff8877fe700 1 -- 172.21.15.70:0/872849597 learned_addr learned my addr 172.21.15.70:0/872849597 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:22.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.779+0000 7ff886ffd700 1 --2- 172.21.15.70:0/872849597 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 0x557f44693cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.780+0000 7ff8867fc700 1 -- 172.21.15.70:0/872849597 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 msgr2=0x557f44693cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:22.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.780+0000 7ff8867fc700 1 --2- 172.21.15.70:0/872849597 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 0x557f44693cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.780+0000 7ff8867fc700 1 -- 172.21.15.70:0/872849597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f4468d320 msgr2=0x557f446914f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:22.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.780+0000 7ff8867fc700 1 --2- 172.21.15.70:0/872849597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f4468d320 0x557f446914f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.780+0000 7ff8867fc700 1 -- 172.21.15.70:0/872849597 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f445258d0 con 0x557f44688260 2024-06-04T05:02:22.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.780+0000 7ff886ffd700 1 --2- 172.21.15.70:0/872849597 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 0x557f44693cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:02:22.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.780+0000 7ff8867fc700 1 --2- 172.21.15.70:0/872849597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f44688260 0x557f44690fb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff87c0e0940 tx=0x7ff87c0a9cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21c50d9a98eca419 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:22.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.780+0000 7ff86aff5700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:02:22.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.780+0000 7ff86aff5700 1 -- 172.21.15.70:0/872849597 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c0aac60 con 0x557f44688260 2024-06-04T05:02:22.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.780+0000 7ff86aff5700 1 -- 172.21.15.70:0/872849597 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ff87c022920 con 0x557f44688260 2024-06-04T05:02:22.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.780+0000 7ff897bfcb80 1 -- 172.21.15.70:0/872849597 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8800092e0 con 0x557f44688260 2024-06-04T05:02:22.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.780+0000 7ff86aff5700 1 -- 172.21.15.70:0/872849597 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff87c0aa560 con 0x557f44688260 2024-06-04T05:02:22.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.780+0000 7ff897bfcb80 10 client.26153 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:22.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.780+0000 7ff897bfcb80 1 -- 172.21.15.70:0/872849597 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557f4458f760 con 0x557f44688260 2024-06-04T05:02:22.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.780+0000 7ff86aff5700 1 -- 172.21.15.70:0/872849597 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff8800092e0 con 0x557f44688260 2024-06-04T05:02:22.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.781+0000 7ff86aff5700 1 -- 172.21.15.70:0/872849597 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ff87c0aa7c0 con 0x557f44688260 2024-06-04T05:02:22.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.781+0000 7ff897bfcb80 10 client.26153 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:22.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.781+0000 7ff897bfcb80 10 client.26153 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:02:22.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.781+0000 7ff897bfcb80 20 client.26153 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:22.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.781+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/872849597 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x557f446a1180 0x557f446a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.781+0000 7ff897bfcb80 4 client.26153 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:02:22.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.781+0000 7ff897bfcb80 1 -- 172.21.15.70:0/872849597 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f445e7cc0 con 0x557f446a1180 2024-06-04T05:02:22.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.781+0000 7ff886ffd700 1 --2- 172.21.15.70:0/872849597 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x557f446a1180 0x557f446a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.781+0000 7ff886ffd700 1 --2- 172.21.15.70:0/872849597 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x557f446a1180 0x557f446a35d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=73a524fe6facabef server_cookie=f7971bc864e9718f in_seq=0 out_seq=0 2024-06-04T05:02:22.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.781+0000 7ff86aff5700 10 client.26153 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:02:22.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.782+0000 7ff86aff5700 1 -- 172.21.15.70:0/872849597 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557f445e7cc0 con 0x557f446a1180 2024-06-04T05:02:22.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.782+0000 7ff86aff5700 10 client.26153 handle_command_reply: tid=0 2024-06-04T05:02:22.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.782+0000 7ff897bfcb80 1 client.26153 shutdown 2024-06-04T05:02:22.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.782+0000 7ff897bfcb80 1 -- 172.21.15.70:0/872849597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f44688260 msgr2=0x557f44690fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:22.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.782+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/872849597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f44688260 0x557f44690fb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff87c0e0940 tx=0x7ff87c0a9cd0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.782+0000 7ff897bfcb80 1 -- 172.21.15.70:0/872849597 shutdown_connections 2024-06-04T05:02:22.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.782+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/872849597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f4468d320 0x557f446914f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.782+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/872849597 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x557f446a1180 0x557f446a35d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.782+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/872849597 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 0x557f44693cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.782+0000 7ff86aff5700 0 client.26153 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:02:22.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.782+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/872849597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f44688260 0x557f44690fb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.782+0000 7ff897bfcb80 1 -- 172.21.15.70:0/872849597 >> 172.21.15.70:0/872849597 conn(0x557f445b3240 msgr2=0x557f44681af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:22.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.782+0000 7ff897bfcb80 1 -- 172.21.15.70:0/872849597 shutdown_connections 2024-06-04T05:02:22.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.782+0000 7ff897bfcb80 1 -- 172.21.15.70:0/872849597 wait complete. 2024-06-04T05:02:22.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.783+0000 7ff897bfcb80 20 client.26153 trim_cache size 0 max 16384 2024-06-04T05:02:22.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.799+0000 7ff897bfcb80 1 Processor -- start 2024-06-04T05:02:22.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.799+0000 7ff897bfcb80 1 -- start start 2024-06-04T05:02:22.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.799+0000 7ff897bfcb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 0x557f446a3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.799+0000 7ff897bfcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 0x557f44682270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.799+0000 7ff897bfcb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f4468d320 0x557f4468c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.799+0000 7ff897bfcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff8881abb30 con 0x557f44688260 2024-06-04T05:02:22.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.799+0000 7ff897bfcb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8881ad670 con 0x557f4468d320 2024-06-04T05:02:22.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.799+0000 7ff897bfcb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff8881ac200 con 0x557f446878a0 2024-06-04T05:02:22.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.799+0000 7ff8867fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 0x557f44682270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.799+0000 7ff886ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 0x557f446a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.799+0000 7ff886ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 0x557f446a3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:49302/0 (socket says 172.21.15.70:49302) 2024-06-04T05:02:22.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.799+0000 7ff8867fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 0x557f44682270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39110/0 (socket says 172.21.15.70:39110) 2024-06-04T05:02:22.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.799+0000 7ff886ffd700 1 -- 172.21.15.70:0/2171694185 learned_addr learned my addr 172.21.15.70:0/2171694185 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:22.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.800+0000 7ff8877fe700 1 --2- 172.21.15.70:0/2171694185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f4468d320 0x557f4468c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.800+0000 7ff886ffd700 1 -- 172.21.15.70:0/2171694185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f4468d320 msgr2=0x557f4468c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:22.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.800+0000 7ff886ffd700 1 --2- 172.21.15.70:0/2171694185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f4468d320 0x557f4468c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.800+0000 7ff886ffd700 1 -- 172.21.15.70:0/2171694185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 msgr2=0x557f44682270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:22.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.800+0000 7ff886ffd700 1 --2- 172.21.15.70:0/2171694185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 0x557f44682270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.800+0000 7ff886ffd700 1 -- 172.21.15.70:0/2171694185 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f446832b0 con 0x557f446878a0 2024-06-04T05:02:22.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.800+0000 7ff8867fc700 1 --2- 172.21.15.70:0/2171694185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 0x557f44682270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:22.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.800+0000 7ff886ffd700 1 --2- 172.21.15.70:0/2171694185 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 0x557f446a3570 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x557f446a3e80 tx=0x7ff878014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8de01443ad26c315 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:22.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.800+0000 7ff86cff9700 1 -- 172.21.15.70:0/2171694185 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff878004030 con 0x557f446878a0 2024-06-04T05:02:22.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.800+0000 7ff86cff9700 1 -- 172.21.15.70:0/2171694185 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff87801b7c0 con 0x557f446878a0 2024-06-04T05:02:22.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.800+0000 7ff86cff9700 1 -- 172.21.15.70:0/2171694185 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff878003a80 con 0x557f446878a0 2024-06-04T05:02:22.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.801+0000 7ff897bfcb80 1 -- 172.21.15.70:0/2171694185 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 msgr2=0x557f446a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:22.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.801+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/2171694185 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 0x557f446a3570 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x557f446a3e80 tx=0x7ff878014000 comp rx=0 tx=0).stop 2024-06-04T05:02:22.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.801+0000 7ff897bfcb80 1 -- 172.21.15.70:0/2171694185 shutdown_connections 2024-06-04T05:02:22.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.801+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/2171694185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f4468d320 0x557f4468c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.801+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/2171694185 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f446878a0 0x557f446a3570 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.801+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/2171694185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 0x557f44682270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.801+0000 7ff897bfcb80 1 -- 172.21.15.70:0/2171694185 >> 172.21.15.70:0/2171694185 conn(0x557f445b3240 msgr2=0x557f4467e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:22.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.801+0000 7ff897bfcb80 1 -- 172.21.15.70:0/2171694185 shutdown_connections 2024-06-04T05:02:22.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.801+0000 7ff897bfcb80 1 -- 172.21.15.70:0/2171694185 wait complete. 2024-06-04T05:02:22.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.801+0000 7ff897bfcb80 1 -- start start 2024-06-04T05:02:22.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff897bfcb80 1 Processor -- start 2024-06-04T05:02:22.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff897bfcb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f446878a0 0x557f44699ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff897bfcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 0x557f446a8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff897bfcb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f4468d320 0x557f446a8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff897bfcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff8881acfa0 con 0x557f44688260 2024-06-04T05:02:22.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff897bfcb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8881ac8d0 con 0x557f446878a0 2024-06-04T05:02:22.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff897bfcb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff8881a9ad0 con 0x557f4468d320 2024-06-04T05:02:22.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff8867fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 0x557f446a8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff886ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f446878a0 0x557f44699ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff886ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f446878a0 0x557f44699ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:43310/0 (socket says 172.21.15.70:43310) 2024-06-04T05:02:22.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff897bfcb80 10 client.0 _ll_register_callbacks cb 0x557f44570950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:22.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff8867fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 0x557f446a8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39116/0 (socket says 172.21.15.70:39116) 2024-06-04T05:02:22.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff886ffd700 1 -- 172.21.15.70:0/68558992 learned_addr learned my addr 172.21.15.70:0/68558992 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:22.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff8877fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f4468d320 0x557f446a8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff886ffd700 1 -- 172.21.15.70:0/68558992 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f4468d320 msgr2=0x557f446a8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:22.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff886ffd700 1 --2- 172.21.15.70:0/68558992 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f4468d320 0x557f446a8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff886ffd700 1 -- 172.21.15.70:0/68558992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 msgr2=0x557f446a8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:22.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff886ffd700 1 --2- 172.21.15.70:0/68558992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 0x557f446a8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff886ffd700 1 -- 172.21.15.70:0/68558992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f446a9290 con 0x557f446878a0 2024-06-04T05:02:22.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.802+0000 7ff8877fe700 1 --2- 172.21.15.70:0/68558992 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f4468d320 0x557f446a8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:22.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.803+0000 7ff886ffd700 1 --2- 172.21.15.70:0/68558992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f446878a0 0x557f44699ee0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff87801b710 tx=0x7ff878009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50eeae79884a442 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:22.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.803+0000 7ff86aff5700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:02:22.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.803+0000 7ff86aff5700 1 -- 172.21.15.70:0/68558992 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87801bb70 con 0x557f446878a0 2024-06-04T05:02:22.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.803+0000 7ff897bfcb80 1 -- 172.21.15.70:0/68558992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff87801b7c0 con 0x557f446878a0 2024-06-04T05:02:22.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.803+0000 7ff86aff5700 1 -- 172.21.15.70:0/68558992 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ff87801bd10 con 0x557f446878a0 2024-06-04T05:02:22.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.803+0000 7ff86aff5700 1 -- 172.21.15.70:0/68558992 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff878041050 con 0x557f446878a0 2024-06-04T05:02:22.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.803+0000 7ff897bfcb80 10 client.26159 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:22.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.803+0000 7ff897bfcb80 1 -- 172.21.15.70:0/68558992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557f4470bdb0 con 0x557f446878a0 2024-06-04T05:02:22.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.803+0000 7ff86aff5700 1 -- 172.21.15.70:0/68558992 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff87801b7c0 con 0x557f446878a0 2024-06-04T05:02:22.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff86aff5700 1 -- 172.21.15.70:0/68558992 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ff878003c20 con 0x557f446878a0 2024-06-04T05:02:22.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff897bfcb80 10 client.26159 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:22.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff897bfcb80 10 client.26159 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:02:22.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff897bfcb80 20 client.26159 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:22.843 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:02:22.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/68558992 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x557f4470c2f0 0x557f4470e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:22.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff897bfcb80 4 client.26159 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:02:22.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff897bfcb80 1 -- 172.21.15.70:0/68558992 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557f445e14a0 con 0x557f4470c2f0 2024-06-04T05:02:22.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff8867fc700 1 --2- 172.21.15.70:0/68558992 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x557f4470c2f0 0x557f4470e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:22.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff86aff5700 10 client.26159 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:02:22.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff8867fc700 1 --2- 172.21.15.70:0/68558992 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x557f4470c2f0 0x557f4470e6e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=83459d5b11a75f80 server_cookie=411b96d8702b6bd1 in_seq=0 out_seq=0 2024-06-04T05:02:22.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff86aff5700 1 -- 172.21.15.70:0/68558992 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557f445e14a0 con 0x557f4470c2f0 2024-06-04T05:02:22.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff86aff5700 10 client.26159 handle_command_reply: tid=0 2024-06-04T05:02:22.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff897bfcb80 1 client.26159 shutdown 2024-06-04T05:02:22.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff897bfcb80 1 -- 172.21.15.70:0/68558992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f446878a0 msgr2=0x557f44699ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:22.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/68558992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f446878a0 0x557f44699ee0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff87801b710 tx=0x7ff878009b20 comp rx=0 tx=0).stop 2024-06-04T05:02:22.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff897bfcb80 1 -- 172.21.15.70:0/68558992 shutdown_connections 2024-06-04T05:02:22.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/68558992 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x557f4470c2f0 0x557f4470e6e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/68558992 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f4468d320 0x557f446a8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/68558992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f446878a0 0x557f44699ee0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff86aff5700 0 client.26159 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:02:22.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.804+0000 7ff897bfcb80 1 --2- 172.21.15.70:0/68558992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f44688260 0x557f446a8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.805+0000 7ff897bfcb80 1 -- 172.21.15.70:0/68558992 >> 172.21.15.70:0/68558992 conn(0x557f445b3240 msgr2=0x557f4468c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:22.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.805+0000 7ff897bfcb80 1 -- 172.21.15.70:0/68558992 shutdown_connections 2024-06-04T05:02:22.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.805+0000 7ff897bfcb80 1 -- 172.21.15.70:0/68558992 wait complete. 2024-06-04T05:02:22.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.805+0000 7ff897bfcb80 20 client.26159 trim_cache size 0 max 16384 2024-06-04T05:02:22.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.805+0000 7ff88d367700 1 -- 172.21.15.70:0/3428864557 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8700b2a70 msgr2=0x7ff8700b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:22.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.805+0000 7ff88d367700 1 --2- 172.21.15.70:0/3428864557 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8700b2a70 0x7ff8700b4f30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff88806ae90 tx=0x7ff87800f040 comp rx=0 tx=0).stop 2024-06-04T05:02:22.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.805+0000 7ff88d367700 1 -- 172.21.15.70:0/3428864557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff888103600 msgr2=0x7ff88819db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:22.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.805+0000 7ff88d367700 1 --2- 172.21.15.70:0/3428864557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff888103600 0x7ff88819db90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7ff87c0098e0 tx=0x7ff87c0057c0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.805+0000 7ff88d367700 1 -- 172.21.15.70:0/3428864557 shutdown_connections 2024-06-04T05:02:22.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.805+0000 7ff88d367700 1 --2- 172.21.15.70:0/3428864557 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8700b2a70 0x7ff8700b4f30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.805+0000 7ff88d367700 1 --2- 172.21.15.70:0/3428864557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8881084c0 0x7ff8881977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.805+0000 7ff88d367700 1 --2- 172.21.15.70:0/3428864557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff888102c40 0x7ff88819d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.805+0000 7ff88d367700 1 --2- 172.21.15.70:0/3428864557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff888103600 0x7ff88819db90 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:22.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.805+0000 7ff88d367700 1 -- 172.21.15.70:0/3428864557 >> 172.21.15.70:0/3428864557 conn(0x7ff888078ab0 msgr2=0x7ff888100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:22.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.805+0000 7ff88d367700 1 -- 172.21.15.70:0/3428864557 shutdown_connections 2024-06-04T05:02:22.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:22.805+0000 7ff88d367700 1 -- 172.21.15.70:0/3428864557 wait complete. 2024-06-04T05:02:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:23 smithi098 ceph-mon[35569]: pgmap v159: 97 pgs: 97 active+clean; 59 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 2.1 MiB/s wr, 201 op/s 2024-06-04T05:02:23.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:23 smithi070 ceph-mon[32631]: pgmap v159: 97 pgs: 97 active+clean; 59 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 2.1 MiB/s wr, 201 op/s 2024-06-04T05:02:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:23 smithi112 ceph-mon[36453]: pgmap v159: 97 pgs: 97 active+clean; 59 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 2.1 MiB/s wr, 201 op/s 2024-06-04T05:02:23.707 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:02:23.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 75055 2.11 MB/sec execute 76 sec latency 13.210 ms 2024-06-04T05:02:23.973 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:02:24.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 75523 2.09 MB/sec execute 77 sec latency 15.251 ms 2024-06-04T05:02:25.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:25 smithi098 ceph-mon[35569]: pgmap v160: 97 pgs: 97 active+clean; 59 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-06-04T05:02:25.516 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:25 smithi070 ceph-mon[32631]: pgmap v160: 97 pgs: 97 active+clean; 59 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-06-04T05:02:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:25 smithi112 ceph-mon[36453]: pgmap v160: 97 pgs: 97 active+clean; 59 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-06-04T05:02:25.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 76138 2.12 MB/sec execute 78 sec latency 12.118 ms 2024-06-04T05:02:25.848 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.846+0000 7f0328980640 1 -- 172.21.15.70:0/293338122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0324102340 msgr2=0x7f032410e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.846+0000 7f0328980640 1 --2- 172.21.15.70:0/293338122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0324102340 0x7f032410e800 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f031c009cd0 tx=0x7f031c02a8e0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.846+0000 7f0328980640 1 -- 172.21.15.70:0/293338122 shutdown_connections 2024-06-04T05:02:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.846+0000 7f0328980640 1 --2- 172.21.15.70:0/293338122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0324101980 0x7f0324101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.846+0000 7f0328980640 1 --2- 172.21.15.70:0/293338122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0324102340 0x7f032410e800 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.846+0000 7f0328980640 1 --2- 172.21.15.70:0/293338122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0324107940 0x7f0324107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.846+0000 7f0328980640 1 -- 172.21.15.70:0/293338122 >> 172.21.15.70:0/293338122 conn(0x7f0324078ad0 msgr2=0x7f0324078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.846+0000 7f0328980640 1 -- 172.21.15.70:0/293338122 shutdown_connections 2024-06-04T05:02:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.846+0000 7f0328980640 1 -- 172.21.15.70:0/293338122 wait complete. 2024-06-04T05:02:25.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.847+0000 7f0328980640 1 Processor -- start 2024-06-04T05:02:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.847+0000 7f0328980640 1 -- start start 2024-06-04T05:02:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.847+0000 7f0328980640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0324101980 0x7f0324197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.847+0000 7f0328980640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0324102340 0x7f03241984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.847+0000 7f0328980640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0324107940 0x7f032419c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.847+0000 7f0328980640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0324111ab0 con 0x7f0324107940 2024-06-04T05:02:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.847+0000 7f0328980640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0324111930 con 0x7f0324101980 2024-06-04T05:02:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.847+0000 7f0328980640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0324111c30 con 0x7f0324102340 2024-06-04T05:02:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.847+0000 7f0322d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0324107940 0x7f032419c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.847+0000 7f0321d9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0324102340 0x7f03241984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.847+0000 7f0322d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0324107940 0x7f032419c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39134/0 (socket says 172.21.15.70:39134) 2024-06-04T05:02:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.847+0000 7f0322d9d640 1 -- 172.21.15.70:0/875892776 learned_addr learned my addr 172.21.15.70:0/875892776 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.847+0000 7f032259c640 1 --2- 172.21.15.70:0/875892776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0324101980 0x7f0324197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.848+0000 7f0322d9d640 1 -- 172.21.15.70:0/875892776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0324102340 msgr2=0x7f03241984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.848+0000 7f0322d9d640 1 --2- 172.21.15.70:0/875892776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0324102340 0x7f03241984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.848+0000 7f0322d9d640 1 -- 172.21.15.70:0/875892776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0324101980 msgr2=0x7f0324197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.848+0000 7f0322d9d640 1 --2- 172.21.15.70:0/875892776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0324101980 0x7f0324197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.848+0000 7f0322d9d640 1 -- 172.21.15.70:0/875892776 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032419cf80 con 0x7f0324107940 2024-06-04T05:02:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.848+0000 7f0321d9b640 1 --2- 172.21.15.70:0/875892776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0324102340 0x7f03241984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.848+0000 7f0322d9d640 1 --2- 172.21.15.70:0/875892776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0324107940 0x7f032419c850 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f031c005cc0 tx=0x7f031c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.848+0000 7f031b7fe640 1 -- 172.21.15.70:0/875892776 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c004240 con 0x7f0324107940 2024-06-04T05:02:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.848+0000 7f0328980640 1 -- 172.21.15.70:0/875892776 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f032419d210 con 0x7f0324107940 2024-06-04T05:02:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.848+0000 7f031b7fe640 1 -- 172.21.15.70:0/875892776 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f031c003e10 con 0x7f0324107940 2024-06-04T05:02:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.848+0000 7f031b7fe640 1 -- 172.21.15.70:0/875892776 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c030c30 con 0x7f0324107940 2024-06-04T05:02:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.848+0000 7f0328980640 1 -- 172.21.15.70:0/875892776 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03241a4b00 con 0x7f0324107940 2024-06-04T05:02:25.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.851+0000 7f031b7fe640 1 -- 172.21.15.70:0/875892776 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f031c0043e0 con 0x7f0324107940 2024-06-04T05:02:25.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.851+0000 7f032b770740 1 Processor -- start 2024-06-04T05:02:25.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.851+0000 7f032b770740 1 -- start start 2024-06-04T05:02:25.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.851+0000 7f032b770740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c854096e0 0x558c85409ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.851+0000 7f032b770740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c8540a000 0x558c854166d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.851+0000 7f032b770740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c85416c10 0x558c85419000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.851+0000 7f032b770740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558c8537bc30 con 0x558c854096e0 2024-06-04T05:02:25.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.851+0000 7f032b770740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558c8536b7d0 con 0x558c85416c10 2024-06-04T05:02:25.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.851+0000 7f032b770740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558c8536f810 con 0x558c8540a000 2024-06-04T05:02:25.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.852+0000 7f032259c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c8540a000 0x558c854166d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:25.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.852+0000 7f032259c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c8540a000 0x558c854166d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:49352/0 (socket says 172.21.15.70:49352) 2024-06-04T05:02:25.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.852+0000 7f032259c640 1 -- 172.21.15.70:0/1528911569 learned_addr learned my addr 172.21.15.70:0/1528911569 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.852+0000 7f031b7fe640 1 --2- 172.21.15.70:0/875892776 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f03100b2a10 0x7f03100b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.852+0000 7f032259c640 1 -- 172.21.15.70:0/1528911569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c85416c10 msgr2=0x558c85419000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-04T05:02:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.852+0000 7f032259c640 1 --2- 172.21.15.70:0/1528911569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c85416c10 0x558c85419000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.852+0000 7f032259c640 1 -- 172.21.15.70:0/1528911569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c854096e0 msgr2=0x558c85409ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-04T05:02:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.852+0000 7f032259c640 1 --2- 172.21.15.70:0/1528911569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c854096e0 0x558c85409ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.852+0000 7f032259c640 1 -- 172.21.15.70:0/1528911569 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c85080210 con 0x558c8540a000 2024-06-04T05:02:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.852+0000 7f031b7fe640 1 -- 172.21.15.70:0/875892776 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f031c02e030 con 0x7f0324107940 2024-06-04T05:02:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.852+0000 7f0322d9d640 1 --2- 172.21.15.70:0/875892776 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f03100b2a10 0x7f03100b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.852+0000 7f032259c640 1 --2- 172.21.15.70:0/1528911569 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c8540a000 0x558c854166d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0314009ca0 tx=0x7f0314011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d6215a5ef6ef63f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.852+0000 7f0318ff9640 1 -- 172.21.15.70:0/1528911569 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031401e070 con 0x558c8540a000 2024-06-04T05:02:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.852+0000 7f0318ff9640 1 -- 172.21.15.70:0/1528911569 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0314019ac0 con 0x558c8540a000 2024-06-04T05:02:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.852+0000 7f0318ff9640 1 -- 172.21.15.70:0/1528911569 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031400e930 con 0x558c8540a000 2024-06-04T05:02:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.853+0000 7f032b770740 1 -- 172.21.15.70:0/1528911569 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c8540a000 msgr2=0x558c854166d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.853+0000 7f0322d9d640 1 --2- 172.21.15.70:0/875892776 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f03100b2a10 0x7f03100b4ed0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f031c0e4380 tx=0x7f031c02a8e0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.853+0000 7f032b770740 1 --2- 172.21.15.70:0/1528911569 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c8540a000 0x558c854166d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0314009ca0 tx=0x7f0314011810 comp rx=0 tx=0).stop 2024-06-04T05:02:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.853+0000 7f032b770740 1 -- 172.21.15.70:0/1528911569 shutdown_connections 2024-06-04T05:02:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.853+0000 7f032b770740 1 --2- 172.21.15.70:0/1528911569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c85416c10 0x558c85419000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.853+0000 7f032b770740 1 --2- 172.21.15.70:0/1528911569 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c8540a000 0x558c854166d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.853+0000 7f032b770740 1 --2- 172.21.15.70:0/1528911569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c854096e0 0x558c85409ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.853+0000 7f032b770740 1 -- 172.21.15.70:0/1528911569 >> 172.21.15.70:0/1528911569 conn(0x558c85405d30 msgr2=0x558c85406110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.853+0000 7f032b770740 1 -- 172.21.15.70:0/1528911569 shutdown_connections 2024-06-04T05:02:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.853+0000 7f032b770740 1 -- 172.21.15.70:0/1528911569 wait complete. 2024-06-04T05:02:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.854+0000 7f032b770740 1 -- start start 2024-06-04T05:02:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.854+0000 7f032b770740 1 Processor -- start 2024-06-04T05:02:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.854+0000 7f032b770740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c854096e0 0x558c854121c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.854+0000 7f032b770740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c8540a000 0x558c85412700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.854+0000 7f032b770740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c85416c10 0x558c85424990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.854+0000 7f032b770740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558c85051790 con 0x558c8540a000 2024-06-04T05:02:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.854+0000 7f032b770740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558c85327890 con 0x558c85416c10 2024-06-04T05:02:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.854+0000 7f032b770740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558c8540c700 con 0x558c854096e0 2024-06-04T05:02:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.854+0000 7f0321d9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c854096e0 0x558c854121c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.854+0000 7f032b770740 10 client.0 _ll_register_callbacks cb 0x558c8507fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.854+0000 7f032259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c8540a000 0x558c85412700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.854+0000 7f0321d9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c854096e0 0x558c854121c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:49368/0 (socket says 172.21.15.70:49368) 2024-06-04T05:02:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.854+0000 7f0321d9b640 1 -- 172.21.15.70:0/409628494 learned_addr learned my addr 172.21.15.70:0/409628494 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.854+0000 7f0321d9b640 1 --2- 172.21.15.70:0/409628494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c85416c10 0x558c85424990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.855+0000 7f0321d9b640 1 -- 172.21.15.70:0/409628494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c85416c10 msgr2=0x558c85424990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.855+0000 7f0321d9b640 1 --2- 172.21.15.70:0/409628494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c85416c10 0x558c85424990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.855+0000 7f0321d9b640 1 -- 172.21.15.70:0/409628494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c8540a000 msgr2=0x558c85412700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.855+0000 7f0321d9b640 1 --2- 172.21.15.70:0/409628494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c8540a000 0x558c85412700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.855+0000 7f0321d9b640 1 -- 172.21.15.70:0/409628494 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c8514c620 con 0x558c854096e0 2024-06-04T05:02:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.855+0000 7f0321d9b640 1 --2- 172.21.15.70:0/409628494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c854096e0 0x558c854121c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f030c004650 tx=0x7f030c009770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e076e6ddba885d7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.855+0000 7f0309ffb640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:02:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.855+0000 7f0309ffb640 1 -- 172.21.15.70:0/409628494 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030c011050 con 0x558c854096e0 2024-06-04T05:02:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.855+0000 7f0309ffb640 1 -- 172.21.15.70:0/409628494 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f030c0111f0 con 0x558c854096e0 2024-06-04T05:02:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.855+0000 7f032b770740 1 -- 172.21.15.70:0/409628494 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0314019ac0 con 0x558c854096e0 2024-06-04T05:02:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.855+0000 7f0309ffb640 1 -- 172.21.15.70:0/409628494 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f030c010430 con 0x558c854096e0 2024-06-04T05:02:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.855+0000 7f032b770740 10 client.26224 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.855+0000 7f032b770740 1 -- 172.21.15.70:0/409628494 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558c85427870 con 0x558c854096e0 2024-06-04T05:02:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.855+0000 7f0309ffb640 1 -- 172.21.15.70:0/409628494 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0314019ac0 con 0x558c854096e0 2024-06-04T05:02:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.856+0000 7f0309ffb640 1 -- 172.21.15.70:0/409628494 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f030c028510 con 0x558c854096e0 2024-06-04T05:02:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.856+0000 7f032b770740 10 client.26224 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.856+0000 7f032b770740 10 client.26224 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:02:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.856+0000 7f032b770740 20 client.26224 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.856+0000 7f032b770740 1 --2- 172.21.15.70:0/409628494 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558c85428d30 0x558c8542b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.856+0000 7f032b770740 4 client.26224 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:02:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.856+0000 7f032b770740 1 -- 172.21.15.70:0/409628494 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c8537db80 con 0x558c85428d30 2024-06-04T05:02:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.856+0000 7f032259c640 1 --2- 172.21.15.70:0/409628494 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558c85428d30 0x558c8542b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.856+0000 7f032259c640 1 --2- 172.21.15.70:0/409628494 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558c85428d30 0x558c8542b120 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce6bdeb1de26d9ae server_cookie=1898000d63f74748 in_seq=0 out_seq=0 2024-06-04T05:02:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.856+0000 7f0309ffb640 10 client.26224 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.857+0000 7f0309ffb640 1 -- 172.21.15.70:0/409628494 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558c8537db80 con 0x558c85428d30 2024-06-04T05:02:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.857+0000 7f0309ffb640 10 client.26224 handle_command_reply: tid=0 2024-06-04T05:02:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.857+0000 7f032b770740 1 client.26224 shutdown 2024-06-04T05:02:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.857+0000 7f032b770740 1 -- 172.21.15.70:0/409628494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c854096e0 msgr2=0x558c854121c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.857+0000 7f032b770740 1 --2- 172.21.15.70:0/409628494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c854096e0 0x558c854121c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f030c004650 tx=0x7f030c009770 comp rx=0 tx=0).stop 2024-06-04T05:02:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.857+0000 7f032b770740 1 -- 172.21.15.70:0/409628494 shutdown_connections 2024-06-04T05:02:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.857+0000 7f032b770740 1 --2- 172.21.15.70:0/409628494 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558c85428d30 0x558c8542b120 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.857+0000 7f032b770740 1 --2- 172.21.15.70:0/409628494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c85416c10 0x558c85424990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.857+0000 7f032b770740 1 --2- 172.21.15.70:0/409628494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c8540a000 0x558c85412700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.857+0000 7f0309ffb640 0 client.26224 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.857+0000 7f032b770740 1 --2- 172.21.15.70:0/409628494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c854096e0 0x558c854121c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.857+0000 7f032b770740 1 -- 172.21.15.70:0/409628494 >> 172.21.15.70:0/409628494 conn(0x558c85405d30 msgr2=0x558c8540bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.857+0000 7f032b770740 1 -- 172.21.15.70:0/409628494 shutdown_connections 2024-06-04T05:02:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.857+0000 7f032b770740 1 -- 172.21.15.70:0/409628494 wait complete. 2024-06-04T05:02:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.857+0000 7f032b770740 20 client.26224 trim_cache size 0 max 16384 2024-06-04T05:02:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.877+0000 7f032b770740 1 Processor -- start 2024-06-04T05:02:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.877+0000 7f032b770740 1 -- start start 2024-06-04T05:02:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.877+0000 7f032b770740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c854096e0 0x558c8540b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.877+0000 7f032b770740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c8540a000 0x558c8540bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.877+0000 7f032b770740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c85416c10 0x558c85430970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.877+0000 7f032b770740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f03241ac330 con 0x558c85416c10 2024-06-04T05:02:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.877+0000 7f032b770740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f03241ade70 con 0x558c8540a000 2024-06-04T05:02:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.877+0000 7f032b770740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f03241aca00 con 0x558c854096e0 2024-06-04T05:02:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.878+0000 7f0321d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c85416c10 0x558c85430970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.878+0000 7f0321d9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c854096e0 0x558c8540b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.878+0000 7f032259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c8540a000 0x558c8540bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.878+0000 7f0321d9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c854096e0 0x558c8540b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:49374/0 (socket says 172.21.15.70:49374) 2024-06-04T05:02:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.878+0000 7f032259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c8540a000 0x558c8540bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:43356/0 (socket says 172.21.15.70:43356) 2024-06-04T05:02:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.878+0000 7f0321d9b640 1 -- 172.21.15.70:0/2250891142 learned_addr learned my addr 172.21.15.70:0/2250891142 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.878+0000 7f032259c640 1 -- 172.21.15.70:0/2250891142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c854096e0 msgr2=0x558c8540b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.878+0000 7f032259c640 1 --2- 172.21.15.70:0/2250891142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c854096e0 0x558c8540b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.878+0000 7f032259c640 1 -- 172.21.15.70:0/2250891142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c85416c10 msgr2=0x558c85430970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.878+0000 7f032259c640 1 --2- 172.21.15.70:0/2250891142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c85416c10 0x558c85430970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.878+0000 7f032259c640 1 -- 172.21.15.70:0/2250891142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c85410fa0 con 0x558c8540a000 2024-06-04T05:02:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.878+0000 7f032259c640 1 --2- 172.21.15.70:0/2250891142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c8540a000 0x558c8540bde0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0314009c70 tx=0x7f03140197a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=519a33f17f2b418 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.878+0000 7f030bfff640 1 -- 172.21.15.70:0/2250891142 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031401e040 con 0x558c8540a000 2024-06-04T05:02:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.878+0000 7f030bfff640 1 -- 172.21.15.70:0/2250891142 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0314019a90 con 0x558c8540a000 2024-06-04T05:02:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.878+0000 7f030bfff640 1 -- 172.21.15.70:0/2250891142 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0314004b50 con 0x558c8540a000 2024-06-04T05:02:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.879+0000 7f032b770740 1 -- 172.21.15.70:0/2250891142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c8540a000 msgr2=0x558c8540bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.879+0000 7f032b770740 1 --2- 172.21.15.70:0/2250891142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c8540a000 0x558c8540bde0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0314009c70 tx=0x7f03140197a0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.879+0000 7f032b770740 1 -- 172.21.15.70:0/2250891142 shutdown_connections 2024-06-04T05:02:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.879+0000 7f032b770740 1 --2- 172.21.15.70:0/2250891142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c8540a000 0x558c8540bde0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.879+0000 7f032b770740 1 --2- 172.21.15.70:0/2250891142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c85416c10 0x558c85430970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.879+0000 7f032b770740 1 --2- 172.21.15.70:0/2250891142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c854096e0 0x558c8540b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.879+0000 7f032b770740 1 -- 172.21.15.70:0/2250891142 >> 172.21.15.70:0/2250891142 conn(0x558c85405d30 msgr2=0x558c8540b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.879+0000 7f032b770740 1 -- 172.21.15.70:0/2250891142 shutdown_connections 2024-06-04T05:02:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.879+0000 7f032b770740 1 -- 172.21.15.70:0/2250891142 wait complete. 2024-06-04T05:02:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f032b770740 1 -- start start 2024-06-04T05:02:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f032b770740 1 Processor -- start 2024-06-04T05:02:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f032b770740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c854096e0 0x558c85420640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f032b770740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c8540a000 0x558c85420b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f032b770740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c85416c10 0x558c85492840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f032b770740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f03241ad7a0 con 0x558c854096e0 2024-06-04T05:02:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f032b770740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f03241ad0d0 con 0x558c8540a000 2024-06-04T05:02:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f032b770740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f03241aa310 con 0x558c85416c10 2024-06-04T05:02:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f032b770740 10 client.0 _ll_register_callbacks cb 0x7f030c009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f0321d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c854096e0 0x558c85420640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f0321d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c854096e0 0x558c85420640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39164/0 (socket says 172.21.15.70:39164) 2024-06-04T05:02:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f0321d9b640 1 -- 172.21.15.70:0/521358473 learned_addr learned my addr 172.21.15.70:0/521358473 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f0321d9b640 1 --2- 172.21.15.70:0/521358473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c85416c10 0x558c85492840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:25.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f032259c640 1 --2- 172.21.15.70:0/521358473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c8540a000 0x558c85420b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:25.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f0321d9b640 1 -- 172.21.15.70:0/521358473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c85416c10 msgr2=0x558c85492840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:25.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f0321d9b640 1 --2- 172.21.15.70:0/521358473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c85416c10 0x558c85492840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f0321d9b640 1 -- 172.21.15.70:0/521358473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c8540a000 msgr2=0x558c85420b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:25.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f0321d9b640 1 --2- 172.21.15.70:0/521358473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c8540a000 0x558c85420b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f0321d9b640 1 -- 172.21.15.70:0/521358473 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c85492e30 con 0x558c854096e0 2024-06-04T05:02:25.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f0321d9b640 1 --2- 172.21.15.70:0/521358473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c854096e0 0x558c85420640 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f030c001eb0 tx=0x7f030c009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e343fe677ced35b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:25.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.880+0000 7f0309ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:02:25.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.881+0000 7f0309ffb640 1 -- 172.21.15.70:0/521358473 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030c0105d0 con 0x558c854096e0 2024-06-04T05:02:25.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.881+0000 7f032b770740 1 -- 172.21.15.70:0/521358473 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0314019a90 con 0x558c854096e0 2024-06-04T05:02:25.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.881+0000 7f0309ffb640 1 -- 172.21.15.70:0/521358473 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f030c010770 con 0x558c854096e0 2024-06-04T05:02:25.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.881+0000 7f0309ffb640 1 -- 172.21.15.70:0/521358473 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f030c010990 con 0x558c854096e0 2024-06-04T05:02:25.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.881+0000 7f032b770740 10 client.16872 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:25.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.881+0000 7f032b770740 1 -- 172.21.15.70:0/521358473 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558c85494120 con 0x558c854096e0 2024-06-04T05:02:25.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.881+0000 7f0309ffb640 1 -- 172.21.15.70:0/521358473 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f030c02f040 con 0x558c854096e0 2024-06-04T05:02:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.881+0000 7f0309ffb640 1 -- 172.21.15.70:0/521358473 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f030c03f5a0 con 0x558c854096e0 2024-06-04T05:02:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.881+0000 7f032b770740 10 client.16872 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.881+0000 7f032b770740 10 client.16872 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:02:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.881+0000 7f032b770740 20 client.16872 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.881+0000 7f032b770740 1 --2- 172.21.15.70:0/521358473 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558c85494810 0x558c85496c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.881+0000 7f032b770740 4 client.16872 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:02:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.881+0000 7f032b770740 1 -- 172.21.15.70:0/521358473 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f0314019a90 con 0x558c85494810 2024-06-04T05:02:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.882+0000 7f032259c640 1 --2- 172.21.15.70:0/521358473 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558c85494810 0x558c85496c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.882+0000 7f0309ffb640 10 client.16872 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.882+0000 7f032259c640 1 --2- 172.21.15.70:0/521358473 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558c85494810 0x558c85496c30 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12a916cceb8bd30d server_cookie=1578be6dfe59ad79 in_seq=0 out_seq=0 2024-06-04T05:02:25.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f0309ffb640 1 -- 172.21.15.70:0/521358473 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x7f0314019a90 con 0x558c85494810 2024-06-04T05:02:25.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f0309ffb640 10 client.16872 handle_command_reply: tid=0 2024-06-04T05:02:25.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f032b770740 1 client.16872 shutdown 2024-06-04T05:02:25.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f032b770740 1 -- 172.21.15.70:0/521358473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c854096e0 msgr2=0x558c85420640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:25.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f032b770740 1 --2- 172.21.15.70:0/521358473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c854096e0 0x558c85420640 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f030c001eb0 tx=0x7f030c009f70 comp rx=0 tx=0).stop 2024-06-04T05:02:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f032b770740 1 -- 172.21.15.70:0/521358473 shutdown_connections 2024-06-04T05:02:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f032b770740 1 --2- 172.21.15.70:0/521358473 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558c85494810 0x558c85496c30 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f0309ffb640 0 client.16872 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f032b770740 1 --2- 172.21.15.70:0/521358473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c8540a000 0x558c85420b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f032b770740 1 --2- 172.21.15.70:0/521358473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c85416c10 0x558c85492840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f032b770740 1 --2- 172.21.15.70:0/521358473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c854096e0 0x558c85420640 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f032b770740 1 -- 172.21.15.70:0/521358473 >> 172.21.15.70:0/521358473 conn(0x558c85405d30 msgr2=0x558c8540ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f032b770740 1 -- 172.21.15.70:0/521358473 shutdown_connections 2024-06-04T05:02:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f032b770740 1 -- 172.21.15.70:0/521358473 wait complete. 2024-06-04T05:02:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f032b770740 20 client.16872 trim_cache size 0 max 16384 2024-06-04T05:02:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:02:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:02:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:02:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f0328980640 1 -- 172.21.15.70:0/875892776 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f03100b2a10 msgr2=0x7f03100b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f0328980640 1 --2- 172.21.15.70:0/875892776 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f03100b2a10 0x7f03100b4ed0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f031c0e4380 tx=0x7f031c02a8e0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f0328980640 1 -- 172.21.15.70:0/875892776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0324107940 msgr2=0x7f032419c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f0328980640 1 --2- 172.21.15.70:0/875892776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0324107940 0x7f032419c850 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f031c005cc0 tx=0x7f031c003c20 comp rx=0 tx=0).stop 2024-06-04T05:02:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f0328980640 1 -- 172.21.15.70:0/875892776 shutdown_connections 2024-06-04T05:02:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f0328980640 1 --2- 172.21.15.70:0/875892776 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f03100b2a10 0x7f03100b4ed0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f0328980640 1 --2- 172.21.15.70:0/875892776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0324107940 0x7f032419c850 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f0328980640 1 --2- 172.21.15.70:0/875892776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0324102340 0x7f03241984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f0328980640 1 --2- 172.21.15.70:0/875892776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0324101980 0x7f0324197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f0328980640 1 -- 172.21.15.70:0/875892776 >> 172.21.15.70:0/875892776 conn(0x7f0324078ad0 msgr2=0x7f03241045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f0328980640 1 -- 172.21.15.70:0/875892776 shutdown_connections 2024-06-04T05:02:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:25.909+0000 7f0328980640 1 -- 172.21.15.70:0/875892776 wait complete. 2024-06-04T05:02:26.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.119+0000 7feeca5a0700 1 -- 172.21.15.70:0/3154530845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec41088f0 msgr2=0x7feec4110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:26.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.119+0000 7feeca5a0700 1 --2- 172.21.15.70:0/3154530845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec41088f0 0x7feec4110470 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7feebc00b0d0 tx=0x7feebc00b3e0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.119+0000 7feeca5a0700 1 -- 172.21.15.70:0/3154530845 shutdown_connections 2024-06-04T05:02:26.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.119+0000 7feeca5a0700 1 --2- 172.21.15.70:0/3154530845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec41088f0 0x7feec4110470 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.119+0000 7feeca5a0700 1 --2- 172.21.15.70:0/3154530845 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feec41037f0 0x7feec4103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.119+0000 7feeca5a0700 1 --2- 172.21.15.70:0/3154530845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feec41041a0 0x7feec41081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.119+0000 7feeca5a0700 1 -- 172.21.15.70:0/3154530845 >> 172.21.15.70:0/3154530845 conn(0x7feec407a6a0 msgr2=0x7feec407aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:26.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.119+0000 7feeca5a0700 1 -- 172.21.15.70:0/3154530845 shutdown_connections 2024-06-04T05:02:26.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.119+0000 7feeca5a0700 1 -- 172.21.15.70:0/3154530845 wait complete. 2024-06-04T05:02:26.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.120+0000 7feeca5a0700 1 Processor -- start 2024-06-04T05:02:26.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.120+0000 7feeca5a0700 1 -- start start 2024-06-04T05:02:26.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.120+0000 7feeca5a0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec41037f0 0x7feec419f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.120+0000 7feeca5a0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feec41041a0 0x7feec419fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.120+0000 7feeca5a0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feec41088f0 0x7feec4199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.120+0000 7feeca5a0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7feec4113590 con 0x7feec41037f0 2024-06-04T05:02:26.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.120+0000 7feeca5a0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feec4113410 con 0x7feec41088f0 2024-06-04T05:02:26.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.120+0000 7feeca5a0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7feec4113710 con 0x7feec41041a0 2024-06-04T05:02:26.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.120+0000 7feec3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec41037f0 0x7feec419f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.120+0000 7feec37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feec41041a0 0x7feec419fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.120+0000 7feec3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec41037f0 0x7feec419f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45330/0 (socket says 172.21.15.70:45330) 2024-06-04T05:02:26.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.120+0000 7feec3fff700 1 -- 172.21.15.70:0/523082905 learned_addr learned my addr 172.21.15.70:0/523082905 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:26.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.120+0000 7feec8b3d700 1 --2- 172.21.15.70:0/523082905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feec41088f0 0x7feec4199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.121+0000 7feec3fff700 1 -- 172.21.15.70:0/523082905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feec41041a0 msgr2=0x7feec419fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:26.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.121+0000 7feec3fff700 1 --2- 172.21.15.70:0/523082905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feec41041a0 0x7feec419fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.121+0000 7feec3fff700 1 -- 172.21.15.70:0/523082905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feec41088f0 msgr2=0x7feec4199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:26.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.121+0000 7feec3fff700 1 --2- 172.21.15.70:0/523082905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feec41088f0 0x7feec4199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.121+0000 7feec3fff700 1 -- 172.21.15.70:0/523082905 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec419a030 con 0x7feec41037f0 2024-06-04T05:02:26.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.121+0000 7feec8b3d700 1 --2- 172.21.15.70:0/523082905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feec41088f0 0x7feec4199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:26.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.121+0000 7feec3fff700 1 --2- 172.21.15.70:0/523082905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec41037f0 0x7feec419f6d0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7feeb400e9c0 tx=0x7feeb400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:26.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.121+0000 7feec17fa700 1 -- 172.21.15.70:0/523082905 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb400cbd0 con 0x7feec41037f0 2024-06-04T05:02:26.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.121+0000 7feeca5a0700 1 -- 172.21.15.70:0/523082905 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feec419a320 con 0x7feec41037f0 2024-06-04T05:02:26.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.121+0000 7feec17fa700 1 -- 172.21.15.70:0/523082905 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feeb400ee60 con 0x7feec41037f0 2024-06-04T05:02:26.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.121+0000 7feec17fa700 1 -- 172.21.15.70:0/523082905 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb40054b0 con 0x7feec41037f0 2024-06-04T05:02:26.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.121+0000 7feeca5a0700 1 -- 172.21.15.70:0/523082905 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feec4075770 con 0x7feec41037f0 2024-06-04T05:02:26.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.124+0000 7feec17fa700 1 -- 172.21.15.70:0/523082905 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7feeb4004750 con 0x7feec41037f0 2024-06-04T05:02:26.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.125+0000 7feec17fa700 1 --2- 172.21.15.70:0/523082905 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feeac0b2a70 0x7feeac0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.125+0000 7feec37fe700 1 --2- 172.21.15.70:0/523082905 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feeac0b2a70 0x7feeac0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.125+0000 7feec17fa700 1 -- 172.21.15.70:0/523082905 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7feeb40e28a0 con 0x7feec41037f0 2024-06-04T05:02:26.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.125+0000 7feec37fe700 1 --2- 172.21.15.70:0/523082905 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feeac0b2a70 0x7feeac0b4f30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feec41031a0 tx=0x7feeb80073d0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:26.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feed4e35b80 1 Processor -- start 2024-06-04T05:02:26.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feed4e35b80 1 -- start start 2024-06-04T05:02:26.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feed4e35b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e741860 0x56063e741c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feed4e35b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e745940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feed4e35b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e7463a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feed4e35b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56063e6a0c90 con 0x56063e745f60 2024-06-04T05:02:26.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feed4e35b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56063e690830 con 0x56063e741860 2024-06-04T05:02:26.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feed4e35b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56063e694870 con 0x56063e742210 2024-06-04T05:02:26.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feec37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e745940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feec8b3d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e7463a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feec37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e745940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:58968/0 (socket says 172.21.15.70:58968) 2024-06-04T05:02:26.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feec37fe700 1 -- 172.21.15.70:0/3589238596 learned_addr learned my addr 172.21.15.70:0/3589238596 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:26.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feec3fff700 1 --2- 172.21.15.70:0/3589238596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e741860 0x56063e741c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feec37fe700 1 -- 172.21.15.70:0/3589238596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e741860 msgr2=0x56063e741c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:26.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feec37fe700 1 --2- 172.21.15.70:0/3589238596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e741860 0x56063e741c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feec37fe700 1 -- 172.21.15.70:0/3589238596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 msgr2=0x56063e7463a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:26.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feec37fe700 1 --2- 172.21.15.70:0/3589238596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e7463a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.126+0000 7feec37fe700 1 -- 172.21.15.70:0/3589238596 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56063e5e08d0 con 0x56063e742210 2024-06-04T05:02:26.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.127+0000 7feec37fe700 1 --2- 172.21.15.70:0/3589238596 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e745940 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7feeb8010e40 tx=0x7feeb8013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=315eae1eb878a641 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:26.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.127+0000 7feeaa7fc700 1 -- 172.21.15.70:0/3589238596 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb8024070 con 0x56063e742210 2024-06-04T05:02:26.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.127+0000 7feeaa7fc700 1 -- 172.21.15.70:0/3589238596 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feeb8015070 con 0x56063e742210 2024-06-04T05:02:26.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.127+0000 7feeaa7fc700 1 -- 172.21.15.70:0/3589238596 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb80117c0 con 0x56063e742210 2024-06-04T05:02:26.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.127+0000 7feed4e35b80 1 -- 172.21.15.70:0/3589238596 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 msgr2=0x56063e745940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:26.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.127+0000 7feed4e35b80 1 --2- 172.21.15.70:0/3589238596 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e745940 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7feeb8010e40 tx=0x7feeb8013000 comp rx=0 tx=0).stop 2024-06-04T05:02:26.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.128+0000 7feed4e35b80 1 -- 172.21.15.70:0/3589238596 shutdown_connections 2024-06-04T05:02:26.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.128+0000 7feed4e35b80 1 --2- 172.21.15.70:0/3589238596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e7463a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.128+0000 7feed4e35b80 1 --2- 172.21.15.70:0/3589238596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e741860 0x56063e741c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.128+0000 7feed4e35b80 1 --2- 172.21.15.70:0/3589238596 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e745940 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.128+0000 7feed4e35b80 1 -- 172.21.15.70:0/3589238596 >> 172.21.15.70:0/3589238596 conn(0x56063e66f080 msgr2=0x56063e73be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.128+0000 7feed4e35b80 1 -- 172.21.15.70:0/3589238596 shutdown_connections 2024-06-04T05:02:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.128+0000 7feed4e35b80 1 -- 172.21.15.70:0/3589238596 wait complete. 2024-06-04T05:02:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.128+0000 7feed4e35b80 1 -- start start 2024-06-04T05:02:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.128+0000 7feed4e35b80 1 Processor -- start 2024-06-04T05:02:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feed4e35b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e741860 0x56063e74d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feed4e35b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e742210 0x56063e74a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feed4e35b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e74ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feed4e35b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56063e596fa0 con 0x56063e745f60 2024-06-04T05:02:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feed4e35b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56063e6327c0 con 0x56063e742210 2024-06-04T05:02:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feed4e35b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56063e74e120 con 0x56063e741860 2024-06-04T05:02:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feed4e35b80 10 client.0 _ll_register_callbacks cb 0x56063e62b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feec3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e741860 0x56063e74d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feec3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e741860 0x56063e74d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:58982/0 (socket says 172.21.15.70:58982) 2024-06-04T05:02:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feec3fff700 1 -- 172.21.15.70:0/974463992 learned_addr learned my addr 172.21.15.70:0/974463992 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feec8b3d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e74ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feec37fe700 1 --2- 172.21.15.70:0/974463992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e742210 0x56063e74a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feec3fff700 1 -- 172.21.15.70:0/974463992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e742210 msgr2=0x56063e74a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feec3fff700 1 --2- 172.21.15.70:0/974463992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e742210 0x56063e74a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feec3fff700 1 -- 172.21.15.70:0/974463992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 msgr2=0x56063e74ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feec3fff700 1 --2- 172.21.15.70:0/974463992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e74ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feec3fff700 1 -- 172.21.15.70:0/974463992 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56063e62f690 con 0x56063e741860 2024-06-04T05:02:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feec37fe700 1 --2- 172.21.15.70:0/974463992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e742210 0x56063e74a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:02:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feec3fff700 1 --2- 172.21.15.70:0/974463992 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e741860 0x56063e74d440 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7feeb40a9b10 tx=0x7feeb40b2320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b450b5fa6c76900f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.129+0000 7feea37fe700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:02:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.130+0000 7feea37fe700 1 -- 172.21.15.70:0/974463992 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb4016540 con 0x56063e741860 2024-06-04T05:02:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.130+0000 7feed4e35b80 1 -- 172.21.15.70:0/974463992 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feeb8015070 con 0x56063e741860 2024-06-04T05:02:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.130+0000 7feea37fe700 1 -- 172.21.15.70:0/974463992 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7feeb40a9de0 con 0x56063e741860 2024-06-04T05:02:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.130+0000 7feea37fe700 1 -- 172.21.15.70:0/974463992 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feeb40a9b80 con 0x56063e741860 2024-06-04T05:02:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.130+0000 7feed4e35b80 10 client.26242 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.130+0000 7feed4e35b80 1 -- 172.21.15.70:0/974463992 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56063e758960 con 0x56063e741860 2024-06-04T05:02:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.130+0000 7feea37fe700 1 -- 172.21.15.70:0/974463992 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feeb8015070 con 0x56063e741860 2024-06-04T05:02:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.130+0000 7feea37fe700 1 -- 172.21.15.70:0/974463992 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7feeb40b2550 con 0x56063e741860 2024-06-04T05:02:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.130+0000 7feed4e35b80 10 client.26242 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.130+0000 7feed4e35b80 10 client.26242 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:02:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.130+0000 7feed4e35b80 20 client.26242 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.130+0000 7feed4e35b80 1 --2- 172.21.15.70:0/974463992 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56063e75aef0 0x56063e75d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.130+0000 7feed4e35b80 4 client.26242 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:02:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.130+0000 7feed4e35b80 1 -- 172.21.15.70:0/974463992 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56063e70d9e0 con 0x56063e75aef0 2024-06-04T05:02:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.131+0000 7feec37fe700 1 --2- 172.21.15.70:0/974463992 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56063e75aef0 0x56063e75d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.131+0000 7feea37fe700 10 client.26242 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:02:26.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.131+0000 7feec37fe700 1 --2- 172.21.15.70:0/974463992 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56063e75aef0 0x56063e75d2e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8ce1ea01540ec4fd server_cookie=20926e4a58c46419 in_seq=0 out_seq=0 2024-06-04T05:02:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.132+0000 7feea37fe700 1 -- 172.21.15.70:0/974463992 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56063e70d9e0 con 0x56063e75aef0 2024-06-04T05:02:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.132+0000 7feea37fe700 10 client.26242 handle_command_reply: tid=0 2024-06-04T05:02:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.132+0000 7feed4e35b80 1 client.26242 shutdown 2024-06-04T05:02:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.132+0000 7feed4e35b80 1 -- 172.21.15.70:0/974463992 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e741860 msgr2=0x56063e74d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.132+0000 7feed4e35b80 1 --2- 172.21.15.70:0/974463992 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e741860 0x56063e74d440 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7feeb40a9b10 tx=0x7feeb40b2320 comp rx=0 tx=0).stop 2024-06-04T05:02:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.132+0000 7feed4e35b80 1 -- 172.21.15.70:0/974463992 shutdown_connections 2024-06-04T05:02:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.132+0000 7feed4e35b80 1 --2- 172.21.15.70:0/974463992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e74ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.132+0000 7feed4e35b80 1 --2- 172.21.15.70:0/974463992 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56063e75aef0 0x56063e75d2e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.132+0000 7feed4e35b80 1 --2- 172.21.15.70:0/974463992 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e741860 0x56063e74d440 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.132+0000 7feed4e35b80 1 --2- 172.21.15.70:0/974463992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e742210 0x56063e74a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.132+0000 7feed4e35b80 1 -- 172.21.15.70:0/974463992 >> 172.21.15.70:0/974463992 conn(0x56063e66f080 msgr2=0x56063e73b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.132+0000 7feea37fe700 0 client.26242 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:02:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.132+0000 7feed4e35b80 1 -- 172.21.15.70:0/974463992 shutdown_connections 2024-06-04T05:02:26.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.132+0000 7feed4e35b80 1 -- 172.21.15.70:0/974463992 wait complete. 2024-06-04T05:02:26.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.132+0000 7feed4e35b80 20 client.26242 trim_cache size 0 max 16384 2024-06-04T05:02:26.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.148+0000 7feed4e35b80 1 Processor -- start 2024-06-04T05:02:26.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feed4e35b80 1 -- start start 2024-06-04T05:02:26.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feed4e35b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e741860 0x56063e744fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feed4e35b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e73bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feed4e35b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e762b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feed4e35b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7feec41abb00 con 0x56063e745f60 2024-06-04T05:02:26.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feed4e35b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feec41ad640 con 0x56063e741860 2024-06-04T05:02:26.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feed4e35b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7feec41ac1d0 con 0x56063e742210 2024-06-04T05:02:26.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feec37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e73bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feec8b3d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e762b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feec37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e73bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:58984/0 (socket says 172.21.15.70:58984) 2024-06-04T05:02:26.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feec8b3d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e762b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45360/0 (socket says 172.21.15.70:45360) 2024-06-04T05:02:26.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feec37fe700 1 -- 172.21.15.70:0/234046410 learned_addr learned my addr 172.21.15.70:0/234046410 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:26.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feec3fff700 1 --2- 172.21.15.70:0/234046410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e741860 0x56063e744fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feec37fe700 1 -- 172.21.15.70:0/234046410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e741860 msgr2=0x56063e744fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:26.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feec37fe700 1 --2- 172.21.15.70:0/234046410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e741860 0x56063e744fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feec37fe700 1 -- 172.21.15.70:0/234046410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 msgr2=0x56063e762b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:26.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feec37fe700 1 --2- 172.21.15.70:0/234046410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e762b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feec37fe700 1 -- 172.21.15.70:0/234046410 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56063e74b680 con 0x56063e742210 2024-06-04T05:02:26.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feec8b3d700 1 --2- 172.21.15.70:0/234046410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e762b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:26.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.149+0000 7feec3fff700 1 --2- 172.21.15.70:0/234046410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e741860 0x56063e744fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:02:26.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.150+0000 7feec37fe700 1 --2- 172.21.15.70:0/234046410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e73bc90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7feeb8004310 tx=0x7feeb8005240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8be0c766753ac91 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:26.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.150+0000 7feea9ffb700 1 -- 172.21.15.70:0/234046410 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb8024070 con 0x56063e742210 2024-06-04T05:02:26.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.150+0000 7feea9ffb700 1 -- 172.21.15.70:0/234046410 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feeb8015040 con 0x56063e742210 2024-06-04T05:02:26.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.150+0000 7feea9ffb700 1 -- 172.21.15.70:0/234046410 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb802d6b0 con 0x56063e742210 2024-06-04T05:02:26.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.150+0000 7feed4e35b80 1 -- 172.21.15.70:0/234046410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 msgr2=0x56063e73bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:26.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.150+0000 7feed4e35b80 1 --2- 172.21.15.70:0/234046410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e73bc90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7feeb8004310 tx=0x7feeb8005240 comp rx=0 tx=0).stop 2024-06-04T05:02:26.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.151+0000 7feed4e35b80 1 -- 172.21.15.70:0/234046410 shutdown_connections 2024-06-04T05:02:26.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.151+0000 7feed4e35b80 1 --2- 172.21.15.70:0/234046410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e762b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.151+0000 7feed4e35b80 1 --2- 172.21.15.70:0/234046410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e741860 0x56063e744fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.151+0000 7feed4e35b80 1 --2- 172.21.15.70:0/234046410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e73bc90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.151+0000 7feed4e35b80 1 -- 172.21.15.70:0/234046410 >> 172.21.15.70:0/234046410 conn(0x56063e66f080 msgr2=0x56063e7448b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:26.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.151+0000 7feed4e35b80 1 -- 172.21.15.70:0/234046410 shutdown_connections 2024-06-04T05:02:26.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.151+0000 7feed4e35b80 1 -- 172.21.15.70:0/234046410 wait complete. 2024-06-04T05:02:26.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.151+0000 7feed4e35b80 1 -- start start 2024-06-04T05:02:26.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.151+0000 7feed4e35b80 1 Processor -- start 2024-06-04T05:02:26.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.151+0000 7feed4e35b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e741860 0x56063e753bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.151+0000 7feed4e35b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e750f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.151+0000 7feed4e35b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e7516b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feed4e35b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7feec41acf70 con 0x56063e745f60 2024-06-04T05:02:26.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feed4e35b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feec41ac8a0 con 0x56063e741860 2024-06-04T05:02:26.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feed4e35b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7feec41a9aa0 con 0x56063e742210 2024-06-04T05:02:26.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feed4e35b80 10 client.0 _ll_register_callbacks cb 0x56063e62b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:26.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feec37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e750f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feec8b3d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e7516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feec37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e750f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:58996/0 (socket says 172.21.15.70:58996) 2024-06-04T05:02:26.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feec37fe700 1 -- 172.21.15.70:0/3179537727 learned_addr learned my addr 172.21.15.70:0/3179537727 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:26.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feec3fff700 1 --2- 172.21.15.70:0/3179537727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e741860 0x56063e753bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feec37fe700 1 -- 172.21.15.70:0/3179537727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e741860 msgr2=0x56063e753bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:26.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feec37fe700 1 --2- 172.21.15.70:0/3179537727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e741860 0x56063e753bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feec37fe700 1 -- 172.21.15.70:0/3179537727 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 msgr2=0x56063e7516b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:26.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feec37fe700 1 --2- 172.21.15.70:0/3179537727 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e7516b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feec37fe700 1 -- 172.21.15.70:0/3179537727 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56063e7c4a40 con 0x56063e742210 2024-06-04T05:02:26.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feec37fe700 1 --2- 172.21.15.70:0/3179537727 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e750f70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7feeb8010e10 tx=0x7feeb8005b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6775199353f14fdb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:26.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feea37fe700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:02:26.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feea37fe700 1 -- 172.21.15.70:0/3179537727 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb8024040 con 0x56063e742210 2024-06-04T05:02:26.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feea37fe700 1 -- 172.21.15.70:0/3179537727 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7feeb8005330 con 0x56063e742210 2024-06-04T05:02:26.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.152+0000 7feed4e35b80 1 -- 172.21.15.70:0/3179537727 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feeb8015040 con 0x56063e742210 2024-06-04T05:02:26.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.153+0000 7feea37fe700 1 -- 172.21.15.70:0/3179537727 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feeb8005530 con 0x56063e742210 2024-06-04T05:02:26.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.153+0000 7feed4e35b80 10 client.26248 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:26.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.153+0000 7feed4e35b80 1 -- 172.21.15.70:0/3179537727 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56063e7c5bc0 con 0x56063e742210 2024-06-04T05:02:26.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.153+0000 7feea37fe700 1 -- 172.21.15.70:0/3179537727 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feeb8015040 con 0x56063e742210 2024-06-04T05:02:26.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.153+0000 7feea37fe700 1 -- 172.21.15.70:0/3179537727 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7feeb802a040 con 0x56063e742210 2024-06-04T05:02:26.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.153+0000 7feed4e35b80 10 client.26248 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:26.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.153+0000 7feed4e35b80 10 client.26248 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:02:26.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.153+0000 7feed4e35b80 20 client.26248 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:26.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.153+0000 7feed4e35b80 1 --2- 172.21.15.70:0/3179537727 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56063e7c6140 0x56063e7c8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:26.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.153+0000 7feed4e35b80 4 client.26248 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:02:26.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.153+0000 7feed4e35b80 1 -- 172.21.15.70:0/3179537727 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56063e6a2be0 con 0x56063e7c6140 2024-06-04T05:02:26.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.153+0000 7feec3fff700 1 --2- 172.21.15.70:0/3179537727 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56063e7c6140 0x56063e7c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:26.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.154+0000 7feea37fe700 10 client.26248 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:02:26.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.154+0000 7feec3fff700 1 --2- 172.21.15.70:0/3179537727 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56063e7c6140 0x56063e7c8560 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5b7b379cff14ced1 server_cookie=e1e3e6b8eaf4d4ac in_seq=0 out_seq=0 2024-06-04T05:02:26.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.154+0000 7feea37fe700 1 -- 172.21.15.70:0/3179537727 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56063e6a2be0 con 0x56063e7c6140 2024-06-04T05:02:26.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.154+0000 7feea37fe700 10 client.26248 handle_command_reply: tid=0 2024-06-04T05:02:26.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.154+0000 7feed4e35b80 1 client.26248 shutdown 2024-06-04T05:02:26.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.154+0000 7feed4e35b80 1 -- 172.21.15.70:0/3179537727 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 msgr2=0x56063e750f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:26.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.154+0000 7feed4e35b80 1 --2- 172.21.15.70:0/3179537727 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e750f70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7feeb8010e10 tx=0x7feeb8005b90 comp rx=0 tx=0).stop 2024-06-04T05:02:26.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.154+0000 7feed4e35b80 1 -- 172.21.15.70:0/3179537727 shutdown_connections 2024-06-04T05:02:26.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.154+0000 7feed4e35b80 1 --2- 172.21.15.70:0/3179537727 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063e745f60 0x56063e7516b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.154+0000 7feed4e35b80 1 --2- 172.21.15.70:0/3179537727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56063e741860 0x56063e753bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.164 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:02:26.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.154+0000 7feed4e35b80 1 --2- 172.21.15.70:0/3179537727 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56063e7c6140 0x56063e7c8560 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.154+0000 7feed4e35b80 1 --2- 172.21.15.70:0/3179537727 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56063e742210 0x56063e750f70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.154+0000 7feed4e35b80 1 -- 172.21.15.70:0/3179537727 >> 172.21.15.70:0/3179537727 conn(0x56063e66f080 msgr2=0x56063e745140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:26.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.154+0000 7feea37fe700 0 client.26248 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:02:26.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.154+0000 7feed4e35b80 1 -- 172.21.15.70:0/3179537727 shutdown_connections 2024-06-04T05:02:26.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.154+0000 7feed4e35b80 1 -- 172.21.15.70:0/3179537727 wait complete. 2024-06-04T05:02:26.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.154+0000 7feed4e35b80 20 client.26248 trim_cache size 0 max 16384 2024-06-04T05:02:26.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.155+0000 7feeca5a0700 1 -- 172.21.15.70:0/523082905 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feeac0b2a70 msgr2=0x7feeac0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:26.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.155+0000 7feeca5a0700 1 --2- 172.21.15.70:0/523082905 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feeac0b2a70 0x7feeac0b4f30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feec41031a0 tx=0x7feeb80073d0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.155+0000 7feeca5a0700 1 -- 172.21.15.70:0/523082905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec41037f0 msgr2=0x7feec419f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:26.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.155+0000 7feeca5a0700 1 --2- 172.21.15.70:0/523082905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec41037f0 0x7feec419f6d0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7feeb400e9c0 tx=0x7feeb400ecd0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.155+0000 7feeca5a0700 1 -- 172.21.15.70:0/523082905 shutdown_connections 2024-06-04T05:02:26.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.155+0000 7feeca5a0700 1 --2- 172.21.15.70:0/523082905 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feeac0b2a70 0x7feeac0b4f30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.155+0000 7feeca5a0700 1 --2- 172.21.15.70:0/523082905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feec41088f0 0x7feec4199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.155+0000 7feeca5a0700 1 --2- 172.21.15.70:0/523082905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec41037f0 0x7feec419f6d0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.155+0000 7feeca5a0700 1 --2- 172.21.15.70:0/523082905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feec41041a0 0x7feec419fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:26.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.155+0000 7feeca5a0700 1 -- 172.21.15.70:0/523082905 >> 172.21.15.70:0/523082905 conn(0x7feec407a6a0 msgr2=0x7feec4101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:26.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.155+0000 7feeca5a0700 1 -- 172.21.15.70:0/523082905 shutdown_connections 2024-06-04T05:02:26.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:26.155+0000 7feeca5a0700 1 -- 172.21.15.70:0/523082905 wait complete. 2024-06-04T05:02:26.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 76838 2.11 MB/sec execute 79 sec latency 16.995 ms 2024-06-04T05:02:27.127 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:02:27.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:27 smithi070 ceph-mon[32631]: pgmap v161: 97 pgs: 97 active+clean; 55 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 1.9 MiB/s wr, 205 op/s 2024-06-04T05:02:27.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:27 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:02:27.459 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:02:27.523 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:27 smithi098 ceph-mon[35569]: pgmap v161: 97 pgs: 97 active+clean; 55 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 1.9 MiB/s wr, 205 op/s 2024-06-04T05:02:27.523 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:27 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:02:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:27 smithi112 ceph-mon[36453]: pgmap v161: 97 pgs: 97 active+clean; 55 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 1.9 MiB/s wr, 205 op/s 2024-06-04T05:02:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:27 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:02:27.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 77319 2.14 MB/sec execute 80 sec latency 15.942 ms 2024-06-04T05:02:28.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:28 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:02:28.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:28 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:02:28.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:28 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:02:28.591 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:28 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:02:28.591 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:28 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:02:28.592 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:28 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:02:28.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:28 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:02:28.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:28 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:02:28.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:28 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:02:28.720 INFO:tasks.workunit.client.0.smithi070.stdout: 1 77877 2.16 MB/sec execute 81 sec latency 56.774 ms 2024-06-04T05:02:29.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 78342 2.18 MB/sec execute 82 sec latency 61.918 ms 2024-06-04T05:02:29.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:29 smithi098 ceph-mon[35569]: pgmap v162: 97 pgs: 97 active+clean; 55 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-06-04T05:02:29.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:29 smithi070 ceph-mon[32631]: pgmap v162: 97 pgs: 97 active+clean; 55 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-06-04T05:02:29.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:29 smithi112 ceph-mon[36453]: pgmap v162: 97 pgs: 97 active+clean; 55 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-06-04T05:02:29.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.449+0000 7ff050e93700 1 -- 172.21.15.70:0/2893027113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff04c102440 msgr2=0x7ff04c1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:29.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.449+0000 7ff050e93700 1 --2- 172.21.15.70:0/2893027113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff04c102440 0x7ff04c1028c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff040009990 tx=0x7ff040009ca0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.449+0000 7ff050e93700 1 -- 172.21.15.70:0/2893027113 shutdown_connections 2024-06-04T05:02:29.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.449+0000 7ff050e93700 1 --2- 172.21.15.70:0/2893027113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff04c102e00 0x7ff04c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.449+0000 7ff050e93700 1 --2- 172.21.15.70:0/2893027113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff04c108520 0x7ff04c108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.449+0000 7ff050e93700 1 --2- 172.21.15.70:0/2893027113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff04c102440 0x7ff04c1028c0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.449+0000 7ff050e93700 1 -- 172.21.15.70:0/2893027113 >> 172.21.15.70:0/2893027113 conn(0x7ff04c0793f0 msgr2=0x7ff04c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:29.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.449+0000 7ff050e93700 1 -- 172.21.15.70:0/2893027113 shutdown_connections 2024-06-04T05:02:29.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.449+0000 7ff050e93700 1 -- 172.21.15.70:0/2893027113 wait complete. 2024-06-04T05:02:29.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.450+0000 7ff050e93700 1 Processor -- start 2024-06-04T05:02:29.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.450+0000 7ff050e93700 1 -- start start 2024-06-04T05:02:29.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.450+0000 7ff050e93700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff04c102440 0x7ff04c19a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.450+0000 7ff050e93700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff04c102e00 0x7ff04c19aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.450+0000 7ff050e93700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff04c108520 0x7ff04c1a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.450+0000 7ff050e93700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c1124a0 con 0x7ff04c102440 2024-06-04T05:02:29.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.450+0000 7ff050e93700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c112320 con 0x7ff04c102e00 2024-06-04T05:02:29.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.450+0000 7ff050e93700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c112620 con 0x7ff04c108520 2024-06-04T05:02:29.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.450+0000 7ff04a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff04c102440 0x7ff04c19a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:29.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.450+0000 7ff04a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff04c102440 0x7ff04c19a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45388/0 (socket says 172.21.15.70:45388) 2024-06-04T05:02:29.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.450+0000 7ff04a59c700 1 -- 172.21.15.70:0/1528357687 learned_addr learned my addr 172.21.15.70:0/1528357687 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:29.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.450+0000 7ff04ad9d700 1 --2- 172.21.15.70:0/1528357687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff04c108520 0x7ff04c1a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:29.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.450+0000 7ff049d9b700 1 --2- 172.21.15.70:0/1528357687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff04c102e00 0x7ff04c19aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:29.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.451+0000 7ff04a59c700 1 -- 172.21.15.70:0/1528357687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff04c108520 msgr2=0x7ff04c1a5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:29.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.451+0000 7ff04a59c700 1 --2- 172.21.15.70:0/1528357687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff04c108520 0x7ff04c1a5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.451+0000 7ff04a59c700 1 -- 172.21.15.70:0/1528357687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff04c102e00 msgr2=0x7ff04c19aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:29.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.451+0000 7ff04a59c700 1 --2- 172.21.15.70:0/1528357687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff04c102e00 0x7ff04c19aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.451+0000 7ff04a59c700 1 -- 172.21.15.70:0/1528357687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff04c1a5f60 con 0x7ff04c102440 2024-06-04T05:02:29.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.451+0000 7ff04a59c700 1 --2- 172.21.15.70:0/1528357687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff04c102440 0x7ff04c19a9a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ff03c00e9c0 tx=0x7ff03c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:29.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.451+0000 7ff03b7fe700 1 -- 172.21.15.70:0/1528357687 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c00cbd0 con 0x7ff04c102440 2024-06-04T05:02:29.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.451+0000 7ff050e93700 1 -- 172.21.15.70:0/1528357687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff04c1a6190 con 0x7ff04c102440 2024-06-04T05:02:29.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.451+0000 7ff03b7fe700 1 -- 172.21.15.70:0/1528357687 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff03c00ee60 con 0x7ff04c102440 2024-06-04T05:02:29.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.451+0000 7ff03b7fe700 1 -- 172.21.15.70:0/1528357687 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0054b0 con 0x7ff04c102440 2024-06-04T05:02:29.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.452+0000 7ff050e93700 1 -- 172.21.15.70:0/1528357687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff04c1a6650 con 0x7ff04c102440 2024-06-04T05:02:29.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.454+0000 7ff03b7fe700 1 -- 172.21.15.70:0/1528357687 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff03c004750 con 0x7ff04c102440 2024-06-04T05:02:29.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.455+0000 7ff03b7fe700 1 --2- 172.21.15.70:0/1528357687 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff0340b29f0 0x7ff0340b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.455+0000 7ff049d9b700 1 --2- 172.21.15.70:0/1528357687 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff0340b29f0 0x7ff0340b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:29.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.455+0000 7ff03b7fe700 1 -- 172.21.15.70:0/1528357687 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ff03c0e2860 con 0x7ff04c102440 2024-06-04T05:02:29.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.455+0000 7ff049d9b700 1 --2- 172.21.15.70:0/1528357687 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff0340b29f0 0x7ff0340b4eb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff04c06bd80 tx=0x7ff040005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:29.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.456+0000 7ff05b728b80 1 Processor -- start 2024-06-04T05:02:29.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.456+0000 7ff05b728b80 1 -- start start 2024-06-04T05:02:29.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.456+0000 7ff05b728b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406b4320 0x564e406b4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.456+0000 7ff05b728b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406ae8a0 0x564e406aed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.456+0000 7ff05b728b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406af260 0x564e406bb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.456+0000 7ff05b728b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564e4060cd70 con 0x564e406af260 2024-06-04T05:02:29.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.456+0000 7ff05b728b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564e405fc910 con 0x564e406ae8a0 2024-06-04T05:02:29.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.456+0000 7ff05b728b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564e40600950 con 0x564e406b4320 2024-06-04T05:02:29.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.456+0000 7ff04a59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406b4320 0x564e406b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:29.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.456+0000 7ff04ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406af260 0x564e406bb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:29.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.456+0000 7ff04a59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406b4320 0x564e406b4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59034/0 (socket says 172.21.15.70:59034) 2024-06-04T05:02:29.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.456+0000 7ff04a59c700 1 -- 172.21.15.70:0/3878458764 learned_addr learned my addr 172.21.15.70:0/3878458764 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:29.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.456+0000 7ff049d9b700 1 --2- 172.21.15.70:0/3878458764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406ae8a0 0x564e406aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:29.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.457+0000 7ff04a59c700 1 -- 172.21.15.70:0/3878458764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406ae8a0 msgr2=0x564e406aed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:29.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.457+0000 7ff04a59c700 1 --2- 172.21.15.70:0/3878458764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406ae8a0 0x564e406aed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.457+0000 7ff04a59c700 1 -- 172.21.15.70:0/3878458764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406af260 msgr2=0x564e406bb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:29.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.457+0000 7ff04a59c700 1 --2- 172.21.15.70:0/3878458764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406af260 0x564e406bb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.457+0000 7ff04a59c700 1 -- 172.21.15.70:0/3878458764 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e405a5fb0 con 0x564e406b4320 2024-06-04T05:02:29.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.457+0000 7ff04a59c700 1 --2- 172.21.15.70:0/3878458764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406b4320 0x564e406b4700 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0a9ad0 tx=0x7ff03c0a9380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7a31ffe2e8996b0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:29.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.457+0000 7ff038ff9700 1 -- 172.21.15.70:0/3878458764 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0c3070 con 0x564e406b4320 2024-06-04T05:02:29.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.457+0000 7ff038ff9700 1 -- 172.21.15.70:0/3878458764 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff03c0b22f0 con 0x564e406b4320 2024-06-04T05:02:29.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.457+0000 7ff038ff9700 1 -- 172.21.15.70:0/3878458764 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0a5dd0 con 0x564e406b4320 2024-06-04T05:02:29.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.458+0000 7ff05b728b80 1 -- 172.21.15.70:0/3878458764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406b4320 msgr2=0x564e406b4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:29.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.458+0000 7ff05b728b80 1 --2- 172.21.15.70:0/3878458764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406b4320 0x564e406b4700 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0a9ad0 tx=0x7ff03c0a9380 comp rx=0 tx=0).stop 2024-06-04T05:02:29.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.458+0000 7ff05b728b80 1 -- 172.21.15.70:0/3878458764 shutdown_connections 2024-06-04T05:02:29.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.458+0000 7ff05b728b80 1 --2- 172.21.15.70:0/3878458764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406af260 0x564e406bb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.458+0000 7ff05b728b80 1 --2- 172.21.15.70:0/3878458764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406b4320 0x564e406b4700 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.458+0000 7ff05b728b80 1 --2- 172.21.15.70:0/3878458764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406ae8a0 0x564e406aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.458+0000 7ff05b728b80 1 -- 172.21.15.70:0/3878458764 >> 172.21.15.70:0/3878458764 conn(0x564e405da240 msgr2=0x564e406a93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:29.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.458+0000 7ff05b728b80 1 -- 172.21.15.70:0/3878458764 shutdown_connections 2024-06-04T05:02:29.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.458+0000 7ff05b728b80 1 -- 172.21.15.70:0/3878458764 wait complete. 2024-06-04T05:02:29.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.458+0000 7ff05b728b80 1 -- start start 2024-06-04T05:02:29.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.458+0000 7ff05b728b80 1 Processor -- start 2024-06-04T05:02:29.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff05b728b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406ae8a0 0x564e406bada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff05b728b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406af260 0x564e406b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff05b728b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406b4320 0x564e406b85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff05b728b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564e40502fa0 con 0x564e406b4320 2024-06-04T05:02:29.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff05b728b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564e40564da0 con 0x564e406ae8a0 2024-06-04T05:02:29.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff05b728b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564e406bb6e0 con 0x564e406af260 2024-06-04T05:02:29.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff04a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406ae8a0 0x564e406bada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:29.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff049d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406af260 0x564e406b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:29.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff04a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406ae8a0 0x564e406bada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:35216/0 (socket says 172.21.15.70:35216) 2024-06-04T05:02:29.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff049d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406af260 0x564e406b80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59042/0 (socket says 172.21.15.70:59042) 2024-06-04T05:02:29.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff04a59c700 1 -- 172.21.15.70:0/3693864180 learned_addr learned my addr 172.21.15.70:0/3693864180 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:29.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff05b728b80 10 client.0 _ll_register_callbacks cb 0x564e40597950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:29.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff04ad9d700 1 --2- 172.21.15.70:0/3693864180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406b4320 0x564e406b85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:29.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff04a59c700 1 -- 172.21.15.70:0/3693864180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406af260 msgr2=0x564e406b80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:29.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff04a59c700 1 --2- 172.21.15.70:0/3693864180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406af260 0x564e406b80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff04a59c700 1 -- 172.21.15.70:0/3693864180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406b4320 msgr2=0x564e406b85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:29.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff04a59c700 1 --2- 172.21.15.70:0/3693864180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406b4320 0x564e406b85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff04a59c700 1 -- 172.21.15.70:0/3693864180 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e4054c8d0 con 0x564e406ae8a0 2024-06-04T05:02:29.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff049d9b700 1 --2- 172.21.15.70:0/3693864180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406af260 0x564e406b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:29.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff04a59c700 1 --2- 172.21.15.70:0/3693864180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406ae8a0 0x564e406bada0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff03c009fd0 tx=0x7ff03c0dff50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=607bf0354fc03988 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:29.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.459+0000 7ff031ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:02:29.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.460+0000 7ff031ffb700 1 -- 172.21.15.70:0/3693864180 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c06d070 con 0x564e406ae8a0 2024-06-04T05:02:29.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.460+0000 7ff031ffb700 1 -- 172.21.15.70:0/3693864180 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ff03c0a9da0 con 0x564e406ae8a0 2024-06-04T05:02:29.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.460+0000 7ff05b728b80 1 -- 172.21.15.70:0/3693864180 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff03c0b22f0 con 0x564e406ae8a0 2024-06-04T05:02:29.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.460+0000 7ff031ffb700 1 -- 172.21.15.70:0/3693864180 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff03c0c3070 con 0x564e406ae8a0 2024-06-04T05:02:29.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.460+0000 7ff05b728b80 10 client.26201 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:29.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.460+0000 7ff05b728b80 1 -- 172.21.15.70:0/3693864180 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564e405b6760 con 0x564e406ae8a0 2024-06-04T05:02:29.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.460+0000 7ff031ffb700 1 -- 172.21.15.70:0/3693864180 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff03c0b22f0 con 0x564e406ae8a0 2024-06-04T05:02:29.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.460+0000 7ff031ffb700 1 -- 172.21.15.70:0/3693864180 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ff03c06f2f0 con 0x564e406ae8a0 2024-06-04T05:02:29.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.460+0000 7ff05b728b80 10 client.26201 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:29.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.460+0000 7ff05b728b80 10 client.26201 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:02:29.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.460+0000 7ff05b728b80 20 client.26201 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:29.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.460+0000 7ff05b728b80 1 --2- 172.21.15.70:0/3693864180 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x564e406c8110 0x564e406ca500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.460+0000 7ff05b728b80 4 client.26201 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:02:29.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.460+0000 7ff05b728b80 1 -- 172.21.15.70:0/3693864180 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e4060ecc0 con 0x564e406c8110 2024-06-04T05:02:29.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.461+0000 7ff049d9b700 1 --2- 172.21.15.70:0/3693864180 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x564e406c8110 0x564e406ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:29.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.461+0000 7ff031ffb700 10 client.26201 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:02:29.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.461+0000 7ff049d9b700 1 --2- 172.21.15.70:0/3693864180 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x564e406c8110 0x564e406ca500 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=90d4ddcccb0a645d server_cookie=539f517d32c22327 in_seq=0 out_seq=0 2024-06-04T05:02:29.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.461+0000 7ff031ffb700 1 -- 172.21.15.70:0/3693864180 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564e4060ecc0 con 0x564e406c8110 2024-06-04T05:02:29.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.461+0000 7ff031ffb700 10 client.26201 handle_command_reply: tid=0 2024-06-04T05:02:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.462+0000 7ff05b728b80 1 client.26201 shutdown 2024-06-04T05:02:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.462+0000 7ff05b728b80 1 -- 172.21.15.70:0/3693864180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406ae8a0 msgr2=0x564e406bada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.462+0000 7ff05b728b80 1 --2- 172.21.15.70:0/3693864180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406ae8a0 0x564e406bada0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff03c009fd0 tx=0x7ff03c0dff50 comp rx=0 tx=0).stop 2024-06-04T05:02:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.462+0000 7ff05b728b80 1 -- 172.21.15.70:0/3693864180 shutdown_connections 2024-06-04T05:02:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.462+0000 7ff05b728b80 1 --2- 172.21.15.70:0/3693864180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406b4320 0x564e406b85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.462+0000 7ff05b728b80 1 --2- 172.21.15.70:0/3693864180 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x564e406c8110 0x564e406ca500 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.462+0000 7ff05b728b80 1 --2- 172.21.15.70:0/3693864180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406ae8a0 0x564e406bada0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.462+0000 7ff05b728b80 1 --2- 172.21.15.70:0/3693864180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406af260 0x564e406b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.462+0000 7ff031ffb700 0 client.26201 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:02:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.462+0000 7ff05b728b80 1 -- 172.21.15.70:0/3693864180 >> 172.21.15.70:0/3693864180 conn(0x564e405da240 msgr2=0x564e406a8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.462+0000 7ff05b728b80 1 -- 172.21.15.70:0/3693864180 shutdown_connections 2024-06-04T05:02:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.462+0000 7ff05b728b80 1 -- 172.21.15.70:0/3693864180 wait complete. 2024-06-04T05:02:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.462+0000 7ff05b728b80 20 client.26201 trim_cache size 0 max 16384 2024-06-04T05:02:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.478+0000 7ff05b728b80 1 Processor -- start 2024-06-04T05:02:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.478+0000 7ff05b728b80 1 -- start start 2024-06-04T05:02:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.479+0000 7ff05b728b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406ae8a0 0x564e406aec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.479+0000 7ff05b728b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406af260 0x564e406b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.479+0000 7ff05b728b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406b4320 0x564e406a9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.479+0000 7ff05b728b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c1add90 con 0x564e406b4320 2024-06-04T05:02:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.479+0000 7ff05b728b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c1af8d0 con 0x564e406af260 2024-06-04T05:02:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.479+0000 7ff05b728b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c1ae460 con 0x564e406ae8a0 2024-06-04T05:02:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.479+0000 7ff04a59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406ae8a0 0x564e406aec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.479+0000 7ff04ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406b4320 0x564e406a9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.479+0000 7ff04ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406b4320 0x564e406a9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45420/0 (socket says 172.21.15.70:45420) 2024-06-04T05:02:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.479+0000 7ff04ad9d700 1 -- 172.21.15.70:0/1206743365 learned_addr learned my addr 172.21.15.70:0/1206743365 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.479+0000 7ff04a59c700 1 -- 172.21.15.70:0/1206743365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406af260 msgr2=0x564e406b38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.479+0000 7ff04a59c700 1 --2- 172.21.15.70:0/1206743365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406af260 0x564e406b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.479+0000 7ff04a59c700 1 -- 172.21.15.70:0/1206743365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406b4320 msgr2=0x564e406a9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.479+0000 7ff04a59c700 1 --2- 172.21.15.70:0/1206743365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406b4320 0x564e406a9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.479+0000 7ff04a59c700 1 -- 172.21.15.70:0/1206743365 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e406b3f40 con 0x564e406ae8a0 2024-06-04T05:02:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.479+0000 7ff04a59c700 1 --2- 172.21.15.70:0/1206743365 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406ae8a0 0x564e406aec80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0cc010 tx=0x7ff03c0a9770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=317351d3ce35d94e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.480+0000 7ff033fff700 1 -- 172.21.15.70:0/1206743365 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c06d070 con 0x564e406ae8a0 2024-06-04T05:02:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.480+0000 7ff033fff700 1 -- 172.21.15.70:0/1206743365 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff03c0b22f0 con 0x564e406ae8a0 2024-06-04T05:02:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.480+0000 7ff033fff700 1 -- 172.21.15.70:0/1206743365 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0a9b40 con 0x564e406ae8a0 2024-06-04T05:02:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.480+0000 7ff05b728b80 1 -- 172.21.15.70:0/1206743365 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406ae8a0 msgr2=0x564e406aec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.480+0000 7ff05b728b80 1 --2- 172.21.15.70:0/1206743365 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406ae8a0 0x564e406aec80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0cc010 tx=0x7ff03c0a9770 comp rx=0 tx=0).stop 2024-06-04T05:02:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff05b728b80 1 -- 172.21.15.70:0/1206743365 shutdown_connections 2024-06-04T05:02:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff05b728b80 1 --2- 172.21.15.70:0/1206743365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406b4320 0x564e406a9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff05b728b80 1 --2- 172.21.15.70:0/1206743365 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406ae8a0 0x564e406aec80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff05b728b80 1 --2- 172.21.15.70:0/1206743365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406af260 0x564e406b38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff05b728b80 1 -- 172.21.15.70:0/1206743365 >> 172.21.15.70:0/1206743365 conn(0x564e405da240 msgr2=0x564e406ae120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff05b728b80 1 -- 172.21.15.70:0/1206743365 shutdown_connections 2024-06-04T05:02:29.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff05b728b80 1 -- 172.21.15.70:0/1206743365 wait complete. 2024-06-04T05:02:29.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff05b728b80 1 -- start start 2024-06-04T05:02:29.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff05b728b80 1 Processor -- start 2024-06-04T05:02:29.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff05b728b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406ae8a0 0x564e4072aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff05b728b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406af260 0x564e4072b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff05b728b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406b4320 0x564e406cc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff05b728b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c1af200 con 0x564e406ae8a0 2024-06-04T05:02:29.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff05b728b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c1aeb30 con 0x564e406b4320 2024-06-04T05:02:29.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff05b728b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c1abd30 con 0x564e406af260 2024-06-04T05:02:29.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff05b728b80 10 client.0 _ll_register_callbacks cb 0x564e40597950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:29.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff049d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406af260 0x564e4072b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:29.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff04a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406ae8a0 0x564e4072aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:29.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff04ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406b4320 0x564e406cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:29.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff049d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406af260 0x564e4072b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59056/0 (socket says 172.21.15.70:59056) 2024-06-04T05:02:29.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff04a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406ae8a0 0x564e4072aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45436/0 (socket says 172.21.15.70:45436) 2024-06-04T05:02:29.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.481+0000 7ff049d9b700 1 -- 172.21.15.70:0/3974407295 learned_addr learned my addr 172.21.15.70:0/3974407295 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:29.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.482+0000 7ff049d9b700 1 -- 172.21.15.70:0/3974407295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406b4320 msgr2=0x564e406cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:29.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.482+0000 7ff049d9b700 1 --2- 172.21.15.70:0/3974407295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406b4320 0x564e406cc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.482+0000 7ff049d9b700 1 -- 172.21.15.70:0/3974407295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406ae8a0 msgr2=0x564e4072aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:29.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.482+0000 7ff049d9b700 1 --2- 172.21.15.70:0/3974407295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406ae8a0 0x564e4072aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.482+0000 7ff049d9b700 1 -- 172.21.15.70:0/3974407295 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e406cca60 con 0x564e406af260 2024-06-04T05:02:29.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.482+0000 7ff049d9b700 1 --2- 172.21.15.70:0/3974407295 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406af260 0x564e4072b420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff040000c00 tx=0x7ff040009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bca29ec5be67ecc server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:29.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.482+0000 7ff031ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:02:29.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.482+0000 7ff031ffb700 1 -- 172.21.15.70:0/3974407295 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04000f660 con 0x564e406af260 2024-06-04T05:02:29.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.482+0000 7ff031ffb700 1 -- 172.21.15.70:0/3974407295 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ff04000f800 con 0x564e406af260 2024-06-04T05:02:29.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.482+0000 7ff05b728b80 1 -- 172.21.15.70:0/3974407295 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff03c0b22f0 con 0x564e406af260 2024-06-04T05:02:29.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.486+0000 7ff031ffb700 1 -- 172.21.15.70:0/3974407295 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff04000fa00 con 0x564e406af260 2024-06-04T05:02:29.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.486+0000 7ff05b728b80 10 client.26266 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:29.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.486+0000 7ff05b728b80 1 -- 172.21.15.70:0/3974407295 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564e406cdd70 con 0x564e406af260 2024-06-04T05:02:29.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.486+0000 7ff031ffb700 1 -- 172.21.15.70:0/3974407295 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff03c0b22f0 con 0x564e406af260 2024-06-04T05:02:29.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.486+0000 7ff031ffb700 1 -- 172.21.15.70:0/3974407295 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ff040021070 con 0x564e406af260 2024-06-04T05:02:29.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.486+0000 7ff05b728b80 10 client.26266 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:29.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.486+0000 7ff05b728b80 10 client.26266 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:02:29.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.486+0000 7ff05b728b80 20 client.26266 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:29.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.486+0000 7ff05b728b80 1 --2- 172.21.15.70:0/3974407295 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x564e40736f40 0x564e40739360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:29.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.486+0000 7ff05b728b80 4 client.26266 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:02:29.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.486+0000 7ff05b728b80 1 -- 172.21.15.70:0/3974407295 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e406084a0 con 0x564e40736f40 2024-06-04T05:02:29.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.487+0000 7ff04a59c700 1 --2- 172.21.15.70:0/3974407295 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x564e40736f40 0x564e40739360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:29.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.487+0000 7ff031ffb700 10 client.26266 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:02:29.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.487+0000 7ff04a59c700 1 --2- 172.21.15.70:0/3974407295 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x564e40736f40 0x564e40739360 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9799303735f6bd97 server_cookie=7e1e256761dc9007 in_seq=0 out_seq=0 2024-06-04T05:02:29.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.487+0000 7ff031ffb700 1 -- 172.21.15.70:0/3974407295 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e406084a0 con 0x564e40736f40 2024-06-04T05:02:29.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.487+0000 7ff031ffb700 10 client.26266 handle_command_reply: tid=0 2024-06-04T05:02:29.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.487+0000 7ff05b728b80 1 client.26266 shutdown 2024-06-04T05:02:29.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.488+0000 7ff05b728b80 1 -- 172.21.15.70:0/3974407295 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406af260 msgr2=0x564e4072b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:29.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.488+0000 7ff05b728b80 1 --2- 172.21.15.70:0/3974407295 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406af260 0x564e4072b420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff040000c00 tx=0x7ff040009310 comp rx=0 tx=0).stop 2024-06-04T05:02:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.488+0000 7ff05b728b80 1 -- 172.21.15.70:0/3974407295 shutdown_connections 2024-06-04T05:02:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.488+0000 7ff05b728b80 1 --2- 172.21.15.70:0/3974407295 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x564e40736f40 0x564e40739360 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.488+0000 7ff05b728b80 1 --2- 172.21.15.70:0/3974407295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e406b4320 0x564e406cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.995 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:02:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.488+0000 7ff031ffb700 0 client.26266 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:02:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.488+0000 7ff05b728b80 1 --2- 172.21.15.70:0/3974407295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564e406ae8a0 0x564e4072aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.488+0000 7ff05b728b80 1 --2- 172.21.15.70:0/3974407295 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564e406af260 0x564e4072b420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.488+0000 7ff05b728b80 1 -- 172.21.15.70:0/3974407295 >> 172.21.15.70:0/3974407295 conn(0x564e405da240 msgr2=0x564e406b1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.488+0000 7ff05b728b80 1 -- 172.21.15.70:0/3974407295 shutdown_connections 2024-06-04T05:02:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.488+0000 7ff05b728b80 1 -- 172.21.15.70:0/3974407295 wait complete. 2024-06-04T05:02:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.488+0000 7ff05b728b80 20 client.26266 trim_cache size 0 max 16384 2024-06-04T05:02:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.489+0000 7ff050e93700 1 -- 172.21.15.70:0/1528357687 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff0340b29f0 msgr2=0x7ff0340b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.489+0000 7ff050e93700 1 --2- 172.21.15.70:0/1528357687 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff0340b29f0 0x7ff0340b4eb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff04c06bd80 tx=0x7ff040005fb0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.489+0000 7ff050e93700 1 -- 172.21.15.70:0/1528357687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff04c102440 msgr2=0x7ff04c19a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.489+0000 7ff050e93700 1 --2- 172.21.15.70:0/1528357687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff04c102440 0x7ff04c19a9a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ff03c00e9c0 tx=0x7ff03c00ecd0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.489+0000 7ff050e93700 1 -- 172.21.15.70:0/1528357687 shutdown_connections 2024-06-04T05:02:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.489+0000 7ff050e93700 1 --2- 172.21.15.70:0/1528357687 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff0340b29f0 0x7ff0340b4eb0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.489+0000 7ff050e93700 1 --2- 172.21.15.70:0/1528357687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff04c108520 0x7ff04c1a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.489+0000 7ff050e93700 1 --2- 172.21.15.70:0/1528357687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff04c102440 0x7ff04c19a9a0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.489+0000 7ff050e93700 1 --2- 172.21.15.70:0/1528357687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff04c102e00 0x7ff04c19aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.489+0000 7ff050e93700 1 -- 172.21.15.70:0/1528357687 >> 172.21.15.70:0/1528357687 conn(0x7ff04c0793f0 msgr2=0x7ff04c100690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.489+0000 7ff050e93700 1 -- 172.21.15.70:0/1528357687 shutdown_connections 2024-06-04T05:02:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:29.489+0000 7ff050e93700 1 -- 172.21.15.70:0/1528357687 wait complete. 2024-06-04T05:02:30.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 78601 2.16 MB/sec execute 83 sec latency 38.966 ms 2024-06-04T05:02:31.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 78991 2.14 MB/sec execute 84 sec latency 86.563 ms 2024-06-04T05:02:31.738 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:31 smithi112 ceph-mon[36453]: pgmap v163: 97 pgs: 97 active+clean; 55 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-06-04T05:02:31.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:31 smithi098 ceph-mon[35569]: pgmap v163: 97 pgs: 97 active+clean; 55 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-06-04T05:02:31.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:31 smithi070 ceph-mon[32631]: pgmap v163: 97 pgs: 97 active+clean; 55 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-06-04T05:02:32.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 79498 2.17 MB/sec execute 85 sec latency 54.245 ms 2024-06-04T05:02:33.035 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:02:33.299 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:02:33.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:33 smithi070 ceph-mon[32631]: pgmap v164: 97 pgs: 97 active+clean; 59 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 1.8 MiB/s wr, 196 op/s 2024-06-04T05:02:33.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 80143 2.15 MB/sec execute 86 sec latency 15.070 ms 2024-06-04T05:02:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:33 smithi098 ceph-mon[35569]: pgmap v164: 97 pgs: 97 active+clean; 59 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 1.8 MiB/s wr, 196 op/s 2024-06-04T05:02:33.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:33 smithi112 ceph-mon[36453]: pgmap v164: 97 pgs: 97 active+clean; 59 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 1.8 MiB/s wr, 196 op/s 2024-06-04T05:02:34.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 80678 2.15 MB/sec execute 87 sec latency 16.076 ms 2024-06-04T05:02:35.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.139+0000 7f57ffe07700 1 -- 172.21.15.70:0/444327655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f81035f0 msgr2=0x7f57f8103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.139+0000 7f57ffe07700 1 --2- 172.21.15.70:0/444327655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f81035f0 0x7f57f8103a50 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f57e8009990 tx=0x7f57e8009ca0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.139+0000 7f57ffe07700 1 -- 172.21.15.70:0/444327655 shutdown_connections 2024-06-04T05:02:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.139+0000 7f57ffe07700 1 --2- 172.21.15.70:0/444327655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f8103f90 0x7f57f8110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.139+0000 7f57ffe07700 1 --2- 172.21.15.70:0/444327655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57f8108500 0x7f57f81088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.139+0000 7f57ffe07700 1 --2- 172.21.15.70:0/444327655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f81035f0 0x7f57f8103a50 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.139+0000 7f57ffe07700 1 -- 172.21.15.70:0/444327655 >> 172.21.15.70:0/444327655 conn(0x7f57f807a6a0 msgr2=0x7f57f807aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.139+0000 7f57ffe07700 1 -- 172.21.15.70:0/444327655 shutdown_connections 2024-06-04T05:02:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.140+0000 7f57ffe07700 1 -- 172.21.15.70:0/444327655 wait complete. 2024-06-04T05:02:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.140+0000 7f57ffe07700 1 Processor -- start 2024-06-04T05:02:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.140+0000 7f57ffe07700 1 -- start start 2024-06-04T05:02:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.140+0000 7f57ffe07700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57f81035f0 0x7f57f8075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.140+0000 7f57ffe07700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f8103f90 0x7f57f8075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.140+0000 7f57ffe07700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f8108500 0x7f57f80762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.140+0000 7f57ffe07700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57f8113690 con 0x7f57f8103f90 2024-06-04T05:02:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57ffe07700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f57f8113510 con 0x7f57f8108500 2024-06-04T05:02:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57ffe07700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f57f8113810 con 0x7f57f81035f0 2024-06-04T05:02:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57fd3a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f8103f90 0x7f57f8075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57fd3a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f8103f90 0x7f57f8075d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45460/0 (socket says 172.21.15.70:45460) 2024-06-04T05:02:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57fd3a2700 1 -- 172.21.15.70:0/2396571713 learned_addr learned my addr 172.21.15.70:0/2396571713 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57fe3a4700 1 --2- 172.21.15.70:0/2396571713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f8108500 0x7f57f80762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57fdba3700 1 --2- 172.21.15.70:0/2396571713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57f81035f0 0x7f57f8075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57fd3a2700 1 -- 172.21.15.70:0/2396571713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57f81035f0 msgr2=0x7f57f8075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57fd3a2700 1 --2- 172.21.15.70:0/2396571713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57f81035f0 0x7f57f8075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57fd3a2700 1 -- 172.21.15.70:0/2396571713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f8108500 msgr2=0x7f57f80762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57fd3a2700 1 --2- 172.21.15.70:0/2396571713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f8108500 0x7f57f80762c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57fd3a2700 1 -- 172.21.15.70:0/2396571713 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f81a8070 con 0x7f57f8103f90 2024-06-04T05:02:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57fd3a2700 1 --2- 172.21.15.70:0/2396571713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f8103f90 0x7f57f8075d80 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f57e8006010 tx=0x7f57e800bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:35.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57eeffd700 1 -- 172.21.15.70:0/2396571713 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8027070 con 0x7f57f8103f90 2024-06-04T05:02:35.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57eeffd700 1 -- 172.21.15.70:0/2396571713 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f57e8022470 con 0x7f57f8103f90 2024-06-04T05:02:35.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57ffe07700 1 -- 172.21.15.70:0/2396571713 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f81a8300 con 0x7f57f8103f90 2024-06-04T05:02:35.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57eeffd700 1 -- 172.21.15.70:0/2396571713 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8017650 con 0x7f57f8103f90 2024-06-04T05:02:35.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.141+0000 7f57ffe07700 1 -- 172.21.15.70:0/2396571713 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f81a8750 con 0x7f57f8103f90 2024-06-04T05:02:35.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.144+0000 7f57eeffd700 1 -- 172.21.15.70:0/2396571713 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f57e8009d10 con 0x7f57f8103f90 2024-06-04T05:02:35.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.144+0000 7f57eeffd700 1 --2- 172.21.15.70:0/2396571713 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f57e40b29f0 0x7f57e40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.145+0000 7f57fdba3700 1 --2- 172.21.15.70:0/2396571713 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f57e40b29f0 0x7f57e40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.145+0000 7f57eeffd700 1 -- 172.21.15.70:0/2396571713 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f57e80e30e0 con 0x7f57f8103f90 2024-06-04T05:02:35.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.145+0000 7f57fdba3700 1 --2- 172.21.15.70:0/2396571713 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f57e40b29f0 0x7f57e40b4eb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f57f81047d0 tx=0x7f57f4009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:35.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.145+0000 7f580a69cb80 1 Processor -- start 2024-06-04T05:02:35.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.145+0000 7f580a69cb80 1 -- start start 2024-06-04T05:02:35.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.146+0000 7f580a69cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a997320 0x56106a997700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.146+0000 7f580a69cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a9918a0 0x56106a991d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.146+0000 7f580a69cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a992260 0x56106a99e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.146+0000 7f580a69cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56106a8efd70 con 0x56106a997320 2024-06-04T05:02:35.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.146+0000 7f580a69cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56106a8df910 con 0x56106a992260 2024-06-04T05:02:35.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.146+0000 7f580a69cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56106a8e3950 con 0x56106a9918a0 2024-06-04T05:02:35.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.146+0000 7f57fd3a2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a9918a0 0x56106a991d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.146+0000 7f57fdba3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a997320 0x56106a997700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.146+0000 7f57fd3a2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a9918a0 0x56106a991d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59074/0 (socket says 172.21.15.70:59074) 2024-06-04T05:02:35.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.146+0000 7f57fd3a2700 1 -- 172.21.15.70:0/2732245888 learned_addr learned my addr 172.21.15.70:0/2732245888 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:35.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.146+0000 7f57fe3a4700 1 --2- 172.21.15.70:0/2732245888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a992260 0x56106a99e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.146+0000 7f57fd3a2700 1 -- 172.21.15.70:0/2732245888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a992260 msgr2=0x56106a99e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:35.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.146+0000 7f57fd3a2700 1 --2- 172.21.15.70:0/2732245888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a992260 0x56106a99e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.146+0000 7f57fd3a2700 1 -- 172.21.15.70:0/2732245888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a997320 msgr2=0x56106a997700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:35.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.146+0000 7f57fd3a2700 1 --2- 172.21.15.70:0/2732245888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a997320 0x56106a997700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.146+0000 7f57fd3a2700 1 -- 172.21.15.70:0/2732245888 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56106a888fb0 con 0x56106a9918a0 2024-06-04T05:02:35.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.146+0000 7f57fd3a2700 1 --2- 172.21.15.70:0/2732245888 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a9918a0 0x56106a991d20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f57e8005b40 tx=0x7f57e80aa0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb283b3d0ff519fb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:35.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.147+0000 7f57e3fff700 1 -- 172.21.15.70:0/2732245888 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e80ae8a0 con 0x56106a9918a0 2024-06-04T05:02:35.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.147+0000 7f57e3fff700 1 -- 172.21.15.70:0/2732245888 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f57e80b32c0 con 0x56106a9918a0 2024-06-04T05:02:35.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.147+0000 7f57e3fff700 1 -- 172.21.15.70:0/2732245888 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e80b2e30 con 0x56106a9918a0 2024-06-04T05:02:35.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.147+0000 7f580a69cb80 1 -- 172.21.15.70:0/2732245888 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a9918a0 msgr2=0x56106a991d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:35.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.147+0000 7f580a69cb80 1 --2- 172.21.15.70:0/2732245888 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a9918a0 0x56106a991d20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f57e8005b40 tx=0x7f57e80aa0d0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.148+0000 7f580a69cb80 1 -- 172.21.15.70:0/2732245888 shutdown_connections 2024-06-04T05:02:35.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.148+0000 7f580a69cb80 1 --2- 172.21.15.70:0/2732245888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a992260 0x56106a99e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.148+0000 7f580a69cb80 1 --2- 172.21.15.70:0/2732245888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a997320 0x56106a997700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.148+0000 7f580a69cb80 1 --2- 172.21.15.70:0/2732245888 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a9918a0 0x56106a991d20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.148+0000 7f580a69cb80 1 -- 172.21.15.70:0/2732245888 >> 172.21.15.70:0/2732245888 conn(0x56106a8bd240 msgr2=0x56106a98c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:35.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.148+0000 7f580a69cb80 1 -- 172.21.15.70:0/2732245888 shutdown_connections 2024-06-04T05:02:35.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.148+0000 7f580a69cb80 1 -- 172.21.15.70:0/2732245888 wait complete. 2024-06-04T05:02:35.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.148+0000 7f580a69cb80 1 -- start start 2024-06-04T05:02:35.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.148+0000 7f580a69cb80 1 Processor -- start 2024-06-04T05:02:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f580a69cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a9918a0 0x56106a99dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f580a69cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a992260 0x56106a99b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f580a69cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a997320 0x56106a99b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f580a69cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56106a7e5fa0 con 0x56106a997320 2024-06-04T05:02:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f580a69cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56106a847da0 con 0x56106a9918a0 2024-06-04T05:02:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f580a69cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56106a8e3950 con 0x56106a992260 2024-06-04T05:02:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f580a69cb80 10 client.0 _ll_register_callbacks cb 0x56106a87a950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f57fdba3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a9918a0 0x56106a99dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f57fd3a2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a992260 0x56106a99b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f57fe3a4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a997320 0x56106a99b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f57fdba3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a9918a0 0x56106a99dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:35268/0 (socket says 172.21.15.70:35268) 2024-06-04T05:02:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f57fdba3700 1 -- 172.21.15.70:0/540046415 learned_addr learned my addr 172.21.15.70:0/540046415 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f57fd3a2700 1 --2- 172.21.15.70:0/540046415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a992260 0x56106a99b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59080/0 (socket says 172.21.15.70:59080) 2024-06-04T05:02:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f57fdba3700 1 -- 172.21.15.70:0/540046415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a992260 msgr2=0x56106a99b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:35.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f57fdba3700 1 --2- 172.21.15.70:0/540046415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a992260 0x56106a99b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f57fdba3700 1 -- 172.21.15.70:0/540046415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a997320 msgr2=0x56106a99b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:35.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f57fdba3700 1 --2- 172.21.15.70:0/540046415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a997320 0x56106a99b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f57fdba3700 1 -- 172.21.15.70:0/540046415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56106a82f8d0 con 0x56106a9918a0 2024-06-04T05:02:35.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.149+0000 7f57fe3a4700 1 --2- 172.21.15.70:0/540046415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a997320 0x56106a99b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:35.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.150+0000 7f57fdba3700 1 --2- 172.21.15.70:0/540046415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a9918a0 0x56106a99dd70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f57f401d8c0 tx=0x7f57f40047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a76b4cfa4d6ae72f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:35.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.150+0000 7f57e17fa700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:02:35.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.150+0000 7f57e17fa700 1 -- 172.21.15.70:0/540046415 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f4004020 con 0x56106a9918a0 2024-06-04T05:02:35.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.150+0000 7f57e17fa700 1 -- 172.21.15.70:0/540046415 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f57f4009040 con 0x56106a9918a0 2024-06-04T05:02:35.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.150+0000 7f580a69cb80 1 -- 172.21.15.70:0/540046415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57e80b32c0 con 0x56106a9918a0 2024-06-04T05:02:35.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.150+0000 7f57e17fa700 1 -- 172.21.15.70:0/540046415 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f57f40041c0 con 0x56106a9918a0 2024-06-04T05:02:35.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.150+0000 7f580a69cb80 10 client.26216 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:35.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.150+0000 7f580a69cb80 1 -- 172.21.15.70:0/540046415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56106a899760 con 0x56106a9918a0 2024-06-04T05:02:35.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.150+0000 7f57e17fa700 1 -- 172.21.15.70:0/540046415 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f57e80b32c0 con 0x56106a9918a0 2024-06-04T05:02:35.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.150+0000 7f57e17fa700 1 -- 172.21.15.70:0/540046415 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f57f4003680 con 0x56106a9918a0 2024-06-04T05:02:35.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.150+0000 7f580a69cb80 10 client.26216 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:35.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.150+0000 7f580a69cb80 10 client.26216 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:02:35.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.150+0000 7f580a69cb80 20 client.26216 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:35.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.151+0000 7f580a69cb80 1 --2- 172.21.15.70:0/540046415 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56106a9aaf90 0x56106a9ad380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.151+0000 7f580a69cb80 4 client.26216 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:02:35.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.151+0000 7f580a69cb80 1 -- 172.21.15.70:0/540046415 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56106a8f1cc0 con 0x56106a9aaf90 2024-06-04T05:02:35.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.151+0000 7f57fd3a2700 1 --2- 172.21.15.70:0/540046415 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56106a9aaf90 0x56106a9ad380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.151+0000 7f57e17fa700 10 client.26216 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.151+0000 7f57fd3a2700 1 --2- 172.21.15.70:0/540046415 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56106a9aaf90 0x56106a9ad380 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6f151b71e95b0dd server_cookie=f4828814f961e203 in_seq=0 out_seq=0 2024-06-04T05:02:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.152+0000 7f57e17fa700 1 -- 172.21.15.70:0/540046415 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56106a8f1cc0 con 0x56106a9aaf90 2024-06-04T05:02:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.152+0000 7f57e17fa700 10 client.26216 handle_command_reply: tid=0 2024-06-04T05:02:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.152+0000 7f580a69cb80 1 client.26216 shutdown 2024-06-04T05:02:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.152+0000 7f580a69cb80 1 -- 172.21.15.70:0/540046415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a9918a0 msgr2=0x56106a99dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.152+0000 7f580a69cb80 1 --2- 172.21.15.70:0/540046415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a9918a0 0x56106a99dd70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f57f401d8c0 tx=0x7f57f40047c0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.152+0000 7f580a69cb80 1 -- 172.21.15.70:0/540046415 shutdown_connections 2024-06-04T05:02:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.152+0000 7f580a69cb80 1 --2- 172.21.15.70:0/540046415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a997320 0x56106a99b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.152+0000 7f580a69cb80 1 --2- 172.21.15.70:0/540046415 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56106a9aaf90 0x56106a9ad380 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.152+0000 7f580a69cb80 1 --2- 172.21.15.70:0/540046415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a9918a0 0x56106a99dd70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.152+0000 7f580a69cb80 1 --2- 172.21.15.70:0/540046415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a992260 0x56106a99b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.152+0000 7f57e17fa700 0 client.26216 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.152+0000 7f580a69cb80 1 -- 172.21.15.70:0/540046415 >> 172.21.15.70:0/540046415 conn(0x56106a8bd240 msgr2=0x56106a98baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.152+0000 7f580a69cb80 1 -- 172.21.15.70:0/540046415 shutdown_connections 2024-06-04T05:02:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.152+0000 7f580a69cb80 1 -- 172.21.15.70:0/540046415 wait complete. 2024-06-04T05:02:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.152+0000 7f580a69cb80 20 client.26216 trim_cache size 0 max 16384 2024-06-04T05:02:35.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.168+0000 7f580a69cb80 1 Processor -- start 2024-06-04T05:02:35.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.168+0000 7f580a69cb80 1 -- start start 2024-06-04T05:02:35.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.168+0000 7f580a69cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a9918a0 0x56106a98ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.168+0000 7f580a69cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a992260 0x56106a996810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.168+0000 7f580a69cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a997320 0x56106a99b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.168+0000 7f580a69cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81afee0 con 0x56106a992260 2024-06-04T05:02:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.168+0000 7f580a69cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81b1a20 con 0x56106a997320 2024-06-04T05:02:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.168+0000 7f580a69cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81b05b0 con 0x56106a9918a0 2024-06-04T05:02:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.168+0000 7f57fd3a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a992260 0x56106a996810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.168+0000 7f57fd3a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a992260 0x56106a996810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45482/0 (socket says 172.21.15.70:45482) 2024-06-04T05:02:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.169+0000 7f57fd3a2700 1 -- 172.21.15.70:0/3925327637 learned_addr learned my addr 172.21.15.70:0/3925327637 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.169+0000 7f57fdba3700 1 --2- 172.21.15.70:0/3925327637 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a9918a0 0x56106a98ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.169+0000 7f57fe3a4700 1 --2- 172.21.15.70:0/3925327637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a997320 0x56106a99b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.169+0000 7f57fd3a2700 1 -- 172.21.15.70:0/3925327637 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a9918a0 msgr2=0x56106a98ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.169+0000 7f57fd3a2700 1 --2- 172.21.15.70:0/3925327637 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a9918a0 0x56106a98ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.169+0000 7f57fd3a2700 1 -- 172.21.15.70:0/3925327637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a997320 msgr2=0x56106a99b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.169+0000 7f57fd3a2700 1 --2- 172.21.15.70:0/3925327637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a997320 0x56106a99b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.169+0000 7f57fd3a2700 1 -- 172.21.15.70:0/3925327637 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56106a98e380 con 0x56106a992260 2024-06-04T05:02:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.169+0000 7f57fdba3700 1 --2- 172.21.15.70:0/3925327637 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a9918a0 0x56106a98ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:35.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.169+0000 7f57fe3a4700 1 --2- 172.21.15.70:0/3925327637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a997320 0x56106a99b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:02:35.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.169+0000 7f57fd3a2700 1 --2- 172.21.15.70:0/3925327637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a992260 0x56106a996810 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f57e80e0e50 tx=0x7f57e80a64e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=166573f663ab0161 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:35.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.169+0000 7f57e37fe700 1 -- 172.21.15.70:0/3925327637 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8022920 con 0x56106a992260 2024-06-04T05:02:35.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.169+0000 7f57e37fe700 1 -- 172.21.15.70:0/3925327637 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f57e8022ac0 con 0x56106a992260 2024-06-04T05:02:35.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.169+0000 7f57e37fe700 1 -- 172.21.15.70:0/3925327637 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e80b2e30 con 0x56106a992260 2024-06-04T05:02:35.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.170+0000 7f580a69cb80 1 -- 172.21.15.70:0/3925327637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a992260 msgr2=0x56106a996810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:35.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.170+0000 7f580a69cb80 1 --2- 172.21.15.70:0/3925327637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a992260 0x56106a996810 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f57e80e0e50 tx=0x7f57e80a64e0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.170+0000 7f580a69cb80 1 -- 172.21.15.70:0/3925327637 shutdown_connections 2024-06-04T05:02:35.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.170+0000 7f580a69cb80 1 --2- 172.21.15.70:0/3925327637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a997320 0x56106a99b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.170+0000 7f580a69cb80 1 --2- 172.21.15.70:0/3925327637 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a9918a0 0x56106a98ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.170+0000 7f580a69cb80 1 --2- 172.21.15.70:0/3925327637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a992260 0x56106a996810 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.170+0000 7f580a69cb80 1 -- 172.21.15.70:0/3925327637 >> 172.21.15.70:0/3925327637 conn(0x56106a8bd240 msgr2=0x56106a991120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:35.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.170+0000 7f580a69cb80 1 -- 172.21.15.70:0/3925327637 shutdown_connections 2024-06-04T05:02:35.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.170+0000 7f580a69cb80 1 -- 172.21.15.70:0/3925327637 wait complete. 2024-06-04T05:02:35.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f580a69cb80 1 -- start start 2024-06-04T05:02:35.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f580a69cb80 1 Processor -- start 2024-06-04T05:02:35.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f580a69cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a9918a0 0x56106a9b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f580a69cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a992260 0x56106a9b22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f580a69cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a997320 0x56106a9a2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f580a69cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81b1350 con 0x56106a9918a0 2024-06-04T05:02:35.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f580a69cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81b0c80 con 0x56106a997320 2024-06-04T05:02:35.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f580a69cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81ade80 con 0x56106a992260 2024-06-04T05:02:35.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f580a69cb80 10 client.0 _ll_register_callbacks cb 0x56106a87a950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:35.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f57fdba3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a9918a0 0x56106a9b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f57fd3a2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a992260 0x56106a9b22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f57fdba3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a9918a0 0x56106a9b1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45488/0 (socket says 172.21.15.70:45488) 2024-06-04T05:02:35.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f57fdba3700 1 -- 172.21.15.70:0/1164738446 learned_addr learned my addr 172.21.15.70:0/1164738446 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:35.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f57fe3a4700 1 --2- 172.21.15.70:0/1164738446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a997320 0x56106a9a2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f57fdba3700 1 -- 172.21.15.70:0/1164738446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a992260 msgr2=0x56106a9b22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:35.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f57fdba3700 1 --2- 172.21.15.70:0/1164738446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a992260 0x56106a9b22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f57fdba3700 1 -- 172.21.15.70:0/1164738446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a997320 msgr2=0x56106a9a2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:35.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.171+0000 7f57fdba3700 1 --2- 172.21.15.70:0/1164738446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a997320 0x56106a9a2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.172+0000 7f57fdba3700 1 -- 172.21.15.70:0/1164738446 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56106a9a2bd0 con 0x56106a9918a0 2024-06-04T05:02:35.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.172+0000 7f57fdba3700 1 --2- 172.21.15.70:0/1164738446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a9918a0 0x56106a9b1d80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f57f4016040 tx=0x7f57f4003700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197528504f7f3489 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:35.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.172+0000 7f57e17fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:02:35.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.172+0000 7f57e17fa700 1 -- 172.21.15.70:0/1164738446 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f4003ed0 con 0x56106a9918a0 2024-06-04T05:02:35.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.172+0000 7f57e17fa700 1 -- 172.21.15.70:0/1164738446 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f57f4009040 con 0x56106a9918a0 2024-06-04T05:02:35.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.172+0000 7f580a69cb80 1 -- 172.21.15.70:0/1164738446 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57e8022ac0 con 0x56106a9918a0 2024-06-04T05:02:35.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.172+0000 7f57e17fa700 1 -- 172.21.15.70:0/1164738446 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f57f400e6c0 con 0x56106a9918a0 2024-06-04T05:02:35.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.172+0000 7f580a69cb80 10 client.16926 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:35.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.172+0000 7f580a69cb80 1 -- 172.21.15.70:0/1164738446 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56106a9a3e40 con 0x56106a9918a0 2024-06-04T05:02:35.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.172+0000 7f57e17fa700 1 -- 172.21.15.70:0/1164738446 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f57e8022ac0 con 0x56106a9918a0 2024-06-04T05:02:35.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.172+0000 7f57e17fa700 1 -- 172.21.15.70:0/1164738446 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f57f40427d0 con 0x56106a9918a0 2024-06-04T05:02:35.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.172+0000 7f580a69cb80 10 client.16926 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:35.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.172+0000 7f580a69cb80 10 client.16926 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:02:35.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.172+0000 7f580a69cb80 20 client.16926 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:35.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.173+0000 7f580a69cb80 1 --2- 172.21.15.70:0/1164738446 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56106aa15b90 0x56106aa17fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:35.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.173+0000 7f580a69cb80 4 client.16926 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:02:35.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.173+0000 7f580a69cb80 1 -- 172.21.15.70:0/1164738446 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56106a8eb4a0 con 0x56106aa15b90 2024-06-04T05:02:35.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.173+0000 7f57fd3a2700 1 --2- 172.21.15.70:0/1164738446 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56106aa15b90 0x56106aa17fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:35.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.173+0000 7f57e17fa700 10 client.16926 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:35.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.173+0000 7f57fd3a2700 1 --2- 172.21.15.70:0/1164738446 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56106aa15b90 0x56106aa17fb0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=427ec1d595d59305 server_cookie=69c96f5528346253 in_seq=0 out_seq=0 2024-06-04T05:02:35.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.173+0000 7f57e17fa700 1 -- 172.21.15.70:0/1164738446 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56106a8eb4a0 con 0x56106aa15b90 2024-06-04T05:02:35.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.173+0000 7f57e17fa700 10 client.16926 handle_command_reply: tid=0 2024-06-04T05:02:35.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.173+0000 7f580a69cb80 1 client.16926 shutdown 2024-06-04T05:02:35.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.174+0000 7f580a69cb80 1 -- 172.21.15.70:0/1164738446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a9918a0 msgr2=0x56106a9b1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:35.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.174+0000 7f580a69cb80 1 --2- 172.21.15.70:0/1164738446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a9918a0 0x56106a9b1d80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f57f4016040 tx=0x7f57f4003700 comp rx=0 tx=0).stop 2024-06-04T05:02:35.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.174+0000 7f580a69cb80 1 -- 172.21.15.70:0/1164738446 shutdown_connections 2024-06-04T05:02:35.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.174+0000 7f580a69cb80 1 --2- 172.21.15.70:0/1164738446 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56106aa15b90 0x56106aa17fb0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.174+0000 7f580a69cb80 1 --2- 172.21.15.70:0/1164738446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56106a997320 0x56106a9a2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.174+0000 7f580a69cb80 1 --2- 172.21.15.70:0/1164738446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56106a9918a0 0x56106a9b1d80 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.174+0000 7f580a69cb80 1 --2- 172.21.15.70:0/1164738446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56106a992260 0x56106a9b22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.174+0000 7f580a69cb80 1 -- 172.21.15.70:0/1164738446 >> 172.21.15.70:0/1164738446 conn(0x56106a8bd240 msgr2=0x56106a99c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:35.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.174+0000 7f57e17fa700 0 client.16926 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:35.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.174+0000 7f580a69cb80 1 -- 172.21.15.70:0/1164738446 shutdown_connections 2024-06-04T05:02:35.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.174+0000 7f580a69cb80 1 -- 172.21.15.70:0/1164738446 wait complete. 2024-06-04T05:02:35.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.174+0000 7f580a69cb80 20 client.16926 trim_cache size 0 max 16384 2024-06-04T05:02:35.181 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:02:35.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.174+0000 7f57ffe07700 1 -- 172.21.15.70:0/2396571713 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f57e40b29f0 msgr2=0x7f57e40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:35.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.174+0000 7f57ffe07700 1 --2- 172.21.15.70:0/2396571713 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f57e40b29f0 0x7f57e40b4eb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f57f81047d0 tx=0x7f57f4009260 comp rx=0 tx=0).stop 2024-06-04T05:02:35.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.174+0000 7f57ffe07700 1 -- 172.21.15.70:0/2396571713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f8103f90 msgr2=0x7f57f8075d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:35.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.174+0000 7f57ffe07700 1 --2- 172.21.15.70:0/2396571713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f8103f90 0x7f57f8075d80 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f57e8006010 tx=0x7f57e800bfc0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.175+0000 7f57ffe07700 1 -- 172.21.15.70:0/2396571713 shutdown_connections 2024-06-04T05:02:35.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.175+0000 7f57ffe07700 1 --2- 172.21.15.70:0/2396571713 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f57e40b29f0 0x7f57e40b4eb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.175+0000 7f57ffe07700 1 --2- 172.21.15.70:0/2396571713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f8108500 0x7f57f80762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.175+0000 7f57ffe07700 1 --2- 172.21.15.70:0/2396571713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57f81035f0 0x7f57f8075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.175+0000 7f57ffe07700 1 --2- 172.21.15.70:0/2396571713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f8103f90 0x7f57f8075d80 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:35.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.175+0000 7f57ffe07700 1 -- 172.21.15.70:0/2396571713 >> 172.21.15.70:0/2396571713 conn(0x7f57f807a6a0 msgr2=0x7f57f8101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:35.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.175+0000 7f57ffe07700 1 -- 172.21.15.70:0/2396571713 shutdown_connections 2024-06-04T05:02:35.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:35.175+0000 7f57ffe07700 1 -- 172.21.15.70:0/2396571713 wait complete. 2024-06-04T05:02:35.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 81400 2.20 MB/sec execute 88 sec latency 15.235 ms 2024-06-04T05:02:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:35 smithi098 ceph-mon[35569]: pgmap v165: 97 pgs: 97 active+clean; 59 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1.2 MiB/s wr, 129 op/s 2024-06-04T05:02:35.783 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:35 smithi070 ceph-mon[32631]: pgmap v165: 97 pgs: 97 active+clean; 59 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1.2 MiB/s wr, 129 op/s 2024-06-04T05:02:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:35 smithi112 ceph-mon[36453]: pgmap v165: 97 pgs: 97 active+clean; 59 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1.2 MiB/s wr, 129 op/s 2024-06-04T05:02:36.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 81882 2.23 MB/sec execute 89 sec latency 15.415 ms 2024-06-04T05:02:36.810 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:02:36.811 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:02:37.067 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:02:37.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 82164 2.21 MB/sec execute 90 sec latency 14.954 ms 2024-06-04T05:02:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:37 smithi098 ceph-mon[35569]: pgmap v166: 97 pgs: 97 active+clean; 61 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-06-04T05:02:37.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:37 smithi070 ceph-mon[32631]: pgmap v166: 97 pgs: 97 active+clean; 61 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-06-04T05:02:37.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:37 smithi112 ceph-mon[36453]: pgmap v166: 97 pgs: 97 active+clean; 61 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-06-04T05:02:38.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 82616 2.19 MB/sec execute 91 sec latency 23.274 ms 2024-06-04T05:02:38.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.900+0000 7f23e6a69700 1 -- 172.21.15.70:0/2809678305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23e006c470 msgr2=0x7f23e01103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:38.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.900+0000 7f23e6a69700 1 --2- 172.21.15.70:0/2809678305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23e006c470 0x7f23e01103c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f23d800b0a0 tx=0x7f23d800b3b0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.900+0000 7f23e6a69700 1 -- 172.21.15.70:0/2809678305 shutdown_connections 2024-06-04T05:02:38.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.900+0000 7f23e6a69700 1 --2- 172.21.15.70:0/2809678305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23e006c470 0x7f23e01103c0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.900+0000 7f23e6a69700 1 --2- 172.21.15.70:0/2809678305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e01095a0 0x7f23e0109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.900+0000 7f23e6a69700 1 --2- 172.21.15.70:0/2809678305 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f23e006bab0 0x7f23e006bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.900+0000 7f23e6a69700 1 -- 172.21.15.70:0/2809678305 >> 172.21.15.70:0/2809678305 conn(0x7f23e00793b0 msgr2=0x7f23e00797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:38.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.900+0000 7f23e6a69700 1 -- 172.21.15.70:0/2809678305 shutdown_connections 2024-06-04T05:02:38.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.900+0000 7f23e6a69700 1 -- 172.21.15.70:0/2809678305 wait complete. 2024-06-04T05:02:38.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.901+0000 7f23e6a69700 1 Processor -- start 2024-06-04T05:02:38.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.901+0000 7f23e6a69700 1 -- start start 2024-06-04T05:02:38.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.901+0000 7f23e6a69700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f23e006bab0 0x7f23e0102500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.901+0000 7f23e6a69700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23e006c470 0x7f23e0102a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.901+0000 7f23e6a69700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e01095a0 0x7f23e0102f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.901+0000 7f23e6a69700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f23e0113590 con 0x7f23e006c470 2024-06-04T05:02:38.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.901+0000 7f23e6a69700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f23e0113410 con 0x7f23e01095a0 2024-06-04T05:02:38.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.901+0000 7f23e6a69700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f23e0113710 con 0x7f23e006bab0 2024-06-04T05:02:38.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.901+0000 7f23dffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23e006c470 0x7f23e0102a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.901+0000 7f23dffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23e006c470 0x7f23e0102a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46700/0 (socket says 172.21.15.70:46700) 2024-06-04T05:02:38.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.901+0000 7f23dffff700 1 -- 172.21.15.70:0/1809009181 learned_addr learned my addr 172.21.15.70:0/1809009181 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:38.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.902+0000 7f23e5006700 1 --2- 172.21.15.70:0/1809009181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e01095a0 0x7f23e0102f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.902+0000 7f23e4805700 1 --2- 172.21.15.70:0/1809009181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f23e006bab0 0x7f23e0102500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.902+0000 7f23dffff700 1 -- 172.21.15.70:0/1809009181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f23e006bab0 msgr2=0x7f23e0102500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:38.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.902+0000 7f23dffff700 1 --2- 172.21.15.70:0/1809009181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f23e006bab0 0x7f23e0102500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.902+0000 7f23dffff700 1 -- 172.21.15.70:0/1809009181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e01095a0 msgr2=0x7f23e0102f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:38.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.902+0000 7f23dffff700 1 --2- 172.21.15.70:0/1809009181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e01095a0 0x7f23e0102f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.902+0000 7f23dffff700 1 -- 172.21.15.70:0/1809009181 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23e01a8030 con 0x7f23e006c470 2024-06-04T05:02:38.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.902+0000 7f23dffff700 1 --2- 172.21.15.70:0/1809009181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23e006c470 0x7f23e0102a40 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f23cc00e9c0 tx=0x7f23cc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:38.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.902+0000 7f23ddffb700 1 -- 172.21.15.70:0/1809009181 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23cc00cbd0 con 0x7f23e006c470 2024-06-04T05:02:38.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.902+0000 7f23e6a69700 1 -- 172.21.15.70:0/1809009181 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23e01a82c0 con 0x7f23e006c470 2024-06-04T05:02:38.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.902+0000 7f23ddffb700 1 -- 172.21.15.70:0/1809009181 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f23cc00ee60 con 0x7f23e006c470 2024-06-04T05:02:38.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.902+0000 7f23ddffb700 1 -- 172.21.15.70:0/1809009181 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23cc0054b0 con 0x7f23e006c470 2024-06-04T05:02:38.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.903+0000 7f23e6a69700 1 -- 172.21.15.70:0/1809009181 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23e01a8800 con 0x7f23e006c470 2024-06-04T05:02:38.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.905+0000 7f23ddffb700 1 -- 172.21.15.70:0/1809009181 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f23cc004750 con 0x7f23e006c470 2024-06-04T05:02:38.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.906+0000 7f23ddffb700 1 --2- 172.21.15.70:0/1809009181 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f23d00b29f0 0x7f23d00b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.906+0000 7f23e4805700 1 --2- 172.21.15.70:0/1809009181 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f23d00b29f0 0x7f23d00b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.906+0000 7f23ddffb700 1 -- 172.21.15.70:0/1809009181 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f23cc0e28f0 con 0x7f23e006c470 2024-06-04T05:02:38.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.906+0000 7f23e4805700 1 --2- 172.21.15.70:0/1809009181 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f23d00b29f0 0x7f23d00b4eb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009be0 tx=0x7f23d4009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:38.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23f12feb80 1 Processor -- start 2024-06-04T05:02:38.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23f12feb80 1 -- start start 2024-06-04T05:02:38.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23f12feb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563383208e0 0x556338320cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23f12feb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556338321290 0x5563383249c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23f12feb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556338324fe0 0x556338325420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23f12feb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55633827fd10 con 0x556338321290 2024-06-04T05:02:38.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23f12feb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55633826f8b0 con 0x556338324fe0 2024-06-04T05:02:38.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23f12feb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5563382738f0 con 0x5563383208e0 2024-06-04T05:02:38.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23dffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556338321290 0x5563383249c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23dffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556338321290 0x5563383249c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46704/0 (socket says 172.21.15.70:46704) 2024-06-04T05:02:38.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23dffff700 1 -- 172.21.15.70:0/2574833534 learned_addr learned my addr 172.21.15.70:0/2574833534 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:38.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23e5006700 1 --2- 172.21.15.70:0/2574833534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556338324fe0 0x556338325420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23e4805700 1 --2- 172.21.15.70:0/2574833534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563383208e0 0x556338320cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23dffff700 1 -- 172.21.15.70:0/2574833534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563383208e0 msgr2=0x556338320cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:38.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23dffff700 1 --2- 172.21.15.70:0/2574833534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563383208e0 0x556338320cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23dffff700 1 -- 172.21.15.70:0/2574833534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556338324fe0 msgr2=0x556338325420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:38.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23dffff700 1 --2- 172.21.15.70:0/2574833534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556338324fe0 0x556338325420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.907+0000 7f23dffff700 1 -- 172.21.15.70:0/2574833534 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563381bf950 con 0x556338321290 2024-06-04T05:02:38.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.908+0000 7f23dffff700 1 --2- 172.21.15.70:0/2574833534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556338321290 0x5563383249c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f23cc0a9b60 tx=0x7f23cc0a95f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce86fbfec614575 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:38.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.908+0000 7f23caffd700 1 -- 172.21.15.70:0/2574833534 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23cc0c3020 con 0x556338321290 2024-06-04T05:02:38.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.908+0000 7f23caffd700 1 -- 172.21.15.70:0/2574833534 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f23cc0b2380 con 0x556338321290 2024-06-04T05:02:38.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.908+0000 7f23caffd700 1 -- 172.21.15.70:0/2574833534 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23cc0b2a30 con 0x556338321290 2024-06-04T05:02:38.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.909+0000 7f23f12feb80 1 -- 172.21.15.70:0/2574833534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556338321290 msgr2=0x5563383249c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:38.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.909+0000 7f23f12feb80 1 --2- 172.21.15.70:0/2574833534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556338321290 0x5563383249c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f23cc0a9b60 tx=0x7f23cc0a95f0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.909+0000 7f23f12feb80 1 -- 172.21.15.70:0/2574833534 shutdown_connections 2024-06-04T05:02:38.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.909+0000 7f23f12feb80 1 --2- 172.21.15.70:0/2574833534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556338324fe0 0x556338325420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.909+0000 7f23f12feb80 1 --2- 172.21.15.70:0/2574833534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563383208e0 0x556338320cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.909+0000 7f23f12feb80 1 --2- 172.21.15.70:0/2574833534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556338321290 0x5563383249c0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.909+0000 7f23f12feb80 1 -- 172.21.15.70:0/2574833534 >> 172.21.15.70:0/2574833534 conn(0x55633824e100 msgr2=0x55633831af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.909+0000 7f23f12feb80 1 -- 172.21.15.70:0/2574833534 shutdown_connections 2024-06-04T05:02:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.909+0000 7f23f12feb80 1 -- 172.21.15.70:0/2574833534 wait complete. 2024-06-04T05:02:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.909+0000 7f23f12feb80 1 -- start start 2024-06-04T05:02:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.909+0000 7f23f12feb80 1 Processor -- start 2024-06-04T05:02:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23f12feb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563383208e0 0x55633832c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23f12feb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556338321290 0x556338329840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23f12feb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556338324fe0 0x556338329d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23f12feb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556338176020 con 0x556338321290 2024-06-04T05:02:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23f12feb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556338211840 con 0x556338324fe0 2024-06-04T05:02:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23f12feb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55633832d1a0 con 0x5563383208e0 2024-06-04T05:02:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23dffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556338321290 0x556338329840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23f12feb80 10 client.0 _ll_register_callbacks cb 0x55633820a9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23dffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556338321290 0x556338329840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46712/0 (socket says 172.21.15.70:46712) 2024-06-04T05:02:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23dffff700 1 -- 172.21.15.70:0/1171313606 learned_addr learned my addr 172.21.15.70:0/1171313606 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23e4805700 1 --2- 172.21.15.70:0/1171313606 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563383208e0 0x55633832c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23e5006700 1 --2- 172.21.15.70:0/1171313606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556338324fe0 0x556338329d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23dffff700 1 -- 172.21.15.70:0/1171313606 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563383208e0 msgr2=0x55633832c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23dffff700 1 --2- 172.21.15.70:0/1171313606 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563383208e0 0x55633832c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23dffff700 1 -- 172.21.15.70:0/1171313606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556338324fe0 msgr2=0x556338329d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23dffff700 1 --2- 172.21.15.70:0/1171313606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556338324fe0 0x556338329d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23dffff700 1 -- 172.21.15.70:0/1171313606 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55633820e710 con 0x556338321290 2024-06-04T05:02:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23e5006700 1 --2- 172.21.15.70:0/1171313606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556338324fe0 0x556338329d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:02:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23dffff700 1 --2- 172.21.15.70:0/1171313606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556338321290 0x556338329840 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f23cc009fd0 tx=0x7f23cc0aa060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=547844e7f608004e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.910+0000 7f23bffff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:02:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.911+0000 7f23bffff700 1 -- 172.21.15.70:0/1171313606 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23cc0c3040 con 0x556338321290 2024-06-04T05:02:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.911+0000 7f23f12feb80 1 -- 172.21.15.70:0/1171313606 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23cc0b2380 con 0x556338321290 2024-06-04T05:02:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.911+0000 7f23bffff700 1 -- 172.21.15.70:0/1171313606 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f23cc0a61c0 con 0x556338321290 2024-06-04T05:02:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.911+0000 7f23bffff700 1 -- 172.21.15.70:0/1171313606 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f23cc0aa2d0 con 0x556338321290 2024-06-04T05:02:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.911+0000 7f23f12feb80 10 client.16938 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.911+0000 7f23f12feb80 1 -- 172.21.15.70:0/1171313606 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5563383379d0 con 0x556338321290 2024-06-04T05:02:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.911+0000 7f23bffff700 1 -- 172.21.15.70:0/1171313606 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f23cc0b2380 con 0x556338321290 2024-06-04T05:02:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.911+0000 7f23bffff700 1 -- 172.21.15.70:0/1171313606 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f23cc0a5040 con 0x556338321290 2024-06-04T05:02:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.911+0000 7f23f12feb80 10 client.16938 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.911+0000 7f23f12feb80 10 client.16938 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:02:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.911+0000 7f23f12feb80 20 client.16938 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.911+0000 7f23f12feb80 1 --2- 172.21.15.70:0/1171313606 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556338339ea0 0x55633833c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.911+0000 7f23f12feb80 4 client.16938 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:02:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.911+0000 7f23f12feb80 1 -- 172.21.15.70:0/1171313606 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563382eca60 con 0x556338339ea0 2024-06-04T05:02:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.911+0000 7f23e4805700 1 --2- 172.21.15.70:0/1171313606 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556338339ea0 0x55633833c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.912+0000 7f23bffff700 10 client.16938 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.912+0000 7f23e4805700 1 --2- 172.21.15.70:0/1171313606 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556338339ea0 0x55633833c290 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fdb54031166938 server_cookie=7a098b8129275d5c in_seq=0 out_seq=0 2024-06-04T05:02:38.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.912+0000 7f23bffff700 1 -- 172.21.15.70:0/1171313606 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5563382eca60 con 0x556338339ea0 2024-06-04T05:02:38.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.912+0000 7f23bffff700 10 client.16938 handle_command_reply: tid=0 2024-06-04T05:02:38.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.912+0000 7f23f12feb80 1 client.16938 shutdown 2024-06-04T05:02:38.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.913+0000 7f23f12feb80 1 -- 172.21.15.70:0/1171313606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556338321290 msgr2=0x556338329840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:38.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.913+0000 7f23f12feb80 1 --2- 172.21.15.70:0/1171313606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556338321290 0x556338329840 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f23cc009fd0 tx=0x7f23cc0aa060 comp rx=0 tx=0).stop 2024-06-04T05:02:38.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.913+0000 7f23f12feb80 1 -- 172.21.15.70:0/1171313606 shutdown_connections 2024-06-04T05:02:38.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.913+0000 7f23f12feb80 1 --2- 172.21.15.70:0/1171313606 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556338339ea0 0x55633833c290 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.913+0000 7f23f12feb80 1 --2- 172.21.15.70:0/1171313606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556338324fe0 0x556338329d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.913+0000 7f23f12feb80 1 --2- 172.21.15.70:0/1171313606 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563383208e0 0x55633832c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.913+0000 7f23f12feb80 1 --2- 172.21.15.70:0/1171313606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556338321290 0x556338329840 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.913+0000 7f23f12feb80 1 -- 172.21.15.70:0/1171313606 >> 172.21.15.70:0/1171313606 conn(0x55633824e100 msgr2=0x55633831a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:38.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.913+0000 7f23bffff700 0 client.16938 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:38.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.913+0000 7f23f12feb80 1 -- 172.21.15.70:0/1171313606 shutdown_connections 2024-06-04T05:02:38.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.913+0000 7f23f12feb80 1 -- 172.21.15.70:0/1171313606 wait complete. 2024-06-04T05:02:38.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.913+0000 7f23f12feb80 20 client.16938 trim_cache size 0 max 16384 2024-06-04T05:02:38.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.937+0000 7f23f12feb80 1 Processor -- start 2024-06-04T05:02:38.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.937+0000 7f23f12feb80 1 -- start start 2024-06-04T05:02:38.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.937+0000 7f23f12feb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563383208e0 0x5563383200d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.937+0000 7f23f12feb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556338324fe0 0x55633833eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.937+0000 7f23f12feb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55633833f420 0x55633831ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.937+0000 7f23f12feb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f23e01aff70 con 0x55633833f420 2024-06-04T05:02:38.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.937+0000 7f23f12feb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f23e01b1ab0 con 0x5563383208e0 2024-06-04T05:02:38.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.937+0000 7f23f12feb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f23e01b0640 con 0x556338324fe0 2024-06-04T05:02:38.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.937+0000 7f23e5006700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55633833f420 0x55633831ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.937+0000 7f23dffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556338324fe0 0x55633833eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.937+0000 7f23dffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556338324fe0 0x55633833eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:38662/0 (socket says 172.21.15.70:38662) 2024-06-04T05:02:38.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.937+0000 7f23e5006700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55633833f420 0x55633831ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46716/0 (socket says 172.21.15.70:46716) 2024-06-04T05:02:38.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.937+0000 7f23dffff700 1 -- 172.21.15.70:0/3616864254 learned_addr learned my addr 172.21.15.70:0/3616864254 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:38.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.937+0000 7f23e4805700 1 --2- 172.21.15.70:0/3616864254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563383208e0 0x5563383200d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.938+0000 7f23dffff700 1 -- 172.21.15.70:0/3616864254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563383208e0 msgr2=0x5563383200d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:38.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.938+0000 7f23dffff700 1 --2- 172.21.15.70:0/3616864254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563383208e0 0x5563383200d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.938+0000 7f23dffff700 1 -- 172.21.15.70:0/3616864254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55633833f420 msgr2=0x55633831ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:38.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.938+0000 7f23dffff700 1 --2- 172.21.15.70:0/3616864254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55633833f420 0x55633831ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.938+0000 7f23dffff700 1 -- 172.21.15.70:0/3616864254 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556338324820 con 0x556338324fe0 2024-06-04T05:02:38.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.938+0000 7f23e5006700 1 --2- 172.21.15.70:0/3616864254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55633833f420 0x55633831ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:38.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.938+0000 7f23dffff700 1 --2- 172.21.15.70:0/3616864254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556338324fe0 0x55633833eee0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f23cc0a98d0 tx=0x7f23cc0e00f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b09fefe1e5febfb1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:38.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.938+0000 7f23ca7fc700 1 -- 172.21.15.70:0/3616864254 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23cc0c3070 con 0x556338324fe0 2024-06-04T05:02:38.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.938+0000 7f23ca7fc700 1 -- 172.21.15.70:0/3616864254 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f23cc0b2a30 con 0x556338324fe0 2024-06-04T05:02:38.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.938+0000 7f23ca7fc700 1 -- 172.21.15.70:0/3616864254 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23cc0adf80 con 0x556338324fe0 2024-06-04T05:02:38.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.939+0000 7f23f12feb80 1 -- 172.21.15.70:0/3616864254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556338324fe0 msgr2=0x55633833eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:38.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.939+0000 7f23f12feb80 1 --2- 172.21.15.70:0/3616864254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556338324fe0 0x55633833eee0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f23cc0a98d0 tx=0x7f23cc0e00f0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.939+0000 7f23f12feb80 1 -- 172.21.15.70:0/3616864254 shutdown_connections 2024-06-04T05:02:38.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.939+0000 7f23f12feb80 1 --2- 172.21.15.70:0/3616864254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55633833f420 0x55633831ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.939+0000 7f23f12feb80 1 --2- 172.21.15.70:0/3616864254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563383208e0 0x5563383200d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.939+0000 7f23f12feb80 1 --2- 172.21.15.70:0/3616864254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556338324fe0 0x55633833eee0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.939+0000 7f23f12feb80 1 -- 172.21.15.70:0/3616864254 >> 172.21.15.70:0/3616864254 conn(0x55633824e100 msgr2=0x556338328890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.939+0000 7f23f12feb80 1 -- 172.21.15.70:0/3616864254 shutdown_connections 2024-06-04T05:02:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.939+0000 7f23f12feb80 1 -- 172.21.15.70:0/3616864254 wait complete. 2024-06-04T05:02:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.940+0000 7f23f12feb80 1 -- start start 2024-06-04T05:02:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.940+0000 7f23f12feb80 1 Processor -- start 2024-06-04T05:02:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.940+0000 7f23f12feb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563383208e0 0x55633839cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.940+0000 7f23f12feb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556338324fe0 0x55633839d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.940+0000 7f23f12feb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55633833f420 0x556338398670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.940+0000 7f23f12feb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f23e01b13e0 con 0x5563383208e0 2024-06-04T05:02:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.940+0000 7f23f12feb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f23e01b0d10 con 0x556338324fe0 2024-06-04T05:02:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.940+0000 7f23f12feb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f23e01adf10 con 0x55633833f420 2024-06-04T05:02:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.940+0000 7f23e4805700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563383208e0 0x55633839cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.940+0000 7f23f12feb80 10 client.0 _ll_register_callbacks cb 0x55633820a9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.940+0000 7f23e4805700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563383208e0 0x55633839cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46718/0 (socket says 172.21.15.70:46718) 2024-06-04T05:02:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.940+0000 7f23e4805700 1 -- 172.21.15.70:0/214973011 learned_addr learned my addr 172.21.15.70:0/214973011 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.940+0000 7f23dffff700 1 --2- 172.21.15.70:0/214973011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556338324fe0 0x55633839d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.940+0000 7f23e5006700 1 --2- 172.21.15.70:0/214973011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55633833f420 0x556338398670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.941+0000 7f23e4805700 1 -- 172.21.15.70:0/214973011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55633833f420 msgr2=0x556338398670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.941+0000 7f23e4805700 1 --2- 172.21.15.70:0/214973011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55633833f420 0x556338398670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.941+0000 7f23e4805700 1 -- 172.21.15.70:0/214973011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556338324fe0 msgr2=0x55633839d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.941+0000 7f23e4805700 1 --2- 172.21.15.70:0/214973011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556338324fe0 0x55633839d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.941+0000 7f23e4805700 1 -- 172.21.15.70:0/214973011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556338398eb0 con 0x5563383208e0 2024-06-04T05:02:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.941+0000 7f23e5006700 1 --2- 172.21.15.70:0/214973011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55633833f420 0x556338398670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.941+0000 7f23e4805700 1 --2- 172.21.15.70:0/214973011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563383208e0 0x55633839cca0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f23d4005950 tx=0x7f23d40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89686f1fa90f9618 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.941+0000 7f23bffff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:02:38.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.941+0000 7f23bffff700 1 -- 172.21.15.70:0/214973011 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d4004030 con 0x5563383208e0 2024-06-04T05:02:38.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.941+0000 7f23f12feb80 1 -- 172.21.15.70:0/214973011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23cc0b2a30 con 0x5563383208e0 2024-06-04T05:02:38.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.941+0000 7f23bffff700 1 -- 172.21.15.70:0/214973011 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f23d40041d0 con 0x5563383208e0 2024-06-04T05:02:38.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.941+0000 7f23bffff700 1 -- 172.21.15.70:0/214973011 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f23d40043d0 con 0x5563383208e0 2024-06-04T05:02:38.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.941+0000 7f23f12feb80 10 client.16944 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:38.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.941+0000 7f23f12feb80 1 -- 172.21.15.70:0/214973011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5563383a5660 con 0x5563383208e0 2024-06-04T05:02:38.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.942+0000 7f23bffff700 1 -- 172.21.15.70:0/214973011 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f23cc0b2a30 con 0x5563383208e0 2024-06-04T05:02:38.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.942+0000 7f23bffff700 1 -- 172.21.15.70:0/214973011 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f23d402a620 con 0x5563383208e0 2024-06-04T05:02:38.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.942+0000 7f23f12feb80 10 client.16944 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:38.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.942+0000 7f23f12feb80 10 client.16944 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:02:38.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.942+0000 7f23f12feb80 20 client.16944 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:38.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.942+0000 7f23f12feb80 1 --2- 172.21.15.70:0/214973011 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5563383a5d80 0x5563383a81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:38.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.942+0000 7f23f12feb80 4 client.16944 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:02:38.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.942+0000 7f23f12feb80 1 -- 172.21.15.70:0/214973011 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556338281c60 con 0x5563383a5d80 2024-06-04T05:02:38.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.942+0000 7f23dffff700 1 --2- 172.21.15.70:0/214973011 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5563383a5d80 0x5563383a81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:38.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.943+0000 7f23dffff700 1 --2- 172.21.15.70:0/214973011 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5563383a5d80 0x5563383a81a0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd4db9873e6b6b29 server_cookie=1ac2ae795b8ef8b9 in_seq=0 out_seq=0 2024-06-04T05:02:38.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.943+0000 7f23bffff700 10 client.16944 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:38.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.943+0000 7f23bffff700 1 -- 172.21.15.70:0/214973011 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556338281c60 con 0x5563383a5d80 2024-06-04T05:02:38.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.943+0000 7f23bffff700 10 client.16944 handle_command_reply: tid=0 2024-06-04T05:02:38.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.943+0000 7f23f12feb80 1 client.16944 shutdown 2024-06-04T05:02:38.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.944+0000 7f23f12feb80 1 -- 172.21.15.70:0/214973011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563383208e0 msgr2=0x55633839cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:38.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.944+0000 7f23f12feb80 1 --2- 172.21.15.70:0/214973011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563383208e0 0x55633839cca0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f23d4005950 tx=0x7f23d40047c0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.944+0000 7f23f12feb80 1 -- 172.21.15.70:0/214973011 shutdown_connections 2024-06-04T05:02:38.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.944+0000 7f23f12feb80 1 --2- 172.21.15.70:0/214973011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55633833f420 0x556338398670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.944+0000 7f23f12feb80 1 --2- 172.21.15.70:0/214973011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563383208e0 0x55633839cca0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.944+0000 7f23f12feb80 1 --2- 172.21.15.70:0/214973011 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5563383a5d80 0x5563383a81a0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.944+0000 7f23f12feb80 1 --2- 172.21.15.70:0/214973011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556338324fe0 0x55633839d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.944+0000 7f23f12feb80 1 -- 172.21.15.70:0/214973011 >> 172.21.15.70:0/214973011 conn(0x55633824e100 msgr2=0x5563383290f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:38.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.944+0000 7f23bffff700 0 client.16944 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:38.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.944+0000 7f23f12feb80 1 -- 172.21.15.70:0/214973011 shutdown_connections 2024-06-04T05:02:38.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.944+0000 7f23f12feb80 1 -- 172.21.15.70:0/214973011 wait complete. 2024-06-04T05:02:38.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.944+0000 7f23f12feb80 20 client.16944 trim_cache size 0 max 16384 2024-06-04T05:02:38.952 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:02:38.952 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:02:38.952 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "d9166179-e851-420b-bf88-bfe6b68f34e7", 2024-06-04T05:02:38.952 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:02:38.952 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:02:38.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.945+0000 7f23e6a69700 1 -- 172.21.15.70:0/1809009181 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f23d00b29f0 msgr2=0x7f23d00b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:38.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.945+0000 7f23e6a69700 1 --2- 172.21.15.70:0/1809009181 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f23d00b29f0 0x7f23d00b4eb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009be0 tx=0x7f23d4009260 comp rx=0 tx=0).stop 2024-06-04T05:02:38.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.945+0000 7f23e6a69700 1 -- 172.21.15.70:0/1809009181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23e006c470 msgr2=0x7f23e0102a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:38.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.945+0000 7f23e6a69700 1 --2- 172.21.15.70:0/1809009181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23e006c470 0x7f23e0102a40 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f23cc00e9c0 tx=0x7f23cc00ecd0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.945+0000 7f23e6a69700 1 -- 172.21.15.70:0/1809009181 shutdown_connections 2024-06-04T05:02:38.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.945+0000 7f23e6a69700 1 --2- 172.21.15.70:0/1809009181 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f23d00b29f0 0x7f23d00b4eb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.945+0000 7f23e6a69700 1 --2- 172.21.15.70:0/1809009181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e01095a0 0x7f23e0102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.945+0000 7f23e6a69700 1 --2- 172.21.15.70:0/1809009181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f23e006bab0 0x7f23e0102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.945+0000 7f23e6a69700 1 --2- 172.21.15.70:0/1809009181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23e006c470 0x7f23e0102a40 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:38.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.945+0000 7f23e6a69700 1 -- 172.21.15.70:0/1809009181 >> 172.21.15.70:0/1809009181 conn(0x7f23e00793b0 msgr2=0x7f23e0101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:38.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.945+0000 7f23e6a69700 1 -- 172.21.15.70:0/1809009181 shutdown_connections 2024-06-04T05:02:38.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:38.945+0000 7f23e6a69700 1 -- 172.21.15.70:0/1809009181 wait complete. 2024-06-04T05:02:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.012+0000 7f7d3756a640 1 -- 172.21.15.70:0/4139815503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d30107940 msgr2=0x7f7d30107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.012+0000 7f7d3756a640 1 --2- 172.21.15.70:0/4139815503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d30107940 0x7f7d30107d20 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009930 tx=0x7f7d2c02a420 comp rx=0 tx=0).stop 2024-06-04T05:02:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.012+0000 7f7d3756a640 1 -- 172.21.15.70:0/4139815503 shutdown_connections 2024-06-04T05:02:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.012+0000 7f7d3756a640 1 --2- 172.21.15.70:0/4139815503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d30101980 0x7f7d30101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.012+0000 7f7d3756a640 1 --2- 172.21.15.70:0/4139815503 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d30102340 0x7f7d3010e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.012+0000 7f7d3756a640 1 --2- 172.21.15.70:0/4139815503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d30107940 0x7f7d30107d20 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.012+0000 7f7d3756a640 1 -- 172.21.15.70:0/4139815503 >> 172.21.15.70:0/4139815503 conn(0x7f7d30078ad0 msgr2=0x7f7d30078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.012+0000 7f7d3756a640 1 -- 172.21.15.70:0/4139815503 shutdown_connections 2024-06-04T05:02:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.012+0000 7f7d3756a640 1 -- 172.21.15.70:0/4139815503 wait complete. 2024-06-04T05:02:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d3756a640 1 Processor -- start 2024-06-04T05:02:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d3756a640 1 -- start start 2024-06-04T05:02:39.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d3756a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d30101980 0x7f7d30199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d3756a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d30102340 0x7f7d3019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d3756a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d30107940 0x7f7d301a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d3756a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d30111ab0 con 0x7f7d30107940 2024-06-04T05:02:39.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d3756a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d30111930 con 0x7f7d30101980 2024-06-04T05:02:39.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d3756a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d30111c30 con 0x7f7d30102340 2024-06-04T05:02:39.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d35b07640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d30107940 0x7f7d301a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d35306640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d30101980 0x7f7d30199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d35b07640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d30107940 0x7f7d301a4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46728/0 (socket says 172.21.15.70:46728) 2024-06-04T05:02:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d35b07640 1 -- 172.21.15.70:0/1515033781 learned_addr learned my addr 172.21.15.70:0/1515033781 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d34b05640 1 --2- 172.21.15.70:0/1515033781 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d30102340 0x7f7d3019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d35b07640 1 -- 172.21.15.70:0/1515033781 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d30102340 msgr2=0x7f7d3019a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d35b07640 1 --2- 172.21.15.70:0/1515033781 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d30102340 0x7f7d3019a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d35b07640 1 -- 172.21.15.70:0/1515033781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d30101980 msgr2=0x7f7d30199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d35b07640 1 --2- 172.21.15.70:0/1515033781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d30101980 0x7f7d30199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d35b07640 1 -- 172.21.15.70:0/1515033781 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d301a5490 con 0x7f7d30107940 2024-06-04T05:02:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d35306640 1 --2- 172.21.15.70:0/1515033781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d30101980 0x7f7d30199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.013+0000 7f7d34b05640 1 --2- 172.21.15.70:0/1515033781 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d30102340 0x7f7d3019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.014+0000 7f7d35b07640 1 --2- 172.21.15.70:0/1515033781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d30107940 0x7f7d301a4f50 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7d2800eb10 tx=0x7f7d2800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.014+0000 7f7d267fc640 1 -- 172.21.15.70:0/1515033781 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d28004b20 con 0x7f7d30107940 2024-06-04T05:02:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.014+0000 7f7d3756a640 1 -- 172.21.15.70:0/1515033781 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d301a56c0 con 0x7f7d30107940 2024-06-04T05:02:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.014+0000 7f7d267fc640 1 -- 172.21.15.70:0/1515033781 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d28004cc0 con 0x7f7d30107940 2024-06-04T05:02:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.014+0000 7f7d267fc640 1 -- 172.21.15.70:0/1515033781 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d28010690 con 0x7f7d30107940 2024-06-04T05:02:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.014+0000 7f7d3756a640 1 -- 172.21.15.70:0/1515033781 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d301a5c30 con 0x7f7d30107940 2024-06-04T05:02:39.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.016+0000 7f7d267fc640 1 -- 172.21.15.70:0/1515033781 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7d28005020 con 0x7f7d30107940 2024-06-04T05:02:39.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.017+0000 7f7d3a283740 1 Processor -- start 2024-06-04T05:02:39.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.017+0000 7f7d3a283740 1 -- start start 2024-06-04T05:02:39.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.017+0000 7f7d267fc640 1 --2- 172.21.15.70:0/1515033781 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d1c0b2a10 0x7f7d1c0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.017+0000 7f7d3a283740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4649820 0x55e5f4649c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.017+0000 7f7d3a283740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f464a140 0x55e5f46544c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.017+0000 7f7d3a283740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4654a00 0x55e5f4656df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.017+0000 7f7d3a283740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e5f45bbc30 con 0x55e5f4649820 2024-06-04T05:02:39.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.017+0000 7f7d3a283740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e5f45ab7d0 con 0x55e5f464a140 2024-06-04T05:02:39.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.017+0000 7f7d3a283740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e5f45af810 con 0x55e5f4654a00 2024-06-04T05:02:39.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.017+0000 7f7d34b05640 1 --2- 172.21.15.70:0/1515033781 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d1c0b2a10 0x7f7d1c0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.017+0000 7f7d35b07640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4654a00 0x55e5f4656df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.018+0000 7f7d35b07640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4654a00 0x55e5f4656df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:38696/0 (socket says 172.21.15.70:38696) 2024-06-04T05:02:39.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.018+0000 7f7d35b07640 1 -- 172.21.15.70:0/855266137 learned_addr learned my addr 172.21.15.70:0/855266137 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:39.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.018+0000 7f7d267fc640 1 -- 172.21.15.70:0/1515033781 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f7d280e2a90 con 0x7f7d30107940 2024-06-04T05:02:39.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.018+0000 7f7d34b05640 1 --2- 172.21.15.70:0/855266137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f464a140 0x55e5f46544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.018+0000 7f7d35306640 1 --2- 172.21.15.70:0/855266137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4649820 0x55e5f4649c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.018+0000 7f7d35b07640 1 -- 172.21.15.70:0/855266137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f464a140 msgr2=0x55e5f46544c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:39.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.018+0000 7f7d35b07640 1 --2- 172.21.15.70:0/855266137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f464a140 0x55e5f46544c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.018+0000 7f7d35b07640 1 -- 172.21.15.70:0/855266137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4649820 msgr2=0x55e5f4649c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:39.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.018+0000 7f7d35b07640 1 --2- 172.21.15.70:0/855266137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4649820 0x55e5f4649c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.018+0000 7f7d35b07640 1 -- 172.21.15.70:0/855266137 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5f42c0210 con 0x55e5f4654a00 2024-06-04T05:02:39.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.018+0000 7f7d34b05640 1 --2- 172.21.15.70:0/1515033781 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d1c0b2a10 0x7f7d1c0b4ed0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d3006acb0 tx=0x7f7d20012c40 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:39.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.018+0000 7f7d35b07640 1 --2- 172.21.15.70:0/855266137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4654a00 0x55e5f4656df0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7d280e0180 tx=0x7f7d28000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6cf970bf2508c5c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:39.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.018+0000 7f7d1b7fe640 1 -- 172.21.15.70:0/855266137 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d280a94e0 con 0x55e5f4654a00 2024-06-04T05:02:39.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.018+0000 7f7d1b7fe640 1 -- 172.21.15.70:0/855266137 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d280a9880 con 0x55e5f4654a00 2024-06-04T05:02:39.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.018+0000 7f7d1b7fe640 1 -- 172.21.15.70:0/855266137 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d28082670 con 0x55e5f4654a00 2024-06-04T05:02:39.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.019+0000 7f7d3a283740 1 -- 172.21.15.70:0/855266137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4654a00 msgr2=0x55e5f4656df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:39.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.019+0000 7f7d3a283740 1 --2- 172.21.15.70:0/855266137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4654a00 0x55e5f4656df0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7d280e0180 tx=0x7f7d28000f40 comp rx=0 tx=0).stop 2024-06-04T05:02:39.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.019+0000 7f7d3a283740 1 -- 172.21.15.70:0/855266137 shutdown_connections 2024-06-04T05:02:39.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.019+0000 7f7d3a283740 1 --2- 172.21.15.70:0/855266137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f464a140 0x55e5f46544c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.019+0000 7f7d3a283740 1 --2- 172.21.15.70:0/855266137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4654a00 0x55e5f4656df0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.019+0000 7f7d3a283740 1 --2- 172.21.15.70:0/855266137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4649820 0x55e5f4649c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.019+0000 7f7d3a283740 1 -- 172.21.15.70:0/855266137 >> 172.21.15.70:0/855266137 conn(0x55e5f4645d30 msgr2=0x55e5f4646110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:39.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.019+0000 7f7d3a283740 1 -- 172.21.15.70:0/855266137 shutdown_connections 2024-06-04T05:02:39.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.019+0000 7f7d3a283740 1 -- 172.21.15.70:0/855266137 wait complete. 2024-06-04T05:02:39.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.019+0000 7f7d3a283740 1 -- start start 2024-06-04T05:02:39.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d3a283740 1 Processor -- start 2024-06-04T05:02:39.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d3a283740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4649820 0x55e5f4651890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d3a283740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f464a140 0x55e5f4651dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d3a283740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f4654a00 0x55e5f4652310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d3a283740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e5f4291790 con 0x55e5f464a140 2024-06-04T05:02:39.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d3a283740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e5f4567890 con 0x55e5f4654a00 2024-06-04T05:02:39.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d3a283740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e5f4657630 con 0x55e5f4649820 2024-06-04T05:02:39.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d3a283740 10 client.0 _ll_register_callbacks cb 0x55e5f42bfa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:39.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d34b05640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f464a140 0x55e5f4651dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d34b05640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f464a140 0x55e5f4651dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46734/0 (socket says 172.21.15.70:46734) 2024-06-04T05:02:39.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d34b05640 1 -- 172.21.15.70:0/450839563 learned_addr learned my addr 172.21.15.70:0/450839563 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:39.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d35306640 1 --2- 172.21.15.70:0/450839563 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4649820 0x55e5f4651890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d35b07640 1 --2- 172.21.15.70:0/450839563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f4654a00 0x55e5f4652310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d34b05640 1 -- 172.21.15.70:0/450839563 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4649820 msgr2=0x55e5f4651890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:39.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d34b05640 1 --2- 172.21.15.70:0/450839563 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4649820 0x55e5f4651890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d34b05640 1 -- 172.21.15.70:0/450839563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f4654a00 msgr2=0x55e5f4652310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:39.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d34b05640 1 --2- 172.21.15.70:0/450839563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f4654a00 0x55e5f4652310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d34b05640 1 -- 172.21.15.70:0/450839563 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5f438c620 con 0x55e5f464a140 2024-06-04T05:02:39.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d35306640 1 --2- 172.21.15.70:0/450839563 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4649820 0x55e5f4651890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:39.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.020+0000 7f7d35b07640 1 --2- 172.21.15.70:0/450839563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f4654a00 0x55e5f4652310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:39.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.021+0000 7f7d34b05640 1 --2- 172.21.15.70:0/450839563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f464a140 0x55e5f4651dd0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000eea0 tx=0x7f7d20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f91aef6cca6b316 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:39.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.021+0000 7f7d18ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:02:39.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.021+0000 7f7d18ff9640 1 -- 172.21.15.70:0/450839563 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d20004220 con 0x55e5f464a140 2024-06-04T05:02:39.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.021+0000 7f7d18ff9640 1 -- 172.21.15.70:0/450839563 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f7d200043c0 con 0x55e5f464a140 2024-06-04T05:02:39.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.021+0000 7f7d3a283740 1 -- 172.21.15.70:0/450839563 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d280a9880 con 0x55e5f464a140 2024-06-04T05:02:39.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.021+0000 7f7d18ff9640 1 -- 172.21.15.70:0/450839563 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7d2000fe10 con 0x55e5f464a140 2024-06-04T05:02:39.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.021+0000 7f7d3a283740 10 client.16956 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:39.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.021+0000 7f7d3a283740 1 -- 172.21.15.70:0/450839563 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e5f46632c0 con 0x55e5f464a140 2024-06-04T05:02:39.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.021+0000 7f7d18ff9640 1 -- 172.21.15.70:0/450839563 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d280a9880 con 0x55e5f464a140 2024-06-04T05:02:39.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.021+0000 7f7d18ff9640 1 -- 172.21.15.70:0/450839563 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f7d20003710 con 0x55e5f464a140 2024-06-04T05:02:39.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.022+0000 7f7d3a283740 10 client.16956 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:39.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.022+0000 7f7d3a283740 10 client.16956 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:02:39.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.022+0000 7f7d3a283740 20 client.16956 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:39.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.022+0000 7f7d3a283740 1 --2- 172.21.15.70:0/450839563 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e5f4664bf0 0x55e5f4667050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.022+0000 7f7d3a283740 4 client.16956 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:02:39.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.022+0000 7f7d3a283740 1 -- 172.21.15.70:0/450839563 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5f45bdb80 con 0x55e5f4664bf0 2024-06-04T05:02:39.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.022+0000 7f7d35306640 1 --2- 172.21.15.70:0/450839563 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e5f4664bf0 0x55e5f4667050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.022+0000 7f7d18ff9640 10 client.16956 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:39.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.022+0000 7f7d35306640 1 --2- 172.21.15.70:0/450839563 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e5f4664bf0 0x55e5f4667050 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ab356bc5d3adb36 server_cookie=9ae46bd7a93b6dc in_seq=0 out_seq=0 2024-06-04T05:02:39.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.023+0000 7f7d18ff9640 1 -- 172.21.15.70:0/450839563 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e5f45bdb80 con 0x55e5f4664bf0 2024-06-04T05:02:39.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.023+0000 7f7d18ff9640 10 client.16956 handle_command_reply: tid=0 2024-06-04T05:02:39.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.023+0000 7f7d3a283740 1 client.16956 shutdown 2024-06-04T05:02:39.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.023+0000 7f7d3a283740 1 -- 172.21.15.70:0/450839563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f464a140 msgr2=0x55e5f4651dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:39.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.023+0000 7f7d3a283740 1 --2- 172.21.15.70:0/450839563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f464a140 0x55e5f4651dd0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000eea0 tx=0x7f7d20004730 comp rx=0 tx=0).stop 2024-06-04T05:02:39.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.023+0000 7f7d3a283740 1 -- 172.21.15.70:0/450839563 shutdown_connections 2024-06-04T05:02:39.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.023+0000 7f7d3a283740 1 --2- 172.21.15.70:0/450839563 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e5f4664bf0 0x55e5f4667050 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.024+0000 7f7d3a283740 1 --2- 172.21.15.70:0/450839563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f4654a00 0x55e5f4652310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.024+0000 7f7d3a283740 1 --2- 172.21.15.70:0/450839563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f464a140 0x55e5f4651dd0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.024+0000 7f7d3a283740 1 --2- 172.21.15.70:0/450839563 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4649820 0x55e5f4651890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.024+0000 7f7d3a283740 1 -- 172.21.15.70:0/450839563 >> 172.21.15.70:0/450839563 conn(0x55e5f4645d30 msgr2=0x55e5f4644fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:39.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.024+0000 7f7d18ff9640 0 client.16956 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:39.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.024+0000 7f7d3a283740 1 -- 172.21.15.70:0/450839563 shutdown_connections 2024-06-04T05:02:39.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.024+0000 7f7d3a283740 1 -- 172.21.15.70:0/450839563 wait complete. 2024-06-04T05:02:39.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.024+0000 7f7d3a283740 20 client.16956 trim_cache size 0 max 16384 2024-06-04T05:02:39.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d3a283740 1 Processor -- start 2024-06-04T05:02:39.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d3a283740 1 -- start start 2024-06-04T05:02:39.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d3a283740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4649820 0x55e5f4645730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d3a283740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f464a140 0x55e5f464c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d3a283740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4654a00 0x55e5f46509f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d3a283740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301b0840 con 0x55e5f4649820 2024-06-04T05:02:39.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d3a283740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301b2380 con 0x55e5f464a140 2024-06-04T05:02:39.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d3a283740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301b0f10 con 0x55e5f4654a00 2024-06-04T05:02:39.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d35306640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4649820 0x55e5f4645730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d35306640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4649820 0x55e5f4645730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46736/0 (socket says 172.21.15.70:46736) 2024-06-04T05:02:39.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d35306640 1 -- 172.21.15.70:0/2378574340 learned_addr learned my addr 172.21.15.70:0/2378574340 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d35b07640 1 --2- 172.21.15.70:0/2378574340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4654a00 0x55e5f46509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d34b05640 1 --2- 172.21.15.70:0/2378574340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f464a140 0x55e5f464c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d35306640 1 -- 172.21.15.70:0/2378574340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4654a00 msgr2=0x55e5f46509f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d35306640 1 --2- 172.21.15.70:0/2378574340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4654a00 0x55e5f46509f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d35306640 1 -- 172.21.15.70:0/2378574340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f464a140 msgr2=0x55e5f464c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d35306640 1 --2- 172.21.15.70:0/2378574340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f464a140 0x55e5f464c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d35306640 1 -- 172.21.15.70:0/2378574340 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5f464fde0 con 0x55e5f4649820 2024-06-04T05:02:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.040+0000 7f7d35306640 1 --2- 172.21.15.70:0/2378574340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4649820 0x55e5f4645730 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009230 tx=0x7f7d2c0058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f61b6c71ac6f7d89 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.041+0000 7f7d1affd640 1 -- 172.21.15.70:0/2378574340 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c035040 con 0x55e5f4649820 2024-06-04T05:02:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.041+0000 7f7d1affd640 1 -- 172.21.15.70:0/2378574340 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d2c04c070 con 0x55e5f4649820 2024-06-04T05:02:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.041+0000 7f7d1affd640 1 -- 172.21.15.70:0/2378574340 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c0473c0 con 0x55e5f4649820 2024-06-04T05:02:39.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.041+0000 7f7d3a283740 1 -- 172.21.15.70:0/2378574340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4649820 msgr2=0x55e5f4645730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:39.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.041+0000 7f7d3a283740 1 --2- 172.21.15.70:0/2378574340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4649820 0x55e5f4645730 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009230 tx=0x7f7d2c0058d0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.041+0000 7f7d3a283740 1 -- 172.21.15.70:0/2378574340 shutdown_connections 2024-06-04T05:02:39.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.041+0000 7f7d3a283740 1 --2- 172.21.15.70:0/2378574340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f464a140 0x55e5f464c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.041+0000 7f7d3a283740 1 --2- 172.21.15.70:0/2378574340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4654a00 0x55e5f46509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.041+0000 7f7d3a283740 1 --2- 172.21.15.70:0/2378574340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4649820 0x55e5f4645730 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.041+0000 7f7d3a283740 1 -- 172.21.15.70:0/2378574340 >> 172.21.15.70:0/2378574340 conn(0x55e5f4645d30 msgr2=0x55e5f4651850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.042+0000 7f7d3a283740 1 -- 172.21.15.70:0/2378574340 shutdown_connections 2024-06-04T05:02:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.042+0000 7f7d3a283740 1 -- 172.21.15.70:0/2378574340 wait complete. 2024-06-04T05:02:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.042+0000 7f7d3a283740 1 -- start start 2024-06-04T05:02:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.042+0000 7f7d3a283740 1 Processor -- start 2024-06-04T05:02:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.042+0000 7f7d3a283740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4649820 0x55e5f4657e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.042+0000 7f7d3a283740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f464a140 0x55e5f4658380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.042+0000 7f7d3a283740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4654a00 0x55e5f46cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.042+0000 7f7d3a283740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301b1cb0 con 0x55e5f4654a00 2024-06-04T05:02:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.042+0000 7f7d3a283740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301b15e0 con 0x55e5f464a140 2024-06-04T05:02:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.042+0000 7f7d3a283740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301ae820 con 0x55e5f4649820 2024-06-04T05:02:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.042+0000 7f7d3a283740 10 client.0 _ll_register_callbacks cb 0x7f7d2000eb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d35b07640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4654a00 0x55e5f46cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d35306640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4649820 0x55e5f4657e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d35b07640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4654a00 0x55e5f46cd7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46740/0 (socket says 172.21.15.70:46740) 2024-06-04T05:02:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d35306640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4649820 0x55e5f4657e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:38716/0 (socket says 172.21.15.70:38716) 2024-06-04T05:02:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d35b07640 1 -- 172.21.15.70:0/2210464765 learned_addr learned my addr 172.21.15.70:0/2210464765 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d34b05640 1 --2- 172.21.15.70:0/2210464765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f464a140 0x55e5f4658380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d35306640 1 -- 172.21.15.70:0/2210464765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f464a140 msgr2=0x55e5f4658380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d35306640 1 --2- 172.21.15.70:0/2210464765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f464a140 0x55e5f4658380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d35306640 1 -- 172.21.15.70:0/2210464765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4654a00 msgr2=0x55e5f46cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d35306640 1 --2- 172.21.15.70:0/2210464765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4654a00 0x55e5f46cd7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d35306640 1 -- 172.21.15.70:0/2210464765 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5f46cde20 con 0x55e5f4649820 2024-06-04T05:02:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d34b05640 1 --2- 172.21.15.70:0/2210464765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f464a140 0x55e5f4658380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:02:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d35b07640 1 --2- 172.21.15.70:0/2210464765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4654a00 0x55e5f46cd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d35306640 1 --2- 172.21.15.70:0/2210464765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4649820 0x55e5f4657e40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c048000 tx=0x7f7d2c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=727aae035e83d52f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d18ff9640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:02:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d18ff9640 1 -- 172.21.15.70:0/2210464765 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c035030 con 0x55e5f4649820 2024-06-04T05:02:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d18ff9640 1 -- 172.21.15.70:0/2210464765 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f7d2c0037f0 con 0x55e5f4649820 2024-06-04T05:02:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.043+0000 7f7d3a283740 1 -- 172.21.15.70:0/2210464765 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d2c04c070 con 0x55e5f4649820 2024-06-04T05:02:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.044+0000 7f7d18ff9640 1 -- 172.21.15.70:0/2210464765 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7d2c004040 con 0x55e5f4649820 2024-06-04T05:02:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.044+0000 7f7d3a283740 10 client.26320 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.044+0000 7f7d3a283740 1 -- 172.21.15.70:0/2210464765 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e5f46cf130 con 0x55e5f4649820 2024-06-04T05:02:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.044+0000 7f7d18ff9640 1 -- 172.21.15.70:0/2210464765 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d2c04c070 con 0x55e5f4649820 2024-06-04T05:02:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.044+0000 7f7d18ff9640 1 -- 172.21.15.70:0/2210464765 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f7d2c047450 con 0x55e5f4649820 2024-06-04T05:02:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.044+0000 7f7d3a283740 10 client.26320 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.044+0000 7f7d3a283740 10 client.26320 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:02:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.044+0000 7f7d3a283740 20 client.26320 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.044+0000 7f7d3a283740 1 --2- 172.21.15.70:0/2210464765 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e5f46cf7a0 0x55e5f46d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.044+0000 7f7d3a283740 4 client.26320 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:02:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.044+0000 7f7d3a283740 1 -- 172.21.15.70:0/2210464765 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e5f45b7360 con 0x55e5f46cf7a0 2024-06-04T05:02:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.045+0000 7f7d34b05640 1 --2- 172.21.15.70:0/2210464765 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e5f46cf7a0 0x55e5f46d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.045+0000 7f7d34b05640 1 --2- 172.21.15.70:0/2210464765 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e5f46cf7a0 0x55e5f46d1b90 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=902bee1ba78d45e0 server_cookie=a564385317cd77d5 in_seq=0 out_seq=0 2024-06-04T05:02:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.045+0000 7f7d18ff9640 10 client.26320 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.064+0000 7f7d18ff9640 1 -- 172.21.15.70:0/2210464765 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55e5f45b7360 con 0x55e5f46cf7a0 2024-06-04T05:02:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.064+0000 7f7d18ff9640 10 client.26320 handle_command_reply: tid=0 2024-06-04T05:02:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.064+0000 7f7d3a283740 1 client.26320 shutdown 2024-06-04T05:02:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.064+0000 7f7d3a283740 1 -- 172.21.15.70:0/2210464765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4649820 msgr2=0x55e5f4657e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.064+0000 7f7d3a283740 1 --2- 172.21.15.70:0/2210464765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4649820 0x55e5f4657e40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c048000 tx=0x7f7d2c000bc0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.064+0000 7f7d3a283740 1 -- 172.21.15.70:0/2210464765 shutdown_connections 2024-06-04T05:02:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.064+0000 7f7d3a283740 1 --2- 172.21.15.70:0/2210464765 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e5f46cf7a0 0x55e5f46d1b90 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.064+0000 7f7d18ff9640 0 client.26320 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.064+0000 7f7d3a283740 1 --2- 172.21.15.70:0/2210464765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f464a140 0x55e5f4658380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.064+0000 7f7d3a283740 1 --2- 172.21.15.70:0/2210464765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5f4654a00 0x55e5f46cd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.064+0000 7f7d3a283740 1 --2- 172.21.15.70:0/2210464765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e5f4649820 0x55e5f4657e40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.064+0000 7f7d3a283740 1 -- 172.21.15.70:0/2210464765 >> 172.21.15.70:0/2210464765 conn(0x55e5f4645d30 msgr2=0x55e5f4650870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:39.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.064+0000 7f7d3a283740 1 -- 172.21.15.70:0/2210464765 shutdown_connections 2024-06-04T05:02:39.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.064+0000 7f7d3a283740 1 -- 172.21.15.70:0/2210464765 wait complete. 2024-06-04T05:02:39.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.065+0000 7f7d3a283740 20 client.26320 trim_cache size 0 max 16384 2024-06-04T05:02:39.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:02:39.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:02:39.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:02:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.065+0000 7f7d3756a640 1 -- 172.21.15.70:0/1515033781 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d1c0b2a10 msgr2=0x7f7d1c0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.065+0000 7f7d3756a640 1 --2- 172.21.15.70:0/1515033781 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d1c0b2a10 0x7f7d1c0b4ed0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d3006acb0 tx=0x7f7d20012c40 comp rx=0 tx=0).stop 2024-06-04T05:02:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.065+0000 7f7d3756a640 1 -- 172.21.15.70:0/1515033781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d30107940 msgr2=0x7f7d301a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.065+0000 7f7d3756a640 1 --2- 172.21.15.70:0/1515033781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d30107940 0x7f7d301a4f50 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7d2800eb10 tx=0x7f7d2800c2d0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.065+0000 7f7d3756a640 1 -- 172.21.15.70:0/1515033781 shutdown_connections 2024-06-04T05:02:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.065+0000 7f7d3756a640 1 --2- 172.21.15.70:0/1515033781 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d1c0b2a10 0x7f7d1c0b4ed0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.065+0000 7f7d3756a640 1 --2- 172.21.15.70:0/1515033781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d30107940 0x7f7d301a4f50 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.065+0000 7f7d3756a640 1 --2- 172.21.15.70:0/1515033781 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d30102340 0x7f7d3019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.065+0000 7f7d3756a640 1 --2- 172.21.15.70:0/1515033781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d30101980 0x7f7d30199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:39.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.065+0000 7f7d3756a640 1 -- 172.21.15.70:0/1515033781 >> 172.21.15.70:0/1515033781 conn(0x7f7d30078ad0 msgr2=0x7f7d30104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:39.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.066+0000 7f7d3756a640 1 -- 172.21.15.70:0/1515033781 shutdown_connections 2024-06-04T05:02:39.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:39.066+0000 7f7d3756a640 1 -- 172.21.15.70:0/1515033781 wait complete. 2024-06-04T05:02:39.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 83186 2.21 MB/sec execute 92 sec latency 22.282 ms 2024-06-04T05:02:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:39 smithi098 ceph-mon[35569]: pgmap v167: 97 pgs: 97 active+clean; 61 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-06-04T05:02:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:39 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:02:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:39 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:39 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:02:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:39 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:39 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:02:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:39 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:39 smithi070 ceph-mon[32631]: pgmap v167: 97 pgs: 97 active+clean; 61 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-06-04T05:02:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:39 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:02:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:39 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:39 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:02:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:39 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:39 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:02:39.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:39 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:39.899 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:02:39.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:39 smithi112 ceph-mon[36453]: pgmap v167: 97 pgs: 97 active+clean; 61 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-06-04T05:02:39.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:39 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:02:39.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:39 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:39.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:39 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:02:39.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:39 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:39.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:39 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:02:39.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:39 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:40.206 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:02:40.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:40 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:02:40.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:40.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:40 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:02:40.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:40 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:02:40.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:40.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:40 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:02:40.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:40.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:40 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:02:40.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:40.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:40.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 83859 2.20 MB/sec execute 93 sec latency 15.644 ms 2024-06-04T05:02:40.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:40 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:02:40.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:40.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:40 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:02:40.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:40 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:02:40.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:40.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:40 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:02:40.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:40.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:40 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:02:40.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:40.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:40.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:40 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:02:40.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:40.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:40 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:02:40.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:40 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:02:40.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:40.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:40 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:02:40.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:40.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:40 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:02:40.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:40.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:02:41.645 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:41 smithi070 ceph-mon[32631]: pgmap v168: 97 pgs: 97 active+clean; 61 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-06-04T05:02:41.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 84412 2.23 MB/sec execute 94 sec latency 15.931 ms 2024-06-04T05:02:41.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:41 smithi112 ceph-mon[36453]: pgmap v168: 97 pgs: 97 active+clean; 61 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-06-04T05:02:41.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:41 smithi098 ceph-mon[35569]: pgmap v168: 97 pgs: 97 active+clean; 61 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-06-04T05:02:42.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.051+0000 7f3631e9e700 1 -- 172.21.15.70:0/3166678701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f362c102440 msgr2=0x7f362c1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:42.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.051+0000 7f3631e9e700 1 --2- 172.21.15.70:0/3166678701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f362c102440 0x7f362c1028c0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f3620009990 tx=0x7f3620009ca0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.051+0000 7f3631e9e700 1 -- 172.21.15.70:0/3166678701 shutdown_connections 2024-06-04T05:02:42.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.051+0000 7f3631e9e700 1 --2- 172.21.15.70:0/3166678701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f362c102e00 0x7f362c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.051+0000 7f3631e9e700 1 --2- 172.21.15.70:0/3166678701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c108520 0x7f362c108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.051+0000 7f3631e9e700 1 --2- 172.21.15.70:0/3166678701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f362c102440 0x7f362c1028c0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.051+0000 7f3631e9e700 1 -- 172.21.15.70:0/3166678701 >> 172.21.15.70:0/3166678701 conn(0x7f362c0793f0 msgr2=0x7f362c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:42.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.051+0000 7f3631e9e700 1 -- 172.21.15.70:0/3166678701 shutdown_connections 2024-06-04T05:02:42.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.051+0000 7f3631e9e700 1 -- 172.21.15.70:0/3166678701 wait complete. 2024-06-04T05:02:42.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.052+0000 7f3631e9e700 1 Processor -- start 2024-06-04T05:02:42.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.052+0000 7f3631e9e700 1 -- start start 2024-06-04T05:02:42.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.052+0000 7f3631e9e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c102440 0x7f362c199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.052+0000 7f3631e9e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f362c102e00 0x7f362c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f3631e9e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f362c108520 0x7f362c19e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f3631e9e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f362c112480 con 0x7f362c102e00 2024-06-04T05:02:42.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f3631e9e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f362c112300 con 0x7f362c102440 2024-06-04T05:02:42.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f3631e9e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f362c112600 con 0x7f362c108520 2024-06-04T05:02:42.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f362affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f362c102e00 0x7f362c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f362affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f362c102e00 0x7f362c19a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46760/0 (socket says 172.21.15.70:46760) 2024-06-04T05:02:42.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f362affd700 1 -- 172.21.15.70:0/370779552 learned_addr learned my addr 172.21.15.70:0/370779552 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:42.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f362b7fe700 1 --2- 172.21.15.70:0/370779552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c102440 0x7f362c199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f362bfff700 1 --2- 172.21.15.70:0/370779552 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f362c108520 0x7f362c19e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f362affd700 1 -- 172.21.15.70:0/370779552 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f362c108520 msgr2=0x7f362c19e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:42.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f362affd700 1 --2- 172.21.15.70:0/370779552 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f362c108520 0x7f362c19e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f362affd700 1 -- 172.21.15.70:0/370779552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c102440 msgr2=0x7f362c199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:42.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f362affd700 1 --2- 172.21.15.70:0/370779552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c102440 0x7f362c199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f362affd700 1 -- 172.21.15.70:0/370779552 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f362c19ec10 con 0x7f362c102e00 2024-06-04T05:02:42.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f362b7fe700 1 --2- 172.21.15.70:0/370779552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c102440 0x7f362c199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:42.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f362bfff700 1 --2- 172.21.15.70:0/370779552 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f362c108520 0x7f362c19e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:42.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f362affd700 1 --2- 172.21.15.70:0/370779552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f362c102e00 0x7f362c19a1a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f3620006010 tx=0x7f36200047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:42.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f3628ff9700 1 -- 172.21.15.70:0/370779552 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620027070 con 0x7f362c102e00 2024-06-04T05:02:42.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.053+0000 7f3631e9e700 1 -- 172.21.15.70:0/370779552 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f362c19eea0 con 0x7f362c102e00 2024-06-04T05:02:42.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.054+0000 7f3628ff9700 1 -- 172.21.15.70:0/370779552 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3620022470 con 0x7f362c102e00 2024-06-04T05:02:42.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.054+0000 7f3631e9e700 1 -- 172.21.15.70:0/370779552 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f362c1a67c0 con 0x7f362c102e00 2024-06-04T05:02:42.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.054+0000 7f3628ff9700 1 -- 172.21.15.70:0/370779552 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620017650 con 0x7f362c102e00 2024-06-04T05:02:42.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.056+0000 7f3628ff9700 1 -- 172.21.15.70:0/370779552 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3620009d10 con 0x7f362c102e00 2024-06-04T05:02:42.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.057+0000 7f3628ff9700 1 --2- 172.21.15.70:0/370779552 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f36140b2a70 0x7f36140b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.057+0000 7f362b7fe700 1 --2- 172.21.15.70:0/370779552 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f36140b2a70 0x7f36140b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.057+0000 7f3628ff9700 1 -- 172.21.15.70:0/370779552 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f36200e2fd0 con 0x7f362c102e00 2024-06-04T05:02:42.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.057+0000 7f362b7fe700 1 --2- 172.21.15.70:0/370779552 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f36140b2a70 0x7f36140b4f30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f361c009fd0 tx=0x7f361c009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:42.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f363c733b80 1 Processor -- start 2024-06-04T05:02:42.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f363c733b80 1 -- start start 2024-06-04T05:02:42.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f363c733b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff82320 0x55ee4ff82700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f363c733b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff7cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f363c733b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff7d260 0x55ee4ff89080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f363c733b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ee4fedad70 con 0x55ee4ff7d260 2024-06-04T05:02:42.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f363c733b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ee4feca910 con 0x55ee4ff82320 2024-06-04T05:02:42.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f363c733b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ee4fece950 con 0x55ee4ff7c8a0 2024-06-04T05:02:42.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f362bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff7d260 0x55ee4ff89080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f362affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff7cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f362bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff7d260 0x55ee4ff89080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46764/0 (socket says 172.21.15.70:46764) 2024-06-04T05:02:42.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f362bfff700 1 -- 172.21.15.70:0/3586212064 learned_addr learned my addr 172.21.15.70:0/3586212064 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:42.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f362b7fe700 1 --2- 172.21.15.70:0/3586212064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff82320 0x55ee4ff82700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f362bfff700 1 -- 172.21.15.70:0/3586212064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff7c8a0 msgr2=0x55ee4ff7cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:42.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f362bfff700 1 --2- 172.21.15.70:0/3586212064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff7cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f362bfff700 1 -- 172.21.15.70:0/3586212064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff82320 msgr2=0x55ee4ff82700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:42.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f362bfff700 1 --2- 172.21.15.70:0/3586212064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff82320 0x55ee4ff82700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f362bfff700 1 -- 172.21.15.70:0/3586212064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee4fe73fb0 con 0x55ee4ff7d260 2024-06-04T05:02:42.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.058+0000 7f362affd700 1 --2- 172.21.15.70:0/3586212064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff7cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:42.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.059+0000 7f362bfff700 1 --2- 172.21.15.70:0/3586212064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff7d260 0x55ee4ff89080 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3624003eb0 tx=0x7f3624009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ccb3bd57274ce9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:42.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.059+0000 7f3611ffb700 1 -- 172.21.15.70:0/3586212064 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3624019020 con 0x55ee4ff7d260 2024-06-04T05:02:42.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.059+0000 7f3611ffb700 1 -- 172.21.15.70:0/3586212064 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f36240092d0 con 0x55ee4ff7d260 2024-06-04T05:02:42.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.059+0000 7f3611ffb700 1 -- 172.21.15.70:0/3586212064 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362400e750 con 0x55ee4ff7d260 2024-06-04T05:02:42.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.060+0000 7f363c733b80 1 -- 172.21.15.70:0/3586212064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff7d260 msgr2=0x55ee4ff89080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:42.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.060+0000 7f363c733b80 1 --2- 172.21.15.70:0/3586212064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff7d260 0x55ee4ff89080 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3624003eb0 tx=0x7f3624009920 comp rx=0 tx=0).stop 2024-06-04T05:02:42.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.060+0000 7f363c733b80 1 -- 172.21.15.70:0/3586212064 shutdown_connections 2024-06-04T05:02:42.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.060+0000 7f363c733b80 1 --2- 172.21.15.70:0/3586212064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff7d260 0x55ee4ff89080 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.060+0000 7f363c733b80 1 --2- 172.21.15.70:0/3586212064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff82320 0x55ee4ff82700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.060+0000 7f363c733b80 1 --2- 172.21.15.70:0/3586212064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff7cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.060+0000 7f363c733b80 1 -- 172.21.15.70:0/3586212064 >> 172.21.15.70:0/3586212064 conn(0x55ee4fea8240 msgr2=0x55ee4ff773e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:42.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.060+0000 7f363c733b80 1 -- 172.21.15.70:0/3586212064 shutdown_connections 2024-06-04T05:02:42.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.060+0000 7f363c733b80 1 -- 172.21.15.70:0/3586212064 wait complete. 2024-06-04T05:02:42.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.060+0000 7f363c733b80 1 -- start start 2024-06-04T05:02:42.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.060+0000 7f363c733b80 1 Processor -- start 2024-06-04T05:02:42.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.060+0000 7f363c733b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff88cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f363c733b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff7d260 0x55ee4ff85fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f363c733b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff82320 0x55ee4ff864f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f363c733b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ee4fdd0fa0 con 0x55ee4ff7d260 2024-06-04T05:02:42.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f363c733b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ee4fe32da0 con 0x55ee4ff7c8a0 2024-06-04T05:02:42.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f363c733b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ee4ff896e0 con 0x55ee4ff82320 2024-06-04T05:02:42.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f363c733b80 10 client.0 _ll_register_callbacks cb 0x55ee4fe65950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:42.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f362b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff88cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f362b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff88cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:51940/0 (socket says 172.21.15.70:51940) 2024-06-04T05:02:42.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f362b7fe700 1 -- 172.21.15.70:0/1401731461 learned_addr learned my addr 172.21.15.70:0/1401731461 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:42.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f362affd700 1 --2- 172.21.15.70:0/1401731461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff7d260 0x55ee4ff85fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f362bfff700 1 --2- 172.21.15.70:0/1401731461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff82320 0x55ee4ff864f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f362b7fe700 1 -- 172.21.15.70:0/1401731461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff82320 msgr2=0x55ee4ff864f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:42.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f362b7fe700 1 --2- 172.21.15.70:0/1401731461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff82320 0x55ee4ff864f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f362b7fe700 1 -- 172.21.15.70:0/1401731461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff7d260 msgr2=0x55ee4ff85fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:42.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f362b7fe700 1 --2- 172.21.15.70:0/1401731461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff7d260 0x55ee4ff85fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f362b7fe700 1 -- 172.21.15.70:0/1401731461 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee4fe1a8d0 con 0x55ee4ff7c8a0 2024-06-04T05:02:42.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f362bfff700 1 --2- 172.21.15.70:0/1401731461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff82320 0x55ee4ff864f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:02:42.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f362affd700 1 --2- 172.21.15.70:0/1401731461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff7d260 0x55ee4ff85fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:42.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f362b7fe700 1 --2- 172.21.15.70:0/1401731461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff88cb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f361c00ec90 tx=0x7f361c0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20516dde82c34244 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:42.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f360affd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:02:42.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.061+0000 7f360affd700 1 -- 172.21.15.70:0/1401731461 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361c01fe10 con 0x55ee4ff7c8a0 2024-06-04T05:02:42.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.062+0000 7f360affd700 1 -- 172.21.15.70:0/1401731461 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f361c004050 con 0x55ee4ff7c8a0 2024-06-04T05:02:42.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.062+0000 7f363c733b80 1 -- 172.21.15.70:0/1401731461 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36240092d0 con 0x55ee4ff7c8a0 2024-06-04T05:02:42.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.062+0000 7f360affd700 1 -- 172.21.15.70:0/1401731461 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f361c004290 con 0x55ee4ff7c8a0 2024-06-04T05:02:42.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.062+0000 7f363c733b80 10 client.26270 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:42.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.062+0000 7f363c733b80 1 -- 172.21.15.70:0/1401731461 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ee4fe84760 con 0x55ee4ff7c8a0 2024-06-04T05:02:42.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.062+0000 7f360affd700 1 -- 172.21.15.70:0/1401731461 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f36240092d0 con 0x55ee4ff7c8a0 2024-06-04T05:02:42.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.062+0000 7f360affd700 1 -- 172.21.15.70:0/1401731461 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f361c026020 con 0x55ee4ff7c8a0 2024-06-04T05:02:42.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.062+0000 7f363c733b80 10 client.26270 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:42.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.062+0000 7f363c733b80 10 client.26270 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:02:42.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.062+0000 7f363c733b80 20 client.26270 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:42.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.062+0000 7f363c733b80 1 --2- 172.21.15.70:0/1401731461 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ee4ff96160 0x55ee4ff98550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.062+0000 7f363c733b80 4 client.26270 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:02:42.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.062+0000 7f363c733b80 1 -- 172.21.15.70:0/1401731461 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee4fedccc0 con 0x55ee4ff96160 2024-06-04T05:02:42.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.062+0000 7f362affd700 1 --2- 172.21.15.70:0/1401731461 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ee4ff96160 0x55ee4ff98550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.063+0000 7f362affd700 1 --2- 172.21.15.70:0/1401731461 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ee4ff96160 0x55ee4ff98550 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78780804becea443 server_cookie=2af7b49fad0883fa in_seq=0 out_seq=0 2024-06-04T05:02:42.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.063+0000 7f360affd700 10 client.26270 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:42.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.064+0000 7f360affd700 1 -- 172.21.15.70:0/1401731461 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ee4fedccc0 con 0x55ee4ff96160 2024-06-04T05:02:42.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.064+0000 7f360affd700 10 client.26270 handle_command_reply: tid=0 2024-06-04T05:02:42.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.064+0000 7f363c733b80 1 client.26270 shutdown 2024-06-04T05:02:42.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.064+0000 7f363c733b80 1 -- 172.21.15.70:0/1401731461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7c8a0 msgr2=0x55ee4ff88cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:42.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.064+0000 7f363c733b80 1 --2- 172.21.15.70:0/1401731461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff88cb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f361c00ec90 tx=0x7f361c0047c0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.064+0000 7f363c733b80 1 -- 172.21.15.70:0/1401731461 shutdown_connections 2024-06-04T05:02:42.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.064+0000 7f363c733b80 1 --2- 172.21.15.70:0/1401731461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff82320 0x55ee4ff864f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.064+0000 7f363c733b80 1 --2- 172.21.15.70:0/1401731461 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ee4ff96160 0x55ee4ff98550 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.064+0000 7f363c733b80 1 --2- 172.21.15.70:0/1401731461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff88cb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.064+0000 7f363c733b80 1 --2- 172.21.15.70:0/1401731461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff7d260 0x55ee4ff85fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.064+0000 7f360affd700 0 client.26270 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:42.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.064+0000 7f363c733b80 1 -- 172.21.15.70:0/1401731461 >> 172.21.15.70:0/1401731461 conn(0x55ee4fea8240 msgr2=0x55ee4ff76af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:42.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.064+0000 7f363c733b80 1 -- 172.21.15.70:0/1401731461 shutdown_connections 2024-06-04T05:02:42.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.064+0000 7f363c733b80 1 -- 172.21.15.70:0/1401731461 wait complete. 2024-06-04T05:02:42.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.064+0000 7f363c733b80 20 client.26270 trim_cache size 0 max 16384 2024-06-04T05:02:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.087+0000 7f363c733b80 1 Processor -- start 2024-06-04T05:02:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.087+0000 7f363c733b80 1 -- start start 2024-06-04T05:02:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.087+0000 7f363c733b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff86bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.087+0000 7f363c733b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff7d260 0x55ee4ff805e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.087+0000 7f363c733b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff82320 0x55ee4ff77270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.087+0000 7f363c733b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f362c1aff10 con 0x55ee4ff82320 2024-06-04T05:02:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.087+0000 7f363c733b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f362c1b1a50 con 0x55ee4ff7c8a0 2024-06-04T05:02:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.087+0000 7f363c733b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f362c1b05e0 con 0x55ee4ff7d260 2024-06-04T05:02:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.087+0000 7f362bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff82320 0x55ee4ff77270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.087+0000 7f362bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff82320 0x55ee4ff77270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46786/0 (socket says 172.21.15.70:46786) 2024-06-04T05:02:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.087+0000 7f362bfff700 1 -- 172.21.15.70:0/4194043490 learned_addr learned my addr 172.21.15.70:0/4194043490 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.087+0000 7f362affd700 1 --2- 172.21.15.70:0/4194043490 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff7d260 0x55ee4ff805e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.088+0000 7f362b7fe700 1 --2- 172.21.15.70:0/4194043490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff86bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.088+0000 7f362bfff700 1 -- 172.21.15.70:0/4194043490 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff7d260 msgr2=0x55ee4ff805e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.088+0000 7f362bfff700 1 --2- 172.21.15.70:0/4194043490 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff7d260 0x55ee4ff805e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.088+0000 7f362bfff700 1 -- 172.21.15.70:0/4194043490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7c8a0 msgr2=0x55ee4ff86bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.088+0000 7f362bfff700 1 --2- 172.21.15.70:0/4194043490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff86bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.088+0000 7f362bfff700 1 -- 172.21.15.70:0/4194043490 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee4ff81d00 con 0x55ee4ff82320 2024-06-04T05:02:42.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.088+0000 7f362bfff700 1 --2- 172.21.15.70:0/4194043490 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff82320 0x55ee4ff77270 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3624009510 tx=0x7f3624006070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c7beddda68dc83b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:42.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.088+0000 7f36117fa700 1 -- 172.21.15.70:0/4194043490 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3624019040 con 0x55ee4ff82320 2024-06-04T05:02:42.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.088+0000 7f36117fa700 1 -- 172.21.15.70:0/4194043490 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3624022070 con 0x55ee4ff82320 2024-06-04T05:02:42.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.088+0000 7f36117fa700 1 -- 172.21.15.70:0/4194043490 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3624007d90 con 0x55ee4ff82320 2024-06-04T05:02:42.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.089+0000 7f363c733b80 1 -- 172.21.15.70:0/4194043490 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff82320 msgr2=0x55ee4ff77270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:42.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.089+0000 7f363c733b80 1 --2- 172.21.15.70:0/4194043490 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff82320 0x55ee4ff77270 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3624009510 tx=0x7f3624006070 comp rx=0 tx=0).stop 2024-06-04T05:02:42.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.089+0000 7f363c733b80 1 -- 172.21.15.70:0/4194043490 shutdown_connections 2024-06-04T05:02:42.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.089+0000 7f363c733b80 1 --2- 172.21.15.70:0/4194043490 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff82320 0x55ee4ff77270 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.089+0000 7f363c733b80 1 --2- 172.21.15.70:0/4194043490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff86bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.089+0000 7f363c733b80 1 --2- 172.21.15.70:0/4194043490 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff7d260 0x55ee4ff805e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.089+0000 7f363c733b80 1 -- 172.21.15.70:0/4194043490 >> 172.21.15.70:0/4194043490 conn(0x55ee4fea8240 msgr2=0x55ee4ff877e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:42.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.089+0000 7f363c733b80 1 -- 172.21.15.70:0/4194043490 shutdown_connections 2024-06-04T05:02:42.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.089+0000 7f363c733b80 1 -- 172.21.15.70:0/4194043490 wait complete. 2024-06-04T05:02:42.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.089+0000 7f363c733b80 1 -- start start 2024-06-04T05:02:42.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.090+0000 7f363c733b80 1 Processor -- start 2024-06-04T05:02:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.090+0000 7f363c733b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff8edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.090+0000 7f363c733b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7d260 0x55ee4ff9d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.090+0000 7f363c733b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff82320 0x55ee4ff9d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.090+0000 7f363c733b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f362c1b1380 con 0x55ee4ff82320 2024-06-04T05:02:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.090+0000 7f363c733b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f362c1b0cb0 con 0x55ee4ff7d260 2024-06-04T05:02:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.090+0000 7f363c733b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f362c1adeb0 con 0x55ee4ff7c8a0 2024-06-04T05:02:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.090+0000 7f363c733b80 10 client.0 _ll_register_callbacks cb 0x55ee4fe65950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.090+0000 7f362bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff82320 0x55ee4ff9d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.090+0000 7f362b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff8edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.090+0000 7f362bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff82320 0x55ee4ff9d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46796/0 (socket says 172.21.15.70:46796) 2024-06-04T05:02:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.090+0000 7f362bfff700 1 -- 172.21.15.70:0/3208700821 learned_addr learned my addr 172.21.15.70:0/3208700821 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.090+0000 7f362affd700 1 --2- 172.21.15.70:0/3208700821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7d260 0x55ee4ff9d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.091+0000 7f362affd700 1 -- 172.21.15.70:0/3208700821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff7c8a0 msgr2=0x55ee4ff8edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.091+0000 7f362affd700 1 --2- 172.21.15.70:0/3208700821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff8edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.091+0000 7f362affd700 1 -- 172.21.15.70:0/3208700821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff82320 msgr2=0x55ee4ff9d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.091+0000 7f362affd700 1 --2- 172.21.15.70:0/3208700821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff82320 0x55ee4ff9d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.091+0000 7f362affd700 1 -- 172.21.15.70:0/3208700821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee4ff9e020 con 0x55ee4ff7d260 2024-06-04T05:02:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.091+0000 7f362b7fe700 1 --2- 172.21.15.70:0/3208700821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff8edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.091+0000 7f362bfff700 1 --2- 172.21.15.70:0/3208700821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff82320 0x55ee4ff9d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.091+0000 7f362affd700 1 --2- 172.21.15.70:0/3208700821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7d260 0x55ee4ff9d4a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f36200e0720 tx=0x7f362001a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d7803640e6fa47 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.091+0000 7f360affd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:02:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.091+0000 7f360affd700 1 -- 172.21.15.70:0/3208700821 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36200aa340 con 0x55ee4ff7d260 2024-06-04T05:02:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.091+0000 7f363c733b80 1 -- 172.21.15.70:0/3208700821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3624022070 con 0x55ee4ff7d260 2024-06-04T05:02:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.091+0000 7f360affd700 1 -- 172.21.15.70:0/3208700821 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f36200aa7b0 con 0x55ee4ff7d260 2024-06-04T05:02:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.091+0000 7f360affd700 1 -- 172.21.15.70:0/3208700821 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f36200ae6f0 con 0x55ee4ff7d260 2024-06-04T05:02:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.091+0000 7f363c733b80 10 client.26276 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.091+0000 7f363c733b80 1 -- 172.21.15.70:0/3208700821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ee50000b60 con 0x55ee4ff7d260 2024-06-04T05:02:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.091+0000 7f360affd700 1 -- 172.21.15.70:0/3208700821 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3624022070 con 0x55ee4ff7d260 2024-06-04T05:02:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.092+0000 7f360affd700 1 -- 172.21.15.70:0/3208700821 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f36200c4ba0 con 0x55ee4ff7d260 2024-06-04T05:02:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.092+0000 7f363c733b80 10 client.26276 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.092+0000 7f363c733b80 10 client.26276 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:02:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.092+0000 7f363c733b80 20 client.26276 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.092+0000 7f363c733b80 1 --2- 172.21.15.70:0/3208700821 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ee50001220 0x55ee50003640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.092+0000 7f363c733b80 4 client.26276 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:02:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.092+0000 7f363c733b80 1 -- 172.21.15.70:0/3208700821 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ee4fed64a0 con 0x55ee50001220 2024-06-04T05:02:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.092+0000 7f362b7fe700 1 --2- 172.21.15.70:0/3208700821 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ee50001220 0x55ee50003640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.092+0000 7f360affd700 10 client.26276 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.092+0000 7f362b7fe700 1 --2- 172.21.15.70:0/3208700821 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ee50001220 0x55ee50003640 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5bb8e6cc21f721 server_cookie=8886e1aa7c79d25f in_seq=0 out_seq=0 2024-06-04T05:02:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.093+0000 7f360affd700 1 -- 172.21.15.70:0/3208700821 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ee4fed64a0 con 0x55ee50001220 2024-06-04T05:02:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.093+0000 7f360affd700 10 client.26276 handle_command_reply: tid=0 2024-06-04T05:02:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.093+0000 7f363c733b80 1 client.26276 shutdown 2024-06-04T05:02:42.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.093+0000 7f363c733b80 1 -- 172.21.15.70:0/3208700821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7d260 msgr2=0x55ee4ff9d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:42.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.093+0000 7f363c733b80 1 --2- 172.21.15.70:0/3208700821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7d260 0x55ee4ff9d4a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f36200e0720 tx=0x7f362001a040 comp rx=0 tx=0).stop 2024-06-04T05:02:42.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.093+0000 7f363c733b80 1 -- 172.21.15.70:0/3208700821 shutdown_connections 2024-06-04T05:02:42.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.093+0000 7f363c733b80 1 --2- 172.21.15.70:0/3208700821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee4ff82320 0x55ee4ff9d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.093+0000 7f363c733b80 1 --2- 172.21.15.70:0/3208700821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee4ff7c8a0 0x55ee4ff8edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.093+0000 7f363c733b80 1 --2- 172.21.15.70:0/3208700821 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ee50001220 0x55ee50003640 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.093+0000 7f363c733b80 1 --2- 172.21.15.70:0/3208700821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee4ff7d260 0x55ee4ff9d4a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.093+0000 7f360affd700 0 client.26276 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:42.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.093+0000 7f363c733b80 1 -- 172.21.15.70:0/3208700821 >> 172.21.15.70:0/3208700821 conn(0x55ee4fea8240 msgr2=0x55ee4ff7c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:42.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.093+0000 7f363c733b80 1 -- 172.21.15.70:0/3208700821 shutdown_connections 2024-06-04T05:02:42.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.093+0000 7f363c733b80 1 -- 172.21.15.70:0/3208700821 wait complete. 2024-06-04T05:02:42.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.093+0000 7f363c733b80 20 client.26276 trim_cache size 0 max 16384 2024-06-04T05:02:42.100 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:02:42.101 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:02:42.101 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:02:42.101 INFO:teuthology.orchestra.run.smithi070.stdout: "d9166179-e851-420b-bf88-bfe6b68f34e7": { 2024-06-04T05:02:42.101 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:02:42.101 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "d9166179-e851-420b-bf88-bfe6b68f34e7", 2024-06-04T05:02:42.101 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:02:42.101 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:02:42.101 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:02:42.101 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:02:42.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.094+0000 7f3631e9e700 1 -- 172.21.15.70:0/370779552 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f36140b2a70 msgr2=0x7f36140b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:42.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.094+0000 7f3631e9e700 1 --2- 172.21.15.70:0/370779552 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f36140b2a70 0x7f36140b4f30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f361c009fd0 tx=0x7f361c009260 comp rx=0 tx=0).stop 2024-06-04T05:02:42.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.094+0000 7f3631e9e700 1 -- 172.21.15.70:0/370779552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f362c102e00 msgr2=0x7f362c19a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:42.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.094+0000 7f3631e9e700 1 --2- 172.21.15.70:0/370779552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f362c102e00 0x7f362c19a1a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f3620006010 tx=0x7f36200047b0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.094+0000 7f3631e9e700 1 -- 172.21.15.70:0/370779552 shutdown_connections 2024-06-04T05:02:42.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.094+0000 7f3631e9e700 1 --2- 172.21.15.70:0/370779552 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f36140b2a70 0x7f36140b4f30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.094+0000 7f3631e9e700 1 --2- 172.21.15.70:0/370779552 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f362c108520 0x7f362c19e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.094+0000 7f3631e9e700 1 --2- 172.21.15.70:0/370779552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c102440 0x7f362c199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.094+0000 7f3631e9e700 1 --2- 172.21.15.70:0/370779552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f362c102e00 0x7f362c19a1a0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:42.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.094+0000 7f3631e9e700 1 -- 172.21.15.70:0/370779552 >> 172.21.15.70:0/370779552 conn(0x7f362c0793f0 msgr2=0x7f362c06b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:42.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.094+0000 7f3631e9e700 1 -- 172.21.15.70:0/370779552 shutdown_connections 2024-06-04T05:02:42.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:02:42.094+0000 7f3631e9e700 1 -- 172.21.15.70:0/370779552 wait complete. 2024-06-04T05:02:42.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 84974 2.24 MB/sec execute 95 sec latency 15.823 ms 2024-06-04T05:02:42.957 INFO:tasks.cephfs.filesystem:scrub status for tag:d9166179-e851-420b-bf88-bfe6b68f34e7 - {'path': '/', 'tag': 'd9166179-e851-420b-bf88-bfe6b68f34e7', 'options': 'recursive,force'} 2024-06-04T05:02:43.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 85477 2.26 MB/sec execute 96 sec latency 12.103 ms 2024-06-04T05:02:43.797 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:43 smithi098 ceph-mon[35569]: pgmap v169: 97 pgs: 97 active+clean; 57 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.9 MiB/s wr, 199 op/s 2024-06-04T05:02:43.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:43 smithi070 ceph-mon[32631]: pgmap v169: 97 pgs: 97 active+clean; 57 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.9 MiB/s wr, 199 op/s 2024-06-04T05:02:43.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:43 smithi112 ceph-mon[36453]: pgmap v169: 97 pgs: 97 active+clean; 57 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.9 MiB/s wr, 199 op/s 2024-06-04T05:02:44.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 85763 2.24 MB/sec execute 97 sec latency 12.694 ms 2024-06-04T05:02:45.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 86312 2.24 MB/sec execute 98 sec latency 11.082 ms 2024-06-04T05:02:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:45 smithi070 ceph-mon[32631]: pgmap v170: 97 pgs: 97 active+clean; 57 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.2 MiB/s wr, 141 op/s 2024-06-04T05:02:45.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:45 smithi112 ceph-mon[36453]: pgmap v170: 97 pgs: 97 active+clean; 57 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.2 MiB/s wr, 141 op/s 2024-06-04T05:02:45.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:45 smithi098 ceph-mon[35569]: pgmap v170: 97 pgs: 97 active+clean; 57 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.2 MiB/s wr, 141 op/s 2024-06-04T05:02:46.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 86887 2.25 MB/sec execute 99 sec latency 15.363 ms 2024-06-04T05:02:47.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 87567 2.25 MB/sec execute 100 sec latency 15.778 ms 2024-06-04T05:02:47.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:47 smithi070 ceph-mon[32631]: pgmap v171: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-06-04T05:02:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:47 smithi112 ceph-mon[36453]: pgmap v171: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-06-04T05:02:47.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:47 smithi098 ceph-mon[35569]: pgmap v171: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-06-04T05:02:48.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 88196 2.28 MB/sec execute 101 sec latency 15.503 ms 2024-06-04T05:02:48.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:48 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:02:48.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:48 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:02:48.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:48 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:02:48.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:48 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:02:48.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:48 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:02:48.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:48 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:02:49.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 88756 2.31 MB/sec execute 102 sec latency 15.203 ms 2024-06-04T05:02:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:49 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:02:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:49 smithi070 ceph-mon[32631]: pgmap v172: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 1.3 MiB/s wr, 146 op/s 2024-06-04T05:02:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:49 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:02:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:49 smithi112 ceph-mon[36453]: pgmap v172: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 1.3 MiB/s wr, 146 op/s 2024-06-04T05:02:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:49 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:02:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:49 smithi098 ceph-mon[35569]: pgmap v172: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 1.3 MiB/s wr, 146 op/s 2024-06-04T05:02:50.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 89125 2.30 MB/sec execute 103 sec latency 11.894 ms 2024-06-04T05:02:51.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 89516 2.28 MB/sec execute 104 sec latency 10.987 ms 2024-06-04T05:02:51.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:51 smithi070 ceph-mon[32631]: pgmap v173: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 1.3 MiB/s wr, 146 op/s 2024-06-04T05:02:51.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:51 smithi112 ceph-mon[36453]: pgmap v173: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 1.3 MiB/s wr, 146 op/s 2024-06-04T05:02:51.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:51 smithi098 ceph-mon[35569]: pgmap v173: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 1.3 MiB/s wr, 146 op/s 2024-06-04T05:02:52.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.162+0000 7fcfcb2e3640 1 -- 172.21.15.70:0/966032536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfc4102340 msgr2=0x7fcfc410e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.162+0000 7fcfcb2e3640 1 --2- 172.21.15.70:0/966032536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfc4102340 0x7fcfc410e800 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc009cd0 tx=0x7fcfbc02a8e0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.162+0000 7fcfcb2e3640 1 -- 172.21.15.70:0/966032536 shutdown_connections 2024-06-04T05:02:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.162+0000 7fcfcb2e3640 1 --2- 172.21.15.70:0/966032536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfc4101980 0x7fcfc4101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.162+0000 7fcfcb2e3640 1 --2- 172.21.15.70:0/966032536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfc4102340 0x7fcfc410e800 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.162+0000 7fcfcb2e3640 1 --2- 172.21.15.70:0/966032536 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcfc4107940 0x7fcfc4107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.162+0000 7fcfcb2e3640 1 -- 172.21.15.70:0/966032536 >> 172.21.15.70:0/966032536 conn(0x7fcfc4078ad0 msgr2=0x7fcfc4078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.162+0000 7fcfcb2e3640 1 -- 172.21.15.70:0/966032536 shutdown_connections 2024-06-04T05:02:52.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.162+0000 7fcfcb2e3640 1 -- 172.21.15.70:0/966032536 wait complete. 2024-06-04T05:02:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.163+0000 7fcfcb2e3640 1 Processor -- start 2024-06-04T05:02:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.163+0000 7fcfcb2e3640 1 -- start start 2024-06-04T05:02:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.163+0000 7fcfcb2e3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfc4101980 0x7fcfc4197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.163+0000 7fcfcb2e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfc4102340 0x7fcfc4198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.163+0000 7fcfcb2e3640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcfc4107940 0x7fcfc4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.163+0000 7fcfcb2e3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc4111ab0 con 0x7fcfc4101980 2024-06-04T05:02:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.163+0000 7fcfcb2e3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc4111930 con 0x7fcfc4102340 2024-06-04T05:02:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.163+0000 7fcfcb2e3640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc4111c30 con 0x7fcfc4107940 2024-06-04T05:02:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.163+0000 7fcfc907f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfc4101980 0x7fcfc4197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.163+0000 7fcfc907f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfc4101980 0x7fcfc4197ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58846/0 (socket says 172.21.15.70:58846) 2024-06-04T05:02:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.163+0000 7fcfc907f640 1 -- 172.21.15.70:0/603625881 learned_addr learned my addr 172.21.15.70:0/603625881 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.163+0000 7fcfc9880640 1 --2- 172.21.15.70:0/603625881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcfc4107940 0x7fcfc4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.163+0000 7fcfc887e640 1 --2- 172.21.15.70:0/603625881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfc4102340 0x7fcfc4198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.163+0000 7fcfc907f640 1 -- 172.21.15.70:0/603625881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcfc4107940 msgr2=0x7fcfc4198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.163+0000 7fcfc907f640 1 --2- 172.21.15.70:0/603625881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcfc4107940 0x7fcfc4198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.163+0000 7fcfc907f640 1 -- 172.21.15.70:0/603625881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfc4102340 msgr2=0x7fcfc4198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.163+0000 7fcfc907f640 1 --2- 172.21.15.70:0/603625881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfc4102340 0x7fcfc4198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.164+0000 7fcfc907f640 1 -- 172.21.15.70:0/603625881 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfc41a7550 con 0x7fcfc4101980 2024-06-04T05:02:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.164+0000 7fcfc907f640 1 --2- 172.21.15.70:0/603625881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfc4101980 0x7fcfc4197ce0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fcfc000e9f0 tx=0x7fcfc000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.164+0000 7fcfba7fc640 1 -- 172.21.15.70:0/603625881 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc000cd80 con 0x7fcfc4101980 2024-06-04T05:02:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.164+0000 7fcfcb2e3640 1 -- 172.21.15.70:0/603625881 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfc41a7840 con 0x7fcfc4101980 2024-06-04T05:02:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.164+0000 7fcfba7fc640 1 -- 172.21.15.70:0/603625881 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcfc00054b0 con 0x7fcfc4101980 2024-06-04T05:02:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.164+0000 7fcfba7fc640 1 -- 172.21.15.70:0/603625881 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0010430 con 0x7fcfc4101980 2024-06-04T05:02:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.164+0000 7fcfcb2e3640 1 -- 172.21.15.70:0/603625881 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfc41a7d40 con 0x7fcfc4101980 2024-06-04T05:02:52.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.166+0000 7fcfba7fc640 1 -- 172.21.15.70:0/603625881 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fcfc00046c0 con 0x7fcfc4101980 2024-06-04T05:02:52.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.167+0000 7fcfcdffd740 1 Processor -- start 2024-06-04T05:02:52.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.167+0000 7fcfcdffd740 1 -- start start 2024-06-04T05:02:52.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.167+0000 7fcfcdffd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9d5820 0x56007e9d5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.167+0000 7fcfcdffd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d6140 0x56007e9e04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.167+0000 7fcfcdffd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9e0a00 0x56007e9e2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.167+0000 7fcfcdffd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56007e947c30 con 0x56007e9d6140 2024-06-04T05:02:52.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.167+0000 7fcfcdffd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56007e9377d0 con 0x56007e9e0a00 2024-06-04T05:02:52.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.167+0000 7fcfcdffd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56007e93b810 con 0x56007e9d5820 2024-06-04T05:02:52.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.167+0000 7fcfba7fc640 1 --2- 172.21.15.70:0/603625881 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcfb00b2a10 0x7fcfb00b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.167+0000 7fcfc907f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d6140 0x56007e9e04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.167+0000 7fcfc9880640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9d5820 0x56007e9d5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.167+0000 7fcfc9880640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9d5820 0x56007e9d5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:56392/0 (socket says 172.21.15.70:56392) 2024-06-04T05:02:52.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.167+0000 7fcfc9880640 1 -- 172.21.15.70:0/2001371183 learned_addr learned my addr 172.21.15.70:0/2001371183 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:52.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.168+0000 7fcfc9880640 1 --2- 172.21.15.70:0/603625881 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcfb00b2a10 0x7fcfb00b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.168+0000 7fcfc887e640 1 --2- 172.21.15.70:0/2001371183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9e0a00 0x56007e9e2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.168+0000 7fcfba7fc640 1 -- 172.21.15.70:0/603625881 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fcfc00e2910 con 0x7fcfc4101980 2024-06-04T05:02:52.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.168+0000 7fcfc9880640 1 -- 172.21.15.70:0/2001371183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9e0a00 msgr2=0x56007e9e2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:52.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.168+0000 7fcfc9880640 1 --2- 172.21.15.70:0/2001371183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9e0a00 0x56007e9e2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.168+0000 7fcfc9880640 1 -- 172.21.15.70:0/2001371183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d6140 msgr2=0x56007e9e04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.168+0000 7fcfc9880640 1 --2- 172.21.15.70:0/2001371183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d6140 0x56007e9e04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.168+0000 7fcfc9880640 1 -- 172.21.15.70:0/2001371183 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56007e64c210 con 0x56007e9d5820 2024-06-04T05:02:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.168+0000 7fcfc9880640 1 --2- 172.21.15.70:0/2001371183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9d5820 0x56007e9d5c00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc005fe0 tx=0x7fcfbc031b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7af9fc7078725068 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.168+0000 7fcfc9880640 1 --2- 172.21.15.70:0/603625881 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcfb00b2a10 0x7fcfb00b4ed0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc004ab0 tx=0x7fcfbc036000 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.168+0000 7fcfaf7fe640 1 -- 172.21.15.70:0/2001371183 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc04d070 con 0x56007e9d5820 2024-06-04T05:02:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.168+0000 7fcfaf7fe640 1 -- 172.21.15.70:0/2001371183 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcfbc036070 con 0x56007e9d5820 2024-06-04T05:02:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.168+0000 7fcfaf7fe640 1 -- 172.21.15.70:0/2001371183 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc030c90 con 0x56007e9d5820 2024-06-04T05:02:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.169+0000 7fcfcdffd740 1 -- 172.21.15.70:0/2001371183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9d5820 msgr2=0x56007e9d5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.169+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/2001371183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9d5820 0x56007e9d5c00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc005fe0 tx=0x7fcfbc031b10 comp rx=0 tx=0).stop 2024-06-04T05:02:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.169+0000 7fcfcdffd740 1 -- 172.21.15.70:0/2001371183 shutdown_connections 2024-06-04T05:02:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.169+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/2001371183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9e0a00 0x56007e9e2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.169+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/2001371183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d6140 0x56007e9e04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.169+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/2001371183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9d5820 0x56007e9d5c00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.169+0000 7fcfcdffd740 1 -- 172.21.15.70:0/2001371183 >> 172.21.15.70:0/2001371183 conn(0x56007e9d1d30 msgr2=0x56007e9d2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:52.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.169+0000 7fcfcdffd740 1 -- 172.21.15.70:0/2001371183 shutdown_connections 2024-06-04T05:02:52.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.169+0000 7fcfcdffd740 1 -- 172.21.15.70:0/2001371183 wait complete. 2024-06-04T05:02:52.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.170+0000 7fcfcdffd740 1 -- start start 2024-06-04T05:02:52.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.170+0000 7fcfcdffd740 1 Processor -- start 2024-06-04T05:02:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.170+0000 7fcfcdffd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d5820 0x56007e9dda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.170+0000 7fcfcdffd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9d6140 0x56007e9ddf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.170+0000 7fcfcdffd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9e0a00 0x56007e9de4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.170+0000 7fcfcdffd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56007e61d790 con 0x56007e9d5820 2024-06-04T05:02:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.170+0000 7fcfcdffd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56007e8f3890 con 0x56007e9d6140 2024-06-04T05:02:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.170+0000 7fcfcdffd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56007e9e3570 con 0x56007e9e0a00 2024-06-04T05:02:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.170+0000 7fcfc907f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d5820 0x56007e9dda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.170+0000 7fcfc907f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d5820 0x56007e9dda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58868/0 (socket says 172.21.15.70:58868) 2024-06-04T05:02:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.170+0000 7fcfc907f640 1 -- 172.21.15.70:0/2686311487 learned_addr learned my addr 172.21.15.70:0/2686311487 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.170+0000 7fcfcdffd740 10 client.0 _ll_register_callbacks cb 0x56007e64ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.170+0000 7fcfc887e640 1 --2- 172.21.15.70:0/2686311487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9d6140 0x56007e9ddf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.170+0000 7fcfc9880640 1 --2- 172.21.15.70:0/2686311487 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9e0a00 0x56007e9de4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.171+0000 7fcfc907f640 1 -- 172.21.15.70:0/2686311487 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9e0a00 msgr2=0x56007e9de4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.171+0000 7fcfc907f640 1 --2- 172.21.15.70:0/2686311487 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9e0a00 0x56007e9de4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.171+0000 7fcfc907f640 1 -- 172.21.15.70:0/2686311487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9d6140 msgr2=0x56007e9ddf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.171+0000 7fcfc907f640 1 --2- 172.21.15.70:0/2686311487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9d6140 0x56007e9ddf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.171+0000 7fcfc907f640 1 -- 172.21.15.70:0/2686311487 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56007e718620 con 0x56007e9d5820 2024-06-04T05:02:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.171+0000 7fcfc9880640 1 --2- 172.21.15.70:0/2686311487 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9e0a00 0x56007e9de4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.171+0000 7fcfc887e640 1 --2- 172.21.15.70:0/2686311487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9d6140 0x56007e9ddf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.171+0000 7fcfc907f640 1 --2- 172.21.15.70:0/2686311487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d5820 0x56007e9dda50 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00a9b80 tx=0x7fcfc0073550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f88eda38f691ea server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.171+0000 7fcfacff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:02:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.171+0000 7fcfacff9640 1 -- 172.21.15.70:0/2686311487 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc00a99c0 con 0x56007e9d5820 2024-06-04T05:02:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.171+0000 7fcfacff9640 1 -- 172.21.15.70:0/2686311487 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fcfc0076110 con 0x56007e9d5820 2024-06-04T05:02:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.171+0000 7fcfcdffd740 1 -- 172.21.15.70:0/2686311487 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfbc036070 con 0x56007e9d5820 2024-06-04T05:02:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.171+0000 7fcfacff9640 1 -- 172.21.15.70:0/2686311487 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcfc00a9360 con 0x56007e9d5820 2024-06-04T05:02:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.171+0000 7fcfcdffd740 10 client.16992 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.171+0000 7fcfcdffd740 1 -- 172.21.15.70:0/2686311487 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56007e9ef380 con 0x56007e9d5820 2024-06-04T05:02:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.171+0000 7fcfacff9640 1 -- 172.21.15.70:0/2686311487 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcfbc036070 con 0x56007e9d5820 2024-06-04T05:02:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.171+0000 7fcfacff9640 1 -- 172.21.15.70:0/2686311487 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fcfc00730d0 con 0x56007e9d5820 2024-06-04T05:02:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.172+0000 7fcfcdffd740 10 client.16992 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.172+0000 7fcfcdffd740 10 client.16992 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:02:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.172+0000 7fcfcdffd740 20 client.16992 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.172+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/2686311487 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56007e9f0840 0x56007e9f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.172+0000 7fcfcdffd740 4 client.16992 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:02:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.172+0000 7fcfcdffd740 1 -- 172.21.15.70:0/2686311487 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56007e949b80 con 0x56007e9f0840 2024-06-04T05:02:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.172+0000 7fcfc887e640 1 --2- 172.21.15.70:0/2686311487 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56007e9f0840 0x56007e9f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.172+0000 7fcfc887e640 1 --2- 172.21.15.70:0/2686311487 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56007e9f0840 0x56007e9f2c30 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b38f9c4d64819c47 server_cookie=d48146ee14adb543 in_seq=0 out_seq=0 2024-06-04T05:02:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.172+0000 7fcfacff9640 10 client.16992 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.173+0000 7fcfacff9640 1 -- 172.21.15.70:0/2686311487 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56007e949b80 con 0x56007e9f0840 2024-06-04T05:02:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.173+0000 7fcfacff9640 10 client.16992 handle_command_reply: tid=0 2024-06-04T05:02:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.173+0000 7fcfcdffd740 1 client.16992 shutdown 2024-06-04T05:02:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.173+0000 7fcfcdffd740 1 -- 172.21.15.70:0/2686311487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d5820 msgr2=0x56007e9dda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.173+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/2686311487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d5820 0x56007e9dda50 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00a9b80 tx=0x7fcfc0073550 comp rx=0 tx=0).stop 2024-06-04T05:02:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.173+0000 7fcfcdffd740 1 -- 172.21.15.70:0/2686311487 shutdown_connections 2024-06-04T05:02:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.173+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/2686311487 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56007e9f0840 0x56007e9f2c30 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.173+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/2686311487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9d6140 0x56007e9ddf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.173+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/2686311487 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9e0a00 0x56007e9de4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.173+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/2686311487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d5820 0x56007e9dda50 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.173+0000 7fcfcdffd740 1 -- 172.21.15.70:0/2686311487 >> 172.21.15.70:0/2686311487 conn(0x56007e9d1d30 msgr2=0x56007e9d0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.173+0000 7fcfacff9640 0 client.16992 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.173+0000 7fcfcdffd740 1 -- 172.21.15.70:0/2686311487 shutdown_connections 2024-06-04T05:02:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.174+0000 7fcfcdffd740 1 -- 172.21.15.70:0/2686311487 wait complete. 2024-06-04T05:02:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.174+0000 7fcfcdffd740 20 client.16992 trim_cache size 0 max 16384 2024-06-04T05:02:52.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfcdffd740 1 Processor -- start 2024-06-04T05:02:52.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfcdffd740 1 -- start start 2024-06-04T05:02:52.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfcdffd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9d5820 0x56007e9de2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfcdffd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9d6140 0x56007e9d7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfcdffd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9e0a00 0x56007e9f84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfcdffd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc41b06f0 con 0x56007e9e0a00 2024-06-04T05:02:52.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfcdffd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc41b2230 con 0x56007e9d6140 2024-06-04T05:02:52.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfcdffd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc41b0dc0 con 0x56007e9d5820 2024-06-04T05:02:52.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfc9880640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9e0a00 0x56007e9f84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfc9880640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9e0a00 0x56007e9f84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58882/0 (socket says 172.21.15.70:58882) 2024-06-04T05:02:52.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfc9880640 1 -- 172.21.15.70:0/776565645 learned_addr learned my addr 172.21.15.70:0/776565645 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:52.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfc907f640 1 --2- 172.21.15.70:0/776565645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9d5820 0x56007e9de2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfc887e640 1 --2- 172.21.15.70:0/776565645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9d6140 0x56007e9d7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfc9880640 1 -- 172.21.15.70:0/776565645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9d5820 msgr2=0x56007e9de2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:52.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfc9880640 1 --2- 172.21.15.70:0/776565645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9d5820 0x56007e9de2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfc9880640 1 -- 172.21.15.70:0/776565645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9d6140 msgr2=0x56007e9d7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:52.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfc9880640 1 --2- 172.21.15.70:0/776565645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9d6140 0x56007e9d7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfc9880640 1 -- 172.21.15.70:0/776565645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56007e9f3210 con 0x56007e9e0a00 2024-06-04T05:02:52.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.191+0000 7fcfc907f640 1 --2- 172.21.15.70:0/776565645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9d5820 0x56007e9de2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:02:52.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.192+0000 7fcfc9880640 1 --2- 172.21.15.70:0/776565645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9e0a00 0x56007e9f84b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc006010 tx=0x7fcfbc004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b78c8b0df9e4ca6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:52.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.192+0000 7fcfaeffd640 1 -- 172.21.15.70:0/776565645 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc061070 con 0x56007e9e0a00 2024-06-04T05:02:52.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.192+0000 7fcfaeffd640 1 -- 172.21.15.70:0/776565645 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcfbc009820 con 0x56007e9e0a00 2024-06-04T05:02:52.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.192+0000 7fcfaeffd640 1 -- 172.21.15.70:0/776565645 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc04d040 con 0x56007e9e0a00 2024-06-04T05:02:52.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.193+0000 7fcfcdffd740 1 -- 172.21.15.70:0/776565645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9e0a00 msgr2=0x56007e9f84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:52.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.193+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/776565645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9e0a00 0x56007e9f84b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc006010 tx=0x7fcfbc004740 comp rx=0 tx=0).stop 2024-06-04T05:02:52.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.193+0000 7fcfcdffd740 1 -- 172.21.15.70:0/776565645 shutdown_connections 2024-06-04T05:02:52.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.193+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/776565645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9d6140 0x56007e9d7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.193+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/776565645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9e0a00 0x56007e9f84b0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.193+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/776565645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9d5820 0x56007e9de2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.193+0000 7fcfcdffd740 1 -- 172.21.15.70:0/776565645 >> 172.21.15.70:0/776565645 conn(0x56007e9d1d30 msgr2=0x56007e9d7220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:52.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.193+0000 7fcfcdffd740 1 -- 172.21.15.70:0/776565645 shutdown_connections 2024-06-04T05:02:52.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.193+0000 7fcfcdffd740 1 -- 172.21.15.70:0/776565645 wait complete. 2024-06-04T05:02:52.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.193+0000 7fcfcdffd740 1 -- start start 2024-06-04T05:02:52.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.193+0000 7fcfcdffd740 1 Processor -- start 2024-06-04T05:02:52.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.193+0000 7fcfcdffd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9d5820 0x56007e9e3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.193+0000 7fcfcdffd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d6140 0x56007e9e42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.193+0000 7fcfcdffd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9e0a00 0x56007ea59370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfcdffd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc41b1b60 con 0x56007e9d6140 2024-06-04T05:02:52.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfcdffd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc41b1490 con 0x56007e9d5820 2024-06-04T05:02:52.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfcdffd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc41ae6d0 con 0x56007e9e0a00 2024-06-04T05:02:52.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfcdffd740 10 client.0 _ll_register_callbacks cb 0x7fcfc0004a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:02:52.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfc887e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d6140 0x56007e9e42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfc887e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d6140 0x56007e9e42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58892/0 (socket says 172.21.15.70:58892) 2024-06-04T05:02:52.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfc887e640 1 -- 172.21.15.70:0/1899681537 learned_addr learned my addr 172.21.15.70:0/1899681537 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:02:52.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfc9880640 1 --2- 172.21.15.70:0/1899681537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9e0a00 0x56007ea59370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfc907f640 1 --2- 172.21.15.70:0/1899681537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9d5820 0x56007e9e3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfc887e640 1 -- 172.21.15.70:0/1899681537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9e0a00 msgr2=0x56007ea59370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:52.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfc887e640 1 --2- 172.21.15.70:0/1899681537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9e0a00 0x56007ea59370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfc887e640 1 -- 172.21.15.70:0/1899681537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9d5820 msgr2=0x56007e9e3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:02:52.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfc887e640 1 --2- 172.21.15.70:0/1899681537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9d5820 0x56007e9e3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfc887e640 1 -- 172.21.15.70:0/1899681537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56007ea598e0 con 0x56007e9d6140 2024-06-04T05:02:52.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfc9880640 1 --2- 172.21.15.70:0/1899681537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9e0a00 0x56007ea59370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:02:52.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfc887e640 1 --2- 172.21.15.70:0/1899681537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d6140 0x56007e9e42a0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400b840 tx=0x7fcfb400bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e898e8217b00824 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:02:52.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfacff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:02:52.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfacff9640 1 -- 172.21.15.70:0/1899681537 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb401d460 con 0x56007e9d6140 2024-06-04T05:02:52.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfacff9640 1 -- 172.21.15.70:0/1899681537 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fcfb401d600 con 0x56007e9d6140 2024-06-04T05:02:52.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.194+0000 7fcfcdffd740 1 -- 172.21.15.70:0/1899681537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfbc009820 con 0x56007e9d6140 2024-06-04T05:02:52.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.195+0000 7fcfacff9640 1 -- 172.21.15.70:0/1899681537 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcfb401d820 con 0x56007e9d6140 2024-06-04T05:02:52.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.195+0000 7fcfcdffd740 10 client.16998 fetch_fsmap learned FSMap version 32 2024-06-04T05:02:52.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.195+0000 7fcfcdffd740 1 -- 172.21.15.70:0/1899681537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56007ea5ac60 con 0x56007e9d6140 2024-06-04T05:02:52.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.195+0000 7fcfacff9640 1 -- 172.21.15.70:0/1899681537 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcfbc009820 con 0x56007e9d6140 2024-06-04T05:02:52.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.195+0000 7fcfacff9640 1 -- 172.21.15.70:0/1899681537 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fcfb4011980 con 0x56007e9d6140 2024-06-04T05:02:52.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.195+0000 7fcfcdffd740 10 client.16998 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:02:52.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.195+0000 7fcfcdffd740 10 client.16998 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:02:52.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.195+0000 7fcfcdffd740 20 client.16998 populate_metadata read hostname 'smithi070' 2024-06-04T05:02:52.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.195+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/1899681537 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56007ea5b410 0x56007ea5d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:02:52.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.195+0000 7fcfcdffd740 4 client.16998 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:02:52.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.195+0000 7fcfcdffd740 1 -- 172.21.15.70:0/1899681537 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56007e943360 con 0x56007ea5b410 2024-06-04T05:02:52.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.195+0000 7fcfc907f640 1 --2- 172.21.15.70:0/1899681537 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56007ea5b410 0x56007ea5d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:02:52.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.196+0000 7fcfc907f640 1 --2- 172.21.15.70:0/1899681537 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56007ea5b410 0x56007ea5d830 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf4157e758e24480 server_cookie=f5874cf435812a2a in_seq=0 out_seq=0 2024-06-04T05:02:52.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.196+0000 7fcfacff9640 10 client.16998 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:52.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.220+0000 7fcfacff9640 1 -- 172.21.15.70:0/1899681537 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56007e943360 con 0x56007ea5b410 2024-06-04T05:02:52.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.220+0000 7fcfacff9640 10 client.16998 handle_command_reply: tid=0 2024-06-04T05:02:52.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.220+0000 7fcfcdffd740 1 client.16998 shutdown 2024-06-04T05:02:52.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.220+0000 7fcfcdffd740 1 -- 172.21.15.70:0/1899681537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d6140 msgr2=0x56007e9e42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:52.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.220+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/1899681537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d6140 0x56007e9e42a0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400b840 tx=0x7fcfb400bde0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.220+0000 7fcfcdffd740 1 -- 172.21.15.70:0/1899681537 shutdown_connections 2024-06-04T05:02:52.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.220+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/1899681537 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56007ea5b410 0x56007ea5d830 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.220+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/1899681537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56007e9e0a00 0x56007ea59370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.220+0000 7fcfacff9640 0 client.16998 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:02:52.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.220+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/1899681537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56007e9d6140 0x56007e9e42a0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.220+0000 7fcfcdffd740 1 --2- 172.21.15.70:0/1899681537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007e9d5820 0x56007e9e3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.220+0000 7fcfcdffd740 1 -- 172.21.15.70:0/1899681537 >> 172.21.15.70:0/1899681537 conn(0x56007e9d1d30 msgr2=0x56007e9d7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:52.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.220+0000 7fcfcdffd740 1 -- 172.21.15.70:0/1899681537 shutdown_connections 2024-06-04T05:02:52.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.220+0000 7fcfcdffd740 1 -- 172.21.15.70:0/1899681537 wait complete. 2024-06-04T05:02:52.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.220+0000 7fcfcdffd740 20 client.16998 trim_cache size 0 max 16384 2024-06-04T05:02:52.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:02:52.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:02:52.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:02:52.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.221+0000 7fcfcb2e3640 1 -- 172.21.15.70:0/603625881 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcfb00b2a10 msgr2=0x7fcfb00b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:52.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.221+0000 7fcfcb2e3640 1 --2- 172.21.15.70:0/603625881 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcfb00b2a10 0x7fcfb00b4ed0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc004ab0 tx=0x7fcfbc036000 comp rx=0 tx=0).stop 2024-06-04T05:02:52.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.221+0000 7fcfcb2e3640 1 -- 172.21.15.70:0/603625881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfc4101980 msgr2=0x7fcfc4197ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:02:52.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.221+0000 7fcfcb2e3640 1 --2- 172.21.15.70:0/603625881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfc4101980 0x7fcfc4197ce0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fcfc000e9f0 tx=0x7fcfc000edf0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.221+0000 7fcfcb2e3640 1 -- 172.21.15.70:0/603625881 shutdown_connections 2024-06-04T05:02:52.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.221+0000 7fcfcb2e3640 1 --2- 172.21.15.70:0/603625881 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcfb00b2a10 0x7fcfb00b4ed0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.221+0000 7fcfcb2e3640 1 --2- 172.21.15.70:0/603625881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfc4102340 0x7fcfc4198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.221+0000 7fcfcb2e3640 1 --2- 172.21.15.70:0/603625881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcfc4107940 0x7fcfc4198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.221+0000 7fcfcb2e3640 1 --2- 172.21.15.70:0/603625881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfc4101980 0x7fcfc4197ce0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:02:52.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.221+0000 7fcfcb2e3640 1 -- 172.21.15.70:0/603625881 >> 172.21.15.70:0/603625881 conn(0x7fcfc4078ad0 msgr2=0x7fcfc41045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:02:52.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.221+0000 7fcfcb2e3640 1 -- 172.21.15.70:0/603625881 shutdown_connections 2024-06-04T05:02:52.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:02:52.221+0000 7fcfcb2e3640 1 -- 172.21.15.70:0/603625881 wait complete. 2024-06-04T05:02:52.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 90029 2.28 MB/sec execute 105 sec latency 25.305 ms 2024-06-04T05:02:53.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 90729 2.29 MB/sec execute 106 sec latency 15.499 ms 2024-06-04T05:02:53.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:53 smithi070 ceph-mon[32631]: pgmap v174: 97 pgs: 97 active+clean; 63 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 1.9 MiB/s wr, 210 op/s 2024-06-04T05:02:53.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:53 smithi112 ceph-mon[36453]: pgmap v174: 97 pgs: 97 active+clean; 63 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 1.9 MiB/s wr, 210 op/s 2024-06-04T05:02:53.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:53 smithi098 ceph-mon[35569]: pgmap v174: 97 pgs: 97 active+clean; 63 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 1.9 MiB/s wr, 210 op/s 2024-06-04T05:02:54.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 91303 2.29 MB/sec execute 107 sec latency 14.660 ms 2024-06-04T05:02:55.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 92036 2.33 MB/sec execute 108 sec latency 12.482 ms 2024-06-04T05:02:55.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:55 smithi070 ceph-mon[32631]: pgmap v175: 97 pgs: 97 active+clean; 63 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-06-04T05:02:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:55 smithi112 ceph-mon[36453]: pgmap v175: 97 pgs: 97 active+clean; 63 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-06-04T05:02:55.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:55 smithi098 ceph-mon[35569]: pgmap v175: 97 pgs: 97 active+clean; 63 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-06-04T05:02:56.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 92510 2.34 MB/sec execute 109 sec latency 14.798 ms 2024-06-04T05:02:57.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 92799 2.33 MB/sec execute 110 sec latency 13.991 ms 2024-06-04T05:02:57.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:57 smithi070 ceph-mon[32631]: pgmap v176: 97 pgs: 97 active+clean; 61 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 2.1 MiB/s wr, 197 op/s 2024-06-04T05:02:57.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:57 smithi112 ceph-mon[36453]: pgmap v176: 97 pgs: 97 active+clean; 61 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 2.1 MiB/s wr, 197 op/s 2024-06-04T05:02:57.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:57 smithi098 ceph-mon[35569]: pgmap v176: 97 pgs: 97 active+clean; 61 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 2.1 MiB/s wr, 197 op/s 2024-06-04T05:02:58.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 93248 2.31 MB/sec execute 111 sec latency 16.517 ms 2024-06-04T05:02:59.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 93828 2.33 MB/sec execute 112 sec latency 15.865 ms 2024-06-04T05:02:59.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:02:59 smithi070 ceph-mon[32631]: pgmap v177: 97 pgs: 97 active+clean; 61 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 1.3 MiB/s wr, 130 op/s 2024-06-04T05:02:59.935 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:02:59 smithi112 ceph-mon[36453]: pgmap v177: 97 pgs: 97 active+clean; 61 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 1.3 MiB/s wr, 130 op/s 2024-06-04T05:02:59.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:02:59 smithi098 ceph-mon[35569]: pgmap v177: 97 pgs: 97 active+clean; 61 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 1.3 MiB/s wr, 130 op/s 2024-06-04T05:03:00.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 94520 2.32 MB/sec execute 113 sec latency 15.980 ms 2024-06-04T05:03:01.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 95163 2.35 MB/sec execute 114 sec latency 14.739 ms 2024-06-04T05:03:01.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:01 smithi070 ceph-mon[32631]: pgmap v178: 97 pgs: 97 active+clean; 61 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 1.3 MiB/s wr, 130 op/s 2024-06-04T05:03:01.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:01 smithi112 ceph-mon[36453]: pgmap v178: 97 pgs: 97 active+clean; 61 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 1.3 MiB/s wr, 130 op/s 2024-06-04T05:03:01.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:01 smithi098 ceph-mon[35569]: pgmap v178: 97 pgs: 97 active+clean; 61 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 1.3 MiB/s wr, 130 op/s 2024-06-04T05:03:02.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 95646 2.35 MB/sec execute 115 sec latency 15.487 ms 2024-06-04T05:03:03.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 96132 2.37 MB/sec execute 116 sec latency 12.890 ms 2024-06-04T05:03:03.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:03 smithi070 ceph-mon[32631]: pgmap v179: 97 pgs: 97 active+clean; 62 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.9 MiB/s wr, 214 op/s 2024-06-04T05:03:03.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:03 smithi112 ceph-mon[36453]: pgmap v179: 97 pgs: 97 active+clean; 62 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.9 MiB/s wr, 214 op/s 2024-06-04T05:03:03.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:03 smithi098 ceph-mon[35569]: pgmap v179: 97 pgs: 97 active+clean; 62 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.9 MiB/s wr, 214 op/s 2024-06-04T05:03:04.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 96415 2.35 MB/sec execute 117 sec latency 13.169 ms 2024-06-04T05:03:05.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.313+0000 7ffa21bda640 1 -- 172.21.15.70:0/965328871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa1c1033f0 msgr2=0x7ffa1c10f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:05.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.313+0000 7ffa21bda640 1 --2- 172.21.15.70:0/965328871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa1c1033f0 0x7ffa1c10f900 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7ffa14009cd0 tx=0x7ffa1402a8e0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.313+0000 7ffa21bda640 1 -- 172.21.15.70:0/965328871 shutdown_connections 2024-06-04T05:03:05.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.313+0000 7ffa21bda640 1 --2- 172.21.15.70:0/965328871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa1c102a30 0x7ffa1c102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.313+0000 7ffa21bda640 1 --2- 172.21.15.70:0/965328871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa1c1033f0 0x7ffa1c10f900 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.313+0000 7ffa21bda640 1 --2- 172.21.15.70:0/965328871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa1c108a40 0x7ffa1c108e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.313+0000 7ffa21bda640 1 -- 172.21.15.70:0/965328871 >> 172.21.15.70:0/965328871 conn(0x7ffa1c0fe500 msgr2=0x7ffa1c100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.313+0000 7ffa21bda640 1 -- 172.21.15.70:0/965328871 shutdown_connections 2024-06-04T05:03:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.313+0000 7ffa21bda640 1 -- 172.21.15.70:0/965328871 wait complete. 2024-06-04T05:03:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa21bda640 1 Processor -- start 2024-06-04T05:03:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa21bda640 1 -- start start 2024-06-04T05:03:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa21bda640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa1c102a30 0x7ffa1c074d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa21bda640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa1c1033f0 0x7ffa1c075290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa21bda640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa1c108a40 0x7ffa1c0757d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa21bda640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c112b80 con 0x7ffa1c108a40 2024-06-04T05:03:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa21bda640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c112a00 con 0x7ffa1c1033f0 2024-06-04T05:03:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa21bda640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c112d00 con 0x7ffa1c102a30 2024-06-04T05:03:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa1bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa1c108a40 0x7ffa1c0757d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa1bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa1c108a40 0x7ffa1c0757d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60798/0 (socket says 172.21.15.70:60798) 2024-06-04T05:03:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa1b7fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa1c102a30 0x7ffa1c074d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa1bfff640 1 -- 172.21.15.70:0/182347181 learned_addr learned my addr 172.21.15.70:0/182347181 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa1affd640 1 --2- 172.21.15.70:0/182347181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa1c1033f0 0x7ffa1c075290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa1bfff640 1 -- 172.21.15.70:0/182347181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa1c102a30 msgr2=0x7ffa1c074d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa1bfff640 1 --2- 172.21.15.70:0/182347181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa1c102a30 0x7ffa1c074d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa1bfff640 1 -- 172.21.15.70:0/182347181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa1c1033f0 msgr2=0x7ffa1c075290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa1bfff640 1 --2- 172.21.15.70:0/182347181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa1c1033f0 0x7ffa1c075290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa1bfff640 1 -- 172.21.15.70:0/182347181 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa1c1a75a0 con 0x7ffa1c108a40 2024-06-04T05:03:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.314+0000 7ffa1b7fe640 1 --2- 172.21.15.70:0/182347181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa1c102a30 0x7ffa1c074d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.315+0000 7ffa1bfff640 1 --2- 172.21.15.70:0/182347181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa1c108a40 0x7ffa1c0757d0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7ffa14005cc0 tx=0x7ffa14003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.315+0000 7ffa18ff9640 1 -- 172.21.15.70:0/182347181 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa14004040 con 0x7ffa1c108a40 2024-06-04T05:03:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.315+0000 7ffa21bda640 1 -- 172.21.15.70:0/182347181 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa1c1a7830 con 0x7ffa1c108a40 2024-06-04T05:03:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.315+0000 7ffa18ff9640 1 -- 172.21.15.70:0/182347181 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffa14004670 con 0x7ffa1c108a40 2024-06-04T05:03:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.315+0000 7ffa18ff9640 1 -- 172.21.15.70:0/182347181 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa14030ce0 con 0x7ffa1c108a40 2024-06-04T05:03:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.315+0000 7ffa21bda640 1 -- 172.21.15.70:0/182347181 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa1c1a7dd0 con 0x7ffa1c108a40 2024-06-04T05:03:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.318+0000 7ffa18ff9640 1 -- 172.21.15.70:0/182347181 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ffa140041e0 con 0x7ffa1c108a40 2024-06-04T05:03:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.318+0000 7ffa249ce740 1 Processor -- start 2024-06-04T05:03:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.318+0000 7ffa249ce740 1 -- start start 2024-06-04T05:03:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.318+0000 7ffa249ce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1b9820 0x55975c1b9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.318+0000 7ffa249ce740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1ba140 0x55975c1c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.318+0000 7ffa249ce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1c4a00 0x55975c1c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.318+0000 7ffa249ce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55975c12bc30 con 0x55975c1c4a00 2024-06-04T05:03:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.318+0000 7ffa249ce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55975c11b7d0 con 0x55975c1b9820 2024-06-04T05:03:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.318+0000 7ffa249ce740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55975c11f810 con 0x55975c1ba140 2024-06-04T05:03:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.318+0000 7ffa1affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1b9820 0x55975c1b9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa1affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1c4a00 0x55975c1c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa1affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1b9820 0x55975c1b9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:37266/0 (socket says 172.21.15.70:37266) 2024-06-04T05:03:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa1affd640 1 -- 172.21.15.70:0/1977040969 learned_addr learned my addr 172.21.15.70:0/1977040969 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa1b7fe640 1 --2- 172.21.15.70:0/1977040969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1ba140 0x55975c1c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa18ff9640 1 --2- 172.21.15.70:0/182347181 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ffa040b2a10 0x7ffa040b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa1affd640 1 -- 172.21.15.70:0/1977040969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1ba140 msgr2=0x55975c1c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa1bfff640 1 --2- 172.21.15.70:0/182347181 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ffa040b2a10 0x7ffa040b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa1affd640 1 --2- 172.21.15.70:0/1977040969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1ba140 0x55975c1c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa1affd640 1 -- 172.21.15.70:0/1977040969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1c4a00 msgr2=0x55975c1c6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa1affd640 1 --2- 172.21.15.70:0/1977040969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1c4a00 0x55975c1c6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa1affd640 1 -- 172.21.15.70:0/1977040969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55975be30210 con 0x55975c1b9820 2024-06-04T05:03:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa1b7fe640 1 --2- 172.21.15.70:0/1977040969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1ba140 0x55975c1c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa18ff9640 1 -- 172.21.15.70:0/182347181 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ffa1402e030 con 0x7ffa1c108a40 2024-06-04T05:03:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa1affd640 1 --2- 172.21.15.70:0/1977040969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1b9820 0x55975c1b9c00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffa1000bfd0 tx=0x7ffa100046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f3c3162b4ae0c70 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa01ffb640 1 -- 172.21.15.70:0/1977040969 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10004050 con 0x55975c1b9820 2024-06-04T05:03:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa1bfff640 1 --2- 172.21.15.70:0/182347181 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ffa040b2a10 0x7ffa040b4ed0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ffa14100e30 tx=0x7ffa140ed040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa01ffb640 1 -- 172.21.15.70:0/1977040969 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffa10005020 con 0x55975c1b9820 2024-06-04T05:03:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.319+0000 7ffa01ffb640 1 -- 172.21.15.70:0/1977040969 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10004a90 con 0x55975c1b9820 2024-06-04T05:03:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.320+0000 7ffa249ce740 1 -- 172.21.15.70:0/1977040969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1b9820 msgr2=0x55975c1b9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.320+0000 7ffa249ce740 1 --2- 172.21.15.70:0/1977040969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1b9820 0x55975c1b9c00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffa1000bfd0 tx=0x7ffa100046c0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.320+0000 7ffa249ce740 1 -- 172.21.15.70:0/1977040969 shutdown_connections 2024-06-04T05:03:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.320+0000 7ffa249ce740 1 --2- 172.21.15.70:0/1977040969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1c4a00 0x55975c1c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.320+0000 7ffa249ce740 1 --2- 172.21.15.70:0/1977040969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1ba140 0x55975c1c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.320+0000 7ffa249ce740 1 --2- 172.21.15.70:0/1977040969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1b9820 0x55975c1b9c00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.320+0000 7ffa249ce740 1 -- 172.21.15.70:0/1977040969 >> 172.21.15.70:0/1977040969 conn(0x55975c1b5d30 msgr2=0x55975c1b6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.320+0000 7ffa249ce740 1 -- 172.21.15.70:0/1977040969 shutdown_connections 2024-06-04T05:03:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.320+0000 7ffa249ce740 1 -- 172.21.15.70:0/1977040969 wait complete. 2024-06-04T05:03:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.320+0000 7ffa249ce740 1 -- start start 2024-06-04T05:03:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa249ce740 1 Processor -- start 2024-06-04T05:03:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa249ce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1b9820 0x55975c1c1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa249ce740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1ba140 0x55975c1c1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa249ce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1c4a00 0x55975c1c2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa249ce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55975be01790 con 0x55975c1b9820 2024-06-04T05:03:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa249ce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55975c0d7890 con 0x55975c1c4a00 2024-06-04T05:03:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa249ce740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55975c1c7630 con 0x55975c1ba140 2024-06-04T05:03:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa249ce740 10 client.0 _ll_register_callbacks cb 0x55975be2fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa1affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1b9820 0x55975c1c1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa1affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1b9820 0x55975c1c1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60826/0 (socket says 172.21.15.70:60826) 2024-06-04T05:03:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa1affd640 1 -- 172.21.15.70:0/2329565449 learned_addr learned my addr 172.21.15.70:0/2329565449 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa1b7fe640 1 --2- 172.21.15.70:0/2329565449 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1ba140 0x55975c1c1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa1affd640 1 --2- 172.21.15.70:0/2329565449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1c4a00 0x55975c1c2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa1affd640 1 -- 172.21.15.70:0/2329565449 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1ba140 msgr2=0x55975c1c1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa1affd640 1 --2- 172.21.15.70:0/2329565449 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1ba140 0x55975c1c1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa1affd640 1 -- 172.21.15.70:0/2329565449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1c4a00 msgr2=0x55975c1c2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa1affd640 1 --2- 172.21.15.70:0/2329565449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1c4a00 0x55975c1c2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa1affd640 1 -- 172.21.15.70:0/2329565449 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55975befc620 con 0x55975c1b9820 2024-06-04T05:03:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ffa1affd640 1 --2- 172.21.15.70:0/2329565449 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1b9820 0x55975c1c1980 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7ffa10004770 tx=0x7ffa1000b570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f0573f62211712 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.321+0000 7ff9faffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.322+0000 7ff9faffd640 1 -- 172.21.15.70:0/2329565449 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10016070 con 0x55975c1b9820 2024-06-04T05:03:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.322+0000 7ff9faffd640 1 -- 172.21.15.70:0/2329565449 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ffa10004460 con 0x55975c1b9820 2024-06-04T05:03:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.322+0000 7ffa249ce740 1 -- 172.21.15.70:0/2329565449 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa10005020 con 0x55975c1b9820 2024-06-04T05:03:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.322+0000 7ff9faffd640 1 -- 172.21.15.70:0/2329565449 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffa10033050 con 0x55975c1b9820 2024-06-04T05:03:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.322+0000 7ffa249ce740 10 client.17010 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.322+0000 7ffa249ce740 1 -- 172.21.15.70:0/2329565449 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55975c1d33d0 con 0x55975c1b9820 2024-06-04T05:03:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.322+0000 7ff9faffd640 1 -- 172.21.15.70:0/2329565449 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffa10005020 con 0x55975c1b9820 2024-06-04T05:03:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.322+0000 7ff9faffd640 1 -- 172.21.15.70:0/2329565449 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ffa10012ce0 con 0x55975c1b9820 2024-06-04T05:03:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.322+0000 7ffa249ce740 10 client.17010 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.322+0000 7ffa249ce740 10 client.17010 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.322+0000 7ffa249ce740 20 client.17010 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.322+0000 7ffa249ce740 1 --2- 172.21.15.70:0/2329565449 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55975c1d4770 0x55975c1d6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.322+0000 7ffa249ce740 4 client.17010 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:03:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.322+0000 7ffa249ce740 1 -- 172.21.15.70:0/2329565449 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55975c12db80 con 0x55975c1d4770 2024-06-04T05:03:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.323+0000 7ffa1b7fe640 1 --2- 172.21.15.70:0/2329565449 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55975c1d4770 0x55975c1d6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.323+0000 7ff9faffd640 10 client.17010 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.323+0000 7ffa1b7fe640 1 --2- 172.21.15.70:0/2329565449 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55975c1d4770 0x55975c1d6b60 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68c4fd3d16891e58 server_cookie=87a404a0a3c9ce92 in_seq=0 out_seq=0 2024-06-04T05:03:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.324+0000 7ff9faffd640 1 -- 172.21.15.70:0/2329565449 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55975c12db80 con 0x55975c1d4770 2024-06-04T05:03:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.324+0000 7ff9faffd640 10 client.17010 handle_command_reply: tid=0 2024-06-04T05:03:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.324+0000 7ffa249ce740 1 client.17010 shutdown 2024-06-04T05:03:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.324+0000 7ffa249ce740 1 -- 172.21.15.70:0/2329565449 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1b9820 msgr2=0x55975c1c1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.324+0000 7ffa249ce740 1 --2- 172.21.15.70:0/2329565449 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1b9820 0x55975c1c1980 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7ffa10004770 tx=0x7ffa1000b570 comp rx=0 tx=0).stop 2024-06-04T05:03:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.324+0000 7ffa249ce740 1 -- 172.21.15.70:0/2329565449 shutdown_connections 2024-06-04T05:03:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.324+0000 7ffa249ce740 1 --2- 172.21.15.70:0/2329565449 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55975c1d4770 0x55975c1d6b60 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.324+0000 7ffa249ce740 1 --2- 172.21.15.70:0/2329565449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1c4a00 0x55975c1c2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.324+0000 7ffa249ce740 1 --2- 172.21.15.70:0/2329565449 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1ba140 0x55975c1c1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.324+0000 7ff9faffd640 0 client.17010 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.324+0000 7ffa249ce740 1 --2- 172.21.15.70:0/2329565449 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1b9820 0x55975c1c1980 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.324+0000 7ffa249ce740 1 -- 172.21.15.70:0/2329565449 >> 172.21.15.70:0/2329565449 conn(0x55975c1b5d30 msgr2=0x55975c1b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.324+0000 7ffa249ce740 1 -- 172.21.15.70:0/2329565449 shutdown_connections 2024-06-04T05:03:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.324+0000 7ffa249ce740 1 -- 172.21.15.70:0/2329565449 wait complete. 2024-06-04T05:03:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.324+0000 7ffa249ce740 20 client.17010 trim_cache size 0 max 16384 2024-06-04T05:03:05.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.341+0000 7ffa249ce740 1 Processor -- start 2024-06-04T05:03:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.341+0000 7ffa249ce740 1 -- start start 2024-06-04T05:03:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.341+0000 7ffa249ce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1b9820 0x55975c1c0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.341+0000 7ffa249ce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1ba140 0x55975c1c2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.341+0000 7ffa249ce740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1c4a00 0x55975c1bb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.341+0000 7ffa249ce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c1b06f0 con 0x55975c1b9820 2024-06-04T05:03:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.341+0000 7ffa249ce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c1b2230 con 0x55975c1ba140 2024-06-04T05:03:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.341+0000 7ffa249ce740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c1b0dc0 con 0x55975c1c4a00 2024-06-04T05:03:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.342+0000 7ffa1affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1b9820 0x55975c1c0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.342+0000 7ffa1affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1b9820 0x55975c1c0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60830/0 (socket says 172.21.15.70:60830) 2024-06-04T05:03:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.342+0000 7ffa1affd640 1 -- 172.21.15.70:0/809225514 learned_addr learned my addr 172.21.15.70:0/809225514 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.342+0000 7ffa1b7fe640 1 --2- 172.21.15.70:0/809225514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1ba140 0x55975c1c2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.342+0000 7ffa1affd640 1 --2- 172.21.15.70:0/809225514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1c4a00 0x55975c1bb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.342+0000 7ffa1affd640 1 -- 172.21.15.70:0/809225514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1c4a00 msgr2=0x55975c1bb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.342+0000 7ffa1affd640 1 --2- 172.21.15.70:0/809225514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1c4a00 0x55975c1bb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.342+0000 7ffa1affd640 1 -- 172.21.15.70:0/809225514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1ba140 msgr2=0x55975c1c2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.342+0000 7ffa1affd640 1 --2- 172.21.15.70:0/809225514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1ba140 0x55975c1c2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.342+0000 7ffa1affd640 1 -- 172.21.15.70:0/809225514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55975c1bc7a0 con 0x55975c1b9820 2024-06-04T05:03:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.342+0000 7ffa1b7fe640 1 --2- 172.21.15.70:0/809225514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1ba140 0x55975c1c2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.342+0000 7ffa1affd640 1 --2- 172.21.15.70:0/809225514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1b9820 0x55975c1c0ae0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7ffa100124f0 tx=0x7ffa10014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccdb2f41e08dd1b3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.342+0000 7ffa017fa640 1 -- 172.21.15.70:0/809225514 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10016070 con 0x55975c1b9820 2024-06-04T05:03:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.342+0000 7ffa017fa640 1 -- 172.21.15.70:0/809225514 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffa1000b390 con 0x55975c1b9820 2024-06-04T05:03:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.342+0000 7ffa017fa640 1 -- 172.21.15.70:0/809225514 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa100039c0 con 0x55975c1b9820 2024-06-04T05:03:05.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.343+0000 7ffa249ce740 1 -- 172.21.15.70:0/809225514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1b9820 msgr2=0x55975c1c0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:05.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.343+0000 7ffa249ce740 1 --2- 172.21.15.70:0/809225514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1b9820 0x55975c1c0ae0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7ffa100124f0 tx=0x7ffa10014040 comp rx=0 tx=0).stop 2024-06-04T05:03:05.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.343+0000 7ffa249ce740 1 -- 172.21.15.70:0/809225514 shutdown_connections 2024-06-04T05:03:05.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.343+0000 7ffa249ce740 1 --2- 172.21.15.70:0/809225514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1ba140 0x55975c1c2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.343+0000 7ffa249ce740 1 --2- 172.21.15.70:0/809225514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1c4a00 0x55975c1bb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.343+0000 7ffa249ce740 1 --2- 172.21.15.70:0/809225514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1b9820 0x55975c1c0ae0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.343+0000 7ffa249ce740 1 -- 172.21.15.70:0/809225514 >> 172.21.15.70:0/809225514 conn(0x55975c1b5d30 msgr2=0x55975c1bb1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.343+0000 7ffa249ce740 1 -- 172.21.15.70:0/809225514 shutdown_connections 2024-06-04T05:03:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.343+0000 7ffa249ce740 1 -- 172.21.15.70:0/809225514 wait complete. 2024-06-04T05:03:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.343+0000 7ffa249ce740 1 -- start start 2024-06-04T05:03:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa249ce740 1 Processor -- start 2024-06-04T05:03:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa249ce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1b9820 0x55975c1bf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa249ce740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1ba140 0x55975c1cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa249ce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1c4a00 0x55975c1ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa249ce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c1b1b60 con 0x55975c1c4a00 2024-06-04T05:03:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa249ce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c1b1490 con 0x55975c1b9820 2024-06-04T05:03:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa249ce740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c1ae6d0 con 0x55975c1ba140 2024-06-04T05:03:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa249ce740 10 client.0 _ll_register_callbacks cb 0x7ffa1000be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa1affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1c4a00 0x55975c1ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa1affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1c4a00 0x55975c1ccfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60838/0 (socket says 172.21.15.70:60838) 2024-06-04T05:03:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa1affd640 1 -- 172.21.15.70:0/1456763047 learned_addr learned my addr 172.21.15.70:0/1456763047 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa1b7fe640 1 --2- 172.21.15.70:0/1456763047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1ba140 0x55975c1cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa1affd640 1 --2- 172.21.15.70:0/1456763047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1b9820 0x55975c1bf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa1affd640 1 -- 172.21.15.70:0/1456763047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1ba140 msgr2=0x55975c1cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa1affd640 1 --2- 172.21.15.70:0/1456763047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1ba140 0x55975c1cc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa1affd640 1 -- 172.21.15.70:0/1456763047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1b9820 msgr2=0x55975c1bf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa1affd640 1 --2- 172.21.15.70:0/1456763047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1b9820 0x55975c1bf620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa1affd640 1 -- 172.21.15.70:0/1456763047 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55975c1bdc70 con 0x55975c1c4a00 2024-06-04T05:03:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.344+0000 7ffa1b7fe640 1 --2- 172.21.15.70:0/1456763047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1ba140 0x55975c1cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.345+0000 7ffa1affd640 1 --2- 172.21.15.70:0/1456763047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1c4a00 0x55975c1ccfa0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7ffa100124c0 tx=0x7ffa10000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e045039a24da1ae server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.345+0000 7ff9faffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.345+0000 7ff9faffd640 1 -- 172.21.15.70:0/1456763047 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10016070 con 0x55975c1c4a00 2024-06-04T05:03:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.345+0000 7ff9faffd640 1 -- 172.21.15.70:0/1456763047 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7ffa1000d6b0 con 0x55975c1c4a00 2024-06-04T05:03:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.345+0000 7ffa249ce740 1 -- 172.21.15.70:0/1456763047 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa1000b390 con 0x55975c1c4a00 2024-06-04T05:03:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.345+0000 7ff9faffd640 1 -- 172.21.15.70:0/1456763047 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffa1004b050 con 0x55975c1c4a00 2024-06-04T05:03:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.345+0000 7ffa249ce740 10 client.17016 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.345+0000 7ffa249ce740 1 -- 172.21.15.70:0/1456763047 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55975c23e600 con 0x55975c1c4a00 2024-06-04T05:03:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.345+0000 7ff9faffd640 1 -- 172.21.15.70:0/1456763047 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffa1000b390 con 0x55975c1c4a00 2024-06-04T05:03:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.345+0000 7ff9faffd640 1 -- 172.21.15.70:0/1456763047 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ffa10004770 con 0x55975c1c4a00 2024-06-04T05:03:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.345+0000 7ffa249ce740 10 client.17016 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.345+0000 7ffa249ce740 10 client.17016 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.345+0000 7ffa249ce740 20 client.17016 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.345+0000 7ffa249ce740 1 --2- 172.21.15.70:0/1456763047 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55975c23ec30 0x55975c241050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.345+0000 7ffa249ce740 4 client.17016 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:03:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.345+0000 7ffa249ce740 1 -- 172.21.15.70:0/1456763047 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55975c127360 con 0x55975c23ec30 2024-06-04T05:03:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.346+0000 7ffa1b7fe640 1 --2- 172.21.15.70:0/1456763047 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55975c23ec30 0x55975c241050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.346+0000 7ff9faffd640 10 client.17016 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.346+0000 7ffa1b7fe640 1 --2- 172.21.15.70:0/1456763047 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55975c23ec30 0x55975c241050 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96640e3aac36b3dd server_cookie=64dfc625ab16270e in_seq=0 out_seq=0 2024-06-04T05:03:05.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.369+0000 7ff9faffd640 1 -- 172.21.15.70:0/1456763047 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55975c127360 con 0x55975c23ec30 2024-06-04T05:03:05.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.369+0000 7ff9faffd640 10 client.17016 handle_command_reply: tid=0 2024-06-04T05:03:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.369+0000 7ffa249ce740 1 client.17016 shutdown 2024-06-04T05:03:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.369+0000 7ffa249ce740 1 -- 172.21.15.70:0/1456763047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1c4a00 msgr2=0x55975c1ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.369+0000 7ffa249ce740 1 --2- 172.21.15.70:0/1456763047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1c4a00 0x55975c1ccfa0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7ffa100124c0 tx=0x7ffa10000bc0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.369+0000 7ffa249ce740 1 -- 172.21.15.70:0/1456763047 shutdown_connections 2024-06-04T05:03:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.369+0000 7ffa249ce740 1 --2- 172.21.15.70:0/1456763047 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55975c23ec30 0x55975c241050 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.369+0000 7ffa249ce740 1 --2- 172.21.15.70:0/1456763047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55975c1c4a00 0x55975c1ccfa0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.369+0000 7ffa249ce740 1 --2- 172.21.15.70:0/1456763047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55975c1ba140 0x55975c1cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.369+0000 7ff9faffd640 0 client.17016 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.369+0000 7ffa249ce740 1 --2- 172.21.15.70:0/1456763047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55975c1b9820 0x55975c1bf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.369+0000 7ffa249ce740 1 -- 172.21.15.70:0/1456763047 >> 172.21.15.70:0/1456763047 conn(0x55975c1b5d30 msgr2=0x55975c1c0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.369+0000 7ffa249ce740 1 -- 172.21.15.70:0/1456763047 shutdown_connections 2024-06-04T05:03:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.369+0000 7ffa249ce740 1 -- 172.21.15.70:0/1456763047 wait complete. 2024-06-04T05:03:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.369+0000 7ffa249ce740 20 client.17016 trim_cache size 0 max 16384 2024-06-04T05:03:05.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:03:05.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:03:05.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:03:05.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.370+0000 7ffa21bda640 1 -- 172.21.15.70:0/182347181 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ffa040b2a10 msgr2=0x7ffa040b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:05.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.370+0000 7ffa21bda640 1 --2- 172.21.15.70:0/182347181 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ffa040b2a10 0x7ffa040b4ed0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ffa14100e30 tx=0x7ffa140ed040 comp rx=0 tx=0).stop 2024-06-04T05:03:05.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.370+0000 7ffa21bda640 1 -- 172.21.15.70:0/182347181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa1c108a40 msgr2=0x7ffa1c0757d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:05.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.370+0000 7ffa21bda640 1 --2- 172.21.15.70:0/182347181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa1c108a40 0x7ffa1c0757d0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7ffa14005cc0 tx=0x7ffa14003c00 comp rx=0 tx=0).stop 2024-06-04T05:03:05.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.370+0000 7ffa21bda640 1 -- 172.21.15.70:0/182347181 shutdown_connections 2024-06-04T05:03:05.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.370+0000 7ffa21bda640 1 --2- 172.21.15.70:0/182347181 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ffa040b2a10 0x7ffa040b4ed0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.370+0000 7ffa21bda640 1 --2- 172.21.15.70:0/182347181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa1c1033f0 0x7ffa1c075290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.370+0000 7ffa21bda640 1 --2- 172.21.15.70:0/182347181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa1c108a40 0x7ffa1c0757d0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.370+0000 7ffa21bda640 1 --2- 172.21.15.70:0/182347181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa1c102a30 0x7ffa1c074d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:05.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.370+0000 7ffa21bda640 1 -- 172.21.15.70:0/182347181 >> 172.21.15.70:0/182347181 conn(0x7ffa1c0fe500 msgr2=0x7ffa1c0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:05.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.370+0000 7ffa21bda640 1 -- 172.21.15.70:0/182347181 shutdown_connections 2024-06-04T05:03:05.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:05.370+0000 7ffa21bda640 1 -- 172.21.15.70:0/182347181 wait complete. 2024-06-04T05:03:05.721 INFO:tasks.workunit.client.0.smithi070.stdout: 1 96947 2.35 MB/sec execute 118 sec latency 19.164 ms 2024-06-04T05:03:05.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:05 smithi070 ceph-mon[32631]: pgmap v180: 97 pgs: 97 active+clean; 62 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-06-04T05:03:05.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:05 smithi112 ceph-mon[36453]: pgmap v180: 97 pgs: 97 active+clean; 62 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-06-04T05:03:05.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:05 smithi098 ceph-mon[35569]: pgmap v180: 97 pgs: 97 active+clean; 62 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-06-04T05:03:06.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 97201 2.35 MB/sec execute 119 sec latency 617.099 ms 2024-06-04T05:03:07.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 97508 2.34 MB/sec execute 120 sec latency 972.758 ms 2024-06-04T05:03:07.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:07 smithi070 ceph-mon[32631]: pgmap v181: 97 pgs: 97 active+clean; 67 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 2.0 MiB/s wr, 220 op/s 2024-06-04T05:03:07.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:07 smithi112 ceph-mon[36453]: pgmap v181: 97 pgs: 97 active+clean; 67 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 2.0 MiB/s wr, 220 op/s 2024-06-04T05:03:07.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:07 smithi098 ceph-mon[35569]: pgmap v181: 97 pgs: 97 active+clean; 67 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 2.0 MiB/s wr, 220 op/s 2024-06-04T05:03:08.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 98153 2.33 MB/sec execute 121 sec latency 19.260 ms 2024-06-04T05:03:09.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 98781 2.36 MB/sec execute 122 sec latency 15.529 ms 2024-06-04T05:03:09.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:09 smithi070 ceph-mon[32631]: pgmap v182: 97 pgs: 97 active+clean; 67 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.3 MiB/s wr, 154 op/s 2024-06-04T05:03:09.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:09 smithi112 ceph-mon[36453]: pgmap v182: 97 pgs: 97 active+clean; 67 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.3 MiB/s wr, 154 op/s 2024-06-04T05:03:09.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:09 smithi098 ceph-mon[35569]: pgmap v182: 97 pgs: 97 active+clean; 67 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.3 MiB/s wr, 154 op/s 2024-06-04T05:03:10.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 99330 2.38 MB/sec execute 123 sec latency 14.761 ms 2024-06-04T05:03:11.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 99722 2.37 MB/sec execute 124 sec latency 15.891 ms 2024-06-04T05:03:11.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:11 smithi070 ceph-mon[32631]: pgmap v183: 97 pgs: 97 active+clean; 67 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.3 MiB/s wr, 154 op/s 2024-06-04T05:03:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:11 smithi112 ceph-mon[36453]: pgmap v183: 97 pgs: 97 active+clean; 67 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.3 MiB/s wr, 154 op/s 2024-06-04T05:03:11.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:11 smithi098 ceph-mon[35569]: pgmap v183: 97 pgs: 97 active+clean; 67 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.3 MiB/s wr, 154 op/s 2024-06-04T05:03:12.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 100023 2.35 MB/sec execute 125 sec latency 14.539 ms 2024-06-04T05:03:12.958 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:03:13.221 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:03:13.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:13 smithi070 ceph-mon[32631]: pgmap v184: 97 pgs: 97 active+clean; 68 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 1.8 MiB/s wr, 205 op/s 2024-06-04T05:03:13.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 100544 2.35 MB/sec execute 126 sec latency 15.175 ms 2024-06-04T05:03:13.797 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:13 smithi098 ceph-mon[35569]: pgmap v184: 97 pgs: 97 active+clean; 68 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 1.8 MiB/s wr, 205 op/s 2024-06-04T05:03:13.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:13 smithi112 ceph-mon[36453]: pgmap v184: 97 pgs: 97 active+clean; 68 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 1.8 MiB/s wr, 205 op/s 2024-06-04T05:03:14.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 101068 2.36 MB/sec execute 127 sec latency 19.662 ms 2024-06-04T05:03:15.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.021+0000 7f61c1025700 1 -- 172.21.15.70:0/2557220141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61bc102e00 msgr2=0x7f61bc10f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:15.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.021+0000 7f61c1025700 1 --2- 172.21.15.70:0/2557220141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61bc102e00 0x7f61bc10f2f0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f61b400b0a0 tx=0x7f61b400b3b0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.021+0000 7f61c1025700 1 -- 172.21.15.70:0/2557220141 shutdown_connections 2024-06-04T05:03:15.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.021+0000 7f61c1025700 1 --2- 172.21.15.70:0/2557220141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61bc102e00 0x7f61bc10f2f0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.021+0000 7f61c1025700 1 --2- 172.21.15.70:0/2557220141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61bc108520 0x7f61bc108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.021+0000 7f61c1025700 1 --2- 172.21.15.70:0/2557220141 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61bc102440 0x7f61bc1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.021+0000 7f61c1025700 1 -- 172.21.15.70:0/2557220141 >> 172.21.15.70:0/2557220141 conn(0x7f61bc0793f0 msgr2=0x7f61bc079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:15.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.021+0000 7f61c1025700 1 -- 172.21.15.70:0/2557220141 shutdown_connections 2024-06-04T05:03:15.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.021+0000 7f61c1025700 1 -- 172.21.15.70:0/2557220141 wait complete. 2024-06-04T05:03:15.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.021+0000 7f61c1025700 1 Processor -- start 2024-06-04T05:03:15.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.022+0000 7f61c1025700 1 -- start start 2024-06-04T05:03:15.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.022+0000 7f61c1025700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61bc102440 0x7f61bc19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.022+0000 7f61c1025700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61bc102e00 0x7f61bc19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.022+0000 7f61c1025700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61bc108520 0x7f61bc1998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.022+0000 7f61c1025700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc112480 con 0x7f61bc108520 2024-06-04T05:03:15.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.022+0000 7f61c1025700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc112300 con 0x7f61bc102440 2024-06-04T05:03:15.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.022+0000 7f61c1025700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc112600 con 0x7f61bc102e00 2024-06-04T05:03:15.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.022+0000 7f61bb59e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61bc108520 0x7f61bc1998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.022+0000 7f61bb59e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61bc108520 0x7f61bc1998e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46862/0 (socket says 172.21.15.70:46862) 2024-06-04T05:03:15.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.022+0000 7f61bad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61bc102440 0x7f61bc19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.022+0000 7f61bb59e700 1 -- 172.21.15.70:0/4263740287 learned_addr learned my addr 172.21.15.70:0/4263740287 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:15.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.022+0000 7f61ba59c700 1 --2- 172.21.15.70:0/4263740287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61bc102e00 0x7f61bc19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.022+0000 7f61bb59e700 1 -- 172.21.15.70:0/4263740287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61bc102e00 msgr2=0x7f61bc19fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:15.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.022+0000 7f61bb59e700 1 --2- 172.21.15.70:0/4263740287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61bc102e00 0x7f61bc19fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.022+0000 7f61bb59e700 1 -- 172.21.15.70:0/4263740287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61bc102440 msgr2=0x7f61bc19f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:15.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.022+0000 7f61bb59e700 1 --2- 172.21.15.70:0/4263740287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61bc102440 0x7f61bc19f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.022+0000 7f61bb59e700 1 -- 172.21.15.70:0/4263740287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61bc19a040 con 0x7f61bc108520 2024-06-04T05:03:15.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.023+0000 7f61bb59e700 1 --2- 172.21.15.70:0/4263740287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61bc108520 0x7f61bc1998e0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f61b400b780 tx=0x7f61b4004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:15.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.023+0000 7f61a3fff700 1 -- 172.21.15.70:0/4263740287 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b4021070 con 0x7f61bc108520 2024-06-04T05:03:15.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.023+0000 7f61c1025700 1 -- 172.21.15.70:0/4263740287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61bc19a2d0 con 0x7f61bc108520 2024-06-04T05:03:15.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.023+0000 7f61c1025700 1 -- 172.21.15.70:0/4263740287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61bc1a65f0 con 0x7f61bc108520 2024-06-04T05:03:15.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.023+0000 7f61a3fff700 1 -- 172.21.15.70:0/4263740287 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f61b4007b80 con 0x7f61bc108520 2024-06-04T05:03:15.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.023+0000 7f61a3fff700 1 -- 172.21.15.70:0/4263740287 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b40052a0 con 0x7f61bc108520 2024-06-04T05:03:15.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.025+0000 7f61a3fff700 1 -- 172.21.15.70:0/4263740287 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f61b4024070 con 0x7f61bc108520 2024-06-04T05:03:15.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.026+0000 7f61a3fff700 1 --2- 172.21.15.70:0/4263740287 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f61a40b2a70 0x7f61a40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.026+0000 7f61bad9d700 1 --2- 172.21.15.70:0/4263740287 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f61a40b2a70 0x7f61a40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.026+0000 7f61a3fff700 1 -- 172.21.15.70:0/4263740287 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f61b400f020 con 0x7f61bc108520 2024-06-04T05:03:15.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61cb8bab80 1 Processor -- start 2024-06-04T05:03:15.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61bad9d700 1 --2- 172.21.15.70:0/4263740287 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f61a40b2a70 0x7f61a40b4f30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f61bc19aa20 tx=0x7f61ac00b410 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:15.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61cb8bab80 1 -- start start 2024-06-04T05:03:15.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61cb8bab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d90dc70 0x561c9d90e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61cb8bab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90e620 0x561c9d911d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61cb8bab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d912370 0x561c9d9127b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61cb8bab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561c9d868c90 con 0x561c9d90e620 2024-06-04T05:03:15.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61cb8bab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561c9d858830 con 0x561c9d90dc70 2024-06-04T05:03:15.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61cb8bab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561c9d85c870 con 0x561c9d912370 2024-06-04T05:03:15.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61ba59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90e620 0x561c9d911d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61bb59e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d912370 0x561c9d9127b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61bb59e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d912370 0x561c9d9127b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:42256/0 (socket says 172.21.15.70:42256) 2024-06-04T05:03:15.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61ba59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90e620 0x561c9d911d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46874/0 (socket says 172.21.15.70:46874) 2024-06-04T05:03:15.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61bb59e700 1 -- 172.21.15.70:0/2475853525 learned_addr learned my addr 172.21.15.70:0/2475853525 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:15.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61bad9d700 1 --2- 172.21.15.70:0/2475853525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d90dc70 0x561c9d90e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61bb59e700 1 -- 172.21.15.70:0/2475853525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d90dc70 msgr2=0x561c9d90e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:15.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61bb59e700 1 --2- 172.21.15.70:0/2475853525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d90dc70 0x561c9d90e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.027+0000 7f61bb59e700 1 -- 172.21.15.70:0/2475853525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90e620 msgr2=0x561c9d911d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:15.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.028+0000 7f61bb59e700 1 --2- 172.21.15.70:0/2475853525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90e620 0x561c9d911d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.028+0000 7f61bb59e700 1 -- 172.21.15.70:0/2475853525 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c9d7a88d0 con 0x561c9d912370 2024-06-04T05:03:15.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.028+0000 7f61bb59e700 1 --2- 172.21.15.70:0/2475853525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d912370 0x561c9d9127b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f61b40a6f20 tx=0x7f61b40aa6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e38aa599b8e6134 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:15.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.028+0000 7f61a17fa700 1 -- 172.21.15.70:0/2475853525 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b40aaea0 con 0x561c9d912370 2024-06-04T05:03:15.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.028+0000 7f61a17fa700 1 -- 172.21.15.70:0/2475853525 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f61b40af250 con 0x561c9d912370 2024-06-04T05:03:15.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.028+0000 7f61a17fa700 1 -- 172.21.15.70:0/2475853525 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b40ab360 con 0x561c9d912370 2024-06-04T05:03:15.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.029+0000 7f61cb8bab80 1 -- 172.21.15.70:0/2475853525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d912370 msgr2=0x561c9d9127b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.029+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/2475853525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d912370 0x561c9d9127b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f61b40a6f20 tx=0x7f61b40aa6e0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.029+0000 7f61cb8bab80 1 -- 172.21.15.70:0/2475853525 shutdown_connections 2024-06-04T05:03:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.029+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/2475853525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d912370 0x561c9d9127b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.029+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/2475853525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d90dc70 0x561c9d90e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.029+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/2475853525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90e620 0x561c9d911d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.029+0000 7f61cb8bab80 1 -- 172.21.15.70:0/2475853525 >> 172.21.15.70:0/2475853525 conn(0x561c9d837080 msgr2=0x561c9d903e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.029+0000 7f61cb8bab80 1 -- 172.21.15.70:0/2475853525 shutdown_connections 2024-06-04T05:03:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.029+0000 7f61cb8bab80 1 -- 172.21.15.70:0/2475853525 wait complete. 2024-06-04T05:03:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.029+0000 7f61cb8bab80 1 -- start start 2024-06-04T05:03:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.029+0000 7f61cb8bab80 1 Processor -- start 2024-06-04T05:03:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61cb8bab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d90dc70 0x561c9d9198c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61cb8bab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90e620 0x561c9d916bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61cb8bab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d912370 0x561c9d917100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61cb8bab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561c9d75efa0 con 0x561c9d90e620 2024-06-04T05:03:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61cb8bab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561c9d7fa7c0 con 0x561c9d90dc70 2024-06-04T05:03:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61cb8bab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561c9d91a2f0 con 0x561c9d912370 2024-06-04T05:03:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61cb8bab80 10 client.0 _ll_register_callbacks cb 0x561c9d7f3920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61ba59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90e620 0x561c9d916bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61bad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d90dc70 0x561c9d9198c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61ba59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90e620 0x561c9d916bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46876/0 (socket says 172.21.15.70:46876) 2024-06-04T05:03:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61bad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d90dc70 0x561c9d9198c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54386/0 (socket says 172.21.15.70:54386) 2024-06-04T05:03:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61ba59c700 1 -- 172.21.15.70:0/3779404752 learned_addr learned my addr 172.21.15.70:0/3779404752 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61bb59e700 1 --2- 172.21.15.70:0/3779404752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d912370 0x561c9d917100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61ba59c700 1 -- 172.21.15.70:0/3779404752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d912370 msgr2=0x561c9d917100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61ba59c700 1 --2- 172.21.15.70:0/3779404752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d912370 0x561c9d917100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61ba59c700 1 -- 172.21.15.70:0/3779404752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d90dc70 msgr2=0x561c9d9198c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61ba59c700 1 --2- 172.21.15.70:0/3779404752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d90dc70 0x561c9d9198c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61ba59c700 1 -- 172.21.15.70:0/3779404752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c9d7f7690 con 0x561c9d90e620 2024-06-04T05:03:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61bad9d700 1 --2- 172.21.15.70:0/3779404752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d90dc70 0x561c9d9198c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f61ba59c700 1 --2- 172.21.15.70:0/3779404752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90e620 0x561c9d916bc0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f61b0012740 tx=0x7f61b0012a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e631d7b80bf2635 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.030+0000 7f619eff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.031+0000 7f619eff5700 1 -- 172.21.15.70:0/3779404752 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b0004260 con 0x561c9d90e620 2024-06-04T05:03:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.031+0000 7f619eff5700 1 -- 172.21.15.70:0/3779404752 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f61b0004400 con 0x561c9d90e620 2024-06-04T05:03:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.031+0000 7f61cb8bab80 1 -- 172.21.15.70:0/3779404752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61b40af250 con 0x561c9d90e620 2024-06-04T05:03:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.031+0000 7f619eff5700 1 -- 172.21.15.70:0/3779404752 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f61b0003680 con 0x561c9d90e620 2024-06-04T05:03:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.031+0000 7f61cb8bab80 10 client.17028 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.031+0000 7f61cb8bab80 1 -- 172.21.15.70:0/3779404752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561c9d924ac0 con 0x561c9d90e620 2024-06-04T05:03:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.031+0000 7f619eff5700 1 -- 172.21.15.70:0/3779404752 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f61b40af250 con 0x561c9d90e620 2024-06-04T05:03:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.031+0000 7f619eff5700 1 -- 172.21.15.70:0/3779404752 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f61b0011990 con 0x561c9d90e620 2024-06-04T05:03:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.031+0000 7f61cb8bab80 10 client.17028 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.031+0000 7f61cb8bab80 10 client.17028 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:15.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.031+0000 7f61cb8bab80 20 client.17028 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:15.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.031+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/3779404752 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561c9d927490 0x561c9d9298f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.031+0000 7f61cb8bab80 4 client.17028 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:03:15.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.031+0000 7f61cb8bab80 1 -- 172.21.15.70:0/3779404752 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c9d8d59e0 con 0x561c9d927490 2024-06-04T05:03:15.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.032+0000 7f61bad9d700 1 --2- 172.21.15.70:0/3779404752 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561c9d927490 0x561c9d9298f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.032+0000 7f619eff5700 10 client.17028 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:15.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.032+0000 7f61bad9d700 1 --2- 172.21.15.70:0/3779404752 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561c9d927490 0x561c9d9298f0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d2bfbc2908e4e4 server_cookie=8da26ca70e8f3a19 in_seq=0 out_seq=0 2024-06-04T05:03:15.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.033+0000 7f619eff5700 1 -- 172.21.15.70:0/3779404752 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561c9d8d59e0 con 0x561c9d927490 2024-06-04T05:03:15.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.033+0000 7f619eff5700 10 client.17028 handle_command_reply: tid=0 2024-06-04T05:03:15.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.033+0000 7f61cb8bab80 1 client.17028 shutdown 2024-06-04T05:03:15.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.033+0000 7f61cb8bab80 1 -- 172.21.15.70:0/3779404752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90e620 msgr2=0x561c9d916bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:15.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.033+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/3779404752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90e620 0x561c9d916bc0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f61b0012740 tx=0x7f61b0012a50 comp rx=0 tx=0).stop 2024-06-04T05:03:15.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.033+0000 7f61cb8bab80 1 -- 172.21.15.70:0/3779404752 shutdown_connections 2024-06-04T05:03:15.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.033+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/3779404752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d912370 0x561c9d917100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.033+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/3779404752 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561c9d927490 0x561c9d9298f0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.033+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/3779404752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d90dc70 0x561c9d9198c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.033+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/3779404752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90e620 0x561c9d916bc0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.033+0000 7f61cb8bab80 1 -- 172.21.15.70:0/3779404752 >> 172.21.15.70:0/3779404752 conn(0x561c9d837080 msgr2=0x561c9d903510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:15.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.033+0000 7f619eff5700 0 client.17028 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:15.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.034+0000 7f61cb8bab80 1 -- 172.21.15.70:0/3779404752 shutdown_connections 2024-06-04T05:03:15.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.034+0000 7f61cb8bab80 1 -- 172.21.15.70:0/3779404752 wait complete. 2024-06-04T05:03:15.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.034+0000 7f61cb8bab80 20 client.17028 trim_cache size 0 max 16384 2024-06-04T05:03:15.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61cb8bab80 1 Processor -- start 2024-06-04T05:03:15.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61cb8bab80 1 -- start start 2024-06-04T05:03:15.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61cb8bab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90dc70 0x561c9d923350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61cb8bab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d912370 0x561c9d9113d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61cb8bab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d903c90 0x561c9d9040f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61cb8bab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc1abb20 con 0x561c9d90dc70 2024-06-04T05:03:15.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61cb8bab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc1ad660 con 0x561c9d912370 2024-06-04T05:03:15.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61cb8bab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc1ac1f0 con 0x561c9d903c90 2024-06-04T05:03:15.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61bad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90dc70 0x561c9d923350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61bad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90dc70 0x561c9d923350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46892/0 (socket says 172.21.15.70:46892) 2024-06-04T05:03:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61bad9d700 1 -- 172.21.15.70:0/2305503405 learned_addr learned my addr 172.21.15.70:0/2305503405 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61bb59e700 1 --2- 172.21.15.70:0/2305503405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d903c90 0x561c9d9040f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61ba59c700 1 --2- 172.21.15.70:0/2305503405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d912370 0x561c9d9113d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61bad9d700 1 -- 172.21.15.70:0/2305503405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d903c90 msgr2=0x561c9d9040f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61bad9d700 1 --2- 172.21.15.70:0/2305503405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d903c90 0x561c9d9040f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61bad9d700 1 -- 172.21.15.70:0/2305503405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d912370 msgr2=0x561c9d9113d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61bad9d700 1 --2- 172.21.15.70:0/2305503405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d912370 0x561c9d9113d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.059+0000 7f61bad9d700 1 -- 172.21.15.70:0/2305503405 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c9d8fc5b0 con 0x561c9d90dc70 2024-06-04T05:03:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.060+0000 7f61bad9d700 1 --2- 172.21.15.70:0/2305503405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90dc70 0x561c9d923350 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f61ac005950 tx=0x7f61ac013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce6103964ce47130 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.060+0000 7f61a0ff9700 1 -- 172.21.15.70:0/2305503405 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac020d60 con 0x561c9d90dc70 2024-06-04T05:03:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.060+0000 7f61a0ff9700 1 -- 172.21.15.70:0/2305503405 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f61ac030070 con 0x561c9d90dc70 2024-06-04T05:03:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.060+0000 7f61a0ff9700 1 -- 172.21.15.70:0/2305503405 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac0296c0 con 0x561c9d90dc70 2024-06-04T05:03:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 1 -- 172.21.15.70:0/2305503405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90dc70 msgr2=0x561c9d923350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/2305503405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90dc70 0x561c9d923350 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f61ac005950 tx=0x7f61ac013000 comp rx=0 tx=0).stop 2024-06-04T05:03:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 1 -- 172.21.15.70:0/2305503405 shutdown_connections 2024-06-04T05:03:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/2305503405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d903c90 0x561c9d9040f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/2305503405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d90dc70 0x561c9d923350 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/2305503405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d912370 0x561c9d9113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 1 -- 172.21.15.70:0/2305503405 >> 172.21.15.70:0/2305503405 conn(0x561c9d837080 msgr2=0x561c9d910cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 1 -- 172.21.15.70:0/2305503405 shutdown_connections 2024-06-04T05:03:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 1 -- 172.21.15.70:0/2305503405 wait complete. 2024-06-04T05:03:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 1 -- start start 2024-06-04T05:03:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 1 Processor -- start 2024-06-04T05:03:15.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d903c90 0x561c9d91d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d90dc70 0x561c9d91d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d912370 0x561c9d91dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc1acf90 con 0x561c9d912370 2024-06-04T05:03:15.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc1ac8c0 con 0x561c9d90dc70 2024-06-04T05:03:15.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc1a9ae0 con 0x561c9d903c90 2024-06-04T05:03:15.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.061+0000 7f61cb8bab80 10 client.0 _ll_register_callbacks cb 0x561c9d7f3920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:15.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f61bad9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d903c90 0x561c9d91d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f61bad9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d903c90 0x561c9d91d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:42284/0 (socket says 172.21.15.70:42284) 2024-06-04T05:03:15.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f61bad9d700 1 -- 172.21.15.70:0/1386330706 learned_addr learned my addr 172.21.15.70:0/1386330706 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:15.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f61ba59c700 1 --2- 172.21.15.70:0/1386330706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d90dc70 0x561c9d91d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f61bb59e700 1 --2- 172.21.15.70:0/1386330706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d912370 0x561c9d91dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f61bad9d700 1 -- 172.21.15.70:0/1386330706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d90dc70 msgr2=0x561c9d91d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:15.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f61bad9d700 1 --2- 172.21.15.70:0/1386330706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d90dc70 0x561c9d91d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f61bad9d700 1 -- 172.21.15.70:0/1386330706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d912370 msgr2=0x561c9d91dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:15.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f61bad9d700 1 --2- 172.21.15.70:0/1386330706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d912370 0x561c9d91dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f61bad9d700 1 -- 172.21.15.70:0/1386330706 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c9d905970 con 0x561c9d903c90 2024-06-04T05:03:15.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f61bad9d700 1 --2- 172.21.15.70:0/1386330706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d903c90 0x561c9d91d1d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f61ac02ac40 tx=0x7f61ac02ac70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5db2828f6e6a92b0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:15.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f619eff5700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:03:15.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f619eff5700 1 -- 172.21.15.70:0/1386330706 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac005240 con 0x561c9d903c90 2024-06-04T05:03:15.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f619eff5700 1 -- 172.21.15.70:0/1386330706 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f61ac0053e0 con 0x561c9d903c90 2024-06-04T05:03:15.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f61cb8bab80 1 -- 172.21.15.70:0/1386330706 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61ac030070 con 0x561c9d903c90 2024-06-04T05:03:15.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f619eff5700 1 -- 172.21.15.70:0/1386330706 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f61ac0055e0 con 0x561c9d903c90 2024-06-04T05:03:15.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f61cb8bab80 10 client.26392 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:15.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.062+0000 7f61cb8bab80 1 -- 172.21.15.70:0/1386330706 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561c9d992130 con 0x561c9d903c90 2024-06-04T05:03:15.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.063+0000 7f619eff5700 1 -- 172.21.15.70:0/1386330706 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f61ac030070 con 0x561c9d903c90 2024-06-04T05:03:15.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.063+0000 7f619eff5700 1 -- 172.21.15.70:0/1386330706 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f61ac004a80 con 0x561c9d903c90 2024-06-04T05:03:15.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.063+0000 7f61cb8bab80 10 client.26392 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:15.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.063+0000 7f61cb8bab80 10 client.26392 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:15.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.063+0000 7f61cb8bab80 20 client.26392 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:15.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.063+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/1386330706 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561c9d992750 0x561c9d994b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:15.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.063+0000 7f61cb8bab80 4 client.26392 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:03:15.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.063+0000 7f61cb8bab80 1 -- 172.21.15.70:0/1386330706 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561c9d86abe0 con 0x561c9d992750 2024-06-04T05:03:15.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.063+0000 7f61ba59c700 1 --2- 172.21.15.70:0/1386330706 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561c9d992750 0x561c9d994b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:15.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.063+0000 7f619eff5700 10 client.26392 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:15.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.063+0000 7f61ba59c700 1 --2- 172.21.15.70:0/1386330706 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561c9d992750 0x561c9d994b70 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da81bc7d8cf82566 server_cookie=6a40dd0867dcf331 in_seq=0 out_seq=0 2024-06-04T05:03:15.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.064+0000 7f619eff5700 1 -- 172.21.15.70:0/1386330706 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561c9d86abe0 con 0x561c9d992750 2024-06-04T05:03:15.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.064+0000 7f619eff5700 10 client.26392 handle_command_reply: tid=0 2024-06-04T05:03:15.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.064+0000 7f61cb8bab80 1 client.26392 shutdown 2024-06-04T05:03:15.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.064+0000 7f61cb8bab80 1 -- 172.21.15.70:0/1386330706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d903c90 msgr2=0x561c9d91d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:15.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.064+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/1386330706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d903c90 0x561c9d91d1d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f61ac02ac40 tx=0x7f61ac02ac70 comp rx=0 tx=0).stop 2024-06-04T05:03:15.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.064+0000 7f61cb8bab80 1 -- 172.21.15.70:0/1386330706 shutdown_connections 2024-06-04T05:03:15.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.064+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/1386330706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c9d912370 0x561c9d91dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.064+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/1386330706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c9d903c90 0x561c9d91d1d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.064+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/1386330706 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561c9d992750 0x561c9d994b70 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.064+0000 7f61cb8bab80 1 --2- 172.21.15.70:0/1386330706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c9d90dc70 0x561c9d91d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.064+0000 7f619eff5700 0 client.26392 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:15.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.064+0000 7f61cb8bab80 1 -- 172.21.15.70:0/1386330706 >> 172.21.15.70:0/1386330706 conn(0x561c9d837080 msgr2=0x561c9d903510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:15.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.064+0000 7f61cb8bab80 1 -- 172.21.15.70:0/1386330706 shutdown_connections 2024-06-04T05:03:15.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.064+0000 7f61cb8bab80 1 -- 172.21.15.70:0/1386330706 wait complete. 2024-06-04T05:03:15.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.065+0000 7f61cb8bab80 20 client.26392 trim_cache size 0 max 16384 2024-06-04T05:03:15.072 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:03:15.073 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:03:15.073 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:03:15.073 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:03:15.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.065+0000 7f61c1025700 1 -- 172.21.15.70:0/4263740287 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f61a40b2a70 msgr2=0x7f61a40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:15.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.065+0000 7f61c1025700 1 --2- 172.21.15.70:0/4263740287 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f61a40b2a70 0x7f61a40b4f30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f61bc19aa20 tx=0x7f61ac00b410 comp rx=0 tx=0).stop 2024-06-04T05:03:15.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.065+0000 7f61c1025700 1 -- 172.21.15.70:0/4263740287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61bc108520 msgr2=0x7f61bc1998e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:15.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.065+0000 7f61c1025700 1 --2- 172.21.15.70:0/4263740287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61bc108520 0x7f61bc1998e0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f61b400b780 tx=0x7f61b4004d00 comp rx=0 tx=0).stop 2024-06-04T05:03:15.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.065+0000 7f61c1025700 1 -- 172.21.15.70:0/4263740287 shutdown_connections 2024-06-04T05:03:15.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.065+0000 7f61c1025700 1 --2- 172.21.15.70:0/4263740287 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f61a40b2a70 0x7f61a40b4f30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.065+0000 7f61c1025700 1 --2- 172.21.15.70:0/4263740287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61bc108520 0x7f61bc1998e0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.065+0000 7f61c1025700 1 --2- 172.21.15.70:0/4263740287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61bc102440 0x7f61bc19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.065+0000 7f61c1025700 1 --2- 172.21.15.70:0/4263740287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61bc102e00 0x7f61bc19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:15.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.065+0000 7f61c1025700 1 -- 172.21.15.70:0/4263740287 >> 172.21.15.70:0/4263740287 conn(0x7f61bc0793f0 msgr2=0x7f61bc06b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:15.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.065+0000 7f61c1025700 1 -- 172.21.15.70:0/4263740287 shutdown_connections 2024-06-04T05:03:15.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:15.065+0000 7f61c1025700 1 -- 172.21.15.70:0/4263740287 wait complete. 2024-06-04T05:03:15.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 101699 2.35 MB/sec execute 128 sec latency 57.891 ms 2024-06-04T05:03:15.762 INFO:tasks.quiescer.fs.[cephfs]:Total duration (37.9) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-06-04T05:03:15.762 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 18.95 2024-06-04T05:03:15.762 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:03:15.788 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:15 smithi070 ceph-mon[32631]: pgmap v185: 97 pgs: 97 active+clean; 68 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-06-04T05:03:15.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:15 smithi112 ceph-mon[36453]: pgmap v185: 97 pgs: 97 active+clean; 68 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-06-04T05:03:15.945 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:03:15.945 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:03:15.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:15 smithi098 ceph-mon[35569]: pgmap v185: 97 pgs: 97 active+clean; 68 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-06-04T05:03:16.029 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:03:16.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 102289 2.38 MB/sec execute 129 sec latency 11.662 ms 2024-06-04T05:03:17.174 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:03:17.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 102733 2.37 MB/sec execute 130 sec latency 47.347 ms 2024-06-04T05:03:17.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:17 smithi070 ceph-mon[32631]: pgmap v186: 97 pgs: 97 active+clean; 68 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 1.8 MiB/s wr, 187 op/s 2024-06-04T05:03:17.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:17 smithi112 ceph-mon[36453]: pgmap v186: 97 pgs: 97 active+clean; 68 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 1.8 MiB/s wr, 187 op/s 2024-06-04T05:03:17.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:17 smithi098 ceph-mon[35569]: pgmap v186: 97 pgs: 97 active+clean; 68 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 1.8 MiB/s wr, 187 op/s 2024-06-04T05:03:18.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.464+0000 7fb9c10f1640 1 -- 172.21.15.70:0/175210985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9bc102340 msgr2=0x7fb9bc10e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:18.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.464+0000 7fb9c10f1640 1 --2- 172.21.15.70:0/175210985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9bc102340 0x7fb9bc10e800 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009cd0 tx=0x7fb9b402a8e0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.464+0000 7fb9c10f1640 1 -- 172.21.15.70:0/175210985 shutdown_connections 2024-06-04T05:03:18.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.464+0000 7fb9c10f1640 1 --2- 172.21.15.70:0/175210985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9bc101980 0x7fb9bc101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.464+0000 7fb9c10f1640 1 --2- 172.21.15.70:0/175210985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9bc102340 0x7fb9bc10e800 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.464+0000 7fb9c10f1640 1 --2- 172.21.15.70:0/175210985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9bc107940 0x7fb9bc107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.464+0000 7fb9c10f1640 1 -- 172.21.15.70:0/175210985 >> 172.21.15.70:0/175210985 conn(0x7fb9bc078ad0 msgr2=0x7fb9bc078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:18.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.464+0000 7fb9c10f1640 1 -- 172.21.15.70:0/175210985 shutdown_connections 2024-06-04T05:03:18.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.464+0000 7fb9c10f1640 1 -- 172.21.15.70:0/175210985 wait complete. 2024-06-04T05:03:18.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9c10f1640 1 Processor -- start 2024-06-04T05:03:18.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9c10f1640 1 -- start start 2024-06-04T05:03:18.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9c10f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9bc101980 0x7fb9bc197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9c10f1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9bc102340 0x7fb9bc1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9c10f1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9bc107940 0x7fb9bc19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9c10f1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc111ab0 con 0x7fb9bc101980 2024-06-04T05:03:18.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9c10f1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc111930 con 0x7fb9bc102340 2024-06-04T05:03:18.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9c10f1640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc111c30 con 0x7fb9bc107940 2024-06-04T05:03:18.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9bad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9bc101980 0x7fb9bc197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9bad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9bc101980 0x7fb9bc197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42510/0 (socket says 172.21.15.70:42510) 2024-06-04T05:03:18.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9bad9d640 1 -- 172.21.15.70:0/417813611 learned_addr learned my addr 172.21.15.70:0/417813611 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:18.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9ba59c640 1 --2- 172.21.15.70:0/417813611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9bc102340 0x7fb9bc1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9bb59e640 1 --2- 172.21.15.70:0/417813611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9bc107940 0x7fb9bc19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9bad9d640 1 -- 172.21.15.70:0/417813611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9bc107940 msgr2=0x7fb9bc19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:18.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9bad9d640 1 --2- 172.21.15.70:0/417813611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9bc107940 0x7fb9bc19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9bad9d640 1 -- 172.21.15.70:0/417813611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9bc102340 msgr2=0x7fb9bc1984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:18.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9bad9d640 1 --2- 172.21.15.70:0/417813611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9bc102340 0x7fb9bc1984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.465+0000 7fb9bad9d640 1 -- 172.21.15.70:0/417813611 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9bc19cf60 con 0x7fb9bc101980 2024-06-04T05:03:18.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.466+0000 7fb9bad9d640 1 --2- 172.21.15.70:0/417813611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9bc101980 0x7fb9bc197f70 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac00ea60 tx=0x7fb9ac00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:18.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.466+0000 7fb9a3fff640 1 -- 172.21.15.70:0/417813611 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac00cde0 con 0x7fb9bc101980 2024-06-04T05:03:18.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.466+0000 7fb9a3fff640 1 -- 172.21.15.70:0/417813611 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb9ac0035f0 con 0x7fb9bc101980 2024-06-04T05:03:18.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.466+0000 7fb9c10f1640 1 -- 172.21.15.70:0/417813611 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9bc19d250 con 0x7fb9bc101980 2024-06-04T05:03:18.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.466+0000 7fb9a3fff640 1 -- 172.21.15.70:0/417813611 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac010690 con 0x7fb9bc101980 2024-06-04T05:03:18.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.466+0000 7fb9c10f1640 1 -- 172.21.15.70:0/417813611 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9bc1a4ae0 con 0x7fb9bc101980 2024-06-04T05:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.468+0000 7fb9c3e0b740 1 Processor -- start 2024-06-04T05:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9a3fff640 1 -- 172.21.15.70:0/417813611 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fb9ac004000 con 0x7fb9bc101980 2024-06-04T05:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9c3e0b740 1 -- start start 2024-06-04T05:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9c3e0b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f3760 0x5627917f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9c3e0b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f4080 0x5627918006d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9c3e0b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x562791803000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9c3e0b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562791765c30 con 0x562791800c10 2024-06-04T05:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9c3e0b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5627917557d0 con 0x5627917f4080 2024-06-04T05:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9c3e0b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562791759810 con 0x5627917f3760 2024-06-04T05:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9ba59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x562791803000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9ba59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x562791803000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42524/0 (socket says 172.21.15.70:42524) 2024-06-04T05:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9bad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f4080 0x5627918006d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9ba59c640 1 -- 172.21.15.70:0/2140447901 learned_addr learned my addr 172.21.15.70:0/2140447901 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:18.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9ba59c640 1 -- 172.21.15.70:0/2140447901 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f3760 msgr2=0x5627917f3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:18.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9ba59c640 1 --2- 172.21.15.70:0/2140447901 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f3760 0x5627917f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9ba59c640 1 -- 172.21.15.70:0/2140447901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f4080 msgr2=0x5627918006d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:18.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9ba59c640 1 --2- 172.21.15.70:0/2140447901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f4080 0x5627918006d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9ba59c640 1 -- 172.21.15.70:0/2140447901 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56279146a210 con 0x562791800c10 2024-06-04T05:03:18.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9a3fff640 1 --2- 172.21.15.70:0/417813611 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9a40b2a10 0x7fb9a40b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.469+0000 7fb9ba59c640 1 --2- 172.21.15.70:0/2140447901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x562791803000 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009c90 tx=0x7fb9b0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cdada79751004ee server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:18.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.470+0000 7fb9bb59e640 1 --2- 172.21.15.70:0/417813611 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9a40b2a10 0x7fb9a40b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.470+0000 7fb9a17fa640 1 -- 172.21.15.70:0/2140447901 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b001e070 con 0x562791800c10 2024-06-04T05:03:18.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.470+0000 7fb9a17fa640 1 -- 172.21.15.70:0/2140447901 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb9b0019ac0 con 0x562791800c10 2024-06-04T05:03:18.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.470+0000 7fb9a3fff640 1 -- 172.21.15.70:0/417813611 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb9ac0e2850 con 0x7fb9bc101980 2024-06-04T05:03:18.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.470+0000 7fb9a17fa640 1 -- 172.21.15.70:0/2140447901 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b000e930 con 0x562791800c10 2024-06-04T05:03:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.470+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/2140447901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 msgr2=0x562791803000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.470+0000 7fb9bb59e640 1 --2- 172.21.15.70:0/417813611 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9a40b2a10 0x7fb9a40b4ed0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009430 tx=0x7fb9b4009360 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.470+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/2140447901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x562791803000 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009c90 tx=0x7fb9b0011810 comp rx=0 tx=0).stop 2024-06-04T05:03:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/2140447901 shutdown_connections 2024-06-04T05:03:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/2140447901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f4080 0x5627918006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/2140447901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x562791803000 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/2140447901 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f3760 0x5627917f3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/2140447901 >> 172.21.15.70:0/2140447901 conn(0x5627917efd30 msgr2=0x5627917f0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/2140447901 shutdown_connections 2024-06-04T05:03:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/2140447901 wait complete. 2024-06-04T05:03:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9c3e0b740 1 -- start start 2024-06-04T05:03:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9c3e0b740 1 Processor -- start 2024-06-04T05:03:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9c3e0b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5627917f3760 0x5627917fdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9c3e0b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f4080 0x5627917fe160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9c3e0b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562791800c10 0x5627917fe6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9c3e0b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56279143b790 con 0x5627917f3760 2024-06-04T05:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9c3e0b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562791711890 con 0x562791800c10 2024-06-04T05:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9c3e0b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5627918037e0 con 0x5627917f4080 2024-06-04T05:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9c3e0b740 10 client.0 _ll_register_callbacks cb 0x562791469a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9bad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5627917f3760 0x5627917fdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9bad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5627917f3760 0x5627917fdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42528/0 (socket says 172.21.15.70:42528) 2024-06-04T05:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9bad9d640 1 -- 172.21.15.70:0/1213605382 learned_addr learned my addr 172.21.15.70:0/1213605382 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9ba59c640 1 --2- 172.21.15.70:0/1213605382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f4080 0x5627917fe160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.471+0000 7fb9bb59e640 1 --2- 172.21.15.70:0/1213605382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562791800c10 0x5627917fe6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.472+0000 7fb9bad9d640 1 -- 172.21.15.70:0/1213605382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f4080 msgr2=0x5627917fe160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.472+0000 7fb9bad9d640 1 --2- 172.21.15.70:0/1213605382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f4080 0x5627917fe160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.472+0000 7fb9bad9d640 1 -- 172.21.15.70:0/1213605382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562791800c10 msgr2=0x5627917fe6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.472+0000 7fb9bad9d640 1 --2- 172.21.15.70:0/1213605382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562791800c10 0x5627917fe6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.472+0000 7fb9bad9d640 1 -- 172.21.15.70:0/1213605382 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562791536620 con 0x5627917f3760 2024-06-04T05:03:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.472+0000 7fb9ba59c640 1 --2- 172.21.15.70:0/1213605382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f4080 0x5627917fe160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.472+0000 7fb9bad9d640 1 --2- 172.21.15.70:0/1213605382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5627917f3760 0x5627917fdc20 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac009920 tx=0x7fb9ac0e84d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ed6dcfabd49c40 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.472+0000 7fb99eff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.472+0000 7fb99eff5640 1 -- 172.21.15.70:0/1213605382 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac0aae60 con 0x5627917f3760 2024-06-04T05:03:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.472+0000 7fb99eff5640 1 -- 172.21.15.70:0/1213605382 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb9ac0a5dc0 con 0x5627917f3760 2024-06-04T05:03:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.472+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/1213605382 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9b0019ac0 con 0x5627917f3760 2024-06-04T05:03:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.472+0000 7fb99eff5640 1 -- 172.21.15.70:0/1213605382 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb9ac0a9480 con 0x5627917f3760 2024-06-04T05:03:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.472+0000 7fb9c3e0b740 10 client.17046 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.472+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/1213605382 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56279180f6d0 con 0x5627917f3760 2024-06-04T05:03:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.472+0000 7fb99eff5640 1 -- 172.21.15.70:0/1213605382 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb9b0019ac0 con 0x5627917f3760 2024-06-04T05:03:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.472+0000 7fb99eff5640 1 -- 172.21.15.70:0/1213605382 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fb9ac0a54a0 con 0x5627917f3760 2024-06-04T05:03:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.473+0000 7fb9c3e0b740 10 client.17046 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.473+0000 7fb9c3e0b740 10 client.17046 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.473+0000 7fb9c3e0b740 20 client.17046 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.473+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/1213605382 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562791810b40 0x562791812f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.473+0000 7fb9c3e0b740 4 client.17046 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:03:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.473+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/1213605382 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562791767b80 con 0x562791810b40 2024-06-04T05:03:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.473+0000 7fb9ba59c640 1 --2- 172.21.15.70:0/1213605382 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562791810b40 0x562791812f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.473+0000 7fb9ba59c640 1 --2- 172.21.15.70:0/1213605382 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562791810b40 0x562791812f30 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7b8476467d0e0c9 server_cookie=c6a9d7275634828a in_seq=0 out_seq=0 2024-06-04T05:03:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.473+0000 7fb99eff5640 10 client.17046 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.474+0000 7fb99eff5640 1 -- 172.21.15.70:0/1213605382 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562791767b80 con 0x562791810b40 2024-06-04T05:03:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.474+0000 7fb99eff5640 10 client.17046 handle_command_reply: tid=0 2024-06-04T05:03:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.474+0000 7fb9c3e0b740 1 client.17046 shutdown 2024-06-04T05:03:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.474+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/1213605382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5627917f3760 msgr2=0x5627917fdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.474+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/1213605382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5627917f3760 0x5627917fdc20 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac009920 tx=0x7fb9ac0e84d0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.474+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/1213605382 shutdown_connections 2024-06-04T05:03:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.474+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/1213605382 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562791810b40 0x562791812f30 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.474+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/1213605382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562791800c10 0x5627917fe6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.474+0000 7fb99eff5640 0 client.17046 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.474+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/1213605382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f4080 0x5627917fe160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.474+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/1213605382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5627917f3760 0x5627917fdc20 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.474+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/1213605382 >> 172.21.15.70:0/1213605382 conn(0x5627917efd30 msgr2=0x5627917f5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.474+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/1213605382 shutdown_connections 2024-06-04T05:03:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.474+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/1213605382 wait complete. 2024-06-04T05:03:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.474+0000 7fb9c3e0b740 20 client.17046 trim_cache size 0 max 16384 2024-06-04T05:03:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.492+0000 7fb9c3e0b740 1 Processor -- start 2024-06-04T05:03:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.492+0000 7fb9c3e0b740 1 -- start start 2024-06-04T05:03:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.492+0000 7fb9c3e0b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f3760 0x5627917fe470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.492+0000 7fb9c3e0b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f4080 0x5627918187a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.492+0000 7fb9c3e0b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x562791818ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.492+0000 7fb9c3e0b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc1ac410 con 0x562791800c10 2024-06-04T05:03:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.492+0000 7fb9c3e0b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc1adf50 con 0x5627917f3760 2024-06-04T05:03:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.492+0000 7fb9c3e0b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc1acae0 con 0x5627917f4080 2024-06-04T05:03:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.492+0000 7fb9bb59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x562791818ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.492+0000 7fb9bb59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x562791818ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42534/0 (socket says 172.21.15.70:42534) 2024-06-04T05:03:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.492+0000 7fb9bad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f3760 0x5627917fe470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.492+0000 7fb9bb59e640 1 -- 172.21.15.70:0/1752863573 learned_addr learned my addr 172.21.15.70:0/1752863573 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.493+0000 7fb9ba59c640 1 --2- 172.21.15.70:0/1752863573 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f4080 0x5627918187a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.493+0000 7fb9bad9d640 1 -- 172.21.15.70:0/1752863573 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f4080 msgr2=0x5627918187a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.493+0000 7fb9bad9d640 1 --2- 172.21.15.70:0/1752863573 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f4080 0x5627918187a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.493+0000 7fb9bad9d640 1 -- 172.21.15.70:0/1752863573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 msgr2=0x562791818ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.493+0000 7fb9bad9d640 1 --2- 172.21.15.70:0/1752863573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x562791818ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.493+0000 7fb9bad9d640 1 -- 172.21.15.70:0/1752863573 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627917fcc10 con 0x5627917f3760 2024-06-04T05:03:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.493+0000 7fb9bb59e640 1 --2- 172.21.15.70:0/1752863573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x562791818ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.493+0000 7fb9bad9d640 1 --2- 172.21.15.70:0/1752863573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f3760 0x5627917fe470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac009710 tx=0x7fb9ac0a5950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9cd76c6792f485e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.493+0000 7fb9a0ff9640 1 -- 172.21.15.70:0/1752863573 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac0aae60 con 0x5627917f3760 2024-06-04T05:03:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.493+0000 7fb9a0ff9640 1 -- 172.21.15.70:0/1752863573 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb9ac0aa230 con 0x5627917f3760 2024-06-04T05:03:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.493+0000 7fb9a0ff9640 1 -- 172.21.15.70:0/1752863573 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac0042f0 con 0x5627917f3760 2024-06-04T05:03:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.494+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/1752863573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f3760 msgr2=0x5627917fe470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.494+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/1752863573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f3760 0x5627917fe470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac009710 tx=0x7fb9ac0a5950 comp rx=0 tx=0).stop 2024-06-04T05:03:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.494+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/1752863573 shutdown_connections 2024-06-04T05:03:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.494+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/1752863573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x562791818ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.494+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/1752863573 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f4080 0x5627918187a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.494+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/1752863573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f3760 0x5627917fe470 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.494+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/1752863573 >> 172.21.15.70:0/1752863573 conn(0x5627917efd30 msgr2=0x5627917f0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.494+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/1752863573 shutdown_connections 2024-06-04T05:03:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.494+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/1752863573 wait complete. 2024-06-04T05:03:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.494+0000 7fb9c3e0b740 1 -- start start 2024-06-04T05:03:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.494+0000 7fb9c3e0b740 1 Processor -- start 2024-06-04T05:03:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.494+0000 7fb9c3e0b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f3760 0x5627917fae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.494+0000 7fb9c3e0b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f4080 0x5627917fb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.494+0000 7fb9c3e0b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x5627917fb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9c3e0b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc1ad880 con 0x562791800c10 2024-06-04T05:03:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9c3e0b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc1ad1b0 con 0x5627917f4080 2024-06-04T05:03:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9c3e0b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc1aa3f0 con 0x5627917f3760 2024-06-04T05:03:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9c3e0b740 10 client.0 _ll_register_callbacks cb 0x7fb9ac00e9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9ba59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f4080 0x5627917fb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9bb59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x5627917fb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9bad9d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f3760 0x5627917fae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9ba59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f4080 0x5627917fb3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:51064/0 (socket says 172.21.15.70:51064) 2024-06-04T05:03:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9bad9d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f3760 0x5627917fae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:33602/0 (socket says 172.21.15.70:33602) 2024-06-04T05:03:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9ba59c640 1 -- 172.21.15.70:0/3379680093 learned_addr learned my addr 172.21.15.70:0/3379680093 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9bb59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x5627917fb8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42544/0 (socket says 172.21.15.70:42544) 2024-06-04T05:03:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9ba59c640 1 -- 172.21.15.70:0/3379680093 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f3760 msgr2=0x5627917fae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9ba59c640 1 --2- 172.21.15.70:0/3379680093 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f3760 0x5627917fae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9ba59c640 1 -- 172.21.15.70:0/3379680093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 msgr2=0x5627917fb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9ba59c640 1 --2- 172.21.15.70:0/3379680093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x5627917fb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9ba59c640 1 -- 172.21.15.70:0/3379680093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56279187a8c0 con 0x5627917f4080 2024-06-04T05:03:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9bad9d640 1 --2- 172.21.15.70:0/3379680093 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f3760 0x5627917fae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9bb59e640 1 --2- 172.21.15.70:0/3379680093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x5627917fb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9ba59c640 1 --2- 172.21.15.70:0/3379680093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f4080 0x5627917fb3a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009820 tx=0x7fb9b0009f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f83b9a01c10d5b57 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb99eff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:03:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb99eff5640 1 -- 172.21.15.70:0/3379680093 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b001e040 con 0x5627917f4080 2024-06-04T05:03:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb99eff5640 1 -- 172.21.15.70:0/3379680093 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb9b00054c0 con 0x5627917f4080 2024-06-04T05:03:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/3379680093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562791761360 con 0x5627917f4080 2024-06-04T05:03:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb99eff5640 1 -- 172.21.15.70:0/3379680093 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb9b003e050 con 0x5627917f4080 2024-06-04T05:03:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9c3e0b740 10 client.26348 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.495+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/3379680093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56279187bb90 con 0x5627917f4080 2024-06-04T05:03:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.496+0000 7fb99eff5640 1 -- 172.21.15.70:0/3379680093 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x562791761360 con 0x5627917f4080 2024-06-04T05:03:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.496+0000 7fb99eff5640 1 -- 172.21.15.70:0/3379680093 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fb9b0003c80 con 0x5627917f4080 2024-06-04T05:03:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.496+0000 7fb9c3e0b740 10 client.26348 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.496+0000 7fb9c3e0b740 10 client.26348 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.496+0000 7fb9c3e0b740 20 client.26348 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.496+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/3379680093 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562791810b40 0x56279187e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.496+0000 7fb9c3e0b740 4 client.26348 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:03:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.496+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/3379680093 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5627916db5f0 con 0x562791810b40 2024-06-04T05:03:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.496+0000 7fb9bad9d640 1 --2- 172.21.15.70:0/3379680093 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562791810b40 0x56279187e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.497+0000 7fb9bad9d640 1 --2- 172.21.15.70:0/3379680093 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562791810b40 0x56279187e180 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56a2da84e0e698e9 server_cookie=14531c7c679a47fd in_seq=0 out_seq=0 2024-06-04T05:03:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.497+0000 7fb99eff5640 10 client.26348 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:18.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.516+0000 7fb99eff5640 1 -- 172.21.15.70:0/3379680093 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5627916db5f0 con 0x562791810b40 2024-06-04T05:03:18.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.516+0000 7fb99eff5640 10 client.26348 handle_command_reply: tid=0 2024-06-04T05:03:18.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.516+0000 7fb9c3e0b740 1 client.26348 shutdown 2024-06-04T05:03:18.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/3379680093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f4080 msgr2=0x5627917fb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:18.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/3379680093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f4080 0x5627917fb3a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009820 tx=0x7fb9b0009f00 comp rx=0 tx=0).stop 2024-06-04T05:03:18.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/3379680093 shutdown_connections 2024-06-04T05:03:18.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/3379680093 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562791810b40 0x56279187e180 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/3379680093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627917f4080 0x5627917fb3a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/3379680093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562791800c10 0x5627917fb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb99eff5640 0 client.26348 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:18.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c3e0b740 1 --2- 172.21.15.70:0/3379680093 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5627917f3760 0x5627917fae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/3379680093 >> 172.21.15.70:0/3379680093 conn(0x5627917efd30 msgr2=0x5627917f5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:18.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/3379680093 shutdown_connections 2024-06-04T05:03:18.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c3e0b740 1 -- 172.21.15.70:0/3379680093 wait complete. 2024-06-04T05:03:18.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c3e0b740 20 client.26348 trim_cache size 0 max 16384 2024-06-04T05:03:18.521 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:03:18.521 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:03:18.521 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:03:18.521 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c10f1640 1 -- 172.21.15.70:0/417813611 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9a40b2a10 msgr2=0x7fb9a40b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:18.521 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c10f1640 1 --2- 172.21.15.70:0/417813611 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9a40b2a10 0x7fb9a40b4ed0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009430 tx=0x7fb9b4009360 comp rx=0 tx=0).stop 2024-06-04T05:03:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c10f1640 1 -- 172.21.15.70:0/417813611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9bc101980 msgr2=0x7fb9bc197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c10f1640 1 --2- 172.21.15.70:0/417813611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9bc101980 0x7fb9bc197f70 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac00ea60 tx=0x7fb9ac00ef20 comp rx=0 tx=0).stop 2024-06-04T05:03:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c10f1640 1 -- 172.21.15.70:0/417813611 shutdown_connections 2024-06-04T05:03:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c10f1640 1 --2- 172.21.15.70:0/417813611 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9a40b2a10 0x7fb9a40b4ed0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.517+0000 7fb9c10f1640 1 --2- 172.21.15.70:0/417813611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9bc102340 0x7fb9bc1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.518+0000 7fb9c10f1640 1 --2- 172.21.15.70:0/417813611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9bc107940 0x7fb9bc19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.518+0000 7fb9c10f1640 1 --2- 172.21.15.70:0/417813611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9bc101980 0x7fb9bc197f70 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.518+0000 7fb9c10f1640 1 -- 172.21.15.70:0/417813611 >> 172.21.15.70:0/417813611 conn(0x7fb9bc078ad0 msgr2=0x7fb9bc1044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.518+0000 7fb9c10f1640 1 -- 172.21.15.70:0/417813611 shutdown_connections 2024-06-04T05:03:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:18.518+0000 7fb9c10f1640 1 -- 172.21.15.70:0/417813611 wait complete. 2024-06-04T05:03:18.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 103215 2.39 MB/sec execute 131 sec latency 17.473 ms 2024-06-04T05:03:18.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.856+0000 7f3401345700 1 -- 172.21.15.70:0/72178585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc108520 msgr2=0x7f33fc108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:18.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.856+0000 7f3401345700 1 --2- 172.21.15.70:0/72178585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc108520 0x7f33fc108900 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f33ec0098f0 tx=0x7f33ec009c00 comp rx=0 tx=0).stop 2024-06-04T05:03:18.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.856+0000 7f3401345700 1 -- 172.21.15.70:0/72178585 shutdown_connections 2024-06-04T05:03:18.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.856+0000 7f3401345700 1 --2- 172.21.15.70:0/72178585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33fc102e00 0x7f33fc10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.856+0000 7f3401345700 1 --2- 172.21.15.70:0/72178585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc108520 0x7f33fc108900 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.856+0000 7f3401345700 1 --2- 172.21.15.70:0/72178585 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc102440 0x7f33fc1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.856+0000 7f3401345700 1 -- 172.21.15.70:0/72178585 >> 172.21.15.70:0/72178585 conn(0x7f33fc0793f0 msgr2=0x7f33fc079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:18.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.856+0000 7f3401345700 1 -- 172.21.15.70:0/72178585 shutdown_connections 2024-06-04T05:03:18.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.857+0000 7f3401345700 1 -- 172.21.15.70:0/72178585 wait complete. 2024-06-04T05:03:18.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.857+0000 7f3401345700 1 Processor -- start 2024-06-04T05:03:18.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f3401345700 1 -- start start 2024-06-04T05:03:18.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f3401345700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc102440 0x7f33fc19a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f3401345700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc102e00 0x7f33fc19af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f3401345700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33fc108520 0x7f33fc1a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f3401345700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc1124a0 con 0x7f33fc102e00 2024-06-04T05:03:18.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f3401345700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc112320 con 0x7f33fc102440 2024-06-04T05:03:18.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f3401345700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc112620 con 0x7f33fc108520 2024-06-04T05:03:18.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f33fa7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc102e00 0x7f33fc19af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f33fa7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc102e00 0x7f33fc19af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42572/0 (socket says 172.21.15.70:42572) 2024-06-04T05:03:18.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f33fa7fc700 1 -- 172.21.15.70:0/994920474 learned_addr learned my addr 172.21.15.70:0/994920474 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:18.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f33fb7fe700 1 --2- 172.21.15.70:0/994920474 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33fc108520 0x7f33fc1a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f33faffd700 1 --2- 172.21.15.70:0/994920474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc102440 0x7f33fc19a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f33fa7fc700 1 -- 172.21.15.70:0/994920474 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33fc108520 msgr2=0x7f33fc1a5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:18.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f33fa7fc700 1 --2- 172.21.15.70:0/994920474 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33fc108520 0x7f33fc1a5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f33fa7fc700 1 -- 172.21.15.70:0/994920474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc102440 msgr2=0x7f33fc19a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:18.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f33fa7fc700 1 --2- 172.21.15.70:0/994920474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc102440 0x7f33fc19a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:18.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f33fa7fc700 1 -- 172.21.15.70:0/994920474 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33fc1987b0 con 0x7f33fc102e00 2024-06-04T05:03:18.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.858+0000 7f33faffd700 1 --2- 172.21.15.70:0/994920474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc102440 0x7f33fc19a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:03:18.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.859+0000 7f33fa7fc700 1 --2- 172.21.15.70:0/994920474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc102e00 0x7f33fc19af00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f33f000ec10 tx=0x7f33f000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:18.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.859+0000 7f33e3fff700 1 -- 172.21.15.70:0/994920474 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f000cd40 con 0x7f33fc102e00 2024-06-04T05:03:18.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.859+0000 7f3401345700 1 -- 172.21.15.70:0/994920474 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33fc198aa0 con 0x7f33fc102e00 2024-06-04T05:03:18.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.859+0000 7f3401345700 1 -- 172.21.15.70:0/994920474 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33fc199030 con 0x7f33fc102e00 2024-06-04T05:03:18.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.859+0000 7f33e3fff700 1 -- 172.21.15.70:0/994920474 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f33f00044b0 con 0x7f33fc102e00 2024-06-04T05:03:18.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.859+0000 7f33e3fff700 1 -- 172.21.15.70:0/994920474 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f0010430 con 0x7f33fc102e00 2024-06-04T05:03:18.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.860+0000 7f3401345700 1 -- 172.21.15.70:0/994920474 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33fc06a230 con 0x7f33fc102e00 2024-06-04T05:03:18.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.861+0000 7f33e3fff700 1 -- 172.21.15.70:0/994920474 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f33f0004000 con 0x7f33fc102e00 2024-06-04T05:03:18.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.862+0000 7f33e3fff700 1 --2- 172.21.15.70:0/994920474 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33e40b2a70 0x7f33e40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:18.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.862+0000 7f33faffd700 1 --2- 172.21.15.70:0/994920474 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33e40b2a70 0x7f33e40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:18.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.863+0000 7f33e3fff700 1 -- 172.21.15.70:0/994920474 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f33f00e29e0 con 0x7f33fc102e00 2024-06-04T05:03:18.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.863+0000 7f33faffd700 1 --2- 172.21.15.70:0/994920474 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33e40b2a70 0x7f33e40b4f30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f33ec009fd0 tx=0x7f33ec019040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:18.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:18.866+0000 7f33e3fff700 1 -- 172.21.15.70:0/994920474 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f33f00aa450 con 0x7f33fc102e00 2024-06-04T05:03:19.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.140+0000 7f3401345700 1 -- 172.21.15.70:0/994920474 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33fc19a4c0 con 0x7f33fc102e00 2024-06-04T05:03:19.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.141+0000 7f33e3fff700 1 -- 172.21.15.70:0/994920474 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11929 (secure 0 0 0) 0x7f33f00a9d50 con 0x7f33fc102e00 2024-06-04T05:03:19.143 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:03:19.143 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:01:52.807483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:03:19.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.143+0000 7f3401345700 1 -- 172.21.15.70:0/994920474 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33e40b2a70 msgr2=0x7f33e40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:19.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.143+0000 7f3401345700 1 --2- 172.21.15.70:0/994920474 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33e40b2a70 0x7f33e40b4f30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f33ec009fd0 tx=0x7f33ec019040 comp rx=0 tx=0).stop 2024-06-04T05:03:19.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.143+0000 7f3401345700 1 -- 172.21.15.70:0/994920474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc102e00 msgr2=0x7f33fc19af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:19.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.143+0000 7f3401345700 1 --2- 172.21.15.70:0/994920474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc102e00 0x7f33fc19af00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f33f000ec10 tx=0x7f33f000ef20 comp rx=0 tx=0).stop 2024-06-04T05:03:19.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.143+0000 7f3401345700 1 -- 172.21.15.70:0/994920474 shutdown_connections 2024-06-04T05:03:19.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.143+0000 7f3401345700 1 --2- 172.21.15.70:0/994920474 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33e40b2a70 0x7f33e40b4f30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:19.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.143+0000 7f3401345700 1 --2- 172.21.15.70:0/994920474 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33fc108520 0x7f33fc1a5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:19.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.143+0000 7f3401345700 1 --2- 172.21.15.70:0/994920474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc102440 0x7f33fc19a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:19.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.143+0000 7f3401345700 1 --2- 172.21.15.70:0/994920474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc102e00 0x7f33fc19af00 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:19.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.143+0000 7f3401345700 1 -- 172.21.15.70:0/994920474 >> 172.21.15.70:0/994920474 conn(0x7f33fc0793f0 msgr2=0x7f33fc1013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:19.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.143+0000 7f3401345700 1 -- 172.21.15.70:0/994920474 shutdown_connections 2024-06-04T05:03:19.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.143+0000 7f3401345700 1 -- 172.21.15.70:0/994920474 wait complete. 2024-06-04T05:03:19.147 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 32 2024-06-04T05:03:19.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.328+0000 7fe208c0c700 1 -- 172.21.15.70:0/163859396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe204109520 msgr2=0x7fe204109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:19.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.328+0000 7fe208c0c700 1 --2- 172.21.15.70:0/163859396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe204109520 0x7fe204109900 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe1f40099c0 tx=0x7fe1f4009cd0 comp rx=0 tx=0).stop 2024-06-04T05:03:19.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.328+0000 7fe208c0c700 1 -- 172.21.15.70:0/163859396 shutdown_connections 2024-06-04T05:03:19.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.328+0000 7fe208c0c700 1 --2- 172.21.15.70:0/163859396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2041024f0 0x7fe2041103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:19.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.328+0000 7fe208c0c700 1 --2- 172.21.15.70:0/163859396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe204109520 0x7fe204109900 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:19.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.328+0000 7fe208c0c700 1 --2- 172.21.15.70:0/163859396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe204101b30 0x7fe204101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:19.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.328+0000 7fe208c0c700 1 -- 172.21.15.70:0/163859396 >> 172.21.15.70:0/163859396 conn(0x7fe204078ab0 msgr2=0x7fe2040ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:19.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.328+0000 7fe208c0c700 1 -- 172.21.15.70:0/163859396 shutdown_connections 2024-06-04T05:03:19.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.328+0000 7fe208c0c700 1 -- 172.21.15.70:0/163859396 wait complete. 2024-06-04T05:03:19.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe208c0c700 1 Processor -- start 2024-06-04T05:03:19.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe208c0c700 1 -- start start 2024-06-04T05:03:19.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe208c0c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe204101b30 0x7fe20419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:19.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe208c0c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe2041024f0 0x7fe20419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:19.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe208c0c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe204109520 0x7fe2041998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:19.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe208c0c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe2041135d0 con 0x7fe2041024f0 2024-06-04T05:03:19.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe208c0c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe204113450 con 0x7fe204109520 2024-06-04T05:03:19.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe208c0c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe204113750 con 0x7fe204101b30 2024-06-04T05:03:19.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe201d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe2041024f0 0x7fe20419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:19.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe201d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe2041024f0 0x7fe20419fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42598/0 (socket says 172.21.15.70:42598) 2024-06-04T05:03:19.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe201d9b700 1 -- 172.21.15.70:0/425315354 learned_addr learned my addr 172.21.15.70:0/425315354 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:19.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe20259c700 1 --2- 172.21.15.70:0/425315354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe204101b30 0x7fe20419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:19.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe202d9d700 1 --2- 172.21.15.70:0/425315354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe204109520 0x7fe2041998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:19.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe201d9b700 1 -- 172.21.15.70:0/425315354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe204101b30 msgr2=0x7fe20419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:19.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe201d9b700 1 --2- 172.21.15.70:0/425315354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe204101b30 0x7fe20419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:19.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe201d9b700 1 -- 172.21.15.70:0/425315354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe204109520 msgr2=0x7fe2041998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:19.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe201d9b700 1 --2- 172.21.15.70:0/425315354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe204109520 0x7fe2041998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:19.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.329+0000 7fe201d9b700 1 -- 172.21.15.70:0/425315354 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20419a050 con 0x7fe2041024f0 2024-06-04T05:03:19.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.330+0000 7fe201d9b700 1 --2- 172.21.15.70:0/425315354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe2041024f0 0x7fe20419fc70 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800e9a0 tx=0x7fe1f800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:19.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.330+0000 7fe1f37fe700 1 -- 172.21.15.70:0/425315354 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f800ca10 con 0x7fe2041024f0 2024-06-04T05:03:19.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.330+0000 7fe1f37fe700 1 -- 172.21.15.70:0/425315354 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1f800cbb0 con 0x7fe2041024f0 2024-06-04T05:03:19.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.330+0000 7fe208c0c700 1 -- 172.21.15.70:0/425315354 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe20419a340 con 0x7fe2041024f0 2024-06-04T05:03:19.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.330+0000 7fe1f37fe700 1 -- 172.21.15.70:0/425315354 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8018680 con 0x7fe2041024f0 2024-06-04T05:03:19.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.330+0000 7fe208c0c700 1 -- 172.21.15.70:0/425315354 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2041a6600 con 0x7fe2041024f0 2024-06-04T05:03:19.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.331+0000 7fe208c0c700 1 -- 172.21.15.70:0/425315354 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe204104e50 con 0x7fe2041024f0 2024-06-04T05:03:19.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.332+0000 7fe1f37fe700 1 -- 172.21.15.70:0/425315354 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fe1f8010430 con 0x7fe2041024f0 2024-06-04T05:03:19.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.333+0000 7fe1f37fe700 1 --2- 172.21.15.70:0/425315354 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1ec0b2a70 0x7fe1ec0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:19.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.334+0000 7fe20259c700 1 --2- 172.21.15.70:0/425315354 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1ec0b2a70 0x7fe1ec0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:19.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.334+0000 7fe1f37fe700 1 -- 172.21.15.70:0/425315354 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fe1f80e2bc0 con 0x7fe2041024f0 2024-06-04T05:03:19.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.334+0000 7fe20259c700 1 --2- 172.21.15.70:0/425315354 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1ec0b2a70 0x7fe1ec0b4f30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4000c00 tx=0x7fe1f4005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:19.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.337+0000 7fe1f37fe700 1 -- 172.21.15.70:0/425315354 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe1f80aa540 con 0x7fe2041024f0 2024-06-04T05:03:19.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:19 smithi070 ceph-mon[32631]: pgmap v187: 97 pgs: 97 active+clean; 68 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-06-04T05:03:19.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:19 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/994920474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:03:19.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.626+0000 7fe208c0c700 1 -- 172.21.15.70:0/425315354 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe20402e560 con 0x7fe2041024f0 2024-06-04T05:03:19.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.627+0000 7fe1f37fe700 1 -- 172.21.15.70:0/425315354 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11929 (secure 0 0 0) 0x7fe1f80a9e40 con 0x7fe2041024f0 2024-06-04T05:03:19.629 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:03:19.630 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:01:52.807483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:03:19.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.630+0000 7fe208c0c700 1 -- 172.21.15.70:0/425315354 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1ec0b2a70 msgr2=0x7fe1ec0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:19.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.630+0000 7fe208c0c700 1 --2- 172.21.15.70:0/425315354 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1ec0b2a70 0x7fe1ec0b4f30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4000c00 tx=0x7fe1f4005fb0 comp rx=0 tx=0).stop 2024-06-04T05:03:19.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.630+0000 7fe208c0c700 1 -- 172.21.15.70:0/425315354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe2041024f0 msgr2=0x7fe20419fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:19.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.630+0000 7fe208c0c700 1 --2- 172.21.15.70:0/425315354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe2041024f0 0x7fe20419fc70 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800e9a0 tx=0x7fe1f800ecb0 comp rx=0 tx=0).stop 2024-06-04T05:03:19.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.630+0000 7fe208c0c700 1 -- 172.21.15.70:0/425315354 shutdown_connections 2024-06-04T05:03:19.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.630+0000 7fe208c0c700 1 --2- 172.21.15.70:0/425315354 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1ec0b2a70 0x7fe1ec0b4f30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:19.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.630+0000 7fe208c0c700 1 --2- 172.21.15.70:0/425315354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe204109520 0x7fe2041998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:19.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.630+0000 7fe208c0c700 1 --2- 172.21.15.70:0/425315354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe204101b30 0x7fe20419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:19.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.630+0000 7fe208c0c700 1 --2- 172.21.15.70:0/425315354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe2041024f0 0x7fe20419fc70 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:19.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.630+0000 7fe208c0c700 1 -- 172.21.15.70:0/425315354 >> 172.21.15.70:0/425315354 conn(0x7fe204078ab0 msgr2=0x7fe2041003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:19.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.630+0000 7fe208c0c700 1 -- 172.21.15.70:0/425315354 shutdown_connections 2024-06-04T05:03:19.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:19.630+0000 7fe208c0c700 1 -- 172.21.15.70:0/425315354 wait complete. 2024-06-04T05:03:19.633 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 32 2024-06-04T05:03:19.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 103476 2.37 MB/sec execute 132 sec latency 39.965 ms 2024-06-04T05:03:19.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:19 smithi112 ceph-mon[36453]: pgmap v187: 97 pgs: 97 active+clean; 68 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-06-04T05:03:19.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:19 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/994920474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:03:19.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:19 smithi098 ceph-mon[35569]: pgmap v187: 97 pgs: 97 active+clean; 68 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-06-04T05:03:19.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:19 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/994920474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:03:20.170 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 138.95 --await-for 45.0' 2024-06-04T05:03:20.171 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 138.95 --await-for 45.0 2024-06-04T05:03:20.475 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:03:20.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 103964 2.37 MB/sec execute 133 sec latency 17.036 ms 2024-06-04T05:03:20.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:20 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/425315354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:03:20.930 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:03:20.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:20 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/425315354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:03:20.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:20 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/425315354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:03:21.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 104471 2.38 MB/sec execute 134 sec latency 23.719 ms 2024-06-04T05:03:21.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:21 smithi070 ceph-mon[32631]: pgmap v188: 97 pgs: 97 active+clean; 68 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-06-04T05:03:21.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:21 smithi112 ceph-mon[36453]: pgmap v188: 97 pgs: 97 active+clean; 68 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-06-04T05:03:21.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:21 smithi098 ceph-mon[35569]: pgmap v188: 97 pgs: 97 active+clean; 68 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-06-04T05:03:22.168 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:03:22.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105068 2.36 MB/sec execute 135 sec latency 19.965 ms 2024-06-04T05:03:23.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.384+0000 7f4651755700 1 -- 172.21.15.70:0/3438033823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f464c102440 msgr2=0x7f464c1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:23.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.384+0000 7f4651755700 1 --2- 172.21.15.70:0/3438033823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f464c102440 0x7f464c1028c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f4640009990 tx=0x7f4640009ca0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.384+0000 7f4651755700 1 -- 172.21.15.70:0/3438033823 shutdown_connections 2024-06-04T05:03:23.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.384+0000 7f4651755700 1 --2- 172.21.15.70:0/3438033823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f464c102e00 0x7f464c10f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.384+0000 7f4651755700 1 --2- 172.21.15.70:0/3438033823 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f464c1084c0 0x7f464c1088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.384+0000 7f4651755700 1 --2- 172.21.15.70:0/3438033823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f464c102440 0x7f464c1028c0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.384+0000 7f4651755700 1 -- 172.21.15.70:0/3438033823 >> 172.21.15.70:0/3438033823 conn(0x7f464c0793e0 msgr2=0x7f464c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:23.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.384+0000 7f4651755700 1 -- 172.21.15.70:0/3438033823 shutdown_connections 2024-06-04T05:03:23.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.384+0000 7f4651755700 1 -- 172.21.15.70:0/3438033823 wait complete. 2024-06-04T05:03:23.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.385+0000 7f4651755700 1 Processor -- start 2024-06-04T05:03:23.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.385+0000 7f4651755700 1 -- start start 2024-06-04T05:03:23.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.385+0000 7f4651755700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f464c102440 0x7f464c104090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.385+0000 7f4651755700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f464c102e00 0x7f464c1045d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.385+0000 7f4651755700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f464c1084c0 0x7f464c104b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.385+0000 7f4651755700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f464c112490 con 0x7f464c102440 2024-06-04T05:03:23.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.385+0000 7f4651755700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f464c112310 con 0x7f464c102e00 2024-06-04T05:03:23.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.385+0000 7f4651755700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f464c112610 con 0x7f464c1084c0 2024-06-04T05:03:23.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.385+0000 7f464affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f464c102440 0x7f464c104090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.385+0000 7f464a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f464c102e00 0x7f464c1045d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.385+0000 7f464a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f464c102e00 0x7f464c1045d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:51124/0 (socket says 172.21.15.70:51124) 2024-06-04T05:03:23.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.385+0000 7f464a7fc700 1 -- 172.21.15.70:0/2172619752 learned_addr learned my addr 172.21.15.70:0/2172619752 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:23.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.385+0000 7f464b7fe700 1 --2- 172.21.15.70:0/2172619752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f464c1084c0 0x7f464c104b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.386+0000 7f464affd700 1 -- 172.21.15.70:0/2172619752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f464c1084c0 msgr2=0x7f464c104b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:23.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.386+0000 7f464affd700 1 --2- 172.21.15.70:0/2172619752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f464c1084c0 0x7f464c104b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.386+0000 7f464affd700 1 -- 172.21.15.70:0/2172619752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f464c102e00 msgr2=0x7f464c1045d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:23.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.386+0000 7f464affd700 1 --2- 172.21.15.70:0/2172619752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f464c102e00 0x7f464c1045d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.386+0000 7f464affd700 1 -- 172.21.15.70:0/2172619752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f464c1a8030 con 0x7f464c102440 2024-06-04T05:03:23.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.386+0000 7f464a7fc700 1 --2- 172.21.15.70:0/2172619752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f464c102e00 0x7f464c1045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:23.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.386+0000 7f464b7fe700 1 --2- 172.21.15.70:0/2172619752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f464c1084c0 0x7f464c104b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:23.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.386+0000 7f464affd700 1 --2- 172.21.15.70:0/2172619752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f464c102440 0x7f464c104090 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f463c00e9f0 tx=0x7f463c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:23.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.386+0000 7f4633fff700 1 -- 172.21.15.70:0/2172619752 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c00c9c0 con 0x7f464c102440 2024-06-04T05:03:23.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.386+0000 7f4651755700 1 -- 172.21.15.70:0/2172619752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f464c1a8320 con 0x7f464c102440 2024-06-04T05:03:23.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.386+0000 7f4633fff700 1 -- 172.21.15.70:0/2172619752 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f463c010430 con 0x7f464c102440 2024-06-04T05:03:23.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.386+0000 7f4633fff700 1 -- 172.21.15.70:0/2172619752 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c018680 con 0x7f464c102440 2024-06-04T05:03:23.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.386+0000 7f4651755700 1 -- 172.21.15.70:0/2172619752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f464c1a88b0 con 0x7f464c102440 2024-06-04T05:03:23.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.388+0000 7f4633fff700 1 -- 172.21.15.70:0/2172619752 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f463c018820 con 0x7f464c102440 2024-06-04T05:03:23.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.389+0000 7f4633fff700 1 --2- 172.21.15.70:0/2172619752 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f46340b2aa0 0x7f46340b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.389+0000 7f464a7fc700 1 --2- 172.21.15.70:0/2172619752 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f46340b2aa0 0x7f46340b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.389+0000 7f4633fff700 1 -- 172.21.15.70:0/2172619752 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f463c0e2770 con 0x7f464c102440 2024-06-04T05:03:23.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.390+0000 7f464a7fc700 1 --2- 172.21.15.70:0/2172619752 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f46340b2aa0 0x7f46340b4f60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f464c079c00 tx=0x7f4640005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:23.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.390+0000 7f465bfeab80 1 Processor -- start 2024-06-04T05:03:23.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.390+0000 7f465bfeab80 1 -- start start 2024-06-04T05:03:23.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.390+0000 7f465bfeab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b4666917c0 0x55b466691ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.390+0000 7f465bfeab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668cc90 0x55b46668d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.391+0000 7f465bfeab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46668d650 0x55b4666996c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.391+0000 7f465bfeab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b4665eaf10 con 0x55b46668cc90 2024-06-04T05:03:23.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.391+0000 7f465bfeab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b4665daab0 con 0x55b46668d650 2024-06-04T05:03:23.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.391+0000 7f465bfeab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b4665deaf0 con 0x55b4666917c0 2024-06-04T05:03:23.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.391+0000 7f464a7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668cc90 0x55b46668d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.391+0000 7f464affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b4666917c0 0x55b466691ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.391+0000 7f464a7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668cc90 0x55b46668d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42634/0 (socket says 172.21.15.70:42634) 2024-06-04T05:03:23.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.391+0000 7f464affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b4666917c0 0x55b466691ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:33678/0 (socket says 172.21.15.70:33678) 2024-06-04T05:03:23.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.391+0000 7f464a7fc700 1 -- 172.21.15.70:0/3746290515 learned_addr learned my addr 172.21.15.70:0/3746290515 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.391+0000 7f464b7fe700 1 --2- 172.21.15.70:0/3746290515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46668d650 0x55b4666996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.391+0000 7f464a7fc700 1 -- 172.21.15.70:0/3746290515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b4666917c0 msgr2=0x55b466691ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.391+0000 7f464a7fc700 1 --2- 172.21.15.70:0/3746290515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b4666917c0 0x55b466691ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.391+0000 7f464a7fc700 1 -- 172.21.15.70:0/3746290515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46668d650 msgr2=0x55b4666996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.391+0000 7f464a7fc700 1 --2- 172.21.15.70:0/3746290515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46668d650 0x55b4666996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.391+0000 7f464a7fc700 1 -- 172.21.15.70:0/3746290515 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b466584150 con 0x55b46668cc90 2024-06-04T05:03:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.391+0000 7f464a7fc700 1 --2- 172.21.15.70:0/3746290515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668cc90 0x55b46668d110 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f464000b650 tx=0x7f464001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8274b9f9711e8d0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.391+0000 7f46317fa700 1 -- 172.21.15.70:0/3746290515 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46400055e0 con 0x55b46668cc90 2024-06-04T05:03:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.392+0000 7f46317fa700 1 -- 172.21.15.70:0/3746290515 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f464000f460 con 0x55b46668cc90 2024-06-04T05:03:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.392+0000 7f46317fa700 1 -- 172.21.15.70:0/3746290515 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4640022670 con 0x55b46668cc90 2024-06-04T05:03:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.392+0000 7f465bfeab80 1 -- 172.21.15.70:0/3746290515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668cc90 msgr2=0x55b46668d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.392+0000 7f465bfeab80 1 --2- 172.21.15.70:0/3746290515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668cc90 0x55b46668d110 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f464000b650 tx=0x7f464001f000 comp rx=0 tx=0).stop 2024-06-04T05:03:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.392+0000 7f465bfeab80 1 -- 172.21.15.70:0/3746290515 shutdown_connections 2024-06-04T05:03:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.392+0000 7f465bfeab80 1 --2- 172.21.15.70:0/3746290515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46668d650 0x55b4666996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.392+0000 7f465bfeab80 1 --2- 172.21.15.70:0/3746290515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b4666917c0 0x55b466691ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.392+0000 7f465bfeab80 1 --2- 172.21.15.70:0/3746290515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668cc90 0x55b46668d110 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.392+0000 7f465bfeab80 1 -- 172.21.15.70:0/3746290515 >> 172.21.15.70:0/3746290515 conn(0x55b4665b83e0 msgr2=0x55b466687980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.393+0000 7f465bfeab80 1 -- 172.21.15.70:0/3746290515 shutdown_connections 2024-06-04T05:03:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.393+0000 7f465bfeab80 1 -- 172.21.15.70:0/3746290515 wait complete. 2024-06-04T05:03:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.393+0000 7f465bfeab80 1 -- start start 2024-06-04T05:03:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.393+0000 7f465bfeab80 1 Processor -- start 2024-06-04T05:03:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.393+0000 7f465bfeab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46668cc90 0x55b466699370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.393+0000 7f465bfeab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668d650 0x55b466696670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.393+0000 7f465bfeab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b4666917c0 0x55b466696bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.393+0000 7f465bfeab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b4664e1140 con 0x55b46668d650 2024-06-04T05:03:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.393+0000 7f465bfeab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b466542f40 con 0x55b4666917c0 2024-06-04T05:03:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.393+0000 7f465bfeab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b466699c60 con 0x55b46668cc90 2024-06-04T05:03:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.393+0000 7f465bfeab80 10 client.0 _ll_register_callbacks cb 0x55b466575af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.393+0000 7f464a7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668d650 0x55b466696670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.393+0000 7f464a7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668d650 0x55b466696670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42642/0 (socket says 172.21.15.70:42642) 2024-06-04T05:03:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.393+0000 7f464a7fc700 1 -- 172.21.15.70:0/2767445282 learned_addr learned my addr 172.21.15.70:0/2767445282 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.393+0000 7f464affd700 1 --2- 172.21.15.70:0/2767445282 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46668cc90 0x55b466699370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.393+0000 7f464b7fe700 1 --2- 172.21.15.70:0/2767445282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b4666917c0 0x55b466696bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.394+0000 7f464a7fc700 1 -- 172.21.15.70:0/2767445282 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46668cc90 msgr2=0x55b466699370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.394+0000 7f464a7fc700 1 --2- 172.21.15.70:0/2767445282 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46668cc90 0x55b466699370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.394+0000 7f464a7fc700 1 -- 172.21.15.70:0/2767445282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b4666917c0 msgr2=0x55b466696bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.394+0000 7f464a7fc700 1 --2- 172.21.15.70:0/2767445282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b4666917c0 0x55b466696bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.394+0000 7f464a7fc700 1 -- 172.21.15.70:0/2767445282 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b46652aa70 con 0x55b46668d650 2024-06-04T05:03:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.394+0000 7f464a7fc700 1 --2- 172.21.15.70:0/2767445282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668d650 0x55b466696670 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4640005040 tx=0x7f4640005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=188983b6f62712b8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.394+0000 7f462eff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.394+0000 7f462eff5700 1 -- 172.21.15.70:0/2767445282 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464003a060 con 0x55b46668d650 2024-06-04T05:03:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.394+0000 7f465bfeab80 1 -- 172.21.15.70:0/2767445282 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f464000f460 con 0x55b46668d650 2024-06-04T05:03:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.394+0000 7f462eff5700 1 -- 172.21.15.70:0/2767445282 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f464000f820 con 0x55b46668d650 2024-06-04T05:03:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.394+0000 7f462eff5700 1 -- 172.21.15.70:0/2767445282 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f464000fa60 con 0x55b46668d650 2024-06-04T05:03:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.394+0000 7f465bfeab80 10 client.17076 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.394+0000 7f465bfeab80 1 -- 172.21.15.70:0/2767445282 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b466594900 con 0x55b46668d650 2024-06-04T05:03:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.395+0000 7f462eff5700 1 -- 172.21.15.70:0/2767445282 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f464000f460 con 0x55b46668d650 2024-06-04T05:03:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.395+0000 7f462eff5700 1 -- 172.21.15.70:0/2767445282 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f4640035400 con 0x55b46668d650 2024-06-04T05:03:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.395+0000 7f465bfeab80 10 client.17076 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.395+0000 7f465bfeab80 10 client.17076 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:03:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.395+0000 7f465bfeab80 20 client.17076 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.395+0000 7f465bfeab80 1 --2- 172.21.15.70:0/2767445282 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b4666a6690 0x55b4666a8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.395+0000 7f465bfeab80 4 client.17076 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:03:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.395+0000 7f465bfeab80 1 -- 172.21.15.70:0/2767445282 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4665ece60 con 0x55b4666a6690 2024-06-04T05:03:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.395+0000 7f464affd700 1 --2- 172.21.15.70:0/2767445282 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b4666a6690 0x55b4666a8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.395+0000 7f462eff5700 10 client.17076 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.395+0000 7f464affd700 1 --2- 172.21.15.70:0/2767445282 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b4666a6690 0x55b4666a8a80 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b20fb3e181bb42d server_cookie=9e4da192fc215b6a in_seq=0 out_seq=0 2024-06-04T05:03:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.396+0000 7f462eff5700 1 -- 172.21.15.70:0/2767445282 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b4665ece60 con 0x55b4666a6690 2024-06-04T05:03:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.396+0000 7f462eff5700 10 client.17076 handle_command_reply: tid=0 2024-06-04T05:03:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.396+0000 7f465bfeab80 1 client.17076 shutdown 2024-06-04T05:03:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.396+0000 7f465bfeab80 1 -- 172.21.15.70:0/2767445282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668d650 msgr2=0x55b466696670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.396+0000 7f465bfeab80 1 --2- 172.21.15.70:0/2767445282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668d650 0x55b466696670 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4640005040 tx=0x7f4640005810 comp rx=0 tx=0).stop 2024-06-04T05:03:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.396+0000 7f465bfeab80 1 -- 172.21.15.70:0/2767445282 shutdown_connections 2024-06-04T05:03:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.396+0000 7f465bfeab80 1 --2- 172.21.15.70:0/2767445282 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b4666a6690 0x55b4666a8a80 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.396+0000 7f465bfeab80 1 --2- 172.21.15.70:0/2767445282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b4666917c0 0x55b466696bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.396+0000 7f465bfeab80 1 --2- 172.21.15.70:0/2767445282 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46668cc90 0x55b466699370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.396+0000 7f465bfeab80 1 --2- 172.21.15.70:0/2767445282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668d650 0x55b466696670 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.396+0000 7f465bfeab80 1 -- 172.21.15.70:0/2767445282 >> 172.21.15.70:0/2767445282 conn(0x55b4665b83e0 msgr2=0x55b46668bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.396+0000 7f462eff5700 0 client.17076 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.396+0000 7f465bfeab80 1 -- 172.21.15.70:0/2767445282 shutdown_connections 2024-06-04T05:03:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.396+0000 7f465bfeab80 1 -- 172.21.15.70:0/2767445282 wait complete. 2024-06-04T05:03:23.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.397+0000 7f465bfeab80 20 client.17076 trim_cache size 0 max 16384 2024-06-04T05:03:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.419+0000 7f465bfeab80 1 Processor -- start 2024-06-04T05:03:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.419+0000 7f465bfeab80 1 -- start start 2024-06-04T05:03:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.419+0000 7f465bfeab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668cc90 0x55b466681f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.419+0000 7f465bfeab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46668d650 0x55b466690810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.419+0000 7f465bfeab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b4666917c0 0x55b466690d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.419+0000 7f465bfeab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f464c1afe00 con 0x55b46668cc90 2024-06-04T05:03:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.419+0000 7f465bfeab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f464c1b1940 con 0x55b4666917c0 2024-06-04T05:03:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.419+0000 7f465bfeab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f464c1b04d0 con 0x55b46668d650 2024-06-04T05:03:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.419+0000 7f464affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668cc90 0x55b466681f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.419+0000 7f464affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668cc90 0x55b466681f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42656/0 (socket says 172.21.15.70:42656) 2024-06-04T05:03:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.419+0000 7f464a7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46668d650 0x55b466690810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.419+0000 7f464affd700 1 -- 172.21.15.70:0/3497509465 learned_addr learned my addr 172.21.15.70:0/3497509465 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.420+0000 7f464b7fe700 1 --2- 172.21.15.70:0/3497509465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b4666917c0 0x55b466690d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.420+0000 7f464a7fc700 1 -- 172.21.15.70:0/3497509465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b4666917c0 msgr2=0x55b466690d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.420+0000 7f464a7fc700 1 --2- 172.21.15.70:0/3497509465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b4666917c0 0x55b466690d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.420+0000 7f464a7fc700 1 -- 172.21.15.70:0/3497509465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668cc90 msgr2=0x55b466681f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.420+0000 7f464a7fc700 1 --2- 172.21.15.70:0/3497509465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668cc90 0x55b466681f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.420+0000 7f464a7fc700 1 -- 172.21.15.70:0/3497509465 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b466694c00 con 0x55b46668d650 2024-06-04T05:03:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.420+0000 7f464b7fe700 1 --2- 172.21.15.70:0/3497509465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b4666917c0 0x55b466690d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:03:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.420+0000 7f464affd700 1 --2- 172.21.15.70:0/3497509465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668cc90 0x55b466681f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.420+0000 7f464a7fc700 1 --2- 172.21.15.70:0/3497509465 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46668d650 0x55b466690810 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f46400098e0 tx=0x7f4640005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33458f8d6a919f4b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.420+0000 7f4630ff9700 1 -- 172.21.15.70:0/3497509465 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464003a070 con 0x55b46668d650 2024-06-04T05:03:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.420+0000 7f4630ff9700 1 -- 172.21.15.70:0/3497509465 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f464000fc00 con 0x55b46668d650 2024-06-04T05:03:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.420+0000 7f4630ff9700 1 -- 172.21.15.70:0/3497509465 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464003e8f0 con 0x55b46668d650 2024-06-04T05:03:23.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.421+0000 7f465bfeab80 1 -- 172.21.15.70:0/3497509465 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46668d650 msgr2=0x55b466690810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:23.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.421+0000 7f465bfeab80 1 --2- 172.21.15.70:0/3497509465 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46668d650 0x55b466690810 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f46400098e0 tx=0x7f4640005be0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.421+0000 7f465bfeab80 1 -- 172.21.15.70:0/3497509465 shutdown_connections 2024-06-04T05:03:23.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.421+0000 7f465bfeab80 1 --2- 172.21.15.70:0/3497509465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b4666917c0 0x55b466690d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.421+0000 7f465bfeab80 1 --2- 172.21.15.70:0/3497509465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668cc90 0x55b466681f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.421+0000 7f465bfeab80 1 --2- 172.21.15.70:0/3497509465 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46668d650 0x55b466690810 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.421+0000 7f465bfeab80 1 -- 172.21.15.70:0/3497509465 >> 172.21.15.70:0/3497509465 conn(0x55b4665b83e0 msgr2=0x55b466694f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.421+0000 7f465bfeab80 1 -- 172.21.15.70:0/3497509465 shutdown_connections 2024-06-04T05:03:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.421+0000 7f465bfeab80 1 -- 172.21.15.70:0/3497509465 wait complete. 2024-06-04T05:03:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.421+0000 7f465bfeab80 1 -- start start 2024-06-04T05:03:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.421+0000 7f465bfeab80 1 Processor -- start 2024-06-04T05:03:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f465bfeab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46668cc90 0x55b4666a77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f465bfeab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668d650 0x55b4666a7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f465bfeab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b4666917c0 0x55b46670fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f465bfeab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f464c1b1270 con 0x55b46668d650 2024-06-04T05:03:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f465bfeab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f464c1b0ba0 con 0x55b46668cc90 2024-06-04T05:03:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f465bfeab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f464c1adda0 con 0x55b4666917c0 2024-06-04T05:03:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f465bfeab80 10 client.0 _ll_register_callbacks cb 0x55b466575af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f464a7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668d650 0x55b4666a7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f464a7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668d650 0x55b4666a7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42672/0 (socket says 172.21.15.70:42672) 2024-06-04T05:03:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f464a7fc700 1 -- 172.21.15.70:0/1354040722 learned_addr learned my addr 172.21.15.70:0/1354040722 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f464b7fe700 1 --2- 172.21.15.70:0/1354040722 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b4666917c0 0x55b46670fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f464affd700 1 --2- 172.21.15.70:0/1354040722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46668cc90 0x55b4666a77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f464a7fc700 1 -- 172.21.15.70:0/1354040722 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b4666917c0 msgr2=0x55b46670fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f464a7fc700 1 --2- 172.21.15.70:0/1354040722 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b4666917c0 0x55b46670fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f464a7fc700 1 -- 172.21.15.70:0/1354040722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46668cc90 msgr2=0x55b4666a77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f464a7fc700 1 --2- 172.21.15.70:0/1354040722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46668cc90 0x55b4666a77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f464a7fc700 1 -- 172.21.15.70:0/1354040722 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b466710400 con 0x55b46668d650 2024-06-04T05:03:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f464a7fc700 1 --2- 172.21.15.70:0/1354040722 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668d650 0x55b4666a7d20 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f4640003fd0 tx=0x7f46400059d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f465b1dc2f6d0c11 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.422+0000 7f462eff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.423+0000 7f462eff5700 1 -- 172.21.15.70:0/1354040722 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4640026070 con 0x55b46668d650 2024-06-04T05:03:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.423+0000 7f462eff5700 1 -- 172.21.15.70:0/1354040722 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f4640035d00 con 0x55b46668d650 2024-06-04T05:03:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.423+0000 7f465bfeab80 1 -- 172.21.15.70:0/1354040722 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f464000fc00 con 0x55b46668d650 2024-06-04T05:03:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.423+0000 7f462eff5700 1 -- 172.21.15.70:0/1354040722 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f464003a070 con 0x55b46668d650 2024-06-04T05:03:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.423+0000 7f465bfeab80 10 client.17082 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.423+0000 7f465bfeab80 1 -- 172.21.15.70:0/1354040722 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b466711590 con 0x55b46668d650 2024-06-04T05:03:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.423+0000 7f462eff5700 1 -- 172.21.15.70:0/1354040722 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f464000fc00 con 0x55b46668d650 2024-06-04T05:03:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.423+0000 7f462eff5700 1 -- 172.21.15.70:0/1354040722 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f4640022410 con 0x55b46668d650 2024-06-04T05:03:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.423+0000 7f465bfeab80 10 client.17082 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.423+0000 7f465bfeab80 10 client.17082 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:03:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.423+0000 7f465bfeab80 20 client.17082 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.423+0000 7f465bfeab80 1 --2- 172.21.15.70:0/1354040722 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b466711a10 0x55b466713e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.423+0000 7f465bfeab80 4 client.17082 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.95, "await_for": 45.0}] 2024-06-04T05:03:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.423+0000 7f465bfeab80 1 -- 172.21.15.70:0/1354040722 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.95, "await_for": 45.0}) -- 0x55b4665e6640 con 0x55b466711a10 2024-06-04T05:03:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.424+0000 7f464affd700 1 --2- 172.21.15.70:0/1354040722 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b466711a10 0x55b466713e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.424+0000 7f464affd700 1 --2- 172.21.15.70:0/1354040722 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b466711a10 0x55b466713e00 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5671c56e34ebfcb6 server_cookie=c291f87ff05e6b6c in_seq=0 out_seq=0 2024-06-04T05:03:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.424+0000 7f462eff5700 10 client.17082 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:23.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.503+0000 7f462eff5700 1 -- 172.21.15.70:0/1354040722 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55b4665e6640 con 0x55b466711a10 2024-06-04T05:03:23.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.503+0000 7f462eff5700 10 client.17082 handle_command_reply: tid=0 2024-06-04T05:03:23.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.503+0000 7f465bfeab80 1 client.17082 shutdown 2024-06-04T05:03:23.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.503+0000 7f465bfeab80 1 -- 172.21.15.70:0/1354040722 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668d650 msgr2=0x55b4666a7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:23.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.503+0000 7f465bfeab80 1 --2- 172.21.15.70:0/1354040722 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668d650 0x55b4666a7d20 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f4640003fd0 tx=0x7f46400059d0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.503+0000 7f465bfeab80 1 -- 172.21.15.70:0/1354040722 shutdown_connections 2024-06-04T05:03:23.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.503+0000 7f465bfeab80 1 --2- 172.21.15.70:0/1354040722 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b4666917c0 0x55b46670fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.503+0000 7f465bfeab80 1 --2- 172.21.15.70:0/1354040722 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b466711a10 0x55b466713e00 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.503+0000 7f465bfeab80 1 --2- 172.21.15.70:0/1354040722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46668cc90 0x55b4666a77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.503+0000 7f465bfeab80 1 --2- 172.21.15.70:0/1354040722 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46668d650 0x55b4666a7d20 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.503+0000 7f465bfeab80 1 -- 172.21.15.70:0/1354040722 >> 172.21.15.70:0/1354040722 conn(0x55b4665b83e0 msgr2=0x55b466690810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:23.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.503+0000 7f462eff5700 0 client.17082 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:23.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.503+0000 7f465bfeab80 1 -- 172.21.15.70:0/1354040722 shutdown_connections 2024-06-04T05:03:23.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.503+0000 7f465bfeab80 1 -- 172.21.15.70:0/1354040722 wait complete. 2024-06-04T05:03:23.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.503+0000 7f465bfeab80 20 client.17082 trim_cache size 0 max 16384 2024-06-04T05:03:23.507 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:03:23.507 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 32, 2024-06-04T05:03:23.507 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:03:23.507 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 14, 2024-06-04T05:03:23.507 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:03:23.507 INFO:teuthology.orchestra.run.smithi070.stdout: "70640af4": { 2024-06-04T05:03:23.507 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 14, 2024-06-04T05:03:23.507 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:03:23.508 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:03:23.508 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:03:23.508 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:03:23.508 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:03:23.508 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:03:23.508 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 138.9, 2024-06-04T05:03:23.508 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:03:23.508 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:03:23.508 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:03:23.508 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:03:23.508 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:03:23.508 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:03:23.508 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:03:23.508 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:03:23.509 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:03:23.509 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:03:23.509 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:03:23.509 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:03:23.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.504+0000 7f4651755700 1 -- 172.21.15.70:0/2172619752 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f46340b2aa0 msgr2=0x7f46340b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:23.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.504+0000 7f4651755700 1 --2- 172.21.15.70:0/2172619752 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f46340b2aa0 0x7f46340b4f60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f464c079c00 tx=0x7f4640005960 comp rx=0 tx=0).stop 2024-06-04T05:03:23.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.504+0000 7f4651755700 1 -- 172.21.15.70:0/2172619752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f464c102440 msgr2=0x7f464c104090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:23.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.504+0000 7f4651755700 1 --2- 172.21.15.70:0/2172619752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f464c102440 0x7f464c104090 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f463c00e9f0 tx=0x7f463c00ed00 comp rx=0 tx=0).stop 2024-06-04T05:03:23.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.504+0000 7f4651755700 1 -- 172.21.15.70:0/2172619752 shutdown_connections 2024-06-04T05:03:23.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.504+0000 7f4651755700 1 --2- 172.21.15.70:0/2172619752 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f46340b2aa0 0x7f46340b4f60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.504+0000 7f4651755700 1 --2- 172.21.15.70:0/2172619752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f464c1084c0 0x7f464c104b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.504+0000 7f4651755700 1 --2- 172.21.15.70:0/2172619752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f464c102440 0x7f464c104090 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.504+0000 7f4651755700 1 --2- 172.21.15.70:0/2172619752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f464c102e00 0x7f464c1045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:23.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.504+0000 7f4651755700 1 -- 172.21.15.70:0/2172619752 >> 172.21.15.70:0/2172619752 conn(0x7f464c0793e0 msgr2=0x7f464c06b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:23.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.504+0000 7f4651755700 1 -- 172.21.15.70:0/2172619752 shutdown_connections 2024-06-04T05:03:23.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:23.504+0000 7f4651755700 1 -- 172.21.15.70:0/2172619752 wait complete. 2024-06-04T05:03:23.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.36 MB/sec execute 136 sec latency 217.756 ms 2024-06-04T05:03:23.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:23 smithi070 ceph-mon[32631]: pgmap v189: 97 pgs: 97 active+clean; 70 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 1.8 MiB/s wr, 183 op/s 2024-06-04T05:03:23.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:23 smithi112 ceph-mon[36453]: pgmap v189: 97 pgs: 97 active+clean; 70 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 1.8 MiB/s wr, 183 op/s 2024-06-04T05:03:23.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:23 smithi098 ceph-mon[35569]: pgmap v189: 97 pgs: 97 active+clean; 70 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 1.8 MiB/s wr, 183 op/s 2024-06-04T05:03:24.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.34 MB/sec execute 137 sec latency 1217.781 ms 2024-06-04T05:03:25.032 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '70640af4', quiesce took 9.3 seconds. Will release after: 9.649999999999999 2024-06-04T05:03:25.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.242+0000 7f58d192c700 1 -- 172.21.15.70:0/939512356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58cc102e00 msgr2=0x7f58cc10f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:25.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.242+0000 7f58d192c700 1 --2- 172.21.15.70:0/939512356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58cc102e00 0x7f58cc10f2f0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f58c400b0a0 tx=0x7f58c400b3b0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.242+0000 7f58d192c700 1 -- 172.21.15.70:0/939512356 shutdown_connections 2024-06-04T05:03:25.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.242+0000 7f58d192c700 1 --2- 172.21.15.70:0/939512356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58cc102e00 0x7f58cc10f2f0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.242+0000 7f58d192c700 1 --2- 172.21.15.70:0/939512356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58cc108520 0x7f58cc108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.242+0000 7f58d192c700 1 --2- 172.21.15.70:0/939512356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58cc102440 0x7f58cc1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.242+0000 7f58d192c700 1 -- 172.21.15.70:0/939512356 >> 172.21.15.70:0/939512356 conn(0x7f58cc0793f0 msgr2=0x7f58cc079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:25.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.242+0000 7f58d192c700 1 -- 172.21.15.70:0/939512356 shutdown_connections 2024-06-04T05:03:25.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.242+0000 7f58d192c700 1 -- 172.21.15.70:0/939512356 wait complete. 2024-06-04T05:03:25.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58d192c700 1 Processor -- start 2024-06-04T05:03:25.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58d192c700 1 -- start start 2024-06-04T05:03:25.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58d192c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58cc102440 0x7f58cc19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58d192c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58cc102e00 0x7f58cc19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58d192c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58cc108520 0x7f58cc198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58d192c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc112500 con 0x7f58cc102440 2024-06-04T05:03:25.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58d192c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc112380 con 0x7f58cc102e00 2024-06-04T05:03:25.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58d192c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc112680 con 0x7f58cc108520 2024-06-04T05:03:25.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58caffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58cc102440 0x7f58cc19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58caffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58cc102440 0x7f58cc19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42686/0 (socket says 172.21.15.70:42686) 2024-06-04T05:03:25.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58caffd700 1 -- 172.21.15.70:0/1944635645 learned_addr learned my addr 172.21.15.70:0/1944635645 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:25.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58cb7fe700 1 --2- 172.21.15.70:0/1944635645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58cc108520 0x7f58cc198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58c3fff700 1 --2- 172.21.15.70:0/1944635645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58cc102e00 0x7f58cc19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58caffd700 1 -- 172.21.15.70:0/1944635645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58cc108520 msgr2=0x7f58cc198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:25.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58caffd700 1 --2- 172.21.15.70:0/1944635645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58cc108520 0x7f58cc198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58caffd700 1 -- 172.21.15.70:0/1944635645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58cc102e00 msgr2=0x7f58cc19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:25.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58caffd700 1 --2- 172.21.15.70:0/1944635645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58cc102e00 0x7f58cc19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.243+0000 7f58caffd700 1 -- 172.21.15.70:0/1944635645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58cc199000 con 0x7f58cc102440 2024-06-04T05:03:25.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.244+0000 7f58caffd700 1 --2- 172.21.15.70:0/1944635645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58cc102440 0x7f58cc19e650 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f58bc00b530 tx=0x7f58bc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:25.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.244+0000 7f58d092a700 1 -- 172.21.15.70:0/1944635645 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58bc015020 con 0x7f58cc102440 2024-06-04T05:03:25.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.244+0000 7f58d092a700 1 -- 172.21.15.70:0/1944635645 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f58bc011cc0 con 0x7f58cc102440 2024-06-04T05:03:25.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.244+0000 7f58d192c700 1 -- 172.21.15.70:0/1944635645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58cc199290 con 0x7f58cc102440 2024-06-04T05:03:25.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.244+0000 7f58d092a700 1 -- 172.21.15.70:0/1944635645 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58bc010c80 con 0x7f58cc102440 2024-06-04T05:03:25.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.244+0000 7f58d192c700 1 -- 172.21.15.70:0/1944635645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58cc1a54b0 con 0x7f58cc102440 2024-06-04T05:03:25.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.246+0000 7f58d092a700 1 -- 172.21.15.70:0/1944635645 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f58bc018020 con 0x7f58cc102440 2024-06-04T05:03:25.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.247+0000 7f58d092a700 1 --2- 172.21.15.70:0/1944635645 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f58b80b2a70 0x7f58b80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.247+0000 7f58c3fff700 1 --2- 172.21.15.70:0/1944635645 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f58b80b2a70 0x7f58b80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.247+0000 7f58d092a700 1 -- 172.21.15.70:0/1944635645 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f58bc0e1b40 con 0x7f58cc102440 2024-06-04T05:03:25.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.247+0000 7f58dc1c1b80 1 Processor -- start 2024-06-04T05:03:25.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.247+0000 7f58c3fff700 1 --2- 172.21.15.70:0/1944635645 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f58b80b2a70 0x7f58b80b4f30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f58cc0799d0 tx=0x7f58b400a380 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:25.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.247+0000 7f58dc1c1b80 1 -- start start 2024-06-04T05:03:25.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.247+0000 7f58dc1c1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d0588e320 0x556d0588e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.247+0000 7f58dc1c1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d058888a0 0x556d05888d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.247+0000 7f58dc1c1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d05889260 0x556d05895080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.247+0000 7f58dc1c1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556d057e6d70 con 0x556d05889260 2024-06-04T05:03:25.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.247+0000 7f58dc1c1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556d057d6910 con 0x556d058888a0 2024-06-04T05:03:25.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.247+0000 7f58dc1c1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556d057da950 con 0x556d0588e320 2024-06-04T05:03:25.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.248+0000 7f58cb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d05889260 0x556d05895080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.248+0000 7f58cb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d05889260 0x556d05895080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42702/0 (socket says 172.21.15.70:42702) 2024-06-04T05:03:25.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.248+0000 7f58cb7fe700 1 -- 172.21.15.70:0/2584986546 learned_addr learned my addr 172.21.15.70:0/2584986546 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:25.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.248+0000 7f58caffd700 1 --2- 172.21.15.70:0/2584986546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d0588e320 0x556d0588e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.248+0000 7f58c3fff700 1 --2- 172.21.15.70:0/2584986546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d058888a0 0x556d05888d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.248+0000 7f58cb7fe700 1 -- 172.21.15.70:0/2584986546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d0588e320 msgr2=0x556d0588e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:25.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.248+0000 7f58cb7fe700 1 --2- 172.21.15.70:0/2584986546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d0588e320 0x556d0588e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.248+0000 7f58cb7fe700 1 -- 172.21.15.70:0/2584986546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d058888a0 msgr2=0x556d05888d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:25.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.248+0000 7f58cb7fe700 1 --2- 172.21.15.70:0/2584986546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d058888a0 0x556d05888d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.248+0000 7f58cb7fe700 1 -- 172.21.15.70:0/2584986546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d0577ffb0 con 0x556d05889260 2024-06-04T05:03:25.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.248+0000 7f58cb7fe700 1 --2- 172.21.15.70:0/2584986546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d05889260 0x556d05895080 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f58c40093c0 tx=0x7f58c40093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcce3fd2de4adef9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:25.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.248+0000 7f58c1ffb700 1 -- 172.21.15.70:0/2584986546 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c4020070 con 0x556d05889260 2024-06-04T05:03:25.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.248+0000 7f58c1ffb700 1 -- 172.21.15.70:0/2584986546 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f58c40074e0 con 0x556d05889260 2024-06-04T05:03:25.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.248+0000 7f58c1ffb700 1 -- 172.21.15.70:0/2584986546 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c4005260 con 0x556d05889260 2024-06-04T05:03:25.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.249+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/2584986546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d05889260 msgr2=0x556d05895080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:25.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.249+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/2584986546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d05889260 0x556d05895080 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f58c40093c0 tx=0x7f58c40093f0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.249+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/2584986546 shutdown_connections 2024-06-04T05:03:25.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.249+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/2584986546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d05889260 0x556d05895080 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.249+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/2584986546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d0588e320 0x556d0588e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.249+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/2584986546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d058888a0 0x556d05888d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.249+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/2584986546 >> 172.21.15.70:0/2584986546 conn(0x556d057b4240 msgr2=0x556d058833e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:25.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.249+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/2584986546 shutdown_connections 2024-06-04T05:03:25.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.249+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/2584986546 wait complete. 2024-06-04T05:03:25.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.249+0000 7f58dc1c1b80 1 -- start start 2024-06-04T05:03:25.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58dc1c1b80 1 Processor -- start 2024-06-04T05:03:25.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58dc1c1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d058888a0 0x556d05894ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58dc1c1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d05889260 0x556d05892030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58dc1c1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d0588e320 0x556d05892570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58dc1c1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556d056dcfa0 con 0x556d05889260 2024-06-04T05:03:25.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58dc1c1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556d0573eda0 con 0x556d058888a0 2024-06-04T05:03:25.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58dc1c1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556d05895680 con 0x556d0588e320 2024-06-04T05:03:25.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58dc1c1b80 10 client.0 _ll_register_callbacks cb 0x556d05771950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:25.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58c3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d05889260 0x556d05892030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58c3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d05889260 0x556d05892030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42706/0 (socket says 172.21.15.70:42706) 2024-06-04T05:03:25.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58c3fff700 1 -- 172.21.15.70:0/3540121516 learned_addr learned my addr 172.21.15.70:0/3540121516 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:25.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58cb7fe700 1 --2- 172.21.15.70:0/3540121516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d0588e320 0x556d05892570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58caffd700 1 --2- 172.21.15.70:0/3540121516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d058888a0 0x556d05894ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58c3fff700 1 -- 172.21.15.70:0/3540121516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d0588e320 msgr2=0x556d05892570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:25.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58c3fff700 1 --2- 172.21.15.70:0/3540121516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d0588e320 0x556d05892570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58c3fff700 1 -- 172.21.15.70:0/3540121516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d058888a0 msgr2=0x556d05894ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:25.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58c3fff700 1 --2- 172.21.15.70:0/3540121516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d058888a0 0x556d05894ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.250+0000 7f58c3fff700 1 -- 172.21.15.70:0/3540121516 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d057268d0 con 0x556d05889260 2024-06-04T05:03:25.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.251+0000 7f58c3fff700 1 --2- 172.21.15.70:0/3540121516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d05889260 0x556d05892030 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f58b40046d0 tx=0x7f58b40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faae2ac916414e05 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:25.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.251+0000 7f58aaffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:25.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.251+0000 7f58aaffd700 1 -- 172.21.15.70:0/3540121516 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b4016070 con 0x556d05889260 2024-06-04T05:03:25.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.251+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/3540121516 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58c40074e0 con 0x556d05889260 2024-06-04T05:03:25.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.251+0000 7f58aaffd700 1 -- 172.21.15.70:0/3540121516 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f58b4010840 con 0x556d05889260 2024-06-04T05:03:25.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.251+0000 7f58aaffd700 1 -- 172.21.15.70:0/3540121516 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f58b4010a40 con 0x556d05889260 2024-06-04T05:03:25.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.251+0000 7f58dc1c1b80 10 client.17094 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:25.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.251+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/3540121516 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556d05790760 con 0x556d05889260 2024-06-04T05:03:25.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.251+0000 7f58aaffd700 1 -- 172.21.15.70:0/3540121516 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f58c40074e0 con 0x556d05889260 2024-06-04T05:03:25.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.251+0000 7f58aaffd700 1 -- 172.21.15.70:0/3540121516 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f58b4019070 con 0x556d05889260 2024-06-04T05:03:25.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.251+0000 7f58dc1c1b80 10 client.17094 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:25.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.251+0000 7f58dc1c1b80 10 client.17094 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:03:25.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.251+0000 7f58dc1c1b80 20 client.17094 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:25.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.251+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/3540121516 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x556d058a2170 0x556d058a4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.251+0000 7f58dc1c1b80 4 client.17094 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:03:25.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.251+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/3540121516 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d057e8cc0 con 0x556d058a2170 2024-06-04T05:03:25.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.252+0000 7f58caffd700 1 --2- 172.21.15.70:0/3540121516 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x556d058a2170 0x556d058a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.252+0000 7f58aaffd700 10 client.17094 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:03:25.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.252+0000 7f58caffd700 1 --2- 172.21.15.70:0/3540121516 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x556d058a2170 0x556d058a4580 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6bf3182319a09a17 server_cookie=e5761ef245fe8575 in_seq=0 out_seq=0 2024-06-04T05:03:25.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.253+0000 7f58aaffd700 1 -- 172.21.15.70:0/3540121516 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556d057e8cc0 con 0x556d058a2170 2024-06-04T05:03:25.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.253+0000 7f58aaffd700 10 client.17094 handle_command_reply: tid=0 2024-06-04T05:03:25.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.253+0000 7f58dc1c1b80 1 client.17094 shutdown 2024-06-04T05:03:25.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.253+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/3540121516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d05889260 msgr2=0x556d05892030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:25.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.253+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/3540121516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d05889260 0x556d05892030 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f58b40046d0 tx=0x7f58b40047c0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.253+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/3540121516 shutdown_connections 2024-06-04T05:03:25.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.253+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/3540121516 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x556d058a2170 0x556d058a4580 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.253+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/3540121516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d0588e320 0x556d05892570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.253+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/3540121516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d058888a0 0x556d05894ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.253+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/3540121516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d05889260 0x556d05892030 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.253+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/3540121516 >> 172.21.15.70:0/3540121516 conn(0x556d057b4240 msgr2=0x556d05882af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:25.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.253+0000 7f58aaffd700 0 client.17094 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:03:25.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.253+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/3540121516 shutdown_connections 2024-06-04T05:03:25.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.253+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/3540121516 wait complete. 2024-06-04T05:03:25.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.253+0000 7f58dc1c1b80 20 client.17094 trim_cache size 0 max 16384 2024-06-04T05:03:25.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.268+0000 7f58dc1c1b80 1 Processor -- start 2024-06-04T05:03:25.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.268+0000 7f58dc1c1b80 1 -- start start 2024-06-04T05:03:25.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58dc1c1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d058888a0 0x556d0588d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58dc1c1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d05889260 0x556d05883270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58dc1c1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d0588e320 0x556d058a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58dc1c1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc1acc20 con 0x556d0588e320 2024-06-04T05:03:25.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58dc1c1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc1ae760 con 0x556d058888a0 2024-06-04T05:03:25.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58dc1c1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc1ad2f0 con 0x556d05889260 2024-06-04T05:03:25.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58cb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d0588e320 0x556d058a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58cb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d0588e320 0x556d058a9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42708/0 (socket says 172.21.15.70:42708) 2024-06-04T05:03:25.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58cb7fe700 1 -- 172.21.15.70:0/800301975 learned_addr learned my addr 172.21.15.70:0/800301975 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:25.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58caffd700 1 --2- 172.21.15.70:0/800301975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d058888a0 0x556d0588d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58c3fff700 1 --2- 172.21.15.70:0/800301975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d05889260 0x556d05883270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58cb7fe700 1 -- 172.21.15.70:0/800301975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d05889260 msgr2=0x556d05883270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:25.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58cb7fe700 1 --2- 172.21.15.70:0/800301975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d05889260 0x556d05883270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58cb7fe700 1 -- 172.21.15.70:0/800301975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d058888a0 msgr2=0x556d0588d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:25.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58cb7fe700 1 --2- 172.21.15.70:0/800301975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d058888a0 0x556d0588d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58cb7fe700 1 -- 172.21.15.70:0/800301975 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d05894920 con 0x556d0588e320 2024-06-04T05:03:25.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58cb7fe700 1 --2- 172.21.15.70:0/800301975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d0588e320 0x556d058a9e90 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009fd0 tx=0x7f58c4006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b9717fe0b0bffe server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:25.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58c17fa700 1 -- 172.21.15.70:0/800301975 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c402c070 con 0x556d0588e320 2024-06-04T05:03:25.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58c17fa700 1 -- 172.21.15.70:0/800301975 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f58c4003cc0 con 0x556d0588e320 2024-06-04T05:03:25.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.269+0000 7f58c17fa700 1 -- 172.21.15.70:0/800301975 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c4020040 con 0x556d0588e320 2024-06-04T05:03:25.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.270+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/800301975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d0588e320 msgr2=0x556d058a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:25.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.270+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/800301975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d0588e320 0x556d058a9e90 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009fd0 tx=0x7f58c4006000 comp rx=0 tx=0).stop 2024-06-04T05:03:25.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.270+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/800301975 shutdown_connections 2024-06-04T05:03:25.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.270+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/800301975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d0588e320 0x556d058a9e90 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.270+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/800301975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d058888a0 0x556d0588d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.270+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/800301975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d05889260 0x556d05883270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.270+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/800301975 >> 172.21.15.70:0/800301975 conn(0x556d057b4240 msgr2=0x556d0588d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:25.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.270+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/800301975 shutdown_connections 2024-06-04T05:03:25.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.270+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/800301975 wait complete. 2024-06-04T05:03:25.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.270+0000 7f58dc1c1b80 1 -- start start 2024-06-04T05:03:25.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58dc1c1b80 1 Processor -- start 2024-06-04T05:03:25.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58dc1c1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d058888a0 0x556d0589ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58dc1c1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d05889260 0x556d0589b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58dc1c1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d0588e320 0x556d058a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58dc1c1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc1ae090 con 0x556d058888a0 2024-06-04T05:03:25.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58dc1c1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc1ad9c0 con 0x556d0588e320 2024-06-04T05:03:25.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58dc1c1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc1aabc0 con 0x556d05889260 2024-06-04T05:03:25.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58dc1c1b80 10 client.0 _ll_register_callbacks cb 0x556d05771950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:25.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58caffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d058888a0 0x556d0589ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58caffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d058888a0 0x556d0589ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42722/0 (socket says 172.21.15.70:42722) 2024-06-04T05:03:25.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58caffd700 1 -- 172.21.15.70:0/776396693 learned_addr learned my addr 172.21.15.70:0/776396693 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:25.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58cb7fe700 1 --2- 172.21.15.70:0/776396693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d0588e320 0x556d058a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58c3fff700 1 --2- 172.21.15.70:0/776396693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d05889260 0x556d0589b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58caffd700 1 -- 172.21.15.70:0/776396693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d05889260 msgr2=0x556d0589b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:25.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58caffd700 1 --2- 172.21.15.70:0/776396693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d05889260 0x556d0589b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58caffd700 1 -- 172.21.15.70:0/776396693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d0588e320 msgr2=0x556d058a94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:25.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58caffd700 1 --2- 172.21.15.70:0/776396693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d0588e320 0x556d058a94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58caffd700 1 -- 172.21.15.70:0/776396693 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d058a9cd0 con 0x556d058888a0 2024-06-04T05:03:25.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.271+0000 7f58cb7fe700 1 --2- 172.21.15.70:0/776396693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d0588e320 0x556d058a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:03:25.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.272+0000 7f58caffd700 1 --2- 172.21.15.70:0/776396693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d058888a0 0x556d0589ab90 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f58bc00b500 tx=0x7f58bc0a8e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=affb1fe05fe9da19 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:25.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.272+0000 7f58aaffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:25.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.272+0000 7f58aaffd700 1 -- 172.21.15.70:0/776396693 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58bc0a51c0 con 0x556d058888a0 2024-06-04T05:03:25.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.272+0000 7f58aaffd700 1 -- 172.21.15.70:0/776396693 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f58bc0a9190 con 0x556d058888a0 2024-06-04T05:03:25.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.272+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/776396693 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58c4003cc0 con 0x556d058888a0 2024-06-04T05:03:25.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.272+0000 7f58aaffd700 1 -- 172.21.15.70:0/776396693 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f58bc0a5360 con 0x556d058888a0 2024-06-04T05:03:25.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.272+0000 7f58dc1c1b80 10 client.17100 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:25.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.272+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/776396693 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556d0590c960 con 0x556d058888a0 2024-06-04T05:03:25.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.272+0000 7f58aaffd700 1 -- 172.21.15.70:0/776396693 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f58c4003cc0 con 0x556d058888a0 2024-06-04T05:03:25.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.272+0000 7f58aaffd700 1 -- 172.21.15.70:0/776396693 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f58bc0a48a0 con 0x556d058888a0 2024-06-04T05:03:25.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.272+0000 7f58dc1c1b80 10 client.17100 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:25.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.272+0000 7f58dc1c1b80 10 client.17100 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:03:25.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.272+0000 7f58dc1c1b80 20 client.17100 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:25.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.272+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/776396693 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x556d0590cff0 0x556d0590f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:25.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.272+0000 7f58dc1c1b80 4 client.17100 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:03:25.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.272+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/776396693 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556d057e24a0 con 0x556d0590cff0 2024-06-04T05:03:25.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.273+0000 7f58c3fff700 1 --2- 172.21.15.70:0/776396693 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x556d0590cff0 0x556d0590f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:25.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.273+0000 7f58c3fff700 1 --2- 172.21.15.70:0/776396693 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x556d0590cff0 0x556d0590f410 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=79c8b51468af9a97 server_cookie=3e3ce5c0a72cefee in_seq=0 out_seq=0 2024-06-04T05:03:25.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.273+0000 7f58aaffd700 10 client.17100 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:03:25.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.273+0000 7f58aaffd700 1 -- 172.21.15.70:0/776396693 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556d057e24a0 con 0x556d0590cff0 2024-06-04T05:03:25.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.273+0000 7f58aaffd700 10 client.17100 handle_command_reply: tid=0 2024-06-04T05:03:25.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.273+0000 7f58dc1c1b80 1 client.17100 shutdown 2024-06-04T05:03:25.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.273+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/776396693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d058888a0 msgr2=0x556d0589ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:25.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.273+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/776396693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d058888a0 0x556d0589ab90 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f58bc00b500 tx=0x7f58bc0a8e30 comp rx=0 tx=0).stop 2024-06-04T05:03:25.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.273+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/776396693 shutdown_connections 2024-06-04T05:03:25.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.273+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/776396693 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x556d0590cff0 0x556d0590f410 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.273+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/776396693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d0588e320 0x556d058a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58aaffd700 0 client.17100 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:03:25.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/776396693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d058888a0 0x556d0589ab90 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58dc1c1b80 1 --2- 172.21.15.70:0/776396693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d05889260 0x556d0589b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/776396693 >> 172.21.15.70:0/776396693 conn(0x556d057b4240 msgr2=0x556d0588bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:25.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/776396693 shutdown_connections 2024-06-04T05:03:25.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58dc1c1b80 1 -- 172.21.15.70:0/776396693 wait complete. 2024-06-04T05:03:25.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58dc1c1b80 20 client.17100 trim_cache size 0 max 16384 2024-06-04T05:03:25.281 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:03:25.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58d192c700 1 -- 172.21.15.70:0/1944635645 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f58b80b2a70 msgr2=0x7f58b80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:25.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58d192c700 1 --2- 172.21.15.70:0/1944635645 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f58b80b2a70 0x7f58b80b4f30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f58cc0799d0 tx=0x7f58b400a380 comp rx=0 tx=0).stop 2024-06-04T05:03:25.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58d192c700 1 -- 172.21.15.70:0/1944635645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58cc102440 msgr2=0x7f58cc19e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:25.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58d192c700 1 --2- 172.21.15.70:0/1944635645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58cc102440 0x7f58cc19e650 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f58bc00b530 tx=0x7f58bc00b840 comp rx=0 tx=0).stop 2024-06-04T05:03:25.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58d192c700 1 -- 172.21.15.70:0/1944635645 shutdown_connections 2024-06-04T05:03:25.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58d192c700 1 --2- 172.21.15.70:0/1944635645 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f58b80b2a70 0x7f58b80b4f30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58d192c700 1 --2- 172.21.15.70:0/1944635645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58cc108520 0x7f58cc198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58d192c700 1 --2- 172.21.15.70:0/1944635645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58cc102440 0x7f58cc19e650 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58d192c700 1 --2- 172.21.15.70:0/1944635645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58cc102e00 0x7f58cc19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:25.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58d192c700 1 -- 172.21.15.70:0/1944635645 >> 172.21.15.70:0/1944635645 conn(0x7f58cc0793f0 msgr2=0x7f58cc100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:25.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58d192c700 1 -- 172.21.15.70:0/1944635645 shutdown_connections 2024-06-04T05:03:25.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:25.274+0000 7f58d192c700 1 -- 172.21.15.70:0/1944635645 wait complete. 2024-06-04T05:03:25.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.33 MB/sec execute 138 sec latency 2218.113 ms 2024-06-04T05:03:25.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:25 smithi070 ceph-mon[32631]: pgmap v190: 97 pgs: 97 active+clean; 70 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-06-04T05:03:25.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:03:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:25 smithi112 ceph-mon[36453]: pgmap v190: 97 pgs: 97 active+clean; 70 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-06-04T05:03:25.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:25 smithi098 ceph-mon[35569]: pgmap v190: 97 pgs: 97 active+clean; 70 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-06-04T05:03:26.144 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:03:26.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.31 MB/sec execute 139 sec latency 3218.156 ms 2024-06-04T05:03:27.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:27 smithi070 ceph-mon[32631]: pgmap v191: 97 pgs: 97 active+clean; 68 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-06-04T05:03:27.722 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.29 MB/sec execute 140 sec latency 4218.202 ms 2024-06-04T05:03:27.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:27 smithi112 ceph-mon[36453]: pgmap v191: 97 pgs: 97 active+clean; 68 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-06-04T05:03:27.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.947+0000 7f4e9c532700 1 -- 172.21.15.70:0/2984620380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e94102440 msgr2=0x7f4e941028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:27.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.947+0000 7f4e9c532700 1 --2- 172.21.15.70:0/2984620380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e94102440 0x7f4e941028c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f4e84009990 tx=0x7f4e84009ca0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.947+0000 7f4e9c532700 1 -- 172.21.15.70:0/2984620380 shutdown_connections 2024-06-04T05:03:27.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.947+0000 7f4e9c532700 1 --2- 172.21.15.70:0/2984620380 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e94102e00 0x7f4e9410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.947+0000 7f4e9c532700 1 --2- 172.21.15.70:0/2984620380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e94108520 0x7f4e94108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.947+0000 7f4e9c532700 1 --2- 172.21.15.70:0/2984620380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e94102440 0x7f4e941028c0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.947+0000 7f4e9c532700 1 -- 172.21.15.70:0/2984620380 >> 172.21.15.70:0/2984620380 conn(0x7f4e940793f0 msgr2=0x7f4e94079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:27.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.947+0000 7f4e9c532700 1 -- 172.21.15.70:0/2984620380 shutdown_connections 2024-06-04T05:03:27.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.947+0000 7f4e9c532700 1 -- 172.21.15.70:0/2984620380 wait complete. 2024-06-04T05:03:27.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.947+0000 7f4e9c532700 1 Processor -- start 2024-06-04T05:03:27.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.947+0000 7f4e9c532700 1 -- start start 2024-06-04T05:03:27.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e9c532700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e94102440 0x7f4e94198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e9c532700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e94102e00 0x7f4e94198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e9c532700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e94108520 0x7f4e941991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e9c532700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e941124a0 con 0x7f4e94102e00 2024-06-04T05:03:27.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e9c532700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e94112320 con 0x7f4e94102440 2024-06-04T05:03:27.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e9c532700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4e94112620 con 0x7f4e94108520 2024-06-04T05:03:27.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e99acd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e94102e00 0x7f4e94198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e9a2ce700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e94102440 0x7f4e94198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e99acd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e94102e00 0x7f4e94198cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43262/0 (socket says 172.21.15.70:43262) 2024-06-04T05:03:27.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e9a2ce700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e94102440 0x7f4e94198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33638/0 (socket says 172.21.15.70:33638) 2024-06-04T05:03:27.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e99acd700 1 -- 172.21.15.70:0/2895993074 learned_addr learned my addr 172.21.15.70:0/2895993074 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:27.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e9aacf700 1 --2- 172.21.15.70:0/2895993074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e94108520 0x7f4e941991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e9a2ce700 1 -- 172.21.15.70:0/2895993074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e94108520 msgr2=0x7f4e941991f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e9a2ce700 1 --2- 172.21.15.70:0/2895993074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e94108520 0x7f4e941991f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e9a2ce700 1 -- 172.21.15.70:0/2895993074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e94102e00 msgr2=0x7f4e94198cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e9a2ce700 1 --2- 172.21.15.70:0/2895993074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e94102e00 0x7f4e94198cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e9a2ce700 1 -- 172.21.15.70:0/2895993074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e941a7fe0 con 0x7f4e94102440 2024-06-04T05:03:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e99acd700 1 --2- 172.21.15.70:0/2895993074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e94102e00 0x7f4e94198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.948+0000 7f4e9aacf700 1 --2- 172.21.15.70:0/2895993074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e94108520 0x7f4e941991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.949+0000 7f4e9a2ce700 1 --2- 172.21.15.70:0/2895993074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e94102440 0x7f4e94198770 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000e9f0 tx=0x7f4e9000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.949+0000 7f4e8b7fe700 1 -- 172.21.15.70:0/2895993074 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9000c9c0 con 0x7f4e94102440 2024-06-04T05:03:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.949+0000 7f4e8b7fe700 1 -- 172.21.15.70:0/2895993074 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e90010430 con 0x7f4e94102440 2024-06-04T05:03:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.949+0000 7f4e9c532700 1 -- 172.21.15.70:0/2895993074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e941a82d0 con 0x7f4e94102440 2024-06-04T05:03:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.949+0000 7f4e8b7fe700 1 -- 172.21.15.70:0/2895993074 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e90018680 con 0x7f4e94102440 2024-06-04T05:03:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.949+0000 7f4e9c532700 1 -- 172.21.15.70:0/2895993074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e941a8810 con 0x7f4e94102440 2024-06-04T05:03:27.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.952+0000 7f4e8b7fe700 1 -- 172.21.15.70:0/2895993074 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f4e9000cb60 con 0x7f4e94102440 2024-06-04T05:03:27.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.952+0000 7f4e8b7fe700 1 --2- 172.21.15.70:0/2895993074 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e800b29f0 0x7f4e800b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.952+0000 7f4ea6dc7b80 1 Processor -- start 2024-06-04T05:03:27.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.952+0000 7f4e99acd700 1 --2- 172.21.15.70:0/2895993074 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e800b29f0 0x7f4e800b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4e8b7fe700 1 -- 172.21.15.70:0/2895993074 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f4e900e26c0 con 0x7f4e94102440 2024-06-04T05:03:27.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4ea6dc7b80 1 -- start start 2024-06-04T05:03:27.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4ea6dc7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e31860 0x55b813e31c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4ea6dc7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e32210 0x55b813e35940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4ea6dc7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e35f60 0x55b813e363a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4ea6dc7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b813d90c90 con 0x55b813e32210 2024-06-04T05:03:27.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4ea6dc7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b813d80830 con 0x55b813e35f60 2024-06-04T05:03:27.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4ea6dc7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b813d84870 con 0x55b813e31860 2024-06-04T05:03:27.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4e99acd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e32210 0x55b813e35940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4e99acd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e32210 0x55b813e35940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43266/0 (socket says 172.21.15.70:43266) 2024-06-04T05:03:27.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4e99acd700 1 -- 172.21.15.70:0/140725791 learned_addr learned my addr 172.21.15.70:0/140725791 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:27.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4e9aacf700 1 --2- 172.21.15.70:0/140725791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e35f60 0x55b813e363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4e99acd700 1 --2- 172.21.15.70:0/2895993074 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e800b29f0 0x7f4e800b4eb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4e9419a8d0 tx=0x7f4e8401a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:27.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4e9a2ce700 1 --2- 172.21.15.70:0/140725791 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e31860 0x55b813e31c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4e9aacf700 1 -- 172.21.15.70:0/140725791 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e31860 msgr2=0x55b813e31c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:27.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4e9aacf700 1 --2- 172.21.15.70:0/140725791 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e31860 0x55b813e31c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4e9aacf700 1 -- 172.21.15.70:0/140725791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e32210 msgr2=0x55b813e35940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:27.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4e9aacf700 1 --2- 172.21.15.70:0/140725791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e32210 0x55b813e35940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4e9aacf700 1 -- 172.21.15.70:0/140725791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b813cd08d0 con 0x55b813e35f60 2024-06-04T05:03:27.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.953+0000 7f4e99acd700 1 --2- 172.21.15.70:0/140725791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e32210 0x55b813e35940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:27.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.954+0000 7f4e9aacf700 1 --2- 172.21.15.70:0/140725791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e35f60 0x55b813e363a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c003db0 tx=0x7f4e8c009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28c87540ddf2d4ec server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:27.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.954+0000 7f4e88ff9700 1 -- 172.21.15.70:0/140725791 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c019030 con 0x55b813e35f60 2024-06-04T05:03:27.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.954+0000 7f4e88ff9700 1 -- 172.21.15.70:0/140725791 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e8c004bb0 con 0x55b813e35f60 2024-06-04T05:03:27.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.954+0000 7f4e88ff9700 1 -- 172.21.15.70:0/140725791 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c007c80 con 0x55b813e35f60 2024-06-04T05:03:27.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.955+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/140725791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e35f60 msgr2=0x55b813e363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:27.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.955+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/140725791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e35f60 0x55b813e363a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c003db0 tx=0x7f4e8c009920 comp rx=0 tx=0).stop 2024-06-04T05:03:27.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.955+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/140725791 shutdown_connections 2024-06-04T05:03:27.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.955+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/140725791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e35f60 0x55b813e363a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.955+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/140725791 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e31860 0x55b813e31c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.955+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/140725791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e32210 0x55b813e35940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.955+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/140725791 >> 172.21.15.70:0/140725791 conn(0x55b813d5f080 msgr2=0x55b813e2be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:27.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.955+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/140725791 shutdown_connections 2024-06-04T05:03:27.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.955+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/140725791 wait complete. 2024-06-04T05:03:27.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.955+0000 7f4ea6dc7b80 1 -- start start 2024-06-04T05:03:27.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.955+0000 7f4ea6dc7b80 1 Processor -- start 2024-06-04T05:03:27.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.955+0000 7f4ea6dc7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e31860 0x55b813e3d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.955+0000 7f4ea6dc7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e32210 0x55b813e3a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.955+0000 7f4ea6dc7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e35f60 0x55b813e3ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.955+0000 7f4ea6dc7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b813c86fa0 con 0x55b813e31860 2024-06-04T05:03:27.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.955+0000 7f4ea6dc7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b813d227c0 con 0x55b813e32210 2024-06-04T05:03:27.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.955+0000 7f4ea6dc7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b813e3e120 con 0x55b813e35f60 2024-06-04T05:03:27.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4ea6dc7b80 10 client.0 _ll_register_callbacks cb 0x55b813d1b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:27.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4e9a2ce700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e31860 0x55b813e3d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4e99acd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e32210 0x55b813e3a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4e9a2ce700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e31860 0x55b813e3d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43282/0 (socket says 172.21.15.70:43282) 2024-06-04T05:03:27.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4e99acd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e32210 0x55b813e3a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33658/0 (socket says 172.21.15.70:33658) 2024-06-04T05:03:27.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4e9a2ce700 1 -- 172.21.15.70:0/1823767685 learned_addr learned my addr 172.21.15.70:0/1823767685 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:27.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4e9aacf700 1 --2- 172.21.15.70:0/1823767685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e35f60 0x55b813e3ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4e9a2ce700 1 -- 172.21.15.70:0/1823767685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e35f60 msgr2=0x55b813e3ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:27.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4e9a2ce700 1 --2- 172.21.15.70:0/1823767685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e35f60 0x55b813e3ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4e9a2ce700 1 -- 172.21.15.70:0/1823767685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e32210 msgr2=0x55b813e3a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:27.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4e9a2ce700 1 --2- 172.21.15.70:0/1823767685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e32210 0x55b813e3a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4e9a2ce700 1 -- 172.21.15.70:0/1823767685 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b813d1f690 con 0x55b813e31860 2024-06-04T05:03:27.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4e9aacf700 1 --2- 172.21.15.70:0/1823767685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e35f60 0x55b813e3ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:27.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4e99acd700 1 --2- 172.21.15.70:0/1823767685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e32210 0x55b813e3a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:27.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4e9a2ce700 1 --2- 172.21.15.70:0/1823767685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e31860 0x55b813e3d2d0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f4e900a6bb0 tx=0x7f4e900a9180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a2c18a8c1929ac server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:27.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4e7dffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:27.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4e7dffb700 1 -- 172.21.15.70:0/1823767685 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9008fe60 con 0x55b813e31860 2024-06-04T05:03:27.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4e7dffb700 1 -- 172.21.15.70:0/1823767685 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f4e900a9db0 con 0x55b813e31860 2024-06-04T05:03:27.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.956+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/1823767685 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e8c004bb0 con 0x55b813e31860 2024-06-04T05:03:27.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.957+0000 7f4e7dffb700 1 -- 172.21.15.70:0/1823767685 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4e900a9450 con 0x55b813e31860 2024-06-04T05:03:27.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.957+0000 7f4ea6dc7b80 10 client.17112 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:27.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.957+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/1823767685 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b813e487f0 con 0x55b813e31860 2024-06-04T05:03:27.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.957+0000 7f4e7dffb700 1 -- 172.21.15.70:0/1823767685 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e8c004bb0 con 0x55b813e31860 2024-06-04T05:03:27.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.957+0000 7f4e7dffb700 1 -- 172.21.15.70:0/1823767685 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f4e900b2310 con 0x55b813e31860 2024-06-04T05:03:27.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.957+0000 7f4ea6dc7b80 10 client.17112 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:27.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.957+0000 7f4ea6dc7b80 10 client.17112 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:03:27.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.957+0000 7f4ea6dc7b80 20 client.17112 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:27.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.957+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/1823767685 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55b813e4ae30 0x55b813e4d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.957+0000 7f4ea6dc7b80 4 client.17112 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:03:27.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.957+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/1823767685 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b813dfd9e0 con 0x55b813e4ae30 2024-06-04T05:03:27.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.957+0000 7f4e99acd700 1 --2- 172.21.15.70:0/1823767685 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55b813e4ae30 0x55b813e4d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.958+0000 7f4e99acd700 1 --2- 172.21.15.70:0/1823767685 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55b813e4ae30 0x55b813e4d220 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bebf4678045c9789 server_cookie=40c57548c419497e in_seq=0 out_seq=0 2024-06-04T05:03:27.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.958+0000 7f4e7dffb700 10 client.17112 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:03:27.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.959+0000 7f4e7dffb700 1 -- 172.21.15.70:0/1823767685 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b813dfd9e0 con 0x55b813e4ae30 2024-06-04T05:03:27.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.959+0000 7f4e7dffb700 10 client.17112 handle_command_reply: tid=0 2024-06-04T05:03:27.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.959+0000 7f4ea6dc7b80 1 client.17112 shutdown 2024-06-04T05:03:27.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.959+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/1823767685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e31860 msgr2=0x55b813e3d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:27.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.959+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/1823767685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e31860 0x55b813e3d2d0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f4e900a6bb0 tx=0x7f4e900a9180 comp rx=0 tx=0).stop 2024-06-04T05:03:27.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.959+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/1823767685 shutdown_connections 2024-06-04T05:03:27.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.959+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/1823767685 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55b813e4ae30 0x55b813e4d220 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.959+0000 7f4e7dffb700 0 client.17112 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:03:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.959+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/1823767685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e35f60 0x55b813e3ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.959+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/1823767685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e31860 0x55b813e3d2d0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.959+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/1823767685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e32210 0x55b813e3a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.959+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/1823767685 >> 172.21.15.70:0/1823767685 conn(0x55b813d5f080 msgr2=0x55b813e2b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.959+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/1823767685 shutdown_connections 2024-06-04T05:03:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.959+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/1823767685 wait complete. 2024-06-04T05:03:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.959+0000 7f4ea6dc7b80 20 client.17112 trim_cache size 0 max 16384 2024-06-04T05:03:27.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.974+0000 7f4ea6dc7b80 1 Processor -- start 2024-06-04T05:03:27.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.974+0000 7f4ea6dc7b80 1 -- start start 2024-06-04T05:03:27.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.974+0000 7f4ea6dc7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e31860 0x55b813e34c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.974+0000 7f4ea6dc7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e32210 0x55b813e39580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.974+0000 7f4ea6dc7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e35f60 0x55b813e2bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.974+0000 7f4ea6dc7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e941b0080 con 0x55b813e35f60 2024-06-04T05:03:27.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.974+0000 7f4ea6dc7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e941b1bc0 con 0x55b813e31860 2024-06-04T05:03:27.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.974+0000 7f4ea6dc7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4e941b0750 con 0x55b813e32210 2024-06-04T05:03:27.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.975+0000 7f4e9aacf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e35f60 0x55b813e2bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.975+0000 7f4e9aacf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e35f60 0x55b813e2bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43290/0 (socket says 172.21.15.70:43290) 2024-06-04T05:03:27.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.975+0000 7f4e9aacf700 1 -- 172.21.15.70:0/3698190672 learned_addr learned my addr 172.21.15.70:0/3698190672 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:27.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.975+0000 7f4e9a2ce700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e31860 0x55b813e34c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.975+0000 7f4e99acd700 1 --2- 172.21.15.70:0/3698190672 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e32210 0x55b813e39580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.975+0000 7f4e9a2ce700 1 -- 172.21.15.70:0/3698190672 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e32210 msgr2=0x55b813e39580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:27.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.975+0000 7f4e9a2ce700 1 --2- 172.21.15.70:0/3698190672 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e32210 0x55b813e39580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.975+0000 7f4e9a2ce700 1 -- 172.21.15.70:0/3698190672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e35f60 msgr2=0x55b813e2bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:27.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.975+0000 7f4e9a2ce700 1 --2- 172.21.15.70:0/3698190672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e35f60 0x55b813e2bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.975+0000 7f4e9a2ce700 1 -- 172.21.15.70:0/3698190672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b813e2cd40 con 0x55b813e31860 2024-06-04T05:03:27.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.975+0000 7f4e9aacf700 1 --2- 172.21.15.70:0/3698190672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e35f60 0x55b813e2bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:27.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.975+0000 7f4e9a2ce700 1 --2- 172.21.15.70:0/3698190672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e31860 0x55b813e34c20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000e9c0 tx=0x7f4e900a9840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4c93846b9fa66c0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:27.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.975+0000 7f4e7ffff700 1 -- 172.21.15.70:0/3698190672 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9008fe60 con 0x55b813e31860 2024-06-04T05:03:27.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.975+0000 7f4e7ffff700 1 -- 172.21.15.70:0/3698190672 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e900aa040 con 0x55b813e31860 2024-06-04T05:03:27.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.975+0000 7f4e7ffff700 1 -- 172.21.15.70:0/3698190672 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e900a8ba0 con 0x55b813e31860 2024-06-04T05:03:27.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.976+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/3698190672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e31860 msgr2=0x55b813e34c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:27.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.976+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/3698190672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e31860 0x55b813e34c20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000e9c0 tx=0x7f4e900a9840 comp rx=0 tx=0).stop 2024-06-04T05:03:27.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.976+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/3698190672 shutdown_connections 2024-06-04T05:03:27.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.976+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/3698190672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e35f60 0x55b813e2bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.976+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/3698190672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e31860 0x55b813e34c20 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.976+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/3698190672 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e32210 0x55b813e39580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.976+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/3698190672 >> 172.21.15.70:0/3698190672 conn(0x55b813d5f080 msgr2=0x55b813e30f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:27.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.976+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/3698190672 shutdown_connections 2024-06-04T05:03:27.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.976+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/3698190672 wait complete. 2024-06-04T05:03:27.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.976+0000 7f4ea6dc7b80 1 -- start start 2024-06-04T05:03:27.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.976+0000 7f4ea6dc7b80 1 Processor -- start 2024-06-04T05:03:27.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4ea6dc7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e31860 0x55b813e43800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4ea6dc7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e32210 0x55b813e40ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4ea6dc7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e35f60 0x55b813e410e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4ea6dc7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e941b14f0 con 0x55b813e31860 2024-06-04T05:03:27.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4ea6dc7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e941b0e20 con 0x55b813e32210 2024-06-04T05:03:27.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4ea6dc7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4e941ae020 con 0x55b813e35f60 2024-06-04T05:03:27.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4ea6dc7b80 10 client.0 _ll_register_callbacks cb 0x55b813d1b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:27.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4e9a2ce700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e31860 0x55b813e43800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4e99acd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e32210 0x55b813e40ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4e99acd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e32210 0x55b813e40ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33668/0 (socket says 172.21.15.70:33668) 2024-06-04T05:03:27.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4e9a2ce700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e31860 0x55b813e43800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43300/0 (socket says 172.21.15.70:43300) 2024-06-04T05:03:27.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4e99acd700 1 -- 172.21.15.70:0/1978147289 learned_addr learned my addr 172.21.15.70:0/1978147289 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:27.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4e9aacf700 1 --2- 172.21.15.70:0/1978147289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e35f60 0x55b813e410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4e99acd700 1 -- 172.21.15.70:0/1978147289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e35f60 msgr2=0x55b813e410e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:27.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4e99acd700 1 --2- 172.21.15.70:0/1978147289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e35f60 0x55b813e410e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4e99acd700 1 -- 172.21.15.70:0/1978147289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e31860 msgr2=0x55b813e43800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:27.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4e99acd700 1 --2- 172.21.15.70:0/1978147289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e31860 0x55b813e43800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4e99acd700 1 -- 172.21.15.70:0/1978147289 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b813e41620 con 0x55b813e32210 2024-06-04T05:03:27.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4e99acd700 1 --2- 172.21.15.70:0/1978147289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e32210 0x55b813e40ba0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4e84000c00 tx=0x7f4e84009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db471f57479c4735 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:27.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4e7dffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:03:27.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.977+0000 7f4e7dffb700 1 -- 172.21.15.70:0/1978147289 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e84033870 con 0x55b813e32210 2024-06-04T05:03:27.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.978+0000 7f4e7dffb700 1 -- 172.21.15.70:0/1978147289 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f4e84033a10 con 0x55b813e32210 2024-06-04T05:03:27.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.978+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/1978147289 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e900aa040 con 0x55b813e32210 2024-06-04T05:03:27.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.978+0000 7f4e7dffb700 1 -- 172.21.15.70:0/1978147289 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4e84033c10 con 0x55b813e32210 2024-06-04T05:03:27.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.978+0000 7f4ea6dc7b80 10 client.26408 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:27.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.978+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/1978147289 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b813eb5740 con 0x55b813e32210 2024-06-04T05:03:27.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.978+0000 7f4e7dffb700 1 -- 172.21.15.70:0/1978147289 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e900aa040 con 0x55b813e32210 2024-06-04T05:03:27.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.978+0000 7f4e7dffb700 1 -- 172.21.15.70:0/1978147289 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f4e8400fd10 con 0x55b813e32210 2024-06-04T05:03:27.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.978+0000 7f4ea6dc7b80 10 client.26408 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:27.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.978+0000 7f4ea6dc7b80 10 client.26408 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:03:27.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.978+0000 7f4ea6dc7b80 20 client.26408 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:27.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.978+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/1978147289 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55b813eb5eb0 0x55b813eb82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:27.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.978+0000 7f4ea6dc7b80 4 client.26408 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:03:27.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.978+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/1978147289 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b813d92be0 con 0x55b813eb5eb0 2024-06-04T05:03:27.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.978+0000 7f4e9a2ce700 1 --2- 172.21.15.70:0/1978147289 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55b813eb5eb0 0x55b813eb82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:27.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.979+0000 7f4e7dffb700 10 client.26408 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:03:27.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.979+0000 7f4e9a2ce700 1 --2- 172.21.15.70:0/1978147289 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55b813eb5eb0 0x55b813eb82d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3c8940e73be3fad0 server_cookie=6ae26014bc9bf94e in_seq=0 out_seq=0 2024-06-04T05:03:27.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.979+0000 7f4e7dffb700 1 -- 172.21.15.70:0/1978147289 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b813d92be0 con 0x55b813eb5eb0 2024-06-04T05:03:27.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.979+0000 7f4e7dffb700 10 client.26408 handle_command_reply: tid=0 2024-06-04T05:03:27.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.979+0000 7f4ea6dc7b80 1 client.26408 shutdown 2024-06-04T05:03:27.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.979+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/1978147289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e32210 msgr2=0x55b813e40ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:27.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.979+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/1978147289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e32210 0x55b813e40ba0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4e84000c00 tx=0x7f4e84009310 comp rx=0 tx=0).stop 2024-06-04T05:03:27.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.979+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/1978147289 shutdown_connections 2024-06-04T05:03:27.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.979+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/1978147289 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55b813eb5eb0 0x55b813eb82d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.979+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/1978147289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b813e35f60 0x55b813e410e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.979+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/1978147289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b813e31860 0x55b813e43800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.979+0000 7f4e7dffb700 0 client.26408 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:03:27.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.979+0000 7f4ea6dc7b80 1 --2- 172.21.15.70:0/1978147289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b813e32210 0x55b813e40ba0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.979+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/1978147289 >> 172.21.15.70:0/1978147289 conn(0x55b813d5f080 msgr2=0x55b813e3be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:27.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.979+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/1978147289 shutdown_connections 2024-06-04T05:03:27.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.979+0000 7f4ea6dc7b80 1 -- 172.21.15.70:0/1978147289 wait complete. 2024-06-04T05:03:27.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.979+0000 7f4ea6dc7b80 20 client.26408 trim_cache size 0 max 16384 2024-06-04T05:03:27.987 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:03:27.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.980+0000 7f4e9c532700 1 -- 172.21.15.70:0/2895993074 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e800b29f0 msgr2=0x7f4e800b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:27.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.980+0000 7f4e9c532700 1 --2- 172.21.15.70:0/2895993074 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e800b29f0 0x7f4e800b4eb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4e9419a8d0 tx=0x7f4e8401a040 comp rx=0 tx=0).stop 2024-06-04T05:03:27.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.980+0000 7f4e9c532700 1 -- 172.21.15.70:0/2895993074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e94102440 msgr2=0x7f4e94198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:27.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.980+0000 7f4e9c532700 1 --2- 172.21.15.70:0/2895993074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e94102440 0x7f4e94198770 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000e9f0 tx=0x7f4e9000ed00 comp rx=0 tx=0).stop 2024-06-04T05:03:27.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.980+0000 7f4e9c532700 1 -- 172.21.15.70:0/2895993074 shutdown_connections 2024-06-04T05:03:27.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.980+0000 7f4e9c532700 1 --2- 172.21.15.70:0/2895993074 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e800b29f0 0x7f4e800b4eb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.980+0000 7f4e9c532700 1 --2- 172.21.15.70:0/2895993074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e94108520 0x7f4e941991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.980+0000 7f4e9c532700 1 --2- 172.21.15.70:0/2895993074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e94102440 0x7f4e94198770 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.980+0000 7f4e9c532700 1 --2- 172.21.15.70:0/2895993074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e94102e00 0x7f4e94198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:27.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.980+0000 7f4e9c532700 1 -- 172.21.15.70:0/2895993074 >> 172.21.15.70:0/2895993074 conn(0x7f4e940793f0 msgr2=0x7f4e94100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:27.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.980+0000 7f4e9c532700 1 -- 172.21.15.70:0/2895993074 shutdown_connections 2024-06-04T05:03:27.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:27.980+0000 7f4e9c532700 1 -- 172.21.15.70:0/2895993074 wait complete. 2024-06-04T05:03:27.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:27 smithi098 ceph-mon[35569]: pgmap v191: 97 pgs: 97 active+clean; 68 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-06-04T05:03:28.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.28 MB/sec execute 141 sec latency 5218.260 ms 2024-06-04T05:03:28.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:28 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:03:28.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:28 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:03:28.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:28 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:03:28.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:28 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:03:28.773 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:03:28.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:28 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:03:28.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:28 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:03:28.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:28 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:03:28.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:28 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:03:28.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:28 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:03:28.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:28 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:03:28.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:28 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:03:28.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:28 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:03:29.010 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:03:29.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.26 MB/sec execute 142 sec latency 6218.312 ms 2024-06-04T05:03:29.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:29 smithi070 ceph-mon[32631]: pgmap v192: 97 pgs: 97 active+clean; 68 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 1012 KiB/s wr, 105 op/s 2024-06-04T05:03:29.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:29 smithi112 ceph-mon[36453]: pgmap v192: 97 pgs: 97 active+clean; 68 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 1012 KiB/s wr, 105 op/s 2024-06-04T05:03:29.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:29 smithi098 ceph-mon[35569]: pgmap v192: 97 pgs: 97 active+clean; 68 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 1012 KiB/s wr, 105 op/s 2024-06-04T05:03:30.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.25 MB/sec execute 143 sec latency 7218.354 ms 2024-06-04T05:03:31.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.174+0000 7f94cd7b4700 1 -- 172.21.15.70:0/3393531670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94c8108520 msgr2=0x7f94c8108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.174+0000 7f94cd7b4700 1 --2- 172.21.15.70:0/3393531670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94c8108520 0x7f94c8108900 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009990 tx=0x7f94b8009ca0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.174+0000 7f94cd7b4700 1 -- 172.21.15.70:0/3393531670 shutdown_connections 2024-06-04T05:03:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.174+0000 7f94cd7b4700 1 --2- 172.21.15.70:0/3393531670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c8102e00 0x7f94c810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.174+0000 7f94cd7b4700 1 --2- 172.21.15.70:0/3393531670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94c8108520 0x7f94c8108900 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.174+0000 7f94cd7b4700 1 --2- 172.21.15.70:0/3393531670 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f94c8102440 0x7f94c81028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.174+0000 7f94cd7b4700 1 -- 172.21.15.70:0/3393531670 >> 172.21.15.70:0/3393531670 conn(0x7f94c80793f0 msgr2=0x7f94c8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.174+0000 7f94cd7b4700 1 -- 172.21.15.70:0/3393531670 shutdown_connections 2024-06-04T05:03:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.174+0000 7f94cd7b4700 1 -- 172.21.15.70:0/3393531670 wait complete. 2024-06-04T05:03:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.175+0000 7f94cd7b4700 1 Processor -- start 2024-06-04T05:03:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.175+0000 7f94cd7b4700 1 -- start start 2024-06-04T05:03:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.175+0000 7f94cd7b4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c8102440 0x7f94c819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.175+0000 7f94cd7b4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94c8102e00 0x7f94c819ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.175+0000 7f94cd7b4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f94c8108520 0x7f94c81a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.175+0000 7f94cd7b4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f94c8112480 con 0x7f94c8102e00 2024-06-04T05:03:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.175+0000 7f94cd7b4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f94c8112300 con 0x7f94c8102440 2024-06-04T05:03:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.175+0000 7f94cd7b4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f94c8112600 con 0x7f94c8108520 2024-06-04T05:03:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.175+0000 7f94bffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94c8102e00 0x7f94c819ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.175+0000 7f94bffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94c8102e00 0x7f94c819ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43316/0 (socket says 172.21.15.70:43316) 2024-06-04T05:03:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.175+0000 7f94bffff700 1 -- 172.21.15.70:0/1240688560 learned_addr learned my addr 172.21.15.70:0/1240688560 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.175+0000 7f94c6ffd700 1 --2- 172.21.15.70:0/1240688560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c8102440 0x7f94c819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.175+0000 7f94c77fe700 1 --2- 172.21.15.70:0/1240688560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f94c8108520 0x7f94c81a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.176+0000 7f94bffff700 1 -- 172.21.15.70:0/1240688560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f94c8108520 msgr2=0x7f94c81a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.176+0000 7f94bffff700 1 --2- 172.21.15.70:0/1240688560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f94c8108520 0x7f94c81a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.176+0000 7f94bffff700 1 -- 172.21.15.70:0/1240688560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c8102440 msgr2=0x7f94c819a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.176+0000 7f94bffff700 1 --2- 172.21.15.70:0/1240688560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c8102440 0x7f94c819a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.176+0000 7f94bffff700 1 -- 172.21.15.70:0/1240688560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c81a5ee0 con 0x7f94c8102e00 2024-06-04T05:03:31.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.176+0000 7f94bffff700 1 --2- 172.21.15.70:0/1240688560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94c8102e00 0x7f94c819ae60 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f94b000d950 tx=0x7f94b000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:31.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.176+0000 7f94c4ff9700 1 -- 172.21.15.70:0/1240688560 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b0016070 con 0x7f94c8102e00 2024-06-04T05:03:31.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.176+0000 7f94cd7b4700 1 -- 172.21.15.70:0/1240688560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94c81a6110 con 0x7f94c8102e00 2024-06-04T05:03:31.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.176+0000 7f94c4ff9700 1 -- 172.21.15.70:0/1240688560 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94b0009dd0 con 0x7f94c8102e00 2024-06-04T05:03:31.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.176+0000 7f94c4ff9700 1 -- 172.21.15.70:0/1240688560 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b000f3c0 con 0x7f94c8102e00 2024-06-04T05:03:31.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.176+0000 7f94cd7b4700 1 -- 172.21.15.70:0/1240688560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94c81a6650 con 0x7f94c8102e00 2024-06-04T05:03:31.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.178+0000 7f94c4ff9700 1 -- 172.21.15.70:0/1240688560 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f94b0019070 con 0x7f94c8102e00 2024-06-04T05:03:31.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.179+0000 7f94c4ff9700 1 --2- 172.21.15.70:0/1240688560 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f94b40b29f0 0x7f94b40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.179+0000 7f94c6ffd700 1 --2- 172.21.15.70:0/1240688560 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f94b40b29f0 0x7f94b40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.179+0000 7f94c4ff9700 1 -- 172.21.15.70:0/1240688560 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f94b00e2290 con 0x7f94c8102e00 2024-06-04T05:03:31.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.179+0000 7f94d8049b80 1 Processor -- start 2024-06-04T05:03:31.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.179+0000 7f94d8049b80 1 -- start start 2024-06-04T05:03:31.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.179+0000 7f94c6ffd700 1 --2- 172.21.15.70:0/1240688560 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f94b40b29f0 0x7f94b40b4eb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f94b800b5c0 tx=0x7f94b8005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:31.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94d8049b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94544860 0x556f94544c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94d8049b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94545210 0x556f94548940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94d8049b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94548f60 0x556f945493a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94d8049b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556f944a3c90 con 0x556f94545210 2024-06-04T05:03:31.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94d8049b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556f94493830 con 0x556f94548f60 2024-06-04T05:03:31.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94d8049b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556f94497870 con 0x556f94544860 2024-06-04T05:03:31.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94bffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94545210 0x556f94548940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94bffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94545210 0x556f94548940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43322/0 (socket says 172.21.15.70:43322) 2024-06-04T05:03:31.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94bffff700 1 -- 172.21.15.70:0/4079685767 learned_addr learned my addr 172.21.15.70:0/4079685767 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:31.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94c77fe700 1 --2- 172.21.15.70:0/4079685767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94548f60 0x556f945493a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94c6ffd700 1 --2- 172.21.15.70:0/4079685767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94544860 0x556f94544c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94bffff700 1 -- 172.21.15.70:0/4079685767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94544860 msgr2=0x556f94544c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:31.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94bffff700 1 --2- 172.21.15.70:0/4079685767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94544860 0x556f94544c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94bffff700 1 -- 172.21.15.70:0/4079685767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94548f60 msgr2=0x556f945493a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:31.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94bffff700 1 --2- 172.21.15.70:0/4079685767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94548f60 0x556f945493a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94bffff700 1 -- 172.21.15.70:0/4079685767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f943e38d0 con 0x556f94545210 2024-06-04T05:03:31.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94c77fe700 1 --2- 172.21.15.70:0/4079685767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94548f60 0x556f945493a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:03:31.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94bffff700 1 --2- 172.21.15.70:0/4079685767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94545210 0x556f94548940 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f94b00a55f0 tx=0x7f94b00a9c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8144170ab57b58ac server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:31.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.180+0000 7f94bd7fa700 1 -- 172.21.15.70:0/4079685767 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b0020070 con 0x556f94545210 2024-06-04T05:03:31.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.181+0000 7f94bd7fa700 1 -- 172.21.15.70:0/4079685767 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94b00b23d0 con 0x556f94545210 2024-06-04T05:03:31.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.181+0000 7f94bd7fa700 1 -- 172.21.15.70:0/4079685767 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b00b1f40 con 0x556f94545210 2024-06-04T05:03:31.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.181+0000 7f94d8049b80 1 -- 172.21.15.70:0/4079685767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94545210 msgr2=0x556f94548940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.181+0000 7f94d8049b80 1 --2- 172.21.15.70:0/4079685767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94545210 0x556f94548940 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f94b00a55f0 tx=0x7f94b00a9c70 comp rx=0 tx=0).stop 2024-06-04T05:03:31.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.181+0000 7f94d8049b80 1 -- 172.21.15.70:0/4079685767 shutdown_connections 2024-06-04T05:03:31.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.181+0000 7f94d8049b80 1 --2- 172.21.15.70:0/4079685767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94548f60 0x556f945493a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.181+0000 7f94d8049b80 1 --2- 172.21.15.70:0/4079685767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94544860 0x556f94544c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.181+0000 7f94d8049b80 1 --2- 172.21.15.70:0/4079685767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94545210 0x556f94548940 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.181+0000 7f94d8049b80 1 -- 172.21.15.70:0/4079685767 >> 172.21.15.70:0/4079685767 conn(0x556f94472080 msgr2=0x556f9453ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.181+0000 7f94d8049b80 1 -- 172.21.15.70:0/4079685767 shutdown_connections 2024-06-04T05:03:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.181+0000 7f94d8049b80 1 -- 172.21.15.70:0/4079685767 wait complete. 2024-06-04T05:03:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94d8049b80 1 -- start start 2024-06-04T05:03:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94d8049b80 1 Processor -- start 2024-06-04T05:03:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94d8049b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94544860 0x556f94550470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94d8049b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94545210 0x556f9454d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94d8049b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94548f60 0x556f9454dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94d8049b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556f94399fa0 con 0x556f94544860 2024-06-04T05:03:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94d8049b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556f944357c0 con 0x556f94545210 2024-06-04T05:03:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94d8049b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556f94551120 con 0x556f94548f60 2024-06-04T05:03:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94d8049b80 10 client.0 _ll_register_callbacks cb 0x556f9442e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94c6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94544860 0x556f94550470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94c6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94544860 0x556f94550470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43330/0 (socket says 172.21.15.70:43330) 2024-06-04T05:03:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94c6ffd700 1 -- 172.21.15.70:0/202822153 learned_addr learned my addr 172.21.15.70:0/202822153 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94bffff700 1 --2- 172.21.15.70:0/202822153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94545210 0x556f9454d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94c77fe700 1 --2- 172.21.15.70:0/202822153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94548f60 0x556f9454dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94c6ffd700 1 -- 172.21.15.70:0/202822153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94548f60 msgr2=0x556f9454dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94c6ffd700 1 --2- 172.21.15.70:0/202822153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94548f60 0x556f9454dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94c6ffd700 1 -- 172.21.15.70:0/202822153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94545210 msgr2=0x556f9454d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94c6ffd700 1 --2- 172.21.15.70:0/202822153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94545210 0x556f9454d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94c6ffd700 1 -- 172.21.15.70:0/202822153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f94432690 con 0x556f94544860 2024-06-04T05:03:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94bffff700 1 --2- 172.21.15.70:0/202822153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94545210 0x556f9454d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:03:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.182+0000 7f94c77fe700 1 --2- 172.21.15.70:0/202822153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94548f60 0x556f9454dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:03:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.183+0000 7f94c6ffd700 1 --2- 172.21.15.70:0/202822153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94544860 0x556f94550470 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f94b8015000 tx=0x7f94b8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52c42f0650edffd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.183+0000 7f94ae7fc700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.183+0000 7f94ae7fc700 1 -- 172.21.15.70:0/202822153 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b8033060 con 0x556f94544860 2024-06-04T05:03:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.183+0000 7f94ae7fc700 1 -- 172.21.15.70:0/202822153 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f94b800fa70 con 0x556f94544860 2024-06-04T05:03:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.183+0000 7f94d8049b80 1 -- 172.21.15.70:0/202822153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94b00b23d0 con 0x556f94544860 2024-06-04T05:03:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.183+0000 7f94ae7fc700 1 -- 172.21.15.70:0/202822153 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f94b800fc70 con 0x556f94544860 2024-06-04T05:03:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.183+0000 7f94d8049b80 10 client.17130 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.183+0000 7f94d8049b80 1 -- 172.21.15.70:0/202822153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556f9455b8f0 con 0x556f94544860 2024-06-04T05:03:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.183+0000 7f94ae7fc700 1 -- 172.21.15.70:0/202822153 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94b00b23d0 con 0x556f94544860 2024-06-04T05:03:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.183+0000 7f94ae7fc700 1 -- 172.21.15.70:0/202822153 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f94b802e3f0 con 0x556f94544860 2024-06-04T05:03:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.183+0000 7f94d8049b80 10 client.17130 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.183+0000 7f94d8049b80 10 client.17130 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:03:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.183+0000 7f94d8049b80 20 client.17130 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.183+0000 7f94d8049b80 1 --2- 172.21.15.70:0/202822153 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556f9455df80 0x556f945603b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.183+0000 7f94d8049b80 4 client.17130 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:03:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.183+0000 7f94d8049b80 1 -- 172.21.15.70:0/202822153 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f945109e0 con 0x556f9455df80 2024-06-04T05:03:31.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.184+0000 7f94bffff700 1 --2- 172.21.15.70:0/202822153 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556f9455df80 0x556f945603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.184+0000 7f94ae7fc700 10 client.17130 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:03:31.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.184+0000 7f94bffff700 1 --2- 172.21.15.70:0/202822153 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556f9455df80 0x556f945603b0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6ee3820d79523a64 server_cookie=f5b750f3556dcf11 in_seq=0 out_seq=0 2024-06-04T05:03:31.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.185+0000 7f94ae7fc700 1 -- 172.21.15.70:0/202822153 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556f945109e0 con 0x556f9455df80 2024-06-04T05:03:31.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.185+0000 7f94ae7fc700 10 client.17130 handle_command_reply: tid=0 2024-06-04T05:03:31.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.185+0000 7f94d8049b80 1 client.17130 shutdown 2024-06-04T05:03:31.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.185+0000 7f94d8049b80 1 -- 172.21.15.70:0/202822153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94544860 msgr2=0x556f94550470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.185+0000 7f94d8049b80 1 --2- 172.21.15.70:0/202822153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94544860 0x556f94550470 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f94b8015000 tx=0x7f94b8009310 comp rx=0 tx=0).stop 2024-06-04T05:03:31.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.185+0000 7f94d8049b80 1 -- 172.21.15.70:0/202822153 shutdown_connections 2024-06-04T05:03:31.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.185+0000 7f94d8049b80 1 --2- 172.21.15.70:0/202822153 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556f9455df80 0x556f945603b0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.185+0000 7f94d8049b80 1 --2- 172.21.15.70:0/202822153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94548f60 0x556f9454dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.185+0000 7f94d8049b80 1 --2- 172.21.15.70:0/202822153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94544860 0x556f94550470 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.185+0000 7f94d8049b80 1 --2- 172.21.15.70:0/202822153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94545210 0x556f9454d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.185+0000 7f94ae7fc700 0 client.17130 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:03:31.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.185+0000 7f94d8049b80 1 -- 172.21.15.70:0/202822153 >> 172.21.15.70:0/202822153 conn(0x556f94472080 msgr2=0x556f9453e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.185+0000 7f94d8049b80 1 -- 172.21.15.70:0/202822153 shutdown_connections 2024-06-04T05:03:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.185+0000 7f94d8049b80 1 -- 172.21.15.70:0/202822153 wait complete. 2024-06-04T05:03:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.185+0000 7f94d8049b80 20 client.17130 trim_cache size 0 max 16384 2024-06-04T05:03:31.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.200+0000 7f94d8049b80 1 Processor -- start 2024-06-04T05:03:31.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.200+0000 7f94d8049b80 1 -- start start 2024-06-04T05:03:31.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94d8049b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94544860 0x556f9455a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94d8049b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94545210 0x556f94544050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94d8049b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94548f60 0x556f9453ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94d8049b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f94c81add90 con 0x556f94544860 2024-06-04T05:03:31.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94d8049b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f94c81af8d0 con 0x556f94545210 2024-06-04T05:03:31.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94d8049b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f94c81ae460 con 0x556f94548f60 2024-06-04T05:03:31.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94c6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94544860 0x556f9455a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94c6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94544860 0x556f9455a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43342/0 (socket says 172.21.15.70:43342) 2024-06-04T05:03:31.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94c6ffd700 1 -- 172.21.15.70:0/1572437763 learned_addr learned my addr 172.21.15.70:0/1572437763 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:31.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94c77fe700 1 --2- 172.21.15.70:0/1572437763 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94548f60 0x556f9453ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94c6ffd700 1 -- 172.21.15.70:0/1572437763 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94548f60 msgr2=0x556f9453ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:31.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94bffff700 1 --2- 172.21.15.70:0/1572437763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94545210 0x556f94544050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94c6ffd700 1 --2- 172.21.15.70:0/1572437763 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94548f60 0x556f9453ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94c6ffd700 1 -- 172.21.15.70:0/1572437763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94545210 msgr2=0x556f94544050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:31.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94c6ffd700 1 --2- 172.21.15.70:0/1572437763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94545210 0x556f94544050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94c6ffd700 1 -- 172.21.15.70:0/1572437763 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f9454b5a0 con 0x556f94544860 2024-06-04T05:03:31.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94c77fe700 1 --2- 172.21.15.70:0/1572437763 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94548f60 0x556f9453ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:03:31.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94bffff700 1 --2- 172.21.15.70:0/1572437763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94545210 0x556f94544050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:03:31.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94c6ffd700 1 --2- 172.21.15.70:0/1572437763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94544860 0x556f9455a1c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f94b802e7e0 tx=0x7f94b8003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a0ebfe3f4767678 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:31.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94bcff9700 1 -- 172.21.15.70:0/1572437763 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b8045070 con 0x556f94544860 2024-06-04T05:03:31.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94bcff9700 1 -- 172.21.15.70:0/1572437763 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94b8040480 con 0x556f94544860 2024-06-04T05:03:31.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.201+0000 7f94bcff9700 1 -- 172.21.15.70:0/1572437763 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b8033070 con 0x556f94544860 2024-06-04T05:03:31.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.202+0000 7f94d8049b80 1 -- 172.21.15.70:0/1572437763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94544860 msgr2=0x556f9455a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.202+0000 7f94d8049b80 1 --2- 172.21.15.70:0/1572437763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94544860 0x556f9455a1c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f94b802e7e0 tx=0x7f94b8003680 comp rx=0 tx=0).stop 2024-06-04T05:03:31.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.202+0000 7f94d8049b80 1 -- 172.21.15.70:0/1572437763 shutdown_connections 2024-06-04T05:03:31.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.202+0000 7f94d8049b80 1 --2- 172.21.15.70:0/1572437763 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94548f60 0x556f9453ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.202+0000 7f94d8049b80 1 --2- 172.21.15.70:0/1572437763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94544860 0x556f9455a1c0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.202+0000 7f94d8049b80 1 --2- 172.21.15.70:0/1572437763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94545210 0x556f94544050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.202+0000 7f94d8049b80 1 -- 172.21.15.70:0/1572437763 >> 172.21.15.70:0/1572437763 conn(0x556f94472080 msgr2=0x556f9454f830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:31.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.202+0000 7f94d8049b80 1 -- 172.21.15.70:0/1572437763 shutdown_connections 2024-06-04T05:03:31.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.202+0000 7f94d8049b80 1 -- 172.21.15.70:0/1572437763 wait complete. 2024-06-04T05:03:31.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94d8049b80 1 -- start start 2024-06-04T05:03:31.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94d8049b80 1 Processor -- start 2024-06-04T05:03:31.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94d8049b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94544860 0x556f94556890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94d8049b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94545210 0x556f94553c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94d8049b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94548f60 0x556f94554170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94d8049b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f94c81af200 con 0x556f94548f60 2024-06-04T05:03:31.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94d8049b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f94c81aeb30 con 0x556f94544860 2024-06-04T05:03:31.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94d8049b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f94c81abd30 con 0x556f94545210 2024-06-04T05:03:31.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94c77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94548f60 0x556f94554170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94c77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94548f60 0x556f94554170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43356/0 (socket says 172.21.15.70:43356) 2024-06-04T05:03:31.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94c77fe700 1 -- 172.21.15.70:0/854225126 learned_addr learned my addr 172.21.15.70:0/854225126 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:31.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94d8049b80 10 client.0 _ll_register_callbacks cb 0x556f9442e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:31.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94c6ffd700 1 --2- 172.21.15.70:0/854225126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94544860 0x556f94556890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94bffff700 1 --2- 172.21.15.70:0/854225126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94545210 0x556f94553c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94c77fe700 1 -- 172.21.15.70:0/854225126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94545210 msgr2=0x556f94553c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:31.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94c77fe700 1 --2- 172.21.15.70:0/854225126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94545210 0x556f94553c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94c77fe700 1 -- 172.21.15.70:0/854225126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94544860 msgr2=0x556f94556890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:31.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94c77fe700 1 --2- 172.21.15.70:0/854225126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94544860 0x556f94556890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94c77fe700 1 -- 172.21.15.70:0/854225126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f945547b0 con 0x556f94548f60 2024-06-04T05:03:31.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.203+0000 7f94c6ffd700 1 --2- 172.21.15.70:0/854225126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94544860 0x556f94556890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:03:31.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.204+0000 7f94c77fe700 1 --2- 172.21.15.70:0/854225126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94548f60 0x556f94554170 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f94c0008ba0 tx=0x7f94c0008f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0faca457e3863e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:31.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.204+0000 7f94ae7fc700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:31.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.204+0000 7f94ae7fc700 1 -- 172.21.15.70:0/854225126 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c001b020 con 0x556f94548f60 2024-06-04T05:03:31.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.204+0000 7f94ae7fc700 1 -- 172.21.15.70:0/854225126 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f94c0012a70 con 0x556f94548f60 2024-06-04T05:03:31.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.204+0000 7f94d8049b80 1 -- 172.21.15.70:0/854225126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94b8040480 con 0x556f94548f60 2024-06-04T05:03:31.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.204+0000 7f94ae7fc700 1 -- 172.21.15.70:0/854225126 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f94c0026050 con 0x556f94548f60 2024-06-04T05:03:31.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.204+0000 7f94d8049b80 10 client.17136 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:31.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.204+0000 7f94d8049b80 1 -- 172.21.15.70:0/854225126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556f945c89a0 con 0x556f94548f60 2024-06-04T05:03:31.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.204+0000 7f94ae7fc700 1 -- 172.21.15.70:0/854225126 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94b8040480 con 0x556f94548f60 2024-06-04T05:03:31.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.204+0000 7f94ae7fc700 1 -- 172.21.15.70:0/854225126 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f94c00163d0 con 0x556f94548f60 2024-06-04T05:03:31.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.204+0000 7f94d8049b80 10 client.17136 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:31.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.204+0000 7f94d8049b80 10 client.17136 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:03:31.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.204+0000 7f94d8049b80 20 client.17136 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:31.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.204+0000 7f94d8049b80 1 --2- 172.21.15.70:0/854225126 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556f945c90a0 0x556f945cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.204+0000 7f94d8049b80 4 client.17136 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:03:31.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.204+0000 7f94d8049b80 1 -- 172.21.15.70:0/854225126 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556f944a5be0 con 0x556f945c90a0 2024-06-04T05:03:31.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.205+0000 7f94c6ffd700 1 --2- 172.21.15.70:0/854225126 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556f945c90a0 0x556f945cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.205+0000 7f94c6ffd700 1 --2- 172.21.15.70:0/854225126 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556f945c90a0 0x556f945cb4c0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=94ce86f5b2c5494b server_cookie=388af0e871715a01 in_seq=0 out_seq=0 2024-06-04T05:03:31.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.205+0000 7f94ae7fc700 10 client.17136 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:03:31.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.205+0000 7f94ae7fc700 1 -- 172.21.15.70:0/854225126 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556f944a5be0 con 0x556f945c90a0 2024-06-04T05:03:31.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.205+0000 7f94ae7fc700 10 client.17136 handle_command_reply: tid=0 2024-06-04T05:03:31.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.205+0000 7f94d8049b80 1 client.17136 shutdown 2024-06-04T05:03:31.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.205+0000 7f94d8049b80 1 -- 172.21.15.70:0/854225126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94548f60 msgr2=0x556f94554170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.205+0000 7f94d8049b80 1 --2- 172.21.15.70:0/854225126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94548f60 0x556f94554170 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f94c0008ba0 tx=0x7f94c0008f60 comp rx=0 tx=0).stop 2024-06-04T05:03:31.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.205+0000 7f94d8049b80 1 -- 172.21.15.70:0/854225126 shutdown_connections 2024-06-04T05:03:31.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.205+0000 7f94d8049b80 1 --2- 172.21.15.70:0/854225126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f94548f60 0x556f94554170 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.205+0000 7f94d8049b80 1 --2- 172.21.15.70:0/854225126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f94544860 0x556f94556890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.205+0000 7f94d8049b80 1 --2- 172.21.15.70:0/854225126 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x556f945c90a0 0x556f945cb4c0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.205+0000 7f94d8049b80 1 --2- 172.21.15.70:0/854225126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f94545210 0x556f94553c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.205+0000 7f94ae7fc700 0 client.17136 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:03:31.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.205+0000 7f94d8049b80 1 -- 172.21.15.70:0/854225126 >> 172.21.15.70:0/854225126 conn(0x556f94472080 msgr2=0x556f9454efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:31.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.206+0000 7f94d8049b80 1 -- 172.21.15.70:0/854225126 shutdown_connections 2024-06-04T05:03:31.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.206+0000 7f94d8049b80 1 -- 172.21.15.70:0/854225126 wait complete. 2024-06-04T05:03:31.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.206+0000 7f94d8049b80 20 client.17136 trim_cache size 0 max 16384 2024-06-04T05:03:31.213 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:03:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.206+0000 7f94cd7b4700 1 -- 172.21.15.70:0/1240688560 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f94b40b29f0 msgr2=0x7f94b40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.206+0000 7f94cd7b4700 1 --2- 172.21.15.70:0/1240688560 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f94b40b29f0 0x7f94b40b4eb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f94b800b5c0 tx=0x7f94b8005960 comp rx=0 tx=0).stop 2024-06-04T05:03:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.206+0000 7f94cd7b4700 1 -- 172.21.15.70:0/1240688560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94c8102e00 msgr2=0x7f94c819ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.206+0000 7f94cd7b4700 1 --2- 172.21.15.70:0/1240688560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94c8102e00 0x7f94c819ae60 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f94b000d950 tx=0x7f94b000dc60 comp rx=0 tx=0).stop 2024-06-04T05:03:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.206+0000 7f94cd7b4700 1 -- 172.21.15.70:0/1240688560 shutdown_connections 2024-06-04T05:03:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.206+0000 7f94cd7b4700 1 --2- 172.21.15.70:0/1240688560 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f94b40b29f0 0x7f94b40b4eb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.206+0000 7f94cd7b4700 1 --2- 172.21.15.70:0/1240688560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f94c8108520 0x7f94c81a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.206+0000 7f94cd7b4700 1 --2- 172.21.15.70:0/1240688560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c8102440 0x7f94c819a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.206+0000 7f94cd7b4700 1 --2- 172.21.15.70:0/1240688560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94c8102e00 0x7f94c819ae60 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.206+0000 7f94cd7b4700 1 -- 172.21.15.70:0/1240688560 >> 172.21.15.70:0/1240688560 conn(0x7f94c80793f0 msgr2=0x7f94c8101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:31.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.206+0000 7f94cd7b4700 1 -- 172.21.15.70:0/1240688560 shutdown_connections 2024-06-04T05:03:31.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:31.206+0000 7f94cd7b4700 1 -- 172.21.15.70:0/1240688560 wait complete. 2024-06-04T05:03:31.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.607+0000 7f68aa1b6640 1 -- 172.21.15.70:0/3817745994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68a41033f0 msgr2=0x7f68a410f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.607+0000 7f68aa1b6640 1 --2- 172.21.15.70:0/3817745994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68a41033f0 0x7f68a410f900 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f689c009cd0 tx=0x7f689c02a8e0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.607+0000 7f68aa1b6640 1 -- 172.21.15.70:0/3817745994 shutdown_connections 2024-06-04T05:03:31.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.607+0000 7f68aa1b6640 1 --2- 172.21.15.70:0/3817745994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68a4102a30 0x7f68a4102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.607+0000 7f68aa1b6640 1 --2- 172.21.15.70:0/3817745994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68a41033f0 0x7f68a410f900 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.607+0000 7f68aa1b6640 1 --2- 172.21.15.70:0/3817745994 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68a4108a40 0x7f68a4108e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.607+0000 7f68aa1b6640 1 -- 172.21.15.70:0/3817745994 >> 172.21.15.70:0/3817745994 conn(0x7f68a40fe500 msgr2=0x7f68a4100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:31.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.607+0000 7f68aa1b6640 1 -- 172.21.15.70:0/3817745994 shutdown_connections 2024-06-04T05:03:31.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.607+0000 7f68aa1b6640 1 -- 172.21.15.70:0/3817745994 wait complete. 2024-06-04T05:03:31.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.607+0000 7f68aa1b6640 1 Processor -- start 2024-06-04T05:03:31.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.607+0000 7f68aa1b6640 1 -- start start 2024-06-04T05:03:31.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68aa1b6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68a4102a30 0x7f68a4199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68aa1b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68a41033f0 0x7f68a41995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68aa1b6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68a4108a40 0x7f68a419d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68aa1b6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f68a4112bb0 con 0x7f68a4108a40 2024-06-04T05:03:31.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68aa1b6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68a4112a30 con 0x7f68a41033f0 2024-06-04T05:03:31.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68aa1b6640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f68a4112d30 con 0x7f68a4102a30 2024-06-04T05:03:31.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68a3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68a4108a40 0x7f68a419d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68a3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68a4108a40 0x7f68a419d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43386/0 (socket says 172.21.15.70:43386) 2024-06-04T05:03:31.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68a3fff640 1 -- 172.21.15.70:0/1645947997 learned_addr learned my addr 172.21.15.70:0/1645947997 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:31.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68a37fe640 1 --2- 172.21.15.70:0/1645947997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68a4102a30 0x7f68a4199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f689bfff640 1 --2- 172.21.15.70:0/1645947997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68a41033f0 0x7f68a41995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68a3fff640 1 -- 172.21.15.70:0/1645947997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68a4102a30 msgr2=0x7f68a4199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68a3fff640 1 --2- 172.21.15.70:0/1645947997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68a4102a30 0x7f68a4199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68a3fff640 1 -- 172.21.15.70:0/1645947997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68a41033f0 msgr2=0x7f68a41995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68a3fff640 1 --2- 172.21.15.70:0/1645947997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68a41033f0 0x7f68a41995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68a3fff640 1 -- 172.21.15.70:0/1645947997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a419e080 con 0x7f68a4108a40 2024-06-04T05:03:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68a3fff640 1 --2- 172.21.15.70:0/1645947997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68a4108a40 0x7f68a419d950 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f689c006010 tx=0x7f689c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68a17fa640 1 -- 172.21.15.70:0/1645947997 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c004450 con 0x7f68a4108a40 2024-06-04T05:03:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68a17fa640 1 -- 172.21.15.70:0/1645947997 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f689c003e10 con 0x7f68a4108a40 2024-06-04T05:03:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68aa1b6640 1 -- 172.21.15.70:0/1645947997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68a419e310 con 0x7f68a4108a40 2024-06-04T05:03:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68a17fa640 1 -- 172.21.15.70:0/1645947997 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c046480 con 0x7f68a4108a40 2024-06-04T05:03:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.608+0000 7f68aa1b6640 1 -- 172.21.15.70:0/1645947997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68a41a5c00 con 0x7f68a4108a40 2024-06-04T05:03:31.619 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.611+0000 7f68acfaa740 1 Processor -- start 2024-06-04T05:03:31.619 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.611+0000 7f68a17fa640 1 -- 172.21.15.70:0/1645947997 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f689c0305a0 con 0x7f68a4108a40 2024-06-04T05:03:31.619 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.611+0000 7f68acfaa740 1 -- start start 2024-06-04T05:03:31.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.611+0000 7f68acfaa740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565888820 0x55e565888c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.611+0000 7f68acfaa740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565889140 0x55e5658934c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.611+0000 7f68acfaa740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565893a00 0x55e565895df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.611+0000 7f68acfaa740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e5657fac30 con 0x55e565889140 2024-06-04T05:03:31.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.611+0000 7f68acfaa740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e5657ea7d0 con 0x55e565893a00 2024-06-04T05:03:31.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.611+0000 7f68acfaa740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e5657ee810 con 0x55e565888820 2024-06-04T05:03:31.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.611+0000 7f68a37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565889140 0x55e5658934c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.611+0000 7f68a37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565889140 0x55e5658934c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43402/0 (socket says 172.21.15.70:43402) 2024-06-04T05:03:31.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.611+0000 7f68a37fe640 1 -- 172.21.15.70:0/558753943 learned_addr learned my addr 172.21.15.70:0/558753943 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:31.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.612+0000 7f689bfff640 1 --2- 172.21.15.70:0/558753943 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565888820 0x55e565888c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.612+0000 7f689bfff640 1 --2- 172.21.15.70:0/558753943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565893a00 0x55e565895df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.612+0000 7f68a37fe640 1 -- 172.21.15.70:0/558753943 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565888820 msgr2=0x55e565888c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:31.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.612+0000 7f68a37fe640 1 --2- 172.21.15.70:0/558753943 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565888820 0x55e565888c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.612+0000 7f68a37fe640 1 -- 172.21.15.70:0/558753943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565893a00 msgr2=0x55e565895df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:31.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.612+0000 7f68a37fe640 1 --2- 172.21.15.70:0/558753943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565893a00 0x55e565895df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.612+0000 7f68a37fe640 1 -- 172.21.15.70:0/558753943 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5654ff210 con 0x55e565889140 2024-06-04T05:03:31.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.612+0000 7f689bfff640 1 --2- 172.21.15.70:0/558753943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565893a00 0x55e565895df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:03:31.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.612+0000 7f68a17fa640 1 --2- 172.21.15.70:0/1645947997 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68900b2a10 0x7f68900b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.612+0000 7f689bfff640 1 --2- 172.21.15.70:0/1645947997 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68900b2a10 0x7f68900b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.612+0000 7f68a37fe640 1 --2- 172.21.15.70:0/558753943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565889140 0x55e5658934c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f689400ad00 tx=0x7f689400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8291d8785ff04c29 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:31.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.612+0000 7f68a17fa640 1 -- 172.21.15.70:0/1645947997 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f689c02e030 con 0x7f68a4108a40 2024-06-04T05:03:31.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.612+0000 7f6899ffb640 1 -- 172.21.15.70:0/558753943 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689401d070 con 0x55e565889140 2024-06-04T05:03:31.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.612+0000 7f6899ffb640 1 -- 172.21.15.70:0/558753943 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6894018ad0 con 0x55e565889140 2024-06-04T05:03:31.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.612+0000 7f6899ffb640 1 -- 172.21.15.70:0/558753943 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68940109d0 con 0x55e565889140 2024-06-04T05:03:31.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.613+0000 7f68acfaa740 1 -- 172.21.15.70:0/558753943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565889140 msgr2=0x55e5658934c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.613+0000 7f689bfff640 1 --2- 172.21.15.70:0/1645947997 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68900b2a10 0x7f68900b4ed0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x55e565886150 tx=0x7f688c008040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.613+0000 7f68acfaa740 1 --2- 172.21.15.70:0/558753943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565889140 0x55e5658934c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f689400ad00 tx=0x7f689400f790 comp rx=0 tx=0).stop 2024-06-04T05:03:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.613+0000 7f68acfaa740 1 -- 172.21.15.70:0/558753943 shutdown_connections 2024-06-04T05:03:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.613+0000 7f68acfaa740 1 --2- 172.21.15.70:0/558753943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565893a00 0x55e565895df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.613+0000 7f68acfaa740 1 --2- 172.21.15.70:0/558753943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565889140 0x55e5658934c0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.613+0000 7f68acfaa740 1 --2- 172.21.15.70:0/558753943 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565888820 0x55e565888c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.613+0000 7f68acfaa740 1 -- 172.21.15.70:0/558753943 >> 172.21.15.70:0/558753943 conn(0x55e565884d30 msgr2=0x55e565885110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.613+0000 7f68acfaa740 1 -- 172.21.15.70:0/558753943 shutdown_connections 2024-06-04T05:03:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.613+0000 7f68acfaa740 1 -- 172.21.15.70:0/558753943 wait complete. 2024-06-04T05:03:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.613+0000 7f68acfaa740 1 -- start start 2024-06-04T05:03:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.613+0000 7f68acfaa740 1 Processor -- start 2024-06-04T05:03:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68acfaa740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565888820 0x55e5658909e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68acfaa740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565889140 0x55e565890f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68acfaa740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565893a00 0x55e565891460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68acfaa740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e5654d0790 con 0x55e565893a00 2024-06-04T05:03:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68acfaa740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e5657a6890 con 0x55e565888820 2024-06-04T05:03:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68acfaa740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e565896630 con 0x55e565889140 2024-06-04T05:03:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68acfaa740 10 client.0 _ll_register_callbacks cb 0x55e5654fea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68a3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565893a00 0x55e565891460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68a3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565893a00 0x55e565891460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43418/0 (socket says 172.21.15.70:43418) 2024-06-04T05:03:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68a3fff640 1 -- 172.21.15.70:0/703971625 learned_addr learned my addr 172.21.15.70:0/703971625 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68a37fe640 1 --2- 172.21.15.70:0/703971625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565888820 0x55e5658909e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f689bfff640 1 --2- 172.21.15.70:0/703971625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565889140 0x55e565890f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68a3fff640 1 -- 172.21.15.70:0/703971625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565889140 msgr2=0x55e565890f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68a3fff640 1 --2- 172.21.15.70:0/703971625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565889140 0x55e565890f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68a3fff640 1 -- 172.21.15.70:0/703971625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565888820 msgr2=0x55e5658909e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68a3fff640 1 --2- 172.21.15.70:0/703971625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565888820 0x55e5658909e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68a3fff640 1 -- 172.21.15.70:0/703971625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5655cb620 con 0x55e565893a00 2024-06-04T05:03:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68a3fff640 1 --2- 172.21.15.70:0/703971625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565893a00 0x55e565891460 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f689c0c7250 tx=0x7f689c0cb0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a7107d6dbc6684 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f6882ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f6882ffd640 1 -- 172.21.15.70:0/703971625 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c0e5060 con 0x55e565893a00 2024-06-04T05:03:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f6882ffd640 1 -- 172.21.15.70:0/703971625 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f689c0cf170 con 0x55e565893a00 2024-06-04T05:03:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f68acfaa740 1 -- 172.21.15.70:0/703971625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6894018ad0 con 0x55e565893a00 2024-06-04T05:03:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.614+0000 7f6882ffd640 1 -- 172.21.15.70:0/703971625 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f689c0cf370 con 0x55e565893a00 2024-06-04T05:03:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.615+0000 7f68acfaa740 10 client.17148 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.615+0000 7f68acfaa740 1 -- 172.21.15.70:0/703971625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e5658a2380 con 0x55e565893a00 2024-06-04T05:03:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.615+0000 7f6882ffd640 1 -- 172.21.15.70:0/703971625 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6894018ad0 con 0x55e565893a00 2024-06-04T05:03:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.615+0000 7f6882ffd640 1 -- 172.21.15.70:0/703971625 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f689c04e070 con 0x55e565893a00 2024-06-04T05:03:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.615+0000 7f68acfaa740 10 client.17148 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.615+0000 7f68acfaa740 10 client.17148 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.615+0000 7f68acfaa740 20 client.17148 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.615+0000 7f68acfaa740 1 --2- 172.21.15.70:0/703971625 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e5658a3830 0x55e5658a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.615+0000 7f68acfaa740 4 client.17148 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:03:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.615+0000 7f68acfaa740 1 -- 172.21.15.70:0/703971625 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5657fcb80 con 0x55e5658a3830 2024-06-04T05:03:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.615+0000 7f68a37fe640 1 --2- 172.21.15.70:0/703971625 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e5658a3830 0x55e5658a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.616+0000 7f6882ffd640 10 client.17148 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.616+0000 7f68a37fe640 1 --2- 172.21.15.70:0/703971625 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e5658a3830 0x55e5658a5c60 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ee8f1a8a2ffb736 server_cookie=aa70dc1f054639fb in_seq=0 out_seq=0 2024-06-04T05:03:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.616+0000 7f6882ffd640 1 -- 172.21.15.70:0/703971625 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e5657fcb80 con 0x55e5658a3830 2024-06-04T05:03:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.617+0000 7f6882ffd640 10 client.17148 handle_command_reply: tid=0 2024-06-04T05:03:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.617+0000 7f68acfaa740 1 client.17148 shutdown 2024-06-04T05:03:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.617+0000 7f68acfaa740 1 -- 172.21.15.70:0/703971625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565893a00 msgr2=0x55e565891460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.617+0000 7f68acfaa740 1 --2- 172.21.15.70:0/703971625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565893a00 0x55e565891460 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f689c0c7250 tx=0x7f689c0cb0d0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.617+0000 7f68acfaa740 1 -- 172.21.15.70:0/703971625 shutdown_connections 2024-06-04T05:03:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.617+0000 7f68acfaa740 1 --2- 172.21.15.70:0/703971625 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e5658a3830 0x55e5658a5c60 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.617+0000 7f68acfaa740 1 --2- 172.21.15.70:0/703971625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565893a00 0x55e565891460 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.617+0000 7f6882ffd640 0 client.17148 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.617+0000 7f68acfaa740 1 --2- 172.21.15.70:0/703971625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565889140 0x55e565890f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.617+0000 7f68acfaa740 1 --2- 172.21.15.70:0/703971625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565888820 0x55e5658909e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.617+0000 7f68acfaa740 1 -- 172.21.15.70:0/703971625 >> 172.21.15.70:0/703971625 conn(0x55e565884d30 msgr2=0x55e565883fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.617+0000 7f68acfaa740 1 -- 172.21.15.70:0/703971625 shutdown_connections 2024-06-04T05:03:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.617+0000 7f68acfaa740 1 -- 172.21.15.70:0/703971625 wait complete. 2024-06-04T05:03:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.617+0000 7f68acfaa740 20 client.17148 trim_cache size 0 max 16384 2024-06-04T05:03:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.632+0000 7f68acfaa740 1 Processor -- start 2024-06-04T05:03:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.632+0000 7f68acfaa740 1 -- start start 2024-06-04T05:03:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.632+0000 7f68acfaa740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565888820 0x55e56588b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.632+0000 7f68acfaa740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565889140 0x55e56589ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.632+0000 7f68acfaa740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565893a00 0x55e56589f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.632+0000 7f68acfaa740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f68a41ae4c0 con 0x55e565888820 2024-06-04T05:03:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.632+0000 7f68acfaa740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68a41b0000 con 0x55e565893a00 2024-06-04T05:03:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.632+0000 7f68acfaa740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f68a41aeb90 con 0x55e565889140 2024-06-04T05:03:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.632+0000 7f68a37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565888820 0x55e56588b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.632+0000 7f68a37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565888820 0x55e56588b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43420/0 (socket says 172.21.15.70:43420) 2024-06-04T05:03:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.632+0000 7f68a37fe640 1 -- 172.21.15.70:0/1602580072 learned_addr learned my addr 172.21.15.70:0/1602580072 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.632+0000 7f689bfff640 1 --2- 172.21.15.70:0/1602580072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565889140 0x55e56589ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.632+0000 7f68a37fe640 1 -- 172.21.15.70:0/1602580072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565889140 msgr2=0x55e56589ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.632+0000 7f68a37fe640 1 --2- 172.21.15.70:0/1602580072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565889140 0x55e56589ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.633+0000 7f68a37fe640 1 -- 172.21.15.70:0/1602580072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565893a00 msgr2=0x55e56589f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.633+0000 7f68a37fe640 1 --2- 172.21.15.70:0/1602580072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565893a00 0x55e56589f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.633+0000 7f68a37fe640 1 -- 172.21.15.70:0/1602580072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5658806f0 con 0x55e565888820 2024-06-04T05:03:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.633+0000 7f68a37fe640 1 --2- 172.21.15.70:0/1602580072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565888820 0x55e56588b1e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f6894000c00 tx=0x7f689400a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d18225b8297b7b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.633+0000 7f68997fa640 1 -- 172.21.15.70:0/1602580072 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6894032070 con 0x55e565888820 2024-06-04T05:03:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.633+0000 7f68997fa640 1 -- 172.21.15.70:0/1602580072 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f689400a730 con 0x55e565888820 2024-06-04T05:03:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.633+0000 7f68997fa640 1 -- 172.21.15.70:0/1602580072 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689401d040 con 0x55e565888820 2024-06-04T05:03:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.633+0000 7f68acfaa740 1 -- 172.21.15.70:0/1602580072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565888820 msgr2=0x55e56588b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.633+0000 7f68acfaa740 1 --2- 172.21.15.70:0/1602580072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565888820 0x55e56588b1e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f6894000c00 tx=0x7f689400a510 comp rx=0 tx=0).stop 2024-06-04T05:03:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.633+0000 7f68acfaa740 1 -- 172.21.15.70:0/1602580072 shutdown_connections 2024-06-04T05:03:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.633+0000 7f68acfaa740 1 --2- 172.21.15.70:0/1602580072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565893a00 0x55e56589f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.633+0000 7f68acfaa740 1 --2- 172.21.15.70:0/1602580072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565889140 0x55e56589ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.633+0000 7f68acfaa740 1 --2- 172.21.15.70:0/1602580072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565888820 0x55e56588b1e0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.633+0000 7f68acfaa740 1 -- 172.21.15.70:0/1602580072 >> 172.21.15.70:0/1602580072 conn(0x55e565884d30 msgr2=0x55e56589e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.633+0000 7f68acfaa740 1 -- 172.21.15.70:0/1602580072 shutdown_connections 2024-06-04T05:03:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.634+0000 7f68acfaa740 1 -- 172.21.15.70:0/1602580072 wait complete. 2024-06-04T05:03:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.634+0000 7f68acfaa740 1 -- start start 2024-06-04T05:03:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.634+0000 7f68acfaa740 1 Processor -- start 2024-06-04T05:03:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.634+0000 7f68acfaa740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565888820 0x55e5658977c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.634+0000 7f68acfaa740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565889140 0x55e565897d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.634+0000 7f68acfaa740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565893a00 0x55e565898240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.634+0000 7f68acfaa740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f68a41af930 con 0x55e565888820 2024-06-04T05:03:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.634+0000 7f68acfaa740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68a41af260 con 0x55e565893a00 2024-06-04T05:03:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.634+0000 7f68acfaa740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f68a41ac4a0 con 0x55e565889140 2024-06-04T05:03:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.634+0000 7f68acfaa740 10 client.0 _ll_register_callbacks cb 0x7f689c009360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.634+0000 7f68a37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565888820 0x55e5658977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.634+0000 7f68a37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565888820 0x55e5658977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43432/0 (socket says 172.21.15.70:43432) 2024-06-04T05:03:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.634+0000 7f68a37fe640 1 -- 172.21.15.70:0/2486691246 learned_addr learned my addr 172.21.15.70:0/2486691246 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f68a37fe640 1 -- 172.21.15.70:0/2486691246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565889140 msgr2=0x55e565897d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f68a37fe640 1 --2- 172.21.15.70:0/2486691246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565889140 0x55e565897d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f68a37fe640 1 -- 172.21.15.70:0/2486691246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565893a00 msgr2=0x55e565898240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f68a37fe640 1 --2- 172.21.15.70:0/2486691246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565893a00 0x55e565898240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f68a37fe640 1 -- 172.21.15.70:0/2486691246 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e56590d690 con 0x55e565888820 2024-06-04T05:03:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f68a37fe640 1 --2- 172.21.15.70:0/2486691246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565888820 0x55e5658977c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f689401b040 tx=0x7f6894018900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15eb001316f816a0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f6882ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f6882ffd640 1 -- 172.21.15.70:0/2486691246 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689401d030 con 0x55e565888820 2024-06-04T05:03:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f6882ffd640 1 -- 172.21.15.70:0/2486691246 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f6894005040 con 0x55e565888820 2024-06-04T05:03:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f68acfaa740 1 -- 172.21.15.70:0/2486691246 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f689400a730 con 0x55e565888820 2024-06-04T05:03:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f6882ffd640 1 -- 172.21.15.70:0/2486691246 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6894032070 con 0x55e565888820 2024-06-04T05:03:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f68acfaa740 10 client.17154 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f68acfaa740 1 -- 172.21.15.70:0/2486691246 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e56590e940 con 0x55e565888820 2024-06-04T05:03:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f6882ffd640 1 -- 172.21.15.70:0/2486691246 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f689400a730 con 0x55e565888820 2024-06-04T05:03:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f6882ffd640 1 -- 172.21.15.70:0/2486691246 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f6894004c00 con 0x55e565888820 2024-06-04T05:03:31.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f68acfaa740 10 client.17154 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:31.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f68acfaa740 10 client.17154 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:31.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f68acfaa740 20 client.17154 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:31.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f68acfaa740 1 --2- 172.21.15.70:0/2486691246 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e56590ee10 0x55e565911260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:31.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f68acfaa740 4 client.17154 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:03:31.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.635+0000 7f68acfaa740 1 -- 172.21.15.70:0/2486691246 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e5657f6360 con 0x55e56590ee10 2024-06-04T05:03:31.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.636+0000 7f689bfff640 1 --2- 172.21.15.70:0/2486691246 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e56590ee10 0x55e565911260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:31.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.636+0000 7f6882ffd640 10 client.17154 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:31.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.636+0000 7f689bfff640 1 --2- 172.21.15.70:0/2486691246 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e56590ee10 0x55e565911260 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78892bf009244742 server_cookie=4b60564716009970 in_seq=0 out_seq=0 2024-06-04T05:03:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.658+0000 7f6882ffd640 1 -- 172.21.15.70:0/2486691246 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55e5657f6360 con 0x55e56590ee10 2024-06-04T05:03:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.658+0000 7f6882ffd640 10 client.17154 handle_command_reply: tid=0 2024-06-04T05:03:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.658+0000 7f68acfaa740 1 client.17154 shutdown 2024-06-04T05:03:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.658+0000 7f68acfaa740 1 -- 172.21.15.70:0/2486691246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565888820 msgr2=0x55e5658977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.658+0000 7f68acfaa740 1 --2- 172.21.15.70:0/2486691246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565888820 0x55e5658977c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f689401b040 tx=0x7f6894018900 comp rx=0 tx=0).stop 2024-06-04T05:03:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.658+0000 7f68acfaa740 1 -- 172.21.15.70:0/2486691246 shutdown_connections 2024-06-04T05:03:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.658+0000 7f68acfaa740 1 --2- 172.21.15.70:0/2486691246 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e56590ee10 0x55e565911260 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.658+0000 7f68acfaa740 1 --2- 172.21.15.70:0/2486691246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e565893a00 0x55e565898240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.658+0000 7f6882ffd640 0 client.17154 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.658+0000 7f68acfaa740 1 --2- 172.21.15.70:0/2486691246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e565889140 0x55e565897d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.658+0000 7f68acfaa740 1 --2- 172.21.15.70:0/2486691246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e565888820 0x55e5658977c0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.658+0000 7f68acfaa740 1 -- 172.21.15.70:0/2486691246 >> 172.21.15.70:0/2486691246 conn(0x55e565884d30 msgr2=0x55e56589ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.658+0000 7f68acfaa740 1 -- 172.21.15.70:0/2486691246 shutdown_connections 2024-06-04T05:03:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.658+0000 7f68acfaa740 1 -- 172.21.15.70:0/2486691246 wait complete. 2024-06-04T05:03:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.659+0000 7f68acfaa740 20 client.17154 trim_cache size 0 max 16384 2024-06-04T05:03:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:03:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:03:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:03:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.659+0000 7f68aa1b6640 1 -- 172.21.15.70:0/1645947997 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68900b2a10 msgr2=0x7f68900b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.659+0000 7f68aa1b6640 1 --2- 172.21.15.70:0/1645947997 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68900b2a10 0x7f68900b4ed0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x55e565886150 tx=0x7f688c008040 comp rx=0 tx=0).stop 2024-06-04T05:03:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.659+0000 7f68aa1b6640 1 -- 172.21.15.70:0/1645947997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68a4108a40 msgr2=0x7f68a419d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.659+0000 7f68aa1b6640 1 --2- 172.21.15.70:0/1645947997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68a4108a40 0x7f68a419d950 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f689c006010 tx=0x7f689c003c20 comp rx=0 tx=0).stop 2024-06-04T05:03:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.659+0000 7f68aa1b6640 1 -- 172.21.15.70:0/1645947997 shutdown_connections 2024-06-04T05:03:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.659+0000 7f68aa1b6640 1 --2- 172.21.15.70:0/1645947997 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68900b2a10 0x7f68900b4ed0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.659+0000 7f68aa1b6640 1 --2- 172.21.15.70:0/1645947997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68a41033f0 0x7f68a41995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.659+0000 7f68aa1b6640 1 --2- 172.21.15.70:0/1645947997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68a4108a40 0x7f68a419d950 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.659+0000 7f68aa1b6640 1 --2- 172.21.15.70:0/1645947997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68a4102a30 0x7f68a4199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.659+0000 7f68aa1b6640 1 -- 172.21.15.70:0/1645947997 >> 172.21.15.70:0/1645947997 conn(0x7f68a40fe500 msgr2=0x7f68a40fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.659+0000 7f68aa1b6640 1 -- 172.21.15.70:0/1645947997 shutdown_connections 2024-06-04T05:03:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:31.659+0000 7f68aa1b6640 1 -- 172.21.15.70:0/1645947997 wait complete. 2024-06-04T05:03:31.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.23 MB/sec execute 144 sec latency 8218.405 ms 2024-06-04T05:03:31.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:31 smithi070 ceph-mon[32631]: pgmap v193: 97 pgs: 97 active+clean; 68 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 1012 KiB/s wr, 105 op/s 2024-06-04T05:03:31.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:31 smithi112 ceph-mon[36453]: pgmap v193: 97 pgs: 97 active+clean; 68 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 1012 KiB/s wr, 105 op/s 2024-06-04T05:03:31.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:31 smithi098 ceph-mon[35569]: pgmap v193: 97 pgs: 97 active+clean; 68 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 1012 KiB/s wr, 105 op/s 2024-06-04T05:03:32.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:03:32.493 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:03:32.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.21 MB/sec execute 145 sec latency 9218.442 ms 2024-06-04T05:03:33.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:33 smithi070 ceph-mon[32631]: pgmap v194: 97 pgs: 97 active+clean; 68 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 1023 KiB/s wr, 111 op/s 2024-06-04T05:03:33.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.20 MB/sec execute 146 sec latency 10218.478 ms 2024-06-04T05:03:33.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:33 smithi112 ceph-mon[36453]: pgmap v194: 97 pgs: 97 active+clean; 68 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 1023 KiB/s wr, 111 op/s 2024-06-04T05:03:33.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:33 smithi098 ceph-mon[35569]: pgmap v194: 97 pgs: 97 active+clean; 68 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 1023 KiB/s wr, 111 op/s 2024-06-04T05:03:34.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.325+0000 7f8fc2b81700 1 -- 172.21.15.70:0/2254041604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fbc108520 msgr2=0x7f8fbc108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:34.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.325+0000 7f8fc2b81700 1 --2- 172.21.15.70:0/2254041604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fbc108520 0x7f8fbc108900 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f8fb800b0a0 tx=0x7f8fb800b3b0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.325+0000 7f8fc2b81700 1 -- 172.21.15.70:0/2254041604 shutdown_connections 2024-06-04T05:03:34.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.325+0000 7f8fc2b81700 1 --2- 172.21.15.70:0/2254041604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fbc102e00 0x7f8fbc10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.325+0000 7f8fc2b81700 1 --2- 172.21.15.70:0/2254041604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fbc108520 0x7f8fbc108900 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.325+0000 7f8fc2b81700 1 --2- 172.21.15.70:0/2254041604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fbc102440 0x7f8fbc1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.325+0000 7f8fc2b81700 1 -- 172.21.15.70:0/2254041604 >> 172.21.15.70:0/2254041604 conn(0x7f8fbc0793f0 msgr2=0x7f8fbc079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:34.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.325+0000 7f8fc2b81700 1 -- 172.21.15.70:0/2254041604 shutdown_connections 2024-06-04T05:03:34.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.325+0000 7f8fc2b81700 1 -- 172.21.15.70:0/2254041604 wait complete. 2024-06-04T05:03:34.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fc2b81700 1 Processor -- start 2024-06-04T05:03:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fc2b81700 1 -- start start 2024-06-04T05:03:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fc2b81700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fbc102440 0x7f8fbc19f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fc2b81700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fbc102e00 0x7f8fbc19fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fc2b81700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fbc108520 0x7f8fbc199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fc2b81700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc1124a0 con 0x7f8fbc108520 2024-06-04T05:03:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fc2b81700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc112320 con 0x7f8fbc102440 2024-06-04T05:03:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fc2b81700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc112620 con 0x7f8fbc102e00 2024-06-04T05:03:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fc111e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fbc108520 0x7f8fbc199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fc111e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fbc108520 0x7f8fbc199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43448/0 (socket says 172.21.15.70:43448) 2024-06-04T05:03:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fc111e700 1 -- 172.21.15.70:0/305355513 learned_addr learned my addr 172.21.15.70:0/305355513 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fc091d700 1 --2- 172.21.15.70:0/305355513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fbc102440 0x7f8fbc19f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fb3fff700 1 --2- 172.21.15.70:0/305355513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fbc102e00 0x7f8fbc19fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fc111e700 1 -- 172.21.15.70:0/305355513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fbc102e00 msgr2=0x7f8fbc19fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:34.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fc111e700 1 --2- 172.21.15.70:0/305355513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fbc102e00 0x7f8fbc19fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fc111e700 1 -- 172.21.15.70:0/305355513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fbc102440 msgr2=0x7f8fbc19f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:34.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fc111e700 1 --2- 172.21.15.70:0/305355513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fbc102440 0x7f8fbc19f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fc111e700 1 -- 172.21.15.70:0/305355513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fbc19a0b0 con 0x7f8fbc108520 2024-06-04T05:03:34.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.326+0000 7f8fb3fff700 1 --2- 172.21.15.70:0/305355513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fbc102e00 0x7f8fbc19fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:03:34.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.327+0000 7f8fc111e700 1 --2- 172.21.15.70:0/305355513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fbc108520 0x7f8fbc199950 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400d980 tx=0x7f8fb400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:34.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.327+0000 7f8fb1ffb700 1 -- 172.21.15.70:0/305355513 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4017070 con 0x7f8fbc108520 2024-06-04T05:03:34.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.327+0000 7f8fc2b81700 1 -- 172.21.15.70:0/305355513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fbc19a340 con 0x7f8fbc108520 2024-06-04T05:03:34.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.327+0000 7f8fb1ffb700 1 -- 172.21.15.70:0/305355513 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8fb4005020 con 0x7f8fbc108520 2024-06-04T05:03:34.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.327+0000 7f8fb1ffb700 1 -- 172.21.15.70:0/305355513 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4009990 con 0x7f8fbc108520 2024-06-04T05:03:34.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.327+0000 7f8fc2b81700 1 -- 172.21.15.70:0/305355513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fbc1a6660 con 0x7f8fbc108520 2024-06-04T05:03:34.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.329+0000 7f8fb1ffb700 1 -- 172.21.15.70:0/305355513 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f8fb4009b30 con 0x7f8fbc108520 2024-06-04T05:03:34.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.330+0000 7f8fb1ffb700 1 --2- 172.21.15.70:0/305355513 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8fa40b29f0 0x7f8fa40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.330+0000 7f8fc091d700 1 --2- 172.21.15.70:0/305355513 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8fa40b29f0 0x7f8fa40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.330+0000 7f8fb1ffb700 1 -- 172.21.15.70:0/305355513 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f8fb40e2380 con 0x7f8fbc108520 2024-06-04T05:03:34.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.330+0000 7f8fc091d700 1 --2- 172.21.15.70:0/305355513 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8fa40b29f0 0x7f8fa40b4eb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8009e60 tx=0x7f8fb800b7b0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:34.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.330+0000 7f8fcd416b80 1 Processor -- start 2024-06-04T05:03:34.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fcd416b80 1 -- start start 2024-06-04T05:03:34.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fcd416b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef2fc70 0x55e2eef30050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fcd416b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef30620 0x55e2eef33d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fcd416b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef34370 0x55e2eef347b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fcd416b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e2eee8ac90 con 0x55e2eef30620 2024-06-04T05:03:34.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fcd416b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e2eee7a830 con 0x55e2eef2fc70 2024-06-04T05:03:34.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fcd416b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e2eee7e870 con 0x55e2eef34370 2024-06-04T05:03:34.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fb3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef30620 0x55e2eef33d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fb3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef30620 0x55e2eef33d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43452/0 (socket says 172.21.15.70:43452) 2024-06-04T05:03:34.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fb3fff700 1 -- 172.21.15.70:0/3166005247 learned_addr learned my addr 172.21.15.70:0/3166005247 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:34.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fc111e700 1 --2- 172.21.15.70:0/3166005247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef34370 0x55e2eef347b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fc091d700 1 --2- 172.21.15.70:0/3166005247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef2fc70 0x55e2eef30050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fb3fff700 1 -- 172.21.15.70:0/3166005247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef34370 msgr2=0x55e2eef347b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:34.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fb3fff700 1 --2- 172.21.15.70:0/3166005247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef34370 0x55e2eef347b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fb3fff700 1 -- 172.21.15.70:0/3166005247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef2fc70 msgr2=0x55e2eef30050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:34.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fb3fff700 1 --2- 172.21.15.70:0/3166005247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef2fc70 0x55e2eef30050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fb3fff700 1 -- 172.21.15.70:0/3166005247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2eedca8d0 con 0x55e2eef30620 2024-06-04T05:03:34.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.331+0000 7f8fb3fff700 1 --2- 172.21.15.70:0/3166005247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef30620 0x55e2eef33d50 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8fac009cb0 tx=0x7f8fac00e2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb724d30cc8d82b1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:34.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.332+0000 7f8fa2ffd700 1 -- 172.21.15.70:0/3166005247 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac016070 con 0x55e2eef30620 2024-06-04T05:03:34.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.332+0000 7f8fa2ffd700 1 -- 172.21.15.70:0/3166005247 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8fac011470 con 0x55e2eef30620 2024-06-04T05:03:34.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.332+0000 7f8fa2ffd700 1 -- 172.21.15.70:0/3166005247 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac010630 con 0x55e2eef30620 2024-06-04T05:03:34.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.332+0000 7f8fcd416b80 1 -- 172.21.15.70:0/3166005247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef30620 msgr2=0x55e2eef33d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:34.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.332+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/3166005247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef30620 0x55e2eef33d50 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8fac009cb0 tx=0x7f8fac00e2f0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.332+0000 7f8fcd416b80 1 -- 172.21.15.70:0/3166005247 shutdown_connections 2024-06-04T05:03:34.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.332+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/3166005247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef34370 0x55e2eef347b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.332+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/3166005247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef2fc70 0x55e2eef30050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.332+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/3166005247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef30620 0x55e2eef33d50 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.332+0000 7f8fcd416b80 1 -- 172.21.15.70:0/3166005247 >> 172.21.15.70:0/3166005247 conn(0x55e2eee59080 msgr2=0x55e2eef25e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:34.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.332+0000 7f8fcd416b80 1 -- 172.21.15.70:0/3166005247 shutdown_connections 2024-06-04T05:03:34.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.332+0000 7f8fcd416b80 1 -- 172.21.15.70:0/3166005247 wait complete. 2024-06-04T05:03:34.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.333+0000 7f8fcd416b80 1 -- start start 2024-06-04T05:03:34.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.333+0000 7f8fcd416b80 1 Processor -- start 2024-06-04T05:03:34.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.333+0000 7f8fcd416b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef2fc70 0x55e2eef3b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.333+0000 7f8fcd416b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef30620 0x55e2eef38b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.333+0000 7f8fcd416b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef34370 0x55e2eef39070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.333+0000 7f8fcd416b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e2eed80fa0 con 0x55e2eef30620 2024-06-04T05:03:34.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.333+0000 7f8fcd416b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e2eee1c7c0 con 0x55e2eef2fc70 2024-06-04T05:03:34.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.333+0000 7f8fcd416b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e2eef3c530 con 0x55e2eef34370 2024-06-04T05:03:34.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.333+0000 7f8fcd416b80 10 client.0 _ll_register_callbacks cb 0x55e2eee15920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:34.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.333+0000 7f8fb3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef30620 0x55e2eef38b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.333+0000 7f8fb3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef30620 0x55e2eef38b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43468/0 (socket says 172.21.15.70:43468) 2024-06-04T05:03:34.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.333+0000 7f8fb3fff700 1 -- 172.21.15.70:0/2522321626 learned_addr learned my addr 172.21.15.70:0/2522321626 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:34.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.333+0000 7f8fc111e700 1 --2- 172.21.15.70:0/2522321626 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef34370 0x55e2eef39070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.334+0000 7f8fc091d700 1 --2- 172.21.15.70:0/2522321626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef2fc70 0x55e2eef3b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.334+0000 7f8fb3fff700 1 -- 172.21.15.70:0/2522321626 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef34370 msgr2=0x55e2eef39070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:34.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.334+0000 7f8fb3fff700 1 --2- 172.21.15.70:0/2522321626 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef34370 0x55e2eef39070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.334+0000 7f8fb3fff700 1 -- 172.21.15.70:0/2522321626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef2fc70 msgr2=0x55e2eef3b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:34.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.334+0000 7f8fb3fff700 1 --2- 172.21.15.70:0/2522321626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef2fc70 0x55e2eef3b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.334+0000 7f8fb3fff700 1 -- 172.21.15.70:0/2522321626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2eee19690 con 0x55e2eef30620 2024-06-04T05:03:34.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.334+0000 7f8fc091d700 1 --2- 172.21.15.70:0/2522321626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef2fc70 0x55e2eef3b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:03:34.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.334+0000 7f8fb3fff700 1 --2- 172.21.15.70:0/2522321626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef30620 0x55e2eef38b30 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8fac011bb0 tx=0x7f8fac005ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcd8c6c14461eb75 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:34.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.334+0000 7f8f9bfff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:34.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.334+0000 7f8f9bfff700 1 -- 172.21.15.70:0/2522321626 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac016030 con 0x55e2eef30620 2024-06-04T05:03:34.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.334+0000 7f8f9bfff700 1 -- 172.21.15.70:0/2522321626 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f8fac00ed80 con 0x55e2eef30620 2024-06-04T05:03:34.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.334+0000 7f8fcd416b80 1 -- 172.21.15.70:0/2522321626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fac011470 con 0x55e2eef30620 2024-06-04T05:03:34.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.334+0000 7f8f9bfff700 1 -- 172.21.15.70:0/2522321626 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8fac005050 con 0x55e2eef30620 2024-06-04T05:03:34.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.334+0000 7f8fcd416b80 10 client.17166 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:34.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.334+0000 7f8fcd416b80 1 -- 172.21.15.70:0/2522321626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e2eef46c90 con 0x55e2eef30620 2024-06-04T05:03:34.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.334+0000 7f8f9bfff700 1 -- 172.21.15.70:0/2522321626 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8fac011470 con 0x55e2eef30620 2024-06-04T05:03:34.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.334+0000 7f8f9bfff700 1 -- 172.21.15.70:0/2522321626 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f8fac00f040 con 0x55e2eef30620 2024-06-04T05:03:34.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.335+0000 7f8fcd416b80 10 client.17166 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:34.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.335+0000 7f8fcd416b80 10 client.17166 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:03:34.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.335+0000 7f8fcd416b80 20 client.17166 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:34.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.335+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/2522321626 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55e2eef491c0 0x55e2eef4b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.335+0000 7f8fcd416b80 4 client.17166 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:03:34.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.335+0000 7f8fcd416b80 1 -- 172.21.15.70:0/2522321626 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2eeef79e0 con 0x55e2eef491c0 2024-06-04T05:03:34.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.335+0000 7f8fc091d700 1 --2- 172.21.15.70:0/2522321626 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55e2eef491c0 0x55e2eef4b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.335+0000 7f8fc091d700 1 --2- 172.21.15.70:0/2522321626 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55e2eef491c0 0x55e2eef4b5b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=76693eec15a0b5ec server_cookie=6c284517df03ca5 in_seq=0 out_seq=0 2024-06-04T05:03:34.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.335+0000 7f8f9bfff700 10 client.17166 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:03:34.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.336+0000 7f8f9bfff700 1 -- 172.21.15.70:0/2522321626 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e2eeef79e0 con 0x55e2eef491c0 2024-06-04T05:03:34.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.336+0000 7f8f9bfff700 10 client.17166 handle_command_reply: tid=0 2024-06-04T05:03:34.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.336+0000 7f8fcd416b80 1 client.17166 shutdown 2024-06-04T05:03:34.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.336+0000 7f8fcd416b80 1 -- 172.21.15.70:0/2522321626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef30620 msgr2=0x55e2eef38b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:34.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.336+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/2522321626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef30620 0x55e2eef38b30 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8fac011bb0 tx=0x7f8fac005ee0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.336+0000 7f8fcd416b80 1 -- 172.21.15.70:0/2522321626 shutdown_connections 2024-06-04T05:03:34.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.336+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/2522321626 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef34370 0x55e2eef39070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.336+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/2522321626 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55e2eef491c0 0x55e2eef4b5b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.336+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/2522321626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef2fc70 0x55e2eef3b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.336+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/2522321626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef30620 0x55e2eef38b30 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.336+0000 7f8f9bfff700 0 client.17166 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:03:34.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.336+0000 7f8fcd416b80 1 -- 172.21.15.70:0/2522321626 >> 172.21.15.70:0/2522321626 conn(0x55e2eee59080 msgr2=0x55e2eef35760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:34.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.336+0000 7f8fcd416b80 1 -- 172.21.15.70:0/2522321626 shutdown_connections 2024-06-04T05:03:34.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.336+0000 7f8fcd416b80 1 -- 172.21.15.70:0/2522321626 wait complete. 2024-06-04T05:03:34.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.336+0000 7f8fcd416b80 20 client.17166 trim_cache size 0 max 16384 2024-06-04T05:03:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.351+0000 7f8fcd416b80 1 Processor -- start 2024-06-04T05:03:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.351+0000 7f8fcd416b80 1 -- start start 2024-06-04T05:03:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fcd416b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef2fc70 0x55e2eef3aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fcd416b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef30620 0x55e2eef25c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fcd416b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef34370 0x55e2eef50cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fcd416b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc1abb70 con 0x55e2eef34370 2024-06-04T05:03:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fcd416b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc1ad6b0 con 0x55e2eef30620 2024-06-04T05:03:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fcd416b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc1ac240 con 0x55e2eef2fc70 2024-06-04T05:03:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fc111e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef34370 0x55e2eef50cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fc111e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef34370 0x55e2eef50cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43482/0 (socket says 172.21.15.70:43482) 2024-06-04T05:03:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fc111e700 1 -- 172.21.15.70:0/1297064767 learned_addr learned my addr 172.21.15.70:0/1297064767 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:34.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fb3fff700 1 --2- 172.21.15.70:0/1297064767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef30620 0x55e2eef25c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fc091d700 1 --2- 172.21.15.70:0/1297064767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef2fc70 0x55e2eef3aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fc111e700 1 -- 172.21.15.70:0/1297064767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef2fc70 msgr2=0x55e2eef3aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fc111e700 1 --2- 172.21.15.70:0/1297064767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef2fc70 0x55e2eef3aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fc111e700 1 -- 172.21.15.70:0/1297064767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef30620 msgr2=0x55e2eef25c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fc111e700 1 --2- 172.21.15.70:0/1297064767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef30620 0x55e2eef25c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fc111e700 1 -- 172.21.15.70:0/1297064767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2eef302a0 con 0x55e2eef34370 2024-06-04T05:03:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fb3fff700 1 --2- 172.21.15.70:0/1297064767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef30620 0x55e2eef25c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:03:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fc111e700 1 --2- 172.21.15.70:0/1297064767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef34370 0x55e2eef50cd0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4009520 tx=0x7f8fb40a5650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bba33d1cc59dbfc server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fa27fc700 1 -- 172.21.15.70:0/1297064767 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb40c2070 con 0x55e2eef34370 2024-06-04T05:03:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fa27fc700 1 -- 172.21.15.70:0/1297064767 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8fb40ada10 con 0x55e2eef34370 2024-06-04T05:03:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.352+0000 7f8fa27fc700 1 -- 172.21.15.70:0/1297064767 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb40a98c0 con 0x55e2eef34370 2024-06-04T05:03:34.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.353+0000 7f8fcd416b80 1 -- 172.21.15.70:0/1297064767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef34370 msgr2=0x55e2eef50cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.353+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/1297064767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef34370 0x55e2eef50cd0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4009520 tx=0x7f8fb40a5650 comp rx=0 tx=0).stop 2024-06-04T05:03:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.353+0000 7f8fcd416b80 1 -- 172.21.15.70:0/1297064767 shutdown_connections 2024-06-04T05:03:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.353+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/1297064767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef34370 0x55e2eef50cd0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.353+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/1297064767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef2fc70 0x55e2eef3aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.353+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/1297064767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef30620 0x55e2eef25c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.353+0000 7f8fcd416b80 1 -- 172.21.15.70:0/1297064767 >> 172.21.15.70:0/1297064767 conn(0x55e2eee59080 msgr2=0x55e2eef3a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.353+0000 7f8fcd416b80 1 -- 172.21.15.70:0/1297064767 shutdown_connections 2024-06-04T05:03:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.353+0000 7f8fcd416b80 1 -- 172.21.15.70:0/1297064767 wait complete. 2024-06-04T05:03:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.353+0000 7f8fcd416b80 1 -- start start 2024-06-04T05:03:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fcd416b80 1 Processor -- start 2024-06-04T05:03:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fcd416b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef2fc70 0x55e2eef41d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fcd416b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef30620 0x55e2eef31530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fcd416b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef34370 0x55e2eef31a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fcd416b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc1acfe0 con 0x55e2eef2fc70 2024-06-04T05:03:34.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fcd416b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc1ac910 con 0x55e2eef30620 2024-06-04T05:03:34.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fcd416b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc1a9b10 con 0x55e2eef34370 2024-06-04T05:03:34.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fcd416b80 10 client.0 _ll_register_callbacks cb 0x55e2eee15920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:34.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fc091d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef2fc70 0x55e2eef41d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fc091d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef2fc70 0x55e2eef41d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43486/0 (socket says 172.21.15.70:43486) 2024-06-04T05:03:34.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fc091d700 1 -- 172.21.15.70:0/1196721479 learned_addr learned my addr 172.21.15.70:0/1196721479 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:34.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fb3fff700 1 --2- 172.21.15.70:0/1196721479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef30620 0x55e2eef31530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fc111e700 1 --2- 172.21.15.70:0/1196721479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef34370 0x55e2eef31a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fc091d700 1 -- 172.21.15.70:0/1196721479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef34370 msgr2=0x55e2eef31a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:34.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fc091d700 1 --2- 172.21.15.70:0/1196721479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef34370 0x55e2eef31a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fc091d700 1 -- 172.21.15.70:0/1196721479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef30620 msgr2=0x55e2eef31530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:34.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fc091d700 1 --2- 172.21.15.70:0/1196721479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef30620 0x55e2eef31530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fc091d700 1 -- 172.21.15.70:0/1196721479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2eef32110 con 0x55e2eef2fc70 2024-06-04T05:03:34.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.354+0000 7f8fb3fff700 1 --2- 172.21.15.70:0/1196721479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef30620 0x55e2eef31530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:03:34.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.355+0000 7f8fc091d700 1 --2- 172.21.15.70:0/1196721479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef2fc70 0x55e2eef41d70 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f8fb80091e0 tx=0x7f8fb800b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b54a06b148d6d34 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:34.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.355+0000 7f8f9bfff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:34.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.355+0000 7f8f9bfff700 1 -- 172.21.15.70:0/1196721479 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8039060 con 0x55e2eef2fc70 2024-06-04T05:03:34.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.355+0000 7f8f9bfff700 1 -- 172.21.15.70:0/1196721479 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f8fb8007c40 con 0x55e2eef2fc70 2024-06-04T05:03:34.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.355+0000 7f8fcd416b80 1 -- 172.21.15.70:0/1196721479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fb40ada10 con 0x55e2eef2fc70 2024-06-04T05:03:34.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.355+0000 7f8f9bfff700 1 -- 172.21.15.70:0/1196721479 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8fb8033440 con 0x55e2eef2fc70 2024-06-04T05:03:34.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.355+0000 7f8fcd416b80 10 client.17172 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:34.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.355+0000 7f8fcd416b80 1 -- 172.21.15.70:0/1196721479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e2eefb3a00 con 0x55e2eef2fc70 2024-06-04T05:03:34.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.355+0000 7f8f9bfff700 1 -- 172.21.15.70:0/1196721479 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8fb40ada10 con 0x55e2eef2fc70 2024-06-04T05:03:34.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.355+0000 7f8f9bfff700 1 -- 172.21.15.70:0/1196721479 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f8fb8011070 con 0x55e2eef2fc70 2024-06-04T05:03:34.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.355+0000 7f8fcd416b80 10 client.17172 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.355+0000 7f8fcd416b80 10 client.17172 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:03:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.355+0000 7f8fcd416b80 20 client.17172 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.356+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/1196721479 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55e2eefb40f0 0x55e2eefb6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.356+0000 7f8fcd416b80 4 client.17172 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:03:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.356+0000 7f8fcd416b80 1 -- 172.21.15.70:0/1196721479 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e2eee8cbe0 con 0x55e2eefb40f0 2024-06-04T05:03:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.356+0000 7f8fb3fff700 1 --2- 172.21.15.70:0/1196721479 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55e2eefb40f0 0x55e2eefb6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.356+0000 7f8fb3fff700 1 --2- 172.21.15.70:0/1196721479 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55e2eefb40f0 0x55e2eefb6510 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=57d4477f03f40866 server_cookie=c782d7f320572261 in_seq=0 out_seq=0 2024-06-04T05:03:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.356+0000 7f8f9bfff700 10 client.17172 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:03:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.356+0000 7f8f9bfff700 1 -- 172.21.15.70:0/1196721479 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e2eee8cbe0 con 0x55e2eefb40f0 2024-06-04T05:03:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.356+0000 7f8f9bfff700 10 client.17172 handle_command_reply: tid=0 2024-06-04T05:03:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.356+0000 7f8fcd416b80 1 client.17172 shutdown 2024-06-04T05:03:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.357+0000 7f8fcd416b80 1 -- 172.21.15.70:0/1196721479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef2fc70 msgr2=0x55e2eef41d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.357+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/1196721479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef2fc70 0x55e2eef41d70 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f8fb80091e0 tx=0x7f8fb800b3b0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.357+0000 7f8fcd416b80 1 -- 172.21.15.70:0/1196721479 shutdown_connections 2024-06-04T05:03:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.357+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/1196721479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2eef34370 0x55e2eef31a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.357+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/1196721479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2eef2fc70 0x55e2eef41d70 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.357+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/1196721479 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55e2eefb40f0 0x55e2eefb6510 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.357+0000 7f8fcd416b80 1 --2- 172.21.15.70:0/1196721479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2eef30620 0x55e2eef31530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.357+0000 7f8f9bfff700 0 client.17172 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:03:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.357+0000 7f8fcd416b80 1 -- 172.21.15.70:0/1196721479 >> 172.21.15.70:0/1196721479 conn(0x55e2eee59080 msgr2=0x55e2eef2af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.357+0000 7f8fcd416b80 1 -- 172.21.15.70:0/1196721479 shutdown_connections 2024-06-04T05:03:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.357+0000 7f8fcd416b80 1 -- 172.21.15.70:0/1196721479 wait complete. 2024-06-04T05:03:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.357+0000 7f8fcd416b80 20 client.17172 trim_cache size 0 max 16384 2024-06-04T05:03:34.365 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:03:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.358+0000 7f8fc2b81700 1 -- 172.21.15.70:0/305355513 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8fa40b29f0 msgr2=0x7f8fa40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.358+0000 7f8fc2b81700 1 --2- 172.21.15.70:0/305355513 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8fa40b29f0 0x7f8fa40b4eb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8009e60 tx=0x7f8fb800b7b0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.358+0000 7f8fc2b81700 1 -- 172.21.15.70:0/305355513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fbc108520 msgr2=0x7f8fbc199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:34.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.358+0000 7f8fc2b81700 1 --2- 172.21.15.70:0/305355513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fbc108520 0x7f8fbc199950 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400d980 tx=0x7f8fb400dc90 comp rx=0 tx=0).stop 2024-06-04T05:03:34.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.358+0000 7f8fc2b81700 1 -- 172.21.15.70:0/305355513 shutdown_connections 2024-06-04T05:03:34.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.358+0000 7f8fc2b81700 1 --2- 172.21.15.70:0/305355513 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8fa40b29f0 0x7f8fa40b4eb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.358+0000 7f8fc2b81700 1 --2- 172.21.15.70:0/305355513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fbc108520 0x7f8fbc199950 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.358+0000 7f8fc2b81700 1 --2- 172.21.15.70:0/305355513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fbc102440 0x7f8fbc19f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.358+0000 7f8fc2b81700 1 --2- 172.21.15.70:0/305355513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fbc102e00 0x7f8fbc19fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:34.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.358+0000 7f8fc2b81700 1 -- 172.21.15.70:0/305355513 >> 172.21.15.70:0/305355513 conn(0x7f8fbc0793f0 msgr2=0x7f8fbc106850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:34.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.358+0000 7f8fc2b81700 1 -- 172.21.15.70:0/305355513 shutdown_connections 2024-06-04T05:03:34.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:34.358+0000 7f8fc2b81700 1 -- 172.21.15.70:0/305355513 wait complete. 2024-06-04T05:03:34.684 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:03:34.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.18 MB/sec execute 147 sec latency 11218.518 ms 2024-06-04T05:03:35.164 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:03:35.256 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:03:35.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:35 smithi070 ceph-mon[32631]: pgmap v195: 97 pgs: 97 active+clean; 68 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 277 KiB/s wr, 45 op/s 2024-06-04T05:03:35.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.17 MB/sec execute 148 sec latency 12218.543 ms 2024-06-04T05:03:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:35 smithi112 ceph-mon[36453]: pgmap v195: 97 pgs: 97 active+clean; 68 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 277 KiB/s wr, 45 op/s 2024-06-04T05:03:35.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:35 smithi098 ceph-mon[35569]: pgmap v195: 97 pgs: 97 active+clean; 68 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 277 KiB/s wr, 45 op/s 2024-06-04T05:03:36.517 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:03:36.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.15 MB/sec execute 149 sec latency 13218.579 ms 2024-06-04T05:03:37.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.14 MB/sec execute 150 sec latency 14218.617 ms 2024-06-04T05:03:37.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:37 smithi070 ceph-mon[32631]: pgmap v196: 97 pgs: 97 active+clean; 69 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 292 KiB/s wr, 53 op/s 2024-06-04T05:03:37.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:37 smithi112 ceph-mon[36453]: pgmap v196: 97 pgs: 97 active+clean; 69 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 292 KiB/s wr, 53 op/s 2024-06-04T05:03:37.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:37 smithi098 ceph-mon[35569]: pgmap v196: 97 pgs: 97 active+clean; 69 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 292 KiB/s wr, 53 op/s 2024-06-04T05:03:38.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.13 MB/sec execute 151 sec latency 15218.651 ms 2024-06-04T05:03:39.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.323+0000 7f30f1975700 1 -- 172.21.15.70:0/2526180465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30ec06bab0 msgr2=0x7f30ec06bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:39.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.323+0000 7f30f1975700 1 --2- 172.21.15.70:0/2526180465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30ec06bab0 0x7f30ec06bf30 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f30e0009990 tx=0x7f30e0009ca0 comp rx=0 tx=0).stop 2024-06-04T05:03:39.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.323+0000 7f30f1975700 1 -- 172.21.15.70:0/2526180465 shutdown_connections 2024-06-04T05:03:39.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.323+0000 7f30f1975700 1 --2- 172.21.15.70:0/2526180465 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30ec06c470 0x7f30ec1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:39.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.323+0000 7f30f1975700 1 --2- 172.21.15.70:0/2526180465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ec1095a0 0x7f30ec109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:39.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.323+0000 7f30f1975700 1 --2- 172.21.15.70:0/2526180465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30ec06bab0 0x7f30ec06bf30 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:39.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.324+0000 7f30f1975700 1 -- 172.21.15.70:0/2526180465 >> 172.21.15.70:0/2526180465 conn(0x7f30ec0793b0 msgr2=0x7f30ec0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:39.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.324+0000 7f30f1975700 1 -- 172.21.15.70:0/2526180465 shutdown_connections 2024-06-04T05:03:39.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.324+0000 7f30f1975700 1 -- 172.21.15.70:0/2526180465 wait complete. 2024-06-04T05:03:39.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.324+0000 7f30f1975700 1 Processor -- start 2024-06-04T05:03:39.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.325+0000 7f30f1975700 1 -- start start 2024-06-04T05:03:39.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.325+0000 7f30f1975700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ec06bab0 0x7f30ec19f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:39.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.325+0000 7f30f1975700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30ec06c470 0x7f30ec19fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:39.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.325+0000 7f30f1975700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30ec1095a0 0x7f30ec199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:39.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.325+0000 7f30f1975700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec113660 con 0x7f30ec1095a0 2024-06-04T05:03:39.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.325+0000 7f30f1975700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec1134e0 con 0x7f30ec06bab0 2024-06-04T05:03:39.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.325+0000 7f30f1975700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec1137e0 con 0x7f30ec06c470 2024-06-04T05:03:39.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.325+0000 7f30eb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30ec1095a0 0x7f30ec199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:39.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.325+0000 7f30eb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30ec1095a0 0x7f30ec199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48732/0 (socket says 172.21.15.70:48732) 2024-06-04T05:03:39.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.325+0000 7f30eb7fe700 1 -- 172.21.15.70:0/1434332708 learned_addr learned my addr 172.21.15.70:0/1434332708 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:39.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.325+0000 7f30ea7fc700 1 --2- 172.21.15.70:0/1434332708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30ec06c470 0x7f30ec19fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:39.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.325+0000 7f30eaffd700 1 --2- 172.21.15.70:0/1434332708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ec06bab0 0x7f30ec19f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:39.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.325+0000 7f30eb7fe700 1 -- 172.21.15.70:0/1434332708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30ec06c470 msgr2=0x7f30ec19fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:39.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.325+0000 7f30eb7fe700 1 --2- 172.21.15.70:0/1434332708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30ec06c470 0x7f30ec19fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:39.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.325+0000 7f30eb7fe700 1 -- 172.21.15.70:0/1434332708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ec06bab0 msgr2=0x7f30ec19f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:39.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.325+0000 7f30eb7fe700 1 --2- 172.21.15.70:0/1434332708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ec06bab0 0x7f30ec19f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:39.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.325+0000 7f30eb7fe700 1 -- 172.21.15.70:0/1434332708 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ec19a200 con 0x7f30ec1095a0 2024-06-04T05:03:39.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.326+0000 7f30eb7fe700 1 --2- 172.21.15.70:0/1434332708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30ec1095a0 0x7f30ec199aa0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f30e400b970 tx=0x7f30e400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:39.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.326+0000 7f30f0973700 1 -- 172.21.15.70:0/1434332708 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e400d860 con 0x7f30ec1095a0 2024-06-04T05:03:39.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.326+0000 7f30f1975700 1 -- 172.21.15.70:0/1434332708 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30ec19a4f0 con 0x7f30ec1095a0 2024-06-04T05:03:39.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.326+0000 7f30f0973700 1 -- 172.21.15.70:0/1434332708 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f30e400e430 con 0x7f30ec1095a0 2024-06-04T05:03:39.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.326+0000 7f30f0973700 1 -- 172.21.15.70:0/1434332708 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e4017660 con 0x7f30ec1095a0 2024-06-04T05:03:39.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.326+0000 7f30f1975700 1 -- 172.21.15.70:0/1434332708 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30ec1a67b0 con 0x7f30ec1095a0 2024-06-04T05:03:39.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.327+0000 7f30f1975700 1 -- 172.21.15.70:0/1434332708 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30ec10ba80 con 0x7f30ec1095a0 2024-06-04T05:03:39.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.328+0000 7f30f0973700 1 -- 172.21.15.70:0/1434332708 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f30e400da00 con 0x7f30ec1095a0 2024-06-04T05:03:39.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.329+0000 7f30f0973700 1 --2- 172.21.15.70:0/1434332708 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30d40b29f0 0x7f30d40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:39.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.329+0000 7f30eaffd700 1 --2- 172.21.15.70:0/1434332708 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30d40b29f0 0x7f30d40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:39.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.329+0000 7f30f0973700 1 -- 172.21.15.70:0/1434332708 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f30e40e2380 con 0x7f30ec1095a0 2024-06-04T05:03:39.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.329+0000 7f30eaffd700 1 --2- 172.21.15.70:0/1434332708 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30d40b29f0 0x7f30d40b4eb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f30ec19abe0 tx=0x7f30dc009190 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:39.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.332+0000 7f30f0973700 1 -- 172.21.15.70:0/1434332708 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f30e4053df0 con 0x7f30ec1095a0 2024-06-04T05:03:39.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.589+0000 7f30f1975700 1 -- 172.21.15.70:0/1434332708 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30ec19afd0 con 0x7f30ec1095a0 2024-06-04T05:03:39.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.590+0000 7f30f0973700 1 -- 172.21.15.70:0/1434332708 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11929 (secure 0 0 0) 0x7f30e4013030 con 0x7f30ec1095a0 2024-06-04T05:03:39.592 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:03:39.592 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:01:52.807483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:03:39.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.592+0000 7f30f1975700 1 -- 172.21.15.70:0/1434332708 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30d40b29f0 msgr2=0x7f30d40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:39.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.592+0000 7f30f1975700 1 --2- 172.21.15.70:0/1434332708 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30d40b29f0 0x7f30d40b4eb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f30ec19abe0 tx=0x7f30dc009190 comp rx=0 tx=0).stop 2024-06-04T05:03:39.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.592+0000 7f30f1975700 1 -- 172.21.15.70:0/1434332708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30ec1095a0 msgr2=0x7f30ec199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:39.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.592+0000 7f30f1975700 1 --2- 172.21.15.70:0/1434332708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30ec1095a0 0x7f30ec199aa0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f30e400b970 tx=0x7f30e400bc80 comp rx=0 tx=0).stop 2024-06-04T05:03:39.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.592+0000 7f30f1975700 1 -- 172.21.15.70:0/1434332708 shutdown_connections 2024-06-04T05:03:39.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.592+0000 7f30f1975700 1 --2- 172.21.15.70:0/1434332708 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30d40b29f0 0x7f30d40b4eb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:39.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.592+0000 7f30f1975700 1 --2- 172.21.15.70:0/1434332708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30ec1095a0 0x7f30ec199aa0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:39.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.592+0000 7f30f1975700 1 --2- 172.21.15.70:0/1434332708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ec06bab0 0x7f30ec19f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:39.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.592+0000 7f30f1975700 1 --2- 172.21.15.70:0/1434332708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30ec06c470 0x7f30ec19fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:39.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.592+0000 7f30f1975700 1 -- 172.21.15.70:0/1434332708 >> 172.21.15.70:0/1434332708 conn(0x7f30ec0793b0 msgr2=0x7f30ec101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:39.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.592+0000 7f30f1975700 1 -- 172.21.15.70:0/1434332708 shutdown_connections 2024-06-04T05:03:39.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:39.592+0000 7f30f1975700 1 -- 172.21.15.70:0/1434332708 wait complete. 2024-06-04T05:03:39.596 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 32 2024-06-04T05:03:39.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.11 MB/sec execute 152 sec latency 16218.701 ms 2024-06-04T05:03:39.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:39 smithi070 ceph-mon[32631]: pgmap v197: 97 pgs: 97 active+clean; 69 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-06-04T05:03:39.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:39 smithi112 ceph-mon[36453]: pgmap v197: 97 pgs: 97 active+clean; 69 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-06-04T05:03:39.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:39 smithi098 ceph-mon[35569]: pgmap v197: 97 pgs: 97 active+clean; 69 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-06-04T05:03:40.689 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id 70640af4 --release --await' 2024-06-04T05:03:40.690 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id 70640af4 --release --await 2024-06-04T05:03:40.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.10 MB/sec execute 153 sec latency 17218.739 ms 2024-06-04T05:03:40.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:40 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1434332708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:03:40.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:40 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1434332708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:03:40.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:40 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1434332708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:03:41.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.174+0000 7f6ceca91700 1 -- 172.21.15.70:0/1699982237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce81095b0 msgr2=0x7f6ce8109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:41.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.174+0000 7f6ceca91700 1 --2- 172.21.15.70:0/1699982237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce81095b0 0x7f6ce8109990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8009990 tx=0x7f6cd8009ca0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.174+0000 7f6ceca91700 1 -- 172.21.15.70:0/1699982237 shutdown_connections 2024-06-04T05:03:41.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.174+0000 7f6ceca91700 1 --2- 172.21.15.70:0/1699982237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce806bbe0 0x7f6ce81103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.174+0000 7f6ceca91700 1 --2- 172.21.15.70:0/1699982237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce81095b0 0x7f6ce8109990 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.174+0000 7f6ceca91700 1 --2- 172.21.15.70:0/1699982237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ce806b220 0x7f6ce806b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.174+0000 7f6ceca91700 1 -- 172.21.15.70:0/1699982237 >> 172.21.15.70:0/1699982237 conn(0x7f6ce80793f0 msgr2=0x7f6ce8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:41.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.174+0000 7f6ceca91700 1 -- 172.21.15.70:0/1699982237 shutdown_connections 2024-06-04T05:03:41.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.174+0000 7f6ceca91700 1 -- 172.21.15.70:0/1699982237 wait complete. 2024-06-04T05:03:41.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.175+0000 7f6ceca91700 1 Processor -- start 2024-06-04T05:03:41.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.175+0000 7f6ceca91700 1 -- start start 2024-06-04T05:03:41.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.176+0000 7f6ceca91700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce806b220 0x7f6ce8102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.176+0000 7f6ceca91700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ce806bbe0 0x7f6ce8102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.176+0000 7f6ceca91700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce81095b0 0x7f6ce8102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.176+0000 7f6ceca91700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce8113590 con 0x7f6ce81095b0 2024-06-04T05:03:41.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.176+0000 7f6ceca91700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce8113410 con 0x7f6ce806b220 2024-06-04T05:03:41.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.176+0000 7f6ceca91700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce8113710 con 0x7f6ce806bbe0 2024-06-04T05:03:41.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.176+0000 7f6ce6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce81095b0 0x7f6ce8102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:41.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.176+0000 7f6ce6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce81095b0 0x7f6ce8102f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48748/0 (socket says 172.21.15.70:48748) 2024-06-04T05:03:41.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.176+0000 7f6ce6d9d700 1 -- 172.21.15.70:0/3012936076 learned_addr learned my addr 172.21.15.70:0/3012936076 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:41.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.176+0000 7f6ce659c700 1 --2- 172.21.15.70:0/3012936076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce806b220 0x7f6ce8102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:41.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.176+0000 7f6ce5d9b700 1 --2- 172.21.15.70:0/3012936076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ce806bbe0 0x7f6ce8102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:41.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.176+0000 7f6ce6d9d700 1 -- 172.21.15.70:0/3012936076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ce806bbe0 msgr2=0x7f6ce8102a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:41.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.176+0000 7f6ce6d9d700 1 --2- 172.21.15.70:0/3012936076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ce806bbe0 0x7f6ce8102a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.176+0000 7f6ce6d9d700 1 -- 172.21.15.70:0/3012936076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce806b220 msgr2=0x7f6ce8102510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:41.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.176+0000 7f6ce6d9d700 1 --2- 172.21.15.70:0/3012936076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce806b220 0x7f6ce8102510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.176+0000 7f6ce6d9d700 1 -- 172.21.15.70:0/3012936076 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce81a8070 con 0x7f6ce81095b0 2024-06-04T05:03:41.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.176+0000 7f6ce6d9d700 1 --2- 172.21.15.70:0/3012936076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce81095b0 0x7f6ce8102f90 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0007ee0 tx=0x7f6ce000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:41.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.177+0000 7f6cd77fe700 1 -- 172.21.15.70:0/3012936076 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0004bb0 con 0x7f6ce81095b0 2024-06-04T05:03:41.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.177+0000 7f6cd77fe700 1 -- 172.21.15.70:0/3012936076 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ce0004d50 con 0x7f6ce81095b0 2024-06-04T05:03:41.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.177+0000 7f6cd77fe700 1 -- 172.21.15.70:0/3012936076 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce00058b0 con 0x7f6ce81095b0 2024-06-04T05:03:41.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.177+0000 7f6ceca91700 1 -- 172.21.15.70:0/3012936076 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ce81a8360 con 0x7f6ce81095b0 2024-06-04T05:03:41.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.177+0000 7f6ceca91700 1 -- 172.21.15.70:0/3012936076 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ce81a8850 con 0x7f6ce81095b0 2024-06-04T05:03:41.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.180+0000 7f6cd77fe700 1 -- 172.21.15.70:0/3012936076 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f6ce001a020 con 0x7f6ce81095b0 2024-06-04T05:03:41.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.180+0000 7f6cd77fe700 1 --2- 172.21.15.70:0/3012936076 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6cd00b2a70 0x7f6cd00b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.180+0000 7f6cd77fe700 1 -- 172.21.15.70:0/3012936076 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f6ce00e1c10 con 0x7f6ce81095b0 2024-06-04T05:03:41.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.180+0000 7f6ce659c700 1 --2- 172.21.15.70:0/3012936076 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6cd00b2a70 0x7f6cd00b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:41.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.180+0000 7f6cf7326b80 1 Processor -- start 2024-06-04T05:03:41.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.181+0000 7f6cf7326b80 1 -- start start 2024-06-04T05:03:41.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.181+0000 7f6ce659c700 1 --2- 172.21.15.70:0/3012936076 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6cd00b2a70 0x7f6cd00b4f30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6cd800b5c0 tx=0x7f6cd8005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:41.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.181+0000 7f6cf7326b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 0x560bd078e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.181+0000 7f6cf7326b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd07888a0 0x560bd0788d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.181+0000 7f6cf7326b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd0789260 0x560bd0795080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.181+0000 7f6cf7326b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560bd06e6d70 con 0x560bd078e320 2024-06-04T05:03:41.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.181+0000 7f6cf7326b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560bd06d6910 con 0x560bd07888a0 2024-06-04T05:03:41.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.181+0000 7f6cf7326b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560bd06da950 con 0x560bd0789260 2024-06-04T05:03:41.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.181+0000 7f6ce5d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd07888a0 0x560bd0788d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:41.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.181+0000 7f6ce659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 0x560bd078e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:41.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.181+0000 7f6ce5d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd07888a0 0x560bd0788d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45302/0 (socket says 172.21.15.70:45302) 2024-06-04T05:03:41.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.181+0000 7f6ce5d9b700 1 -- 172.21.15.70:0/3540939208 learned_addr learned my addr 172.21.15.70:0/3540939208 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:41.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.182+0000 7f6ce5d9b700 1 -- 172.21.15.70:0/3540939208 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd0789260 msgr2=0x560bd0795080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:41.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.182+0000 7f6ce5d9b700 1 --2- 172.21.15.70:0/3540939208 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd0789260 0x560bd0795080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.182+0000 7f6ce5d9b700 1 -- 172.21.15.70:0/3540939208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 msgr2=0x560bd078e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:41.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.182+0000 7f6ce5d9b700 1 --2- 172.21.15.70:0/3540939208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 0x560bd078e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.182+0000 7f6ce5d9b700 1 -- 172.21.15.70:0/3540939208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bd067ffb0 con 0x560bd07888a0 2024-06-04T05:03:41.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.182+0000 7f6ce5d9b700 1 --2- 172.21.15.70:0/3540939208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd07888a0 0x560bd0788d20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc0046c0 tx=0x7f6cdc00b9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=877aa1c2ea3be1e1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:41.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.182+0000 7f6cd4ff9700 1 -- 172.21.15.70:0/3540939208 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc004a30 con 0x560bd07888a0 2024-06-04T05:03:41.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.182+0000 7f6cd4ff9700 1 -- 172.21.15.70:0/3540939208 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6cdc015070 con 0x560bd07888a0 2024-06-04T05:03:41.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.182+0000 7f6cd4ff9700 1 -- 172.21.15.70:0/3540939208 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc003ab0 con 0x560bd07888a0 2024-06-04T05:03:41.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.183+0000 7f6cf7326b80 1 -- 172.21.15.70:0/3540939208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd07888a0 msgr2=0x560bd0788d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:41.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.183+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/3540939208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd07888a0 0x560bd0788d20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc0046c0 tx=0x7f6cdc00b9e0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.183+0000 7f6cf7326b80 1 -- 172.21.15.70:0/3540939208 shutdown_connections 2024-06-04T05:03:41.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.183+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/3540939208 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd0789260 0x560bd0795080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.183+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/3540939208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 0x560bd078e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.183+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/3540939208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd07888a0 0x560bd0788d20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.183+0000 7f6cf7326b80 1 -- 172.21.15.70:0/3540939208 >> 172.21.15.70:0/3540939208 conn(0x560bd06b4240 msgr2=0x560bd07833e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:41.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.183+0000 7f6cf7326b80 1 -- 172.21.15.70:0/3540939208 shutdown_connections 2024-06-04T05:03:41.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.183+0000 7f6cf7326b80 1 -- 172.21.15.70:0/3540939208 wait complete. 2024-06-04T05:03:41.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6cf7326b80 1 -- start start 2024-06-04T05:03:41.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6cf7326b80 1 Processor -- start 2024-06-04T05:03:41.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6cf7326b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd07888a0 0x560bd0794d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6cf7326b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd0789260 0x560bd0792060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6cf7326b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 0x560bd07925a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6cf7326b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560bd05dcfa0 con 0x560bd078e320 2024-06-04T05:03:41.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6cf7326b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560bd063eda0 con 0x560bd07888a0 2024-06-04T05:03:41.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6cf7326b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560bd07956e0 con 0x560bd0789260 2024-06-04T05:03:41.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6cf7326b80 10 client.0 _ll_register_callbacks cb 0x560bd0671950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:41.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6ce6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 0x560bd07925a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:41.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6ce6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 0x560bd07925a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48772/0 (socket says 172.21.15.70:48772) 2024-06-04T05:03:41.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6ce6d9d700 1 -- 172.21.15.70:0/1501275522 learned_addr learned my addr 172.21.15.70:0/1501275522 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:41.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6ce659c700 1 --2- 172.21.15.70:0/1501275522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd07888a0 0x560bd0794d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:41.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6ce5d9b700 1 --2- 172.21.15.70:0/1501275522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd0789260 0x560bd0792060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:41.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6ce6d9d700 1 -- 172.21.15.70:0/1501275522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd0789260 msgr2=0x560bd0792060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:41.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6ce6d9d700 1 --2- 172.21.15.70:0/1501275522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd0789260 0x560bd0792060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6ce6d9d700 1 -- 172.21.15.70:0/1501275522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd07888a0 msgr2=0x560bd0794d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:41.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6ce6d9d700 1 --2- 172.21.15.70:0/1501275522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd07888a0 0x560bd0794d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.184+0000 7f6ce6d9d700 1 -- 172.21.15.70:0/1501275522 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bd06268d0 con 0x560bd078e320 2024-06-04T05:03:41.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.185+0000 7f6ce6d9d700 1 --2- 172.21.15.70:0/1501275522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 0x560bd07925a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00a8bf0 tx=0x7f6ce00a8930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0ad9d8fca0da72 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:41.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.185+0000 7f6ccdffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:41.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.185+0000 7f6ccdffb700 1 -- 172.21.15.70:0/1501275522 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce00a8ef0 con 0x560bd078e320 2024-06-04T05:03:41.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.185+0000 7f6ccdffb700 1 -- 172.21.15.70:0/1501275522 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f6ce00a9360 con 0x560bd078e320 2024-06-04T05:03:41.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.185+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1501275522 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cdc015070 con 0x560bd078e320 2024-06-04T05:03:41.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.185+0000 7f6ccdffb700 1 -- 172.21.15.70:0/1501275522 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6ce00ad2a0 con 0x560bd078e320 2024-06-04T05:03:41.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.185+0000 7f6cf7326b80 10 client.17190 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.185+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1501275522 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560bd0690760 con 0x560bd078e320 2024-06-04T05:03:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.185+0000 7f6ccdffb700 1 -- 172.21.15.70:0/1501275522 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ce00b1d50 con 0x560bd078e320 2024-06-04T05:03:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.185+0000 7f6ccdffb700 1 -- 172.21.15.70:0/1501275522 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f6ce00a4860 con 0x560bd078e320 2024-06-04T05:03:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.185+0000 7f6cf7326b80 10 client.17190 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.185+0000 7f6cf7326b80 10 client.17190 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.185+0000 7f6cf7326b80 20 client.17190 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.185+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/1501275522 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560bd07a22c0 0x560bd07a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.185+0000 7f6cf7326b80 4 client.17190 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:03:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.185+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1501275522 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6cdc015070 con 0x560bd07a22c0 2024-06-04T05:03:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.186+0000 7f6ce659c700 1 --2- 172.21.15.70:0/1501275522 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560bd07a22c0 0x560bd07a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.186+0000 7f6ccdffb700 10 client.17190 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.186+0000 7f6ce659c700 1 --2- 172.21.15.70:0/1501275522 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560bd07a22c0 0x560bd07a46e0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=716eae29ed1f92bd server_cookie=25c59a1b88fb487e in_seq=0 out_seq=0 2024-06-04T05:03:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.187+0000 7f6ccdffb700 1 -- 172.21.15.70:0/1501275522 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f6cdc015070 con 0x560bd07a22c0 2024-06-04T05:03:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.187+0000 7f6ccdffb700 10 client.17190 handle_command_reply: tid=0 2024-06-04T05:03:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.187+0000 7f6cf7326b80 1 client.17190 shutdown 2024-06-04T05:03:41.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.187+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1501275522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 msgr2=0x560bd07925a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:41.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.187+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/1501275522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 0x560bd07925a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00a8bf0 tx=0x7f6ce00a8930 comp rx=0 tx=0).stop 2024-06-04T05:03:41.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.187+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1501275522 shutdown_connections 2024-06-04T05:03:41.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.187+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/1501275522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 0x560bd07925a0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.187+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/1501275522 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560bd07a22c0 0x560bd07a46e0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.187+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/1501275522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd07888a0 0x560bd0794d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.187+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/1501275522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd0789260 0x560bd0792060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.187+0000 7f6ccdffb700 0 client.17190 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:41.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.187+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1501275522 >> 172.21.15.70:0/1501275522 conn(0x560bd06b4240 msgr2=0x560bd0782af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:41.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.187+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1501275522 shutdown_connections 2024-06-04T05:03:41.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.187+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1501275522 wait complete. 2024-06-04T05:03:41.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.187+0000 7f6cf7326b80 20 client.17190 trim_cache size 0 max 16384 2024-06-04T05:03:41.203 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:03:41.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.203+0000 7f6cf7326b80 1 Processor -- start 2024-06-04T05:03:41.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.203+0000 7f6cf7326b80 1 -- start start 2024-06-04T05:03:41.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.203+0000 7f6cf7326b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd07888a0 0x560bd079e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.203+0000 7f6cf7326b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd0789260 0x560bd0783270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.203+0000 7f6cf7326b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd078e320 0x560bd0787fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.203+0000 7f6cf7326b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce81affb0 con 0x560bd0789260 2024-06-04T05:03:41.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.203+0000 7f6cf7326b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce81b1af0 con 0x560bd078e320 2024-06-04T05:03:41.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.203+0000 7f6cf7326b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce81b0680 con 0x560bd07888a0 2024-06-04T05:03:41.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.203+0000 7f6ce5d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd0789260 0x560bd0783270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:41.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.204+0000 7f6ce5d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd0789260 0x560bd0783270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48774/0 (socket says 172.21.15.70:48774) 2024-06-04T05:03:41.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.204+0000 7f6ce5d9b700 1 -- 172.21.15.70:0/1511568207 learned_addr learned my addr 172.21.15.70:0/1511568207 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:41.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.204+0000 7f6ce659c700 1 --2- 172.21.15.70:0/1511568207 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd07888a0 0x560bd079e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:41.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.204+0000 7f6ce6d9d700 1 --2- 172.21.15.70:0/1511568207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd078e320 0x560bd0787fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:41.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.204+0000 7f6ce5d9b700 1 -- 172.21.15.70:0/1511568207 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd07888a0 msgr2=0x560bd079e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:41.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.204+0000 7f6ce5d9b700 1 --2- 172.21.15.70:0/1511568207 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd07888a0 0x560bd079e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.204+0000 7f6ce5d9b700 1 -- 172.21.15.70:0/1511568207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd078e320 msgr2=0x560bd0787fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:41.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.204+0000 7f6ce5d9b700 1 --2- 172.21.15.70:0/1511568207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd078e320 0x560bd0787fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.204+0000 7f6ce5d9b700 1 -- 172.21.15.70:0/1511568207 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bd0792f60 con 0x560bd0789260 2024-06-04T05:03:41.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.204+0000 7f6ce6d9d700 1 --2- 172.21.15.70:0/1511568207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd078e320 0x560bd0787fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:03:41.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.204+0000 7f6ce5d9b700 1 --2- 172.21.15.70:0/1511568207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd0789260 0x560bd0783270 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc0045e0 tx=0x7f6cdc0051d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e280c0c75f7508 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:41.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.204+0000 7f6ccffff700 1 -- 172.21.15.70:0/1511568207 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc028070 con 0x560bd0789260 2024-06-04T05:03:41.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.204+0000 7f6ccffff700 1 -- 172.21.15.70:0/1511568207 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6cdc015040 con 0x560bd0789260 2024-06-04T05:03:41.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.204+0000 7f6ccffff700 1 -- 172.21.15.70:0/1511568207 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc0039a0 con 0x560bd0789260 2024-06-04T05:03:41.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.205+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1511568207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd0789260 msgr2=0x560bd0783270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:41.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.205+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/1511568207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd0789260 0x560bd0783270 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc0045e0 tx=0x7f6cdc0051d0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.205+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1511568207 shutdown_connections 2024-06-04T05:03:41.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.205+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/1511568207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd078e320 0x560bd0787fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.205+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/1511568207 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd07888a0 0x560bd079e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.205+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/1511568207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd0789260 0x560bd0783270 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.205+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1511568207 >> 172.21.15.70:0/1511568207 conn(0x560bd06b4240 msgr2=0x560bd0787890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:41.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.205+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1511568207 shutdown_connections 2024-06-04T05:03:41.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.205+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1511568207 wait complete. 2024-06-04T05:03:41.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.205+0000 7f6cf7326b80 1 -- start start 2024-06-04T05:03:41.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6cf7326b80 1 Processor -- start 2024-06-04T05:03:41.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6cf7326b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd07888a0 0x560bd079b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6cf7326b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd0789260 0x560bd07a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6cf7326b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 0x560bd07aa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6cf7326b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce81b1420 con 0x560bd078e320 2024-06-04T05:03:41.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6cf7326b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce81b0d50 con 0x560bd0789260 2024-06-04T05:03:41.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6ce6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 0x560bd07aa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:41.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6cf7326b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce81adf50 con 0x560bd07888a0 2024-06-04T05:03:41.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6ce6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 0x560bd07aa0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48784/0 (socket says 172.21.15.70:48784) 2024-06-04T05:03:41.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6ce6d9d700 1 -- 172.21.15.70:0/1676683134 learned_addr learned my addr 172.21.15.70:0/1676683134 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:41.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6ce659c700 1 --2- 172.21.15.70:0/1676683134 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd07888a0 0x560bd079b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:41.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6ce5d9b700 1 --2- 172.21.15.70:0/1676683134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd0789260 0x560bd07a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:41.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6cf7326b80 10 client.0 _ll_register_callbacks cb 0x560bd0671950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:41.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6ce6d9d700 1 -- 172.21.15.70:0/1676683134 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd07888a0 msgr2=0x560bd079b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:41.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6ce6d9d700 1 --2- 172.21.15.70:0/1676683134 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd07888a0 0x560bd079b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6ce6d9d700 1 -- 172.21.15.70:0/1676683134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd0789260 msgr2=0x560bd07a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:41.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6ce6d9d700 1 --2- 172.21.15.70:0/1676683134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd0789260 0x560bd07a9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6ce6d9d700 1 -- 172.21.15.70:0/1676683134 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bd0798c80 con 0x560bd078e320 2024-06-04T05:03:41.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6ce659c700 1 --2- 172.21.15.70:0/1676683134 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd07888a0 0x560bd079b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:41.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.206+0000 7f6ce5d9b700 1 --2- 172.21.15.70:0/1676683134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd0789260 0x560bd07a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:41.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.207+0000 7f6ce6d9d700 1 --2- 172.21.15.70:0/1676683134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 0x560bd07aa0a0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00e1f30 tx=0x7f6ce0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed8a933c937eceb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:41.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.207+0000 7f6ccdffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:41.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.207+0000 7f6ccdffb700 1 -- 172.21.15.70:0/1676683134 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0087070 con 0x560bd078e320 2024-06-04T05:03:41.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.207+0000 7f6ccdffb700 1 -- 172.21.15.70:0/1676683134 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f6ce00dfb80 con 0x560bd078e320 2024-06-04T05:03:41.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.207+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1676683134 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cdc015040 con 0x560bd078e320 2024-06-04T05:03:41.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.207+0000 7f6ccdffb700 1 -- 172.21.15.70:0/1676683134 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6ce00b1ad0 con 0x560bd078e320 2024-06-04T05:03:41.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.207+0000 7f6cf7326b80 10 client.17196 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:41.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.207+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1676683134 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560bd0815880 con 0x560bd078e320 2024-06-04T05:03:41.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.207+0000 7f6ccdffb700 1 -- 172.21.15.70:0/1676683134 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6cdc015040 con 0x560bd078e320 2024-06-04T05:03:41.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.207+0000 7f6ccdffb700 1 -- 172.21.15.70:0/1676683134 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f6ce000db20 con 0x560bd078e320 2024-06-04T05:03:41.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.207+0000 7f6cf7326b80 10 client.17196 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:41.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.207+0000 7f6cf7326b80 10 client.17196 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.207+0000 7f6cf7326b80 20 client.17196 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.207+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/1676683134 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560bd0815e00 0x560bd0818220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.207+0000 7f6cf7326b80 4 client.17196 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:03:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.207+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1676683134 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560bd06e8cc0 con 0x560bd0815e00 2024-06-04T05:03:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.208+0000 7f6ce659c700 1 --2- 172.21.15.70:0/1676683134 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560bd0815e00 0x560bd0818220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.208+0000 7f6ce659c700 1 --2- 172.21.15.70:0/1676683134 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560bd0815e00 0x560bd0818220 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38f53b57f441ed29 server_cookie=d516c79d40beb718 in_seq=0 out_seq=0 2024-06-04T05:03:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.208+0000 7f6ccdffb700 10 client.17196 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.208+0000 7f6ccdffb700 1 -- 172.21.15.70:0/1676683134 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560bd06e8cc0 con 0x560bd0815e00 2024-06-04T05:03:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.208+0000 7f6ccdffb700 10 client.17196 handle_command_reply: tid=0 2024-06-04T05:03:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.208+0000 7f6cf7326b80 1 client.17196 shutdown 2024-06-04T05:03:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.208+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1676683134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 msgr2=0x560bd07aa0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.208+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/1676683134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 0x560bd07aa0a0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00e1f30 tx=0x7f6ce0009260 comp rx=0 tx=0).stop 2024-06-04T05:03:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.208+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1676683134 shutdown_connections 2024-06-04T05:03:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.208+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/1676683134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560bd078e320 0x560bd07aa0a0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.208+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/1676683134 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560bd07888a0 0x560bd079b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.208+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/1676683134 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560bd0815e00 0x560bd0818220 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.208+0000 7f6cf7326b80 1 --2- 172.21.15.70:0/1676683134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd0789260 0x560bd07a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.208+0000 7f6ccdffb700 0 client.17196 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:41.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.208+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1676683134 >> 172.21.15.70:0/1676683134 conn(0x560bd06b4240 msgr2=0x560bd078d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:41.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.209+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1676683134 shutdown_connections 2024-06-04T05:03:41.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.209+0000 7f6cf7326b80 1 -- 172.21.15.70:0/1676683134 wait complete. 2024-06-04T05:03:41.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.209+0000 7f6cf7326b80 20 client.17196 trim_cache size 0 max 16384 2024-06-04T05:03:41.215 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:03:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.209+0000 7f6ceca91700 1 -- 172.21.15.70:0/3012936076 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6cd00b2a70 msgr2=0x7f6cd00b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.209+0000 7f6ceca91700 1 --2- 172.21.15.70:0/3012936076 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6cd00b2a70 0x7f6cd00b4f30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6cd800b5c0 tx=0x7f6cd8005960 comp rx=0 tx=0).stop 2024-06-04T05:03:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.209+0000 7f6ceca91700 1 -- 172.21.15.70:0/3012936076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce81095b0 msgr2=0x7f6ce8102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.209+0000 7f6ceca91700 1 --2- 172.21.15.70:0/3012936076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce81095b0 0x7f6ce8102f90 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0007ee0 tx=0x7f6ce000d3b0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.209+0000 7f6ceca91700 1 -- 172.21.15.70:0/3012936076 shutdown_connections 2024-06-04T05:03:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.209+0000 7f6ceca91700 1 --2- 172.21.15.70:0/3012936076 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6cd00b2a70 0x7f6cd00b4f30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.209+0000 7f6ceca91700 1 --2- 172.21.15.70:0/3012936076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce81095b0 0x7f6ce8102f90 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.209+0000 7f6ceca91700 1 --2- 172.21.15.70:0/3012936076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce806b220 0x7f6ce8102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.209+0000 7f6ceca91700 1 --2- 172.21.15.70:0/3012936076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ce806bbe0 0x7f6ce8102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.209+0000 7f6ceca91700 1 -- 172.21.15.70:0/3012936076 >> 172.21.15.70:0/3012936076 conn(0x7f6ce80793f0 msgr2=0x7f6ce8101330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.210+0000 7f6ceca91700 1 -- 172.21.15.70:0/3012936076 shutdown_connections 2024-06-04T05:03:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:41.210+0000 7f6ceca91700 1 -- 172.21.15.70:0/3012936076 wait complete. 2024-06-04T05:03:41.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.08 MB/sec execute 154 sec latency 18218.769 ms 2024-06-04T05:03:41.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:41 smithi070 ceph-mon[32631]: pgmap v198: 97 pgs: 97 active+clean; 69 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-06-04T05:03:41.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:41 smithi112 ceph-mon[36453]: pgmap v198: 97 pgs: 97 active+clean; 69 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-06-04T05:03:41.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:41 smithi098 ceph-mon[35569]: pgmap v198: 97 pgs: 97 active+clean; 69 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-06-04T05:03:42.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.07 MB/sec execute 155 sec latency 19218.806 ms 2024-06-04T05:03:42.938 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:03:42.939 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:03:43.269 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:03:43.713 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:43 smithi070 ceph-mon[32631]: pgmap v199: 97 pgs: 97 active+clean; 69 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 18 op/s 2024-06-04T05:03:43.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.06 MB/sec execute 156 sec latency 20218.838 ms 2024-06-04T05:03:43.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:43 smithi112 ceph-mon[36453]: pgmap v199: 97 pgs: 97 active+clean; 69 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 18 op/s 2024-06-04T05:03:43.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:43 smithi098 ceph-mon[35569]: pgmap v199: 97 pgs: 97 active+clean; 69 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 18 op/s 2024-06-04T05:03:44.724 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105418 2.04 MB/sec execute 157 sec latency 21218.880 ms 2024-06-04T05:03:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.752+0000 7f72959d0640 1 -- 172.21.15.70:0/3409685233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72901033f0 msgr2=0x7f729010f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.752+0000 7f72959d0640 1 --2- 172.21.15.70:0/3409685233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72901033f0 0x7f729010f900 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f7288009cd0 tx=0x7f728802a8e0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.752+0000 7f72959d0640 1 -- 172.21.15.70:0/3409685233 shutdown_connections 2024-06-04T05:03:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.752+0000 7f72959d0640 1 --2- 172.21.15.70:0/3409685233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7290102a30 0x7f7290102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.752+0000 7f72959d0640 1 --2- 172.21.15.70:0/3409685233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72901033f0 0x7f729010f900 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.752+0000 7f72959d0640 1 --2- 172.21.15.70:0/3409685233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7290108a40 0x7f7290108e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.752+0000 7f72959d0640 1 -- 172.21.15.70:0/3409685233 >> 172.21.15.70:0/3409685233 conn(0x7f72900fe500 msgr2=0x7f7290100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.752+0000 7f72959d0640 1 -- 172.21.15.70:0/3409685233 shutdown_connections 2024-06-04T05:03:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.752+0000 7f72959d0640 1 -- 172.21.15.70:0/3409685233 wait complete. 2024-06-04T05:03:44.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.753+0000 7f72959d0640 1 Processor -- start 2024-06-04T05:03:44.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.753+0000 7f72959d0640 1 -- start start 2024-06-04T05:03:44.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.753+0000 7f72959d0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7290102a30 0x7f7290199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.753+0000 7f72959d0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72901033f0 0x7f72901996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.753+0000 7f72959d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7290108a40 0x7f729019d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.753+0000 7f72959d0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7290112bd0 con 0x7f72901033f0 2024-06-04T05:03:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.753+0000 7f72959d0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7290112a50 con 0x7f7290108a40 2024-06-04T05:03:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.753+0000 7f72959d0640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7290112d50 con 0x7f7290102a30 2024-06-04T05:03:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.754+0000 7f728e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72901033f0 0x7f72901996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.754+0000 7f728e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72901033f0 0x7f72901996b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48808/0 (socket says 172.21.15.70:48808) 2024-06-04T05:03:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.754+0000 7f728e7fc640 1 -- 172.21.15.70:0/928092642 learned_addr learned my addr 172.21.15.70:0/928092642 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.754+0000 7f728effd640 1 --2- 172.21.15.70:0/928092642 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7290102a30 0x7f7290199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.754+0000 7f728f7fe640 1 --2- 172.21.15.70:0/928092642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7290108a40 0x7f729019d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.754+0000 7f728e7fc640 1 -- 172.21.15.70:0/928092642 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7290102a30 msgr2=0x7f7290199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.754+0000 7f728e7fc640 1 --2- 172.21.15.70:0/928092642 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7290102a30 0x7f7290199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.754+0000 7f728e7fc640 1 -- 172.21.15.70:0/928092642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7290108a40 msgr2=0x7f729019d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.754+0000 7f728e7fc640 1 --2- 172.21.15.70:0/928092642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7290108a40 0x7f729019d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.754+0000 7f728e7fc640 1 -- 172.21.15.70:0/928092642 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729019e110 con 0x7f72901033f0 2024-06-04T05:03:44.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.754+0000 7f728e7fc640 1 --2- 172.21.15.70:0/928092642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72901033f0 0x7f72901996b0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f728400b4a0 tx=0x7f728400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:44.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.754+0000 7f72949ce640 1 -- 172.21.15.70:0/928092642 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7284014070 con 0x7f72901033f0 2024-06-04T05:03:44.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.754+0000 7f72949ce640 1 -- 172.21.15.70:0/928092642 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7284010aa0 con 0x7f72901033f0 2024-06-04T05:03:44.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.754+0000 7f72959d0640 1 -- 172.21.15.70:0/928092642 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f729019e400 con 0x7f72901033f0 2024-06-04T05:03:44.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.754+0000 7f72949ce640 1 -- 172.21.15.70:0/928092642 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7284017b70 con 0x7f72901033f0 2024-06-04T05:03:44.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.754+0000 7f72959d0640 1 -- 172.21.15.70:0/928092642 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72901a5c90 con 0x7f72901033f0 2024-06-04T05:03:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.756+0000 7f72949ce640 1 -- 172.21.15.70:0/928092642 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7284017070 con 0x7f72901033f0 2024-06-04T05:03:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f72987c5740 1 Processor -- start 2024-06-04T05:03:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f72987c5740 1 -- start start 2024-06-04T05:03:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f72987c5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57104820 0x55fe57104c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f72987c5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe57105140 0x55fe5710f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f72987c5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe5710fa00 0x55fe57111df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f72987c5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fe57076c30 con 0x55fe57104820 2024-06-04T05:03:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f72987c5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fe570667d0 con 0x55fe5710fa00 2024-06-04T05:03:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f72987c5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fe5706a810 con 0x55fe57105140 2024-06-04T05:03:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f728f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57104820 0x55fe57104c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f728f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57104820 0x55fe57104c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48824/0 (socket says 172.21.15.70:48824) 2024-06-04T05:03:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f728f7fe640 1 -- 172.21.15.70:0/1414775222 learned_addr learned my addr 172.21.15.70:0/1414775222 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f72949ce640 1 --2- 172.21.15.70:0/928092642 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f72780b2a10 0x7f72780b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f728f7fe640 1 --2- 172.21.15.70:0/928092642 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f72780b2a10 0x7f72780b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f728effd640 1 --2- 172.21.15.70:0/1414775222 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe57105140 0x55fe5710f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f72949ce640 1 -- 172.21.15.70:0/928092642 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f72840e1be0 con 0x7f72901033f0 2024-06-04T05:03:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f728e7fc640 1 --2- 172.21.15.70:0/1414775222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe5710fa00 0x55fe57111df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f728f7fe640 1 -- 172.21.15.70:0/1414775222 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe57105140 msgr2=0x55fe5710f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f728f7fe640 1 --2- 172.21.15.70:0/1414775222 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe57105140 0x55fe5710f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f728f7fe640 1 -- 172.21.15.70:0/1414775222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe5710fa00 msgr2=0x55fe57111df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f728f7fe640 1 --2- 172.21.15.70:0/1414775222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe5710fa00 0x55fe57111df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f728f7fe640 1 -- 172.21.15.70:0/1414775222 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe56d7b210 con 0x55fe57104820 2024-06-04T05:03:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f728effd640 1 --2- 172.21.15.70:0/1414775222 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe57105140 0x55fe5710f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:03:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.757+0000 7f728e7fc640 1 --2- 172.21.15.70:0/1414775222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe5710fa00 0x55fe57111df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:03:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.758+0000 7f728f7fe640 1 --2- 172.21.15.70:0/1414775222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57104820 0x55fe57104c00 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f7288009430 tx=0x7f7288031af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d2b97c6bd472149 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.758+0000 7f7275ffb640 1 -- 172.21.15.70:0/1414775222 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7288004d40 con 0x55fe57104820 2024-06-04T05:03:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.758+0000 7f7275ffb640 1 -- 172.21.15.70:0/1414775222 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7288036040 con 0x55fe57104820 2024-06-04T05:03:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.758+0000 7f728f7fe640 1 --2- 172.21.15.70:0/928092642 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f72780b2a10 0x7f72780b4ed0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f728804d000 tx=0x7f728804b000 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.758+0000 7f7275ffb640 1 -- 172.21.15.70:0/1414775222 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72880307e0 con 0x55fe57104820 2024-06-04T05:03:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.758+0000 7f72987c5740 1 -- 172.21.15.70:0/1414775222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57104820 msgr2=0x55fe57104c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.758+0000 7f72987c5740 1 --2- 172.21.15.70:0/1414775222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57104820 0x55fe57104c00 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f7288009430 tx=0x7f7288031af0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.759+0000 7f72987c5740 1 -- 172.21.15.70:0/1414775222 shutdown_connections 2024-06-04T05:03:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.759+0000 7f72987c5740 1 --2- 172.21.15.70:0/1414775222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe5710fa00 0x55fe57111df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.759+0000 7f72987c5740 1 --2- 172.21.15.70:0/1414775222 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe57105140 0x55fe5710f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.759+0000 7f72987c5740 1 --2- 172.21.15.70:0/1414775222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57104820 0x55fe57104c00 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.759+0000 7f72987c5740 1 -- 172.21.15.70:0/1414775222 >> 172.21.15.70:0/1414775222 conn(0x55fe57100d30 msgr2=0x55fe57101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.759+0000 7f72987c5740 1 -- 172.21.15.70:0/1414775222 shutdown_connections 2024-06-04T05:03:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.759+0000 7f72987c5740 1 -- 172.21.15.70:0/1414775222 wait complete. 2024-06-04T05:03:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.759+0000 7f72987c5740 1 -- start start 2024-06-04T05:03:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.759+0000 7f72987c5740 1 Processor -- start 2024-06-04T05:03:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.759+0000 7f72987c5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe57104820 0x55fe5710c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.759+0000 7f72987c5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57105140 0x55fe5710ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.759+0000 7f72987c5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe5710fa00 0x55fe5710d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.759+0000 7f72987c5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fe56d4c790 con 0x55fe57105140 2024-06-04T05:03:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.759+0000 7f72987c5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fe57022890 con 0x55fe57104820 2024-06-04T05:03:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.759+0000 7f72987c5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fe57112630 con 0x55fe5710fa00 2024-06-04T05:03:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.759+0000 7f72987c5740 10 client.0 _ll_register_callbacks cb 0x55fe56d7aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.759+0000 7f728e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57105140 0x55fe5710ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f728e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57105140 0x55fe5710ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48840/0 (socket says 172.21.15.70:48840) 2024-06-04T05:03:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f728e7fc640 1 -- 172.21.15.70:0/3701593520 learned_addr learned my addr 172.21.15.70:0/3701593520 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f728f7fe640 1 --2- 172.21.15.70:0/3701593520 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe5710fa00 0x55fe5710d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f728effd640 1 --2- 172.21.15.70:0/3701593520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe57104820 0x55fe5710c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f728e7fc640 1 -- 172.21.15.70:0/3701593520 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe5710fa00 msgr2=0x55fe5710d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f728e7fc640 1 --2- 172.21.15.70:0/3701593520 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe5710fa00 0x55fe5710d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f728e7fc640 1 -- 172.21.15.70:0/3701593520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe57104820 msgr2=0x55fe5710c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f728e7fc640 1 --2- 172.21.15.70:0/3701593520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe57104820 0x55fe5710c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f728e7fc640 1 -- 172.21.15.70:0/3701593520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe56e47620 con 0x55fe57105140 2024-06-04T05:03:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f728effd640 1 --2- 172.21.15.70:0/3701593520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe57104820 0x55fe5710c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:03:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f728e7fc640 1 --2- 172.21.15.70:0/3701593520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57105140 0x55fe5710ce60 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f72840a8f60 tx=0x7f7284017f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839a0f06603abc44 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f726effd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f726effd640 1 -- 172.21.15.70:0/3701593520 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728401e070 con 0x55fe57105140 2024-06-04T05:03:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f726effd640 1 -- 172.21.15.70:0/3701593520 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f72840df5b0 con 0x55fe57105140 2024-06-04T05:03:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f72987c5740 1 -- 172.21.15.70:0/3701593520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7288036040 con 0x55fe57105140 2024-06-04T05:03:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f726effd640 1 -- 172.21.15.70:0/3701593520 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f72840a96d0 con 0x55fe57105140 2024-06-04T05:03:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f72987c5740 10 client.17208 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f72987c5740 1 -- 172.21.15.70:0/3701593520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fe5711e2b0 con 0x55fe57105140 2024-06-04T05:03:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.760+0000 7f726effd640 1 -- 172.21.15.70:0/3701593520 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7288036040 con 0x55fe57105140 2024-06-04T05:03:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.761+0000 7f726effd640 1 -- 172.21.15.70:0/3701593520 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f72840b19a0 con 0x55fe57105140 2024-06-04T05:03:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.761+0000 7f72987c5740 10 client.17208 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.761+0000 7f72987c5740 10 client.17208 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.761+0000 7f72987c5740 20 client.17208 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.761+0000 7f72987c5740 1 --2- 172.21.15.70:0/3701593520 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fe5711f720 0x55fe57121b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.761+0000 7f72987c5740 4 client.17208 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:03:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.761+0000 7f72987c5740 1 -- 172.21.15.70:0/3701593520 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe57078b80 con 0x55fe5711f720 2024-06-04T05:03:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.761+0000 7f728effd640 1 --2- 172.21.15.70:0/3701593520 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fe5711f720 0x55fe57121b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.761+0000 7f728effd640 1 --2- 172.21.15.70:0/3701593520 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fe5711f720 0x55fe57121b10 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f792f902c5ec53f4 server_cookie=f49cc2c36cfe240f in_seq=0 out_seq=0 2024-06-04T05:03:44.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.761+0000 7f726effd640 10 client.17208 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:44.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.762+0000 7f726effd640 1 -- 172.21.15.70:0/3701593520 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fe57078b80 con 0x55fe5711f720 2024-06-04T05:03:44.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.762+0000 7f726effd640 10 client.17208 handle_command_reply: tid=0 2024-06-04T05:03:44.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.762+0000 7f72987c5740 1 client.17208 shutdown 2024-06-04T05:03:44.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.762+0000 7f72987c5740 1 -- 172.21.15.70:0/3701593520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57105140 msgr2=0x55fe5710ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:44.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.762+0000 7f72987c5740 1 --2- 172.21.15.70:0/3701593520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57105140 0x55fe5710ce60 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f72840a8f60 tx=0x7f7284017f90 comp rx=0 tx=0).stop 2024-06-04T05:03:44.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.762+0000 7f72987c5740 1 -- 172.21.15.70:0/3701593520 shutdown_connections 2024-06-04T05:03:44.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.762+0000 7f72987c5740 1 --2- 172.21.15.70:0/3701593520 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fe5711f720 0x55fe57121b10 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.762+0000 7f72987c5740 1 --2- 172.21.15.70:0/3701593520 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe5710fa00 0x55fe5710d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.763+0000 7f726effd640 0 client.17208 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:44.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.763+0000 7f72987c5740 1 --2- 172.21.15.70:0/3701593520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57105140 0x55fe5710ce60 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.763+0000 7f72987c5740 1 --2- 172.21.15.70:0/3701593520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe57104820 0x55fe5710c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.763+0000 7f72987c5740 1 -- 172.21.15.70:0/3701593520 >> 172.21.15.70:0/3701593520 conn(0x55fe57100d30 msgr2=0x55fe570fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.763+0000 7f72987c5740 1 -- 172.21.15.70:0/3701593520 shutdown_connections 2024-06-04T05:03:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.763+0000 7f72987c5740 1 -- 172.21.15.70:0/3701593520 wait complete. 2024-06-04T05:03:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.763+0000 7f72987c5740 20 client.17208 trim_cache size 0 max 16384 2024-06-04T05:03:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f72987c5740 1 Processor -- start 2024-06-04T05:03:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f72987c5740 1 -- start start 2024-06-04T05:03:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f72987c5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe57104820 0x55fe571273d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f72987c5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe57105140 0x55fe57127910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f72987c5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe5710fa00 0x55fe5710c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f72987c5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f72901ae570 con 0x55fe5710fa00 2024-06-04T05:03:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f72987c5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f72901b00b0 con 0x55fe57104820 2024-06-04T05:03:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f72987c5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f72901aec40 con 0x55fe57105140 2024-06-04T05:03:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f728f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe5710fa00 0x55fe5710c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f728f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe5710fa00 0x55fe5710c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48848/0 (socket says 172.21.15.70:48848) 2024-06-04T05:03:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f728f7fe640 1 -- 172.21.15.70:0/2250200385 learned_addr learned my addr 172.21.15.70:0/2250200385 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f728effd640 1 --2- 172.21.15.70:0/2250200385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe57104820 0x55fe571273d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f728e7fc640 1 --2- 172.21.15.70:0/2250200385 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe57105140 0x55fe57127910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f728f7fe640 1 -- 172.21.15.70:0/2250200385 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe57105140 msgr2=0x55fe57127910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f728f7fe640 1 --2- 172.21.15.70:0/2250200385 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe57105140 0x55fe57127910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f728f7fe640 1 -- 172.21.15.70:0/2250200385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe57104820 msgr2=0x55fe571273d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f728f7fe640 1 --2- 172.21.15.70:0/2250200385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe57104820 0x55fe571273d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f728f7fe640 1 -- 172.21.15.70:0/2250200385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe57104f70 con 0x55fe5710fa00 2024-06-04T05:03:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.778+0000 7f728f7fe640 1 --2- 172.21.15.70:0/2250200385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe5710fa00 0x55fe5710c8e0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f72880314a0 tx=0x7f7288005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336dbfa906cd99e7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.779+0000 7f72757fa640 1 -- 172.21.15.70:0/2250200385 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72880305a0 con 0x55fe5710fa00 2024-06-04T05:03:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.779+0000 7f72757fa640 1 -- 172.21.15.70:0/2250200385 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7288036030 con 0x55fe5710fa00 2024-06-04T05:03:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.779+0000 7f72757fa640 1 -- 172.21.15.70:0/2250200385 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7288030e30 con 0x55fe5710fa00 2024-06-04T05:03:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.779+0000 7f72987c5740 1 -- 172.21.15.70:0/2250200385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe5710fa00 msgr2=0x55fe5710c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.779+0000 7f72987c5740 1 --2- 172.21.15.70:0/2250200385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe5710fa00 0x55fe5710c8e0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f72880314a0 tx=0x7f7288005c30 comp rx=0 tx=0).stop 2024-06-04T05:03:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.779+0000 7f72987c5740 1 -- 172.21.15.70:0/2250200385 shutdown_connections 2024-06-04T05:03:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.779+0000 7f72987c5740 1 --2- 172.21.15.70:0/2250200385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe5710fa00 0x55fe5710c8e0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.779+0000 7f72987c5740 1 --2- 172.21.15.70:0/2250200385 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe57105140 0x55fe57127910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.779+0000 7f72987c5740 1 --2- 172.21.15.70:0/2250200385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe57104820 0x55fe571273d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.779+0000 7f72987c5740 1 -- 172.21.15.70:0/2250200385 >> 172.21.15.70:0/2250200385 conn(0x55fe57100d30 msgr2=0x55fe57107b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.779+0000 7f72987c5740 1 -- 172.21.15.70:0/2250200385 shutdown_connections 2024-06-04T05:03:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.779+0000 7f72987c5740 1 -- 172.21.15.70:0/2250200385 wait complete. 2024-06-04T05:03:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.779+0000 7f72987c5740 1 -- start start 2024-06-04T05:03:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f72987c5740 1 Processor -- start 2024-06-04T05:03:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f72987c5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57104820 0x55fe571135f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f72987c5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe57105140 0x55fe57113b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f72987c5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe5710fa00 0x55fe57114070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f72987c5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f72901af9e0 con 0x55fe57104820 2024-06-04T05:03:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f72987c5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f72901af310 con 0x55fe5710fa00 2024-06-04T05:03:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f72987c5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f72901ac550 con 0x55fe57105140 2024-06-04T05:03:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f72987c5740 10 client.0 _ll_register_callbacks cb 0x7f72840092f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f728effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57104820 0x55fe571135f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f728effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57104820 0x55fe571135f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48850/0 (socket says 172.21.15.70:48850) 2024-06-04T05:03:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f728effd640 1 -- 172.21.15.70:0/3435231398 learned_addr learned my addr 172.21.15.70:0/3435231398 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f728e7fc640 1 --2- 172.21.15.70:0/3435231398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe57105140 0x55fe57113b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f728f7fe640 1 --2- 172.21.15.70:0/3435231398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe5710fa00 0x55fe57114070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f728effd640 1 -- 172.21.15.70:0/3435231398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe57105140 msgr2=0x55fe57113b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f728effd640 1 --2- 172.21.15.70:0/3435231398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe57105140 0x55fe57113b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f728effd640 1 -- 172.21.15.70:0/3435231398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe5710fa00 msgr2=0x55fe57114070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f728effd640 1 --2- 172.21.15.70:0/3435231398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe5710fa00 0x55fe57114070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f728effd640 1 -- 172.21.15.70:0/3435231398 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe571892d0 con 0x55fe57104820 2024-06-04T05:03:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.780+0000 7f728effd640 1 --2- 172.21.15.70:0/3435231398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57104820 0x55fe571135f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x55fe57122340 tx=0x7f728000bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e772e9afb4d01ac6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.781+0000 7f726effd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.781+0000 7f726effd640 1 -- 172.21.15.70:0/3435231398 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728001b330 con 0x55fe57104820 2024-06-04T05:03:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.781+0000 7f726effd640 1 -- 172.21.15.70:0/3435231398 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f728001b4d0 con 0x55fe57104820 2024-06-04T05:03:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.781+0000 7f72987c5740 1 -- 172.21.15.70:0/3435231398 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7288036030 con 0x55fe57104820 2024-06-04T05:03:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.781+0000 7f726effd640 1 -- 172.21.15.70:0/3435231398 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f728001b6d0 con 0x55fe57104820 2024-06-04T05:03:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.781+0000 7f72987c5740 10 client.17214 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.781+0000 7f72987c5740 1 -- 172.21.15.70:0/3435231398 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fe5718a620 con 0x55fe57104820 2024-06-04T05:03:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.781+0000 7f726effd640 1 -- 172.21.15.70:0/3435231398 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7288036030 con 0x55fe57104820 2024-06-04T05:03:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.781+0000 7f726effd640 1 -- 172.21.15.70:0/3435231398 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f728002dc00 con 0x55fe57104820 2024-06-04T05:03:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.781+0000 7f72987c5740 10 client.17214 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.781+0000 7f72987c5740 10 client.17214 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.781+0000 7f72987c5740 20 client.17214 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.781+0000 7f72987c5740 1 --2- 172.21.15.70:0/3435231398 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fe5718ad50 0x55fe5718d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.781+0000 7f72987c5740 4 client.17214 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:03:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.781+0000 7f72987c5740 1 -- 172.21.15.70:0/3435231398 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fe57072360 con 0x55fe5718ad50 2024-06-04T05:03:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.781+0000 7f728e7fc640 1 --2- 172.21.15.70:0/3435231398 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fe5718ad50 0x55fe5718d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.782+0000 7f728e7fc640 1 --2- 172.21.15.70:0/3435231398 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fe5718ad50 0x55fe5718d170 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=823395da889a71c2 server_cookie=4243bce6d13fd8b2 in_seq=0 out_seq=0 2024-06-04T05:03:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.782+0000 7f726effd640 10 client.17214 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f726effd640 1 -- 172.21.15.70:0/3435231398 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55fe57072360 con 0x55fe5718ad50 2024-06-04T05:03:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f726effd640 10 client.17214 handle_command_reply: tid=0 2024-06-04T05:03:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72987c5740 1 client.17214 shutdown 2024-06-04T05:03:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72987c5740 1 -- 172.21.15.70:0/3435231398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57104820 msgr2=0x55fe571135f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72987c5740 1 --2- 172.21.15.70:0/3435231398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57104820 0x55fe571135f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x55fe57122340 tx=0x7f728000bf50 comp rx=0 tx=0).stop 2024-06-04T05:03:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72987c5740 1 -- 172.21.15.70:0/3435231398 shutdown_connections 2024-06-04T05:03:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72987c5740 1 --2- 172.21.15.70:0/3435231398 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fe5718ad50 0x55fe5718d170 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72987c5740 1 --2- 172.21.15.70:0/3435231398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe5710fa00 0x55fe57114070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72987c5740 1 --2- 172.21.15.70:0/3435231398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fe57105140 0x55fe57113b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f726effd640 0 client.17214 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72987c5740 1 --2- 172.21.15.70:0/3435231398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe57104820 0x55fe571135f0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72987c5740 1 -- 172.21.15.70:0/3435231398 >> 172.21.15.70:0/3435231398 conn(0x55fe57100d30 msgr2=0x55fe5710e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72987c5740 1 -- 172.21.15.70:0/3435231398 shutdown_connections 2024-06-04T05:03:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72987c5740 1 -- 172.21.15.70:0/3435231398 wait complete. 2024-06-04T05:03:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72987c5740 20 client.17214 trim_cache size 0 max 16384 2024-06-04T05:03:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:03:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:03:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:03:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72959d0640 1 -- 172.21.15.70:0/928092642 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f72780b2a10 msgr2=0x7f72780b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72959d0640 1 --2- 172.21.15.70:0/928092642 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f72780b2a10 0x7f72780b4ed0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f728804d000 tx=0x7f728804b000 comp rx=0 tx=0).stop 2024-06-04T05:03:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72959d0640 1 -- 172.21.15.70:0/928092642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72901033f0 msgr2=0x7f72901996b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72959d0640 1 --2- 172.21.15.70:0/928092642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72901033f0 0x7f72901996b0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f728400b4a0 tx=0x7f728400b960 comp rx=0 tx=0).stop 2024-06-04T05:03:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72959d0640 1 -- 172.21.15.70:0/928092642 shutdown_connections 2024-06-04T05:03:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72959d0640 1 --2- 172.21.15.70:0/928092642 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f72780b2a10 0x7f72780b4ed0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72959d0640 1 --2- 172.21.15.70:0/928092642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7290108a40 0x7f729019d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72959d0640 1 --2- 172.21.15.70:0/928092642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72901033f0 0x7f72901996b0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72959d0640 1 --2- 172.21.15.70:0/928092642 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7290102a30 0x7f7290199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72959d0640 1 -- 172.21.15.70:0/928092642 >> 172.21.15.70:0/928092642 conn(0x7f72900fe500 msgr2=0x7f72900fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72959d0640 1 -- 172.21.15.70:0/928092642 shutdown_connections 2024-06-04T05:03:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:44.799+0000 7f72959d0640 1 -- 172.21.15.70:0/928092642 wait complete. 2024-06-04T05:03:45.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.048+0000 7f698c78f700 1 -- 172.21.15.70:0/518294859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f698406bcb0 msgr2=0x7f698406c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:45.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.048+0000 7f698c78f700 1 --2- 172.21.15.70:0/518294859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f698406bcb0 0x7f698406c130 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f6974009990 tx=0x7f6974009ca0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.048+0000 7f698c78f700 1 -- 172.21.15.70:0/518294859 shutdown_connections 2024-06-04T05:03:45.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.048+0000 7f698c78f700 1 --2- 172.21.15.70:0/518294859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f698406c670 0x7f6984110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.048+0000 7f698c78f700 1 --2- 172.21.15.70:0/518294859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6984109620 0x7f6984109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.048+0000 7f698c78f700 1 --2- 172.21.15.70:0/518294859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f698406bcb0 0x7f698406c130 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.048+0000 7f698c78f700 1 -- 172.21.15.70:0/518294859 >> 172.21.15.70:0/518294859 conn(0x7f6984078490 msgr2=0x7f69840788a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:45.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.048+0000 7f698c78f700 1 -- 172.21.15.70:0/518294859 shutdown_connections 2024-06-04T05:03:45.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.048+0000 7f698c78f700 1 -- 172.21.15.70:0/518294859 wait complete. 2024-06-04T05:03:45.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f698c78f700 1 Processor -- start 2024-06-04T05:03:45.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f698c78f700 1 -- start start 2024-06-04T05:03:45.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f698c78f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f698406bcb0 0x7f698419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f698c78f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f698406c670 0x7f698419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f698c78f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6984109620 0x7f6984199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f698c78f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f69841135f0 con 0x7f698406bcb0 2024-06-04T05:03:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f698c78f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6984113470 con 0x7f698406c670 2024-06-04T05:03:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f698c78f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6984113770 con 0x7f6984109620 2024-06-04T05:03:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f698a52b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f698406bcb0 0x7f698419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f698a52b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f698406bcb0 0x7f698419f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48856/0 (socket says 172.21.15.70:48856) 2024-06-04T05:03:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f698a52b700 1 -- 172.21.15.70:0/747860766 learned_addr learned my addr 172.21.15.70:0/747860766 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f6989d2a700 1 --2- 172.21.15.70:0/747860766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f698406c670 0x7f698419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f698ad2c700 1 --2- 172.21.15.70:0/747860766 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6984109620 0x7f6984199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f698a52b700 1 -- 172.21.15.70:0/747860766 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6984109620 msgr2=0x7f6984199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f698a52b700 1 --2- 172.21.15.70:0/747860766 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6984109620 0x7f6984199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f698a52b700 1 -- 172.21.15.70:0/747860766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f698406c670 msgr2=0x7f698419fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f698a52b700 1 --2- 172.21.15.70:0/747860766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f698406c670 0x7f698419fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.049+0000 7f698a52b700 1 -- 172.21.15.70:0/747860766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6984199fd0 con 0x7f698406bcb0 2024-06-04T05:03:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.050+0000 7f698a52b700 1 --2- 172.21.15.70:0/747860766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f698406bcb0 0x7f698419f6b0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f698000c9b0 tx=0x7f698000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.050+0000 7f697b7fe700 1 -- 172.21.15.70:0/747860766 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6980007b00 con 0x7f698406bcb0 2024-06-04T05:03:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.050+0000 7f698c78f700 1 -- 172.21.15.70:0/747860766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f698419a2c0 con 0x7f698406bcb0 2024-06-04T05:03:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.050+0000 7f697b7fe700 1 -- 172.21.15.70:0/747860766 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f698000ce50 con 0x7f698406bcb0 2024-06-04T05:03:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.050+0000 7f697b7fe700 1 -- 172.21.15.70:0/747860766 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6980018660 con 0x7f698406bcb0 2024-06-04T05:03:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.050+0000 7f698c78f700 1 -- 172.21.15.70:0/747860766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69841024d0 con 0x7f698406bcb0 2024-06-04T05:03:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.052+0000 7f697b7fe700 1 -- 172.21.15.70:0/747860766 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f698000f450 con 0x7f698406bcb0 2024-06-04T05:03:45.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.053+0000 7f697b7fe700 1 --2- 172.21.15.70:0/747860766 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f69700b29f0 0x7f69700b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.053+0000 7f6989d2a700 1 --2- 172.21.15.70:0/747860766 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f69700b29f0 0x7f69700b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:45.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.053+0000 7f697b7fe700 1 -- 172.21.15.70:0/747860766 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f69800e2550 con 0x7f698406bcb0 2024-06-04T05:03:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.053+0000 7f6997024b80 1 Processor -- start 2024-06-04T05:03:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.053+0000 7f6989d2a700 1 --2- 172.21.15.70:0/747860766 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f69700b29f0 0x7f69700b4eb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f697400b5c0 tx=0x7f6974005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f6997024b80 1 -- start start 2024-06-04T05:03:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f6997024b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f691193720 0x55f691193b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f6997024b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118ebf0 0x55f69118f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f6997024b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f69119b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f6997024b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f6910ece70 con 0x55f69118f5b0 2024-06-04T05:03:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f6997024b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f6910dca10 con 0x55f69118ebf0 2024-06-04T05:03:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f6997024b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f6910e0a50 con 0x55f691193720 2024-06-04T05:03:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f698ad2c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f69119b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f698ad2c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f69119b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48858/0 (socket says 172.21.15.70:48858) 2024-06-04T05:03:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f698ad2c700 1 -- 172.21.15.70:0/3547321219 learned_addr learned my addr 172.21.15.70:0/3547321219 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f6989d2a700 1 --2- 172.21.15.70:0/3547321219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118ebf0 0x55f69118f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f698a52b700 1 --2- 172.21.15.70:0/3547321219 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f691193720 0x55f691193b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f698ad2c700 1 -- 172.21.15.70:0/3547321219 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f691193720 msgr2=0x55f691193b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f698ad2c700 1 --2- 172.21.15.70:0/3547321219 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f691193720 0x55f691193b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f698ad2c700 1 -- 172.21.15.70:0/3547321219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118ebf0 msgr2=0x55f69118f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f698ad2c700 1 --2- 172.21.15.70:0/3547321219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118ebf0 0x55f69118f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f698ad2c700 1 -- 172.21.15.70:0/3547321219 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6910860b0 con 0x55f69118f5b0 2024-06-04T05:03:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.054+0000 7f6989d2a700 1 --2- 172.21.15.70:0/3547321219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118ebf0 0x55f69118f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:03:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.055+0000 7f698ad2c700 1 --2- 172.21.15.70:0/3547321219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f69119b620 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f697c0098f0 tx=0x7f697c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce5613d5b124c39 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.055+0000 7f6978ff9700 1 -- 172.21.15.70:0/3547321219 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c004bb0 con 0x55f69118f5b0 2024-06-04T05:03:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.055+0000 7f6978ff9700 1 -- 172.21.15.70:0/3547321219 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f697c004d50 con 0x55f69118f5b0 2024-06-04T05:03:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.055+0000 7f6978ff9700 1 -- 172.21.15.70:0/3547321219 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c007db0 con 0x55f69118f5b0 2024-06-04T05:03:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 1 -- 172.21.15.70:0/3547321219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 msgr2=0x55f69119b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 1 --2- 172.21.15.70:0/3547321219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f69119b620 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f697c0098f0 tx=0x7f697c009c00 comp rx=0 tx=0).stop 2024-06-04T05:03:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 1 -- 172.21.15.70:0/3547321219 shutdown_connections 2024-06-04T05:03:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 1 --2- 172.21.15.70:0/3547321219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f69119b620 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 1 --2- 172.21.15.70:0/3547321219 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f691193720 0x55f691193b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 1 --2- 172.21.15.70:0/3547321219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118ebf0 0x55f69118f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 1 -- 172.21.15.70:0/3547321219 >> 172.21.15.70:0/3547321219 conn(0x55f6910ba340 msgr2=0x55f6911898e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 1 -- 172.21.15.70:0/3547321219 shutdown_connections 2024-06-04T05:03:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 1 -- 172.21.15.70:0/3547321219 wait complete. 2024-06-04T05:03:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 1 -- start start 2024-06-04T05:03:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 1 Processor -- start 2024-06-04T05:03:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118ebf0 0x55f69119b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f6911985d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f691193720 0x55f691198b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f690fe30a0 con 0x55f69118f5b0 2024-06-04T05:03:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f691044ea0 con 0x55f69118ebf0 2024-06-04T05:03:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f69119bc80 con 0x55f691193720 2024-06-04T05:03:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.056+0000 7f6997024b80 10 client.0 _ll_register_callbacks cb 0x55f691077a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f6989d2a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f6911985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f6989d2a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f6911985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48874/0 (socket says 172.21.15.70:48874) 2024-06-04T05:03:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f6989d2a700 1 -- 172.21.15.70:0/2743825242 learned_addr learned my addr 172.21.15.70:0/2743825242 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f698a52b700 1 --2- 172.21.15.70:0/2743825242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118ebf0 0x55f69119b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f698ad2c700 1 --2- 172.21.15.70:0/2743825242 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f691193720 0x55f691198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f6989d2a700 1 -- 172.21.15.70:0/2743825242 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f691193720 msgr2=0x55f691198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f6989d2a700 1 --2- 172.21.15.70:0/2743825242 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f691193720 0x55f691198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f6989d2a700 1 -- 172.21.15.70:0/2743825242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118ebf0 msgr2=0x55f69119b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f6989d2a700 1 --2- 172.21.15.70:0/2743825242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118ebf0 0x55f69119b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f6989d2a700 1 -- 172.21.15.70:0/2743825242 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f69102c9d0 con 0x55f69118f5b0 2024-06-04T05:03:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f6989d2a700 1 --2- 172.21.15.70:0/2743825242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f6911985d0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f6974015000 tx=0x7f6974009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654e371a4ecfebac server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f696dffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f696dffb700 1 -- 172.21.15.70:0/2743825242 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974034060 con 0x55f69118f5b0 2024-06-04T05:03:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f696dffb700 1 -- 172.21.15.70:0/2743825242 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f697400fd00 con 0x55f69118f5b0 2024-06-04T05:03:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f6997024b80 1 -- 172.21.15.70:0/2743825242 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f697c004d50 con 0x55f69118f5b0 2024-06-04T05:03:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f696dffb700 1 -- 172.21.15.70:0/2743825242 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6974024420 con 0x55f69118f5b0 2024-06-04T05:03:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f6997024b80 10 client.17226 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.057+0000 7f6997024b80 1 -- 172.21.15.70:0/2743825242 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f691096860 con 0x55f69118f5b0 2024-06-04T05:03:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.058+0000 7f696dffb700 1 -- 172.21.15.70:0/2743825242 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f697c004d50 con 0x55f69118f5b0 2024-06-04T05:03:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.058+0000 7f696dffb700 1 -- 172.21.15.70:0/2743825242 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f697402f3f0 con 0x55f69118f5b0 2024-06-04T05:03:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.058+0000 7f6997024b80 10 client.17226 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.058+0000 7f6997024b80 10 client.17226 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:03:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.058+0000 7f6997024b80 20 client.17226 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.058+0000 7f6997024b80 1 --2- 172.21.15.70:0/2743825242 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f6911a8820 0x55f6911aac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.058+0000 7f6997024b80 4 client.17226 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:03:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.058+0000 7f6997024b80 1 -- 172.21.15.70:0/2743825242 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6910eedc0 con 0x55f6911a8820 2024-06-04T05:03:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.058+0000 7f698a52b700 1 --2- 172.21.15.70:0/2743825242 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f6911a8820 0x55f6911aac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.058+0000 7f696dffb700 10 client.17226 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.058+0000 7f698a52b700 1 --2- 172.21.15.70:0/2743825242 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f6911a8820 0x55f6911aac30 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb3a0c0932f3d905 server_cookie=e03e5ad43a84756f in_seq=0 out_seq=0 2024-06-04T05:03:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.059+0000 7f696dffb700 1 -- 172.21.15.70:0/2743825242 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f6910eedc0 con 0x55f6911a8820 2024-06-04T05:03:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.059+0000 7f696dffb700 10 client.17226 handle_command_reply: tid=0 2024-06-04T05:03:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.059+0000 7f6997024b80 1 client.17226 shutdown 2024-06-04T05:03:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.059+0000 7f6997024b80 1 -- 172.21.15.70:0/2743825242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 msgr2=0x55f6911985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.059+0000 7f6997024b80 1 --2- 172.21.15.70:0/2743825242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f6911985d0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f6974015000 tx=0x7f6974009310 comp rx=0 tx=0).stop 2024-06-04T05:03:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.059+0000 7f6997024b80 1 -- 172.21.15.70:0/2743825242 shutdown_connections 2024-06-04T05:03:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.059+0000 7f6997024b80 1 --2- 172.21.15.70:0/2743825242 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f691193720 0x55f691198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.059+0000 7f6997024b80 1 --2- 172.21.15.70:0/2743825242 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f6911a8820 0x55f6911aac30 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.059+0000 7f6997024b80 1 --2- 172.21.15.70:0/2743825242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118ebf0 0x55f69119b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.059+0000 7f696dffb700 0 client.17226 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:45.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.059+0000 7f6997024b80 1 --2- 172.21.15.70:0/2743825242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f6911985d0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.059+0000 7f6997024b80 1 -- 172.21.15.70:0/2743825242 >> 172.21.15.70:0/2743825242 conn(0x55f6910ba340 msgr2=0x55f69118dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:45.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.059+0000 7f6997024b80 1 -- 172.21.15.70:0/2743825242 shutdown_connections 2024-06-04T05:03:45.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.059+0000 7f6997024b80 1 -- 172.21.15.70:0/2743825242 wait complete. 2024-06-04T05:03:45.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.059+0000 7f6997024b80 20 client.17226 trim_cache size 0 max 16384 2024-06-04T05:03:45.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.081+0000 7f6997024b80 1 Processor -- start 2024-06-04T05:03:45.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.081+0000 7f6997024b80 1 -- start start 2024-06-04T05:03:45.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.081+0000 7f6997024b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118ebf0 0x55f691183e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.081+0000 7f6997024b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f69118e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.081+0000 7f6997024b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f691193720 0x55f6911b0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.081+0000 7f6997024b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f69841afe20 con 0x55f69118f5b0 2024-06-04T05:03:45.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.081+0000 7f6997024b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f69841b1960 con 0x55f69118ebf0 2024-06-04T05:03:45.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.081+0000 7f6997024b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f69841b04f0 con 0x55f691193720 2024-06-04T05:03:45.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.081+0000 7f6989d2a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f69118e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:45.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.081+0000 7f6989d2a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f69118e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48882/0 (socket says 172.21.15.70:48882) 2024-06-04T05:03:45.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.081+0000 7f6989d2a700 1 -- 172.21.15.70:0/3161572239 learned_addr learned my addr 172.21.15.70:0/3161572239 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:45.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.082+0000 7f698a52b700 1 --2- 172.21.15.70:0/3161572239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118ebf0 0x55f691183e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:45.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.082+0000 7f6989d2a700 1 -- 172.21.15.70:0/3161572239 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f691193720 msgr2=0x55f6911b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:45.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.082+0000 7f6989d2a700 1 --2- 172.21.15.70:0/3161572239 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f691193720 0x55f6911b0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.082+0000 7f6989d2a700 1 -- 172.21.15.70:0/3161572239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118ebf0 msgr2=0x55f691183e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:45.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.082+0000 7f6989d2a700 1 --2- 172.21.15.70:0/3161572239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118ebf0 0x55f691183e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.082+0000 7f6989d2a700 1 -- 172.21.15.70:0/3161572239 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f69118b3f0 con 0x55f69118f5b0 2024-06-04T05:03:45.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.082+0000 7f698a52b700 1 --2- 172.21.15.70:0/3161572239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118ebf0 0x55f691183e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:03:45.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.082+0000 7f6989d2a700 1 --2- 172.21.15.70:0/3161572239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f69118e2f0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f697402f890 tx=0x7f69740059f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7cf01a06fae5337 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:45.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.082+0000 7f696ffff700 1 -- 172.21.15.70:0/3161572239 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974030e60 con 0x55f69118f5b0 2024-06-04T05:03:45.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.082+0000 7f696ffff700 1 -- 172.21.15.70:0/3161572239 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6974028070 con 0x55f69118f5b0 2024-06-04T05:03:45.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.082+0000 7f696ffff700 1 -- 172.21.15.70:0/3161572239 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974034040 con 0x55f69118f5b0 2024-06-04T05:03:45.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.083+0000 7f6997024b80 1 -- 172.21.15.70:0/3161572239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 msgr2=0x55f69118e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:45.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.083+0000 7f6997024b80 1 --2- 172.21.15.70:0/3161572239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f69118e2f0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f697402f890 tx=0x7f69740059f0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.083+0000 7f6997024b80 1 -- 172.21.15.70:0/3161572239 shutdown_connections 2024-06-04T05:03:45.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.083+0000 7f6997024b80 1 --2- 172.21.15.70:0/3161572239 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f691193720 0x55f6911b0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.083+0000 7f6997024b80 1 --2- 172.21.15.70:0/3161572239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118ebf0 0x55f691183e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.083+0000 7f6997024b80 1 --2- 172.21.15.70:0/3161572239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f69118f5b0 0x55f69118e2f0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.083+0000 7f6997024b80 1 -- 172.21.15.70:0/3161572239 >> 172.21.15.70:0/3161572239 conn(0x55f6910ba340 msgr2=0x55f69118dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:45.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.083+0000 7f6997024b80 1 -- 172.21.15.70:0/3161572239 shutdown_connections 2024-06-04T05:03:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.083+0000 7f6997024b80 1 -- 172.21.15.70:0/3161572239 wait complete. 2024-06-04T05:03:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.083+0000 7f6997024b80 1 -- start start 2024-06-04T05:03:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.083+0000 7f6997024b80 1 Processor -- start 2024-06-04T05:03:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f6997024b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f69118ebf0 0x55f6911a1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f6997024b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118f5b0 0x55f69119f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f6997024b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f691193720 0x55f69119fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f6997024b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f69841b1290 con 0x55f691193720 2024-06-04T05:03:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f6997024b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f69841b0bc0 con 0x55f69118f5b0 2024-06-04T05:03:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f6997024b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f69841adde0 con 0x55f69118ebf0 2024-06-04T05:03:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f6997024b80 10 client.0 _ll_register_callbacks cb 0x55f691077a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f698ad2c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f691193720 0x55f69119fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f698ad2c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f691193720 0x55f69119fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48890/0 (socket says 172.21.15.70:48890) 2024-06-04T05:03:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f698ad2c700 1 -- 172.21.15.70:0/2631104982 learned_addr learned my addr 172.21.15.70:0/2631104982 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f6989d2a700 1 --2- 172.21.15.70:0/2631104982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118f5b0 0x55f69119f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:45.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f698a52b700 1 --2- 172.21.15.70:0/2631104982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f69118ebf0 0x55f6911a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:45.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f698ad2c700 1 -- 172.21.15.70:0/2631104982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f69118ebf0 msgr2=0x55f6911a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:45.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f698ad2c700 1 --2- 172.21.15.70:0/2631104982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f69118ebf0 0x55f6911a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f698ad2c700 1 -- 172.21.15.70:0/2631104982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118f5b0 msgr2=0x55f69119f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:45.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f698ad2c700 1 --2- 172.21.15.70:0/2631104982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118f5b0 0x55f69119f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f698ad2c700 1 -- 172.21.15.70:0/2631104982 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6911a0080 con 0x55f691193720 2024-06-04T05:03:45.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f698ad2c700 1 --2- 172.21.15.70:0/2631104982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f691193720 0x55f69119fb40 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f697c007730 tx=0x7f697c003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875c7c22f4442182 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:45.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f696dffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:45.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f696dffb700 1 -- 172.21.15.70:0/2631104982 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c023020 con 0x55f691193720 2024-06-04T05:03:45.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f696dffb700 1 -- 172.21.15.70:0/2631104982 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f697c004ca0 con 0x55f691193720 2024-06-04T05:03:45.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.084+0000 7f6997024b80 1 -- 172.21.15.70:0/2631104982 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6974028070 con 0x55f691193720 2024-06-04T05:03:45.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.085+0000 7f696dffb700 1 -- 172.21.15.70:0/2631104982 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f697c02e050 con 0x55f691193720 2024-06-04T05:03:45.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.085+0000 7f6997024b80 10 client.17232 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:45.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.085+0000 7f6997024b80 1 -- 172.21.15.70:0/2631104982 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f691214170 con 0x55f691193720 2024-06-04T05:03:45.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.085+0000 7f696dffb700 1 -- 172.21.15.70:0/2631104982 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6974028070 con 0x55f691193720 2024-06-04T05:03:45.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.085+0000 7f696dffb700 1 -- 172.21.15.70:0/2631104982 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f697c015610 con 0x55f691193720 2024-06-04T05:03:45.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.085+0000 7f6997024b80 10 client.17232 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:45.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.085+0000 7f6997024b80 10 client.17232 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:03:45.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.085+0000 7f6997024b80 20 client.17232 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:45.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.085+0000 7f6997024b80 1 --2- 172.21.15.70:0/2631104982 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f691214490 0x55f6912168b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:45.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.085+0000 7f6997024b80 4 client.17232 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "70640af4", "release": true, "await": true}] 2024-06-04T05:03:45.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.085+0000 7f6997024b80 1 -- 172.21.15.70:0/2631104982 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "70640af4", "release": true, "await": true}) -- 0x55f6910e85a0 con 0x55f691214490 2024-06-04T05:03:45.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.085+0000 7f698a52b700 1 --2- 172.21.15.70:0/2631104982 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f691214490 0x55f6912168b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:45.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.086+0000 7f698a52b700 1 --2- 172.21.15.70:0/2631104982 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f691214490 0x55f6912168b0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=183130b92a275c97 server_cookie=a4d03072acfb4ed8 in_seq=0 out_seq=0 2024-06-04T05:03:45.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.086+0000 7f696dffb700 10 client.17232 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:45.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.092+0000 7f696dffb700 1 -- 172.21.15.70:0/2631104982 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55f6910e85a0 con 0x55f691214490 2024-06-04T05:03:45.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.092+0000 7f696dffb700 10 client.17232 handle_command_reply: tid=0 2024-06-04T05:03:45.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.092+0000 7f6997024b80 1 client.17232 shutdown 2024-06-04T05:03:45.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.092+0000 7f6997024b80 1 -- 172.21.15.70:0/2631104982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f691193720 msgr2=0x55f69119fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:45.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.092+0000 7f6997024b80 1 --2- 172.21.15.70:0/2631104982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f691193720 0x55f69119fb40 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f697c007730 tx=0x7f697c003f90 comp rx=0 tx=0).stop 2024-06-04T05:03:45.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.092+0000 7f6997024b80 1 -- 172.21.15.70:0/2631104982 shutdown_connections 2024-06-04T05:03:45.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.092+0000 7f6997024b80 1 --2- 172.21.15.70:0/2631104982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f691193720 0x55f69119fb40 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.092+0000 7f6997024b80 1 --2- 172.21.15.70:0/2631104982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f69118ebf0 0x55f6911a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.092+0000 7f6997024b80 1 --2- 172.21.15.70:0/2631104982 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f691214490 0x55f6912168b0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.093+0000 7f6997024b80 1 --2- 172.21.15.70:0/2631104982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f69118f5b0 0x55f69119f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.093+0000 7f6997024b80 1 -- 172.21.15.70:0/2631104982 >> 172.21.15.70:0/2631104982 conn(0x55f6910ba340 msgr2=0x55f691199e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:45.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.093+0000 7f696dffb700 0 client.17232 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:45.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.093+0000 7f6997024b80 1 -- 172.21.15.70:0/2631104982 shutdown_connections 2024-06-04T05:03:45.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.093+0000 7f6997024b80 1 -- 172.21.15.70:0/2631104982 wait complete. 2024-06-04T05:03:45.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.093+0000 7f6997024b80 20 client.17232 trim_cache size 0 max 16384 2024-06-04T05:03:45.097 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:03:45.097 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 32, 2024-06-04T05:03:45.098 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:03:45.098 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 16, 2024-06-04T05:03:45.098 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:03:45.098 INFO:teuthology.orchestra.run.smithi070.stdout: "70640af4": { 2024-06-04T05:03:45.098 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 16, 2024-06-04T05:03:45.098 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:03:45.098 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:03:45.098 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:03:45.098 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:03:45.098 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:03:45.098 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:03:45.098 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 138.9, 2024-06-04T05:03:45.098 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:03:45.099 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:03:45.099 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:03:45.099 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:03:45.099 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:03:45.099 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:03:45.099 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:03:45.099 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:03:45.099 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:03:45.099 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:03:45.099 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:03:45.099 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:03:45.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.093+0000 7f698c78f700 1 -- 172.21.15.70:0/747860766 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f69700b29f0 msgr2=0x7f69700b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:45.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.093+0000 7f698c78f700 1 --2- 172.21.15.70:0/747860766 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f69700b29f0 0x7f69700b4eb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f697400b5c0 tx=0x7f6974005960 comp rx=0 tx=0).stop 2024-06-04T05:03:45.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.093+0000 7f698c78f700 1 -- 172.21.15.70:0/747860766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f698406bcb0 msgr2=0x7f698419f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:45.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.093+0000 7f698c78f700 1 --2- 172.21.15.70:0/747860766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f698406bcb0 0x7f698419f6b0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f698000c9b0 tx=0x7f698000ccc0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.093+0000 7f698c78f700 1 -- 172.21.15.70:0/747860766 shutdown_connections 2024-06-04T05:03:45.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.093+0000 7f698c78f700 1 --2- 172.21.15.70:0/747860766 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f69700b29f0 0x7f69700b4eb0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.093+0000 7f698c78f700 1 --2- 172.21.15.70:0/747860766 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6984109620 0x7f6984199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.093+0000 7f698c78f700 1 --2- 172.21.15.70:0/747860766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f698406bcb0 0x7f698419f6b0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.093+0000 7f698c78f700 1 --2- 172.21.15.70:0/747860766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f698406c670 0x7f698419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:45.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.093+0000 7f698c78f700 1 -- 172.21.15.70:0/747860766 >> 172.21.15.70:0/747860766 conn(0x7f6984078490 msgr2=0x7f6984101130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:45.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.094+0000 7f698c78f700 1 -- 172.21.15.70:0/747860766 shutdown_connections 2024-06-04T05:03:45.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:45.094+0000 7f698c78f700 1 -- 172.21.15.70:0/747860766 wait complete. 2024-06-04T05:03:45.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 105891 2.06 MB/sec execute 158 sec latency 21587.215 ms 2024-06-04T05:03:45.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:45 smithi070 ceph-mon[32631]: pgmap v200: 97 pgs: 97 active+clean; 69 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 12 op/s 2024-06-04T05:03:45.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:45 smithi112 ceph-mon[36453]: pgmap v200: 97 pgs: 97 active+clean; 69 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 12 op/s 2024-06-04T05:03:45.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:45 smithi098 ceph-mon[35569]: pgmap v200: 97 pgs: 97 active+clean; 69 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 12 op/s 2024-06-04T05:03:46.015 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '70640af4', total seconds elapsed: 30.3 2024-06-04T05:03:46.015 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 18.95 2024-06-04T05:03:46.015 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:03:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.291+0000 7efddf8f2700 1 -- 172.21.15.70:0/2249882307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdd806b220 msgr2=0x7efdd806b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.291+0000 7efddf8f2700 1 --2- 172.21.15.70:0/2249882307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdd806b220 0x7efdd806b6a0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7efdc8009990 tx=0x7efdc8009ca0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.291+0000 7efddf8f2700 1 -- 172.21.15.70:0/2249882307 shutdown_connections 2024-06-04T05:03:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.291+0000 7efddf8f2700 1 --2- 172.21.15.70:0/2249882307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd806bbe0 0x7efdd81103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.291+0000 7efddf8f2700 1 --2- 172.21.15.70:0/2249882307 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efdd81095b0 0x7efdd8109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.291+0000 7efddf8f2700 1 --2- 172.21.15.70:0/2249882307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdd806b220 0x7efdd806b6a0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.291+0000 7efddf8f2700 1 -- 172.21.15.70:0/2249882307 >> 172.21.15.70:0/2249882307 conn(0x7efdd80793f0 msgr2=0x7efdd8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:46.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.292+0000 7efddf8f2700 1 -- 172.21.15.70:0/2249882307 shutdown_connections 2024-06-04T05:03:46.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.292+0000 7efddf8f2700 1 -- 172.21.15.70:0/2249882307 wait complete. 2024-06-04T05:03:46.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.292+0000 7efddf8f2700 1 Processor -- start 2024-06-04T05:03:46.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.292+0000 7efddf8f2700 1 -- start start 2024-06-04T05:03:46.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.293+0000 7efddf8f2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efdd806b220 0x7efdd8102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.293+0000 7efddf8f2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd806bbe0 0x7efdd8102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.293+0000 7efddf8f2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdd81095b0 0x7efdd8102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.293+0000 7efddf8f2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efdd8113570 con 0x7efdd81095b0 2024-06-04T05:03:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.293+0000 7efddf8f2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdd81133f0 con 0x7efdd806bbe0 2024-06-04T05:03:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.293+0000 7efddf8f2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efdd81136f0 con 0x7efdd806b220 2024-06-04T05:03:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.293+0000 7efddd68e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efdd806b220 0x7efdd8102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.293+0000 7efddde8f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdd81095b0 0x7efdd8102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.293+0000 7efddd68e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efdd806b220 0x7efdd8102510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:57766/0 (socket says 172.21.15.70:57766) 2024-06-04T05:03:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.293+0000 7efddd68e700 1 -- 172.21.15.70:0/1987681748 learned_addr learned my addr 172.21.15.70:0/1987681748 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.293+0000 7efddd68e700 1 -- 172.21.15.70:0/1987681748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd806bbe0 msgr2=0x7efdd8102a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.293+0000 7efddd68e700 1 --2- 172.21.15.70:0/1987681748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd806bbe0 0x7efdd8102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.293+0000 7efddd68e700 1 -- 172.21.15.70:0/1987681748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdd81095b0 msgr2=0x7efdd8102f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.293+0000 7efddd68e700 1 --2- 172.21.15.70:0/1987681748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdd81095b0 0x7efdd8102f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.293+0000 7efddd68e700 1 -- 172.21.15.70:0/1987681748 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd81a8050 con 0x7efdd806b220 2024-06-04T05:03:46.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.293+0000 7efddd68e700 1 --2- 172.21.15.70:0/1987681748 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efdd806b220 0x7efdd8102510 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efdd400e9a0 tx=0x7efdd400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:46.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.294+0000 7efdce7fc700 1 -- 172.21.15.70:0/1987681748 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd400cbd0 con 0x7efdd806b220 2024-06-04T05:03:46.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.294+0000 7efddf8f2700 1 -- 172.21.15.70:0/1987681748 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdd81a8340 con 0x7efdd806b220 2024-06-04T05:03:46.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.294+0000 7efdce7fc700 1 -- 172.21.15.70:0/1987681748 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efdd400cd70 con 0x7efdd806b220 2024-06-04T05:03:46.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.294+0000 7efddf8f2700 1 -- 172.21.15.70:0/1987681748 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdd81a88d0 con 0x7efdd806b220 2024-06-04T05:03:46.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.294+0000 7efdce7fc700 1 -- 172.21.15.70:0/1987681748 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd4018680 con 0x7efdd806b220 2024-06-04T05:03:46.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.296+0000 7efdce7fc700 1 -- 172.21.15.70:0/1987681748 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7efdd4018820 con 0x7efdd806b220 2024-06-04T05:03:46.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.297+0000 7efdce7fc700 1 --2- 172.21.15.70:0/1987681748 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efdc40b2a70 0x7efdc40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.297+0000 7efddce8d700 1 --2- 172.21.15.70:0/1987681748 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efdc40b2a70 0x7efdc40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:46.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.297+0000 7efdce7fc700 1 -- 172.21.15.70:0/1987681748 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7efdd40e2690 con 0x7efdd806b220 2024-06-04T05:03:46.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.298+0000 7efdea187b80 1 Processor -- start 2024-06-04T05:03:46.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.298+0000 7efdea187b80 1 -- start start 2024-06-04T05:03:46.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.298+0000 7efddce8d700 1 --2- 172.21.15.70:0/1987681748 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efdc40b2a70 0x7efdc40b4f30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efdc80098e0 tx=0x7efdc800b610 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:46.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.298+0000 7efdea187b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6e0f0 0x558abde6e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.298+0000 7efdea187b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde6eaa0 0x558abde721d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.298+0000 7efdea187b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde727f0 0x558abde72c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.298+0000 7efdea187b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558abddc4d10 con 0x558abde727f0 2024-06-04T05:03:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.298+0000 7efdea187b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558abddb48b0 con 0x558abde6e0f0 2024-06-04T05:03:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.298+0000 7efdea187b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558abddb88f0 con 0x558abde6eaa0 2024-06-04T05:03:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.298+0000 7efddce8d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde6eaa0 0x558abde721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.298+0000 7efddde8f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde727f0 0x558abde72c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.298+0000 7efddce8d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde6eaa0 0x558abde721d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:57770/0 (socket says 172.21.15.70:57770) 2024-06-04T05:03:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.298+0000 7efddce8d700 1 -- 172.21.15.70:0/3298348788 learned_addr learned my addr 172.21.15.70:0/3298348788 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.298+0000 7efddd68e700 1 --2- 172.21.15.70:0/3298348788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6e0f0 0x558abde6e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.299+0000 7efddce8d700 1 -- 172.21.15.70:0/3298348788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6e0f0 msgr2=0x558abde6e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.299+0000 7efddce8d700 1 --2- 172.21.15.70:0/3298348788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6e0f0 0x558abde6e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.299+0000 7efddce8d700 1 -- 172.21.15.70:0/3298348788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde727f0 msgr2=0x558abde72c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.299+0000 7efddce8d700 1 --2- 172.21.15.70:0/3298348788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde727f0 0x558abde72c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.299+0000 7efddce8d700 1 -- 172.21.15.70:0/3298348788 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558abdd04950 con 0x558abde6eaa0 2024-06-04T05:03:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.299+0000 7efddce8d700 1 --2- 172.21.15.70:0/3298348788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde6eaa0 0x558abde721d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efdc8009990 tx=0x7efdc801d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=379ee32e622f750 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.299+0000 7efdc37fe700 1 -- 172.21.15.70:0/3298348788 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc80056c0 con 0x558abde6eaa0 2024-06-04T05:03:46.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.299+0000 7efdc37fe700 1 -- 172.21.15.70:0/3298348788 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efdc800b6b0 con 0x558abde6eaa0 2024-06-04T05:03:46.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.299+0000 7efdc37fe700 1 -- 172.21.15.70:0/3298348788 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc802e460 con 0x558abde6eaa0 2024-06-04T05:03:46.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.300+0000 7efdea187b80 1 -- 172.21.15.70:0/3298348788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde6eaa0 msgr2=0x558abde721d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:46.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.300+0000 7efdea187b80 1 --2- 172.21.15.70:0/3298348788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde6eaa0 0x558abde721d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efdc8009990 tx=0x7efdc801d000 comp rx=0 tx=0).stop 2024-06-04T05:03:46.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.300+0000 7efdea187b80 1 -- 172.21.15.70:0/3298348788 shutdown_connections 2024-06-04T05:03:46.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.300+0000 7efdea187b80 1 --2- 172.21.15.70:0/3298348788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde727f0 0x558abde72c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.300+0000 7efdea187b80 1 --2- 172.21.15.70:0/3298348788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6e0f0 0x558abde6e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.300+0000 7efdea187b80 1 --2- 172.21.15.70:0/3298348788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde6eaa0 0x558abde721d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.300+0000 7efdea187b80 1 -- 172.21.15.70:0/3298348788 >> 172.21.15.70:0/3298348788 conn(0x558abdd93100 msgr2=0x558abde5ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:46.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.300+0000 7efdea187b80 1 -- 172.21.15.70:0/3298348788 shutdown_connections 2024-06-04T05:03:46.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.300+0000 7efdea187b80 1 -- 172.21.15.70:0/3298348788 wait complete. 2024-06-04T05:03:46.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.301+0000 7efdea187b80 1 -- start start 2024-06-04T05:03:46.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.301+0000 7efdea187b80 1 Processor -- start 2024-06-04T05:03:46.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.301+0000 7efdea187b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde6e0f0 0x558abde79bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.301+0000 7efdea187b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6eaa0 0x558abde76ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.301+0000 7efdea187b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde727f0 0x558abde77400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.301+0000 7efdea187b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558abdcbb020 con 0x558abde6e0f0 2024-06-04T05:03:46.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.301+0000 7efdea187b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558abdd56840 con 0x558abde6eaa0 2024-06-04T05:03:46.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.301+0000 7efdea187b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558abde7a950 con 0x558abde727f0 2024-06-04T05:03:46.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.302+0000 7efdea187b80 10 client.0 _ll_register_callbacks cb 0x558abdd4f9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:46.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.302+0000 7efddde8f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde727f0 0x558abde77400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:46.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.302+0000 7efddde8f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde727f0 0x558abde77400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:57772/0 (socket says 172.21.15.70:57772) 2024-06-04T05:03:46.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.302+0000 7efddde8f700 1 -- 172.21.15.70:0/289277669 learned_addr learned my addr 172.21.15.70:0/289277669 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:46.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.302+0000 7efddd68e700 1 --2- 172.21.15.70:0/289277669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde6e0f0 0x558abde79bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:46.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.302+0000 7efddce8d700 1 --2- 172.21.15.70:0/289277669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6eaa0 0x558abde76ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:46.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.302+0000 7efddde8f700 1 -- 172.21.15.70:0/289277669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6eaa0 msgr2=0x558abde76ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:46.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.302+0000 7efddde8f700 1 --2- 172.21.15.70:0/289277669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6eaa0 0x558abde76ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.302+0000 7efddde8f700 1 -- 172.21.15.70:0/289277669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde6e0f0 msgr2=0x558abde79bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:46.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.302+0000 7efddde8f700 1 --2- 172.21.15.70:0/289277669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde6e0f0 0x558abde79bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.302+0000 7efddde8f700 1 -- 172.21.15.70:0/289277669 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558abdd53710 con 0x558abde727f0 2024-06-04T05:03:46.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.302+0000 7efddd68e700 1 --2- 172.21.15.70:0/289277669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde6e0f0 0x558abde79bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:46.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.302+0000 7efddde8f700 1 --2- 172.21.15.70:0/289277669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde727f0 0x558abde77400 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efdd00042a0 tx=0x7efdd00042d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d2e5c8688c0e825 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:46.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.302+0000 7efdc0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:03:46.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.303+0000 7efdc0ff9700 1 -- 172.21.15.70:0/289277669 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd001a070 con 0x558abde727f0 2024-06-04T05:03:46.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.303+0000 7efdc0ff9700 1 -- 172.21.15.70:0/289277669 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7efdd0016840 con 0x558abde727f0 2024-06-04T05:03:46.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.303+0000 7efdea187b80 1 -- 172.21.15.70:0/289277669 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdc800b6b0 con 0x558abde727f0 2024-06-04T05:03:46.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.303+0000 7efdc0ff9700 1 -- 172.21.15.70:0/289277669 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7efdd0026050 con 0x558abde727f0 2024-06-04T05:03:46.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.303+0000 7efdea187b80 10 client.26554 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:46.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.303+0000 7efdea187b80 1 -- 172.21.15.70:0/289277669 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558abde850c0 con 0x558abde727f0 2024-06-04T05:03:46.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.303+0000 7efdc0ff9700 1 -- 172.21.15.70:0/289277669 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efdc800b6b0 con 0x558abde727f0 2024-06-04T05:03:46.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.303+0000 7efdc0ff9700 1 -- 172.21.15.70:0/289277669 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7efdd0007990 con 0x558abde727f0 2024-06-04T05:03:46.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.303+0000 7efdea187b80 10 client.26554 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:46.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.303+0000 7efdea187b80 10 client.26554 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:46.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.303+0000 7efdea187b80 20 client.26554 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:46.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.303+0000 7efdea187b80 1 --2- 172.21.15.70:0/289277669 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558abde876f0 0x558abde89b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.303+0000 7efdea187b80 4 client.26554 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:03:46.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.303+0000 7efdea187b80 1 -- 172.21.15.70:0/289277669 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558abde31a60 con 0x558abde876f0 2024-06-04T05:03:46.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.304+0000 7efddd68e700 1 --2- 172.21.15.70:0/289277669 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558abde876f0 0x558abde89b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:46.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.304+0000 7efddd68e700 1 --2- 172.21.15.70:0/289277669 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558abde876f0 0x558abde89b50 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f575db8bc36de863 server_cookie=6646d6fdd07f8eee in_seq=0 out_seq=0 2024-06-04T05:03:46.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.304+0000 7efdc0ff9700 10 client.26554 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:46.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.305+0000 7efdc0ff9700 1 -- 172.21.15.70:0/289277669 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558abde31a60 con 0x558abde876f0 2024-06-04T05:03:46.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.305+0000 7efdc0ff9700 10 client.26554 handle_command_reply: tid=0 2024-06-04T05:03:46.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.305+0000 7efdea187b80 1 client.26554 shutdown 2024-06-04T05:03:46.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.305+0000 7efdea187b80 1 -- 172.21.15.70:0/289277669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde727f0 msgr2=0x558abde77400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:46.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.305+0000 7efdea187b80 1 --2- 172.21.15.70:0/289277669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde727f0 0x558abde77400 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efdd00042a0 tx=0x7efdd00042d0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.305+0000 7efdea187b80 1 -- 172.21.15.70:0/289277669 shutdown_connections 2024-06-04T05:03:46.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.305+0000 7efdea187b80 1 --2- 172.21.15.70:0/289277669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde727f0 0x558abde77400 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.305+0000 7efdea187b80 1 --2- 172.21.15.70:0/289277669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde6e0f0 0x558abde79bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.305+0000 7efdea187b80 1 --2- 172.21.15.70:0/289277669 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558abde876f0 0x558abde89b50 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.305+0000 7efdea187b80 1 --2- 172.21.15.70:0/289277669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6eaa0 0x558abde76ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.305+0000 7efdc0ff9700 0 client.26554 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:46.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.305+0000 7efdea187b80 1 -- 172.21.15.70:0/289277669 >> 172.21.15.70:0/289277669 conn(0x558abdd93100 msgr2=0x558abde5f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:46.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.305+0000 7efdea187b80 1 -- 172.21.15.70:0/289277669 shutdown_connections 2024-06-04T05:03:46.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.305+0000 7efdea187b80 1 -- 172.21.15.70:0/289277669 wait complete. 2024-06-04T05:03:46.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.305+0000 7efdea187b80 20 client.26554 trim_cache size 0 max 16384 2024-06-04T05:03:46.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.333+0000 7efdea187b80 1 Processor -- start 2024-06-04T05:03:46.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.333+0000 7efdea187b80 1 -- start start 2024-06-04T05:03:46.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.333+0000 7efdea187b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde6e0f0 0x558abde78e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.333+0000 7efdea187b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6eaa0 0x558abde5fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.333+0000 7efdea187b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde727f0 0x558abde8f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.333+0000 7efdea187b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc00d640 con 0x558abde6e0f0 2024-06-04T05:03:46.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.333+0000 7efdea187b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc00f180 con 0x558abde6eaa0 2024-06-04T05:03:46.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.333+0000 7efdea187b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc00dd10 con 0x558abde727f0 2024-06-04T05:03:46.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.334+0000 7efddd68e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde6e0f0 0x558abde78e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:46.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.334+0000 7efddd68e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde6e0f0 0x558abde78e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49650/0 (socket says 172.21.15.70:49650) 2024-06-04T05:03:46.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.334+0000 7efddd68e700 1 -- 172.21.15.70:0/54534637 learned_addr learned my addr 172.21.15.70:0/54534637 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:46.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.334+0000 7efddde8f700 1 --2- 172.21.15.70:0/54534637 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde727f0 0x558abde8f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:46.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.334+0000 7efddce8d700 1 --2- 172.21.15.70:0/54534637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6eaa0 0x558abde5fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:46.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.334+0000 7efddd68e700 1 -- 172.21.15.70:0/54534637 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde727f0 msgr2=0x558abde8f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:46.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.334+0000 7efddd68e700 1 --2- 172.21.15.70:0/54534637 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde727f0 0x558abde8f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.334+0000 7efddd68e700 1 -- 172.21.15.70:0/54534637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6eaa0 msgr2=0x558abde5fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:46.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.334+0000 7efddd68e700 1 --2- 172.21.15.70:0/54534637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6eaa0 0x558abde5fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.334+0000 7efddd68e700 1 -- 172.21.15.70:0/54534637 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558abde72510 con 0x558abde6e0f0 2024-06-04T05:03:46.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.334+0000 7efddce8d700 1 --2- 172.21.15.70:0/54534637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6eaa0 0x558abde5fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:03:46.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.334+0000 7efddd68e700 1 --2- 172.21.15.70:0/54534637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde6e0f0 0x558abde78e00 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7efdd400e970 tx=0x7efdd400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66355dc3da69d618 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:46.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.334+0000 7efdc2ffd700 1 -- 172.21.15.70:0/54534637 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd40b2770 con 0x558abde6e0f0 2024-06-04T05:03:46.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.334+0000 7efdc2ffd700 1 -- 172.21.15.70:0/54534637 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efdd40af0c0 con 0x558abde6e0f0 2024-06-04T05:03:46.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.334+0000 7efdc2ffd700 1 -- 172.21.15.70:0/54534637 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd40cabc0 con 0x558abde6e0f0 2024-06-04T05:03:46.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.335+0000 7efdea187b80 1 -- 172.21.15.70:0/54534637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde6e0f0 msgr2=0x558abde78e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:46.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.335+0000 7efdea187b80 1 --2- 172.21.15.70:0/54534637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde6e0f0 0x558abde78e00 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7efdd400e970 tx=0x7efdd400cf20 comp rx=0 tx=0).stop 2024-06-04T05:03:46.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.335+0000 7efdea187b80 1 -- 172.21.15.70:0/54534637 shutdown_connections 2024-06-04T05:03:46.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.335+0000 7efdea187b80 1 --2- 172.21.15.70:0/54534637 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde727f0 0x558abde8f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.335+0000 7efdea187b80 1 --2- 172.21.15.70:0/54534637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde6e0f0 0x558abde78e00 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.335+0000 7efdea187b80 1 --2- 172.21.15.70:0/54534637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6eaa0 0x558abde5fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.335+0000 7efdea187b80 1 -- 172.21.15.70:0/54534637 >> 172.21.15.70:0/54534637 conn(0x558abdd93100 msgr2=0x558abde786f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:46.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.335+0000 7efdea187b80 1 -- 172.21.15.70:0/54534637 shutdown_connections 2024-06-04T05:03:46.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.335+0000 7efdea187b80 1 -- 172.21.15.70:0/54534637 wait complete. 2024-06-04T05:03:46.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.335+0000 7efdea187b80 1 -- start start 2024-06-04T05:03:46.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.336+0000 7efdea187b80 1 Processor -- start 2024-06-04T05:03:46.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.336+0000 7efdea187b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6e0f0 0x558abde7f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.336+0000 7efdea187b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde6eaa0 0x558abde7f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.336+0000 7efdea187b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde727f0 0x558abdef2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.336+0000 7efdea187b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc00eab0 con 0x558abde727f0 2024-06-04T05:03:46.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.336+0000 7efdea187b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc00e3e0 con 0x558abde6e0f0 2024-06-04T05:03:46.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.336+0000 7efddce8d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde6eaa0 0x558abde7f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:46.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.336+0000 7efddce8d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde6eaa0 0x558abde7f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:57788/0 (socket says 172.21.15.70:57788) 2024-06-04T05:03:46.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.336+0000 7efddce8d700 1 -- 172.21.15.70:0/502359311 learned_addr learned my addr 172.21.15.70:0/502359311 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:46.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.336+0000 7efdea187b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc00b5e0 con 0x558abde6eaa0 2024-06-04T05:03:46.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.336+0000 7efddde8f700 1 --2- 172.21.15.70:0/502359311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde727f0 0x558abdef2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:46.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.337+0000 7efdea187b80 10 client.0 _ll_register_callbacks cb 0x558abdd4f9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:46.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.337+0000 7efddd68e700 1 --2- 172.21.15.70:0/502359311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6e0f0 0x558abde7f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:46.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.337+0000 7efddde8f700 1 -- 172.21.15.70:0/502359311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde6eaa0 msgr2=0x558abde7f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:46.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.337+0000 7efddde8f700 1 --2- 172.21.15.70:0/502359311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde6eaa0 0x558abde7f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.337+0000 7efddde8f700 1 -- 172.21.15.70:0/502359311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6e0f0 msgr2=0x558abde7f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:46.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.337+0000 7efddde8f700 1 --2- 172.21.15.70:0/502359311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6e0f0 0x558abde7f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.337+0000 7efddde8f700 1 -- 172.21.15.70:0/502359311 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558abdef28a0 con 0x558abde727f0 2024-06-04T05:03:46.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.337+0000 7efddd68e700 1 --2- 172.21.15.70:0/502359311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6e0f0 0x558abde7f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:46.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.337+0000 7efddde8f700 1 --2- 172.21.15.70:0/502359311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde727f0 0x558abdef2310 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7efdd0007d50 tx=0x7efdd0004640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da95371f0a5a3ca9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:46.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.337+0000 7efdc0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:46.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.337+0000 7efdc0ff9700 1 -- 172.21.15.70:0/502359311 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd002e070 con 0x558abde727f0 2024-06-04T05:03:46.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.337+0000 7efdc0ff9700 1 -- 172.21.15.70:0/502359311 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7efdd00048f0 con 0x558abde727f0 2024-06-04T05:03:46.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.337+0000 7efdea187b80 1 -- 172.21.15.70:0/502359311 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdd40af0c0 con 0x558abde727f0 2024-06-04T05:03:46.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.338+0000 7efdc0ff9700 1 -- 172.21.15.70:0/502359311 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7efdd003a050 con 0x558abde727f0 2024-06-04T05:03:46.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.338+0000 7efdea187b80 10 client.17250 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:46.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.338+0000 7efdea187b80 1 -- 172.21.15.70:0/502359311 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558abdef3b50 con 0x558abde727f0 2024-06-04T05:03:46.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.338+0000 7efdc0ff9700 1 -- 172.21.15.70:0/502359311 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efdd40af0c0 con 0x558abde727f0 2024-06-04T05:03:46.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.338+0000 7efdc0ff9700 1 -- 172.21.15.70:0/502359311 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7efdd0016640 con 0x558abde727f0 2024-06-04T05:03:46.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.338+0000 7efdea187b80 10 client.17250 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:46.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.338+0000 7efdea187b80 10 client.17250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:46.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.338+0000 7efdea187b80 20 client.17250 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:46.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.338+0000 7efdea187b80 1 --2- 172.21.15.70:0/502359311 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558abdef4260 0x558abdef6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:46.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.338+0000 7efdea187b80 4 client.17250 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:03:46.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.338+0000 7efdea187b80 1 -- 172.21.15.70:0/502359311 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558abddc6c60 con 0x558abdef4260 2024-06-04T05:03:46.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.339+0000 7efddd68e700 1 --2- 172.21.15.70:0/502359311 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558abdef4260 0x558abdef6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:46.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.339+0000 7efdc0ff9700 10 client.17250 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:46.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.339+0000 7efddd68e700 1 --2- 172.21.15.70:0/502359311 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558abdef4260 0x558abdef6680 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c910462b72bb5a3 server_cookie=88c8239ff4178419 in_seq=0 out_seq=0 2024-06-04T05:03:46.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.340+0000 7efdc0ff9700 1 -- 172.21.15.70:0/502359311 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558abddc6c60 con 0x558abdef4260 2024-06-04T05:03:46.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.340+0000 7efdc0ff9700 10 client.17250 handle_command_reply: tid=0 2024-06-04T05:03:46.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.340+0000 7efdea187b80 1 client.17250 shutdown 2024-06-04T05:03:46.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.340+0000 7efdea187b80 1 -- 172.21.15.70:0/502359311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde727f0 msgr2=0x558abdef2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:46.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.340+0000 7efdea187b80 1 --2- 172.21.15.70:0/502359311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde727f0 0x558abdef2310 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7efdd0007d50 tx=0x7efdd0004640 comp rx=0 tx=0).stop 2024-06-04T05:03:46.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.340+0000 7efdea187b80 1 -- 172.21.15.70:0/502359311 shutdown_connections 2024-06-04T05:03:46.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.340+0000 7efdea187b80 1 --2- 172.21.15.70:0/502359311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558abde727f0 0x558abdef2310 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.340+0000 7efdea187b80 1 --2- 172.21.15.70:0/502359311 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558abdef4260 0x558abdef6680 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.340+0000 7efdea187b80 1 --2- 172.21.15.70:0/502359311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558abde6e0f0 0x558abde7f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.340+0000 7efdea187b80 1 --2- 172.21.15.70:0/502359311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558abde6eaa0 0x558abde7f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.340+0000 7efdc0ff9700 0 client.17250 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:46.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.340+0000 7efdea187b80 1 -- 172.21.15.70:0/502359311 >> 172.21.15.70:0/502359311 conn(0x558abdd93100 msgr2=0x558abde64f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:46.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.341+0000 7efdea187b80 1 -- 172.21.15.70:0/502359311 shutdown_connections 2024-06-04T05:03:46.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.341+0000 7efdea187b80 1 -- 172.21.15.70:0/502359311 wait complete. 2024-06-04T05:03:46.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.341+0000 7efdea187b80 20 client.17250 trim_cache size 0 max 16384 2024-06-04T05:03:46.346 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:03:46.347 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:03:46.347 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "0b10a296-edf0-4f03-99c8-02076b4f7236", 2024-06-04T05:03:46.347 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:03:46.347 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:03:46.347 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:03:46.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.341+0000 7efddf8f2700 1 -- 172.21.15.70:0/1987681748 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efdc40b2a70 msgr2=0x7efdc40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:46.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.341+0000 7efddf8f2700 1 --2- 172.21.15.70:0/1987681748 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efdc40b2a70 0x7efdc40b4f30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efdc80098e0 tx=0x7efdc800b610 comp rx=0 tx=0).stop 2024-06-04T05:03:46.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.341+0000 7efddf8f2700 1 -- 172.21.15.70:0/1987681748 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efdd806b220 msgr2=0x7efdd8102510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:46.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.341+0000 7efddf8f2700 1 --2- 172.21.15.70:0/1987681748 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efdd806b220 0x7efdd8102510 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efdd400e9a0 tx=0x7efdd400ecb0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.342+0000 7efddf8f2700 1 -- 172.21.15.70:0/1987681748 shutdown_connections 2024-06-04T05:03:46.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.342+0000 7efddf8f2700 1 --2- 172.21.15.70:0/1987681748 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efdc40b2a70 0x7efdc40b4f30 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.342+0000 7efddf8f2700 1 --2- 172.21.15.70:0/1987681748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdd81095b0 0x7efdd8102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.342+0000 7efddf8f2700 1 --2- 172.21.15.70:0/1987681748 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efdd806b220 0x7efdd8102510 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.342+0000 7efddf8f2700 1 --2- 172.21.15.70:0/1987681748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd806bbe0 0x7efdd8102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:46.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.342+0000 7efddf8f2700 1 -- 172.21.15.70:0/1987681748 >> 172.21.15.70:0/1987681748 conn(0x7efdd80793f0 msgr2=0x7efdd8101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:46.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.342+0000 7efddf8f2700 1 -- 172.21.15.70:0/1987681748 shutdown_connections 2024-06-04T05:03:46.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:46.342+0000 7efddf8f2700 1 -- 172.21.15.70:0/1987681748 wait complete. 2024-06-04T05:03:46.686 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:03:46 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:03:46.353+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000215 [...8,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL/ auth{1=1} v17572 ap=2 f(v706 m2024-06-04T05:03:19.727679+0000 11=11+0) n(v22 rc2024-06-04T05:03:19.727679+0000 b2861056 12=11+1) old_inodes=4 (inest sync->mix g=1) caps={15231=pAsLsXsFs/-@15510} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55d6deff1b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000215:[<0x10000000009/COREL v50>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x10000000215:[<0x10000000009/COREL v17572>,<0x10000000008/~dmtmp v15429>,<0x10000000007/client0 v541>,<0x10000000006/clients v524>,<0x10000000005/tmp v506>,<0x10000000003/client.0 v483>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v505>,<0x10000000001/sv_0 v496>,<0x10000000000/qa v495>,<0x1/volumes v481>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:03:19.717578+0000 12=12+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:03:19.727679+0000 b2861056 12=11+1)","memory_value.dirstat":"f(v706 m2024-06-04T05:03:19.727679+0000 11=11+0)","memory_value.rstat":"n(v22 rc2024-06-04T05:03:19.727679+0000 b2861056 12=11+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:03:46.686 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:03:46 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:03:46.358+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000208 [...8,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/WORD/ auth{2=1} v17580 ap=2 f(v648 m2024-06-04T05:03:21.199930+0000 5=5+0) n(v21 rc2024-06-04T05:03:21.199930+0000 b814326 6=5+1) old_inodes=4 (inest sync->mix g=2) caps={15231=pAsLsXsFs/-@15362} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55d6decef700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000208:[<0x10000000009/WORD v8>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x10000000208:[<0x10000000009/WORD v17580>,<0x10000000008/~dmtmp v15429>,<0x10000000007/client0 v541>,<0x10000000006/clients v524>,<0x10000000005/tmp v506>,<0x10000000003/client.0 v483>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v505>,<0x10000000001/sv_0 v496>,<0x10000000000/qa v495>,<0x1/volumes v481>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:03:21.188000+0000 6=6+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:03:21.199930+0000 b814326 6=5+1)","memory_value.dirstat":"f(v648 m2024-06-04T05:03:21.199930+0000 5=5+0)","memory_value.rstat":"n(v21 rc2024-06-04T05:03:21.199930+0000 b814326 6=5+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:03:46.686 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:03:46 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:03:46.362+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000229 [...8,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PARADOX/ auth{1=1,2=1,3=1,4=1} fragtree_t(*^3) v17574 ap=2 f(v981 m2024-06-04T05:03:22.855507+0000 24=24+0) n(v22 rc2024-06-04T05:03:22.855507+0000 b5230239 25=24+1) old_inodes=4 (inest sync->mix g=1,2,3,4) caps={15231=pAsLsXsFs/-@18285} | dirtyscattered=0 request=0 lock=0 dirfrag=8 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55d6df003080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000229:[<0x10000000009/PARADOX v108>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x10000000229:[<0x10000000009/PARADOX v17574>,<0x10000000008/~dmtmp v15429>,<0x10000000007/client0 v541>,<0x10000000006/clients v524>,<0x10000000005/tmp v506>,<0x10000000003/client.0 v483>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v505>,<0x10000000001/sv_0 v496>,<0x10000000000/qa v495>,<0x1/volumes v481>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:03:22.839765+0000 25=25+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:03:22.855507+0000 b5230239 25=24+1)","memory_value.dirstat":"f(v981 m2024-06-04T05:03:22.855507+0000 24=24+0)","memory_value.rstat":"n(v22 rc2024-06-04T05:03:22.855507+0000 b5230239 25=24+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:03:46.687 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:03:46 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:03:46.366+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000225 [...8,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PM/ auth{2=1} v17576 ap=2 f(v1998 m2024-06-04T05:03:19.795885+0000 3=3+0) n(v21 rc2024-06-04T05:03:19.795885+0000 b780288 4=3+1) old_inodes=4 (inest sync->mix g=2) caps={15231=pAsLsXsFs/-@21054} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55d6dea57700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000225:[<0x10000000009/PM v66>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x10000000225:[<0x10000000009/PM v17576>,<0x10000000008/~dmtmp v15431>,<0x10000000007/client0 v543>,<0x10000000006/clients v526>,<0x10000000005/tmp v508>,<0x10000000003/client.0 v485>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v507>,<0x10000000001/sv_0 v498>,<0x10000000000/qa v497>,<0x1/volumes v484>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:03:19.427973+0000 4=4+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:03:19.795885+0000 b780288 4=3+1)","memory_value.dirstat":"f(v1998 m2024-06-04T05:03:19.795885+0000 3=3+0)","memory_value.rstat":"n(v21 rc2024-06-04T05:03:19.795885+0000 b780288 4=3+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:03:46.687 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:03:46 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:03:46.371+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000020f [...8,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ auth{2=1} v17578 ap=2 f(v756 m2024-06-04T05:03:20.640201+0000 7=7+0) n(v21 rc2024-06-04T05:03:20.640201+0000 b1462582 8=7+1) old_inodes=4 (inest sync->mix g=2) caps={15231=pAsLsXsFs/-@15277} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55d6de801700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000020f:[<0x10000000009/PWRPNT v234>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x1000000020f:[<0x10000000009/PWRPNT v17578>,<0x10000000008/~dmtmp v15437>,<0x10000000007/client0 v543>,<0x10000000006/clients v526>,<0x10000000005/tmp v508>,<0x10000000003/client.0 v485>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v507>,<0x10000000001/sv_0 v498>,<0x10000000000/qa v497>,<0x1/volumes v484>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:03:20.619057+0000 8=8+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:03:20.640201+0000 b1462582 8=7+1)","memory_value.dirstat":"f(v756 m2024-06-04T05:03:20.640201+0000 7=7+0)","memory_value.rstat":"n(v21 rc2024-06-04T05:03:20.640201+0000 b1462582 8=7+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:03:46.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 106422 2.08 MB/sec execute 159 sec latency 29.659 ms 2024-06-04T05:03:47.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 106815 2.08 MB/sec execute 160 sec latency 11.700 ms 2024-06-04T05:03:47.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: pgmap v201: 97 pgs: 97 active+clean; 71 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 205 KiB/s wr, 29 op/s 2024-06-04T05:03:47.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:03:47.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:03:47.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:03:47.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:03:47.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:03:47.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.i log and `damage ls` output for details 2024-06-04T05:03:47.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:03:47.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:03:47.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: Scrub error on inode 0x10000000208 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/WORD) see mds.i log and `damage ls` output for details 2024-06-04T05:03:47.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:03:47.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: Scrub error on inode 0x10000000229 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PARADOX) see mds.i log and `damage ls` output for details 2024-06-04T05:03:47.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: Scrub error on inode 0x10000000225 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PM) see mds.i log and `damage ls` output for details 2024-06-04T05:03:47.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT) see mds.i log and `damage ls` output for details 2024-06-04T05:03:47.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:47 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.860 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:03:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: pgmap v201: 97 pgs: 97 active+clean; 71 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 205 KiB/s wr, 29 op/s 2024-06-04T05:03:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:03:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:03:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:03:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:03:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:03:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.i log and `damage ls` output for details 2024-06-04T05:03:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:03:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:03:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: Scrub error on inode 0x10000000208 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/WORD) see mds.i log and `damage ls` output for details 2024-06-04T05:03:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:03:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: Scrub error on inode 0x10000000229 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PARADOX) see mds.i log and `damage ls` output for details 2024-06-04T05:03:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: Scrub error on inode 0x10000000225 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PM) see mds.i log and `damage ls` output for details 2024-06-04T05:03:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT) see mds.i log and `damage ls` output for details 2024-06-04T05:03:47.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:47 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: pgmap v201: 97 pgs: 97 active+clean; 71 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 205 KiB/s wr, 29 op/s 2024-06-04T05:03:47.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:03:47.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:03:47.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:03:47.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:03:47.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:03:47.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.i log and `damage ls` output for details 2024-06-04T05:03:47.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:03:47.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:03:47.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: Scrub error on inode 0x10000000208 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/WORD) see mds.i log and `damage ls` output for details 2024-06-04T05:03:47.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:03:47.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: Scrub error on inode 0x10000000229 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PARADOX) see mds.i log and `damage ls` output for details 2024-06-04T05:03:47.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:47.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: Scrub error on inode 0x10000000225 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PM) see mds.i log and `damage ls` output for details 2024-06-04T05:03:47.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT) see mds.i log and `damage ls` output for details 2024-06-04T05:03:47.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:47 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:03:48.724 INFO:tasks.workunit.client.0.smithi070.stdout: 1 107112 2.07 MB/sec execute 161 sec latency 15.508 ms 2024-06-04T05:03:48.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.980+0000 7fa090c85700 1 -- 172.21.15.70:0/3900334079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa08c1039b0 msgr2=0x7fa08c10f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:48.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.980+0000 7fa090c85700 1 --2- 172.21.15.70:0/3900334079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa08c1039b0 0x7fa08c10f310 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fa08400b0a0 tx=0x7fa08400b3b0 comp rx=0 tx=0).stop 2024-06-04T05:03:48.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.980+0000 7fa090c85700 1 -- 172.21.15.70:0/3900334079 shutdown_connections 2024-06-04T05:03:48.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.980+0000 7fa090c85700 1 --2- 172.21.15.70:0/3900334079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa08c1039b0 0x7fa08c10f310 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:48.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.980+0000 7fa090c85700 1 --2- 172.21.15.70:0/3900334079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa08c107470 0x7fa08c107850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:48.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.980+0000 7fa090c85700 1 --2- 172.21.15.70:0/3900334079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa08c102ff0 0x7fa08c103470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:48.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.980+0000 7fa090c85700 1 -- 172.21.15.70:0/3900334079 >> 172.21.15.70:0/3900334079 conn(0x7fa08c0777e0 msgr2=0x7fa08c077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:48.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.980+0000 7fa090c85700 1 -- 172.21.15.70:0/3900334079 shutdown_connections 2024-06-04T05:03:48.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.980+0000 7fa090c85700 1 -- 172.21.15.70:0/3900334079 wait complete. 2024-06-04T05:03:48.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.981+0000 7fa090c85700 1 Processor -- start 2024-06-04T05:03:48.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.981+0000 7fa090c85700 1 -- start start 2024-06-04T05:03:48.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.981+0000 7fa090c85700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa08c102ff0 0x7fa08c19a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:48.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.981+0000 7fa090c85700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa08c1039b0 0x7fa08c19ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:48.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.981+0000 7fa090c85700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa08c107470 0x7fa08c1a59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:48.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.981+0000 7fa090c85700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa08c1124b0 con 0x7fa08c102ff0 2024-06-04T05:03:48.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.981+0000 7fa090c85700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa08c112330 con 0x7fa08c1039b0 2024-06-04T05:03:48.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.981+0000 7fa090c85700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa08c112630 con 0x7fa08c107470 2024-06-04T05:03:48.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.982+0000 7fa089d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa08c1039b0 0x7fa08c19ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:48.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.982+0000 7fa08a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa08c102ff0 0x7fa08c19a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:48.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.982+0000 7fa089d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa08c1039b0 0x7fa08c19ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:52276/0 (socket says 172.21.15.70:52276) 2024-06-04T05:03:48.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.982+0000 7fa08a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa08c102ff0 0x7fa08c19a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49672/0 (socket says 172.21.15.70:49672) 2024-06-04T05:03:48.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.982+0000 7fa08ad9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa08c107470 0x7fa08c1a59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:48.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.982+0000 7fa089d9b700 1 -- 172.21.15.70:0/2355753803 learned_addr learned my addr 172.21.15.70:0/2355753803 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:48.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.982+0000 7fa089d9b700 1 -- 172.21.15.70:0/2355753803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa08c107470 msgr2=0x7fa08c1a59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:48.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.982+0000 7fa089d9b700 1 --2- 172.21.15.70:0/2355753803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa08c107470 0x7fa08c1a59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:48.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.982+0000 7fa089d9b700 1 -- 172.21.15.70:0/2355753803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa08c102ff0 msgr2=0x7fa08c19a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:48.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.982+0000 7fa089d9b700 1 --2- 172.21.15.70:0/2355753803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa08c102ff0 0x7fa08c19a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:48.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.982+0000 7fa089d9b700 1 -- 172.21.15.70:0/2355753803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08c1a5ef0 con 0x7fa08c1039b0 2024-06-04T05:03:48.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.982+0000 7fa089d9b700 1 --2- 172.21.15.70:0/2355753803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa08c1039b0 0x7fa08c19ae70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa08000d950 tx=0x7fa08000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:48.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.982+0000 7fa07b7fe700 1 -- 172.21.15.70:0/2355753803 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa080016070 con 0x7fa08c1039b0 2024-06-04T05:03:48.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.982+0000 7fa090c85700 1 -- 172.21.15.70:0/2355753803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa08c1a6120 con 0x7fa08c1039b0 2024-06-04T05:03:48.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.982+0000 7fa07b7fe700 1 -- 172.21.15.70:0/2355753803 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa08000fd20 con 0x7fa08c1039b0 2024-06-04T05:03:48.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.982+0000 7fa07b7fe700 1 -- 172.21.15.70:0/2355753803 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa08000e3b0 con 0x7fa08c1039b0 2024-06-04T05:03:48.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.982+0000 7fa090c85700 1 -- 172.21.15.70:0/2355753803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa08c1a6630 con 0x7fa08c1039b0 2024-06-04T05:03:48.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.983+0000 7fa090c85700 1 -- 172.21.15.70:0/2355753803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa08c06b7a0 con 0x7fa08c1039b0 2024-06-04T05:03:48.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.985+0000 7fa07b7fe700 1 -- 172.21.15.70:0/2355753803 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fa080019070 con 0x7fa08c1039b0 2024-06-04T05:03:48.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.986+0000 7fa07b7fe700 1 --2- 172.21.15.70:0/2355753803 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa0740b29f0 0x7fa0740b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:48.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.986+0000 7fa07b7fe700 1 -- 172.21.15.70:0/2355753803 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fa0800e2140 con 0x7fa08c1039b0 2024-06-04T05:03:48.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.986+0000 7fa08a59c700 1 --2- 172.21.15.70:0/2355753803 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa0740b29f0 0x7fa0740b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:48.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.987+0000 7fa08a59c700 1 --2- 172.21.15.70:0/2355753803 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa0740b29f0 0x7fa0740b4eb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009680 tx=0x7fa07c009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:48.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:48.990+0000 7fa07b7fe700 1 -- 172.21.15.70:0/2355753803 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa0800a9ac0 con 0x7fa08c1039b0 2024-06-04T05:03:49.010 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:03:49.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:49.288+0000 7fa090c85700 1 -- 172.21.15.70:0/2355753803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa08c1024d0 con 0x7fa08c1039b0 2024-06-04T05:03:49.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:49.289+0000 7fa07b7fe700 1 -- 172.21.15.70:0/2355753803 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11929 (secure 0 0 0) 0x7fa0800a93c0 con 0x7fa08c1039b0 2024-06-04T05:03:49.292 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:03:49.292 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:01:52.807483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:03:49.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:49.291+0000 7fa090c85700 1 -- 172.21.15.70:0/2355753803 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa0740b29f0 msgr2=0x7fa0740b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:49.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:49.291+0000 7fa090c85700 1 --2- 172.21.15.70:0/2355753803 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa0740b29f0 0x7fa0740b4eb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009680 tx=0x7fa07c009260 comp rx=0 tx=0).stop 2024-06-04T05:03:49.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:49.291+0000 7fa090c85700 1 -- 172.21.15.70:0/2355753803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa08c1039b0 msgr2=0x7fa08c19ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:49.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:49.291+0000 7fa090c85700 1 --2- 172.21.15.70:0/2355753803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa08c1039b0 0x7fa08c19ae70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa08000d950 tx=0x7fa08000dc60 comp rx=0 tx=0).stop 2024-06-04T05:03:49.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:49.292+0000 7fa090c85700 1 -- 172.21.15.70:0/2355753803 shutdown_connections 2024-06-04T05:03:49.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:49.292+0000 7fa090c85700 1 --2- 172.21.15.70:0/2355753803 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa0740b29f0 0x7fa0740b4eb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:49.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:49.292+0000 7fa090c85700 1 --2- 172.21.15.70:0/2355753803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa08c107470 0x7fa08c1a59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:49.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:49.292+0000 7fa090c85700 1 --2- 172.21.15.70:0/2355753803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa08c102ff0 0x7fa08c19a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:49.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:49.292+0000 7fa090c85700 1 --2- 172.21.15.70:0/2355753803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa08c1039b0 0x7fa08c19ae70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:49.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:49.292+0000 7fa090c85700 1 -- 172.21.15.70:0/2355753803 >> 172.21.15.70:0/2355753803 conn(0x7fa08c0777e0 msgr2=0x7fa08c0fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:49.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:49.292+0000 7fa090c85700 1 -- 172.21.15.70:0/2355753803 shutdown_connections 2024-06-04T05:03:49.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:49.292+0000 7fa090c85700 1 -- 172.21.15.70:0/2355753803 wait complete. 2024-06-04T05:03:49.296 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 32 2024-06-04T05:03:49.719 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:49 smithi070 ceph-mon[32631]: pgmap v202: 97 pgs: 97 active+clean; 71 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 190 KiB/s wr, 21 op/s 2024-06-04T05:03:49.719 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:49 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2355753803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:03:49.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 107626 2.06 MB/sec execute 162 sec latency 27.813 ms 2024-06-04T05:03:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:49 smithi112 ceph-mon[36453]: pgmap v202: 97 pgs: 97 active+clean; 71 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 190 KiB/s wr, 21 op/s 2024-06-04T05:03:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:49 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2355753803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:03:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:49 smithi098 ceph-mon[35569]: pgmap v202: 97 pgs: 97 active+clean; 71 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 190 KiB/s wr, 21 op/s 2024-06-04T05:03:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:49 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2355753803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:03:50.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 108155 2.07 MB/sec execute 163 sec latency 19.043 ms 2024-06-04T05:03:51.311 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 138.95 --await-for 45.0' 2024-06-04T05:03:51.312 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 138.95 --await-for 45.0 2024-06-04T05:03:51.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.553+0000 7fb7b7545700 1 -- 172.21.15.70:0/778880468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7b0108510 msgr2=0x7fb7b01088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:51.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.553+0000 7fb7b7545700 1 --2- 172.21.15.70:0/778880468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7b0108510 0x7fb7b01088f0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8009990 tx=0x7fb7a8009ca0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.553+0000 7fb7b7545700 1 -- 172.21.15.70:0/778880468 shutdown_connections 2024-06-04T05:03:51.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.553+0000 7fb7b7545700 1 --2- 172.21.15.70:0/778880468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7b0102df0 0x7fb7b010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.553+0000 7fb7b7545700 1 --2- 172.21.15.70:0/778880468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7b0108510 0x7fb7b01088f0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.553+0000 7fb7b7545700 1 --2- 172.21.15.70:0/778880468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7b0102430 0x7fb7b01028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.553+0000 7fb7b7545700 1 -- 172.21.15.70:0/778880468 >> 172.21.15.70:0/778880468 conn(0x7fb7b00793b0 msgr2=0x7fb7b00797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:51.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.553+0000 7fb7b7545700 1 -- 172.21.15.70:0/778880468 shutdown_connections 2024-06-04T05:03:51.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.553+0000 7fb7b7545700 1 -- 172.21.15.70:0/778880468 wait complete. 2024-06-04T05:03:51.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.554+0000 7fb7b7545700 1 Processor -- start 2024-06-04T05:03:51.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.554+0000 7fb7b7545700 1 -- start start 2024-06-04T05:03:51.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.554+0000 7fb7b7545700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7b0102430 0x7fb7b019f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.554+0000 7fb7b7545700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7b0102df0 0x7fb7b019fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.554+0000 7fb7b7545700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7b0108510 0x7fb7b0199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.554+0000 7fb7b7545700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b0112470 con 0x7fb7b0108510 2024-06-04T05:03:51.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.554+0000 7fb7b7545700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b01122f0 con 0x7fb7b0102430 2024-06-04T05:03:51.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.554+0000 7fb7b7545700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b01125f0 con 0x7fb7b0102df0 2024-06-04T05:03:51.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.554+0000 7fb7b52e1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7b0102430 0x7fb7b019f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:51.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.554+0000 7fb7b4ae0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7b0102df0 0x7fb7b019fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:51.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.554+0000 7fb7b52e1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7b0102430 0x7fb7b019f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:52294/0 (socket says 172.21.15.70:52294) 2024-06-04T05:03:51.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.554+0000 7fb7b52e1700 1 -- 172.21.15.70:0/946882106 learned_addr learned my addr 172.21.15.70:0/946882106 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:51.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.554+0000 7fb7b4ae0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7b0102df0 0x7fb7b019fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:57820/0 (socket says 172.21.15.70:57820) 2024-06-04T05:03:51.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.554+0000 7fb7b5ae2700 1 --2- 172.21.15.70:0/946882106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7b0108510 0x7fb7b0199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:51.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.555+0000 7fb7b4ae0700 1 -- 172.21.15.70:0/946882106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7b0102430 msgr2=0x7fb7b019f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:51.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.555+0000 7fb7b4ae0700 1 --2- 172.21.15.70:0/946882106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7b0102430 0x7fb7b019f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.555+0000 7fb7b4ae0700 1 -- 172.21.15.70:0/946882106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7b0108510 msgr2=0x7fb7b0199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:51.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.555+0000 7fb7b4ae0700 1 --2- 172.21.15.70:0/946882106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7b0108510 0x7fb7b0199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.555+0000 7fb7b4ae0700 1 -- 172.21.15.70:0/946882106 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b019a0e0 con 0x7fb7b0102df0 2024-06-04T05:03:51.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.555+0000 7fb7b4ae0700 1 --2- 172.21.15.70:0/946882106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7b0102df0 0x7fb7b019fd00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4007a60 tx=0x7fb7a4007d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:51.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.555+0000 7fb7ae7fc700 1 -- 172.21.15.70:0/946882106 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a4004020 con 0x7fb7b0102df0 2024-06-04T05:03:51.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.555+0000 7fb7ae7fc700 1 -- 172.21.15.70:0/946882106 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb7a4004830 con 0x7fb7b0102df0 2024-06-04T05:03:51.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.555+0000 7fb7b7545700 1 -- 172.21.15.70:0/946882106 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7b019a3d0 con 0x7fb7b0102df0 2024-06-04T05:03:51.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.555+0000 7fb7ae7fc700 1 -- 172.21.15.70:0/946882106 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a4005390 con 0x7fb7b0102df0 2024-06-04T05:03:51.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.555+0000 7fb7b7545700 1 -- 172.21.15.70:0/946882106 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7b0103fa0 con 0x7fb7b0102df0 2024-06-04T05:03:51.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.558+0000 7fb7ae7fc700 1 -- 172.21.15.70:0/946882106 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fb7a4021020 con 0x7fb7b0102df0 2024-06-04T05:03:51.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.559+0000 7fb7ae7fc700 1 --2- 172.21.15.70:0/946882106 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb7980b2a70 0x7fb7980b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.560+0000 7fb7b52e1700 1 --2- 172.21.15.70:0/946882106 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb7980b2a70 0x7fb7980b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:51.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.560+0000 7fb7ae7fc700 1 -- 172.21.15.70:0/946882106 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb7a40e57a0 con 0x7fb7b0102df0 2024-06-04T05:03:51.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.560+0000 7fb7c1ddab80 1 Processor -- start 2024-06-04T05:03:51.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.560+0000 7fb7c1ddab80 1 -- start start 2024-06-04T05:03:51.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.561+0000 7fb7c1ddab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a63041320 0x560a63041700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.561+0000 7fb7c1ddab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a6303b8a0 0x560a6303bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.561+0000 7fb7b52e1700 1 --2- 172.21.15.70:0/946882106 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb7980b2a70 0x7fb7980b4f30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb7b019aac0 tx=0x7fb7a801a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:51.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.561+0000 7fb7c1ddab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6303c260 0x560a63048080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.561+0000 7fb7c1ddab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560a62f99d70 con 0x560a6303c260 2024-06-04T05:03:51.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.561+0000 7fb7c1ddab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560a62f89910 con 0x560a6303b8a0 2024-06-04T05:03:51.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.561+0000 7fb7c1ddab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560a62f8d950 con 0x560a63041320 2024-06-04T05:03:51.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.561+0000 7fb7b5ae2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6303c260 0x560a63048080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:51.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.561+0000 7fb7b5ae2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6303c260 0x560a63048080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49706/0 (socket says 172.21.15.70:49706) 2024-06-04T05:03:51.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.561+0000 7fb7b5ae2700 1 -- 172.21.15.70:0/1835661883 learned_addr learned my addr 172.21.15.70:0/1835661883 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:51.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.561+0000 7fb7b4ae0700 1 --2- 172.21.15.70:0/1835661883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a6303b8a0 0x560a6303bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:51.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.561+0000 7fb7b5ae2700 1 -- 172.21.15.70:0/1835661883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a63041320 msgr2=0x560a63041700 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:03:51.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.561+0000 7fb7b5ae2700 1 --2- 172.21.15.70:0/1835661883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a63041320 0x560a63041700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.561+0000 7fb7b5ae2700 1 -- 172.21.15.70:0/1835661883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a6303b8a0 msgr2=0x560a6303bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:51.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.561+0000 7fb7b5ae2700 1 --2- 172.21.15.70:0/1835661883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a6303b8a0 0x560a6303bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.561+0000 7fb7b5ae2700 1 -- 172.21.15.70:0/1835661883 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a62f32fb0 con 0x560a6303c260 2024-06-04T05:03:51.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.561+0000 7fb7b4ae0700 1 --2- 172.21.15.70:0/1835661883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a6303b8a0 0x560a6303bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:51.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.562+0000 7fb7b5ae2700 1 --2- 172.21.15.70:0/1835661883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6303c260 0x560a63048080 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fb7a0011680 tx=0x7fb7a0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5a1cbc04c3f3d5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:51.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.562+0000 7fb7977fe700 1 -- 172.21.15.70:0/1835661883 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a0019030 con 0x560a6303c260 2024-06-04T05:03:51.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.562+0000 7fb7977fe700 1 -- 172.21.15.70:0/1835661883 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb7a00092e0 con 0x560a6303c260 2024-06-04T05:03:51.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.562+0000 7fb7977fe700 1 -- 172.21.15.70:0/1835661883 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a0007500 con 0x560a6303c260 2024-06-04T05:03:51.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.563+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/1835661883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6303c260 msgr2=0x560a63048080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:51.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.563+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/1835661883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6303c260 0x560a63048080 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fb7a0011680 tx=0x7fb7a0009920 comp rx=0 tx=0).stop 2024-06-04T05:03:51.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.563+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/1835661883 shutdown_connections 2024-06-04T05:03:51.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.563+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/1835661883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6303c260 0x560a63048080 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.563+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/1835661883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a63041320 0x560a63041700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.563+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/1835661883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a6303b8a0 0x560a6303bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.563+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/1835661883 >> 172.21.15.70:0/1835661883 conn(0x560a62f67240 msgr2=0x560a630363e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:51.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.563+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/1835661883 shutdown_connections 2024-06-04T05:03:51.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.563+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/1835661883 wait complete. 2024-06-04T05:03:51.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.563+0000 7fb7c1ddab80 1 -- start start 2024-06-04T05:03:51.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.563+0000 7fb7c1ddab80 1 Processor -- start 2024-06-04T05:03:51.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7c1ddab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a6303b8a0 0x560a63047dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7c1ddab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303c260 0x560a630450c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7c1ddab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a63041320 0x560a63045600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7c1ddab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560a62e8ffa0 con 0x560a63041320 2024-06-04T05:03:51.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7c1ddab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560a62ef1da0 con 0x560a6303b8a0 2024-06-04T05:03:51.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7c1ddab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560a630486e0 con 0x560a6303c260 2024-06-04T05:03:51.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7b5ae2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a63041320 0x560a63045600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:51.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7b5ae2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a63041320 0x560a63045600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49720/0 (socket says 172.21.15.70:49720) 2024-06-04T05:03:51.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7b5ae2700 1 -- 172.21.15.70:0/2076363025 learned_addr learned my addr 172.21.15.70:0/2076363025 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:51.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7c1ddab80 10 client.0 _ll_register_callbacks cb 0x560a62f24950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:51.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7b4ae0700 1 --2- 172.21.15.70:0/2076363025 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303c260 0x560a630450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:51.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7b52e1700 1 --2- 172.21.15.70:0/2076363025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a6303b8a0 0x560a63047dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:51.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7b4ae0700 1 -- 172.21.15.70:0/2076363025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a6303b8a0 msgr2=0x560a63047dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:51.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7b4ae0700 1 --2- 172.21.15.70:0/2076363025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a6303b8a0 0x560a63047dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7b4ae0700 1 -- 172.21.15.70:0/2076363025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a63041320 msgr2=0x560a63045600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:51.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7b4ae0700 1 --2- 172.21.15.70:0/2076363025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a63041320 0x560a63045600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7b4ae0700 1 -- 172.21.15.70:0/2076363025 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a62ed98d0 con 0x560a6303c260 2024-06-04T05:03:51.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7b52e1700 1 --2- 172.21.15.70:0/2076363025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a6303b8a0 0x560a63047dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:03:51.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb7b4ae0700 1 --2- 172.21.15.70:0/2076363025 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303c260 0x560a630450c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb7a40e2e90 tx=0x7fb7a40ad1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=154c3e92ac0fa6fe server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.564+0000 7fb794ff9700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:03:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.565+0000 7fb794ff9700 1 -- 172.21.15.70:0/2076363025 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a401e040 con 0x560a6303c260 2024-06-04T05:03:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.565+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2076363025 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7a00092e0 con 0x560a6303c260 2024-06-04T05:03:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.565+0000 7fb794ff9700 1 -- 172.21.15.70:0/2076363025 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb7a4005530 con 0x560a6303c260 2024-06-04T05:03:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.565+0000 7fb794ff9700 1 -- 172.21.15.70:0/2076363025 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb7a4005730 con 0x560a6303c260 2024-06-04T05:03:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.565+0000 7fb7c1ddab80 10 client.26575 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.565+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2076363025 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560a62f43760 con 0x560a6303c260 2024-06-04T05:03:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.565+0000 7fb794ff9700 1 -- 172.21.15.70:0/2076363025 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb7a00092e0 con 0x560a6303c260 2024-06-04T05:03:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.565+0000 7fb794ff9700 1 -- 172.21.15.70:0/2076363025 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fb7a40b53f0 con 0x560a6303c260 2024-06-04T05:03:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.565+0000 7fb7c1ddab80 10 client.26575 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.565+0000 7fb7c1ddab80 10 client.26575 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.565+0000 7fb7c1ddab80 20 client.26575 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.565+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/2076363025 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560a6305ded0 0x560a63060310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.565+0000 7fb7c1ddab80 4 client.26575 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:03:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.565+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2076363025 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a62f9bcc0 con 0x560a6305ded0 2024-06-04T05:03:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.566+0000 7fb7b52e1700 1 --2- 172.21.15.70:0/2076363025 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560a6305ded0 0x560a63060310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:51.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.566+0000 7fb794ff9700 10 client.26575 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:51.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.566+0000 7fb7b52e1700 1 --2- 172.21.15.70:0/2076363025 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560a6305ded0 0x560a63060310 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7342424a3942acf server_cookie=dad2f123b262915b in_seq=0 out_seq=0 2024-06-04T05:03:51.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.567+0000 7fb794ff9700 1 -- 172.21.15.70:0/2076363025 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560a62f9bcc0 con 0x560a6305ded0 2024-06-04T05:03:51.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.567+0000 7fb794ff9700 10 client.26575 handle_command_reply: tid=0 2024-06-04T05:03:51.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.567+0000 7fb7c1ddab80 1 client.26575 shutdown 2024-06-04T05:03:51.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.567+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2076363025 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303c260 msgr2=0x560a630450c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:51.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.567+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/2076363025 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303c260 0x560a630450c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb7a40e2e90 tx=0x7fb7a40ad1a0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.567+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2076363025 shutdown_connections 2024-06-04T05:03:51.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.567+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/2076363025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a63041320 0x560a63045600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.567+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/2076363025 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560a6305ded0 0x560a63060310 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.567+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/2076363025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a6303b8a0 0x560a63047dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.567+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/2076363025 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303c260 0x560a630450c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.567+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2076363025 >> 172.21.15.70:0/2076363025 conn(0x560a62f67240 msgr2=0x560a63035af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:51.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.567+0000 7fb794ff9700 0 client.26575 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:51.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.567+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2076363025 shutdown_connections 2024-06-04T05:03:51.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.567+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2076363025 wait complete. 2024-06-04T05:03:51.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.567+0000 7fb7c1ddab80 20 client.26575 trim_cache size 0 max 16384 2024-06-04T05:03:51.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.597+0000 7fb7c1ddab80 1 Processor -- start 2024-06-04T05:03:51.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.597+0000 7fb7c1ddab80 1 -- start start 2024-06-04T05:03:51.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.597+0000 7fb7c1ddab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6303b8a0 0x560a63044390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.597+0000 7fb7c1ddab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303c260 0x560a63036270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.597+0000 7fb7c1ddab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a63041320 0x560a63042010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.597+0000 7fb7c1ddab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b01abc40 con 0x560a6303b8a0 2024-06-04T05:03:51.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.597+0000 7fb7c1ddab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b01ad780 con 0x560a63041320 2024-06-04T05:03:51.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.597+0000 7fb7c1ddab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b01ac310 con 0x560a6303c260 2024-06-04T05:03:51.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.597+0000 7fb7b52e1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6303b8a0 0x560a63044390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:51.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.597+0000 7fb7b4ae0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303c260 0x560a63036270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:51.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.598+0000 7fb7b4ae0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303c260 0x560a63036270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:57850/0 (socket says 172.21.15.70:57850) 2024-06-04T05:03:51.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.598+0000 7fb7b4ae0700 1 -- 172.21.15.70:0/2597900263 learned_addr learned my addr 172.21.15.70:0/2597900263 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:51.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.598+0000 7fb7b5ae2700 1 --2- 172.21.15.70:0/2597900263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a63041320 0x560a63042010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:51.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.598+0000 7fb7b52e1700 1 -- 172.21.15.70:0/2597900263 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303c260 msgr2=0x560a63036270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:51.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.598+0000 7fb7b52e1700 1 --2- 172.21.15.70:0/2597900263 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303c260 0x560a63036270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.598+0000 7fb7b52e1700 1 -- 172.21.15.70:0/2597900263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a63041320 msgr2=0x560a63042010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:51.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.598+0000 7fb7b52e1700 1 --2- 172.21.15.70:0/2597900263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a63041320 0x560a63042010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.598+0000 7fb7b52e1700 1 -- 172.21.15.70:0/2597900263 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a63038380 con 0x560a6303b8a0 2024-06-04T05:03:51.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.598+0000 7fb7b52e1700 1 --2- 172.21.15.70:0/2597900263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6303b8a0 0x560a63044390 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8006010 tx=0x7fb7a801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc0bee62f693f7b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:51.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.598+0000 7fb796ffd700 1 -- 172.21.15.70:0/2597900263 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8036070 con 0x560a6303b8a0 2024-06-04T05:03:51.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.598+0000 7fb796ffd700 1 -- 172.21.15.70:0/2597900263 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb7a800bd10 con 0x560a6303b8a0 2024-06-04T05:03:51.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.598+0000 7fb796ffd700 1 -- 172.21.15.70:0/2597900263 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a800f7e0 con 0x560a6303b8a0 2024-06-04T05:03:51.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.599+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2597900263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6303b8a0 msgr2=0x560a63044390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:51.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.599+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/2597900263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6303b8a0 0x560a63044390 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8006010 tx=0x7fb7a801f000 comp rx=0 tx=0).stop 2024-06-04T05:03:51.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.599+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2597900263 shutdown_connections 2024-06-04T05:03:51.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.599+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/2597900263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a63041320 0x560a63042010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.599+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/2597900263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6303b8a0 0x560a63044390 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.599+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/2597900263 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303c260 0x560a63036270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.599+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2597900263 >> 172.21.15.70:0/2597900263 conn(0x560a62f67240 msgr2=0x560a6303b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:51.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.599+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2597900263 shutdown_connections 2024-06-04T05:03:51.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.599+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2597900263 wait complete. 2024-06-04T05:03:51.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.600+0000 7fb7c1ddab80 1 -- start start 2024-06-04T05:03:51.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.600+0000 7fb7c1ddab80 1 Processor -- start 2024-06-04T05:03:51.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.600+0000 7fb7c1ddab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303b8a0 0x560a63056580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.600+0000 7fb7c1ddab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a6303c260 0x560a630651f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.600+0000 7fb7c1ddab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a63041320 0x560a63065730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.600+0000 7fb7c1ddab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b01ad0b0 con 0x560a63041320 2024-06-04T05:03:51.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.600+0000 7fb7c1ddab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b01ac9e0 con 0x560a6303c260 2024-06-04T05:03:51.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.600+0000 7fb7c1ddab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b01a9c00 con 0x560a6303b8a0 2024-06-04T05:03:51.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.600+0000 7fb7c1ddab80 10 client.0 _ll_register_callbacks cb 0x560a62f24950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:51.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.601+0000 7fb7b52e1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303b8a0 0x560a63056580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:51.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.601+0000 7fb7b5ae2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a63041320 0x560a63065730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:51.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.601+0000 7fb7b52e1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303b8a0 0x560a63056580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:57866/0 (socket says 172.21.15.70:57866) 2024-06-04T05:03:51.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.601+0000 7fb7b52e1700 1 -- 172.21.15.70:0/2025115765 learned_addr learned my addr 172.21.15.70:0/2025115765 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:51.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.601+0000 7fb7b4ae0700 1 --2- 172.21.15.70:0/2025115765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a6303c260 0x560a630651f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:51.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.601+0000 7fb7b52e1700 1 -- 172.21.15.70:0/2025115765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a6303c260 msgr2=0x560a630651f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:51.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.601+0000 7fb7b52e1700 1 --2- 172.21.15.70:0/2025115765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a6303c260 0x560a630651f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.601+0000 7fb7b52e1700 1 -- 172.21.15.70:0/2025115765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a63041320 msgr2=0x560a63065730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:51.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.601+0000 7fb7b52e1700 1 --2- 172.21.15.70:0/2025115765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a63041320 0x560a63065730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.601+0000 7fb7b52e1700 1 -- 172.21.15.70:0/2025115765 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a63065c70 con 0x560a6303b8a0 2024-06-04T05:03:51.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.601+0000 7fb7b52e1700 1 --2- 172.21.15.70:0/2025115765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303b8a0 0x560a63056580 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb7a80093a0 tx=0x7fb7a800bb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=941ad9e88d475a2b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:51.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.601+0000 7fb794ff9700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:03:51.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.601+0000 7fb794ff9700 1 -- 172.21.15.70:0/2025115765 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8036070 con 0x560a6303b8a0 2024-06-04T05:03:51.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.601+0000 7fb794ff9700 1 -- 172.21.15.70:0/2025115765 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fb7a8003bd0 con 0x560a6303b8a0 2024-06-04T05:03:51.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.601+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2025115765 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7a800bd10 con 0x560a6303b8a0 2024-06-04T05:03:51.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.602+0000 7fb794ff9700 1 -- 172.21.15.70:0/2025115765 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb7a804d050 con 0x560a6303b8a0 2024-06-04T05:03:51.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.602+0000 7fb7c1ddab80 10 client.26581 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:51.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.602+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2025115765 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560a630c8830 con 0x560a6303b8a0 2024-06-04T05:03:51.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.602+0000 7fb794ff9700 1 -- 172.21.15.70:0/2025115765 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb7a800bd10 con 0x560a6303b8a0 2024-06-04T05:03:51.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.602+0000 7fb794ff9700 1 -- 172.21.15.70:0/2025115765 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fb7a803b450 con 0x560a6303b8a0 2024-06-04T05:03:51.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.602+0000 7fb7c1ddab80 10 client.26581 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:51.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.602+0000 7fb7c1ddab80 10 client.26581 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:51.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.602+0000 7fb7c1ddab80 20 client.26581 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:51.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.602+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/2025115765 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560a630c8ea0 0x560a630cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:51.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.602+0000 7fb7c1ddab80 4 client.26581 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:03:51.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.602+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2025115765 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560a62f954a0 con 0x560a630c8ea0 2024-06-04T05:03:51.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.602+0000 7fb7b4ae0700 1 --2- 172.21.15.70:0/2025115765 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560a630c8ea0 0x560a630cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:51.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.603+0000 7fb7b4ae0700 1 --2- 172.21.15.70:0/2025115765 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560a630c8ea0 0x560a630cb2c0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93eba5bcd7f002e8 server_cookie=607d2db172927361 in_seq=0 out_seq=0 2024-06-04T05:03:51.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.603+0000 7fb794ff9700 10 client.26581 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:51.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.603+0000 7fb794ff9700 1 -- 172.21.15.70:0/2025115765 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560a62f954a0 con 0x560a630c8ea0 2024-06-04T05:03:51.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.603+0000 7fb794ff9700 10 client.26581 handle_command_reply: tid=0 2024-06-04T05:03:51.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.603+0000 7fb7c1ddab80 1 client.26581 shutdown 2024-06-04T05:03:51.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.603+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2025115765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303b8a0 msgr2=0x560a63056580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:51.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.603+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/2025115765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303b8a0 0x560a63056580 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb7a80093a0 tx=0x7fb7a800bb10 comp rx=0 tx=0).stop 2024-06-04T05:03:51.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.603+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2025115765 shutdown_connections 2024-06-04T05:03:51.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.603+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/2025115765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a63041320 0x560a63065730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.603+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/2025115765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a6303b8a0 0x560a63056580 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.603+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/2025115765 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560a630c8ea0 0x560a630cb2c0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.603+0000 7fb7c1ddab80 1 --2- 172.21.15.70:0/2025115765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a6303c260 0x560a630651f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.603+0000 7fb794ff9700 0 client.26581 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:51.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.603+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2025115765 >> 172.21.15.70:0/2025115765 conn(0x560a62f67240 msgr2=0x560a6303ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:51.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.603+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2025115765 shutdown_connections 2024-06-04T05:03:51.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.603+0000 7fb7c1ddab80 1 -- 172.21.15.70:0/2025115765 wait complete. 2024-06-04T05:03:51.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.604+0000 7fb7c1ddab80 20 client.26581 trim_cache size 0 max 16384 2024-06-04T05:03:51.610 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:03:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:03:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:03:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: "0b10a296-edf0-4f03-99c8-02076b4f7236": { 2024-06-04T05:03:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:03:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "0b10a296-edf0-4f03-99c8-02076b4f7236", 2024-06-04T05:03:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:03:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:03:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:03:51.611 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:03:51.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.604+0000 7fb7b7545700 1 -- 172.21.15.70:0/946882106 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb7980b2a70 msgr2=0x7fb7980b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:51.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.604+0000 7fb7b7545700 1 --2- 172.21.15.70:0/946882106 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb7980b2a70 0x7fb7980b4f30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb7b019aac0 tx=0x7fb7a801a040 comp rx=0 tx=0).stop 2024-06-04T05:03:51.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.604+0000 7fb7b7545700 1 -- 172.21.15.70:0/946882106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7b0102df0 msgr2=0x7fb7b019fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:51.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.604+0000 7fb7b7545700 1 --2- 172.21.15.70:0/946882106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7b0102df0 0x7fb7b019fd00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4007a60 tx=0x7fb7a4007d70 comp rx=0 tx=0).stop 2024-06-04T05:03:51.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.604+0000 7fb7b7545700 1 -- 172.21.15.70:0/946882106 shutdown_connections 2024-06-04T05:03:51.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.604+0000 7fb7b7545700 1 --2- 172.21.15.70:0/946882106 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb7980b2a70 0x7fb7980b4f30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.604+0000 7fb7b7545700 1 --2- 172.21.15.70:0/946882106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7b0108510 0x7fb7b0199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.604+0000 7fb7b7545700 1 --2- 172.21.15.70:0/946882106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7b0102430 0x7fb7b019f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.604+0000 7fb7b7545700 1 --2- 172.21.15.70:0/946882106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7b0102df0 0x7fb7b019fd00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:51.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.604+0000 7fb7b7545700 1 -- 172.21.15.70:0/946882106 >> 172.21.15.70:0/946882106 conn(0x7fb7b00793b0 msgr2=0x7fb7b006bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:51.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.605+0000 7fb7b7545700 1 -- 172.21.15.70:0/946882106 shutdown_connections 2024-06-04T05:03:51.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:51.605+0000 7fb7b7545700 1 -- 172.21.15.70:0/946882106 wait complete. 2024-06-04T05:03:51.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 108787 2.07 MB/sec execute 164 sec latency 64.811 ms 2024-06-04T05:03:51.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:51 smithi070 ceph-mon[32631]: pgmap v203: 97 pgs: 97 active+clean; 71 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 190 KiB/s wr, 21 op/s 2024-06-04T05:03:51.909 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:03:51.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:51 smithi112 ceph-mon[36453]: pgmap v203: 97 pgs: 97 active+clean; 71 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 190 KiB/s wr, 21 op/s 2024-06-04T05:03:51.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:51 smithi098 ceph-mon[35569]: pgmap v203: 97 pgs: 97 active+clean; 71 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 190 KiB/s wr, 21 op/s 2024-06-04T05:03:52.724 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109388 2.09 MB/sec execute 165 sec latency 15.357 ms 2024-06-04T05:03:53.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.665+0000 7fad4ef19700 1 -- 172.21.15.70:0/2149315083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad481085a0 msgr2=0x7fad48108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:53.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.665+0000 7fad4ef19700 1 --2- 172.21.15.70:0/2149315083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad481085a0 0x7fad48108980 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fad3c0099c0 tx=0x7fad3c009cd0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.665+0000 7fad4ef19700 1 -- 172.21.15.70:0/2149315083 shutdown_connections 2024-06-04T05:03:53.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.665+0000 7fad4ef19700 1 --2- 172.21.15.70:0/2149315083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad48103fb0 0x7fad481104a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.665+0000 7fad4ef19700 1 --2- 172.21.15.70:0/2149315083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad481085a0 0x7fad48108980 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.665+0000 7fad4ef19700 1 --2- 172.21.15.70:0/2149315083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad481035f0 0x7fad48103a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.665+0000 7fad4ef19700 1 -- 172.21.15.70:0/2149315083 >> 172.21.15.70:0/2149315083 conn(0x7fad4807a6f0 msgr2=0x7fad4807ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.665+0000 7fad4ef19700 1 -- 172.21.15.70:0/2149315083 shutdown_connections 2024-06-04T05:03:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.665+0000 7fad4ef19700 1 -- 172.21.15.70:0/2149315083 wait complete. 2024-06-04T05:03:53.669 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:53 smithi070 ceph-mon[32631]: pgmap v204: 97 pgs: 97 active+clean; 71 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 740 KiB/s wr, 90 op/s 2024-06-04T05:03:53.669 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:53 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:03:53.669 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:53 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:03:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.666+0000 7fad4ef19700 1 Processor -- start 2024-06-04T05:03:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.666+0000 7fad4ef19700 1 -- start start 2024-06-04T05:03:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad4ef19700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad481035f0 0x7fad48075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad4ef19700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad48103fb0 0x7fad48075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad4ef19700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad481085a0 0x7fad480762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad4ef19700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fad48113670 con 0x7fad48103fb0 2024-06-04T05:03:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad4ef19700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad481134f0 con 0x7fad481035f0 2024-06-04T05:03:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad4ef19700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fad481137f0 con 0x7fad481085a0 2024-06-04T05:03:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad47fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad48103fb0 0x7fad48075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad4ccb5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad481035f0 0x7fad48075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad47fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad48103fb0 0x7fad48075d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49762/0 (socket says 172.21.15.70:49762) 2024-06-04T05:03:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad4ccb5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad481035f0 0x7fad48075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:52364/0 (socket says 172.21.15.70:52364) 2024-06-04T05:03:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad47fff700 1 -- 172.21.15.70:0/3220414432 learned_addr learned my addr 172.21.15.70:0/3220414432 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad4d4b6700 1 --2- 172.21.15.70:0/3220414432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad481085a0 0x7fad480762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad47fff700 1 -- 172.21.15.70:0/3220414432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad481085a0 msgr2=0x7fad480762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad47fff700 1 --2- 172.21.15.70:0/3220414432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad481085a0 0x7fad480762c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad47fff700 1 -- 172.21.15.70:0/3220414432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad481035f0 msgr2=0x7fad48075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad47fff700 1 --2- 172.21.15.70:0/3220414432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad481035f0 0x7fad48075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad47fff700 1 -- 172.21.15.70:0/3220414432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad481ac4d0 con 0x7fad48103fb0 2024-06-04T05:03:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad4ccb5700 1 --2- 172.21.15.70:0/3220414432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad481035f0 0x7fad48075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.667+0000 7fad47fff700 1 --2- 172.21.15.70:0/3220414432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad48103fb0 0x7fad48075d80 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fad3400e9a0 tx=0x7fad3400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.668+0000 7fad45ffb700 1 -- 172.21.15.70:0/3220414432 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3400ca10 con 0x7fad48103fb0 2024-06-04T05:03:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.668+0000 7fad4ef19700 1 -- 172.21.15.70:0/3220414432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad481ac760 con 0x7fad48103fb0 2024-06-04T05:03:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.668+0000 7fad45ffb700 1 -- 172.21.15.70:0/3220414432 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad3400cbb0 con 0x7fad48103fb0 2024-06-04T05:03:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.668+0000 7fad45ffb700 1 -- 172.21.15.70:0/3220414432 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad34018680 con 0x7fad48103fb0 2024-06-04T05:03:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.668+0000 7fad4ef19700 1 -- 172.21.15.70:0/3220414432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad481accf0 con 0x7fad48103fb0 2024-06-04T05:03:53.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.670+0000 7fad45ffb700 1 -- 172.21.15.70:0/3220414432 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fad34010430 con 0x7fad48103fb0 2024-06-04T05:03:53.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.671+0000 7fad45ffb700 1 --2- 172.21.15.70:0/3220414432 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fad380b2b20 0x7fad380b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.671+0000 7fad4ccb5700 1 --2- 172.21.15.70:0/3220414432 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fad380b2b20 0x7fad380b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:53.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.671+0000 7fad45ffb700 1 -- 172.21.15.70:0/3220414432 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fad340e2750 con 0x7fad48103fb0 2024-06-04T05:03:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.672+0000 7fad4ccb5700 1 --2- 172.21.15.70:0/3220414432 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fad380b2b20 0x7fad380b4fe0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fad3c000c00 tx=0x7fad3c005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.672+0000 7fad597aeb80 1 Processor -- start 2024-06-04T05:03:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.672+0000 7fad597aeb80 1 -- start start 2024-06-04T05:03:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.672+0000 7fad597aeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd0177c0 0x5559bd017ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.672+0000 7fad597aeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd012c90 0x5559bd013110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.672+0000 7fad597aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd013650 0x5559bd01f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.672+0000 7fad597aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5559bcf70f10 con 0x5559bd013650 2024-06-04T05:03:53.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.672+0000 7fad597aeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5559bcf60ab0 con 0x5559bd012c90 2024-06-04T05:03:53.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.672+0000 7fad597aeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5559bcf64af0 con 0x5559bd0177c0 2024-06-04T05:03:53.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.672+0000 7fad4ccb5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd0177c0 0x5559bd017ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:53.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.672+0000 7fad4d4b6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd013650 0x5559bd01f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:53.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.672+0000 7fad4ccb5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd0177c0 0x5559bd017ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:57892/0 (socket says 172.21.15.70:57892) 2024-06-04T05:03:53.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.672+0000 7fad4ccb5700 1 -- 172.21.15.70:0/389828600 learned_addr learned my addr 172.21.15.70:0/389828600 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:53.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.672+0000 7fad47fff700 1 --2- 172.21.15.70:0/389828600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd012c90 0x5559bd013110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:53.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.673+0000 7fad4ccb5700 1 -- 172.21.15.70:0/389828600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd012c90 msgr2=0x5559bd013110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:53.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.673+0000 7fad4ccb5700 1 --2- 172.21.15.70:0/389828600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd012c90 0x5559bd013110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.673+0000 7fad4ccb5700 1 -- 172.21.15.70:0/389828600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd013650 msgr2=0x5559bd01f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.673+0000 7fad4ccb5700 1 --2- 172.21.15.70:0/389828600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd013650 0x5559bd01f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.673+0000 7fad4ccb5700 1 -- 172.21.15.70:0/389828600 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559bcf0a150 con 0x5559bd0177c0 2024-06-04T05:03:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.673+0000 7fad4d4b6700 1 --2- 172.21.15.70:0/389828600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd013650 0x5559bd01f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.673+0000 7fad4ccb5700 1 --2- 172.21.15.70:0/389828600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd0177c0 0x5559bd017ba0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fad48104800 tx=0x7fad3c020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=854cb65bc97fdd39 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.673+0000 7fad32ffd700 1 -- 172.21.15.70:0/389828600 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c005690 con 0x5559bd0177c0 2024-06-04T05:03:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.673+0000 7fad32ffd700 1 -- 172.21.15.70:0/389828600 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad3c00bdf0 con 0x5559bd0177c0 2024-06-04T05:03:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.673+0000 7fad32ffd700 1 -- 172.21.15.70:0/389828600 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c024ac0 con 0x5559bd0177c0 2024-06-04T05:03:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.674+0000 7fad597aeb80 1 -- 172.21.15.70:0/389828600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd0177c0 msgr2=0x5559bd017ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.674+0000 7fad597aeb80 1 --2- 172.21.15.70:0/389828600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd0177c0 0x5559bd017ba0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fad48104800 tx=0x7fad3c020000 comp rx=0 tx=0).stop 2024-06-04T05:03:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.674+0000 7fad597aeb80 1 -- 172.21.15.70:0/389828600 shutdown_connections 2024-06-04T05:03:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.674+0000 7fad597aeb80 1 --2- 172.21.15.70:0/389828600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd013650 0x5559bd01f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.674+0000 7fad597aeb80 1 --2- 172.21.15.70:0/389828600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd0177c0 0x5559bd017ba0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.674+0000 7fad597aeb80 1 --2- 172.21.15.70:0/389828600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd012c90 0x5559bd013110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.674+0000 7fad597aeb80 1 -- 172.21.15.70:0/389828600 >> 172.21.15.70:0/389828600 conn(0x5559bcf3e3e0 msgr2=0x5559bd00d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:53.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.674+0000 7fad597aeb80 1 -- 172.21.15.70:0/389828600 shutdown_connections 2024-06-04T05:03:53.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.674+0000 7fad597aeb80 1 -- 172.21.15.70:0/389828600 wait complete. 2024-06-04T05:03:53.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.674+0000 7fad597aeb80 1 -- start start 2024-06-04T05:03:53.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.674+0000 7fad597aeb80 1 Processor -- start 2024-06-04T05:03:53.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad597aeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd012c90 0x5559bd01f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad597aeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd013650 0x5559bd01c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad597aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd0177c0 0x5559bd01cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad597aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5559bce67140 con 0x5559bd0177c0 2024-06-04T05:03:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad597aeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5559bcec8f40 con 0x5559bd012c90 2024-06-04T05:03:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad597aeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5559bd01fc60 con 0x5559bd013650 2024-06-04T05:03:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad597aeb80 10 client.0 _ll_register_callbacks cb 0x5559bcefbaf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad4ccb5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd012c90 0x5559bd01f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad4ccb5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd012c90 0x5559bd01f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:52382/0 (socket says 172.21.15.70:52382) 2024-06-04T05:03:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad4ccb5700 1 -- 172.21.15.70:0/2321364420 learned_addr learned my addr 172.21.15.70:0/2321364420 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad4d4b6700 1 --2- 172.21.15.70:0/2321364420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd0177c0 0x5559bd01cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad47fff700 1 --2- 172.21.15.70:0/2321364420 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd013650 0x5559bd01c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad4ccb5700 1 -- 172.21.15.70:0/2321364420 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd013650 msgr2=0x5559bd01c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad4ccb5700 1 --2- 172.21.15.70:0/2321364420 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd013650 0x5559bd01c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad4ccb5700 1 -- 172.21.15.70:0/2321364420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd0177c0 msgr2=0x5559bd01cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad4ccb5700 1 --2- 172.21.15.70:0/2321364420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd0177c0 0x5559bd01cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad4ccb5700 1 -- 172.21.15.70:0/2321364420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559bceb0a70 con 0x5559bd012c90 2024-06-04T05:03:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.675+0000 7fad47fff700 1 --2- 172.21.15.70:0/2321364420 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd013650 0x5559bd01c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:03:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.676+0000 7fad4ccb5700 1 --2- 172.21.15.70:0/2321364420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd012c90 0x5559bd01f390 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fad3c00bfe0 tx=0x7fad3c003f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fea54b84b170bac9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.676+0000 7fad27fff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:03:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.676+0000 7fad27fff700 1 -- 172.21.15.70:0/2321364420 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c03c060 con 0x5559bd012c90 2024-06-04T05:03:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.676+0000 7fad27fff700 1 -- 172.21.15.70:0/2321364420 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fad3c024700 con 0x5559bd012c90 2024-06-04T05:03:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.676+0000 7fad597aeb80 1 -- 172.21.15.70:0/2321364420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad3c00bdf0 con 0x5559bd012c90 2024-06-04T05:03:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.676+0000 7fad27fff700 1 -- 172.21.15.70:0/2321364420 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fad3c024900 con 0x5559bd012c90 2024-06-04T05:03:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.676+0000 7fad597aeb80 10 client.26522 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.676+0000 7fad597aeb80 1 -- 172.21.15.70:0/2321364420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5559bcf1a900 con 0x5559bd012c90 2024-06-04T05:03:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.676+0000 7fad27fff700 1 -- 172.21.15.70:0/2321364420 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad3c00bdf0 con 0x5559bd012c90 2024-06-04T05:03:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.676+0000 7fad27fff700 1 -- 172.21.15.70:0/2321364420 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fad3c040a30 con 0x5559bd012c90 2024-06-04T05:03:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.676+0000 7fad597aeb80 10 client.26522 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.676+0000 7fad597aeb80 10 client.26522 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:03:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.676+0000 7fad597aeb80 20 client.26522 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.677+0000 7fad597aeb80 1 --2- 172.21.15.70:0/2321364420 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5559bd02c6b0 0x5559bd02eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.677+0000 7fad597aeb80 4 client.26522 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:03:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.677+0000 7fad597aeb80 1 -- 172.21.15.70:0/2321364420 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559bcf72e60 con 0x5559bd02c6b0 2024-06-04T05:03:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.677+0000 7fad47fff700 1 --2- 172.21.15.70:0/2321364420 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5559bd02c6b0 0x5559bd02eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.677+0000 7fad27fff700 10 client.26522 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.677+0000 7fad47fff700 1 --2- 172.21.15.70:0/2321364420 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5559bd02c6b0 0x5559bd02eaa0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=886aa04bfedb9171 server_cookie=e9092eaa8f8c7df5 in_seq=0 out_seq=0 2024-06-04T05:03:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.678+0000 7fad27fff700 1 -- 172.21.15.70:0/2321364420 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5559bcf72e60 con 0x5559bd02c6b0 2024-06-04T05:03:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.678+0000 7fad27fff700 10 client.26522 handle_command_reply: tid=0 2024-06-04T05:03:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.678+0000 7fad597aeb80 1 client.26522 shutdown 2024-06-04T05:03:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.678+0000 7fad597aeb80 1 -- 172.21.15.70:0/2321364420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd012c90 msgr2=0x5559bd01f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.678+0000 7fad597aeb80 1 --2- 172.21.15.70:0/2321364420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd012c90 0x5559bd01f390 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fad3c00bfe0 tx=0x7fad3c003f50 comp rx=0 tx=0).stop 2024-06-04T05:03:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.678+0000 7fad597aeb80 1 -- 172.21.15.70:0/2321364420 shutdown_connections 2024-06-04T05:03:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.678+0000 7fad597aeb80 1 --2- 172.21.15.70:0/2321364420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd0177c0 0x5559bd01cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.678+0000 7fad597aeb80 1 --2- 172.21.15.70:0/2321364420 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5559bd02c6b0 0x5559bd02eaa0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.678+0000 7fad597aeb80 1 --2- 172.21.15.70:0/2321364420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd012c90 0x5559bd01f390 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.678+0000 7fad597aeb80 1 --2- 172.21.15.70:0/2321364420 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd013650 0x5559bd01c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.678+0000 7fad27fff700 0 client.26522 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.678+0000 7fad597aeb80 1 -- 172.21.15.70:0/2321364420 >> 172.21.15.70:0/2321364420 conn(0x5559bcf3e3e0 msgr2=0x5559bd011c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.678+0000 7fad597aeb80 1 -- 172.21.15.70:0/2321364420 shutdown_connections 2024-06-04T05:03:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.678+0000 7fad597aeb80 1 -- 172.21.15.70:0/2321364420 wait complete. 2024-06-04T05:03:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.678+0000 7fad597aeb80 20 client.26522 trim_cache size 0 max 16384 2024-06-04T05:03:53.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.703+0000 7fad597aeb80 1 Processor -- start 2024-06-04T05:03:53.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.703+0000 7fad597aeb80 1 -- start start 2024-06-04T05:03:53.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.703+0000 7fad597aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd012c90 0x5559bd016bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.703+0000 7fad597aeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd013650 0x5559bd007f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.703+0000 7fad597aeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd0177c0 0x5559bd019d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.703+0000 7fad597aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fad481b4450 con 0x5559bd012c90 2024-06-04T05:03:53.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.703+0000 7fad597aeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad481b5f90 con 0x5559bd013650 2024-06-04T05:03:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.703+0000 7fad597aeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fad481b4b20 con 0x5559bd0177c0 2024-06-04T05:03:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.703+0000 7fad4ccb5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd012c90 0x5559bd016bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.703+0000 7fad4d4b6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd0177c0 0x5559bd019d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.703+0000 7fad4ccb5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd012c90 0x5559bd016bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49790/0 (socket says 172.21.15.70:49790) 2024-06-04T05:03:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.703+0000 7fad4ccb5700 1 -- 172.21.15.70:0/2613378468 learned_addr learned my addr 172.21.15.70:0/2613378468 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.703+0000 7fad47fff700 1 --2- 172.21.15.70:0/2613378468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd013650 0x5559bd007f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.704+0000 7fad4d4b6700 1 -- 172.21.15.70:0/2613378468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd013650 msgr2=0x5559bd007f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.704+0000 7fad4d4b6700 1 --2- 172.21.15.70:0/2613378468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd013650 0x5559bd007f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.704+0000 7fad4d4b6700 1 -- 172.21.15.70:0/2613378468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd012c90 msgr2=0x5559bd016bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.704+0000 7fad4d4b6700 1 --2- 172.21.15.70:0/2613378468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd012c90 0x5559bd016bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.704+0000 7fad4d4b6700 1 -- 172.21.15.70:0/2613378468 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559bd01d630 con 0x5559bd0177c0 2024-06-04T05:03:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.704+0000 7fad4ccb5700 1 --2- 172.21.15.70:0/2613378468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd012c90 0x5559bd016bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.704+0000 7fad47fff700 1 --2- 172.21.15.70:0/2613378468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd013650 0x5559bd007f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.704+0000 7fad4d4b6700 1 --2- 172.21.15.70:0/2613378468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd0177c0 0x5559bd019d80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fad400177b0 tx=0x7fad40009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfe28853034146b8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:53.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.704+0000 7fad327fc700 1 -- 172.21.15.70:0/2613378468 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad40004bb0 con 0x5559bd0177c0 2024-06-04T05:03:53.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.704+0000 7fad327fc700 1 -- 172.21.15.70:0/2613378468 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad40004d50 con 0x5559bd0177c0 2024-06-04T05:03:53.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.704+0000 7fad327fc700 1 -- 172.21.15.70:0/2613378468 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad40004030 con 0x5559bd0177c0 2024-06-04T05:03:53.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.705+0000 7fad597aeb80 1 -- 172.21.15.70:0/2613378468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd0177c0 msgr2=0x5559bd019d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:53.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.705+0000 7fad597aeb80 1 --2- 172.21.15.70:0/2613378468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd0177c0 0x5559bd019d80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fad400177b0 tx=0x7fad40009920 comp rx=0 tx=0).stop 2024-06-04T05:03:53.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.705+0000 7fad597aeb80 1 -- 172.21.15.70:0/2613378468 shutdown_connections 2024-06-04T05:03:53.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.705+0000 7fad597aeb80 1 --2- 172.21.15.70:0/2613378468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd0177c0 0x5559bd019d80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.705+0000 7fad597aeb80 1 --2- 172.21.15.70:0/2613378468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd012c90 0x5559bd016bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.705+0000 7fad597aeb80 1 --2- 172.21.15.70:0/2613378468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd013650 0x5559bd007f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.705+0000 7fad597aeb80 1 -- 172.21.15.70:0/2613378468 >> 172.21.15.70:0/2613378468 conn(0x5559bcf3e3e0 msgr2=0x5559bd019670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:53.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.705+0000 7fad597aeb80 1 -- 172.21.15.70:0/2613378468 shutdown_connections 2024-06-04T05:03:53.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.705+0000 7fad597aeb80 1 -- 172.21.15.70:0/2613378468 wait complete. 2024-06-04T05:03:53.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.705+0000 7fad597aeb80 1 -- start start 2024-06-04T05:03:53.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.705+0000 7fad597aeb80 1 Processor -- start 2024-06-04T05:03:53.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad597aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd012c90 0x5559bd08f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad597aeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd013650 0x5559bd08f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad597aeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd0177c0 0x5559bd090190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad597aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fad481b58c0 con 0x5559bd012c90 2024-06-04T05:03:53.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad597aeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad481b51f0 con 0x5559bd0177c0 2024-06-04T05:03:53.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad597aeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fad481b23f0 con 0x5559bd013650 2024-06-04T05:03:53.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad597aeb80 10 client.0 _ll_register_callbacks cb 0x5559bcefbaf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:53.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad4ccb5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd012c90 0x5559bd08f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:53.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad4ccb5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd012c90 0x5559bd08f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49806/0 (socket says 172.21.15.70:49806) 2024-06-04T05:03:53.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad4ccb5700 1 -- 172.21.15.70:0/2236529229 learned_addr learned my addr 172.21.15.70:0/2236529229 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:53.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad47fff700 1 --2- 172.21.15.70:0/2236529229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd013650 0x5559bd08f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:53.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad4ccb5700 1 -- 172.21.15.70:0/2236529229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd013650 msgr2=0x5559bd08f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:53.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad4ccb5700 1 --2- 172.21.15.70:0/2236529229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd013650 0x5559bd08f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad4ccb5700 1 -- 172.21.15.70:0/2236529229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd0177c0 msgr2=0x5559bd090190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:53.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad4ccb5700 1 --2- 172.21.15.70:0/2236529229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd0177c0 0x5559bd090190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad4ccb5700 1 -- 172.21.15.70:0/2236529229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559bd0909d0 con 0x5559bd012c90 2024-06-04T05:03:53.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad47fff700 1 --2- 172.21.15.70:0/2236529229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd013650 0x5559bd08f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:03:53.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad4ccb5700 1 --2- 172.21.15.70:0/2236529229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd012c90 0x5559bd08f410 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fad3c040e80 tx=0x7fad3c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b90e511dd1a897af server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:53.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.706+0000 7fad27fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:53.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.707+0000 7fad27fff700 1 -- 172.21.15.70:0/2236529229 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c029060 con 0x5559bd012c90 2024-06-04T05:03:53.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.707+0000 7fad27fff700 1 -- 172.21.15.70:0/2236529229 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7fad3c02f800 con 0x5559bd012c90 2024-06-04T05:03:53.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.707+0000 7fad597aeb80 1 -- 172.21.15.70:0/2236529229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad40004d50 con 0x5559bd012c90 2024-06-04T05:03:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.707+0000 7fad27fff700 1 -- 172.21.15.70:0/2236529229 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fad3c03c040 con 0x5559bd012c90 2024-06-04T05:03:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.707+0000 7fad597aeb80 10 client.17292 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.707+0000 7fad597aeb80 1 -- 172.21.15.70:0/2236529229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5559bd097d10 con 0x5559bd012c90 2024-06-04T05:03:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.707+0000 7fad27fff700 1 -- 172.21.15.70:0/2236529229 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad40004d50 con 0x5559bd012c90 2024-06-04T05:03:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.707+0000 7fad27fff700 1 -- 172.21.15.70:0/2236529229 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fad3c040460 con 0x5559bd012c90 2024-06-04T05:03:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.707+0000 7fad597aeb80 10 client.17292 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.707+0000 7fad597aeb80 10 client.17292 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:03:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.707+0000 7fad597aeb80 20 client.17292 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.707+0000 7fad597aeb80 1 --2- 172.21.15.70:0/2236529229 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5559bd0983e0 0x5559bd09a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.707+0000 7fad597aeb80 4 client.17292 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.95, "await_for": 45.0}] 2024-06-04T05:03:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.707+0000 7fad597aeb80 1 -- 172.21.15.70:0/2236529229 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.95, "await_for": 45.0}) -- 0x5559bcf6c640 con 0x5559bd0983e0 2024-06-04T05:03:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.708+0000 7fad47fff700 1 --2- 172.21.15.70:0/2236529229 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5559bd0983e0 0x5559bd09a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.708+0000 7fad27fff700 10 client.17292 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.708+0000 7fad47fff700 1 --2- 172.21.15.70:0/2236529229 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5559bd0983e0 0x5559bd09a800 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4477ddcd21275116 server_cookie=840c1c015dd6ee47 in_seq=0 out_seq=0 2024-06-04T05:03:53.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109962 2.11 MB/sec execute 166 sec latency 16.648 ms 2024-06-04T05:03:53.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.791+0000 7fad27fff700 1 -- 172.21.15.70:0/2236529229 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5559bcf6c640 con 0x5559bd0983e0 2024-06-04T05:03:53.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.791+0000 7fad27fff700 10 client.17292 handle_command_reply: tid=0 2024-06-04T05:03:53.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.791+0000 7fad597aeb80 1 client.17292 shutdown 2024-06-04T05:03:53.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.791+0000 7fad597aeb80 1 -- 172.21.15.70:0/2236529229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd012c90 msgr2=0x5559bd08f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:53.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.791+0000 7fad597aeb80 1 --2- 172.21.15.70:0/2236529229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd012c90 0x5559bd08f410 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fad3c040e80 tx=0x7fad3c003c10 comp rx=0 tx=0).stop 2024-06-04T05:03:53.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.791+0000 7fad597aeb80 1 -- 172.21.15.70:0/2236529229 shutdown_connections 2024-06-04T05:03:53.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.791+0000 7fad597aeb80 1 --2- 172.21.15.70:0/2236529229 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5559bd0983e0 0x5559bd09a800 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.791+0000 7fad597aeb80 1 --2- 172.21.15.70:0/2236529229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559bd0177c0 0x5559bd090190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.791+0000 7fad597aeb80 1 --2- 172.21.15.70:0/2236529229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559bd012c90 0x5559bd08f410 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.791+0000 7fad27fff700 0 client.17292 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:53.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.791+0000 7fad597aeb80 1 --2- 172.21.15.70:0/2236529229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5559bd013650 0x5559bd08f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.791+0000 7fad597aeb80 1 -- 172.21.15.70:0/2236529229 >> 172.21.15.70:0/2236529229 conn(0x5559bcf3e3e0 msgr2=0x5559bd019f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:53.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.791+0000 7fad597aeb80 1 -- 172.21.15.70:0/2236529229 shutdown_connections 2024-06-04T05:03:53.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.791+0000 7fad597aeb80 1 -- 172.21.15.70:0/2236529229 wait complete. 2024-06-04T05:03:53.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.791+0000 7fad597aeb80 20 client.17292 trim_cache size 0 max 16384 2024-06-04T05:03:53.795 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:03:53.795 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 32, 2024-06-04T05:03:53.795 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:03:53.795 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 18, 2024-06-04T05:03:53.795 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:03:53.795 INFO:teuthology.orchestra.run.smithi070.stdout: "b977e96e": { 2024-06-04T05:03:53.795 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 18, 2024-06-04T05:03:53.795 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:03:53.795 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:03:53.795 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:03:53.795 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:03:53.795 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:03:53.795 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:03:53.795 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 138.9, 2024-06-04T05:03:53.795 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:03:53.796 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:03:53.796 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:03:53.796 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:03:53.796 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:03:53.796 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:03:53.796 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:03:53.796 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:03:53.796 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:03:53.796 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:03:53.796 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:03:53.796 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:03:53.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.792+0000 7fad4ef19700 1 -- 172.21.15.70:0/3220414432 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fad380b2b20 msgr2=0x7fad380b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:53.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.792+0000 7fad4ef19700 1 --2- 172.21.15.70:0/3220414432 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fad380b2b20 0x7fad380b4fe0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fad3c000c00 tx=0x7fad3c005fb0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.792+0000 7fad4ef19700 1 -- 172.21.15.70:0/3220414432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad48103fb0 msgr2=0x7fad48075d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:53.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.792+0000 7fad4ef19700 1 --2- 172.21.15.70:0/3220414432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad48103fb0 0x7fad48075d80 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fad3400e9a0 tx=0x7fad3400ecb0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.792+0000 7fad4ef19700 1 -- 172.21.15.70:0/3220414432 shutdown_connections 2024-06-04T05:03:53.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.792+0000 7fad4ef19700 1 --2- 172.21.15.70:0/3220414432 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fad380b2b20 0x7fad380b4fe0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.792+0000 7fad4ef19700 1 --2- 172.21.15.70:0/3220414432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad481085a0 0x7fad480762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.792+0000 7fad4ef19700 1 --2- 172.21.15.70:0/3220414432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad481035f0 0x7fad48075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.792+0000 7fad4ef19700 1 --2- 172.21.15.70:0/3220414432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad48103fb0 0x7fad48075d80 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:53.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.792+0000 7fad4ef19700 1 -- 172.21.15.70:0/3220414432 >> 172.21.15.70:0/3220414432 conn(0x7fad4807a6f0 msgr2=0x7fad481017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:53.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.792+0000 7fad4ef19700 1 -- 172.21.15.70:0/3220414432 shutdown_connections 2024-06-04T05:03:53.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:03:53.792+0000 7fad4ef19700 1 -- 172.21.15.70:0/3220414432 wait complete. 2024-06-04T05:03:53.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:53 smithi112 ceph-mon[36453]: pgmap v204: 97 pgs: 97 active+clean; 71 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 740 KiB/s wr, 90 op/s 2024-06-04T05:03:53.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:53 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:03:53.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:53 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:03:53.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:53 smithi098 ceph-mon[35569]: pgmap v204: 97 pgs: 97 active+clean; 71 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 740 KiB/s wr, 90 op/s 2024-06-04T05:03:53.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:53 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:03:53.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:53 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:03:54.115 INFO:tasks.cephfs.filesystem:scrub status for tag:0b10a296-edf0-4f03-99c8-02076b4f7236 - {'path': '/', 'tag': '0b10a296-edf0-4f03-99c8-02076b4f7236', 'options': 'recursive,force'} 2024-06-04T05:03:54.723 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109965 2.09 MB/sec execute 167 sec latency 935.000 ms 2024-06-04T05:03:54.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:54 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:03:54.823 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b977e96e', quiesce took 8.8 seconds. Will release after: 10.149999999999999 2024-06-04T05:03:54.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:54 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:03:54.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:54 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:03:55.724 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109965 2.08 MB/sec execute 168 sec latency 1935.037 ms 2024-06-04T05:03:55.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:55 smithi070 ceph-mon[32631]: pgmap v205: 97 pgs: 97 active+clean; 71 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 737 KiB/s wr, 86 op/s 2024-06-04T05:03:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:55 smithi112 ceph-mon[36453]: pgmap v205: 97 pgs: 97 active+clean; 71 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 737 KiB/s wr, 86 op/s 2024-06-04T05:03:55.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:55 smithi098 ceph-mon[35569]: pgmap v205: 97 pgs: 97 active+clean; 71 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 737 KiB/s wr, 86 op/s 2024-06-04T05:03:56.724 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109965 2.07 MB/sec execute 169 sec latency 2935.124 ms 2024-06-04T05:03:57.724 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109965 2.06 MB/sec execute 170 sec latency 3935.207 ms 2024-06-04T05:03:57.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:57 smithi070 ceph-mon[32631]: pgmap v206: 97 pgs: 97 active+clean; 74 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-06-04T05:03:57.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.896+0000 7f9735d6a640 1 -- 172.21.15.70:0/267475086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9730107940 msgr2=0x7f9730107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:57.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.896+0000 7f9735d6a640 1 --2- 172.21.15.70:0/267475086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9730107940 0x7f9730107d20 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f9720009930 tx=0x7f972002a420 comp rx=0 tx=0).stop 2024-06-04T05:03:57.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.896+0000 7f9735d6a640 1 -- 172.21.15.70:0/267475086 shutdown_connections 2024-06-04T05:03:57.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.896+0000 7f9735d6a640 1 --2- 172.21.15.70:0/267475086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9730101980 0x7f9730101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.896+0000 7f9735d6a640 1 --2- 172.21.15.70:0/267475086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9730102340 0x7f973010e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.896+0000 7f9735d6a640 1 --2- 172.21.15.70:0/267475086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9730107940 0x7f9730107d20 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.896+0000 7f9735d6a640 1 -- 172.21.15.70:0/267475086 >> 172.21.15.70:0/267475086 conn(0x7f9730078ad0 msgr2=0x7f9730078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:57.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.896+0000 7f9735d6a640 1 -- 172.21.15.70:0/267475086 shutdown_connections 2024-06-04T05:03:57.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.896+0000 7f9735d6a640 1 -- 172.21.15.70:0/267475086 wait complete. 2024-06-04T05:03:57.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.896+0000 7f9735d6a640 1 Processor -- start 2024-06-04T05:03:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.897+0000 7f9735d6a640 1 -- start start 2024-06-04T05:03:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.897+0000 7f9735d6a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9730101980 0x7f9730199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.897+0000 7f9735d6a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9730102340 0x7f97301996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.897+0000 7f9735d6a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9730107940 0x7f973019da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.897+0000 7f9735d6a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9730112c00 con 0x7f9730102340 2024-06-04T05:03:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.897+0000 7f9735d6a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9730112a80 con 0x7f9730107940 2024-06-04T05:03:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.897+0000 7f9735d6a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9730112d80 con 0x7f9730101980 2024-06-04T05:03:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.897+0000 7f972effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9730102340 0x7f97301996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.897+0000 7f972effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9730102340 0x7f97301996a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39928/0 (socket says 172.21.15.70:39928) 2024-06-04T05:03:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.897+0000 7f972effd640 1 -- 172.21.15.70:0/3775593258 learned_addr learned my addr 172.21.15.70:0/3775593258 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.897+0000 7f972ffff640 1 --2- 172.21.15.70:0/3775593258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9730107940 0x7f973019da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.897+0000 7f972f7fe640 1 --2- 172.21.15.70:0/3775593258 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9730101980 0x7f9730199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.897+0000 7f972effd640 1 -- 172.21.15.70:0/3775593258 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9730101980 msgr2=0x7f9730199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.897+0000 7f972effd640 1 --2- 172.21.15.70:0/3775593258 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9730101980 0x7f9730199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.897+0000 7f972effd640 1 -- 172.21.15.70:0/3775593258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9730107940 msgr2=0x7f973019da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.897+0000 7f972effd640 1 --2- 172.21.15.70:0/3775593258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9730107940 0x7f973019da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.897+0000 7f972effd640 1 -- 172.21.15.70:0/3775593258 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f973019e150 con 0x7f9730102340 2024-06-04T05:03:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.898+0000 7f972effd640 1 --2- 172.21.15.70:0/3775593258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9730102340 0x7f97301996a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f972400e9f0 tx=0x7f972400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.898+0000 7f972cff9640 1 -- 172.21.15.70:0/3775593258 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f972400cd80 con 0x7f9730102340 2024-06-04T05:03:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.898+0000 7f972cff9640 1 -- 172.21.15.70:0/3775593258 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97240108c0 con 0x7f9730102340 2024-06-04T05:03:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.898+0000 7f9735d6a640 1 -- 172.21.15.70:0/3775593258 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f973019e440 con 0x7f9730102340 2024-06-04T05:03:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.898+0000 7f972cff9640 1 -- 172.21.15.70:0/3775593258 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97240187c0 con 0x7f9730102340 2024-06-04T05:03:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.898+0000 7f9735d6a640 1 -- 172.21.15.70:0/3775593258 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97301a5cd0 con 0x7f9730102340 2024-06-04T05:03:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.901+0000 7f9738a84740 1 Processor -- start 2024-06-04T05:03:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.901+0000 7f972cff9640 1 -- 172.21.15.70:0/3775593258 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f9724010430 con 0x7f9730102340 2024-06-04T05:03:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.901+0000 7f9738a84740 1 -- start start 2024-06-04T05:03:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.901+0000 7f9738a84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084da700 0x5634084daae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.901+0000 7f9738a84740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084db020 0x5634084e98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.901+0000 7f9738a84740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084e9e10 0x5634084ec200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.901+0000 7f9738a84740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56340844cc30 con 0x5634084da700 2024-06-04T05:03:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.901+0000 7f9738a84740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56340843c7d0 con 0x5634084db020 2024-06-04T05:03:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.901+0000 7f9738a84740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563408440810 con 0x5634084e9e10 2024-06-04T05:03:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.901+0000 7f972ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084da700 0x5634084daae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.901+0000 7f972ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084da700 0x5634084daae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39934/0 (socket says 172.21.15.70:39934) 2024-06-04T05:03:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.902+0000 7f972ffff640 1 -- 172.21.15.70:0/2709628213 learned_addr learned my addr 172.21.15.70:0/2709628213 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.902+0000 7f972effd640 1 --2- 172.21.15.70:0/2709628213 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084e9e10 0x5634084ec200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.902+0000 7f972f7fe640 1 --2- 172.21.15.70:0/2709628213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084db020 0x5634084e98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.902+0000 7f972ffff640 1 -- 172.21.15.70:0/2709628213 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084e9e10 msgr2=0x5634084ec200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:57.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.902+0000 7f972ffff640 1 --2- 172.21.15.70:0/2709628213 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084e9e10 0x5634084ec200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.902+0000 7f972ffff640 1 -- 172.21.15.70:0/2709628213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084db020 msgr2=0x5634084e98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:57.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.902+0000 7f972ffff640 1 --2- 172.21.15.70:0/2709628213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084db020 0x5634084e98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.902+0000 7f972ffff640 1 -- 172.21.15.70:0/2709628213 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563408151210 con 0x5634084da700 2024-06-04T05:03:57.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.902+0000 7f972cff9640 1 --2- 172.21.15.70:0/3775593258 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97180b2a10 0x7f97180b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.902+0000 7f972ffff640 1 --2- 172.21.15.70:0/2709628213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084da700 0x5634084daae0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f9728009fa0 tx=0x7f9728010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6269a80083847ed6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:57.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.902+0000 7f972f7fe640 1 --2- 172.21.15.70:0/3775593258 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97180b2a10 0x7f97180b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.902+0000 7f9715ffb640 1 -- 172.21.15.70:0/2709628213 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9728016070 con 0x5634084da700 2024-06-04T05:03:57.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.902+0000 7f9715ffb640 1 -- 172.21.15.70:0/2709628213 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9728004d20 con 0x5634084da700 2024-06-04T05:03:57.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.902+0000 7f9715ffb640 1 -- 172.21.15.70:0/2709628213 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97280127d0 con 0x5634084da700 2024-06-04T05:03:57.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.902+0000 7f972cff9640 1 -- 172.21.15.70:0/3775593258 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f97240e2a30 con 0x7f9730102340 2024-06-04T05:03:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.903+0000 7f9738a84740 1 -- 172.21.15.70:0/2709628213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084da700 msgr2=0x5634084daae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.903+0000 7f972f7fe640 1 --2- 172.21.15.70:0/3775593258 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97180b2a10 0x7f97180b4ed0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f97200057c0 tx=0x7f9720005a20 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.903+0000 7f9738a84740 1 --2- 172.21.15.70:0/2709628213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084da700 0x5634084daae0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f9728009fa0 tx=0x7f9728010800 comp rx=0 tx=0).stop 2024-06-04T05:03:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.903+0000 7f9738a84740 1 -- 172.21.15.70:0/2709628213 shutdown_connections 2024-06-04T05:03:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.903+0000 7f9738a84740 1 --2- 172.21.15.70:0/2709628213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084db020 0x5634084e98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.903+0000 7f9738a84740 1 --2- 172.21.15.70:0/2709628213 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084e9e10 0x5634084ec200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.903+0000 7f9738a84740 1 --2- 172.21.15.70:0/2709628213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084da700 0x5634084daae0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.903+0000 7f9738a84740 1 -- 172.21.15.70:0/2709628213 >> 172.21.15.70:0/2709628213 conn(0x5634084d6d30 msgr2=0x5634084d7110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.903+0000 7f9738a84740 1 -- 172.21.15.70:0/2709628213 shutdown_connections 2024-06-04T05:03:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.903+0000 7f9738a84740 1 -- 172.21.15.70:0/2709628213 wait complete. 2024-06-04T05:03:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.903+0000 7f9738a84740 1 -- start start 2024-06-04T05:03:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f9738a84740 1 Processor -- start 2024-06-04T05:03:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f9738a84740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084da700 0x5634084e6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f9738a84740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084db020 0x5634084e73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f9738a84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084e9e10 0x5634084e78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f9738a84740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563408122790 con 0x5634084e9e10 2024-06-04T05:03:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f9738a84740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5634083f8890 con 0x5634084db020 2024-06-04T05:03:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f9738a84740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5634084eca40 con 0x5634084da700 2024-06-04T05:03:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f9738a84740 10 client.0 _ll_register_callbacks cb 0x563408150a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f972ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084e9e10 0x5634084e78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f972ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084e9e10 0x5634084e78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39944/0 (socket says 172.21.15.70:39944) 2024-06-04T05:03:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f972ffff640 1 -- 172.21.15.70:0/3703791184 learned_addr learned my addr 172.21.15.70:0/3703791184 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f972effd640 1 --2- 172.21.15.70:0/3703791184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084db020 0x5634084e73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f972f7fe640 1 --2- 172.21.15.70:0/3703791184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084da700 0x5634084e6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f972ffff640 1 -- 172.21.15.70:0/3703791184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084da700 msgr2=0x5634084e6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f972ffff640 1 --2- 172.21.15.70:0/3703791184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084da700 0x5634084e6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f972ffff640 1 -- 172.21.15.70:0/3703791184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084db020 msgr2=0x5634084e73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f972ffff640 1 --2- 172.21.15.70:0/3703791184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084db020 0x5634084e73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f972ffff640 1 -- 172.21.15.70:0/3703791184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56340821d620 con 0x5634084e9e10 2024-06-04T05:03:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f972ffff640 1 --2- 172.21.15.70:0/3703791184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084e9e10 0x5634084e78e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f9728006010 tx=0x7f9728004f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f53c323ab11e83be server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.904+0000 7f970effd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.905+0000 7f970effd640 1 -- 172.21.15.70:0/3703791184 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9728016070 con 0x5634084e9e10 2024-06-04T05:03:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.905+0000 7f970effd640 1 -- 172.21.15.70:0/3703791184 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f9728003d80 con 0x5634084e9e10 2024-06-04T05:03:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.905+0000 7f9738a84740 1 -- 172.21.15.70:0/3703791184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9728004d20 con 0x5634084e9e10 2024-06-04T05:03:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.905+0000 7f970effd640 1 -- 172.21.15.70:0/3703791184 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9728005140 con 0x5634084e9e10 2024-06-04T05:03:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.905+0000 7f9738a84740 10 client.17304 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.905+0000 7f9738a84740 1 -- 172.21.15.70:0/3703791184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5634084f88d0 con 0x5634084e9e10 2024-06-04T05:03:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.905+0000 7f970effd640 1 -- 172.21.15.70:0/3703791184 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9728004d20 con 0x5634084e9e10 2024-06-04T05:03:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.905+0000 7f970effd640 1 -- 172.21.15.70:0/3703791184 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f972800f040 con 0x5634084e9e10 2024-06-04T05:03:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.905+0000 7f9738a84740 10 client.17304 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.905+0000 7f9738a84740 10 client.17304 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.905+0000 7f9738a84740 20 client.17304 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.905+0000 7f9738a84740 1 --2- 172.21.15.70:0/3703791184 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5634084f9d30 0x5634084fc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.905+0000 7f9738a84740 4 client.17304 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:03:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.905+0000 7f9738a84740 1 -- 172.21.15.70:0/3703791184 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56340844eb80 con 0x5634084f9d30 2024-06-04T05:03:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.906+0000 7f972f7fe640 1 --2- 172.21.15.70:0/3703791184 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5634084f9d30 0x5634084fc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.906+0000 7f970effd640 10 client.17304 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.906+0000 7f972f7fe640 1 --2- 172.21.15.70:0/3703791184 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5634084f9d30 0x5634084fc120 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c8aadba16fec178 server_cookie=6c14d67167c3c86b in_seq=0 out_seq=0 2024-06-04T05:03:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.907+0000 7f970effd640 1 -- 172.21.15.70:0/3703791184 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56340844eb80 con 0x5634084f9d30 2024-06-04T05:03:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.907+0000 7f970effd640 10 client.17304 handle_command_reply: tid=0 2024-06-04T05:03:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.907+0000 7f9738a84740 1 client.17304 shutdown 2024-06-04T05:03:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.907+0000 7f9738a84740 1 -- 172.21.15.70:0/3703791184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084e9e10 msgr2=0x5634084e78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.907+0000 7f9738a84740 1 --2- 172.21.15.70:0/3703791184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084e9e10 0x5634084e78e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f9728006010 tx=0x7f9728004f50 comp rx=0 tx=0).stop 2024-06-04T05:03:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.907+0000 7f9738a84740 1 -- 172.21.15.70:0/3703791184 shutdown_connections 2024-06-04T05:03:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.907+0000 7f9738a84740 1 --2- 172.21.15.70:0/3703791184 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5634084f9d30 0x5634084fc120 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.907+0000 7f9738a84740 1 --2- 172.21.15.70:0/3703791184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084db020 0x5634084e73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.907+0000 7f970effd640 0 client.17304 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.907+0000 7f9738a84740 1 --2- 172.21.15.70:0/3703791184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084e9e10 0x5634084e78e0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.907+0000 7f9738a84740 1 --2- 172.21.15.70:0/3703791184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084da700 0x5634084e6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.907+0000 7f9738a84740 1 -- 172.21.15.70:0/3703791184 >> 172.21.15.70:0/3703791184 conn(0x5634084d6d30 msgr2=0x5634084dcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.907+0000 7f9738a84740 1 -- 172.21.15.70:0/3703791184 shutdown_connections 2024-06-04T05:03:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.907+0000 7f9738a84740 1 -- 172.21.15.70:0/3703791184 wait complete. 2024-06-04T05:03:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.907+0000 7f9738a84740 20 client.17304 trim_cache size 0 max 16384 2024-06-04T05:03:57.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.922+0000 7f9738a84740 1 Processor -- start 2024-06-04T05:03:57.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.922+0000 7f9738a84740 1 -- start start 2024-06-04T05:03:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.922+0000 7f9738a84740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084da700 0x5634084f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.922+0000 7f9738a84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084db020 0x5634084f5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.922+0000 7f9738a84740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084e9e10 0x563408501900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.922+0000 7f9738a84740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f97301ac3e0 con 0x5634084db020 2024-06-04T05:03:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.922+0000 7f9738a84740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97301adf20 con 0x5634084e9e10 2024-06-04T05:03:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.922+0000 7f9738a84740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f97301acab0 con 0x5634084da700 2024-06-04T05:03:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.923+0000 7f972effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084db020 0x5634084f5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.923+0000 7f972effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084db020 0x5634084f5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39954/0 (socket says 172.21.15.70:39954) 2024-06-04T05:03:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.923+0000 7f972effd640 1 -- 172.21.15.70:0/4047149870 learned_addr learned my addr 172.21.15.70:0/4047149870 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.923+0000 7f972f7fe640 1 --2- 172.21.15.70:0/4047149870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084da700 0x5634084f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.923+0000 7f972effd640 1 -- 172.21.15.70:0/4047149870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084da700 msgr2=0x5634084f51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.923+0000 7f972effd640 1 --2- 172.21.15.70:0/4047149870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084da700 0x5634084f51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.923+0000 7f972effd640 1 -- 172.21.15.70:0/4047149870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084e9e10 msgr2=0x563408501900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.923+0000 7f972ffff640 1 --2- 172.21.15.70:0/4047149870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084e9e10 0x563408501900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.923+0000 7f972effd640 1 --2- 172.21.15.70:0/4047149870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084e9e10 0x563408501900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.923+0000 7f972effd640 1 -- 172.21.15.70:0/4047149870 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634084fc490 con 0x5634084db020 2024-06-04T05:03:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.923+0000 7f972f7fe640 1 --2- 172.21.15.70:0/4047149870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084da700 0x5634084f51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:03:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.923+0000 7f972effd640 1 --2- 172.21.15.70:0/4047149870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084db020 0x5634084f5720 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f97240a9ca0 tx=0x7f9724010cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae8bd298015a6ff server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.923+0000 7f97157fa640 1 -- 172.21.15.70:0/4047149870 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97240a9820 con 0x5634084db020 2024-06-04T05:03:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.923+0000 7f97157fa640 1 -- 172.21.15.70:0/4047149870 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97240aa410 con 0x5634084db020 2024-06-04T05:03:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.923+0000 7f97157fa640 1 -- 172.21.15.70:0/4047149870 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97240a5fa0 con 0x5634084db020 2024-06-04T05:03:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.924+0000 7f9738a84740 1 -- 172.21.15.70:0/4047149870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084db020 msgr2=0x5634084f5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.924+0000 7f9738a84740 1 --2- 172.21.15.70:0/4047149870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084db020 0x5634084f5720 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f97240a9ca0 tx=0x7f9724010cc0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.924+0000 7f9738a84740 1 -- 172.21.15.70:0/4047149870 shutdown_connections 2024-06-04T05:03:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.924+0000 7f9738a84740 1 --2- 172.21.15.70:0/4047149870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084e9e10 0x563408501900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.924+0000 7f9738a84740 1 --2- 172.21.15.70:0/4047149870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084db020 0x5634084f5720 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.924+0000 7f9738a84740 1 --2- 172.21.15.70:0/4047149870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084da700 0x5634084f51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.924+0000 7f9738a84740 1 -- 172.21.15.70:0/4047149870 >> 172.21.15.70:0/4047149870 conn(0x5634084d6d30 msgr2=0x5634084f4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.924+0000 7f9738a84740 1 -- 172.21.15.70:0/4047149870 shutdown_connections 2024-06-04T05:03:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.924+0000 7f9738a84740 1 -- 172.21.15.70:0/4047149870 wait complete. 2024-06-04T05:03:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.924+0000 7f9738a84740 1 -- start start 2024-06-04T05:03:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.924+0000 7f9738a84740 1 Processor -- start 2024-06-04T05:03:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f9738a84740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084da700 0x5634084efa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f9738a84740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084db020 0x5634084eff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f9738a84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084e9e10 0x5634084f04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f9738a84740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f97301ad850 con 0x5634084e9e10 2024-06-04T05:03:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f9738a84740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97301ad180 con 0x5634084da700 2024-06-04T05:03:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f9738a84740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f97301aa3c0 con 0x5634084db020 2024-06-04T05:03:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f9738a84740 10 client.0 _ll_register_callbacks cb 0x7f9728009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:03:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f972ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084e9e10 0x5634084f04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f972ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084e9e10 0x5634084f04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39966/0 (socket says 172.21.15.70:39966) 2024-06-04T05:03:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f972ffff640 1 -- 172.21.15.70:0/2221825434 learned_addr learned my addr 172.21.15.70:0/2221825434 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:03:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f972f7fe640 1 --2- 172.21.15.70:0/2221825434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084da700 0x5634084efa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f972effd640 1 --2- 172.21.15.70:0/2221825434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084db020 0x5634084eff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f972ffff640 1 -- 172.21.15.70:0/2221825434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084db020 msgr2=0x5634084eff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f972ffff640 1 --2- 172.21.15.70:0/2221825434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084db020 0x5634084eff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f972ffff640 1 -- 172.21.15.70:0/2221825434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084da700 msgr2=0x5634084efa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:03:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f972ffff640 1 --2- 172.21.15.70:0/2221825434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084da700 0x5634084efa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f972ffff640 1 -- 172.21.15.70:0/2221825434 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563408563810 con 0x5634084e9e10 2024-06-04T05:03:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f972effd640 1 --2- 172.21.15.70:0/2221825434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084db020 0x5634084eff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:03:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f972ffff640 1 --2- 172.21.15.70:0/2221825434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084e9e10 0x5634084f04d0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f9728009a80 tx=0x7f97280050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b077c5c04ce5a27 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:03:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f970effd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:03:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f970effd640 1 -- 172.21.15.70:0/2221825434 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9728016070 con 0x5634084e9e10 2024-06-04T05:03:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f970effd640 1 -- 172.21.15.70:0/2221825434 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8527+0+0 (secure 0 0 0) 0x7f97280056f0 con 0x5634084e9e10 2024-06-04T05:03:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f9738a84740 1 -- 172.21.15.70:0/2221825434 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97240aa410 con 0x5634084e9e10 2024-06-04T05:03:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.925+0000 7f970effd640 1 -- 172.21.15.70:0/2221825434 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f972803d050 con 0x5634084e9e10 2024-06-04T05:03:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.926+0000 7f9738a84740 10 client.17310 fetch_fsmap learned FSMap version 32 2024-06-04T05:03:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.926+0000 7f9738a84740 1 -- 172.21.15.70:0/2221825434 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563408564a70 con 0x5634084e9e10 2024-06-04T05:03:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.926+0000 7f970effd640 1 -- 172.21.15.70:0/2221825434 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97240aa410 con 0x5634084e9e10 2024-06-04T05:03:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.926+0000 7f970effd640 1 -- 172.21.15.70:0/2221825434 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f972801b450 con 0x5634084e9e10 2024-06-04T05:03:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.926+0000 7f9738a84740 10 client.17310 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:03:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.926+0000 7f9738a84740 10 client.17310 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:03:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.926+0000 7f9738a84740 20 client.17310 populate_metadata read hostname 'smithi070' 2024-06-04T05:03:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.926+0000 7f9738a84740 1 --2- 172.21.15.70:0/2221825434 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5634085651b0 0x5634085675d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:03:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.926+0000 7f9738a84740 4 client.17310 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:03:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.926+0000 7f9738a84740 1 -- 172.21.15.70:0/2221825434 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563408448360 con 0x5634085651b0 2024-06-04T05:03:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.926+0000 7f972f7fe640 1 --2- 172.21.15.70:0/2221825434 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5634085651b0 0x5634085675d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:03:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.926+0000 7f972f7fe640 1 --2- 172.21.15.70:0/2221825434 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5634085651b0 0x5634085675d0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f81f7c0fd5bbf89 server_cookie=3c5670e7acc41c41 in_seq=0 out_seq=0 2024-06-04T05:03:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.926+0000 7f970effd640 10 client.17310 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:57.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:57 smithi112 ceph-mon[36453]: pgmap v206: 97 pgs: 97 active+clean; 74 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-06-04T05:03:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.944+0000 7f970effd640 1 -- 172.21.15.70:0/2221825434 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x563408448360 con 0x5634085651b0 2024-06-04T05:03:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.944+0000 7f970effd640 10 client.17310 handle_command_reply: tid=0 2024-06-04T05:03:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.944+0000 7f9738a84740 1 client.17310 shutdown 2024-06-04T05:03:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.944+0000 7f9738a84740 1 -- 172.21.15.70:0/2221825434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084e9e10 msgr2=0x5634084f04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.944+0000 7f9738a84740 1 --2- 172.21.15.70:0/2221825434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084e9e10 0x5634084f04d0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f9728009a80 tx=0x7f97280050b0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.944+0000 7f9738a84740 1 -- 172.21.15.70:0/2221825434 shutdown_connections 2024-06-04T05:03:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.944+0000 7f9738a84740 1 --2- 172.21.15.70:0/2221825434 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5634085651b0 0x5634085675d0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.944+0000 7f9738a84740 1 --2- 172.21.15.70:0/2221825434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634084e9e10 0x5634084f04d0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.944+0000 7f9738a84740 1 --2- 172.21.15.70:0/2221825434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634084db020 0x5634084eff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.944+0000 7f9738a84740 1 --2- 172.21.15.70:0/2221825434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634084da700 0x5634084efa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.944+0000 7f970effd640 0 client.17310 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:03:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.944+0000 7f9738a84740 1 -- 172.21.15.70:0/2221825434 >> 172.21.15.70:0/2221825434 conn(0x5634084d6d30 msgr2=0x5634084f5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.944+0000 7f9738a84740 1 -- 172.21.15.70:0/2221825434 shutdown_connections 2024-06-04T05:03:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.944+0000 7f9738a84740 1 -- 172.21.15.70:0/2221825434 wait complete. 2024-06-04T05:03:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.945+0000 7f9738a84740 20 client.17310 trim_cache size 0 max 16384 2024-06-04T05:03:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:03:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:03:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:03:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.945+0000 7f9735d6a640 1 -- 172.21.15.70:0/3775593258 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97180b2a10 msgr2=0x7f97180b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.945+0000 7f9735d6a640 1 --2- 172.21.15.70:0/3775593258 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97180b2a10 0x7f97180b4ed0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f97200057c0 tx=0x7f9720005a20 comp rx=0 tx=0).stop 2024-06-04T05:03:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.945+0000 7f9735d6a640 1 -- 172.21.15.70:0/3775593258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9730102340 msgr2=0x7f97301996a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:03:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.945+0000 7f9735d6a640 1 --2- 172.21.15.70:0/3775593258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9730102340 0x7f97301996a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f972400e9f0 tx=0x7f972400eeb0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.945+0000 7f9735d6a640 1 -- 172.21.15.70:0/3775593258 shutdown_connections 2024-06-04T05:03:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.945+0000 7f9735d6a640 1 --2- 172.21.15.70:0/3775593258 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97180b2a10 0x7f97180b4ed0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.945+0000 7f9735d6a640 1 --2- 172.21.15.70:0/3775593258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9730107940 0x7f973019da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.945+0000 7f9735d6a640 1 --2- 172.21.15.70:0/3775593258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9730102340 0x7f97301996a0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.945+0000 7f9735d6a640 1 --2- 172.21.15.70:0/3775593258 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9730101980 0x7f9730199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:03:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.945+0000 7f9735d6a640 1 -- 172.21.15.70:0/3775593258 >> 172.21.15.70:0/3775593258 conn(0x7f9730078ad0 msgr2=0x7f9730104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:03:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.945+0000 7f9735d6a640 1 -- 172.21.15.70:0/3775593258 shutdown_connections 2024-06-04T05:03:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:03:57.945+0000 7f9735d6a640 1 -- 172.21.15.70:0/3775593258 wait complete. 2024-06-04T05:03:57.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:57 smithi098 ceph-mon[35569]: pgmap v206: 97 pgs: 97 active+clean; 74 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-06-04T05:03:58.724 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109965 2.05 MB/sec execute 171 sec latency 4935.249 ms 2024-06-04T05:03:59.724 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109965 2.03 MB/sec execute 172 sec latency 5935.302 ms 2024-06-04T05:03:59.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:03:59 smithi070 ceph-mon[32631]: pgmap v207: 97 pgs: 97 active+clean; 74 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-06-04T05:03:59.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:03:59 smithi112 ceph-mon[36453]: pgmap v207: 97 pgs: 97 active+clean; 74 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-06-04T05:03:59.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:03:59 smithi098 ceph-mon[35569]: pgmap v207: 97 pgs: 97 active+clean; 74 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-06-04T05:04:00.724 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109965 2.02 MB/sec execute 173 sec latency 6935.399 ms 2024-06-04T05:04:01.724 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109965 2.01 MB/sec execute 174 sec latency 7935.465 ms 2024-06-04T05:04:01.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:01 smithi112 ceph-mon[36453]: pgmap v208: 97 pgs: 97 active+clean; 74 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-06-04T05:04:01.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:01 smithi098 ceph-mon[35569]: pgmap v208: 97 pgs: 97 active+clean; 74 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-06-04T05:04:02.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:01 smithi070 ceph-mon[32631]: pgmap v208: 97 pgs: 97 active+clean; 74 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-06-04T05:04:02.724 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109965 2.00 MB/sec execute 175 sec latency 8935.526 ms 2024-06-04T05:04:03.724 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109965 1.99 MB/sec execute 176 sec latency 9935.589 ms 2024-06-04T05:04:03.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:03 smithi112 ceph-mon[36453]: pgmap v209: 97 pgs: 97 active+clean; 70 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 1.1 MiB/s wr, 137 op/s 2024-06-04T05:04:03.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:03 smithi098 ceph-mon[35569]: pgmap v209: 97 pgs: 97 active+clean; 70 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 1.1 MiB/s wr, 137 op/s 2024-06-04T05:04:04.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:03 smithi070 ceph-mon[32631]: pgmap v209: 97 pgs: 97 active+clean; 70 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 1.1 MiB/s wr, 137 op/s 2024-06-04T05:04:04.724 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109965 1.98 MB/sec execute 177 sec latency 10935.679 ms 2024-06-04T05:04:04.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:04 smithi112 ceph-mon[36453]: osdmap e83: 12 total, 12 up, 12 in 2024-06-04T05:04:04.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:04:04.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:04 smithi098 ceph-mon[35569]: osdmap e83: 12 total, 12 up, 12 in 2024-06-04T05:04:05.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:04 smithi070 ceph-mon[32631]: osdmap e83: 12 total, 12 up, 12 in 2024-06-04T05:04:05.242 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:04:05.724 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109965 1.97 MB/sec execute 178 sec latency 11935.766 ms 2024-06-04T05:04:05.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:05 smithi112 ceph-mon[36453]: pgmap v211: 97 pgs: 97 active+clean; 70 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 665 KiB/s wr, 81 op/s 2024-06-04T05:04:05.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:05 smithi098 ceph-mon[35569]: pgmap v211: 97 pgs: 97 active+clean; 70 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 665 KiB/s wr, 81 op/s 2024-06-04T05:04:06.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:05 smithi070 ceph-mon[32631]: pgmap v211: 97 pgs: 97 active+clean; 70 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 665 KiB/s wr, 81 op/s 2024-06-04T05:04:06.724 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109965 1.95 MB/sec execute 179 sec latency 12935.854 ms 2024-06-04T05:04:07.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.022+0000 7ff69a4b7700 1 -- 172.21.15.70:0/3968292143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff694108620 msgr2=0x7ff694108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:07.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.022+0000 7ff69a4b7700 1 --2- 172.21.15.70:0/3968292143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff694108620 0x7ff694108a00 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7ff684009990 tx=0x7ff684009ca0 comp rx=0 tx=0).stop 2024-06-04T05:04:07.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.023+0000 7ff69a4b7700 1 -- 172.21.15.70:0/3968292143 shutdown_connections 2024-06-04T05:04:07.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.023+0000 7ff69a4b7700 1 --2- 172.21.15.70:0/3968292143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6941028c0 0x7ff694110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:07.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.023+0000 7ff69a4b7700 1 --2- 172.21.15.70:0/3968292143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff694108620 0x7ff694108a00 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:07.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.023+0000 7ff69a4b7700 1 --2- 172.21.15.70:0/3968292143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff694101f00 0x7ff694102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:07.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.023+0000 7ff69a4b7700 1 -- 172.21.15.70:0/3968292143 >> 172.21.15.70:0/3968292143 conn(0x7ff694077780 msgr2=0x7ff694077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:07.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.023+0000 7ff69a4b7700 1 -- 172.21.15.70:0/3968292143 shutdown_connections 2024-06-04T05:04:07.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.023+0000 7ff69a4b7700 1 -- 172.21.15.70:0/3968292143 wait complete. 2024-06-04T05:04:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.023+0000 7ff69a4b7700 1 Processor -- start 2024-06-04T05:04:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.023+0000 7ff69a4b7700 1 -- start start 2024-06-04T05:04:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.024+0000 7ff69a4b7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff694101f00 0x7ff6941030e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.024+0000 7ff69a4b7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6941028c0 0x7ff694103620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.024+0000 7ff69a4b7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff694108620 0x7ff694103b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.024+0000 7ff69a4b7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff6941136c0 con 0x7ff6941028c0 2024-06-04T05:04:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.024+0000 7ff69a4b7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff694113540 con 0x7ff694101f00 2024-06-04T05:04:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.024+0000 7ff69a4b7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff694113840 con 0x7ff694108620 2024-06-04T05:04:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.024+0000 7ff6937fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6941028c0 0x7ff694103620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.024+0000 7ff6937fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6941028c0 0x7ff694103620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36714/0 (socket says 172.21.15.70:36714) 2024-06-04T05:04:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.024+0000 7ff6937fe700 1 -- 172.21.15.70:0/18721558 learned_addr learned my addr 172.21.15.70:0/18721558 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.024+0000 7ff693fff700 1 --2- 172.21.15.70:0/18721558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff694101f00 0x7ff6941030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.024+0000 7ff698a54700 1 --2- 172.21.15.70:0/18721558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff694108620 0x7ff694103b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.024+0000 7ff6937fe700 1 -- 172.21.15.70:0/18721558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff694108620 msgr2=0x7ff694103b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.024+0000 7ff6937fe700 1 --2- 172.21.15.70:0/18721558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff694108620 0x7ff694103b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.024+0000 7ff6937fe700 1 -- 172.21.15.70:0/18721558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff694101f00 msgr2=0x7ff6941030e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.024+0000 7ff6937fe700 1 --2- 172.21.15.70:0/18721558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff694101f00 0x7ff6941030e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.024+0000 7ff6937fe700 1 -- 172.21.15.70:0/18721558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6941a8220 con 0x7ff6941028c0 2024-06-04T05:04:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.025+0000 7ff6937fe700 1 --2- 172.21.15.70:0/18721558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6941028c0 0x7ff694103620 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7ff68800b530 tx=0x7ff68800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.025+0000 7ff6917fa700 1 -- 172.21.15.70:0/18721558 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff688015020 con 0x7ff6941028c0 2024-06-04T05:04:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.025+0000 7ff69a4b7700 1 -- 172.21.15.70:0/18721558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6941a8510 con 0x7ff6941028c0 2024-06-04T05:04:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.025+0000 7ff6917fa700 1 -- 172.21.15.70:0/18721558 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff688011cc0 con 0x7ff6941028c0 2024-06-04T05:04:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.025+0000 7ff6917fa700 1 -- 172.21.15.70:0/18721558 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff688010c80 con 0x7ff6941028c0 2024-06-04T05:04:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.025+0000 7ff69a4b7700 1 -- 172.21.15.70:0/18721558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6941a8a50 con 0x7ff6941028c0 2024-06-04T05:04:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.026+0000 7ff69a4b7700 1 -- 172.21.15.70:0/18721558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff69406a230 con 0x7ff6941028c0 2024-06-04T05:04:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.028+0000 7ff6917fa700 1 -- 172.21.15.70:0/18721558 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff688018020 con 0x7ff6941028c0 2024-06-04T05:04:07.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.028+0000 7ff6917fa700 1 --2- 172.21.15.70:0/18721558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff67c0b2a70 0x7ff67c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:07.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.028+0000 7ff693fff700 1 --2- 172.21.15.70:0/18721558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff67c0b2a70 0x7ff67c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:07.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.028+0000 7ff6917fa700 1 -- 172.21.15.70:0/18721558 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8615+0+0 (secure 0 0 0) 0x7ff6880e2130 con 0x7ff6941028c0 2024-06-04T05:04:07.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.029+0000 7ff693fff700 1 --2- 172.21.15.70:0/18721558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff67c0b2a70 0x7ff67c0b4f30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff68400b5c0 tx=0x7ff684005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:07.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.032+0000 7ff6917fa700 1 -- 172.21.15.70:0/18721558 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff68801f020 con 0x7ff6941028c0 2024-06-04T05:04:07.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.282+0000 7ff69a4b7700 1 -- 172.21.15.70:0/18721558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff694104610 con 0x7ff6941028c0 2024-06-04T05:04:07.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.283+0000 7ff6917fa700 1 -- 172.21.15.70:0/18721558 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11929 (secure 0 0 0) 0x7ff6880a9c60 con 0x7ff6941028c0 2024-06-04T05:04:07.286 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:04:07.286 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:01:52.807483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:04:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.286+0000 7ff69a4b7700 1 -- 172.21.15.70:0/18721558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff67c0b2a70 msgr2=0x7ff67c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.286+0000 7ff69a4b7700 1 --2- 172.21.15.70:0/18721558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff67c0b2a70 0x7ff67c0b4f30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff68400b5c0 tx=0x7ff684005960 comp rx=0 tx=0).stop 2024-06-04T05:04:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.286+0000 7ff69a4b7700 1 -- 172.21.15.70:0/18721558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6941028c0 msgr2=0x7ff694103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.286+0000 7ff69a4b7700 1 --2- 172.21.15.70:0/18721558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6941028c0 0x7ff694103620 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7ff68800b530 tx=0x7ff68800b840 comp rx=0 tx=0).stop 2024-06-04T05:04:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.286+0000 7ff69a4b7700 1 -- 172.21.15.70:0/18721558 shutdown_connections 2024-06-04T05:04:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.286+0000 7ff69a4b7700 1 --2- 172.21.15.70:0/18721558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff67c0b2a70 0x7ff67c0b4f30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.286+0000 7ff69a4b7700 1 --2- 172.21.15.70:0/18721558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff694108620 0x7ff694103b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.286+0000 7ff69a4b7700 1 --2- 172.21.15.70:0/18721558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff694101f00 0x7ff6941030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.286+0000 7ff69a4b7700 1 --2- 172.21.15.70:0/18721558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6941028c0 0x7ff694103620 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.286+0000 7ff69a4b7700 1 -- 172.21.15.70:0/18721558 >> 172.21.15.70:0/18721558 conn(0x7ff694077780 msgr2=0x7ff694100c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.286+0000 7ff69a4b7700 1 -- 172.21.15.70:0/18721558 shutdown_connections 2024-06-04T05:04:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:07.286+0000 7ff69a4b7700 1 -- 172.21.15.70:0/18721558 wait complete. 2024-06-04T05:04:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 32 2024-06-04T05:04:07.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109965 1.94 MB/sec execute 180 sec latency 13935.907 ms 2024-06-04T05:04:07.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:07 smithi070 ceph-mon[32631]: pgmap v212: 97 pgs: 97 active+clean; 70 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 87 KiB/s wr, 37 op/s 2024-06-04T05:04:07.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:07 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/18721558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:04:07.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:07 smithi112 ceph-mon[36453]: pgmap v212: 97 pgs: 97 active+clean; 70 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 87 KiB/s wr, 37 op/s 2024-06-04T05:04:07.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:07 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/18721558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:04:07.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:07 smithi098 ceph-mon[35569]: pgmap v212: 97 pgs: 97 active+clean; 70 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 87 KiB/s wr, 37 op/s 2024-06-04T05:04:07.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:07 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/18721558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:04:07.998 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id b977e96e --release --await' 2024-06-04T05:04:07.999 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id b977e96e --release --await 2024-06-04T05:04:08.280 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:04:08.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109965 1.93 MB/sec execute 181 sec latency 14936.042 ms 2024-06-04T05:04:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:08 smithi070 ceph-mon[32631]: osdmap e84: 12 total, 12 up, 12 in 2024-06-04T05:04:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:08 smithi070 ceph-mon[32631]: pgmap v214: 97 pgs: 97 active+clean; 70 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 109 KiB/s wr, 46 op/s 2024-06-04T05:04:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:08 smithi112 ceph-mon[36453]: osdmap e84: 12 total, 12 up, 12 in 2024-06-04T05:04:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:08 smithi112 ceph-mon[36453]: pgmap v214: 97 pgs: 97 active+clean; 70 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 109 KiB/s wr, 46 op/s 2024-06-04T05:04:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:08 smithi098 ceph-mon[35569]: osdmap e84: 12 total, 12 up, 12 in 2024-06-04T05:04:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:08 smithi098 ceph-mon[35569]: pgmap v214: 97 pgs: 97 active+clean; 70 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 109 KiB/s wr, 46 op/s 2024-06-04T05:04:09.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109965 1.92 MB/sec execute 182 sec latency 15936.069 ms 2024-06-04T05:04:10.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 109965 1.91 MB/sec execute 183 sec latency 16936.096 ms 2024-06-04T05:04:10.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.927+0000 7fd4cc759700 1 -- 172.21.15.70:0/1609023751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4c4102e30 msgr2=0x7fd4c410f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:10.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.927+0000 7fd4cc759700 1 --2- 172.21.15.70:0/1609023751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4c4102e30 0x7fd4c410f2f0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc00b0a0 tx=0x7fd4bc00b3b0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.928+0000 7fd4cc759700 1 -- 172.21.15.70:0/1609023751 shutdown_connections 2024-06-04T05:04:10.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.928+0000 7fd4cc759700 1 --2- 172.21.15.70:0/1609023751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4c4102e30 0x7fd4c410f2f0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.928+0000 7fd4cc759700 1 --2- 172.21.15.70:0/1609023751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4c41084b0 0x7fd4c4108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.928+0000 7fd4cc759700 1 --2- 172.21.15.70:0/1609023751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4c4102470 0x7fd4c41028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.928+0000 7fd4cc759700 1 -- 172.21.15.70:0/1609023751 >> 172.21.15.70:0/1609023751 conn(0x7fd4c40784b0 msgr2=0x7fd4c40788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:10.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.928+0000 7fd4cc759700 1 -- 172.21.15.70:0/1609023751 shutdown_connections 2024-06-04T05:04:10.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.928+0000 7fd4cc759700 1 -- 172.21.15.70:0/1609023751 wait complete. 2024-06-04T05:04:10.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.928+0000 7fd4cc759700 1 Processor -- start 2024-06-04T05:04:10.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.928+0000 7fd4cc759700 1 -- start start 2024-06-04T05:04:10.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.928+0000 7fd4cc759700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4c4102470 0x7fd4c419a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.928+0000 7fd4cc759700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4c4102e30 0x7fd4c419aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.928+0000 7fd4cc759700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4c41084b0 0x7fd4c41a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4cc759700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c4112470 con 0x7fd4c4102470 2024-06-04T05:04:10.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4cc759700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c41122f0 con 0x7fd4c41084b0 2024-06-04T05:04:10.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4cc759700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c41125f0 con 0x7fd4c4102e30 2024-06-04T05:04:10.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4ca4f5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4c4102470 0x7fd4c419a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4ca4f5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4c4102470 0x7fd4c419a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36732/0 (socket says 172.21.15.70:36732) 2024-06-04T05:04:10.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4ca4f5700 1 -- 172.21.15.70:0/1131723789 learned_addr learned my addr 172.21.15.70:0/1131723789 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:10.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4c9cf4700 1 --2- 172.21.15.70:0/1131723789 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4c4102e30 0x7fd4c419aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4cacf6700 1 --2- 172.21.15.70:0/1131723789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4c41084b0 0x7fd4c41a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4ca4f5700 1 -- 172.21.15.70:0/1131723789 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4c4102e30 msgr2=0x7fd4c419aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:10.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4ca4f5700 1 --2- 172.21.15.70:0/1131723789 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4c4102e30 0x7fd4c419aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4ca4f5700 1 -- 172.21.15.70:0/1131723789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4c41084b0 msgr2=0x7fd4c41a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:10.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4ca4f5700 1 --2- 172.21.15.70:0/1131723789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4c41084b0 0x7fd4c41a59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4ca4f5700 1 -- 172.21.15.70:0/1131723789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c41a5f30 con 0x7fd4c4102470 2024-06-04T05:04:10.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4cacf6700 1 --2- 172.21.15.70:0/1131723789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4c41084b0 0x7fd4c41a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:04:10.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4ca4f5700 1 --2- 172.21.15.70:0/1131723789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4c4102470 0x7fd4c419a970 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000d980 tx=0x7fd4c000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:10.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4bb7fe700 1 -- 172.21.15.70:0/1131723789 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0016070 con 0x7fd4c4102470 2024-06-04T05:04:10.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4cc759700 1 -- 172.21.15.70:0/1131723789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c41a6160 con 0x7fd4c4102470 2024-06-04T05:04:10.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4bb7fe700 1 -- 172.21.15.70:0/1131723789 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd4c0004830 con 0x7fd4c4102470 2024-06-04T05:04:10.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4bb7fe700 1 -- 172.21.15.70:0/1131723789 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0005440 con 0x7fd4c4102470 2024-06-04T05:04:10.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.929+0000 7fd4cc759700 1 -- 172.21.15.70:0/1131723789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c41a66c0 con 0x7fd4c4102470 2024-06-04T05:04:10.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.933+0000 7fd4bb7fe700 1 -- 172.21.15.70:0/1131723789 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fd4c0004220 con 0x7fd4c4102470 2024-06-04T05:04:10.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.933+0000 7fd4bb7fe700 1 --2- 172.21.15.70:0/1131723789 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd4b00b29f0 0x7fd4b00b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.933+0000 7fd4c9cf4700 1 --2- 172.21.15.70:0/1131723789 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd4b00b29f0 0x7fd4b00b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.933+0000 7fd4d6feeb80 1 Processor -- start 2024-06-04T05:04:10.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.933+0000 7fd4bb7fe700 1 -- 172.21.15.70:0/1131723789 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fd4c00e2380 con 0x7fd4c4102470 2024-06-04T05:04:10.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4d6feeb80 1 -- start start 2024-06-04T05:04:10.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4d6feeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990fa2b0 0x5612990fa690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4d6feeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990f48e0 0x5612990f4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4d6feeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f52a0 0x5612991010e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4d6feeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561299053d90 con 0x5612990f52a0 2024-06-04T05:04:10.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4d6feeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561299043930 con 0x5612990fa2b0 2024-06-04T05:04:10.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4d6feeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561299047970 con 0x5612990f48e0 2024-06-04T05:04:10.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4c9cf4700 1 --2- 172.21.15.70:0/1131723789 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd4b00b29f0 0x7fd4b00b4eb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4078b00 tx=0x7fd4b400a300 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:10.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4cacf6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f52a0 0x5612991010e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4cacf6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f52a0 0x5612991010e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36734/0 (socket says 172.21.15.70:36734) 2024-06-04T05:04:10.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4cacf6700 1 -- 172.21.15.70:0/403766914 learned_addr learned my addr 172.21.15.70:0/403766914 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:10.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4ca4f5700 1 --2- 172.21.15.70:0/403766914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990fa2b0 0x5612990fa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4c9cf4700 1 --2- 172.21.15.70:0/403766914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990f48e0 0x5612990f4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4cacf6700 1 -- 172.21.15.70:0/403766914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990f48e0 msgr2=0x5612990f4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:10.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4cacf6700 1 --2- 172.21.15.70:0/403766914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990f48e0 0x5612990f4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4cacf6700 1 -- 172.21.15.70:0/403766914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990fa2b0 msgr2=0x5612990fa690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:10.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4cacf6700 1 --2- 172.21.15.70:0/403766914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990fa2b0 0x5612990fa690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.934+0000 7fd4cacf6700 1 -- 172.21.15.70:0/403766914 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561298f939d0 con 0x5612990f52a0 2024-06-04T05:04:10.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.935+0000 7fd4cacf6700 1 --2- 172.21.15.70:0/403766914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f52a0 0x5612991010e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc00b830 tx=0x7fd4bc009170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccbef1d164f37d91 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:10.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.935+0000 7fd4b8ff9700 1 -- 172.21.15.70:0/403766914 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc022070 con 0x5612990f52a0 2024-06-04T05:04:10.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.935+0000 7fd4b8ff9700 1 -- 172.21.15.70:0/403766914 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd4bc0074e0 con 0x5612990f52a0 2024-06-04T05:04:10.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.935+0000 7fd4b8ff9700 1 -- 172.21.15.70:0/403766914 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc01c660 con 0x5612990f52a0 2024-06-04T05:04:10.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.935+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/403766914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f52a0 msgr2=0x5612991010e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:10.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.935+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/403766914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f52a0 0x5612991010e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc00b830 tx=0x7fd4bc009170 comp rx=0 tx=0).stop 2024-06-04T05:04:10.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.936+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/403766914 shutdown_connections 2024-06-04T05:04:10.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.936+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/403766914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f52a0 0x5612991010e0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.936+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/403766914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990fa2b0 0x5612990fa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.936+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/403766914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990f48e0 0x5612990f4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.936+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/403766914 >> 172.21.15.70:0/403766914 conn(0x561299022180 msgr2=0x5612990ef1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:10.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.936+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/403766914 shutdown_connections 2024-06-04T05:04:10.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.936+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/403766914 wait complete. 2024-06-04T05:04:10.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.936+0000 7fd4d6feeb80 1 -- start start 2024-06-04T05:04:10.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.936+0000 7fd4d6feeb80 1 Processor -- start 2024-06-04T05:04:10.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.936+0000 7fd4d6feeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f48e0 0x561299100d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.936+0000 7fd4d6feeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990f52a0 0x5612990fe050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.936+0000 7fd4d6feeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990fa2b0 0x5612990fe590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.936+0000 7fd4d6feeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561298f4a0a0 con 0x5612990f48e0 2024-06-04T05:04:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.936+0000 7fd4d6feeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561298fe58c0 con 0x5612990fa2b0 2024-06-04T05:04:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.936+0000 7fd4d6feeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561299101680 con 0x5612990f52a0 2024-06-04T05:04:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.936+0000 7fd4d6feeb80 10 client.0 _ll_register_callbacks cb 0x561298fdea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.936+0000 7fd4ca4f5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f48e0 0x561299100d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4ca4f5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f48e0 0x561299100d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36746/0 (socket says 172.21.15.70:36746) 2024-06-04T05:04:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4ca4f5700 1 -- 172.21.15.70:0/1215883268 learned_addr learned my addr 172.21.15.70:0/1215883268 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4c9cf4700 1 --2- 172.21.15.70:0/1215883268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990f52a0 0x5612990fe050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4cacf6700 1 --2- 172.21.15.70:0/1215883268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990fa2b0 0x5612990fe590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4ca4f5700 1 -- 172.21.15.70:0/1215883268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990f52a0 msgr2=0x5612990fe050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4ca4f5700 1 --2- 172.21.15.70:0/1215883268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990f52a0 0x5612990fe050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4ca4f5700 1 -- 172.21.15.70:0/1215883268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990fa2b0 msgr2=0x5612990fe590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4ca4f5700 1 --2- 172.21.15.70:0/1215883268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990fa2b0 0x5612990fe590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4ca4f5700 1 -- 172.21.15.70:0/1215883268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561298fe2790 con 0x5612990f48e0 2024-06-04T05:04:10.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4cacf6700 1 --2- 172.21.15.70:0/1215883268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990fa2b0 0x5612990fe590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:04:10.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4ca4f5700 1 --2- 172.21.15.70:0/1215883268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f48e0 0x561299100d50 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00a56b0 tx=0x7fd4c0004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cbd7992c3fdc9d0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:10.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4adffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:04:10.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4adffb700 1 -- 172.21.15.70:0/1215883268 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c00c3060 con 0x5612990f48e0 2024-06-04T05:04:10.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4adffb700 1 -- 172.21.15.70:0/1215883268 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fd4c00910e0 con 0x5612990f48e0 2024-06-04T05:04:10.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/1215883268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4bc0074e0 con 0x5612990f48e0 2024-06-04T05:04:10.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4adffb700 1 -- 172.21.15.70:0/1215883268 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd4c00a9630 con 0x5612990f48e0 2024-06-04T05:04:10.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4d6feeb80 10 client.17328 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:10.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/1215883268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56129910be10 con 0x5612990f48e0 2024-06-04T05:04:10.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.937+0000 7fd4adffb700 1 -- 172.21.15.70:0/1215883268 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd4bc0074e0 con 0x5612990f48e0 2024-06-04T05:04:10.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.938+0000 7fd4adffb700 1 -- 172.21.15.70:0/1215883268 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fd4c00a9890 con 0x5612990f48e0 2024-06-04T05:04:10.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.938+0000 7fd4d6feeb80 10 client.17328 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:10.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.938+0000 7fd4d6feeb80 10 client.17328 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:04:10.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.938+0000 7fd4d6feeb80 20 client.17328 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:10.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.938+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/1215883268 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56129910e450 0x561299110840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.938+0000 7fd4d6feeb80 4 client.17328 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:04:10.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.938+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/1215883268 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612990c0ae0 con 0x56129910e450 2024-06-04T05:04:10.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.938+0000 7fd4c9cf4700 1 --2- 172.21.15.70:0/1215883268 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56129910e450 0x561299110840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.938+0000 7fd4adffb700 10 client.17328 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:10.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.938+0000 7fd4c9cf4700 1 --2- 172.21.15.70:0/1215883268 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56129910e450 0x561299110840 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae4701113614c1de server_cookie=bd48ab3a66a33dc2 in_seq=0 out_seq=0 2024-06-04T05:04:10.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.939+0000 7fd4adffb700 1 -- 172.21.15.70:0/1215883268 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5612990c0ae0 con 0x56129910e450 2024-06-04T05:04:10.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.939+0000 7fd4adffb700 10 client.17328 handle_command_reply: tid=0 2024-06-04T05:04:10.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.939+0000 7fd4d6feeb80 1 client.17328 shutdown 2024-06-04T05:04:10.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.939+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/1215883268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f48e0 msgr2=0x561299100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:10.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.939+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/1215883268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f48e0 0x561299100d50 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00a56b0 tx=0x7fd4c0004ef0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.939+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/1215883268 shutdown_connections 2024-06-04T05:04:10.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.939+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/1215883268 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56129910e450 0x561299110840 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.939+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/1215883268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990fa2b0 0x5612990fe590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.939+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/1215883268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f48e0 0x561299100d50 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.939+0000 7fd4adffb700 0 client.17328 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:10.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.939+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/1215883268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990f52a0 0x5612990fe050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.939+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/1215883268 >> 172.21.15.70:0/1215883268 conn(0x561299022180 msgr2=0x5612990e9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:10.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.939+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/1215883268 shutdown_connections 2024-06-04T05:04:10.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.939+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/1215883268 wait complete. 2024-06-04T05:04:10.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.940+0000 7fd4d6feeb80 20 client.17328 trim_cache size 0 max 16384 2024-06-04T05:04:10.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.959+0000 7fd4d6feeb80 1 Processor -- start 2024-06-04T05:04:10.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.959+0000 7fd4d6feeb80 1 -- start start 2024-06-04T05:04:10.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4d6feeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f48e0 0x561299100890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4d6feeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990f52a0 0x5612991160c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4d6feeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990fa2b0 0x561299116600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4d6feeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c41adcf0 con 0x5612990f48e0 2024-06-04T05:04:10.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4d6feeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c41af830 con 0x5612990fa2b0 2024-06-04T05:04:10.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4d6feeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c41ae3c0 con 0x5612990f52a0 2024-06-04T05:04:10.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4ca4f5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f48e0 0x561299100890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4ca4f5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f48e0 0x561299100890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36750/0 (socket says 172.21.15.70:36750) 2024-06-04T05:04:10.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4ca4f5700 1 -- 172.21.15.70:0/4059206712 learned_addr learned my addr 172.21.15.70:0/4059206712 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:10.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4c9cf4700 1 --2- 172.21.15.70:0/4059206712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990f52a0 0x5612991160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4cacf6700 1 --2- 172.21.15.70:0/4059206712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990fa2b0 0x561299116600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4ca4f5700 1 -- 172.21.15.70:0/4059206712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990f52a0 msgr2=0x5612991160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:10.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4ca4f5700 1 --2- 172.21.15.70:0/4059206712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990f52a0 0x5612991160c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4ca4f5700 1 -- 172.21.15.70:0/4059206712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990fa2b0 msgr2=0x561299116600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:10.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4ca4f5700 1 --2- 172.21.15.70:0/4059206712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990fa2b0 0x561299116600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4ca4f5700 1 -- 172.21.15.70:0/4059206712 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612990ef730 con 0x5612990f48e0 2024-06-04T05:04:10.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4cacf6700 1 --2- 172.21.15.70:0/4059206712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990fa2b0 0x561299116600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:04:10.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4ca4f5700 1 --2- 172.21.15.70:0/4059206712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f48e0 0x561299100890 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fd4c0004c70 tx=0x7fd4c0004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf69fdf091988c33 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:10.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.960+0000 7fd4affff700 1 -- 172.21.15.70:0/4059206712 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c00c3070 con 0x5612990f48e0 2024-06-04T05:04:10.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.961+0000 7fd4affff700 1 -- 172.21.15.70:0/4059206712 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd4c00b2490 con 0x5612990f48e0 2024-06-04T05:04:10.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.961+0000 7fd4affff700 1 -- 172.21.15.70:0/4059206712 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c00b2000 con 0x5612990f48e0 2024-06-04T05:04:10.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.961+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/4059206712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f48e0 msgr2=0x561299100890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:10.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.961+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/4059206712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f48e0 0x561299100890 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fd4c0004c70 tx=0x7fd4c0004f90 comp rx=0 tx=0).stop 2024-06-04T05:04:10.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.961+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/4059206712 shutdown_connections 2024-06-04T05:04:10.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.961+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/4059206712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990fa2b0 0x561299116600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.961+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/4059206712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f48e0 0x561299100890 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.961+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/4059206712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990f52a0 0x5612991160c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.961+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/4059206712 >> 172.21.15.70:0/4059206712 conn(0x561299022180 msgr2=0x5612990f4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:10.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.961+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/4059206712 shutdown_connections 2024-06-04T05:04:10.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.961+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/4059206712 wait complete. 2024-06-04T05:04:10.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.962+0000 7fd4d6feeb80 1 -- start start 2024-06-04T05:04:10.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.962+0000 7fd4d6feeb80 1 Processor -- start 2024-06-04T05:04:10.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.962+0000 7fd4d6feeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990f48e0 0x5612991070e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.962+0000 7fd4d6feeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f52a0 0x5612991038e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.962+0000 7fd4d6feeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990fa2b0 0x561299103ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.962+0000 7fd4d6feeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c41af160 con 0x5612990f52a0 2024-06-04T05:04:10.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.962+0000 7fd4d6feeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c41aea90 con 0x5612990f48e0 2024-06-04T05:04:10.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.962+0000 7fd4d6feeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c41abcb0 con 0x5612990fa2b0 2024-06-04T05:04:10.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.962+0000 7fd4d6feeb80 10 client.0 _ll_register_callbacks cb 0x561298fdea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:10.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.962+0000 7fd4c9cf4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f52a0 0x5612991038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.962+0000 7fd4c9cf4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f52a0 0x5612991038e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36756/0 (socket says 172.21.15.70:36756) 2024-06-04T05:04:10.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.963+0000 7fd4c9cf4700 1 -- 172.21.15.70:0/2074019974 learned_addr learned my addr 172.21.15.70:0/2074019974 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:10.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.963+0000 7fd4cacf6700 1 --2- 172.21.15.70:0/2074019974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990fa2b0 0x561299103ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.963+0000 7fd4ca4f5700 1 --2- 172.21.15.70:0/2074019974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990f48e0 0x5612991070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.963+0000 7fd4c9cf4700 1 -- 172.21.15.70:0/2074019974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990fa2b0 msgr2=0x561299103ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:10.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.963+0000 7fd4c9cf4700 1 --2- 172.21.15.70:0/2074019974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990fa2b0 0x561299103ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.963+0000 7fd4c9cf4700 1 -- 172.21.15.70:0/2074019974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990f48e0 msgr2=0x5612991070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:10.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.963+0000 7fd4c9cf4700 1 --2- 172.21.15.70:0/2074019974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990f48e0 0x5612991070e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.963+0000 7fd4c9cf4700 1 -- 172.21.15.70:0/2074019974 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561299104710 con 0x5612990f52a0 2024-06-04T05:04:10.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.963+0000 7fd4c9cf4700 1 --2- 172.21.15.70:0/2074019974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f52a0 0x5612991038e0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fd4b400a9d0 tx=0x7fd4b40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=345a97387d9258d5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:10.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.963+0000 7fd4adffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:04:10.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.963+0000 7fd4adffb700 1 -- 172.21.15.70:0/2074019974 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4016070 con 0x5612990f52a0 2024-06-04T05:04:10.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.963+0000 7fd4adffb700 1 -- 172.21.15.70:0/2074019974 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fd4b401c830 con 0x5612990f52a0 2024-06-04T05:04:10.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.963+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/2074019974 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4c00b2490 con 0x5612990f52a0 2024-06-04T05:04:10.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.963+0000 7fd4adffb700 1 -- 172.21.15.70:0/2074019974 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd4b403f050 con 0x5612990f52a0 2024-06-04T05:04:10.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.963+0000 7fd4d6feeb80 10 client.17334 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:10.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.963+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/2074019974 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56129917e550 con 0x5612990f52a0 2024-06-04T05:04:10.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.964+0000 7fd4adffb700 1 -- 172.21.15.70:0/2074019974 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd4c00b2490 con 0x5612990f52a0 2024-06-04T05:04:10.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.964+0000 7fd4adffb700 1 -- 172.21.15.70:0/2074019974 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fd4b40298b0 con 0x5612990f52a0 2024-06-04T05:04:10.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.964+0000 7fd4d6feeb80 10 client.17334 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:10.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.964+0000 7fd4d6feeb80 10 client.17334 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:04:10.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.964+0000 7fd4d6feeb80 20 client.17334 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:10.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.964+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/2074019974 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56129917ec40 0x561299181060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:10.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.964+0000 7fd4d6feeb80 4 client.17334 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b977e96e", "release": true, "await": true}] 2024-06-04T05:04:10.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.964+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/2074019974 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b977e96e", "release": true, "await": true}) -- 0x561299055ce0 con 0x56129917ec40 2024-06-04T05:04:10.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.964+0000 7fd4ca4f5700 1 --2- 172.21.15.70:0/2074019974 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56129917ec40 0x561299181060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:10.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.965+0000 7fd4ca4f5700 1 --2- 172.21.15.70:0/2074019974 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56129917ec40 0x561299181060 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f23e0b2245a5aa17 server_cookie=6e5ac3e6587d4d92 in_seq=0 out_seq=0 2024-06-04T05:04:10.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.965+0000 7fd4adffb700 10 client.17334 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:10.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.970+0000 7fd4adffb700 1 -- 172.21.15.70:0/2074019974 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x561299055ce0 con 0x56129917ec40 2024-06-04T05:04:10.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.970+0000 7fd4adffb700 10 client.17334 handle_command_reply: tid=0 2024-06-04T05:04:10.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.970+0000 7fd4d6feeb80 1 client.17334 shutdown 2024-06-04T05:04:10.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.970+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/2074019974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f52a0 msgr2=0x5612991038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:10.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.970+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/2074019974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f52a0 0x5612991038e0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fd4b400a9d0 tx=0x7fd4b40047c0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.970+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/2074019974 shutdown_connections 2024-06-04T05:04:10.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.970+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/2074019974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5612990fa2b0 0x561299103ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.970+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/2074019974 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56129917ec40 0x561299181060 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.970+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/2074019974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612990f48e0 0x5612991070e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.970+0000 7fd4d6feeb80 1 --2- 172.21.15.70:0/2074019974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612990f52a0 0x5612991038e0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.970+0000 7fd4adffb700 0 client.17334 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:10.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.970+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/2074019974 >> 172.21.15.70:0/2074019974 conn(0x561299022180 msgr2=0x5612990e9410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:10.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.970+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/2074019974 shutdown_connections 2024-06-04T05:04:10.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.970+0000 7fd4d6feeb80 1 -- 172.21.15.70:0/2074019974 wait complete. 2024-06-04T05:04:10.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.970+0000 7fd4d6feeb80 20 client.17334 trim_cache size 0 max 16384 2024-06-04T05:04:10.974 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:04:10.974 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 32, 2024-06-04T05:04:10.974 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:04:10.974 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 20, 2024-06-04T05:04:10.974 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:04:10.974 INFO:teuthology.orchestra.run.smithi070.stdout: "b977e96e": { 2024-06-04T05:04:10.974 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 20, 2024-06-04T05:04:10.974 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:04:10.974 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:04:10.974 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:04:10.975 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:04:10.975 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:04:10.975 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:04:10.975 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 138.9, 2024-06-04T05:04:10.975 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:04:10.975 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:04:10.975 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:04:10.975 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:04:10.975 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:04:10.975 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:04:10.975 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:04:10.975 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:04:10.975 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:04:10.975 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:04:10.975 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:04:10.975 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:04:10.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.971+0000 7fd4cc759700 1 -- 172.21.15.70:0/1131723789 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd4b00b29f0 msgr2=0x7fd4b00b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:10.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.971+0000 7fd4cc759700 1 --2- 172.21.15.70:0/1131723789 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd4b00b29f0 0x7fd4b00b4eb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4078b00 tx=0x7fd4b400a300 comp rx=0 tx=0).stop 2024-06-04T05:04:10.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.971+0000 7fd4cc759700 1 -- 172.21.15.70:0/1131723789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4c4102470 msgr2=0x7fd4c419a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:10.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.971+0000 7fd4cc759700 1 --2- 172.21.15.70:0/1131723789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4c4102470 0x7fd4c419a970 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000d980 tx=0x7fd4c000dc90 comp rx=0 tx=0).stop 2024-06-04T05:04:10.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.971+0000 7fd4cc759700 1 -- 172.21.15.70:0/1131723789 shutdown_connections 2024-06-04T05:04:10.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.971+0000 7fd4cc759700 1 --2- 172.21.15.70:0/1131723789 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd4b00b29f0 0x7fd4b00b4eb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.971+0000 7fd4cc759700 1 --2- 172.21.15.70:0/1131723789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4c41084b0 0x7fd4c41a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.971+0000 7fd4cc759700 1 --2- 172.21.15.70:0/1131723789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4c4102470 0x7fd4c419a970 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.971+0000 7fd4cc759700 1 --2- 172.21.15.70:0/1131723789 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4c4102e30 0x7fd4c419aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:10.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.971+0000 7fd4cc759700 1 -- 172.21.15.70:0/1131723789 >> 172.21.15.70:0/1131723789 conn(0x7fd4c40784b0 msgr2=0x7fd4c41005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:10.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.971+0000 7fd4cc759700 1 -- 172.21.15.70:0/1131723789 shutdown_connections 2024-06-04T05:04:10.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:10.971+0000 7fd4cc759700 1 -- 172.21.15.70:0/1131723789 wait complete. 2024-06-04T05:04:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.046+0000 7f56f1596640 1 -- 172.21.15.70:0/4197656396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56ec1033f0 msgr2=0x7f56ec10f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.046+0000 7f56f1596640 1 --2- 172.21.15.70:0/4197656396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56ec1033f0 0x7f56ec10f900 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f56e4009cd0 tx=0x7f56e402a8e0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.046+0000 7f56f1596640 1 -- 172.21.15.70:0/4197656396 shutdown_connections 2024-06-04T05:04:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.046+0000 7f56f1596640 1 --2- 172.21.15.70:0/4197656396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56ec1033f0 0x7f56ec10f900 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.046+0000 7f56f1596640 1 --2- 172.21.15.70:0/4197656396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f56ec102a30 0x7f56ec102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.046+0000 7f56f1596640 1 --2- 172.21.15.70:0/4197656396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56ec108a40 0x7f56ec108e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.046+0000 7f56f1596640 1 -- 172.21.15.70:0/4197656396 >> 172.21.15.70:0/4197656396 conn(0x7f56ec0fe500 msgr2=0x7f56ec100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:11.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.046+0000 7f56f1596640 1 -- 172.21.15.70:0/4197656396 shutdown_connections 2024-06-04T05:04:11.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.046+0000 7f56f1596640 1 -- 172.21.15.70:0/4197656396 wait complete. 2024-06-04T05:04:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56f1596640 1 Processor -- start 2024-06-04T05:04:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56f1596640 1 -- start start 2024-06-04T05:04:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56f1596640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56ec102a30 0x7f56ec199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56f1596640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56ec1033f0 0x7f56ec199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56f1596640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f56ec108a40 0x7f56ec19d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56f1596640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec112b80 con 0x7f56ec1033f0 2024-06-04T05:04:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56f1596640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec112a00 con 0x7f56ec102a30 2024-06-04T05:04:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56f1596640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec112d00 con 0x7f56ec108a40 2024-06-04T05:04:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56e3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56ec1033f0 0x7f56ec199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56e3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56ec1033f0 0x7f56ec199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36772/0 (socket says 172.21.15.70:36772) 2024-06-04T05:04:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56e3fff640 1 -- 172.21.15.70:0/1017771603 learned_addr learned my addr 172.21.15.70:0/1017771603 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:11.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56eb7fe640 1 --2- 172.21.15.70:0/1017771603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f56ec108a40 0x7f56ec19d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56eaffd640 1 --2- 172.21.15.70:0/1017771603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56ec102a30 0x7f56ec199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56e3fff640 1 -- 172.21.15.70:0/1017771603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f56ec108a40 msgr2=0x7f56ec19d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:11.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56e3fff640 1 --2- 172.21.15.70:0/1017771603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f56ec108a40 0x7f56ec19d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56e3fff640 1 -- 172.21.15.70:0/1017771603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56ec102a30 msgr2=0x7f56ec199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:11.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56e3fff640 1 --2- 172.21.15.70:0/1017771603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56ec102a30 0x7f56ec199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56e3fff640 1 -- 172.21.15.70:0/1017771603 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56ec19e0c0 con 0x7f56ec1033f0 2024-06-04T05:04:11.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.047+0000 7f56e3fff640 1 --2- 172.21.15.70:0/1017771603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56ec1033f0 0x7f56ec199660 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f56d400d950 tx=0x7f56d400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:11.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.048+0000 7f56e8ff9640 1 -- 172.21.15.70:0/1017771603 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d4017070 con 0x7f56ec1033f0 2024-06-04T05:04:11.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.048+0000 7f56f1596640 1 -- 172.21.15.70:0/1017771603 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56ec19e3b0 con 0x7f56ec1033f0 2024-06-04T05:04:11.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.048+0000 7f56e8ff9640 1 -- 172.21.15.70:0/1017771603 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f56d4009c30 con 0x7f56ec1033f0 2024-06-04T05:04:11.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.048+0000 7f56e8ff9640 1 -- 172.21.15.70:0/1017771603 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d400f620 con 0x7f56ec1033f0 2024-06-04T05:04:11.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.048+0000 7f56f1596640 1 -- 172.21.15.70:0/1017771603 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56ec1a5c40 con 0x7f56ec1033f0 2024-06-04T05:04:11.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.050+0000 7f56f4388740 1 Processor -- start 2024-06-04T05:04:11.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.050+0000 7f56e8ff9640 1 -- 172.21.15.70:0/1017771603 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f56d4010460 con 0x7f56ec1033f0 2024-06-04T05:04:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.050+0000 7f56f4388740 1 -- start start 2024-06-04T05:04:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56f4388740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bd9740 0x558582bd9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56f4388740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582bda060 0x558582be88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56f4388740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582be8e10 0x558582beb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56f4388740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558582b4bc30 con 0x558582bd9740 2024-06-04T05:04:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56f4388740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558582b3b7d0 con 0x558582bda060 2024-06-04T05:04:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56f4388740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558582b3f810 con 0x558582be8e10 2024-06-04T05:04:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56eb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bd9740 0x558582bd9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56eb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bd9740 0x558582bd9b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36786/0 (socket says 172.21.15.70:36786) 2024-06-04T05:04:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56eb7fe640 1 -- 172.21.15.70:0/2678105826 learned_addr learned my addr 172.21.15.70:0/2678105826 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56e3fff640 1 --2- 172.21.15.70:0/2678105826 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582be8e10 0x558582beb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56eaffd640 1 --2- 172.21.15.70:0/2678105826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582bda060 0x558582be88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56eb7fe640 1 -- 172.21.15.70:0/2678105826 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582be8e10 msgr2=0x558582beb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56eb7fe640 1 --2- 172.21.15.70:0/2678105826 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582be8e10 0x558582beb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56eb7fe640 1 -- 172.21.15.70:0/2678105826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582bda060 msgr2=0x558582be88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56eb7fe640 1 --2- 172.21.15.70:0/2678105826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582bda060 0x558582be88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56eb7fe640 1 -- 172.21.15.70:0/2678105826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558582850210 con 0x558582bd9740 2024-06-04T05:04:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56eaffd640 1 --2- 172.21.15.70:0/2678105826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582bda060 0x558582be88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:04:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.051+0000 7f56eb7fe640 1 --2- 172.21.15.70:0/2678105826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bd9740 0x558582bd9b20 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f56e4004ab0 tx=0x7f56e4009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1bdf2ebf4396dab server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.052+0000 7f56e8ff9640 1 --2- 172.21.15.70:0/1017771603 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56d80b2a10 0x7f56d80b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.052+0000 7f56e17fa640 1 -- 172.21.15.70:0/2678105826 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e4048070 con 0x558582bd9740 2024-06-04T05:04:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.052+0000 7f56eaffd640 1 --2- 172.21.15.70:0/1017771603 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56d80b2a10 0x7f56d80b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.052+0000 7f56e17fa640 1 -- 172.21.15.70:0/2678105826 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f56e4036070 con 0x558582bd9740 2024-06-04T05:04:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.052+0000 7f56e17fa640 1 -- 172.21.15.70:0/2678105826 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e4004460 con 0x558582bd9740 2024-06-04T05:04:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.052+0000 7f56e8ff9640 1 -- 172.21.15.70:0/1017771603 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f56d40e2260 con 0x7f56ec1033f0 2024-06-04T05:04:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 1 -- 172.21.15.70:0/2678105826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bd9740 msgr2=0x558582bd9b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56eaffd640 1 --2- 172.21.15.70:0/1017771603 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56d80b2a10 0x7f56d80b4ed0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f56dc005e00 tx=0x7f56dc005d50 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 1 --2- 172.21.15.70:0/2678105826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bd9740 0x558582bd9b20 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f56e4004ab0 tx=0x7f56e4009a20 comp rx=0 tx=0).stop 2024-06-04T05:04:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 1 -- 172.21.15.70:0/2678105826 shutdown_connections 2024-06-04T05:04:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 1 --2- 172.21.15.70:0/2678105826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582bda060 0x558582be88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 1 --2- 172.21.15.70:0/2678105826 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582be8e10 0x558582beb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 1 --2- 172.21.15.70:0/2678105826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bd9740 0x558582bd9b20 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 1 -- 172.21.15.70:0/2678105826 >> 172.21.15.70:0/2678105826 conn(0x558582bd5d30 msgr2=0x558582bd6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 1 -- 172.21.15.70:0/2678105826 shutdown_connections 2024-06-04T05:04:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 1 -- 172.21.15.70:0/2678105826 wait complete. 2024-06-04T05:04:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 1 -- start start 2024-06-04T05:04:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 1 Processor -- start 2024-06-04T05:04:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582bd9740 0x558582be5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bda060 0x558582be6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582be8e10 0x558582be6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558582821790 con 0x558582bda060 2024-06-04T05:04:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558582af7890 con 0x558582be8e10 2024-06-04T05:04:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558582beb9e0 con 0x558582bd9740 2024-06-04T05:04:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.053+0000 7f56f4388740 10 client.0 _ll_register_callbacks cb 0x55858284fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56e3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bda060 0x558582be6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56e3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bda060 0x558582be6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36788/0 (socket says 172.21.15.70:36788) 2024-06-04T05:04:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56e3fff640 1 -- 172.21.15.70:0/34177103 learned_addr learned my addr 172.21.15.70:0/34177103 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56eaffd640 1 --2- 172.21.15.70:0/34177103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582bd9740 0x558582be5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56eb7fe640 1 --2- 172.21.15.70:0/34177103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582be8e10 0x558582be6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56e3fff640 1 -- 172.21.15.70:0/34177103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582bd9740 msgr2=0x558582be5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56e3fff640 1 --2- 172.21.15.70:0/34177103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582bd9740 0x558582be5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56e3fff640 1 -- 172.21.15.70:0/34177103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582be8e10 msgr2=0x558582be6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56e3fff640 1 --2- 172.21.15.70:0/34177103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582be8e10 0x558582be6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56e3fff640 1 -- 172.21.15.70:0/34177103 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55858291c620 con 0x558582bda060 2024-06-04T05:04:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56eb7fe640 1 --2- 172.21.15.70:0/34177103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582be8e10 0x558582be6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:04:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56e3fff640 1 --2- 172.21.15.70:0/34177103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bda060 0x558582be6340 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f56d40a9210 tx=0x7f56d40dfe80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac353a76045005f1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56d27fc640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:04:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56d27fc640 1 -- 172.21.15.70:0/34177103 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d4020070 con 0x558582bda060 2024-06-04T05:04:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56d27fc640 1 -- 172.21.15.70:0/34177103 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f56d40b1ee0 con 0x558582bda060 2024-06-04T05:04:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56f4388740 1 -- 172.21.15.70:0/34177103 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56e4036070 con 0x558582bda060 2024-06-04T05:04:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56d27fc640 1 -- 172.21.15.70:0/34177103 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f56d40a8e60 con 0x558582bda060 2024-06-04T05:04:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56f4388740 10 client.17346 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56f4388740 1 -- 172.21.15.70:0/34177103 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558582bf7820 con 0x558582bda060 2024-06-04T05:04:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.054+0000 7f56d27fc640 1 -- 172.21.15.70:0/34177103 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f56e4036070 con 0x558582bda060 2024-06-04T05:04:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.055+0000 7f56d27fc640 1 -- 172.21.15.70:0/34177103 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f56d40a4e80 con 0x558582bda060 2024-06-04T05:04:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.055+0000 7f56f4388740 10 client.17346 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.055+0000 7f56f4388740 10 client.17346 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:04:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.055+0000 7f56f4388740 20 client.17346 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.055+0000 7f56f4388740 1 --2- 172.21.15.70:0/34177103 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558582bf8df0 0x558582bfb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.055+0000 7f56f4388740 4 client.17346 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:04:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.055+0000 7f56f4388740 1 -- 172.21.15.70:0/34177103 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558582b4db80 con 0x558582bf8df0 2024-06-04T05:04:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.055+0000 7f56eaffd640 1 --2- 172.21.15.70:0/34177103 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558582bf8df0 0x558582bfb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.055+0000 7f56d27fc640 10 client.17346 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.055+0000 7f56eaffd640 1 --2- 172.21.15.70:0/34177103 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558582bf8df0 0x558582bfb250 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=757ff62471f394d2 server_cookie=45820a3722d266cb in_seq=0 out_seq=0 2024-06-04T05:04:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.056+0000 7f56d27fc640 1 -- 172.21.15.70:0/34177103 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558582b4db80 con 0x558582bf8df0 2024-06-04T05:04:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.056+0000 7f56d27fc640 10 client.17346 handle_command_reply: tid=0 2024-06-04T05:04:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.056+0000 7f56f4388740 1 client.17346 shutdown 2024-06-04T05:04:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.056+0000 7f56f4388740 1 -- 172.21.15.70:0/34177103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bda060 msgr2=0x558582be6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.056+0000 7f56f4388740 1 --2- 172.21.15.70:0/34177103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bda060 0x558582be6340 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f56d40a9210 tx=0x7f56d40dfe80 comp rx=0 tx=0).stop 2024-06-04T05:04:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.056+0000 7f56f4388740 1 -- 172.21.15.70:0/34177103 shutdown_connections 2024-06-04T05:04:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.056+0000 7f56f4388740 1 --2- 172.21.15.70:0/34177103 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558582bf8df0 0x558582bfb250 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.056+0000 7f56f4388740 1 --2- 172.21.15.70:0/34177103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582be8e10 0x558582be6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.056+0000 7f56f4388740 1 --2- 172.21.15.70:0/34177103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bda060 0x558582be6340 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.056+0000 7f56d27fc640 0 client.17346 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.056+0000 7f56f4388740 1 --2- 172.21.15.70:0/34177103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582bd9740 0x558582be5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.056+0000 7f56f4388740 1 -- 172.21.15.70:0/34177103 >> 172.21.15.70:0/34177103 conn(0x558582bd5d30 msgr2=0x558582bdbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.056+0000 7f56f4388740 1 -- 172.21.15.70:0/34177103 shutdown_connections 2024-06-04T05:04:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.056+0000 7f56f4388740 1 -- 172.21.15.70:0/34177103 wait complete. 2024-06-04T05:04:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.056+0000 7f56f4388740 20 client.17346 trim_cache size 0 max 16384 2024-06-04T05:04:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.074+0000 7f56f4388740 1 Processor -- start 2024-06-04T05:04:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.074+0000 7f56f4388740 1 -- start start 2024-06-04T05:04:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.074+0000 7f56f4388740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582bd9740 0x558582bdb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.074+0000 7f56f4388740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582bda060 0x558582c00ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.074+0000 7f56f4388740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582be8e10 0x558582c01010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.074+0000 7f56f4388740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec1ac310 con 0x558582be8e10 2024-06-04T05:04:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.074+0000 7f56f4388740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec1ade50 con 0x558582bd9740 2024-06-04T05:04:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.075+0000 7f56f4388740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec1ac9e0 con 0x558582bda060 2024-06-04T05:04:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.075+0000 7f56e3fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582bda060 0x558582c00ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.075+0000 7f56eb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582be8e10 0x558582c01010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.075+0000 7f56e3fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582bda060 0x558582c00ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:54076/0 (socket says 172.21.15.70:54076) 2024-06-04T05:04:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.075+0000 7f56eb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582be8e10 0x558582c01010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36800/0 (socket says 172.21.15.70:36800) 2024-06-04T05:04:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.075+0000 7f56e3fff640 1 -- 172.21.15.70:0/265342958 learned_addr learned my addr 172.21.15.70:0/265342958 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:11.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.075+0000 7f56eaffd640 1 --2- 172.21.15.70:0/265342958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582bd9740 0x558582bdb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.075+0000 7f56eb7fe640 1 -- 172.21.15.70:0/265342958 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582bda060 msgr2=0x558582c00ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:11.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.075+0000 7f56eb7fe640 1 --2- 172.21.15.70:0/265342958 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582bda060 0x558582c00ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.075+0000 7f56eb7fe640 1 -- 172.21.15.70:0/265342958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582bd9740 msgr2=0x558582bdb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:11.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.075+0000 7f56eb7fe640 1 --2- 172.21.15.70:0/265342958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582bd9740 0x558582bdb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.075+0000 7f56eb7fe640 1 -- 172.21.15.70:0/265342958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558582bd5b60 con 0x558582be8e10 2024-06-04T05:04:11.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.075+0000 7f56eb7fe640 1 --2- 172.21.15.70:0/265342958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582be8e10 0x558582c01010 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f56e402ae00 tx=0x7f56e4000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a44cf4975f6aae5e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:11.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.075+0000 7f56e0ff9640 1 -- 172.21.15.70:0/265342958 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e4048040 con 0x558582be8e10 2024-06-04T05:04:11.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.075+0000 7f56e0ff9640 1 -- 172.21.15.70:0/265342958 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f56e4036040 con 0x558582be8e10 2024-06-04T05:04:11.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.075+0000 7f56e0ff9640 1 -- 172.21.15.70:0/265342958 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e4004280 con 0x558582be8e10 2024-06-04T05:04:11.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.076+0000 7f56f4388740 1 -- 172.21.15.70:0/265342958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582be8e10 msgr2=0x558582c01010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:11.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.076+0000 7f56f4388740 1 --2- 172.21.15.70:0/265342958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582be8e10 0x558582c01010 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f56e402ae00 tx=0x7f56e4000f40 comp rx=0 tx=0).stop 2024-06-04T05:04:11.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.076+0000 7f56f4388740 1 -- 172.21.15.70:0/265342958 shutdown_connections 2024-06-04T05:04:11.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.076+0000 7f56f4388740 1 --2- 172.21.15.70:0/265342958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582be8e10 0x558582c01010 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.076+0000 7f56f4388740 1 --2- 172.21.15.70:0/265342958 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582bda060 0x558582c00ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.076+0000 7f56f4388740 1 --2- 172.21.15.70:0/265342958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582bd9740 0x558582bdb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.076+0000 7f56f4388740 1 -- 172.21.15.70:0/265342958 >> 172.21.15.70:0/265342958 conn(0x558582bd5d30 msgr2=0x558582bdb250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.076+0000 7f56f4388740 1 -- 172.21.15.70:0/265342958 shutdown_connections 2024-06-04T05:04:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.076+0000 7f56f4388740 1 -- 172.21.15.70:0/265342958 wait complete. 2024-06-04T05:04:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56f4388740 1 -- start start 2024-06-04T05:04:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56f4388740 1 Processor -- start 2024-06-04T05:04:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56f4388740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bd9740 0x558582bed200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56f4388740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582bda060 0x558582bed740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56f4388740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582be8e10 0x558582c629a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56f4388740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec1ad780 con 0x558582bd9740 2024-06-04T05:04:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56f4388740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec1ad0b0 con 0x558582be8e10 2024-06-04T05:04:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56f4388740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec1aa310 con 0x558582bda060 2024-06-04T05:04:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56f4388740 10 client.0 _ll_register_callbacks cb 0x7f56d400df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56eaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bd9740 0x558582bed200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56eaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bd9740 0x558582bed200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36810/0 (socket says 172.21.15.70:36810) 2024-06-04T05:04:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56eaffd640 1 -- 172.21.15.70:0/3636331194 learned_addr learned my addr 172.21.15.70:0/3636331194 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56e3fff640 1 --2- 172.21.15.70:0/3636331194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582bda060 0x558582bed740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56eb7fe640 1 --2- 172.21.15.70:0/3636331194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582be8e10 0x558582c629a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56eaffd640 1 -- 172.21.15.70:0/3636331194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582bda060 msgr2=0x558582bed740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56eaffd640 1 --2- 172.21.15.70:0/3636331194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582bda060 0x558582bed740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56eaffd640 1 -- 172.21.15.70:0/3636331194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582be8e10 msgr2=0x558582c629a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56eaffd640 1 --2- 172.21.15.70:0/3636331194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582be8e10 0x558582c629a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56eaffd640 1 -- 172.21.15.70:0/3636331194 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558582c62f30 con 0x558582bd9740 2024-06-04T05:04:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.077+0000 7f56eb7fe640 1 --2- 172.21.15.70:0/3636331194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582be8e10 0x558582c629a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:04:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.078+0000 7f56eaffd640 1 --2- 172.21.15.70:0/3636331194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bd9740 0x558582bed200 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x558582bdc340 tx=0x7f56dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf84cae0f0372d64 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.078+0000 7f56d27fc640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:04:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.078+0000 7f56d27fc640 1 -- 172.21.15.70:0/3636331194 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56dc0035f0 con 0x558582bd9740 2024-06-04T05:04:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.078+0000 7f56d27fc640 1 -- 172.21.15.70:0/3636331194 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f56dc003790 con 0x558582bd9740 2024-06-04T05:04:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.078+0000 7f56f4388740 1 -- 172.21.15.70:0/3636331194 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56e4036040 con 0x558582bd9740 2024-06-04T05:04:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.078+0000 7f56d27fc640 1 -- 172.21.15.70:0/3636331194 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f56dc037050 con 0x558582bd9740 2024-06-04T05:04:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.078+0000 7f56f4388740 10 client.17352 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.078+0000 7f56f4388740 1 -- 172.21.15.70:0/3636331194 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558582c64280 con 0x558582bd9740 2024-06-04T05:04:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.078+0000 7f56d27fc640 1 -- 172.21.15.70:0/3636331194 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f56e4036040 con 0x558582bd9740 2024-06-04T05:04:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.078+0000 7f56d27fc640 1 -- 172.21.15.70:0/3636331194 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f56dc0288a0 con 0x558582bd9740 2024-06-04T05:04:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.078+0000 7f56f4388740 10 client.17352 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.078+0000 7f56f4388740 10 client.17352 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:04:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.078+0000 7f56f4388740 20 client.17352 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.078+0000 7f56f4388740 1 --2- 172.21.15.70:0/3636331194 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558582c64a30 0x558582c66e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.078+0000 7f56f4388740 4 client.17352 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:04:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.078+0000 7f56f4388740 1 -- 172.21.15.70:0/3636331194 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558582b47360 con 0x558582c64a30 2024-06-04T05:04:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.079+0000 7f56e3fff640 1 --2- 172.21.15.70:0/3636331194 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558582c64a30 0x558582c66e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.079+0000 7f56e3fff640 1 --2- 172.21.15.70:0/3636331194 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558582c64a30 0x558582c66e50 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcd9ef3d1c6f75c7 server_cookie=2d5d3bc1d2883de9 in_seq=0 out_seq=0 2024-06-04T05:04:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.079+0000 7f56d27fc640 10 client.17352 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.097+0000 7f56d27fc640 1 -- 172.21.15.70:0/3636331194 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x558582b47360 con 0x558582c64a30 2024-06-04T05:04:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.097+0000 7f56d27fc640 10 client.17352 handle_command_reply: tid=0 2024-06-04T05:04:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.097+0000 7f56f4388740 1 client.17352 shutdown 2024-06-04T05:04:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.097+0000 7f56f4388740 1 -- 172.21.15.70:0/3636331194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bd9740 msgr2=0x558582bed200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.097+0000 7f56f4388740 1 --2- 172.21.15.70:0/3636331194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bd9740 0x558582bed200 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x558582bdc340 tx=0x7f56dc0046c0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.097+0000 7f56f4388740 1 -- 172.21.15.70:0/3636331194 shutdown_connections 2024-06-04T05:04:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.097+0000 7f56f4388740 1 --2- 172.21.15.70:0/3636331194 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558582c64a30 0x558582c66e50 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.097+0000 7f56f4388740 1 --2- 172.21.15.70:0/3636331194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558582be8e10 0x558582c629a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.097+0000 7f56d27fc640 0 client.17352 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.097+0000 7f56f4388740 1 --2- 172.21.15.70:0/3636331194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558582bda060 0x558582bed740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.097+0000 7f56f4388740 1 --2- 172.21.15.70:0/3636331194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558582bd9740 0x558582bed200 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.097+0000 7f56f4388740 1 -- 172.21.15.70:0/3636331194 >> 172.21.15.70:0/3636331194 conn(0x558582bd5d30 msgr2=0x558582bdbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.097+0000 7f56f4388740 1 -- 172.21.15.70:0/3636331194 shutdown_connections 2024-06-04T05:04:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.097+0000 7f56f4388740 1 -- 172.21.15.70:0/3636331194 wait complete. 2024-06-04T05:04:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.097+0000 7f56f4388740 20 client.17352 trim_cache size 0 max 16384 2024-06-04T05:04:11.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:04:11.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:04:11.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:04:11.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.098+0000 7f56f1596640 1 -- 172.21.15.70:0/1017771603 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56d80b2a10 msgr2=0x7f56d80b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:11.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.098+0000 7f56f1596640 1 --2- 172.21.15.70:0/1017771603 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56d80b2a10 0x7f56d80b4ed0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f56dc005e00 tx=0x7f56dc005d50 comp rx=0 tx=0).stop 2024-06-04T05:04:11.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.098+0000 7f56f1596640 1 -- 172.21.15.70:0/1017771603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56ec1033f0 msgr2=0x7f56ec199660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:11.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.098+0000 7f56f1596640 1 --2- 172.21.15.70:0/1017771603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56ec1033f0 0x7f56ec199660 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f56d400d950 tx=0x7f56d400de10 comp rx=0 tx=0).stop 2024-06-04T05:04:11.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.098+0000 7f56f1596640 1 -- 172.21.15.70:0/1017771603 shutdown_connections 2024-06-04T05:04:11.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.098+0000 7f56f1596640 1 --2- 172.21.15.70:0/1017771603 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56d80b2a10 0x7f56d80b4ed0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.098+0000 7f56f1596640 1 --2- 172.21.15.70:0/1017771603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f56ec108a40 0x7f56ec19d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.098+0000 7f56f1596640 1 --2- 172.21.15.70:0/1017771603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56ec1033f0 0x7f56ec199660 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.098+0000 7f56f1596640 1 --2- 172.21.15.70:0/1017771603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56ec102a30 0x7f56ec199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:11.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.098+0000 7f56f1596640 1 -- 172.21.15.70:0/1017771603 >> 172.21.15.70:0/1017771603 conn(0x7f56ec0fe500 msgr2=0x7f56ec0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:11.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.098+0000 7f56f1596640 1 -- 172.21.15.70:0/1017771603 shutdown_connections 2024-06-04T05:04:11.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:11.098+0000 7f56f1596640 1 -- 172.21.15.70:0/1017771603 wait complete. 2024-06-04T05:04:11.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:11 smithi070 ceph-mon[32631]: pgmap v215: 97 pgs: 97 active+clean; 70 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 8 op/s 2024-06-04T05:04:11.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:11 smithi112 ceph-mon[36453]: pgmap v215: 97 pgs: 97 active+clean; 70 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 8 op/s 2024-06-04T05:04:11.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:11 smithi098 ceph-mon[35569]: pgmap v215: 97 pgs: 97 active+clean; 70 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 8 op/s 2024-06-04T05:04:11.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 110275 1.91 MB/sec execute 184 sec latency 17189.404 ms 2024-06-04T05:04:11.877 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'b977e96e', total seconds elapsed: 25.9 2024-06-04T05:04:12.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 110555 1.90 MB/sec execute 185 sec latency 15.844 ms 2024-06-04T05:04:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:13 smithi070 ceph-mon[32631]: pgmap v216: 97 pgs: 97 active+clean; 71 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 25 KiB/s wr, 15 op/s 2024-06-04T05:04:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:13 smithi112 ceph-mon[36453]: pgmap v216: 97 pgs: 97 active+clean; 71 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 25 KiB/s wr, 15 op/s 2024-06-04T05:04:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:13 smithi098 ceph-mon[35569]: pgmap v216: 97 pgs: 97 active+clean; 71 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 25 KiB/s wr, 15 op/s 2024-06-04T05:04:13.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 111051 1.90 MB/sec execute 186 sec latency 15.726 ms 2024-06-04T05:04:14.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 111584 1.91 MB/sec execute 187 sec latency 20.227 ms 2024-06-04T05:04:15.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:15 smithi070 ceph-mon[32631]: pgmap v217: 97 pgs: 97 active+clean; 71 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-06-04T05:04:15.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:15 smithi112 ceph-mon[36453]: pgmap v217: 97 pgs: 97 active+clean; 71 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-06-04T05:04:15.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:15 smithi098 ceph-mon[35569]: pgmap v217: 97 pgs: 97 active+clean; 71 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-06-04T05:04:15.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 112193 1.90 MB/sec execute 188 sec latency 18.909 ms 2024-06-04T05:04:16.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 112777 1.92 MB/sec execute 189 sec latency 12.247 ms 2024-06-04T05:04:17.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:17 smithi112 ceph-mon[36453]: pgmap v218: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 709 KiB/s wr, 84 op/s 2024-06-04T05:04:17.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:17 smithi098 ceph-mon[35569]: pgmap v218: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 709 KiB/s wr, 84 op/s 2024-06-04T05:04:17.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:17 smithi070 ceph-mon[32631]: pgmap v218: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 709 KiB/s wr, 84 op/s 2024-06-04T05:04:17.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 113347 1.93 MB/sec execute 190 sec latency 15.198 ms 2024-06-04T05:04:18.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 113826 1.94 MB/sec execute 191 sec latency 15.084 ms 2024-06-04T05:04:19.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:19 smithi112 ceph-mon[36453]: pgmap v219: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 691 KiB/s wr, 82 op/s 2024-06-04T05:04:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:19 smithi098 ceph-mon[35569]: pgmap v219: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 691 KiB/s wr, 82 op/s 2024-06-04T05:04:19.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:19 smithi070 ceph-mon[32631]: pgmap v219: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 691 KiB/s wr, 82 op/s 2024-06-04T05:04:19.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 114113 1.93 MB/sec execute 192 sec latency 15.031 ms 2024-06-04T05:04:20.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 114603 1.93 MB/sec execute 193 sec latency 16.415 ms 2024-06-04T05:04:21.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:21 smithi112 ceph-mon[36453]: pgmap v220: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 591 KiB/s wr, 70 op/s 2024-06-04T05:04:21.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:21 smithi098 ceph-mon[35569]: pgmap v220: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 591 KiB/s wr, 70 op/s 2024-06-04T05:04:21.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:21 smithi070 ceph-mon[32631]: pgmap v220: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 591 KiB/s wr, 70 op/s 2024-06-04T05:04:21.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 115155 1.94 MB/sec execute 194 sec latency 18.791 ms 2024-06-04T05:04:22.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 115767 1.93 MB/sec execute 195 sec latency 17.379 ms 2024-06-04T05:04:23.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:23 smithi112 ceph-mon[36453]: pgmap v221: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 1.3 MiB/s wr, 137 op/s 2024-06-04T05:04:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:23 smithi098 ceph-mon[35569]: pgmap v221: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 1.3 MiB/s wr, 137 op/s 2024-06-04T05:04:23.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:23 smithi070 ceph-mon[32631]: pgmap v221: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 1.3 MiB/s wr, 137 op/s 2024-06-04T05:04:23.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 116317 1.95 MB/sec execute 196 sec latency 16.660 ms 2024-06-04T05:04:24.116 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:04:24.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.197+0000 7fd8dff46640 1 -- 172.21.15.70:0/3495083531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8d8107df0 msgr2=0x7fd8d810f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.197+0000 7fd8dff46640 1 --2- 172.21.15.70:0/3495083531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8d8107df0 0x7fd8d810f970 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009ca0 tx=0x7fd8d002a8c0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.197+0000 7fd8dff46640 1 -- 172.21.15.70:0/3495083531 shutdown_connections 2024-06-04T05:04:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.197+0000 7fd8dff46640 1 --2- 172.21.15.70:0/3495083531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8d8107df0 0x7fd8d810f970 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.197+0000 7fd8dff46640 1 --2- 172.21.15.70:0/3495083531 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8d8101b90 0x7fd8d81076c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.197+0000 7fd8dff46640 1 --2- 172.21.15.70:0/3495083531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d81011e0 0x7fd8d81015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.197+0000 7fd8dff46640 1 -- 172.21.15.70:0/3495083531 >> 172.21.15.70:0/3495083531 conn(0x7fd8d8077ee0 msgr2=0x7fd8d80ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.197+0000 7fd8dff46640 1 -- 172.21.15.70:0/3495083531 shutdown_connections 2024-06-04T05:04:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.197+0000 7fd8dff46640 1 -- 172.21.15.70:0/3495083531 wait complete. 2024-06-04T05:04:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8dff46640 1 Processor -- start 2024-06-04T05:04:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8dff46640 1 -- start start 2024-06-04T05:04:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8dff46640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8d81011e0 0x7fd8d8198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8dff46640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8d8107df0 0x7fd8d8199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8dff46640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d819d8a0 0x7fd8d819dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8dff46640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d8112c00 con 0x7fd8d8107df0 2024-06-04T05:04:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8dff46640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d8112a80 con 0x7fd8d819d8a0 2024-06-04T05:04:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8dff46640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d8112d80 con 0x7fd8d81011e0 2024-06-04T05:04:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8dd4e1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8d8107df0 0x7fd8d8199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8ddce2640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8d81011e0 0x7fd8d8198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8dd4e1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8d8107df0 0x7fd8d8199520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49776/0 (socket says 172.21.15.70:49776) 2024-06-04T05:04:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8dd4e1640 1 -- 172.21.15.70:0/1014783813 learned_addr learned my addr 172.21.15.70:0/1014783813 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8de4e3640 1 --2- 172.21.15.70:0/1014783813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d819d8a0 0x7fd8d819dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8dd4e1640 1 -- 172.21.15.70:0/1014783813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8d81011e0 msgr2=0x7fd8d8198fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8dd4e1640 1 --2- 172.21.15.70:0/1014783813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8d81011e0 0x7fd8d8198fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8dd4e1640 1 -- 172.21.15.70:0/1014783813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d819d8a0 msgr2=0x7fd8d819dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8dd4e1640 1 --2- 172.21.15.70:0/1014783813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d819d8a0 0x7fd8d819dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8dd4e1640 1 -- 172.21.15.70:0/1014783813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d819e390 con 0x7fd8d8107df0 2024-06-04T05:04:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.198+0000 7fd8ddce2640 1 --2- 172.21.15.70:0/1014783813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8d81011e0 0x7fd8d8198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.199+0000 7fd8dd4e1640 1 --2- 172.21.15.70:0/1014783813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8d8107df0 0x7fd8d8199520 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800c8c0 tx=0x7fd8c800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.199+0000 7fd8ceffd640 1 -- 172.21.15.70:0/1014783813 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c8007bb0 con 0x7fd8d8107df0 2024-06-04T05:04:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.199+0000 7fd8dff46640 1 -- 172.21.15.70:0/1014783813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8d819e680 con 0x7fd8d8107df0 2024-06-04T05:04:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.199+0000 7fd8ceffd640 1 -- 172.21.15.70:0/1014783813 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8c8007d50 con 0x7fd8d8107df0 2024-06-04T05:04:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.199+0000 7fd8dff46640 1 -- 172.21.15.70:0/1014783813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8d81a5ec0 con 0x7fd8d8107df0 2024-06-04T05:04:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.199+0000 7fd8ceffd640 1 -- 172.21.15.70:0/1014783813 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c8018660 con 0x7fd8d8107df0 2024-06-04T05:04:24.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.202+0000 7fd8e2c67740 1 Processor -- start 2024-06-04T05:04:24.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.202+0000 7fd8e2c67740 1 -- start start 2024-06-04T05:04:24.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.202+0000 7fd8e2c67740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b82700 0x55f998b82ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.202+0000 7fd8e2c67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b83020 0x55f998b918d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.202+0000 7fd8e2c67740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b91e10 0x55f998b94200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.202+0000 7fd8e2c67740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f998af4c30 con 0x55f998b91e10 2024-06-04T05:04:24.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.202+0000 7fd8e2c67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f998ae47d0 con 0x55f998b83020 2024-06-04T05:04:24.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.202+0000 7fd8e2c67740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f998ae8810 con 0x55f998b82700 2024-06-04T05:04:24.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.202+0000 7fd8de4e3640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b82700 0x55f998b82ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.202+0000 7fd8de4e3640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b82700 0x55f998b82ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:34240/0 (socket says 172.21.15.70:34240) 2024-06-04T05:04:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.202+0000 7fd8ceffd640 1 -- 172.21.15.70:0/1014783813 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fd8c80188e0 con 0x7fd8d8107df0 2024-06-04T05:04:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.202+0000 7fd8de4e3640 1 -- 172.21.15.70:0/3112097074 learned_addr learned my addr 172.21.15.70:0/3112097074 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.202+0000 7fd8ddce2640 1 --2- 172.21.15.70:0/3112097074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b83020 0x55f998b918d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.202+0000 7fd8dd4e1640 1 --2- 172.21.15.70:0/3112097074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b91e10 0x55f998b94200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.203+0000 7fd8de4e3640 1 -- 172.21.15.70:0/3112097074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b83020 msgr2=0x55f998b918d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.203+0000 7fd8de4e3640 1 --2- 172.21.15.70:0/3112097074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b83020 0x55f998b918d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.203+0000 7fd8de4e3640 1 -- 172.21.15.70:0/3112097074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b91e10 msgr2=0x55f998b94200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.203+0000 7fd8de4e3640 1 --2- 172.21.15.70:0/3112097074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b91e10 0x55f998b94200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.203+0000 7fd8de4e3640 1 -- 172.21.15.70:0/3112097074 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9987f9210 con 0x55f998b82700 2024-06-04T05:04:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.203+0000 7fd8de4e3640 1 --2- 172.21.15.70:0/3112097074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b82700 0x55f998b82ae0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009430 tx=0x7fd8d0009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4470b0b272ffb7e6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.203+0000 7fd8c3fff640 1 -- 172.21.15.70:0/3112097074 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0031980 con 0x55f998b82700 2024-06-04T05:04:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.203+0000 7fd8c3fff640 1 -- 172.21.15.70:0/3112097074 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8d0036040 con 0x55f998b82700 2024-06-04T05:04:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.203+0000 7fd8c3fff640 1 -- 172.21.15.70:0/3112097074 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0004500 con 0x55f998b82700 2024-06-04T05:04:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.203+0000 7fd8ceffd640 1 --2- 172.21.15.70:0/1014783813 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd8c40b2a10 0x7fd8c40b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.203+0000 7fd8ddce2640 1 --2- 172.21.15.70:0/1014783813 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd8c40b2a10 0x7fd8c40b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.203+0000 7fd8ceffd640 1 -- 172.21.15.70:0/1014783813 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fd8c80e25c0 con 0x7fd8d8107df0 2024-06-04T05:04:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.204+0000 7fd8e2c67740 1 -- 172.21.15.70:0/3112097074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b82700 msgr2=0x55f998b82ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.204+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/3112097074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b82700 0x55f998b82ae0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009430 tx=0x7fd8d0009a20 comp rx=0 tx=0).stop 2024-06-04T05:04:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.204+0000 7fd8ddce2640 1 --2- 172.21.15.70:0/1014783813 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd8c40b2a10 0x7fd8c40b4ed0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4011630 tx=0x7fd8d400b390 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.204+0000 7fd8e2c67740 1 -- 172.21.15.70:0/3112097074 shutdown_connections 2024-06-04T05:04:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.204+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/3112097074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b83020 0x55f998b918d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.204+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/3112097074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b91e10 0x55f998b94200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.204+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/3112097074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b82700 0x55f998b82ae0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.204+0000 7fd8e2c67740 1 -- 172.21.15.70:0/3112097074 >> 172.21.15.70:0/3112097074 conn(0x55f998b7ed30 msgr2=0x55f998b7f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.204+0000 7fd8e2c67740 1 -- 172.21.15.70:0/3112097074 shutdown_connections 2024-06-04T05:04:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.204+0000 7fd8e2c67740 1 -- 172.21.15.70:0/3112097074 wait complete. 2024-06-04T05:04:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.204+0000 7fd8e2c67740 1 -- start start 2024-06-04T05:04:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.204+0000 7fd8e2c67740 1 Processor -- start 2024-06-04T05:04:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.205+0000 7fd8e2c67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b82700 0x55f998b8ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.205+0000 7fd8e2c67740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b83020 0x55f998b8f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.205+0000 7fd8e2c67740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b91e10 0x55f998b8f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.205+0000 7fd8e2c67740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f9987ca790 con 0x55f998b83020 2024-06-04T05:04:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.205+0000 7fd8e2c67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f998aa0890 con 0x55f998b82700 2024-06-04T05:04:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.205+0000 7fd8e2c67740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f998b949e0 con 0x55f998b91e10 2024-06-04T05:04:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.205+0000 7fd8e2c67740 10 client.0 _ll_register_callbacks cb 0x55f9987f8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.205+0000 7fd8dd4e1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b83020 0x55f998b8f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.205+0000 7fd8de4e3640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b91e10 0x55f998b8f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.205+0000 7fd8de4e3640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b91e10 0x55f998b8f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:34242/0 (socket says 172.21.15.70:34242) 2024-06-04T05:04:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.205+0000 7fd8dd4e1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b83020 0x55f998b8f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49794/0 (socket says 172.21.15.70:49794) 2024-06-04T05:04:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.205+0000 7fd8de4e3640 1 -- 172.21.15.70:0/2699703898 learned_addr learned my addr 172.21.15.70:0/2699703898 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.205+0000 7fd8ddce2640 1 --2- 172.21.15.70:0/2699703898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b82700 0x55f998b8ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.206+0000 7fd8de4e3640 1 -- 172.21.15.70:0/2699703898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b82700 msgr2=0x55f998b8ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.206+0000 7fd8de4e3640 1 --2- 172.21.15.70:0/2699703898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b82700 0x55f998b8ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.206+0000 7fd8de4e3640 1 -- 172.21.15.70:0/2699703898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b83020 msgr2=0x55f998b8f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.206+0000 7fd8de4e3640 1 --2- 172.21.15.70:0/2699703898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b83020 0x55f998b8f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.206+0000 7fd8de4e3640 1 -- 172.21.15.70:0/2699703898 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9988c5620 con 0x55f998b91e10 2024-06-04T05:04:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.206+0000 7fd8dd4e1640 1 --2- 172.21.15.70:0/2699703898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b83020 0x55f998b8f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.206+0000 7fd8de4e3640 1 --2- 172.21.15.70:0/2699703898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b91e10 0x55f998b8f730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd8d002aeb0 tx=0x7fd8d00317a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3c26f3a7522e295 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.206+0000 7fd8c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:04:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.206+0000 7fd8c17fa640 1 -- 172.21.15.70:0/2699703898 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d004a040 con 0x55f998b91e10 2024-06-04T05:04:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.206+0000 7fd8c17fa640 1 -- 172.21.15.70:0/2699703898 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fd8d0003bf0 con 0x55f998b91e10 2024-06-04T05:04:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.206+0000 7fd8e2c67740 1 -- 172.21.15.70:0/2699703898 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8d0036040 con 0x55f998b91e10 2024-06-04T05:04:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.207+0000 7fd8c17fa640 1 -- 172.21.15.70:0/2699703898 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd8d005c050 con 0x55f998b91e10 2024-06-04T05:04:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.207+0000 7fd8e2c67740 10 client.26659 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.207+0000 7fd8e2c67740 1 -- 172.21.15.70:0/2699703898 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f998ba0730 con 0x55f998b91e10 2024-06-04T05:04:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.207+0000 7fd8c17fa640 1 -- 172.21.15.70:0/2699703898 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8d0053070 con 0x55f998b91e10 2024-06-04T05:04:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.207+0000 7fd8c17fa640 1 -- 172.21.15.70:0/2699703898 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fd8d004d070 con 0x55f998b91e10 2024-06-04T05:04:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.207+0000 7fd8e2c67740 10 client.26659 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.207+0000 7fd8e2c67740 10 client.26659 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:04:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.207+0000 7fd8e2c67740 20 client.26659 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.207+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/2699703898 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f998ba1a80 0x55f998ba3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.207+0000 7fd8e2c67740 4 client.26659 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:04:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.207+0000 7fd8e2c67740 1 -- 172.21.15.70:0/2699703898 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd8d0036040 con 0x55f998ba1a80 2024-06-04T05:04:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.207+0000 7fd8ddce2640 1 --2- 172.21.15.70:0/2699703898 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f998ba1a80 0x55f998ba3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.208+0000 7fd8ddce2640 1 --2- 172.21.15.70:0/2699703898 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f998ba1a80 0x55f998ba3e70 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b1861132eba1c1e server_cookie=438ee47da63b90ac in_seq=0 out_seq=0 2024-06-04T05:04:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.208+0000 7fd8c17fa640 10 client.26659 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:24.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.208+0000 7fd8c17fa640 1 -- 172.21.15.70:0/2699703898 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fd8d0036040 con 0x55f998ba1a80 2024-06-04T05:04:24.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.208+0000 7fd8c17fa640 10 client.26659 handle_command_reply: tid=0 2024-06-04T05:04:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.208+0000 7fd8e2c67740 1 client.26659 shutdown 2024-06-04T05:04:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.209+0000 7fd8e2c67740 1 -- 172.21.15.70:0/2699703898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b91e10 msgr2=0x55f998b8f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.209+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/2699703898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b91e10 0x55f998b8f730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd8d002aeb0 tx=0x7fd8d00317a0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.209+0000 7fd8e2c67740 1 -- 172.21.15.70:0/2699703898 shutdown_connections 2024-06-04T05:04:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.209+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/2699703898 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f998ba1a80 0x55f998ba3e70 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.209+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/2699703898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b91e10 0x55f998b8f730 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.209+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/2699703898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b83020 0x55f998b8f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.209+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/2699703898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b82700 0x55f998b8ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.209+0000 7fd8c17fa640 0 client.26659 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.209+0000 7fd8e2c67740 1 -- 172.21.15.70:0/2699703898 >> 172.21.15.70:0/2699703898 conn(0x55f998b7ed30 msgr2=0x55f998b84b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.209+0000 7fd8e2c67740 1 -- 172.21.15.70:0/2699703898 shutdown_connections 2024-06-04T05:04:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.209+0000 7fd8e2c67740 1 -- 172.21.15.70:0/2699703898 wait complete. 2024-06-04T05:04:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.209+0000 7fd8e2c67740 20 client.26659 trim_cache size 0 max 16384 2024-06-04T05:04:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.226+0000 7fd8e2c67740 1 Processor -- start 2024-06-04T05:04:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.226+0000 7fd8e2c67740 1 -- start start 2024-06-04T05:04:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.226+0000 7fd8e2c67740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b82700 0x55f998ba94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.226+0000 7fd8e2c67740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b83020 0x55f998ba99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.226+0000 7fd8e2c67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b91e10 0x55f998b8bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.226+0000 7fd8e2c67740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd8bc00e860 con 0x55f998b82700 2024-06-04T05:04:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.226+0000 7fd8e2c67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8bc0103a0 con 0x55f998b91e10 2024-06-04T05:04:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.226+0000 7fd8e2c67740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd8bc00ef30 con 0x55f998b83020 2024-06-04T05:04:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.227+0000 7fd8ddce2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b82700 0x55f998ba94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.227+0000 7fd8ddce2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b82700 0x55f998ba94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49810/0 (socket says 172.21.15.70:49810) 2024-06-04T05:04:24.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.227+0000 7fd8ddce2640 1 -- 172.21.15.70:0/2889218687 learned_addr learned my addr 172.21.15.70:0/2889218687 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:24.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.227+0000 7fd8dd4e1640 1 --2- 172.21.15.70:0/2889218687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b83020 0x55f998ba99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.227+0000 7fd8de4e3640 1 --2- 172.21.15.70:0/2889218687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b91e10 0x55f998b8bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.227+0000 7fd8ddce2640 1 -- 172.21.15.70:0/2889218687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b83020 msgr2=0x55f998ba99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:24.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.227+0000 7fd8ddce2640 1 --2- 172.21.15.70:0/2889218687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b83020 0x55f998ba99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.227+0000 7fd8ddce2640 1 -- 172.21.15.70:0/2889218687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b91e10 msgr2=0x55f998b8bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:24.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.227+0000 7fd8ddce2640 1 --2- 172.21.15.70:0/2889218687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b91e10 0x55f998b8bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.227+0000 7fd8ddce2640 1 -- 172.21.15.70:0/2889218687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f998b7eb60 con 0x55f998b82700 2024-06-04T05:04:24.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.227+0000 7fd8ddce2640 1 --2- 172.21.15.70:0/2889218687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b82700 0x55f998ba94a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4010d10 tx=0x7fd8d4011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327e6b53c977b960 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:24.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.227+0000 7fd8c37fe640 1 -- 172.21.15.70:0/2889218687 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d4010e00 con 0x55f998b82700 2024-06-04T05:04:24.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.227+0000 7fd8c37fe640 1 -- 172.21.15.70:0/2889218687 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8d4022030 con 0x55f998b82700 2024-06-04T05:04:24.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.227+0000 7fd8c37fe640 1 -- 172.21.15.70:0/2889218687 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d402a430 con 0x55f998b82700 2024-06-04T05:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.228+0000 7fd8e2c67740 1 -- 172.21.15.70:0/2889218687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b82700 msgr2=0x55f998ba94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.228+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/2889218687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b82700 0x55f998ba94a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4010d10 tx=0x7fd8d4011f90 comp rx=0 tx=0).stop 2024-06-04T05:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.228+0000 7fd8e2c67740 1 -- 172.21.15.70:0/2889218687 shutdown_connections 2024-06-04T05:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.228+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/2889218687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b91e10 0x55f998b8bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.228+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/2889218687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b83020 0x55f998ba99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.228+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/2889218687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b82700 0x55f998ba94a0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.228+0000 7fd8e2c67740 1 -- 172.21.15.70:0/2889218687 >> 172.21.15.70:0/2889218687 conn(0x55f998b7ed30 msgr2=0x55f998b8f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.228+0000 7fd8e2c67740 1 -- 172.21.15.70:0/2889218687 shutdown_connections 2024-06-04T05:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.228+0000 7fd8e2c67740 1 -- 172.21.15.70:0/2889218687 wait complete. 2024-06-04T05:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.228+0000 7fd8e2c67740 1 -- start start 2024-06-04T05:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8e2c67740 1 Processor -- start 2024-06-04T05:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8e2c67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b82700 0x55f998b9a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8e2c67740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b83020 0x55f998b956f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8e2c67740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b91e10 0x55f998b95c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8e2c67740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd8bc00fcd0 con 0x55f998b83020 2024-06-04T05:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8e2c67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8bc00f600 con 0x55f998b82700 2024-06-04T05:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8e2c67740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd8bc00c860 con 0x55f998b91e10 2024-06-04T05:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8e2c67740 10 client.0 _ll_register_callbacks cb 0x7fd8d400bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8ddce2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b82700 0x55f998b9a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8ddce2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b82700 0x55f998b9a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:44714/0 (socket says 172.21.15.70:44714) 2024-06-04T05:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8ddce2640 1 -- 172.21.15.70:0/1177821831 learned_addr learned my addr 172.21.15.70:0/1177821831 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8dd4e1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b83020 0x55f998b956f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8de4e3640 1 --2- 172.21.15.70:0/1177821831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b91e10 0x55f998b95c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8ddce2640 1 -- 172.21.15.70:0/1177821831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b91e10 msgr2=0x55f998b95c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8ddce2640 1 --2- 172.21.15.70:0/1177821831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b91e10 0x55f998b95c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8ddce2640 1 -- 172.21.15.70:0/1177821831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b83020 msgr2=0x55f998b956f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8ddce2640 1 --2- 172.21.15.70:0/1177821831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b83020 0x55f998b956f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8ddce2640 1 -- 172.21.15.70:0/1177821831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f998b96440 con 0x55f998b82700 2024-06-04T05:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8de4e3640 1 --2- 172.21.15.70:0/1177821831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b91e10 0x55f998b95c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.229+0000 7fd8dd4e1640 1 --2- 172.21.15.70:0/1177821831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b83020 0x55f998b956f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.230+0000 7fd8ddce2640 1 --2- 172.21.15.70:0/1177821831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b82700 0x55f998b9a380 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8d400b5f0 tx=0x7fd8d4010bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e84c58d037a0bc7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.230+0000 7fd8c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.230+0000 7fd8c17fa640 1 -- 172.21.15.70:0/1177821831 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d4039020 con 0x55f998b82700 2024-06-04T05:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.230+0000 7fd8e2c67740 1 -- 172.21.15.70:0/1177821831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8d4022030 con 0x55f998b82700 2024-06-04T05:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.230+0000 7fd8c17fa640 1 -- 172.21.15.70:0/1177821831 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fd8d401e920 con 0x55f998b82700 2024-06-04T05:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.230+0000 7fd8c17fa640 1 -- 172.21.15.70:0/1177821831 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd8d4044050 con 0x55f998b82700 2024-06-04T05:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.230+0000 7fd8e2c67740 10 client.26597 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.230+0000 7fd8e2c67740 1 -- 172.21.15.70:0/1177821831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f998c0c350 con 0x55f998b82700 2024-06-04T05:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.230+0000 7fd8c17fa640 1 -- 172.21.15.70:0/1177821831 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8d4022030 con 0x55f998b82700 2024-06-04T05:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.230+0000 7fd8c17fa640 1 -- 172.21.15.70:0/1177821831 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fd8d4034450 con 0x55f998b82700 2024-06-04T05:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.230+0000 7fd8e2c67740 10 client.26597 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.230+0000 7fd8e2c67740 10 client.26597 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.230+0000 7fd8e2c67740 20 client.26597 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.230+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/1177821831 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f998c0ca00 0x55f998c0edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.230+0000 7fd8e2c67740 4 client.26597 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.230+0000 7fd8e2c67740 1 -- 172.21.15.70:0/1177821831 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f998af6b80 con 0x55f998c0ca00 2024-06-04T05:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.231+0000 7fd8dd4e1640 1 --2- 172.21.15.70:0/1177821831 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f998c0ca00 0x55f998c0edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.231+0000 7fd8c17fa640 10 client.26597 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.231+0000 7fd8dd4e1640 1 --2- 172.21.15.70:0/1177821831 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f998c0ca00 0x55f998c0edf0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8069f2b455d4934b server_cookie=724cd42ddaca9cc9 in_seq=0 out_seq=0 2024-06-04T05:04:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.254+0000 7fd8c17fa640 1 -- 172.21.15.70:0/1177821831 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f998af6b80 con 0x55f998c0ca00 2024-06-04T05:04:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.254+0000 7fd8c17fa640 10 client.26597 handle_command_reply: tid=0 2024-06-04T05:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.254+0000 7fd8e2c67740 1 client.26597 shutdown 2024-06-04T05:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.254+0000 7fd8e2c67740 1 -- 172.21.15.70:0/1177821831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b82700 msgr2=0x55f998b9a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.254+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/1177821831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b82700 0x55f998b9a380 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8d400b5f0 tx=0x7fd8d4010bb0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.254+0000 7fd8e2c67740 1 -- 172.21.15.70:0/1177821831 shutdown_connections 2024-06-04T05:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.254+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/1177821831 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f998c0ca00 0x55f998c0edf0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.254+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/1177821831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f998b91e10 0x55f998b95c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.254+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/1177821831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f998b83020 0x55f998b956f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.254+0000 7fd8c17fa640 0 client.26597 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.254+0000 7fd8e2c67740 1 --2- 172.21.15.70:0/1177821831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f998b82700 0x55f998b9a380 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.254+0000 7fd8e2c67740 1 -- 172.21.15.70:0/1177821831 >> 172.21.15.70:0/1177821831 conn(0x55f998b7ed30 msgr2=0x55f998b7d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.254+0000 7fd8e2c67740 1 -- 172.21.15.70:0/1177821831 shutdown_connections 2024-06-04T05:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.254+0000 7fd8e2c67740 1 -- 172.21.15.70:0/1177821831 wait complete. 2024-06-04T05:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.254+0000 7fd8e2c67740 20 client.26597 trim_cache size 0 max 16384 2024-06-04T05:04:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:04:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:04:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:04:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.255+0000 7fd8dff46640 1 -- 172.21.15.70:0/1014783813 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd8c40b2a10 msgr2=0x7fd8c40b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.255+0000 7fd8dff46640 1 --2- 172.21.15.70:0/1014783813 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd8c40b2a10 0x7fd8c40b4ed0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4011630 tx=0x7fd8d400b390 comp rx=0 tx=0).stop 2024-06-04T05:04:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.255+0000 7fd8dff46640 1 -- 172.21.15.70:0/1014783813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8d8107df0 msgr2=0x7fd8d8199520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.255+0000 7fd8dff46640 1 --2- 172.21.15.70:0/1014783813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8d8107df0 0x7fd8d8199520 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800c8c0 tx=0x7fd8c800cd80 comp rx=0 tx=0).stop 2024-06-04T05:04:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.255+0000 7fd8dff46640 1 -- 172.21.15.70:0/1014783813 shutdown_connections 2024-06-04T05:04:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.255+0000 7fd8dff46640 1 --2- 172.21.15.70:0/1014783813 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd8c40b2a10 0x7fd8c40b4ed0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.255+0000 7fd8dff46640 1 --2- 172.21.15.70:0/1014783813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d819d8a0 0x7fd8d819dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.255+0000 7fd8dff46640 1 --2- 172.21.15.70:0/1014783813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8d8107df0 0x7fd8d8199520 secure :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800c8c0 tx=0x7fd8c800cd80 comp rx=0 tx=0).stop 2024-06-04T05:04:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.255+0000 7fd8dff46640 1 --2- 172.21.15.70:0/1014783813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8d81011e0 0x7fd8d8198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.255+0000 7fd8dff46640 1 -- 172.21.15.70:0/1014783813 >> 172.21.15.70:0/1014783813 conn(0x7fd8d8077ee0 msgr2=0x7fd8d81065c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.255+0000 7fd8dff46640 1 -- 172.21.15.70:0/1014783813 shutdown_connections 2024-06-04T05:04:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:24.255+0000 7fd8dff46640 1 -- 172.21.15.70:0/1014783813 wait complete. 2024-06-04T05:04:24.399 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:04:24.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 116876 1.95 MB/sec execute 197 sec latency 22.519 ms 2024-06-04T05:04:25.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:25 smithi112 ceph-mon[36453]: pgmap v222: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-06-04T05:04:25.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:25 smithi098 ceph-mon[35569]: pgmap v222: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-06-04T05:04:25.552 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:25 smithi070 ceph-mon[32631]: pgmap v222: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-06-04T05:04:25.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 117367 1.97 MB/sec execute 198 sec latency 12.751 ms 2024-06-04T05:04:26.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.440+0000 7f6c69575700 1 -- 172.21.15.70:0/1624537007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c64107460 msgr2=0x7f6c64107840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:26.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.440+0000 7f6c69575700 1 --2- 172.21.15.70:0/1624537007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c64107460 0x7f6c64107840 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009990 tx=0x7f6c54009ca0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.440+0000 7f6c69575700 1 -- 172.21.15.70:0/1624537007 shutdown_connections 2024-06-04T05:04:26.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.440+0000 7f6c69575700 1 --2- 172.21.15.70:0/1624537007 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c641030e0 0x7f6c6410f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.440+0000 7f6c69575700 1 --2- 172.21.15.70:0/1624537007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c64107460 0x7f6c64107840 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.440+0000 7f6c69575700 1 --2- 172.21.15.70:0/1624537007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c64102720 0x7f6c64102ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.440+0000 7f6c69575700 1 -- 172.21.15.70:0/1624537007 >> 172.21.15.70:0/1624537007 conn(0x7f6c64079300 msgr2=0x7f6c64079710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:26.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.440+0000 7f6c69575700 1 -- 172.21.15.70:0/1624537007 shutdown_connections 2024-06-04T05:04:26.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.440+0000 7f6c69575700 1 -- 172.21.15.70:0/1624537007 wait complete. 2024-06-04T05:04:26.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.440+0000 7f6c69575700 1 Processor -- start 2024-06-04T05:04:26.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c69575700 1 -- start start 2024-06-04T05:04:26.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c69575700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c64102720 0x7f6c6419f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c69575700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c641030e0 0x7f6c6419fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c69575700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c64107460 0x7f6c64199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c69575700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6c64112650 con 0x7f6c64107460 2024-06-04T05:04:26.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c69575700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6c641124d0 con 0x7f6c64102720 2024-06-04T05:04:26.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c69575700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6c641127d0 con 0x7f6c641030e0 2024-06-04T05:04:26.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c637fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c64107460 0x7f6c64199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c637fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c64107460 0x7f6c64199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54820/0 (socket says 172.21.15.70:54820) 2024-06-04T05:04:26.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c637fe700 1 -- 172.21.15.70:0/3943605960 learned_addr learned my addr 172.21.15.70:0/3943605960 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:26.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c62ffd700 1 --2- 172.21.15.70:0/3943605960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c64102720 0x7f6c6419f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c627fc700 1 --2- 172.21.15.70:0/3943605960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c641030e0 0x7f6c6419fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c637fe700 1 -- 172.21.15.70:0/3943605960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c641030e0 msgr2=0x7f6c6419fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:26.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c637fe700 1 --2- 172.21.15.70:0/3943605960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c641030e0 0x7f6c6419fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c637fe700 1 -- 172.21.15.70:0/3943605960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c64102720 msgr2=0x7f6c6419f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:26.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c637fe700 1 --2- 172.21.15.70:0/3943605960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c64102720 0x7f6c6419f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c637fe700 1 -- 172.21.15.70:0/3943605960 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c6419a200 con 0x7f6c64107460 2024-06-04T05:04:26.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.441+0000 7f6c62ffd700 1 --2- 172.21.15.70:0/3943605960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c64102720 0x7f6c6419f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:26.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.442+0000 7f6c637fe700 1 --2- 172.21.15.70:0/3943605960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c64107460 0x7f6c64199aa0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c00efc0 tx=0x7f6c5c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:26.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.442+0000 7f6c4bfff700 1 -- 172.21.15.70:0/3943605960 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c0092e0 con 0x7f6c64107460 2024-06-04T05:04:26.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.442+0000 7f6c69575700 1 -- 172.21.15.70:0/3943605960 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c6419a490 con 0x7f6c64107460 2024-06-04T05:04:26.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.442+0000 7f6c4bfff700 1 -- 172.21.15.70:0/3943605960 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6c5c004bb0 con 0x7f6c64107460 2024-06-04T05:04:26.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.442+0000 7f6c4bfff700 1 -- 172.21.15.70:0/3943605960 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c00ce40 con 0x7f6c64107460 2024-06-04T05:04:26.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.442+0000 7f6c69575700 1 -- 172.21.15.70:0/3943605960 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c641a67b0 con 0x7f6c64107460 2024-06-04T05:04:26.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.444+0000 7f6c4bfff700 1 -- 172.21.15.70:0/3943605960 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f6c5c01b030 con 0x7f6c64107460 2024-06-04T05:04:26.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.445+0000 7f6c4bfff700 1 --2- 172.21.15.70:0/3943605960 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6c4c0b29f0 0x7f6c4c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.445+0000 7f6c62ffd700 1 --2- 172.21.15.70:0/3943605960 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6c4c0b29f0 0x7f6c4c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.445+0000 7f6c4bfff700 1 -- 172.21.15.70:0/3943605960 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f6c5c0e1f70 con 0x7f6c64107460 2024-06-04T05:04:26.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.445+0000 7f6c62ffd700 1 --2- 172.21.15.70:0/3943605960 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6c4c0b29f0 0x7f6c4c0b4eb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6c5400b2e0 tx=0x7f6c5401a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:26.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.446+0000 7f6c73e0ab80 1 Processor -- start 2024-06-04T05:04:26.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.446+0000 7f6c73e0ab80 1 -- start start 2024-06-04T05:04:26.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.446+0000 7f6c73e0ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14574c70 0x55bd14575050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.446+0000 7f6c73e0ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14575620 0x55bd14578d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.446+0000 7f6c73e0ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14579370 0x55bd145797b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.446+0000 7f6c73e0ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bd144cfc90 con 0x55bd14575620 2024-06-04T05:04:26.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.446+0000 7f6c73e0ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bd144bf830 con 0x55bd14579370 2024-06-04T05:04:26.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.446+0000 7f6c73e0ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bd144c3870 con 0x55bd14574c70 2024-06-04T05:04:26.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.446+0000 7f6c627fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14575620 0x55bd14578d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.446+0000 7f6c627fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14575620 0x55bd14578d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54830/0 (socket says 172.21.15.70:54830) 2024-06-04T05:04:26.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.446+0000 7f6c62ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14574c70 0x55bd14575050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.446+0000 7f6c627fc700 1 -- 172.21.15.70:0/4243416537 learned_addr learned my addr 172.21.15.70:0/4243416537 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:26.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.446+0000 7f6c637fe700 1 --2- 172.21.15.70:0/4243416537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14579370 0x55bd145797b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.446+0000 7f6c627fc700 1 -- 172.21.15.70:0/4243416537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14574c70 msgr2=0x55bd14575050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:26.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.447+0000 7f6c627fc700 1 --2- 172.21.15.70:0/4243416537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14574c70 0x55bd14575050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.447+0000 7f6c627fc700 1 -- 172.21.15.70:0/4243416537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14579370 msgr2=0x55bd145797b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:26.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.447+0000 7f6c627fc700 1 --2- 172.21.15.70:0/4243416537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14579370 0x55bd145797b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.447+0000 7f6c627fc700 1 -- 172.21.15.70:0/4243416537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd1440f8d0 con 0x55bd14575620 2024-06-04T05:04:26.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.447+0000 7f6c62ffd700 1 --2- 172.21.15.70:0/4243416537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14574c70 0x55bd14575050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:26.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.447+0000 7f6c627fc700 1 --2- 172.21.15.70:0/4243416537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14575620 0x55bd14578d50 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f6c58009d20 tx=0x7f6c58011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ffb77eacd104a2a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:26.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.447+0000 7f6c497fa700 1 -- 172.21.15.70:0/4243416537 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5801e070 con 0x55bd14575620 2024-06-04T05:04:26.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.447+0000 7f6c497fa700 1 -- 172.21.15.70:0/4243416537 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6c58019ac0 con 0x55bd14575620 2024-06-04T05:04:26.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.447+0000 7f6c497fa700 1 -- 172.21.15.70:0/4243416537 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5800ea30 con 0x55bd14575620 2024-06-04T05:04:26.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.448+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/4243416537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14575620 msgr2=0x55bd14578d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:26.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.448+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/4243416537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14575620 0x55bd14578d50 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f6c58009d20 tx=0x7f6c58011720 comp rx=0 tx=0).stop 2024-06-04T05:04:26.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.448+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/4243416537 shutdown_connections 2024-06-04T05:04:26.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.448+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/4243416537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14579370 0x55bd145797b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.448+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/4243416537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14574c70 0x55bd14575050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.448+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/4243416537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14575620 0x55bd14578d50 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.448+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/4243416537 >> 172.21.15.70:0/4243416537 conn(0x55bd1449e080 msgr2=0x55bd1456ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:26.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.448+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/4243416537 shutdown_connections 2024-06-04T05:04:26.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.448+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/4243416537 wait complete. 2024-06-04T05:04:26.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.448+0000 7f6c73e0ab80 1 -- start start 2024-06-04T05:04:26.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.448+0000 7f6c73e0ab80 1 Processor -- start 2024-06-04T05:04:26.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c73e0ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14574c70 0x55bd145807b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c73e0ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14575620 0x55bd1457dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c73e0ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14579370 0x55bd1457dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c73e0ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bd143c5fa0 con 0x55bd14579370 2024-06-04T05:04:26.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c73e0ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bd144617c0 con 0x55bd14574c70 2024-06-04T05:04:26.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c73e0ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bd14581530 con 0x55bd14575620 2024-06-04T05:04:26.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c73e0ab80 10 client.0 _ll_register_callbacks cb 0x55bd1445a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:26.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c637fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14579370 0x55bd1457dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c637fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14579370 0x55bd1457dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54838/0 (socket says 172.21.15.70:54838) 2024-06-04T05:04:26.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c627fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14575620 0x55bd1457dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c637fe700 1 -- 172.21.15.70:0/1342597863 learned_addr learned my addr 172.21.15.70:0/1342597863 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:26.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c62ffd700 1 --2- 172.21.15.70:0/1342597863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14574c70 0x55bd145807b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c627fc700 1 -- 172.21.15.70:0/1342597863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14574c70 msgr2=0x55bd145807b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:26.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c627fc700 1 --2- 172.21.15.70:0/1342597863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14574c70 0x55bd145807b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c627fc700 1 -- 172.21.15.70:0/1342597863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14579370 msgr2=0x55bd1457dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:26.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c627fc700 1 --2- 172.21.15.70:0/1342597863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14579370 0x55bd1457dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c627fc700 1 -- 172.21.15.70:0/1342597863 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd1445e690 con 0x55bd14575620 2024-06-04T05:04:26.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c627fc700 1 --2- 172.21.15.70:0/1342597863 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14575620 0x55bd1457dab0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6c5801c040 tx=0x7f6c58005df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbcf78d274d078c7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:26.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c46ff5700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:04:26.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c46ff5700 1 -- 172.21.15.70:0/1342597863 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5801e050 con 0x55bd14575620 2024-06-04T05:04:26.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c46ff5700 1 -- 172.21.15.70:0/1342597863 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f6c58004030 con 0x55bd14575620 2024-06-04T05:04:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.449+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/1342597863 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c58019ac0 con 0x55bd14575620 2024-06-04T05:04:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.450+0000 7f6c46ff5700 1 -- 172.21.15.70:0/1342597863 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6c58032050 con 0x55bd14575620 2024-06-04T05:04:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.450+0000 7f6c73e0ab80 10 client.26677 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.450+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/1342597863 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bd1458bc20 con 0x55bd14575620 2024-06-04T05:04:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.450+0000 7f6c46ff5700 1 -- 172.21.15.70:0/1342597863 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6c58019ac0 con 0x55bd14575620 2024-06-04T05:04:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.450+0000 7f6c46ff5700 1 -- 172.21.15.70:0/1342597863 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f6c5802d030 con 0x55bd14575620 2024-06-04T05:04:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.450+0000 7f6c73e0ab80 10 client.26677 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.450+0000 7f6c73e0ab80 10 client.26677 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:04:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.450+0000 7f6c73e0ab80 20 client.26677 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.450+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/1342597863 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bd1458e1a0 0x55bd14590590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.450+0000 7f6c73e0ab80 4 client.26677 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:04:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.450+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/1342597863 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd1453c9e0 con 0x55bd1458e1a0 2024-06-04T05:04:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.450+0000 7f6c62ffd700 1 --2- 172.21.15.70:0/1342597863 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bd1458e1a0 0x55bd14590590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.451+0000 7f6c62ffd700 1 --2- 172.21.15.70:0/1342597863 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bd1458e1a0 0x55bd14590590 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=659b5fe160fb8a81 server_cookie=72e0221ee641780a in_seq=0 out_seq=0 2024-06-04T05:04:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.451+0000 7f6c46ff5700 10 client.26677 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:26.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.451+0000 7f6c46ff5700 1 -- 172.21.15.70:0/1342597863 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bd1453c9e0 con 0x55bd1458e1a0 2024-06-04T05:04:26.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.451+0000 7f6c46ff5700 10 client.26677 handle_command_reply: tid=0 2024-06-04T05:04:26.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.451+0000 7f6c73e0ab80 1 client.26677 shutdown 2024-06-04T05:04:26.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.452+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/1342597863 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14575620 msgr2=0x55bd1457dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:26.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.452+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/1342597863 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14575620 0x55bd1457dab0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6c5801c040 tx=0x7f6c58005df0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.452+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/1342597863 shutdown_connections 2024-06-04T05:04:26.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.452+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/1342597863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14579370 0x55bd1457dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.452+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/1342597863 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bd1458e1a0 0x55bd14590590 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.452+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/1342597863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14574c70 0x55bd145807b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.452+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/1342597863 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14575620 0x55bd1457dab0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.452+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/1342597863 >> 172.21.15.70:0/1342597863 conn(0x55bd1449e080 msgr2=0x55bd1457a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:26.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.452+0000 7f6c46ff5700 0 client.26677 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:26.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.452+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/1342597863 shutdown_connections 2024-06-04T05:04:26.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.452+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/1342597863 wait complete. 2024-06-04T05:04:26.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.452+0000 7f6c73e0ab80 20 client.26677 trim_cache size 0 max 16384 2024-06-04T05:04:26.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c73e0ab80 1 Processor -- start 2024-06-04T05:04:26.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c73e0ab80 1 -- start start 2024-06-04T05:04:26.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c73e0ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14574c70 0x55bd1457c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c73e0ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14575620 0x55bd14595cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c73e0ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14579370 0x55bd14596210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c73e0ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6c641abd70 con 0x55bd14579370 2024-06-04T05:04:26.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c73e0ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6c641ad8b0 con 0x55bd14575620 2024-06-04T05:04:26.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c73e0ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6c641ac440 con 0x55bd14574c70 2024-06-04T05:04:26.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c637fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14579370 0x55bd14596210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c637fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14579370 0x55bd14596210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54844/0 (socket says 172.21.15.70:54844) 2024-06-04T05:04:26.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c637fe700 1 -- 172.21.15.70:0/680421929 learned_addr learned my addr 172.21.15.70:0/680421929 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:26.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c62ffd700 1 --2- 172.21.15.70:0/680421929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14574c70 0x55bd1457c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c627fc700 1 --2- 172.21.15.70:0/680421929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14575620 0x55bd14595cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c637fe700 1 -- 172.21.15.70:0/680421929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14574c70 msgr2=0x55bd1457c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:26.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c637fe700 1 --2- 172.21.15.70:0/680421929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14574c70 0x55bd1457c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c637fe700 1 -- 172.21.15.70:0/680421929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14575620 msgr2=0x55bd14595cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:26.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c637fe700 1 --2- 172.21.15.70:0/680421929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14575620 0x55bd14595cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.475+0000 7f6c637fe700 1 -- 172.21.15.70:0/680421929 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd1456fce0 con 0x55bd14579370 2024-06-04T05:04:26.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.476+0000 7f6c62ffd700 1 --2- 172.21.15.70:0/680421929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14574c70 0x55bd1457c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:26.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.476+0000 7f6c637fe700 1 --2- 172.21.15.70:0/680421929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14579370 0x55bd14596210 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c000f80 tx=0x7f6c5c00c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2afbcb0c45b084db server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:26.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.476+0000 7f6c48ff9700 1 -- 172.21.15.70:0/680421929 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c0c3070 con 0x55bd14579370 2024-06-04T05:04:26.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.476+0000 7f6c48ff9700 1 -- 172.21.15.70:0/680421929 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6c5c0a9210 con 0x55bd14579370 2024-06-04T05:04:26.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.476+0000 7f6c48ff9700 1 -- 172.21.15.70:0/680421929 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c0a9470 con 0x55bd14579370 2024-06-04T05:04:26.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/680421929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14579370 msgr2=0x55bd14596210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:26.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/680421929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14579370 0x55bd14596210 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c000f80 tx=0x7f6c5c00c870 comp rx=0 tx=0).stop 2024-06-04T05:04:26.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/680421929 shutdown_connections 2024-06-04T05:04:26.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/680421929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14579370 0x55bd14596210 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/680421929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14574c70 0x55bd1457c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/680421929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14575620 0x55bd14595cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/680421929 >> 172.21.15.70:0/680421929 conn(0x55bd1449e080 msgr2=0x55bd1457c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:26.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/680421929 shutdown_connections 2024-06-04T05:04:26.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/680421929 wait complete. 2024-06-04T05:04:26.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 1 -- start start 2024-06-04T05:04:26.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 1 Processor -- start 2024-06-04T05:04:26.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14574c70 0x55bd145830c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14575620 0x55bd14583600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14579370 0x55bd145f0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6c641ad1e0 con 0x55bd14574c70 2024-06-04T05:04:26.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6c641acb10 con 0x55bd14575620 2024-06-04T05:04:26.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6c641a9d50 con 0x55bd14579370 2024-06-04T05:04:26.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c73e0ab80 10 client.0 _ll_register_callbacks cb 0x55bd1445a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:26.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.477+0000 7f6c62ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14574c70 0x55bd145830c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.478+0000 7f6c62ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14574c70 0x55bd145830c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54856/0 (socket says 172.21.15.70:54856) 2024-06-04T05:04:26.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.478+0000 7f6c62ffd700 1 -- 172.21.15.70:0/2865941056 learned_addr learned my addr 172.21.15.70:0/2865941056 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:26.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.478+0000 7f6c627fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14575620 0x55bd14583600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.478+0000 7f6c637fe700 1 --2- 172.21.15.70:0/2865941056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14579370 0x55bd145f0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.478+0000 7f6c62ffd700 1 -- 172.21.15.70:0/2865941056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14579370 msgr2=0x55bd145f0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:26.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.478+0000 7f6c62ffd700 1 --2- 172.21.15.70:0/2865941056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14579370 0x55bd145f0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.478+0000 7f6c62ffd700 1 -- 172.21.15.70:0/2865941056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14575620 msgr2=0x55bd14583600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:26.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.478+0000 7f6c62ffd700 1 --2- 172.21.15.70:0/2865941056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14575620 0x55bd14583600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.478+0000 7f6c62ffd700 1 -- 172.21.15.70:0/2865941056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd145f13a0 con 0x55bd14574c70 2024-06-04T05:04:26.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.478+0000 7f6c62ffd700 1 --2- 172.21.15.70:0/2865941056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14574c70 0x55bd145830c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f6c540098e0 tx=0x7f6c54009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5346d9ed540124e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:26.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.478+0000 7f6c46ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:04:26.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.478+0000 7f6c46ff5700 1 -- 172.21.15.70:0/2865941056 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c54004e80 con 0x55bd14574c70 2024-06-04T05:04:26.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.478+0000 7f6c46ff5700 1 -- 172.21.15.70:0/2865941056 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f6c54034400 con 0x55bd14574c70 2024-06-04T05:04:26.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.478+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/2865941056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c5c0a9210 con 0x55bd14574c70 2024-06-04T05:04:26.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.478+0000 7f6c46ff5700 1 -- 172.21.15.70:0/2865941056 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6c54034640 con 0x55bd14574c70 2024-06-04T05:04:26.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.478+0000 7f6c73e0ab80 10 client.17388 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:26.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.478+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/2865941056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bd145f97f0 con 0x55bd14574c70 2024-06-04T05:04:26.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.479+0000 7f6c46ff5700 1 -- 172.21.15.70:0/2865941056 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6c5c0a9210 con 0x55bd14574c70 2024-06-04T05:04:26.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.479+0000 7f6c46ff5700 1 -- 172.21.15.70:0/2865941056 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f6c54044a50 con 0x55bd14574c70 2024-06-04T05:04:26.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.479+0000 7f6c73e0ab80 10 client.17388 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:26.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.479+0000 7f6c73e0ab80 10 client.17388 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:04:26.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.479+0000 7f6c73e0ab80 20 client.17388 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:26.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.479+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/2865941056 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bd145f9eb0 0x55bd145fc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:26.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.479+0000 7f6c73e0ab80 4 client.17388 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:04:26.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.479+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/2865941056 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bd144d1be0 con 0x55bd145f9eb0 2024-06-04T05:04:26.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.479+0000 7f6c627fc700 1 --2- 172.21.15.70:0/2865941056 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bd145f9eb0 0x55bd145fc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:26.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.479+0000 7f6c46ff5700 10 client.17388 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:26.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.479+0000 7f6c627fc700 1 --2- 172.21.15.70:0/2865941056 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bd145f9eb0 0x55bd145fc2d0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3743785780164bc7 server_cookie=83aae18b80d1e76d in_seq=0 out_seq=0 2024-06-04T05:04:26.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.480+0000 7f6c46ff5700 1 -- 172.21.15.70:0/2865941056 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bd144d1be0 con 0x55bd145f9eb0 2024-06-04T05:04:26.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.480+0000 7f6c46ff5700 10 client.17388 handle_command_reply: tid=0 2024-06-04T05:04:26.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.480+0000 7f6c73e0ab80 1 client.17388 shutdown 2024-06-04T05:04:26.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.480+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/2865941056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14574c70 msgr2=0x55bd145830c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:26.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.480+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/2865941056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14574c70 0x55bd145830c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f6c540098e0 tx=0x7f6c54009310 comp rx=0 tx=0).stop 2024-06-04T05:04:26.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.480+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/2865941056 shutdown_connections 2024-06-04T05:04:26.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.480+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/2865941056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd14579370 0x55bd145f0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.480+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/2865941056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd14574c70 0x55bd145830c0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.480+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/2865941056 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bd145f9eb0 0x55bd145fc2d0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.480+0000 7f6c73e0ab80 1 --2- 172.21.15.70:0/2865941056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd14575620 0x55bd14583600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.480+0000 7f6c46ff5700 0 client.17388 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:26.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.480+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/2865941056 >> 172.21.15.70:0/2865941056 conn(0x55bd1449e080 msgr2=0x55bd1457fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:26.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.480+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/2865941056 shutdown_connections 2024-06-04T05:04:26.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.480+0000 7f6c73e0ab80 1 -- 172.21.15.70:0/2865941056 wait complete. 2024-06-04T05:04:26.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.480+0000 7f6c73e0ab80 20 client.17388 trim_cache size 0 max 16384 2024-06-04T05:04:26.487 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:04:26.487 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:04:26.487 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:04:26.487 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:04:26.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.481+0000 7f6c69575700 1 -- 172.21.15.70:0/3943605960 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6c4c0b29f0 msgr2=0x7f6c4c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:26.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.481+0000 7f6c69575700 1 --2- 172.21.15.70:0/3943605960 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6c4c0b29f0 0x7f6c4c0b4eb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6c5400b2e0 tx=0x7f6c5401a040 comp rx=0 tx=0).stop 2024-06-04T05:04:26.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.481+0000 7f6c69575700 1 -- 172.21.15.70:0/3943605960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c64107460 msgr2=0x7f6c64199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:26.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.481+0000 7f6c69575700 1 --2- 172.21.15.70:0/3943605960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c64107460 0x7f6c64199aa0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c00efc0 tx=0x7f6c5c00c3a0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.481+0000 7f6c69575700 1 -- 172.21.15.70:0/3943605960 shutdown_connections 2024-06-04T05:04:26.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.481+0000 7f6c69575700 1 --2- 172.21.15.70:0/3943605960 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6c4c0b29f0 0x7f6c4c0b4eb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.481+0000 7f6c69575700 1 --2- 172.21.15.70:0/3943605960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c64107460 0x7f6c64199aa0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.481+0000 7f6c69575700 1 --2- 172.21.15.70:0/3943605960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c64102720 0x7f6c6419f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.481+0000 7f6c69575700 1 --2- 172.21.15.70:0/3943605960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c641030e0 0x7f6c6419fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:26.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.481+0000 7f6c69575700 1 -- 172.21.15.70:0/3943605960 >> 172.21.15.70:0/3943605960 conn(0x7f6c64079300 msgr2=0x7f6c6406c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:26.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.481+0000 7f6c69575700 1 -- 172.21.15.70:0/3943605960 shutdown_connections 2024-06-04T05:04:26.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:26.481+0000 7f6c69575700 1 -- 172.21.15.70:0/3943605960 wait complete. 2024-06-04T05:04:26.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 117613 1.96 MB/sec execute 199 sec latency 42.661 ms 2024-06-04T05:04:27.288 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:04:27.288 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:04:27.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:27 smithi112 ceph-mon[36453]: pgmap v223: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 2.0 MiB/s wr, 195 op/s 2024-06-04T05:04:27.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:27 smithi098 ceph-mon[35569]: pgmap v223: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 2.0 MiB/s wr, 195 op/s 2024-06-04T05:04:27.545 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:04:27.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:27 smithi070 ceph-mon[32631]: pgmap v223: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 2.0 MiB/s wr, 195 op/s 2024-06-04T05:04:27.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 118004 1.95 MB/sec execute 200 sec latency 19.663 ms 2024-06-04T05:04:28.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 118542 1.96 MB/sec execute 201 sec latency 39.446 ms 2024-06-04T05:04:29.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:29 smithi070 ceph-mon[32631]: pgmap v224: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 1.5 MiB/s wr, 129 op/s 2024-06-04T05:04:29.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:29 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:04:29.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:29 smithi112 ceph-mon[36453]: pgmap v224: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 1.5 MiB/s wr, 129 op/s 2024-06-04T05:04:29.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:29 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:04:29.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.488+0000 7f7381cf3700 1 -- 172.21.15.70:0/1178603663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f737c1095b0 msgr2=0x7f737c109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:29.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.488+0000 7f7381cf3700 1 --2- 172.21.15.70:0/1178603663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f737c1095b0 0x7f737c109990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f736c009990 tx=0x7f736c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:04:29.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.488+0000 7f7381cf3700 1 -- 172.21.15.70:0/1178603663 shutdown_connections 2024-06-04T05:04:29.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.488+0000 7f7381cf3700 1 --2- 172.21.15.70:0/1178603663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737c06bbe0 0x7f737c1103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:29.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.488+0000 7f7381cf3700 1 --2- 172.21.15.70:0/1178603663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f737c1095b0 0x7f737c109990 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:29.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.488+0000 7f7381cf3700 1 --2- 172.21.15.70:0/1178603663 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f737c06b220 0x7f737c06b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:29.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.488+0000 7f7381cf3700 1 -- 172.21.15.70:0/1178603663 >> 172.21.15.70:0/1178603663 conn(0x7f737c0793f0 msgr2=0x7f737c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:29.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.488+0000 7f7381cf3700 1 -- 172.21.15.70:0/1178603663 shutdown_connections 2024-06-04T05:04:29.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.488+0000 7f7381cf3700 1 -- 172.21.15.70:0/1178603663 wait complete. 2024-06-04T05:04:29.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.489+0000 7f7381cf3700 1 Processor -- start 2024-06-04T05:04:29.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.489+0000 7f7381cf3700 1 -- start start 2024-06-04T05:04:29.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f7381cf3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737c06b220 0x7f737c19f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:29.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f7381cf3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f737c06bbe0 0x7f737c19fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:29.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f7381cf3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f737c1095b0 0x7f737c1999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:29.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f7381cf3700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f737c113570 con 0x7f737c06bbe0 2024-06-04T05:04:29.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f7381cf3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f737c1133f0 con 0x7f737c06b220 2024-06-04T05:04:29.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f7381cf3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f737c1136f0 con 0x7f737c1095b0 2024-06-04T05:04:29.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f737affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f737c06bbe0 0x7f737c19fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:29.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f737b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737c06b220 0x7f737c19f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:29.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f737affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f737c06bbe0 0x7f737c19fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54872/0 (socket says 172.21.15.70:54872) 2024-06-04T05:04:29.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f737affd700 1 -- 172.21.15.70:0/3743782877 learned_addr learned my addr 172.21.15.70:0/3743782877 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:29.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f737bfff700 1 --2- 172.21.15.70:0/3743782877 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f737c1095b0 0x7f737c1999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:29.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f737affd700 1 -- 172.21.15.70:0/3743782877 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f737c1095b0 msgr2=0x7f737c1999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:29.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f737affd700 1 --2- 172.21.15.70:0/3743782877 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f737c1095b0 0x7f737c1999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:29.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f737affd700 1 -- 172.21.15.70:0/3743782877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737c06b220 msgr2=0x7f737c19f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:29.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f737affd700 1 --2- 172.21.15.70:0/3743782877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737c06b220 0x7f737c19f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:29.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f737affd700 1 -- 172.21.15.70:0/3743782877 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f737c19a100 con 0x7f737c06bbe0 2024-06-04T05:04:29.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f737bfff700 1 --2- 172.21.15.70:0/3743782877 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f737c1095b0 0x7f737c1999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:29.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f737b7fe700 1 --2- 172.21.15.70:0/3743782877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737c06b220 0x7f737c19f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:29.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f737affd700 1 --2- 172.21.15.70:0/3743782877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f737c06bbe0 0x7f737c19fd20 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f737000e9a0 tx=0x7f737000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:29.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.490+0000 7f7378ff9700 1 -- 172.21.15.70:0/3743782877 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737000ca10 con 0x7f737c06bbe0 2024-06-04T05:04:29.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.491+0000 7f7378ff9700 1 -- 172.21.15.70:0/3743782877 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f737000cbb0 con 0x7f737c06bbe0 2024-06-04T05:04:29.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.491+0000 7f7378ff9700 1 -- 172.21.15.70:0/3743782877 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7370018680 con 0x7f737c06bbe0 2024-06-04T05:04:29.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.491+0000 7f7381cf3700 1 -- 172.21.15.70:0/3743782877 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f737c19a3f0 con 0x7f737c06bbe0 2024-06-04T05:04:29.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.491+0000 7f7381cf3700 1 -- 172.21.15.70:0/3743782877 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f737c1a66b0 con 0x7f737c06bbe0 2024-06-04T05:04:29.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:29 smithi098 ceph-mon[35569]: pgmap v224: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 1.5 MiB/s wr, 129 op/s 2024-06-04T05:04:29.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:29 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:04:29.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.492+0000 7f7381cf3700 1 -- 172.21.15.70:0/3743782877 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f737c10ba90 con 0x7f737c06bbe0 2024-06-04T05:04:29.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.493+0000 7f7378ff9700 1 -- 172.21.15.70:0/3743782877 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7370010430 con 0x7f737c06bbe0 2024-06-04T05:04:29.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.494+0000 7f7378ff9700 1 --2- 172.21.15.70:0/3743782877 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73640b2a70 0x7f73640b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:29.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.494+0000 7f737b7fe700 1 --2- 172.21.15.70:0/3743782877 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73640b2a70 0x7f73640b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:29.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.494+0000 7f7378ff9700 1 -- 172.21.15.70:0/3743782877 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f73700e2740 con 0x7f737c06bbe0 2024-06-04T05:04:29.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.495+0000 7f737b7fe700 1 --2- 172.21.15.70:0/3743782877 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73640b2a70 0x7f73640b4f30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f736c00b2e0 tx=0x7f736c01a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:29.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.500+0000 7f7378ff9700 1 -- 172.21.15.70:0/3743782877 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f73700aa0f0 con 0x7f737c06bbe0 2024-06-04T05:04:29.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 119061 1.96 MB/sec execute 202 sec latency 21.138 ms 2024-06-04T05:04:29.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.799+0000 7f7381cf3700 1 -- 172.21.15.70:0/3743782877 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f737c06a230 con 0x7f737c06bbe0 2024-06-04T05:04:29.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.800+0000 7f7378ff9700 1 -- 172.21.15.70:0/3743782877 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11929 (secure 0 0 0) 0x7f73700b2170 con 0x7f737c06bbe0 2024-06-04T05:04:29.802 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:04:29.802 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:01:52.807483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:04:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.802+0000 7f7381cf3700 1 -- 172.21.15.70:0/3743782877 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73640b2a70 msgr2=0x7f73640b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.802+0000 7f7381cf3700 1 --2- 172.21.15.70:0/3743782877 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73640b2a70 0x7f73640b4f30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f736c00b2e0 tx=0x7f736c01a040 comp rx=0 tx=0).stop 2024-06-04T05:04:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.802+0000 7f7381cf3700 1 -- 172.21.15.70:0/3743782877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f737c06bbe0 msgr2=0x7f737c19fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.802+0000 7f7381cf3700 1 --2- 172.21.15.70:0/3743782877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f737c06bbe0 0x7f737c19fd20 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f737000e9a0 tx=0x7f737000ecb0 comp rx=0 tx=0).stop 2024-06-04T05:04:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.803+0000 7f7381cf3700 1 -- 172.21.15.70:0/3743782877 shutdown_connections 2024-06-04T05:04:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.803+0000 7f7381cf3700 1 --2- 172.21.15.70:0/3743782877 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73640b2a70 0x7f73640b4f30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.803+0000 7f7381cf3700 1 --2- 172.21.15.70:0/3743782877 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f737c1095b0 0x7f737c1999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.803+0000 7f7381cf3700 1 --2- 172.21.15.70:0/3743782877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737c06b220 0x7f737c19f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.803+0000 7f7381cf3700 1 --2- 172.21.15.70:0/3743782877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f737c06bbe0 0x7f737c19fd20 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.803+0000 7f7381cf3700 1 -- 172.21.15.70:0/3743782877 >> 172.21.15.70:0/3743782877 conn(0x7f737c0793f0 msgr2=0x7f737c101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:29.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.803+0000 7f7381cf3700 1 -- 172.21.15.70:0/3743782877 shutdown_connections 2024-06-04T05:04:29.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:29.803+0000 7f7381cf3700 1 -- 172.21.15.70:0/3743782877 wait complete. 2024-06-04T05:04:29.806 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 32 2024-06-04T05:04:30.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:30 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:04:30.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:30 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:04:30.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:30 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:04:30.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:30 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3743782877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:04:30.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:30 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:04:30.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:30 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:04:30.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:30 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:04:30.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:30 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3743782877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:04:30.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:30 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:04:30.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:30 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:04:30.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:30 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:04:30.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:30 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3743782877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:04:30.543 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:04:30.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 119590 1.96 MB/sec execute 203 sec latency 17.814 ms 2024-06-04T05:04:30.802 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:04:31.577 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:31 smithi070 ceph-mon[32631]: pgmap v225: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 1.5 MiB/s wr, 129 op/s 2024-06-04T05:04:31.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:31 smithi112 ceph-mon[36453]: pgmap v225: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 1.5 MiB/s wr, 129 op/s 2024-06-04T05:04:31.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 120378 1.98 MB/sec execute 204 sec latency 15.685 ms 2024-06-04T05:04:31.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:31 smithi098 ceph-mon[35569]: pgmap v225: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 1.5 MiB/s wr, 129 op/s 2024-06-04T05:04:32.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 120785 1.99 MB/sec execute 205 sec latency 43.745 ms 2024-06-04T05:04:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.874+0000 7f65a4e83700 1 -- 172.21.15.70:0/282207101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65a0108520 msgr2=0x7f65a0108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.874+0000 7f65a4e83700 1 --2- 172.21.15.70:0/282207101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65a0108520 0x7f65a0108900 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f6590009990 tx=0x7f6590009ca0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.874+0000 7f65a4e83700 1 -- 172.21.15.70:0/282207101 shutdown_connections 2024-06-04T05:04:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.874+0000 7f65a4e83700 1 --2- 172.21.15.70:0/282207101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65a0102e00 0x7f65a010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.874+0000 7f65a4e83700 1 --2- 172.21.15.70:0/282207101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65a0108520 0x7f65a0108900 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.874+0000 7f65a4e83700 1 --2- 172.21.15.70:0/282207101 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65a0102440 0x7f65a01028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.874+0000 7f65a4e83700 1 -- 172.21.15.70:0/282207101 >> 172.21.15.70:0/282207101 conn(0x7f65a00793f0 msgr2=0x7f65a0079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.874+0000 7f65a4e83700 1 -- 172.21.15.70:0/282207101 shutdown_connections 2024-06-04T05:04:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.874+0000 7f65a4e83700 1 -- 172.21.15.70:0/282207101 wait complete. 2024-06-04T05:04:32.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f65a4e83700 1 Processor -- start 2024-06-04T05:04:32.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f65a4e83700 1 -- start start 2024-06-04T05:04:32.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f65a4e83700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65a0102440 0x7f65a0104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f65a4e83700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65a0102e00 0x7f65a01045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f65a4e83700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65a0108520 0x7f65a0104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f65a4e83700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65a01124a0 con 0x7f65a0102e00 2024-06-04T05:04:32.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f65a4e83700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f65a0112320 con 0x7f65a0102440 2024-06-04T05:04:32.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f65a4e83700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f65a0112620 con 0x7f65a0108520 2024-06-04T05:04:32.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f659dd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65a0102e00 0x7f65a01045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f659dd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65a0102e00 0x7f65a01045c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54888/0 (socket says 172.21.15.70:54888) 2024-06-04T05:04:32.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f659dd9b700 1 -- 172.21.15.70:0/3648155257 learned_addr learned my addr 172.21.15.70:0/3648155257 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:32.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f659e59c700 1 --2- 172.21.15.70:0/3648155257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65a0102440 0x7f65a0104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f659ed9d700 1 --2- 172.21.15.70:0/3648155257 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65a0108520 0x7f65a0104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f659dd9b700 1 -- 172.21.15.70:0/3648155257 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65a0108520 msgr2=0x7f65a0104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:32.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f659dd9b700 1 --2- 172.21.15.70:0/3648155257 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65a0108520 0x7f65a0104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f659dd9b700 1 -- 172.21.15.70:0/3648155257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65a0102440 msgr2=0x7f65a0104080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:32.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f659dd9b700 1 --2- 172.21.15.70:0/3648155257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65a0102440 0x7f65a0104080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.875+0000 7f659dd9b700 1 -- 172.21.15.70:0/3648155257 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a01a80a0 con 0x7f65a0102e00 2024-06-04T05:04:32.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.876+0000 7f659dd9b700 1 --2- 172.21.15.70:0/3648155257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65a0102e00 0x7f65a01045c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f659400e9c0 tx=0x7f659400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:32.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.876+0000 7f658f7fe700 1 -- 172.21.15.70:0/3648155257 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659400cbd0 con 0x7f65a0102e00 2024-06-04T05:04:32.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.876+0000 7f65a4e83700 1 -- 172.21.15.70:0/3648155257 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65a01a8390 con 0x7f65a0102e00 2024-06-04T05:04:32.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.876+0000 7f658f7fe700 1 -- 172.21.15.70:0/3648155257 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f659400ee60 con 0x7f65a0102e00 2024-06-04T05:04:32.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.876+0000 7f658f7fe700 1 -- 172.21.15.70:0/3648155257 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65940054b0 con 0x7f65a0102e00 2024-06-04T05:04:32.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.876+0000 7f65a4e83700 1 -- 172.21.15.70:0/3648155257 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65a01a8880 con 0x7f65a0102e00 2024-06-04T05:04:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.878+0000 7f658f7fe700 1 -- 172.21.15.70:0/3648155257 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f6594004750 con 0x7f65a0102e00 2024-06-04T05:04:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.879+0000 7f658f7fe700 1 --2- 172.21.15.70:0/3648155257 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65880b29f0 0x7f65880b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.879+0000 7f659e59c700 1 --2- 172.21.15.70:0/3648155257 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65880b29f0 0x7f65880b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.879+0000 7f658f7fe700 1 -- 172.21.15.70:0/3648155257 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f65940e2980 con 0x7f65a0102e00 2024-06-04T05:04:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.880+0000 7f659e59c700 1 --2- 172.21.15.70:0/3648155257 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65880b29f0 0x7f65880b4eb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f659000b2e0 tx=0x7f6590005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.880+0000 7f65af718b80 1 Processor -- start 2024-06-04T05:04:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.880+0000 7f65af718b80 1 -- start start 2024-06-04T05:04:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.880+0000 7f65af718b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f4860 0x5629d49f4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.880+0000 7f65af718b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d49f8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.880+0000 7f65af718b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f8f60 0x5629d49f93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.880+0000 7f65af718b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5629d4953c90 con 0x5629d49f5210 2024-06-04T05:04:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.880+0000 7f65af718b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5629d4943830 con 0x5629d49f8f60 2024-06-04T05:04:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.880+0000 7f65af718b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5629d4947870 con 0x5629d49f4860 2024-06-04T05:04:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.881+0000 7f659dd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d49f8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.881+0000 7f659dd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d49f8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54892/0 (socket says 172.21.15.70:54892) 2024-06-04T05:04:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.881+0000 7f659e59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f4860 0x5629d49f4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.881+0000 7f659dd9b700 1 -- 172.21.15.70:0/380259701 learned_addr learned my addr 172.21.15.70:0/380259701 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.881+0000 7f659ed9d700 1 --2- 172.21.15.70:0/380259701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f8f60 0x5629d49f93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.881+0000 7f659e59c700 1 -- 172.21.15.70:0/380259701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f8f60 msgr2=0x5629d49f93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.881+0000 7f659e59c700 1 --2- 172.21.15.70:0/380259701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f8f60 0x5629d49f93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.881+0000 7f659e59c700 1 -- 172.21.15.70:0/380259701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 msgr2=0x5629d49f8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.881+0000 7f659e59c700 1 --2- 172.21.15.70:0/380259701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d49f8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.881+0000 7f659e59c700 1 -- 172.21.15.70:0/380259701 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629d48938d0 con 0x5629d49f4860 2024-06-04T05:04:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.881+0000 7f659dd9b700 1 --2- 172.21.15.70:0/380259701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d49f8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.881+0000 7f659e59c700 1 --2- 172.21.15.70:0/380259701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f4860 0x5629d49f4c40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f659000b650 tx=0x7f659001f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3c4002122bba1b6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.881+0000 7f658cff9700 1 -- 172.21.15.70:0/380259701 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659000b6b0 con 0x5629d49f4860 2024-06-04T05:04:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.881+0000 7f658cff9700 1 -- 172.21.15.70:0/380259701 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f659000bce0 con 0x5629d49f4860 2024-06-04T05:04:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.881+0000 7f658cff9700 1 -- 172.21.15.70:0/380259701 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6590025c50 con 0x5629d49f4860 2024-06-04T05:04:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.882+0000 7f65af718b80 1 -- 172.21.15.70:0/380259701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f4860 msgr2=0x5629d49f4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.882+0000 7f65af718b80 1 --2- 172.21.15.70:0/380259701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f4860 0x5629d49f4c40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f659000b650 tx=0x7f659001f000 comp rx=0 tx=0).stop 2024-06-04T05:04:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.882+0000 7f65af718b80 1 -- 172.21.15.70:0/380259701 shutdown_connections 2024-06-04T05:04:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.882+0000 7f65af718b80 1 --2- 172.21.15.70:0/380259701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f8f60 0x5629d49f93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.882+0000 7f65af718b80 1 --2- 172.21.15.70:0/380259701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f4860 0x5629d49f4c40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.882+0000 7f65af718b80 1 --2- 172.21.15.70:0/380259701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d49f8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.882+0000 7f65af718b80 1 -- 172.21.15.70:0/380259701 >> 172.21.15.70:0/380259701 conn(0x5629d4922080 msgr2=0x5629d49eee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.882+0000 7f65af718b80 1 -- 172.21.15.70:0/380259701 shutdown_connections 2024-06-04T05:04:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.882+0000 7f65af718b80 1 -- 172.21.15.70:0/380259701 wait complete. 2024-06-04T05:04:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f65af718b80 1 -- start start 2024-06-04T05:04:32.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f65af718b80 1 Processor -- start 2024-06-04T05:04:32.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f65af718b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f4860 0x5629d4a00390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f65af718b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d49fd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f65af718b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f8f60 0x5629d49fdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f65af718b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5629d4849fa0 con 0x5629d49f5210 2024-06-04T05:04:32.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f65af718b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5629d48e57c0 con 0x5629d49f8f60 2024-06-04T05:04:32.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f65af718b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5629d4a010c0 con 0x5629d49f4860 2024-06-04T05:04:32.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f65af718b80 10 client.0 _ll_register_callbacks cb 0x5629d48de920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:32.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f659dd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d49fd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f659e59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f4860 0x5629d4a00390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f659dd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d49fd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54902/0 (socket says 172.21.15.70:54902) 2024-06-04T05:04:32.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f659dd9b700 1 -- 172.21.15.70:0/196374709 learned_addr learned my addr 172.21.15.70:0/196374709 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:32.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f659ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f8f60 0x5629d49fdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f659e59c700 1 -- 172.21.15.70:0/196374709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f8f60 msgr2=0x5629d49fdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:32.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f659e59c700 1 --2- 172.21.15.70:0/196374709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f8f60 0x5629d49fdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f659e59c700 1 -- 172.21.15.70:0/196374709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 msgr2=0x5629d49fd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:32.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f659e59c700 1 --2- 172.21.15.70:0/196374709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d49fd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f659e59c700 1 -- 172.21.15.70:0/196374709 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629d48e2690 con 0x5629d49f4860 2024-06-04T05:04:32.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.883+0000 7f659dd9b700 1 --2- 172.21.15.70:0/196374709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d49fd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:32.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.884+0000 7f659e59c700 1 --2- 172.21.15.70:0/196374709 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f4860 0x5629d4a00390 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f659000b650 tx=0x7f65900256a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ebf2b3fb664a1b1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:32.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.884+0000 7f6585ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:04:32.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.884+0000 7f6585ffb700 1 -- 172.21.15.70:0/196374709 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659000be90 con 0x5629d49f4860 2024-06-04T05:04:32.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.884+0000 7f6585ffb700 1 -- 172.21.15.70:0/196374709 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f6590003dc0 con 0x5629d49f4860 2024-06-04T05:04:32.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.884+0000 7f65af718b80 1 -- 172.21.15.70:0/196374709 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f659000bce0 con 0x5629d49f4860 2024-06-04T05:04:32.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.884+0000 7f6585ffb700 1 -- 172.21.15.70:0/196374709 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6590025ad0 con 0x5629d49f4860 2024-06-04T05:04:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.884+0000 7f65af718b80 10 client.26701 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.884+0000 7f65af718b80 1 -- 172.21.15.70:0/196374709 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5629d4a0b7d0 con 0x5629d49f4860 2024-06-04T05:04:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.884+0000 7f6585ffb700 1 -- 172.21.15.70:0/196374709 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f659000bce0 con 0x5629d49f4860 2024-06-04T05:04:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.884+0000 7f6585ffb700 1 -- 172.21.15.70:0/196374709 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f659002e020 con 0x5629d49f4860 2024-06-04T05:04:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.885+0000 7f65af718b80 10 client.26701 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.885+0000 7f65af718b80 10 client.26701 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:04:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.885+0000 7f65af718b80 20 client.26701 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.885+0000 7f65af718b80 1 --2- 172.21.15.70:0/196374709 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5629d4a0dca0 0x5629d4a10090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.885+0000 7f65af718b80 4 client.26701 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:04:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.885+0000 7f65af718b80 1 -- 172.21.15.70:0/196374709 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629d49c09e0 con 0x5629d4a0dca0 2024-06-04T05:04:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.885+0000 7f659dd9b700 1 --2- 172.21.15.70:0/196374709 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5629d4a0dca0 0x5629d4a10090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.885+0000 7f659dd9b700 1 --2- 172.21.15.70:0/196374709 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5629d4a0dca0 0x5629d4a10090 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cc76179308366bb5 server_cookie=98b214dac428807a in_seq=0 out_seq=0 2024-06-04T05:04:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.885+0000 7f6585ffb700 10 client.26701 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:04:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.886+0000 7f6585ffb700 1 -- 172.21.15.70:0/196374709 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5629d49c09e0 con 0x5629d4a0dca0 2024-06-04T05:04:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.886+0000 7f6585ffb700 10 client.26701 handle_command_reply: tid=0 2024-06-04T05:04:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.886+0000 7f65af718b80 1 client.26701 shutdown 2024-06-04T05:04:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.886+0000 7f65af718b80 1 -- 172.21.15.70:0/196374709 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f4860 msgr2=0x5629d4a00390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.886+0000 7f65af718b80 1 --2- 172.21.15.70:0/196374709 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f4860 0x5629d4a00390 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f659000b650 tx=0x7f65900256a0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.886+0000 7f65af718b80 1 -- 172.21.15.70:0/196374709 shutdown_connections 2024-06-04T05:04:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.886+0000 7f65af718b80 1 --2- 172.21.15.70:0/196374709 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5629d4a0dca0 0x5629d4a10090 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.886+0000 7f65af718b80 1 --2- 172.21.15.70:0/196374709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f8f60 0x5629d49fdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.886+0000 7f65af718b80 1 --2- 172.21.15.70:0/196374709 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f4860 0x5629d4a00390 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.886+0000 7f65af718b80 1 --2- 172.21.15.70:0/196374709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d49fd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.886+0000 7f6585ffb700 0 client.26701 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:04:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.886+0000 7f65af718b80 1 -- 172.21.15.70:0/196374709 >> 172.21.15.70:0/196374709 conn(0x5629d4922080 msgr2=0x5629d49ee510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.886+0000 7f65af718b80 1 -- 172.21.15.70:0/196374709 shutdown_connections 2024-06-04T05:04:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.886+0000 7f65af718b80 1 -- 172.21.15.70:0/196374709 wait complete. 2024-06-04T05:04:32.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.886+0000 7f65af718b80 20 client.26701 trim_cache size 0 max 16384 2024-06-04T05:04:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f65af718b80 1 Processor -- start 2024-06-04T05:04:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f65af718b80 1 -- start start 2024-06-04T05:04:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f65af718b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f4860 0x5629d49ff5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f65af718b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f5210 0x5629d49eec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f65af718b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f8f60 0x5629d49ef1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f65af718b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65a01b0000 con 0x5629d49f4860 2024-06-04T05:04:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f65af718b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f65a01b1b40 con 0x5629d49f5210 2024-06-04T05:04:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f65af718b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f65a01b06d0 con 0x5629d49f8f60 2024-06-04T05:04:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f659e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f4860 0x5629d49ff5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f659e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f4860 0x5629d49ff5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54910/0 (socket says 172.21.15.70:54910) 2024-06-04T05:04:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f659e59c700 1 -- 172.21.15.70:0/1618537493 learned_addr learned my addr 172.21.15.70:0/1618537493 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f659ed9d700 1 --2- 172.21.15.70:0/1618537493 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f8f60 0x5629d49ef1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f659dd9b700 1 --2- 172.21.15.70:0/1618537493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f5210 0x5629d49eec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f659e59c700 1 -- 172.21.15.70:0/1618537493 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f8f60 msgr2=0x5629d49ef1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f659e59c700 1 --2- 172.21.15.70:0/1618537493 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f8f60 0x5629d49ef1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f659e59c700 1 -- 172.21.15.70:0/1618537493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f5210 msgr2=0x5629d49eec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f659e59c700 1 --2- 172.21.15.70:0/1618537493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f5210 0x5629d49eec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.903+0000 7f659e59c700 1 -- 172.21.15.70:0/1618537493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629d49eff70 con 0x5629d49f4860 2024-06-04T05:04:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.904+0000 7f659e59c700 1 --2- 172.21.15.70:0/1618537493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f4860 0x5629d49ff5d0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6590009350 tx=0x7f65900096f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac423f53b1a7994 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.904+0000 7f6587fff700 1 -- 172.21.15.70:0/1618537493 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6590046070 con 0x5629d49f4860 2024-06-04T05:04:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.904+0000 7f6587fff700 1 -- 172.21.15.70:0/1618537493 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6590005bc0 con 0x5629d49f4860 2024-06-04T05:04:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.904+0000 7f6587fff700 1 -- 172.21.15.70:0/1618537493 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659002a020 con 0x5629d49f4860 2024-06-04T05:04:32.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.905+0000 7f65af718b80 1 -- 172.21.15.70:0/1618537493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f4860 msgr2=0x5629d49ff5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:32.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.905+0000 7f65af718b80 1 --2- 172.21.15.70:0/1618537493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f4860 0x5629d49ff5d0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6590009350 tx=0x7f65900096f0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.905+0000 7f65af718b80 1 -- 172.21.15.70:0/1618537493 shutdown_connections 2024-06-04T05:04:32.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.905+0000 7f65af718b80 1 --2- 172.21.15.70:0/1618537493 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f8f60 0x5629d49ef1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.905+0000 7f65af718b80 1 --2- 172.21.15.70:0/1618537493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f4860 0x5629d49ff5d0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.905+0000 7f65af718b80 1 --2- 172.21.15.70:0/1618537493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f5210 0x5629d49eec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.905+0000 7f65af718b80 1 -- 172.21.15.70:0/1618537493 >> 172.21.15.70:0/1618537493 conn(0x5629d4922080 msgr2=0x5629d49feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:32.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.905+0000 7f65af718b80 1 -- 172.21.15.70:0/1618537493 shutdown_connections 2024-06-04T05:04:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.905+0000 7f65af718b80 1 -- 172.21.15.70:0/1618537493 wait complete. 2024-06-04T05:04:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.905+0000 7f65af718b80 1 -- start start 2024-06-04T05:04:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.905+0000 7f65af718b80 1 Processor -- start 2024-06-04T05:04:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f65af718b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f4860 0x5629d4a126b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f65af718b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d4a12bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f65af718b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f8f60 0x5629d4a78d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f65af718b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65a01b1470 con 0x5629d49f5210 2024-06-04T05:04:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f65af718b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f65a01b0da0 con 0x5629d49f4860 2024-06-04T05:04:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f65af718b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f65a01adfc0 con 0x5629d49f8f60 2024-06-04T05:04:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f65af718b80 10 client.0 _ll_register_callbacks cb 0x5629d48de920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f659dd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d4a12bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f659dd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d4a12bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54916/0 (socket says 172.21.15.70:54916) 2024-06-04T05:04:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f659dd9b700 1 -- 172.21.15.70:0/2276627204 learned_addr learned my addr 172.21.15.70:0/2276627204 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f659ed9d700 1 --2- 172.21.15.70:0/2276627204 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f8f60 0x5629d4a78d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f659dd9b700 1 -- 172.21.15.70:0/2276627204 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f8f60 msgr2=0x5629d4a78d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f659e59c700 1 --2- 172.21.15.70:0/2276627204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f4860 0x5629d4a126b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f659dd9b700 1 --2- 172.21.15.70:0/2276627204 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f8f60 0x5629d4a78d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f659dd9b700 1 -- 172.21.15.70:0/2276627204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f4860 msgr2=0x5629d4a126b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f659dd9b700 1 --2- 172.21.15.70:0/2276627204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f4860 0x5629d4a126b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f659dd9b700 1 -- 172.21.15.70:0/2276627204 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629d4a792f0 con 0x5629d49f5210 2024-06-04T05:04:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f659ed9d700 1 --2- 172.21.15.70:0/2276627204 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f8f60 0x5629d4a78d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:04:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f659e59c700 1 --2- 172.21.15.70:0/2276627204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f4860 0x5629d4a126b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:04:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f659dd9b700 1 --2- 172.21.15.70:0/2276627204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d4a12bf0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f65940a5c50 tx=0x7f65940097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66aea6e80de6a8a2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.906+0000 7f6585ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:04:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.907+0000 7f6585ffb700 1 -- 172.21.15.70:0/2276627204 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65940c42c0 con 0x5629d49f5210 2024-06-04T05:04:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.907+0000 7f6585ffb700 1 -- 172.21.15.70:0/2276627204 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f65940a9e30 con 0x5629d49f5210 2024-06-04T05:04:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.907+0000 7f65af718b80 1 -- 172.21.15.70:0/2276627204 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6590005bc0 con 0x5629d49f5210 2024-06-04T05:04:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.907+0000 7f6585ffb700 1 -- 172.21.15.70:0/2276627204 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6594016540 con 0x5629d49f5210 2024-06-04T05:04:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.907+0000 7f65af718b80 10 client.17412 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.907+0000 7f65af718b80 1 -- 172.21.15.70:0/2276627204 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5629d4a806a0 con 0x5629d49f5210 2024-06-04T05:04:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.907+0000 7f6585ffb700 1 -- 172.21.15.70:0/2276627204 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6590005bc0 con 0x5629d49f5210 2024-06-04T05:04:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.907+0000 7f6585ffb700 1 -- 172.21.15.70:0/2276627204 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f65940a5040 con 0x5629d49f5210 2024-06-04T05:04:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.907+0000 7f65af718b80 10 client.17412 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.907+0000 7f65af718b80 10 client.17412 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:04:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.907+0000 7f65af718b80 20 client.17412 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.907+0000 7f65af718b80 1 --2- 172.21.15.70:0/2276627204 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5629d4a80d90 0x5629d4a831b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.907+0000 7f65af718b80 4 client.17412 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:04:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.907+0000 7f65af718b80 1 -- 172.21.15.70:0/2276627204 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5629d4955be0 con 0x5629d4a80d90 2024-06-04T05:04:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.908+0000 7f659e59c700 1 --2- 172.21.15.70:0/2276627204 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5629d4a80d90 0x5629d4a831b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.908+0000 7f659e59c700 1 --2- 172.21.15.70:0/2276627204 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5629d4a80d90 0x5629d4a831b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1f05965cd38f6f5b server_cookie=23675b0d1f78a05f in_seq=0 out_seq=0 2024-06-04T05:04:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.908+0000 7f6585ffb700 10 client.17412 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:04:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.908+0000 7f6585ffb700 1 -- 172.21.15.70:0/2276627204 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5629d4955be0 con 0x5629d4a80d90 2024-06-04T05:04:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.908+0000 7f6585ffb700 10 client.17412 handle_command_reply: tid=0 2024-06-04T05:04:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.908+0000 7f65af718b80 1 client.17412 shutdown 2024-06-04T05:04:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.908+0000 7f65af718b80 1 -- 172.21.15.70:0/2276627204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 msgr2=0x5629d4a12bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.908+0000 7f65af718b80 1 --2- 172.21.15.70:0/2276627204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d4a12bf0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f65940a5c50 tx=0x7f65940097b0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.909+0000 7f65af718b80 1 -- 172.21.15.70:0/2276627204 shutdown_connections 2024-06-04T05:04:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.909+0000 7f65af718b80 1 --2- 172.21.15.70:0/2276627204 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x5629d4a80d90 0x5629d4a831b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.909+0000 7f65af718b80 1 --2- 172.21.15.70:0/2276627204 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5629d49f8f60 0x5629d4a78d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.909+0000 7f6585ffb700 0 client.17412 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:04:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.909+0000 7f65af718b80 1 --2- 172.21.15.70:0/2276627204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5629d49f4860 0x5629d4a126b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.909+0000 7f65af718b80 1 --2- 172.21.15.70:0/2276627204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629d49f5210 0x5629d4a12bf0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.909+0000 7f65af718b80 1 -- 172.21.15.70:0/2276627204 >> 172.21.15.70:0/2276627204 conn(0x5629d4922080 msgr2=0x5629d49f3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.909+0000 7f65af718b80 1 -- 172.21.15.70:0/2276627204 shutdown_connections 2024-06-04T05:04:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.909+0000 7f65af718b80 1 -- 172.21.15.70:0/2276627204 wait complete. 2024-06-04T05:04:32.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.909+0000 7f65af718b80 20 client.17412 trim_cache size 0 max 16384 2024-06-04T05:04:32.916 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:04:32.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.909+0000 7f65a4e83700 1 -- 172.21.15.70:0/3648155257 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65880b29f0 msgr2=0x7f65880b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.909+0000 7f65a4e83700 1 --2- 172.21.15.70:0/3648155257 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65880b29f0 0x7f65880b4eb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f659000b2e0 tx=0x7f6590005fb0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.909+0000 7f65a4e83700 1 -- 172.21.15.70:0/3648155257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65a0102e00 msgr2=0x7f65a01045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.909+0000 7f65a4e83700 1 --2- 172.21.15.70:0/3648155257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65a0102e00 0x7f65a01045c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f659400e9c0 tx=0x7f659400ecd0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.910+0000 7f65a4e83700 1 -- 172.21.15.70:0/3648155257 shutdown_connections 2024-06-04T05:04:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.910+0000 7f65a4e83700 1 --2- 172.21.15.70:0/3648155257 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65880b29f0 0x7f65880b4eb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.910+0000 7f65a4e83700 1 --2- 172.21.15.70:0/3648155257 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65a0108520 0x7f65a0104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.910+0000 7f65a4e83700 1 --2- 172.21.15.70:0/3648155257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65a0102440 0x7f65a0104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.910+0000 7f65a4e83700 1 --2- 172.21.15.70:0/3648155257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65a0102e00 0x7f65a01045c0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.910+0000 7f65a4e83700 1 -- 172.21.15.70:0/3648155257 >> 172.21.15.70:0/3648155257 conn(0x7f65a00793f0 msgr2=0x7f65a006b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.910+0000 7f65a4e83700 1 -- 172.21.15.70:0/3648155257 shutdown_connections 2024-06-04T05:04:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:32.910+0000 7f65a4e83700 1 -- 172.21.15.70:0/3648155257 wait complete. 2024-06-04T05:04:33.587 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:33 smithi070 ceph-mon[32631]: pgmap v226: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 2.0 MiB/s wr, 190 op/s 2024-06-04T05:04:33.613 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:04:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:33 smithi112 ceph-mon[36453]: pgmap v226: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 2.0 MiB/s wr, 190 op/s 2024-06-04T05:04:33.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 121070 1.98 MB/sec execute 206 sec latency 33.101 ms 2024-06-04T05:04:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:33 smithi098 ceph-mon[35569]: pgmap v226: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 2.0 MiB/s wr, 190 op/s 2024-06-04T05:04:33.872 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:04:34.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 121462 1.98 MB/sec execute 207 sec latency 15.692 ms 2024-06-04T05:04:35.515 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:35 smithi070 ceph-mon[32631]: pgmap v227: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-06-04T05:04:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:35 smithi112 ceph-mon[36453]: pgmap v227: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-06-04T05:04:35.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 122043 1.99 MB/sec execute 208 sec latency 16.568 ms 2024-06-04T05:04:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:35 smithi098 ceph-mon[35569]: pgmap v227: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-06-04T05:04:35.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.757+0000 7fbf58958700 1 -- 172.21.15.70:0/1932443084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf541035d0 msgr2=0x7fbf54103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:35.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.757+0000 7fbf58958700 1 --2- 172.21.15.70:0/1932443084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf541035d0 0x7fbf54103a50 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fbf480099c0 tx=0x7fbf48009cd0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.757+0000 7fbf58958700 1 -- 172.21.15.70:0/1932443084 shutdown_connections 2024-06-04T05:04:35.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.757+0000 7fbf58958700 1 --2- 172.21.15.70:0/1932443084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf54103f90 0x7fbf54110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.757+0000 7fbf58958700 1 --2- 172.21.15.70:0/1932443084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf54109670 0x7fbf54109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.757+0000 7fbf58958700 1 --2- 172.21.15.70:0/1932443084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf541035d0 0x7fbf54103a50 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.757+0000 7fbf58958700 1 -- 172.21.15.70:0/1932443084 >> 172.21.15.70:0/1932443084 conn(0x7fbf5407a730 msgr2=0x7fbf5407ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:35.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.757+0000 7fbf58958700 1 -- 172.21.15.70:0/1932443084 shutdown_connections 2024-06-04T05:04:35.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.757+0000 7fbf58958700 1 -- 172.21.15.70:0/1932443084 wait complete. 2024-06-04T05:04:35.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.758+0000 7fbf58958700 1 Processor -- start 2024-06-04T05:04:35.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.758+0000 7fbf58958700 1 -- start start 2024-06-04T05:04:35.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.758+0000 7fbf58958700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf541035d0 0x7fbf5419f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.758+0000 7fbf58958700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf54103f90 0x7fbf5419fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.758+0000 7fbf58958700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf54109670 0x7fbf541999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.758+0000 7fbf58958700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf541136d0 con 0x7fbf54103f90 2024-06-04T05:04:35.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.758+0000 7fbf58958700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf54113550 con 0x7fbf54109670 2024-06-04T05:04:35.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.758+0000 7fbf58958700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbf54113850 con 0x7fbf541035d0 2024-06-04T05:04:35.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.758+0000 7fbf51d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf54103f90 0x7fbf5419fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.758+0000 7fbf5259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf541035d0 0x7fbf5419f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.758+0000 7fbf51d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf54103f90 0x7fbf5419fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54926/0 (socket says 172.21.15.70:54926) 2024-06-04T05:04:35.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.758+0000 7fbf51d9b700 1 -- 172.21.15.70:0/2128714848 learned_addr learned my addr 172.21.15.70:0/2128714848 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:35.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.758+0000 7fbf52d9d700 1 --2- 172.21.15.70:0/2128714848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf54109670 0x7fbf541999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.758+0000 7fbf5259c700 1 -- 172.21.15.70:0/2128714848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf54109670 msgr2=0x7fbf541999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:35.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.758+0000 7fbf5259c700 1 --2- 172.21.15.70:0/2128714848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf54109670 0x7fbf541999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.759+0000 7fbf5259c700 1 -- 172.21.15.70:0/2128714848 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf54103f90 msgr2=0x7fbf5419fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:35.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.759+0000 7fbf5259c700 1 --2- 172.21.15.70:0/2128714848 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf54103f90 0x7fbf5419fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.759+0000 7fbf5259c700 1 -- 172.21.15.70:0/2128714848 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf5419a150 con 0x7fbf541035d0 2024-06-04T05:04:35.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.759+0000 7fbf51d9b700 1 --2- 172.21.15.70:0/2128714848 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf54103f90 0x7fbf5419fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:35.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.759+0000 7fbf52d9d700 1 --2- 172.21.15.70:0/2128714848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf54109670 0x7fbf541999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:04:35.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.759+0000 7fbf5259c700 1 --2- 172.21.15.70:0/2128714848 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf541035d0 0x7fbf5419f830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400e9f0 tx=0x7fbf4400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:35.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.759+0000 7fbf437fe700 1 -- 172.21.15.70:0/2128714848 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4400cbd0 con 0x7fbf541035d0 2024-06-04T05:04:35.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.759+0000 7fbf437fe700 1 -- 172.21.15.70:0/2128714848 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf4400cd70 con 0x7fbf541035d0 2024-06-04T05:04:35.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.759+0000 7fbf58958700 1 -- 172.21.15.70:0/2128714848 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf5419a440 con 0x7fbf541035d0 2024-06-04T05:04:35.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.759+0000 7fbf437fe700 1 -- 172.21.15.70:0/2128714848 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf44018680 con 0x7fbf541035d0 2024-06-04T05:04:35.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.759+0000 7fbf58958700 1 -- 172.21.15.70:0/2128714848 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf541a6700 con 0x7fbf541035d0 2024-06-04T05:04:35.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.762+0000 7fbf437fe700 1 -- 172.21.15.70:0/2128714848 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fbf44010430 con 0x7fbf541035d0 2024-06-04T05:04:35.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.763+0000 7fbf437fe700 1 --2- 172.21.15.70:0/2128714848 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf3c0b29f0 0x7fbf3c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.763+0000 7fbf51d9b700 1 --2- 172.21.15.70:0/2128714848 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf3c0b29f0 0x7fbf3c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.763+0000 7fbf437fe700 1 -- 172.21.15.70:0/2128714848 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fbf440e2820 con 0x7fbf541035d0 2024-06-04T05:04:35.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.763+0000 7fbf631edb80 1 Processor -- start 2024-06-04T05:04:35.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.764+0000 7fbf631edb80 1 -- start start 2024-06-04T05:04:35.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.764+0000 7fbf51d9b700 1 --2- 172.21.15.70:0/2128714848 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf3c0b29f0 0x7fbf3c0b4eb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbf5407ae70 tx=0x7fbf48005fd0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:35.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.764+0000 7fbf631edb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 0x55d3810f6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.764+0000 7fbf631edb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f0cb0 0x55d3810f1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.764+0000 7fbf631edb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 0x55d3810fd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.764+0000 7fbf631edb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d38104ad70 con 0x55d3810f66c0 2024-06-04T05:04:35.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.764+0000 7fbf631edb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d38103a910 con 0x55d3810f0cb0 2024-06-04T05:04:35.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.764+0000 7fbf631edb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d38103e950 con 0x55d3810f1670 2024-06-04T05:04:35.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.764+0000 7fbf5259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 0x55d3810f6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.764+0000 7fbf52d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 0x55d3810fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.764+0000 7fbf52d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 0x55d3810fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:37992/0 (socket says 172.21.15.70:37992) 2024-06-04T05:04:35.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.764+0000 7fbf51d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f0cb0 0x55d3810f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.764+0000 7fbf5259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 0x55d3810f6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54936/0 (socket says 172.21.15.70:54936) 2024-06-04T05:04:35.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.764+0000 7fbf52d9d700 1 -- 172.21.15.70:0/3906786921 learned_addr learned my addr 172.21.15.70:0/3906786921 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:35.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.765+0000 7fbf51d9b700 1 -- 172.21.15.70:0/3906786921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 msgr2=0x55d3810fd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:35.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.765+0000 7fbf51d9b700 1 --2- 172.21.15.70:0/3906786921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 0x55d3810fd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.765+0000 7fbf51d9b700 1 -- 172.21.15.70:0/3906786921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 msgr2=0x55d3810f6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.765+0000 7fbf51d9b700 1 --2- 172.21.15.70:0/3906786921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 0x55d3810f6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.765+0000 7fbf51d9b700 1 -- 172.21.15.70:0/3906786921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d380fe3fb0 con 0x55d3810f0cb0 2024-06-04T05:04:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.765+0000 7fbf5259c700 1 --2- 172.21.15.70:0/3906786921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 0x55d3810f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.765+0000 7fbf51d9b700 1 --2- 172.21.15.70:0/3906786921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f0cb0 0x55d3810f1130 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800a010 tx=0x7fbf48020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc79a5843ae07aea server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.765+0000 7fbf40ff9700 1 -- 172.21.15.70:0/3906786921 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4800be10 con 0x55d3810f0cb0 2024-06-04T05:04:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.765+0000 7fbf40ff9700 1 -- 172.21.15.70:0/3906786921 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf48003c50 con 0x55d3810f0cb0 2024-06-04T05:04:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.765+0000 7fbf40ff9700 1 -- 172.21.15.70:0/3906786921 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf48024ab0 con 0x55d3810f0cb0 2024-06-04T05:04:35.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.766+0000 7fbf631edb80 1 -- 172.21.15.70:0/3906786921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f0cb0 msgr2=0x55d3810f1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:35.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.766+0000 7fbf631edb80 1 --2- 172.21.15.70:0/3906786921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f0cb0 0x55d3810f1130 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800a010 tx=0x7fbf48020000 comp rx=0 tx=0).stop 2024-06-04T05:04:35.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.766+0000 7fbf631edb80 1 -- 172.21.15.70:0/3906786921 shutdown_connections 2024-06-04T05:04:35.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.766+0000 7fbf631edb80 1 --2- 172.21.15.70:0/3906786921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 0x55d3810fd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.766+0000 7fbf631edb80 1 --2- 172.21.15.70:0/3906786921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 0x55d3810f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.766+0000 7fbf631edb80 1 --2- 172.21.15.70:0/3906786921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f0cb0 0x55d3810f1130 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.766+0000 7fbf631edb80 1 -- 172.21.15.70:0/3906786921 >> 172.21.15.70:0/3906786921 conn(0x55d381018240 msgr2=0x55d3810e73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:35.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.766+0000 7fbf631edb80 1 -- 172.21.15.70:0/3906786921 shutdown_connections 2024-06-04T05:04:35.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.766+0000 7fbf631edb80 1 -- 172.21.15.70:0/3906786921 wait complete. 2024-06-04T05:04:35.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.766+0000 7fbf631edb80 1 -- start start 2024-06-04T05:04:35.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf631edb80 1 Processor -- start 2024-06-04T05:04:35.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf631edb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f0cb0 0x55d3810fd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf631edb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f1670 0x55d3810fa370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf631edb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f66c0 0x55d3810fa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf631edb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d380f40fa0 con 0x55d3810f1670 2024-06-04T05:04:35.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf631edb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d380fa2da0 con 0x55d3810f66c0 2024-06-04T05:04:35.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf631edb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d3810fda90 con 0x55d3810f0cb0 2024-06-04T05:04:35.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf631edb80 10 client.0 _ll_register_callbacks cb 0x55d380fd5950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:35.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf51d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f1670 0x55d3810fa370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf51d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f1670 0x55d3810fa370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54946/0 (socket says 172.21.15.70:54946) 2024-06-04T05:04:35.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf51d9b700 1 -- 172.21.15.70:0/2361934956 learned_addr learned my addr 172.21.15.70:0/2361934956 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:35.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf5259c700 1 --2- 172.21.15.70:0/2361934956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f0cb0 0x55d3810fd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf52d9d700 1 --2- 172.21.15.70:0/2361934956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f66c0 0x55d3810fa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf51d9b700 1 -- 172.21.15.70:0/2361934956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f0cb0 msgr2=0x55d3810fd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:35.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf51d9b700 1 --2- 172.21.15.70:0/2361934956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f0cb0 0x55d3810fd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf51d9b700 1 -- 172.21.15.70:0/2361934956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f66c0 msgr2=0x55d3810fa8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:35.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf51d9b700 1 --2- 172.21.15.70:0/2361934956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f66c0 0x55d3810fa8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.767+0000 7fbf51d9b700 1 -- 172.21.15.70:0/2361934956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d380f8a8d0 con 0x55d3810f1670 2024-06-04T05:04:35.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.768+0000 7fbf51d9b700 1 --2- 172.21.15.70:0/2361934956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f1670 0x55d3810fa370 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009d10 tx=0x7fbf48003e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684f9436dfde05a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:35.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.768+0000 7fbf39ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:04:35.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.768+0000 7fbf39ffb700 1 -- 172.21.15.70:0/2361934956 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4803c060 con 0x55d3810f1670 2024-06-04T05:04:35.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.768+0000 7fbf631edb80 1 -- 172.21.15.70:0/2361934956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf48003c50 con 0x55d3810f1670 2024-06-04T05:04:35.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.768+0000 7fbf39ffb700 1 -- 172.21.15.70:0/2361934956 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fbf48024750 con 0x55d3810f1670 2024-06-04T05:04:35.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.768+0000 7fbf39ffb700 1 -- 172.21.15.70:0/2361934956 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbf48024990 con 0x55d3810f1670 2024-06-04T05:04:35.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.768+0000 7fbf631edb80 10 client.17424 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:35.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.768+0000 7fbf631edb80 1 -- 172.21.15.70:0/2361934956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d380ff4760 con 0x55d3810f1670 2024-06-04T05:04:35.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.768+0000 7fbf39ffb700 1 -- 172.21.15.70:0/2361934956 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf48003c50 con 0x55d3810f1670 2024-06-04T05:04:35.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.769+0000 7fbf39ffb700 1 -- 172.21.15.70:0/2361934956 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fbf4802f400 con 0x55d3810f1670 2024-06-04T05:04:35.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.769+0000 7fbf631edb80 10 client.17424 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:35.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.769+0000 7fbf631edb80 10 client.17424 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:04:35.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.769+0000 7fbf631edb80 20 client.17424 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:35.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.769+0000 7fbf631edb80 1 --2- 172.21.15.70:0/2361934956 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d38110a410 0x55d38110c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.769+0000 7fbf631edb80 4 client.17424 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:04:35.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.769+0000 7fbf631edb80 1 -- 172.21.15.70:0/2361934956 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d38104ccc0 con 0x55d38110a410 2024-06-04T05:04:35.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.769+0000 7fbf5259c700 1 --2- 172.21.15.70:0/2361934956 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d38110a410 0x55d38110c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.769+0000 7fbf5259c700 1 --2- 172.21.15.70:0/2361934956 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d38110a410 0x55d38110c800 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=be5a5c90c37fc55c server_cookie=afc81603331ca75d in_seq=0 out_seq=0 2024-06-04T05:04:35.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.769+0000 7fbf39ffb700 10 client.17424 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:04:35.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.770+0000 7fbf39ffb700 1 -- 172.21.15.70:0/2361934956 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d38104ccc0 con 0x55d38110a410 2024-06-04T05:04:35.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.770+0000 7fbf39ffb700 10 client.17424 handle_command_reply: tid=0 2024-06-04T05:04:35.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.770+0000 7fbf631edb80 1 client.17424 shutdown 2024-06-04T05:04:35.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.770+0000 7fbf631edb80 1 -- 172.21.15.70:0/2361934956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f1670 msgr2=0x55d3810fa370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:35.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.770+0000 7fbf631edb80 1 --2- 172.21.15.70:0/2361934956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f1670 0x55d3810fa370 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009d10 tx=0x7fbf48003e80 comp rx=0 tx=0).stop 2024-06-04T05:04:35.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.770+0000 7fbf631edb80 1 -- 172.21.15.70:0/2361934956 shutdown_connections 2024-06-04T05:04:35.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.770+0000 7fbf631edb80 1 --2- 172.21.15.70:0/2361934956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f66c0 0x55d3810fa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.770+0000 7fbf631edb80 1 --2- 172.21.15.70:0/2361934956 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d38110a410 0x55d38110c800 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.770+0000 7fbf39ffb700 0 client.17424 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:04:35.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.770+0000 7fbf631edb80 1 --2- 172.21.15.70:0/2361934956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f0cb0 0x55d3810fd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.770+0000 7fbf631edb80 1 --2- 172.21.15.70:0/2361934956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f1670 0x55d3810fa370 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.770+0000 7fbf631edb80 1 -- 172.21.15.70:0/2361934956 >> 172.21.15.70:0/2361934956 conn(0x55d381018240 msgr2=0x55d3810e6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:35.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.771+0000 7fbf631edb80 1 -- 172.21.15.70:0/2361934956 shutdown_connections 2024-06-04T05:04:35.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.771+0000 7fbf631edb80 1 -- 172.21.15.70:0/2361934956 wait complete. 2024-06-04T05:04:35.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.771+0000 7fbf631edb80 20 client.17424 trim_cache size 0 max 16384 2024-06-04T05:04:35.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.787+0000 7fbf631edb80 1 Processor -- start 2024-06-04T05:04:35.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.787+0000 7fbf631edb80 1 -- start start 2024-06-04T05:04:35.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.787+0000 7fbf631edb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f0cb0 0x55d3810e7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.787+0000 7fbf631edb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 0x55d38110f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.787+0000 7fbf631edb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 0x55d38110fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.787+0000 7fbf631edb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf541abbd0 con 0x55d3810f66c0 2024-06-04T05:04:35.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.787+0000 7fbf631edb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf541ad710 con 0x55d3810f0cb0 2024-06-04T05:04:35.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.787+0000 7fbf631edb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbf541ac2a0 con 0x55d3810f1670 2024-06-04T05:04:35.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.787+0000 7fbf52d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 0x55d38110fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.787+0000 7fbf51d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 0x55d38110f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.787+0000 7fbf52d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 0x55d38110fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54950/0 (socket says 172.21.15.70:54950) 2024-06-04T05:04:35.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.787+0000 7fbf52d9d700 1 -- 172.21.15.70:0/1881278164 learned_addr learned my addr 172.21.15.70:0/1881278164 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:35.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.787+0000 7fbf5259c700 1 --2- 172.21.15.70:0/1881278164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f0cb0 0x55d3810e7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.787+0000 7fbf51d9b700 1 -- 172.21.15.70:0/1881278164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f0cb0 msgr2=0x55d3810e7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:35.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.788+0000 7fbf51d9b700 1 --2- 172.21.15.70:0/1881278164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f0cb0 0x55d3810e7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.788+0000 7fbf51d9b700 1 -- 172.21.15.70:0/1881278164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 msgr2=0x55d38110fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:35.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.788+0000 7fbf51d9b700 1 --2- 172.21.15.70:0/1881278164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 0x55d38110fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.788+0000 7fbf51d9b700 1 -- 172.21.15.70:0/1881278164 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3810f8360 con 0x55d3810f1670 2024-06-04T05:04:35.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.788+0000 7fbf5259c700 1 --2- 172.21.15.70:0/1881278164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f0cb0 0x55d3810e7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:35.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.788+0000 7fbf52d9d700 1 --2- 172.21.15.70:0/1881278164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 0x55d38110fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:35.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.788+0000 7fbf51d9b700 1 --2- 172.21.15.70:0/1881278164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 0x55d38110f820 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbf48003760 tx=0x7fbf48009290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd2c15007d80c700 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:35.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.788+0000 7fbf3bfff700 1 -- 172.21.15.70:0/1881278164 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4803c030 con 0x55d3810f1670 2024-06-04T05:04:35.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.788+0000 7fbf3bfff700 1 -- 172.21.15.70:0/1881278164 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf48024720 con 0x55d3810f1670 2024-06-04T05:04:35.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.788+0000 7fbf3bfff700 1 -- 172.21.15.70:0/1881278164 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4802fe60 con 0x55d3810f1670 2024-06-04T05:04:35.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.789+0000 7fbf631edb80 1 -- 172.21.15.70:0/1881278164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 msgr2=0x55d38110f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:35.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.789+0000 7fbf631edb80 1 --2- 172.21.15.70:0/1881278164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 0x55d38110f820 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbf48003760 tx=0x7fbf48009290 comp rx=0 tx=0).stop 2024-06-04T05:04:35.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.789+0000 7fbf631edb80 1 -- 172.21.15.70:0/1881278164 shutdown_connections 2024-06-04T05:04:35.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.789+0000 7fbf631edb80 1 --2- 172.21.15.70:0/1881278164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 0x55d38110fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.789+0000 7fbf631edb80 1 --2- 172.21.15.70:0/1881278164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f0cb0 0x55d3810e7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.789+0000 7fbf631edb80 1 --2- 172.21.15.70:0/1881278164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 0x55d38110f820 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.789+0000 7fbf631edb80 1 -- 172.21.15.70:0/1881278164 >> 172.21.15.70:0/1881278164 conn(0x55d381018240 msgr2=0x55d3810e8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:35.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.789+0000 7fbf631edb80 1 -- 172.21.15.70:0/1881278164 shutdown_connections 2024-06-04T05:04:35.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.789+0000 7fbf631edb80 1 -- 172.21.15.70:0/1881278164 wait complete. 2024-06-04T05:04:35.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.789+0000 7fbf631edb80 1 -- start start 2024-06-04T05:04:35.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.789+0000 7fbf631edb80 1 Processor -- start 2024-06-04T05:04:35.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf631edb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f0cb0 0x55d38110cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf631edb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 0x55d38110d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf631edb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 0x55d38116cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf631edb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf541ad040 con 0x55d3810f66c0 2024-06-04T05:04:35.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf631edb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf541ac970 con 0x55d3810f0cb0 2024-06-04T05:04:35.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf631edb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbf541a9b90 con 0x55d3810f1670 2024-06-04T05:04:35.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf631edb80 10 client.0 _ll_register_callbacks cb 0x55d380fd5950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:35.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf51d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 0x55d38110d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf51d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 0x55d38110d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:38024/0 (socket says 172.21.15.70:38024) 2024-06-04T05:04:35.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf51d9b700 1 -- 172.21.15.70:0/4157618583 learned_addr learned my addr 172.21.15.70:0/4157618583 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:35.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf52d9d700 1 --2- 172.21.15.70:0/4157618583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 0x55d38116cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf5259c700 1 --2- 172.21.15.70:0/4157618583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f0cb0 0x55d38110cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf51d9b700 1 -- 172.21.15.70:0/4157618583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f0cb0 msgr2=0x55d38110cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:35.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf51d9b700 1 --2- 172.21.15.70:0/4157618583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f0cb0 0x55d38110cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf51d9b700 1 -- 172.21.15.70:0/4157618583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 msgr2=0x55d38116cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:35.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf51d9b700 1 --2- 172.21.15.70:0/4157618583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 0x55d38116cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf51d9b700 1 -- 172.21.15.70:0/4157618583 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d38116d250 con 0x55d3810f1670 2024-06-04T05:04:35.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf51d9b700 1 --2- 172.21.15.70:0/4157618583 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 0x55d38110d320 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbf48000c00 tx=0x7fbf480037c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb8aed31cd8bf404 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:35.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf39ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:04:35.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.790+0000 7fbf39ffb700 1 -- 172.21.15.70:0/4157618583 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4803c040 con 0x55d3810f1670 2024-06-04T05:04:35.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.791+0000 7fbf631edb80 1 -- 172.21.15.70:0/4157618583 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf48024720 con 0x55d3810f1670 2024-06-04T05:04:35.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.791+0000 7fbf39ffb700 1 -- 172.21.15.70:0/4157618583 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fbf4802fa40 con 0x55d3810f1670 2024-06-04T05:04:35.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.791+0000 7fbf39ffb700 1 -- 172.21.15.70:0/4157618583 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbf4802fc80 con 0x55d3810f1670 2024-06-04T05:04:35.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.791+0000 7fbf631edb80 10 client.26722 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:35.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.791+0000 7fbf631edb80 1 -- 172.21.15.70:0/4157618583 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d3811745e0 con 0x55d3810f1670 2024-06-04T05:04:35.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.791+0000 7fbf39ffb700 1 -- 172.21.15.70:0/4157618583 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf48024720 con 0x55d3810f1670 2024-06-04T05:04:35.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.791+0000 7fbf39ffb700 1 -- 172.21.15.70:0/4157618583 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fbf48010470 con 0x55d3810f1670 2024-06-04T05:04:35.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.791+0000 7fbf631edb80 10 client.26722 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:35.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.791+0000 7fbf631edb80 10 client.26722 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:04:35.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.791+0000 7fbf631edb80 20 client.26722 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:35.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.791+0000 7fbf631edb80 1 --2- 172.21.15.70:0/4157618583 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d381174b40 0x55d381176f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:35.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.791+0000 7fbf631edb80 4 client.26722 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:04:35.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.791+0000 7fbf631edb80 1 -- 172.21.15.70:0/4157618583 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d3810464a0 con 0x55d381174b40 2024-06-04T05:04:35.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.791+0000 7fbf5259c700 1 --2- 172.21.15.70:0/4157618583 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d381174b40 0x55d381176f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:35.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.792+0000 7fbf39ffb700 10 client.26722 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:04:35.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.792+0000 7fbf5259c700 1 --2- 172.21.15.70:0/4157618583 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d381174b40 0x55d381176f30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ef6da2574fc77b86 server_cookie=133aade315f1c1fe in_seq=0 out_seq=0 2024-06-04T05:04:35.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.792+0000 7fbf39ffb700 1 -- 172.21.15.70:0/4157618583 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d3810464a0 con 0x55d381174b40 2024-06-04T05:04:35.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.792+0000 7fbf39ffb700 10 client.26722 handle_command_reply: tid=0 2024-06-04T05:04:35.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.792+0000 7fbf631edb80 1 client.26722 shutdown 2024-06-04T05:04:35.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.792+0000 7fbf631edb80 1 -- 172.21.15.70:0/4157618583 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 msgr2=0x55d38110d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:35.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.792+0000 7fbf631edb80 1 --2- 172.21.15.70:0/4157618583 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 0x55d38110d320 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbf48000c00 tx=0x7fbf480037c0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.792+0000 7fbf631edb80 1 -- 172.21.15.70:0/4157618583 shutdown_connections 2024-06-04T05:04:35.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.792+0000 7fbf631edb80 1 --2- 172.21.15.70:0/4157618583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3810f66c0 0x55d38116cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.792+0000 7fbf631edb80 1 --2- 172.21.15.70:0/4157618583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3810f0cb0 0x55d38110cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.792+0000 7fbf631edb80 1 --2- 172.21.15.70:0/4157618583 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55d381174b40 0x55d381176f30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.792+0000 7fbf631edb80 1 --2- 172.21.15.70:0/4157618583 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d3810f1670 0x55d38110d320 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.792+0000 7fbf631edb80 1 -- 172.21.15.70:0/4157618583 >> 172.21.15.70:0/4157618583 conn(0x55d381018240 msgr2=0x55d3810f54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:35.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.792+0000 7fbf39ffb700 0 client.26722 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:04:35.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.792+0000 7fbf631edb80 1 -- 172.21.15.70:0/4157618583 shutdown_connections 2024-06-04T05:04:35.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.792+0000 7fbf631edb80 1 -- 172.21.15.70:0/4157618583 wait complete. 2024-06-04T05:04:35.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.792+0000 7fbf631edb80 20 client.26722 trim_cache size 0 max 16384 2024-06-04T05:04:35.800 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:04:35.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.793+0000 7fbf58958700 1 -- 172.21.15.70:0/2128714848 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf3c0b29f0 msgr2=0x7fbf3c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:35.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.793+0000 7fbf58958700 1 --2- 172.21.15.70:0/2128714848 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf3c0b29f0 0x7fbf3c0b4eb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbf5407ae70 tx=0x7fbf48005fd0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.793+0000 7fbf58958700 1 -- 172.21.15.70:0/2128714848 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf541035d0 msgr2=0x7fbf5419f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:35.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.793+0000 7fbf58958700 1 --2- 172.21.15.70:0/2128714848 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf541035d0 0x7fbf5419f830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400e9f0 tx=0x7fbf4400ed00 comp rx=0 tx=0).stop 2024-06-04T05:04:35.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.793+0000 7fbf58958700 1 -- 172.21.15.70:0/2128714848 shutdown_connections 2024-06-04T05:04:35.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.793+0000 7fbf58958700 1 --2- 172.21.15.70:0/2128714848 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf3c0b29f0 0x7fbf3c0b4eb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.793+0000 7fbf58958700 1 --2- 172.21.15.70:0/2128714848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf54109670 0x7fbf541999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.793+0000 7fbf58958700 1 --2- 172.21.15.70:0/2128714848 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf541035d0 0x7fbf5419f830 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.793+0000 7fbf58958700 1 --2- 172.21.15.70:0/2128714848 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf54103f90 0x7fbf5419fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:35.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.793+0000 7fbf58958700 1 -- 172.21.15.70:0/2128714848 >> 172.21.15.70:0/2128714848 conn(0x7fbf5407a730 msgr2=0x7fbf54102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:35.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.793+0000 7fbf58958700 1 -- 172.21.15.70:0/2128714848 shutdown_connections 2024-06-04T05:04:35.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:35.793+0000 7fbf58958700 1 -- 172.21.15.70:0/2128714848 wait complete. 2024-06-04T05:04:36.526 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:04:36.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 122600 1.98 MB/sec execute 209 sec latency 19.772 ms 2024-06-04T05:04:36.792 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:04:37.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.356+0000 7fc9cb4e1640 1 -- 172.21.15.70:0/467654325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9c4101980 msgr2=0x7fc9c4101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:37.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.356+0000 7fc9cb4e1640 1 --2- 172.21.15.70:0/467654325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9c4101980 0x7fc9c4101e00 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4009930 tx=0x7fc9b402a420 comp rx=0 tx=0).stop 2024-06-04T05:04:37.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.356+0000 7fc9cb4e1640 1 -- 172.21.15.70:0/467654325 shutdown_connections 2024-06-04T05:04:37.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.356+0000 7fc9cb4e1640 1 --2- 172.21.15.70:0/467654325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9c4102340 0x7fc9c410e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.356+0000 7fc9cb4e1640 1 --2- 172.21.15.70:0/467654325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9c4101980 0x7fc9c4101e00 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.356+0000 7fc9cb4e1640 1 --2- 172.21.15.70:0/467654325 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9c4107940 0x7fc9c4107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.356+0000 7fc9cb4e1640 1 -- 172.21.15.70:0/467654325 >> 172.21.15.70:0/467654325 conn(0x7fc9c4078ad0 msgr2=0x7fc9c4078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:37.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.356+0000 7fc9cb4e1640 1 -- 172.21.15.70:0/467654325 shutdown_connections 2024-06-04T05:04:37.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.356+0000 7fc9cb4e1640 1 -- 172.21.15.70:0/467654325 wait complete. 2024-06-04T05:04:37.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.357+0000 7fc9cb4e1640 1 Processor -- start 2024-06-04T05:04:37.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.357+0000 7fc9cb4e1640 1 -- start start 2024-06-04T05:04:37.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.357+0000 7fc9cb4e1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9c4101980 0x7fc9c4199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.357+0000 7fc9cb4e1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9c4102340 0x7fc9c419a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.357+0000 7fc9cb4e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9c4107940 0x7fc9c41a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.357+0000 7fc9cb4e1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c4111ab0 con 0x7fc9c4102340 2024-06-04T05:04:37.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.357+0000 7fc9cb4e1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c4111930 con 0x7fc9c4107940 2024-06-04T05:04:37.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.357+0000 7fc9cb4e1640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c4111c30 con 0x7fc9c4101980 2024-06-04T05:04:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.357+0000 7fc9c927d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9c4101980 0x7fc9c4199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.357+0000 7fc9c927d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9c4101980 0x7fc9c4199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:54938/0 (socket says 172.21.15.70:54938) 2024-06-04T05:04:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.357+0000 7fc9c927d640 1 -- 172.21.15.70:0/251422840 learned_addr learned my addr 172.21.15.70:0/251422840 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.357+0000 7fc9c8a7c640 1 --2- 172.21.15.70:0/251422840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9c4102340 0x7fc9c419a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.357+0000 7fc9c9a7e640 1 --2- 172.21.15.70:0/251422840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9c4107940 0x7fc9c41a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.357+0000 7fc9c927d640 1 -- 172.21.15.70:0/251422840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9c4107940 msgr2=0x7fc9c41a4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.357+0000 7fc9c927d640 1 --2- 172.21.15.70:0/251422840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9c4107940 0x7fc9c41a4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.357+0000 7fc9c927d640 1 -- 172.21.15.70:0/251422840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9c4102340 msgr2=0x7fc9c419a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.357+0000 7fc9c927d640 1 --2- 172.21.15.70:0/251422840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9c4102340 0x7fc9c419a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.358+0000 7fc9c927d640 1 -- 172.21.15.70:0/251422840 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9c41a5530 con 0x7fc9c4101980 2024-06-04T05:04:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.358+0000 7fc9c927d640 1 --2- 172.21.15.70:0/251422840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9c4101980 0x7fc9c4199f70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc9c000e9f0 tx=0x7fc9c000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.358+0000 7fc9ba7fc640 1 -- 172.21.15.70:0/251422840 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c000cd80 con 0x7fc9c4101980 2024-06-04T05:04:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.358+0000 7fc9ba7fc640 1 -- 172.21.15.70:0/251422840 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9c00108c0 con 0x7fc9c4101980 2024-06-04T05:04:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.358+0000 7fc9ba7fc640 1 -- 172.21.15.70:0/251422840 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c00187c0 con 0x7fc9c4101980 2024-06-04T05:04:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.358+0000 7fc9cb4e1640 1 -- 172.21.15.70:0/251422840 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9c41a5700 con 0x7fc9c4101980 2024-06-04T05:04:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.358+0000 7fc9cb4e1640 1 -- 172.21.15.70:0/251422840 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9c41a5c20 con 0x7fc9c4101980 2024-06-04T05:04:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.361+0000 7fc9ce200740 1 Processor -- start 2024-06-04T05:04:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.361+0000 7fc9ce200740 1 -- start start 2024-06-04T05:04:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.361+0000 7fc9ba7fc640 1 -- 172.21.15.70:0/251422840 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fc9c0010430 con 0x7fc9c4101980 2024-06-04T05:04:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.361+0000 7fc9ce200740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4b9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.361+0000 7fc9ce200740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4ba020 0x56395d4c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.361+0000 7fc9ce200740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4c8e10 0x56395d4cb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.361+0000 7fc9ce200740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56395d42bc30 con 0x56395d4ba020 2024-06-04T05:04:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.361+0000 7fc9ce200740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56395d41b7d0 con 0x56395d4c8e10 2024-06-04T05:04:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.361+0000 7fc9ce200740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56395d41f810 con 0x56395d4b9700 2024-06-04T05:04:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.361+0000 7fc9c927d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4ba020 0x56395d4c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.361+0000 7fc9c927d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4ba020 0x56395d4c88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54256/0 (socket says 172.21.15.70:54256) 2024-06-04T05:04:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.361+0000 7fc9c927d640 1 -- 172.21.15.70:0/3279549297 learned_addr learned my addr 172.21.15.70:0/3279549297 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.361+0000 7fc9c8a7c640 1 --2- 172.21.15.70:0/3279549297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4c8e10 0x56395d4cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.361+0000 7fc9c9a7e640 1 --2- 172.21.15.70:0/3279549297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4b9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.362+0000 7fc9c927d640 1 -- 172.21.15.70:0/3279549297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 msgr2=0x56395d4b9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.362+0000 7fc9c927d640 1 --2- 172.21.15.70:0/3279549297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4b9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.362+0000 7fc9c927d640 1 -- 172.21.15.70:0/3279549297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4c8e10 msgr2=0x56395d4cb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.362+0000 7fc9c927d640 1 --2- 172.21.15.70:0/3279549297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4c8e10 0x56395d4cb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.362+0000 7fc9c927d640 1 -- 172.21.15.70:0/3279549297 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56395d130210 con 0x56395d4ba020 2024-06-04T05:04:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.362+0000 7fc9ba7fc640 1 --2- 172.21.15.70:0/251422840 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9b00b2a10 0x7fc9b00b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.362+0000 7fc9c9a7e640 1 --2- 172.21.15.70:0/251422840 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9b00b2a10 0x7fc9b00b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.362+0000 7fc9ba7fc640 1 -- 172.21.15.70:0/251422840 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fc9c00e29d0 con 0x7fc9c4101980 2024-06-04T05:04:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.362+0000 7fc9c927d640 1 --2- 172.21.15.70:0/3279549297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4ba020 0x56395d4c88d0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fc9c00e8ba0 tx=0x7fc9c00a6320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c09e0eea8f55a55a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.362+0000 7fc9af7fe640 1 -- 172.21.15.70:0/3279549297 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c00a9aa0 con 0x56395d4ba020 2024-06-04T05:04:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.362+0000 7fc9c9a7e640 1 --2- 172.21.15.70:0/251422840 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9b00b2a10 0x7fc9b00b4ed0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc005fd0 tx=0x7fc9bc009500 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.363+0000 7fc9af7fe640 1 -- 172.21.15.70:0/3279549297 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9c00b23c0 con 0x56395d4ba020 2024-06-04T05:04:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.363+0000 7fc9af7fe640 1 -- 172.21.15.70:0/3279549297 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c0010720 con 0x56395d4ba020 2024-06-04T05:04:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.363+0000 7fc9ce200740 1 -- 172.21.15.70:0/3279549297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4ba020 msgr2=0x56395d4c88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.363+0000 7fc9ce200740 1 --2- 172.21.15.70:0/3279549297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4ba020 0x56395d4c88d0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fc9c00e8ba0 tx=0x7fc9c00a6320 comp rx=0 tx=0).stop 2024-06-04T05:04:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.363+0000 7fc9ce200740 1 -- 172.21.15.70:0/3279549297 shutdown_connections 2024-06-04T05:04:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.363+0000 7fc9ce200740 1 --2- 172.21.15.70:0/3279549297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4c8e10 0x56395d4cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.363+0000 7fc9ce200740 1 --2- 172.21.15.70:0/3279549297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4ba020 0x56395d4c88d0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.363+0000 7fc9ce200740 1 --2- 172.21.15.70:0/3279549297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4b9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.363+0000 7fc9ce200740 1 -- 172.21.15.70:0/3279549297 >> 172.21.15.70:0/3279549297 conn(0x56395d4b5d30 msgr2=0x56395d4b6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.363+0000 7fc9ce200740 1 -- 172.21.15.70:0/3279549297 shutdown_connections 2024-06-04T05:04:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.363+0000 7fc9ce200740 1 -- 172.21.15.70:0/3279549297 wait complete. 2024-06-04T05:04:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.364+0000 7fc9ce200740 1 -- start start 2024-06-04T05:04:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.364+0000 7fc9ce200740 1 Processor -- start 2024-06-04T05:04:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.364+0000 7fc9ce200740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.364+0000 7fc9ce200740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4ba020 0x56395d4c6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.364+0000 7fc9ce200740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4c8e10 0x56395d4c6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.364+0000 7fc9ce200740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56395d101790 con 0x56395d4c8e10 2024-06-04T05:04:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.364+0000 7fc9ce200740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56395d3d7890 con 0x56395d4ba020 2024-06-04T05:04:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.364+0000 7fc9ce200740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56395d4cb9e0 con 0x56395d4b9700 2024-06-04T05:04:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.364+0000 7fc9ce200740 10 client.0 _ll_register_callbacks cb 0x56395d12fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.364+0000 7fc9c9a7e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4c8e10 0x56395d4c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.364+0000 7fc9c927d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.364+0000 7fc9c8a7c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4ba020 0x56395d4c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.364+0000 7fc9c927d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4c5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:54970/0 (socket says 172.21.15.70:54970) 2024-06-04T05:04:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.364+0000 7fc9c927d640 1 -- 172.21.15.70:0/2996675722 learned_addr learned my addr 172.21.15.70:0/2996675722 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.365+0000 7fc9c927d640 1 -- 172.21.15.70:0/2996675722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4ba020 msgr2=0x56395d4c6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.365+0000 7fc9c927d640 1 --2- 172.21.15.70:0/2996675722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4ba020 0x56395d4c6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.365+0000 7fc9c927d640 1 -- 172.21.15.70:0/2996675722 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4c8e10 msgr2=0x56395d4c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.365+0000 7fc9c927d640 1 --2- 172.21.15.70:0/2996675722 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4c8e10 0x56395d4c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.365+0000 7fc9c927d640 1 -- 172.21.15.70:0/2996675722 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56395d1fc620 con 0x56395d4b9700 2024-06-04T05:04:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.365+0000 7fc9c927d640 1 --2- 172.21.15.70:0/2996675722 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4c5e10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc9c00a97e0 tx=0x7fc9c00e0540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9fd81fd1ccf69d0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.365+0000 7fc9acff9640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:04:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.365+0000 7fc9acff9640 1 -- 172.21.15.70:0/2996675722 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c00a9440 con 0x56395d4b9700 2024-06-04T05:04:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.365+0000 7fc9acff9640 1 -- 172.21.15.70:0/2996675722 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fc9c00a5040 con 0x56395d4b9700 2024-06-04T05:04:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.365+0000 7fc9ce200740 1 -- 172.21.15.70:0/2996675722 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56395d42db80 con 0x56395d4b9700 2024-06-04T05:04:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.365+0000 7fc9acff9640 1 -- 172.21.15.70:0/2996675722 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc9c00a9620 con 0x56395d4b9700 2024-06-04T05:04:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.365+0000 7fc9ce200740 10 client.26734 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.365+0000 7fc9ce200740 1 -- 172.21.15.70:0/2996675722 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56395d4d7810 con 0x56395d4b9700 2024-06-04T05:04:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.365+0000 7fc9acff9640 1 -- 172.21.15.70:0/2996675722 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x56395d42db80 con 0x56395d4b9700 2024-06-04T05:04:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.365+0000 7fc9acff9640 1 -- 172.21.15.70:0/2996675722 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fc9c00a9f30 con 0x56395d4b9700 2024-06-04T05:04:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.366+0000 7fc9ce200740 10 client.26734 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.366+0000 7fc9ce200740 10 client.26734 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:04:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.366+0000 7fc9ce200740 20 client.26734 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.366+0000 7fc9ce200740 1 --2- 172.21.15.70:0/2996675722 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56395d4d8b60 0x56395d4daf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.366+0000 7fc9ce200740 4 client.26734 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:04:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.366+0000 7fc9ce200740 1 -- 172.21.15.70:0/2996675722 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56395d427360 con 0x56395d4d8b60 2024-06-04T05:04:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.366+0000 7fc9c8a7c640 1 --2- 172.21.15.70:0/2996675722 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56395d4d8b60 0x56395d4daf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.366+0000 7fc9acff9640 10 client.26734 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.366+0000 7fc9c8a7c640 1 --2- 172.21.15.70:0/2996675722 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56395d4d8b60 0x56395d4daf50 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3e16452558103a5 server_cookie=e9485ccff66d058d in_seq=0 out_seq=0 2024-06-04T05:04:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.367+0000 7fc9acff9640 1 -- 172.21.15.70:0/2996675722 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56395d427360 con 0x56395d4d8b60 2024-06-04T05:04:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.367+0000 7fc9acff9640 10 client.26734 handle_command_reply: tid=0 2024-06-04T05:04:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.367+0000 7fc9ce200740 1 client.26734 shutdown 2024-06-04T05:04:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.367+0000 7fc9ce200740 1 -- 172.21.15.70:0/2996675722 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 msgr2=0x56395d4c5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.367+0000 7fc9ce200740 1 --2- 172.21.15.70:0/2996675722 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4c5e10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc9c00a97e0 tx=0x7fc9c00e0540 comp rx=0 tx=0).stop 2024-06-04T05:04:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.367+0000 7fc9ce200740 1 -- 172.21.15.70:0/2996675722 shutdown_connections 2024-06-04T05:04:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.367+0000 7fc9ce200740 1 --2- 172.21.15.70:0/2996675722 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56395d4d8b60 0x56395d4daf50 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.367+0000 7fc9ce200740 1 --2- 172.21.15.70:0/2996675722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4ba020 0x56395d4c6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.367+0000 7fc9ce200740 1 --2- 172.21.15.70:0/2996675722 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4c8e10 0x56395d4c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.367+0000 7fc9ce200740 1 --2- 172.21.15.70:0/2996675722 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4c5e10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.367+0000 7fc9acff9640 0 client.26734 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.367+0000 7fc9ce200740 1 -- 172.21.15.70:0/2996675722 >> 172.21.15.70:0/2996675722 conn(0x56395d4b5d30 msgr2=0x56395d4bbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.367+0000 7fc9ce200740 1 -- 172.21.15.70:0/2996675722 shutdown_connections 2024-06-04T05:04:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.367+0000 7fc9ce200740 1 -- 172.21.15.70:0/2996675722 wait complete. 2024-06-04T05:04:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.367+0000 7fc9ce200740 20 client.26734 trim_cache size 0 max 16384 2024-06-04T05:04:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.385+0000 7fc9ce200740 1 Processor -- start 2024-06-04T05:04:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.385+0000 7fc9ce200740 1 -- start start 2024-06-04T05:04:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9ce200740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9ce200740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4ba020 0x56395d4c3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9ce200740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4c8e10 0x56395d4c35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9ce200740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c41ac350 con 0x56395d4c8e10 2024-06-04T05:04:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9ce200740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c41ade90 con 0x56395d4ba020 2024-06-04T05:04:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9ce200740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c41aca20 con 0x56395d4b9700 2024-06-04T05:04:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9c9a7e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4c8e10 0x56395d4c35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9c9a7e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4c8e10 0x56395d4c35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54282/0 (socket says 172.21.15.70:54282) 2024-06-04T05:04:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9c9a7e640 1 -- 172.21.15.70:0/1192044287 learned_addr learned my addr 172.21.15.70:0/1192044287 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9c927d640 1 --2- 172.21.15.70:0/1192044287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9c8a7c640 1 --2- 172.21.15.70:0/1192044287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4ba020 0x56395d4c3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9c9a7e640 1 -- 172.21.15.70:0/1192044287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 msgr2=0x56395d4c64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9c9a7e640 1 --2- 172.21.15.70:0/1192044287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4c64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9c9a7e640 1 -- 172.21.15.70:0/1192044287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4ba020 msgr2=0x56395d4c3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9c9a7e640 1 --2- 172.21.15.70:0/1192044287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4ba020 0x56395d4c3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9c9a7e640 1 -- 172.21.15.70:0/1192044287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56395d4b5b60 con 0x56395d4c8e10 2024-06-04T05:04:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9c927d640 1 --2- 172.21.15.70:0/1192044287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4c64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9c9a7e640 1 --2- 172.21.15.70:0/1192044287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4c8e10 0x56395d4c35b0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc003ea0 tx=0x7fc9bc01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9371b045cc132905 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9aeffd640 1 -- 172.21.15.70:0/1192044287 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc004b20 con 0x56395d4c8e10 2024-06-04T05:04:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9aeffd640 1 -- 172.21.15.70:0/1192044287 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9bc004cc0 con 0x56395d4c8e10 2024-06-04T05:04:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.386+0000 7fc9aeffd640 1 -- 172.21.15.70:0/1192044287 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc0053a0 con 0x56395d4c8e10 2024-06-04T05:04:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.387+0000 7fc9ce200740 1 -- 172.21.15.70:0/1192044287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4c8e10 msgr2=0x56395d4c35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.387+0000 7fc9ce200740 1 --2- 172.21.15.70:0/1192044287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4c8e10 0x56395d4c35b0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc003ea0 tx=0x7fc9bc01a000 comp rx=0 tx=0).stop 2024-06-04T05:04:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.387+0000 7fc9ce200740 1 -- 172.21.15.70:0/1192044287 shutdown_connections 2024-06-04T05:04:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.387+0000 7fc9ce200740 1 --2- 172.21.15.70:0/1192044287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4ba020 0x56395d4c3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.387+0000 7fc9ce200740 1 --2- 172.21.15.70:0/1192044287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4c8e10 0x56395d4c35b0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.387+0000 7fc9ce200740 1 --2- 172.21.15.70:0/1192044287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4c64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.387+0000 7fc9ce200740 1 -- 172.21.15.70:0/1192044287 >> 172.21.15.70:0/1192044287 conn(0x56395d4b5d30 msgr2=0x56395d4c5dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.387+0000 7fc9ce200740 1 -- 172.21.15.70:0/1192044287 shutdown_connections 2024-06-04T05:04:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.387+0000 7fc9ce200740 1 -- 172.21.15.70:0/1192044287 wait complete. 2024-06-04T05:04:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9ce200740 1 -- start start 2024-06-04T05:04:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9ce200740 1 Processor -- start 2024-06-04T05:04:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9ce200740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4dbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9ce200740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4ba020 0x56395d4dc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9ce200740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4c8e10 0x56395d4dc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9ce200740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c41ad7c0 con 0x56395d4ba020 2024-06-04T05:04:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9ce200740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c41ad0f0 con 0x56395d4c8e10 2024-06-04T05:04:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9ce200740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c41aa350 con 0x56395d4b9700 2024-06-04T05:04:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9ce200740 10 client.0 _ll_register_callbacks cb 0x7fc9c0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9c8a7c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4ba020 0x56395d4dc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9c8a7c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4ba020 0x56395d4dc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54296/0 (socket says 172.21.15.70:54296) 2024-06-04T05:04:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9c8a7c640 1 -- 172.21.15.70:0/3803960300 learned_addr learned my addr 172.21.15.70:0/3803960300 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9c927d640 1 --2- 172.21.15.70:0/3803960300 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4dbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9c9a7e640 1 --2- 172.21.15.70:0/3803960300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4c8e10 0x56395d4dc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9c8a7c640 1 -- 172.21.15.70:0/3803960300 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 msgr2=0x56395d4dbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9c8a7c640 1 --2- 172.21.15.70:0/3803960300 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4dbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9c8a7c640 1 -- 172.21.15.70:0/3803960300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4c8e10 msgr2=0x56395d4dc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9c8a7c640 1 --2- 172.21.15.70:0/3803960300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4c8e10 0x56395d4dc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.388+0000 7fc9c8a7c640 1 -- 172.21.15.70:0/3803960300 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56395d4d4b70 con 0x56395d4ba020 2024-06-04T05:04:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.389+0000 7fc9c8a7c640 1 --2- 172.21.15.70:0/3803960300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4ba020 0x56395d4dc330 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4005ec0 tx=0x7fc9b4004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9c32603ef06bf1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.389+0000 7fc9acff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:04:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.389+0000 7fc9acff9640 1 -- 172.21.15.70:0/3803960300 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4035040 con 0x56395d4ba020 2024-06-04T05:04:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.389+0000 7fc9acff9640 1 -- 172.21.15.70:0/3803960300 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fc9b4031810 con 0x56395d4ba020 2024-06-04T05:04:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.389+0000 7fc9ce200740 1 -- 172.21.15.70:0/3803960300 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9bc004cc0 con 0x56395d4ba020 2024-06-04T05:04:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.389+0000 7fc9acff9640 1 -- 172.21.15.70:0/3803960300 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc9b40035f0 con 0x56395d4ba020 2024-06-04T05:04:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.389+0000 7fc9ce200740 10 client.17448 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.389+0000 7fc9ce200740 1 -- 172.21.15.70:0/3803960300 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56395d543640 con 0x56395d4ba020 2024-06-04T05:04:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.389+0000 7fc9acff9640 1 -- 172.21.15.70:0/3803960300 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9bc004cc0 con 0x56395d4ba020 2024-06-04T05:04:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.389+0000 7fc9acff9640 1 -- 172.21.15.70:0/3803960300 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fc9b4055070 con 0x56395d4ba020 2024-06-04T05:04:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.389+0000 7fc9ce200740 10 client.17448 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.389+0000 7fc9ce200740 10 client.17448 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:04:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.389+0000 7fc9ce200740 20 client.17448 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.389+0000 7fc9ce200740 1 --2- 172.21.15.70:0/3803960300 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56395d543df0 0x56395d546210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.389+0000 7fc9ce200740 4 client.17448 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:04:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.389+0000 7fc9ce200740 1 -- 172.21.15.70:0/3803960300 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56395d3a15f0 con 0x56395d543df0 2024-06-04T05:04:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.390+0000 7fc9c927d640 1 --2- 172.21.15.70:0/3803960300 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56395d543df0 0x56395d546210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.390+0000 7fc9acff9640 10 client.17448 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.390+0000 7fc9c927d640 1 --2- 172.21.15.70:0/3803960300 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56395d543df0 0x56395d546210 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf59154e05aa77b9 server_cookie=7999a910c7f056e4 in_seq=0 out_seq=0 2024-06-04T05:04:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.411+0000 7fc9acff9640 1 -- 172.21.15.70:0/3803960300 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56395d3a15f0 con 0x56395d543df0 2024-06-04T05:04:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.411+0000 7fc9acff9640 10 client.17448 handle_command_reply: tid=0 2024-06-04T05:04:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.411+0000 7fc9ce200740 1 client.17448 shutdown 2024-06-04T05:04:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.411+0000 7fc9ce200740 1 -- 172.21.15.70:0/3803960300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4ba020 msgr2=0x56395d4dc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.411+0000 7fc9ce200740 1 --2- 172.21.15.70:0/3803960300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4ba020 0x56395d4dc330 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4005ec0 tx=0x7fc9b4004440 comp rx=0 tx=0).stop 2024-06-04T05:04:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.411+0000 7fc9ce200740 1 -- 172.21.15.70:0/3803960300 shutdown_connections 2024-06-04T05:04:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.411+0000 7fc9ce200740 1 --2- 172.21.15.70:0/3803960300 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56395d543df0 0x56395d546210 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.411+0000 7fc9ce200740 1 --2- 172.21.15.70:0/3803960300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56395d4c8e10 0x56395d4dc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.411+0000 7fc9acff9640 0 client.17448 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.411+0000 7fc9ce200740 1 --2- 172.21.15.70:0/3803960300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d4ba020 0x56395d4dc330 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.411+0000 7fc9ce200740 1 --2- 172.21.15.70:0/3803960300 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56395d4b9700 0x56395d4dbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.411+0000 7fc9ce200740 1 -- 172.21.15.70:0/3803960300 >> 172.21.15.70:0/3803960300 conn(0x56395d4b5d30 msgr2=0x56395d4b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.412+0000 7fc9ce200740 1 -- 172.21.15.70:0/3803960300 shutdown_connections 2024-06-04T05:04:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.412+0000 7fc9ce200740 1 -- 172.21.15.70:0/3803960300 wait complete. 2024-06-04T05:04:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.412+0000 7fc9ce200740 20 client.17448 trim_cache size 0 max 16384 2024-06-04T05:04:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:04:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:04:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:04:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.412+0000 7fc9cb4e1640 1 -- 172.21.15.70:0/251422840 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9b00b2a10 msgr2=0x7fc9b00b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.412+0000 7fc9cb4e1640 1 --2- 172.21.15.70:0/251422840 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9b00b2a10 0x7fc9b00b4ed0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc005fd0 tx=0x7fc9bc009500 comp rx=0 tx=0).stop 2024-06-04T05:04:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.412+0000 7fc9cb4e1640 1 -- 172.21.15.70:0/251422840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9c4101980 msgr2=0x7fc9c4199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.412+0000 7fc9cb4e1640 1 --2- 172.21.15.70:0/251422840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9c4101980 0x7fc9c4199f70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc9c000e9f0 tx=0x7fc9c000eeb0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.412+0000 7fc9cb4e1640 1 -- 172.21.15.70:0/251422840 shutdown_connections 2024-06-04T05:04:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.412+0000 7fc9cb4e1640 1 --2- 172.21.15.70:0/251422840 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9b00b2a10 0x7fc9b00b4ed0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.412+0000 7fc9cb4e1640 1 --2- 172.21.15.70:0/251422840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9c4107940 0x7fc9c41a4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.412+0000 7fc9cb4e1640 1 --2- 172.21.15.70:0/251422840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9c4102340 0x7fc9c419a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.412+0000 7fc9cb4e1640 1 --2- 172.21.15.70:0/251422840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9c4101980 0x7fc9c4199f70 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.412+0000 7fc9cb4e1640 1 -- 172.21.15.70:0/251422840 >> 172.21.15.70:0/251422840 conn(0x7fc9c4078ad0 msgr2=0x7fc9c4104540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.412+0000 7fc9cb4e1640 1 -- 172.21.15.70:0/251422840 shutdown_connections 2024-06-04T05:04:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:37.412+0000 7fc9cb4e1640 1 -- 172.21.15.70:0/251422840 wait complete. 2024-06-04T05:04:37.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:37 smithi070 ceph-mon[32631]: pgmap v228: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 1.9 MiB/s wr, 191 op/s 2024-06-04T05:04:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:37 smithi112 ceph-mon[36453]: pgmap v228: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 1.9 MiB/s wr, 191 op/s 2024-06-04T05:04:37.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 123088 1.98 MB/sec execute 210 sec latency 73.738 ms 2024-06-04T05:04:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:37 smithi098 ceph-mon[35569]: pgmap v228: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 1.9 MiB/s wr, 191 op/s 2024-06-04T05:04:38.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 123876 2.00 MB/sec execute 211 sec latency 60.685 ms 2024-06-04T05:04:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:39 smithi070 ceph-mon[32631]: pgmap v229: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-06-04T05:04:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:39 smithi112 ceph-mon[36453]: pgmap v229: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-06-04T05:04:39.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 124211 2.01 MB/sec execute 212 sec latency 88.530 ms 2024-06-04T05:04:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:39 smithi098 ceph-mon[35569]: pgmap v229: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-06-04T05:04:40.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 124567 2.01 MB/sec execute 213 sec latency 14.630 ms 2024-06-04T05:04:41.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.282+0000 7f08e8ac9700 1 -- 172.21.15.70:0/368590603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e4101b30 msgr2=0x7f08e4101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:41.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.282+0000 7f08e8ac9700 1 --2- 172.21.15.70:0/368590603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e4101b30 0x7f08e4101fb0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f08d80099c0 tx=0x7f08d8009cd0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.282+0000 7f08e8ac9700 1 -- 172.21.15.70:0/368590603 shutdown_connections 2024-06-04T05:04:41.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.282+0000 7f08e8ac9700 1 --2- 172.21.15.70:0/368590603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08e41024f0 0x7f08e41103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.282+0000 7f08e8ac9700 1 --2- 172.21.15.70:0/368590603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e4109520 0x7f08e4109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.282+0000 7f08e8ac9700 1 --2- 172.21.15.70:0/368590603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e4101b30 0x7f08e4101fb0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.282+0000 7f08e8ac9700 1 -- 172.21.15.70:0/368590603 >> 172.21.15.70:0/368590603 conn(0x7f08e4078ab0 msgr2=0x7f08e40ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:41.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.282+0000 7f08e8ac9700 1 -- 172.21.15.70:0/368590603 shutdown_connections 2024-06-04T05:04:41.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.282+0000 7f08e8ac9700 1 -- 172.21.15.70:0/368590603 wait complete. 2024-06-04T05:04:41.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.283+0000 7f08e8ac9700 1 Processor -- start 2024-06-04T05:04:41.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.283+0000 7f08e8ac9700 1 -- start start 2024-06-04T05:04:41.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.283+0000 7f08e8ac9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e4101b30 0x7f08e4102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.283+0000 7f08e8ac9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08e41024f0 0x7f08e4103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.283+0000 7f08e8ac9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e4109520 0x7f08e4103790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.283+0000 7f08e8ac9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08e4113610 con 0x7f08e4109520 2024-06-04T05:04:41.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.283+0000 7f08e8ac9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08e4113490 con 0x7f08e4101b30 2024-06-04T05:04:41.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.283+0000 7f08e8ac9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f08e4113790 con 0x7f08e41024f0 2024-06-04T05:04:41.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.283+0000 7f08e2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e4109520 0x7f08e4103790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.283+0000 7f08e2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e4109520 0x7f08e4103790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54328/0 (socket says 172.21.15.70:54328) 2024-06-04T05:04:41.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.284+0000 7f08e2d9d700 1 -- 172.21.15.70:0/3946332542 learned_addr learned my addr 172.21.15.70:0/3946332542 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:41.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.284+0000 7f08e259c700 1 --2- 172.21.15.70:0/3946332542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e4101b30 0x7f08e4102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.284+0000 7f08e1d9b700 1 --2- 172.21.15.70:0/3946332542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08e41024f0 0x7f08e4103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.284+0000 7f08e2d9d700 1 -- 172.21.15.70:0/3946332542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08e41024f0 msgr2=0x7f08e4103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:41.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.284+0000 7f08e2d9d700 1 --2- 172.21.15.70:0/3946332542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08e41024f0 0x7f08e4103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.284+0000 7f08e2d9d700 1 -- 172.21.15.70:0/3946332542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e4101b30 msgr2=0x7f08e4102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:41.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.284+0000 7f08e2d9d700 1 --2- 172.21.15.70:0/3946332542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e4101b30 0x7f08e4102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.284+0000 7f08e2d9d700 1 -- 172.21.15.70:0/3946332542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e41a8050 con 0x7f08e4109520 2024-06-04T05:04:41.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.284+0000 7f08e259c700 1 --2- 172.21.15.70:0/3946332542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e4101b30 0x7f08e4102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:41.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.284+0000 7f08e1d9b700 1 --2- 172.21.15.70:0/3946332542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08e41024f0 0x7f08e4103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:04:41.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.284+0000 7f08e2d9d700 1 --2- 172.21.15.70:0/3946332542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e4109520 0x7f08e4103790 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00ef30 tx=0x7f08dc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:41.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.284+0000 7f08d37fe700 1 -- 172.21.15.70:0/3946332542 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc0092e0 con 0x7f08e4109520 2024-06-04T05:04:41.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.284+0000 7f08d37fe700 1 -- 172.21.15.70:0/3946332542 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f08dc004bb0 con 0x7f08e4109520 2024-06-04T05:04:41.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.284+0000 7f08e8ac9700 1 -- 172.21.15.70:0/3946332542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08e41a82e0 con 0x7f08e4109520 2024-06-04T05:04:41.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.284+0000 7f08d37fe700 1 -- 172.21.15.70:0/3946332542 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc004020 con 0x7f08e4109520 2024-06-04T05:04:41.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.284+0000 7f08e8ac9700 1 -- 172.21.15.70:0/3946332542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08e41a87f0 con 0x7f08e4109520 2024-06-04T05:04:41.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.287+0000 7f08d37fe700 1 -- 172.21.15.70:0/3946332542 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f08dc01b030 con 0x7f08e4109520 2024-06-04T05:04:41.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.287+0000 7f08d37fe700 1 --2- 172.21.15.70:0/3946332542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08cc0b29f0 0x7f08cc0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.287+0000 7f08e259c700 1 --2- 172.21.15.70:0/3946332542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08cc0b29f0 0x7f08cc0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.287+0000 7f08d37fe700 1 -- 172.21.15.70:0/3946332542 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f08dc0e2100 con 0x7f08e4109520 2024-06-04T05:04:41.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.288+0000 7f08e259c700 1 --2- 172.21.15.70:0/3946332542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08cc0b29f0 0x7f08cc0b4eb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f08d4009fd0 tx=0x7f08d4009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:41.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.288+0000 7f08f335eb80 1 Processor -- start 2024-06-04T05:04:41.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.288+0000 7f08f335eb80 1 -- start start 2024-06-04T05:04:41.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.288+0000 7f08f335eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e834860 0x56241e834c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.288+0000 7f08f335eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e835210 0x56241e838940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.288+0000 7f08f335eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e838f60 0x56241e8393a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.288+0000 7f08f335eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56241e793c90 con 0x56241e838f60 2024-06-04T05:04:41.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.288+0000 7f08f335eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56241e783830 con 0x56241e835210 2024-06-04T05:04:41.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.288+0000 7f08f335eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56241e787870 con 0x56241e834860 2024-06-04T05:04:41.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.289+0000 7f08e2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e838f60 0x56241e8393a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.289+0000 7f08e2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e838f60 0x56241e8393a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54338/0 (socket says 172.21.15.70:54338) 2024-06-04T05:04:41.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.289+0000 7f08e2d9d700 1 -- 172.21.15.70:0/2403974787 learned_addr learned my addr 172.21.15.70:0/2403974787 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:41.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.289+0000 7f08e1d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e835210 0x56241e838940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.289+0000 7f08e259c700 1 --2- 172.21.15.70:0/2403974787 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e834860 0x56241e834c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.289+0000 7f08e2d9d700 1 -- 172.21.15.70:0/2403974787 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e834860 msgr2=0x56241e834c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:41.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.289+0000 7f08e2d9d700 1 --2- 172.21.15.70:0/2403974787 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e834860 0x56241e834c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.289+0000 7f08e2d9d700 1 -- 172.21.15.70:0/2403974787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e835210 msgr2=0x56241e838940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:41.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.289+0000 7f08e2d9d700 1 --2- 172.21.15.70:0/2403974787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e835210 0x56241e838940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.289+0000 7f08e2d9d700 1 -- 172.21.15.70:0/2403974787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56241e6d38d0 con 0x56241e838f60 2024-06-04T05:04:41.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.289+0000 7f08e1d9b700 1 --2- 172.21.15.70:0/2403974787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e835210 0x56241e838940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:41.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.289+0000 7f08e2d9d700 1 --2- 172.21.15.70:0/2403974787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e838f60 0x56241e8393a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f08dc0a9470 tx=0x7f08dc0a90c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6f77b6227b02ce8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:41.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.289+0000 7f08d0ff9700 1 -- 172.21.15.70:0/2403974787 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc0ad890 con 0x56241e838f60 2024-06-04T05:04:41.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.289+0000 7f08d0ff9700 1 -- 172.21.15.70:0/2403974787 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f08dc0b22b0 con 0x56241e838f60 2024-06-04T05:04:41.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.289+0000 7f08d0ff9700 1 -- 172.21.15.70:0/2403974787 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc0b0b80 con 0x56241e838f60 2024-06-04T05:04:41.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.290+0000 7f08f335eb80 1 -- 172.21.15.70:0/2403974787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e838f60 msgr2=0x56241e8393a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:41.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.290+0000 7f08f335eb80 1 --2- 172.21.15.70:0/2403974787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e838f60 0x56241e8393a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f08dc0a9470 tx=0x7f08dc0a90c0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.290+0000 7f08f335eb80 1 -- 172.21.15.70:0/2403974787 shutdown_connections 2024-06-04T05:04:41.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.290+0000 7f08f335eb80 1 --2- 172.21.15.70:0/2403974787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e838f60 0x56241e8393a0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.290+0000 7f08f335eb80 1 --2- 172.21.15.70:0/2403974787 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e834860 0x56241e834c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.290+0000 7f08f335eb80 1 --2- 172.21.15.70:0/2403974787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e835210 0x56241e838940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.290+0000 7f08f335eb80 1 -- 172.21.15.70:0/2403974787 >> 172.21.15.70:0/2403974787 conn(0x56241e762080 msgr2=0x56241e82ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:41.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.290+0000 7f08f335eb80 1 -- 172.21.15.70:0/2403974787 shutdown_connections 2024-06-04T05:04:41.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.290+0000 7f08f335eb80 1 -- 172.21.15.70:0/2403974787 wait complete. 2024-06-04T05:04:41.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08f335eb80 1 -- start start 2024-06-04T05:04:41.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08f335eb80 1 Processor -- start 2024-06-04T05:04:41.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08f335eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e834860 0x56241e840480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08f335eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e835210 0x56241e83d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08f335eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e838f60 0x56241e83dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08f335eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56241e689fa0 con 0x56241e834860 2024-06-04T05:04:41.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08f335eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56241e7257c0 con 0x56241e838f60 2024-06-04T05:04:41.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08f335eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56241e8410c0 con 0x56241e835210 2024-06-04T05:04:41.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08f335eb80 10 client.0 _ll_register_callbacks cb 0x56241e71e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:41.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08e259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e834860 0x56241e840480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08e259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e834860 0x56241e840480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54346/0 (socket says 172.21.15.70:54346) 2024-06-04T05:04:41.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08e259c700 1 -- 172.21.15.70:0/2605016654 learned_addr learned my addr 172.21.15.70:0/2605016654 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:41.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08e2d9d700 1 --2- 172.21.15.70:0/2605016654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e838f60 0x56241e83dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08e1d9b700 1 --2- 172.21.15.70:0/2605016654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e835210 0x56241e83d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08e259c700 1 -- 172.21.15.70:0/2605016654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e835210 msgr2=0x56241e83d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:41.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08e259c700 1 --2- 172.21.15.70:0/2605016654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e835210 0x56241e83d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08e259c700 1 -- 172.21.15.70:0/2605016654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e838f60 msgr2=0x56241e83dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:41.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08e259c700 1 --2- 172.21.15.70:0/2605016654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e838f60 0x56241e83dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08e259c700 1 -- 172.21.15.70:0/2605016654 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56241e722690 con 0x56241e834860 2024-06-04T05:04:41.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.291+0000 7f08e2d9d700 1 --2- 172.21.15.70:0/2605016654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e838f60 0x56241e83dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:41.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.292+0000 7f08c9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:04:41.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.292+0000 7f08e259c700 1 --2- 172.21.15.70:0/2605016654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e834860 0x56241e840480 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f08d400ec90 tx=0x7f08d40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b65d54aff9535320 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:41.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.292+0000 7f08c9ffb700 1 -- 172.21.15.70:0/2605016654 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d4004520 con 0x56241e834860 2024-06-04T05:04:41.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.292+0000 7f08c9ffb700 1 -- 172.21.15.70:0/2605016654 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f08d401f890 con 0x56241e834860 2024-06-04T05:04:41.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.292+0000 7f08f335eb80 1 -- 172.21.15.70:0/2605016654 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08dc0b22b0 con 0x56241e834860 2024-06-04T05:04:41.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.292+0000 7f08c9ffb700 1 -- 172.21.15.70:0/2605016654 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f08d401fad0 con 0x56241e834860 2024-06-04T05:04:41.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.292+0000 7f08f335eb80 10 client.17460 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:41.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.292+0000 7f08f335eb80 1 -- 172.21.15.70:0/2605016654 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56241e84b8f0 con 0x56241e834860 2024-06-04T05:04:41.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.293+0000 7f08c9ffb700 1 -- 172.21.15.70:0/2605016654 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f08dc0b22b0 con 0x56241e834860 2024-06-04T05:04:41.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.293+0000 7f08c9ffb700 1 -- 172.21.15.70:0/2605016654 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f08d402f070 con 0x56241e834860 2024-06-04T05:04:41.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.293+0000 7f08f335eb80 10 client.17460 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:41.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.293+0000 7f08f335eb80 10 client.17460 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:04:41.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.293+0000 7f08f335eb80 20 client.17460 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:41.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.293+0000 7f08f335eb80 1 --2- 172.21.15.70:0/2605016654 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56241e84de80 0x56241e850270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.293+0000 7f08f335eb80 4 client.17460 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:04:41.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.293+0000 7f08f335eb80 1 -- 172.21.15.70:0/2605016654 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56241e8009e0 con 0x56241e84de80 2024-06-04T05:04:41.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.293+0000 7f08e1d9b700 1 --2- 172.21.15.70:0/2605016654 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56241e84de80 0x56241e850270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.294+0000 7f08c9ffb700 10 client.17460 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:04:41.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.294+0000 7f08e1d9b700 1 --2- 172.21.15.70:0/2605016654 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56241e84de80 0x56241e850270 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ea2699fceff8bc73 server_cookie=8996874b4cc0d55a in_seq=0 out_seq=0 2024-06-04T05:04:41.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.294+0000 7f08c9ffb700 1 -- 172.21.15.70:0/2605016654 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56241e8009e0 con 0x56241e84de80 2024-06-04T05:04:41.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.294+0000 7f08c9ffb700 10 client.17460 handle_command_reply: tid=0 2024-06-04T05:04:41.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.294+0000 7f08f335eb80 1 client.17460 shutdown 2024-06-04T05:04:41.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.294+0000 7f08f335eb80 1 -- 172.21.15.70:0/2605016654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e834860 msgr2=0x56241e840480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:41.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.294+0000 7f08f335eb80 1 --2- 172.21.15.70:0/2605016654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e834860 0x56241e840480 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f08d400ec90 tx=0x7f08d40047c0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.294+0000 7f08f335eb80 1 -- 172.21.15.70:0/2605016654 shutdown_connections 2024-06-04T05:04:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.294+0000 7f08f335eb80 1 --2- 172.21.15.70:0/2605016654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e838f60 0x56241e83dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.294+0000 7f08f335eb80 1 --2- 172.21.15.70:0/2605016654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e834860 0x56241e840480 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.294+0000 7f08f335eb80 1 --2- 172.21.15.70:0/2605016654 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56241e84de80 0x56241e850270 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.294+0000 7f08f335eb80 1 --2- 172.21.15.70:0/2605016654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e835210 0x56241e83d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.294+0000 7f08f335eb80 1 -- 172.21.15.70:0/2605016654 >> 172.21.15.70:0/2605016654 conn(0x56241e762080 msgr2=0x56241e82e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.294+0000 7f08c9ffb700 0 client.17460 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:04:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.295+0000 7f08f335eb80 1 -- 172.21.15.70:0/2605016654 shutdown_connections 2024-06-04T05:04:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.295+0000 7f08f335eb80 1 -- 172.21.15.70:0/2605016654 wait complete. 2024-06-04T05:04:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.295+0000 7f08f335eb80 20 client.17460 trim_cache size 0 max 16384 2024-06-04T05:04:41.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.311+0000 7f08f335eb80 1 Processor -- start 2024-06-04T05:04:41.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.311+0000 7f08f335eb80 1 -- start start 2024-06-04T05:04:41.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.311+0000 7f08f335eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e835210 0x56241e84a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.311+0000 7f08f335eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e838f60 0x56241e83f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.311+0000 7f08f335eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e82ec90 0x56241e82f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.311+0000 7f08f335eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08e41b00b0 con 0x56241e82ec90 2024-06-04T05:04:41.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.311+0000 7f08f335eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08e41b1bf0 con 0x56241e838f60 2024-06-04T05:04:41.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.311+0000 7f08f335eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f08e41b0780 con 0x56241e835210 2024-06-04T05:04:41.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.311+0000 7f08e2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e82ec90 0x56241e82f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.311+0000 7f08e2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e82ec90 0x56241e82f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54350/0 (socket says 172.21.15.70:54350) 2024-06-04T05:04:41.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.311+0000 7f08e2d9d700 1 -- 172.21.15.70:0/3560284692 learned_addr learned my addr 172.21.15.70:0/3560284692 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:41.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.311+0000 7f08e1d9b700 1 --2- 172.21.15.70:0/3560284692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e838f60 0x56241e83f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.311+0000 7f08e259c700 1 --2- 172.21.15.70:0/3560284692 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e835210 0x56241e84a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.312+0000 7f08e2d9d700 1 -- 172.21.15.70:0/3560284692 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e835210 msgr2=0x56241e84a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:41.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.312+0000 7f08e2d9d700 1 --2- 172.21.15.70:0/3560284692 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e835210 0x56241e84a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.312+0000 7f08e2d9d700 1 -- 172.21.15.70:0/3560284692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e838f60 msgr2=0x56241e83f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:41.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.312+0000 7f08e2d9d700 1 --2- 172.21.15.70:0/3560284692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e838f60 0x56241e83f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.312+0000 7f08e2d9d700 1 -- 172.21.15.70:0/3560284692 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56241e834e90 con 0x56241e82ec90 2024-06-04T05:04:41.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.312+0000 7f08e2d9d700 1 --2- 172.21.15.70:0/3560284692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e82ec90 0x56241e82f0b0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00ef00 tx=0x7f08dc009480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=865a7f26253b8857 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:41.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.312+0000 7f08cbfff700 1 -- 172.21.15.70:0/3560284692 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc0ad890 con 0x56241e82ec90 2024-06-04T05:04:41.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.312+0000 7f08cbfff700 1 -- 172.21.15.70:0/3560284692 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f08dc0a9740 con 0x56241e82ec90 2024-06-04T05:04:41.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.312+0000 7f08cbfff700 1 -- 172.21.15.70:0/3560284692 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc0a4e50 con 0x56241e82ec90 2024-06-04T05:04:41.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.313+0000 7f08f335eb80 1 -- 172.21.15.70:0/3560284692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e82ec90 msgr2=0x56241e82f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:41.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.313+0000 7f08f335eb80 1 --2- 172.21.15.70:0/3560284692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e82ec90 0x56241e82f0b0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00ef00 tx=0x7f08dc009480 comp rx=0 tx=0).stop 2024-06-04T05:04:41.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.313+0000 7f08f335eb80 1 -- 172.21.15.70:0/3560284692 shutdown_connections 2024-06-04T05:04:41.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.313+0000 7f08f335eb80 1 --2- 172.21.15.70:0/3560284692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e82ec90 0x56241e82f0b0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.313+0000 7f08f335eb80 1 --2- 172.21.15.70:0/3560284692 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e835210 0x56241e84a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.313+0000 7f08f335eb80 1 --2- 172.21.15.70:0/3560284692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e838f60 0x56241e83f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.313+0000 7f08f335eb80 1 -- 172.21.15.70:0/3560284692 >> 172.21.15.70:0/3560284692 conn(0x56241e762080 msgr2=0x56241e83efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:41.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.313+0000 7f08f335eb80 1 -- 172.21.15.70:0/3560284692 shutdown_connections 2024-06-04T05:04:41.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.313+0000 7f08f335eb80 1 -- 172.21.15.70:0/3560284692 wait complete. 2024-06-04T05:04:41.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.313+0000 7f08f335eb80 1 -- start start 2024-06-04T05:04:41.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08f335eb80 1 Processor -- start 2024-06-04T05:04:41.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08f335eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e82ec90 0x56241e8467e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08f335eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e835210 0x56241e843b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08f335eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e838f60 0x56241e8440c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08f335eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08e41b1520 con 0x56241e82ec90 2024-06-04T05:04:41.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08f335eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08e41b0e50 con 0x56241e838f60 2024-06-04T05:04:41.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08f335eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f08e41ae070 con 0x56241e835210 2024-06-04T05:04:41.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08f335eb80 10 client.0 _ll_register_callbacks cb 0x56241e71e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:41.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08e259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e82ec90 0x56241e8467e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08e259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e82ec90 0x56241e8467e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54356/0 (socket says 172.21.15.70:54356) 2024-06-04T05:04:41.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08e259c700 1 -- 172.21.15.70:0/1058808436 learned_addr learned my addr 172.21.15.70:0/1058808436 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:41.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08e1d9b700 1 --2- 172.21.15.70:0/1058808436 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e835210 0x56241e843b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08e259c700 1 -- 172.21.15.70:0/1058808436 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e835210 msgr2=0x56241e843b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:41.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08e2d9d700 1 --2- 172.21.15.70:0/1058808436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e838f60 0x56241e8440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08e259c700 1 --2- 172.21.15.70:0/1058808436 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e835210 0x56241e843b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08e259c700 1 -- 172.21.15.70:0/1058808436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e838f60 msgr2=0x56241e8440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:41.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08e259c700 1 --2- 172.21.15.70:0/1058808436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e838f60 0x56241e8440c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08e259c700 1 -- 172.21.15.70:0/1058808436 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56241e8446b0 con 0x56241e82ec90 2024-06-04T05:04:41.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08e2d9d700 1 --2- 172.21.15.70:0/1058808436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e838f60 0x56241e8440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:04:41.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08e259c700 1 --2- 172.21.15.70:0/1058808436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e82ec90 0x56241e8467e0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f08d400e680 tx=0x7f08d401f990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f31d16b8290c84b2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:41.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.314+0000 7f08c9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:04:41.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.315+0000 7f08c9ffb700 1 -- 172.21.15.70:0/1058808436 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d4031b40 con 0x56241e82ec90 2024-06-04T05:04:41.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.315+0000 7f08c9ffb700 1 -- 172.21.15.70:0/1058808436 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f08d4031ce0 con 0x56241e82ec90 2024-06-04T05:04:41.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.315+0000 7f08f335eb80 1 -- 172.21.15.70:0/1058808436 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08dc0a9740 con 0x56241e82ec90 2024-06-04T05:04:41.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.315+0000 7f08c9ffb700 1 -- 172.21.15.70:0/1058808436 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f08d4003680 con 0x56241e82ec90 2024-06-04T05:04:41.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.315+0000 7f08f335eb80 10 client.17466 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:41.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.315+0000 7f08f335eb80 1 -- 172.21.15.70:0/1058808436 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56241e8b87e0 con 0x56241e82ec90 2024-06-04T05:04:41.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.315+0000 7f08c9ffb700 1 -- 172.21.15.70:0/1058808436 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f08dc0a9740 con 0x56241e82ec90 2024-06-04T05:04:41.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.315+0000 7f08c9ffb700 1 -- 172.21.15.70:0/1058808436 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f08d4042070 con 0x56241e82ec90 2024-06-04T05:04:41.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.315+0000 7f08f335eb80 10 client.17466 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:41.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.315+0000 7f08f335eb80 10 client.17466 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:04:41.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.315+0000 7f08f335eb80 20 client.17466 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:41.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.315+0000 7f08f335eb80 1 --2- 172.21.15.70:0/1058808436 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56241e8b8e60 0x56241e8bb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:41.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.315+0000 7f08f335eb80 4 client.17466 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:04:41.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.315+0000 7f08f335eb80 1 -- 172.21.15.70:0/1058808436 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56241e795be0 con 0x56241e8b8e60 2024-06-04T05:04:41.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.315+0000 7f08e1d9b700 1 --2- 172.21.15.70:0/1058808436 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56241e8b8e60 0x56241e8bb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:41.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.316+0000 7f08c9ffb700 10 client.17466 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:04:41.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.316+0000 7f08e1d9b700 1 --2- 172.21.15.70:0/1058808436 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56241e8b8e60 0x56241e8bb280 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c4cf0f5958b85517 server_cookie=427dbc1d8f9cc3d4 in_seq=0 out_seq=0 2024-06-04T05:04:41.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.316+0000 7f08c9ffb700 1 -- 172.21.15.70:0/1058808436 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56241e795be0 con 0x56241e8b8e60 2024-06-04T05:04:41.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.316+0000 7f08c9ffb700 10 client.17466 handle_command_reply: tid=0 2024-06-04T05:04:41.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.316+0000 7f08f335eb80 1 client.17466 shutdown 2024-06-04T05:04:41.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.316+0000 7f08f335eb80 1 -- 172.21.15.70:0/1058808436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e82ec90 msgr2=0x56241e8467e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:41.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.316+0000 7f08f335eb80 1 --2- 172.21.15.70:0/1058808436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e82ec90 0x56241e8467e0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f08d400e680 tx=0x7f08d401f990 comp rx=0 tx=0).stop 2024-06-04T05:04:41.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.316+0000 7f08f335eb80 1 -- 172.21.15.70:0/1058808436 shutdown_connections 2024-06-04T05:04:41.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.316+0000 7f08f335eb80 1 --2- 172.21.15.70:0/1058808436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241e838f60 0x56241e8440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.316+0000 7f08f335eb80 1 --2- 172.21.15.70:0/1058808436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56241e82ec90 0x56241e8467e0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.316+0000 7f08f335eb80 1 --2- 172.21.15.70:0/1058808436 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x56241e8b8e60 0x56241e8bb280 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.316+0000 7f08f335eb80 1 --2- 172.21.15.70:0/1058808436 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56241e835210 0x56241e843b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.316+0000 7f08c9ffb700 0 client.17466 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:04:41.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.316+0000 7f08f335eb80 1 -- 172.21.15.70:0/1058808436 >> 172.21.15.70:0/1058808436 conn(0x56241e762080 msgr2=0x56241e8378b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:41.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.316+0000 7f08f335eb80 1 -- 172.21.15.70:0/1058808436 shutdown_connections 2024-06-04T05:04:41.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.316+0000 7f08f335eb80 1 -- 172.21.15.70:0/1058808436 wait complete. 2024-06-04T05:04:41.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.316+0000 7f08f335eb80 20 client.17466 trim_cache size 0 max 16384 2024-06-04T05:04:41.324 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:04:41.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.317+0000 7f08e8ac9700 1 -- 172.21.15.70:0/3946332542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08cc0b29f0 msgr2=0x7f08cc0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:41.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.317+0000 7f08e8ac9700 1 --2- 172.21.15.70:0/3946332542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08cc0b29f0 0x7f08cc0b4eb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f08d4009fd0 tx=0x7f08d4009260 comp rx=0 tx=0).stop 2024-06-04T05:04:41.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.317+0000 7f08e8ac9700 1 -- 172.21.15.70:0/3946332542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e4109520 msgr2=0x7f08e4103790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:41.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.317+0000 7f08e8ac9700 1 --2- 172.21.15.70:0/3946332542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e4109520 0x7f08e4103790 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00ef30 tx=0x7f08dc00c3a0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.317+0000 7f08e8ac9700 1 -- 172.21.15.70:0/3946332542 shutdown_connections 2024-06-04T05:04:41.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.317+0000 7f08e8ac9700 1 --2- 172.21.15.70:0/3946332542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08cc0b29f0 0x7f08cc0b4eb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.317+0000 7f08e8ac9700 1 --2- 172.21.15.70:0/3946332542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e4109520 0x7f08e4103790 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.317+0000 7f08e8ac9700 1 --2- 172.21.15.70:0/3946332542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e4101b30 0x7f08e4102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.317+0000 7f08e8ac9700 1 --2- 172.21.15.70:0/3946332542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08e41024f0 0x7f08e4103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:41.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.317+0000 7f08e8ac9700 1 -- 172.21.15.70:0/3946332542 >> 172.21.15.70:0/3946332542 conn(0x7f08e4078ab0 msgr2=0x7f08e41003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:41.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.317+0000 7f08e8ac9700 1 -- 172.21.15.70:0/3946332542 shutdown_connections 2024-06-04T05:04:41.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:41.317+0000 7f08e8ac9700 1 -- 172.21.15.70:0/3946332542 wait complete. 2024-06-04T05:04:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:41 smithi070 ceph-mon[32631]: pgmap v230: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-06-04T05:04:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:41 smithi112 ceph-mon[36453]: pgmap v230: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-06-04T05:04:41.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 124877 2.00 MB/sec execute 214 sec latency 21.188 ms 2024-06-04T05:04:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:41 smithi098 ceph-mon[35569]: pgmap v230: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-06-04T05:04:42.115 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:04:42.382 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:04:42.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 125409 2.00 MB/sec execute 215 sec latency 14.987 ms 2024-06-04T05:04:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:43 smithi070 ceph-mon[32631]: pgmap v231: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-06-04T05:04:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:43 smithi112 ceph-mon[36453]: pgmap v231: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-06-04T05:04:43.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 125997 2.01 MB/sec execute 216 sec latency 19.686 ms 2024-06-04T05:04:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:43 smithi098 ceph-mon[35569]: pgmap v231: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-06-04T05:04:44.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.085+0000 7f39d0b35700 1 -- 172.21.15.70:0/352626536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39cc102440 msgr2=0x7f39cc1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:44.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.085+0000 7f39d0b35700 1 --2- 172.21.15.70:0/352626536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39cc102440 0x7f39cc1028c0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f39c0009990 tx=0x7f39c0009ca0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.085+0000 7f39d0b35700 1 -- 172.21.15.70:0/352626536 shutdown_connections 2024-06-04T05:04:44.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.085+0000 7f39d0b35700 1 --2- 172.21.15.70:0/352626536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39cc102e00 0x7f39cc10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.085+0000 7f39d0b35700 1 --2- 172.21.15.70:0/352626536 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39cc108520 0x7f39cc108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.085+0000 7f39d0b35700 1 --2- 172.21.15.70:0/352626536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39cc102440 0x7f39cc1028c0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.085+0000 7f39d0b35700 1 -- 172.21.15.70:0/352626536 >> 172.21.15.70:0/352626536 conn(0x7f39cc0793f0 msgr2=0x7f39cc079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:44.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.085+0000 7f39d0b35700 1 -- 172.21.15.70:0/352626536 shutdown_connections 2024-06-04T05:04:44.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.085+0000 7f39d0b35700 1 -- 172.21.15.70:0/352626536 wait complete. 2024-06-04T05:04:44.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.086+0000 7f39d0b35700 1 Processor -- start 2024-06-04T05:04:44.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.086+0000 7f39d0b35700 1 -- start start 2024-06-04T05:04:44.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.086+0000 7f39d0b35700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39cc102440 0x7f39cc19bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.086+0000 7f39d0b35700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39cc102e00 0x7f39cc19c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.086+0000 7f39d0b35700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39cc108520 0x7f39cc1a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.086+0000 7f39d0b35700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc112480 con 0x7f39cc108520 2024-06-04T05:04:44.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.086+0000 7f39d0b35700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc112300 con 0x7f39cc102440 2024-06-04T05:04:44.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.086+0000 7f39d0b35700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc112600 con 0x7f39cc102e00 2024-06-04T05:04:44.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.086+0000 7f39cad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39cc108520 0x7f39cc1a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.086+0000 7f39c9d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39cc102e00 0x7f39cc19c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.086+0000 7f39cad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39cc108520 0x7f39cc1a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54368/0 (socket says 172.21.15.70:54368) 2024-06-04T05:04:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.086+0000 7f39cad9d700 1 -- 172.21.15.70:0/3946189845 learned_addr learned my addr 172.21.15.70:0/3946189845 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.086+0000 7f39ca59c700 1 --2- 172.21.15.70:0/3946189845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39cc102440 0x7f39cc19bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.087+0000 7f39c9d9b700 1 -- 172.21.15.70:0/3946189845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39cc102440 msgr2=0x7f39cc19bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.087+0000 7f39c9d9b700 1 --2- 172.21.15.70:0/3946189845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39cc102440 0x7f39cc19bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.087+0000 7f39c9d9b700 1 -- 172.21.15.70:0/3946189845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39cc108520 msgr2=0x7f39cc1a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.087+0000 7f39c9d9b700 1 --2- 172.21.15.70:0/3946189845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39cc108520 0x7f39cc1a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.087+0000 7f39c9d9b700 1 -- 172.21.15.70:0/3946189845 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39cc1a0d60 con 0x7f39cc102e00 2024-06-04T05:04:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.087+0000 7f39cad9d700 1 --2- 172.21.15.70:0/3946189845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39cc108520 0x7f39cc1a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.087+0000 7f39c9d9b700 1 --2- 172.21.15.70:0/3946189845 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39cc102e00 0x7f39cc19c2e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f39c0000c00 tx=0x7f39c00047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.087+0000 7f39bb7fe700 1 -- 172.21.15.70:0/3946189845 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0026070 con 0x7f39cc102e00 2024-06-04T05:04:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.087+0000 7f39bb7fe700 1 -- 172.21.15.70:0/3946189845 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39c0021410 con 0x7f39cc102e00 2024-06-04T05:04:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.087+0000 7f39d0b35700 1 -- 172.21.15.70:0/3946189845 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39cc1a0ff0 con 0x7f39cc102e00 2024-06-04T05:04:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.087+0000 7f39bb7fe700 1 -- 172.21.15.70:0/3946189845 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0017650 con 0x7f39cc102e00 2024-06-04T05:04:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.087+0000 7f39d0b35700 1 -- 172.21.15.70:0/3946189845 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39cc1a88d0 con 0x7f39cc102e00 2024-06-04T05:04:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.090+0000 7f39bb7fe700 1 -- 172.21.15.70:0/3946189845 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f39c0029070 con 0x7f39cc102e00 2024-06-04T05:04:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.090+0000 7f39bb7fe700 1 --2- 172.21.15.70:0/3946189845 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39b40b2a70 0x7f39b40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.091+0000 7f39ca59c700 1 --2- 172.21.15.70:0/3946189845 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39b40b2a70 0x7f39b40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.091+0000 7f39bb7fe700 1 -- 172.21.15.70:0/3946189845 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f39c00e33a0 con 0x7f39cc102e00 2024-06-04T05:04:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.091+0000 7f39ca59c700 1 --2- 172.21.15.70:0/3946189845 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39b40b2a70 0x7f39b40b4f30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f39bc009be0 tx=0x7f39bc009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.092+0000 7f39db3cab80 1 Processor -- start 2024-06-04T05:04:44.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.092+0000 7f39db3cab80 1 -- start start 2024-06-04T05:04:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.092+0000 7f39db3cab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e4f9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.092+0000 7f39db3cab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fa210 0x55ef7e4fd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.092+0000 7f39db3cab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4fdf60 0x55ef7e4fe3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.092+0000 7f39db3cab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ef7e458c90 con 0x55ef7e4f9860 2024-06-04T05:04:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.092+0000 7f39db3cab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ef7e448830 con 0x55ef7e4fdf60 2024-06-04T05:04:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.092+0000 7f39db3cab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ef7e44c870 con 0x55ef7e4fa210 2024-06-04T05:04:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.092+0000 7f39ca59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e4f9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.092+0000 7f39ca59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e4f9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54382/0 (socket says 172.21.15.70:54382) 2024-06-04T05:04:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.092+0000 7f39ca59c700 1 -- 172.21.15.70:0/69185142 learned_addr learned my addr 172.21.15.70:0/69185142 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.092+0000 7f39c9d9b700 1 --2- 172.21.15.70:0/69185142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fa210 0x55ef7e4fd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.093+0000 7f39ca59c700 1 -- 172.21.15.70:0/69185142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fa210 msgr2=0x55ef7e4fd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.093+0000 7f39ca59c700 1 --2- 172.21.15.70:0/69185142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fa210 0x55ef7e4fd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.093+0000 7f39ca59c700 1 -- 172.21.15.70:0/69185142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4fdf60 msgr2=0x55ef7e4fe3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.093+0000 7f39cad9d700 1 --2- 172.21.15.70:0/69185142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4fdf60 0x55ef7e4fe3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.093+0000 7f39ca59c700 1 --2- 172.21.15.70:0/69185142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4fdf60 0x55ef7e4fe3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.093+0000 7f39ca59c700 1 -- 172.21.15.70:0/69185142 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef7e3988d0 con 0x55ef7e4f9860 2024-06-04T05:04:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.093+0000 7f39cad9d700 1 --2- 172.21.15.70:0/69185142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4fdf60 0x55ef7e4fe3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:04:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.093+0000 7f39ca59c700 1 --2- 172.21.15.70:0/69185142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e4f9c40 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f39bc011d70 tx=0x7f39bc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cccd9e8e6bd4f8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.093+0000 7f39b8ff9700 1 -- 172.21.15.70:0/69185142 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc011da0 con 0x55ef7e4f9860 2024-06-04T05:04:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.093+0000 7f39b8ff9700 1 -- 172.21.15.70:0/69185142 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39bc004020 con 0x55ef7e4f9860 2024-06-04T05:04:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.093+0000 7f39b8ff9700 1 -- 172.21.15.70:0/69185142 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc003c10 con 0x55ef7e4f9860 2024-06-04T05:04:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.094+0000 7f39db3cab80 1 -- 172.21.15.70:0/69185142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 msgr2=0x55ef7e4f9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.094+0000 7f39db3cab80 1 --2- 172.21.15.70:0/69185142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e4f9c40 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f39bc011d70 tx=0x7f39bc014000 comp rx=0 tx=0).stop 2024-06-04T05:04:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.094+0000 7f39db3cab80 1 -- 172.21.15.70:0/69185142 shutdown_connections 2024-06-04T05:04:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.094+0000 7f39db3cab80 1 --2- 172.21.15.70:0/69185142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4fdf60 0x55ef7e4fe3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.094+0000 7f39db3cab80 1 --2- 172.21.15.70:0/69185142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e4f9c40 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.094+0000 7f39db3cab80 1 --2- 172.21.15.70:0/69185142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fa210 0x55ef7e4fd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.094+0000 7f39db3cab80 1 -- 172.21.15.70:0/69185142 >> 172.21.15.70:0/69185142 conn(0x55ef7e427080 msgr2=0x55ef7e4f3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.094+0000 7f39db3cab80 1 -- 172.21.15.70:0/69185142 shutdown_connections 2024-06-04T05:04:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.094+0000 7f39db3cab80 1 -- 172.21.15.70:0/69185142 wait complete. 2024-06-04T05:04:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.095+0000 7f39db3cab80 1 -- start start 2024-06-04T05:04:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.095+0000 7f39db3cab80 1 Processor -- start 2024-06-04T05:04:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.095+0000 7f39db3cab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e505440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.095+0000 7f39db3cab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4fa210 0x55ef7e502740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.095+0000 7f39db3cab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fdf60 0x55ef7e502c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.095+0000 7f39db3cab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ef7e34efa0 con 0x55ef7e4f9860 2024-06-04T05:04:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.095+0000 7f39db3cab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ef7e3ea7c0 con 0x55ef7e4fa210 2024-06-04T05:04:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.095+0000 7f39db3cab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ef7e506120 con 0x55ef7e4fdf60 2024-06-04T05:04:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.095+0000 7f39db3cab80 10 client.0 _ll_register_callbacks cb 0x55ef7e3e3920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.095+0000 7f39ca59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e505440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.095+0000 7f39ca59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e505440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54394/0 (socket says 172.21.15.70:54394) 2024-06-04T05:04:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.095+0000 7f39ca59c700 1 -- 172.21.15.70:0/4248554158 learned_addr learned my addr 172.21.15.70:0/4248554158 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.095+0000 7f39cad9d700 1 --2- 172.21.15.70:0/4248554158 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fdf60 0x55ef7e502c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.095+0000 7f39c9d9b700 1 --2- 172.21.15.70:0/4248554158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4fa210 0x55ef7e502740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.096+0000 7f39ca59c700 1 -- 172.21.15.70:0/4248554158 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fdf60 msgr2=0x55ef7e502c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.096+0000 7f39ca59c700 1 --2- 172.21.15.70:0/4248554158 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fdf60 0x55ef7e502c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.096+0000 7f39ca59c700 1 -- 172.21.15.70:0/4248554158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4fa210 msgr2=0x55ef7e502740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.096+0000 7f39ca59c700 1 --2- 172.21.15.70:0/4248554158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4fa210 0x55ef7e502740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.096+0000 7f39ca59c700 1 -- 172.21.15.70:0/4248554158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef7e3e7690 con 0x55ef7e4f9860 2024-06-04T05:04:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.096+0000 7f39ca59c700 1 --2- 172.21.15.70:0/4248554158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e505440 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0037f0 tx=0x7f39bc005490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef529e5c7523343 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.096+0000 7f39b1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:04:44.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.096+0000 7f39b1ffb700 1 -- 172.21.15.70:0/4248554158 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc003c60 con 0x55ef7e4f9860 2024-06-04T05:04:44.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.096+0000 7f39b1ffb700 1 -- 172.21.15.70:0/4248554158 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f39bc003e00 con 0x55ef7e4f9860 2024-06-04T05:04:44.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.096+0000 7f39db3cab80 1 -- 172.21.15.70:0/4248554158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39bc004020 con 0x55ef7e4f9860 2024-06-04T05:04:44.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.096+0000 7f39b1ffb700 1 -- 172.21.15.70:0/4248554158 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f39bc005680 con 0x55ef7e4f9860 2024-06-04T05:04:44.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.096+0000 7f39db3cab80 10 client.17478 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:44.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.096+0000 7f39db3cab80 1 -- 172.21.15.70:0/4248554158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ef7e510850 con 0x55ef7e4f9860 2024-06-04T05:04:44.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.096+0000 7f39b1ffb700 1 -- 172.21.15.70:0/4248554158 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39bc0398d0 con 0x55ef7e4f9860 2024-06-04T05:04:44.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.097+0000 7f39b1ffb700 1 -- 172.21.15.70:0/4248554158 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f39bc01e600 con 0x55ef7e4f9860 2024-06-04T05:04:44.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.097+0000 7f39db3cab80 10 client.17478 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:44.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.097+0000 7f39db3cab80 10 client.17478 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:04:44.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.097+0000 7f39db3cab80 20 client.17478 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:44.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.097+0000 7f39db3cab80 1 --2- 172.21.15.70:0/4248554158 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ef7e512d20 0x55ef7e515110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.097+0000 7f39db3cab80 4 client.17478 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:04:44.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.097+0000 7f39db3cab80 1 -- 172.21.15.70:0/4248554158 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f39bc004020 con 0x55ef7e512d20 2024-06-04T05:04:44.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.097+0000 7f39c9d9b700 1 --2- 172.21.15.70:0/4248554158 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ef7e512d20 0x55ef7e515110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.097+0000 7f39c9d9b700 1 --2- 172.21.15.70:0/4248554158 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ef7e512d20 0x55ef7e515110 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3b9baad7e36caae1 server_cookie=800c1da9b6b09e62 in_seq=0 out_seq=0 2024-06-04T05:04:44.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.097+0000 7f39b1ffb700 10 client.17478 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:04:44.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.098+0000 7f39b1ffb700 1 -- 172.21.15.70:0/4248554158 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f39bc004020 con 0x55ef7e512d20 2024-06-04T05:04:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.098+0000 7f39b1ffb700 10 client.17478 handle_command_reply: tid=0 2024-06-04T05:04:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.098+0000 7f39db3cab80 1 client.17478 shutdown 2024-06-04T05:04:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.098+0000 7f39db3cab80 1 -- 172.21.15.70:0/4248554158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 msgr2=0x55ef7e505440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.098+0000 7f39db3cab80 1 --2- 172.21.15.70:0/4248554158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e505440 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0037f0 tx=0x7f39bc005490 comp rx=0 tx=0).stop 2024-06-04T05:04:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.098+0000 7f39db3cab80 1 -- 172.21.15.70:0/4248554158 shutdown_connections 2024-06-04T05:04:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.098+0000 7f39db3cab80 1 --2- 172.21.15.70:0/4248554158 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fdf60 0x55ef7e502c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.098+0000 7f39db3cab80 1 --2- 172.21.15.70:0/4248554158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e505440 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.098+0000 7f39db3cab80 1 --2- 172.21.15.70:0/4248554158 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ef7e512d20 0x55ef7e515110 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.098+0000 7f39db3cab80 1 --2- 172.21.15.70:0/4248554158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4fa210 0x55ef7e502740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.098+0000 7f39db3cab80 1 -- 172.21.15.70:0/4248554158 >> 172.21.15.70:0/4248554158 conn(0x55ef7e427080 msgr2=0x55ef7e4f3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.098+0000 7f39b1ffb700 0 client.17478 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:04:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.099+0000 7f39db3cab80 1 -- 172.21.15.70:0/4248554158 shutdown_connections 2024-06-04T05:04:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.099+0000 7f39db3cab80 1 -- 172.21.15.70:0/4248554158 wait complete. 2024-06-04T05:04:44.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.099+0000 7f39db3cab80 20 client.17478 trim_cache size 0 max 16384 2024-06-04T05:04:44.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.115+0000 7f39db3cab80 1 Processor -- start 2024-06-04T05:04:44.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.115+0000 7f39db3cab80 1 -- start start 2024-06-04T05:04:44.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.115+0000 7f39db3cab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4f9860 0x55ef7e50f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.115+0000 7f39db3cab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fa210 0x55ef7e4f9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.115+0000 7f39db3cab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4fdf60 0x55ef7e51a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.115+0000 7f39db3cab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc1affb0 con 0x55ef7e4fdf60 2024-06-04T05:04:44.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.115+0000 7f39db3cab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc1b1af0 con 0x55ef7e4f9860 2024-06-04T05:04:44.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.115+0000 7f39db3cab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc1b0680 con 0x55ef7e4fa210 2024-06-04T05:04:44.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.115+0000 7f39ca59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4f9860 0x55ef7e50f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.115+0000 7f39ca59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4f9860 0x55ef7e50f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:47204/0 (socket says 172.21.15.70:47204) 2024-06-04T05:04:44.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.115+0000 7f39c9d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fa210 0x55ef7e4f9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.115+0000 7f39ca59c700 1 -- 172.21.15.70:0/181587515 learned_addr learned my addr 172.21.15.70:0/181587515 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:44.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.115+0000 7f39cad9d700 1 --2- 172.21.15.70:0/181587515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4fdf60 0x55ef7e51a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.116+0000 7f39ca59c700 1 -- 172.21.15.70:0/181587515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fa210 msgr2=0x55ef7e4f9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:44.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.116+0000 7f39ca59c700 1 --2- 172.21.15.70:0/181587515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fa210 0x55ef7e4f9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.116+0000 7f39ca59c700 1 -- 172.21.15.70:0/181587515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4fdf60 msgr2=0x55ef7e51a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:44.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.116+0000 7f39ca59c700 1 --2- 172.21.15.70:0/181587515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4fdf60 0x55ef7e51a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.116+0000 7f39ca59c700 1 -- 172.21.15.70:0/181587515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef7e4f5520 con 0x55ef7e4f9860 2024-06-04T05:04:44.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.116+0000 7f39ca59c700 1 --2- 172.21.15.70:0/181587515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4f9860 0x55ef7e50f1f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f39bc01ea50 tx=0x7f39bc0093c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c8d47ae4c86cbf2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:44.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.116+0000 7f39b3fff700 1 -- 172.21.15.70:0/181587515 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc0119e0 con 0x55ef7e4f9860 2024-06-04T05:04:44.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.116+0000 7f39b3fff700 1 -- 172.21.15.70:0/181587515 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39bc01ec40 con 0x55ef7e4f9860 2024-06-04T05:04:44.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.116+0000 7f39b3fff700 1 -- 172.21.15.70:0/181587515 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc0396a0 con 0x55ef7e4f9860 2024-06-04T05:04:44.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.117+0000 7f39db3cab80 1 -- 172.21.15.70:0/181587515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4f9860 msgr2=0x55ef7e50f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:44.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.117+0000 7f39db3cab80 1 --2- 172.21.15.70:0/181587515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4f9860 0x55ef7e50f1f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f39bc01ea50 tx=0x7f39bc0093c0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.117+0000 7f39db3cab80 1 -- 172.21.15.70:0/181587515 shutdown_connections 2024-06-04T05:04:44.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.117+0000 7f39db3cab80 1 --2- 172.21.15.70:0/181587515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4fdf60 0x55ef7e51a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.117+0000 7f39db3cab80 1 --2- 172.21.15.70:0/181587515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4f9860 0x55ef7e50f1f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.117+0000 7f39db3cab80 1 --2- 172.21.15.70:0/181587515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fa210 0x55ef7e4f9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.117+0000 7f39db3cab80 1 -- 172.21.15.70:0/181587515 >> 172.21.15.70:0/181587515 conn(0x55ef7e427080 msgr2=0x55ef7e504800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.117+0000 7f39db3cab80 1 -- 172.21.15.70:0/181587515 shutdown_connections 2024-06-04T05:04:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.117+0000 7f39db3cab80 1 -- 172.21.15.70:0/181587515 wait complete. 2024-06-04T05:04:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.117+0000 7f39db3cab80 1 -- start start 2024-06-04T05:04:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.117+0000 7f39db3cab80 1 Processor -- start 2024-06-04T05:04:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.117+0000 7f39db3cab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e50b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39db3cab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4fa210 0x55ef7e508c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39db3cab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fdf60 0x55ef7e5091c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39db3cab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc1b1420 con 0x55ef7e4f9860 2024-06-04T05:04:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39db3cab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc1b0d50 con 0x55ef7e4fa210 2024-06-04T05:04:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39db3cab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc1adf70 con 0x55ef7e4fdf60 2024-06-04T05:04:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39db3cab80 10 client.0 _ll_register_callbacks cb 0x55ef7e3e3920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39ca59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e50b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39ca59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e50b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54406/0 (socket says 172.21.15.70:54406) 2024-06-04T05:04:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39ca59c700 1 -- 172.21.15.70:0/476993073 learned_addr learned my addr 172.21.15.70:0/476993073 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39c9d9b700 1 --2- 172.21.15.70:0/476993073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4fa210 0x55ef7e508c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39cad9d700 1 --2- 172.21.15.70:0/476993073 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fdf60 0x55ef7e5091c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39c9d9b700 1 -- 172.21.15.70:0/476993073 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fdf60 msgr2=0x55ef7e5091c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39c9d9b700 1 --2- 172.21.15.70:0/476993073 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fdf60 0x55ef7e5091c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39c9d9b700 1 -- 172.21.15.70:0/476993073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 msgr2=0x55ef7e50b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39c9d9b700 1 --2- 172.21.15.70:0/476993073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e50b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39c9d9b700 1 -- 172.21.15.70:0/476993073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef7e509800 con 0x55ef7e4fa210 2024-06-04T05:04:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39ca59c700 1 --2- 172.21.15.70:0/476993073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e50b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39c9d9b700 1 --2- 172.21.15.70:0/476993073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4fa210 0x55ef7e508c80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f39c00e0a60 tx=0x7f39c001a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2894641caa2032a9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39b1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:04:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39b1ffb700 1 -- 172.21.15.70:0/476993073 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c00aa680 con 0x55ef7e4fa210 2024-06-04T05:04:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.118+0000 7f39b1ffb700 1 -- 172.21.15.70:0/476993073 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f39c00a6910 con 0x55ef7e4fa210 2024-06-04T05:04:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.119+0000 7f39db3cab80 1 -- 172.21.15.70:0/476993073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39bc01ec40 con 0x55ef7e4fa210 2024-06-04T05:04:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.119+0000 7f39b1ffb700 1 -- 172.21.15.70:0/476993073 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f39c00aea30 con 0x55ef7e4fa210 2024-06-04T05:04:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.119+0000 7f39db3cab80 10 client.26699 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.119+0000 7f39db3cab80 1 -- 172.21.15.70:0/476993073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ef7e585c40 con 0x55ef7e4fa210 2024-06-04T05:04:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.119+0000 7f39b1ffb700 1 -- 172.21.15.70:0/476993073 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39bc01ec40 con 0x55ef7e4fa210 2024-06-04T05:04:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.119+0000 7f39b1ffb700 1 -- 172.21.15.70:0/476993073 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f39c00218c0 con 0x55ef7e4fa210 2024-06-04T05:04:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.119+0000 7f39db3cab80 10 client.26699 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.119+0000 7f39db3cab80 10 client.26699 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:04:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.119+0000 7f39db3cab80 20 client.26699 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.119+0000 7f39db3cab80 1 --2- 172.21.15.70:0/476993073 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ef7e586380 0x55ef7e5887a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.119+0000 7f39db3cab80 4 client.26699 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:04:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.119+0000 7f39db3cab80 1 -- 172.21.15.70:0/476993073 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ef7e4c59e0 con 0x55ef7e586380 2024-06-04T05:04:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.119+0000 7f39ca59c700 1 --2- 172.21.15.70:0/476993073 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ef7e586380 0x55ef7e5887a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.120+0000 7f39b1ffb700 10 client.26699 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:04:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.120+0000 7f39ca59c700 1 --2- 172.21.15.70:0/476993073 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ef7e586380 0x55ef7e5887a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=37952543632f0c53 server_cookie=6b1620b7c72511a1 in_seq=0 out_seq=0 2024-06-04T05:04:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.120+0000 7f39b1ffb700 1 -- 172.21.15.70:0/476993073 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ef7e4c59e0 con 0x55ef7e586380 2024-06-04T05:04:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.120+0000 7f39b1ffb700 10 client.26699 handle_command_reply: tid=0 2024-06-04T05:04:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.120+0000 7f39db3cab80 1 client.26699 shutdown 2024-06-04T05:04:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.120+0000 7f39db3cab80 1 -- 172.21.15.70:0/476993073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4fa210 msgr2=0x55ef7e508c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.120+0000 7f39db3cab80 1 --2- 172.21.15.70:0/476993073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4fa210 0x55ef7e508c80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f39c00e0a60 tx=0x7f39c001a040 comp rx=0 tx=0).stop 2024-06-04T05:04:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.120+0000 7f39db3cab80 1 -- 172.21.15.70:0/476993073 shutdown_connections 2024-06-04T05:04:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.120+0000 7f39db3cab80 1 --2- 172.21.15.70:0/476993073 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef7e4fdf60 0x55ef7e5091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.120+0000 7f39db3cab80 1 --2- 172.21.15.70:0/476993073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef7e4f9860 0x55ef7e50b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.120+0000 7f39db3cab80 1 --2- 172.21.15.70:0/476993073 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ef7e586380 0x55ef7e5887a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.120+0000 7f39db3cab80 1 --2- 172.21.15.70:0/476993073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef7e4fa210 0x55ef7e508c80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.120+0000 7f39db3cab80 1 -- 172.21.15.70:0/476993073 >> 172.21.15.70:0/476993073 conn(0x55ef7e427080 msgr2=0x55ef7e503f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.120+0000 7f39b1ffb700 0 client.26699 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:04:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.120+0000 7f39db3cab80 1 -- 172.21.15.70:0/476993073 shutdown_connections 2024-06-04T05:04:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.120+0000 7f39db3cab80 1 -- 172.21.15.70:0/476993073 wait complete. 2024-06-04T05:04:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.120+0000 7f39db3cab80 20 client.26699 trim_cache size 0 max 16384 2024-06-04T05:04:44.129 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:04:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.121+0000 7f39d0b35700 1 -- 172.21.15.70:0/3946189845 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39b40b2a70 msgr2=0x7f39b40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.121+0000 7f39d0b35700 1 --2- 172.21.15.70:0/3946189845 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39b40b2a70 0x7f39b40b4f30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f39bc009be0 tx=0x7f39bc009260 comp rx=0 tx=0).stop 2024-06-04T05:04:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.121+0000 7f39d0b35700 1 -- 172.21.15.70:0/3946189845 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39cc102e00 msgr2=0x7f39cc19c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.121+0000 7f39d0b35700 1 --2- 172.21.15.70:0/3946189845 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39cc102e00 0x7f39cc19c2e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f39c0000c00 tx=0x7f39c00047b0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.121+0000 7f39d0b35700 1 -- 172.21.15.70:0/3946189845 shutdown_connections 2024-06-04T05:04:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.121+0000 7f39d0b35700 1 --2- 172.21.15.70:0/3946189845 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39b40b2a70 0x7f39b40b4f30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.121+0000 7f39d0b35700 1 --2- 172.21.15.70:0/3946189845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39cc108520 0x7f39cc1a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.121+0000 7f39d0b35700 1 --2- 172.21.15.70:0/3946189845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39cc102440 0x7f39cc19bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.121+0000 7f39d0b35700 1 --2- 172.21.15.70:0/3946189845 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39cc102e00 0x7f39cc19c2e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.121+0000 7f39d0b35700 1 -- 172.21.15.70:0/3946189845 >> 172.21.15.70:0/3946189845 conn(0x7f39cc0793f0 msgr2=0x7f39cc06b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.121+0000 7f39d0b35700 1 -- 172.21.15.70:0/3946189845 shutdown_connections 2024-06-04T05:04:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:44.121+0000 7f39d0b35700 1 -- 172.21.15.70:0/3946189845 wait complete. 2024-06-04T05:04:44.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 126579 2.00 MB/sec execute 217 sec latency 39.335 ms 2024-06-04T05:04:44.819 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:04:45.083 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:04:45.535 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:45 smithi070 ceph-mon[32631]: pgmap v232: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-06-04T05:04:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:45 smithi112 ceph-mon[36453]: pgmap v232: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-06-04T05:04:45.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 127196 2.02 MB/sec execute 218 sec latency 14.855 ms 2024-06-04T05:04:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:45 smithi098 ceph-mon[35569]: pgmap v232: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-06-04T05:04:46.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 127764 2.03 MB/sec execute 219 sec latency 12.040 ms 2024-06-04T05:04:46.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.807+0000 7f2317316700 1 -- 172.21.15.70:0/1544694201 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2310102ff0 msgr2=0x7f2310103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:46.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.807+0000 7f2317316700 1 --2- 172.21.15.70:0/1544694201 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2310102ff0 0x7f2310103470 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f2300009990 tx=0x7f2300009ca0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.807+0000 7f2317316700 1 -- 172.21.15.70:0/1544694201 shutdown_connections 2024-06-04T05:04:46.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.807+0000 7f2317316700 1 --2- 172.21.15.70:0/1544694201 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f23101039b0 0x7f231010f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.807+0000 7f2317316700 1 --2- 172.21.15.70:0/1544694201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2310107470 0x7f2310107850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.807+0000 7f2317316700 1 --2- 172.21.15.70:0/1544694201 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2310102ff0 0x7f2310103470 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.807+0000 7f2317316700 1 -- 172.21.15.70:0/1544694201 >> 172.21.15.70:0/1544694201 conn(0x7f23100777e0 msgr2=0x7f2310077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:46.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.807+0000 7f2317316700 1 -- 172.21.15.70:0/1544694201 shutdown_connections 2024-06-04T05:04:46.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.808+0000 7f2317316700 1 -- 172.21.15.70:0/1544694201 wait complete. 2024-06-04T05:04:46.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.808+0000 7f2317316700 1 Processor -- start 2024-06-04T05:04:46.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.808+0000 7f2317316700 1 -- start start 2024-06-04T05:04:46.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.808+0000 7f2317316700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2310102ff0 0x7f2310104170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.808+0000 7f2317316700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23101039b0 0x7f23101046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.808+0000 7f2317316700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2310107470 0x7f2310104bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.808+0000 7f2317316700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f23101124e0 con 0x7f23101039b0 2024-06-04T05:04:46.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.808+0000 7f2317316700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2310112360 con 0x7f2310107470 2024-06-04T05:04:46.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.808+0000 7f2317316700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2310112660 con 0x7f2310102ff0 2024-06-04T05:04:46.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.809+0000 7f23148b1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23101039b0 0x7f23101046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.809+0000 7f23148b1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23101039b0 0x7f23101046b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33750/0 (socket says 172.21.15.70:33750) 2024-06-04T05:04:46.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.809+0000 7f23150b2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2310102ff0 0x7f2310104170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.809+0000 7f23148b1700 1 -- 172.21.15.70:0/1420298029 learned_addr learned my addr 172.21.15.70:0/1420298029 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:46.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.809+0000 7f23158b3700 1 --2- 172.21.15.70:0/1420298029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2310107470 0x7f2310104bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.809+0000 7f23148b1700 1 -- 172.21.15.70:0/1420298029 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2310102ff0 msgr2=0x7f2310104170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:46.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.809+0000 7f23148b1700 1 --2- 172.21.15.70:0/1420298029 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2310102ff0 0x7f2310104170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.809+0000 7f23148b1700 1 -- 172.21.15.70:0/1420298029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2310107470 msgr2=0x7f2310104bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:46.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.809+0000 7f23148b1700 1 --2- 172.21.15.70:0/1420298029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2310107470 0x7f2310104bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.809+0000 7f23148b1700 1 -- 172.21.15.70:0/1420298029 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23101a8090 con 0x7f23101039b0 2024-06-04T05:04:46.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.809+0000 7f23150b2700 1 --2- 172.21.15.70:0/1420298029 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2310102ff0 0x7f2310104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:46.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.809+0000 7f23148b1700 1 --2- 172.21.15.70:0/1420298029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23101039b0 0x7f23101046b0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f2300006010 tx=0x7f2300003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:46.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.809+0000 7f23067fc700 1 -- 172.21.15.70:0/1420298029 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300027070 con 0x7f23101039b0 2024-06-04T05:04:46.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.809+0000 7f23067fc700 1 -- 172.21.15.70:0/1420298029 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f230000be10 con 0x7f23101039b0 2024-06-04T05:04:46.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.809+0000 7f2317316700 1 -- 172.21.15.70:0/1420298029 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23101a8320 con 0x7f23101039b0 2024-06-04T05:04:46.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.809+0000 7f23067fc700 1 -- 172.21.15.70:0/1420298029 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300022e00 con 0x7f23101039b0 2024-06-04T05:04:46.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.809+0000 7f2317316700 1 -- 172.21.15.70:0/1420298029 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23101a8770 con 0x7f23101039b0 2024-06-04T05:04:46.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.812+0000 7f23067fc700 1 -- 172.21.15.70:0/1420298029 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f230002a070 con 0x7f23101039b0 2024-06-04T05:04:46.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.812+0000 7f23067fc700 1 --2- 172.21.15.70:0/1420298029 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f22fc0b29f0 0x7f22fc0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.812+0000 7f23067fc700 1 -- 172.21.15.70:0/1420298029 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f23000e3670 con 0x7f23101039b0 2024-06-04T05:04:46.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.812+0000 7f23150b2700 1 --2- 172.21.15.70:0/1420298029 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f22fc0b29f0 0x7f22fc0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.813+0000 7f23150b2700 1 --2- 172.21.15.70:0/1420298029 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f22fc0b29f0 0x7f22fc0b4eb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f230c005950 tx=0x7f230c00a300 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:46.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f2321babb80 1 Processor -- start 2024-06-04T05:04:46.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f2321babb80 1 -- start start 2024-06-04T05:04:46.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f2321babb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf15320 0x557fdaf15700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f2321babb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf0fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f2321babb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf10260 0x557fdaf1c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f2321babb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557fdae6dd70 con 0x557fdaf15320 2024-06-04T05:04:46.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f2321babb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557fdae5d910 con 0x557fdaf10260 2024-06-04T05:04:46.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f2321babb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557fdae61950 con 0x557fdaf0f8a0 2024-06-04T05:04:46.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f23150b2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf15320 0x557fdaf15700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f23148b1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf0fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f23150b2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf15320 0x557fdaf15700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33754/0 (socket says 172.21.15.70:33754) 2024-06-04T05:04:46.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f23148b1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf0fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:34784/0 (socket says 172.21.15.70:34784) 2024-06-04T05:04:46.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f23150b2700 1 -- 172.21.15.70:0/773319181 learned_addr learned my addr 172.21.15.70:0/773319181 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:46.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f23158b3700 1 --2- 172.21.15.70:0/773319181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf10260 0x557fdaf1c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f23148b1700 1 -- 172.21.15.70:0/773319181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf10260 msgr2=0x557fdaf1c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:46.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f23148b1700 1 --2- 172.21.15.70:0/773319181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf10260 0x557fdaf1c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f23148b1700 1 -- 172.21.15.70:0/773319181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf15320 msgr2=0x557fdaf15700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:46.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f23148b1700 1 --2- 172.21.15.70:0/773319181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf15320 0x557fdaf15700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.814+0000 7f23148b1700 1 -- 172.21.15.70:0/773319181 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fdae06fb0 con 0x557fdaf0f8a0 2024-06-04T05:04:46.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.815+0000 7f23148b1700 1 --2- 172.21.15.70:0/773319181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf0fd20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f23000e0d90 tx=0x7f23000aa1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb1db160c85cb62a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:46.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.815+0000 7f22fb7fe700 1 -- 172.21.15.70:0/773319181 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23000aa4c0 con 0x557fdaf0f8a0 2024-06-04T05:04:46.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.815+0000 7f22fb7fe700 1 -- 172.21.15.70:0/773319181 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f23000ab0b0 con 0x557fdaf0f8a0 2024-06-04T05:04:46.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.815+0000 7f22fb7fe700 1 -- 172.21.15.70:0/773319181 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23000b10d0 con 0x557fdaf0f8a0 2024-06-04T05:04:46.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.816+0000 7f2321babb80 1 -- 172.21.15.70:0/773319181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 msgr2=0x557fdaf0fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:46.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.816+0000 7f2321babb80 1 --2- 172.21.15.70:0/773319181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf0fd20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f23000e0d90 tx=0x7f23000aa1a0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.816+0000 7f2321babb80 1 -- 172.21.15.70:0/773319181 shutdown_connections 2024-06-04T05:04:46.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.816+0000 7f2321babb80 1 --2- 172.21.15.70:0/773319181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf10260 0x557fdaf1c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.816+0000 7f2321babb80 1 --2- 172.21.15.70:0/773319181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf15320 0x557fdaf15700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.816+0000 7f2321babb80 1 --2- 172.21.15.70:0/773319181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf0fd20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.816+0000 7f2321babb80 1 -- 172.21.15.70:0/773319181 >> 172.21.15.70:0/773319181 conn(0x557fdae3b240 msgr2=0x557fdaf0a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:46.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.816+0000 7f2321babb80 1 -- 172.21.15.70:0/773319181 shutdown_connections 2024-06-04T05:04:46.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.816+0000 7f2321babb80 1 -- 172.21.15.70:0/773319181 wait complete. 2024-06-04T05:04:46.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.816+0000 7f2321babb80 1 -- start start 2024-06-04T05:04:46.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.816+0000 7f2321babb80 1 Processor -- start 2024-06-04T05:04:46.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f2321babb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf1bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f2321babb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf10260 0x557fdaf19030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f2321babb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf15320 0x557fdaf19570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f2321babb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557fdad63fa0 con 0x557fdaf10260 2024-06-04T05:04:46.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f2321babb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557fdadc5da0 con 0x557fdaf15320 2024-06-04T05:04:46.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f2321babb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557fdae61950 con 0x557fdaf0f8a0 2024-06-04T05:04:46.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f2321babb80 10 client.0 _ll_register_callbacks cb 0x557fdadf8950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:46.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f23150b2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf1bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f23148b1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf10260 0x557fdaf19030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f23150b2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf1bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:34798/0 (socket says 172.21.15.70:34798) 2024-06-04T05:04:46.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f23150b2700 1 -- 172.21.15.70:0/1801248408 learned_addr learned my addr 172.21.15.70:0/1801248408 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:46.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f23158b3700 1 --2- 172.21.15.70:0/1801248408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf15320 0x557fdaf19570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f23150b2700 1 -- 172.21.15.70:0/1801248408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf15320 msgr2=0x557fdaf19570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:46.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f23150b2700 1 --2- 172.21.15.70:0/1801248408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf15320 0x557fdaf19570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f23150b2700 1 -- 172.21.15.70:0/1801248408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf10260 msgr2=0x557fdaf19030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:46.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f23150b2700 1 --2- 172.21.15.70:0/1801248408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf10260 0x557fdaf19030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f23150b2700 1 -- 172.21.15.70:0/1801248408 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fdadad8d0 con 0x557fdaf0f8a0 2024-06-04T05:04:46.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f23148b1700 1 --2- 172.21.15.70:0/1801248408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf10260 0x557fdaf19030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:46.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f23150b2700 1 --2- 172.21.15.70:0/1801248408 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf1bd30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f230c01de30 tx=0x7f230c0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51aec4310aeb238c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:46.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.817+0000 7f22f8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:04:46.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.818+0000 7f22f8ff9700 1 -- 172.21.15.70:0/1801248408 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c01de60 con 0x557fdaf0f8a0 2024-06-04T05:04:46.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.818+0000 7f22f8ff9700 1 -- 172.21.15.70:0/1801248408 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f230c010cd0 con 0x557fdaf0f8a0 2024-06-04T05:04:46.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.818+0000 7f2321babb80 1 -- 172.21.15.70:0/1801248408 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23000ab0b0 con 0x557fdaf0f8a0 2024-06-04T05:04:46.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.818+0000 7f22f8ff9700 1 -- 172.21.15.70:0/1801248408 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f230c016070 con 0x557fdaf0f8a0 2024-06-04T05:04:46.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.818+0000 7f2321babb80 10 client.26788 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:46.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.818+0000 7f2321babb80 1 -- 172.21.15.70:0/1801248408 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557fdae17760 con 0x557fdaf0f8a0 2024-06-04T05:04:46.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.818+0000 7f22f8ff9700 1 -- 172.21.15.70:0/1801248408 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f23000ab0b0 con 0x557fdaf0f8a0 2024-06-04T05:04:46.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.818+0000 7f22f8ff9700 1 -- 172.21.15.70:0/1801248408 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f230c024030 con 0x557fdaf0f8a0 2024-06-04T05:04:46.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.818+0000 7f2321babb80 10 client.26788 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:46.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.818+0000 7f2321babb80 10 client.26788 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:04:46.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.818+0000 7f2321babb80 20 client.26788 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:46.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.818+0000 7f2321babb80 1 --2- 172.21.15.70:0/1801248408 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557fdaf28fc0 0x557fdaf2b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.818+0000 7f2321babb80 4 client.26788 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:04:46.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.818+0000 7f2321babb80 1 -- 172.21.15.70:0/1801248408 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557fdae6fcc0 con 0x557fdaf28fc0 2024-06-04T05:04:46.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.818+0000 7f23148b1700 1 --2- 172.21.15.70:0/1801248408 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557fdaf28fc0 0x557fdaf2b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.819+0000 7f23148b1700 1 --2- 172.21.15.70:0/1801248408 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557fdaf28fc0 0x557fdaf2b3b0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5991ce7180ffb5c server_cookie=d9aba83442b4ec71 in_seq=0 out_seq=0 2024-06-04T05:04:46.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.819+0000 7f22f8ff9700 10 client.26788 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:46.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.819+0000 7f22f8ff9700 1 -- 172.21.15.70:0/1801248408 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557fdae6fcc0 con 0x557fdaf28fc0 2024-06-04T05:04:46.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.819+0000 7f22f8ff9700 10 client.26788 handle_command_reply: tid=0 2024-06-04T05:04:46.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.820+0000 7f2321babb80 1 client.26788 shutdown 2024-06-04T05:04:46.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.820+0000 7f2321babb80 1 -- 172.21.15.70:0/1801248408 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 msgr2=0x557fdaf1bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:46.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.820+0000 7f2321babb80 1 --2- 172.21.15.70:0/1801248408 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf1bd30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f230c01de30 tx=0x7f230c0047c0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.820+0000 7f2321babb80 1 -- 172.21.15.70:0/1801248408 shutdown_connections 2024-06-04T05:04:46.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.820+0000 7f2321babb80 1 --2- 172.21.15.70:0/1801248408 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557fdaf28fc0 0x557fdaf2b3b0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.820+0000 7f2321babb80 1 --2- 172.21.15.70:0/1801248408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf15320 0x557fdaf19570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.820+0000 7f2321babb80 1 --2- 172.21.15.70:0/1801248408 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf1bd30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.820+0000 7f2321babb80 1 --2- 172.21.15.70:0/1801248408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf10260 0x557fdaf19030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.820+0000 7f2321babb80 1 -- 172.21.15.70:0/1801248408 >> 172.21.15.70:0/1801248408 conn(0x557fdae3b240 msgr2=0x557fdaf09af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:46.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.820+0000 7f22f8ff9700 0 client.26788 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:46.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.820+0000 7f2321babb80 1 -- 172.21.15.70:0/1801248408 shutdown_connections 2024-06-04T05:04:46.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.820+0000 7f2321babb80 1 -- 172.21.15.70:0/1801248408 wait complete. 2024-06-04T05:04:46.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.820+0000 7f2321babb80 20 client.26788 trim_cache size 0 max 16384 2024-06-04T05:04:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.837+0000 7f2321babb80 1 Processor -- start 2024-06-04T05:04:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.837+0000 7f2321babb80 1 -- start start 2024-06-04T05:04:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f2321babb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf0f8a0 0x557fdaf25360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f2321babb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf10260 0x557fdaf17f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f2321babb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf15320 0x557fdaf0a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f2321babb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f23101aff60 con 0x557fdaf15320 2024-06-04T05:04:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f2321babb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f23101b1aa0 con 0x557fdaf0f8a0 2024-06-04T05:04:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f2321babb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f23101b0630 con 0x557fdaf10260 2024-06-04T05:04:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f23150b2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf0f8a0 0x557fdaf25360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f23148b1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf10260 0x557fdaf17f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f23150b2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf0f8a0 0x557fdaf25360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46772/0 (socket says 172.21.15.70:46772) 2024-06-04T05:04:46.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f23148b1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf10260 0x557fdaf17f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:34810/0 (socket says 172.21.15.70:34810) 2024-06-04T05:04:46.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f23150b2700 1 -- 172.21.15.70:0/839437217 learned_addr learned my addr 172.21.15.70:0/839437217 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:46.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f23158b3700 1 --2- 172.21.15.70:0/839437217 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf15320 0x557fdaf0a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f23150b2700 1 -- 172.21.15.70:0/839437217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf10260 msgr2=0x557fdaf17f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:46.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f23150b2700 1 --2- 172.21.15.70:0/839437217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf10260 0x557fdaf17f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f23150b2700 1 -- 172.21.15.70:0/839437217 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf15320 msgr2=0x557fdaf0a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:46.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f23150b2700 1 --2- 172.21.15.70:0/839437217 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf15320 0x557fdaf0a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f23150b2700 1 -- 172.21.15.70:0/839437217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fdaf0c380 con 0x557fdaf0f8a0 2024-06-04T05:04:46.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f23150b2700 1 --2- 172.21.15.70:0/839437217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf0f8a0 0x557fdaf25360 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f230c000c00 tx=0x7f230c005770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb5f0a716bd47a68 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:46.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.838+0000 7f22faffd700 1 -- 172.21.15.70:0/839437217 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c03b070 con 0x557fdaf0f8a0 2024-06-04T05:04:46.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.839+0000 7f22faffd700 1 -- 172.21.15.70:0/839437217 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f230c010cd0 con 0x557fdaf0f8a0 2024-06-04T05:04:46.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.839+0000 7f22faffd700 1 -- 172.21.15.70:0/839437217 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c01de60 con 0x557fdaf0f8a0 2024-06-04T05:04:46.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.839+0000 7f2321babb80 1 -- 172.21.15.70:0/839437217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf0f8a0 msgr2=0x557fdaf25360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:46.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.839+0000 7f2321babb80 1 --2- 172.21.15.70:0/839437217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf0f8a0 0x557fdaf25360 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f230c000c00 tx=0x7f230c005770 comp rx=0 tx=0).stop 2024-06-04T05:04:46.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.839+0000 7f2321babb80 1 -- 172.21.15.70:0/839437217 shutdown_connections 2024-06-04T05:04:46.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.839+0000 7f2321babb80 1 --2- 172.21.15.70:0/839437217 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf15320 0x557fdaf0a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.840+0000 7f2321babb80 1 --2- 172.21.15.70:0/839437217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf0f8a0 0x557fdaf25360 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.840+0000 7f2321babb80 1 --2- 172.21.15.70:0/839437217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf10260 0x557fdaf17f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.840+0000 7f2321babb80 1 -- 172.21.15.70:0/839437217 >> 172.21.15.70:0/839437217 conn(0x557fdae3b240 msgr2=0x557fdaf1a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:46.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.840+0000 7f2321babb80 1 -- 172.21.15.70:0/839437217 shutdown_connections 2024-06-04T05:04:46.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.840+0000 7f2321babb80 1 -- 172.21.15.70:0/839437217 wait complete. 2024-06-04T05:04:46.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.840+0000 7f2321babb80 1 -- start start 2024-06-04T05:04:46.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.840+0000 7f2321babb80 1 Processor -- start 2024-06-04T05:04:46.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.840+0000 7f2321babb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf2fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.840+0000 7f2321babb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf10260 0x557fdaf301d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.840+0000 7f2321babb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf15320 0x557fdaf30710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.840+0000 7f2321babb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f23101b13d0 con 0x557fdaf15320 2024-06-04T05:04:46.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.840+0000 7f2321babb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f23101b0d00 con 0x557fdaf10260 2024-06-04T05:04:46.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.840+0000 7f2321babb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f23101adf20 con 0x557fdaf0f8a0 2024-06-04T05:04:46.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.841+0000 7f2321babb80 10 client.0 _ll_register_callbacks cb 0x557fdadf8950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:46.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.841+0000 7f23150b2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf2fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.841+0000 7f23158b3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf15320 0x557fdaf30710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.841+0000 7f23150b2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf2fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:34824/0 (socket says 172.21.15.70:34824) 2024-06-04T05:04:46.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.841+0000 7f23150b2700 1 -- 172.21.15.70:0/553479126 learned_addr learned my addr 172.21.15.70:0/553479126 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:46.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.841+0000 7f23148b1700 1 --2- 172.21.15.70:0/553479126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf10260 0x557fdaf301d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.841+0000 7f23150b2700 1 -- 172.21.15.70:0/553479126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf10260 msgr2=0x557fdaf301d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:46.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.841+0000 7f23150b2700 1 --2- 172.21.15.70:0/553479126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf10260 0x557fdaf301d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.841+0000 7f23150b2700 1 -- 172.21.15.70:0/553479126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf15320 msgr2=0x557fdaf30710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:46.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.841+0000 7f23150b2700 1 --2- 172.21.15.70:0/553479126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf15320 0x557fdaf30710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.841+0000 7f23150b2700 1 -- 172.21.15.70:0/553479126 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fdaf20650 con 0x557fdaf0f8a0 2024-06-04T05:04:46.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.841+0000 7f23158b3700 1 --2- 172.21.15.70:0/553479126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf15320 0x557fdaf30710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:46.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.841+0000 7f23150b2700 1 --2- 172.21.15.70:0/553479126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf2fc90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f230c0055f0 tx=0x7f230c010e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54bacf0215ef3a16 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:46.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.841+0000 7f22f8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:04:46.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.841+0000 7f22f8ff9700 1 -- 172.21.15.70:0/553479126 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c03b070 con 0x557fdaf0f8a0 2024-06-04T05:04:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.841+0000 7f22f8ff9700 1 -- 172.21.15.70:0/553479126 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f230c036890 con 0x557fdaf0f8a0 2024-06-04T05:04:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.841+0000 7f2321babb80 1 -- 172.21.15.70:0/553479126 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f230c010cd0 con 0x557fdaf0f8a0 2024-06-04T05:04:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.842+0000 7f22f8ff9700 1 -- 172.21.15.70:0/553479126 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f230c03f450 con 0x557fdaf0f8a0 2024-06-04T05:04:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.842+0000 7f2321babb80 10 client.26794 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.842+0000 7f2321babb80 1 -- 172.21.15.70:0/553479126 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557fdaf21930 con 0x557fdaf0f8a0 2024-06-04T05:04:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.842+0000 7f22f8ff9700 1 -- 172.21.15.70:0/553479126 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f230c010cd0 con 0x557fdaf0f8a0 2024-06-04T05:04:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.842+0000 7f22f8ff9700 1 -- 172.21.15.70:0/553479126 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f230c024040 con 0x557fdaf0f8a0 2024-06-04T05:04:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.842+0000 7f2321babb80 10 client.26794 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.842+0000 7f2321babb80 10 client.26794 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:04:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.842+0000 7f2321babb80 20 client.26794 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.842+0000 7f2321babb80 1 --2- 172.21.15.70:0/553479126 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557fdaf93b70 0x557fdaf95f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.842+0000 7f2321babb80 4 client.26794 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:04:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.842+0000 7f2321babb80 1 -- 172.21.15.70:0/553479126 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557fdae694a0 con 0x557fdaf93b70 2024-06-04T05:04:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.842+0000 7f23148b1700 1 --2- 172.21.15.70:0/553479126 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557fdaf93b70 0x557fdaf95f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.842+0000 7f22f8ff9700 10 client.26794 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.843+0000 7f23148b1700 1 --2- 172.21.15.70:0/553479126 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557fdaf93b70 0x557fdaf95f60 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=107e94be0ab48b7e server_cookie=8d1bd23f86a3a124 in_seq=0 out_seq=0 2024-06-04T05:04:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.843+0000 7f22f8ff9700 1 -- 172.21.15.70:0/553479126 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557fdae694a0 con 0x557fdaf93b70 2024-06-04T05:04:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.843+0000 7f22f8ff9700 10 client.26794 handle_command_reply: tid=0 2024-06-04T05:04:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.843+0000 7f2321babb80 1 client.26794 shutdown 2024-06-04T05:04:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.843+0000 7f2321babb80 1 -- 172.21.15.70:0/553479126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 msgr2=0x557fdaf2fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.843+0000 7f2321babb80 1 --2- 172.21.15.70:0/553479126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf2fc90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f230c0055f0 tx=0x7f230c010e80 comp rx=0 tx=0).stop 2024-06-04T05:04:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.843+0000 7f2321babb80 1 -- 172.21.15.70:0/553479126 shutdown_connections 2024-06-04T05:04:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.843+0000 7f2321babb80 1 --2- 172.21.15.70:0/553479126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557fdaf15320 0x557fdaf30710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.843+0000 7f2321babb80 1 --2- 172.21.15.70:0/553479126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557fdaf0f8a0 0x557fdaf2fc90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.843+0000 7f2321babb80 1 --2- 172.21.15.70:0/553479126 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557fdaf93b70 0x557fdaf95f60 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.843+0000 7f2321babb80 1 --2- 172.21.15.70:0/553479126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fdaf10260 0x557fdaf301d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.843+0000 7f2321babb80 1 -- 172.21.15.70:0/553479126 >> 172.21.15.70:0/553479126 conn(0x557fdae3b240 msgr2=0x557fdaf0f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.843+0000 7f22f8ff9700 0 client.26794 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.843+0000 7f2321babb80 1 -- 172.21.15.70:0/553479126 shutdown_connections 2024-06-04T05:04:46.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.843+0000 7f2321babb80 1 -- 172.21.15.70:0/553479126 wait complete. 2024-06-04T05:04:46.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.843+0000 7f2321babb80 20 client.26794 trim_cache size 0 max 16384 2024-06-04T05:04:46.852 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:04:46.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.844+0000 7f2317316700 1 -- 172.21.15.70:0/1420298029 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f22fc0b29f0 msgr2=0x7f22fc0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:46.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.844+0000 7f2317316700 1 --2- 172.21.15.70:0/1420298029 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f22fc0b29f0 0x7f22fc0b4eb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f230c005950 tx=0x7f230c00a300 comp rx=0 tx=0).stop 2024-06-04T05:04:46.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.844+0000 7f2317316700 1 -- 172.21.15.70:0/1420298029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23101039b0 msgr2=0x7f23101046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:46.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.844+0000 7f2317316700 1 --2- 172.21.15.70:0/1420298029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23101039b0 0x7f23101046b0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f2300006010 tx=0x7f2300003c20 comp rx=0 tx=0).stop 2024-06-04T05:04:46.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.844+0000 7f2317316700 1 -- 172.21.15.70:0/1420298029 shutdown_connections 2024-06-04T05:04:46.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.844+0000 7f2317316700 1 --2- 172.21.15.70:0/1420298029 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f22fc0b29f0 0x7f22fc0b4eb0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.844+0000 7f2317316700 1 --2- 172.21.15.70:0/1420298029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2310107470 0x7f2310104bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.844+0000 7f2317316700 1 --2- 172.21.15.70:0/1420298029 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2310102ff0 0x7f2310104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.844+0000 7f2317316700 1 --2- 172.21.15.70:0/1420298029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23101039b0 0x7f23101046b0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:46.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.844+0000 7f2317316700 1 -- 172.21.15.70:0/1420298029 >> 172.21.15.70:0/1420298029 conn(0x7f23100777e0 msgr2=0x7f23101020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:46.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.844+0000 7f2317316700 1 -- 172.21.15.70:0/1420298029 shutdown_connections 2024-06-04T05:04:46.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:46.844+0000 7f2317316700 1 -- 172.21.15.70:0/1420298029 wait complete. 2024-06-04T05:04:47.577 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:47 smithi070 ceph-mon[32631]: pgmap v233: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 2.0 MiB/s wr, 193 op/s 2024-06-04T05:04:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:47 smithi112 ceph-mon[36453]: pgmap v233: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 2.0 MiB/s wr, 193 op/s 2024-06-04T05:04:47.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 128119 2.03 MB/sec execute 220 sec latency 13.173 ms 2024-06-04T05:04:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:47 smithi098 ceph-mon[35569]: pgmap v233: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 2.0 MiB/s wr, 193 op/s 2024-06-04T05:04:48.605 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:04:48.605 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:04:48.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 128450 2.02 MB/sec execute 221 sec latency 18.828 ms 2024-06-04T05:04:48.865 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:04:49.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:49 smithi070 ceph-mon[32631]: pgmap v234: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-06-04T05:04:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:49 smithi112 ceph-mon[36453]: pgmap v234: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-06-04T05:04:49.725 INFO:tasks.workunit.client.0.smithi070.stdout: 1 128944 2.02 MB/sec execute 222 sec latency 15.922 ms 2024-06-04T05:04:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:49 smithi098 ceph-mon[35569]: pgmap v234: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-06-04T05:04:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.509+0000 7ff3166c5640 1 -- 172.21.15.70:0/997700664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff31010d040 msgr2=0x7ff31010f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.509+0000 7ff3166c5640 1 --2- 172.21.15.70:0/997700664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff31010d040 0x7ff31010f4b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff308009be0 tx=0x7ff30802a750 comp rx=0 tx=0).stop 2024-06-04T05:04:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.509+0000 7ff3166c5640 1 -- 172.21.15.70:0/997700664 shutdown_connections 2024-06-04T05:04:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.509+0000 7ff3166c5640 1 --2- 172.21.15.70:0/997700664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff310104620 0x7ff31010cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.509+0000 7ff3166c5640 1 --2- 172.21.15.70:0/997700664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff31010d040 0x7ff31010f4b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.509+0000 7ff3166c5640 1 --2- 172.21.15.70:0/997700664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff310103d00 0x7ff3101040e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.509+0000 7ff3166c5640 1 -- 172.21.15.70:0/997700664 >> 172.21.15.70:0/997700664 conn(0x7ff310078ad0 msgr2=0x7ff310078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.509+0000 7ff3166c5640 1 -- 172.21.15.70:0/997700664 shutdown_connections 2024-06-04T05:04:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.509+0000 7ff3166c5640 1 -- 172.21.15.70:0/997700664 wait complete. 2024-06-04T05:04:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff3166c5640 1 Processor -- start 2024-06-04T05:04:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff3166c5640 1 -- start start 2024-06-04T05:04:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff3166c5640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff310103d00 0x7ff3101990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff3166c5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff310104620 0x7ff3101995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff3166c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff31010d040 0x7ff31019d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff3166c5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff310112b70 con 0x7ff310104620 2024-06-04T05:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff3166c5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff3101129f0 con 0x7ff31010d040 2024-06-04T05:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff3166c5640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff310112cf0 con 0x7ff310103d00 2024-06-04T05:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff307fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff310104620 0x7ff3101995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff30ffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff310103d00 0x7ff3101990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff307fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff310104620 0x7ff3101995f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33802/0 (socket says 172.21.15.70:33802) 2024-06-04T05:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff307fff640 1 -- 172.21.15.70:0/2404892568 learned_addr learned my addr 172.21.15.70:0/2404892568 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff30ffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff310103d00 0x7ff3101990b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:34828/0 (socket says 172.21.15.70:34828) 2024-06-04T05:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff314c62640 1 --2- 172.21.15.70:0/2404892568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff31010d040 0x7ff31019d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff307fff640 1 -- 172.21.15.70:0/2404892568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff310103d00 msgr2=0x7ff3101990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff307fff640 1 --2- 172.21.15.70:0/2404892568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff310103d00 0x7ff3101990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff307fff640 1 -- 172.21.15.70:0/2404892568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff31010d040 msgr2=0x7ff31019d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.510+0000 7ff307fff640 1 --2- 172.21.15.70:0/2404892568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff31010d040 0x7ff31019d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.511+0000 7ff307fff640 1 -- 172.21.15.70:0/2404892568 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff31019e0a0 con 0x7ff310104620 2024-06-04T05:04:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.511+0000 7ff307fff640 1 --2- 172.21.15.70:0/2404892568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff310104620 0x7ff3101995f0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7ff2f800d770 tx=0x7ff2f800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.511+0000 7ff30dffb640 1 -- 172.21.15.70:0/2404892568 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8017070 con 0x7ff310104620 2024-06-04T05:04:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.511+0000 7ff3166c5640 1 -- 172.21.15.70:0/2404892568 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff31019e390 con 0x7ff310104620 2024-06-04T05:04:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.511+0000 7ff30dffb640 1 -- 172.21.15.70:0/2404892568 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff2f80054b0 con 0x7ff310104620 2024-06-04T05:04:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.511+0000 7ff3166c5640 1 -- 172.21.15.70:0/2404892568 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3101a5c20 con 0x7ff310104620 2024-06-04T05:04:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.511+0000 7ff30dffb640 1 -- 172.21.15.70:0/2404892568 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8004c60 con 0x7ff310104620 2024-06-04T05:04:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.514+0000 7ff3193d6740 1 Processor -- start 2024-06-04T05:04:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.514+0000 7ff30dffb640 1 -- 172.21.15.70:0/2404892568 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff2f8004e00 con 0x7ff310104620 2024-06-04T05:04:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.514+0000 7ff3193d6740 1 -- start start 2024-06-04T05:04:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.514+0000 7ff3193d6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee5820 0x557141ee5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.514+0000 7ff3193d6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ee6140 0x557141ef04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.514+0000 7ff3193d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ef0a00 0x557141ef2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.514+0000 7ff3193d6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557141e57c30 con 0x557141ee5820 2024-06-04T05:04:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.514+0000 7ff3193d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557141e477d0 con 0x557141ef0a00 2024-06-04T05:04:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.514+0000 7ff3193d6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557141e4b810 con 0x557141ee6140 2024-06-04T05:04:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.514+0000 7ff314c62640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee5820 0x557141ee5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.514+0000 7ff307fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ef0a00 0x557141ef2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.514+0000 7ff314c62640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee5820 0x557141ee5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33804/0 (socket says 172.21.15.70:33804) 2024-06-04T05:04:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.514+0000 7ff30ffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ee6140 0x557141ef04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.514+0000 7ff314c62640 1 -- 172.21.15.70:0/911996647 learned_addr learned my addr 172.21.15.70:0/911996647 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.515+0000 7ff314c62640 1 -- 172.21.15.70:0/911996647 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ee6140 msgr2=0x557141ef04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.515+0000 7ff314c62640 1 --2- 172.21.15.70:0/911996647 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ee6140 0x557141ef04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.515+0000 7ff314c62640 1 -- 172.21.15.70:0/911996647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ef0a00 msgr2=0x557141ef2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.515+0000 7ff314c62640 1 --2- 172.21.15.70:0/911996647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ef0a00 0x557141ef2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.515+0000 7ff314c62640 1 -- 172.21.15.70:0/911996647 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557141b5c210 con 0x557141ee5820 2024-06-04T05:04:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.515+0000 7ff30dffb640 1 --2- 172.21.15.70:0/2404892568 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff2fc0b2a10 0x7ff2fc0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.515+0000 7ff314c62640 1 --2- 172.21.15.70:0/911996647 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee5820 0x557141ee5c00 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff308009be0 tx=0x7ff308005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc04cc0e4dd135be server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.515+0000 7ff30ffff640 1 --2- 172.21.15.70:0/2404892568 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff2fc0b2a10 0x7ff2fc0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.515+0000 7ff3067fc640 1 -- 172.21.15.70:0/911996647 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff308041030 con 0x557141ee5820 2024-06-04T05:04:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.515+0000 7ff3067fc640 1 -- 172.21.15.70:0/911996647 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff308037040 con 0x557141ee5820 2024-06-04T05:04:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.515+0000 7ff30dffb640 1 -- 172.21.15.70:0/2404892568 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7ff2f80e1f80 con 0x7ff310104620 2024-06-04T05:04:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.515+0000 7ff3067fc640 1 -- 172.21.15.70:0/911996647 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff308036040 con 0x557141ee5820 2024-06-04T05:04:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.516+0000 7ff3193d6740 1 -- 172.21.15.70:0/911996647 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee5820 msgr2=0x557141ee5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.516+0000 7ff30ffff640 1 --2- 172.21.15.70:0/2404892568 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff2fc0b2a10 0x7ff2fc0b4ed0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x557141ee6dc0 tx=0x7ff300009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.516+0000 7ff3193d6740 1 --2- 172.21.15.70:0/911996647 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee5820 0x557141ee5c00 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff308009be0 tx=0x7ff308005eb0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.516+0000 7ff3193d6740 1 -- 172.21.15.70:0/911996647 shutdown_connections 2024-06-04T05:04:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.516+0000 7ff3193d6740 1 --2- 172.21.15.70:0/911996647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ef0a00 0x557141ef2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.516+0000 7ff3193d6740 1 --2- 172.21.15.70:0/911996647 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ee6140 0x557141ef04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.516+0000 7ff3193d6740 1 --2- 172.21.15.70:0/911996647 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee5820 0x557141ee5c00 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.516+0000 7ff3193d6740 1 -- 172.21.15.70:0/911996647 >> 172.21.15.70:0/911996647 conn(0x557141ee1d30 msgr2=0x557141ee2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.516+0000 7ff3193d6740 1 -- 172.21.15.70:0/911996647 shutdown_connections 2024-06-04T05:04:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.516+0000 7ff3193d6740 1 -- 172.21.15.70:0/911996647 wait complete. 2024-06-04T05:04:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff3193d6740 1 -- start start 2024-06-04T05:04:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff3193d6740 1 Processor -- start 2024-06-04T05:04:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff3193d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee5820 0x557141eed960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff3193d6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee6140 0x557141eedea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff3193d6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ef0a00 0x557141eee3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff3193d6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557141b2d790 con 0x557141ee6140 2024-06-04T05:04:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff3193d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557141e03890 con 0x557141ee5820 2024-06-04T05:04:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff3193d6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557141ef3620 con 0x557141ef0a00 2024-06-04T05:04:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff3193d6740 10 client.0 _ll_register_callbacks cb 0x557141b5ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff307fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee6140 0x557141eedea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff307fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee6140 0x557141eedea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33816/0 (socket says 172.21.15.70:33816) 2024-06-04T05:04:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff307fff640 1 -- 172.21.15.70:0/3739137887 learned_addr learned my addr 172.21.15.70:0/3739137887 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff314c62640 1 --2- 172.21.15.70:0/3739137887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ef0a00 0x557141eee3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff30ffff640 1 --2- 172.21.15.70:0/3739137887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee5820 0x557141eed960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff307fff640 1 -- 172.21.15.70:0/3739137887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ef0a00 msgr2=0x557141eee3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff307fff640 1 --2- 172.21.15.70:0/3739137887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ef0a00 0x557141eee3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff307fff640 1 -- 172.21.15.70:0/3739137887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee5820 msgr2=0x557141eed960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff307fff640 1 --2- 172.21.15.70:0/3739137887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee5820 0x557141eed960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.517+0000 7ff307fff640 1 -- 172.21.15.70:0/3739137887 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557141c28620 con 0x557141ee6140 2024-06-04T05:04:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.518+0000 7ff307fff640 1 --2- 172.21.15.70:0/3739137887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee6140 0x557141eedea0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7ff2f800d740 tx=0x7ff2f80e56c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ecd4aa054890b53 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.518+0000 7ff2ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:04:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.518+0000 7ff2ef7fe640 1 -- 172.21.15.70:0/3739137887 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8020070 con 0x557141ee6140 2024-06-04T05:04:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.518+0000 7ff2ef7fe640 1 -- 172.21.15.70:0/3739137887 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7ff2f80a5570 con 0x557141ee6140 2024-06-04T05:04:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.518+0000 7ff3193d6740 1 -- 172.21.15.70:0/3739137887 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff308037040 con 0x557141ee6140 2024-06-04T05:04:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.518+0000 7ff2ef7fe640 1 -- 172.21.15.70:0/3739137887 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff2f80a90b0 con 0x557141ee6140 2024-06-04T05:04:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.518+0000 7ff3193d6740 10 client.17514 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.518+0000 7ff3193d6740 1 -- 172.21.15.70:0/3739137887 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557141eff370 con 0x557141ee6140 2024-06-04T05:04:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.518+0000 7ff2ef7fe640 1 -- 172.21.15.70:0/3739137887 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff308037040 con 0x557141ee6140 2024-06-04T05:04:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.518+0000 7ff2ef7fe640 1 -- 172.21.15.70:0/3739137887 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ff2f80e40d0 con 0x557141ee6140 2024-06-04T05:04:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.518+0000 7ff3193d6740 10 client.17514 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.518+0000 7ff3193d6740 10 client.17514 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:04:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.518+0000 7ff3193d6740 20 client.17514 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.518+0000 7ff3193d6740 1 --2- 172.21.15.70:0/3739137887 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557141f007e0 0x557141f02bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.518+0000 7ff3193d6740 4 client.17514 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:04:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.518+0000 7ff3193d6740 1 -- 172.21.15.70:0/3739137887 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557141e59b80 con 0x557141f007e0 2024-06-04T05:04:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.519+0000 7ff30ffff640 1 --2- 172.21.15.70:0/3739137887 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557141f007e0 0x557141f02bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.519+0000 7ff2ef7fe640 10 client.17514 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.519+0000 7ff30ffff640 1 --2- 172.21.15.70:0/3739137887 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557141f007e0 0x557141f02bd0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b3cab0d4b126c87 server_cookie=e65e98c9ff8b46d4 in_seq=0 out_seq=0 2024-06-04T05:04:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.520+0000 7ff2ef7fe640 1 -- 172.21.15.70:0/3739137887 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557141e59b80 con 0x557141f007e0 2024-06-04T05:04:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.520+0000 7ff2ef7fe640 10 client.17514 handle_command_reply: tid=0 2024-06-04T05:04:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.520+0000 7ff3193d6740 1 client.17514 shutdown 2024-06-04T05:04:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.520+0000 7ff3193d6740 1 -- 172.21.15.70:0/3739137887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee6140 msgr2=0x557141eedea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.520+0000 7ff3193d6740 1 --2- 172.21.15.70:0/3739137887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee6140 0x557141eedea0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7ff2f800d740 tx=0x7ff2f80e56c0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.520+0000 7ff3193d6740 1 -- 172.21.15.70:0/3739137887 shutdown_connections 2024-06-04T05:04:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.520+0000 7ff3193d6740 1 --2- 172.21.15.70:0/3739137887 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557141f007e0 0x557141f02bd0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.520+0000 7ff3193d6740 1 --2- 172.21.15.70:0/3739137887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ef0a00 0x557141eee3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.520+0000 7ff2ef7fe640 0 client.17514 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.520+0000 7ff3193d6740 1 --2- 172.21.15.70:0/3739137887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee6140 0x557141eedea0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.520+0000 7ff3193d6740 1 --2- 172.21.15.70:0/3739137887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee5820 0x557141eed960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.520+0000 7ff3193d6740 1 -- 172.21.15.70:0/3739137887 >> 172.21.15.70:0/3739137887 conn(0x557141ee1d30 msgr2=0x557141ee0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.521+0000 7ff3193d6740 1 -- 172.21.15.70:0/3739137887 shutdown_connections 2024-06-04T05:04:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.521+0000 7ff3193d6740 1 -- 172.21.15.70:0/3739137887 wait complete. 2024-06-04T05:04:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.521+0000 7ff3193d6740 20 client.17514 trim_cache size 0 max 16384 2024-06-04T05:04:50.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff3193d6740 1 Processor -- start 2024-06-04T05:04:50.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff3193d6740 1 -- start start 2024-06-04T05:04:50.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff3193d6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee5820 0x557141ee78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff3193d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee6140 0x557141ee7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff3193d6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ef0a00 0x557141f084a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff3193d6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff3101ae5e0 con 0x557141ee5820 2024-06-04T05:04:50.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff3193d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff3101b0120 con 0x557141ee6140 2024-06-04T05:04:50.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff3193d6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff3101aecb0 con 0x557141ef0a00 2024-06-04T05:04:50.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff30ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee5820 0x557141ee78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff307fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee6140 0x557141ee7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff30ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee5820 0x557141ee78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33818/0 (socket says 172.21.15.70:33818) 2024-06-04T05:04:50.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff314c62640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ef0a00 0x557141f084a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff30ffff640 1 -- 172.21.15.70:0/3374716391 learned_addr learned my addr 172.21.15.70:0/3374716391 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:50.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff307fff640 1 -- 172.21.15.70:0/3374716391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ef0a00 msgr2=0x557141f084a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:50.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff307fff640 1 --2- 172.21.15.70:0/3374716391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ef0a00 0x557141f084a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff307fff640 1 -- 172.21.15.70:0/3374716391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee5820 msgr2=0x557141ee78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:50.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff307fff640 1 --2- 172.21.15.70:0/3374716391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee5820 0x557141ee78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff307fff640 1 -- 172.21.15.70:0/3374716391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557141ee5f70 con 0x557141ee6140 2024-06-04T05:04:50.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.540+0000 7ff30ffff640 1 --2- 172.21.15.70:0/3374716391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee5820 0x557141ee78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:50.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.541+0000 7ff307fff640 1 --2- 172.21.15.70:0/3374716391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee6140 0x557141ee7e20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff2f80e5730 tx=0x7ff2f80df750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b900d09264fa0cce server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:50.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.541+0000 7ff305ffb640 1 -- 172.21.15.70:0/3374716391 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8020070 con 0x557141ee6140 2024-06-04T05:04:50.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.541+0000 7ff305ffb640 1 -- 172.21.15.70:0/3374716391 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff2f80ad5e0 con 0x557141ee6140 2024-06-04T05:04:50.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.541+0000 7ff305ffb640 1 -- 172.21.15.70:0/3374716391 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f80a8a50 con 0x557141ee6140 2024-06-04T05:04:50.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.541+0000 7ff3193d6740 1 -- 172.21.15.70:0/3374716391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee6140 msgr2=0x557141ee7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.541+0000 7ff3193d6740 1 --2- 172.21.15.70:0/3374716391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee6140 0x557141ee7e20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff2f80e5730 tx=0x7ff2f80df750 comp rx=0 tx=0).stop 2024-06-04T05:04:50.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.541+0000 7ff3193d6740 1 -- 172.21.15.70:0/3374716391 shutdown_connections 2024-06-04T05:04:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.541+0000 7ff3193d6740 1 --2- 172.21.15.70:0/3374716391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee6140 0x557141ee7e20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.541+0000 7ff3193d6740 1 --2- 172.21.15.70:0/3374716391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ef0a00 0x557141f084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.541+0000 7ff3193d6740 1 --2- 172.21.15.70:0/3374716391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee5820 0x557141ee78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.541+0000 7ff3193d6740 1 -- 172.21.15.70:0/3374716391 >> 172.21.15.70:0/3374716391 conn(0x557141ee1d30 msgr2=0x557141ee71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.542+0000 7ff3193d6740 1 -- 172.21.15.70:0/3374716391 shutdown_connections 2024-06-04T05:04:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.542+0000 7ff3193d6740 1 -- 172.21.15.70:0/3374716391 wait complete. 2024-06-04T05:04:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.542+0000 7ff3193d6740 1 -- start start 2024-06-04T05:04:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.542+0000 7ff3193d6740 1 Processor -- start 2024-06-04T05:04:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.542+0000 7ff3193d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee5820 0x557141ef47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.542+0000 7ff3193d6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee6140 0x557141ef4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.542+0000 7ff3193d6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ef0a00 0x557141ef5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.542+0000 7ff3193d6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff3101afa50 con 0x557141ee6140 2024-06-04T05:04:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.542+0000 7ff3193d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff3101af380 con 0x557141ee5820 2024-06-04T05:04:50.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.542+0000 7ff3193d6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff3101ac5e0 con 0x557141ef0a00 2024-06-04T05:04:50.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.542+0000 7ff3193d6740 10 client.0 _ll_register_callbacks cb 0x7ff2f80e57c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:50.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.542+0000 7ff307fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee6140 0x557141ef4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.542+0000 7ff30ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee5820 0x557141ef47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.542+0000 7ff307fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee6140 0x557141ef4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33834/0 (socket says 172.21.15.70:33834) 2024-06-04T05:04:50.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.542+0000 7ff30ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee5820 0x557141ef47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46854/0 (socket says 172.21.15.70:46854) 2024-06-04T05:04:50.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.542+0000 7ff307fff640 1 -- 172.21.15.70:0/3541581666 learned_addr learned my addr 172.21.15.70:0/3541581666 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:50.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.542+0000 7ff314c62640 1 --2- 172.21.15.70:0/3541581666 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ef0a00 0x557141ef5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.543+0000 7ff30ffff640 1 -- 172.21.15.70:0/3541581666 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ef0a00 msgr2=0x557141ef5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:50.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.543+0000 7ff30ffff640 1 --2- 172.21.15.70:0/3541581666 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ef0a00 0x557141ef5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.543+0000 7ff30ffff640 1 -- 172.21.15.70:0/3541581666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee6140 msgr2=0x557141ef4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:50.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.543+0000 7ff30ffff640 1 --2- 172.21.15.70:0/3541581666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee6140 0x557141ef4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.543+0000 7ff30ffff640 1 -- 172.21.15.70:0/3541581666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557141f6a5c0 con 0x557141ee5820 2024-06-04T05:04:50.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.543+0000 7ff307fff640 1 --2- 172.21.15.70:0/3541581666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee6140 0x557141ef4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:50.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.543+0000 7ff30ffff640 1 --2- 172.21.15.70:0/3541581666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee5820 0x557141ef47b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x557141f03440 tx=0x7ff300004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0bafa7ba43d52da server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:50.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.543+0000 7ff2ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:04:50.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.543+0000 7ff2ef7fe640 1 -- 172.21.15.70:0/3541581666 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30001dce0 con 0x557141ee5820 2024-06-04T05:04:50.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.543+0000 7ff2ef7fe640 1 -- 172.21.15.70:0/3541581666 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7ff300018030 con 0x557141ee5820 2024-06-04T05:04:50.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.543+0000 7ff3193d6740 1 -- 172.21.15.70:0/3541581666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2f80ad5e0 con 0x557141ee5820 2024-06-04T05:04:50.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.543+0000 7ff2ef7fe640 1 -- 172.21.15.70:0/3541581666 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff300004020 con 0x557141ee5820 2024-06-04T05:04:50.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.543+0000 7ff3193d6740 10 client.26732 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:50.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.543+0000 7ff3193d6740 1 -- 172.21.15.70:0/3541581666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557141f6b8d0 con 0x557141ee5820 2024-06-04T05:04:50.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.544+0000 7ff2ef7fe640 1 -- 172.21.15.70:0/3541581666 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff2f80ad5e0 con 0x557141ee5820 2024-06-04T05:04:50.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.544+0000 7ff2ef7fe640 1 -- 172.21.15.70:0/3541581666 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ff300035420 con 0x557141ee5820 2024-06-04T05:04:50.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.544+0000 7ff3193d6740 10 client.26732 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:50.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.544+0000 7ff3193d6740 10 client.26732 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:04:50.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.544+0000 7ff3193d6740 20 client.26732 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:50.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.544+0000 7ff3193d6740 1 --2- 172.21.15.70:0/3541581666 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557141f6c020 0x557141f6e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.544+0000 7ff3193d6740 4 client.26732 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:04:50.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.544+0000 7ff3193d6740 1 -- 172.21.15.70:0/3541581666 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557141e53360 con 0x557141f6c020 2024-06-04T05:04:50.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.544+0000 7ff307fff640 1 --2- 172.21.15.70:0/3541581666 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557141f6c020 0x557141f6e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.545+0000 7ff307fff640 1 --2- 172.21.15.70:0/3541581666 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557141f6c020 0x557141f6e440 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f95c55d7d4a22a5b server_cookie=8f40f6d02716269c in_seq=0 out_seq=0 2024-06-04T05:04:50.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.545+0000 7ff2ef7fe640 10 client.26732 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:50.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.568+0000 7ff2ef7fe640 1 -- 172.21.15.70:0/3541581666 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x557141e53360 con 0x557141f6c020 2024-06-04T05:04:50.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.568+0000 7ff2ef7fe640 10 client.26732 handle_command_reply: tid=0 2024-06-04T05:04:50.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.568+0000 7ff3193d6740 1 client.26732 shutdown 2024-06-04T05:04:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.568+0000 7ff3193d6740 1 -- 172.21.15.70:0/3541581666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee5820 msgr2=0x557141ef47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.568+0000 7ff3193d6740 1 --2- 172.21.15.70:0/3541581666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee5820 0x557141ef47b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x557141f03440 tx=0x7ff300004730 comp rx=0 tx=0).stop 2024-06-04T05:04:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.568+0000 7ff3193d6740 1 -- 172.21.15.70:0/3541581666 shutdown_connections 2024-06-04T05:04:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.568+0000 7ff3193d6740 1 --2- 172.21.15.70:0/3541581666 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557141f6c020 0x557141f6e440 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.568+0000 7ff3193d6740 1 --2- 172.21.15.70:0/3541581666 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557141ef0a00 0x557141ef5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.568+0000 7ff2ef7fe640 0 client.26732 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.568+0000 7ff3193d6740 1 --2- 172.21.15.70:0/3541581666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557141ee6140 0x557141ef4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.568+0000 7ff3193d6740 1 --2- 172.21.15.70:0/3541581666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557141ee5820 0x557141ef47b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.568+0000 7ff3193d6740 1 -- 172.21.15.70:0/3541581666 >> 172.21.15.70:0/3541581666 conn(0x557141ee1d30 msgr2=0x557141ee71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.568+0000 7ff3193d6740 1 -- 172.21.15.70:0/3541581666 shutdown_connections 2024-06-04T05:04:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.568+0000 7ff3193d6740 1 -- 172.21.15.70:0/3541581666 wait complete. 2024-06-04T05:04:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.568+0000 7ff3193d6740 20 client.26732 trim_cache size 0 max 16384 2024-06-04T05:04:50.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:04:50.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:04:50.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:04:50.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.569+0000 7ff3166c5640 1 -- 172.21.15.70:0/2404892568 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff2fc0b2a10 msgr2=0x7ff2fc0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.569+0000 7ff3166c5640 1 --2- 172.21.15.70:0/2404892568 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff2fc0b2a10 0x7ff2fc0b4ed0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x557141ee6dc0 tx=0x7ff300009260 comp rx=0 tx=0).stop 2024-06-04T05:04:50.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.569+0000 7ff3166c5640 1 -- 172.21.15.70:0/2404892568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff310104620 msgr2=0x7ff3101995f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.569+0000 7ff3166c5640 1 --2- 172.21.15.70:0/2404892568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff310104620 0x7ff3101995f0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7ff2f800d770 tx=0x7ff2f800dc30 comp rx=0 tx=0).stop 2024-06-04T05:04:50.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.569+0000 7ff3166c5640 1 -- 172.21.15.70:0/2404892568 shutdown_connections 2024-06-04T05:04:50.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.569+0000 7ff3166c5640 1 --2- 172.21.15.70:0/2404892568 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff2fc0b2a10 0x7ff2fc0b4ed0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.569+0000 7ff3166c5640 1 --2- 172.21.15.70:0/2404892568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff31010d040 0x7ff31019d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.569+0000 7ff3166c5640 1 --2- 172.21.15.70:0/2404892568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff310104620 0x7ff3101995f0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.569+0000 7ff3166c5640 1 --2- 172.21.15.70:0/2404892568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff310103d00 0x7ff3101990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.569+0000 7ff3166c5640 1 -- 172.21.15.70:0/2404892568 >> 172.21.15.70:0/2404892568 conn(0x7ff310078ad0 msgr2=0x7ff3100fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:50.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.569+0000 7ff3166c5640 1 -- 172.21.15.70:0/2404892568 shutdown_connections 2024-06-04T05:04:50.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:04:50.569+0000 7ff3166c5640 1 -- 172.21.15.70:0/2404892568 wait complete. 2024-06-04T05:04:50.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.587+0000 7f37f5f9f700 1 -- 172.21.15.70:0/2445161527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37f0102e00 msgr2=0x7f37f010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.587+0000 7f37f5f9f700 1 --2- 172.21.15.70:0/2445161527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37f0102e00 0x7f37f010f2f0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f37e800b0a0 tx=0x7f37e800b3b0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.587+0000 7f37f5f9f700 1 -- 172.21.15.70:0/2445161527 shutdown_connections 2024-06-04T05:04:50.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.587+0000 7f37f5f9f700 1 --2- 172.21.15.70:0/2445161527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37f0102e00 0x7f37f010f2f0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.587+0000 7f37f5f9f700 1 --2- 172.21.15.70:0/2445161527 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f0108520 0x7f37f0108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.587+0000 7f37f5f9f700 1 --2- 172.21.15.70:0/2445161527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f0102440 0x7f37f01028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.587+0000 7f37f5f9f700 1 -- 172.21.15.70:0/2445161527 >> 172.21.15.70:0/2445161527 conn(0x7f37f00793f0 msgr2=0x7f37f0079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:50.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.587+0000 7f37f5f9f700 1 -- 172.21.15.70:0/2445161527 shutdown_connections 2024-06-04T05:04:50.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.587+0000 7f37f5f9f700 1 -- 172.21.15.70:0/2445161527 wait complete. 2024-06-04T05:04:50.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.588+0000 7f37f5f9f700 1 Processor -- start 2024-06-04T05:04:50.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37f5f9f700 1 -- start start 2024-06-04T05:04:50.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37f5f9f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37f0102440 0x7f37f019f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37f5f9f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f0102e00 0x7f37f019fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37f5f9f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f0108520 0x7f37f0199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37f5f9f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f37f01124a0 con 0x7f37f0102440 2024-06-04T05:04:50.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37f5f9f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37f0112320 con 0x7f37f0102e00 2024-06-04T05:04:50.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37f5f9f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f37f0112620 con 0x7f37f0108520 2024-06-04T05:04:50.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37ef7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37f0102440 0x7f37f019f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37ef7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37f0102440 0x7f37f019f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33864/0 (socket says 172.21.15.70:33864) 2024-06-04T05:04:50.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37ef7fe700 1 -- 172.21.15.70:0/3971255460 learned_addr learned my addr 172.21.15.70:0/3971255460 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:50.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37eeffd700 1 --2- 172.21.15.70:0/3971255460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f0102e00 0x7f37f019fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37effff700 1 --2- 172.21.15.70:0/3971255460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f0108520 0x7f37f0199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37eeffd700 1 -- 172.21.15.70:0/3971255460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f0108520 msgr2=0x7f37f0199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37eeffd700 1 --2- 172.21.15.70:0/3971255460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f0108520 0x7f37f0199900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37eeffd700 1 -- 172.21.15.70:0/3971255460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37f0102440 msgr2=0x7f37f019f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37eeffd700 1 --2- 172.21.15.70:0/3971255460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37f0102440 0x7f37f019f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37eeffd700 1 -- 172.21.15.70:0/3971255460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f019a060 con 0x7f37f0102e00 2024-06-04T05:04:50.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37ef7fe700 1 --2- 172.21.15.70:0/3971255460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37f0102440 0x7f37f019f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:50.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.589+0000 7f37effff700 1 --2- 172.21.15.70:0/3971255460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f0108520 0x7f37f0199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:50.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.590+0000 7f37eeffd700 1 --2- 172.21.15.70:0/3971255460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f0102e00 0x7f37f019fc80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f37e400e9f0 tx=0x7f37e400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:50.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.590+0000 7f37ecff9700 1 -- 172.21.15.70:0/3971255460 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e400c9c0 con 0x7f37f0102e00 2024-06-04T05:04:50.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.590+0000 7f37ecff9700 1 -- 172.21.15.70:0/3971255460 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f37e4010430 con 0x7f37f0102e00 2024-06-04T05:04:50.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.590+0000 7f37ecff9700 1 -- 172.21.15.70:0/3971255460 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4018680 con 0x7f37f0102e00 2024-06-04T05:04:50.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.590+0000 7f37f5f9f700 1 -- 172.21.15.70:0/3971255460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37f019a2f0 con 0x7f37f0102e00 2024-06-04T05:04:50.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.590+0000 7f37f5f9f700 1 -- 172.21.15.70:0/3971255460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37f01a6610 con 0x7f37f0102e00 2024-06-04T05:04:50.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.593+0000 7f37ecff9700 1 -- 172.21.15.70:0/3971255460 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f37e400cb60 con 0x7f37f0102e00 2024-06-04T05:04:50.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.594+0000 7f37ecff9700 1 --2- 172.21.15.70:0/3971255460 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f37d80b2a70 0x7f37d80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.594+0000 7f37ef7fe700 1 --2- 172.21.15.70:0/3971255460 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f37d80b2a70 0x7f37d80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.594+0000 7f37ecff9700 1 -- 172.21.15.70:0/3971255460 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f37e40e26e0 con 0x7f37f0102e00 2024-06-04T05:04:50.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.594+0000 7f3800834b80 1 Processor -- start 2024-06-04T05:04:50.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.594+0000 7f3800834b80 1 -- start start 2024-06-04T05:04:50.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.594+0000 7f37ef7fe700 1 --2- 172.21.15.70:0/3971255460 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f37d80b2a70 0x7f37d80b4f30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f37f019aa40 tx=0x7f37e0009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:50.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f3800834b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123ea3a0 0x55a4123ea780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f3800834b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 0x55a4123e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f3800834b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123e52e0 0x55a4123f1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f3800834b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a412342df0 con 0x55a4123e52e0 2024-06-04T05:04:50.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f3800834b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a412332990 con 0x55a4123ea3a0 2024-06-04T05:04:50.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f3800834b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a4123369d0 con 0x55a4123e4920 2024-06-04T05:04:50.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f37ef7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123ea3a0 0x55a4123ea780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f37effff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123e52e0 0x55a4123f1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f37eeffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 0x55a4123e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f37ef7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123ea3a0 0x55a4123ea780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46874/0 (socket says 172.21.15.70:46874) 2024-06-04T05:04:50.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f37ef7fe700 1 -- 172.21.15.70:0/3055152725 learned_addr learned my addr 172.21.15.70:0/3055152725 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:50.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f37eeffd700 1 -- 172.21.15.70:0/3055152725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123ea3a0 msgr2=0x55a4123ea780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:50.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f37eeffd700 1 --2- 172.21.15.70:0/3055152725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123ea3a0 0x55a4123ea780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f37eeffd700 1 -- 172.21.15.70:0/3055152725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123e52e0 msgr2=0x55a4123f1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:50.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f37eeffd700 1 --2- 172.21.15.70:0/3055152725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123e52e0 0x55a4123f1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f37eeffd700 1 -- 172.21.15.70:0/3055152725 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4122dc030 con 0x55a4123e4920 2024-06-04T05:04:50.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f37eeffd700 1 --2- 172.21.15.70:0/3055152725 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 0x55a4123e4da0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f37e40dfe00 tx=0x7f37e40a9180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a0f5c194493c362 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:50.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f37d5ffb700 1 -- 172.21.15.70:0/3055152725 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e40aa090 con 0x55a4123e4920 2024-06-04T05:04:50.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f37d5ffb700 1 -- 172.21.15.70:0/3055152725 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f37e40a9990 con 0x55a4123e4920 2024-06-04T05:04:50.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.595+0000 7f37d5ffb700 1 -- 172.21.15.70:0/3055152725 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e40a9bf0 con 0x55a4123e4920 2024-06-04T05:04:50.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.596+0000 7f3800834b80 1 -- 172.21.15.70:0/3055152725 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 msgr2=0x55a4123e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.596+0000 7f3800834b80 1 --2- 172.21.15.70:0/3055152725 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 0x55a4123e4da0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f37e40dfe00 tx=0x7f37e40a9180 comp rx=0 tx=0).stop 2024-06-04T05:04:50.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.596+0000 7f3800834b80 1 -- 172.21.15.70:0/3055152725 shutdown_connections 2024-06-04T05:04:50.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.596+0000 7f3800834b80 1 --2- 172.21.15.70:0/3055152725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123e52e0 0x55a4123f1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.596+0000 7f3800834b80 1 --2- 172.21.15.70:0/3055152725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123ea3a0 0x55a4123ea780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.596+0000 7f3800834b80 1 --2- 172.21.15.70:0/3055152725 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 0x55a4123e4da0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.596+0000 7f3800834b80 1 -- 172.21.15.70:0/3055152725 >> 172.21.15.70:0/3055152725 conn(0x55a4123102c0 msgr2=0x55a4123df460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:50.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.596+0000 7f3800834b80 1 -- 172.21.15.70:0/3055152725 shutdown_connections 2024-06-04T05:04:50.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.596+0000 7f3800834b80 1 -- 172.21.15.70:0/3055152725 wait complete. 2024-06-04T05:04:50.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.597+0000 7f3800834b80 1 -- start start 2024-06-04T05:04:50.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.597+0000 7f3800834b80 1 Processor -- start 2024-06-04T05:04:50.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.597+0000 7f3800834b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 0x55a4123f0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.597+0000 7f3800834b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123e52e0 0x55a4123ee170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.597+0000 7f3800834b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123ea3a0 0x55a4123ee6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.597+0000 7f3800834b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a412239020 con 0x55a4123ea3a0 2024-06-04T05:04:50.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.597+0000 7f3800834b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a41229ae20 con 0x55a4123e52e0 2024-06-04T05:04:50.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.597+0000 7f3800834b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a4123f1760 con 0x55a4123e4920 2024-06-04T05:04:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.597+0000 7f3800834b80 10 client.0 _ll_register_callbacks cb 0x55a4122cd9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.597+0000 7f37ef7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 0x55a4123f0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.597+0000 7f37ef7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 0x55a4123f0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:34886/0 (socket says 172.21.15.70:34886) 2024-06-04T05:04:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.597+0000 7f37effff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123ea3a0 0x55a4123ee6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.597+0000 7f37ef7fe700 1 -- 172.21.15.70:0/3901978227 learned_addr learned my addr 172.21.15.70:0/3901978227 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.598+0000 7f37eeffd700 1 --2- 172.21.15.70:0/3901978227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123e52e0 0x55a4123ee170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.598+0000 7f37ef7fe700 1 -- 172.21.15.70:0/3901978227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123e52e0 msgr2=0x55a4123ee170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.598+0000 7f37ef7fe700 1 --2- 172.21.15.70:0/3901978227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123e52e0 0x55a4123ee170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.598+0000 7f37ef7fe700 1 -- 172.21.15.70:0/3901978227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123ea3a0 msgr2=0x55a4123ee6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.598+0000 7f37ef7fe700 1 --2- 172.21.15.70:0/3901978227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123ea3a0 0x55a4123ee6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.598+0000 7f37ef7fe700 1 -- 172.21.15.70:0/3901978227 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a412282950 con 0x55a4123e4920 2024-06-04T05:04:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.598+0000 7f37effff700 1 --2- 172.21.15.70:0/3901978227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123ea3a0 0x55a4123ee6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.598+0000 7f37ef7fe700 1 --2- 172.21.15.70:0/3901978227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 0x55a4123f0e70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f37e000fe10 tx=0x7f37e00047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d2514a2e648c642 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.598+0000 7f37ceffd700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:04:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.598+0000 7f37ceffd700 1 -- 172.21.15.70:0/3901978227 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e001fe60 con 0x55a4123e4920 2024-06-04T05:04:50.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.598+0000 7f37ceffd700 1 -- 172.21.15.70:0/3901978227 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f37e000fe40 con 0x55a4123e4920 2024-06-04T05:04:50.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.598+0000 7f3800834b80 1 -- 172.21.15.70:0/3901978227 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37e40a9990 con 0x55a4123e4920 2024-06-04T05:04:50.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.598+0000 7f37ceffd700 1 -- 172.21.15.70:0/3901978227 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f37e0024450 con 0x55a4123e4920 2024-06-04T05:04:50.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.598+0000 7f3800834b80 10 client.26824 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:50.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.598+0000 7f3800834b80 1 -- 172.21.15.70:0/3901978227 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a4122ec7e0 con 0x55a4123e4920 2024-06-04T05:04:50.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.598+0000 7f37ceffd700 1 -- 172.21.15.70:0/3901978227 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f37e40a9990 con 0x55a4123e4920 2024-06-04T05:04:50.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.599+0000 7f37ceffd700 1 -- 172.21.15.70:0/3901978227 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f37e002c680 con 0x55a4123e4920 2024-06-04T05:04:50.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.599+0000 7f3800834b80 10 client.26824 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:50.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.599+0000 7f3800834b80 10 client.26824 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:04:50.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.599+0000 7f3800834b80 20 client.26824 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:50.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.599+0000 7f3800834b80 1 --2- 172.21.15.70:0/3901978227 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a4123fe440 0x55a412400830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.599+0000 7f3800834b80 4 client.26824 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:04:50.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.599+0000 7f3800834b80 1 -- 172.21.15.70:0/3901978227 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a412344d40 con 0x55a4123fe440 2024-06-04T05:04:50.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.599+0000 7f37eeffd700 1 --2- 172.21.15.70:0/3901978227 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a4123fe440 0x55a412400830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.599+0000 7f37ceffd700 10 client.26824 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:50.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.599+0000 7f37eeffd700 1 --2- 172.21.15.70:0/3901978227 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a4123fe440 0x55a412400830 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49995ba0c038effc server_cookie=3598576bf2c6437e in_seq=0 out_seq=0 2024-06-04T05:04:50.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.600+0000 7f37ceffd700 1 -- 172.21.15.70:0/3901978227 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a412344d40 con 0x55a4123fe440 2024-06-04T05:04:50.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.600+0000 7f37ceffd700 10 client.26824 handle_command_reply: tid=0 2024-06-04T05:04:50.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.600+0000 7f3800834b80 1 client.26824 shutdown 2024-06-04T05:04:50.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.600+0000 7f3800834b80 1 -- 172.21.15.70:0/3901978227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 msgr2=0x55a4123f0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.600+0000 7f3800834b80 1 --2- 172.21.15.70:0/3901978227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 0x55a4123f0e70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f37e000fe10 tx=0x7f37e00047c0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.600+0000 7f3800834b80 1 -- 172.21.15.70:0/3901978227 shutdown_connections 2024-06-04T05:04:50.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.600+0000 7f3800834b80 1 --2- 172.21.15.70:0/3901978227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123ea3a0 0x55a4123ee6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.600+0000 7f3800834b80 1 --2- 172.21.15.70:0/3901978227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 0x55a4123f0e70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.600+0000 7f3800834b80 1 --2- 172.21.15.70:0/3901978227 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a4123fe440 0x55a412400830 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.600+0000 7f37ceffd700 0 client.26824 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:50.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.600+0000 7f3800834b80 1 --2- 172.21.15.70:0/3901978227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123e52e0 0x55a4123ee170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.600+0000 7f3800834b80 1 -- 172.21.15.70:0/3901978227 >> 172.21.15.70:0/3901978227 conn(0x55a4123102c0 msgr2=0x55a4123deb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:50.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.601+0000 7f3800834b80 1 -- 172.21.15.70:0/3901978227 shutdown_connections 2024-06-04T05:04:50.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.601+0000 7f3800834b80 1 -- 172.21.15.70:0/3901978227 wait complete. 2024-06-04T05:04:50.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.601+0000 7f3800834b80 20 client.26824 trim_cache size 0 max 16384 2024-06-04T05:04:50.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.625+0000 7f3800834b80 1 Processor -- start 2024-06-04T05:04:50.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.625+0000 7f3800834b80 1 -- start start 2024-06-04T05:04:50.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f3800834b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123e4920 0x55a4123fa7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f3800834b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e52e0 0x55a4123df2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f3800834b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123ea3a0 0x55a4123f00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f3800834b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f37f01abcf0 con 0x55a4123ea3a0 2024-06-04T05:04:50.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f3800834b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37f01ad830 con 0x55a4123e4920 2024-06-04T05:04:50.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f3800834b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f37f01ac3c0 con 0x55a4123e52e0 2024-06-04T05:04:50.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f37effff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123ea3a0 0x55a4123f00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f37effff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123ea3a0 0x55a4123f00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33888/0 (socket says 172.21.15.70:33888) 2024-06-04T05:04:50.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f37effff700 1 -- 172.21.15.70:0/1696810789 learned_addr learned my addr 172.21.15.70:0/1696810789 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:50.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f37eeffd700 1 --2- 172.21.15.70:0/1696810789 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e52e0 0x55a4123df2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f37ef7fe700 1 --2- 172.21.15.70:0/1696810789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123e4920 0x55a4123fa7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f37effff700 1 -- 172.21.15.70:0/1696810789 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e52e0 msgr2=0x55a4123df2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:50.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f37effff700 1 --2- 172.21.15.70:0/1696810789 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e52e0 0x55a4123df2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f37effff700 1 -- 172.21.15.70:0/1696810789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123e4920 msgr2=0x55a4123fa7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:50.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f37effff700 1 --2- 172.21.15.70:0/1696810789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123e4920 0x55a4123fa7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f37effff700 1 -- 172.21.15.70:0/1696810789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4123f15b0 con 0x55a4123ea3a0 2024-06-04T05:04:50.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f37effff700 1 --2- 172.21.15.70:0/1696810789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123ea3a0 0x55a4123f00b0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f37e8003a30 tx=0x7f37e800b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a383cd9736506790 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:50.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.626+0000 7f37d57fa700 1 -- 172.21.15.70:0/1696810789 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e802b070 con 0x55a4123ea3a0 2024-06-04T05:04:50.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.627+0000 7f37d57fa700 1 -- 172.21.15.70:0/1696810789 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f37e8003d90 con 0x55a4123ea3a0 2024-06-04T05:04:50.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.627+0000 7f37d57fa700 1 -- 172.21.15.70:0/1696810789 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e8005440 con 0x55a4123ea3a0 2024-06-04T05:04:50.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.627+0000 7f3800834b80 1 -- 172.21.15.70:0/1696810789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123ea3a0 msgr2=0x55a4123f00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.627+0000 7f3800834b80 1 --2- 172.21.15.70:0/1696810789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123ea3a0 0x55a4123f00b0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f37e8003a30 tx=0x7f37e800b720 comp rx=0 tx=0).stop 2024-06-04T05:04:50.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.627+0000 7f3800834b80 1 -- 172.21.15.70:0/1696810789 shutdown_connections 2024-06-04T05:04:50.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.627+0000 7f3800834b80 1 --2- 172.21.15.70:0/1696810789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123ea3a0 0x55a4123f00b0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.627+0000 7f3800834b80 1 --2- 172.21.15.70:0/1696810789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123e4920 0x55a4123fa7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.627+0000 7f3800834b80 1 --2- 172.21.15.70:0/1696810789 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e52e0 0x55a4123df2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.627+0000 7f3800834b80 1 -- 172.21.15.70:0/1696810789 >> 172.21.15.70:0/1696810789 conn(0x55a4123102c0 msgr2=0x55a4123ef9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:50.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.627+0000 7f3800834b80 1 -- 172.21.15.70:0/1696810789 shutdown_connections 2024-06-04T05:04:50.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.628+0000 7f3800834b80 1 -- 172.21.15.70:0/1696810789 wait complete. 2024-06-04T05:04:50.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.628+0000 7f3800834b80 1 -- start start 2024-06-04T05:04:50.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.628+0000 7f3800834b80 1 Processor -- start 2024-06-04T05:04:50.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.628+0000 7f3800834b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 0x55a4123f6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.628+0000 7f3800834b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123e52e0 0x55a4123f70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.628+0000 7f3800834b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123ea3a0 0x55a4123f1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.628+0000 7f3800834b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f37f01ad160 con 0x55a4123e52e0 2024-06-04T05:04:50.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.628+0000 7f3800834b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37f01aca90 con 0x55a4123ea3a0 2024-06-04T05:04:50.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.628+0000 7f3800834b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f37f01a9cb0 con 0x55a4123e4920 2024-06-04T05:04:50.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.628+0000 7f3800834b80 10 client.0 _ll_register_callbacks cb 0x55a4122cd9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:50.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.628+0000 7f37eeffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123e52e0 0x55a4123f70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.628+0000 7f37eeffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123e52e0 0x55a4123f70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33890/0 (socket says 172.21.15.70:33890) 2024-06-04T05:04:50.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.628+0000 7f37ef7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 0x55a4123f6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.628+0000 7f37eeffd700 1 -- 172.21.15.70:0/3038996245 learned_addr learned my addr 172.21.15.70:0/3038996245 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:50.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.628+0000 7f37effff700 1 --2- 172.21.15.70:0/3038996245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123ea3a0 0x55a4123f1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.629+0000 7f37ef7fe700 1 -- 172.21.15.70:0/3038996245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123ea3a0 msgr2=0x55a4123f1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:50.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.629+0000 7f37ef7fe700 1 --2- 172.21.15.70:0/3038996245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123ea3a0 0x55a4123f1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.629+0000 7f37ef7fe700 1 -- 172.21.15.70:0/3038996245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123e52e0 msgr2=0x55a4123f70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:50.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.629+0000 7f37ef7fe700 1 --2- 172.21.15.70:0/3038996245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123e52e0 0x55a4123f70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.629+0000 7f37ef7fe700 1 -- 172.21.15.70:0/3038996245 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4123f2140 con 0x55a4123e4920 2024-06-04T05:04:50.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.629+0000 7f37eeffd700 1 --2- 172.21.15.70:0/3038996245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123e52e0 0x55a4123f70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:50.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.629+0000 7f37effff700 1 --2- 172.21.15.70:0/3038996245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123ea3a0 0x55a4123f1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:50.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.629+0000 7f37ef7fe700 1 --2- 172.21.15.70:0/3038996245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 0x55a4123f6ba0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f37e002ccb0 tx=0x7f37e00346a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd58de2af563acf7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:50.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.629+0000 7f37ceffd700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:04:50.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.629+0000 7f37ceffd700 1 -- 172.21.15.70:0/3038996245 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e001faa0 con 0x55a4123e4920 2024-06-04T05:04:50.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.629+0000 7f37ceffd700 1 -- 172.21.15.70:0/3038996245 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f37e000fe40 con 0x55a4123e4920 2024-06-04T05:04:50.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.629+0000 7f3800834b80 1 -- 172.21.15.70:0/3038996245 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37e8003d90 con 0x55a4123e4920 2024-06-04T05:04:50.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.629+0000 7f37ceffd700 1 -- 172.21.15.70:0/3038996245 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f37e001fe60 con 0x55a4123e4920 2024-06-04T05:04:50.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.629+0000 7f3800834b80 10 client.26830 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:50.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.629+0000 7f3800834b80 1 -- 172.21.15.70:0/3038996245 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a412469af0 con 0x55a4123e4920 2024-06-04T05:04:50.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.630+0000 7f37ceffd700 1 -- 172.21.15.70:0/3038996245 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f37e8003d90 con 0x55a4123e4920 2024-06-04T05:04:50.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.630+0000 7f37ceffd700 1 -- 172.21.15.70:0/3038996245 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f37e0024a80 con 0x55a4123e4920 2024-06-04T05:04:50.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.630+0000 7f3800834b80 10 client.26830 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:50.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.630+0000 7f3800834b80 10 client.26830 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:04:50.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.630+0000 7f3800834b80 20 client.26830 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:50.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.630+0000 7f3800834b80 1 --2- 172.21.15.70:0/3038996245 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a4123fe440 0x55a41246c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:50.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.630+0000 7f3800834b80 4 client.26830 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:04:50.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.630+0000 7f3800834b80 1 -- 172.21.15.70:0/3038996245 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a41233e520 con 0x55a4123fe440 2024-06-04T05:04:50.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.630+0000 7f37eeffd700 1 --2- 172.21.15.70:0/3038996245 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a4123fe440 0x55a41246c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:50.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.631+0000 7f37ceffd700 10 client.26830 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:50.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.631+0000 7f37eeffd700 1 --2- 172.21.15.70:0/3038996245 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a4123fe440 0x55a41246c180 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef342ee259173c79 server_cookie=9049811ed2548f82 in_seq=0 out_seq=0 2024-06-04T05:04:50.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.633+0000 7f37ceffd700 1 -- 172.21.15.70:0/3038996245 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a41233e520 con 0x55a4123fe440 2024-06-04T05:04:50.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.633+0000 7f37ceffd700 10 client.26830 handle_command_reply: tid=0 2024-06-04T05:04:50.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.633+0000 7f3800834b80 1 client.26830 shutdown 2024-06-04T05:04:50.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.633+0000 7f3800834b80 1 -- 172.21.15.70:0/3038996245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 msgr2=0x55a4123f6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.633+0000 7f3800834b80 1 --2- 172.21.15.70:0/3038996245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 0x55a4123f6ba0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f37e002ccb0 tx=0x7f37e00346a0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.633+0000 7f3800834b80 1 -- 172.21.15.70:0/3038996245 shutdown_connections 2024-06-04T05:04:50.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.633+0000 7f3800834b80 1 --2- 172.21.15.70:0/3038996245 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a4123fe440 0x55a41246c180 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.633+0000 7f3800834b80 1 --2- 172.21.15.70:0/3038996245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4123ea3a0 0x55a4123f1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.633+0000 7f3800834b80 1 --2- 172.21.15.70:0/3038996245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a4123e4920 0x55a4123f6ba0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.633+0000 7f37ceffd700 0 client.26830 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:50.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.633+0000 7f3800834b80 1 --2- 172.21.15.70:0/3038996245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4123e52e0 0x55a4123f70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.633+0000 7f3800834b80 1 -- 172.21.15.70:0/3038996245 >> 172.21.15.70:0/3038996245 conn(0x55a4123102c0 msgr2=0x55a4123f0230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:50.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.633+0000 7f3800834b80 1 -- 172.21.15.70:0/3038996245 shutdown_connections 2024-06-04T05:04:50.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.633+0000 7f3800834b80 1 -- 172.21.15.70:0/3038996245 wait complete. 2024-06-04T05:04:50.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.633+0000 7f3800834b80 20 client.26830 trim_cache size 0 max 16384 2024-06-04T05:04:50.638 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:04:50.638 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:04:50.639 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "83046a07-2799-458e-abac-dafb988bc251", 2024-06-04T05:04:50.639 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:04:50.639 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:04:50.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.634+0000 7f37f5f9f700 1 -- 172.21.15.70:0/3971255460 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f37d80b2a70 msgr2=0x7f37d80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.634+0000 7f37f5f9f700 1 --2- 172.21.15.70:0/3971255460 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f37d80b2a70 0x7f37d80b4f30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f37f019aa40 tx=0x7f37e0009260 comp rx=0 tx=0).stop 2024-06-04T05:04:50.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.634+0000 7f37f5f9f700 1 -- 172.21.15.70:0/3971255460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f0102e00 msgr2=0x7f37f019fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:50.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.634+0000 7f37f5f9f700 1 --2- 172.21.15.70:0/3971255460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f0102e00 0x7f37f019fc80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f37e400e9f0 tx=0x7f37e400ed00 comp rx=0 tx=0).stop 2024-06-04T05:04:50.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.634+0000 7f37f5f9f700 1 -- 172.21.15.70:0/3971255460 shutdown_connections 2024-06-04T05:04:50.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.634+0000 7f37f5f9f700 1 --2- 172.21.15.70:0/3971255460 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f37d80b2a70 0x7f37d80b4f30 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.634+0000 7f37f5f9f700 1 --2- 172.21.15.70:0/3971255460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f0108520 0x7f37f0199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.634+0000 7f37f5f9f700 1 --2- 172.21.15.70:0/3971255460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37f0102440 0x7f37f019f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.634+0000 7f37f5f9f700 1 --2- 172.21.15.70:0/3971255460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f0102e00 0x7f37f019fc80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:50.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.634+0000 7f37f5f9f700 1 -- 172.21.15.70:0/3971255460 >> 172.21.15.70:0/3971255460 conn(0x7f37f00793f0 msgr2=0x7f37f006b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:50.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.634+0000 7f37f5f9f700 1 -- 172.21.15.70:0/3971255460 shutdown_connections 2024-06-04T05:04:50.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:50.634+0000 7f37f5f9f700 1 -- 172.21.15.70:0/3971255460 wait complete. 2024-06-04T05:04:50.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 129503 2.03 MB/sec execute 223 sec latency 26.180 ms 2024-06-04T05:04:51.408 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:04:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:51 smithi070 ceph-mon[32631]: pgmap v235: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-06-04T05:04:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:51 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:04:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:51 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:51 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:04:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:51 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:04:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:51 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:51 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:04:51.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:51 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:04:51.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:51 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:51 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:51 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:04:51.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:51 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:04:51.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:51 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:51 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:04:51.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:51 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:51 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:51 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:51 smithi112 ceph-mon[36453]: pgmap v235: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-06-04T05:04:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:51 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:04:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:51 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:51 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:04:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:51 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:04:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:51 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:51 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:04:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:51 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:04:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:51 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:51 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:51 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:04:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:51 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:04:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:51 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:51 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:04:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:51 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:51 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:51 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.714 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:04:51.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 130100 2.03 MB/sec execute 224 sec latency 19.412 ms 2024-06-04T05:04:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:51 smithi098 ceph-mon[35569]: pgmap v235: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-06-04T05:04:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:51 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:04:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:51 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:51 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:04:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:51 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:04:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:51 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:51 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:04:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:51 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:04:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:51 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:51 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:51 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:04:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:51 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:04:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:51 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:51 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:04:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:51 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:51 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:51 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:04:52.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 130722 2.04 MB/sec execute 225 sec latency 15.166 ms 2024-06-04T05:04:53.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.438+0000 7fbb02c98700 1 -- 172.21.15.70:0/2199332535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbafc103fa0 msgr2=0x7fbafc110450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:53.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.438+0000 7fbb02c98700 1 --2- 172.21.15.70:0/2199332535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbafc103fa0 0x7fbafc110450 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400b0d0 tx=0x7fbaf400b3e0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.438+0000 7fbb02c98700 1 -- 172.21.15.70:0/2199332535 shutdown_connections 2024-06-04T05:04:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.438+0000 7fbb02c98700 1 --2- 172.21.15.70:0/2199332535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbafc103fa0 0x7fbafc110450 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.438+0000 7fbb02c98700 1 --2- 172.21.15.70:0/2199332535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbafc109680 0x7fbafc109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.438+0000 7fbb02c98700 1 --2- 172.21.15.70:0/2199332535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbafc1035e0 0x7fbafc103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.438+0000 7fbb02c98700 1 -- 172.21.15.70:0/2199332535 >> 172.21.15.70:0/2199332535 conn(0x7fbafc07a6f0 msgr2=0x7fbafc07ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.438+0000 7fbb02c98700 1 -- 172.21.15.70:0/2199332535 shutdown_connections 2024-06-04T05:04:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.438+0000 7fbb02c98700 1 -- 172.21.15.70:0/2199332535 wait complete. 2024-06-04T05:04:53.442 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:53 smithi070 ceph-mon[32631]: pgmap v236: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 1.9 MiB/s wr, 202 op/s 2024-06-04T05:04:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.438+0000 7fbb02c98700 1 Processor -- start 2024-06-04T05:04:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbb02c98700 1 -- start start 2024-06-04T05:04:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbb02c98700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbafc1035e0 0x7fbafc1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbb02c98700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbafc103fa0 0x7fbafc1a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbb02c98700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbafc109680 0x7fbafc10d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbb02c98700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbafc113650 con 0x7fbafc109680 2024-06-04T05:04:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbb02c98700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbafc1134d0 con 0x7fbafc1035e0 2024-06-04T05:04:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbb02c98700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbafc1137d0 con 0x7fbafc103fa0 2024-06-04T05:04:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbb01235700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbafc109680 0x7fbafc10d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbb01235700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbafc109680 0x7fbafc10d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33902/0 (socket says 172.21.15.70:33902) 2024-06-04T05:04:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbb01235700 1 -- 172.21.15.70:0/454157470 learned_addr learned my addr 172.21.15.70:0/454157470 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbafbfff700 1 --2- 172.21.15.70:0/454157470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbafc103fa0 0x7fbafc1a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbb00a34700 1 --2- 172.21.15.70:0/454157470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbafc1035e0 0x7fbafc1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbb01235700 1 -- 172.21.15.70:0/454157470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbafc103fa0 msgr2=0x7fbafc1a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbb01235700 1 --2- 172.21.15.70:0/454157470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbafc103fa0 0x7fbafc1a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbb01235700 1 -- 172.21.15.70:0/454157470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbafc1035e0 msgr2=0x7fbafc1a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbb01235700 1 --2- 172.21.15.70:0/454157470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbafc1035e0 0x7fbafc1a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbb01235700 1 -- 172.21.15.70:0/454157470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbafc10dd50 con 0x7fbafc109680 2024-06-04T05:04:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbafbfff700 1 --2- 172.21.15.70:0/454157470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbafc103fa0 0x7fbafc1a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.439+0000 7fbb00a34700 1 --2- 172.21.15.70:0/454157470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbafc1035e0 0x7fbafc1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:04:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.440+0000 7fbb01235700 1 --2- 172.21.15.70:0/454157470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbafc109680 0x7fbafc10d4b0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4009850 tx=0x7fbaf4007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.440+0000 7fbaf9ffb700 1 -- 172.21.15.70:0/454157470 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf401f070 con 0x7fbafc109680 2024-06-04T05:04:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.440+0000 7fbb02c98700 1 -- 172.21.15.70:0/454157470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbafc10dfe0 con 0x7fbafc109680 2024-06-04T05:04:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.440+0000 7fbaf9ffb700 1 -- 172.21.15.70:0/454157470 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbaf4004c50 con 0x7fbafc109680 2024-06-04T05:04:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.440+0000 7fbaf9ffb700 1 -- 172.21.15.70:0/454157470 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf401b690 con 0x7fbafc109680 2024-06-04T05:04:53.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.440+0000 7fbb02c98700 1 -- 172.21.15.70:0/454157470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbafc19c340 con 0x7fbafc109680 2024-06-04T05:04:53.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.443+0000 7fbaf9ffb700 1 -- 172.21.15.70:0/454157470 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fbaf4022070 con 0x7fbafc109680 2024-06-04T05:04:53.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.443+0000 7fbaf9ffb700 1 --2- 172.21.15.70:0/454157470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbaec0b29f0 0x7fbaec0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.443+0000 7fbb00a34700 1 --2- 172.21.15.70:0/454157470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbaec0b29f0 0x7fbaec0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.443+0000 7fbaf9ffb700 1 -- 172.21.15.70:0/454157470 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fbaf400f020 con 0x7fbafc109680 2024-06-04T05:04:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.444+0000 7fbb0d52db80 1 Processor -- start 2024-06-04T05:04:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.444+0000 7fbb0d52db80 1 -- start start 2024-06-04T05:04:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.444+0000 7fbb00a34700 1 --2- 172.21.15.70:0/454157470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbaec0b29f0 0x7fbaec0b4eb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0005950 tx=0x7fbaf00058e0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.444+0000 7fbb0d52db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b53935f860 0x55b53935fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.444+0000 7fbb0d52db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b539360210 0x55b539363940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.444+0000 7fbb0d52db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b539363f60 0x55b5393643a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.444+0000 7fbb0d52db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b5392bec90 con 0x55b539360210 2024-06-04T05:04:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.444+0000 7fbb0d52db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b5392ae830 con 0x55b53935f860 2024-06-04T05:04:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.444+0000 7fbb0d52db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b5392b2870 con 0x55b539363f60 2024-06-04T05:04:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.444+0000 7fbafbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b539360210 0x55b539363940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.445+0000 7fbafbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b539360210 0x55b539363940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33908/0 (socket says 172.21.15.70:33908) 2024-06-04T05:04:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.445+0000 7fbafbfff700 1 -- 172.21.15.70:0/3218793325 learned_addr learned my addr 172.21.15.70:0/3218793325 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.445+0000 7fbb01235700 1 --2- 172.21.15.70:0/3218793325 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b539363f60 0x55b5393643a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.445+0000 7fbb00a34700 1 --2- 172.21.15.70:0/3218793325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b53935f860 0x55b53935fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.445+0000 7fbafbfff700 1 -- 172.21.15.70:0/3218793325 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b539363f60 msgr2=0x55b5393643a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.445+0000 7fbafbfff700 1 --2- 172.21.15.70:0/3218793325 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b539363f60 0x55b5393643a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.445+0000 7fbafbfff700 1 -- 172.21.15.70:0/3218793325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b53935f860 msgr2=0x55b53935fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.445+0000 7fbafbfff700 1 --2- 172.21.15.70:0/3218793325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b53935f860 0x55b53935fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.445+0000 7fbafbfff700 1 -- 172.21.15.70:0/3218793325 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5391fe8d0 con 0x55b539360210 2024-06-04T05:04:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.445+0000 7fbafbfff700 1 --2- 172.21.15.70:0/3218793325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b539360210 0x55b539363940 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fbae80115f0 tx=0x7fbae8011900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5598f83d05405708 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.445+0000 7fbae6ffd700 1 -- 172.21.15.70:0/3218793325 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae8004220 con 0x55b539360210 2024-06-04T05:04:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.445+0000 7fbae6ffd700 1 -- 172.21.15.70:0/3218793325 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbae80043c0 con 0x55b539360210 2024-06-04T05:04:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.445+0000 7fbae6ffd700 1 -- 172.21.15.70:0/3218793325 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae8019810 con 0x55b539360210 2024-06-04T05:04:53.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.446+0000 7fbb0d52db80 1 -- 172.21.15.70:0/3218793325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b539360210 msgr2=0x55b539363940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:53.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.446+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/3218793325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b539360210 0x55b539363940 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fbae80115f0 tx=0x7fbae8011900 comp rx=0 tx=0).stop 2024-06-04T05:04:53.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.446+0000 7fbb0d52db80 1 -- 172.21.15.70:0/3218793325 shutdown_connections 2024-06-04T05:04:53.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.446+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/3218793325 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b539363f60 0x55b5393643a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.446+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/3218793325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b53935f860 0x55b53935fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.446+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/3218793325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b539360210 0x55b539363940 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.446+0000 7fbb0d52db80 1 -- 172.21.15.70:0/3218793325 >> 172.21.15.70:0/3218793325 conn(0x55b53928d080 msgr2=0x55b539359e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:53.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.446+0000 7fbb0d52db80 1 -- 172.21.15.70:0/3218793325 shutdown_connections 2024-06-04T05:04:53.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.446+0000 7fbb0d52db80 1 -- 172.21.15.70:0/3218793325 wait complete. 2024-06-04T05:04:53.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.447+0000 7fbb0d52db80 1 -- start start 2024-06-04T05:04:53.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.447+0000 7fbb0d52db80 1 Processor -- start 2024-06-04T05:04:53.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.447+0000 7fbb0d52db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b53935f860 0x55b53936b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.447+0000 7fbb0d52db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b539360210 0x55b5393686c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.447+0000 7fbb0d52db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b539363f60 0x55b539368c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.447+0000 7fbb0d52db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b5391b4fa0 con 0x55b53935f860 2024-06-04T05:04:53.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.447+0000 7fbb0d52db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b5392507c0 con 0x55b539363f60 2024-06-04T05:04:53.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.447+0000 7fbb0d52db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b53936c120 con 0x55b539360210 2024-06-04T05:04:53.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.447+0000 7fbb0d52db80 10 client.0 _ll_register_callbacks cb 0x55b539249920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:53.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.447+0000 7fbb00a34700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b53935f860 0x55b53936b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.447+0000 7fbb00a34700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b53935f860 0x55b53936b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33924/0 (socket says 172.21.15.70:33924) 2024-06-04T05:04:53.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.447+0000 7fbb00a34700 1 -- 172.21.15.70:0/320042360 learned_addr learned my addr 172.21.15.70:0/320042360 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:53.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.447+0000 7fbafbfff700 1 --2- 172.21.15.70:0/320042360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b539360210 0x55b5393686c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.448+0000 7fbb01235700 1 --2- 172.21.15.70:0/320042360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b539363f60 0x55b539368c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.448+0000 7fbafbfff700 1 -- 172.21.15.70:0/320042360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b539363f60 msgr2=0x55b539368c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:53.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.448+0000 7fbafbfff700 1 --2- 172.21.15.70:0/320042360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b539363f60 0x55b539368c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.448+0000 7fbafbfff700 1 -- 172.21.15.70:0/320042360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b53935f860 msgr2=0x55b53936b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.448+0000 7fbafbfff700 1 --2- 172.21.15.70:0/320042360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b53935f860 0x55b53936b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.448+0000 7fbafbfff700 1 -- 172.21.15.70:0/320042360 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b53924d690 con 0x55b539360210 2024-06-04T05:04:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.448+0000 7fbb00a34700 1 --2- 172.21.15.70:0/320042360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b53935f860 0x55b53936b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:04:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.448+0000 7fbafbfff700 1 --2- 172.21.15.70:0/320042360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b539360210 0x55b5393686c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbae8006010 tx=0x7fbae8005720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2f776a0fd400604 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.448+0000 7fbadbfff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:04:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.448+0000 7fbadbfff700 1 -- 172.21.15.70:0/320042360 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae8027070 con 0x55b539360210 2024-06-04T05:04:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.448+0000 7fbadbfff700 1 -- 172.21.15.70:0/320042360 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fbae8004be0 con 0x55b539360210 2024-06-04T05:04:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.448+0000 7fbb0d52db80 1 -- 172.21.15.70:0/320042360 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbae80043c0 con 0x55b539360210 2024-06-04T05:04:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.448+0000 7fbadbfff700 1 -- 172.21.15.70:0/320042360 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbae8033050 con 0x55b539360210 2024-06-04T05:04:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.448+0000 7fbb0d52db80 10 client.26842 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.448+0000 7fbb0d52db80 1 -- 172.21.15.70:0/320042360 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b539376830 con 0x55b539360210 2024-06-04T05:04:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.449+0000 7fbadbfff700 1 -- 172.21.15.70:0/320042360 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbae80043c0 con 0x55b539360210 2024-06-04T05:04:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.449+0000 7fbadbfff700 1 -- 172.21.15.70:0/320042360 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fbae8019430 con 0x55b539360210 2024-06-04T05:04:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.449+0000 7fbb0d52db80 10 client.26842 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.449+0000 7fbb0d52db80 10 client.26842 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:04:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.449+0000 7fbb0d52db80 20 client.26842 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.449+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/320042360 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b539378d60 0x55b53937b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.449+0000 7fbb0d52db80 4 client.26842 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:04:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.449+0000 7fbb0d52db80 1 -- 172.21.15.70:0/320042360 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b53932b9e0 con 0x55b539378d60 2024-06-04T05:04:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.449+0000 7fbb00a34700 1 --2- 172.21.15.70:0/320042360 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b539378d60 0x55b53937b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.449+0000 7fbb00a34700 1 --2- 172.21.15.70:0/320042360 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b539378d60 0x55b53937b150 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e58fb2c9640a46b5 server_cookie=ddc1510086581f99 in_seq=0 out_seq=0 2024-06-04T05:04:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.449+0000 7fbadbfff700 10 client.26842 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.450+0000 7fbadbfff700 1 -- 172.21.15.70:0/320042360 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b53932b9e0 con 0x55b539378d60 2024-06-04T05:04:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.450+0000 7fbadbfff700 10 client.26842 handle_command_reply: tid=0 2024-06-04T05:04:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.450+0000 7fbb0d52db80 1 client.26842 shutdown 2024-06-04T05:04:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.450+0000 7fbb0d52db80 1 -- 172.21.15.70:0/320042360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b539360210 msgr2=0x55b5393686c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.450+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/320042360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b539360210 0x55b5393686c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbae8006010 tx=0x7fbae8005720 comp rx=0 tx=0).stop 2024-06-04T05:04:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.450+0000 7fbb0d52db80 1 -- 172.21.15.70:0/320042360 shutdown_connections 2024-06-04T05:04:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.450+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/320042360 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b539378d60 0x55b53937b150 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.450+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/320042360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b539363f60 0x55b539368c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.450+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/320042360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b53935f860 0x55b53936b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.450+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/320042360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b539360210 0x55b5393686c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.450+0000 7fbb0d52db80 1 -- 172.21.15.70:0/320042360 >> 172.21.15.70:0/320042360 conn(0x55b53928d080 msgr2=0x55b539365350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.450+0000 7fbadbfff700 0 client.26842 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.450+0000 7fbb0d52db80 1 -- 172.21.15.70:0/320042360 shutdown_connections 2024-06-04T05:04:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.450+0000 7fbb0d52db80 1 -- 172.21.15.70:0/320042360 wait complete. 2024-06-04T05:04:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.451+0000 7fbb0d52db80 20 client.26842 trim_cache size 0 max 16384 2024-06-04T05:04:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.472+0000 7fbb0d52db80 1 Processor -- start 2024-06-04T05:04:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.472+0000 7fbb0d52db80 1 -- start start 2024-06-04T05:04:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.472+0000 7fbb0d52db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b53935f860 0x55b539367320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.472+0000 7fbb0d52db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b539360210 0x55b539359db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.472+0000 7fbb0d52db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b539363f60 0x55b539380890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.472+0000 7fbb0d52db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbafc1bcad0 con 0x55b53935f860 2024-06-04T05:04:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.472+0000 7fbb0d52db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbafc1be610 con 0x55b539363f60 2024-06-04T05:04:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.472+0000 7fbb0d52db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbafc1bd1a0 con 0x55b539360210 2024-06-04T05:04:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.473+0000 7fbb00a34700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b53935f860 0x55b539367320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.473+0000 7fbb00a34700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b53935f860 0x55b539367320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33930/0 (socket says 172.21.15.70:33930) 2024-06-04T05:04:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.473+0000 7fbb00a34700 1 -- 172.21.15.70:0/1596462784 learned_addr learned my addr 172.21.15.70:0/1596462784 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.473+0000 7fbafbfff700 1 --2- 172.21.15.70:0/1596462784 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b539360210 0x55b539359db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.473+0000 7fbb01235700 1 --2- 172.21.15.70:0/1596462784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b539363f60 0x55b539380890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.473+0000 7fbb00a34700 1 -- 172.21.15.70:0/1596462784 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b539360210 msgr2=0x55b539359db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.473+0000 7fbb00a34700 1 --2- 172.21.15.70:0/1596462784 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b539360210 0x55b539359db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.473+0000 7fbb00a34700 1 -- 172.21.15.70:0/1596462784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b539363f60 msgr2=0x55b539380890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.473+0000 7fbb00a34700 1 --2- 172.21.15.70:0/1596462784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b539363f60 0x55b539380890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.473+0000 7fbb00a34700 1 -- 172.21.15.70:0/1596462784 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5393695a0 con 0x55b53935f860 2024-06-04T05:04:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.473+0000 7fbb00a34700 1 --2- 172.21.15.70:0/1596462784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b53935f860 0x55b539367320 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fbaf000fab0 tx=0x7fbaf0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa24487cdfc28a9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.473+0000 7fbae67fc700 1 -- 172.21.15.70:0/1596462784 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf002e070 con 0x55b53935f860 2024-06-04T05:04:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.473+0000 7fbae67fc700 1 -- 172.21.15.70:0/1596462784 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbaf0003d10 con 0x55b53935f860 2024-06-04T05:04:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.473+0000 7fbae67fc700 1 -- 172.21.15.70:0/1596462784 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0029be0 con 0x55b53935f860 2024-06-04T05:04:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.474+0000 7fbb0d52db80 1 -- 172.21.15.70:0/1596462784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b53935f860 msgr2=0x55b539367320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.474+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/1596462784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b53935f860 0x55b539367320 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fbaf000fab0 tx=0x7fbaf0015000 comp rx=0 tx=0).stop 2024-06-04T05:04:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.474+0000 7fbb0d52db80 1 -- 172.21.15.70:0/1596462784 shutdown_connections 2024-06-04T05:04:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.474+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/1596462784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b539363f60 0x55b539380890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.474+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/1596462784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b53935f860 0x55b539367320 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.474+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/1596462784 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b539360210 0x55b539359db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.474+0000 7fbb0d52db80 1 -- 172.21.15.70:0/1596462784 >> 172.21.15.70:0/1596462784 conn(0x55b53928d080 msgr2=0x55b5393596a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.474+0000 7fbb0d52db80 1 -- 172.21.15.70:0/1596462784 shutdown_connections 2024-06-04T05:04:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.474+0000 7fbb0d52db80 1 -- 172.21.15.70:0/1596462784 wait complete. 2024-06-04T05:04:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.475+0000 7fbb0d52db80 1 -- start start 2024-06-04T05:04:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.475+0000 7fbb0d52db80 1 Processor -- start 2024-06-04T05:04:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.475+0000 7fbb0d52db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b53935f860 0x55b539371a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.475+0000 7fbb0d52db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b539360210 0x55b539361250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.475+0000 7fbb0d52db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b539363f60 0x55b539361790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.475+0000 7fbb0d52db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbafc1bdf40 con 0x55b539360210 2024-06-04T05:04:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.475+0000 7fbb0d52db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbafc1bd870 con 0x55b539363f60 2024-06-04T05:04:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.475+0000 7fbb0d52db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbafc1baa90 con 0x55b53935f860 2024-06-04T05:04:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.475+0000 7fbb0d52db80 10 client.0 _ll_register_callbacks cb 0x55b539249920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:04:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.475+0000 7fbafbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b539360210 0x55b539361250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.475+0000 7fbb00a34700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b53935f860 0x55b539371a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.475+0000 7fbb00a34700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b53935f860 0x55b539371a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:34940/0 (socket says 172.21.15.70:34940) 2024-06-04T05:04:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.475+0000 7fbafbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b539360210 0x55b539361250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33944/0 (socket says 172.21.15.70:33944) 2024-06-04T05:04:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.475+0000 7fbb00a34700 1 -- 172.21.15.70:0/1643875318 learned_addr learned my addr 172.21.15.70:0/1643875318 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:04:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.475+0000 7fbb01235700 1 --2- 172.21.15.70:0/1643875318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b539363f60 0x55b539361790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.476+0000 7fbb00a34700 1 -- 172.21.15.70:0/1643875318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b539363f60 msgr2=0x55b539361790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.476+0000 7fbb00a34700 1 --2- 172.21.15.70:0/1643875318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b539363f60 0x55b539361790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.476+0000 7fbb00a34700 1 -- 172.21.15.70:0/1643875318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b539360210 msgr2=0x55b539361250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:04:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.476+0000 7fbb00a34700 1 --2- 172.21.15.70:0/1643875318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b539360210 0x55b539361250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.476+0000 7fbb00a34700 1 -- 172.21.15.70:0/1643875318 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b539361f90 con 0x55b53935f860 2024-06-04T05:04:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.476+0000 7fbb00a34700 1 --2- 172.21.15.70:0/1643875318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b53935f860 0x55b539371a40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0010010 tx=0x7fbaf0004630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87e03af704ffc751 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:04:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.476+0000 7fbadbfff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:04:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.476+0000 7fbadbfff700 1 -- 172.21.15.70:0/1643875318 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf002e040 con 0x55b53935f860 2024-06-04T05:04:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.476+0000 7fbadbfff700 1 -- 172.21.15.70:0/1643875318 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fbaf0003680 con 0x55b53935f860 2024-06-04T05:04:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.476+0000 7fbb0d52db80 1 -- 172.21.15.70:0/1643875318 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbaf0003d10 con 0x55b53935f860 2024-06-04T05:04:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.476+0000 7fbadbfff700 1 -- 172.21.15.70:0/1643875318 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbaf0041050 con 0x55b53935f860 2024-06-04T05:04:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.476+0000 7fbb0d52db80 10 client.26848 fetch_fsmap learned FSMap version 32 2024-06-04T05:04:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.476+0000 7fbb0d52db80 1 -- 172.21.15.70:0/1643875318 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b5393e36e0 con 0x55b53935f860 2024-06-04T05:04:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.476+0000 7fbadbfff700 1 -- 172.21.15.70:0/1643875318 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbaf0003d10 con 0x55b53935f860 2024-06-04T05:04:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.477+0000 7fbadbfff700 1 -- 172.21.15.70:0/1643875318 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fbaf0011440 con 0x55b53935f860 2024-06-04T05:04:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.477+0000 7fbb0d52db80 10 client.26848 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:04:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.477+0000 7fbb0d52db80 10 client.26848 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:04:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.477+0000 7fbb0d52db80 20 client.26848 populate_metadata read hostname 'smithi070' 2024-06-04T05:04:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.477+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/1643875318 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b5393e3d30 0x55b5393e6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:04:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.477+0000 7fbb0d52db80 4 client.26848 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:04:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.477+0000 7fbb0d52db80 1 -- 172.21.15.70:0/1643875318 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b5392c0be0 con 0x55b5393e3d30 2024-06-04T05:04:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.477+0000 7fbafbfff700 1 --2- 172.21.15.70:0/1643875318 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b5393e3d30 0x55b5393e6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:04:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.477+0000 7fbadbfff700 10 client.26848 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.477+0000 7fbafbfff700 1 --2- 172.21.15.70:0/1643875318 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b5393e3d30 0x55b5393e6150 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d8f34efd37d83f1 server_cookie=bb48683db048a938 in_seq=0 out_seq=0 2024-06-04T05:04:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.477+0000 7fbadbfff700 1 -- 172.21.15.70:0/1643875318 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b5392c0be0 con 0x55b5393e3d30 2024-06-04T05:04:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.477+0000 7fbadbfff700 10 client.26848 handle_command_reply: tid=0 2024-06-04T05:04:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.477+0000 7fbb0d52db80 1 client.26848 shutdown 2024-06-04T05:04:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.477+0000 7fbb0d52db80 1 -- 172.21.15.70:0/1643875318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b53935f860 msgr2=0x55b539371a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.477+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/1643875318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b53935f860 0x55b539371a40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0010010 tx=0x7fbaf0004630 comp rx=0 tx=0).stop 2024-06-04T05:04:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.477+0000 7fbb0d52db80 1 -- 172.21.15.70:0/1643875318 shutdown_connections 2024-06-04T05:04:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.477+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/1643875318 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b5393e3d30 0x55b5393e6150 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.478+0000 7fbadbfff700 0 client.26848 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:04:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.478+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/1643875318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b539363f60 0x55b539361790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.478+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/1643875318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b53935f860 0x55b539371a40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.478+0000 7fbb0d52db80 1 --2- 172.21.15.70:0/1643875318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b539360210 0x55b539361250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.478+0000 7fbb0d52db80 1 -- 172.21.15.70:0/1643875318 >> 172.21.15.70:0/1643875318 conn(0x55b53928d080 msgr2=0x55b53935ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.478+0000 7fbb0d52db80 1 -- 172.21.15.70:0/1643875318 shutdown_connections 2024-06-04T05:04:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.478+0000 7fbb0d52db80 1 -- 172.21.15.70:0/1643875318 wait complete. 2024-06-04T05:04:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.478+0000 7fbb0d52db80 20 client.26848 trim_cache size 0 max 16384 2024-06-04T05:04:53.487 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:04:53.487 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:04:53.487 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:04:53.487 INFO:teuthology.orchestra.run.smithi070.stdout: "83046a07-2799-458e-abac-dafb988bc251": { 2024-06-04T05:04:53.487 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:04:53.487 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "83046a07-2799-458e-abac-dafb988bc251", 2024-06-04T05:04:53.487 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:04:53.487 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:04:53.487 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:04:53.487 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:04:53.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.478+0000 7fbb02c98700 1 -- 172.21.15.70:0/454157470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbaec0b29f0 msgr2=0x7fbaec0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:53.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.478+0000 7fbb02c98700 1 --2- 172.21.15.70:0/454157470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbaec0b29f0 0x7fbaec0b4eb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0005950 tx=0x7fbaf00058e0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.478+0000 7fbb02c98700 1 -- 172.21.15.70:0/454157470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbafc109680 msgr2=0x7fbafc10d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:04:53.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.478+0000 7fbb02c98700 1 --2- 172.21.15.70:0/454157470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbafc109680 0x7fbafc10d4b0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4009850 tx=0x7fbaf4007a20 comp rx=0 tx=0).stop 2024-06-04T05:04:53.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.479+0000 7fbb02c98700 1 -- 172.21.15.70:0/454157470 shutdown_connections 2024-06-04T05:04:53.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.479+0000 7fbb02c98700 1 --2- 172.21.15.70:0/454157470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbaec0b29f0 0x7fbaec0b4eb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.479+0000 7fbb02c98700 1 --2- 172.21.15.70:0/454157470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbafc109680 0x7fbafc10d4b0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.479+0000 7fbb02c98700 1 --2- 172.21.15.70:0/454157470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbafc1035e0 0x7fbafc1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.479+0000 7fbb02c98700 1 --2- 172.21.15.70:0/454157470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbafc103fa0 0x7fbafc1a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:04:53.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.479+0000 7fbb02c98700 1 -- 172.21.15.70:0/454157470 >> 172.21.15.70:0/454157470 conn(0x7fbafc07a6f0 msgr2=0x7fbafc10e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:04:53.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.479+0000 7fbb02c98700 1 -- 172.21.15.70:0/454157470 shutdown_connections 2024-06-04T05:04:53.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:04:53.479+0000 7fbb02c98700 1 -- 172.21.15.70:0/454157470 wait complete. 2024-06-04T05:04:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:53 smithi112 ceph-mon[36453]: pgmap v236: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 1.9 MiB/s wr, 202 op/s 2024-06-04T05:04:53.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 131249 2.06 MB/sec execute 226 sec latency 60.580 ms 2024-06-04T05:04:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:53 smithi098 ceph-mon[35569]: pgmap v236: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 1.9 MiB/s wr, 202 op/s 2024-06-04T05:04:54.222 INFO:tasks.cephfs.filesystem:scrub status for tag:83046a07-2799-458e-abac-dafb988bc251 - {'path': '/', 'tag': '83046a07-2799-458e-abac-dafb988bc251', 'options': 'recursive,force'} 2024-06-04T05:04:54.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 131637 2.05 MB/sec execute 227 sec latency 12.088 ms 2024-06-04T05:04:55.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:55 smithi070 ceph-mon[32631]: pgmap v237: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 1.2 MiB/s wr, 139 op/s 2024-06-04T05:04:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:55 smithi112 ceph-mon[36453]: pgmap v237: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 1.2 MiB/s wr, 139 op/s 2024-06-04T05:04:55.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 131942 2.05 MB/sec execute 228 sec latency 14.081 ms 2024-06-04T05:04:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:55 smithi098 ceph-mon[35569]: pgmap v237: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 1.2 MiB/s wr, 139 op/s 2024-06-04T05:04:56.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 132484 2.05 MB/sec execute 229 sec latency 12.321 ms 2024-06-04T05:04:57.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:57 smithi070 ceph-mon[32631]: pgmap v238: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 2.0 MiB/s wr, 202 op/s 2024-06-04T05:04:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:57 smithi112 ceph-mon[36453]: pgmap v238: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 2.0 MiB/s wr, 202 op/s 2024-06-04T05:04:57.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 133052 2.05 MB/sec execute 230 sec latency 19.289 ms 2024-06-04T05:04:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:57 smithi098 ceph-mon[35569]: pgmap v238: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 2.0 MiB/s wr, 202 op/s 2024-06-04T05:04:58.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 133667 2.05 MB/sec execute 231 sec latency 19.468 ms 2024-06-04T05:04:59.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:58 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:04:59.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:58 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:04:59.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:58 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:04:59.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:04:58 smithi070 ceph-mon[32631]: pgmap v239: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-06-04T05:04:59.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:58 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:04:59.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:58 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:04:59.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:58 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:04:59.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:04:58 smithi112 ceph-mon[36453]: pgmap v239: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-06-04T05:04:59.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:58 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:04:59.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:58 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:04:59.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:58 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:04:59.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:04:58 smithi098 ceph-mon[35569]: pgmap v239: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-06-04T05:04:59.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 134277 2.06 MB/sec execute 232 sec latency 16.173 ms 2024-06-04T05:05:00.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 134827 2.08 MB/sec execute 233 sec latency 18.566 ms 2024-06-04T05:05:01.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:01 smithi112 ceph-mon[36453]: pgmap v240: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-06-04T05:05:01.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:01 smithi098 ceph-mon[35569]: pgmap v240: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-06-04T05:05:01.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:01 smithi070 ceph-mon[32631]: pgmap v240: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-06-04T05:05:01.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 135198 2.07 MB/sec execute 234 sec latency 12.454 ms 2024-06-04T05:05:02.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 135488 2.07 MB/sec execute 235 sec latency 17.934 ms 2024-06-04T05:05:03.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:03 smithi112 ceph-mon[36453]: pgmap v241: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 2.0 MiB/s wr, 212 op/s 2024-06-04T05:05:03.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:03 smithi098 ceph-mon[35569]: pgmap v241: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 2.0 MiB/s wr, 212 op/s 2024-06-04T05:05:03.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:03 smithi070 ceph-mon[32631]: pgmap v241: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 2.0 MiB/s wr, 212 op/s 2024-06-04T05:05:03.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.664+0000 7fcee4e2a640 1 -- 172.21.15.70:0/2080095775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcee0102340 msgr2=0x7fcee010e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:03.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.664+0000 7fcee4e2a640 1 --2- 172.21.15.70:0/2080095775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcee0102340 0x7fcee010e800 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fced8009cd0 tx=0x7fced802a8e0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.664+0000 7fcee4e2a640 1 -- 172.21.15.70:0/2080095775 shutdown_connections 2024-06-04T05:05:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.664+0000 7fcee4e2a640 1 --2- 172.21.15.70:0/2080095775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcee0102340 0x7fcee010e800 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.664+0000 7fcee4e2a640 1 --2- 172.21.15.70:0/2080095775 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcee0101980 0x7fcee0101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.664+0000 7fcee4e2a640 1 --2- 172.21.15.70:0/2080095775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee0107940 0x7fcee0107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.664+0000 7fcee4e2a640 1 -- 172.21.15.70:0/2080095775 >> 172.21.15.70:0/2080095775 conn(0x7fcee0078ad0 msgr2=0x7fcee0078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.664+0000 7fcee4e2a640 1 -- 172.21.15.70:0/2080095775 shutdown_connections 2024-06-04T05:05:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.664+0000 7fcee4e2a640 1 -- 172.21.15.70:0/2080095775 wait complete. 2024-06-04T05:05:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fcee4e2a640 1 Processor -- start 2024-06-04T05:05:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fcee4e2a640 1 -- start start 2024-06-04T05:05:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fcee4e2a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcee0101980 0x7fcee0199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fcee4e2a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcee0102340 0x7fcee019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fcee4e2a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee0107940 0x7fcee01a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fcee4e2a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcee0111ab0 con 0x7fcee0102340 2024-06-04T05:05:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fcee4e2a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcee0111930 con 0x7fcee0107940 2024-06-04T05:05:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fcee4e2a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcee0111c30 con 0x7fcee0101980 2024-06-04T05:05:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fceddd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcee0102340 0x7fcee019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fceddd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcee0102340 0x7fcee019a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34408/0 (socket says 172.21.15.70:34408) 2024-06-04T05:05:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fceddd9b640 1 -- 172.21.15.70:0/2048149215 learned_addr learned my addr 172.21.15.70:0/2048149215 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fcede59c640 1 --2- 172.21.15.70:0/2048149215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcee0101980 0x7fcee0199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fceded9d640 1 --2- 172.21.15.70:0/2048149215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee0107940 0x7fcee01a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fceddd9b640 1 -- 172.21.15.70:0/2048149215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcee0101980 msgr2=0x7fcee0199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fceddd9b640 1 --2- 172.21.15.70:0/2048149215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcee0101980 0x7fcee0199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fceddd9b640 1 -- 172.21.15.70:0/2048149215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee0107940 msgr2=0x7fcee01a4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fceddd9b640 1 --2- 172.21.15.70:0/2048149215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee0107940 0x7fcee01a4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fceddd9b640 1 -- 172.21.15.70:0/2048149215 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee01a5490 con 0x7fcee0102340 2024-06-04T05:05:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.665+0000 7fcede59c640 1 --2- 172.21.15.70:0/2048149215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcee0101980 0x7fcee0199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.666+0000 7fceddd9b640 1 --2- 172.21.15.70:0/2048149215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcee0102340 0x7fcee019a410 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fced400e9f0 tx=0x7fced400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.666+0000 7fcecf7fe640 1 -- 172.21.15.70:0/2048149215 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced400cd80 con 0x7fcee0102340 2024-06-04T05:05:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.666+0000 7fcee4e2a640 1 -- 172.21.15.70:0/2048149215 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcee01a5660 con 0x7fcee0102340 2024-06-04T05:05:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.666+0000 7fcecf7fe640 1 -- 172.21.15.70:0/2048149215 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fced40054b0 con 0x7fcee0102340 2024-06-04T05:05:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.666+0000 7fcecf7fe640 1 -- 172.21.15.70:0/2048149215 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced4010430 con 0x7fcee0102340 2024-06-04T05:05:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.666+0000 7fcee4e2a640 1 -- 172.21.15.70:0/2048149215 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcee01a5b80 con 0x7fcee0102340 2024-06-04T05:05:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.668+0000 7fcecf7fe640 1 -- 172.21.15.70:0/2048149215 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fced40046c0 con 0x7fcee0102340 2024-06-04T05:05:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.668+0000 7fcee7c22740 1 Processor -- start 2024-06-04T05:05:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fcee7c22740 1 -- start start 2024-06-04T05:05:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fcee7c22740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e675700 0x560b7e675ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fcee7c22740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e676020 0x560b7e6848d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fcee7c22740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e684e10 0x560b7e687200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fcee7c22740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560b7e5e7c30 con 0x560b7e684e10 2024-06-04T05:05:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fcee7c22740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560b7e5d77d0 con 0x560b7e675700 2024-06-04T05:05:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fcee7c22740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560b7e5db810 con 0x560b7e676020 2024-06-04T05:05:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fceded9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e675700 0x560b7e675ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fceddd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e684e10 0x560b7e687200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fceded9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e675700 0x560b7e675ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:53012/0 (socket says 172.21.15.70:53012) 2024-06-04T05:05:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fceded9d640 1 -- 172.21.15.70:0/2112206042 learned_addr learned my addr 172.21.15.70:0/2112206042 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fcecf7fe640 1 --2- 172.21.15.70:0/2048149215 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcec80b2a10 0x7fcec80b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fcede59c640 1 --2- 172.21.15.70:0/2112206042 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e676020 0x560b7e6848d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fceded9d640 1 --2- 172.21.15.70:0/2048149215 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcec80b2a10 0x7fcec80b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fceded9d640 1 -- 172.21.15.70:0/2112206042 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e676020 msgr2=0x560b7e6848d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fcecf7fe640 1 -- 172.21.15.70:0/2048149215 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fced40e2990 con 0x7fcee0102340 2024-06-04T05:05:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fceded9d640 1 --2- 172.21.15.70:0/2112206042 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e676020 0x560b7e6848d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.669+0000 7fceded9d640 1 -- 172.21.15.70:0/2112206042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e684e10 msgr2=0x560b7e687200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.670+0000 7fceded9d640 1 --2- 172.21.15.70:0/2112206042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e684e10 0x560b7e687200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.670+0000 7fceded9d640 1 -- 172.21.15.70:0/2112206042 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b7e2ec210 con 0x560b7e675700 2024-06-04T05:05:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.670+0000 7fceddd9b640 1 --2- 172.21.15.70:0/2112206042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e684e10 0x560b7e687200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.670+0000 7fceded9d640 1 --2- 172.21.15.70:0/2112206042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e675700 0x560b7e675ae0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fced8035000 tx=0x7fced8031920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5c521c2eb455302 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.670+0000 7fceded9d640 1 --2- 172.21.15.70:0/2048149215 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcec80b2a10 0x7fcec80b4ed0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fced8047000 tx=0x7fced804a000 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.670+0000 7fceccff9640 1 -- 172.21.15.70:0/2112206042 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8004410 con 0x560b7e675700 2024-06-04T05:05:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.670+0000 7fceccff9640 1 -- 172.21.15.70:0/2112206042 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fced8036070 con 0x560b7e675700 2024-06-04T05:05:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.670+0000 7fceccff9640 1 -- 172.21.15.70:0/2112206042 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8030d20 con 0x560b7e675700 2024-06-04T05:05:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.671+0000 7fcee7c22740 1 -- 172.21.15.70:0/2112206042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e675700 msgr2=0x560b7e675ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.671+0000 7fcee7c22740 1 --2- 172.21.15.70:0/2112206042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e675700 0x560b7e675ae0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fced8035000 tx=0x7fced8031920 comp rx=0 tx=0).stop 2024-06-04T05:05:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.671+0000 7fcee7c22740 1 -- 172.21.15.70:0/2112206042 shutdown_connections 2024-06-04T05:05:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.671+0000 7fcee7c22740 1 --2- 172.21.15.70:0/2112206042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e684e10 0x560b7e687200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.671+0000 7fcee7c22740 1 --2- 172.21.15.70:0/2112206042 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e676020 0x560b7e6848d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.671+0000 7fcee7c22740 1 --2- 172.21.15.70:0/2112206042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e675700 0x560b7e675ae0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.671+0000 7fcee7c22740 1 -- 172.21.15.70:0/2112206042 >> 172.21.15.70:0/2112206042 conn(0x560b7e671d30 msgr2=0x560b7e672110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.671+0000 7fcee7c22740 1 -- 172.21.15.70:0/2112206042 shutdown_connections 2024-06-04T05:05:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.671+0000 7fcee7c22740 1 -- 172.21.15.70:0/2112206042 wait complete. 2024-06-04T05:05:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.671+0000 7fcee7c22740 1 -- start start 2024-06-04T05:05:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.671+0000 7fcee7c22740 1 Processor -- start 2024-06-04T05:05:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fcee7c22740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e675700 0x560b7e684880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fcee7c22740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e676020 0x560b7e681dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fcee7c22740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e684e10 0x560b7e682310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fcee7c22740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560b7e2bd790 con 0x560b7e675700 2024-06-04T05:05:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fcee7c22740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560b7e593890 con 0x560b7e676020 2024-06-04T05:05:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fcee7c22740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560b7e6879e0 con 0x560b7e684e10 2024-06-04T05:05:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fcede59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e675700 0x560b7e684880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fcede59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e675700 0x560b7e684880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34434/0 (socket says 172.21.15.70:34434) 2024-06-04T05:05:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fcede59c640 1 -- 172.21.15.70:0/1328151282 learned_addr learned my addr 172.21.15.70:0/1328151282 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fceddd9b640 1 --2- 172.21.15.70:0/1328151282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e676020 0x560b7e681dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fcee7c22740 10 client.0 _ll_register_callbacks cb 0x560b7e2eba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fceded9d640 1 --2- 172.21.15.70:0/1328151282 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e684e10 0x560b7e682310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fceddd9b640 1 -- 172.21.15.70:0/1328151282 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e684e10 msgr2=0x560b7e682310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fceddd9b640 1 --2- 172.21.15.70:0/1328151282 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e684e10 0x560b7e682310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fceddd9b640 1 -- 172.21.15.70:0/1328151282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e675700 msgr2=0x560b7e684880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fceddd9b640 1 --2- 172.21.15.70:0/1328151282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e675700 0x560b7e684880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fceddd9b640 1 -- 172.21.15.70:0/1328151282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b7e3b8620 con 0x560b7e676020 2024-06-04T05:05:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.672+0000 7fcede59c640 1 --2- 172.21.15.70:0/1328151282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e675700 0x560b7e684880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.673+0000 7fceddd9b640 1 --2- 172.21.15.70:0/1328151282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e676020 0x560b7e681dd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fced40e00b0 tx=0x7fced40aa190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d58e14336f6b9cee server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.673+0000 7fcec5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:05:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.673+0000 7fcec5ffb640 1 -- 172.21.15.70:0/1328151282 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced40a93b0 con 0x560b7e676020 2024-06-04T05:05:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.673+0000 7fcec5ffb640 1 -- 172.21.15.70:0/1328151282 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fced40a8ea0 con 0x560b7e676020 2024-06-04T05:05:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.673+0000 7fcee7c22740 1 -- 172.21.15.70:0/1328151282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fced8036070 con 0x560b7e676020 2024-06-04T05:05:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.673+0000 7fcec5ffb640 1 -- 172.21.15.70:0/1328151282 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fced40a53d0 con 0x560b7e676020 2024-06-04T05:05:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.673+0000 7fcee7c22740 10 client.26777 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.673+0000 7fcee7c22740 1 -- 172.21.15.70:0/1328151282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560b7e697c00 con 0x560b7e676020 2024-06-04T05:05:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.673+0000 7fcec5ffb640 1 -- 172.21.15.70:0/1328151282 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fced8036070 con 0x560b7e676020 2024-06-04T05:05:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.674+0000 7fcec5ffb640 1 -- 172.21.15.70:0/1328151282 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fced40a8910 con 0x560b7e676020 2024-06-04T05:05:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.674+0000 7fcee7c22740 10 client.26777 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.674+0000 7fcee7c22740 10 client.26777 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:05:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.674+0000 7fcee7c22740 20 client.26777 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:03.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.674+0000 7fcee7c22740 1 --2- 172.21.15.70:0/1328151282 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560b7e699400 0x560b7e69b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.674+0000 7fcee7c22740 4 client.26777 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:05:03.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.674+0000 7fcee7c22740 1 -- 172.21.15.70:0/1328151282 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b7e5e9b80 con 0x560b7e699400 2024-06-04T05:05:03.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.674+0000 7fcede59c640 1 --2- 172.21.15.70:0/1328151282 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560b7e699400 0x560b7e69b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.674+0000 7fcec5ffb640 10 client.26777 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:03.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.674+0000 7fcede59c640 1 --2- 172.21.15.70:0/1328151282 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560b7e699400 0x560b7e69b8a0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a055ee6e51f58c5 server_cookie=7b1dc48cd230be4a in_seq=0 out_seq=0 2024-06-04T05:05:03.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.675+0000 7fcec5ffb640 1 -- 172.21.15.70:0/1328151282 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560b7e5e9b80 con 0x560b7e699400 2024-06-04T05:05:03.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.675+0000 7fcec5ffb640 10 client.26777 handle_command_reply: tid=0 2024-06-04T05:05:03.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.675+0000 7fcee7c22740 1 client.26777 shutdown 2024-06-04T05:05:03.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.675+0000 7fcee7c22740 1 -- 172.21.15.70:0/1328151282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e676020 msgr2=0x560b7e681dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:03.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.675+0000 7fcee7c22740 1 --2- 172.21.15.70:0/1328151282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e676020 0x560b7e681dd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fced40e00b0 tx=0x7fced40aa190 comp rx=0 tx=0).stop 2024-06-04T05:05:03.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.675+0000 7fcee7c22740 1 -- 172.21.15.70:0/1328151282 shutdown_connections 2024-06-04T05:05:03.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.675+0000 7fcee7c22740 1 --2- 172.21.15.70:0/1328151282 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560b7e699400 0x560b7e69b8a0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.675+0000 7fcee7c22740 1 --2- 172.21.15.70:0/1328151282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e676020 0x560b7e681dd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.675+0000 7fcec5ffb640 0 client.26777 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:03.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.675+0000 7fcee7c22740 1 --2- 172.21.15.70:0/1328151282 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e684e10 0x560b7e682310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.675+0000 7fcee7c22740 1 --2- 172.21.15.70:0/1328151282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e675700 0x560b7e684880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.675+0000 7fcee7c22740 1 -- 172.21.15.70:0/1328151282 >> 172.21.15.70:0/1328151282 conn(0x560b7e671d30 msgr2=0x560b7e677ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:03.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.675+0000 7fcee7c22740 1 -- 172.21.15.70:0/1328151282 shutdown_connections 2024-06-04T05:05:03.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.675+0000 7fcee7c22740 1 -- 172.21.15.70:0/1328151282 wait complete. 2024-06-04T05:05:03.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.676+0000 7fcee7c22740 20 client.26777 trim_cache size 0 max 16384 2024-06-04T05:05:03.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.693+0000 7fcee7c22740 1 Processor -- start 2024-06-04T05:05:03.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.693+0000 7fcee7c22740 1 -- start start 2024-06-04T05:05:03.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.693+0000 7fcee7c22740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e671d30 0x560b7e684610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.693+0000 7fcee7c22740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e675700 0x560b7e670e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.693+0000 7fcee7c22740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e676020 0x560b7e6a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.693+0000 7fcee7c22740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcee01ac450 con 0x560b7e675700 2024-06-04T05:05:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.693+0000 7fcee7c22740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcee01adf90 con 0x560b7e676020 2024-06-04T05:05:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.693+0000 7fcee7c22740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcee01acb20 con 0x560b7e671d30 2024-06-04T05:05:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.694+0000 7fceddd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e675700 0x560b7e670e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.694+0000 7fceded9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e676020 0x560b7e6a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.694+0000 7fceddd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e675700 0x560b7e670e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34448/0 (socket says 172.21.15.70:34448) 2024-06-04T05:05:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.694+0000 7fceded9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e676020 0x560b7e6a11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:53024/0 (socket says 172.21.15.70:53024) 2024-06-04T05:05:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.694+0000 7fcede59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e671d30 0x560b7e684610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.694+0000 7fceddd9b640 1 -- 172.21.15.70:0/1676931164 learned_addr learned my addr 172.21.15.70:0/1676931164 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.694+0000 7fceded9d640 1 -- 172.21.15.70:0/1676931164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e671d30 msgr2=0x560b7e684610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.694+0000 7fceded9d640 1 --2- 172.21.15.70:0/1676931164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e671d30 0x560b7e684610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.694+0000 7fceded9d640 1 -- 172.21.15.70:0/1676931164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e675700 msgr2=0x560b7e670e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:03.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.694+0000 7fceded9d640 1 --2- 172.21.15.70:0/1676931164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e675700 0x560b7e670e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.694+0000 7fceded9d640 1 -- 172.21.15.70:0/1676931164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b7e69bf20 con 0x560b7e676020 2024-06-04T05:05:03.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.694+0000 7fceded9d640 1 --2- 172.21.15.70:0/1676931164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e676020 0x560b7e6a11b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fced8009430 tx=0x7fced802b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d73cb4c2480f34fe server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:03.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.694+0000 7fcec7fff640 1 -- 172.21.15.70:0/1676931164 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8031a40 con 0x560b7e676020 2024-06-04T05:05:03.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.694+0000 7fcec7fff640 1 -- 172.21.15.70:0/1676931164 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fced8036040 con 0x560b7e676020 2024-06-04T05:05:03.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.694+0000 7fcec7fff640 1 -- 172.21.15.70:0/1676931164 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8030b80 con 0x560b7e676020 2024-06-04T05:05:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.695+0000 7fcee7c22740 1 -- 172.21.15.70:0/1676931164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e676020 msgr2=0x560b7e6a11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.695+0000 7fcee7c22740 1 --2- 172.21.15.70:0/1676931164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e676020 0x560b7e6a11b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fced8009430 tx=0x7fced802b040 comp rx=0 tx=0).stop 2024-06-04T05:05:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.695+0000 7fcee7c22740 1 -- 172.21.15.70:0/1676931164 shutdown_connections 2024-06-04T05:05:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.695+0000 7fcee7c22740 1 --2- 172.21.15.70:0/1676931164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e676020 0x560b7e6a11b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.695+0000 7fcee7c22740 1 --2- 172.21.15.70:0/1676931164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e675700 0x560b7e670e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.695+0000 7fcee7c22740 1 --2- 172.21.15.70:0/1676931164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e671d30 0x560b7e684610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.695+0000 7fcee7c22740 1 -- 172.21.15.70:0/1676931164 >> 172.21.15.70:0/1676931164 conn(0x560b7e682470 msgr2=0x560b7e670720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.695+0000 7fcee7c22740 1 -- 172.21.15.70:0/1676931164 shutdown_connections 2024-06-04T05:05:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.695+0000 7fcee7c22740 1 -- 172.21.15.70:0/1676931164 wait complete. 2024-06-04T05:05:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.695+0000 7fcee7c22740 1 -- start start 2024-06-04T05:05:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcee7c22740 1 Processor -- start 2024-06-04T05:05:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcee7c22740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e671d30 0x560b7e691400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcee7c22740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e675700 0x560b7e691940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcee7c22740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e676020 0x560b7e691e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcee7c22740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcee01ad8c0 con 0x560b7e676020 2024-06-04T05:05:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcee7c22740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcee01ad1f0 con 0x560b7e671d30 2024-06-04T05:05:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcee7c22740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcee01aa450 con 0x560b7e675700 2024-06-04T05:05:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcee7c22740 10 client.0 _ll_register_callbacks cb 0x7fced4004a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcede59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e671d30 0x560b7e691400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fceded9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e676020 0x560b7e691e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcede59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e671d30 0x560b7e691400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:53040/0 (socket says 172.21.15.70:53040) 2024-06-04T05:05:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcede59c640 1 -- 172.21.15.70:0/4213151311 learned_addr learned my addr 172.21.15.70:0/4213151311 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fceddd9b640 1 --2- 172.21.15.70:0/4213151311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e675700 0x560b7e691940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcede59c640 1 -- 172.21.15.70:0/4213151311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e675700 msgr2=0x560b7e691940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcede59c640 1 --2- 172.21.15.70:0/4213151311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e675700 0x560b7e691940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcede59c640 1 -- 172.21.15.70:0/4213151311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e676020 msgr2=0x560b7e691e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcede59c640 1 --2- 172.21.15.70:0/4213151311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e676020 0x560b7e691e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcede59c640 1 -- 172.21.15.70:0/4213151311 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b7e671b60 con 0x560b7e671d30 2024-06-04T05:05:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fceddd9b640 1 --2- 172.21.15.70:0/4213151311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e675700 0x560b7e691940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fceded9d640 1 --2- 172.21.15.70:0/4213151311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e676020 0x560b7e691e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcede59c640 1 --2- 172.21.15.70:0/4213151311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e671d30 0x560b7e691400 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fced0000c00 tx=0x7fced00044d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7e3eb936971af1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.696+0000 7fcec5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:05:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.697+0000 7fcec5ffb640 1 -- 172.21.15.70:0/4213151311 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced0022030 con 0x560b7e671d30 2024-06-04T05:05:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.697+0000 7fcee7c22740 1 -- 172.21.15.70:0/4213151311 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fced8036040 con 0x560b7e671d30 2024-06-04T05:05:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.697+0000 7fcec5ffb640 1 -- 172.21.15.70:0/4213151311 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fced0005020 con 0x560b7e671d30 2024-06-04T05:05:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.697+0000 7fcec5ffb640 1 -- 172.21.15.70:0/4213151311 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fced002e050 con 0x560b7e671d30 2024-06-04T05:05:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.697+0000 7fcee7c22740 10 client.26783 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.697+0000 7fcee7c22740 1 -- 172.21.15.70:0/4213151311 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560b7e7041d0 con 0x560b7e671d30 2024-06-04T05:05:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.697+0000 7fcec5ffb640 1 -- 172.21.15.70:0/4213151311 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fced8036040 con 0x560b7e671d30 2024-06-04T05:05:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.697+0000 7fcec5ffb640 1 -- 172.21.15.70:0/4213151311 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fced00047a0 con 0x560b7e671d30 2024-06-04T05:05:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.697+0000 7fcee7c22740 10 client.26783 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.697+0000 7fcee7c22740 10 client.26783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:05:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.697+0000 7fcee7c22740 20 client.26783 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.697+0000 7fcee7c22740 1 --2- 172.21.15.70:0/4213151311 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560b7e684e10 0x560b7e7068b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.697+0000 7fcee7c22740 4 client.26783 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:05:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.697+0000 7fcee7c22740 1 -- 172.21.15.70:0/4213151311 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560b7e5e3360 con 0x560b7e684e10 2024-06-04T05:05:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.697+0000 7fceddd9b640 1 --2- 172.21.15.70:0/4213151311 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560b7e684e10 0x560b7e7068b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.698+0000 7fcec5ffb640 10 client.26783 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.698+0000 7fceddd9b640 1 --2- 172.21.15.70:0/4213151311 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560b7e684e10 0x560b7e7068b0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37985a738853fc36 server_cookie=4baf87b0332d9d1b in_seq=0 out_seq=0 2024-06-04T05:05:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.721+0000 7fcec5ffb640 1 -- 172.21.15.70:0/4213151311 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x560b7e5e3360 con 0x560b7e684e10 2024-06-04T05:05:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.721+0000 7fcec5ffb640 10 client.26783 handle_command_reply: tid=0 2024-06-04T05:05:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.721+0000 7fcee7c22740 1 client.26783 shutdown 2024-06-04T05:05:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.721+0000 7fcee7c22740 1 -- 172.21.15.70:0/4213151311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e671d30 msgr2=0x560b7e691400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.721+0000 7fcee7c22740 1 --2- 172.21.15.70:0/4213151311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e671d30 0x560b7e691400 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fced0000c00 tx=0x7fced00044d0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.721+0000 7fcee7c22740 1 -- 172.21.15.70:0/4213151311 shutdown_connections 2024-06-04T05:05:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.721+0000 7fcee7c22740 1 --2- 172.21.15.70:0/4213151311 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560b7e684e10 0x560b7e7068b0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.721+0000 7fcee7c22740 1 --2- 172.21.15.70:0/4213151311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b7e676020 0x560b7e691e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.721+0000 7fcee7c22740 1 --2- 172.21.15.70:0/4213151311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560b7e675700 0x560b7e691940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.721+0000 7fcec5ffb640 0 client.26783 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.721+0000 7fcee7c22740 1 --2- 172.21.15.70:0/4213151311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b7e671d30 0x560b7e691400 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.721+0000 7fcee7c22740 1 -- 172.21.15.70:0/4213151311 >> 172.21.15.70:0/4213151311 conn(0x560b7e682470 msgr2=0x560b7e670fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.721+0000 7fcee7c22740 1 -- 172.21.15.70:0/4213151311 shutdown_connections 2024-06-04T05:05:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.721+0000 7fcee7c22740 1 -- 172.21.15.70:0/4213151311 wait complete. 2024-06-04T05:05:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.721+0000 7fcee7c22740 20 client.26783 trim_cache size 0 max 16384 2024-06-04T05:05:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:05:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:05:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:05:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.722+0000 7fcee4e2a640 1 -- 172.21.15.70:0/2048149215 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcec80b2a10 msgr2=0x7fcec80b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.722+0000 7fcee4e2a640 1 --2- 172.21.15.70:0/2048149215 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcec80b2a10 0x7fcec80b4ed0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fced8047000 tx=0x7fced804a000 comp rx=0 tx=0).stop 2024-06-04T05:05:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.722+0000 7fcee4e2a640 1 -- 172.21.15.70:0/2048149215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcee0102340 msgr2=0x7fcee019a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.722+0000 7fcee4e2a640 1 --2- 172.21.15.70:0/2048149215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcee0102340 0x7fcee019a410 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fced400e9f0 tx=0x7fced400edf0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.722+0000 7fcee4e2a640 1 -- 172.21.15.70:0/2048149215 shutdown_connections 2024-06-04T05:05:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.722+0000 7fcee4e2a640 1 --2- 172.21.15.70:0/2048149215 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcec80b2a10 0x7fcec80b4ed0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.722+0000 7fcee4e2a640 1 --2- 172.21.15.70:0/2048149215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee0107940 0x7fcee01a4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.722+0000 7fcee4e2a640 1 --2- 172.21.15.70:0/2048149215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcee0102340 0x7fcee019a410 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.722+0000 7fcee4e2a640 1 --2- 172.21.15.70:0/2048149215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcee0101980 0x7fcee0199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.722+0000 7fcee4e2a640 1 -- 172.21.15.70:0/2048149215 >> 172.21.15.70:0/2048149215 conn(0x7fcee0078ad0 msgr2=0x7fcee0104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.722+0000 7fcee4e2a640 1 -- 172.21.15.70:0/2048149215 shutdown_connections 2024-06-04T05:05:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:03.722+0000 7fcee4e2a640 1 -- 172.21.15.70:0/2048149215 wait complete. 2024-06-04T05:05:03.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 135985 2.07 MB/sec execute 236 sec latency 30.873 ms 2024-06-04T05:05:04.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 136512 2.07 MB/sec execute 237 sec latency 25.145 ms 2024-06-04T05:05:05.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:05 smithi112 ceph-mon[36453]: pgmap v242: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-06-04T05:05:05.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:05 smithi098 ceph-mon[35569]: pgmap v242: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-06-04T05:05:05.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:05 smithi070 ceph-mon[32631]: pgmap v242: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-06-04T05:05:05.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 137180 2.07 MB/sec execute 238 sec latency 18.340 ms 2024-06-04T05:05:06.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 137794 2.08 MB/sec execute 239 sec latency 11.871 ms 2024-06-04T05:05:07.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:07 smithi112 ceph-mon[36453]: pgmap v243: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 2.0 MiB/s wr, 204 op/s 2024-06-04T05:05:07.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:07 smithi098 ceph-mon[35569]: pgmap v243: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 2.0 MiB/s wr, 204 op/s 2024-06-04T05:05:07.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:07 smithi070 ceph-mon[32631]: pgmap v243: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 2.0 MiB/s wr, 204 op/s 2024-06-04T05:05:07.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 138355 2.09 MB/sec execute 240 sec latency 15.355 ms 2024-06-04T05:05:08.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 138736 2.09 MB/sec execute 241 sec latency 12.778 ms 2024-06-04T05:05:09.410 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:09 smithi070 ceph-mon[32631]: pgmap v244: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1.3 MiB/s wr, 141 op/s 2024-06-04T05:05:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:09 smithi112 ceph-mon[36453]: pgmap v244: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1.3 MiB/s wr, 141 op/s 2024-06-04T05:05:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:09 smithi098 ceph-mon[35569]: pgmap v244: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1.3 MiB/s wr, 141 op/s 2024-06-04T05:05:09.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 139046 2.09 MB/sec execute 242 sec latency 16.101 ms 2024-06-04T05:05:10.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 139543 2.08 MB/sec execute 243 sec latency 53.025 ms 2024-06-04T05:05:11.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:11 smithi112 ceph-mon[36453]: pgmap v245: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1.3 MiB/s wr, 141 op/s 2024-06-04T05:05:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:11 smithi098 ceph-mon[35569]: pgmap v245: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1.3 MiB/s wr, 141 op/s 2024-06-04T05:05:11.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:11 smithi070 ceph-mon[32631]: pgmap v245: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1.3 MiB/s wr, 141 op/s 2024-06-04T05:05:11.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 140074 2.09 MB/sec execute 244 sec latency 19.060 ms 2024-06-04T05:05:12.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 140730 2.09 MB/sec execute 245 sec latency 20.702 ms 2024-06-04T05:05:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:13 smithi112 ceph-mon[36453]: pgmap v246: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 2.1 MiB/s wr, 208 op/s 2024-06-04T05:05:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:13 smithi098 ceph-mon[35569]: pgmap v246: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 2.1 MiB/s wr, 208 op/s 2024-06-04T05:05:13.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:13 smithi070 ceph-mon[32631]: pgmap v246: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 2.1 MiB/s wr, 208 op/s 2024-06-04T05:05:13.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 141353 2.10 MB/sec execute 246 sec latency 15.527 ms 2024-06-04T05:05:14.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 141912 2.11 MB/sec execute 247 sec latency 15.690 ms 2024-06-04T05:05:15.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:15 smithi112 ceph-mon[36453]: pgmap v247: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-06-04T05:05:15.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:15 smithi098 ceph-mon[35569]: pgmap v247: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-06-04T05:05:15.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:15 smithi070 ceph-mon[32631]: pgmap v247: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-06-04T05:05:15.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 142286 2.11 MB/sec execute 248 sec latency 15.181 ms 2024-06-04T05:05:16.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 142537 2.10 MB/sec execute 249 sec latency 15.779 ms 2024-06-04T05:05:16.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.824+0000 7ff7d4159640 1 -- 172.21.15.70:0/3055787696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7cc10d020 msgr2=0x7ff7cc10f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:16.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.824+0000 7ff7d4159640 1 --2- 172.21.15.70:0/3055787696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7cc10d020 0x7ff7cc10f4c0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4009860 tx=0x7ff7c402a460 comp rx=0 tx=0).stop 2024-06-04T05:05:16.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.824+0000 7ff7d4159640 1 -- 172.21.15.70:0/3055787696 shutdown_connections 2024-06-04T05:05:16.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.824+0000 7ff7d4159640 1 --2- 172.21.15.70:0/3055787696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7cc10d020 0x7ff7cc10f4c0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.824+0000 7ff7d4159640 1 --2- 172.21.15.70:0/3055787696 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7cc104600 0x7ff7cc10cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.824+0000 7ff7d4159640 1 --2- 172.21.15.70:0/3055787696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc103ce0 0x7ff7cc1040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.824+0000 7ff7d4159640 1 -- 172.21.15.70:0/3055787696 >> 172.21.15.70:0/3055787696 conn(0x7ff7cc078ad0 msgr2=0x7ff7cc078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:16.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.824+0000 7ff7d4159640 1 -- 172.21.15.70:0/3055787696 shutdown_connections 2024-06-04T05:05:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.825+0000 7ff7d4159640 1 -- 172.21.15.70:0/3055787696 wait complete. 2024-06-04T05:05:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.825+0000 7ff7d4159640 1 Processor -- start 2024-06-04T05:05:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.825+0000 7ff7d4159640 1 -- start start 2024-06-04T05:05:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d4159640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc103ce0 0x7ff7cc101930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d4159640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7cc104600 0x7ff7cc101e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d4159640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7cc10d020 0x7ff7cc1023b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d4159640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc112bd0 con 0x7ff7cc104600 2024-06-04T05:05:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d4159640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc112a50 con 0x7ff7cc103ce0 2024-06-04T05:05:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d4159640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc112d50 con 0x7ff7cc10d020 2024-06-04T05:05:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d16f4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7cc104600 0x7ff7cc101e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d16f4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7cc104600 0x7ff7cc101e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53144/0 (socket says 172.21.15.70:53144) 2024-06-04T05:05:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d16f4640 1 -- 172.21.15.70:0/2805603631 learned_addr learned my addr 172.21.15.70:0/2805603631 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d1ef5640 1 --2- 172.21.15.70:0/2805603631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc103ce0 0x7ff7cc101930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d26f6640 1 --2- 172.21.15.70:0/2805603631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7cc10d020 0x7ff7cc1023b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d16f4640 1 -- 172.21.15.70:0/2805603631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7cc10d020 msgr2=0x7ff7cc1023b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d16f4640 1 --2- 172.21.15.70:0/2805603631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7cc10d020 0x7ff7cc1023b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d16f4640 1 -- 172.21.15.70:0/2805603631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc103ce0 msgr2=0x7ff7cc101930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d16f4640 1 --2- 172.21.15.70:0/2805603631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc103ce0 0x7ff7cc101930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d16f4640 1 -- 172.21.15.70:0/2805603631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7cc1a75b0 con 0x7ff7cc104600 2024-06-04T05:05:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d1ef5640 1 --2- 172.21.15.70:0/2805603631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc103ce0 0x7ff7cc101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.826+0000 7ff7d16f4640 1 --2- 172.21.15.70:0/2805603631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7cc104600 0x7ff7cc101e70 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000eb90 tx=0x7ff7c000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.827+0000 7ff7caffd640 1 -- 172.21.15.70:0/2805603631 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0004b20 con 0x7ff7cc104600 2024-06-04T05:05:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.827+0000 7ff7d4159640 1 -- 172.21.15.70:0/2805603631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7cc1a78a0 con 0x7ff7cc104600 2024-06-04T05:05:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.827+0000 7ff7caffd640 1 -- 172.21.15.70:0/2805603631 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff7c0004cc0 con 0x7ff7cc104600 2024-06-04T05:05:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.827+0000 7ff7caffd640 1 -- 172.21.15.70:0/2805603631 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0010690 con 0x7ff7cc104600 2024-06-04T05:05:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.827+0000 7ff7d4159640 1 -- 172.21.15.70:0/2805603631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7cc1a7df0 con 0x7ff7cc104600 2024-06-04T05:05:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.829+0000 7ff7caffd640 1 -- 172.21.15.70:0/2805603631 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff7c0005020 con 0x7ff7cc104600 2024-06-04T05:05:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.829+0000 7ff7d6e70740 1 Processor -- start 2024-06-04T05:05:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7d6e70740 1 -- start start 2024-06-04T05:05:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7d6e70740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108328820 0x561108328c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7d6e70740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108329140 0x5611083334c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7d6e70740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108333a00 0x561108335df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7d6e70740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56110829ac30 con 0x561108333a00 2024-06-04T05:05:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7d6e70740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56110828a7d0 con 0x561108328820 2024-06-04T05:05:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7d6e70740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56110828e810 con 0x561108329140 2024-06-04T05:05:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7d1ef5640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108329140 0x5611083334c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7d1ef5640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108329140 0x5611083334c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48154/0 (socket says 172.21.15.70:48154) 2024-06-04T05:05:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7d1ef5640 1 -- 172.21.15.70:0/2110121138 learned_addr learned my addr 172.21.15.70:0/2110121138 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7d16f4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108333a00 0x561108335df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7d26f6640 1 --2- 172.21.15.70:0/2110121138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108328820 0x561108328c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7d1ef5640 1 -- 172.21.15.70:0/2110121138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108328820 msgr2=0x561108328c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7d1ef5640 1 --2- 172.21.15.70:0/2110121138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108328820 0x561108328c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7d1ef5640 1 -- 172.21.15.70:0/2110121138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108333a00 msgr2=0x561108335df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7d1ef5640 1 --2- 172.21.15.70:0/2110121138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108333a00 0x561108335df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7caffd640 1 --2- 172.21.15.70:0/2805603631 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff7b40b2a10 0x7ff7b40b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.830+0000 7ff7d1ef5640 1 -- 172.21.15.70:0/2110121138 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561107f9f210 con 0x561108329140 2024-06-04T05:05:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.831+0000 7ff7d26f6640 1 --2- 172.21.15.70:0/2805603631 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff7b40b2a10 0x7ff7b40b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.831+0000 7ff7d1ef5640 1 --2- 172.21.15.70:0/2110121138 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108329140 0x5611083334c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc011510 tx=0x7ff7bc0119d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddbfc3753a77146a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:16.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.831+0000 7ff7caffd640 1 -- 172.21.15.70:0/2805603631 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7ff7c00e2c80 con 0x7ff7cc104600 2024-06-04T05:05:16.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.831+0000 7ff7b3fff640 1 -- 172.21.15.70:0/2110121138 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc004020 con 0x561108329140 2024-06-04T05:05:16.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.831+0000 7ff7b3fff640 1 -- 172.21.15.70:0/2110121138 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff7bc0041c0 con 0x561108329140 2024-06-04T05:05:16.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.831+0000 7ff7b3fff640 1 -- 172.21.15.70:0/2110121138 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc00f820 con 0x561108329140 2024-06-04T05:05:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.832+0000 7ff7d6e70740 1 -- 172.21.15.70:0/2110121138 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108329140 msgr2=0x5611083334c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.832+0000 7ff7d26f6640 1 --2- 172.21.15.70:0/2805603631 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff7b40b2a10 0x7ff7b40b4ed0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4005e00 tx=0x7ff7c4005d30 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.832+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/2110121138 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108329140 0x5611083334c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc011510 tx=0x7ff7bc0119d0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.832+0000 7ff7d6e70740 1 -- 172.21.15.70:0/2110121138 shutdown_connections 2024-06-04T05:05:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.832+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/2110121138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108333a00 0x561108335df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.832+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/2110121138 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108329140 0x5611083334c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.832+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/2110121138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108328820 0x561108328c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.832+0000 7ff7d6e70740 1 -- 172.21.15.70:0/2110121138 >> 172.21.15.70:0/2110121138 conn(0x561108324d30 msgr2=0x561108325110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.832+0000 7ff7d6e70740 1 -- 172.21.15.70:0/2110121138 shutdown_connections 2024-06-04T05:05:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.832+0000 7ff7d6e70740 1 -- 172.21.15.70:0/2110121138 wait complete. 2024-06-04T05:05:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.832+0000 7ff7d6e70740 1 -- start start 2024-06-04T05:05:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.832+0000 7ff7d6e70740 1 Processor -- start 2024-06-04T05:05:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d6e70740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108328820 0x561108330960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d6e70740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108329140 0x561108330ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d6e70740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108333a00 0x5611083313e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d6e70740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561107f70790 con 0x561108328820 2024-06-04T05:05:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d6e70740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561108246890 con 0x561108333a00 2024-06-04T05:05:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d6e70740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561108336630 con 0x561108329140 2024-06-04T05:05:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d6e70740 10 client.0 _ll_register_callbacks cb 0x561107f9ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d1ef5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108328820 0x561108330960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d16f4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108329140 0x561108330ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d16f4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108329140 0x561108330ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48160/0 (socket says 172.21.15.70:48160) 2024-06-04T05:05:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d1ef5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108328820 0x561108330960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53168/0 (socket says 172.21.15.70:53168) 2024-06-04T05:05:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d16f4640 1 -- 172.21.15.70:0/2324058785 learned_addr learned my addr 172.21.15.70:0/2324058785 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d26f6640 1 --2- 172.21.15.70:0/2324058785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108333a00 0x5611083313e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d16f4640 1 -- 172.21.15.70:0/2324058785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108333a00 msgr2=0x5611083313e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d16f4640 1 --2- 172.21.15.70:0/2324058785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108333a00 0x5611083313e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d16f4640 1 -- 172.21.15.70:0/2324058785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108328820 msgr2=0x561108330960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d16f4640 1 --2- 172.21.15.70:0/2324058785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108328820 0x561108330960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d16f4640 1 -- 172.21.15.70:0/2324058785 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56110806b620 con 0x561108329140 2024-06-04T05:05:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d1ef5640 1 --2- 172.21.15.70:0/2324058785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108328820 0x561108330960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d16f4640 1 --2- 172.21.15.70:0/2324058785 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108329140 0x561108330ea0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00a71a0 tx=0x7ff7c00e08d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27c856b504485b6d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:05:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7b17fa640 1 -- 172.21.15.70:0/2324058785 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c00a9880 con 0x561108329140 2024-06-04T05:05:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7b17fa640 1 -- 172.21.15.70:0/2324058785 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7ff7c00a61c0 con 0x561108329140 2024-06-04T05:05:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d6e70740 1 -- 172.21.15.70:0/2324058785 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7bc0041c0 con 0x561108329140 2024-06-04T05:05:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7b17fa640 1 -- 172.21.15.70:0/2324058785 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff7c006e050 con 0x561108329140 2024-06-04T05:05:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d6e70740 10 client.26875 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.833+0000 7ff7d6e70740 1 -- 172.21.15.70:0/2324058785 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5611083423a0 con 0x561108329140 2024-06-04T05:05:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.834+0000 7ff7b17fa640 1 -- 172.21.15.70:0/2324058785 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff7bc0041c0 con 0x561108329140 2024-06-04T05:05:16.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.834+0000 7ff7b17fa640 1 -- 172.21.15.70:0/2324058785 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ff7c00aa190 con 0x561108329140 2024-06-04T05:05:16.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.834+0000 7ff7d6e70740 10 client.26875 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:16.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.834+0000 7ff7d6e70740 10 client.26875 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:05:16.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.834+0000 7ff7d6e70740 20 client.26875 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:16.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.834+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/2324058785 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561108343860 0x561108345c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.834+0000 7ff7d6e70740 4 client.26875 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:05:16.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.834+0000 7ff7d6e70740 1 -- 172.21.15.70:0/2324058785 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56110829cb80 con 0x561108343860 2024-06-04T05:05:16.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.834+0000 7ff7d1ef5640 1 --2- 172.21.15.70:0/2324058785 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561108343860 0x561108345c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.834+0000 7ff7b17fa640 10 client.26875 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:16.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.834+0000 7ff7d1ef5640 1 --2- 172.21.15.70:0/2324058785 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561108343860 0x561108345c50 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5b3db740f4c931a server_cookie=9063decb57e2e8fa in_seq=0 out_seq=0 2024-06-04T05:05:16.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.835+0000 7ff7b17fa640 1 -- 172.21.15.70:0/2324058785 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56110829cb80 con 0x561108343860 2024-06-04T05:05:16.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.835+0000 7ff7b17fa640 10 client.26875 handle_command_reply: tid=0 2024-06-04T05:05:16.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.835+0000 7ff7d6e70740 1 client.26875 shutdown 2024-06-04T05:05:16.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.835+0000 7ff7d6e70740 1 -- 172.21.15.70:0/2324058785 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108329140 msgr2=0x561108330ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:16.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.835+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/2324058785 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108329140 0x561108330ea0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00a71a0 tx=0x7ff7c00e08d0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.835+0000 7ff7d6e70740 1 -- 172.21.15.70:0/2324058785 shutdown_connections 2024-06-04T05:05:16.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.835+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/2324058785 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561108343860 0x561108345c50 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.835+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/2324058785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108333a00 0x5611083313e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.835+0000 7ff7b17fa640 0 client.26875 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:16.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.835+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/2324058785 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108329140 0x561108330ea0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.835+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/2324058785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108328820 0x561108330960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.835+0000 7ff7d6e70740 1 -- 172.21.15.70:0/2324058785 >> 172.21.15.70:0/2324058785 conn(0x561108324d30 msgr2=0x561108323fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:16.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.836+0000 7ff7d6e70740 1 -- 172.21.15.70:0/2324058785 shutdown_connections 2024-06-04T05:05:16.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.836+0000 7ff7d6e70740 1 -- 172.21.15.70:0/2324058785 wait complete. 2024-06-04T05:05:16.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.836+0000 7ff7d6e70740 20 client.26875 trim_cache size 0 max 16384 2024-06-04T05:05:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.855+0000 7ff7d6e70740 1 Processor -- start 2024-06-04T05:05:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.855+0000 7ff7d6e70740 1 -- start start 2024-06-04T05:05:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.856+0000 7ff7d6e70740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108328820 0x56110834b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.856+0000 7ff7d6e70740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108329140 0x56110834ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.856+0000 7ff7d6e70740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108333a00 0x561108330920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.856+0000 7ff7d6e70740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc1b08c0 con 0x561108333a00 2024-06-04T05:05:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.856+0000 7ff7d6e70740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc1b2400 con 0x561108329140 2024-06-04T05:05:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.856+0000 7ff7d6e70740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc1b0f90 con 0x561108328820 2024-06-04T05:05:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.856+0000 7ff7d26f6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108333a00 0x561108330920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.856+0000 7ff7d26f6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108333a00 0x561108330920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53180/0 (socket says 172.21.15.70:53180) 2024-06-04T05:05:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.856+0000 7ff7d26f6640 1 -- 172.21.15.70:0/1660663044 learned_addr learned my addr 172.21.15.70:0/1660663044 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.856+0000 7ff7d16f4640 1 --2- 172.21.15.70:0/1660663044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108329140 0x56110834ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.856+0000 7ff7d1ef5640 1 --2- 172.21.15.70:0/1660663044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108328820 0x56110834b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.856+0000 7ff7d26f6640 1 -- 172.21.15.70:0/1660663044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108328820 msgr2=0x56110834b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.856+0000 7ff7d26f6640 1 --2- 172.21.15.70:0/1660663044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108328820 0x56110834b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.856+0000 7ff7d26f6640 1 -- 172.21.15.70:0/1660663044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108329140 msgr2=0x56110834ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.856+0000 7ff7d26f6640 1 --2- 172.21.15.70:0/1660663044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108329140 0x56110834ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.856+0000 7ff7d26f6640 1 -- 172.21.15.70:0/1660663044 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561108345fc0 con 0x561108333a00 2024-06-04T05:05:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.856+0000 7ff7d26f6640 1 --2- 172.21.15.70:0/1660663044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108333a00 0x561108330920 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4034000 tx=0x7ff7c4040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21f6ef8b2e294f1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.857+0000 7ff7b37fe640 1 -- 172.21.15.70:0/1660663044 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4004310 con 0x561108333a00 2024-06-04T05:05:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.857+0000 7ff7b37fe640 1 -- 172.21.15.70:0/1660663044 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff7c40044b0 con 0x561108333a00 2024-06-04T05:05:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.857+0000 7ff7b37fe640 1 -- 172.21.15.70:0/1660663044 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4003cd0 con 0x561108333a00 2024-06-04T05:05:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.857+0000 7ff7d6e70740 1 -- 172.21.15.70:0/1660663044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108333a00 msgr2=0x561108330920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.857+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/1660663044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108333a00 0x561108330920 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4034000 tx=0x7ff7c4040000 comp rx=0 tx=0).stop 2024-06-04T05:05:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.857+0000 7ff7d6e70740 1 -- 172.21.15.70:0/1660663044 shutdown_connections 2024-06-04T05:05:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.857+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/1660663044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108329140 0x56110834ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.857+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/1660663044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108333a00 0x561108330920 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.857+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/1660663044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108328820 0x56110834b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.857+0000 7ff7d6e70740 1 -- 172.21.15.70:0/1660663044 >> 172.21.15.70:0/1660663044 conn(0x561108324d30 msgr2=0x56110831fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.858+0000 7ff7d6e70740 1 -- 172.21.15.70:0/1660663044 shutdown_connections 2024-06-04T05:05:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.858+0000 7ff7d6e70740 1 -- 172.21.15.70:0/1660663044 wait complete. 2024-06-04T05:05:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.858+0000 7ff7d6e70740 1 -- start start 2024-06-04T05:05:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.858+0000 7ff7d6e70740 1 Processor -- start 2024-06-04T05:05:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.858+0000 7ff7d6e70740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108328820 0x56110833bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.858+0000 7ff7d6e70740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108329140 0x561108337280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.858+0000 7ff7d6e70740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108333a00 0x5611083377c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.858+0000 7ff7d6e70740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc1b1d30 con 0x561108329140 2024-06-04T05:05:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.858+0000 7ff7d6e70740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc1b1660 con 0x561108328820 2024-06-04T05:05:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.858+0000 7ff7d6e70740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc1ae8c0 con 0x561108333a00 2024-06-04T05:05:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.858+0000 7ff7d6e70740 10 client.0 _ll_register_callbacks cb 0x7ff7c0005ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.858+0000 7ff7d16f4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108329140 0x561108337280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.858+0000 7ff7d16f4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108329140 0x561108337280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53194/0 (socket says 172.21.15.70:53194) 2024-06-04T05:05:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7d16f4640 1 -- 172.21.15.70:0/1901323418 learned_addr learned my addr 172.21.15.70:0/1901323418 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7d1ef5640 1 --2- 172.21.15.70:0/1901323418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108328820 0x56110833bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7d26f6640 1 --2- 172.21.15.70:0/1901323418 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108333a00 0x5611083377c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7d16f4640 1 -- 172.21.15.70:0/1901323418 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108333a00 msgr2=0x5611083377c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7d16f4640 1 --2- 172.21.15.70:0/1901323418 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108333a00 0x5611083377c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7d16f4640 1 -- 172.21.15.70:0/1901323418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108328820 msgr2=0x56110833bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7d16f4640 1 --2- 172.21.15.70:0/1901323418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108328820 0x56110833bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7d16f4640 1 -- 172.21.15.70:0/1901323418 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561108337ff0 con 0x561108329140 2024-06-04T05:05:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7d1ef5640 1 --2- 172.21.15.70:0/1901323418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108328820 0x56110833bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7d16f4640 1 --2- 172.21.15.70:0/1901323418 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108329140 0x561108337280 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00e09a0 tx=0x7ff7c00b2c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab554318a7fdab84 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:05:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7b17fa640 1 -- 172.21.15.70:0/1901323418 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c00a96a0 con 0x561108329140 2024-06-04T05:05:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7b17fa640 1 -- 172.21.15.70:0/1901323418 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7ff7c00a6520 con 0x561108329140 2024-06-04T05:05:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7d6e70740 1 -- 172.21.15.70:0/1901323418 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7c40044b0 con 0x561108329140 2024-06-04T05:05:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7b17fa640 1 -- 172.21.15.70:0/1901323418 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff7c00a9880 con 0x561108329140 2024-06-04T05:05:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7d6e70740 10 client.17592 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7d6e70740 1 -- 172.21.15.70:0/1901323418 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5611083ae420 con 0x561108329140 2024-06-04T05:05:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.859+0000 7ff7b17fa640 1 -- 172.21.15.70:0/1901323418 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff7c40044b0 con 0x561108329140 2024-06-04T05:05:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.860+0000 7ff7b17fa640 1 -- 172.21.15.70:0/1901323418 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ff7c00e0af0 con 0x561108329140 2024-06-04T05:05:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.860+0000 7ff7d6e70740 10 client.17592 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.860+0000 7ff7d6e70740 10 client.17592 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:05:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.860+0000 7ff7d6e70740 20 client.17592 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.860+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/1901323418 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5611083aea40 0x5611083b0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.860+0000 7ff7d6e70740 4 client.17592 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:05:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.860+0000 7ff7d6e70740 1 -- 172.21.15.70:0/1901323418 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561108296360 con 0x5611083aea40 2024-06-04T05:05:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.860+0000 7ff7d1ef5640 1 --2- 172.21.15.70:0/1901323418 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5611083aea40 0x5611083b0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.860+0000 7ff7b17fa640 10 client.17592 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.860+0000 7ff7d1ef5640 1 --2- 172.21.15.70:0/1901323418 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5611083aea40 0x5611083b0e60 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7187ccb14f03c9c server_cookie=516e0a53d9195268 in_seq=0 out_seq=0 2024-06-04T05:05:16.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.878+0000 7ff7b17fa640 1 -- 172.21.15.70:0/1901323418 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x561108296360 con 0x5611083aea40 2024-06-04T05:05:16.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.878+0000 7ff7b17fa640 10 client.17592 handle_command_reply: tid=0 2024-06-04T05:05:16.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.879+0000 7ff7d6e70740 1 client.17592 shutdown 2024-06-04T05:05:16.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.879+0000 7ff7d6e70740 1 -- 172.21.15.70:0/1901323418 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108329140 msgr2=0x561108337280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:16.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.879+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/1901323418 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108329140 0x561108337280 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00e09a0 tx=0x7ff7c00b2c80 comp rx=0 tx=0).stop 2024-06-04T05:05:16.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.879+0000 7ff7d6e70740 1 -- 172.21.15.70:0/1901323418 shutdown_connections 2024-06-04T05:05:16.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.879+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/1901323418 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5611083aea40 0x5611083b0e60 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.879+0000 7ff7b17fa640 0 client.17592 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:16.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.879+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/1901323418 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561108333a00 0x5611083377c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.879+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/1901323418 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561108329140 0x561108337280 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.879+0000 7ff7d6e70740 1 --2- 172.21.15.70:0/1901323418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561108328820 0x56110833bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.879+0000 7ff7d6e70740 1 -- 172.21.15.70:0/1901323418 >> 172.21.15.70:0/1901323418 conn(0x561108324d30 msgr2=0x56110832a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:16.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.879+0000 7ff7d6e70740 1 -- 172.21.15.70:0/1901323418 shutdown_connections 2024-06-04T05:05:16.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.879+0000 7ff7d6e70740 1 -- 172.21.15.70:0/1901323418 wait complete. 2024-06-04T05:05:16.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.879+0000 7ff7d6e70740 20 client.17592 trim_cache size 0 max 16384 2024-06-04T05:05:16.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:05:16.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:05:16.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:05:16.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.879+0000 7ff7d4159640 1 -- 172.21.15.70:0/2805603631 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff7b40b2a10 msgr2=0x7ff7b40b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:16.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.879+0000 7ff7d4159640 1 --2- 172.21.15.70:0/2805603631 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff7b40b2a10 0x7ff7b40b4ed0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4005e00 tx=0x7ff7c4005d30 comp rx=0 tx=0).stop 2024-06-04T05:05:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.879+0000 7ff7d4159640 1 -- 172.21.15.70:0/2805603631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7cc104600 msgr2=0x7ff7cc101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.879+0000 7ff7d4159640 1 --2- 172.21.15.70:0/2805603631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7cc104600 0x7ff7cc101e70 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000eb90 tx=0x7ff7c000ef90 comp rx=0 tx=0).stop 2024-06-04T05:05:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.880+0000 7ff7d4159640 1 -- 172.21.15.70:0/2805603631 shutdown_connections 2024-06-04T05:05:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.880+0000 7ff7d4159640 1 --2- 172.21.15.70:0/2805603631 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff7b40b2a10 0x7ff7b40b4ed0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.880+0000 7ff7d4159640 1 --2- 172.21.15.70:0/2805603631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7cc10d020 0x7ff7cc1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.880+0000 7ff7d4159640 1 --2- 172.21.15.70:0/2805603631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7cc104600 0x7ff7cc101e70 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.880+0000 7ff7d4159640 1 --2- 172.21.15.70:0/2805603631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc103ce0 0x7ff7cc101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.880+0000 7ff7d4159640 1 -- 172.21.15.70:0/2805603631 >> 172.21.15.70:0/2805603631 conn(0x7ff7cc078ad0 msgr2=0x7ff7cc0fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.880+0000 7ff7d4159640 1 -- 172.21.15.70:0/2805603631 shutdown_connections 2024-06-04T05:05:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:16.880+0000 7ff7d4159640 1 -- 172.21.15.70:0/2805603631 wait complete. 2024-06-04T05:05:17.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:17 smithi112 ceph-mon[36453]: pgmap v248: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.1 MiB/s wr, 195 op/s 2024-06-04T05:05:17.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:17 smithi098 ceph-mon[35569]: pgmap v248: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.1 MiB/s wr, 195 op/s 2024-06-04T05:05:17.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:17 smithi070 ceph-mon[32631]: pgmap v248: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.1 MiB/s wr, 195 op/s 2024-06-04T05:05:17.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 143086 2.10 MB/sec execute 250 sec latency 16.092 ms 2024-06-04T05:05:18.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 143615 2.11 MB/sec execute 251 sec latency 20.062 ms 2024-06-04T05:05:19.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:19 smithi112 ceph-mon[36453]: pgmap v249: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-06-04T05:05:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:19 smithi098 ceph-mon[35569]: pgmap v249: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-06-04T05:05:19.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:19 smithi070 ceph-mon[32631]: pgmap v249: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-06-04T05:05:19.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 144274 2.11 MB/sec execute 252 sec latency 18.500 ms 2024-06-04T05:05:20.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 144896 2.12 MB/sec execute 253 sec latency 15.374 ms 2024-06-04T05:05:21.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:21 smithi112 ceph-mon[36453]: pgmap v250: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-06-04T05:05:21.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:21 smithi098 ceph-mon[35569]: pgmap v250: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-06-04T05:05:21.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:21 smithi070 ceph-mon[32631]: pgmap v250: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-06-04T05:05:21.726 INFO:tasks.workunit.client.0.smithi070.stdout: 1 145419 2.13 MB/sec execute 254 sec latency 15.660 ms 2024-06-04T05:05:22.263 INFO:tasks.quiescer.fs.[cephfs]:Total duration (47.2) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-06-04T05:05:22.264 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.6 2024-06-04T05:05:22.264 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:05:22.531 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:05:22.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 145782 2.13 MB/sec execute 255 sec latency 15.271 ms 2024-06-04T05:05:23.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:23 smithi070 ceph-mon[32631]: pgmap v251: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 2.2 MiB/s wr, 195 op/s 2024-06-04T05:05:23.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:23 smithi112 ceph-mon[36453]: pgmap v251: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 2.2 MiB/s wr, 195 op/s 2024-06-04T05:05:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:23 smithi098 ceph-mon[35569]: pgmap v251: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 2.2 MiB/s wr, 195 op/s 2024-06-04T05:05:23.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 146064 2.12 MB/sec execute 256 sec latency 16.728 ms 2024-06-04T05:05:24.224 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:05:24.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.397+0000 7f5e8e5f9700 1 -- 172.21.15.70:0/2798786891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e88101e60 msgr2=0x7f5e88102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:24.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.397+0000 7f5e8e5f9700 1 --2- 172.21.15.70:0/2798786891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e88101e60 0x7f5e88102240 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f5e7800aad0 tx=0x7f5e7800ade0 comp rx=0 tx=0).stop 2024-06-04T05:05:24.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.397+0000 7f5e8e5f9700 1 -- 172.21.15.70:0/2798786891 shutdown_connections 2024-06-04T05:05:24.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.397+0000 7f5e8e5f9700 1 --2- 172.21.15.70:0/2798786891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e88106ef0 0x7f5e8810eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:24.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.397+0000 7f5e8e5f9700 1 --2- 172.21.15.70:0/2798786891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e88101e60 0x7f5e88102240 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:24.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.397+0000 7f5e8e5f9700 1 --2- 172.21.15.70:0/2798786891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e88102810 0x7f5e88106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:24.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.397+0000 7f5e8e5f9700 1 -- 172.21.15.70:0/2798786891 >> 172.21.15.70:0/2798786891 conn(0x7f5e88078ab0 msgr2=0x7f5e880ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:24.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.397+0000 7f5e8e5f9700 1 -- 172.21.15.70:0/2798786891 shutdown_connections 2024-06-04T05:05:24.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.397+0000 7f5e8e5f9700 1 -- 172.21.15.70:0/2798786891 wait complete. 2024-06-04T05:05:24.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.399+0000 7f5e8e5f9700 1 Processor -- start 2024-06-04T05:05:24.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.399+0000 7f5e8e5f9700 1 -- start start 2024-06-04T05:05:24.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.399+0000 7f5e8e5f9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e88101e60 0x7f5e88111c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:24.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.399+0000 7f5e8e5f9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e88102810 0x7f5e88112190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:24.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.399+0000 7f5e8e5f9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e88106ef0 0x7f5e881126d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.399+0000 7f5e8e5f9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5e88111400 con 0x7f5e88102810 2024-06-04T05:05:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.399+0000 7f5e8e5f9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e88111280 con 0x7f5e88106ef0 2024-06-04T05:05:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.399+0000 7f5e8e5f9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5e88111580 con 0x7f5e88101e60 2024-06-04T05:05:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.399+0000 7f5e8cb96700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e88106ef0 0x7f5e881126d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.400+0000 7f5e8cb96700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e88106ef0 0x7f5e881126d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:41140/0 (socket says 172.21.15.70:41140) 2024-06-04T05:05:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.400+0000 7f5e8cb96700 1 -- 172.21.15.70:0/419211376 learned_addr learned my addr 172.21.15.70:0/419211376 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.400+0000 7f5e877fe700 1 --2- 172.21.15.70:0/419211376 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e88102810 0x7f5e88112190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.400+0000 7f5e87fff700 1 --2- 172.21.15.70:0/419211376 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e88101e60 0x7f5e88111c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.400+0000 7f5e8cb96700 1 -- 172.21.15.70:0/419211376 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e88101e60 msgr2=0x7f5e88111c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.400+0000 7f5e8cb96700 1 --2- 172.21.15.70:0/419211376 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e88101e60 0x7f5e88111c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.400+0000 7f5e8cb96700 1 -- 172.21.15.70:0/419211376 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e88102810 msgr2=0x7f5e88112190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.400+0000 7f5e8cb96700 1 --2- 172.21.15.70:0/419211376 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e88102810 0x7f5e88112190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.400+0000 7f5e8cb96700 1 -- 172.21.15.70:0/419211376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e881a8060 con 0x7f5e88106ef0 2024-06-04T05:05:24.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.400+0000 7f5e8cb96700 1 --2- 172.21.15.70:0/419211376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e88106ef0 0x7f5e881126d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000ef30 tx=0x7f5e8000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:24.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.400+0000 7f5e857fa700 1 -- 172.21.15.70:0/419211376 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e800092e0 con 0x7f5e88106ef0 2024-06-04T05:05:24.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.400+0000 7f5e8e5f9700 1 -- 172.21.15.70:0/419211376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e881a82f0 con 0x7f5e88106ef0 2024-06-04T05:05:24.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.400+0000 7f5e857fa700 1 -- 172.21.15.70:0/419211376 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5e80004bb0 con 0x7f5e88106ef0 2024-06-04T05:05:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.401+0000 7f5e857fa700 1 -- 172.21.15.70:0/419211376 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e80004020 con 0x7f5e88106ef0 2024-06-04T05:05:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.401+0000 7f5e8e5f9700 1 -- 172.21.15.70:0/419211376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e881a8830 con 0x7f5e88106ef0 2024-06-04T05:05:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.402+0000 7f5e8e5f9700 1 -- 172.21.15.70:0/419211376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e881032f0 con 0x7f5e88106ef0 2024-06-04T05:05:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.403+0000 7f5e857fa700 1 -- 172.21.15.70:0/419211376 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f5e8001b030 con 0x7f5e88106ef0 2024-06-04T05:05:24.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.404+0000 7f5e857fa700 1 --2- 172.21.15.70:0/419211376 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5e700b2a70 0x7f5e700b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:24.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.404+0000 7f5e857fa700 1 -- 172.21.15.70:0/419211376 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f5e800e2130 con 0x7f5e88106ef0 2024-06-04T05:05:24.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.404+0000 7f5e87fff700 1 --2- 172.21.15.70:0/419211376 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5e700b2a70 0x7f5e700b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:24.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.405+0000 7f5e87fff700 1 --2- 172.21.15.70:0/419211376 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5e700b2a70 0x7f5e700b4f30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5e78005cd0 tx=0x7f5e78005c20 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:24.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.409+0000 7f5e857fa700 1 -- 172.21.15.70:0/419211376 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5e80022020 con 0x7f5e88106ef0 2024-06-04T05:05:24.522 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:05:24.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 146515 2.12 MB/sec execute 257 sec latency 66.727 ms 2024-06-04T05:05:24.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.734+0000 7f5e8e5f9700 1 -- 172.21.15.70:0/419211376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e8802e720 con 0x7f5e88106ef0 2024-06-04T05:05:24.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.734+0000 7f5e857fa700 1 -- 172.21.15.70:0/419211376 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11929 (secure 0 0 0) 0x7f5e80010040 con 0x7f5e88106ef0 2024-06-04T05:05:24.737 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:05:24.737 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:01:52.807483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:05:24.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.737+0000 7f5e8e5f9700 1 -- 172.21.15.70:0/419211376 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5e700b2a70 msgr2=0x7f5e700b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:24.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.737+0000 7f5e8e5f9700 1 --2- 172.21.15.70:0/419211376 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5e700b2a70 0x7f5e700b4f30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5e78005cd0 tx=0x7f5e78005c20 comp rx=0 tx=0).stop 2024-06-04T05:05:24.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.737+0000 7f5e8e5f9700 1 -- 172.21.15.70:0/419211376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e88106ef0 msgr2=0x7f5e881126d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:24.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.737+0000 7f5e8e5f9700 1 --2- 172.21.15.70:0/419211376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e88106ef0 0x7f5e881126d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000ef30 tx=0x7f5e8000c3a0 comp rx=0 tx=0).stop 2024-06-04T05:05:24.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.737+0000 7f5e8e5f9700 1 -- 172.21.15.70:0/419211376 shutdown_connections 2024-06-04T05:05:24.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.737+0000 7f5e8e5f9700 1 --2- 172.21.15.70:0/419211376 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5e700b2a70 0x7f5e700b4f30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:24.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.737+0000 7f5e8e5f9700 1 --2- 172.21.15.70:0/419211376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e88106ef0 0x7f5e881126d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:24.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.737+0000 7f5e8e5f9700 1 --2- 172.21.15.70:0/419211376 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e88101e60 0x7f5e88111c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:24.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.737+0000 7f5e8e5f9700 1 --2- 172.21.15.70:0/419211376 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e88102810 0x7f5e88112190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:24.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.737+0000 7f5e8e5f9700 1 -- 172.21.15.70:0/419211376 >> 172.21.15.70:0/419211376 conn(0x7f5e88078ab0 msgr2=0x7f5e880ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:24.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.738+0000 7f5e8e5f9700 1 -- 172.21.15.70:0/419211376 shutdown_connections 2024-06-04T05:05:24.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:24.738+0000 7f5e8e5f9700 1 -- 172.21.15.70:0/419211376 wait complete. 2024-06-04T05:05:24.741 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 32 2024-06-04T05:05:25.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:25 smithi112 ceph-mon[36453]: pgmap v252: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-06-04T05:05:25.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:25 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/419211376' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:05:25.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:25 smithi098 ceph-mon[35569]: pgmap v252: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-06-04T05:05:25.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:25 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/419211376' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:05:25.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:25 smithi070 ceph-mon[32631]: pgmap v252: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-06-04T05:05:25.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:25 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/419211376' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:05:25.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 147027 2.12 MB/sec execute 258 sec latency 17.544 ms 2024-06-04T05:05:25.930 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 143.6 --await-for 45.0' 2024-06-04T05:05:25.930 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 143.6 --await-for 45.0 2024-06-04T05:05:26.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 147640 2.12 MB/sec execute 259 sec latency 19.907 ms 2024-06-04T05:05:27.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.035+0000 7f71321c4700 1 -- 172.21.15.70:0/36141164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f712c108520 msgr2=0x7f712c108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:27.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.035+0000 7f71321c4700 1 --2- 172.21.15.70:0/36141164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f712c108520 0x7f712c108900 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f711c009990 tx=0x7f711c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.035+0000 7f71321c4700 1 -- 172.21.15.70:0/36141164 shutdown_connections 2024-06-04T05:05:27.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.035+0000 7f71321c4700 1 --2- 172.21.15.70:0/36141164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f712c102e00 0x7f712c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.035+0000 7f71321c4700 1 --2- 172.21.15.70:0/36141164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f712c108520 0x7f712c108900 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.035+0000 7f71321c4700 1 --2- 172.21.15.70:0/36141164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c102440 0x7f712c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.035+0000 7f71321c4700 1 -- 172.21.15.70:0/36141164 >> 172.21.15.70:0/36141164 conn(0x7f712c0793f0 msgr2=0x7f712c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:27.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.035+0000 7f71321c4700 1 -- 172.21.15.70:0/36141164 shutdown_connections 2024-06-04T05:05:27.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.035+0000 7f71321c4700 1 -- 172.21.15.70:0/36141164 wait complete. 2024-06-04T05:05:27.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.036+0000 7f71321c4700 1 Processor -- start 2024-06-04T05:05:27.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.036+0000 7f71321c4700 1 -- start start 2024-06-04T05:05:27.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.036+0000 7f71321c4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f712c102440 0x7f712c110a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.036+0000 7f71321c4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c102e00 0x7f712c110fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.036+0000 7f71321c4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f712c108520 0x7f712c111500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.037+0000 7f71321c4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f712c113590 con 0x7f712c102440 2024-06-04T05:05:27.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.037+0000 7f71321c4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f712c113410 con 0x7f712c102e00 2024-06-04T05:05:27.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.037+0000 7f71321c4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f712c113710 con 0x7f712c108520 2024-06-04T05:05:27.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.037+0000 7f712bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f712c108520 0x7f712c111500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.037+0000 7f712bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f712c108520 0x7f712c111500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36514/0 (socket says 172.21.15.70:36514) 2024-06-04T05:05:27.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.037+0000 7f712bfff700 1 -- 172.21.15.70:0/1942477813 learned_addr learned my addr 172.21.15.70:0/1942477813 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:27.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.037+0000 7f712b7fe700 1 --2- 172.21.15.70:0/1942477813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f712c102440 0x7f712c110a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.037+0000 7f712affd700 1 --2- 172.21.15.70:0/1942477813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c102e00 0x7f712c110fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.037+0000 7f712bfff700 1 -- 172.21.15.70:0/1942477813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c102e00 msgr2=0x7f712c110fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:27.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.037+0000 7f712bfff700 1 --2- 172.21.15.70:0/1942477813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c102e00 0x7f712c110fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.037+0000 7f712bfff700 1 -- 172.21.15.70:0/1942477813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f712c102440 msgr2=0x7f712c110a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:27.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.037+0000 7f712bfff700 1 --2- 172.21.15.70:0/1942477813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f712c102440 0x7f712c110a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.037+0000 7f712bfff700 1 -- 172.21.15.70:0/1942477813 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f712c1a8030 con 0x7f712c108520 2024-06-04T05:05:27.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.037+0000 7f712bfff700 1 --2- 172.21.15.70:0/1942477813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f712c108520 0x7f712c111500 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f712400ef30 tx=0x7f712400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:27.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.037+0000 7f7128ff9700 1 -- 172.21.15.70:0/1942477813 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71240092e0 con 0x7f712c108520 2024-06-04T05:05:27.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.038+0000 7f7128ff9700 1 -- 172.21.15.70:0/1942477813 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7124004bb0 con 0x7f712c108520 2024-06-04T05:05:27.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.038+0000 7f71321c4700 1 -- 172.21.15.70:0/1942477813 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f712c1a82c0 con 0x7f712c108520 2024-06-04T05:05:27.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.038+0000 7f7128ff9700 1 -- 172.21.15.70:0/1942477813 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7124004020 con 0x7f712c108520 2024-06-04T05:05:27.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.038+0000 7f71321c4700 1 -- 172.21.15.70:0/1942477813 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f712c1a8800 con 0x7f712c108520 2024-06-04T05:05:27.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.040+0000 7f7128ff9700 1 -- 172.21.15.70:0/1942477813 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f712401b030 con 0x7f712c108520 2024-06-04T05:05:27.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.041+0000 7f7128ff9700 1 --2- 172.21.15.70:0/1942477813 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f71140b29f0 0x7f71140b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.041+0000 7f712b7fe700 1 --2- 172.21.15.70:0/1942477813 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f71140b29f0 0x7f71140b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.041+0000 7f7128ff9700 1 -- 172.21.15.70:0/1942477813 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f71240e2070 con 0x7f712c108520 2024-06-04T05:05:27.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.042+0000 7f712b7fe700 1 --2- 172.21.15.70:0/1942477813 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f71140b29f0 0x7f71140b4eb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f711c00b2e0 tx=0x7f711c005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:27.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.043+0000 7f713ca59b80 1 Processor -- start 2024-06-04T05:05:27.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.043+0000 7f713ca59b80 1 -- start start 2024-06-04T05:05:27.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.043+0000 7f713ca59b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc20320 0x557ecfc20700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.043+0000 7f713ca59b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1a8a0 0x557ecfc1ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.043+0000 7f713ca59b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1b260 0x557ecfc27080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.043+0000 7f713ca59b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557ecfb78d70 con 0x557ecfc20320 2024-06-04T05:05:27.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.043+0000 7f712b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc20320 0x557ecfc20700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.043+0000 7f713ca59b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557ecfb68910 con 0x557ecfc1b260 2024-06-04T05:05:27.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.043+0000 7f713ca59b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557ecfb6c950 con 0x557ecfc1a8a0 2024-06-04T05:05:27.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.043+0000 7f712affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1a8a0 0x557ecfc1ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.043+0000 7f712b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc20320 0x557ecfc20700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37920/0 (socket says 172.21.15.70:37920) 2024-06-04T05:05:27.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.043+0000 7f712b7fe700 1 -- 172.21.15.70:0/3001540522 learned_addr learned my addr 172.21.15.70:0/3001540522 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:27.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.044+0000 7f712bfff700 1 --2- 172.21.15.70:0/3001540522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1b260 0x557ecfc27080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.044+0000 7f712affd700 1 -- 172.21.15.70:0/3001540522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1b260 msgr2=0x557ecfc27080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:27.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.044+0000 7f712affd700 1 --2- 172.21.15.70:0/3001540522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1b260 0x557ecfc27080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.044+0000 7f712affd700 1 -- 172.21.15.70:0/3001540522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc20320 msgr2=0x557ecfc20700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:27.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.044+0000 7f712affd700 1 --2- 172.21.15.70:0/3001540522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc20320 0x557ecfc20700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.044+0000 7f712affd700 1 -- 172.21.15.70:0/3001540522 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ecfb11fb0 con 0x557ecfc1a8a0 2024-06-04T05:05:27.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.044+0000 7f712affd700 1 --2- 172.21.15.70:0/3001540522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1a8a0 0x557ecfc1ad20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f712c112510 tx=0x7f7120009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b173270038410935 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:27.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.044+0000 7f7111ffb700 1 -- 172.21.15.70:0/3001540522 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712001e070 con 0x557ecfc1a8a0 2024-06-04T05:05:27.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.044+0000 7f7111ffb700 1 -- 172.21.15.70:0/3001540522 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7120011c80 con 0x557ecfc1a8a0 2024-06-04T05:05:27.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.045+0000 7f7111ffb700 1 -- 172.21.15.70:0/3001540522 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712000e610 con 0x557ecfc1a8a0 2024-06-04T05:05:27.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.045+0000 7f713ca59b80 1 -- 172.21.15.70:0/3001540522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1a8a0 msgr2=0x557ecfc1ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:27.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.045+0000 7f713ca59b80 1 --2- 172.21.15.70:0/3001540522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1a8a0 0x557ecfc1ad20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f712c112510 tx=0x7f7120009f60 comp rx=0 tx=0).stop 2024-06-04T05:05:27.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.046+0000 7f713ca59b80 1 -- 172.21.15.70:0/3001540522 shutdown_connections 2024-06-04T05:05:27.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.046+0000 7f713ca59b80 1 --2- 172.21.15.70:0/3001540522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1b260 0x557ecfc27080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.046+0000 7f713ca59b80 1 --2- 172.21.15.70:0/3001540522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc20320 0x557ecfc20700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.046+0000 7f713ca59b80 1 --2- 172.21.15.70:0/3001540522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1a8a0 0x557ecfc1ad20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.046+0000 7f713ca59b80 1 -- 172.21.15.70:0/3001540522 >> 172.21.15.70:0/3001540522 conn(0x557ecfb46240 msgr2=0x557ecfc153e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:27.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.046+0000 7f713ca59b80 1 -- 172.21.15.70:0/3001540522 shutdown_connections 2024-06-04T05:05:27.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.046+0000 7f713ca59b80 1 -- 172.21.15.70:0/3001540522 wait complete. 2024-06-04T05:05:27.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.046+0000 7f713ca59b80 1 -- start start 2024-06-04T05:05:27.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.047+0000 7f713ca59b80 1 Processor -- start 2024-06-04T05:05:27.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.048+0000 7f713ca59b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc1a8a0 0x557ecfc26c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.048+0000 7f713ca59b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1b260 0x557ecfc23f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.048+0000 7f713ca59b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc20320 0x557ecfc24460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.048+0000 7f712b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc1a8a0 0x557ecfc26c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.048+0000 7f712b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc1a8a0 0x557ecfc26c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37934/0 (socket says 172.21.15.70:37934) 2024-06-04T05:05:27.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.048+0000 7f712b7fe700 1 -- 172.21.15.70:0/4022619962 learned_addr learned my addr 172.21.15.70:0/4022619962 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:27.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.048+0000 7f713ca59b80 1 -- 172.21.15.70:0/4022619962 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557ecfa6efa0 con 0x557ecfc1a8a0 2024-06-04T05:05:27.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.048+0000 7f712affd700 1 --2- 172.21.15.70:0/4022619962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1b260 0x557ecfc23f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.048+0000 7f712bfff700 1 --2- 172.21.15.70:0/4022619962 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc20320 0x557ecfc24460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.048+0000 7f713ca59b80 1 -- 172.21.15.70:0/4022619962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557ecfad0da0 con 0x557ecfc1b260 2024-06-04T05:05:27.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.048+0000 7f713ca59b80 1 -- 172.21.15.70:0/4022619962 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557ecfc27780 con 0x557ecfc20320 2024-06-04T05:05:27.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.048+0000 7f713ca59b80 10 client.0 _ll_register_callbacks cb 0x557ecfb03950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:27.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.049+0000 7f712affd700 1 -- 172.21.15.70:0/4022619962 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc20320 msgr2=0x557ecfc24460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:27.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.049+0000 7f712affd700 1 --2- 172.21.15.70:0/4022619962 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc20320 0x557ecfc24460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.049+0000 7f712affd700 1 -- 172.21.15.70:0/4022619962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc1a8a0 msgr2=0x557ecfc26c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:27.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.049+0000 7f712affd700 1 --2- 172.21.15.70:0/4022619962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc1a8a0 0x557ecfc26c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.049+0000 7f712affd700 1 -- 172.21.15.70:0/4022619962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ecfab88d0 con 0x557ecfc1b260 2024-06-04T05:05:27.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.049+0000 7f712bfff700 1 --2- 172.21.15.70:0/4022619962 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc20320 0x557ecfc24460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:27.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.049+0000 7f712b7fe700 1 --2- 172.21.15.70:0/4022619962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc1a8a0 0x557ecfc26c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:27.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.049+0000 7f712affd700 1 --2- 172.21.15.70:0/4022619962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1b260 0x557ecfc23f20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7120000c00 tx=0x7f7120005500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=722294592fea2a0b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:27.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.049+0000 7f710affd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:05:27.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.049+0000 7f710affd700 1 -- 172.21.15.70:0/4022619962 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7120026070 con 0x557ecfc1b260 2024-06-04T05:05:27.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.049+0000 7f713ca59b80 1 -- 172.21.15.70:0/4022619962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7120011c80 con 0x557ecfc1b260 2024-06-04T05:05:27.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.049+0000 7f710affd700 1 -- 172.21.15.70:0/4022619962 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f7120004960 con 0x557ecfc1b260 2024-06-04T05:05:27.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.049+0000 7f710affd700 1 -- 172.21.15.70:0/4022619962 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7120030050 con 0x557ecfc1b260 2024-06-04T05:05:27.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.050+0000 7f713ca59b80 10 client.26819 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:27.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.050+0000 7f713ca59b80 1 -- 172.21.15.70:0/4022619962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557ecfb22760 con 0x557ecfc1b260 2024-06-04T05:05:27.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.050+0000 7f710affd700 1 -- 172.21.15.70:0/4022619962 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7120011c80 con 0x557ecfc1b260 2024-06-04T05:05:27.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.050+0000 7f710affd700 1 -- 172.21.15.70:0/4022619962 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f7120033020 con 0x557ecfc1b260 2024-06-04T05:05:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.051+0000 7f713ca59b80 10 client.26819 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.051+0000 7f713ca59b80 10 client.26819 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:05:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.051+0000 7f713ca59b80 20 client.26819 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.051+0000 7f713ca59b80 1 --2- 172.21.15.70:0/4022619962 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557ecfc343c0 0x557ecfc36860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.051+0000 7f713ca59b80 4 client.26819 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:05:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.051+0000 7f713ca59b80 1 -- 172.21.15.70:0/4022619962 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ecfb7acc0 con 0x557ecfc343c0 2024-06-04T05:05:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.051+0000 7f712b7fe700 1 --2- 172.21.15.70:0/4022619962 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557ecfc343c0 0x557ecfc36860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.051+0000 7f712b7fe700 1 --2- 172.21.15.70:0/4022619962 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557ecfc343c0 0x557ecfc36860 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8d362897889929c server_cookie=9639ebeb1079d891 in_seq=0 out_seq=0 2024-06-04T05:05:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.051+0000 7f710affd700 10 client.26819 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.052+0000 7f710affd700 1 -- 172.21.15.70:0/4022619962 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557ecfb7acc0 con 0x557ecfc343c0 2024-06-04T05:05:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.052+0000 7f710affd700 10 client.26819 handle_command_reply: tid=0 2024-06-04T05:05:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.052+0000 7f713ca59b80 1 client.26819 shutdown 2024-06-04T05:05:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.053+0000 7f713ca59b80 1 -- 172.21.15.70:0/4022619962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1b260 msgr2=0x557ecfc23f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.053+0000 7f713ca59b80 1 --2- 172.21.15.70:0/4022619962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1b260 0x557ecfc23f20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7120000c00 tx=0x7f7120005500 comp rx=0 tx=0).stop 2024-06-04T05:05:27.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.053+0000 7f713ca59b80 1 -- 172.21.15.70:0/4022619962 shutdown_connections 2024-06-04T05:05:27.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.053+0000 7f713ca59b80 1 --2- 172.21.15.70:0/4022619962 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc20320 0x557ecfc24460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.053+0000 7f713ca59b80 1 --2- 172.21.15.70:0/4022619962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc1a8a0 0x557ecfc26c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.053+0000 7f713ca59b80 1 --2- 172.21.15.70:0/4022619962 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557ecfc343c0 0x557ecfc36860 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.053+0000 7f713ca59b80 1 --2- 172.21.15.70:0/4022619962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1b260 0x557ecfc23f20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.053+0000 7f710affd700 0 client.26819 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:27.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.053+0000 7f713ca59b80 1 -- 172.21.15.70:0/4022619962 >> 172.21.15.70:0/4022619962 conn(0x557ecfb46240 msgr2=0x557ecfc14af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:27.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.053+0000 7f713ca59b80 1 -- 172.21.15.70:0/4022619962 shutdown_connections 2024-06-04T05:05:27.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.053+0000 7f713ca59b80 1 -- 172.21.15.70:0/4022619962 wait complete. 2024-06-04T05:05:27.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.054+0000 7f713ca59b80 20 client.26819 trim_cache size 0 max 16384 2024-06-04T05:05:27.075 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:05:27.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.082+0000 7f713ca59b80 1 Processor -- start 2024-06-04T05:05:27.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.083+0000 7f713ca59b80 1 -- start start 2024-06-04T05:05:27.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.083+0000 7f713ca59b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1a8a0 0x557ecfc1ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.083+0000 7f713ca59b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1b260 0x557ecfc15eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.083+0000 7f713ca59b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc20320 0x557ecfc15270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.083+0000 7f713ca59b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f712c1b00f0 con 0x557ecfc20320 2024-06-04T05:05:27.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.083+0000 7f713ca59b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f712c1b1c30 con 0x557ecfc1a8a0 2024-06-04T05:05:27.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.083+0000 7f713ca59b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f712c1b07c0 con 0x557ecfc1b260 2024-06-04T05:05:27.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.083+0000 7f712affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1b260 0x557ecfc15eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.083+0000 7f712affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1b260 0x557ecfc15eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36532/0 (socket says 172.21.15.70:36532) 2024-06-04T05:05:27.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.083+0000 7f712affd700 1 -- 172.21.15.70:0/3121035741 learned_addr learned my addr 172.21.15.70:0/3121035741 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:27.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.083+0000 7f712bfff700 1 --2- 172.21.15.70:0/3121035741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc20320 0x557ecfc15270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.083+0000 7f712b7fe700 1 --2- 172.21.15.70:0/3121035741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1a8a0 0x557ecfc1ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.083+0000 7f712affd700 1 -- 172.21.15.70:0/3121035741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1a8a0 msgr2=0x557ecfc1ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:27.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.083+0000 7f712affd700 1 --2- 172.21.15.70:0/3121035741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1a8a0 0x557ecfc1ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.083+0000 7f712affd700 1 -- 172.21.15.70:0/3121035741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc20320 msgr2=0x557ecfc15270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:27.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.083+0000 7f712affd700 1 --2- 172.21.15.70:0/3121035741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc20320 0x557ecfc15270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.083+0000 7f712affd700 1 -- 172.21.15.70:0/3121035741 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ecfc26760 con 0x557ecfc1b260 2024-06-04T05:05:27.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.084+0000 7f712affd700 1 --2- 172.21.15.70:0/3121035741 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1b260 0x557ecfc15eb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7120019dd0 tx=0x7f7120001e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b27c7f9ecd4e053c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:27.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.084+0000 7f71117fa700 1 -- 172.21.15.70:0/3121035741 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7120039070 con 0x557ecfc1b260 2024-06-04T05:05:27.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.084+0000 7f71117fa700 1 -- 172.21.15.70:0/3121035741 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7120022400 con 0x557ecfc1b260 2024-06-04T05:05:27.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.084+0000 7f71117fa700 1 -- 172.21.15.70:0/3121035741 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7120015030 con 0x557ecfc1b260 2024-06-04T05:05:27.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.085+0000 7f713ca59b80 1 -- 172.21.15.70:0/3121035741 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1b260 msgr2=0x557ecfc15eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:27.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.085+0000 7f713ca59b80 1 --2- 172.21.15.70:0/3121035741 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1b260 0x557ecfc15eb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7120019dd0 tx=0x7f7120001e50 comp rx=0 tx=0).stop 2024-06-04T05:05:27.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.085+0000 7f713ca59b80 1 -- 172.21.15.70:0/3121035741 shutdown_connections 2024-06-04T05:05:27.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.085+0000 7f713ca59b80 1 --2- 172.21.15.70:0/3121035741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc20320 0x557ecfc15270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.085+0000 7f713ca59b80 1 --2- 172.21.15.70:0/3121035741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1a8a0 0x557ecfc1ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.085+0000 7f713ca59b80 1 --2- 172.21.15.70:0/3121035741 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1b260 0x557ecfc15eb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.085+0000 7f713ca59b80 1 -- 172.21.15.70:0/3121035741 >> 172.21.15.70:0/3121035741 conn(0x557ecfb46240 msgr2=0x557ecfc11760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:27.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.085+0000 7f713ca59b80 1 -- 172.21.15.70:0/3121035741 shutdown_connections 2024-06-04T05:05:27.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.085+0000 7f713ca59b80 1 -- 172.21.15.70:0/3121035741 wait complete. 2024-06-04T05:05:27.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.085+0000 7f713ca59b80 1 -- start start 2024-06-04T05:05:27.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.086+0000 7f713ca59b80 1 Processor -- start 2024-06-04T05:05:27.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.086+0000 7f713ca59b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1a8a0 0x557ecfc3b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.086+0000 7f713ca59b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1b260 0x557ecfc3b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.086+0000 7f713ca59b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc20320 0x557ecfc2ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.086+0000 7f713ca59b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f712c1b1560 con 0x557ecfc20320 2024-06-04T05:05:27.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.086+0000 7f712b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1a8a0 0x557ecfc3b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.086+0000 7f712b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1a8a0 0x557ecfc3b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36548/0 (socket says 172.21.15.70:36548) 2024-06-04T05:05:27.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.086+0000 7f712b7fe700 1 -- 172.21.15.70:0/224339153 learned_addr learned my addr 172.21.15.70:0/224339153 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:27.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.086+0000 7f713ca59b80 1 -- 172.21.15.70:0/224339153 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f712c1b0e90 con 0x557ecfc1b260 2024-06-04T05:05:27.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.086+0000 7f713ca59b80 1 -- 172.21.15.70:0/224339153 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f712c1ae0b0 con 0x557ecfc1a8a0 2024-06-04T05:05:27.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.086+0000 7f712bfff700 1 --2- 172.21.15.70:0/224339153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc20320 0x557ecfc2ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.086+0000 7f713ca59b80 10 client.0 _ll_register_callbacks cb 0x557ecfb03950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:27.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.086+0000 7f712affd700 1 --2- 172.21.15.70:0/224339153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1b260 0x557ecfc3b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.087+0000 7f712b7fe700 1 -- 172.21.15.70:0/224339153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1b260 msgr2=0x557ecfc3b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:27.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.087+0000 7f712b7fe700 1 --2- 172.21.15.70:0/224339153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1b260 0x557ecfc3b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.087+0000 7f712b7fe700 1 -- 172.21.15.70:0/224339153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc20320 msgr2=0x557ecfc2ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:27.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.087+0000 7f712b7fe700 1 --2- 172.21.15.70:0/224339153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc20320 0x557ecfc2ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.087+0000 7f712b7fe700 1 -- 172.21.15.70:0/224339153 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ecfc3bfd0 con 0x557ecfc1a8a0 2024-06-04T05:05:27.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.087+0000 7f712b7fe700 1 --2- 172.21.15.70:0/224339153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1a8a0 0x557ecfc3b1d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f711c00b650 tx=0x7f711c009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9559c58f1c0dad13 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:27.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.087+0000 7f710affd700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:05:27.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.087+0000 7f710affd700 1 -- 172.21.15.70:0/224339153 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711c00f870 con 0x557ecfc1a8a0 2024-06-04T05:05:27.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.087+0000 7f710affd700 1 -- 172.21.15.70:0/224339153 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f711c00fa10 con 0x557ecfc1a8a0 2024-06-04T05:05:27.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.087+0000 7f713ca59b80 1 -- 172.21.15.70:0/224339153 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7120022400 con 0x557ecfc1a8a0 2024-06-04T05:05:27.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.087+0000 7f710affd700 1 -- 172.21.15.70:0/224339153 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f711c042050 con 0x557ecfc1a8a0 2024-06-04T05:05:27.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.087+0000 7f713ca59b80 10 client.26902 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:27.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.087+0000 7f713ca59b80 1 -- 172.21.15.70:0/224339153 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557ecfc2d0a0 con 0x557ecfc1a8a0 2024-06-04T05:05:27.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.088+0000 7f710affd700 1 -- 172.21.15.70:0/224339153 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7120022400 con 0x557ecfc1a8a0 2024-06-04T05:05:27.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.088+0000 7f710affd700 1 -- 172.21.15.70:0/224339153 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f711c021070 con 0x557ecfc1a8a0 2024-06-04T05:05:27.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.088+0000 7f713ca59b80 10 client.26902 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:27.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.088+0000 7f713ca59b80 10 client.26902 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:05:27.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.088+0000 7f713ca59b80 20 client.26902 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:27.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.088+0000 7f713ca59b80 1 --2- 172.21.15.70:0/224339153 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557ecfc343c0 0x557ecfca1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:27.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.088+0000 7f713ca59b80 4 client.26902 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:05:27.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.088+0000 7f713ca59b80 1 -- 172.21.15.70:0/224339153 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557ecfb744a0 con 0x557ecfc343c0 2024-06-04T05:05:27.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.088+0000 7f712affd700 1 --2- 172.21.15.70:0/224339153 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557ecfc343c0 0x557ecfca1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:27.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.088+0000 7f710affd700 10 client.26902 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:27.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.089+0000 7f712affd700 1 --2- 172.21.15.70:0/224339153 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557ecfc343c0 0x557ecfca1020 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbc0cd1beeb36bdf server_cookie=a9dbdc9c9b74549 in_seq=0 out_seq=0 2024-06-04T05:05:27.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.090+0000 7f710affd700 1 -- 172.21.15.70:0/224339153 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557ecfb744a0 con 0x557ecfc343c0 2024-06-04T05:05:27.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.090+0000 7f710affd700 10 client.26902 handle_command_reply: tid=0 2024-06-04T05:05:27.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.090+0000 7f713ca59b80 1 client.26902 shutdown 2024-06-04T05:05:27.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.090+0000 7f713ca59b80 1 -- 172.21.15.70:0/224339153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1a8a0 msgr2=0x557ecfc3b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:27.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.090+0000 7f713ca59b80 1 --2- 172.21.15.70:0/224339153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1a8a0 0x557ecfc3b1d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f711c00b650 tx=0x7f711c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.090+0000 7f713ca59b80 1 -- 172.21.15.70:0/224339153 shutdown_connections 2024-06-04T05:05:27.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.090+0000 7f713ca59b80 1 --2- 172.21.15.70:0/224339153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ecfc20320 0x557ecfc2ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.090+0000 7f713ca59b80 1 --2- 172.21.15.70:0/224339153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecfc1a8a0 0x557ecfc3b1d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.090+0000 7f713ca59b80 1 --2- 172.21.15.70:0/224339153 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557ecfc343c0 0x557ecfca1020 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.090+0000 7f713ca59b80 1 --2- 172.21.15.70:0/224339153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ecfc1b260 0x557ecfc3b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.090+0000 7f710affd700 0 client.26902 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:27.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.090+0000 7f713ca59b80 1 -- 172.21.15.70:0/224339153 >> 172.21.15.70:0/224339153 conn(0x557ecfb46240 msgr2=0x557ecfc1dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:27.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.091+0000 7f713ca59b80 1 -- 172.21.15.70:0/224339153 shutdown_connections 2024-06-04T05:05:27.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.091+0000 7f713ca59b80 1 -- 172.21.15.70:0/224339153 wait complete. 2024-06-04T05:05:27.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.091+0000 7f713ca59b80 20 client.26902 trim_cache size 0 max 16384 2024-06-04T05:05:27.097 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:05:27.097 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:05:27.097 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:05:27.097 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:05:27.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.092+0000 7f71321c4700 1 -- 172.21.15.70:0/1942477813 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f71140b29f0 msgr2=0x7f71140b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:27.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.092+0000 7f71321c4700 1 --2- 172.21.15.70:0/1942477813 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f71140b29f0 0x7f71140b4eb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f711c00b2e0 tx=0x7f711c005fb0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.092+0000 7f71321c4700 1 -- 172.21.15.70:0/1942477813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f712c108520 msgr2=0x7f712c111500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:27.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.092+0000 7f71321c4700 1 --2- 172.21.15.70:0/1942477813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f712c108520 0x7f712c111500 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f712400ef30 tx=0x7f712400c3a0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.092+0000 7f71321c4700 1 -- 172.21.15.70:0/1942477813 shutdown_connections 2024-06-04T05:05:27.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.092+0000 7f71321c4700 1 --2- 172.21.15.70:0/1942477813 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f71140b29f0 0x7f71140b4eb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.092+0000 7f71321c4700 1 --2- 172.21.15.70:0/1942477813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f712c108520 0x7f712c111500 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.092+0000 7f71321c4700 1 --2- 172.21.15.70:0/1942477813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f712c102440 0x7f712c110a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.092+0000 7f71321c4700 1 --2- 172.21.15.70:0/1942477813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c102e00 0x7f712c110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:27.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.092+0000 7f71321c4700 1 -- 172.21.15.70:0/1942477813 >> 172.21.15.70:0/1942477813 conn(0x7f712c0793f0 msgr2=0x7f712c101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:27.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.092+0000 7f71321c4700 1 -- 172.21.15.70:0/1942477813 shutdown_connections 2024-06-04T05:05:27.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:27.092+0000 7f71321c4700 1 -- 172.21.15.70:0/1942477813 wait complete. 2024-06-04T05:05:27.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:27 smithi070 ceph-mon[32631]: pgmap v253: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 2.0 MiB/s wr, 191 op/s 2024-06-04T05:05:27.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:27 smithi112 ceph-mon[36453]: pgmap v253: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 2.0 MiB/s wr, 191 op/s 2024-06-04T05:05:27.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:27 smithi098 ceph-mon[35569]: pgmap v253: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 2.0 MiB/s wr, 191 op/s 2024-06-04T05:05:27.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 148081 2.12 MB/sec execute 260 sec latency 17.057 ms 2024-06-04T05:05:28.719 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:05:28.719 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:05:28.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 148783 2.13 MB/sec execute 261 sec latency 15.657 ms 2024-06-04T05:05:29.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:29 smithi112 ceph-mon[36453]: pgmap v254: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 1.3 MiB/s wr, 130 op/s 2024-06-04T05:05:29.447 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:29 smithi070 ceph-mon[32631]: pgmap v254: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 1.3 MiB/s wr, 130 op/s 2024-06-04T05:05:29.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:29 smithi098 ceph-mon[35569]: pgmap v254: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 1.3 MiB/s wr, 130 op/s 2024-06-04T05:05:29.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149255 2.14 MB/sec execute 262 sec latency 27.824 ms 2024-06-04T05:05:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.772+0000 7f6966354700 1 -- 172.21.15.70:0/2525011905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6960102470 msgr2=0x7f69601028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.772+0000 7f6966354700 1 --2- 172.21.15.70:0/2525011905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6960102470 0x7f69601028f0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f6954009990 tx=0x7f6954009ca0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.772+0000 7f6966354700 1 -- 172.21.15.70:0/2525011905 shutdown_connections 2024-06-04T05:05:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.772+0000 7f6966354700 1 --2- 172.21.15.70:0/2525011905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6960102e30 0x7f696010f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.772+0000 7f6966354700 1 --2- 172.21.15.70:0/2525011905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69601073d0 0x7f69601077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.772+0000 7f6966354700 1 --2- 172.21.15.70:0/2525011905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6960102470 0x7f69601028f0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.772+0000 7f6966354700 1 -- 172.21.15.70:0/2525011905 >> 172.21.15.70:0/2525011905 conn(0x7f6960079480 msgr2=0x7f6960079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.772+0000 7f6966354700 1 -- 172.21.15.70:0/2525011905 shutdown_connections 2024-06-04T05:05:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.772+0000 7f6966354700 1 -- 172.21.15.70:0/2525011905 wait complete. 2024-06-04T05:05:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.773+0000 7f6966354700 1 Processor -- start 2024-06-04T05:05:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.773+0000 7f6966354700 1 -- start start 2024-06-04T05:05:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.774+0000 7f6966354700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6960102470 0x7f696019d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.774+0000 7f6966354700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6960102e30 0x7f696019db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.774+0000 7f6966354700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69601073d0 0x7f69601976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.774+0000 7f6966354700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6960111420 con 0x7f6960102e30 2024-06-04T05:05:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.774+0000 7f6966354700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f69601112a0 con 0x7f6960102470 2024-06-04T05:05:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.774+0000 7f6966354700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f69601115a0 con 0x7f69601073d0 2024-06-04T05:05:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.774+0000 7f695ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6960102470 0x7f696019d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.774+0000 7f695ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6960102470 0x7f696019d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33270/0 (socket says 172.21.15.70:33270) 2024-06-04T05:05:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.774+0000 7f695ffff700 1 -- 172.21.15.70:0/1100819428 learned_addr learned my addr 172.21.15.70:0/1100819428 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.774+0000 7f695f7fe700 1 --2- 172.21.15.70:0/1100819428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6960102e30 0x7f696019db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.774+0000 7f69648f1700 1 --2- 172.21.15.70:0/1100819428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69601073d0 0x7f69601976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.775+0000 7f695ffff700 1 -- 172.21.15.70:0/1100819428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69601073d0 msgr2=0x7f69601976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.775+0000 7f695ffff700 1 --2- 172.21.15.70:0/1100819428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69601073d0 0x7f69601976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.775+0000 7f695ffff700 1 -- 172.21.15.70:0/1100819428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6960102e30 msgr2=0x7f696019db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.775+0000 7f695ffff700 1 --2- 172.21.15.70:0/1100819428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6960102e30 0x7f696019db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.775+0000 7f695ffff700 1 -- 172.21.15.70:0/1100819428 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6960197fc0 con 0x7f6960102470 2024-06-04T05:05:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.775+0000 7f695ffff700 1 --2- 172.21.15.70:0/1100819428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6960102470 0x7f696019d5c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6950009f60 tx=0x7f6950009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.775+0000 7f695d7fa700 1 -- 172.21.15.70:0/1100819428 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6950013ca0 con 0x7f6960102470 2024-06-04T05:05:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.775+0000 7f695d7fa700 1 -- 172.21.15.70:0/1100819428 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6950013e40 con 0x7f6960102470 2024-06-04T05:05:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.775+0000 7f695d7fa700 1 -- 172.21.15.70:0/1100819428 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6950003680 con 0x7f6960102470 2024-06-04T05:05:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.775+0000 7f6966354700 1 -- 172.21.15.70:0/1100819428 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69601982b0 con 0x7f6960102470 2024-06-04T05:05:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.776+0000 7f6966354700 1 -- 172.21.15.70:0/1100819428 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69601a4540 con 0x7f6960102470 2024-06-04T05:05:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.780+0000 7f695d7fa700 1 -- 172.21.15.70:0/1100819428 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f6950022020 con 0x7f6960102470 2024-06-04T05:05:29.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.781+0000 7f695d7fa700 1 --2- 172.21.15.70:0/1100819428 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f69480b2b90 0x7f69480b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.781+0000 7f6970be9b80 1 Processor -- start 2024-06-04T05:05:29.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.781+0000 7f695d7fa700 1 -- 172.21.15.70:0/1100819428 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f69500e64e0 con 0x7f6960102470 2024-06-04T05:05:29.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.781+0000 7f6970be9b80 1 -- start start 2024-06-04T05:05:29.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.782+0000 7f695f7fe700 1 --2- 172.21.15.70:0/1100819428 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f69480b2b90 0x7f69480b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.782+0000 7f6970be9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e844fb0 0x55e11e845390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.782+0000 7f6970be9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e840490 0x55e11e840910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.782+0000 7f6970be9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e840e50 0x55e11e84cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.782+0000 7f6970be9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e11e795f00 con 0x55e11e840e50 2024-06-04T05:05:29.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.782+0000 7f6970be9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e11e785aa0 con 0x55e11e840490 2024-06-04T05:05:29.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.782+0000 7f6970be9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e11e789ae0 con 0x55e11e844fb0 2024-06-04T05:05:29.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.782+0000 7f69648f1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e840e50 0x55e11e84cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.782+0000 7f69648f1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e840e50 0x55e11e84cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37976/0 (socket says 172.21.15.70:37976) 2024-06-04T05:05:29.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.782+0000 7f69648f1700 1 -- 172.21.15.70:0/1078006929 learned_addr learned my addr 172.21.15.70:0/1078006929 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:29.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.782+0000 7f695ffff700 1 --2- 172.21.15.70:0/1078006929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e844fb0 0x55e11e845390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.782+0000 7f695ffff700 1 -- 172.21.15.70:0/1078006929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e840490 msgr2=0x55e11e840910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:05:29.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.782+0000 7f695ffff700 1 --2- 172.21.15.70:0/1078006929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e840490 0x55e11e840910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.782+0000 7f695ffff700 1 -- 172.21.15.70:0/1078006929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e840e50 msgr2=0x55e11e84cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:29.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.782+0000 7f695ffff700 1 --2- 172.21.15.70:0/1078006929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e840e50 0x55e11e84cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.782+0000 7f695ffff700 1 -- 172.21.15.70:0/1078006929 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e11e72f140 con 0x55e11e844fb0 2024-06-04T05:05:29.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.783+0000 7f695ffff700 1 --2- 172.21.15.70:0/1078006929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e844fb0 0x55e11e845390 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f69500e3cb0 tx=0x7f69500ad350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b84955fe4c36e14a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:29.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.783+0000 7f695f7fe700 1 --2- 172.21.15.70:0/1100819428 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f69480b2b90 0x7f69480b5050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6960198e50 tx=0x7f695401a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:29.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.783+0000 7f69467fc700 1 -- 172.21.15.70:0/1078006929 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69500ad840 con 0x55e11e844fb0 2024-06-04T05:05:29.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.783+0000 7f69467fc700 1 -- 172.21.15.70:0/1078006929 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69500b1bf0 con 0x55e11e844fb0 2024-06-04T05:05:29.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.783+0000 7f69467fc700 1 -- 172.21.15.70:0/1078006929 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69500add00 con 0x55e11e844fb0 2024-06-04T05:05:29.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.784+0000 7f6970be9b80 1 -- 172.21.15.70:0/1078006929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e844fb0 msgr2=0x55e11e845390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:29.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.784+0000 7f6970be9b80 1 --2- 172.21.15.70:0/1078006929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e844fb0 0x55e11e845390 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f69500e3cb0 tx=0x7f69500ad350 comp rx=0 tx=0).stop 2024-06-04T05:05:29.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.784+0000 7f6970be9b80 1 -- 172.21.15.70:0/1078006929 shutdown_connections 2024-06-04T05:05:29.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.784+0000 7f6970be9b80 1 --2- 172.21.15.70:0/1078006929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e840e50 0x55e11e84cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.784+0000 7f6970be9b80 1 --2- 172.21.15.70:0/1078006929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e844fb0 0x55e11e845390 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.784+0000 7f6970be9b80 1 --2- 172.21.15.70:0/1078006929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e840490 0x55e11e840910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.784+0000 7f6970be9b80 1 -- 172.21.15.70:0/1078006929 >> 172.21.15.70:0/1078006929 conn(0x55e11e7633d0 msgr2=0x55e11e832970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:29.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.784+0000 7f6970be9b80 1 -- 172.21.15.70:0/1078006929 shutdown_connections 2024-06-04T05:05:29.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.784+0000 7f6970be9b80 1 -- 172.21.15.70:0/1078006929 wait complete. 2024-06-04T05:05:29.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.784+0000 7f6970be9b80 1 -- start start 2024-06-04T05:05:29.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.785+0000 7f6970be9b80 1 Processor -- start 2024-06-04T05:05:29.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.785+0000 7f6970be9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e840490 0x55e11e84ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.785+0000 7f6970be9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e840e50 0x55e11e849fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.785+0000 7f6970be9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e844fb0 0x55e11e84a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.785+0000 7f695f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e840e50 0x55e11e849fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.785+0000 7f6970be9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e11e68c130 con 0x55e11e840e50 2024-06-04T05:05:29.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.785+0000 7f6970be9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e11e6edf30 con 0x55e11e840490 2024-06-04T05:05:29.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.785+0000 7f6970be9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e11e84d510 con 0x55e11e844fb0 2024-06-04T05:05:29.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.785+0000 7f695ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e840490 0x55e11e84ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.785+0000 7f695ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e840490 0x55e11e84ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33280/0 (socket says 172.21.15.70:33280) 2024-06-04T05:05:29.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.785+0000 7f695ffff700 1 -- 172.21.15.70:0/2744275863 learned_addr learned my addr 172.21.15.70:0/2744275863 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:29.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.785+0000 7f6970be9b80 10 client.0 _ll_register_callbacks cb 0x55e11e720ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:29.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.785+0000 7f69648f1700 1 --2- 172.21.15.70:0/2744275863 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e844fb0 0x55e11e84a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.785+0000 7f695ffff700 1 -- 172.21.15.70:0/2744275863 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e844fb0 msgr2=0x55e11e84a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:29.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.785+0000 7f695ffff700 1 --2- 172.21.15.70:0/2744275863 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e844fb0 0x55e11e84a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.785+0000 7f695ffff700 1 -- 172.21.15.70:0/2744275863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e840e50 msgr2=0x55e11e849fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:29.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.786+0000 7f695ffff700 1 --2- 172.21.15.70:0/2744275863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e840e50 0x55e11e849fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.786+0000 7f695ffff700 1 -- 172.21.15.70:0/2744275863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e11e6d5a60 con 0x55e11e840490 2024-06-04T05:05:29.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.786+0000 7f69648f1700 1 --2- 172.21.15.70:0/2744275863 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e844fb0 0x55e11e84a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:29.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.786+0000 7f695f7fe700 1 --2- 172.21.15.70:0/2744275863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e840e50 0x55e11e849fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:29.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.786+0000 7f695ffff700 1 --2- 172.21.15.70:0/2744275863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e840490 0x55e11e84ccb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6950009230 tx=0x7f6950005bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ef23c30e6d2da8f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:29.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.786+0000 7f693f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:05:29.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.786+0000 7f693f7fe700 1 -- 172.21.15.70:0/2744275863 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69500b66a0 con 0x55e11e840490 2024-06-04T05:05:29.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.786+0000 7f6970be9b80 1 -- 172.21.15.70:0/2744275863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69500b1bf0 con 0x55e11e840490 2024-06-04T05:05:29.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.786+0000 7f693f7fe700 1 -- 172.21.15.70:0/2744275863 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f69500aeb70 con 0x55e11e840490 2024-06-04T05:05:29.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.786+0000 7f693f7fe700 1 -- 172.21.15.70:0/2744275863 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f69500ad840 con 0x55e11e840490 2024-06-04T05:05:29.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.786+0000 7f6970be9b80 10 client.26834 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:29.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.786+0000 7f6970be9b80 1 -- 172.21.15.70:0/2744275863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e11e73f8f0 con 0x55e11e840490 2024-06-04T05:05:29.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.786+0000 7f693f7fe700 1 -- 172.21.15.70:0/2744275863 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69500b1bf0 con 0x55e11e840490 2024-06-04T05:05:29.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.787+0000 7f693f7fe700 1 -- 172.21.15.70:0/2744275863 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f69500e5070 con 0x55e11e840490 2024-06-04T05:05:29.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.787+0000 7f6970be9b80 10 client.26834 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:29.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.787+0000 7f6970be9b80 10 client.26834 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:05:29.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.787+0000 7f6970be9b80 20 client.26834 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:29.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.787+0000 7f6970be9b80 1 --2- 172.21.15.70:0/2744275863 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e11e85a040 0x55e11e85c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.787+0000 7f6970be9b80 4 client.26834 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:05:29.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.787+0000 7f6970be9b80 1 -- 172.21.15.70:0/2744275863 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e11e797e50 con 0x55e11e85a040 2024-06-04T05:05:29.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.787+0000 7f695f7fe700 1 --2- 172.21.15.70:0/2744275863 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e11e85a040 0x55e11e85c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.787+0000 7f693f7fe700 10 client.26834 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:29.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.787+0000 7f695f7fe700 1 --2- 172.21.15.70:0/2744275863 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e11e85a040 0x55e11e85c430 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27537a0b9595d576 server_cookie=2ea5226b7c59a8b8 in_seq=0 out_seq=0 2024-06-04T05:05:29.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.788+0000 7f693f7fe700 1 -- 172.21.15.70:0/2744275863 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e11e797e50 con 0x55e11e85a040 2024-06-04T05:05:29.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.788+0000 7f693f7fe700 10 client.26834 handle_command_reply: tid=0 2024-06-04T05:05:29.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.788+0000 7f6970be9b80 1 client.26834 shutdown 2024-06-04T05:05:29.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.789+0000 7f6970be9b80 1 -- 172.21.15.70:0/2744275863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e840490 msgr2=0x55e11e84ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:29.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.789+0000 7f6970be9b80 1 --2- 172.21.15.70:0/2744275863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e840490 0x55e11e84ccb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6950009230 tx=0x7f6950005bb0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.789+0000 7f6970be9b80 1 -- 172.21.15.70:0/2744275863 shutdown_connections 2024-06-04T05:05:29.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.789+0000 7f6970be9b80 1 --2- 172.21.15.70:0/2744275863 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e844fb0 0x55e11e84a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.789+0000 7f6970be9b80 1 --2- 172.21.15.70:0/2744275863 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e11e85a040 0x55e11e85c430 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.789+0000 7f6970be9b80 1 --2- 172.21.15.70:0/2744275863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e840490 0x55e11e84ccb0 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6950009230 tx=0x7f6950005bb0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.789+0000 7f6970be9b80 1 --2- 172.21.15.70:0/2744275863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e840e50 0x55e11e849fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.789+0000 7f693f7fe700 0 client.26834 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:29.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.789+0000 7f6970be9b80 1 -- 172.21.15.70:0/2744275863 >> 172.21.15.70:0/2744275863 conn(0x55e11e7633d0 msgr2=0x55e11e836c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:29.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.789+0000 7f6970be9b80 1 -- 172.21.15.70:0/2744275863 shutdown_connections 2024-06-04T05:05:29.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.789+0000 7f6970be9b80 1 -- 172.21.15.70:0/2744275863 wait complete. 2024-06-04T05:05:29.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.789+0000 7f6970be9b80 20 client.26834 trim_cache size 0 max 16384 2024-06-04T05:05:29.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.815+0000 7f6970be9b80 1 Processor -- start 2024-06-04T05:05:29.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.815+0000 7f6970be9b80 1 -- start start 2024-06-04T05:05:29.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.815+0000 7f6970be9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e840e50 0x55e11e82cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.815+0000 7f6970be9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e844fb0 0x55e11e84a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.815+0000 7f6970be9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e85f080 0x55e11e85f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.815+0000 7f6970be9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f69601abc10 con 0x55e11e85f080 2024-06-04T05:05:29.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.815+0000 7f6970be9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f69601ad750 con 0x55e11e844fb0 2024-06-04T05:05:29.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.815+0000 7f6970be9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f69601ac2e0 con 0x55e11e840e50 2024-06-04T05:05:29.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.815+0000 7f69648f1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e85f080 0x55e11e85f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.815+0000 7f69648f1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e85f080 0x55e11e85f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38006/0 (socket says 172.21.15.70:38006) 2024-06-04T05:05:29.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.815+0000 7f695f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e844fb0 0x55e11e84a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.815+0000 7f69648f1700 1 -- 172.21.15.70:0/1212217388 learned_addr learned my addr 172.21.15.70:0/1212217388 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:29.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.815+0000 7f695ffff700 1 --2- 172.21.15.70:0/1212217388 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e840e50 0x55e11e82cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.816+0000 7f695f7fe700 1 -- 172.21.15.70:0/1212217388 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e840e50 msgr2=0x55e11e82cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:29.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.816+0000 7f695f7fe700 1 --2- 172.21.15.70:0/1212217388 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e840e50 0x55e11e82cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.816+0000 7f695f7fe700 1 -- 172.21.15.70:0/1212217388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e85f080 msgr2=0x55e11e85f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:29.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.816+0000 7f695f7fe700 1 --2- 172.21.15.70:0/1212217388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e85f080 0x55e11e85f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.816+0000 7f695f7fe700 1 -- 172.21.15.70:0/1212217388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e11e848520 con 0x55e11e844fb0 2024-06-04T05:05:29.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.816+0000 7f69648f1700 1 --2- 172.21.15.70:0/1212217388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e85f080 0x55e11e85f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:29.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.816+0000 7f695f7fe700 1 --2- 172.21.15.70:0/1212217388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e844fb0 0x55e11e84a120 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f695400b650 tx=0x7f6954022000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d20d5c918e3e9e82 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:29.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.816+0000 7f6945ffb700 1 -- 172.21.15.70:0/1212217388 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6954030620 con 0x55e11e844fb0 2024-06-04T05:05:29.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.816+0000 7f6945ffb700 1 -- 172.21.15.70:0/1212217388 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6954030c50 con 0x55e11e844fb0 2024-06-04T05:05:29.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.816+0000 7f6945ffb700 1 -- 172.21.15.70:0/1212217388 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6954024a50 con 0x55e11e844fb0 2024-06-04T05:05:29.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.817+0000 7f6970be9b80 1 -- 172.21.15.70:0/1212217388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e844fb0 msgr2=0x55e11e84a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.817+0000 7f6970be9b80 1 --2- 172.21.15.70:0/1212217388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e844fb0 0x55e11e84a120 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f695400b650 tx=0x7f6954022000 comp rx=0 tx=0).stop 2024-06-04T05:05:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.817+0000 7f6970be9b80 1 -- 172.21.15.70:0/1212217388 shutdown_connections 2024-06-04T05:05:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.817+0000 7f6970be9b80 1 --2- 172.21.15.70:0/1212217388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e85f080 0x55e11e85f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.817+0000 7f6970be9b80 1 --2- 172.21.15.70:0/1212217388 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e840e50 0x55e11e82cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.817+0000 7f6970be9b80 1 --2- 172.21.15.70:0/1212217388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e844fb0 0x55e11e84a120 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.817+0000 7f6970be9b80 1 -- 172.21.15.70:0/1212217388 >> 172.21.15.70:0/1212217388 conn(0x55e11e7633d0 msgr2=0x55e11e8372e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.817+0000 7f6970be9b80 1 -- 172.21.15.70:0/1212217388 shutdown_connections 2024-06-04T05:05:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.817+0000 7f6970be9b80 1 -- 172.21.15.70:0/1212217388 wait complete. 2024-06-04T05:05:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.818+0000 7f6970be9b80 1 -- start start 2024-06-04T05:05:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.818+0000 7f6970be9b80 1 Processor -- start 2024-06-04T05:05:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.818+0000 7f6970be9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e840e50 0x55e11e8be090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.819+0000 7f6970be9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e844fb0 0x55e11e8be5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.819+0000 7f6970be9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e85f080 0x55e11e8b8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.819+0000 7f695ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e840e50 0x55e11e8be090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.819+0000 7f695ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e840e50 0x55e11e8be090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36620/0 (socket says 172.21.15.70:36620) 2024-06-04T05:05:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.819+0000 7f695f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e844fb0 0x55e11e8be5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.819+0000 7f695f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e844fb0 0x55e11e8be5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33298/0 (socket says 172.21.15.70:33298) 2024-06-04T05:05:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.819+0000 7f695ffff700 1 -- 172.21.15.70:0/1114438712 learned_addr learned my addr 172.21.15.70:0/1114438712 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.819+0000 7f6970be9b80 1 -- 172.21.15.70:0/1114438712 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f69601ad080 con 0x55e11e85f080 2024-06-04T05:05:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.819+0000 7f69648f1700 1 --2- 172.21.15.70:0/1114438712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e85f080 0x55e11e8b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.819+0000 7f6970be9b80 1 -- 172.21.15.70:0/1114438712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f69601ac9b0 con 0x55e11e844fb0 2024-06-04T05:05:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.819+0000 7f6970be9b80 1 -- 172.21.15.70:0/1114438712 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f69601a9bd0 con 0x55e11e840e50 2024-06-04T05:05:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.819+0000 7f6970be9b80 10 client.0 _ll_register_callbacks cb 0x55e11e720ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.819+0000 7f695ffff700 1 -- 172.21.15.70:0/1114438712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e844fb0 msgr2=0x55e11e8be5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.819+0000 7f695ffff700 1 --2- 172.21.15.70:0/1114438712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e844fb0 0x55e11e8be5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.819+0000 7f695ffff700 1 -- 172.21.15.70:0/1114438712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e85f080 msgr2=0x55e11e8b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:29.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.819+0000 7f695ffff700 1 --2- 172.21.15.70:0/1114438712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e85f080 0x55e11e8b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.820+0000 7f695ffff700 1 -- 172.21.15.70:0/1114438712 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e11e8b9020 con 0x55e11e840e50 2024-06-04T05:05:29.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.820+0000 7f695f7fe700 1 --2- 172.21.15.70:0/1114438712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e844fb0 0x55e11e8be5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:29.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.820+0000 7f695ffff700 1 --2- 172.21.15.70:0/1114438712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e840e50 0x55e11e8be090 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f69500ad7d0 tx=0x7f69500a9560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e70d311e8a196db server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:29.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.820+0000 7f693f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:05:29.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.820+0000 7f693f7fe700 1 -- 172.21.15.70:0/1114438712 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69500ad610 con 0x55e11e840e50 2024-06-04T05:05:29.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.820+0000 7f693f7fe700 1 -- 172.21.15.70:0/1114438712 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f69500ac860 con 0x55e11e840e50 2024-06-04T05:05:29.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.820+0000 7f6970be9b80 1 -- 172.21.15.70:0/1114438712 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6954030c50 con 0x55e11e840e50 2024-06-04T05:05:29.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.820+0000 7f693f7fe700 1 -- 172.21.15.70:0/1114438712 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f69500b66a0 con 0x55e11e840e50 2024-06-04T05:05:29.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.820+0000 7f6970be9b80 10 client.26917 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:29.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.820+0000 7f6970be9b80 1 -- 172.21.15.70:0/1114438712 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e11e8c5770 con 0x55e11e840e50 2024-06-04T05:05:29.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.820+0000 7f693f7fe700 1 -- 172.21.15.70:0/1114438712 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6954030c50 con 0x55e11e840e50 2024-06-04T05:05:29.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.821+0000 7f693f7fe700 1 -- 172.21.15.70:0/1114438712 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f69500e5070 con 0x55e11e840e50 2024-06-04T05:05:29.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.821+0000 7f6970be9b80 10 client.26917 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:29.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.821+0000 7f6970be9b80 10 client.26917 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:05:29.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.821+0000 7f6970be9b80 20 client.26917 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:29.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.821+0000 7f6970be9b80 1 --2- 172.21.15.70:0/1114438712 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e11e8c5c50 0x55e11e8c8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.821+0000 7f6970be9b80 4 client.26917 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.6, "await_for": 45.0}] 2024-06-04T05:05:29.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.821+0000 7f6970be9b80 1 -- 172.21.15.70:0/1114438712 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.6, "await_for": 45.0}) -- 0x55e11e791630 con 0x55e11e8c5c50 2024-06-04T05:05:29.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.821+0000 7f695f7fe700 1 --2- 172.21.15.70:0/1114438712 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e11e8c5c50 0x55e11e8c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.821+0000 7f693f7fe700 10 client.26917 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:29.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.821+0000 7f695f7fe700 1 --2- 172.21.15.70:0/1114438712 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e11e8c5c50 0x55e11e8c8040 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8acdaa77ddb1089c server_cookie=ebe95cd71bfa9f45 in_seq=0 out_seq=0 2024-06-04T05:05:29.859 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:05:29.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.922+0000 7f693f7fe700 1 -- 172.21.15.70:0/1114438712 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e11e791630 con 0x55e11e8c5c50 2024-06-04T05:05:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.922+0000 7f693f7fe700 10 client.26917 handle_command_reply: tid=0 2024-06-04T05:05:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.922+0000 7f6970be9b80 1 client.26917 shutdown 2024-06-04T05:05:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.922+0000 7f6970be9b80 1 -- 172.21.15.70:0/1114438712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e840e50 msgr2=0x55e11e8be090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.922+0000 7f6970be9b80 1 --2- 172.21.15.70:0/1114438712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e840e50 0x55e11e8be090 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f69500ad7d0 tx=0x7f69500a9560 comp rx=0 tx=0).stop 2024-06-04T05:05:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.922+0000 7f6970be9b80 1 -- 172.21.15.70:0/1114438712 shutdown_connections 2024-06-04T05:05:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.922+0000 7f6970be9b80 1 --2- 172.21.15.70:0/1114438712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e11e85f080 0x55e11e8b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.922+0000 7f6970be9b80 1 --2- 172.21.15.70:0/1114438712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e11e840e50 0x55e11e8be090 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.922+0000 7f6970be9b80 1 --2- 172.21.15.70:0/1114438712 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e11e8c5c50 0x55e11e8c8040 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.922+0000 7f6970be9b80 1 --2- 172.21.15.70:0/1114438712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e11e844fb0 0x55e11e8be5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.922+0000 7f693f7fe700 0 client.26917 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:29.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.922+0000 7f6970be9b80 1 -- 172.21.15.70:0/1114438712 >> 172.21.15.70:0/1114438712 conn(0x55e11e7633d0 msgr2=0x55e11e849890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:29.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.922+0000 7f6970be9b80 1 -- 172.21.15.70:0/1114438712 shutdown_connections 2024-06-04T05:05:29.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.922+0000 7f6970be9b80 1 -- 172.21.15.70:0/1114438712 wait complete. 2024-06-04T05:05:29.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.922+0000 7f6970be9b80 20 client.26917 trim_cache size 0 max 16384 2024-06-04T05:05:29.927 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:05:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 32, 2024-06-04T05:05:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:05:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 22, 2024-06-04T05:05:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:05:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "18106a7e": { 2024-06-04T05:05:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 22, 2024-06-04T05:05:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:05:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:05:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:05:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:05:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:05:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:05:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 143.6, 2024-06-04T05:05:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:05:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:05:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:05:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:05:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:05:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:05:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:05:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:05:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:05:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:05:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:05:29.929 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:05:29.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.923+0000 7f6966354700 1 -- 172.21.15.70:0/1100819428 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f69480b2b90 msgr2=0x7f69480b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:29.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.923+0000 7f6966354700 1 --2- 172.21.15.70:0/1100819428 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f69480b2b90 0x7f69480b5050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6960198e50 tx=0x7f695401a040 comp rx=0 tx=0).stop 2024-06-04T05:05:29.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.923+0000 7f6966354700 1 -- 172.21.15.70:0/1100819428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6960102470 msgr2=0x7f696019d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:29.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.923+0000 7f6966354700 1 --2- 172.21.15.70:0/1100819428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6960102470 0x7f696019d5c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6950009f60 tx=0x7f6950009f90 comp rx=0 tx=0).stop 2024-06-04T05:05:29.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.923+0000 7f6966354700 1 -- 172.21.15.70:0/1100819428 shutdown_connections 2024-06-04T05:05:29.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.923+0000 7f6966354700 1 --2- 172.21.15.70:0/1100819428 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f69480b2b90 0x7f69480b5050 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.923+0000 7f6966354700 1 --2- 172.21.15.70:0/1100819428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69601073d0 0x7f69601976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.923+0000 7f6966354700 1 --2- 172.21.15.70:0/1100819428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6960102470 0x7f696019d5c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.923+0000 7f6966354700 1 --2- 172.21.15.70:0/1100819428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6960102e30 0x7f696019db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.923+0000 7f6966354700 1 -- 172.21.15.70:0/1100819428 >> 172.21.15.70:0/1100819428 conn(0x7f6960079480 msgr2=0x7f6960100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:29.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.923+0000 7f6966354700 1 -- 172.21.15.70:0/1100819428 shutdown_connections 2024-06-04T05:05:29.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:29.923+0000 7f6966354700 1 -- 172.21.15.70:0/1100819428 wait complete. 2024-06-04T05:05:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.983+0000 7f4e0fb6a640 1 -- 172.21.15.70:0/2100517014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e08107940 msgr2=0x7f4e08107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.983+0000 7f4e0fb6a640 1 --2- 172.21.15.70:0/2100517014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e08107940 0x7f4e08107d20 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f4e04009cd0 tx=0x7f4e0402a8e0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.983+0000 7f4e0fb6a640 1 -- 172.21.15.70:0/2100517014 shutdown_connections 2024-06-04T05:05:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.983+0000 7f4e0fb6a640 1 --2- 172.21.15.70:0/2100517014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e08102340 0x7f4e0810e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.983+0000 7f4e0fb6a640 1 --2- 172.21.15.70:0/2100517014 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e08101980 0x7f4e08101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.983+0000 7f4e0fb6a640 1 --2- 172.21.15.70:0/2100517014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e08107940 0x7f4e08107d20 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.983+0000 7f4e0fb6a640 1 -- 172.21.15.70:0/2100517014 >> 172.21.15.70:0/2100517014 conn(0x7f4e08078ad0 msgr2=0x7f4e08078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.983+0000 7f4e0fb6a640 1 -- 172.21.15.70:0/2100517014 shutdown_connections 2024-06-04T05:05:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.983+0000 7f4e0fb6a640 1 -- 172.21.15.70:0/2100517014 wait complete. 2024-06-04T05:05:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.984+0000 7f4e0fb6a640 1 Processor -- start 2024-06-04T05:05:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.984+0000 7f4e0fb6a640 1 -- start start 2024-06-04T05:05:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.984+0000 7f4e0fb6a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e08101980 0x7f4e08199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.984+0000 7f4e0fb6a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e08102340 0x7f4e0819a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.984+0000 7f4e0fb6a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e08107940 0x7f4e081a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.984+0000 7f4e0fb6a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e08111a80 con 0x7f4e08102340 2024-06-04T05:05:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.984+0000 7f4e0fb6a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e08111900 con 0x7f4e08101980 2024-06-04T05:05:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.984+0000 7f4e0fb6a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4e08111c00 con 0x7f4e08107940 2024-06-04T05:05:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.984+0000 7f4e0d105640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e08102340 0x7f4e0819a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.984+0000 7f4e0d105640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e08102340 0x7f4e0819a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38022/0 (socket says 172.21.15.70:38022) 2024-06-04T05:05:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.984+0000 7f4e0d105640 1 -- 172.21.15.70:0/3973664810 learned_addr learned my addr 172.21.15.70:0/3973664810 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.984+0000 7f4e0d906640 1 --2- 172.21.15.70:0/3973664810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e08101980 0x7f4e08199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.984+0000 7f4e0e107640 1 --2- 172.21.15.70:0/3973664810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e08107940 0x7f4e081a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.985+0000 7f4e0d105640 1 -- 172.21.15.70:0/3973664810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e08107940 msgr2=0x7f4e081a4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.985+0000 7f4e0d105640 1 --2- 172.21.15.70:0/3973664810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e08107940 0x7f4e081a4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.985+0000 7f4e0d105640 1 -- 172.21.15.70:0/3973664810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e08101980 msgr2=0x7f4e08199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.985+0000 7f4e0d105640 1 --2- 172.21.15.70:0/3973664810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e08101980 0x7f4e08199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.985+0000 7f4e0d105640 1 -- 172.21.15.70:0/3973664810 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e081a54c0 con 0x7f4e08102340 2024-06-04T05:05:29.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.985+0000 7f4e0d906640 1 --2- 172.21.15.70:0/3973664810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e08101980 0x7f4e08199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:29.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.985+0000 7f4e0d105640 1 --2- 172.21.15.70:0/3973664810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e08102340 0x7f4e0819a440 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f4df800e9f0 tx=0x7f4df800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:29.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.985+0000 7f4dfeffd640 1 -- 172.21.15.70:0/3973664810 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df800cd80 con 0x7f4e08102340 2024-06-04T05:05:29.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.985+0000 7f4dfeffd640 1 -- 172.21.15.70:0/3973664810 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4df80108c0 con 0x7f4e08102340 2024-06-04T05:05:29.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.985+0000 7f4dfeffd640 1 -- 172.21.15.70:0/3973664810 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df80187c0 con 0x7f4e08102340 2024-06-04T05:05:29.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.985+0000 7f4e0fb6a640 1 -- 172.21.15.70:0/3973664810 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e081a5690 con 0x7f4e08102340 2024-06-04T05:05:29.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.985+0000 7f4e0fb6a640 1 -- 172.21.15.70:0/3973664810 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e081a5b90 con 0x7f4e08102340 2024-06-04T05:05:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.988+0000 7f4e12881740 1 Processor -- start 2024-06-04T05:05:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.988+0000 7f4dfeffd640 1 -- 172.21.15.70:0/3973664810 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f4df8010430 con 0x7f4e08102340 2024-06-04T05:05:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.988+0000 7f4e12881740 1 -- start start 2024-06-04T05:05:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.988+0000 7f4e12881740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 0x5637b9148c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.988+0000 7f4e12881740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 0x5637b91534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.988+0000 7f4e12881740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 0x5637b9155df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.988+0000 7f4e12881740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5637b90bac30 con 0x5637b9148820 2024-06-04T05:05:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.988+0000 7f4e12881740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5637b90aa7d0 con 0x5637b9153a00 2024-06-04T05:05:29.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.988+0000 7f4e12881740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5637b90ae810 con 0x5637b9149140 2024-06-04T05:05:29.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.989+0000 7f4e0d906640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 0x5637b91534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.989+0000 7f4e0d906640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 0x5637b91534c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36650/0 (socket says 172.21.15.70:36650) 2024-06-04T05:05:29.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.989+0000 7f4e0d906640 1 -- 172.21.15.70:0/1411758084 learned_addr learned my addr 172.21.15.70:0/1411758084 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:29.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.989+0000 7f4e0e107640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 0x5637b9148c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.989+0000 7f4e0d105640 1 --2- 172.21.15.70:0/1411758084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 0x5637b9155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.989+0000 7f4e0d906640 1 -- 172.21.15.70:0/1411758084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 msgr2=0x5637b9155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:29.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.989+0000 7f4e0d906640 1 --2- 172.21.15.70:0/1411758084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 0x5637b9155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.989+0000 7f4e0d906640 1 -- 172.21.15.70:0/1411758084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 msgr2=0x5637b9148c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:29.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.989+0000 7f4e0d906640 1 --2- 172.21.15.70:0/1411758084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 0x5637b9148c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:29.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.989+0000 7f4e0d906640 1 -- 172.21.15.70:0/1411758084 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637b8dbf210 con 0x5637b9149140 2024-06-04T05:05:29.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.989+0000 7f4e0e107640 1 --2- 172.21.15.70:0/1411758084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 0x5637b9148c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:29.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.989+0000 7f4dfeffd640 1 --2- 172.21.15.70:0/3973664810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4df40b2a90 0x7f4df40b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:29.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.989+0000 7f4dfeffd640 1 -- 172.21.15.70:0/3973664810 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f4df80e2920 con 0x7f4e08102340 2024-06-04T05:05:29.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.989+0000 7f4e0d906640 1 --2- 172.21.15.70:0/1411758084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 0x5637b91534c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4e04009cd0 tx=0x7f4e04005f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67f0d586c8e33f3d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:29.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.989+0000 7f4e0e107640 1 --2- 172.21.15.70:0/3973664810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4df40b2a90 0x7f4df40b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:29.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.990+0000 7f4df3fff640 1 -- 172.21.15.70:0/1411758084 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e04048070 con 0x5637b9149140 2024-06-04T05:05:29.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.990+0000 7f4df3fff640 1 -- 172.21.15.70:0/1411758084 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e04036070 con 0x5637b9149140 2024-06-04T05:05:29.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.990+0000 7f4df3fff640 1 -- 172.21.15.70:0/1411758084 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e04004710 con 0x5637b9149140 2024-06-04T05:05:30.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.990+0000 7f4e12881740 1 -- 172.21.15.70:0/1411758084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 msgr2=0x5637b91534c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:30.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.990+0000 7f4e12881740 1 --2- 172.21.15.70:0/1411758084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 0x5637b91534c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4e04009cd0 tx=0x7f4e04005f60 comp rx=0 tx=0).stop 2024-06-04T05:05:30.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.990+0000 7f4e0e107640 1 --2- 172.21.15.70:0/3973664810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4df40b2a90 0x7f4df40b4f50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4e00009ef0 tx=0x7f4e00005f90 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.991+0000 7f4e12881740 1 -- 172.21.15.70:0/1411758084 shutdown_connections 2024-06-04T05:05:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.991+0000 7f4e12881740 1 --2- 172.21.15.70:0/1411758084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 0x5637b9155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.991+0000 7f4e12881740 1 --2- 172.21.15.70:0/1411758084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 0x5637b91534c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.991+0000 7f4e12881740 1 --2- 172.21.15.70:0/1411758084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 0x5637b9148c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.991+0000 7f4e12881740 1 -- 172.21.15.70:0/1411758084 >> 172.21.15.70:0/1411758084 conn(0x5637b9144d30 msgr2=0x5637b9145110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.991+0000 7f4e12881740 1 -- 172.21.15.70:0/1411758084 shutdown_connections 2024-06-04T05:05:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.991+0000 7f4e12881740 1 -- 172.21.15.70:0/1411758084 wait complete. 2024-06-04T05:05:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.991+0000 7f4e12881740 1 -- start start 2024-06-04T05:05:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.991+0000 7f4e12881740 1 Processor -- start 2024-06-04T05:05:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.991+0000 7f4e12881740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 0x5637b91509d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.991+0000 7f4e12881740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 0x5637b9150f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.991+0000 7f4e12881740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 0x5637b9151450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.991+0000 7f4e12881740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5637b8d90790 con 0x5637b9148820 2024-06-04T05:05:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4e12881740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5637b9066890 con 0x5637b9153a00 2024-06-04T05:05:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4e12881740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5637b91565d0 con 0x5637b9149140 2024-06-04T05:05:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4e12881740 10 client.0 _ll_register_callbacks cb 0x5637b8dbea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4e0d906640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 0x5637b91509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4e0d906640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 0x5637b91509d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38050/0 (socket says 172.21.15.70:38050) 2024-06-04T05:05:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4e0d906640 1 -- 172.21.15.70:0/3458746268 learned_addr learned my addr 172.21.15.70:0/3458746268 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4e0d105640 1 --2- 172.21.15.70:0/3458746268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 0x5637b9150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4e0e107640 1 --2- 172.21.15.70:0/3458746268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 0x5637b9151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4e0d906640 1 -- 172.21.15.70:0/3458746268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 msgr2=0x5637b9150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4e0d906640 1 --2- 172.21.15.70:0/3458746268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 0x5637b9150f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4e0d906640 1 -- 172.21.15.70:0/3458746268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 msgr2=0x5637b9151450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4e0d906640 1 --2- 172.21.15.70:0/3458746268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 0x5637b9151450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4e0d906640 1 -- 172.21.15.70:0/3458746268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637b8e8b620 con 0x5637b9148820 2024-06-04T05:05:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4e0d906640 1 --2- 172.21.15.70:0/3458746268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 0x5637b91509d0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f4e04046040 tx=0x7f4e040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf5bca9e896fb4e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4df17fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:05:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4df17fa640 1 -- 172.21.15.70:0/3458746268 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e04048040 con 0x5637b9148820 2024-06-04T05:05:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4df17fa640 1 -- 172.21.15.70:0/3458746268 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f4e04003970 con 0x5637b9148820 2024-06-04T05:05:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4e12881740 1 -- 172.21.15.70:0/3458746268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e04036070 con 0x5637b9148820 2024-06-04T05:05:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4df17fa640 1 -- 172.21.15.70:0/3458746268 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4e04003b70 con 0x5637b9148820 2024-06-04T05:05:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4e12881740 10 client.17646 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.992+0000 7f4e12881740 1 -- 172.21.15.70:0/3458746268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5637b9162370 con 0x5637b9148820 2024-06-04T05:05:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.993+0000 7f4df17fa640 1 -- 172.21.15.70:0/3458746268 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e04036070 con 0x5637b9148820 2024-06-04T05:05:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.993+0000 7f4df17fa640 1 -- 172.21.15.70:0/3458746268 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f4e04042030 con 0x5637b9148820 2024-06-04T05:05:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.993+0000 7f4e12881740 10 client.17646 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.993+0000 7f4e12881740 10 client.17646 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:05:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.993+0000 7f4e12881740 20 client.17646 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.993+0000 7f4e12881740 1 --2- 172.21.15.70:0/3458746268 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5637b91636c0 0x5637b9165ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.993+0000 7f4e12881740 4 client.17646 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:05:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.993+0000 7f4e12881740 1 -- 172.21.15.70:0/3458746268 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637b90bcb80 con 0x5637b91636c0 2024-06-04T05:05:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.993+0000 7f4e0d105640 1 --2- 172.21.15.70:0/3458746268 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5637b91636c0 0x5637b9165ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.993+0000 7f4e0d105640 1 --2- 172.21.15.70:0/3458746268 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5637b91636c0 0x5637b9165ab0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f3149b1fcf72e2a server_cookie=f3c6e2543d52761e in_seq=0 out_seq=0 2024-06-04T05:05:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.993+0000 7f4df17fa640 10 client.17646 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.994+0000 7f4df17fa640 1 -- 172.21.15.70:0/3458746268 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5637b90bcb80 con 0x5637b91636c0 2024-06-04T05:05:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.994+0000 7f4df17fa640 10 client.17646 handle_command_reply: tid=0 2024-06-04T05:05:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.994+0000 7f4e12881740 1 client.17646 shutdown 2024-06-04T05:05:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.994+0000 7f4e12881740 1 -- 172.21.15.70:0/3458746268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 msgr2=0x5637b91509d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.994+0000 7f4e12881740 1 --2- 172.21.15.70:0/3458746268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 0x5637b91509d0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f4e04046040 tx=0x7f4e040046c0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.994+0000 7f4e12881740 1 -- 172.21.15.70:0/3458746268 shutdown_connections 2024-06-04T05:05:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.994+0000 7f4e12881740 1 --2- 172.21.15.70:0/3458746268 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5637b91636c0 0x5637b9165ab0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.994+0000 7f4e12881740 1 --2- 172.21.15.70:0/3458746268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 0x5637b9151450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.994+0000 7f4e12881740 1 --2- 172.21.15.70:0/3458746268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 0x5637b9150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.994+0000 7f4df17fa640 0 client.17646 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.994+0000 7f4e12881740 1 --2- 172.21.15.70:0/3458746268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 0x5637b91509d0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.994+0000 7f4e12881740 1 -- 172.21.15.70:0/3458746268 >> 172.21.15.70:0/3458746268 conn(0x5637b9144d30 msgr2=0x5637b914aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.994+0000 7f4e12881740 1 -- 172.21.15.70:0/3458746268 shutdown_connections 2024-06-04T05:05:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.994+0000 7f4e12881740 1 -- 172.21.15.70:0/3458746268 wait complete. 2024-06-04T05:05:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:29.995+0000 7f4e12881740 20 client.17646 trim_cache size 0 max 16384 2024-06-04T05:05:30.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.009+0000 7f4e12881740 1 Processor -- start 2024-06-04T05:05:30.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.009+0000 7f4e12881740 1 -- start start 2024-06-04T05:05:30.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4e12881740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9148820 0x5637b9151270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:30.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4e12881740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9149140 0x5637b915ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:30.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4e12881740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 0x5637b914dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:30.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4e12881740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e081b0810 con 0x5637b9149140 2024-06-04T05:05:30.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4e12881740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e081b2350 con 0x5637b9153a00 2024-06-04T05:05:30.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4e12881740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4e081b0ee0 con 0x5637b9148820 2024-06-04T05:05:30.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4e0e107640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 0x5637b914dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:30.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4e0d105640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9149140 0x5637b915ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:30.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4e0e107640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 0x5637b914dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33352/0 (socket says 172.21.15.70:33352) 2024-06-04T05:05:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4e0e107640 1 -- 172.21.15.70:0/4200892004 learned_addr learned my addr 172.21.15.70:0/4200892004 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4e0d906640 1 --2- 172.21.15.70:0/4200892004 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9148820 0x5637b9151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4e0d105640 1 -- 172.21.15.70:0/4200892004 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9148820 msgr2=0x5637b9151270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4e0d105640 1 --2- 172.21.15.70:0/4200892004 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9148820 0x5637b9151270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4e0d105640 1 -- 172.21.15.70:0/4200892004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 msgr2=0x5637b914dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4e0d105640 1 --2- 172.21.15.70:0/4200892004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 0x5637b914dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4e0d105640 1 -- 172.21.15.70:0/4200892004 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637b91404b0 con 0x5637b9149140 2024-06-04T05:05:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4e0d105640 1 --2- 172.21.15.70:0/4200892004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9149140 0x5637b915ef30 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f4df80e0040 tx=0x7f4df80e4e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab59e1d4868dc1e1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.010+0000 7f4df37fe640 1 -- 172.21.15.70:0/4200892004 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df80a93f0 con 0x5637b9149140 2024-06-04T05:05:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.011+0000 7f4df37fe640 1 -- 172.21.15.70:0/4200892004 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4df80b2370 con 0x5637b9149140 2024-06-04T05:05:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.011+0000 7f4df37fe640 1 -- 172.21.15.70:0/4200892004 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df80a5410 con 0x5637b9149140 2024-06-04T05:05:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.011+0000 7f4e12881740 1 -- 172.21.15.70:0/4200892004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9149140 msgr2=0x5637b915ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.011+0000 7f4e12881740 1 --2- 172.21.15.70:0/4200892004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9149140 0x5637b915ef30 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f4df80e0040 tx=0x7f4df80e4e30 comp rx=0 tx=0).stop 2024-06-04T05:05:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.011+0000 7f4e12881740 1 -- 172.21.15.70:0/4200892004 shutdown_connections 2024-06-04T05:05:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.011+0000 7f4e12881740 1 --2- 172.21.15.70:0/4200892004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 0x5637b914dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.011+0000 7f4e12881740 1 --2- 172.21.15.70:0/4200892004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9149140 0x5637b915ef30 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.011+0000 7f4e12881740 1 --2- 172.21.15.70:0/4200892004 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9148820 0x5637b9151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.011+0000 7f4e12881740 1 -- 172.21.15.70:0/4200892004 >> 172.21.15.70:0/4200892004 conn(0x5637b9144d30 msgr2=0x5637b915e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.011+0000 7f4e12881740 1 -- 172.21.15.70:0/4200892004 shutdown_connections 2024-06-04T05:05:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.011+0000 7f4e12881740 1 -- 172.21.15.70:0/4200892004 wait complete. 2024-06-04T05:05:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.012+0000 7f4e12881740 1 -- start start 2024-06-04T05:05:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.012+0000 7f4e12881740 1 Processor -- start 2024-06-04T05:05:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.012+0000 7f4e12881740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 0x5637b9169330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.012+0000 7f4e12881740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 0x5637b9166760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.012+0000 7f4e12881740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 0x5637b9166e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.012+0000 7f4e12881740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e081b1c80 con 0x5637b9148820 2024-06-04T05:05:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.012+0000 7f4e12881740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e081b15b0 con 0x5637b9153a00 2024-06-04T05:05:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.012+0000 7f4e12881740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4e081ae810 con 0x5637b9149140 2024-06-04T05:05:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.012+0000 7f4e12881740 10 client.0 _ll_register_callbacks cb 0x7f4e04046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.012+0000 7f4e0d906640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 0x5637b9169330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.012+0000 7f4e0d906640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 0x5637b9169330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38064/0 (socket says 172.21.15.70:38064) 2024-06-04T05:05:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.012+0000 7f4e0d906640 1 -- 172.21.15.70:0/4079620989 learned_addr learned my addr 172.21.15.70:0/4079620989 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.012+0000 7f4e0e107640 1 --2- 172.21.15.70:0/4079620989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 0x5637b9166e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.012+0000 7f4e0d105640 1 --2- 172.21.15.70:0/4079620989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 0x5637b9166760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.013+0000 7f4e0d906640 1 -- 172.21.15.70:0/4079620989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 msgr2=0x5637b9166760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.013+0000 7f4e0d906640 1 --2- 172.21.15.70:0/4079620989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 0x5637b9166760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.013+0000 7f4e0d906640 1 -- 172.21.15.70:0/4079620989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 msgr2=0x5637b9166e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.013+0000 7f4e0d906640 1 --2- 172.21.15.70:0/4079620989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 0x5637b9166e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.013+0000 7f4e0d906640 1 -- 172.21.15.70:0/4079620989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637b91ccf00 con 0x5637b9148820 2024-06-04T05:05:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.013+0000 7f4e0d105640 1 --2- 172.21.15.70:0/4079620989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 0x5637b9166760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.013+0000 7f4e0d906640 1 --2- 172.21.15.70:0/4079620989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 0x5637b9169330 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f4e04030c00 tx=0x7f4e04031bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd8500b057d788e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.013+0000 7f4df17fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:05:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.013+0000 7f4df17fa640 1 -- 172.21.15.70:0/4079620989 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e04048070 con 0x5637b9148820 2024-06-04T05:05:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.013+0000 7f4df17fa640 1 -- 172.21.15.70:0/4079620989 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f4e04003970 con 0x5637b9148820 2024-06-04T05:05:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.013+0000 7f4e12881740 1 -- 172.21.15.70:0/4079620989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5637b90b6360 con 0x5637b9148820 2024-06-04T05:05:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.013+0000 7f4df17fa640 1 -- 172.21.15.70:0/4079620989 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4e04030c30 con 0x5637b9148820 2024-06-04T05:05:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.013+0000 7f4e12881740 10 client.17652 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.013+0000 7f4e12881740 1 -- 172.21.15.70:0/4079620989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5637b91ce090 con 0x5637b9148820 2024-06-04T05:05:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.013+0000 7f4df17fa640 1 -- 172.21.15.70:0/4079620989 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x5637b90b6360 con 0x5637b9148820 2024-06-04T05:05:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.014+0000 7f4df17fa640 1 -- 172.21.15.70:0/4079620989 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f4e0405d020 con 0x5637b9148820 2024-06-04T05:05:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.014+0000 7f4e12881740 10 client.17652 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.014+0000 7f4e12881740 10 client.17652 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:05:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.014+0000 7f4e12881740 20 client.17652 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.014+0000 7f4e12881740 1 --2- 172.21.15.70:0/4079620989 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5637b91636c0 0x5637b91d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.014+0000 7f4e12881740 4 client.17652 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:05:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.014+0000 7f4e12881740 1 -- 172.21.15.70:0/4079620989 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5637b90305f0 con 0x5637b91636c0 2024-06-04T05:05:30.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.014+0000 7f4e0d105640 1 --2- 172.21.15.70:0/4079620989 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5637b91636c0 0x5637b91d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:30.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.014+0000 7f4e0d105640 1 --2- 172.21.15.70:0/4079620989 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5637b91636c0 0x5637b91d08a0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8722a49eca430779 server_cookie=a334f96b48f511e in_seq=0 out_seq=0 2024-06-04T05:05:30.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.014+0000 7f4df17fa640 10 client.17652 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.033+0000 7f4df17fa640 1 -- 172.21.15.70:0/4079620989 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5637b90305f0 con 0x5637b91636c0 2024-06-04T05:05:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.033+0000 7f4df17fa640 10 client.17652 handle_command_reply: tid=0 2024-06-04T05:05:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.033+0000 7f4e12881740 1 client.17652 shutdown 2024-06-04T05:05:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.033+0000 7f4e12881740 1 -- 172.21.15.70:0/4079620989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 msgr2=0x5637b9169330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.034+0000 7f4e12881740 1 --2- 172.21.15.70:0/4079620989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 0x5637b9169330 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f4e04030c00 tx=0x7f4e04031bb0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.034+0000 7f4e12881740 1 -- 172.21.15.70:0/4079620989 shutdown_connections 2024-06-04T05:05:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.034+0000 7f4e12881740 1 --2- 172.21.15.70:0/4079620989 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5637b91636c0 0x5637b91d08a0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.034+0000 7f4e12881740 1 --2- 172.21.15.70:0/4079620989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637b9153a00 0x5637b9166e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.034+0000 7f4e12881740 1 --2- 172.21.15.70:0/4079620989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5637b9149140 0x5637b9166760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.034+0000 7f4e12881740 1 --2- 172.21.15.70:0/4079620989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5637b9148820 0x5637b9169330 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.034+0000 7f4e12881740 1 -- 172.21.15.70:0/4079620989 >> 172.21.15.70:0/4079620989 conn(0x5637b9144d30 msgr2=0x5637b9143720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.034+0000 7f4df17fa640 0 client.17652 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.034+0000 7f4e12881740 1 -- 172.21.15.70:0/4079620989 shutdown_connections 2024-06-04T05:05:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.034+0000 7f4e12881740 1 -- 172.21.15.70:0/4079620989 wait complete. 2024-06-04T05:05:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.034+0000 7f4e12881740 20 client.17652 trim_cache size 0 max 16384 2024-06-04T05:05:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:05:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:05:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:05:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.035+0000 7f4e0fb6a640 1 -- 172.21.15.70:0/3973664810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4df40b2a90 msgr2=0x7f4df40b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.035+0000 7f4e0fb6a640 1 --2- 172.21.15.70:0/3973664810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4df40b2a90 0x7f4df40b4f50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4e00009ef0 tx=0x7f4e00005f90 comp rx=0 tx=0).stop 2024-06-04T05:05:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.035+0000 7f4e0fb6a640 1 -- 172.21.15.70:0/3973664810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e08102340 msgr2=0x7f4e0819a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.035+0000 7f4e0fb6a640 1 --2- 172.21.15.70:0/3973664810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e08102340 0x7f4e0819a440 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f4df800e9f0 tx=0x7f4df800eeb0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.035+0000 7f4e0fb6a640 1 -- 172.21.15.70:0/3973664810 shutdown_connections 2024-06-04T05:05:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.035+0000 7f4e0fb6a640 1 --2- 172.21.15.70:0/3973664810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4df40b2a90 0x7f4df40b4f50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.035+0000 7f4e0fb6a640 1 --2- 172.21.15.70:0/3973664810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e08107940 0x7f4e081a4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.035+0000 7f4e0fb6a640 1 --2- 172.21.15.70:0/3973664810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e08102340 0x7f4e0819a440 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.035+0000 7f4e0fb6a640 1 --2- 172.21.15.70:0/3973664810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e08101980 0x7f4e08199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.035+0000 7f4e0fb6a640 1 -- 172.21.15.70:0/3973664810 >> 172.21.15.70:0/3973664810 conn(0x7f4e08078ad0 msgr2=0x7f4e081045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.035+0000 7f4e0fb6a640 1 -- 172.21.15.70:0/3973664810 shutdown_connections 2024-06-04T05:05:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:30.035+0000 7f4e0fb6a640 1 -- 172.21.15.70:0/3973664810 wait complete. 2024-06-04T05:05:30.320 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:30 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:05:30.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:30 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:05:30.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:30 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:05:30.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.14 MB/sec execute 263 sec latency 827.266 ms 2024-06-04T05:05:31.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:31 smithi070 ceph-mon[32631]: pgmap v255: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 1.3 MiB/s wr, 130 op/s 2024-06-04T05:05:31.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:31 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:05:31.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:31 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:05:31.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:31 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:05:31.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:31 smithi112 ceph-mon[36453]: pgmap v255: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 1.3 MiB/s wr, 130 op/s 2024-06-04T05:05:31.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:31 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:05:31.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:31 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:05:31.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:31 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:05:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:31 smithi098 ceph-mon[35569]: pgmap v255: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 1.3 MiB/s wr, 130 op/s 2024-06-04T05:05:31.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:31 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:05:31.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:31 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:05:31.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:31 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:05:31.550 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '18106a7e', quiesce took 9.3 seconds. Will release after: 14.3 2024-06-04T05:05:31.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.13 MB/sec execute 264 sec latency 1827.393 ms 2024-06-04T05:05:32.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.507+0000 7f17912d5700 1 -- 172.21.15.70:0/3555916080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f178c102470 msgr2=0x7f178c1028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:32.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.507+0000 7f17912d5700 1 --2- 172.21.15.70:0/3555916080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f178c102470 0x7f178c1028f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f1780009990 tx=0x7f1780009ca0 comp rx=0 tx=0).stop 2024-06-04T05:05:32.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.507+0000 7f17912d5700 1 -- 172.21.15.70:0/3555916080 shutdown_connections 2024-06-04T05:05:32.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.507+0000 7f17912d5700 1 --2- 172.21.15.70:0/3555916080 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f178c102e30 0x7f178c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:32.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.507+0000 7f17912d5700 1 --2- 172.21.15.70:0/3555916080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f178c1084b0 0x7f178c108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:32.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.507+0000 7f17912d5700 1 --2- 172.21.15.70:0/3555916080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f178c102470 0x7f178c1028f0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:32.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.507+0000 7f17912d5700 1 -- 172.21.15.70:0/3555916080 >> 172.21.15.70:0/3555916080 conn(0x7f178c0784b0 msgr2=0x7f178c0788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:32.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.507+0000 7f17912d5700 1 -- 172.21.15.70:0/3555916080 shutdown_connections 2024-06-04T05:05:32.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.507+0000 7f17912d5700 1 -- 172.21.15.70:0/3555916080 wait complete. 2024-06-04T05:05:32.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.508+0000 7f17912d5700 1 Processor -- start 2024-06-04T05:05:32.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.508+0000 7f17912d5700 1 -- start start 2024-06-04T05:05:32.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.508+0000 7f17912d5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f178c102470 0x7f178c19f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:32.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.508+0000 7f17912d5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f178c102e30 0x7f178c19fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.508+0000 7f17912d5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f178c1084b0 0x7f178c1999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.508+0000 7f17912d5700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f178c1124a0 con 0x7f178c1084b0 2024-06-04T05:05:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.508+0000 7f17912d5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f178c112320 con 0x7f178c102e30 2024-06-04T05:05:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.508+0000 7f17912d5700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f178c112620 con 0x7f178c102470 2024-06-04T05:05:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.509+0000 7f178b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f178c1084b0 0x7f178c1999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.509+0000 7f178b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f178c1084b0 0x7f178c1999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38074/0 (socket says 172.21.15.70:38074) 2024-06-04T05:05:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.509+0000 7f178b7fe700 1 -- 172.21.15.70:0/4279547758 learned_addr learned my addr 172.21.15.70:0/4279547758 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.509+0000 7f178affd700 1 --2- 172.21.15.70:0/4279547758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f178c102470 0x7f178c19f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.509+0000 7f178a7fc700 1 --2- 172.21.15.70:0/4279547758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f178c102e30 0x7f178c19fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.509+0000 7f178b7fe700 1 -- 172.21.15.70:0/4279547758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f178c102470 msgr2=0x7f178c19f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.509+0000 7f178b7fe700 1 --2- 172.21.15.70:0/4279547758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f178c102470 0x7f178c19f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.509+0000 7f178b7fe700 1 -- 172.21.15.70:0/4279547758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f178c102e30 msgr2=0x7f178c19fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.509+0000 7f178b7fe700 1 --2- 172.21.15.70:0/4279547758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f178c102e30 0x7f178c19fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.509+0000 7f178b7fe700 1 -- 172.21.15.70:0/4279547758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f178c19a130 con 0x7f178c1084b0 2024-06-04T05:05:32.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.509+0000 7f178a7fc700 1 --2- 172.21.15.70:0/4279547758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f178c102e30 0x7f178c19fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:05:32.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.509+0000 7f178b7fe700 1 --2- 172.21.15.70:0/4279547758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f178c1084b0 0x7f178c1999d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f1784007ee0 tx=0x7f178400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:32.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.509+0000 7f1773fff700 1 -- 172.21.15.70:0/4279547758 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178400dbd0 con 0x7f178c1084b0 2024-06-04T05:05:32.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.509+0000 7f17912d5700 1 -- 172.21.15.70:0/4279547758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f178c19a420 con 0x7f178c1084b0 2024-06-04T05:05:32.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.509+0000 7f1773fff700 1 -- 172.21.15.70:0/4279547758 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f178400dd70 con 0x7f178c1084b0 2024-06-04T05:05:32.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.509+0000 7f1773fff700 1 -- 172.21.15.70:0/4279547758 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17840058d0 con 0x7f178c1084b0 2024-06-04T05:05:32.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.509+0000 7f17912d5700 1 -- 172.21.15.70:0/4279547758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f178c1a6690 con 0x7f178c1084b0 2024-06-04T05:05:32.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.510+0000 7f17912d5700 1 -- 172.21.15.70:0/4279547758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f178c06a230 con 0x7f178c1084b0 2024-06-04T05:05:32.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.512+0000 7f1773fff700 1 -- 172.21.15.70:0/4279547758 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f178401a020 con 0x7f178c1084b0 2024-06-04T05:05:32.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.512+0000 7f1773fff700 1 --2- 172.21.15.70:0/4279547758 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f17740b29f0 0x7f17740b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:32.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.513+0000 7f178affd700 1 --2- 172.21.15.70:0/4279547758 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f17740b29f0 0x7f17740b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:32.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.513+0000 7f1773fff700 1 -- 172.21.15.70:0/4279547758 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f17840e24a0 con 0x7f178c1084b0 2024-06-04T05:05:32.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.513+0000 7f178affd700 1 --2- 172.21.15.70:0/4279547758 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f17740b29f0 0x7f17740b4eb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f177c005950 tx=0x7f177c00a300 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:32.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.516+0000 7f1773fff700 1 -- 172.21.15.70:0/4279547758 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1784010070 con 0x7f178c1084b0 2024-06-04T05:05:32.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.12 MB/sec execute 265 sec latency 2827.457 ms 2024-06-04T05:05:32.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.763+0000 7f17912d5700 1 -- 172.21.15.70:0/4279547758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f178c02e560 con 0x7f178c1084b0 2024-06-04T05:05:32.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.764+0000 7f1773fff700 1 -- 172.21.15.70:0/4279547758 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11929 (secure 0 0 0) 0x7f17840a9e50 con 0x7f178c1084b0 2024-06-04T05:05:32.766 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:05:32.767 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:01:52.807483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:05:32.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.767+0000 7f17912d5700 1 -- 172.21.15.70:0/4279547758 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f17740b29f0 msgr2=0x7f17740b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:32.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.767+0000 7f17912d5700 1 --2- 172.21.15.70:0/4279547758 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f17740b29f0 0x7f17740b4eb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f177c005950 tx=0x7f177c00a300 comp rx=0 tx=0).stop 2024-06-04T05:05:32.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.767+0000 7f17912d5700 1 -- 172.21.15.70:0/4279547758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f178c1084b0 msgr2=0x7f178c1999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:32.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.767+0000 7f17912d5700 1 --2- 172.21.15.70:0/4279547758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f178c1084b0 0x7f178c1999d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f1784007ee0 tx=0x7f178400d3b0 comp rx=0 tx=0).stop 2024-06-04T05:05:32.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.767+0000 7f17912d5700 1 -- 172.21.15.70:0/4279547758 shutdown_connections 2024-06-04T05:05:32.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.767+0000 7f17912d5700 1 --2- 172.21.15.70:0/4279547758 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f17740b29f0 0x7f17740b4eb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:32.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.767+0000 7f17912d5700 1 --2- 172.21.15.70:0/4279547758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f178c1084b0 0x7f178c1999d0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:32.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.767+0000 7f17912d5700 1 --2- 172.21.15.70:0/4279547758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f178c102470 0x7f178c19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:32.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.767+0000 7f17912d5700 1 --2- 172.21.15.70:0/4279547758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f178c102e30 0x7f178c19fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:32.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.767+0000 7f17912d5700 1 -- 172.21.15.70:0/4279547758 >> 172.21.15.70:0/4279547758 conn(0x7f178c0784b0 msgr2=0x7f178c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:32.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.767+0000 7f17912d5700 1 -- 172.21.15.70:0/4279547758 shutdown_connections 2024-06-04T05:05:32.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:32.767+0000 7f17912d5700 1 -- 172.21.15.70:0/4279547758 wait complete. 2024-06-04T05:05:32.770 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 32 2024-06-04T05:05:33.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:33 smithi112 ceph-mon[36453]: pgmap v256: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.9 MiB/s wr, 186 op/s 2024-06-04T05:05:33.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:33 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/4279547758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:05:33.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:33 smithi098 ceph-mon[35569]: pgmap v256: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.9 MiB/s wr, 186 op/s 2024-06-04T05:05:33.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:33 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/4279547758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:05:33.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:33 smithi070 ceph-mon[32631]: pgmap v256: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.9 MiB/s wr, 186 op/s 2024-06-04T05:05:33.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:33 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/4279547758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:05:33.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.11 MB/sec execute 266 sec latency 3827.534 ms 2024-06-04T05:05:33.764 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:05:34.009 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:05:34.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.10 MB/sec execute 267 sec latency 4827.607 ms 2024-06-04T05:05:35.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:35 smithi070 ceph-mon[32631]: pgmap v257: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 1.2 MiB/s wr, 119 op/s 2024-06-04T05:05:35.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:35 smithi112 ceph-mon[36453]: pgmap v257: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 1.2 MiB/s wr, 119 op/s 2024-06-04T05:05:35.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:35 smithi098 ceph-mon[35569]: pgmap v257: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 1.2 MiB/s wr, 119 op/s 2024-06-04T05:05:35.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.10 MB/sec execute 268 sec latency 5827.664 ms 2024-06-04T05:05:36.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.841+0000 7f1ba700c700 1 -- 172.21.15.70:0/3109045071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ba0103430 msgr2=0x7f1ba01070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:36.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.841+0000 7f1ba700c700 1 --2- 172.21.15.70:0/3109045071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ba0103430 0x7f1ba01070a0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0099c0 tx=0x7f1b8c009cd0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.841+0000 7f1ba700c700 1 -- 172.21.15.70:0/3109045071 shutdown_connections 2024-06-04T05:05:36.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.841+0000 7f1ba700c700 1 --2- 172.21.15.70:0/3109045071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba01077d0 0x7f1ba010f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.841+0000 7f1ba700c700 1 --2- 172.21.15.70:0/3109045071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ba0102a80 0x7f1ba0102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.841+0000 7f1ba700c700 1 --2- 172.21.15.70:0/3109045071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ba0103430 0x7f1ba01070a0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.841+0000 7f1ba700c700 1 -- 172.21.15.70:0/3109045071 >> 172.21.15.70:0/3109045071 conn(0x7f1ba00793f0 msgr2=0x7f1ba0079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:36.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.841+0000 7f1ba700c700 1 -- 172.21.15.70:0/3109045071 shutdown_connections 2024-06-04T05:05:36.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.841+0000 7f1ba700c700 1 -- 172.21.15.70:0/3109045071 wait complete. 2024-06-04T05:05:36.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.842+0000 7f1ba700c700 1 Processor -- start 2024-06-04T05:05:36.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.842+0000 7f1ba700c700 1 -- start start 2024-06-04T05:05:36.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.842+0000 7f1ba700c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ba0102a80 0x7f1ba019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.842+0000 7f1ba700c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba0103430 0x7f1ba019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.842+0000 7f1ba700c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ba01077d0 0x7f1ba01988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.842+0000 7f1ba700c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba01124c0 con 0x7f1ba01077d0 2024-06-04T05:05:36.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.842+0000 7f1ba700c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba0112340 con 0x7f1ba0103430 2024-06-04T05:05:36.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.842+0000 7f1ba700c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba0112640 con 0x7f1ba0102a80 2024-06-04T05:05:36.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.843+0000 7f1ba55a9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ba01077d0 0x7f1ba01988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:36.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.843+0000 7f1ba55a9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ba01077d0 0x7f1ba01988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38092/0 (socket says 172.21.15.70:38092) 2024-06-04T05:05:36.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.843+0000 7f1ba55a9700 1 -- 172.21.15.70:0/601674277 learned_addr learned my addr 172.21.15.70:0/601674277 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:36.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.843+0000 7f1ba4da8700 1 --2- 172.21.15.70:0/601674277 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ba0102a80 0x7f1ba019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:36.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.843+0000 7f1b9ffff700 1 --2- 172.21.15.70:0/601674277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba0103430 0x7f1ba019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:36.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.843+0000 7f1ba55a9700 1 -- 172.21.15.70:0/601674277 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ba0102a80 msgr2=0x7f1ba019e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:36.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.843+0000 7f1ba55a9700 1 --2- 172.21.15.70:0/601674277 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ba0102a80 0x7f1ba019e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.843+0000 7f1ba55a9700 1 -- 172.21.15.70:0/601674277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba0103430 msgr2=0x7f1ba019ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:36.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.843+0000 7f1ba55a9700 1 --2- 172.21.15.70:0/601674277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba0103430 0x7f1ba019ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.843+0000 7f1ba55a9700 1 -- 172.21.15.70:0/601674277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba0199050 con 0x7f1ba01077d0 2024-06-04T05:05:36.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.843+0000 7f1b9ffff700 1 --2- 172.21.15.70:0/601674277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba0103430 0x7f1ba019ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:05:36.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.843+0000 7f1ba55a9700 1 --2- 172.21.15.70:0/601674277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ba01077d0 0x7f1ba01988f0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f1b98007ee0 tx=0x7f1b9800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:36.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.843+0000 7f1b9dffb700 1 -- 172.21.15.70:0/601674277 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9800ddd0 con 0x7f1ba01077d0 2024-06-04T05:05:36.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.843+0000 7f1b9dffb700 1 -- 172.21.15.70:0/601674277 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b98004020 con 0x7f1ba01077d0 2024-06-04T05:05:36.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.843+0000 7f1b9dffb700 1 -- 172.21.15.70:0/601674277 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b98015660 con 0x7f1ba01077d0 2024-06-04T05:05:36.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.843+0000 7f1ba700c700 1 -- 172.21.15.70:0/601674277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ba0199340 con 0x7f1ba01077d0 2024-06-04T05:05:36.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.843+0000 7f1ba700c700 1 -- 172.21.15.70:0/601674277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ba01a55b0 con 0x7f1ba01077d0 2024-06-04T05:05:36.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.846+0000 7f1b9dffb700 1 -- 172.21.15.70:0/601674277 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f1b980041d0 con 0x7f1ba01077d0 2024-06-04T05:05:36.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.846+0000 7f1b9dffb700 1 --2- 172.21.15.70:0/601674277 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1b900b29f0 0x7f1b900b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.846+0000 7f1ba4da8700 1 --2- 172.21.15.70:0/601674277 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1b900b29f0 0x7f1b900b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:36.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.846+0000 7f1b9dffb700 1 -- 172.21.15.70:0/601674277 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f1b980e2050 con 0x7f1ba01077d0 2024-06-04T05:05:36.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.847+0000 7f1ba4da8700 1 --2- 172.21.15.70:0/601674277 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1b900b29f0 0x7f1b900b4eb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0107360 tx=0x7f1b9400b3c0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:36.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.847+0000 7f1bb18a1b80 1 Processor -- start 2024-06-04T05:05:36.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.847+0000 7f1bb18a1b80 1 -- start start 2024-06-04T05:05:36.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.847+0000 7f1bb18a1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40cee320 0x555b40cee700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.847+0000 7f1bb18a1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40ce88a0 0x555b40ce8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.847+0000 7f1bb18a1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 0x555b40cf5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.847+0000 7f1bb18a1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555b40c46d70 con 0x555b40cee320 2024-06-04T05:05:36.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.847+0000 7f1bb18a1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555b40c36910 con 0x555b40ce88a0 2024-06-04T05:05:36.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.847+0000 7f1bb18a1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555b40c3a950 con 0x555b40ce9260 2024-06-04T05:05:36.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.847+0000 7f1ba4da8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40cee320 0x555b40cee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:36.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.847+0000 7f1ba4da8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40cee320 0x555b40cee700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38098/0 (socket says 172.21.15.70:38098) 2024-06-04T05:05:36.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.847+0000 7f1ba4da8700 1 -- 172.21.15.70:0/269731838 learned_addr learned my addr 172.21.15.70:0/269731838 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:36.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.847+0000 7f1b9ffff700 1 --2- 172.21.15.70:0/269731838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40ce88a0 0x555b40ce8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:36.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.847+0000 7f1ba55a9700 1 --2- 172.21.15.70:0/269731838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 0x555b40cf5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:36.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.848+0000 7f1ba4da8700 1 -- 172.21.15.70:0/269731838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 msgr2=0x555b40cf5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:36.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.848+0000 7f1ba4da8700 1 --2- 172.21.15.70:0/269731838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 0x555b40cf5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.848+0000 7f1ba4da8700 1 -- 172.21.15.70:0/269731838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40ce88a0 msgr2=0x555b40ce8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:36.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.848+0000 7f1ba4da8700 1 --2- 172.21.15.70:0/269731838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40ce88a0 0x555b40ce8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.848+0000 7f1ba4da8700 1 -- 172.21.15.70:0/269731838 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b40bdffb0 con 0x555b40cee320 2024-06-04T05:05:36.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.848+0000 7f1b9ffff700 1 --2- 172.21.15.70:0/269731838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40ce88a0 0x555b40ce8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:05:36.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.848+0000 7f1ba4da8700 1 --2- 172.21.15.70:0/269731838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40cee320 0x555b40cee700 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f1b94005d70 tx=0x7f1b94013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f61437837a8963 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:36.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.848+0000 7f1b8affd700 1 -- 172.21.15.70:0/269731838 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b94011ad0 con 0x555b40cee320 2024-06-04T05:05:36.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.848+0000 7f1b8affd700 1 -- 172.21.15.70:0/269731838 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b94003e90 con 0x555b40cee320 2024-06-04T05:05:36.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.848+0000 7f1b8affd700 1 -- 172.21.15.70:0/269731838 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9401c3b0 con 0x555b40cee320 2024-06-04T05:05:36.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.849+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/269731838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40cee320 msgr2=0x555b40cee700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:36.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.849+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/269731838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40cee320 0x555b40cee700 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f1b94005d70 tx=0x7f1b94013000 comp rx=0 tx=0).stop 2024-06-04T05:05:36.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.849+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/269731838 shutdown_connections 2024-06-04T05:05:36.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.849+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/269731838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 0x555b40cf5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.849+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/269731838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40cee320 0x555b40cee700 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.849+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/269731838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40ce88a0 0x555b40ce8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.849+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/269731838 >> 172.21.15.70:0/269731838 conn(0x555b40c14240 msgr2=0x555b40ce33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:36.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.849+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/269731838 shutdown_connections 2024-06-04T05:05:36.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.849+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/269731838 wait complete. 2024-06-04T05:05:36.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.849+0000 7f1bb18a1b80 1 -- start start 2024-06-04T05:05:36.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.849+0000 7f1bb18a1b80 1 Processor -- start 2024-06-04T05:05:36.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1bb18a1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40ce88a0 0x555b40cf4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1bb18a1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 0x555b40cf1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1bb18a1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40cee320 0x555b40cf2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1bb18a1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555b40b3cfa0 con 0x555b40ce88a0 2024-06-04T05:05:36.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1bb18a1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555b40b9eda0 con 0x555b40cee320 2024-06-04T05:05:36.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1bb18a1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555b40cf56e0 con 0x555b40ce9260 2024-06-04T05:05:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1bb18a1b80 10 client.0 _ll_register_callbacks cb 0x555b40bd1950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1ba4da8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40ce88a0 0x555b40cf4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1ba4da8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40ce88a0 0x555b40cf4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38114/0 (socket says 172.21.15.70:38114) 2024-06-04T05:05:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1ba4da8700 1 -- 172.21.15.70:0/812740174 learned_addr learned my addr 172.21.15.70:0/812740174 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1b9ffff700 1 --2- 172.21.15.70:0/812740174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 0x555b40cf1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1ba55a9700 1 --2- 172.21.15.70:0/812740174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40cee320 0x555b40cf2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1ba4da8700 1 -- 172.21.15.70:0/812740174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 msgr2=0x555b40cf1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1ba4da8700 1 --2- 172.21.15.70:0/812740174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 0x555b40cf1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1ba4da8700 1 -- 172.21.15.70:0/812740174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40cee320 msgr2=0x555b40cf2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1ba4da8700 1 --2- 172.21.15.70:0/812740174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40cee320 0x555b40cf2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1ba4da8700 1 -- 172.21.15.70:0/812740174 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b40b868d0 con 0x555b40ce88a0 2024-06-04T05:05:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1ba55a9700 1 --2- 172.21.15.70:0/812740174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40cee320 0x555b40cf2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:05:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1ba4da8700 1 --2- 172.21.15.70:0/812740174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40ce88a0 0x555b40cf4cf0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f1b9401eb80 tx=0x7f1b9401ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=370b66173e8a450e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1b7ffff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:05:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1b7ffff700 1 -- 172.21.15.70:0/812740174 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b94036060 con 0x555b40ce88a0 2024-06-04T05:05:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1b7ffff700 1 -- 172.21.15.70:0/812740174 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f1b9401e630 con 0x555b40ce88a0 2024-06-04T05:05:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/812740174 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b94003e90 con 0x555b40ce88a0 2024-06-04T05:05:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1b7ffff700 1 -- 172.21.15.70:0/812740174 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1b940118d0 con 0x555b40ce88a0 2024-06-04T05:05:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1bb18a1b80 10 client.17670 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.850+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/812740174 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555b40bf0760 con 0x555b40ce88a0 2024-06-04T05:05:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.851+0000 7f1b7ffff700 1 -- 172.21.15.70:0/812740174 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b94003e90 con 0x555b40ce88a0 2024-06-04T05:05:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.851+0000 7f1b7ffff700 1 -- 172.21.15.70:0/812740174 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f1b94026a00 con 0x555b40ce88a0 2024-06-04T05:05:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.851+0000 7f1bb18a1b80 10 client.17670 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.851+0000 7f1bb18a1b80 10 client.17670 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:05:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.851+0000 7f1bb18a1b80 20 client.17670 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.851+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/812740174 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555b40d02190 0x555b40d04580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.851+0000 7f1bb18a1b80 4 client.17670 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:05:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.851+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/812740174 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b40c48cc0 con 0x555b40d02190 2024-06-04T05:05:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.851+0000 7f1b9ffff700 1 --2- 172.21.15.70:0/812740174 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555b40d02190 0x555b40d04580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.852+0000 7f1b7ffff700 10 client.17670 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:05:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.852+0000 7f1b9ffff700 1 --2- 172.21.15.70:0/812740174 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555b40d02190 0x555b40d04580 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=20194c9a398396bd server_cookie=258be8ba72895f39 in_seq=0 out_seq=0 2024-06-04T05:05:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.852+0000 7f1b7ffff700 1 -- 172.21.15.70:0/812740174 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555b40c48cc0 con 0x555b40d02190 2024-06-04T05:05:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.852+0000 7f1b7ffff700 10 client.17670 handle_command_reply: tid=0 2024-06-04T05:05:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.852+0000 7f1bb18a1b80 1 client.17670 shutdown 2024-06-04T05:05:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.852+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/812740174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40ce88a0 msgr2=0x555b40cf4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.852+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/812740174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40ce88a0 0x555b40cf4cf0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f1b9401eb80 tx=0x7f1b9401ee10 comp rx=0 tx=0).stop 2024-06-04T05:05:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.853+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/812740174 shutdown_connections 2024-06-04T05:05:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.853+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/812740174 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555b40d02190 0x555b40d04580 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.853+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/812740174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40cee320 0x555b40cf2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.853+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/812740174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40ce88a0 0x555b40cf4cf0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.853+0000 7f1b7ffff700 0 client.17670 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:05:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.853+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/812740174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 0x555b40cf1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.853+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/812740174 >> 172.21.15.70:0/812740174 conn(0x555b40c14240 msgr2=0x555b40ce2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.853+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/812740174 shutdown_connections 2024-06-04T05:05:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.853+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/812740174 wait complete. 2024-06-04T05:05:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.853+0000 7f1bb18a1b80 20 client.17670 trim_cache size 0 max 16384 2024-06-04T05:05:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.867+0000 7f1bb18a1b80 1 Processor -- start 2024-06-04T05:05:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.867+0000 7f1bb18a1b80 1 -- start start 2024-06-04T05:05:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1bb18a1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40ce88a0 0x555b40cf4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1bb18a1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 0x555b40cec5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1bb18a1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40cee320 0x555b40ce3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1bb18a1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba01ade70 con 0x555b40cee320 2024-06-04T05:05:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1bb18a1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba01af9b0 con 0x555b40ce88a0 2024-06-04T05:05:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1bb18a1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba01ae540 con 0x555b40ce9260 2024-06-04T05:05:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1ba55a9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40cee320 0x555b40ce3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1ba55a9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40cee320 0x555b40ce3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38120/0 (socket says 172.21.15.70:38120) 2024-06-04T05:05:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1ba55a9700 1 -- 172.21.15.70:0/2953705084 learned_addr learned my addr 172.21.15.70:0/2953705084 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:36.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1ba4da8700 1 --2- 172.21.15.70:0/2953705084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40ce88a0 0x555b40cf4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:36.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1b9ffff700 1 --2- 172.21.15.70:0/2953705084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 0x555b40cec5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:36.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1ba55a9700 1 -- 172.21.15.70:0/2953705084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 msgr2=0x555b40cec5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:36.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1ba55a9700 1 --2- 172.21.15.70:0/2953705084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 0x555b40cec5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1ba55a9700 1 -- 172.21.15.70:0/2953705084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40ce88a0 msgr2=0x555b40cf4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:36.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1ba55a9700 1 --2- 172.21.15.70:0/2953705084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40ce88a0 0x555b40cf4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1ba55a9700 1 -- 172.21.15.70:0/2953705084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b40cf4930 con 0x555b40cee320 2024-06-04T05:05:36.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1b9ffff700 1 --2- 172.21.15.70:0/2953705084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 0x555b40cec5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:05:36.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1ba55a9700 1 --2- 172.21.15.70:0/2953705084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40cee320 0x555b40ce3270 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f1b980dfd70 tx=0x7f1b980a9ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c6f3baf03cd4ac server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:36.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1b8a7fc700 1 -- 172.21.15.70:0/2953705084 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b980b2160 con 0x555b40cee320 2024-06-04T05:05:36.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1b8a7fc700 1 -- 172.21.15.70:0/2953705084 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b980a5590 con 0x555b40cee320 2024-06-04T05:05:36.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.868+0000 7f1b8a7fc700 1 -- 172.21.15.70:0/2953705084 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b98004e30 con 0x555b40cee320 2024-06-04T05:05:36.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.869+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/2953705084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40cee320 msgr2=0x555b40ce3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:36.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.869+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/2953705084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40cee320 0x555b40ce3270 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f1b980dfd70 tx=0x7f1b980a9ad0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.869+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/2953705084 shutdown_connections 2024-06-04T05:05:36.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.869+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/2953705084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40cee320 0x555b40ce3270 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.869+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/2953705084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40ce88a0 0x555b40cf4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.869+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/2953705084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 0x555b40cec5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.869+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/2953705084 >> 172.21.15.70:0/2953705084 conn(0x555b40c14240 msgr2=0x555b40ce7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:36.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.869+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/2953705084 shutdown_connections 2024-06-04T05:05:36.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.869+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/2953705084 wait complete. 2024-06-04T05:05:36.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.869+0000 7f1bb18a1b80 1 -- start start 2024-06-04T05:05:36.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1bb18a1b80 1 Processor -- start 2024-06-04T05:05:36.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1bb18a1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40ce88a0 0x555b40d09100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1bb18a1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 0x555b40d09640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1bb18a1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40cee320 0x555b40d6ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1bb18a1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba01af2e0 con 0x555b40ce88a0 2024-06-04T05:05:36.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1bb18a1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba01aec10 con 0x555b40cee320 2024-06-04T05:05:36.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1bb18a1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba01abe30 con 0x555b40ce9260 2024-06-04T05:05:36.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1bb18a1b80 10 client.0 _ll_register_callbacks cb 0x555b40bd1950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:36.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1ba4da8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40ce88a0 0x555b40d09100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:36.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1ba4da8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40ce88a0 0x555b40d09100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38126/0 (socket says 172.21.15.70:38126) 2024-06-04T05:05:36.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1ba4da8700 1 -- 172.21.15.70:0/1379771985 learned_addr learned my addr 172.21.15.70:0/1379771985 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:36.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1b9ffff700 1 --2- 172.21.15.70:0/1379771985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 0x555b40d09640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:36.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1ba4da8700 1 -- 172.21.15.70:0/1379771985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 msgr2=0x555b40d09640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:36.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1ba4da8700 1 --2- 172.21.15.70:0/1379771985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 0x555b40d09640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1ba4da8700 1 -- 172.21.15.70:0/1379771985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40cee320 msgr2=0x555b40d6ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:36.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1ba4da8700 1 --2- 172.21.15.70:0/1379771985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40cee320 0x555b40d6ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1ba4da8700 1 -- 172.21.15.70:0/1379771985 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b40d6f010 con 0x555b40ce88a0 2024-06-04T05:05:36.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1ba4da8700 1 --2- 172.21.15.70:0/1379771985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40ce88a0 0x555b40d09100 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f1b9401eb30 tx=0x7f1b940396c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14791116debe0c4d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:36.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.870+0000 7f1b7ffff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:05:36.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.871+0000 7f1b7ffff700 1 -- 172.21.15.70:0/1379771985 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b94036070 con 0x555b40ce88a0 2024-06-04T05:05:36.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.871+0000 7f1b7ffff700 1 -- 172.21.15.70:0/1379771985 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f1b94011c80 con 0x555b40ce88a0 2024-06-04T05:05:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.871+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/1379771985 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b980a5590 con 0x555b40ce88a0 2024-06-04T05:05:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.871+0000 7f1b7ffff700 1 -- 172.21.15.70:0/1379771985 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1b94049050 con 0x555b40ce88a0 2024-06-04T05:05:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.871+0000 7f1bb18a1b80 10 client.17676 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.871+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/1379771985 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555b40d70260 con 0x555b40ce88a0 2024-06-04T05:05:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.871+0000 7f1b7ffff700 1 -- 172.21.15.70:0/1379771985 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b980a5590 con 0x555b40ce88a0 2024-06-04T05:05:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.871+0000 7f1b7ffff700 1 -- 172.21.15.70:0/1379771985 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f1b94005460 con 0x555b40ce88a0 2024-06-04T05:05:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.871+0000 7f1bb18a1b80 10 client.17676 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.871+0000 7f1bb18a1b80 10 client.17676 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:05:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.871+0000 7f1bb18a1b80 20 client.17676 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.871+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/1379771985 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555b40d708b0 0x555b40d72cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.871+0000 7f1bb18a1b80 4 client.17676 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:05:36.030 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:05:36.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.871+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/1379771985 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555b40c424a0 con 0x555b40d708b0 2024-06-04T05:05:36.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.871+0000 7f1b9ffff700 1 --2- 172.21.15.70:0/1379771985 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555b40d708b0 0x555b40d72cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:36.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.872+0000 7f1b9ffff700 1 --2- 172.21.15.70:0/1379771985 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555b40d708b0 0x555b40d72cd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=eff81c53d43af491 server_cookie=5e5d9033a7d7b26d in_seq=0 out_seq=0 2024-06-04T05:05:36.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.872+0000 7f1b7ffff700 10 client.17676 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:05:36.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.872+0000 7f1b7ffff700 1 -- 172.21.15.70:0/1379771985 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555b40c424a0 con 0x555b40d708b0 2024-06-04T05:05:36.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.872+0000 7f1b7ffff700 10 client.17676 handle_command_reply: tid=0 2024-06-04T05:05:36.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.872+0000 7f1bb18a1b80 1 client.17676 shutdown 2024-06-04T05:05:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.872+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/1379771985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40ce88a0 msgr2=0x555b40d09100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.872+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/1379771985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40ce88a0 0x555b40d09100 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f1b9401eb30 tx=0x7f1b940396c0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.872+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/1379771985 shutdown_connections 2024-06-04T05:05:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.872+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/1379771985 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x555b40d708b0 0x555b40d72cd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.872+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/1379771985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b40cee320 0x555b40d6ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.872+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/1379771985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555b40ce88a0 0x555b40d09100 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.872+0000 7f1b7ffff700 0 client.17676 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:05:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.872+0000 7f1bb18a1b80 1 --2- 172.21.15.70:0/1379771985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555b40ce9260 0x555b40d09640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.872+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/1379771985 >> 172.21.15.70:0/1379771985 conn(0x555b40c14240 msgr2=0x555b40ce8120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.872+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/1379771985 shutdown_connections 2024-06-04T05:05:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.872+0000 7f1bb18a1b80 1 -- 172.21.15.70:0/1379771985 wait complete. 2024-06-04T05:05:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.872+0000 7f1bb18a1b80 20 client.17676 trim_cache size 0 max 16384 2024-06-04T05:05:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.873+0000 7f1ba700c700 1 -- 172.21.15.70:0/601674277 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1b900b29f0 msgr2=0x7f1b900b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:36.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.873+0000 7f1ba700c700 1 --2- 172.21.15.70:0/601674277 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1b900b29f0 0x7f1b900b4eb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0107360 tx=0x7f1b9400b3c0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.873+0000 7f1ba700c700 1 -- 172.21.15.70:0/601674277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ba01077d0 msgr2=0x7f1ba01988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:36.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.873+0000 7f1ba700c700 1 --2- 172.21.15.70:0/601674277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ba01077d0 0x7f1ba01988f0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f1b98007ee0 tx=0x7f1b9800d3b0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.873+0000 7f1ba700c700 1 -- 172.21.15.70:0/601674277 shutdown_connections 2024-06-04T05:05:36.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.873+0000 7f1ba700c700 1 --2- 172.21.15.70:0/601674277 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1b900b29f0 0x7f1b900b4eb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.873+0000 7f1ba700c700 1 --2- 172.21.15.70:0/601674277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ba01077d0 0x7f1ba01988f0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.873+0000 7f1ba700c700 1 --2- 172.21.15.70:0/601674277 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ba0102a80 0x7f1ba019e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.873+0000 7f1ba700c700 1 --2- 172.21.15.70:0/601674277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba0103430 0x7f1ba019ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:36.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.873+0000 7f1ba700c700 1 -- 172.21.15.70:0/601674277 >> 172.21.15.70:0/601674277 conn(0x7f1ba00793f0 msgr2=0x7f1ba0100d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:36.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.873+0000 7f1ba700c700 1 -- 172.21.15.70:0/601674277 shutdown_connections 2024-06-04T05:05:36.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:35.873+0000 7f1ba700c700 1 -- 172.21.15.70:0/601674277 wait complete. 2024-06-04T05:05:36.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.09 MB/sec execute 269 sec latency 6827.737 ms 2024-06-04T05:05:36.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:05:37.175 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:05:37.407 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:37 smithi070 ceph-mon[32631]: pgmap v258: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-06-04T05:05:37.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:37 smithi112 ceph-mon[36453]: pgmap v258: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-06-04T05:05:37.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:37 smithi098 ceph-mon[35569]: pgmap v258: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-06-04T05:05:37.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.08 MB/sec execute 270 sec latency 7827.777 ms 2024-06-04T05:05:38.727 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.07 MB/sec execute 271 sec latency 8827.816 ms 2024-06-04T05:05:38.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.951+0000 7fb6fa95b700 1 -- 172.21.15.70:0/1678500142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6f41041a0 msgr2=0x7fb6f41081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:38.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.951+0000 7fb6fa95b700 1 --2- 172.21.15.70:0/1678500142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6f41041a0 0x7fb6f41081c0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb6e80099c0 tx=0x7fb6e8009cd0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.951+0000 7fb6fa95b700 1 -- 172.21.15.70:0/1678500142 shutdown_connections 2024-06-04T05:05:38.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.951+0000 7fb6fa95b700 1 --2- 172.21.15.70:0/1678500142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f41088f0 0x7fb6f4110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.951+0000 7fb6fa95b700 1 --2- 172.21.15.70:0/1678500142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6f41037f0 0x7fb6f4103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.951+0000 7fb6fa95b700 1 --2- 172.21.15.70:0/1678500142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6f41041a0 0x7fb6f41081c0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.951+0000 7fb6fa95b700 1 -- 172.21.15.70:0/1678500142 >> 172.21.15.70:0/1678500142 conn(0x7fb6f407a6a0 msgr2=0x7fb6f407aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:38.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.951+0000 7fb6fa95b700 1 -- 172.21.15.70:0/1678500142 shutdown_connections 2024-06-04T05:05:38.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.952+0000 7fb6fa95b700 1 -- 172.21.15.70:0/1678500142 wait complete. 2024-06-04T05:05:38.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.952+0000 7fb6fa95b700 1 Processor -- start 2024-06-04T05:05:38.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.952+0000 7fb6fa95b700 1 -- start start 2024-06-04T05:05:38.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.952+0000 7fb6fa95b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6f41037f0 0x7fb6f4075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.952+0000 7fb6fa95b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f41041a0 0x7fb6f4075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.952+0000 7fb6fa95b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6f41088f0 0x7fb6f40762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.952+0000 7fb6fa95b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f4113600 con 0x7fb6f41088f0 2024-06-04T05:05:38.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.952+0000 7fb6fa95b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f4113480 con 0x7fb6f41041a0 2024-06-04T05:05:38.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.952+0000 7fb6fa95b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f4113780 con 0x7fb6f41037f0 2024-06-04T05:05:38.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.953+0000 7fb6f8ef8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6f41088f0 0x7fb6f40762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.953+0000 7fb6f8ef8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6f41088f0 0x7fb6f40762c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56332/0 (socket says 172.21.15.70:56332) 2024-06-04T05:05:38.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.953+0000 7fb6f8ef8700 1 -- 172.21.15.70:0/716192363 learned_addr learned my addr 172.21.15.70:0/716192363 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:38.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.953+0000 7fb6f3fff700 1 --2- 172.21.15.70:0/716192363 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6f41037f0 0x7fb6f4075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.953+0000 7fb6f37fe700 1 --2- 172.21.15.70:0/716192363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f41041a0 0x7fb6f4075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.953+0000 7fb6f8ef8700 1 -- 172.21.15.70:0/716192363 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6f41037f0 msgr2=0x7fb6f4075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:38.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.953+0000 7fb6f8ef8700 1 --2- 172.21.15.70:0/716192363 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6f41037f0 0x7fb6f4075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.953+0000 7fb6f8ef8700 1 -- 172.21.15.70:0/716192363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f41041a0 msgr2=0x7fb6f4075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:38.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.953+0000 7fb6f8ef8700 1 --2- 172.21.15.70:0/716192363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f41041a0 0x7fb6f4075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.953+0000 7fb6f8ef8700 1 -- 172.21.15.70:0/716192363 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6f41a8060 con 0x7fb6f41088f0 2024-06-04T05:05:38.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.953+0000 7fb6f8ef8700 1 --2- 172.21.15.70:0/716192363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6f41088f0 0x7fb6f40762c0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec00b970 tx=0x7fb6ec00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:38.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.953+0000 7fb6f17fa700 1 -- 172.21.15.70:0/716192363 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec00d860 con 0x7fb6f41088f0 2024-06-04T05:05:38.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.953+0000 7fb6fa95b700 1 -- 172.21.15.70:0/716192363 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6f41a8350 con 0x7fb6f41088f0 2024-06-04T05:05:38.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.953+0000 7fb6f17fa700 1 -- 172.21.15.70:0/716192363 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb6ec00e430 con 0x7fb6f41088f0 2024-06-04T05:05:38.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.953+0000 7fb6f17fa700 1 -- 172.21.15.70:0/716192363 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec017660 con 0x7fb6f41088f0 2024-06-04T05:05:38.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.953+0000 7fb6fa95b700 1 -- 172.21.15.70:0/716192363 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6f41a8860 con 0x7fb6f41088f0 2024-06-04T05:05:38.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.956+0000 7fb6f17fa700 1 -- 172.21.15.70:0/716192363 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fb6ec00da00 con 0x7fb6f41088f0 2024-06-04T05:05:38.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.956+0000 7fb6f17fa700 1 --2- 172.21.15.70:0/716192363 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb6dc0b29f0 0x7fb6dc0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.956+0000 7fb6f3fff700 1 --2- 172.21.15.70:0/716192363 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb6dc0b29f0 0x7fb6dc0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.957+0000 7fb6f17fa700 1 -- 172.21.15.70:0/716192363 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fb6ec0e2280 con 0x7fb6f41088f0 2024-06-04T05:05:38.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.957+0000 7fb7051f0b80 1 Processor -- start 2024-06-04T05:05:38.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.957+0000 7fb6f3fff700 1 --2- 172.21.15.70:0/716192363 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb6dc0b29f0 0x7fb6dc0b4eb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb6e4005950 tx=0x7fb6e400a300 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:38.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.957+0000 7fb7051f0b80 1 -- start start 2024-06-04T05:05:38.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.957+0000 7fb7051f0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b3c860 0x561e40b3cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.957+0000 7fb7051f0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3d210 0x561e40b40940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.957+0000 7fb7051f0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b40f60 0x561e40b413a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.957+0000 7fb7051f0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561e40a9bc90 con 0x561e40b40f60 2024-06-04T05:05:38.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.957+0000 7fb7051f0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561e40a8b830 con 0x561e40b3d210 2024-06-04T05:05:38.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.957+0000 7fb7051f0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561e40a8f870 con 0x561e40b3c860 2024-06-04T05:05:38.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.957+0000 7fb6f8ef8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b40f60 0x561e40b413a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.957+0000 7fb6f8ef8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b40f60 0x561e40b413a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56348/0 (socket says 172.21.15.70:56348) 2024-06-04T05:05:38.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.957+0000 7fb6f8ef8700 1 -- 172.21.15.70:0/3917752123 learned_addr learned my addr 172.21.15.70:0/3917752123 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:38.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.957+0000 7fb6f3fff700 1 --2- 172.21.15.70:0/3917752123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b3c860 0x561e40b3cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.957+0000 7fb6f37fe700 1 --2- 172.21.15.70:0/3917752123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3d210 0x561e40b40940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.958+0000 7fb6f8ef8700 1 -- 172.21.15.70:0/3917752123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b3c860 msgr2=0x561e40b3cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:38.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.958+0000 7fb6f8ef8700 1 --2- 172.21.15.70:0/3917752123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b3c860 0x561e40b3cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.958+0000 7fb6f8ef8700 1 -- 172.21.15.70:0/3917752123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3d210 msgr2=0x561e40b40940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:38.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.958+0000 7fb6f8ef8700 1 --2- 172.21.15.70:0/3917752123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3d210 0x561e40b40940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.958+0000 7fb6f8ef8700 1 -- 172.21.15.70:0/3917752123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e409db8d0 con 0x561e40b40f60 2024-06-04T05:05:38.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.958+0000 7fb6f8ef8700 1 --2- 172.21.15.70:0/3917752123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b40f60 0x561e40b413a0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec0a55b0 tx=0x7fb6ec0a9cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dbebb4872f8ff17 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:38.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.958+0000 7fb6da7fc700 1 -- 172.21.15.70:0/3917752123 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec0b2390 con 0x561e40b40f60 2024-06-04T05:05:38.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.958+0000 7fb6da7fc700 1 -- 172.21.15.70:0/3917752123 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb6ec0dfc20 con 0x561e40b40f60 2024-06-04T05:05:38.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.958+0000 7fb6da7fc700 1 -- 172.21.15.70:0/3917752123 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec015300 con 0x561e40b40f60 2024-06-04T05:05:38.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.959+0000 7fb7051f0b80 1 -- 172.21.15.70:0/3917752123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b40f60 msgr2=0x561e40b413a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:38.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.959+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/3917752123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b40f60 0x561e40b413a0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec0a55b0 tx=0x7fb6ec0a9cb0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.959+0000 7fb7051f0b80 1 -- 172.21.15.70:0/3917752123 shutdown_connections 2024-06-04T05:05:38.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.959+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/3917752123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b40f60 0x561e40b413a0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.959+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/3917752123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b3c860 0x561e40b3cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.959+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/3917752123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3d210 0x561e40b40940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.959+0000 7fb7051f0b80 1 -- 172.21.15.70:0/3917752123 >> 172.21.15.70:0/3917752123 conn(0x561e40a6a080 msgr2=0x561e40b36e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:38.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.959+0000 7fb7051f0b80 1 -- 172.21.15.70:0/3917752123 shutdown_connections 2024-06-04T05:05:38.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.959+0000 7fb7051f0b80 1 -- 172.21.15.70:0/3917752123 wait complete. 2024-06-04T05:05:38.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.959+0000 7fb7051f0b80 1 -- start start 2024-06-04T05:05:38.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.959+0000 7fb7051f0b80 1 Processor -- start 2024-06-04T05:05:38.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb7051f0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3c860 0x561e40b48350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb7051f0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b3d210 0x561e40b45650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb7051f0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b40f60 0x561e40b45b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb7051f0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561e40991fa0 con 0x561e40b3d210 2024-06-04T05:05:38.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb7051f0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561e40a2d7c0 con 0x561e40b3c860 2024-06-04T05:05:38.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb7051f0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561e40b49120 con 0x561e40b40f60 2024-06-04T05:05:38.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb7051f0b80 10 client.0 _ll_register_callbacks cb 0x561e40a26920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:38.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb6f37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b3d210 0x561e40b45650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb6f37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b3d210 0x561e40b45650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56364/0 (socket says 172.21.15.70:56364) 2024-06-04T05:05:38.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb6f37fe700 1 -- 172.21.15.70:0/4011567977 learned_addr learned my addr 172.21.15.70:0/4011567977 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:38.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb6f3fff700 1 --2- 172.21.15.70:0/4011567977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3c860 0x561e40b48350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb6f8ef8700 1 --2- 172.21.15.70:0/4011567977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b40f60 0x561e40b45b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb6f37fe700 1 -- 172.21.15.70:0/4011567977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b40f60 msgr2=0x561e40b45b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:38.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb6f37fe700 1 --2- 172.21.15.70:0/4011567977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b40f60 0x561e40b45b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb6f37fe700 1 -- 172.21.15.70:0/4011567977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3c860 msgr2=0x561e40b48350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:38.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb6f37fe700 1 --2- 172.21.15.70:0/4011567977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3c860 0x561e40b48350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb6f37fe700 1 -- 172.21.15.70:0/4011567977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e40a2a690 con 0x561e40b3d210 2024-06-04T05:05:38.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb6f37fe700 1 --2- 172.21.15.70:0/4011567977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b3d210 0x561e40b45650 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fb6e80099c0 tx=0x7fb6e8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=837878cfa6a76ce1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:38.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb6d37fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:05:38.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb6d37fe700 1 -- 172.21.15.70:0/4011567977 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8028060 con 0x561e40b3d210 2024-06-04T05:05:38.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb6d37fe700 1 -- 172.21.15.70:0/4011567977 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fb6e800bba0 con 0x561e40b3d210 2024-06-04T05:05:38.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.960+0000 7fb7051f0b80 1 -- 172.21.15.70:0/4011567977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6ec0dfc20 con 0x561e40b3d210 2024-06-04T05:05:38.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.961+0000 7fb6d37fe700 1 -- 172.21.15.70:0/4011567977 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb6e800bda0 con 0x561e40b3d210 2024-06-04T05:05:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.961+0000 7fb7051f0b80 10 client.17688 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.961+0000 7fb7051f0b80 1 -- 172.21.15.70:0/4011567977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561e40b537c0 con 0x561e40b3d210 2024-06-04T05:05:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.961+0000 7fb6d37fe700 1 -- 172.21.15.70:0/4011567977 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb6ec0dfc20 con 0x561e40b3d210 2024-06-04T05:05:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.961+0000 7fb6d37fe700 1 -- 172.21.15.70:0/4011567977 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fb6e800f580 con 0x561e40b3d210 2024-06-04T05:05:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.961+0000 7fb7051f0b80 10 client.17688 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.961+0000 7fb7051f0b80 10 client.17688 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:05:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.961+0000 7fb7051f0b80 20 client.17688 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.961+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/4011567977 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x561e40b55e00 0x561e40b581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.961+0000 7fb7051f0b80 4 client.17688 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:05:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.961+0000 7fb7051f0b80 1 -- 172.21.15.70:0/4011567977 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e40b089e0 con 0x561e40b55e00 2024-06-04T05:05:38.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.961+0000 7fb6f3fff700 1 --2- 172.21.15.70:0/4011567977 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x561e40b55e00 0x561e40b581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.962+0000 7fb6d37fe700 10 client.17688 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:05:38.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.962+0000 7fb6f3fff700 1 --2- 172.21.15.70:0/4011567977 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x561e40b55e00 0x561e40b581f0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c350192572088c86 server_cookie=f6e41268935c9b17 in_seq=0 out_seq=0 2024-06-04T05:05:38.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.962+0000 7fb6d37fe700 1 -- 172.21.15.70:0/4011567977 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561e40b089e0 con 0x561e40b55e00 2024-06-04T05:05:38.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.962+0000 7fb6d37fe700 10 client.17688 handle_command_reply: tid=0 2024-06-04T05:05:38.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.963+0000 7fb7051f0b80 1 client.17688 shutdown 2024-06-04T05:05:38.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.963+0000 7fb7051f0b80 1 -- 172.21.15.70:0/4011567977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b3d210 msgr2=0x561e40b45650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:38.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.963+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/4011567977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b3d210 0x561e40b45650 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fb6e80099c0 tx=0x7fb6e8005800 comp rx=0 tx=0).stop 2024-06-04T05:05:38.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.963+0000 7fb7051f0b80 1 -- 172.21.15.70:0/4011567977 shutdown_connections 2024-06-04T05:05:38.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.963+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/4011567977 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x561e40b55e00 0x561e40b581f0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.963+0000 7fb6d37fe700 0 client.17688 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:05:38.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.963+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/4011567977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b40f60 0x561e40b45b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.963+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/4011567977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3c860 0x561e40b48350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.963+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/4011567977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b3d210 0x561e40b45650 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.963+0000 7fb7051f0b80 1 -- 172.21.15.70:0/4011567977 >> 172.21.15.70:0/4011567977 conn(0x561e40a6a080 msgr2=0x561e40b36510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:38.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.963+0000 7fb7051f0b80 1 -- 172.21.15.70:0/4011567977 shutdown_connections 2024-06-04T05:05:38.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.963+0000 7fb7051f0b80 1 -- 172.21.15.70:0/4011567977 wait complete. 2024-06-04T05:05:38.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.963+0000 7fb7051f0b80 20 client.17688 trim_cache size 0 max 16384 2024-06-04T05:05:38.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb7051f0b80 1 Processor -- start 2024-06-04T05:05:38.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb7051f0b80 1 -- start start 2024-06-04T05:05:38.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb7051f0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b3c860 0x561e40b3ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb7051f0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3d210 0x561e40b36c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb7051f0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b40f60 0x561e40b371d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb7051f0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f41b0030 con 0x561e40b40f60 2024-06-04T05:05:38.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb7051f0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f41b1b70 con 0x561e40b3d210 2024-06-04T05:05:38.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb7051f0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f41b0700 con 0x561e40b3c860 2024-06-04T05:05:38.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb6f8ef8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b40f60 0x561e40b371d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb6f8ef8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b40f60 0x561e40b371d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56366/0 (socket says 172.21.15.70:56366) 2024-06-04T05:05:38.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb6f8ef8700 1 -- 172.21.15.70:0/1457448812 learned_addr learned my addr 172.21.15.70:0/1457448812 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:38.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb6f3fff700 1 --2- 172.21.15.70:0/1457448812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b3c860 0x561e40b3ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb6f37fe700 1 --2- 172.21.15.70:0/1457448812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3d210 0x561e40b36c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb6f8ef8700 1 -- 172.21.15.70:0/1457448812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b3c860 msgr2=0x561e40b3ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:38.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb6f8ef8700 1 --2- 172.21.15.70:0/1457448812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b3c860 0x561e40b3ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb6f8ef8700 1 -- 172.21.15.70:0/1457448812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3d210 msgr2=0x561e40b36c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:38.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb6f8ef8700 1 --2- 172.21.15.70:0/1457448812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3d210 0x561e40b36c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb6f8ef8700 1 -- 172.21.15.70:0/1457448812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e40b45420 con 0x561e40b40f60 2024-06-04T05:05:38.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.978+0000 7fb6f3fff700 1 --2- 172.21.15.70:0/1457448812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b3c860 0x561e40b3ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:05:38.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.979+0000 7fb6f8ef8700 1 --2- 172.21.15.70:0/1457448812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b40f60 0x561e40b371d0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec0098f0 tx=0x7fb6ec00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0377719d21f98c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:38.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.979+0000 7fb6d9ffb700 1 -- 172.21.15.70:0/1457448812 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec0c8070 con 0x561e40b40f60 2024-06-04T05:05:38.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.979+0000 7fb6d9ffb700 1 -- 172.21.15.70:0/1457448812 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb6ec0a5b20 con 0x561e40b40f60 2024-06-04T05:05:38.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.979+0000 7fb6d9ffb700 1 -- 172.21.15.70:0/1457448812 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec0a5950 con 0x561e40b40f60 2024-06-04T05:05:38.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.980+0000 7fb7051f0b80 1 -- 172.21.15.70:0/1457448812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b40f60 msgr2=0x561e40b371d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:38.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.980+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/1457448812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b40f60 0x561e40b371d0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec0098f0 tx=0x7fb6ec00ef90 comp rx=0 tx=0).stop 2024-06-04T05:05:38.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.980+0000 7fb7051f0b80 1 -- 172.21.15.70:0/1457448812 shutdown_connections 2024-06-04T05:05:38.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.980+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/1457448812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b40f60 0x561e40b371d0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.980+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/1457448812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b3c860 0x561e40b3ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.980+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/1457448812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3d210 0x561e40b36c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.980+0000 7fb7051f0b80 1 -- 172.21.15.70:0/1457448812 >> 172.21.15.70:0/1457448812 conn(0x561e40a6a080 msgr2=0x561e40b3f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:38.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.980+0000 7fb7051f0b80 1 -- 172.21.15.70:0/1457448812 shutdown_connections 2024-06-04T05:05:38.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.980+0000 7fb7051f0b80 1 -- 172.21.15.70:0/1457448812 wait complete. 2024-06-04T05:05:38.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.980+0000 7fb7051f0b80 1 -- start start 2024-06-04T05:05:38.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.980+0000 7fb7051f0b80 1 Processor -- start 2024-06-04T05:05:38.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb7051f0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3c860 0x561e40b4eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb7051f0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b3d210 0x561e40bb94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb7051f0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b40f60 0x561e40bb99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb7051f0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f41b14a0 con 0x561e40b3d210 2024-06-04T05:05:38.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb7051f0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f41b0dd0 con 0x561e40b3c860 2024-06-04T05:05:38.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb7051f0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f41adff0 con 0x561e40b40f60 2024-06-04T05:05:38.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb7051f0b80 10 client.0 _ll_register_callbacks cb 0x561e40a26920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:38.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb6f37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b3d210 0x561e40bb94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb6f37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b3d210 0x561e40bb94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56382/0 (socket says 172.21.15.70:56382) 2024-06-04T05:05:38.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb6f37fe700 1 -- 172.21.15.70:0/90805403 learned_addr learned my addr 172.21.15.70:0/90805403 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:38.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb6f3fff700 1 --2- 172.21.15.70:0/90805403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3c860 0x561e40b4eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb6f8ef8700 1 --2- 172.21.15.70:0/90805403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b40f60 0x561e40bb99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb6f37fe700 1 -- 172.21.15.70:0/90805403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b40f60 msgr2=0x561e40bb99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:38.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb6f37fe700 1 --2- 172.21.15.70:0/90805403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b40f60 0x561e40bb99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb6f37fe700 1 -- 172.21.15.70:0/90805403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3c860 msgr2=0x561e40b4eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:38.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb6f37fe700 1 --2- 172.21.15.70:0/90805403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3c860 0x561e40b4eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb6f37fe700 1 -- 172.21.15.70:0/90805403 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e40bba220 con 0x561e40b3d210 2024-06-04T05:05:38.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb6f3fff700 1 --2- 172.21.15.70:0/90805403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3c860 0x561e40b4eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:38.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb6f8ef8700 1 --2- 172.21.15.70:0/90805403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b40f60 0x561e40bb99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:38.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb6f37fe700 1 --2- 172.21.15.70:0/90805403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b3d210 0x561e40bb94a0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fb6e800fb30 tx=0x7fb6e8009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52bd779fdefccfb8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:38.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.981+0000 7fb6d37fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:05:38.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.982+0000 7fb6d37fe700 1 -- 172.21.15.70:0/90805403 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8028070 con 0x561e40b3d210 2024-06-04T05:05:38.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.982+0000 7fb7051f0b80 1 -- 172.21.15.70:0/90805403 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6ec0a5b20 con 0x561e40b3d210 2024-06-04T05:05:38.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.982+0000 7fb6d37fe700 1 -- 172.21.15.70:0/90805403 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fb6e800f660 con 0x561e40b3d210 2024-06-04T05:05:38.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.982+0000 7fb6d37fe700 1 -- 172.21.15.70:0/90805403 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb6e803b050 con 0x561e40b3d210 2024-06-04T05:05:38.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.982+0000 7fb7051f0b80 10 client.17694 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:38.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.982+0000 7fb7051f0b80 1 -- 172.21.15.70:0/90805403 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561e40bc91a0 con 0x561e40b3d210 2024-06-04T05:05:38.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.982+0000 7fb6d37fe700 1 -- 172.21.15.70:0/90805403 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb6ec0a5b20 con 0x561e40b3d210 2024-06-04T05:05:38.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.982+0000 7fb6d37fe700 1 -- 172.21.15.70:0/90805403 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fb6e80374e0 con 0x561e40b3d210 2024-06-04T05:05:38.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.982+0000 7fb7051f0b80 10 client.17694 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:38.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.982+0000 7fb7051f0b80 10 client.17694 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:05:38.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.982+0000 7fb7051f0b80 20 client.17694 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:38.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.982+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/90805403 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x561e40b55e00 0x561e40bcb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:38.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.982+0000 7fb7051f0b80 4 client.17694 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:05:38.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.982+0000 7fb7051f0b80 1 -- 172.21.15.70:0/90805403 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561e40a9dbe0 con 0x561e40b55e00 2024-06-04T05:05:38.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.982+0000 7fb6f3fff700 1 --2- 172.21.15.70:0/90805403 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x561e40b55e00 0x561e40bcb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:38.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.983+0000 7fb6d37fe700 10 client.17694 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:05:38.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.983+0000 7fb6f3fff700 1 --2- 172.21.15.70:0/90805403 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x561e40b55e00 0x561e40bcb860 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=65c167e56213e7d0 server_cookie=2ae4dc880302069c in_seq=0 out_seq=0 2024-06-04T05:05:38.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.983+0000 7fb6d37fe700 1 -- 172.21.15.70:0/90805403 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561e40a9dbe0 con 0x561e40b55e00 2024-06-04T05:05:38.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.983+0000 7fb6d37fe700 10 client.17694 handle_command_reply: tid=0 2024-06-04T05:05:38.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.983+0000 7fb7051f0b80 1 client.17694 shutdown 2024-06-04T05:05:38.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.983+0000 7fb7051f0b80 1 -- 172.21.15.70:0/90805403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b3d210 msgr2=0x561e40bb94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:38.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.983+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/90805403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b3d210 0x561e40bb94a0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fb6e800fb30 tx=0x7fb6e8009290 comp rx=0 tx=0).stop 2024-06-04T05:05:38.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.983+0000 7fb7051f0b80 1 -- 172.21.15.70:0/90805403 shutdown_connections 2024-06-04T05:05:38.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.983+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/90805403 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x561e40b55e00 0x561e40bcb860 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.983+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/90805403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e40b40f60 0x561e40bb99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.983+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/90805403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e40b3c860 0x561e40b4eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.983+0000 7fb6d37fe700 0 client.17694 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:05:38.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.983+0000 7fb7051f0b80 1 --2- 172.21.15.70:0/90805403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e40b3d210 0x561e40bb94a0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.983+0000 7fb7051f0b80 1 -- 172.21.15.70:0/90805403 >> 172.21.15.70:0/90805403 conn(0x561e40a6a080 msgr2=0x561e40b46e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:38.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.983+0000 7fb7051f0b80 1 -- 172.21.15.70:0/90805403 shutdown_connections 2024-06-04T05:05:38.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.983+0000 7fb7051f0b80 1 -- 172.21.15.70:0/90805403 wait complete. 2024-06-04T05:05:38.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.983+0000 7fb7051f0b80 20 client.17694 trim_cache size 0 max 16384 2024-06-04T05:05:38.992 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:05:38.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.984+0000 7fb6fa95b700 1 -- 172.21.15.70:0/716192363 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb6dc0b29f0 msgr2=0x7fb6dc0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:38.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.984+0000 7fb6fa95b700 1 --2- 172.21.15.70:0/716192363 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb6dc0b29f0 0x7fb6dc0b4eb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb6e4005950 tx=0x7fb6e400a300 comp rx=0 tx=0).stop 2024-06-04T05:05:38.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.984+0000 7fb6fa95b700 1 -- 172.21.15.70:0/716192363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6f41088f0 msgr2=0x7fb6f40762c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:38.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.984+0000 7fb6fa95b700 1 --2- 172.21.15.70:0/716192363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6f41088f0 0x7fb6f40762c0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec00b970 tx=0x7fb6ec00bc80 comp rx=0 tx=0).stop 2024-06-04T05:05:38.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.984+0000 7fb6fa95b700 1 -- 172.21.15.70:0/716192363 shutdown_connections 2024-06-04T05:05:38.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.984+0000 7fb6fa95b700 1 --2- 172.21.15.70:0/716192363 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb6dc0b29f0 0x7fb6dc0b4eb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.984+0000 7fb6fa95b700 1 --2- 172.21.15.70:0/716192363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6f41088f0 0x7fb6f40762c0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.984+0000 7fb6fa95b700 1 --2- 172.21.15.70:0/716192363 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6f41037f0 0x7fb6f4075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.984+0000 7fb6fa95b700 1 --2- 172.21.15.70:0/716192363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f41041a0 0x7fb6f4075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:38.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.984+0000 7fb6fa95b700 1 -- 172.21.15.70:0/716192363 >> 172.21.15.70:0/716192363 conn(0x7fb6f407a6a0 msgr2=0x7fb6f4101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:38.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.984+0000 7fb6fa95b700 1 -- 172.21.15.70:0/716192363 shutdown_connections 2024-06-04T05:05:38.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:38.984+0000 7fb6fa95b700 1 -- 172.21.15.70:0/716192363 wait complete. 2024-06-04T05:05:39.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:39 smithi070 ceph-mon[32631]: pgmap v259: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 686 KiB/s wr, 62 op/s 2024-06-04T05:05:39.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:39 smithi112 ceph-mon[36453]: pgmap v259: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 686 KiB/s wr, 62 op/s 2024-06-04T05:05:39.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:39 smithi098 ceph-mon[35569]: pgmap v259: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 686 KiB/s wr, 62 op/s 2024-06-04T05:05:39.640 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:05:39.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.06 MB/sec execute 272 sec latency 9827.843 ms 2024-06-04T05:05:39.880 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:05:40.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.06 MB/sec execute 273 sec latency 10827.876 ms 2024-06-04T05:05:41.340 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:41 smithi070 ceph-mon[32631]: pgmap v260: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 686 KiB/s wr, 62 op/s 2024-06-04T05:05:41.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:41 smithi112 ceph-mon[36453]: pgmap v260: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 686 KiB/s wr, 62 op/s 2024-06-04T05:05:41.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:41 smithi098 ceph-mon[35569]: pgmap v260: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 686 KiB/s wr, 62 op/s 2024-06-04T05:05:41.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.616+0000 7fb3d424c700 1 -- 172.21.15.70:0/3056014064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3cc102470 msgr2=0x7fb3cc1028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:41.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.616+0000 7fb3d424c700 1 --2- 172.21.15.70:0/3056014064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3cc102470 0x7fb3cc1028f0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80099c0 tx=0x7fb3c8009cd0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.616+0000 7fb3d424c700 1 -- 172.21.15.70:0/3056014064 shutdown_connections 2024-06-04T05:05:41.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.616+0000 7fb3d424c700 1 --2- 172.21.15.70:0/3056014064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb3cc102e30 0x7fb3cc10f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.616+0000 7fb3d424c700 1 --2- 172.21.15.70:0/3056014064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3cc108550 0x7fb3cc108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.616+0000 7fb3d424c700 1 --2- 172.21.15.70:0/3056014064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3cc102470 0x7fb3cc1028f0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.616+0000 7fb3d424c700 1 -- 172.21.15.70:0/3056014064 >> 172.21.15.70:0/3056014064 conn(0x7fb3cc079480 msgr2=0x7fb3cc079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:41.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.616+0000 7fb3d424c700 1 -- 172.21.15.70:0/3056014064 shutdown_connections 2024-06-04T05:05:41.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.616+0000 7fb3d424c700 1 -- 172.21.15.70:0/3056014064 wait complete. 2024-06-04T05:05:41.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d424c700 1 Processor -- start 2024-06-04T05:05:41.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d424c700 1 -- start start 2024-06-04T05:05:41.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d424c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb3cc102470 0x7fb3cc19e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d424c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3cc102e30 0x7fb3cc19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d424c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3cc108550 0x7fb3cc198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d424c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc1124d0 con 0x7fb3cc102e30 2024-06-04T05:05:41.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d424c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc112350 con 0x7fb3cc108550 2024-06-04T05:05:41.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d424c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc112650 con 0x7fb3cc102470 2024-06-04T05:05:41.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d17e7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3cc102e30 0x7fb3cc19ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d17e7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3cc102e30 0x7fb3cc19ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56404/0 (socket says 172.21.15.70:56404) 2024-06-04T05:05:41.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d17e7700 1 -- 172.21.15.70:0/2468778030 learned_addr learned my addr 172.21.15.70:0/2468778030 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:41.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d27e9700 1 --2- 172.21.15.70:0/2468778030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3cc108550 0x7fb3cc198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d1fe8700 1 --2- 172.21.15.70:0/2468778030 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb3cc102470 0x7fb3cc19e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d17e7700 1 -- 172.21.15.70:0/2468778030 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb3cc102470 msgr2=0x7fb3cc19e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:41.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d17e7700 1 --2- 172.21.15.70:0/2468778030 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb3cc102470 0x7fb3cc19e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d17e7700 1 -- 172.21.15.70:0/2468778030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3cc108550 msgr2=0x7fb3cc198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:41.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d17e7700 1 --2- 172.21.15.70:0/2468778030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3cc108550 0x7fb3cc198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d17e7700 1 -- 172.21.15.70:0/2468778030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3cc198fd0 con 0x7fb3cc102e30 2024-06-04T05:05:41.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.617+0000 7fb3d1fe8700 1 --2- 172.21.15.70:0/2468778030 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb3cc102470 0x7fb3cc19e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:05:41.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.618+0000 7fb3d17e7700 1 --2- 172.21.15.70:0/2468778030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3cc102e30 0x7fb3cc19ebf0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8006010 tx=0x7fb3c8004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:41.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.618+0000 7fb3c2ffd700 1 -- 172.21.15.70:0/2468778030 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8026070 con 0x7fb3cc102e30 2024-06-04T05:05:41.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.618+0000 7fb3c2ffd700 1 -- 172.21.15.70:0/2468778030 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3c800f460 con 0x7fb3cc102e30 2024-06-04T05:05:41.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.618+0000 7fb3d424c700 1 -- 172.21.15.70:0/2468778030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3cc199260 con 0x7fb3cc102e30 2024-06-04T05:05:41.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.618+0000 7fb3c2ffd700 1 -- 172.21.15.70:0/2468778030 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8009e60 con 0x7fb3cc102e30 2024-06-04T05:05:41.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.618+0000 7fb3d424c700 1 -- 172.21.15.70:0/2468778030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3cc1a5580 con 0x7fb3cc102e30 2024-06-04T05:05:41.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.620+0000 7fb3c2ffd700 1 -- 172.21.15.70:0/2468778030 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fb3c8029070 con 0x7fb3cc102e30 2024-06-04T05:05:41.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.621+0000 7fb3c2ffd700 1 --2- 172.21.15.70:0/2468778030 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3b80b2a70 0x7fb3b80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.622+0000 7fb3d1fe8700 1 --2- 172.21.15.70:0/2468778030 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3b80b2a70 0x7fb3b80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.622+0000 7fb3c2ffd700 1 -- 172.21.15.70:0/2468778030 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fb3c80e2fe0 con 0x7fb3cc102e30 2024-06-04T05:05:41.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.622+0000 7fb3deae1b80 1 Processor -- start 2024-06-04T05:05:41.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.622+0000 7fb3d1fe8700 1 --2- 172.21.15.70:0/2468778030 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3b80b2a70 0x7fb3b80b4f30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc005e90 tx=0x7fb3bc005d40 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:41.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.622+0000 7fb3deae1b80 1 -- start start 2024-06-04T05:05:41.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.622+0000 7fb3deae1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895d70 0x55c4d6896150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.622+0000 7fb3deae1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6894680 0x55c4d6894b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.622+0000 7fb3deae1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6895040 0x55c4d68a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.622+0000 7fb3deae1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c4d67f3c90 con 0x55c4d6894680 2024-06-04T05:05:41.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.622+0000 7fb3deae1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c4d67e3830 con 0x55c4d6895040 2024-06-04T05:05:41.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.622+0000 7fb3deae1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c4d67e7870 con 0x55c4d6895d70 2024-06-04T05:05:41.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.623+0000 7fb3d17e7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6894680 0x55c4d6894b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.623+0000 7fb3d17e7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6894680 0x55c4d6894b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56414/0 (socket says 172.21.15.70:56414) 2024-06-04T05:05:41.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.623+0000 7fb3d17e7700 1 -- 172.21.15.70:0/443578213 learned_addr learned my addr 172.21.15.70:0/443578213 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:41.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.623+0000 7fb3d27e9700 1 --2- 172.21.15.70:0/443578213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6895040 0x55c4d68a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.623+0000 7fb3d1fe8700 1 --2- 172.21.15.70:0/443578213 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895d70 0x55c4d6896150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.623+0000 7fb3d17e7700 1 -- 172.21.15.70:0/443578213 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895d70 msgr2=0x55c4d6896150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:41.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.623+0000 7fb3d17e7700 1 --2- 172.21.15.70:0/443578213 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895d70 0x55c4d6896150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.623+0000 7fb3d17e7700 1 -- 172.21.15.70:0/443578213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6895040 msgr2=0x55c4d68a4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:41.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.623+0000 7fb3d17e7700 1 --2- 172.21.15.70:0/443578213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6895040 0x55c4d68a4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.623+0000 7fb3d17e7700 1 -- 172.21.15.70:0/443578213 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4d67338d0 con 0x55c4d6894680 2024-06-04T05:05:41.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.623+0000 7fb3d1fe8700 1 --2- 172.21.15.70:0/443578213 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895d70 0x55c4d6896150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:05:41.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.623+0000 7fb3d17e7700 1 --2- 172.21.15.70:0/443578213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6894680 0x55c4d6894b00 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80aa290 tx=0x7fb3c80a9d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248005d80fffe12d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:41.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.623+0000 7fb3b7fff700 1 -- 172.21.15.70:0/443578213 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c80aaa00 con 0x55c4d6894680 2024-06-04T05:05:41.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.623+0000 7fb3b7fff700 1 -- 172.21.15.70:0/443578213 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3c80aa300 con 0x55c4d6894680 2024-06-04T05:05:41.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.623+0000 7fb3b7fff700 1 -- 172.21.15.70:0/443578213 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c80aa560 con 0x55c4d6894680 2024-06-04T05:05:41.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.624+0000 7fb3deae1b80 1 -- 172.21.15.70:0/443578213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6894680 msgr2=0x55c4d6894b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:41.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.624+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/443578213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6894680 0x55c4d6894b00 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80aa290 tx=0x7fb3c80a9d50 comp rx=0 tx=0).stop 2024-06-04T05:05:41.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.624+0000 7fb3deae1b80 1 -- 172.21.15.70:0/443578213 shutdown_connections 2024-06-04T05:05:41.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.624+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/443578213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6895040 0x55c4d68a4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.624+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/443578213 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895d70 0x55c4d6896150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.624+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/443578213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6894680 0x55c4d6894b00 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.624+0000 7fb3deae1b80 1 -- 172.21.15.70:0/443578213 >> 172.21.15.70:0/443578213 conn(0x55c4d67c2080 msgr2=0x55c4d688ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:41.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.624+0000 7fb3deae1b80 1 -- 172.21.15.70:0/443578213 shutdown_connections 2024-06-04T05:05:41.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.624+0000 7fb3deae1b80 1 -- 172.21.15.70:0/443578213 wait complete. 2024-06-04T05:05:41.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.624+0000 7fb3deae1b80 1 -- start start 2024-06-04T05:05:41.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.625+0000 7fb3deae1b80 1 Processor -- start 2024-06-04T05:05:41.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.625+0000 7fb3deae1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6894680 0x55c4d68a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.625+0000 7fb3deae1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895040 0x55c4d68a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.625+0000 7fb3deae1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6895d70 0x55c4d68a2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.625+0000 7fb3deae1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c4d66e9fa0 con 0x55c4d6894680 2024-06-04T05:05:41.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.625+0000 7fb3deae1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c4d67857c0 con 0x55c4d6895d70 2024-06-04T05:05:41.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.625+0000 7fb3deae1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c4d68a5390 con 0x55c4d6895040 2024-06-04T05:05:41.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.625+0000 7fb3deae1b80 10 client.0 _ll_register_callbacks cb 0x55c4d677e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:41.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.625+0000 7fb3d1fe8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6894680 0x55c4d68a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.625+0000 7fb3d1fe8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6894680 0x55c4d68a4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56416/0 (socket says 172.21.15.70:56416) 2024-06-04T05:05:41.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.625+0000 7fb3d1fe8700 1 -- 172.21.15.70:0/2188135823 learned_addr learned my addr 172.21.15.70:0/2188135823 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:41.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.625+0000 7fb3d27e9700 1 --2- 172.21.15.70:0/2188135823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6895d70 0x55c4d68a2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.625+0000 7fb3d17e7700 1 --2- 172.21.15.70:0/2188135823 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895040 0x55c4d68a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.626+0000 7fb3d1fe8700 1 -- 172.21.15.70:0/2188135823 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895040 msgr2=0x55c4d68a1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:41.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.626+0000 7fb3d1fe8700 1 --2- 172.21.15.70:0/2188135823 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895040 0x55c4d68a1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.626+0000 7fb3d1fe8700 1 -- 172.21.15.70:0/2188135823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6895d70 msgr2=0x55c4d68a2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:41.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.626+0000 7fb3d1fe8700 1 --2- 172.21.15.70:0/2188135823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6895d70 0x55c4d68a2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.626+0000 7fb3d1fe8700 1 -- 172.21.15.70:0/2188135823 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4d6782690 con 0x55c4d6894680 2024-06-04T05:05:41.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.626+0000 7fb3d17e7700 1 --2- 172.21.15.70:0/2188135823 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895040 0x55c4d68a1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:05:41.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.626+0000 7fb3d1fe8700 1 --2- 172.21.15.70:0/2188135823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6894680 0x55c4d68a4ac0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc011750 tx=0x7fb3bc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9555b9d80a4bf2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:41.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.626+0000 7fb3b57fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:05:41.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.626+0000 7fb3b57fa700 1 -- 172.21.15.70:0/2188135823 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc023070 con 0x55c4d6894680 2024-06-04T05:05:41.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.626+0000 7fb3b57fa700 1 -- 172.21.15.70:0/2188135823 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fb3bc004a30 con 0x55c4d6894680 2024-06-04T05:05:41.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.626+0000 7fb3deae1b80 1 -- 172.21.15.70:0/2188135823 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3c80aa300 con 0x55c4d6894680 2024-06-04T05:05:41.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.626+0000 7fb3b57fa700 1 -- 172.21.15.70:0/2188135823 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb3bc02e050 con 0x55c4d6894680 2024-06-04T05:05:41.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.626+0000 7fb3deae1b80 10 client.17706 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:41.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.626+0000 7fb3deae1b80 1 -- 172.21.15.70:0/2188135823 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c4d68afb60 con 0x55c4d6894680 2024-06-04T05:05:41.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.626+0000 7fb3b57fa700 1 -- 172.21.15.70:0/2188135823 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3c80aa300 con 0x55c4d6894680 2024-06-04T05:05:41.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.627+0000 7fb3b57fa700 1 -- 172.21.15.70:0/2188135823 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fb3bc00f040 con 0x55c4d6894680 2024-06-04T05:05:41.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.627+0000 7fb3deae1b80 10 client.17706 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:41.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.627+0000 7fb3deae1b80 10 client.17706 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:05:41.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.627+0000 7fb3deae1b80 20 client.17706 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:41.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.627+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/2188135823 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c4d68b2190 0x55c4d68b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.627+0000 7fb3deae1b80 4 client.17706 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:05:41.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.627+0000 7fb3deae1b80 1 -- 172.21.15.70:0/2188135823 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4d68609e0 con 0x55c4d68b2190 2024-06-04T05:05:41.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.627+0000 7fb3d17e7700 1 --2- 172.21.15.70:0/2188135823 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c4d68b2190 0x55c4d68b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.627+0000 7fb3d17e7700 1 --2- 172.21.15.70:0/2188135823 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c4d68b2190 0x55c4d68b45e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5e2625a33c4af2ef server_cookie=53bbd755537e5bb4 in_seq=0 out_seq=0 2024-06-04T05:05:41.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.627+0000 7fb3b57fa700 10 client.17706 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:05:41.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.628+0000 7fb3b57fa700 1 -- 172.21.15.70:0/2188135823 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c4d68609e0 con 0x55c4d68b2190 2024-06-04T05:05:41.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.628+0000 7fb3b57fa700 10 client.17706 handle_command_reply: tid=0 2024-06-04T05:05:41.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.628+0000 7fb3deae1b80 1 client.17706 shutdown 2024-06-04T05:05:41.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.628+0000 7fb3deae1b80 1 -- 172.21.15.70:0/2188135823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6894680 msgr2=0x55c4d68a4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:41.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.628+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/2188135823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6894680 0x55c4d68a4ac0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc011750 tx=0x7fb3bc004750 comp rx=0 tx=0).stop 2024-06-04T05:05:41.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.628+0000 7fb3deae1b80 1 -- 172.21.15.70:0/2188135823 shutdown_connections 2024-06-04T05:05:41.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.628+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/2188135823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6895d70 0x55c4d68a2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.628+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/2188135823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6894680 0x55c4d68a4ac0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.628+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/2188135823 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c4d68b2190 0x55c4d68b45e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.628+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/2188135823 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895040 0x55c4d68a1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.628+0000 7fb3deae1b80 1 -- 172.21.15.70:0/2188135823 >> 172.21.15.70:0/2188135823 conn(0x55c4d67c2080 msgr2=0x55c4d688e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:41.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.628+0000 7fb3b57fa700 0 client.17706 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:05:41.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.628+0000 7fb3deae1b80 1 -- 172.21.15.70:0/2188135823 shutdown_connections 2024-06-04T05:05:41.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.628+0000 7fb3deae1b80 1 -- 172.21.15.70:0/2188135823 wait complete. 2024-06-04T05:05:41.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.629+0000 7fb3deae1b80 20 client.17706 trim_cache size 0 max 16384 2024-06-04T05:05:41.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.643+0000 7fb3deae1b80 1 Processor -- start 2024-06-04T05:05:41.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.643+0000 7fb3deae1b80 1 -- start start 2024-06-04T05:05:41.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3deae1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6894680 0x55c4d6893d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3deae1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6895040 0x55c4d68a4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3deae1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895d70 0x55c4d689bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3deae1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc1abbb0 con 0x55c4d6895040 2024-06-04T05:05:41.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3deae1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc1ad6f0 con 0x55c4d6894680 2024-06-04T05:05:41.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3deae1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc1ac280 con 0x55c4d6895d70 2024-06-04T05:05:41.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3d17e7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6895040 0x55c4d68a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3d17e7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6895040 0x55c4d68a4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56426/0 (socket says 172.21.15.70:56426) 2024-06-04T05:05:41.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3d17e7700 1 -- 172.21.15.70:0/596126929 learned_addr learned my addr 172.21.15.70:0/596126929 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:41.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3d1fe8700 1 --2- 172.21.15.70:0/596126929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6894680 0x55c4d6893d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3d27e9700 1 --2- 172.21.15.70:0/596126929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895d70 0x55c4d689bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3d17e7700 1 -- 172.21.15.70:0/596126929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895d70 msgr2=0x55c4d689bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:41.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3d17e7700 1 --2- 172.21.15.70:0/596126929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895d70 0x55c4d689bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3d17e7700 1 -- 172.21.15.70:0/596126929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6894680 msgr2=0x55c4d6893d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:41.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3d17e7700 1 --2- 172.21.15.70:0/596126929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6894680 0x55c4d6893d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3d17e7700 1 -- 172.21.15.70:0/596126929 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4d688fd40 con 0x55c4d6895040 2024-06-04T05:05:41.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3d27e9700 1 --2- 172.21.15.70:0/596126929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895d70 0x55c4d689bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:05:41.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.644+0000 7fb3d17e7700 1 --2- 172.21.15.70:0/596126929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6895040 0x55c4d68a4600 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80059d0 tx=0x7fb3c80a9c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5abb378c0af502 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:41.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.645+0000 7fb3b77fe700 1 -- 172.21.15.70:0/596126929 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c80aaa00 con 0x55c4d6895040 2024-06-04T05:05:41.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.645+0000 7fb3b77fe700 1 -- 172.21.15.70:0/596126929 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3c80a6590 con 0x55c4d6895040 2024-06-04T05:05:41.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.645+0000 7fb3b77fe700 1 -- 172.21.15.70:0/596126929 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c80b2c40 con 0x55c4d6895040 2024-06-04T05:05:41.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.645+0000 7fb3deae1b80 1 -- 172.21.15.70:0/596126929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6895040 msgr2=0x55c4d68a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:41.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.645+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/596126929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6895040 0x55c4d68a4600 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80059d0 tx=0x7fb3c80a9c50 comp rx=0 tx=0).stop 2024-06-04T05:05:41.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.645+0000 7fb3deae1b80 1 -- 172.21.15.70:0/596126929 shutdown_connections 2024-06-04T05:05:41.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.645+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/596126929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6895d70 0x55c4d689bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.645+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/596126929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6894680 0x55c4d6893d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.645+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/596126929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6895040 0x55c4d68a4600 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.645+0000 7fb3deae1b80 1 -- 172.21.15.70:0/596126929 >> 172.21.15.70:0/596126929 conn(0x55c4d67c2080 msgr2=0x55c4d6893670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:41.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.645+0000 7fb3deae1b80 1 -- 172.21.15.70:0/596126929 shutdown_connections 2024-06-04T05:05:41.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.646+0000 7fb3deae1b80 1 -- 172.21.15.70:0/596126929 wait complete. 2024-06-04T05:05:41.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.646+0000 7fb3deae1b80 1 -- start start 2024-06-04T05:05:41.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.646+0000 7fb3deae1b80 1 Processor -- start 2024-06-04T05:05:41.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.646+0000 7fb3deae1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6894680 0x55c4d68aa8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.646+0000 7fb3deae1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6895040 0x55c4d68aae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.646+0000 7fb3deae1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6895d70 0x55c4d689e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.646+0000 7fb3deae1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc1ad020 con 0x55c4d6895d70 2024-06-04T05:05:41.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.646+0000 7fb3deae1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc1ac950 con 0x55c4d6895040 2024-06-04T05:05:41.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.646+0000 7fb3deae1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc1a9b70 con 0x55c4d6894680 2024-06-04T05:05:41.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.646+0000 7fb3deae1b80 10 client.0 _ll_register_callbacks cb 0x55c4d677e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:41.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.646+0000 7fb3d27e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6895d70 0x55c4d689e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.646+0000 7fb3d27e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6895d70 0x55c4d689e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56442/0 (socket says 172.21.15.70:56442) 2024-06-04T05:05:41.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.646+0000 7fb3d27e9700 1 -- 172.21.15.70:0/1551661656 learned_addr learned my addr 172.21.15.70:0/1551661656 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:41.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.647+0000 7fb3d17e7700 1 --2- 172.21.15.70:0/1551661656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6895040 0x55c4d68aae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.647+0000 7fb3d1fe8700 1 --2- 172.21.15.70:0/1551661656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6894680 0x55c4d68aa8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.647+0000 7fb3d27e9700 1 -- 172.21.15.70:0/1551661656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6894680 msgr2=0x55c4d68aa8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:41.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.647+0000 7fb3d27e9700 1 --2- 172.21.15.70:0/1551661656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6894680 0x55c4d68aa8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.647+0000 7fb3d27e9700 1 -- 172.21.15.70:0/1551661656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6895040 msgr2=0x55c4d68aae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:41.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.647+0000 7fb3d27e9700 1 --2- 172.21.15.70:0/1551661656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6895040 0x55c4d68aae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.647+0000 7fb3d27e9700 1 -- 172.21.15.70:0/1551661656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4d689f120 con 0x55c4d6895d70 2024-06-04T05:05:41.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.647+0000 7fb3d27e9700 1 --2- 172.21.15.70:0/1551661656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6895d70 0x55c4d689e950 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4007f00 tx=0x7fb3c4007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4942e3fde802332b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:41.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.647+0000 7fb3b57fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:05:41.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.647+0000 7fb3b57fa700 1 -- 172.21.15.70:0/1551661656 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4016820 con 0x55c4d6895d70 2024-06-04T05:05:41.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.647+0000 7fb3b57fa700 1 -- 172.21.15.70:0/1551661656 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fb3c40169c0 con 0x55c4d6895d70 2024-06-04T05:05:41.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.647+0000 7fb3deae1b80 1 -- 172.21.15.70:0/1551661656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3c80a6590 con 0x55c4d6895d70 2024-06-04T05:05:41.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.647+0000 7fb3b57fa700 1 -- 172.21.15.70:0/1551661656 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb3c4016bc0 con 0x55c4d6895d70 2024-06-04T05:05:41.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.647+0000 7fb3deae1b80 10 client.17712 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:41.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.647+0000 7fb3deae1b80 1 -- 172.21.15.70:0/1551661656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c4d68aa250 con 0x55c4d6895d70 2024-06-04T05:05:41.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.648+0000 7fb3b57fa700 1 -- 172.21.15.70:0/1551661656 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3c80a6590 con 0x55c4d6895d70 2024-06-04T05:05:41.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.648+0000 7fb3b57fa700 1 -- 172.21.15.70:0/1551661656 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fb3c4017410 con 0x55c4d6895d70 2024-06-04T05:05:41.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.648+0000 7fb3deae1b80 10 client.17712 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:41.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.648+0000 7fb3deae1b80 10 client.17712 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:05:41.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.648+0000 7fb3deae1b80 20 client.17712 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:41.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.648+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/1551661656 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c4d691d0d0 0x55c4d691f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:41.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.648+0000 7fb3deae1b80 4 client.17712 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:05:41.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.648+0000 7fb3deae1b80 1 -- 172.21.15.70:0/1551661656 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c4d67f5be0 con 0x55c4d691d0d0 2024-06-04T05:05:41.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.648+0000 7fb3d1fe8700 1 --2- 172.21.15.70:0/1551661656 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c4d691d0d0 0x55c4d691f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:41.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.649+0000 7fb3b57fa700 10 client.17712 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:05:41.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.649+0000 7fb3d1fe8700 1 --2- 172.21.15.70:0/1551661656 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c4d691d0d0 0x55c4d691f4f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=eb11e5359b7a6539 server_cookie=edb9dc46537585dd in_seq=0 out_seq=0 2024-06-04T05:05:41.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.649+0000 7fb3b57fa700 1 -- 172.21.15.70:0/1551661656 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c4d67f5be0 con 0x55c4d691d0d0 2024-06-04T05:05:41.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.649+0000 7fb3b57fa700 10 client.17712 handle_command_reply: tid=0 2024-06-04T05:05:41.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.649+0000 7fb3deae1b80 1 client.17712 shutdown 2024-06-04T05:05:41.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.649+0000 7fb3deae1b80 1 -- 172.21.15.70:0/1551661656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6895d70 msgr2=0x55c4d689e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:41.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.649+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/1551661656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6895d70 0x55c4d689e950 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4007f00 tx=0x7fb3c4007f30 comp rx=0 tx=0).stop 2024-06-04T05:05:41.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.649+0000 7fb3deae1b80 1 -- 172.21.15.70:0/1551661656 shutdown_connections 2024-06-04T05:05:41.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.649+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/1551661656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4d6895d70 0x55c4d689e950 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.649+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/1551661656 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c4d691d0d0 0x55c4d691f4f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.649+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/1551661656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4d6894680 0x55c4d68aa8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.649+0000 7fb3deae1b80 1 --2- 172.21.15.70:0/1551661656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4d6895040 0x55c4d68aae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.649+0000 7fb3b57fa700 0 client.17712 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:05:41.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.649+0000 7fb3deae1b80 1 -- 172.21.15.70:0/1551661656 >> 172.21.15.70:0/1551661656 conn(0x55c4d67c2080 msgr2=0x55c4d6893f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:41.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.649+0000 7fb3deae1b80 1 -- 172.21.15.70:0/1551661656 shutdown_connections 2024-06-04T05:05:41.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.649+0000 7fb3deae1b80 1 -- 172.21.15.70:0/1551661656 wait complete. 2024-06-04T05:05:41.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.649+0000 7fb3deae1b80 20 client.17712 trim_cache size 0 max 16384 2024-06-04T05:05:41.656 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:05:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.650+0000 7fb3d424c700 1 -- 172.21.15.70:0/2468778030 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3b80b2a70 msgr2=0x7fb3b80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.650+0000 7fb3d424c700 1 --2- 172.21.15.70:0/2468778030 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3b80b2a70 0x7fb3b80b4f30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc005e90 tx=0x7fb3bc005d40 comp rx=0 tx=0).stop 2024-06-04T05:05:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.650+0000 7fb3d424c700 1 -- 172.21.15.70:0/2468778030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3cc102e30 msgr2=0x7fb3cc19ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.650+0000 7fb3d424c700 1 --2- 172.21.15.70:0/2468778030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3cc102e30 0x7fb3cc19ebf0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8006010 tx=0x7fb3c8004690 comp rx=0 tx=0).stop 2024-06-04T05:05:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.650+0000 7fb3d424c700 1 -- 172.21.15.70:0/2468778030 shutdown_connections 2024-06-04T05:05:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.650+0000 7fb3d424c700 1 --2- 172.21.15.70:0/2468778030 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3b80b2a70 0x7fb3b80b4f30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.650+0000 7fb3d424c700 1 --2- 172.21.15.70:0/2468778030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3cc108550 0x7fb3cc198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.650+0000 7fb3d424c700 1 --2- 172.21.15.70:0/2468778030 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb3cc102470 0x7fb3cc19e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.650+0000 7fb3d424c700 1 --2- 172.21.15.70:0/2468778030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3cc102e30 0x7fb3cc19ebf0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:41.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.650+0000 7fb3d424c700 1 -- 172.21.15.70:0/2468778030 >> 172.21.15.70:0/2468778030 conn(0x7fb3cc079480 msgr2=0x7fb3cc100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:41.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.650+0000 7fb3d424c700 1 -- 172.21.15.70:0/2468778030 shutdown_connections 2024-06-04T05:05:41.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:41.650+0000 7fb3d424c700 1 -- 172.21.15.70:0/2468778030 wait complete. 2024-06-04T05:05:41.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.05 MB/sec execute 274 sec latency 11827.919 ms 2024-06-04T05:05:42.336 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:05:42.585 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:05:42.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.04 MB/sec execute 275 sec latency 12827.952 ms 2024-06-04T05:05:43.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.127+0000 7f0d07e54640 1 -- 172.21.15.70:0/1290802735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d00104600 msgr2=0x7f0d0010cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.127+0000 7f0d07e54640 1 --2- 172.21.15.70:0/1290802735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d00104600 0x7f0d0010cae0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4009900 tx=0x7f0cf402a510 comp rx=0 tx=0).stop 2024-06-04T05:05:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.127+0000 7f0d07e54640 1 -- 172.21.15.70:0/1290802735 shutdown_connections 2024-06-04T05:05:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.127+0000 7f0d07e54640 1 --2- 172.21.15.70:0/1290802735 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d0010d020 0x7f0d0010f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.127+0000 7f0d07e54640 1 --2- 172.21.15.70:0/1290802735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d00104600 0x7f0d0010cae0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.127+0000 7f0d07e54640 1 --2- 172.21.15.70:0/1290802735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d00103ce0 0x7f0d001040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.127+0000 7f0d07e54640 1 -- 172.21.15.70:0/1290802735 >> 172.21.15.70:0/1290802735 conn(0x7f0d00078ad0 msgr2=0x7f0d00078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.127+0000 7f0d07e54640 1 -- 172.21.15.70:0/1290802735 shutdown_connections 2024-06-04T05:05:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.127+0000 7f0d07e54640 1 -- 172.21.15.70:0/1290802735 wait complete. 2024-06-04T05:05:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d07e54640 1 Processor -- start 2024-06-04T05:05:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d07e54640 1 -- start start 2024-06-04T05:05:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d07e54640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d00103ce0 0x7f0d001990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d07e54640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d00104600 0x7f0d00199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d07e54640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d0010d020 0x7f0d0019d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d07e54640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0d00112c10 con 0x7f0d0010d020 2024-06-04T05:05:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d07e54640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0d00112a90 con 0x7f0d00104600 2024-06-04T05:05:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d07e54640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0d00112d90 con 0x7f0d00103ce0 2024-06-04T05:05:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d063f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d0010d020 0x7f0d0019d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d063f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d0010d020 0x7f0d0019d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56464/0 (socket says 172.21.15.70:56464) 2024-06-04T05:05:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d063f1640 1 -- 172.21.15.70:0/387981233 learned_addr learned my addr 172.21.15.70:0/387981233 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d053ef640 1 --2- 172.21.15.70:0/387981233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d00104600 0x7f0d00199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d05bf0640 1 --2- 172.21.15.70:0/387981233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d00103ce0 0x7f0d001990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d063f1640 1 -- 172.21.15.70:0/387981233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d00103ce0 msgr2=0x7f0d001990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d063f1640 1 --2- 172.21.15.70:0/387981233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d00103ce0 0x7f0d001990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d063f1640 1 -- 172.21.15.70:0/387981233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d00104600 msgr2=0x7f0d00199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d063f1640 1 --2- 172.21.15.70:0/387981233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d00104600 0x7f0d00199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.128+0000 7f0d063f1640 1 -- 172.21.15.70:0/387981233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d0019e0c0 con 0x7f0d0010d020 2024-06-04T05:05:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.129+0000 7f0d053ef640 1 --2- 172.21.15.70:0/387981233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d00104600 0x7f0d00199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:05:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.129+0000 7f0d063f1640 1 --2- 172.21.15.70:0/387981233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d0010d020 0x7f0d0019d990 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8009fd0 tx=0x7f0cf800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.129+0000 7f0cfeffd640 1 -- 172.21.15.70:0/387981233 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8011830 con 0x7f0d0010d020 2024-06-04T05:05:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.129+0000 7f0cfeffd640 1 -- 172.21.15.70:0/387981233 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0cf8016070 con 0x7f0d0010d020 2024-06-04T05:05:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.129+0000 7f0d07e54640 1 -- 172.21.15.70:0/387981233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d0019e3b0 con 0x7f0d0010d020 2024-06-04T05:05:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.129+0000 7f0cfeffd640 1 -- 172.21.15.70:0/387981233 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8010d20 con 0x7f0d0010d020 2024-06-04T05:05:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.129+0000 7f0d07e54640 1 -- 172.21.15.70:0/387981233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d001a5c40 con 0x7f0d0010d020 2024-06-04T05:05:43.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.131+0000 7f0cfeffd640 1 -- 172.21.15.70:0/387981233 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f0cf801b460 con 0x7f0d0010d020 2024-06-04T05:05:43.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.131+0000 7f0d0ab69740 1 Processor -- start 2024-06-04T05:05:43.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.132+0000 7f0d0ab69740 1 -- start start 2024-06-04T05:05:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.132+0000 7f0d0ab69740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82ca8720 0x556c82ca8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.132+0000 7f0d0ab69740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82ca9040 0x556c82cb78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.132+0000 7f0d0ab69740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82cb7e10 0x556c82cba200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.132+0000 7f0d0ab69740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556c82c1ac30 con 0x556c82cb7e10 2024-06-04T05:05:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.132+0000 7f0d0ab69740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556c82c0a7d0 con 0x556c82ca8720 2024-06-04T05:05:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.132+0000 7f0d0ab69740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556c82c0e810 con 0x556c82ca9040 2024-06-04T05:05:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.132+0000 7f0d053ef640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82cb7e10 0x556c82cba200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.132+0000 7f0d053ef640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82cb7e10 0x556c82cba200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56478/0 (socket says 172.21.15.70:56478) 2024-06-04T05:05:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.132+0000 7f0d053ef640 1 -- 172.21.15.70:0/2347941680 learned_addr learned my addr 172.21.15.70:0/2347941680 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.132+0000 7f0d053ef640 1 --2- 172.21.15.70:0/2347941680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82ca8720 0x556c82ca8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.132+0000 7f0d05bf0640 1 --2- 172.21.15.70:0/2347941680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82ca9040 0x556c82cb78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.132+0000 7f0cfeffd640 1 --2- 172.21.15.70:0/387981233 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0ce80b2a10 0x7f0ce80b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.132+0000 7f0d063f1640 1 --2- 172.21.15.70:0/387981233 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0ce80b2a10 0x7f0ce80b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.133+0000 7f0d053ef640 1 -- 172.21.15.70:0/2347941680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82ca9040 msgr2=0x556c82cb78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.133+0000 7f0d053ef640 1 --2- 172.21.15.70:0/2347941680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82ca9040 0x556c82cb78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.133+0000 7f0d053ef640 1 -- 172.21.15.70:0/2347941680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82ca8720 msgr2=0x556c82ca8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.133+0000 7f0d053ef640 1 --2- 172.21.15.70:0/2347941680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82ca8720 0x556c82ca8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.133+0000 7f0d053ef640 1 -- 172.21.15.70:0/2347941680 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c8291f210 con 0x556c82cb7e10 2024-06-04T05:05:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.133+0000 7f0cfeffd640 1 -- 172.21.15.70:0/387981233 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f0cf80e1c10 con 0x7f0d0010d020 2024-06-04T05:05:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.133+0000 7f0d053ef640 1 --2- 172.21.15.70:0/2347941680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82cb7e10 0x556c82cba200 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f0cf402ab00 tx=0x7f0cf4009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d9fa31700c0e1d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.133+0000 7f0d063f1640 1 --2- 172.21.15.70:0/387981233 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0ce80b2a10 0x7f0ce80b4ed0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80a8cd0 tx=0x7f0cf8075900 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.133+0000 7f0ce7fff640 1 -- 172.21.15.70:0/2347941680 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4041070 con 0x556c82cb7e10 2024-06-04T05:05:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.133+0000 7f0ce7fff640 1 -- 172.21.15.70:0/2347941680 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0cf4045070 con 0x556c82cb7e10 2024-06-04T05:05:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.133+0000 7f0ce7fff640 1 -- 172.21.15.70:0/2347941680 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4035050 con 0x556c82cb7e10 2024-06-04T05:05:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.134+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2347941680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82cb7e10 msgr2=0x556c82cba200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.134+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/2347941680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82cb7e10 0x556c82cba200 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f0cf402ab00 tx=0x7f0cf4009600 comp rx=0 tx=0).stop 2024-06-04T05:05:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.134+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2347941680 shutdown_connections 2024-06-04T05:05:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.134+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/2347941680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82cb7e10 0x556c82cba200 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.134+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/2347941680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82ca9040 0x556c82cb78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.134+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/2347941680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82ca8720 0x556c82ca8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.134+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2347941680 >> 172.21.15.70:0/2347941680 conn(0x556c82ca4d30 msgr2=0x556c82ca5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.134+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2347941680 shutdown_connections 2024-06-04T05:05:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.134+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2347941680 wait complete. 2024-06-04T05:05:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.134+0000 7f0d0ab69740 1 -- start start 2024-06-04T05:05:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.134+0000 7f0d0ab69740 1 Processor -- start 2024-06-04T05:05:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d0ab69740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82ca8720 0x556c82cb4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d0ab69740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82ca9040 0x556c82cb5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d0ab69740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82cb7e10 0x556c82cb5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d0ab69740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556c828f0790 con 0x556c82cb7e10 2024-06-04T05:05:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d0ab69740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556c82bc6890 con 0x556c82ca8720 2024-06-04T05:05:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d0ab69740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556c82cba9e0 con 0x556c82ca9040 2024-06-04T05:05:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d0ab69740 10 client.0 _ll_register_callbacks cb 0x556c8291ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d053ef640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82cb7e10 0x556c82cb5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d053ef640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82cb7e10 0x556c82cb5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56482/0 (socket says 172.21.15.70:56482) 2024-06-04T05:05:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d053ef640 1 -- 172.21.15.70:0/2314806289 learned_addr learned my addr 172.21.15.70:0/2314806289 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d053ef640 1 --2- 172.21.15.70:0/2314806289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82ca8720 0x556c82cb4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d05bf0640 1 --2- 172.21.15.70:0/2314806289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82ca9040 0x556c82cb5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d053ef640 1 -- 172.21.15.70:0/2314806289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82ca9040 msgr2=0x556c82cb5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d053ef640 1 --2- 172.21.15.70:0/2314806289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82ca9040 0x556c82cb5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d053ef640 1 -- 172.21.15.70:0/2314806289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82ca8720 msgr2=0x556c82cb4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d053ef640 1 --2- 172.21.15.70:0/2314806289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82ca8720 0x556c82cb4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d053ef640 1 -- 172.21.15.70:0/2314806289 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c829eb620 con 0x556c82cb7e10 2024-06-04T05:05:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d053ef640 1 --2- 172.21.15.70:0/2314806289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82cb7e10 0x556c82cb5850 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f0cf403c860 tx=0x7f0cf403c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c647bb82076fc0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0ce57fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:05:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0ce57fa640 1 -- 172.21.15.70:0/2314806289 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4041070 con 0x556c82cb7e10 2024-06-04T05:05:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0ce57fa640 1 -- 172.21.15.70:0/2314806289 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f0cf4003b80 con 0x556c82cb7e10 2024-06-04T05:05:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.135+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2314806289 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cf4045070 con 0x556c82cb7e10 2024-06-04T05:05:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.136+0000 7f0ce57fa640 1 -- 172.21.15.70:0/2314806289 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0cf4035040 con 0x556c82cb7e10 2024-06-04T05:05:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.136+0000 7f0d0ab69740 10 client.17724 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.136+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2314806289 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556c82cc67c0 con 0x556c82cb7e10 2024-06-04T05:05:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.136+0000 7f0ce57fa640 1 -- 172.21.15.70:0/2314806289 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0cf4045070 con 0x556c82cb7e10 2024-06-04T05:05:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.136+0000 7f0ce57fa640 1 -- 172.21.15.70:0/2314806289 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f0cf403dc80 con 0x556c82cb7e10 2024-06-04T05:05:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.136+0000 7f0d0ab69740 10 client.17724 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.136+0000 7f0d0ab69740 10 client.17724 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:05:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.136+0000 7f0d0ab69740 20 client.17724 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.136+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/2314806289 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556c82cc7b10 0x556c82cc9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.136+0000 7f0d0ab69740 4 client.17724 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:05:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.136+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2314806289 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c82c1cb80 con 0x556c82cc7b10 2024-06-04T05:05:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.136+0000 7f0d05bf0640 1 --2- 172.21.15.70:0/2314806289 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556c82cc7b10 0x556c82cc9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.137+0000 7f0d05bf0640 1 --2- 172.21.15.70:0/2314806289 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556c82cc7b10 0x556c82cc9f00 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33d616ac897d8bd5 server_cookie=79499436b61ec18e in_seq=0 out_seq=0 2024-06-04T05:05:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.137+0000 7f0ce57fa640 10 client.17724 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.137+0000 7f0ce57fa640 1 -- 172.21.15.70:0/2314806289 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556c82c1cb80 con 0x556c82cc7b10 2024-06-04T05:05:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.137+0000 7f0ce57fa640 10 client.17724 handle_command_reply: tid=0 2024-06-04T05:05:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.137+0000 7f0d0ab69740 1 client.17724 shutdown 2024-06-04T05:05:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.137+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2314806289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82cb7e10 msgr2=0x556c82cb5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.137+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/2314806289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82cb7e10 0x556c82cb5850 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f0cf403c860 tx=0x7f0cf403c760 comp rx=0 tx=0).stop 2024-06-04T05:05:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.137+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2314806289 shutdown_connections 2024-06-04T05:05:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.137+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/2314806289 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556c82cc7b10 0x556c82cc9f00 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.138+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/2314806289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82cb7e10 0x556c82cb5850 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.138+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/2314806289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82ca9040 0x556c82cb5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.138+0000 7f0ce57fa640 0 client.17724 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.138+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/2314806289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82ca8720 0x556c82cb4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.138+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2314806289 >> 172.21.15.70:0/2314806289 conn(0x556c82ca4d30 msgr2=0x556c82caaba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.138+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2314806289 shutdown_connections 2024-06-04T05:05:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.138+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2314806289 wait complete. 2024-06-04T05:05:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.138+0000 7f0d0ab69740 20 client.17724 trim_cache size 0 max 16384 2024-06-04T05:05:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d0ab69740 1 Processor -- start 2024-06-04T05:05:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d0ab69740 1 -- start start 2024-06-04T05:05:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d0ab69740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82ca8720 0x556c82cc3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d0ab69740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82ca9040 0x556c82cccf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d0ab69740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82cb7e10 0x556c82ccd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d0ab69740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0d001ac3f0 con 0x556c82ca9040 2024-06-04T05:05:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d0ab69740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0d001adf30 con 0x556c82ca8720 2024-06-04T05:05:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d0ab69740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0d001acac0 con 0x556c82cb7e10 2024-06-04T05:05:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d05bf0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82ca9040 0x556c82cccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d05bf0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82ca9040 0x556c82cccf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56488/0 (socket says 172.21.15.70:56488) 2024-06-04T05:05:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d05bf0640 1 -- 172.21.15.70:0/3188700479 learned_addr learned my addr 172.21.15.70:0/3188700479 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d05bf0640 1 -- 172.21.15.70:0/3188700479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82cb7e10 msgr2=0x556c82ccd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d05bf0640 1 --2- 172.21.15.70:0/3188700479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82cb7e10 0x556c82ccd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d05bf0640 1 -- 172.21.15.70:0/3188700479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82ca8720 msgr2=0x556c82cc3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d053ef640 1 --2- 172.21.15.70:0/3188700479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82ca8720 0x556c82cc3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d05bf0640 1 --2- 172.21.15.70:0/3188700479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82ca8720 0x556c82cc3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d05bf0640 1 -- 172.21.15.70:0/3188700479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c82ca8e70 con 0x556c82ca9040 2024-06-04T05:05:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0d05bf0640 1 --2- 172.21.15.70:0/3188700479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82ca9040 0x556c82cccf20 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x556c82cb4630 tx=0x7f0cf0004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bfc121f942608cc server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0ce77fe640 1 -- 172.21.15.70:0/3188700479 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0004b20 con 0x556c82ca9040 2024-06-04T05:05:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0ce77fe640 1 -- 172.21.15.70:0/3188700479 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0cf0004cc0 con 0x556c82ca9040 2024-06-04T05:05:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.153+0000 7f0ce77fe640 1 -- 172.21.15.70:0/3188700479 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0005260 con 0x556c82ca9040 2024-06-04T05:05:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.154+0000 7f0d0ab69740 1 -- 172.21.15.70:0/3188700479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82ca9040 msgr2=0x556c82cccf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.154+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/3188700479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82ca9040 0x556c82cccf20 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x556c82cb4630 tx=0x7f0cf0004410 comp rx=0 tx=0).stop 2024-06-04T05:05:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.154+0000 7f0d0ab69740 1 -- 172.21.15.70:0/3188700479 shutdown_connections 2024-06-04T05:05:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.154+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/3188700479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82cb7e10 0x556c82ccd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.154+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/3188700479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82ca9040 0x556c82cccf20 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.154+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/3188700479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82ca8720 0x556c82cc3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.154+0000 7f0d0ab69740 1 -- 172.21.15.70:0/3188700479 >> 172.21.15.70:0/3188700479 conn(0x556c82ca4d30 msgr2=0x556c82cc2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.154+0000 7f0d0ab69740 1 -- 172.21.15.70:0/3188700479 shutdown_connections 2024-06-04T05:05:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.154+0000 7f0d0ab69740 1 -- 172.21.15.70:0/3188700479 wait complete. 2024-06-04T05:05:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.154+0000 7f0d0ab69740 1 -- start start 2024-06-04T05:05:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.154+0000 7f0d0ab69740 1 Processor -- start 2024-06-04T05:05:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0d0ab69740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82ca8720 0x556c82ccacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0d0ab69740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82ca9040 0x556c82ccb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0d0ab69740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82cb7e10 0x556c82ccb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0d0ab69740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0d001ad860 con 0x556c82ca8720 2024-06-04T05:05:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0d0ab69740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0d001ad190 con 0x556c82cb7e10 2024-06-04T05:05:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0d0ab69740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0d001aa410 con 0x556c82ca9040 2024-06-04T05:05:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0d0ab69740 10 client.0 _ll_register_callbacks cb 0x7f0cf00075a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0d053ef640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82ca8720 0x556c82ccacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0d053ef640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82ca8720 0x556c82ccacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56490/0 (socket says 172.21.15.70:56490) 2024-06-04T05:05:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0d053ef640 1 -- 172.21.15.70:0/2710550775 learned_addr learned my addr 172.21.15.70:0/2710550775 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0d053ef640 1 -- 172.21.15.70:0/2710550775 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82ca9040 msgr2=0x556c82ccb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0d05bf0640 1 --2- 172.21.15.70:0/2710550775 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82ca9040 0x556c82ccb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0d053ef640 1 --2- 172.21.15.70:0/2710550775 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82ca9040 0x556c82ccb230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0d053ef640 1 -- 172.21.15.70:0/2710550775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82cb7e10 msgr2=0x556c82ccb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0d053ef640 1 --2- 172.21.15.70:0/2710550775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82cb7e10 0x556c82ccb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0d053ef640 1 -- 172.21.15.70:0/2710550775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c82d313a0 con 0x556c82ca8720 2024-06-04T05:05:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0d053ef640 1 --2- 172.21.15.70:0/2710550775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82ca8720 0x556c82ccacf0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f0cf403c860 tx=0x7f0cf404ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cdae6a8bb51423b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.155+0000 7f0ce57fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:05:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.156+0000 7f0ce57fa640 1 -- 172.21.15.70:0/2710550775 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4065060 con 0x556c82ca8720 2024-06-04T05:05:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.156+0000 7f0ce57fa640 1 -- 172.21.15.70:0/2710550775 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f0cf4003b80 con 0x556c82ca8720 2024-06-04T05:05:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.156+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2710550775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cf0004cc0 con 0x556c82ca8720 2024-06-04T05:05:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.156+0000 7f0ce57fa640 1 -- 172.21.15.70:0/2710550775 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0cf4041040 con 0x556c82ca8720 2024-06-04T05:05:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.156+0000 7f0d0ab69740 10 client.17730 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.156+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2710550775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556c82d326e0 con 0x556c82ca8720 2024-06-04T05:05:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.156+0000 7f0ce57fa640 1 -- 172.21.15.70:0/2710550775 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0cf0004cc0 con 0x556c82ca8720 2024-06-04T05:05:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.156+0000 7f0ce57fa640 1 -- 172.21.15.70:0/2710550775 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f0cf4003d80 con 0x556c82ca8720 2024-06-04T05:05:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.156+0000 7f0d0ab69740 10 client.17730 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.156+0000 7f0d0ab69740 10 client.17730 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:05:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.156+0000 7f0d0ab69740 20 client.17730 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.156+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/2710550775 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556c82d32e60 0x556c82d35280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.156+0000 7f0d0ab69740 4 client.17730 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:05:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.156+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2710550775 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556c82c16360 con 0x556c82d32e60 2024-06-04T05:05:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.156+0000 7f0d05bf0640 1 --2- 172.21.15.70:0/2710550775 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556c82d32e60 0x556c82d35280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.157+0000 7f0d05bf0640 1 --2- 172.21.15.70:0/2710550775 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556c82d32e60 0x556c82d35280 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2189cb1534f87ce server_cookie=dfb500dcc9c6ccd7 in_seq=0 out_seq=0 2024-06-04T05:05:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.157+0000 7f0ce57fa640 10 client.17730 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.175+0000 7f0ce57fa640 1 -- 172.21.15.70:0/2710550775 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x556c82c16360 con 0x556c82d32e60 2024-06-04T05:05:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.175+0000 7f0ce57fa640 10 client.17730 handle_command_reply: tid=0 2024-06-04T05:05:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.175+0000 7f0d0ab69740 1 client.17730 shutdown 2024-06-04T05:05:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.175+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2710550775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82ca8720 msgr2=0x556c82ccacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.175+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/2710550775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82ca8720 0x556c82ccacf0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f0cf403c860 tx=0x7f0cf404ca10 comp rx=0 tx=0).stop 2024-06-04T05:05:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.175+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2710550775 shutdown_connections 2024-06-04T05:05:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.175+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/2710550775 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556c82d32e60 0x556c82d35280 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.175+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/2710550775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c82cb7e10 0x556c82ccb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.175+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/2710550775 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556c82ca9040 0x556c82ccb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.175+0000 7f0ce57fa640 0 client.17730 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.175+0000 7f0d0ab69740 1 --2- 172.21.15.70:0/2710550775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c82ca8720 0x556c82ccacf0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.175+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2710550775 >> 172.21.15.70:0/2710550775 conn(0x556c82ca4d30 msgr2=0x556c82cc3500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.175+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2710550775 shutdown_connections 2024-06-04T05:05:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.175+0000 7f0d0ab69740 1 -- 172.21.15.70:0/2710550775 wait complete. 2024-06-04T05:05:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.176+0000 7f0d0ab69740 20 client.17730 trim_cache size 0 max 16384 2024-06-04T05:05:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:05:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:05:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:05:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.176+0000 7f0d07e54640 1 -- 172.21.15.70:0/387981233 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0ce80b2a10 msgr2=0x7f0ce80b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.176+0000 7f0d07e54640 1 --2- 172.21.15.70:0/387981233 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0ce80b2a10 0x7f0ce80b4ed0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80a8cd0 tx=0x7f0cf8075900 comp rx=0 tx=0).stop 2024-06-04T05:05:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.176+0000 7f0d07e54640 1 -- 172.21.15.70:0/387981233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d0010d020 msgr2=0x7f0d0019d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.176+0000 7f0d07e54640 1 --2- 172.21.15.70:0/387981233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d0010d020 0x7f0d0019d990 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8009fd0 tx=0x7f0cf800b8e0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.176+0000 7f0d07e54640 1 -- 172.21.15.70:0/387981233 shutdown_connections 2024-06-04T05:05:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.176+0000 7f0d07e54640 1 --2- 172.21.15.70:0/387981233 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0ce80b2a10 0x7f0ce80b4ed0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.176+0000 7f0d07e54640 1 --2- 172.21.15.70:0/387981233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d00104600 0x7f0d00199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.176+0000 7f0d07e54640 1 --2- 172.21.15.70:0/387981233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d0010d020 0x7f0d0019d990 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.176+0000 7f0d07e54640 1 --2- 172.21.15.70:0/387981233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d00103ce0 0x7f0d001990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.176+0000 7f0d07e54640 1 -- 172.21.15.70:0/387981233 >> 172.21.15.70:0/387981233 conn(0x7f0d00078ad0 msgr2=0x7f0d000ffad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.176+0000 7f0d07e54640 1 -- 172.21.15.70:0/387981233 shutdown_connections 2024-06-04T05:05:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:43.176+0000 7f0d07e54640 1 -- 172.21.15.70:0/387981233 wait complete. 2024-06-04T05:05:43.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:43 smithi112 ceph-mon[36453]: pgmap v261: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 697 KiB/s wr, 68 op/s 2024-06-04T05:05:43.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:43 smithi098 ceph-mon[35569]: pgmap v261: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 697 KiB/s wr, 68 op/s 2024-06-04T05:05:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:43 smithi070 ceph-mon[32631]: pgmap v261: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 697 KiB/s wr, 68 op/s 2024-06-04T05:05:43.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.03 MB/sec execute 276 sec latency 13827.981 ms 2024-06-04T05:05:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.415+0000 7ff965cb5700 1 -- 172.21.15.70:0/899755533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff960107460 msgr2=0x7ff960107840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.415+0000 7ff965cb5700 1 --2- 172.21.15.70:0/899755533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff960107460 0x7ff960107840 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ff948009990 tx=0x7ff948009ca0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.415+0000 7ff965cb5700 1 -- 172.21.15.70:0/899755533 shutdown_connections 2024-06-04T05:05:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.415+0000 7ff965cb5700 1 --2- 172.21.15.70:0/899755533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9601030e0 0x7ff96010f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.415+0000 7ff965cb5700 1 --2- 172.21.15.70:0/899755533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff960107460 0x7ff960107840 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.415+0000 7ff965cb5700 1 --2- 172.21.15.70:0/899755533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff960102720 0x7ff960102ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.415+0000 7ff965cb5700 1 -- 172.21.15.70:0/899755533 >> 172.21.15.70:0/899755533 conn(0x7ff960079300 msgr2=0x7ff960079710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:44.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.415+0000 7ff965cb5700 1 -- 172.21.15.70:0/899755533 shutdown_connections 2024-06-04T05:05:44.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.415+0000 7ff965cb5700 1 -- 172.21.15.70:0/899755533 wait complete. 2024-06-04T05:05:44.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.416+0000 7ff965cb5700 1 Processor -- start 2024-06-04T05:05:44.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.416+0000 7ff965cb5700 1 -- start start 2024-06-04T05:05:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.416+0000 7ff965cb5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff960102720 0x7ff96019aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.416+0000 7ff965cb5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9601030e0 0x7ff96019af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.416+0000 7ff965cb5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff960107460 0x7ff9601a5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.416+0000 7ff965cb5700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9601125f0 con 0x7ff9601030e0 2024-06-04T05:05:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.416+0000 7ff965cb5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff960112470 con 0x7ff960107460 2024-06-04T05:05:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.416+0000 7ff965cb5700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff960112770 con 0x7ff960102720 2024-06-04T05:05:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.416+0000 7ff95effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9601030e0 0x7ff96019af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.416+0000 7ff95effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9601030e0 0x7ff96019af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56514/0 (socket says 172.21.15.70:56514) 2024-06-04T05:05:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.416+0000 7ff95effd700 1 -- 172.21.15.70:0/4027813713 learned_addr learned my addr 172.21.15.70:0/4027813713 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.417+0000 7ff95f7fe700 1 --2- 172.21.15.70:0/4027813713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff960102720 0x7ff96019aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.417+0000 7ff95ffff700 1 --2- 172.21.15.70:0/4027813713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff960107460 0x7ff9601a5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.417+0000 7ff95effd700 1 -- 172.21.15.70:0/4027813713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff960102720 msgr2=0x7ff96019aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.417+0000 7ff95effd700 1 --2- 172.21.15.70:0/4027813713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff960102720 0x7ff96019aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.417+0000 7ff95effd700 1 -- 172.21.15.70:0/4027813713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff960107460 msgr2=0x7ff9601a5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.417+0000 7ff95effd700 1 --2- 172.21.15.70:0/4027813713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff960107460 0x7ff9601a5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.417+0000 7ff95effd700 1 -- 172.21.15.70:0/4027813713 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9601a5ff0 con 0x7ff9601030e0 2024-06-04T05:05:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.417+0000 7ff95ffff700 1 --2- 172.21.15.70:0/4027813713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff960107460 0x7ff9601a5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:05:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.417+0000 7ff95effd700 1 --2- 172.21.15.70:0/4027813713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9601030e0 0x7ff96019af70 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ff95000d950 tx=0x7ff95000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.417+0000 7ff95cff9700 1 -- 172.21.15.70:0/4027813713 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950016070 con 0x7ff9601030e0 2024-06-04T05:05:44.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.417+0000 7ff95cff9700 1 -- 172.21.15.70:0/4027813713 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff950010460 con 0x7ff9601030e0 2024-06-04T05:05:44.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.417+0000 7ff965cb5700 1 -- 172.21.15.70:0/4027813713 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9601a6220 con 0x7ff9601030e0 2024-06-04T05:05:44.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.417+0000 7ff95cff9700 1 -- 172.21.15.70:0/4027813713 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95000f620 con 0x7ff9601030e0 2024-06-04T05:05:44.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.417+0000 7ff965cb5700 1 -- 172.21.15.70:0/4027813713 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9601a6780 con 0x7ff9601030e0 2024-06-04T05:05:44.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.420+0000 7ff95cff9700 1 -- 172.21.15.70:0/4027813713 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff950019070 con 0x7ff9601030e0 2024-06-04T05:05:44.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.421+0000 7ff95cff9700 1 --2- 172.21.15.70:0/4027813713 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff94c0b2a70 0x7ff94c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.421+0000 7ff95f7fe700 1 --2- 172.21.15.70:0/4027813713 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff94c0b2a70 0x7ff94c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.421+0000 7ff95cff9700 1 -- 172.21.15.70:0/4027813713 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7ff9500e21f0 con 0x7ff9601030e0 2024-06-04T05:05:44.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.421+0000 7ff97054ab80 1 Processor -- start 2024-06-04T05:05:44.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.421+0000 7ff97054ab80 1 -- start start 2024-06-04T05:05:44.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.421+0000 7ff95f7fe700 1 --2- 172.21.15.70:0/4027813713 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff94c0b2a70 0x7ff94c0b4f30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff94800b5c0 tx=0x7ff94801a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:44.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.421+0000 7ff97054ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a313c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff97054ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a314210 0x55b46a317940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff97054ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a317f60 0x55b46a3183a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff97054ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b46a272c90 con 0x55b46a313860 2024-06-04T05:05:44.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff97054ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b46a262830 con 0x55b46a317f60 2024-06-04T05:05:44.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff97054ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b46a266870 con 0x55b46a314210 2024-06-04T05:05:44.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff95f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a313c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff95f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a313c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56520/0 (socket says 172.21.15.70:56520) 2024-06-04T05:05:44.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff95f7fe700 1 -- 172.21.15.70:0/694842341 learned_addr learned my addr 172.21.15.70:0/694842341 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:44.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff95effd700 1 --2- 172.21.15.70:0/694842341 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a314210 0x55b46a317940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff95ffff700 1 --2- 172.21.15.70:0/694842341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a317f60 0x55b46a3183a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff95f7fe700 1 -- 172.21.15.70:0/694842341 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a314210 msgr2=0x55b46a317940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff95f7fe700 1 --2- 172.21.15.70:0/694842341 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a314210 0x55b46a317940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff95f7fe700 1 -- 172.21.15.70:0/694842341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a317f60 msgr2=0x55b46a3183a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff95f7fe700 1 --2- 172.21.15.70:0/694842341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a317f60 0x55b46a3183a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff95f7fe700 1 -- 172.21.15.70:0/694842341 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b46a1b28d0 con 0x55b46a313860 2024-06-04T05:05:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff95f7fe700 1 --2- 172.21.15.70:0/694842341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a313c40 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ff948015000 tx=0x7ff94801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acd8c07ce12a599f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff955ffb700 1 -- 172.21.15.70:0/694842341 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94800f8f0 con 0x55b46a313860 2024-06-04T05:05:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff955ffb700 1 -- 172.21.15.70:0/694842341 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff948032070 con 0x55b46a313860 2024-06-04T05:05:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.422+0000 7ff955ffb700 1 -- 172.21.15.70:0/694842341 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff948023d30 con 0x55b46a313860 2024-06-04T05:05:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.423+0000 7ff97054ab80 1 -- 172.21.15.70:0/694842341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 msgr2=0x55b46a313c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.423+0000 7ff97054ab80 1 --2- 172.21.15.70:0/694842341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a313c40 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ff948015000 tx=0x7ff94801e000 comp rx=0 tx=0).stop 2024-06-04T05:05:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.423+0000 7ff97054ab80 1 -- 172.21.15.70:0/694842341 shutdown_connections 2024-06-04T05:05:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.423+0000 7ff97054ab80 1 --2- 172.21.15.70:0/694842341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a317f60 0x55b46a3183a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.423+0000 7ff97054ab80 1 --2- 172.21.15.70:0/694842341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a313c40 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.423+0000 7ff97054ab80 1 --2- 172.21.15.70:0/694842341 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a314210 0x55b46a317940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.423+0000 7ff97054ab80 1 -- 172.21.15.70:0/694842341 >> 172.21.15.70:0/694842341 conn(0x55b46a241080 msgr2=0x55b46a30de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.423+0000 7ff97054ab80 1 -- 172.21.15.70:0/694842341 shutdown_connections 2024-06-04T05:05:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.423+0000 7ff97054ab80 1 -- 172.21.15.70:0/694842341 wait complete. 2024-06-04T05:05:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.424+0000 7ff97054ab80 1 -- start start 2024-06-04T05:05:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.424+0000 7ff97054ab80 1 Processor -- start 2024-06-04T05:05:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.424+0000 7ff97054ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a31f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.424+0000 7ff97054ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a314210 0x55b46a31c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.424+0000 7ff97054ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a317f60 0x55b46a31cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.424+0000 7ff97054ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b46a168fa0 con 0x55b46a313860 2024-06-04T05:05:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.424+0000 7ff97054ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b46a2047c0 con 0x55b46a317f60 2024-06-04T05:05:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.424+0000 7ff97054ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b46a3200c0 con 0x55b46a314210 2024-06-04T05:05:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.424+0000 7ff97054ab80 10 client.0 _ll_register_callbacks cb 0x55b46a1fd920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.424+0000 7ff95f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a31f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.424+0000 7ff95f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a31f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56530/0 (socket says 172.21.15.70:56530) 2024-06-04T05:05:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.424+0000 7ff95f7fe700 1 -- 172.21.15.70:0/2413561328 learned_addr learned my addr 172.21.15.70:0/2413561328 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.424+0000 7ff95ffff700 1 --2- 172.21.15.70:0/2413561328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a317f60 0x55b46a31cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.424+0000 7ff95effd700 1 --2- 172.21.15.70:0/2413561328 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a314210 0x55b46a31c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.425+0000 7ff95f7fe700 1 -- 172.21.15.70:0/2413561328 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a314210 msgr2=0x55b46a31c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.425+0000 7ff95f7fe700 1 --2- 172.21.15.70:0/2413561328 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a314210 0x55b46a31c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.425+0000 7ff95f7fe700 1 -- 172.21.15.70:0/2413561328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a317f60 msgr2=0x55b46a31cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.425+0000 7ff95f7fe700 1 --2- 172.21.15.70:0/2413561328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a317f60 0x55b46a31cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.425+0000 7ff95f7fe700 1 -- 172.21.15.70:0/2413561328 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b46a201690 con 0x55b46a313860 2024-06-04T05:05:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.425+0000 7ff95effd700 1 --2- 172.21.15.70:0/2413561328 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a314210 0x55b46a31c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:05:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.425+0000 7ff95f7fe700 1 --2- 172.21.15.70:0/2413561328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a31f370 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7ff948000c00 tx=0x7ff948023870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e222ea75bc15a08f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.425+0000 7ff93effd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:05:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.425+0000 7ff93effd700 1 -- 172.21.15.70:0/2413561328 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94800f8f0 con 0x55b46a313860 2024-06-04T05:05:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.425+0000 7ff93effd700 1 -- 172.21.15.70:0/2413561328 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7ff94800fa90 con 0x55b46a313860 2024-06-04T05:05:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.425+0000 7ff97054ab80 1 -- 172.21.15.70:0/2413561328 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff948032070 con 0x55b46a313860 2024-06-04T05:05:44.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.425+0000 7ff93effd700 1 -- 172.21.15.70:0/2413561328 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff94800fc90 con 0x55b46a313860 2024-06-04T05:05:44.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.425+0000 7ff97054ab80 10 client.17742 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:44.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.425+0000 7ff97054ab80 1 -- 172.21.15.70:0/2413561328 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b46a32a780 con 0x55b46a313860 2024-06-04T05:05:44.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.425+0000 7ff93effd700 1 -- 172.21.15.70:0/2413561328 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff948032070 con 0x55b46a313860 2024-06-04T05:05:44.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.425+0000 7ff93effd700 1 -- 172.21.15.70:0/2413561328 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ff9480053b0 con 0x55b46a313860 2024-06-04T05:05:44.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.426+0000 7ff97054ab80 10 client.17742 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:44.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.426+0000 7ff97054ab80 10 client.17742 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:05:44.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.426+0000 7ff97054ab80 20 client.17742 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:44.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.426+0000 7ff97054ab80 1 --2- 172.21.15.70:0/2413561328 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55b46a32cc50 0x55b46a32f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.426+0000 7ff97054ab80 4 client.17742 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:05:44.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.426+0000 7ff97054ab80 1 -- 172.21.15.70:0/2413561328 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b46a2df9e0 con 0x55b46a32cc50 2024-06-04T05:05:44.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.426+0000 7ff95effd700 1 --2- 172.21.15.70:0/2413561328 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55b46a32cc50 0x55b46a32f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.426+0000 7ff93effd700 10 client.17742 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:05:44.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.426+0000 7ff95effd700 1 --2- 172.21.15.70:0/2413561328 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55b46a32cc50 0x55b46a32f040 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e7ce08ad70030a90 server_cookie=999ef86f411747da in_seq=0 out_seq=0 2024-06-04T05:05:44.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.427+0000 7ff93effd700 1 -- 172.21.15.70:0/2413561328 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b46a2df9e0 con 0x55b46a32cc50 2024-06-04T05:05:44.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.427+0000 7ff93effd700 10 client.17742 handle_command_reply: tid=0 2024-06-04T05:05:44.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.427+0000 7ff97054ab80 1 client.17742 shutdown 2024-06-04T05:05:44.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.427+0000 7ff97054ab80 1 -- 172.21.15.70:0/2413561328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 msgr2=0x55b46a31f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:44.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.427+0000 7ff97054ab80 1 --2- 172.21.15.70:0/2413561328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a31f370 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7ff948000c00 tx=0x7ff948023870 comp rx=0 tx=0).stop 2024-06-04T05:05:44.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.427+0000 7ff97054ab80 1 -- 172.21.15.70:0/2413561328 shutdown_connections 2024-06-04T05:05:44.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.427+0000 7ff97054ab80 1 --2- 172.21.15.70:0/2413561328 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55b46a32cc50 0x55b46a32f040 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.427+0000 7ff97054ab80 1 --2- 172.21.15.70:0/2413561328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a317f60 0x55b46a31cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.427+0000 7ff97054ab80 1 --2- 172.21.15.70:0/2413561328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a31f370 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.427+0000 7ff97054ab80 1 --2- 172.21.15.70:0/2413561328 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a314210 0x55b46a31c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.427+0000 7ff97054ab80 1 -- 172.21.15.70:0/2413561328 >> 172.21.15.70:0/2413561328 conn(0x55b46a241080 msgr2=0x55b46a30d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:44.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.427+0000 7ff93effd700 0 client.17742 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:05:44.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.427+0000 7ff97054ab80 1 -- 172.21.15.70:0/2413561328 shutdown_connections 2024-06-04T05:05:44.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.427+0000 7ff97054ab80 1 -- 172.21.15.70:0/2413561328 wait complete. 2024-06-04T05:05:44.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.427+0000 7ff97054ab80 20 client.17742 trim_cache size 0 max 16384 2024-06-04T05:05:44.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.442+0000 7ff97054ab80 1 Processor -- start 2024-06-04T05:05:44.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.442+0000 7ff97054ab80 1 -- start start 2024-06-04T05:05:44.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.442+0000 7ff97054ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a312d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.442+0000 7ff97054ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a314210 0x55b46a30dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.442+0000 7ff97054ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a317f60 0x55b46a328350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.443+0000 7ff97054ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9601ade90 con 0x55b46a313860 2024-06-04T05:05:44.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.443+0000 7ff97054ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff9601af9d0 con 0x55b46a314210 2024-06-04T05:05:44.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.443+0000 7ff97054ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff9601ae560 con 0x55b46a317f60 2024-06-04T05:05:44.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.443+0000 7ff95f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a312d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.443+0000 7ff95f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a312d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56538/0 (socket says 172.21.15.70:56538) 2024-06-04T05:05:44.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.443+0000 7ff95ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a317f60 0x55b46a328350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.443+0000 7ff95f7fe700 1 -- 172.21.15.70:0/1255558763 learned_addr learned my addr 172.21.15.70:0/1255558763 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:44.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.443+0000 7ff95effd700 1 --2- 172.21.15.70:0/1255558763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a314210 0x55b46a30dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.443+0000 7ff95f7fe700 1 -- 172.21.15.70:0/1255558763 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a317f60 msgr2=0x55b46a328350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:44.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.443+0000 7ff95f7fe700 1 --2- 172.21.15.70:0/1255558763 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a317f60 0x55b46a328350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.443+0000 7ff95f7fe700 1 -- 172.21.15.70:0/1255558763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a314210 msgr2=0x55b46a30dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:44.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.443+0000 7ff95f7fe700 1 --2- 172.21.15.70:0/1255558763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a314210 0x55b46a30dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.443+0000 7ff95f7fe700 1 -- 172.21.15.70:0/1255558763 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b46a31f6e0 con 0x55b46a313860 2024-06-04T05:05:44.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.443+0000 7ff95f7fe700 1 --2- 172.21.15.70:0/1255558763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a312d80 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff948009990 tx=0x7ff9480405d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b59ad88abba56d8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:44.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.443+0000 7ff9557fa700 1 -- 172.21.15.70:0/1255558763 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94804a070 con 0x55b46a313860 2024-06-04T05:05:44.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.443+0000 7ff9557fa700 1 -- 172.21.15.70:0/1255558763 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff948032040 con 0x55b46a313860 2024-06-04T05:05:44.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.443+0000 7ff9557fa700 1 -- 172.21.15.70:0/1255558763 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9480402e0 con 0x55b46a313860 2024-06-04T05:05:44.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.444+0000 7ff97054ab80 1 -- 172.21.15.70:0/1255558763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 msgr2=0x55b46a312d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:44.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.444+0000 7ff97054ab80 1 --2- 172.21.15.70:0/1255558763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a312d80 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff948009990 tx=0x7ff9480405d0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.444+0000 7ff97054ab80 1 -- 172.21.15.70:0/1255558763 shutdown_connections 2024-06-04T05:05:44.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.444+0000 7ff97054ab80 1 --2- 172.21.15.70:0/1255558763 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a317f60 0x55b46a328350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.444+0000 7ff97054ab80 1 --2- 172.21.15.70:0/1255558763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a313860 0x55b46a312d80 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.444+0000 7ff97054ab80 1 --2- 172.21.15.70:0/1255558763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a314210 0x55b46a30dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.444+0000 7ff97054ab80 1 -- 172.21.15.70:0/1255558763 >> 172.21.15.70:0/1255558763 conn(0x55b46a241080 msgr2=0x55b46a312670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:44.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.444+0000 7ff97054ab80 1 -- 172.21.15.70:0/1255558763 shutdown_connections 2024-06-04T05:05:44.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.444+0000 7ff97054ab80 1 -- 172.21.15.70:0/1255558763 wait complete. 2024-06-04T05:05:44.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.444+0000 7ff97054ab80 1 -- start start 2024-06-04T05:05:44.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff97054ab80 1 Processor -- start 2024-06-04T05:05:44.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff97054ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a313860 0x55b46a38e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff97054ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a314210 0x55b46a38ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff97054ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a317f60 0x55b46a331210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff97054ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9601af300 con 0x55b46a314210 2024-06-04T05:05:44.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff97054ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff9601aec30 con 0x55b46a317f60 2024-06-04T05:05:44.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff97054ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff9601abe50 con 0x55b46a313860 2024-06-04T05:05:44.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff97054ab80 10 client.0 _ll_register_callbacks cb 0x55b46a1fd920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:44.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff95effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a314210 0x55b46a38ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff95effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a314210 0x55b46a38ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56550/0 (socket says 172.21.15.70:56550) 2024-06-04T05:05:44.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff95effd700 1 -- 172.21.15.70:0/243544374 learned_addr learned my addr 172.21.15.70:0/243544374 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:44.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff95f7fe700 1 --2- 172.21.15.70:0/243544374 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a313860 0x55b46a38e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff95ffff700 1 --2- 172.21.15.70:0/243544374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a317f60 0x55b46a331210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff95effd700 1 -- 172.21.15.70:0/243544374 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a313860 msgr2=0x55b46a38e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:44.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff95effd700 1 --2- 172.21.15.70:0/243544374 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a313860 0x55b46a38e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff95effd700 1 -- 172.21.15.70:0/243544374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a317f60 msgr2=0x55b46a331210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:44.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff95effd700 1 --2- 172.21.15.70:0/243544374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a317f60 0x55b46a331210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff95effd700 1 -- 172.21.15.70:0/243544374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b46a331a50 con 0x55b46a314210 2024-06-04T05:05:44.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.445+0000 7ff95ffff700 1 --2- 172.21.15.70:0/243544374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a317f60 0x55b46a331210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:05:44.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.446+0000 7ff95effd700 1 --2- 172.21.15.70:0/243544374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a314210 0x55b46a38ed50 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ff9500df9c0 tx=0x7ff95000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c73d5be1ff2c1a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:44.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.446+0000 7ff93effd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:05:44.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.446+0000 7ff93effd700 1 -- 172.21.15.70:0/243544374 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950014070 con 0x55b46a314210 2024-06-04T05:05:44.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.446+0000 7ff93effd700 1 -- 172.21.15.70:0/243544374 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7ff9500a9700 con 0x55b46a314210 2024-06-04T05:05:44.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.446+0000 7ff97054ab80 1 -- 172.21.15.70:0/243544374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff948032040 con 0x55b46a314210 2024-06-04T05:05:44.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.446+0000 7ff93effd700 1 -- 172.21.15.70:0/243544374 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff950020040 con 0x55b46a314210 2024-06-04T05:05:44.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.446+0000 7ff97054ab80 10 client.17748 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:44.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.446+0000 7ff97054ab80 1 -- 172.21.15.70:0/243544374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b46a396fd0 con 0x55b46a314210 2024-06-04T05:05:44.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.446+0000 7ff93effd700 1 -- 172.21.15.70:0/243544374 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff948032040 con 0x55b46a314210 2024-06-04T05:05:44.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.446+0000 7ff93effd700 1 -- 172.21.15.70:0/243544374 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ff9500b1e70 con 0x55b46a314210 2024-06-04T05:05:44.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.446+0000 7ff97054ab80 10 client.17748 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:44.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.446+0000 7ff97054ab80 10 client.17748 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:05:44.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.446+0000 7ff97054ab80 20 client.17748 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:44.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.446+0000 7ff97054ab80 1 --2- 172.21.15.70:0/243544374 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55b46a397690 0x55b46a399ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:44.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.446+0000 7ff97054ab80 4 client.17748 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:05:44.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.446+0000 7ff97054ab80 1 -- 172.21.15.70:0/243544374 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b46a274be0 con 0x55b46a397690 2024-06-04T05:05:44.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff95f7fe700 1 --2- 172.21.15.70:0/243544374 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55b46a397690 0x55b46a399ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:44.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff95f7fe700 1 --2- 172.21.15.70:0/243544374 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55b46a397690 0x55b46a399ab0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3962c28e201c289 server_cookie=74f68887f722cf88 in_seq=0 out_seq=0 2024-06-04T05:05:44.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff93effd700 10 client.17748 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:05:44.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff93effd700 1 -- 172.21.15.70:0/243544374 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b46a274be0 con 0x55b46a397690 2024-06-04T05:05:44.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff93effd700 10 client.17748 handle_command_reply: tid=0 2024-06-04T05:05:44.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff97054ab80 1 client.17748 shutdown 2024-06-04T05:05:44.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff97054ab80 1 -- 172.21.15.70:0/243544374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a314210 msgr2=0x55b46a38ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:44.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff97054ab80 1 --2- 172.21.15.70:0/243544374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a314210 0x55b46a38ed50 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ff9500df9c0 tx=0x7ff95000b5b0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff97054ab80 1 -- 172.21.15.70:0/243544374 shutdown_connections 2024-06-04T05:05:44.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff97054ab80 1 --2- 172.21.15.70:0/243544374 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55b46a397690 0x55b46a399ab0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff97054ab80 1 --2- 172.21.15.70:0/243544374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b46a317f60 0x55b46a331210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff97054ab80 1 --2- 172.21.15.70:0/243544374 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b46a313860 0x55b46a38e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff97054ab80 1 --2- 172.21.15.70:0/243544374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b46a314210 0x55b46a38ed50 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff97054ab80 1 -- 172.21.15.70:0/243544374 >> 172.21.15.70:0/243544374 conn(0x55b46a241080 msgr2=0x55b46a312f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:44.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff93effd700 0 client.17748 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:05:44.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff97054ab80 1 -- 172.21.15.70:0/243544374 shutdown_connections 2024-06-04T05:05:44.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff97054ab80 1 -- 172.21.15.70:0/243544374 wait complete. 2024-06-04T05:05:44.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.447+0000 7ff97054ab80 20 client.17748 trim_cache size 0 max 16384 2024-06-04T05:05:44.455 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:05:44.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.448+0000 7ff965cb5700 1 -- 172.21.15.70:0/4027813713 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff94c0b2a70 msgr2=0x7ff94c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:44.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.448+0000 7ff965cb5700 1 --2- 172.21.15.70:0/4027813713 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff94c0b2a70 0x7ff94c0b4f30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff94800b5c0 tx=0x7ff94801a040 comp rx=0 tx=0).stop 2024-06-04T05:05:44.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.448+0000 7ff965cb5700 1 -- 172.21.15.70:0/4027813713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9601030e0 msgr2=0x7ff96019af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:44.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.448+0000 7ff965cb5700 1 --2- 172.21.15.70:0/4027813713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9601030e0 0x7ff96019af70 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ff95000d950 tx=0x7ff95000dd10 comp rx=0 tx=0).stop 2024-06-04T05:05:44.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.448+0000 7ff965cb5700 1 -- 172.21.15.70:0/4027813713 shutdown_connections 2024-06-04T05:05:44.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.448+0000 7ff965cb5700 1 --2- 172.21.15.70:0/4027813713 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff94c0b2a70 0x7ff94c0b4f30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.448+0000 7ff965cb5700 1 --2- 172.21.15.70:0/4027813713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff960107460 0x7ff9601a5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.448+0000 7ff965cb5700 1 --2- 172.21.15.70:0/4027813713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff960102720 0x7ff96019aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.448+0000 7ff965cb5700 1 --2- 172.21.15.70:0/4027813713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9601030e0 0x7ff96019af70 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:44.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.448+0000 7ff965cb5700 1 -- 172.21.15.70:0/4027813713 >> 172.21.15.70:0/4027813713 conn(0x7ff960079300 msgr2=0x7ff960100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:44.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.448+0000 7ff965cb5700 1 -- 172.21.15.70:0/4027813713 shutdown_connections 2024-06-04T05:05:44.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:44.448+0000 7ff965cb5700 1 -- 172.21.15.70:0/4027813713 wait complete. 2024-06-04T05:05:44.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.03 MB/sec execute 277 sec latency 14828.034 ms 2024-06-04T05:05:45.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:45 smithi112 ceph-mon[36453]: pgmap v262: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-06-04T05:05:45.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:45 smithi098 ceph-mon[35569]: pgmap v262: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-06-04T05:05:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:45 smithi070 ceph-mon[32631]: pgmap v262: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-06-04T05:05:45.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.02 MB/sec execute 278 sec latency 15828.119 ms 2024-06-04T05:05:45.851 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:05:46.617 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:05:46.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.01 MB/sec execute 279 sec latency 16828.164 ms 2024-06-04T05:05:46.741 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:05:47.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:47 smithi112 ceph-mon[36453]: pgmap v263: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 42 KiB/s wr, 19 op/s 2024-06-04T05:05:47.459 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:05:47.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:47 smithi098 ceph-mon[35569]: pgmap v263: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 42 KiB/s wr, 19 op/s 2024-06-04T05:05:47.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:47 smithi070 ceph-mon[32631]: pgmap v263: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 42 KiB/s wr, 19 op/s 2024-06-04T05:05:47.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.01 MB/sec execute 280 sec latency 17828.239 ms 2024-06-04T05:05:48.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 2.00 MB/sec execute 281 sec latency 18828.284 ms 2024-06-04T05:05:49.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.213+0000 7f3bee5d0700 1 -- 172.21.15.70:0/2186999630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3be8102e20 msgr2=0x7f3be810f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:49.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.213+0000 7f3bee5d0700 1 --2- 172.21.15.70:0/2186999630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3be8102e20 0x7f3be810f330 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f3be000b0a0 tx=0x7f3be000b3b0 comp rx=0 tx=0).stop 2024-06-04T05:05:49.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.213+0000 7f3bee5d0700 1 -- 172.21.15.70:0/2186999630 shutdown_connections 2024-06-04T05:05:49.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.213+0000 7f3bee5d0700 1 --2- 172.21.15.70:0/2186999630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3be8102e20 0x7f3be810f330 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:49.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.213+0000 7f3bee5d0700 1 --2- 172.21.15.70:0/2186999630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3be81084f0 0x7f3be81088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:49.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.213+0000 7f3bee5d0700 1 --2- 172.21.15.70:0/2186999630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3be8102460 0x7f3be81028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:49.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.213+0000 7f3bee5d0700 1 -- 172.21.15.70:0/2186999630 >> 172.21.15.70:0/2186999630 conn(0x7f3be80794e0 msgr2=0x7f3be80798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:49.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.213+0000 7f3bee5d0700 1 -- 172.21.15.70:0/2186999630 shutdown_connections 2024-06-04T05:05:49.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.213+0000 7f3bee5d0700 1 -- 172.21.15.70:0/2186999630 wait complete. 2024-06-04T05:05:49.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.214+0000 7f3bee5d0700 1 Processor -- start 2024-06-04T05:05:49.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.214+0000 7f3bee5d0700 1 -- start start 2024-06-04T05:05:49.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.214+0000 7f3bee5d0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3be8102460 0x7f3be819f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:49.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.214+0000 7f3bee5d0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3be8102e20 0x7f3be819fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:49.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.214+0000 7f3bee5d0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3be81084f0 0x7f3be8199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:49.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.214+0000 7f3bee5d0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3be81124b0 con 0x7f3be8102e20 2024-06-04T05:05:49.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.214+0000 7f3bee5d0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3be8112330 con 0x7f3be8102460 2024-06-04T05:05:49.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.214+0000 7f3bee5d0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3be8112630 con 0x7f3be81084f0 2024-06-04T05:05:49.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.215+0000 7f3be77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3be8102e20 0x7f3be819fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:49.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.215+0000 7f3be77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3be8102e20 0x7f3be819fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45582/0 (socket says 172.21.15.70:45582) 2024-06-04T05:05:49.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.215+0000 7f3be77fe700 1 -- 172.21.15.70:0/507013668 learned_addr learned my addr 172.21.15.70:0/507013668 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:49.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.215+0000 7f3be7fff700 1 --2- 172.21.15.70:0/507013668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3be8102460 0x7f3be819f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:49.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.215+0000 7f3becb6d700 1 --2- 172.21.15.70:0/507013668 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3be81084f0 0x7f3be8199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:49.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.215+0000 7f3be77fe700 1 -- 172.21.15.70:0/507013668 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3be81084f0 msgr2=0x7f3be8199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:49.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.215+0000 7f3be77fe700 1 --2- 172.21.15.70:0/507013668 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3be81084f0 0x7f3be8199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:49.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.215+0000 7f3be77fe700 1 -- 172.21.15.70:0/507013668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3be8102460 msgr2=0x7f3be819f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:49.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.215+0000 7f3be77fe700 1 --2- 172.21.15.70:0/507013668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3be8102460 0x7f3be819f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:49.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.215+0000 7f3be77fe700 1 -- 172.21.15.70:0/507013668 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be819a010 con 0x7f3be8102e20 2024-06-04T05:05:49.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.215+0000 7f3be77fe700 1 --2- 172.21.15.70:0/507013668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3be8102e20 0x7f3be819fbf0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc00c9b0 tx=0x7f3bdc00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:49.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.215+0000 7f3be57fa700 1 -- 172.21.15.70:0/507013668 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc007b00 con 0x7f3be8102e20 2024-06-04T05:05:49.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.215+0000 7f3be57fa700 1 -- 172.21.15.70:0/507013668 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3bdc007ca0 con 0x7f3be8102e20 2024-06-04T05:05:49.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.215+0000 7f3bee5d0700 1 -- 172.21.15.70:0/507013668 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be819a300 con 0x7f3be8102e20 2024-06-04T05:05:49.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.215+0000 7f3be57fa700 1 -- 172.21.15.70:0/507013668 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc018660 con 0x7f3be8102e20 2024-06-04T05:05:49.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.215+0000 7f3bee5d0700 1 -- 172.21.15.70:0/507013668 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be81a6510 con 0x7f3be8102e20 2024-06-04T05:05:49.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.216+0000 7f3bee5d0700 1 -- 172.21.15.70:0/507013668 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3be8103ba0 con 0x7f3be8102e20 2024-06-04T05:05:49.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.219+0000 7f3be57fa700 1 -- 172.21.15.70:0/507013668 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3bdc00f450 con 0x7f3be8102e20 2024-06-04T05:05:49.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.219+0000 7f3be57fa700 1 --2- 172.21.15.70:0/507013668 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3bd00b29f0 0x7f3bd00b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:49.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.219+0000 7f3be7fff700 1 --2- 172.21.15.70:0/507013668 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3bd00b29f0 0x7f3bd00b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:49.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.219+0000 7f3be57fa700 1 -- 172.21.15.70:0/507013668 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f3bdc0e2990 con 0x7f3be8102e20 2024-06-04T05:05:49.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.220+0000 7f3be7fff700 1 --2- 172.21.15.70:0/507013668 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3bd00b29f0 0x7f3bd00b4eb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800abd0 tx=0x7f3bd800a380 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:49.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.223+0000 7f3be57fa700 1 -- 172.21.15.70:0/507013668 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3bdc014030 con 0x7f3be8102e20 2024-06-04T05:05:49.416 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:49 smithi070 ceph-mon[32631]: pgmap v264: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 32 KiB/s wr, 13 op/s 2024-06-04T05:05:49.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:49 smithi112 ceph-mon[36453]: pgmap v264: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 32 KiB/s wr, 13 op/s 2024-06-04T05:05:49.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.479+0000 7f3bee5d0700 1 -- 172.21.15.70:0/507013668 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3be802e560 con 0x7f3be8102e20 2024-06-04T05:05:49.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.479+0000 7f3be57fa700 1 -- 172.21.15.70:0/507013668 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11929 (secure 0 0 0) 0x7f3bdc0aa340 con 0x7f3be8102e20 2024-06-04T05:05:49.482 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:05:49.482 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:01:52.807483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:05:49.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.481+0000 7f3bee5d0700 1 -- 172.21.15.70:0/507013668 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3bd00b29f0 msgr2=0x7f3bd00b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:49.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.481+0000 7f3bee5d0700 1 --2- 172.21.15.70:0/507013668 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3bd00b29f0 0x7f3bd00b4eb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800abd0 tx=0x7f3bd800a380 comp rx=0 tx=0).stop 2024-06-04T05:05:49.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.481+0000 7f3bee5d0700 1 -- 172.21.15.70:0/507013668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3be8102e20 msgr2=0x7f3be819fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:49.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.481+0000 7f3bee5d0700 1 --2- 172.21.15.70:0/507013668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3be8102e20 0x7f3be819fbf0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc00c9b0 tx=0x7f3bdc00ccc0 comp rx=0 tx=0).stop 2024-06-04T05:05:49.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.482+0000 7f3bee5d0700 1 -- 172.21.15.70:0/507013668 shutdown_connections 2024-06-04T05:05:49.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.482+0000 7f3bee5d0700 1 --2- 172.21.15.70:0/507013668 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3bd00b29f0 0x7f3bd00b4eb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:49.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.482+0000 7f3bee5d0700 1 --2- 172.21.15.70:0/507013668 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3be81084f0 0x7f3be8199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:49.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.482+0000 7f3bee5d0700 1 --2- 172.21.15.70:0/507013668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3be8102460 0x7f3be819f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:49.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.482+0000 7f3bee5d0700 1 --2- 172.21.15.70:0/507013668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3be8102e20 0x7f3be819fbf0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:49.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.482+0000 7f3bee5d0700 1 -- 172.21.15.70:0/507013668 >> 172.21.15.70:0/507013668 conn(0x7f3be80794e0 msgr2=0x7f3be806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:49.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.482+0000 7f3bee5d0700 1 -- 172.21.15.70:0/507013668 shutdown_connections 2024-06-04T05:05:49.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:49.482+0000 7f3bee5d0700 1 -- 172.21.15.70:0/507013668 wait complete. 2024-06-04T05:05:49.487 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 32 2024-06-04T05:05:49.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:49 smithi098 ceph-mon[35569]: pgmap v264: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 32 KiB/s wr, 13 op/s 2024-06-04T05:05:49.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 1.99 MB/sec execute 282 sec latency 19828.341 ms 2024-06-04T05:05:50.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:50 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/507013668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:05:50.494 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:50 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/507013668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:05:50.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:50 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/507013668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:05:50.523 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id 18106a7e --release --await' 2024-06-04T05:05:50.523 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id 18106a7e --release --await 2024-06-04T05:05:50.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 1.98 MB/sec execute 283 sec latency 20828.378 ms 2024-06-04T05:05:51.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.016+0000 7f1b05be4700 1 -- 172.21.15.70:0/3358230114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b00107a00 msgr2=0x7f1b0010f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:51.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.016+0000 7f1b05be4700 1 --2- 172.21.15.70:0/3358230114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b00107a00 0x7f1b0010f310 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f1af800b0a0 tx=0x7f1af800b3b0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.016+0000 7f1b05be4700 1 -- 172.21.15.70:0/3358230114 shutdown_connections 2024-06-04T05:05:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.016+0000 7f1b05be4700 1 --2- 172.21.15.70:0/3358230114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b00107a00 0x7f1b0010f310 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.016+0000 7f1b05be4700 1 --2- 172.21.15.70:0/3358230114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b001029e0 0x7f1b00102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.016+0000 7f1b05be4700 1 --2- 172.21.15.70:0/3358230114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b00107060 0x7f1b001074c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.016+0000 7f1b05be4700 1 -- 172.21.15.70:0/3358230114 >> 172.21.15.70:0/3358230114 conn(0x7f1b00079330 msgr2=0x7f1b00079740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.016+0000 7f1b05be4700 1 -- 172.21.15.70:0/3358230114 shutdown_connections 2024-06-04T05:05:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.016+0000 7f1b05be4700 1 -- 172.21.15.70:0/3358230114 wait complete. 2024-06-04T05:05:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.017+0000 7f1b05be4700 1 Processor -- start 2024-06-04T05:05:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.017+0000 7f1b05be4700 1 -- start start 2024-06-04T05:05:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.017+0000 7f1b05be4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b001029e0 0x7f1b0019a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.017+0000 7f1b05be4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b00107060 0x7f1b0019add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.017+0000 7f1b05be4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b00107a00 0x7f1b001a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.017+0000 7f1b05be4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1b00112510 con 0x7f1b001029e0 2024-06-04T05:05:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.017+0000 7f1b05be4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b00112390 con 0x7f1b00107060 2024-06-04T05:05:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.017+0000 7f1b05be4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1b00112690 con 0x7f1b00107a00 2024-06-04T05:05:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.017+0000 7f1aff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b001029e0 0x7f1b0019a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.017+0000 7f1aff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b001029e0 0x7f1b0019a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45600/0 (socket says 172.21.15.70:45600) 2024-06-04T05:05:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.017+0000 7f1aff7fe700 1 -- 172.21.15.70:0/941941192 learned_addr learned my addr 172.21.15.70:0/941941192 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.017+0000 7f1af7fff700 1 --2- 172.21.15.70:0/941941192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b00107060 0x7f1b0019add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.017+0000 7f1aff7fe700 1 -- 172.21.15.70:0/941941192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b00107a00 msgr2=0x7f1b001a5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.017+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/941941192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b00107a00 0x7f1b001a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.017+0000 7f1aff7fe700 1 -- 172.21.15.70:0/941941192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b00107060 msgr2=0x7f1b0019add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.017+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/941941192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b00107060 0x7f1b0019add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.017+0000 7f1aff7fe700 1 -- 172.21.15.70:0/941941192 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b001a5e50 con 0x7f1b001029e0 2024-06-04T05:05:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.018+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/941941192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b001029e0 0x7f1b0019a890 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f1af000b530 tx=0x7f1af000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.018+0000 7f1afd7fa700 1 -- 172.21.15.70:0/941941192 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0015020 con 0x7f1b001029e0 2024-06-04T05:05:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.018+0000 7f1afd7fa700 1 -- 172.21.15.70:0/941941192 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1af0011cc0 con 0x7f1b001029e0 2024-06-04T05:05:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.018+0000 7f1b05be4700 1 -- 172.21.15.70:0/941941192 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b001a6080 con 0x7f1b001029e0 2024-06-04T05:05:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.018+0000 7f1afd7fa700 1 -- 172.21.15.70:0/941941192 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0010d30 con 0x7f1b001029e0 2024-06-04T05:05:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.018+0000 7f1b05be4700 1 -- 172.21.15.70:0/941941192 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b001a6540 con 0x7f1b001029e0 2024-06-04T05:05:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.021+0000 7f1afd7fa700 1 -- 172.21.15.70:0/941941192 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f1af0018020 con 0x7f1b001029e0 2024-06-04T05:05:51.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.022+0000 7f1afd7fa700 1 --2- 172.21.15.70:0/941941192 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1aec0b29f0 0x7f1aec0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.022+0000 7f1af7fff700 1 --2- 172.21.15.70:0/941941192 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1aec0b29f0 0x7f1aec0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:51.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.022+0000 7f1afd7fa700 1 -- 172.21.15.70:0/941941192 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f1af00e1e00 con 0x7f1b001029e0 2024-06-04T05:05:51.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.022+0000 7f1b10479b80 1 Processor -- start 2024-06-04T05:05:51.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.022+0000 7f1b10479b80 1 -- start start 2024-06-04T05:05:51.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.022+0000 7f1af7fff700 1 --2- 172.21.15.70:0/941941192 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1aec0b29f0 0x7f1aec0b4eb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8005ea0 tx=0x7f1ae8005e30 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:51.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.022+0000 7f1b10479b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f54320 0x55c3b9f54700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.022+0000 7f1b10479b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4e8a0 0x55c3b9f4ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.022+0000 7f1b10479b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f4f260 0x55c3b9f5b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.022+0000 7f1b10479b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c3b9eacd70 con 0x55c3b9f54320 2024-06-04T05:05:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.022+0000 7f1b10479b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c3b9e9c910 con 0x55c3b9f4e8a0 2024-06-04T05:05:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.022+0000 7f1b10479b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c3b9ea0950 con 0x55c3b9f4f260 2024-06-04T05:05:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.022+0000 7f1aff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f54320 0x55c3b9f54700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.022+0000 7f1aff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f54320 0x55c3b9f54700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45606/0 (socket says 172.21.15.70:45606) 2024-06-04T05:05:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.022+0000 7f1aff7fe700 1 -- 172.21.15.70:0/1779495115 learned_addr learned my addr 172.21.15.70:0/1779495115 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.022+0000 7f1af7fff700 1 --2- 172.21.15.70:0/1779495115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4e8a0 0x55c3b9f4ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.023+0000 7f1afffff700 1 --2- 172.21.15.70:0/1779495115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f4f260 0x55c3b9f5b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.023+0000 7f1aff7fe700 1 -- 172.21.15.70:0/1779495115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f4f260 msgr2=0x55c3b9f5b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.023+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/1779495115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f4f260 0x55c3b9f5b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.023+0000 7f1aff7fe700 1 -- 172.21.15.70:0/1779495115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4e8a0 msgr2=0x55c3b9f4ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.023+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/1779495115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4e8a0 0x55c3b9f4ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.023+0000 7f1aff7fe700 1 -- 172.21.15.70:0/1779495115 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3b9e45fb0 con 0x55c3b9f54320 2024-06-04T05:05:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.023+0000 7f1af7fff700 1 --2- 172.21.15.70:0/1779495115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4e8a0 0x55c3b9f4ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:05:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.023+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/1779495115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f54320 0x55c3b9f54700 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f1af00a5240 tx=0x7f1af00a8a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e62557a7a2e4c7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.023+0000 7f1af5ffb700 1 -- 172.21.15.70:0/1779495115 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af00a91c0 con 0x55c3b9f54320 2024-06-04T05:05:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.023+0000 7f1af5ffb700 1 -- 172.21.15.70:0/1779495115 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1af00ad570 con 0x55c3b9f54320 2024-06-04T05:05:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.023+0000 7f1af5ffb700 1 -- 172.21.15.70:0/1779495115 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af00a9660 con 0x55c3b9f54320 2024-06-04T05:05:51.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.024+0000 7f1b10479b80 1 -- 172.21.15.70:0/1779495115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f54320 msgr2=0x55c3b9f54700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:51.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.024+0000 7f1b10479b80 1 --2- 172.21.15.70:0/1779495115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f54320 0x55c3b9f54700 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f1af00a5240 tx=0x7f1af00a8a30 comp rx=0 tx=0).stop 2024-06-04T05:05:51.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.024+0000 7f1b10479b80 1 -- 172.21.15.70:0/1779495115 shutdown_connections 2024-06-04T05:05:51.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.024+0000 7f1b10479b80 1 --2- 172.21.15.70:0/1779495115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f4f260 0x55c3b9f5b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.024+0000 7f1b10479b80 1 --2- 172.21.15.70:0/1779495115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f54320 0x55c3b9f54700 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.024+0000 7f1b10479b80 1 --2- 172.21.15.70:0/1779495115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4e8a0 0x55c3b9f4ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.024+0000 7f1b10479b80 1 -- 172.21.15.70:0/1779495115 >> 172.21.15.70:0/1779495115 conn(0x55c3b9e7a240 msgr2=0x55c3b9f493e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:51.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.024+0000 7f1b10479b80 1 -- 172.21.15.70:0/1779495115 shutdown_connections 2024-06-04T05:05:51.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.024+0000 7f1b10479b80 1 -- 172.21.15.70:0/1779495115 wait complete. 2024-06-04T05:05:51.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.024+0000 7f1b10479b80 1 -- start start 2024-06-04T05:05:51.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.024+0000 7f1b10479b80 1 Processor -- start 2024-06-04T05:05:51.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1b10479b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f4e8a0 0x55c3b9f5ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1b10479b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f4f260 0x55c3b9f58040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1b10479b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f54320 0x55c3b9f58580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1b10479b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c3b9da2fa0 con 0x55c3b9f4f260 2024-06-04T05:05:51.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1b10479b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c3b9e04da0 con 0x55c3b9f54320 2024-06-04T05:05:51.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1b10479b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c3b9f5b6e0 con 0x55c3b9f4e8a0 2024-06-04T05:05:51.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1b10479b80 10 client.0 _ll_register_callbacks cb 0x55c3b9e37950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:51.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1af7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f4f260 0x55c3b9f58040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:51.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1af7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f4f260 0x55c3b9f58040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45610/0 (socket says 172.21.15.70:45610) 2024-06-04T05:05:51.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1af7fff700 1 -- 172.21.15.70:0/387472142 learned_addr learned my addr 172.21.15.70:0/387472142 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:51.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/387472142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f4e8a0 0x55c3b9f5ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:51.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1afffff700 1 --2- 172.21.15.70:0/387472142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f54320 0x55c3b9f58580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:51.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1af7fff700 1 -- 172.21.15.70:0/387472142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f4e8a0 msgr2=0x55c3b9f5ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:51.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1af7fff700 1 --2- 172.21.15.70:0/387472142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f4e8a0 0x55c3b9f5ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1af7fff700 1 -- 172.21.15.70:0/387472142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f54320 msgr2=0x55c3b9f58580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:51.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1af7fff700 1 --2- 172.21.15.70:0/387472142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f54320 0x55c3b9f58580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1af7fff700 1 -- 172.21.15.70:0/387472142 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3b9dec8d0 con 0x55c3b9f4f260 2024-06-04T05:05:51.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1afffff700 1 --2- 172.21.15.70:0/387472142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f54320 0x55c3b9f58580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:05:51.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1af7fff700 1 --2- 172.21.15.70:0/387472142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f4f260 0x55c3b9f58040 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8010800 tx=0x7f1ae80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63cf0d1ecd9214f0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:51.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1adeffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:05:51.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1adeffd700 1 -- 172.21.15.70:0/387472142 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae800f850 con 0x55c3b9f4f260 2024-06-04T05:05:51.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.025+0000 7f1adeffd700 1 -- 172.21.15.70:0/387472142 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f1ae800f9f0 con 0x55c3b9f4f260 2024-06-04T05:05:51.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.026+0000 7f1b10479b80 1 -- 172.21.15.70:0/387472142 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1af00ad570 con 0x55c3b9f4f260 2024-06-04T05:05:51.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.026+0000 7f1adeffd700 1 -- 172.21.15.70:0/387472142 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1ae800fbf0 con 0x55c3b9f4f260 2024-06-04T05:05:51.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.026+0000 7f1b10479b80 10 client.17766 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:51.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.026+0000 7f1b10479b80 1 -- 172.21.15.70:0/387472142 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c3b9e56760 con 0x55c3b9f4f260 2024-06-04T05:05:51.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.026+0000 7f1adeffd700 1 -- 172.21.15.70:0/387472142 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1af00ad570 con 0x55c3b9f4f260 2024-06-04T05:05:51.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.026+0000 7f1adeffd700 1 -- 172.21.15.70:0/387472142 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f1ae8026400 con 0x55c3b9f4f260 2024-06-04T05:05:51.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.026+0000 7f1b10479b80 10 client.17766 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.026+0000 7f1b10479b80 10 client.17766 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:05:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.026+0000 7f1b10479b80 20 client.17766 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.026+0000 7f1b10479b80 1 --2- 172.21.15.70:0/387472142 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c3b9f682f0 0x55c3b9f6a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.026+0000 7f1b10479b80 4 client.17766 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:05:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.026+0000 7f1b10479b80 1 -- 172.21.15.70:0/387472142 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3b9eaecc0 con 0x55c3b9f682f0 2024-06-04T05:05:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.026+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/387472142 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c3b9f682f0 0x55c3b9f6a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.027+0000 7f1adeffd700 10 client.17766 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.027+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/387472142 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c3b9f682f0 0x55c3b9f6a6e0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f35d5c901f5e3d64 server_cookie=479c5b1835d0ecfd in_seq=0 out_seq=0 2024-06-04T05:05:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.027+0000 7f1adeffd700 1 -- 172.21.15.70:0/387472142 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c3b9eaecc0 con 0x55c3b9f682f0 2024-06-04T05:05:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.027+0000 7f1adeffd700 10 client.17766 handle_command_reply: tid=0 2024-06-04T05:05:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.027+0000 7f1b10479b80 1 client.17766 shutdown 2024-06-04T05:05:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.028+0000 7f1b10479b80 1 -- 172.21.15.70:0/387472142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f4f260 msgr2=0x55c3b9f58040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.028+0000 7f1b10479b80 1 --2- 172.21.15.70:0/387472142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f4f260 0x55c3b9f58040 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8010800 tx=0x7f1ae80047c0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.028+0000 7f1b10479b80 1 -- 172.21.15.70:0/387472142 shutdown_connections 2024-06-04T05:05:51.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.028+0000 7f1b10479b80 1 --2- 172.21.15.70:0/387472142 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c3b9f682f0 0x55c3b9f6a6e0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.028+0000 7f1b10479b80 1 --2- 172.21.15.70:0/387472142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f54320 0x55c3b9f58580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.028+0000 7f1b10479b80 1 --2- 172.21.15.70:0/387472142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f4e8a0 0x55c3b9f5ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.028+0000 7f1b10479b80 1 --2- 172.21.15.70:0/387472142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f4f260 0x55c3b9f58040 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.028+0000 7f1b10479b80 1 -- 172.21.15.70:0/387472142 >> 172.21.15.70:0/387472142 conn(0x55c3b9e7a240 msgr2=0x55c3b9f48af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:51.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.028+0000 7f1adeffd700 0 client.17766 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:51.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.028+0000 7f1b10479b80 1 -- 172.21.15.70:0/387472142 shutdown_connections 2024-06-04T05:05:51.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.028+0000 7f1b10479b80 1 -- 172.21.15.70:0/387472142 wait complete. 2024-06-04T05:05:51.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.028+0000 7f1b10479b80 20 client.17766 trim_cache size 0 max 16384 2024-06-04T05:05:51.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.043+0000 7f1b10479b80 1 Processor -- start 2024-06-04T05:05:51.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.043+0000 7f1b10479b80 1 -- start start 2024-06-04T05:05:51.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.043+0000 7f1b10479b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4e8a0 0x55c3b9f64740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.043+0000 7f1b10479b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f4f260 0x55c3b9f538a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.043+0000 7f1b10479b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f54320 0x55c3b9f49270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.043+0000 7f1b10479b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001ade80 con 0x55c3b9f4f260 2024-06-04T05:05:51.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.043+0000 7f1b10479b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001af9c0 con 0x55c3b9f4e8a0 2024-06-04T05:05:51.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.043+0000 7f1b10479b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001ae550 con 0x55c3b9f54320 2024-06-04T05:05:51.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.043+0000 7f1af7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f4f260 0x55c3b9f538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:51.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.043+0000 7f1af7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f4f260 0x55c3b9f538a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45618/0 (socket says 172.21.15.70:45618) 2024-06-04T05:05:51.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.043+0000 7f1af7fff700 1 -- 172.21.15.70:0/2451259055 learned_addr learned my addr 172.21.15.70:0/2451259055 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:51.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.043+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/2451259055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4e8a0 0x55c3b9f64740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:51.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.043+0000 7f1af7fff700 1 -- 172.21.15.70:0/2451259055 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f54320 msgr2=0x55c3b9f49270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:51.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.043+0000 7f1afffff700 1 --2- 172.21.15.70:0/2451259055 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f54320 0x55c3b9f49270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:51.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.044+0000 7f1af7fff700 1 --2- 172.21.15.70:0/2451259055 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f54320 0x55c3b9f49270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.044+0000 7f1af7fff700 1 -- 172.21.15.70:0/2451259055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4e8a0 msgr2=0x55c3b9f64740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:51.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.044+0000 7f1af7fff700 1 --2- 172.21.15.70:0/2451259055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4e8a0 0x55c3b9f64740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.044+0000 7f1af7fff700 1 -- 172.21.15.70:0/2451259055 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3b9f53f40 con 0x55c3b9f4f260 2024-06-04T05:05:51.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.044+0000 7f1afffff700 1 --2- 172.21.15.70:0/2451259055 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f54320 0x55c3b9f49270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:05:51.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.044+0000 7f1af7fff700 1 --2- 172.21.15.70:0/2451259055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f4f260 0x55c3b9f538a0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8005cb0 tx=0x7f1ae800a910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72be4c3c8c2e3ad server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:51.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.044+0000 7f1af57fa700 1 -- 172.21.15.70:0/2451259055 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8032040 con 0x55c3b9f4f260 2024-06-04T05:05:51.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.044+0000 7f1af57fa700 1 -- 172.21.15.70:0/2451259055 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1ae8026a60 con 0x55c3b9f4f260 2024-06-04T05:05:51.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.044+0000 7f1af57fa700 1 -- 172.21.15.70:0/2451259055 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8036970 con 0x55c3b9f4f260 2024-06-04T05:05:51.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.045+0000 7f1b10479b80 1 -- 172.21.15.70:0/2451259055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f4f260 msgr2=0x55c3b9f538a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:51.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.045+0000 7f1b10479b80 1 --2- 172.21.15.70:0/2451259055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f4f260 0x55c3b9f538a0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8005cb0 tx=0x7f1ae800a910 comp rx=0 tx=0).stop 2024-06-04T05:05:51.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.045+0000 7f1b10479b80 1 -- 172.21.15.70:0/2451259055 shutdown_connections 2024-06-04T05:05:51.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.045+0000 7f1b10479b80 1 --2- 172.21.15.70:0/2451259055 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f54320 0x55c3b9f49270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.045+0000 7f1b10479b80 1 --2- 172.21.15.70:0/2451259055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4e8a0 0x55c3b9f64740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.045+0000 7f1b10479b80 1 --2- 172.21.15.70:0/2451259055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f4f260 0x55c3b9f538a0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.045+0000 7f1b10479b80 1 -- 172.21.15.70:0/2451259055 >> 172.21.15.70:0/2451259055 conn(0x55c3b9e7a240 msgr2=0x55c3b9f53190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:51.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.045+0000 7f1b10479b80 1 -- 172.21.15.70:0/2451259055 shutdown_connections 2024-06-04T05:05:51.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.045+0000 7f1b10479b80 1 -- 172.21.15.70:0/2451259055 wait complete. 2024-06-04T05:05:51.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.045+0000 7f1b10479b80 1 -- start start 2024-06-04T05:05:51.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.045+0000 7f1b10479b80 1 Processor -- start 2024-06-04T05:05:51.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1b10479b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f4e8a0 0x55c3b9f60b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1b10479b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4f260 0x55c3b9f61050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1b10479b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f54320 0x55c3b9f6f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1b10479b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001af2f0 con 0x55c3b9f54320 2024-06-04T05:05:51.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1b10479b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001aec20 con 0x55c3b9f4f260 2024-06-04T05:05:51.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1b10479b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001abe40 con 0x55c3b9f4e8a0 2024-06-04T05:05:51.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1b10479b80 10 client.0 _ll_register_callbacks cb 0x55c3b9e37950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:51.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1afffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f54320 0x55c3b9f6f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:51.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1af7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4f260 0x55c3b9f61050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:51.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1af7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4f260 0x55c3b9f61050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:38580/0 (socket says 172.21.15.70:38580) 2024-06-04T05:05:51.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1afffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f54320 0x55c3b9f6f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45628/0 (socket says 172.21.15.70:45628) 2024-06-04T05:05:51.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1af7fff700 1 -- 172.21.15.70:0/1605309360 learned_addr learned my addr 172.21.15.70:0/1605309360 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:51.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/1605309360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f4e8a0 0x55c3b9f60b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:51.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1af7fff700 1 -- 172.21.15.70:0/1605309360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f4e8a0 msgr2=0x55c3b9f60b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:51.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1af7fff700 1 --2- 172.21.15.70:0/1605309360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f4e8a0 0x55c3b9f60b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1af7fff700 1 -- 172.21.15.70:0/1605309360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f54320 msgr2=0x55c3b9f6f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:51.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1af7fff700 1 --2- 172.21.15.70:0/1605309360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f54320 0x55c3b9f6f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1af7fff700 1 -- 172.21.15.70:0/1605309360 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3b9f6fbc0 con 0x55c3b9f4f260 2024-06-04T05:05:51.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1afffff700 1 --2- 172.21.15.70:0/1605309360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f54320 0x55c3b9f6f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:51.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1af7fff700 1 --2- 172.21.15.70:0/1605309360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4f260 0x55c3b9f61050 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8026a30 tx=0x7f1ae8026ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e642cadebd451e6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:51.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.046+0000 7f1adeffd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:05:51.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.047+0000 7f1adeffd700 1 -- 172.21.15.70:0/1605309360 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8032040 con 0x55c3b9f4f260 2024-06-04T05:05:51.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.047+0000 7f1b10479b80 1 -- 172.21.15.70:0/1605309360 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ae8026a60 con 0x55c3b9f4f260 2024-06-04T05:05:51.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.047+0000 7f1adeffd700 1 -- 172.21.15.70:0/1605309360 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f1ae8005040 con 0x55c3b9f4f260 2024-06-04T05:05:51.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.047+0000 7f1adeffd700 1 -- 172.21.15.70:0/1605309360 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1ae8005240 con 0x55c3b9f4f260 2024-06-04T05:05:51.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.047+0000 7f1b10479b80 10 client.26942 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:51.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.047+0000 7f1b10479b80 1 -- 172.21.15.70:0/1605309360 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c3b9fd2910 con 0x55c3b9f4f260 2024-06-04T05:05:51.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.047+0000 7f1adeffd700 1 -- 172.21.15.70:0/1605309360 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1ae8026a60 con 0x55c3b9f4f260 2024-06-04T05:05:51.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.047+0000 7f1adeffd700 1 -- 172.21.15.70:0/1605309360 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f1ae8035070 con 0x55c3b9f4f260 2024-06-04T05:05:51.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.047+0000 7f1b10479b80 10 client.26942 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:51.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.047+0000 7f1b10479b80 10 client.26942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:05:51.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.047+0000 7f1b10479b80 20 client.26942 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:51.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.047+0000 7f1b10479b80 1 --2- 172.21.15.70:0/1605309360 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c3b9fd2f20 0x55c3b9fd5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.047+0000 7f1b10479b80 4 client.26942 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:05:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.047+0000 7f1b10479b80 1 -- 172.21.15.70:0/1605309360 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c3b9ea84a0 con 0x55c3b9fd2f20 2024-06-04T05:05:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.048+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/1605309360 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c3b9fd2f20 0x55c3b9fd5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.048+0000 7f1adeffd700 10 client.26942 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.048+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/1605309360 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c3b9fd2f20 0x55c3b9fd5340 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c92186404b697dd server_cookie=d670b794aa89bc86 in_seq=0 out_seq=0 2024-06-04T05:05:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.048+0000 7f1adeffd700 1 -- 172.21.15.70:0/1605309360 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c3b9ea84a0 con 0x55c3b9fd2f20 2024-06-04T05:05:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.048+0000 7f1adeffd700 10 client.26942 handle_command_reply: tid=0 2024-06-04T05:05:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.048+0000 7f1b10479b80 1 client.26942 shutdown 2024-06-04T05:05:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.048+0000 7f1b10479b80 1 -- 172.21.15.70:0/1605309360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4f260 msgr2=0x55c3b9f61050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.048+0000 7f1b10479b80 1 --2- 172.21.15.70:0/1605309360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4f260 0x55c3b9f61050 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8026a30 tx=0x7f1ae8026ca0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b10479b80 1 -- 172.21.15.70:0/1605309360 shutdown_connections 2024-06-04T05:05:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b10479b80 1 --2- 172.21.15.70:0/1605309360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3b9f54320 0x55c3b9f6f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b10479b80 1 --2- 172.21.15.70:0/1605309360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3b9f4e8a0 0x55c3b9f60b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b10479b80 1 --2- 172.21.15.70:0/1605309360 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c3b9fd2f20 0x55c3b9fd5340 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b10479b80 1 --2- 172.21.15.70:0/1605309360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3b9f4f260 0x55c3b9f61050 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1adeffd700 0 client.26942 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:51.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b10479b80 1 -- 172.21.15.70:0/1605309360 >> 172.21.15.70:0/1605309360 conn(0x55c3b9e7a240 msgr2=0x55c3b9f51d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:51.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b10479b80 1 -- 172.21.15.70:0/1605309360 shutdown_connections 2024-06-04T05:05:51.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b10479b80 1 -- 172.21.15.70:0/1605309360 wait complete. 2024-06-04T05:05:51.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b10479b80 20 client.26942 trim_cache size 0 max 16384 2024-06-04T05:05:51.056 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:05:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b05be4700 1 -- 172.21.15.70:0/941941192 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1aec0b29f0 msgr2=0x7f1aec0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b05be4700 1 --2- 172.21.15.70:0/941941192 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1aec0b29f0 0x7f1aec0b4eb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8005ea0 tx=0x7f1ae8005e30 comp rx=0 tx=0).stop 2024-06-04T05:05:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b05be4700 1 -- 172.21.15.70:0/941941192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b001029e0 msgr2=0x7f1b0019a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b05be4700 1 --2- 172.21.15.70:0/941941192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b001029e0 0x7f1b0019a890 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f1af000b530 tx=0x7f1af000b8f0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b05be4700 1 -- 172.21.15.70:0/941941192 shutdown_connections 2024-06-04T05:05:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b05be4700 1 --2- 172.21.15.70:0/941941192 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1aec0b29f0 0x7f1aec0b4eb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b05be4700 1 --2- 172.21.15.70:0/941941192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b00107a00 0x7f1b001a5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b05be4700 1 --2- 172.21.15.70:0/941941192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b001029e0 0x7f1b0019a890 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b05be4700 1 --2- 172.21.15.70:0/941941192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b00107060 0x7f1b0019add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:51.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.049+0000 7f1b05be4700 1 -- 172.21.15.70:0/941941192 >> 172.21.15.70:0/941941192 conn(0x7f1b00079330 msgr2=0x7f1b00100730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:51.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.050+0000 7f1b05be4700 1 -- 172.21.15.70:0/941941192 shutdown_connections 2024-06-04T05:05:51.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:51.050+0000 7f1b05be4700 1 -- 172.21.15.70:0/941941192 wait complete. 2024-06-04T05:05:51.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:51 smithi070 ceph-mon[32631]: pgmap v265: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 32 KiB/s wr, 13 op/s 2024-06-04T05:05:51.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:51 smithi112 ceph-mon[36453]: pgmap v265: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 32 KiB/s wr, 13 op/s 2024-06-04T05:05:51.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:51 smithi098 ceph-mon[35569]: pgmap v265: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 32 KiB/s wr, 13 op/s 2024-06-04T05:05:51.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 1.98 MB/sec execute 284 sec latency 21828.425 ms 2024-06-04T05:05:51.914 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:05:52.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149285 1.97 MB/sec execute 285 sec latency 22828.473 ms 2024-06-04T05:05:52.842 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:05:52.842 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:05:53.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:53 smithi070 ceph-mon[32631]: pgmap v266: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 32 KiB/s wr, 17 op/s 2024-06-04T05:05:53.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:53 smithi112 ceph-mon[36453]: pgmap v266: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 32 KiB/s wr, 17 op/s 2024-06-04T05:05:53.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:53 smithi098 ceph-mon[35569]: pgmap v266: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 32 KiB/s wr, 17 op/s 2024-06-04T05:05:53.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.666+0000 7fbc9733e700 1 -- 172.21.15.70:0/587939227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc90102430 msgr2=0x7fbc901028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.666+0000 7fbc9733e700 1 --2- 172.21.15.70:0/587939227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc90102430 0x7fbc901028b0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fbc80009990 tx=0x7fbc80009ca0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.666+0000 7fbc9733e700 1 -- 172.21.15.70:0/587939227 shutdown_connections 2024-06-04T05:05:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.666+0000 7fbc9733e700 1 --2- 172.21.15.70:0/587939227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc90102df0 0x7fbc9010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.666+0000 7fbc9733e700 1 --2- 172.21.15.70:0/587939227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc90108510 0x7fbc901088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.666+0000 7fbc9733e700 1 --2- 172.21.15.70:0/587939227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc90102430 0x7fbc901028b0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.666+0000 7fbc9733e700 1 -- 172.21.15.70:0/587939227 >> 172.21.15.70:0/587939227 conn(0x7fbc900793b0 msgr2=0x7fbc900797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.666+0000 7fbc9733e700 1 -- 172.21.15.70:0/587939227 shutdown_connections 2024-06-04T05:05:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.666+0000 7fbc9733e700 1 -- 172.21.15.70:0/587939227 wait complete. 2024-06-04T05:05:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.667+0000 7fbc9733e700 1 Processor -- start 2024-06-04T05:05:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.667+0000 7fbc9733e700 1 -- start start 2024-06-04T05:05:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.667+0000 7fbc9733e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc90102430 0x7fbc9019a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.667+0000 7fbc9733e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc90102df0 0x7fbc9019aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.667+0000 7fbc9733e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc90108510 0x7fbc901a5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.667+0000 7fbc9733e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbc901124f0 con 0x7fbc90102430 2024-06-04T05:05:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.667+0000 7fbc9733e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc90112370 con 0x7fbc90108510 2024-06-04T05:05:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.667+0000 7fbc9733e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbc90112670 con 0x7fbc90102df0 2024-06-04T05:05:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.667+0000 7fbc950da700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc90102430 0x7fbc9019a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.667+0000 7fbc950da700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc90102430 0x7fbc9019a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45652/0 (socket says 172.21.15.70:45652) 2024-06-04T05:05:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.667+0000 7fbc950da700 1 -- 172.21.15.70:0/580694972 learned_addr learned my addr 172.21.15.70:0/580694972 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.667+0000 7fbc958db700 1 --2- 172.21.15.70:0/580694972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc90108510 0x7fbc901a5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.667+0000 7fbc950da700 1 -- 172.21.15.70:0/580694972 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc90102df0 msgr2=0x7fbc9019aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.667+0000 7fbc948d9700 1 --2- 172.21.15.70:0/580694972 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc90102df0 0x7fbc9019aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.667+0000 7fbc950da700 1 --2- 172.21.15.70:0/580694972 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc90102df0 0x7fbc9019aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.668+0000 7fbc950da700 1 -- 172.21.15.70:0/580694972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc90108510 msgr2=0x7fbc901a5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.668+0000 7fbc950da700 1 --2- 172.21.15.70:0/580694972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc90108510 0x7fbc901a5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.668+0000 7fbc950da700 1 -- 172.21.15.70:0/580694972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc901a5f70 con 0x7fbc90102430 2024-06-04T05:05:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.668+0000 7fbc958db700 1 --2- 172.21.15.70:0/580694972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc90108510 0x7fbc901a5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:05:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.668+0000 7fbc950da700 1 --2- 172.21.15.70:0/580694972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc90102430 0x7fbc9019a9b0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c00b530 tx=0x7fbc8c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.668+0000 7fbc867fc700 1 -- 172.21.15.70:0/580694972 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c015020 con 0x7fbc90102430 2024-06-04T05:05:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.668+0000 7fbc867fc700 1 -- 172.21.15.70:0/580694972 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbc8c011cc0 con 0x7fbc90102430 2024-06-04T05:05:53.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.668+0000 7fbc867fc700 1 -- 172.21.15.70:0/580694972 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c010d30 con 0x7fbc90102430 2024-06-04T05:05:53.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.668+0000 7fbc9733e700 1 -- 172.21.15.70:0/580694972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc901a61a0 con 0x7fbc90102430 2024-06-04T05:05:53.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.668+0000 7fbc9733e700 1 -- 172.21.15.70:0/580694972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc901a6660 con 0x7fbc90102430 2024-06-04T05:05:53.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.671+0000 7fbc867fc700 1 -- 172.21.15.70:0/580694972 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fbc8c018020 con 0x7fbc90102430 2024-06-04T05:05:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.672+0000 7fbc867fc700 1 --2- 172.21.15.70:0/580694972 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbc7c0b29f0 0x7fbc7c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.672+0000 7fbc948d9700 1 --2- 172.21.15.70:0/580694972 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbc7c0b29f0 0x7fbc7c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.672+0000 7fbc867fc700 1 -- 172.21.15.70:0/580694972 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fbc8c0e1d90 con 0x7fbc90102430 2024-06-04T05:05:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.672+0000 7fbca1bd3b80 1 Processor -- start 2024-06-04T05:05:53.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.672+0000 7fbca1bd3b80 1 -- start start 2024-06-04T05:05:53.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.672+0000 7fbc948d9700 1 --2- 172.21.15.70:0/580694972 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbc7c0b29f0 0x7fbc7c0b4eb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbc80006010 tx=0x7fbc80005c40 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:53.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbca1bd3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5fa16c0 0x55a6e5fa1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbca1bd3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5f9c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbca1bd3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5fa84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbca1bd3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a6e5ef6d90 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbca1bd3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a6e5ee6930 con 0x55a6e5f9bcf0 2024-06-04T05:05:53.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbca1bd3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a6e5eea970 con 0x55a6e5fa16c0 2024-06-04T05:05:53.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbc958db700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5fa84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbc958db700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5fa84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45662/0 (socket says 172.21.15.70:45662) 2024-06-04T05:05:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbc958db700 1 -- 172.21.15.70:0/4080451629 learned_addr learned my addr 172.21.15.70:0/4080451629 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbc950da700 1 --2- 172.21.15.70:0/4080451629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5fa16c0 0x55a6e5fa1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbc948d9700 1 --2- 172.21.15.70:0/4080451629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5f9c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbc958db700 1 -- 172.21.15.70:0/4080451629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5fa16c0 msgr2=0x55a6e5fa1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbc958db700 1 --2- 172.21.15.70:0/4080451629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5fa16c0 0x55a6e5fa1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbc958db700 1 -- 172.21.15.70:0/4080451629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5f9bcf0 msgr2=0x55a6e5f9c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbc958db700 1 --2- 172.21.15.70:0/4080451629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5f9c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbc958db700 1 -- 172.21.15.70:0/4080451629 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6e5e369d0 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbc948d9700 1 --2- 172.21.15.70:0/4080451629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5f9c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:05:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbc958db700 1 --2- 172.21.15.70:0/4080451629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5fa84f0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fbc8800e260 tx=0x7fbc8800e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3fef8f7c11e49a2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.673+0000 7fbc7b7fe700 1 -- 172.21.15.70:0/4080451629 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8800eda0 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.674+0000 7fbc7b7fe700 1 -- 172.21.15.70:0/4080451629 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbc88004bb0 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.674+0000 7fbc7b7fe700 1 -- 172.21.15.70:0/4080451629 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc88014d90 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.674+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/4080451629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 msgr2=0x55a6e5fa84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.674+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/4080451629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5fa84f0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fbc8800e260 tx=0x7fbc8800e570 comp rx=0 tx=0).stop 2024-06-04T05:05:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.674+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/4080451629 shutdown_connections 2024-06-04T05:05:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.674+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/4080451629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5fa84f0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.674+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/4080451629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5fa16c0 0x55a6e5fa1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.674+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/4080451629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5f9c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.674+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/4080451629 >> 172.21.15.70:0/4080451629 conn(0x55a6e5ec5180 msgr2=0x55a6e5f921f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.674+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/4080451629 shutdown_connections 2024-06-04T05:05:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.674+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/4080451629 wait complete. 2024-06-04T05:05:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.675+0000 7fbca1bd3b80 1 -- start start 2024-06-04T05:05:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.675+0000 7fbca1bd3b80 1 Processor -- start 2024-06-04T05:05:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.675+0000 7fbca1bd3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5fa8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.675+0000 7fbca1bd3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5fa5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.675+0000 7fbca1bd3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5fa16c0 0x55a6e5fa5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.675+0000 7fbca1bd3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a6e5ded0a0 con 0x55a6e5f9bcf0 2024-06-04T05:05:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.675+0000 7fbca1bd3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a6e5e888c0 con 0x55a6e5fa16c0 2024-06-04T05:05:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.675+0000 7fbca1bd3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a6e5fa8af0 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.675+0000 7fbca1bd3b80 10 client.0 _ll_register_callbacks cb 0x55a6e5e81a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.675+0000 7fbc950da700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5fa8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.675+0000 7fbc950da700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5fa8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45672/0 (socket says 172.21.15.70:45672) 2024-06-04T05:05:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbc950da700 1 -- 172.21.15.70:0/1213279350 learned_addr learned my addr 172.21.15.70:0/1213279350 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbc948d9700 1 --2- 172.21.15.70:0/1213279350 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5fa5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbc958db700 1 --2- 172.21.15.70:0/1213279350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5fa16c0 0x55a6e5fa5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbc950da700 1 -- 172.21.15.70:0/1213279350 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9c6b0 msgr2=0x55a6e5fa5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbc950da700 1 --2- 172.21.15.70:0/1213279350 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5fa5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbc950da700 1 -- 172.21.15.70:0/1213279350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5fa16c0 msgr2=0x55a6e5fa5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbc950da700 1 --2- 172.21.15.70:0/1213279350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5fa16c0 0x55a6e5fa5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbc950da700 1 -- 172.21.15.70:0/1213279350 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6e5e85790 con 0x55a6e5f9bcf0 2024-06-04T05:05:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbc958db700 1 --2- 172.21.15.70:0/1213279350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5fa16c0 0x55a6e5fa5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:05:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbc948d9700 1 --2- 172.21.15.70:0/1213279350 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5fa5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:05:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbc950da700 1 --2- 172.21.15.70:0/1213279350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5fa8220 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c0a90e0 tx=0x7fbc8c0106d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b7de15e1cb74124 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbc78ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:05:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbc78ff9700 1 -- 172.21.15.70:0/1213279350 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c0c3020 con 0x55a6e5f9bcf0 2024-06-04T05:05:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbc78ff9700 1 -- 172.21.15.70:0/1213279350 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fbc8c0a53e0 con 0x55a6e5f9bcf0 2024-06-04T05:05:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/1213279350 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc88004bb0 con 0x55a6e5f9bcf0 2024-06-04T05:05:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbc78ff9700 1 -- 172.21.15.70:0/1213279350 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbc8c0ad500 con 0x55a6e5f9bcf0 2024-06-04T05:05:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbca1bd3b80 10 client.17784 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/1213279350 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a6e5fb3300 con 0x55a6e5f9bcf0 2024-06-04T05:05:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.676+0000 7fbc78ff9700 1 -- 172.21.15.70:0/1213279350 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbc88004bb0 con 0x55a6e5f9bcf0 2024-06-04T05:05:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.677+0000 7fbc78ff9700 1 -- 172.21.15.70:0/1213279350 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fbc8c0b1b20 con 0x55a6e5f9bcf0 2024-06-04T05:05:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.677+0000 7fbca1bd3b80 10 client.17784 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.677+0000 7fbca1bd3b80 10 client.17784 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:05:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.677+0000 7fbca1bd3b80 20 client.17784 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.677+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/1213279350 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a6e5fb5890 0x55a6e5fb7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.677+0000 7fbca1bd3b80 4 client.17784 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:05:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.677+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/1213279350 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6e5f63ae0 con 0x55a6e5fb5890 2024-06-04T05:05:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.677+0000 7fbc948d9700 1 --2- 172.21.15.70:0/1213279350 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a6e5fb5890 0x55a6e5fb7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.677+0000 7fbc948d9700 1 --2- 172.21.15.70:0/1213279350 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a6e5fb5890 0x55a6e5fb7ce0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=458f1b83d6738c75 server_cookie=a20559f312960ca0 in_seq=0 out_seq=0 2024-06-04T05:05:53.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.677+0000 7fbc78ff9700 10 client.17784 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:53.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.678+0000 7fbc78ff9700 1 -- 172.21.15.70:0/1213279350 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a6e5f63ae0 con 0x55a6e5fb5890 2024-06-04T05:05:53.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.678+0000 7fbc78ff9700 10 client.17784 handle_command_reply: tid=0 2024-06-04T05:05:53.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.678+0000 7fbca1bd3b80 1 client.17784 shutdown 2024-06-04T05:05:53.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.678+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/1213279350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9bcf0 msgr2=0x55a6e5fa8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:53.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.678+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/1213279350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5fa8220 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c0a90e0 tx=0x7fbc8c0106d0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.678+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/1213279350 shutdown_connections 2024-06-04T05:05:53.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.678+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/1213279350 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a6e5fb5890 0x55a6e5fb7ce0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.678+0000 7fbc78ff9700 0 client.17784 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:53.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.678+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/1213279350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5fa16c0 0x55a6e5fa5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.678+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/1213279350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5fa8220 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.678+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/1213279350 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5fa5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.678+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/1213279350 >> 172.21.15.70:0/1213279350 conn(0x55a6e5ec5180 msgr2=0x55a6e5f8cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:53.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.678+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/1213279350 shutdown_connections 2024-06-04T05:05:53.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.678+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/1213279350 wait complete. 2024-06-04T05:05:53.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.679+0000 7fbca1bd3b80 20 client.17784 trim_cache size 0 max 16384 2024-06-04T05:05:53.696 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:05:53.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.699+0000 7fbca1bd3b80 1 Processor -- start 2024-06-04T05:05:53.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.699+0000 7fbca1bd3b80 1 -- start start 2024-06-04T05:05:53.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.699+0000 7fbca1bd3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5fb1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.699+0000 7fbca1bd3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5fa0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.699+0000 7fbca1bd3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5fa16c0 0x55a6e5f8ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.699+0000 7fbca1bd3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbc901abc70 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.699+0000 7fbca1bd3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc901ad7b0 con 0x55a6e5fa16c0 2024-06-04T05:05:53.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.699+0000 7fbca1bd3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbc901ac340 con 0x55a6e5f9bcf0 2024-06-04T05:05:53.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.700+0000 7fbc948d9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5fa0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.700+0000 7fbc948d9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5fa0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45680/0 (socket says 172.21.15.70:45680) 2024-06-04T05:05:53.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.700+0000 7fbc948d9700 1 -- 172.21.15.70:0/4037469478 learned_addr learned my addr 172.21.15.70:0/4037469478 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:53.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.700+0000 7fbc958db700 1 --2- 172.21.15.70:0/4037469478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5fa16c0 0x55a6e5f8ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.700+0000 7fbc950da700 1 --2- 172.21.15.70:0/4037469478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5fb1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.700+0000 7fbc948d9700 1 -- 172.21.15.70:0/4037469478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9bcf0 msgr2=0x55a6e5fb1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:53.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.700+0000 7fbc948d9700 1 --2- 172.21.15.70:0/4037469478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5fb1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.700+0000 7fbc948d9700 1 -- 172.21.15.70:0/4037469478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5fa16c0 msgr2=0x55a6e5f8ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:53.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.700+0000 7fbc948d9700 1 --2- 172.21.15.70:0/4037469478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5fa16c0 0x55a6e5f8ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.700+0000 7fbc948d9700 1 -- 172.21.15.70:0/4037469478 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6e5fa6460 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.700+0000 7fbc950da700 1 --2- 172.21.15.70:0/4037469478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5fb1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:05:53.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.700+0000 7fbc948d9700 1 --2- 172.21.15.70:0/4037469478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5fa0dc0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fbc80005170 tx=0x7fbc8001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99838a6aafdd2cf1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:53.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.700+0000 7fbc7affd700 1 -- 172.21.15.70:0/4037469478 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc80036070 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.700+0000 7fbc7affd700 1 -- 172.21.15.70:0/4037469478 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbc8000bc60 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.700+0000 7fbc7affd700 1 -- 172.21.15.70:0/4037469478 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc800319b0 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.701+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/4037469478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 msgr2=0x55a6e5fa0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.701+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/4037469478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5fa0dc0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fbc80005170 tx=0x7fbc8001d000 comp rx=0 tx=0).stop 2024-06-04T05:05:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.701+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/4037469478 shutdown_connections 2024-06-04T05:05:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.701+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/4037469478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5fa16c0 0x55a6e5f8ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.701+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/4037469478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5fb1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.701+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/4037469478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5fa0dc0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.701+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/4037469478 >> 172.21.15.70:0/4037469478 conn(0x55a6e5ec5180 msgr2=0x55a6e5fa75e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.701+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/4037469478 shutdown_connections 2024-06-04T05:05:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.701+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/4037469478 wait complete. 2024-06-04T05:05:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.702+0000 7fbca1bd3b80 1 -- start start 2024-06-04T05:05:53.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.702+0000 7fbca1bd3b80 1 Processor -- start 2024-06-04T05:05:53.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.702+0000 7fbca1bd3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5fae410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.702+0000 7fbca1bd3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5faa7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.702+0000 7fbca1bd3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5fa16c0 0x55a6e5faad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.702+0000 7fbca1bd3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbc901ad0e0 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.702+0000 7fbca1bd3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc901aca10 con 0x55a6e5fa16c0 2024-06-04T05:05:53.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.702+0000 7fbca1bd3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbc901a9c30 con 0x55a6e5f9bcf0 2024-06-04T05:05:53.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.702+0000 7fbca1bd3b80 10 client.0 _ll_register_callbacks cb 0x55a6e5e81a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:53.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.702+0000 7fbc948d9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5faa7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.702+0000 7fbc948d9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5faa7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45684/0 (socket says 172.21.15.70:45684) 2024-06-04T05:05:53.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.702+0000 7fbc948d9700 1 -- 172.21.15.70:0/1571872223 learned_addr learned my addr 172.21.15.70:0/1571872223 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:53.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.703+0000 7fbc950da700 1 --2- 172.21.15.70:0/1571872223 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5fae410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.703+0000 7fbc958db700 1 --2- 172.21.15.70:0/1571872223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5fa16c0 0x55a6e5faad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.703+0000 7fbc948d9700 1 -- 172.21.15.70:0/1571872223 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9bcf0 msgr2=0x55a6e5fae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:53.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.703+0000 7fbc948d9700 1 --2- 172.21.15.70:0/1571872223 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5fae410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.703+0000 7fbc948d9700 1 -- 172.21.15.70:0/1571872223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5fa16c0 msgr2=0x55a6e5faad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:53.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.703+0000 7fbc948d9700 1 --2- 172.21.15.70:0/1571872223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5fa16c0 0x55a6e5faad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.703+0000 7fbc948d9700 1 -- 172.21.15.70:0/1571872223 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6e5fab550 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.703+0000 7fbc950da700 1 --2- 172.21.15.70:0/1571872223 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5fae410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:53.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.703+0000 7fbc948d9700 1 --2- 172.21.15.70:0/1571872223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5faa7f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fbc80009ce0 tx=0x7fbc80004f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8da3e237ce5e1bf server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:53.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.703+0000 7fbc78ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:05:53.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.703+0000 7fbc78ff9700 1 -- 172.21.15.70:0/1571872223 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc80036070 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.703+0000 7fbc78ff9700 1 -- 172.21.15.70:0/1571872223 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fbc8000b770 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.703+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/1571872223 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc8000bc60 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.703+0000 7fbc78ff9700 1 -- 172.21.15.70:0/1571872223 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbc80031400 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.703+0000 7fbca1bd3b80 10 client.17790 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:53.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.703+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/1571872223 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a6e6021510 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.704+0000 7fbc78ff9700 1 -- 172.21.15.70:0/1571872223 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbc8000bc60 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.704+0000 7fbc78ff9700 1 -- 172.21.15.70:0/1571872223 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fbc8000f510 con 0x55a6e5f9c6b0 2024-06-04T05:05:53.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.704+0000 7fbca1bd3b80 10 client.17790 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:53.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.704+0000 7fbca1bd3b80 10 client.17790 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:05:53.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.704+0000 7fbca1bd3b80 20 client.17790 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:53.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.704+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/1571872223 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a6e6021b80 0x55a6e6023fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:53.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.704+0000 7fbca1bd3b80 4 client.17790 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "18106a7e", "release": true, "await": true}] 2024-06-04T05:05:53.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.704+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/1571872223 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "18106a7e", "release": true, "await": true}) -- 0x55a6e5ef8ce0 con 0x55a6e6021b80 2024-06-04T05:05:53.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.704+0000 7fbc950da700 1 --2- 172.21.15.70:0/1571872223 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a6e6021b80 0x55a6e6023fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:53.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.705+0000 7fbc950da700 1 --2- 172.21.15.70:0/1571872223 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a6e6021b80 0x55a6e6023fa0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73871e7b89a9c0f2 server_cookie=c4a00688399cd57d in_seq=0 out_seq=0 2024-06-04T05:05:53.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.705+0000 7fbc78ff9700 10 client.17790 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.710+0000 7fbc78ff9700 1 -- 172.21.15.70:0/1571872223 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55a6e5ef8ce0 con 0x55a6e6021b80 2024-06-04T05:05:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.710+0000 7fbc78ff9700 10 client.17790 handle_command_reply: tid=0 2024-06-04T05:05:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.710+0000 7fbca1bd3b80 1 client.17790 shutdown 2024-06-04T05:05:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.710+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/1571872223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 msgr2=0x55a6e5faa7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.710+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/1571872223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5faa7f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fbc80009ce0 tx=0x7fbc80004f10 comp rx=0 tx=0).stop 2024-06-04T05:05:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.710+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/1571872223 shutdown_connections 2024-06-04T05:05:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.710+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/1571872223 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a6e6021b80 0x55a6e6023fa0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.710+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/1571872223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6e5fa16c0 0x55a6e5faad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.710+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/1571872223 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6e5f9bcf0 0x55a6e5fae410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.710+0000 7fbca1bd3b80 1 --2- 172.21.15.70:0/1571872223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6e5f9c6b0 0x55a6e5faa7f0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.710+0000 7fbc78ff9700 0 client.17790 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.710+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/1571872223 >> 172.21.15.70:0/1571872223 conn(0x55a6e5ec5180 msgr2=0x55a6e5f9f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:53.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.711+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/1571872223 shutdown_connections 2024-06-04T05:05:53.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.711+0000 7fbca1bd3b80 1 -- 172.21.15.70:0/1571872223 wait complete. 2024-06-04T05:05:53.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.711+0000 7fbca1bd3b80 20 client.17790 trim_cache size 0 max 16384 2024-06-04T05:05:53.714 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:05:53.714 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 32, 2024-06-04T05:05:53.714 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:05:53.714 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 24, 2024-06-04T05:05:53.714 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:05:53.714 INFO:teuthology.orchestra.run.smithi070.stdout: "18106a7e": { 2024-06-04T05:05:53.715 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 24, 2024-06-04T05:05:53.715 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:05:53.715 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:05:53.715 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:05:53.715 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:05:53.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:05:53.715 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:05:53.715 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 143.6, 2024-06-04T05:05:53.715 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:05:53.715 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:05:53.715 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:05:53.715 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:05:53.715 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:05:53.715 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:05:53.716 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:05:53.716 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:05:53.716 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:05:53.716 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:05:53.716 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:05:53.716 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:05:53.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.711+0000 7fbc9733e700 1 -- 172.21.15.70:0/580694972 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbc7c0b29f0 msgr2=0x7fbc7c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:53.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.711+0000 7fbc9733e700 1 --2- 172.21.15.70:0/580694972 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbc7c0b29f0 0x7fbc7c0b4eb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbc80006010 tx=0x7fbc80005c40 comp rx=0 tx=0).stop 2024-06-04T05:05:53.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.711+0000 7fbc9733e700 1 -- 172.21.15.70:0/580694972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc90102430 msgr2=0x7fbc9019a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:53.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.711+0000 7fbc9733e700 1 --2- 172.21.15.70:0/580694972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc90102430 0x7fbc9019a9b0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c00b530 tx=0x7fbc8c00b8f0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.711+0000 7fbc9733e700 1 -- 172.21.15.70:0/580694972 shutdown_connections 2024-06-04T05:05:53.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.711+0000 7fbc9733e700 1 --2- 172.21.15.70:0/580694972 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbc7c0b29f0 0x7fbc7c0b4eb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.711+0000 7fbc9733e700 1 --2- 172.21.15.70:0/580694972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc90108510 0x7fbc901a5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.711+0000 7fbc9733e700 1 --2- 172.21.15.70:0/580694972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc90102430 0x7fbc9019a9b0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.712+0000 7fbc9733e700 1 --2- 172.21.15.70:0/580694972 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc90102df0 0x7fbc9019aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:53.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.712+0000 7fbc9733e700 1 -- 172.21.15.70:0/580694972 >> 172.21.15.70:0/580694972 conn(0x7fbc900793b0 msgr2=0x7fbc90100640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:53.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.712+0000 7fbc9733e700 1 -- 172.21.15.70:0/580694972 shutdown_connections 2024-06-04T05:05:53.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:53.712+0000 7fbc9733e700 1 -- 172.21.15.70:0/580694972 wait complete. 2024-06-04T05:05:53.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149287 1.96 MB/sec execute 286 sec latency 23818.843 ms 2024-06-04T05:05:54.598 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '18106a7e', total seconds elapsed: 32.3 2024-06-04T05:05:54.598 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.6 2024-06-04T05:05:54.598 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:05:54.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149540 1.96 MB/sec execute 287 sec latency 15.220 ms 2024-06-04T05:05:55.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:55 smithi112 ceph-mon[36453]: pgmap v267: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-06-04T05:05:55.480 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:55 smithi070 ceph-mon[32631]: pgmap v267: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-06-04T05:05:55.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:55 smithi098 ceph-mon[35569]: pgmap v267: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-06-04T05:05:55.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 149971 1.95 MB/sec execute 288 sec latency 16.688 ms 2024-06-04T05:05:55.833 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:05:56.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.277+0000 7fed39fda640 1 -- 172.21.15.70:0/3211200800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed34101980 msgr2=0x7fed34101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.277+0000 7fed39fda640 1 --2- 172.21.15.70:0/3211200800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed34101980 0x7fed34101e00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fed28009860 tx=0x7fed2802a340 comp rx=0 tx=0).stop 2024-06-04T05:05:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.277+0000 7fed39fda640 1 -- 172.21.15.70:0/3211200800 shutdown_connections 2024-06-04T05:05:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.277+0000 7fed39fda640 1 --2- 172.21.15.70:0/3211200800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed34101980 0x7fed34101e00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.277+0000 7fed39fda640 1 --2- 172.21.15.70:0/3211200800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed34102340 0x7fed3410e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.277+0000 7fed39fda640 1 --2- 172.21.15.70:0/3211200800 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed34107940 0x7fed34107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.277+0000 7fed39fda640 1 -- 172.21.15.70:0/3211200800 >> 172.21.15.70:0/3211200800 conn(0x7fed34078ad0 msgr2=0x7fed34078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.277+0000 7fed39fda640 1 -- 172.21.15.70:0/3211200800 shutdown_connections 2024-06-04T05:05:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.277+0000 7fed39fda640 1 -- 172.21.15.70:0/3211200800 wait complete. 2024-06-04T05:05:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.278+0000 7fed39fda640 1 Processor -- start 2024-06-04T05:05:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.278+0000 7fed39fda640 1 -- start start 2024-06-04T05:05:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.278+0000 7fed39fda640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed34101980 0x7fed34199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.278+0000 7fed39fda640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed34102340 0x7fed34199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.278+0000 7fed39fda640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed34107940 0x7fed3419d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.278+0000 7fed39fda640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fed34111ab0 con 0x7fed34102340 2024-06-04T05:05:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.278+0000 7fed39fda640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed34111930 con 0x7fed34101980 2024-06-04T05:05:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.278+0000 7fed39fda640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fed34111c30 con 0x7fed34107940 2024-06-04T05:05:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.278+0000 7fed32ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed34102340 0x7fed34199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.278+0000 7fed32ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed34102340 0x7fed34199640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53052/0 (socket says 172.21.15.70:53052) 2024-06-04T05:05:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.278+0000 7fed32ffd640 1 -- 172.21.15.70:0/1204513351 learned_addr learned my addr 172.21.15.70:0/1204513351 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.278+0000 7fed33fff640 1 --2- 172.21.15.70:0/1204513351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed34107940 0x7fed3419d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.278+0000 7fed337fe640 1 --2- 172.21.15.70:0/1204513351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed34101980 0x7fed34199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.279+0000 7fed32ffd640 1 -- 172.21.15.70:0/1204513351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed34107940 msgr2=0x7fed3419d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.279+0000 7fed32ffd640 1 --2- 172.21.15.70:0/1204513351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed34107940 0x7fed3419d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.279+0000 7fed32ffd640 1 -- 172.21.15.70:0/1204513351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed34101980 msgr2=0x7fed34199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:56.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.279+0000 7fed32ffd640 1 --2- 172.21.15.70:0/1204513351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed34101980 0x7fed34199100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.279+0000 7fed32ffd640 1 -- 172.21.15.70:0/1204513351 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3419e0f0 con 0x7fed34102340 2024-06-04T05:05:56.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.279+0000 7fed32ffd640 1 --2- 172.21.15.70:0/1204513351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed34102340 0x7fed34199640 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fed28035040 tx=0x7fed28030d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:56.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.279+0000 7fed30ff9640 1 -- 172.21.15.70:0/1204513351 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed2803f020 con 0x7fed34102340 2024-06-04T05:05:56.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.279+0000 7fed39fda640 1 -- 172.21.15.70:0/1204513351 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed3419e380 con 0x7fed34102340 2024-06-04T05:05:56.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.279+0000 7fed30ff9640 1 -- 172.21.15.70:0/1204513351 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed280044d0 con 0x7fed34102340 2024-06-04T05:05:56.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.279+0000 7fed30ff9640 1 -- 172.21.15.70:0/1204513351 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed28037040 con 0x7fed34102340 2024-06-04T05:05:56.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.279+0000 7fed39fda640 1 -- 172.21.15.70:0/1204513351 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed341a5c70 con 0x7fed34102340 2024-06-04T05:05:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed30ff9640 1 -- 172.21.15.70:0/1204513351 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fed28042020 con 0x7fed34102340 2024-06-04T05:05:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed3cdc9740 1 Processor -- start 2024-06-04T05:05:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed3cdc9740 1 -- start start 2024-06-04T05:05:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed3cdc9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff472820 0x5598ff472c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed3cdc9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff473140 0x5598ff47d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed3cdc9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff47da00 0x5598ff47fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed3cdc9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5598ff3e4c30 con 0x5598ff472820 2024-06-04T05:05:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed3cdc9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5598ff3d47d0 con 0x5598ff47da00 2024-06-04T05:05:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed3cdc9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5598ff3d8810 con 0x5598ff473140 2024-06-04T05:05:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed33fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff472820 0x5598ff472c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed33fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff472820 0x5598ff472c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53054/0 (socket says 172.21.15.70:53054) 2024-06-04T05:05:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed33fff640 1 -- 172.21.15.70:0/2992924415 learned_addr learned my addr 172.21.15.70:0/2992924415 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed337fe640 1 --2- 172.21.15.70:0/2992924415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff473140 0x5598ff47d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed32ffd640 1 --2- 172.21.15.70:0/2992924415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff47da00 0x5598ff47fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed337fe640 1 -- 172.21.15.70:0/2992924415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff47da00 msgr2=0x5598ff47fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed337fe640 1 --2- 172.21.15.70:0/2992924415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff47da00 0x5598ff47fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed337fe640 1 -- 172.21.15.70:0/2992924415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff472820 msgr2=0x5598ff472c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed337fe640 1 --2- 172.21.15.70:0/2992924415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff472820 0x5598ff472c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed337fe640 1 -- 172.21.15.70:0/2992924415 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598ff0e9210 con 0x5598ff473140 2024-06-04T05:05:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.282+0000 7fed33fff640 1 --2- 172.21.15.70:0/2992924415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff472820 0x5598ff472c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.283+0000 7fed30ff9640 1 --2- 172.21.15.70:0/1204513351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fed1c0b2a10 0x7fed1c0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.283+0000 7fed33fff640 1 --2- 172.21.15.70:0/1204513351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fed1c0b2a10 0x7fed1c0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.283+0000 7fed337fe640 1 --2- 172.21.15.70:0/2992924415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff473140 0x5598ff47d4c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fed24005fd0 tx=0x7fed240107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfd82eee9484b010 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.283+0000 7fed30ff9640 1 -- 172.21.15.70:0/1204513351 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fed28102330 con 0x7fed34102340 2024-06-04T05:05:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.283+0000 7fed19ffb640 1 -- 172.21.15.70:0/2992924415 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed2401c480 con 0x5598ff473140 2024-06-04T05:05:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.283+0000 7fed19ffb640 1 -- 172.21.15.70:0/2992924415 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed2401cab0 con 0x5598ff473140 2024-06-04T05:05:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.283+0000 7fed19ffb640 1 -- 172.21.15.70:0/2992924415 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed24011a10 con 0x5598ff473140 2024-06-04T05:05:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.284+0000 7fed3cdc9740 1 -- 172.21.15.70:0/2992924415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff473140 msgr2=0x5598ff47d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.284+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/2992924415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff473140 0x5598ff47d4c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fed24005fd0 tx=0x7fed240107a0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.284+0000 7fed33fff640 1 --2- 172.21.15.70:0/1204513351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fed1c0b2a10 0x7fed1c0b4ed0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fed2c005c40 tx=0x7fed2c010040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.284+0000 7fed3cdc9740 1 -- 172.21.15.70:0/2992924415 shutdown_connections 2024-06-04T05:05:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.284+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/2992924415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff47da00 0x5598ff47fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.284+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/2992924415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff473140 0x5598ff47d4c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.284+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/2992924415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff472820 0x5598ff472c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.284+0000 7fed3cdc9740 1 -- 172.21.15.70:0/2992924415 >> 172.21.15.70:0/2992924415 conn(0x5598ff46ed30 msgr2=0x5598ff46f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.284+0000 7fed3cdc9740 1 -- 172.21.15.70:0/2992924415 shutdown_connections 2024-06-04T05:05:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.284+0000 7fed3cdc9740 1 -- 172.21.15.70:0/2992924415 wait complete. 2024-06-04T05:05:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.284+0000 7fed3cdc9740 1 -- start start 2024-06-04T05:05:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.284+0000 7fed3cdc9740 1 Processor -- start 2024-06-04T05:05:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed3cdc9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff472820 0x5598ff47aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed3cdc9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 0x5598ff47af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed3cdc9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff47da00 0x5598ff47b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed3cdc9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5598ff0ba790 con 0x5598ff473140 2024-06-04T05:05:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed3cdc9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5598ff390890 con 0x5598ff472820 2024-06-04T05:05:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed3cdc9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5598ff480630 con 0x5598ff47da00 2024-06-04T05:05:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed3cdc9740 10 client.0 _ll_register_callbacks cb 0x5598ff0e8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed337fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff472820 0x5598ff47aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed32ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 0x5598ff47af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed337fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff472820 0x5598ff47aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:58136/0 (socket says 172.21.15.70:58136) 2024-06-04T05:05:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed337fe640 1 -- 172.21.15.70:0/1218431778 learned_addr learned my addr 172.21.15.70:0/1218431778 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed33fff640 1 --2- 172.21.15.70:0/1218431778 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff47da00 0x5598ff47b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed337fe640 1 -- 172.21.15.70:0/1218431778 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff47da00 msgr2=0x5598ff47b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed337fe640 1 --2- 172.21.15.70:0/1218431778 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff47da00 0x5598ff47b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed337fe640 1 -- 172.21.15.70:0/1218431778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 msgr2=0x5598ff47af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed337fe640 1 --2- 172.21.15.70:0/1218431778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 0x5598ff47af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed337fe640 1 -- 172.21.15.70:0/1218431778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598ff1b5620 con 0x5598ff472820 2024-06-04T05:05:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed32ffd640 1 --2- 172.21.15.70:0/1218431778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 0x5598ff47af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed337fe640 1 --2- 172.21.15.70:0/1218431778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff472820 0x5598ff47aa40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fed24000c00 tx=0x7fed2400bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90932c231a230af5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.285+0000 7fed12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:05:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.286+0000 7fed12ffd640 1 -- 172.21.15.70:0/1218431778 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed2402b060 con 0x5598ff472820 2024-06-04T05:05:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.286+0000 7fed3cdc9740 1 -- 172.21.15.70:0/1218431778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed2401cab0 con 0x5598ff472820 2024-06-04T05:05:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.286+0000 7fed12ffd640 1 -- 172.21.15.70:0/1218431778 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fed2401c690 con 0x5598ff472820 2024-06-04T05:05:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.286+0000 7fed12ffd640 1 -- 172.21.15.70:0/1218431778 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fed2401c890 con 0x5598ff472820 2024-06-04T05:05:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.286+0000 7fed3cdc9740 10 client.26969 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.286+0000 7fed3cdc9740 1 -- 172.21.15.70:0/1218431778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5598ff48c440 con 0x5598ff472820 2024-06-04T05:05:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.286+0000 7fed12ffd640 1 -- 172.21.15.70:0/1218431778 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed2401cab0 con 0x5598ff472820 2024-06-04T05:05:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.286+0000 7fed12ffd640 1 -- 172.21.15.70:0/1218431778 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fed2402e670 con 0x5598ff472820 2024-06-04T05:05:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.286+0000 7fed3cdc9740 10 client.26969 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.286+0000 7fed3cdc9740 10 client.26969 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:05:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.286+0000 7fed3cdc9740 20 client.26969 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.286+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/1218431778 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5598ff48d8a0 0x5598ff48fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.286+0000 7fed3cdc9740 4 client.26969 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:05:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.286+0000 7fed3cdc9740 1 -- 172.21.15.70:0/1218431778 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598ff3e6b80 con 0x5598ff48d8a0 2024-06-04T05:05:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.287+0000 7fed32ffd640 1 --2- 172.21.15.70:0/1218431778 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5598ff48d8a0 0x5598ff48fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.287+0000 7fed12ffd640 10 client.26969 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.287+0000 7fed32ffd640 1 --2- 172.21.15.70:0/1218431778 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5598ff48d8a0 0x5598ff48fc90 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=522353d3d21d1222 server_cookie=dbb08135d3276d72 in_seq=0 out_seq=0 2024-06-04T05:05:56.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.287+0000 7fed12ffd640 1 -- 172.21.15.70:0/1218431778 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5598ff3e6b80 con 0x5598ff48d8a0 2024-06-04T05:05:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.287+0000 7fed12ffd640 10 client.26969 handle_command_reply: tid=0 2024-06-04T05:05:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.288+0000 7fed3cdc9740 1 client.26969 shutdown 2024-06-04T05:05:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.288+0000 7fed3cdc9740 1 -- 172.21.15.70:0/1218431778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff472820 msgr2=0x5598ff47aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.288+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/1218431778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff472820 0x5598ff47aa40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fed24000c00 tx=0x7fed2400bf50 comp rx=0 tx=0).stop 2024-06-04T05:05:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.288+0000 7fed3cdc9740 1 -- 172.21.15.70:0/1218431778 shutdown_connections 2024-06-04T05:05:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.288+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/1218431778 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5598ff48d8a0 0x5598ff48fc90 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.288+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/1218431778 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff47da00 0x5598ff47b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.288+0000 7fed12ffd640 0 client.26969 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.288+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/1218431778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 0x5598ff47af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.288+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/1218431778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff472820 0x5598ff47aa40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.288+0000 7fed3cdc9740 1 -- 172.21.15.70:0/1218431778 >> 172.21.15.70:0/1218431778 conn(0x5598ff46ed30 msgr2=0x5598ff474ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.288+0000 7fed3cdc9740 1 -- 172.21.15.70:0/1218431778 shutdown_connections 2024-06-04T05:05:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.288+0000 7fed3cdc9740 1 -- 172.21.15.70:0/1218431778 wait complete. 2024-06-04T05:05:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.288+0000 7fed3cdc9740 20 client.26969 trim_cache size 0 max 16384 2024-06-04T05:05:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed3cdc9740 1 Processor -- start 2024-06-04T05:05:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed3cdc9740 1 -- start start 2024-06-04T05:05:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed3cdc9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff472820 0x5598ff47b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed3cdc9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 0x5598ff4748e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed3cdc9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff47da00 0x5598ff474e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed3cdc9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fed341ae600 con 0x5598ff473140 2024-06-04T05:05:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed3cdc9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed341b0140 con 0x5598ff47da00 2024-06-04T05:05:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed3cdc9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fed341aecd0 con 0x5598ff472820 2024-06-04T05:05:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed32ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 0x5598ff4748e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed32ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 0x5598ff4748e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53068/0 (socket says 172.21.15.70:53068) 2024-06-04T05:05:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed32ffd640 1 -- 172.21.15.70:0/2932363185 learned_addr learned my addr 172.21.15.70:0/2932363185 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed33fff640 1 --2- 172.21.15.70:0/2932363185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff47da00 0x5598ff474e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed32ffd640 1 -- 172.21.15.70:0/2932363185 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff472820 msgr2=0x5598ff47b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed337fe640 1 --2- 172.21.15.70:0/2932363185 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff472820 0x5598ff47b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed32ffd640 1 --2- 172.21.15.70:0/2932363185 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff472820 0x5598ff47b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed32ffd640 1 -- 172.21.15.70:0/2932363185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff47da00 msgr2=0x5598ff474e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed32ffd640 1 --2- 172.21.15.70:0/2932363185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff47da00 0x5598ff474e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.306+0000 7fed32ffd640 1 -- 172.21.15.70:0/2932363185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598ff480510 con 0x5598ff473140 2024-06-04T05:05:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.307+0000 7fed337fe640 1 --2- 172.21.15.70:0/2932363185 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff472820 0x5598ff47b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:05:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.307+0000 7fed32ffd640 1 --2- 172.21.15.70:0/2932363185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 0x5598ff4748e0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fed280c9390 tx=0x7fed280ec040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31fddb91dabde4c8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.307+0000 7fed197fa640 1 -- 172.21.15.70:0/2932363185 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed280c5550 con 0x5598ff473140 2024-06-04T05:05:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.307+0000 7fed197fa640 1 -- 172.21.15.70:0/2932363185 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed280c91a0 con 0x5598ff473140 2024-06-04T05:05:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.307+0000 7fed197fa640 1 -- 172.21.15.70:0/2932363185 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed280c8e00 con 0x5598ff473140 2024-06-04T05:05:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.308+0000 7fed3cdc9740 1 -- 172.21.15.70:0/2932363185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 msgr2=0x5598ff4748e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.308+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/2932363185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 0x5598ff4748e0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fed280c9390 tx=0x7fed280ec040 comp rx=0 tx=0).stop 2024-06-04T05:05:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.308+0000 7fed3cdc9740 1 -- 172.21.15.70:0/2932363185 shutdown_connections 2024-06-04T05:05:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.308+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/2932363185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff47da00 0x5598ff474e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.308+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/2932363185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 0x5598ff4748e0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.308+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/2932363185 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff472820 0x5598ff47b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.308+0000 7fed3cdc9740 1 -- 172.21.15.70:0/2932363185 >> 172.21.15.70:0/2932363185 conn(0x5598ff46ed30 msgr2=0x5598ff4741d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.308+0000 7fed3cdc9740 1 -- 172.21.15.70:0/2932363185 shutdown_connections 2024-06-04T05:05:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.308+0000 7fed3cdc9740 1 -- 172.21.15.70:0/2932363185 wait complete. 2024-06-04T05:05:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.308+0000 7fed3cdc9740 1 -- start start 2024-06-04T05:05:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.308+0000 7fed3cdc9740 1 Processor -- start 2024-06-04T05:05:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.308+0000 7fed3cdc9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff472820 0x5598ff477ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.308+0000 7fed3cdc9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 0x5598ff478000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.308+0000 7fed3cdc9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff47da00 0x5598ff478540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.308+0000 7fed3cdc9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fed341afa70 con 0x5598ff473140 2024-06-04T05:05:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.308+0000 7fed3cdc9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed341af3a0 con 0x5598ff47da00 2024-06-04T05:05:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.308+0000 7fed3cdc9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fed341ac600 con 0x5598ff472820 2024-06-04T05:05:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed3cdc9740 10 client.0 _ll_register_callbacks cb 0x7fed2c009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed32ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 0x5598ff478000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed32ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 0x5598ff478000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53084/0 (socket says 172.21.15.70:53084) 2024-06-04T05:05:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed337fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff472820 0x5598ff477ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed32ffd640 1 -- 172.21.15.70:0/3995820674 learned_addr learned my addr 172.21.15.70:0/3995820674 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed33fff640 1 --2- 172.21.15.70:0/3995820674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff47da00 0x5598ff478540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed32ffd640 1 -- 172.21.15.70:0/3995820674 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff472820 msgr2=0x5598ff477ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed32ffd640 1 --2- 172.21.15.70:0/3995820674 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff472820 0x5598ff477ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed32ffd640 1 -- 172.21.15.70:0/3995820674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff47da00 msgr2=0x5598ff478540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed32ffd640 1 --2- 172.21.15.70:0/3995820674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff47da00 0x5598ff478540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed32ffd640 1 -- 172.21.15.70:0/3995820674 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598ff4f7160 con 0x5598ff473140 2024-06-04T05:05:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed337fe640 1 --2- 172.21.15.70:0/3995820674 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff472820 0x5598ff477ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed32ffd640 1 --2- 172.21.15.70:0/3995820674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 0x5598ff478000 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fed280c6900 tx=0x7fed2802ae20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb2ee3779a0072e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:05:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed12ffd640 1 -- 172.21.15.70:0/3995820674 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed280c8fe0 con 0x5598ff473140 2024-06-04T05:05:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed12ffd640 1 -- 172.21.15.70:0/3995820674 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fed28030460 con 0x5598ff473140 2024-06-04T05:05:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed3cdc9740 1 -- 172.21.15.70:0/3995820674 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed280c91a0 con 0x5598ff473140 2024-06-04T05:05:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed12ffd640 1 -- 172.21.15.70:0/3995820674 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fed280c9460 con 0x5598ff473140 2024-06-04T05:05:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed3cdc9740 10 client.17808 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.309+0000 7fed3cdc9740 1 -- 172.21.15.70:0/3995820674 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5598ff4f83c0 con 0x5598ff473140 2024-06-04T05:05:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.310+0000 7fed12ffd640 1 -- 172.21.15.70:0/3995820674 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed280c91a0 con 0x5598ff473140 2024-06-04T05:05:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.310+0000 7fed12ffd640 1 -- 172.21.15.70:0/3995820674 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fed2803a030 con 0x5598ff473140 2024-06-04T05:05:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.310+0000 7fed3cdc9740 10 client.17808 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.310+0000 7fed3cdc9740 10 client.17808 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:05:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.310+0000 7fed3cdc9740 20 client.17808 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.310+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/3995820674 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5598ff48d8a0 0x5598ff4fa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.310+0000 7fed3cdc9740 4 client.17808 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:05:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.310+0000 7fed3cdc9740 1 -- 172.21.15.70:0/3995820674 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5598ff3e0360 con 0x5598ff48d8a0 2024-06-04T05:05:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.310+0000 7fed337fe640 1 --2- 172.21.15.70:0/3995820674 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5598ff48d8a0 0x5598ff4fa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.310+0000 7fed337fe640 1 --2- 172.21.15.70:0/3995820674 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5598ff48d8a0 0x5598ff4fa8f0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a52236ba3de7b657 server_cookie=b69dceed8a8421a9 in_seq=0 out_seq=0 2024-06-04T05:05:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.310+0000 7fed12ffd640 10 client.17808 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.325+0000 7fed12ffd640 1 -- 172.21.15.70:0/3995820674 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5598ff3e0360 con 0x5598ff48d8a0 2024-06-04T05:05:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.325+0000 7fed12ffd640 10 client.17808 handle_command_reply: tid=0 2024-06-04T05:05:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.325+0000 7fed3cdc9740 1 client.17808 shutdown 2024-06-04T05:05:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.326+0000 7fed3cdc9740 1 -- 172.21.15.70:0/3995820674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 msgr2=0x5598ff478000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.326+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/3995820674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 0x5598ff478000 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fed280c6900 tx=0x7fed2802ae20 comp rx=0 tx=0).stop 2024-06-04T05:05:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.326+0000 7fed3cdc9740 1 -- 172.21.15.70:0/3995820674 shutdown_connections 2024-06-04T05:05:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.326+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/3995820674 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5598ff48d8a0 0x5598ff4fa8f0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.326+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/3995820674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598ff47da00 0x5598ff478540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.326+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/3995820674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598ff473140 0x5598ff478000 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.326+0000 7fed3cdc9740 1 --2- 172.21.15.70:0/3995820674 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598ff472820 0x5598ff477ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.326+0000 7fed3cdc9740 1 -- 172.21.15.70:0/3995820674 >> 172.21.15.70:0/3995820674 conn(0x5598ff46ed30 msgr2=0x5598ff474a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.326+0000 7fed12ffd640 0 client.17808 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.326+0000 7fed3cdc9740 1 -- 172.21.15.70:0/3995820674 shutdown_connections 2024-06-04T05:05:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.326+0000 7fed3cdc9740 1 -- 172.21.15.70:0/3995820674 wait complete. 2024-06-04T05:05:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.326+0000 7fed3cdc9740 20 client.17808 trim_cache size 0 max 16384 2024-06-04T05:05:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:05:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:05:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:05:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.327+0000 7fed39fda640 1 -- 172.21.15.70:0/1204513351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fed1c0b2a10 msgr2=0x7fed1c0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.327+0000 7fed39fda640 1 --2- 172.21.15.70:0/1204513351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fed1c0b2a10 0x7fed1c0b4ed0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fed2c005c40 tx=0x7fed2c010040 comp rx=0 tx=0).stop 2024-06-04T05:05:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.327+0000 7fed39fda640 1 -- 172.21.15.70:0/1204513351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed34102340 msgr2=0x7fed34199640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.327+0000 7fed39fda640 1 --2- 172.21.15.70:0/1204513351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed34102340 0x7fed34199640 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fed28035040 tx=0x7fed28030d20 comp rx=0 tx=0).stop 2024-06-04T05:05:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.327+0000 7fed39fda640 1 -- 172.21.15.70:0/1204513351 shutdown_connections 2024-06-04T05:05:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.327+0000 7fed39fda640 1 --2- 172.21.15.70:0/1204513351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fed1c0b2a10 0x7fed1c0b4ed0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.327+0000 7fed39fda640 1 --2- 172.21.15.70:0/1204513351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed34107940 0x7fed3419d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.327+0000 7fed39fda640 1 --2- 172.21.15.70:0/1204513351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed34102340 0x7fed34199640 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.327+0000 7fed39fda640 1 --2- 172.21.15.70:0/1204513351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed34101980 0x7fed34199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.327+0000 7fed39fda640 1 -- 172.21.15.70:0/1204513351 >> 172.21.15.70:0/1204513351 conn(0x7fed34078ad0 msgr2=0x7fed34100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.327+0000 7fed39fda640 1 -- 172.21.15.70:0/1204513351 shutdown_connections 2024-06-04T05:05:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:05:56.327+0000 7fed39fda640 1 -- 172.21.15.70:0/1204513351 wait complete. 2024-06-04T05:05:56.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 150493 1.96 MB/sec execute 289 sec latency 20.805 ms 2024-06-04T05:05:57.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:57 smithi070 ceph-mon[32631]: pgmap v268: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 302 KiB/s wr, 42 op/s 2024-06-04T05:05:57.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:57 smithi112 ceph-mon[36453]: pgmap v268: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 302 KiB/s wr, 42 op/s 2024-06-04T05:05:57.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.467+0000 7f6876e6f700 1 -- 172.21.15.70:0/2280445134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6870102e00 msgr2=0x7f687010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:57.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.467+0000 7f6876e6f700 1 --2- 172.21.15.70:0/2280445134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6870102e00 0x7f687010f2f0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f686800b0a0 tx=0x7f686800b3b0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.467+0000 7f6876e6f700 1 -- 172.21.15.70:0/2280445134 shutdown_connections 2024-06-04T05:05:57.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.467+0000 7f6876e6f700 1 --2- 172.21.15.70:0/2280445134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6870102e00 0x7f687010f2f0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.467+0000 7f6876e6f700 1 --2- 172.21.15.70:0/2280445134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6870108520 0x7f6870108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.467+0000 7f6876e6f700 1 --2- 172.21.15.70:0/2280445134 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6870102440 0x7f68701028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.467+0000 7f6876e6f700 1 -- 172.21.15.70:0/2280445134 >> 172.21.15.70:0/2280445134 conn(0x7f68700793f0 msgr2=0x7f6870079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:57.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.467+0000 7f6876e6f700 1 -- 172.21.15.70:0/2280445134 shutdown_connections 2024-06-04T05:05:57.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.468+0000 7f6876e6f700 1 -- 172.21.15.70:0/2280445134 wait complete. 2024-06-04T05:05:57.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.468+0000 7f6876e6f700 1 Processor -- start 2024-06-04T05:05:57.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.468+0000 7f6876e6f700 1 -- start start 2024-06-04T05:05:57.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f6876e6f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6870102440 0x7f687019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f6876e6f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6870102e00 0x7f687019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f6876e6f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6870108520 0x7f68701987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f6876e6f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6870112480 con 0x7f6870108520 2024-06-04T05:05:57.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f6876e6f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6870112300 con 0x7f6870102e00 2024-06-04T05:05:57.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f6876e6f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6870112600 con 0x7f6870102440 2024-06-04T05:05:57.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f6874c0b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6870102440 0x7f687019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f686ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6870102e00 0x7f687019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f686ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6870102e00 0x7f687019eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:58186/0 (socket says 172.21.15.70:58186) 2024-06-04T05:05:57.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f6874c0b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6870102440 0x7f687019e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46396/0 (socket says 172.21.15.70:46396) 2024-06-04T05:05:57.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f686ffff700 1 -- 172.21.15.70:0/4203376607 learned_addr learned my addr 172.21.15.70:0/4203376607 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:57.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f687540c700 1 --2- 172.21.15.70:0/4203376607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6870108520 0x7f68701987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f687540c700 1 -- 172.21.15.70:0/4203376607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6870102440 msgr2=0x7f687019e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:57.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f687540c700 1 --2- 172.21.15.70:0/4203376607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6870102440 0x7f687019e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f687540c700 1 -- 172.21.15.70:0/4203376607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6870102e00 msgr2=0x7f687019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:57.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f687540c700 1 --2- 172.21.15.70:0/4203376607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6870102e00 0x7f687019eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f687540c700 1 -- 172.21.15.70:0/4203376607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6870198fa0 con 0x7f6870108520 2024-06-04T05:05:57.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f686ffff700 1 --2- 172.21.15.70:0/4203376607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6870102e00 0x7f687019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:57.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.469+0000 7f687540c700 1 --2- 172.21.15.70:0/4203376607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6870108520 0x7f68701987c0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f686800b0a0 tx=0x7f6868004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:57.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.470+0000 7f686dffb700 1 -- 172.21.15.70:0/4203376607 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6868021070 con 0x7f6870108520 2024-06-04T05:05:57.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.470+0000 7f6876e6f700 1 -- 172.21.15.70:0/4203376607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6870199230 con 0x7f6870108520 2024-06-04T05:05:57.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.470+0000 7f686dffb700 1 -- 172.21.15.70:0/4203376607 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6868007b80 con 0x7f6870108520 2024-06-04T05:05:57.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.470+0000 7f686dffb700 1 -- 172.21.15.70:0/4203376607 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6868005040 con 0x7f6870108520 2024-06-04T05:05:57.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.470+0000 7f6876e6f700 1 -- 172.21.15.70:0/4203376607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f687006b220 con 0x7f6870108520 2024-06-04T05:05:57.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.472+0000 7f686dffb700 1 -- 172.21.15.70:0/4203376607 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f6868024070 con 0x7f6870108520 2024-06-04T05:05:57.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.473+0000 7f686dffb700 1 --2- 172.21.15.70:0/4203376607 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68600b2a70 0x7f68600b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.473+0000 7f6874c0b700 1 --2- 172.21.15.70:0/4203376607 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68600b2a70 0x7f68600b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.473+0000 7f686dffb700 1 -- 172.21.15.70:0/4203376607 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f686800f020 con 0x7f6870108520 2024-06-04T05:05:57.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.474+0000 7f6881704b80 1 Processor -- start 2024-06-04T05:05:57.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.474+0000 7f6881704b80 1 -- start start 2024-06-04T05:05:57.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.474+0000 7f6874c0b700 1 --2- 172.21.15.70:0/4203376607 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68600b2a70 0x7f68600b4f30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f686400ffb0 tx=0x7f686400f7a0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:57.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.474+0000 7f6881704b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee118e0 0x556f9ee11cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.474+0000 7f6881704b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee12290 0x556f9ee159c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.474+0000 7f6881704b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee15fe0 0x556f9ee16420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.474+0000 7f6881704b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556f9ed70d10 con 0x556f9ee118e0 2024-06-04T05:05:57.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.474+0000 7f6881704b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556f9ed608b0 con 0x556f9ee15fe0 2024-06-04T05:05:57.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.474+0000 7f6881704b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556f9ed648f0 con 0x556f9ee12290 2024-06-04T05:05:57.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.475+0000 7f686ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee12290 0x556f9ee159c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.475+0000 7f686ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee12290 0x556f9ee159c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46412/0 (socket says 172.21.15.70:46412) 2024-06-04T05:05:57.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.475+0000 7f686ffff700 1 -- 172.21.15.70:0/3683160127 learned_addr learned my addr 172.21.15.70:0/3683160127 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:57.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.475+0000 7f6874c0b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee118e0 0x556f9ee11cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.475+0000 7f687540c700 1 --2- 172.21.15.70:0/3683160127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee15fe0 0x556f9ee16420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.475+0000 7f686ffff700 1 -- 172.21.15.70:0/3683160127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee15fe0 msgr2=0x556f9ee16420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:57.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.475+0000 7f686ffff700 1 --2- 172.21.15.70:0/3683160127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee15fe0 0x556f9ee16420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.475+0000 7f686ffff700 1 -- 172.21.15.70:0/3683160127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee118e0 msgr2=0x556f9ee11cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:57.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.475+0000 7f686ffff700 1 --2- 172.21.15.70:0/3683160127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee118e0 0x556f9ee11cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.475+0000 7f686ffff700 1 -- 172.21.15.70:0/3683160127 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f9ecb0950 con 0x556f9ee12290 2024-06-04T05:05:57.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.475+0000 7f6874c0b700 1 --2- 172.21.15.70:0/3683160127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee118e0 0x556f9ee11cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:57.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.475+0000 7f686ffff700 1 --2- 172.21.15.70:0/3683160127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee12290 0x556f9ee159c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f685c00d410 tx=0x7f685c00c2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5e1e0266901ea82 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:57.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.475+0000 7f685affd700 1 -- 172.21.15.70:0/3683160127 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c00ce30 con 0x556f9ee12290 2024-06-04T05:05:57.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.475+0000 7f685affd700 1 -- 172.21.15.70:0/3683160127 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f685c016910 con 0x556f9ee12290 2024-06-04T05:05:57.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.475+0000 7f685affd700 1 -- 172.21.15.70:0/3683160127 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c0158e0 con 0x556f9ee12290 2024-06-04T05:05:57.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.476+0000 7f6881704b80 1 -- 172.21.15.70:0/3683160127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee12290 msgr2=0x556f9ee159c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:57.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.476+0000 7f6881704b80 1 --2- 172.21.15.70:0/3683160127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee12290 0x556f9ee159c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f685c00d410 tx=0x7f685c00c2f0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.476+0000 7f6881704b80 1 -- 172.21.15.70:0/3683160127 shutdown_connections 2024-06-04T05:05:57.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.476+0000 7f6881704b80 1 --2- 172.21.15.70:0/3683160127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee15fe0 0x556f9ee16420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.476+0000 7f6881704b80 1 --2- 172.21.15.70:0/3683160127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee118e0 0x556f9ee11cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.476+0000 7f6881704b80 1 --2- 172.21.15.70:0/3683160127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee12290 0x556f9ee159c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.476+0000 7f6881704b80 1 -- 172.21.15.70:0/3683160127 >> 172.21.15.70:0/3683160127 conn(0x556f9ed3f100 msgr2=0x556f9ee0bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:57.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.476+0000 7f6881704b80 1 -- 172.21.15.70:0/3683160127 shutdown_connections 2024-06-04T05:05:57.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.476+0000 7f6881704b80 1 -- 172.21.15.70:0/3683160127 wait complete. 2024-06-04T05:05:57.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.477+0000 7f6881704b80 1 -- start start 2024-06-04T05:05:57.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.477+0000 7f6881704b80 1 Processor -- start 2024-06-04T05:05:57.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.477+0000 7f6881704b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee118e0 0x556f9ee1d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.477+0000 7f6881704b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 0x556f9ee1a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.477+0000 7f6881704b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee15fe0 0x556f9ee1ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.477+0000 7f6881704b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556f9ec67020 con 0x556f9ee12290 2024-06-04T05:05:57.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.477+0000 7f6881704b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556f9ed02840 con 0x556f9ee118e0 2024-06-04T05:05:57.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.477+0000 7f6881704b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556f9ee1e1a0 con 0x556f9ee15fe0 2024-06-04T05:05:57.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.477+0000 7f6881704b80 10 client.0 _ll_register_callbacks cb 0x556f9ecfb9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:57.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.477+0000 7f686ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 0x556f9ee1a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.477+0000 7f686ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 0x556f9ee1a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53112/0 (socket says 172.21.15.70:53112) 2024-06-04T05:05:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.477+0000 7f687540c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee15fe0 0x556f9ee1ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.478+0000 7f686ffff700 1 -- 172.21.15.70:0/3082311663 learned_addr learned my addr 172.21.15.70:0/3082311663 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.478+0000 7f6874c0b700 1 --2- 172.21.15.70:0/3082311663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee118e0 0x556f9ee1d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.478+0000 7f687540c700 1 -- 172.21.15.70:0/3082311663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee118e0 msgr2=0x556f9ee1d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.478+0000 7f687540c700 1 --2- 172.21.15.70:0/3082311663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee118e0 0x556f9ee1d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.478+0000 7f687540c700 1 -- 172.21.15.70:0/3082311663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 msgr2=0x556f9ee1a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.478+0000 7f687540c700 1 --2- 172.21.15.70:0/3082311663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 0x556f9ee1a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.478+0000 7f687540c700 1 -- 172.21.15.70:0/3082311663 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f9ecff710 con 0x556f9ee15fe0 2024-06-04T05:05:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.478+0000 7f686ffff700 1 --2- 172.21.15.70:0/3082311663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 0x556f9ee1a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.478+0000 7f687540c700 1 --2- 172.21.15.70:0/3082311663 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee15fe0 0x556f9ee1ac00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f68680e11a0 tx=0x7f68680aa530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bcd29e025bc6afa server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.478+0000 7f684ffff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:05:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.478+0000 7f684ffff700 1 -- 172.21.15.70:0/3082311663 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68680a6ce0 con 0x556f9ee15fe0 2024-06-04T05:05:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.478+0000 7f684ffff700 1 -- 172.21.15.70:0/3082311663 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f68680b3700 con 0x556f9ee15fe0 2024-06-04T05:05:57.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.478+0000 7f6881704b80 1 -- 172.21.15.70:0/3082311663 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f685c016910 con 0x556f9ee15fe0 2024-06-04T05:05:57.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.478+0000 7f684ffff700 1 -- 172.21.15.70:0/3082311663 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f68680ab4c0 con 0x556f9ee15fe0 2024-06-04T05:05:57.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.478+0000 7f6881704b80 10 client.27058 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:57.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.478+0000 7f6881704b80 1 -- 172.21.15.70:0/3082311663 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556f9ee28840 con 0x556f9ee15fe0 2024-06-04T05:05:57.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.479+0000 7f684ffff700 1 -- 172.21.15.70:0/3082311663 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68680c1020 con 0x556f9ee15fe0 2024-06-04T05:05:57.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.479+0000 7f684ffff700 1 -- 172.21.15.70:0/3082311663 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f6868011070 con 0x556f9ee15fe0 2024-06-04T05:05:57.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.479+0000 7f6881704b80 10 client.27058 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:57.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.479+0000 7f6881704b80 10 client.27058 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:05:57.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.479+0000 7f6881704b80 20 client.27058 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:57.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.479+0000 7f6881704b80 1 --2- 172.21.15.70:0/3082311663 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556f9ee2aeb0 0x556f9ee2d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.479+0000 7f6881704b80 4 client.27058 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:05:57.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.479+0000 7f6881704b80 1 -- 172.21.15.70:0/3082311663 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f9eddda60 con 0x556f9ee2aeb0 2024-06-04T05:05:57.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.479+0000 7f6874c0b700 1 --2- 172.21.15.70:0/3082311663 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556f9ee2aeb0 0x556f9ee2d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.479+0000 7f684ffff700 10 client.27058 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.479+0000 7f6874c0b700 1 --2- 172.21.15.70:0/3082311663 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556f9ee2aeb0 0x556f9ee2d2d0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce77d6130132fec4 server_cookie=5b6d9420807ad746 in_seq=0 out_seq=0 2024-06-04T05:05:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.480+0000 7f684ffff700 1 -- 172.21.15.70:0/3082311663 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556f9eddda60 con 0x556f9ee2aeb0 2024-06-04T05:05:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.480+0000 7f684ffff700 10 client.27058 handle_command_reply: tid=0 2024-06-04T05:05:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.480+0000 7f6881704b80 1 client.27058 shutdown 2024-06-04T05:05:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.480+0000 7f6881704b80 1 -- 172.21.15.70:0/3082311663 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee15fe0 msgr2=0x556f9ee1ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.480+0000 7f6881704b80 1 --2- 172.21.15.70:0/3082311663 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee15fe0 0x556f9ee1ac00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f68680e11a0 tx=0x7f68680aa530 comp rx=0 tx=0).stop 2024-06-04T05:05:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.480+0000 7f6881704b80 1 -- 172.21.15.70:0/3082311663 shutdown_connections 2024-06-04T05:05:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.480+0000 7f6881704b80 1 --2- 172.21.15.70:0/3082311663 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee15fe0 0x556f9ee1ac00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.480+0000 7f6881704b80 1 --2- 172.21.15.70:0/3082311663 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556f9ee2aeb0 0x556f9ee2d2d0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.480+0000 7f6881704b80 1 --2- 172.21.15.70:0/3082311663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee118e0 0x556f9ee1d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.480+0000 7f6881704b80 1 --2- 172.21.15.70:0/3082311663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 0x556f9ee1a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.480+0000 7f684ffff700 0 client.27058 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.480+0000 7f6881704b80 1 -- 172.21.15.70:0/3082311663 >> 172.21.15.70:0/3082311663 conn(0x556f9ed3f100 msgr2=0x556f9ee173d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:57.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.481+0000 7f6881704b80 1 -- 172.21.15.70:0/3082311663 shutdown_connections 2024-06-04T05:05:57.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.481+0000 7f6881704b80 1 -- 172.21.15.70:0/3082311663 wait complete. 2024-06-04T05:05:57.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.481+0000 7f6881704b80 20 client.27058 trim_cache size 0 max 16384 2024-06-04T05:05:57.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:57 smithi098 ceph-mon[35569]: pgmap v268: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 302 KiB/s wr, 42 op/s 2024-06-04T05:05:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.503+0000 7f6881704b80 1 Processor -- start 2024-06-04T05:05:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.503+0000 7f6881704b80 1 -- start start 2024-06-04T05:05:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.503+0000 7f6881704b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee118e0 0x556f9ee196a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.503+0000 7f6881704b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 0x556f9ee1c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.503+0000 7f6881704b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee15fe0 0x556f9ee32ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.503+0000 7f6881704b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f68701abb10 con 0x556f9ee12290 2024-06-04T05:05:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.503+0000 7f6881704b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68701ad650 con 0x556f9ee118e0 2024-06-04T05:05:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.503+0000 7f6881704b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f68701ac1e0 con 0x556f9ee15fe0 2024-06-04T05:05:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.503+0000 7f686ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 0x556f9ee1c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.503+0000 7f6874c0b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee118e0 0x556f9ee196a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.503+0000 7f686ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 0x556f9ee1c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53128/0 (socket says 172.21.15.70:53128) 2024-06-04T05:05:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.503+0000 7f686ffff700 1 -- 172.21.15.70:0/3394409021 learned_addr learned my addr 172.21.15.70:0/3394409021 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.503+0000 7f687540c700 1 --2- 172.21.15.70:0/3394409021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee15fe0 0x556f9ee32ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.504+0000 7f6874c0b700 1 -- 172.21.15.70:0/3394409021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee15fe0 msgr2=0x556f9ee32ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.504+0000 7f6874c0b700 1 --2- 172.21.15.70:0/3394409021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee15fe0 0x556f9ee32ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.504+0000 7f6874c0b700 1 -- 172.21.15.70:0/3394409021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 msgr2=0x556f9ee1c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.504+0000 7f6874c0b700 1 --2- 172.21.15.70:0/3394409021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 0x556f9ee1c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.504+0000 7f6874c0b700 1 -- 172.21.15.70:0/3394409021 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f9ee0cdc0 con 0x556f9ee118e0 2024-06-04T05:05:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.504+0000 7f6874c0b700 1 --2- 172.21.15.70:0/3394409021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee118e0 0x556f9ee196a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x556f9ee2dae0 tx=0x7f6864011050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bb4f8193d45325b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.504+0000 7f685a7fc700 1 -- 172.21.15.70:0/3394409021 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686401bd10 con 0x556f9ee118e0 2024-06-04T05:05:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.504+0000 7f685a7fc700 1 -- 172.21.15.70:0/3394409021 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f686402f6e0 con 0x556f9ee118e0 2024-06-04T05:05:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.504+0000 7f685a7fc700 1 -- 172.21.15.70:0/3394409021 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6864029410 con 0x556f9ee118e0 2024-06-04T05:05:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.505+0000 7f6881704b80 1 -- 172.21.15.70:0/3394409021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee118e0 msgr2=0x556f9ee196a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.505+0000 7f6881704b80 1 --2- 172.21.15.70:0/3394409021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee118e0 0x556f9ee196a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x556f9ee2dae0 tx=0x7f6864011050 comp rx=0 tx=0).stop 2024-06-04T05:05:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.505+0000 7f6881704b80 1 -- 172.21.15.70:0/3394409021 shutdown_connections 2024-06-04T05:05:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.505+0000 7f6881704b80 1 --2- 172.21.15.70:0/3394409021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee15fe0 0x556f9ee32ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.505+0000 7f6881704b80 1 --2- 172.21.15.70:0/3394409021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee118e0 0x556f9ee196a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.505+0000 7f6881704b80 1 --2- 172.21.15.70:0/3394409021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 0x556f9ee1c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.505+0000 7f6881704b80 1 -- 172.21.15.70:0/3394409021 >> 172.21.15.70:0/3394409021 conn(0x556f9ed3f100 msgr2=0x556f9ee1bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:57.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.505+0000 7f6881704b80 1 -- 172.21.15.70:0/3394409021 shutdown_connections 2024-06-04T05:05:57.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.505+0000 7f6881704b80 1 -- 172.21.15.70:0/3394409021 wait complete. 2024-06-04T05:05:57.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.505+0000 7f6881704b80 1 -- start start 2024-06-04T05:05:57.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.506+0000 7f6881704b80 1 Processor -- start 2024-06-04T05:05:57.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.506+0000 7f6881704b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee118e0 0x556f9ee22520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.506+0000 7f6881704b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 0x556f9ee22a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f6881704b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee15fe0 0x556f9ee95a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f6881704b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f68701acf80 con 0x556f9ee12290 2024-06-04T05:05:57.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f6881704b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68701ac8b0 con 0x556f9ee118e0 2024-06-04T05:05:57.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f6881704b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f68701a9ad0 con 0x556f9ee15fe0 2024-06-04T05:05:57.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f6881704b80 10 client.0 _ll_register_callbacks cb 0x556f9ecfb9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:05:57.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f686ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 0x556f9ee22a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f6874c0b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee118e0 0x556f9ee22520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f686ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 0x556f9ee22a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53146/0 (socket says 172.21.15.70:53146) 2024-06-04T05:05:57.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f686ffff700 1 -- 172.21.15.70:0/850312907 learned_addr learned my addr 172.21.15.70:0/850312907 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:57.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f687540c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee15fe0 0x556f9ee95a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f6874c0b700 1 -- 172.21.15.70:0/850312907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee15fe0 msgr2=0x556f9ee95a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:57.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f6874c0b700 1 --2- 172.21.15.70:0/850312907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee15fe0 0x556f9ee95a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f6874c0b700 1 -- 172.21.15.70:0/850312907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 msgr2=0x556f9ee22a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:05:57.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f6874c0b700 1 --2- 172.21.15.70:0/850312907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 0x556f9ee22a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f6874c0b700 1 -- 172.21.15.70:0/850312907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f9ee95fd0 con 0x556f9ee118e0 2024-06-04T05:05:57.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f687540c700 1 --2- 172.21.15.70:0/850312907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee15fe0 0x556f9ee95a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:05:57.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f6874c0b700 1 --2- 172.21.15.70:0/850312907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee118e0 0x556f9ee22520 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x556f9ee2dae0 tx=0x7f686401b540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32d5e5c4a668afbe server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:57.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.507+0000 7f684ffff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:05:57.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.508+0000 7f684ffff700 1 -- 172.21.15.70:0/850312907 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6864034070 con 0x556f9ee118e0 2024-06-04T05:05:57.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.508+0000 7f6881704b80 1 -- 172.21.15.70:0/850312907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f686402f6e0 con 0x556f9ee118e0 2024-06-04T05:05:57.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.508+0000 7f684ffff700 1 -- 172.21.15.70:0/850312907 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f686402f890 con 0x556f9ee118e0 2024-06-04T05:05:57.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.508+0000 7f684ffff700 1 -- 172.21.15.70:0/850312907 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f686402fa90 con 0x556f9ee118e0 2024-06-04T05:05:57.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.508+0000 7f6881704b80 10 client.26993 fetch_fsmap learned FSMap version 32 2024-06-04T05:05:57.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.508+0000 7f6881704b80 1 -- 172.21.15.70:0/850312907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556f9ee97170 con 0x556f9ee118e0 2024-06-04T05:05:57.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.508+0000 7f684ffff700 1 -- 172.21.15.70:0/850312907 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f686402f6e0 con 0x556f9ee118e0 2024-06-04T05:05:57.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.508+0000 7f684ffff700 1 -- 172.21.15.70:0/850312907 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f6864004870 con 0x556f9ee118e0 2024-06-04T05:05:57.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.509+0000 7f6881704b80 10 client.26993 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:05:57.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.509+0000 7f6881704b80 10 client.26993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:05:57.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.509+0000 7f6881704b80 20 client.26993 populate_metadata read hostname 'smithi070' 2024-06-04T05:05:57.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.509+0000 7f6881704b80 1 --2- 172.21.15.70:0/850312907 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556f9ee97890 0x556f9ee99cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:57.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.509+0000 7f6881704b80 4 client.26993 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:05:57.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.509+0000 7f6881704b80 1 -- 172.21.15.70:0/850312907 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556f9ed72c60 con 0x556f9ee97890 2024-06-04T05:05:57.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.509+0000 7f686ffff700 1 --2- 172.21.15.70:0/850312907 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556f9ee97890 0x556f9ee99cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:57.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.509+0000 7f684ffff700 10 client.26993 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:57.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.509+0000 7f686ffff700 1 --2- 172.21.15.70:0/850312907 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556f9ee97890 0x556f9ee99cb0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef6f27ff979d1617 server_cookie=f928a0714fdc4705 in_seq=0 out_seq=0 2024-06-04T05:05:57.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.510+0000 7f684ffff700 1 -- 172.21.15.70:0/850312907 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556f9ed72c60 con 0x556f9ee97890 2024-06-04T05:05:57.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.510+0000 7f684ffff700 10 client.26993 handle_command_reply: tid=0 2024-06-04T05:05:57.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.510+0000 7f6881704b80 1 client.26993 shutdown 2024-06-04T05:05:57.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.511+0000 7f6881704b80 1 -- 172.21.15.70:0/850312907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee118e0 msgr2=0x556f9ee22520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:57.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.511+0000 7f6881704b80 1 --2- 172.21.15.70:0/850312907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee118e0 0x556f9ee22520 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x556f9ee2dae0 tx=0x7f686401b540 comp rx=0 tx=0).stop 2024-06-04T05:05:57.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.511+0000 7f6881704b80 1 -- 172.21.15.70:0/850312907 shutdown_connections 2024-06-04T05:05:57.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.511+0000 7f6881704b80 1 --2- 172.21.15.70:0/850312907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f9ee15fe0 0x556f9ee95a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.511+0000 7f6881704b80 1 --2- 172.21.15.70:0/850312907 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556f9ee97890 0x556f9ee99cb0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.511+0000 7f6881704b80 1 --2- 172.21.15.70:0/850312907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f9ee118e0 0x556f9ee22520 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x556f9ee2dae0 tx=0x7f686401b540 comp rx=0 tx=0).stop 2024-06-04T05:05:57.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.511+0000 7f6881704b80 1 --2- 172.21.15.70:0/850312907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f9ee12290 0x556f9ee22a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.511+0000 7f684ffff700 0 client.26993 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:05:57.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.511+0000 7f6881704b80 1 -- 172.21.15.70:0/850312907 >> 172.21.15.70:0/850312907 conn(0x556f9ed3f100 msgr2=0x556f9ee10f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:57.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.511+0000 7f6881704b80 1 -- 172.21.15.70:0/850312907 shutdown_connections 2024-06-04T05:05:57.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.511+0000 7f6881704b80 1 -- 172.21.15.70:0/850312907 wait complete. 2024-06-04T05:05:57.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.511+0000 7f6881704b80 20 client.26993 trim_cache size 0 max 16384 2024-06-04T05:05:57.516 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:05:57.516 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:05:57.516 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "a4b108d0-bc8c-492b-ae38-7d7ce99731ba", 2024-06-04T05:05:57.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:05:57.516 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:05:57.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.512+0000 7f6876e6f700 1 -- 172.21.15.70:0/4203376607 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68600b2a70 msgr2=0x7f68600b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:57.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.512+0000 7f6876e6f700 1 --2- 172.21.15.70:0/4203376607 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68600b2a70 0x7f68600b4f30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f686400ffb0 tx=0x7f686400f7a0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.512+0000 7f6876e6f700 1 -- 172.21.15.70:0/4203376607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6870108520 msgr2=0x7f68701987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:57.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.512+0000 7f6876e6f700 1 --2- 172.21.15.70:0/4203376607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6870108520 0x7f68701987c0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f686800b0a0 tx=0x7f6868004ce0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.512+0000 7f6876e6f700 1 -- 172.21.15.70:0/4203376607 shutdown_connections 2024-06-04T05:05:57.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.512+0000 7f6876e6f700 1 --2- 172.21.15.70:0/4203376607 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68600b2a70 0x7f68600b4f30 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.512+0000 7f6876e6f700 1 --2- 172.21.15.70:0/4203376607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6870108520 0x7f68701987c0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.512+0000 7f6876e6f700 1 --2- 172.21.15.70:0/4203376607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6870102440 0x7f687019e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.512+0000 7f6876e6f700 1 --2- 172.21.15.70:0/4203376607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6870102e00 0x7f687019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:57.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.512+0000 7f6876e6f700 1 -- 172.21.15.70:0/4203376607 >> 172.21.15.70:0/4203376607 conn(0x7f68700793f0 msgr2=0x7f68701005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:57.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.512+0000 7f6876e6f700 1 -- 172.21.15.70:0/4203376607 shutdown_connections 2024-06-04T05:05:57.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:57.512+0000 7f6876e6f700 1 -- 172.21.15.70:0/4203376607 wait complete. 2024-06-04T05:05:57.728 INFO:tasks.workunit.client.0.smithi070.stdout: 1 151046 1.96 MB/sec execute 290 sec latency 28.271 ms 2024-06-04T05:05:58.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.065+0000 7f47eefc1700 1 -- 172.21.15.70:0/3954660737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47e81084c0 msgr2=0x7f47e81088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:58.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.065+0000 7f47eefc1700 1 --2- 172.21.15.70:0/3954660737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47e81084c0 0x7f47e81088a0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f47dc009990 tx=0x7f47dc009ca0 comp rx=0 tx=0).stop 2024-06-04T05:05:58.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.065+0000 7f47eefc1700 1 -- 172.21.15.70:0/3954660737 shutdown_connections 2024-06-04T05:05:58.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.065+0000 7f47eefc1700 1 --2- 172.21.15.70:0/3954660737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e8103600 0x7f47e810f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:58.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.065+0000 7f47eefc1700 1 --2- 172.21.15.70:0/3954660737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47e81084c0 0x7f47e81088a0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:58.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.065+0000 7f47eefc1700 1 --2- 172.21.15.70:0/3954660737 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47e8102c40 0x7f47e81030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:58.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.065+0000 7f47eefc1700 1 -- 172.21.15.70:0/3954660737 >> 172.21.15.70:0/3954660737 conn(0x7f47e8078ab0 msgr2=0x7f47e80ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:58.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.065+0000 7f47eefc1700 1 -- 172.21.15.70:0/3954660737 shutdown_connections 2024-06-04T05:05:58.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.066+0000 7f47eefc1700 1 -- 172.21.15.70:0/3954660737 wait complete. 2024-06-04T05:05:58.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.066+0000 7f47eefc1700 1 Processor -- start 2024-06-04T05:05:58.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.066+0000 7f47eefc1700 1 -- start start 2024-06-04T05:05:58.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.066+0000 7f47eefc1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47e8102c40 0x7f47e819f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:58.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.066+0000 7f47eefc1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47e8103600 0x7f47e819fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:58.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.066+0000 7f47eefc1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e81084c0 0x7f47e8199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:58.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.066+0000 7f47eefc1700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f47e81124d0 con 0x7f47e8103600 2024-06-04T05:05:58.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.066+0000 7f47eefc1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47e8112350 con 0x7f47e81084c0 2024-06-04T05:05:58.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.066+0000 7f47eefc1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f47e8112650 con 0x7f47e8102c40 2024-06-04T05:05:58.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.067+0000 7f47e7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47e8103600 0x7f47e819fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:58.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.067+0000 7f47e7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47e8103600 0x7f47e819fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53164/0 (socket says 172.21.15.70:53164) 2024-06-04T05:05:58.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.067+0000 7f47ecd5d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47e8102c40 0x7f47e819f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:58.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.067+0000 7f47ed55e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e81084c0 0x7f47e8199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:58.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.067+0000 7f47e7fff700 1 -- 172.21.15.70:0/709767551 learned_addr learned my addr 172.21.15.70:0/709767551 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:05:58.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.067+0000 7f47e7fff700 1 -- 172.21.15.70:0/709767551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47e8102c40 msgr2=0x7f47e819f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:58.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.067+0000 7f47e7fff700 1 --2- 172.21.15.70:0/709767551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47e8102c40 0x7f47e819f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:58.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.067+0000 7f47e7fff700 1 -- 172.21.15.70:0/709767551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e81084c0 msgr2=0x7f47e8199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:58.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.067+0000 7f47e7fff700 1 --2- 172.21.15.70:0/709767551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e81084c0 0x7f47e8199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:58.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.067+0000 7f47e7fff700 1 -- 172.21.15.70:0/709767551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e819a0c0 con 0x7f47e8103600 2024-06-04T05:05:58.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.067+0000 7f47e7fff700 1 --2- 172.21.15.70:0/709767551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47e8103600 0x7f47e819fce0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f47d400e9a0 tx=0x7f47d400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:58.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.067+0000 7f47e5ffb700 1 -- 172.21.15.70:0/709767551 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d400ca10 con 0x7f47e8103600 2024-06-04T05:05:58.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.067+0000 7f47eefc1700 1 -- 172.21.15.70:0/709767551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47e819a3b0 con 0x7f47e8103600 2024-06-04T05:05:58.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.067+0000 7f47e5ffb700 1 -- 172.21.15.70:0/709767551 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47d400cbb0 con 0x7f47e8103600 2024-06-04T05:05:58.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.067+0000 7f47e5ffb700 1 -- 172.21.15.70:0/709767551 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d4018680 con 0x7f47e8103600 2024-06-04T05:05:58.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.067+0000 7f47eefc1700 1 -- 172.21.15.70:0/709767551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47e81a6670 con 0x7f47e8103600 2024-06-04T05:05:58.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.068+0000 7f47eefc1700 1 -- 172.21.15.70:0/709767551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47e8103c50 con 0x7f47e8103600 2024-06-04T05:05:58.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.070+0000 7f47e5ffb700 1 -- 172.21.15.70:0/709767551 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f47d4010430 con 0x7f47e8103600 2024-06-04T05:05:58.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.071+0000 7f47e5ffb700 1 --2- 172.21.15.70:0/709767551 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f47d80b29f0 0x7f47d80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:05:58.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.071+0000 7f47ecd5d700 1 --2- 172.21.15.70:0/709767551 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f47d80b29f0 0x7f47d80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:05:58.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.071+0000 7f47e5ffb700 1 -- 172.21.15.70:0/709767551 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7f47d40e2bf0 con 0x7f47e8103600 2024-06-04T05:05:58.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.072+0000 7f47ecd5d700 1 --2- 172.21.15.70:0/709767551 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f47d80b29f0 0x7f47d80b4eb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00b650 tx=0x7f47dc01a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:05:58.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.076+0000 7f47e5ffb700 1 -- 172.21.15.70:0/709767551 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f47d40aa5a0 con 0x7f47e8103600 2024-06-04T05:05:58.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:58 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:05:58.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:58 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:05:58.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:58 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:05:58.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:58 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:05:58.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:58 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:05:58.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:58 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:05:58.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:58 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:05:58.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:58 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:05:58.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.380+0000 7f47eefc1700 1 -- 172.21.15.70:0/709767551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47e806a230 con 0x7f47e8103600 2024-06-04T05:05:58.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.380+0000 7f47e5ffb700 1 -- 172.21.15.70:0/709767551 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11929 (secure 0 0 0) 0x7f47e806a230 con 0x7f47e8103600 2024-06-04T05:05:58.383 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:05:58.383 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:01:52.807483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:05:58.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.383+0000 7f47eefc1700 1 -- 172.21.15.70:0/709767551 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f47d80b29f0 msgr2=0x7f47d80b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:58.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.383+0000 7f47eefc1700 1 --2- 172.21.15.70:0/709767551 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f47d80b29f0 0x7f47d80b4eb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00b650 tx=0x7f47dc01a040 comp rx=0 tx=0).stop 2024-06-04T05:05:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.383+0000 7f47eefc1700 1 -- 172.21.15.70:0/709767551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47e8103600 msgr2=0x7f47e819fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:05:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.383+0000 7f47eefc1700 1 --2- 172.21.15.70:0/709767551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47e8103600 0x7f47e819fce0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f47d400e9a0 tx=0x7f47d400ecb0 comp rx=0 tx=0).stop 2024-06-04T05:05:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.383+0000 7f47eefc1700 1 -- 172.21.15.70:0/709767551 shutdown_connections 2024-06-04T05:05:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.383+0000 7f47eefc1700 1 --2- 172.21.15.70:0/709767551 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f47d80b29f0 0x7f47d80b4eb0 secure :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00b650 tx=0x7f47dc01a040 comp rx=0 tx=0).stop 2024-06-04T05:05:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.383+0000 7f47eefc1700 1 --2- 172.21.15.70:0/709767551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e81084c0 0x7f47e8199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.383+0000 7f47eefc1700 1 --2- 172.21.15.70:0/709767551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47e8102c40 0x7f47e819f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.383+0000 7f47eefc1700 1 --2- 172.21.15.70:0/709767551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47e8103600 0x7f47e819fce0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:05:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.383+0000 7f47eefc1700 1 -- 172.21.15.70:0/709767551 >> 172.21.15.70:0/709767551 conn(0x7f47e8078ab0 msgr2=0x7f47e8100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:05:58.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.384+0000 7f47eefc1700 1 -- 172.21.15.70:0/709767551 shutdown_connections 2024-06-04T05:05:58.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:05:58.385+0000 7f47eefc1700 1 -- 172.21.15.70:0/709767551 wait complete. 2024-06-04T05:05:58.388 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 32 2024-06-04T05:05:58.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:58 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:05:58.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:58 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:05:58.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:58 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:05:58.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:58 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:05:58.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:58 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:05:58.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:58 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:05:58.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:58 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:05:58.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:58 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:05:58.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:58 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:05:58.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:58 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:05:58.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:58 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:05:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:58 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:05:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:58 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:05:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:58 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:05:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:58 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:05:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:58 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:05:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:05:58.729 INFO:tasks.workunit.client.0.smithi070.stdout: 1 151501 1.96 MB/sec execute 291 sec latency 63.931 ms 2024-06-04T05:05:58.738 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:05:59.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:59 smithi070 ceph-mon[32631]: pgmap v269: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 282 KiB/s wr, 34 op/s 2024-06-04T05:05:59.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:05:59 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/709767551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:05:59.485 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 143.6 --await-for 45.0' 2024-06-04T05:05:59.485 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 143.6 --await-for 45.0 2024-06-04T05:05:59.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:59 smithi098 ceph-mon[35569]: pgmap v269: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 282 KiB/s wr, 34 op/s 2024-06-04T05:05:59.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:05:59 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/709767551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:05:59.547 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:05:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:59 smithi112 ceph-mon[36453]: pgmap v269: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 282 KiB/s wr, 34 op/s 2024-06-04T05:05:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:05:59 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/709767551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:05:59.729 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152275 1.97 MB/sec execute 292 sec latency 53.152 ms 2024-06-04T05:06:00.729 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152365 1.97 MB/sec execute 293 sec latency 471.301 ms 2024-06-04T05:06:00.823 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:06:01.314 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:01 smithi070 ceph-mon[32631]: pgmap v270: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 282 KiB/s wr, 34 op/s 2024-06-04T05:06:01.487 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:01 smithi112 ceph-mon[36453]: pgmap v270: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 282 KiB/s wr, 34 op/s 2024-06-04T05:06:01.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:01 smithi098 ceph-mon[35569]: pgmap v270: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 282 KiB/s wr, 34 op/s 2024-06-04T05:06:01.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.493+0000 7fc0e5ebe700 1 -- 172.21.15.70:0/3874493637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0e0102440 msgr2=0x7fc0e01028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:01.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.493+0000 7fc0e5ebe700 1 --2- 172.21.15.70:0/3874493637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0e0102440 0x7fc0e01028c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009990 tx=0x7fc0d4009ca0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.494+0000 7fc0e5ebe700 1 -- 172.21.15.70:0/3874493637 shutdown_connections 2024-06-04T05:06:01.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.494+0000 7fc0e5ebe700 1 --2- 172.21.15.70:0/3874493637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0e0102e00 0x7fc0e010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.494+0000 7fc0e5ebe700 1 --2- 172.21.15.70:0/3874493637 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0e0108520 0x7fc0e0108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.494+0000 7fc0e5ebe700 1 --2- 172.21.15.70:0/3874493637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0e0102440 0x7fc0e01028c0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.494+0000 7fc0e5ebe700 1 -- 172.21.15.70:0/3874493637 >> 172.21.15.70:0/3874493637 conn(0x7fc0e00793f0 msgr2=0x7fc0e0079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:01.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.494+0000 7fc0e5ebe700 1 -- 172.21.15.70:0/3874493637 shutdown_connections 2024-06-04T05:06:01.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.494+0000 7fc0e5ebe700 1 -- 172.21.15.70:0/3874493637 wait complete. 2024-06-04T05:06:01.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.494+0000 7fc0e5ebe700 1 Processor -- start 2024-06-04T05:06:01.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.494+0000 7fc0e5ebe700 1 -- start start 2024-06-04T05:06:01.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0e5ebe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0e0102440 0x7fc0e01988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0e5ebe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0e0102e00 0x7fc0e0198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0e5ebe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0e0108520 0x7fc0e0199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0e5ebe700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e0112580 con 0x7fc0e0102e00 2024-06-04T05:06:01.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0e5ebe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e0112400 con 0x7fc0e0108520 2024-06-04T05:06:01.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0e5ebe700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e0112700 con 0x7fc0e0102440 2024-06-04T05:06:01.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0deffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0e0102e00 0x7fc0e0198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0deffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0e0102e00 0x7fc0e0198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53192/0 (socket says 172.21.15.70:53192) 2024-06-04T05:06:01.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0deffd700 1 -- 172.21.15.70:0/3422898409 learned_addr learned my addr 172.21.15.70:0/3422898409 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:01.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0df7fe700 1 --2- 172.21.15.70:0/3422898409 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0e0102440 0x7fc0e01988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0dffff700 1 --2- 172.21.15.70:0/3422898409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0e0108520 0x7fc0e0199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0deffd700 1 -- 172.21.15.70:0/3422898409 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0e0102440 msgr2=0x7fc0e01988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:01.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0deffd700 1 --2- 172.21.15.70:0/3422898409 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0e0102440 0x7fc0e01988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0deffd700 1 -- 172.21.15.70:0/3422898409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0e0108520 msgr2=0x7fc0e0199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:01.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0deffd700 1 --2- 172.21.15.70:0/3422898409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0e0108520 0x7fc0e0199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0deffd700 1 -- 172.21.15.70:0/3422898409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e01a8120 con 0x7fc0e0102e00 2024-06-04T05:06:01.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0deffd700 1 --2- 172.21.15.70:0/3422898409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0e0102e00 0x7fc0e0198df0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4000c00 tx=0x7fc0d40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:01.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0dcff9700 1 -- 172.21.15.70:0/3422898409 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4027070 con 0x7fc0e0102e00 2024-06-04T05:06:01.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0dcff9700 1 -- 172.21.15.70:0/3422898409 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc0d400fcc0 con 0x7fc0e0102e00 2024-06-04T05:06:01.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0e5ebe700 1 -- 172.21.15.70:0/3422898409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e01a83b0 con 0x7fc0e0102e00 2024-06-04T05:06:01.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.495+0000 7fc0dcff9700 1 -- 172.21.15.70:0/3422898409 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4022d70 con 0x7fc0e0102e00 2024-06-04T05:06:01.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.496+0000 7fc0e5ebe700 1 -- 172.21.15.70:0/3422898409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e01a8860 con 0x7fc0e0102e00 2024-06-04T05:06:01.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.498+0000 7fc0dcff9700 1 -- 172.21.15.70:0/3422898409 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fc0d402a070 con 0x7fc0e0102e00 2024-06-04T05:06:01.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.499+0000 7fc0dcff9700 1 --2- 172.21.15.70:0/3422898409 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc0c80b2a70 0x7fc0c80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.499+0000 7fc0df7fe700 1 --2- 172.21.15.70:0/3422898409 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc0c80b2a70 0x7fc0c80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.499+0000 7fc0dcff9700 1 -- 172.21.15.70:0/3422898409 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fc0d40e3570 con 0x7fc0e0102e00 2024-06-04T05:06:01.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.499+0000 7fc0f0753b80 1 Processor -- start 2024-06-04T05:06:01.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.499+0000 7fc0f0753b80 1 -- start start 2024-06-04T05:06:01.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.499+0000 7fc0df7fe700 1 --2- 172.21.15.70:0/3422898409 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc0c80b2a70 0x7fc0c80b4f30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0009a10 tx=0x7fc0d0009190 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:01.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.499+0000 7fc0f0753b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab0615f860 0x55ab0615fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.499+0000 7fc0f0753b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06160210 0x55ab06163940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.499+0000 7fc0f0753b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06163f60 0x55ab061643a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.499+0000 7fc0f0753b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ab060bec90 con 0x55ab06160210 2024-06-04T05:06:01.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.499+0000 7fc0f0753b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ab060ae830 con 0x55ab06163f60 2024-06-04T05:06:01.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.499+0000 7fc0f0753b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ab060b2870 con 0x55ab0615f860 2024-06-04T05:06:01.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.500+0000 7fc0deffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06160210 0x55ab06163940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.500+0000 7fc0deffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06160210 0x55ab06163940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53200/0 (socket says 172.21.15.70:53200) 2024-06-04T05:06:01.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.500+0000 7fc0deffd700 1 -- 172.21.15.70:0/862557321 learned_addr learned my addr 172.21.15.70:0/862557321 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:01.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.500+0000 7fc0df7fe700 1 --2- 172.21.15.70:0/862557321 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab0615f860 0x55ab0615fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.500+0000 7fc0dffff700 1 --2- 172.21.15.70:0/862557321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06163f60 0x55ab061643a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.500+0000 7fc0deffd700 1 -- 172.21.15.70:0/862557321 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab0615f860 msgr2=0x55ab0615fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:01.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.500+0000 7fc0deffd700 1 --2- 172.21.15.70:0/862557321 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab0615f860 0x55ab0615fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.500+0000 7fc0deffd700 1 -- 172.21.15.70:0/862557321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06163f60 msgr2=0x55ab061643a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:01.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.500+0000 7fc0deffd700 1 --2- 172.21.15.70:0/862557321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06163f60 0x55ab061643a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.500+0000 7fc0deffd700 1 -- 172.21.15.70:0/862557321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab05ffe8d0 con 0x55ab06160210 2024-06-04T05:06:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.500+0000 7fc0dffff700 1 --2- 172.21.15.70:0/862557321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06163f60 0x55ab061643a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:06:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.500+0000 7fc0deffd700 1 --2- 172.21.15.70:0/862557321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06160210 0x55ab06163940 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fc0d40a6930 tx=0x7fc0d40aa780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d81de4fb44d4515f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.500+0000 7fc0c5ffb700 1 -- 172.21.15.70:0/862557321 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d40aa3c0 con 0x55ab06160210 2024-06-04T05:06:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.500+0000 7fc0c5ffb700 1 -- 172.21.15.70:0/862557321 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc0d40aafb0 con 0x55ab06160210 2024-06-04T05:06:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.500+0000 7fc0c5ffb700 1 -- 172.21.15.70:0/862557321 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4022500 con 0x55ab06160210 2024-06-04T05:06:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.501+0000 7fc0f0753b80 1 -- 172.21.15.70:0/862557321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06160210 msgr2=0x55ab06163940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.501+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/862557321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06160210 0x55ab06163940 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fc0d40a6930 tx=0x7fc0d40aa780 comp rx=0 tx=0).stop 2024-06-04T05:06:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.501+0000 7fc0f0753b80 1 -- 172.21.15.70:0/862557321 shutdown_connections 2024-06-04T05:06:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.501+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/862557321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06163f60 0x55ab061643a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.501+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/862557321 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab0615f860 0x55ab0615fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.501+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/862557321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06160210 0x55ab06163940 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.501+0000 7fc0f0753b80 1 -- 172.21.15.70:0/862557321 >> 172.21.15.70:0/862557321 conn(0x55ab0608d080 msgr2=0x55ab06159e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.501+0000 7fc0f0753b80 1 -- 172.21.15.70:0/862557321 shutdown_connections 2024-06-04T05:06:01.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.501+0000 7fc0f0753b80 1 -- 172.21.15.70:0/862557321 wait complete. 2024-06-04T05:06:01.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.501+0000 7fc0f0753b80 1 -- start start 2024-06-04T05:06:01.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.502+0000 7fc0f0753b80 1 Processor -- start 2024-06-04T05:06:01.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.502+0000 7fc0f0753b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab0615f860 0x55ab0616b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.502+0000 7fc0f0753b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06160210 0x55ab06168720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.502+0000 7fc0f0753b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06163f60 0x55ab06168c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.502+0000 7fc0f0753b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ab05fb4fa0 con 0x55ab06163f60 2024-06-04T05:06:01.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.502+0000 7fc0f0753b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ab060507c0 con 0x55ab06160210 2024-06-04T05:06:01.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.502+0000 7fc0f0753b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ab0616bee0 con 0x55ab0615f860 2024-06-04T05:06:01.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.502+0000 7fc0f0753b80 10 client.0 _ll_register_callbacks cb 0x55ab06049920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:01.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.502+0000 7fc0dffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06163f60 0x55ab06168c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.502+0000 7fc0dffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06163f60 0x55ab06168c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53202/0 (socket says 172.21.15.70:53202) 2024-06-04T05:06:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.502+0000 7fc0dffff700 1 -- 172.21.15.70:0/2435742873 learned_addr learned my addr 172.21.15.70:0/2435742873 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.502+0000 7fc0df7fe700 1 --2- 172.21.15.70:0/2435742873 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab0615f860 0x55ab0616b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.502+0000 7fc0deffd700 1 --2- 172.21.15.70:0/2435742873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06160210 0x55ab06168720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.502+0000 7fc0dffff700 1 -- 172.21.15.70:0/2435742873 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab0615f860 msgr2=0x55ab0616b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.502+0000 7fc0dffff700 1 --2- 172.21.15.70:0/2435742873 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab0615f860 0x55ab0616b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.503+0000 7fc0dffff700 1 -- 172.21.15.70:0/2435742873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06160210 msgr2=0x55ab06168720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.503+0000 7fc0dffff700 1 --2- 172.21.15.70:0/2435742873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06160210 0x55ab06168720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.503+0000 7fc0dffff700 1 -- 172.21.15.70:0/2435742873 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab0604d690 con 0x55ab06163f60 2024-06-04T05:06:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.503+0000 7fc0df7fe700 1 --2- 172.21.15.70:0/2435742873 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab0615f860 0x55ab0616b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:06:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.503+0000 7fc0dffff700 1 --2- 172.21.15.70:0/2435742873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06163f60 0x55ab06168c60 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fc0d800abd0 tx=0x7fc0d800af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa48ce2c7a8ed8a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.503+0000 7fc0beffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.503+0000 7fc0beffd700 1 -- 172.21.15.70:0/2435742873 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8007a20 con 0x55ab06163f60 2024-06-04T05:06:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.503+0000 7fc0beffd700 1 -- 172.21.15.70:0/2435742873 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fc0d8007bc0 con 0x55ab06163f60 2024-06-04T05:06:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.503+0000 7fc0f0753b80 1 -- 172.21.15.70:0/2435742873 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0d40aafb0 con 0x55ab06163f60 2024-06-04T05:06:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.503+0000 7fc0beffd700 1 -- 172.21.15.70:0/2435742873 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc0d8007dc0 con 0x55ab06163f60 2024-06-04T05:06:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.503+0000 7fc0f0753b80 10 client.17844 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.503+0000 7fc0f0753b80 1 -- 172.21.15.70:0/2435742873 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ab06176520 con 0x55ab06163f60 2024-06-04T05:06:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.503+0000 7fc0beffd700 1 -- 172.21.15.70:0/2435742873 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc0d801a070 con 0x55ab06163f60 2024-06-04T05:06:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.503+0000 7fc0beffd700 1 -- 172.21.15.70:0/2435742873 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fc0d801e440 con 0x55ab06163f60 2024-06-04T05:06:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.503+0000 7fc0f0753b80 10 client.17844 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.503+0000 7fc0f0753b80 10 client.17844 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:06:01.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.503+0000 7fc0f0753b80 20 client.17844 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:01.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.504+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/2435742873 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ab06178b60 0x55ab0617af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.504+0000 7fc0f0753b80 4 client.17844 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:06:01.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.504+0000 7fc0f0753b80 1 -- 172.21.15.70:0/2435742873 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc0d40aafb0 con 0x55ab06178b60 2024-06-04T05:06:01.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.504+0000 7fc0df7fe700 1 --2- 172.21.15.70:0/2435742873 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ab06178b60 0x55ab0617af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.504+0000 7fc0df7fe700 1 --2- 172.21.15.70:0/2435742873 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ab06178b60 0x55ab0617af90 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee726c6f7a942d74 server_cookie=a8dbe96bd80635f8 in_seq=0 out_seq=0 2024-06-04T05:06:01.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.504+0000 7fc0beffd700 10 client.17844 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:01.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.505+0000 7fc0beffd700 1 -- 172.21.15.70:0/2435742873 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fc0d40aafb0 con 0x55ab06178b60 2024-06-04T05:06:01.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.505+0000 7fc0beffd700 10 client.17844 handle_command_reply: tid=0 2024-06-04T05:06:01.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.505+0000 7fc0f0753b80 1 client.17844 shutdown 2024-06-04T05:06:01.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.505+0000 7fc0f0753b80 1 -- 172.21.15.70:0/2435742873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06163f60 msgr2=0x55ab06168c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:01.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.505+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/2435742873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06163f60 0x55ab06168c60 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fc0d800abd0 tx=0x7fc0d800af90 comp rx=0 tx=0).stop 2024-06-04T05:06:01.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.505+0000 7fc0f0753b80 1 -- 172.21.15.70:0/2435742873 shutdown_connections 2024-06-04T05:06:01.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.505+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/2435742873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06163f60 0x55ab06168c60 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.505+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/2435742873 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab0615f860 0x55ab0616b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.505+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/2435742873 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ab06178b60 0x55ab0617af90 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.505+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/2435742873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06160210 0x55ab06168720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.505+0000 7fc0f0753b80 1 -- 172.21.15.70:0/2435742873 >> 172.21.15.70:0/2435742873 conn(0x55ab0608d080 msgr2=0x55ab06159510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:01.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.505+0000 7fc0beffd700 0 client.17844 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:01.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.505+0000 7fc0f0753b80 1 -- 172.21.15.70:0/2435742873 shutdown_connections 2024-06-04T05:06:01.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.505+0000 7fc0f0753b80 1 -- 172.21.15.70:0/2435742873 wait complete. 2024-06-04T05:06:01.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.506+0000 7fc0f0753b80 20 client.17844 trim_cache size 0 max 16384 2024-06-04T05:06:01.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.526+0000 7fc0f0753b80 1 Processor -- start 2024-06-04T05:06:01.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.526+0000 7fc0f0753b80 1 -- start start 2024-06-04T05:06:01.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.526+0000 7fc0f0753b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab0615f860 0x55ab06162fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.526+0000 7fc0f0753b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab06160210 0x55ab06159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.526+0000 7fc0f0753b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06163f60 0x55ab06180860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.526+0000 7fc0f0753b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01b0020 con 0x55ab0615f860 2024-06-04T05:06:01.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.526+0000 7fc0f0753b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01b1b60 con 0x55ab06163f60 2024-06-04T05:06:01.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.526+0000 7fc0f0753b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01b06f0 con 0x55ab06160210 2024-06-04T05:06:01.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.527+0000 7fc0df7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab0615f860 0x55ab06162fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.527+0000 7fc0df7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab0615f860 0x55ab06162fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53216/0 (socket says 172.21.15.70:53216) 2024-06-04T05:06:01.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.527+0000 7fc0df7fe700 1 -- 172.21.15.70:0/1629824965 learned_addr learned my addr 172.21.15.70:0/1629824965 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:01.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.527+0000 7fc0deffd700 1 --2- 172.21.15.70:0/1629824965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab06160210 0x55ab06159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.527+0000 7fc0dffff700 1 --2- 172.21.15.70:0/1629824965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06163f60 0x55ab06180860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.527+0000 7fc0df7fe700 1 -- 172.21.15.70:0/1629824965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab06160210 msgr2=0x55ab06159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:01.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.527+0000 7fc0df7fe700 1 --2- 172.21.15.70:0/1629824965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab06160210 0x55ab06159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.527+0000 7fc0df7fe700 1 -- 172.21.15.70:0/1629824965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06163f60 msgr2=0x55ab06180860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:01.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.527+0000 7fc0df7fe700 1 --2- 172.21.15.70:0/1629824965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06163f60 0x55ab06180860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.527+0000 7fc0df7fe700 1 -- 172.21.15.70:0/1629824965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab0615fe90 con 0x55ab0615f860 2024-06-04T05:06:01.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.527+0000 7fc0df7fe700 1 --2- 172.21.15.70:0/1629824965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab0615f860 0x55ab06162fc0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x55ab0617b800 tx=0x7fc0d0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe045465a487269 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:01.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.527+0000 7fc0c57fa700 1 -- 172.21.15.70:0/1629824965 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d002d070 con 0x55ab0615f860 2024-06-04T05:06:01.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.527+0000 7fc0c57fa700 1 -- 172.21.15.70:0/1629824965 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc0d0015990 con 0x55ab0615f860 2024-06-04T05:06:01.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.527+0000 7fc0c57fa700 1 -- 172.21.15.70:0/1629824965 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d001b040 con 0x55ab0615f860 2024-06-04T05:06:01.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.528+0000 7fc0f0753b80 1 -- 172.21.15.70:0/1629824965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab0615f860 msgr2=0x55ab06162fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:01.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.528+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/1629824965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab0615f860 0x55ab06162fc0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x55ab0617b800 tx=0x7fc0d0013000 comp rx=0 tx=0).stop 2024-06-04T05:06:01.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.528+0000 7fc0f0753b80 1 -- 172.21.15.70:0/1629824965 shutdown_connections 2024-06-04T05:06:01.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.528+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/1629824965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06163f60 0x55ab06180860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.528+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/1629824965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab0615f860 0x55ab06162fc0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.528+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/1629824965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab06160210 0x55ab06159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.528+0000 7fc0f0753b80 1 -- 172.21.15.70:0/1629824965 >> 172.21.15.70:0/1629824965 conn(0x55ab0608d080 msgr2=0x55ab061628b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:01.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.528+0000 7fc0f0753b80 1 -- 172.21.15.70:0/1629824965 shutdown_connections 2024-06-04T05:06:01.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.528+0000 7fc0f0753b80 1 -- 172.21.15.70:0/1629824965 wait complete. 2024-06-04T05:06:01.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.528+0000 7fc0f0753b80 1 -- start start 2024-06-04T05:06:01.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0f0753b80 1 Processor -- start 2024-06-04T05:06:01.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0f0753b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab0615f860 0x55ab0616f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0f0753b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06160210 0x55ab0616fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0f0753b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06163f60 0x55ab061e2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0f0753b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01b1490 con 0x55ab06160210 2024-06-04T05:06:01.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0f0753b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01b0dc0 con 0x55ab06163f60 2024-06-04T05:06:01.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0f0753b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01adfe0 con 0x55ab0615f860 2024-06-04T05:06:01.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0f0753b80 10 client.0 _ll_register_callbacks cb 0x55ab06049920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:01.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0deffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06160210 0x55ab0616fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0deffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06160210 0x55ab0616fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53220/0 (socket says 172.21.15.70:53220) 2024-06-04T05:06:01.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0deffd700 1 -- 172.21.15.70:0/1097571376 learned_addr learned my addr 172.21.15.70:0/1097571376 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:01.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0df7fe700 1 --2- 172.21.15.70:0/1097571376 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab0615f860 0x55ab0616f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0dffff700 1 --2- 172.21.15.70:0/1097571376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06163f60 0x55ab061e2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0deffd700 1 -- 172.21.15.70:0/1097571376 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab0615f860 msgr2=0x55ab0616f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:01.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0deffd700 1 --2- 172.21.15.70:0/1097571376 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab0615f860 0x55ab0616f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0deffd700 1 -- 172.21.15.70:0/1097571376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06163f60 msgr2=0x55ab061e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:01.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0deffd700 1 --2- 172.21.15.70:0/1097571376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06163f60 0x55ab061e2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0deffd700 1 -- 172.21.15.70:0/1097571376 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab061e2d80 con 0x55ab06160210 2024-06-04T05:06:01.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.529+0000 7fc0dffff700 1 --2- 172.21.15.70:0/1097571376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06163f60 0x55ab061e2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:06:01.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.530+0000 7fc0deffd700 1 --2- 172.21.15.70:0/1097571376 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06160210 0x55ab0616fae0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc0d40e0d40 tx=0x7fc0d4022470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a263508bc306315 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:01.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.530+0000 7fc0beffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:01.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.530+0000 7fc0beffd700 1 -- 172.21.15.70:0/1097571376 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d40aafb0 con 0x55ab06160210 2024-06-04T05:06:01.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.530+0000 7fc0beffd700 1 -- 172.21.15.70:0/1097571376 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fc0d40abbe0 con 0x55ab06160210 2024-06-04T05:06:01.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.530+0000 7fc0f0753b80 1 -- 172.21.15.70:0/1097571376 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0d0015990 con 0x55ab06160210 2024-06-04T05:06:01.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.530+0000 7fc0beffd700 1 -- 172.21.15.70:0/1097571376 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc0d40c7070 con 0x55ab06160210 2024-06-04T05:06:01.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.530+0000 7fc0f0753b80 10 client.17850 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:01.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.530+0000 7fc0f0753b80 1 -- 172.21.15.70:0/1097571376 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ab061e4070 con 0x55ab06160210 2024-06-04T05:06:01.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.530+0000 7fc0beffd700 1 -- 172.21.15.70:0/1097571376 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc0d0015990 con 0x55ab06160210 2024-06-04T05:06:01.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.530+0000 7fc0beffd700 1 -- 172.21.15.70:0/1097571376 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fc0d40aab10 con 0x55ab06160210 2024-06-04T05:06:01.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.530+0000 7fc0f0753b80 10 client.17850 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:01.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.530+0000 7fc0f0753b80 10 client.17850 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:06:01.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.530+0000 7fc0f0753b80 20 client.17850 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:01.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.530+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/1097571376 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ab061e4690 0x55ab061e6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:01.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.530+0000 7fc0f0753b80 4 client.17850 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:06:01.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.530+0000 7fc0f0753b80 1 -- 172.21.15.70:0/1097571376 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ab0612b9e0 con 0x55ab061e4690 2024-06-04T05:06:01.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.531+0000 7fc0df7fe700 1 --2- 172.21.15.70:0/1097571376 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ab061e4690 0x55ab061e6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:01.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.531+0000 7fc0df7fe700 1 --2- 172.21.15.70:0/1097571376 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ab061e4690 0x55ab061e6ab0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae08666f1adb28ca server_cookie=99d9296318f64a9e in_seq=0 out_seq=0 2024-06-04T05:06:01.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.531+0000 7fc0beffd700 10 client.17850 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:01.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.531+0000 7fc0beffd700 1 -- 172.21.15.70:0/1097571376 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ab0612b9e0 con 0x55ab061e4690 2024-06-04T05:06:01.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.531+0000 7fc0beffd700 10 client.17850 handle_command_reply: tid=0 2024-06-04T05:06:01.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.531+0000 7fc0f0753b80 1 client.17850 shutdown 2024-06-04T05:06:01.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.531+0000 7fc0f0753b80 1 -- 172.21.15.70:0/1097571376 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06160210 msgr2=0x55ab0616fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:01.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.531+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/1097571376 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06160210 0x55ab0616fae0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc0d40e0d40 tx=0x7fc0d4022470 comp rx=0 tx=0).stop 2024-06-04T05:06:01.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.531+0000 7fc0f0753b80 1 -- 172.21.15.70:0/1097571376 shutdown_connections 2024-06-04T05:06:01.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.531+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/1097571376 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ab061e4690 0x55ab061e6ab0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.531+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/1097571376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab06163f60 0x55ab061e2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.531+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/1097571376 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ab0615f860 0x55ab0616f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.531+0000 7fc0f0753b80 1 --2- 172.21.15.70:0/1097571376 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ab06160210 0x55ab0616fae0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.531+0000 7fc0f0753b80 1 -- 172.21.15.70:0/1097571376 >> 172.21.15.70:0/1097571376 conn(0x55ab0608d080 msgr2=0x55ab06163140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:01.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.531+0000 7fc0beffd700 0 client.17850 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:01.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.532+0000 7fc0f0753b80 1 -- 172.21.15.70:0/1097571376 shutdown_connections 2024-06-04T05:06:01.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.532+0000 7fc0f0753b80 1 -- 172.21.15.70:0/1097571376 wait complete. 2024-06-04T05:06:01.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.532+0000 7fc0f0753b80 20 client.17850 trim_cache size 0 max 16384 2024-06-04T05:06:01.542 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:06:01.543 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:06:01.543 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:06:01.543 INFO:teuthology.orchestra.run.smithi070.stdout: "a4b108d0-bc8c-492b-ae38-7d7ce99731ba": { 2024-06-04T05:06:01.543 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:06:01.543 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "a4b108d0-bc8c-492b-ae38-7d7ce99731ba", 2024-06-04T05:06:01.543 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:06:01.544 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:06:01.544 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:06:01.544 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:06:01.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.532+0000 7fc0e5ebe700 1 -- 172.21.15.70:0/3422898409 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc0c80b2a70 msgr2=0x7fc0c80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:01.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.532+0000 7fc0e5ebe700 1 --2- 172.21.15.70:0/3422898409 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc0c80b2a70 0x7fc0c80b4f30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0009a10 tx=0x7fc0d0009190 comp rx=0 tx=0).stop 2024-06-04T05:06:01.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.532+0000 7fc0e5ebe700 1 -- 172.21.15.70:0/3422898409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0e0102e00 msgr2=0x7fc0e0198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:01.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.532+0000 7fc0e5ebe700 1 --2- 172.21.15.70:0/3422898409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0e0102e00 0x7fc0e0198df0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4000c00 tx=0x7fc0d40047b0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.532+0000 7fc0e5ebe700 1 -- 172.21.15.70:0/3422898409 shutdown_connections 2024-06-04T05:06:01.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.532+0000 7fc0e5ebe700 1 --2- 172.21.15.70:0/3422898409 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc0c80b2a70 0x7fc0c80b4f30 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.532+0000 7fc0e5ebe700 1 --2- 172.21.15.70:0/3422898409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0e0108520 0x7fc0e0199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.532+0000 7fc0e5ebe700 1 --2- 172.21.15.70:0/3422898409 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0e0102440 0x7fc0e01988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.532+0000 7fc0e5ebe700 1 --2- 172.21.15.70:0/3422898409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0e0102e00 0x7fc0e0198df0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:01.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.532+0000 7fc0e5ebe700 1 -- 172.21.15.70:0/3422898409 >> 172.21.15.70:0/3422898409 conn(0x7fc0e00793f0 msgr2=0x7fc0e0100610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:01.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.532+0000 7fc0e5ebe700 1 -- 172.21.15.70:0/3422898409 shutdown_connections 2024-06-04T05:06:01.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:01.532+0000 7fc0e5ebe700 1 -- 172.21.15.70:0/3422898409 wait complete. 2024-06-04T05:06:01.729 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152365 1.96 MB/sec execute 294 sec latency 1471.418 ms 2024-06-04T05:06:02.729 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152365 1.95 MB/sec execute 295 sec latency 2471.496 ms 2024-06-04T05:06:03.288 INFO:tasks.cephfs.filesystem:scrub status for tag:a4b108d0-bc8c-492b-ae38-7d7ce99731ba - {'path': '/', 'tag': 'a4b108d0-bc8c-492b-ae38-7d7ce99731ba', 'options': 'recursive,force'} 2024-06-04T05:06:03.447 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:03 smithi070 ceph-mon[32631]: pgmap v271: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 853 KiB/s wr, 93 op/s 2024-06-04T05:06:03.448 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:03 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:06:03.448 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:03 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:06:03.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:03 smithi098 ceph-mon[35569]: pgmap v271: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 853 KiB/s wr, 93 op/s 2024-06-04T05:06:03.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:03 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:06:03.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:03 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:06:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:03 smithi112 ceph-mon[36453]: pgmap v271: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 853 KiB/s wr, 93 op/s 2024-06-04T05:06:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:03 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:06:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:03 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:06:03.729 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152378 1.95 MB/sec execute 296 sec latency 3422.129 ms 2024-06-04T05:06:03.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.882+0000 7fba88bbb700 1 -- 172.21.15.70:0/2190367446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba84102ae0 msgr2=0x7fba84102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:03.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.882+0000 7fba88bbb700 1 --2- 172.21.15.70:0/2190367446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba84102ae0 0x7fba84102ec0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fba740099c0 tx=0x7fba74009cd0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.882+0000 7fba88bbb700 1 -- 172.21.15.70:0/2190367446 shutdown_connections 2024-06-04T05:06:03.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.882+0000 7fba88bbb700 1 --2- 172.21.15.70:0/2190367446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba84107a30 0x7fba8410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.882+0000 7fba88bbb700 1 --2- 172.21.15.70:0/2190367446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba84102ae0 0x7fba84102ec0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.882+0000 7fba88bbb700 1 --2- 172.21.15.70:0/2190367446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba84107070 0x7fba841074f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.882+0000 7fba88bbb700 1 -- 172.21.15.70:0/2190367446 >> 172.21.15.70:0/2190367446 conn(0x7fba840793f0 msgr2=0x7fba84079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:03.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.882+0000 7fba88bbb700 1 -- 172.21.15.70:0/2190367446 shutdown_connections 2024-06-04T05:06:03.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.882+0000 7fba88bbb700 1 -- 172.21.15.70:0/2190367446 wait complete. 2024-06-04T05:06:03.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.883+0000 7fba88bbb700 1 Processor -- start 2024-06-04T05:06:03.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.883+0000 7fba88bbb700 1 -- start start 2024-06-04T05:06:03.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.883+0000 7fba88bbb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba84102ae0 0x7fba84104110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.883+0000 7fba88bbb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba84107070 0x7fba84104650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.883+0000 7fba88bbb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba84107a30 0x7fba84104b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.883+0000 7fba88bbb700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fba841124c0 con 0x7fba84107a30 2024-06-04T05:06:03.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.883+0000 7fba88bbb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fba84112340 con 0x7fba84102ae0 2024-06-04T05:06:03.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.883+0000 7fba88bbb700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fba84112640 con 0x7fba84107070 2024-06-04T05:06:03.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.883+0000 7fba82d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba84107a30 0x7fba84104b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.883+0000 7fba82d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba84107a30 0x7fba84104b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53238/0 (socket says 172.21.15.70:53238) 2024-06-04T05:06:03.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.883+0000 7fba81d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba84107070 0x7fba84104650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.883+0000 7fba82d9d700 1 -- 172.21.15.70:0/1059839190 learned_addr learned my addr 172.21.15.70:0/1059839190 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:03.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.883+0000 7fba8259c700 1 --2- 172.21.15.70:0/1059839190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba84102ae0 0x7fba84104110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.884+0000 7fba82d9d700 1 -- 172.21.15.70:0/1059839190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba84107070 msgr2=0x7fba84104650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:03.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.884+0000 7fba82d9d700 1 --2- 172.21.15.70:0/1059839190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba84107070 0x7fba84104650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.884+0000 7fba82d9d700 1 -- 172.21.15.70:0/1059839190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba84102ae0 msgr2=0x7fba84104110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:03.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.884+0000 7fba82d9d700 1 --2- 172.21.15.70:0/1059839190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba84102ae0 0x7fba84104110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.884+0000 7fba82d9d700 1 -- 172.21.15.70:0/1059839190 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba841a7fa0 con 0x7fba84107a30 2024-06-04T05:06:03.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.884+0000 7fba82d9d700 1 --2- 172.21.15.70:0/1059839190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba84107a30 0x7fba84104b90 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fba7c00ef30 tx=0x7fba7c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:03.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.884+0000 7fba737fe700 1 -- 172.21.15.70:0/1059839190 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c0092e0 con 0x7fba84107a30 2024-06-04T05:06:03.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.884+0000 7fba737fe700 1 -- 172.21.15.70:0/1059839190 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba7c004bb0 con 0x7fba84107a30 2024-06-04T05:06:03.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.884+0000 7fba88bbb700 1 -- 172.21.15.70:0/1059839190 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba841a8230 con 0x7fba84107a30 2024-06-04T05:06:03.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.884+0000 7fba737fe700 1 -- 172.21.15.70:0/1059839190 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c004020 con 0x7fba84107a30 2024-06-04T05:06:03.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.884+0000 7fba88bbb700 1 -- 172.21.15.70:0/1059839190 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba841a8770 con 0x7fba84107a30 2024-06-04T05:06:03.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.886+0000 7fba737fe700 1 -- 172.21.15.70:0/1059839190 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fba7c01b030 con 0x7fba84107a30 2024-06-04T05:06:03.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.887+0000 7fba737fe700 1 --2- 172.21.15.70:0/1059839190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fba6c0b2aa0 0x7fba6c0b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.887+0000 7fba8259c700 1 --2- 172.21.15.70:0/1059839190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fba6c0b2aa0 0x7fba6c0b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.887+0000 7fba737fe700 1 -- 172.21.15.70:0/1059839190 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fba7c0e2070 con 0x7fba84107a30 2024-06-04T05:06:03.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.888+0000 7fba8259c700 1 --2- 172.21.15.70:0/1059839190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fba6c0b2aa0 0x7fba6c0b4f60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fba74003760 tx=0x7fba74005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:03.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.888+0000 7fba93450b80 1 Processor -- start 2024-06-04T05:06:03.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.888+0000 7fba93450b80 1 -- start start 2024-06-04T05:06:03.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.888+0000 7fba93450b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b597b0 0x555809b59b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.888+0000 7fba93450b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b54c80 0x555809b55100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.888+0000 7fba93450b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b55640 0x555809b616b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.888+0000 7fba93450b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555809ab2f00 con 0x555809b597b0 2024-06-04T05:06:03.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.888+0000 7fba93450b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555809aa2aa0 con 0x555809b54c80 2024-06-04T05:06:03.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.888+0000 7fba93450b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555809aa6ae0 con 0x555809b55640 2024-06-04T05:06:03.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.888+0000 7fba8259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b597b0 0x555809b59b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.888+0000 7fba8259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b597b0 0x555809b59b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53246/0 (socket says 172.21.15.70:53246) 2024-06-04T05:06:03.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.888+0000 7fba8259c700 1 -- 172.21.15.70:0/1818271575 learned_addr learned my addr 172.21.15.70:0/1818271575 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:03.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.888+0000 7fba81d9b700 1 --2- 172.21.15.70:0/1818271575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b54c80 0x555809b55100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.888+0000 7fba82d9d700 1 --2- 172.21.15.70:0/1818271575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b55640 0x555809b616b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.889+0000 7fba8259c700 1 -- 172.21.15.70:0/1818271575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b55640 msgr2=0x555809b616b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:03.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.889+0000 7fba8259c700 1 --2- 172.21.15.70:0/1818271575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b55640 0x555809b616b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.889+0000 7fba8259c700 1 -- 172.21.15.70:0/1818271575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b54c80 msgr2=0x555809b55100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:03.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.889+0000 7fba8259c700 1 --2- 172.21.15.70:0/1818271575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b54c80 0x555809b55100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.889+0000 7fba8259c700 1 -- 172.21.15.70:0/1818271575 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555809a4c140 con 0x555809b597b0 2024-06-04T05:06:03.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.889+0000 7fba8259c700 1 --2- 172.21.15.70:0/1818271575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b597b0 0x555809b59b90 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fba740099c0 tx=0x7fba7401a070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ed7041a3143d1d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:03.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.889+0000 7fba70ff9700 1 -- 172.21.15.70:0/1818271575 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74003c20 con 0x555809b597b0 2024-06-04T05:06:03.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.889+0000 7fba70ff9700 1 -- 172.21.15.70:0/1818271575 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba74003dc0 con 0x555809b597b0 2024-06-04T05:06:03.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.889+0000 7fba70ff9700 1 -- 172.21.15.70:0/1818271575 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba740239b0 con 0x555809b597b0 2024-06-04T05:06:03.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.890+0000 7fba93450b80 1 -- 172.21.15.70:0/1818271575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b597b0 msgr2=0x555809b59b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:03.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.890+0000 7fba93450b80 1 --2- 172.21.15.70:0/1818271575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b597b0 0x555809b59b90 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fba740099c0 tx=0x7fba7401a070 comp rx=0 tx=0).stop 2024-06-04T05:06:03.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.890+0000 7fba93450b80 1 -- 172.21.15.70:0/1818271575 shutdown_connections 2024-06-04T05:06:03.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.890+0000 7fba93450b80 1 --2- 172.21.15.70:0/1818271575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b55640 0x555809b616b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.890+0000 7fba93450b80 1 --2- 172.21.15.70:0/1818271575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b597b0 0x555809b59b90 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.890+0000 7fba93450b80 1 --2- 172.21.15.70:0/1818271575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b54c80 0x555809b55100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.890+0000 7fba93450b80 1 -- 172.21.15.70:0/1818271575 >> 172.21.15.70:0/1818271575 conn(0x555809a803d0 msgr2=0x555809b4f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:03.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.890+0000 7fba93450b80 1 -- 172.21.15.70:0/1818271575 shutdown_connections 2024-06-04T05:06:03.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.890+0000 7fba93450b80 1 -- 172.21.15.70:0/1818271575 wait complete. 2024-06-04T05:06:03.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba93450b80 1 -- start start 2024-06-04T05:06:03.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba93450b80 1 Processor -- start 2024-06-04T05:06:03.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba93450b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b54c80 0x555809b613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba93450b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b55640 0x555809b5e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba93450b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b597b0 0x555809b5ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba93450b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5558099a9130 con 0x555809b597b0 2024-06-04T05:06:03.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba93450b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555809a0af30 con 0x555809b54c80 2024-06-04T05:06:03.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba93450b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555809b61c50 con 0x555809b55640 2024-06-04T05:06:03.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba82d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b597b0 0x555809b5ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba81d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b55640 0x555809b5e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba81d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b55640 0x555809b5e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46542/0 (socket says 172.21.15.70:46542) 2024-06-04T05:06:03.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba93450b80 10 client.0 _ll_register_callbacks cb 0x555809a3dae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:03.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba81d9b700 1 -- 172.21.15.70:0/1620780816 learned_addr learned my addr 172.21.15.70:0/1620780816 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:03.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba8259c700 1 --2- 172.21.15.70:0/1620780816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b54c80 0x555809b613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba81d9b700 1 -- 172.21.15.70:0/1620780816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b54c80 msgr2=0x555809b613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:03.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba81d9b700 1 --2- 172.21.15.70:0/1620780816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b54c80 0x555809b613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba81d9b700 1 -- 172.21.15.70:0/1620780816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b597b0 msgr2=0x555809b5ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:03.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba81d9b700 1 --2- 172.21.15.70:0/1620780816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b597b0 0x555809b5ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.891+0000 7fba81d9b700 1 -- 172.21.15.70:0/1620780816 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558099f2a60 con 0x555809b55640 2024-06-04T05:06:03.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.892+0000 7fba81d9b700 1 --2- 172.21.15.70:0/1620780816 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b55640 0x555809b5e6e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fba7800bb70 tx=0x7fba7800be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4eb0acc7277d1dd6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:03.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.892+0000 7fba69ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:06:03.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.892+0000 7fba69ffb700 1 -- 172.21.15.70:0/1620780816 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba78003d00 con 0x555809b55640 2024-06-04T05:06:03.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.892+0000 7fba69ffb700 1 -- 172.21.15.70:0/1620780816 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fba78004830 con 0x555809b55640 2024-06-04T05:06:03.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.892+0000 7fba93450b80 1 -- 172.21.15.70:0/1620780816 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba74003dc0 con 0x555809b55640 2024-06-04T05:06:03.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.892+0000 7fba69ffb700 1 -- 172.21.15.70:0/1620780816 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fba78004a50 con 0x555809b55640 2024-06-04T05:06:03.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.892+0000 7fba93450b80 10 client.27097 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:03.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.892+0000 7fba93450b80 1 -- 172.21.15.70:0/1620780816 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555809a5c8f0 con 0x555809b55640 2024-06-04T05:06:03.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.892+0000 7fba69ffb700 1 -- 172.21.15.70:0/1620780816 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba74003dc0 con 0x555809b55640 2024-06-04T05:06:03.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.892+0000 7fba69ffb700 1 -- 172.21.15.70:0/1620780816 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fba78022070 con 0x555809b55640 2024-06-04T05:06:03.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.892+0000 7fba93450b80 10 client.27097 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:03.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.892+0000 7fba93450b80 10 client.27097 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:06:03.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.893+0000 7fba93450b80 20 client.27097 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:03.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.893+0000 7fba93450b80 1 --2- 172.21.15.70:0/1620780816 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555809b6e8d0 0x555809b70cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.893+0000 7fba93450b80 4 client.27097 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:06:03.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.893+0000 7fba93450b80 1 -- 172.21.15.70:0/1620780816 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555809ab4e50 con 0x555809b6e8d0 2024-06-04T05:06:03.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.893+0000 7fba8259c700 1 --2- 172.21.15.70:0/1620780816 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555809b6e8d0 0x555809b70cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.893+0000 7fba69ffb700 10 client.27097 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:03.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.893+0000 7fba8259c700 1 --2- 172.21.15.70:0/1620780816 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555809b6e8d0 0x555809b70cc0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=183c4200d731f1cd server_cookie=fdc5ef8f24d1846d in_seq=0 out_seq=0 2024-06-04T05:06:03.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.894+0000 7fba69ffb700 1 -- 172.21.15.70:0/1620780816 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555809ab4e50 con 0x555809b6e8d0 2024-06-04T05:06:03.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.894+0000 7fba69ffb700 10 client.27097 handle_command_reply: tid=0 2024-06-04T05:06:03.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.894+0000 7fba93450b80 1 client.27097 shutdown 2024-06-04T05:06:03.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.894+0000 7fba93450b80 1 -- 172.21.15.70:0/1620780816 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b55640 msgr2=0x555809b5e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:03.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.894+0000 7fba93450b80 1 --2- 172.21.15.70:0/1620780816 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b55640 0x555809b5e6e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fba7800bb70 tx=0x7fba7800be80 comp rx=0 tx=0).stop 2024-06-04T05:06:03.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.894+0000 7fba93450b80 1 -- 172.21.15.70:0/1620780816 shutdown_connections 2024-06-04T05:06:03.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.894+0000 7fba93450b80 1 --2- 172.21.15.70:0/1620780816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b597b0 0x555809b5ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.894+0000 7fba93450b80 1 --2- 172.21.15.70:0/1620780816 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555809b6e8d0 0x555809b70cc0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.894+0000 7fba93450b80 1 --2- 172.21.15.70:0/1620780816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b54c80 0x555809b613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.894+0000 7fba69ffb700 0 client.27097 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:03.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.894+0000 7fba93450b80 1 --2- 172.21.15.70:0/1620780816 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b55640 0x555809b5e6e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.894+0000 7fba93450b80 1 -- 172.21.15.70:0/1620780816 >> 172.21.15.70:0/1620780816 conn(0x555809a803d0 msgr2=0x555809b53c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:03.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.894+0000 7fba93450b80 1 -- 172.21.15.70:0/1620780816 shutdown_connections 2024-06-04T05:06:03.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.894+0000 7fba93450b80 1 -- 172.21.15.70:0/1620780816 wait complete. 2024-06-04T05:06:03.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.894+0000 7fba93450b80 20 client.27097 trim_cache size 0 max 16384 2024-06-04T05:06:03.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.918+0000 7fba93450b80 1 Processor -- start 2024-06-04T05:06:03.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.918+0000 7fba93450b80 1 -- start start 2024-06-04T05:06:03.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.918+0000 7fba93450b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b54c80 0x555809b58cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.918+0000 7fba93450b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b55640 0x555809b6acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.918+0000 7fba93450b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b597b0 0x555809b49f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.918+0000 7fba93450b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fba841b0060 con 0x555809b54c80 2024-06-04T05:06:03.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.918+0000 7fba93450b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fba841b1ba0 con 0x555809b55640 2024-06-04T05:06:03.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.918+0000 7fba93450b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fba841b0730 con 0x555809b597b0 2024-06-04T05:06:03.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.918+0000 7fba8259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b54c80 0x555809b58cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.918+0000 7fba8259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b54c80 0x555809b58cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53260/0 (socket says 172.21.15.70:53260) 2024-06-04T05:06:03.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.918+0000 7fba8259c700 1 -- 172.21.15.70:0/1276373944 learned_addr learned my addr 172.21.15.70:0/1276373944 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:03.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.918+0000 7fba82d9d700 1 --2- 172.21.15.70:0/1276373944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b597b0 0x555809b49f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.918+0000 7fba81d9b700 1 --2- 172.21.15.70:0/1276373944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b55640 0x555809b6acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.918+0000 7fba8259c700 1 -- 172.21.15.70:0/1276373944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b597b0 msgr2=0x555809b49f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:03.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.918+0000 7fba8259c700 1 --2- 172.21.15.70:0/1276373944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b597b0 0x555809b49f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.918+0000 7fba8259c700 1 -- 172.21.15.70:0/1276373944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b55640 msgr2=0x555809b6acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:03.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.918+0000 7fba8259c700 1 --2- 172.21.15.70:0/1276373944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b55640 0x555809b6acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.919+0000 7fba8259c700 1 -- 172.21.15.70:0/1276373944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555809b712f0 con 0x555809b54c80 2024-06-04T05:06:03.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.919+0000 7fba81d9b700 1 --2- 172.21.15.70:0/1276373944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b55640 0x555809b6acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:06:03.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.919+0000 7fba8259c700 1 --2- 172.21.15.70:0/1276373944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b54c80 0x555809b58cb0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fba74009750 tx=0x7fba74025040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d44f3d19eb50b82 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:03.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.919+0000 7fba6bfff700 1 -- 172.21.15.70:0/1276373944 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74005630 con 0x555809b54c80 2024-06-04T05:06:03.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.919+0000 7fba6bfff700 1 -- 172.21.15.70:0/1276373944 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba74023a90 con 0x555809b54c80 2024-06-04T05:06:03.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.919+0000 7fba6bfff700 1 -- 172.21.15.70:0/1276373944 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74039410 con 0x555809b54c80 2024-06-04T05:06:03.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.920+0000 7fba93450b80 1 -- 172.21.15.70:0/1276373944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b54c80 msgr2=0x555809b58cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:03.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.920+0000 7fba93450b80 1 --2- 172.21.15.70:0/1276373944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b54c80 0x555809b58cb0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fba74009750 tx=0x7fba74025040 comp rx=0 tx=0).stop 2024-06-04T05:06:03.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.920+0000 7fba93450b80 1 -- 172.21.15.70:0/1276373944 shutdown_connections 2024-06-04T05:06:03.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.920+0000 7fba93450b80 1 --2- 172.21.15.70:0/1276373944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b597b0 0x555809b49f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.920+0000 7fba93450b80 1 --2- 172.21.15.70:0/1276373944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b54c80 0x555809b58cb0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.920+0000 7fba93450b80 1 --2- 172.21.15.70:0/1276373944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b55640 0x555809b6acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.920+0000 7fba93450b80 1 -- 172.21.15.70:0/1276373944 >> 172.21.15.70:0/1276373944 conn(0x555809a803d0 msgr2=0x555809b5a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:03.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.920+0000 7fba93450b80 1 -- 172.21.15.70:0/1276373944 shutdown_connections 2024-06-04T05:06:03.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.920+0000 7fba93450b80 1 -- 172.21.15.70:0/1276373944 wait complete. 2024-06-04T05:06:03.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.920+0000 7fba93450b80 1 -- start start 2024-06-04T05:06:03.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.920+0000 7fba93450b80 1 Processor -- start 2024-06-04T05:06:03.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.920+0000 7fba93450b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b54c80 0x555809b76060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.920+0000 7fba93450b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b55640 0x555809b765a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.920+0000 7fba93450b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b597b0 0x555809b76ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.920+0000 7fba93450b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fba841b14d0 con 0x555809b54c80 2024-06-04T05:06:03.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.920+0000 7fba93450b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fba841b0e00 con 0x555809b55640 2024-06-04T05:06:03.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.920+0000 7fba93450b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fba841ae020 con 0x555809b597b0 2024-06-04T05:06:03.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba93450b80 10 client.0 _ll_register_callbacks cb 0x555809a3dae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:03.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba8259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b54c80 0x555809b76060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba8259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b54c80 0x555809b76060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53276/0 (socket says 172.21.15.70:53276) 2024-06-04T05:06:03.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba8259c700 1 -- 172.21.15.70:0/3621204302 learned_addr learned my addr 172.21.15.70:0/3621204302 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:03.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba82d9d700 1 --2- 172.21.15.70:0/3621204302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b597b0 0x555809b76ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba81d9b700 1 --2- 172.21.15.70:0/3621204302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b55640 0x555809b765a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba8259c700 1 -- 172.21.15.70:0/3621204302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b597b0 msgr2=0x555809b76ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:03.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba8259c700 1 --2- 172.21.15.70:0/3621204302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b597b0 0x555809b76ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba8259c700 1 -- 172.21.15.70:0/3621204302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b55640 msgr2=0x555809b765a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:03.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba8259c700 1 --2- 172.21.15.70:0/3621204302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b55640 0x555809b765a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba8259c700 1 -- 172.21.15.70:0/3621204302 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555809bd8420 con 0x555809b54c80 2024-06-04T05:06:03.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba8259c700 1 --2- 172.21.15.70:0/3621204302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b54c80 0x555809b76060 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fba74005090 tx=0x7fba74003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=776e866d14be6542 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:03.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba69ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:03.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba69ffb700 1 -- 172.21.15.70:0/3621204302 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba740056b0 con 0x555809b54c80 2024-06-04T05:06:03.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba69ffb700 1 -- 172.21.15.70:0/3621204302 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8571+0+0 (secure 0 0 0) 0x7fba7400bde0 con 0x555809b54c80 2024-06-04T05:06:03.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba93450b80 1 -- 172.21.15.70:0/3621204302 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba74023a90 con 0x555809b54c80 2024-06-04T05:06:03.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba69ffb700 1 -- 172.21.15.70:0/3621204302 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fba7400b6b0 con 0x555809b54c80 2024-06-04T05:06:03.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.921+0000 7fba93450b80 10 client.17868 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:03.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.922+0000 7fba93450b80 1 -- 172.21.15.70:0/3621204302 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555809bd96e0 con 0x555809b54c80 2024-06-04T05:06:03.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.922+0000 7fba69ffb700 1 -- 172.21.15.70:0/3621204302 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba74023a90 con 0x555809b54c80 2024-06-04T05:06:03.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.922+0000 7fba69ffb700 1 -- 172.21.15.70:0/3621204302 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fba7404d870 con 0x555809b54c80 2024-06-04T05:06:03.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.922+0000 7fba93450b80 10 client.17868 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:03.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.922+0000 7fba93450b80 10 client.17868 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:06:03.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.922+0000 7fba93450b80 20 client.17868 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:03.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.922+0000 7fba93450b80 1 --2- 172.21.15.70:0/3621204302 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555809bd9960 0x555809bdbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:03.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.922+0000 7fba93450b80 4 client.17868 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.6, "await_for": 45.0}] 2024-06-04T05:06:03.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.922+0000 7fba93450b80 1 -- 172.21.15.70:0/3621204302 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.6, "await_for": 45.0}) -- 0x555809aae630 con 0x555809bd9960 2024-06-04T05:06:03.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.922+0000 7fba81d9b700 1 --2- 172.21.15.70:0/3621204302 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555809bd9960 0x555809bdbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:03.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.922+0000 7fba81d9b700 1 --2- 172.21.15.70:0/3621204302 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555809bd9960 0x555809bdbd50 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=871b0397e0a7e66b server_cookie=137fe58a156e4d8b in_seq=0 out_seq=0 2024-06-04T05:06:03.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.922+0000 7fba69ffb700 10 client.17868 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:03.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.981+0000 7fba69ffb700 1 -- 172.21.15.70:0/3621204302 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x555809aae630 con 0x555809bd9960 2024-06-04T05:06:03.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.981+0000 7fba69ffb700 10 client.17868 handle_command_reply: tid=0 2024-06-04T05:06:03.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.981+0000 7fba93450b80 1 client.17868 shutdown 2024-06-04T05:06:03.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.982+0000 7fba93450b80 1 -- 172.21.15.70:0/3621204302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b54c80 msgr2=0x555809b76060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:03.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.982+0000 7fba93450b80 1 --2- 172.21.15.70:0/3621204302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b54c80 0x555809b76060 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fba74005090 tx=0x7fba74003dc0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.982+0000 7fba93450b80 1 -- 172.21.15.70:0/3621204302 shutdown_connections 2024-06-04T05:06:03.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.982+0000 7fba93450b80 1 --2- 172.21.15.70:0/3621204302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555809b597b0 0x555809b76ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.982+0000 7fba93450b80 1 --2- 172.21.15.70:0/3621204302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555809b54c80 0x555809b76060 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.982+0000 7fba93450b80 1 --2- 172.21.15.70:0/3621204302 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555809bd9960 0x555809bdbd50 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.982+0000 7fba69ffb700 0 client.17868 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:03.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.982+0000 7fba93450b80 1 --2- 172.21.15.70:0/3621204302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555809b55640 0x555809b765a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.982+0000 7fba93450b80 1 -- 172.21.15.70:0/3621204302 >> 172.21.15.70:0/3621204302 conn(0x555809a803d0 msgr2=0x555809b5ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:03.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.982+0000 7fba93450b80 1 -- 172.21.15.70:0/3621204302 shutdown_connections 2024-06-04T05:06:03.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.982+0000 7fba93450b80 1 -- 172.21.15.70:0/3621204302 wait complete. 2024-06-04T05:06:03.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.982+0000 7fba93450b80 20 client.17868 trim_cache size 0 max 16384 2024-06-04T05:06:03.990 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:06:03.990 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 32, 2024-06-04T05:06:03.990 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:06:03.990 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 26, 2024-06-04T05:06:03.990 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:06:03.990 INFO:teuthology.orchestra.run.smithi070.stdout: "24943b9a": { 2024-06-04T05:06:03.991 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 26, 2024-06-04T05:06:03.991 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:06:03.991 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:06:03.991 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:06:03.991 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:06:03.991 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:06:03.991 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:06:03.991 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 143.6, 2024-06-04T05:06:03.991 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:06:03.991 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:06:03.991 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:06:03.991 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:06:03.992 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:06:03.992 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:06:03.992 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:06:03.992 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:06:03.992 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:06:03.992 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:06:03.992 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:06:03.992 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:06:03.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.983+0000 7fba88bbb700 1 -- 172.21.15.70:0/1059839190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fba6c0b2aa0 msgr2=0x7fba6c0b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:03.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.983+0000 7fba88bbb700 1 --2- 172.21.15.70:0/1059839190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fba6c0b2aa0 0x7fba6c0b4f60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fba74003760 tx=0x7fba74005fb0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.983+0000 7fba88bbb700 1 -- 172.21.15.70:0/1059839190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba84107a30 msgr2=0x7fba84104b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:03.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.983+0000 7fba88bbb700 1 --2- 172.21.15.70:0/1059839190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba84107a30 0x7fba84104b90 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fba7c00ef30 tx=0x7fba7c00c3a0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.983+0000 7fba88bbb700 1 -- 172.21.15.70:0/1059839190 shutdown_connections 2024-06-04T05:06:03.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.983+0000 7fba88bbb700 1 --2- 172.21.15.70:0/1059839190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fba6c0b2aa0 0x7fba6c0b4f60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.983+0000 7fba88bbb700 1 --2- 172.21.15.70:0/1059839190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba84107a30 0x7fba84104b90 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.983+0000 7fba88bbb700 1 --2- 172.21.15.70:0/1059839190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba84102ae0 0x7fba84104110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.983+0000 7fba88bbb700 1 --2- 172.21.15.70:0/1059839190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba84107070 0x7fba84104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:03.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.983+0000 7fba88bbb700 1 -- 172.21.15.70:0/1059839190 >> 172.21.15.70:0/1059839190 conn(0x7fba840793f0 msgr2=0x7fba84106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:03.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.983+0000 7fba88bbb700 1 -- 172.21.15.70:0/1059839190 shutdown_connections 2024-06-04T05:06:03.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:03.983+0000 7fba88bbb700 1 -- 172.21.15.70:0/1059839190 wait complete. 2024-06-04T05:06:04.456 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:04 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:06:04.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:04 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:06:04.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:04 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:06:04.729 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.95 MB/sec execute 297 sec latency 731.722 ms 2024-06-04T05:06:04.761 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '24943b9a', quiesce took 10.2 seconds. Will release after: 13.400000000000002 2024-06-04T05:06:05.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:05 smithi098 ceph-mon[35569]: pgmap v272: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 853 KiB/s wr, 89 op/s 2024-06-04T05:06:05.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:05 smithi070 ceph-mon[32631]: pgmap v272: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 853 KiB/s wr, 89 op/s 2024-06-04T05:06:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:05 smithi112 ceph-mon[36453]: pgmap v272: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 853 KiB/s wr, 89 op/s 2024-06-04T05:06:05.729 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.94 MB/sec execute 298 sec latency 1731.765 ms 2024-06-04T05:06:06.729 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.94 MB/sec execute 299 sec latency 2731.867 ms 2024-06-04T05:06:07.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:07 smithi098 ceph-mon[35569]: pgmap v273: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 938 KiB/s wr, 119 op/s 2024-06-04T05:06:07.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:07 smithi070 ceph-mon[32631]: pgmap v273: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 938 KiB/s wr, 119 op/s 2024-06-04T05:06:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:07 smithi112 ceph-mon[36453]: pgmap v273: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 938 KiB/s wr, 119 op/s 2024-06-04T05:06:07.729 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.93 MB/sec execute 300 sec latency 3731.929 ms 2024-06-04T05:06:08.729 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.92 MB/sec execute 301 sec latency 4732.045 ms 2024-06-04T05:06:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.420+0000 7f1f61485640 1 -- 172.21.15.70:0/2272097493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f5c1068f0 msgr2=0x7f1f5c106cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.420+0000 7f1f61485640 1 --2- 172.21.15.70:0/2272097493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f5c1068f0 0x7f1f5c106cd0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c009900 tx=0x7f1f4c02a410 comp rx=0 tx=0).stop 2024-06-04T05:06:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.420+0000 7f1f61485640 1 -- 172.21.15.70:0/2272097493 shutdown_connections 2024-06-04T05:06:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.420+0000 7f1f61485640 1 --2- 172.21.15.70:0/2272097493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f5c102660 0x7f1f5c10e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.420+0000 7f1f61485640 1 --2- 172.21.15.70:0/2272097493 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f5c101ca0 0x7f1f5c102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.420+0000 7f1f61485640 1 --2- 172.21.15.70:0/2272097493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f5c1068f0 0x7f1f5c106cd0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.420+0000 7f1f61485640 1 -- 172.21.15.70:0/2272097493 >> 172.21.15.70:0/2272097493 conn(0x7f1f5c077750 msgr2=0x7f1f5c077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.420+0000 7f1f61485640 1 -- 172.21.15.70:0/2272097493 shutdown_connections 2024-06-04T05:06:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.420+0000 7f1f61485640 1 -- 172.21.15.70:0/2272097493 wait complete. 2024-06-04T05:06:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.420+0000 7f1f61485640 1 Processor -- start 2024-06-04T05:06:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.421+0000 7f1f61485640 1 -- start start 2024-06-04T05:06:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.421+0000 7f1f61485640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f5c101ca0 0x7f1f5c199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.421+0000 7f1f61485640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f5c102660 0x7f1f5c19a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.421+0000 7f1f61485640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f5c1068f0 0x7f1f5c1a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.421+0000 7f1f61485640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c111a80 con 0x7f1f5c102660 2024-06-04T05:06:09.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.421+0000 7f1f61485640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c111900 con 0x7f1f5c1068f0 2024-06-04T05:06:09.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.421+0000 7f1f61485640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c111c00 con 0x7f1f5c101ca0 2024-06-04T05:06:09.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.421+0000 7f1f5a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f5c102660 0x7f1f5c19a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:09.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.421+0000 7f1f5a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f5c102660 0x7f1f5c19a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53002/0 (socket says 172.21.15.70:53002) 2024-06-04T05:06:09.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.421+0000 7f1f5a7fc640 1 -- 172.21.15.70:0/2125957887 learned_addr learned my addr 172.21.15.70:0/2125957887 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:09.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.421+0000 7f1f5affd640 1 --2- 172.21.15.70:0/2125957887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f5c101ca0 0x7f1f5c199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:09.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.421+0000 7f1f5b7fe640 1 --2- 172.21.15.70:0/2125957887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f5c1068f0 0x7f1f5c1a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:09.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.421+0000 7f1f5a7fc640 1 -- 172.21.15.70:0/2125957887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f5c101ca0 msgr2=0x7f1f5c199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:09.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.421+0000 7f1f5a7fc640 1 --2- 172.21.15.70:0/2125957887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f5c101ca0 0x7f1f5c199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.421+0000 7f1f5a7fc640 1 -- 172.21.15.70:0/2125957887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f5c1068f0 msgr2=0x7f1f5c1a4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:09.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.421+0000 7f1f5a7fc640 1 --2- 172.21.15.70:0/2125957887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f5c1068f0 0x7f1f5c1a4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.421+0000 7f1f5a7fc640 1 -- 172.21.15.70:0/2125957887 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f5c1a5420 con 0x7f1f5c102660 2024-06-04T05:06:09.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.422+0000 7f1f5a7fc640 1 --2- 172.21.15.70:0/2125957887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f5c102660 0x7f1f5c19a3a0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000e9f0 tx=0x7f1f5000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:09.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.422+0000 7f1f43fff640 1 -- 172.21.15.70:0/2125957887 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5000cd80 con 0x7f1f5c102660 2024-06-04T05:06:09.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.422+0000 7f1f43fff640 1 -- 172.21.15.70:0/2125957887 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f500054b0 con 0x7f1f5c102660 2024-06-04T05:06:09.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.422+0000 7f1f61485640 1 -- 172.21.15.70:0/2125957887 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f5c1a55f0 con 0x7f1f5c102660 2024-06-04T05:06:09.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.422+0000 7f1f43fff640 1 -- 172.21.15.70:0/2125957887 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f50010430 con 0x7f1f5c102660 2024-06-04T05:06:09.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.422+0000 7f1f61485640 1 -- 172.21.15.70:0/2125957887 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f5c1a5af0 con 0x7f1f5c102660 2024-06-04T05:06:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.424+0000 7f1f43fff640 1 -- 172.21.15.70:0/2125957887 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f1f500046c0 con 0x7f1f5c102660 2024-06-04T05:06:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.424+0000 7f1f6419f740 1 Processor -- start 2024-06-04T05:06:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.424+0000 7f1f6419f740 1 -- start start 2024-06-04T05:06:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.424+0000 7f1f6419f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd0b820 0x55728fd0bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f6419f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd164c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f6419f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd16a00 0x55728fd18df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f6419f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55728fc7dc30 con 0x55728fd0c140 2024-06-04T05:06:09.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f6419f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55728fc6d7d0 con 0x55728fd0b820 2024-06-04T05:06:09.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f6419f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55728fc71810 con 0x55728fd16a00 2024-06-04T05:06:09.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f5affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd164c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:09.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f5affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd164c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53008/0 (socket says 172.21.15.70:53008) 2024-06-04T05:06:09.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f5affd640 1 -- 172.21.15.70:0/2517240338 learned_addr learned my addr 172.21.15.70:0/2517240338 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:09.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f5a7fc640 1 --2- 172.21.15.70:0/2517240338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd16a00 0x55728fd18df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:09.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f5b7fe640 1 --2- 172.21.15.70:0/2517240338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd0b820 0x55728fd0bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:09.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f5affd640 1 -- 172.21.15.70:0/2517240338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd16a00 msgr2=0x55728fd18df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:09.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f5affd640 1 --2- 172.21.15.70:0/2517240338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd16a00 0x55728fd18df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f5affd640 1 -- 172.21.15.70:0/2517240338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd0b820 msgr2=0x55728fd0bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:09.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f5affd640 1 --2- 172.21.15.70:0/2517240338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd0b820 0x55728fd0bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f5affd640 1 -- 172.21.15.70:0/2517240338 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55728f982210 con 0x55728fd0c140 2024-06-04T05:06:09.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f5b7fe640 1 --2- 172.21.15.70:0/2517240338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd0b820 0x55728fd0bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:06:09.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f43fff640 1 --2- 172.21.15.70:0/2125957887 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1f440b2a10 0x7f1f440b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f5b7fe640 1 --2- 172.21.15.70:0/2125957887 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1f440b2a10 0x7f1f440b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:09.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f5affd640 1 --2- 172.21.15.70:0/2517240338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd164c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c02a930 tx=0x7f1f4c032b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f60d132bd0a03fe server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:09.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f43fff640 1 -- 172.21.15.70:0/2125957887 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8615+0+0 (secure 0 0 0) 0x7f1f500e2b50 con 0x7f1f5c102660 2024-06-04T05:06:09.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f417fa640 1 -- 172.21.15.70:0/2517240338 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c03f070 con 0x55728fd0c140 2024-06-04T05:06:09.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f417fa640 1 -- 172.21.15.70:0/2517240338 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f4c004030 con 0x55728fd0c140 2024-06-04T05:06:09.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.425+0000 7f1f417fa640 1 -- 172.21.15.70:0/2517240338 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c035070 con 0x55728fd0c140 2024-06-04T05:06:09.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.426+0000 7f1f6419f740 1 -- 172.21.15.70:0/2517240338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 msgr2=0x55728fd164c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:09.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.426+0000 7f1f5b7fe640 1 --2- 172.21.15.70:0/2125957887 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1f440b2a10 0x7f1f440b4ed0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1f54005fd0 tx=0x7f1f54009500 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:09.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.426+0000 7f1f6419f740 1 --2- 172.21.15.70:0/2517240338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd164c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c02a930 tx=0x7f1f4c032b00 comp rx=0 tx=0).stop 2024-06-04T05:06:09.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.426+0000 7f1f6419f740 1 -- 172.21.15.70:0/2517240338 shutdown_connections 2024-06-04T05:06:09.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.426+0000 7f1f6419f740 1 --2- 172.21.15.70:0/2517240338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd16a00 0x55728fd18df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.436 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.426+0000 7f1f6419f740 1 --2- 172.21.15.70:0/2517240338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd164c0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.436 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.426+0000 7f1f6419f740 1 --2- 172.21.15.70:0/2517240338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd0b820 0x55728fd0bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.436 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.426+0000 7f1f6419f740 1 -- 172.21.15.70:0/2517240338 >> 172.21.15.70:0/2517240338 conn(0x55728fd07d30 msgr2=0x55728fd08110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:09.436 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.426+0000 7f1f6419f740 1 -- 172.21.15.70:0/2517240338 shutdown_connections 2024-06-04T05:06:09.436 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.426+0000 7f1f6419f740 1 -- 172.21.15.70:0/2517240338 wait complete. 2024-06-04T05:06:09.436 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f6419f740 1 -- start start 2024-06-04T05:06:09.436 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f6419f740 1 Processor -- start 2024-06-04T05:06:09.436 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f6419f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd0b820 0x55728fd139e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.436 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f6419f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd13f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.436 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f6419f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd16a00 0x55728fd14460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.436 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f6419f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55728f953790 con 0x55728fd0c140 2024-06-04T05:06:09.436 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f6419f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55728fc29890 con 0x55728fd0b820 2024-06-04T05:06:09.436 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f6419f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55728fd195d0 con 0x55728fd16a00 2024-06-04T05:06:09.436 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f6419f740 10 client.0 _ll_register_callbacks cb 0x55728f981a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:09.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f5a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd13f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:09.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f5a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd13f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53020/0 (socket says 172.21.15.70:53020) 2024-06-04T05:06:09.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f5a7fc640 1 -- 172.21.15.70:0/2320938322 learned_addr learned my addr 172.21.15.70:0/2320938322 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:09.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f5affd640 1 --2- 172.21.15.70:0/2320938322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd0b820 0x55728fd139e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:09.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f5b7fe640 1 --2- 172.21.15.70:0/2320938322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd16a00 0x55728fd14460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:09.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f5a7fc640 1 -- 172.21.15.70:0/2320938322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd16a00 msgr2=0x55728fd14460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:09.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f5a7fc640 1 --2- 172.21.15.70:0/2320938322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd16a00 0x55728fd14460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f5a7fc640 1 -- 172.21.15.70:0/2320938322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd0b820 msgr2=0x55728fd139e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:09.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f5a7fc640 1 --2- 172.21.15.70:0/2320938322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd0b820 0x55728fd139e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f5a7fc640 1 -- 172.21.15.70:0/2320938322 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55728fa4e620 con 0x55728fd0c140 2024-06-04T05:06:09.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f5a7fc640 1 --2- 172.21.15.70:0/2320938322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd13f20 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000e9c0 tx=0x7f1f500e6280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=703ccb508d9eb224 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:09.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.427+0000 7f1f3eff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:09.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.428+0000 7f1f3eff5640 1 -- 172.21.15.70:0/2320938322 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f500a9ba0 con 0x55728fd0c140 2024-06-04T05:06:09.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.428+0000 7f1f3eff5640 1 -- 172.21.15.70:0/2320938322 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8615+0+0 (secure 0 0 0) 0x7f1f500b29b0 con 0x55728fd0c140 2024-06-04T05:06:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.428+0000 7f1f6419f740 1 -- 172.21.15.70:0/2320938322 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f4c004030 con 0x55728fd0c140 2024-06-04T05:06:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.428+0000 7f1f3eff5640 1 -- 172.21.15.70:0/2320938322 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1f500a9540 con 0x55728fd0c140 2024-06-04T05:06:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.428+0000 7f1f6419f740 10 client.17880 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.428+0000 7f1f6419f740 1 -- 172.21.15.70:0/2320938322 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55728fd253d0 con 0x55728fd0c140 2024-06-04T05:06:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.428+0000 7f1f3eff5640 1 -- 172.21.15.70:0/2320938322 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f4c004030 con 0x55728fd0c140 2024-06-04T05:06:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.428+0000 7f1f3eff5640 1 -- 172.21.15.70:0/2320938322 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f1f50091040 con 0x55728fd0c140 2024-06-04T05:06:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.428+0000 7f1f6419f740 10 client.17880 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.428+0000 7f1f6419f740 10 client.17880 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:06:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.428+0000 7f1f6419f740 20 client.17880 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.428+0000 7f1f6419f740 1 --2- 172.21.15.70:0/2320938322 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55728fd268e0 0x55728fd28cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.428+0000 7f1f6419f740 4 client.17880 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:06:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.428+0000 7f1f6419f740 1 -- 172.21.15.70:0/2320938322 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55728fc7fb80 con 0x55728fd268e0 2024-06-04T05:06:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.428+0000 7f1f5affd640 1 --2- 172.21.15.70:0/2320938322 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55728fd268e0 0x55728fd28cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.429+0000 7f1f3eff5640 10 client.17880 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.429+0000 7f1f5affd640 1 --2- 172.21.15.70:0/2320938322 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55728fd268e0 0x55728fd28cd0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=495e0281d0764fc9 server_cookie=8a6007b923b83ff5 in_seq=0 out_seq=0 2024-06-04T05:06:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.430+0000 7f1f3eff5640 1 -- 172.21.15.70:0/2320938322 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55728fc7fb80 con 0x55728fd268e0 2024-06-04T05:06:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.430+0000 7f1f3eff5640 10 client.17880 handle_command_reply: tid=0 2024-06-04T05:06:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.430+0000 7f1f6419f740 1 client.17880 shutdown 2024-06-04T05:06:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.430+0000 7f1f6419f740 1 -- 172.21.15.70:0/2320938322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 msgr2=0x55728fd13f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.430+0000 7f1f6419f740 1 --2- 172.21.15.70:0/2320938322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd13f20 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000e9c0 tx=0x7f1f500e6280 comp rx=0 tx=0).stop 2024-06-04T05:06:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.430+0000 7f1f6419f740 1 -- 172.21.15.70:0/2320938322 shutdown_connections 2024-06-04T05:06:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.430+0000 7f1f6419f740 1 --2- 172.21.15.70:0/2320938322 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55728fd268e0 0x55728fd28cd0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.430+0000 7f1f6419f740 1 --2- 172.21.15.70:0/2320938322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd16a00 0x55728fd14460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.430+0000 7f1f6419f740 1 --2- 172.21.15.70:0/2320938322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd13f20 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.430+0000 7f1f6419f740 1 --2- 172.21.15.70:0/2320938322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd0b820 0x55728fd139e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.430+0000 7f1f3eff5640 0 client.17880 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.430+0000 7f1f6419f740 1 -- 172.21.15.70:0/2320938322 >> 172.21.15.70:0/2320938322 conn(0x55728fd07d30 msgr2=0x55728fd06fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.430+0000 7f1f6419f740 1 -- 172.21.15.70:0/2320938322 shutdown_connections 2024-06-04T05:06:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.430+0000 7f1f6419f740 1 -- 172.21.15.70:0/2320938322 wait complete. 2024-06-04T05:06:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.430+0000 7f1f6419f740 20 client.17880 trim_cache size 0 max 16384 2024-06-04T05:06:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f6419f740 1 Processor -- start 2024-06-04T05:06:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f6419f740 1 -- start start 2024-06-04T05:06:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f6419f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd0b820 0x55728fd02e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f6419f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd0c140 0x55728fd2e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f6419f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd16a00 0x55728fd2eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f6419f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c1ae630 con 0x55728fd16a00 2024-06-04T05:06:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f6419f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c1b0170 con 0x55728fd0c140 2024-06-04T05:06:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f6419f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c1aed00 con 0x55728fd0b820 2024-06-04T05:06:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f5b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd16a00 0x55728fd2eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f5b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd16a00 0x55728fd2eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53026/0 (socket says 172.21.15.70:53026) 2024-06-04T05:06:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f5b7fe640 1 -- 172.21.15.70:0/23912812 learned_addr learned my addr 172.21.15.70:0/23912812 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f5a7fc640 1 --2- 172.21.15.70:0/23912812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd0c140 0x55728fd2e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f5b7fe640 1 -- 172.21.15.70:0/23912812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd0b820 msgr2=0x55728fd02e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f5b7fe640 1 --2- 172.21.15.70:0/23912812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd0b820 0x55728fd02e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f5b7fe640 1 -- 172.21.15.70:0/23912812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd0c140 msgr2=0x55728fd2e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f5b7fe640 1 --2- 172.21.15.70:0/23912812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd0c140 0x55728fd2e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f5b7fe640 1 -- 172.21.15.70:0/23912812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55728fd12990 con 0x55728fd16a00 2024-06-04T05:06:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f5a7fc640 1 --2- 172.21.15.70:0/23912812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd0c140 0x55728fd2e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:06:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.445+0000 7f1f5b7fe640 1 --2- 172.21.15.70:0/23912812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd16a00 0x55728fd2eb00 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f1f54012880 tx=0x7f1f54014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3516c20dfe19e33 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.446+0000 7f1f40ff9640 1 -- 172.21.15.70:0/23912812 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f54003b70 con 0x55728fd16a00 2024-06-04T05:06:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.446+0000 7f1f40ff9640 1 -- 172.21.15.70:0/23912812 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f54020dc0 con 0x55728fd16a00 2024-06-04T05:06:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.446+0000 7f1f40ff9640 1 -- 172.21.15.70:0/23912812 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f540046c0 con 0x55728fd16a00 2024-06-04T05:06:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.446+0000 7f1f6419f740 1 -- 172.21.15.70:0/23912812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd16a00 msgr2=0x55728fd2eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.446+0000 7f1f6419f740 1 --2- 172.21.15.70:0/23912812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd16a00 0x55728fd2eb00 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f1f54012880 tx=0x7f1f54014000 comp rx=0 tx=0).stop 2024-06-04T05:06:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.446+0000 7f1f6419f740 1 -- 172.21.15.70:0/23912812 shutdown_connections 2024-06-04T05:06:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.446+0000 7f1f6419f740 1 --2- 172.21.15.70:0/23912812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd0c140 0x55728fd2e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.446+0000 7f1f6419f740 1 --2- 172.21.15.70:0/23912812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd16a00 0x55728fd2eb00 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.446+0000 7f1f6419f740 1 --2- 172.21.15.70:0/23912812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd0b820 0x55728fd02e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.446+0000 7f1f6419f740 1 -- 172.21.15.70:0/23912812 >> 172.21.15.70:0/23912812 conn(0x55728fd07d30 msgr2=0x55728fd0dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.446+0000 7f1f6419f740 1 -- 172.21.15.70:0/23912812 shutdown_connections 2024-06-04T05:06:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.446+0000 7f1f6419f740 1 -- 172.21.15.70:0/23912812 wait complete. 2024-06-04T05:06:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.446+0000 7f1f6419f740 1 -- start start 2024-06-04T05:06:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.446+0000 7f1f6419f740 1 Processor -- start 2024-06-04T05:06:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f6419f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd0b820 0x55728fd1ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f6419f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd1b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f6419f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd16a00 0x55728fd90490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f6419f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c1afaa0 con 0x55728fd0c140 2024-06-04T05:06:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f6419f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c1af3d0 con 0x55728fd16a00 2024-06-04T05:06:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f6419f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c1ac630 con 0x55728fd0b820 2024-06-04T05:06:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f6419f740 10 client.0 _ll_register_callbacks cb 0x7f1f500e6380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f5a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd1b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f5a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd1b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53040/0 (socket says 172.21.15.70:53040) 2024-06-04T05:06:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f5a7fc640 1 -- 172.21.15.70:0/939255120 learned_addr learned my addr 172.21.15.70:0/939255120 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f5b7fe640 1 --2- 172.21.15.70:0/939255120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd16a00 0x55728fd90490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f5affd640 1 --2- 172.21.15.70:0/939255120 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd0b820 0x55728fd1ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f5a7fc640 1 -- 172.21.15.70:0/939255120 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd0b820 msgr2=0x55728fd1ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f5a7fc640 1 --2- 172.21.15.70:0/939255120 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd0b820 0x55728fd1ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f5a7fc640 1 -- 172.21.15.70:0/939255120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd16a00 msgr2=0x55728fd90490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f5a7fc640 1 --2- 172.21.15.70:0/939255120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd16a00 0x55728fd90490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f5a7fc640 1 -- 172.21.15.70:0/939255120 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55728fd0bf70 con 0x55728fd0c140 2024-06-04T05:06:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f5affd640 1 --2- 172.21.15.70:0/939255120 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd0b820 0x55728fd1ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:06:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f5a7fc640 1 --2- 172.21.15.70:0/939255120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd1b280 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1f500a5e50 tx=0x7f1f50009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e528d5fba3514a65 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f3eff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f3eff5640 1 -- 172.21.15.70:0/939255120 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f500a9720 con 0x55728fd0c140 2024-06-04T05:06:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f3eff5640 1 -- 172.21.15.70:0/939255120 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8615+0+0 (secure 0 0 0) 0x7f1f500a5950 con 0x55728fd0c140 2024-06-04T05:06:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.447+0000 7f1f6419f740 1 -- 172.21.15.70:0/939255120 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f54020dc0 con 0x55728fd0c140 2024-06-04T05:06:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.448+0000 7f1f3eff5640 1 -- 172.21.15.70:0/939255120 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1f500a9ba0 con 0x55728fd0c140 2024-06-04T05:06:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.448+0000 7f1f6419f740 10 client.17886 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.448+0000 7f1f6419f740 1 -- 172.21.15.70:0/939255120 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55728fd91b00 con 0x55728fd0c140 2024-06-04T05:06:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.448+0000 7f1f3eff5640 1 -- 172.21.15.70:0/939255120 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f54020dc0 con 0x55728fd0c140 2024-06-04T05:06:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.448+0000 7f1f3eff5640 1 -- 172.21.15.70:0/939255120 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f1f50091040 con 0x55728fd0c140 2024-06-04T05:06:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.448+0000 7f1f6419f740 10 client.17886 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.448+0000 7f1f6419f740 10 client.17886 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:06:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.448+0000 7f1f6419f740 20 client.17886 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.448+0000 7f1f6419f740 1 --2- 172.21.15.70:0/939255120 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55728fd920a0 0x55728fd944c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.448+0000 7f1f6419f740 4 client.17886 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:06:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.448+0000 7f1f6419f740 1 -- 172.21.15.70:0/939255120 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55728fc79360 con 0x55728fd920a0 2024-06-04T05:06:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.448+0000 7f1f5affd640 1 --2- 172.21.15.70:0/939255120 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55728fd920a0 0x55728fd944c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.448+0000 7f1f5affd640 1 --2- 172.21.15.70:0/939255120 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55728fd920a0 0x55728fd944c0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85cb37c4aa4a0115 server_cookie=70e74ab9fe1a83d2 in_seq=0 out_seq=0 2024-06-04T05:06:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.448+0000 7f1f3eff5640 10 client.17886 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:09.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.470+0000 7f1f3eff5640 1 -- 172.21.15.70:0/939255120 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55728fc79360 con 0x55728fd920a0 2024-06-04T05:06:09.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.470+0000 7f1f3eff5640 10 client.17886 handle_command_reply: tid=0 2024-06-04T05:06:09.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.470+0000 7f1f6419f740 1 client.17886 shutdown 2024-06-04T05:06:09.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.470+0000 7f1f6419f740 1 -- 172.21.15.70:0/939255120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 msgr2=0x55728fd1b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:09.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.470+0000 7f1f6419f740 1 --2- 172.21.15.70:0/939255120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd1b280 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1f500a5e50 tx=0x7f1f50009310 comp rx=0 tx=0).stop 2024-06-04T05:06:09.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.470+0000 7f1f6419f740 1 -- 172.21.15.70:0/939255120 shutdown_connections 2024-06-04T05:06:09.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.470+0000 7f1f6419f740 1 --2- 172.21.15.70:0/939255120 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55728fd920a0 0x55728fd944c0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.470+0000 7f1f6419f740 1 --2- 172.21.15.70:0/939255120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55728fd16a00 0x55728fd90490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.470+0000 7f1f6419f740 1 --2- 172.21.15.70:0/939255120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728fd0c140 0x55728fd1b280 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.470+0000 7f1f3eff5640 0 client.17886 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:09.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.470+0000 7f1f6419f740 1 --2- 172.21.15.70:0/939255120 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55728fd0b820 0x55728fd1ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.470+0000 7f1f6419f740 1 -- 172.21.15.70:0/939255120 >> 172.21.15.70:0/939255120 conn(0x55728fd07d30 msgr2=0x55728fd139a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:09.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.470+0000 7f1f6419f740 1 -- 172.21.15.70:0/939255120 shutdown_connections 2024-06-04T05:06:09.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.470+0000 7f1f6419f740 1 -- 172.21.15.70:0/939255120 wait complete. 2024-06-04T05:06:09.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.470+0000 7f1f6419f740 20 client.17886 trim_cache size 0 max 16384 2024-06-04T05:06:09.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:06:09.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:06:09.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:06:09.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.471+0000 7f1f61485640 1 -- 172.21.15.70:0/2125957887 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1f440b2a10 msgr2=0x7f1f440b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:09.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.471+0000 7f1f61485640 1 --2- 172.21.15.70:0/2125957887 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1f440b2a10 0x7f1f440b4ed0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1f54005fd0 tx=0x7f1f54009500 comp rx=0 tx=0).stop 2024-06-04T05:06:09.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.471+0000 7f1f61485640 1 -- 172.21.15.70:0/2125957887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f5c102660 msgr2=0x7f1f5c19a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:09.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.471+0000 7f1f61485640 1 --2- 172.21.15.70:0/2125957887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f5c102660 0x7f1f5c19a3a0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000e9f0 tx=0x7f1f5000edf0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.471+0000 7f1f61485640 1 -- 172.21.15.70:0/2125957887 shutdown_connections 2024-06-04T05:06:09.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.471+0000 7f1f61485640 1 --2- 172.21.15.70:0/2125957887 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1f440b2a10 0x7f1f440b4ed0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.471+0000 7f1f61485640 1 --2- 172.21.15.70:0/2125957887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f5c1068f0 0x7f1f5c1a4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.471+0000 7f1f61485640 1 --2- 172.21.15.70:0/2125957887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f5c102660 0x7f1f5c19a3a0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.471+0000 7f1f61485640 1 --2- 172.21.15.70:0/2125957887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f5c101ca0 0x7f1f5c199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.471+0000 7f1f61485640 1 -- 172.21.15.70:0/2125957887 >> 172.21.15.70:0/2125957887 conn(0x7f1f5c077750 msgr2=0x7f1f5c0ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.471+0000 7f1f61485640 1 -- 172.21.15.70:0/2125957887 shutdown_connections 2024-06-04T05:06:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:09.471+0000 7f1f61485640 1 -- 172.21.15.70:0/2125957887 wait complete. 2024-06-04T05:06:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:09 smithi098 ceph-mon[35569]: pgmap v274: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 656 KiB/s wr, 89 op/s 2024-06-04T05:06:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:09 smithi098 ceph-mon[35569]: osdmap e85: 12 total, 12 up, 12 in 2024-06-04T05:06:09.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:09 smithi070 ceph-mon[32631]: pgmap v274: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 656 KiB/s wr, 89 op/s 2024-06-04T05:06:09.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:09 smithi070 ceph-mon[32631]: osdmap e85: 12 total, 12 up, 12 in 2024-06-04T05:06:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:09 smithi112 ceph-mon[36453]: pgmap v274: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 656 KiB/s wr, 89 op/s 2024-06-04T05:06:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:09 smithi112 ceph-mon[36453]: osdmap e85: 12 total, 12 up, 12 in 2024-06-04T05:06:09.729 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.92 MB/sec execute 302 sec latency 5732.162 ms 2024-06-04T05:06:10.729 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.91 MB/sec execute 303 sec latency 6732.225 ms 2024-06-04T05:06:11.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:11 smithi098 ceph-mon[35569]: pgmap v276: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 788 KiB/s wr, 107 op/s 2024-06-04T05:06:11.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:11 smithi070 ceph-mon[32631]: pgmap v276: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 788 KiB/s wr, 107 op/s 2024-06-04T05:06:11.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:11 smithi112 ceph-mon[36453]: pgmap v276: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 788 KiB/s wr, 107 op/s 2024-06-04T05:06:11.729 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.91 MB/sec execute 304 sec latency 7732.279 ms 2024-06-04T05:06:12.729 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.90 MB/sec execute 305 sec latency 8732.340 ms 2024-06-04T05:06:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:13 smithi098 ceph-mon[35569]: pgmap v277: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 140 KiB/s wr, 46 op/s 2024-06-04T05:06:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:13 smithi098 ceph-mon[35569]: osdmap e86: 12 total, 12 up, 12 in 2024-06-04T05:06:13.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:13 smithi070 ceph-mon[32631]: pgmap v277: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 140 KiB/s wr, 46 op/s 2024-06-04T05:06:13.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:13 smithi070 ceph-mon[32631]: osdmap e86: 12 total, 12 up, 12 in 2024-06-04T05:06:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:13 smithi112 ceph-mon[36453]: pgmap v277: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 140 KiB/s wr, 46 op/s 2024-06-04T05:06:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:13 smithi112 ceph-mon[36453]: osdmap e86: 12 total, 12 up, 12 in 2024-06-04T05:06:13.729 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.89 MB/sec execute 306 sec latency 9732.421 ms 2024-06-04T05:06:14.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.89 MB/sec execute 307 sec latency 10732.507 ms 2024-06-04T05:06:15.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:15 smithi098 ceph-mon[35569]: pgmap v279: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 48 KiB/s wr, 13 op/s 2024-06-04T05:06:15.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:15 smithi070 ceph-mon[32631]: pgmap v279: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 48 KiB/s wr, 13 op/s 2024-06-04T05:06:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:15 smithi112 ceph-mon[36453]: pgmap v279: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 48 KiB/s wr, 13 op/s 2024-06-04T05:06:15.729 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.88 MB/sec execute 308 sec latency 11732.565 ms 2024-06-04T05:06:16.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.87 MB/sec execute 309 sec latency 12732.667 ms 2024-06-04T05:06:17.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:17 smithi098 ceph-mon[35569]: pgmap v280: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 61 KiB/s wr, 21 op/s 2024-06-04T05:06:17.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:17 smithi070 ceph-mon[32631]: pgmap v280: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 61 KiB/s wr, 21 op/s 2024-06-04T05:06:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:17 smithi112 ceph-mon[36453]: pgmap v280: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 61 KiB/s wr, 21 op/s 2024-06-04T05:06:17.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.87 MB/sec execute 310 sec latency 13732.793 ms 2024-06-04T05:06:18.163 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:06:18.422 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:06:18.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.86 MB/sec execute 311 sec latency 14732.833 ms 2024-06-04T05:06:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:19 smithi098 ceph-mon[35569]: pgmap v281: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 50 KiB/s wr, 17 op/s 2024-06-04T05:06:19.568 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:19 smithi070 ceph-mon[32631]: pgmap v281: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 50 KiB/s wr, 17 op/s 2024-06-04T05:06:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:19 smithi112 ceph-mon[36453]: pgmap v281: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 50 KiB/s wr, 17 op/s 2024-06-04T05:06:19.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.86 MB/sec execute 312 sec latency 15732.861 ms 2024-06-04T05:06:20.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.230+0000 7f16993d7700 1 -- 172.21.15.70:0/1327152708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1694102430 msgr2=0x7f16941028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.230+0000 7f16993d7700 1 --2- 172.21.15.70:0/1327152708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1694102430 0x7f16941028b0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f167c009990 tx=0x7f167c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:06:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.230+0000 7f16993d7700 1 -- 172.21.15.70:0/1327152708 shutdown_connections 2024-06-04T05:06:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.230+0000 7f16993d7700 1 --2- 172.21.15.70:0/1327152708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1694102df0 0x7f169410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.230+0000 7f16993d7700 1 --2- 172.21.15.70:0/1327152708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1694108510 0x7f16941088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.230+0000 7f16993d7700 1 --2- 172.21.15.70:0/1327152708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1694102430 0x7f16941028b0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.230+0000 7f16993d7700 1 -- 172.21.15.70:0/1327152708 >> 172.21.15.70:0/1327152708 conn(0x7f16940793b0 msgr2=0x7f16940797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.230+0000 7f16993d7700 1 -- 172.21.15.70:0/1327152708 shutdown_connections 2024-06-04T05:06:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.230+0000 7f16993d7700 1 -- 172.21.15.70:0/1327152708 wait complete. 2024-06-04T05:06:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.230+0000 7f16993d7700 1 Processor -- start 2024-06-04T05:06:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f16993d7700 1 -- start start 2024-06-04T05:06:20.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f16993d7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1694102430 0x7f1694104070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:20.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f16993d7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1694102df0 0x7f16941045b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:20.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f16993d7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1694108510 0x7f1694104af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:20.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f16993d7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1694112490 con 0x7f1694108510 2024-06-04T05:06:20.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f16993d7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1694112310 con 0x7f1694102430 2024-06-04T05:06:20.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f16993d7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1694112610 con 0x7f1694102df0 2024-06-04T05:06:20.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f16937fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1694108510 0x7f1694104af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:20.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f16937fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1694108510 0x7f1694104af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44850/0 (socket says 172.21.15.70:44850) 2024-06-04T05:06:20.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f16937fe700 1 -- 172.21.15.70:0/469638168 learned_addr learned my addr 172.21.15.70:0/469638168 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f1692ffd700 1 --2- 172.21.15.70:0/469638168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1694102430 0x7f1694104070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f16927fc700 1 --2- 172.21.15.70:0/469638168 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1694102df0 0x7f16941045b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f16937fe700 1 -- 172.21.15.70:0/469638168 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1694102df0 msgr2=0x7f16941045b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f16937fe700 1 --2- 172.21.15.70:0/469638168 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1694102df0 0x7f16941045b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f16937fe700 1 -- 172.21.15.70:0/469638168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1694102430 msgr2=0x7f1694104070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f16937fe700 1 --2- 172.21.15.70:0/469638168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1694102430 0x7f1694104070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f16937fe700 1 -- 172.21.15.70:0/469638168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16941a8060 con 0x7f1694108510 2024-06-04T05:06:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.231+0000 7f16937fe700 1 --2- 172.21.15.70:0/469638168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1694108510 0x7f1694104af0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f168400d980 tx=0x7f168400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.232+0000 7f168bfff700 1 -- 172.21.15.70:0/469638168 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1684017070 con 0x7f1694108510 2024-06-04T05:06:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.232+0000 7f16993d7700 1 -- 172.21.15.70:0/469638168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16941a8350 con 0x7f1694108510 2024-06-04T05:06:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.232+0000 7f168bfff700 1 -- 172.21.15.70:0/469638168 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1684005020 con 0x7f1694108510 2024-06-04T05:06:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.232+0000 7f168bfff700 1 -- 172.21.15.70:0/469638168 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1684009990 con 0x7f1694108510 2024-06-04T05:06:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.232+0000 7f16993d7700 1 -- 172.21.15.70:0/469638168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16941a8890 con 0x7f1694108510 2024-06-04T05:06:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.233+0000 7f16993d7700 1 -- 172.21.15.70:0/469638168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f169406a230 con 0x7f1694108510 2024-06-04T05:06:20.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.234+0000 7f168bfff700 1 -- 172.21.15.70:0/469638168 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f1684004a30 con 0x7f1694108510 2024-06-04T05:06:20.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.235+0000 7f168bfff700 1 --2- 172.21.15.70:0/469638168 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f16800b29f0 0x7f16800b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:20.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.235+0000 7f1692ffd700 1 --2- 172.21.15.70:0/469638168 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f16800b29f0 0x7f16800b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:20.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.235+0000 7f168bfff700 1 -- 172.21.15.70:0/469638168 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f16840e2860 con 0x7f1694108510 2024-06-04T05:06:20.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.236+0000 7f1692ffd700 1 --2- 172.21.15.70:0/469638168 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f16800b29f0 0x7f16800b4eb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f168c003f10 tx=0x7f168c009180 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:20.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.239+0000 7f168bfff700 1 -- 172.21.15.70:0/469638168 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1684020020 con 0x7f1694108510 2024-06-04T05:06:20.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.488+0000 7f16993d7700 1 -- 172.21.15.70:0/469638168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f169402e560 con 0x7f1694108510 2024-06-04T05:06:20.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.489+0000 7f168bfff700 1 -- 172.21.15.70:0/469638168 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11929 (secure 0 0 0) 0x7f16840aa210 con 0x7f1694108510 2024-06-04T05:06:20.491 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:06:20.491 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:01:52.807483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:06:20.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.491+0000 7f16993d7700 1 -- 172.21.15.70:0/469638168 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f16800b29f0 msgr2=0x7f16800b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:20.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.491+0000 7f16993d7700 1 --2- 172.21.15.70:0/469638168 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f16800b29f0 0x7f16800b4eb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f168c003f10 tx=0x7f168c009180 comp rx=0 tx=0).stop 2024-06-04T05:06:20.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.491+0000 7f16993d7700 1 -- 172.21.15.70:0/469638168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1694108510 msgr2=0x7f1694104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:20.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.491+0000 7f16993d7700 1 --2- 172.21.15.70:0/469638168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1694108510 0x7f1694104af0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f168400d980 tx=0x7f168400dc90 comp rx=0 tx=0).stop 2024-06-04T05:06:20.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.491+0000 7f16993d7700 1 -- 172.21.15.70:0/469638168 shutdown_connections 2024-06-04T05:06:20.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.491+0000 7f16993d7700 1 --2- 172.21.15.70:0/469638168 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f16800b29f0 0x7f16800b4eb0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:20.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.491+0000 7f16993d7700 1 --2- 172.21.15.70:0/469638168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1694108510 0x7f1694104af0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:20.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.491+0000 7f16993d7700 1 --2- 172.21.15.70:0/469638168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1694102430 0x7f1694104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:20.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.491+0000 7f16993d7700 1 --2- 172.21.15.70:0/469638168 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1694102df0 0x7f16941045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:20.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.491+0000 7f16993d7700 1 -- 172.21.15.70:0/469638168 >> 172.21.15.70:0/469638168 conn(0x7f16940793b0 msgr2=0x7f169406be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:20.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.492+0000 7f16993d7700 1 -- 172.21.15.70:0/469638168 shutdown_connections 2024-06-04T05:06:20.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:20.492+0000 7f16993d7700 1 -- 172.21.15.70:0/469638168 wait complete. 2024-06-04T05:06:20.495 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 32 2024-06-04T05:06:20.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.85 MB/sec execute 313 sec latency 16732.973 ms 2024-06-04T05:06:21.319 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id 24943b9a --release --await' 2024-06-04T05:06:21.319 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id 24943b9a --release --await 2024-06-04T05:06:21.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:21 smithi098 ceph-mon[35569]: pgmap v282: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 49 KiB/s wr, 17 op/s 2024-06-04T05:06:21.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:21 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/469638168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:06:21.560 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:06:21.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:21 smithi070 ceph-mon[32631]: pgmap v282: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 49 KiB/s wr, 17 op/s 2024-06-04T05:06:21.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:21 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/469638168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:06:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:21 smithi112 ceph-mon[36453]: pgmap v282: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 49 KiB/s wr, 17 op/s 2024-06-04T05:06:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:21 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/469638168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:06:21.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.84 MB/sec execute 314 sec latency 17733.090 ms 2024-06-04T05:06:22.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.567+0000 7f0afdf04640 1 -- 172.21.15.70:0/925214646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0af8107940 msgr2=0x7f0af8107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.567+0000 7f0afdf04640 1 --2- 172.21.15.70:0/925214646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0af8107940 0x7f0af8107d20 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8009930 tx=0x7f0ae802a420 comp rx=0 tx=0).stop 2024-06-04T05:06:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.567+0000 7f0afdf04640 1 -- 172.21.15.70:0/925214646 shutdown_connections 2024-06-04T05:06:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.567+0000 7f0afdf04640 1 --2- 172.21.15.70:0/925214646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0af8102340 0x7f0af810e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.567+0000 7f0afdf04640 1 --2- 172.21.15.70:0/925214646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0af8101980 0x7f0af8101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.567+0000 7f0afdf04640 1 --2- 172.21.15.70:0/925214646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0af8107940 0x7f0af8107d20 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.567+0000 7f0afdf04640 1 -- 172.21.15.70:0/925214646 >> 172.21.15.70:0/925214646 conn(0x7f0af8078ad0 msgr2=0x7f0af8078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.567+0000 7f0afdf04640 1 -- 172.21.15.70:0/925214646 shutdown_connections 2024-06-04T05:06:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.567+0000 7f0afdf04640 1 -- 172.21.15.70:0/925214646 wait complete. 2024-06-04T05:06:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.568+0000 7f0afdf04640 1 Processor -- start 2024-06-04T05:06:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.568+0000 7f0afdf04640 1 -- start start 2024-06-04T05:06:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.568+0000 7f0afdf04640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0af8101980 0x7f0af8199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.568+0000 7f0afdf04640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0af8102340 0x7f0af819a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.568+0000 7f0afdf04640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0af8107940 0x7f0af81a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.568+0000 7f0afdf04640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0af8111ab0 con 0x7f0af8102340 2024-06-04T05:06:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.568+0000 7f0afdf04640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0af8111930 con 0x7f0af8107940 2024-06-04T05:06:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.568+0000 7f0afdf04640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0af8111c30 con 0x7f0af8101980 2024-06-04T05:06:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.568+0000 7f0af6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0af8102340 0x7f0af819a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.568+0000 7f0af6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0af8102340 0x7f0af819a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44864/0 (socket says 172.21.15.70:44864) 2024-06-04T05:06:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.568+0000 7f0af6ffd640 1 -- 172.21.15.70:0/3057109721 learned_addr learned my addr 172.21.15.70:0/3057109721 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.568+0000 7f0af77fe640 1 --2- 172.21.15.70:0/3057109721 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0af8101980 0x7f0af8199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.568+0000 7f0af7fff640 1 --2- 172.21.15.70:0/3057109721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0af8107940 0x7f0af81a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.569+0000 7f0af6ffd640 1 -- 172.21.15.70:0/3057109721 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0af8101980 msgr2=0x7f0af8199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.569+0000 7f0af6ffd640 1 --2- 172.21.15.70:0/3057109721 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0af8101980 0x7f0af8199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.569+0000 7f0af6ffd640 1 -- 172.21.15.70:0/3057109721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0af8107940 msgr2=0x7f0af81a4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.569+0000 7f0af6ffd640 1 --2- 172.21.15.70:0/3057109721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0af8107940 0x7f0af81a4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.569+0000 7f0af6ffd640 1 -- 172.21.15.70:0/3057109721 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0af81a54d0 con 0x7f0af8102340 2024-06-04T05:06:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.569+0000 7f0af7fff640 1 --2- 172.21.15.70:0/3057109721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0af8107940 0x7f0af81a4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:06:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.569+0000 7f0af6ffd640 1 --2- 172.21.15.70:0/3057109721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0af8102340 0x7f0af819a450 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f0aec00d770 tx=0x7f0aec00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.569+0000 7f0af4ff9640 1 -- 172.21.15.70:0/3057109721 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec017070 con 0x7f0af8102340 2024-06-04T05:06:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.569+0000 7f0af4ff9640 1 -- 172.21.15.70:0/3057109721 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0aec0054b0 con 0x7f0af8102340 2024-06-04T05:06:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.569+0000 7f0afdf04640 1 -- 172.21.15.70:0/3057109721 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0af81a56a0 con 0x7f0af8102340 2024-06-04T05:06:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.569+0000 7f0af4ff9640 1 -- 172.21.15.70:0/3057109721 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec004c60 con 0x7f0af8102340 2024-06-04T05:06:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.569+0000 7f0afdf04640 1 -- 172.21.15.70:0/3057109721 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0af81a5bf0 con 0x7f0af8102340 2024-06-04T05:06:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.571+0000 7f0b00c1c740 1 Processor -- start 2024-06-04T05:06:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.571+0000 7f0af4ff9640 1 -- 172.21.15.70:0/3057109721 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f0aec004e00 con 0x7f0af8102340 2024-06-04T05:06:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0b00c1c740 1 -- start start 2024-06-04T05:06:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0b00c1c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260d6820 0x55f1260d6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0b00c1c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d7140 0x55f1260e14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0b00c1c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260e3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0b00c1c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f126048c30 con 0x55f1260e1a00 2024-06-04T05:06:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0b00c1c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f1260387d0 con 0x55f1260d7140 2024-06-04T05:06:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0b00c1c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f12603c810 con 0x55f1260d6820 2024-06-04T05:06:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0af6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260e3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0af6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260e3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44872/0 (socket says 172.21.15.70:44872) 2024-06-04T05:06:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0af6ffd640 1 -- 172.21.15.70:0/1103174638 learned_addr learned my addr 172.21.15.70:0/1103174638 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0af7fff640 1 --2- 172.21.15.70:0/1103174638 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260d6820 0x55f1260d6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0af77fe640 1 --2- 172.21.15.70:0/1103174638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d7140 0x55f1260e14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0af6ffd640 1 -- 172.21.15.70:0/1103174638 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260d6820 msgr2=0x55f1260d6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0af6ffd640 1 --2- 172.21.15.70:0/1103174638 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260d6820 0x55f1260d6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0af4ff9640 1 --2- 172.21.15.70:0/3057109721 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0ae00b2a90 0x7f0ae00b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0af6ffd640 1 -- 172.21.15.70:0/1103174638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d7140 msgr2=0x55f1260e14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0af6ffd640 1 --2- 172.21.15.70:0/1103174638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d7140 0x55f1260e14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0af6ffd640 1 -- 172.21.15.70:0/1103174638 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f125d4d210 con 0x55f1260e1a00 2024-06-04T05:06:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0af77fe640 1 --2- 172.21.15.70:0/1103174638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d7140 0x55f1260e14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:06:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.572+0000 7f0af7fff640 1 --2- 172.21.15.70:0/1103174638 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260d6820 0x55f1260d6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.573+0000 7f0af7fff640 1 --2- 172.21.15.70:0/3057109721 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0ae00b2a90 0x7f0ae00b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.573+0000 7f0af6ffd640 1 --2- 172.21.15.70:0/1103174638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260e3df0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f0aec0df6f0 tx=0x7f0aec0ba9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16704191ee0102df server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.573+0000 7f0af4ff9640 1 -- 172.21.15.70:0/3057109721 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f0aec0e2060 con 0x7f0af8102340 2024-06-04T05:06:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.573+0000 7f0addffb640 1 -- 172.21.15.70:0/1103174638 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec020070 con 0x55f1260e1a00 2024-06-04T05:06:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.573+0000 7f0addffb640 1 -- 172.21.15.70:0/1103174638 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0aec0a8e20 con 0x55f1260e1a00 2024-06-04T05:06:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.573+0000 7f0addffb640 1 -- 172.21.15.70:0/1103174638 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec0a8c60 con 0x55f1260e1a00 2024-06-04T05:06:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.574+0000 7f0b00c1c740 1 -- 172.21.15.70:0/1103174638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 msgr2=0x55f1260e3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.574+0000 7f0af7fff640 1 --2- 172.21.15.70:0/3057109721 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0ae00b2a90 0x7f0ae00b4f50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0af0009fd0 tx=0x7f0af0009680 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.574+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/1103174638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260e3df0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f0aec0df6f0 tx=0x7f0aec0ba9c0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.574+0000 7f0b00c1c740 1 -- 172.21.15.70:0/1103174638 shutdown_connections 2024-06-04T05:06:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.574+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/1103174638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d7140 0x55f1260e14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.574+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/1103174638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260e3df0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.574+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/1103174638 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260d6820 0x55f1260d6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.574+0000 7f0b00c1c740 1 -- 172.21.15.70:0/1103174638 >> 172.21.15.70:0/1103174638 conn(0x55f1260d2d30 msgr2=0x55f1260d3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.574+0000 7f0b00c1c740 1 -- 172.21.15.70:0/1103174638 shutdown_connections 2024-06-04T05:06:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.574+0000 7f0b00c1c740 1 -- 172.21.15.70:0/1103174638 wait complete. 2024-06-04T05:06:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.574+0000 7f0b00c1c740 1 -- start start 2024-06-04T05:06:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.574+0000 7f0b00c1c740 1 Processor -- start 2024-06-04T05:06:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0b00c1c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260d6820 0x55f1260dea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0b00c1c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d7140 0x55f1260defa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0b00c1c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260e1a00 0x55f1260df4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0b00c1c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f125d1e790 con 0x55f1260d6820 2024-06-04T05:06:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0b00c1c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f125ff4890 con 0x55f1260d7140 2024-06-04T05:06:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0b00c1c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f1260e45d0 con 0x55f1260e1a00 2024-06-04T05:06:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0b00c1c740 10 client.0 _ll_register_callbacks cb 0x55f125d4ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0af77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260d6820 0x55f1260dea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0af77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260d6820 0x55f1260dea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44886/0 (socket says 172.21.15.70:44886) 2024-06-04T05:06:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0af77fe640 1 -- 172.21.15.70:0/2587513015 learned_addr learned my addr 172.21.15.70:0/2587513015 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0af7fff640 1 --2- 172.21.15.70:0/2587513015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260e1a00 0x55f1260df4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0af6ffd640 1 --2- 172.21.15.70:0/2587513015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d7140 0x55f1260defa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0af77fe640 1 -- 172.21.15.70:0/2587513015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260e1a00 msgr2=0x55f1260df4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0af77fe640 1 --2- 172.21.15.70:0/2587513015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260e1a00 0x55f1260df4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0af77fe640 1 -- 172.21.15.70:0/2587513015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d7140 msgr2=0x55f1260defa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0af77fe640 1 --2- 172.21.15.70:0/2587513015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d7140 0x55f1260defa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0af77fe640 1 -- 172.21.15.70:0/2587513015 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f125e19620 con 0x55f1260d6820 2024-06-04T05:06:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0af77fe640 1 --2- 172.21.15.70:0/2587513015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260d6820 0x55f1260dea60 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8006010 tx=0x7f0ae8031ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37212d6be9d62111 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0ad6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0ad6ffd640 1 -- 172.21.15.70:0/2587513015 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae8035070 con 0x55f1260d6820 2024-06-04T05:06:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0ad6ffd640 1 -- 172.21.15.70:0/2587513015 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f0ae8004040 con 0x55f1260d6820 2024-06-04T05:06:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.575+0000 7f0b00c1c740 1 -- 172.21.15.70:0/2587513015 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0aec0a8e20 con 0x55f1260d6820 2024-06-04T05:06:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.576+0000 7f0ad6ffd640 1 -- 172.21.15.70:0/2587513015 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0ae8004240 con 0x55f1260d6820 2024-06-04T05:06:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.576+0000 7f0b00c1c740 10 client.17904 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.576+0000 7f0b00c1c740 1 -- 172.21.15.70:0/2587513015 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f1260f0400 con 0x55f1260d6820 2024-06-04T05:06:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.576+0000 7f0ad6ffd640 1 -- 172.21.15.70:0/2587513015 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0aec0a8e20 con 0x55f1260d6820 2024-06-04T05:06:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.576+0000 7f0ad6ffd640 1 -- 172.21.15.70:0/2587513015 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f0ae8043020 con 0x55f1260d6820 2024-06-04T05:06:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.576+0000 7f0b00c1c740 10 client.17904 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.576+0000 7f0b00c1c740 10 client.17904 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:06:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.576+0000 7f0b00c1c740 20 client.17904 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.576+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/2587513015 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f1260f18c0 0x55f1260f3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.576+0000 7f0b00c1c740 4 client.17904 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:06:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.576+0000 7f0b00c1c740 1 -- 172.21.15.70:0/2587513015 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f12604ab80 con 0x55f1260f18c0 2024-06-04T05:06:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.576+0000 7f0af6ffd640 1 --2- 172.21.15.70:0/2587513015 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f1260f18c0 0x55f1260f3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.577+0000 7f0af6ffd640 1 --2- 172.21.15.70:0/2587513015 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f1260f18c0 0x55f1260f3cb0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3e2bd620677b0a1 server_cookie=c53faf909b44546 in_seq=0 out_seq=0 2024-06-04T05:06:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.577+0000 7f0ad6ffd640 10 client.17904 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.577+0000 7f0ad6ffd640 1 -- 172.21.15.70:0/2587513015 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f12604ab80 con 0x55f1260f18c0 2024-06-04T05:06:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.577+0000 7f0ad6ffd640 10 client.17904 handle_command_reply: tid=0 2024-06-04T05:06:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.577+0000 7f0b00c1c740 1 client.17904 shutdown 2024-06-04T05:06:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.578+0000 7f0b00c1c740 1 -- 172.21.15.70:0/2587513015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260d6820 msgr2=0x55f1260dea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.578+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/2587513015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260d6820 0x55f1260dea60 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8006010 tx=0x7f0ae8031ef0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.578+0000 7f0b00c1c740 1 -- 172.21.15.70:0/2587513015 shutdown_connections 2024-06-04T05:06:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.578+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/2587513015 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f1260f18c0 0x55f1260f3cb0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.578+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/2587513015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d7140 0x55f1260defa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.578+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/2587513015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260e1a00 0x55f1260df4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.578+0000 7f0ad6ffd640 0 client.17904 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.578+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/2587513015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260d6820 0x55f1260dea60 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.578+0000 7f0b00c1c740 1 -- 172.21.15.70:0/2587513015 >> 172.21.15.70:0/2587513015 conn(0x55f1260d2d30 msgr2=0x55f1260d1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.578+0000 7f0b00c1c740 1 -- 172.21.15.70:0/2587513015 shutdown_connections 2024-06-04T05:06:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.578+0000 7f0b00c1c740 1 -- 172.21.15.70:0/2587513015 wait complete. 2024-06-04T05:06:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.578+0000 7f0b00c1c740 20 client.17904 trim_cache size 0 max 16384 2024-06-04T05:06:22.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0b00c1c740 1 Processor -- start 2024-06-04T05:06:22.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0b00c1c740 1 -- start start 2024-06-04T05:06:22.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0b00c1c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d6820 0x55f1260d91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0b00c1c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260d7140 0x55f1260df300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0b00c1c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260f9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0b00c1c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0af81ae610 con 0x55f1260e1a00 2024-06-04T05:06:22.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0b00c1c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0af81b0150 con 0x55f1260d6820 2024-06-04T05:06:22.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0b00c1c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0af81aece0 con 0x55f1260d7140 2024-06-04T05:06:22.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0af7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260f9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0af7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260f9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44890/0 (socket says 172.21.15.70:44890) 2024-06-04T05:06:22.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0af7fff640 1 -- 172.21.15.70:0/104876684 learned_addr learned my addr 172.21.15.70:0/104876684 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:22.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0af6ffd640 1 --2- 172.21.15.70:0/104876684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260d7140 0x55f1260df300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0af77fe640 1 --2- 172.21.15.70:0/104876684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d6820 0x55f1260d91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0af7fff640 1 -- 172.21.15.70:0/104876684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260d7140 msgr2=0x55f1260df300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:22.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0af7fff640 1 --2- 172.21.15.70:0/104876684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260d7140 0x55f1260df300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0af7fff640 1 -- 172.21.15.70:0/104876684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d6820 msgr2=0x55f1260d91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:22.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0af7fff640 1 --2- 172.21.15.70:0/104876684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d6820 0x55f1260d91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0af7fff640 1 -- 172.21.15.70:0/104876684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1260dda10 con 0x55f1260e1a00 2024-06-04T05:06:22.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.593+0000 7f0af6ffd640 1 --2- 172.21.15.70:0/104876684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260d7140 0x55f1260df300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:06:22.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.594+0000 7f0af7fff640 1 --2- 172.21.15.70:0/104876684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260f9580 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f0af00128c0 tx=0x7f0af0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d61a35bd4d2a291 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:22.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.594+0000 7f0add7fa640 1 -- 172.21.15.70:0/104876684 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af000f8c0 con 0x55f1260e1a00 2024-06-04T05:06:22.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.594+0000 7f0add7fa640 1 -- 172.21.15.70:0/104876684 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0af0027470 con 0x55f1260e1a00 2024-06-04T05:06:22.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.594+0000 7f0add7fa640 1 -- 172.21.15.70:0/104876684 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af001a670 con 0x55f1260e1a00 2024-06-04T05:06:22.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.594+0000 7f0b00c1c740 1 -- 172.21.15.70:0/104876684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 msgr2=0x55f1260f9580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:22.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.594+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/104876684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260f9580 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f0af00128c0 tx=0x7f0af0014000 comp rx=0 tx=0).stop 2024-06-04T05:06:22.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.594+0000 7f0b00c1c740 1 -- 172.21.15.70:0/104876684 shutdown_connections 2024-06-04T05:06:22.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.594+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/104876684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260f9580 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.594+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/104876684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260d7140 0x55f1260df300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.594+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/104876684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d6820 0x55f1260d91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.594+0000 7f0b00c1c740 1 -- 172.21.15.70:0/104876684 >> 172.21.15.70:0/104876684 conn(0x55f1260d2d30 msgr2=0x55f1260eceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:22.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.594+0000 7f0b00c1c740 1 -- 172.21.15.70:0/104876684 shutdown_connections 2024-06-04T05:06:22.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.594+0000 7f0b00c1c740 1 -- 172.21.15.70:0/104876684 wait complete. 2024-06-04T05:06:22.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.595+0000 7f0b00c1c740 1 -- start start 2024-06-04T05:06:22.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.595+0000 7f0b00c1c740 1 Processor -- start 2024-06-04T05:06:22.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.595+0000 7f0b00c1c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d6820 0x55f1260e9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.595+0000 7f0b00c1c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260d7140 0x55f1260e5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.595+0000 7f0b00c1c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260e5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.595+0000 7f0b00c1c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0af81afa80 con 0x55f1260e1a00 2024-06-04T05:06:22.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.595+0000 7f0b00c1c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0af81af3b0 con 0x55f1260d6820 2024-06-04T05:06:22.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.595+0000 7f0b00c1c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0af81ac610 con 0x55f1260d7140 2024-06-04T05:06:22.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.595+0000 7f0b00c1c740 10 client.0 _ll_register_callbacks cb 0x7f0ae8005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:22.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.595+0000 7f0af7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260e5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.595+0000 7f0af7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260e5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44898/0 (socket says 172.21.15.70:44898) 2024-06-04T05:06:22.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.595+0000 7f0af7fff640 1 -- 172.21.15.70:0/2347964231 learned_addr learned my addr 172.21.15.70:0/2347964231 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:22.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0af6ffd640 1 --2- 172.21.15.70:0/2347964231 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260d7140 0x55f1260e5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0af77fe640 1 --2- 172.21.15.70:0/2347964231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d6820 0x55f1260e9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0af7fff640 1 -- 172.21.15.70:0/2347964231 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260d7140 msgr2=0x55f1260e5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:22.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0af7fff640 1 --2- 172.21.15.70:0/2347964231 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260d7140 0x55f1260e5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0af7fff640 1 -- 172.21.15.70:0/2347964231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d6820 msgr2=0x55f1260e9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:22.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0af7fff640 1 --2- 172.21.15.70:0/2347964231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d6820 0x55f1260e9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0af7fff640 1 -- 172.21.15.70:0/2347964231 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1260e6040 con 0x55f1260e1a00 2024-06-04T05:06:22.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0af77fe640 1 --2- 172.21.15.70:0/2347964231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d6820 0x55f1260e9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:06:22.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0af7fff640 1 --2- 172.21.15.70:0/2347964231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260e5860 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0af0013f80 tx=0x7f0af0027bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13c75bb0bbb6ba2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:22.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0ad6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0ad6ffd640 1 -- 172.21.15.70:0/2347964231 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af00278f0 con 0x55f1260e1a00 2024-06-04T05:06:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0ad6ffd640 1 -- 172.21.15.70:0/2347964231 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f0af000f8c0 con 0x55f1260e1a00 2024-06-04T05:06:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0b00c1c740 1 -- 172.21.15.70:0/2347964231 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0af0027470 con 0x55f1260e1a00 2024-06-04T05:06:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0ad6ffd640 1 -- 172.21.15.70:0/2347964231 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0af000fb00 con 0x55f1260e1a00 2024-06-04T05:06:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0b00c1c740 10 client.17910 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0b00c1c740 1 -- 172.21.15.70:0/2347964231 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f12615c470 con 0x55f1260e1a00 2024-06-04T05:06:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0ad6ffd640 1 -- 172.21.15.70:0/2347964231 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0af0027470 con 0x55f1260e1a00 2024-06-04T05:06:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0ad6ffd640 1 -- 172.21.15.70:0/2347964231 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f0af002f070 con 0x55f1260e1a00 2024-06-04T05:06:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.596+0000 7f0b00c1c740 10 client.17910 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.597+0000 7f0b00c1c740 10 client.17910 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:06:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.597+0000 7f0b00c1c740 20 client.17910 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.597+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/2347964231 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f12615cb40 0x55f12615ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.597+0000 7f0b00c1c740 4 client.17910 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:06:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.597+0000 7f0b00c1c740 1 -- 172.21.15.70:0/2347964231 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f126044360 con 0x55f12615cb40 2024-06-04T05:06:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.597+0000 7f0af77fe640 1 --2- 172.21.15.70:0/2347964231 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f12615cb40 0x55f12615ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.597+0000 7f0ad6ffd640 10 client.17910 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.597+0000 7f0af77fe640 1 --2- 172.21.15.70:0/2347964231 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f12615cb40 0x55f12615ef60 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=398e0637503cf0e1 server_cookie=217a8c6e6349876f in_seq=0 out_seq=0 2024-06-04T05:06:22.618 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.615+0000 7f0ad6ffd640 1 -- 172.21.15.70:0/2347964231 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f126044360 con 0x55f12615cb40 2024-06-04T05:06:22.618 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.615+0000 7f0ad6ffd640 10 client.17910 handle_command_reply: tid=0 2024-06-04T05:06:22.619 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.615+0000 7f0b00c1c740 1 client.17910 shutdown 2024-06-04T05:06:22.619 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.615+0000 7f0b00c1c740 1 -- 172.21.15.70:0/2347964231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 msgr2=0x55f1260e5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:22.619 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.615+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/2347964231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260e5860 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0af0013f80 tx=0x7f0af0027bd0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.619 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.615+0000 7f0b00c1c740 1 -- 172.21.15.70:0/2347964231 shutdown_connections 2024-06-04T05:06:22.619 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.615+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/2347964231 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f12615cb40 0x55f12615ef60 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.619 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.615+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/2347964231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1260e1a00 0x55f1260e5860 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.615+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/2347964231 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1260d7140 0x55f1260e5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.615+0000 7f0ad6ffd640 0 client.17910 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:22.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.615+0000 7f0b00c1c740 1 --2- 172.21.15.70:0/2347964231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f1260d6820 0x55f1260e9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.615+0000 7f0b00c1c740 1 -- 172.21.15.70:0/2347964231 >> 172.21.15.70:0/2347964231 conn(0x55f1260d2d30 msgr2=0x55f1260d1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:22.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.615+0000 7f0b00c1c740 1 -- 172.21.15.70:0/2347964231 shutdown_connections 2024-06-04T05:06:22.620 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.615+0000 7f0b00c1c740 1 -- 172.21.15.70:0/2347964231 wait complete. 2024-06-04T05:06:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.615+0000 7f0b00c1c740 20 client.17910 trim_cache size 0 max 16384 2024-06-04T05:06:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:06:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:06:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:06:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.616+0000 7f0afdf04640 1 -- 172.21.15.70:0/3057109721 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0ae00b2a90 msgr2=0x7f0ae00b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.616+0000 7f0afdf04640 1 --2- 172.21.15.70:0/3057109721 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0ae00b2a90 0x7f0ae00b4f50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0af0009fd0 tx=0x7f0af0009680 comp rx=0 tx=0).stop 2024-06-04T05:06:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.616+0000 7f0afdf04640 1 -- 172.21.15.70:0/3057109721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0af8102340 msgr2=0x7f0af819a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.616+0000 7f0afdf04640 1 --2- 172.21.15.70:0/3057109721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0af8102340 0x7f0af819a450 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f0aec00d770 tx=0x7f0aec00dc30 comp rx=0 tx=0).stop 2024-06-04T05:06:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.616+0000 7f0afdf04640 1 -- 172.21.15.70:0/3057109721 shutdown_connections 2024-06-04T05:06:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.616+0000 7f0afdf04640 1 --2- 172.21.15.70:0/3057109721 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0ae00b2a90 0x7f0ae00b4f50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.616+0000 7f0afdf04640 1 --2- 172.21.15.70:0/3057109721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0af8107940 0x7f0af81a4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.616+0000 7f0afdf04640 1 --2- 172.21.15.70:0/3057109721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0af8102340 0x7f0af819a450 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.616+0000 7f0afdf04640 1 --2- 172.21.15.70:0/3057109721 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0af8101980 0x7f0af8199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:22.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.616+0000 7f0afdf04640 1 -- 172.21.15.70:0/3057109721 >> 172.21.15.70:0/3057109721 conn(0x7f0af8078ad0 msgr2=0x7f0af81045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:22.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.616+0000 7f0afdf04640 1 -- 172.21.15.70:0/3057109721 shutdown_connections 2024-06-04T05:06:22.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:22.616+0000 7f0afdf04640 1 -- 172.21.15.70:0/3057109721 wait complete. 2024-06-04T05:06:22.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.84 MB/sec execute 315 sec latency 18733.137 ms 2024-06-04T05:06:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:23 smithi098 ceph-mon[35569]: pgmap v283: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-06-04T05:06:23.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:23 smithi070 ceph-mon[32631]: pgmap v283: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-06-04T05:06:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:23 smithi112 ceph-mon[36453]: pgmap v283: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-06-04T05:06:23.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152548 1.83 MB/sec execute 316 sec latency 19733.247 ms 2024-06-04T05:06:24.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.051+0000 7f19c47e5700 1 -- 172.21.15.70:0/1668052897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19bc06bc90 msgr2=0x7f19bc06c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:24.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.051+0000 7f19c47e5700 1 --2- 172.21.15.70:0/1668052897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19bc06bc90 0x7f19bc06c110 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0099c0 tx=0x7f19ac009cd0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.051+0000 7f19c47e5700 1 -- 172.21.15.70:0/1668052897 shutdown_connections 2024-06-04T05:06:24.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.051+0000 7f19c47e5700 1 --2- 172.21.15.70:0/1668052897 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19bc06c650 0x7f19bc1103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.051+0000 7f19c47e5700 1 --2- 172.21.15.70:0/1668052897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19bc109650 0x7f19bc109a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.051+0000 7f19c47e5700 1 --2- 172.21.15.70:0/1668052897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19bc06bc90 0x7f19bc06c110 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.051+0000 7f19c47e5700 1 -- 172.21.15.70:0/1668052897 >> 172.21.15.70:0/1668052897 conn(0x7f19bc079330 msgr2=0x7f19bc079740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:24.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.051+0000 7f19c47e5700 1 -- 172.21.15.70:0/1668052897 shutdown_connections 2024-06-04T05:06:24.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.051+0000 7f19c47e5700 1 -- 172.21.15.70:0/1668052897 wait complete. 2024-06-04T05:06:24.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.052+0000 7f19c47e5700 1 Processor -- start 2024-06-04T05:06:24.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.052+0000 7f19c47e5700 1 -- start start 2024-06-04T05:06:24.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c47e5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19bc06bc90 0x7f19bc19f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c47e5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19bc06c650 0x7f19bc19fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c47e5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19bc109650 0x7f19bc1997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c47e5700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc113610 con 0x7f19bc06bc90 2024-06-04T05:06:24.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c47e5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc113490 con 0x7f19bc06c650 2024-06-04T05:06:24.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c47e5700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc113790 con 0x7f19bc109650 2024-06-04T05:06:24.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c2581700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19bc06bc90 0x7f19bc19f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c2581700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19bc06bc90 0x7f19bc19f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44912/0 (socket says 172.21.15.70:44912) 2024-06-04T05:06:24.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c2581700 1 -- 172.21.15.70:0/2501459670 learned_addr learned my addr 172.21.15.70:0/2501459670 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:24.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c1d80700 1 --2- 172.21.15.70:0/2501459670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19bc06c650 0x7f19bc19fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c2d82700 1 --2- 172.21.15.70:0/2501459670 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19bc109650 0x7f19bc1997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c2581700 1 -- 172.21.15.70:0/2501459670 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19bc109650 msgr2=0x7f19bc1997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:24.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c2581700 1 --2- 172.21.15.70:0/2501459670 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19bc109650 0x7f19bc1997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c2581700 1 -- 172.21.15.70:0/2501459670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19bc06c650 msgr2=0x7f19bc19fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:24.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c2581700 1 --2- 172.21.15.70:0/2501459670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19bc06c650 0x7f19bc19fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c2581700 1 -- 172.21.15.70:0/2501459670 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19bc199fb0 con 0x7f19bc06bc90 2024-06-04T05:06:24.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c2d82700 1 --2- 172.21.15.70:0/2501459670 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19bc109650 0x7f19bc1997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:06:24.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c2581700 1 --2- 172.21.15.70:0/2501459670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19bc06bc90 0x7f19bc19f650 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f19b800c9b0 tx=0x7f19b800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:24.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19b37fe700 1 -- 172.21.15.70:0/2501459670 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b8007b00 con 0x7f19bc06bc90 2024-06-04T05:06:24.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c47e5700 1 -- 172.21.15.70:0/2501459670 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19bc19a2a0 con 0x7f19bc06bc90 2024-06-04T05:06:24.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19b37fe700 1 -- 172.21.15.70:0/2501459670 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f19b8007ca0 con 0x7f19bc06bc90 2024-06-04T05:06:24.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19b37fe700 1 -- 172.21.15.70:0/2501459670 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b8018660 con 0x7f19bc06bc90 2024-06-04T05:06:24.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.053+0000 7f19c47e5700 1 -- 172.21.15.70:0/2501459670 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19bc1a64b0 con 0x7f19bc06bc90 2024-06-04T05:06:24.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.056+0000 7f19b37fe700 1 -- 172.21.15.70:0/2501459670 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f19b800f450 con 0x7f19bc06bc90 2024-06-04T05:06:24.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.056+0000 7f19b37fe700 1 --2- 172.21.15.70:0/2501459670 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f19a80b29f0 0x7f19a80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.056+0000 7f19c1d80700 1 --2- 172.21.15.70:0/2501459670 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f19a80b29f0 0x7f19a80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.057+0000 7f19b37fe700 1 -- 172.21.15.70:0/2501459670 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f19b80e2630 con 0x7f19bc06bc90 2024-06-04T05:06:24.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.057+0000 7f19c1d80700 1 --2- 172.21.15.70:0/2501459670 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f19a80b29f0 0x7f19a80b4eb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f19ac000c00 tx=0x7f19ac005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:24.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.057+0000 7f19cf07ab80 1 Processor -- start 2024-06-04T05:06:24.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.057+0000 7f19cf07ab80 1 -- start start 2024-06-04T05:06:24.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.057+0000 7f19cf07ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb830a52b0 0x55bb830a5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.057+0000 7f19cf07ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb8309f8e0 0x55bb8309fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.057+0000 7f19cf07ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.057+0000 7f19cf07ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bb82ffed90 con 0x55bb830a02a0 2024-06-04T05:06:24.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.057+0000 7f19cf07ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bb82fee930 con 0x55bb830a52b0 2024-06-04T05:06:24.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.057+0000 7f19cf07ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bb82ff2970 con 0x55bb8309f8e0 2024-06-04T05:06:24.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.058+0000 7f19c2d82700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.058+0000 7f19c2d82700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ac0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44922/0 (socket says 172.21.15.70:44922) 2024-06-04T05:06:24.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.058+0000 7f19c2d82700 1 -- 172.21.15.70:0/3806483678 learned_addr learned my addr 172.21.15.70:0/3806483678 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:24.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.058+0000 7f19c2581700 1 --2- 172.21.15.70:0/3806483678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb830a52b0 0x55bb830a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.058+0000 7f19c1d80700 1 --2- 172.21.15.70:0/3806483678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb8309f8e0 0x55bb8309fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.058+0000 7f19c2d82700 1 -- 172.21.15.70:0/3806483678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb8309f8e0 msgr2=0x55bb8309fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:24.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.058+0000 7f19c2d82700 1 --2- 172.21.15.70:0/3806483678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb8309f8e0 0x55bb8309fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.058+0000 7f19c2d82700 1 -- 172.21.15.70:0/3806483678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb830a52b0 msgr2=0x55bb830a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:24.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.058+0000 7f19c2d82700 1 --2- 172.21.15.70:0/3806483678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb830a52b0 0x55bb830a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.058+0000 7f19c2d82700 1 -- 172.21.15.70:0/3806483678 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb82f3e9d0 con 0x55bb830a02a0 2024-06-04T05:06:24.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.058+0000 7f19c2d82700 1 --2- 172.21.15.70:0/3806483678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ac0e0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f19b400d300 tx=0x7f19b400d610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ae6b9a677b28cef server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:24.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.058+0000 7f19b0ff9700 1 -- 172.21.15.70:0/3806483678 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b4016020 con 0x55bb830a02a0 2024-06-04T05:06:24.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.058+0000 7f19b0ff9700 1 -- 172.21.15.70:0/3806483678 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f19b400de40 con 0x55bb830a02a0 2024-06-04T05:06:24.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.058+0000 7f19b0ff9700 1 -- 172.21.15.70:0/3806483678 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b40118e0 con 0x55bb830a02a0 2024-06-04T05:06:24.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.059+0000 7f19cf07ab80 1 -- 172.21.15.70:0/3806483678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 msgr2=0x55bb830ac0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:24.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.059+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/3806483678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ac0e0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f19b400d300 tx=0x7f19b400d610 comp rx=0 tx=0).stop 2024-06-04T05:06:24.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.059+0000 7f19cf07ab80 1 -- 172.21.15.70:0/3806483678 shutdown_connections 2024-06-04T05:06:24.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.059+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/3806483678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ac0e0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.059+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/3806483678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb830a52b0 0x55bb830a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.059+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/3806483678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb8309f8e0 0x55bb8309fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.059+0000 7f19cf07ab80 1 -- 172.21.15.70:0/3806483678 >> 172.21.15.70:0/3806483678 conn(0x55bb82fcd180 msgr2=0x55bb8309a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:24.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.059+0000 7f19cf07ab80 1 -- 172.21.15.70:0/3806483678 shutdown_connections 2024-06-04T05:06:24.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.059+0000 7f19cf07ab80 1 -- 172.21.15.70:0/3806483678 wait complete. 2024-06-04T05:06:24.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.059+0000 7f19cf07ab80 1 -- start start 2024-06-04T05:06:24.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19cf07ab80 1 Processor -- start 2024-06-04T05:06:24.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19cf07ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb8309f8e0 0x55bb830abd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19cf07ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb830a02a0 0x55bb830a9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19cf07ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a52b0 0x55bb830a95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19cf07ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bb82ef50a0 con 0x55bb830a52b0 2024-06-04T05:06:24.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19cf07ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bb82f908c0 con 0x55bb830a02a0 2024-06-04T05:06:24.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19cf07ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bb830ac6e0 con 0x55bb8309f8e0 2024-06-04T05:06:24.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19cf07ab80 10 client.0 _ll_register_callbacks cb 0x55bb82f89a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:24.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19c2d82700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a52b0 0x55bb830a95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19c2d82700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a52b0 0x55bb830a95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44938/0 (socket says 172.21.15.70:44938) 2024-06-04T05:06:24.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19c2d82700 1 -- 172.21.15.70:0/1885622172 learned_addr learned my addr 172.21.15.70:0/1885622172 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:24.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19c2581700 1 --2- 172.21.15.70:0/1885622172 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb8309f8e0 0x55bb830abd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19c1d80700 1 --2- 172.21.15.70:0/1885622172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb830a02a0 0x55bb830a9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19c2d82700 1 -- 172.21.15.70:0/1885622172 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb8309f8e0 msgr2=0x55bb830abd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:24.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19c2d82700 1 --2- 172.21.15.70:0/1885622172 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb8309f8e0 0x55bb830abd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19c2d82700 1 -- 172.21.15.70:0/1885622172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb830a02a0 msgr2=0x55bb830a9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:24.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19c2d82700 1 --2- 172.21.15.70:0/1885622172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb830a02a0 0x55bb830a9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19c2d82700 1 -- 172.21.15.70:0/1885622172 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb82f8d790 con 0x55bb830a52b0 2024-06-04T05:06:24.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19c2581700 1 --2- 172.21.15.70:0/1885622172 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb8309f8e0 0x55bb830abd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:06:24.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19c1d80700 1 --2- 172.21.15.70:0/1885622172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb830a02a0 0x55bb830a9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:06:24.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19c2d82700 1 --2- 172.21.15.70:0/1885622172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a52b0 0x55bb830a95c0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f19b400d2d0 tx=0x7f19b4012c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ab729a2b35ad68 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:24.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.060+0000 7f19a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:24.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.061+0000 7f19a5ffb700 1 -- 172.21.15.70:0/1885622172 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b4016030 con 0x55bb830a52b0 2024-06-04T05:06:24.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.061+0000 7f19cf07ab80 1 -- 172.21.15.70:0/1885622172 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bb8306bae0 con 0x55bb830a52b0 2024-06-04T05:06:24.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.061+0000 7f19a5ffb700 1 -- 172.21.15.70:0/1885622172 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f19b4004c90 con 0x55bb830a52b0 2024-06-04T05:06:24.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.061+0000 7f19a5ffb700 1 -- 172.21.15.70:0/1885622172 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f19b4011500 con 0x55bb830a52b0 2024-06-04T05:06:24.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.061+0000 7f19cf07ab80 10 client.17922 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:24.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.061+0000 7f19cf07ab80 1 -- 172.21.15.70:0/1885622172 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bb830b6df0 con 0x55bb830a52b0 2024-06-04T05:06:24.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.061+0000 7f19a5ffb700 1 -- 172.21.15.70:0/1885622172 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55bb8306bae0 con 0x55bb830a52b0 2024-06-04T05:06:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.061+0000 7f19a5ffb700 1 -- 172.21.15.70:0/1885622172 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f19b4023070 con 0x55bb830a52b0 2024-06-04T05:06:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.061+0000 7f19cf07ab80 10 client.17922 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.061+0000 7f19cf07ab80 10 client.17922 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:06:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.061+0000 7f19cf07ab80 20 client.17922 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.061+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/1885622172 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bb830b92c0 0x55bb830bb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.061+0000 7f19cf07ab80 4 client.17922 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:06:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.061+0000 7f19cf07ab80 1 -- 172.21.15.70:0/1885622172 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb83000ce0 con 0x55bb830b92c0 2024-06-04T05:06:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.062+0000 7f19c2581700 1 --2- 172.21.15.70:0/1885622172 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bb830b92c0 0x55bb830bb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.062+0000 7f19a5ffb700 10 client.17922 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.062+0000 7f19c2581700 1 --2- 172.21.15.70:0/1885622172 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bb830b92c0 0x55bb830bb6b0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1973b2398fcf5b4 server_cookie=93e091d02989f95a in_seq=0 out_seq=0 2024-06-04T05:06:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.063+0000 7f19a5ffb700 1 -- 172.21.15.70:0/1885622172 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bb83000ce0 con 0x55bb830b92c0 2024-06-04T05:06:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.063+0000 7f19a5ffb700 10 client.17922 handle_command_reply: tid=0 2024-06-04T05:06:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.063+0000 7f19cf07ab80 1 client.17922 shutdown 2024-06-04T05:06:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.063+0000 7f19cf07ab80 1 -- 172.21.15.70:0/1885622172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a52b0 msgr2=0x55bb830a95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:24.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.063+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/1885622172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a52b0 0x55bb830a95c0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f19b400d2d0 tx=0x7f19b4012c30 comp rx=0 tx=0).stop 2024-06-04T05:06:24.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.063+0000 7f19cf07ab80 1 -- 172.21.15.70:0/1885622172 shutdown_connections 2024-06-04T05:06:24.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.063+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/1885622172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a52b0 0x55bb830a95c0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.063+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/1885622172 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb8309f8e0 0x55bb830abd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.063+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/1885622172 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bb830b92c0 0x55bb830bb6b0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.063+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/1885622172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb830a02a0 0x55bb830a9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.063+0000 7f19cf07ab80 1 -- 172.21.15.70:0/1885622172 >> 172.21.15.70:0/1885622172 conn(0x55bb82fcd180 msgr2=0x55bb83094ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:24.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.063+0000 7f19a5ffb700 0 client.17922 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:24.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.063+0000 7f19cf07ab80 1 -- 172.21.15.70:0/1885622172 shutdown_connections 2024-06-04T05:06:24.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.063+0000 7f19cf07ab80 1 -- 172.21.15.70:0/1885622172 wait complete. 2024-06-04T05:06:24.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.063+0000 7f19cf07ab80 20 client.17922 trim_cache size 0 max 16384 2024-06-04T05:06:24.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.083+0000 7f19cf07ab80 1 Processor -- start 2024-06-04T05:06:24.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.083+0000 7f19cf07ab80 1 -- start start 2024-06-04T05:06:24.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.083+0000 7f19cf07ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb8309f8e0 0x55bb830a4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.083+0000 7f19cf07ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19cf07ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb830a52b0 0x55bb830c0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19cf07ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1add00 con 0x55bb830a02a0 2024-06-04T05:06:24.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19cf07ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1af840 con 0x55bb8309f8e0 2024-06-04T05:06:24.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19cf07ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1ae3d0 con 0x55bb830a52b0 2024-06-04T05:06:24.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19c1d80700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19c1d80700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ab8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44948/0 (socket says 172.21.15.70:44948) 2024-06-04T05:06:24.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19c1d80700 1 -- 172.21.15.70:0/2745614604 learned_addr learned my addr 172.21.15.70:0/2745614604 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:24.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19c2581700 1 --2- 172.21.15.70:0/2745614604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb8309f8e0 0x55bb830a4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19c2d82700 1 --2- 172.21.15.70:0/2745614604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb830a52b0 0x55bb830c0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19c1d80700 1 -- 172.21.15.70:0/2745614604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb830a52b0 msgr2=0x55bb830c0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:24.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19c1d80700 1 --2- 172.21.15.70:0/2745614604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb830a52b0 0x55bb830c0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19c1d80700 1 -- 172.21.15.70:0/2745614604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb8309f8e0 msgr2=0x55bb830a4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:24.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19c1d80700 1 --2- 172.21.15.70:0/2745614604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb8309f8e0 0x55bb830a4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19c1d80700 1 -- 172.21.15.70:0/2745614604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb830a75f0 con 0x55bb830a02a0 2024-06-04T05:06:24.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19c2d82700 1 --2- 172.21.15.70:0/2745614604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb830a52b0 0x55bb830c0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:06:24.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19c1d80700 1 --2- 172.21.15.70:0/2745614604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ab8c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0093d0 tx=0x7f19ac005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3501aadc2ceaf2e9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:24.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19a7fff700 1 -- 172.21.15.70:0/2745614604 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ac034070 con 0x55bb830a02a0 2024-06-04T05:06:24.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19a7fff700 1 -- 172.21.15.70:0/2745614604 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f19ac003b10 con 0x55bb830a02a0 2024-06-04T05:06:24.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.084+0000 7f19a7fff700 1 -- 172.21.15.70:0/2745614604 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ac00f9c0 con 0x55bb830a02a0 2024-06-04T05:06:24.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.085+0000 7f19cf07ab80 1 -- 172.21.15.70:0/2745614604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 msgr2=0x55bb830ab8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:24.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.085+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/2745614604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ab8c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0093d0 tx=0x7f19ac005000 comp rx=0 tx=0).stop 2024-06-04T05:06:24.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.085+0000 7f19cf07ab80 1 -- 172.21.15.70:0/2745614604 shutdown_connections 2024-06-04T05:06:24.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.085+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/2745614604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb830a52b0 0x55bb830c0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.085+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/2745614604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb8309f8e0 0x55bb830a4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.085+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/2745614604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ab8c0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.085+0000 7f19cf07ab80 1 -- 172.21.15.70:0/2745614604 >> 172.21.15.70:0/2745614604 conn(0x55bb82fcd180 msgr2=0x55bb830a3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:24.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.085+0000 7f19cf07ab80 1 -- 172.21.15.70:0/2745614604 shutdown_connections 2024-06-04T05:06:24.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.085+0000 7f19cf07ab80 1 -- 172.21.15.70:0/2745614604 wait complete. 2024-06-04T05:06:24.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.085+0000 7f19cf07ab80 1 -- start start 2024-06-04T05:06:24.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.085+0000 7f19cf07ab80 1 Processor -- start 2024-06-04T05:06:24.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19cf07ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb8309f8e0 0x55bb830b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19cf07ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ae940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19cf07ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb830a52b0 0x55bb830aef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19cf07ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1af170 con 0x55bb830a02a0 2024-06-04T05:06:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19cf07ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1aeaa0 con 0x55bb830a52b0 2024-06-04T05:06:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19cf07ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1abcc0 con 0x55bb8309f8e0 2024-06-04T05:06:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19cf07ab80 10 client.0 _ll_register_callbacks cb 0x55bb82f89a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19c1d80700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ae940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19c1d80700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ae940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44962/0 (socket says 172.21.15.70:44962) 2024-06-04T05:06:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19c1d80700 1 -- 172.21.15.70:0/3338560524 learned_addr learned my addr 172.21.15.70:0/3338560524 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19c2581700 1 --2- 172.21.15.70:0/3338560524 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb8309f8e0 0x55bb830b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19c2d82700 1 --2- 172.21.15.70:0/3338560524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb830a52b0 0x55bb830aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19c1d80700 1 -- 172.21.15.70:0/3338560524 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb8309f8e0 msgr2=0x55bb830b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:24.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19c1d80700 1 --2- 172.21.15.70:0/3338560524 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb8309f8e0 0x55bb830b2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19c1d80700 1 -- 172.21.15.70:0/3338560524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb830a52b0 msgr2=0x55bb830aef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:24.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19c1d80700 1 --2- 172.21.15.70:0/3338560524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb830a52b0 0x55bb830aef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19c1d80700 1 -- 172.21.15.70:0/3338560524 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb830af740 con 0x55bb830a02a0 2024-06-04T05:06:24.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19c2d82700 1 --2- 172.21.15.70:0/3338560524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb830a52b0 0x55bb830aef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:06:24.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19c1d80700 1 --2- 172.21.15.70:0/3338560524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ae940 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f19ac032040 tx=0x7f19ac003940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc8c92aa6fe474e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:24.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:24.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19a5ffb700 1 -- 172.21.15.70:0/3338560524 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ac034040 con 0x55bb830a02a0 2024-06-04T05:06:24.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19cf07ab80 1 -- 172.21.15.70:0/3338560524 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19ac003b10 con 0x55bb830a02a0 2024-06-04T05:06:24.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.086+0000 7f19a5ffb700 1 -- 172.21.15.70:0/3338560524 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f19ac00f7e0 con 0x55bb830a02a0 2024-06-04T05:06:24.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.087+0000 7f19a5ffb700 1 -- 172.21.15.70:0/3338560524 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f19ac046050 con 0x55bb830a02a0 2024-06-04T05:06:24.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.087+0000 7f19cf07ab80 10 client.17928 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:24.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.087+0000 7f19cf07ab80 1 -- 172.21.15.70:0/3338560524 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bb83124ca0 con 0x55bb830a02a0 2024-06-04T05:06:24.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.087+0000 7f19a5ffb700 1 -- 172.21.15.70:0/3338560524 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f19ac003b10 con 0x55bb830a02a0 2024-06-04T05:06:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.087+0000 7f19a5ffb700 1 -- 172.21.15.70:0/3338560524 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f19ac038a60 con 0x55bb830a02a0 2024-06-04T05:06:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.087+0000 7f19cf07ab80 10 client.17928 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.087+0000 7f19cf07ab80 10 client.17928 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:06:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.087+0000 7f19cf07ab80 20 client.17928 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.087+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/3338560524 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bb83125230 0x55bb83127620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.087+0000 7f19cf07ab80 4 client.17928 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "24943b9a", "release": true, "await": true}] 2024-06-04T05:06:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.087+0000 7f19cf07ab80 1 -- 172.21.15.70:0/3338560524 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "24943b9a", "release": true, "await": true}) -- 0x55bb82ffa4c0 con 0x55bb83125230 2024-06-04T05:06:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.087+0000 7f19c2581700 1 --2- 172.21.15.70:0/3338560524 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bb83125230 0x55bb83127620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.088+0000 7f19a5ffb700 10 client.17928 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.088+0000 7f19c2581700 1 --2- 172.21.15.70:0/3338560524 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bb83125230 0x55bb83127620 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ed49612b240825d server_cookie=3942d66641092b2a in_seq=0 out_seq=0 2024-06-04T05:06:24.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.094+0000 7f19a5ffb700 1 -- 172.21.15.70:0/3338560524 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55bb82ffa4c0 con 0x55bb83125230 2024-06-04T05:06:24.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.094+0000 7f19a5ffb700 10 client.17928 handle_command_reply: tid=0 2024-06-04T05:06:24.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.094+0000 7f19cf07ab80 1 client.17928 shutdown 2024-06-04T05:06:24.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.094+0000 7f19cf07ab80 1 -- 172.21.15.70:0/3338560524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 msgr2=0x55bb830ae940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:24.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.094+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/3338560524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ae940 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f19ac032040 tx=0x7f19ac003940 comp rx=0 tx=0).stop 2024-06-04T05:06:24.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.094+0000 7f19cf07ab80 1 -- 172.21.15.70:0/3338560524 shutdown_connections 2024-06-04T05:06:24.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.094+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/3338560524 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bb83125230 0x55bb83127620 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.094+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/3338560524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb830a52b0 0x55bb830aef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.094+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/3338560524 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb8309f8e0 0x55bb830b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.094+0000 7f19cf07ab80 1 --2- 172.21.15.70:0/3338560524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb830a02a0 0x55bb830ae940 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.094+0000 7f19cf07ab80 1 -- 172.21.15.70:0/3338560524 >> 172.21.15.70:0/3338560524 conn(0x55bb82fcd180 msgr2=0x55bb8309e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:24.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.094+0000 7f19a5ffb700 0 client.17928 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:24.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.094+0000 7f19cf07ab80 1 -- 172.21.15.70:0/3338560524 shutdown_connections 2024-06-04T05:06:24.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.094+0000 7f19cf07ab80 1 -- 172.21.15.70:0/3338560524 wait complete. 2024-06-04T05:06:24.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.094+0000 7f19cf07ab80 20 client.17928 trim_cache size 0 max 16384 2024-06-04T05:06:24.098 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:06:24.098 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 32, 2024-06-04T05:06:24.098 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:06:24.098 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 28, 2024-06-04T05:06:24.098 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:06:24.098 INFO:teuthology.orchestra.run.smithi070.stdout: "24943b9a": { 2024-06-04T05:06:24.098 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 28, 2024-06-04T05:06:24.098 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:06:24.099 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:06:24.099 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:06:24.099 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:06:24.099 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:06:24.099 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:06:24.099 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 143.6, 2024-06-04T05:06:24.099 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:06:24.099 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:06:24.099 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:06:24.099 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:06:24.099 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:06:24.099 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:06:24.099 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:06:24.099 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:06:24.099 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:06:24.099 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:06:24.100 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:06:24.100 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:06:24.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.095+0000 7f19c47e5700 1 -- 172.21.15.70:0/2501459670 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f19a80b29f0 msgr2=0x7f19a80b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:24.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.095+0000 7f19c47e5700 1 --2- 172.21.15.70:0/2501459670 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f19a80b29f0 0x7f19a80b4eb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f19ac000c00 tx=0x7f19ac005fb0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.095+0000 7f19c47e5700 1 -- 172.21.15.70:0/2501459670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19bc06bc90 msgr2=0x7f19bc19f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:24.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.095+0000 7f19c47e5700 1 --2- 172.21.15.70:0/2501459670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19bc06bc90 0x7f19bc19f650 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f19b800c9b0 tx=0x7f19b800ccc0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.095+0000 7f19c47e5700 1 -- 172.21.15.70:0/2501459670 shutdown_connections 2024-06-04T05:06:24.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.095+0000 7f19c47e5700 1 --2- 172.21.15.70:0/2501459670 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f19a80b29f0 0x7f19a80b4eb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.095+0000 7f19c47e5700 1 --2- 172.21.15.70:0/2501459670 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19bc109650 0x7f19bc1997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.095+0000 7f19c47e5700 1 --2- 172.21.15.70:0/2501459670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19bc06bc90 0x7f19bc19f650 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.095+0000 7f19c47e5700 1 --2- 172.21.15.70:0/2501459670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19bc06c650 0x7f19bc19fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:24.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.095+0000 7f19c47e5700 1 -- 172.21.15.70:0/2501459670 >> 172.21.15.70:0/2501459670 conn(0x7f19bc079330 msgr2=0x7f19bc1011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:24.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.095+0000 7f19c47e5700 1 -- 172.21.15.70:0/2501459670 shutdown_connections 2024-06-04T05:06:24.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:24.095+0000 7f19c47e5700 1 -- 172.21.15.70:0/2501459670 wait complete. 2024-06-04T05:06:24.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 152801 1.83 MB/sec execute 317 sec latency 20104.636 ms 2024-06-04T05:06:24.992 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '24943b9a', total seconds elapsed: 30.4 2024-06-04T05:06:25.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:25 smithi098 ceph-mon[35569]: pgmap v284: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-06-04T05:06:25.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:25 smithi070 ceph-mon[32631]: pgmap v284: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-06-04T05:06:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:25 smithi112 ceph-mon[36453]: pgmap v284: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-06-04T05:06:25.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 153089 1.83 MB/sec execute 318 sec latency 11.468 ms 2024-06-04T05:06:26.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 153565 1.82 MB/sec execute 319 sec latency 15.454 ms 2024-06-04T05:06:27.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:27 smithi098 ceph-mon[35569]: pgmap v285: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 266 KiB/s wr, 40 op/s 2024-06-04T05:06:27.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:27 smithi070 ceph-mon[32631]: pgmap v285: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 266 KiB/s wr, 40 op/s 2024-06-04T05:06:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:27 smithi112 ceph-mon[36453]: pgmap v285: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 266 KiB/s wr, 40 op/s 2024-06-04T05:06:27.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 154109 1.83 MB/sec execute 320 sec latency 17.983 ms 2024-06-04T05:06:28.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 154726 1.83 MB/sec execute 321 sec latency 20.633 ms 2024-06-04T05:06:29.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:29 smithi098 ceph-mon[35569]: pgmap v286: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 257 KiB/s wr, 35 op/s 2024-06-04T05:06:29.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:29 smithi070 ceph-mon[32631]: pgmap v286: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 257 KiB/s wr, 35 op/s 2024-06-04T05:06:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:29 smithi112 ceph-mon[36453]: pgmap v286: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 257 KiB/s wr, 35 op/s 2024-06-04T05:06:29.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 155308 1.84 MB/sec execute 322 sec latency 12.498 ms 2024-06-04T05:06:30.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 155877 1.84 MB/sec execute 323 sec latency 15.477 ms 2024-06-04T05:06:31.488 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:31 smithi112 ceph-mon[36453]: pgmap v287: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 257 KiB/s wr, 35 op/s 2024-06-04T05:06:31.488 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:31 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:06:31.488 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:31 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:06:31.488 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:31 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:06:31.488 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:31 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:06:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:31 smithi098 ceph-mon[35569]: pgmap v287: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 257 KiB/s wr, 35 op/s 2024-06-04T05:06:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:31 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:06:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:31 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:06:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:31 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:06:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:31 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:06:31.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:31 smithi070 ceph-mon[32631]: pgmap v287: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 257 KiB/s wr, 35 op/s 2024-06-04T05:06:31.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:31 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:06:31.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:31 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:06:31.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:31 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:06:31.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:31 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:06:31.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 156354 1.85 MB/sec execute 324 sec latency 15.125 ms 2024-06-04T05:06:32.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 156637 1.85 MB/sec execute 325 sec latency 14.947 ms 2024-06-04T05:06:33.289 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:06:33.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:33 smithi098 ceph-mon[35569]: pgmap v288: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 990 KiB/s wr, 95 op/s 2024-06-04T05:06:33.551 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:06:33.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:33 smithi070 ceph-mon[32631]: pgmap v288: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 990 KiB/s wr, 95 op/s 2024-06-04T05:06:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:33 smithi112 ceph-mon[36453]: pgmap v288: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 990 KiB/s wr, 95 op/s 2024-06-04T05:06:33.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 157096 1.84 MB/sec execute 326 sec latency 17.366 ms 2024-06-04T05:06:34.730 INFO:tasks.workunit.client.0.smithi070.stdout: 1 157657 1.85 MB/sec execute 327 sec latency 20.720 ms 2024-06-04T05:06:35.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.341+0000 7facda79f700 1 -- 172.21.15.70:0/2284369047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd406b220 msgr2=0x7facd406b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.341+0000 7facda79f700 1 --2- 172.21.15.70:0/2284369047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd406b220 0x7facd406b6a0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7facc8009990 tx=0x7facc8009ca0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.342+0000 7facda79f700 1 -- 172.21.15.70:0/2284369047 shutdown_connections 2024-06-04T05:06:35.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.342+0000 7facda79f700 1 --2- 172.21.15.70:0/2284369047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7facd406bbe0 0x7facd41103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.342+0000 7facda79f700 1 --2- 172.21.15.70:0/2284369047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd41095b0 0x7facd4109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.342+0000 7facda79f700 1 --2- 172.21.15.70:0/2284369047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd406b220 0x7facd406b6a0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.342+0000 7facda79f700 1 -- 172.21.15.70:0/2284369047 >> 172.21.15.70:0/2284369047 conn(0x7facd40793f0 msgr2=0x7facd4079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:35.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.342+0000 7facda79f700 1 -- 172.21.15.70:0/2284369047 shutdown_connections 2024-06-04T05:06:35.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.342+0000 7facda79f700 1 -- 172.21.15.70:0/2284369047 wait complete. 2024-06-04T05:06:35.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.342+0000 7facda79f700 1 Processor -- start 2024-06-04T05:06:35.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.342+0000 7facda79f700 1 -- start start 2024-06-04T05:06:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.343+0000 7facda79f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd406b220 0x7facd4102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.343+0000 7facda79f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7facd406bbe0 0x7facd4102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.343+0000 7facda79f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd41095b0 0x7facd4102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.343+0000 7facda79f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7facd4113590 con 0x7facd41095b0 2024-06-04T05:06:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.343+0000 7facda79f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7facd4113410 con 0x7facd406b220 2024-06-04T05:06:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.343+0000 7facda79f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7facd4113710 con 0x7facd406bbe0 2024-06-04T05:06:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.343+0000 7facd8d3c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd41095b0 0x7facd4102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.343+0000 7facd8d3c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd41095b0 0x7facd4102f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46190/0 (socket says 172.21.15.70:46190) 2024-06-04T05:06:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.343+0000 7facd8d3c700 1 -- 172.21.15.70:0/3229772533 learned_addr learned my addr 172.21.15.70:0/3229772533 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.343+0000 7facd37fe700 1 --2- 172.21.15.70:0/3229772533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7facd406bbe0 0x7facd4102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.343+0000 7facd3fff700 1 --2- 172.21.15.70:0/3229772533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd406b220 0x7facd4102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.343+0000 7facd8d3c700 1 -- 172.21.15.70:0/3229772533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7facd406bbe0 msgr2=0x7facd4102a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.343+0000 7facd8d3c700 1 --2- 172.21.15.70:0/3229772533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7facd406bbe0 0x7facd4102a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.343+0000 7facd8d3c700 1 -- 172.21.15.70:0/3229772533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd406b220 msgr2=0x7facd4102510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.343+0000 7facd8d3c700 1 --2- 172.21.15.70:0/3229772533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd406b220 0x7facd4102510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.343+0000 7facd8d3c700 1 -- 172.21.15.70:0/3229772533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd41a8030 con 0x7facd41095b0 2024-06-04T05:06:35.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.343+0000 7facd37fe700 1 --2- 172.21.15.70:0/3229772533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7facd406bbe0 0x7facd4102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:35.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.344+0000 7facd8d3c700 1 --2- 172.21.15.70:0/3229772533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd41095b0 0x7facd4102f90 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7faccc00ef30 tx=0x7faccc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:35.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.344+0000 7facd17fa700 1 -- 172.21.15.70:0/3229772533 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faccc0092e0 con 0x7facd41095b0 2024-06-04T05:06:35.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.344+0000 7facda79f700 1 -- 172.21.15.70:0/3229772533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facd41a82c0 con 0x7facd41095b0 2024-06-04T05:06:35.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.344+0000 7facd17fa700 1 -- 172.21.15.70:0/3229772533 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faccc004bb0 con 0x7facd41095b0 2024-06-04T05:06:35.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.344+0000 7facd17fa700 1 -- 172.21.15.70:0/3229772533 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faccc004020 con 0x7facd41095b0 2024-06-04T05:06:35.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.344+0000 7facda79f700 1 -- 172.21.15.70:0/3229772533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facd41a87d0 con 0x7facd41095b0 2024-06-04T05:06:35.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.346+0000 7facd17fa700 1 -- 172.21.15.70:0/3229772533 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7faccc01b030 con 0x7facd41095b0 2024-06-04T05:06:35.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.347+0000 7facd17fa700 1 --2- 172.21.15.70:0/3229772533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7facbc0b29f0 0x7facbc0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.347+0000 7facd3fff700 1 --2- 172.21.15.70:0/3229772533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7facbc0b29f0 0x7facbc0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.347+0000 7facd17fa700 1 -- 172.21.15.70:0/3229772533 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7faccc0e2100 con 0x7facd41095b0 2024-06-04T05:06:35.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7face5034b80 1 Processor -- start 2024-06-04T05:06:35.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7facd3fff700 1 --2- 172.21.15.70:0/3229772533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7facbc0b29f0 0x7facbc0b4eb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7facc400abd0 tx=0x7facc400a380 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:35.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7face5034b80 1 -- start start 2024-06-04T05:06:35.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7face5034b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab6f860 0x5652cab6fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7face5034b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab70210 0x5652cab73940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7face5034b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab73f60 0x5652cab743a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7face5034b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5652caacec90 con 0x5652cab70210 2024-06-04T05:06:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7face5034b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5652caabe830 con 0x5652cab73f60 2024-06-04T05:06:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7face5034b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5652caac2870 con 0x5652cab6f860 2024-06-04T05:06:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7facd37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab70210 0x5652cab73940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7facd37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab70210 0x5652cab73940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46206/0 (socket says 172.21.15.70:46206) 2024-06-04T05:06:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7facd37fe700 1 -- 172.21.15.70:0/2845172252 learned_addr learned my addr 172.21.15.70:0/2845172252 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7facd3fff700 1 --2- 172.21.15.70:0/2845172252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab6f860 0x5652cab6fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7facd8d3c700 1 --2- 172.21.15.70:0/2845172252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab73f60 0x5652cab743a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7facd37fe700 1 -- 172.21.15.70:0/2845172252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab6f860 msgr2=0x5652cab6fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7facd37fe700 1 --2- 172.21.15.70:0/2845172252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab6f860 0x5652cab6fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7facd37fe700 1 -- 172.21.15.70:0/2845172252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab73f60 msgr2=0x5652cab743a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7facd37fe700 1 --2- 172.21.15.70:0/2845172252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab73f60 0x5652cab743a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7facd37fe700 1 -- 172.21.15.70:0/2845172252 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652caa0e8d0 con 0x5652cab70210 2024-06-04T05:06:35.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.348+0000 7facd8d3c700 1 --2- 172.21.15.70:0/2845172252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab73f60 0x5652cab743a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:06:35.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.349+0000 7facd37fe700 1 --2- 172.21.15.70:0/2845172252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab70210 0x5652cab73940 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7facc8009990 tx=0x7facc8005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607017433c4fad88 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:35.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.349+0000 7facba7fc700 1 -- 172.21.15.70:0/2845172252 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc8027070 con 0x5652cab70210 2024-06-04T05:06:35.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.349+0000 7facba7fc700 1 -- 172.21.15.70:0/2845172252 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7facc8005530 con 0x5652cab70210 2024-06-04T05:06:35.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.349+0000 7facba7fc700 1 -- 172.21.15.70:0/2845172252 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc800f460 con 0x5652cab70210 2024-06-04T05:06:35.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.350+0000 7face5034b80 1 -- 172.21.15.70:0/2845172252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab70210 msgr2=0x5652cab73940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.350+0000 7face5034b80 1 --2- 172.21.15.70:0/2845172252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab70210 0x5652cab73940 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7facc8009990 tx=0x7facc8005fb0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.350+0000 7face5034b80 1 -- 172.21.15.70:0/2845172252 shutdown_connections 2024-06-04T05:06:35.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.350+0000 7face5034b80 1 --2- 172.21.15.70:0/2845172252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab73f60 0x5652cab743a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.350+0000 7face5034b80 1 --2- 172.21.15.70:0/2845172252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab6f860 0x5652cab6fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.350+0000 7face5034b80 1 --2- 172.21.15.70:0/2845172252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab70210 0x5652cab73940 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.350+0000 7face5034b80 1 -- 172.21.15.70:0/2845172252 >> 172.21.15.70:0/2845172252 conn(0x5652caa9d080 msgr2=0x5652cab69e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:35.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.350+0000 7face5034b80 1 -- 172.21.15.70:0/2845172252 shutdown_connections 2024-06-04T05:06:35.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.350+0000 7face5034b80 1 -- 172.21.15.70:0/2845172252 wait complete. 2024-06-04T05:06:35.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.350+0000 7face5034b80 1 -- start start 2024-06-04T05:06:35.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7face5034b80 1 Processor -- start 2024-06-04T05:06:35.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7face5034b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab6f860 0x5652cab7b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7face5034b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab70210 0x5652cab78690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7face5034b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab73f60 0x5652cab78bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7face5034b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5652ca9c4fa0 con 0x5652cab73f60 2024-06-04T05:06:35.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7face5034b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5652caa607c0 con 0x5652cab70210 2024-06-04T05:06:35.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7face5034b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5652cab7c0c0 con 0x5652cab6f860 2024-06-04T05:06:35.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7face5034b80 10 client.0 _ll_register_callbacks cb 0x5652caa59920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:35.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7facd3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab6f860 0x5652cab7b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7facd3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab6f860 0x5652cab7b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51396/0 (socket says 172.21.15.70:51396) 2024-06-04T05:06:35.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7facd3fff700 1 -- 172.21.15.70:0/3996831562 learned_addr learned my addr 172.21.15.70:0/3996831562 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:35.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7facd8d3c700 1 --2- 172.21.15.70:0/3996831562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab73f60 0x5652cab78bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7facd37fe700 1 --2- 172.21.15.70:0/3996831562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab70210 0x5652cab78690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7facd8d3c700 1 -- 172.21.15.70:0/3996831562 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab6f860 msgr2=0x5652cab7b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:35.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7facd8d3c700 1 --2- 172.21.15.70:0/3996831562 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab6f860 0x5652cab7b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7facd8d3c700 1 -- 172.21.15.70:0/3996831562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab70210 msgr2=0x5652cab78690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:35.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7facd8d3c700 1 --2- 172.21.15.70:0/3996831562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab70210 0x5652cab78690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7facd8d3c700 1 -- 172.21.15.70:0/3996831562 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652caa5d690 con 0x5652cab73f60 2024-06-04T05:06:35.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.351+0000 7facd3fff700 1 --2- 172.21.15.70:0/3996831562 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab6f860 0x5652cab7b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:35.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.352+0000 7facd8d3c700 1 --2- 172.21.15.70:0/3996831562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab73f60 0x5652cab78bd0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7faccc0df820 tx=0x7faccc0a9980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d39c5f6633db45 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:35.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.352+0000 7facb37fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:35.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.352+0000 7facb37fe700 1 -- 172.21.15.70:0/3996831562 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faccc0a8f50 con 0x5652cab73f60 2024-06-04T05:06:35.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.352+0000 7facb37fe700 1 -- 172.21.15.70:0/3996831562 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7faccc0a3d70 con 0x5652cab73f60 2024-06-04T05:06:35.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.352+0000 7face5034b80 1 -- 172.21.15.70:0/3996831562 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facc8005530 con 0x5652cab73f60 2024-06-04T05:06:35.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.352+0000 7facb37fe700 1 -- 172.21.15.70:0/3996831562 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7faccc068e60 con 0x5652cab73f60 2024-06-04T05:06:35.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.352+0000 7face5034b80 10 client.17940 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:35.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.352+0000 7face5034b80 1 -- 172.21.15.70:0/3996831562 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5652cab86770 con 0x5652cab73f60 2024-06-04T05:06:35.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.352+0000 7facb37fe700 1 -- 172.21.15.70:0/3996831562 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7facc8005530 con 0x5652cab73f60 2024-06-04T05:06:35.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.352+0000 7facb37fe700 1 -- 172.21.15.70:0/3996831562 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7faccc0b0ae0 con 0x5652cab73f60 2024-06-04T05:06:35.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.353+0000 7face5034b80 10 client.17940 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:35.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.353+0000 7face5034b80 10 client.17940 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:06:35.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.353+0000 7face5034b80 20 client.17940 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:35.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.353+0000 7face5034b80 1 --2- 172.21.15.70:0/3996831562 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5652cab88d50 0x5652cab8b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.353+0000 7face5034b80 4 client.17940 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:06:35.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.353+0000 7face5034b80 1 -- 172.21.15.70:0/3996831562 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652cab3b9e0 con 0x5652cab88d50 2024-06-04T05:06:35.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.353+0000 7facd3fff700 1 --2- 172.21.15.70:0/3996831562 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5652cab88d50 0x5652cab8b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.353+0000 7facb37fe700 10 client.17940 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:35.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.353+0000 7facd3fff700 1 --2- 172.21.15.70:0/3996831562 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5652cab88d50 0x5652cab8b180 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe59436fb353a569 server_cookie=30f6136993114e84 in_seq=0 out_seq=0 2024-06-04T05:06:35.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.354+0000 7facb37fe700 1 -- 172.21.15.70:0/3996831562 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5652cab3b9e0 con 0x5652cab88d50 2024-06-04T05:06:35.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.354+0000 7facb37fe700 10 client.17940 handle_command_reply: tid=0 2024-06-04T05:06:35.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.354+0000 7face5034b80 1 client.17940 shutdown 2024-06-04T05:06:35.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.354+0000 7face5034b80 1 -- 172.21.15.70:0/3996831562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab73f60 msgr2=0x5652cab78bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.354+0000 7face5034b80 1 --2- 172.21.15.70:0/3996831562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab73f60 0x5652cab78bd0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7faccc0df820 tx=0x7faccc0a9980 comp rx=0 tx=0).stop 2024-06-04T05:06:35.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.354+0000 7face5034b80 1 -- 172.21.15.70:0/3996831562 shutdown_connections 2024-06-04T05:06:35.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.354+0000 7face5034b80 1 --2- 172.21.15.70:0/3996831562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab73f60 0x5652cab78bd0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.354+0000 7face5034b80 1 --2- 172.21.15.70:0/3996831562 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab6f860 0x5652cab7b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.354+0000 7face5034b80 1 --2- 172.21.15.70:0/3996831562 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5652cab88d50 0x5652cab8b180 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.354+0000 7face5034b80 1 --2- 172.21.15.70:0/3996831562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab70210 0x5652cab78690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.354+0000 7face5034b80 1 -- 172.21.15.70:0/3996831562 >> 172.21.15.70:0/3996831562 conn(0x5652caa9d080 msgr2=0x5652cab69510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:35.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.354+0000 7facb37fe700 0 client.17940 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:35.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.354+0000 7face5034b80 1 -- 172.21.15.70:0/3996831562 shutdown_connections 2024-06-04T05:06:35.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.354+0000 7face5034b80 1 -- 172.21.15.70:0/3996831562 wait complete. 2024-06-04T05:06:35.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.354+0000 7face5034b80 20 client.17940 trim_cache size 0 max 16384 2024-06-04T05:06:35.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.377+0000 7face5034b80 1 Processor -- start 2024-06-04T05:06:35.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.377+0000 7face5034b80 1 -- start start 2024-06-04T05:06:35.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7face5034b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab6f860 0x5652cab73290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7face5034b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab70210 0x5652cab69c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7face5034b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab73f60 0x5652cab909d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7face5034b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7facd41affb0 con 0x5652cab6f860 2024-06-04T05:06:35.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7face5034b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7facd41b1af0 con 0x5652cab73f60 2024-06-04T05:06:35.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7face5034b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7facd41b0680 con 0x5652cab70210 2024-06-04T05:06:35.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7facd3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab6f860 0x5652cab73290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7facd37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab70210 0x5652cab69c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7facd3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab6f860 0x5652cab73290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46232/0 (socket says 172.21.15.70:46232) 2024-06-04T05:06:35.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7facd3fff700 1 -- 172.21.15.70:0/3978491297 learned_addr learned my addr 172.21.15.70:0/3978491297 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:35.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7facd8d3c700 1 --2- 172.21.15.70:0/3978491297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab73f60 0x5652cab909d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7facd37fe700 1 -- 172.21.15.70:0/3978491297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab73f60 msgr2=0x5652cab909d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:35.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7facd37fe700 1 --2- 172.21.15.70:0/3978491297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab73f60 0x5652cab909d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7facd37fe700 1 -- 172.21.15.70:0/3978491297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab6f860 msgr2=0x5652cab73290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:35.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7facd37fe700 1 --2- 172.21.15.70:0/3978491297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab6f860 0x5652cab73290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7facd37fe700 1 -- 172.21.15.70:0/3978491297 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652cab79040 con 0x5652cab70210 2024-06-04T05:06:35.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7facd8d3c700 1 --2- 172.21.15.70:0/3978491297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab73f60 0x5652cab909d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:06:35.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7facd3fff700 1 --2- 172.21.15.70:0/3978491297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab6f860 0x5652cab73290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:35.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.378+0000 7facd37fe700 1 --2- 172.21.15.70:0/3978491297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab70210 0x5652cab69c90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7facc8009750 tx=0x7facc80058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8babe824577e22dd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:35.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.379+0000 7facb9ffb700 1 -- 172.21.15.70:0/3978491297 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc8027040 con 0x5652cab70210 2024-06-04T05:06:35.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.379+0000 7facb9ffb700 1 -- 172.21.15.70:0/3978491297 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7facc8003680 con 0x5652cab70210 2024-06-04T05:06:35.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.379+0000 7facb9ffb700 1 -- 172.21.15.70:0/3978491297 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc800f6c0 con 0x5652cab70210 2024-06-04T05:06:35.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.379+0000 7face5034b80 1 -- 172.21.15.70:0/3978491297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab70210 msgr2=0x5652cab69c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.379+0000 7face5034b80 1 --2- 172.21.15.70:0/3978491297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab70210 0x5652cab69c90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7facc8009750 tx=0x7facc80058e0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.379+0000 7face5034b80 1 -- 172.21.15.70:0/3978491297 shutdown_connections 2024-06-04T05:06:35.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.379+0000 7face5034b80 1 --2- 172.21.15.70:0/3978491297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab73f60 0x5652cab909d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.379+0000 7face5034b80 1 --2- 172.21.15.70:0/3978491297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab6f860 0x5652cab73290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.379+0000 7face5034b80 1 --2- 172.21.15.70:0/3978491297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab70210 0x5652cab69c90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.379+0000 7face5034b80 1 -- 172.21.15.70:0/3978491297 >> 172.21.15.70:0/3978491297 conn(0x5652caa9d080 msgr2=0x5652cab7a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:35.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.379+0000 7face5034b80 1 -- 172.21.15.70:0/3978491297 shutdown_connections 2024-06-04T05:06:35.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.379+0000 7face5034b80 1 -- 172.21.15.70:0/3978491297 wait complete. 2024-06-04T05:06:35.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.380+0000 7face5034b80 1 -- start start 2024-06-04T05:06:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.380+0000 7face5034b80 1 Processor -- start 2024-06-04T05:06:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.380+0000 7face5034b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab6f860 0x5652cab81300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.380+0000 7face5034b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab70210 0x5652cab81840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.380+0000 7face5034b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab73f60 0x5652cab7f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.380+0000 7face5034b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7facd41b1420 con 0x5652cab6f860 2024-06-04T05:06:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.380+0000 7face5034b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7facd41b0d50 con 0x5652cab73f60 2024-06-04T05:06:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.380+0000 7face5034b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7facd41adf70 con 0x5652cab70210 2024-06-04T05:06:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7facd37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab70210 0x5652cab81840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7face5034b80 10 client.0 _ll_register_callbacks cb 0x5652caa59920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7facd37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab70210 0x5652cab81840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51410/0 (socket says 172.21.15.70:51410) 2024-06-04T05:06:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7facd37fe700 1 -- 172.21.15.70:0/638660237 learned_addr learned my addr 172.21.15.70:0/638660237 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7facd3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab6f860 0x5652cab81300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7facd8d3c700 1 --2- 172.21.15.70:0/638660237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab73f60 0x5652cab7f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7facd3fff700 1 -- 172.21.15.70:0/638660237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab70210 msgr2=0x5652cab81840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7facd3fff700 1 --2- 172.21.15.70:0/638660237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab70210 0x5652cab81840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7facd3fff700 1 -- 172.21.15.70:0/638660237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab73f60 msgr2=0x5652cab7f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7facd3fff700 1 --2- 172.21.15.70:0/638660237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab73f60 0x5652cab7f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7facd3fff700 1 -- 172.21.15.70:0/638660237 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652cab7ffd0 con 0x5652cab6f860 2024-06-04T05:06:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7facd37fe700 1 --2- 172.21.15.70:0/638660237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab70210 0x5652cab81840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7facd3fff700 1 --2- 172.21.15.70:0/638660237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab6f860 0x5652cab81300 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7facc400f750 tx=0x7facc40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f031e41ff62c19c4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7facb37fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7facb37fe700 1 -- 172.21.15.70:0/638660237 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc4016070 con 0x5652cab6f860 2024-06-04T05:06:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7face5034b80 1 -- 172.21.15.70:0/638660237 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facc8003680 con 0x5652cab6f860 2024-06-04T05:06:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7facb37fe700 1 -- 172.21.15.70:0/638660237 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7facc4029610 con 0x5652cab6f860 2024-06-04T05:06:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7facb37fe700 1 -- 172.21.15.70:0/638660237 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7facc4039050 con 0x5652cab6f860 2024-06-04T05:06:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7face5034b80 10 client.17946 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.381+0000 7face5034b80 1 -- 172.21.15.70:0/638660237 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5652cabf43f0 con 0x5652cab6f860 2024-06-04T05:06:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.382+0000 7facb37fe700 1 -- 172.21.15.70:0/638660237 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7facc8003680 con 0x5652cab6f860 2024-06-04T05:06:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.382+0000 7facb37fe700 1 -- 172.21.15.70:0/638660237 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7facc4010ca0 con 0x5652cab6f860 2024-06-04T05:06:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.382+0000 7face5034b80 10 client.17946 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.382+0000 7face5034b80 10 client.17946 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:06:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.382+0000 7face5034b80 20 client.17946 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.382+0000 7face5034b80 1 --2- 172.21.15.70:0/638660237 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5652cabf4b60 0x5652cabf6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.382+0000 7face5034b80 4 client.17946 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:06:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.382+0000 7face5034b80 1 -- 172.21.15.70:0/638660237 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5652caad0be0 con 0x5652cabf4b60 2024-06-04T05:06:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.382+0000 7facd37fe700 1 --2- 172.21.15.70:0/638660237 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5652cabf4b60 0x5652cabf6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.383+0000 7facb37fe700 10 client.17946 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.383+0000 7facd37fe700 1 --2- 172.21.15.70:0/638660237 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5652cabf4b60 0x5652cabf6f80 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f18484cba0464f32 server_cookie=bc253c357098eba in_seq=0 out_seq=0 2024-06-04T05:06:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.383+0000 7facb37fe700 1 -- 172.21.15.70:0/638660237 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5652caad0be0 con 0x5652cabf4b60 2024-06-04T05:06:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.383+0000 7facb37fe700 10 client.17946 handle_command_reply: tid=0 2024-06-04T05:06:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.383+0000 7face5034b80 1 client.17946 shutdown 2024-06-04T05:06:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.383+0000 7face5034b80 1 -- 172.21.15.70:0/638660237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab6f860 msgr2=0x5652cab81300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.383+0000 7face5034b80 1 --2- 172.21.15.70:0/638660237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab6f860 0x5652cab81300 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7facc400f750 tx=0x7facc40047c0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.383+0000 7face5034b80 1 -- 172.21.15.70:0/638660237 shutdown_connections 2024-06-04T05:06:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.383+0000 7face5034b80 1 --2- 172.21.15.70:0/638660237 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5652cabf4b60 0x5652cabf6f80 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.383+0000 7face5034b80 1 --2- 172.21.15.70:0/638660237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652cab73f60 0x5652cab7f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.383+0000 7face5034b80 1 --2- 172.21.15.70:0/638660237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652cab6f860 0x5652cab81300 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.383+0000 7face5034b80 1 --2- 172.21.15.70:0/638660237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652cab70210 0x5652cab81840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.383+0000 7face5034b80 1 -- 172.21.15.70:0/638660237 >> 172.21.15.70:0/638660237 conn(0x5652caa9d080 msgr2=0x5652cab84150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.383+0000 7facb37fe700 0 client.17946 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.383+0000 7face5034b80 1 -- 172.21.15.70:0/638660237 shutdown_connections 2024-06-04T05:06:35.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.383+0000 7face5034b80 1 -- 172.21.15.70:0/638660237 wait complete. 2024-06-04T05:06:35.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.383+0000 7face5034b80 20 client.17946 trim_cache size 0 max 16384 2024-06-04T05:06:35.391 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:06:35.391 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:06:35.391 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:06:35.391 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:06:35.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.384+0000 7facda79f700 1 -- 172.21.15.70:0/3229772533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7facbc0b29f0 msgr2=0x7facbc0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.384+0000 7facda79f700 1 --2- 172.21.15.70:0/3229772533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7facbc0b29f0 0x7facbc0b4eb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7facc400abd0 tx=0x7facc400a380 comp rx=0 tx=0).stop 2024-06-04T05:06:35.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.384+0000 7facda79f700 1 -- 172.21.15.70:0/3229772533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd41095b0 msgr2=0x7facd4102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.384+0000 7facda79f700 1 --2- 172.21.15.70:0/3229772533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd41095b0 0x7facd4102f90 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7faccc00ef30 tx=0x7faccc00c3a0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.384+0000 7facda79f700 1 -- 172.21.15.70:0/3229772533 shutdown_connections 2024-06-04T05:06:35.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.384+0000 7facda79f700 1 --2- 172.21.15.70:0/3229772533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7facbc0b29f0 0x7facbc0b4eb0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.384+0000 7facda79f700 1 --2- 172.21.15.70:0/3229772533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facd41095b0 0x7facd4102f90 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.384+0000 7facda79f700 1 --2- 172.21.15.70:0/3229772533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd406b220 0x7facd4102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.384+0000 7facda79f700 1 --2- 172.21.15.70:0/3229772533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7facd406bbe0 0x7facd4102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.384+0000 7facda79f700 1 -- 172.21.15.70:0/3229772533 >> 172.21.15.70:0/3229772533 conn(0x7facd40793f0 msgr2=0x7facd4101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.384+0000 7facda79f700 1 -- 172.21.15.70:0/3229772533 shutdown_connections 2024-06-04T05:06:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:35.384+0000 7facda79f700 1 -- 172.21.15.70:0/3229772533 wait complete. 2024-06-04T05:06:35.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:35 smithi098 ceph-mon[35569]: pgmap v289: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 987 KiB/s wr, 90 op/s 2024-06-04T05:06:35.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:35 smithi070 ceph-mon[32631]: pgmap v289: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 987 KiB/s wr, 90 op/s 2024-06-04T05:06:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:35 smithi112 ceph-mon[36453]: pgmap v289: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 987 KiB/s wr, 90 op/s 2024-06-04T05:06:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.717+0000 7fdc2c21d640 1 -- 172.21.15.70:0/278939797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc24107970 msgr2=0x7fdc24107d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.717+0000 7fdc2c21d640 1 --2- 172.21.15.70:0/278939797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc24107970 0x7fdc24107d50 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009930 tx=0x7fdc2002a420 comp rx=0 tx=0).stop 2024-06-04T05:06:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.717+0000 7fdc2c21d640 1 -- 172.21.15.70:0/278939797 shutdown_connections 2024-06-04T05:06:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.717+0000 7fdc2c21d640 1 --2- 172.21.15.70:0/278939797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc24102320 0x7fdc2410e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.717+0000 7fdc2c21d640 1 --2- 172.21.15.70:0/278939797 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc24101960 0x7fdc24101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.717+0000 7fdc2c21d640 1 --2- 172.21.15.70:0/278939797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc24107970 0x7fdc24107d50 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.717+0000 7fdc2c21d640 1 -- 172.21.15.70:0/278939797 >> 172.21.15.70:0/278939797 conn(0x7fdc24078b60 msgr2=0x7fdc24078f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.718+0000 7fdc2c21d640 1 -- 172.21.15.70:0/278939797 shutdown_connections 2024-06-04T05:06:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.718+0000 7fdc2c21d640 1 -- 172.21.15.70:0/278939797 wait complete. 2024-06-04T05:06:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.718+0000 7fdc2c21d640 1 Processor -- start 2024-06-04T05:06:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.718+0000 7fdc2c21d640 1 -- start start 2024-06-04T05:06:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.718+0000 7fdc2c21d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc24101960 0x7fdc24199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.718+0000 7fdc2c21d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc24102320 0x7fdc2419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.718+0000 7fdc2c21d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc24107970 0x7fdc241a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.718+0000 7fdc2c21d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc24111ae0 con 0x7fdc24102320 2024-06-04T05:06:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.718+0000 7fdc2c21d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc24111960 con 0x7fdc24107970 2024-06-04T05:06:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.718+0000 7fdc2c21d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdc24111c60 con 0x7fdc24101960 2024-06-04T05:06:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.719+0000 7fdc29fb9640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc24101960 0x7fdc24199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.719+0000 7fdc297b8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc24102320 0x7fdc2419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.719+0000 7fdc29fb9640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc24101960 0x7fdc24199f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51432/0 (socket says 172.21.15.70:51432) 2024-06-04T05:06:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.719+0000 7fdc29fb9640 1 -- 172.21.15.70:0/1447385779 learned_addr learned my addr 172.21.15.70:0/1447385779 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.719+0000 7fdc2a7ba640 1 --2- 172.21.15.70:0/1447385779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc24107970 0x7fdc241a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.719+0000 7fdc29fb9640 1 -- 172.21.15.70:0/1447385779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc24107970 msgr2=0x7fdc241a5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.719+0000 7fdc29fb9640 1 --2- 172.21.15.70:0/1447385779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc24107970 0x7fdc241a5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.719+0000 7fdc29fb9640 1 -- 172.21.15.70:0/1447385779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc24102320 msgr2=0x7fdc2419a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.719+0000 7fdc29fb9640 1 --2- 172.21.15.70:0/1447385779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc24102320 0x7fdc2419a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.719+0000 7fdc29fb9640 1 -- 172.21.15.70:0/1447385779 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc241a5540 con 0x7fdc24101960 2024-06-04T05:06:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.719+0000 7fdc297b8640 1 --2- 172.21.15.70:0/1447385779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc24102320 0x7fdc2419a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.719+0000 7fdc29fb9640 1 --2- 172.21.15.70:0/1447385779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc24101960 0x7fdc24199f80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdc20005940 tx=0x7fdc20004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.719+0000 7fdc1affd640 1 -- 172.21.15.70:0/1447385779 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2003f030 con 0x7fdc24101960 2024-06-04T05:06:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.719+0000 7fdc1affd640 1 -- 172.21.15.70:0/1447385779 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc20004440 con 0x7fdc24101960 2024-06-04T05:06:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.719+0000 7fdc2c21d640 1 -- 172.21.15.70:0/1447385779 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc241a5710 con 0x7fdc24101960 2024-06-04T05:06:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.719+0000 7fdc1affd640 1 -- 172.21.15.70:0/1447385779 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc20035070 con 0x7fdc24101960 2024-06-04T05:06:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.719+0000 7fdc2c21d640 1 -- 172.21.15.70:0/1447385779 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc241a5b70 con 0x7fdc24101960 2024-06-04T05:06:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.722+0000 7fdc1affd640 1 -- 172.21.15.70:0/1447385779 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fdc20042030 con 0x7fdc24101960 2024-06-04T05:06:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.722+0000 7fdc2f010740 1 Processor -- start 2024-06-04T05:06:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.722+0000 7fdc2f010740 1 -- start start 2024-06-04T05:06:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.722+0000 7fdc2f010740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943d820 0x55f47943dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.722+0000 7fdc2f010740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943e140 0x55f4794484c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.722+0000 7fdc2f010740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 0x55f47944adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.722+0000 7fdc2f010740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f4793afc30 con 0x55f479448a00 2024-06-04T05:06:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.722+0000 7fdc2f010740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f47939f7d0 con 0x55f47943e140 2024-06-04T05:06:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.722+0000 7fdc2f010740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f4793a3810 con 0x55f47943d820 2024-06-04T05:06:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc2a7ba640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943d820 0x55f47943dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc2a7ba640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943d820 0x55f47943dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51438/0 (socket says 172.21.15.70:51438) 2024-06-04T05:06:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc2a7ba640 1 -- 172.21.15.70:0/126255283 learned_addr learned my addr 172.21.15.70:0/126255283 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc297b8640 1 --2- 172.21.15.70:0/126255283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 0x55f47944adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc29fb9640 1 --2- 172.21.15.70:0/126255283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943e140 0x55f4794484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc2a7ba640 1 -- 172.21.15.70:0/126255283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943e140 msgr2=0x55f4794484c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc2a7ba640 1 --2- 172.21.15.70:0/126255283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943e140 0x55f4794484c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc2a7ba640 1 -- 172.21.15.70:0/126255283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 msgr2=0x55f47944adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc2a7ba640 1 --2- 172.21.15.70:0/126255283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 0x55f47944adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc2a7ba640 1 -- 172.21.15.70:0/126255283 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4790b4210 con 0x55f47943d820 2024-06-04T05:06:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc297b8640 1 --2- 172.21.15.70:0/126255283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 0x55f47944adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:06:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc1affd640 1 --2- 172.21.15.70:0/1447385779 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdc100b2a10 0x7fdc100b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc29fb9640 1 --2- 172.21.15.70:0/1447385779 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdc100b2a10 0x7fdc100b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc2a7ba640 1 --2- 172.21.15.70:0/126255283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943d820 0x55f47943dc00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009f50 tx=0x7fdc1c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d55ab8e698dad177 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc1affd640 1 -- 172.21.15.70:0/1447385779 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7fdc201029e0 con 0x7fdc24101960 2024-06-04T05:06:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc0ffff640 1 -- 172.21.15.70:0/126255283 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c004d20 con 0x55f47943d820 2024-06-04T05:06:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc0ffff640 1 -- 172.21.15.70:0/126255283 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc1c004050 con 0x55f47943d820 2024-06-04T05:06:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.723+0000 7fdc0ffff640 1 -- 172.21.15.70:0/126255283 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c005260 con 0x55f47943d820 2024-06-04T05:06:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.724+0000 7fdc2f010740 1 -- 172.21.15.70:0/126255283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943d820 msgr2=0x55f47943dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.724+0000 7fdc29fb9640 1 --2- 172.21.15.70:0/1447385779 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdc100b2a10 0x7fdc100b4ed0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdc201001b0 tx=0x7fdc20107040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.724+0000 7fdc2f010740 1 --2- 172.21.15.70:0/126255283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943d820 0x55f47943dc00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009f50 tx=0x7fdc1c0118d0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.724+0000 7fdc2f010740 1 -- 172.21.15.70:0/126255283 shutdown_connections 2024-06-04T05:06:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.724+0000 7fdc2f010740 1 --2- 172.21.15.70:0/126255283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943e140 0x55f4794484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.724+0000 7fdc2f010740 1 --2- 172.21.15.70:0/126255283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 0x55f47944adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.724+0000 7fdc2f010740 1 --2- 172.21.15.70:0/126255283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943d820 0x55f47943dc00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.724+0000 7fdc2f010740 1 -- 172.21.15.70:0/126255283 >> 172.21.15.70:0/126255283 conn(0x55f479439d30 msgr2=0x55f47943a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.724+0000 7fdc2f010740 1 -- 172.21.15.70:0/126255283 shutdown_connections 2024-06-04T05:06:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.724+0000 7fdc2f010740 1 -- 172.21.15.70:0/126255283 wait complete. 2024-06-04T05:06:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.725+0000 7fdc2f010740 1 -- start start 2024-06-04T05:06:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.725+0000 7fdc2f010740 1 Processor -- start 2024-06-04T05:06:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.725+0000 7fdc2f010740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943d820 0x55f479452920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.725+0000 7fdc2f010740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47943e140 0x55f479452e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.725+0000 7fdc2f010740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f479448a00 0x55f47944cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.725+0000 7fdc2f010740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f479085790 con 0x55f47943e140 2024-06-04T05:06:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.725+0000 7fdc2f010740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f47935b890 con 0x55f47943d820 2024-06-04T05:06:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.725+0000 7fdc2f010740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f47944b630 con 0x55f479448a00 2024-06-04T05:06:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.725+0000 7fdc2f010740 10 client.0 _ll_register_callbacks cb 0x55f4790b3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.725+0000 7fdc2a7ba640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f479448a00 0x55f47944cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.725+0000 7fdc2a7ba640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f479448a00 0x55f47944cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51444/0 (socket says 172.21.15.70:51444) 2024-06-04T05:06:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.725+0000 7fdc2a7ba640 1 -- 172.21.15.70:0/2076983733 learned_addr learned my addr 172.21.15.70:0/2076983733 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.725+0000 7fdc297b8640 1 --2- 172.21.15.70:0/2076983733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47943e140 0x55f479452e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.725+0000 7fdc2a7ba640 1 --2- 172.21.15.70:0/2076983733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943d820 0x55f479452920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.726+0000 7fdc2a7ba640 1 -- 172.21.15.70:0/2076983733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943d820 msgr2=0x55f479452920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.726+0000 7fdc2a7ba640 1 --2- 172.21.15.70:0/2076983733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943d820 0x55f479452920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.726+0000 7fdc2a7ba640 1 -- 172.21.15.70:0/2076983733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47943e140 msgr2=0x55f479452e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.726+0000 7fdc2a7ba640 1 --2- 172.21.15.70:0/2076983733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47943e140 0x55f479452e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.726+0000 7fdc2a7ba640 1 -- 172.21.15.70:0/2076983733 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f479180620 con 0x55f479448a00 2024-06-04T05:06:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.726+0000 7fdc2a7ba640 1 --2- 172.21.15.70:0/2076983733 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f479448a00 0x55f47944cc90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c0048f0 tx=0x7fdc1c004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5539048ccfd0d5c6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.726+0000 7fdc0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:06:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.726+0000 7fdc0d7fa640 1 -- 172.21.15.70:0/2076983733 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c003c40 con 0x55f479448a00 2024-06-04T05:06:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.726+0000 7fdc0d7fa640 1 -- 172.21.15.70:0/2076983733 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7fdc1c003de0 con 0x55f479448a00 2024-06-04T05:06:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.726+0000 7fdc2f010740 1 -- 172.21.15.70:0/2076983733 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc1c004050 con 0x55f479448a00 2024-06-04T05:06:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.726+0000 7fdc0d7fa640 1 -- 172.21.15.70:0/2076983733 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdc1c036050 con 0x55f479448a00 2024-06-04T05:06:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.726+0000 7fdc2f010740 10 client.27178 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.726+0000 7fdc2f010740 1 -- 172.21.15.70:0/2076983733 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f47945b820 con 0x55f479448a00 2024-06-04T05:06:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.727+0000 7fdc0d7fa640 1 -- 172.21.15.70:0/2076983733 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc1c004050 con 0x55f479448a00 2024-06-04T05:06:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.727+0000 7fdc0d7fa640 1 -- 172.21.15.70:0/2076983733 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fdc1c00f480 con 0x55f479448a00 2024-06-04T05:06:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.727+0000 7fdc2f010740 10 client.27178 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.727+0000 7fdc2f010740 10 client.27178 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:06:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.727+0000 7fdc2f010740 20 client.27178 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.727+0000 7fdc2f010740 1 --2- 172.21.15.70:0/2076983733 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f47945cc80 0x55f47945f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.727+0000 7fdc2f010740 4 client.27178 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:06:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.727+0000 7fdc2f010740 1 -- 172.21.15.70:0/2076983733 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4793b1b80 con 0x55f47945cc80 2024-06-04T05:06:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.727+0000 7fdc297b8640 1 --2- 172.21.15.70:0/2076983733 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f47945cc80 0x55f47945f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.727+0000 7fdc0d7fa640 10 client.27178 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.727+0000 7fdc297b8640 1 --2- 172.21.15.70:0/2076983733 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f47945cc80 0x55f47945f070 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15a88fa20439c2b2 server_cookie=b1ff4fa69991fd66 in_seq=0 out_seq=0 2024-06-04T05:06:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.728+0000 7fdc0d7fa640 1 -- 172.21.15.70:0/2076983733 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f4793b1b80 con 0x55f47945cc80 2024-06-04T05:06:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.728+0000 7fdc0d7fa640 10 client.27178 handle_command_reply: tid=0 2024-06-04T05:06:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.728+0000 7fdc2f010740 1 client.27178 shutdown 2024-06-04T05:06:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.728+0000 7fdc2f010740 1 -- 172.21.15.70:0/2076983733 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f479448a00 msgr2=0x55f47944cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.728+0000 7fdc2f010740 1 --2- 172.21.15.70:0/2076983733 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f479448a00 0x55f47944cc90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c0048f0 tx=0x7fdc1c004950 comp rx=0 tx=0).stop 2024-06-04T05:06:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.728+0000 7fdc2f010740 1 -- 172.21.15.70:0/2076983733 shutdown_connections 2024-06-04T05:06:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.728+0000 7fdc2f010740 1 --2- 172.21.15.70:0/2076983733 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f47945cc80 0x55f47945f070 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.728+0000 7fdc2f010740 1 --2- 172.21.15.70:0/2076983733 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f479448a00 0x55f47944cc90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.728+0000 7fdc2f010740 1 --2- 172.21.15.70:0/2076983733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47943e140 0x55f479452e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.728+0000 7fdc0d7fa640 0 client.27178 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.728+0000 7fdc2f010740 1 --2- 172.21.15.70:0/2076983733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943d820 0x55f479452920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.728+0000 7fdc2f010740 1 -- 172.21.15.70:0/2076983733 >> 172.21.15.70:0/2076983733 conn(0x55f479439d30 msgr2=0x55f479438fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:35.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 158236 1.85 MB/sec execute 328 sec latency 22.815 ms 2024-06-04T05:06:35.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.728+0000 7fdc2f010740 1 -- 172.21.15.70:0/2076983733 shutdown_connections 2024-06-04T05:06:35.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.729+0000 7fdc2f010740 1 -- 172.21.15.70:0/2076983733 wait complete. 2024-06-04T05:06:35.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.729+0000 7fdc2f010740 20 client.27178 trim_cache size 0 max 16384 2024-06-04T05:06:35.752 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.749+0000 7fdc2f010740 1 Processor -- start 2024-06-04T05:06:35.752 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.749+0000 7fdc2f010740 1 -- start start 2024-06-04T05:06:35.752 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.750+0000 7fdc2f010740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943d820 0x55f479452d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.750+0000 7fdc2f010740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943e140 0x55f47943f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.750+0000 7fdc2f010740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 0x55f479447070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.750+0000 7fdc2f010740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc241ac420 con 0x55f479448a00 2024-06-04T05:06:35.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.750+0000 7fdc2f010740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc241adf60 con 0x55f47943e140 2024-06-04T05:06:35.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.750+0000 7fdc2f010740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdc241acaf0 con 0x55f47943d820 2024-06-04T05:06:35.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.750+0000 7fdc2a7ba640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943d820 0x55f479452d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.750+0000 7fdc2a7ba640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943d820 0x55f479452d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51458/0 (socket says 172.21.15.70:51458) 2024-06-04T05:06:35.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.750+0000 7fdc2a7ba640 1 -- 172.21.15.70:0/2553000193 learned_addr learned my addr 172.21.15.70:0/2553000193 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:35.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.750+0000 7fdc2a7ba640 1 --2- 172.21.15.70:0/2553000193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 0x55f479447070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.750+0000 7fdc297b8640 1 --2- 172.21.15.70:0/2553000193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943e140 0x55f47943f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.750+0000 7fdc2a7ba640 1 -- 172.21.15.70:0/2553000193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943e140 msgr2=0x55f47943f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:35.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.750+0000 7fdc2a7ba640 1 --2- 172.21.15.70:0/2553000193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943e140 0x55f47943f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.750+0000 7fdc2a7ba640 1 -- 172.21.15.70:0/2553000193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 msgr2=0x55f479447070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:35.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.750+0000 7fdc2a7ba640 1 --2- 172.21.15.70:0/2553000193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 0x55f479447070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.750+0000 7fdc2a7ba640 1 -- 172.21.15.70:0/2553000193 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f479448360 con 0x55f47943d820 2024-06-04T05:06:35.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.750+0000 7fdc2a7ba640 1 --2- 172.21.15.70:0/2553000193 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943d820 0x55f479452d70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c011ed0 tx=0x7fdc1c009230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd63ab2cff6af596 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.751+0000 7fdc0f7fe640 1 -- 172.21.15.70:0/2553000193 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c004a70 con 0x55f47943d820 2024-06-04T05:06:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.751+0000 7fdc0f7fe640 1 -- 172.21.15.70:0/2553000193 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc1c02d070 con 0x55f47943d820 2024-06-04T05:06:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.751+0000 7fdc0f7fe640 1 -- 172.21.15.70:0/2553000193 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c028c20 con 0x55f47943d820 2024-06-04T05:06:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.752+0000 7fdc2f010740 1 -- 172.21.15.70:0/2553000193 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943d820 msgr2=0x55f479452d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.752+0000 7fdc2f010740 1 --2- 172.21.15.70:0/2553000193 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943d820 0x55f479452d70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c011ed0 tx=0x7fdc1c009230 comp rx=0 tx=0).stop 2024-06-04T05:06:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.752+0000 7fdc2f010740 1 -- 172.21.15.70:0/2553000193 shutdown_connections 2024-06-04T05:06:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.752+0000 7fdc2f010740 1 --2- 172.21.15.70:0/2553000193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943e140 0x55f47943f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.752+0000 7fdc2f010740 1 --2- 172.21.15.70:0/2553000193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 0x55f479447070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.752+0000 7fdc2f010740 1 --2- 172.21.15.70:0/2553000193 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943d820 0x55f479452d70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.752+0000 7fdc2f010740 1 -- 172.21.15.70:0/2553000193 >> 172.21.15.70:0/2553000193 conn(0x55f479439d30 msgr2=0x55f47943f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.752+0000 7fdc2f010740 1 -- 172.21.15.70:0/2553000193 shutdown_connections 2024-06-04T05:06:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.752+0000 7fdc2f010740 1 -- 172.21.15.70:0/2553000193 wait complete. 2024-06-04T05:06:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.752+0000 7fdc2f010740 1 -- start start 2024-06-04T05:06:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.752+0000 7fdc2f010740 1 Processor -- start 2024-06-04T05:06:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.752+0000 7fdc2f010740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943d820 0x55f479456ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.752+0000 7fdc2f010740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943e140 0x55f479457020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.752+0000 7fdc2f010740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 0x55f479457560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.752+0000 7fdc2f010740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc241ad890 con 0x55f479448a00 2024-06-04T05:06:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.752+0000 7fdc2f010740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc241ad1c0 con 0x55f47943d820 2024-06-04T05:06:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.752+0000 7fdc2f010740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdc241aa420 con 0x55f47943e140 2024-06-04T05:06:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.753+0000 7fdc2f010740 10 client.0 _ll_register_callbacks cb 0x7fdc1c009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.753+0000 7fdc2a7ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 0x55f479457560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.753+0000 7fdc2a7ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 0x55f479457560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46308/0 (socket says 172.21.15.70:46308) 2024-06-04T05:06:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.753+0000 7fdc297b8640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943e140 0x55f479457020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.753+0000 7fdc2a7ba640 1 -- 172.21.15.70:0/276877319 learned_addr learned my addr 172.21.15.70:0/276877319 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.753+0000 7fdc2a7ba640 1 --2- 172.21.15.70:0/276877319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943d820 0x55f479456ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.753+0000 7fdc297b8640 1 -- 172.21.15.70:0/276877319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943d820 msgr2=0x55f479456ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.753+0000 7fdc297b8640 1 --2- 172.21.15.70:0/276877319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943d820 0x55f479456ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.753+0000 7fdc297b8640 1 -- 172.21.15.70:0/276877319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 msgr2=0x55f479457560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.753+0000 7fdc297b8640 1 --2- 172.21.15.70:0/276877319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 0x55f479457560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.753+0000 7fdc297b8640 1 -- 172.21.15.70:0/276877319 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4794c6730 con 0x55f47943e140 2024-06-04T05:06:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.753+0000 7fdc2a7ba640 1 --2- 172.21.15.70:0/276877319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 0x55f479457560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.753+0000 7fdc297b8640 1 --2- 172.21.15.70:0/276877319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943e140 0x55f479457020 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x55f47944c4f0 tx=0x7fdc14004b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a725387a5cb1a18 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.753+0000 7fdc0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:06:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.753+0000 7fdc0d7fa640 1 -- 172.21.15.70:0/276877319 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc14005710 con 0x55f47943e140 2024-06-04T05:06:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.753+0000 7fdc0d7fa640 1 -- 172.21.15.70:0/276877319 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7fdc14012690 con 0x55f47943e140 2024-06-04T05:06:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.753+0000 7fdc2f010740 1 -- 172.21.15.70:0/276877319 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc1c02d070 con 0x55f47943e140 2024-06-04T05:06:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.754+0000 7fdc0d7fa640 1 -- 172.21.15.70:0/276877319 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdc14012890 con 0x55f47943e140 2024-06-04T05:06:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.754+0000 7fdc2f010740 10 client.27184 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.754+0000 7fdc2f010740 1 -- 172.21.15.70:0/276877319 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f4794c79f0 con 0x55f47943e140 2024-06-04T05:06:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.754+0000 7fdc0d7fa640 1 -- 172.21.15.70:0/276877319 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc1c02d070 con 0x55f47943e140 2024-06-04T05:06:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.754+0000 7fdc0d7fa640 1 -- 172.21.15.70:0/276877319 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fdc1402e070 con 0x55f47943e140 2024-06-04T05:06:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.754+0000 7fdc2f010740 10 client.27184 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.754+0000 7fdc2f010740 10 client.27184 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:06:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.754+0000 7fdc2f010740 20 client.27184 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.754+0000 7fdc2f010740 1 --2- 172.21.15.70:0/276877319 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f4794c8170 0x55f4794ca590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.754+0000 7fdc2f010740 4 client.27184 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:06:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.754+0000 7fdc2f010740 1 -- 172.21.15.70:0/276877319 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f4793ab360 con 0x55f4794c8170 2024-06-04T05:06:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.754+0000 7fdc2a7ba640 1 --2- 172.21.15.70:0/276877319 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f4794c8170 0x55f4794ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.755+0000 7fdc0d7fa640 10 client.27184 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.755+0000 7fdc2a7ba640 1 --2- 172.21.15.70:0/276877319 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f4794c8170 0x55f4794ca590 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e05d900b3bc4fc75 server_cookie=14517ca792581569 in_seq=0 out_seq=0 2024-06-04T05:06:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.771+0000 7fdc0d7fa640 1 -- 172.21.15.70:0/276877319 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f4793ab360 con 0x55f4794c8170 2024-06-04T05:06:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.771+0000 7fdc0d7fa640 10 client.27184 handle_command_reply: tid=0 2024-06-04T05:06:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.771+0000 7fdc2f010740 1 client.27184 shutdown 2024-06-04T05:06:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.771+0000 7fdc2f010740 1 -- 172.21.15.70:0/276877319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943e140 msgr2=0x55f479457020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.771+0000 7fdc2f010740 1 --2- 172.21.15.70:0/276877319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943e140 0x55f479457020 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x55f47944c4f0 tx=0x7fdc14004b70 comp rx=0 tx=0).stop 2024-06-04T05:06:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.771+0000 7fdc2f010740 1 -- 172.21.15.70:0/276877319 shutdown_connections 2024-06-04T05:06:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.771+0000 7fdc2f010740 1 --2- 172.21.15.70:0/276877319 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f4794c8170 0x55f4794ca590 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.771+0000 7fdc2f010740 1 --2- 172.21.15.70:0/276877319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f479448a00 0x55f479457560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.771+0000 7fdc2f010740 1 --2- 172.21.15.70:0/276877319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47943e140 0x55f479457020 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.771+0000 7fdc0d7fa640 0 client.27184 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:35.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.771+0000 7fdc2f010740 1 --2- 172.21.15.70:0/276877319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47943d820 0x55f479456ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.771+0000 7fdc2f010740 1 -- 172.21.15.70:0/276877319 >> 172.21.15.70:0/276877319 conn(0x55f479439d30 msgr2=0x55f479452bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:35.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.772+0000 7fdc2f010740 1 -- 172.21.15.70:0/276877319 shutdown_connections 2024-06-04T05:06:35.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.772+0000 7fdc2f010740 1 -- 172.21.15.70:0/276877319 wait complete. 2024-06-04T05:06:35.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.772+0000 7fdc2f010740 20 client.27184 trim_cache size 0 max 16384 2024-06-04T05:06:35.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:06:35.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:06:35.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:06:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.772+0000 7fdc2c21d640 1 -- 172.21.15.70:0/1447385779 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdc100b2a10 msgr2=0x7fdc100b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.772+0000 7fdc2c21d640 1 --2- 172.21.15.70:0/1447385779 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdc100b2a10 0x7fdc100b4ed0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdc201001b0 tx=0x7fdc20107040 comp rx=0 tx=0).stop 2024-06-04T05:06:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.772+0000 7fdc2c21d640 1 -- 172.21.15.70:0/1447385779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc24101960 msgr2=0x7fdc24199f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.772+0000 7fdc2c21d640 1 --2- 172.21.15.70:0/1447385779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc24101960 0x7fdc24199f80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdc20005940 tx=0x7fdc20004040 comp rx=0 tx=0).stop 2024-06-04T05:06:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.772+0000 7fdc2c21d640 1 -- 172.21.15.70:0/1447385779 shutdown_connections 2024-06-04T05:06:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.773+0000 7fdc2c21d640 1 --2- 172.21.15.70:0/1447385779 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdc100b2a10 0x7fdc100b4ed0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.773+0000 7fdc2c21d640 1 --2- 172.21.15.70:0/1447385779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc24107970 0x7fdc241a5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.773+0000 7fdc2c21d640 1 --2- 172.21.15.70:0/1447385779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc24102320 0x7fdc2419a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.773+0000 7fdc2c21d640 1 --2- 172.21.15.70:0/1447385779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc24101960 0x7fdc24199f80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.773+0000 7fdc2c21d640 1 -- 172.21.15.70:0/1447385779 >> 172.21.15.70:0/1447385779 conn(0x7fdc24078b60 msgr2=0x7fdc24104630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.773+0000 7fdc2c21d640 1 -- 172.21.15.70:0/1447385779 shutdown_connections 2024-06-04T05:06:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:35.773+0000 7fdc2c21d640 1 -- 172.21.15.70:0/1447385779 wait complete. 2024-06-04T05:06:36.124 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:06:36.125 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:06:36.395 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:06:36.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 158702 1.85 MB/sec execute 329 sec latency 15.695 ms 2024-06-04T05:06:37.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:37 smithi098 ceph-mon[35569]: pgmap v290: 97 pgs: 97 active+clean; 98 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.5 MiB/s wr, 159 op/s 2024-06-04T05:06:37.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:37 smithi070 ceph-mon[32631]: pgmap v290: 97 pgs: 97 active+clean; 98 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.5 MiB/s wr, 159 op/s 2024-06-04T05:06:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:37 smithi112 ceph-mon[36453]: pgmap v290: 97 pgs: 97 active+clean; 98 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.5 MiB/s wr, 159 op/s 2024-06-04T05:06:37.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 159406 1.86 MB/sec execute 330 sec latency 15.653 ms 2024-06-04T05:06:38.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.284+0000 7fab4e880700 1 -- 172.21.15.70:0/485827886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab481095b0 msgr2=0x7fab48109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:38.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.284+0000 7fab4e880700 1 --2- 172.21.15.70:0/485827886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab481095b0 0x7fab48109990 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fab38009990 tx=0x7fab38009ca0 comp rx=0 tx=0).stop 2024-06-04T05:06:38.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.284+0000 7fab4e880700 1 -- 172.21.15.70:0/485827886 shutdown_connections 2024-06-04T05:06:38.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.284+0000 7fab4e880700 1 --2- 172.21.15.70:0/485827886 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab4806bbe0 0x7fab481103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:38.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.284+0000 7fab4e880700 1 --2- 172.21.15.70:0/485827886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab481095b0 0x7fab48109990 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:38.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.284+0000 7fab4e880700 1 --2- 172.21.15.70:0/485827886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4806b220 0x7fab4806b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:38.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.284+0000 7fab4e880700 1 -- 172.21.15.70:0/485827886 >> 172.21.15.70:0/485827886 conn(0x7fab480793f0 msgr2=0x7fab48079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:38.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.284+0000 7fab4e880700 1 -- 172.21.15.70:0/485827886 shutdown_connections 2024-06-04T05:06:38.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.284+0000 7fab4e880700 1 -- 172.21.15.70:0/485827886 wait complete. 2024-06-04T05:06:38.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.284+0000 7fab4e880700 1 Processor -- start 2024-06-04T05:06:38.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.284+0000 7fab4e880700 1 -- start start 2024-06-04T05:06:38.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab4e880700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4806b220 0x7fab4819f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:38.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab4e880700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab4806bbe0 0x7fab4819fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:38.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab4e880700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab481095b0 0x7fab48199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:38.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab4e880700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab48113650 con 0x7fab481095b0 2024-06-04T05:06:38.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab4e880700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fab481134d0 con 0x7fab4806b220 2024-06-04T05:06:38.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab4e880700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fab481137d0 con 0x7fab4806bbe0 2024-06-04T05:06:38.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab4ce1d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab481095b0 0x7fab48199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:38.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab47fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4806b220 0x7fab4819f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:38.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab4ce1d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab481095b0 0x7fab48199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50806/0 (socket says 172.21.15.70:50806) 2024-06-04T05:06:38.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab47fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4806b220 0x7fab4819f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:37926/0 (socket says 172.21.15.70:37926) 2024-06-04T05:06:38.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab4ce1d700 1 -- 172.21.15.70:0/140758774 learned_addr learned my addr 172.21.15.70:0/140758774 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:38.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab477fe700 1 --2- 172.21.15.70:0/140758774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab4806bbe0 0x7fab4819fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:38.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab4ce1d700 1 -- 172.21.15.70:0/140758774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab4806bbe0 msgr2=0x7fab4819fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:38.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab4ce1d700 1 --2- 172.21.15.70:0/140758774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab4806bbe0 0x7fab4819fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:38.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab4ce1d700 1 -- 172.21.15.70:0/140758774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4806b220 msgr2=0x7fab4819f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:38.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab4ce1d700 1 --2- 172.21.15.70:0/140758774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4806b220 0x7fab4819f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:38.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab4ce1d700 1 -- 172.21.15.70:0/140758774 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4819a1f0 con 0x7fab481095b0 2024-06-04T05:06:38.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab4ce1d700 1 --2- 172.21.15.70:0/140758774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab481095b0 0x7fab48199a90 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fab4000ef30 tx=0x7fab4000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:38.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.285+0000 7fab457fa700 1 -- 172.21.15.70:0/140758774 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab400092e0 con 0x7fab481095b0 2024-06-04T05:06:38.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.286+0000 7fab457fa700 1 -- 172.21.15.70:0/140758774 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab40004bb0 con 0x7fab481095b0 2024-06-04T05:06:38.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.286+0000 7fab4e880700 1 -- 172.21.15.70:0/140758774 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab4819a480 con 0x7fab481095b0 2024-06-04T05:06:38.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.286+0000 7fab457fa700 1 -- 172.21.15.70:0/140758774 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab40004020 con 0x7fab481095b0 2024-06-04T05:06:38.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.286+0000 7fab4e880700 1 -- 172.21.15.70:0/140758774 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab481a67a0 con 0x7fab481095b0 2024-06-04T05:06:38.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.287+0000 7fab4e880700 1 -- 172.21.15.70:0/140758774 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab4810ba90 con 0x7fab481095b0 2024-06-04T05:06:38.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.288+0000 7fab457fa700 1 -- 172.21.15.70:0/140758774 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fab4001b030 con 0x7fab481095b0 2024-06-04T05:06:38.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.289+0000 7fab457fa700 1 --2- 172.21.15.70:0/140758774 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab300b2a70 0x7fab300b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:38.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.289+0000 7fab47fff700 1 --2- 172.21.15.70:0/140758774 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab300b2a70 0x7fab300b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:38.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.289+0000 7fab457fa700 1 -- 172.21.15.70:0/140758774 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7fab400e2a50 con 0x7fab481095b0 2024-06-04T05:06:38.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.289+0000 7fab47fff700 1 --2- 172.21.15.70:0/140758774 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab300b2a70 0x7fab300b4f30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fab3800b2e0 tx=0x7fab3801a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:38.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.293+0000 7fab457fa700 1 -- 172.21.15.70:0/140758774 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fab40022020 con 0x7fab481095b0 2024-06-04T05:06:38.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.572+0000 7fab4e880700 1 -- 172.21.15.70:0/140758774 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab4806a230 con 0x7fab481095b0 2024-06-04T05:06:38.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.573+0000 7fab457fa700 1 -- 172.21.15.70:0/140758774 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11929 (secure 0 0 0) 0x7fab40010040 con 0x7fab481095b0 2024-06-04T05:06:38.576 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:06:38.576 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:01:52.807483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:06:38.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.575+0000 7fab4e880700 1 -- 172.21.15.70:0/140758774 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab300b2a70 msgr2=0x7fab300b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:38.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.575+0000 7fab4e880700 1 --2- 172.21.15.70:0/140758774 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab300b2a70 0x7fab300b4f30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fab3800b2e0 tx=0x7fab3801a040 comp rx=0 tx=0).stop 2024-06-04T05:06:38.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.575+0000 7fab4e880700 1 -- 172.21.15.70:0/140758774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab481095b0 msgr2=0x7fab48199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:38.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.575+0000 7fab4e880700 1 --2- 172.21.15.70:0/140758774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab481095b0 0x7fab48199a90 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fab4000ef30 tx=0x7fab4000c3a0 comp rx=0 tx=0).stop 2024-06-04T05:06:38.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.575+0000 7fab4e880700 1 -- 172.21.15.70:0/140758774 shutdown_connections 2024-06-04T05:06:38.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.575+0000 7fab4e880700 1 --2- 172.21.15.70:0/140758774 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab300b2a70 0x7fab300b4f30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:38.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.575+0000 7fab4e880700 1 --2- 172.21.15.70:0/140758774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab481095b0 0x7fab48199a90 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:38.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.575+0000 7fab4e880700 1 --2- 172.21.15.70:0/140758774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4806b220 0x7fab4819f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:38.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.575+0000 7fab4e880700 1 --2- 172.21.15.70:0/140758774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab4806bbe0 0x7fab4819fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:38.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.575+0000 7fab4e880700 1 -- 172.21.15.70:0/140758774 >> 172.21.15.70:0/140758774 conn(0x7fab480793f0 msgr2=0x7fab48101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:38.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.576+0000 7fab4e880700 1 -- 172.21.15.70:0/140758774 shutdown_connections 2024-06-04T05:06:38.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:38.576+0000 7fab4e880700 1 -- 172.21.15.70:0/140758774 wait complete. 2024-06-04T05:06:38.580 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 32 2024-06-04T05:06:38.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 159861 1.87 MB/sec execute 331 sec latency 63.426 ms 2024-06-04T05:06:39.383 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:06:39.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:39 smithi098 ceph-mon[35569]: pgmap v291: 97 pgs: 97 active+clean; 98 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-06-04T05:06:39.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:39 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/140758774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:06:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:39 smithi070 ceph-mon[32631]: pgmap v291: 97 pgs: 97 active+clean; 98 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-06-04T05:06:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:39 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/140758774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:06:39.635 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:06:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:39 smithi112 ceph-mon[36453]: pgmap v291: 97 pgs: 97 active+clean; 98 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-06-04T05:06:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:39 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/140758774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:06:39.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 160115 1.87 MB/sec execute 332 sec latency 12.664 ms 2024-06-04T05:06:40.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 160576 1.86 MB/sec execute 333 sec latency 17.289 ms 2024-06-04T05:06:41.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.450+0000 7fe91f194700 1 -- 172.21.15.70:0/2200348519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe91806bc90 msgr2=0x7fe91806c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:41.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.450+0000 7fe91f194700 1 --2- 172.21.15.70:0/2200348519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe91806bc90 0x7fe91806c110 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fe904009990 tx=0x7fe904009ca0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.450+0000 7fe91f194700 1 -- 172.21.15.70:0/2200348519 shutdown_connections 2024-06-04T05:06:41.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.450+0000 7fe91f194700 1 --2- 172.21.15.70:0/2200348519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe91806c650 0x7fe918110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.450+0000 7fe91f194700 1 --2- 172.21.15.70:0/2200348519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe918108580 0x7fe918108960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.450+0000 7fe91f194700 1 --2- 172.21.15.70:0/2200348519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe91806bc90 0x7fe91806c110 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.450+0000 7fe91f194700 1 -- 172.21.15.70:0/2200348519 >> 172.21.15.70:0/2200348519 conn(0x7fe918079300 msgr2=0x7fe918079710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:41.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.450+0000 7fe91f194700 1 -- 172.21.15.70:0/2200348519 shutdown_connections 2024-06-04T05:06:41.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.450+0000 7fe91f194700 1 -- 172.21.15.70:0/2200348519 wait complete. 2024-06-04T05:06:41.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.451+0000 7fe91f194700 1 Processor -- start 2024-06-04T05:06:41.455 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:41 smithi070 ceph-mon[32631]: pgmap v292: 97 pgs: 97 active+clean; 98 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-06-04T05:06:41.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.451+0000 7fe91f194700 1 -- start start 2024-06-04T05:06:41.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.451+0000 7fe91f194700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe91806bc90 0x7fe9181027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.451+0000 7fe91f194700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe91806c650 0x7fe918102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.451+0000 7fe91f194700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe918108580 0x7fe918103270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.451+0000 7fe91f194700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe918113660 con 0x7fe91806c650 2024-06-04T05:06:41.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.451+0000 7fe917fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe91806c650 0x7fe918102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.451+0000 7fe917fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe91806c650 0x7fe918102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50824/0 (socket says 172.21.15.70:50824) 2024-06-04T05:06:41.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.451+0000 7fe917fff700 1 -- 172.21.15.70:0/2328529343 learned_addr learned my addr 172.21.15.70:0/2328529343 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:41.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.451+0000 7fe91f194700 1 -- 172.21.15.70:0/2328529343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe9181134e0 con 0x7fe918108580 2024-06-04T05:06:41.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.451+0000 7fe91f194700 1 -- 172.21.15.70:0/2328529343 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe9181137e0 con 0x7fe91806bc90 2024-06-04T05:06:41.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.451+0000 7fe91cf30700 1 --2- 172.21.15.70:0/2328529343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe91806bc90 0x7fe9181027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.451+0000 7fe91d731700 1 --2- 172.21.15.70:0/2328529343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe918108580 0x7fe918103270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.452+0000 7fe91cf30700 1 -- 172.21.15.70:0/2328529343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe918108580 msgr2=0x7fe918103270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:41.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.452+0000 7fe91cf30700 1 --2- 172.21.15.70:0/2328529343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe918108580 0x7fe918103270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.452+0000 7fe91cf30700 1 -- 172.21.15.70:0/2328529343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe91806c650 msgr2=0x7fe918102d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:41.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.452+0000 7fe91cf30700 1 --2- 172.21.15.70:0/2328529343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe91806c650 0x7fe918102d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.452+0000 7fe91cf30700 1 -- 172.21.15.70:0/2328529343 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9181a8040 con 0x7fe91806bc90 2024-06-04T05:06:41.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.452+0000 7fe91d731700 1 --2- 172.21.15.70:0/2328529343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe918108580 0x7fe918103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.452+0000 7fe91cf30700 1 --2- 172.21.15.70:0/2328529343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe91806bc90 0x7fe9181027f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00e9c0 tx=0x7fe90c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.452+0000 7fe915ffb700 1 -- 172.21.15.70:0/2328529343 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c004230 con 0x7fe91806bc90 2024-06-04T05:06:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.452+0000 7fe915ffb700 1 -- 172.21.15.70:0/2328529343 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe90c0043d0 con 0x7fe91806bc90 2024-06-04T05:06:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.452+0000 7fe91f194700 1 -- 172.21.15.70:0/2328529343 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9181a8330 con 0x7fe91806bc90 2024-06-04T05:06:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.452+0000 7fe915ffb700 1 -- 172.21.15.70:0/2328529343 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c003d30 con 0x7fe91806bc90 2024-06-04T05:06:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.452+0000 7fe91f194700 1 -- 172.21.15.70:0/2328529343 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9181a87f0 con 0x7fe91806bc90 2024-06-04T05:06:41.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.455+0000 7fe915ffb700 1 -- 172.21.15.70:0/2328529343 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fe90c00c7c0 con 0x7fe91806bc90 2024-06-04T05:06:41.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.456+0000 7fe915ffb700 1 --2- 172.21.15.70:0/2328529343 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9080b29f0 0x7fe9080b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.456+0000 7fe917fff700 1 --2- 172.21.15.70:0/2328529343 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9080b29f0 0x7fe9080b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.456+0000 7fe915ffb700 1 -- 172.21.15.70:0/2328529343 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7fe90c0e2830 con 0x7fe91806bc90 2024-06-04T05:06:41.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.456+0000 7fe917fff700 1 --2- 172.21.15.70:0/2328529343 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9080b29f0 0x7fe9080b4eb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe918104280 tx=0x7fe904005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:41.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.456+0000 7fe929a29b80 1 Processor -- start 2024-06-04T05:06:41.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe929a29b80 1 -- start start 2024-06-04T05:06:41.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe929a29b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af3a320 0x559c7af3a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe929a29b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af348a0 0x559c7af34d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe929a29b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af41080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe929a29b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559c7ae92d70 con 0x559c7af35260 2024-06-04T05:06:41.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe929a29b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559c7ae82910 con 0x559c7af348a0 2024-06-04T05:06:41.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe929a29b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559c7ae86950 con 0x559c7af3a320 2024-06-04T05:06:41.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe91d731700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af41080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe91cf30700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af3a320 0x559c7af3a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe91d731700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af41080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50830/0 (socket says 172.21.15.70:50830) 2024-06-04T05:06:41.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe91d731700 1 -- 172.21.15.70:0/1643769606 learned_addr learned my addr 172.21.15.70:0/1643769606 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:41.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe917fff700 1 --2- 172.21.15.70:0/1643769606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af348a0 0x559c7af34d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe91cf30700 1 -- 172.21.15.70:0/1643769606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af348a0 msgr2=0x559c7af34d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:41.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe91cf30700 1 --2- 172.21.15.70:0/1643769606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af348a0 0x559c7af34d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe91cf30700 1 -- 172.21.15.70:0/1643769606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 msgr2=0x559c7af41080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:41.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe91cf30700 1 --2- 172.21.15.70:0/1643769606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af41080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe91cf30700 1 -- 172.21.15.70:0/1643769606 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c7ae2bfb0 con 0x559c7af3a320 2024-06-04T05:06:41.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe91d731700 1 --2- 172.21.15.70:0/1643769606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af41080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:41.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.457+0000 7fe91cf30700 1 --2- 172.21.15.70:0/1643769606 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af3a320 0x559c7af3a700 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0dff50 tx=0x7fe90c0a9250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e2913171ca5a06b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:41.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.458+0000 7fe902ffd700 1 -- 172.21.15.70:0/1643769606 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c0a95f0 con 0x559c7af3a320 2024-06-04T05:06:41.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.458+0000 7fe902ffd700 1 -- 172.21.15.70:0/1643769606 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe90c0aa1e0 con 0x559c7af3a320 2024-06-04T05:06:41.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.458+0000 7fe902ffd700 1 -- 172.21.15.70:0/1643769606 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c0a9d40 con 0x559c7af3a320 2024-06-04T05:06:41.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.458+0000 7fe929a29b80 1 -- 172.21.15.70:0/1643769606 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af3a320 msgr2=0x559c7af3a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:41.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.458+0000 7fe929a29b80 1 --2- 172.21.15.70:0/1643769606 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af3a320 0x559c7af3a700 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0dff50 tx=0x7fe90c0a9250 comp rx=0 tx=0).stop 2024-06-04T05:06:41.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.458+0000 7fe929a29b80 1 -- 172.21.15.70:0/1643769606 shutdown_connections 2024-06-04T05:06:41.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.458+0000 7fe929a29b80 1 --2- 172.21.15.70:0/1643769606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af41080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.458+0000 7fe929a29b80 1 --2- 172.21.15.70:0/1643769606 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af3a320 0x559c7af3a700 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.458+0000 7fe929a29b80 1 --2- 172.21.15.70:0/1643769606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af348a0 0x559c7af34d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.458+0000 7fe929a29b80 1 -- 172.21.15.70:0/1643769606 >> 172.21.15.70:0/1643769606 conn(0x559c7ae60240 msgr2=0x559c7af2f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.458+0000 7fe929a29b80 1 -- 172.21.15.70:0/1643769606 shutdown_connections 2024-06-04T05:06:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.459+0000 7fe929a29b80 1 -- 172.21.15.70:0/1643769606 wait complete. 2024-06-04T05:06:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.459+0000 7fe929a29b80 1 -- start start 2024-06-04T05:06:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.459+0000 7fe929a29b80 1 Processor -- start 2024-06-04T05:06:41.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.459+0000 7fe929a29b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af348a0 0x559c7af40da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.459+0000 7fe929a29b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af35260 0x559c7af3e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.459+0000 7fe929a29b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af3a320 0x559c7af3e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.459+0000 7fe929a29b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559c7ad88fa0 con 0x559c7af348a0 2024-06-04T05:06:41.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.459+0000 7fe929a29b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559c7adeada0 con 0x559c7af35260 2024-06-04T05:06:41.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.459+0000 7fe929a29b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559c7af41680 con 0x559c7af3a320 2024-06-04T05:06:41.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.459+0000 7fe929a29b80 10 client.0 _ll_register_callbacks cb 0x559c7ae1d950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:41.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.459+0000 7fe91cf30700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af348a0 0x559c7af40da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.459+0000 7fe91cf30700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af348a0 0x559c7af40da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50844/0 (socket says 172.21.15.70:50844) 2024-06-04T05:06:41.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.459+0000 7fe91cf30700 1 -- 172.21.15.70:0/1050585213 learned_addr learned my addr 172.21.15.70:0/1050585213 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:41.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.460+0000 7fe917fff700 1 --2- 172.21.15.70:0/1050585213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af35260 0x559c7af3e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.460+0000 7fe91d731700 1 --2- 172.21.15.70:0/1050585213 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af3a320 0x559c7af3e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.460+0000 7fe917fff700 1 -- 172.21.15.70:0/1050585213 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af3a320 msgr2=0x559c7af3e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:41.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.460+0000 7fe917fff700 1 --2- 172.21.15.70:0/1050585213 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af3a320 0x559c7af3e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.460+0000 7fe917fff700 1 -- 172.21.15.70:0/1050585213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af348a0 msgr2=0x559c7af40da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:41.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.460+0000 7fe917fff700 1 --2- 172.21.15.70:0/1050585213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af348a0 0x559c7af40da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.460+0000 7fe917fff700 1 -- 172.21.15.70:0/1050585213 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c7add28d0 con 0x559c7af35260 2024-06-04T05:06:41.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.460+0000 7fe91cf30700 1 --2- 172.21.15.70:0/1050585213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af348a0 0x559c7af40da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:41.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.460+0000 7fe917fff700 1 --2- 172.21.15.70:0/1050585213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af35260 0x559c7af3e0a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe904006010 tx=0x7fe904009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=140aab47c0bbb106 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:41.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.460+0000 7fe8f7fff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:06:41.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.460+0000 7fe8f7fff700 1 -- 172.21.15.70:0/1050585213 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904035060 con 0x559c7af35260 2024-06-04T05:06:41.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.460+0000 7fe929a29b80 1 -- 172.21.15.70:0/1050585213 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe90c0aa1e0 con 0x559c7af35260 2024-06-04T05:06:41.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.460+0000 7fe8f7fff700 1 -- 172.21.15.70:0/1050585213 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7fe904030480 con 0x559c7af35260 2024-06-04T05:06:41.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.460+0000 7fe8f7fff700 1 -- 172.21.15.70:0/1050585213 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe904030680 con 0x559c7af35260 2024-06-04T05:06:41.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.460+0000 7fe929a29b80 10 client.27116 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:41.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.460+0000 7fe929a29b80 1 -- 172.21.15.70:0/1050585213 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559c7ae3c760 con 0x559c7af35260 2024-06-04T05:06:41.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.461+0000 7fe8f7fff700 1 -- 172.21.15.70:0/1050585213 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe90c0aa1e0 con 0x559c7af35260 2024-06-04T05:06:41.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.461+0000 7fe8f7fff700 1 -- 172.21.15.70:0/1050585213 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fe90400f760 con 0x559c7af35260 2024-06-04T05:06:41.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.461+0000 7fe929a29b80 10 client.27116 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:41.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.461+0000 7fe929a29b80 10 client.27116 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:06:41.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.461+0000 7fe929a29b80 20 client.27116 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:41.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.461+0000 7fe929a29b80 1 --2- 172.21.15.70:0/1050585213 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x559c7af4e1e0 0x559c7af505d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.461+0000 7fe929a29b80 4 client.27116 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:06:41.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.461+0000 7fe929a29b80 1 -- 172.21.15.70:0/1050585213 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c7ae94cc0 con 0x559c7af4e1e0 2024-06-04T05:06:41.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.461+0000 7fe91cf30700 1 --2- 172.21.15.70:0/1050585213 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x559c7af4e1e0 0x559c7af505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.461+0000 7fe8f7fff700 10 client.27116 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:06:41.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.461+0000 7fe91cf30700 1 --2- 172.21.15.70:0/1050585213 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x559c7af4e1e0 0x559c7af505d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6ed23dc49e632238 server_cookie=381b1b1f96b7b274 in_seq=0 out_seq=0 2024-06-04T05:06:41.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.462+0000 7fe8f7fff700 1 -- 172.21.15.70:0/1050585213 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559c7ae94cc0 con 0x559c7af4e1e0 2024-06-04T05:06:41.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.462+0000 7fe8f7fff700 10 client.27116 handle_command_reply: tid=0 2024-06-04T05:06:41.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.462+0000 7fe929a29b80 1 client.27116 shutdown 2024-06-04T05:06:41.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.462+0000 7fe929a29b80 1 -- 172.21.15.70:0/1050585213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af35260 msgr2=0x559c7af3e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:41.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.462+0000 7fe929a29b80 1 --2- 172.21.15.70:0/1050585213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af35260 0x559c7af3e0a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe904006010 tx=0x7fe904009ca0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.462+0000 7fe929a29b80 1 -- 172.21.15.70:0/1050585213 shutdown_connections 2024-06-04T05:06:41.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.462+0000 7fe929a29b80 1 --2- 172.21.15.70:0/1050585213 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x559c7af4e1e0 0x559c7af505d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.462+0000 7fe929a29b80 1 --2- 172.21.15.70:0/1050585213 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af3a320 0x559c7af3e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.462+0000 7fe929a29b80 1 --2- 172.21.15.70:0/1050585213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af348a0 0x559c7af40da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.462+0000 7fe8f7fff700 0 client.27116 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:06:41.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.462+0000 7fe929a29b80 1 --2- 172.21.15.70:0/1050585213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af35260 0x559c7af3e0a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.462+0000 7fe929a29b80 1 -- 172.21.15.70:0/1050585213 >> 172.21.15.70:0/1050585213 conn(0x559c7ae60240 msgr2=0x559c7af2eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:41.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.463+0000 7fe929a29b80 1 -- 172.21.15.70:0/1050585213 shutdown_connections 2024-06-04T05:06:41.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.463+0000 7fe929a29b80 1 -- 172.21.15.70:0/1050585213 wait complete. 2024-06-04T05:06:41.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.463+0000 7fe929a29b80 20 client.27116 trim_cache size 0 max 16384 2024-06-04T05:06:41.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.479+0000 7fe929a29b80 1 Processor -- start 2024-06-04T05:06:41.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.479+0000 7fe929a29b80 1 -- start start 2024-06-04T05:06:41.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe929a29b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af348a0 0x559c7af505d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe929a29b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af398e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe929a29b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af3a320 0x559c7af2f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe929a29b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe9181b00f0 con 0x559c7af35260 2024-06-04T05:06:41.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe929a29b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe9181b1c30 con 0x559c7af3a320 2024-06-04T05:06:41.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe929a29b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe9181b07c0 con 0x559c7af348a0 2024-06-04T05:06:41.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe917fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af398e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe917fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af398e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50856/0 (socket says 172.21.15.70:50856) 2024-06-04T05:06:41.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe917fff700 1 -- 172.21.15.70:0/3830526936 learned_addr learned my addr 172.21.15.70:0/3830526936 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:41.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe91cf30700 1 --2- 172.21.15.70:0/3830526936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af348a0 0x559c7af505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe91d731700 1 --2- 172.21.15.70:0/3830526936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af3a320 0x559c7af2f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe917fff700 1 -- 172.21.15.70:0/3830526936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af348a0 msgr2=0x559c7af505d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:41.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe917fff700 1 --2- 172.21.15.70:0/3830526936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af348a0 0x559c7af505d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe917fff700 1 -- 172.21.15.70:0/3830526936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af3a320 msgr2=0x559c7af2f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:41.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe917fff700 1 --2- 172.21.15.70:0/3830526936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af3a320 0x559c7af2f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe917fff700 1 -- 172.21.15.70:0/3830526936 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c7af50c30 con 0x559c7af35260 2024-06-04T05:06:41.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe917fff700 1 --2- 172.21.15.70:0/3830526936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af398e0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fe904005850 tx=0x7fe904005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b447f5d7ef9d5e8d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:41.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.480+0000 7fe9027fc700 1 -- 172.21.15.70:0/3830526936 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904035070 con 0x559c7af35260 2024-06-04T05:06:41.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.481+0000 7fe9027fc700 1 -- 172.21.15.70:0/3830526936 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe90400f8f0 con 0x559c7af35260 2024-06-04T05:06:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.481+0000 7fe9027fc700 1 -- 172.21.15.70:0/3830526936 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904030e30 con 0x559c7af35260 2024-06-04T05:06:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.481+0000 7fe929a29b80 1 -- 172.21.15.70:0/3830526936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 msgr2=0x559c7af398e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.481+0000 7fe929a29b80 1 --2- 172.21.15.70:0/3830526936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af398e0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fe904005850 tx=0x7fe904005a50 comp rx=0 tx=0).stop 2024-06-04T05:06:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.481+0000 7fe929a29b80 1 -- 172.21.15.70:0/3830526936 shutdown_connections 2024-06-04T05:06:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.481+0000 7fe929a29b80 1 --2- 172.21.15.70:0/3830526936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af3a320 0x559c7af2f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.481+0000 7fe929a29b80 1 --2- 172.21.15.70:0/3830526936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af348a0 0x559c7af505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.481+0000 7fe929a29b80 1 --2- 172.21.15.70:0/3830526936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af398e0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.481+0000 7fe929a29b80 1 -- 172.21.15.70:0/3830526936 >> 172.21.15.70:0/3830526936 conn(0x559c7ae60240 msgr2=0x559c7af391d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.481+0000 7fe929a29b80 1 -- 172.21.15.70:0/3830526936 shutdown_connections 2024-06-04T05:06:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.481+0000 7fe929a29b80 1 -- 172.21.15.70:0/3830526936 wait complete. 2024-06-04T05:06:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.482+0000 7fe929a29b80 1 -- start start 2024-06-04T05:06:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.482+0000 7fe929a29b80 1 Processor -- start 2024-06-04T05:06:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.482+0000 7fe929a29b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af348a0 0x559c7af54e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.482+0000 7fe929a29b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af553b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.482+0000 7fe929a29b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af3a320 0x559c7af558f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.482+0000 7fe929a29b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe9181b1560 con 0x559c7af35260 2024-06-04T05:06:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.482+0000 7fe929a29b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe9181b0e90 con 0x559c7af348a0 2024-06-04T05:06:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.482+0000 7fe929a29b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe9181ae0b0 con 0x559c7af3a320 2024-06-04T05:06:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.482+0000 7fe929a29b80 10 client.0 _ll_register_callbacks cb 0x559c7ae1d950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.482+0000 7fe917fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.482+0000 7fe91cf30700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af348a0 0x559c7af54e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.482+0000 7fe917fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af553b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50860/0 (socket says 172.21.15.70:50860) 2024-06-04T05:06:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.482+0000 7fe917fff700 1 -- 172.21.15.70:0/733096974 learned_addr learned my addr 172.21.15.70:0/733096974 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.483+0000 7fe91d731700 1 --2- 172.21.15.70:0/733096974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af3a320 0x559c7af558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.483+0000 7fe917fff700 1 -- 172.21.15.70:0/733096974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af3a320 msgr2=0x559c7af558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.483+0000 7fe917fff700 1 --2- 172.21.15.70:0/733096974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af3a320 0x559c7af558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.483+0000 7fe917fff700 1 -- 172.21.15.70:0/733096974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af348a0 msgr2=0x559c7af54e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.483+0000 7fe917fff700 1 --2- 172.21.15.70:0/733096974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af348a0 0x559c7af54e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.483+0000 7fe917fff700 1 -- 172.21.15.70:0/733096974 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c7af45b10 con 0x559c7af35260 2024-06-04T05:06:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.483+0000 7fe91cf30700 1 --2- 172.21.15.70:0/733096974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af348a0 0x559c7af54e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.483+0000 7fe917fff700 1 --2- 172.21.15.70:0/733096974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af553b0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fe90400b2e0 tx=0x7fe904030590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a36d12720e215a6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.483+0000 7fe8f7fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.483+0000 7fe8f7fff700 1 -- 172.21.15.70:0/733096974 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904035050 con 0x559c7af35260 2024-06-04T05:06:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.483+0000 7fe8f7fff700 1 -- 172.21.15.70:0/733096974 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7fe90400f660 con 0x559c7af35260 2024-06-04T05:06:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.483+0000 7fe929a29b80 1 -- 172.21.15.70:0/733096974 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe90400f8f0 con 0x559c7af35260 2024-06-04T05:06:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.483+0000 7fe8f7fff700 1 -- 172.21.15.70:0/733096974 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe904030820 con 0x559c7af35260 2024-06-04T05:06:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.483+0000 7fe929a29b80 10 client.17988 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.483+0000 7fe929a29b80 1 -- 172.21.15.70:0/733096974 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559c7af46df0 con 0x559c7af35260 2024-06-04T05:06:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.484+0000 7fe8f7fff700 1 -- 172.21.15.70:0/733096974 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe90400f8f0 con 0x559c7af35260 2024-06-04T05:06:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.484+0000 7fe8f7fff700 1 -- 172.21.15.70:0/733096974 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fe9040249f0 con 0x559c7af35260 2024-06-04T05:06:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.484+0000 7fe929a29b80 10 client.17988 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.484+0000 7fe929a29b80 10 client.17988 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:06:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.484+0000 7fe929a29b80 20 client.17988 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.484+0000 7fe929a29b80 1 --2- 172.21.15.70:0/733096974 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x559c7af472a0 0x559c7afbad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.484+0000 7fe929a29b80 4 client.17988 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:06:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.484+0000 7fe929a29b80 1 -- 172.21.15.70:0/733096974 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559c7ae8e4a0 con 0x559c7af472a0 2024-06-04T05:06:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.484+0000 7fe91cf30700 1 --2- 172.21.15.70:0/733096974 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x559c7af472a0 0x559c7afbad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.484+0000 7fe91cf30700 1 --2- 172.21.15.70:0/733096974 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x559c7af472a0 0x559c7afbad60 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b21ce208c406b57e server_cookie=c3d42b632d8b8cfa in_seq=0 out_seq=0 2024-06-04T05:06:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.484+0000 7fe8f7fff700 10 client.17988 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:06:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.485+0000 7fe8f7fff700 1 -- 172.21.15.70:0/733096974 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559c7ae8e4a0 con 0x559c7af472a0 2024-06-04T05:06:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.485+0000 7fe8f7fff700 10 client.17988 handle_command_reply: tid=0 2024-06-04T05:06:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.485+0000 7fe929a29b80 1 client.17988 shutdown 2024-06-04T05:06:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.485+0000 7fe929a29b80 1 -- 172.21.15.70:0/733096974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 msgr2=0x559c7af553b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.485+0000 7fe929a29b80 1 --2- 172.21.15.70:0/733096974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af553b0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fe90400b2e0 tx=0x7fe904030590 comp rx=0 tx=0).stop 2024-06-04T05:06:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.485+0000 7fe929a29b80 1 -- 172.21.15.70:0/733096974 shutdown_connections 2024-06-04T05:06:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.485+0000 7fe929a29b80 1 --2- 172.21.15.70:0/733096974 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x559c7af472a0 0x559c7afbad60 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.485+0000 7fe8f7fff700 0 client.17988 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:06:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.485+0000 7fe929a29b80 1 --2- 172.21.15.70:0/733096974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c7af3a320 0x559c7af558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.485+0000 7fe929a29b80 1 --2- 172.21.15.70:0/733096974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c7af348a0 0x559c7af54e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.485+0000 7fe929a29b80 1 --2- 172.21.15.70:0/733096974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c7af35260 0x559c7af553b0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.485+0000 7fe929a29b80 1 -- 172.21.15.70:0/733096974 >> 172.21.15.70:0/733096974 conn(0x559c7ae60240 msgr2=0x559c7af33890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.485+0000 7fe929a29b80 1 -- 172.21.15.70:0/733096974 shutdown_connections 2024-06-04T05:06:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.485+0000 7fe929a29b80 1 -- 172.21.15.70:0/733096974 wait complete. 2024-06-04T05:06:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.485+0000 7fe929a29b80 20 client.17988 trim_cache size 0 max 16384 2024-06-04T05:06:41.504 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:06:41.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.486+0000 7fe91f194700 1 -- 172.21.15.70:0/2328529343 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9080b29f0 msgr2=0x7fe9080b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:41.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.486+0000 7fe91f194700 1 --2- 172.21.15.70:0/2328529343 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9080b29f0 0x7fe9080b4eb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe918104280 tx=0x7fe904005960 comp rx=0 tx=0).stop 2024-06-04T05:06:41.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.486+0000 7fe91f194700 1 -- 172.21.15.70:0/2328529343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe91806bc90 msgr2=0x7fe9181027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:41.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.486+0000 7fe91f194700 1 --2- 172.21.15.70:0/2328529343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe91806bc90 0x7fe9181027f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00e9c0 tx=0x7fe90c00ecd0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.486+0000 7fe91f194700 1 -- 172.21.15.70:0/2328529343 shutdown_connections 2024-06-04T05:06:41.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.486+0000 7fe91f194700 1 --2- 172.21.15.70:0/2328529343 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9080b29f0 0x7fe9080b4eb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.486+0000 7fe91f194700 1 --2- 172.21.15.70:0/2328529343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe918108580 0x7fe918103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.486+0000 7fe91f194700 1 --2- 172.21.15.70:0/2328529343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe91806bc90 0x7fe9181027f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.486+0000 7fe91f194700 1 --2- 172.21.15.70:0/2328529343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe91806c650 0x7fe918102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:41.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.486+0000 7fe91f194700 1 -- 172.21.15.70:0/2328529343 >> 172.21.15.70:0/2328529343 conn(0x7fe918079300 msgr2=0x7fe9181007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:41.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.486+0000 7fe91f194700 1 -- 172.21.15.70:0/2328529343 shutdown_connections 2024-06-04T05:06:41.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:41.486+0000 7fe91f194700 1 -- 172.21.15.70:0/2328529343 wait complete. 2024-06-04T05:06:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:41 smithi112 ceph-mon[36453]: pgmap v292: 97 pgs: 97 active+clean; 98 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-06-04T05:06:41.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 161091 1.87 MB/sec execute 334 sec latency 68.544 ms 2024-06-04T05:06:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:41 smithi098 ceph-mon[35569]: pgmap v292: 97 pgs: 97 active+clean; 98 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-06-04T05:06:42.189 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:06:42.447 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:06:42.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 161607 1.86 MB/sec execute 335 sec latency 20.644 ms 2024-06-04T05:06:43.547 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:43 smithi098 ceph-mon[35569]: pgmap v293: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 2.1 MiB/s wr, 192 op/s 2024-06-04T05:06:43.588 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:43 smithi070 ceph-mon[32631]: pgmap v293: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 2.1 MiB/s wr, 192 op/s 2024-06-04T05:06:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:43 smithi112 ceph-mon[36453]: pgmap v293: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 2.1 MiB/s wr, 192 op/s 2024-06-04T05:06:43.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 162139 1.87 MB/sec execute 336 sec latency 18.734 ms 2024-06-04T05:06:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.122+0000 7fa973c6d700 1 -- 172.21.15.70:0/876752168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa96c101b30 msgr2=0x7fa96c101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.122+0000 7fa973c6d700 1 --2- 172.21.15.70:0/876752168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa96c101b30 0x7fa96c101fb0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fa95c0099c0 tx=0x7fa95c009cd0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.122+0000 7fa973c6d700 1 -- 172.21.15.70:0/876752168 shutdown_connections 2024-06-04T05:06:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.122+0000 7fa973c6d700 1 --2- 172.21.15.70:0/876752168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c1024f0 0x7fa96c1103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.122+0000 7fa973c6d700 1 --2- 172.21.15.70:0/876752168 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa96c109520 0x7fa96c109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.122+0000 7fa973c6d700 1 --2- 172.21.15.70:0/876752168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa96c101b30 0x7fa96c101fb0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.122+0000 7fa973c6d700 1 -- 172.21.15.70:0/876752168 >> 172.21.15.70:0/876752168 conn(0x7fa96c078ab0 msgr2=0x7fa96c0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.123+0000 7fa973c6d700 1 -- 172.21.15.70:0/876752168 shutdown_connections 2024-06-04T05:06:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.123+0000 7fa973c6d700 1 -- 172.21.15.70:0/876752168 wait complete. 2024-06-04T05:06:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.123+0000 7fa973c6d700 1 Processor -- start 2024-06-04T05:06:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.123+0000 7fa973c6d700 1 -- start start 2024-06-04T05:06:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.124+0000 7fa973c6d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101b30 0x7fa96c19f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.124+0000 7fa973c6d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa96c1024f0 0x7fa96c19fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.124+0000 7fa973c6d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa96c109520 0x7fa96c199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.124+0000 7fa973c6d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1135b0 con 0x7fa96c1024f0 2024-06-04T05:06:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.124+0000 7fa973c6d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c113430 con 0x7fa96c101b30 2024-06-04T05:06:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.124+0000 7fa973c6d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c113730 con 0x7fa96c109520 2024-06-04T05:06:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.124+0000 7fa971a09700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101b30 0x7fa96c19f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.124+0000 7fa971a09700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101b30 0x7fa96c19f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:37982/0 (socket says 172.21.15.70:37982) 2024-06-04T05:06:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.124+0000 7fa971a09700 1 -- 172.21.15.70:0/3533866758 learned_addr learned my addr 172.21.15.70:0/3533866758 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.124+0000 7fa971208700 1 --2- 172.21.15.70:0/3533866758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa96c1024f0 0x7fa96c19fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.124+0000 7fa971a09700 1 -- 172.21.15.70:0/3533866758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa96c109520 msgr2=0x7fa96c199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.124+0000 7fa971a09700 1 --2- 172.21.15.70:0/3533866758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa96c109520 0x7fa96c199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.124+0000 7fa971a09700 1 -- 172.21.15.70:0/3533866758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa96c1024f0 msgr2=0x7fa96c19fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.124+0000 7fa971a09700 1 --2- 172.21.15.70:0/3533866758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa96c1024f0 0x7fa96c19fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.124+0000 7fa971a09700 1 -- 172.21.15.70:0/3533866758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa96c19a070 con 0x7fa96c101b30 2024-06-04T05:06:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.124+0000 7fa971a09700 1 --2- 172.21.15.70:0/3533866758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101b30 0x7fa96c19f750 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa96800d900 tx=0x7fa96800dc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.124+0000 7fa962ffd700 1 -- 172.21.15.70:0/3533866758 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa968016070 con 0x7fa96c101b30 2024-06-04T05:06:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.125+0000 7fa973c6d700 1 -- 172.21.15.70:0/3533866758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa96c19a360 con 0x7fa96c101b30 2024-06-04T05:06:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.125+0000 7fa962ffd700 1 -- 172.21.15.70:0/3533866758 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa968004830 con 0x7fa96c101b30 2024-06-04T05:06:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.125+0000 7fa962ffd700 1 -- 172.21.15.70:0/3533866758 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa968005350 con 0x7fa96c101b30 2024-06-04T05:06:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.125+0000 7fa973c6d700 1 -- 172.21.15.70:0/3533866758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa96c1a6620 con 0x7fa96c101b30 2024-06-04T05:06:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.127+0000 7fa962ffd700 1 -- 172.21.15.70:0/3533866758 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fa9680054f0 con 0x7fa96c101b30 2024-06-04T05:06:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.128+0000 7fa962ffd700 1 --2- 172.21.15.70:0/3533866758 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa9580b29f0 0x7fa9580b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.128+0000 7fa971208700 1 --2- 172.21.15.70:0/3533866758 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa9580b29f0 0x7fa9580b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.129+0000 7fa97e502b80 1 Processor -- start 2024-06-04T05:06:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.129+0000 7fa962ffd700 1 -- 172.21.15.70:0/3533866758 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7fa9680e23a0 con 0x7fa96c101b30 2024-06-04T05:06:44.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.129+0000 7fa97e502b80 1 -- start start 2024-06-04T05:06:44.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.129+0000 7fa97e502b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 0x55ee80186c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.129+0000 7fa97e502b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 0x55ee8018a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.129+0000 7fa97e502b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee8018af60 0x55ee8018b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.129+0000 7fa97e502b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ee800e5c90 con 0x55ee80187210 2024-06-04T05:06:44.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.129+0000 7fa97e502b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ee800d5830 con 0x55ee8018af60 2024-06-04T05:06:44.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.129+0000 7fa97e502b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ee800d9870 con 0x55ee80186860 2024-06-04T05:06:44.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.129+0000 7fa971208700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 0x55ee8018a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.129+0000 7fa971a09700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 0x55ee80186c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.129+0000 7fa971a09700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 0x55ee80186c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:44544/0 (socket says 172.21.15.70:44544) 2024-06-04T05:06:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.129+0000 7fa971208700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 0x55ee8018a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50892/0 (socket says 172.21.15.70:50892) 2024-06-04T05:06:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.129+0000 7fa971a09700 1 -- 172.21.15.70:0/33184640 learned_addr learned my addr 172.21.15.70:0/33184640 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.129+0000 7fa97220a700 1 --2- 172.21.15.70:0/33184640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee8018af60 0x55ee8018b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.129+0000 7fa971208700 1 --2- 172.21.15.70:0/3533866758 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa9580b29f0 0x7fa9580b4eb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa96c0ffc40 tx=0x7fa95c005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.130+0000 7fa971a09700 1 -- 172.21.15.70:0/33184640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee8018af60 msgr2=0x55ee8018b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.130+0000 7fa971a09700 1 --2- 172.21.15.70:0/33184640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee8018af60 0x55ee8018b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.130+0000 7fa971a09700 1 -- 172.21.15.70:0/33184640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 msgr2=0x55ee8018a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.130+0000 7fa971a09700 1 --2- 172.21.15.70:0/33184640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 0x55ee8018a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.130+0000 7fa971a09700 1 -- 172.21.15.70:0/33184640 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee800258d0 con 0x55ee80186860 2024-06-04T05:06:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.130+0000 7fa971208700 1 --2- 172.21.15.70:0/33184640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 0x55ee8018a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.130+0000 7fa971a09700 1 --2- 172.21.15.70:0/33184640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 0x55ee80186c40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa9680a9580 tx=0x7fa9680a9050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e292966c8181a8d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:44.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.130+0000 7fa957fff700 1 -- 172.21.15.70:0/33184640 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9680ad9a0 con 0x55ee80186860 2024-06-04T05:06:44.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.130+0000 7fa957fff700 1 -- 172.21.15.70:0/33184640 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa9680b2450 con 0x55ee80186860 2024-06-04T05:06:44.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.130+0000 7fa957fff700 1 -- 172.21.15.70:0/33184640 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9680b1fc0 con 0x55ee80186860 2024-06-04T05:06:44.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.131+0000 7fa97e502b80 1 -- 172.21.15.70:0/33184640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 msgr2=0x55ee80186c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:44.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.131+0000 7fa97e502b80 1 --2- 172.21.15.70:0/33184640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 0x55ee80186c40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa9680a9580 tx=0x7fa9680a9050 comp rx=0 tx=0).stop 2024-06-04T05:06:44.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.131+0000 7fa97e502b80 1 -- 172.21.15.70:0/33184640 shutdown_connections 2024-06-04T05:06:44.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.131+0000 7fa97e502b80 1 --2- 172.21.15.70:0/33184640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee8018af60 0x55ee8018b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.131+0000 7fa97e502b80 1 --2- 172.21.15.70:0/33184640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 0x55ee80186c40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.131+0000 7fa97e502b80 1 --2- 172.21.15.70:0/33184640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 0x55ee8018a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.131+0000 7fa97e502b80 1 -- 172.21.15.70:0/33184640 >> 172.21.15.70:0/33184640 conn(0x55ee800b4080 msgr2=0x55ee80180e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:44.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.131+0000 7fa97e502b80 1 -- 172.21.15.70:0/33184640 shutdown_connections 2024-06-04T05:06:44.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.131+0000 7fa97e502b80 1 -- 172.21.15.70:0/33184640 wait complete. 2024-06-04T05:06:44.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.131+0000 7fa97e502b80 1 -- start start 2024-06-04T05:06:44.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa97e502b80 1 Processor -- start 2024-06-04T05:06:44.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa97e502b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee80186860 0x55ee801923b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa97e502b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 0x55ee8018f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa97e502b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee8018af60 0x55ee8018fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa97e502b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ee7ffdbfa0 con 0x55ee80187210 2024-06-04T05:06:44.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa97e502b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ee800777c0 con 0x55ee80186860 2024-06-04T05:06:44.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa97e502b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ee80193120 con 0x55ee8018af60 2024-06-04T05:06:44.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa97e502b80 10 client.0 _ll_register_callbacks cb 0x55ee80070920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:44.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa971208700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 0x55ee8018f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:44.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa97220a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee8018af60 0x55ee8018fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:44.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa971208700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 0x55ee8018f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50894/0 (socket says 172.21.15.70:50894) 2024-06-04T05:06:44.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa971a09700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee80186860 0x55ee801923b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:44.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa971208700 1 -- 172.21.15.70:0/110258870 learned_addr learned my addr 172.21.15.70:0/110258870 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:44.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa971a09700 1 -- 172.21.15.70:0/110258870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee8018af60 msgr2=0x55ee8018fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:44.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa971a09700 1 --2- 172.21.15.70:0/110258870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee8018af60 0x55ee8018fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa971a09700 1 -- 172.21.15.70:0/110258870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 msgr2=0x55ee8018f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:44.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa971a09700 1 --2- 172.21.15.70:0/110258870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 0x55ee8018f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.132+0000 7fa971a09700 1 -- 172.21.15.70:0/110258870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee80074690 con 0x55ee80186860 2024-06-04T05:06:44.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.133+0000 7fa971a09700 1 --2- 172.21.15.70:0/110258870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee80186860 0x55ee801923b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa9680a9580 tx=0x7fa968005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e8d8585c03c8bba server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:44.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.133+0000 7fa9557fa700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:06:44.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.133+0000 7fa9557fa700 1 -- 172.21.15.70:0/110258870 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9680c8060 con 0x55ee80186860 2024-06-04T05:06:44.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.133+0000 7fa9557fa700 1 -- 172.21.15.70:0/110258870 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7fa9680c3070 con 0x55ee80186860 2024-06-04T05:06:44.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.133+0000 7fa97e502b80 1 -- 172.21.15.70:0/110258870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9680b2450 con 0x55ee80186860 2024-06-04T05:06:44.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.133+0000 7fa9557fa700 1 -- 172.21.15.70:0/110258870 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa968088050 con 0x55ee80186860 2024-06-04T05:06:44.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.133+0000 7fa97e502b80 10 client.27134 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:44.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.133+0000 7fa97e502b80 1 -- 172.21.15.70:0/110258870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ee8019d820 con 0x55ee80186860 2024-06-04T05:06:44.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.133+0000 7fa9557fa700 1 -- 172.21.15.70:0/110258870 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa9680b2450 con 0x55ee80186860 2024-06-04T05:06:44.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.133+0000 7fa9557fa700 1 -- 172.21.15.70:0/110258870 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fa9680b1fc0 con 0x55ee80186860 2024-06-04T05:06:44.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.133+0000 7fa97e502b80 10 client.27134 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:44.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.133+0000 7fa97e502b80 10 client.27134 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:06:44.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.133+0000 7fa97e502b80 20 client.27134 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:44.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.134+0000 7fa97e502b80 1 --2- 172.21.15.70:0/110258870 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55ee8019fcf0 0x55ee801a20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.134+0000 7fa97e502b80 4 client.27134 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:06:44.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.134+0000 7fa97e502b80 1 -- 172.21.15.70:0/110258870 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee801529e0 con 0x55ee8019fcf0 2024-06-04T05:06:44.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.134+0000 7fa971208700 1 --2- 172.21.15.70:0/110258870 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55ee8019fcf0 0x55ee801a20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:44.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.134+0000 7fa9557fa700 10 client.27134 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:06:44.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.134+0000 7fa971208700 1 --2- 172.21.15.70:0/110258870 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55ee8019fcf0 0x55ee801a20e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a77cf0ec59692135 server_cookie=c6832ea102cb39a7 in_seq=0 out_seq=0 2024-06-04T05:06:44.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.135+0000 7fa9557fa700 1 -- 172.21.15.70:0/110258870 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ee801529e0 con 0x55ee8019fcf0 2024-06-04T05:06:44.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.135+0000 7fa9557fa700 10 client.27134 handle_command_reply: tid=0 2024-06-04T05:06:44.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.135+0000 7fa97e502b80 1 client.27134 shutdown 2024-06-04T05:06:44.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.135+0000 7fa97e502b80 1 -- 172.21.15.70:0/110258870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee80186860 msgr2=0x55ee801923b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:44.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.135+0000 7fa97e502b80 1 --2- 172.21.15.70:0/110258870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee80186860 0x55ee801923b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa9680a9580 tx=0x7fa968005850 comp rx=0 tx=0).stop 2024-06-04T05:06:44.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.135+0000 7fa97e502b80 1 -- 172.21.15.70:0/110258870 shutdown_connections 2024-06-04T05:06:44.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.135+0000 7fa97e502b80 1 --2- 172.21.15.70:0/110258870 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55ee8019fcf0 0x55ee801a20e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.135+0000 7fa9557fa700 0 client.27134 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:06:44.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.135+0000 7fa97e502b80 1 --2- 172.21.15.70:0/110258870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee8018af60 0x55ee8018fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.135+0000 7fa97e502b80 1 --2- 172.21.15.70:0/110258870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee80186860 0x55ee801923b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.135+0000 7fa97e502b80 1 --2- 172.21.15.70:0/110258870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 0x55ee8018f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.135+0000 7fa97e502b80 1 -- 172.21.15.70:0/110258870 >> 172.21.15.70:0/110258870 conn(0x55ee800b4080 msgr2=0x55ee80180510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:44.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.136+0000 7fa97e502b80 1 -- 172.21.15.70:0/110258870 shutdown_connections 2024-06-04T05:06:44.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.136+0000 7fa97e502b80 1 -- 172.21.15.70:0/110258870 wait complete. 2024-06-04T05:06:44.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.136+0000 7fa97e502b80 20 client.27134 trim_cache size 0 max 16384 2024-06-04T05:06:44.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa97e502b80 1 Processor -- start 2024-06-04T05:06:44.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa97e502b80 1 -- start start 2024-06-04T05:06:44.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa97e502b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 0x55ee801915f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa97e502b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 0x55ee80180c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa97e502b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee8018af60 0x55ee801811d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa97e502b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1adec0 con 0x55ee80187210 2024-06-04T05:06:44.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa97e502b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1afa00 con 0x55ee8018af60 2024-06-04T05:06:44.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa97e502b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1ae590 con 0x55ee80186860 2024-06-04T05:06:44.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa971208700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 0x55ee80180c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:44.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa971208700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 0x55ee80180c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50896/0 (socket says 172.21.15.70:50896) 2024-06-04T05:06:44.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa971208700 1 -- 172.21.15.70:0/3033826971 learned_addr learned my addr 172.21.15.70:0/3033826971 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:44.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa971a09700 1 --2- 172.21.15.70:0/3033826971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 0x55ee801915f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:44.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa97220a700 1 --2- 172.21.15.70:0/3033826971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee8018af60 0x55ee801811d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:44.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa971208700 1 -- 172.21.15.70:0/3033826971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 msgr2=0x55ee801915f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:44.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa971208700 1 --2- 172.21.15.70:0/3033826971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 0x55ee801915f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa971208700 1 -- 172.21.15.70:0/3033826971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee8018af60 msgr2=0x55ee801811d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:44.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa971208700 1 --2- 172.21.15.70:0/3033826971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee8018af60 0x55ee801811d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.154+0000 7fa971208700 1 -- 172.21.15.70:0/3033826971 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee80190560 con 0x55ee80187210 2024-06-04T05:06:44.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.155+0000 7fa971208700 1 --2- 172.21.15.70:0/3033826971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 0x55ee80180c90 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009990 tx=0x7fa95c01a070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa4fde4a19df02e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:44.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.155+0000 7fa9577fe700 1 -- 172.21.15.70:0/3033826971 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c00b990 con 0x55ee80187210 2024-06-04T05:06:44.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.155+0000 7fa9577fe700 1 -- 172.21.15.70:0/3033826971 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa95c023460 con 0x55ee80187210 2024-06-04T05:06:44.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.155+0000 7fa9577fe700 1 -- 172.21.15.70:0/3033826971 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c0106d0 con 0x55ee80187210 2024-06-04T05:06:44.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.156+0000 7fa97e502b80 1 -- 172.21.15.70:0/3033826971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 msgr2=0x55ee80180c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:44.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.156+0000 7fa97e502b80 1 --2- 172.21.15.70:0/3033826971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 0x55ee80180c90 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009990 tx=0x7fa95c01a070 comp rx=0 tx=0).stop 2024-06-04T05:06:44.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.156+0000 7fa97e502b80 1 -- 172.21.15.70:0/3033826971 shutdown_connections 2024-06-04T05:06:44.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.156+0000 7fa97e502b80 1 --2- 172.21.15.70:0/3033826971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee8018af60 0x55ee801811d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.156+0000 7fa97e502b80 1 --2- 172.21.15.70:0/3033826971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 0x55ee801915f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.156+0000 7fa97e502b80 1 --2- 172.21.15.70:0/3033826971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee80187210 0x55ee80180c90 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.156+0000 7fa97e502b80 1 -- 172.21.15.70:0/3033826971 >> 172.21.15.70:0/3033826971 conn(0x55ee800b4080 msgr2=0x55ee80190ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:44.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.156+0000 7fa97e502b80 1 -- 172.21.15.70:0/3033826971 shutdown_connections 2024-06-04T05:06:44.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.156+0000 7fa97e502b80 1 -- 172.21.15.70:0/3033826971 wait complete. 2024-06-04T05:06:44.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.156+0000 7fa97e502b80 1 -- start start 2024-06-04T05:06:44.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.156+0000 7fa97e502b80 1 Processor -- start 2024-06-04T05:06:44.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa97e502b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 0x55ee80201960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa97e502b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee80187210 0x55ee80201ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa97e502b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8018af60 0x55ee801a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa97e502b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1af330 con 0x55ee8018af60 2024-06-04T05:06:44.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa97e502b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1aec60 con 0x55ee80187210 2024-06-04T05:06:44.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa97e502b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1abe80 con 0x55ee80186860 2024-06-04T05:06:44.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa97e502b80 10 client.0 _ll_register_callbacks cb 0x55ee80070920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:44.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa971a09700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 0x55ee80201960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:44.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa971208700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee80187210 0x55ee80201ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:44.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa971a09700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 0x55ee80201960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:44574/0 (socket says 172.21.15.70:44574) 2024-06-04T05:06:44.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa971208700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee80187210 0x55ee80201ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:38026/0 (socket says 172.21.15.70:38026) 2024-06-04T05:06:44.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa971a09700 1 -- 172.21.15.70:0/848741558 learned_addr learned my addr 172.21.15.70:0/848741558 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:44.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa97220a700 1 --2- 172.21.15.70:0/848741558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8018af60 0x55ee801a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:44.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa971a09700 1 -- 172.21.15.70:0/848741558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee80187210 msgr2=0x55ee80201ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:44.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa971a09700 1 --2- 172.21.15.70:0/848741558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee80187210 0x55ee80201ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa971a09700 1 -- 172.21.15.70:0/848741558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8018af60 msgr2=0x55ee801a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:44.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa971a09700 1 --2- 172.21.15.70:0/848741558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8018af60 0x55ee801a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa971a09700 1 -- 172.21.15.70:0/848741558 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee801a50f0 con 0x55ee80186860 2024-06-04T05:06:44.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa97220a700 1 --2- 172.21.15.70:0/848741558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8018af60 0x55ee801a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:44.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa971208700 1 --2- 172.21.15.70:0/848741558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee80187210 0x55ee80201ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:44.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa971a09700 1 --2- 172.21.15.70:0/848741558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 0x55ee80201960 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa96800d8d0 tx=0x7fa9680dffc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f15b5b5b8442127 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:44.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.157+0000 7fa9557fa700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:06:44.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.158+0000 7fa9557fa700 1 -- 172.21.15.70:0/848741558 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9680c8040 con 0x55ee80186860 2024-06-04T05:06:44.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.158+0000 7fa97e502b80 1 -- 172.21.15.70:0/848741558 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa95c023460 con 0x55ee80186860 2024-06-04T05:06:44.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.158+0000 7fa9557fa700 1 -- 172.21.15.70:0/848741558 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7fa9680c3040 con 0x55ee80186860 2024-06-04T05:06:44.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.158+0000 7fa9557fa700 1 -- 172.21.15.70:0/848741558 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa96806d030 con 0x55ee80186860 2024-06-04T05:06:44.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.158+0000 7fa97e502b80 10 client.27223 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:44.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.158+0000 7fa97e502b80 1 -- 172.21.15.70:0/848741558 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ee8020a1f0 con 0x55ee80186860 2024-06-04T05:06:44.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.158+0000 7fa9557fa700 1 -- 172.21.15.70:0/848741558 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa95c023460 con 0x55ee80186860 2024-06-04T05:06:44.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.158+0000 7fa9557fa700 1 -- 172.21.15.70:0/848741558 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fa9680a9850 con 0x55ee80186860 2024-06-04T05:06:44.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.158+0000 7fa97e502b80 10 client.27223 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:44.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.158+0000 7fa97e502b80 10 client.27223 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:06:44.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.158+0000 7fa97e502b80 20 client.27223 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:44.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.158+0000 7fa97e502b80 1 --2- 172.21.15.70:0/848741558 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55ee8019fcf0 0x55ee8020c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:44.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.158+0000 7fa97e502b80 4 client.27223 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:06:44.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.158+0000 7fa97e502b80 1 -- 172.21.15.70:0/848741558 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ee800e7be0 con 0x55ee8019fcf0 2024-06-04T05:06:44.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.158+0000 7fa971208700 1 --2- 172.21.15.70:0/848741558 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55ee8019fcf0 0x55ee8020c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:44.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.159+0000 7fa9557fa700 10 client.27223 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:06:44.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.159+0000 7fa971208700 1 --2- 172.21.15.70:0/848741558 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55ee8019fcf0 0x55ee8020c640 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=64524f199e469905 server_cookie=5733514e77720c3c in_seq=0 out_seq=0 2024-06-04T05:06:44.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.159+0000 7fa9557fa700 1 -- 172.21.15.70:0/848741558 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ee800e7be0 con 0x55ee8019fcf0 2024-06-04T05:06:44.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.159+0000 7fa9557fa700 10 client.27223 handle_command_reply: tid=0 2024-06-04T05:06:44.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.159+0000 7fa97e502b80 1 client.27223 shutdown 2024-06-04T05:06:44.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.159+0000 7fa97e502b80 1 -- 172.21.15.70:0/848741558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 msgr2=0x55ee80201960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:44.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.159+0000 7fa97e502b80 1 --2- 172.21.15.70:0/848741558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 0x55ee80201960 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa96800d8d0 tx=0x7fa9680dffc0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.159+0000 7fa97e502b80 1 -- 172.21.15.70:0/848741558 shutdown_connections 2024-06-04T05:06:44.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.159+0000 7fa97e502b80 1 --2- 172.21.15.70:0/848741558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8018af60 0x55ee801a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.159+0000 7fa97e502b80 1 --2- 172.21.15.70:0/848741558 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55ee8019fcf0 0x55ee8020c640 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.159+0000 7fa97e502b80 1 --2- 172.21.15.70:0/848741558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee80186860 0x55ee80201960 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.159+0000 7fa97e502b80 1 --2- 172.21.15.70:0/848741558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee80187210 0x55ee80201ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.159+0000 7fa9557fa700 0 client.27223 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:06:44.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.159+0000 7fa97e502b80 1 -- 172.21.15.70:0/848741558 >> 172.21.15.70:0/848741558 conn(0x55ee800b4080 msgr2=0x55ee80185f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:44.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.159+0000 7fa97e502b80 1 -- 172.21.15.70:0/848741558 shutdown_connections 2024-06-04T05:06:44.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.160+0000 7fa97e502b80 1 -- 172.21.15.70:0/848741558 wait complete. 2024-06-04T05:06:44.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.160+0000 7fa97e502b80 20 client.27223 trim_cache size 0 max 16384 2024-06-04T05:06:44.168 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:06:44.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.160+0000 7fa973c6d700 1 -- 172.21.15.70:0/3533866758 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa9580b29f0 msgr2=0x7fa9580b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:44.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.160+0000 7fa973c6d700 1 --2- 172.21.15.70:0/3533866758 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa9580b29f0 0x7fa9580b4eb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa96c0ffc40 tx=0x7fa95c005fb0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.160+0000 7fa973c6d700 1 -- 172.21.15.70:0/3533866758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101b30 msgr2=0x7fa96c19f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:44.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.160+0000 7fa973c6d700 1 --2- 172.21.15.70:0/3533866758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101b30 0x7fa96c19f750 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa96800d900 tx=0x7fa96800dc10 comp rx=0 tx=0).stop 2024-06-04T05:06:44.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.160+0000 7fa973c6d700 1 -- 172.21.15.70:0/3533866758 shutdown_connections 2024-06-04T05:06:44.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.160+0000 7fa973c6d700 1 --2- 172.21.15.70:0/3533866758 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa9580b29f0 0x7fa9580b4eb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.160+0000 7fa973c6d700 1 --2- 172.21.15.70:0/3533866758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa96c109520 0x7fa96c199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.160+0000 7fa973c6d700 1 --2- 172.21.15.70:0/3533866758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101b30 0x7fa96c19f750 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.160+0000 7fa973c6d700 1 --2- 172.21.15.70:0/3533866758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa96c1024f0 0x7fa96c19fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:44.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.160+0000 7fa973c6d700 1 -- 172.21.15.70:0/3533866758 >> 172.21.15.70:0/3533866758 conn(0x7fa96c078ab0 msgr2=0x7fa96c1003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:44.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.160+0000 7fa973c6d700 1 -- 172.21.15.70:0/3533866758 shutdown_connections 2024-06-04T05:06:44.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:44.161+0000 7fa973c6d700 1 -- 172.21.15.70:0/3533866758 wait complete. 2024-06-04T05:06:44.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 162921 1.88 MB/sec execute 337 sec latency 15.079 ms 2024-06-04T05:06:44.885 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:06:45.157 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:06:45.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:45 smithi070 ceph-mon[32631]: pgmap v294: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-06-04T05:06:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:45 smithi112 ceph-mon[36453]: pgmap v294: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-06-04T05:06:45.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 163343 1.88 MB/sec execute 338 sec latency 14.673 ms 2024-06-04T05:06:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:45 smithi098 ceph-mon[35569]: pgmap v294: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-06-04T05:06:46.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 163609 1.88 MB/sec execute 339 sec latency 38.591 ms 2024-06-04T05:06:46.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.888+0000 7f7cea732700 1 -- 172.21.15.70:0/1751334810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ce41084b0 msgr2=0x7f7ce4108890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:46.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.888+0000 7f7cea732700 1 --2- 172.21.15.70:0/1751334810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ce41084b0 0x7f7ce4108890 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009990 tx=0x7f7cd4009ca0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.888+0000 7f7cea732700 1 -- 172.21.15.70:0/1751334810 shutdown_connections 2024-06-04T05:06:46.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.888+0000 7f7cea732700 1 --2- 172.21.15.70:0/1751334810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce4102e30 0x7f7ce410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.888+0000 7f7cea732700 1 --2- 172.21.15.70:0/1751334810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ce41084b0 0x7f7ce4108890 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.888+0000 7f7cea732700 1 --2- 172.21.15.70:0/1751334810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ce4102470 0x7f7ce41028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.888+0000 7f7cea732700 1 -- 172.21.15.70:0/1751334810 >> 172.21.15.70:0/1751334810 conn(0x7f7ce4079410 msgr2=0x7f7ce4079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:46.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.888+0000 7f7cea732700 1 -- 172.21.15.70:0/1751334810 shutdown_connections 2024-06-04T05:06:46.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.888+0000 7f7cea732700 1 -- 172.21.15.70:0/1751334810 wait complete. 2024-06-04T05:06:46.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.888+0000 7f7cea732700 1 Processor -- start 2024-06-04T05:06:46.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.889+0000 7f7cea732700 1 -- start start 2024-06-04T05:06:46.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.889+0000 7f7cea732700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce4102470 0x7f7ce419f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.889+0000 7f7cea732700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ce4102e30 0x7f7ce419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.889+0000 7f7cea732700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ce41084b0 0x7f7ce4199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.889+0000 7f7cea732700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce4112490 con 0x7f7ce41084b0 2024-06-04T05:06:46.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.889+0000 7f7cea732700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce4112310 con 0x7f7ce4102470 2024-06-04T05:06:46.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.889+0000 7f7cea732700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce4112610 con 0x7f7ce4102e30 2024-06-04T05:06:46.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.889+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce4102470 0x7f7ce419f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:46.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.889+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce4102470 0x7f7ce419f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54648/0 (socket says 172.21.15.70:54648) 2024-06-04T05:06:46.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.889+0000 7f7ce3fff700 1 -- 172.21.15.70:0/2252119683 learned_addr learned my addr 172.21.15.70:0/2252119683 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:46.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.889+0000 7f7ce8ccf700 1 --2- 172.21.15.70:0/2252119683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ce41084b0 0x7f7ce4199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:46.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.889+0000 7f7ce37fe700 1 --2- 172.21.15.70:0/2252119683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ce4102e30 0x7f7ce419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:46.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.889+0000 7f7ce3fff700 1 -- 172.21.15.70:0/2252119683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ce4102e30 msgr2=0x7f7ce419fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:46.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.889+0000 7f7ce3fff700 1 --2- 172.21.15.70:0/2252119683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ce4102e30 0x7f7ce419fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.890+0000 7f7ce3fff700 1 -- 172.21.15.70:0/2252119683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ce41084b0 msgr2=0x7f7ce4199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:46.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.890+0000 7f7ce3fff700 1 --2- 172.21.15.70:0/2252119683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ce41084b0 0x7f7ce4199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.890+0000 7f7ce3fff700 1 -- 172.21.15.70:0/2252119683 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce419a0d0 con 0x7f7ce4102470 2024-06-04T05:06:46.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.890+0000 7f7ce3fff700 1 --2- 172.21.15.70:0/2252119683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce4102470 0x7f7ce419f760 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4006010 tx=0x7f7cd400bfc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:46.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.890+0000 7f7ce17fa700 1 -- 172.21.15.70:0/2252119683 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4027070 con 0x7f7ce4102470 2024-06-04T05:06:46.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.890+0000 7f7ce17fa700 1 -- 172.21.15.70:0/2252119683 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cd4022470 con 0x7f7ce4102470 2024-06-04T05:06:46.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.890+0000 7f7cea732700 1 -- 172.21.15.70:0/2252119683 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce419a360 con 0x7f7ce4102470 2024-06-04T05:06:46.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.890+0000 7f7ce17fa700 1 -- 172.21.15.70:0/2252119683 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4017650 con 0x7f7ce4102470 2024-06-04T05:06:46.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.890+0000 7f7cea732700 1 -- 172.21.15.70:0/2252119683 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce41a6630 con 0x7f7ce4102470 2024-06-04T05:06:46.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.892+0000 7f7ce17fa700 1 -- 172.21.15.70:0/2252119683 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7cd4009d10 con 0x7f7ce4102470 2024-06-04T05:06:46.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.893+0000 7f7ce17fa700 1 --2- 172.21.15.70:0/2252119683 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ccc0b29f0 0x7f7ccc0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.893+0000 7f7ce37fe700 1 --2- 172.21.15.70:0/2252119683 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ccc0b29f0 0x7f7ccc0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:46.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.893+0000 7f7ce17fa700 1 -- 172.21.15.70:0/2252119683 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f7cd40e3110 con 0x7f7ce4102470 2024-06-04T05:06:46.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.894+0000 7f7cf4fc7b80 1 Processor -- start 2024-06-04T05:06:46.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.894+0000 7f7cf4fc7b80 1 -- start start 2024-06-04T05:06:46.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.894+0000 7f7cf4fc7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb87c730 0x5609eb87cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.894+0000 7f7cf4fc7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb876cb0 0x5609eb877130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.894+0000 7f7cf4fc7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb877670 0x5609eb883490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.894+0000 7f7cf4fc7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5609eb7d0d70 con 0x5609eb876cb0 2024-06-04T05:06:46.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.894+0000 7f7cf4fc7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5609eb7c0910 con 0x5609eb87c730 2024-06-04T05:06:46.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.894+0000 7f7cf4fc7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5609eb7c4950 con 0x5609eb877670 2024-06-04T05:06:46.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.894+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb87c730 0x5609eb87cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:46.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.894+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb87c730 0x5609eb87cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54662/0 (socket says 172.21.15.70:54662) 2024-06-04T05:06:46.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.894+0000 7f7ce3fff700 1 -- 172.21.15.70:0/3699336046 learned_addr learned my addr 172.21.15.70:0/3699336046 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:46.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.894+0000 7f7ce8ccf700 1 --2- 172.21.15.70:0/3699336046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb877670 0x5609eb883490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:46.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.895+0000 7f7ce3fff700 1 -- 172.21.15.70:0/3699336046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb877670 msgr2=0x5609eb883490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:46.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.895+0000 7f7ce3fff700 1 --2- 172.21.15.70:0/3699336046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb877670 0x5609eb883490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.895+0000 7f7ce3fff700 1 -- 172.21.15.70:0/3699336046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb876cb0 msgr2=0x5609eb877130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:46.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.895+0000 7f7ce3fff700 1 --2- 172.21.15.70:0/3699336046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb876cb0 0x5609eb877130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.895+0000 7f7ce3fff700 1 -- 172.21.15.70:0/3699336046 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609eb769fb0 con 0x5609eb87c730 2024-06-04T05:06:46.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.895+0000 7f7ce37fe700 1 --2- 172.21.15.70:0/2252119683 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ccc0b29f0 0x7f7ccc0b4eb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7ce406cce0 tx=0x7f7cd8009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:46.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.895+0000 7f7ce3fff700 1 --2- 172.21.15.70:0/3699336046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb87c730 0x5609eb87cb10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40aa1f0 tx=0x7f7cd40aa0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8945c366612d5b5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:46.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.895+0000 7f7cca7fc700 1 -- 172.21.15.70:0/3699336046 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd40aa4f0 con 0x5609eb87c730 2024-06-04T05:06:46.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.895+0000 7f7cca7fc700 1 -- 172.21.15.70:0/3699336046 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cd40ae8a0 con 0x5609eb87c730 2024-06-04T05:06:46.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.895+0000 7f7cca7fc700 1 -- 172.21.15.70:0/3699336046 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd40b2e30 con 0x5609eb87c730 2024-06-04T05:06:46.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.896+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/3699336046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb87c730 msgr2=0x5609eb87cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:46.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.896+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/3699336046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb87c730 0x5609eb87cb10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40aa1f0 tx=0x7f7cd40aa0d0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.896+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/3699336046 shutdown_connections 2024-06-04T05:06:46.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.896+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/3699336046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb877670 0x5609eb883490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.896+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/3699336046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb87c730 0x5609eb87cb10 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.896+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/3699336046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb876cb0 0x5609eb877130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.896+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/3699336046 >> 172.21.15.70:0/3699336046 conn(0x5609eb79e240 msgr2=0x5609eb86d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:46.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.896+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/3699336046 shutdown_connections 2024-06-04T05:06:46.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.896+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/3699336046 wait complete. 2024-06-04T05:06:46.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.897+0000 7f7cf4fc7b80 1 -- start start 2024-06-04T05:06:46.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.897+0000 7f7cf4fc7b80 1 Processor -- start 2024-06-04T05:06:46.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.897+0000 7f7cf4fc7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb876cb0 0x5609eb883210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.897+0000 7f7cf4fc7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb877670 0x5609eb880510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.897+0000 7f7cf4fc7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb87c730 0x5609eb880a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.897+0000 7f7cf4fc7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5609eb6c6fa0 con 0x5609eb87c730 2024-06-04T05:06:46.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.897+0000 7f7cf4fc7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5609eb728da0 con 0x5609eb876cb0 2024-06-04T05:06:46.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.897+0000 7f7cf4fc7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5609eb7c0910 con 0x5609eb877670 2024-06-04T05:06:46.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.897+0000 7f7cf4fc7b80 10 client.0 _ll_register_callbacks cb 0x5609eb75b950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:46.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.897+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb876cb0 0x5609eb883210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:46.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.897+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb876cb0 0x5609eb883210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54674/0 (socket says 172.21.15.70:54674) 2024-06-04T05:06:46.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.897+0000 7f7ce3fff700 1 -- 172.21.15.70:0/1409378501 learned_addr learned my addr 172.21.15.70:0/1409378501 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:46.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.897+0000 7f7ce8ccf700 1 --2- 172.21.15.70:0/1409378501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb87c730 0x5609eb880a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:46.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.897+0000 7f7ce37fe700 1 --2- 172.21.15.70:0/1409378501 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb877670 0x5609eb880510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:46.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.898+0000 7f7ce3fff700 1 -- 172.21.15.70:0/1409378501 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb877670 msgr2=0x5609eb880510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:46.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.898+0000 7f7ce3fff700 1 --2- 172.21.15.70:0/1409378501 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb877670 0x5609eb880510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.898+0000 7f7ce3fff700 1 -- 172.21.15.70:0/1409378501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb87c730 msgr2=0x5609eb880a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:46.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.898+0000 7f7ce3fff700 1 --2- 172.21.15.70:0/1409378501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb87c730 0x5609eb880a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.898+0000 7f7ce3fff700 1 -- 172.21.15.70:0/1409378501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609eb7108d0 con 0x5609eb876cb0 2024-06-04T05:06:46.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.898+0000 7f7ce3fff700 1 --2- 172.21.15.70:0/1409378501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb876cb0 0x5609eb883210 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40a7760 tx=0x7f7cd40aaa30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7777599a9b895e6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:46.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.898+0000 7f7cc37fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:06:46.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.898+0000 7f7cc37fe700 1 -- 172.21.15.70:0/1409378501 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd40c3020 con 0x5609eb876cb0 2024-06-04T05:06:46.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.898+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/1409378501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cd40ae8a0 con 0x5609eb876cb0 2024-06-04T05:06:46.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.898+0000 7f7cc37fe700 1 -- 172.21.15.70:0/1409378501 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f7cd40b3040 con 0x5609eb876cb0 2024-06-04T05:06:46.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.898+0000 7f7cc37fe700 1 -- 172.21.15.70:0/1409378501 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7cd40b32c0 con 0x5609eb876cb0 2024-06-04T05:06:46.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.898+0000 7f7cf4fc7b80 10 client.27152 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:46.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.898+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/1409378501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5609eb77a760 con 0x5609eb876cb0 2024-06-04T05:06:46.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.898+0000 7f7cc37fe700 1 -- 172.21.15.70:0/1409378501 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cd40ae8a0 con 0x5609eb876cb0 2024-06-04T05:06:46.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.899+0000 7f7cc37fe700 1 -- 172.21.15.70:0/1409378501 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f7cd40cb310 con 0x5609eb876cb0 2024-06-04T05:06:46.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.899+0000 7f7cf4fc7b80 10 client.27152 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:46.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.899+0000 7f7cf4fc7b80 10 client.27152 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:06:46.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.899+0000 7f7cf4fc7b80 20 client.27152 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:46.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.899+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/1409378501 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5609eb8902e0 0x5609eb8926d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.899+0000 7f7cf4fc7b80 4 client.27152 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:06:46.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.899+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/1409378501 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609eb7d2cc0 con 0x5609eb8902e0 2024-06-04T05:06:46.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.899+0000 7f7ce37fe700 1 --2- 172.21.15.70:0/1409378501 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5609eb8902e0 0x5609eb8926d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:46.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.899+0000 7f7cc37fe700 10 client.27152 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:06:46.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.899+0000 7f7ce37fe700 1 --2- 172.21.15.70:0/1409378501 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5609eb8902e0 0x5609eb8926d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7b4a7a954043b66d server_cookie=d083525d791f0de6 in_seq=0 out_seq=0 2024-06-04T05:06:46.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.900+0000 7f7cc37fe700 1 -- 172.21.15.70:0/1409378501 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5609eb7d2cc0 con 0x5609eb8902e0 2024-06-04T05:06:46.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.900+0000 7f7cc37fe700 10 client.27152 handle_command_reply: tid=0 2024-06-04T05:06:46.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.900+0000 7f7cf4fc7b80 1 client.27152 shutdown 2024-06-04T05:06:46.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.900+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/1409378501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb876cb0 msgr2=0x5609eb883210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:46.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.900+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/1409378501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb876cb0 0x5609eb883210 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40a7760 tx=0x7f7cd40aaa30 comp rx=0 tx=0).stop 2024-06-04T05:06:46.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.900+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/1409378501 shutdown_connections 2024-06-04T05:06:46.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.900+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/1409378501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb87c730 0x5609eb880a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.900+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/1409378501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb876cb0 0x5609eb883210 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.900+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/1409378501 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5609eb8902e0 0x5609eb8926d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.900+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/1409378501 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb877670 0x5609eb880510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.900+0000 7f7cc37fe700 0 client.27152 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:06:46.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.900+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/1409378501 >> 172.21.15.70:0/1409378501 conn(0x5609eb79e240 msgr2=0x5609eb86caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:46.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.900+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/1409378501 shutdown_connections 2024-06-04T05:06:46.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.901+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/1409378501 wait complete. 2024-06-04T05:06:46.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.901+0000 7f7cf4fc7b80 20 client.27152 trim_cache size 0 max 16384 2024-06-04T05:06:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.920+0000 7f7cf4fc7b80 1 Processor -- start 2024-06-04T05:06:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.920+0000 7f7cf4fc7b80 1 -- start start 2024-06-04T05:06:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.920+0000 7f7cf4fc7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb876cb0 0x5609eb87cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.920+0000 7f7cf4fc7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb877670 0x5609eb86d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.920+0000 7f7cf4fc7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb87c730 0x5609eb87f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.920+0000 7f7cf4fc7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce41abb30 con 0x5609eb87c730 2024-06-04T05:06:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.920+0000 7f7cf4fc7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce41ad670 con 0x5609eb877670 2024-06-04T05:06:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.920+0000 7f7cf4fc7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce41ac200 con 0x5609eb876cb0 2024-06-04T05:06:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.920+0000 7f7ce8ccf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb87c730 0x5609eb87f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.921+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb876cb0 0x5609eb87cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.921+0000 7f7ce37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb877670 0x5609eb86d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.921+0000 7f7ce8ccf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb87c730 0x5609eb87f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40692/0 (socket says 172.21.15.70:40692) 2024-06-04T05:06:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.921+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb876cb0 0x5609eb87cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:58854/0 (socket says 172.21.15.70:58854) 2024-06-04T05:06:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.921+0000 7f7ce8ccf700 1 -- 172.21.15.70:0/3686163233 learned_addr learned my addr 172.21.15.70:0/3686163233 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.921+0000 7f7ce8ccf700 1 -- 172.21.15.70:0/3686163233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb876cb0 msgr2=0x5609eb87cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.921+0000 7f7ce8ccf700 1 --2- 172.21.15.70:0/3686163233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb876cb0 0x5609eb87cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.921+0000 7f7ce8ccf700 1 -- 172.21.15.70:0/3686163233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb877670 msgr2=0x5609eb86d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.921+0000 7f7ce8ccf700 1 --2- 172.21.15.70:0/3686163233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb877670 0x5609eb86d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.921+0000 7f7ce8ccf700 1 -- 172.21.15.70:0/3686163233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609eb87c110 con 0x5609eb87c730 2024-06-04T05:06:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.921+0000 7f7ce3fff700 1 --2- 172.21.15.70:0/3686163233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb876cb0 0x5609eb87cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.921+0000 7f7ce37fe700 1 --2- 172.21.15.70:0/3686163233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb877670 0x5609eb86d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.921+0000 7f7ce8ccf700 1 --2- 172.21.15.70:0/3686163233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb87c730 0x5609eb87f400 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009d50 tx=0x7f7cdc009d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=954662382a9d70d9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.921+0000 7f7cc9ffb700 1 -- 172.21.15.70:0/3686163233 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc007d80 con 0x5609eb87c730 2024-06-04T05:06:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.921+0000 7f7cc9ffb700 1 -- 172.21.15.70:0/3686163233 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cdc01b070 con 0x5609eb87c730 2024-06-04T05:06:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.921+0000 7f7cc9ffb700 1 -- 172.21.15.70:0/3686163233 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc017670 con 0x5609eb87c730 2024-06-04T05:06:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.922+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/3686163233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb87c730 msgr2=0x5609eb87f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.922+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/3686163233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb87c730 0x5609eb87f400 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009d50 tx=0x7f7cdc009d80 comp rx=0 tx=0).stop 2024-06-04T05:06:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.922+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/3686163233 shutdown_connections 2024-06-04T05:06:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.922+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/3686163233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb87c730 0x5609eb87f400 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.922+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/3686163233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb876cb0 0x5609eb87cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.922+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/3686163233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb877670 0x5609eb86d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.922+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/3686163233 >> 172.21.15.70:0/3686163233 conn(0x5609eb79e240 msgr2=0x5609eb87ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.922+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/3686163233 shutdown_connections 2024-06-04T05:06:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.922+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/3686163233 wait complete. 2024-06-04T05:06:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.923+0000 7f7cf4fc7b80 1 -- start start 2024-06-04T05:06:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.923+0000 7f7cf4fc7b80 1 Processor -- start 2024-06-04T05:06:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.923+0000 7f7cf4fc7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb876cb0 0x5609eb890800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.923+0000 7f7cf4fc7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb877670 0x5609eb890d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.923+0000 7f7cf4fc7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb87c730 0x5609eb8f8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.923+0000 7f7cf4fc7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce41acfa0 con 0x5609eb87c730 2024-06-04T05:06:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.923+0000 7f7cf4fc7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce41ac8d0 con 0x5609eb876cb0 2024-06-04T05:06:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.923+0000 7f7cf4fc7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce41a9b10 con 0x5609eb877670 2024-06-04T05:06:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.923+0000 7f7cf4fc7b80 10 client.0 _ll_register_callbacks cb 0x5609eb75b950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.923+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb876cb0 0x5609eb890800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.923+0000 7f7ce8ccf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb87c730 0x5609eb8f8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.923+0000 7f7ce37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb877670 0x5609eb890d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.923+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb876cb0 0x5609eb890800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54696/0 (socket says 172.21.15.70:54696) 2024-06-04T05:06:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.923+0000 7f7ce3fff700 1 -- 172.21.15.70:0/2769411586 learned_addr learned my addr 172.21.15.70:0/2769411586 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.924+0000 7f7ce3fff700 1 -- 172.21.15.70:0/2769411586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb877670 msgr2=0x5609eb890d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.924+0000 7f7ce3fff700 1 --2- 172.21.15.70:0/2769411586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb877670 0x5609eb890d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.924+0000 7f7ce3fff700 1 -- 172.21.15.70:0/2769411586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb87c730 msgr2=0x5609eb8f8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.924+0000 7f7ce3fff700 1 --2- 172.21.15.70:0/2769411586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb87c730 0x5609eb8f8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.924+0000 7f7ce3fff700 1 -- 172.21.15.70:0/2769411586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609eb8f9040 con 0x5609eb876cb0 2024-06-04T05:06:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.924+0000 7f7ce3fff700 1 --2- 172.21.15.70:0/2769411586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb876cb0 0x5609eb890800 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40059d0 tx=0x7f7cd40a6150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28769de6e5178694 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.924+0000 7f7cc37fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:06:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.924+0000 7f7cc37fe700 1 -- 172.21.15.70:0/2769411586 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd40c3070 con 0x5609eb876cb0 2024-06-04T05:06:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.924+0000 7f7cc37fe700 1 -- 172.21.15.70:0/2769411586 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f7cd40a94c0 con 0x5609eb876cb0 2024-06-04T05:06:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.924+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/2769411586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cdc01b070 con 0x5609eb876cb0 2024-06-04T05:06:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.924+0000 7f7cc37fe700 1 -- 172.21.15.70:0/2769411586 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7cd40aa4f0 con 0x5609eb876cb0 2024-06-04T05:06:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.924+0000 7f7cf4fc7b80 10 client.27158 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.924+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/2769411586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5609eb8f9040 con 0x5609eb876cb0 2024-06-04T05:06:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.924+0000 7f7cc37fe700 1 -- 172.21.15.70:0/2769411586 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cdc01b070 con 0x5609eb876cb0 2024-06-04T05:06:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.925+0000 7f7cc37fe700 1 -- 172.21.15.70:0/2769411586 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f7cd40b1b90 con 0x5609eb876cb0 2024-06-04T05:06:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.925+0000 7f7cf4fc7b80 10 client.27158 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.925+0000 7f7cf4fc7b80 10 client.27158 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:06:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.925+0000 7f7cf4fc7b80 20 client.27158 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.925+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/2769411586 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5609eb8fa530 0x5609eb8fc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.925+0000 7f7cf4fc7b80 4 client.27158 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:06:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.925+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/2769411586 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5609eb7cc4a0 con 0x5609eb8fa530 2024-06-04T05:06:46.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.925+0000 7f7ce37fe700 1 --2- 172.21.15.70:0/2769411586 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5609eb8fa530 0x5609eb8fc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.925+0000 7f7cc37fe700 10 client.27158 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:06:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.925+0000 7f7ce37fe700 1 --2- 172.21.15.70:0/2769411586 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5609eb8fa530 0x5609eb8fc920 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bd5b50f0f6d38816 server_cookie=eca23693e8d547a5 in_seq=0 out_seq=0 2024-06-04T05:06:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.925+0000 7f7cc37fe700 1 -- 172.21.15.70:0/2769411586 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5609eb7cc4a0 con 0x5609eb8fa530 2024-06-04T05:06:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.925+0000 7f7cc37fe700 10 client.27158 handle_command_reply: tid=0 2024-06-04T05:06:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.925+0000 7f7cf4fc7b80 1 client.27158 shutdown 2024-06-04T05:06:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.926+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/2769411586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb876cb0 msgr2=0x5609eb890800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.926+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/2769411586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb876cb0 0x5609eb890800 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40059d0 tx=0x7f7cd40a6150 comp rx=0 tx=0).stop 2024-06-04T05:06:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.926+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/2769411586 shutdown_connections 2024-06-04T05:06:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.926+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/2769411586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609eb87c730 0x5609eb8f8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.926+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/2769411586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609eb876cb0 0x5609eb890800 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.926+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/2769411586 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5609eb8fa530 0x5609eb8fc920 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.926+0000 7f7cf4fc7b80 1 --2- 172.21.15.70:0/2769411586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5609eb877670 0x5609eb890d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.926+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/2769411586 >> 172.21.15.70:0/2769411586 conn(0x5609eb79e240 msgr2=0x5609eb8825d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.926+0000 7f7cc37fe700 0 client.27158 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:06:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.926+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/2769411586 shutdown_connections 2024-06-04T05:06:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.926+0000 7f7cf4fc7b80 1 -- 172.21.15.70:0/2769411586 wait complete. 2024-06-04T05:06:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.926+0000 7f7cf4fc7b80 20 client.27158 trim_cache size 0 max 16384 2024-06-04T05:06:46.933 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:06:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.927+0000 7f7cea732700 1 -- 172.21.15.70:0/2252119683 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ccc0b29f0 msgr2=0x7f7ccc0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.927+0000 7f7cea732700 1 --2- 172.21.15.70:0/2252119683 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ccc0b29f0 0x7f7ccc0b4eb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7ce406cce0 tx=0x7f7cd8009260 comp rx=0 tx=0).stop 2024-06-04T05:06:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.927+0000 7f7cea732700 1 -- 172.21.15.70:0/2252119683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce4102470 msgr2=0x7f7ce419f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.927+0000 7f7cea732700 1 --2- 172.21.15.70:0/2252119683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce4102470 0x7f7ce419f760 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4006010 tx=0x7f7cd400bfc0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.927+0000 7f7cea732700 1 -- 172.21.15.70:0/2252119683 shutdown_connections 2024-06-04T05:06:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.927+0000 7f7cea732700 1 --2- 172.21.15.70:0/2252119683 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ccc0b29f0 0x7f7ccc0b4eb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.927+0000 7f7cea732700 1 --2- 172.21.15.70:0/2252119683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ce41084b0 0x7f7ce4199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.927+0000 7f7cea732700 1 --2- 172.21.15.70:0/2252119683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce4102470 0x7f7ce419f760 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.927+0000 7f7cea732700 1 --2- 172.21.15.70:0/2252119683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ce4102e30 0x7f7ce419fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.927+0000 7f7cea732700 1 -- 172.21.15.70:0/2252119683 >> 172.21.15.70:0/2252119683 conn(0x7f7ce4079410 msgr2=0x7f7ce4106440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:46.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.927+0000 7f7cea732700 1 -- 172.21.15.70:0/2252119683 shutdown_connections 2024-06-04T05:06:46.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:46.927+0000 7f7cea732700 1 -- 172.21.15.70:0/2252119683 wait complete. 2024-06-04T05:06:47.570 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:47 smithi070 ceph-mon[32631]: pgmap v295: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 2.1 MiB/s wr, 192 op/s 2024-06-04T05:06:47.598 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:06:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:47 smithi112 ceph-mon[36453]: pgmap v295: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 2.1 MiB/s wr, 192 op/s 2024-06-04T05:06:47.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 164007 1.88 MB/sec execute 340 sec latency 14.425 ms 2024-06-04T05:06:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:47 smithi098 ceph-mon[35569]: pgmap v295: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 2.1 MiB/s wr, 192 op/s 2024-06-04T05:06:47.858 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:06:48.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 164586 1.89 MB/sec execute 341 sec latency 20.552 ms 2024-06-04T05:06:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.872+0000 7f7d73b89640 1 -- 172.21.15.70:0/1244060984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d6c102340 msgr2=0x7f7d6c10e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.872+0000 7f7d73b89640 1 --2- 172.21.15.70:0/1244060984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d6c102340 0x7f7d6c10e800 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f7d64009cd0 tx=0x7f7d6402a8e0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.872+0000 7f7d73b89640 1 -- 172.21.15.70:0/1244060984 shutdown_connections 2024-06-04T05:06:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.873+0000 7f7d73b89640 1 --2- 172.21.15.70:0/1244060984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d6c102340 0x7f7d6c10e800 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.873+0000 7f7d73b89640 1 --2- 172.21.15.70:0/1244060984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d6c101980 0x7f7d6c101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.873+0000 7f7d73b89640 1 --2- 172.21.15.70:0/1244060984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d6c107940 0x7f7d6c107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.873+0000 7f7d73b89640 1 -- 172.21.15.70:0/1244060984 >> 172.21.15.70:0/1244060984 conn(0x7f7d6c078ad0 msgr2=0x7f7d6c078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.873+0000 7f7d73b89640 1 -- 172.21.15.70:0/1244060984 shutdown_connections 2024-06-04T05:06:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.873+0000 7f7d73b89640 1 -- 172.21.15.70:0/1244060984 wait complete. 2024-06-04T05:06:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.873+0000 7f7d73b89640 1 Processor -- start 2024-06-04T05:06:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.873+0000 7f7d73b89640 1 -- start start 2024-06-04T05:06:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.873+0000 7f7d73b89640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d6c101980 0x7f7d6c199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.873+0000 7f7d73b89640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d6c102340 0x7f7d6c1995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.873+0000 7f7d73b89640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d6c107940 0x7f7d6c19d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.873+0000 7f7d73b89640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c111ab0 con 0x7f7d6c101980 2024-06-04T05:06:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.873+0000 7f7d73b89640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c111930 con 0x7f7d6c107940 2024-06-04T05:06:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.873+0000 7f7d73b89640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c111c30 con 0x7f7d6c102340 2024-06-04T05:06:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.874+0000 7f7d71925640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d6c101980 0x7f7d6c199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.874+0000 7f7d71925640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d6c101980 0x7f7d6c199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40714/0 (socket says 172.21.15.70:40714) 2024-06-04T05:06:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.874+0000 7f7d71124640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d6c102340 0x7f7d6c1995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.874+0000 7f7d71925640 1 -- 172.21.15.70:0/668963161 learned_addr learned my addr 172.21.15.70:0/668963161 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.874+0000 7f7d72126640 1 --2- 172.21.15.70:0/668963161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d6c107940 0x7f7d6c19d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.874+0000 7f7d71925640 1 -- 172.21.15.70:0/668963161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d6c102340 msgr2=0x7f7d6c1995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.874+0000 7f7d71925640 1 --2- 172.21.15.70:0/668963161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d6c102340 0x7f7d6c1995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.874+0000 7f7d71925640 1 -- 172.21.15.70:0/668963161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d6c107940 msgr2=0x7f7d6c19d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.874+0000 7f7d71925640 1 --2- 172.21.15.70:0/668963161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d6c107940 0x7f7d6c19d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.874+0000 7f7d71925640 1 -- 172.21.15.70:0/668963161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6c19e080 con 0x7f7d6c101980 2024-06-04T05:06:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.874+0000 7f7d71925640 1 --2- 172.21.15.70:0/668963161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d6c101980 0x7f7d6c199090 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800e9f0 tx=0x7f7d6800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.874+0000 7f7d62ffd640 1 -- 172.21.15.70:0/668963161 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6800cd80 con 0x7f7d6c101980 2024-06-04T05:06:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.874+0000 7f7d73b89640 1 -- 172.21.15.70:0/668963161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d6c19e370 con 0x7f7d6c101980 2024-06-04T05:06:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.874+0000 7f7d62ffd640 1 -- 172.21.15.70:0/668963161 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d680108c0 con 0x7f7d6c101980 2024-06-04T05:06:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.874+0000 7f7d62ffd640 1 -- 172.21.15.70:0/668963161 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d680187c0 con 0x7f7d6c101980 2024-06-04T05:06:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.874+0000 7f7d73b89640 1 -- 172.21.15.70:0/668963161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d6c1a5c00 con 0x7f7d6c101980 2024-06-04T05:06:48.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.877+0000 7f7d62ffd640 1 -- 172.21.15.70:0/668963161 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7d68010430 con 0x7f7d6c101980 2024-06-04T05:06:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.877+0000 7f7d7697b740 1 Processor -- start 2024-06-04T05:06:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.878+0000 7f7d7697b740 1 -- start start 2024-06-04T05:06:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.878+0000 7f7d7697b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d339820 0x55a82d339c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.878+0000 7f7d7697b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d33a140 0x55a82d3444c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.878+0000 7f7d7697b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d344a00 0x55a82d346df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.878+0000 7f7d7697b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a82d2abc30 con 0x55a82d339820 2024-06-04T05:06:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.878+0000 7f7d7697b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a82d29b7d0 con 0x55a82d344a00 2024-06-04T05:06:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.878+0000 7f7d7697b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a82d29f810 con 0x55a82d33a140 2024-06-04T05:06:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.878+0000 7f7d62ffd640 1 --2- 172.21.15.70:0/668963161 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d580b2a10 0x7f7d580b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.878+0000 7f7d72126640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d339820 0x55a82d339c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.878+0000 7f7d72126640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d339820 0x55a82d339c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40720/0 (socket says 172.21.15.70:40720) 2024-06-04T05:06:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.878+0000 7f7d72126640 1 -- 172.21.15.70:0/2788660051 learned_addr learned my addr 172.21.15.70:0/2788660051 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.878+0000 7f7d71925640 1 --2- 172.21.15.70:0/2788660051 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d33a140 0x55a82d3444c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.878+0000 7f7d71124640 1 --2- 172.21.15.70:0/2788660051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d344a00 0x55a82d346df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.878+0000 7f7d72126640 1 --2- 172.21.15.70:0/668963161 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d580b2a10 0x7f7d580b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.879+0000 7f7d71124640 1 -- 172.21.15.70:0/2788660051 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d33a140 msgr2=0x55a82d3444c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.879+0000 7f7d62ffd640 1 -- 172.21.15.70:0/668963161 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f7d680e2970 con 0x7f7d6c101980 2024-06-04T05:06:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.879+0000 7f7d71124640 1 --2- 172.21.15.70:0/2788660051 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d33a140 0x55a82d3444c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.879+0000 7f7d71124640 1 -- 172.21.15.70:0/2788660051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d339820 msgr2=0x55a82d339c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.879+0000 7f7d71124640 1 --2- 172.21.15.70:0/2788660051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d339820 0x55a82d339c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.879+0000 7f7d71124640 1 -- 172.21.15.70:0/2788660051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a82cfb0210 con 0x55a82d344a00 2024-06-04T05:06:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.879+0000 7f7d71124640 1 --2- 172.21.15.70:0/2788660051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d344a00 0x55a82d346df0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c009bd0 tx=0x7f7d5c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c63ad04f659bffa8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.879+0000 7f7d72126640 1 --2- 172.21.15.70:0/668963161 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d580b2a10 0x7f7d580b4ed0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7d64031730 tx=0x7f7d64005f10 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.879+0000 7f7d57fff640 1 -- 172.21.15.70:0/2788660051 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c004230 con 0x55a82d344a00 2024-06-04T05:06:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.879+0000 7f7d57fff640 1 -- 172.21.15.70:0/2788660051 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d5c009e40 con 0x55a82d344a00 2024-06-04T05:06:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.879+0000 7f7d57fff640 1 -- 172.21.15.70:0/2788660051 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c004b60 con 0x55a82d344a00 2024-06-04T05:06:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.880+0000 7f7d7697b740 1 -- 172.21.15.70:0/2788660051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d344a00 msgr2=0x55a82d346df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.880+0000 7f7d7697b740 1 --2- 172.21.15.70:0/2788660051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d344a00 0x55a82d346df0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c009bd0 tx=0x7f7d5c011810 comp rx=0 tx=0).stop 2024-06-04T05:06:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.880+0000 7f7d7697b740 1 -- 172.21.15.70:0/2788660051 shutdown_connections 2024-06-04T05:06:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.880+0000 7f7d7697b740 1 --2- 172.21.15.70:0/2788660051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d344a00 0x55a82d346df0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.880+0000 7f7d7697b740 1 --2- 172.21.15.70:0/2788660051 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d33a140 0x55a82d3444c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.880+0000 7f7d7697b740 1 --2- 172.21.15.70:0/2788660051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d339820 0x55a82d339c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.880+0000 7f7d7697b740 1 -- 172.21.15.70:0/2788660051 >> 172.21.15.70:0/2788660051 conn(0x55a82d335d30 msgr2=0x55a82d336110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.880+0000 7f7d7697b740 1 -- 172.21.15.70:0/2788660051 shutdown_connections 2024-06-04T05:06:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.880+0000 7f7d7697b740 1 -- 172.21.15.70:0/2788660051 wait complete. 2024-06-04T05:06:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.880+0000 7f7d7697b740 1 -- start start 2024-06-04T05:06:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.880+0000 7f7d7697b740 1 Processor -- start 2024-06-04T05:06:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d7697b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d339820 0x55a82d341930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d7697b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d33a140 0x55a82d341e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d7697b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d344a00 0x55a82d3423b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d7697b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a82cf81790 con 0x55a82d33a140 2024-06-04T05:06:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d7697b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a82d257890 con 0x55a82d344a00 2024-06-04T05:06:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d7697b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a82d347630 con 0x55a82d339820 2024-06-04T05:06:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d7697b740 10 client.0 _ll_register_callbacks cb 0x55a82cfafa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d71124640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d33a140 0x55a82d341e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d71124640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d33a140 0x55a82d341e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40732/0 (socket says 172.21.15.70:40732) 2024-06-04T05:06:48.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d71124640 1 -- 172.21.15.70:0/131977528 learned_addr learned my addr 172.21.15.70:0/131977528 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:48.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d71925640 1 --2- 172.21.15.70:0/131977528 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d339820 0x55a82d341930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d72126640 1 --2- 172.21.15.70:0/131977528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d344a00 0x55a82d3423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d71124640 1 -- 172.21.15.70:0/131977528 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d339820 msgr2=0x55a82d341930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:48.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d71124640 1 --2- 172.21.15.70:0/131977528 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d339820 0x55a82d341930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d71124640 1 -- 172.21.15.70:0/131977528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d344a00 msgr2=0x55a82d3423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:48.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d71124640 1 --2- 172.21.15.70:0/131977528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d344a00 0x55a82d3423b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d71124640 1 -- 172.21.15.70:0/131977528 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a82d07c620 con 0x55a82d33a140 2024-06-04T05:06:48.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d71925640 1 --2- 172.21.15.70:0/131977528 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d339820 0x55a82d341930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:48.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d71124640 1 --2- 172.21.15.70:0/131977528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d33a140 0x55a82d341e70 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c005890 tx=0x7f7d5c019ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b9c4c32564653d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:48.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d557fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:48.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.881+0000 7f7d557fa640 1 -- 172.21.15.70:0/131977528 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c027020 con 0x55a82d33a140 2024-06-04T05:06:48.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.882+0000 7f7d7697b740 1 -- 172.21.15.70:0/131977528 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a82d2adb80 con 0x55a82d33a140 2024-06-04T05:06:48.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.882+0000 7f7d557fa640 1 -- 172.21.15.70:0/131977528 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f7d5c023610 con 0x55a82d33a140 2024-06-04T05:06:48.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.882+0000 7f7d557fa640 1 -- 172.21.15.70:0/131977528 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7d5c032050 con 0x55a82d33a140 2024-06-04T05:06:48.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.882+0000 7f7d7697b740 10 client.18033 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:48.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.882+0000 7f7d7697b740 1 -- 172.21.15.70:0/131977528 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a82d353320 con 0x55a82d33a140 2024-06-04T05:06:48.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.882+0000 7f7d557fa640 1 -- 172.21.15.70:0/131977528 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55a82d2adb80 con 0x55a82d33a140 2024-06-04T05:06:48.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.882+0000 7f7d557fa640 1 -- 172.21.15.70:0/131977528 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f7d5c018070 con 0x55a82d33a140 2024-06-04T05:06:48.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.882+0000 7f7d7697b740 10 client.18033 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:48.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.882+0000 7f7d7697b740 10 client.18033 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:06:48.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.882+0000 7f7d7697b740 20 client.18033 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:48.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.882+0000 7f7d7697b740 1 --2- 172.21.15.70:0/131977528 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a82d3546d0 0x55a82d356ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.882+0000 7f7d7697b740 4 client.18033 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:06:48.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.882+0000 7f7d7697b740 1 -- 172.21.15.70:0/131977528 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a82d2a7360 con 0x55a82d3546d0 2024-06-04T05:06:48.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.882+0000 7f7d71925640 1 --2- 172.21.15.70:0/131977528 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a82d3546d0 0x55a82d356ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.883+0000 7f7d71925640 1 --2- 172.21.15.70:0/131977528 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a82d3546d0 0x55a82d356ac0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e27955b90df42ceb server_cookie=6445cfd33588bb17 in_seq=0 out_seq=0 2024-06-04T05:06:48.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.883+0000 7f7d557fa640 10 client.18033 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:48.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.883+0000 7f7d557fa640 1 -- 172.21.15.70:0/131977528 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a82d2a7360 con 0x55a82d3546d0 2024-06-04T05:06:48.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.883+0000 7f7d557fa640 10 client.18033 handle_command_reply: tid=0 2024-06-04T05:06:48.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.883+0000 7f7d7697b740 1 client.18033 shutdown 2024-06-04T05:06:48.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.883+0000 7f7d7697b740 1 -- 172.21.15.70:0/131977528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d33a140 msgr2=0x55a82d341e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:48.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.883+0000 7f7d7697b740 1 --2- 172.21.15.70:0/131977528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d33a140 0x55a82d341e70 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c005890 tx=0x7f7d5c019ad0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.884+0000 7f7d7697b740 1 -- 172.21.15.70:0/131977528 shutdown_connections 2024-06-04T05:06:48.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.884+0000 7f7d7697b740 1 --2- 172.21.15.70:0/131977528 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a82d3546d0 0x55a82d356ac0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.884+0000 7f7d7697b740 1 --2- 172.21.15.70:0/131977528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d344a00 0x55a82d3423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.884+0000 7f7d7697b740 1 --2- 172.21.15.70:0/131977528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d33a140 0x55a82d341e70 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.884+0000 7f7d7697b740 1 --2- 172.21.15.70:0/131977528 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d339820 0x55a82d341930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.884+0000 7f7d557fa640 0 client.18033 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:48.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.884+0000 7f7d7697b740 1 -- 172.21.15.70:0/131977528 >> 172.21.15.70:0/131977528 conn(0x55a82d335d30 msgr2=0x55a82d33bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:48.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.884+0000 7f7d7697b740 1 -- 172.21.15.70:0/131977528 shutdown_connections 2024-06-04T05:06:48.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.884+0000 7f7d7697b740 1 -- 172.21.15.70:0/131977528 wait complete. 2024-06-04T05:06:48.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.884+0000 7f7d7697b740 20 client.18033 trim_cache size 0 max 16384 2024-06-04T05:06:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.899+0000 7f7d7697b740 1 Processor -- start 2024-06-04T05:06:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.899+0000 7f7d7697b740 1 -- start start 2024-06-04T05:06:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d7697b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d339820 0x55a82d35c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d7697b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d33a140 0x55a82d35c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d7697b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d344a00 0x55a82d33b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d7697b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c1ac4f0 con 0x55a82d344a00 2024-06-04T05:06:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d7697b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c1ae030 con 0x55a82d339820 2024-06-04T05:06:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d7697b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c1acbc0 con 0x55a82d33a140 2024-06-04T05:06:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d72126640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d344a00 0x55a82d33b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d71124640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d33a140 0x55a82d35c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d72126640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d344a00 0x55a82d33b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40742/0 (socket says 172.21.15.70:40742) 2024-06-04T05:06:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d72126640 1 -- 172.21.15.70:0/4139302095 learned_addr learned my addr 172.21.15.70:0/4139302095 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d71124640 1 --2- 172.21.15.70:0/4139302095 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d33a140 0x55a82d35c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:58914/0 (socket says 172.21.15.70:58914) 2024-06-04T05:06:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d71925640 1 --2- 172.21.15.70:0/4139302095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d339820 0x55a82d35c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d71124640 1 -- 172.21.15.70:0/4139302095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d339820 msgr2=0x55a82d35c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d71124640 1 --2- 172.21.15.70:0/4139302095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d339820 0x55a82d35c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d71124640 1 -- 172.21.15.70:0/4139302095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d344a00 msgr2=0x55a82d33b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d71124640 1 --2- 172.21.15.70:0/4139302095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d344a00 0x55a82d33b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d71124640 1 -- 172.21.15.70:0/4139302095 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a82d357080 con 0x55a82d33a140 2024-06-04T05:06:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d72126640 1 --2- 172.21.15.70:0/4139302095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d344a00 0x55a82d33b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d71925640 1 --2- 172.21.15.70:0/4139302095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d339820 0x55a82d35c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d71124640 1 --2- 172.21.15.70:0/4139302095 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d33a140 0x55a82d35c840 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0187c0 tx=0x7f7d5c011db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=920195b4841304fc server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.900+0000 7f7d577fe640 1 -- 172.21.15.70:0/4139302095 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c027040 con 0x55a82d33a140 2024-06-04T05:06:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.901+0000 7f7d577fe640 1 -- 172.21.15.70:0/4139302095 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d5c019b40 con 0x55a82d33a140 2024-06-04T05:06:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.901+0000 7f7d577fe640 1 -- 172.21.15.70:0/4139302095 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c01e040 con 0x55a82d33a140 2024-06-04T05:06:48.907 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.901+0000 7f7d7697b740 1 -- 172.21.15.70:0/4139302095 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d33a140 msgr2=0x55a82d35c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:48.907 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.901+0000 7f7d7697b740 1 --2- 172.21.15.70:0/4139302095 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d33a140 0x55a82d35c840 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0187c0 tx=0x7f7d5c011db0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.907 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.901+0000 7f7d7697b740 1 -- 172.21.15.70:0/4139302095 shutdown_connections 2024-06-04T05:06:48.907 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.901+0000 7f7d7697b740 1 --2- 172.21.15.70:0/4139302095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d344a00 0x55a82d33b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.901+0000 7f7d7697b740 1 --2- 172.21.15.70:0/4139302095 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d33a140 0x55a82d35c840 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.901+0000 7f7d7697b740 1 --2- 172.21.15.70:0/4139302095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d339820 0x55a82d35c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.901+0000 7f7d7697b740 1 -- 172.21.15.70:0/4139302095 >> 172.21.15.70:0/4139302095 conn(0x55a82d335d30 msgr2=0x55a82d330e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.901+0000 7f7d7697b740 1 -- 172.21.15.70:0/4139302095 shutdown_connections 2024-06-04T05:06:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.901+0000 7f7d7697b740 1 -- 172.21.15.70:0/4139302095 wait complete. 2024-06-04T05:06:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.902+0000 7f7d7697b740 1 -- start start 2024-06-04T05:06:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.902+0000 7f7d7697b740 1 Processor -- start 2024-06-04T05:06:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.902+0000 7f7d7697b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d339820 0x55a82d347980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.902+0000 7f7d7697b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d33a140 0x55a82d347ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.902+0000 7f7d7697b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d344a00 0x55a82d3bd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.902+0000 7f7d7697b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c1ad960 con 0x55a82d33a140 2024-06-04T05:06:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.902+0000 7f7d7697b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c1ad290 con 0x55a82d339820 2024-06-04T05:06:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.902+0000 7f7d7697b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c1aa4f0 con 0x55a82d344a00 2024-06-04T05:06:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.902+0000 7f7d7697b740 10 client.0 _ll_register_callbacks cb 0x7f7d5c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.902+0000 7f7d71124640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d33a140 0x55a82d347ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.902+0000 7f7d71124640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d33a140 0x55a82d347ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40748/0 (socket says 172.21.15.70:40748) 2024-06-04T05:06:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.902+0000 7f7d72126640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d344a00 0x55a82d3bd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.902+0000 7f7d71124640 1 -- 172.21.15.70:0/2096042560 learned_addr learned my addr 172.21.15.70:0/2096042560 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.902+0000 7f7d71925640 1 --2- 172.21.15.70:0/2096042560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d339820 0x55a82d347980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d71124640 1 -- 172.21.15.70:0/2096042560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d344a00 msgr2=0x55a82d3bd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d71124640 1 --2- 172.21.15.70:0/2096042560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d344a00 0x55a82d3bd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d71124640 1 -- 172.21.15.70:0/2096042560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d339820 msgr2=0x55a82d347980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d71124640 1 --2- 172.21.15.70:0/2096042560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d339820 0x55a82d347980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d71124640 1 -- 172.21.15.70:0/2096042560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a82d3bd700 con 0x55a82d33a140 2024-06-04T05:06:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d72126640 1 --2- 172.21.15.70:0/2096042560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d344a00 0x55a82d3bd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d71124640 1 --2- 172.21.15.70:0/2096042560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d33a140 0x55a82d347ec0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c018930 tx=0x7f7d5c019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad03d21cf7a2520b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d557fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d557fa640 1 -- 172.21.15.70:0/2096042560 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c027070 con 0x55a82d33a140 2024-06-04T05:06:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d7697b740 1 -- 172.21.15.70:0/2096042560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d5c019b40 con 0x55a82d33a140 2024-06-04T05:06:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d557fa640 1 -- 172.21.15.70:0/2096042560 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f7d5c005240 con 0x55a82d33a140 2024-06-04T05:06:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d557fa640 1 -- 172.21.15.70:0/2096042560 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7d5c046050 con 0x55a82d33a140 2024-06-04T05:06:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d7697b740 10 client.18039 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d7697b740 1 -- 172.21.15.70:0/2096042560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a82d3be910 con 0x55a82d33a140 2024-06-04T05:06:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d557fa640 1 -- 172.21.15.70:0/2096042560 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d5c019b40 con 0x55a82d33a140 2024-06-04T05:06:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d557fa640 1 -- 172.21.15.70:0/2096042560 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f7d5c018050 con 0x55a82d33a140 2024-06-04T05:06:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d7697b740 10 client.18039 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d7697b740 10 client.18039 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:06:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.903+0000 7f7d7697b740 20 client.18039 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.904+0000 7f7d7697b740 1 --2- 172.21.15.70:0/2096042560 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a82d3bef80 0x55a82d3c13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.904+0000 7f7d7697b740 4 client.18039 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:06:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.904+0000 7f7d7697b740 1 -- 172.21.15.70:0/2096042560 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a82d2215f0 con 0x55a82d3bef80 2024-06-04T05:06:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.904+0000 7f7d71925640 1 --2- 172.21.15.70:0/2096042560 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a82d3bef80 0x55a82d3c13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.904+0000 7f7d71925640 1 --2- 172.21.15.70:0/2096042560 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a82d3bef80 0x55a82d3c13a0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ead72c43fdf326d server_cookie=54335d0e9f78aea8 in_seq=0 out_seq=0 2024-06-04T05:06:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.904+0000 7f7d557fa640 10 client.18039 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:48.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.927+0000 7f7d557fa640 1 -- 172.21.15.70:0/2096042560 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55a82d2215f0 con 0x55a82d3bef80 2024-06-04T05:06:48.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.927+0000 7f7d557fa640 10 client.18039 handle_command_reply: tid=0 2024-06-04T05:06:48.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.927+0000 7f7d7697b740 1 client.18039 shutdown 2024-06-04T05:06:48.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.927+0000 7f7d7697b740 1 -- 172.21.15.70:0/2096042560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d33a140 msgr2=0x55a82d347ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:48.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.927+0000 7f7d7697b740 1 --2- 172.21.15.70:0/2096042560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d33a140 0x55a82d347ec0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c018930 tx=0x7f7d5c019700 comp rx=0 tx=0).stop 2024-06-04T05:06:48.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.927+0000 7f7d7697b740 1 -- 172.21.15.70:0/2096042560 shutdown_connections 2024-06-04T05:06:48.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.927+0000 7f7d7697b740 1 --2- 172.21.15.70:0/2096042560 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a82d3bef80 0x55a82d3c13a0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.927+0000 7f7d557fa640 0 client.18039 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:48.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.927+0000 7f7d7697b740 1 --2- 172.21.15.70:0/2096042560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a82d344a00 0x55a82d3bd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.927+0000 7f7d7697b740 1 --2- 172.21.15.70:0/2096042560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a82d33a140 0x55a82d347ec0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.927+0000 7f7d7697b740 1 --2- 172.21.15.70:0/2096042560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a82d339820 0x55a82d347980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.927+0000 7f7d7697b740 1 -- 172.21.15.70:0/2096042560 >> 172.21.15.70:0/2096042560 conn(0x55a82d335d30 msgr2=0x55a82d3418f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:48.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.927+0000 7f7d7697b740 1 -- 172.21.15.70:0/2096042560 shutdown_connections 2024-06-04T05:06:48.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.927+0000 7f7d7697b740 1 -- 172.21.15.70:0/2096042560 wait complete. 2024-06-04T05:06:48.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.927+0000 7f7d7697b740 20 client.18039 trim_cache size 0 max 16384 2024-06-04T05:06:48.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:06:48.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:06:48.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:06:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.928+0000 7f7d73b89640 1 -- 172.21.15.70:0/668963161 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d580b2a10 msgr2=0x7f7d580b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.928+0000 7f7d73b89640 1 --2- 172.21.15.70:0/668963161 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d580b2a10 0x7f7d580b4ed0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7d64031730 tx=0x7f7d64005f10 comp rx=0 tx=0).stop 2024-06-04T05:06:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.928+0000 7f7d73b89640 1 -- 172.21.15.70:0/668963161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d6c101980 msgr2=0x7f7d6c199090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.928+0000 7f7d73b89640 1 --2- 172.21.15.70:0/668963161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d6c101980 0x7f7d6c199090 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800e9f0 tx=0x7f7d6800eeb0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.928+0000 7f7d73b89640 1 -- 172.21.15.70:0/668963161 shutdown_connections 2024-06-04T05:06:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.928+0000 7f7d73b89640 1 --2- 172.21.15.70:0/668963161 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d580b2a10 0x7f7d580b4ed0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.928+0000 7f7d73b89640 1 --2- 172.21.15.70:0/668963161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d6c107940 0x7f7d6c19d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.928+0000 7f7d73b89640 1 --2- 172.21.15.70:0/668963161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d6c102340 0x7f7d6c1995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.928+0000 7f7d73b89640 1 --2- 172.21.15.70:0/668963161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d6c101980 0x7f7d6c199090 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:48.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.928+0000 7f7d73b89640 1 -- 172.21.15.70:0/668963161 >> 172.21.15.70:0/668963161 conn(0x7f7d6c078ad0 msgr2=0x7f7d6c100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:48.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.928+0000 7f7d73b89640 1 -- 172.21.15.70:0/668963161 shutdown_connections 2024-06-04T05:06:48.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:06:48.928+0000 7f7d73b89640 1 -- 172.21.15.70:0/668963161 wait complete. 2024-06-04T05:06:49.518 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:49 smithi070 ceph-mon[32631]: pgmap v296: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.5 MiB/s wr, 123 op/s 2024-06-04T05:06:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:49 smithi112 ceph-mon[36453]: pgmap v296: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.5 MiB/s wr, 123 op/s 2024-06-04T05:06:49.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 165055 1.88 MB/sec execute 342 sec latency 39.277 ms 2024-06-04T05:06:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:49 smithi098 ceph-mon[35569]: pgmap v296: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.5 MiB/s wr, 123 op/s 2024-06-04T05:06:49.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.761+0000 7f091d35b700 1 -- 172.21.15.70:0/422913832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0918114140 msgr2=0x7f09181165e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:49.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.761+0000 7f091d35b700 1 --2- 172.21.15.70:0/422913832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0918114140 0x7f09181165e0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f091000b0a0 tx=0x7f091000b3b0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.761+0000 7f091d35b700 1 -- 172.21.15.70:0/422913832 shutdown_connections 2024-06-04T05:06:49.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.761+0000 7f091d35b700 1 --2- 172.21.15.70:0/422913832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0918114140 0x7f09181165e0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.761+0000 7f091d35b700 1 --2- 172.21.15.70:0/422913832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0918101d90 0x7f0918102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.761+0000 7f091d35b700 1 --2- 172.21.15.70:0/422913832 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0918102740 0x7f0918113a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.761+0000 7f091d35b700 1 -- 172.21.15.70:0/422913832 >> 172.21.15.70:0/422913832 conn(0x7f0918078ab0 msgr2=0x7f09180ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:49.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.762+0000 7f091d35b700 1 -- 172.21.15.70:0/422913832 shutdown_connections 2024-06-04T05:06:49.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.762+0000 7f091d35b700 1 -- 172.21.15.70:0/422913832 wait complete. 2024-06-04T05:06:49.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.762+0000 7f091d35b700 1 Processor -- start 2024-06-04T05:06:49.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.762+0000 7f091d35b700 1 -- start start 2024-06-04T05:06:49.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f091d35b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0918101d90 0x7f09181a00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f091d35b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0918102740 0x7f09181a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f091d35b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0918114140 0x7f09181a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f091d35b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0918119bb0 con 0x7f0918101d90 2024-06-04T05:06:49.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f091d35b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0918119a30 con 0x7f0918114140 2024-06-04T05:06:49.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f091d35b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0918119d30 con 0x7f0918102740 2024-06-04T05:06:49.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f0916ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0918101d90 0x7f09181a00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f0916ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0918101d90 0x7f09181a00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40764/0 (socket says 172.21.15.70:40764) 2024-06-04T05:06:49.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f0916ffd700 1 -- 172.21.15.70:0/3673532012 learned_addr learned my addr 172.21.15.70:0/3673532012 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:49.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f09167fc700 1 --2- 172.21.15.70:0/3673532012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0918102740 0x7f09181a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f09177fe700 1 --2- 172.21.15.70:0/3673532012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0918114140 0x7f09181a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f0916ffd700 1 -- 172.21.15.70:0/3673532012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0918102740 msgr2=0x7f09181a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:49.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f0916ffd700 1 --2- 172.21.15.70:0/3673532012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0918102740 0x7f09181a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f0916ffd700 1 -- 172.21.15.70:0/3673532012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0918114140 msgr2=0x7f09181a4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:49.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f0916ffd700 1 --2- 172.21.15.70:0/3673532012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0918114140 0x7f09181a4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f0916ffd700 1 -- 172.21.15.70:0/3673532012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09181a50a0 con 0x7f0918101d90 2024-06-04T05:06:49.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f09167fc700 1 --2- 172.21.15.70:0/3673532012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0918102740 0x7f09181a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:49.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.763+0000 7f0916ffd700 1 --2- 172.21.15.70:0/3673532012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0918101d90 0x7f09181a00e0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f090800d980 tx=0x7f090800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:49.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.764+0000 7f08fffff700 1 -- 172.21.15.70:0/3673532012 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0908016070 con 0x7f0918101d90 2024-06-04T05:06:49.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.764+0000 7f091d35b700 1 -- 172.21.15.70:0/3673532012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09181a5330 con 0x7f0918101d90 2024-06-04T05:06:49.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.764+0000 7f08fffff700 1 -- 172.21.15.70:0/3673532012 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0908004830 con 0x7f0918101d90 2024-06-04T05:06:49.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.764+0000 7f08fffff700 1 -- 172.21.15.70:0/3673532012 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09080054c0 con 0x7f0918101d90 2024-06-04T05:06:49.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.764+0000 7f091d35b700 1 -- 172.21.15.70:0/3673532012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09181acc10 con 0x7f0918101d90 2024-06-04T05:06:49.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.766+0000 7f08fffff700 1 -- 172.21.15.70:0/3673532012 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f0908004220 con 0x7f0918101d90 2024-06-04T05:06:49.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.767+0000 7f08fffff700 1 --2- 172.21.15.70:0/3673532012 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f09000b29f0 0x7f09000b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.767+0000 7f09167fc700 1 --2- 172.21.15.70:0/3673532012 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f09000b29f0 0x7f09000b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.767+0000 7f08fffff700 1 -- 172.21.15.70:0/3673532012 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f09080e2390 con 0x7f0918101d90 2024-06-04T05:06:49.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.768+0000 7f09167fc700 1 --2- 172.21.15.70:0/3673532012 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f09000b29f0 0x7f09000b4eb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0918111bb0 tx=0x7f090c009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:49.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.768+0000 7f0927bf0b80 1 Processor -- start 2024-06-04T05:06:49.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.768+0000 7f0927bf0b80 1 -- start start 2024-06-04T05:06:49.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.768+0000 7f0927bf0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c98d320 0x55ac6c98d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.768+0000 7f0927bf0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c9878a0 0x55ac6c987d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.768+0000 7f0927bf0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c988260 0x55ac6c994080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.768+0000 7f0927bf0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ac6c8e5d70 con 0x55ac6c988260 2024-06-04T05:06:49.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.768+0000 7f0927bf0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ac6c8d5910 con 0x55ac6c98d320 2024-06-04T05:06:49.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.768+0000 7f0927bf0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ac6c8d9950 con 0x55ac6c9878a0 2024-06-04T05:06:49.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.768+0000 7f09167fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c9878a0 0x55ac6c987d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.768+0000 7f0916ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c98d320 0x55ac6c98d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.768+0000 7f09177fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c988260 0x55ac6c994080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.768+0000 7f09167fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c9878a0 0x55ac6c987d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:58948/0 (socket says 172.21.15.70:58948) 2024-06-04T05:06:49.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.768+0000 7f09167fc700 1 -- 172.21.15.70:0/3336513765 learned_addr learned my addr 172.21.15.70:0/3336513765 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:49.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.769+0000 7f09167fc700 1 -- 172.21.15.70:0/3336513765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c98d320 msgr2=0x55ac6c98d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:49.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.769+0000 7f09167fc700 1 --2- 172.21.15.70:0/3336513765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c98d320 0x55ac6c98d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.769+0000 7f09167fc700 1 -- 172.21.15.70:0/3336513765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c988260 msgr2=0x55ac6c994080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:49.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.769+0000 7f09167fc700 1 --2- 172.21.15.70:0/3336513765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c988260 0x55ac6c994080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.769+0000 7f09167fc700 1 -- 172.21.15.70:0/3336513765 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac6c87efb0 con 0x55ac6c9878a0 2024-06-04T05:06:49.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.769+0000 7f09167fc700 1 --2- 172.21.15.70:0/3336513765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c9878a0 0x55ac6c987d20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f090c00e700 tx=0x7f090c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ca3e3b584e80ec5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:49.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.769+0000 7f08fd7fa700 1 -- 172.21.15.70:0/3336513765 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c026600 con 0x55ac6c9878a0 2024-06-04T05:06:49.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.769+0000 7f08fd7fa700 1 -- 172.21.15.70:0/3336513765 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f090c026c30 con 0x55ac6c9878a0 2024-06-04T05:06:49.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.769+0000 7f08fd7fa700 1 -- 172.21.15.70:0/3336513765 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c01eb40 con 0x55ac6c9878a0 2024-06-04T05:06:49.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.770+0000 7f0927bf0b80 1 -- 172.21.15.70:0/3336513765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c9878a0 msgr2=0x55ac6c987d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:49.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.770+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/3336513765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c9878a0 0x55ac6c987d20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f090c00e700 tx=0x7f090c014000 comp rx=0 tx=0).stop 2024-06-04T05:06:49.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.770+0000 7f0927bf0b80 1 -- 172.21.15.70:0/3336513765 shutdown_connections 2024-06-04T05:06:49.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.770+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/3336513765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c988260 0x55ac6c994080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.770+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/3336513765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c98d320 0x55ac6c98d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.770+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/3336513765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c9878a0 0x55ac6c987d20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.770+0000 7f0927bf0b80 1 -- 172.21.15.70:0/3336513765 >> 172.21.15.70:0/3336513765 conn(0x55ac6c8b3240 msgr2=0x55ac6c9823e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:49.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.770+0000 7f0927bf0b80 1 -- 172.21.15.70:0/3336513765 shutdown_connections 2024-06-04T05:06:49.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.770+0000 7f0927bf0b80 1 -- 172.21.15.70:0/3336513765 wait complete. 2024-06-04T05:06:49.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.771+0000 7f0927bf0b80 1 -- start start 2024-06-04T05:06:49.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.771+0000 7f0927bf0b80 1 Processor -- start 2024-06-04T05:06:49.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.771+0000 7f0927bf0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c9878a0 0x55ac6c993d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.771+0000 7f0927bf0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c988260 0x55ac6c991000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.771+0000 7f0927bf0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c98d320 0x55ac6c991540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.771+0000 7f0927bf0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ac6c7dbfa0 con 0x55ac6c9878a0 2024-06-04T05:06:49.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.771+0000 7f0927bf0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ac6c83dda0 con 0x55ac6c98d320 2024-06-04T05:06:49.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.771+0000 7f0927bf0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ac6c9946e0 con 0x55ac6c988260 2024-06-04T05:06:49.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.771+0000 7f0927bf0b80 10 client.0 _ll_register_callbacks cb 0x55ac6c870950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:49.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.771+0000 7f0916ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c9878a0 0x55ac6c993d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.771+0000 7f09167fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c988260 0x55ac6c991000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.771+0000 7f0916ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c9878a0 0x55ac6c993d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40790/0 (socket says 172.21.15.70:40790) 2024-06-04T05:06:49.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.771+0000 7f0916ffd700 1 -- 172.21.15.70:0/2885626428 learned_addr learned my addr 172.21.15.70:0/2885626428 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:49.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.771+0000 7f09177fe700 1 --2- 172.21.15.70:0/2885626428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c98d320 0x55ac6c991540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.772+0000 7f09167fc700 1 -- 172.21.15.70:0/2885626428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c98d320 msgr2=0x55ac6c991540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:49.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.772+0000 7f09167fc700 1 --2- 172.21.15.70:0/2885626428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c98d320 0x55ac6c991540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.772+0000 7f09167fc700 1 -- 172.21.15.70:0/2885626428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c9878a0 msgr2=0x55ac6c993d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:49.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.772+0000 7f09167fc700 1 --2- 172.21.15.70:0/2885626428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c9878a0 0x55ac6c993d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.772+0000 7f09167fc700 1 -- 172.21.15.70:0/2885626428 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac6c8258d0 con 0x55ac6c988260 2024-06-04T05:06:49.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.772+0000 7f09167fc700 1 --2- 172.21.15.70:0/2885626428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c988260 0x55ac6c991000 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f090c00e6d0 tx=0x7f090c0268a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6116d4673f4bc69b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:49.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.772+0000 7f08faff5700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:06:49.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.772+0000 7f08faff5700 1 -- 172.21.15.70:0/2885626428 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c01e3f0 con 0x55ac6c988260 2024-06-04T05:06:49.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.772+0000 7f0927bf0b80 1 -- 172.21.15.70:0/2885626428 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f090c026c30 con 0x55ac6c988260 2024-06-04T05:06:49.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.772+0000 7f08faff5700 1 -- 172.21.15.70:0/2885626428 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f090c005050 con 0x55ac6c988260 2024-06-04T05:06:49.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.772+0000 7f08faff5700 1 -- 172.21.15.70:0/2885626428 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f090c005250 con 0x55ac6c988260 2024-06-04T05:06:49.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.773+0000 7f0927bf0b80 10 client.27271 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:49.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.773+0000 7f0927bf0b80 1 -- 172.21.15.70:0/2885626428 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ac6c88f760 con 0x55ac6c988260 2024-06-04T05:06:49.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.773+0000 7f08faff5700 1 -- 172.21.15.70:0/2885626428 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f090c026c30 con 0x55ac6c988260 2024-06-04T05:06:49.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.773+0000 7f08faff5700 1 -- 172.21.15.70:0/2885626428 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f090c004760 con 0x55ac6c988260 2024-06-04T05:06:49.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.773+0000 7f0927bf0b80 10 client.27271 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:49.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.773+0000 7f0927bf0b80 10 client.27271 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:06:49.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.773+0000 7f0927bf0b80 20 client.27271 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:49.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.773+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/2885626428 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ac6c9a10f0 0x55ac6c9a34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.773+0000 7f0927bf0b80 4 client.27271 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:06:49.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.773+0000 7f0927bf0b80 1 -- 172.21.15.70:0/2885626428 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac6c8e7cc0 con 0x55ac6c9a10f0 2024-06-04T05:06:49.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.773+0000 7f0916ffd700 1 --2- 172.21.15.70:0/2885626428 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ac6c9a10f0 0x55ac6c9a34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.774+0000 7f08faff5700 10 client.27271 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:06:49.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.774+0000 7f0916ffd700 1 --2- 172.21.15.70:0/2885626428 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ac6c9a10f0 0x55ac6c9a34e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f7b4f4b3338a669c server_cookie=6c1908348e70f3cd in_seq=0 out_seq=0 2024-06-04T05:06:49.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.774+0000 7f08faff5700 1 -- 172.21.15.70:0/2885626428 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ac6c8e7cc0 con 0x55ac6c9a10f0 2024-06-04T05:06:49.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.774+0000 7f08faff5700 10 client.27271 handle_command_reply: tid=0 2024-06-04T05:06:49.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.774+0000 7f0927bf0b80 1 client.27271 shutdown 2024-06-04T05:06:49.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.775+0000 7f0927bf0b80 1 -- 172.21.15.70:0/2885626428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c988260 msgr2=0x55ac6c991000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:49.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.775+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/2885626428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c988260 0x55ac6c991000 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f090c00e6d0 tx=0x7f090c0268a0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.775+0000 7f0927bf0b80 1 -- 172.21.15.70:0/2885626428 shutdown_connections 2024-06-04T05:06:49.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.775+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/2885626428 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ac6c9a10f0 0x55ac6c9a34e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.775+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/2885626428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c98d320 0x55ac6c991540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.775+0000 7f08faff5700 0 client.27271 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:06:49.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.775+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/2885626428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c9878a0 0x55ac6c993d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.775+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/2885626428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c988260 0x55ac6c991000 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.775+0000 7f0927bf0b80 1 -- 172.21.15.70:0/2885626428 >> 172.21.15.70:0/2885626428 conn(0x55ac6c8b3240 msgr2=0x55ac6c981af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:49.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.775+0000 7f0927bf0b80 1 -- 172.21.15.70:0/2885626428 shutdown_connections 2024-06-04T05:06:49.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.775+0000 7f0927bf0b80 1 -- 172.21.15.70:0/2885626428 wait complete. 2024-06-04T05:06:49.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.775+0000 7f0927bf0b80 20 client.27271 trim_cache size 0 max 16384 2024-06-04T05:06:49.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.795+0000 7f0927bf0b80 1 Processor -- start 2024-06-04T05:06:49.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.795+0000 7f0927bf0b80 1 -- start start 2024-06-04T05:06:49.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.795+0000 7f0927bf0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c9878a0 0x55ac6c98c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.795+0000 7f0927bf0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c988260 0x55ac6c982270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.795+0000 7f0927bf0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c98d320 0x55ac6c992830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.795+0000 7f0927bf0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f09181b4440 con 0x55ac6c9878a0 2024-06-04T05:06:49.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.795+0000 7f0927bf0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09181b5f80 con 0x55ac6c98d320 2024-06-04T05:06:49.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.795+0000 7f0927bf0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f09181b4b10 con 0x55ac6c988260 2024-06-04T05:06:49.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.795+0000 7f0916ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c9878a0 0x55ac6c98c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.795+0000 7f0916ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c9878a0 0x55ac6c98c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40806/0 (socket says 172.21.15.70:40806) 2024-06-04T05:06:49.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.795+0000 7f0916ffd700 1 -- 172.21.15.70:0/1136798435 learned_addr learned my addr 172.21.15.70:0/1136798435 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:49.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.795+0000 7f09177fe700 1 --2- 172.21.15.70:0/1136798435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c98d320 0x55ac6c992830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.795+0000 7f09167fc700 1 --2- 172.21.15.70:0/1136798435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c988260 0x55ac6c982270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.796+0000 7f0916ffd700 1 -- 172.21.15.70:0/1136798435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c988260 msgr2=0x55ac6c982270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:49.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.796+0000 7f0916ffd700 1 --2- 172.21.15.70:0/1136798435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c988260 0x55ac6c982270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.796+0000 7f0916ffd700 1 -- 172.21.15.70:0/1136798435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c98d320 msgr2=0x55ac6c992830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:49.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.796+0000 7f0916ffd700 1 --2- 172.21.15.70:0/1136798435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c98d320 0x55ac6c992830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.796+0000 7f0916ffd700 1 -- 172.21.15.70:0/1136798435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac6c984380 con 0x55ac6c9878a0 2024-06-04T05:06:49.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.796+0000 7f09177fe700 1 --2- 172.21.15.70:0/1136798435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c98d320 0x55ac6c992830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:49.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.796+0000 7f0916ffd700 1 --2- 172.21.15.70:0/1136798435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c9878a0 0x55ac6c98c840 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f09080a5670 tx=0x7f0908007640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19cb220ee725e172 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:49.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.796+0000 7f08fcff9700 1 -- 172.21.15.70:0/1136798435 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09080b24a0 con 0x55ac6c9878a0 2024-06-04T05:06:49.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.796+0000 7f08fcff9700 1 -- 172.21.15.70:0/1136798435 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0908004cc0 con 0x55ac6c9878a0 2024-06-04T05:06:49.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.796+0000 7f08fcff9700 1 -- 172.21.15.70:0/1136798435 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09080a98a0 con 0x55ac6c9878a0 2024-06-04T05:06:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.797+0000 7f0927bf0b80 1 -- 172.21.15.70:0/1136798435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c9878a0 msgr2=0x55ac6c98c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.797+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/1136798435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c9878a0 0x55ac6c98c840 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f09080a5670 tx=0x7f0908007640 comp rx=0 tx=0).stop 2024-06-04T05:06:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.797+0000 7f0927bf0b80 1 -- 172.21.15.70:0/1136798435 shutdown_connections 2024-06-04T05:06:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.797+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/1136798435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c98d320 0x55ac6c992830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.797+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/1136798435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c9878a0 0x55ac6c98c840 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.797+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/1136798435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c988260 0x55ac6c982270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.797+0000 7f0927bf0b80 1 -- 172.21.15.70:0/1136798435 >> 172.21.15.70:0/1136798435 conn(0x55ac6c8b3240 msgr2=0x55ac6c98c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.797+0000 7f0927bf0b80 1 -- 172.21.15.70:0/1136798435 shutdown_connections 2024-06-04T05:06:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.797+0000 7f0927bf0b80 1 -- 172.21.15.70:0/1136798435 wait complete. 2024-06-04T05:06:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.797+0000 7f0927bf0b80 1 -- start start 2024-06-04T05:06:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.797+0000 7f0927bf0b80 1 Processor -- start 2024-06-04T05:06:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f0927bf0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c9878a0 0x55ac6c9a2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f0927bf0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c988260 0x55ac6c9a2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f0927bf0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c98d320 0x55ac6c9a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f0927bf0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f09181b58b0 con 0x55ac6c98d320 2024-06-04T05:06:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f0927bf0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09181b51e0 con 0x55ac6c9878a0 2024-06-04T05:06:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f0927bf0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f09181b2400 con 0x55ac6c988260 2024-06-04T05:06:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f09177fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c98d320 0x55ac6c9a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f0916ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c9878a0 0x55ac6c9a2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f09177fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c98d320 0x55ac6c9a2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40814/0 (socket says 172.21.15.70:40814) 2024-06-04T05:06:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f0916ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c9878a0 0x55ac6c9a2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54794/0 (socket says 172.21.15.70:54794) 2024-06-04T05:06:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f09177fe700 1 -- 172.21.15.70:0/939150421 learned_addr learned my addr 172.21.15.70:0/939150421 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f0927bf0b80 10 client.0 _ll_register_callbacks cb 0x55ac6c870950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f09167fc700 1 --2- 172.21.15.70:0/939150421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c988260 0x55ac6c9a2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f09177fe700 1 -- 172.21.15.70:0/939150421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c988260 msgr2=0x55ac6c9a2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f09177fe700 1 --2- 172.21.15.70:0/939150421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c988260 0x55ac6c9a2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f09177fe700 1 -- 172.21.15.70:0/939150421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c9878a0 msgr2=0x55ac6c9a2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f09177fe700 1 --2- 172.21.15.70:0/939150421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c9878a0 0x55ac6c9a2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f09177fe700 1 -- 172.21.15.70:0/939150421 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac6c98f020 con 0x55ac6c98d320 2024-06-04T05:06:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f0916ffd700 1 --2- 172.21.15.70:0/939150421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c9878a0 0x55ac6c9a2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.798+0000 7f09167fc700 1 --2- 172.21.15.70:0/939150421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c988260 0x55ac6c9a2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.799+0000 7f09177fe700 1 --2- 172.21.15.70:0/939150421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c98d320 0x55ac6c9a2dc0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f091000bfd0 tx=0x7f0910003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79c4680634032a87 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.799+0000 7f08faff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.799+0000 7f08faff5700 1 -- 172.21.15.70:0/939150421 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091002c060 con 0x55ac6c98d320 2024-06-04T05:06:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.799+0000 7f08faff5700 1 -- 172.21.15.70:0/939150421 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f09100074e0 con 0x55ac6c98d320 2024-06-04T05:06:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.799+0000 7f0927bf0b80 1 -- 172.21.15.70:0/939150421 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0908004cc0 con 0x55ac6c98d320 2024-06-04T05:06:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.799+0000 7f08faff5700 1 -- 172.21.15.70:0/939150421 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f09100076e0 con 0x55ac6c98d320 2024-06-04T05:06:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.799+0000 7f0927bf0b80 10 client.18057 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.799+0000 7f0927bf0b80 1 -- 172.21.15.70:0/939150421 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ac6ca0bb30 con 0x55ac6c98d320 2024-06-04T05:06:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.799+0000 7f08faff5700 1 -- 172.21.15.70:0/939150421 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0908004cc0 con 0x55ac6c98d320 2024-06-04T05:06:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.799+0000 7f08faff5700 1 -- 172.21.15.70:0/939150421 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f0910011070 con 0x55ac6c98d320 2024-06-04T05:06:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.800+0000 7f0927bf0b80 10 client.18057 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.800+0000 7f0927bf0b80 10 client.18057 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:06:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.800+0000 7f0927bf0b80 20 client.18057 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.800+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/939150421 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ac6ca0c1e0 0x55ac6ca0e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.800+0000 7f0927bf0b80 4 client.18057 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:06:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.800+0000 7f0927bf0b80 1 -- 172.21.15.70:0/939150421 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ac6c8e14a0 con 0x55ac6ca0c1e0 2024-06-04T05:06:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.800+0000 7f0916ffd700 1 --2- 172.21.15.70:0/939150421 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ac6ca0c1e0 0x55ac6ca0e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.800+0000 7f08faff5700 10 client.18057 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:06:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.800+0000 7f0916ffd700 1 --2- 172.21.15.70:0/939150421 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ac6ca0c1e0 0x55ac6ca0e600 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cd2b636f026fee07 server_cookie=3bbce8b2efb1aa9 in_seq=0 out_seq=0 2024-06-04T05:06:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.801+0000 7f08faff5700 1 -- 172.21.15.70:0/939150421 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ac6c8e14a0 con 0x55ac6ca0c1e0 2024-06-04T05:06:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.801+0000 7f08faff5700 10 client.18057 handle_command_reply: tid=0 2024-06-04T05:06:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.801+0000 7f0927bf0b80 1 client.18057 shutdown 2024-06-04T05:06:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.801+0000 7f0927bf0b80 1 -- 172.21.15.70:0/939150421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c98d320 msgr2=0x55ac6c9a2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.801+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/939150421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c98d320 0x55ac6c9a2dc0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f091000bfd0 tx=0x7f0910003c30 comp rx=0 tx=0).stop 2024-06-04T05:06:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.801+0000 7f0927bf0b80 1 -- 172.21.15.70:0/939150421 shutdown_connections 2024-06-04T05:06:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.801+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/939150421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac6c98d320 0x55ac6c9a2dc0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.801+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/939150421 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ac6ca0c1e0 0x55ac6ca0e600 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.801+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/939150421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6c9878a0 0x55ac6c9a2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.801+0000 7f0927bf0b80 1 --2- 172.21.15.70:0/939150421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac6c988260 0x55ac6c9a2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.801+0000 7f08faff5700 0 client.18057 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:06:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.801+0000 7f0927bf0b80 1 -- 172.21.15.70:0/939150421 >> 172.21.15.70:0/939150421 conn(0x55ac6c8b3240 msgr2=0x55ac6c98ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.801+0000 7f0927bf0b80 1 -- 172.21.15.70:0/939150421 shutdown_connections 2024-06-04T05:06:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.801+0000 7f0927bf0b80 1 -- 172.21.15.70:0/939150421 wait complete. 2024-06-04T05:06:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.801+0000 7f0927bf0b80 20 client.18057 trim_cache size 0 max 16384 2024-06-04T05:06:49.809 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:06:49.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.802+0000 7f091d35b700 1 -- 172.21.15.70:0/3673532012 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f09000b29f0 msgr2=0x7f09000b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:49.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.802+0000 7f091d35b700 1 --2- 172.21.15.70:0/3673532012 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f09000b29f0 0x7f09000b4eb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0918111bb0 tx=0x7f090c009260 comp rx=0 tx=0).stop 2024-06-04T05:06:49.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.802+0000 7f091d35b700 1 -- 172.21.15.70:0/3673532012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0918101d90 msgr2=0x7f09181a00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:49.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.802+0000 7f091d35b700 1 --2- 172.21.15.70:0/3673532012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0918101d90 0x7f09181a00e0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f090800d980 tx=0x7f090800dc90 comp rx=0 tx=0).stop 2024-06-04T05:06:49.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.802+0000 7f091d35b700 1 -- 172.21.15.70:0/3673532012 shutdown_connections 2024-06-04T05:06:49.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.802+0000 7f091d35b700 1 --2- 172.21.15.70:0/3673532012 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f09000b29f0 0x7f09000b4eb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.802+0000 7f091d35b700 1 --2- 172.21.15.70:0/3673532012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0918114140 0x7f09181a4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.802+0000 7f091d35b700 1 --2- 172.21.15.70:0/3673532012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0918101d90 0x7f09181a00e0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.802+0000 7f091d35b700 1 --2- 172.21.15.70:0/3673532012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0918102740 0x7f09181a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:49.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.802+0000 7f091d35b700 1 -- 172.21.15.70:0/3673532012 >> 172.21.15.70:0/3673532012 conn(0x7f0918078ab0 msgr2=0x7f09180fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:49.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.802+0000 7f091d35b700 1 -- 172.21.15.70:0/3673532012 shutdown_connections 2024-06-04T05:06:49.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:49.802+0000 7f091d35b700 1 -- 172.21.15.70:0/3673532012 wait complete. 2024-06-04T05:06:50.454 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:06:50.715 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:06:50.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 165624 1.88 MB/sec execute 343 sec latency 19.076 ms 2024-06-04T05:06:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:51 smithi070 ceph-mon[32631]: pgmap v297: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.5 MiB/s wr, 123 op/s 2024-06-04T05:06:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:51 smithi112 ceph-mon[36453]: pgmap v297: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.5 MiB/s wr, 123 op/s 2024-06-04T05:06:51.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 166384 1.89 MB/sec execute 344 sec latency 16.066 ms 2024-06-04T05:06:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:51 smithi098 ceph-mon[35569]: pgmap v297: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.5 MiB/s wr, 123 op/s 2024-06-04T05:06:52.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.505+0000 7f5a0c041700 1 -- 172.21.15.70:0/855412810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a041041a0 msgr2=0x7f5a041081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:52.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.505+0000 7f5a0c041700 1 --2- 172.21.15.70:0/855412810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a041041a0 0x7f5a041081c0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f59f40099c0 tx=0x7f59f4009cd0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.505+0000 7f5a0c041700 1 -- 172.21.15.70:0/855412810 shutdown_connections 2024-06-04T05:06:52.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.505+0000 7f5a0c041700 1 --2- 172.21.15.70:0/855412810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a041088f0 0x7f5a04110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.505+0000 7f5a0c041700 1 --2- 172.21.15.70:0/855412810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a041037f0 0x7f5a04103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.505+0000 7f5a0c041700 1 --2- 172.21.15.70:0/855412810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a041041a0 0x7f5a041081c0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.505+0000 7f5a0c041700 1 -- 172.21.15.70:0/855412810 >> 172.21.15.70:0/855412810 conn(0x7f5a0407a6a0 msgr2=0x7f5a0407aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:52.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.505+0000 7f5a0c041700 1 -- 172.21.15.70:0/855412810 shutdown_connections 2024-06-04T05:06:52.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.505+0000 7f5a0c041700 1 -- 172.21.15.70:0/855412810 wait complete. 2024-06-04T05:06:52.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.505+0000 7f5a0c041700 1 Processor -- start 2024-06-04T05:06:52.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a0c041700 1 -- start start 2024-06-04T05:06:52.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a0c041700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a041037f0 0x7f5a0419f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a0c041700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a041041a0 0x7f5a0419fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a0c041700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a041088f0 0x7f5a041997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a0c041700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5a04113580 con 0x7f5a041037f0 2024-06-04T05:06:52.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a0c041700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5a04113400 con 0x7f5a041041a0 2024-06-04T05:06:52.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a0c041700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5a04113700 con 0x7f5a041088f0 2024-06-04T05:06:52.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a09ddd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a041037f0 0x7f5a0419f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a09ddd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a041037f0 0x7f5a0419f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40846/0 (socket says 172.21.15.70:40846) 2024-06-04T05:06:52.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a09ddd700 1 -- 172.21.15.70:0/186701276 learned_addr learned my addr 172.21.15.70:0/186701276 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:52.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a0a5de700 1 --2- 172.21.15.70:0/186701276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a041088f0 0x7f5a041997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a095dc700 1 --2- 172.21.15.70:0/186701276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a041041a0 0x7f5a0419fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a09ddd700 1 -- 172.21.15.70:0/186701276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a041088f0 msgr2=0x7f5a041997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:52.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a09ddd700 1 --2- 172.21.15.70:0/186701276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a041088f0 0x7f5a041997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a09ddd700 1 -- 172.21.15.70:0/186701276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a041041a0 msgr2=0x7f5a0419fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:52.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a09ddd700 1 --2- 172.21.15.70:0/186701276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a041041a0 0x7f5a0419fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a09ddd700 1 -- 172.21.15.70:0/186701276 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a04199fc0 con 0x7f5a041037f0 2024-06-04T05:06:52.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a0a5de700 1 --2- 172.21.15.70:0/186701276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a041088f0 0x7f5a041997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:52.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.506+0000 7f5a09ddd700 1 --2- 172.21.15.70:0/186701276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a041037f0 0x7f5a0419f660 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f5a0000c9b0 tx=0x7f5a0000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:52.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.507+0000 7f59faffd700 1 -- 172.21.15.70:0/186701276 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00007b00 con 0x7f5a041037f0 2024-06-04T05:06:52.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.507+0000 7f5a0c041700 1 -- 172.21.15.70:0/186701276 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a0419a2b0 con 0x7f5a041037f0 2024-06-04T05:06:52.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.507+0000 7f59faffd700 1 -- 172.21.15.70:0/186701276 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a0000ce50 con 0x7f5a041037f0 2024-06-04T05:06:52.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.507+0000 7f59faffd700 1 -- 172.21.15.70:0/186701276 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00018660 con 0x7f5a041037f0 2024-06-04T05:06:52.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.507+0000 7f5a0c041700 1 -- 172.21.15.70:0/186701276 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a041a64c0 con 0x7f5a041037f0 2024-06-04T05:06:52.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.509+0000 7f59faffd700 1 -- 172.21.15.70:0/186701276 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f5a0000f450 con 0x7f5a041037f0 2024-06-04T05:06:52.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.510+0000 7f59faffd700 1 --2- 172.21.15.70:0/186701276 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f59f00b2a70 0x7f59f00b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.510+0000 7f59faffd700 1 -- 172.21.15.70:0/186701276 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f5a000e2510 con 0x7f5a041037f0 2024-06-04T05:06:52.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.510+0000 7f5a095dc700 1 --2- 172.21.15.70:0/186701276 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f59f00b2a70 0x7f59f00b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.510+0000 7f5a095dc700 1 --2- 172.21.15.70:0/186701276 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f59f00b2a70 0x7f59f00b4f30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5a04108190 tx=0x7f59f4005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:52.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.510+0000 7f5a168d6b80 1 Processor -- start 2024-06-04T05:06:52.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.511+0000 7f5a168d6b80 1 -- start start 2024-06-04T05:06:52.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.511+0000 7f5a168d6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c670c70 0x55f23c671050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.511+0000 7f5a168d6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c671620 0x55f23c674d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.511+0000 7f5a168d6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c675370 0x55f23c6757b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.511+0000 7f5a168d6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f23c5cbc90 con 0x55f23c671620 2024-06-04T05:06:52.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.511+0000 7f5a168d6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f23c5bb830 con 0x55f23c675370 2024-06-04T05:06:52.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.511+0000 7f5a168d6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f23c5bf870 con 0x55f23c670c70 2024-06-04T05:06:52.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.511+0000 7f5a095dc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c671620 0x55f23c674d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.511+0000 7f5a095dc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c671620 0x55f23c674d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40854/0 (socket says 172.21.15.70:40854) 2024-06-04T05:06:52.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.511+0000 7f5a095dc700 1 -- 172.21.15.70:0/3172788880 learned_addr learned my addr 172.21.15.70:0/3172788880 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:52.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.511+0000 7f5a09ddd700 1 --2- 172.21.15.70:0/3172788880 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c670c70 0x55f23c671050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.511+0000 7f5a0a5de700 1 --2- 172.21.15.70:0/3172788880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c675370 0x55f23c6757b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.511+0000 7f5a09ddd700 1 -- 172.21.15.70:0/3172788880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c675370 msgr2=0x55f23c6757b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:52.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.511+0000 7f5a09ddd700 1 --2- 172.21.15.70:0/3172788880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c675370 0x55f23c6757b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.511+0000 7f5a09ddd700 1 -- 172.21.15.70:0/3172788880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c671620 msgr2=0x55f23c674d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:52.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.511+0000 7f5a09ddd700 1 --2- 172.21.15.70:0/3172788880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c671620 0x55f23c674d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.511+0000 7f5a09ddd700 1 -- 172.21.15.70:0/3172788880 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f23c50b8d0 con 0x55f23c670c70 2024-06-04T05:06:52.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.512+0000 7f5a095dc700 1 --2- 172.21.15.70:0/3172788880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c671620 0x55f23c674d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:52.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.512+0000 7f5a09ddd700 1 --2- 172.21.15.70:0/3172788880 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c670c70 0x55f23c671050 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5a000dfc30 tx=0x7f5a000a92d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=699f9683b2288119 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:52.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.512+0000 7f59effff700 1 -- 172.21.15.70:0/3172788880 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a000a97c0 con 0x55f23c670c70 2024-06-04T05:06:52.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.512+0000 7f59effff700 1 -- 172.21.15.70:0/3172788880 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a000adb70 con 0x55f23c670c70 2024-06-04T05:06:52.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.512+0000 7f59effff700 1 -- 172.21.15.70:0/3172788880 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00091110 con 0x55f23c670c70 2024-06-04T05:06:52.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.513+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3172788880 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c670c70 msgr2=0x55f23c671050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:52.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.513+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/3172788880 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c670c70 0x55f23c671050 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5a000dfc30 tx=0x7f5a000a92d0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.513+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3172788880 shutdown_connections 2024-06-04T05:06:52.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.513+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/3172788880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c675370 0x55f23c6757b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.513+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/3172788880 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c670c70 0x55f23c671050 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.513+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/3172788880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c671620 0x55f23c674d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.513+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3172788880 >> 172.21.15.70:0/3172788880 conn(0x55f23c59a080 msgr2=0x55f23c666e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:52.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.513+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3172788880 shutdown_connections 2024-06-04T05:06:52.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.513+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3172788880 wait complete. 2024-06-04T05:06:52.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.513+0000 7f5a168d6b80 1 -- start start 2024-06-04T05:06:52.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.513+0000 7f5a168d6b80 1 Processor -- start 2024-06-04T05:06:52.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a168d6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c670c70 0x55f23c67c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a168d6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c671620 0x55f23c679b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a168d6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c675370 0x55f23c67a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a168d6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f23c4c1fa0 con 0x55f23c670c70 2024-06-04T05:06:52.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a168d6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f23c55d7c0 con 0x55f23c671620 2024-06-04T05:06:52.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a168d6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f23c67d530 con 0x55f23c675370 2024-06-04T05:06:52.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a168d6b80 10 client.0 _ll_register_callbacks cb 0x55f23c556920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:52.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a0a5de700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c675370 0x55f23c67a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a09ddd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c670c70 0x55f23c67c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a0a5de700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c675370 0x55f23c67a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:58994/0 (socket says 172.21.15.70:58994) 2024-06-04T05:06:52.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a0a5de700 1 -- 172.21.15.70:0/3998517385 learned_addr learned my addr 172.21.15.70:0/3998517385 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:52.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a095dc700 1 --2- 172.21.15.70:0/3998517385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c671620 0x55f23c679b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a09ddd700 1 -- 172.21.15.70:0/3998517385 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c675370 msgr2=0x55f23c67a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:52.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a09ddd700 1 --2- 172.21.15.70:0/3998517385 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c675370 0x55f23c67a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a09ddd700 1 -- 172.21.15.70:0/3998517385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c671620 msgr2=0x55f23c679b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:52.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a09ddd700 1 --2- 172.21.15.70:0/3998517385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c671620 0x55f23c679b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a09ddd700 1 -- 172.21.15.70:0/3998517385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f23c55a690 con 0x55f23c670c70 2024-06-04T05:06:52.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.514+0000 7f5a095dc700 1 --2- 172.21.15.70:0/3998517385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c671620 0x55f23c679b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:06:52.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.515+0000 7f59ed7fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:52.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.515+0000 7f5a09ddd700 1 --2- 172.21.15.70:0/3998517385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c670c70 0x55f23c67c820 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x55f23c67d200 tx=0x7f5a00007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7120459e36f5b5f4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:52.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.515+0000 7f59ed7fa700 1 -- 172.21.15.70:0/3998517385 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a000b2620 con 0x55f23c670c70 2024-06-04T05:06:52.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.515+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3998517385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a000adb70 con 0x55f23c670c70 2024-06-04T05:06:52.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.515+0000 7f59ed7fa700 1 -- 172.21.15.70:0/3998517385 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f5a000a5db0 con 0x55f23c670c70 2024-06-04T05:06:52.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.515+0000 7f59ed7fa700 1 -- 172.21.15.70:0/3998517385 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5a00091320 con 0x55f23c670c70 2024-06-04T05:06:52.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.515+0000 7f5a168d6b80 10 client.18069 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:52.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.515+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3998517385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f23c687c90 con 0x55f23c670c70 2024-06-04T05:06:52.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.515+0000 7f59ed7fa700 1 -- 172.21.15.70:0/3998517385 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a000adb70 con 0x55f23c670c70 2024-06-04T05:06:52.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.515+0000 7f59ed7fa700 1 -- 172.21.15.70:0/3998517385 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f5a0008fce0 con 0x55f23c670c70 2024-06-04T05:06:52.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.515+0000 7f5a168d6b80 10 client.18069 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:52.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.515+0000 7f5a168d6b80 10 client.18069 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:06:52.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.515+0000 7f5a168d6b80 20 client.18069 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:52.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.515+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/3998517385 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f23c68a160 0x55f23c68c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.515+0000 7f5a168d6b80 4 client.18069 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:06:52.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.516+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3998517385 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f23c6389e0 con 0x55f23c68a160 2024-06-04T05:06:52.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.516+0000 7f5a095dc700 1 --2- 172.21.15.70:0/3998517385 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f23c68a160 0x55f23c68c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.516+0000 7f5a095dc700 1 --2- 172.21.15.70:0/3998517385 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f23c68a160 0x55f23c68c550 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d1edc0d15879876 server_cookie=187bd1eb70368b57 in_seq=0 out_seq=0 2024-06-04T05:06:52.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.516+0000 7f59ed7fa700 10 client.18069 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:52.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.517+0000 7f59ed7fa700 1 -- 172.21.15.70:0/3998517385 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f23c6389e0 con 0x55f23c68a160 2024-06-04T05:06:52.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.517+0000 7f59ed7fa700 10 client.18069 handle_command_reply: tid=0 2024-06-04T05:06:52.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.517+0000 7f5a168d6b80 1 client.18069 shutdown 2024-06-04T05:06:52.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.517+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3998517385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c670c70 msgr2=0x55f23c67c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:52.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.517+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/3998517385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c670c70 0x55f23c67c820 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x55f23c67d200 tx=0x7f5a00007ec0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.517+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3998517385 shutdown_connections 2024-06-04T05:06:52.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.517+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/3998517385 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c675370 0x55f23c67a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.517+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/3998517385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c670c70 0x55f23c67c820 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.517+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/3998517385 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f23c68a160 0x55f23c68c550 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.517+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/3998517385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c671620 0x55f23c679b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.517+0000 7f59ed7fa700 0 client.18069 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:52.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.517+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3998517385 >> 172.21.15.70:0/3998517385 conn(0x55f23c59a080 msgr2=0x55f23c666510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:52.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.517+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3998517385 shutdown_connections 2024-06-04T05:06:52.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.517+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3998517385 wait complete. 2024-06-04T05:06:52.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.517+0000 7f5a168d6b80 20 client.18069 trim_cache size 0 max 16384 2024-06-04T05:06:52.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.534+0000 7f5a168d6b80 1 Processor -- start 2024-06-04T05:06:52.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.534+0000 7f5a168d6b80 1 -- start start 2024-06-04T05:06:52.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.534+0000 7f5a168d6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c670c70 0x55f23c67ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.534+0000 7f5a168d6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c671620 0x55f23c666c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.534+0000 7f5a168d6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c675370 0x55f23c6857c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.534+0000 7f5a168d6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5a041aba90 con 0x55f23c675370 2024-06-04T05:06:52.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.534+0000 7f5a168d6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5a041ad5d0 con 0x55f23c670c70 2024-06-04T05:06:52.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.534+0000 7f5a168d6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5a041ac160 con 0x55f23c671620 2024-06-04T05:06:52.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.534+0000 7f5a0a5de700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c675370 0x55f23c6857c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.534+0000 7f5a09ddd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c670c70 0x55f23c67ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.534+0000 7f5a0a5de700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c675370 0x55f23c6857c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40874/0 (socket says 172.21.15.70:40874) 2024-06-04T05:06:52.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.534+0000 7f5a095dc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c671620 0x55f23c666c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.534+0000 7f5a09ddd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c670c70 0x55f23c67ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54858/0 (socket says 172.21.15.70:54858) 2024-06-04T05:06:52.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.534+0000 7f5a0a5de700 1 -- 172.21.15.70:0/1993085702 learned_addr learned my addr 172.21.15.70:0/1993085702 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:52.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.535+0000 7f5a0a5de700 1 -- 172.21.15.70:0/1993085702 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c671620 msgr2=0x55f23c666c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:52.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.535+0000 7f5a0a5de700 1 --2- 172.21.15.70:0/1993085702 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c671620 0x55f23c666c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.535+0000 7f5a0a5de700 1 -- 172.21.15.70:0/1993085702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c670c70 msgr2=0x55f23c67ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:52.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.535+0000 7f5a0a5de700 1 --2- 172.21.15.70:0/1993085702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c670c70 0x55f23c67ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.535+0000 7f5a0a5de700 1 -- 172.21.15.70:0/1993085702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f23c67aa50 con 0x55f23c675370 2024-06-04T05:06:52.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.535+0000 7f5a09ddd700 1 --2- 172.21.15.70:0/1993085702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c670c70 0x55f23c67ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:06:52.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.535+0000 7f5a095dc700 1 --2- 172.21.15.70:0/1993085702 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c671620 0x55f23c666c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:52.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.535+0000 7f5a0a5de700 1 --2- 172.21.15.70:0/1993085702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c675370 0x55f23c6857c0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00e810 tx=0x7f59fc009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92b8b6cd6ef261f2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:52.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.535+0000 7f59ef7fe700 1 -- 172.21.15.70:0/1993085702 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc020670 con 0x55f23c675370 2024-06-04T05:06:52.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.535+0000 7f59ef7fe700 1 -- 172.21.15.70:0/1993085702 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f59fc01b070 con 0x55f23c675370 2024-06-04T05:06:52.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.535+0000 7f59ef7fe700 1 -- 172.21.15.70:0/1993085702 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc008a10 con 0x55f23c675370 2024-06-04T05:06:52.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.536+0000 7f5a168d6b80 1 -- 172.21.15.70:0/1993085702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c675370 msgr2=0x55f23c6857c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:52.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.536+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/1993085702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c675370 0x55f23c6857c0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00e810 tx=0x7f59fc009920 comp rx=0 tx=0).stop 2024-06-04T05:06:52.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.536+0000 7f5a168d6b80 1 -- 172.21.15.70:0/1993085702 shutdown_connections 2024-06-04T05:06:52.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.536+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/1993085702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c675370 0x55f23c6857c0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.536+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/1993085702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c670c70 0x55f23c67ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.536+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/1993085702 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c671620 0x55f23c666c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.536+0000 7f5a168d6b80 1 -- 172.21.15.70:0/1993085702 >> 172.21.15.70:0/1993085702 conn(0x55f23c59a080 msgr2=0x55f23c67b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:52.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.536+0000 7f5a168d6b80 1 -- 172.21.15.70:0/1993085702 shutdown_connections 2024-06-04T05:06:52.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.536+0000 7f5a168d6b80 1 -- 172.21.15.70:0/1993085702 wait complete. 2024-06-04T05:06:52.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.536+0000 7f5a168d6b80 1 -- start start 2024-06-04T05:06:52.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.537+0000 7f5a168d6b80 1 Processor -- start 2024-06-04T05:06:52.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.537+0000 7f5a168d6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c670c70 0x55f23c68e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.537+0000 7f5a168d6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c671620 0x55f23c68eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.537+0000 7f5a168d6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c675370 0x55f23c6f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.537+0000 7f5a168d6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5a041acf00 con 0x55f23c671620 2024-06-04T05:06:52.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.537+0000 7f5a168d6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5a041ac830 con 0x55f23c670c70 2024-06-04T05:06:52.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.537+0000 7f5a168d6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5a041a9a50 con 0x55f23c675370 2024-06-04T05:06:52.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.537+0000 7f5a168d6b80 10 client.0 _ll_register_callbacks cb 0x55f23c556920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:52.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.537+0000 7f5a095dc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c671620 0x55f23c68eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.537+0000 7f5a095dc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c671620 0x55f23c68eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40880/0 (socket says 172.21.15.70:40880) 2024-06-04T05:06:52.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.537+0000 7f5a095dc700 1 -- 172.21.15.70:0/3244446944 learned_addr learned my addr 172.21.15.70:0/3244446944 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:52.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.537+0000 7f5a0a5de700 1 --2- 172.21.15.70:0/3244446944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c675370 0x55f23c6f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.537+0000 7f5a09ddd700 1 --2- 172.21.15.70:0/3244446944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c670c70 0x55f23c68e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f5a095dc700 1 -- 172.21.15.70:0/3244446944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c675370 msgr2=0x55f23c6f29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:52.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f5a095dc700 1 --2- 172.21.15.70:0/3244446944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c675370 0x55f23c6f29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f5a095dc700 1 -- 172.21.15.70:0/3244446944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c670c70 msgr2=0x55f23c68e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:52.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f5a095dc700 1 --2- 172.21.15.70:0/3244446944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c670c70 0x55f23c68e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f5a095dc700 1 -- 172.21.15.70:0/3244446944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f23c6f2f90 con 0x55f23c671620 2024-06-04T05:06:52.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f5a09ddd700 1 --2- 172.21.15.70:0/3244446944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c670c70 0x55f23c68e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:06:52.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f5a095dc700 1 --2- 172.21.15.70:0/3244446944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c671620 0x55f23c68eb90 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f59f4000c00 tx=0x7f59f4009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d8fe5ddce6b7e1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:52.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f59ed7fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:52.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f59ed7fa700 1 -- 172.21.15.70:0/3244446944 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f400bba0 con 0x55f23c671620 2024-06-04T05:06:52.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3244446944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59fc01b070 con 0x55f23c671620 2024-06-04T05:06:52.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f59ed7fa700 1 -- 172.21.15.70:0/3244446944 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f59f400bd40 con 0x55f23c671620 2024-06-04T05:06:52.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f59ed7fa700 1 -- 172.21.15.70:0/3244446944 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f59f403d420 con 0x55f23c671620 2024-06-04T05:06:52.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f5a168d6b80 10 client.18075 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:52.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3244446944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f23c6f4370 con 0x55f23c671620 2024-06-04T05:06:52.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f59ed7fa700 1 -- 172.21.15.70:0/3244446944 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f59fc01b070 con 0x55f23c671620 2024-06-04T05:06:52.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f59ed7fa700 1 -- 172.21.15.70:0/3244446944 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f59f4033410 con 0x55f23c671620 2024-06-04T05:06:52.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f5a168d6b80 10 client.18075 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:52.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f5a168d6b80 10 client.18075 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:06:52.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f5a168d6b80 20 client.18075 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:52.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/3244446944 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f23c6f4a60 0x55f23c6f6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:52.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f5a168d6b80 4 client.18075 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:06:52.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.538+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3244446944 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f23c5cdbe0 con 0x55f23c6f4a60 2024-06-04T05:06:52.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.539+0000 7f5a09ddd700 1 --2- 172.21.15.70:0/3244446944 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f23c6f4a60 0x55f23c6f6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:52.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.539+0000 7f59ed7fa700 10 client.18075 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:52.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.539+0000 7f5a09ddd700 1 --2- 172.21.15.70:0/3244446944 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f23c6f4a60 0x55f23c6f6e80 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e58e427427497e3 server_cookie=5433809031ae783d in_seq=0 out_seq=0 2024-06-04T05:06:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.539+0000 7f59ed7fa700 1 -- 172.21.15.70:0/3244446944 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f23c5cdbe0 con 0x55f23c6f4a60 2024-06-04T05:06:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.539+0000 7f59ed7fa700 10 client.18075 handle_command_reply: tid=0 2024-06-04T05:06:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.539+0000 7f5a168d6b80 1 client.18075 shutdown 2024-06-04T05:06:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.539+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3244446944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c671620 msgr2=0x55f23c68eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.539+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/3244446944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c671620 0x55f23c68eb90 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f59f4000c00 tx=0x7f59f4009cd0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.539+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3244446944 shutdown_connections 2024-06-04T05:06:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.539+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/3244446944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f23c675370 0x55f23c6f29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.539+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/3244446944 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f23c6f4a60 0x55f23c6f6e80 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.539+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/3244446944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f23c670c70 0x55f23c68e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.539+0000 7f59ed7fa700 0 client.18075 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.539+0000 7f5a168d6b80 1 --2- 172.21.15.70:0/3244446944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23c671620 0x55f23c68eb90 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.539+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3244446944 >> 172.21.15.70:0/3244446944 conn(0x55f23c59a080 msgr2=0x55f23c674550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:52.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.540+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3244446944 shutdown_connections 2024-06-04T05:06:52.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.540+0000 7f5a168d6b80 1 -- 172.21.15.70:0/3244446944 wait complete. 2024-06-04T05:06:52.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.540+0000 7f5a168d6b80 20 client.18075 trim_cache size 0 max 16384 2024-06-04T05:06:52.549 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:06:52.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.540+0000 7f5a0c041700 1 -- 172.21.15.70:0/186701276 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f59f00b2a70 msgr2=0x7f59f00b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:52.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.540+0000 7f5a0c041700 1 --2- 172.21.15.70:0/186701276 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f59f00b2a70 0x7f59f00b4f30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5a04108190 tx=0x7f59f4005fb0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.540+0000 7f5a0c041700 1 -- 172.21.15.70:0/186701276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a041037f0 msgr2=0x7f5a0419f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:52.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.540+0000 7f5a0c041700 1 --2- 172.21.15.70:0/186701276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a041037f0 0x7f5a0419f660 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f5a0000c9b0 tx=0x7f5a0000ccc0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.540+0000 7f5a0c041700 1 -- 172.21.15.70:0/186701276 shutdown_connections 2024-06-04T05:06:52.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.540+0000 7f5a0c041700 1 --2- 172.21.15.70:0/186701276 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f59f00b2a70 0x7f59f00b4f30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.540+0000 7f5a0c041700 1 --2- 172.21.15.70:0/186701276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a041088f0 0x7f5a041997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.540+0000 7f5a0c041700 1 --2- 172.21.15.70:0/186701276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a041037f0 0x7f5a0419f660 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.540+0000 7f5a0c041700 1 --2- 172.21.15.70:0/186701276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a041041a0 0x7f5a0419fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:52.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.540+0000 7f5a0c041700 1 -- 172.21.15.70:0/186701276 >> 172.21.15.70:0/186701276 conn(0x7f5a0407a6a0 msgr2=0x7f5a04101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:52.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.540+0000 7f5a0c041700 1 -- 172.21.15.70:0/186701276 shutdown_connections 2024-06-04T05:06:52.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:52.540+0000 7f5a0c041700 1 -- 172.21.15.70:0/186701276 wait complete. 2024-06-04T05:06:52.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 166838 1.90 MB/sec execute 345 sec latency 14.232 ms 2024-06-04T05:06:53.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:53 smithi070 ceph-mon[32631]: pgmap v298: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 2.1 MiB/s wr, 190 op/s 2024-06-04T05:06:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:53 smithi112 ceph-mon[36453]: pgmap v298: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 2.1 MiB/s wr, 190 op/s 2024-06-04T05:06:53.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 167142 1.90 MB/sec execute 346 sec latency 13.193 ms 2024-06-04T05:06:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:53 smithi098 ceph-mon[35569]: pgmap v298: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 2.1 MiB/s wr, 190 op/s 2024-06-04T05:06:54.378 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:06:54.379 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:06:54.645 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:06:54.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 167470 1.90 MB/sec execute 347 sec latency 18.082 ms 2024-06-04T05:06:55.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:55 smithi070 ceph-mon[32631]: pgmap v299: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 1.3 MiB/s wr, 126 op/s 2024-06-04T05:06:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:55 smithi112 ceph-mon[36453]: pgmap v299: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 1.3 MiB/s wr, 126 op/s 2024-06-04T05:06:55.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 167907 1.90 MB/sec execute 348 sec latency 79.593 ms 2024-06-04T05:06:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:55 smithi098 ceph-mon[35569]: pgmap v299: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 1.3 MiB/s wr, 126 op/s 2024-06-04T05:06:56.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 168345 1.90 MB/sec execute 349 sec latency 112.120 ms 2024-06-04T05:06:57.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:57 smithi070 ceph-mon[32631]: pgmap v300: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 2.0 MiB/s wr, 191 op/s 2024-06-04T05:06:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:57 smithi112 ceph-mon[36453]: pgmap v300: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 2.0 MiB/s wr, 191 op/s 2024-06-04T05:06:57.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 168900 1.89 MB/sec execute 350 sec latency 74.397 ms 2024-06-04T05:06:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:57 smithi098 ceph-mon[35569]: pgmap v300: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 2.0 MiB/s wr, 191 op/s 2024-06-04T05:06:58.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 169349 1.90 MB/sec execute 351 sec latency 61.109 ms 2024-06-04T05:06:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.226+0000 7fcdf0916700 1 -- 172.21.15.70:0/4249266938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdec102440 msgr2=0x7fcdec1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.226+0000 7fcdf0916700 1 --2- 172.21.15.70:0/4249266938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdec102440 0x7fcdec1028c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009990 tx=0x7fcde0009ca0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.226+0000 7fcdf0916700 1 -- 172.21.15.70:0/4249266938 shutdown_connections 2024-06-04T05:06:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.226+0000 7fcdf0916700 1 --2- 172.21.15.70:0/4249266938 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec102e00 0x7fcdec10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.226+0000 7fcdf0916700 1 --2- 172.21.15.70:0/4249266938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdec108520 0x7fcdec108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.226+0000 7fcdf0916700 1 --2- 172.21.15.70:0/4249266938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdec102440 0x7fcdec1028c0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.226+0000 7fcdf0916700 1 -- 172.21.15.70:0/4249266938 >> 172.21.15.70:0/4249266938 conn(0x7fcdec0793f0 msgr2=0x7fcdec079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.226+0000 7fcdf0916700 1 -- 172.21.15.70:0/4249266938 shutdown_connections 2024-06-04T05:06:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.226+0000 7fcdf0916700 1 -- 172.21.15.70:0/4249266938 wait complete. 2024-06-04T05:06:59.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.227+0000 7fcdf0916700 1 Processor -- start 2024-06-04T05:06:59.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.227+0000 7fcdf0916700 1 -- start start 2024-06-04T05:06:59.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.227+0000 7fcdf0916700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec102440 0x7fcdec19bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.227+0000 7fcdf0916700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdec102e00 0x7fcdec19c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.227+0000 7fcdf0916700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdec108520 0x7fcdec1a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.227+0000 7fcdf0916700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec1113b0 con 0x7fcdec108520 2024-06-04T05:06:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.227+0000 7fcdf0916700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec111230 con 0x7fcdec102e00 2024-06-04T05:06:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.227+0000 7fcdf0916700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec111530 con 0x7fcdec102440 2024-06-04T05:06:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcdea59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec102440 0x7fcdec19bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcdead9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdec108520 0x7fcdec1a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcde9d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdec102e00 0x7fcdec19c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcdea59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec102440 0x7fcdec19bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:54832/0 (socket says 172.21.15.70:54832) 2024-06-04T05:06:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcdea59c700 1 -- 172.21.15.70:0/3422270291 learned_addr learned my addr 172.21.15.70:0/3422270291 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcde9d9b700 1 -- 172.21.15.70:0/3422270291 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec102440 msgr2=0x7fcdec19bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcde9d9b700 1 --2- 172.21.15.70:0/3422270291 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec102440 0x7fcdec19bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcde9d9b700 1 -- 172.21.15.70:0/3422270291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdec108520 msgr2=0x7fcdec1a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcde9d9b700 1 --2- 172.21.15.70:0/3422270291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdec108520 0x7fcdec1a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcde9d9b700 1 -- 172.21.15.70:0/3422270291 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdec1a0d40 con 0x7fcdec102e00 2024-06-04T05:06:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcdea59c700 1 --2- 172.21.15.70:0/3422270291 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec102440 0x7fcdec19bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcdead9d700 1 --2- 172.21.15.70:0/3422270291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdec108520 0x7fcdec1a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcde9d9b700 1 --2- 172.21.15.70:0/3422270291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdec102e00 0x7fcdec19c270 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcde0000c00 tx=0x7fcde000dd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcddb7fe700 1 -- 172.21.15.70:0/3422270291 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde001a030 con 0x7fcdec102e00 2024-06-04T05:06:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcddb7fe700 1 -- 172.21.15.70:0/3422270291 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcde001c070 con 0x7fcdec102e00 2024-06-04T05:06:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcdf0916700 1 -- 172.21.15.70:0/3422270291 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdec1a0fd0 con 0x7fcdec102e00 2024-06-04T05:06:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcddb7fe700 1 -- 172.21.15.70:0/3422270291 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde0015bb0 con 0x7fcdec102e00 2024-06-04T05:06:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.228+0000 7fcdf0916700 1 -- 172.21.15.70:0/3422270291 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdec1a88b0 con 0x7fcdec102e00 2024-06-04T05:06:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.232+0000 7fcddb7fe700 1 -- 172.21.15.70:0/3422270291 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fcde0027060 con 0x7fcdec102e00 2024-06-04T05:06:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.232+0000 7fcddb7fe700 1 --2- 172.21.15.70:0/3422270291 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcdd40b2a70 0x7fcdd40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.233+0000 7fcdfb1abb80 1 Processor -- start 2024-06-04T05:06:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.233+0000 7fcdea59c700 1 --2- 172.21.15.70:0/3422270291 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcdd40b2a70 0x7fcdd40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.233+0000 7fcddb7fe700 1 -- 172.21.15.70:0/3422270291 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7fcde00e2950 con 0x7fcdec102e00 2024-06-04T05:06:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.233+0000 7fcdfb1abb80 1 -- start start 2024-06-04T05:06:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.233+0000 7fcdfb1abb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a783a0 0x55c910a78780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.233+0000 7fcdfb1abb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a72920 0x55c910a72da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.233+0000 7fcdfb1abb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a732e0 0x55c910a7f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.233+0000 7fcdfb1abb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c9109d0df0 con 0x55c910a72920 2024-06-04T05:06:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.233+0000 7fcdfb1abb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c9109c0990 con 0x55c910a732e0 2024-06-04T05:06:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.233+0000 7fcdfb1abb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c9109c49d0 con 0x55c910a783a0 2024-06-04T05:06:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.233+0000 7fcde9d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a72920 0x55c910a72da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.233+0000 7fcdea59c700 1 --2- 172.21.15.70:0/3422270291 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcdd40b2a70 0x7fcdd40b4f30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcddc011550 tx=0x7fcddc009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.233+0000 7fcdea59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a783a0 0x55c910a78780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.233+0000 7fcde9d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a72920 0x55c910a72da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59726/0 (socket says 172.21.15.70:59726) 2024-06-04T05:06:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.233+0000 7fcde9d9b700 1 -- 172.21.15.70:0/2561320326 learned_addr learned my addr 172.21.15.70:0/2561320326 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.234+0000 7fcdead9d700 1 --2- 172.21.15.70:0/2561320326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a732e0 0x55c910a7f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.234+0000 7fcdea59c700 1 -- 172.21.15.70:0/2561320326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a732e0 msgr2=0x55c910a7f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.234+0000 7fcdea59c700 1 --2- 172.21.15.70:0/2561320326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a732e0 0x55c910a7f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.234+0000 7fcdea59c700 1 -- 172.21.15.70:0/2561320326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a72920 msgr2=0x55c910a72da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.234+0000 7fcdea59c700 1 --2- 172.21.15.70:0/2561320326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a72920 0x55c910a72da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.234+0000 7fcdea59c700 1 -- 172.21.15.70:0/2561320326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c91096a030 con 0x55c910a783a0 2024-06-04T05:06:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.234+0000 7fcdea59c700 1 --2- 172.21.15.70:0/2561320326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a783a0 0x55c910a78780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009880 tx=0x7fcddc014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8baa6e4895e89f0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.234+0000 7fcdd8ff9700 1 -- 172.21.15.70:0/2561320326 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc00edc0 con 0x55c910a783a0 2024-06-04T05:06:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.234+0000 7fcdd8ff9700 1 -- 172.21.15.70:0/2561320326 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcddc01e8b0 con 0x55c910a783a0 2024-06-04T05:06:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.234+0000 7fcdd8ff9700 1 -- 172.21.15.70:0/2561320326 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc01d7e0 con 0x55c910a783a0 2024-06-04T05:06:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.235+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/2561320326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a783a0 msgr2=0x55c910a78780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.235+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/2561320326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a783a0 0x55c910a78780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009880 tx=0x7fcddc014000 comp rx=0 tx=0).stop 2024-06-04T05:06:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.235+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/2561320326 shutdown_connections 2024-06-04T05:06:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.235+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/2561320326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a732e0 0x55c910a7f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.235+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/2561320326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a783a0 0x55c910a78780 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.235+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/2561320326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a72920 0x55c910a72da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.235+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/2561320326 >> 172.21.15.70:0/2561320326 conn(0x55c91099e2c0 msgr2=0x55c910a6d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.235+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/2561320326 shutdown_connections 2024-06-04T05:06:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.235+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/2561320326 wait complete. 2024-06-04T05:06:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.236+0000 7fcdfb1abb80 1 -- start start 2024-06-04T05:06:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.236+0000 7fcdfb1abb80 1 Processor -- start 2024-06-04T05:06:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.236+0000 7fcdfb1abb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a72920 0x55c910a7ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.236+0000 7fcdfb1abb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a732e0 0x55c910a7c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.236+0000 7fcdfb1abb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a783a0 0x55c910a7c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.236+0000 7fcdfb1abb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c9108c7020 con 0x55c910a72920 2024-06-04T05:06:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.236+0000 7fcdfb1abb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c910928e20 con 0x55c910a732e0 2024-06-04T05:06:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.236+0000 7fcdfb1abb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c910a7f760 con 0x55c910a783a0 2024-06-04T05:06:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.236+0000 7fcdea59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a72920 0x55c910a7ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.236+0000 7fcdfb1abb80 10 client.0 _ll_register_callbacks cb 0x55c91095b9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.236+0000 7fcdea59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a72920 0x55c910a7ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59728/0 (socket says 172.21.15.70:59728) 2024-06-04T05:06:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.236+0000 7fcdea59c700 1 -- 172.21.15.70:0/1799242790 learned_addr learned my addr 172.21.15.70:0/1799242790 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.236+0000 7fcdead9d700 1 --2- 172.21.15.70:0/1799242790 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a783a0 0x55c910a7c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.236+0000 7fcde9d9b700 1 --2- 172.21.15.70:0/1799242790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a732e0 0x55c910a7c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.237+0000 7fcdea59c700 1 -- 172.21.15.70:0/1799242790 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a783a0 msgr2=0x55c910a7c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.237+0000 7fcdea59c700 1 --2- 172.21.15.70:0/1799242790 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a783a0 0x55c910a7c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.237+0000 7fcdea59c700 1 -- 172.21.15.70:0/1799242790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a732e0 msgr2=0x55c910a7c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.237+0000 7fcdea59c700 1 --2- 172.21.15.70:0/1799242790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a732e0 0x55c910a7c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.237+0000 7fcdea59c700 1 -- 172.21.15.70:0/1799242790 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c910910950 con 0x55c910a72920 2024-06-04T05:06:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.237+0000 7fcdead9d700 1 --2- 172.21.15.70:0/1799242790 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a783a0 0x55c910a7c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.237+0000 7fcde9d9b700 1 --2- 172.21.15.70:0/1799242790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a732e0 0x55c910a7c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.237+0000 7fcdea59c700 1 --2- 172.21.15.70:0/1799242790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a72920 0x55c910a7ed50 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fcddc011da0 tx=0x7fcddc0090c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59753704570a9729 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.237+0000 7fcdd1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.237+0000 7fcdd1ffb700 1 -- 172.21.15.70:0/1799242790 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc034060 con 0x55c910a72920 2024-06-04T05:06:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.237+0000 7fcdd1ffb700 1 -- 172.21.15.70:0/1799242790 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7fcddc02f480 con 0x55c910a72920 2024-06-04T05:06:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.237+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/1799242790 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcddc01e8b0 con 0x55c910a72920 2024-06-04T05:06:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.238+0000 7fcdd1ffb700 1 -- 172.21.15.70:0/1799242790 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcddc02f680 con 0x55c910a72920 2024-06-04T05:06:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.238+0000 7fcdfb1abb80 10 client.18087 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.238+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/1799242790 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c91097a7e0 con 0x55c910a72920 2024-06-04T05:06:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.238+0000 7fcdd1ffb700 1 -- 172.21.15.70:0/1799242790 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcddc01e8b0 con 0x55c910a72920 2024-06-04T05:06:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.238+0000 7fcdd1ffb700 1 -- 172.21.15.70:0/1799242790 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fcddc037450 con 0x55c910a72920 2024-06-04T05:06:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.238+0000 7fcdfb1abb80 10 client.18087 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.238+0000 7fcdfb1abb80 10 client.18087 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:06:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.238+0000 7fcdfb1abb80 20 client.18087 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.238+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/1799242790 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c910a8c1a0 0x55c910a8e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.238+0000 7fcdfb1abb80 4 client.18087 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:06:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.238+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/1799242790 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9109d2d40 con 0x55c910a8c1a0 2024-06-04T05:06:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.239+0000 7fcde9d9b700 1 --2- 172.21.15.70:0/1799242790 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c910a8c1a0 0x55c910a8e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.239+0000 7fcdd1ffb700 10 client.18087 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.239+0000 7fcde9d9b700 1 --2- 172.21.15.70:0/1799242790 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c910a8c1a0 0x55c910a8e590 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4f22864d8ae750e server_cookie=92635dee29cc527f in_seq=0 out_seq=0 2024-06-04T05:06:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.240+0000 7fcdd1ffb700 1 -- 172.21.15.70:0/1799242790 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c9109d2d40 con 0x55c910a8c1a0 2024-06-04T05:06:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.240+0000 7fcdd1ffb700 10 client.18087 handle_command_reply: tid=0 2024-06-04T05:06:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.240+0000 7fcdfb1abb80 1 client.18087 shutdown 2024-06-04T05:06:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.240+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/1799242790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a72920 msgr2=0x55c910a7ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.240+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/1799242790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a72920 0x55c910a7ed50 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fcddc011da0 tx=0x7fcddc0090c0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.240+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/1799242790 shutdown_connections 2024-06-04T05:06:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.240+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/1799242790 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a783a0 0x55c910a7c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.240+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/1799242790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a72920 0x55c910a7ed50 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.240+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/1799242790 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c910a8c1a0 0x55c910a8e590 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.240+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/1799242790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a732e0 0x55c910a7c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.240+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/1799242790 >> 172.21.15.70:0/1799242790 conn(0x55c91099e2c0 msgr2=0x55c910a6cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.240+0000 7fcdd1ffb700 0 client.18087 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:59.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.240+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/1799242790 shutdown_connections 2024-06-04T05:06:59.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.240+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/1799242790 wait complete. 2024-06-04T05:06:59.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.240+0000 7fcdfb1abb80 20 client.18087 trim_cache size 0 max 16384 2024-06-04T05:06:59.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.262+0000 7fcdfb1abb80 1 Processor -- start 2024-06-04T05:06:59.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.262+0000 7fcdfb1abb80 1 -- start start 2024-06-04T05:06:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.262+0000 7fcdfb1abb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a72920 0x55c910a6d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.262+0000 7fcdfb1abb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a732e0 0x55c910a915b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.262+0000 7fcdfb1abb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a783a0 0x55c910a91af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.262+0000 7fcdfb1abb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec1aff30 con 0x55c910a783a0 2024-06-04T05:06:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.262+0000 7fcdfb1abb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec1b1a70 con 0x55c910a732e0 2024-06-04T05:06:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.262+0000 7fcdfb1abb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec1b0600 con 0x55c910a72920 2024-06-04T05:06:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.263+0000 7fcdea59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a72920 0x55c910a6d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.263+0000 7fcdea59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a72920 0x55c910a6d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:54858/0 (socket says 172.21.15.70:54858) 2024-06-04T05:06:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.263+0000 7fcde9d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a732e0 0x55c910a915b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.263+0000 7fcdea59c700 1 -- 172.21.15.70:0/477109281 learned_addr learned my addr 172.21.15.70:0/477109281 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.263+0000 7fcdead9d700 1 --2- 172.21.15.70:0/477109281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a783a0 0x55c910a91af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.263+0000 7fcdea59c700 1 -- 172.21.15.70:0/477109281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a732e0 msgr2=0x55c910a915b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.263+0000 7fcdea59c700 1 --2- 172.21.15.70:0/477109281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a732e0 0x55c910a915b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.263+0000 7fcdea59c700 1 -- 172.21.15.70:0/477109281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a783a0 msgr2=0x55c910a91af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.263+0000 7fcdea59c700 1 --2- 172.21.15.70:0/477109281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a783a0 0x55c910a91af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.263+0000 7fcdea59c700 1 -- 172.21.15.70:0/477109281 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c910a77fc0 con 0x55c910a72920 2024-06-04T05:06:59.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.263+0000 7fcde9d9b700 1 --2- 172.21.15.70:0/477109281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a732e0 0x55c910a915b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:06:59.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.263+0000 7fcdea59c700 1 --2- 172.21.15.70:0/477109281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a72920 0x55c910a6d2f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009880 tx=0x7fcddc0093b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5665414a49af5e41 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:59.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.267+0000 7fcdd3fff700 1 -- 172.21.15.70:0/477109281 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc034050 con 0x55c910a72920 2024-06-04T05:06:59.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.267+0000 7fcdd3fff700 1 -- 172.21.15.70:0/477109281 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcddc02fcb0 con 0x55c910a72920 2024-06-04T05:06:59.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.267+0000 7fcdd3fff700 1 -- 172.21.15.70:0/477109281 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc01e7a0 con 0x55c910a72920 2024-06-04T05:06:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.267+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/477109281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a72920 msgr2=0x55c910a6d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.267+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/477109281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a72920 0x55c910a6d2f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009880 tx=0x7fcddc0093b0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/477109281 shutdown_connections 2024-06-04T05:06:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/477109281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a783a0 0x55c910a91af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/477109281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a72920 0x55c910a6d2f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/477109281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a732e0 0x55c910a915b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/477109281 >> 172.21.15.70:0/477109281 conn(0x55c91099e2c0 msgr2=0x55c910a721a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/477109281 shutdown_connections 2024-06-04T05:06:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/477109281 wait complete. 2024-06-04T05:06:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcdfb1abb80 1 -- start start 2024-06-04T05:06:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcdfb1abb80 1 Processor -- start 2024-06-04T05:06:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcdfb1abb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a72920 0x55c910a91be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcdfb1abb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a732e0 0x55c910a8ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcdfb1abb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a783a0 0x55c910a904d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcdfb1abb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec1b13a0 con 0x55c910a732e0 2024-06-04T05:06:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcdfb1abb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec1b0cd0 con 0x55c910a72920 2024-06-04T05:06:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcdfb1abb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec1adef0 con 0x55c910a783a0 2024-06-04T05:06:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcdfb1abb80 10 client.0 _ll_register_callbacks cb 0x55c91095b9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:06:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcde9d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a732e0 0x55c910a8ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.268+0000 7fcde9d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a732e0 0x55c910a8ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59740/0 (socket says 172.21.15.70:59740) 2024-06-04T05:06:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.269+0000 7fcde9d9b700 1 -- 172.21.15.70:0/3132349105 learned_addr learned my addr 172.21.15.70:0/3132349105 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:06:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.269+0000 7fcdead9d700 1 --2- 172.21.15.70:0/3132349105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a783a0 0x55c910a904d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.269+0000 7fcdea59c700 1 --2- 172.21.15.70:0/3132349105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a72920 0x55c910a91be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.269+0000 7fcde9d9b700 1 -- 172.21.15.70:0/3132349105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a783a0 msgr2=0x55c910a904d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.269+0000 7fcde9d9b700 1 --2- 172.21.15.70:0/3132349105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a783a0 0x55c910a904d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.269+0000 7fcde9d9b700 1 -- 172.21.15.70:0/3132349105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a72920 msgr2=0x55c910a91be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:06:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.269+0000 7fcde9d9b700 1 --2- 172.21.15.70:0/3132349105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a72920 0x55c910a91be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.269+0000 7fcde9d9b700 1 -- 172.21.15.70:0/3132349105 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c910a90d10 con 0x55c910a732e0 2024-06-04T05:06:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.269+0000 7fcdea59c700 1 --2- 172.21.15.70:0/3132349105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a72920 0x55c910a91be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:06:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.269+0000 7fcde9d9b700 1 --2- 172.21.15.70:0/3132349105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a732e0 0x55c910a8ff90 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fcde00a9d30 tx=0x7fcde00096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b3c91933e4076 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:06:59.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.269+0000 7fcdd1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:06:59.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.269+0000 7fcdd1ffb700 1 -- 172.21.15.70:0/3132349105 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde00aa4a0 con 0x55c910a732e0 2024-06-04T05:06:59.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.269+0000 7fcdd1ffb700 1 -- 172.21.15.70:0/3132349105 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7fcde00c43a0 con 0x55c910a732e0 2024-06-04T05:06:59.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.269+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/3132349105 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcddc02fcb0 con 0x55c910a732e0 2024-06-04T05:06:59.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.269+0000 7fcdd1ffb700 1 -- 172.21.15.70:0/3132349105 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcde00a9da0 con 0x55c910a732e0 2024-06-04T05:06:59.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.270+0000 7fcdfb1abb80 10 client.18093 fetch_fsmap learned FSMap version 32 2024-06-04T05:06:59.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.270+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/3132349105 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c910af79f0 con 0x55c910a732e0 2024-06-04T05:06:59.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.270+0000 7fcdd1ffb700 1 -- 172.21.15.70:0/3132349105 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcde00ae150 con 0x55c910a732e0 2024-06-04T05:06:59.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.270+0000 7fcdd1ffb700 1 -- 172.21.15.70:0/3132349105 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fcde0027be0 con 0x55c910a732e0 2024-06-04T05:06:59.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.270+0000 7fcdfb1abb80 10 client.18093 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:06:59.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.270+0000 7fcdfb1abb80 10 client.18093 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:06:59.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.270+0000 7fcdfb1abb80 20 client.18093 populate_metadata read hostname 'smithi070' 2024-06-04T05:06:59.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.270+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/3132349105 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c910af8180 0x55c910afa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:06:59.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.270+0000 7fcdfb1abb80 4 client.18093 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:06:59.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.270+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/3132349105 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fcddc02fcb0 con 0x55c910af8180 2024-06-04T05:06:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.270+0000 7fcdea59c700 1 --2- 172.21.15.70:0/3132349105 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c910af8180 0x55c910afa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:06:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.271+0000 7fcdd1ffb700 10 client.18093 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.271+0000 7fcdea59c700 1 --2- 172.21.15.70:0/3132349105 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c910af8180 0x55c910afa5a0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2ab2a391c191a1e server_cookie=3f97f9e7a6d0b998 in_seq=0 out_seq=0 2024-06-04T05:06:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.272+0000 7fcdd1ffb700 1 -- 172.21.15.70:0/3132349105 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fcddc02fcb0 con 0x55c910af8180 2024-06-04T05:06:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.272+0000 7fcdd1ffb700 10 client.18093 handle_command_reply: tid=0 2024-06-04T05:06:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.272+0000 7fcdfb1abb80 1 client.18093 shutdown 2024-06-04T05:06:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.272+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/3132349105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a732e0 msgr2=0x55c910a8ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.272+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/3132349105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a732e0 0x55c910a8ff90 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fcde00a9d30 tx=0x7fcde00096c0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.272+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/3132349105 shutdown_connections 2024-06-04T05:06:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.272+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/3132349105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c910a783a0 0x55c910a904d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.272+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/3132349105 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c910af8180 0x55c910afa5a0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.272+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/3132349105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c910a72920 0x55c910a91be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.272+0000 7fcdfb1abb80 1 --2- 172.21.15.70:0/3132349105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c910a732e0 0x55c910a8ff90 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.272+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/3132349105 >> 172.21.15.70:0/3132349105 conn(0x55c91099e2c0 msgr2=0x55c910a7d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:59.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.272+0000 7fcdd1ffb700 0 client.18093 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:06:59.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.273+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/3132349105 shutdown_connections 2024-06-04T05:06:59.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.273+0000 7fcdfb1abb80 1 -- 172.21.15.70:0/3132349105 wait complete. 2024-06-04T05:06:59.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.273+0000 7fcdfb1abb80 20 client.18093 trim_cache size 0 max 16384 2024-06-04T05:06:59.279 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:06:59.279 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:06:59.279 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "6e738ccd-bfb5-40b3-bc3c-e0697675e8c0", 2024-06-04T05:06:59.279 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:06:59.279 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:06:59.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.273+0000 7fcdf0916700 1 -- 172.21.15.70:0/3422270291 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcdd40b2a70 msgr2=0x7fcdd40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:59.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.273+0000 7fcdf0916700 1 --2- 172.21.15.70:0/3422270291 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcdd40b2a70 0x7fcdd40b4f30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcddc011550 tx=0x7fcddc009260 comp rx=0 tx=0).stop 2024-06-04T05:06:59.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.273+0000 7fcdf0916700 1 -- 172.21.15.70:0/3422270291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdec102e00 msgr2=0x7fcdec19c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:06:59.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.273+0000 7fcdf0916700 1 --2- 172.21.15.70:0/3422270291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdec102e00 0x7fcdec19c270 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcde0000c00 tx=0x7fcde000dd90 comp rx=0 tx=0).stop 2024-06-04T05:06:59.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.273+0000 7fcdf0916700 1 -- 172.21.15.70:0/3422270291 shutdown_connections 2024-06-04T05:06:59.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.273+0000 7fcdf0916700 1 --2- 172.21.15.70:0/3422270291 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcdd40b2a70 0x7fcdd40b4f30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.273+0000 7fcdf0916700 1 --2- 172.21.15.70:0/3422270291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdec108520 0x7fcdec1a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.273+0000 7fcdf0916700 1 --2- 172.21.15.70:0/3422270291 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec102440 0x7fcdec19bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.273+0000 7fcdf0916700 1 --2- 172.21.15.70:0/3422270291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdec102e00 0x7fcdec19c270 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:06:59.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.273+0000 7fcdf0916700 1 -- 172.21.15.70:0/3422270291 >> 172.21.15.70:0/3422270291 conn(0x7fcdec0793f0 msgr2=0x7fcdec100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:06:59.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.274+0000 7fcdf0916700 1 -- 172.21.15.70:0/3422270291 shutdown_connections 2024-06-04T05:06:59.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:06:59.274+0000 7fcdf0916700 1 -- 172.21.15.70:0/3422270291 wait complete. 2024-06-04T05:06:59.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:06:59 smithi070 ceph-mon[32631]: pgmap v301: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-06-04T05:06:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:06:59 smithi112 ceph-mon[36453]: pgmap v301: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-06-04T05:06:59.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 170046 1.91 MB/sec execute 352 sec latency 27.292 ms 2024-06-04T05:06:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:06:59 smithi098 ceph-mon[35569]: pgmap v301: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-06-04T05:06:59.989 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:07:00.247 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:07:00.533 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:00 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:07:00.533 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:00 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:00 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:07:00.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:00 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:07:00.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:00 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:00 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:07:00.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:00 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:07:00.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:00 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:00 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:00 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:07:00.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:00 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:07:00.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:00 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:00 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:07:00.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:00 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:00 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:00 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:00 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:07:00.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:00 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:00 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:07:00.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:00 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:07:00.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:00 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:00 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:07:00.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:00 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:07:00.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:00 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:00 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:00 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:07:00.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:00 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:07:00.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:00 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:00 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:07:00.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:00 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:00 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:00 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 170513 1.92 MB/sec execute 353 sec latency 15.768 ms 2024-06-04T05:07:00.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:00 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:07:00.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:00 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:00 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:07:00.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:00 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:07:00.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:00 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:00 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:07:00.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:00 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:07:00.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:00 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:00 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:00 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:07:00.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:00 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:07:00.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:00 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:00 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:07:00.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:00 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:00 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:00.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:00 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:01.587 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:01 smithi112 ceph-mon[36453]: pgmap v302: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-06-04T05:07:01.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:01 smithi070 ceph-mon[32631]: pgmap v302: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-06-04T05:07:01.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 170776 1.91 MB/sec execute 354 sec latency 16.489 ms 2024-06-04T05:07:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:01 smithi098 ceph-mon[35569]: pgmap v302: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-06-04T05:07:02.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.021+0000 7fc7e71c9640 1 -- 172.21.15.70:0/2410127613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7e0102340 msgr2=0x7fc7e010e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.021+0000 7fc7e71c9640 1 --2- 172.21.15.70:0/2410127613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7e0102340 0x7fc7e010e800 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8009c10 tx=0x7fc7d802a6e0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.021+0000 7fc7e71c9640 1 -- 172.21.15.70:0/2410127613 shutdown_connections 2024-06-04T05:07:02.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.021+0000 7fc7e71c9640 1 --2- 172.21.15.70:0/2410127613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7e0101980 0x7fc7e0101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.021+0000 7fc7e71c9640 1 --2- 172.21.15.70:0/2410127613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7e0102340 0x7fc7e010e800 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.021+0000 7fc7e71c9640 1 --2- 172.21.15.70:0/2410127613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7e0107940 0x7fc7e0107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.021+0000 7fc7e71c9640 1 -- 172.21.15.70:0/2410127613 >> 172.21.15.70:0/2410127613 conn(0x7fc7e0078ad0 msgr2=0x7fc7e0078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:02.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.021+0000 7fc7e71c9640 1 -- 172.21.15.70:0/2410127613 shutdown_connections 2024-06-04T05:07:02.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.021+0000 7fc7e71c9640 1 -- 172.21.15.70:0/2410127613 wait complete. 2024-06-04T05:07:02.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.021+0000 7fc7e71c9640 1 Processor -- start 2024-06-04T05:07:02.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.021+0000 7fc7e71c9640 1 -- start start 2024-06-04T05:07:02.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7e71c9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7e0101980 0x7fc7e0198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7e71c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7e0102340 0x7fc7e0198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7e71c9640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7e0107940 0x7fc7e019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7e71c9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e0111a80 con 0x7fc7e0101980 2024-06-04T05:07:02.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7e71c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e0111900 con 0x7fc7e0102340 2024-06-04T05:07:02.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7e71c9640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e0111c00 con 0x7fc7e0107940 2024-06-04T05:07:02.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7e4f65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7e0101980 0x7fc7e0198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7e4f65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7e0101980 0x7fc7e0198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59748/0 (socket says 172.21.15.70:59748) 2024-06-04T05:07:02.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7e4f65640 1 -- 172.21.15.70:0/986607763 learned_addr learned my addr 172.21.15.70:0/986607763 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:02.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7dffff640 1 --2- 172.21.15.70:0/986607763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7e0102340 0x7fc7e0198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7e5766640 1 --2- 172.21.15.70:0/986607763 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7e0107940 0x7fc7e019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7e4f65640 1 -- 172.21.15.70:0/986607763 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7e0107940 msgr2=0x7fc7e019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7e4f65640 1 --2- 172.21.15.70:0/986607763 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7e0107940 0x7fc7e019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7e4f65640 1 -- 172.21.15.70:0/986607763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7e0102340 msgr2=0x7fc7e0198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7e4f65640 1 --2- 172.21.15.70:0/986607763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7e0102340 0x7fc7e0198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7e4f65640 1 -- 172.21.15.70:0/986607763 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7e019cfc0 con 0x7fc7e0101980 2024-06-04T05:07:02.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7e4f65640 1 --2- 172.21.15.70:0/986607763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7e0101980 0x7fc7e0198020 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fc7d400e9c0 tx=0x7fc7d400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:02.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7ddffb640 1 -- 172.21.15.70:0/986607763 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d400cde0 con 0x7fc7e0101980 2024-06-04T05:07:02.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7e71c9640 1 -- 172.21.15.70:0/986607763 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7e019d2b0 con 0x7fc7e0101980 2024-06-04T05:07:02.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7ddffb640 1 -- 172.21.15.70:0/986607763 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc7d40035f0 con 0x7fc7e0101980 2024-06-04T05:07:02.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.022+0000 7fc7ddffb640 1 -- 172.21.15.70:0/986607763 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4010690 con 0x7fc7e0101980 2024-06-04T05:07:02.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.023+0000 7fc7e71c9640 1 -- 172.21.15.70:0/986607763 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7e01a4b40 con 0x7fc7e0101980 2024-06-04T05:07:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.026+0000 7fc7ddffb640 1 -- 172.21.15.70:0/986607763 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fc7d4004000 con 0x7fc7e0101980 2024-06-04T05:07:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.026+0000 7fc7e9fb6740 1 Processor -- start 2024-06-04T05:07:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.026+0000 7fc7e9fb6740 1 -- start start 2024-06-04T05:07:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.026+0000 7fc7e9fb6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b40eb0 0x555df1b3f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.026+0000 7fc7e9fb6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b3fcd0 0x555df1b40150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.026+0000 7fc7e9fb6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b4eb50 0x555df1b50f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.026+0000 7fc7ddffb640 1 --2- 172.21.15.70:0/986607763 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc7d00b2a10 0x7fc7d00b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.026+0000 7fc7e9fb6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555df1ab1c30 con 0x555df1b3fcd0 2024-06-04T05:07:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.026+0000 7fc7e9fb6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555df1aa17d0 con 0x555df1b40eb0 2024-06-04T05:07:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.026+0000 7fc7e9fb6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555df1aa5810 con 0x555df1b4eb50 2024-06-04T05:07:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.027+0000 7fc7e5766640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b40eb0 0x555df1b3f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.027+0000 7fc7dffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b4eb50 0x555df1b50f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.027+0000 7fc7e4f65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b3fcd0 0x555df1b40150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.027+0000 7fc7e5766640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b40eb0 0x555df1b3f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:51004/0 (socket says 172.21.15.70:51004) 2024-06-04T05:07:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.027+0000 7fc7e5766640 1 -- 172.21.15.70:0/2717976085 learned_addr learned my addr 172.21.15.70:0/2717976085 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.027+0000 7fc7dffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b4eb50 0x555df1b50f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:54886/0 (socket says 172.21.15.70:54886) 2024-06-04T05:07:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.027+0000 7fc7e5766640 1 --2- 172.21.15.70:0/986607763 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc7d00b2a10 0x7fc7d00b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.027+0000 7fc7ddffb640 1 -- 172.21.15.70:0/986607763 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7fc7d40e2850 con 0x7fc7e0101980 2024-06-04T05:07:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.027+0000 7fc7dffff640 1 -- 172.21.15.70:0/2717976085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b40eb0 msgr2=0x555df1b3f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.027+0000 7fc7dffff640 1 --2- 172.21.15.70:0/2717976085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b40eb0 0x555df1b3f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.027+0000 7fc7dffff640 1 -- 172.21.15.70:0/2717976085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b3fcd0 msgr2=0x555df1b40150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:02.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.027+0000 7fc7dffff640 1 --2- 172.21.15.70:0/2717976085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b3fcd0 0x555df1b40150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.027+0000 7fc7dffff640 1 -- 172.21.15.70:0/2717976085 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555df17b6210 con 0x555df1b4eb50 2024-06-04T05:07:02.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.027+0000 7fc7dffff640 1 --2- 172.21.15.70:0/2717976085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b4eb50 0x555df1b50f40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009bf0 tx=0x7fc7cc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acbeb68c068d26da server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:02.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.027+0000 7fc7e5766640 1 --2- 172.21.15.70:0/986607763 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc7d00b2a10 0x7fc7d00b4ed0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8005f00 tx=0x7fc7d8005e30 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:02.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.027+0000 7fc7caffd640 1 -- 172.21.15.70:0/2717976085 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc009da0 con 0x555df1b4eb50 2024-06-04T05:07:02.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.028+0000 7fc7caffd640 1 -- 172.21.15.70:0/2717976085 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc7cc005020 con 0x555df1b4eb50 2024-06-04T05:07:02.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.028+0000 7fc7caffd640 1 -- 172.21.15.70:0/2717976085 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0049c0 con 0x555df1b4eb50 2024-06-04T05:07:02.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.028+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/2717976085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b4eb50 msgr2=0x555df1b50f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.028+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/2717976085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b4eb50 0x555df1b50f40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009bf0 tx=0x7fc7cc011810 comp rx=0 tx=0).stop 2024-06-04T05:07:02.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.028+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/2717976085 shutdown_connections 2024-06-04T05:07:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.028+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/2717976085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b4eb50 0x555df1b50f40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.028+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/2717976085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b3fcd0 0x555df1b40150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.028+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/2717976085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b40eb0 0x555df1b3f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.028+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/2717976085 >> 172.21.15.70:0/2717976085 conn(0x555df1b3bd30 msgr2=0x555df1b3c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.028+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/2717976085 shutdown_connections 2024-06-04T05:07:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.028+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/2717976085 wait complete. 2024-06-04T05:07:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.029+0000 7fc7e9fb6740 1 -- start start 2024-06-04T05:07:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.029+0000 7fc7e9fb6740 1 Processor -- start 2024-06-04T05:07:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.029+0000 7fc7e9fb6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b3fcd0 0x555df1b4ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.029+0000 7fc7e9fb6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b40eb0 0x555df1b4bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.029+0000 7fc7e9fb6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b4eb50 0x555df1b4c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.029+0000 7fc7e9fb6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555df1787790 con 0x555df1b40eb0 2024-06-04T05:07:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.029+0000 7fc7e9fb6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555df1a5d890 con 0x555df1b4eb50 2024-06-04T05:07:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.029+0000 7fc7e9fb6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555df1b519c0 con 0x555df1b3fcd0 2024-06-04T05:07:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.029+0000 7fc7e9fb6740 10 client.0 _ll_register_callbacks cb 0x555df17b5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.029+0000 7fc7dffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b40eb0 0x555df1b4bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.029+0000 7fc7e4f65640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b3fcd0 0x555df1b4ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.029+0000 7fc7dffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b40eb0 0x555df1b4bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59774/0 (socket says 172.21.15.70:59774) 2024-06-04T05:07:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.029+0000 7fc7dffff640 1 -- 172.21.15.70:0/4109444838 learned_addr learned my addr 172.21.15.70:0/4109444838 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.029+0000 7fc7e5766640 1 --2- 172.21.15.70:0/4109444838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b4eb50 0x555df1b4c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.030+0000 7fc7e4f65640 1 -- 172.21.15.70:0/4109444838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b4eb50 msgr2=0x555df1b4c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.030+0000 7fc7e4f65640 1 --2- 172.21.15.70:0/4109444838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b4eb50 0x555df1b4c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.030+0000 7fc7e4f65640 1 -- 172.21.15.70:0/4109444838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b40eb0 msgr2=0x555df1b4bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.030+0000 7fc7e4f65640 1 --2- 172.21.15.70:0/4109444838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b40eb0 0x555df1b4bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.030+0000 7fc7e4f65640 1 -- 172.21.15.70:0/4109444838 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555df1882620 con 0x555df1b3fcd0 2024-06-04T05:07:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.030+0000 7fc7e4f65640 1 --2- 172.21.15.70:0/4109444838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b3fcd0 0x555df1b4ba30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40dff70 tx=0x7fc7d40733a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64a072c6311e2eb2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.030+0000 7fc7bffff640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:07:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.030+0000 7fc7bffff640 1 -- 172.21.15.70:0/4109444838 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d40aa200 con 0x555df1b3fcd0 2024-06-04T05:07:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.030+0000 7fc7bffff640 1 -- 172.21.15.70:0/4109444838 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7fc7d40037a0 con 0x555df1b3fcd0 2024-06-04T05:07:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.030+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/4109444838 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7cc005020 con 0x555df1b3fcd0 2024-06-04T05:07:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.030+0000 7fc7bffff640 1 -- 172.21.15.70:0/4109444838 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc7d40a9b00 con 0x555df1b3fcd0 2024-06-04T05:07:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.030+0000 7fc7e9fb6740 10 client.27325 fetch_fsmap learned FSMap version 32 2024-06-04T05:07:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.030+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/4109444838 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555df1b5d820 con 0x555df1b3fcd0 2024-06-04T05:07:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.030+0000 7fc7bffff640 1 -- 172.21.15.70:0/4109444838 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc7cc005020 con 0x555df1b3fcd0 2024-06-04T05:07:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.031+0000 7fc7bffff640 1 -- 172.21.15.70:0/4109444838 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fc7d40a5d90 con 0x555df1b3fcd0 2024-06-04T05:07:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.031+0000 7fc7e9fb6740 10 client.27325 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:07:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.031+0000 7fc7e9fb6740 10 client.27325 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.031+0000 7fc7e9fb6740 20 client.27325 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.031+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/4109444838 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555df1b5ed30 0x555df1b61140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.031+0000 7fc7e9fb6740 4 client.27325 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:07:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.031+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/4109444838 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555df1ab3b80 con 0x555df1b5ed30 2024-06-04T05:07:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.031+0000 7fc7dffff640 1 --2- 172.21.15.70:0/4109444838 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555df1b5ed30 0x555df1b61140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.031+0000 7fc7bffff640 10 client.27325 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.031+0000 7fc7dffff640 1 --2- 172.21.15.70:0/4109444838 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555df1b5ed30 0x555df1b61140 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89a8389e5f9bd210 server_cookie=79579f13e9bcfc7e in_seq=0 out_seq=0 2024-06-04T05:07:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.032+0000 7fc7bffff640 1 -- 172.21.15.70:0/4109444838 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555df1ab3b80 con 0x555df1b5ed30 2024-06-04T05:07:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.032+0000 7fc7bffff640 10 client.27325 handle_command_reply: tid=0 2024-06-04T05:07:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.032+0000 7fc7e9fb6740 1 client.27325 shutdown 2024-06-04T05:07:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.032+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/4109444838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b3fcd0 msgr2=0x555df1b4ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.032+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/4109444838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b3fcd0 0x555df1b4ba30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40dff70 tx=0x7fc7d40733a0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.032+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/4109444838 shutdown_connections 2024-06-04T05:07:02.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.032+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/4109444838 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555df1b5ed30 0x555df1b61140 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.032+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/4109444838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b4eb50 0x555df1b4c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.032+0000 7fc7bffff640 0 client.27325 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:02.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.032+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/4109444838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b40eb0 0x555df1b4bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.032+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/4109444838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b3fcd0 0x555df1b4ba30 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.032+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/4109444838 >> 172.21.15.70:0/4109444838 conn(0x555df1b3bd30 msgr2=0x555df1b41ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:02.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.032+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/4109444838 shutdown_connections 2024-06-04T05:07:02.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.032+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/4109444838 wait complete. 2024-06-04T05:07:02.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.032+0000 7fc7e9fb6740 20 client.27325 trim_cache size 0 max 16384 2024-06-04T05:07:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.048+0000 7fc7e9fb6740 1 Processor -- start 2024-06-04T05:07:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.048+0000 7fc7e9fb6740 1 -- start start 2024-06-04T05:07:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7e9fb6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b3fcd0 0x555df1b50f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7e9fb6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b40eb0 0x555df1b3b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7e9fb6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b4eb50 0x555df1b49190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7e9fb6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01ac420 con 0x555df1b40eb0 2024-06-04T05:07:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7e9fb6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01adf60 con 0x555df1b3fcd0 2024-06-04T05:07:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7e9fb6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01acaf0 con 0x555df1b4eb50 2024-06-04T05:07:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7dffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b40eb0 0x555df1b3b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7dffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b40eb0 0x555df1b3b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59778/0 (socket says 172.21.15.70:59778) 2024-06-04T05:07:02.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7dffff640 1 -- 172.21.15.70:0/87311479 learned_addr learned my addr 172.21.15.70:0/87311479 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:02.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7e5766640 1 --2- 172.21.15.70:0/87311479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b4eb50 0x555df1b49190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7e4f65640 1 --2- 172.21.15.70:0/87311479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b3fcd0 0x555df1b50f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7dffff640 1 -- 172.21.15.70:0/87311479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b4eb50 msgr2=0x555df1b49190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:02.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7dffff640 1 --2- 172.21.15.70:0/87311479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b4eb50 0x555df1b49190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7dffff640 1 -- 172.21.15.70:0/87311479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b3fcd0 msgr2=0x555df1b50f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:02.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7dffff640 1 --2- 172.21.15.70:0/87311479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b3fcd0 0x555df1b50f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7dffff640 1 -- 172.21.15.70:0/87311479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555df1b617c0 con 0x555df1b40eb0 2024-06-04T05:07:02.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7dffff640 1 --2- 172.21.15.70:0/87311479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b40eb0 0x555df1b3b730 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x555df1b444c0 tx=0x7fc7cc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc191ed07a0b0c06 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:02.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7ca7fc640 1 -- 172.21.15.70:0/87311479 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc02e070 con 0x555df1b40eb0 2024-06-04T05:07:02.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7ca7fc640 1 -- 172.21.15.70:0/87311479 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc7cc0054e0 con 0x555df1b40eb0 2024-06-04T05:07:02.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.049+0000 7fc7ca7fc640 1 -- 172.21.15.70:0/87311479 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc029360 con 0x555df1b40eb0 2024-06-04T05:07:02.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.050+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/87311479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b40eb0 msgr2=0x555df1b3b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.050+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/87311479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b40eb0 0x555df1b3b730 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x555df1b444c0 tx=0x7fc7cc011810 comp rx=0 tx=0).stop 2024-06-04T05:07:02.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.050+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/87311479 shutdown_connections 2024-06-04T05:07:02.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.050+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/87311479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b4eb50 0x555df1b49190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.050+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/87311479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b40eb0 0x555df1b3b730 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.050+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/87311479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b3fcd0 0x555df1b50f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.050+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/87311479 >> 172.21.15.70:0/87311479 conn(0x555df1b3bd30 msgr2=0x555df1b48a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:02.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.050+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/87311479 shutdown_connections 2024-06-04T05:07:02.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.050+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/87311479 wait complete. 2024-06-04T05:07:02.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.050+0000 7fc7e9fb6740 1 -- start start 2024-06-04T05:07:02.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e9fb6740 1 Processor -- start 2024-06-04T05:07:02.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e9fb6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b3fcd0 0x555df1b66520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e9fb6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b40eb0 0x555df1b66a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e9fb6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b4eb50 0x555df1b66fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e9fb6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01ad890 con 0x555df1b4eb50 2024-06-04T05:07:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e9fb6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01ad1c0 con 0x555df1b40eb0 2024-06-04T05:07:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e9fb6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01aa420 con 0x555df1b3fcd0 2024-06-04T05:07:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e9fb6740 10 client.0 _ll_register_callbacks cb 0x7fc7d40e2b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e5766640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b4eb50 0x555df1b66fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e4f65640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b3fcd0 0x555df1b66520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e5766640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b4eb50 0x555df1b66fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59788/0 (socket says 172.21.15.70:59788) 2024-06-04T05:07:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e4f65640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b3fcd0 0x555df1b66520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:54914/0 (socket says 172.21.15.70:54914) 2024-06-04T05:07:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e5766640 1 -- 172.21.15.70:0/3296873894 learned_addr learned my addr 172.21.15.70:0/3296873894 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7dffff640 1 --2- 172.21.15.70:0/3296873894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b40eb0 0x555df1b66a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e5766640 1 -- 172.21.15.70:0/3296873894 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b3fcd0 msgr2=0x555df1b66520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e5766640 1 --2- 172.21.15.70:0/3296873894 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b3fcd0 0x555df1b66520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e5766640 1 -- 172.21.15.70:0/3296873894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b40eb0 msgr2=0x555df1b66a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:02.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e5766640 1 --2- 172.21.15.70:0/3296873894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b40eb0 0x555df1b66a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e5766640 1 -- 172.21.15.70:0/3296873894 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555df1b55130 con 0x555df1b4eb50 2024-06-04T05:07:02.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.051+0000 7fc7e4f65640 1 --2- 172.21.15.70:0/3296873894 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b3fcd0 0x555df1b66520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:02.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.052+0000 7fc7e5766640 1 --2- 172.21.15.70:0/3296873894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b4eb50 0x555df1b66fa0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8009be0 tx=0x7fc7d8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a33f82c2b3dcce server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:02.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.052+0000 7fc7bffff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:02.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.052+0000 7fc7bffff640 1 -- 172.21.15.70:0/3296873894 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8036050 con 0x555df1b4eb50 2024-06-04T05:07:02.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.052+0000 7fc7bffff640 1 -- 172.21.15.70:0/3296873894 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7fc7d8031440 con 0x555df1b4eb50 2024-06-04T05:07:02.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.052+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/3296873894 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7cc0054e0 con 0x555df1b4eb50 2024-06-04T05:07:02.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.052+0000 7fc7bffff640 1 -- 172.21.15.70:0/3296873894 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc7d8003e20 con 0x555df1b4eb50 2024-06-04T05:07:02.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.052+0000 7fc7e9fb6740 10 client.18111 fetch_fsmap learned FSMap version 32 2024-06-04T05:07:02.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.052+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/3296873894 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555df1bc9be0 con 0x555df1b4eb50 2024-06-04T05:07:02.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.052+0000 7fc7bffff640 1 -- 172.21.15.70:0/3296873894 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc7cc0054e0 con 0x555df1b4eb50 2024-06-04T05:07:02.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.052+0000 7fc7bffff640 1 -- 172.21.15.70:0/3296873894 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fc7d8055070 con 0x555df1b4eb50 2024-06-04T05:07:02.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.052+0000 7fc7e9fb6740 10 client.18111 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:07:02.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.052+0000 7fc7e9fb6740 10 client.18111 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.052+0000 7fc7e9fb6740 20 client.18111 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.052+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/3296873894 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555df1bca330 0x555df1bcc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.052+0000 7fc7e9fb6740 4 client.18111 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:07:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.052+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/3296873894 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555df1aad360 con 0x555df1bca330 2024-06-04T05:07:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.052+0000 7fc7e4f65640 1 --2- 172.21.15.70:0/3296873894 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555df1bca330 0x555df1bcc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.053+0000 7fc7e4f65640 1 --2- 172.21.15.70:0/3296873894 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555df1bca330 0x555df1bcc750 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7bd62816629772f server_cookie=3eee4e832310c05d in_seq=0 out_seq=0 2024-06-04T05:07:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.053+0000 7fc7bffff640 10 client.18111 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:02.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.072+0000 7f9b412d4700 1 -- 172.21.15.70:0/313510019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b3c06c470 msgr2=0x7f9b3c1103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.072+0000 7f9b412d4700 1 --2- 172.21.15.70:0/313510019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b3c06c470 0x7f9b3c1103c0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400b0a0 tx=0x7f9b3400b3b0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.072+0000 7f9b412d4700 1 -- 172.21.15.70:0/313510019 shutdown_connections 2024-06-04T05:07:02.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.072+0000 7f9b412d4700 1 --2- 172.21.15.70:0/313510019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b3c06c470 0x7f9b3c1103c0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.072+0000 7f9b412d4700 1 --2- 172.21.15.70:0/313510019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b3c1095a0 0x7f9b3c109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.072+0000 7f9b412d4700 1 --2- 172.21.15.70:0/313510019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b3c06bab0 0x7f9b3c06bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.072+0000 7f9b412d4700 1 -- 172.21.15.70:0/313510019 >> 172.21.15.70:0/313510019 conn(0x7f9b3c0793b0 msgr2=0x7f9b3c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:02.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.072+0000 7f9b412d4700 1 -- 172.21.15.70:0/313510019 shutdown_connections 2024-06-04T05:07:02.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.072+0000 7f9b412d4700 1 -- 172.21.15.70:0/313510019 wait complete. 2024-06-04T05:07:02.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.072+0000 7f9b412d4700 1 Processor -- start 2024-06-04T05:07:02.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.073+0000 7f9b412d4700 1 -- start start 2024-06-04T05:07:02.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.073+0000 7f9b412d4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b3c06bab0 0x7f9b3c19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.073+0000 7f9b412d4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b3c06c470 0x7f9b3c19fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.073+0000 7f9b412d4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b3c1095a0 0x7f9b3c1998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.073+0000 7f9b412d4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c113580 con 0x7f9b3c1095a0 2024-06-04T05:07:02.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.073+0000 7f9b412d4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c113400 con 0x7f9b3c06bab0 2024-06-04T05:07:02.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.073+0000 7f9b412d4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c113700 con 0x7f9b3c06c470 2024-06-04T05:07:02.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.073+0000 7f9b3affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b3c06bab0 0x7f9b3c19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.073+0000 7f9b3a7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b3c06c470 0x7f9b3c19fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.073+0000 7f9b3affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b3c06bab0 0x7f9b3c19f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:51046/0 (socket says 172.21.15.70:51046) 2024-06-04T05:07:02.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.073+0000 7f9b3affd700 1 -- 172.21.15.70:0/300302787 learned_addr learned my addr 172.21.15.70:0/300302787 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:02.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.073+0000 7f9b3b7fe700 1 --2- 172.21.15.70:0/300302787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b3c1095a0 0x7f9b3c1998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.073+0000 7f9b3b7fe700 1 -- 172.21.15.70:0/300302787 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b3c06c470 msgr2=0x7f9b3c19fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.074+0000 7f9b3b7fe700 1 --2- 172.21.15.70:0/300302787 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b3c06c470 0x7f9b3c19fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.074+0000 7f9b3b7fe700 1 -- 172.21.15.70:0/300302787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b3c06bab0 msgr2=0x7f9b3c19f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.074+0000 7f9b3b7fe700 1 --2- 172.21.15.70:0/300302787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b3c06bab0 0x7f9b3c19f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.074+0000 7f9b3b7fe700 1 -- 172.21.15.70:0/300302787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b3c19a020 con 0x7f9b3c1095a0 2024-06-04T05:07:02.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.074+0000 7f9b3b7fe700 1 --2- 172.21.15.70:0/300302787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b3c1095a0 0x7f9b3c1998c0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400b0a0 tx=0x7f9b34004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:02.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.074+0000 7f9b23fff700 1 -- 172.21.15.70:0/300302787 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b34021070 con 0x7f9b3c1095a0 2024-06-04T05:07:02.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.074+0000 7f9b23fff700 1 -- 172.21.15.70:0/300302787 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b34007b80 con 0x7f9b3c1095a0 2024-06-04T05:07:02.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.074+0000 7f9b23fff700 1 -- 172.21.15.70:0/300302787 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b34005040 con 0x7f9b3c1095a0 2024-06-04T05:07:02.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.074+0000 7f9b412d4700 1 -- 172.21.15.70:0/300302787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b3c19a2b0 con 0x7f9b3c1095a0 2024-06-04T05:07:02.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.074+0000 7f9b412d4700 1 -- 172.21.15.70:0/300302787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b3c1a65d0 con 0x7f9b3c1095a0 2024-06-04T05:07:02.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.074+0000 7fc7bffff640 1 -- 172.21.15.70:0/3296873894 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x555df1aad360 con 0x555df1bca330 2024-06-04T05:07:02.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.074+0000 7fc7bffff640 10 client.18111 handle_command_reply: tid=0 2024-06-04T05:07:02.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.074+0000 7fc7e9fb6740 1 client.18111 shutdown 2024-06-04T05:07:02.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.075+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/3296873894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b4eb50 msgr2=0x555df1b66fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.075+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/3296873894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b4eb50 0x555df1b66fa0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8009be0 tx=0x7fc7d8005fd0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.075+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/3296873894 shutdown_connections 2024-06-04T05:07:02.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.075+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/3296873894 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555df1bca330 0x555df1bcc750 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.075+0000 7fc7bffff640 0 client.18111 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:02.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.075+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/3296873894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555df1b40eb0 0x555df1b66a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.075+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/3296873894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555df1b4eb50 0x555df1b66fa0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.075+0000 7fc7e9fb6740 1 --2- 172.21.15.70:0/3296873894 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555df1b3fcd0 0x555df1b66520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.075+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/3296873894 >> 172.21.15.70:0/3296873894 conn(0x555df1b3bd30 msgr2=0x555df1b5a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:02.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.076+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/3296873894 shutdown_connections 2024-06-04T05:07:02.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.076+0000 7fc7e9fb6740 1 -- 172.21.15.70:0/3296873894 wait complete. 2024-06-04T05:07:02.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:07:02.081 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:07:02.081 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:07:02.081 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.076+0000 7fc7e9fb6740 20 client.18111 trim_cache size 0 max 16384 2024-06-04T05:07:02.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.077+0000 7fc7e71c9640 1 -- 172.21.15.70:0/986607763 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc7d00b2a10 msgr2=0x7fc7d00b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.077+0000 7fc7e71c9640 1 --2- 172.21.15.70:0/986607763 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc7d00b2a10 0x7fc7d00b4ed0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8005f00 tx=0x7fc7d8005e30 comp rx=0 tx=0).stop 2024-06-04T05:07:02.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.077+0000 7fc7e71c9640 1 -- 172.21.15.70:0/986607763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7e0101980 msgr2=0x7fc7e0198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.077+0000 7fc7e71c9640 1 --2- 172.21.15.70:0/986607763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7e0101980 0x7fc7e0198020 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fc7d400e9c0 tx=0x7fc7d400ee80 comp rx=0 tx=0).stop 2024-06-04T05:07:02.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.077+0000 7fc7e71c9640 1 -- 172.21.15.70:0/986607763 shutdown_connections 2024-06-04T05:07:02.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.077+0000 7fc7e71c9640 1 --2- 172.21.15.70:0/986607763 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc7d00b2a10 0x7fc7d00b4ed0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.077+0000 7fc7e71c9640 1 --2- 172.21.15.70:0/986607763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7e0102340 0x7fc7e0198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.077+0000 7fc7e71c9640 1 --2- 172.21.15.70:0/986607763 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7e0107940 0x7fc7e019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.077+0000 7fc7e71c9640 1 --2- 172.21.15.70:0/986607763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7e0101980 0x7fc7e0198020 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.077+0000 7fc7e71c9640 1 -- 172.21.15.70:0/986607763 >> 172.21.15.70:0/986607763 conn(0x7fc7e0078ad0 msgr2=0x7fc7e0104540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:02.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.078+0000 7fc7e71c9640 1 -- 172.21.15.70:0/986607763 shutdown_connections 2024-06-04T05:07:02.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:02.078+0000 7fc7e71c9640 1 -- 172.21.15.70:0/986607763 wait complete. 2024-06-04T05:07:02.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.079+0000 7f9b4bb69b80 1 Processor -- start 2024-06-04T05:07:02.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.079+0000 7f9b4bb69b80 1 -- start start 2024-06-04T05:07:02.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.079+0000 7f9b23fff700 1 -- 172.21.15.70:0/300302787 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f9b34024070 con 0x7f9b3c1095a0 2024-06-04T05:07:02.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.080+0000 7f9b4bb69b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd7546860 0x55edd7546c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.080+0000 7f9b4bb69b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7547210 0x55edd7549a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.080+0000 7f9b4bb69b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd754a090 0x55edd754a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.080+0000 7f9b4bb69b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55edd74a5c90 con 0x55edd7547210 2024-06-04T05:07:02.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.080+0000 7f9b4bb69b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55edd7495830 con 0x55edd754a090 2024-06-04T05:07:02.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.080+0000 7f9b4bb69b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55edd7499870 con 0x55edd7546860 2024-06-04T05:07:02.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.080+0000 7f9b3affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7547210 0x55edd7549a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.080+0000 7f9b3affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7547210 0x55edd7549a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59810/0 (socket says 172.21.15.70:59810) 2024-06-04T05:07:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.080+0000 7f9b3affd700 1 -- 172.21.15.70:0/2696221099 learned_addr learned my addr 172.21.15.70:0/2696221099 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.080+0000 7f9b3a7fc700 1 --2- 172.21.15.70:0/2696221099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd754a090 0x55edd754a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.080+0000 7f9b3a7fc700 1 --2- 172.21.15.70:0/2696221099 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd7546860 0x55edd7546c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.081+0000 7f9b3affd700 1 -- 172.21.15.70:0/2696221099 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd7546860 msgr2=0x55edd7546c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.081+0000 7f9b3affd700 1 --2- 172.21.15.70:0/2696221099 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd7546860 0x55edd7546c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.081+0000 7f9b3affd700 1 -- 172.21.15.70:0/2696221099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd754a090 msgr2=0x55edd754a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.081+0000 7f9b3affd700 1 --2- 172.21.15.70:0/2696221099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd754a090 0x55edd754a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.081+0000 7f9b3affd700 1 -- 172.21.15.70:0/2696221099 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edd73e58d0 con 0x55edd7547210 2024-06-04T05:07:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.081+0000 7f9b23fff700 1 --2- 172.21.15.70:0/300302787 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9b240b2a70 0x7f9b240b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.081+0000 7f9b3affd700 1 --2- 172.21.15.70:0/2696221099 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7547210 0x55edd7549a70 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009d90 tx=0x7f9b2c011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5d42e52030d1796 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.081+0000 7f9b23fff700 1 -- 172.21.15.70:0/300302787 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f9b3400f020 con 0x7f9b3c1095a0 2024-06-04T05:07:02.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.082+0000 7f9b3a7fc700 1 --2- 172.21.15.70:0/300302787 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9b240b2a70 0x7f9b240b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.082+0000 7f9b217fa700 1 -- 172.21.15.70:0/2696221099 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c01e070 con 0x55edd7547210 2024-06-04T05:07:02.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.082+0000 7f9b217fa700 1 -- 172.21.15.70:0/2696221099 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b2c019ac0 con 0x55edd7547210 2024-06-04T05:07:02.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.082+0000 7f9b217fa700 1 -- 172.21.15.70:0/2696221099 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c00f960 con 0x55edd7547210 2024-06-04T05:07:02.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.083+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/2696221099 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7547210 msgr2=0x55edd7549a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.083+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/2696221099 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7547210 0x55edd7549a70 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009d90 tx=0x7f9b2c011720 comp rx=0 tx=0).stop 2024-06-04T05:07:02.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.083+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/2696221099 shutdown_connections 2024-06-04T05:07:02.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.083+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/2696221099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd754a090 0x55edd754a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.083+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/2696221099 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd7546860 0x55edd7546c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.083+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/2696221099 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7547210 0x55edd7549a70 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.083+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/2696221099 >> 172.21.15.70:0/2696221099 conn(0x55edd7474080 msgr2=0x55edd7540e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:02.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.084+0000 7f9b3a7fc700 1 --2- 172.21.15.70:0/300302787 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9b240b2a70 0x7f9b240b4f30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9b300097c0 tx=0x7f9b30009750 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:02.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.084+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/2696221099 shutdown_connections 2024-06-04T05:07:02.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.084+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/2696221099 wait complete. 2024-06-04T05:07:02.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.084+0000 7f9b4bb69b80 1 -- start start 2024-06-04T05:07:02.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.084+0000 7f9b4bb69b80 1 Processor -- start 2024-06-04T05:07:02.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.085+0000 7f9b4bb69b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7546860 0x55edd755acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.085+0000 7f9b4bb69b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd7547210 0x55edd7558010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.085+0000 7f9b4bb69b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd754a090 0x55edd7558550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.085+0000 7f9b4bb69b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55edd739bfa0 con 0x55edd7546860 2024-06-04T05:07:02.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.085+0000 7f9b4bb69b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55edd74377c0 con 0x55edd7547210 2024-06-04T05:07:02.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.085+0000 7f9b4bb69b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55edd755b9c0 con 0x55edd754a090 2024-06-04T05:07:02.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.085+0000 7f9b4bb69b80 10 client.0 _ll_register_callbacks cb 0x55edd7430920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:02.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.085+0000 7f9b3b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd754a090 0x55edd7558550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.085+0000 7f9b3b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd754a090 0x55edd7558550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:54966/0 (socket says 172.21.15.70:54966) 2024-06-04T05:07:02.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.085+0000 7f9b3b7fe700 1 -- 172.21.15.70:0/3667324262 learned_addr learned my addr 172.21.15.70:0/3667324262 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:02.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.085+0000 7f9b3affd700 1 --2- 172.21.15.70:0/3667324262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7546860 0x55edd755acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.085+0000 7f9b3b7fe700 1 -- 172.21.15.70:0/3667324262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd7547210 msgr2=0x55edd7558010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:07:02.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.085+0000 7f9b3b7fe700 1 --2- 172.21.15.70:0/3667324262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd7547210 0x55edd7558010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.085+0000 7f9b3b7fe700 1 -- 172.21.15.70:0/3667324262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7546860 msgr2=0x55edd755acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:02.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.086+0000 7f9b3b7fe700 1 --2- 172.21.15.70:0/3667324262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7546860 0x55edd755acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.086+0000 7f9b3b7fe700 1 -- 172.21.15.70:0/3667324262 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edd7434690 con 0x55edd754a090 2024-06-04T05:07:02.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.086+0000 7f9b3affd700 1 --2- 172.21.15.70:0/3667324262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7546860 0x55edd755acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:02.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.086+0000 7f9b3b7fe700 1 --2- 172.21.15.70:0/3667324262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd754a090 0x55edd7558550 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9b340e1080 tx=0x7f9b340aa560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62211c5acf162e8f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:02.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.086+0000 7f9b1eff5700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:07:02.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.087+0000 7f9b1eff5700 1 -- 172.21.15.70:0/3667324262 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b340ab400 con 0x55edd754a090 2024-06-04T05:07:02.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.087+0000 7f9b1eff5700 1 -- 172.21.15.70:0/3667324262 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f9b340ab170 con 0x55edd754a090 2024-06-04T05:07:02.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.087+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/3667324262 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b2c019ac0 con 0x55edd754a090 2024-06-04T05:07:02.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.087+0000 7f9b1eff5700 1 -- 172.21.15.70:0/3667324262 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9b340aad00 con 0x55edd754a090 2024-06-04T05:07:02.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.087+0000 7f9b4bb69b80 10 client.27343 fetch_fsmap learned FSMap version 32 2024-06-04T05:07:02.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.087+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/3667324262 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55edd75661c0 con 0x55edd754a090 2024-06-04T05:07:02.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.088+0000 7f9b1eff5700 1 -- 172.21.15.70:0/3667324262 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b2c019ac0 con 0x55edd754a090 2024-06-04T05:07:02.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.088+0000 7f9b1eff5700 1 -- 172.21.15.70:0/3667324262 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f9b34011070 con 0x55edd754a090 2024-06-04T05:07:02.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.088+0000 7f9b4bb69b80 10 client.27343 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:07:02.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.088+0000 7f9b4bb69b80 10 client.27343 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:02.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.088+0000 7f9b4bb69b80 20 client.27343 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:02.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.088+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/3667324262 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55edd75687d0 0x55edd756ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.088+0000 7f9b4bb69b80 4 client.27343 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:07:02.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.088+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/3667324262 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55edd75129e0 con 0x55edd75687d0 2024-06-04T05:07:02.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.088+0000 7f9b3affd700 1 --2- 172.21.15.70:0/3667324262 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55edd75687d0 0x55edd756ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.089+0000 7f9b1eff5700 10 client.27343 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:02.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.089+0000 7f9b3affd700 1 --2- 172.21.15.70:0/3667324262 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55edd75687d0 0x55edd756ac30 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3725cfb64a65875 server_cookie=f5061cdbf48cb910 in_seq=0 out_seq=0 2024-06-04T05:07:02.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.089+0000 7f9b1eff5700 1 -- 172.21.15.70:0/3667324262 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55edd75129e0 con 0x55edd75687d0 2024-06-04T05:07:02.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.089+0000 7f9b1eff5700 10 client.27343 handle_command_reply: tid=0 2024-06-04T05:07:02.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.090+0000 7f9b4bb69b80 1 client.27343 shutdown 2024-06-04T05:07:02.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.090+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/3667324262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd754a090 msgr2=0x55edd7558550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.090+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/3667324262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd754a090 0x55edd7558550 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9b340e1080 tx=0x7f9b340aa560 comp rx=0 tx=0).stop 2024-06-04T05:07:02.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.090+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/3667324262 shutdown_connections 2024-06-04T05:07:02.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.090+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/3667324262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd754a090 0x55edd7558550 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.090+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/3667324262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7546860 0x55edd755acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.090+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/3667324262 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55edd75687d0 0x55edd756ac30 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.090+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/3667324262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd7547210 0x55edd7558010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.090+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/3667324262 >> 172.21.15.70:0/3667324262 conn(0x55edd7474080 msgr2=0x55edd7540510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:02.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.090+0000 7f9b1eff5700 0 client.27343 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:02.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.090+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/3667324262 shutdown_connections 2024-06-04T05:07:02.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.090+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/3667324262 wait complete. 2024-06-04T05:07:02.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.090+0000 7f9b4bb69b80 20 client.27343 trim_cache size 0 max 16384 2024-06-04T05:07:02.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.113+0000 7f9b4bb69b80 1 Processor -- start 2024-06-04T05:07:02.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.113+0000 7f9b4bb69b80 1 -- start start 2024-06-04T05:07:02.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.113+0000 7f9b4bb69b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd7546860 0x55edd7555620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.113+0000 7f9b4bb69b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7547210 0x55edd7557190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.113+0000 7f9b4bb69b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd754a090 0x55edd7559f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.113+0000 7f9b4bb69b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c1abbe0 con 0x55edd7547210 2024-06-04T05:07:02.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.113+0000 7f9b4bb69b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c1ad720 con 0x55edd754a090 2024-06-04T05:07:02.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.113+0000 7f9b4bb69b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c1ac2b0 con 0x55edd7546860 2024-06-04T05:07:02.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.113+0000 7f9b3a7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7547210 0x55edd7557190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.113+0000 7f9b3affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd7546860 0x55edd7555620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.113+0000 7f9b3a7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7547210 0x55edd7557190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59826/0 (socket says 172.21.15.70:59826) 2024-06-04T05:07:02.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.113+0000 7f9b3a7fc700 1 -- 172.21.15.70:0/3349540115 learned_addr learned my addr 172.21.15.70:0/3349540115 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:02.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.113+0000 7f9b3b7fe700 1 --2- 172.21.15.70:0/3349540115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd754a090 0x55edd7559f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.113+0000 7f9b3affd700 1 -- 172.21.15.70:0/3349540115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd754a090 msgr2=0x55edd7559f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:02.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.113+0000 7f9b3affd700 1 --2- 172.21.15.70:0/3349540115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd754a090 0x55edd7559f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.113+0000 7f9b3affd700 1 -- 172.21.15.70:0/3349540115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7547210 msgr2=0x55edd7557190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:02.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.114+0000 7f9b3affd700 1 --2- 172.21.15.70:0/3349540115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7547210 0x55edd7557190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.114+0000 7f9b3affd700 1 -- 172.21.15.70:0/3349540115 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edd7542770 con 0x55edd7546860 2024-06-04T05:07:02.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.114+0000 7f9b3affd700 1 --2- 172.21.15.70:0/3349540115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd7546860 0x55edd7555620 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c00f580 tx=0x7f9b2c0095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbd1db00edbe4342 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:02.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.114+0000 7f9b20ff9700 1 -- 172.21.15.70:0/3349540115 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c01e040 con 0x55edd7546860 2024-06-04T05:07:02.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.114+0000 7f9b20ff9700 1 -- 172.21.15.70:0/3349540115 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b2c0054c0 con 0x55edd7546860 2024-06-04T05:07:02.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.114+0000 7f9b20ff9700 1 -- 172.21.15.70:0/3349540115 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c019bb0 con 0x55edd7546860 2024-06-04T05:07:02.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.115+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/3349540115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd7546860 msgr2=0x55edd7555620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.115+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/3349540115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd7546860 0x55edd7555620 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c00f580 tx=0x7f9b2c0095c0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.115+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/3349540115 shutdown_connections 2024-06-04T05:07:02.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.115+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/3349540115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd754a090 0x55edd7559f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.115+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/3349540115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd7546860 0x55edd7555620 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.115+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/3349540115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd7547210 0x55edd7557190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.115+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/3349540115 >> 172.21.15.70:0/3349540115 conn(0x55edd7474080 msgr2=0x55edd75597f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.115+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/3349540115 shutdown_connections 2024-06-04T05:07:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.115+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/3349540115 wait complete. 2024-06-04T05:07:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.115+0000 7f9b4bb69b80 1 -- start start 2024-06-04T05:07:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.115+0000 7f9b4bb69b80 1 Processor -- start 2024-06-04T05:07:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.115+0000 7f9b4bb69b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd7546860 0x55edd75614d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.115+0000 7f9b4bb69b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd7547210 0x55edd756fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b4bb69b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd754a090 0x55edd7570630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b4bb69b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c1ad050 con 0x55edd754a090 2024-06-04T05:07:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b4bb69b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c1ac980 con 0x55edd7546860 2024-06-04T05:07:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b4bb69b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c1a9ba0 con 0x55edd7547210 2024-06-04T05:07:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b4bb69b80 10 client.0 _ll_register_callbacks cb 0x55edd7430920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b3b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd754a090 0x55edd7570630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b3a7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd7547210 0x55edd756fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b3b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd754a090 0x55edd7570630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59830/0 (socket says 172.21.15.70:59830) 2024-06-04T05:07:02.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b3b7fe700 1 -- 172.21.15.70:0/1144496441 learned_addr learned my addr 172.21.15.70:0/1144496441 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:02.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b3affd700 1 --2- 172.21.15.70:0/1144496441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd7546860 0x55edd75614d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b3b7fe700 1 -- 172.21.15.70:0/1144496441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd7547210 msgr2=0x55edd756fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:02.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b3b7fe700 1 --2- 172.21.15.70:0/1144496441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd7547210 0x55edd756fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b3b7fe700 1 -- 172.21.15.70:0/1144496441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd7546860 msgr2=0x55edd75614d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:02.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b3b7fe700 1 --2- 172.21.15.70:0/1144496441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd7546860 0x55edd75614d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b3b7fe700 1 -- 172.21.15.70:0/1144496441 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edd75d2410 con 0x55edd754a090 2024-06-04T05:07:02.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b3affd700 1 --2- 172.21.15.70:0/1144496441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd7546860 0x55edd75614d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:02.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b3b7fe700 1 --2- 172.21.15.70:0/1144496441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd754a090 0x55edd7570630 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f9b340b3a40 tx=0x7f9b340aab40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f6dd29f4886e27f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:02.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b1eff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:02.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b1eff5700 1 -- 172.21.15.70:0/1144496441 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b340ab400 con 0x55edd754a090 2024-06-04T05:07:02.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b1eff5700 1 -- 172.21.15.70:0/1144496441 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8571+0+0 (secure 0 0 0) 0x7f9b340a72f0 con 0x55edd754a090 2024-06-04T05:07:02.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/1144496441 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b2c0054c0 con 0x55edd754a090 2024-06-04T05:07:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b1eff5700 1 -- 172.21.15.70:0/1144496441 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9b340af0b0 con 0x55edd754a090 2024-06-04T05:07:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b4bb69b80 10 client.18129 fetch_fsmap learned FSMap version 32 2024-06-04T05:07:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.116+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/1144496441 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55edd75d35b0 con 0x55edd754a090 2024-06-04T05:07:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.117+0000 7f9b1eff5700 1 -- 172.21.15.70:0/1144496441 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b2c0054c0 con 0x55edd754a090 2024-06-04T05:07:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.117+0000 7f9b1eff5700 1 -- 172.21.15.70:0/1144496441 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f9b34011070 con 0x55edd754a090 2024-06-04T05:07:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.117+0000 7f9b4bb69b80 10 client.18129 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:07:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.117+0000 7f9b4bb69b80 10 client.18129 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.117+0000 7f9b4bb69b80 20 client.18129 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.117+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/1144496441 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55edd75d3b30 0x55edd75d5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.117+0000 7f9b4bb69b80 4 client.18129 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:07:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.117+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/1144496441 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55edd74a7be0 con 0x55edd75d3b30 2024-06-04T05:07:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.117+0000 7f9b3affd700 1 --2- 172.21.15.70:0/1144496441 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55edd75d3b30 0x55edd75d5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.117+0000 7f9b1eff5700 10 client.18129 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.117+0000 7f9b3affd700 1 --2- 172.21.15.70:0/1144496441 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55edd75d3b30 0x55edd75d5f50 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aabbe62381176d15 server_cookie=fbecac656ea59a63 in_seq=0 out_seq=0 2024-06-04T05:07:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.118+0000 7f9b1eff5700 1 -- 172.21.15.70:0/1144496441 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55edd74a7be0 con 0x55edd75d3b30 2024-06-04T05:07:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.118+0000 7f9b1eff5700 10 client.18129 handle_command_reply: tid=0 2024-06-04T05:07:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.118+0000 7f9b4bb69b80 1 client.18129 shutdown 2024-06-04T05:07:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.118+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/1144496441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd754a090 msgr2=0x55edd7570630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.118+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/1144496441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd754a090 0x55edd7570630 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f9b340b3a40 tx=0x7f9b340aab40 comp rx=0 tx=0).stop 2024-06-04T05:07:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.118+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/1144496441 shutdown_connections 2024-06-04T05:07:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.118+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/1144496441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edd754a090 0x55edd7570630 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.118+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/1144496441 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55edd75d3b30 0x55edd75d5f50 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.118+0000 7f9b1eff5700 0 client.18129 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.118+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/1144496441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd7546860 0x55edd75614d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.118+0000 7f9b4bb69b80 1 --2- 172.21.15.70:0/1144496441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55edd7547210 0x55edd756fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.118+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/1144496441 >> 172.21.15.70:0/1144496441 conn(0x55edd7474080 msgr2=0x55edd75454f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.118+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/1144496441 shutdown_connections 2024-06-04T05:07:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.118+0000 7f9b4bb69b80 1 -- 172.21.15.70:0/1144496441 wait complete. 2024-06-04T05:07:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.118+0000 7f9b4bb69b80 20 client.18129 trim_cache size 0 max 16384 2024-06-04T05:07:02.126 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:07:02.126 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:07:02.126 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:07:02.126 INFO:teuthology.orchestra.run.smithi070.stdout: "6e738ccd-bfb5-40b3-bc3c-e0697675e8c0": { 2024-06-04T05:07:02.126 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:07:02.126 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "6e738ccd-bfb5-40b3-bc3c-e0697675e8c0", 2024-06-04T05:07:02.126 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:07:02.126 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:07:02.126 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:07:02.126 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:07:02.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.119+0000 7f9b412d4700 1 -- 172.21.15.70:0/300302787 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9b240b2a70 msgr2=0x7f9b240b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.119+0000 7f9b412d4700 1 --2- 172.21.15.70:0/300302787 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9b240b2a70 0x7f9b240b4f30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9b300097c0 tx=0x7f9b30009750 comp rx=0 tx=0).stop 2024-06-04T05:07:02.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.119+0000 7f9b412d4700 1 -- 172.21.15.70:0/300302787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b3c1095a0 msgr2=0x7f9b3c1998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:02.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.119+0000 7f9b412d4700 1 --2- 172.21.15.70:0/300302787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b3c1095a0 0x7f9b3c1998c0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400b0a0 tx=0x7f9b34004d00 comp rx=0 tx=0).stop 2024-06-04T05:07:02.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.119+0000 7f9b412d4700 1 -- 172.21.15.70:0/300302787 shutdown_connections 2024-06-04T05:07:02.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.119+0000 7f9b412d4700 1 --2- 172.21.15.70:0/300302787 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9b240b2a70 0x7f9b240b4f30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.119+0000 7f9b412d4700 1 --2- 172.21.15.70:0/300302787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b3c1095a0 0x7f9b3c1998c0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.119+0000 7f9b412d4700 1 --2- 172.21.15.70:0/300302787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b3c06bab0 0x7f9b3c19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.119+0000 7f9b412d4700 1 --2- 172.21.15.70:0/300302787 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b3c06c470 0x7f9b3c19fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:02.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.119+0000 7f9b412d4700 1 -- 172.21.15.70:0/300302787 >> 172.21.15.70:0/300302787 conn(0x7f9b3c0793b0 msgr2=0x7f9b3c101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:02.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.119+0000 7f9b412d4700 1 -- 172.21.15.70:0/300302787 shutdown_connections 2024-06-04T05:07:02.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:02.119+0000 7f9b412d4700 1 -- 172.21.15.70:0/300302787 wait complete. 2024-06-04T05:07:02.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 171175 1.91 MB/sec execute 355 sec latency 37.756 ms 2024-06-04T05:07:02.791 INFO:tasks.cephfs.filesystem:scrub status for tag:6e738ccd-bfb5-40b3-bc3c-e0697675e8c0 - {'path': '/', 'tag': '6e738ccd-bfb5-40b3-bc3c-e0697675e8c0', 'options': 'recursive,force'} 2024-06-04T05:07:03.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:03 smithi070 ceph-mon[32631]: pgmap v303: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 2.0 MiB/s wr, 211 op/s 2024-06-04T05:07:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:03 smithi112 ceph-mon[36453]: pgmap v303: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 2.0 MiB/s wr, 211 op/s 2024-06-04T05:07:03.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 171723 1.91 MB/sec execute 356 sec latency 24.062 ms 2024-06-04T05:07:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:03 smithi098 ceph-mon[35569]: pgmap v303: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 2.0 MiB/s wr, 211 op/s 2024-06-04T05:07:04.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:04 smithi070 ceph-mon[32631]: osdmap e87: 12 total, 12 up, 12 in 2024-06-04T05:07:04.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:04 smithi112 ceph-mon[36453]: osdmap e87: 12 total, 12 up, 12 in 2024-06-04T05:07:04.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 172254 1.91 MB/sec execute 357 sec latency 19.511 ms 2024-06-04T05:07:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:04 smithi098 ceph-mon[35569]: osdmap e87: 12 total, 12 up, 12 in 2024-06-04T05:07:05.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:05 smithi070 ceph-mon[32631]: pgmap v305: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-06-04T05:07:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:05 smithi112 ceph-mon[36453]: pgmap v305: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-06-04T05:07:05.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 172824 1.91 MB/sec execute 358 sec latency 33.178 ms 2024-06-04T05:07:05.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:05 smithi098 ceph-mon[35569]: pgmap v305: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-06-04T05:07:06.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 173569 1.92 MB/sec execute 359 sec latency 15.660 ms 2024-06-04T05:07:07.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:07 smithi070 ceph-mon[32631]: pgmap v306: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-06-04T05:07:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:07 smithi112 ceph-mon[36453]: pgmap v306: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-06-04T05:07:07.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 173979 1.93 MB/sec execute 360 sec latency 15.518 ms 2024-06-04T05:07:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:07 smithi098 ceph-mon[35569]: pgmap v306: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-06-04T05:07:08.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 174260 1.93 MB/sec execute 361 sec latency 21.039 ms 2024-06-04T05:07:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:08 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:07:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:08 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:07:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:08 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:07:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:08 smithi070 ceph-mon[32631]: osdmap e88: 12 total, 12 up, 12 in 2024-06-04T05:07:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:08 smithi070 ceph-mon[32631]: pgmap v308: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 2.0 MiB/s wr, 216 op/s 2024-06-04T05:07:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:08 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:07:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:08 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:07:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:08 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:07:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:08 smithi112 ceph-mon[36453]: osdmap e88: 12 total, 12 up, 12 in 2024-06-04T05:07:09.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:08 smithi112 ceph-mon[36453]: pgmap v308: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 2.0 MiB/s wr, 216 op/s 2024-06-04T05:07:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:08 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:07:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:08 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:07:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:08 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:07:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:08 smithi098 ceph-mon[35569]: osdmap e88: 12 total, 12 up, 12 in 2024-06-04T05:07:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:08 smithi098 ceph-mon[35569]: pgmap v308: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 2.0 MiB/s wr, 216 op/s 2024-06-04T05:07:09.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 174640 1.92 MB/sec execute 362 sec latency 14.999 ms 2024-06-04T05:07:10.731 INFO:tasks.workunit.client.0.smithi070.stdout: 1 175110 1.92 MB/sec execute 363 sec latency 44.503 ms 2024-06-04T05:07:11.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:11 smithi112 ceph-mon[36453]: pgmap v309: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 914 KiB/s wr, 97 op/s 2024-06-04T05:07:11.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:11 smithi098 ceph-mon[35569]: pgmap v309: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 914 KiB/s wr, 97 op/s 2024-06-04T05:07:11.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:11 smithi070 ceph-mon[32631]: pgmap v309: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 914 KiB/s wr, 97 op/s 2024-06-04T05:07:11.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 175669 1.93 MB/sec execute 364 sec latency 22.884 ms 2024-06-04T05:07:12.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 176224 1.93 MB/sec execute 365 sec latency 21.945 ms 2024-06-04T05:07:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:13 smithi112 ceph-mon[36453]: pgmap v310: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 1.8 MiB/s wr, 178 op/s 2024-06-04T05:07:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:13 smithi098 ceph-mon[35569]: pgmap v310: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 1.8 MiB/s wr, 178 op/s 2024-06-04T05:07:13.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:13 smithi070 ceph-mon[32631]: pgmap v310: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 1.8 MiB/s wr, 178 op/s 2024-06-04T05:07:13.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 176819 1.94 MB/sec execute 366 sec latency 16.126 ms 2024-06-04T05:07:14.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 177323 1.94 MB/sec execute 367 sec latency 17.183 ms 2024-06-04T05:07:15.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.176+0000 7f43d8c09640 1 -- 172.21.15.70:0/3422969555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43d4102a70 msgr2=0x7f43d4102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.176+0000 7f43d8c09640 1 --2- 172.21.15.70:0/3422969555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43d4102a70 0x7f43d4102ef0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f43c8009930 tx=0x7f43c802a420 comp rx=0 tx=0).stop 2024-06-04T05:07:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.176+0000 7f43d8c09640 1 -- 172.21.15.70:0/3422969555 shutdown_connections 2024-06-04T05:07:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.176+0000 7f43d8c09640 1 --2- 172.21.15.70:0/3422969555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43d4103430 0x7f43d410f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.176+0000 7f43d8c09640 1 --2- 172.21.15.70:0/3422969555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43d4102a70 0x7f43d4102ef0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.176+0000 7f43d8c09640 1 --2- 172.21.15.70:0/3422969555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d4108a80 0x7f43d4108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.176+0000 7f43d8c09640 1 -- 172.21.15.70:0/3422969555 >> 172.21.15.70:0/3422969555 conn(0x7f43d40fe500 msgr2=0x7f43d4100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.176+0000 7f43d8c09640 1 -- 172.21.15.70:0/3422969555 shutdown_connections 2024-06-04T05:07:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.176+0000 7f43d8c09640 1 -- 172.21.15.70:0/3422969555 wait complete. 2024-06-04T05:07:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.177+0000 7f43d8c09640 1 Processor -- start 2024-06-04T05:07:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.177+0000 7f43d8c09640 1 -- start start 2024-06-04T05:07:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.177+0000 7f43d8c09640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43d4102a70 0x7f43d4199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.177+0000 7f43d8c09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d4103430 0x7f43d4199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.177+0000 7f43d8c09640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43d4108a80 0x7f43d419da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.177+0000 7f43d8c09640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f43d4112bc0 con 0x7f43d4102a70 2024-06-04T05:07:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.177+0000 7f43d8c09640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f43d4112a40 con 0x7f43d4103430 2024-06-04T05:07:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.177+0000 7f43d8c09640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f43d4112d40 con 0x7f43d4108a80 2024-06-04T05:07:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.177+0000 7f43d259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43d4102a70 0x7f43d4199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.177+0000 7f43d259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43d4102a70 0x7f43d4199140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50834/0 (socket says 172.21.15.70:50834) 2024-06-04T05:07:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.177+0000 7f43d259c640 1 -- 172.21.15.70:0/2009899916 learned_addr learned my addr 172.21.15.70:0/2009899916 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.177+0000 7f43d1d9b640 1 --2- 172.21.15.70:0/2009899916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d4103430 0x7f43d4199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.177+0000 7f43d2d9d640 1 --2- 172.21.15.70:0/2009899916 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43d4108a80 0x7f43d419da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.178+0000 7f43d259c640 1 -- 172.21.15.70:0/2009899916 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43d4108a80 msgr2=0x7f43d419da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.178+0000 7f43d259c640 1 --2- 172.21.15.70:0/2009899916 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43d4108a80 0x7f43d419da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.178+0000 7f43d259c640 1 -- 172.21.15.70:0/2009899916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d4103430 msgr2=0x7f43d4199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.178+0000 7f43d259c640 1 --2- 172.21.15.70:0/2009899916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d4103430 0x7f43d4199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.178+0000 7f43d259c640 1 -- 172.21.15.70:0/2009899916 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d419e130 con 0x7f43d4102a70 2024-06-04T05:07:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.178+0000 7f43d1d9b640 1 --2- 172.21.15.70:0/2009899916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d4103430 0x7f43d4199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.178+0000 7f43d259c640 1 --2- 172.21.15.70:0/2009899916 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43d4102a70 0x7f43d4199140 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f43c400e9f0 tx=0x7f43c400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.178+0000 7f43c37fe640 1 -- 172.21.15.70:0/2009899916 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c400cd80 con 0x7f43d4102a70 2024-06-04T05:07:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.178+0000 7f43c37fe640 1 -- 172.21.15.70:0/2009899916 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43c40108c0 con 0x7f43d4102a70 2024-06-04T05:07:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.178+0000 7f43d8c09640 1 -- 172.21.15.70:0/2009899916 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43d419e420 con 0x7f43d4102a70 2024-06-04T05:07:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.178+0000 7f43c37fe640 1 -- 172.21.15.70:0/2009899916 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c40187c0 con 0x7f43d4102a70 2024-06-04T05:07:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.178+0000 7f43d8c09640 1 -- 172.21.15.70:0/2009899916 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43d41a5cb0 con 0x7f43d4102a70 2024-06-04T05:07:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.181+0000 7f43c37fe640 1 -- 172.21.15.70:0/2009899916 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f43c4010430 con 0x7f43d4102a70 2024-06-04T05:07:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.181+0000 7f43dba03740 1 Processor -- start 2024-06-04T05:07:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.181+0000 7f43dba03740 1 -- start start 2024-06-04T05:07:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.181+0000 7f43dba03740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324e9820 0x5603324e9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.181+0000 7f43dba03740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324ea140 0x5603324f44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.181+0000 7f43dba03740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324f4a00 0x5603324f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.181+0000 7f43dba03740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56033245bc30 con 0x5603324f4a00 2024-06-04T05:07:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.181+0000 7f43dba03740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56033244b7d0 con 0x5603324ea140 2024-06-04T05:07:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.181+0000 7f43dba03740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56033244f810 con 0x5603324e9820 2024-06-04T05:07:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.182+0000 7f43d1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324f4a00 0x5603324f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.182+0000 7f43c37fe640 1 --2- 172.21.15.70:0/2009899916 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f43bc0b2a10 0x7f43bc0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.182+0000 7f43d1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324f4a00 0x5603324f6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50840/0 (socket says 172.21.15.70:50840) 2024-06-04T05:07:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.182+0000 7f43d1d9b640 1 -- 172.21.15.70:0/2188583732 learned_addr learned my addr 172.21.15.70:0/2188583732 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.182+0000 7f43d2d9d640 1 --2- 172.21.15.70:0/2188583732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324e9820 0x5603324e9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.182+0000 7f43d259c640 1 --2- 172.21.15.70:0/2188583732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324ea140 0x5603324f44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.182+0000 7f43d2d9d640 1 --2- 172.21.15.70:0/2009899916 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f43bc0b2a10 0x7f43bc0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.182+0000 7f43c37fe640 1 -- 172.21.15.70:0/2009899916 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f43c40e2970 con 0x7f43d4102a70 2024-06-04T05:07:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.182+0000 7f43d1d9b640 1 -- 172.21.15.70:0/2188583732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324e9820 msgr2=0x5603324e9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.182+0000 7f43d1d9b640 1 --2- 172.21.15.70:0/2188583732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324e9820 0x5603324e9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.182+0000 7f43d1d9b640 1 -- 172.21.15.70:0/2188583732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324ea140 msgr2=0x5603324f44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.182+0000 7f43d1d9b640 1 --2- 172.21.15.70:0/2188583732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324ea140 0x5603324f44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.182+0000 7f43d1d9b640 1 -- 172.21.15.70:0/2188583732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560332160210 con 0x5603324f4a00 2024-06-04T05:07:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.182+0000 7f43d2d9d640 1 --2- 172.21.15.70:0/2009899916 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f43bc0b2a10 0x7f43bc0b4ed0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f43cc005c40 tx=0x7f43cc009500 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.182+0000 7f43d1d9b640 1 --2- 172.21.15.70:0/2188583732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324f4a00 0x5603324f6df0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f43c8009230 tx=0x7f43c8032e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58cae1ebfc4273e5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.182+0000 7f43c0ff9640 1 -- 172.21.15.70:0/2188583732 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c8035070 con 0x5603324f4a00 2024-06-04T05:07:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.183+0000 7f43c0ff9640 1 -- 172.21.15.70:0/2188583732 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43c80044c0 con 0x5603324f4a00 2024-06-04T05:07:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.183+0000 7f43c0ff9640 1 -- 172.21.15.70:0/2188583732 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c8003b30 con 0x5603324f4a00 2024-06-04T05:07:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.183+0000 7f43dba03740 1 -- 172.21.15.70:0/2188583732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324f4a00 msgr2=0x5603324f6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.183+0000 7f43dba03740 1 --2- 172.21.15.70:0/2188583732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324f4a00 0x5603324f6df0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f43c8009230 tx=0x7f43c8032e30 comp rx=0 tx=0).stop 2024-06-04T05:07:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.183+0000 7f43dba03740 1 -- 172.21.15.70:0/2188583732 shutdown_connections 2024-06-04T05:07:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.183+0000 7f43dba03740 1 --2- 172.21.15.70:0/2188583732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324ea140 0x5603324f44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.183+0000 7f43dba03740 1 --2- 172.21.15.70:0/2188583732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324f4a00 0x5603324f6df0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.183+0000 7f43dba03740 1 --2- 172.21.15.70:0/2188583732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324e9820 0x5603324e9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.183+0000 7f43dba03740 1 -- 172.21.15.70:0/2188583732 >> 172.21.15.70:0/2188583732 conn(0x5603324e5d30 msgr2=0x5603324e6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.183+0000 7f43dba03740 1 -- 172.21.15.70:0/2188583732 shutdown_connections 2024-06-04T05:07:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.183+0000 7f43dba03740 1 -- 172.21.15.70:0/2188583732 wait complete. 2024-06-04T05:07:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.184+0000 7f43dba03740 1 -- start start 2024-06-04T05:07:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.184+0000 7f43dba03740 1 Processor -- start 2024-06-04T05:07:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.184+0000 7f43dba03740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324e9820 0x5603324f1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.184+0000 7f43dba03740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324ea140 0x5603324f1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.184+0000 7f43dba03740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324f4a00 0x5603324f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.184+0000 7f43dba03740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560332131790 con 0x5603324e9820 2024-06-04T05:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.184+0000 7f43dba03740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560332407890 con 0x5603324ea140 2024-06-04T05:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.184+0000 7f43dba03740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5603324f75d0 con 0x5603324f4a00 2024-06-04T05:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.184+0000 7f43dba03740 10 client.0 _ll_register_callbacks cb 0x56033215fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.184+0000 7f43d259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324e9820 0x5603324f1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.184+0000 7f43d259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324e9820 0x5603324f1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50858/0 (socket says 172.21.15.70:50858) 2024-06-04T05:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.184+0000 7f43d259c640 1 -- 172.21.15.70:0/2985418336 learned_addr learned my addr 172.21.15.70:0/2985418336 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.184+0000 7f43d2d9d640 1 --2- 172.21.15.70:0/2985418336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324f4a00 0x5603324f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.184+0000 7f43d1d9b640 1 --2- 172.21.15.70:0/2985418336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324ea140 0x5603324f1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.185+0000 7f43d259c640 1 -- 172.21.15.70:0/2985418336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324f4a00 msgr2=0x5603324f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.185+0000 7f43d259c640 1 --2- 172.21.15.70:0/2985418336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324f4a00 0x5603324f23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.185+0000 7f43d259c640 1 -- 172.21.15.70:0/2985418336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324ea140 msgr2=0x5603324f1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.185+0000 7f43d259c640 1 --2- 172.21.15.70:0/2985418336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324ea140 0x5603324f1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.185+0000 7f43d259c640 1 -- 172.21.15.70:0/2985418336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56033222c620 con 0x5603324e9820 2024-06-04T05:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.185+0000 7f43d1d9b640 1 --2- 172.21.15.70:0/2985418336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324ea140 0x5603324f1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.185+0000 7f43d259c640 1 --2- 172.21.15.70:0/2985418336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324e9820 0x5603324f1950 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f43c40e0690 tx=0x7f43c4073630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83beda8c3f032326 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.185+0000 7f43b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.185+0000 7f43b9ffb640 1 -- 172.21.15.70:0/2985418336 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c40a9cd0 con 0x5603324e9820 2024-06-04T05:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.185+0000 7f43b9ffb640 1 -- 172.21.15.70:0/2985418336 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f43c40a5f60 con 0x5603324e9820 2024-06-04T05:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.185+0000 7f43dba03740 1 -- 172.21.15.70:0/2985418336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56033245db80 con 0x5603324e9820 2024-06-04T05:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.185+0000 7f43b9ffb640 1 -- 172.21.15.70:0/2985418336 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f43c40adfd0 con 0x5603324e9820 2024-06-04T05:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.185+0000 7f43dba03740 10 client.18141 fetch_fsmap learned FSMap version 32 2024-06-04T05:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.185+0000 7f43dba03740 1 -- 172.21.15.70:0/2985418336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560332503310 con 0x5603324e9820 2024-06-04T05:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.185+0000 7f43b9ffb640 1 -- 172.21.15.70:0/2985418336 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x56033245db80 con 0x5603324e9820 2024-06-04T05:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.186+0000 7f43b9ffb640 1 -- 172.21.15.70:0/2985418336 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f43c40a5640 con 0x5603324e9820 2024-06-04T05:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.186+0000 7f43dba03740 10 client.18141 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.186+0000 7f43dba03740 10 client.18141 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.186+0000 7f43dba03740 20 client.18141 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.186+0000 7f43dba03740 1 --2- 172.21.15.70:0/2985418336 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5603325047d0 0x560332506bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.186+0000 7f43dba03740 4 client.18141 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.186+0000 7f43dba03740 1 -- 172.21.15.70:0/2985418336 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560332457360 con 0x5603325047d0 2024-06-04T05:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.186+0000 7f43d1d9b640 1 --2- 172.21.15.70:0/2985418336 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5603325047d0 0x560332506bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.186+0000 7f43b9ffb640 10 client.18141 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.186+0000 7f43d1d9b640 1 --2- 172.21.15.70:0/2985418336 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5603325047d0 0x560332506bc0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57fefe246bc384e9 server_cookie=1022a0b5623fa72b in_seq=0 out_seq=0 2024-06-04T05:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.187+0000 7f43b9ffb640 1 -- 172.21.15.70:0/2985418336 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560332457360 con 0x5603325047d0 2024-06-04T05:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.187+0000 7f43b9ffb640 10 client.18141 handle_command_reply: tid=0 2024-06-04T05:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.187+0000 7f43dba03740 1 client.18141 shutdown 2024-06-04T05:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.187+0000 7f43dba03740 1 -- 172.21.15.70:0/2985418336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324e9820 msgr2=0x5603324f1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.187+0000 7f43dba03740 1 --2- 172.21.15.70:0/2985418336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324e9820 0x5603324f1950 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f43c40e0690 tx=0x7f43c4073630 comp rx=0 tx=0).stop 2024-06-04T05:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.187+0000 7f43dba03740 1 -- 172.21.15.70:0/2985418336 shutdown_connections 2024-06-04T05:07:15.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.187+0000 7f43dba03740 1 --2- 172.21.15.70:0/2985418336 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5603325047d0 0x560332506bc0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.187+0000 7f43dba03740 1 --2- 172.21.15.70:0/2985418336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324ea140 0x5603324f1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.187+0000 7f43b9ffb640 0 client.18141 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:15.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.187+0000 7f43dba03740 1 --2- 172.21.15.70:0/2985418336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324f4a00 0x5603324f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.187+0000 7f43dba03740 1 --2- 172.21.15.70:0/2985418336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324e9820 0x5603324f1950 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.187+0000 7f43dba03740 1 -- 172.21.15.70:0/2985418336 >> 172.21.15.70:0/2985418336 conn(0x5603324e5d30 msgr2=0x5603324e4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:15.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.187+0000 7f43dba03740 1 -- 172.21.15.70:0/2985418336 shutdown_connections 2024-06-04T05:07:15.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.187+0000 7f43dba03740 1 -- 172.21.15.70:0/2985418336 wait complete. 2024-06-04T05:07:15.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.187+0000 7f43dba03740 20 client.18141 trim_cache size 0 max 16384 2024-06-04T05:07:15.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.204+0000 7f43dba03740 1 Processor -- start 2024-06-04T05:07:15.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43dba03740 1 -- start start 2024-06-04T05:07:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43dba03740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324e9820 0x5603324ff850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43dba03740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324ea140 0x5603324f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43dba03740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324f4a00 0x56033250c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43dba03740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f43d41ae6b0 con 0x5603324ea140 2024-06-04T05:07:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43dba03740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f43d41b01f0 con 0x5603324e9820 2024-06-04T05:07:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43dba03740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f43d41aed80 con 0x5603324f4a00 2024-06-04T05:07:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43d1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324ea140 0x5603324f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43d1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324ea140 0x5603324f21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50864/0 (socket says 172.21.15.70:50864) 2024-06-04T05:07:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43d1d9b640 1 -- 172.21.15.70:0/134610303 learned_addr learned my addr 172.21.15.70:0/134610303 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43d2d9d640 1 --2- 172.21.15.70:0/134610303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324f4a00 0x56033250c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43d259c640 1 --2- 172.21.15.70:0/134610303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324e9820 0x5603324ff850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43d1d9b640 1 -- 172.21.15.70:0/134610303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324f4a00 msgr2=0x56033250c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43d1d9b640 1 --2- 172.21.15.70:0/134610303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324f4a00 0x56033250c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43d1d9b640 1 -- 172.21.15.70:0/134610303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324e9820 msgr2=0x5603324ff850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43d1d9b640 1 --2- 172.21.15.70:0/134610303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324e9820 0x5603324ff850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43d1d9b640 1 -- 172.21.15.70:0/134610303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603324f7160 con 0x5603324ea140 2024-06-04T05:07:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43d1d9b640 1 --2- 172.21.15.70:0/134610303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324ea140 0x5603324f21f0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f43c8033000 tx=0x7f43c80058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21ced5071ccccf00 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43bbfff640 1 -- 172.21.15.70:0/134610303 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c8052070 con 0x5603324ea140 2024-06-04T05:07:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43bbfff640 1 -- 172.21.15.70:0/134610303 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43c804e850 con 0x5603324ea140 2024-06-04T05:07:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.205+0000 7f43bbfff640 1 -- 172.21.15.70:0/134610303 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c8035040 con 0x5603324ea140 2024-06-04T05:07:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.206+0000 7f43dba03740 1 -- 172.21.15.70:0/134610303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324ea140 msgr2=0x5603324f21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.206+0000 7f43dba03740 1 --2- 172.21.15.70:0/134610303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324ea140 0x5603324f21f0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f43c8033000 tx=0x7f43c80058d0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.206+0000 7f43dba03740 1 -- 172.21.15.70:0/134610303 shutdown_connections 2024-06-04T05:07:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.206+0000 7f43dba03740 1 --2- 172.21.15.70:0/134610303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324f4a00 0x56033250c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.206+0000 7f43dba03740 1 --2- 172.21.15.70:0/134610303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324ea140 0x5603324f21f0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.206+0000 7f43dba03740 1 --2- 172.21.15.70:0/134610303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324e9820 0x5603324ff850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.206+0000 7f43dba03740 1 -- 172.21.15.70:0/134610303 >> 172.21.15.70:0/134610303 conn(0x5603324e5d30 msgr2=0x5603324e4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.206+0000 7f43dba03740 1 -- 172.21.15.70:0/134610303 shutdown_connections 2024-06-04T05:07:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.206+0000 7f43dba03740 1 -- 172.21.15.70:0/134610303 wait complete. 2024-06-04T05:07:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.206+0000 7f43dba03740 1 -- start start 2024-06-04T05:07:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.206+0000 7f43dba03740 1 Processor -- start 2024-06-04T05:07:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43dba03740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324e9820 0x5603324fcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43dba03740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324ea140 0x5603324fd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43dba03740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324f4a00 0x5603324f98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43dba03740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f43d41afb20 con 0x5603324f4a00 2024-06-04T05:07:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43dba03740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f43d41af450 con 0x5603324e9820 2024-06-04T05:07:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43dba03740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f43d41ac6b0 con 0x5603324ea140 2024-06-04T05:07:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43dba03740 10 client.0 _ll_register_callbacks cb 0x7f43c40a9620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43d2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324f4a00 0x5603324f98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43d259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324e9820 0x5603324fcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43d2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324f4a00 0x5603324f98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50868/0 (socket says 172.21.15.70:50868) 2024-06-04T05:07:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43d1d9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324ea140 0x5603324fd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43d2d9d640 1 -- 172.21.15.70:0/3464923489 learned_addr learned my addr 172.21.15.70:0/3464923489 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43d2d9d640 1 -- 172.21.15.70:0/3464923489 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324ea140 msgr2=0x5603324fd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43d2d9d640 1 --2- 172.21.15.70:0/3464923489 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324ea140 0x5603324fd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43d2d9d640 1 -- 172.21.15.70:0/3464923489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324e9820 msgr2=0x5603324fcaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43d2d9d640 1 --2- 172.21.15.70:0/3464923489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324e9820 0x5603324fcaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43d2d9d640 1 -- 172.21.15.70:0/3464923489 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603324fa110 con 0x5603324f4a00 2024-06-04T05:07:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43d259c640 1 --2- 172.21.15.70:0/3464923489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324e9820 0x5603324fcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43d2d9d640 1 --2- 172.21.15.70:0/3464923489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324f4a00 0x5603324f98e0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f43cc011910 tx=0x7f43cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f835a6c333ebf60 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.207+0000 7f43b9ffb640 1 -- 172.21.15.70:0/3464923489 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc004b20 con 0x5603324f4a00 2024-06-04T05:07:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.208+0000 7f43b9ffb640 1 -- 172.21.15.70:0/3464923489 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f43cc004cc0 con 0x5603324f4a00 2024-06-04T05:07:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.208+0000 7f43dba03740 1 -- 172.21.15.70:0/3464923489 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43c804e850 con 0x5603324f4a00 2024-06-04T05:07:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.208+0000 7f43b9ffb640 1 -- 172.21.15.70:0/3464923489 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f43cc036050 con 0x5603324f4a00 2024-06-04T05:07:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.208+0000 7f43dba03740 10 client.18147 fetch_fsmap learned FSMap version 32 2024-06-04T05:07:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.208+0000 7f43dba03740 1 -- 172.21.15.70:0/3464923489 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56033256ffa0 con 0x5603324f4a00 2024-06-04T05:07:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.208+0000 7f43b9ffb640 1 -- 172.21.15.70:0/3464923489 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43c804e850 con 0x5603324f4a00 2024-06-04T05:07:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.208+0000 7f43b9ffb640 1 -- 172.21.15.70:0/3464923489 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f43cc01d040 con 0x5603324f4a00 2024-06-04T05:07:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.208+0000 7f43dba03740 10 client.18147 fetch_fsmap finished waiting for FSMap version 32 2024-06-04T05:07:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.208+0000 7f43dba03740 10 client.18147 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.208+0000 7f43dba03740 20 client.18147 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.208+0000 7f43dba03740 1 --2- 172.21.15.70:0/3464923489 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5603325707d0 0x560332572bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.208+0000 7f43dba03740 4 client.18147 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:07:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.208+0000 7f43dba03740 1 -- 172.21.15.70:0/3464923489 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5603323d15f0 con 0x5603325707d0 2024-06-04T05:07:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.208+0000 7f43d259c640 1 --2- 172.21.15.70:0/3464923489 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5603325707d0 0x560332572bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.209+0000 7f43b9ffb640 10 client.18147 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:15.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.209+0000 7f43d259c640 1 --2- 172.21.15.70:0/3464923489 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5603325707d0 0x560332572bf0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d724ab83f47cbc1 server_cookie=1f9fa219541849f6 in_seq=0 out_seq=0 2024-06-04T05:07:15.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.235+0000 7f43b9ffb640 1 -- 172.21.15.70:0/3464923489 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5603323d15f0 con 0x5603325707d0 2024-06-04T05:07:15.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.235+0000 7f43b9ffb640 10 client.18147 handle_command_reply: tid=0 2024-06-04T05:07:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.235+0000 7f43dba03740 1 client.18147 shutdown 2024-06-04T05:07:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.236+0000 7f43dba03740 1 -- 172.21.15.70:0/3464923489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324f4a00 msgr2=0x5603324f98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.236+0000 7f43dba03740 1 --2- 172.21.15.70:0/3464923489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324f4a00 0x5603324f98e0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f43cc011910 tx=0x7f43cc004a40 comp rx=0 tx=0).stop 2024-06-04T05:07:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.236+0000 7f43dba03740 1 -- 172.21.15.70:0/3464923489 shutdown_connections 2024-06-04T05:07:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.236+0000 7f43dba03740 1 --2- 172.21.15.70:0/3464923489 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5603325707d0 0x560332572bf0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.236+0000 7f43dba03740 1 --2- 172.21.15.70:0/3464923489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603324f4a00 0x5603324f98e0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.236+0000 7f43dba03740 1 --2- 172.21.15.70:0/3464923489 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5603324ea140 0x5603324fd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.236+0000 7f43b9ffb640 0 client.18147 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.236+0000 7f43dba03740 1 --2- 172.21.15.70:0/3464923489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603324e9820 0x5603324fcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.236+0000 7f43dba03740 1 -- 172.21.15.70:0/3464923489 >> 172.21.15.70:0/3464923489 conn(0x5603324e5d30 msgr2=0x5603324f3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.236+0000 7f43dba03740 1 -- 172.21.15.70:0/3464923489 shutdown_connections 2024-06-04T05:07:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.236+0000 7f43dba03740 1 -- 172.21.15.70:0/3464923489 wait complete. 2024-06-04T05:07:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.236+0000 7f43dba03740 20 client.18147 trim_cache size 0 max 16384 2024-06-04T05:07:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:07:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:07:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:07:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.236+0000 7f43d8c09640 1 -- 172.21.15.70:0/2009899916 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f43bc0b2a10 msgr2=0x7f43bc0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.236+0000 7f43d8c09640 1 --2- 172.21.15.70:0/2009899916 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f43bc0b2a10 0x7f43bc0b4ed0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f43cc005c40 tx=0x7f43cc009500 comp rx=0 tx=0).stop 2024-06-04T05:07:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.236+0000 7f43d8c09640 1 -- 172.21.15.70:0/2009899916 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43d4102a70 msgr2=0x7f43d4199140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.236+0000 7f43d8c09640 1 --2- 172.21.15.70:0/2009899916 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43d4102a70 0x7f43d4199140 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f43c400e9f0 tx=0x7f43c400eeb0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.237+0000 7f43d8c09640 1 -- 172.21.15.70:0/2009899916 shutdown_connections 2024-06-04T05:07:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.237+0000 7f43d8c09640 1 --2- 172.21.15.70:0/2009899916 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f43bc0b2a10 0x7f43bc0b4ed0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.237+0000 7f43d8c09640 1 --2- 172.21.15.70:0/2009899916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d4103430 0x7f43d4199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.237+0000 7f43d8c09640 1 --2- 172.21.15.70:0/2009899916 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43d4108a80 0x7f43d419da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.237+0000 7f43d8c09640 1 --2- 172.21.15.70:0/2009899916 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43d4102a70 0x7f43d4199140 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.237+0000 7f43d8c09640 1 -- 172.21.15.70:0/2009899916 >> 172.21.15.70:0/2009899916 conn(0x7f43d40fe500 msgr2=0x7f43d40fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.237+0000 7f43d8c09640 1 -- 172.21.15.70:0/2009899916 shutdown_connections 2024-06-04T05:07:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:15.237+0000 7f43d8c09640 1 -- 172.21.15.70:0/2009899916 wait complete. 2024-06-04T05:07:15.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:15 smithi112 ceph-mon[36453]: pgmap v311: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 1.6 MiB/s wr, 157 op/s 2024-06-04T05:07:15.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:15 smithi098 ceph-mon[35569]: pgmap v311: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 1.6 MiB/s wr, 157 op/s 2024-06-04T05:07:15.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:15 smithi070 ceph-mon[32631]: pgmap v311: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 1.6 MiB/s wr, 157 op/s 2024-06-04T05:07:15.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 177655 1.94 MB/sec execute 368 sec latency 29.979 ms 2024-06-04T05:07:16.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 177929 1.94 MB/sec execute 369 sec latency 14.839 ms 2024-06-04T05:07:17.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:17 smithi112 ceph-mon[36453]: pgmap v312: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.7 MiB/s wr, 151 op/s 2024-06-04T05:07:17.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:17 smithi098 ceph-mon[35569]: pgmap v312: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.7 MiB/s wr, 151 op/s 2024-06-04T05:07:17.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:17 smithi070 ceph-mon[32631]: pgmap v312: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.7 MiB/s wr, 151 op/s 2024-06-04T05:07:17.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 178393 1.94 MB/sec execute 370 sec latency 17.541 ms 2024-06-04T05:07:18.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:18 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:07:18.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:18 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:07:18.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:18 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:07:18.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 178899 1.94 MB/sec execute 371 sec latency 20.765 ms 2024-06-04T05:07:19.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:19 smithi112 ceph-mon[36453]: pgmap v313: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 1.7 MiB/s wr, 147 op/s 2024-06-04T05:07:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:19 smithi098 ceph-mon[35569]: pgmap v313: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 1.7 MiB/s wr, 147 op/s 2024-06-04T05:07:19.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:19 smithi070 ceph-mon[32631]: pgmap v313: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 1.7 MiB/s wr, 147 op/s 2024-06-04T05:07:19.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 179507 1.94 MB/sec execute 372 sec latency 22.157 ms 2024-06-04T05:07:20.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 179973 1.94 MB/sec execute 373 sec latency 23.880 ms 2024-06-04T05:07:21.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:21 smithi112 ceph-mon[36453]: pgmap v314: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-06-04T05:07:21.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:21 smithi098 ceph-mon[35569]: pgmap v314: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-06-04T05:07:21.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:21 smithi070 ceph-mon[32631]: pgmap v314: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-06-04T05:07:21.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 180684 1.95 MB/sec execute 374 sec latency 14.871 ms 2024-06-04T05:07:22.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 180982 1.95 MB/sec execute 375 sec latency 466.461 ms 2024-06-04T05:07:23.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:23 smithi112 ceph-mon[36453]: pgmap v315: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 2.1 MiB/s wr, 188 op/s 2024-06-04T05:07:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:23 smithi098 ceph-mon[35569]: pgmap v315: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 2.1 MiB/s wr, 188 op/s 2024-06-04T05:07:23.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:23 smithi070 ceph-mon[32631]: pgmap v315: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 2.1 MiB/s wr, 188 op/s 2024-06-04T05:07:23.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 180982 1.95 MB/sec execute 376 sec latency 1466.492 ms 2024-06-04T05:07:24.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 180982 1.94 MB/sec execute 377 sec latency 2466.581 ms 2024-06-04T05:07:25.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:25 smithi112 ceph-mon[36453]: pgmap v316: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.4 MiB/s wr, 122 op/s 2024-06-04T05:07:25.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:25 smithi098 ceph-mon[35569]: pgmap v316: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.4 MiB/s wr, 122 op/s 2024-06-04T05:07:25.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:25 smithi070 ceph-mon[32631]: pgmap v316: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.4 MiB/s wr, 122 op/s 2024-06-04T05:07:25.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 180982 1.94 MB/sec execute 378 sec latency 3466.637 ms 2024-06-04T05:07:26.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 180982 1.93 MB/sec execute 379 sec latency 4466.744 ms 2024-06-04T05:07:27.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:27 smithi112 ceph-mon[36453]: pgmap v317: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 1.5 MiB/s wr, 144 op/s 2024-06-04T05:07:27.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:27 smithi098 ceph-mon[35569]: pgmap v317: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 1.5 MiB/s wr, 144 op/s 2024-06-04T05:07:27.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:27 smithi070 ceph-mon[32631]: pgmap v317: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 1.5 MiB/s wr, 144 op/s 2024-06-04T05:07:27.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 181049 1.93 MB/sec execute 380 sec latency 4781.910 ms 2024-06-04T05:07:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.332+0000 7f4156454640 1 -- 172.21.15.70:0/152876417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4150104600 msgr2=0x7f415010cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.332+0000 7f4156454640 1 --2- 172.21.15.70:0/152876417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4150104600 0x7f415010cae0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f4138009900 tx=0x7f413802a510 comp rx=0 tx=0).stop 2024-06-04T05:07:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.332+0000 7f4156454640 1 -- 172.21.15.70:0/152876417 shutdown_connections 2024-06-04T05:07:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.332+0000 7f4156454640 1 --2- 172.21.15.70:0/152876417 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f415010d020 0x7f415010f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.332+0000 7f4156454640 1 --2- 172.21.15.70:0/152876417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4150104600 0x7f415010cae0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.332+0000 7f4156454640 1 --2- 172.21.15.70:0/152876417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4150103ce0 0x7f41501040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.332+0000 7f4156454640 1 -- 172.21.15.70:0/152876417 >> 172.21.15.70:0/152876417 conn(0x7f4150078ad0 msgr2=0x7f4150078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.332+0000 7f4156454640 1 -- 172.21.15.70:0/152876417 shutdown_connections 2024-06-04T05:07:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.332+0000 7f4156454640 1 -- 172.21.15.70:0/152876417 wait complete. 2024-06-04T05:07:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.333+0000 7f4156454640 1 Processor -- start 2024-06-04T05:07:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.333+0000 7f4156454640 1 -- start start 2024-06-04T05:07:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.333+0000 7f4156454640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4150103ce0 0x7f4150199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.333+0000 7f4156454640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4150104600 0x7f41501996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.333+0000 7f4156454640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f415010d020 0x7f415019da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.333+0000 7f4156454640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4150112b80 con 0x7f4150103ce0 2024-06-04T05:07:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.333+0000 7f4156454640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4150112a00 con 0x7f415010d020 2024-06-04T05:07:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.333+0000 7f4156454640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4150112d00 con 0x7f4150104600 2024-06-04T05:07:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.333+0000 7f414ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4150103ce0 0x7f4150199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.333+0000 7f414ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4150103ce0 0x7f4150199190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60394/0 (socket says 172.21.15.70:60394) 2024-06-04T05:07:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.333+0000 7f414ffff640 1 -- 172.21.15.70:0/1122831920 learned_addr learned my addr 172.21.15.70:0/1122831920 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:28.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.333+0000 7f414f7fe640 1 --2- 172.21.15.70:0/1122831920 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4150104600 0x7f41501996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:28.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.333+0000 7f41549f1640 1 --2- 172.21.15.70:0/1122831920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f415010d020 0x7f415019da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:28.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.334+0000 7f414ffff640 1 -- 172.21.15.70:0/1122831920 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4150104600 msgr2=0x7f41501996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:28.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.334+0000 7f414ffff640 1 --2- 172.21.15.70:0/1122831920 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4150104600 0x7f41501996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.334+0000 7f414ffff640 1 -- 172.21.15.70:0/1122831920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f415010d020 msgr2=0x7f415019da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:28.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.334+0000 7f414ffff640 1 --2- 172.21.15.70:0/1122831920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f415010d020 0x7f415019da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.334+0000 7f414ffff640 1 -- 172.21.15.70:0/1122831920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f415019e180 con 0x7f4150103ce0 2024-06-04T05:07:28.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.334+0000 7f414ffff640 1 --2- 172.21.15.70:0/1122831920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4150103ce0 0x7f4150199190 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f414000d950 tx=0x7f414000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:28.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.334+0000 7f414d7fa640 1 -- 172.21.15.70:0/1122831920 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4140017030 con 0x7f4150103ce0 2024-06-04T05:07:28.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.334+0000 7f414d7fa640 1 -- 172.21.15.70:0/1122831920 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4140004020 con 0x7f4150103ce0 2024-06-04T05:07:28.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.334+0000 7f414d7fa640 1 -- 172.21.15.70:0/1122831920 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41400055d0 con 0x7f4150103ce0 2024-06-04T05:07:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.334+0000 7f4156454640 1 -- 172.21.15.70:0/1122831920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f415019e470 con 0x7f4150103ce0 2024-06-04T05:07:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.334+0000 7f4156454640 1 -- 172.21.15.70:0/1122831920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41501a5d00 con 0x7f4150103ce0 2024-06-04T05:07:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.336+0000 7f414d7fa640 1 -- 172.21.15.70:0/1122831920 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f4140009a30 con 0x7f4150103ce0 2024-06-04T05:07:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f415916b740 1 Processor -- start 2024-06-04T05:07:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f415916b740 1 -- start start 2024-06-04T05:07:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f415916b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660bfc30 0x560d660c0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f415916b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660c0550 0x560d660ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f415916b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660cae10 0x560d660cd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f415916b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560d6602dc30 con 0x560d660c0550 2024-06-04T05:07:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f415916b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560d6601d7d0 con 0x560d660cae10 2024-06-04T05:07:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f415916b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560d66021810 con 0x560d660bfc30 2024-06-04T05:07:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f414ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660c0550 0x560d660ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f414ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660c0550 0x560d660ca8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60396/0 (socket says 172.21.15.70:60396) 2024-06-04T05:07:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f414ffff640 1 -- 172.21.15.70:0/1541857527 learned_addr learned my addr 172.21.15.70:0/1541857527 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f414f7fe640 1 --2- 172.21.15.70:0/1541857527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660cae10 0x560d660cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f41549f1640 1 --2- 172.21.15.70:0/1541857527 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660bfc30 0x560d660c0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f414d7fa640 1 --2- 172.21.15.70:0/1122831920 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f413c0b2a10 0x7f413c0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.354 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f414ffff640 1 -- 172.21.15.70:0/1541857527 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660bfc30 msgr2=0x560d660c0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:28.354 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f414ffff640 1 --2- 172.21.15.70:0/1541857527 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660bfc30 0x560d660c0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.354 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f414ffff640 1 -- 172.21.15.70:0/1541857527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660cae10 msgr2=0x560d660cd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:28.354 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f414ffff640 1 --2- 172.21.15.70:0/1541857527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660cae10 0x560d660cd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.354 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f414ffff640 1 -- 172.21.15.70:0/1541857527 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d65d32210 con 0x560d660c0550 2024-06-04T05:07:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.337+0000 7f41549f1640 1 --2- 172.21.15.70:0/1541857527 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660bfc30 0x560d660c0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:07:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.338+0000 7f41549f1640 1 --2- 172.21.15.70:0/1122831920 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f413c0b2a10 0x7f413c0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.338+0000 7f414d7fa640 1 -- 172.21.15.70:0/1122831920 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f41400e1fb0 con 0x7f4150103ce0 2024-06-04T05:07:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.338+0000 7f414ffff640 1 --2- 172.21.15.70:0/1541857527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660c0550 0x560d660ca8d0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f41400df640 tx=0x7f41400a5970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd05cd379bb3c945 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.338+0000 7f41467fc640 1 -- 172.21.15.70:0/1541857527 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41400a9030 con 0x560d660c0550 2024-06-04T05:07:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.338+0000 7f41467fc640 1 -- 172.21.15.70:0/1541857527 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f41400a8d70 con 0x560d660c0550 2024-06-04T05:07:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.338+0000 7f41467fc640 1 -- 172.21.15.70:0/1541857527 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41400a89d0 con 0x560d660c0550 2024-06-04T05:07:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.339+0000 7f41549f1640 1 --2- 172.21.15.70:0/1122831920 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f413c0b2a10 0x7f413c0b4ed0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f414800a780 tx=0x7f4148004a40 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.339+0000 7f415916b740 1 -- 172.21.15.70:0/1541857527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660c0550 msgr2=0x560d660ca8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.339+0000 7f415916b740 1 --2- 172.21.15.70:0/1541857527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660c0550 0x560d660ca8d0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f41400df640 tx=0x7f41400a5970 comp rx=0 tx=0).stop 2024-06-04T05:07:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.339+0000 7f415916b740 1 -- 172.21.15.70:0/1541857527 shutdown_connections 2024-06-04T05:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.339+0000 7f415916b740 1 --2- 172.21.15.70:0/1541857527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660cae10 0x560d660cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.339+0000 7f415916b740 1 --2- 172.21.15.70:0/1541857527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660c0550 0x560d660ca8d0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.339+0000 7f415916b740 1 --2- 172.21.15.70:0/1541857527 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660bfc30 0x560d660c0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.339+0000 7f415916b740 1 -- 172.21.15.70:0/1541857527 >> 172.21.15.70:0/1541857527 conn(0x560d660bc140 msgr2=0x560d660bc520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.339+0000 7f415916b740 1 -- 172.21.15.70:0/1541857527 shutdown_connections 2024-06-04T05:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.339+0000 7f415916b740 1 -- 172.21.15.70:0/1541857527 wait complete. 2024-06-04T05:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.339+0000 7f415916b740 1 -- start start 2024-06-04T05:07:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.339+0000 7f415916b740 1 Processor -- start 2024-06-04T05:07:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f415916b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660bfc30 0x560d660c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f415916b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660c0550 0x560d660c8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f415916b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660c8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f415916b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560d65d03790 con 0x560d660cae10 2024-06-04T05:07:28.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f415916b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560d65fd9890 con 0x560d660c0550 2024-06-04T05:07:28.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f415916b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560d660cda40 con 0x560d660bfc30 2024-06-04T05:07:28.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f415916b740 10 client.0 _ll_register_callbacks cb 0x560d65d31a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:28.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f41549f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660c8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:28.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f41549f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660c8890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60408/0 (socket says 172.21.15.70:60408) 2024-06-04T05:07:28.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f41549f1640 1 -- 172.21.15.70:0/1481002505 learned_addr learned my addr 172.21.15.70:0/1481002505 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:28.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f414f7fe640 1 --2- 172.21.15.70:0/1481002505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660c0550 0x560d660c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:28.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f414ffff640 1 --2- 172.21.15.70:0/1481002505 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660bfc30 0x560d660c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:28.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f41549f1640 1 -- 172.21.15.70:0/1481002505 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660bfc30 msgr2=0x560d660c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:28.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f41549f1640 1 --2- 172.21.15.70:0/1481002505 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660bfc30 0x560d660c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f41549f1640 1 -- 172.21.15.70:0/1481002505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660c0550 msgr2=0x560d660c8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f41549f1640 1 --2- 172.21.15.70:0/1481002505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660c0550 0x560d660c8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f41549f1640 1 -- 172.21.15.70:0/1481002505 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d65dfe620 con 0x560d660cae10 2024-06-04T05:07:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f41549f1640 1 --2- 172.21.15.70:0/1481002505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660c8890 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f414800a700 tx=0x7f4148004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a575f17daeadbdb0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f412f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f412f7fe640 1 -- 172.21.15.70:0/1481002505 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4148028060 con 0x560d660cae10 2024-06-04T05:07:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f412f7fe640 1 -- 172.21.15.70:0/1481002505 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f4148015070 con 0x560d660cae10 2024-06-04T05:07:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.340+0000 7f415916b740 1 -- 172.21.15.70:0/1481002505 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41400a8d70 con 0x560d660cae10 2024-06-04T05:07:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.341+0000 7f412f7fe640 1 -- 172.21.15.70:0/1481002505 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4148019ab0 con 0x560d660cae10 2024-06-04T05:07:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.341+0000 7f415916b740 10 client.18159 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.341+0000 7f415916b740 1 -- 172.21.15.70:0/1481002505 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560d660d97c0 con 0x560d660cae10 2024-06-04T05:07:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.341+0000 7f412f7fe640 1 -- 172.21.15.70:0/1481002505 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f41400a8d70 con 0x560d660cae10 2024-06-04T05:07:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.341+0000 7f412f7fe640 1 -- 172.21.15.70:0/1481002505 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f414802b3e0 con 0x560d660cae10 2024-06-04T05:07:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.341+0000 7f415916b740 10 client.18159 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.341+0000 7f415916b740 10 client.18159 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.341+0000 7f415916b740 20 client.18159 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.341+0000 7f415916b740 1 --2- 172.21.15.70:0/1481002505 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560d660dacd0 0x560d660dd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.341+0000 7f415916b740 4 client.18159 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:07:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.341+0000 7f415916b740 1 -- 172.21.15.70:0/1481002505 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d6602fb80 con 0x560d660dacd0 2024-06-04T05:07:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.341+0000 7f414ffff640 1 --2- 172.21.15.70:0/1481002505 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560d660dacd0 0x560d660dd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.342+0000 7f414ffff640 1 --2- 172.21.15.70:0/1481002505 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560d660dacd0 0x560d660dd0c0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fef6de88074004a0 server_cookie=47b4fa895685d6c0 in_seq=0 out_seq=0 2024-06-04T05:07:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.342+0000 7f412f7fe640 10 client.18159 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.342+0000 7f412f7fe640 1 -- 172.21.15.70:0/1481002505 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560d6602fb80 con 0x560d660dacd0 2024-06-04T05:07:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.342+0000 7f412f7fe640 10 client.18159 handle_command_reply: tid=0 2024-06-04T05:07:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.342+0000 7f415916b740 1 client.18159 shutdown 2024-06-04T05:07:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.343+0000 7f415916b740 1 -- 172.21.15.70:0/1481002505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 msgr2=0x560d660c8890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.343+0000 7f415916b740 1 --2- 172.21.15.70:0/1481002505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660c8890 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f414800a700 tx=0x7f4148004ab0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.343+0000 7f415916b740 1 -- 172.21.15.70:0/1481002505 shutdown_connections 2024-06-04T05:07:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.343+0000 7f415916b740 1 --2- 172.21.15.70:0/1481002505 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560d660dacd0 0x560d660dd0c0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.343+0000 7f415916b740 1 --2- 172.21.15.70:0/1481002505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660c0550 0x560d660c8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.343+0000 7f415916b740 1 --2- 172.21.15.70:0/1481002505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660c8890 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.343+0000 7f412f7fe640 0 client.18159 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.343+0000 7f415916b740 1 --2- 172.21.15.70:0/1481002505 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660bfc30 0x560d660c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.343+0000 7f415916b740 1 -- 172.21.15.70:0/1481002505 >> 172.21.15.70:0/1481002505 conn(0x560d660bc140 msgr2=0x560d660b6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.343+0000 7f415916b740 1 -- 172.21.15.70:0/1481002505 shutdown_connections 2024-06-04T05:07:28.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.343+0000 7f415916b740 1 -- 172.21.15.70:0/1481002505 wait complete. 2024-06-04T05:07:28.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.343+0000 7f415916b740 20 client.18159 trim_cache size 0 max 16384 2024-06-04T05:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.358+0000 7f415916b740 1 Processor -- start 2024-06-04T05:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.358+0000 7f415916b740 1 -- start start 2024-06-04T05:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.358+0000 7f415916b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660bfc30 0x560d660c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.358+0000 7f415916b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660c0550 0x560d660c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.358+0000 7f415916b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660e2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.358+0000 7f415916b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f41501ac4e0 con 0x560d660cae10 2024-06-04T05:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.358+0000 7f415916b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f41501ae080 con 0x560d660bfc30 2024-06-04T05:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.358+0000 7f415916b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f41501acbb0 con 0x560d660c0550 2024-06-04T05:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.358+0000 7f41549f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660e2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.358+0000 7f41549f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660e2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60410/0 (socket says 172.21.15.70:60410) 2024-06-04T05:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.358+0000 7f41549f1640 1 -- 172.21.15.70:0/3115925046 learned_addr learned my addr 172.21.15.70:0/3115925046 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.359+0000 7f41549f1640 1 -- 172.21.15.70:0/3115925046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660c0550 msgr2=0x560d660c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.359+0000 7f41549f1640 1 --2- 172.21.15.70:0/3115925046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660c0550 0x560d660c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.359+0000 7f41549f1640 1 -- 172.21.15.70:0/3115925046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660bfc30 msgr2=0x560d660c86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.359+0000 7f41549f1640 1 --2- 172.21.15.70:0/3115925046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660bfc30 0x560d660c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.359+0000 7f41549f1640 1 -- 172.21.15.70:0/3115925046 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d660c0380 con 0x560d660cae10 2024-06-04T05:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.359+0000 7f41549f1640 1 --2- 172.21.15.70:0/3115925046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660e2990 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f414801bb40 tx=0x7f414800a550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b109d53f3512c59 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.359+0000 7f4145ffb640 1 -- 172.21.15.70:0/3115925046 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414802b5e0 con 0x560d660cae10 2024-06-04T05:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.359+0000 7f4145ffb640 1 -- 172.21.15.70:0/3115925046 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4148019c50 con 0x560d660cae10 2024-06-04T05:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.359+0000 7f4145ffb640 1 -- 172.21.15.70:0/3115925046 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414802ca60 con 0x560d660cae10 2024-06-04T05:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.359+0000 7f415916b740 1 -- 172.21.15.70:0/3115925046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 msgr2=0x560d660e2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.359+0000 7f415916b740 1 --2- 172.21.15.70:0/3115925046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660e2990 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f414801bb40 tx=0x7f414800a550 comp rx=0 tx=0).stop 2024-06-04T05:07:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.359+0000 7f415916b740 1 -- 172.21.15.70:0/3115925046 shutdown_connections 2024-06-04T05:07:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.359+0000 7f415916b740 1 --2- 172.21.15.70:0/3115925046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660e2990 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.359+0000 7f415916b740 1 --2- 172.21.15.70:0/3115925046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660c0550 0x560d660c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.359+0000 7f415916b740 1 --2- 172.21.15.70:0/3115925046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660bfc30 0x560d660c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.359+0000 7f415916b740 1 -- 172.21.15.70:0/3115925046 >> 172.21.15.70:0/3115925046 conn(0x560d660bc140 msgr2=0x560d660c9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.360+0000 7f415916b740 1 -- 172.21.15.70:0/3115925046 shutdown_connections 2024-06-04T05:07:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.360+0000 7f415916b740 1 -- 172.21.15.70:0/3115925046 wait complete. 2024-06-04T05:07:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.360+0000 7f415916b740 1 -- start start 2024-06-04T05:07:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.360+0000 7f415916b740 1 Processor -- start 2024-06-04T05:07:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.360+0000 7f415916b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660bfc30 0x560d660cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.360+0000 7f415916b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660c0550 0x560d660cef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.360+0000 7f415916b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.360+0000 7f415916b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f41501ad980 con 0x560d660cae10 2024-06-04T05:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.360+0000 7f415916b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f41501ad280 con 0x560d660c0550 2024-06-04T05:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.360+0000 7f415916b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f41501aa4e0 con 0x560d660bfc30 2024-06-04T05:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.360+0000 7f415916b740 10 client.0 _ll_register_callbacks cb 0x7f414800a850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.360+0000 7f41549f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.360+0000 7f41549f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660cf4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60422/0 (socket says 172.21.15.70:60422) 2024-06-04T05:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.360+0000 7f41549f1640 1 -- 172.21.15.70:0/2161968837 learned_addr learned my addr 172.21.15.70:0/2161968837 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f414f7fe640 1 --2- 172.21.15.70:0/2161968837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660c0550 0x560d660cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f414ffff640 1 --2- 172.21.15.70:0/2161968837 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660bfc30 0x560d660cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f41549f1640 1 -- 172.21.15.70:0/2161968837 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660bfc30 msgr2=0x560d660cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f41549f1640 1 --2- 172.21.15.70:0/2161968837 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660bfc30 0x560d660cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f41549f1640 1 -- 172.21.15.70:0/2161968837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660c0550 msgr2=0x560d660cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f41549f1640 1 --2- 172.21.15.70:0/2161968837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660c0550 0x560d660cef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f41549f1640 1 -- 172.21.15.70:0/2161968837 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d661448e0 con 0x560d660cae10 2024-06-04T05:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f414ffff640 1 --2- 172.21.15.70:0/2161968837 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660bfc30 0x560d660cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:07:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f414f7fe640 1 --2- 172.21.15.70:0/2161968837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660c0550 0x560d660cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:07:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f41549f1640 1 --2- 172.21.15.70:0/2161968837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660cf4c0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f4148010c90 tx=0x7f414802b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=486ef2c3b64d87e8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f412f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f412f7fe640 1 -- 172.21.15.70:0/2161968837 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414802bc90 con 0x560d660cae10 2024-06-04T05:07:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f412f7fe640 1 -- 172.21.15.70:0/2161968837 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f414802be30 con 0x560d660cae10 2024-06-04T05:07:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f415916b740 1 -- 172.21.15.70:0/2161968837 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4148019c50 con 0x560d660cae10 2024-06-04T05:07:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f412f7fe640 1 -- 172.21.15.70:0/2161968837 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4148044050 con 0x560d660cae10 2024-06-04T05:07:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f415916b740 10 client.18165 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f415916b740 1 -- 172.21.15.70:0/2161968837 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560d66145bd0 con 0x560d660cae10 2024-06-04T05:07:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f412f7fe640 1 -- 172.21.15.70:0/2161968837 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4148019c50 con 0x560d660cae10 2024-06-04T05:07:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.361+0000 7f412f7fe640 1 -- 172.21.15.70:0/2161968837 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f4148035550 con 0x560d660cae10 2024-06-04T05:07:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.362+0000 7f415916b740 10 client.18165 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.362+0000 7f415916b740 10 client.18165 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.362+0000 7f415916b740 20 client.18165 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.362+0000 7f415916b740 1 --2- 172.21.15.70:0/2161968837 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560d66146200 0x560d66148620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.362+0000 7f415916b740 4 client.18165 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:07:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.362+0000 7f415916b740 1 -- 172.21.15.70:0/2161968837 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560d66029360 con 0x560d66146200 2024-06-04T05:07:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.362+0000 7f414ffff640 1 --2- 172.21.15.70:0/2161968837 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560d66146200 0x560d66148620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.362+0000 7f414ffff640 1 --2- 172.21.15.70:0/2161968837 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560d66146200 0x560d66148620 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14f3128770bf08cf server_cookie=d0496547c40e84f4 in_seq=0 out_seq=0 2024-06-04T05:07:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:28.362+0000 7f412f7fe640 10 client.18165 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:28.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 181049 1.92 MB/sec execute 381 sec latency 1463.380 ms 2024-06-04T05:07:29.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:29 smithi112 ceph-mon[36453]: pgmap v318: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 835 KiB/s wr, 84 op/s 2024-06-04T05:07:29.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:29 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:07:29.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:29 smithi098 ceph-mon[35569]: pgmap v318: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 835 KiB/s wr, 84 op/s 2024-06-04T05:07:29.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:29 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:07:29.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:29 smithi070 ceph-mon[32631]: pgmap v318: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 835 KiB/s wr, 84 op/s 2024-06-04T05:07:29.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:29 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:07:29.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 181049 1.92 MB/sec execute 382 sec latency 2463.460 ms 2024-06-04T05:07:30.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 181049 1.91 MB/sec execute 383 sec latency 3463.568 ms 2024-06-04T05:07:31.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:31 smithi112 ceph-mon[36453]: pgmap v319: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 835 KiB/s wr, 84 op/s 2024-06-04T05:07:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:31 smithi098 ceph-mon[35569]: pgmap v319: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 835 KiB/s wr, 84 op/s 2024-06-04T05:07:31.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:31 smithi070 ceph-mon[32631]: pgmap v319: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 835 KiB/s wr, 84 op/s 2024-06-04T05:07:31.733 INFO:tasks.workunit.client.0.smithi070.stdout: 1 181049 1.91 MB/sec execute 384 sec latency 4463.607 ms 2024-06-04T05:07:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.048+0000 7f412f7fe640 1 -- 172.21.15.70:0/2161968837 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x560d66029360 con 0x560d66146200 2024-06-04T05:07:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.048+0000 7f412f7fe640 10 client.18165 handle_command_reply: tid=0 2024-06-04T05:07:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.048+0000 7f415916b740 1 client.18165 shutdown 2024-06-04T05:07:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.048+0000 7f415916b740 1 -- 172.21.15.70:0/2161968837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 msgr2=0x560d660cf4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.048+0000 7f415916b740 1 --2- 172.21.15.70:0/2161968837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660cf4c0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f4148010c90 tx=0x7f414802b610 comp rx=0 tx=0).stop 2024-06-04T05:07:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.048+0000 7f415916b740 1 -- 172.21.15.70:0/2161968837 shutdown_connections 2024-06-04T05:07:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.048+0000 7f415916b740 1 --2- 172.21.15.70:0/2161968837 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560d66146200 0x560d66148620 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f415916b740 1 --2- 172.21.15.70:0/2161968837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560d660c0550 0x560d660cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f415916b740 1 --2- 172.21.15.70:0/2161968837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d660cae10 0x560d660cf4c0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f412f7fe640 0 client.18165 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f415916b740 1 --2- 172.21.15.70:0/2161968837 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560d660bfc30 0x560d660cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f415916b740 1 -- 172.21.15.70:0/2161968837 >> 172.21.15.70:0/2161968837 conn(0x560d660bc140 msgr2=0x560d660c9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f415916b740 1 -- 172.21.15.70:0/2161968837 shutdown_connections 2024-06-04T05:07:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f415916b740 1 -- 172.21.15.70:0/2161968837 wait complete. 2024-06-04T05:07:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f415916b740 20 client.18165 trim_cache size 0 max 16384 2024-06-04T05:07:32.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:07:32.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:07:32.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:07:32.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f4156454640 1 -- 172.21.15.70:0/1122831920 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f413c0b2a10 msgr2=0x7f413c0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f4156454640 1 --2- 172.21.15.70:0/1122831920 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f413c0b2a10 0x7f413c0b4ed0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f414800a780 tx=0x7f4148004a40 comp rx=0 tx=0).stop 2024-06-04T05:07:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f4156454640 1 -- 172.21.15.70:0/1122831920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4150103ce0 msgr2=0x7f4150199190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f4156454640 1 --2- 172.21.15.70:0/1122831920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4150103ce0 0x7f4150199190 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f414000d950 tx=0x7f414000de10 comp rx=0 tx=0).stop 2024-06-04T05:07:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f4156454640 1 -- 172.21.15.70:0/1122831920 shutdown_connections 2024-06-04T05:07:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f4156454640 1 --2- 172.21.15.70:0/1122831920 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f413c0b2a10 0x7f413c0b4ed0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f4156454640 1 --2- 172.21.15.70:0/1122831920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f415010d020 0x7f415019da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f4156454640 1 --2- 172.21.15.70:0/1122831920 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4150104600 0x7f41501996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f4156454640 1 --2- 172.21.15.70:0/1122831920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4150103ce0 0x7f4150199190 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f4156454640 1 -- 172.21.15.70:0/1122831920 >> 172.21.15.70:0/1122831920 conn(0x7f4150078ad0 msgr2=0x7f41500fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.049+0000 7f4156454640 1 -- 172.21.15.70:0/1122831920 shutdown_connections 2024-06-04T05:07:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:32.050+0000 7f4156454640 1 -- 172.21.15.70:0/1122831920 wait complete. 2024-06-04T05:07:32.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:32 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:07:32.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:32 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:32 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:32 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:32 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:32 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:32 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:32 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:07:32.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:32 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:32 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:32 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:32 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:32 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:32 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:32 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:07:32.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:32 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:32 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:32 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:32 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:32 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:32 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:32.733 INFO:tasks.workunit.client.0.smithi070.stdout: 1 181251 1.91 MB/sec execute 385 sec latency 4772.176 ms 2024-06-04T05:07:32.792 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:07:33.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:33 smithi098 ceph-mon[35569]: pgmap v320: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 893 KiB/s wr, 100 op/s 2024-06-04T05:07:33.289 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:07:33.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:33 smithi112 ceph-mon[36453]: pgmap v320: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 893 KiB/s wr, 100 op/s 2024-06-04T05:07:33.448 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:33 smithi070 ceph-mon[32631]: pgmap v320: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 893 KiB/s wr, 100 op/s 2024-06-04T05:07:33.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 181512 1.90 MB/sec execute 386 sec latency 15.323 ms 2024-06-04T05:07:34.733 INFO:tasks.workunit.client.0.smithi070.stdout: 1 182010 1.90 MB/sec execute 387 sec latency 15.175 ms 2024-06-04T05:07:34.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:34 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:34.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:34 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:34.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:34 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:34.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:34 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:34.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:34 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:34 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:34 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:34 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:34 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:34 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:34 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:34 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:34 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:34.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:34 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:34.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:34 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:34.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:34 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:34.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:34 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:34 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:34 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:34 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:34 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:34 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:34 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:34 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:34 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:34.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:34 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:34 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:34 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:34 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:34 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:34 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:34 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:34 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:34 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:34 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:34.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:34 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:35.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.151+0000 7f025aaac700 1 -- 172.21.15.70:0/3489695523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f025406c020 msgr2=0x7f025406c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:35.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.151+0000 7f025aaac700 1 --2- 172.21.15.70:0/3489695523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f025406c020 0x7f025406c480 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f0240009990 tx=0x7f0240009ca0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.151+0000 7f025aaac700 1 -- 172.21.15.70:0/3489695523 shutdown_connections 2024-06-04T05:07:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.151+0000 7f025aaac700 1 --2- 172.21.15.70:0/3489695523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0254105f70 0x7f0254106400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.151+0000 7f025aaac700 1 --2- 172.21.15.70:0/3489695523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0254108460 0x7f025406bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.151+0000 7f025aaac700 1 --2- 172.21.15.70:0/3489695523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f025406c020 0x7f025406c480 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.151+0000 7f025aaac700 1 -- 172.21.15.70:0/3489695523 >> 172.21.15.70:0/3489695523 conn(0x7f02540793b0 msgr2=0x7f02540797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.151+0000 7f025aaac700 1 -- 172.21.15.70:0/3489695523 shutdown_connections 2024-06-04T05:07:35.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.151+0000 7f025aaac700 1 -- 172.21.15.70:0/3489695523 wait complete. 2024-06-04T05:07:35.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.152+0000 7f025aaac700 1 Processor -- start 2024-06-04T05:07:35.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.152+0000 7f025aaac700 1 -- start start 2024-06-04T05:07:35.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.152+0000 7f025aaac700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f025406c020 0x7f02541a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.152+0000 7f025aaac700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0254105f70 0x7f02541a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.152+0000 7f025aaac700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0254108460 0x7f02541a4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.152+0000 7f025aaac700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0254119b70 con 0x7f025406c020 2024-06-04T05:07:35.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.152+0000 7f025aaac700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02541199f0 con 0x7f0254105f70 2024-06-04T05:07:35.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.152+0000 7f025aaac700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0254119cf0 con 0x7f0254108460 2024-06-04T05:07:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.152+0000 7f0258848700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f025406c020 0x7f02541a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.152+0000 7f0258848700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f025406c020 0x7f02541a0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60436/0 (socket says 172.21.15.70:60436) 2024-06-04T05:07:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.152+0000 7f0258848700 1 -- 172.21.15.70:0/3714594230 learned_addr learned my addr 172.21.15.70:0/3714594230 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.152+0000 7f0253fff700 1 --2- 172.21.15.70:0/3714594230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0254105f70 0x7f02541a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.153+0000 7f0259049700 1 --2- 172.21.15.70:0/3714594230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0254108460 0x7f02541a4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.153+0000 7f0258848700 1 -- 172.21.15.70:0/3714594230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0254108460 msgr2=0x7f02541a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.153+0000 7f0258848700 1 --2- 172.21.15.70:0/3714594230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0254108460 0x7f02541a4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.153+0000 7f0258848700 1 -- 172.21.15.70:0/3714594230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0254105f70 msgr2=0x7f02541a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.153+0000 7f0258848700 1 --2- 172.21.15.70:0/3714594230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0254105f70 0x7f02541a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.153+0000 7f0258848700 1 -- 172.21.15.70:0/3714594230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02541a50b0 con 0x7f025406c020 2024-06-04T05:07:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.153+0000 7f0258848700 1 --2- 172.21.15.70:0/3714594230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f025406c020 0x7f02541a0140 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f024800e9a0 tx=0x7f024800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.153+0000 7f0251ffb700 1 -- 172.21.15.70:0/3714594230 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024800cbd0 con 0x7f025406c020 2024-06-04T05:07:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.153+0000 7f0251ffb700 1 -- 172.21.15.70:0/3714594230 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f024800cd70 con 0x7f025406c020 2024-06-04T05:07:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.153+0000 7f0251ffb700 1 -- 172.21.15.70:0/3714594230 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0248018680 con 0x7f025406c020 2024-06-04T05:07:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.153+0000 7f025aaac700 1 -- 172.21.15.70:0/3714594230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02541a53a0 con 0x7f025406c020 2024-06-04T05:07:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.154+0000 7f025aaac700 1 -- 172.21.15.70:0/3714594230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02541acc20 con 0x7f025406c020 2024-06-04T05:07:35.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.156+0000 7f0251ffb700 1 -- 172.21.15.70:0/3714594230 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f0248010950 con 0x7f025406c020 2024-06-04T05:07:35.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.158+0000 7f0251ffb700 1 --2- 172.21.15.70:0/3714594230 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02440b2a70 0x7f02440b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.158+0000 7f0251ffb700 1 -- 172.21.15.70:0/3714594230 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f02480e28c0 con 0x7f025406c020 2024-06-04T05:07:35.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.158+0000 7f0265341b80 1 Processor -- start 2024-06-04T05:07:35.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.158+0000 7f0253fff700 1 --2- 172.21.15.70:0/3714594230 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02440b2a70 0x7f02440b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:35.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.158+0000 7f0265341b80 1 -- start start 2024-06-04T05:07:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.159+0000 7f0265341b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d49860 0x557f48d49c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.159+0000 7f0265341b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d4a210 0x557f48d4d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.159+0000 7f0265341b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4df60 0x557f48d4e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.159+0000 7f0265341b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557f48ca8c90 con 0x557f48d4a210 2024-06-04T05:07:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.159+0000 7f0265341b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557f48c98830 con 0x557f48d49860 2024-06-04T05:07:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.159+0000 7f0265341b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557f48c9c870 con 0x557f48d4df60 2024-06-04T05:07:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.159+0000 7f0259049700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4df60 0x557f48d4e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.159+0000 7f0259049700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4df60 0x557f48d4e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:34344/0 (socket says 172.21.15.70:34344) 2024-06-04T05:07:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.159+0000 7f0259049700 1 -- 172.21.15.70:0/3927055094 learned_addr learned my addr 172.21.15.70:0/3927055094 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.159+0000 7f0258848700 1 --2- 172.21.15.70:0/3927055094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d49860 0x557f48d49c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.159+0000 7f0253fff700 1 --2- 172.21.15.70:0/3714594230 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02440b2a70 0x7f02440b4f30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f024000b2e0 tx=0x7f024001a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.159+0000 7f0259049700 1 -- 172.21.15.70:0/3927055094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d49860 msgr2=0x557f48d49c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:35.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.159+0000 7f0259049700 1 --2- 172.21.15.70:0/3927055094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d49860 0x557f48d49c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.159+0000 7f0259049700 1 -- 172.21.15.70:0/3927055094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d4a210 msgr2=0x557f48d4d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:35.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.159+0000 7f0259049700 1 --2- 172.21.15.70:0/3927055094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d4a210 0x557f48d4d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.159+0000 7f0259049700 1 -- 172.21.15.70:0/3927055094 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f48be88d0 con 0x557f48d4df60 2024-06-04T05:07:35.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.159+0000 7f0259049700 1 --2- 172.21.15.70:0/3927055094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4df60 0x557f48d4e3a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f024c009440 tx=0x7f024c009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=887462e100400f8d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:35.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.160+0000 7f023effd700 1 -- 172.21.15.70:0/3927055094 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c013750 con 0x557f48d4df60 2024-06-04T05:07:35.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.160+0000 7f023effd700 1 -- 172.21.15.70:0/3927055094 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f024c013d80 con 0x557f48d4df60 2024-06-04T05:07:35.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.160+0000 7f023effd700 1 -- 172.21.15.70:0/3927055094 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c015dd0 con 0x557f48d4df60 2024-06-04T05:07:35.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.160+0000 7f0265341b80 1 -- 172.21.15.70:0/3927055094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4df60 msgr2=0x557f48d4e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:35.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.160+0000 7f0265341b80 1 --2- 172.21.15.70:0/3927055094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4df60 0x557f48d4e3a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f024c009440 tx=0x7f024c009920 comp rx=0 tx=0).stop 2024-06-04T05:07:35.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.161+0000 7f0265341b80 1 -- 172.21.15.70:0/3927055094 shutdown_connections 2024-06-04T05:07:35.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.161+0000 7f0265341b80 1 --2- 172.21.15.70:0/3927055094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4df60 0x557f48d4e3a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.161+0000 7f0265341b80 1 --2- 172.21.15.70:0/3927055094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d49860 0x557f48d49c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.161+0000 7f0265341b80 1 --2- 172.21.15.70:0/3927055094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d4a210 0x557f48d4d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.161+0000 7f0265341b80 1 -- 172.21.15.70:0/3927055094 >> 172.21.15.70:0/3927055094 conn(0x557f48c77080 msgr2=0x557f48d43e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:35.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.161+0000 7f0265341b80 1 -- 172.21.15.70:0/3927055094 shutdown_connections 2024-06-04T05:07:35.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.161+0000 7f0265341b80 1 -- 172.21.15.70:0/3927055094 wait complete. 2024-06-04T05:07:35.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.161+0000 7f0265341b80 1 -- start start 2024-06-04T05:07:35.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.161+0000 7f0265341b80 1 Processor -- start 2024-06-04T05:07:35.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0265341b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d49860 0x557f48d553c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0265341b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d4a210 0x557f48d526c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0265341b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4df60 0x557f48d52c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0265341b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557f48b9efa0 con 0x557f48d49860 2024-06-04T05:07:35.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0265341b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557f48c3a7c0 con 0x557f48d4a210 2024-06-04T05:07:35.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0265341b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557f48d56120 con 0x557f48d4df60 2024-06-04T05:07:35.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0265341b80 10 client.0 _ll_register_callbacks cb 0x557f48c33920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:35.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0253fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d4a210 0x557f48d526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:35.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0258848700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d49860 0x557f48d553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:35.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0259049700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4df60 0x557f48d52c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:35.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0258848700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d49860 0x557f48d553c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60460/0 (socket says 172.21.15.70:60460) 2024-06-04T05:07:35.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0258848700 1 -- 172.21.15.70:0/1148203506 learned_addr learned my addr 172.21.15.70:0/1148203506 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:35.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0253fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d4a210 0x557f48d526c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:47510/0 (socket says 172.21.15.70:47510) 2024-06-04T05:07:35.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0259049700 1 -- 172.21.15.70:0/1148203506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d4a210 msgr2=0x557f48d526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:35.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0259049700 1 --2- 172.21.15.70:0/1148203506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d4a210 0x557f48d526c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0259049700 1 -- 172.21.15.70:0/1148203506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d49860 msgr2=0x557f48d553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:35.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0259049700 1 --2- 172.21.15.70:0/1148203506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d49860 0x557f48d553c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0259049700 1 -- 172.21.15.70:0/1148203506 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f48c37690 con 0x557f48d4df60 2024-06-04T05:07:35.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0258848700 1 --2- 172.21.15.70:0/1148203506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d49860 0x557f48d553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:35.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0259049700 1 --2- 172.21.15.70:0/1148203506 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4df60 0x557f48d52c00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f024c009510 tx=0x7f024c0156f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a45f9bd0f061b336 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:35.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.162+0000 7f0233fff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:07:35.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.163+0000 7f0233fff700 1 -- 172.21.15.70:0/1148203506 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c023070 con 0x557f48d4df60 2024-06-04T05:07:35.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.163+0000 7f0233fff700 1 -- 172.21.15.70:0/1148203506 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f024c015de0 con 0x557f48d4df60 2024-06-04T05:07:35.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.163+0000 7f0265341b80 1 -- 172.21.15.70:0/1148203506 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f024c013d80 con 0x557f48d4df60 2024-06-04T05:07:35.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.163+0000 7f0233fff700 1 -- 172.21.15.70:0/1148203506 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f024c0095e0 con 0x557f48d4df60 2024-06-04T05:07:35.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.163+0000 7f0265341b80 10 client.27385 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:35.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.163+0000 7f0265341b80 1 -- 172.21.15.70:0/1148203506 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x557f48d60830 con 0x557f48d4df60 2024-06-04T05:07:35.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.163+0000 7f0233fff700 1 -- 172.21.15.70:0/1148203506 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f024c013d80 con 0x557f48d4df60 2024-06-04T05:07:35.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.163+0000 7f0233fff700 1 -- 172.21.15.70:0/1148203506 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f024c028920 con 0x557f48d4df60 2024-06-04T05:07:35.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.164+0000 7f0265341b80 10 client.27385 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:35.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.164+0000 7f0265341b80 10 client.27385 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:35.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.164+0000 7f0265341b80 20 client.27385 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:35.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.164+0000 7f0265341b80 1 --2- 172.21.15.70:0/1148203506 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557f48d62d00 0x557f48d650f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.164+0000 7f0265341b80 4 client.27385 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:07:35.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.164+0000 7f0265341b80 1 -- 172.21.15.70:0/1148203506 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f48d159e0 con 0x557f48d62d00 2024-06-04T05:07:35.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.164+0000 7f0258848700 1 --2- 172.21.15.70:0/1148203506 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557f48d62d00 0x557f48d650f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:35.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.164+0000 7f0233fff700 10 client.27385 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:35.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.164+0000 7f0258848700 1 --2- 172.21.15.70:0/1148203506 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557f48d62d00 0x557f48d650f0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dba44ec691e589a server_cookie=45b3887004b80e78 in_seq=0 out_seq=0 2024-06-04T05:07:35.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.165+0000 7f0233fff700 1 -- 172.21.15.70:0/1148203506 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557f48d159e0 con 0x557f48d62d00 2024-06-04T05:07:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.165+0000 7f0233fff700 10 client.27385 handle_command_reply: tid=0 2024-06-04T05:07:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.165+0000 7f0265341b80 1 client.27385 shutdown 2024-06-04T05:07:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.165+0000 7f0265341b80 1 -- 172.21.15.70:0/1148203506 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4df60 msgr2=0x557f48d52c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.165+0000 7f0265341b80 1 --2- 172.21.15.70:0/1148203506 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4df60 0x557f48d52c00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f024c009510 tx=0x7f024c0156f0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.166+0000 7f0265341b80 1 -- 172.21.15.70:0/1148203506 shutdown_connections 2024-06-04T05:07:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.166+0000 7f0265341b80 1 --2- 172.21.15.70:0/1148203506 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4df60 0x557f48d52c00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.166+0000 7f0265341b80 1 --2- 172.21.15.70:0/1148203506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d49860 0x557f48d553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.166+0000 7f0265341b80 1 --2- 172.21.15.70:0/1148203506 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557f48d62d00 0x557f48d650f0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.166+0000 7f0265341b80 1 --2- 172.21.15.70:0/1148203506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d4a210 0x557f48d526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.166+0000 7f0233fff700 0 client.27385 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.166+0000 7f0265341b80 1 -- 172.21.15.70:0/1148203506 >> 172.21.15.70:0/1148203506 conn(0x557f48c77080 msgr2=0x557f48d4f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.166+0000 7f0265341b80 1 -- 172.21.15.70:0/1148203506 shutdown_connections 2024-06-04T05:07:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.166+0000 7f0265341b80 1 -- 172.21.15.70:0/1148203506 wait complete. 2024-06-04T05:07:35.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.166+0000 7f0265341b80 20 client.27385 trim_cache size 0 max 16384 2024-06-04T05:07:35.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.189+0000 7f0265341b80 1 Processor -- start 2024-06-04T05:07:35.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.189+0000 7f0265341b80 1 -- start start 2024-06-04T05:07:35.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.190+0000 7f0265341b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d49860 0x557f48d6a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.190+0000 7f0265341b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4a210 0x557f48d6abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.190+0000 7f0265341b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d4df60 0x557f48d4b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.190+0000 7f0265341b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f02541b4460 con 0x557f48d4df60 2024-06-04T05:07:35.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.190+0000 7f0265341b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02541b6000 con 0x557f48d49860 2024-06-04T05:07:35.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.190+0000 7f0265341b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f02541b4b30 con 0x557f48d4a210 2024-06-04T05:07:35.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.190+0000 7f0259049700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d4df60 0x557f48d4b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:35.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.190+0000 7f0253fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4a210 0x557f48d6abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:35.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.190+0000 7f0259049700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d4df60 0x557f48d4b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60462/0 (socket says 172.21.15.70:60462) 2024-06-04T05:07:35.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.190+0000 7f0253fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4a210 0x557f48d6abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:34358/0 (socket says 172.21.15.70:34358) 2024-06-04T05:07:35.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.190+0000 7f0259049700 1 -- 172.21.15.70:0/179705582 learned_addr learned my addr 172.21.15.70:0/179705582 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:35.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.190+0000 7f0259049700 1 -- 172.21.15.70:0/179705582 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4a210 msgr2=0x557f48d6abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:35.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.190+0000 7f0259049700 1 --2- 172.21.15.70:0/179705582 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4a210 0x557f48d6abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.190+0000 7f0259049700 1 -- 172.21.15.70:0/179705582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d49860 msgr2=0x557f48d6a690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:07:35.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.190+0000 7f0259049700 1 --2- 172.21.15.70:0/179705582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d49860 0x557f48d6a690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.190+0000 7f0259049700 1 -- 172.21.15.70:0/179705582 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f48d52490 con 0x557f48d4df60 2024-06-04T05:07:35.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.191+0000 7f0259049700 1 --2- 172.21.15.70:0/179705582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d4df60 0x557f48d4b8b0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f024c015810 tx=0x7f024c015840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c60d7ea627d20b3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:35.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.191+0000 7f023e7fc700 1 -- 172.21.15.70:0/179705582 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c037070 con 0x557f48d4df60 2024-06-04T05:07:35.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.191+0000 7f023e7fc700 1 -- 172.21.15.70:0/179705582 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f024c0288f0 con 0x557f48d4df60 2024-06-04T05:07:35.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.191+0000 7f023e7fc700 1 -- 172.21.15.70:0/179705582 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c023070 con 0x557f48d4df60 2024-06-04T05:07:35.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.192+0000 7f0265341b80 1 -- 172.21.15.70:0/179705582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d4df60 msgr2=0x557f48d4b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:35.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.192+0000 7f0265341b80 1 --2- 172.21.15.70:0/179705582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d4df60 0x557f48d4b8b0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f024c015810 tx=0x7f024c015840 comp rx=0 tx=0).stop 2024-06-04T05:07:35.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.192+0000 7f0265341b80 1 -- 172.21.15.70:0/179705582 shutdown_connections 2024-06-04T05:07:35.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.192+0000 7f0265341b80 1 --2- 172.21.15.70:0/179705582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d4df60 0x557f48d4b8b0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.192+0000 7f0265341b80 1 --2- 172.21.15.70:0/179705582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d49860 0x557f48d6a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.192+0000 7f0265341b80 1 --2- 172.21.15.70:0/179705582 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4a210 0x557f48d6abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.192+0000 7f0265341b80 1 -- 172.21.15.70:0/179705582 >> 172.21.15.70:0/179705582 conn(0x557f48c77080 msgr2=0x557f48d43e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:35.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.192+0000 7f0265341b80 1 -- 172.21.15.70:0/179705582 shutdown_connections 2024-06-04T05:07:35.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.192+0000 7f0265341b80 1 -- 172.21.15.70:0/179705582 wait complete. 2024-06-04T05:07:35.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.192+0000 7f0265341b80 1 -- start start 2024-06-04T05:07:35.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.193+0000 7f0265341b80 1 Processor -- start 2024-06-04T05:07:35.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.193+0000 7f0265341b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d49860 0x557f48d5b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.193+0000 7f0265341b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4a210 0x557f48d5bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.193+0000 7f0265341b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d4df60 0x557f48d58380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.193+0000 7f0265341b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f02541b5900 con 0x557f48d4df60 2024-06-04T05:07:35.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.193+0000 7f0253fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4a210 0x557f48d5bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:35.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.193+0000 7f0259049700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d4df60 0x557f48d58380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:35.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.193+0000 7f0258848700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d49860 0x557f48d5b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:35.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.193+0000 7f0258848700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d49860 0x557f48d5b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:47528/0 (socket says 172.21.15.70:47528) 2024-06-04T05:07:35.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.193+0000 7f0258848700 1 -- 172.21.15.70:0/1928695250 learned_addr learned my addr 172.21.15.70:0/1928695250 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:35.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.193+0000 7f0265341b80 1 -- 172.21.15.70:0/1928695250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02541b5200 con 0x557f48d49860 2024-06-04T05:07:35.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.193+0000 7f0265341b80 1 -- 172.21.15.70:0/1928695250 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f02541b2420 con 0x557f48d4a210 2024-06-04T05:07:35.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.194+0000 7f0259049700 1 -- 172.21.15.70:0/1928695250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4a210 msgr2=0x557f48d5bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:35.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.194+0000 7f0259049700 1 --2- 172.21.15.70:0/1928695250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4a210 0x557f48d5bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.194+0000 7f0259049700 1 -- 172.21.15.70:0/1928695250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d49860 msgr2=0x557f48d5b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:35.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.194+0000 7f0259049700 1 --2- 172.21.15.70:0/1928695250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d49860 0x557f48d5b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.194+0000 7f0265341b80 10 client.0 _ll_register_callbacks cb 0x557f48c33920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:35.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.194+0000 7f0259049700 1 -- 172.21.15.70:0/1928695250 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f48d58a80 con 0x557f48d4df60 2024-06-04T05:07:35.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.194+0000 7f0265341b80 1 -- 172.21.15.70:0/1928695250 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f024c0288f0 con 0x557f48d4df60 2024-06-04T05:07:35.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.194+0000 7f0259049700 1 --2- 172.21.15.70:0/1928695250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d4df60 0x557f48d58380 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f024c009440 tx=0x7f024c006160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d1802b0d25c8f5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:35.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.194+0000 7f0233fff700 10 client.18180 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:35.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.194+0000 7f0233fff700 1 -- 172.21.15.70:0/1928695250 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c037030 con 0x557f48d4df60 2024-06-04T05:07:35.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.194+0000 7f0233fff700 1 -- 172.21.15.70:0/1928695250 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f024c028460 con 0x557f48d4df60 2024-06-04T05:07:35.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.194+0000 7f0233fff700 1 -- 172.21.15.70:0/1928695250 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f024c023030 con 0x557f48d4df60 2024-06-04T05:07:35.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.194+0000 7f0265341b80 10 client.18180 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:35.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.194+0000 7f0265341b80 1 -- 172.21.15.70:0/1928695250 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({fsmap=34}) -- 0x557f48dcde00 con 0x557f48d4df60 2024-06-04T05:07:35.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.195+0000 7f0233fff700 1 -- 172.21.15.70:0/1928695250 <== mon.0 v2:172.21.15.70:3300/0 4 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f024c013550 con 0x557f48d4df60 2024-06-04T05:07:35.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.195+0000 7f0265341b80 10 client.18180 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:35.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.195+0000 7f0265341b80 10 client.18180 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:35.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.195+0000 7f0265341b80 20 client.18180 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:35.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.195+0000 7f0265341b80 1 --2- 172.21.15.70:0/1928695250 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557f48d62d00 0x557f48dd0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:35.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.195+0000 7f0265341b80 4 client.18180 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:07:35.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.195+0000 7f0265341b80 1 -- 172.21.15.70:0/1928695250 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557f48caabe0 con 0x557f48d62d00 2024-06-04T05:07:35.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.195+0000 7f0258848700 1 --2- 172.21.15.70:0/1928695250 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557f48d62d00 0x557f48dd0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:35.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.195+0000 7f0233fff700 10 client.18180 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:35.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.195+0000 7f0258848700 1 --2- 172.21.15.70:0/1928695250 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557f48d62d00 0x557f48dd0350 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=933623ff421488e7 server_cookie=3303ebc2c0c7a6ff in_seq=0 out_seq=0 2024-06-04T05:07:35.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.196+0000 7f0233fff700 1 -- 172.21.15.70:0/1928695250 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557f48caabe0 con 0x557f48d62d00 2024-06-04T05:07:35.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.196+0000 7f0233fff700 10 client.18180 handle_command_reply: tid=0 2024-06-04T05:07:35.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.196+0000 7f0265341b80 1 client.18180 shutdown 2024-06-04T05:07:35.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.197+0000 7f0265341b80 1 -- 172.21.15.70:0/1928695250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d4df60 msgr2=0x557f48d58380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:35.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.197+0000 7f0265341b80 1 --2- 172.21.15.70:0/1928695250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d4df60 0x557f48d58380 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f024c009440 tx=0x7f024c006160 comp rx=0 tx=0).stop 2024-06-04T05:07:35.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.197+0000 7f0265341b80 1 -- 172.21.15.70:0/1928695250 shutdown_connections 2024-06-04T05:07:35.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.197+0000 7f0265341b80 1 --2- 172.21.15.70:0/1928695250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f48d4df60 0x557f48d58380 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.197+0000 7f0265341b80 1 --2- 172.21.15.70:0/1928695250 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557f48d62d00 0x557f48dd0350 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.197+0000 7f0265341b80 1 --2- 172.21.15.70:0/1928695250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557f48d49860 0x557f48d5b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.197+0000 7f0265341b80 1 --2- 172.21.15.70:0/1928695250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f48d4a210 0x557f48d5bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.197+0000 7f0233fff700 0 client.18180 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:35.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.197+0000 7f0265341b80 1 -- 172.21.15.70:0/1928695250 >> 172.21.15.70:0/1928695250 conn(0x557f48c77080 msgr2=0x557f48d43530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:35.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.197+0000 7f0265341b80 1 -- 172.21.15.70:0/1928695250 shutdown_connections 2024-06-04T05:07:35.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.197+0000 7f0265341b80 1 -- 172.21.15.70:0/1928695250 wait complete. 2024-06-04T05:07:35.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.197+0000 7f0265341b80 20 client.18180 trim_cache size 0 max 16384 2024-06-04T05:07:35.204 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:07:35.204 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:07:35.204 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:07:35.204 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:07:35.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.198+0000 7f025aaac700 1 -- 172.21.15.70:0/3714594230 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02440b2a70 msgr2=0x7f02440b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:35.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.198+0000 7f025aaac700 1 --2- 172.21.15.70:0/3714594230 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02440b2a70 0x7f02440b4f30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f024000b2e0 tx=0x7f024001a040 comp rx=0 tx=0).stop 2024-06-04T05:07:35.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.198+0000 7f025aaac700 1 -- 172.21.15.70:0/3714594230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f025406c020 msgr2=0x7f02541a0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:35.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.198+0000 7f025aaac700 1 --2- 172.21.15.70:0/3714594230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f025406c020 0x7f02541a0140 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f024800e9a0 tx=0x7f024800ed60 comp rx=0 tx=0).stop 2024-06-04T05:07:35.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.198+0000 7f025aaac700 1 -- 172.21.15.70:0/3714594230 shutdown_connections 2024-06-04T05:07:35.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.198+0000 7f025aaac700 1 --2- 172.21.15.70:0/3714594230 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02440b2a70 0x7f02440b4f30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.198+0000 7f025aaac700 1 --2- 172.21.15.70:0/3714594230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0254108460 0x7f02541a4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.198+0000 7f025aaac700 1 --2- 172.21.15.70:0/3714594230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f025406c020 0x7f02541a0140 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.198+0000 7f025aaac700 1 --2- 172.21.15.70:0/3714594230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0254105f70 0x7f02541a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:35.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.198+0000 7f025aaac700 1 -- 172.21.15.70:0/3714594230 >> 172.21.15.70:0/3714594230 conn(0x7f02540793b0 msgr2=0x7f0254104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:35.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.199+0000 7f025aaac700 1 -- 172.21.15.70:0/3714594230 shutdown_connections 2024-06-04T05:07:35.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:35.199+0000 7f025aaac700 1 -- 172.21.15.70:0/3714594230 wait complete. 2024-06-04T05:07:35.733 INFO:tasks.workunit.client.0.smithi070.stdout: 1 182473 1.90 MB/sec execute 388 sec latency 67.543 ms 2024-06-04T05:07:35.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:35 smithi070 ceph-mon[32631]: pgmap v321: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 227 KiB/s wr, 37 op/s 2024-06-04T05:07:35.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:35 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi098 to 176.1M 2024-06-04T05:07:35.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:35 smithi070 ceph-mon[32631]: Unable to set osd_memory_target on smithi098 to 184732979: error parsing value: Value '184732979' is below minimum 939524096 2024-06-04T05:07:35.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:35 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi112 to 176.1M 2024-06-04T05:07:35.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:35 smithi070 ceph-mon[32631]: Unable to set osd_memory_target on smithi112 to 184710758: error parsing value: Value '184710758' is below minimum 939524096 2024-06-04T05:07:35.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:35 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:35.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:35 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:35 smithi112 ceph-mon[36453]: pgmap v321: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 227 KiB/s wr, 37 op/s 2024-06-04T05:07:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:35 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi098 to 176.1M 2024-06-04T05:07:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:35 smithi112 ceph-mon[36453]: Unable to set osd_memory_target on smithi098 to 184732979: error parsing value: Value '184732979' is below minimum 939524096 2024-06-04T05:07:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:35 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi112 to 176.1M 2024-06-04T05:07:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:35 smithi112 ceph-mon[36453]: Unable to set osd_memory_target on smithi112 to 184710758: error parsing value: Value '184710758' is below minimum 939524096 2024-06-04T05:07:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:35 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:35.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:35 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:35.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:35 smithi098 ceph-mon[35569]: pgmap v321: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 227 KiB/s wr, 37 op/s 2024-06-04T05:07:35.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:35 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi098 to 176.1M 2024-06-04T05:07:35.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:35 smithi098 ceph-mon[35569]: Unable to set osd_memory_target on smithi098 to 184732979: error parsing value: Value '184732979' is below minimum 939524096 2024-06-04T05:07:35.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:35 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi112 to 176.1M 2024-06-04T05:07:35.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:35 smithi098 ceph-mon[35569]: Unable to set osd_memory_target on smithi112 to 184710758: error parsing value: Value '184710758' is below minimum 939524096 2024-06-04T05:07:35.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:35 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:35.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:35 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:36.033 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:07:36.033 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:07:36.318 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:07:36.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:36 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:36.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:36 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:36.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:36 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:36.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:36 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:36.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:36 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:07:36.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:36 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:07:36.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:36 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:36.732 INFO:tasks.workunit.client.0.smithi070.stdout: 1 183008 1.90 MB/sec execute 389 sec latency 19.140 ms 2024-06-04T05:07:36.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:36 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:36.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:36 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:36.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:36 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:36.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:36 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:36.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:36 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:07:36.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:36 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:07:36.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:36 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:36.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:36 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:36.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:36 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:36.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:36 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:36.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:36 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:07:36.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:36 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:07:36.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:36 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:07:36.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:36 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:07:37.733 INFO:tasks.workunit.client.0.smithi070.stdout: 1 183527 1.90 MB/sec execute 390 sec latency 17.272 ms 2024-06-04T05:07:37.811 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:37 smithi070 ceph-mon[32631]: pgmap v322: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 690 KiB/s wr, 89 op/s 2024-06-04T05:07:37.811 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:37 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi070 to 176.1M 2024-06-04T05:07:37.811 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:37 smithi070 ceph-mon[32631]: Unable to set osd_memory_target on smithi070 to 184717209: error parsing value: Value '184717209' is below minimum 939524096 2024-06-04T05:07:37.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:37 smithi112 ceph-mon[36453]: pgmap v322: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 690 KiB/s wr, 89 op/s 2024-06-04T05:07:37.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:37 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi070 to 176.1M 2024-06-04T05:07:37.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:37 smithi112 ceph-mon[36453]: Unable to set osd_memory_target on smithi070 to 184717209: error parsing value: Value '184717209' is below minimum 939524096 2024-06-04T05:07:37.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:37 smithi098 ceph-mon[35569]: pgmap v322: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 690 KiB/s wr, 89 op/s 2024-06-04T05:07:37.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:37 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi070 to 176.1M 2024-06-04T05:07:37.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:37 smithi098 ceph-mon[35569]: Unable to set osd_memory_target on smithi070 to 184717209: error parsing value: Value '184717209' is below minimum 939524096 2024-06-04T05:07:38.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.201+0000 7fd8c705d700 1 -- 172.21.15.70:0/2048799288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8c0108500 msgr2=0x7fd8c01088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:38.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.201+0000 7fd8c705d700 1 --2- 172.21.15.70:0/2048799288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8c0108500 0x7fd8c01088e0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009990 tx=0x7fd8b4009ca0 comp rx=0 tx=0).stop 2024-06-04T05:07:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.201+0000 7fd8c705d700 1 -- 172.21.15.70:0/2048799288 shutdown_connections 2024-06-04T05:07:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.201+0000 7fd8c705d700 1 --2- 172.21.15.70:0/2048799288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c0103f90 0x7fd8c0110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.201+0000 7fd8c705d700 1 --2- 172.21.15.70:0/2048799288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8c0108500 0x7fd8c01088e0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.201+0000 7fd8c705d700 1 --2- 172.21.15.70:0/2048799288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8c01035f0 0x7fd8c0103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.201+0000 7fd8c705d700 1 -- 172.21.15.70:0/2048799288 >> 172.21.15.70:0/2048799288 conn(0x7fd8c007a6a0 msgr2=0x7fd8c007aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.201+0000 7fd8c705d700 1 -- 172.21.15.70:0/2048799288 shutdown_connections 2024-06-04T05:07:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.201+0000 7fd8c705d700 1 -- 172.21.15.70:0/2048799288 wait complete. 2024-06-04T05:07:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.201+0000 7fd8c705d700 1 Processor -- start 2024-06-04T05:07:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8c705d700 1 -- start start 2024-06-04T05:07:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8c705d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c01035f0 0x7fd8c0075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8c705d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8c0103f90 0x7fd8c0075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8c705d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8c0108500 0x7fd8c00762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8c705d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0113650 con 0x7fd8c0103f90 2024-06-04T05:07:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8c705d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01134d0 con 0x7fd8c01035f0 2024-06-04T05:07:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8c705d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01137d0 con 0x7fd8c0108500 2024-06-04T05:07:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8bffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8c0103f90 0x7fd8c0075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8bffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8c0103f90 0x7fd8c0075d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44496/0 (socket says 172.21.15.70:44496) 2024-06-04T05:07:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8bffff700 1 -- 172.21.15.70:0/3459561868 learned_addr learned my addr 172.21.15.70:0/3459561868 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8c4df9700 1 --2- 172.21.15.70:0/3459561868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c01035f0 0x7fd8c0075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8c55fa700 1 --2- 172.21.15.70:0/3459561868 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8c0108500 0x7fd8c00762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8c4df9700 1 -- 172.21.15.70:0/3459561868 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8c0108500 msgr2=0x7fd8c00762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8c4df9700 1 --2- 172.21.15.70:0/3459561868 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8c0108500 0x7fd8c00762c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8c4df9700 1 -- 172.21.15.70:0/3459561868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8c0103f90 msgr2=0x7fd8c0075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8c4df9700 1 --2- 172.21.15.70:0/3459561868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8c0103f90 0x7fd8c0075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8c4df9700 1 -- 172.21.15.70:0/3459561868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c01a80d0 con 0x7fd8c01035f0 2024-06-04T05:07:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.202+0000 7fd8bffff700 1 --2- 172.21.15.70:0/3459561868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8c0103f90 0x7fd8c0075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.203+0000 7fd8c4df9700 1 --2- 172.21.15.70:0/3459561868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c01035f0 0x7fd8c0075840 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4000c00 tx=0x7fd8b40047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.203+0000 7fd8bdffb700 1 -- 172.21.15.70:0/3459561868 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4026070 con 0x7fd8c01035f0 2024-06-04T05:07:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.203+0000 7fd8c705d700 1 -- 172.21.15.70:0/3459561868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c01a8360 con 0x7fd8c01035f0 2024-06-04T05:07:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.203+0000 7fd8bdffb700 1 -- 172.21.15.70:0/3459561868 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8b4021410 con 0x7fd8c01035f0 2024-06-04T05:07:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.203+0000 7fd8bdffb700 1 -- 172.21.15.70:0/3459561868 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4017650 con 0x7fd8c01035f0 2024-06-04T05:07:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.203+0000 7fd8c705d700 1 -- 172.21.15.70:0/3459561868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c01a8810 con 0x7fd8c01035f0 2024-06-04T05:07:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.204+0000 7fd8c705d700 1 -- 172.21.15.70:0/3459561868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c010bb30 con 0x7fd8c01035f0 2024-06-04T05:07:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.206+0000 7fd8bdffb700 1 -- 172.21.15.70:0/3459561868 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fd8b4029070 con 0x7fd8c01035f0 2024-06-04T05:07:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.206+0000 7fd8bdffb700 1 --2- 172.21.15.70:0/3459561868 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd8b00b29f0 0x7fd8b00b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.206+0000 7fd8bffff700 1 --2- 172.21.15.70:0/3459561868 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd8b00b29f0 0x7fd8b00b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.206+0000 7fd8bdffb700 1 -- 172.21.15.70:0/3459561868 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7fd8b40e3bb0 con 0x7fd8c01035f0 2024-06-04T05:07:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.207+0000 7fd8bffff700 1 --2- 172.21.15.70:0/3459561868 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd8b00b29f0 0x7fd8b00b4eb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd8c007ade0 tx=0x7fd8ac009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:38.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.210+0000 7fd8bdffb700 1 -- 172.21.15.70:0/3459561868 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd8b4030070 con 0x7fd8c01035f0 2024-06-04T05:07:38.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.497+0000 7fd8c705d700 1 -- 172.21.15.70:0/3459561868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8c006a230 con 0x7fd8c01035f0 2024-06-04T05:07:38.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.498+0000 7fd8bdffb700 1 -- 172.21.15.70:0/3459561868 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11931 (secure 0 0 0) 0x7fd8b4013070 con 0x7fd8c01035f0 2024-06-04T05:07:38.501 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:07:38.501 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:07:27.814997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24436,24410,24440,24448,24442]},"id":1}]} 2024-06-04T05:07:38.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.501+0000 7fd8c705d700 1 -- 172.21.15.70:0/3459561868 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd8b00b29f0 msgr2=0x7fd8b00b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:38.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.501+0000 7fd8c705d700 1 --2- 172.21.15.70:0/3459561868 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd8b00b29f0 0x7fd8b00b4eb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd8c007ade0 tx=0x7fd8ac009260 comp rx=0 tx=0).stop 2024-06-04T05:07:38.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.501+0000 7fd8c705d700 1 -- 172.21.15.70:0/3459561868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c01035f0 msgr2=0x7fd8c0075840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:38.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.501+0000 7fd8c705d700 1 --2- 172.21.15.70:0/3459561868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c01035f0 0x7fd8c0075840 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4000c00 tx=0x7fd8b40047b0 comp rx=0 tx=0).stop 2024-06-04T05:07:38.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.501+0000 7fd8c705d700 1 -- 172.21.15.70:0/3459561868 shutdown_connections 2024-06-04T05:07:38.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.501+0000 7fd8c705d700 1 --2- 172.21.15.70:0/3459561868 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd8b00b29f0 0x7fd8b00b4eb0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:38.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.501+0000 7fd8c705d700 1 --2- 172.21.15.70:0/3459561868 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8c0108500 0x7fd8c00762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:38.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.501+0000 7fd8c705d700 1 --2- 172.21.15.70:0/3459561868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c01035f0 0x7fd8c0075840 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:38.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.501+0000 7fd8c705d700 1 --2- 172.21.15.70:0/3459561868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8c0103f90 0x7fd8c0075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:38.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.501+0000 7fd8c705d700 1 -- 172.21.15.70:0/3459561868 >> 172.21.15.70:0/3459561868 conn(0x7fd8c007a6a0 msgr2=0x7fd8c0101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:38.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.501+0000 7fd8c705d700 1 -- 172.21.15.70:0/3459561868 shutdown_connections 2024-06-04T05:07:38.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:38.501+0000 7fd8c705d700 1 -- 172.21.15.70:0/3459561868 wait complete. 2024-06-04T05:07:38.505 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 34 2024-06-04T05:07:38.733 INFO:tasks.workunit.client.0.smithi070.stdout: 1 184229 1.91 MB/sec execute 391 sec latency 22.343 ms 2024-06-04T05:07:38.821 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:38 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3459561868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:07:38.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:38 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3459561868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:07:38.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:38 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3459561868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:07:39.261 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:07:39.502 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:07:39.733 INFO:tasks.workunit.client.0.smithi070.stdout: 1 184528 1.92 MB/sec execute 392 sec latency 423.995 ms 2024-06-04T05:07:39.902 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:39 smithi070 ceph-mon[32631]: pgmap v323: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 521 KiB/s wr, 67 op/s 2024-06-04T05:07:39.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:39 smithi112 ceph-mon[36453]: pgmap v323: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 521 KiB/s wr, 67 op/s 2024-06-04T05:07:39.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:39 smithi098 ceph-mon[35569]: pgmap v323: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 521 KiB/s wr, 67 op/s 2024-06-04T05:07:40.733 INFO:tasks.workunit.client.0.smithi070.stdout: 1 184528 1.91 MB/sec execute 393 sec latency 1424.020 ms 2024-06-04T05:07:41.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.263+0000 7ff82d82b700 1 -- 172.21.15.70:0/618857154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8281037f0 msgr2=0x7ff828103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:41.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.263+0000 7ff82d82b700 1 --2- 172.21.15.70:0/618857154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8281037f0 0x7ff828103bd0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7ff8180099c0 tx=0x7ff818009cd0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.263+0000 7ff82d82b700 1 -- 172.21.15.70:0/618857154 shutdown_connections 2024-06-04T05:07:41.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.263+0000 7ff82d82b700 1 --2- 172.21.15.70:0/618857154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8281088f0 0x7ff828110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.263+0000 7ff82d82b700 1 --2- 172.21.15.70:0/618857154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8281037f0 0x7ff828103bd0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.263+0000 7ff82d82b700 1 --2- 172.21.15.70:0/618857154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8281041a0 0x7ff8281081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.263+0000 7ff82d82b700 1 -- 172.21.15.70:0/618857154 >> 172.21.15.70:0/618857154 conn(0x7ff82807a6a0 msgr2=0x7ff82807aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:41.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.263+0000 7ff82d82b700 1 -- 172.21.15.70:0/618857154 shutdown_connections 2024-06-04T05:07:41.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.263+0000 7ff82d82b700 1 -- 172.21.15.70:0/618857154 wait complete. 2024-06-04T05:07:41.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.263+0000 7ff82d82b700 1 Processor -- start 2024-06-04T05:07:41.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff82d82b700 1 -- start start 2024-06-04T05:07:41.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff82d82b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8281037f0 0x7ff82819f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff82d82b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8281041a0 0x7ff82819fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff82d82b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8281088f0 0x7ff828199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff82d82b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff828113570 con 0x7ff8281037f0 2024-06-04T05:07:41.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff82d82b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8281133f0 con 0x7ff8281088f0 2024-06-04T05:07:41.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff82d82b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff8281136f0 con 0x7ff8281041a0 2024-06-04T05:07:41.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff826ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8281037f0 0x7ff82819f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff826ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8281037f0 0x7ff82819f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44522/0 (socket says 172.21.15.70:44522) 2024-06-04T05:07:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff826ffd700 1 -- 172.21.15.70:0/1624721992 learned_addr learned my addr 172.21.15.70:0/1624721992 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff81ffff700 1 --2- 172.21.15.70:0/1624721992 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8281041a0 0x7ff82819fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff8277fe700 1 --2- 172.21.15.70:0/1624721992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8281088f0 0x7ff828199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff826ffd700 1 -- 172.21.15.70:0/1624721992 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8281041a0 msgr2=0x7ff82819fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff826ffd700 1 --2- 172.21.15.70:0/1624721992 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8281041a0 0x7ff82819fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff826ffd700 1 -- 172.21.15.70:0/1624721992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8281088f0 msgr2=0x7ff828199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:41.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff826ffd700 1 --2- 172.21.15.70:0/1624721992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8281088f0 0x7ff828199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff826ffd700 1 -- 172.21.15.70:0/1624721992 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff82819a030 con 0x7ff8281037f0 2024-06-04T05:07:41.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff826ffd700 1 --2- 172.21.15.70:0/1624721992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8281037f0 0x7ff82819f6d0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7ff818000c00 tx=0x7ff8180047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:41.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.264+0000 7ff82c829700 1 -- 172.21.15.70:0/1624721992 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff818004830 con 0x7ff8281037f0 2024-06-04T05:07:41.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.265+0000 7ff82d82b700 1 -- 172.21.15.70:0/1624721992 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff82819a2c0 con 0x7ff8281037f0 2024-06-04T05:07:41.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.265+0000 7ff82c829700 1 -- 172.21.15.70:0/1624721992 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff81800f7f0 con 0x7ff8281037f0 2024-06-04T05:07:41.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.265+0000 7ff82c829700 1 -- 172.21.15.70:0/1624721992 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8180053a0 con 0x7ff8281037f0 2024-06-04T05:07:41.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.265+0000 7ff82d82b700 1 -- 172.21.15.70:0/1624721992 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8281a6530 con 0x7ff8281037f0 2024-06-04T05:07:41.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.267+0000 7ff82c829700 1 -- 172.21.15.70:0/1624721992 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff818029060 con 0x7ff8281037f0 2024-06-04T05:07:41.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.268+0000 7ff82c829700 1 --2- 172.21.15.70:0/1624721992 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8140b29f0 0x7ff8140b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.268+0000 7ff81ffff700 1 --2- 172.21.15.70:0/1624721992 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8140b29f0 0x7ff8140b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.268+0000 7ff82c829700 1 -- 172.21.15.70:0/1624721992 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7ff8180e3090 con 0x7ff8281037f0 2024-06-04T05:07:41.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff81ffff700 1 --2- 172.21.15.70:0/1624721992 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8140b29f0 0x7ff8140b4eb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff828103180 tx=0x7ff81000b410 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:41.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff8380c0b80 1 Processor -- start 2024-06-04T05:07:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff8380c0b80 1 -- start start 2024-06-04T05:07:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff8380c0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24490860 0x55cf24490c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff8380c0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24491210 0x55cf24494940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff8380c0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24494f60 0x55cf244953a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff8380c0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cf243efc90 con 0x55cf24494f60 2024-06-04T05:07:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff8380c0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cf243df830 con 0x55cf24490860 2024-06-04T05:07:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff8380c0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cf243e3870 con 0x55cf24491210 2024-06-04T05:07:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff8277fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24494f60 0x55cf244953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff8277fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24494f60 0x55cf244953a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44524/0 (socket says 172.21.15.70:44524) 2024-06-04T05:07:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff8277fe700 1 -- 172.21.15.70:0/3823653375 learned_addr learned my addr 172.21.15.70:0/3823653375 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff81ffff700 1 --2- 172.21.15.70:0/3823653375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24491210 0x55cf24494940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff826ffd700 1 --2- 172.21.15.70:0/3823653375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24490860 0x55cf24490c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff8277fe700 1 -- 172.21.15.70:0/3823653375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24491210 msgr2=0x55cf24494940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:41.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff8277fe700 1 --2- 172.21.15.70:0/3823653375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24491210 0x55cf24494940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff8277fe700 1 -- 172.21.15.70:0/3823653375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24490860 msgr2=0x55cf24490c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:41.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff8277fe700 1 --2- 172.21.15.70:0/3823653375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24490860 0x55cf24490c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff8277fe700 1 -- 172.21.15.70:0/3823653375 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf2432f8d0 con 0x55cf24494f60 2024-06-04T05:07:41.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.269+0000 7ff81ffff700 1 --2- 172.21.15.70:0/3823653375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24491210 0x55cf24494940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:07:41.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.270+0000 7ff8277fe700 1 --2- 172.21.15.70:0/3823653375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24494f60 0x55cf244953a0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7ff82000f5a0 tx=0x7ff82000f8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e00f4f5a9a21822 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:41.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.270+0000 7ff81dffb700 1 -- 172.21.15.70:0/3823653375 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff820018030 con 0x55cf24494f60 2024-06-04T05:07:41.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.270+0000 7ff81dffb700 1 -- 172.21.15.70:0/3823653375 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff820004bb0 con 0x55cf24494f60 2024-06-04T05:07:41.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.270+0000 7ff81dffb700 1 -- 172.21.15.70:0/3823653375 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff820005020 con 0x55cf24494f60 2024-06-04T05:07:41.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.270+0000 7ff8380c0b80 1 -- 172.21.15.70:0/3823653375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24494f60 msgr2=0x55cf244953a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:41.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.270+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/3823653375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24494f60 0x55cf244953a0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7ff82000f5a0 tx=0x7ff82000f8b0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.271+0000 7ff8380c0b80 1 -- 172.21.15.70:0/3823653375 shutdown_connections 2024-06-04T05:07:41.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.271+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/3823653375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24494f60 0x55cf244953a0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.271+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/3823653375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24490860 0x55cf24490c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.271+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/3823653375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24491210 0x55cf24494940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.271+0000 7ff8380c0b80 1 -- 172.21.15.70:0/3823653375 >> 172.21.15.70:0/3823653375 conn(0x55cf243be080 msgr2=0x55cf2448ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.271+0000 7ff8380c0b80 1 -- 172.21.15.70:0/3823653375 shutdown_connections 2024-06-04T05:07:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.271+0000 7ff8380c0b80 1 -- 172.21.15.70:0/3823653375 wait complete. 2024-06-04T05:07:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.271+0000 7ff8380c0b80 1 -- start start 2024-06-04T05:07:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.271+0000 7ff8380c0b80 1 Processor -- start 2024-06-04T05:07:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.271+0000 7ff8380c0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24490860 0x55cf2449c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.271+0000 7ff8380c0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24491210 0x55cf244996b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.271+0000 7ff8380c0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24494f60 0x55cf24499bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.271+0000 7ff8380c0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cf242e5fa0 con 0x55cf24490860 2024-06-04T05:07:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff8380c0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cf243817c0 con 0x55cf24494f60 2024-06-04T05:07:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff8380c0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cf2449d120 con 0x55cf24491210 2024-06-04T05:07:41.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff8380c0b80 10 client.0 _ll_register_callbacks cb 0x55cf2437a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:41.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff826ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24490860 0x55cf2449c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff826ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24490860 0x55cf2449c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44528/0 (socket says 172.21.15.70:44528) 2024-06-04T05:07:41.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff826ffd700 1 -- 172.21.15.70:0/2865083843 learned_addr learned my addr 172.21.15.70:0/2865083843 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:41.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff81ffff700 1 --2- 172.21.15.70:0/2865083843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24491210 0x55cf244996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff8277fe700 1 --2- 172.21.15.70:0/2865083843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24494f60 0x55cf24499bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff826ffd700 1 -- 172.21.15.70:0/2865083843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24491210 msgr2=0x55cf244996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:41.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff826ffd700 1 --2- 172.21.15.70:0/2865083843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24491210 0x55cf244996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff826ffd700 1 -- 172.21.15.70:0/2865083843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24494f60 msgr2=0x55cf24499bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:41.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff826ffd700 1 --2- 172.21.15.70:0/2865083843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24494f60 0x55cf24499bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff826ffd700 1 -- 172.21.15.70:0/2865083843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf2437e690 con 0x55cf24490860 2024-06-04T05:07:41.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff81ffff700 1 --2- 172.21.15.70:0/2865083843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24491210 0x55cf244996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:41.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff826ffd700 1 --2- 172.21.15.70:0/2865083843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24490860 0x55cf2449c3b0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7ff81800ffd0 tx=0x7ff8180e0c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24a82e03a829c069 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:41.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff806ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:41.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff806ffd700 1 -- 172.21.15.70:0/2865083843 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8180aaa70 con 0x55cf24490860 2024-06-04T05:07:41.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff806ffd700 1 -- 172.21.15.70:0/2865083843 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7ff8180aa7e0 con 0x55cf24490860 2024-06-04T05:07:41.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.272+0000 7ff8380c0b80 1 -- 172.21.15.70:0/2865083843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff820004bb0 con 0x55cf24490860 2024-06-04T05:07:41.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.273+0000 7ff806ffd700 1 -- 172.21.15.70:0/2865083843 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff818016e30 con 0x55cf24490860 2024-06-04T05:07:41.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.273+0000 7ff8380c0b80 10 client.18198 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:41.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.273+0000 7ff8380c0b80 1 -- 172.21.15.70:0/2865083843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cf244abc30 con 0x55cf24490860 2024-06-04T05:07:41.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.273+0000 7ff806ffd700 1 -- 172.21.15.70:0/2865083843 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff820004bb0 con 0x55cf24490860 2024-06-04T05:07:41.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.273+0000 7ff806ffd700 1 -- 172.21.15.70:0/2865083843 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7ff8180c4bd0 con 0x55cf24490860 2024-06-04T05:07:41.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.273+0000 7ff8380c0b80 10 client.18198 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:41.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.273+0000 7ff8380c0b80 10 client.18198 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:07:41.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.273+0000 7ff8380c0b80 20 client.18198 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:41.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.273+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/2865083843 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55cf244ae150 0x55cf244b0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.273+0000 7ff8380c0b80 4 client.18198 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:07:41.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.273+0000 7ff8380c0b80 1 -- 172.21.15.70:0/2865083843 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf2445c9e0 con 0x55cf244ae150 2024-06-04T05:07:41.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.273+0000 7ff81ffff700 1 --2- 172.21.15.70:0/2865083843 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55cf244ae150 0x55cf244b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.274+0000 7ff806ffd700 10 client.18198 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:07:41.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.274+0000 7ff81ffff700 1 --2- 172.21.15.70:0/2865083843 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55cf244ae150 0x55cf244b0580 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2e0ac333510079ea server_cookie=6752cd7b3d1708c8 in_seq=0 out_seq=0 2024-06-04T05:07:41.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.274+0000 7ff806ffd700 1 -- 172.21.15.70:0/2865083843 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cf2445c9e0 con 0x55cf244ae150 2024-06-04T05:07:41.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.274+0000 7ff806ffd700 10 client.18198 handle_command_reply: tid=0 2024-06-04T05:07:41.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.274+0000 7ff8380c0b80 1 client.18198 shutdown 2024-06-04T05:07:41.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.275+0000 7ff8380c0b80 1 -- 172.21.15.70:0/2865083843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24490860 msgr2=0x55cf2449c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:41.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.275+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/2865083843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24490860 0x55cf2449c3b0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7ff81800ffd0 tx=0x7ff8180e0c50 comp rx=0 tx=0).stop 2024-06-04T05:07:41.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.275+0000 7ff8380c0b80 1 -- 172.21.15.70:0/2865083843 shutdown_connections 2024-06-04T05:07:41.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.275+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/2865083843 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55cf244ae150 0x55cf244b0580 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.275+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/2865083843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24494f60 0x55cf24499bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.275+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/2865083843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24490860 0x55cf2449c3b0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.275+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/2865083843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24491210 0x55cf244996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.275+0000 7ff8380c0b80 1 -- 172.21.15.70:0/2865083843 >> 172.21.15.70:0/2865083843 conn(0x55cf243be080 msgr2=0x55cf2448a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:41.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.275+0000 7ff806ffd700 0 client.18198 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:07:41.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.275+0000 7ff8380c0b80 1 -- 172.21.15.70:0/2865083843 shutdown_connections 2024-06-04T05:07:41.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.275+0000 7ff8380c0b80 1 -- 172.21.15.70:0/2865083843 wait complete. 2024-06-04T05:07:41.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.275+0000 7ff8380c0b80 20 client.18198 trim_cache size 0 max 16384 2024-06-04T05:07:41.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.293+0000 7ff8380c0b80 1 Processor -- start 2024-06-04T05:07:41.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.293+0000 7ff8380c0b80 1 -- start start 2024-06-04T05:07:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.293+0000 7ff8380c0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24490860 0x55cf2449b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.293+0000 7ff8380c0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24491210 0x55cf2448ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.293+0000 7ff8380c0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24494f60 0x55cf2448b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.293+0000 7ff8380c0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff8281aba00 con 0x55cf24490860 2024-06-04T05:07:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.293+0000 7ff8380c0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8281ad5a0 con 0x55cf24491210 2024-06-04T05:07:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.293+0000 7ff8380c0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff8281ac0d0 con 0x55cf24494f60 2024-06-04T05:07:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.293+0000 7ff826ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24490860 0x55cf2449b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.293+0000 7ff826ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24490860 0x55cf2449b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44536/0 (socket says 172.21.15.70:44536) 2024-06-04T05:07:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.293+0000 7ff826ffd700 1 -- 172.21.15.70:0/83715415 learned_addr learned my addr 172.21.15.70:0/83715415 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.294+0000 7ff81ffff700 1 --2- 172.21.15.70:0/83715415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24491210 0x55cf2448ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.294+0000 7ff8277fe700 1 --2- 172.21.15.70:0/83715415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24494f60 0x55cf2448b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.294+0000 7ff826ffd700 1 -- 172.21.15.70:0/83715415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24494f60 msgr2=0x55cf2448b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:41.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.294+0000 7ff826ffd700 1 --2- 172.21.15.70:0/83715415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24494f60 0x55cf2448b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.294+0000 7ff826ffd700 1 -- 172.21.15.70:0/83715415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24491210 msgr2=0x55cf2448ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:41.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.294+0000 7ff826ffd700 1 --2- 172.21.15.70:0/83715415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24491210 0x55cf2448ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.294+0000 7ff826ffd700 1 -- 172.21.15.70:0/83715415 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf2448bd40 con 0x55cf24490860 2024-06-04T05:07:41.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.294+0000 7ff81ffff700 1 --2- 172.21.15.70:0/83715415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24491210 0x55cf2448ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:07:41.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.294+0000 7ff826ffd700 1 --2- 172.21.15.70:0/83715415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24490860 0x55cf2449b5f0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7ff818009fd0 tx=0x7ff818009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b58917a1a47470 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:41.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.294+0000 7ff81d7fa700 1 -- 172.21.15.70:0/83715415 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8180a9ae0 con 0x55cf24490860 2024-06-04T05:07:41.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.294+0000 7ff81d7fa700 1 -- 172.21.15.70:0/83715415 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff8180aaa70 con 0x55cf24490860 2024-06-04T05:07:41.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.294+0000 7ff81d7fa700 1 -- 172.21.15.70:0/83715415 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8180049d0 con 0x55cf24490860 2024-06-04T05:07:41.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.295+0000 7ff8380c0b80 1 -- 172.21.15.70:0/83715415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24490860 msgr2=0x55cf2449b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:41.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.295+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/83715415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24490860 0x55cf2449b5f0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7ff818009fd0 tx=0x7ff818009cd0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.295+0000 7ff8380c0b80 1 -- 172.21.15.70:0/83715415 shutdown_connections 2024-06-04T05:07:41.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.295+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/83715415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24494f60 0x55cf2448b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.295+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/83715415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24490860 0x55cf2449b5f0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.295+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/83715415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24491210 0x55cf2448ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.295+0000 7ff8380c0b80 1 -- 172.21.15.70:0/83715415 >> 172.21.15.70:0/83715415 conn(0x55cf243be080 msgr2=0x55cf2449aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:41.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.295+0000 7ff8380c0b80 1 -- 172.21.15.70:0/83715415 shutdown_connections 2024-06-04T05:07:41.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.295+0000 7ff8380c0b80 1 -- 172.21.15.70:0/83715415 wait complete. 2024-06-04T05:07:41.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.295+0000 7ff8380c0b80 1 -- start start 2024-06-04T05:07:41.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.295+0000 7ff8380c0b80 1 Processor -- start 2024-06-04T05:07:41.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.295+0000 7ff8380c0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24490860 0x55cf244a26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.295+0000 7ff8380c0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24491210 0x55cf244a2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.295+0000 7ff8380c0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24494f60 0x55cf2449fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff8380c0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff8281acea0 con 0x55cf24494f60 2024-06-04T05:07:41.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff8380c0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8281ac7a0 con 0x55cf24490860 2024-06-04T05:07:41.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff8380c0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff8281a99c0 con 0x55cf24491210 2024-06-04T05:07:41.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff8380c0b80 10 client.0 _ll_register_callbacks cb 0x55cf2437a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:41.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff8277fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24494f60 0x55cf2449fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff8277fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24494f60 0x55cf2449fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44552/0 (socket says 172.21.15.70:44552) 2024-06-04T05:07:41.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff8277fe700 1 -- 172.21.15.70:0/3963387320 learned_addr learned my addr 172.21.15.70:0/3963387320 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:41.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff826ffd700 1 --2- 172.21.15.70:0/3963387320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24490860 0x55cf244a26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff81ffff700 1 --2- 172.21.15.70:0/3963387320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24491210 0x55cf244a2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff8277fe700 1 -- 172.21.15.70:0/3963387320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24491210 msgr2=0x55cf244a2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:41.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff8277fe700 1 --2- 172.21.15.70:0/3963387320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24491210 0x55cf244a2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff8277fe700 1 -- 172.21.15.70:0/3963387320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24490860 msgr2=0x55cf244a26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:41.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff8277fe700 1 --2- 172.21.15.70:0/3963387320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24490860 0x55cf244a26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff8277fe700 1 -- 172.21.15.70:0/3963387320 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf244a04a0 con 0x55cf24494f60 2024-06-04T05:07:41.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff8277fe700 1 --2- 172.21.15.70:0/3963387320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24494f60 0x55cf2449fc90 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7ff82000f5a0 tx=0x7ff8200146a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b61a93fc3d7bec6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:41.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff806ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:41.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff806ffd700 1 -- 172.21.15.70:0/3963387320 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff820023070 con 0x55cf24494f60 2024-06-04T05:07:41.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff806ffd700 1 -- 172.21.15.70:0/3963387320 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7ff820004040 con 0x55cf24494f60 2024-06-04T05:07:41.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff8380c0b80 1 -- 172.21.15.70:0/3963387320 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8180aaa70 con 0x55cf24494f60 2024-06-04T05:07:41.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff806ffd700 1 -- 172.21.15.70:0/3963387320 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff820004240 con 0x55cf24494f60 2024-06-04T05:07:41.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff8380c0b80 10 client.18204 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:41.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.296+0000 7ff8380c0b80 1 -- 172.21.15.70:0/3963387320 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cf245189b0 con 0x55cf24494f60 2024-06-04T05:07:41.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.297+0000 7ff806ffd700 1 -- 172.21.15.70:0/3963387320 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff8180aaa70 con 0x55cf24494f60 2024-06-04T05:07:41.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.297+0000 7ff806ffd700 1 -- 172.21.15.70:0/3963387320 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7ff82001d7c0 con 0x55cf24494f60 2024-06-04T05:07:41.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.297+0000 7ff8380c0b80 10 client.18204 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:41.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.297+0000 7ff8380c0b80 10 client.18204 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:07:41.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.297+0000 7ff8380c0b80 20 client.18204 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:41.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.297+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/3963387320 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55cf24518de0 0x55cf2451b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:41.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.297+0000 7ff8380c0b80 4 client.18204 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:07:41.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.297+0000 7ff8380c0b80 1 -- 172.21.15.70:0/3963387320 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cf243f1be0 con 0x55cf24518de0 2024-06-04T05:07:41.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.297+0000 7ff826ffd700 1 --2- 172.21.15.70:0/3963387320 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55cf24518de0 0x55cf2451b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:41.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.298+0000 7ff806ffd700 10 client.18204 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:07:41.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.298+0000 7ff826ffd700 1 --2- 172.21.15.70:0/3963387320 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55cf24518de0 0x55cf2451b1d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7900e7c4f0de924f server_cookie=9528dd0b86ef2c5e in_seq=0 out_seq=0 2024-06-04T05:07:41.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.298+0000 7ff806ffd700 1 -- 172.21.15.70:0/3963387320 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cf243f1be0 con 0x55cf24518de0 2024-06-04T05:07:41.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.298+0000 7ff806ffd700 10 client.18204 handle_command_reply: tid=0 2024-06-04T05:07:41.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.298+0000 7ff8380c0b80 1 client.18204 shutdown 2024-06-04T05:07:41.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.298+0000 7ff8380c0b80 1 -- 172.21.15.70:0/3963387320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24494f60 msgr2=0x55cf2449fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:41.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.298+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/3963387320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24494f60 0x55cf2449fc90 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7ff82000f5a0 tx=0x7ff8200146a0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.298+0000 7ff8380c0b80 1 -- 172.21.15.70:0/3963387320 shutdown_connections 2024-06-04T05:07:41.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.298+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/3963387320 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55cf24518de0 0x55cf2451b1d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.298+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/3963387320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf24494f60 0x55cf2449fc90 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.298+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/3963387320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf24490860 0x55cf244a26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.298+0000 7ff8380c0b80 1 --2- 172.21.15.70:0/3963387320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cf24491210 0x55cf244a2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.298+0000 7ff806ffd700 0 client.18204 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:07:41.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.298+0000 7ff8380c0b80 1 -- 172.21.15.70:0/3963387320 >> 172.21.15.70:0/3963387320 conn(0x55cf243be080 msgr2=0x55cf2449b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:41.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.298+0000 7ff8380c0b80 1 -- 172.21.15.70:0/3963387320 shutdown_connections 2024-06-04T05:07:41.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.298+0000 7ff8380c0b80 1 -- 172.21.15.70:0/3963387320 wait complete. 2024-06-04T05:07:41.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.298+0000 7ff8380c0b80 20 client.18204 trim_cache size 0 max 16384 2024-06-04T05:07:41.307 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:07:41.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.299+0000 7ff82d82b700 1 -- 172.21.15.70:0/1624721992 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8140b29f0 msgr2=0x7ff8140b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:41.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.299+0000 7ff82d82b700 1 --2- 172.21.15.70:0/1624721992 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8140b29f0 0x7ff8140b4eb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff828103180 tx=0x7ff81000b410 comp rx=0 tx=0).stop 2024-06-04T05:07:41.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.299+0000 7ff82d82b700 1 -- 172.21.15.70:0/1624721992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8281037f0 msgr2=0x7ff82819f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:41.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.299+0000 7ff82d82b700 1 --2- 172.21.15.70:0/1624721992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8281037f0 0x7ff82819f6d0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7ff818000c00 tx=0x7ff8180047b0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.299+0000 7ff82d82b700 1 -- 172.21.15.70:0/1624721992 shutdown_connections 2024-06-04T05:07:41.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.299+0000 7ff82d82b700 1 --2- 172.21.15.70:0/1624721992 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8140b29f0 0x7ff8140b4eb0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.299+0000 7ff82d82b700 1 --2- 172.21.15.70:0/1624721992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8281088f0 0x7ff828199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.299+0000 7ff82d82b700 1 --2- 172.21.15.70:0/1624721992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8281037f0 0x7ff82819f6d0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.299+0000 7ff82d82b700 1 --2- 172.21.15.70:0/1624721992 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8281041a0 0x7ff82819fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:41.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.299+0000 7ff82d82b700 1 -- 172.21.15.70:0/1624721992 >> 172.21.15.70:0/1624721992 conn(0x7ff82807a6a0 msgr2=0x7ff8281018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:41.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.299+0000 7ff82d82b700 1 -- 172.21.15.70:0/1624721992 shutdown_connections 2024-06-04T05:07:41.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:41.299+0000 7ff82d82b700 1 -- 172.21.15.70:0/1624721992 wait complete. 2024-06-04T05:07:41.733 INFO:tasks.workunit.client.0.smithi070.stdout: 1 184528 1.91 MB/sec execute 394 sec latency 2424.090 ms 2024-06-04T05:07:41.903 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:41 smithi070 ceph-mon[32631]: pgmap v324: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 521 KiB/s wr, 67 op/s 2024-06-04T05:07:41.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:41 smithi112 ceph-mon[36453]: pgmap v324: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 521 KiB/s wr, 67 op/s 2024-06-04T05:07:41.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:41 smithi098 ceph-mon[35569]: pgmap v324: 97 pgs: 97 active+clean; 111 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 521 KiB/s wr, 67 op/s 2024-06-04T05:07:42.062 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:07:42.327 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:07:42.733 INFO:tasks.workunit.client.0.smithi070.stdout: 1 184595 1.90 MB/sec execute 395 sec latency 2736.230 ms 2024-06-04T05:07:43.733 INFO:tasks.workunit.client.0.smithi070.stdout: 1 184595 1.90 MB/sec execute 396 sec latency 1451.759 ms 2024-06-04T05:07:43.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:43 smithi070 ceph-mon[32631]: pgmap v325: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-06-04T05:07:43.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:43 smithi112 ceph-mon[36453]: pgmap v325: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-06-04T05:07:43.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:43 smithi098 ceph-mon[35569]: pgmap v325: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-06-04T05:07:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.089+0000 7fb0ce1eb700 1 -- 172.21.15.70:0/128708003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0c81041a0 msgr2=0x7fb0c81081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.089+0000 7fb0ce1eb700 1 --2- 172.21.15.70:0/128708003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0c81041a0 0x7fb0c81081c0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00099c0 tx=0x7fb0b0009cd0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.090+0000 7fb0ce1eb700 1 -- 172.21.15.70:0/128708003 shutdown_connections 2024-06-04T05:07:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.090+0000 7fb0ce1eb700 1 --2- 172.21.15.70:0/128708003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0c81088f0 0x7fb0c8110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.090+0000 7fb0ce1eb700 1 --2- 172.21.15.70:0/128708003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c81037f0 0x7fb0c8103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.090+0000 7fb0ce1eb700 1 --2- 172.21.15.70:0/128708003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0c81041a0 0x7fb0c81081c0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.090+0000 7fb0ce1eb700 1 -- 172.21.15.70:0/128708003 >> 172.21.15.70:0/128708003 conn(0x7fb0c807a6a0 msgr2=0x7fb0c807aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.090+0000 7fb0ce1eb700 1 -- 172.21.15.70:0/128708003 shutdown_connections 2024-06-04T05:07:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.090+0000 7fb0ce1eb700 1 -- 172.21.15.70:0/128708003 wait complete. 2024-06-04T05:07:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.090+0000 7fb0ce1eb700 1 Processor -- start 2024-06-04T05:07:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.090+0000 7fb0ce1eb700 1 -- start start 2024-06-04T05:07:44.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.090+0000 7fb0ce1eb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0c81037f0 0x7fb0c819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.090+0000 7fb0ce1eb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0c81041a0 0x7fb0c819fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.090+0000 7fb0ce1eb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c81088f0 0x7fb0c81998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.090+0000 7fb0ce1eb700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c8113580 con 0x7fb0c81037f0 2024-06-04T05:07:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.090+0000 7fb0ce1eb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c8113400 con 0x7fb0c81088f0 2024-06-04T05:07:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.090+0000 7fb0ce1eb700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c8113700 con 0x7fb0c81041a0 2024-06-04T05:07:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.091+0000 7fb0c77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0c81037f0 0x7fb0c819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.091+0000 7fb0c77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0c81037f0 0x7fb0c819f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44570/0 (socket says 172.21.15.70:44570) 2024-06-04T05:07:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.091+0000 7fb0c77fe700 1 -- 172.21.15.70:0/3360132574 learned_addr learned my addr 172.21.15.70:0/3360132574 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.091+0000 7fb0bffff700 1 --2- 172.21.15.70:0/3360132574 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0c81041a0 0x7fb0c819fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.091+0000 7fb0c7fff700 1 --2- 172.21.15.70:0/3360132574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c81088f0 0x7fb0c81998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.091+0000 7fb0c77fe700 1 -- 172.21.15.70:0/3360132574 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0c81041a0 msgr2=0x7fb0c819fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.091+0000 7fb0c77fe700 1 --2- 172.21.15.70:0/3360132574 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0c81041a0 0x7fb0c819fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.091+0000 7fb0c77fe700 1 -- 172.21.15.70:0/3360132574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c81088f0 msgr2=0x7fb0c81998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.091+0000 7fb0c77fe700 1 --2- 172.21.15.70:0/3360132574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c81088f0 0x7fb0c81998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.091+0000 7fb0c77fe700 1 -- 172.21.15.70:0/3360132574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c819a000 con 0x7fb0c81037f0 2024-06-04T05:07:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.091+0000 7fb0c7fff700 1 --2- 172.21.15.70:0/3360132574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c81088f0 0x7fb0c81998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:07:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.091+0000 7fb0c77fe700 1 --2- 172.21.15.70:0/3360132574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0c81037f0 0x7fb0c819f6e0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fb0b800e9a0 tx=0x7fb0b800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.091+0000 7fb0c57fa700 1 -- 172.21.15.70:0/3360132574 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b800cbd0 con 0x7fb0c81037f0 2024-06-04T05:07:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.091+0000 7fb0c57fa700 1 -- 172.21.15.70:0/3360132574 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb0b800cd70 con 0x7fb0c81037f0 2024-06-04T05:07:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.091+0000 7fb0ce1eb700 1 -- 172.21.15.70:0/3360132574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c819a2f0 con 0x7fb0c81037f0 2024-06-04T05:07:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.091+0000 7fb0c57fa700 1 -- 172.21.15.70:0/3360132574 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8018680 con 0x7fb0c81037f0 2024-06-04T05:07:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.091+0000 7fb0ce1eb700 1 -- 172.21.15.70:0/3360132574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0c81a65b0 con 0x7fb0c81037f0 2024-06-04T05:07:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.094+0000 7fb0c57fa700 1 -- 172.21.15.70:0/3360132574 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fb0b8010430 con 0x7fb0c81037f0 2024-06-04T05:07:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.094+0000 7fb0c57fa700 1 --2- 172.21.15.70:0/3360132574 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb0b40b29f0 0x7fb0b40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.094+0000 7fb0bffff700 1 --2- 172.21.15.70:0/3360132574 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb0b40b29f0 0x7fb0b40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.094+0000 7fb0c57fa700 1 -- 172.21.15.70:0/3360132574 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7fb0b80e28e0 con 0x7fb0c81037f0 2024-06-04T05:07:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.095+0000 7fb0bffff700 1 --2- 172.21.15.70:0/3360132574 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb0b40b29f0 0x7fb0b40b4eb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8103220 tx=0x7fb0b0005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.095+0000 7fb0d8a80b80 1 Processor -- start 2024-06-04T05:07:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.095+0000 7fb0d8a80b80 1 -- start start 2024-06-04T05:07:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.095+0000 7fb0d8a80b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa6e610 0x557c8aa6e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.095+0000 7fb0d8a80b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6efc0 0x557c8aa68cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.095+0000 7fb0d8a80b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa692d0 0x557c8aa69730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.095+0000 7fb0d8a80b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557c8a9c2d70 con 0x557c8aa6efc0 2024-06-04T05:07:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.095+0000 7fb0d8a80b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557c8a9b2910 con 0x557c8aa6e610 2024-06-04T05:07:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.095+0000 7fb0d8a80b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557c8a9b6950 con 0x557c8aa692d0 2024-06-04T05:07:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.095+0000 7fb0bffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6efc0 0x557c8aa68cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.095+0000 7fb0bffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6efc0 0x557c8aa68cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44582/0 (socket says 172.21.15.70:44582) 2024-06-04T05:07:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.096+0000 7fb0bffff700 1 -- 172.21.15.70:0/4040965687 learned_addr learned my addr 172.21.15.70:0/4040965687 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.096+0000 7fb0c7fff700 1 --2- 172.21.15.70:0/4040965687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa692d0 0x557c8aa69730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.096+0000 7fb0c77fe700 1 --2- 172.21.15.70:0/4040965687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa6e610 0x557c8aa6e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.096+0000 7fb0bffff700 1 -- 172.21.15.70:0/4040965687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa692d0 msgr2=0x557c8aa69730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.096+0000 7fb0bffff700 1 --2- 172.21.15.70:0/4040965687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa692d0 0x557c8aa69730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.096+0000 7fb0bffff700 1 -- 172.21.15.70:0/4040965687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa6e610 msgr2=0x557c8aa6e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.096+0000 7fb0bffff700 1 --2- 172.21.15.70:0/4040965687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa6e610 0x557c8aa6e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.096+0000 7fb0bffff700 1 -- 172.21.15.70:0/4040965687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c8a95bfb0 con 0x557c8aa6efc0 2024-06-04T05:07:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.096+0000 7fb0bffff700 1 --2- 172.21.15.70:0/4040965687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6efc0 0x557c8aa68cb0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0015000 tx=0x7fb0b001a070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61aed65fb43e828c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.096+0000 7fb0bdffb700 1 -- 172.21.15.70:0/4040965687 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0009e60 con 0x557c8aa6efc0 2024-06-04T05:07:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.096+0000 7fb0bdffb700 1 -- 172.21.15.70:0/4040965687 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb0b0003c50 con 0x557c8aa6efc0 2024-06-04T05:07:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.096+0000 7fb0bdffb700 1 -- 172.21.15.70:0/4040965687 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b00238d0 con 0x557c8aa6efc0 2024-06-04T05:07:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.097+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4040965687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6efc0 msgr2=0x557c8aa68cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.097+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/4040965687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6efc0 0x557c8aa68cb0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0015000 tx=0x7fb0b001a070 comp rx=0 tx=0).stop 2024-06-04T05:07:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.097+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4040965687 shutdown_connections 2024-06-04T05:07:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.097+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/4040965687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa692d0 0x557c8aa69730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.097+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/4040965687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa6e610 0x557c8aa6e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.097+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/4040965687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6efc0 0x557c8aa68cb0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.097+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4040965687 >> 172.21.15.70:0/4040965687 conn(0x557c8a990240 msgr2=0x557c8aa5f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:44.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.097+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4040965687 shutdown_connections 2024-06-04T05:07:44.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.097+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4040965687 wait complete. 2024-06-04T05:07:44.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.097+0000 7fb0d8a80b80 1 -- start start 2024-06-04T05:07:44.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.098+0000 7fb0d8a80b80 1 Processor -- start 2024-06-04T05:07:44.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.098+0000 7fb0d8a80b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa692d0 0x557c8aa74d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.098+0000 7fb0d8a80b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6e610 0x557c8aa72090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.098+0000 7fb0d8a80b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa6efc0 0x557c8aa725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.098+0000 7fb0d8a80b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557c8a8b8fa0 con 0x557c8aa6e610 2024-06-04T05:07:44.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.098+0000 7fb0d8a80b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557c8a91ada0 con 0x557c8aa692d0 2024-06-04T05:07:44.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.098+0000 7fb0d8a80b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557c8aa759e0 con 0x557c8aa6efc0 2024-06-04T05:07:44.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.098+0000 7fb0d8a80b80 10 client.0 _ll_register_callbacks cb 0x557c8a94d950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:44.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.098+0000 7fb0bffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6e610 0x557c8aa72090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.098+0000 7fb0bffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6e610 0x557c8aa72090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44588/0 (socket says 172.21.15.70:44588) 2024-06-04T05:07:44.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.098+0000 7fb0c7fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa6efc0 0x557c8aa725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.098+0000 7fb0bffff700 1 -- 172.21.15.70:0/4042181641 learned_addr learned my addr 172.21.15.70:0/4042181641 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:44.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.098+0000 7fb0c77fe700 1 --2- 172.21.15.70:0/4042181641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa692d0 0x557c8aa74d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.099+0000 7fb0c7fff700 1 -- 172.21.15.70:0/4042181641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa692d0 msgr2=0x557c8aa74d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:44.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.099+0000 7fb0c7fff700 1 --2- 172.21.15.70:0/4042181641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa692d0 0x557c8aa74d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.099+0000 7fb0c7fff700 1 -- 172.21.15.70:0/4042181641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6e610 msgr2=0x557c8aa72090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:44.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.099+0000 7fb0c7fff700 1 --2- 172.21.15.70:0/4042181641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6e610 0x557c8aa72090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.099+0000 7fb0c7fff700 1 -- 172.21.15.70:0/4042181641 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c8a9028d0 con 0x557c8aa6efc0 2024-06-04T05:07:44.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.099+0000 7fb0c77fe700 1 --2- 172.21.15.70:0/4042181641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa692d0 0x557c8aa74d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:44.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.099+0000 7fb0bffff700 1 --2- 172.21.15.70:0/4042181641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6e610 0x557c8aa72090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:44.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.099+0000 7fb0c7fff700 1 --2- 172.21.15.70:0/4042181641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa6efc0 0x557c8aa725d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0007a70 tx=0x7fb0c0007aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8127d6b7386605ec server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:44.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.099+0000 7fb0a6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:07:44.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.099+0000 7fb0a6ffd700 1 -- 172.21.15.70:0/4042181641 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c001a070 con 0x557c8aa6efc0 2024-06-04T05:07:44.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.099+0000 7fb0a6ffd700 1 -- 172.21.15.70:0/4042181641 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7fb0c0003c00 con 0x557c8aa6efc0 2024-06-04T05:07:44.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.099+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4042181641 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0b0003c50 con 0x557c8aa6efc0 2024-06-04T05:07:44.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.099+0000 7fb0a6ffd700 1 -- 172.21.15.70:0/4042181641 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb0c0025050 con 0x557c8aa6efc0 2024-06-04T05:07:44.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.099+0000 7fb0d8a80b80 10 client.27424 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:44.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.099+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4042181641 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x557c8a96c760 con 0x557c8aa6efc0 2024-06-04T05:07:44.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.100+0000 7fb0a6ffd700 1 -- 172.21.15.70:0/4042181641 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb0b0003c50 con 0x557c8aa6efc0 2024-06-04T05:07:44.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.100+0000 7fb0a6ffd700 1 -- 172.21.15.70:0/4042181641 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7fb0c0016850 con 0x557c8aa6efc0 2024-06-04T05:07:44.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.100+0000 7fb0d8a80b80 10 client.27424 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:44.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.100+0000 7fb0d8a80b80 10 client.27424 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:07:44.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.100+0000 7fb0d8a80b80 20 client.27424 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:44.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.100+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/4042181641 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x557c8aa825e0 0x557c8aa84a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.100+0000 7fb0d8a80b80 4 client.27424 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:07:44.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.100+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4042181641 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c8a9c4cc0 con 0x557c8aa825e0 2024-06-04T05:07:44.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.100+0000 7fb0c77fe700 1 --2- 172.21.15.70:0/4042181641 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x557c8aa825e0 0x557c8aa84a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.101+0000 7fb0c77fe700 1 --2- 172.21.15.70:0/4042181641 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x557c8aa825e0 0x557c8aa84a00 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d06dcf9ff1004a89 server_cookie=37912156d403e472 in_seq=0 out_seq=0 2024-06-04T05:07:44.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.101+0000 7fb0a6ffd700 10 client.27424 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:07:44.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.101+0000 7fb0a6ffd700 1 -- 172.21.15.70:0/4042181641 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557c8a9c4cc0 con 0x557c8aa825e0 2024-06-04T05:07:44.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.101+0000 7fb0a6ffd700 10 client.27424 handle_command_reply: tid=0 2024-06-04T05:07:44.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.101+0000 7fb0d8a80b80 1 client.27424 shutdown 2024-06-04T05:07:44.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.101+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4042181641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa6efc0 msgr2=0x557c8aa725d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:44.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.101+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/4042181641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa6efc0 0x557c8aa725d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0007a70 tx=0x7fb0c0007aa0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.101+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4042181641 shutdown_connections 2024-06-04T05:07:44.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.101+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/4042181641 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x557c8aa825e0 0x557c8aa84a00 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.101+0000 7fb0a6ffd700 0 client.27424 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:07:44.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.102+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/4042181641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa6efc0 0x557c8aa725d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.102+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/4042181641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa692d0 0x557c8aa74d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.102+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/4042181641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6e610 0x557c8aa72090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.102+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4042181641 >> 172.21.15.70:0/4042181641 conn(0x557c8a990240 msgr2=0x557c8aa5eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:44.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.102+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4042181641 shutdown_connections 2024-06-04T05:07:44.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.102+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4042181641 wait complete. 2024-06-04T05:07:44.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.102+0000 7fb0d8a80b80 20 client.27424 trim_cache size 0 max 16384 2024-06-04T05:07:44.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0d8a80b80 1 Processor -- start 2024-06-04T05:07:44.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0d8a80b80 1 -- start start 2024-06-04T05:07:44.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0d8a80b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa692d0 0x557c8aa71010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0d8a80b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6e610 0x557c8aa6fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0d8a80b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa6efc0 0x557c8aa5f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0d8a80b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c81abc50 con 0x557c8aa6e610 2024-06-04T05:07:44.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0d8a80b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c81ad7f0 con 0x557c8aa6efc0 2024-06-04T05:07:44.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0d8a80b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c81ac320 con 0x557c8aa692d0 2024-06-04T05:07:44.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0bffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6e610 0x557c8aa6fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0bffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6e610 0x557c8aa6fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44600/0 (socket says 172.21.15.70:44600) 2024-06-04T05:07:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0bffff700 1 -- 172.21.15.70:0/4143389060 learned_addr learned my addr 172.21.15.70:0/4143389060 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0c77fe700 1 --2- 172.21.15.70:0/4143389060 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa692d0 0x557c8aa71010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0c7fff700 1 --2- 172.21.15.70:0/4143389060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa6efc0 0x557c8aa5f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0bffff700 1 -- 172.21.15.70:0/4143389060 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa692d0 msgr2=0x557c8aa71010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0bffff700 1 --2- 172.21.15.70:0/4143389060 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa692d0 0x557c8aa71010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0bffff700 1 -- 172.21.15.70:0/4143389060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa6efc0 msgr2=0x557c8aa5f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0bffff700 1 --2- 172.21.15.70:0/4143389060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa6efc0 0x557c8aa5f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.121+0000 7fb0bffff700 1 -- 172.21.15.70:0/4143389060 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c8aa84d70 con 0x557c8aa6e610 2024-06-04T05:07:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.122+0000 7fb0bffff700 1 --2- 172.21.15.70:0/4143389060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6e610 0x557c8aa6fc30 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009ff0 tx=0x7fb0b0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2fba6bd63139e00 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.122+0000 7fb0bd7fa700 1 -- 172.21.15.70:0/4143389060 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b003d070 con 0x557c8aa6e610 2024-06-04T05:07:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.122+0000 7fb0bd7fa700 1 -- 172.21.15.70:0/4143389060 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb0b00238f0 con 0x557c8aa6e610 2024-06-04T05:07:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.122+0000 7fb0bd7fa700 1 -- 172.21.15.70:0/4143389060 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b000bc80 con 0x557c8aa6e610 2024-06-04T05:07:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.122+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4143389060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6e610 msgr2=0x557c8aa6fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.122+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/4143389060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6e610 0x557c8aa6fc30 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009ff0 tx=0x7fb0b0009310 comp rx=0 tx=0).stop 2024-06-04T05:07:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.122+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4143389060 shutdown_connections 2024-06-04T05:07:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.122+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/4143389060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa6efc0 0x557c8aa5f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.122+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/4143389060 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa692d0 0x557c8aa71010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.122+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/4143389060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6e610 0x557c8aa6fc30 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.122+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4143389060 >> 172.21.15.70:0/4143389060 conn(0x557c8a990240 msgr2=0x557c8aa6d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.123+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4143389060 shutdown_connections 2024-06-04T05:07:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.123+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/4143389060 wait complete. 2024-06-04T05:07:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.123+0000 7fb0d8a80b80 1 -- start start 2024-06-04T05:07:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.123+0000 7fb0d8a80b80 1 Processor -- start 2024-06-04T05:07:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.123+0000 7fb0d8a80b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa692d0 0x557c8aa7ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.123+0000 7fb0d8a80b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa6e610 0x557c8aa7b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.123+0000 7fb0d8a80b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6efc0 0x557c8aa89380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.123+0000 7fb0d8a80b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c81ad0f0 con 0x557c8aa6efc0 2024-06-04T05:07:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.123+0000 7fb0d8a80b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c81ac9f0 con 0x557c8aa692d0 2024-06-04T05:07:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.123+0000 7fb0d8a80b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c81a9c30 con 0x557c8aa6e610 2024-06-04T05:07:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.123+0000 7fb0d8a80b80 10 client.0 _ll_register_callbacks cb 0x557c8a94d950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.123+0000 7fb0c7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6efc0 0x557c8aa89380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.123+0000 7fb0c7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6efc0 0x557c8aa89380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44602/0 (socket says 172.21.15.70:44602) 2024-06-04T05:07:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.123+0000 7fb0c7fff700 1 -- 172.21.15.70:0/3373676859 learned_addr learned my addr 172.21.15.70:0/3373676859 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0bffff700 1 --2- 172.21.15.70:0/3373676859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa6e610 0x557c8aa7b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0c77fe700 1 --2- 172.21.15.70:0/3373676859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa692d0 0x557c8aa7ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0c7fff700 1 -- 172.21.15.70:0/3373676859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa6e610 msgr2=0x557c8aa7b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0c7fff700 1 --2- 172.21.15.70:0/3373676859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa6e610 0x557c8aa7b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0c7fff700 1 -- 172.21.15.70:0/3373676859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa692d0 msgr2=0x557c8aa7ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0c7fff700 1 --2- 172.21.15.70:0/3373676859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa692d0 0x557c8aa7ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0c7fff700 1 -- 172.21.15.70:0/3373676859 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c8aa89940 con 0x557c8aa6efc0 2024-06-04T05:07:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0bffff700 1 --2- 172.21.15.70:0/3373676859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa6e610 0x557c8aa7b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:07:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0c77fe700 1 --2- 172.21.15.70:0/3373676859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa692d0 0x557c8aa7ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:07:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0c7fff700 1 --2- 172.21.15.70:0/3373676859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6efc0 0x557c8aa89380 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0016c20 tx=0x7fb0c000aef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df95fb2560e23f2c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0a6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0a6ffd700 1 -- 172.21.15.70:0/3373676859 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c002f020 con 0x557c8aa6efc0 2024-06-04T05:07:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0a6ffd700 1 -- 172.21.15.70:0/3373676859 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7fb0c0016840 con 0x557c8aa6efc0 2024-06-04T05:07:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/3373676859 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0b00238f0 con 0x557c8aa6efc0 2024-06-04T05:07:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0a6ffd700 1 -- 172.21.15.70:0/3373676859 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb0c001a040 con 0x557c8aa6efc0 2024-06-04T05:07:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0d8a80b80 10 client.18222 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/3373676859 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x557c8aaeca40 con 0x557c8aa6efc0 2024-06-04T05:07:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.124+0000 7fb0a6ffd700 1 -- 172.21.15.70:0/3373676859 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb0b00238f0 con 0x557c8aa6efc0 2024-06-04T05:07:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.125+0000 7fb0a6ffd700 1 -- 172.21.15.70:0/3373676859 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7fb0c0015a90 con 0x557c8aa6efc0 2024-06-04T05:07:44.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.125+0000 7fb0d8a80b80 10 client.18222 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:44.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.125+0000 7fb0d8a80b80 10 client.18222 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:07:44.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.125+0000 7fb0d8a80b80 20 client.18222 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:44.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.125+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/3373676859 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x557c8aa825e0 0x557c8aaef180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:44.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.125+0000 7fb0d8a80b80 4 client.18222 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:07:44.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.125+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/3373676859 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557c8a9be4a0 con 0x557c8aa825e0 2024-06-04T05:07:44.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.125+0000 7fb0c77fe700 1 --2- 172.21.15.70:0/3373676859 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x557c8aa825e0 0x557c8aaef180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:44.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.125+0000 7fb0a6ffd700 10 client.18222 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:07:44.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.125+0000 7fb0c77fe700 1 --2- 172.21.15.70:0/3373676859 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x557c8aa825e0 0x557c8aaef180 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=701de9b7b74030ca server_cookie=5c50f91e1ee4039c in_seq=0 out_seq=0 2024-06-04T05:07:44.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.125+0000 7fb0a6ffd700 1 -- 172.21.15.70:0/3373676859 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557c8a9be4a0 con 0x557c8aa825e0 2024-06-04T05:07:44.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.125+0000 7fb0a6ffd700 10 client.18222 handle_command_reply: tid=0 2024-06-04T05:07:44.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.125+0000 7fb0d8a80b80 1 client.18222 shutdown 2024-06-04T05:07:44.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/3373676859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6efc0 msgr2=0x557c8aa89380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:44.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/3373676859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6efc0 0x557c8aa89380 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0016c20 tx=0x7fb0c000aef0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/3373676859 shutdown_connections 2024-06-04T05:07:44.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/3373676859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c8aa6efc0 0x557c8aa89380 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/3373676859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c8aa692d0 0x557c8aa7ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/3373676859 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x557c8aa825e0 0x557c8aaef180 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0d8a80b80 1 --2- 172.21.15.70:0/3373676859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c8aa6e610 0x557c8aa7b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/3373676859 >> 172.21.15.70:0/3373676859 conn(0x557c8a990240 msgr2=0x557c8aa64120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:44.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0a6ffd700 0 client.18222 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:07:44.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/3373676859 shutdown_connections 2024-06-04T05:07:44.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0d8a80b80 1 -- 172.21.15.70:0/3373676859 wait complete. 2024-06-04T05:07:44.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0d8a80b80 20 client.18222 trim_cache size 0 max 16384 2024-06-04T05:07:44.133 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:07:44.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0ce1eb700 1 -- 172.21.15.70:0/3360132574 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb0b40b29f0 msgr2=0x7fb0b40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:44.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0ce1eb700 1 --2- 172.21.15.70:0/3360132574 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb0b40b29f0 0x7fb0b40b4eb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8103220 tx=0x7fb0b0005fb0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0ce1eb700 1 -- 172.21.15.70:0/3360132574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0c81037f0 msgr2=0x7fb0c819f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:44.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0ce1eb700 1 --2- 172.21.15.70:0/3360132574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0c81037f0 0x7fb0c819f6e0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fb0b800e9a0 tx=0x7fb0b800ecb0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0ce1eb700 1 -- 172.21.15.70:0/3360132574 shutdown_connections 2024-06-04T05:07:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0ce1eb700 1 --2- 172.21.15.70:0/3360132574 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb0b40b29f0 0x7fb0b40b4eb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0ce1eb700 1 --2- 172.21.15.70:0/3360132574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c81088f0 0x7fb0c81998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.126+0000 7fb0ce1eb700 1 --2- 172.21.15.70:0/3360132574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0c81037f0 0x7fb0c819f6e0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.127+0000 7fb0ce1eb700 1 --2- 172.21.15.70:0/3360132574 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0c81041a0 0x7fb0c819fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.127+0000 7fb0ce1eb700 1 -- 172.21.15.70:0/3360132574 >> 172.21.15.70:0/3360132574 conn(0x7fb0c807a6a0 msgr2=0x7fb0c81019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.127+0000 7fb0ce1eb700 1 -- 172.21.15.70:0/3360132574 shutdown_connections 2024-06-04T05:07:44.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:44.127+0000 7fb0ce1eb700 1 -- 172.21.15.70:0/3360132574 wait complete. 2024-06-04T05:07:44.733 INFO:tasks.workunit.client.0.smithi070.stdout: 1 184595 1.89 MB/sec execute 397 sec latency 2451.785 ms 2024-06-04T05:07:44.842 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:07:45.081 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:07:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.145+0000 7f1882c07640 1 -- 172.21.15.70:0/1692527014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f187c101ad0 msgr2=0x7f187c101eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.145+0000 7f1882c07640 1 --2- 172.21.15.70:0/1692527014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f187c101ad0 0x7f187c101eb0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f1870009900 tx=0x7f187002a510 comp rx=0 tx=0).stop 2024-06-04T05:07:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.145+0000 7f1882c07640 1 -- 172.21.15.70:0/1692527014 shutdown_connections 2024-06-04T05:07:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.145+0000 7f1882c07640 1 --2- 172.21.15.70:0/1692527014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f187c1023f0 0x7f187c1068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.145+0000 7f1882c07640 1 --2- 172.21.15.70:0/1692527014 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f187c106df0 0x7f187c107280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.145+0000 7f1882c07640 1 --2- 172.21.15.70:0/1692527014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f187c101ad0 0x7f187c101eb0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.145+0000 7f1882c07640 1 -- 172.21.15.70:0/1692527014 >> 172.21.15.70:0/1692527014 conn(0x7f187c078ad0 msgr2=0x7f187c078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.145+0000 7f1882c07640 1 -- 172.21.15.70:0/1692527014 shutdown_connections 2024-06-04T05:07:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.145+0000 7f1882c07640 1 -- 172.21.15.70:0/1692527014 wait complete. 2024-06-04T05:07:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.145+0000 7f1882c07640 1 Processor -- start 2024-06-04T05:07:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.145+0000 7f1882c07640 1 -- start start 2024-06-04T05:07:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.145+0000 7f1882c07640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f187c101ad0 0x7f187c19f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.145+0000 7f1882c07640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f187c1023f0 0x7f187c19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.145+0000 7f1882c07640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f187c106df0 0x7f187c1a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.145+0000 7f1882c07640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f187c119170 con 0x7f187c101ad0 2024-06-04T05:07:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.145+0000 7f1882c07640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f187c118ff0 con 0x7f187c106df0 2024-06-04T05:07:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.145+0000 7f1882c07640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f187c1192f0 con 0x7f187c1023f0 2024-06-04T05:07:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.146+0000 7f18809a3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f187c101ad0 0x7f187c19f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.146+0000 7f18809a3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f187c101ad0 0x7f187c19f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44610/0 (socket says 172.21.15.70:44610) 2024-06-04T05:07:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.146+0000 7f18809a3640 1 -- 172.21.15.70:0/913988316 learned_addr learned my addr 172.21.15.70:0/913988316 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.146+0000 7f187bfff640 1 --2- 172.21.15.70:0/913988316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f187c1023f0 0x7f187c19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.146+0000 7f18809a3640 1 -- 172.21.15.70:0/913988316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f187c1023f0 msgr2=0x7f187c19fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.146+0000 7f18809a3640 1 --2- 172.21.15.70:0/913988316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f187c1023f0 0x7f187c19fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.146+0000 7f18809a3640 1 -- 172.21.15.70:0/913988316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f187c106df0 msgr2=0x7f187c1a3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.146+0000 7f18809a3640 1 --2- 172.21.15.70:0/913988316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f187c106df0 0x7f187c1a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.146+0000 7f18809a3640 1 -- 172.21.15.70:0/913988316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f187c1a46a0 con 0x7f187c101ad0 2024-06-04T05:07:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.146+0000 7f18809a3640 1 --2- 172.21.15.70:0/913988316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f187c101ad0 0x7f187c19f6b0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f1870009230 tx=0x7f1870004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.146+0000 7f1879ffb640 1 -- 172.21.15.70:0/913988316 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f187003f030 con 0x7f187c101ad0 2024-06-04T05:07:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.146+0000 7f1879ffb640 1 -- 172.21.15.70:0/913988316 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1870003a80 con 0x7f187c101ad0 2024-06-04T05:07:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.146+0000 7f1882c07640 1 -- 172.21.15.70:0/913988316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f187c1a4930 con 0x7f187c101ad0 2024-06-04T05:07:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.146+0000 7f1879ffb640 1 -- 172.21.15.70:0/913988316 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1870035070 con 0x7f187c101ad0 2024-06-04T05:07:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.146+0000 7f1882c07640 1 -- 172.21.15.70:0/913988316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f187c1ac220 con 0x7f187c101ad0 2024-06-04T05:07:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.149+0000 7f1879ffb640 1 -- 172.21.15.70:0/913988316 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f1870042030 con 0x7f187c101ad0 2024-06-04T05:07:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.149+0000 7f18859f6740 1 Processor -- start 2024-06-04T05:07:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.149+0000 7f18859f6740 1 -- start start 2024-06-04T05:07:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.149+0000 7f18859f6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d730b820 0x5592d730bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.149+0000 7f18859f6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730c140 0x5592d73164c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.149+0000 7f18859f6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d7316a00 0x5592d7318df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.149+0000 7f18859f6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5592d727dc30 con 0x5592d730b820 2024-06-04T05:07:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.149+0000 7f18859f6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5592d726d7d0 con 0x5592d7316a00 2024-06-04T05:07:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.149+0000 7f18859f6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5592d7271810 con 0x5592d730c140 2024-06-04T05:07:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.149+0000 7f18811a4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d730b820 0x5592d730bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.149+0000 7f18811a4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d730b820 0x5592d730bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44614/0 (socket says 172.21.15.70:44614) 2024-06-04T05:07:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.149+0000 7f18811a4640 1 -- 172.21.15.70:0/449686444 learned_addr learned my addr 172.21.15.70:0/449686444 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.149+0000 7f187bfff640 1 --2- 172.21.15.70:0/449686444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d7316a00 0x5592d7318df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.149+0000 7f18809a3640 1 --2- 172.21.15.70:0/449686444 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730c140 0x5592d73164c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.149+0000 7f1879ffb640 1 --2- 172.21.15.70:0/913988316 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f186c0b2a10 0x7f186c0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.150+0000 7f18811a4640 1 -- 172.21.15.70:0/449686444 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730c140 msgr2=0x5592d73164c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.150+0000 7f18811a4640 1 --2- 172.21.15.70:0/449686444 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730c140 0x5592d73164c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.150+0000 7f18811a4640 1 -- 172.21.15.70:0/449686444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d7316a00 msgr2=0x5592d7318df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.150+0000 7f18811a4640 1 --2- 172.21.15.70:0/449686444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d7316a00 0x5592d7318df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.150+0000 7f18811a4640 1 -- 172.21.15.70:0/449686444 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592d6f82210 con 0x5592d730b820 2024-06-04T05:07:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.150+0000 7f1879ffb640 1 -- 172.21.15.70:0/913988316 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f1870102e80 con 0x7f187c101ad0 2024-06-04T05:07:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.150+0000 7f18811a4640 1 --2- 172.21.15.70:0/913988316 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f186c0b2a10 0x7f186c0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.150+0000 7f18811a4640 1 --2- 172.21.15.70:0/449686444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d730b820 0x5592d730bc00 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f1874009e90 tx=0x7f187400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d4315b6c996947 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.150+0000 7f1866ffd640 1 -- 172.21.15.70:0/449686444 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f187400ee10 con 0x5592d730b820 2024-06-04T05:07:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.150+0000 7f18811a4640 1 --2- 172.21.15.70:0/913988316 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f186c0b2a10 0x7f186c0b4ed0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f187401f000 tx=0x7f1874020000 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.150+0000 7f1866ffd640 1 -- 172.21.15.70:0/449686444 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f187401b9e0 con 0x5592d730b820 2024-06-04T05:07:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.150+0000 7f1866ffd640 1 -- 172.21.15.70:0/449686444 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18740118f0 con 0x5592d730b820 2024-06-04T05:07:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.151+0000 7f18859f6740 1 -- 172.21.15.70:0/449686444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d730b820 msgr2=0x5592d730bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.151+0000 7f18859f6740 1 --2- 172.21.15.70:0/449686444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d730b820 0x5592d730bc00 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f1874009e90 tx=0x7f187400e6c0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.151+0000 7f18859f6740 1 -- 172.21.15.70:0/449686444 shutdown_connections 2024-06-04T05:07:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.151+0000 7f18859f6740 1 --2- 172.21.15.70:0/449686444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d7316a00 0x5592d7318df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.151+0000 7f18859f6740 1 --2- 172.21.15.70:0/449686444 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730c140 0x5592d73164c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.151+0000 7f18859f6740 1 --2- 172.21.15.70:0/449686444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d730b820 0x5592d730bc00 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.151+0000 7f18859f6740 1 -- 172.21.15.70:0/449686444 >> 172.21.15.70:0/449686444 conn(0x5592d7307d30 msgr2=0x5592d7308110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.151+0000 7f18859f6740 1 -- 172.21.15.70:0/449686444 shutdown_connections 2024-06-04T05:07:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.151+0000 7f18859f6740 1 -- 172.21.15.70:0/449686444 wait complete. 2024-06-04T05:07:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.151+0000 7f18859f6740 1 -- start start 2024-06-04T05:07:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.151+0000 7f18859f6740 1 Processor -- start 2024-06-04T05:07:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f18859f6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730b820 0x5592d73139a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f18859f6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d730c140 0x5592d7313ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f18859f6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d7316a00 0x5592d7314420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f18859f6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5592d6f53790 con 0x5592d7316a00 2024-06-04T05:07:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f18859f6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5592d7229890 con 0x5592d730c140 2024-06-04T05:07:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f18859f6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5592d7319630 con 0x5592d730b820 2024-06-04T05:07:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f18859f6740 10 client.0 _ll_register_callbacks cb 0x5592d6f81a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f18811a4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d7316a00 0x5592d7314420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f18811a4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d7316a00 0x5592d7314420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44626/0 (socket says 172.21.15.70:44626) 2024-06-04T05:07:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f18811a4640 1 -- 172.21.15.70:0/4019502782 learned_addr learned my addr 172.21.15.70:0/4019502782 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f187bfff640 1 --2- 172.21.15.70:0/4019502782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d730c140 0x5592d7313ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f18809a3640 1 --2- 172.21.15.70:0/4019502782 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730b820 0x5592d73139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f18811a4640 1 -- 172.21.15.70:0/4019502782 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730b820 msgr2=0x5592d73139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f18811a4640 1 --2- 172.21.15.70:0/4019502782 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730b820 0x5592d73139a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f18811a4640 1 -- 172.21.15.70:0/4019502782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d730c140 msgr2=0x5592d7313ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f18811a4640 1 --2- 172.21.15.70:0/4019502782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d730c140 0x5592d7313ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f18811a4640 1 -- 172.21.15.70:0/4019502782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592d704e620 con 0x5592d7316a00 2024-06-04T05:07:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f187bfff640 1 --2- 172.21.15.70:0/4019502782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d730c140 0x5592d7313ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f18811a4640 1 --2- 172.21.15.70:0/4019502782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d7316a00 0x5592d7314420 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f1874005c10 tx=0x7f187401b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa0770cc94e31337 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f185bfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.152+0000 7f185bfff640 1 -- 172.21.15.70:0/4019502782 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1874033060 con 0x5592d7316a00 2024-06-04T05:07:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.153+0000 7f185bfff640 1 -- 172.21.15.70:0/4019502782 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f1874021050 con 0x5592d7316a00 2024-06-04T05:07:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.153+0000 7f18859f6740 1 -- 172.21.15.70:0/4019502782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f187401b9e0 con 0x5592d7316a00 2024-06-04T05:07:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.153+0000 7f185bfff640 1 -- 172.21.15.70:0/4019502782 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f187400ee10 con 0x5592d7316a00 2024-06-04T05:07:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.153+0000 7f18859f6740 10 client.18234 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.153+0000 7f18859f6740 1 -- 172.21.15.70:0/4019502782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5592d73253e0 con 0x5592d7316a00 2024-06-04T05:07:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.153+0000 7f185bfff640 1 -- 172.21.15.70:0/4019502782 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f187401b9e0 con 0x5592d7316a00 2024-06-04T05:07:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.153+0000 7f185bfff640 1 -- 172.21.15.70:0/4019502782 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f187400f040 con 0x5592d7316a00 2024-06-04T05:07:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.153+0000 7f18859f6740 10 client.18234 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.153+0000 7f18859f6740 10 client.18234 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.153+0000 7f18859f6740 20 client.18234 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.153+0000 7f18859f6740 1 --2- 172.21.15.70:0/4019502782 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5592d7326840 0x5592d7328c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.153+0000 7f18859f6740 4 client.18234 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:07:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.153+0000 7f18859f6740 1 -- 172.21.15.70:0/4019502782 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592d727fb80 con 0x5592d7326840 2024-06-04T05:07:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.153+0000 7f18809a3640 1 --2- 172.21.15.70:0/4019502782 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5592d7326840 0x5592d7328c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.154+0000 7f185bfff640 10 client.18234 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.154+0000 7f18809a3640 1 --2- 172.21.15.70:0/4019502782 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5592d7326840 0x5592d7328c30 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=903a19066b9beedd server_cookie=1062ae75bbcce0ec in_seq=0 out_seq=0 2024-06-04T05:07:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.154+0000 7f185bfff640 1 -- 172.21.15.70:0/4019502782 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5592d727fb80 con 0x5592d7326840 2024-06-04T05:07:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.154+0000 7f185bfff640 10 client.18234 handle_command_reply: tid=0 2024-06-04T05:07:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.154+0000 7f18859f6740 1 client.18234 shutdown 2024-06-04T05:07:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.155+0000 7f18859f6740 1 -- 172.21.15.70:0/4019502782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d7316a00 msgr2=0x5592d7314420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.155+0000 7f18859f6740 1 --2- 172.21.15.70:0/4019502782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d7316a00 0x5592d7314420 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f1874005c10 tx=0x7f187401b780 comp rx=0 tx=0).stop 2024-06-04T05:07:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.155+0000 7f18859f6740 1 -- 172.21.15.70:0/4019502782 shutdown_connections 2024-06-04T05:07:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.155+0000 7f18859f6740 1 --2- 172.21.15.70:0/4019502782 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5592d7326840 0x5592d7328c30 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.155+0000 7f18859f6740 1 --2- 172.21.15.70:0/4019502782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d730c140 0x5592d7313ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.155+0000 7f18859f6740 1 --2- 172.21.15.70:0/4019502782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d7316a00 0x5592d7314420 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.155+0000 7f18859f6740 1 --2- 172.21.15.70:0/4019502782 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730b820 0x5592d73139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.155+0000 7f185bfff640 0 client.18234 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.155+0000 7f18859f6740 1 -- 172.21.15.70:0/4019502782 >> 172.21.15.70:0/4019502782 conn(0x5592d7307d30 msgr2=0x5592d7306fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.155+0000 7f18859f6740 1 -- 172.21.15.70:0/4019502782 shutdown_connections 2024-06-04T05:07:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.155+0000 7f18859f6740 1 -- 172.21.15.70:0/4019502782 wait complete. 2024-06-04T05:07:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.155+0000 7f18859f6740 20 client.18234 trim_cache size 0 max 16384 2024-06-04T05:07:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18859f6740 1 Processor -- start 2024-06-04T05:07:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18859f6740 1 -- start start 2024-06-04T05:07:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18859f6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d730b820 0x5592d7314430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18859f6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730c140 0x5592d730e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18859f6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d7316a00 0x5592d732e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18859f6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f187c1b4c60 con 0x5592d730b820 2024-06-04T05:07:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18859f6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f187c1b6800 con 0x5592d7316a00 2024-06-04T05:07:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18859f6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f187c1b5330 con 0x5592d730c140 2024-06-04T05:07:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18809a3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d730b820 0x5592d7314430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18809a3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d730b820 0x5592d7314430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44642/0 (socket says 172.21.15.70:44642) 2024-06-04T05:07:45.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18809a3640 1 -- 172.21.15.70:0/1218175671 learned_addr learned my addr 172.21.15.70:0/1218175671 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:45.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18809a3640 1 -- 172.21.15.70:0/1218175671 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730c140 msgr2=0x5592d730e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:45.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18809a3640 1 --2- 172.21.15.70:0/1218175671 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730c140 0x5592d730e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18809a3640 1 -- 172.21.15.70:0/1218175671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d7316a00 msgr2=0x5592d732e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:45.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18809a3640 1 --2- 172.21.15.70:0/1218175671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d7316a00 0x5592d732e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18809a3640 1 -- 172.21.15.70:0/1218175671 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592d730cab0 con 0x5592d730b820 2024-06-04T05:07:45.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18809a3640 1 --2- 172.21.15.70:0/1218175671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d730b820 0x5592d7314430 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f18700c9e30 tx=0x7f18700c9d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3486828683ad478e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:45.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18667fc640 1 -- 172.21.15.70:0/1218175671 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1870049040 con 0x5592d730b820 2024-06-04T05:07:45.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.170+0000 7f18667fc640 1 -- 172.21.15.70:0/1218175671 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f18700ce4e0 con 0x5592d730b820 2024-06-04T05:07:45.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.171+0000 7f18667fc640 1 -- 172.21.15.70:0/1218175671 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18700c63c0 con 0x5592d730b820 2024-06-04T05:07:45.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.171+0000 7f18859f6740 1 -- 172.21.15.70:0/1218175671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d730b820 msgr2=0x5592d7314430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:45.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.171+0000 7f18859f6740 1 --2- 172.21.15.70:0/1218175671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d730b820 0x5592d7314430 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f18700c9e30 tx=0x7f18700c9d00 comp rx=0 tx=0).stop 2024-06-04T05:07:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.171+0000 7f18859f6740 1 -- 172.21.15.70:0/1218175671 shutdown_connections 2024-06-04T05:07:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.171+0000 7f18859f6740 1 --2- 172.21.15.70:0/1218175671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d7316a00 0x5592d732e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.171+0000 7f18859f6740 1 --2- 172.21.15.70:0/1218175671 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730c140 0x5592d730e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.171+0000 7f18859f6740 1 --2- 172.21.15.70:0/1218175671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d730b820 0x5592d7314430 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.171+0000 7f18859f6740 1 -- 172.21.15.70:0/1218175671 >> 172.21.15.70:0/1218175671 conn(0x5592d7307d30 msgr2=0x5592d7306fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.171+0000 7f18859f6740 1 -- 172.21.15.70:0/1218175671 shutdown_connections 2024-06-04T05:07:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.171+0000 7f18859f6740 1 -- 172.21.15.70:0/1218175671 wait complete. 2024-06-04T05:07:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.171+0000 7f18859f6740 1 -- start start 2024-06-04T05:07:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.171+0000 7f18859f6740 1 Processor -- start 2024-06-04T05:07:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f18859f6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730b820 0x5592d731f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f18859f6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d730c140 0x5592d731a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f18859f6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d7316a00 0x5592d731abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f18859f6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f187c1b6100 con 0x5592d7316a00 2024-06-04T05:07:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f18859f6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f187c1b5a00 con 0x5592d730c140 2024-06-04T05:07:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f18859f6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f187c1b2c60 con 0x5592d730b820 2024-06-04T05:07:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f18859f6740 10 client.0 _ll_register_callbacks cb 0x7f187401e020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f18811a4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d7316a00 0x5592d731abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f18811a4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d7316a00 0x5592d731abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44648/0 (socket says 172.21.15.70:44648) 2024-06-04T05:07:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f18811a4640 1 -- 172.21.15.70:0/412348833 learned_addr learned my addr 172.21.15.70:0/412348833 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f18809a3640 1 --2- 172.21.15.70:0/412348833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730b820 0x5592d731f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f187bfff640 1 --2- 172.21.15.70:0/412348833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d730c140 0x5592d731a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f18811a4640 1 -- 172.21.15.70:0/412348833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730b820 msgr2=0x5592d731f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f18811a4640 1 --2- 172.21.15.70:0/412348833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730b820 0x5592d731f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f18811a4640 1 -- 172.21.15.70:0/412348833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d730c140 msgr2=0x5592d731a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f18811a4640 1 --2- 172.21.15.70:0/412348833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d730c140 0x5592d731a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f18811a4640 1 -- 172.21.15.70:0/412348833 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592d7390190 con 0x5592d7316a00 2024-06-04T05:07:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f18811a4640 1 --2- 172.21.15.70:0/412348833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d7316a00 0x5592d731abd0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f1874025820 tx=0x7f1874009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff8dfc6b0739d1d9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.172+0000 7f185bfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.173+0000 7f185bfff640 1 -- 172.21.15.70:0/412348833 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1874033040 con 0x5592d7316a00 2024-06-04T05:07:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.173+0000 7f18859f6740 1 -- 172.21.15.70:0/412348833 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18700ce4e0 con 0x5592d7316a00 2024-06-04T05:07:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.173+0000 7f185bfff640 1 -- 172.21.15.70:0/412348833 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f187402b040 con 0x5592d7316a00 2024-06-04T05:07:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.173+0000 7f185bfff640 1 -- 172.21.15.70:0/412348833 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f1874046050 con 0x5592d7316a00 2024-06-04T05:07:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.173+0000 7f18859f6740 10 client.18240 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.173+0000 7f18859f6740 1 -- 172.21.15.70:0/412348833 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5592d7391360 con 0x5592d7316a00 2024-06-04T05:07:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.173+0000 7f185bfff640 1 -- 172.21.15.70:0/412348833 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f18700ce4e0 con 0x5592d7316a00 2024-06-04T05:07:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.173+0000 7f185bfff640 1 -- 172.21.15.70:0/412348833 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f187401b760 con 0x5592d7316a00 2024-06-04T05:07:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.173+0000 7f18859f6740 10 client.18240 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.173+0000 7f18859f6740 10 client.18240 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.173+0000 7f18859f6740 20 client.18240 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:45.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.173+0000 7f18859f6740 1 --2- 172.21.15.70:0/412348833 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5592d73919a0 0x5592d7393dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:45.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.173+0000 7f18859f6740 4 client.18240 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:07:45.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.173+0000 7f18859f6740 1 -- 172.21.15.70:0/412348833 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5592d7279360 con 0x5592d73919a0 2024-06-04T05:07:45.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.173+0000 7f18809a3640 1 --2- 172.21.15.70:0/412348833 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5592d73919a0 0x5592d7393dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:45.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.174+0000 7f185bfff640 10 client.18240 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:45.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.174+0000 7f18809a3640 1 --2- 172.21.15.70:0/412348833 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5592d73919a0 0x5592d7393dc0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0ea9978d05c5cb4 server_cookie=3398b0130c3d56b7 in_seq=0 out_seq=0 2024-06-04T05:07:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.203+0000 7f185bfff640 1 -- 172.21.15.70:0/412348833 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5592d7279360 con 0x5592d73919a0 2024-06-04T05:07:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.203+0000 7f185bfff640 10 client.18240 handle_command_reply: tid=0 2024-06-04T05:07:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.203+0000 7f18859f6740 1 client.18240 shutdown 2024-06-04T05:07:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.203+0000 7f18859f6740 1 -- 172.21.15.70:0/412348833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d7316a00 msgr2=0x5592d731abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.203+0000 7f18859f6740 1 --2- 172.21.15.70:0/412348833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d7316a00 0x5592d731abd0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f1874025820 tx=0x7f1874009550 comp rx=0 tx=0).stop 2024-06-04T05:07:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.203+0000 7f18859f6740 1 -- 172.21.15.70:0/412348833 shutdown_connections 2024-06-04T05:07:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.203+0000 7f18859f6740 1 --2- 172.21.15.70:0/412348833 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5592d73919a0 0x5592d7393dc0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.203+0000 7f18859f6740 1 --2- 172.21.15.70:0/412348833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5592d730c140 0x5592d731a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.203+0000 7f185bfff640 0 client.18240 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.203+0000 7f18859f6740 1 --2- 172.21.15.70:0/412348833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592d7316a00 0x5592d731abd0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.203+0000 7f18859f6740 1 --2- 172.21.15.70:0/412348833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5592d730b820 0x5592d731f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.203+0000 7f18859f6740 1 -- 172.21.15.70:0/412348833 >> 172.21.15.70:0/412348833 conn(0x5592d7307d30 msgr2=0x5592d7321680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.204+0000 7f18859f6740 1 -- 172.21.15.70:0/412348833 shutdown_connections 2024-06-04T05:07:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.204+0000 7f18859f6740 1 -- 172.21.15.70:0/412348833 wait complete. 2024-06-04T05:07:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.204+0000 7f18859f6740 20 client.18240 trim_cache size 0 max 16384 2024-06-04T05:07:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:07:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:07:45.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:07:45.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.204+0000 7f1882c07640 1 -- 172.21.15.70:0/913988316 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f186c0b2a10 msgr2=0x7f186c0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:45.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.204+0000 7f1882c07640 1 --2- 172.21.15.70:0/913988316 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f186c0b2a10 0x7f186c0b4ed0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f187401f000 tx=0x7f1874020000 comp rx=0 tx=0).stop 2024-06-04T05:07:45.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.204+0000 7f1882c07640 1 -- 172.21.15.70:0/913988316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f187c101ad0 msgr2=0x7f187c19f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.204+0000 7f1882c07640 1 --2- 172.21.15.70:0/913988316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f187c101ad0 0x7f187c19f6b0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f1870009230 tx=0x7f1870004270 comp rx=0 tx=0).stop 2024-06-04T05:07:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.204+0000 7f1882c07640 1 -- 172.21.15.70:0/913988316 shutdown_connections 2024-06-04T05:07:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.204+0000 7f1882c07640 1 --2- 172.21.15.70:0/913988316 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f186c0b2a10 0x7f186c0b4ed0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.204+0000 7f1882c07640 1 --2- 172.21.15.70:0/913988316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f187c106df0 0x7f187c1a3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.204+0000 7f1882c07640 1 --2- 172.21.15.70:0/913988316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f187c1023f0 0x7f187c19fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.204+0000 7f1882c07640 1 --2- 172.21.15.70:0/913988316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f187c101ad0 0x7f187c19f6b0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.204+0000 7f1882c07640 1 -- 172.21.15.70:0/913988316 >> 172.21.15.70:0/913988316 conn(0x7f187c078ad0 msgr2=0x7f187c104880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.204+0000 7f1882c07640 1 -- 172.21.15.70:0/913988316 shutdown_connections 2024-06-04T05:07:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:45.204+0000 7f1882c07640 1 -- 172.21.15.70:0/913988316 wait complete. 2024-06-04T05:07:45.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 184595 1.89 MB/sec execute 398 sec latency 3451.860 ms 2024-06-04T05:07:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:45 smithi070 ceph-mon[32631]: pgmap v326: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 989 KiB/s wr, 92 op/s 2024-06-04T05:07:45.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:45 smithi112 ceph-mon[36453]: pgmap v326: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 989 KiB/s wr, 92 op/s 2024-06-04T05:07:45.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:45 smithi098 ceph-mon[35569]: pgmap v326: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 989 KiB/s wr, 92 op/s 2024-06-04T05:07:46.733 INFO:tasks.workunit.client.0.smithi070.stdout: 1 184595 1.88 MB/sec execute 399 sec latency 4451.924 ms 2024-06-04T05:07:46.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.828+0000 7f80501f4700 1 -- 172.21.15.70:0/4288062929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80481084b0 msgr2=0x7f8048108890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:46.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.828+0000 7f80501f4700 1 --2- 172.21.15.70:0/4288062929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80481084b0 0x7f8048108890 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f8044009990 tx=0x7f8044009ca0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.828+0000 7f80501f4700 1 -- 172.21.15.70:0/4288062929 shutdown_connections 2024-06-04T05:07:46.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.828+0000 7f80501f4700 1 --2- 172.21.15.70:0/4288062929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8048102e30 0x7f804810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.828+0000 7f80501f4700 1 --2- 172.21.15.70:0/4288062929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80481084b0 0x7f8048108890 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.828+0000 7f80501f4700 1 --2- 172.21.15.70:0/4288062929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8048102470 0x7f80481028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.828+0000 7f80501f4700 1 -- 172.21.15.70:0/4288062929 >> 172.21.15.70:0/4288062929 conn(0x7f8048079410 msgr2=0x7f8048079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.828+0000 7f80501f4700 1 -- 172.21.15.70:0/4288062929 shutdown_connections 2024-06-04T05:07:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.828+0000 7f80501f4700 1 -- 172.21.15.70:0/4288062929 wait complete. 2024-06-04T05:07:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.828+0000 7f80501f4700 1 Processor -- start 2024-06-04T05:07:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f80501f4700 1 -- start start 2024-06-04T05:07:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f80501f4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8048102470 0x7f804819e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f80501f4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8048102e30 0x7f804819ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f80501f4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80481084b0 0x7f8048198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f80501f4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f80481124f0 con 0x7f80481084b0 2024-06-04T05:07:46.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f80501f4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8048112370 con 0x7f8048102e30 2024-06-04T05:07:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f80501f4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8048112670 con 0x7f8048102470 2024-06-04T05:07:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f804e791700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80481084b0 0x7f8048198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f804e791700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80481084b0 0x7f8048198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36426/0 (socket says 172.21.15.70:36426) 2024-06-04T05:07:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f804e791700 1 -- 172.21.15.70:0/3705575506 learned_addr learned my addr 172.21.15.70:0/3705575506 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f804df90700 1 --2- 172.21.15.70:0/3705575506 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8048102470 0x7f804819e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f804d78f700 1 --2- 172.21.15.70:0/3705575506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8048102e30 0x7f804819ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f804e791700 1 -- 172.21.15.70:0/3705575506 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8048102470 msgr2=0x7f804819e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f804e791700 1 --2- 172.21.15.70:0/3705575506 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8048102470 0x7f804819e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f804e791700 1 -- 172.21.15.70:0/3705575506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8048102e30 msgr2=0x7f804819ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f804e791700 1 --2- 172.21.15.70:0/3705575506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8048102e30 0x7f804819ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f804e791700 1 -- 172.21.15.70:0/3705575506 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80481990c0 con 0x7f80481084b0 2024-06-04T05:07:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f804d78f700 1 --2- 172.21.15.70:0/3705575506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8048102e30 0x7f804819ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:07:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f804e791700 1 --2- 172.21.15.70:0/3705575506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80481084b0 0x7f8048198960 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f804000db90 tx=0x7f804000dea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:46.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.829+0000 7f803effd700 1 -- 172.21.15.70:0/3705575506 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804000e750 con 0x7f80481084b0 2024-06-04T05:07:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.830+0000 7f80501f4700 1 -- 172.21.15.70:0/3705575506 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8048199350 con 0x7f80481084b0 2024-06-04T05:07:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.830+0000 7f803effd700 1 -- 172.21.15.70:0/3705575506 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f804000ed80 con 0x7f80481084b0 2024-06-04T05:07:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.830+0000 7f803effd700 1 -- 172.21.15.70:0/3705575506 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80400163f0 con 0x7f80481084b0 2024-06-04T05:07:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.830+0000 7f80501f4700 1 -- 172.21.15.70:0/3705575506 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f804806bc90 con 0x7f80481084b0 2024-06-04T05:07:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.832+0000 7f803effd700 1 -- 172.21.15.70:0/3705575506 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f804000e8f0 con 0x7f80481084b0 2024-06-04T05:07:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.833+0000 7f803effd700 1 --2- 172.21.15.70:0/3705575506 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f80340b29f0 0x7f80340b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.833+0000 7f804df90700 1 --2- 172.21.15.70:0/3705575506 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f80340b29f0 0x7f80340b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.833+0000 7f803effd700 1 -- 172.21.15.70:0/3705575506 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f80400e3a20 con 0x7f80481084b0 2024-06-04T05:07:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.833+0000 7f805aa89b80 1 Processor -- start 2024-06-04T05:07:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.833+0000 7f805aa89b80 1 -- start start 2024-06-04T05:07:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.833+0000 7f804df90700 1 --2- 172.21.15.70:0/3705575506 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f80340b29f0 0x7f80340b4eb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f804400b5c0 tx=0x7f804401a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f805aa89b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d0860 0x5562086d0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f805aa89b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d1210 0x5562086d4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f805aa89b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086d53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f805aa89b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55620862fc90 con 0x5562086d4f60 2024-06-04T05:07:46.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f805aa89b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55620861f830 con 0x5562086d1210 2024-06-04T05:07:46.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f805aa89b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556208623870 con 0x5562086d0860 2024-06-04T05:07:46.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f804e791700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086d53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:46.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f804e791700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086d53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36430/0 (socket says 172.21.15.70:36430) 2024-06-04T05:07:46.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f804e791700 1 -- 172.21.15.70:0/448134519 learned_addr learned my addr 172.21.15.70:0/448134519 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:46.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f804df90700 1 --2- 172.21.15.70:0/448134519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d0860 0x5562086d0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:46.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f804d78f700 1 --2- 172.21.15.70:0/448134519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d1210 0x5562086d4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:46.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f804e791700 1 -- 172.21.15.70:0/448134519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d0860 msgr2=0x5562086d0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:46.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f804e791700 1 --2- 172.21.15.70:0/448134519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d0860 0x5562086d0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f804e791700 1 -- 172.21.15.70:0/448134519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d1210 msgr2=0x5562086d4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:46.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f804e791700 1 --2- 172.21.15.70:0/448134519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d1210 0x5562086d4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f804e791700 1 -- 172.21.15.70:0/448134519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55620856f8d0 con 0x5562086d4f60 2024-06-04T05:07:46.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f804e791700 1 --2- 172.21.15.70:0/448134519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086d53a0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f80400a6e60 tx=0x7f8040004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29bf7fd58fc5e256 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:46.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f8033fff700 1 -- 172.21.15.70:0/448134519 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80400ae070 con 0x5562086d4f60 2024-06-04T05:07:46.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f8033fff700 1 -- 172.21.15.70:0/448134519 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f80400aa8f0 con 0x5562086d4f60 2024-06-04T05:07:46.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.834+0000 7f8033fff700 1 -- 172.21.15.70:0/448134519 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80400ab2a0 con 0x5562086d4f60 2024-06-04T05:07:46.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.835+0000 7f805aa89b80 1 -- 172.21.15.70:0/448134519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 msgr2=0x5562086d53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:46.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.835+0000 7f805aa89b80 1 --2- 172.21.15.70:0/448134519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086d53a0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f80400a6e60 tx=0x7f8040004ad0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.835+0000 7f805aa89b80 1 -- 172.21.15.70:0/448134519 shutdown_connections 2024-06-04T05:07:46.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.835+0000 7f805aa89b80 1 --2- 172.21.15.70:0/448134519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086d53a0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.835+0000 7f805aa89b80 1 --2- 172.21.15.70:0/448134519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d0860 0x5562086d0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.835+0000 7f805aa89b80 1 --2- 172.21.15.70:0/448134519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d1210 0x5562086d4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.835+0000 7f805aa89b80 1 -- 172.21.15.70:0/448134519 >> 172.21.15.70:0/448134519 conn(0x5562085fe080 msgr2=0x5562086cae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:46.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.835+0000 7f805aa89b80 1 -- 172.21.15.70:0/448134519 shutdown_connections 2024-06-04T05:07:46.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.835+0000 7f805aa89b80 1 -- 172.21.15.70:0/448134519 wait complete. 2024-06-04T05:07:46.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f805aa89b80 1 -- start start 2024-06-04T05:07:46.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f805aa89b80 1 Processor -- start 2024-06-04T05:07:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f805aa89b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d0860 0x5562086dc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f805aa89b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d1210 0x5562086d9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f805aa89b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086d9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f805aa89b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556208525fa0 con 0x5562086d4f60 2024-06-04T05:07:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f805aa89b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5562085c17c0 con 0x5562086d0860 2024-06-04T05:07:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f805aa89b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5562086dd120 con 0x5562086d1210 2024-06-04T05:07:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f805aa89b80 10 client.0 _ll_register_callbacks cb 0x5562085ba920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f804e791700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086d9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f804e791700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086d9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36442/0 (socket says 172.21.15.70:36442) 2024-06-04T05:07:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f804e791700 1 -- 172.21.15.70:0/1847063402 learned_addr learned my addr 172.21.15.70:0/1847063402 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f804d78f700 1 --2- 172.21.15.70:0/1847063402 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d1210 0x5562086d9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f804df90700 1 --2- 172.21.15.70:0/1847063402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d0860 0x5562086dc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f804e791700 1 -- 172.21.15.70:0/1847063402 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d1210 msgr2=0x5562086d9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:46.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f804e791700 1 --2- 172.21.15.70:0/1847063402 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d1210 0x5562086d9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f804e791700 1 -- 172.21.15.70:0/1847063402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d0860 msgr2=0x5562086dc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f804e791700 1 --2- 172.21.15.70:0/1847063402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d0860 0x5562086dc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.836+0000 7f804e791700 1 -- 172.21.15.70:0/1847063402 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562085be690 con 0x5562086d4f60 2024-06-04T05:07:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.837+0000 7f804e791700 1 --2- 172.21.15.70:0/1847063402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086d9cb0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f804000b0a0 tx=0x7f80400098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e88551e76f88d78 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.837+0000 7f80317fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.837+0000 7f80317fa700 1 -- 172.21.15.70:0/1847063402 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804006e020 con 0x5562086d4f60 2024-06-04T05:07:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.837+0000 7f80317fa700 1 -- 172.21.15.70:0/1847063402 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f80400b37b0 con 0x5562086d4f60 2024-06-04T05:07:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.837+0000 7f805aa89b80 1 -- 172.21.15.70:0/1847063402 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80400aa8f0 con 0x5562086d4f60 2024-06-04T05:07:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.837+0000 7f80317fa700 1 -- 172.21.15.70:0/1847063402 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f80400ae040 con 0x5562086d4f60 2024-06-04T05:07:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.837+0000 7f805aa89b80 10 client.18252 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.837+0000 7f805aa89b80 1 -- 172.21.15.70:0/1847063402 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5562086e78f0 con 0x5562086d4f60 2024-06-04T05:07:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.837+0000 7f80317fa700 1 -- 172.21.15.70:0/1847063402 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f80400aa8f0 con 0x5562086d4f60 2024-06-04T05:07:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.837+0000 7f80317fa700 1 -- 172.21.15.70:0/1847063402 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f80400a6750 con 0x5562086d4f60 2024-06-04T05:07:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.837+0000 7f805aa89b80 10 client.18252 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.837+0000 7f805aa89b80 10 client.18252 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:07:46.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.837+0000 7f805aa89b80 20 client.18252 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.837+0000 7f805aa89b80 1 --2- 172.21.15.70:0/1847063402 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5562086e9dc0 0x5562086ec1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.837+0000 7f805aa89b80 4 client.18252 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:07:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.837+0000 7f805aa89b80 1 -- 172.21.15.70:0/1847063402 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55620869c9e0 con 0x5562086e9dc0 2024-06-04T05:07:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.838+0000 7f804df90700 1 --2- 172.21.15.70:0/1847063402 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5562086e9dc0 0x5562086ec1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.838+0000 7f80317fa700 10 client.18252 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:07:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.838+0000 7f804df90700 1 --2- 172.21.15.70:0/1847063402 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5562086e9dc0 0x5562086ec1b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9c5cb7dfa4e732af server_cookie=9cb72282839b2d84 in_seq=0 out_seq=0 2024-06-04T05:07:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.839+0000 7f80317fa700 1 -- 172.21.15.70:0/1847063402 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55620869c9e0 con 0x5562086e9dc0 2024-06-04T05:07:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.839+0000 7f80317fa700 10 client.18252 handle_command_reply: tid=0 2024-06-04T05:07:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.839+0000 7f805aa89b80 1 client.18252 shutdown 2024-06-04T05:07:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.839+0000 7f805aa89b80 1 -- 172.21.15.70:0/1847063402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 msgr2=0x5562086d9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.839+0000 7f805aa89b80 1 --2- 172.21.15.70:0/1847063402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086d9cb0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f804000b0a0 tx=0x7f80400098e0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.839+0000 7f805aa89b80 1 -- 172.21.15.70:0/1847063402 shutdown_connections 2024-06-04T05:07:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.839+0000 7f805aa89b80 1 --2- 172.21.15.70:0/1847063402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086d9cb0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.839+0000 7f805aa89b80 1 --2- 172.21.15.70:0/1847063402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d0860 0x5562086dc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.839+0000 7f805aa89b80 1 --2- 172.21.15.70:0/1847063402 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5562086e9dc0 0x5562086ec1b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.839+0000 7f805aa89b80 1 --2- 172.21.15.70:0/1847063402 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d1210 0x5562086d9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.839+0000 7f805aa89b80 1 -- 172.21.15.70:0/1847063402 >> 172.21.15.70:0/1847063402 conn(0x5562085fe080 msgr2=0x5562086ca510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.839+0000 7f80317fa700 0 client.18252 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:07:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.839+0000 7f805aa89b80 1 -- 172.21.15.70:0/1847063402 shutdown_connections 2024-06-04T05:07:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.839+0000 7f805aa89b80 1 -- 172.21.15.70:0/1847063402 wait complete. 2024-06-04T05:07:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.839+0000 7f805aa89b80 20 client.18252 trim_cache size 0 max 16384 2024-06-04T05:07:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.854+0000 7f805aa89b80 1 Processor -- start 2024-06-04T05:07:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.854+0000 7f805aa89b80 1 -- start start 2024-06-04T05:07:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.854+0000 7f805aa89b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d0860 0x5562086d9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.854+0000 7f805aa89b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d1210 0x5562086cac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.854+0000 7f805aa89b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086e53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.854+0000 7f805aa89b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f80481abd30 con 0x5562086d4f60 2024-06-04T05:07:46.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.854+0000 7f805aa89b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80481ad8d0 con 0x5562086d1210 2024-06-04T05:07:46.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.854+0000 7f805aa89b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f80481ac400 con 0x5562086d0860 2024-06-04T05:07:46.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.854+0000 7f804e791700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086e53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:46.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.854+0000 7f804e791700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086e53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36448/0 (socket says 172.21.15.70:36448) 2024-06-04T05:07:46.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.854+0000 7f804e791700 1 -- 172.21.15.70:0/3064053207 learned_addr learned my addr 172.21.15.70:0/3064053207 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:46.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.855+0000 7f804d78f700 1 --2- 172.21.15.70:0/3064053207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d1210 0x5562086cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:46.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.855+0000 7f804e791700 1 -- 172.21.15.70:0/3064053207 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d0860 msgr2=0x5562086d9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:46.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.855+0000 7f804e791700 1 --2- 172.21.15.70:0/3064053207 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d0860 0x5562086d9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.855+0000 7f804e791700 1 -- 172.21.15.70:0/3064053207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d1210 msgr2=0x5562086cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:46.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.855+0000 7f804e791700 1 --2- 172.21.15.70:0/3064053207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d1210 0x5562086cac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.855+0000 7f804e791700 1 -- 172.21.15.70:0/3064053207 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562086c35b0 con 0x5562086d4f60 2024-06-04T05:07:46.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.855+0000 7f804d78f700 1 --2- 172.21.15.70:0/3064053207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d1210 0x5562086cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:07:46.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.855+0000 7f804e791700 1 --2- 172.21.15.70:0/3064053207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086e53b0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f804000db60 tx=0x7f8040009950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f5359b61537847 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:46.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.855+0000 7f80337fe700 1 -- 172.21.15.70:0/3064053207 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804006e040 con 0x5562086d4f60 2024-06-04T05:07:46.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.855+0000 7f80337fe700 1 -- 172.21.15.70:0/3064053207 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f80400aa8f0 con 0x5562086d4f60 2024-06-04T05:07:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.855+0000 7f80337fe700 1 -- 172.21.15.70:0/3064053207 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80400af190 con 0x5562086d4f60 2024-06-04T05:07:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 1 -- 172.21.15.70:0/3064053207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 msgr2=0x5562086e53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 1 --2- 172.21.15.70:0/3064053207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086e53b0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f804000db60 tx=0x7f8040009950 comp rx=0 tx=0).stop 2024-06-04T05:07:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 1 -- 172.21.15.70:0/3064053207 shutdown_connections 2024-06-04T05:07:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 1 --2- 172.21.15.70:0/3064053207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086e53b0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 1 --2- 172.21.15.70:0/3064053207 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d0860 0x5562086d9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 1 --2- 172.21.15.70:0/3064053207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d1210 0x5562086cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 1 -- 172.21.15.70:0/3064053207 >> 172.21.15.70:0/3064053207 conn(0x5562085fe080 msgr2=0x5562086db830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 1 -- 172.21.15.70:0/3064053207 shutdown_connections 2024-06-04T05:07:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 1 -- 172.21.15.70:0/3064053207 wait complete. 2024-06-04T05:07:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 1 -- start start 2024-06-04T05:07:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 1 Processor -- start 2024-06-04T05:07:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d0860 0x55620874be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d1210 0x55620874c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086eedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f80481ad1d0 con 0x5562086d4f60 2024-06-04T05:07:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80481acad0 con 0x5562086d0860 2024-06-04T05:07:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f80481a9cf0 con 0x5562086d1210 2024-06-04T05:07:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.856+0000 7f805aa89b80 10 client.0 _ll_register_callbacks cb 0x5562085ba920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f804e791700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086eedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f804e791700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086eedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36460/0 (socket says 172.21.15.70:36460) 2024-06-04T05:07:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f804e791700 1 -- 172.21.15.70:0/1391390776 learned_addr learned my addr 172.21.15.70:0/1391390776 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f804df90700 1 --2- 172.21.15.70:0/1391390776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d0860 0x55620874be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f804d78f700 1 --2- 172.21.15.70:0/1391390776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d1210 0x55620874c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f804e791700 1 -- 172.21.15.70:0/1391390776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d1210 msgr2=0x55620874c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f804e791700 1 --2- 172.21.15.70:0/1391390776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d1210 0x55620874c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f804e791700 1 -- 172.21.15.70:0/1391390776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d0860 msgr2=0x55620874be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f804e791700 1 --2- 172.21.15.70:0/1391390776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d0860 0x55620874be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f804e791700 1 -- 172.21.15.70:0/1391390776 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562086cc970 con 0x5562086d4f60 2024-06-04T05:07:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f804e791700 1 --2- 172.21.15.70:0/1391390776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086eedb0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f80400aaae0 tx=0x7f8040004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e39b00c29c357b53 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f80317fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f80317fa700 1 -- 172.21.15.70:0/1391390776 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804006e040 con 0x5562086d4f60 2024-06-04T05:07:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f80317fa700 1 -- 172.21.15.70:0/1391390776 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f80400a9bd0 con 0x5562086d4f60 2024-06-04T05:07:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f805aa89b80 1 -- 172.21.15.70:0/1391390776 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80400aa8f0 con 0x5562086d4f60 2024-06-04T05:07:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f80317fa700 1 -- 172.21.15.70:0/1391390776 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f80400ab4e0 con 0x5562086d4f60 2024-06-04T05:07:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f805aa89b80 10 client.18258 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f805aa89b80 1 -- 172.21.15.70:0/1391390776 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5562086cc970 con 0x5562086d4f60 2024-06-04T05:07:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.857+0000 7f80317fa700 1 -- 172.21.15.70:0/1391390776 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f80400aa8f0 con 0x5562086d4f60 2024-06-04T05:07:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.858+0000 7f80317fa700 1 -- 172.21.15.70:0/1391390776 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f804008c020 con 0x5562086d4f60 2024-06-04T05:07:46.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.858+0000 7f805aa89b80 10 client.18258 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:46.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.858+0000 7f805aa89b80 10 client.18258 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:07:46.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.858+0000 7f805aa89b80 20 client.18258 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:46.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.858+0000 7f805aa89b80 1 --2- 172.21.15.70:0/1391390776 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5562087547b0 0x556208756ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:46.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.858+0000 7f805aa89b80 4 client.18258 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:07:46.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.858+0000 7f805aa89b80 1 -- 172.21.15.70:0/1391390776 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556208631be0 con 0x5562087547b0 2024-06-04T05:07:46.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.858+0000 7f804df90700 1 --2- 172.21.15.70:0/1391390776 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5562087547b0 0x556208756ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:46.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.858+0000 7f80317fa700 10 client.18258 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:07:46.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.858+0000 7f804df90700 1 --2- 172.21.15.70:0/1391390776 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5562087547b0 0x556208756ba0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7c1d08d8bbed6148 server_cookie=46a286ec16071345 in_seq=0 out_seq=0 2024-06-04T05:07:46.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.858+0000 7f80317fa700 1 -- 172.21.15.70:0/1391390776 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556208631be0 con 0x5562087547b0 2024-06-04T05:07:46.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.858+0000 7f80317fa700 10 client.18258 handle_command_reply: tid=0 2024-06-04T05:07:46.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.858+0000 7f805aa89b80 1 client.18258 shutdown 2024-06-04T05:07:46.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f805aa89b80 1 -- 172.21.15.70:0/1391390776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 msgr2=0x5562086eedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:46.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f805aa89b80 1 --2- 172.21.15.70:0/1391390776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086eedb0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f80400aaae0 tx=0x7f8040004ad0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f805aa89b80 1 -- 172.21.15.70:0/1391390776 shutdown_connections 2024-06-04T05:07:46.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f805aa89b80 1 --2- 172.21.15.70:0/1391390776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562086d4f60 0x5562086eedb0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f805aa89b80 1 --2- 172.21.15.70:0/1391390776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562086d0860 0x55620874be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f805aa89b80 1 --2- 172.21.15.70:0/1391390776 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5562087547b0 0x556208756ba0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f805aa89b80 1 --2- 172.21.15.70:0/1391390776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5562086d1210 0x55620874c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f80317fa700 0 client.18258 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:07:46.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f805aa89b80 1 -- 172.21.15.70:0/1391390776 >> 172.21.15.70:0/1391390776 conn(0x5562085fe080 msgr2=0x5562086dafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:46.936 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:07:46.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f805aa89b80 1 -- 172.21.15.70:0/1391390776 shutdown_connections 2024-06-04T05:07:46.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f805aa89b80 1 -- 172.21.15.70:0/1391390776 wait complete. 2024-06-04T05:07:46.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f805aa89b80 20 client.18258 trim_cache size 0 max 16384 2024-06-04T05:07:46.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f80501f4700 1 -- 172.21.15.70:0/3705575506 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f80340b29f0 msgr2=0x7f80340b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:46.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f80501f4700 1 --2- 172.21.15.70:0/3705575506 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f80340b29f0 0x7f80340b4eb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f804400b5c0 tx=0x7f804401a040 comp rx=0 tx=0).stop 2024-06-04T05:07:46.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f80501f4700 1 -- 172.21.15.70:0/3705575506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80481084b0 msgr2=0x7f8048198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:46.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f80501f4700 1 --2- 172.21.15.70:0/3705575506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80481084b0 0x7f8048198960 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f804000db90 tx=0x7f804000dea0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f80501f4700 1 -- 172.21.15.70:0/3705575506 shutdown_connections 2024-06-04T05:07:46.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f80501f4700 1 --2- 172.21.15.70:0/3705575506 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f80340b29f0 0x7f80340b4eb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f80501f4700 1 --2- 172.21.15.70:0/3705575506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80481084b0 0x7f8048198960 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f80501f4700 1 --2- 172.21.15.70:0/3705575506 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8048102470 0x7f804819e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f80501f4700 1 --2- 172.21.15.70:0/3705575506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8048102e30 0x7f804819ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:46.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f80501f4700 1 -- 172.21.15.70:0/3705575506 >> 172.21.15.70:0/3705575506 conn(0x7f8048079410 msgr2=0x7f8048100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:46.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.859+0000 7f80501f4700 1 -- 172.21.15.70:0/3705575506 shutdown_connections 2024-06-04T05:07:46.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:46.860+0000 7f80501f4700 1 -- 172.21.15.70:0/3705575506 wait complete. 2024-06-04T05:07:47.665 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:07:47.733 INFO:tasks.workunit.client.0.smithi070.stdout: 1 184804 1.88 MB/sec execute 400 sec latency 4765.503 ms 2024-06-04T05:07:47.931 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:07:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:47 smithi112 ceph-mon[36453]: pgmap v327: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.0 MiB/s wr, 108 op/s 2024-06-04T05:07:47.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:47 smithi098 ceph-mon[35569]: pgmap v327: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.0 MiB/s wr, 108 op/s 2024-06-04T05:07:48.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:47 smithi070 ceph-mon[32631]: pgmap v327: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.0 MiB/s wr, 108 op/s 2024-06-04T05:07:48.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 185082 1.88 MB/sec execute 401 sec latency 42.891 ms 2024-06-04T05:07:49.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.538+0000 7f33bba20700 1 -- 172.21.15.70:0/3533182178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33b4108920 msgr2=0x7f33b41104a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:49.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.538+0000 7f33bba20700 1 --2- 172.21.15.70:0/3533182178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33b4108920 0x7f33b41104a0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f33ac00b0d0 tx=0x7f33ac00b3e0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.539+0000 7f33bba20700 1 -- 172.21.15.70:0/3533182178 shutdown_connections 2024-06-04T05:07:49.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.539+0000 7f33bba20700 1 --2- 172.21.15.70:0/3533182178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33b4108920 0x7f33b41104a0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.539+0000 7f33bba20700 1 --2- 172.21.15.70:0/3533182178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33b4103850 0x7f33b4103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.539+0000 7f33bba20700 1 --2- 172.21.15.70:0/3533182178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33b4104200 0x7f33b41081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.539+0000 7f33bba20700 1 -- 172.21.15.70:0/3533182178 >> 172.21.15.70:0/3533182178 conn(0x7f33b407a5d0 msgr2=0x7f33b407a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:49.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.539+0000 7f33bba20700 1 -- 172.21.15.70:0/3533182178 shutdown_connections 2024-06-04T05:07:49.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.539+0000 7f33bba20700 1 -- 172.21.15.70:0/3533182178 wait complete. 2024-06-04T05:07:49.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.539+0000 7f33bba20700 1 Processor -- start 2024-06-04T05:07:49.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.539+0000 7f33bba20700 1 -- start start 2024-06-04T05:07:49.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33bba20700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33b4103850 0x7f33b419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33bba20700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33b4104200 0x7f33b419fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33bba20700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33b4108920 0x7f33b4199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33bba20700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33b41135c0 con 0x7f33b4104200 2024-06-04T05:07:49.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33bba20700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f33b4113440 con 0x7f33b4108920 2024-06-04T05:07:49.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33bba20700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f33b4113740 con 0x7f33b4103850 2024-06-04T05:07:49.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33b8fbb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33b4104200 0x7f33b419fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33b8fbb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33b4104200 0x7f33b419fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36480/0 (socket says 172.21.15.70:36480) 2024-06-04T05:07:49.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33b8fbb700 1 -- 172.21.15.70:0/1393006973 learned_addr learned my addr 172.21.15.70:0/1393006973 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:49.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33b97bc700 1 --2- 172.21.15.70:0/1393006973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33b4103850 0x7f33b419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33b9fbd700 1 --2- 172.21.15.70:0/1393006973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33b4108920 0x7f33b4199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33b8fbb700 1 -- 172.21.15.70:0/1393006973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33b4103850 msgr2=0x7f33b419f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:49.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33b8fbb700 1 --2- 172.21.15.70:0/1393006973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33b4103850 0x7f33b419f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33b8fbb700 1 -- 172.21.15.70:0/1393006973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33b4108920 msgr2=0x7f33b4199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:49.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33b8fbb700 1 --2- 172.21.15.70:0/1393006973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33b4108920 0x7f33b4199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33b8fbb700 1 -- 172.21.15.70:0/1393006973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b4199fe0 con 0x7f33b4104200 2024-06-04T05:07:49.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33b97bc700 1 --2- 172.21.15.70:0/1393006973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33b4103850 0x7f33b419f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:49.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33b8fbb700 1 --2- 172.21.15.70:0/1393006973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33b4104200 0x7f33b419fc00 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f33a400e9c0 tx=0x7f33a400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:49.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33aa7fc700 1 -- 172.21.15.70:0/1393006973 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a400cbd0 con 0x7f33b4104200 2024-06-04T05:07:49.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33aa7fc700 1 -- 172.21.15.70:0/1393006973 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f33a400ee60 con 0x7f33b4104200 2024-06-04T05:07:49.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.540+0000 7f33aa7fc700 1 -- 172.21.15.70:0/1393006973 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a40054b0 con 0x7f33b4104200 2024-06-04T05:07:49.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.541+0000 7f33bba20700 1 -- 172.21.15.70:0/1393006973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33b419a2d0 con 0x7f33b4104200 2024-06-04T05:07:49.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.541+0000 7f33bba20700 1 -- 172.21.15.70:0/1393006973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33b41a6590 con 0x7f33b4104200 2024-06-04T05:07:49.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.544+0000 7f33aa7fc700 1 -- 172.21.15.70:0/1393006973 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f33a4021440 con 0x7f33b4104200 2024-06-04T05:07:49.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.545+0000 7f33aa7fc700 1 --2- 172.21.15.70:0/1393006973 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33a00b29f0 0x7f33a00b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.545+0000 7f33b97bc700 1 --2- 172.21.15.70:0/1393006973 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33a00b29f0 0x7f33a00b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.545+0000 7f33aa7fc700 1 -- 172.21.15.70:0/1393006973 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f33a40e29d0 con 0x7f33b4104200 2024-06-04T05:07:49.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.545+0000 7f33c62b5b80 1 Processor -- start 2024-06-04T05:07:49.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.545+0000 7f33c62b5b80 1 -- start start 2024-06-04T05:07:49.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.545+0000 7f33b97bc700 1 --2- 172.21.15.70:0/1393006973 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33a00b29f0 0x7f33a00b4eb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f33b419a9c0 tx=0x7f33b0009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:49.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.545+0000 7f33c62b5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd436320 0x5646dd436700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.545+0000 7f33c62b5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd4308a0 0x5646dd430d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.545+0000 7f33c62b5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd431260 0x5646dd43d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f33c62b5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5646dd38ed70 con 0x5646dd431260 2024-06-04T05:07:49.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f33c62b5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5646dd37e910 con 0x5646dd4308a0 2024-06-04T05:07:49.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f33c62b5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5646dd382950 con 0x5646dd436320 2024-06-04T05:07:49.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f33b97bc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd436320 0x5646dd436700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f33b97bc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd436320 0x5646dd436700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:56156/0 (socket says 172.21.15.70:56156) 2024-06-04T05:07:49.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f33b97bc700 1 -- 172.21.15.70:0/1210941738 learned_addr learned my addr 172.21.15.70:0/1210941738 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:49.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f33b9fbd700 1 --2- 172.21.15.70:0/1210941738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd431260 0x5646dd43d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f33b8fbb700 1 --2- 172.21.15.70:0/1210941738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd4308a0 0x5646dd430d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f33b97bc700 1 -- 172.21.15.70:0/1210941738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd4308a0 msgr2=0x5646dd430d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:49.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f33b97bc700 1 --2- 172.21.15.70:0/1210941738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd4308a0 0x5646dd430d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f33b97bc700 1 -- 172.21.15.70:0/1210941738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd431260 msgr2=0x5646dd43d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:49.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f33b97bc700 1 --2- 172.21.15.70:0/1210941738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd431260 0x5646dd43d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f33b97bc700 1 -- 172.21.15.70:0/1210941738 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646dd327fb0 con 0x5646dd436320 2024-06-04T05:07:49.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f33b9fbd700 1 --2- 172.21.15.70:0/1210941738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd431260 0x5646dd43d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:07:49.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f33b97bc700 1 --2- 172.21.15.70:0/1210941738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd436320 0x5646dd436700 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f33b00043c0 tx=0x7f33b0014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17e1c5d7ef811296 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:49.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f339f7fe700 1 -- 172.21.15.70:0/1210941738 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b0003dc0 con 0x5646dd436320 2024-06-04T05:07:49.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f339f7fe700 1 -- 172.21.15.70:0/1210941738 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f33b001e8b0 con 0x5646dd436320 2024-06-04T05:07:49.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.546+0000 7f339f7fe700 1 -- 172.21.15.70:0/1210941738 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b00267f0 con 0x5646dd436320 2024-06-04T05:07:49.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.547+0000 7f33c62b5b80 1 -- 172.21.15.70:0/1210941738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd436320 msgr2=0x5646dd436700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:49.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.547+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/1210941738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd436320 0x5646dd436700 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f33b00043c0 tx=0x7f33b0014000 comp rx=0 tx=0).stop 2024-06-04T05:07:49.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.547+0000 7f33c62b5b80 1 -- 172.21.15.70:0/1210941738 shutdown_connections 2024-06-04T05:07:49.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.547+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/1210941738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd431260 0x5646dd43d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.547+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/1210941738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd436320 0x5646dd436700 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.547+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/1210941738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd4308a0 0x5646dd430d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.547+0000 7f33c62b5b80 1 -- 172.21.15.70:0/1210941738 >> 172.21.15.70:0/1210941738 conn(0x5646dd35c240 msgr2=0x5646dd42b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:49.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.547+0000 7f33c62b5b80 1 -- 172.21.15.70:0/1210941738 shutdown_connections 2024-06-04T05:07:49.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.548+0000 7f33c62b5b80 1 -- 172.21.15.70:0/1210941738 wait complete. 2024-06-04T05:07:49.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.548+0000 7f33c62b5b80 1 -- start start 2024-06-04T05:07:49.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.548+0000 7f33c62b5b80 1 Processor -- start 2024-06-04T05:07:49.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.548+0000 7f33c62b5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd4308a0 0x5646dd43cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.548+0000 7f33c62b5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd431260 0x5646dd43a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.548+0000 7f33c62b5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd436320 0x5646dd43a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.548+0000 7f33c62b5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5646dd284fa0 con 0x5646dd431260 2024-06-04T05:07:49.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.548+0000 7f33c62b5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5646dd2e6da0 con 0x5646dd4308a0 2024-06-04T05:07:49.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.548+0000 7f33c62b5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5646dd43d6e0 con 0x5646dd436320 2024-06-04T05:07:49.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.549+0000 7f33c62b5b80 10 client.0 _ll_register_callbacks cb 0x5646dd319950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:49.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.549+0000 7f33b8fbb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd431260 0x5646dd43a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.549+0000 7f33b8fbb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd431260 0x5646dd43a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36494/0 (socket says 172.21.15.70:36494) 2024-06-04T05:07:49.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.549+0000 7f33b8fbb700 1 -- 172.21.15.70:0/3938673016 learned_addr learned my addr 172.21.15.70:0/3938673016 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:49.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.549+0000 7f33b97bc700 1 --2- 172.21.15.70:0/3938673016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd4308a0 0x5646dd43cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.549+0000 7f33b9fbd700 1 --2- 172.21.15.70:0/3938673016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd436320 0x5646dd43a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.549+0000 7f33b8fbb700 1 -- 172.21.15.70:0/3938673016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd436320 msgr2=0x5646dd43a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:49.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.549+0000 7f33b8fbb700 1 --2- 172.21.15.70:0/3938673016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd436320 0x5646dd43a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.549+0000 7f33b8fbb700 1 -- 172.21.15.70:0/3938673016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd4308a0 msgr2=0x5646dd43cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:49.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.549+0000 7f33b8fbb700 1 --2- 172.21.15.70:0/3938673016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd4308a0 0x5646dd43cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.549+0000 7f33b8fbb700 1 -- 172.21.15.70:0/3938673016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646dd2ce8d0 con 0x5646dd431260 2024-06-04T05:07:49.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.549+0000 7f33b97bc700 1 --2- 172.21.15.70:0/3938673016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd4308a0 0x5646dd43cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:49.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.549+0000 7f33b8fbb700 1 --2- 172.21.15.70:0/3938673016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd431260 0x5646dd43a030 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f33a40a6ef0 tx=0x7f33a40a5410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17b66671b1264cee server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:49.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.549+0000 7f339cff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:49.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.550+0000 7f339cff9700 1 -- 172.21.15.70:0/3938673016 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a40aa380 con 0x5646dd431260 2024-06-04T05:07:49.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.550+0000 7f33c62b5b80 1 -- 172.21.15.70:0/3938673016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33b001e8b0 con 0x5646dd431260 2024-06-04T05:07:49.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.550+0000 7f339cff9700 1 -- 172.21.15.70:0/3938673016 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f33a40a6270 con 0x5646dd431260 2024-06-04T05:07:49.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.550+0000 7f339cff9700 1 -- 172.21.15.70:0/3938673016 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f33a40a9c80 con 0x5646dd431260 2024-06-04T05:07:49.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.550+0000 7f33c62b5b80 10 client.18270 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:49.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.550+0000 7f33c62b5b80 1 -- 172.21.15.70:0/3938673016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5646dd338760 con 0x5646dd431260 2024-06-04T05:07:49.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.550+0000 7f339cff9700 1 -- 172.21.15.70:0/3938673016 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f33b001e8b0 con 0x5646dd431260 2024-06-04T05:07:49.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.551+0000 7f339cff9700 1 -- 172.21.15.70:0/3938673016 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f33a40b2650 con 0x5646dd431260 2024-06-04T05:07:49.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.551+0000 7f33c62b5b80 10 client.18270 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:49.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.551+0000 7f33c62b5b80 10 client.18270 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:07:49.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.551+0000 7f33c62b5b80 20 client.18270 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:49.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.551+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/3938673016 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5646dd44a2f0 0x5646dd44c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.551+0000 7f33c62b5b80 4 client.18270 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:07:49.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.551+0000 7f33c62b5b80 1 -- 172.21.15.70:0/3938673016 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646dd390cc0 con 0x5646dd44a2f0 2024-06-04T05:07:49.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.551+0000 7f33b97bc700 1 --2- 172.21.15.70:0/3938673016 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5646dd44a2f0 0x5646dd44c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.551+0000 7f339cff9700 10 client.18270 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:07:49.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.552+0000 7f33b97bc700 1 --2- 172.21.15.70:0/3938673016 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5646dd44a2f0 0x5646dd44c6e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c12a98cc2283e599 server_cookie=d18284ad59a28a0c in_seq=0 out_seq=0 2024-06-04T05:07:49.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.552+0000 7f339cff9700 1 -- 172.21.15.70:0/3938673016 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5646dd390cc0 con 0x5646dd44a2f0 2024-06-04T05:07:49.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.552+0000 7f339cff9700 10 client.18270 handle_command_reply: tid=0 2024-06-04T05:07:49.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.552+0000 7f33c62b5b80 1 client.18270 shutdown 2024-06-04T05:07:49.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.553+0000 7f33c62b5b80 1 -- 172.21.15.70:0/3938673016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd431260 msgr2=0x5646dd43a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:49.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.553+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/3938673016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd431260 0x5646dd43a030 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f33a40a6ef0 tx=0x7f33a40a5410 comp rx=0 tx=0).stop 2024-06-04T05:07:49.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.553+0000 7f33c62b5b80 1 -- 172.21.15.70:0/3938673016 shutdown_connections 2024-06-04T05:07:49.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.553+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/3938673016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd436320 0x5646dd43a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.553+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/3938673016 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5646dd44a2f0 0x5646dd44c6e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.553+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/3938673016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd4308a0 0x5646dd43cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.553+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/3938673016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd431260 0x5646dd43a030 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.553+0000 7f33c62b5b80 1 -- 172.21.15.70:0/3938673016 >> 172.21.15.70:0/3938673016 conn(0x5646dd35c240 msgr2=0x5646dd42aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:49.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.553+0000 7f339cff9700 0 client.18270 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:07:49.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.553+0000 7f33c62b5b80 1 -- 172.21.15.70:0/3938673016 shutdown_connections 2024-06-04T05:07:49.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.553+0000 7f33c62b5b80 1 -- 172.21.15.70:0/3938673016 wait complete. 2024-06-04T05:07:49.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.553+0000 7f33c62b5b80 20 client.18270 trim_cache size 0 max 16384 2024-06-04T05:07:49.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.570+0000 7f33c62b5b80 1 Processor -- start 2024-06-04T05:07:49.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.570+0000 7f33c62b5b80 1 -- start start 2024-06-04T05:07:49.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.570+0000 7f33c62b5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd4308a0 0x5646dd438f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.570+0000 7f33c62b5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd436320 0x5646dd42b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.570+0000 7f33c62b5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd427760 0x5646dd437010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.570+0000 7f33c62b5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33b41ade40 con 0x5646dd4308a0 2024-06-04T05:07:49.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.570+0000 7f33c62b5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f33b41af9e0 con 0x5646dd427760 2024-06-04T05:07:49.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.570+0000 7f33c62b5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f33b41ae510 con 0x5646dd436320 2024-06-04T05:07:49.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.570+0000 7f33b97bc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd4308a0 0x5646dd438f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.570+0000 7f33b97bc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd4308a0 0x5646dd438f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36498/0 (socket says 172.21.15.70:36498) 2024-06-04T05:07:49.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.570+0000 7f33b97bc700 1 -- 172.21.15.70:0/2118118982 learned_addr learned my addr 172.21.15.70:0/2118118982 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:49.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.571+0000 7f33b8fbb700 1 --2- 172.21.15.70:0/2118118982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd436320 0x5646dd42b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.571+0000 7f33b9fbd700 1 --2- 172.21.15.70:0/2118118982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd427760 0x5646dd437010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.571+0000 7f33b97bc700 1 -- 172.21.15.70:0/2118118982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd436320 msgr2=0x5646dd42b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:49.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.571+0000 7f33b97bc700 1 --2- 172.21.15.70:0/2118118982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd436320 0x5646dd42b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.571+0000 7f33b97bc700 1 -- 172.21.15.70:0/2118118982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd427760 msgr2=0x5646dd437010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:49.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.571+0000 7f33b97bc700 1 --2- 172.21.15.70:0/2118118982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd427760 0x5646dd437010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.571+0000 7f33b97bc700 1 -- 172.21.15.70:0/2118118982 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646dd4385a0 con 0x5646dd4308a0 2024-06-04T05:07:49.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.571+0000 7f33b97bc700 1 --2- 172.21.15.70:0/2118118982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd4308a0 0x5646dd438f90 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f33b0009680 tx=0x7f33b00051d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13ba3c5480b503d0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:49.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.571+0000 7f339effd700 1 -- 172.21.15.70:0/2118118982 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b0005440 con 0x5646dd4308a0 2024-06-04T05:07:49.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.571+0000 7f339effd700 1 -- 172.21.15.70:0/2118118982 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f33b00055e0 con 0x5646dd4308a0 2024-06-04T05:07:49.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.571+0000 7f339effd700 1 -- 172.21.15.70:0/2118118982 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b001e8a0 con 0x5646dd4308a0 2024-06-04T05:07:49.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.572+0000 7f33c62b5b80 1 -- 172.21.15.70:0/2118118982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd4308a0 msgr2=0x5646dd438f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:49.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.572+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/2118118982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd4308a0 0x5646dd438f90 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f33b0009680 tx=0x7f33b00051d0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.572+0000 7f33c62b5b80 1 -- 172.21.15.70:0/2118118982 shutdown_connections 2024-06-04T05:07:49.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.572+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/2118118982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd427760 0x5646dd437010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.572+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/2118118982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd4308a0 0x5646dd438f90 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.572+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/2118118982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd436320 0x5646dd42b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.572+0000 7f33c62b5b80 1 -- 172.21.15.70:0/2118118982 >> 172.21.15.70:0/2118118982 conn(0x5646dd35c240 msgr2=0x5646dd430120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:49.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.572+0000 7f33c62b5b80 1 -- 172.21.15.70:0/2118118982 shutdown_connections 2024-06-04T05:07:49.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.572+0000 7f33c62b5b80 1 -- 172.21.15.70:0/2118118982 wait complete. 2024-06-04T05:07:49.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.573+0000 7f33c62b5b80 1 -- start start 2024-06-04T05:07:49.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.573+0000 7f33c62b5b80 1 Processor -- start 2024-06-04T05:07:49.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.573+0000 7f33c62b5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd427760 0x5646dd442940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.573+0000 7f33c62b5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd4308a0 0x5646dd442e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.573+0000 7f33c62b5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd436320 0x5646dd43d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.574+0000 7f33c62b5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33b41af2e0 con 0x5646dd436320 2024-06-04T05:07:49.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.574+0000 7f33c62b5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f33b41aebe0 con 0x5646dd4308a0 2024-06-04T05:07:49.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.574+0000 7f33b97bc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd427760 0x5646dd442940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.574+0000 7f33b9fbd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd436320 0x5646dd43d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.574+0000 7f33b97bc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd427760 0x5646dd442940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:56188/0 (socket says 172.21.15.70:56188) 2024-06-04T05:07:49.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.574+0000 7f33b97bc700 1 -- 172.21.15.70:0/2712381958 learned_addr learned my addr 172.21.15.70:0/2712381958 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:49.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.574+0000 7f33c62b5b80 1 -- 172.21.15.70:0/2712381958 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f33b41abe00 con 0x5646dd427760 2024-06-04T05:07:49.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.574+0000 7f33b8fbb700 1 --2- 172.21.15.70:0/2712381958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd4308a0 0x5646dd442e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.574+0000 7f33c62b5b80 10 client.0 _ll_register_callbacks cb 0x5646dd319950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:49.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.574+0000 7f33b9fbd700 1 -- 172.21.15.70:0/2712381958 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd427760 msgr2=0x5646dd442940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:49.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.574+0000 7f33b9fbd700 1 --2- 172.21.15.70:0/2712381958 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd427760 0x5646dd442940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.574+0000 7f33b9fbd700 1 -- 172.21.15.70:0/2712381958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd4308a0 msgr2=0x5646dd442e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:49.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.574+0000 7f33b9fbd700 1 --2- 172.21.15.70:0/2712381958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd4308a0 0x5646dd442e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.574+0000 7f33b9fbd700 1 -- 172.21.15.70:0/2712381958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646dd452a20 con 0x5646dd436320 2024-06-04T05:07:49.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.574+0000 7f33b9fbd700 1 --2- 172.21.15.70:0/2712381958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd436320 0x5646dd43d8f0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f33ac00b770 tx=0x7f33ac00bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946b26e03be0822e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:49.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.574+0000 7f339cff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:49.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.574+0000 7f339cff9700 1 -- 172.21.15.70:0/2712381958 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ac003c00 con 0x5646dd436320 2024-06-04T05:07:49.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.575+0000 7f339cff9700 1 -- 172.21.15.70:0/2712381958 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f33ac003da0 con 0x5646dd436320 2024-06-04T05:07:49.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.575+0000 7f33c62b5b80 1 -- 172.21.15.70:0/2712381958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33b00055e0 con 0x5646dd436320 2024-06-04T05:07:49.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.575+0000 7f339cff9700 1 -- 172.21.15.70:0/2712381958 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f33ac0074e0 con 0x5646dd436320 2024-06-04T05:07:49.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.575+0000 7f33c62b5b80 10 client.18276 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:49.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.575+0000 7f33c62b5b80 1 -- 172.21.15.70:0/2712381958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5646dd4b5330 con 0x5646dd436320 2024-06-04T05:07:49.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.575+0000 7f339cff9700 1 -- 172.21.15.70:0/2712381958 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f33b00055e0 con 0x5646dd436320 2024-06-04T05:07:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.575+0000 7f339cff9700 1 -- 172.21.15.70:0/2712381958 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f33ac011070 con 0x5646dd436320 2024-06-04T05:07:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.575+0000 7f33c62b5b80 10 client.18276 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.575+0000 7f33c62b5b80 10 client.18276 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:07:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.575+0000 7f33c62b5b80 20 client.18276 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.575+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/2712381958 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5646dd4b5a70 0x5646dd4b7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.575+0000 7f33c62b5b80 4 client.18276 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:07:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.575+0000 7f33c62b5b80 1 -- 172.21.15.70:0/2712381958 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5646dd38a4a0 con 0x5646dd4b5a70 2024-06-04T05:07:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.576+0000 7f33b97bc700 1 --2- 172.21.15.70:0/2712381958 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5646dd4b5a70 0x5646dd4b7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.576+0000 7f33b97bc700 1 --2- 172.21.15.70:0/2712381958 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5646dd4b5a70 0x5646dd4b7e90 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7b2f5009d1a4077e server_cookie=b0ec32f3825c178e in_seq=0 out_seq=0 2024-06-04T05:07:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.576+0000 7f339cff9700 10 client.18276 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:07:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.576+0000 7f339cff9700 1 -- 172.21.15.70:0/2712381958 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5646dd38a4a0 con 0x5646dd4b5a70 2024-06-04T05:07:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.576+0000 7f339cff9700 10 client.18276 handle_command_reply: tid=0 2024-06-04T05:07:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.576+0000 7f33c62b5b80 1 client.18276 shutdown 2024-06-04T05:07:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.576+0000 7f33c62b5b80 1 -- 172.21.15.70:0/2712381958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd436320 msgr2=0x5646dd43d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.576+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/2712381958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd436320 0x5646dd43d8f0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f33ac00b770 tx=0x7f33ac00bcd0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.576+0000 7f33c62b5b80 1 -- 172.21.15.70:0/2712381958 shutdown_connections 2024-06-04T05:07:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.576+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/2712381958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646dd436320 0x5646dd43d8f0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.576+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/2712381958 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5646dd427760 0x5646dd442940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.576+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/2712381958 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5646dd4b5a70 0x5646dd4b7e90 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.576+0000 7f33c62b5b80 1 --2- 172.21.15.70:0/2712381958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646dd4308a0 0x5646dd442e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.576+0000 7f33c62b5b80 1 -- 172.21.15.70:0/2712381958 >> 172.21.15.70:0/2712381958 conn(0x5646dd35c240 msgr2=0x5646dd433ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.576+0000 7f339cff9700 0 client.18276 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:07:49.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.577+0000 7f33c62b5b80 1 -- 172.21.15.70:0/2712381958 shutdown_connections 2024-06-04T05:07:49.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.577+0000 7f33c62b5b80 1 -- 172.21.15.70:0/2712381958 wait complete. 2024-06-04T05:07:49.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.577+0000 7f33c62b5b80 20 client.18276 trim_cache size 0 max 16384 2024-06-04T05:07:49.585 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:07:49.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.577+0000 7f33bba20700 1 -- 172.21.15.70:0/1393006973 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33a00b29f0 msgr2=0x7f33a00b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:49.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.577+0000 7f33bba20700 1 --2- 172.21.15.70:0/1393006973 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33a00b29f0 0x7f33a00b4eb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f33b419a9c0 tx=0x7f33b0009260 comp rx=0 tx=0).stop 2024-06-04T05:07:49.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.577+0000 7f33bba20700 1 -- 172.21.15.70:0/1393006973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33b4104200 msgr2=0x7f33b419fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:49.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.577+0000 7f33bba20700 1 --2- 172.21.15.70:0/1393006973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33b4104200 0x7f33b419fc00 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f33a400e9c0 tx=0x7f33a400ecd0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.577+0000 7f33bba20700 1 -- 172.21.15.70:0/1393006973 shutdown_connections 2024-06-04T05:07:49.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.577+0000 7f33bba20700 1 --2- 172.21.15.70:0/1393006973 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33a00b29f0 0x7f33a00b4eb0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.577+0000 7f33bba20700 1 --2- 172.21.15.70:0/1393006973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33b4108920 0x7f33b4199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.577+0000 7f33bba20700 1 --2- 172.21.15.70:0/1393006973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33b4103850 0x7f33b419f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.577+0000 7f33bba20700 1 --2- 172.21.15.70:0/1393006973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33b4104200 0x7f33b419fc00 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:49.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.577+0000 7f33bba20700 1 -- 172.21.15.70:0/1393006973 >> 172.21.15.70:0/1393006973 conn(0x7f33b407a5d0 msgr2=0x7f33b41018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:49.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.578+0000 7f33bba20700 1 -- 172.21.15.70:0/1393006973 shutdown_connections 2024-06-04T05:07:49.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:49.578+0000 7f33bba20700 1 -- 172.21.15.70:0/1393006973 wait complete. 2024-06-04T05:07:49.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 185640 1.88 MB/sec execute 402 sec latency 16.463 ms 2024-06-04T05:07:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:49 smithi070 ceph-mon[32631]: pgmap v328: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 580 KiB/s wr, 56 op/s 2024-06-04T05:07:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:49 smithi112 ceph-mon[36453]: pgmap v328: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 580 KiB/s wr, 56 op/s 2024-06-04T05:07:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:49 smithi098 ceph-mon[35569]: pgmap v328: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 580 KiB/s wr, 56 op/s 2024-06-04T05:07:50.383 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:07:50.644 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:07:50.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 186198 1.88 MB/sec execute 403 sec latency 18.592 ms 2024-06-04T05:07:51.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 186821 1.88 MB/sec execute 404 sec latency 17.940 ms 2024-06-04T05:07:51.846 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:51 smithi070 ceph-mon[32631]: pgmap v329: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 580 KiB/s wr, 56 op/s 2024-06-04T05:07:51.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:51 smithi112 ceph-mon[36453]: pgmap v329: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 580 KiB/s wr, 56 op/s 2024-06-04T05:07:51.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:51 smithi098 ceph-mon[35569]: pgmap v329: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 580 KiB/s wr, 56 op/s 2024-06-04T05:07:52.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.408+0000 7fcb1b244700 1 -- 172.21.15.70:0/1240696829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb14102e00 msgr2=0x7fcb1410f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:52.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.408+0000 7fcb1b244700 1 --2- 172.21.15.70:0/1240696829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb14102e00 0x7fcb1410f2f0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c00b0a0 tx=0x7fcb0c00b3b0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.409+0000 7fcb1b244700 1 -- 172.21.15.70:0/1240696829 shutdown_connections 2024-06-04T05:07:52.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.409+0000 7fcb1b244700 1 --2- 172.21.15.70:0/1240696829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb14102e00 0x7fcb1410f2f0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.409+0000 7fcb1b244700 1 --2- 172.21.15.70:0/1240696829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb14108520 0x7fcb14108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.409+0000 7fcb1b244700 1 --2- 172.21.15.70:0/1240696829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb14102440 0x7fcb141028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.409+0000 7fcb1b244700 1 -- 172.21.15.70:0/1240696829 >> 172.21.15.70:0/1240696829 conn(0x7fcb140793f0 msgr2=0x7fcb14079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:52.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.409+0000 7fcb1b244700 1 -- 172.21.15.70:0/1240696829 shutdown_connections 2024-06-04T05:07:52.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.409+0000 7fcb1b244700 1 -- 172.21.15.70:0/1240696829 wait complete. 2024-06-04T05:07:52.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.409+0000 7fcb1b244700 1 Processor -- start 2024-06-04T05:07:52.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.409+0000 7fcb1b244700 1 -- start start 2024-06-04T05:07:52.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.409+0000 7fcb1b244700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb14102440 0x7fcb1419a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.409+0000 7fcb1b244700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb14102e00 0x7fcb1419aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.409+0000 7fcb1b244700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb14108520 0x7fcb141a5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.410+0000 7fcb1b244700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcb141124f0 con 0x7fcb14102e00 2024-06-04T05:07:52.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.410+0000 7fcb1b244700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb14112370 con 0x7fcb14102440 2024-06-04T05:07:52.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.410+0000 7fcb1b244700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcb14112670 con 0x7fcb14108520 2024-06-04T05:07:52.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.410+0000 7fcb13fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb14102e00 0x7fcb1419aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.410+0000 7fcb13fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb14102e00 0x7fcb1419aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36518/0 (socket says 172.21.15.70:36518) 2024-06-04T05:07:52.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.410+0000 7fcb13fff700 1 -- 172.21.15.70:0/1074193426 learned_addr learned my addr 172.21.15.70:0/1074193426 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:52.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.410+0000 7fcb197e1700 1 --2- 172.21.15.70:0/1074193426 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb14108520 0x7fcb141a5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.410+0000 7fcb18fe0700 1 --2- 172.21.15.70:0/1074193426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb14102440 0x7fcb1419a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.410+0000 7fcb13fff700 1 -- 172.21.15.70:0/1074193426 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb14108520 msgr2=0x7fcb141a5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:52.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.410+0000 7fcb13fff700 1 --2- 172.21.15.70:0/1074193426 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb14108520 0x7fcb141a5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.410+0000 7fcb13fff700 1 -- 172.21.15.70:0/1074193426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb14102440 msgr2=0x7fcb1419a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:52.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.410+0000 7fcb13fff700 1 --2- 172.21.15.70:0/1074193426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb14102440 0x7fcb1419a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.410+0000 7fcb13fff700 1 -- 172.21.15.70:0/1074193426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb141a5f70 con 0x7fcb14102e00 2024-06-04T05:07:52.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.410+0000 7fcb18fe0700 1 --2- 172.21.15.70:0/1074193426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb14102440 0x7fcb1419a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:07:52.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.410+0000 7fcb13fff700 1 --2- 172.21.15.70:0/1074193426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb14102e00 0x7fcb1419aef0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fcb0000e9c0 tx=0x7fcb0000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:52.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.411+0000 7fcb11ffb700 1 -- 172.21.15.70:0/1074193426 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0000cbd0 con 0x7fcb14102e00 2024-06-04T05:07:52.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.411+0000 7fcb11ffb700 1 -- 172.21.15.70:0/1074193426 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb0000cd70 con 0x7fcb14102e00 2024-06-04T05:07:52.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.411+0000 7fcb1b244700 1 -- 172.21.15.70:0/1074193426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb141a6140 con 0x7fcb14102e00 2024-06-04T05:07:52.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.411+0000 7fcb11ffb700 1 -- 172.21.15.70:0/1074193426 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb00005540 con 0x7fcb14102e00 2024-06-04T05:07:52.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.411+0000 7fcb1b244700 1 -- 172.21.15.70:0/1074193426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb141a6620 con 0x7fcb14102e00 2024-06-04T05:07:52.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.413+0000 7fcb11ffb700 1 -- 172.21.15.70:0/1074193426 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fcb00004750 con 0x7fcb14102e00 2024-06-04T05:07:52.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.414+0000 7fcb11ffb700 1 --2- 172.21.15.70:0/1074193426 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcb040b2a70 0x7fcb040b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.414+0000 7fcb11ffb700 1 -- 172.21.15.70:0/1074193426 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7fcb000e2830 con 0x7fcb14102e00 2024-06-04T05:07:52.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.414+0000 7fcb18fe0700 1 --2- 172.21.15.70:0/1074193426 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcb040b2a70 0x7fcb040b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb25ad9b80 1 Processor -- start 2024-06-04T05:07:52.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb18fe0700 1 --2- 172.21.15.70:0/1074193426 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcb040b2a70 0x7fcb040b4f30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcb08009b50 tx=0x7fcb08009550 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:52.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb25ad9b80 1 -- start start 2024-06-04T05:07:52.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb25ad9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771b99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb25ad9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b9fc0 0x55fd771b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb25ad9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b42d0 0x55fd771b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb25ad9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fd7710dd70 con 0x55fd771b9610 2024-06-04T05:07:52.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb25ad9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fd770fd910 con 0x55fd771b42d0 2024-06-04T05:07:52.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb25ad9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fd77101950 con 0x55fd771b9fc0 2024-06-04T05:07:52.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb18fe0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771b99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb18fe0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771b99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36534/0 (socket says 172.21.15.70:36534) 2024-06-04T05:07:52.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb13fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b9fc0 0x55fd771b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb18fe0700 1 -- 172.21.15.70:0/3740573306 learned_addr learned my addr 172.21.15.70:0/3740573306 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:52.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb197e1700 1 --2- 172.21.15.70:0/3740573306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b42d0 0x55fd771b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb13fff700 1 -- 172.21.15.70:0/3740573306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b42d0 msgr2=0x55fd771b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:52.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb13fff700 1 --2- 172.21.15.70:0/3740573306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b42d0 0x55fd771b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb13fff700 1 -- 172.21.15.70:0/3740573306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 msgr2=0x55fd771b99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb13fff700 1 --2- 172.21.15.70:0/3740573306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771b99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.415+0000 7fcb13fff700 1 -- 172.21.15.70:0/3740573306 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd770a6fb0 con 0x55fd771b9fc0 2024-06-04T05:07:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.416+0000 7fcb18fe0700 1 --2- 172.21.15.70:0/3740573306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771b99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.416+0000 7fcb13fff700 1 --2- 172.21.15.70:0/3740573306 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b9fc0 0x55fd771b3cb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcb000dff50 tx=0x7fcb000a9530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82e862d4c1c1a0bf server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.416+0000 7fcafeffd700 1 -- 172.21.15.70:0/3740573306 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb000b2290 con 0x55fd771b9fc0 2024-06-04T05:07:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.416+0000 7fcafeffd700 1 -- 172.21.15.70:0/3740573306 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb000a95f0 con 0x55fd771b9fc0 2024-06-04T05:07:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.416+0000 7fcafeffd700 1 -- 172.21.15.70:0/3740573306 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb000a9f40 con 0x55fd771b9fc0 2024-06-04T05:07:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.417+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/3740573306 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b9fc0 msgr2=0x55fd771b3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.417+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/3740573306 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b9fc0 0x55fd771b3cb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcb000dff50 tx=0x7fcb000a9530 comp rx=0 tx=0).stop 2024-06-04T05:07:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.417+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/3740573306 shutdown_connections 2024-06-04T05:07:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.417+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/3740573306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b42d0 0x55fd771b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.417+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/3740573306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771b99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.417+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/3740573306 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b9fc0 0x55fd771b3cb0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.417+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/3740573306 >> 172.21.15.70:0/3740573306 conn(0x55fd770db240 msgr2=0x55fd771aa3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:52.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.417+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/3740573306 shutdown_connections 2024-06-04T05:07:52.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.417+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/3740573306 wait complete. 2024-06-04T05:07:52.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.417+0000 7fcb25ad9b80 1 -- start start 2024-06-04T05:07:52.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.417+0000 7fcb25ad9b80 1 Processor -- start 2024-06-04T05:07:52.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb25ad9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b42d0 0x55fd771bfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb25ad9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b9610 0x55fd771bd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb25ad9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9fc0 0x55fd771bd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb25ad9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fd77003fa0 con 0x55fd771b9fc0 2024-06-04T05:07:52.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb25ad9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fd77065da0 con 0x55fd771b9610 2024-06-04T05:07:52.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb25ad9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fd771c0aa0 con 0x55fd771b42d0 2024-06-04T05:07:52.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb25ad9b80 10 client.0 _ll_register_callbacks cb 0x55fd77098950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:52.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb18fe0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b42d0 0x55fd771bfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb197e1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9fc0 0x55fd771bd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb18fe0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b42d0 0x55fd771bfdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:56230/0 (socket says 172.21.15.70:56230) 2024-06-04T05:07:52.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb197e1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9fc0 0x55fd771bd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36538/0 (socket says 172.21.15.70:36538) 2024-06-04T05:07:52.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb18fe0700 1 -- 172.21.15.70:0/82433693 learned_addr learned my addr 172.21.15.70:0/82433693 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:52.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb13fff700 1 --2- 172.21.15.70:0/82433693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b9610 0x55fd771bd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb18fe0700 1 -- 172.21.15.70:0/82433693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b9610 msgr2=0x55fd771bd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:52.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb18fe0700 1 --2- 172.21.15.70:0/82433693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b9610 0x55fd771bd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb18fe0700 1 -- 172.21.15.70:0/82433693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9fc0 msgr2=0x55fd771bd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:52.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb18fe0700 1 --2- 172.21.15.70:0/82433693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9fc0 0x55fd771bd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb18fe0700 1 -- 172.21.15.70:0/82433693 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd7704d8d0 con 0x55fd771b42d0 2024-06-04T05:07:52.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.418+0000 7fcb13fff700 1 --2- 172.21.15.70:0/82433693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b9610 0x55fd771bd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:07:52.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.419+0000 7fcb197e1700 1 --2- 172.21.15.70:0/82433693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9fc0 0x55fd771bd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:52.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.419+0000 7fcb18fe0700 1 --2- 172.21.15.70:0/82433693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b42d0 0x55fd771bfdf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcb08011710 tx=0x7fcb08004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39b77e7794c2738b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:52.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.419+0000 7fcaf3fff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:07:52.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.419+0000 7fcaf3fff700 1 -- 172.21.15.70:0/82433693 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08004220 con 0x55fd771b42d0 2024-06-04T05:07:52.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.419+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/82433693 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb000a95f0 con 0x55fd771b42d0 2024-06-04T05:07:52.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.419+0000 7fcaf3fff700 1 -- 172.21.15.70:0/82433693 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7fcb080043c0 con 0x55fd771b42d0 2024-06-04T05:07:52.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.419+0000 7fcaf3fff700 1 -- 172.21.15.70:0/82433693 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fcb08003680 con 0x55fd771b42d0 2024-06-04T05:07:52.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.419+0000 7fcb25ad9b80 10 client.27484 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:52.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.419+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/82433693 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55fd770b7760 con 0x55fd771b42d0 2024-06-04T05:07:52.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.420+0000 7fcaf3fff700 1 -- 172.21.15.70:0/82433693 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb000a95f0 con 0x55fd771b42d0 2024-06-04T05:07:52.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.420+0000 7fcaf3fff700 1 -- 172.21.15.70:0/82433693 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7fcb0800f040 con 0x55fd771b42d0 2024-06-04T05:07:52.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.420+0000 7fcb25ad9b80 10 client.27484 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:52.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.420+0000 7fcb25ad9b80 10 client.27484 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:52.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.420+0000 7fcb25ad9b80 20 client.27484 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:52.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.420+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/82433693 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fd771cd7c0 0x55fd771cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.420+0000 7fcb25ad9b80 4 client.27484 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:07:52.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.420+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/82433693 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd7710fcc0 con 0x55fd771cd7c0 2024-06-04T05:07:52.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.420+0000 7fcb13fff700 1 --2- 172.21.15.70:0/82433693 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fd771cd7c0 0x55fd771cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.420+0000 7fcb13fff700 1 --2- 172.21.15.70:0/82433693 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fd771cd7c0 0x55fd771cfc10 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64b70c3b482d699b server_cookie=98033ef71a2affde in_seq=0 out_seq=0 2024-06-04T05:07:52.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.420+0000 7fcaf3fff700 10 client.27484 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:52.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.421+0000 7fcaf3fff700 1 -- 172.21.15.70:0/82433693 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fd7710fcc0 con 0x55fd771cd7c0 2024-06-04T05:07:52.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.421+0000 7fcaf3fff700 10 client.27484 handle_command_reply: tid=0 2024-06-04T05:07:52.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.421+0000 7fcb25ad9b80 1 client.27484 shutdown 2024-06-04T05:07:52.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.421+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/82433693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b42d0 msgr2=0x55fd771bfdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:52.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.421+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/82433693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b42d0 0x55fd771bfdf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcb08011710 tx=0x7fcb08004750 comp rx=0 tx=0).stop 2024-06-04T05:07:52.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.421+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/82433693 shutdown_connections 2024-06-04T05:07:52.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.421+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/82433693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9fc0 0x55fd771bd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.421+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/82433693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b42d0 0x55fd771bfdf0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.421+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/82433693 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fd771cd7c0 0x55fd771cfc10 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.422+0000 7fcaf3fff700 0 client.27484 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:52.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.422+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/82433693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b9610 0x55fd771bd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.422+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/82433693 >> 172.21.15.70:0/82433693 conn(0x55fd770db240 msgr2=0x55fd771a9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:52.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.422+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/82433693 shutdown_connections 2024-06-04T05:07:52.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.422+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/82433693 wait complete. 2024-06-04T05:07:52.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.422+0000 7fcb25ad9b80 20 client.27484 trim_cache size 0 max 16384 2024-06-04T05:07:52.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.437+0000 7fcb25ad9b80 1 Processor -- start 2024-06-04T05:07:52.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.437+0000 7fcb25ad9b80 1 -- start start 2024-06-04T05:07:52.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcb25ad9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b42d0 0x55fd771c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcb25ad9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771baab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcb25ad9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b9fc0 0x55fd771aa270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcb25ad9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcb141abbe0 con 0x55fd771b9610 2024-06-04T05:07:52.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcb25ad9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb141ad780 con 0x55fd771b9fc0 2024-06-04T05:07:52.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcb25ad9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcb141ac2b0 con 0x55fd771b42d0 2024-06-04T05:07:52.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcb13fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771baab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcb13fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771baab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36546/0 (socket says 172.21.15.70:36546) 2024-06-04T05:07:52.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcb13fff700 1 -- 172.21.15.70:0/3916491340 learned_addr learned my addr 172.21.15.70:0/3916491340 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:52.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcb18fe0700 1 --2- 172.21.15.70:0/3916491340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b42d0 0x55fd771c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcb197e1700 1 --2- 172.21.15.70:0/3916491340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b9fc0 0x55fd771aa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcb13fff700 1 -- 172.21.15.70:0/3916491340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b42d0 msgr2=0x55fd771c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:52.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcb13fff700 1 --2- 172.21.15.70:0/3916491340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b42d0 0x55fd771c9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcb13fff700 1 -- 172.21.15.70:0/3916491340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b9fc0 msgr2=0x55fd771aa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:52.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcb13fff700 1 --2- 172.21.15.70:0/3916491340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b9fc0 0x55fd771aa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcb13fff700 1 -- 172.21.15.70:0/3916491340 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd771b9c40 con 0x55fd771b9610 2024-06-04T05:07:52.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcb13fff700 1 --2- 172.21.15.70:0/3916491340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771baab0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fcb000dffb0 tx=0x7fcb00005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ec5e6e48a85042 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:52.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcafe7fc700 1 -- 172.21.15.70:0/3916491340 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb000a95f0 con 0x55fd771b9610 2024-06-04T05:07:52.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcafe7fc700 1 -- 172.21.15.70:0/3916491340 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb000c2040 con 0x55fd771b9610 2024-06-04T05:07:52.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.438+0000 7fcafe7fc700 1 -- 172.21.15.70:0/3916491340 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb000b24b0 con 0x55fd771b9610 2024-06-04T05:07:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.439+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/3916491340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 msgr2=0x55fd771baab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.439+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/3916491340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771baab0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fcb000dffb0 tx=0x7fcb00005020 comp rx=0 tx=0).stop 2024-06-04T05:07:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.439+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/3916491340 shutdown_connections 2024-06-04T05:07:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.439+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/3916491340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b9fc0 0x55fd771aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.439+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/3916491340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b42d0 0x55fd771c9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.439+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/3916491340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771baab0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.439+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/3916491340 >> 172.21.15.70:0/3916491340 conn(0x55fd770db240 msgr2=0x55fd771ba3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.439+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/3916491340 shutdown_connections 2024-06-04T05:07:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.439+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/3916491340 wait complete. 2024-06-04T05:07:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.440+0000 7fcb25ad9b80 1 -- start start 2024-06-04T05:07:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.440+0000 7fcb25ad9b80 1 Processor -- start 2024-06-04T05:07:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.440+0000 7fcb25ad9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b42d0 0x55fd771c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.440+0000 7fcb25ad9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771c6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.440+0000 7fcb25ad9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b9fc0 0x55fd771d44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.440+0000 7fcb25ad9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcb141ad080 con 0x55fd771b9610 2024-06-04T05:07:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.440+0000 7fcb25ad9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb141ac980 con 0x55fd771b42d0 2024-06-04T05:07:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.440+0000 7fcb25ad9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcb141a9ba0 con 0x55fd771b9fc0 2024-06-04T05:07:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.440+0000 7fcb25ad9b80 10 client.0 _ll_register_callbacks cb 0x55fd77098950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.440+0000 7fcb13fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.440+0000 7fcb13fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771c6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36548/0 (socket says 172.21.15.70:36548) 2024-06-04T05:07:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.440+0000 7fcb13fff700 1 -- 172.21.15.70:0/646662084 learned_addr learned my addr 172.21.15.70:0/646662084 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.440+0000 7fcb197e1700 1 --2- 172.21.15.70:0/646662084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b9fc0 0x55fd771d44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.440+0000 7fcb18fe0700 1 --2- 172.21.15.70:0/646662084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b42d0 0x55fd771c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.441+0000 7fcb13fff700 1 -- 172.21.15.70:0/646662084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b9fc0 msgr2=0x55fd771d44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.441+0000 7fcb13fff700 1 --2- 172.21.15.70:0/646662084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b9fc0 0x55fd771d44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.441+0000 7fcb13fff700 1 -- 172.21.15.70:0/646662084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b42d0 msgr2=0x55fd771c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.441+0000 7fcb13fff700 1 --2- 172.21.15.70:0/646662084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b42d0 0x55fd771c5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.441+0000 7fcb13fff700 1 -- 172.21.15.70:0/646662084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd771d6000 con 0x55fd771b9610 2024-06-04T05:07:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.441+0000 7fcb13fff700 1 --2- 172.21.15.70:0/646662084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771c6270 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fcb000e04d0 tx=0x7fcb0000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d7e649e34069139 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.441+0000 7fcaf3fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.441+0000 7fcaf3fff700 1 -- 172.21.15.70:0/646662084 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb000b2290 con 0x55fd771b9610 2024-06-04T05:07:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.441+0000 7fcaf3fff700 1 -- 172.21.15.70:0/646662084 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7fcb000a9f50 con 0x55fd771b9610 2024-06-04T05:07:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.441+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/646662084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb000c2040 con 0x55fd771b9610 2024-06-04T05:07:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.441+0000 7fcaf3fff700 1 -- 172.21.15.70:0/646662084 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fcb000ade90 con 0x55fd771b9610 2024-06-04T05:07:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.441+0000 7fcb25ad9b80 10 client.18294 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.441+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/646662084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55fd771d6000 con 0x55fd771b9610 2024-06-04T05:07:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.441+0000 7fcaf3fff700 1 -- 172.21.15.70:0/646662084 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb000c2040 con 0x55fd771b9610 2024-06-04T05:07:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.442+0000 7fcaf3fff700 1 -- 172.21.15.70:0/646662084 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7fcb00088060 con 0x55fd771b9610 2024-06-04T05:07:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.442+0000 7fcb25ad9b80 10 client.18294 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.442+0000 7fcb25ad9b80 10 client.18294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.442+0000 7fcb25ad9b80 20 client.18294 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.442+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/646662084 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fd771cd7c0 0x55fd772399a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.442+0000 7fcb25ad9b80 4 client.18294 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:07:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.442+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/646662084 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fd771094a0 con 0x55fd771cd7c0 2024-06-04T05:07:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.442+0000 7fcb18fe0700 1 --2- 172.21.15.70:0/646662084 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fd771cd7c0 0x55fd772399a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.442+0000 7fcaf3fff700 10 client.18294 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.442+0000 7fcb18fe0700 1 --2- 172.21.15.70:0/646662084 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fd771cd7c0 0x55fd772399a0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdfa0f4b601ede62 server_cookie=ac9a51525c3b7796 in_seq=0 out_seq=0 2024-06-04T05:07:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.443+0000 7fcaf3fff700 1 -- 172.21.15.70:0/646662084 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fd771094a0 con 0x55fd771cd7c0 2024-06-04T05:07:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.443+0000 7fcaf3fff700 10 client.18294 handle_command_reply: tid=0 2024-06-04T05:07:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.443+0000 7fcb25ad9b80 1 client.18294 shutdown 2024-06-04T05:07:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.443+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/646662084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 msgr2=0x55fd771c6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.443+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/646662084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771c6270 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fcb000e04d0 tx=0x7fcb0000ef10 comp rx=0 tx=0).stop 2024-06-04T05:07:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.443+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/646662084 shutdown_connections 2024-06-04T05:07:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.443+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/646662084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd771b9fc0 0x55fd771d44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.443+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/646662084 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fd771cd7c0 0x55fd772399a0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.443+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/646662084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd771b42d0 0x55fd771c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.443+0000 7fcb25ad9b80 1 --2- 172.21.15.70:0/646662084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd771b9610 0x55fd771c6270 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.443+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/646662084 >> 172.21.15.70:0/646662084 conn(0x55fd770db240 msgr2=0x55fd771bac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.443+0000 7fcaf3fff700 0 client.18294 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.443+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/646662084 shutdown_connections 2024-06-04T05:07:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.443+0000 7fcb25ad9b80 1 -- 172.21.15.70:0/646662084 wait complete. 2024-06-04T05:07:52.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.443+0000 7fcb25ad9b80 20 client.18294 trim_cache size 0 max 16384 2024-06-04T05:07:52.451 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:07:52.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.444+0000 7fcb1b244700 1 -- 172.21.15.70:0/1074193426 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcb040b2a70 msgr2=0x7fcb040b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:52.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.444+0000 7fcb1b244700 1 --2- 172.21.15.70:0/1074193426 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcb040b2a70 0x7fcb040b4f30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcb08009b50 tx=0x7fcb08009550 comp rx=0 tx=0).stop 2024-06-04T05:07:52.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.444+0000 7fcb1b244700 1 -- 172.21.15.70:0/1074193426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb14102e00 msgr2=0x7fcb1419aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:52.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.444+0000 7fcb1b244700 1 --2- 172.21.15.70:0/1074193426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb14102e00 0x7fcb1419aef0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fcb0000e9c0 tx=0x7fcb0000ecd0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.444+0000 7fcb1b244700 1 -- 172.21.15.70:0/1074193426 shutdown_connections 2024-06-04T05:07:52.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.444+0000 7fcb1b244700 1 --2- 172.21.15.70:0/1074193426 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcb040b2a70 0x7fcb040b4f30 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.444+0000 7fcb1b244700 1 --2- 172.21.15.70:0/1074193426 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb14108520 0x7fcb141a5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.444+0000 7fcb1b244700 1 --2- 172.21.15.70:0/1074193426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb14102440 0x7fcb1419a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.444+0000 7fcb1b244700 1 --2- 172.21.15.70:0/1074193426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb14102e00 0x7fcb1419aef0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:52.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.444+0000 7fcb1b244700 1 -- 172.21.15.70:0/1074193426 >> 172.21.15.70:0/1074193426 conn(0x7fcb140793f0 msgr2=0x7fcb14100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:52.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.444+0000 7fcb1b244700 1 -- 172.21.15.70:0/1074193426 shutdown_connections 2024-06-04T05:07:52.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:52.444+0000 7fcb1b244700 1 -- 172.21.15.70:0/1074193426 wait complete. 2024-06-04T05:07:52.650 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-06-04T05:07:52.650 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:07:52.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 187430 1.89 MB/sec execute 405 sec latency 21.065 ms 2024-06-04T05:07:53.245 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:07:53.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 187982 1.90 MB/sec execute 406 sec latency 16.696 ms 2024-06-04T05:07:53.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:53 smithi112 ceph-mon[36453]: pgmap v330: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-06-04T05:07:53.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:53 smithi098 ceph-mon[35569]: pgmap v330: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-06-04T05:07:54.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:53 smithi070 ceph-mon[32631]: pgmap v330: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-06-04T05:07:54.144 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:07:54.144 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:07:54.496 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:07:54.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 188073 1.89 MB/sec execute 407 sec latency 835.833 ms 2024-06-04T05:07:55.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.221+0000 7f3031499700 1 -- 172.21.15.70:0/3205561974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f302c06bc80 msgr2=0x7f302c06c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:55.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.221+0000 7f3031499700 1 --2- 172.21.15.70:0/3205561974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f302c06bc80 0x7f302c06c100 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f3020009990 tx=0x7f3020009ca0 comp rx=0 tx=0).stop 2024-06-04T05:07:55.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.221+0000 7f3031499700 1 -- 172.21.15.70:0/3205561974 shutdown_connections 2024-06-04T05:07:55.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.221+0000 7f3031499700 1 --2- 172.21.15.70:0/3205561974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f302c06c640 0x7f302c110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:55.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.221+0000 7f3031499700 1 --2- 172.21.15.70:0/3205561974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f302c1096e0 0x7f302c109ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:55.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.221+0000 7f3031499700 1 --2- 172.21.15.70:0/3205561974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f302c06bc80 0x7f302c06c100 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:55.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.221+0000 7f3031499700 1 -- 172.21.15.70:0/3205561974 >> 172.21.15.70:0/3205561974 conn(0x7f302c079240 msgr2=0x7f302c079650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:55.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.221+0000 7f3031499700 1 -- 172.21.15.70:0/3205561974 shutdown_connections 2024-06-04T05:07:55.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.221+0000 7f3031499700 1 -- 172.21.15.70:0/3205561974 wait complete. 2024-06-04T05:07:55.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.222+0000 7f3031499700 1 Processor -- start 2024-06-04T05:07:55.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.222+0000 7f3031499700 1 -- start start 2024-06-04T05:07:55.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.222+0000 7f3031499700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f302c06bc80 0x7f302c102810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:55.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.222+0000 7f3031499700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f302c06c640 0x7f302c102d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:55.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.222+0000 7f3031499700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f302c1096e0 0x7f302c103290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:55.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.222+0000 7f3031499700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f302c113660 con 0x7f302c06c640 2024-06-04T05:07:55.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.222+0000 7f3031499700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f302c1134e0 con 0x7f302c06bc80 2024-06-04T05:07:55.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.222+0000 7f3031499700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f302c1137e0 con 0x7f302c1096e0 2024-06-04T05:07:55.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.222+0000 7f302a7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f302c06c640 0x7f302c102d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:55.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.222+0000 7f302a7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f302c06c640 0x7f302c102d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36570/0 (socket says 172.21.15.70:36570) 2024-06-04T05:07:55.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.222+0000 7f302a7fc700 1 -- 172.21.15.70:0/377839755 learned_addr learned my addr 172.21.15.70:0/377839755 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:55.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.222+0000 7f302b7fe700 1 --2- 172.21.15.70:0/377839755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f302c1096e0 0x7f302c103290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:55.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.222+0000 7f302affd700 1 --2- 172.21.15.70:0/377839755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f302c06bc80 0x7f302c102810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:55.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.223+0000 7f302a7fc700 1 -- 172.21.15.70:0/377839755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f302c1096e0 msgr2=0x7f302c103290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:55.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.223+0000 7f302a7fc700 1 --2- 172.21.15.70:0/377839755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f302c1096e0 0x7f302c103290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:55.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.223+0000 7f302a7fc700 1 -- 172.21.15.70:0/377839755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f302c06bc80 msgr2=0x7f302c102810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:55.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.223+0000 7f302a7fc700 1 --2- 172.21.15.70:0/377839755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f302c06bc80 0x7f302c102810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:55.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.223+0000 7f302a7fc700 1 -- 172.21.15.70:0/377839755 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f302c1a8120 con 0x7f302c06c640 2024-06-04T05:07:55.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.223+0000 7f302b7fe700 1 --2- 172.21.15.70:0/377839755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f302c1096e0 0x7f302c103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:07:55.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.223+0000 7f302a7fc700 1 --2- 172.21.15.70:0/377839755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f302c06c640 0x7f302c102d50 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f30200057c0 tx=0x7f3020009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:55.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.223+0000 7f3013fff700 1 -- 172.21.15.70:0/377839755 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3020027070 con 0x7f302c06c640 2024-06-04T05:07:55.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.223+0000 7f3031499700 1 -- 172.21.15.70:0/377839755 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f302c1a83b0 con 0x7f302c06c640 2024-06-04T05:07:55.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.223+0000 7f3013fff700 1 -- 172.21.15.70:0/377839755 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f302000fda0 con 0x7f302c06c640 2024-06-04T05:07:55.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.223+0000 7f3013fff700 1 -- 172.21.15.70:0/377839755 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3020022ca0 con 0x7f302c06c640 2024-06-04T05:07:55.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.223+0000 7f3031499700 1 -- 172.21.15.70:0/377839755 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f302c1a88e0 con 0x7f302c06c640 2024-06-04T05:07:55.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.224+0000 7f3031499700 1 -- 172.21.15.70:0/377839755 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f302c06a230 con 0x7f302c06c640 2024-06-04T05:07:55.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.225+0000 7f3013fff700 1 -- 172.21.15.70:0/377839755 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f302002a070 con 0x7f302c06c640 2024-06-04T05:07:55.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.226+0000 7f3013fff700 1 --2- 172.21.15.70:0/377839755 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30140b29f0 0x7f30140b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:55.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.226+0000 7f3013fff700 1 -- 172.21.15.70:0/377839755 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f30200e34c0 con 0x7f302c06c640 2024-06-04T05:07:55.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.226+0000 7f302affd700 1 --2- 172.21.15.70:0/377839755 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30140b29f0 0x7f30140b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:55.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.226+0000 7f302affd700 1 --2- 172.21.15.70:0/377839755 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30140b29f0 0x7f30140b4eb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f301c00bbb0 tx=0x7f301c00b590 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:55.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.231+0000 7f3013fff700 1 -- 172.21.15.70:0/377839755 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3020013070 con 0x7f302c06c640 2024-06-04T05:07:55.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.483+0000 7f3031499700 1 -- 172.21.15.70:0/377839755 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f302c103d40 con 0x7f302c06c640 2024-06-04T05:07:55.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.484+0000 7f3013fff700 1 -- 172.21.15.70:0/377839755 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11931 (secure 0 0 0) 0x7f30200aaf00 con 0x7f302c06c640 2024-06-04T05:07:55.487 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:07:55.487 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:07:27.814997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24436,24410,24440,24448,24442]},"id":1}]} 2024-06-04T05:07:55.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.486+0000 7f3031499700 1 -- 172.21.15.70:0/377839755 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30140b29f0 msgr2=0x7f30140b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:55.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.486+0000 7f3031499700 1 --2- 172.21.15.70:0/377839755 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30140b29f0 0x7f30140b4eb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f301c00bbb0 tx=0x7f301c00b590 comp rx=0 tx=0).stop 2024-06-04T05:07:55.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.486+0000 7f3031499700 1 -- 172.21.15.70:0/377839755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f302c06c640 msgr2=0x7f302c102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:55.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.486+0000 7f3031499700 1 --2- 172.21.15.70:0/377839755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f302c06c640 0x7f302c102d50 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f30200057c0 tx=0x7f3020009f90 comp rx=0 tx=0).stop 2024-06-04T05:07:55.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.486+0000 7f3031499700 1 -- 172.21.15.70:0/377839755 shutdown_connections 2024-06-04T05:07:55.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.486+0000 7f3031499700 1 --2- 172.21.15.70:0/377839755 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30140b29f0 0x7f30140b4eb0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:55.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.486+0000 7f3031499700 1 --2- 172.21.15.70:0/377839755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f302c1096e0 0x7f302c103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:55.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.486+0000 7f3031499700 1 --2- 172.21.15.70:0/377839755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f302c06bc80 0x7f302c102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:55.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.486+0000 7f3031499700 1 --2- 172.21.15.70:0/377839755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f302c06c640 0x7f302c102d50 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:55.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.486+0000 7f3031499700 1 -- 172.21.15.70:0/377839755 >> 172.21.15.70:0/377839755 conn(0x7f302c079240 msgr2=0x7f302c1014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:55.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.487+0000 7f3031499700 1 -- 172.21.15.70:0/377839755 shutdown_connections 2024-06-04T05:07:55.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:55.487+0000 7f3031499700 1 -- 172.21.15.70:0/377839755 wait complete. 2024-06-04T05:07:55.490 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 34 2024-06-04T05:07:55.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:55 smithi070 ceph-mon[32631]: pgmap v331: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 525 KiB/s wr, 71 op/s 2024-06-04T05:07:55.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:55 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/377839755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:07:55.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 188073 1.89 MB/sec execute 408 sec latency 1836.083 ms 2024-06-04T05:07:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:55 smithi112 ceph-mon[36453]: pgmap v331: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 525 KiB/s wr, 71 op/s 2024-06-04T05:07:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:55 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/377839755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:07:55.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:55 smithi098 ceph-mon[35569]: pgmap v331: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 525 KiB/s wr, 71 op/s 2024-06-04T05:07:55.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:55 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/377839755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:07:56.357 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 130 --await-for 45.0' 2024-06-04T05:07:56.357 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 130 --await-for 45.0 2024-06-04T05:07:56.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 188073 1.88 MB/sec execute 409 sec latency 2836.133 ms 2024-06-04T05:07:57.492 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:07:57.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.491+0000 7f0fd3d8a700 1 -- 172.21.15.70:0/1596318076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0fcc104200 msgr2=0x7f0fcc1081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:57.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.491+0000 7f0fd3d8a700 1 --2- 172.21.15.70:0/1596318076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0fcc104200 0x7f0fcc1081f0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0099c0 tx=0x7f0fbc009cd0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.491+0000 7f0fd3d8a700 1 -- 172.21.15.70:0/1596318076 shutdown_connections 2024-06-04T05:07:57.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.491+0000 7f0fd3d8a700 1 --2- 172.21.15.70:0/1596318076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0fcc108920 0x7f0fcc1104a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.491+0000 7f0fd3d8a700 1 --2- 172.21.15.70:0/1596318076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fcc103850 0x7f0fcc103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.491+0000 7f0fd3d8a700 1 --2- 172.21.15.70:0/1596318076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0fcc104200 0x7f0fcc1081f0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.491+0000 7f0fd3d8a700 1 -- 172.21.15.70:0/1596318076 >> 172.21.15.70:0/1596318076 conn(0x7f0fcc07a5d0 msgr2=0x7f0fcc07a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:57.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.491+0000 7f0fd3d8a700 1 -- 172.21.15.70:0/1596318076 shutdown_connections 2024-06-04T05:07:57.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.491+0000 7f0fd3d8a700 1 -- 172.21.15.70:0/1596318076 wait complete. 2024-06-04T05:07:57.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.492+0000 7f0fd3d8a700 1 Processor -- start 2024-06-04T05:07:57.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.492+0000 7f0fd3d8a700 1 -- start start 2024-06-04T05:07:57.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.492+0000 7f0fd3d8a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0fcc103850 0x7f0fcc19f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.492+0000 7f0fd3d8a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fcc104200 0x7f0fcc19fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.492+0000 7f0fd3d8a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0fcc108920 0x7f0fcc199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.492+0000 7f0fd3d8a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc1135f0 con 0x7f0fcc108920 2024-06-04T05:07:57.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.492+0000 7f0fd3d8a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc113470 con 0x7f0fcc104200 2024-06-04T05:07:57.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.492+0000 7f0fd3d8a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc113770 con 0x7f0fcc103850 2024-06-04T05:07:57.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.492+0000 7f0fd1325700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fcc104200 0x7f0fcc19fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.492+0000 7f0fd2327700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0fcc108920 0x7f0fcc199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.492+0000 7f0fd1325700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fcc104200 0x7f0fcc19fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33478/0 (socket says 172.21.15.70:33478) 2024-06-04T05:07:57.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.492+0000 7f0fd1325700 1 -- 172.21.15.70:0/3501525556 learned_addr learned my addr 172.21.15.70:0/3501525556 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:57.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.492+0000 7f0fd2327700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0fcc108920 0x7f0fcc199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45318/0 (socket says 172.21.15.70:45318) 2024-06-04T05:07:57.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.493+0000 7f0fd1b26700 1 --2- 172.21.15.70:0/3501525556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0fcc103850 0x7f0fcc19f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.493+0000 7f0fd2327700 1 -- 172.21.15.70:0/3501525556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0fcc103850 msgr2=0x7f0fcc19f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:57.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.493+0000 7f0fd2327700 1 --2- 172.21.15.70:0/3501525556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0fcc103850 0x7f0fcc19f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.493+0000 7f0fd2327700 1 -- 172.21.15.70:0/3501525556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fcc104200 msgr2=0x7f0fcc19fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:57.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.493+0000 7f0fd2327700 1 --2- 172.21.15.70:0/3501525556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fcc104200 0x7f0fcc19fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.493+0000 7f0fd2327700 1 -- 172.21.15.70:0/3501525556 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fcc19a070 con 0x7f0fcc108920 2024-06-04T05:07:57.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.493+0000 7f0fd1325700 1 --2- 172.21.15.70:0/3501525556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fcc104200 0x7f0fcc19fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:57.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.493+0000 7f0fd2327700 1 --2- 172.21.15.70:0/3501525556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0fcc108920 0x7f0fcc199910 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400ab40 tx=0x7f0fc400ae50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:57.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.493+0000 7f0fc2ffd700 1 -- 172.21.15.70:0/3501525556 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4017020 con 0x7f0fcc108920 2024-06-04T05:07:57.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.493+0000 7f0fd3d8a700 1 -- 172.21.15.70:0/3501525556 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fcc19a360 con 0x7f0fcc108920 2024-06-04T05:07:57.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.493+0000 7f0fc2ffd700 1 -- 172.21.15.70:0/3501525556 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0fc4004bb0 con 0x7f0fcc108920 2024-06-04T05:07:57.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.494+0000 7f0fc2ffd700 1 -- 172.21.15.70:0/3501525556 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4005280 con 0x7f0fcc108920 2024-06-04T05:07:57.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.494+0000 7f0fd3d8a700 1 -- 172.21.15.70:0/3501525556 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fcc1a6620 con 0x7f0fcc108920 2024-06-04T05:07:57.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.496+0000 7f0fc2ffd700 1 -- 172.21.15.70:0/3501525556 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f0fc400c580 con 0x7f0fcc108920 2024-06-04T05:07:57.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.496+0000 7f0fc2ffd700 1 --2- 172.21.15.70:0/3501525556 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0fb80b29f0 0x7f0fb80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.497+0000 7f0fd1b26700 1 --2- 172.21.15.70:0/3501525556 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0fb80b29f0 0x7f0fb80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.497+0000 7f0fc2ffd700 1 -- 172.21.15.70:0/3501525556 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f0fc40e1ba0 con 0x7f0fcc108920 2024-06-04T05:07:57.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.497+0000 7f0fd1b26700 1 --2- 172.21.15.70:0/3501525556 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0fb80b29f0 0x7f0fb80b4eb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8005950 tx=0x7f0fc800a300 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:57.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.498+0000 7f0fde61fb80 1 Processor -- start 2024-06-04T05:07:57.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.498+0000 7f0fde61fb80 1 -- start start 2024-06-04T05:07:57.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.498+0000 7f0fde61fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de83a0 0x563a25de8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.498+0000 7f0fde61fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25de2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.498+0000 7f0fde61fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de32e0 0x563a25def100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.498+0000 7f0fde61fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563a25d40df0 con 0x563a25de83a0 2024-06-04T05:07:57.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.498+0000 7f0fde61fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563a25d30990 con 0x563a25de32e0 2024-06-04T05:07:57.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.498+0000 7f0fde61fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563a25d349d0 con 0x563a25de2920 2024-06-04T05:07:57.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.498+0000 7f0fd1b26700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de83a0 0x563a25de8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.498+0000 7f0fd1b26700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de83a0 0x563a25de8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45322/0 (socket says 172.21.15.70:45322) 2024-06-04T05:07:57.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.498+0000 7f0fd1b26700 1 -- 172.21.15.70:0/1563013315 learned_addr learned my addr 172.21.15.70:0/1563013315 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:57.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.499+0000 7f0fd1325700 1 --2- 172.21.15.70:0/1563013315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25de2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.499+0000 7f0fd2327700 1 --2- 172.21.15.70:0/1563013315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de32e0 0x563a25def100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.499+0000 7f0fd1b26700 1 -- 172.21.15.70:0/1563013315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 msgr2=0x563a25de2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:57.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.499+0000 7f0fd1b26700 1 --2- 172.21.15.70:0/1563013315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25de2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.499+0000 7f0fd1b26700 1 -- 172.21.15.70:0/1563013315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de32e0 msgr2=0x563a25def100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:57.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.499+0000 7f0fd1b26700 1 --2- 172.21.15.70:0/1563013315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de32e0 0x563a25def100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.499+0000 7f0fd1b26700 1 -- 172.21.15.70:0/1563013315 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a25cda030 con 0x563a25de83a0 2024-06-04T05:07:57.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.499+0000 7f0fd2327700 1 --2- 172.21.15.70:0/1563013315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de32e0 0x563a25def100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:07:57.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.499+0000 7f0fd1b26700 1 --2- 172.21.15.70:0/1563013315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de83a0 0x563a25de8780 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f0fc800fda0 tx=0x7f0fc8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9631a7a0d3584bd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:57.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.499+0000 7f0fb7fff700 1 -- 172.21.15.70:0/1563013315 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc8016070 con 0x563a25de83a0 2024-06-04T05:07:57.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.499+0000 7f0fb7fff700 1 -- 172.21.15.70:0/1563013315 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0fc800fe80 con 0x563a25de83a0 2024-06-04T05:07:57.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.499+0000 7f0fb7fff700 1 -- 172.21.15.70:0/1563013315 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc8025e30 con 0x563a25de83a0 2024-06-04T05:07:57.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.500+0000 7f0fde61fb80 1 -- 172.21.15.70:0/1563013315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de83a0 msgr2=0x563a25de8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:57.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.500+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/1563013315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de83a0 0x563a25de8780 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f0fc800fda0 tx=0x7f0fc8012000 comp rx=0 tx=0).stop 2024-06-04T05:07:57.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.500+0000 7f0fde61fb80 1 -- 172.21.15.70:0/1563013315 shutdown_connections 2024-06-04T05:07:57.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.500+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/1563013315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de32e0 0x563a25def100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.500+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/1563013315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de83a0 0x563a25de8780 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.500+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/1563013315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25de2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.500+0000 7f0fde61fb80 1 -- 172.21.15.70:0/1563013315 >> 172.21.15.70:0/1563013315 conn(0x563a25d0e2c0 msgr2=0x563a25ddd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:57.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.501+0000 7f0fde61fb80 1 -- 172.21.15.70:0/1563013315 shutdown_connections 2024-06-04T05:07:57.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.501+0000 7f0fde61fb80 1 -- 172.21.15.70:0/1563013315 wait complete. 2024-06-04T05:07:57.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.501+0000 7f0fde61fb80 1 -- start start 2024-06-04T05:07:57.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.501+0000 7f0fde61fb80 1 Processor -- start 2024-06-04T05:07:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fde61fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25deedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fde61fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de32e0 0x563a25dec0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fde61fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de83a0 0x563a25dec5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fde61fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563a25c37020 con 0x563a25de32e0 2024-06-04T05:07:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fde61fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563a25c98e20 con 0x563a25de83a0 2024-06-04T05:07:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fde61fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563a25def760 con 0x563a25de2920 2024-06-04T05:07:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fd1325700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de32e0 0x563a25dec0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fd1325700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de32e0 0x563a25dec0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45336/0 (socket says 172.21.15.70:45336) 2024-06-04T05:07:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fde61fb80 10 client.0 _ll_register_callbacks cb 0x563a25ccb9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fd1325700 1 -- 172.21.15.70:0/4218712980 learned_addr learned my addr 172.21.15.70:0/4218712980 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fd1b26700 1 --2- 172.21.15.70:0/4218712980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25deedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fd2327700 1 --2- 172.21.15.70:0/4218712980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de83a0 0x563a25dec5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fd1b26700 1 -- 172.21.15.70:0/4218712980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de83a0 msgr2=0x563a25dec5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fd1b26700 1 --2- 172.21.15.70:0/4218712980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de83a0 0x563a25dec5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fd1b26700 1 -- 172.21.15.70:0/4218712980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de32e0 msgr2=0x563a25dec0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fd1b26700 1 --2- 172.21.15.70:0/4218712980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de32e0 0x563a25dec0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fd1b26700 1 -- 172.21.15.70:0/4218712980 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a25c80950 con 0x563a25de2920 2024-06-04T05:07:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fd2327700 1 --2- 172.21.15.70:0/4218712980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de83a0 0x563a25dec5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:07:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fd1b26700 1 --2- 172.21.15.70:0/4218712980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25deedb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0fc800fda0 tx=0x7f0fc8026c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd71530f8046649d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fb57fa700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:07:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fb57fa700 1 -- 172.21.15.70:0/4218712980 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc8010cd0 con 0x563a25de2920 2024-06-04T05:07:57.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fde61fb80 1 -- 172.21.15.70:0/4218712980 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563a25d42d40 con 0x563a25de2920 2024-06-04T05:07:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.502+0000 7f0fb57fa700 1 -- 172.21.15.70:0/4218712980 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f0fc801c420 con 0x563a25de2920 2024-06-04T05:07:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.503+0000 7f0fb57fa700 1 -- 172.21.15.70:0/4218712980 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0fc8004870 con 0x563a25de2920 2024-06-04T05:07:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.503+0000 7f0fde61fb80 10 client.27505 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.503+0000 7f0fde61fb80 1 -- 172.21.15.70:0/4218712980 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563a25cea7e0 con 0x563a25de2920 2024-06-04T05:07:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.503+0000 7f0fb57fa700 1 -- 172.21.15.70:0/4218712980 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x563a25d42d40 con 0x563a25de2920 2024-06-04T05:07:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.503+0000 7f0fb57fa700 1 -- 172.21.15.70:0/4218712980 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f0fc8005350 con 0x563a25de2920 2024-06-04T05:07:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.503+0000 7f0fde61fb80 10 client.27505 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.503+0000 7f0fde61fb80 10 client.27505 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.503+0000 7f0fde61fb80 20 client.27505 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.503+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/4218712980 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563a25dfc200 0x563a25dfe5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.503+0000 7f0fde61fb80 4 client.27505 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:07:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.503+0000 7f0fde61fb80 1 -- 172.21.15.70:0/4218712980 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a25d3c520 con 0x563a25dfc200 2024-06-04T05:07:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.503+0000 7f0fd1325700 1 --2- 172.21.15.70:0/4218712980 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563a25dfc200 0x563a25dfe5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.504+0000 7f0fd1325700 1 --2- 172.21.15.70:0/4218712980 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563a25dfc200 0x563a25dfe5f0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32aa98cff1835365 server_cookie=d4b5fb021c0fff41 in_seq=0 out_seq=0 2024-06-04T05:07:57.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.504+0000 7f0fb57fa700 10 client.27505 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:57.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.504+0000 7f0fb57fa700 1 -- 172.21.15.70:0/4218712980 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563a25d3c520 con 0x563a25dfc200 2024-06-04T05:07:57.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.504+0000 7f0fb57fa700 10 client.27505 handle_command_reply: tid=0 2024-06-04T05:07:57.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.504+0000 7f0fde61fb80 1 client.27505 shutdown 2024-06-04T05:07:57.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.504+0000 7f0fde61fb80 1 -- 172.21.15.70:0/4218712980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 msgr2=0x563a25deedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:57.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.504+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/4218712980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25deedb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0fc800fda0 tx=0x7f0fc8026c50 comp rx=0 tx=0).stop 2024-06-04T05:07:57.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.505+0000 7f0fde61fb80 1 -- 172.21.15.70:0/4218712980 shutdown_connections 2024-06-04T05:07:57.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.505+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/4218712980 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563a25dfc200 0x563a25dfe5f0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.505+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/4218712980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de83a0 0x563a25dec5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.505+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/4218712980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25deedb0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.505+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/4218712980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de32e0 0x563a25dec0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.505+0000 7f0fde61fb80 1 -- 172.21.15.70:0/4218712980 >> 172.21.15.70:0/4218712980 conn(0x563a25d0e2c0 msgr2=0x563a25ddcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:57.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.505+0000 7f0fb57fa700 0 client.27505 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:57.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.505+0000 7f0fde61fb80 1 -- 172.21.15.70:0/4218712980 shutdown_connections 2024-06-04T05:07:57.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.505+0000 7f0fde61fb80 1 -- 172.21.15.70:0/4218712980 wait complete. 2024-06-04T05:07:57.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.505+0000 7f0fde61fb80 20 client.27505 trim_cache size 0 max 16384 2024-06-04T05:07:57.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.527+0000 7f0fde61fb80 1 Processor -- start 2024-06-04T05:07:57.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.527+0000 7f0fde61fb80 1 -- start start 2024-06-04T05:07:57.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.528+0000 7f0fde61fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25df87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.528+0000 7f0fde61fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de32e0 0x563a25ddd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.528+0000 7f0fde61fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de83a0 0x563a25dedff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.528+0000 7f0fde61fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc1abc50 con 0x563a25de83a0 2024-06-04T05:07:57.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.528+0000 7f0fde61fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc1ad7f0 con 0x563a25de32e0 2024-06-04T05:07:57.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.528+0000 7f0fde61fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc1ac320 con 0x563a25de2920 2024-06-04T05:07:57.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.528+0000 7f0fd1b26700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25df87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.528+0000 7f0fd1b26700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25df87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59346/0 (socket says 172.21.15.70:59346) 2024-06-04T05:07:57.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.528+0000 7f0fd1b26700 1 -- 172.21.15.70:0/2650967216 learned_addr learned my addr 172.21.15.70:0/2650967216 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:57.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.528+0000 7f0fd2327700 1 --2- 172.21.15.70:0/2650967216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de83a0 0x563a25dedff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.528+0000 7f0fd1325700 1 --2- 172.21.15.70:0/2650967216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de32e0 0x563a25ddd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.528+0000 7f0fd2327700 1 -- 172.21.15.70:0/2650967216 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 msgr2=0x563a25df87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:57.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.528+0000 7f0fd2327700 1 --2- 172.21.15.70:0/2650967216 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25df87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.528+0000 7f0fd2327700 1 -- 172.21.15.70:0/2650967216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de32e0 msgr2=0x563a25ddd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:57.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.528+0000 7f0fd2327700 1 --2- 172.21.15.70:0/2650967216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de32e0 0x563a25ddd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.528+0000 7f0fd2327700 1 -- 172.21.15.70:0/2650967216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a25def5b0 con 0x563a25de83a0 2024-06-04T05:07:57.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.529+0000 7f0fd2327700 1 --2- 172.21.15.70:0/2650967216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de83a0 0x563a25dedff0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f0fc40df2c0 tx=0x7f0fc40a86d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa021d87d9bcf3e5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:57.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.529+0000 7f0fb77fe700 1 -- 172.21.15.70:0/2650967216 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc40a8ee0 con 0x563a25de83a0 2024-06-04T05:07:57.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.529+0000 7f0fb77fe700 1 -- 172.21.15.70:0/2650967216 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0fc40ad290 con 0x563a25de83a0 2024-06-04T05:07:57.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.529+0000 7f0fb77fe700 1 -- 172.21.15.70:0/2650967216 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc40a93a0 con 0x563a25de83a0 2024-06-04T05:07:57.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.530+0000 7f0fde61fb80 1 -- 172.21.15.70:0/2650967216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de83a0 msgr2=0x563a25dedff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:57.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.530+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/2650967216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de83a0 0x563a25dedff0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f0fc40df2c0 tx=0x7f0fc40a86d0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.530+0000 7f0fde61fb80 1 -- 172.21.15.70:0/2650967216 shutdown_connections 2024-06-04T05:07:57.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.530+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/2650967216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de83a0 0x563a25dedff0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.530+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/2650967216 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25df87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.530+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/2650967216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de32e0 0x563a25ddd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.530+0000 7f0fde61fb80 1 -- 172.21.15.70:0/2650967216 >> 172.21.15.70:0/2650967216 conn(0x563a25d0e2c0 msgr2=0x563a25ded8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.530+0000 7f0fde61fb80 1 -- 172.21.15.70:0/2650967216 shutdown_connections 2024-06-04T05:07:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.530+0000 7f0fde61fb80 1 -- 172.21.15.70:0/2650967216 wait complete. 2024-06-04T05:07:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.530+0000 7f0fde61fb80 1 -- start start 2024-06-04T05:07:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.530+0000 7f0fde61fb80 1 Processor -- start 2024-06-04T05:07:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fde61fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25df49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fde61fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de32e0 0x563a25df4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fde61fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de83a0 0x563a25def9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fde61fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc1ad0f0 con 0x563a25de32e0 2024-06-04T05:07:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fde61fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc1ac9f0 con 0x563a25de83a0 2024-06-04T05:07:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fde61fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc1a9c30 con 0x563a25de2920 2024-06-04T05:07:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fde61fb80 10 client.0 _ll_register_callbacks cb 0x563a25ccb9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fd1325700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de32e0 0x563a25df4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fd1325700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de32e0 0x563a25df4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45346/0 (socket says 172.21.15.70:45346) 2024-06-04T05:07:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fd1325700 1 -- 172.21.15.70:0/3856184279 learned_addr learned my addr 172.21.15.70:0/3856184279 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fd1b26700 1 --2- 172.21.15.70:0/3856184279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25df49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fd2327700 1 --2- 172.21.15.70:0/3856184279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de83a0 0x563a25def9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fd1325700 1 -- 172.21.15.70:0/3856184279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 msgr2=0x563a25df49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fd1325700 1 --2- 172.21.15.70:0/3856184279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25df49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fd1325700 1 -- 172.21.15.70:0/3856184279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de83a0 msgr2=0x563a25def9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fd1325700 1 --2- 172.21.15.70:0/3856184279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de83a0 0x563a25def9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fd1325700 1 -- 172.21.15.70:0/3856184279 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a25df0050 con 0x563a25de32e0 2024-06-04T05:07:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fd1b26700 1 --2- 172.21.15.70:0/3856184279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25df49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:07:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fd2327700 1 --2- 172.21.15.70:0/3856184279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de83a0 0x563a25def9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:07:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fd1325700 1 --2- 172.21.15.70:0/3856184279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de32e0 0x563a25df4f30 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009990 tx=0x7f0fbc005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b823eef59dc86aa server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fb57fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fb57fa700 1 -- 172.21.15.70:0/3856184279 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc005680 con 0x563a25de32e0 2024-06-04T05:07:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fb57fa700 1 -- 172.21.15.70:0/3856184279 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f0fbc003680 con 0x563a25de32e0 2024-06-04T05:07:57.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.531+0000 7f0fde61fb80 1 -- 172.21.15.70:0/3856184279 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fc40ad290 con 0x563a25de32e0 2024-06-04T05:07:57.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.532+0000 7f0fb57fa700 1 -- 172.21.15.70:0/3856184279 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0fbc03b050 con 0x563a25de32e0 2024-06-04T05:07:57.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.532+0000 7f0fde61fb80 10 client.18318 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:57.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.532+0000 7f0fde61fb80 1 -- 172.21.15.70:0/3856184279 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563a25e67730 con 0x563a25de32e0 2024-06-04T05:07:57.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.532+0000 7f0fb57fa700 1 -- 172.21.15.70:0/3856184279 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0fc40ad290 con 0x563a25de32e0 2024-06-04T05:07:57.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.532+0000 7f0fb57fa700 1 -- 172.21.15.70:0/3856184279 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f0fbc02c820 con 0x563a25de32e0 2024-06-04T05:07:57.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.532+0000 7f0fde61fb80 10 client.18318 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:57.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.532+0000 7f0fde61fb80 10 client.18318 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:57.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.532+0000 7f0fde61fb80 20 client.18318 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:57.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.532+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/3856184279 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563a25e67ea0 0x563a25e6a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:57.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.532+0000 7f0fde61fb80 4 client.18318 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:07:57.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.532+0000 7f0fde61fb80 1 -- 172.21.15.70:0/3856184279 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563a25bb32d0 con 0x563a25e67ea0 2024-06-04T05:07:57.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.532+0000 7f0fd1b26700 1 --2- 172.21.15.70:0/3856184279 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563a25e67ea0 0x563a25e6a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:57.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.533+0000 7f0fd1b26700 1 --2- 172.21.15.70:0/3856184279 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563a25e67ea0 0x563a25e6a2c0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d0708ae3d7b966f server_cookie=85a5349b2baeda6f in_seq=0 out_seq=0 2024-06-04T05:07:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.533+0000 7f0fb57fa700 10 client.18318 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.533+0000 7f0fb57fa700 1 -- 172.21.15.70:0/3856184279 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563a25bb32d0 con 0x563a25e67ea0 2024-06-04T05:07:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.533+0000 7f0fb57fa700 10 client.18318 handle_command_reply: tid=0 2024-06-04T05:07:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.533+0000 7f0fde61fb80 1 client.18318 shutdown 2024-06-04T05:07:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.533+0000 7f0fde61fb80 1 -- 172.21.15.70:0/3856184279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de32e0 msgr2=0x563a25df4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.533+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/3856184279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de32e0 0x563a25df4f30 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009990 tx=0x7f0fbc005fb0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fde61fb80 1 -- 172.21.15.70:0/3856184279 shutdown_connections 2024-06-04T05:07:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/3856184279 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563a25e67ea0 0x563a25e6a2c0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/3856184279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a25de83a0 0x563a25def9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/3856184279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563a25de2920 0x563a25df49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fb57fa700 0 client.18318 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fde61fb80 1 --2- 172.21.15.70:0/3856184279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a25de32e0 0x563a25df4f30 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fde61fb80 1 -- 172.21.15.70:0/3856184279 >> 172.21.15.70:0/3856184279 conn(0x563a25d0e2c0 msgr2=0x563a25dee170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fde61fb80 1 -- 172.21.15.70:0/3856184279 shutdown_connections 2024-06-04T05:07:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fde61fb80 1 -- 172.21.15.70:0/3856184279 wait complete. 2024-06-04T05:07:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fde61fb80 20 client.18318 trim_cache size 0 max 16384 2024-06-04T05:07:57.541 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:07:57.541 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:07:57.541 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "45bb262b-4741-4606-a59b-8159ac404175", 2024-06-04T05:07:57.541 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:07:57.541 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:07:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fd3d8a700 1 -- 172.21.15.70:0/3501525556 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0fb80b29f0 msgr2=0x7f0fb80b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fd3d8a700 1 --2- 172.21.15.70:0/3501525556 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0fb80b29f0 0x7f0fb80b4eb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8005950 tx=0x7f0fc800a300 comp rx=0 tx=0).stop 2024-06-04T05:07:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fd3d8a700 1 -- 172.21.15.70:0/3501525556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0fcc108920 msgr2=0x7f0fcc199910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fd3d8a700 1 --2- 172.21.15.70:0/3501525556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0fcc108920 0x7f0fcc199910 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400ab40 tx=0x7f0fc400ae50 comp rx=0 tx=0).stop 2024-06-04T05:07:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fd3d8a700 1 -- 172.21.15.70:0/3501525556 shutdown_connections 2024-06-04T05:07:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fd3d8a700 1 --2- 172.21.15.70:0/3501525556 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0fb80b29f0 0x7f0fb80b4eb0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fd3d8a700 1 --2- 172.21.15.70:0/3501525556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0fcc108920 0x7f0fcc199910 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fd3d8a700 1 --2- 172.21.15.70:0/3501525556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0fcc103850 0x7f0fcc19f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fd3d8a700 1 --2- 172.21.15.70:0/3501525556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fcc104200 0x7f0fcc19fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fd3d8a700 1 -- 172.21.15.70:0/3501525556 >> 172.21.15.70:0/3501525556 conn(0x7f0fcc07a5d0 msgr2=0x7f0fcc1019a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fd3d8a700 1 -- 172.21.15.70:0/3501525556 shutdown_connections 2024-06-04T05:07:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:57.534+0000 7f0fd3d8a700 1 -- 172.21.15.70:0/3501525556 wait complete. 2024-06-04T05:07:57.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 188223 1.88 MB/sec execute 410 sec latency 3147.706 ms 2024-06-04T05:07:57.823 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:57 smithi070 ceph-mon[32631]: pgmap v332: 97 pgs: 97 active+clean; 119 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-06-04T05:07:57.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:57 smithi112 ceph-mon[36453]: pgmap v332: 97 pgs: 97 active+clean; 119 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-06-04T05:07:57.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:57 smithi098 ceph-mon[35569]: pgmap v332: 97 pgs: 97 active+clean; 119 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-06-04T05:07:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.301+0000 7f003401d640 1 -- 172.21.15.70:0/1728646241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f002c107940 msgr2=0x7f002c107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.301+0000 7f003401d640 1 --2- 172.21.15.70:0/1728646241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f002c107940 0x7f002c107d20 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f0028009930 tx=0x7f002802a420 comp rx=0 tx=0).stop 2024-06-04T05:07:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.301+0000 7f003401d640 1 -- 172.21.15.70:0/1728646241 shutdown_connections 2024-06-04T05:07:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.301+0000 7f003401d640 1 --2- 172.21.15.70:0/1728646241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f002c101980 0x7f002c101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.301+0000 7f003401d640 1 --2- 172.21.15.70:0/1728646241 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f002c102340 0x7f002c10e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.301+0000 7f003401d640 1 --2- 172.21.15.70:0/1728646241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f002c107940 0x7f002c107d20 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.301+0000 7f003401d640 1 -- 172.21.15.70:0/1728646241 >> 172.21.15.70:0/1728646241 conn(0x7f002c078ad0 msgr2=0x7f002c078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.301+0000 7f003401d640 1 -- 172.21.15.70:0/1728646241 shutdown_connections 2024-06-04T05:07:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.301+0000 7f003401d640 1 -- 172.21.15.70:0/1728646241 wait complete. 2024-06-04T05:07:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.302+0000 7f003401d640 1 Processor -- start 2024-06-04T05:07:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.302+0000 7f003401d640 1 -- start start 2024-06-04T05:07:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.302+0000 7f003401d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f002c101980 0x7f002c196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.302+0000 7f003401d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f002c102340 0x7f002c197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.302+0000 7f003401d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f002c107940 0x7f002c19b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.302+0000 7f003401d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f002c1109a0 con 0x7f002c107940 2024-06-04T05:07:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.302+0000 7f003401d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f002c110820 con 0x7f002c102340 2024-06-04T05:07:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.302+0000 7f003401d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f002c110b20 con 0x7f002c101980 2024-06-04T05:07:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.302+0000 7f00325ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f002c107940 0x7f002c19b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.303+0000 7f0031db9640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f002c101980 0x7f002c196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.303+0000 7f00325ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f002c107940 0x7f002c19b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45370/0 (socket says 172.21.15.70:45370) 2024-06-04T05:07:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.303+0000 7f00325ba640 1 -- 172.21.15.70:0/3441988523 learned_addr learned my addr 172.21.15.70:0/3441988523 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.303+0000 7f0031db9640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f002c101980 0x7f002c196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59380/0 (socket says 172.21.15.70:59380) 2024-06-04T05:07:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.303+0000 7f00315b8640 1 --2- 172.21.15.70:0/3441988523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f002c102340 0x7f002c197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.303+0000 7f00325ba640 1 -- 172.21.15.70:0/3441988523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f002c101980 msgr2=0x7f002c196f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.303+0000 7f00325ba640 1 --2- 172.21.15.70:0/3441988523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f002c101980 0x7f002c196f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.303+0000 7f00325ba640 1 -- 172.21.15.70:0/3441988523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f002c102340 msgr2=0x7f002c197440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.303+0000 7f00325ba640 1 --2- 172.21.15.70:0/3441988523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f002c102340 0x7f002c197440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.303+0000 7f00325ba640 1 -- 172.21.15.70:0/3441988523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f002c19bef0 con 0x7f002c107940 2024-06-04T05:07:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.303+0000 7f00315b8640 1 --2- 172.21.15.70:0/3441988523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f002c102340 0x7f002c197440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:07:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.303+0000 7f00325ba640 1 --2- 172.21.15.70:0/3441988523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f002c107940 0x7f002c19b7c0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f002400cbb0 tx=0x7f0024010490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.303+0000 7f0022ffd640 1 -- 172.21.15.70:0/3441988523 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002401a030 con 0x7f002c107940 2024-06-04T05:07:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.303+0000 7f0022ffd640 1 -- 172.21.15.70:0/3441988523 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0024004b20 con 0x7f002c107940 2024-06-04T05:07:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.303+0000 7f003401d640 1 -- 172.21.15.70:0/3441988523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f002c19c1e0 con 0x7f002c107940 2024-06-04T05:07:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.303+0000 7f0022ffd640 1 -- 172.21.15.70:0/3441988523 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00240057e0 con 0x7f002c107940 2024-06-04T05:07:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.303+0000 7f003401d640 1 -- 172.21.15.70:0/3441988523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f002c1a3a70 con 0x7f002c107940 2024-06-04T05:07:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.306+0000 7f0036e0c740 1 Processor -- start 2024-06-04T05:07:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.306+0000 7f0022ffd640 1 -- 172.21.15.70:0/3441988523 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f002401d070 con 0x7f002c107940 2024-06-04T05:07:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.306+0000 7f0036e0c740 1 -- start start 2024-06-04T05:07:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.306+0000 7f0036e0c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d43820 0x55d569d43c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.306+0000 7f0036e0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d44140 0x55d569d4e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.306+0000 7f0036e0c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d4ea00 0x55d569d50df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.306+0000 7f0036e0c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d569cb5c30 con 0x55d569d44140 2024-06-04T05:07:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.306+0000 7f0036e0c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d569ca57d0 con 0x55d569d4ea00 2024-06-04T05:07:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.306+0000 7f0036e0c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d569ca9810 con 0x55d569d43820 2024-06-04T05:07:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.306+0000 7f0031db9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d44140 0x55d569d4e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.307+0000 7f00315b8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d4ea00 0x55d569d50df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.307+0000 7f0031db9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d44140 0x55d569d4e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45380/0 (socket says 172.21.15.70:45380) 2024-06-04T05:07:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.307+0000 7f0031db9640 1 -- 172.21.15.70:0/271973148 learned_addr learned my addr 172.21.15.70:0/271973148 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.307+0000 7f00315b8640 1 --2- 172.21.15.70:0/271973148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d43820 0x55d569d43c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.307+0000 7f0022ffd640 1 --2- 172.21.15.70:0/3441988523 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f00180b2a10 0x7f00180b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.307+0000 7f00315b8640 1 -- 172.21.15.70:0/271973148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d43820 msgr2=0x55d569d43c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.307+0000 7f00315b8640 1 --2- 172.21.15.70:0/271973148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d43820 0x55d569d43c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.307+0000 7f00315b8640 1 -- 172.21.15.70:0/271973148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d44140 msgr2=0x55d569d4e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.307+0000 7f00315b8640 1 --2- 172.21.15.70:0/271973148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d44140 0x55d569d4e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.307+0000 7f00315b8640 1 -- 172.21.15.70:0/271973148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5699ba210 con 0x55d569d4ea00 2024-06-04T05:07:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.307+0000 7f0031db9640 1 --2- 172.21.15.70:0/271973148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d44140 0x55d569d4e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.307+0000 7f0022ffd640 1 -- 172.21.15.70:0/3441988523 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f00240e43b0 con 0x7f002c107940 2024-06-04T05:07:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.307+0000 7f00325ba640 1 --2- 172.21.15.70:0/3441988523 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f00180b2a10 0x7f00180b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.307+0000 7f00315b8640 1 --2- 172.21.15.70:0/271973148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d4ea00 0x55d569d50df0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f001c012750 tx=0x7f001c009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5b3f9f0beefac4d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.307+0000 7f0017fff640 1 -- 172.21.15.70:0/271973148 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c012cd0 con 0x55d569d4ea00 2024-06-04T05:07:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.307+0000 7f0017fff640 1 -- 172.21.15.70:0/271973148 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f001c0044d0 con 0x55d569d4ea00 2024-06-04T05:07:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.307+0000 7f0017fff640 1 -- 172.21.15.70:0/271973148 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c0035f0 con 0x55d569d4ea00 2024-06-04T05:07:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.308+0000 7f0036e0c740 1 -- 172.21.15.70:0/271973148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d4ea00 msgr2=0x55d569d50df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.308+0000 7f0036e0c740 1 --2- 172.21.15.70:0/271973148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d4ea00 0x55d569d50df0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f001c012750 tx=0x7f001c009680 comp rx=0 tx=0).stop 2024-06-04T05:07:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.308+0000 7f00325ba640 1 --2- 172.21.15.70:0/3441988523 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f00180b2a10 0x7f00180b4ed0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f00240a76e0 tx=0x7f0024000f40 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.308+0000 7f0036e0c740 1 -- 172.21.15.70:0/271973148 shutdown_connections 2024-06-04T05:07:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.308+0000 7f0036e0c740 1 --2- 172.21.15.70:0/271973148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d4ea00 0x55d569d50df0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.308+0000 7f0036e0c740 1 --2- 172.21.15.70:0/271973148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d44140 0x55d569d4e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.308+0000 7f0036e0c740 1 --2- 172.21.15.70:0/271973148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d43820 0x55d569d43c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.308+0000 7f0036e0c740 1 -- 172.21.15.70:0/271973148 >> 172.21.15.70:0/271973148 conn(0x55d569d3fd30 msgr2=0x55d569d40110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.308+0000 7f0036e0c740 1 -- 172.21.15.70:0/271973148 shutdown_connections 2024-06-04T05:07:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.308+0000 7f0036e0c740 1 -- 172.21.15.70:0/271973148 wait complete. 2024-06-04T05:07:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.309+0000 7f0036e0c740 1 -- start start 2024-06-04T05:07:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.309+0000 7f0036e0c740 1 Processor -- start 2024-06-04T05:07:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.309+0000 7f0036e0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d43820 0x55d569d4b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.309+0000 7f0036e0c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d44140 0x55d569d4bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.309+0000 7f0036e0c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d4ea00 0x55d569d4c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.309+0000 7f0036e0c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d56998b790 con 0x55d569d43820 2024-06-04T05:07:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.309+0000 7f0036e0c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d569c61890 con 0x55d569d44140 2024-06-04T05:07:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.309+0000 7f0036e0c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d569d51630 con 0x55d569d4ea00 2024-06-04T05:07:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.309+0000 7f0036e0c740 10 client.0 _ll_register_callbacks cb 0x55d5699b9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.309+0000 7f00315b8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d43820 0x55d569d4b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.309+0000 7f00315b8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d43820 0x55d569d4b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45382/0 (socket says 172.21.15.70:45382) 2024-06-04T05:07:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.309+0000 7f00315b8640 1 -- 172.21.15.70:0/2791875352 learned_addr learned my addr 172.21.15.70:0/2791875352 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.309+0000 7f00315b8640 1 --2- 172.21.15.70:0/2791875352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d4ea00 0x55d569d4c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.309+0000 7f0031db9640 1 --2- 172.21.15.70:0/2791875352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d44140 0x55d569d4bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.310+0000 7f00315b8640 1 -- 172.21.15.70:0/2791875352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d4ea00 msgr2=0x55d569d4c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.310+0000 7f00315b8640 1 --2- 172.21.15.70:0/2791875352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d4ea00 0x55d569d4c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.310+0000 7f00315b8640 1 -- 172.21.15.70:0/2791875352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d44140 msgr2=0x55d569d4bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.310+0000 7f00315b8640 1 --2- 172.21.15.70:0/2791875352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d44140 0x55d569d4bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.310+0000 7f00315b8640 1 -- 172.21.15.70:0/2791875352 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d569a86620 con 0x55d569d43820 2024-06-04T05:07:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.310+0000 7f0031db9640 1 --2- 172.21.15.70:0/2791875352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d44140 0x55d569d4bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:07:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.310+0000 7f00315b8640 1 --2- 172.21.15.70:0/2791875352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d43820 0x55d569d4b9a0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f001c010d40 tx=0x7f001c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18814bd43f2e03b7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.310+0000 7f00157fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.310+0000 7f00157fa640 1 -- 172.21.15.70:0/2791875352 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c005460 con 0x55d569d43820 2024-06-04T05:07:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.310+0000 7f00157fa640 1 -- 172.21.15.70:0/2791875352 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f001c0046c0 con 0x55d569d43820 2024-06-04T05:07:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.310+0000 7f0036e0c740 1 -- 172.21.15.70:0/2791875352 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f001c0044d0 con 0x55d569d43820 2024-06-04T05:07:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.310+0000 7f00157fa640 1 -- 172.21.15.70:0/2791875352 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f001c0048c0 con 0x55d569d43820 2024-06-04T05:07:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.310+0000 7f0036e0c740 10 client.18330 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.310+0000 7f0036e0c740 1 -- 172.21.15.70:0/2791875352 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55d569d5d410 con 0x55d569d43820 2024-06-04T05:07:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.310+0000 7f00157fa640 1 -- 172.21.15.70:0/2791875352 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f001c0044d0 con 0x55d569d43820 2024-06-04T05:07:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.311+0000 7f00157fa640 1 -- 172.21.15.70:0/2791875352 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f001c02e070 con 0x55d569d43820 2024-06-04T05:07:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.311+0000 7f0036e0c740 10 client.18330 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.311+0000 7f0036e0c740 10 client.18330 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.311+0000 7f0036e0c740 20 client.18330 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.311+0000 7f0036e0c740 1 --2- 172.21.15.70:0/2791875352 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d569d5e760 0x55d569d60b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.311+0000 7f0036e0c740 4 client.18330 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:07:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.311+0000 7f0036e0c740 1 -- 172.21.15.70:0/2791875352 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d569cb7b80 con 0x55d569d5e760 2024-06-04T05:07:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.311+0000 7f0031db9640 1 --2- 172.21.15.70:0/2791875352 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d569d5e760 0x55d569d60b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.311+0000 7f0031db9640 1 --2- 172.21.15.70:0/2791875352 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d569d5e760 0x55d569d60b50 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fb95cdc261a1fd8 server_cookie=2a0c32c86583f5e3 in_seq=0 out_seq=0 2024-06-04T05:07:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.311+0000 7f00157fa640 10 client.18330 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.312+0000 7f00157fa640 1 -- 172.21.15.70:0/2791875352 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d569cb7b80 con 0x55d569d5e760 2024-06-04T05:07:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.312+0000 7f00157fa640 10 client.18330 handle_command_reply: tid=0 2024-06-04T05:07:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.312+0000 7f0036e0c740 1 client.18330 shutdown 2024-06-04T05:07:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.312+0000 7f0036e0c740 1 -- 172.21.15.70:0/2791875352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d43820 msgr2=0x55d569d4b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.312+0000 7f0036e0c740 1 --2- 172.21.15.70:0/2791875352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d43820 0x55d569d4b9a0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f001c010d40 tx=0x7f001c005fd0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.312+0000 7f0036e0c740 1 -- 172.21.15.70:0/2791875352 shutdown_connections 2024-06-04T05:07:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.312+0000 7f0036e0c740 1 --2- 172.21.15.70:0/2791875352 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d569d5e760 0x55d569d60b50 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.312+0000 7f0036e0c740 1 --2- 172.21.15.70:0/2791875352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d44140 0x55d569d4bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.312+0000 7f0036e0c740 1 --2- 172.21.15.70:0/2791875352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d4ea00 0x55d569d4c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.312+0000 7f0036e0c740 1 --2- 172.21.15.70:0/2791875352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d43820 0x55d569d4b9a0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.312+0000 7f00157fa640 0 client.18330 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.312+0000 7f0036e0c740 1 -- 172.21.15.70:0/2791875352 >> 172.21.15.70:0/2791875352 conn(0x55d569d3fd30 msgr2=0x55d569d3efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.312+0000 7f0036e0c740 1 -- 172.21.15.70:0/2791875352 shutdown_connections 2024-06-04T05:07:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.312+0000 7f0036e0c740 1 -- 172.21.15.70:0/2791875352 wait complete. 2024-06-04T05:07:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.312+0000 7f0036e0c740 20 client.18330 trim_cache size 0 max 16384 2024-06-04T05:07:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.329+0000 7f0036e0c740 1 Processor -- start 2024-06-04T05:07:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.329+0000 7f0036e0c740 1 -- start start 2024-06-04T05:07:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f0036e0c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d43820 0x55d569d4c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f0036e0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d44140 0x55d569d59dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f0036e0c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d4ea00 0x55d569d5a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f0036e0c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f002c1ac520 con 0x55d569d44140 2024-06-04T05:07:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f0036e0c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f002c1ae0c0 con 0x55d569d4ea00 2024-06-04T05:07:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f0036e0c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f002c1acbf0 con 0x55d569d43820 2024-06-04T05:07:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f0031db9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d44140 0x55d569d59dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f0031db9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d44140 0x55d569d59dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45388/0 (socket says 172.21.15.70:45388) 2024-06-04T05:07:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f0031db9640 1 -- 172.21.15.70:0/617539188 learned_addr learned my addr 172.21.15.70:0/617539188 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f00315b8640 1 --2- 172.21.15.70:0/617539188 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d43820 0x55d569d4c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f00315b8640 1 --2- 172.21.15.70:0/617539188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d4ea00 0x55d569d5a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f0031db9640 1 -- 172.21.15.70:0/617539188 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d43820 msgr2=0x55d569d4c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f0031db9640 1 --2- 172.21.15.70:0/617539188 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d43820 0x55d569d4c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f0031db9640 1 -- 172.21.15.70:0/617539188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d4ea00 msgr2=0x55d569d5a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f0031db9640 1 --2- 172.21.15.70:0/617539188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d4ea00 0x55d569d5a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f0031db9640 1 -- 172.21.15.70:0/617539188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d569d49ef0 con 0x55d569d44140 2024-06-04T05:07:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f0031db9640 1 --2- 172.21.15.70:0/617539188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d44140 0x55d569d59dc0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f0028005940 tx=0x7f00280058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4585d93896da01be server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f00177fe640 1 -- 172.21.15.70:0/617539188 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0028049020 con 0x55d569d44140 2024-06-04T05:07:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f00177fe640 1 -- 172.21.15.70:0/617539188 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0028031890 con 0x55d569d44140 2024-06-04T05:07:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.330+0000 7f00177fe640 1 -- 172.21.15.70:0/617539188 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0028035040 con 0x55d569d44140 2024-06-04T05:07:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.331+0000 7f0036e0c740 1 -- 172.21.15.70:0/617539188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d44140 msgr2=0x55d569d59dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.331+0000 7f0036e0c740 1 --2- 172.21.15.70:0/617539188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d44140 0x55d569d59dc0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f0028005940 tx=0x7f00280058d0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.331+0000 7f0036e0c740 1 -- 172.21.15.70:0/617539188 shutdown_connections 2024-06-04T05:07:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.331+0000 7f0036e0c740 1 --2- 172.21.15.70:0/617539188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d4ea00 0x55d569d5a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.331+0000 7f0036e0c740 1 --2- 172.21.15.70:0/617539188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d44140 0x55d569d59dc0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.331+0000 7f0036e0c740 1 --2- 172.21.15.70:0/617539188 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d43820 0x55d569d4c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.331+0000 7f0036e0c740 1 -- 172.21.15.70:0/617539188 >> 172.21.15.70:0/617539188 conn(0x55d569d3fd30 msgr2=0x55d569d596b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.331+0000 7f0036e0c740 1 -- 172.21.15.70:0/617539188 shutdown_connections 2024-06-04T05:07:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.331+0000 7f0036e0c740 1 -- 172.21.15.70:0/617539188 wait complete. 2024-06-04T05:07:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.332+0000 7f0036e0c740 1 -- start start 2024-06-04T05:07:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.332+0000 7f0036e0c740 1 Processor -- start 2024-06-04T05:07:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.332+0000 7f0036e0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d43820 0x55d569d56f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.332+0000 7f0036e0c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d44140 0x55d569d52300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.332+0000 7f0036e0c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d4ea00 0x55d569d52840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.332+0000 7f0036e0c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f002c1ad9c0 con 0x55d569d43820 2024-06-04T05:07:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.332+0000 7f0036e0c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f002c1ad2c0 con 0x55d569d4ea00 2024-06-04T05:07:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.332+0000 7f0036e0c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f002c1aa520 con 0x55d569d44140 2024-06-04T05:07:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.332+0000 7f0036e0c740 10 client.0 _ll_register_callbacks cb 0x7f00240093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.332+0000 7f00315b8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d43820 0x55d569d56f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.332+0000 7f00315b8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d43820 0x55d569d56f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45392/0 (socket says 172.21.15.70:45392) 2024-06-04T05:07:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.332+0000 7f00315b8640 1 -- 172.21.15.70:0/2778693548 learned_addr learned my addr 172.21.15.70:0/2778693548 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.332+0000 7f0031db9640 1 --2- 172.21.15.70:0/2778693548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d44140 0x55d569d52300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.332+0000 7f00315b8640 1 --2- 172.21.15.70:0/2778693548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d4ea00 0x55d569d52840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f00315b8640 1 -- 172.21.15.70:0/2778693548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d44140 msgr2=0x55d569d52300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f00315b8640 1 --2- 172.21.15.70:0/2778693548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d44140 0x55d569d52300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f00315b8640 1 -- 172.21.15.70:0/2778693548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d4ea00 msgr2=0x55d569d52840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f00315b8640 1 --2- 172.21.15.70:0/2778693548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d4ea00 0x55d569d52840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f00315b8640 1 -- 172.21.15.70:0/2778693548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d569d53020 con 0x55d569d43820 2024-06-04T05:07:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f00315b8640 1 --2- 172.21.15.70:0/2778693548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d43820 0x55d569d56f90 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f001c021010 tx=0x7f001c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40976132227ff74e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f00157fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:07:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f00157fa640 1 -- 172.21.15.70:0/2778693548 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c034070 con 0x55d569d43820 2024-06-04T05:07:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f00157fa640 1 -- 172.21.15.70:0/2778693548 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7f001c005260 con 0x55d569d43820 2024-06-04T05:07:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f0036e0c740 1 -- 172.21.15.70:0/2778693548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0028031890 con 0x55d569d43820 2024-06-04T05:07:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f00157fa640 1 -- 172.21.15.70:0/2778693548 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f001c005460 con 0x55d569d43820 2024-06-04T05:07:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f0036e0c740 10 client.18336 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f0036e0c740 1 -- 172.21.15.70:0/2778693548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55d569dc9150 con 0x55d569d43820 2024-06-04T05:07:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f00157fa640 1 -- 172.21.15.70:0/2778693548 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0028031890 con 0x55d569d43820 2024-06-04T05:07:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f00157fa640 1 -- 172.21.15.70:0/2778693548 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f001c041070 con 0x55d569d43820 2024-06-04T05:07:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f0036e0c740 10 client.18336 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f0036e0c740 10 client.18336 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:07:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f0036e0c740 20 client.18336 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f0036e0c740 1 --2- 172.21.15.70:0/2778693548 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d569dc97e0 0x55d569dcbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f0036e0c740 4 client.18336 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:07:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.333+0000 7f0036e0c740 1 -- 172.21.15.70:0/2778693548 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d569cb1360 con 0x55d569dc97e0 2024-06-04T05:07:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.334+0000 7f0031db9640 1 --2- 172.21.15.70:0/2778693548 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d569dc97e0 0x55d569dcbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.334+0000 7f00157fa640 10 client.18336 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.334+0000 7f0031db9640 1 --2- 172.21.15.70:0/2778693548 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d569dc97e0 0x55d569dcbc00 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16f76b61bf78898a server_cookie=d4466b16652ae23c in_seq=0 out_seq=0 2024-06-04T05:07:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.360+0000 7f00157fa640 1 -- 172.21.15.70:0/2778693548 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55d569cb1360 con 0x55d569dc97e0 2024-06-04T05:07:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.360+0000 7f00157fa640 10 client.18336 handle_command_reply: tid=0 2024-06-04T05:07:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.360+0000 7f0036e0c740 1 client.18336 shutdown 2024-06-04T05:07:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.360+0000 7f0036e0c740 1 -- 172.21.15.70:0/2778693548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d43820 msgr2=0x55d569d56f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.360+0000 7f0036e0c740 1 --2- 172.21.15.70:0/2778693548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d43820 0x55d569d56f90 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f001c021010 tx=0x7f001c005fd0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.360+0000 7f0036e0c740 1 -- 172.21.15.70:0/2778693548 shutdown_connections 2024-06-04T05:07:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.360+0000 7f0036e0c740 1 --2- 172.21.15.70:0/2778693548 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d569dc97e0 0x55d569dcbc00 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.360+0000 7f00157fa640 0 client.18336 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.360+0000 7f0036e0c740 1 --2- 172.21.15.70:0/2778693548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d569d4ea00 0x55d569d52840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.360+0000 7f0036e0c740 1 --2- 172.21.15.70:0/2778693548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d569d44140 0x55d569d52300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.360+0000 7f0036e0c740 1 --2- 172.21.15.70:0/2778693548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d569d43820 0x55d569d56f90 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.360+0000 7f0036e0c740 1 -- 172.21.15.70:0/2778693548 >> 172.21.15.70:0/2778693548 conn(0x55d569d3fd30 msgr2=0x55d569d59f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.360+0000 7f0036e0c740 1 -- 172.21.15.70:0/2778693548 shutdown_connections 2024-06-04T05:07:58.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.360+0000 7f0036e0c740 1 -- 172.21.15.70:0/2778693548 wait complete. 2024-06-04T05:07:58.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.360+0000 7f0036e0c740 20 client.18336 trim_cache size 0 max 16384 2024-06-04T05:07:58.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:07:58.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:07:58.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:07:58.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.361+0000 7f003401d640 1 -- 172.21.15.70:0/3441988523 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f00180b2a10 msgr2=0x7f00180b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:58.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.361+0000 7f003401d640 1 --2- 172.21.15.70:0/3441988523 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f00180b2a10 0x7f00180b4ed0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f00240a76e0 tx=0x7f0024000f40 comp rx=0 tx=0).stop 2024-06-04T05:07:58.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.361+0000 7f003401d640 1 -- 172.21.15.70:0/3441988523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f002c107940 msgr2=0x7f002c19b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:58.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.361+0000 7f003401d640 1 --2- 172.21.15.70:0/3441988523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f002c107940 0x7f002c19b7c0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f002400cbb0 tx=0x7f0024010490 comp rx=0 tx=0).stop 2024-06-04T05:07:58.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.361+0000 7f003401d640 1 -- 172.21.15.70:0/3441988523 shutdown_connections 2024-06-04T05:07:58.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.361+0000 7f003401d640 1 --2- 172.21.15.70:0/3441988523 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f00180b2a10 0x7f00180b4ed0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.361+0000 7f003401d640 1 --2- 172.21.15.70:0/3441988523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f002c102340 0x7f002c197440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.361+0000 7f003401d640 1 --2- 172.21.15.70:0/3441988523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f002c107940 0x7f002c19b7c0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.361+0000 7f003401d640 1 --2- 172.21.15.70:0/3441988523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f002c101980 0x7f002c196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:58.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.361+0000 7f003401d640 1 -- 172.21.15.70:0/3441988523 >> 172.21.15.70:0/3441988523 conn(0x7f002c078ad0 msgr2=0x7f002c104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:58.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.361+0000 7f003401d640 1 -- 172.21.15.70:0/3441988523 shutdown_connections 2024-06-04T05:07:58.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:07:58.361+0000 7f003401d640 1 -- 172.21.15.70:0/3441988523 wait complete. 2024-06-04T05:07:58.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 188467 1.88 MB/sec execute 411 sec latency 15.659 ms 2024-06-04T05:07:58.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:58 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:07:58.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:58.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:58 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:07:58.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:58 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:07:58.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:58 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:07:58.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:58.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:58 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:07:58.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:58.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:58 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:07:58.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:58.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:58.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:58.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:58 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:07:58.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:58.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:58 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:07:58.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:58 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:07:58.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:58 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:07:58.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:58.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:58 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:07:58.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:58.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:58 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:07:58.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:58.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:58.937 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:58.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:58 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:07:58.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:58 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:07:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:58 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:07:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:58 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:07:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:58 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:07:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:58 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:07:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:07:59.403 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:07:59.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 188869 1.88 MB/sec execute 412 sec latency 55.449 ms 2024-06-04T05:07:59.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.902+0000 7fa8a821e700 1 -- 172.21.15.70:0/615036731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8a006b480 msgr2=0x7fa8a006b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:59.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.902+0000 7fa8a821e700 1 --2- 172.21.15.70:0/615036731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8a006b480 0x7fa8a006b860 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fa89c0099c0 tx=0x7fa89c009cd0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.902+0000 7fa8a821e700 1 -- 172.21.15.70:0/615036731 shutdown_connections 2024-06-04T05:07:59.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.902+0000 7fa8a821e700 1 --2- 172.21.15.70:0/615036731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8a0108900 0x7fa8a0110480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.902+0000 7fa8a821e700 1 --2- 172.21.15.70:0/615036731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8a006b480 0x7fa8a006b860 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.902+0000 7fa8a821e700 1 --2- 172.21.15.70:0/615036731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8a006be30 0x7fa8a01081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.902+0000 7fa8a821e700 1 -- 172.21.15.70:0/615036731 >> 172.21.15.70:0/615036731 conn(0x7fa8a00793e0 msgr2=0x7fa8a00797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:59.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.902+0000 7fa8a821e700 1 -- 172.21.15.70:0/615036731 shutdown_connections 2024-06-04T05:07:59.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.902+0000 7fa8a821e700 1 -- 172.21.15.70:0/615036731 wait complete. 2024-06-04T05:07:59.905 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:07:59 smithi070 ceph-mon[32631]: pgmap v333: 97 pgs: 97 active+clean; 119 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 981 KiB/s wr, 93 op/s 2024-06-04T05:07:59.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.903+0000 7fa8a821e700 1 Processor -- start 2024-06-04T05:07:59.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.903+0000 7fa8a821e700 1 -- start start 2024-06-04T05:07:59.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.903+0000 7fa8a821e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8a006b480 0x7fa8a019f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.903+0000 7fa8a821e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8a006be30 0x7fa8a019fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.903+0000 7fa8a821e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8a0108900 0x7fa8a0199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.903+0000 7fa8a821e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a0113590 con 0x7fa8a006be30 2024-06-04T05:07:59.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.903+0000 7fa8a821e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a0113410 con 0x7fa8a006b480 2024-06-04T05:07:59.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.903+0000 7fa8a821e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a0113710 con 0x7fa8a0108900 2024-06-04T05:07:59.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.903+0000 7fa8a67bb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8a0108900 0x7fa8a0199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.903+0000 7fa8a57b9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8a006be30 0x7fa8a019fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.904+0000 7fa8a67bb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8a0108900 0x7fa8a0199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59426/0 (socket says 172.21.15.70:59426) 2024-06-04T05:07:59.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.904+0000 7fa8a67bb700 1 -- 172.21.15.70:0/1574066354 learned_addr learned my addr 172.21.15.70:0/1574066354 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:59.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.904+0000 7fa8a57b9700 1 --2- 172.21.15.70:0/1574066354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8a006be30 0x7fa8a019fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45402/0 (socket says 172.21.15.70:45402) 2024-06-04T05:07:59.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.904+0000 7fa8a5fba700 1 --2- 172.21.15.70:0/1574066354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8a006b480 0x7fa8a019f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.904+0000 7fa8a67bb700 1 -- 172.21.15.70:0/1574066354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8a006b480 msgr2=0x7fa8a019f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:59.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.904+0000 7fa8a67bb700 1 --2- 172.21.15.70:0/1574066354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8a006b480 0x7fa8a019f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.904+0000 7fa8a67bb700 1 -- 172.21.15.70:0/1574066354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8a006be30 msgr2=0x7fa8a019fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:59.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.904+0000 7fa8a67bb700 1 --2- 172.21.15.70:0/1574066354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8a006be30 0x7fa8a019fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.904+0000 7fa8a67bb700 1 -- 172.21.15.70:0/1574066354 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8a019a030 con 0x7fa8a0108900 2024-06-04T05:07:59.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.904+0000 7fa8a57b9700 1 --2- 172.21.15.70:0/1574066354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8a006be30 0x7fa8a019fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:59.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.904+0000 7fa8a67bb700 1 --2- 172.21.15.70:0/1574066354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8a0108900 0x7fa8a0199850 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa89800ef30 tx=0x7fa89800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:59.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.904+0000 7fa896ffd700 1 -- 172.21.15.70:0/1574066354 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8980092e0 con 0x7fa8a0108900 2024-06-04T05:07:59.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.905+0000 7fa896ffd700 1 -- 172.21.15.70:0/1574066354 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa898004bb0 con 0x7fa8a0108900 2024-06-04T05:07:59.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.905+0000 7fa896ffd700 1 -- 172.21.15.70:0/1574066354 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa898004020 con 0x7fa8a0108900 2024-06-04T05:07:59.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.905+0000 7fa8a821e700 1 -- 172.21.15.70:0/1574066354 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8a019a2c0 con 0x7fa8a0108900 2024-06-04T05:07:59.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.905+0000 7fa8a821e700 1 -- 172.21.15.70:0/1574066354 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8a0102440 con 0x7fa8a0108900 2024-06-04T05:07:59.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.909+0000 7fa896ffd700 1 -- 172.21.15.70:0/1574066354 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fa89801b030 con 0x7fa8a0108900 2024-06-04T05:07:59.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.910+0000 7fa896ffd700 1 --2- 172.21.15.70:0/1574066354 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa88c0b2ba0 0x7fa88c0b5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.910+0000 7fa8b2ab3b80 1 Processor -- start 2024-06-04T05:07:59.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.910+0000 7fa8a5fba700 1 --2- 172.21.15.70:0/1574066354 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa88c0b2ba0 0x7fa88c0b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.910+0000 7fa8b2ab3b80 1 -- start start 2024-06-04T05:07:59.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.910+0000 7fa896ffd700 1 -- 172.21.15.70:0/1574066354 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7fa8980e2100 con 0x7fa8a0108900 2024-06-04T05:07:59.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.910+0000 7fa8b2ab3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4487340 0x5568e4487720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.910+0000 7fa8b2ab3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4481970 0x5568e4481df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.910+0000 7fa8b2ab3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 0x5568e448e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa8b2ab3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5568e43e0e20 con 0x5568e4487340 2024-06-04T05:07:59.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa8b2ab3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5568e43d09c0 con 0x5568e4481970 2024-06-04T05:07:59.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa8b2ab3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5568e43d4a00 con 0x5568e4482330 2024-06-04T05:07:59.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa8a5fba700 1 --2- 172.21.15.70:0/1574066354 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa88c0b2ba0 0x7fa88c0b5060 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa89c003760 tx=0x7fa89c005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:59.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa8a67bb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 0x5568e448e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa8a67bb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 0x5568e448e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59434/0 (socket says 172.21.15.70:59434) 2024-06-04T05:07:59.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa8a67bb700 1 -- 172.21.15.70:0/1253091171 learned_addr learned my addr 172.21.15.70:0/1253091171 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:59.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa8a5fba700 1 --2- 172.21.15.70:0/1253091171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4487340 0x5568e4487720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa8a57b9700 1 --2- 172.21.15.70:0/1253091171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4481970 0x5568e4481df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa8a67bb700 1 -- 172.21.15.70:0/1253091171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4481970 msgr2=0x5568e4481df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:59.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa8a67bb700 1 --2- 172.21.15.70:0/1253091171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4481970 0x5568e4481df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa8a67bb700 1 -- 172.21.15.70:0/1253091171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4487340 msgr2=0x5568e4487720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:59.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa8a67bb700 1 --2- 172.21.15.70:0/1253091171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4487340 0x5568e4487720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa8a67bb700 1 -- 172.21.15.70:0/1253091171 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568e4320a60 con 0x5568e4482330 2024-06-04T05:07:59.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa8a57b9700 1 --2- 172.21.15.70:0/1253091171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4481970 0x5568e4481df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:07:59.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa8a67bb700 1 --2- 172.21.15.70:0/1253091171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 0x5568e448e170 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa8980a5460 tx=0x7fa8980a8c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef750e339e1e3d25 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:59.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa88bfff700 1 -- 172.21.15.70:0/1253091171 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8980ad790 con 0x5568e4482330 2024-06-04T05:07:59.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa88bfff700 1 -- 172.21.15.70:0/1253091171 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa8980b21b0 con 0x5568e4482330 2024-06-04T05:07:59.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.911+0000 7fa88bfff700 1 -- 172.21.15.70:0/1253091171 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8980a9640 con 0x5568e4482330 2024-06-04T05:07:59.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.912+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/1253091171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 msgr2=0x5568e448e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:59.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.912+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/1253091171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 0x5568e448e170 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa8980a5460 tx=0x7fa8980a8c80 comp rx=0 tx=0).stop 2024-06-04T05:07:59.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.912+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/1253091171 shutdown_connections 2024-06-04T05:07:59.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.912+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/1253091171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 0x5568e448e170 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.912+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/1253091171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4487340 0x5568e4487720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.912+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/1253091171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4481970 0x5568e4481df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.912+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/1253091171 >> 172.21.15.70:0/1253091171 conn(0x5568e43af210 msgr2=0x5568e447c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:59.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.912+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/1253091171 shutdown_connections 2024-06-04T05:07:59.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.912+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/1253091171 wait complete. 2024-06-04T05:07:59.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.913+0000 7fa8b2ab3b80 1 -- start start 2024-06-04T05:07:59.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.913+0000 7fa8b2ab3b80 1 Processor -- start 2024-06-04T05:07:59.917 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:07:59.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.913+0000 7fa8b2ab3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4481970 0x5568e448de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.913+0000 7fa8b2ab3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4482330 0x5568e448b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8b2ab3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4487340 0x5568e448b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8a5fba700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4481970 0x5568e448de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8a5fba700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4481970 0x5568e448de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59438/0 (socket says 172.21.15.70:59438) 2024-06-04T05:07:59.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/3632571530 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5568e42d7130 con 0x5568e4487340 2024-06-04T05:07:59.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8a5fba700 1 -- 172.21.15.70:0/3632571530 learned_addr learned my addr 172.21.15.70:0/3632571530 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:59.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/3632571530 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5568e4372950 con 0x5568e4482330 2024-06-04T05:07:59.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/3632571530 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5568e448e770 con 0x5568e4481970 2024-06-04T05:07:59.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8a67bb700 1 --2- 172.21.15.70:0/3632571530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4487340 0x5568e448b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8a57b9700 1 --2- 172.21.15.70:0/3632571530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4482330 0x5568e448b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8b2ab3b80 10 client.0 _ll_register_callbacks cb 0x5568e436bab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:59.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8a5fba700 1 -- 172.21.15.70:0/3632571530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4482330 msgr2=0x5568e448b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:59.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8a5fba700 1 --2- 172.21.15.70:0/3632571530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4482330 0x5568e448b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8a5fba700 1 -- 172.21.15.70:0/3632571530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4487340 msgr2=0x5568e448b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:59.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8a5fba700 1 --2- 172.21.15.70:0/3632571530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4487340 0x5568e448b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8a5fba700 1 -- 172.21.15.70:0/3632571530 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568e436f820 con 0x5568e4481970 2024-06-04T05:07:59.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8a67bb700 1 --2- 172.21.15.70:0/3632571530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4487340 0x5568e448b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:59.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8a5fba700 1 --2- 172.21.15.70:0/3632571530 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4481970 0x5568e448de90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa89c0098e0 tx=0x7fa89c01a070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=455c22ae32625e0a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:59.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.914+0000 7fa8897fa700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:07:59.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.915+0000 7fa8897fa700 1 -- 172.21.15.70:0/3632571530 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa89c00bdf0 con 0x5568e4481970 2024-06-04T05:07:59.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.915+0000 7fa8897fa700 1 -- 172.21.15.70:0/3632571530 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7fa89c024470 con 0x5568e4481970 2024-06-04T05:07:59.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.915+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/3632571530 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8980b21b0 con 0x5568e4481970 2024-06-04T05:07:59.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.915+0000 7fa8897fa700 1 -- 172.21.15.70:0/3632571530 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa89c0246b0 con 0x5568e4481970 2024-06-04T05:07:59.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.915+0000 7fa8b2ab3b80 10 client.27541 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:59.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.915+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/3632571530 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5568e4498e50 con 0x5568e4481970 2024-06-04T05:07:59.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.915+0000 7fa8897fa700 1 -- 172.21.15.70:0/3632571530 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa8980b21b0 con 0x5568e4481970 2024-06-04T05:07:59.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.915+0000 7fa8897fa700 1 -- 172.21.15.70:0/3632571530 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7fa89c038610 con 0x5568e4481970 2024-06-04T05:07:59.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.915+0000 7fa8b2ab3b80 10 client.27541 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:59.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.915+0000 7fa8b2ab3b80 10 client.27541 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:07:59.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.915+0000 7fa8b2ab3b80 20 client.27541 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:59.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.915+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/3632571530 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5568e449b4e0 0x5568e449d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.915+0000 7fa8b2ab3b80 4 client.27541 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:07:59.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.915+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/3632571530 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568e444db70 con 0x5568e449b4e0 2024-06-04T05:07:59.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.915+0000 7fa8a57b9700 1 --2- 172.21.15.70:0/3632571530 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5568e449b4e0 0x5568e449d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.916+0000 7fa8897fa700 10 client.27541 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:59.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.916+0000 7fa8a57b9700 1 --2- 172.21.15.70:0/3632571530 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5568e449b4e0 0x5568e449d920 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87380c69c48fe93d server_cookie=c0b52ff2ebec49c1 in_seq=0 out_seq=0 2024-06-04T05:07:59.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.917+0000 7fa8897fa700 1 -- 172.21.15.70:0/3632571530 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5568e444db70 con 0x5568e449b4e0 2024-06-04T05:07:59.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.917+0000 7fa8897fa700 10 client.27541 handle_command_reply: tid=0 2024-06-04T05:07:59.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.917+0000 7fa8b2ab3b80 1 client.27541 shutdown 2024-06-04T05:07:59.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.917+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/3632571530 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4481970 msgr2=0x5568e448de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:59.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.917+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/3632571530 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4481970 0x5568e448de90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa89c0098e0 tx=0x7fa89c01a070 comp rx=0 tx=0).stop 2024-06-04T05:07:59.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.917+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/3632571530 shutdown_connections 2024-06-04T05:07:59.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.917+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/3632571530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4487340 0x5568e448b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.917+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/3632571530 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4481970 0x5568e448de90 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.917+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/3632571530 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5568e449b4e0 0x5568e449d920 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.917+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/3632571530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4482330 0x5568e448b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.917+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/3632571530 >> 172.21.15.70:0/3632571530 conn(0x5568e43af210 msgr2=0x5568e4476d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:59.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.917+0000 7fa8897fa700 0 client.27541 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:59.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.918+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/3632571530 shutdown_connections 2024-06-04T05:07:59.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.918+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/3632571530 wait complete. 2024-06-04T05:07:59.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.918+0000 7fa8b2ab3b80 20 client.27541 trim_cache size 0 max 16384 2024-06-04T05:07:59.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.948+0000 7fa8b2ab3b80 1 Processor -- start 2024-06-04T05:07:59.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.948+0000 7fa8b2ab3b80 1 -- start start 2024-06-04T05:07:59.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.948+0000 7fa8b2ab3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4481970 0x5568e448d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.948+0000 7fa8b2ab3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 0x5568e44a31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.948+0000 7fa8b2ab3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4487340 0x5568e44a3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.948+0000 7fa8b2ab3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a01afe50 con 0x5568e4481970 2024-06-04T05:07:59.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.948+0000 7fa8b2ab3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a01b19f0 con 0x5568e4487340 2024-06-04T05:07:59.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.948+0000 7fa8b2ab3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a01b0520 con 0x5568e4482330 2024-06-04T05:07:59.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.948+0000 7fa8a57b9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 0x5568e44a31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.948+0000 7fa8a57b9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 0x5568e44a31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59450/0 (socket says 172.21.15.70:59450) 2024-06-04T05:07:59.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.948+0000 7fa8a57b9700 1 -- 172.21.15.70:0/1693339990 learned_addr learned my addr 172.21.15.70:0/1693339990 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:59.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.948+0000 7fa8a5fba700 1 --2- 172.21.15.70:0/1693339990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4481970 0x5568e448d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.948+0000 7fa8a67bb700 1 --2- 172.21.15.70:0/1693339990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4487340 0x5568e44a3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.949+0000 7fa8a67bb700 1 -- 172.21.15.70:0/1693339990 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 msgr2=0x5568e44a31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:59.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.949+0000 7fa8a67bb700 1 --2- 172.21.15.70:0/1693339990 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 0x5568e44a31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.949+0000 7fa8a67bb700 1 -- 172.21.15.70:0/1693339990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4481970 msgr2=0x5568e448d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:59.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.949+0000 7fa8a67bb700 1 --2- 172.21.15.70:0/1693339990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4481970 0x5568e448d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.949+0000 7fa8a67bb700 1 -- 172.21.15.70:0/1693339990 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568e448e690 con 0x5568e4487340 2024-06-04T05:07:59.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.949+0000 7fa8a67bb700 1 --2- 172.21.15.70:0/1693339990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4487340 0x5568e44a3730 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa8980a5460 tx=0x7fa898009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bad5edcd7aa3994 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:59.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.949+0000 7fa88b7fe700 1 -- 172.21.15.70:0/1693339990 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8980ad790 con 0x5568e4487340 2024-06-04T05:07:59.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.949+0000 7fa88b7fe700 1 -- 172.21.15.70:0/1693339990 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa8980ca190 con 0x5568e4487340 2024-06-04T05:07:59.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.949+0000 7fa88b7fe700 1 -- 172.21.15.70:0/1693339990 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8980b1f80 con 0x5568e4487340 2024-06-04T05:07:59.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.950+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/1693339990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4487340 msgr2=0x5568e44a3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:07:59.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.950+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/1693339990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4487340 0x5568e44a3730 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa8980a5460 tx=0x7fa898009ed0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.951+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/1693339990 shutdown_connections 2024-06-04T05:07:59.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.951+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/1693339990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4487340 0x5568e44a3730 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.951+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/1693339990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4481970 0x5568e448d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.951+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/1693339990 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 0x5568e44a31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.951+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/1693339990 >> 172.21.15.70:0/1693339990 conn(0x5568e43af210 msgr2=0x5568e4476d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:07:59.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.951+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/1693339990 shutdown_connections 2024-06-04T05:07:59.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.951+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/1693339990 wait complete. 2024-06-04T05:07:59.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.951+0000 7fa8b2ab3b80 1 -- start start 2024-06-04T05:07:59.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.951+0000 7fa8b2ab3b80 1 Processor -- start 2024-06-04T05:07:59.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.952+0000 7fa8b2ab3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4481970 0x5568e4493be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.952+0000 7fa8b2ab3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 0x5568e4494120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.952+0000 7fa8b2ab3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4487340 0x5568e44909d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.952+0000 7fa8b2ab3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a01b12f0 con 0x5568e4481970 2024-06-04T05:07:59.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.952+0000 7fa8b2ab3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a01b0bf0 con 0x5568e4487340 2024-06-04T05:07:59.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.952+0000 7fa8b2ab3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a01ade10 con 0x5568e4482330 2024-06-04T05:07:59.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.952+0000 7fa8b2ab3b80 10 client.0 _ll_register_callbacks cb 0x5568e436bab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:07:59.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.952+0000 7fa8a57b9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 0x5568e4494120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.952+0000 7fa8a5fba700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4481970 0x5568e4493be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.952+0000 7fa8a67bb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4487340 0x5568e44909d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.952+0000 7fa8a67bb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4487340 0x5568e44909d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33600/0 (socket says 172.21.15.70:33600) 2024-06-04T05:07:59.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.952+0000 7fa8a67bb700 1 -- 172.21.15.70:0/2520353646 learned_addr learned my addr 172.21.15.70:0/2520353646 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:07:59.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.952+0000 7fa8a57b9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 0x5568e4494120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:59464/0 (socket says 172.21.15.70:59464) 2024-06-04T05:07:59.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.953+0000 7fa8a67bb700 1 -- 172.21.15.70:0/2520353646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 msgr2=0x5568e4494120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:59.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.953+0000 7fa8a67bb700 1 --2- 172.21.15.70:0/2520353646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 0x5568e4494120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.953+0000 7fa8a67bb700 1 -- 172.21.15.70:0/2520353646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4481970 msgr2=0x5568e4493be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:07:59.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.953+0000 7fa8a67bb700 1 --2- 172.21.15.70:0/2520353646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4481970 0x5568e4493be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:07:59.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.953+0000 7fa8a67bb700 1 -- 172.21.15.70:0/2520353646 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568e4491210 con 0x5568e4487340 2024-06-04T05:07:59.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.953+0000 7fa8a5fba700 1 --2- 172.21.15.70:0/2520353646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4481970 0x5568e4493be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:07:59.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.953+0000 7fa8a67bb700 1 --2- 172.21.15.70:0/2520353646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4487340 0x5568e44909d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa8980dfd90 tx=0x7fa8980b1fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8f42f26caba15b7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:07:59.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.953+0000 7fa8897fa700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:07:59.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.953+0000 7fa8897fa700 1 -- 172.21.15.70:0/2520353646 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8980c8060 con 0x5568e4487340 2024-06-04T05:07:59.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.953+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/2520353646 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8980ca190 con 0x5568e4487340 2024-06-04T05:07:59.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.953+0000 7fa8897fa700 1 -- 172.21.15.70:0/2520353646 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7fa8980a8850 con 0x5568e4487340 2024-06-04T05:07:59.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.953+0000 7fa8897fa700 1 -- 172.21.15.70:0/2520353646 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa8980ad790 con 0x5568e4487340 2024-06-04T05:07:59.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.953+0000 7fa8b2ab3b80 10 client.27431 fetch_fsmap learned FSMap version 34 2024-06-04T05:07:59.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.953+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/2520353646 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5568e4506b70 con 0x5568e4487340 2024-06-04T05:07:59.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.953+0000 7fa8897fa700 1 -- 172.21.15.70:0/2520353646 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa8980ca190 con 0x5568e4487340 2024-06-04T05:07:59.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.954+0000 7fa8897fa700 1 -- 172.21.15.70:0/2520353646 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7fa8980a81d0 con 0x5568e4487340 2024-06-04T05:07:59.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.954+0000 7fa8b2ab3b80 10 client.27431 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:07:59.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.954+0000 7fa8b2ab3b80 10 client.27431 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:07:59.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.954+0000 7fa8b2ab3b80 20 client.27431 populate_metadata read hostname 'smithi070' 2024-06-04T05:07:59.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.954+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/2520353646 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5568e4506d40 0x5568e4509130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:07:59.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.954+0000 7fa8b2ab3b80 4 client.27431 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}] 2024-06-04T05:07:59.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.954+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/2520353646 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}) -- 0x5568e43e2d70 con 0x5568e4506d40 2024-06-04T05:07:59.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.954+0000 7fa8a5fba700 1 --2- 172.21.15.70:0/2520353646 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5568e4506d40 0x5568e4509130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:07:59.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.955+0000 7fa8a5fba700 1 --2- 172.21.15.70:0/2520353646 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5568e4506d40 0x5568e4509130 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc6613f171b9cadf server_cookie=8158de06dd4fd199 in_seq=0 out_seq=0 2024-06-04T05:07:59.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:07:59.955+0000 7fa8897fa700 10 client.27431 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:07:59.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:07:59 smithi098 ceph-mon[35569]: pgmap v333: 97 pgs: 97 active+clean; 119 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 981 KiB/s wr, 93 op/s 2024-06-04T05:08:00.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.021+0000 7fa8897fa700 1 -- 172.21.15.70:0/2520353646 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5568e43e2d70 con 0x5568e4506d40 2024-06-04T05:08:00.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.021+0000 7fa8897fa700 10 client.27431 handle_command_reply: tid=0 2024-06-04T05:08:00.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.021+0000 7fa8b2ab3b80 1 client.27431 shutdown 2024-06-04T05:08:00.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.022+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/2520353646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4487340 msgr2=0x5568e44909d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:00.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.022+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/2520353646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4487340 0x5568e44909d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa8980dfd90 tx=0x7fa8980b1fd0 comp rx=0 tx=0).stop 2024-06-04T05:08:00.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.022+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/2520353646 shutdown_connections 2024-06-04T05:08:00.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.022+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/2520353646 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5568e4506d40 0x5568e4509130 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:00.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.022+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/2520353646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5568e4487340 0x5568e44909d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:00.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.022+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/2520353646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568e4481970 0x5568e4493be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:00.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.022+0000 7fa8897fa700 0 client.27431 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:00.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.022+0000 7fa8b2ab3b80 1 --2- 172.21.15.70:0/2520353646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5568e4482330 0x5568e4494120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:00.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.022+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/2520353646 >> 172.21.15.70:0/2520353646 conn(0x5568e43af210 msgr2=0x5568e4484ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:00.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.022+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/2520353646 shutdown_connections 2024-06-04T05:08:00.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.022+0000 7fa8b2ab3b80 1 -- 172.21.15.70:0/2520353646 wait complete. 2024-06-04T05:08:00.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.022+0000 7fa8b2ab3b80 20 client.27431 trim_cache size 0 max 16384 2024-06-04T05:08:00.026 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:08:00.026 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 34, 2024-06-04T05:08:00.026 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:08:00.026 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 30, 2024-06-04T05:08:00.026 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:08:00.026 INFO:teuthology.orchestra.run.smithi070.stdout: "c91c6358": { 2024-06-04T05:08:00.026 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 30, 2024-06-04T05:08:00.026 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:08:00.026 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:08:00.026 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:08:00.026 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:08:00.026 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:08:00.027 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:08:00.027 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 130.0, 2024-06-04T05:08:00.027 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:08:00.027 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:08:00.027 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:08:00.027 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:08:00.027 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:08:00.027 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:08:00.027 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:00.027 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:00.027 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:00.027 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:00.027 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:00.027 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:08:00.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.022+0000 7fa8a821e700 1 -- 172.21.15.70:0/1574066354 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa88c0b2ba0 msgr2=0x7fa88c0b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:00.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.022+0000 7fa8a821e700 1 --2- 172.21.15.70:0/1574066354 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa88c0b2ba0 0x7fa88c0b5060 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa89c003760 tx=0x7fa89c005fb0 comp rx=0 tx=0).stop 2024-06-04T05:08:00.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.022+0000 7fa8a821e700 1 -- 172.21.15.70:0/1574066354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8a0108900 msgr2=0x7fa8a0199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:00.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.022+0000 7fa8a821e700 1 --2- 172.21.15.70:0/1574066354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8a0108900 0x7fa8a0199850 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa89800ef30 tx=0x7fa89800c3a0 comp rx=0 tx=0).stop 2024-06-04T05:08:00.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.023+0000 7fa8a821e700 1 -- 172.21.15.70:0/1574066354 shutdown_connections 2024-06-04T05:08:00.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.023+0000 7fa8a821e700 1 --2- 172.21.15.70:0/1574066354 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa88c0b2ba0 0x7fa88c0b5060 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:00.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.023+0000 7fa8a821e700 1 --2- 172.21.15.70:0/1574066354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8a0108900 0x7fa8a0199850 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:00.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.023+0000 7fa8a821e700 1 --2- 172.21.15.70:0/1574066354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8a006b480 0x7fa8a019f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:00.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.023+0000 7fa8a821e700 1 --2- 172.21.15.70:0/1574066354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8a006be30 0x7fa8a019fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:00.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.023+0000 7fa8a821e700 1 -- 172.21.15.70:0/1574066354 >> 172.21.15.70:0/1574066354 conn(0x7fa8a00793e0 msgr2=0x7fa8a01005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:00.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.023+0000 7fa8a821e700 1 -- 172.21.15.70:0/1574066354 shutdown_connections 2024-06-04T05:08:00.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:00.023+0000 7fa8a821e700 1 -- 172.21.15.70:0/1574066354 wait complete. 2024-06-04T05:08:00.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:07:59 smithi112 ceph-mon[36453]: pgmap v333: 97 pgs: 97 active+clean; 119 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 981 KiB/s wr, 93 op/s 2024-06-04T05:08:00.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 188972 1.87 MB/sec execute 413 sec latency 721.898 ms 2024-06-04T05:08:00.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:00 smithi070 ceph-mon[32631]: pgmap v334: 97 pgs: 97 active+clean; 119 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 981 KiB/s wr, 93 op/s 2024-06-04T05:08:00.932 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c91c6358', quiesce took 8.3 seconds. Will release after: 1.6999999999999993 2024-06-04T05:08:00.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:00 smithi098 ceph-mon[35569]: pgmap v334: 97 pgs: 97 active+clean; 119 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 981 KiB/s wr, 93 op/s 2024-06-04T05:08:01.087 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:00 smithi112 ceph-mon[36453]: pgmap v334: 97 pgs: 97 active+clean; 119 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 981 KiB/s wr, 93 op/s 2024-06-04T05:08:01.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 188972 1.87 MB/sec execute 414 sec latency 1721.971 ms 2024-06-04T05:08:02.633 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:08:02.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 188972 1.86 MB/sec execute 415 sec latency 2722.015 ms 2024-06-04T05:08:02.922 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:08:03.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:03 smithi070 ceph-mon[32631]: pgmap v335: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 1.4 MiB/s wr, 169 op/s 2024-06-04T05:08:03.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:03 smithi112 ceph-mon[36453]: pgmap v335: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 1.4 MiB/s wr, 169 op/s 2024-06-04T05:08:03.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:03 smithi098 ceph-mon[35569]: pgmap v335: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 1.4 MiB/s wr, 169 op/s 2024-06-04T05:08:03.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 188972 1.86 MB/sec execute 416 sec latency 3722.052 ms 2024-06-04T05:08:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.405+0000 7faf88733700 1 -- 172.21.15.70:0/2708853211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf801088f0 msgr2=0x7faf80110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.405+0000 7faf88733700 1 --2- 172.21.15.70:0/2708853211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf801088f0 0x7faf80110470 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7faf7800b0d0 tx=0x7faf7800b3e0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.405+0000 7faf88733700 1 -- 172.21.15.70:0/2708853211 shutdown_connections 2024-06-04T05:08:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.405+0000 7faf88733700 1 --2- 172.21.15.70:0/2708853211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf801088f0 0x7faf80110470 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.405+0000 7faf88733700 1 --2- 172.21.15.70:0/2708853211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf801037f0 0x7faf80103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.405+0000 7faf88733700 1 --2- 172.21.15.70:0/2708853211 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faf801041a0 0x7faf801081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.405+0000 7faf88733700 1 -- 172.21.15.70:0/2708853211 >> 172.21.15.70:0/2708853211 conn(0x7faf8007a6a0 msgr2=0x7faf8007aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.405+0000 7faf88733700 1 -- 172.21.15.70:0/2708853211 shutdown_connections 2024-06-04T05:08:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.405+0000 7faf88733700 1 -- 172.21.15.70:0/2708853211 wait complete. 2024-06-04T05:08:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.406+0000 7faf88733700 1 Processor -- start 2024-06-04T05:08:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.406+0000 7faf88733700 1 -- start start 2024-06-04T05:08:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.406+0000 7faf88733700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faf801037f0 0x7faf80075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.406+0000 7faf88733700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf801041a0 0x7faf80075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.406+0000 7faf88733700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf801088f0 0x7faf800762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.406+0000 7faf88733700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faf80113570 con 0x7faf801088f0 2024-06-04T05:08:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.406+0000 7faf88733700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faf801133f0 con 0x7faf801041a0 2024-06-04T05:08:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.406+0000 7faf88733700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7faf801136f0 con 0x7faf801037f0 2024-06-04T05:08:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.406+0000 7faf86cd0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf801088f0 0x7faf800762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.406+0000 7faf86cd0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf801088f0 0x7faf800762c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45456/0 (socket says 172.21.15.70:45456) 2024-06-04T05:08:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.406+0000 7faf86cd0700 1 -- 172.21.15.70:0/1643887315 learned_addr learned my addr 172.21.15.70:0/1643887315 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.406+0000 7faf864cf700 1 --2- 172.21.15.70:0/1643887315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faf801037f0 0x7faf80075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.406+0000 7faf85cce700 1 --2- 172.21.15.70:0/1643887315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf801041a0 0x7faf80075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.407+0000 7faf86cd0700 1 -- 172.21.15.70:0/1643887315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faf801037f0 msgr2=0x7faf80075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.407+0000 7faf86cd0700 1 --2- 172.21.15.70:0/1643887315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faf801037f0 0x7faf80075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.407+0000 7faf86cd0700 1 -- 172.21.15.70:0/1643887315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf801041a0 msgr2=0x7faf80075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.407+0000 7faf86cd0700 1 --2- 172.21.15.70:0/1643887315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf801041a0 0x7faf80075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.407+0000 7faf86cd0700 1 -- 172.21.15.70:0/1643887315 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf801a7e40 con 0x7faf801088f0 2024-06-04T05:08:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.407+0000 7faf864cf700 1 --2- 172.21.15.70:0/1643887315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faf801037f0 0x7faf80075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.407+0000 7faf85cce700 1 --2- 172.21.15.70:0/1643887315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf801041a0 0x7faf80075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.407+0000 7faf86cd0700 1 --2- 172.21.15.70:0/1643887315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf801088f0 0x7faf800762c0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7faf78007d90 tx=0x7faf78007e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.407+0000 7faf777fe700 1 -- 172.21.15.70:0/1643887315 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7801f030 con 0x7faf801088f0 2024-06-04T05:08:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.407+0000 7faf777fe700 1 -- 172.21.15.70:0/1643887315 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faf7801bb90 con 0x7faf801088f0 2024-06-04T05:08:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.407+0000 7faf88733700 1 -- 172.21.15.70:0/1643887315 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf801a80d0 con 0x7faf801088f0 2024-06-04T05:08:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.407+0000 7faf777fe700 1 -- 172.21.15.70:0/1643887315 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf78012c90 con 0x7faf801088f0 2024-06-04T05:08:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.407+0000 7faf88733700 1 -- 172.21.15.70:0/1643887315 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf801a8530 con 0x7faf801088f0 2024-06-04T05:08:04.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.409+0000 7faf777fe700 1 -- 172.21.15.70:0/1643887315 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7faf78022030 con 0x7faf801088f0 2024-06-04T05:08:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.410+0000 7faf777fe700 1 --2- 172.21.15.70:0/1643887315 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7faf6c0b29f0 0x7faf6c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.410+0000 7faf864cf700 1 --2- 172.21.15.70:0/1643887315 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7faf6c0b29f0 0x7faf6c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.410+0000 7faf777fe700 1 -- 172.21.15.70:0/1643887315 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7faf780e3830 con 0x7faf801088f0 2024-06-04T05:08:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.411+0000 7faf92fc8b80 1 Processor -- start 2024-06-04T05:08:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.411+0000 7faf864cf700 1 --2- 172.21.15.70:0/1643887315 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7faf6c0b29f0 0x7faf6c0b4eb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7faf7c005fd0 tx=0x7faf7c005e30 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.411+0000 7faf92fc8b80 1 -- start start 2024-06-04T05:08:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.411+0000 7faf92fc8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203c0320 0x5598203c0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.411+0000 7faf92fc8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203ba8a0 0x5598203bad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.411+0000 7faf92fc8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203bb260 0x5598203c7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.411+0000 7faf92fc8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559820318d70 con 0x5598203bb260 2024-06-04T05:08:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.411+0000 7faf92fc8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559820308910 con 0x5598203c0320 2024-06-04T05:08:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.411+0000 7faf92fc8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55982030c950 con 0x5598203ba8a0 2024-06-04T05:08:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.411+0000 7faf86cd0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203bb260 0x5598203c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.411+0000 7faf86cd0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203bb260 0x5598203c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45462/0 (socket says 172.21.15.70:45462) 2024-06-04T05:08:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.411+0000 7faf86cd0700 1 -- 172.21.15.70:0/3564341615 learned_addr learned my addr 172.21.15.70:0/3564341615 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.411+0000 7faf85cce700 1 --2- 172.21.15.70:0/3564341615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203ba8a0 0x5598203bad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.411+0000 7faf864cf700 1 --2- 172.21.15.70:0/3564341615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203c0320 0x5598203c0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.412+0000 7faf86cd0700 1 -- 172.21.15.70:0/3564341615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203ba8a0 msgr2=0x5598203bad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.412+0000 7faf86cd0700 1 --2- 172.21.15.70:0/3564341615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203ba8a0 0x5598203bad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.412+0000 7faf86cd0700 1 -- 172.21.15.70:0/3564341615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203c0320 msgr2=0x5598203c0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.412+0000 7faf86cd0700 1 --2- 172.21.15.70:0/3564341615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203c0320 0x5598203c0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.412+0000 7faf86cd0700 1 -- 172.21.15.70:0/3564341615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598202b1fb0 con 0x5598203bb260 2024-06-04T05:08:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.412+0000 7faf86cd0700 1 --2- 172.21.15.70:0/3564341615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203bb260 0x5598203c7080 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7faf780a6bf0 tx=0x7faf780aaa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac599ad6ac25437b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.412+0000 7faf74ff9700 1 -- 172.21.15.70:0/3564341615 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf78029070 con 0x5598203bb260 2024-06-04T05:08:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.412+0000 7faf74ff9700 1 -- 172.21.15.70:0/3564341615 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faf780aab70 con 0x5598203bb260 2024-06-04T05:08:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.412+0000 7faf74ff9700 1 -- 172.21.15.70:0/3564341615 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf780e11d0 con 0x5598203bb260 2024-06-04T05:08:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.413+0000 7faf92fc8b80 1 -- 172.21.15.70:0/3564341615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203bb260 msgr2=0x5598203c7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.413+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/3564341615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203bb260 0x5598203c7080 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7faf780a6bf0 tx=0x7faf780aaa70 comp rx=0 tx=0).stop 2024-06-04T05:08:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.413+0000 7faf92fc8b80 1 -- 172.21.15.70:0/3564341615 shutdown_connections 2024-06-04T05:08:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.413+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/3564341615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203bb260 0x5598203c7080 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.413+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/3564341615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203c0320 0x5598203c0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.413+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/3564341615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203ba8a0 0x5598203bad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.413+0000 7faf92fc8b80 1 -- 172.21.15.70:0/3564341615 >> 172.21.15.70:0/3564341615 conn(0x5598202e6240 msgr2=0x5598203b53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.413+0000 7faf92fc8b80 1 -- 172.21.15.70:0/3564341615 shutdown_connections 2024-06-04T05:08:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.413+0000 7faf92fc8b80 1 -- 172.21.15.70:0/3564341615 wait complete. 2024-06-04T05:08:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.413+0000 7faf92fc8b80 1 -- start start 2024-06-04T05:08:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.413+0000 7faf92fc8b80 1 Processor -- start 2024-06-04T05:08:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf92fc8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203ba8a0 0x5598203c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf92fc8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203bb260 0x5598203ca250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf92fc8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203c0320 0x5598203c4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf92fc8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55982020efa0 con 0x5598203ba8a0 2024-06-04T05:08:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf92fc8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559820270da0 con 0x5598203bb260 2024-06-04T05:08:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf92fc8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559820318d70 con 0x5598203c0320 2024-06-04T05:08:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf92fc8b80 10 client.0 _ll_register_callbacks cb 0x5598202a3950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf864cf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203ba8a0 0x5598203c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf864cf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203ba8a0 0x5598203c9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45478/0 (socket says 172.21.15.70:45478) 2024-06-04T05:08:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf864cf700 1 -- 172.21.15.70:0/747273227 learned_addr learned my addr 172.21.15.70:0/747273227 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf85cce700 1 --2- 172.21.15.70:0/747273227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203bb260 0x5598203ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf86cd0700 1 --2- 172.21.15.70:0/747273227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203c0320 0x5598203c4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf864cf700 1 -- 172.21.15.70:0/747273227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203c0320 msgr2=0x5598203c4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf864cf700 1 --2- 172.21.15.70:0/747273227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203c0320 0x5598203c4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf864cf700 1 -- 172.21.15.70:0/747273227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203bb260 msgr2=0x5598203ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf864cf700 1 --2- 172.21.15.70:0/747273227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203bb260 0x5598203ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf864cf700 1 -- 172.21.15.70:0/747273227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598202588d0 con 0x5598203ba8a0 2024-06-04T05:08:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf864cf700 1 --2- 172.21.15.70:0/747273227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203ba8a0 0x5598203c9d10 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7faf7c010990 tx=0x7faf7c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee11ea55fc1b8532 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:04.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf69ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:04.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf69ffb700 1 -- 172.21.15.70:0/747273227 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7c01d950 con 0x5598203ba8a0 2024-06-04T05:08:04.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf69ffb700 1 -- 172.21.15.70:0/747273227 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7faf7c01daf0 con 0x5598203ba8a0 2024-06-04T05:08:04.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.414+0000 7faf92fc8b80 1 -- 172.21.15.70:0/747273227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf780aab70 con 0x5598203ba8a0 2024-06-04T05:08:04.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.415+0000 7faf69ffb700 1 -- 172.21.15.70:0/747273227 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7faf7c01dcf0 con 0x5598203ba8a0 2024-06-04T05:08:04.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.415+0000 7faf92fc8b80 10 client.18366 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:04.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.415+0000 7faf92fc8b80 1 -- 172.21.15.70:0/747273227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5598202c2760 con 0x5598203ba8a0 2024-06-04T05:08:04.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.415+0000 7faf69ffb700 1 -- 172.21.15.70:0/747273227 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faf780aab70 con 0x5598203ba8a0 2024-06-04T05:08:04.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.415+0000 7faf69ffb700 1 -- 172.21.15.70:0/747273227 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7faf7c0263f0 con 0x5598203ba8a0 2024-06-04T05:08:04.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.415+0000 7faf92fc8b80 10 client.18366 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:04.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.415+0000 7faf92fc8b80 10 client.18366 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:08:04.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.415+0000 7faf92fc8b80 20 client.18366 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:04.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.415+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/747273227 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5598203d4010 0x5598203d6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.415+0000 7faf92fc8b80 4 client.18366 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:08:04.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.415+0000 7faf92fc8b80 1 -- 172.21.15.70:0/747273227 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55982031acc0 con 0x5598203d4010 2024-06-04T05:08:04.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.415+0000 7faf85cce700 1 --2- 172.21.15.70:0/747273227 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5598203d4010 0x5598203d6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:04.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.416+0000 7faf85cce700 1 --2- 172.21.15.70:0/747273227 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5598203d4010 0x5598203d6490 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26eb22741299fcc0 server_cookie=a2ad5a7a3b6b4952 in_seq=0 out_seq=0 2024-06-04T05:08:04.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.416+0000 7faf69ffb700 10 client.18366 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:04.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.416+0000 7faf69ffb700 1 -- 172.21.15.70:0/747273227 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55982031acc0 con 0x5598203d4010 2024-06-04T05:08:04.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.416+0000 7faf69ffb700 10 client.18366 handle_command_reply: tid=0 2024-06-04T05:08:04.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.416+0000 7faf92fc8b80 1 client.18366 shutdown 2024-06-04T05:08:04.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.416+0000 7faf92fc8b80 1 -- 172.21.15.70:0/747273227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203ba8a0 msgr2=0x5598203c9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:04.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.416+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/747273227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203ba8a0 0x5598203c9d10 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7faf7c010990 tx=0x7faf7c004750 comp rx=0 tx=0).stop 2024-06-04T05:08:04.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.416+0000 7faf92fc8b80 1 -- 172.21.15.70:0/747273227 shutdown_connections 2024-06-04T05:08:04.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.416+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/747273227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203c0320 0x5598203c4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.416+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/747273227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203ba8a0 0x5598203c9d10 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.417+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/747273227 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5598203d4010 0x5598203d6490 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.417+0000 7faf69ffb700 0 client.18366 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:04.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.417+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/747273227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203bb260 0x5598203ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.417+0000 7faf92fc8b80 1 -- 172.21.15.70:0/747273227 >> 172.21.15.70:0/747273227 conn(0x5598202e6240 msgr2=0x5598203b4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:04.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.417+0000 7faf92fc8b80 1 -- 172.21.15.70:0/747273227 shutdown_connections 2024-06-04T05:08:04.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.417+0000 7faf92fc8b80 1 -- 172.21.15.70:0/747273227 wait complete. 2024-06-04T05:08:04.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.417+0000 7faf92fc8b80 20 client.18366 trim_cache size 0 max 16384 2024-06-04T05:08:04.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.437+0000 7faf92fc8b80 1 Processor -- start 2024-06-04T05:08:04.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.437+0000 7faf92fc8b80 1 -- start start 2024-06-04T05:08:04.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.437+0000 7faf92fc8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203ba8a0 0x5598203c41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.437+0000 7faf92fc8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203bb260 0x5598203c4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.437+0000 7faf92fc8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203c0320 0x5598203bf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.437+0000 7faf92fc8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faf801afe50 con 0x5598203ba8a0 2024-06-04T05:08:04.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.437+0000 7faf92fc8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faf801b19f0 con 0x5598203bb260 2024-06-04T05:08:04.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.437+0000 7faf92fc8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7faf801b0520 con 0x5598203c0320 2024-06-04T05:08:04.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.437+0000 7faf864cf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203ba8a0 0x5598203c41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:04.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.437+0000 7faf864cf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203ba8a0 0x5598203c41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45482/0 (socket says 172.21.15.70:45482) 2024-06-04T05:08:04.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.437+0000 7faf864cf700 1 -- 172.21.15.70:0/2453639904 learned_addr learned my addr 172.21.15.70:0/2453639904 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:04.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.437+0000 7faf864cf700 1 -- 172.21.15.70:0/2453639904 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203c0320 msgr2=0x5598203bf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:04.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.437+0000 7faf864cf700 1 --2- 172.21.15.70:0/2453639904 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203c0320 0x5598203bf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.437+0000 7faf864cf700 1 -- 172.21.15.70:0/2453639904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203bb260 msgr2=0x5598203c4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:04.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.437+0000 7faf864cf700 1 --2- 172.21.15.70:0/2453639904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203bb260 0x5598203c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.437+0000 7faf864cf700 1 -- 172.21.15.70:0/2453639904 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598203ca5c0 con 0x5598203ba8a0 2024-06-04T05:08:04.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.437+0000 7faf864cf700 1 --2- 172.21.15.70:0/2453639904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203ba8a0 0x5598203c41d0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7faf7c010960 tx=0x7faf7c00a9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d6ad7c56d7a126 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:04.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.438+0000 7faf6bfff700 1 -- 172.21.15.70:0/2453639904 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7c01de90 con 0x5598203ba8a0 2024-06-04T05:08:04.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.438+0000 7faf6bfff700 1 -- 172.21.15.70:0/2453639904 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faf7c03b070 con 0x5598203ba8a0 2024-06-04T05:08:04.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.438+0000 7faf6bfff700 1 -- 172.21.15.70:0/2453639904 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7c01e690 con 0x5598203ba8a0 2024-06-04T05:08:04.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.438+0000 7faf92fc8b80 1 -- 172.21.15.70:0/2453639904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203ba8a0 msgr2=0x5598203c41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:04.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.438+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/2453639904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203ba8a0 0x5598203c41d0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7faf7c010960 tx=0x7faf7c00a9d0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.438+0000 7faf92fc8b80 1 -- 172.21.15.70:0/2453639904 shutdown_connections 2024-06-04T05:08:04.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.438+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/2453639904 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203c0320 0x5598203bf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.438+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/2453639904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203ba8a0 0x5598203c41d0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.438+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/2453639904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203bb260 0x5598203c4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.438+0000 7faf92fc8b80 1 -- 172.21.15.70:0/2453639904 >> 172.21.15.70:0/2453639904 conn(0x5598202e6240 msgr2=0x5598203b9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:04.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.438+0000 7faf92fc8b80 1 -- 172.21.15.70:0/2453639904 shutdown_connections 2024-06-04T05:08:04.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.438+0000 7faf92fc8b80 1 -- 172.21.15.70:0/2453639904 wait complete. 2024-06-04T05:08:04.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf92fc8b80 1 -- start start 2024-06-04T05:08:04.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf92fc8b80 1 Processor -- start 2024-06-04T05:08:04.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf92fc8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203ba8a0 0x5598203cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf92fc8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203bb260 0x5598203cc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf92fc8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203c0320 0x5598203cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf92fc8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faf801b12f0 con 0x5598203bb260 2024-06-04T05:08:04.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf92fc8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faf801b0bf0 con 0x5598203c0320 2024-06-04T05:08:04.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf92fc8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7faf801ade10 con 0x5598203ba8a0 2024-06-04T05:08:04.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf92fc8b80 10 client.0 _ll_register_callbacks cb 0x5598202a3950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:04.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf85cce700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203bb260 0x5598203cc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:04.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf85cce700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203bb260 0x5598203cc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45498/0 (socket says 172.21.15.70:45498) 2024-06-04T05:08:04.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf85cce700 1 -- 172.21.15.70:0/3509065359 learned_addr learned my addr 172.21.15.70:0/3509065359 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:04.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf85cce700 1 -- 172.21.15.70:0/3509065359 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203ba8a0 msgr2=0x5598203cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:04.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf85cce700 1 --2- 172.21.15.70:0/3509065359 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203ba8a0 0x5598203cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf85cce700 1 -- 172.21.15.70:0/3509065359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203c0320 msgr2=0x5598203cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:04.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf85cce700 1 --2- 172.21.15.70:0/3509065359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203c0320 0x5598203cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf85cce700 1 -- 172.21.15.70:0/3509065359 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55982043ddf0 con 0x5598203bb260 2024-06-04T05:08:04.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf85cce700 1 --2- 172.21.15.70:0/3509065359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203bb260 0x5598203cc530 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7faf70005fd0 tx=0x7faf70012a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=864e1278460713fe server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:04.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.439+0000 7faf69ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:04.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.440+0000 7faf69ffb700 1 -- 172.21.15.70:0/3509065359 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7001c630 con 0x5598203bb260 2024-06-04T05:08:04.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.440+0000 7faf69ffb700 1 -- 172.21.15.70:0/3509065359 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7faf7001c7d0 con 0x5598203bb260 2024-06-04T05:08:04.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.440+0000 7faf92fc8b80 1 -- 172.21.15.70:0/3509065359 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf7c03b070 con 0x5598203bb260 2024-06-04T05:08:04.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.440+0000 7faf69ffb700 1 -- 172.21.15.70:0/3509065359 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7faf7002b050 con 0x5598203bb260 2024-06-04T05:08:04.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.440+0000 7faf92fc8b80 10 client.18372 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:04.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.440+0000 7faf92fc8b80 1 -- 172.21.15.70:0/3509065359 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55982043f120 con 0x5598203bb260 2024-06-04T05:08:04.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.440+0000 7faf69ffb700 1 -- 172.21.15.70:0/3509065359 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faf7c03b070 con 0x5598203bb260 2024-06-04T05:08:04.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.440+0000 7faf69ffb700 1 -- 172.21.15.70:0/3509065359 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7faf70010c10 con 0x5598203bb260 2024-06-04T05:08:04.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.440+0000 7faf92fc8b80 10 client.18372 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:04.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.440+0000 7faf92fc8b80 10 client.18372 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:08:04.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.440+0000 7faf92fc8b80 20 client.18372 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:04.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.440+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/3509065359 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5598203d4010 0x559820441870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:04.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.440+0000 7faf92fc8b80 4 client.18372 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:08:04.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.440+0000 7faf92fc8b80 1 -- 172.21.15.70:0/3509065359 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5598203144a0 con 0x5598203d4010 2024-06-04T05:08:04.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.441+0000 7faf864cf700 1 --2- 172.21.15.70:0/3509065359 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5598203d4010 0x559820441870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:04.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.441+0000 7faf864cf700 1 --2- 172.21.15.70:0/3509065359 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5598203d4010 0x559820441870 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c49f9d028395d72 server_cookie=8bbd03121d38bc6 in_seq=0 out_seq=0 2024-06-04T05:08:04.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.441+0000 7faf69ffb700 10 client.18372 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:04.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.441+0000 7faf69ffb700 1 -- 172.21.15.70:0/3509065359 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5598203144a0 con 0x5598203d4010 2024-06-04T05:08:04.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.441+0000 7faf69ffb700 10 client.18372 handle_command_reply: tid=0 2024-06-04T05:08:04.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.441+0000 7faf92fc8b80 1 client.18372 shutdown 2024-06-04T05:08:04.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.441+0000 7faf92fc8b80 1 -- 172.21.15.70:0/3509065359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203bb260 msgr2=0x5598203cc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:04.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.441+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/3509065359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203bb260 0x5598203cc530 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7faf70005fd0 tx=0x7faf70012a60 comp rx=0 tx=0).stop 2024-06-04T05:08:04.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.441+0000 7faf92fc8b80 1 -- 172.21.15.70:0/3509065359 shutdown_connections 2024-06-04T05:08:04.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.441+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/3509065359 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5598203d4010 0x559820441870 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.441+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/3509065359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598203c0320 0x5598203cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.441+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/3509065359 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5598203ba8a0 0x5598203cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.441+0000 7faf69ffb700 0 client.18372 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:04.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.441+0000 7faf92fc8b80 1 --2- 172.21.15.70:0/3509065359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598203bb260 0x5598203cc530 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.441+0000 7faf92fc8b80 1 -- 172.21.15.70:0/3509065359 >> 172.21.15.70:0/3509065359 conn(0x5598202e6240 msgr2=0x5598203c3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:04.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.441+0000 7faf92fc8b80 1 -- 172.21.15.70:0/3509065359 shutdown_connections 2024-06-04T05:08:04.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.441+0000 7faf92fc8b80 1 -- 172.21.15.70:0/3509065359 wait complete. 2024-06-04T05:08:04.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.442+0000 7faf92fc8b80 20 client.18372 trim_cache size 0 max 16384 2024-06-04T05:08:04.450 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:08:04.450 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:08:04.450 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:08:04.450 INFO:teuthology.orchestra.run.smithi070.stdout: "45bb262b-4741-4606-a59b-8159ac404175": { 2024-06-04T05:08:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:08:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "45bb262b-4741-4606-a59b-8159ac404175", 2024-06-04T05:08:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:08:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:04.451 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:08:04.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.442+0000 7faf88733700 1 -- 172.21.15.70:0/1643887315 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7faf6c0b29f0 msgr2=0x7faf6c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:04.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.442+0000 7faf88733700 1 --2- 172.21.15.70:0/1643887315 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7faf6c0b29f0 0x7faf6c0b4eb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7faf7c005fd0 tx=0x7faf7c005e30 comp rx=0 tx=0).stop 2024-06-04T05:08:04.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.442+0000 7faf88733700 1 -- 172.21.15.70:0/1643887315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf801088f0 msgr2=0x7faf800762c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:04.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.442+0000 7faf88733700 1 --2- 172.21.15.70:0/1643887315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf801088f0 0x7faf800762c0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7faf78007d90 tx=0x7faf78007e70 comp rx=0 tx=0).stop 2024-06-04T05:08:04.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.442+0000 7faf88733700 1 -- 172.21.15.70:0/1643887315 shutdown_connections 2024-06-04T05:08:04.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.442+0000 7faf88733700 1 --2- 172.21.15.70:0/1643887315 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7faf6c0b29f0 0x7faf6c0b4eb0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.442+0000 7faf88733700 1 --2- 172.21.15.70:0/1643887315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf801088f0 0x7faf800762c0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.442+0000 7faf88733700 1 --2- 172.21.15.70:0/1643887315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faf801037f0 0x7faf80075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.442+0000 7faf88733700 1 --2- 172.21.15.70:0/1643887315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf801041a0 0x7faf80075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:04.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.442+0000 7faf88733700 1 -- 172.21.15.70:0/1643887315 >> 172.21.15.70:0/1643887315 conn(0x7faf8007a6a0 msgr2=0x7faf80101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:04.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.442+0000 7faf88733700 1 -- 172.21.15.70:0/1643887315 shutdown_connections 2024-06-04T05:08:04.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:04.442+0000 7faf88733700 1 -- 172.21.15.70:0/1643887315 wait complete. 2024-06-04T05:08:04.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 188972 1.85 MB/sec execute 417 sec latency 4722.140 ms 2024-06-04T05:08:05.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:05 smithi112 ceph-mon[36453]: pgmap v336: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 911 KiB/s wr, 114 op/s 2024-06-04T05:08:05.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:05 smithi098 ceph-mon[35569]: pgmap v336: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 911 KiB/s wr, 114 op/s 2024-06-04T05:08:05.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:05 smithi070 ceph-mon[32631]: pgmap v336: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 911 KiB/s wr, 114 op/s 2024-06-04T05:08:05.533 INFO:tasks.cephfs.filesystem:scrub status for tag:45bb262b-4741-4606-a59b-8159ac404175 - {'path': '/', 'tag': '45bb262b-4741-4606-a59b-8159ac404175', 'options': 'recursive,force'} 2024-06-04T05:08:05.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 188972 1.85 MB/sec execute 418 sec latency 5722.253 ms 2024-06-04T05:08:06.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.581+0000 7fdd144c1700 1 -- 172.21.15.70:0/3393803609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd0c1041a0 msgr2=0x7fdd0c1081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:06.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.581+0000 7fdd144c1700 1 --2- 172.21.15.70:0/3393803609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd0c1041a0 0x7fdd0c1081c0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fdd000099c0 tx=0x7fdd00009cd0 comp rx=0 tx=0).stop 2024-06-04T05:08:06.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.581+0000 7fdd144c1700 1 -- 172.21.15.70:0/3393803609 shutdown_connections 2024-06-04T05:08:06.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.581+0000 7fdd144c1700 1 --2- 172.21.15.70:0/3393803609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd0c1088f0 0x7fdd0c110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:06.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.581+0000 7fdd144c1700 1 --2- 172.21.15.70:0/3393803609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd0c1037f0 0x7fdd0c103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:06.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.581+0000 7fdd144c1700 1 --2- 172.21.15.70:0/3393803609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd0c1041a0 0x7fdd0c1081c0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.581+0000 7fdd144c1700 1 -- 172.21.15.70:0/3393803609 >> 172.21.15.70:0/3393803609 conn(0x7fdd0c07a6a0 msgr2=0x7fdd0c07aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.582+0000 7fdd144c1700 1 -- 172.21.15.70:0/3393803609 shutdown_connections 2024-06-04T05:08:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.582+0000 7fdd144c1700 1 -- 172.21.15.70:0/3393803609 wait complete. 2024-06-04T05:08:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.582+0000 7fdd144c1700 1 Processor -- start 2024-06-04T05:08:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.582+0000 7fdd144c1700 1 -- start start 2024-06-04T05:08:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.582+0000 7fdd144c1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd0c1037f0 0x7fdd0c075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.582+0000 7fdd144c1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd0c1041a0 0x7fdd0c075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.582+0000 7fdd144c1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd0c1088f0 0x7fdd0c0762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.582+0000 7fdd144c1700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0c113580 con 0x7fdd0c1037f0 2024-06-04T05:08:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.582+0000 7fdd144c1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0c113400 con 0x7fdd0c1041a0 2024-06-04T05:08:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.582+0000 7fdd144c1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0c113700 con 0x7fdd0c1088f0 2024-06-04T05:08:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.583+0000 7fdd1225d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd0c1037f0 0x7fdd0c075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.583+0000 7fdd1225d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd0c1037f0 0x7fdd0c075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58516/0 (socket says 172.21.15.70:58516) 2024-06-04T05:08:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.583+0000 7fdd1225d700 1 -- 172.21.15.70:0/238438842 learned_addr learned my addr 172.21.15.70:0/238438842 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.583+0000 7fdd12a5e700 1 --2- 172.21.15.70:0/238438842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd0c1088f0 0x7fdd0c0762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.583+0000 7fdd11a5c700 1 --2- 172.21.15.70:0/238438842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd0c1041a0 0x7fdd0c075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.583+0000 7fdd1225d700 1 -- 172.21.15.70:0/238438842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd0c1088f0 msgr2=0x7fdd0c0762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.583+0000 7fdd1225d700 1 --2- 172.21.15.70:0/238438842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd0c1088f0 0x7fdd0c0762c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.583+0000 7fdd1225d700 1 -- 172.21.15.70:0/238438842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd0c1041a0 msgr2=0x7fdd0c075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.583+0000 7fdd1225d700 1 --2- 172.21.15.70:0/238438842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd0c1041a0 0x7fdd0c075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.583+0000 7fdd1225d700 1 -- 172.21.15.70:0/238438842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd0c1a7fb0 con 0x7fdd0c1037f0 2024-06-04T05:08:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.583+0000 7fdd1225d700 1 --2- 172.21.15.70:0/238438842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd0c1037f0 0x7fdd0c075840 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fdd0800d950 tx=0x7fdd0800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:06.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.583+0000 7fdcff7fe700 1 -- 172.21.15.70:0/238438842 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd08016070 con 0x7fdd0c1037f0 2024-06-04T05:08:06.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.583+0000 7fdcff7fe700 1 -- 172.21.15.70:0/238438842 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd08010460 con 0x7fdd0c1037f0 2024-06-04T05:08:06.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.583+0000 7fdd144c1700 1 -- 172.21.15.70:0/238438842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd0c1a82a0 con 0x7fdd0c1037f0 2024-06-04T05:08:06.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.583+0000 7fdcff7fe700 1 -- 172.21.15.70:0/238438842 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0800f620 con 0x7fdd0c1037f0 2024-06-04T05:08:06.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.583+0000 7fdd144c1700 1 -- 172.21.15.70:0/238438842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd0c1a8800 con 0x7fdd0c1037f0 2024-06-04T05:08:06.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.584+0000 7fdd144c1700 1 -- 172.21.15.70:0/238438842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd0c06a230 con 0x7fdd0c1037f0 2024-06-04T05:08:06.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.586+0000 7fdcff7fe700 1 -- 172.21.15.70:0/238438842 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fdd08019070 con 0x7fdd0c1037f0 2024-06-04T05:08:06.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.587+0000 7fdcff7fe700 1 --2- 172.21.15.70:0/238438842 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdcf80b2a70 0x7fdcf80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:06.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.587+0000 7fdd11a5c700 1 --2- 172.21.15.70:0/238438842 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdcf80b2a70 0x7fdcf80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:06.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.587+0000 7fdcff7fe700 1 -- 172.21.15.70:0/238438842 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8571+0+0 (secure 0 0 0) 0x7fdd080e2590 con 0x7fdd0c1037f0 2024-06-04T05:08:06.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.587+0000 7fdd11a5c700 1 --2- 172.21.15.70:0/238438842 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdcf80b2a70 0x7fdcf80b4f30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdd00000c00 tx=0x7fdd0001a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:06.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.591+0000 7fdcff7fe700 1 -- 172.21.15.70:0/238438842 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdd080a9f40 con 0x7fdd0c1037f0 2024-06-04T05:08:06.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 188972 1.84 MB/sec execute 419 sec latency 6722.281 ms 2024-06-04T05:08:06.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.841+0000 7fdd144c1700 1 -- 172.21.15.70:0/238438842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd0c02e720 con 0x7fdd0c1037f0 2024-06-04T05:08:06.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.842+0000 7fdcff7fe700 1 -- 172.21.15.70:0/238438842 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11931 (secure 0 0 0) 0x7fdd080a9840 con 0x7fdd0c1037f0 2024-06-04T05:08:06.844 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:08:06.844 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:07:27.814997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24436,24410,24440,24448,24442]},"id":1}]} 2024-06-04T05:08:06.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.844+0000 7fdd144c1700 1 -- 172.21.15.70:0/238438842 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdcf80b2a70 msgr2=0x7fdcf80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:06.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.844+0000 7fdd144c1700 1 --2- 172.21.15.70:0/238438842 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdcf80b2a70 0x7fdcf80b4f30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdd00000c00 tx=0x7fdd0001a040 comp rx=0 tx=0).stop 2024-06-04T05:08:06.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.844+0000 7fdd144c1700 1 -- 172.21.15.70:0/238438842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd0c1037f0 msgr2=0x7fdd0c075840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:06.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.844+0000 7fdd144c1700 1 --2- 172.21.15.70:0/238438842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd0c1037f0 0x7fdd0c075840 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fdd0800d950 tx=0x7fdd0800dd10 comp rx=0 tx=0).stop 2024-06-04T05:08:06.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.844+0000 7fdd144c1700 1 -- 172.21.15.70:0/238438842 shutdown_connections 2024-06-04T05:08:06.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.844+0000 7fdd144c1700 1 --2- 172.21.15.70:0/238438842 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdcf80b2a70 0x7fdcf80b4f30 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:06.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.844+0000 7fdd144c1700 1 --2- 172.21.15.70:0/238438842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd0c1088f0 0x7fdd0c0762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:06.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.844+0000 7fdd144c1700 1 --2- 172.21.15.70:0/238438842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd0c1037f0 0x7fdd0c075840 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:06.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.844+0000 7fdd144c1700 1 --2- 172.21.15.70:0/238438842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd0c1041a0 0x7fdd0c075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:06.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.844+0000 7fdd144c1700 1 -- 172.21.15.70:0/238438842 >> 172.21.15.70:0/238438842 conn(0x7fdd0c07a6a0 msgr2=0x7fdd0c101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:06.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.844+0000 7fdd144c1700 1 -- 172.21.15.70:0/238438842 shutdown_connections 2024-06-04T05:08:06.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:06.844+0000 7fdd144c1700 1 -- 172.21.15.70:0/238438842 wait complete. 2024-06-04T05:08:06.847 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 34 2024-06-04T05:08:07.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:07 smithi112 ceph-mon[36453]: pgmap v337: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 920 KiB/s wr, 120 op/s 2024-06-04T05:08:07.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:07 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/238438842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:08:07.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:07 smithi098 ceph-mon[35569]: pgmap v337: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 920 KiB/s wr, 120 op/s 2024-06-04T05:08:07.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:07 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/238438842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:08:07.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:07 smithi070 ceph-mon[32631]: pgmap v337: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 920 KiB/s wr, 120 op/s 2024-06-04T05:08:07.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:07 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/238438842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:08:07.551 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id c91c6358 --release --await' 2024-06-04T05:08:07.551 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id c91c6358 --release --await 2024-06-04T05:08:07.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 188972 1.84 MB/sec execute 420 sec latency 7722.330 ms 2024-06-04T05:08:07.797 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:08:08.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:08 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:08:08.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:08 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:08:08.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:08 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:08:08.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:08 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:08:08.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:08 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:08:08.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:08 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:08:08.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 188972 1.84 MB/sec execute 421 sec latency 8722.388 ms 2024-06-04T05:08:09.283 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:09 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:08:09.283 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:09 smithi070 ceph-mon[32631]: pgmap v338: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 410 KiB/s wr, 82 op/s 2024-06-04T05:08:09.283 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:09 smithi070 ceph-mon[32631]: osdmap e89: 12 total, 12 up, 12 in 2024-06-04T05:08:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:09 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:08:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:09 smithi112 ceph-mon[36453]: pgmap v338: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 410 KiB/s wr, 82 op/s 2024-06-04T05:08:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:09 smithi112 ceph-mon[36453]: osdmap e89: 12 total, 12 up, 12 in 2024-06-04T05:08:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:09 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:08:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:09 smithi098 ceph-mon[35569]: pgmap v338: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 410 KiB/s wr, 82 op/s 2024-06-04T05:08:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:09 smithi098 ceph-mon[35569]: osdmap e89: 12 total, 12 up, 12 in 2024-06-04T05:08:09.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.517+0000 7f62ad3a3700 1 -- 172.21.15.70:0/1538173921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62a806bbe0 msgr2=0x7f62a81103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:09.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.517+0000 7f62ad3a3700 1 --2- 172.21.15.70:0/1538173921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62a806bbe0 0x7f62a81103d0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f62a0009990 tx=0x7f62a0009ca0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.517+0000 7f62ad3a3700 1 -- 172.21.15.70:0/1538173921 shutdown_connections 2024-06-04T05:08:09.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.517+0000 7f62ad3a3700 1 --2- 172.21.15.70:0/1538173921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62a806bbe0 0x7f62a81103d0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.517+0000 7f62ad3a3700 1 --2- 172.21.15.70:0/1538173921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a81095b0 0x7f62a8109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.517+0000 7f62ad3a3700 1 --2- 172.21.15.70:0/1538173921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62a806b220 0x7f62a806b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.517+0000 7f62ad3a3700 1 -- 172.21.15.70:0/1538173921 >> 172.21.15.70:0/1538173921 conn(0x7f62a80793f0 msgr2=0x7f62a8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:09.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.517+0000 7f62ad3a3700 1 -- 172.21.15.70:0/1538173921 shutdown_connections 2024-06-04T05:08:09.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.517+0000 7f62ad3a3700 1 -- 172.21.15.70:0/1538173921 wait complete. 2024-06-04T05:08:09.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.517+0000 7f62ad3a3700 1 Processor -- start 2024-06-04T05:08:09.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62ad3a3700 1 -- start start 2024-06-04T05:08:09.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62ad3a3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62a806b220 0x7f62a8102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62ad3a3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62a806bbe0 0x7f62a8102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62ad3a3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a81095b0 0x7f62a8102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62ad3a3700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62a81135a0 con 0x7f62a806bbe0 2024-06-04T05:08:09.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62ad3a3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62a8113420 con 0x7f62a81095b0 2024-06-04T05:08:09.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62ad3a3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f62a8113720 con 0x7f62a806b220 2024-06-04T05:08:09.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62a67fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62a806bbe0 0x7f62a8102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62a67fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62a806bbe0 0x7f62a8102a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58520/0 (socket says 172.21.15.70:58520) 2024-06-04T05:08:09.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62a67fc700 1 -- 172.21.15.70:0/3512361246 learned_addr learned my addr 172.21.15.70:0/3512361246 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:09.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62a77fe700 1 --2- 172.21.15.70:0/3512361246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a81095b0 0x7f62a8102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62a67fc700 1 -- 172.21.15.70:0/3512361246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62a806b220 msgr2=0x7f62a8102510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:09.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62a6ffd700 1 --2- 172.21.15.70:0/3512361246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62a806b220 0x7f62a8102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62a67fc700 1 --2- 172.21.15.70:0/3512361246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62a806b220 0x7f62a8102510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62a67fc700 1 -- 172.21.15.70:0/3512361246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a81095b0 msgr2=0x7f62a8102f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:09.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62a67fc700 1 --2- 172.21.15.70:0/3512361246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a81095b0 0x7f62a8102f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62a67fc700 1 -- 172.21.15.70:0/3512361246 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62a81a8020 con 0x7f62a806bbe0 2024-06-04T05:08:09.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62a6ffd700 1 --2- 172.21.15.70:0/3512361246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62a806b220 0x7f62a8102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:08:09.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.518+0000 7f62a77fe700 1 --2- 172.21.15.70:0/3512361246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a81095b0 0x7f62a8102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:09.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.519+0000 7f62a67fc700 1 --2- 172.21.15.70:0/3512361246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62a806bbe0 0x7f62a8102a50 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f629c00d980 tx=0x7f629c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:09.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.519+0000 7f628ffff700 1 -- 172.21.15.70:0/3512361246 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629c017070 con 0x7f62a806bbe0 2024-06-04T05:08:09.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.519+0000 7f62ad3a3700 1 -- 172.21.15.70:0/3512361246 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62a81a8310 con 0x7f62a806bbe0 2024-06-04T05:08:09.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.519+0000 7f628ffff700 1 -- 172.21.15.70:0/3512361246 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f629c004830 con 0x7f62a806bbe0 2024-06-04T05:08:09.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.519+0000 7f628ffff700 1 -- 172.21.15.70:0/3512361246 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629c0053a0 con 0x7f62a806bbe0 2024-06-04T05:08:09.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.519+0000 7f62ad3a3700 1 -- 172.21.15.70:0/3512361246 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62a81a8850 con 0x7f62a806bbe0 2024-06-04T05:08:09.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.521+0000 7f628ffff700 1 -- 172.21.15.70:0/3512361246 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f629c005540 con 0x7f62a806bbe0 2024-06-04T05:08:09.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.522+0000 7f628ffff700 1 --2- 172.21.15.70:0/3512361246 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62900b29f0 0x7f62900b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.522+0000 7f62a6ffd700 1 --2- 172.21.15.70:0/3512361246 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62900b29f0 0x7f62900b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.522+0000 7f628ffff700 1 -- 172.21.15.70:0/3512361246 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8615+0+0 (secure 0 0 0) 0x7f629c0e2620 con 0x7f62a806bbe0 2024-06-04T05:08:09.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.522+0000 7f62b7c38b80 1 Processor -- start 2024-06-04T05:08:09.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.522+0000 7f62b7c38b80 1 -- start start 2024-06-04T05:08:09.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62a6ffd700 1 --2- 172.21.15.70:0/3512361246 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62900b29f0 0x7f62900b4eb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f62980060b0 tx=0x7f6298007620 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:09.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62b7c38b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62b7c38b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0bcbf0 0x555ceb0bd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62b7c38b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bd5b0 0x555ceb0c9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62b7c38b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555ceb01ae70 con 0x555ceb0c1720 2024-06-04T05:08:09.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62b7c38b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555ceb00aa10 con 0x555ceb0bcbf0 2024-06-04T05:08:09.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62b7c38b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555ceb00ea50 con 0x555ceb0bd5b0 2024-06-04T05:08:09.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62a6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62a6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58530/0 (socket says 172.21.15.70:58530) 2024-06-04T05:08:09.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62a6ffd700 1 -- 172.21.15.70:0/2688025828 learned_addr learned my addr 172.21.15.70:0/2688025828 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:09.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62a67fc700 1 --2- 172.21.15.70:0/2688025828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0bcbf0 0x555ceb0bd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62a77fe700 1 --2- 172.21.15.70:0/2688025828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bd5b0 0x555ceb0c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62a6ffd700 1 -- 172.21.15.70:0/2688025828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bd5b0 msgr2=0x555ceb0c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:09.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62a6ffd700 1 --2- 172.21.15.70:0/2688025828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bd5b0 0x555ceb0c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62a6ffd700 1 -- 172.21.15.70:0/2688025828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0bcbf0 msgr2=0x555ceb0bd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:09.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62a6ffd700 1 --2- 172.21.15.70:0/2688025828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0bcbf0 0x555ceb0bd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62a6ffd700 1 -- 172.21.15.70:0/2688025828 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ceafb40b0 con 0x555ceb0c1720 2024-06-04T05:08:09.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f62a6ffd700 1 --2- 172.21.15.70:0/2688025828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c1b00 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f62980177c0 tx=0x7f6298010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa66e37de829f867 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:09.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.523+0000 7f628d7fa700 1 -- 172.21.15.70:0/2688025828 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6298025070 con 0x555ceb0c1720 2024-06-04T05:08:09.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.524+0000 7f628d7fa700 1 -- 172.21.15.70:0/2688025828 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f62980172e0 con 0x555ceb0c1720 2024-06-04T05:08:09.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.524+0000 7f628d7fa700 1 -- 172.21.15.70:0/2688025828 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6298020670 con 0x555ceb0c1720 2024-06-04T05:08:09.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.524+0000 7f62b7c38b80 1 -- 172.21.15.70:0/2688025828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 msgr2=0x555ceb0c1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:09.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.524+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/2688025828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c1b00 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f62980177c0 tx=0x7f6298010000 comp rx=0 tx=0).stop 2024-06-04T05:08:09.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.524+0000 7f62b7c38b80 1 -- 172.21.15.70:0/2688025828 shutdown_connections 2024-06-04T05:08:09.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.524+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/2688025828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bd5b0 0x555ceb0c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.524+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/2688025828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c1b00 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.524+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/2688025828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0bcbf0 0x555ceb0bd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.524+0000 7f62b7c38b80 1 -- 172.21.15.70:0/2688025828 >> 172.21.15.70:0/2688025828 conn(0x555ceafe8340 msgr2=0x555ceb0b78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:09.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.524+0000 7f62b7c38b80 1 -- 172.21.15.70:0/2688025828 shutdown_connections 2024-06-04T05:08:09.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.524+0000 7f62b7c38b80 1 -- 172.21.15.70:0/2688025828 wait complete. 2024-06-04T05:08:09.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62b7c38b80 1 -- start start 2024-06-04T05:08:09.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62b7c38b80 1 Processor -- start 2024-06-04T05:08:09.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62b7c38b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bcbf0 0x555ceb0c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62b7c38b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0bd5b0 0x555ceb0c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62b7c38b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62b7c38b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555ceaf110a0 con 0x555ceb0c1720 2024-06-04T05:08:09.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62b7c38b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555ceaf72ea0 con 0x555ceb0bd5b0 2024-06-04T05:08:09.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62b7c38b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555ceb0c9c80 con 0x555ceb0bcbf0 2024-06-04T05:08:09.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62b7c38b80 10 client.0 _ll_register_callbacks cb 0x555ceafa5a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:09.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62a77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62a77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58538/0 (socket says 172.21.15.70:58538) 2024-06-04T05:08:09.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62a77fe700 1 -- 172.21.15.70:0/1695799966 learned_addr learned my addr 172.21.15.70:0/1695799966 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:09.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62a6ffd700 1 --2- 172.21.15.70:0/1695799966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bcbf0 0x555ceb0c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62a67fc700 1 --2- 172.21.15.70:0/1695799966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0bd5b0 0x555ceb0c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62a77fe700 1 -- 172.21.15.70:0/1695799966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bcbf0 msgr2=0x555ceb0c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:09.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62a77fe700 1 --2- 172.21.15.70:0/1695799966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bcbf0 0x555ceb0c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62a77fe700 1 -- 172.21.15.70:0/1695799966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0bd5b0 msgr2=0x555ceb0c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:09.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62a77fe700 1 --2- 172.21.15.70:0/1695799966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0bd5b0 0x555ceb0c65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62a77fe700 1 -- 172.21.15.70:0/1695799966 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ceaf5a9d0 con 0x555ceb0c1720 2024-06-04T05:08:09.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.525+0000 7f62a67fc700 1 --2- 172.21.15.70:0/1695799966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0bd5b0 0x555ceb0c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:08:09.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.526+0000 7f62a77fe700 1 --2- 172.21.15.70:0/1695799966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c6b10 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f62a0015000 tx=0x7f62a0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f71fd106b75075d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:09.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.526+0000 7f628aff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:09.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.526+0000 7f628aff5700 1 -- 172.21.15.70:0/1695799966 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a0003bd0 con 0x555ceb0c1720 2024-06-04T05:08:09.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.526+0000 7f62b7c38b80 1 -- 172.21.15.70:0/1695799966 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555ceb01cdc0 con 0x555ceb0c1720 2024-06-04T05:08:09.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.526+0000 7f628aff5700 1 -- 172.21.15.70:0/1695799966 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8615+0+0 (secure 0 0 0) 0x7f62a0003d70 con 0x555ceb0c1720 2024-06-04T05:08:09.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.526+0000 7f628aff5700 1 -- 172.21.15.70:0/1695799966 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f62a0031050 con 0x555ceb0c1720 2024-06-04T05:08:09.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.526+0000 7f62b7c38b80 10 client.18390 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:09.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.526+0000 7f62b7c38b80 1 -- 172.21.15.70:0/1695799966 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x555ceafc4860 con 0x555ceb0c1720 2024-06-04T05:08:09.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.526+0000 7f628aff5700 1 -- 172.21.15.70:0/1695799966 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x555ceb01cdc0 con 0x555ceb0c1720 2024-06-04T05:08:09.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.526+0000 7f628aff5700 1 -- 172.21.15.70:0/1695799966 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f62a0023730 con 0x555ceb0c1720 2024-06-04T05:08:09.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.526+0000 7f62b7c38b80 10 client.18390 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:09.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.526+0000 7f62b7c38b80 10 client.18390 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:08:09.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.526+0000 7f62b7c38b80 20 client.18390 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:09.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.526+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/1695799966 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555ceb0d6910 0x555ceb0d8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.526+0000 7f62b7c38b80 4 client.18390 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:08:09.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.526+0000 7f62b7c38b80 1 -- 172.21.15.70:0/1695799966 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ceb0165a0 con 0x555ceb0d6910 2024-06-04T05:08:09.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.527+0000 7f62a6ffd700 1 --2- 172.21.15.70:0/1695799966 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555ceb0d6910 0x555ceb0d8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.527+0000 7f628aff5700 10 client.18390 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:09.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.527+0000 7f62a6ffd700 1 --2- 172.21.15.70:0/1695799966 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555ceb0d6910 0x555ceb0d8d00 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=307a13677b384771 server_cookie=c110a06350e45c24 in_seq=0 out_seq=0 2024-06-04T05:08:09.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.528+0000 7f628aff5700 1 -- 172.21.15.70:0/1695799966 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555ceb0165a0 con 0x555ceb0d6910 2024-06-04T05:08:09.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.528+0000 7f628aff5700 10 client.18390 handle_command_reply: tid=0 2024-06-04T05:08:09.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.528+0000 7f62b7c38b80 1 client.18390 shutdown 2024-06-04T05:08:09.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.528+0000 7f62b7c38b80 1 -- 172.21.15.70:0/1695799966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 msgr2=0x555ceb0c6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:09.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.528+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/1695799966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c6b10 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f62a0015000 tx=0x7f62a0005800 comp rx=0 tx=0).stop 2024-06-04T05:08:09.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.528+0000 7f62b7c38b80 1 -- 172.21.15.70:0/1695799966 shutdown_connections 2024-06-04T05:08:09.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.528+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/1695799966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c6b10 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.528+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/1695799966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bcbf0 0x555ceb0c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.528+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/1695799966 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555ceb0d6910 0x555ceb0d8d00 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.528+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/1695799966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0bd5b0 0x555ceb0c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.528+0000 7f62b7c38b80 1 -- 172.21.15.70:0/1695799966 >> 172.21.15.70:0/1695799966 conn(0x555ceafe8340 msgr2=0x555ceb0bbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:09.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.528+0000 7f628aff5700 0 client.18390 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:09.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.528+0000 7f62b7c38b80 1 -- 172.21.15.70:0/1695799966 shutdown_connections 2024-06-04T05:08:09.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.528+0000 7f62b7c38b80 1 -- 172.21.15.70:0/1695799966 wait complete. 2024-06-04T05:08:09.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.528+0000 7f62b7c38b80 20 client.18390 trim_cache size 0 max 16384 2024-06-04T05:08:09.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62b7c38b80 1 Processor -- start 2024-06-04T05:08:09.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62b7c38b80 1 -- start start 2024-06-04T05:08:09.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62b7c38b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bcbf0 0x555ceb0b1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62b7c38b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0bd5b0 0x555ceb0c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62b7c38b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62b7c38b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62a81affb0 con 0x555ceb0c1720 2024-06-04T05:08:09.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62b7c38b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62a81b1b50 con 0x555ceb0bd5b0 2024-06-04T05:08:09.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62b7c38b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f62a81b0680 con 0x555ceb0bcbf0 2024-06-04T05:08:09.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62a77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62a77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58550/0 (socket says 172.21.15.70:58550) 2024-06-04T05:08:09.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62a77fe700 1 -- 172.21.15.70:0/807663984 learned_addr learned my addr 172.21.15.70:0/807663984 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:09.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62a67fc700 1 --2- 172.21.15.70:0/807663984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0bd5b0 0x555ceb0c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62a6ffd700 1 --2- 172.21.15.70:0/807663984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bcbf0 0x555ceb0b1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62a77fe700 1 -- 172.21.15.70:0/807663984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bcbf0 msgr2=0x555ceb0b1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:09.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62a77fe700 1 --2- 172.21.15.70:0/807663984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bcbf0 0x555ceb0b1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62a77fe700 1 -- 172.21.15.70:0/807663984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0bd5b0 msgr2=0x555ceb0c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:09.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62a77fe700 1 --2- 172.21.15.70:0/807663984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0bd5b0 0x555ceb0c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62a77fe700 1 -- 172.21.15.70:0/807663984 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ceb0c9ab0 con 0x555ceb0c1720 2024-06-04T05:08:09.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62a6ffd700 1 --2- 172.21.15.70:0/807663984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bcbf0 0x555ceb0b1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:08:09.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.548+0000 7f62a67fc700 1 --2- 172.21.15.70:0/807663984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0bd5b0 0x555ceb0c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:09.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.549+0000 7f62a77fe700 1 --2- 172.21.15.70:0/807663984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c0d10 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f62a000f480 tx=0x7f62a0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23f238dad71ce6d4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:09.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.549+0000 7f628cff9700 1 -- 172.21.15.70:0/807663984 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a00039d0 con 0x555ceb0c1720 2024-06-04T05:08:09.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.549+0000 7f628cff9700 1 -- 172.21.15.70:0/807663984 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f62a0039030 con 0x555ceb0c1720 2024-06-04T05:08:09.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.549+0000 7f628cff9700 1 -- 172.21.15.70:0/807663984 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a000baf0 con 0x555ceb0c1720 2024-06-04T05:08:09.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.549+0000 7f62b7c38b80 1 -- 172.21.15.70:0/807663984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 msgr2=0x555ceb0c0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:09.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.549+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/807663984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c0d10 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f62a000f480 tx=0x7f62a0005fd0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62b7c38b80 1 -- 172.21.15.70:0/807663984 shutdown_connections 2024-06-04T05:08:09.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/807663984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0c1720 0x555ceb0c0d10 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/807663984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bcbf0 0x555ceb0b1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/807663984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0bd5b0 0x555ceb0c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62b7c38b80 1 -- 172.21.15.70:0/807663984 >> 172.21.15.70:0/807663984 conn(0x555ceafe8340 msgr2=0x555ceb0c5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62b7c38b80 1 -- 172.21.15.70:0/807663984 shutdown_connections 2024-06-04T05:08:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62b7c38b80 1 -- 172.21.15.70:0/807663984 wait complete. 2024-06-04T05:08:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62b7c38b80 1 -- start start 2024-06-04T05:08:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62b7c38b80 1 Processor -- start 2024-06-04T05:08:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62b7c38b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0bcbf0 0x555ceb0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62b7c38b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bd5b0 0x555ceb0cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62b7c38b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0c1720 0x555ceb0cb820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62b7c38b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62a81b1450 con 0x555ceb0bcbf0 2024-06-04T05:08:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62b7c38b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62a81b0d50 con 0x555ceb0c1720 2024-06-04T05:08:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62b7c38b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f62a81adf70 con 0x555ceb0bd5b0 2024-06-04T05:08:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62b7c38b80 10 client.0 _ll_register_callbacks cb 0x555ceafa5a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62a6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0bcbf0 0x555ceb0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62a6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0bcbf0 0x555ceb0cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58564/0 (socket says 172.21.15.70:58564) 2024-06-04T05:08:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.550+0000 7f62a6ffd700 1 -- 172.21.15.70:0/691515995 learned_addr learned my addr 172.21.15.70:0/691515995 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.551+0000 7f62a67fc700 1 --2- 172.21.15.70:0/691515995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bd5b0 0x555ceb0cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.551+0000 7f62a77fe700 1 --2- 172.21.15.70:0/691515995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0c1720 0x555ceb0cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.551+0000 7f62a6ffd700 1 -- 172.21.15.70:0/691515995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bd5b0 msgr2=0x555ceb0cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.551+0000 7f62a6ffd700 1 --2- 172.21.15.70:0/691515995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bd5b0 0x555ceb0cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.551+0000 7f62a6ffd700 1 -- 172.21.15.70:0/691515995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0c1720 msgr2=0x555ceb0cb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.551+0000 7f62a6ffd700 1 --2- 172.21.15.70:0/691515995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0c1720 0x555ceb0cb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.551+0000 7f62a6ffd700 1 -- 172.21.15.70:0/691515995 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ceb140660 con 0x555ceb0bcbf0 2024-06-04T05:08:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.551+0000 7f62a77fe700 1 --2- 172.21.15.70:0/691515995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0c1720 0x555ceb0cb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:08:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.551+0000 7f62a6ffd700 1 --2- 172.21.15.70:0/691515995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0bcbf0 0x555ceb0cada0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f6298004b80 tx=0x7f6298008f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc71482510b6674 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.551+0000 7f628aff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.551+0000 7f628aff5700 1 -- 172.21.15.70:0/691515995 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6298025040 con 0x555ceb0bcbf0 2024-06-04T05:08:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.551+0000 7f628aff5700 1 -- 172.21.15.70:0/691515995 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8615+0+0 (secure 0 0 0) 0x7f629802d730 con 0x555ceb0bcbf0 2024-06-04T05:08:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.551+0000 7f62b7c38b80 1 -- 172.21.15.70:0/691515995 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62a0039030 con 0x555ceb0bcbf0 2024-06-04T05:08:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.551+0000 7f628aff5700 1 -- 172.21.15.70:0/691515995 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f629802d930 con 0x555ceb0bcbf0 2024-06-04T05:08:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.551+0000 7f62b7c38b80 10 client.18396 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.551+0000 7f62b7c38b80 1 -- 172.21.15.70:0/691515995 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x555ceb1418f0 con 0x555ceb0bcbf0 2024-06-04T05:08:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.552+0000 7f628aff5700 1 -- 172.21.15.70:0/691515995 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f62a0039030 con 0x555ceb0bcbf0 2024-06-04T05:08:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.552+0000 7f628aff5700 1 -- 172.21.15.70:0/691515995 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f6298015a90 con 0x555ceb0bcbf0 2024-06-04T05:08:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.552+0000 7f62b7c38b80 10 client.18396 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.552+0000 7f62b7c38b80 10 client.18396 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:08:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.552+0000 7f62b7c38b80 20 client.18396 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:09.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.552+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/691515995 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555ceb141fa0 0x555ceb144390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:09.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.552+0000 7f62b7c38b80 4 client.18396 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c91c6358", "release": true, "await": true}] 2024-06-04T05:08:09.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.552+0000 7f62b7c38b80 1 -- 172.21.15.70:0/691515995 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c91c6358", "release": true, "await": true}) -- 0x555ceae8d350 con 0x555ceb141fa0 2024-06-04T05:08:09.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.552+0000 7f62a67fc700 1 --2- 172.21.15.70:0/691515995 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555ceb141fa0 0x555ceb144390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:09.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.553+0000 7f628aff5700 10 client.18396 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:09.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.553+0000 7f62a67fc700 1 --2- 172.21.15.70:0/691515995 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555ceb141fa0 0x555ceb144390 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbd0d8b8694880cc server_cookie=27df9f2df519397f in_seq=0 out_seq=0 2024-06-04T05:08:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.558+0000 7f628aff5700 1 -- 172.21.15.70:0/691515995 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x555ceae8d350 con 0x555ceb141fa0 2024-06-04T05:08:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.558+0000 7f628aff5700 10 client.18396 handle_command_reply: tid=0 2024-06-04T05:08:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.558+0000 7f62b7c38b80 1 client.18396 shutdown 2024-06-04T05:08:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.558+0000 7f62b7c38b80 1 -- 172.21.15.70:0/691515995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0bcbf0 msgr2=0x555ceb0cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.558+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/691515995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0bcbf0 0x555ceb0cada0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f6298004b80 tx=0x7f6298008f10 comp rx=0 tx=0).stop 2024-06-04T05:08:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.558+0000 7f62b7c38b80 1 -- 172.21.15.70:0/691515995 shutdown_connections 2024-06-04T05:08:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.558+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/691515995 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555ceb141fa0 0x555ceb144390 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.558+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/691515995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ceb0c1720 0x555ceb0cb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.558+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/691515995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ceb0bcbf0 0x555ceb0cada0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.558+0000 7f62b7c38b80 1 --2- 172.21.15.70:0/691515995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ceb0bd5b0 0x555ceb0cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.558+0000 7f628aff5700 0 client.18396 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:09.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.558+0000 7f62b7c38b80 1 -- 172.21.15.70:0/691515995 >> 172.21.15.70:0/691515995 conn(0x555ceafe8340 msgr2=0x555ceb0c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:09.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.558+0000 7f62b7c38b80 1 -- 172.21.15.70:0/691515995 shutdown_connections 2024-06-04T05:08:09.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.558+0000 7f62b7c38b80 1 -- 172.21.15.70:0/691515995 wait complete. 2024-06-04T05:08:09.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.559+0000 7f62b7c38b80 20 client.18396 trim_cache size 0 max 16384 2024-06-04T05:08:09.562 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:08:09.562 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 34, 2024-06-04T05:08:09.563 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:08:09.563 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 32, 2024-06-04T05:08:09.563 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:08:09.563 INFO:teuthology.orchestra.run.smithi070.stdout: "c91c6358": { 2024-06-04T05:08:09.563 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 32, 2024-06-04T05:08:09.563 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:08:09.563 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:08:09.563 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:08:09.563 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:08:09.563 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:08:09.563 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:08:09.563 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 130.0, 2024-06-04T05:08:09.563 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:08:09.564 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:08:09.564 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:08:09.564 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:08:09.564 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:08:09.564 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:08:09.564 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:09.564 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:09.564 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:09.564 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:09.564 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:09.564 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:08:09.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.559+0000 7f62ad3a3700 1 -- 172.21.15.70:0/3512361246 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62900b29f0 msgr2=0x7f62900b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:09.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.559+0000 7f62ad3a3700 1 --2- 172.21.15.70:0/3512361246 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62900b29f0 0x7f62900b4eb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f62980060b0 tx=0x7f6298007620 comp rx=0 tx=0).stop 2024-06-04T05:08:09.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.559+0000 7f62ad3a3700 1 -- 172.21.15.70:0/3512361246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62a806bbe0 msgr2=0x7f62a8102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:09.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.559+0000 7f62ad3a3700 1 --2- 172.21.15.70:0/3512361246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62a806bbe0 0x7f62a8102a50 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f629c00d980 tx=0x7f629c00dd40 comp rx=0 tx=0).stop 2024-06-04T05:08:09.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.559+0000 7f62ad3a3700 1 -- 172.21.15.70:0/3512361246 shutdown_connections 2024-06-04T05:08:09.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.559+0000 7f62ad3a3700 1 --2- 172.21.15.70:0/3512361246 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62900b29f0 0x7f62900b4eb0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.559+0000 7f62ad3a3700 1 --2- 172.21.15.70:0/3512361246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a81095b0 0x7f62a8102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.559+0000 7f62ad3a3700 1 --2- 172.21.15.70:0/3512361246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62a806b220 0x7f62a8102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.559+0000 7f62ad3a3700 1 --2- 172.21.15.70:0/3512361246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62a806bbe0 0x7f62a8102a50 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:09.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.559+0000 7f62ad3a3700 1 -- 172.21.15.70:0/3512361246 >> 172.21.15.70:0/3512361246 conn(0x7f62a80793f0 msgr2=0x7f62a8101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:09.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.560+0000 7f62ad3a3700 1 -- 172.21.15.70:0/3512361246 shutdown_connections 2024-06-04T05:08:09.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:09.560+0000 7f62ad3a3700 1 -- 172.21.15.70:0/3512361246 wait complete. 2024-06-04T05:08:09.734 INFO:tasks.workunit.client.0.smithi070.stdout: 1 189064 1.83 MB/sec execute 422 sec latency 9546.748 ms 2024-06-04T05:08:10.277 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c91c6358', total seconds elapsed: 17.6 2024-06-04T05:08:10.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 189618 1.84 MB/sec execute 423 sec latency 20.485 ms 2024-06-04T05:08:11.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:11 smithi112 ceph-mon[36453]: pgmap v340: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 492 KiB/s wr, 98 op/s 2024-06-04T05:08:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.456+0000 7f2172740640 1 -- 172.21.15.70:0/660015842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f216c101bc0 msgr2=0x7f216c102020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.456+0000 7f2172740640 1 --2- 172.21.15.70:0/660015842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f216c101bc0 0x7f216c102020 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f2154009900 tx=0x7f215402a410 comp rx=0 tx=0).stop 2024-06-04T05:08:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.456+0000 7f2172740640 1 -- 172.21.15.70:0/660015842 shutdown_connections 2024-06-04T05:08:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.456+0000 7f2172740640 1 --2- 172.21.15.70:0/660015842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f216c102560 0x7f216c10e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.456+0000 7f2172740640 1 --2- 172.21.15.70:0/660015842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f216c101bc0 0x7f216c102020 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.456+0000 7f2172740640 1 --2- 172.21.15.70:0/660015842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f216c1079f0 0x7f216c107dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.456+0000 7f2172740640 1 -- 172.21.15.70:0/660015842 >> 172.21.15.70:0/660015842 conn(0x7f216c078830 msgr2=0x7f216c078c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.456+0000 7f2172740640 1 -- 172.21.15.70:0/660015842 shutdown_connections 2024-06-04T05:08:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.456+0000 7f2172740640 1 -- 172.21.15.70:0/660015842 wait complete. 2024-06-04T05:08:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.457+0000 7f2172740640 1 Processor -- start 2024-06-04T05:08:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.457+0000 7f2172740640 1 -- start start 2024-06-04T05:08:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.457+0000 7f2172740640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f216c101bc0 0x7f216c1990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.457+0000 7f2172740640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f216c102560 0x7f216c199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.457+0000 7f2172740640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f216c1079f0 0x7f216c19d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.457+0000 7f2172740640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f216c111ad0 con 0x7f216c102560 2024-06-04T05:08:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.457+0000 7f2172740640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f216c111950 con 0x7f216c101bc0 2024-06-04T05:08:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.457+0000 7f2172740640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f216c111c50 con 0x7f216c1079f0 2024-06-04T05:08:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.457+0000 7f2163fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f216c102560 0x7f216c199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.457+0000 7f2163fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f216c102560 0x7f216c199600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58578/0 (socket says 172.21.15.70:58578) 2024-06-04T05:08:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.457+0000 7f2163fff640 1 -- 172.21.15.70:0/2989690072 learned_addr learned my addr 172.21.15.70:0/2989690072 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.457+0000 7f2170cdd640 1 --2- 172.21.15.70:0/2989690072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f216c1079f0 0x7f216c19d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.457+0000 7f216bfff640 1 --2- 172.21.15.70:0/2989690072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f216c101bc0 0x7f216c1990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.458+0000 7f2163fff640 1 -- 172.21.15.70:0/2989690072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f216c1079f0 msgr2=0x7f216c19d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.458+0000 7f2163fff640 1 --2- 172.21.15.70:0/2989690072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f216c1079f0 0x7f216c19d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.458+0000 7f2163fff640 1 -- 172.21.15.70:0/2989690072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f216c101bc0 msgr2=0x7f216c1990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.458+0000 7f2163fff640 1 --2- 172.21.15.70:0/2989690072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f216c101bc0 0x7f216c1990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.458+0000 7f2163fff640 1 -- 172.21.15.70:0/2989690072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f216c19e060 con 0x7f216c102560 2024-06-04T05:08:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.458+0000 7f2163fff640 1 --2- 172.21.15.70:0/2989690072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f216c102560 0x7f216c199600 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f2154009230 tx=0x7f21540319a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.458+0000 7f2169ffb640 1 -- 172.21.15.70:0/2989690072 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215403f030 con 0x7f216c102560 2024-06-04T05:08:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.458+0000 7f2169ffb640 1 -- 172.21.15.70:0/2989690072 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21540043d0 con 0x7f216c102560 2024-06-04T05:08:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.458+0000 7f2172740640 1 -- 172.21.15.70:0/2989690072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f216c19e2f0 con 0x7f216c102560 2024-06-04T05:08:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.458+0000 7f2169ffb640 1 -- 172.21.15.70:0/2989690072 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2154035070 con 0x7f216c102560 2024-06-04T05:08:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.458+0000 7f2172740640 1 -- 172.21.15.70:0/2989690072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f216c1a5be0 con 0x7f216c102560 2024-06-04T05:08:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.460+0000 7f2169ffb640 1 -- 172.21.15.70:0/2989690072 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f2154042030 con 0x7f216c102560 2024-06-04T05:08:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f217545c740 1 Processor -- start 2024-06-04T05:08:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f217545c740 1 -- start start 2024-06-04T05:08:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f217545c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738690820 0x559738690c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f217545c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559738691140 0x55973869b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f217545c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55973869ba00 0x55973869ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f217545c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559738602c30 con 0x559738690820 2024-06-04T05:08:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f217545c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5597385f27d0 con 0x55973869ba00 2024-06-04T05:08:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f217545c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5597385f6810 con 0x559738691140 2024-06-04T05:08:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f2170cdd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738690820 0x559738690c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f216bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559738691140 0x55973869b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f2170cdd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738690820 0x559738690c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58592/0 (socket says 172.21.15.70:58592) 2024-06-04T05:08:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f2170cdd640 1 -- 172.21.15.70:0/3606548003 learned_addr learned my addr 172.21.15.70:0/3606548003 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f2163fff640 1 --2- 172.21.15.70:0/3606548003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55973869ba00 0x55973869ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f2170cdd640 1 -- 172.21.15.70:0/3606548003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559738691140 msgr2=0x55973869b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f2170cdd640 1 --2- 172.21.15.70:0/3606548003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559738691140 0x55973869b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f2169ffb640 1 --2- 172.21.15.70:0/2989690072 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f21580bb2a0 0x7f21580bd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f2170cdd640 1 -- 172.21.15.70:0/3606548003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55973869ba00 msgr2=0x55973869ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f2170cdd640 1 --2- 172.21.15.70:0/3606548003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55973869ba00 0x55973869ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f2170cdd640 1 -- 172.21.15.70:0/3606548003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559738307210 con 0x559738690820 2024-06-04T05:08:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f2163fff640 1 --2- 172.21.15.70:0/3606548003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55973869ba00 0x55973869ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.461+0000 7f216bfff640 1 --2- 172.21.15.70:0/3606548003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559738691140 0x55973869b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.462+0000 7f2169ffb640 1 -- 172.21.15.70:0/2989690072 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8615+0+0 (secure 0 0 0) 0x7f21541028e0 con 0x7f216c102560 2024-06-04T05:08:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.462+0000 7f2170cdd640 1 --2- 172.21.15.70:0/3606548003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738690820 0x559738690c00 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f2164009fd0 tx=0x7f21640118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303040371ecf69a0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.462+0000 7f2170cdd640 1 --2- 172.21.15.70:0/2989690072 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f21580bb2a0 0x7f21580bd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.462+0000 7f21627fc640 1 -- 172.21.15.70:0/3606548003 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216401e070 con 0x559738690820 2024-06-04T05:08:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.462+0000 7f21627fc640 1 -- 172.21.15.70:0/3606548003 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2164003d90 con 0x559738690820 2024-06-04T05:08:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.462+0000 7f21627fc640 1 -- 172.21.15.70:0/3606548003 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216400fc30 con 0x559738690820 2024-06-04T05:08:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.463+0000 7f217545c740 1 -- 172.21.15.70:0/3606548003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738690820 msgr2=0x559738690c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.463+0000 7f2170cdd640 1 --2- 172.21.15.70:0/2989690072 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f21580bb2a0 0x7f21580bd760 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2164003fd0 tx=0x7f216400feb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.463+0000 7f217545c740 1 --2- 172.21.15.70:0/3606548003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738690820 0x559738690c00 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f2164009fd0 tx=0x7f21640118d0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.463+0000 7f217545c740 1 -- 172.21.15.70:0/3606548003 shutdown_connections 2024-06-04T05:08:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.463+0000 7f217545c740 1 --2- 172.21.15.70:0/3606548003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55973869ba00 0x55973869ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.463+0000 7f217545c740 1 --2- 172.21.15.70:0/3606548003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559738691140 0x55973869b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.463+0000 7f217545c740 1 --2- 172.21.15.70:0/3606548003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738690820 0x559738690c00 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.463+0000 7f217545c740 1 -- 172.21.15.70:0/3606548003 >> 172.21.15.70:0/3606548003 conn(0x55973868cd30 msgr2=0x55973868d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.463+0000 7f217545c740 1 -- 172.21.15.70:0/3606548003 shutdown_connections 2024-06-04T05:08:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.463+0000 7f217545c740 1 -- 172.21.15.70:0/3606548003 wait complete. 2024-06-04T05:08:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.463+0000 7f217545c740 1 -- start start 2024-06-04T05:08:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.463+0000 7f217545c740 1 Processor -- start 2024-06-04T05:08:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f217545c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738690820 0x559738698a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f217545c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559738691140 0x559738698f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f217545c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55973869ba00 0x559738699480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f217545c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5597382d8790 con 0x559738690820 2024-06-04T05:08:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f217545c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5597385ae890 con 0x55973869ba00 2024-06-04T05:08:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f217545c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55973869e630 con 0x559738691140 2024-06-04T05:08:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f217545c740 10 client.0 _ll_register_callbacks cb 0x559738306a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f216bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738690820 0x559738698a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f216bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738690820 0x559738698a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58596/0 (socket says 172.21.15.70:58596) 2024-06-04T05:08:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f216bfff640 1 -- 172.21.15.70:0/1088795947 learned_addr learned my addr 172.21.15.70:0/1088795947 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f2170cdd640 1 --2- 172.21.15.70:0/1088795947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55973869ba00 0x559738699480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f2163fff640 1 --2- 172.21.15.70:0/1088795947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559738691140 0x559738698f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f216bfff640 1 -- 172.21.15.70:0/1088795947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559738691140 msgr2=0x559738698f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f216bfff640 1 --2- 172.21.15.70:0/1088795947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559738691140 0x559738698f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f216bfff640 1 -- 172.21.15.70:0/1088795947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55973869ba00 msgr2=0x559738699480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f216bfff640 1 --2- 172.21.15.70:0/1088795947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55973869ba00 0x559738699480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f216bfff640 1 -- 172.21.15.70:0/1088795947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597383d3620 con 0x559738690820 2024-06-04T05:08:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f2170cdd640 1 --2- 172.21.15.70:0/1088795947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55973869ba00 0x559738699480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f216bfff640 1 --2- 172.21.15.70:0/1088795947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738690820 0x559738698a00 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f215c0126e0 tx=0x7f215c012ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7866fed582f6b4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f214b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.464+0000 7f214b7fe640 1 -- 172.21.15.70:0/1088795947 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c004020 con 0x559738690820 2024-06-04T05:08:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.465+0000 7f214b7fe640 1 -- 172.21.15.70:0/1088795947 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8615+0+0 (secure 0 0 0) 0x7f215c0041c0 con 0x559738690820 2024-06-04T05:08:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.465+0000 7f217545c740 1 -- 172.21.15.70:0/1088795947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2164003d90 con 0x559738690820 2024-06-04T05:08:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.465+0000 7f214b7fe640 1 -- 172.21.15.70:0/1088795947 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f215c0043c0 con 0x559738690820 2024-06-04T05:08:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.465+0000 7f217545c740 10 client.18408 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.465+0000 7f217545c740 1 -- 172.21.15.70:0/1088795947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5597386aa440 con 0x559738690820 2024-06-04T05:08:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.465+0000 7f214b7fe640 1 -- 172.21.15.70:0/1088795947 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2164003d90 con 0x559738690820 2024-06-04T05:08:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.465+0000 7f214b7fe640 1 -- 172.21.15.70:0/1088795947 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f215c0037b0 con 0x559738690820 2024-06-04T05:08:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.465+0000 7f217545c740 10 client.18408 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.465+0000 7f217545c740 10 client.18408 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:08:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.465+0000 7f217545c740 20 client.18408 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.465+0000 7f217545c740 1 --2- 172.21.15.70:0/1088795947 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5597386ab950 0x5597386add60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.465+0000 7f217545c740 4 client.18408 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:08:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.465+0000 7f217545c740 1 -- 172.21.15.70:0/1088795947 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559738604b80 con 0x5597386ab950 2024-06-04T05:08:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.465+0000 7f2163fff640 1 --2- 172.21.15.70:0/1088795947 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5597386ab950 0x5597386add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.466+0000 7f214b7fe640 10 client.18408 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.466+0000 7f2163fff640 1 --2- 172.21.15.70:0/1088795947 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5597386ab950 0x5597386add60 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf44ee0d2fc0dbda server_cookie=2396b83369d3ff8b in_seq=0 out_seq=0 2024-06-04T05:08:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.467+0000 7f214b7fe640 1 -- 172.21.15.70:0/1088795947 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559738604b80 con 0x5597386ab950 2024-06-04T05:08:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.467+0000 7f214b7fe640 10 client.18408 handle_command_reply: tid=0 2024-06-04T05:08:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.467+0000 7f217545c740 1 client.18408 shutdown 2024-06-04T05:08:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.467+0000 7f217545c740 1 -- 172.21.15.70:0/1088795947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738690820 msgr2=0x559738698a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.467+0000 7f217545c740 1 --2- 172.21.15.70:0/1088795947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738690820 0x559738698a00 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f215c0126e0 tx=0x7f215c012ae0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.467+0000 7f217545c740 1 -- 172.21.15.70:0/1088795947 shutdown_connections 2024-06-04T05:08:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.467+0000 7f217545c740 1 --2- 172.21.15.70:0/1088795947 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5597386ab950 0x5597386add60 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.467+0000 7f217545c740 1 --2- 172.21.15.70:0/1088795947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55973869ba00 0x559738699480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.467+0000 7f217545c740 1 --2- 172.21.15.70:0/1088795947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559738691140 0x559738698f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.467+0000 7f214b7fe640 0 client.18408 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.467+0000 7f217545c740 1 --2- 172.21.15.70:0/1088795947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738690820 0x559738698a00 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.467+0000 7f217545c740 1 -- 172.21.15.70:0/1088795947 >> 172.21.15.70:0/1088795947 conn(0x55973868cd30 msgr2=0x55973868bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.467+0000 7f217545c740 1 -- 172.21.15.70:0/1088795947 shutdown_connections 2024-06-04T05:08:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.467+0000 7f217545c740 1 -- 172.21.15.70:0/1088795947 wait complete. 2024-06-04T05:08:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.467+0000 7f217545c740 20 client.18408 trim_cache size 0 max 16384 2024-06-04T05:08:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f217545c740 1 Processor -- start 2024-06-04T05:08:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f217545c740 1 -- start start 2024-06-04T05:08:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f217545c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559738690820 0x55973868c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f217545c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559738691140 0x5597386992a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f217545c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55973869ba00 0x5597386928e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f217545c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1ac490 con 0x55973869ba00 2024-06-04T05:08:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f217545c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1ae030 con 0x559738691140 2024-06-04T05:08:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f217545c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1acb60 con 0x559738690820 2024-06-04T05:08:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f2170cdd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55973869ba00 0x5597386928e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f2170cdd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55973869ba00 0x5597386928e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58598/0 (socket says 172.21.15.70:58598) 2024-06-04T05:08:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f2170cdd640 1 -- 172.21.15.70:0/3825802464 learned_addr learned my addr 172.21.15.70:0/3825802464 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f216bfff640 1 --2- 172.21.15.70:0/3825802464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559738690820 0x55973868c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f2163fff640 1 --2- 172.21.15.70:0/3825802464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559738691140 0x5597386992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f2170cdd640 1 -- 172.21.15.70:0/3825802464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559738690820 msgr2=0x55973868c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f2170cdd640 1 --2- 172.21.15.70:0/3825802464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559738690820 0x55973868c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f2170cdd640 1 -- 172.21.15.70:0/3825802464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559738691140 msgr2=0x5597386992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f2170cdd640 1 --2- 172.21.15.70:0/3825802464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559738691140 0x5597386992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f2170cdd640 1 -- 172.21.15.70:0/3825802464 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55973868d670 con 0x55973869ba00 2024-06-04T05:08:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.485+0000 7f216bfff640 1 --2- 172.21.15.70:0/3825802464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559738690820 0x55973868c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.486+0000 7f2170cdd640 1 --2- 172.21.15.70:0/3825802464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55973869ba00 0x5597386928e0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f216400fda0 tx=0x7f2164027000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2886cb48b0fd8953 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.486+0000 7f2161ffb640 1 -- 172.21.15.70:0/3825802464 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2164037070 con 0x55973869ba00 2024-06-04T05:08:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.486+0000 7f2161ffb640 1 -- 172.21.15.70:0/3825802464 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f216402a040 con 0x55973869ba00 2024-06-04T05:08:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.486+0000 7f2161ffb640 1 -- 172.21.15.70:0/3825802464 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216401e040 con 0x55973869ba00 2024-06-04T05:08:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.486+0000 7f217545c740 1 -- 172.21.15.70:0/3825802464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55973869ba00 msgr2=0x5597386928e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.486+0000 7f217545c740 1 --2- 172.21.15.70:0/3825802464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55973869ba00 0x5597386928e0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f216400fda0 tx=0x7f2164027000 comp rx=0 tx=0).stop 2024-06-04T05:08:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.487+0000 7f217545c740 1 -- 172.21.15.70:0/3825802464 shutdown_connections 2024-06-04T05:08:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.487+0000 7f217545c740 1 --2- 172.21.15.70:0/3825802464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559738691140 0x5597386992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.487+0000 7f217545c740 1 --2- 172.21.15.70:0/3825802464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55973869ba00 0x5597386928e0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.487+0000 7f217545c740 1 --2- 172.21.15.70:0/3825802464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559738690820 0x55973868c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.487+0000 7f217545c740 1 -- 172.21.15.70:0/3825802464 >> 172.21.15.70:0/3825802464 conn(0x55973868cd30 msgr2=0x5597386921d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.487+0000 7f217545c740 1 -- 172.21.15.70:0/3825802464 shutdown_connections 2024-06-04T05:08:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.487+0000 7f217545c740 1 -- 172.21.15.70:0/3825802464 wait complete. 2024-06-04T05:08:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.487+0000 7f217545c740 1 -- start start 2024-06-04T05:08:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.487+0000 7f217545c740 1 Processor -- start 2024-06-04T05:08:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.487+0000 7f217545c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559738690820 0x5597386a39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.487+0000 7f217545c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738691140 0x5597386a3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.487+0000 7f217545c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55973869ba00 0x55973869ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.487+0000 7f217545c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1ad930 con 0x559738691140 2024-06-04T05:08:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.487+0000 7f217545c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1ad230 con 0x559738690820 2024-06-04T05:08:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.487+0000 7f217545c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1aa490 con 0x55973869ba00 2024-06-04T05:08:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.487+0000 7f217545c740 10 client.0 _ll_register_callbacks cb 0x7f215c009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f2163fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738691140 0x5597386a3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f2163fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738691140 0x5597386a3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58600/0 (socket says 172.21.15.70:58600) 2024-06-04T05:08:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f2163fff640 1 -- 172.21.15.70:0/951014683 learned_addr learned my addr 172.21.15.70:0/951014683 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f2170cdd640 1 --2- 172.21.15.70:0/951014683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55973869ba00 0x55973869ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f216bfff640 1 --2- 172.21.15.70:0/951014683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559738690820 0x5597386a39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f2163fff640 1 -- 172.21.15.70:0/951014683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55973869ba00 msgr2=0x55973869ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f2163fff640 1 --2- 172.21.15.70:0/951014683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55973869ba00 0x55973869ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f2163fff640 1 -- 172.21.15.70:0/951014683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559738690820 msgr2=0x5597386a39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f2163fff640 1 --2- 172.21.15.70:0/951014683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559738690820 0x5597386a39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f2163fff640 1 -- 172.21.15.70:0/951014683 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55973869f620 con 0x559738691140 2024-06-04T05:08:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f216bfff640 1 --2- 172.21.15.70:0/951014683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559738690820 0x5597386a39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f2163fff640 1 --2- 172.21.15.70:0/951014683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738691140 0x5597386a3f10 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f21540c6e80 tx=0x7f21540ec040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7145ff88100031bf server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f214b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f214b7fe640 1 -- 172.21.15.70:0/951014683 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21540e3060 con 0x559738691140 2024-06-04T05:08:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f214b7fe640 1 -- 172.21.15.70:0/951014683 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8615+0+0 (secure 0 0 0) 0x7f2154003bf0 con 0x559738691140 2024-06-04T05:08:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f217545c740 1 -- 172.21.15.70:0/951014683 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f216402a040 con 0x559738691140 2024-06-04T05:08:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f214b7fe640 1 -- 172.21.15.70:0/951014683 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f2154049040 con 0x559738691140 2024-06-04T05:08:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f217545c740 10 client.18414 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f217545c740 1 -- 172.21.15.70:0/951014683 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5597387160c0 con 0x559738691140 2024-06-04T05:08:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.488+0000 7f214b7fe640 1 -- 172.21.15.70:0/951014683 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f216402a040 con 0x559738691140 2024-06-04T05:08:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.489+0000 7f214b7fe640 1 -- 172.21.15.70:0/951014683 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f21540c9e70 con 0x559738691140 2024-06-04T05:08:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.489+0000 7f217545c740 10 client.18414 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.489+0000 7f217545c740 10 client.18414 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:08:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.489+0000 7f217545c740 20 client.18414 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.489+0000 7f217545c740 1 --2- 172.21.15.70:0/951014683 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559738716820 0x559738718c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.489+0000 7f217545c740 4 client.18414 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:08:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.489+0000 7f217545c740 1 -- 172.21.15.70:0/951014683 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5597385fe360 con 0x559738716820 2024-06-04T05:08:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.489+0000 7f216bfff640 1 --2- 172.21.15.70:0/951014683 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559738716820 0x559738718c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.489+0000 7f214b7fe640 10 client.18414 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.489+0000 7f216bfff640 1 --2- 172.21.15.70:0/951014683 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559738716820 0x559738718c40 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58d9a0b48c263369 server_cookie=f6952fbe1e91d5fd in_seq=0 out_seq=0 2024-06-04T05:08:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:11 smithi098 ceph-mon[35569]: pgmap v340: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 492 KiB/s wr, 98 op/s 2024-06-04T05:08:11.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.514+0000 7f214b7fe640 1 -- 172.21.15.70:0/951014683 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5597385fe360 con 0x559738716820 2024-06-04T05:08:11.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.514+0000 7f214b7fe640 10 client.18414 handle_command_reply: tid=0 2024-06-04T05:08:11.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.514+0000 7f217545c740 1 client.18414 shutdown 2024-06-04T05:08:11.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.514+0000 7f217545c740 1 -- 172.21.15.70:0/951014683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738691140 msgr2=0x5597386a3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:11.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.514+0000 7f217545c740 1 --2- 172.21.15.70:0/951014683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738691140 0x5597386a3f10 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f21540c6e80 tx=0x7f21540ec040 comp rx=0 tx=0).stop 2024-06-04T05:08:11.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.514+0000 7f217545c740 1 -- 172.21.15.70:0/951014683 shutdown_connections 2024-06-04T05:08:11.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.514+0000 7f217545c740 1 --2- 172.21.15.70:0/951014683 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559738716820 0x559738718c40 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.514+0000 7f217545c740 1 --2- 172.21.15.70:0/951014683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55973869ba00 0x55973869ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.514+0000 7f217545c740 1 --2- 172.21.15.70:0/951014683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559738691140 0x5597386a3f10 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.514+0000 7f217545c740 1 --2- 172.21.15.70:0/951014683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559738690820 0x5597386a39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.514+0000 7f214b7fe640 0 client.18414 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:11.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.514+0000 7f217545c740 1 -- 172.21.15.70:0/951014683 >> 172.21.15.70:0/951014683 conn(0x55973868cd30 msgr2=0x559738692a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:11.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.514+0000 7f217545c740 1 -- 172.21.15.70:0/951014683 shutdown_connections 2024-06-04T05:08:11.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.514+0000 7f217545c740 1 -- 172.21.15.70:0/951014683 wait complete. 2024-06-04T05:08:11.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.514+0000 7f217545c740 20 client.18414 trim_cache size 0 max 16384 2024-06-04T05:08:11.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:08:11.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:08:11.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:08:11.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.515+0000 7f2172740640 1 -- 172.21.15.70:0/2989690072 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f21580bb2a0 msgr2=0x7f21580bd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:11.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.515+0000 7f2172740640 1 --2- 172.21.15.70:0/2989690072 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f21580bb2a0 0x7f21580bd760 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2164003fd0 tx=0x7f216400feb0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.515+0000 7f2172740640 1 -- 172.21.15.70:0/2989690072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f216c102560 msgr2=0x7f216c199600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:11.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.515+0000 7f2172740640 1 --2- 172.21.15.70:0/2989690072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f216c102560 0x7f216c199600 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f2154009230 tx=0x7f21540319a0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.515+0000 7f2172740640 1 -- 172.21.15.70:0/2989690072 shutdown_connections 2024-06-04T05:08:11.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.515+0000 7f2172740640 1 --2- 172.21.15.70:0/2989690072 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f21580bb2a0 0x7f21580bd760 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.515+0000 7f2172740640 1 --2- 172.21.15.70:0/2989690072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f216c1079f0 0x7f216c19d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.515+0000 7f2172740640 1 --2- 172.21.15.70:0/2989690072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f216c102560 0x7f216c199600 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.515+0000 7f2172740640 1 --2- 172.21.15.70:0/2989690072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f216c101bc0 0x7f216c1990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:11.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.515+0000 7f2172740640 1 -- 172.21.15.70:0/2989690072 >> 172.21.15.70:0/2989690072 conn(0x7f216c078830 msgr2=0x7f216c105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:11.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.515+0000 7f2172740640 1 -- 172.21.15.70:0/2989690072 shutdown_connections 2024-06-04T05:08:11.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:11.515+0000 7f2172740640 1 -- 172.21.15.70:0/2989690072 wait complete. 2024-06-04T05:08:11.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:11 smithi070 ceph-mon[32631]: pgmap v340: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 492 KiB/s wr, 98 op/s 2024-06-04T05:08:11.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 190211 1.83 MB/sec execute 424 sec latency 26.165 ms 2024-06-04T05:08:12.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 190879 1.84 MB/sec execute 425 sec latency 14.099 ms 2024-06-04T05:08:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:13 smithi112 ceph-mon[36453]: pgmap v341: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 228 KiB/s wr, 32 op/s 2024-06-04T05:08:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:13 smithi112 ceph-mon[36453]: osdmap e90: 12 total, 12 up, 12 in 2024-06-04T05:08:13.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:13 smithi098 ceph-mon[35569]: pgmap v341: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 228 KiB/s wr, 32 op/s 2024-06-04T05:08:13.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:13 smithi098 ceph-mon[35569]: osdmap e90: 12 total, 12 up, 12 in 2024-06-04T05:08:13.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:13 smithi070 ceph-mon[32631]: pgmap v341: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 228 KiB/s wr, 32 op/s 2024-06-04T05:08:13.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:13 smithi070 ceph-mon[32631]: osdmap e90: 12 total, 12 up, 12 in 2024-06-04T05:08:13.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 191363 1.85 MB/sec execute 426 sec latency 15.520 ms 2024-06-04T05:08:14.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 191620 1.85 MB/sec execute 427 sec latency 638.697 ms 2024-06-04T05:08:15.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:15 smithi112 ceph-mon[36453]: pgmap v343: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 273 KiB/s wr, 30 op/s 2024-06-04T05:08:15.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:15 smithi098 ceph-mon[35569]: pgmap v343: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 273 KiB/s wr, 30 op/s 2024-06-04T05:08:15.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:15 smithi070 ceph-mon[32631]: pgmap v343: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 273 KiB/s wr, 30 op/s 2024-06-04T05:08:15.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 191620 1.85 MB/sec execute 428 sec latency 1638.778 ms 2024-06-04T05:08:16.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 191620 1.84 MB/sec execute 429 sec latency 2638.881 ms 2024-06-04T05:08:17.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:17 smithi112 ceph-mon[36453]: pgmap v344: 97 pgs: 97 active+clean; 113 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 1.0 MiB/s wr, 93 op/s 2024-06-04T05:08:17.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:17 smithi098 ceph-mon[35569]: pgmap v344: 97 pgs: 97 active+clean; 113 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 1.0 MiB/s wr, 93 op/s 2024-06-04T05:08:17.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:17 smithi070 ceph-mon[32631]: pgmap v344: 97 pgs: 97 active+clean; 113 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 1.0 MiB/s wr, 93 op/s 2024-06-04T05:08:17.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 191687 1.84 MB/sec execute 430 sec latency 2949.805 ms 2024-06-04T05:08:18.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 191948 1.84 MB/sec execute 431 sec latency 562.648 ms 2024-06-04T05:08:19.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:19 smithi112 ceph-mon[36453]: pgmap v345: 97 pgs: 97 active+clean; 113 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 851 KiB/s wr, 75 op/s 2024-06-04T05:08:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:19 smithi098 ceph-mon[35569]: pgmap v345: 97 pgs: 97 active+clean; 113 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 851 KiB/s wr, 75 op/s 2024-06-04T05:08:19.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:19 smithi070 ceph-mon[32631]: pgmap v345: 97 pgs: 97 active+clean; 113 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 851 KiB/s wr, 75 op/s 2024-06-04T05:08:19.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 192270 1.83 MB/sec execute 432 sec latency 15.709 ms 2024-06-04T05:08:20.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 192801 1.83 MB/sec execute 433 sec latency 16.042 ms 2024-06-04T05:08:21.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:21 smithi112 ceph-mon[36453]: pgmap v346: 97 pgs: 97 active+clean; 113 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 846 KiB/s wr, 74 op/s 2024-06-04T05:08:21.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:21 smithi098 ceph-mon[35569]: pgmap v346: 97 pgs: 97 active+clean; 113 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 846 KiB/s wr, 74 op/s 2024-06-04T05:08:21.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:21 smithi070 ceph-mon[32631]: pgmap v346: 97 pgs: 97 active+clean; 113 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 846 KiB/s wr, 74 op/s 2024-06-04T05:08:21.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 193379 1.84 MB/sec execute 434 sec latency 21.899 ms 2024-06-04T05:08:22.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 193950 1.84 MB/sec execute 435 sec latency 17.658 ms 2024-06-04T05:08:23.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:23 smithi112 ceph-mon[36453]: pgmap v347: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 1.2 MiB/s wr, 111 op/s 2024-06-04T05:08:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:23 smithi098 ceph-mon[35569]: pgmap v347: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 1.2 MiB/s wr, 111 op/s 2024-06-04T05:08:23.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:23 smithi070 ceph-mon[32631]: pgmap v347: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 1.2 MiB/s wr, 111 op/s 2024-06-04T05:08:23.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 194570 1.84 MB/sec execute 436 sec latency 15.449 ms 2024-06-04T05:08:24.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.608+0000 7f13b3272640 1 -- 172.21.15.70:0/976362535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13ac107940 msgr2=0x7f13ac107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:24.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.608+0000 7f13b3272640 1 --2- 172.21.15.70:0/976362535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13ac107940 0x7f13ac107d20 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f13a8009930 tx=0x7f13a802a420 comp rx=0 tx=0).stop 2024-06-04T05:08:24.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.608+0000 7f13b3272640 1 -- 172.21.15.70:0/976362535 shutdown_connections 2024-06-04T05:08:24.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.608+0000 7f13b3272640 1 --2- 172.21.15.70:0/976362535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13ac102340 0x7f13ac10e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.608+0000 7f13b3272640 1 --2- 172.21.15.70:0/976362535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13ac101980 0x7f13ac101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.608+0000 7f13b3272640 1 --2- 172.21.15.70:0/976362535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13ac107940 0x7f13ac107d20 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.608+0000 7f13b3272640 1 -- 172.21.15.70:0/976362535 >> 172.21.15.70:0/976362535 conn(0x7f13ac078ad0 msgr2=0x7f13ac078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:24.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.608+0000 7f13b3272640 1 -- 172.21.15.70:0/976362535 shutdown_connections 2024-06-04T05:08:24.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.608+0000 7f13b3272640 1 -- 172.21.15.70:0/976362535 wait complete. 2024-06-04T05:08:24.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.608+0000 7f13b3272640 1 Processor -- start 2024-06-04T05:08:24.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b3272640 1 -- start start 2024-06-04T05:08:24.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b3272640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13ac101980 0x7f13ac0fe910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b3272640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13ac102340 0x7f13ac0fee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b3272640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13ac107940 0x7f13ac0ff390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b3272640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac111a80 con 0x7f13ac102340 2024-06-04T05:08:24.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b3272640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac111900 con 0x7f13ac107940 2024-06-04T05:08:24.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b3272640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac111c00 con 0x7f13ac101980 2024-06-04T05:08:24.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b100e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13ac101980 0x7f13ac0fe910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b080d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13ac102340 0x7f13ac0fee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b100e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13ac101980 0x7f13ac0fe910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:50010/0 (socket says 172.21.15.70:50010) 2024-06-04T05:08:24.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b100e640 1 -- 172.21.15.70:0/605120494 learned_addr learned my addr 172.21.15.70:0/605120494 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:24.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b080d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13ac102340 0x7f13ac0fee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57396/0 (socket says 172.21.15.70:57396) 2024-06-04T05:08:24.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b180f640 1 --2- 172.21.15.70:0/605120494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13ac107940 0x7f13ac0ff390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b080d640 1 -- 172.21.15.70:0/605120494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13ac101980 msgr2=0x7f13ac0fe910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:24.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b080d640 1 --2- 172.21.15.70:0/605120494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13ac101980 0x7f13ac0fe910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b080d640 1 -- 172.21.15.70:0/605120494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13ac107940 msgr2=0x7f13ac0ff390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:24.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b080d640 1 --2- 172.21.15.70:0/605120494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13ac107940 0x7f13ac0ff390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b080d640 1 -- 172.21.15.70:0/605120494 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ac1a7630 con 0x7f13ac102340 2024-06-04T05:08:24.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b100e640 1 --2- 172.21.15.70:0/605120494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13ac101980 0x7f13ac0fe910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:24.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.609+0000 7f13b080d640 1 --2- 172.21.15.70:0/605120494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13ac102340 0x7f13ac0fee50 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f139c00e9f0 tx=0x7f139c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:24.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.610+0000 7f13a27fc640 1 -- 172.21.15.70:0/605120494 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c00cd80 con 0x7f13ac102340 2024-06-04T05:08:24.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.610+0000 7f13a27fc640 1 -- 172.21.15.70:0/605120494 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f139c0035f0 con 0x7f13ac102340 2024-06-04T05:08:24.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.610+0000 7f13b3272640 1 -- 172.21.15.70:0/605120494 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13ac1a7920 con 0x7f13ac102340 2024-06-04T05:08:24.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.610+0000 7f13a27fc640 1 -- 172.21.15.70:0/605120494 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c010670 con 0x7f13ac102340 2024-06-04T05:08:24.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.610+0000 7f13b3272640 1 -- 172.21.15.70:0/605120494 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13ac1a7e70 con 0x7f13ac102340 2024-06-04T05:08:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.612+0000 7f13a27fc640 1 -- 172.21.15.70:0/605120494 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f139c004000 con 0x7f13ac102340 2024-06-04T05:08:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b5f8d740 1 Processor -- start 2024-06-04T05:08:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b5f8d740 1 -- start start 2024-06-04T05:08:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b5f8d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a62820 0x563850a62c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b5f8d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a63140 0x563850a6d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b5f8d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a6da00 0x563850a6fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b5f8d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5638509d4c30 con 0x563850a62820 2024-06-04T05:08:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b5f8d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5638509c47d0 con 0x563850a6da00 2024-06-04T05:08:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b5f8d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5638509c8810 con 0x563850a63140 2024-06-04T05:08:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13a27fc640 1 --2- 172.21.15.70:0/605120494 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f13980b2a40 0x7f13980b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b100e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a63140 0x563850a6d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b180f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a62820 0x563850a62c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b100e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a63140 0x563850a6d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:50014/0 (socket says 172.21.15.70:50014) 2024-06-04T05:08:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b100e640 1 -- 172.21.15.70:0/1630944826 learned_addr learned my addr 172.21.15.70:0/1630944826 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b180f640 1 --2- 172.21.15.70:0/605120494 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f13980b2a40 0x7f13980b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b080d640 1 --2- 172.21.15.70:0/1630944826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a6da00 0x563850a6fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13a27fc640 1 -- 172.21.15.70:0/605120494 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f139c0e28c0 con 0x7f13ac102340 2024-06-04T05:08:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b100e640 1 -- 172.21.15.70:0/1630944826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a6da00 msgr2=0x563850a6fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b100e640 1 --2- 172.21.15.70:0/1630944826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a6da00 0x563850a6fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b100e640 1 -- 172.21.15.70:0/1630944826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a62820 msgr2=0x563850a62c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.613+0000 7f13b100e640 1 --2- 172.21.15.70:0/1630944826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a62820 0x563850a62c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.614+0000 7f13b100e640 1 -- 172.21.15.70:0/1630944826 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638506d9210 con 0x563850a63140 2024-06-04T05:08:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.614+0000 7f13b180f640 1 --2- 172.21.15.70:0/605120494 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f13980b2a40 0x7f13980b4f00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f13a4005fd0 tx=0x7f13a4009500 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.614+0000 7f13b100e640 1 --2- 172.21.15.70:0/1630944826 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a63140 0x563850a6d4c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f13a8006010 tx=0x7f13a8032e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b62bbc1009e354ad server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.614+0000 7f13977fe640 1 -- 172.21.15.70:0/1630944826 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a8035070 con 0x563850a63140 2024-06-04T05:08:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.614+0000 7f13977fe640 1 -- 172.21.15.70:0/1630944826 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f13a80044c0 con 0x563850a63140 2024-06-04T05:08:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.614+0000 7f13977fe640 1 -- 172.21.15.70:0/1630944826 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a8003d20 con 0x563850a63140 2024-06-04T05:08:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.615+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1630944826 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a63140 msgr2=0x563850a6d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.615+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/1630944826 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a63140 0x563850a6d4c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f13a8006010 tx=0x7f13a8032e30 comp rx=0 tx=0).stop 2024-06-04T05:08:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.615+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1630944826 shutdown_connections 2024-06-04T05:08:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.615+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/1630944826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a6da00 0x563850a6fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.615+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/1630944826 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a63140 0x563850a6d4c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.615+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/1630944826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a62820 0x563850a62c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.615+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1630944826 >> 172.21.15.70:0/1630944826 conn(0x563850a5ed30 msgr2=0x563850a5f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.615+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1630944826 shutdown_connections 2024-06-04T05:08:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.615+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1630944826 wait complete. 2024-06-04T05:08:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.615+0000 7f13b5f8d740 1 -- start start 2024-06-04T05:08:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.615+0000 7f13b5f8d740 1 Processor -- start 2024-06-04T05:08:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b5f8d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a62820 0x563850a6a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b5f8d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a63140 0x563850a6ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b5f8d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a6da00 0x563850a6b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b5f8d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5638506aa790 con 0x563850a63140 2024-06-04T05:08:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b5f8d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563850980890 con 0x563850a6da00 2024-06-04T05:08:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b5f8d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563850a705d0 con 0x563850a62820 2024-06-04T05:08:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b5f8d740 10 client.0 _ll_register_callbacks cb 0x5638506d8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b080d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a63140 0x563850a6ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b100e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a62820 0x563850a6a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b080d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a63140 0x563850a6ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57412/0 (socket says 172.21.15.70:57412) 2024-06-04T05:08:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b080d640 1 -- 172.21.15.70:0/3696655396 learned_addr learned my addr 172.21.15.70:0/3696655396 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b180f640 1 --2- 172.21.15.70:0/3696655396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a6da00 0x563850a6b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b080d640 1 -- 172.21.15.70:0/3696655396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a62820 msgr2=0x563850a6a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b080d640 1 --2- 172.21.15.70:0/3696655396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a62820 0x563850a6a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b080d640 1 -- 172.21.15.70:0/3696655396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a6da00 msgr2=0x563850a6b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b080d640 1 --2- 172.21.15.70:0/3696655396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a6da00 0x563850a6b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b080d640 1 -- 172.21.15.70:0/3696655396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638507a5620 con 0x563850a63140 2024-06-04T05:08:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b180f640 1 --2- 172.21.15.70:0/3696655396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a6da00 0x563850a6b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:08:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b080d640 1 --2- 172.21.15.70:0/3696655396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a63140 0x563850a6ae60 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f139c0a9870 tx=0x7f139c003cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d3364567d350d2c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f1394ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f1394ff9640 1 -- 172.21.15.70:0/3696655396 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c0a9b70 con 0x563850a63140 2024-06-04T05:08:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f1394ff9640 1 -- 172.21.15.70:0/3696655396 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f139c0e0260 con 0x563850a63140 2024-06-04T05:08:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.616+0000 7f13b5f8d740 1 -- 172.21.15.70:0/3696655396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5638509d6b80 con 0x563850a63140 2024-06-04T05:08:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.617+0000 7f1394ff9640 1 -- 172.21.15.70:0/3696655396 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f139c0adf20 con 0x563850a63140 2024-06-04T05:08:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.617+0000 7f13b5f8d740 10 client.18426 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.617+0000 7f13b5f8d740 1 -- 172.21.15.70:0/3696655396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563850a7c270 con 0x563850a63140 2024-06-04T05:08:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.617+0000 7f1394ff9640 1 -- 172.21.15.70:0/3696655396 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x5638509d6b80 con 0x563850a63140 2024-06-04T05:08:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.617+0000 7f1394ff9640 1 -- 172.21.15.70:0/3696655396 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f139c0bc290 con 0x563850a63140 2024-06-04T05:08:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.617+0000 7f13b5f8d740 10 client.18426 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.617+0000 7f13b5f8d740 10 client.18426 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:08:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.617+0000 7f13b5f8d740 20 client.18426 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.617+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/3696655396 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563850a7d7a0 0x563850a7fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.617+0000 7f13b5f8d740 4 client.18426 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:08:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.617+0000 7f13b5f8d740 1 -- 172.21.15.70:0/3696655396 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638509d0360 con 0x563850a7d7a0 2024-06-04T05:08:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.617+0000 7f13b100e640 1 --2- 172.21.15.70:0/3696655396 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563850a7d7a0 0x563850a7fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.618+0000 7f1394ff9640 10 client.18426 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.618+0000 7f13b100e640 1 --2- 172.21.15.70:0/3696655396 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563850a7d7a0 0x563850a7fbd0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba90aaa7f528191f server_cookie=9dc9f12df2ccaeb1 in_seq=0 out_seq=0 2024-06-04T05:08:24.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.619+0000 7f1394ff9640 1 -- 172.21.15.70:0/3696655396 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5638509d0360 con 0x563850a7d7a0 2024-06-04T05:08:24.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.619+0000 7f1394ff9640 10 client.18426 handle_command_reply: tid=0 2024-06-04T05:08:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.619+0000 7f13b5f8d740 1 client.18426 shutdown 2024-06-04T05:08:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.619+0000 7f13b5f8d740 1 -- 172.21.15.70:0/3696655396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a63140 msgr2=0x563850a6ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.619+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/3696655396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a63140 0x563850a6ae60 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f139c0a9870 tx=0x7f139c003cf0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.619+0000 7f13b5f8d740 1 -- 172.21.15.70:0/3696655396 shutdown_connections 2024-06-04T05:08:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.619+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/3696655396 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563850a7d7a0 0x563850a7fbd0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.619+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/3696655396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a6da00 0x563850a6b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.619+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/3696655396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a63140 0x563850a6ae60 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.619+0000 7f1394ff9640 0 client.18426 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.619+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/3696655396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a62820 0x563850a6a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.619+0000 7f13b5f8d740 1 -- 172.21.15.70:0/3696655396 >> 172.21.15.70:0/3696655396 conn(0x563850a5ed30 msgr2=0x563850a64ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.619+0000 7f13b5f8d740 1 -- 172.21.15.70:0/3696655396 shutdown_connections 2024-06-04T05:08:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.619+0000 7f13b5f8d740 1 -- 172.21.15.70:0/3696655396 wait complete. 2024-06-04T05:08:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.619+0000 7f13b5f8d740 20 client.18426 trim_cache size 0 max 16384 2024-06-04T05:08:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b5f8d740 1 Processor -- start 2024-06-04T05:08:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b5f8d740 1 -- start start 2024-06-04T05:08:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b5f8d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a62820 0x563850a651e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b5f8d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a63140 0x563850a6aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b5f8d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a6da00 0x563850a6b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b5f8d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1b0840 con 0x563850a6da00 2024-06-04T05:08:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b5f8d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1b23e0 con 0x563850a63140 2024-06-04T05:08:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b5f8d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1b0f10 con 0x563850a62820 2024-06-04T05:08:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b180f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a6da00 0x563850a6b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b180f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a6da00 0x563850a6b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57426/0 (socket says 172.21.15.70:57426) 2024-06-04T05:08:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b100e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a62820 0x563850a651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b180f640 1 -- 172.21.15.70:0/1060310539 learned_addr learned my addr 172.21.15.70:0/1060310539 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b080d640 1 --2- 172.21.15.70:0/1060310539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a63140 0x563850a6aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b180f640 1 -- 172.21.15.70:0/1060310539 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a62820 msgr2=0x563850a651e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b180f640 1 --2- 172.21.15.70:0/1060310539 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a62820 0x563850a651e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b180f640 1 -- 172.21.15.70:0/1060310539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a63140 msgr2=0x563850a6aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b180f640 1 --2- 172.21.15.70:0/1060310539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a63140 0x563850a6aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b180f640 1 -- 172.21.15.70:0/1060310539 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563850a5eb60 con 0x563850a6da00 2024-06-04T05:08:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b100e640 1 --2- 172.21.15.70:0/1060310539 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a62820 0x563850a651e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.636+0000 7f13b080d640 1 --2- 172.21.15.70:0/1060310539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a63140 0x563850a6aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.637+0000 7f13b180f640 1 --2- 172.21.15.70:0/1060310539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a6da00 0x563850a6b530 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f13a4005ee0 tx=0x7f13a4008000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965018c0efd45084 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.637+0000 7f1396ffd640 1 -- 172.21.15.70:0/1060310539 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a400fdb0 con 0x563850a6da00 2024-06-04T05:08:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.637+0000 7f1396ffd640 1 -- 172.21.15.70:0/1060310539 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f13a4010dc0 con 0x563850a6da00 2024-06-04T05:08:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.637+0000 7f1396ffd640 1 -- 172.21.15.70:0/1060310539 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a4029dd0 con 0x563850a6da00 2024-06-04T05:08:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.637+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1060310539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a6da00 msgr2=0x563850a6b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.637+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/1060310539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a6da00 0x563850a6b530 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f13a4005ee0 tx=0x7f13a4008000 comp rx=0 tx=0).stop 2024-06-04T05:08:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.638+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1060310539 shutdown_connections 2024-06-04T05:08:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.638+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/1060310539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a63140 0x563850a6aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.638+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/1060310539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a6da00 0x563850a6b530 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.638+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/1060310539 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a62820 0x563850a651e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.638+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1060310539 >> 172.21.15.70:0/1060310539 conn(0x563850a5ed30 msgr2=0x563850a6a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.638+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1060310539 shutdown_connections 2024-06-04T05:08:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.638+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1060310539 wait complete. 2024-06-04T05:08:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.638+0000 7f13b5f8d740 1 -- start start 2024-06-04T05:08:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.638+0000 7f13b5f8d740 1 Processor -- start 2024-06-04T05:08:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.638+0000 7f13b5f8d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a62820 0x563850a67940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.638+0000 7f13b5f8d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a63140 0x563850a67e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.638+0000 7f13b5f8d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a6da00 0x563850a683c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.638+0000 7f13b5f8d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1b1ce0 con 0x563850a6da00 2024-06-04T05:08:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.638+0000 7f13b5f8d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1b15e0 con 0x563850a62820 2024-06-04T05:08:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.638+0000 7f13b5f8d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1ae840 con 0x563850a63140 2024-06-04T05:08:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f13b5f8d740 10 client.0 _ll_register_callbacks cb 0x7f139c0a5a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f13b180f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a6da00 0x563850a683c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f13b180f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a6da00 0x563850a683c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57430/0 (socket says 172.21.15.70:57430) 2024-06-04T05:08:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f13b180f640 1 -- 172.21.15.70:0/1915192765 learned_addr learned my addr 172.21.15.70:0/1915192765 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f13b080d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a63140 0x563850a67e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f13b100e640 1 --2- 172.21.15.70:0/1915192765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a62820 0x563850a67940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f13b180f640 1 -- 172.21.15.70:0/1915192765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a63140 msgr2=0x563850a67e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f13b180f640 1 --2- 172.21.15.70:0/1915192765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a63140 0x563850a67e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f13b180f640 1 -- 172.21.15.70:0/1915192765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a62820 msgr2=0x563850a67940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f13b180f640 1 --2- 172.21.15.70:0/1915192765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a62820 0x563850a67940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f13b180f640 1 -- 172.21.15.70:0/1915192765 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563850ae7370 con 0x563850a6da00 2024-06-04T05:08:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f13b080d640 1 --2- 172.21.15.70:0/1915192765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a63140 0x563850a67e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f13b100e640 1 --2- 172.21.15.70:0/1915192765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a62820 0x563850a67940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f13b180f640 1 --2- 172.21.15.70:0/1915192765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a6da00 0x563850a683c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f13a4009270 tx=0x7f13a4005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24a77874cb0d1812 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f1394ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f1394ff9640 1 -- 172.21.15.70:0/1915192765 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a4038060 con 0x563850a6da00 2024-06-04T05:08:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f1394ff9640 1 -- 172.21.15.70:0/1915192765 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f13a4026040 con 0x563850a6da00 2024-06-04T05:08:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1915192765 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13a4010dc0 con 0x563850a6da00 2024-06-04T05:08:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f1394ff9640 1 -- 172.21.15.70:0/1915192765 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f13a4029400 con 0x563850a6da00 2024-06-04T05:08:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.639+0000 7f13b5f8d740 10 client.18432 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.640+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1915192765 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563850ae8660 con 0x563850a6da00 2024-06-04T05:08:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.640+0000 7f1394ff9640 1 -- 172.21.15.70:0/1915192765 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f13a4010dc0 con 0x563850a6da00 2024-06-04T05:08:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.640+0000 7f1394ff9640 1 -- 172.21.15.70:0/1915192765 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f13a401d070 con 0x563850a6da00 2024-06-04T05:08:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.640+0000 7f13b5f8d740 10 client.18432 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.640+0000 7f13b5f8d740 10 client.18432 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:08:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.640+0000 7f13b5f8d740 20 client.18432 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.640+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/1915192765 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563850ae8cd0 0x563850aeb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.640+0000 7f13b5f8d740 4 client.18432 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:08:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.640+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1915192765 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56385094a5f0 con 0x563850ae8cd0 2024-06-04T05:08:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.640+0000 7f13b100e640 1 --2- 172.21.15.70:0/1915192765 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563850ae8cd0 0x563850aeb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.640+0000 7f13b100e640 1 --2- 172.21.15.70:0/1915192765 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563850ae8cd0 0x563850aeb0f0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab31e53204b7933f server_cookie=5fd715a2443fd161 in_seq=0 out_seq=0 2024-06-04T05:08:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.640+0000 7f1394ff9640 10 client.18432 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:24.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.670+0000 7f1394ff9640 1 -- 172.21.15.70:0/1915192765 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56385094a5f0 con 0x563850ae8cd0 2024-06-04T05:08:24.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.670+0000 7f1394ff9640 10 client.18432 handle_command_reply: tid=0 2024-06-04T05:08:24.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.671+0000 7f13b5f8d740 1 client.18432 shutdown 2024-06-04T05:08:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.671+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1915192765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a6da00 msgr2=0x563850a683c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.671+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/1915192765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a6da00 0x563850a683c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f13a4009270 tx=0x7f13a4005a50 comp rx=0 tx=0).stop 2024-06-04T05:08:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.671+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1915192765 shutdown_connections 2024-06-04T05:08:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.671+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/1915192765 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563850ae8cd0 0x563850aeb0f0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.671+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/1915192765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563850a6da00 0x563850a683c0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.671+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/1915192765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563850a63140 0x563850a67e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.671+0000 7f1394ff9640 0 client.18432 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.671+0000 7f13b5f8d740 1 --2- 172.21.15.70:0/1915192765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563850a62820 0x563850a67940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.671+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1915192765 >> 172.21.15.70:0/1915192765 conn(0x563850a5ed30 msgr2=0x563850a6b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.671+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1915192765 shutdown_connections 2024-06-04T05:08:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.671+0000 7f13b5f8d740 1 -- 172.21.15.70:0/1915192765 wait complete. 2024-06-04T05:08:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.671+0000 7f13b5f8d740 20 client.18432 trim_cache size 0 max 16384 2024-06-04T05:08:24.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:08:24.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:08:24.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:08:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.672+0000 7f13b3272640 1 -- 172.21.15.70:0/605120494 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f13980b2a40 msgr2=0x7f13980b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.672+0000 7f13b3272640 1 --2- 172.21.15.70:0/605120494 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f13980b2a40 0x7f13980b4f00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f13a4005fd0 tx=0x7f13a4009500 comp rx=0 tx=0).stop 2024-06-04T05:08:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.672+0000 7f13b3272640 1 -- 172.21.15.70:0/605120494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13ac102340 msgr2=0x7f13ac0fee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.672+0000 7f13b3272640 1 --2- 172.21.15.70:0/605120494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13ac102340 0x7f13ac0fee50 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f139c00e9f0 tx=0x7f139c00eeb0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.672+0000 7f13b3272640 1 -- 172.21.15.70:0/605120494 shutdown_connections 2024-06-04T05:08:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.672+0000 7f13b3272640 1 --2- 172.21.15.70:0/605120494 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f13980b2a40 0x7f13980b4f00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.672+0000 7f13b3272640 1 --2- 172.21.15.70:0/605120494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13ac107940 0x7f13ac0ff390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.672+0000 7f13b3272640 1 --2- 172.21.15.70:0/605120494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13ac102340 0x7f13ac0fee50 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.672+0000 7f13b3272640 1 --2- 172.21.15.70:0/605120494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13ac101980 0x7f13ac0fe910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.672+0000 7f13b3272640 1 -- 172.21.15.70:0/605120494 >> 172.21.15.70:0/605120494 conn(0x7f13ac078ad0 msgr2=0x7f13ac06a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.672+0000 7f13b3272640 1 -- 172.21.15.70:0/605120494 shutdown_connections 2024-06-04T05:08:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:24.672+0000 7f13b3272640 1 -- 172.21.15.70:0/605120494 wait complete. 2024-06-04T05:08:24.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 195108 1.85 MB/sec execute 437 sec latency 19.356 ms 2024-06-04T05:08:25.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:25 smithi098 ceph-mon[35569]: pgmap v348: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-06-04T05:08:25.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:25 smithi070 ceph-mon[32631]: pgmap v348: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-06-04T05:08:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:25 smithi112 ceph-mon[36453]: pgmap v348: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-06-04T05:08:25.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 195165 1.85 MB/sec execute 438 sec latency 936.623 ms 2024-06-04T05:08:26.735 INFO:tasks.workunit.client.0.smithi070.stdout: 1 195165 1.84 MB/sec execute 439 sec latency 1936.723 ms 2024-06-04T05:08:27.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:27 smithi098 ceph-mon[35569]: pgmap v349: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 1.5 MiB/s wr, 141 op/s 2024-06-04T05:08:27.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:27 smithi070 ceph-mon[32631]: pgmap v349: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 1.5 MiB/s wr, 141 op/s 2024-06-04T05:08:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:27 smithi112 ceph-mon[36453]: pgmap v349: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 1.5 MiB/s wr, 141 op/s 2024-06-04T05:08:27.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 195232 1.84 MB/sec execute 440 sec latency 2249.988 ms 2024-06-04T05:08:28.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 195232 1.83 MB/sec execute 441 sec latency 1468.501 ms 2024-06-04T05:08:28.850 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 14.9 2024-06-04T05:08:28.851 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:08:29.107 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:08:29.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:29 smithi098 ceph-mon[35569]: pgmap v350: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-06-04T05:08:29.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:29 smithi070 ceph-mon[32631]: pgmap v350: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-06-04T05:08:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:29 smithi112 ceph-mon[36453]: pgmap v350: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-06-04T05:08:29.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 195232 1.83 MB/sec execute 442 sec latency 2468.539 ms 2024-06-04T05:08:30.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 195232 1.83 MB/sec execute 443 sec latency 3468.584 ms 2024-06-04T05:08:31.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.247+0000 7f7883c67700 1 -- 172.21.15.70:0/96326563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f787c108560 msgr2=0x7f787c06bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:31.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.247+0000 7f7883c67700 1 --2- 172.21.15.70:0/96326563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f787c108560 0x7f787c06bcb0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f7878009990 tx=0x7f7878009ca0 comp rx=0 tx=0).stop 2024-06-04T05:08:31.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.247+0000 7f7883c67700 1 -- 172.21.15.70:0/96326563 shutdown_connections 2024-06-04T05:08:31.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.247+0000 7f7883c67700 1 --2- 172.21.15.70:0/96326563 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f787c105fd0 0x7f787c106450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:31.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.247+0000 7f7883c67700 1 --2- 172.21.15.70:0/96326563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f787c108560 0x7f787c06bcb0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:31.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.247+0000 7f7883c67700 1 --2- 172.21.15.70:0/96326563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787c06c1f0 0x7f787c06c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.247+0000 7f7883c67700 1 -- 172.21.15.70:0/96326563 >> 172.21.15.70:0/96326563 conn(0x7f787c078490 msgr2=0x7f787c0788a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.247+0000 7f7883c67700 1 -- 172.21.15.70:0/96326563 shutdown_connections 2024-06-04T05:08:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.247+0000 7f7883c67700 1 -- 172.21.15.70:0/96326563 wait complete. 2024-06-04T05:08:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.248+0000 7f7883c67700 1 Processor -- start 2024-06-04T05:08:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.248+0000 7f7883c67700 1 -- start start 2024-06-04T05:08:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.248+0000 7f7883c67700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787c06c1f0 0x7f787c1a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.248+0000 7f7883c67700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f787c105fd0 0x7f787c1a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.248+0000 7f7883c67700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f787c108560 0x7f787c1a49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.248+0000 7f7883c67700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f787c119bd0 con 0x7f787c108560 2024-06-04T05:08:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.248+0000 7f7883c67700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f787c119a50 con 0x7f787c06c1f0 2024-06-04T05:08:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.248+0000 7f7883c67700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f787c119d50 con 0x7f787c105fd0 2024-06-04T05:08:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.248+0000 7f7882204700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f787c108560 0x7f787c1a49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.248+0000 7f7882204700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f787c108560 0x7f787c1a49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45364/0 (socket says 172.21.15.70:45364) 2024-06-04T05:08:31.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.248+0000 7f7882204700 1 -- 172.21.15.70:0/344337245 learned_addr learned my addr 172.21.15.70:0/344337245 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:31.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.249+0000 7f7881a03700 1 --2- 172.21.15.70:0/344337245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787c06c1f0 0x7f787c1a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:31.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.249+0000 7f7881202700 1 --2- 172.21.15.70:0/344337245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f787c105fd0 0x7f787c1a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:31.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.249+0000 7f7882204700 1 -- 172.21.15.70:0/344337245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f787c105fd0 msgr2=0x7f787c1a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:31.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.249+0000 7f7882204700 1 --2- 172.21.15.70:0/344337245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f787c105fd0 0x7f787c1a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:31.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.249+0000 7f7882204700 1 -- 172.21.15.70:0/344337245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787c06c1f0 msgr2=0x7f787c1a0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:31.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.249+0000 7f7882204700 1 --2- 172.21.15.70:0/344337245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787c06c1f0 0x7f787c1a0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:31.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.249+0000 7f7882204700 1 -- 172.21.15.70:0/344337245 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f787c1a5100 con 0x7f787c108560 2024-06-04T05:08:31.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.249+0000 7f7882204700 1 --2- 172.21.15.70:0/344337245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f787c108560 0x7f787c1a49c0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f7874007ee0 tx=0x7f787400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:31.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.249+0000 7f7872ffd700 1 -- 172.21.15.70:0/344337245 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787400ddd0 con 0x7f787c108560 2024-06-04T05:08:31.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.249+0000 7f7883c67700 1 -- 172.21.15.70:0/344337245 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f787c1a53f0 con 0x7f787c108560 2024-06-04T05:08:31.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.249+0000 7f7872ffd700 1 -- 172.21.15.70:0/344337245 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7874004020 con 0x7f787c108560 2024-06-04T05:08:31.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.249+0000 7f7872ffd700 1 -- 172.21.15.70:0/344337245 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7874015660 con 0x7f787c108560 2024-06-04T05:08:31.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.249+0000 7f7883c67700 1 -- 172.21.15.70:0/344337245 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f787c1acc70 con 0x7f787c108560 2024-06-04T05:08:31.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.250+0000 7f7883c67700 1 -- 172.21.15.70:0/344337245 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f787c06a230 con 0x7f787c108560 2024-06-04T05:08:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.251+0000 7f7872ffd700 1 -- 172.21.15.70:0/344337245 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f78740041d0 con 0x7f787c108560 2024-06-04T05:08:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.252+0000 7f7872ffd700 1 --2- 172.21.15.70:0/344337245 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f78680b29f0 0x7f78680b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.252+0000 7f7881a03700 1 --2- 172.21.15.70:0/344337245 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f78680b29f0 0x7f78680b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.252+0000 7f7872ffd700 1 -- 172.21.15.70:0/344337245 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f78740e2950 con 0x7f787c108560 2024-06-04T05:08:31.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.253+0000 7f7881a03700 1 --2- 172.21.15.70:0/344337245 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f78680b29f0 0x7f78680b4eb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f787800b2e0 tx=0x7f787801a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:31.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.256+0000 7f7872ffd700 1 -- 172.21.15.70:0/344337245 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7874010070 con 0x7f787c108560 2024-06-04T05:08:31.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:31 smithi070 ceph-mon[32631]: pgmap v351: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-06-04T05:08:31.488 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:31 smithi112 ceph-mon[36453]: pgmap v351: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-06-04T05:08:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:31 smithi098 ceph-mon[35569]: pgmap v351: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-06-04T05:08:31.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.503+0000 7f7883c67700 1 -- 172.21.15.70:0/344337245 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f787c1ace40 con 0x7f787c108560 2024-06-04T05:08:31.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.504+0000 7f7872ffd700 1 -- 172.21.15.70:0/344337245 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11931 (secure 0 0 0) 0x7f78740aa300 con 0x7f787c108560 2024-06-04T05:08:31.506 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:08:31.506 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:07:27.814997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24436,24410,24440,24448,24442]},"id":1}]} 2024-06-04T05:08:31.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.506+0000 7f7883c67700 1 -- 172.21.15.70:0/344337245 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f78680b29f0 msgr2=0x7f78680b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:31.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.506+0000 7f7883c67700 1 --2- 172.21.15.70:0/344337245 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f78680b29f0 0x7f78680b4eb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f787800b2e0 tx=0x7f787801a040 comp rx=0 tx=0).stop 2024-06-04T05:08:31.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.506+0000 7f7883c67700 1 -- 172.21.15.70:0/344337245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f787c108560 msgr2=0x7f787c1a49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:31.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.506+0000 7f7883c67700 1 --2- 172.21.15.70:0/344337245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f787c108560 0x7f787c1a49c0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f7874007ee0 tx=0x7f787400d3b0 comp rx=0 tx=0).stop 2024-06-04T05:08:31.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.506+0000 7f7883c67700 1 -- 172.21.15.70:0/344337245 shutdown_connections 2024-06-04T05:08:31.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.506+0000 7f7883c67700 1 --2- 172.21.15.70:0/344337245 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f78680b29f0 0x7f78680b4eb0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:31.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.506+0000 7f7883c67700 1 --2- 172.21.15.70:0/344337245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f787c108560 0x7f787c1a49c0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:31.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.506+0000 7f7883c67700 1 --2- 172.21.15.70:0/344337245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787c06c1f0 0x7f787c1a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:31.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.506+0000 7f7883c67700 1 --2- 172.21.15.70:0/344337245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f787c105fd0 0x7f787c1a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:31.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.506+0000 7f7883c67700 1 -- 172.21.15.70:0/344337245 >> 172.21.15.70:0/344337245 conn(0x7f787c078490 msgr2=0x7f787c103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:31.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.506+0000 7f7883c67700 1 -- 172.21.15.70:0/344337245 shutdown_connections 2024-06-04T05:08:31.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:31.506+0000 7f7883c67700 1 -- 172.21.15.70:0/344337245 wait complete. 2024-06-04T05:08:31.509 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 34 2024-06-04T05:08:31.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 195232 1.82 MB/sec execute 444 sec latency 4468.665 ms 2024-06-04T05:08:32.303 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 134.9 --await-for 45.0' 2024-06-04T05:08:32.303 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 134.9 --await-for 45.0 2024-06-04T05:08:32.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:32 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/344337245' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:08:32.568 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:08:32.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:32 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/344337245' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:08:32.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:32 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/344337245' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:08:32.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 195421 1.82 MB/sec execute 445 sec latency 4780.599 ms 2024-06-04T05:08:33.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:33 smithi098 ceph-mon[35569]: pgmap v352: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-06-04T05:08:33.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:33 smithi070 ceph-mon[32631]: pgmap v352: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-06-04T05:08:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:33 smithi112 ceph-mon[36453]: pgmap v352: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-06-04T05:08:33.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 195693 1.82 MB/sec execute 446 sec latency 16.302 ms 2024-06-04T05:08:34.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.485+0000 7fab70a71700 1 -- 172.21.15.70:0/257571563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab6c101b30 msgr2=0x7fab6c101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:34.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.485+0000 7fab70a71700 1 --2- 172.21.15.70:0/257571563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab6c101b30 0x7fab6c101fb0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fab60009990 tx=0x7fab60009ca0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.485+0000 7fab70a71700 1 -- 172.21.15.70:0/257571563 shutdown_connections 2024-06-04T05:08:34.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.485+0000 7fab70a71700 1 --2- 172.21.15.70:0/257571563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab6c1024f0 0x7fab6c1103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.485+0000 7fab70a71700 1 --2- 172.21.15.70:0/257571563 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab6c1095b0 0x7fab6c109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.485+0000 7fab70a71700 1 --2- 172.21.15.70:0/257571563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab6c101b30 0x7fab6c101fb0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.485+0000 7fab70a71700 1 -- 172.21.15.70:0/257571563 >> 172.21.15.70:0/257571563 conn(0x7fab6c078ae0 msgr2=0x7fab6c0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:34.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.485+0000 7fab70a71700 1 -- 172.21.15.70:0/257571563 shutdown_connections 2024-06-04T05:08:34.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.485+0000 7fab70a71700 1 -- 172.21.15.70:0/257571563 wait complete. 2024-06-04T05:08:34.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.486+0000 7fab70a71700 1 Processor -- start 2024-06-04T05:08:34.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.486+0000 7fab70a71700 1 -- start start 2024-06-04T05:08:34.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.486+0000 7fab70a71700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab6c101b30 0x7fab6c102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.486+0000 7fab70a71700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab6c1024f0 0x7fab6c103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.486+0000 7fab70a71700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab6c1095b0 0x7fab6c103790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.486+0000 7fab70a71700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c113610 con 0x7fab6c1024f0 2024-06-04T05:08:34.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.486+0000 7fab70a71700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c113490 con 0x7fab6c101b30 2024-06-04T05:08:34.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.486+0000 7fab70a71700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c113790 con 0x7fab6c1095b0 2024-06-04T05:08:34.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.486+0000 7fab69d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab6c1024f0 0x7fab6c103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.486+0000 7fab69d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab6c1024f0 0x7fab6c103250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45378/0 (socket says 172.21.15.70:45378) 2024-06-04T05:08:34.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.486+0000 7fab69d9b700 1 -- 172.21.15.70:0/725167988 learned_addr learned my addr 172.21.15.70:0/725167988 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:34.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.487+0000 7fab6a59c700 1 --2- 172.21.15.70:0/725167988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab6c101b30 0x7fab6c102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.487+0000 7fab6ad9d700 1 --2- 172.21.15.70:0/725167988 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab6c1095b0 0x7fab6c103790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.487+0000 7fab69d9b700 1 -- 172.21.15.70:0/725167988 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab6c1095b0 msgr2=0x7fab6c103790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:34.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.487+0000 7fab69d9b700 1 --2- 172.21.15.70:0/725167988 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab6c1095b0 0x7fab6c103790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.487+0000 7fab69d9b700 1 -- 172.21.15.70:0/725167988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab6c101b30 msgr2=0x7fab6c102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:34.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.487+0000 7fab69d9b700 1 --2- 172.21.15.70:0/725167988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab6c101b30 0x7fab6c102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.487+0000 7fab69d9b700 1 -- 172.21.15.70:0/725167988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab6c1a7f90 con 0x7fab6c1024f0 2024-06-04T05:08:34.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.487+0000 7fab6a59c700 1 --2- 172.21.15.70:0/725167988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab6c101b30 0x7fab6c102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:34.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.487+0000 7fab69d9b700 1 --2- 172.21.15.70:0/725167988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab6c1024f0 0x7fab6c103250 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fab60006010 tx=0x7fab600047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:34.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.487+0000 7fab5b7fe700 1 -- 172.21.15.70:0/725167988 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab60027070 con 0x7fab6c1024f0 2024-06-04T05:08:34.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.487+0000 7fab70a71700 1 -- 172.21.15.70:0/725167988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab6c1a8220 con 0x7fab6c1024f0 2024-06-04T05:08:34.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.487+0000 7fab5b7fe700 1 -- 172.21.15.70:0/725167988 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab60022470 con 0x7fab6c1024f0 2024-06-04T05:08:34.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.487+0000 7fab5b7fe700 1 -- 172.21.15.70:0/725167988 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab60017650 con 0x7fab6c1024f0 2024-06-04T05:08:34.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.487+0000 7fab70a71700 1 -- 172.21.15.70:0/725167988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab6c1a86d0 con 0x7fab6c1024f0 2024-06-04T05:08:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.490+0000 7fab5b7fe700 1 -- 172.21.15.70:0/725167988 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fab60009d10 con 0x7fab6c1024f0 2024-06-04T05:08:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.490+0000 7fab5b7fe700 1 --2- 172.21.15.70:0/725167988 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab540b2b20 0x7fab540b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.491+0000 7fab6a59c700 1 --2- 172.21.15.70:0/725167988 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab540b2b20 0x7fab540b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.491+0000 7fab5b7fe700 1 -- 172.21.15.70:0/725167988 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7fab600e3000 con 0x7fab6c1024f0 2024-06-04T05:08:34.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.491+0000 7fab7b306b80 1 Processor -- start 2024-06-04T05:08:34.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.491+0000 7fab6a59c700 1 --2- 172.21.15.70:0/725167988 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab540b2b20 0x7fab540b4fe0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009fd0 tx=0x7fab5c009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:34.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.491+0000 7fab7b306b80 1 -- start start 2024-06-04T05:08:34.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab7b306b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327aa7b0 0x562e327aab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab7b306b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a5c80 0x562e327a6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab7b306b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a6640 0x562e327b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab7b306b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562e32703f00 con 0x562e327a6640 2024-06-04T05:08:34.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab7b306b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562e326f3aa0 con 0x562e327a5c80 2024-06-04T05:08:34.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab7b306b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562e326f7ae0 con 0x562e327aa7b0 2024-06-04T05:08:34.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab69d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a5c80 0x562e327a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab6a59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327aa7b0 0x562e327aab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab69d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a5c80 0x562e327a6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:43878/0 (socket says 172.21.15.70:43878) 2024-06-04T05:08:34.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab69d9b700 1 -- 172.21.15.70:0/1263194889 learned_addr learned my addr 172.21.15.70:0/1263194889 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:34.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab6ad9d700 1 --2- 172.21.15.70:0/1263194889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a6640 0x562e327b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab69d9b700 1 -- 172.21.15.70:0/1263194889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327aa7b0 msgr2=0x562e327aab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:34.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab69d9b700 1 --2- 172.21.15.70:0/1263194889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327aa7b0 0x562e327aab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab69d9b700 1 -- 172.21.15.70:0/1263194889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a6640 msgr2=0x562e327b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:34.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab69d9b700 1 --2- 172.21.15.70:0/1263194889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a6640 0x562e327b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab69d9b700 1 -- 172.21.15.70:0/1263194889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e3269d140 con 0x562e327a5c80 2024-06-04T05:08:34.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab69d9b700 1 --2- 172.21.15.70:0/1263194889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a5c80 0x562e327a6100 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fab60005b40 tx=0x7fab600aa220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a31efdf0030ccc2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:34.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab58ff9700 1 -- 172.21.15.70:0/1263194889 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab600ae6f0 con 0x562e327a5c80 2024-06-04T05:08:34.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab58ff9700 1 -- 172.21.15.70:0/1263194889 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab600b3110 con 0x562e327a5c80 2024-06-04T05:08:34.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.492+0000 7fab58ff9700 1 -- 172.21.15.70:0/1263194889 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab600b2c80 con 0x562e327a5c80 2024-06-04T05:08:34.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.493+0000 7fab7b306b80 1 -- 172.21.15.70:0/1263194889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a5c80 msgr2=0x562e327a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:34.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.493+0000 7fab7b306b80 1 --2- 172.21.15.70:0/1263194889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a5c80 0x562e327a6100 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fab60005b40 tx=0x7fab600aa220 comp rx=0 tx=0).stop 2024-06-04T05:08:34.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.493+0000 7fab7b306b80 1 -- 172.21.15.70:0/1263194889 shutdown_connections 2024-06-04T05:08:34.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.493+0000 7fab7b306b80 1 --2- 172.21.15.70:0/1263194889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a6640 0x562e327b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.493+0000 7fab7b306b80 1 --2- 172.21.15.70:0/1263194889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327aa7b0 0x562e327aab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.493+0000 7fab7b306b80 1 --2- 172.21.15.70:0/1263194889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a5c80 0x562e327a6100 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.493+0000 7fab7b306b80 1 -- 172.21.15.70:0/1263194889 >> 172.21.15.70:0/1263194889 conn(0x562e326d13d0 msgr2=0x562e327a0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:34.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.493+0000 7fab7b306b80 1 -- 172.21.15.70:0/1263194889 shutdown_connections 2024-06-04T05:08:34.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.494+0000 7fab7b306b80 1 -- 172.21.15.70:0/1263194889 wait complete. 2024-06-04T05:08:34.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.494+0000 7fab7b306b80 1 -- start start 2024-06-04T05:08:34.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.494+0000 7fab7b306b80 1 Processor -- start 2024-06-04T05:08:34.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.494+0000 7fab7b306b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a5c80 0x562e327b2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.494+0000 7fab7b306b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a6640 0x562e327af720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.494+0000 7fab7b306b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327aa7b0 0x562e327afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.494+0000 7fab7b306b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562e325fa130 con 0x562e327a6640 2024-06-04T05:08:34.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.494+0000 7fab7b306b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562e3265bf30 con 0x562e327a5c80 2024-06-04T05:08:34.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.494+0000 7fab7b306b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562e326f3aa0 con 0x562e327aa7b0 2024-06-04T05:08:34.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.494+0000 7fab7b306b80 10 client.0 _ll_register_callbacks cb 0x562e3268eae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:34.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.494+0000 7fab69d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a6640 0x562e327af720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.494+0000 7fab69d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a6640 0x562e327af720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45410/0 (socket says 172.21.15.70:45410) 2024-06-04T05:08:34.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.494+0000 7fab69d9b700 1 -- 172.21.15.70:0/3787912918 learned_addr learned my addr 172.21.15.70:0/3787912918 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:34.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.494+0000 7fab6ad9d700 1 --2- 172.21.15.70:0/3787912918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327aa7b0 0x562e327afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.494+0000 7fab6a59c700 1 --2- 172.21.15.70:0/3787912918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a5c80 0x562e327b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.495+0000 7fab69d9b700 1 -- 172.21.15.70:0/3787912918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327aa7b0 msgr2=0x562e327afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:34.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.495+0000 7fab69d9b700 1 --2- 172.21.15.70:0/3787912918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327aa7b0 0x562e327afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.495+0000 7fab69d9b700 1 -- 172.21.15.70:0/3787912918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a5c80 msgr2=0x562e327b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:34.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.495+0000 7fab69d9b700 1 --2- 172.21.15.70:0/3787912918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a5c80 0x562e327b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.495+0000 7fab69d9b700 1 -- 172.21.15.70:0/3787912918 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e32643a60 con 0x562e327a6640 2024-06-04T05:08:34.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.495+0000 7fab69d9b700 1 --2- 172.21.15.70:0/3787912918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a6640 0x562e327af720 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fab60005f50 tx=0x7fab600e06c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69684e85d856b774 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:34.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.495+0000 7fab51ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:34.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.495+0000 7fab51ffb700 1 -- 172.21.15.70:0/3787912918 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab600ae6f0 con 0x562e327a6640 2024-06-04T05:08:34.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.495+0000 7fab51ffb700 1 -- 172.21.15.70:0/3787912918 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7fab600b2c80 con 0x562e327a6640 2024-06-04T05:08:34.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.495+0000 7fab7b306b80 1 -- 172.21.15.70:0/3787912918 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab600b3110 con 0x562e327a6640 2024-06-04T05:08:34.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.495+0000 7fab51ffb700 1 -- 172.21.15.70:0/3787912918 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fab600aa7b0 con 0x562e327a6640 2024-06-04T05:08:34.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.495+0000 7fab7b306b80 10 client.18450 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:34.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.495+0000 7fab7b306b80 1 -- 172.21.15.70:0/3787912918 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562e326ad8f0 con 0x562e327a6640 2024-06-04T05:08:34.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.495+0000 7fab51ffb700 1 -- 172.21.15.70:0/3787912918 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab600b3110 con 0x562e327a6640 2024-06-04T05:08:34.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.496+0000 7fab51ffb700 1 -- 172.21.15.70:0/3787912918 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7fab600a5cb0 con 0x562e327a6640 2024-06-04T05:08:34.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.496+0000 7fab7b306b80 10 client.18450 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:34.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.496+0000 7fab7b306b80 10 client.18450 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:08:34.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.496+0000 7fab7b306b80 20 client.18450 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:34.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.496+0000 7fab7b306b80 1 --2- 172.21.15.70:0/3787912918 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562e327bf530 0x562e327c1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.496+0000 7fab7b306b80 4 client.18450 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:08:34.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.496+0000 7fab7b306b80 1 -- 172.21.15.70:0/3787912918 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e32705e50 con 0x562e327bf530 2024-06-04T05:08:34.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.496+0000 7fab6a59c700 1 --2- 172.21.15.70:0/3787912918 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562e327bf530 0x562e327c1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.496+0000 7fab51ffb700 10 client.18450 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:34.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.496+0000 7fab6a59c700 1 --2- 172.21.15.70:0/3787912918 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562e327bf530 0x562e327c1920 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d187332ac0527b6 server_cookie=b0de359355a8b9c0 in_seq=0 out_seq=0 2024-06-04T05:08:34.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.497+0000 7fab51ffb700 1 -- 172.21.15.70:0/3787912918 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562e32705e50 con 0x562e327bf530 2024-06-04T05:08:34.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.497+0000 7fab51ffb700 10 client.18450 handle_command_reply: tid=0 2024-06-04T05:08:34.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.497+0000 7fab7b306b80 1 client.18450 shutdown 2024-06-04T05:08:34.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.497+0000 7fab7b306b80 1 -- 172.21.15.70:0/3787912918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a6640 msgr2=0x562e327af720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:34.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.497+0000 7fab7b306b80 1 --2- 172.21.15.70:0/3787912918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a6640 0x562e327af720 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fab60005f50 tx=0x7fab600e06c0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.497+0000 7fab7b306b80 1 -- 172.21.15.70:0/3787912918 shutdown_connections 2024-06-04T05:08:34.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.497+0000 7fab7b306b80 1 --2- 172.21.15.70:0/3787912918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327aa7b0 0x562e327afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.497+0000 7fab7b306b80 1 --2- 172.21.15.70:0/3787912918 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562e327bf530 0x562e327c1920 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.497+0000 7fab7b306b80 1 --2- 172.21.15.70:0/3787912918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a5c80 0x562e327b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.497+0000 7fab7b306b80 1 --2- 172.21.15.70:0/3787912918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a6640 0x562e327af720 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.497+0000 7fab7b306b80 1 -- 172.21.15.70:0/3787912918 >> 172.21.15.70:0/3787912918 conn(0x562e326d13d0 msgr2=0x562e327a4c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:34.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.497+0000 7fab51ffb700 0 client.18450 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:34.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.497+0000 7fab7b306b80 1 -- 172.21.15.70:0/3787912918 shutdown_connections 2024-06-04T05:08:34.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.497+0000 7fab7b306b80 1 -- 172.21.15.70:0/3787912918 wait complete. 2024-06-04T05:08:34.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.497+0000 7fab7b306b80 20 client.18450 trim_cache size 0 max 16384 2024-06-04T05:08:34.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.520+0000 7fab7b306b80 1 Processor -- start 2024-06-04T05:08:34.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.520+0000 7fab7b306b80 1 -- start start 2024-06-04T05:08:34.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.520+0000 7fab7b306b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a5c80 0x562e327bbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.520+0000 7fab7b306b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327a6640 0x562e3279af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.520+0000 7fab7b306b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327aa7b0 0x562e327abad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.520+0000 7fab7b306b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c1afdc0 con 0x562e327aa7b0 2024-06-04T05:08:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.520+0000 7fab7b306b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c1b1960 con 0x562e327a5c80 2024-06-04T05:08:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.520+0000 7fab7b306b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c1b0490 con 0x562e327a6640 2024-06-04T05:08:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.520+0000 7fab6ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327aa7b0 0x562e327abad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.520+0000 7fab6ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327aa7b0 0x562e327abad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45412/0 (socket says 172.21.15.70:45412) 2024-06-04T05:08:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.520+0000 7fab6ad9d700 1 -- 172.21.15.70:0/3832946104 learned_addr learned my addr 172.21.15.70:0/3832946104 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.520+0000 7fab6a59c700 1 --2- 172.21.15.70:0/3832946104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a5c80 0x562e327bbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.520+0000 7fab69d9b700 1 --2- 172.21.15.70:0/3832946104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327a6640 0x562e3279af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.521+0000 7fab6ad9d700 1 -- 172.21.15.70:0/3832946104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327a6640 msgr2=0x562e3279af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.521+0000 7fab6ad9d700 1 --2- 172.21.15.70:0/3832946104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327a6640 0x562e3279af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.521+0000 7fab6ad9d700 1 -- 172.21.15.70:0/3832946104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a5c80 msgr2=0x562e327bbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.521+0000 7fab6ad9d700 1 --2- 172.21.15.70:0/3832946104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a5c80 0x562e327bbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.521+0000 7fab6ad9d700 1 -- 172.21.15.70:0/3832946104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e327a5890 con 0x562e327aa7b0 2024-06-04T05:08:34.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.521+0000 7fab6ad9d700 1 --2- 172.21.15.70:0/3832946104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327aa7b0 0x562e327abad0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fab640118d0 tx=0x7fab64009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d73d588d1c0f2d07 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:34.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.521+0000 7fab53fff700 1 -- 172.21.15.70:0/3832946104 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab64003a00 con 0x562e327aa7b0 2024-06-04T05:08:34.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.521+0000 7fab53fff700 1 -- 172.21.15.70:0/3832946104 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab64003ba0 con 0x562e327aa7b0 2024-06-04T05:08:34.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.521+0000 7fab53fff700 1 -- 172.21.15.70:0/3832946104 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab640177b0 con 0x562e327aa7b0 2024-06-04T05:08:34.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.522+0000 7fab7b306b80 1 -- 172.21.15.70:0/3832946104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327aa7b0 msgr2=0x562e327abad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:34.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.522+0000 7fab7b306b80 1 --2- 172.21.15.70:0/3832946104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327aa7b0 0x562e327abad0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fab640118d0 tx=0x7fab64009920 comp rx=0 tx=0).stop 2024-06-04T05:08:34.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.522+0000 7fab7b306b80 1 -- 172.21.15.70:0/3832946104 shutdown_connections 2024-06-04T05:08:34.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.522+0000 7fab7b306b80 1 --2- 172.21.15.70:0/3832946104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327aa7b0 0x562e327abad0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.522+0000 7fab7b306b80 1 --2- 172.21.15.70:0/3832946104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a5c80 0x562e327bbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.522+0000 7fab7b306b80 1 --2- 172.21.15.70:0/3832946104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327a6640 0x562e3279af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.522+0000 7fab7b306b80 1 -- 172.21.15.70:0/3832946104 >> 172.21.15.70:0/3832946104 conn(0x562e326d13d0 msgr2=0x562e327ab3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:34.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.522+0000 7fab7b306b80 1 -- 172.21.15.70:0/3832946104 shutdown_connections 2024-06-04T05:08:34.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.522+0000 7fab7b306b80 1 -- 172.21.15.70:0/3832946104 wait complete. 2024-06-04T05:08:34.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.522+0000 7fab7b306b80 1 -- start start 2024-06-04T05:08:34.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.522+0000 7fab7b306b80 1 Processor -- start 2024-06-04T05:08:34.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab7b306b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a5c80 0x562e327b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab7b306b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a6640 0x562e327b5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab7b306b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327aa7b0 0x562e327b5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab7b306b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c1b1260 con 0x562e327a5c80 2024-06-04T05:08:34.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab7b306b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c1b0b60 con 0x562e327a6640 2024-06-04T05:08:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab7b306b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c1add80 con 0x562e327aa7b0 2024-06-04T05:08:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab7b306b80 10 client.0 _ll_register_callbacks cb 0x562e3268eae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab6a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a5c80 0x562e327b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab6a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a5c80 0x562e327b4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45416/0 (socket says 172.21.15.70:45416) 2024-06-04T05:08:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab6a59c700 1 -- 172.21.15.70:0/1673817878 learned_addr learned my addr 172.21.15.70:0/1673817878 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab6ad9d700 1 --2- 172.21.15.70:0/1673817878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327aa7b0 0x562e327b5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab69d9b700 1 --2- 172.21.15.70:0/1673817878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a6640 0x562e327b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab6a59c700 1 -- 172.21.15.70:0/1673817878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327aa7b0 msgr2=0x562e327b5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab6a59c700 1 --2- 172.21.15.70:0/1673817878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327aa7b0 0x562e327b5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab6a59c700 1 -- 172.21.15.70:0/1673817878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a6640 msgr2=0x562e327b5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab6a59c700 1 --2- 172.21.15.70:0/1673817878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a6640 0x562e327b5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab6a59c700 1 -- 172.21.15.70:0/1673817878 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e3282a060 con 0x562e327a5c80 2024-06-04T05:08:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab6ad9d700 1 --2- 172.21.15.70:0/1673817878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327aa7b0 0x562e327b5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab6a59c700 1 --2- 172.21.15.70:0/1673817878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a5c80 0x562e327b4ad0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x562e327a2610 tx=0x7fab5c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76215471b3da805f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.523+0000 7fab51ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.524+0000 7fab51ffb700 1 -- 172.21.15.70:0/1673817878 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab5c02e030 con 0x562e327a5c80 2024-06-04T05:08:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.524+0000 7fab51ffb700 1 -- 172.21.15.70:0/1673817878 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7fab5c003e00 con 0x562e327a5c80 2024-06-04T05:08:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.524+0000 7fab7b306b80 1 -- 172.21.15.70:0/1673817878 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab64003ba0 con 0x562e327a5c80 2024-06-04T05:08:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.524+0000 7fab51ffb700 1 -- 172.21.15.70:0/1673817878 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fab5c02a620 con 0x562e327a5c80 2024-06-04T05:08:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.524+0000 7fab7b306b80 10 client.18456 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.524+0000 7fab7b306b80 1 -- 172.21.15.70:0/1673817878 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562e3282b3a0 con 0x562e327a5c80 2024-06-04T05:08:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.524+0000 7fab51ffb700 1 -- 172.21.15.70:0/1673817878 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab5c02ac50 con 0x562e327a5c80 2024-06-04T05:08:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.525+0000 7fab51ffb700 1 -- 172.21.15.70:0/1673817878 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7fab5c033c50 con 0x562e327a5c80 2024-06-04T05:08:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.525+0000 7fab7b306b80 10 client.18456 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.525+0000 7fab7b306b80 10 client.18456 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:08:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.525+0000 7fab7b306b80 20 client.18456 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.525+0000 7fab7b306b80 1 --2- 172.21.15.70:0/1673817878 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562e3282bb50 0x562e3282df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.525+0000 7fab7b306b80 4 client.18456 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 134.9, "await_for": 45.0}] 2024-06-04T05:08:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.525+0000 7fab7b306b80 1 -- 172.21.15.70:0/1673817878 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 134.9, "await_for": 45.0}) -- 0x7fab64003ba0 con 0x562e3282bb50 2024-06-04T05:08:34.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.525+0000 7fab69d9b700 1 --2- 172.21.15.70:0/1673817878 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562e3282bb50 0x562e3282df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:34.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.526+0000 7fab51ffb700 10 client.18456 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:34.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.526+0000 7fab69d9b700 1 --2- 172.21.15.70:0/1673817878 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562e3282bb50 0x562e3282df70 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=298f8876f6530554 server_cookie=9e7ed96139ea28ce in_seq=0 out_seq=0 2024-06-04T05:08:34.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.578+0000 7fab51ffb700 1 -- 172.21.15.70:0/1673817878 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x7fab64003ba0 con 0x562e3282bb50 2024-06-04T05:08:34.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.578+0000 7fab51ffb700 10 client.18456 handle_command_reply: tid=0 2024-06-04T05:08:34.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.578+0000 7fab7b306b80 1 client.18456 shutdown 2024-06-04T05:08:34.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.578+0000 7fab7b306b80 1 -- 172.21.15.70:0/1673817878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a5c80 msgr2=0x562e327b4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:34.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.578+0000 7fab7b306b80 1 --2- 172.21.15.70:0/1673817878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a5c80 0x562e327b4ad0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x562e327a2610 tx=0x7fab5c0047c0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.578+0000 7fab7b306b80 1 -- 172.21.15.70:0/1673817878 shutdown_connections 2024-06-04T05:08:34.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.578+0000 7fab7b306b80 1 --2- 172.21.15.70:0/1673817878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e327aa7b0 0x562e327b5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.578+0000 7fab7b306b80 1 --2- 172.21.15.70:0/1673817878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e327a5c80 0x562e327b4ad0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.578+0000 7fab7b306b80 1 --2- 172.21.15.70:0/1673817878 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562e3282bb50 0x562e3282df70 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.578+0000 7fab7b306b80 1 --2- 172.21.15.70:0/1673817878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e327a6640 0x562e327b5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.578+0000 7fab51ffb700 0 client.18456 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:34.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.578+0000 7fab7b306b80 1 -- 172.21.15.70:0/1673817878 >> 172.21.15.70:0/1673817878 conn(0x562e326d13d0 msgr2=0x562e327b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:34.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.578+0000 7fab7b306b80 1 -- 172.21.15.70:0/1673817878 shutdown_connections 2024-06-04T05:08:34.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.579+0000 7fab7b306b80 1 -- 172.21.15.70:0/1673817878 wait complete. 2024-06-04T05:08:34.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.579+0000 7fab7b306b80 20 client.18456 trim_cache size 0 max 16384 2024-06-04T05:08:34.582 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:08:34.582 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 34, 2024-06-04T05:08:34.582 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:08:34.582 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 34, 2024-06-04T05:08:34.582 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:08:34.582 INFO:teuthology.orchestra.run.smithi070.stdout: "af20084": { 2024-06-04T05:08:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 34, 2024-06-04T05:08:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:08:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:08:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:08:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:08:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:08:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:08:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 134.9, 2024-06-04T05:08:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:08:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:08:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:08:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:08:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:08:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:08:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:34.584 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:08:34.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.579+0000 7fab70a71700 1 -- 172.21.15.70:0/725167988 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab540b2b20 msgr2=0x7fab540b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:34.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.579+0000 7fab70a71700 1 --2- 172.21.15.70:0/725167988 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab540b2b20 0x7fab540b4fe0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009fd0 tx=0x7fab5c009260 comp rx=0 tx=0).stop 2024-06-04T05:08:34.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.579+0000 7fab70a71700 1 -- 172.21.15.70:0/725167988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab6c1024f0 msgr2=0x7fab6c103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:34.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.579+0000 7fab70a71700 1 --2- 172.21.15.70:0/725167988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab6c1024f0 0x7fab6c103250 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fab60006010 tx=0x7fab600047b0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.579+0000 7fab70a71700 1 -- 172.21.15.70:0/725167988 shutdown_connections 2024-06-04T05:08:34.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.579+0000 7fab70a71700 1 --2- 172.21.15.70:0/725167988 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab540b2b20 0x7fab540b4fe0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.579+0000 7fab70a71700 1 --2- 172.21.15.70:0/725167988 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab6c1095b0 0x7fab6c103790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.579+0000 7fab70a71700 1 --2- 172.21.15.70:0/725167988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab6c101b30 0x7fab6c102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.579+0000 7fab70a71700 1 --2- 172.21.15.70:0/725167988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab6c1024f0 0x7fab6c103250 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:34.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.579+0000 7fab70a71700 1 -- 172.21.15.70:0/725167988 >> 172.21.15.70:0/725167988 conn(0x7fab6c078ae0 msgr2=0x7fab6c100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:34.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.579+0000 7fab70a71700 1 -- 172.21.15.70:0/725167988 shutdown_connections 2024-06-04T05:08:34.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:34.579+0000 7fab70a71700 1 -- 172.21.15.70:0/725167988 wait complete. 2024-06-04T05:08:34.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.82 MB/sec execute 447 sec latency 152.754 ms 2024-06-04T05:08:35.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:35 smithi098 ceph-mon[35569]: pgmap v353: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 594 KiB/s wr, 60 op/s 2024-06-04T05:08:35.534 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:08:35.563 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:35 smithi070 ceph-mon[32631]: pgmap v353: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 594 KiB/s wr, 60 op/s 2024-06-04T05:08:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:35 smithi112 ceph-mon[36453]: pgmap v353: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 594 KiB/s wr, 60 op/s 2024-06-04T05:08:35.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.81 MB/sec execute 448 sec latency 1152.792 ms 2024-06-04T05:08:36.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.81 MB/sec execute 449 sec latency 2152.820 ms 2024-06-04T05:08:36.742 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'af20084', quiesce took 7.9 seconds. Will release after: 7.0 2024-06-04T05:08:36.833 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:08:37.234 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:37 smithi070 ceph-mon[32631]: pgmap v354: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 931 KiB/s wr, 104 op/s 2024-06-04T05:08:37.234 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:37 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:08:37.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:37 smithi098 ceph-mon[35569]: pgmap v354: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 931 KiB/s wr, 104 op/s 2024-06-04T05:08:37.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:37 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:08:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:37 smithi112 ceph-mon[36453]: pgmap v354: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 931 KiB/s wr, 104 op/s 2024-06-04T05:08:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:37 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:08:37.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.80 MB/sec execute 450 sec latency 3152.871 ms 2024-06-04T05:08:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.760+0000 7f3bff578640 1 -- 172.21.15.70:0/995305741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bf8101980 msgr2=0x7f3bf8101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.760+0000 7f3bff578640 1 --2- 172.21.15.70:0/995305741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bf8101980 0x7f3bf8101e00 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f3bec009cd0 tx=0x7f3bec02a8e0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.760+0000 7f3bff578640 1 -- 172.21.15.70:0/995305741 shutdown_connections 2024-06-04T05:08:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.760+0000 7f3bff578640 1 --2- 172.21.15.70:0/995305741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bf8102340 0x7f3bf810e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.761+0000 7f3bff578640 1 --2- 172.21.15.70:0/995305741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bf8101980 0x7f3bf8101e00 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.761+0000 7f3bff578640 1 --2- 172.21.15.70:0/995305741 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bf8107940 0x7f3bf8107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.761+0000 7f3bff578640 1 -- 172.21.15.70:0/995305741 >> 172.21.15.70:0/995305741 conn(0x7f3bf8078ad0 msgr2=0x7f3bf8078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.761+0000 7f3bff578640 1 -- 172.21.15.70:0/995305741 shutdown_connections 2024-06-04T05:08:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.761+0000 7f3bff578640 1 -- 172.21.15.70:0/995305741 wait complete. 2024-06-04T05:08:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.761+0000 7f3bff578640 1 Processor -- start 2024-06-04T05:08:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.761+0000 7f3bff578640 1 -- start start 2024-06-04T05:08:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.761+0000 7f3bff578640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bf8101980 0x7f3bf8199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.761+0000 7f3bff578640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bf8102340 0x7f3bf819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.761+0000 7f3bff578640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bf8107940 0x7f3bf81a5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.761+0000 7f3bff578640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf8111ab0 con 0x7f3bf8102340 2024-06-04T05:08:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.761+0000 7f3bff578640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf8111930 con 0x7f3bf8107940 2024-06-04T05:08:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.761+0000 7f3bff578640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf8111c30 con 0x7f3bf8101980 2024-06-04T05:08:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.761+0000 7f3bfcb13640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bf8102340 0x7f3bf819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.762+0000 7f3bfcb13640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bf8102340 0x7f3bf819a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46090/0 (socket says 172.21.15.70:46090) 2024-06-04T05:08:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.762+0000 7f3bfcb13640 1 -- 172.21.15.70:0/618479819 learned_addr learned my addr 172.21.15.70:0/618479819 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.762+0000 7f3bfdb15640 1 --2- 172.21.15.70:0/618479819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bf8107940 0x7f3bf81a5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.762+0000 7f3bfd314640 1 --2- 172.21.15.70:0/618479819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bf8101980 0x7f3bf8199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.762+0000 7f3bfcb13640 1 -- 172.21.15.70:0/618479819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bf8101980 msgr2=0x7f3bf8199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.762+0000 7f3bfcb13640 1 --2- 172.21.15.70:0/618479819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bf8101980 0x7f3bf8199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.762+0000 7f3bfcb13640 1 -- 172.21.15.70:0/618479819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bf8107940 msgr2=0x7f3bf81a5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.762+0000 7f3bfcb13640 1 --2- 172.21.15.70:0/618479819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bf8107940 0x7f3bf81a5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.762+0000 7f3bfcb13640 1 -- 172.21.15.70:0/618479819 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf81a5550 con 0x7f3bf8102340 2024-06-04T05:08:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.762+0000 7f3bfd314640 1 --2- 172.21.15.70:0/618479819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bf8101980 0x7f3bf8199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:08:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.762+0000 7f3bfcb13640 1 --2- 172.21.15.70:0/618479819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bf8102340 0x7f3bf819a4d0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0098d0 tx=0x7f3bec003b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.762+0000 7f3bea7fc640 1 -- 172.21.15.70:0/618479819 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec004040 con 0x7f3bf8102340 2024-06-04T05:08:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.762+0000 7f3bea7fc640 1 -- 172.21.15.70:0/618479819 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3bec004670 con 0x7f3bf8102340 2024-06-04T05:08:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.762+0000 7f3bff578640 1 -- 172.21.15.70:0/618479819 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bf81a5720 con 0x7f3bf8102340 2024-06-04T05:08:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.762+0000 7f3bea7fc640 1 -- 172.21.15.70:0/618479819 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec030da0 con 0x7f3bf8102340 2024-06-04T05:08:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.762+0000 7f3bff578640 1 -- 172.21.15.70:0/618479819 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bf81a5be0 con 0x7f3bf8102340 2024-06-04T05:08:37.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.765+0000 7f3c0236c740 1 Processor -- start 2024-06-04T05:08:37.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.765+0000 7f3c0236c740 1 -- start start 2024-06-04T05:08:37.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.765+0000 7f3c0236c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f744820 0x55bb5f744c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3c0236c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f745140 0x55bb5f74f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3c0236c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f74fa00 0x55bb5f751df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3c0236c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bb5f6b6c30 con 0x55bb5f74fa00 2024-06-04T05:08:37.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3c0236c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bb5f6a67d0 con 0x55bb5f745140 2024-06-04T05:08:37.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3c0236c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bb5f6aa810 con 0x55bb5f744820 2024-06-04T05:08:37.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3bea7fc640 1 -- 172.21.15.70:0/618479819 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3bec0305a0 con 0x7f3bf8102340 2024-06-04T05:08:37.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3bfdb15640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f744820 0x55bb5f744c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3bfcb13640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f74fa00 0x55bb5f751df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3bfdb15640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f744820 0x55bb5f744c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51764/0 (socket says 172.21.15.70:51764) 2024-06-04T05:08:37.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3bfdb15640 1 -- 172.21.15.70:0/456508305 learned_addr learned my addr 172.21.15.70:0/456508305 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:37.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3bfd314640 1 --2- 172.21.15.70:0/456508305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f745140 0x55bb5f74f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3bfdb15640 1 -- 172.21.15.70:0/456508305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f745140 msgr2=0x55bb5f74f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:37.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3bfdb15640 1 --2- 172.21.15.70:0/456508305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f745140 0x55bb5f74f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3bfdb15640 1 -- 172.21.15.70:0/456508305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f74fa00 msgr2=0x55bb5f751df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3bfdb15640 1 --2- 172.21.15.70:0/456508305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f74fa00 0x55bb5f751df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3bfdb15640 1 -- 172.21.15.70:0/456508305 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb5f3bb210 con 0x55bb5f744820 2024-06-04T05:08:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3bfcb13640 1 --2- 172.21.15.70:0/456508305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f74fa00 0x55bb5f751df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3bfdb15640 1 --2- 172.21.15.70:0/456508305 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f744820 0x55bb5f744c00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400b6b0 tx=0x7f3bf400bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=103c16a470f50b5b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3be37fe640 1 -- 172.21.15.70:0/456508305 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4003bd0 con 0x55bb5f744820 2024-06-04T05:08:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3bea7fc640 1 --2- 172.21.15.70:0/618479819 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3be40b2a10 0x7f3be40b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3be37fe640 1 -- 172.21.15.70:0/456508305 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3bf4015070 con 0x55bb5f744820 2024-06-04T05:08:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.766+0000 7f3be37fe640 1 -- 172.21.15.70:0/456508305 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf40116a0 con 0x55bb5f744820 2024-06-04T05:08:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.767+0000 7f3bfd314640 1 --2- 172.21.15.70:0/618479819 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3be40b2a10 0x7f3be40b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.767+0000 7f3bea7fc640 1 -- 172.21.15.70:0/618479819 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f3bec02e030 con 0x7f3bf8102340 2024-06-04T05:08:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.767+0000 7f3bfd314640 1 --2- 172.21.15.70:0/618479819 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3be40b2a10 0x7f3be40b4ed0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55bb5f745dc0 tx=0x7f3bf000b370 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.767+0000 7f3c0236c740 1 -- 172.21.15.70:0/456508305 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f744820 msgr2=0x55bb5f744c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.767+0000 7f3c0236c740 1 --2- 172.21.15.70:0/456508305 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f744820 0x55bb5f744c00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400b6b0 tx=0x7f3bf400bb70 comp rx=0 tx=0).stop 2024-06-04T05:08:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.767+0000 7f3c0236c740 1 -- 172.21.15.70:0/456508305 shutdown_connections 2024-06-04T05:08:37.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.767+0000 7f3c0236c740 1 --2- 172.21.15.70:0/456508305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f745140 0x55bb5f74f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.767+0000 7f3c0236c740 1 --2- 172.21.15.70:0/456508305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f74fa00 0x55bb5f751df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.767+0000 7f3c0236c740 1 --2- 172.21.15.70:0/456508305 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f744820 0x55bb5f744c00 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.767+0000 7f3c0236c740 1 -- 172.21.15.70:0/456508305 >> 172.21.15.70:0/456508305 conn(0x55bb5f740d30 msgr2=0x55bb5f741110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:37.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.767+0000 7f3c0236c740 1 -- 172.21.15.70:0/456508305 shutdown_connections 2024-06-04T05:08:37.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.767+0000 7f3c0236c740 1 -- 172.21.15.70:0/456508305 wait complete. 2024-06-04T05:08:37.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3c0236c740 1 -- start start 2024-06-04T05:08:37.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3c0236c740 1 Processor -- start 2024-06-04T05:08:37.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3c0236c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f744820 0x55bb5f74ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3c0236c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f745140 0x55bb5f74cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3c0236c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f74fa00 0x55bb5f74d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3c0236c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bb5f38c790 con 0x55bb5f74fa00 2024-06-04T05:08:37.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3c0236c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bb5f662890 con 0x55bb5f744820 2024-06-04T05:08:37.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3c0236c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bb5f752630 con 0x55bb5f745140 2024-06-04T05:08:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3c0236c740 10 client.0 _ll_register_callbacks cb 0x55bb5f3baa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3bfdb15640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f74fa00 0x55bb5f74d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3bfdb15640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f74fa00 0x55bb5f74d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46112/0 (socket says 172.21.15.70:46112) 2024-06-04T05:08:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3bfdb15640 1 -- 172.21.15.70:0/111283973 learned_addr learned my addr 172.21.15.70:0/111283973 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3bfcb13640 1 --2- 172.21.15.70:0/111283973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f745140 0x55bb5f74cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3bfd314640 1 --2- 172.21.15.70:0/111283973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f744820 0x55bb5f74ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3bfdb15640 1 -- 172.21.15.70:0/111283973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f745140 msgr2=0x55bb5f74cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3bfdb15640 1 --2- 172.21.15.70:0/111283973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f745140 0x55bb5f74cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3bfdb15640 1 -- 172.21.15.70:0/111283973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f744820 msgr2=0x55bb5f74ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3bfdb15640 1 --2- 172.21.15.70:0/111283973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f744820 0x55bb5f74ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.768+0000 7f3bfdb15640 1 -- 172.21.15.70:0/111283973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb5f487620 con 0x55bb5f74fa00 2024-06-04T05:08:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.769+0000 7f3bfdb15640 1 --2- 172.21.15.70:0/111283973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f74fa00 0x55bb5f74d4b0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4000c00 tx=0x7f3bf401abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1a6366767e573d5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.769+0000 7f3be0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.769+0000 7f3be0ff9640 1 -- 172.21.15.70:0/111283973 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf402a060 con 0x55bb5f74fa00 2024-06-04T05:08:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.769+0000 7f3be0ff9640 1 -- 172.21.15.70:0/111283973 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f3bf4003de0 con 0x55bb5f74fa00 2024-06-04T05:08:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.769+0000 7f3c0236c740 1 -- 172.21.15.70:0/111283973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bf4015070 con 0x55bb5f74fa00 2024-06-04T05:08:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.769+0000 7f3be0ff9640 1 -- 172.21.15.70:0/111283973 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f3bf401a470 con 0x55bb5f74fa00 2024-06-04T05:08:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.769+0000 7f3c0236c740 10 client.18468 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.769+0000 7f3c0236c740 1 -- 172.21.15.70:0/111283973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55bb5f75e410 con 0x55bb5f74fa00 2024-06-04T05:08:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.769+0000 7f3be0ff9640 1 -- 172.21.15.70:0/111283973 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3bf4015070 con 0x55bb5f74fa00 2024-06-04T05:08:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.769+0000 7f3be0ff9640 1 -- 172.21.15.70:0/111283973 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f3bf4011ad0 con 0x55bb5f74fa00 2024-06-04T05:08:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.769+0000 7f3c0236c740 10 client.18468 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.769+0000 7f3c0236c740 10 client.18468 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:08:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.769+0000 7f3c0236c740 20 client.18468 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.769+0000 7f3c0236c740 1 --2- 172.21.15.70:0/111283973 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bb5f75f870 0x55bb5f761c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.769+0000 7f3c0236c740 4 client.18468 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:08:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.769+0000 7f3c0236c740 1 -- 172.21.15.70:0/111283973 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb5f6b8b80 con 0x55bb5f75f870 2024-06-04T05:08:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.770+0000 7f3bfd314640 1 --2- 172.21.15.70:0/111283973 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bb5f75f870 0x55bb5f761c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.770+0000 7f3be0ff9640 10 client.18468 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.770+0000 7f3bfd314640 1 --2- 172.21.15.70:0/111283973 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bb5f75f870 0x55bb5f761c60 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=499b9d2c686a8677 server_cookie=528bb508da074adf in_seq=0 out_seq=0 2024-06-04T05:08:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.771+0000 7f3be0ff9640 1 -- 172.21.15.70:0/111283973 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bb5f6b8b80 con 0x55bb5f75f870 2024-06-04T05:08:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.771+0000 7f3be0ff9640 10 client.18468 handle_command_reply: tid=0 2024-06-04T05:08:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.771+0000 7f3c0236c740 1 client.18468 shutdown 2024-06-04T05:08:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.771+0000 7f3c0236c740 1 -- 172.21.15.70:0/111283973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f74fa00 msgr2=0x55bb5f74d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.771+0000 7f3c0236c740 1 --2- 172.21.15.70:0/111283973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f74fa00 0x55bb5f74d4b0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4000c00 tx=0x7f3bf401abe0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.771+0000 7f3c0236c740 1 -- 172.21.15.70:0/111283973 shutdown_connections 2024-06-04T05:08:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.771+0000 7f3c0236c740 1 --2- 172.21.15.70:0/111283973 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bb5f75f870 0x55bb5f761c60 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.771+0000 7f3be0ff9640 0 client.18468 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.771+0000 7f3c0236c740 1 --2- 172.21.15.70:0/111283973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f74fa00 0x55bb5f74d4b0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.771+0000 7f3c0236c740 1 --2- 172.21.15.70:0/111283973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f745140 0x55bb5f74cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.771+0000 7f3c0236c740 1 --2- 172.21.15.70:0/111283973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f744820 0x55bb5f74ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.771+0000 7f3c0236c740 1 -- 172.21.15.70:0/111283973 >> 172.21.15.70:0/111283973 conn(0x55bb5f740d30 msgr2=0x55bb5f73ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.771+0000 7f3c0236c740 1 -- 172.21.15.70:0/111283973 shutdown_connections 2024-06-04T05:08:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.771+0000 7f3c0236c740 1 -- 172.21.15.70:0/111283973 wait complete. 2024-06-04T05:08:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.771+0000 7f3c0236c740 20 client.18468 trim_cache size 0 max 16384 2024-06-04T05:08:37.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3c0236c740 1 Processor -- start 2024-06-04T05:08:37.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3c0236c740 1 -- start start 2024-06-04T05:08:37.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3c0236c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f744820 0x55bb5f767260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3c0236c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f745140 0x55bb5f7677a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3c0236c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f74fa00 0x55bb5f749cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3c0236c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf81ae690 con 0x55bb5f744820 2024-06-04T05:08:37.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3c0236c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf81b0230 con 0x55bb5f745140 2024-06-04T05:08:37.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3c0236c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf81aed60 con 0x55bb5f74fa00 2024-06-04T05:08:37.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3bfd314640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f744820 0x55bb5f767260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3bfd314640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f744820 0x55bb5f767260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46114/0 (socket says 172.21.15.70:46114) 2024-06-04T05:08:37.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3bfd314640 1 -- 172.21.15.70:0/1541367475 learned_addr learned my addr 172.21.15.70:0/1541367475 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:37.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3bfcb13640 1 --2- 172.21.15.70:0/1541367475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f745140 0x55bb5f7677a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3bfdb15640 1 --2- 172.21.15.70:0/1541367475 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f74fa00 0x55bb5f749cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3bfd314640 1 -- 172.21.15.70:0/1541367475 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f74fa00 msgr2=0x55bb5f749cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:37.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3bfd314640 1 --2- 172.21.15.70:0/1541367475 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f74fa00 0x55bb5f749cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3bfd314640 1 -- 172.21.15.70:0/1541367475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f745140 msgr2=0x55bb5f7677a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:37.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3bfd314640 1 --2- 172.21.15.70:0/1541367475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f745140 0x55bb5f7677a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3bfd314640 1 -- 172.21.15.70:0/1541367475 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb5f745bf0 con 0x55bb5f744820 2024-06-04T05:08:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3bfd314640 1 --2- 172.21.15.70:0/1541367475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f744820 0x55bb5f767260 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00058c0 tx=0x7f3bf0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7d5a61817dbc36 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3be2ffd640 1 -- 172.21.15.70:0/1541367475 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0031440 con 0x55bb5f744820 2024-06-04T05:08:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3be2ffd640 1 -- 172.21.15.70:0/1541367475 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3bf0031a70 con 0x55bb5f744820 2024-06-04T05:08:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.787+0000 7f3be2ffd640 1 -- 172.21.15.70:0/1541367475 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0027a70 con 0x55bb5f744820 2024-06-04T05:08:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.788+0000 7f3c0236c740 1 -- 172.21.15.70:0/1541367475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f744820 msgr2=0x55bb5f767260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.788+0000 7f3c0236c740 1 --2- 172.21.15.70:0/1541367475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f744820 0x55bb5f767260 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00058c0 tx=0x7f3bf0013000 comp rx=0 tx=0).stop 2024-06-04T05:08:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.788+0000 7f3c0236c740 1 -- 172.21.15.70:0/1541367475 shutdown_connections 2024-06-04T05:08:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.788+0000 7f3c0236c740 1 --2- 172.21.15.70:0/1541367475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f745140 0x55bb5f7677a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.788+0000 7f3c0236c740 1 --2- 172.21.15.70:0/1541367475 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f74fa00 0x55bb5f749cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.788+0000 7f3c0236c740 1 --2- 172.21.15.70:0/1541367475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f744820 0x55bb5f767260 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.788+0000 7f3c0236c740 1 -- 172.21.15.70:0/1541367475 >> 172.21.15.70:0/1541367475 conn(0x55bb5f740d30 msgr2=0x55bb5f74d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.788+0000 7f3c0236c740 1 -- 172.21.15.70:0/1541367475 shutdown_connections 2024-06-04T05:08:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.788+0000 7f3c0236c740 1 -- 172.21.15.70:0/1541367475 wait complete. 2024-06-04T05:08:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.788+0000 7f3c0236c740 1 -- start start 2024-06-04T05:08:37.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.788+0000 7f3c0236c740 1 Processor -- start 2024-06-04T05:08:37.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3c0236c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f744820 0x55bb5f758030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3c0236c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f745140 0x55bb5f7497b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3c0236c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f74fa00 0x55bb5f749cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3c0236c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf81afb30 con 0x55bb5f745140 2024-06-04T05:08:37.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3c0236c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf81af430 con 0x55bb5f744820 2024-06-04T05:08:37.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3c0236c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf81ac690 con 0x55bb5f74fa00 2024-06-04T05:08:37.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3c0236c740 10 client.0 _ll_register_callbacks cb 0x7f3bec109420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:37.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3bfcb13640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f745140 0x55bb5f7497b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3bfcb13640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f745140 0x55bb5f7497b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46126/0 (socket says 172.21.15.70:46126) 2024-06-04T05:08:37.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3bfcb13640 1 -- 172.21.15.70:0/1392088914 learned_addr learned my addr 172.21.15.70:0/1392088914 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:37.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3bfdb15640 1 --2- 172.21.15.70:0/1392088914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f74fa00 0x55bb5f749cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3bfd314640 1 --2- 172.21.15.70:0/1392088914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f744820 0x55bb5f758030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3bfcb13640 1 -- 172.21.15.70:0/1392088914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f74fa00 msgr2=0x55bb5f749cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3bfcb13640 1 --2- 172.21.15.70:0/1392088914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f74fa00 0x55bb5f749cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3bfcb13640 1 -- 172.21.15.70:0/1392088914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f744820 msgr2=0x55bb5f758030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3bfcb13640 1 --2- 172.21.15.70:0/1392088914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f744820 0x55bb5f758030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3bfcb13640 1 -- 172.21.15.70:0/1392088914 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb5f74a4f0 con 0x55bb5f745140 2024-06-04T05:08:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3bfcb13640 1 --2- 172.21.15.70:0/1392088914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f745140 0x55bb5f7497b0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f3bec101520 tx=0x7f3bec1094d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee5f6805b9ecd0e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.789+0000 7f3be0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.790+0000 7f3be0ff9640 1 -- 172.21.15.70:0/1392088914 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec0ca510 con 0x55bb5f745140 2024-06-04T05:08:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.790+0000 7f3be0ff9640 1 -- 172.21.15.70:0/1392088914 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f3bec0d37a0 con 0x55bb5f745140 2024-06-04T05:08:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.790+0000 7f3c0236c740 1 -- 172.21.15.70:0/1392088914 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bf0031a70 con 0x55bb5f745140 2024-06-04T05:08:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.790+0000 7f3be0ff9640 1 -- 172.21.15.70:0/1392088914 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f3bec0c6350 con 0x55bb5f745140 2024-06-04T05:08:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.790+0000 7f3c0236c740 10 client.18474 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.790+0000 7f3c0236c740 1 -- 172.21.15.70:0/1392088914 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55bb5f7ca1d0 con 0x55bb5f745140 2024-06-04T05:08:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.790+0000 7f3be0ff9640 1 -- 172.21.15.70:0/1392088914 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3bf0031a70 con 0x55bb5f745140 2024-06-04T05:08:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.790+0000 7f3be0ff9640 1 -- 172.21.15.70:0/1392088914 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f3bec0d3040 con 0x55bb5f745140 2024-06-04T05:08:37.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.790+0000 7f3c0236c740 10 client.18474 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:37.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.790+0000 7f3c0236c740 10 client.18474 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:08:37.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.790+0000 7f3c0236c740 20 client.18474 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:37.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.790+0000 7f3c0236c740 1 --2- 172.21.15.70:0/1392088914 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bb5f75f870 0x55bb5f7cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:37.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.790+0000 7f3c0236c740 4 client.18474 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:08:37.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.790+0000 7f3c0236c740 1 -- 172.21.15.70:0/1392088914 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bb5f6b2360 con 0x55bb5f75f870 2024-06-04T05:08:37.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.790+0000 7f3bfd314640 1 --2- 172.21.15.70:0/1392088914 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bb5f75f870 0x55bb5f7cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:37.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.791+0000 7f3bfd314640 1 --2- 172.21.15.70:0/1392088914 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bb5f75f870 0x55bb5f7cc9c0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e72804264bd88cb server_cookie=df4c15497ae63948 in_seq=0 out_seq=0 2024-06-04T05:08:37.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.791+0000 7f3be0ff9640 10 client.18474 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.815+0000 7f3be0ff9640 1 -- 172.21.15.70:0/1392088914 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55bb5f6b2360 con 0x55bb5f75f870 2024-06-04T05:08:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.815+0000 7f3be0ff9640 10 client.18474 handle_command_reply: tid=0 2024-06-04T05:08:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.815+0000 7f3c0236c740 1 client.18474 shutdown 2024-06-04T05:08:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.815+0000 7f3c0236c740 1 -- 172.21.15.70:0/1392088914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f745140 msgr2=0x55bb5f7497b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.815+0000 7f3c0236c740 1 --2- 172.21.15.70:0/1392088914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f745140 0x55bb5f7497b0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f3bec101520 tx=0x7f3bec1094d0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.815+0000 7f3c0236c740 1 -- 172.21.15.70:0/1392088914 shutdown_connections 2024-06-04T05:08:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.815+0000 7f3c0236c740 1 --2- 172.21.15.70:0/1392088914 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bb5f75f870 0x55bb5f7cc9c0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.815+0000 7f3c0236c740 1 --2- 172.21.15.70:0/1392088914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb5f74fa00 0x55bb5f749cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.815+0000 7f3c0236c740 1 --2- 172.21.15.70:0/1392088914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb5f745140 0x55bb5f7497b0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.815+0000 7f3be0ff9640 0 client.18474 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.815+0000 7f3c0236c740 1 --2- 172.21.15.70:0/1392088914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb5f744820 0x55bb5f758030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.815+0000 7f3c0236c740 1 -- 172.21.15.70:0/1392088914 >> 172.21.15.70:0/1392088914 conn(0x55bb5f740d30 msgr2=0x55bb5f74c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.815+0000 7f3c0236c740 1 -- 172.21.15.70:0/1392088914 shutdown_connections 2024-06-04T05:08:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.815+0000 7f3c0236c740 1 -- 172.21.15.70:0/1392088914 wait complete. 2024-06-04T05:08:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.815+0000 7f3c0236c740 20 client.18474 trim_cache size 0 max 16384 2024-06-04T05:08:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:08:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:08:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:08:37.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.816+0000 7f3bff578640 1 -- 172.21.15.70:0/618479819 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3be40b2a10 msgr2=0x7f3be40b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:37.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.816+0000 7f3bff578640 1 --2- 172.21.15.70:0/618479819 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3be40b2a10 0x7f3be40b4ed0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55bb5f745dc0 tx=0x7f3bf000b370 comp rx=0 tx=0).stop 2024-06-04T05:08:37.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.816+0000 7f3bff578640 1 -- 172.21.15.70:0/618479819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bf8102340 msgr2=0x7f3bf819a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:37.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.816+0000 7f3bff578640 1 --2- 172.21.15.70:0/618479819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bf8102340 0x7f3bf819a4d0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0098d0 tx=0x7f3bec003b80 comp rx=0 tx=0).stop 2024-06-04T05:08:37.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.816+0000 7f3bff578640 1 -- 172.21.15.70:0/618479819 shutdown_connections 2024-06-04T05:08:37.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.816+0000 7f3bff578640 1 --2- 172.21.15.70:0/618479819 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3be40b2a10 0x7f3be40b4ed0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.816+0000 7f3bff578640 1 --2- 172.21.15.70:0/618479819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bf8107940 0x7f3bf81a5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.816+0000 7f3bff578640 1 --2- 172.21.15.70:0/618479819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bf8102340 0x7f3bf819a4d0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.816+0000 7f3bff578640 1 --2- 172.21.15.70:0/618479819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bf8101980 0x7f3bf8199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:37.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.816+0000 7f3bff578640 1 -- 172.21.15.70:0/618479819 >> 172.21.15.70:0/618479819 conn(0x7f3bf8078ad0 msgr2=0x7f3bf81045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:37.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.816+0000 7f3bff578640 1 -- 172.21.15.70:0/618479819 shutdown_connections 2024-06-04T05:08:37.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:37.816+0000 7f3bff578640 1 -- 172.21.15.70:0/618479819 wait complete. 2024-06-04T05:08:38.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:38 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:08:38.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:38 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:08:38.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:38 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:08:38.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:38 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:08:38.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:38 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:08:38.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:38 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:08:38.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.587+0000 7f7c6502a700 1 -- 172.21.15.70:0/332893923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c60108920 msgr2=0x7f7c601104a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:38.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.587+0000 7f7c6502a700 1 --2- 172.21.15.70:0/332893923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c60108920 0x7f7c601104a0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800b0d0 tx=0x7f7c5800b3e0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.587+0000 7f7c6502a700 1 -- 172.21.15.70:0/332893923 shutdown_connections 2024-06-04T05:08:38.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.587+0000 7f7c6502a700 1 --2- 172.21.15.70:0/332893923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c60108920 0x7f7c601104a0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.587+0000 7f7c6502a700 1 --2- 172.21.15.70:0/332893923 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c60103850 0x7f7c60103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.587+0000 7f7c6502a700 1 --2- 172.21.15.70:0/332893923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c60104200 0x7f7c601081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.587+0000 7f7c6502a700 1 -- 172.21.15.70:0/332893923 >> 172.21.15.70:0/332893923 conn(0x7f7c6007a5d0 msgr2=0x7f7c6007a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:38.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.587+0000 7f7c6502a700 1 -- 172.21.15.70:0/332893923 shutdown_connections 2024-06-04T05:08:38.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.587+0000 7f7c6502a700 1 -- 172.21.15.70:0/332893923 wait complete. 2024-06-04T05:08:38.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.588+0000 7f7c6502a700 1 Processor -- start 2024-06-04T05:08:38.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.588+0000 7f7c6502a700 1 -- start start 2024-06-04T05:08:38.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.588+0000 7f7c6502a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c60103850 0x7f7c6019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.588+0000 7f7c6502a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c60104200 0x7f7c6019fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.588+0000 7f7c6502a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c60108920 0x7f7c60199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.588+0000 7f7c6502a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7c601135c0 con 0x7f7c60104200 2024-06-04T05:08:38.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.588+0000 7f7c6502a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c60113440 con 0x7f7c60108920 2024-06-04T05:08:38.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.588+0000 7f7c6502a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7c60113740 con 0x7f7c60103850 2024-06-04T05:08:38.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.589+0000 7f7c5e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c60104200 0x7f7c6019fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.589+0000 7f7c5e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c60104200 0x7f7c6019fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46152/0 (socket says 172.21.15.70:46152) 2024-06-04T05:08:38.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.589+0000 7f7c5e59c700 1 -- 172.21.15.70:0/883800438 learned_addr learned my addr 172.21.15.70:0/883800438 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:38.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.589+0000 7f7c5ed9d700 1 --2- 172.21.15.70:0/883800438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c60103850 0x7f7c6019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.589+0000 7f7c5f59e700 1 --2- 172.21.15.70:0/883800438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c60108920 0x7f7c60199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.589+0000 7f7c5e59c700 1 -- 172.21.15.70:0/883800438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c60103850 msgr2=0x7f7c6019f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:38.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.589+0000 7f7c5e59c700 1 --2- 172.21.15.70:0/883800438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c60103850 0x7f7c6019f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.589+0000 7f7c5e59c700 1 -- 172.21.15.70:0/883800438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c60108920 msgr2=0x7f7c60199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:38.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.589+0000 7f7c5e59c700 1 --2- 172.21.15.70:0/883800438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c60108920 0x7f7c60199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.589+0000 7f7c5e59c700 1 -- 172.21.15.70:0/883800438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c6019a060 con 0x7f7c60104200 2024-06-04T05:08:38.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.589+0000 7f7c5f59e700 1 --2- 172.21.15.70:0/883800438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c60108920 0x7f7c60199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:08:38.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.589+0000 7f7c5e59c700 1 --2- 172.21.15.70:0/883800438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c60104200 0x7f7c6019fbf0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f7c5400d950 tx=0x7f7c5400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:38.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.589+0000 7f7c47fff700 1 -- 172.21.15.70:0/883800438 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c54016070 con 0x7f7c60104200 2024-06-04T05:08:38.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.589+0000 7f7c47fff700 1 -- 172.21.15.70:0/883800438 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c54009dd0 con 0x7f7c60104200 2024-06-04T05:08:38.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.589+0000 7f7c6502a700 1 -- 172.21.15.70:0/883800438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c6019a350 con 0x7f7c60104200 2024-06-04T05:08:38.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.589+0000 7f7c47fff700 1 -- 172.21.15.70:0/883800438 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5400f3c0 con 0x7f7c60104200 2024-06-04T05:08:38.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.589+0000 7f7c6502a700 1 -- 172.21.15.70:0/883800438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c601a6510 con 0x7f7c60104200 2024-06-04T05:08:38.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.592+0000 7f7c47fff700 1 -- 172.21.15.70:0/883800438 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7c54019070 con 0x7f7c60104200 2024-06-04T05:08:38.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.592+0000 7f7c47fff700 1 --2- 172.21.15.70:0/883800438 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7c480b29f0 0x7f7c480b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.593+0000 7f7c5ed9d700 1 --2- 172.21.15.70:0/883800438 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7c480b29f0 0x7f7c480b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.593+0000 7f7c47fff700 1 -- 172.21.15.70:0/883800438 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f7c540e22c0 con 0x7f7c60104200 2024-06-04T05:08:38.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.593+0000 7f7c6f8bfb80 1 Processor -- start 2024-06-04T05:08:38.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.593+0000 7f7c6f8bfb80 1 -- start start 2024-06-04T05:08:38.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.593+0000 7f7c5ed9d700 1 --2- 172.21.15.70:0/883800438 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7c480b29f0 0x7f7c480b4eb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7c50005950 tx=0x7f7c5000a300 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:38.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.593+0000 7f7c6f8bfb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60c0320 0x557ed60c0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.593+0000 7f7c6f8bfb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60ba8a0 0x557ed60bad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.593+0000 7f7c6f8bfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60bb260 0x557ed60c7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.593+0000 7f7c6f8bfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557ed6018d70 con 0x557ed60bb260 2024-06-04T05:08:38.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.593+0000 7f7c6f8bfb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557ed6008910 con 0x557ed60ba8a0 2024-06-04T05:08:38.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.593+0000 7f7c6f8bfb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557ed600c950 con 0x557ed60c0320 2024-06-04T05:08:38.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.593+0000 7f7c5f59e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60bb260 0x557ed60c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.593+0000 7f7c5f59e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60bb260 0x557ed60c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46164/0 (socket says 172.21.15.70:46164) 2024-06-04T05:08:38.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.593+0000 7f7c5f59e700 1 -- 172.21.15.70:0/682745771 learned_addr learned my addr 172.21.15.70:0/682745771 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:38.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.594+0000 7f7c5ed9d700 1 --2- 172.21.15.70:0/682745771 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60c0320 0x557ed60c0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.594+0000 7f7c5e59c700 1 --2- 172.21.15.70:0/682745771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60ba8a0 0x557ed60bad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.594+0000 7f7c5f59e700 1 -- 172.21.15.70:0/682745771 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60c0320 msgr2=0x557ed60c0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:38.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.594+0000 7f7c5f59e700 1 --2- 172.21.15.70:0/682745771 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60c0320 0x557ed60c0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.594+0000 7f7c5f59e700 1 -- 172.21.15.70:0/682745771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60ba8a0 msgr2=0x557ed60bad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:38.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.594+0000 7f7c5f59e700 1 --2- 172.21.15.70:0/682745771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60ba8a0 0x557ed60bad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.594+0000 7f7c5f59e700 1 -- 172.21.15.70:0/682745771 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ed5fb1fb0 con 0x557ed60bb260 2024-06-04T05:08:38.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.594+0000 7f7c5ed9d700 1 --2- 172.21.15.70:0/682745771 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60c0320 0x557ed60c0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:38.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.594+0000 7f7c5f59e700 1 --2- 172.21.15.70:0/682745771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60bb260 0x557ed60c7080 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f7c58009dc0 tx=0x7f7c58009df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4994115409ac505a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:38.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.594+0000 7f7c457fa700 1 -- 172.21.15.70:0/682745771 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c58021020 con 0x557ed60bb260 2024-06-04T05:08:38.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.594+0000 7f7c457fa700 1 -- 172.21.15.70:0/682745771 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c580093d0 con 0x557ed60bb260 2024-06-04T05:08:38.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.594+0000 7f7c457fa700 1 -- 172.21.15.70:0/682745771 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c580074e0 con 0x557ed60bb260 2024-06-04T05:08:38.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.595+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/682745771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60bb260 msgr2=0x557ed60c7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:38.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.595+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/682745771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60bb260 0x557ed60c7080 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f7c58009dc0 tx=0x7f7c58009df0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.595+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/682745771 shutdown_connections 2024-06-04T05:08:38.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.595+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/682745771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60bb260 0x557ed60c7080 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.595+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/682745771 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60c0320 0x557ed60c0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.595+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/682745771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60ba8a0 0x557ed60bad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.595+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/682745771 >> 172.21.15.70:0/682745771 conn(0x557ed5fe6240 msgr2=0x557ed60b53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:38.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.595+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/682745771 shutdown_connections 2024-06-04T05:08:38.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.595+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/682745771 wait complete. 2024-06-04T05:08:38.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.595+0000 7f7c6f8bfb80 1 -- start start 2024-06-04T05:08:38.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c6f8bfb80 1 Processor -- start 2024-06-04T05:08:38.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c6f8bfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60ba8a0 0x557ed60c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c6f8bfb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60bb260 0x557ed60c3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c6f8bfb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60c0320 0x557ed60c44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c6f8bfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557ed5f0efa0 con 0x557ed60ba8a0 2024-06-04T05:08:38.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c6f8bfb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557ed5f70da0 con 0x557ed60bb260 2024-06-04T05:08:38.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c6f8bfb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557ed60c7680 con 0x557ed60c0320 2024-06-04T05:08:38.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c6f8bfb80 10 client.0 _ll_register_callbacks cb 0x557ed5fa3950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:38.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c5ed9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60ba8a0 0x557ed60c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c5ed9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60ba8a0 0x557ed60c6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46178/0 (socket says 172.21.15.70:46178) 2024-06-04T05:08:38.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c5ed9d700 1 -- 172.21.15.70:0/3785027395 learned_addr learned my addr 172.21.15.70:0/3785027395 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:38.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c5e59c700 1 --2- 172.21.15.70:0/3785027395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60bb260 0x557ed60c3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c5f59e700 1 --2- 172.21.15.70:0/3785027395 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60c0320 0x557ed60c44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c5ed9d700 1 -- 172.21.15.70:0/3785027395 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60c0320 msgr2=0x557ed60c44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:38.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c5ed9d700 1 --2- 172.21.15.70:0/3785027395 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60c0320 0x557ed60c44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c5ed9d700 1 -- 172.21.15.70:0/3785027395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60bb260 msgr2=0x557ed60c3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:38.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c5ed9d700 1 --2- 172.21.15.70:0/3785027395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60bb260 0x557ed60c3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c5ed9d700 1 -- 172.21.15.70:0/3785027395 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ed5f588d0 con 0x557ed60ba8a0 2024-06-04T05:08:38.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.596+0000 7f7c5f59e700 1 --2- 172.21.15.70:0/3785027395 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60c0320 0x557ed60c44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:38.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.597+0000 7f7c5ed9d700 1 --2- 172.21.15.70:0/3785027395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60ba8a0 0x557ed60c6cb0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f7c5001dac0 tx=0x7f7c500047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c0e170c5217ded server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:38.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.597+0000 7f7c42ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:38.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.597+0000 7f7c42ff5700 1 -- 172.21.15.70:0/3785027395 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c50016070 con 0x557ed60ba8a0 2024-06-04T05:08:38.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.597+0000 7f7c42ff5700 1 -- 172.21.15.70:0/3785027395 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f7c50004a30 con 0x557ed60ba8a0 2024-06-04T05:08:38.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.597+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/3785027395 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c580093d0 con 0x557ed60ba8a0 2024-06-04T05:08:38.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.597+0000 7f7c42ff5700 1 -- 172.21.15.70:0/3785027395 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f7c50004c30 con 0x557ed60ba8a0 2024-06-04T05:08:38.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.597+0000 7f7c6f8bfb80 10 client.18486 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:38.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.597+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/3785027395 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x557ed5fc2760 con 0x557ed60ba8a0 2024-06-04T05:08:38.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.597+0000 7f7c42ff5700 1 -- 172.21.15.70:0/3785027395 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c580093d0 con 0x557ed60ba8a0 2024-06-04T05:08:38.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.597+0000 7f7c42ff5700 1 -- 172.21.15.70:0/3785027395 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f7c5002d070 con 0x557ed60ba8a0 2024-06-04T05:08:38.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.597+0000 7f7c6f8bfb80 10 client.18486 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:38.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.597+0000 7f7c6f8bfb80 10 client.18486 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:08:38.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.597+0000 7f7c6f8bfb80 20 client.18486 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:38.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.597+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/3785027395 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557ed60d41e0 0x557ed60d65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.598+0000 7f7c6f8bfb80 4 client.18486 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:08:38.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.598+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/3785027395 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ed601acc0 con 0x557ed60d41e0 2024-06-04T05:08:38.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.598+0000 7f7c5e59c700 1 --2- 172.21.15.70:0/3785027395 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557ed60d41e0 0x557ed60d65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.598+0000 7f7c5e59c700 1 --2- 172.21.15.70:0/3785027395 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557ed60d41e0 0x557ed60d65d0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce4537b98b896c31 server_cookie=39a5c9d48034d35b in_seq=0 out_seq=0 2024-06-04T05:08:38.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.598+0000 7f7c42ff5700 10 client.18486 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:38.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.599+0000 7f7c42ff5700 1 -- 172.21.15.70:0/3785027395 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557ed601acc0 con 0x557ed60d41e0 2024-06-04T05:08:38.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.599+0000 7f7c42ff5700 10 client.18486 handle_command_reply: tid=0 2024-06-04T05:08:38.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.599+0000 7f7c6f8bfb80 1 client.18486 shutdown 2024-06-04T05:08:38.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.599+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/3785027395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60ba8a0 msgr2=0x557ed60c6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:38.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.599+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/3785027395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60ba8a0 0x557ed60c6cb0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f7c5001dac0 tx=0x7f7c500047c0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.599+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/3785027395 shutdown_connections 2024-06-04T05:08:38.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.599+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/3785027395 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60c0320 0x557ed60c44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.599+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/3785027395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60ba8a0 0x557ed60c6cb0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.599+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/3785027395 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557ed60d41e0 0x557ed60d65d0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.599+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/3785027395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60bb260 0x557ed60c3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.599+0000 7f7c42ff5700 0 client.18486 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:38.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.599+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/3785027395 >> 172.21.15.70:0/3785027395 conn(0x557ed5fe6240 msgr2=0x557ed60b4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:38.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.599+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/3785027395 shutdown_connections 2024-06-04T05:08:38.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.599+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/3785027395 wait complete. 2024-06-04T05:08:38.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.599+0000 7f7c6f8bfb80 20 client.18486 trim_cache size 0 max 16384 2024-06-04T05:08:38.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.619+0000 7f7c6f8bfb80 1 Processor -- start 2024-06-04T05:08:38.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.619+0000 7f7c6f8bfb80 1 -- start start 2024-06-04T05:08:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.619+0000 7f7c6f8bfb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60ba8a0 0x557ed60b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.619+0000 7f7c6f8bfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60bb260 0x557ed60c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.619+0000 7f7c6f8bfb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60c0320 0x557ed60bcd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.619+0000 7f7c6f8bfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7c601adc70 con 0x557ed60bb260 2024-06-04T05:08:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.619+0000 7f7c6f8bfb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c601af810 con 0x557ed60c0320 2024-06-04T05:08:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.619+0000 7f7c6f8bfb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7c601ae340 con 0x557ed60ba8a0 2024-06-04T05:08:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.620+0000 7f7c5e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60bb260 0x557ed60c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.620+0000 7f7c5e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60bb260 0x557ed60c5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46182/0 (socket says 172.21.15.70:46182) 2024-06-04T05:08:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.620+0000 7f7c5e59c700 1 -- 172.21.15.70:0/2253110496 learned_addr learned my addr 172.21.15.70:0/2253110496 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.620+0000 7f7c5f59e700 1 --2- 172.21.15.70:0/2253110496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60c0320 0x557ed60bcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.620+0000 7f7c5ed9d700 1 --2- 172.21.15.70:0/2253110496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60ba8a0 0x557ed60b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.620+0000 7f7c5e59c700 1 -- 172.21.15.70:0/2253110496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60ba8a0 msgr2=0x557ed60b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.620+0000 7f7c5e59c700 1 --2- 172.21.15.70:0/2253110496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60ba8a0 0x557ed60b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.620+0000 7f7c5e59c700 1 -- 172.21.15.70:0/2253110496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60c0320 msgr2=0x557ed60bcd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.620+0000 7f7c5e59c700 1 --2- 172.21.15.70:0/2253110496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60c0320 0x557ed60bcd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.620+0000 7f7c5e59c700 1 -- 172.21.15.70:0/2253110496 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ed60b6e10 con 0x557ed60bb260 2024-06-04T05:08:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.620+0000 7f7c5ed9d700 1 --2- 172.21.15.70:0/2253110496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60ba8a0 0x557ed60b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.620+0000 7f7c5f59e700 1 --2- 172.21.15.70:0/2253110496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60c0320 0x557ed60bcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.620+0000 7f7c5e59c700 1 --2- 172.21.15.70:0/2253110496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60bb260 0x557ed60c5ef0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f7c540a55f0 tx=0x7f7c54010f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=131ae7cc4a7f25 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.620+0000 7f7c44ff9700 1 -- 172.21.15.70:0/2253110496 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5406d070 con 0x557ed60bb260 2024-06-04T05:08:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.620+0000 7f7c44ff9700 1 -- 172.21.15.70:0/2253110496 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c54010460 con 0x557ed60bb260 2024-06-04T05:08:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.620+0000 7f7c44ff9700 1 -- 172.21.15.70:0/2253110496 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c540c3070 con 0x557ed60bb260 2024-06-04T05:08:38.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.621+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/2253110496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60bb260 msgr2=0x557ed60c5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:38.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.621+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/2253110496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60bb260 0x557ed60c5ef0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f7c540a55f0 tx=0x7f7c54010f50 comp rx=0 tx=0).stop 2024-06-04T05:08:38.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.621+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/2253110496 shutdown_connections 2024-06-04T05:08:38.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.621+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/2253110496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60c0320 0x557ed60bcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.621+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/2253110496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60ba8a0 0x557ed60b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.621+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/2253110496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60bb260 0x557ed60c5ef0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.621+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/2253110496 >> 172.21.15.70:0/2253110496 conn(0x557ed5fe6240 msgr2=0x557ed60c57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:38.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.621+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/2253110496 shutdown_connections 2024-06-04T05:08:38.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.621+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/2253110496 wait complete. 2024-06-04T05:08:38.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.621+0000 7f7c6f8bfb80 1 -- start start 2024-06-04T05:08:38.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.621+0000 7f7c6f8bfb80 1 Processor -- start 2024-06-04T05:08:38.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c6f8bfb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60ba8a0 0x557ed60ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c6f8bfb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60bb260 0x557ed60cd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c6f8bfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60c0320 0x557ed60c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c6f8bfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7c601af110 con 0x557ed60c0320 2024-06-04T05:08:38.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c6f8bfb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c601aea10 con 0x557ed60bb260 2024-06-04T05:08:38.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c6f8bfb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7c601abc30 con 0x557ed60ba8a0 2024-06-04T05:08:38.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c6f8bfb80 10 client.0 _ll_register_callbacks cb 0x557ed5fa3950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:38.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c5f59e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60c0320 0x557ed60c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c5f59e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60c0320 0x557ed60c79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46192/0 (socket says 172.21.15.70:46192) 2024-06-04T05:08:38.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c5f59e700 1 -- 172.21.15.70:0/2108515135 learned_addr learned my addr 172.21.15.70:0/2108515135 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:38.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c5e59c700 1 --2- 172.21.15.70:0/2108515135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60bb260 0x557ed60cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c5ed9d700 1 --2- 172.21.15.70:0/2108515135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60ba8a0 0x557ed60ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c5f59e700 1 -- 172.21.15.70:0/2108515135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60ba8a0 msgr2=0x557ed60ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:38.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c5f59e700 1 --2- 172.21.15.70:0/2108515135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60ba8a0 0x557ed60ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c5f59e700 1 -- 172.21.15.70:0/2108515135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60bb260 msgr2=0x557ed60cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:38.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c5f59e700 1 --2- 172.21.15.70:0/2108515135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60bb260 0x557ed60cd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c5f59e700 1 -- 172.21.15.70:0/2108515135 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ed60c8190 con 0x557ed60c0320 2024-06-04T05:08:38.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c5f59e700 1 --2- 172.21.15.70:0/2108515135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60c0320 0x557ed60c79b0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800b0d0 tx=0x7f7c58005820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d665d9d417fbc5f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:38.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c42ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:38.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c42ff5700 1 -- 172.21.15.70:0/2108515135 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c58021040 con 0x557ed60c0320 2024-06-04T05:08:38.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/2108515135 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c54010460 con 0x557ed60c0320 2024-06-04T05:08:38.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.622+0000 7f7c42ff5700 1 -- 172.21.15.70:0/2108515135 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f7c580078b0 con 0x557ed60c0320 2024-06-04T05:08:38.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.623+0000 7f7c42ff5700 1 -- 172.21.15.70:0/2108515135 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f7c5800b820 con 0x557ed60c0320 2024-06-04T05:08:38.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.623+0000 7f7c6f8bfb80 10 client.18492 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:38.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.623+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/2108515135 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x557ed613f920 con 0x557ed60c0320 2024-06-04T05:08:38.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.623+0000 7f7c42ff5700 1 -- 172.21.15.70:0/2108515135 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c54010460 con 0x557ed60c0320 2024-06-04T05:08:38.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.623+0000 7f7c42ff5700 1 -- 172.21.15.70:0/2108515135 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f7c58011070 con 0x557ed60c0320 2024-06-04T05:08:38.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.623+0000 7f7c6f8bfb80 10 client.18492 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:38.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.623+0000 7f7c6f8bfb80 10 client.18492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:08:38.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.623+0000 7f7c6f8bfb80 20 client.18492 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:38.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.623+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/2108515135 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557ed60d41e0 0x557ed6141bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:38.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.623+0000 7f7c6f8bfb80 4 client.18492 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:08:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.623+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/2108515135 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557ed60144a0 con 0x557ed60d41e0 2024-06-04T05:08:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.623+0000 7f7c5ed9d700 1 --2- 172.21.15.70:0/2108515135 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557ed60d41e0 0x557ed6141bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.624+0000 7f7c5ed9d700 1 --2- 172.21.15.70:0/2108515135 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557ed60d41e0 0x557ed6141bb0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b69e811165e00c12 server_cookie=c4511832bed62ff5 in_seq=0 out_seq=0 2024-06-04T05:08:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.624+0000 7f7c42ff5700 10 client.18492 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.624+0000 7f7c42ff5700 1 -- 172.21.15.70:0/2108515135 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557ed60144a0 con 0x557ed60d41e0 2024-06-04T05:08:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.624+0000 7f7c42ff5700 10 client.18492 handle_command_reply: tid=0 2024-06-04T05:08:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.624+0000 7f7c6f8bfb80 1 client.18492 shutdown 2024-06-04T05:08:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.624+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/2108515135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60c0320 msgr2=0x557ed60c79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.624+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/2108515135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60c0320 0x557ed60c79b0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800b0d0 tx=0x7f7c58005820 comp rx=0 tx=0).stop 2024-06-04T05:08:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.624+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/2108515135 shutdown_connections 2024-06-04T05:08:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.624+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/2108515135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557ed60c0320 0x557ed60c79b0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.624+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/2108515135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ed60ba8a0 0x557ed60ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.624+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/2108515135 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557ed60d41e0 0x557ed6141bb0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.624+0000 7f7c6f8bfb80 1 --2- 172.21.15.70:0/2108515135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ed60bb260 0x557ed60cd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.624+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/2108515135 >> 172.21.15.70:0/2108515135 conn(0x557ed5fe6240 msgr2=0x557ed60c18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:38.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.624+0000 7f7c42ff5700 0 client.18492 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:38.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.624+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/2108515135 shutdown_connections 2024-06-04T05:08:38.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.624+0000 7f7c6f8bfb80 1 -- 172.21.15.70:0/2108515135 wait complete. 2024-06-04T05:08:38.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.624+0000 7f7c6f8bfb80 20 client.18492 trim_cache size 0 max 16384 2024-06-04T05:08:38.633 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:08:38.633 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:08:38.634 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:08:38.634 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:08:38.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.625+0000 7f7c6502a700 1 -- 172.21.15.70:0/883800438 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7c480b29f0 msgr2=0x7f7c480b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:38.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.625+0000 7f7c6502a700 1 --2- 172.21.15.70:0/883800438 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7c480b29f0 0x7f7c480b4eb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7c50005950 tx=0x7f7c5000a300 comp rx=0 tx=0).stop 2024-06-04T05:08:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.625+0000 7f7c6502a700 1 -- 172.21.15.70:0/883800438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c60104200 msgr2=0x7f7c6019fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.625+0000 7f7c6502a700 1 --2- 172.21.15.70:0/883800438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c60104200 0x7f7c6019fbf0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f7c5400d950 tx=0x7f7c5400dc60 comp rx=0 tx=0).stop 2024-06-04T05:08:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.625+0000 7f7c6502a700 1 -- 172.21.15.70:0/883800438 shutdown_connections 2024-06-04T05:08:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.625+0000 7f7c6502a700 1 --2- 172.21.15.70:0/883800438 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7c480b29f0 0x7f7c480b4eb0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.625+0000 7f7c6502a700 1 --2- 172.21.15.70:0/883800438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c60108920 0x7f7c60199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.625+0000 7f7c6502a700 1 --2- 172.21.15.70:0/883800438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c60103850 0x7f7c6019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.625+0000 7f7c6502a700 1 --2- 172.21.15.70:0/883800438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c60104200 0x7f7c6019fbf0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.625+0000 7f7c6502a700 1 -- 172.21.15.70:0/883800438 >> 172.21.15.70:0/883800438 conn(0x7f7c6007a5d0 msgr2=0x7f7c60101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.625+0000 7f7c6502a700 1 -- 172.21.15.70:0/883800438 shutdown_connections 2024-06-04T05:08:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:38.625+0000 7f7c6502a700 1 -- 172.21.15.70:0/883800438 wait complete. 2024-06-04T05:08:38.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:38 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:08:38.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:38 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:08:38.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:38 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:08:38.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.80 MB/sec execute 451 sec latency 4152.894 ms 2024-06-04T05:08:39.384 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:08:39.385 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:08:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:39 smithi070 ceph-mon[32631]: pgmap v355: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 374 KiB/s wr, 55 op/s 2024-06-04T05:08:39.634 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:08:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:39 smithi112 ceph-mon[36453]: pgmap v355: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 374 KiB/s wr, 55 op/s 2024-06-04T05:08:39.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.80 MB/sec execute 452 sec latency 5152.951 ms 2024-06-04T05:08:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:39 smithi098 ceph-mon[35569]: pgmap v355: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 374 KiB/s wr, 55 op/s 2024-06-04T05:08:40.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.79 MB/sec execute 453 sec latency 6152.987 ms 2024-06-04T05:08:41.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.355+0000 7f534791e700 1 -- 172.21.15.70:0/3147052479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5340103010 msgr2=0x7f5340103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:41.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.355+0000 7f534791e700 1 --2- 172.21.15.70:0/3147052479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5340103010 0x7f5340103490 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f53300099c0 tx=0x7f5330009cd0 comp rx=0 tx=0).stop 2024-06-04T05:08:41.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.355+0000 7f534791e700 1 -- 172.21.15.70:0/3147052479 shutdown_connections 2024-06-04T05:08:41.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.355+0000 7f534791e700 1 --2- 172.21.15.70:0/3147052479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53401039d0 0x7f534010f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:41.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.355+0000 7f534791e700 1 --2- 172.21.15.70:0/3147052479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5340108730 0x7f5340108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:41.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.355+0000 7f534791e700 1 --2- 172.21.15.70:0/3147052479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5340103010 0x7f5340103490 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:41.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.355+0000 7f534791e700 1 -- 172.21.15.70:0/3147052479 >> 172.21.15.70:0/3147052479 conn(0x7f5340077780 msgr2=0x7f5340077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:41.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.355+0000 7f534791e700 1 -- 172.21.15.70:0/3147052479 shutdown_connections 2024-06-04T05:08:41.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.355+0000 7f534791e700 1 -- 172.21.15.70:0/3147052479 wait complete. 2024-06-04T05:08:41.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.355+0000 7f534791e700 1 Processor -- start 2024-06-04T05:08:41.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.355+0000 7f534791e700 1 -- start start 2024-06-04T05:08:41.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.355+0000 7f534791e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5340103010 0x7f53401989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:41.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.355+0000 7f534791e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53401039d0 0x7f5340198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:41.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f534791e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5340108730 0x7f5340199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:41.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f534791e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5340112700 con 0x7f53401039d0 2024-06-04T05:08:41.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f534791e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5340112580 con 0x7f5340103010 2024-06-04T05:08:41.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f534791e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5340112880 con 0x7f5340108730 2024-06-04T05:08:41.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f5344eb9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53401039d0 0x7f5340198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:41.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f5344eb9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53401039d0 0x7f5340198ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46208/0 (socket says 172.21.15.70:46208) 2024-06-04T05:08:41.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f5344eb9700 1 -- 172.21.15.70:0/198979558 learned_addr learned my addr 172.21.15.70:0/198979558 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:41.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f5345ebb700 1 --2- 172.21.15.70:0/198979558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5340108730 0x7f5340199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:41.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f5344eb9700 1 -- 172.21.15.70:0/198979558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5340108730 msgr2=0x7f5340199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:41.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f5344eb9700 1 --2- 172.21.15.70:0/198979558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5340108730 0x7f5340199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:41.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f5344eb9700 1 -- 172.21.15.70:0/198979558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5340103010 msgr2=0x7f53401989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:41.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f53456ba700 1 --2- 172.21.15.70:0/198979558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5340103010 0x7f53401989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:41.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f5344eb9700 1 --2- 172.21.15.70:0/198979558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5340103010 0x7f53401989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:41.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f5344eb9700 1 -- 172.21.15.70:0/198979558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53401a81c0 con 0x7f53401039d0 2024-06-04T05:08:41.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f5344eb9700 1 --2- 172.21.15.70:0/198979558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53401039d0 0x7f5340198ef0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f5330000c00 tx=0x7f533000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:41.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f53367fc700 1 -- 172.21.15.70:0/198979558 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5330026070 con 0x7f53401039d0 2024-06-04T05:08:41.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f53367fc700 1 -- 172.21.15.70:0/198979558 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f53300054c0 con 0x7f53401039d0 2024-06-04T05:08:41.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f534791e700 1 -- 172.21.15.70:0/198979558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53401a8450 con 0x7f53401039d0 2024-06-04T05:08:41.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f53367fc700 1 -- 172.21.15.70:0/198979558 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5330004c50 con 0x7f53401039d0 2024-06-04T05:08:41.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.356+0000 7f534791e700 1 -- 172.21.15.70:0/198979558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53401a88a0 con 0x7f53401039d0 2024-06-04T05:08:41.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.357+0000 7f534791e700 1 -- 172.21.15.70:0/198979558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f534006a230 con 0x7f53401039d0 2024-06-04T05:08:41.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.359+0000 7f53367fc700 1 -- 172.21.15.70:0/198979558 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f5330029070 con 0x7f53401039d0 2024-06-04T05:08:41.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.359+0000 7f53367fc700 1 --2- 172.21.15.70:0/198979558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f532c0b29f0 0x7f532c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:41.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.359+0000 7f53456ba700 1 --2- 172.21.15.70:0/198979558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f532c0b29f0 0x7f532c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:41.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.359+0000 7f53367fc700 1 -- 172.21.15.70:0/198979558 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f53300e3660 con 0x7f53401039d0 2024-06-04T05:08:41.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.360+0000 7f53456ba700 1 --2- 172.21.15.70:0/198979558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f532c0b29f0 0x7f532c0b4eb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f533c006fd0 tx=0x7f533c008040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:41.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.363+0000 7f53367fc700 1 -- 172.21.15.70:0/198979558 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5330030020 con 0x7f53401039d0 2024-06-04T05:08:41.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:41 smithi070 ceph-mon[32631]: pgmap v356: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 374 KiB/s wr, 55 op/s 2024-06-04T05:08:41.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.612+0000 7f534791e700 1 -- 172.21.15.70:0/198979558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f534002e560 con 0x7f53401039d0 2024-06-04T05:08:41.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.613+0000 7f53367fc700 1 -- 172.21.15.70:0/198979558 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11931 (secure 0 0 0) 0x7f5330013070 con 0x7f53401039d0 2024-06-04T05:08:41.615 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:08:41.615 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:07:27.814997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24436,24410,24440,24448,24442]},"id":1}]} 2024-06-04T05:08:41.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.615+0000 7f534791e700 1 -- 172.21.15.70:0/198979558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f532c0b29f0 msgr2=0x7f532c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:41.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.615+0000 7f534791e700 1 --2- 172.21.15.70:0/198979558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f532c0b29f0 0x7f532c0b4eb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f533c006fd0 tx=0x7f533c008040 comp rx=0 tx=0).stop 2024-06-04T05:08:41.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.615+0000 7f534791e700 1 -- 172.21.15.70:0/198979558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53401039d0 msgr2=0x7f5340198ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:41.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.615+0000 7f534791e700 1 --2- 172.21.15.70:0/198979558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53401039d0 0x7f5340198ef0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f5330000c00 tx=0x7f533000f740 comp rx=0 tx=0).stop 2024-06-04T05:08:41.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.615+0000 7f534791e700 1 -- 172.21.15.70:0/198979558 shutdown_connections 2024-06-04T05:08:41.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.615+0000 7f534791e700 1 --2- 172.21.15.70:0/198979558 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f532c0b29f0 0x7f532c0b4eb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:41.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.615+0000 7f534791e700 1 --2- 172.21.15.70:0/198979558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5340108730 0x7f5340199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:41.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.615+0000 7f534791e700 1 --2- 172.21.15.70:0/198979558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5340103010 0x7f53401989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:41.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.615+0000 7f534791e700 1 --2- 172.21.15.70:0/198979558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53401039d0 0x7f5340198ef0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:41.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.615+0000 7f534791e700 1 -- 172.21.15.70:0/198979558 >> 172.21.15.70:0/198979558 conn(0x7f5340077780 msgr2=0x7f53400fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:41.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.615+0000 7f534791e700 1 -- 172.21.15.70:0/198979558 shutdown_connections 2024-06-04T05:08:41.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:41.615+0000 7f534791e700 1 -- 172.21.15.70:0/198979558 wait complete. 2024-06-04T05:08:41.619 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 34 2024-06-04T05:08:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:41 smithi112 ceph-mon[36453]: pgmap v356: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 374 KiB/s wr, 55 op/s 2024-06-04T05:08:41.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.79 MB/sec execute 454 sec latency 7153.074 ms 2024-06-04T05:08:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:41 smithi098 ceph-mon[35569]: pgmap v356: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 374 KiB/s wr, 55 op/s 2024-06-04T05:08:42.458 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:08:42.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:42 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/198979558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:08:42.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:42 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/198979558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:08:42.704 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:08:42.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.78 MB/sec execute 455 sec latency 8153.113 ms 2024-06-04T05:08:42.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:42 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/198979558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:08:43.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:43 smithi070 ceph-mon[32631]: pgmap v357: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 409 KiB/s wr, 67 op/s 2024-06-04T05:08:43.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:43 smithi098 ceph-mon[35569]: pgmap v357: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 409 KiB/s wr, 67 op/s 2024-06-04T05:08:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:43 smithi112 ceph-mon[36453]: pgmap v357: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 409 KiB/s wr, 67 op/s 2024-06-04T05:08:43.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.78 MB/sec execute 456 sec latency 9153.224 ms 2024-06-04T05:08:43.744 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:08:44.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.575+0000 7fa257d3d700 1 -- 172.21.15.70:0/445918110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa250101e60 msgr2=0x7fa250102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:44.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.575+0000 7fa257d3d700 1 --2- 172.21.15.70:0/445918110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa250101e60 0x7fa250102240 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fa24000aad0 tx=0x7fa24000ade0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.575+0000 7fa257d3d700 1 -- 172.21.15.70:0/445918110 shutdown_connections 2024-06-04T05:08:44.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.575+0000 7fa257d3d700 1 --2- 172.21.15.70:0/445918110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa250106ef0 0x7fa25010eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.575+0000 7fa257d3d700 1 --2- 172.21.15.70:0/445918110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa250101e60 0x7fa250102240 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.575+0000 7fa257d3d700 1 --2- 172.21.15.70:0/445918110 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa250102810 0x7fa250106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.575+0000 7fa257d3d700 1 -- 172.21.15.70:0/445918110 >> 172.21.15.70:0/445918110 conn(0x7fa250078ab0 msgr2=0x7fa2500ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:44.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.575+0000 7fa257d3d700 1 -- 172.21.15.70:0/445918110 shutdown_connections 2024-06-04T05:08:44.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.575+0000 7fa257d3d700 1 -- 172.21.15.70:0/445918110 wait complete. 2024-06-04T05:08:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.576+0000 7fa257d3d700 1 Processor -- start 2024-06-04T05:08:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.576+0000 7fa257d3d700 1 -- start start 2024-06-04T05:08:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.576+0000 7fa257d3d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa250101e60 0x7fa25019f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.576+0000 7fa257d3d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa250102810 0x7fa25019fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.576+0000 7fa257d3d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa250106ef0 0x7fa250199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.576+0000 7fa257d3d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa250111400 con 0x7fa250106ef0 2024-06-04T05:08:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.576+0000 7fa257d3d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa250111280 con 0x7fa250102810 2024-06-04T05:08:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.576+0000 7fa257d3d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa250111580 con 0x7fa250101e60 2024-06-04T05:08:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.576+0000 7fa2562da700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa250106ef0 0x7fa250199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.576+0000 7fa2562da700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa250106ef0 0x7fa250199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46216/0 (socket says 172.21.15.70:46216) 2024-06-04T05:08:44.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.576+0000 7fa2562da700 1 -- 172.21.15.70:0/1862924294 learned_addr learned my addr 172.21.15.70:0/1862924294 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.576+0000 7fa2552d8700 1 --2- 172.21.15.70:0/1862924294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa250102810 0x7fa25019fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.576+0000 7fa255ad9700 1 --2- 172.21.15.70:0/1862924294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa250101e60 0x7fa25019f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.577+0000 7fa2562da700 1 -- 172.21.15.70:0/1862924294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa250101e60 msgr2=0x7fa25019f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.577+0000 7fa2562da700 1 --2- 172.21.15.70:0/1862924294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa250101e60 0x7fa25019f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.577+0000 7fa2562da700 1 -- 172.21.15.70:0/1862924294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa250102810 msgr2=0x7fa25019fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.577+0000 7fa2562da700 1 --2- 172.21.15.70:0/1862924294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa250102810 0x7fa25019fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.577+0000 7fa2562da700 1 -- 172.21.15.70:0/1862924294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25019a0b0 con 0x7fa250106ef0 2024-06-04T05:08:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.577+0000 7fa255ad9700 1 --2- 172.21.15.70:0/1862924294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa250101e60 0x7fa25019f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.577+0000 7fa2562da700 1 --2- 172.21.15.70:0/1862924294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa250106ef0 0x7fa250199950 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fa248007d70 tx=0x7fa24800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:44.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.577+0000 7fa246ffd700 1 -- 172.21.15.70:0/1862924294 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa248016020 con 0x7fa250106ef0 2024-06-04T05:08:44.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.577+0000 7fa246ffd700 1 -- 172.21.15.70:0/1862924294 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa24800ad40 con 0x7fa250106ef0 2024-06-04T05:08:44.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.577+0000 7fa246ffd700 1 -- 172.21.15.70:0/1862924294 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2480115e0 con 0x7fa250106ef0 2024-06-04T05:08:44.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.577+0000 7fa257d3d700 1 -- 172.21.15.70:0/1862924294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa25019a3a0 con 0x7fa250106ef0 2024-06-04T05:08:44.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.577+0000 7fa257d3d700 1 -- 172.21.15.70:0/1862924294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa250197670 con 0x7fa250106ef0 2024-06-04T05:08:44.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.579+0000 7fa246ffd700 1 -- 172.21.15.70:0/1862924294 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fa248019020 con 0x7fa250106ef0 2024-06-04T05:08:44.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.580+0000 7fa246ffd700 1 --2- 172.21.15.70:0/1862924294 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa23c0b2a70 0x7fa23c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.581+0000 7fa255ad9700 1 --2- 172.21.15.70:0/1862924294 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa23c0b2a70 0x7fa23c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.581+0000 7fa246ffd700 1 -- 172.21.15.70:0/1862924294 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7fa2480e23f0 con 0x7fa250106ef0 2024-06-04T05:08:44.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.581+0000 7fa255ad9700 1 --2- 172.21.15.70:0/1862924294 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa23c0b2a70 0x7fa23c0b4f30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa250106b00 tx=0x7fa24000a700 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:44.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.581+0000 7fa2625d2b80 1 Processor -- start 2024-06-04T05:08:44.584 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:08:44.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.581+0000 7fa2625d2b80 1 -- start start 2024-06-04T05:08:44.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa2625d2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f29819320 0x561f29819700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa2625d2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f298138a0 0x561f29813d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa2625d2b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f29820080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa2625d2b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561f29771d70 con 0x561f29814260 2024-06-04T05:08:44.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa2625d2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561f29761910 con 0x561f298138a0 2024-06-04T05:08:44.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa2625d2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561f29765950 con 0x561f29819320 2024-06-04T05:08:44.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa2562da700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f29820080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa2562da700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f29820080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46222/0 (socket says 172.21.15.70:46222) 2024-06-04T05:08:44.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa2562da700 1 -- 172.21.15.70:0/1080859707 learned_addr learned my addr 172.21.15.70:0/1080859707 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:44.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa2552d8700 1 --2- 172.21.15.70:0/1080859707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f298138a0 0x561f29813d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa255ad9700 1 --2- 172.21.15.70:0/1080859707 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f29819320 0x561f29819700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa2562da700 1 -- 172.21.15.70:0/1080859707 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f29819320 msgr2=0x561f29819700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:44.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa2562da700 1 --2- 172.21.15.70:0/1080859707 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f29819320 0x561f29819700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa2562da700 1 -- 172.21.15.70:0/1080859707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f298138a0 msgr2=0x561f29813d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:44.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa2562da700 1 --2- 172.21.15.70:0/1080859707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f298138a0 0x561f29813d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa2562da700 1 -- 172.21.15.70:0/1080859707 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f2970afb0 con 0x561f29814260 2024-06-04T05:08:44.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa2562da700 1 --2- 172.21.15.70:0/1080859707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f29820080 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fa2480a5830 tx=0x7fa2480a8fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65aa39cffecbd6a7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:44.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa23bfff700 1 -- 172.21.15.70:0/1080859707 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2480a9eb0 con 0x561f29814260 2024-06-04T05:08:44.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa23bfff700 1 -- 172.21.15.70:0/1080859707 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa2480a97b0 con 0x561f29814260 2024-06-04T05:08:44.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.582+0000 7fa23bfff700 1 -- 172.21.15.70:0/1080859707 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2480910d0 con 0x561f29814260 2024-06-04T05:08:44.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.583+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1080859707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 msgr2=0x561f29820080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:44.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.583+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1080859707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f29820080 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fa2480a5830 tx=0x7fa2480a8fa0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.583+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1080859707 shutdown_connections 2024-06-04T05:08:44.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.583+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1080859707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f29820080 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.583+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1080859707 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f29819320 0x561f29819700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.583+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1080859707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f298138a0 0x561f29813d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.583+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1080859707 >> 172.21.15.70:0/1080859707 conn(0x561f2973f240 msgr2=0x561f2980e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:44.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.583+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1080859707 shutdown_connections 2024-06-04T05:08:44.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.583+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1080859707 wait complete. 2024-06-04T05:08:44.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.584+0000 7fa2625d2b80 1 -- start start 2024-06-04T05:08:44.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.584+0000 7fa2625d2b80 1 Processor -- start 2024-06-04T05:08:44.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.584+0000 7fa2625d2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f298138a0 0x561f2981fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.584+0000 7fa2625d2b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f2981d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.584+0000 7fa2625d2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f29819320 0x561f2981d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.584+0000 7fa2625d2b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561f29667fa0 con 0x561f29814260 2024-06-04T05:08:44.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.584+0000 7fa2625d2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561f296c9da0 con 0x561f298138a0 2024-06-04T05:08:44.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.584+0000 7fa2625d2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561f29820680 con 0x561f29819320 2024-06-04T05:08:44.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.584+0000 7fa2625d2b80 10 client.0 _ll_register_callbacks cb 0x561f296fc950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:44.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.585+0000 7fa2552d8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f2981d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.585+0000 7fa2552d8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f2981d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46226/0 (socket says 172.21.15.70:46226) 2024-06-04T05:08:44.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.585+0000 7fa2552d8700 1 -- 172.21.15.70:0/1383309256 learned_addr learned my addr 172.21.15.70:0/1383309256 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:44.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.585+0000 7fa255ad9700 1 --2- 172.21.15.70:0/1383309256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f298138a0 0x561f2981fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.585+0000 7fa2562da700 1 --2- 172.21.15.70:0/1383309256 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f29819320 0x561f2981d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.585+0000 7fa2552d8700 1 -- 172.21.15.70:0/1383309256 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f29819320 msgr2=0x561f2981d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:44.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.585+0000 7fa2552d8700 1 --2- 172.21.15.70:0/1383309256 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f29819320 0x561f2981d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.585+0000 7fa2552d8700 1 -- 172.21.15.70:0/1383309256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f298138a0 msgr2=0x561f2981fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:44.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.585+0000 7fa2552d8700 1 --2- 172.21.15.70:0/1383309256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f298138a0 0x561f2981fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.585+0000 7fa2552d8700 1 -- 172.21.15.70:0/1383309256 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f296b18d0 con 0x561f29814260 2024-06-04T05:08:44.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.585+0000 7fa2552d8700 1 --2- 172.21.15.70:0/1383309256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f2981d080 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0107f0 tx=0x7fa24c010b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11477e291439c63a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:44.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.585+0000 7fa2397fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:44.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.585+0000 7fa2397fa700 1 -- 172.21.15.70:0/1383309256 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c01e060 con 0x561f29814260 2024-06-04T05:08:44.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.585+0000 7fa2397fa700 1 -- 172.21.15.70:0/1383309256 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7fa24c00f7d0 con 0x561f29814260 2024-06-04T05:08:44.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.585+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1383309256 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2480a97b0 con 0x561f29814260 2024-06-04T05:08:44.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.586+0000 7fa2397fa700 1 -- 172.21.15.70:0/1383309256 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa24c00f9d0 con 0x561f29814260 2024-06-04T05:08:44.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.586+0000 7fa2625d2b80 10 client.18510 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:44.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.586+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1383309256 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x561f2971b760 con 0x561f29814260 2024-06-04T05:08:44.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.586+0000 7fa2397fa700 1 -- 172.21.15.70:0/1383309256 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa2480a97b0 con 0x561f29814260 2024-06-04T05:08:44.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.586+0000 7fa2397fa700 1 -- 172.21.15.70:0/1383309256 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7fa24c021cc0 con 0x561f29814260 2024-06-04T05:08:44.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.586+0000 7fa2625d2b80 10 client.18510 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:44.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.586+0000 7fa2625d2b80 10 client.18510 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:08:44.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.586+0000 7fa2625d2b80 20 client.18510 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:44.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.586+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1383309256 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x561f2982d390 0x561f2982f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.586+0000 7fa2625d2b80 4 client.18510 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:08:44.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.586+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1383309256 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f29773cc0 con 0x561f2982d390 2024-06-04T05:08:44.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.587+0000 7fa255ad9700 1 --2- 172.21.15.70:0/1383309256 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x561f2982d390 0x561f2982f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.587+0000 7fa2397fa700 10 client.18510 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:08:44.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.587+0000 7fa255ad9700 1 --2- 172.21.15.70:0/1383309256 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x561f2982d390 0x561f2982f780 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=41d016c401ce2ac3 server_cookie=501c9f884dea4fbb in_seq=0 out_seq=0 2024-06-04T05:08:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.588+0000 7fa2397fa700 1 -- 172.21.15.70:0/1383309256 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561f29773cc0 con 0x561f2982d390 2024-06-04T05:08:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.588+0000 7fa2397fa700 10 client.18510 handle_command_reply: tid=0 2024-06-04T05:08:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.588+0000 7fa2625d2b80 1 client.18510 shutdown 2024-06-04T05:08:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.588+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1383309256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 msgr2=0x561f2981d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.588+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1383309256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f2981d080 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0107f0 tx=0x7fa24c010b00 comp rx=0 tx=0).stop 2024-06-04T05:08:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.588+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1383309256 shutdown_connections 2024-06-04T05:08:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.588+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1383309256 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x561f2982d390 0x561f2982f780 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.588+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1383309256 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f29819320 0x561f2981d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.588+0000 7fa2397fa700 0 client.18510 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:08:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.588+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1383309256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f298138a0 0x561f2981fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.588+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1383309256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f2981d080 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.588+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1383309256 >> 172.21.15.70:0/1383309256 conn(0x561f2973f240 msgr2=0x561f2980daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:44.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.588+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1383309256 shutdown_connections 2024-06-04T05:08:44.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.588+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1383309256 wait complete. 2024-06-04T05:08:44.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.588+0000 7fa2625d2b80 20 client.18510 trim_cache size 0 max 16384 2024-06-04T05:08:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.604+0000 7fa2625d2b80 1 Processor -- start 2024-06-04T05:08:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.604+0000 7fa2625d2b80 1 -- start start 2024-06-04T05:08:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.604+0000 7fa2625d2b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f298138a0 0x561f2982f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.604+0000 7fa2625d2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f29814260 0x561f298188c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.604+0000 7fa2625d2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f29819320 0x561f298296e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.604+0000 7fa2625d2b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501affd0 con 0x561f298138a0 2024-06-04T05:08:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.604+0000 7fa2625d2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501b1b70 con 0x561f29819320 2024-06-04T05:08:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.604+0000 7fa2625d2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501b06a0 con 0x561f29814260 2024-06-04T05:08:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.605+0000 7fa255ad9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f298138a0 0x561f2982f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.605+0000 7fa255ad9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f298138a0 0x561f2982f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46232/0 (socket says 172.21.15.70:46232) 2024-06-04T05:08:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.605+0000 7fa255ad9700 1 -- 172.21.15.70:0/1163119604 learned_addr learned my addr 172.21.15.70:0/1163119604 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.605+0000 7fa2552d8700 1 --2- 172.21.15.70:0/1163119604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f29814260 0x561f298188c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.605+0000 7fa255ad9700 1 -- 172.21.15.70:0/1163119604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f29814260 msgr2=0x561f298188c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.605+0000 7fa255ad9700 1 --2- 172.21.15.70:0/1163119604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f29814260 0x561f298188c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.605+0000 7fa255ad9700 1 -- 172.21.15.70:0/1163119604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f29819320 msgr2=0x561f298296e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.605+0000 7fa2562da700 1 --2- 172.21.15.70:0/1163119604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f29819320 0x561f298296e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.605+0000 7fa255ad9700 1 --2- 172.21.15.70:0/1163119604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f29819320 0x561f298296e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.605+0000 7fa255ad9700 1 -- 172.21.15.70:0/1163119604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f2982ad00 con 0x561f298138a0 2024-06-04T05:08:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.605+0000 7fa255ad9700 1 --2- 172.21.15.70:0/1163119604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f298138a0 0x561f2982f780 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fa24000c690 tx=0x7fa240023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d0371e8a64987c4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.605+0000 7fa23b7fe700 1 -- 172.21.15.70:0/1163119604 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24003d070 con 0x561f298138a0 2024-06-04T05:08:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.605+0000 7fa23b7fe700 1 -- 172.21.15.70:0/1163119604 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa240026040 con 0x561f298138a0 2024-06-04T05:08:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.605+0000 7fa23b7fe700 1 -- 172.21.15.70:0/1163119604 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24000e820 con 0x561f298138a0 2024-06-04T05:08:44.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.606+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1163119604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f298138a0 msgr2=0x561f2982f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.606+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1163119604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f298138a0 0x561f2982f780 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fa24000c690 tx=0x7fa240023000 comp rx=0 tx=0).stop 2024-06-04T05:08:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.606+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1163119604 shutdown_connections 2024-06-04T05:08:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.606+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1163119604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f29819320 0x561f298296e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.606+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1163119604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f298138a0 0x561f2982f780 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.606+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1163119604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f29814260 0x561f298188c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.606+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1163119604 >> 172.21.15.70:0/1163119604 conn(0x561f2973f240 msgr2=0x561f298181b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.606+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1163119604 shutdown_connections 2024-06-04T05:08:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.606+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1163119604 wait complete. 2024-06-04T05:08:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.607+0000 7fa2625d2b80 1 -- start start 2024-06-04T05:08:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.607+0000 7fa2625d2b80 1 Processor -- start 2024-06-04T05:08:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.607+0000 7fa2625d2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f298138a0 0x561f298258d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.607+0000 7fa2625d2b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f29825e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.607+0000 7fa2625d2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f29819320 0x561f29833fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.607+0000 7fa2625d2b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501b1470 con 0x561f29814260 2024-06-04T05:08:44.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.607+0000 7fa2625d2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501b0d70 con 0x561f29819320 2024-06-04T05:08:44.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.607+0000 7fa2552d8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f29825e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.607+0000 7fa2625d2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501adf90 con 0x561f298138a0 2024-06-04T05:08:44.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.607+0000 7fa2552d8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f29825e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46242/0 (socket says 172.21.15.70:46242) 2024-06-04T05:08:44.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.607+0000 7fa2625d2b80 10 client.0 _ll_register_callbacks cb 0x561f296fc950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:44.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.607+0000 7fa2552d8700 1 -- 172.21.15.70:0/1761196254 learned_addr learned my addr 172.21.15.70:0/1761196254 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:44.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.607+0000 7fa255ad9700 1 --2- 172.21.15.70:0/1761196254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f298138a0 0x561f298258d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.608+0000 7fa2552d8700 1 -- 172.21.15.70:0/1761196254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f298138a0 msgr2=0x561f298258d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:44.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.608+0000 7fa2552d8700 1 --2- 172.21.15.70:0/1761196254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f298138a0 0x561f298258d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.608+0000 7fa2552d8700 1 -- 172.21.15.70:0/1761196254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f29819320 msgr2=0x561f29833fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:44.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.608+0000 7fa2562da700 1 --2- 172.21.15.70:0/1761196254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f29819320 0x561f29833fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.608+0000 7fa2552d8700 1 --2- 172.21.15.70:0/1761196254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f29819320 0x561f29833fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.608+0000 7fa2552d8700 1 -- 172.21.15.70:0/1761196254 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f29834520 con 0x561f29814260 2024-06-04T05:08:44.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.608+0000 7fa255ad9700 1 --2- 172.21.15.70:0/1761196254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f298138a0 0x561f298258d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:44.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.608+0000 7fa2562da700 1 --2- 172.21.15.70:0/1761196254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f29819320 0x561f29833fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:08:44.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.608+0000 7fa2552d8700 1 --2- 172.21.15.70:0/1761196254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f29825e10 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fa24c000c00 tx=0x7fa24c005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2401003a6b6866fe server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:44.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.608+0000 7fa2397fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:44.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.608+0000 7fa2397fa700 1 -- 172.21.15.70:0/1761196254 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c0055b0 con 0x561f29814260 2024-06-04T05:08:44.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.608+0000 7fa2397fa700 1 -- 172.21.15.70:0/1761196254 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7fa24c0046f0 con 0x561f29814260 2024-06-04T05:08:44.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.608+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1761196254 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa240026040 con 0x561f29814260 2024-06-04T05:08:44.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.609+0000 7fa2397fa700 1 -- 172.21.15.70:0/1761196254 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa24c03a050 con 0x561f29814260 2024-06-04T05:08:44.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.609+0000 7fa2625d2b80 10 client.18516 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:44.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.609+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1761196254 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x561f29897620 con 0x561f29814260 2024-06-04T05:08:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.609+0000 7fa2397fa700 1 -- 172.21.15.70:0/1761196254 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa240026040 con 0x561f29814260 2024-06-04T05:08:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.610+0000 7fa2397fa700 1 -- 172.21.15.70:0/1761196254 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7fa24c00e940 con 0x561f29814260 2024-06-04T05:08:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.610+0000 7fa2625d2b80 10 client.18516 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.610+0000 7fa2625d2b80 10 client.18516 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:08:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.610+0000 7fa2625d2b80 20 client.18516 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.610+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1761196254 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x561f29897cc0 0x561f2989a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.610+0000 7fa2625d2b80 4 client.18516 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:08:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.610+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1761196254 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561f2976d4a0 con 0x561f29897cc0 2024-06-04T05:08:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.610+0000 7fa255ad9700 1 --2- 172.21.15.70:0/1761196254 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x561f29897cc0 0x561f2989a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.610+0000 7fa2397fa700 10 client.18516 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:08:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.610+0000 7fa255ad9700 1 --2- 172.21.15.70:0/1761196254 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x561f29897cc0 0x561f2989a0e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fc4b286fe36ed54d server_cookie=62c5d8634aece32d in_seq=0 out_seq=0 2024-06-04T05:08:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.611+0000 7fa2397fa700 1 -- 172.21.15.70:0/1761196254 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561f2976d4a0 con 0x561f29897cc0 2024-06-04T05:08:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.611+0000 7fa2397fa700 10 client.18516 handle_command_reply: tid=0 2024-06-04T05:08:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.611+0000 7fa2625d2b80 1 client.18516 shutdown 2024-06-04T05:08:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.611+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1761196254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 msgr2=0x561f29825e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:44.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.611+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1761196254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f29825e10 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fa24c000c00 tx=0x7fa24c005040 comp rx=0 tx=0).stop 2024-06-04T05:08:44.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.611+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1761196254 shutdown_connections 2024-06-04T05:08:44.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.611+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1761196254 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x561f29897cc0 0x561f2989a0e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.611+0000 7fa2397fa700 0 client.18516 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:08:44.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.611+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1761196254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f29819320 0x561f29833fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.611+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1761196254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561f298138a0 0x561f298258d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.611+0000 7fa2625d2b80 1 --2- 172.21.15.70:0/1761196254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f29814260 0x561f29825e10 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.611+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1761196254 >> 172.21.15.70:0/1761196254 conn(0x561f2973f240 msgr2=0x561f29816d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:44.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.611+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1761196254 shutdown_connections 2024-06-04T05:08:44.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.611+0000 7fa2625d2b80 1 -- 172.21.15.70:0/1761196254 wait complete. 2024-06-04T05:08:44.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.611+0000 7fa2625d2b80 20 client.18516 trim_cache size 0 max 16384 2024-06-04T05:08:44.617 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:08:44.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.612+0000 7fa257d3d700 1 -- 172.21.15.70:0/1862924294 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa23c0b2a70 msgr2=0x7fa23c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:44.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.612+0000 7fa257d3d700 1 --2- 172.21.15.70:0/1862924294 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa23c0b2a70 0x7fa23c0b4f30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa250106b00 tx=0x7fa24000a700 comp rx=0 tx=0).stop 2024-06-04T05:08:44.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.612+0000 7fa257d3d700 1 -- 172.21.15.70:0/1862924294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa250106ef0 msgr2=0x7fa250199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:44.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.612+0000 7fa257d3d700 1 --2- 172.21.15.70:0/1862924294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa250106ef0 0x7fa250199950 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fa248007d70 tx=0x7fa24800a340 comp rx=0 tx=0).stop 2024-06-04T05:08:44.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.612+0000 7fa257d3d700 1 -- 172.21.15.70:0/1862924294 shutdown_connections 2024-06-04T05:08:44.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.612+0000 7fa257d3d700 1 --2- 172.21.15.70:0/1862924294 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa23c0b2a70 0x7fa23c0b4f30 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.612+0000 7fa257d3d700 1 --2- 172.21.15.70:0/1862924294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa250106ef0 0x7fa250199950 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.612+0000 7fa257d3d700 1 --2- 172.21.15.70:0/1862924294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa250101e60 0x7fa25019f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.612+0000 7fa257d3d700 1 --2- 172.21.15.70:0/1862924294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa250102810 0x7fa25019fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:44.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.612+0000 7fa257d3d700 1 -- 172.21.15.70:0/1862924294 >> 172.21.15.70:0/1862924294 conn(0x7fa250078ab0 msgr2=0x7fa2500fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:44.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.612+0000 7fa257d3d700 1 -- 172.21.15.70:0/1862924294 shutdown_connections 2024-06-04T05:08:44.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:44.612+0000 7fa257d3d700 1 -- 172.21.15.70:0/1862924294 wait complete. 2024-06-04T05:08:44.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.78 MB/sec execute 457 sec latency 10153.267 ms 2024-06-04T05:08:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:45 smithi070 ceph-mon[32631]: pgmap v358: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 372 KiB/s wr, 55 op/s 2024-06-04T05:08:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:45 smithi112 ceph-mon[36453]: pgmap v358: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 372 KiB/s wr, 55 op/s 2024-06-04T05:08:45.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.77 MB/sec execute 458 sec latency 11153.350 ms 2024-06-04T05:08:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:45 smithi098 ceph-mon[35569]: pgmap v358: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 372 KiB/s wr, 55 op/s 2024-06-04T05:08:46.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:08:46.736 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.77 MB/sec execute 459 sec latency 12153.400 ms 2024-06-04T05:08:47.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.217+0000 7fb98c22c700 1 -- 172.21.15.70:0/1746182970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb984101ee0 msgr2=0x7fb984102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:47.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.217+0000 7fb98c22c700 1 --2- 172.21.15.70:0/1746182970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb984101ee0 0x7fb984102360 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fb9740099c0 tx=0x7fb974009cd0 comp rx=0 tx=0).stop 2024-06-04T05:08:47.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.217+0000 7fb98c22c700 1 -- 172.21.15.70:0/1746182970 shutdown_connections 2024-06-04T05:08:47.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.217+0000 7fb98c22c700 1 --2- 172.21.15.70:0/1746182970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9841028a0 0x7fb984110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:47.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.217+0000 7fb98c22c700 1 --2- 172.21.15.70:0/1746182970 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9841085b0 0x7fb984108990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:47.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.217+0000 7fb98c22c700 1 --2- 172.21.15.70:0/1746182970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb984101ee0 0x7fb984102360 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:47.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.217+0000 7fb98c22c700 1 -- 172.21.15.70:0/1746182970 >> 172.21.15.70:0/1746182970 conn(0x7fb9840777e0 msgr2=0x7fb984077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:47.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.217+0000 7fb98c22c700 1 -- 172.21.15.70:0/1746182970 shutdown_connections 2024-06-04T05:08:47.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.217+0000 7fb98c22c700 1 -- 172.21.15.70:0/1746182970 wait complete. 2024-06-04T05:08:47.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.218+0000 7fb98c22c700 1 Processor -- start 2024-06-04T05:08:47.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.218+0000 7fb98c22c700 1 -- start start 2024-06-04T05:08:47.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.218+0000 7fb98c22c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb984101ee0 0x7fb9841030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:47.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.218+0000 7fb98c22c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9841028a0 0x7fb984103600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:47.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.218+0000 7fb98c22c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9841085b0 0x7fb984103b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:47.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.218+0000 7fb98c22c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb984113620 con 0x7fb9841028a0 2024-06-04T05:08:47.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.218+0000 7fb98c22c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9841134a0 con 0x7fb9841085b0 2024-06-04T05:08:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.218+0000 7fb98c22c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb9841137a0 con 0x7fb984101ee0 2024-06-04T05:08:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.218+0000 7fb9897c7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9841028a0 0x7fb984103600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.219+0000 7fb9897c7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9841028a0 0x7fb984103600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38112/0 (socket says 172.21.15.70:38112) 2024-06-04T05:08:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.219+0000 7fb9897c7700 1 -- 172.21.15.70:0/840770072 learned_addr learned my addr 172.21.15.70:0/840770072 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.219+0000 7fb98a7c9700 1 --2- 172.21.15.70:0/840770072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9841085b0 0x7fb984103b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.219+0000 7fb989fc8700 1 --2- 172.21.15.70:0/840770072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb984101ee0 0x7fb9841030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.219+0000 7fb9897c7700 1 -- 172.21.15.70:0/840770072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb984101ee0 msgr2=0x7fb9841030c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.219+0000 7fb9897c7700 1 --2- 172.21.15.70:0/840770072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb984101ee0 0x7fb9841030c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.219+0000 7fb9897c7700 1 -- 172.21.15.70:0/840770072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9841085b0 msgr2=0x7fb984103b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.219+0000 7fb9897c7700 1 --2- 172.21.15.70:0/840770072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9841085b0 0x7fb984103b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.219+0000 7fb9897c7700 1 -- 172.21.15.70:0/840770072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9841a80b0 con 0x7fb9841028a0 2024-06-04T05:08:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.219+0000 7fb989fc8700 1 --2- 172.21.15.70:0/840770072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb984101ee0 0x7fb9841030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.219+0000 7fb9897c7700 1 --2- 172.21.15.70:0/840770072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9841028a0 0x7fb984103600 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fb974000c00 tx=0x7fb97400f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.219+0000 7fb97affd700 1 -- 172.21.15.70:0/840770072 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97400f940 con 0x7fb9841028a0 2024-06-04T05:08:47.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.219+0000 7fb98c22c700 1 -- 172.21.15.70:0/840770072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9841a8340 con 0x7fb9841028a0 2024-06-04T05:08:47.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.219+0000 7fb97affd700 1 -- 172.21.15.70:0/840770072 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb974004830 con 0x7fb9841028a0 2024-06-04T05:08:47.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.219+0000 7fb97affd700 1 -- 172.21.15.70:0/840770072 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9740053a0 con 0x7fb9841028a0 2024-06-04T05:08:47.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.219+0000 7fb98c22c700 1 -- 172.21.15.70:0/840770072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9841a8790 con 0x7fb9841028a0 2024-06-04T05:08:47.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.221+0000 7fb98c22c700 1 -- 172.21.15.70:0/840770072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb98410bb60 con 0x7fb9841028a0 2024-06-04T05:08:47.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.222+0000 7fb97affd700 1 -- 172.21.15.70:0/840770072 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fb974029060 con 0x7fb9841028a0 2024-06-04T05:08:47.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.222+0000 7fb97affd700 1 --2- 172.21.15.70:0/840770072 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9700b29f0 0x7fb9700b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:47.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.223+0000 7fb989fc8700 1 --2- 172.21.15.70:0/840770072 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9700b29f0 0x7fb9700b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:47.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.223+0000 7fb97affd700 1 -- 172.21.15.70:0/840770072 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7fb9740e3130 con 0x7fb9841028a0 2024-06-04T05:08:47.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.223+0000 7fb989fc8700 1 --2- 172.21.15.70:0/840770072 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9700b29f0 0x7fb9700b4eb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb980009fa0 tx=0x7fb980009570 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:47.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.228+0000 7fb97affd700 1 -- 172.21.15.70:0/840770072 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb974013070 con 0x7fb9841028a0 2024-06-04T05:08:47.256 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:08:47.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:47 smithi070 ceph-mon[32631]: pgmap v359: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 383 KiB/s wr, 60 op/s 2024-06-04T05:08:47.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.489+0000 7fb98c22c700 1 -- 172.21.15.70:0/840770072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9841045a0 con 0x7fb9841028a0 2024-06-04T05:08:47.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.490+0000 7fb97affd700 1 -- 172.21.15.70:0/840770072 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11931 (secure 0 0 0) 0x7fb9740aab70 con 0x7fb9841028a0 2024-06-04T05:08:47.492 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:08:47.492 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:07:27.814997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24436,24410,24440,24448,24442]},"id":1}]} 2024-06-04T05:08:47.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.492+0000 7fb98c22c700 1 -- 172.21.15.70:0/840770072 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9700b29f0 msgr2=0x7fb9700b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:47.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.492+0000 7fb98c22c700 1 --2- 172.21.15.70:0/840770072 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9700b29f0 0x7fb9700b4eb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb980009fa0 tx=0x7fb980009570 comp rx=0 tx=0).stop 2024-06-04T05:08:47.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.492+0000 7fb98c22c700 1 -- 172.21.15.70:0/840770072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9841028a0 msgr2=0x7fb984103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:47.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.492+0000 7fb98c22c700 1 --2- 172.21.15.70:0/840770072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9841028a0 0x7fb984103600 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fb974000c00 tx=0x7fb97400f510 comp rx=0 tx=0).stop 2024-06-04T05:08:47.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.492+0000 7fb98c22c700 1 -- 172.21.15.70:0/840770072 shutdown_connections 2024-06-04T05:08:47.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.492+0000 7fb98c22c700 1 --2- 172.21.15.70:0/840770072 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9700b29f0 0x7fb9700b4eb0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:47.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.492+0000 7fb98c22c700 1 --2- 172.21.15.70:0/840770072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9841085b0 0x7fb984103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:47.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.492+0000 7fb98c22c700 1 --2- 172.21.15.70:0/840770072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb984101ee0 0x7fb9841030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:47.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.492+0000 7fb98c22c700 1 --2- 172.21.15.70:0/840770072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9841028a0 0x7fb984103600 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:47.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.492+0000 7fb98c22c700 1 -- 172.21.15.70:0/840770072 >> 172.21.15.70:0/840770072 conn(0x7fb9840777e0 msgr2=0x7fb9840fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:47.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.493+0000 7fb98c22c700 1 -- 172.21.15.70:0/840770072 shutdown_connections 2024-06-04T05:08:47.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:47.493+0000 7fb98c22c700 1 -- 172.21.15.70:0/840770072 wait complete. 2024-06-04T05:08:47.496 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 34 2024-06-04T05:08:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:47 smithi112 ceph-mon[36453]: pgmap v359: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 383 KiB/s wr, 60 op/s 2024-06-04T05:08:47.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.76 MB/sec execute 460 sec latency 13153.473 ms 2024-06-04T05:08:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:47 smithi098 ceph-mon[35569]: pgmap v359: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 383 KiB/s wr, 60 op/s 2024-06-04T05:08:48.324 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id af20084 --release --await' 2024-06-04T05:08:48.324 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id af20084 --release --await 2024-06-04T05:08:48.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:48 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/840770072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:08:48.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:48 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/840770072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:08:48.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.76 MB/sec execute 461 sec latency 14153.540 ms 2024-06-04T05:08:48.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:48 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/840770072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:08:49.208 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:08:49.461 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:49 smithi070 ceph-mon[32631]: pgmap v360: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 45 KiB/s wr, 16 op/s 2024-06-04T05:08:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:49 smithi112 ceph-mon[36453]: pgmap v360: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 45 KiB/s wr, 16 op/s 2024-06-04T05:08:49.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.76 MB/sec execute 462 sec latency 15153.582 ms 2024-06-04T05:08:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:49 smithi098 ceph-mon[35569]: pgmap v360: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 45 KiB/s wr, 16 op/s 2024-06-04T05:08:50.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.381+0000 7f517d426700 1 -- 172.21.15.70:0/987574291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f517806c630 msgr2=0x7f51781103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.381+0000 7f517d426700 1 --2- 172.21.15.70:0/987574291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f517806c630 0x7f51781103d0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f517000b0a0 tx=0x7f517000b3b0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.381+0000 7f517d426700 1 -- 172.21.15.70:0/987574291 shutdown_connections 2024-06-04T05:08:50.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.381+0000 7f517d426700 1 --2- 172.21.15.70:0/987574291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f517806c630 0x7f51781103d0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.381+0000 7f517d426700 1 --2- 172.21.15.70:0/987574291 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51781095b0 0x7f5178109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.381+0000 7f517d426700 1 --2- 172.21.15.70:0/987574291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f517806bc70 0x7f517806c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.381+0000 7f517d426700 1 -- 172.21.15.70:0/987574291 >> 172.21.15.70:0/987574291 conn(0x7f51780793f0 msgr2=0x7f5178079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.381+0000 7f517d426700 1 -- 172.21.15.70:0/987574291 shutdown_connections 2024-06-04T05:08:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.381+0000 7f517d426700 1 -- 172.21.15.70:0/987574291 wait complete. 2024-06-04T05:08:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f517d426700 1 Processor -- start 2024-06-04T05:08:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f517d426700 1 -- start start 2024-06-04T05:08:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f517d426700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f517806bc70 0x7f5178102540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f517d426700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f517806c630 0x7f5178102a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f517d426700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51781095b0 0x7f5178102fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f517d426700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f51781135f0 con 0x7f51781095b0 2024-06-04T05:08:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f517d426700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5178113470 con 0x7f517806bc70 2024-06-04T05:08:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f517d426700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5178113770 con 0x7f517806c630 2024-06-04T05:08:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f51777fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51781095b0 0x7f5178102fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f51777fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51781095b0 0x7f5178102fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38130/0 (socket says 172.21.15.70:38130) 2024-06-04T05:08:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f51777fe700 1 -- 172.21.15.70:0/3932849712 learned_addr learned my addr 172.21.15.70:0/3932849712 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f5176ffd700 1 --2- 172.21.15.70:0/3932849712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f517806bc70 0x7f5178102540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f51777fe700 1 -- 172.21.15.70:0/3932849712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f517806c630 msgr2=0x7f5178102a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f51777fe700 1 --2- 172.21.15.70:0/3932849712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f517806c630 0x7f5178102a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f51777fe700 1 -- 172.21.15.70:0/3932849712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f517806bc70 msgr2=0x7f5178102540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f51777fe700 1 --2- 172.21.15.70:0/3932849712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f517806bc70 0x7f5178102540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f51777fe700 1 -- 172.21.15.70:0/3932849712 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51781a8090 con 0x7f51781095b0 2024-06-04T05:08:50.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f5176ffd700 1 --2- 172.21.15.70:0/3932849712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f517806bc70 0x7f5178102540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:08:50.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.382+0000 7f51777fe700 1 --2- 172.21.15.70:0/3932849712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51781095b0 0x7f5178102fc0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f51700062a0 tx=0x7f5170004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:50.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.383+0000 7f5174ff9700 1 -- 172.21.15.70:0/3932849712 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5170021070 con 0x7f51781095b0 2024-06-04T05:08:50.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.383+0000 7f517d426700 1 -- 172.21.15.70:0/3932849712 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51781a8320 con 0x7f51781095b0 2024-06-04T05:08:50.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.383+0000 7f5174ff9700 1 -- 172.21.15.70:0/3932849712 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5170007d00 con 0x7f51781095b0 2024-06-04T05:08:50.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.383+0000 7f517d426700 1 -- 172.21.15.70:0/3932849712 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51781a87d0 con 0x7f51781095b0 2024-06-04T05:08:50.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.383+0000 7f5174ff9700 1 -- 172.21.15.70:0/3932849712 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5170005280 con 0x7f51781095b0 2024-06-04T05:08:50.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.385+0000 7f5174ff9700 1 -- 172.21.15.70:0/3932849712 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f5170024070 con 0x7f51781095b0 2024-06-04T05:08:50.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.386+0000 7f5174ff9700 1 --2- 172.21.15.70:0/3932849712 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f51640b29f0 0x7f51640b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.386+0000 7f5176ffd700 1 --2- 172.21.15.70:0/3932849712 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f51640b29f0 0x7f51640b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.386+0000 7f5174ff9700 1 -- 172.21.15.70:0/3932849712 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f517000f020 con 0x7f51781095b0 2024-06-04T05:08:50.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.386+0000 7f5187cbbb80 1 Processor -- start 2024-06-04T05:08:50.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.386+0000 7f5187cbbb80 1 -- start start 2024-06-04T05:08:50.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.386+0000 7f5176ffd700 1 --2- 172.21.15.70:0/3932849712 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f51640b29f0 0x7f51640b4eb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5168005fd0 tx=0x7f5168005dc0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f5187cbbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1f320 0x55c36bc1f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f5187cbbb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc198a0 0x55c36bc19d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f5187cbbb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc1a260 0x55c36bc26080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f5187cbbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c36bb77d70 con 0x55c36bc1f320 2024-06-04T05:08:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f5187cbbb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c36bb67910 con 0x55c36bc1a260 2024-06-04T05:08:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f5187cbbb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c36bb6b950 con 0x55c36bc198a0 2024-06-04T05:08:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f5176ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1f320 0x55c36bc1f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f5176ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1f320 0x55c36bc1f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38132/0 (socket says 172.21.15.70:38132) 2024-06-04T05:08:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f5176ffd700 1 -- 172.21.15.70:0/4137871073 learned_addr learned my addr 172.21.15.70:0/4137871073 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f516ffff700 1 --2- 172.21.15.70:0/4137871073 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc198a0 0x55c36bc19d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f51777fe700 1 --2- 172.21.15.70:0/4137871073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc1a260 0x55c36bc26080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f5176ffd700 1 -- 172.21.15.70:0/4137871073 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc198a0 msgr2=0x55c36bc19d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f5176ffd700 1 --2- 172.21.15.70:0/4137871073 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc198a0 0x55c36bc19d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f5176ffd700 1 -- 172.21.15.70:0/4137871073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc1a260 msgr2=0x55c36bc26080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f5176ffd700 1 --2- 172.21.15.70:0/4137871073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc1a260 0x55c36bc26080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f5176ffd700 1 -- 172.21.15.70:0/4137871073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c36bb10fb0 con 0x55c36bc1f320 2024-06-04T05:08:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f51777fe700 1 --2- 172.21.15.70:0/4137871073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc1a260 0x55c36bc26080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f5176ffd700 1 --2- 172.21.15.70:0/4137871073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1f320 0x55c36bc1f700 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f516800db90 tx=0x7f5168010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1357fd7f9e4ed57 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f516d7fa700 1 -- 172.21.15.70:0/4137871073 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5168020070 con 0x55c36bc1f320 2024-06-04T05:08:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f516d7fa700 1 -- 172.21.15.70:0/4137871073 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f516800fd40 con 0x55c36bc1f320 2024-06-04T05:08:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.387+0000 7f516d7fa700 1 -- 172.21.15.70:0/4137871073 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516801cda0 con 0x55c36bc1f320 2024-06-04T05:08:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.388+0000 7f5187cbbb80 1 -- 172.21.15.70:0/4137871073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1f320 msgr2=0x55c36bc1f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.388+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/4137871073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1f320 0x55c36bc1f700 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f516800db90 tx=0x7f5168010000 comp rx=0 tx=0).stop 2024-06-04T05:08:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.388+0000 7f5187cbbb80 1 -- 172.21.15.70:0/4137871073 shutdown_connections 2024-06-04T05:08:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.388+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/4137871073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc1a260 0x55c36bc26080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.388+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/4137871073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1f320 0x55c36bc1f700 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.388+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/4137871073 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc198a0 0x55c36bc19d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.388+0000 7f5187cbbb80 1 -- 172.21.15.70:0/4137871073 >> 172.21.15.70:0/4137871073 conn(0x55c36bb45240 msgr2=0x55c36bc143e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.388+0000 7f5187cbbb80 1 -- 172.21.15.70:0/4137871073 shutdown_connections 2024-06-04T05:08:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.388+0000 7f5187cbbb80 1 -- 172.21.15.70:0/4137871073 wait complete. 2024-06-04T05:08:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.389+0000 7f5187cbbb80 1 -- start start 2024-06-04T05:08:50.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.389+0000 7f5187cbbb80 1 Processor -- start 2024-06-04T05:08:50.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.389+0000 7f5187cbbb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc198a0 0x55c36bc25d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.389+0000 7f5187cbbb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc1a260 0x55c36bc23000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.389+0000 7f5187cbbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1f320 0x55c36bc23540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.389+0000 7f5187cbbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c36ba6dfa0 con 0x55c36bc1f320 2024-06-04T05:08:50.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.389+0000 7f5187cbbb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c36bacfda0 con 0x55c36bc198a0 2024-06-04T05:08:50.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.389+0000 7f5187cbbb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c36bc266e0 con 0x55c36bc1a260 2024-06-04T05:08:50.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.389+0000 7f5187cbbb80 10 client.0 _ll_register_callbacks cb 0x55c36bb02950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:50.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.389+0000 7f51777fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1f320 0x55c36bc23540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.389+0000 7f51777fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1f320 0x55c36bc23540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38146/0 (socket says 172.21.15.70:38146) 2024-06-04T05:08:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.389+0000 7f51777fe700 1 -- 172.21.15.70:0/4255438617 learned_addr learned my addr 172.21.15.70:0/4255438617 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.389+0000 7f5176ffd700 1 --2- 172.21.15.70:0/4255438617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc198a0 0x55c36bc25d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f516ffff700 1 --2- 172.21.15.70:0/4255438617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc1a260 0x55c36bc23000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f51777fe700 1 -- 172.21.15.70:0/4255438617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc1a260 msgr2=0x55c36bc23000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f51777fe700 1 --2- 172.21.15.70:0/4255438617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc1a260 0x55c36bc23000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f51777fe700 1 -- 172.21.15.70:0/4255438617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc198a0 msgr2=0x55c36bc25d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f51777fe700 1 --2- 172.21.15.70:0/4255438617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc198a0 0x55c36bc25d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f51777fe700 1 -- 172.21.15.70:0/4255438617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c36bab78d0 con 0x55c36bc1f320 2024-06-04T05:08:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f51777fe700 1 --2- 172.21.15.70:0/4255438617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1f320 0x55c36bc23540 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f51700a8390 tx=0x7f51700a7630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5690d0aea0a6fd55 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f515e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f515e7fc700 1 -- 172.21.15.70:0/4255438617 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51700c5060 con 0x55c36bc1f320 2024-06-04T05:08:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f515e7fc700 1 -- 172.21.15.70:0/4255438617 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f51700ab590 con 0x55c36bc1f320 2024-06-04T05:08:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f5187cbbb80 1 -- 172.21.15.70:0/4255438617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f516800fd40 con 0x55c36bc1f320 2024-06-04T05:08:50.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f515e7fc700 1 -- 172.21.15.70:0/4255438617 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f51700af4d0 con 0x55c36bc1f320 2024-06-04T05:08:50.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f5187cbbb80 10 client.18534 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:50.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f5187cbbb80 1 -- 172.21.15.70:0/4255438617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55c36bb21760 con 0x55c36bc1f320 2024-06-04T05:08:50.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f515e7fc700 1 -- 172.21.15.70:0/4255438617 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f516800fd40 con 0x55c36bc1f320 2024-06-04T05:08:50.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f515e7fc700 1 -- 172.21.15.70:0/4255438617 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f5170011070 con 0x55c36bc1f320 2024-06-04T05:08:50.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f5187cbbb80 10 client.18534 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:50.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.390+0000 7f5187cbbb80 10 client.18534 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:08:50.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.391+0000 7f5187cbbb80 20 client.18534 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:50.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.391+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/4255438617 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55c36bc330f0 0x55c36bc35550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.391+0000 7f5187cbbb80 4 client.18534 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:08:50.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.391+0000 7f5187cbbb80 1 -- 172.21.15.70:0/4255438617 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c36bb79cc0 con 0x55c36bc330f0 2024-06-04T05:08:50.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.391+0000 7f5176ffd700 1 --2- 172.21.15.70:0/4255438617 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55c36bc330f0 0x55c36bc35550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.391+0000 7f5176ffd700 1 --2- 172.21.15.70:0/4255438617 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55c36bc330f0 0x55c36bc35550 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9e5d7b789d3709f7 server_cookie=7401b5cf68897bb8 in_seq=0 out_seq=0 2024-06-04T05:08:50.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.391+0000 7f515e7fc700 10 client.18534 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:08:50.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.392+0000 7f515e7fc700 1 -- 172.21.15.70:0/4255438617 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c36bb79cc0 con 0x55c36bc330f0 2024-06-04T05:08:50.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.392+0000 7f515e7fc700 10 client.18534 handle_command_reply: tid=0 2024-06-04T05:08:50.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.392+0000 7f5187cbbb80 1 client.18534 shutdown 2024-06-04T05:08:50.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.392+0000 7f5187cbbb80 1 -- 172.21.15.70:0/4255438617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1f320 msgr2=0x55c36bc23540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.392+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/4255438617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1f320 0x55c36bc23540 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f51700a8390 tx=0x7f51700a7630 comp rx=0 tx=0).stop 2024-06-04T05:08:50.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.392+0000 7f5187cbbb80 1 -- 172.21.15.70:0/4255438617 shutdown_connections 2024-06-04T05:08:50.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.392+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/4255438617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1f320 0x55c36bc23540 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.392+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/4255438617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc198a0 0x55c36bc25d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.392+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/4255438617 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55c36bc330f0 0x55c36bc35550 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.392+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/4255438617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc1a260 0x55c36bc23000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.392+0000 7f5187cbbb80 1 -- 172.21.15.70:0/4255438617 >> 172.21.15.70:0/4255438617 conn(0x55c36bb45240 msgr2=0x55c36bc13af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:50.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.392+0000 7f515e7fc700 0 client.18534 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:08:50.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.392+0000 7f5187cbbb80 1 -- 172.21.15.70:0/4255438617 shutdown_connections 2024-06-04T05:08:50.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.392+0000 7f5187cbbb80 1 -- 172.21.15.70:0/4255438617 wait complete. 2024-06-04T05:08:50.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.392+0000 7f5187cbbb80 20 client.18534 trim_cache size 0 max 16384 2024-06-04T05:08:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.407+0000 7f5187cbbb80 1 Processor -- start 2024-06-04T05:08:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.407+0000 7f5187cbbb80 1 -- start start 2024-06-04T05:08:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f5187cbbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc198a0 0x55c36bc22180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f5187cbbb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc1a260 0x55c36bc14270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f5187cbbb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc1f320 0x55c36bc18890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f5187cbbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f51781b0060 con 0x55c36bc198a0 2024-06-04T05:08:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f5187cbbb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51781b1c00 con 0x55c36bc1f320 2024-06-04T05:08:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f5187cbbb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f51781b0730 con 0x55c36bc1a260 2024-06-04T05:08:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f5176ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc198a0 0x55c36bc22180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f5176ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc198a0 0x55c36bc22180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38152/0 (socket says 172.21.15.70:38152) 2024-06-04T05:08:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f5176ffd700 1 -- 172.21.15.70:0/2126512065 learned_addr learned my addr 172.21.15.70:0/2126512065 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f5176ffd700 1 -- 172.21.15.70:0/2126512065 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc1a260 msgr2=0x55c36bc14270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f5176ffd700 1 --2- 172.21.15.70:0/2126512065 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc1a260 0x55c36bc14270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f5176ffd700 1 -- 172.21.15.70:0/2126512065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc1f320 msgr2=0x55c36bc18890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f5176ffd700 1 --2- 172.21.15.70:0/2126512065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc1f320 0x55c36bc18890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f5176ffd700 1 -- 172.21.15.70:0/2126512065 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c36bc25940 con 0x55c36bc198a0 2024-06-04T05:08:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f5176ffd700 1 --2- 172.21.15.70:0/2126512065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc198a0 0x55c36bc22180 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f5168005950 tx=0x7f516801c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c86c5643605313 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f516cff9700 1 -- 172.21.15.70:0/2126512065 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5168013040 con 0x55c36bc198a0 2024-06-04T05:08:50.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f516cff9700 1 -- 172.21.15.70:0/2126512065 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f516800f6b0 con 0x55c36bc198a0 2024-06-04T05:08:50.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.408+0000 7f516cff9700 1 -- 172.21.15.70:0/2126512065 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5168020040 con 0x55c36bc198a0 2024-06-04T05:08:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.409+0000 7f5187cbbb80 1 -- 172.21.15.70:0/2126512065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc198a0 msgr2=0x55c36bc22180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.409+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/2126512065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc198a0 0x55c36bc22180 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f5168005950 tx=0x7f516801c710 comp rx=0 tx=0).stop 2024-06-04T05:08:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.409+0000 7f5187cbbb80 1 -- 172.21.15.70:0/2126512065 shutdown_connections 2024-06-04T05:08:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.409+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/2126512065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc1f320 0x55c36bc18890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.409+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/2126512065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc198a0 0x55c36bc22180 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.409+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/2126512065 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc1a260 0x55c36bc14270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.409+0000 7f5187cbbb80 1 -- 172.21.15.70:0/2126512065 >> 172.21.15.70:0/2126512065 conn(0x55c36bb45240 msgr2=0x55c36bc1e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.409+0000 7f5187cbbb80 1 -- 172.21.15.70:0/2126512065 shutdown_connections 2024-06-04T05:08:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.409+0000 7f5187cbbb80 1 -- 172.21.15.70:0/2126512065 wait complete. 2024-06-04T05:08:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.409+0000 7f5187cbbb80 1 -- start start 2024-06-04T05:08:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.409+0000 7f5187cbbb80 1 Processor -- start 2024-06-04T05:08:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f5187cbbb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc198a0 0x55c36bc2b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f5187cbbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1a260 0x55c36bc2be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f5187cbbb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc1f320 0x55c36bc268f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f5187cbbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f51781b1500 con 0x55c36bc1a260 2024-06-04T05:08:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f5187cbbb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51781b0e00 con 0x55c36bc198a0 2024-06-04T05:08:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f5187cbbb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f51781ae020 con 0x55c36bc1f320 2024-06-04T05:08:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f5187cbbb80 10 client.0 _ll_register_callbacks cb 0x55c36bb02950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f516ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1a260 0x55c36bc2be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f516ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1a260 0x55c36bc2be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38164/0 (socket says 172.21.15.70:38164) 2024-06-04T05:08:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f516ffff700 1 -- 172.21.15.70:0/3252878950 learned_addr learned my addr 172.21.15.70:0/3252878950 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f5176ffd700 1 --2- 172.21.15.70:0/3252878950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc198a0 0x55c36bc2b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f51777fe700 1 --2- 172.21.15.70:0/3252878950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc1f320 0x55c36bc268f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f516ffff700 1 -- 172.21.15.70:0/3252878950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc1f320 msgr2=0x55c36bc268f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f516ffff700 1 --2- 172.21.15.70:0/3252878950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc1f320 0x55c36bc268f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f516ffff700 1 -- 172.21.15.70:0/3252878950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc198a0 msgr2=0x55c36bc2b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:50.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f516ffff700 1 --2- 172.21.15.70:0/3252878950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc198a0 0x55c36bc2b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f516ffff700 1 -- 172.21.15.70:0/3252878950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c36bc3b960 con 0x55c36bc1a260 2024-06-04T05:08:50.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f516ffff700 1 --2- 172.21.15.70:0/3252878950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1a260 0x55c36bc2be80 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x55c36bc245d0 tx=0x7f5160010520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c589beb196694eec server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:50.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f515e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:50.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f515e7fc700 1 -- 172.21.15.70:0/3252878950 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5160018070 con 0x55c36bc1a260 2024-06-04T05:08:50.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f515e7fc700 1 -- 172.21.15.70:0/3252878950 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f51600143f0 con 0x55c36bc1a260 2024-06-04T05:08:50.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.410+0000 7f5187cbbb80 1 -- 172.21.15.70:0/3252878950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f516800f6b0 con 0x55c36bc1a260 2024-06-04T05:08:50.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.411+0000 7f515e7fc700 1 -- 172.21.15.70:0/3252878950 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f5160024050 con 0x55c36bc1a260 2024-06-04T05:08:50.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.411+0000 7f5187cbbb80 10 client.18540 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:50.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.411+0000 7f5187cbbb80 1 -- 172.21.15.70:0/3252878950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55c36bc9e310 con 0x55c36bc1a260 2024-06-04T05:08:50.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.411+0000 7f515e7fc700 1 -- 172.21.15.70:0/3252878950 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f516800f6b0 con 0x55c36bc1a260 2024-06-04T05:08:50.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.411+0000 7f515e7fc700 1 -- 172.21.15.70:0/3252878950 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f516001c640 con 0x55c36bc1a260 2024-06-04T05:08:50.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.411+0000 7f5187cbbb80 10 client.18540 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.411+0000 7f5187cbbb80 10 client.18540 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:08:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.411+0000 7f5187cbbb80 20 client.18540 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.411+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/3252878950 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55c36bc9ea80 0x55c36bca0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.411+0000 7f5187cbbb80 4 client.18540 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:08:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.411+0000 7f5187cbbb80 1 -- 172.21.15.70:0/3252878950 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c36bb734a0 con 0x55c36bc9ea80 2024-06-04T05:08:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.411+0000 7f5176ffd700 1 --2- 172.21.15.70:0/3252878950 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55c36bc9ea80 0x55c36bca0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.412+0000 7f515e7fc700 10 client.18540 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:08:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.412+0000 7f5176ffd700 1 --2- 172.21.15.70:0/3252878950 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55c36bc9ea80 0x55c36bca0e70 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=972a202bc0a63752 server_cookie=c61f678b57136af in_seq=0 out_seq=0 2024-06-04T05:08:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.412+0000 7f515e7fc700 1 -- 172.21.15.70:0/3252878950 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c36bb734a0 con 0x55c36bc9ea80 2024-06-04T05:08:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.412+0000 7f515e7fc700 10 client.18540 handle_command_reply: tid=0 2024-06-04T05:08:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.412+0000 7f5187cbbb80 1 client.18540 shutdown 2024-06-04T05:08:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.412+0000 7f5187cbbb80 1 -- 172.21.15.70:0/3252878950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1a260 msgr2=0x55c36bc2be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.412+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/3252878950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1a260 0x55c36bc2be80 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x55c36bc245d0 tx=0x7f5160010520 comp rx=0 tx=0).stop 2024-06-04T05:08:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.412+0000 7f5187cbbb80 1 -- 172.21.15.70:0/3252878950 shutdown_connections 2024-06-04T05:08:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.412+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/3252878950 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55c36bc9ea80 0x55c36bca0e70 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.412+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/3252878950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c36bc1f320 0x55c36bc268f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.412+0000 7f515e7fc700 0 client.18540 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:08:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.412+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/3252878950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c36bc198a0 0x55c36bc2b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.412+0000 7f5187cbbb80 1 --2- 172.21.15.70:0/3252878950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c36bc1a260 0x55c36bc2be80 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.412+0000 7f5187cbbb80 1 -- 172.21.15.70:0/3252878950 >> 172.21.15.70:0/3252878950 conn(0x55c36bb45240 msgr2=0x55c36bc1cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.412+0000 7f5187cbbb80 1 -- 172.21.15.70:0/3252878950 shutdown_connections 2024-06-04T05:08:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.412+0000 7f5187cbbb80 1 -- 172.21.15.70:0/3252878950 wait complete. 2024-06-04T05:08:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.412+0000 7f5187cbbb80 20 client.18540 trim_cache size 0 max 16384 2024-06-04T05:08:50.419 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:08:50.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.413+0000 7f517d426700 1 -- 172.21.15.70:0/3932849712 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f51640b29f0 msgr2=0x7f51640b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.413+0000 7f517d426700 1 --2- 172.21.15.70:0/3932849712 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f51640b29f0 0x7f51640b4eb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5168005fd0 tx=0x7f5168005dc0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.413+0000 7f517d426700 1 -- 172.21.15.70:0/3932849712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51781095b0 msgr2=0x7f5178102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.413+0000 7f517d426700 1 --2- 172.21.15.70:0/3932849712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51781095b0 0x7f5178102fc0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f51700062a0 tx=0x7f5170004ed0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.413+0000 7f517d426700 1 -- 172.21.15.70:0/3932849712 shutdown_connections 2024-06-04T05:08:50.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.413+0000 7f517d426700 1 --2- 172.21.15.70:0/3932849712 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f51640b29f0 0x7f51640b4eb0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.413+0000 7f517d426700 1 --2- 172.21.15.70:0/3932849712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51781095b0 0x7f5178102fc0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.413+0000 7f517d426700 1 --2- 172.21.15.70:0/3932849712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f517806bc70 0x7f5178102540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.413+0000 7f517d426700 1 --2- 172.21.15.70:0/3932849712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f517806c630 0x7f5178102a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.413+0000 7f517d426700 1 -- 172.21.15.70:0/3932849712 >> 172.21.15.70:0/3932849712 conn(0x7f51780793f0 msgr2=0x7f5178101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:50.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.413+0000 7f517d426700 1 -- 172.21.15.70:0/3932849712 shutdown_connections 2024-06-04T05:08:50.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:50.413+0000 7f517d426700 1 -- 172.21.15.70:0/3932849712 wait complete. 2024-06-04T05:08:50.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.75 MB/sec execute 463 sec latency 16153.627 ms 2024-06-04T05:08:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.906+0000 7fb0d682a640 1 -- 172.21.15.70:0/4122038339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0d0107e30 msgr2=0x7fb0d010f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.906+0000 7fb0d682a640 1 --2- 172.21.15.70:0/4122038339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0d0107e30 0x7fb0d010f9b0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8009ca0 tx=0x7fb0c802a8c0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.906+0000 7fb0d682a640 1 -- 172.21.15.70:0/4122038339 shutdown_connections 2024-06-04T05:08:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.906+0000 7fb0d682a640 1 --2- 172.21.15.70:0/4122038339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d0101b70 0x7fb0d0107700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.906+0000 7fb0d682a640 1 --2- 172.21.15.70:0/4122038339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0d0107e30 0x7fb0d010f9b0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.906+0000 7fb0d682a640 1 --2- 172.21.15.70:0/4122038339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0d01011c0 0x7fb0d01015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.906+0000 7fb0d682a640 1 -- 172.21.15.70:0/4122038339 >> 172.21.15.70:0/4122038339 conn(0x7fb0d0077eb0 msgr2=0x7fb0d00ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.907+0000 7fb0d682a640 1 -- 172.21.15.70:0/4122038339 shutdown_connections 2024-06-04T05:08:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.907+0000 7fb0d682a640 1 -- 172.21.15.70:0/4122038339 wait complete. 2024-06-04T05:08:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.907+0000 7fb0d682a640 1 Processor -- start 2024-06-04T05:08:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.907+0000 7fb0d682a640 1 -- start start 2024-06-04T05:08:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.907+0000 7fb0d682a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0d01011c0 0x7fb0d0199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.907+0000 7fb0d682a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0d0107e30 0x7fb0d01995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.907+0000 7fb0d682a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d019d940 0x7fb0d019ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.907+0000 7fb0d682a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d0112c40 con 0x7fb0d0107e30 2024-06-04T05:08:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.907+0000 7fb0d682a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d0112ac0 con 0x7fb0d019d940 2024-06-04T05:08:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.907+0000 7fb0d682a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d0112dc0 con 0x7fb0d01011c0 2024-06-04T05:08:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.908+0000 7fb0cf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0d0107e30 0x7fb0d01995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.908+0000 7fb0cf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0d0107e30 0x7fb0d01995c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38180/0 (socket says 172.21.15.70:38180) 2024-06-04T05:08:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.908+0000 7fb0cf7fe640 1 -- 172.21.15.70:0/3568903270 learned_addr learned my addr 172.21.15.70:0/3568903270 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.908+0000 7fb0cffff640 1 --2- 172.21.15.70:0/3568903270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0d01011c0 0x7fb0d0199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.908+0000 7fb0cf7fe640 1 -- 172.21.15.70:0/3568903270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0d01011c0 msgr2=0x7fb0d0199080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.908+0000 7fb0cf7fe640 1 --2- 172.21.15.70:0/3568903270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0d01011c0 0x7fb0d0199080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.908+0000 7fb0cf7fe640 1 -- 172.21.15.70:0/3568903270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d019d940 msgr2=0x7fb0d019ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.908+0000 7fb0cf7fe640 1 --2- 172.21.15.70:0/3568903270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d019d940 0x7fb0d019ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.908+0000 7fb0cf7fe640 1 -- 172.21.15.70:0/3568903270 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d019e430 con 0x7fb0d0107e30 2024-06-04T05:08:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.908+0000 7fb0cf7fe640 1 --2- 172.21.15.70:0/3568903270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0d0107e30 0x7fb0d01995c0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fb0c400d740 tx=0x7fb0c400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.908+0000 7fb0cd7fa640 1 -- 172.21.15.70:0/3568903270 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4017070 con 0x7fb0d0107e30 2024-06-04T05:08:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.908+0000 7fb0d682a640 1 -- 172.21.15.70:0/3568903270 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d019e720 con 0x7fb0d0107e30 2024-06-04T05:08:50.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.908+0000 7fb0cd7fa640 1 -- 172.21.15.70:0/3568903270 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb0c400de50 con 0x7fb0d0107e30 2024-06-04T05:08:50.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.908+0000 7fb0cd7fa640 1 -- 172.21.15.70:0/3568903270 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c400f620 con 0x7fb0d0107e30 2024-06-04T05:08:50.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.908+0000 7fb0d682a640 1 -- 172.21.15.70:0/3568903270 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d01a5f60 con 0x7fb0d0107e30 2024-06-04T05:08:50.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.910+0000 7fb0d9624740 1 Processor -- start 2024-06-04T05:08:50.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0d9624740 1 -- start start 2024-06-04T05:08:50.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0cd7fa640 1 -- 172.21.15.70:0/3568903270 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fb0c401a070 con 0x7fb0d0107e30 2024-06-04T05:08:50.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0d9624740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0d9624740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85c6140 0x562ee85d04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0d9624740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85d0a00 0x562ee85d2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0d9624740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562ee8537c30 con 0x562ee85c5820 2024-06-04T05:08:50.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0d9624740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562ee85277d0 con 0x562ee85c6140 2024-06-04T05:08:50.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0d9624740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562ee852b810 con 0x562ee85d0a00 2024-06-04T05:08:50.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0d4dc7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0d4dc7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85c5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38196/0 (socket says 172.21.15.70:38196) 2024-06-04T05:08:50.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0d4dc7640 1 -- 172.21.15.70:0/2357584435 learned_addr learned my addr 172.21.15.70:0/2357584435 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:50.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0cf7fe640 1 --2- 172.21.15.70:0/2357584435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85d0a00 0x562ee85d2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0cffff640 1 --2- 172.21.15.70:0/2357584435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85c6140 0x562ee85d04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0d4dc7640 1 -- 172.21.15.70:0/2357584435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85d0a00 msgr2=0x562ee85d2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:50.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0d4dc7640 1 --2- 172.21.15.70:0/2357584435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85d0a00 0x562ee85d2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0d4dc7640 1 -- 172.21.15.70:0/2357584435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85c6140 msgr2=0x562ee85d04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:50.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0d4dc7640 1 --2- 172.21.15.70:0/2357584435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85c6140 0x562ee85d04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0d4dc7640 1 -- 172.21.15.70:0/2357584435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ee823c210 con 0x562ee85c5820 2024-06-04T05:08:50.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0cffff640 1 --2- 172.21.15.70:0/2357584435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85c6140 0x562ee85d04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:08:50.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0d4dc7640 1 --2- 172.21.15.70:0/2357584435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85c5c00 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8009dc0 tx=0x7fb0c8031450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2927ae3b33a348d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:50.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.911+0000 7fb0cd7fa640 1 --2- 172.21.15.70:0/3568903270 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb0b80b2a10 0x7fb0b80b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.912+0000 7fb0b67fc640 1 -- 172.21.15.70:0/2357584435 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8031980 con 0x562ee85c5820 2024-06-04T05:08:50.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.912+0000 7fb0b67fc640 1 -- 172.21.15.70:0/2357584435 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb0c803f070 con 0x562ee85c5820 2024-06-04T05:08:50.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.912+0000 7fb0b67fc640 1 -- 172.21.15.70:0/2357584435 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8004350 con 0x562ee85c5820 2024-06-04T05:08:50.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.912+0000 7fb0cffff640 1 --2- 172.21.15.70:0/3568903270 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb0b80b2a10 0x7fb0b80b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.912+0000 7fb0cd7fa640 1 -- 172.21.15.70:0/3568903270 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7fb0c40e20b0 con 0x7fb0d0107e30 2024-06-04T05:08:50.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.912+0000 7fb0d9624740 1 -- 172.21.15.70:0/2357584435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 msgr2=0x562ee85c5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.912+0000 7fb0cffff640 1 --2- 172.21.15.70:0/3568903270 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb0b80b2a10 0x7fb0b80b4ed0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x562ee85c6dc0 tx=0x7fb0c0005f50 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:50.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.912+0000 7fb0d9624740 1 --2- 172.21.15.70:0/2357584435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85c5c00 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8009dc0 tx=0x7fb0c8031450 comp rx=0 tx=0).stop 2024-06-04T05:08:50.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.912+0000 7fb0d9624740 1 -- 172.21.15.70:0/2357584435 shutdown_connections 2024-06-04T05:08:50.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.912+0000 7fb0d9624740 1 --2- 172.21.15.70:0/2357584435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85c6140 0x562ee85d04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.912+0000 7fb0d9624740 1 --2- 172.21.15.70:0/2357584435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85d0a00 0x562ee85d2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.912+0000 7fb0d9624740 1 --2- 172.21.15.70:0/2357584435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85c5c00 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.912+0000 7fb0d9624740 1 -- 172.21.15.70:0/2357584435 >> 172.21.15.70:0/2357584435 conn(0x562ee85c1d30 msgr2=0x562ee85c2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:50.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.912+0000 7fb0d9624740 1 -- 172.21.15.70:0/2357584435 shutdown_connections 2024-06-04T05:08:50.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.912+0000 7fb0d9624740 1 -- 172.21.15.70:0/2357584435 wait complete. 2024-06-04T05:08:50.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.913+0000 7fb0d9624740 1 -- start start 2024-06-04T05:08:50.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.913+0000 7fb0d9624740 1 Processor -- start 2024-06-04T05:08:50.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.913+0000 7fb0d9624740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85cda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.913+0000 7fb0d9624740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85c6140 0x562ee85cdf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.913+0000 7fb0d9624740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85d0a00 0x562ee85ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.913+0000 7fb0d9624740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562ee820d790 con 0x562ee85c5820 2024-06-04T05:08:50.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.913+0000 7fb0d9624740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562ee84e3890 con 0x562ee85d0a00 2024-06-04T05:08:50.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.913+0000 7fb0d9624740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562ee85d35d0 con 0x562ee85c6140 2024-06-04T05:08:50.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.913+0000 7fb0d9624740 10 client.0 _ll_register_callbacks cb 0x562ee823ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:50.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.913+0000 7fb0cffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.913+0000 7fb0cffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85cda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38198/0 (socket says 172.21.15.70:38198) 2024-06-04T05:08:50.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.913+0000 7fb0cffff640 1 -- 172.21.15.70:0/4170710706 learned_addr learned my addr 172.21.15.70:0/4170710706 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:50.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.913+0000 7fb0cffff640 1 -- 172.21.15.70:0/4170710706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85c6140 msgr2=0x562ee85cdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:50.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.913+0000 7fb0cffff640 1 --2- 172.21.15.70:0/4170710706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85c6140 0x562ee85cdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.913+0000 7fb0cffff640 1 -- 172.21.15.70:0/4170710706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85d0a00 msgr2=0x562ee85ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:50.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.913+0000 7fb0cffff640 1 --2- 172.21.15.70:0/4170710706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85d0a00 0x562ee85ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.914+0000 7fb0cffff640 1 -- 172.21.15.70:0/4170710706 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ee8308620 con 0x562ee85c5820 2024-06-04T05:08:50.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.914+0000 7fb0cffff640 1 --2- 172.21.15.70:0/4170710706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85cda40 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fb0c00124f0 tx=0x7fb0c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a21df27e338162 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:50.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.914+0000 7fb0af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:50.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.914+0000 7fb0af7fe640 1 -- 172.21.15.70:0/4170710706 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0023070 con 0x562ee85c5820 2024-06-04T05:08:50.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.914+0000 7fb0af7fe640 1 -- 172.21.15.70:0/4170710706 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7fb0c001f860 con 0x562ee85c5820 2024-06-04T05:08:50.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.914+0000 7fb0d9624740 1 -- 172.21.15.70:0/4170710706 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0c803f070 con 0x562ee85c5820 2024-06-04T05:08:50.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.914+0000 7fb0af7fe640 1 -- 172.21.15.70:0/4170710706 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb0c002f050 con 0x562ee85c5820 2024-06-04T05:08:50.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.914+0000 7fb0d9624740 10 client.18552 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:50.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.914+0000 7fb0d9624740 1 -- 172.21.15.70:0/4170710706 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562ee85df3a0 con 0x562ee85c5820 2024-06-04T05:08:50.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.914+0000 7fb0af7fe640 1 -- 172.21.15.70:0/4170710706 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb0c803f070 con 0x562ee85c5820 2024-06-04T05:08:50.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.915+0000 7fb0af7fe640 1 -- 172.21.15.70:0/4170710706 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7fb0c000f040 con 0x562ee85c5820 2024-06-04T05:08:50.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.915+0000 7fb0d9624740 10 client.18552 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:50.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.915+0000 7fb0d9624740 10 client.18552 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:08:50.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.915+0000 7fb0d9624740 20 client.18552 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:50.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.915+0000 7fb0d9624740 1 --2- 172.21.15.70:0/4170710706 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562ee85e08d0 0x562ee85e2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.915+0000 7fb0d9624740 4 client.18552 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:08:50.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.915+0000 7fb0d9624740 1 -- 172.21.15.70:0/4170710706 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ee8539b80 con 0x562ee85e08d0 2024-06-04T05:08:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.915+0000 7fb0cf7fe640 1 --2- 172.21.15.70:0/4170710706 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562ee85e08d0 0x562ee85e2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.916+0000 7fb0af7fe640 10 client.18552 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.916+0000 7fb0cf7fe640 1 --2- 172.21.15.70:0/4170710706 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562ee85e08d0 0x562ee85e2d60 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4dba1fb2b63cf4 server_cookie=1905d83fb9599dd2 in_seq=0 out_seq=0 2024-06-04T05:08:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.916+0000 7fb0af7fe640 1 -- 172.21.15.70:0/4170710706 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562ee8539b80 con 0x562ee85e08d0 2024-06-04T05:08:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.916+0000 7fb0af7fe640 10 client.18552 handle_command_reply: tid=0 2024-06-04T05:08:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.916+0000 7fb0d9624740 1 client.18552 shutdown 2024-06-04T05:08:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.916+0000 7fb0d9624740 1 -- 172.21.15.70:0/4170710706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 msgr2=0x562ee85cda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.916+0000 7fb0d9624740 1 --2- 172.21.15.70:0/4170710706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85cda40 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fb0c00124f0 tx=0x7fb0c0004730 comp rx=0 tx=0).stop 2024-06-04T05:08:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.916+0000 7fb0d9624740 1 -- 172.21.15.70:0/4170710706 shutdown_connections 2024-06-04T05:08:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.916+0000 7fb0d9624740 1 --2- 172.21.15.70:0/4170710706 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562ee85e08d0 0x562ee85e2d60 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.917+0000 7fb0d9624740 1 --2- 172.21.15.70:0/4170710706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85d0a00 0x562ee85ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.917+0000 7fb0d9624740 1 --2- 172.21.15.70:0/4170710706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85c6140 0x562ee85cdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.917+0000 7fb0af7fe640 0 client.18552 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.917+0000 7fb0d9624740 1 --2- 172.21.15.70:0/4170710706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85cda40 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.917+0000 7fb0d9624740 1 -- 172.21.15.70:0/4170710706 >> 172.21.15.70:0/4170710706 conn(0x562ee85c1d30 msgr2=0x562ee85c0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.917+0000 7fb0d9624740 1 -- 172.21.15.70:0/4170710706 shutdown_connections 2024-06-04T05:08:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.917+0000 7fb0d9624740 1 -- 172.21.15.70:0/4170710706 wait complete. 2024-06-04T05:08:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.917+0000 7fb0d9624740 20 client.18552 trim_cache size 0 max 16384 2024-06-04T05:08:50.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.931+0000 7fb0d9624740 1 Processor -- start 2024-06-04T05:08:50.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.931+0000 7fb0d9624740 1 -- start start 2024-06-04T05:08:50.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0d9624740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85c5820 0x562ee85c1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0d9624740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85c6140 0x562ee85c7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0d9624740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85d0a00 0x562ee85e8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0d9624740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d01ae6b0 con 0x562ee85d0a00 2024-06-04T05:08:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0d9624740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d01b0250 con 0x562ee85c5820 2024-06-04T05:08:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0d9624740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d01aed80 con 0x562ee85c6140 2024-06-04T05:08:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0d4dc7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85d0a00 0x562ee85e8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0d4dc7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85d0a00 0x562ee85e8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38200/0 (socket says 172.21.15.70:38200) 2024-06-04T05:08:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0d4dc7640 1 -- 172.21.15.70:0/4057604520 learned_addr learned my addr 172.21.15.70:0/4057604520 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0cf7fe640 1 --2- 172.21.15.70:0/4057604520 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85c6140 0x562ee85c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0d4dc7640 1 -- 172.21.15.70:0/4057604520 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85c6140 msgr2=0x562ee85c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0cffff640 1 --2- 172.21.15.70:0/4057604520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85c5820 0x562ee85c1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0d4dc7640 1 --2- 172.21.15.70:0/4057604520 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85c6140 0x562ee85c7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0d4dc7640 1 -- 172.21.15.70:0/4057604520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85c5820 msgr2=0x562ee85c1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0d4dc7640 1 --2- 172.21.15.70:0/4057604520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85c5820 0x562ee85c1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0d4dc7640 1 -- 172.21.15.70:0/4057604520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ee85d3160 con 0x562ee85d0a00 2024-06-04T05:08:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0cf7fe640 1 --2- 172.21.15.70:0/4057604520 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85c6140 0x562ee85c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:08:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0cffff640 1 --2- 172.21.15.70:0/4057604520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85c5820 0x562ee85c1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:08:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0d4dc7640 1 --2- 172.21.15.70:0/4057604520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85d0a00 0x562ee85e8630 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8009d90 tx=0x7fb0c8031620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90fe584090b0e46a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0b5ffb640 1 -- 172.21.15.70:0/4057604520 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8031b80 con 0x562ee85d0a00 2024-06-04T05:08:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0b5ffb640 1 -- 172.21.15.70:0/4057604520 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb0c803f050 con 0x562ee85d0a00 2024-06-04T05:08:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.932+0000 7fb0b5ffb640 1 -- 172.21.15.70:0/4057604520 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c80043a0 con 0x562ee85d0a00 2024-06-04T05:08:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.933+0000 7fb0d9624740 1 -- 172.21.15.70:0/4057604520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85d0a00 msgr2=0x562ee85e8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.933+0000 7fb0d9624740 1 --2- 172.21.15.70:0/4057604520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85d0a00 0x562ee85e8630 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8009d90 tx=0x7fb0c8031620 comp rx=0 tx=0).stop 2024-06-04T05:08:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.933+0000 7fb0d9624740 1 -- 172.21.15.70:0/4057604520 shutdown_connections 2024-06-04T05:08:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.933+0000 7fb0d9624740 1 --2- 172.21.15.70:0/4057604520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85d0a00 0x562ee85e8630 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.933+0000 7fb0d9624740 1 --2- 172.21.15.70:0/4057604520 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85c6140 0x562ee85c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.933+0000 7fb0d9624740 1 --2- 172.21.15.70:0/4057604520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85c5820 0x562ee85c1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.933+0000 7fb0d9624740 1 -- 172.21.15.70:0/4057604520 >> 172.21.15.70:0/4057604520 conn(0x562ee85c1d30 msgr2=0x562ee85c7220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:50.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.933+0000 7fb0d9624740 1 -- 172.21.15.70:0/4057604520 shutdown_connections 2024-06-04T05:08:50.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.933+0000 7fb0d9624740 1 -- 172.21.15.70:0/4057604520 wait complete. 2024-06-04T05:08:50.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.933+0000 7fb0d9624740 1 -- start start 2024-06-04T05:08:50.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.933+0000 7fb0d9624740 1 Processor -- start 2024-06-04T05:08:50.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0d9624740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85d44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0d9624740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85c6140 0x562ee85d4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0d9624740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85d0a00 0x562ee85d4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0d9624740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d01afb50 con 0x562ee85c5820 2024-06-04T05:08:50.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0d9624740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d01af450 con 0x562ee85d0a00 2024-06-04T05:08:50.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0d9624740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d01ac6b0 con 0x562ee85c6140 2024-06-04T05:08:50.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0d9624740 10 client.0 _ll_register_callbacks cb 0x7fb0c0016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:50.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0cffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85d44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0cffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85d44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38204/0 (socket says 172.21.15.70:38204) 2024-06-04T05:08:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0cffff640 1 -- 172.21.15.70:0/1937374438 learned_addr learned my addr 172.21.15.70:0/1937374438 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0d4dc7640 1 --2- 172.21.15.70:0/1937374438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85d0a00 0x562ee85d4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0cffff640 1 -- 172.21.15.70:0/1937374438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85c6140 msgr2=0x562ee85d4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0cf7fe640 1 --2- 172.21.15.70:0/1937374438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85c6140 0x562ee85d4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0cffff640 1 --2- 172.21.15.70:0/1937374438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85c6140 0x562ee85d4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0cffff640 1 -- 172.21.15.70:0/1937374438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85d0a00 msgr2=0x562ee85d4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0cffff640 1 --2- 172.21.15.70:0/1937374438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85d0a00 0x562ee85d4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0cffff640 1 -- 172.21.15.70:0/1937374438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ee864a500 con 0x562ee85c5820 2024-06-04T05:08:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0d4dc7640 1 --2- 172.21.15.70:0/1937374438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85d0a00 0x562ee85d4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0cffff640 1 --2- 172.21.15.70:0/1937374438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85d44f0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000b430 tx=0x7fb0c001f910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b2493388feae283 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0af7fe640 1 -- 172.21.15.70:0/1937374438 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0023040 con 0x562ee85c5820 2024-06-04T05:08:50.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0af7fe640 1 -- 172.21.15.70:0/1937374438 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7fb0c0028440 con 0x562ee85c5820 2024-06-04T05:08:50.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.934+0000 7fb0d9624740 1 -- 172.21.15.70:0/1937374438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0c803f050 con 0x562ee85c5820 2024-06-04T05:08:50.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.935+0000 7fb0af7fe640 1 -- 172.21.15.70:0/1937374438 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb0c0011dd0 con 0x562ee85c5820 2024-06-04T05:08:50.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.935+0000 7fb0d9624740 10 client.18558 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:50.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.935+0000 7fb0d9624740 1 -- 172.21.15.70:0/1937374438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562ee864b700 con 0x562ee85c5820 2024-06-04T05:08:50.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.935+0000 7fb0af7fe640 1 -- 172.21.15.70:0/1937374438 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb0c803f050 con 0x562ee85c5820 2024-06-04T05:08:50.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.935+0000 7fb0af7fe640 1 -- 172.21.15.70:0/1937374438 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7fb0c0042070 con 0x562ee85c5820 2024-06-04T05:08:50.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.935+0000 7fb0d9624740 10 client.18558 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:50.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.935+0000 7fb0d9624740 10 client.18558 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:08:50.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.935+0000 7fb0d9624740 20 client.18558 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:50.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.935+0000 7fb0d9624740 1 --2- 172.21.15.70:0/1937374438 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562ee864be20 0x562ee864e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:50.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.935+0000 7fb0d9624740 4 client.18558 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:08:50.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.935+0000 7fb0d9624740 1 -- 172.21.15.70:0/1937374438 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562ee8533360 con 0x562ee864be20 2024-06-04T05:08:50.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.935+0000 7fb0cf7fe640 1 --2- 172.21.15.70:0/1937374438 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562ee864be20 0x562ee864e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:50.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.936+0000 7fb0af7fe640 10 client.18558 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:50.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.936+0000 7fb0cf7fe640 1 --2- 172.21.15.70:0/1937374438 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562ee864be20 0x562ee864e240 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a60867de4bbb3e8 server_cookie=fc436efaed4c5a73 in_seq=0 out_seq=0 2024-06-04T05:08:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.956+0000 7fb0af7fe640 1 -- 172.21.15.70:0/1937374438 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x562ee8533360 con 0x562ee864be20 2024-06-04T05:08:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.956+0000 7fb0af7fe640 10 client.18558 handle_command_reply: tid=0 2024-06-04T05:08:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.956+0000 7fb0d9624740 1 client.18558 shutdown 2024-06-04T05:08:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.956+0000 7fb0d9624740 1 -- 172.21.15.70:0/1937374438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 msgr2=0x562ee85d44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.956+0000 7fb0d9624740 1 --2- 172.21.15.70:0/1937374438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85d44f0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000b430 tx=0x7fb0c001f910 comp rx=0 tx=0).stop 2024-06-04T05:08:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.956+0000 7fb0d9624740 1 -- 172.21.15.70:0/1937374438 shutdown_connections 2024-06-04T05:08:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.956+0000 7fb0d9624740 1 --2- 172.21.15.70:0/1937374438 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562ee864be20 0x562ee864e240 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.956+0000 7fb0d9624740 1 --2- 172.21.15.70:0/1937374438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ee85d0a00 0x562ee85d4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.956+0000 7fb0d9624740 1 --2- 172.21.15.70:0/1937374438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562ee85c6140 0x562ee85d4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.956+0000 7fb0af7fe640 0 client.18558 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.956+0000 7fb0d9624740 1 --2- 172.21.15.70:0/1937374438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ee85c5820 0x562ee85d44f0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.956+0000 7fb0d9624740 1 -- 172.21.15.70:0/1937374438 >> 172.21.15.70:0/1937374438 conn(0x562ee85c1d30 msgr2=0x562ee85ce290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.956+0000 7fb0d9624740 1 -- 172.21.15.70:0/1937374438 shutdown_connections 2024-06-04T05:08:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.956+0000 7fb0d9624740 1 -- 172.21.15.70:0/1937374438 wait complete. 2024-06-04T05:08:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.957+0000 7fb0d9624740 20 client.18558 trim_cache size 0 max 16384 2024-06-04T05:08:50.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:08:50.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:08:50.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:08:50.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.957+0000 7fb0d682a640 1 -- 172.21.15.70:0/3568903270 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb0b80b2a10 msgr2=0x7fb0b80b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.957+0000 7fb0d682a640 1 --2- 172.21.15.70:0/3568903270 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb0b80b2a10 0x7fb0b80b4ed0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x562ee85c6dc0 tx=0x7fb0c0005f50 comp rx=0 tx=0).stop 2024-06-04T05:08:50.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.957+0000 7fb0d682a640 1 -- 172.21.15.70:0/3568903270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0d0107e30 msgr2=0x7fb0d01995c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:50.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.957+0000 7fb0d682a640 1 --2- 172.21.15.70:0/3568903270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0d0107e30 0x7fb0d01995c0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fb0c400d740 tx=0x7fb0c400dc00 comp rx=0 tx=0).stop 2024-06-04T05:08:50.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.957+0000 7fb0d682a640 1 -- 172.21.15.70:0/3568903270 shutdown_connections 2024-06-04T05:08:50.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.957+0000 7fb0d682a640 1 --2- 172.21.15.70:0/3568903270 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb0b80b2a10 0x7fb0b80b4ed0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.957+0000 7fb0d682a640 1 --2- 172.21.15.70:0/3568903270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d019d940 0x7fb0d019ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.957+0000 7fb0d682a640 1 --2- 172.21.15.70:0/3568903270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0d0107e30 0x7fb0d01995c0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.957+0000 7fb0d682a640 1 --2- 172.21.15.70:0/3568903270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0d01011c0 0x7fb0d0199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:50.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.957+0000 7fb0d682a640 1 -- 172.21.15.70:0/3568903270 >> 172.21.15.70:0/3568903270 conn(0x7fb0d0077eb0 msgr2=0x7fb0d0105ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:50.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.957+0000 7fb0d682a640 1 -- 172.21.15.70:0/3568903270 shutdown_connections 2024-06-04T05:08:50.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:08:50.957+0000 7fb0d682a640 1 -- 172.21.15.70:0/3568903270 wait complete. 2024-06-04T05:08:51.178 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:08:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:51 smithi070 ceph-mon[32631]: pgmap v361: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 45 KiB/s wr, 16 op/s 2024-06-04T05:08:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:51 smithi112 ceph-mon[36453]: pgmap v361: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 45 KiB/s wr, 16 op/s 2024-06-04T05:08:51.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196104 1.75 MB/sec execute 464 sec latency 17153.705 ms 2024-06-04T05:08:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:51 smithi098 ceph-mon[35569]: pgmap v361: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 45 KiB/s wr, 16 op/s 2024-06-04T05:08:52.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.319+0000 7fe88d79a700 1 -- 172.21.15.70:0/1173408517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8881037f0 msgr2=0x7fe888103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:52.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.319+0000 7fe88d79a700 1 --2- 172.21.15.70:0/1173408517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8881037f0 0x7fe888103bd0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fe8780099c0 tx=0x7fe878009cd0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.319+0000 7fe88d79a700 1 -- 172.21.15.70:0/1173408517 shutdown_connections 2024-06-04T05:08:52.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.319+0000 7fe88d79a700 1 --2- 172.21.15.70:0/1173408517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8881088f0 0x7fe888110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.319+0000 7fe88d79a700 1 --2- 172.21.15.70:0/1173408517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8881037f0 0x7fe888103bd0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.319+0000 7fe88d79a700 1 --2- 172.21.15.70:0/1173408517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8881041a0 0x7fe8881081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.319+0000 7fe88d79a700 1 -- 172.21.15.70:0/1173408517 >> 172.21.15.70:0/1173408517 conn(0x7fe88807a6a0 msgr2=0x7fe88807aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:52.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.319+0000 7fe88d79a700 1 -- 172.21.15.70:0/1173408517 shutdown_connections 2024-06-04T05:08:52.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.319+0000 7fe88d79a700 1 -- 172.21.15.70:0/1173408517 wait complete. 2024-06-04T05:08:52.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.319+0000 7fe88d79a700 1 Processor -- start 2024-06-04T05:08:52.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.320+0000 7fe88d79a700 1 -- start start 2024-06-04T05:08:52.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.320+0000 7fe88d79a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8881037f0 0x7fe88819f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.320+0000 7fe88d79a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8881041a0 0x7fe88819fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.320+0000 7fe88d79a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8881088f0 0x7fe888199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.320+0000 7fe88d79a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe888113570 con 0x7fe8881088f0 2024-06-04T05:08:52.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.320+0000 7fe88d79a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe8881133f0 con 0x7fe8881041a0 2024-06-04T05:08:52.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.320+0000 7fe88d79a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe8881136f0 con 0x7fe8881037f0 2024-06-04T05:08:52.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.320+0000 7fe8877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8881088f0 0x7fe888199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:52.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.320+0000 7fe8877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8881088f0 0x7fe888199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38216/0 (socket says 172.21.15.70:38216) 2024-06-04T05:08:52.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.320+0000 7fe8877fe700 1 -- 172.21.15.70:0/1878249666 learned_addr learned my addr 172.21.15.70:0/1878249666 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:52.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.320+0000 7fe87e7fc700 1 --2- 172.21.15.70:0/1878249666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8881041a0 0x7fe88819fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:52.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.320+0000 7fe886ffd700 1 --2- 172.21.15.70:0/1878249666 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8881037f0 0x7fe88819f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:52.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.321+0000 7fe8877fe700 1 -- 172.21.15.70:0/1878249666 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8881037f0 msgr2=0x7fe88819f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:52.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.321+0000 7fe8877fe700 1 --2- 172.21.15.70:0/1878249666 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8881037f0 0x7fe88819f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.321+0000 7fe8877fe700 1 -- 172.21.15.70:0/1878249666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8881041a0 msgr2=0x7fe88819fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:52.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.321+0000 7fe8877fe700 1 --2- 172.21.15.70:0/1878249666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8881041a0 0x7fe88819fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.321+0000 7fe8877fe700 1 -- 172.21.15.70:0/1878249666 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe88819a030 con 0x7fe8881088f0 2024-06-04T05:08:52.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.321+0000 7fe886ffd700 1 --2- 172.21.15.70:0/1878249666 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8881037f0 0x7fe88819f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:52.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.321+0000 7fe87e7fc700 1 --2- 172.21.15.70:0/1878249666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8881041a0 0x7fe88819fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:52.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.321+0000 7fe8877fe700 1 --2- 172.21.15.70:0/1878249666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8881088f0 0x7fe888199850 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fe880007ee0 tx=0x7fe88000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:52.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.321+0000 7fe884ff9700 1 -- 172.21.15.70:0/1878249666 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88000dbd0 con 0x7fe8881088f0 2024-06-04T05:08:52.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.321+0000 7fe88d79a700 1 -- 172.21.15.70:0/1878249666 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe88819a320 con 0x7fe8881088f0 2024-06-04T05:08:52.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.321+0000 7fe884ff9700 1 -- 172.21.15.70:0/1878249666 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe88000dd70 con 0x7fe8881088f0 2024-06-04T05:08:52.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.321+0000 7fe884ff9700 1 -- 172.21.15.70:0/1878249666 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8800058d0 con 0x7fe8881088f0 2024-06-04T05:08:52.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.321+0000 7fe88d79a700 1 -- 172.21.15.70:0/1878249666 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8881a6530 con 0x7fe8881088f0 2024-06-04T05:08:52.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.324+0000 7fe884ff9700 1 -- 172.21.15.70:0/1878249666 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fe88001a020 con 0x7fe8881088f0 2024-06-04T05:08:52.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.325+0000 7fe884ff9700 1 --2- 172.21.15.70:0/1878249666 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe8740b29f0 0x7fe8740b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.325+0000 7fe886ffd700 1 --2- 172.21.15.70:0/1878249666 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe8740b29f0 0x7fe8740b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:52.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.325+0000 7fe884ff9700 1 -- 172.21.15.70:0/1878249666 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7fe8800e1d80 con 0x7fe8881088f0 2024-06-04T05:08:52.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.325+0000 7fe89802fb80 1 Processor -- start 2024-06-04T05:08:52.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.325+0000 7fe89802fb80 1 -- start start 2024-06-04T05:08:52.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe89802fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560080876310 0x5600808766f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe89802fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560080867bf0 0x560080868070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe89802fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600808685b0 0x56008087d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe89802fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5600807c5e70 con 0x5600808685b0 2024-06-04T05:08:52.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe89802fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5600807b5a10 con 0x560080867bf0 2024-06-04T05:08:52.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe89802fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5600807b9a50 con 0x560080876310 2024-06-04T05:08:52.329 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:08:52.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe886ffd700 1 --2- 172.21.15.70:0/1878249666 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe8740b29f0 0x7fe8740b4eb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe8780058e0 tx=0x7fe87801a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:52.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe8877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600808685b0 0x56008087d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:52.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe8877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600808685b0 0x56008087d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38226/0 (socket says 172.21.15.70:38226) 2024-06-04T05:08:52.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe8877fe700 1 -- 172.21.15.70:0/680936800 learned_addr learned my addr 172.21.15.70:0/680936800 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:52.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe886ffd700 1 --2- 172.21.15.70:0/680936800 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560080876310 0x5600808766f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:52.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe87e7fc700 1 --2- 172.21.15.70:0/680936800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560080867bf0 0x560080868070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:52.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe8877fe700 1 -- 172.21.15.70:0/680936800 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560080876310 msgr2=0x5600808766f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:52.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe8877fe700 1 --2- 172.21.15.70:0/680936800 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560080876310 0x5600808766f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe8877fe700 1 -- 172.21.15.70:0/680936800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560080867bf0 msgr2=0x560080868070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:52.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe8877fe700 1 --2- 172.21.15.70:0/680936800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560080867bf0 0x560080868070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe8877fe700 1 -- 172.21.15.70:0/680936800 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56008075f0b0 con 0x5600808685b0 2024-06-04T05:08:52.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.326+0000 7fe886ffd700 1 --2- 172.21.15.70:0/680936800 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560080876310 0x5600808766f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:52.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.327+0000 7fe8877fe700 1 --2- 172.21.15.70:0/680936800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600808685b0 0x56008087d040 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fe8800dfaa0 tx=0x7fe8800a5020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1304166f96a9dfe3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:52.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.327+0000 7fe87d7fa700 1 -- 172.21.15.70:0/680936800 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8800ad3e0 con 0x5600808685b0 2024-06-04T05:08:52.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.327+0000 7fe87d7fa700 1 -- 172.21.15.70:0/680936800 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe8800b1e90 con 0x5600808685b0 2024-06-04T05:08:52.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.327+0000 7fe87d7fa700 1 -- 172.21.15.70:0/680936800 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe880004ad0 con 0x5600808685b0 2024-06-04T05:08:52.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.328+0000 7fe89802fb80 1 -- 172.21.15.70:0/680936800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600808685b0 msgr2=0x56008087d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:52.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.328+0000 7fe89802fb80 1 --2- 172.21.15.70:0/680936800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600808685b0 0x56008087d040 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fe8800dfaa0 tx=0x7fe8800a5020 comp rx=0 tx=0).stop 2024-06-04T05:08:52.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.328+0000 7fe89802fb80 1 -- 172.21.15.70:0/680936800 shutdown_connections 2024-06-04T05:08:52.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.328+0000 7fe89802fb80 1 --2- 172.21.15.70:0/680936800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600808685b0 0x56008087d040 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.328+0000 7fe89802fb80 1 --2- 172.21.15.70:0/680936800 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560080876310 0x5600808766f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.328+0000 7fe89802fb80 1 --2- 172.21.15.70:0/680936800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560080867bf0 0x560080868070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.328+0000 7fe89802fb80 1 -- 172.21.15.70:0/680936800 >> 172.21.15.70:0/680936800 conn(0x560080793340 msgr2=0x560080862ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:52.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.328+0000 7fe89802fb80 1 -- 172.21.15.70:0/680936800 shutdown_connections 2024-06-04T05:08:52.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.328+0000 7fe89802fb80 1 -- 172.21.15.70:0/680936800 wait complete. 2024-06-04T05:08:52.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.329+0000 7fe89802fb80 1 -- start start 2024-06-04T05:08:52.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.329+0000 7fe89802fb80 1 Processor -- start 2024-06-04T05:08:52.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.329+0000 7fe89802fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560080867bf0 0x56008087cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.329+0000 7fe89802fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5600808685b0 0x56008087a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.329+0000 7fe89802fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080876310 0x56008087a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.329+0000 7fe89802fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5600806bc0a0 con 0x560080876310 2024-06-04T05:08:52.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.329+0000 7fe89802fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56008071dea0 con 0x5600808685b0 2024-06-04T05:08:52.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.329+0000 7fe89802fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56008087d6a0 con 0x560080867bf0 2024-06-04T05:08:52.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.329+0000 7fe89802fb80 10 client.0 _ll_register_callbacks cb 0x560080750a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:52.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.329+0000 7fe8877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080876310 0x56008087a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:52.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.329+0000 7fe8877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080876310 0x56008087a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38236/0 (socket says 172.21.15.70:38236) 2024-06-04T05:08:52.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.329+0000 7fe8877fe700 1 -- 172.21.15.70:0/1293647766 learned_addr learned my addr 172.21.15.70:0/1293647766 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:52.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.329+0000 7fe886ffd700 1 --2- 172.21.15.70:0/1293647766 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560080867bf0 0x56008087cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:52.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.330+0000 7fe8877fe700 1 -- 172.21.15.70:0/1293647766 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560080867bf0 msgr2=0x56008087cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:52.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.330+0000 7fe8877fe700 1 --2- 172.21.15.70:0/1293647766 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560080867bf0 0x56008087cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.330+0000 7fe8877fe700 1 -- 172.21.15.70:0/1293647766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5600808685b0 msgr2=0x56008087a010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-04T05:08:52.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.330+0000 7fe8877fe700 1 --2- 172.21.15.70:0/1293647766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5600808685b0 0x56008087a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.330+0000 7fe8877fe700 1 -- 172.21.15.70:0/1293647766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600807059d0 con 0x560080876310 2024-06-04T05:08:52.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.330+0000 7fe8877fe700 1 --2- 172.21.15.70:0/1293647766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080876310 0x56008087a550 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fe8800a8fc0 tx=0x7fe8800a90f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e116d1133d98e183 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:52.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.330+0000 7fe86e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:52.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.330+0000 7fe86e7fc700 1 -- 172.21.15.70:0/1293647766 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8800ad3e0 con 0x560080876310 2024-06-04T05:08:52.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.330+0000 7fe86e7fc700 1 -- 172.21.15.70:0/1293647766 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7fe8800b1c10 con 0x560080876310 2024-06-04T05:08:52.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.330+0000 7fe89802fb80 1 -- 172.21.15.70:0/1293647766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8800b1e90 con 0x560080876310 2024-06-04T05:08:52.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.330+0000 7fe86e7fc700 1 -- 172.21.15.70:0/1293647766 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fe8800a9290 con 0x560080876310 2024-06-04T05:08:52.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.330+0000 7fe89802fb80 10 client.18570 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:52.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.330+0000 7fe89802fb80 1 -- 172.21.15.70:0/1293647766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56008076f860 con 0x560080876310 2024-06-04T05:08:52.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.331+0000 7fe86e7fc700 1 -- 172.21.15.70:0/1293647766 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe8800b1e90 con 0x560080876310 2024-06-04T05:08:52.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.331+0000 7fe86e7fc700 1 -- 172.21.15.70:0/1293647766 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7fe8800050d0 con 0x560080876310 2024-06-04T05:08:52.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.331+0000 7fe89802fb80 10 client.18570 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:52.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.331+0000 7fe89802fb80 10 client.18570 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:08:52.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.331+0000 7fe89802fb80 20 client.18570 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:52.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.331+0000 7fe89802fb80 1 --2- 172.21.15.70:0/1293647766 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56008088a110 0x56008088c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.331+0000 7fe89802fb80 4 client.18570 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:08:52.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.331+0000 7fe89802fb80 1 -- 172.21.15.70:0/1293647766 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600807c7dc0 con 0x56008088a110 2024-06-04T05:08:52.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.331+0000 7fe886ffd700 1 --2- 172.21.15.70:0/1293647766 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56008088a110 0x56008088c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:52.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.332+0000 7fe86e7fc700 10 client.18570 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:52.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.332+0000 7fe886ffd700 1 --2- 172.21.15.70:0/1293647766 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56008088a110 0x56008088c500 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b8847e9b232ec21 server_cookie=7616082fe890af3f in_seq=0 out_seq=0 2024-06-04T05:08:52.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.332+0000 7fe86e7fc700 1 -- 172.21.15.70:0/1293647766 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5600807c7dc0 con 0x56008088a110 2024-06-04T05:08:52.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.332+0000 7fe86e7fc700 10 client.18570 handle_command_reply: tid=0 2024-06-04T05:08:52.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.332+0000 7fe89802fb80 1 client.18570 shutdown 2024-06-04T05:08:52.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.332+0000 7fe89802fb80 1 -- 172.21.15.70:0/1293647766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080876310 msgr2=0x56008087a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:52.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.333+0000 7fe89802fb80 1 --2- 172.21.15.70:0/1293647766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080876310 0x56008087a550 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fe8800a8fc0 tx=0x7fe8800a90f0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.333+0000 7fe89802fb80 1 -- 172.21.15.70:0/1293647766 shutdown_connections 2024-06-04T05:08:52.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.333+0000 7fe89802fb80 1 --2- 172.21.15.70:0/1293647766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080876310 0x56008087a550 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.333+0000 7fe89802fb80 1 --2- 172.21.15.70:0/1293647766 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560080867bf0 0x56008087cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.333+0000 7fe89802fb80 1 --2- 172.21.15.70:0/1293647766 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56008088a110 0x56008088c500 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.333+0000 7fe89802fb80 1 --2- 172.21.15.70:0/1293647766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5600808685b0 0x56008087a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.333+0000 7fe86e7fc700 0 client.18570 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:52.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.333+0000 7fe89802fb80 1 -- 172.21.15.70:0/1293647766 >> 172.21.15.70:0/1293647766 conn(0x560080793340 msgr2=0x560080866be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:52.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.333+0000 7fe89802fb80 1 -- 172.21.15.70:0/1293647766 shutdown_connections 2024-06-04T05:08:52.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.333+0000 7fe89802fb80 1 -- 172.21.15.70:0/1293647766 wait complete. 2024-06-04T05:08:52.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.333+0000 7fe89802fb80 20 client.18570 trim_cache size 0 max 16384 2024-06-04T05:08:52.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.357+0000 7fe89802fb80 1 Processor -- start 2024-06-04T05:08:52.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.357+0000 7fe89802fb80 1 -- start start 2024-06-04T05:08:52.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe89802fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080867bf0 0x56008085ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe89802fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5600808685b0 0x5600808759c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe89802fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560080876310 0x560080891b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe89802fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe86800d690 con 0x560080867bf0 2024-06-04T05:08:52.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe89802fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe86800f230 con 0x560080876310 2024-06-04T05:08:52.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe89802fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe86800dd60 con 0x5600808685b0 2024-06-04T05:08:52.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe886ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080867bf0 0x56008085ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:52.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe886ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080867bf0 0x56008085ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38244/0 (socket says 172.21.15.70:38244) 2024-06-04T05:08:52.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe886ffd700 1 -- 172.21.15.70:0/3176690386 learned_addr learned my addr 172.21.15.70:0/3176690386 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:52.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe87e7fc700 1 --2- 172.21.15.70:0/3176690386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5600808685b0 0x5600808759c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:52.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe8877fe700 1 --2- 172.21.15.70:0/3176690386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560080876310 0x560080891b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:52.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe886ffd700 1 -- 172.21.15.70:0/3176690386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5600808685b0 msgr2=0x5600808759c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:52.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe886ffd700 1 --2- 172.21.15.70:0/3176690386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5600808685b0 0x5600808759c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe886ffd700 1 -- 172.21.15.70:0/3176690386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560080876310 msgr2=0x560080891b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:52.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe886ffd700 1 --2- 172.21.15.70:0/3176690386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560080876310 0x560080891b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe886ffd700 1 -- 172.21.15.70:0/3176690386 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56008087a6a0 con 0x560080867bf0 2024-06-04T05:08:52.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe87e7fc700 1 --2- 172.21.15.70:0/3176690386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5600808685b0 0x5600808759c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:52.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.358+0000 7fe886ffd700 1 --2- 172.21.15.70:0/3176690386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080867bf0 0x56008085ce70 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fe8780099c0 tx=0x7fe87801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37fe016f2a293577 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:52.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.359+0000 7fe87cff9700 1 -- 172.21.15.70:0/3176690386 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87800fda0 con 0x560080867bf0 2024-06-04T05:08:52.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.359+0000 7fe87cff9700 1 -- 172.21.15.70:0/3176690386 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe8780238a0 con 0x560080867bf0 2024-06-04T05:08:52.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.359+0000 7fe87cff9700 1 -- 172.21.15.70:0/3176690386 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878031730 con 0x560080867bf0 2024-06-04T05:08:52.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.359+0000 7fe89802fb80 1 -- 172.21.15.70:0/3176690386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080867bf0 msgr2=0x56008085ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:52.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.359+0000 7fe89802fb80 1 --2- 172.21.15.70:0/3176690386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080867bf0 0x56008085ce70 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fe8780099c0 tx=0x7fe87801e000 comp rx=0 tx=0).stop 2024-06-04T05:08:52.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.360+0000 7fe89802fb80 1 -- 172.21.15.70:0/3176690386 shutdown_connections 2024-06-04T05:08:52.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.360+0000 7fe89802fb80 1 --2- 172.21.15.70:0/3176690386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560080876310 0x560080891b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.360+0000 7fe89802fb80 1 --2- 172.21.15.70:0/3176690386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080867bf0 0x56008085ce70 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.360+0000 7fe89802fb80 1 --2- 172.21.15.70:0/3176690386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5600808685b0 0x5600808759c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.360+0000 7fe89802fb80 1 -- 172.21.15.70:0/3176690386 >> 172.21.15.70:0/3176690386 conn(0x560080793340 msgr2=0x560080867250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:52.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.360+0000 7fe89802fb80 1 -- 172.21.15.70:0/3176690386 shutdown_connections 2024-06-04T05:08:52.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.360+0000 7fe89802fb80 1 -- 172.21.15.70:0/3176690386 wait complete. 2024-06-04T05:08:52.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.360+0000 7fe89802fb80 1 -- start start 2024-06-04T05:08:52.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.360+0000 7fe89802fb80 1 Processor -- start 2024-06-04T05:08:52.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe89802fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560080867bf0 0x560080882fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe89802fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5600808685b0 0x56008087f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe89802fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080876310 0x56008087fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe89802fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe86800eb30 con 0x560080876310 2024-06-04T05:08:52.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe89802fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe86800e430 con 0x5600808685b0 2024-06-04T05:08:52.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe89802fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe86800b650 con 0x560080867bf0 2024-06-04T05:08:52.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe89802fb80 10 client.0 _ll_register_callbacks cb 0x560080750a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:52.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe8877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080876310 0x56008087fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:52.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe8877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080876310 0x56008087fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38250/0 (socket says 172.21.15.70:38250) 2024-06-04T05:08:52.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe8877fe700 1 -- 172.21.15.70:0/105054813 learned_addr learned my addr 172.21.15.70:0/105054813 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:52.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe87e7fc700 1 --2- 172.21.15.70:0/105054813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5600808685b0 0x56008087f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:52.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe886ffd700 1 --2- 172.21.15.70:0/105054813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560080867bf0 0x560080882fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:52.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe8877fe700 1 -- 172.21.15.70:0/105054813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560080867bf0 msgr2=0x560080882fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:52.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe8877fe700 1 --2- 172.21.15.70:0/105054813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560080867bf0 0x560080882fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe8877fe700 1 -- 172.21.15.70:0/105054813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5600808685b0 msgr2=0x56008087f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:52.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe8877fe700 1 --2- 172.21.15.70:0/105054813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5600808685b0 0x56008087f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe8877fe700 1 -- 172.21.15.70:0/105054813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560080880660 con 0x560080876310 2024-06-04T05:08:52.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe8877fe700 1 --2- 172.21.15.70:0/105054813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080876310 0x56008087fe60 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fe8800dfa20 tx=0x7fe8800a4c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dbe43a955a1a86f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:52.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.361+0000 7fe86e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:52.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.362+0000 7fe86e7fc700 1 -- 172.21.15.70:0/105054813 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8800a4ef0 con 0x560080876310 2024-06-04T05:08:52.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.362+0000 7fe89802fb80 1 -- 172.21.15.70:0/105054813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8780238a0 con 0x560080876310 2024-06-04T05:08:52.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.362+0000 7fe86e7fc700 1 -- 172.21.15.70:0/105054813 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7fe880004f80 con 0x560080876310 2024-06-04T05:08:52.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.362+0000 7fe86e7fc700 1 -- 172.21.15.70:0/105054813 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fe8800ad3e0 con 0x560080876310 2024-06-04T05:08:52.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.362+0000 7fe89802fb80 10 client.18576 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:52.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.362+0000 7fe89802fb80 1 -- 172.21.15.70:0/105054813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5600808f5a00 con 0x560080876310 2024-06-04T05:08:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.362+0000 7fe86e7fc700 1 -- 172.21.15.70:0/105054813 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe8780238a0 con 0x560080876310 2024-06-04T05:08:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.362+0000 7fe86e7fc700 1 -- 172.21.15.70:0/105054813 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7fe88008f150 con 0x560080876310 2024-06-04T05:08:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.362+0000 7fe89802fb80 10 client.18576 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.362+0000 7fe89802fb80 10 client.18576 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:08:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.362+0000 7fe89802fb80 20 client.18576 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.362+0000 7fe89802fb80 1 --2- 172.21.15.70:0/105054813 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5600808f5f50 0x5600808f8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.362+0000 7fe89802fb80 4 client.18576 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "af20084", "release": true, "await": true}] 2024-06-04T05:08:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.362+0000 7fe89802fb80 1 -- 172.21.15.70:0/105054813 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "af20084", "release": true, "await": true}) -- 0x5600807c15a0 con 0x5600808f5f50 2024-06-04T05:08:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.363+0000 7fe886ffd700 1 --2- 172.21.15.70:0/105054813 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5600808f5f50 0x5600808f8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.363+0000 7fe86e7fc700 10 client.18576 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.363+0000 7fe886ffd700 1 --2- 172.21.15.70:0/105054813 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5600808f5f50 0x5600808f8370 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e907ccf65a2f58b server_cookie=b1a0553563f5c55b in_seq=0 out_seq=0 2024-06-04T05:08:52.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.369+0000 7fe86e7fc700 1 -- 172.21.15.70:0/105054813 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5600807c15a0 con 0x5600808f5f50 2024-06-04T05:08:52.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.369+0000 7fe86e7fc700 10 client.18576 handle_command_reply: tid=0 2024-06-04T05:08:52.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.369+0000 7fe89802fb80 1 client.18576 shutdown 2024-06-04T05:08:52.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.369+0000 7fe89802fb80 1 -- 172.21.15.70:0/105054813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080876310 msgr2=0x56008087fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:52.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.369+0000 7fe89802fb80 1 --2- 172.21.15.70:0/105054813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080876310 0x56008087fe60 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fe8800dfa20 tx=0x7fe8800a4c70 comp rx=0 tx=0).stop 2024-06-04T05:08:52.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.369+0000 7fe89802fb80 1 -- 172.21.15.70:0/105054813 shutdown_connections 2024-06-04T05:08:52.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.369+0000 7fe89802fb80 1 --2- 172.21.15.70:0/105054813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560080876310 0x56008087fe60 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.369+0000 7fe89802fb80 1 --2- 172.21.15.70:0/105054813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560080867bf0 0x560080882fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.369+0000 7fe89802fb80 1 --2- 172.21.15.70:0/105054813 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5600808f5f50 0x5600808f8370 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.369+0000 7fe89802fb80 1 --2- 172.21.15.70:0/105054813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5600808685b0 0x56008087f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.369+0000 7fe89802fb80 1 -- 172.21.15.70:0/105054813 >> 172.21.15.70:0/105054813 conn(0x560080793340 msgr2=0x560080873c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:52.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.369+0000 7fe86e7fc700 0 client.18576 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:08:52.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.369+0000 7fe89802fb80 1 -- 172.21.15.70:0/105054813 shutdown_connections 2024-06-04T05:08:52.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.369+0000 7fe89802fb80 1 -- 172.21.15.70:0/105054813 wait complete. 2024-06-04T05:08:52.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.369+0000 7fe89802fb80 20 client.18576 trim_cache size 0 max 16384 2024-06-04T05:08:52.373 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:08:52.373 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 34, 2024-06-04T05:08:52.373 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:08:52.373 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 36, 2024-06-04T05:08:52.373 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:08:52.373 INFO:teuthology.orchestra.run.smithi070.stdout: "af20084": { 2024-06-04T05:08:52.373 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 36, 2024-06-04T05:08:52.374 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:08:52.374 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:08:52.374 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:08:52.374 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:08:52.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:08:52.374 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:08:52.374 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 134.9, 2024-06-04T05:08:52.374 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:08:52.374 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:08:52.374 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:08:52.374 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:08:52.375 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:08:52.375 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:08:52.375 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:52.375 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:52.375 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:52.375 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:52.375 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:08:52.375 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:08:52.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.370+0000 7fe88d79a700 1 -- 172.21.15.70:0/1878249666 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe8740b29f0 msgr2=0x7fe8740b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:52.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.370+0000 7fe88d79a700 1 --2- 172.21.15.70:0/1878249666 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe8740b29f0 0x7fe8740b4eb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe8780058e0 tx=0x7fe87801a040 comp rx=0 tx=0).stop 2024-06-04T05:08:52.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.370+0000 7fe88d79a700 1 -- 172.21.15.70:0/1878249666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8881088f0 msgr2=0x7fe888199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:52.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.370+0000 7fe88d79a700 1 --2- 172.21.15.70:0/1878249666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8881088f0 0x7fe888199850 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fe880007ee0 tx=0x7fe88000d3b0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.370+0000 7fe88d79a700 1 -- 172.21.15.70:0/1878249666 shutdown_connections 2024-06-04T05:08:52.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.370+0000 7fe88d79a700 1 --2- 172.21.15.70:0/1878249666 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe8740b29f0 0x7fe8740b4eb0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.370+0000 7fe88d79a700 1 --2- 172.21.15.70:0/1878249666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8881088f0 0x7fe888199850 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.370+0000 7fe88d79a700 1 --2- 172.21.15.70:0/1878249666 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8881037f0 0x7fe88819f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.370+0000 7fe88d79a700 1 --2- 172.21.15.70:0/1878249666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8881041a0 0x7fe88819fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.370+0000 7fe88d79a700 1 -- 172.21.15.70:0/1878249666 >> 172.21.15.70:0/1878249666 conn(0x7fe88807a6a0 msgr2=0x7fe8881018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.370+0000 7fe88d79a700 1 -- 172.21.15.70:0/1878249666 shutdown_connections 2024-06-04T05:08:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:52.370+0000 7fe88d79a700 1 -- 172.21.15.70:0/1878249666 wait complete. 2024-06-04T05:08:52.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196275 1.75 MB/sec execute 465 sec latency 17785.724 ms 2024-06-04T05:08:53.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:53 smithi070 ceph-mon[32631]: pgmap v362: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 68 KiB/s wr, 25 op/s 2024-06-04T05:08:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:53 smithi112 ceph-mon[36453]: pgmap v362: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 68 KiB/s wr, 25 op/s 2024-06-04T05:08:53.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 196817 1.75 MB/sec execute 466 sec latency 18.236 ms 2024-06-04T05:08:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:53 smithi098 ceph-mon[35569]: pgmap v362: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 68 KiB/s wr, 25 op/s 2024-06-04T05:08:53.821 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'af20084', total seconds elapsed: 25.0 2024-06-04T05:08:54.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 197459 1.75 MB/sec execute 467 sec latency 29.189 ms 2024-06-04T05:08:54.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.962+0000 7f3a017c2700 1 -- 172.21.15.70:0/3932364925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39fc06bab0 msgr2=0x7f39fc06bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:54.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.962+0000 7f3a017c2700 1 --2- 172.21.15.70:0/3932364925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39fc06bab0 0x7f39fc06bf30 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f39f0009990 tx=0x7f39f0009ca0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.962+0000 7f3a017c2700 1 -- 172.21.15.70:0/3932364925 shutdown_connections 2024-06-04T05:08:54.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.962+0000 7f3a017c2700 1 --2- 172.21.15.70:0/3932364925 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39fc06c470 0x7f39fc1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.962+0000 7f3a017c2700 1 --2- 172.21.15.70:0/3932364925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39fc1095a0 0x7f39fc109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.962+0000 7f3a017c2700 1 --2- 172.21.15.70:0/3932364925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39fc06bab0 0x7f39fc06bf30 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.962+0000 7f3a017c2700 1 -- 172.21.15.70:0/3932364925 >> 172.21.15.70:0/3932364925 conn(0x7f39fc0793b0 msgr2=0x7f39fc0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:54.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.962+0000 7f3a017c2700 1 -- 172.21.15.70:0/3932364925 shutdown_connections 2024-06-04T05:08:54.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.962+0000 7f3a017c2700 1 -- 172.21.15.70:0/3932364925 wait complete. 2024-06-04T05:08:54.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.962+0000 7f3a017c2700 1 Processor -- start 2024-06-04T05:08:54.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.963+0000 7f3a017c2700 1 -- start start 2024-06-04T05:08:54.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.963+0000 7f3a017c2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39fc06bab0 0x7f39fc19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:54.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.963+0000 7f3a017c2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39fc06c470 0x7f39fc19fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:54.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.963+0000 7f3a017c2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39fc1095a0 0x7f39fc1999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:54.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.963+0000 7f3a017c2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39fc113580 con 0x7f39fc06bab0 2024-06-04T05:08:54.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.963+0000 7f3a017c2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39fc113400 con 0x7f39fc06c470 2024-06-04T05:08:54.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.963+0000 7f3a017c2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f39fc113700 con 0x7f39fc1095a0 2024-06-04T05:08:54.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.963+0000 7f39faffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39fc06bab0 0x7f39fc19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:54.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.963+0000 7f39faffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39fc06bab0 0x7f39fc19f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38276/0 (socket says 172.21.15.70:38276) 2024-06-04T05:08:54.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.963+0000 7f39faffd700 1 -- 172.21.15.70:0/1833597604 learned_addr learned my addr 172.21.15.70:0/1833597604 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:54.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.963+0000 7f39fb7fe700 1 --2- 172.21.15.70:0/1833597604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39fc1095a0 0x7f39fc1999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:54.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.963+0000 7f39fa7fc700 1 --2- 172.21.15.70:0/1833597604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39fc06c470 0x7f39fc19fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:54.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.964+0000 7f39fb7fe700 1 -- 172.21.15.70:0/1833597604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39fc06c470 msgr2=0x7f39fc19fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:54.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.964+0000 7f39fb7fe700 1 --2- 172.21.15.70:0/1833597604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39fc06c470 0x7f39fc19fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.964+0000 7f39fb7fe700 1 -- 172.21.15.70:0/1833597604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39fc06bab0 msgr2=0x7f39fc19f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:54.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.964+0000 7f39fb7fe700 1 --2- 172.21.15.70:0/1833597604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39fc06bab0 0x7f39fc19f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.964+0000 7f39fb7fe700 1 -- 172.21.15.70:0/1833597604 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39fc19a120 con 0x7f39fc1095a0 2024-06-04T05:08:54.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.964+0000 7f39faffd700 1 --2- 172.21.15.70:0/1833597604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39fc06bab0 0x7f39fc19f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:54.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.964+0000 7f39fb7fe700 1 --2- 172.21.15.70:0/1833597604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39fc1095a0 0x7f39fc1999c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f39f400efc0 tx=0x7f39f400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:54.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.964+0000 7f39e3fff700 1 -- 172.21.15.70:0/1833597604 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f40092e0 con 0x7f39fc1095a0 2024-06-04T05:08:54.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.964+0000 7f39e3fff700 1 -- 172.21.15.70:0/1833597604 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39f4004bb0 con 0x7f39fc1095a0 2024-06-04T05:08:54.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.964+0000 7f39e3fff700 1 -- 172.21.15.70:0/1833597604 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f4004020 con 0x7f39fc1095a0 2024-06-04T05:08:54.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.964+0000 7f3a017c2700 1 -- 172.21.15.70:0/1833597604 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39fc19a3b0 con 0x7f39fc1095a0 2024-06-04T05:08:54.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.964+0000 7f3a017c2700 1 -- 172.21.15.70:0/1833597604 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39fc102430 con 0x7f39fc1095a0 2024-06-04T05:08:54.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.967+0000 7f39e3fff700 1 -- 172.21.15.70:0/1833597604 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f39f401b030 con 0x7f39fc1095a0 2024-06-04T05:08:54.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.967+0000 7f39e3fff700 1 --2- 172.21.15.70:0/1833597604 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39e40b29f0 0x7f39e40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:54.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.968+0000 7f39faffd700 1 --2- 172.21.15.70:0/1833597604 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39e40b29f0 0x7f39e40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:54.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.968+0000 7f39e3fff700 1 -- 172.21.15.70:0/1833597604 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f39f40e2140 con 0x7f39fc1095a0 2024-06-04T05:08:54.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.968+0000 7f3a0c057b80 1 Processor -- start 2024-06-04T05:08:54.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.968+0000 7f3a0c057b80 1 -- start start 2024-06-04T05:08:54.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.968+0000 7f39faffd700 1 --2- 172.21.15.70:0/1833597604 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39e40b29f0 0x7f39e40b4eb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f39fc19ab00 tx=0x7f39ec00b500 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:54.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.968+0000 7f3a0c057b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 0x55ef0f145c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:54.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.968+0000 7f3a0c057b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f146210 0x55ef0f149940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:54.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.968+0000 7f3a0c057b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f149f60 0x55ef0f14a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:54.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.968+0000 7f3a0c057b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ef0f0a4c90 con 0x55ef0f145860 2024-06-04T05:08:54.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.968+0000 7f3a0c057b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ef0f094830 con 0x55ef0f146210 2024-06-04T05:08:54.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.968+0000 7f3a0c057b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ef0f098870 con 0x55ef0f149f60 2024-06-04T05:08:54.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.968+0000 7f39faffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 0x55ef0f145c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:54.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.968+0000 7f39faffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 0x55ef0f145c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38282/0 (socket says 172.21.15.70:38282) 2024-06-04T05:08:54.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.968+0000 7f39faffd700 1 -- 172.21.15.70:0/935583615 learned_addr learned my addr 172.21.15.70:0/935583615 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:54.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.968+0000 7f39fb7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f149f60 0x55ef0f14a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:54.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.968+0000 7f39fa7fc700 1 --2- 172.21.15.70:0/935583615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f146210 0x55ef0f149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:54.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.969+0000 7f39fb7fe700 1 -- 172.21.15.70:0/935583615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f146210 msgr2=0x55ef0f149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:54.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.969+0000 7f39fb7fe700 1 --2- 172.21.15.70:0/935583615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f146210 0x55ef0f149940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.969+0000 7f39fb7fe700 1 -- 172.21.15.70:0/935583615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 msgr2=0x55ef0f145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:54.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.969+0000 7f39fb7fe700 1 --2- 172.21.15.70:0/935583615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 0x55ef0f145c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.969+0000 7f39fb7fe700 1 -- 172.21.15.70:0/935583615 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef0efe48d0 con 0x55ef0f149f60 2024-06-04T05:08:54.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.969+0000 7f39fa7fc700 1 --2- 172.21.15.70:0/935583615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f146210 0x55ef0f149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:08:54.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.969+0000 7f39fb7fe700 1 --2- 172.21.15.70:0/935583615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f149f60 0x55ef0f14a3a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f39f40df800 tx=0x7f39f4004de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bdea99bdf2a361c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:54.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.969+0000 7f39e17fa700 1 -- 172.21.15.70:0/935583615 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f40a9420 con 0x55ef0f149f60 2024-06-04T05:08:54.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.969+0000 7f39e17fa700 1 -- 172.21.15.70:0/935583615 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39f40ad7d0 con 0x55ef0f149f60 2024-06-04T05:08:54.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.969+0000 7f39e17fa700 1 -- 172.21.15.70:0/935583615 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f40a8680 con 0x55ef0f149f60 2024-06-04T05:08:54.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.970+0000 7f3a0c057b80 1 -- 172.21.15.70:0/935583615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f149f60 msgr2=0x55ef0f14a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:54.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.970+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/935583615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f149f60 0x55ef0f14a3a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f39f40df800 tx=0x7f39f4004de0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.970+0000 7f3a0c057b80 1 -- 172.21.15.70:0/935583615 shutdown_connections 2024-06-04T05:08:54.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.970+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/935583615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f149f60 0x55ef0f14a3a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.970+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/935583615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 0x55ef0f145c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.970+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/935583615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f146210 0x55ef0f149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.970+0000 7f3a0c057b80 1 -- 172.21.15.70:0/935583615 >> 172.21.15.70:0/935583615 conn(0x55ef0f073080 msgr2=0x55ef0f13fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:54.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.970+0000 7f3a0c057b80 1 -- 172.21.15.70:0/935583615 shutdown_connections 2024-06-04T05:08:54.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.970+0000 7f3a0c057b80 1 -- 172.21.15.70:0/935583615 wait complete. 2024-06-04T05:08:54.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.970+0000 7f3a0c057b80 1 -- start start 2024-06-04T05:08:54.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.970+0000 7f3a0c057b80 1 Processor -- start 2024-06-04T05:08:54.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f3a0c057b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f145860 0x55ef0f151500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:54.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f3a0c057b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f146210 0x55ef0f14e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:54.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f3a0c057b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f149f60 0x55ef0f14edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:54.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f3a0c057b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ef0ef9afa0 con 0x55ef0f146210 2024-06-04T05:08:54.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f3a0c057b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ef0f0367c0 con 0x55ef0f149f60 2024-06-04T05:08:54.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f3a0c057b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ef0f152120 con 0x55ef0f145860 2024-06-04T05:08:54.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f3a0c057b80 10 client.0 _ll_register_callbacks cb 0x55ef0f02f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:54.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f39fa7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f146210 0x55ef0f14e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:54.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f39faffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f145860 0x55ef0f151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:54.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f39fa7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f146210 0x55ef0f14e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38296/0 (socket says 172.21.15.70:38296) 2024-06-04T05:08:54.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f39fa7fc700 1 -- 172.21.15.70:0/2307347303 learned_addr learned my addr 172.21.15.70:0/2307347303 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:54.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f39fb7fe700 1 --2- 172.21.15.70:0/2307347303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f149f60 0x55ef0f14edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:54.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f39fa7fc700 1 -- 172.21.15.70:0/2307347303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f145860 msgr2=0x55ef0f151500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:54.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f39fa7fc700 1 --2- 172.21.15.70:0/2307347303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f145860 0x55ef0f151500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f39fa7fc700 1 -- 172.21.15.70:0/2307347303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f149f60 msgr2=0x55ef0f14edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:54.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f39fa7fc700 1 --2- 172.21.15.70:0/2307347303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f149f60 0x55ef0f14edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f39fa7fc700 1 -- 172.21.15.70:0/2307347303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef0f033690 con 0x55ef0f146210 2024-06-04T05:08:54.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f39faffd700 1 --2- 172.21.15.70:0/2307347303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f145860 0x55ef0f151500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:54.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f39fb7fe700 1 --2- 172.21.15.70:0/2307347303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f149f60 0x55ef0f14edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:54.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f39fa7fc700 1 --2- 172.21.15.70:0/2307347303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f146210 0x55ef0f14e870 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f39f0009990 tx=0x7f39f0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d70dd9be4079501 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:54.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f39deff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:54.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.971+0000 7f39deff5700 1 -- 172.21.15.70:0/2307347303 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f0003ce0 con 0x55ef0f146210 2024-06-04T05:08:54.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.972+0000 7f39deff5700 1 -- 172.21.15.70:0/2307347303 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f39f000b6b0 con 0x55ef0f146210 2024-06-04T05:08:54.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.972+0000 7f3a0c057b80 1 -- 172.21.15.70:0/2307347303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39f40ad7d0 con 0x55ef0f146210 2024-06-04T05:08:54.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.972+0000 7f39deff5700 1 -- 172.21.15.70:0/2307347303 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f39f000b8b0 con 0x55ef0f146210 2024-06-04T05:08:54.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.972+0000 7f3a0c057b80 10 client.18588 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:54.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.972+0000 7f3a0c057b80 1 -- 172.21.15.70:0/2307347303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ef0f165200 con 0x55ef0f146210 2024-06-04T05:08:54.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.972+0000 7f39deff5700 1 -- 172.21.15.70:0/2307347303 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39f40ad7d0 con 0x55ef0f146210 2024-06-04T05:08:54.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.972+0000 7f39deff5700 1 -- 172.21.15.70:0/2307347303 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f39f000f7d0 con 0x55ef0f146210 2024-06-04T05:08:54.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.972+0000 7f3a0c057b80 10 client.18588 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:54.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.972+0000 7f3a0c057b80 10 client.18588 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:08:54.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.972+0000 7f3a0c057b80 20 client.18588 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:54.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.972+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/2307347303 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55ef0f167890 0x55ef0f169cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:54.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.972+0000 7f3a0c057b80 4 client.18588 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:08:54.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.972+0000 7f3a0c057b80 1 -- 172.21.15.70:0/2307347303 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef0f1119e0 con 0x55ef0f167890 2024-06-04T05:08:54.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.972+0000 7f39faffd700 1 --2- 172.21.15.70:0/2307347303 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55ef0f167890 0x55ef0f169cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:54.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.973+0000 7f39deff5700 10 client.18588 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:08:54.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.973+0000 7f39faffd700 1 --2- 172.21.15.70:0/2307347303 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55ef0f167890 0x55ef0f169cb0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7b3162a8e1c4c676 server_cookie=3bed88ae5f9b8eb6 in_seq=0 out_seq=0 2024-06-04T05:08:54.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.973+0000 7f39deff5700 1 -- 172.21.15.70:0/2307347303 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ef0f1119e0 con 0x55ef0f167890 2024-06-04T05:08:54.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.973+0000 7f39deff5700 10 client.18588 handle_command_reply: tid=0 2024-06-04T05:08:54.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.973+0000 7f3a0c057b80 1 client.18588 shutdown 2024-06-04T05:08:54.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.974+0000 7f3a0c057b80 1 -- 172.21.15.70:0/2307347303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f146210 msgr2=0x55ef0f14e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:54.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.974+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/2307347303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f146210 0x55ef0f14e870 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f39f0009990 tx=0x7f39f0005800 comp rx=0 tx=0).stop 2024-06-04T05:08:54.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.974+0000 7f3a0c057b80 1 -- 172.21.15.70:0/2307347303 shutdown_connections 2024-06-04T05:08:54.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.974+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/2307347303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f149f60 0x55ef0f14edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.974+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/2307347303 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55ef0f167890 0x55ef0f169cb0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.974+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/2307347303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f145860 0x55ef0f151500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.974+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/2307347303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f146210 0x55ef0f14e870 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.974+0000 7f39deff5700 0 client.18588 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:08:54.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.974+0000 7f3a0c057b80 1 -- 172.21.15.70:0/2307347303 >> 172.21.15.70:0/2307347303 conn(0x55ef0f073080 msgr2=0x55ef0f13f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:54.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.974+0000 7f3a0c057b80 1 -- 172.21.15.70:0/2307347303 shutdown_connections 2024-06-04T05:08:54.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.974+0000 7f3a0c057b80 1 -- 172.21.15.70:0/2307347303 wait complete. 2024-06-04T05:08:54.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.974+0000 7f3a0c057b80 20 client.18588 trim_cache size 0 max 16384 2024-06-04T05:08:54.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.990+0000 7f3a0c057b80 1 Processor -- start 2024-06-04T05:08:54.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.990+0000 7f3a0c057b80 1 -- start start 2024-06-04T05:08:54.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.990+0000 7f3a0c057b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 0x55ef0f163a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.990+0000 7f3a0c057b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f146210 0x55ef0f13fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.990+0000 7f3a0c057b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f149f60 0x55ef0f16f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.990+0000 7f3a0c057b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39fc1abcd0 con 0x55ef0f145860 2024-06-04T05:08:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.990+0000 7f3a0c057b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39fc1ad870 con 0x55ef0f149f60 2024-06-04T05:08:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.990+0000 7f3a0c057b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f39fc1ac3a0 con 0x55ef0f146210 2024-06-04T05:08:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.990+0000 7f39faffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 0x55ef0f163a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.990+0000 7f39faffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 0x55ef0f163a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38304/0 (socket says 172.21.15.70:38304) 2024-06-04T05:08:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.991+0000 7f39faffd700 1 -- 172.21.15.70:0/2120634555 learned_addr learned my addr 172.21.15.70:0/2120634555 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.991+0000 7f39fa7fc700 1 --2- 172.21.15.70:0/2120634555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f146210 0x55ef0f13fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.991+0000 7f39fb7fe700 1 --2- 172.21.15.70:0/2120634555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f149f60 0x55ef0f16f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.991+0000 7f39faffd700 1 -- 172.21.15.70:0/2120634555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f146210 msgr2=0x55ef0f13fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.991+0000 7f39faffd700 1 --2- 172.21.15.70:0/2120634555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f146210 0x55ef0f13fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.991+0000 7f39faffd700 1 -- 172.21.15.70:0/2120634555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f149f60 msgr2=0x55ef0f16f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.991+0000 7f39faffd700 1 --2- 172.21.15.70:0/2120634555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f149f60 0x55ef0f16f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:54.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.991+0000 7f39faffd700 1 -- 172.21.15.70:0/2120634555 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef0f141970 con 0x55ef0f145860 2024-06-04T05:08:54.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.991+0000 7f39fa7fc700 1 --2- 172.21.15.70:0/2120634555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f146210 0x55ef0f13fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:54.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.991+0000 7f39faffd700 1 --2- 172.21.15.70:0/2120634555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 0x55ef0f163a50 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f39ec005c20 tx=0x7f39ec013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7999876f36b0f9b5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:54.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.991+0000 7f39e0ff9700 1 -- 172.21.15.70:0/2120634555 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ec02a670 con 0x55ef0f145860 2024-06-04T05:08:54.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.991+0000 7f39e0ff9700 1 -- 172.21.15.70:0/2120634555 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39ec030070 con 0x55ef0f145860 2024-06-04T05:08:54.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.991+0000 7f39e0ff9700 1 -- 172.21.15.70:0/2120634555 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ec01fc60 con 0x55ef0f145860 2024-06-04T05:08:55.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.992+0000 7f3a0c057b80 1 -- 172.21.15.70:0/2120634555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 msgr2=0x55ef0f163a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:55.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.992+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/2120634555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 0x55ef0f163a50 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f39ec005c20 tx=0x7f39ec013000 comp rx=0 tx=0).stop 2024-06-04T05:08:55.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.992+0000 7f3a0c057b80 1 -- 172.21.15.70:0/2120634555 shutdown_connections 2024-06-04T05:08:55.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.992+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/2120634555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f149f60 0x55ef0f16f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:55.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.992+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/2120634555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 0x55ef0f163a50 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:55.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.992+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/2120634555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f146210 0x55ef0f13fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:55.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.992+0000 7f3a0c057b80 1 -- 172.21.15.70:0/2120634555 >> 172.21.15.70:0/2120634555 conn(0x55ef0f073080 msgr2=0x55ef0f150930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:55.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.992+0000 7f3a0c057b80 1 -- 172.21.15.70:0/2120634555 shutdown_connections 2024-06-04T05:08:55.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.992+0000 7f3a0c057b80 1 -- 172.21.15.70:0/2120634555 wait complete. 2024-06-04T05:08:55.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.993+0000 7f3a0c057b80 1 -- start start 2024-06-04T05:08:55.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.993+0000 7f3a0c057b80 1 Processor -- start 2024-06-04T05:08:55.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.993+0000 7f3a0c057b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 0x55ef0f15fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:55.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.993+0000 7f3a0c057b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f146210 0x55ef0f1603d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:55.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.993+0000 7f3a0c057b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f149f60 0x55ef0f14b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:55.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.993+0000 7f3a0c057b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39fc1ad170 con 0x55ef0f145860 2024-06-04T05:08:55.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.993+0000 7f3a0c057b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39fc1aca70 con 0x55ef0f149f60 2024-06-04T05:08:55.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.993+0000 7f3a0c057b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f39fc1a9ce0 con 0x55ef0f146210 2024-06-04T05:08:55.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.993+0000 7f3a0c057b80 10 client.0 _ll_register_callbacks cb 0x7f39f0005350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:55.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.993+0000 7f39faffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 0x55ef0f15fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:55.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.993+0000 7f39fa7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f146210 0x55ef0f1603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:55.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.993+0000 7f39faffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 0x55ef0f15fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38318/0 (socket says 172.21.15.70:38318) 2024-06-04T05:08:55.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.993+0000 7f39fa7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f146210 0x55ef0f1603d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:55704/0 (socket says 172.21.15.70:55704) 2024-06-04T05:08:55.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.993+0000 7f39faffd700 1 -- 172.21.15.70:0/3506073483 learned_addr learned my addr 172.21.15.70:0/3506073483 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:55.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.993+0000 7f39fb7fe700 1 --2- 172.21.15.70:0/3506073483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f149f60 0x55ef0f14b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:55.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.994+0000 7f39fa7fc700 1 -- 172.21.15.70:0/3506073483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f149f60 msgr2=0x55ef0f14b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:55.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.994+0000 7f39fa7fc700 1 --2- 172.21.15.70:0/3506073483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f149f60 0x55ef0f14b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:55.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.994+0000 7f39fa7fc700 1 -- 172.21.15.70:0/3506073483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 msgr2=0x55ef0f15fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:55.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.994+0000 7f39fa7fc700 1 --2- 172.21.15.70:0/3506073483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 0x55ef0f15fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:55.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.994+0000 7f39fa7fc700 1 -- 172.21.15.70:0/3506073483 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef0f14ba10 con 0x55ef0f146210 2024-06-04T05:08:55.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.994+0000 7f39faffd700 1 --2- 172.21.15.70:0/3506073483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 0x55ef0f15fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:08:55.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.994+0000 7f39fa7fc700 1 --2- 172.21.15.70:0/3506073483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f146210 0x55ef0f1603d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f39f000fcc0 tx=0x7f39f00235b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cb9e23ccc82b6e1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:55.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.994+0000 7f39deff5700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:08:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.994+0000 7f39deff5700 1 -- 172.21.15.70:0/3506073483 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f000be70 con 0x55ef0f146210 2024-06-04T05:08:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.994+0000 7f3a0c057b80 1 -- 172.21.15.70:0/3506073483 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39ec030070 con 0x55ef0f146210 2024-06-04T05:08:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.994+0000 7f39deff5700 1 -- 172.21.15.70:0/3506073483 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f39f0003b70 con 0x55ef0f146210 2024-06-04T05:08:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.994+0000 7f39deff5700 1 -- 172.21.15.70:0/3506073483 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f39f0043050 con 0x55ef0f146210 2024-06-04T05:08:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.994+0000 7f3a0c057b80 10 client.27727 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.994+0000 7f3a0c057b80 1 -- 172.21.15.70:0/3506073483 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ef0f1d1f70 con 0x55ef0f146210 2024-06-04T05:08:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.994+0000 7f39deff5700 1 -- 172.21.15.70:0/3506073483 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39ec030070 con 0x55ef0f146210 2024-06-04T05:08:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.995+0000 7f39deff5700 1 -- 172.21.15.70:0/3506073483 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f39f0023740 con 0x55ef0f146210 2024-06-04T05:08:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.995+0000 7f3a0c057b80 10 client.27727 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.995+0000 7f3a0c057b80 10 client.27727 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:08:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.995+0000 7f3a0c057b80 20 client.27727 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.995+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/3506073483 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55ef0f1d25b0 0x55ef0f1d49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.995+0000 7f3a0c057b80 4 client.27727 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:08:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.995+0000 7f3a0c057b80 1 -- 172.21.15.70:0/3506073483 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ef0f0a6be0 con 0x55ef0f1d25b0 2024-06-04T05:08:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.995+0000 7f39faffd700 1 --2- 172.21.15.70:0/3506073483 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55ef0f1d25b0 0x55ef0f1d49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.995+0000 7f39deff5700 10 client.27727 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:08:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.995+0000 7f39faffd700 1 --2- 172.21.15.70:0/3506073483 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55ef0f1d25b0 0x55ef0f1d49d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=939e224317409354 server_cookie=1d7be9a36e6093a8 in_seq=0 out_seq=0 2024-06-04T05:08:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.995+0000 7f39deff5700 1 -- 172.21.15.70:0/3506073483 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ef0f0a6be0 con 0x55ef0f1d25b0 2024-06-04T05:08:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f39deff5700 10 client.27727 handle_command_reply: tid=0 2024-06-04T05:08:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f3a0c057b80 1 client.27727 shutdown 2024-06-04T05:08:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f3a0c057b80 1 -- 172.21.15.70:0/3506073483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f146210 msgr2=0x55ef0f1603d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/3506073483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f146210 0x55ef0f1603d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f39f000fcc0 tx=0x7f39f00235b0 comp rx=0 tx=0).stop 2024-06-04T05:08:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f3a0c057b80 1 -- 172.21.15.70:0/3506073483 shutdown_connections 2024-06-04T05:08:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/3506073483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef0f149f60 0x55ef0f14b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/3506073483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef0f145860 0x55ef0f15fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/3506073483 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55ef0f1d25b0 0x55ef0f1d49d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f3a0c057b80 1 --2- 172.21.15.70:0/3506073483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef0f146210 0x55ef0f1603d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f39deff5700 0 client.27727 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:08:55.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f3a0c057b80 1 -- 172.21.15.70:0/3506073483 >> 172.21.15.70:0/3506073483 conn(0x55ef0f073080 msgr2=0x55ef0f1488b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:55.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f3a0c057b80 1 -- 172.21.15.70:0/3506073483 shutdown_connections 2024-06-04T05:08:55.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f3a0c057b80 1 -- 172.21.15.70:0/3506073483 wait complete. 2024-06-04T05:08:55.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f3a0c057b80 20 client.27727 trim_cache size 0 max 16384 2024-06-04T05:08:55.005 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:08:55.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f3a017c2700 1 -- 172.21.15.70:0/1833597604 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39e40b29f0 msgr2=0x7f39e40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:55.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f3a017c2700 1 --2- 172.21.15.70:0/1833597604 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39e40b29f0 0x7f39e40b4eb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f39fc19ab00 tx=0x7f39ec00b500 comp rx=0 tx=0).stop 2024-06-04T05:08:55.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f3a017c2700 1 -- 172.21.15.70:0/1833597604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39fc1095a0 msgr2=0x7f39fc1999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:55.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.996+0000 7f3a017c2700 1 --2- 172.21.15.70:0/1833597604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39fc1095a0 0x7f39fc1999c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f39f400efc0 tx=0x7f39f400c3a0 comp rx=0 tx=0).stop 2024-06-04T05:08:55.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.997+0000 7f3a017c2700 1 -- 172.21.15.70:0/1833597604 shutdown_connections 2024-06-04T05:08:55.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.997+0000 7f3a017c2700 1 --2- 172.21.15.70:0/1833597604 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39e40b29f0 0x7f39e40b4eb0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:55.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.997+0000 7f3a017c2700 1 --2- 172.21.15.70:0/1833597604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39fc1095a0 0x7f39fc1999c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:55.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.997+0000 7f3a017c2700 1 --2- 172.21.15.70:0/1833597604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39fc06bab0 0x7f39fc19f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:55.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.997+0000 7f3a017c2700 1 --2- 172.21.15.70:0/1833597604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39fc06c470 0x7f39fc19fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:55.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.997+0000 7f3a017c2700 1 -- 172.21.15.70:0/1833597604 >> 172.21.15.70:0/1833597604 conn(0x7f39fc0793b0 msgr2=0x7f39fc1012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:55.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.998+0000 7f3a017c2700 1 -- 172.21.15.70:0/1833597604 shutdown_connections 2024-06-04T05:08:55.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:54.998+0000 7f3a017c2700 1 -- 172.21.15.70:0/1833597604 wait complete. 2024-06-04T05:08:55.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:55 smithi070 ceph-mon[32631]: pgmap v363: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 33 KiB/s wr, 13 op/s 2024-06-04T05:08:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:55 smithi112 ceph-mon[36453]: pgmap v363: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 33 KiB/s wr, 13 op/s 2024-06-04T05:08:55.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 198061 1.76 MB/sec execute 468 sec latency 37.053 ms 2024-06-04T05:08:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:55 smithi098 ceph-mon[35569]: pgmap v363: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 33 KiB/s wr, 13 op/s 2024-06-04T05:08:55.751 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:08:56.014 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:08:56.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 198591 1.76 MB/sec execute 469 sec latency 15.139 ms 2024-06-04T05:08:57.436 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:57 smithi070 ceph-mon[32631]: pgmap v364: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 569 KiB/s wr, 53 op/s 2024-06-04T05:08:57.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.671+0000 7f4c77513700 1 -- 172.21.15.70:0/2008404159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c70102470 msgr2=0x7f4c701028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:57.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.671+0000 7f4c77513700 1 --2- 172.21.15.70:0/2008404159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c70102470 0x7f4c701028f0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f4c60009990 tx=0x7f4c60009ca0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.671+0000 7f4c77513700 1 -- 172.21.15.70:0/2008404159 shutdown_connections 2024-06-04T05:08:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.671+0000 7f4c77513700 1 --2- 172.21.15.70:0/2008404159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c70102e30 0x7f4c7010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.671+0000 7f4c77513700 1 --2- 172.21.15.70:0/2008404159 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c701084b0 0x7f4c70108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.671+0000 7f4c77513700 1 --2- 172.21.15.70:0/2008404159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c70102470 0x7f4c701028f0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.671+0000 7f4c77513700 1 -- 172.21.15.70:0/2008404159 >> 172.21.15.70:0/2008404159 conn(0x7f4c700793f0 msgr2=0x7f4c70079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.671+0000 7f4c77513700 1 -- 172.21.15.70:0/2008404159 shutdown_connections 2024-06-04T05:08:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.671+0000 7f4c77513700 1 -- 172.21.15.70:0/2008404159 wait complete. 2024-06-04T05:08:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.671+0000 7f4c77513700 1 Processor -- start 2024-06-04T05:08:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.671+0000 7f4c77513700 1 -- start start 2024-06-04T05:08:57.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c77513700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c70102470 0x7f4c70104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c77513700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c70102e30 0x7f4c701045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c77513700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c701084b0 0x7f4c70104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c77513700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4c701124f0 con 0x7f4c701084b0 2024-06-04T05:08:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c77513700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4c70112370 con 0x7f4c70102e30 2024-06-04T05:08:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c77513700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4c70112670 con 0x7f4c70102470 2024-06-04T05:08:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c75ab0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c701084b0 0x7f4c70104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c75ab0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c701084b0 0x7f4c70104b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50954/0 (socket says 172.21.15.70:50954) 2024-06-04T05:08:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c75ab0700 1 -- 172.21.15.70:0/2845329753 learned_addr learned my addr 172.21.15.70:0/2845329753 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c74aae700 1 --2- 172.21.15.70:0/2845329753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c70102e30 0x7f4c701045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c752af700 1 --2- 172.21.15.70:0/2845329753 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c70102470 0x7f4c70104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c75ab0700 1 -- 172.21.15.70:0/2845329753 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c70102470 msgr2=0x7f4c70104080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c75ab0700 1 --2- 172.21.15.70:0/2845329753 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c70102470 0x7f4c70104080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c75ab0700 1 -- 172.21.15.70:0/2845329753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c70102e30 msgr2=0x7f4c701045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c75ab0700 1 --2- 172.21.15.70:0/2845329753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c70102e30 0x7f4c701045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c75ab0700 1 -- 172.21.15.70:0/2845329753 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c701a80f0 con 0x7f4c701084b0 2024-06-04T05:08:57.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c75ab0700 1 --2- 172.21.15.70:0/2845329753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c701084b0 0x7f4c70104b00 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f4c68007d70 tx=0x7f4c6800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:57.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c667fc700 1 -- 172.21.15.70:0/2845329753 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c68016020 con 0x7f4c701084b0 2024-06-04T05:08:57.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c667fc700 1 -- 172.21.15.70:0/2845329753 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4c6800ac90 con 0x7f4c701084b0 2024-06-04T05:08:57.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c77513700 1 -- 172.21.15.70:0/2845329753 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c701a83e0 con 0x7f4c701084b0 2024-06-04T05:08:57.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.672+0000 7f4c667fc700 1 -- 172.21.15.70:0/2845329753 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c680115e0 con 0x7f4c701084b0 2024-06-04T05:08:57.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.673+0000 7f4c77513700 1 -- 172.21.15.70:0/2845329753 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c701a88f0 con 0x7f4c701084b0 2024-06-04T05:08:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.675+0000 7f4c667fc700 1 -- 172.21.15.70:0/2845329753 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f4c68019020 con 0x7f4c701084b0 2024-06-04T05:08:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.675+0000 7f4c667fc700 1 --2- 172.21.15.70:0/2845329753 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4c5c0b2a70 0x7f4c5c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.675+0000 7f4c752af700 1 --2- 172.21.15.70:0/2845329753 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4c5c0b2a70 0x7f4c5c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.676+0000 7f4c667fc700 1 -- 172.21.15.70:0/2845329753 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f4c680e2440 con 0x7f4c701084b0 2024-06-04T05:08:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.676+0000 7f4c752af700 1 --2- 172.21.15.70:0/2845329753 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4c5c0b2a70 0x7f4c5c0b4f30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009a10 tx=0x7f4c6c009190 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.676+0000 7f4c81da8b80 1 Processor -- start 2024-06-04T05:08:57.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.676+0000 7f4c81da8b80 1 -- start start 2024-06-04T05:08:57.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.676+0000 7f4c81da8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08536320 0x55cd08536700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.676+0000 7f4c81da8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd08530d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.676+0000 7f4c81da8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08531260 0x55cd0853d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.676+0000 7f4c81da8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cd0848ed70 con 0x55cd08536320 2024-06-04T05:08:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.676+0000 7f4c81da8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cd0847e910 con 0x55cd085308a0 2024-06-04T05:08:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.676+0000 7f4c81da8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cd08482950 con 0x55cd08531260 2024-06-04T05:08:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.677+0000 7f4c752af700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08536320 0x55cd08536700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.677+0000 7f4c752af700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08536320 0x55cd08536700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50958/0 (socket says 172.21.15.70:50958) 2024-06-04T05:08:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.677+0000 7f4c752af700 1 -- 172.21.15.70:0/714802042 learned_addr learned my addr 172.21.15.70:0/714802042 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.677+0000 7f4c75ab0700 1 --2- 172.21.15.70:0/714802042 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08531260 0x55cd0853d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.677+0000 7f4c74aae700 1 --2- 172.21.15.70:0/714802042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd08530d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.677+0000 7f4c752af700 1 -- 172.21.15.70:0/714802042 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08531260 msgr2=0x55cd0853d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.677+0000 7f4c752af700 1 --2- 172.21.15.70:0/714802042 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08531260 0x55cd0853d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.677+0000 7f4c752af700 1 -- 172.21.15.70:0/714802042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 msgr2=0x55cd08530d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.677+0000 7f4c752af700 1 --2- 172.21.15.70:0/714802042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd08530d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.677+0000 7f4c752af700 1 -- 172.21.15.70:0/714802042 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd08427fb0 con 0x55cd08536320 2024-06-04T05:08:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.677+0000 7f4c752af700 1 --2- 172.21.15.70:0/714802042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08536320 0x55cd08536700 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c00ea00 tx=0x7f4c6c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56410b70b7a980bd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.677+0000 7f4c5b7fe700 1 -- 172.21.15.70:0/714802042 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c01b040 con 0x55cd08536320 2024-06-04T05:08:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.677+0000 7f4c5b7fe700 1 -- 172.21.15.70:0/714802042 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4c6c024070 con 0x55cd08536320 2024-06-04T05:08:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.677+0000 7f4c5b7fe700 1 -- 172.21.15.70:0/714802042 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c013650 con 0x55cd08536320 2024-06-04T05:08:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.678+0000 7f4c81da8b80 1 -- 172.21.15.70:0/714802042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08536320 msgr2=0x55cd08536700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.678+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/714802042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08536320 0x55cd08536700 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c00ea00 tx=0x7f4c6c011000 comp rx=0 tx=0).stop 2024-06-04T05:08:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.678+0000 7f4c81da8b80 1 -- 172.21.15.70:0/714802042 shutdown_connections 2024-06-04T05:08:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.678+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/714802042 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08531260 0x55cd0853d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.678+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/714802042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08536320 0x55cd08536700 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.678+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/714802042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd08530d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.678+0000 7f4c81da8b80 1 -- 172.21.15.70:0/714802042 >> 172.21.15.70:0/714802042 conn(0x55cd0845c240 msgr2=0x55cd0852b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.678+0000 7f4c81da8b80 1 -- 172.21.15.70:0/714802042 shutdown_connections 2024-06-04T05:08:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.678+0000 7f4c81da8b80 1 -- 172.21.15.70:0/714802042 wait complete. 2024-06-04T05:08:57.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.678+0000 7f4c81da8b80 1 -- start start 2024-06-04T05:08:57.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.678+0000 7f4c81da8b80 1 Processor -- start 2024-06-04T05:08:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c81da8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd0853cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c81da8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08531260 0x55cd08539fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c81da8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08536320 0x55cd0853a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c81da8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cd08384fa0 con 0x55cd08536320 2024-06-04T05:08:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c81da8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cd083e6da0 con 0x55cd085308a0 2024-06-04T05:08:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c81da8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cd0853d6e0 con 0x55cd08531260 2024-06-04T05:08:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c81da8b80 10 client.0 _ll_register_callbacks cb 0x55cd08419950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c75ab0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08536320 0x55cd0853a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c75ab0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08536320 0x55cd0853a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50974/0 (socket says 172.21.15.70:50974) 2024-06-04T05:08:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c75ab0700 1 -- 172.21.15.70:0/2182517560 learned_addr learned my addr 172.21.15.70:0/2182517560 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c752af700 1 --2- 172.21.15.70:0/2182517560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd0853cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c74aae700 1 --2- 172.21.15.70:0/2182517560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08531260 0x55cd08539fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c75ab0700 1 -- 172.21.15.70:0/2182517560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08531260 msgr2=0x55cd08539fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c75ab0700 1 --2- 172.21.15.70:0/2182517560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08531260 0x55cd08539fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c75ab0700 1 -- 172.21.15.70:0/2182517560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 msgr2=0x55cd0853cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c75ab0700 1 --2- 172.21.15.70:0/2182517560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd0853cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c75ab0700 1 -- 172.21.15.70:0/2182517560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd083ce8d0 con 0x55cd08536320 2024-06-04T05:08:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c75ab0700 1 --2- 172.21.15.70:0/2182517560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08536320 0x55cd0853a4e0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f4c680e0160 tx=0x7f4c6800ae40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a543fd19902dec95 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.679+0000 7f4c58ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:08:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.680+0000 7f4c58ff9700 1 -- 172.21.15.70:0/2182517560 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c680c3060 con 0x55cd08536320 2024-06-04T05:08:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.680+0000 7f4c58ff9700 1 -- 172.21.15.70:0/2182517560 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f4c680b21d0 con 0x55cd08536320 2024-06-04T05:08:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.680+0000 7f4c81da8b80 1 -- 172.21.15.70:0/2182517560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c6c024070 con 0x55cd08536320 2024-06-04T05:08:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.680+0000 7f4c58ff9700 1 -- 172.21.15.70:0/2182517560 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4c680a9f00 con 0x55cd08536320 2024-06-04T05:08:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.680+0000 7f4c81da8b80 10 client.18606 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.680+0000 7f4c81da8b80 1 -- 172.21.15.70:0/2182517560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cd08438760 con 0x55cd08536320 2024-06-04T05:08:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.680+0000 7f4c58ff9700 1 -- 172.21.15.70:0/2182517560 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4c6c024070 con 0x55cd08536320 2024-06-04T05:08:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.680+0000 7f4c58ff9700 1 -- 172.21.15.70:0/2182517560 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f4c680910f0 con 0x55cd08536320 2024-06-04T05:08:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.680+0000 7f4c81da8b80 10 client.18606 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.680+0000 7f4c81da8b80 10 client.18606 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:08:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.680+0000 7f4c81da8b80 20 client.18606 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.680+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/2182517560 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55cd0854a240 0x55cd0854c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.680+0000 7f4c81da8b80 4 client.18606 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:08:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.680+0000 7f4c81da8b80 1 -- 172.21.15.70:0/2182517560 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd08490cc0 con 0x55cd0854a240 2024-06-04T05:08:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.681+0000 7f4c752af700 1 --2- 172.21.15.70:0/2182517560 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55cd0854a240 0x55cd0854c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.681+0000 7f4c58ff9700 10 client.18606 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:08:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.681+0000 7f4c752af700 1 --2- 172.21.15.70:0/2182517560 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55cd0854a240 0x55cd0854c650 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=385d71e49c4b96aa server_cookie=e02815354e76ca9 in_seq=0 out_seq=0 2024-06-04T05:08:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.682+0000 7f4c58ff9700 1 -- 172.21.15.70:0/2182517560 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cd08490cc0 con 0x55cd0854a240 2024-06-04T05:08:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.682+0000 7f4c58ff9700 10 client.18606 handle_command_reply: tid=0 2024-06-04T05:08:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.682+0000 7f4c81da8b80 1 client.18606 shutdown 2024-06-04T05:08:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.682+0000 7f4c81da8b80 1 -- 172.21.15.70:0/2182517560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08536320 msgr2=0x55cd0853a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.682+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/2182517560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08536320 0x55cd0853a4e0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f4c680e0160 tx=0x7f4c6800ae40 comp rx=0 tx=0).stop 2024-06-04T05:08:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.682+0000 7f4c81da8b80 1 -- 172.21.15.70:0/2182517560 shutdown_connections 2024-06-04T05:08:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.682+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/2182517560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08536320 0x55cd0853a4e0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.682+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/2182517560 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55cd0854a240 0x55cd0854c650 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.682+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/2182517560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd0853cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.682+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/2182517560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08531260 0x55cd08539fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.682+0000 7f4c58ff9700 0 client.18606 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:08:57.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.682+0000 7f4c81da8b80 1 -- 172.21.15.70:0/2182517560 >> 172.21.15.70:0/2182517560 conn(0x55cd0845c240 msgr2=0x55cd0852aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:57.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.682+0000 7f4c81da8b80 1 -- 172.21.15.70:0/2182517560 shutdown_connections 2024-06-04T05:08:57.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.682+0000 7f4c81da8b80 1 -- 172.21.15.70:0/2182517560 wait complete. 2024-06-04T05:08:57.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.682+0000 7f4c81da8b80 20 client.18606 trim_cache size 0 max 16384 2024-06-04T05:08:57.690 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:57 smithi112 ceph-mon[36453]: pgmap v364: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 569 KiB/s wr, 53 op/s 2024-06-04T05:08:57.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.697+0000 7f4c81da8b80 1 Processor -- start 2024-06-04T05:08:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.697+0000 7f4c81da8b80 1 -- start start 2024-06-04T05:08:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.697+0000 7f4c81da8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd085345e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.697+0000 7f4c81da8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08531260 0x55cd0852b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.697+0000 7f4c81da8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08536320 0x55cd0852f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.697+0000 7f4c81da8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4c701b0040 con 0x55cd08531260 2024-06-04T05:08:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.697+0000 7f4c81da8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4c701b1be0 con 0x55cd085308a0 2024-06-04T05:08:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.697+0000 7f4c81da8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4c701b0710 con 0x55cd08536320 2024-06-04T05:08:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.697+0000 7f4c74aae700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08531260 0x55cd0852b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.697+0000 7f4c74aae700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08531260 0x55cd0852b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50980/0 (socket says 172.21.15.70:50980) 2024-06-04T05:08:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.698+0000 7f4c74aae700 1 -- 172.21.15.70:0/3835487254 learned_addr learned my addr 172.21.15.70:0/3835487254 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.698+0000 7f4c752af700 1 --2- 172.21.15.70:0/3835487254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd085345e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.698+0000 7f4c75ab0700 1 --2- 172.21.15.70:0/3835487254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08536320 0x55cd0852f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.698+0000 7f4c74aae700 1 -- 172.21.15.70:0/3835487254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08536320 msgr2=0x55cd0852f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.698+0000 7f4c74aae700 1 --2- 172.21.15.70:0/3835487254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08536320 0x55cd0852f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.698+0000 7f4c74aae700 1 -- 172.21.15.70:0/3835487254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 msgr2=0x55cd085345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.698+0000 7f4c74aae700 1 --2- 172.21.15.70:0/3835487254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd085345e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.698+0000 7f4c74aae700 1 -- 172.21.15.70:0/3835487254 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd0854cc80 con 0x55cd08531260 2024-06-04T05:08:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.698+0000 7f4c74aae700 1 --2- 172.21.15.70:0/3835487254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08531260 0x55cd0852b270 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f4c60009990 tx=0x7f4c60009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c157d4b9d558106 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.698+0000 7f4c5affd700 1 -- 172.21.15.70:0/3835487254 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c60028070 con 0x55cd08531260 2024-06-04T05:08:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.698+0000 7f4c5affd700 1 -- 172.21.15.70:0/3835487254 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4c60005040 con 0x55cd08531260 2024-06-04T05:08:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.698+0000 7f4c5affd700 1 -- 172.21.15.70:0/3835487254 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6000ba30 con 0x55cd08531260 2024-06-04T05:08:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.699+0000 7f4c81da8b80 1 -- 172.21.15.70:0/3835487254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08531260 msgr2=0x55cd0852b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.699+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/3835487254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08531260 0x55cd0852b270 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f4c60009990 tx=0x7f4c60009f90 comp rx=0 tx=0).stop 2024-06-04T05:08:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.699+0000 7f4c81da8b80 1 -- 172.21.15.70:0/3835487254 shutdown_connections 2024-06-04T05:08:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.699+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/3835487254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08536320 0x55cd0852f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.699+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/3835487254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd085345e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.699+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/3835487254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08531260 0x55cd0852b270 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.699+0000 7f4c81da8b80 1 -- 172.21.15.70:0/3835487254 >> 172.21.15.70:0/3835487254 conn(0x55cd0845c240 msgr2=0x55cd08535740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.699+0000 7f4c81da8b80 1 -- 172.21.15.70:0/3835487254 shutdown_connections 2024-06-04T05:08:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.699+0000 7f4c81da8b80 1 -- 172.21.15.70:0/3835487254 wait complete. 2024-06-04T05:08:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.699+0000 7f4c81da8b80 1 -- start start 2024-06-04T05:08:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.699+0000 7f4c81da8b80 1 Processor -- start 2024-06-04T05:08:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.700+0000 7f4c81da8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd08542960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.700+0000 7f4c81da8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08531260 0x55cd08542ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.700+0000 7f4c81da8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08536320 0x55cd0853f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.700+0000 7f4c81da8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4c701b14e0 con 0x55cd08531260 2024-06-04T05:08:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.700+0000 7f4c81da8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4c701b0de0 con 0x55cd085308a0 2024-06-04T05:08:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.700+0000 7f4c81da8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4c701ae000 con 0x55cd08536320 2024-06-04T05:08:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.700+0000 7f4c81da8b80 10 client.0 _ll_register_callbacks cb 0x55cd08419950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:08:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.700+0000 7f4c752af700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd08542960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.700+0000 7f4c752af700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd08542960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46454/0 (socket says 172.21.15.70:46454) 2024-06-04T05:08:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.700+0000 7f4c752af700 1 -- 172.21.15.70:0/1717946799 learned_addr learned my addr 172.21.15.70:0/1717946799 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:08:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.700+0000 7f4c75ab0700 1 --2- 172.21.15.70:0/1717946799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08536320 0x55cd0853f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.700+0000 7f4c752af700 1 -- 172.21.15.70:0/1717946799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08536320 msgr2=0x55cd0853f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:08:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.700+0000 7f4c752af700 1 --2- 172.21.15.70:0/1717946799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08536320 0x55cd0853f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.700+0000 7f4c752af700 1 -- 172.21.15.70:0/1717946799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08531260 msgr2=0x55cd08542ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-04T05:08:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.700+0000 7f4c752af700 1 --2- 172.21.15.70:0/1717946799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08531260 0x55cd08542ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.700+0000 7f4c752af700 1 -- 172.21.15.70:0/1717946799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd0853fe70 con 0x55cd085308a0 2024-06-04T05:08:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.700+0000 7f4c75ab0700 1 --2- 172.21.15.70:0/1717946799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08536320 0x55cd0853f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:08:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.701+0000 7f4c752af700 1 --2- 172.21.15.70:0/1717946799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd08542960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c015e30 tx=0x7f4c6c006e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4480504a762ea1e6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:08:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.701+0000 7f4c58ff9700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:08:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.701+0000 7f4c58ff9700 1 -- 172.21.15.70:0/1717946799 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c038070 con 0x55cd085308a0 2024-06-04T05:08:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.701+0000 7f4c81da8b80 1 -- 172.21.15.70:0/1717946799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c60005040 con 0x55cd085308a0 2024-06-04T05:08:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.701+0000 7f4c58ff9700 1 -- 172.21.15.70:0/1717946799 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7f4c6c015a10 con 0x55cd085308a0 2024-06-04T05:08:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.701+0000 7f4c58ff9700 1 -- 172.21.15.70:0/1717946799 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4c6c044050 con 0x55cd085308a0 2024-06-04T05:08:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.701+0000 7f4c81da8b80 10 client.27623 fetch_fsmap learned FSMap version 34 2024-06-04T05:08:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.701+0000 7f4c81da8b80 1 -- 172.21.15.70:0/1717946799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cd085b5530 con 0x55cd085308a0 2024-06-04T05:08:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.701+0000 7f4c58ff9700 1 -- 172.21.15.70:0/1717946799 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4c60005040 con 0x55cd085308a0 2024-06-04T05:08:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.701+0000 7f4c58ff9700 1 -- 172.21.15.70:0/1717946799 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f4c6c01e030 con 0x55cd085308a0 2024-06-04T05:08:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.702+0000 7f4c81da8b80 10 client.27623 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:08:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.702+0000 7f4c81da8b80 10 client.27623 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:08:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.702+0000 7f4c81da8b80 20 client.27623 populate_metadata read hostname 'smithi070' 2024-06-04T05:08:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.702+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/1717946799 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55cd085b59b0 0x55cd085b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:08:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.702+0000 7f4c81da8b80 4 client.27623 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:08:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.702+0000 7f4c81da8b80 1 -- 172.21.15.70:0/1717946799 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cd0848a4a0 con 0x55cd085b59b0 2024-06-04T05:08:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.702+0000 7f4c74aae700 1 --2- 172.21.15.70:0/1717946799 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55cd085b59b0 0x55cd085b7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:08:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.702+0000 7f4c74aae700 1 --2- 172.21.15.70:0/1717946799 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55cd085b59b0 0x55cd085b7df0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=479926b81119bd1 server_cookie=8ca60bae9ffc07c1 in_seq=0 out_seq=0 2024-06-04T05:08:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.702+0000 7f4c58ff9700 10 client.27623 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:08:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.702+0000 7f4c58ff9700 1 -- 172.21.15.70:0/1717946799 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cd0848a4a0 con 0x55cd085b59b0 2024-06-04T05:08:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.702+0000 7f4c58ff9700 10 client.27623 handle_command_reply: tid=0 2024-06-04T05:08:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.702+0000 7f4c81da8b80 1 client.27623 shutdown 2024-06-04T05:08:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.703+0000 7f4c81da8b80 1 -- 172.21.15.70:0/1717946799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 msgr2=0x55cd08542960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.703+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/1717946799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd08542960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c015e30 tx=0x7f4c6c006e10 comp rx=0 tx=0).stop 2024-06-04T05:08:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.703+0000 7f4c81da8b80 1 -- 172.21.15.70:0/1717946799 shutdown_connections 2024-06-04T05:08:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.703+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/1717946799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cd08536320 0x55cd0853f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.703+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/1717946799 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55cd085b59b0 0x55cd085b7df0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.703+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/1717946799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd085308a0 0x55cd08542960 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.703+0000 7f4c81da8b80 1 --2- 172.21.15.70:0/1717946799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd08531260 0x55cd08542ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.703+0000 7f4c81da8b80 1 -- 172.21.15.70:0/1717946799 >> 172.21.15.70:0/1717946799 conn(0x55cd0845c240 msgr2=0x55cd0852f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.703+0000 7f4c58ff9700 0 client.27623 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:08:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.703+0000 7f4c81da8b80 1 -- 172.21.15.70:0/1717946799 shutdown_connections 2024-06-04T05:08:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.703+0000 7f4c81da8b80 1 -- 172.21.15.70:0/1717946799 wait complete. 2024-06-04T05:08:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.703+0000 7f4c81da8b80 20 client.27623 trim_cache size 0 max 16384 2024-06-04T05:08:57.711 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:08:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.703+0000 7f4c77513700 1 -- 172.21.15.70:0/2845329753 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4c5c0b2a70 msgr2=0x7f4c5c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.703+0000 7f4c77513700 1 --2- 172.21.15.70:0/2845329753 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4c5c0b2a70 0x7f4c5c0b4f30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009a10 tx=0x7f4c6c009190 comp rx=0 tx=0).stop 2024-06-04T05:08:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.703+0000 7f4c77513700 1 -- 172.21.15.70:0/2845329753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c701084b0 msgr2=0x7f4c70104b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:08:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.703+0000 7f4c77513700 1 --2- 172.21.15.70:0/2845329753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c701084b0 0x7f4c70104b00 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f4c68007d70 tx=0x7f4c6800a340 comp rx=0 tx=0).stop 2024-06-04T05:08:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.703+0000 7f4c77513700 1 -- 172.21.15.70:0/2845329753 shutdown_connections 2024-06-04T05:08:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.704+0000 7f4c77513700 1 --2- 172.21.15.70:0/2845329753 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4c5c0b2a70 0x7f4c5c0b4f30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.704+0000 7f4c77513700 1 --2- 172.21.15.70:0/2845329753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c701084b0 0x7f4c70104b00 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.704+0000 7f4c77513700 1 --2- 172.21.15.70:0/2845329753 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c70102470 0x7f4c70104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.704+0000 7f4c77513700 1 --2- 172.21.15.70:0/2845329753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c70102e30 0x7f4c701045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:08:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.704+0000 7f4c77513700 1 -- 172.21.15.70:0/2845329753 >> 172.21.15.70:0/2845329753 conn(0x7f4c700793f0 msgr2=0x7f4c7006c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:08:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.704+0000 7f4c77513700 1 -- 172.21.15.70:0/2845329753 shutdown_connections 2024-06-04T05:08:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:08:57.704+0000 7f4c77513700 1 -- 172.21.15.70:0/2845329753 wait complete. 2024-06-04T05:08:57.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 198777 1.76 MB/sec execute 470 sec latency 481.107 ms 2024-06-04T05:08:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:57 smithi098 ceph-mon[35569]: pgmap v364: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 569 KiB/s wr, 53 op/s 2024-06-04T05:08:58.411 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:08:58.648 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:08:58.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 198777 1.76 MB/sec execute 471 sec latency 1481.152 ms 2024-06-04T05:08:59.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:08:59 smithi070 ceph-mon[32631]: pgmap v365: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 559 KiB/s wr, 48 op/s 2024-06-04T05:08:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:08:59 smithi112 ceph-mon[36453]: pgmap v365: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 559 KiB/s wr, 48 op/s 2024-06-04T05:08:59.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 198851 1.76 MB/sec execute 472 sec latency 2325.970 ms 2024-06-04T05:08:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:08:59 smithi098 ceph-mon[35569]: pgmap v365: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 559 KiB/s wr, 48 op/s 2024-06-04T05:09:00.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.310+0000 7ff23b93b700 1 -- 172.21.15.70:0/1534412290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff234103f80 msgr2=0x7ff234110410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:00.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.310+0000 7ff23b93b700 1 --2- 172.21.15.70:0/1534412290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff234103f80 0x7ff234110410 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7ff22c00b0a0 tx=0x7ff22c00b3b0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.310+0000 7ff23b93b700 1 -- 172.21.15.70:0/1534412290 shutdown_connections 2024-06-04T05:09:00.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.310+0000 7ff23b93b700 1 --2- 172.21.15.70:0/1534412290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff234103f80 0x7ff234110410 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.310+0000 7ff23b93b700 1 --2- 172.21.15.70:0/1534412290 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2341095d0 0x7ff2341099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.310+0000 7ff23b93b700 1 --2- 172.21.15.70:0/1534412290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2341035c0 0x7ff234103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.310+0000 7ff23b93b700 1 -- 172.21.15.70:0/1534412290 >> 172.21.15.70:0/1534412290 conn(0x7ff23407a610 msgr2=0x7ff23407aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:00.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.310+0000 7ff23b93b700 1 -- 172.21.15.70:0/1534412290 shutdown_connections 2024-06-04T05:09:00.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.310+0000 7ff23b93b700 1 -- 172.21.15.70:0/1534412290 wait complete. 2024-06-04T05:09:00.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.311+0000 7ff23b93b700 1 Processor -- start 2024-06-04T05:09:00.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.311+0000 7ff23b93b700 1 -- start start 2024-06-04T05:09:00.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.311+0000 7ff23b93b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2341035c0 0x7ff234075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.311+0000 7ff23b93b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff234103f80 0x7ff234075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.311+0000 7ff23b93b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff2341095d0 0x7ff2340762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.311+0000 7ff23b93b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff234113630 con 0x7ff2341095d0 2024-06-04T05:09:00.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.311+0000 7ff23b93b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff2341134b0 con 0x7ff2341035c0 2024-06-04T05:09:00.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.311+0000 7ff23b93b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff2341137b0 con 0x7ff234103f80 2024-06-04T05:09:00.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.311+0000 7ff239ed8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff2341095d0 0x7ff2340762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.311+0000 7ff2396d7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2341035c0 0x7ff234075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.311+0000 7ff2396d7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2341035c0 0x7ff234075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46480/0 (socket says 172.21.15.70:46480) 2024-06-04T05:09:00.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.311+0000 7ff2396d7700 1 -- 172.21.15.70:0/1372706262 learned_addr learned my addr 172.21.15.70:0/1372706262 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:00.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.311+0000 7ff239ed8700 1 --2- 172.21.15.70:0/1372706262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff2341095d0 0x7ff2340762c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50996/0 (socket says 172.21.15.70:50996) 2024-06-04T05:09:00.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.311+0000 7ff238ed6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff234103f80 0x7ff234075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.312+0000 7ff239ed8700 1 -- 172.21.15.70:0/1372706262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff234103f80 msgr2=0x7ff234075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:00.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.312+0000 7ff239ed8700 1 --2- 172.21.15.70:0/1372706262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff234103f80 0x7ff234075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.312+0000 7ff239ed8700 1 -- 172.21.15.70:0/1372706262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2341035c0 msgr2=0x7ff234075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:00.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.312+0000 7ff239ed8700 1 --2- 172.21.15.70:0/1372706262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2341035c0 0x7ff234075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.312+0000 7ff239ed8700 1 -- 172.21.15.70:0/1372706262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2341a7f70 con 0x7ff2341095d0 2024-06-04T05:09:00.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.312+0000 7ff2396d7700 1 --2- 172.21.15.70:0/1372706262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2341035c0 0x7ff234075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:00.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.312+0000 7ff238ed6700 1 --2- 172.21.15.70:0/1372706262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff234103f80 0x7ff234075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:00.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.312+0000 7ff239ed8700 1 --2- 172.21.15.70:0/1372706262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff2341095d0 0x7ff2340762c0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7ff22c00b0a0 tx=0x7ff22c004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:00.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.312+0000 7ff22a7fc700 1 -- 172.21.15.70:0/1372706262 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c021070 con 0x7ff2341095d0 2024-06-04T05:09:00.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.312+0000 7ff22a7fc700 1 -- 172.21.15.70:0/1372706262 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff22c007b80 con 0x7ff2341095d0 2024-06-04T05:09:00.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.312+0000 7ff23b93b700 1 -- 172.21.15.70:0/1372706262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2341a8200 con 0x7ff2341095d0 2024-06-04T05:09:00.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.312+0000 7ff22a7fc700 1 -- 172.21.15.70:0/1372706262 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c005040 con 0x7ff2341095d0 2024-06-04T05:09:00.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.313+0000 7ff23b93b700 1 -- 172.21.15.70:0/1372706262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2341a8600 con 0x7ff2341095d0 2024-06-04T05:09:00.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.316+0000 7ff22a7fc700 1 -- 172.21.15.70:0/1372706262 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff22c024070 con 0x7ff2341095d0 2024-06-04T05:09:00.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.317+0000 7ff22a7fc700 1 --2- 172.21.15.70:0/1372706262 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff2200b29f0 0x7ff2200b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.317+0000 7ff2396d7700 1 --2- 172.21.15.70:0/1372706262 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff2200b29f0 0x7ff2200b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.317+0000 7ff22a7fc700 1 -- 172.21.15.70:0/1372706262 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7ff22c00f020 con 0x7ff2341095d0 2024-06-04T05:09:00.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.317+0000 7ff2461d0b80 1 Processor -- start 2024-06-04T05:09:00.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff2461d0b80 1 -- start start 2024-06-04T05:09:00.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff2396d7700 1 --2- 172.21.15.70:0/1372706262 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff2200b29f0 0x7ff2200b4eb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff230011550 tx=0x7ff230009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:00.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff2461d0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0333860 0x5630c0333c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff2461d0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0334210 0x5630c0337940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff2461d0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c03383a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff2461d0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5630c0292c90 con 0x5630c0337f60 2024-06-04T05:09:00.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff2461d0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5630c0282830 con 0x5630c0333860 2024-06-04T05:09:00.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff2461d0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5630c0286870 con 0x5630c0334210 2024-06-04T05:09:00.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff239ed8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c03383a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff238ed6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0334210 0x5630c0337940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff239ed8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c03383a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51004/0 (socket says 172.21.15.70:51004) 2024-06-04T05:09:00.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff238ed6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0334210 0x5630c0337940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:53316/0 (socket says 172.21.15.70:53316) 2024-06-04T05:09:00.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff239ed8700 1 -- 172.21.15.70:0/1568737103 learned_addr learned my addr 172.21.15.70:0/1568737103 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:00.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff2396d7700 1 --2- 172.21.15.70:0/1568737103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0333860 0x5630c0333c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff239ed8700 1 -- 172.21.15.70:0/1568737103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0334210 msgr2=0x5630c0337940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:00.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff239ed8700 1 --2- 172.21.15.70:0/1568737103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0334210 0x5630c0337940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff239ed8700 1 -- 172.21.15.70:0/1568737103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0333860 msgr2=0x5630c0333c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:00.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff239ed8700 1 --2- 172.21.15.70:0/1568737103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0333860 0x5630c0333c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff239ed8700 1 -- 172.21.15.70:0/1568737103 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630c01d28d0 con 0x5630c0337f60 2024-06-04T05:09:00.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff238ed6700 1 --2- 172.21.15.70:0/1568737103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0334210 0x5630c0337940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:00.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.318+0000 7ff239ed8700 1 --2- 172.21.15.70:0/1568737103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c03383a0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0aac50 tx=0x7ff22c009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945c0a8bfc031431 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:00.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.319+0000 7ff21f7fe700 1 -- 172.21.15.70:0/1568737103 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c0ab3c0 con 0x5630c0337f60 2024-06-04T05:09:00.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.319+0000 7ff21f7fe700 1 -- 172.21.15.70:0/1568737103 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff22c0aacc0 con 0x5630c0337f60 2024-06-04T05:09:00.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.319+0000 7ff21f7fe700 1 -- 172.21.15.70:0/1568737103 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c0aaf20 con 0x5630c0337f60 2024-06-04T05:09:00.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.319+0000 7ff2461d0b80 1 -- 172.21.15.70:0/1568737103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 msgr2=0x5630c03383a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:00.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.319+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/1568737103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c03383a0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0aac50 tx=0x7ff22c009b80 comp rx=0 tx=0).stop 2024-06-04T05:09:00.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.320+0000 7ff2461d0b80 1 -- 172.21.15.70:0/1568737103 shutdown_connections 2024-06-04T05:09:00.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.320+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/1568737103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c03383a0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.320+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/1568737103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0333860 0x5630c0333c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.320+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/1568737103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0334210 0x5630c0337940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.320+0000 7ff2461d0b80 1 -- 172.21.15.70:0/1568737103 >> 172.21.15.70:0/1568737103 conn(0x5630c0261080 msgr2=0x5630c032de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:00.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.320+0000 7ff2461d0b80 1 -- 172.21.15.70:0/1568737103 shutdown_connections 2024-06-04T05:09:00.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.320+0000 7ff2461d0b80 1 -- 172.21.15.70:0/1568737103 wait complete. 2024-06-04T05:09:00.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.320+0000 7ff2461d0b80 1 -- start start 2024-06-04T05:09:00.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.320+0000 7ff2461d0b80 1 Processor -- start 2024-06-04T05:09:00.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.320+0000 7ff2461d0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0333860 0x5630c033f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff2461d0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0334210 0x5630c033c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff2461d0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c033ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff2461d0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5630c0188fa0 con 0x5630c0337f60 2024-06-04T05:09:00.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff2461d0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5630c02247c0 con 0x5630c0334210 2024-06-04T05:09:00.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff2461d0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5630c033fee0 con 0x5630c0333860 2024-06-04T05:09:00.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff2461d0b80 10 client.0 _ll_register_callbacks cb 0x5630c021d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:00.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff239ed8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c033ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff239ed8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c033ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51016/0 (socket says 172.21.15.70:51016) 2024-06-04T05:09:00.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff239ed8700 1 -- 172.21.15.70:0/3435482074 learned_addr learned my addr 172.21.15.70:0/3435482074 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:00.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff238ed6700 1 --2- 172.21.15.70:0/3435482074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0334210 0x5630c033c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff2396d7700 1 --2- 172.21.15.70:0/3435482074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0333860 0x5630c033f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff239ed8700 1 -- 172.21.15.70:0/3435482074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0333860 msgr2=0x5630c033f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:00.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff239ed8700 1 --2- 172.21.15.70:0/3435482074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0333860 0x5630c033f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff239ed8700 1 -- 172.21.15.70:0/3435482074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0334210 msgr2=0x5630c033c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff239ed8700 1 --2- 172.21.15.70:0/3435482074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0334210 0x5630c033c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff239ed8700 1 -- 172.21.15.70:0/3435482074 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630c0221690 con 0x5630c0337f60 2024-06-04T05:09:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff2396d7700 1 --2- 172.21.15.70:0/3435482074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0333860 0x5630c033f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:09:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff239ed8700 1 --2- 172.21.15.70:0/3435482074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c033ccb0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0aa9c0 tx=0x7ff22c0aaab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5abf874dbd9b6a3b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff21cff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff21cff9700 1 -- 172.21.15.70:0/3435482074 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c06f060 con 0x5630c0337f60 2024-06-04T05:09:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.321+0000 7ff21cff9700 1 -- 172.21.15.70:0/3435482074 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7ff22c0a72b0 con 0x5630c0337f60 2024-06-04T05:09:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.322+0000 7ff2461d0b80 1 -- 172.21.15.70:0/3435482074 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff22c0aacc0 con 0x5630c0337f60 2024-06-04T05:09:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.322+0000 7ff21cff9700 1 -- 172.21.15.70:0/3435482074 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff22c0b3a90 con 0x5630c0337f60 2024-06-04T05:09:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.322+0000 7ff2461d0b80 10 client.18621 fetch_fsmap learned FSMap version 34 2024-06-04T05:09:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.322+0000 7ff2461d0b80 1 -- 172.21.15.70:0/3435482074 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5630c034a5e0 con 0x5630c0337f60 2024-06-04T05:09:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.322+0000 7ff21cff9700 1 -- 172.21.15.70:0/3435482074 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff22c0aacc0 con 0x5630c0337f60 2024-06-04T05:09:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.322+0000 7ff21cff9700 1 -- 172.21.15.70:0/3435482074 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7ff22c011070 con 0x5630c0337f60 2024-06-04T05:09:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.322+0000 7ff2461d0b80 10 client.18621 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:09:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.322+0000 7ff2461d0b80 10 client.18621 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.322+0000 7ff2461d0b80 20 client.18621 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.322+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/3435482074 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5630c034cab0 0x5630c034eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.322+0000 7ff2461d0b80 4 client.18621 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:09:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.322+0000 7ff2461d0b80 1 -- 172.21.15.70:0/3435482074 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630c02ff9e0 con 0x5630c034cab0 2024-06-04T05:09:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.323+0000 7ff2396d7700 1 --2- 172.21.15.70:0/3435482074 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5630c034cab0 0x5630c034eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.323+0000 7ff2396d7700 1 --2- 172.21.15.70:0/3435482074 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5630c034cab0 0x5630c034eea0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bca21cd4fba02340 server_cookie=4fb0d02683a0f56 in_seq=0 out_seq=0 2024-06-04T05:09:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.323+0000 7ff21cff9700 10 client.18621 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.324+0000 7ff21cff9700 1 -- 172.21.15.70:0/3435482074 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5630c02ff9e0 con 0x5630c034cab0 2024-06-04T05:09:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.324+0000 7ff21cff9700 10 client.18621 handle_command_reply: tid=0 2024-06-04T05:09:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.324+0000 7ff2461d0b80 1 client.18621 shutdown 2024-06-04T05:09:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.324+0000 7ff2461d0b80 1 -- 172.21.15.70:0/3435482074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 msgr2=0x5630c033ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.324+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/3435482074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c033ccb0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0aa9c0 tx=0x7ff22c0aaab0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.324+0000 7ff2461d0b80 1 -- 172.21.15.70:0/3435482074 shutdown_connections 2024-06-04T05:09:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.324+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/3435482074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c033ccb0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.324+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/3435482074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0333860 0x5630c033f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.324+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/3435482074 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5630c034cab0 0x5630c034eea0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.324+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/3435482074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0334210 0x5630c033c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.324+0000 7ff21cff9700 0 client.18621 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.324+0000 7ff2461d0b80 1 -- 172.21.15.70:0/3435482074 >> 172.21.15.70:0/3435482074 conn(0x5630c0261080 msgr2=0x5630c032d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.324+0000 7ff2461d0b80 1 -- 172.21.15.70:0/3435482074 shutdown_connections 2024-06-04T05:09:00.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.324+0000 7ff2461d0b80 1 -- 172.21.15.70:0/3435482074 wait complete. 2024-06-04T05:09:00.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.324+0000 7ff2461d0b80 20 client.18621 trim_cache size 0 max 16384 2024-06-04T05:09:00.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.343+0000 7ff2461d0b80 1 Processor -- start 2024-06-04T05:09:00.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.343+0000 7ff2461d0b80 1 -- start start 2024-06-04T05:09:00.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.343+0000 7ff2461d0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0333860 0x5630c0332ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.343+0000 7ff2461d0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0334210 0x5630c032dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.343+0000 7ff2461d0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c03546b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.343+0000 7ff2461d0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff2341affb0 con 0x5630c0337f60 2024-06-04T05:09:00.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.343+0000 7ff2461d0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff2341b1b50 con 0x5630c0333860 2024-06-04T05:09:00.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.343+0000 7ff2461d0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff2341b0680 con 0x5630c0334210 2024-06-04T05:09:00.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.343+0000 7ff239ed8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c03546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.344+0000 7ff239ed8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c03546b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51022/0 (socket says 172.21.15.70:51022) 2024-06-04T05:09:00.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.344+0000 7ff239ed8700 1 -- 172.21.15.70:0/184578693 learned_addr learned my addr 172.21.15.70:0/184578693 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:00.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.344+0000 7ff238ed6700 1 --2- 172.21.15.70:0/184578693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0334210 0x5630c032dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.344+0000 7ff2396d7700 1 --2- 172.21.15.70:0/184578693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0333860 0x5630c0332ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.344+0000 7ff239ed8700 1 -- 172.21.15.70:0/184578693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0334210 msgr2=0x5630c032dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:00.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.344+0000 7ff239ed8700 1 --2- 172.21.15.70:0/184578693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0334210 0x5630c032dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.344+0000 7ff239ed8700 1 -- 172.21.15.70:0/184578693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0333860 msgr2=0x5630c0332ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:00.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.344+0000 7ff239ed8700 1 --2- 172.21.15.70:0/184578693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0333860 0x5630c0332ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.344+0000 7ff239ed8700 1 -- 172.21.15.70:0/184578693 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630c033fcf0 con 0x5630c0337f60 2024-06-04T05:09:00.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.344+0000 7ff238ed6700 1 --2- 172.21.15.70:0/184578693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0334210 0x5630c032dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:00.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.344+0000 7ff239ed8700 1 --2- 172.21.15.70:0/184578693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c03546b0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0b3750 tx=0x7ff22c009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df52204b945dbd5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:00.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.344+0000 7ff21effd700 1 -- 172.21.15.70:0/184578693 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c0ab3c0 con 0x5630c0337f60 2024-06-04T05:09:00.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.344+0000 7ff21effd700 1 -- 172.21.15.70:0/184578693 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff22c0aacc0 con 0x5630c0337f60 2024-06-04T05:09:00.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.344+0000 7ff21effd700 1 -- 172.21.15.70:0/184578693 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c0b1310 con 0x5630c0337f60 2024-06-04T05:09:00.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.345+0000 7ff2461d0b80 1 -- 172.21.15.70:0/184578693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 msgr2=0x5630c03546b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:00.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.345+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/184578693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c03546b0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0b3750 tx=0x7ff22c009b80 comp rx=0 tx=0).stop 2024-06-04T05:09:00.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.345+0000 7ff2461d0b80 1 -- 172.21.15.70:0/184578693 shutdown_connections 2024-06-04T05:09:00.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.345+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/184578693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0337f60 0x5630c03546b0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.345+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/184578693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0333860 0x5630c0332ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.345+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/184578693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0334210 0x5630c032dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.345+0000 7ff2461d0b80 1 -- 172.21.15.70:0/184578693 >> 172.21.15.70:0/184578693 conn(0x5630c0261080 msgr2=0x5630c0337140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:00.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.345+0000 7ff2461d0b80 1 -- 172.21.15.70:0/184578693 shutdown_connections 2024-06-04T05:09:00.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.345+0000 7ff2461d0b80 1 -- 172.21.15.70:0/184578693 wait complete. 2024-06-04T05:09:00.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.345+0000 7ff2461d0b80 1 -- start start 2024-06-04T05:09:00.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.345+0000 7ff2461d0b80 1 Processor -- start 2024-06-04T05:09:00.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.346+0000 7ff2461d0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0333860 0x5630c03454c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.346+0000 7ff2461d0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0334210 0x5630c0342860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.346+0000 7ff2461d0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0337f60 0x5630c0342da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.346+0000 7ff2461d0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff2341b1450 con 0x5630c0333860 2024-06-04T05:09:00.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.346+0000 7ff2461d0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff2341b0d50 con 0x5630c0334210 2024-06-04T05:09:00.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.346+0000 7ff2461d0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff2341adf70 con 0x5630c0337f60 2024-06-04T05:09:00.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.346+0000 7ff2461d0b80 10 client.0 _ll_register_callbacks cb 0x5630c021d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:00.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.346+0000 7ff2396d7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0333860 0x5630c03454c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.346+0000 7ff239ed8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0337f60 0x5630c0342da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.346+0000 7ff2396d7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0333860 0x5630c03454c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51038/0 (socket says 172.21.15.70:51038) 2024-06-04T05:09:00.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.346+0000 7ff2396d7700 1 -- 172.21.15.70:0/2756064365 learned_addr learned my addr 172.21.15.70:0/2756064365 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:00.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.346+0000 7ff238ed6700 1 --2- 172.21.15.70:0/2756064365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0334210 0x5630c0342860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.346+0000 7ff239ed8700 1 -- 172.21.15.70:0/2756064365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0334210 msgr2=0x5630c0342860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:00.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.346+0000 7ff239ed8700 1 --2- 172.21.15.70:0/2756064365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0334210 0x5630c0342860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.346+0000 7ff239ed8700 1 -- 172.21.15.70:0/2756064365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0333860 msgr2=0x5630c03454c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:00.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.346+0000 7ff239ed8700 1 --2- 172.21.15.70:0/2756064365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0333860 0x5630c03454c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.346+0000 7ff239ed8700 1 -- 172.21.15.70:0/2756064365 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630c0343360 con 0x5630c0337f60 2024-06-04T05:09:00.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.347+0000 7ff239ed8700 1 --2- 172.21.15.70:0/2756064365 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0337f60 0x5630c0342da0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0a7f30 tx=0x7ff22c00b3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c5275f14231bf8c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:00.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.347+0000 7ff21cff9700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:09:00.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.347+0000 7ff21cff9700 1 -- 172.21.15.70:0/2756064365 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c08b070 con 0x5630c0337f60 2024-06-04T05:09:00.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.347+0000 7ff2461d0b80 1 -- 172.21.15.70:0/2756064365 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff22c0aacc0 con 0x5630c0337f60 2024-06-04T05:09:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.347+0000 7ff21cff9700 1 -- 172.21.15.70:0/2756064365 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8571+0+0 (secure 0 0 0) 0x7ff22c0a9f20 con 0x5630c0337f60 2024-06-04T05:09:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.347+0000 7ff21cff9700 1 -- 172.21.15.70:0/2756064365 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff22c06f050 con 0x5630c0337f60 2024-06-04T05:09:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.347+0000 7ff2461d0b80 10 client.27751 fetch_fsmap learned FSMap version 34 2024-06-04T05:09:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.347+0000 7ff2461d0b80 1 -- 172.21.15.70:0/2756064365 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5630c0343360 con 0x5630c0337f60 2024-06-04T05:09:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.347+0000 7ff21cff9700 1 -- 172.21.15.70:0/2756064365 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff22c0aacc0 con 0x5630c0337f60 2024-06-04T05:09:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.347+0000 7ff21cff9700 1 -- 172.21.15.70:0/2756064365 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7ff22c08e030 con 0x5630c0337f60 2024-06-04T05:09:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.347+0000 7ff2461d0b80 10 client.27751 fetch_fsmap finished waiting for FSMap version 34 2024-06-04T05:09:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.347+0000 7ff2461d0b80 10 client.27751 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.347+0000 7ff2461d0b80 20 client.27751 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.347+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/2756064365 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5630c03b7620 0x5630c03b9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.347+0000 7ff2461d0b80 4 client.27751 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:09:00.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.347+0000 7ff2461d0b80 1 -- 172.21.15.70:0/2756064365 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5630c0294be0 con 0x5630c03b7620 2024-06-04T05:09:00.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.348+0000 7ff2396d7700 1 --2- 172.21.15.70:0/2756064365 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5630c03b7620 0x5630c03b9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:00.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.348+0000 7ff2396d7700 1 --2- 172.21.15.70:0/2756064365 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5630c03b7620 0x5630c03b9a10 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1007b47193908f67 server_cookie=a618a4819f50897f in_seq=0 out_seq=0 2024-06-04T05:09:00.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.348+0000 7ff21cff9700 10 client.27751 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:00.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.348+0000 7ff21cff9700 1 -- 172.21.15.70:0/2756064365 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5630c0294be0 con 0x5630c03b7620 2024-06-04T05:09:00.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.348+0000 7ff21cff9700 10 client.27751 handle_command_reply: tid=0 2024-06-04T05:09:00.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.348+0000 7ff2461d0b80 1 client.27751 shutdown 2024-06-04T05:09:00.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.348+0000 7ff2461d0b80 1 -- 172.21.15.70:0/2756064365 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0337f60 msgr2=0x5630c0342da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:00.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.348+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/2756064365 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0337f60 0x5630c0342da0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0a7f30 tx=0x7ff22c00b3b0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.348+0000 7ff2461d0b80 1 -- 172.21.15.70:0/2756064365 shutdown_connections 2024-06-04T05:09:00.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.348+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/2756064365 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5630c0337f60 0x5630c0342da0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.348+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/2756064365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630c0333860 0x5630c03454c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.348+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/2756064365 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5630c03b7620 0x5630c03b9a10 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.348+0000 7ff2461d0b80 1 --2- 172.21.15.70:0/2756064365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c0334210 0x5630c0342860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.348+0000 7ff2461d0b80 1 -- 172.21.15.70:0/2756064365 >> 172.21.15.70:0/2756064365 conn(0x5630c0261080 msgr2=0x5630c033dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:00.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.348+0000 7ff21cff9700 0 client.27751 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:00.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.349+0000 7ff2461d0b80 1 -- 172.21.15.70:0/2756064365 shutdown_connections 2024-06-04T05:09:00.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.349+0000 7ff2461d0b80 1 -- 172.21.15.70:0/2756064365 wait complete. 2024-06-04T05:09:00.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.349+0000 7ff2461d0b80 20 client.27751 trim_cache size 0 max 16384 2024-06-04T05:09:00.359 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:09:00.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.349+0000 7ff23b93b700 1 -- 172.21.15.70:0/1372706262 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff2200b29f0 msgr2=0x7ff2200b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:00.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.349+0000 7ff23b93b700 1 --2- 172.21.15.70:0/1372706262 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff2200b29f0 0x7ff2200b4eb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff230011550 tx=0x7ff230009260 comp rx=0 tx=0).stop 2024-06-04T05:09:00.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.349+0000 7ff23b93b700 1 -- 172.21.15.70:0/1372706262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff2341095d0 msgr2=0x7ff2340762c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:00.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.349+0000 7ff23b93b700 1 --2- 172.21.15.70:0/1372706262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff2341095d0 0x7ff2340762c0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7ff22c00b0a0 tx=0x7ff22c004d00 comp rx=0 tx=0).stop 2024-06-04T05:09:00.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.349+0000 7ff23b93b700 1 -- 172.21.15.70:0/1372706262 shutdown_connections 2024-06-04T05:09:00.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.349+0000 7ff23b93b700 1 --2- 172.21.15.70:0/1372706262 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff2200b29f0 0x7ff2200b4eb0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.349+0000 7ff23b93b700 1 --2- 172.21.15.70:0/1372706262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff2341095d0 0x7ff2340762c0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.349+0000 7ff23b93b700 1 --2- 172.21.15.70:0/1372706262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2341035c0 0x7ff234075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.349+0000 7ff23b93b700 1 --2- 172.21.15.70:0/1372706262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff234103f80 0x7ff234075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:00.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.349+0000 7ff23b93b700 1 -- 172.21.15.70:0/1372706262 >> 172.21.15.70:0/1372706262 conn(0x7ff23407a610 msgr2=0x7ff234102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:00.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.349+0000 7ff23b93b700 1 -- 172.21.15.70:0/1372706262 shutdown_connections 2024-06-04T05:09:00.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:00.349+0000 7ff23b93b700 1 -- 172.21.15.70:0/1372706262 wait complete. 2024-06-04T05:09:00.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 199087 1.75 MB/sec execute 473 sec latency 46.280 ms 2024-06-04T05:09:01.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:01 smithi070 ceph-mon[32631]: pgmap v366: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 559 KiB/s wr, 48 op/s 2024-06-04T05:09:01.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:01 smithi112 ceph-mon[36453]: pgmap v366: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 559 KiB/s wr, 48 op/s 2024-06-04T05:09:01.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 199502 1.75 MB/sec execute 474 sec latency 17.536 ms 2024-06-04T05:09:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:01 smithi098 ceph-mon[35569]: pgmap v366: 97 pgs: 97 active+clean; 118 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 559 KiB/s wr, 48 op/s 2024-06-04T05:09:02.092 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:09:02.093 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:09:02.362 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:09:02.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 200040 1.76 MB/sec execute 475 sec latency 16.878 ms 2024-06-04T05:09:03.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:03 smithi070 ceph-mon[32631]: pgmap v367: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 944 KiB/s wr, 100 op/s 2024-06-04T05:09:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:03 smithi112 ceph-mon[36453]: pgmap v367: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 944 KiB/s wr, 100 op/s 2024-06-04T05:09:03.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 200529 1.75 MB/sec execute 476 sec latency 24.509 ms 2024-06-04T05:09:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:03 smithi098 ceph-mon[35569]: pgmap v367: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 944 KiB/s wr, 100 op/s 2024-06-04T05:09:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.055+0000 7fd37a9f7640 1 -- 172.21.15.70:0/3077065169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd374107940 msgr2=0x7fd374107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.056+0000 7fd37a9f7640 1 --2- 172.21.15.70:0/3077065169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd374107940 0x7fd374107d20 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fd364009930 tx=0x7fd36402a420 comp rx=0 tx=0).stop 2024-06-04T05:09:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.056+0000 7fd37a9f7640 1 -- 172.21.15.70:0/3077065169 shutdown_connections 2024-06-04T05:09:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.056+0000 7fd37a9f7640 1 --2- 172.21.15.70:0/3077065169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd374101980 0x7fd374101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.056+0000 7fd37a9f7640 1 --2- 172.21.15.70:0/3077065169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd374102340 0x7fd37410e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.056+0000 7fd37a9f7640 1 --2- 172.21.15.70:0/3077065169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd374107940 0x7fd374107d20 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.056+0000 7fd37a9f7640 1 -- 172.21.15.70:0/3077065169 >> 172.21.15.70:0/3077065169 conn(0x7fd374078ad0 msgr2=0x7fd374078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.056+0000 7fd37a9f7640 1 -- 172.21.15.70:0/3077065169 shutdown_connections 2024-06-04T05:09:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.056+0000 7fd37a9f7640 1 -- 172.21.15.70:0/3077065169 wait complete. 2024-06-04T05:09:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.056+0000 7fd37a9f7640 1 Processor -- start 2024-06-04T05:09:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.057+0000 7fd37a9f7640 1 -- start start 2024-06-04T05:09:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.057+0000 7fd37a9f7640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd374101980 0x7fd374199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.057+0000 7fd37a9f7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd374102340 0x7fd37419a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.057+0000 7fd37a9f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd374107940 0x7fd3741a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.057+0000 7fd37a9f7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd374111ab0 con 0x7fd374102340 2024-06-04T05:09:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.057+0000 7fd37a9f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd374111930 con 0x7fd374107940 2024-06-04T05:09:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.057+0000 7fd37a9f7640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd374111c30 con 0x7fd374101980 2024-06-04T05:09:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.057+0000 7fd3737fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd374102340 0x7fd37419a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.057+0000 7fd373fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd374101980 0x7fd374199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.057+0000 7fd373fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd374101980 0x7fd374199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:53354/0 (socket says 172.21.15.70:53354) 2024-06-04T05:09:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.057+0000 7fd3737fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd374102340 0x7fd37419a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51056/0 (socket says 172.21.15.70:51056) 2024-06-04T05:09:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.057+0000 7fd373fff640 1 -- 172.21.15.70:0/2417424125 learned_addr learned my addr 172.21.15.70:0/2417424125 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.058+0000 7fd373fff640 1 -- 172.21.15.70:0/2417424125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd374107940 msgr2=0x7fd3741a4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.058+0000 7fd373fff640 1 --2- 172.21.15.70:0/2417424125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd374107940 0x7fd3741a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.058+0000 7fd373fff640 1 -- 172.21.15.70:0/2417424125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd374102340 msgr2=0x7fd37419a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.058+0000 7fd373fff640 1 --2- 172.21.15.70:0/2417424125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd374102340 0x7fd37419a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.058+0000 7fd373fff640 1 -- 172.21.15.70:0/2417424125 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3741a5530 con 0x7fd374101980 2024-06-04T05:09:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.058+0000 7fd3737fe640 1 --2- 172.21.15.70:0/2417424125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd374102340 0x7fd37419a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:04.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.058+0000 7fd373fff640 1 --2- 172.21.15.70:0/2417424125 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd374101980 0x7fd374199f70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd364005940 tx=0x7fd364004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:04.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.058+0000 7fd3717fa640 1 -- 172.21.15.70:0/2417424125 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36403f030 con 0x7fd374101980 2024-06-04T05:09:04.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.058+0000 7fd3717fa640 1 -- 172.21.15.70:0/2417424125 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd364004440 con 0x7fd374101980 2024-06-04T05:09:04.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.058+0000 7fd3717fa640 1 -- 172.21.15.70:0/2417424125 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd364035070 con 0x7fd374101980 2024-06-04T05:09:04.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.058+0000 7fd37a9f7640 1 -- 172.21.15.70:0/2417424125 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3741a5700 con 0x7fd374101980 2024-06-04T05:09:04.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.058+0000 7fd37a9f7640 1 -- 172.21.15.70:0/2417424125 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3741a5ab0 con 0x7fd374101980 2024-06-04T05:09:04.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.061+0000 7fd3717fa640 1 -- 172.21.15.70:0/2417424125 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fd364042030 con 0x7fd374101980 2024-06-04T05:09:04.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.061+0000 7fd37d7eb740 1 Processor -- start 2024-06-04T05:09:04.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.061+0000 7fd37d7eb740 1 -- start start 2024-06-04T05:09:04.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd37d7eb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e8e820 0x55b784e8ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd37d7eb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 0x55b784e994c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd37d7eb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e99a00 0x55b784e9bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd37d7eb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b784e00c30 con 0x55b784e8f140 2024-06-04T05:09:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd37d7eb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b784df07d0 con 0x55b784e99a00 2024-06-04T05:09:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd37d7eb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b784df4810 con 0x55b784e8e820 2024-06-04T05:09:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd378f94640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e8e820 0x55b784e8ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd373fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 0x55b784e994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd3737fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e99a00 0x55b784e9bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd378f94640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e8e820 0x55b784e8ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:53368/0 (socket says 172.21.15.70:53368) 2024-06-04T05:09:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd373fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 0x55b784e994c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51064/0 (socket says 172.21.15.70:51064) 2024-06-04T05:09:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd378f94640 1 -- 172.21.15.70:0/2544115092 learned_addr learned my addr 172.21.15.70:0/2544115092 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd3717fa640 1 --2- 172.21.15.70:0/2417424125 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd35c0b2a10 0x7fd35c0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd3737fe640 1 -- 172.21.15.70:0/2544115092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e8e820 msgr2=0x55b784e8ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd3737fe640 1 --2- 172.21.15.70:0/2544115092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e8e820 0x55b784e8ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd3737fe640 1 -- 172.21.15.70:0/2544115092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 msgr2=0x55b784e994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd3737fe640 1 --2- 172.21.15.70:0/2544115092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 0x55b784e994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd3737fe640 1 -- 172.21.15.70:0/2544115092 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b784b05210 con 0x55b784e99a00 2024-06-04T05:09:04.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd378f94640 1 --2- 172.21.15.70:0/2544115092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e8e820 0x55b784e8ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:04.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd378f94640 1 --2- 172.21.15.70:0/2417424125 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd35c0b2a10 0x7fd35c0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd3717fa640 1 -- 172.21.15.70:0/2417424125 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8615+0+0 (secure 0 0 0) 0x7fd364102b40 con 0x7fd374101980 2024-06-04T05:09:04.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.062+0000 7fd3737fe640 1 --2- 172.21.15.70:0/2544115092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e99a00 0x55b784e9bdf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd36800bfb0 tx=0x7fd368010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3172271f3504e1fa server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:04.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.063+0000 7fd35a7fc640 1 -- 172.21.15.70:0/2544115092 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd368004220 con 0x55b784e99a00 2024-06-04T05:09:04.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.063+0000 7fd35a7fc640 1 -- 172.21.15.70:0/2544115092 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd3680043c0 con 0x55b784e99a00 2024-06-04T05:09:04.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.063+0000 7fd35a7fc640 1 -- 172.21.15.70:0/2544115092 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd368011980 con 0x55b784e99a00 2024-06-04T05:09:04.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.063+0000 7fd37d7eb740 1 -- 172.21.15.70:0/2544115092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e99a00 msgr2=0x55b784e9bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.063+0000 7fd378f94640 1 --2- 172.21.15.70:0/2417424125 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd35c0b2a10 0x7fd35c0b4ed0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd36c0049d0 tx=0x7fd36c004a40 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:04.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.063+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/2544115092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e99a00 0x55b784e9bdf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd36800bfb0 tx=0x7fd368010860 comp rx=0 tx=0).stop 2024-06-04T05:09:04.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.063+0000 7fd37d7eb740 1 -- 172.21.15.70:0/2544115092 shutdown_connections 2024-06-04T05:09:04.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.063+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/2544115092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e99a00 0x55b784e9bdf0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.063+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/2544115092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 0x55b784e994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.063+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/2544115092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e8e820 0x55b784e8ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.064+0000 7fd37d7eb740 1 -- 172.21.15.70:0/2544115092 >> 172.21.15.70:0/2544115092 conn(0x55b784e8ad30 msgr2=0x55b784e8b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:04.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.064+0000 7fd37d7eb740 1 -- 172.21.15.70:0/2544115092 shutdown_connections 2024-06-04T05:09:04.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.064+0000 7fd37d7eb740 1 -- 172.21.15.70:0/2544115092 wait complete. 2024-06-04T05:09:04.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.064+0000 7fd37d7eb740 1 -- start start 2024-06-04T05:09:04.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.064+0000 7fd37d7eb740 1 Processor -- start 2024-06-04T05:09:04.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.064+0000 7fd37d7eb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e8e820 0x55b784e969d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.064+0000 7fd37d7eb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e8f140 0x55b784e96f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.064+0000 7fd37d7eb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e99a00 0x55b784e97450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.064+0000 7fd37d7eb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b784ad6790 con 0x55b784e99a00 2024-06-04T05:09:04.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.064+0000 7fd37d7eb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b784dac890 con 0x55b784e8e820 2024-06-04T05:09:04.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.064+0000 7fd37d7eb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b784ea0a40 con 0x55b784e8f140 2024-06-04T05:09:04.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.064+0000 7fd37d7eb740 10 client.0 _ll_register_callbacks cb 0x55b784b04a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:04.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.065+0000 7fd3737fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e8f140 0x55b784e96f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.065+0000 7fd3737fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e8f140 0x55b784e96f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:53374/0 (socket says 172.21.15.70:53374) 2024-06-04T05:09:04.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.065+0000 7fd3737fe640 1 -- 172.21.15.70:0/2205129228 learned_addr learned my addr 172.21.15.70:0/2205129228 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:04.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.065+0000 7fd378f94640 1 --2- 172.21.15.70:0/2205129228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e99a00 0x55b784e97450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.065+0000 7fd373fff640 1 --2- 172.21.15.70:0/2205129228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e8e820 0x55b784e969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.065+0000 7fd3737fe640 1 -- 172.21.15.70:0/2205129228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e8e820 msgr2=0x55b784e969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:04.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.065+0000 7fd3737fe640 1 --2- 172.21.15.70:0/2205129228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e8e820 0x55b784e969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.065+0000 7fd3737fe640 1 -- 172.21.15.70:0/2205129228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e99a00 msgr2=0x55b784e97450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:04.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.065+0000 7fd3737fe640 1 --2- 172.21.15.70:0/2205129228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e99a00 0x55b784e97450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.065+0000 7fd3737fe640 1 -- 172.21.15.70:0/2205129228 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b784bd1620 con 0x55b784e8f140 2024-06-04T05:09:04.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.065+0000 7fd3737fe640 1 --2- 172.21.15.70:0/2205129228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e8f140 0x55b784e96f10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd368011620 tx=0x7fd368004230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c136f69efa2c746 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:04.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.065+0000 7fd3537fe640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:09:04.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.065+0000 7fd3537fe640 1 -- 172.21.15.70:0/2205129228 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd368005250 con 0x55b784e8f140 2024-06-04T05:09:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.065+0000 7fd3537fe640 1 -- 172.21.15.70:0/2205129228 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8615+0+0 (secure 0 0 0) 0x7fd3680053f0 con 0x55b784e8f140 2024-06-04T05:09:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.065+0000 7fd37d7eb740 1 -- 172.21.15.70:0/2205129228 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3680043c0 con 0x55b784e8f140 2024-06-04T05:09:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.065+0000 7fd3537fe640 1 -- 172.21.15.70:0/2205129228 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd3680055f0 con 0x55b784e8f140 2024-06-04T05:09:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.066+0000 7fd37d7eb740 10 client.27763 fetch_fsmap learned FSMap version 35 2024-06-04T05:09:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.066+0000 7fd37d7eb740 1 -- 172.21.15.70:0/2205129228 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b784eac7e0 con 0x55b784e8f140 2024-06-04T05:09:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.066+0000 7fd3537fe640 1 -- 172.21.15.70:0/2205129228 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd3680043c0 con 0x55b784e8f140 2024-06-04T05:09:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.066+0000 7fd3537fe640 1 -- 172.21.15.70:0/2205129228 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7fd368004a30 con 0x55b784e8f140 2024-06-04T05:09:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.066+0000 7fd37d7eb740 10 client.27763 fetch_fsmap finished waiting for FSMap version 35 2024-06-04T05:09:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.066+0000 7fd37d7eb740 10 client.27763 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.066+0000 7fd37d7eb740 20 client.27763 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.066+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/2205129228 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b784eadc40 0x55b784eb0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.066+0000 7fd37d7eb740 4 client.27763 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:09:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.066+0000 7fd37d7eb740 1 -- 172.21.15.70:0/2205129228 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b784e02b80 con 0x55b784eadc40 2024-06-04T05:09:04.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.066+0000 7fd373fff640 1 --2- 172.21.15.70:0/2205129228 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b784eadc40 0x55b784eb0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.067+0000 7fd3537fe640 10 client.27763 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:04.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.067+0000 7fd373fff640 1 --2- 172.21.15.70:0/2205129228 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b784eadc40 0x55b784eb0030 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fb8aab06abe045 server_cookie=c25c7a0e0133c9e7 in_seq=0 out_seq=0 2024-06-04T05:09:04.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.068+0000 7fd3537fe640 1 -- 172.21.15.70:0/2205129228 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b784e02b80 con 0x55b784eadc40 2024-06-04T05:09:04.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.068+0000 7fd3537fe640 10 client.27763 handle_command_reply: tid=0 2024-06-04T05:09:04.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.068+0000 7fd37d7eb740 1 client.27763 shutdown 2024-06-04T05:09:04.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.068+0000 7fd37d7eb740 1 -- 172.21.15.70:0/2205129228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e8f140 msgr2=0x55b784e96f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.068+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/2205129228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e8f140 0x55b784e96f10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd368011620 tx=0x7fd368004230 comp rx=0 tx=0).stop 2024-06-04T05:09:04.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.068+0000 7fd37d7eb740 1 -- 172.21.15.70:0/2205129228 shutdown_connections 2024-06-04T05:09:04.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.068+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/2205129228 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b784eadc40 0x55b784eb0030 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.068+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/2205129228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e99a00 0x55b784e97450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.068+0000 7fd3537fe640 0 client.27763 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:04.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.068+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/2205129228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e8f140 0x55b784e96f10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.068+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/2205129228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e8e820 0x55b784e969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.068+0000 7fd37d7eb740 1 -- 172.21.15.70:0/2205129228 >> 172.21.15.70:0/2205129228 conn(0x55b784e8ad30 msgr2=0x55b784e89fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.068+0000 7fd37d7eb740 1 -- 172.21.15.70:0/2205129228 shutdown_connections 2024-06-04T05:09:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.068+0000 7fd37d7eb740 1 -- 172.21.15.70:0/2205129228 wait complete. 2024-06-04T05:09:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.068+0000 7fd37d7eb740 20 client.27763 trim_cache size 0 max 16384 2024-06-04T05:09:04.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.100+0000 7fd37d7eb740 1 Processor -- start 2024-06-04T05:09:04.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.100+0000 7fd37d7eb740 1 -- start start 2024-06-04T05:09:04.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.101+0000 7fd37d7eb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e8e820 0x55b784eb58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.101+0000 7fd37d7eb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 0x55b784eb5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.101+0000 7fd37d7eb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e99a00 0x55b784e93c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.101+0000 7fd37d7eb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd3741ac490 con 0x55b784e8f140 2024-06-04T05:09:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.101+0000 7fd37d7eb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd3741ae060 con 0x55b784e8e820 2024-06-04T05:09:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.101+0000 7fd37d7eb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd3741acb60 con 0x55b784e99a00 2024-06-04T05:09:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.101+0000 7fd373fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e8e820 0x55b784eb58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.101+0000 7fd378f94640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e99a00 0x55b784e93c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.101+0000 7fd373fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e8e820 0x55b784eb58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46570/0 (socket says 172.21.15.70:46570) 2024-06-04T05:09:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.101+0000 7fd373fff640 1 -- 172.21.15.70:0/1154935862 learned_addr learned my addr 172.21.15.70:0/1154935862 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.101+0000 7fd3737fe640 1 --2- 172.21.15.70:0/1154935862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 0x55b784eb5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.101+0000 7fd378f94640 1 -- 172.21.15.70:0/1154935862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e8e820 msgr2=0x55b784eb58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.101+0000 7fd378f94640 1 --2- 172.21.15.70:0/1154935862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e8e820 0x55b784eb58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.101+0000 7fd378f94640 1 -- 172.21.15.70:0/1154935862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 msgr2=0x55b784eb5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.101+0000 7fd378f94640 1 --2- 172.21.15.70:0/1154935862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 0x55b784eb5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.101+0000 7fd378f94640 1 -- 172.21.15.70:0/1154935862 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b784e866f0 con 0x55b784e99a00 2024-06-04T05:09:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.101+0000 7fd378f94640 1 --2- 172.21.15.70:0/1154935862 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e99a00 0x55b784e93c30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd36c005fd0 tx=0x7fd36c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=109c6df8d97e87ab server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.102+0000 7fd359ffb640 1 -- 172.21.15.70:0/1154935862 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36c00e630 con 0x55b784e99a00 2024-06-04T05:09:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.102+0000 7fd359ffb640 1 -- 172.21.15.70:0/1154935862 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd36c016070 con 0x55b784e99a00 2024-06-04T05:09:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.102+0000 7fd359ffb640 1 -- 172.21.15.70:0/1154935862 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36c01e950 con 0x55b784e99a00 2024-06-04T05:09:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.103+0000 7fd37d7eb740 1 -- 172.21.15.70:0/1154935862 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e99a00 msgr2=0x55b784e93c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.103+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/1154935862 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e99a00 0x55b784e93c30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd36c005fd0 tx=0x7fd36c014000 comp rx=0 tx=0).stop 2024-06-04T05:09:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.103+0000 7fd37d7eb740 1 -- 172.21.15.70:0/1154935862 shutdown_connections 2024-06-04T05:09:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.103+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/1154935862 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e99a00 0x55b784e93c30 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.103+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/1154935862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 0x55b784eb5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.103+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/1154935862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e8e820 0x55b784eb58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.103+0000 7fd37d7eb740 1 -- 172.21.15.70:0/1154935862 >> 172.21.15.70:0/1154935862 conn(0x55b784e8ad30 msgr2=0x55b784e90a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.103+0000 7fd37d7eb740 1 -- 172.21.15.70:0/1154935862 shutdown_connections 2024-06-04T05:09:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.103+0000 7fd37d7eb740 1 -- 172.21.15.70:0/1154935862 wait complete. 2024-06-04T05:09:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.103+0000 7fd37d7eb740 1 -- start start 2024-06-04T05:09:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.103+0000 7fd37d7eb740 1 Processor -- start 2024-06-04T05:09:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd37d7eb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e8e820 0x55b784ea64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd37d7eb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 0x55b784ea18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd3737fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 0x55b784ea18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd3737fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 0x55b784ea18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51098/0 (socket says 172.21.15.70:51098) 2024-06-04T05:09:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd3737fe640 1 -- 172.21.15.70:0/3982062753 learned_addr learned my addr 172.21.15.70:0/3982062753 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd373fff640 1 --2- 172.21.15.70:0/3982062753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e8e820 0x55b784ea64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/3982062753 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e99a00 0x55b784ea1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd37d7eb740 1 -- 172.21.15.70:0/3982062753 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd3741ad930 con 0x55b784e8f140 2024-06-04T05:09:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd37d7eb740 1 -- 172.21.15.70:0/3982062753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd3741ad230 con 0x55b784e8e820 2024-06-04T05:09:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd37d7eb740 1 -- 172.21.15.70:0/3982062753 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd3741aa490 con 0x55b784e99a00 2024-06-04T05:09:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd378f94640 1 --2- 172.21.15.70:0/3982062753 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e99a00 0x55b784ea1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd37d7eb740 10 client.0 _ll_register_callbacks cb 0x7fd3641064c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd3737fe640 1 -- 172.21.15.70:0/3982062753 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e99a00 msgr2=0x55b784ea1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd3737fe640 1 --2- 172.21.15.70:0/3982062753 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e99a00 0x55b784ea1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd3737fe640 1 -- 172.21.15.70:0/3982062753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e8e820 msgr2=0x55b784ea64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd3737fe640 1 --2- 172.21.15.70:0/3982062753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e8e820 0x55b784ea64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd3737fe640 1 -- 172.21.15.70:0/3982062753 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b784f177b0 con 0x55b784e8f140 2024-06-04T05:09:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd373fff640 1 --2- 172.21.15.70:0/3982062753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e8e820 0x55b784ea64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd378f94640 1 --2- 172.21.15.70:0/3982062753 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e99a00 0x55b784ea1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:09:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd37d7eb740 1 -- 172.21.15.70:0/3982062753 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd36c016070 con 0x55b784e8f140 2024-06-04T05:09:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd3737fe640 1 --2- 172.21.15.70:0/3982062753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 0x55b784ea18b0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x55b784e97fa0 tx=0x7fd368010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=122726cf482dbac1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.104+0000 7fd3537fe640 10 client.18645 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.105+0000 7fd3537fe640 1 -- 172.21.15.70:0/3982062753 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3680053d0 con 0x55b784e8f140 2024-06-04T05:09:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.105+0000 7fd3537fe640 1 -- 172.21.15.70:0/3982062753 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8615+0+0 (secure 0 0 0) 0x7fd3680042a0 con 0x55b784e8f140 2024-06-04T05:09:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.105+0000 7fd3537fe640 1 -- 172.21.15.70:0/3982062753 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd3680119e0 con 0x55b784e8f140 2024-06-04T05:09:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.105+0000 7fd37d7eb740 10 client.18645 fetch_fsmap learned FSMap version 35 2024-06-04T05:09:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.105+0000 7fd37d7eb740 1 -- 172.21.15.70:0/3982062753 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({fsmap=35}) -- 0x55b784f18840 con 0x55b784e8f140 2024-06-04T05:09:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.105+0000 7fd3537fe640 1 -- 172.21.15.70:0/3982062753 <== mon.0 v2:172.21.15.70:3300/0 4 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7fd368011b80 con 0x55b784e8f140 2024-06-04T05:09:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.105+0000 7fd37d7eb740 10 client.18645 fetch_fsmap finished waiting for FSMap version 35 2024-06-04T05:09:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.105+0000 7fd37d7eb740 10 client.18645 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.105+0000 7fd37d7eb740 20 client.18645 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.105+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/3982062753 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b784f18ea0 0x55b784f1b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.105+0000 7fd37d7eb740 4 client.18645 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:09:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.105+0000 7fd37d7eb740 1 -- 172.21.15.70:0/3982062753 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b784dfc360 con 0x55b784f18ea0 2024-06-04T05:09:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.105+0000 7fd373fff640 1 --2- 172.21.15.70:0/3982062753 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b784f18ea0 0x55b784f1b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.106+0000 7fd3537fe640 10 client.18645 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.106+0000 7fd373fff640 1 --2- 172.21.15.70:0/3982062753 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b784f18ea0 0x55b784f1b2c0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=373f818977a58a37 server_cookie=18d98055fb8b7600 in_seq=0 out_seq=0 2024-06-04T05:09:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.119+0000 7fa50f2ac700 1 -- 172.21.15.70:0/2815642210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa508108510 msgr2=0x7fa5081088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.119+0000 7fa50f2ac700 1 --2- 172.21.15.70:0/2815642210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa508108510 0x7fa5081088f0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fa504009990 tx=0x7fa504009ca0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.119+0000 7fa50f2ac700 1 -- 172.21.15.70:0/2815642210 shutdown_connections 2024-06-04T05:09:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.119+0000 7fa50f2ac700 1 --2- 172.21.15.70:0/2815642210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa508102df0 0x7fa50810f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.119+0000 7fa50f2ac700 1 --2- 172.21.15.70:0/2815642210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa508108510 0x7fa5081088f0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.119+0000 7fa50f2ac700 1 --2- 172.21.15.70:0/2815642210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa508102430 0x7fa5081028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.119+0000 7fa50f2ac700 1 -- 172.21.15.70:0/2815642210 >> 172.21.15.70:0/2815642210 conn(0x7fa5080793b0 msgr2=0x7fa5080797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.119+0000 7fa50f2ac700 1 -- 172.21.15.70:0/2815642210 shutdown_connections 2024-06-04T05:09:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.119+0000 7fa50f2ac700 1 -- 172.21.15.70:0/2815642210 wait complete. 2024-06-04T05:09:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.120+0000 7fa50f2ac700 1 Processor -- start 2024-06-04T05:09:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.120+0000 7fa50f2ac700 1 -- start start 2024-06-04T05:09:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.120+0000 7fa50f2ac700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa508102430 0x7fa50819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.120+0000 7fa50f2ac700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa508102df0 0x7fa50819fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.120+0000 7fa50f2ac700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa508108510 0x7fa5081999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.120+0000 7fa50f2ac700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa508112490 con 0x7fa508102430 2024-06-04T05:09:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.120+0000 7fa50f2ac700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa508112310 con 0x7fa508108510 2024-06-04T05:09:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.120+0000 7fa50f2ac700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa508112610 con 0x7fa508102df0 2024-06-04T05:09:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.120+0000 7fa50d048700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa508102430 0x7fa50819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.120+0000 7fa50d048700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa508102430 0x7fa50819f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51128/0 (socket says 172.21.15.70:51128) 2024-06-04T05:09:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.121+0000 7fa50d048700 1 -- 172.21.15.70:0/2112583079 learned_addr learned my addr 172.21.15.70:0/2112583079 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.121+0000 7fa50c847700 1 --2- 172.21.15.70:0/2112583079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa508102df0 0x7fa50819fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.121+0000 7fa50d849700 1 --2- 172.21.15.70:0/2112583079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa508108510 0x7fa5081999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.121+0000 7fa50d048700 1 -- 172.21.15.70:0/2112583079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa508102df0 msgr2=0x7fa50819fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.121+0000 7fa50d048700 1 --2- 172.21.15.70:0/2112583079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa508102df0 0x7fa50819fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.121+0000 7fa50d048700 1 -- 172.21.15.70:0/2112583079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa508108510 msgr2=0x7fa5081999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.121+0000 7fa50d048700 1 --2- 172.21.15.70:0/2112583079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa508108510 0x7fa5081999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.121+0000 7fa50d048700 1 -- 172.21.15.70:0/2112583079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50819a100 con 0x7fa508102430 2024-06-04T05:09:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.121+0000 7fa50d048700 1 --2- 172.21.15.70:0/2112583079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa508102430 0x7fa50819f790 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fa504006010 tx=0x7fa5040047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.121+0000 7fa4fe7fc700 1 -- 172.21.15.70:0/2112583079 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa504027070 con 0x7fa508102430 2024-06-04T05:09:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.121+0000 7fa50f2ac700 1 -- 172.21.15.70:0/2112583079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa50819a390 con 0x7fa508102430 2024-06-04T05:09:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.121+0000 7fa4fe7fc700 1 -- 172.21.15.70:0/2112583079 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa504022470 con 0x7fa508102430 2024-06-04T05:09:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.121+0000 7fa4fe7fc700 1 -- 172.21.15.70:0/2112583079 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa504017650 con 0x7fa508102430 2024-06-04T05:09:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.121+0000 7fa50f2ac700 1 -- 172.21.15.70:0/2112583079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5081a6660 con 0x7fa508102430 2024-06-04T05:09:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.124+0000 7fa4fe7fc700 1 -- 172.21.15.70:0/2112583079 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fa504009d10 con 0x7fa508102430 2024-06-04T05:09:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.125+0000 7fa4fe7fc700 1 --2- 172.21.15.70:0/2112583079 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa4f40b29f0 0x7fa4f40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.125+0000 7fa50c847700 1 --2- 172.21.15.70:0/2112583079 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa4f40b29f0 0x7fa4f40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.125+0000 7fa4fe7fc700 1 -- 172.21.15.70:0/2112583079 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8615+0+0 (secure 0 0 0) 0x7fa5040e30f0 con 0x7fa508102430 2024-06-04T05:09:04.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.125+0000 7fa519b41b80 1 Processor -- start 2024-06-04T05:09:04.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.125+0000 7fa519b41b80 1 -- start start 2024-06-04T05:09:04.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa50c847700 1 --2- 172.21.15.70:0/2112583079 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa4f40b29f0 0x7fa4f40b4eb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa508079990 tx=0x7fa4f8009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:04.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa519b41b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657046f8e0 0x55657046fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa519b41b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556570470290 0x55657047d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa519b41b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55657047da40 0x55657047dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa519b41b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5565703ced10 con 0x55657046f8e0 2024-06-04T05:09:04.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa519b41b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5565703be8b0 con 0x55657047da40 2024-06-04T05:09:04.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa519b41b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5565703c28f0 con 0x556570470290 2024-06-04T05:09:04.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa50d048700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657046f8e0 0x55657046fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa50d048700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657046f8e0 0x55657046fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51132/0 (socket says 172.21.15.70:51132) 2024-06-04T05:09:04.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa50d048700 1 -- 172.21.15.70:0/2251089460 learned_addr learned my addr 172.21.15.70:0/2251089460 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:04.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa50d849700 1 --2- 172.21.15.70:0/2251089460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55657047da40 0x55657047dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa50c847700 1 --2- 172.21.15.70:0/2251089460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556570470290 0x55657047d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa50d048700 1 -- 172.21.15.70:0/2251089460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556570470290 msgr2=0x55657047d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:04.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa50d048700 1 --2- 172.21.15.70:0/2251089460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556570470290 0x55657047d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa50d048700 1 -- 172.21.15.70:0/2251089460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55657047da40 msgr2=0x55657047dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:04.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa50d048700 1 --2- 172.21.15.70:0/2251089460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55657047da40 0x55657047dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa50d048700 1 -- 172.21.15.70:0/2251089460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55657030e950 con 0x55657046f8e0 2024-06-04T05:09:04.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.126+0000 7fa50d849700 1 --2- 172.21.15.70:0/2251089460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55657047da40 0x55657047dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:04.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fa50d048700 1 --2- 172.21.15.70:0/2251089460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657046f8e0 0x55657046fcc0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fa5040aa390 tx=0x7fa5040a9c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f49d2e15242c89e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:04.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fa4f37fe700 1 -- 172.21.15.70:0/2251089460 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5040aa400 con 0x55657046f8e0 2024-06-04T05:09:04.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fa4f37fe700 1 -- 172.21.15.70:0/2251089460 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa5040ae7b0 con 0x55657046f8e0 2024-06-04T05:09:04.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fa4f37fe700 1 -- 172.21.15.70:0/2251089460 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5040b2d40 con 0x55657046f8e0 2024-06-04T05:09:04.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fd3537fe640 1 -- 172.21.15.70:0/3982062753 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55b784dfc360 con 0x55b784f18ea0 2024-06-04T05:09:04.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fd3537fe640 10 client.18645 handle_command_reply: tid=0 2024-06-04T05:09:04.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fd37d7eb740 1 client.18645 shutdown 2024-06-04T05:09:04.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fd37d7eb740 1 -- 172.21.15.70:0/3982062753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 msgr2=0x55b784ea18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/3982062753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 0x55b784ea18b0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x55b784e97fa0 tx=0x7fd368010860 comp rx=0 tx=0).stop 2024-06-04T05:09:04.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fd37d7eb740 1 -- 172.21.15.70:0/3982062753 shutdown_connections 2024-06-04T05:09:04.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/3982062753 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b784f18ea0 0x55b784f1b2c0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/3982062753 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b784e99a00 0x55b784ea1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/3982062753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b784e8f140 0x55b784ea18b0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fd3537fe640 0 client.18645 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:04.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fd37d7eb740 1 --2- 172.21.15.70:0/3982062753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b784e8e820 0x55b784ea64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fd37d7eb740 1 -- 172.21.15.70:0/3982062753 >> 172.21.15.70:0/3982062753 conn(0x55b784e8ad30 msgr2=0x55b784e96970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:04.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fd37d7eb740 1 -- 172.21.15.70:0/3982062753 shutdown_connections 2024-06-04T05:09:04.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.127+0000 7fd37d7eb740 1 -- 172.21.15.70:0/3982062753 wait complete. 2024-06-04T05:09:04.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fd37d7eb740 20 client.18645 trim_cache size 0 max 16384 2024-06-04T05:09:04.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:09:04.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:09:04.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:09:04.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fa519b41b80 1 -- 172.21.15.70:0/2251089460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657046f8e0 msgr2=0x55657046fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fa519b41b80 1 --2- 172.21.15.70:0/2251089460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657046f8e0 0x55657046fcc0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fa5040aa390 tx=0x7fa5040a9c40 comp rx=0 tx=0).stop 2024-06-04T05:09:04.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fa519b41b80 1 -- 172.21.15.70:0/2251089460 shutdown_connections 2024-06-04T05:09:04.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fa519b41b80 1 --2- 172.21.15.70:0/2251089460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55657047da40 0x55657047dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fa519b41b80 1 --2- 172.21.15.70:0/2251089460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657046f8e0 0x55657046fcc0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fa519b41b80 1 --2- 172.21.15.70:0/2251089460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556570470290 0x55657047d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fa519b41b80 1 -- 172.21.15.70:0/2251089460 >> 172.21.15.70:0/2251089460 conn(0x55657039d100 msgr2=0x556570469f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:04.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fa519b41b80 1 -- 172.21.15.70:0/2251089460 shutdown_connections 2024-06-04T05:09:04.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fa519b41b80 1 -- 172.21.15.70:0/2251089460 wait complete. 2024-06-04T05:09:04.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fd37a9f7640 1 -- 172.21.15.70:0/2417424125 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd35c0b2a10 msgr2=0x7fd35c0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fd37a9f7640 1 --2- 172.21.15.70:0/2417424125 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd35c0b2a10 0x7fd35c0b4ed0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd36c0049d0 tx=0x7fd36c004a40 comp rx=0 tx=0).stop 2024-06-04T05:09:04.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fd37a9f7640 1 -- 172.21.15.70:0/2417424125 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd374101980 msgr2=0x7fd374199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fd37a9f7640 1 --2- 172.21.15.70:0/2417424125 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd374101980 0x7fd374199f70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd364005940 tx=0x7fd364004040 comp rx=0 tx=0).stop 2024-06-04T05:09:04.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fd37a9f7640 1 -- 172.21.15.70:0/2417424125 shutdown_connections 2024-06-04T05:09:04.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fd37a9f7640 1 --2- 172.21.15.70:0/2417424125 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd35c0b2a10 0x7fd35c0b4ed0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fd37a9f7640 1 --2- 172.21.15.70:0/2417424125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd374107940 0x7fd3741a4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fd37a9f7640 1 --2- 172.21.15.70:0/2417424125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd374102340 0x7fd37419a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fd37a9f7640 1 --2- 172.21.15.70:0/2417424125 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd374101980 0x7fd374199f70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fd37a9f7640 1 -- 172.21.15.70:0/2417424125 >> 172.21.15.70:0/2417424125 conn(0x7fd374078ad0 msgr2=0x7fd374104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:04.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fd37a9f7640 1 -- 172.21.15.70:0/2417424125 shutdown_connections 2024-06-04T05:09:04.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fd37a9f7640 1 -- 172.21.15.70:0/2417424125 wait complete. 2024-06-04T05:09:04.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fa519b41b80 1 -- start start 2024-06-04T05:09:04.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.128+0000 7fa519b41b80 1 Processor -- start 2024-06-04T05:09:04.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.129+0000 7fa519b41b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657046f8e0 0x556570483d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.129+0000 7fa519b41b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556570470290 0x556570481040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.129+0000 7fa519b41b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55657047da40 0x556570481580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.129+0000 7fa519b41b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5565702c5020 con 0x55657046f8e0 2024-06-04T05:09:04.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.129+0000 7fa519b41b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556570360840 con 0x55657047da40 2024-06-04T05:09:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.129+0000 7fa519b41b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556570484790 con 0x556570470290 2024-06-04T05:09:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.129+0000 7fa519b41b80 10 client.0 _ll_register_callbacks cb 0x5565703599a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.129+0000 7fa50c847700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556570470290 0x556570481040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.129+0000 7fa50d048700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657046f8e0 0x556570483d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.129+0000 7fa50c847700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556570470290 0x556570481040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:53422/0 (socket says 172.21.15.70:53422) 2024-06-04T05:09:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.129+0000 7fa50c847700 1 -- 172.21.15.70:0/372695995 learned_addr learned my addr 172.21.15.70:0/372695995 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.129+0000 7fa50d849700 1 --2- 172.21.15.70:0/372695995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55657047da40 0x556570481580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.129+0000 7fa50c847700 1 -- 172.21.15.70:0/372695995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55657047da40 msgr2=0x556570481580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.129+0000 7fa50c847700 1 --2- 172.21.15.70:0/372695995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55657047da40 0x556570481580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.129+0000 7fa50c847700 1 -- 172.21.15.70:0/372695995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657046f8e0 msgr2=0x556570483d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.129+0000 7fa50c847700 1 --2- 172.21.15.70:0/372695995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657046f8e0 0x556570483d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.129+0000 7fa50c847700 1 -- 172.21.15.70:0/372695995 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55657035d710 con 0x556570470290 2024-06-04T05:09:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.130+0000 7fa50c847700 1 --2- 172.21.15.70:0/372695995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556570470290 0x556570481040 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8004580 tx=0x7fa4f80047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=864403e49da962d1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.130+0000 7fa4f0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:09:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.130+0000 7fa4f0ff9700 1 -- 172.21.15.70:0/372695995 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8027420 con 0x556570470290 2024-06-04T05:09:04.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.130+0000 7fa519b41b80 1 -- 172.21.15.70:0/372695995 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5040ae7b0 con 0x556570470290 2024-06-04T05:09:04.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.130+0000 7fa4f0ff9700 1 -- 172.21.15.70:0/372695995 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8615+0+0 (secure 0 0 0) 0x7fa4f801d410 con 0x556570470290 2024-06-04T05:09:04.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.130+0000 7fa4f0ff9700 1 -- 172.21.15.70:0/372695995 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa4f801d650 con 0x556570470290 2024-06-04T05:09:04.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.130+0000 7fa519b41b80 10 client.27781 fetch_fsmap learned FSMap version 35 2024-06-04T05:09:04.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.130+0000 7fa519b41b80 1 -- 172.21.15.70:0/372695995 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55657048ef80 con 0x556570470290 2024-06-04T05:09:04.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.130+0000 7fa4f0ff9700 1 -- 172.21.15.70:0/372695995 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa5040ae7b0 con 0x556570470290 2024-06-04T05:09:04.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.131+0000 7fa4f0ff9700 1 -- 172.21.15.70:0/372695995 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7fa4f80278f0 con 0x556570470290 2024-06-04T05:09:04.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.131+0000 7fa519b41b80 10 client.27781 fetch_fsmap finished waiting for FSMap version 35 2024-06-04T05:09:04.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.131+0000 7fa519b41b80 10 client.27781 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:04.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.131+0000 7fa519b41b80 20 client.27781 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:04.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.131+0000 7fa519b41b80 1 --2- 172.21.15.70:0/372695995 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556570491970 0x556570493db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.131+0000 7fa519b41b80 4 client.27781 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:09:04.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.131+0000 7fa519b41b80 1 -- 172.21.15.70:0/372695995 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55657043ba60 con 0x556570491970 2024-06-04T05:09:04.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.131+0000 7fa50d048700 1 --2- 172.21.15.70:0/372695995 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556570491970 0x556570493db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.132+0000 7fa50d048700 1 --2- 172.21.15.70:0/372695995 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556570491970 0x556570493db0 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef8272bb7c6ee483 server_cookie=b334301201b1e036 in_seq=0 out_seq=0 2024-06-04T05:09:04.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.132+0000 7fa4f0ff9700 10 client.27781 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:04.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.132+0000 7fa4f0ff9700 1 -- 172.21.15.70:0/372695995 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55657043ba60 con 0x556570491970 2024-06-04T05:09:04.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.132+0000 7fa4f0ff9700 10 client.27781 handle_command_reply: tid=0 2024-06-04T05:09:04.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.132+0000 7fa519b41b80 1 client.27781 shutdown 2024-06-04T05:09:04.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.133+0000 7fa519b41b80 1 -- 172.21.15.70:0/372695995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556570470290 msgr2=0x556570481040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.133+0000 7fa519b41b80 1 --2- 172.21.15.70:0/372695995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556570470290 0x556570481040 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8004580 tx=0x7fa4f80047c0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.133+0000 7fa519b41b80 1 -- 172.21.15.70:0/372695995 shutdown_connections 2024-06-04T05:09:04.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.133+0000 7fa519b41b80 1 --2- 172.21.15.70:0/372695995 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556570491970 0x556570493db0 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.133+0000 7fa519b41b80 1 --2- 172.21.15.70:0/372695995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55657047da40 0x556570481580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.133+0000 7fa519b41b80 1 --2- 172.21.15.70:0/372695995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657046f8e0 0x556570483d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.133+0000 7fa4f0ff9700 0 client.27781 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:04.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.133+0000 7fa519b41b80 1 --2- 172.21.15.70:0/372695995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556570470290 0x556570481040 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.133+0000 7fa519b41b80 1 -- 172.21.15.70:0/372695995 >> 172.21.15.70:0/372695995 conn(0x55657039d100 msgr2=0x556570469ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:04.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.133+0000 7fa519b41b80 1 -- 172.21.15.70:0/372695995 shutdown_connections 2024-06-04T05:09:04.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.133+0000 7fa519b41b80 1 -- 172.21.15.70:0/372695995 wait complete. 2024-06-04T05:09:04.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.133+0000 7fa519b41b80 20 client.27781 trim_cache size 0 max 16384 2024-06-04T05:09:04.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.157+0000 7fa519b41b80 1 Processor -- start 2024-06-04T05:09:04.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.157+0000 7fa519b41b80 1 -- start start 2024-06-04T05:09:04.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.157+0000 7fa519b41b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55657046f8e0 0x556570483880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.157+0000 7fa519b41b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556570470290 0x556570469d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.157+0000 7fa519b41b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657047da40 0x55657046a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.157+0000 7fa519b41b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa5081abc70 con 0x55657047da40 2024-06-04T05:09:04.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.157+0000 7fa519b41b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa5081ad840 con 0x556570470290 2024-06-04T05:09:04.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.157+0000 7fa519b41b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa5081ac340 con 0x55657046f8e0 2024-06-04T05:09:04.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.157+0000 7fa50d048700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55657046f8e0 0x556570483880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.157+0000 7fa50d849700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657047da40 0x55657046a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.157+0000 7fa50d048700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55657046f8e0 0x556570483880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:53426/0 (socket says 172.21.15.70:53426) 2024-06-04T05:09:04.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.157+0000 7fa50d048700 1 -- 172.21.15.70:0/4019150623 learned_addr learned my addr 172.21.15.70:0/4019150623 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:04.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.158+0000 7fa50c847700 1 --2- 172.21.15.70:0/4019150623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556570470290 0x556570469d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.158+0000 7fa50d048700 1 -- 172.21.15.70:0/4019150623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556570470290 msgr2=0x556570469d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:04.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.158+0000 7fa50d048700 1 --2- 172.21.15.70:0/4019150623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556570470290 0x556570469d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.158+0000 7fa50d048700 1 -- 172.21.15.70:0/4019150623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657047da40 msgr2=0x55657046a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:04.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.158+0000 7fa50d048700 1 --2- 172.21.15.70:0/4019150623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657047da40 0x55657046a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.158+0000 7fa50d048700 1 -- 172.21.15.70:0/4019150623 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55657047f5d0 con 0x55657046f8e0 2024-06-04T05:09:04.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.158+0000 7fa50d849700 1 --2- 172.21.15.70:0/4019150623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657047da40 0x55657046a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:04.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.158+0000 7fa50d048700 1 --2- 172.21.15.70:0/4019150623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55657046f8e0 0x556570483880 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa5040aa100 tx=0x7fa50400bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4995d5aaf1d53bf server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:04.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.158+0000 7fa4f2ffd700 1 -- 172.21.15.70:0/4019150623 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5040aa400 con 0x55657046f8e0 2024-06-04T05:09:04.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.158+0000 7fa4f2ffd700 1 -- 172.21.15.70:0/4019150623 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa5040a69f0 con 0x55657046f8e0 2024-06-04T05:09:04.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.158+0000 7fa4f2ffd700 1 -- 172.21.15.70:0/4019150623 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5040b2d40 con 0x55657046f8e0 2024-06-04T05:09:04.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.159+0000 7fa519b41b80 1 -- 172.21.15.70:0/4019150623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55657046f8e0 msgr2=0x556570483880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.159+0000 7fa519b41b80 1 --2- 172.21.15.70:0/4019150623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55657046f8e0 0x556570483880 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa5040aa100 tx=0x7fa50400bfd0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.159+0000 7fa519b41b80 1 -- 172.21.15.70:0/4019150623 shutdown_connections 2024-06-04T05:09:04.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.159+0000 7fa519b41b80 1 --2- 172.21.15.70:0/4019150623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55657047da40 0x55657046a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.159+0000 7fa519b41b80 1 --2- 172.21.15.70:0/4019150623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55657046f8e0 0x556570483880 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.159+0000 7fa519b41b80 1 --2- 172.21.15.70:0/4019150623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556570470290 0x556570469d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.159+0000 7fa519b41b80 1 -- 172.21.15.70:0/4019150623 >> 172.21.15.70:0/4019150623 conn(0x55657039d100 msgr2=0x55657046ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:04.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.159+0000 7fa519b41b80 1 -- 172.21.15.70:0/4019150623 shutdown_connections 2024-06-04T05:09:04.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.159+0000 7fa519b41b80 1 -- 172.21.15.70:0/4019150623 wait complete. 2024-06-04T05:09:04.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.160+0000 7fa519b41b80 1 -- start start 2024-06-04T05:09:04.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.160+0000 7fa519b41b80 1 Processor -- start 2024-06-04T05:09:04.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.160+0000 7fa519b41b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55657046f8e0 0x5565704899e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.160+0000 7fa519b41b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556570470290 0x556570489f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.160+0000 7fa519b41b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55657047da40 0x5565704869b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.160+0000 7fa519b41b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa5081ad110 con 0x556570470290 2024-06-04T05:09:04.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.160+0000 7fa519b41b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa5081aca10 con 0x55657046f8e0 2024-06-04T05:09:04.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.160+0000 7fa519b41b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa5081a9c50 con 0x55657047da40 2024-06-04T05:09:04.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.160+0000 7fa519b41b80 10 client.0 _ll_register_callbacks cb 0x5565703599a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:04.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.160+0000 7fa50c847700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556570470290 0x556570489f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.160+0000 7fa50c847700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556570470290 0x556570489f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51158/0 (socket says 172.21.15.70:51158) 2024-06-04T05:09:04.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.160+0000 7fa50c847700 1 -- 172.21.15.70:0/805348584 learned_addr learned my addr 172.21.15.70:0/805348584 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:04.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.160+0000 7fa50d849700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55657047da40 0x5565704869b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.160+0000 7fa50d048700 1 --2- 172.21.15.70:0/805348584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55657046f8e0 0x5565704899e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.161+0000 7fa50c847700 1 -- 172.21.15.70:0/805348584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55657047da40 msgr2=0x5565704869b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:04.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.161+0000 7fa50c847700 1 --2- 172.21.15.70:0/805348584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55657047da40 0x5565704869b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.161+0000 7fa50c847700 1 -- 172.21.15.70:0/805348584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55657046f8e0 msgr2=0x5565704899e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:04.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.161+0000 7fa50c847700 1 --2- 172.21.15.70:0/805348584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55657046f8e0 0x5565704899e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.161+0000 7fa50c847700 1 -- 172.21.15.70:0/805348584 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556570486ef0 con 0x556570470290 2024-06-04T05:09:04.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.161+0000 7fa50c847700 1 --2- 172.21.15.70:0/805348584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556570470290 0x556570489f20 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x55657046c2f0 tx=0x7fa4f801dd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21ceffceb2e273 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:04.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.161+0000 7fa4f0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:04.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.161+0000 7fa4f0ff9700 1 -- 172.21.15.70:0/805348584 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8027420 con 0x556570470290 2024-06-04T05:09:04.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.161+0000 7fa4f0ff9700 1 -- 172.21.15.70:0/805348584 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8615+0+0 (secure 0 0 0) 0x7fa4f80275c0 con 0x556570470290 2024-06-04T05:09:04.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.161+0000 7fa519b41b80 1 -- 172.21.15.70:0/805348584 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5040a69f0 con 0x556570470290 2024-06-04T05:09:04.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.161+0000 7fa4f0ff9700 1 -- 172.21.15.70:0/805348584 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa4f80277c0 con 0x556570470290 2024-06-04T05:09:04.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.161+0000 7fa519b41b80 10 client.18663 fetch_fsmap learned FSMap version 35 2024-06-04T05:09:04.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.161+0000 7fa519b41b80 1 -- 172.21.15.70:0/805348584 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5565704fcdf0 con 0x556570470290 2024-06-04T05:09:04.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.161+0000 7fa4f0ff9700 1 -- 172.21.15.70:0/805348584 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa5040a69f0 con 0x556570470290 2024-06-04T05:09:04.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.162+0000 7fa4f0ff9700 1 -- 172.21.15.70:0/805348584 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7fa4f8035070 con 0x556570470290 2024-06-04T05:09:04.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.162+0000 7fa519b41b80 10 client.18663 fetch_fsmap finished waiting for FSMap version 35 2024-06-04T05:09:04.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.162+0000 7fa519b41b80 10 client.18663 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:04.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.162+0000 7fa519b41b80 20 client.18663 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:04.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.162+0000 7fa519b41b80 1 --2- 172.21.15.70:0/805348584 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556570491970 0x5565704ff620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:04.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.162+0000 7fa519b41b80 4 client.18663 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:09:04.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.162+0000 7fa519b41b80 1 -- 172.21.15.70:0/805348584 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5565703d0c60 con 0x556570491970 2024-06-04T05:09:04.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.162+0000 7fa50d048700 1 --2- 172.21.15.70:0/805348584 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556570491970 0x5565704ff620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.162+0000 7fa4f0ff9700 10 client.18663 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.162+0000 7fa50d048700 1 --2- 172.21.15.70:0/805348584 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556570491970 0x5565704ff620 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d1910815cb2c216 server_cookie=deadd54b5268bf3a in_seq=0 out_seq=0 2024-06-04T05:09:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.163+0000 7fa4f0ff9700 1 -- 172.21.15.70:0/805348584 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5565703d0c60 con 0x556570491970 2024-06-04T05:09:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.163+0000 7fa4f0ff9700 10 client.18663 handle_command_reply: tid=0 2024-06-04T05:09:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.163+0000 7fa519b41b80 1 client.18663 shutdown 2024-06-04T05:09:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.163+0000 7fa519b41b80 1 -- 172.21.15.70:0/805348584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556570470290 msgr2=0x556570489f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.163+0000 7fa519b41b80 1 --2- 172.21.15.70:0/805348584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556570470290 0x556570489f20 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x55657046c2f0 tx=0x7fa4f801dd30 comp rx=0 tx=0).stop 2024-06-04T05:09:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.163+0000 7fa519b41b80 1 -- 172.21.15.70:0/805348584 shutdown_connections 2024-06-04T05:09:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.163+0000 7fa519b41b80 1 --2- 172.21.15.70:0/805348584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55657047da40 0x5565704869b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.163+0000 7fa519b41b80 1 --2- 172.21.15.70:0/805348584 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556570491970 0x5565704ff620 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.164+0000 7fa519b41b80 1 --2- 172.21.15.70:0/805348584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55657046f8e0 0x5565704899e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.164+0000 7fa519b41b80 1 --2- 172.21.15.70:0/805348584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556570470290 0x556570489f20 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.164+0000 7fa4f0ff9700 0 client.18663 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.164+0000 7fa519b41b80 1 -- 172.21.15.70:0/805348584 >> 172.21.15.70:0/805348584 conn(0x55657039d100 msgr2=0x55657047b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.164+0000 7fa519b41b80 1 -- 172.21.15.70:0/805348584 shutdown_connections 2024-06-04T05:09:04.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.164+0000 7fa519b41b80 1 -- 172.21.15.70:0/805348584 wait complete. 2024-06-04T05:09:04.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.164+0000 7fa519b41b80 20 client.18663 trim_cache size 0 max 16384 2024-06-04T05:09:04.172 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:09:04.172 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:09:04.172 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "c8a8a279-6825-4a7c-94f1-c3913d3c1186", 2024-06-04T05:09:04.172 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:09:04.172 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:09:04.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.164+0000 7fa50f2ac700 1 -- 172.21.15.70:0/2112583079 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa4f40b29f0 msgr2=0x7fa4f40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.164+0000 7fa50f2ac700 1 --2- 172.21.15.70:0/2112583079 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa4f40b29f0 0x7fa4f40b4eb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa508079990 tx=0x7fa4f8009260 comp rx=0 tx=0).stop 2024-06-04T05:09:04.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.164+0000 7fa50f2ac700 1 -- 172.21.15.70:0/2112583079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa508102430 msgr2=0x7fa50819f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:04.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.164+0000 7fa50f2ac700 1 --2- 172.21.15.70:0/2112583079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa508102430 0x7fa50819f790 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fa504006010 tx=0x7fa5040047b0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.165+0000 7fa50f2ac700 1 -- 172.21.15.70:0/2112583079 shutdown_connections 2024-06-04T05:09:04.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.165+0000 7fa50f2ac700 1 --2- 172.21.15.70:0/2112583079 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa4f40b29f0 0x7fa4f40b4eb0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.165+0000 7fa50f2ac700 1 --2- 172.21.15.70:0/2112583079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa508108510 0x7fa5081999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.165+0000 7fa50f2ac700 1 --2- 172.21.15.70:0/2112583079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa508102430 0x7fa50819f790 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.165+0000 7fa50f2ac700 1 --2- 172.21.15.70:0/2112583079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa508102df0 0x7fa50819fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:04.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.165+0000 7fa50f2ac700 1 -- 172.21.15.70:0/2112583079 >> 172.21.15.70:0/2112583079 conn(0x7fa5080793b0 msgr2=0x7fa50806bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:04.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.165+0000 7fa50f2ac700 1 -- 172.21.15.70:0/2112583079 shutdown_connections 2024-06-04T05:09:04.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:04.165+0000 7fa50f2ac700 1 -- 172.21.15.70:0/2112583079 wait complete. 2024-06-04T05:09:04.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:04 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:09:04.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:04 smithi070 ceph-mon[32631]: osdmap e91: 12 total, 12 up, 12 in 2024-06-04T05:09:04.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:04 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:09:04.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:04 smithi112 ceph-mon[36453]: osdmap e91: 12 total, 12 up, 12 in 2024-06-04T05:09:04.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 201073 1.75 MB/sec execute 477 sec latency 30.177 ms 2024-06-04T05:09:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:04 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:09:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:04 smithi098 ceph-mon[35569]: osdmap e91: 12 total, 12 up, 12 in 2024-06-04T05:09:05.101 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:09:05.408 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:09:05.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: pgmap v369: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 1.1 MiB/s wr, 110 op/s 2024-06-04T05:09:05.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:09:05.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:09:05.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:09:05.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:09:05.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:09:05.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:09:05.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:09:05.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:09:05.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:09:05.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:09:05.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:05 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:09:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: pgmap v369: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 1.1 MiB/s wr, 110 op/s 2024-06-04T05:09:05.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:09:05.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:09:05.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:09:05.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:09:05.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:09:05.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:09:05.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:09:05.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:09:05.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:09:05.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:09:05.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:05 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:09:05.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 201846 1.76 MB/sec execute 478 sec latency 15.606 ms 2024-06-04T05:09:05.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: pgmap v369: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 1.1 MiB/s wr, 110 op/s 2024-06-04T05:09:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:09:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:09:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:09:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:09:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:09:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:09:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:09:05.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:09:05.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:09:05.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:09:05.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:09:05.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:05 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:09:06.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 202255 1.77 MB/sec execute 479 sec latency 78.186 ms 2024-06-04T05:09:07.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:07 smithi070 ceph-mon[32631]: pgmap v370: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-06-04T05:09:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:07 smithi112 ceph-mon[36453]: pgmap v370: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-06-04T05:09:07.737 INFO:tasks.workunit.client.0.smithi070.stdout: 1 202322 1.77 MB/sec execute 480 sec latency 426.867 ms 2024-06-04T05:09:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:07 smithi098 ceph-mon[35569]: pgmap v370: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-06-04T05:09:07.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.277+0000 7f6083652700 1 -- 172.21.15.70:0/2643393017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f607c103f90 msgr2=0x7f607c110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:07.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.277+0000 7f6083652700 1 --2- 172.21.15.70:0/2643393017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f607c103f90 0x7f607c110440 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f607400b0d0 tx=0x7f607400b3e0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.277+0000 7f6083652700 1 -- 172.21.15.70:0/2643393017 shutdown_connections 2024-06-04T05:09:07.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.277+0000 7f6083652700 1 --2- 172.21.15.70:0/2643393017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f607c103f90 0x7f607c110440 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.277+0000 7f6083652700 1 --2- 172.21.15.70:0/2643393017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f607c109670 0x7f607c109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.277+0000 7f6083652700 1 --2- 172.21.15.70:0/2643393017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f607c1035d0 0x7f607c103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.277+0000 7f6083652700 1 -- 172.21.15.70:0/2643393017 >> 172.21.15.70:0/2643393017 conn(0x7f607c07a730 msgr2=0x7f607c07ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:07.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.277+0000 7f6083652700 1 -- 172.21.15.70:0/2643393017 shutdown_connections 2024-06-04T05:09:07.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.277+0000 7f6083652700 1 -- 172.21.15.70:0/2643393017 wait complete. 2024-06-04T05:09:07.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.278+0000 7f6083652700 1 Processor -- start 2024-06-04T05:09:07.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.278+0000 7f6083652700 1 -- start start 2024-06-04T05:09:07.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.278+0000 7f6083652700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f607c1035d0 0x7f607c19f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.278+0000 7f6083652700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f607c103f90 0x7f607c19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.278+0000 7f6083652700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f607c109670 0x7f607c1998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.278+0000 7f6083652700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f607c113680 con 0x7f607c1035d0 2024-06-04T05:09:07.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.278+0000 7f6083652700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f607c113500 con 0x7f607c103f90 2024-06-04T05:09:07.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.278+0000 7f6083652700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f607c113800 con 0x7f607c109670 2024-06-04T05:09:07.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.278+0000 7f60813ee700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f607c1035d0 0x7f607c19f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.278+0000 7f60813ee700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f607c1035d0 0x7f607c19f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49654/0 (socket says 172.21.15.70:49654) 2024-06-04T05:09:07.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.278+0000 7f60813ee700 1 -- 172.21.15.70:0/592629600 learned_addr learned my addr 172.21.15.70:0/592629600 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:07.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.278+0000 7f6080bed700 1 --2- 172.21.15.70:0/592629600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f607c103f90 0x7f607c19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.278+0000 7f6081bef700 1 --2- 172.21.15.70:0/592629600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f607c109670 0x7f607c1998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.278+0000 7f60813ee700 1 -- 172.21.15.70:0/592629600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f607c109670 msgr2=0x7f607c1998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:07.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.278+0000 7f60813ee700 1 --2- 172.21.15.70:0/592629600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f607c109670 0x7f607c1998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.278+0000 7f60813ee700 1 -- 172.21.15.70:0/592629600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f607c103f90 msgr2=0x7f607c19fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:07.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.278+0000 7f60813ee700 1 --2- 172.21.15.70:0/592629600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f607c103f90 0x7f607c19fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.279+0000 7f60813ee700 1 -- 172.21.15.70:0/592629600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607c19a020 con 0x7f607c1035d0 2024-06-04T05:09:07.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.279+0000 7f60813ee700 1 --2- 172.21.15.70:0/592629600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f607c1035d0 0x7f607c19f6b0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f607800d980 tx=0x7f607800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:07.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.279+0000 7f60727fc700 1 -- 172.21.15.70:0/592629600 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078016070 con 0x7f607c1035d0 2024-06-04T05:09:07.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.279+0000 7f6083652700 1 -- 172.21.15.70:0/592629600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f607c19a310 con 0x7f607c1035d0 2024-06-04T05:09:07.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.279+0000 7f60727fc700 1 -- 172.21.15.70:0/592629600 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6078004830 con 0x7f607c1035d0 2024-06-04T05:09:07.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.279+0000 7f60727fc700 1 -- 172.21.15.70:0/592629600 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60780054c0 con 0x7f607c1035d0 2024-06-04T05:09:07.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.279+0000 7f6083652700 1 -- 172.21.15.70:0/592629600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f607c1a6580 con 0x7f607c1035d0 2024-06-04T05:09:07.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.281+0000 7f60727fc700 1 -- 172.21.15.70:0/592629600 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f6078004220 con 0x7f607c1035d0 2024-06-04T05:09:07.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.282+0000 7f60727fc700 1 --2- 172.21.15.70:0/592629600 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f60680b29f0 0x7f60680b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.282+0000 7f6080bed700 1 --2- 172.21.15.70:0/592629600 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f60680b29f0 0x7f60680b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.282+0000 7f60727fc700 1 -- 172.21.15.70:0/592629600 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8615+0+0 (secure 0 0 0) 0x7f60780e2350 con 0x7f607c1035d0 2024-06-04T05:09:07.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.283+0000 7f6080bed700 1 --2- 172.21.15.70:0/592629600 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f60680b29f0 0x7f60680b4eb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f607c07ae10 tx=0x7f606c0073d0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:07.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.283+0000 7f608dee7b80 1 Processor -- start 2024-06-04T05:09:07.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.283+0000 7f608dee7b80 1 -- start start 2024-06-04T05:09:07.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f608dee7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f396c0 0x557a73f39aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f608dee7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 0x557a73f34130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f608dee7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f34670 0x557a73f40460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f608dee7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557a73e8dd70 con 0x557a73f396c0 2024-06-04T05:09:07.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f608dee7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557a73e7d910 con 0x557a73f34670 2024-06-04T05:09:07.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f608dee7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557a73e81950 con 0x557a73f33cb0 2024-06-04T05:09:07.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f60813ee700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f396c0 0x557a73f39aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f6080bed700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 0x557a73f34130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f60813ee700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f396c0 0x557a73f39aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49656/0 (socket says 172.21.15.70:49656) 2024-06-04T05:09:07.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f60813ee700 1 -- 172.21.15.70:0/1309871161 learned_addr learned my addr 172.21.15.70:0/1309871161 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:07.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f6081bef700 1 --2- 172.21.15.70:0/1309871161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f34670 0x557a73f40460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f60813ee700 1 -- 172.21.15.70:0/1309871161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 msgr2=0x557a73f34130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:07.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f60813ee700 1 --2- 172.21.15.70:0/1309871161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 0x557a73f34130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f60813ee700 1 -- 172.21.15.70:0/1309871161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f34670 msgr2=0x557a73f40460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:07.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f60813ee700 1 --2- 172.21.15.70:0/1309871161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f34670 0x557a73f40460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f60813ee700 1 -- 172.21.15.70:0/1309871161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a73e26fb0 con 0x557a73f396c0 2024-06-04T05:09:07.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f6080bed700 1 --2- 172.21.15.70:0/1309871161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 0x557a73f34130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:07.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f6081bef700 1 --2- 172.21.15.70:0/1309871161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f34670 0x557a73f40460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:07.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f60813ee700 1 --2- 172.21.15.70:0/1309871161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f396c0 0x557a73f39aa0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f6078004ff0 tx=0x7f60780a91b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b615200b754552 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:07.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f60677fe700 1 -- 172.21.15.70:0/1309871161 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60780a95d0 con 0x557a73f396c0 2024-06-04T05:09:07.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.284+0000 7f60677fe700 1 -- 172.21.15.70:0/1309871161 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f60780ad980 con 0x557a73f396c0 2024-06-04T05:09:07.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.285+0000 7f60677fe700 1 -- 172.21.15.70:0/1309871161 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60780a9830 con 0x557a73f396c0 2024-06-04T05:09:07.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.285+0000 7f608dee7b80 1 -- 172.21.15.70:0/1309871161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f396c0 msgr2=0x557a73f39aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:07.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.285+0000 7f608dee7b80 1 --2- 172.21.15.70:0/1309871161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f396c0 0x557a73f39aa0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f6078004ff0 tx=0x7f60780a91b0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.285+0000 7f608dee7b80 1 -- 172.21.15.70:0/1309871161 shutdown_connections 2024-06-04T05:09:07.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.285+0000 7f608dee7b80 1 --2- 172.21.15.70:0/1309871161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f34670 0x557a73f40460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.285+0000 7f608dee7b80 1 --2- 172.21.15.70:0/1309871161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f396c0 0x557a73f39aa0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.285+0000 7f608dee7b80 1 --2- 172.21.15.70:0/1309871161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 0x557a73f34130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.286+0000 7f608dee7b80 1 -- 172.21.15.70:0/1309871161 >> 172.21.15.70:0/1309871161 conn(0x557a73e5b240 msgr2=0x557a73f2a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:07.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.286+0000 7f608dee7b80 1 -- 172.21.15.70:0/1309871161 shutdown_connections 2024-06-04T05:09:07.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.286+0000 7f608dee7b80 1 -- 172.21.15.70:0/1309871161 wait complete. 2024-06-04T05:09:07.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.286+0000 7f608dee7b80 1 -- start start 2024-06-04T05:09:07.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.286+0000 7f608dee7b80 1 Processor -- start 2024-06-04T05:09:07.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f608dee7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 0x557a73f400a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f608dee7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f34670 0x557a73f3d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f608dee7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f396c0 0x557a73f3d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f608dee7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557a73d83fa0 con 0x557a73f34670 2024-06-04T05:09:07.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f608dee7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557a73de5da0 con 0x557a73f396c0 2024-06-04T05:09:07.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f608dee7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557a73f40ba0 con 0x557a73f33cb0 2024-06-04T05:09:07.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f608dee7b80 10 client.0 _ll_register_callbacks cb 0x557a73e18950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:07.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f6080bed700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f34670 0x557a73f3d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f6080bed700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f34670 0x557a73f3d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49666/0 (socket says 172.21.15.70:49666) 2024-06-04T05:09:07.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f6080bed700 1 -- 172.21.15.70:0/659937649 learned_addr learned my addr 172.21.15.70:0/659937649 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:07.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f60813ee700 1 --2- 172.21.15.70:0/659937649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 0x557a73f400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f6081bef700 1 --2- 172.21.15.70:0/659937649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f396c0 0x557a73f3d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f60813ee700 1 -- 172.21.15.70:0/659937649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f396c0 msgr2=0x557a73f3d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:07.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f60813ee700 1 --2- 172.21.15.70:0/659937649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f396c0 0x557a73f3d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f60813ee700 1 -- 172.21.15.70:0/659937649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f34670 msgr2=0x557a73f3d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:07.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f60813ee700 1 --2- 172.21.15.70:0/659937649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f34670 0x557a73f3d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f60813ee700 1 -- 172.21.15.70:0/659937649 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a73dcd8d0 con 0x557a73f33cb0 2024-06-04T05:09:07.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f6081bef700 1 --2- 172.21.15.70:0/659937649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f396c0 0x557a73f3d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:09:07.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f6080bed700 1 --2- 172.21.15.70:0/659937649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f34670 0x557a73f3d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:07.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f60813ee700 1 --2- 172.21.15.70:0/659937649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 0x557a73f400a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f60780b25f0 tx=0x7f6078005e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2103a91c6ad6ea3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:07.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.287+0000 7f6064ff9700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:09:07.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.288+0000 7f6064ff9700 1 -- 172.21.15.70:0/659937649 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60780a5bc0 con 0x557a73f33cb0 2024-06-04T05:09:07.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.288+0000 7f6064ff9700 1 -- 172.21.15.70:0/659937649 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8615+0+0 (secure 0 0 0) 0x7f60780a9830 con 0x557a73f33cb0 2024-06-04T05:09:07.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.288+0000 7f608dee7b80 1 -- 172.21.15.70:0/659937649 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60780ad980 con 0x557a73f33cb0 2024-06-04T05:09:07.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.288+0000 7f6064ff9700 1 -- 172.21.15.70:0/659937649 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f60780a9a30 con 0x557a73f33cb0 2024-06-04T05:09:07.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.288+0000 7f608dee7b80 10 client.27799 fetch_fsmap learned FSMap version 36 2024-06-04T05:09:07.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.288+0000 7f608dee7b80 1 -- 172.21.15.70:0/659937649 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x557a73e37760 con 0x557a73f33cb0 2024-06-04T05:09:07.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.288+0000 7f6064ff9700 1 -- 172.21.15.70:0/659937649 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f60780ad980 con 0x557a73f33cb0 2024-06-04T05:09:07.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.288+0000 7f6064ff9700 1 -- 172.21.15.70:0/659937649 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f60780b2010 con 0x557a73f33cb0 2024-06-04T05:09:07.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.288+0000 7f608dee7b80 10 client.27799 fetch_fsmap finished waiting for FSMap version 36 2024-06-04T05:09:07.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.288+0000 7f608dee7b80 10 client.27799 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:07.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.288+0000 7f608dee7b80 20 client.27799 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:07.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.288+0000 7f608dee7b80 1 --2- 172.21.15.70:0/659937649 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557a73f4d5b0 0x557a73f4f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.288+0000 7f608dee7b80 4 client.27799 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:09:07.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.288+0000 7f608dee7b80 1 -- 172.21.15.70:0/659937649 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a73e8fcc0 con 0x557a73f4d5b0 2024-06-04T05:09:07.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.288+0000 7f6080bed700 1 --2- 172.21.15.70:0/659937649 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557a73f4d5b0 0x557a73f4f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.289+0000 7f6064ff9700 10 client.27799 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:07.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.289+0000 7f6080bed700 1 --2- 172.21.15.70:0/659937649 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557a73f4d5b0 0x557a73f4f9a0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd89fb67e58bb3ac server_cookie=84ce357ae8fb78ad in_seq=0 out_seq=0 2024-06-04T05:09:07.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.289+0000 7f6064ff9700 1 -- 172.21.15.70:0/659937649 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557a73e8fcc0 con 0x557a73f4d5b0 2024-06-04T05:09:07.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.289+0000 7f6064ff9700 10 client.27799 handle_command_reply: tid=0 2024-06-04T05:09:07.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.289+0000 7f608dee7b80 1 client.27799 shutdown 2024-06-04T05:09:07.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.290+0000 7f608dee7b80 1 -- 172.21.15.70:0/659937649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 msgr2=0x557a73f400a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:07.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.290+0000 7f608dee7b80 1 --2- 172.21.15.70:0/659937649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 0x557a73f400a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f60780b25f0 tx=0x7f6078005e40 comp rx=0 tx=0).stop 2024-06-04T05:09:07.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.290+0000 7f608dee7b80 1 -- 172.21.15.70:0/659937649 shutdown_connections 2024-06-04T05:09:07.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.290+0000 7f608dee7b80 1 --2- 172.21.15.70:0/659937649 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557a73f4d5b0 0x557a73f4f9a0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.290+0000 7f608dee7b80 1 --2- 172.21.15.70:0/659937649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f396c0 0x557a73f3d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.290+0000 7f6064ff9700 0 client.27799 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:07.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.290+0000 7f608dee7b80 1 --2- 172.21.15.70:0/659937649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 0x557a73f400a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.290+0000 7f608dee7b80 1 --2- 172.21.15.70:0/659937649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f34670 0x557a73f3d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.290+0000 7f608dee7b80 1 -- 172.21.15.70:0/659937649 >> 172.21.15.70:0/659937649 conn(0x557a73e5b240 msgr2=0x557a73f29af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:07.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.290+0000 7f608dee7b80 1 -- 172.21.15.70:0/659937649 shutdown_connections 2024-06-04T05:09:07.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.290+0000 7f608dee7b80 1 -- 172.21.15.70:0/659937649 wait complete. 2024-06-04T05:09:07.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.290+0000 7f608dee7b80 20 client.27799 trim_cache size 0 max 16384 2024-06-04T05:09:07.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.313+0000 7f608dee7b80 1 Processor -- start 2024-06-04T05:09:07.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.313+0000 7f608dee7b80 1 -- start start 2024-06-04T05:09:07.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f608dee7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 0x557a73f2a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f608dee7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f34670 0x557a73f3f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f608dee7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f396c0 0x557a73f529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f608dee7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f607c1abbf0 con 0x557a73f34670 2024-06-04T05:09:07.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f608dee7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f607c1ad7f0 con 0x557a73f396c0 2024-06-04T05:09:07.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f608dee7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f607c1ac2c0 con 0x557a73f33cb0 2024-06-04T05:09:07.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f60813ee700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 0x557a73f2a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f6080bed700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f34670 0x557a73f3f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f60813ee700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 0x557a73f2a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36886/0 (socket says 172.21.15.70:36886) 2024-06-04T05:09:07.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f60813ee700 1 -- 172.21.15.70:0/693214678 learned_addr learned my addr 172.21.15.70:0/693214678 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:07.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f6081bef700 1 --2- 172.21.15.70:0/693214678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f396c0 0x557a73f529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f6081bef700 1 -- 172.21.15.70:0/693214678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 msgr2=0x557a73f2a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f6081bef700 1 --2- 172.21.15.70:0/693214678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 0x557a73f2a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f6081bef700 1 -- 172.21.15.70:0/693214678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f34670 msgr2=0x557a73f3f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f6081bef700 1 --2- 172.21.15.70:0/693214678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f34670 0x557a73f3f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f6081bef700 1 -- 172.21.15.70:0/693214678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a73f342e0 con 0x557a73f396c0 2024-06-04T05:09:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f6080bed700 1 --2- 172.21.15.70:0/693214678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f34670 0x557a73f3f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.314+0000 7f6081bef700 1 --2- 172.21.15.70:0/693214678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f396c0 0x557a73f529c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6074009fd0 tx=0x7f607400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19b238b5a089e29f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.315+0000 7f6066ffd700 1 -- 172.21.15.70:0/693214678 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607402b070 con 0x557a73f396c0 2024-06-04T05:09:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.315+0000 7f6066ffd700 1 -- 172.21.15.70:0/693214678 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6074005070 con 0x557a73f396c0 2024-06-04T05:09:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.315+0000 7f6066ffd700 1 -- 172.21.15.70:0/693214678 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074007740 con 0x557a73f396c0 2024-06-04T05:09:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.315+0000 7f608dee7b80 1 -- 172.21.15.70:0/693214678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f396c0 msgr2=0x557a73f529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.315+0000 7f608dee7b80 1 --2- 172.21.15.70:0/693214678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f396c0 0x557a73f529c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6074009fd0 tx=0x7f607400b040 comp rx=0 tx=0).stop 2024-06-04T05:09:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.315+0000 7f608dee7b80 1 -- 172.21.15.70:0/693214678 shutdown_connections 2024-06-04T05:09:07.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.315+0000 7f608dee7b80 1 --2- 172.21.15.70:0/693214678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f396c0 0x557a73f529c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.315+0000 7f608dee7b80 1 --2- 172.21.15.70:0/693214678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f33cb0 0x557a73f2a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.315+0000 7f608dee7b80 1 --2- 172.21.15.70:0/693214678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f34670 0x557a73f3f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.315+0000 7f608dee7b80 1 -- 172.21.15.70:0/693214678 >> 172.21.15.70:0/693214678 conn(0x557a73e5b240 msgr2=0x557a73f3ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:07.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.316+0000 7f608dee7b80 1 -- 172.21.15.70:0/693214678 shutdown_connections 2024-06-04T05:09:07.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.316+0000 7f608dee7b80 1 -- 172.21.15.70:0/693214678 wait complete. 2024-06-04T05:09:07.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.316+0000 7f608dee7b80 1 -- start start 2024-06-04T05:09:07.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.316+0000 7f608dee7b80 1 Processor -- start 2024-06-04T05:09:07.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.316+0000 7f608dee7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f33cb0 0x557a73f508d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.316+0000 7f608dee7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f34670 0x557a73f50e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.316+0000 7f608dee7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f396c0 0x557a73f51350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.316+0000 7f608dee7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f607c1ad0c0 con 0x557a73f33cb0 2024-06-04T05:09:07.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.316+0000 7f608dee7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f607c1ac9c0 con 0x557a73f34670 2024-06-04T05:09:07.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.316+0000 7f608dee7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f607c1a9bd0 con 0x557a73f396c0 2024-06-04T05:09:07.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.316+0000 7f60813ee700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f33cb0 0x557a73f508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.316+0000 7f608dee7b80 10 client.0 _ll_register_callbacks cb 0x557a73e18950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:07.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f60813ee700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f33cb0 0x557a73f508d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49690/0 (socket says 172.21.15.70:49690) 2024-06-04T05:09:07.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f6081bef700 1 --2- 172.21.15.70:0/3407808052 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f396c0 0x557a73f51350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f60813ee700 1 -- 172.21.15.70:0/3407808052 learned_addr learned my addr 172.21.15.70:0/3407808052 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:07.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f6080bed700 1 --2- 172.21.15.70:0/3407808052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f34670 0x557a73f50e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f60813ee700 1 -- 172.21.15.70:0/3407808052 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f396c0 msgr2=0x557a73f51350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:07.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f60813ee700 1 --2- 172.21.15.70:0/3407808052 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f396c0 0x557a73f51350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f60813ee700 1 -- 172.21.15.70:0/3407808052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f34670 msgr2=0x557a73f50e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:07.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f60813ee700 1 --2- 172.21.15.70:0/3407808052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f34670 0x557a73f50e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f60813ee700 1 -- 172.21.15.70:0/3407808052 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a73fb6fc0 con 0x557a73f33cb0 2024-06-04T05:09:07.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f6081bef700 1 --2- 172.21.15.70:0/3407808052 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f396c0 0x557a73f51350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:07.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f6080bed700 1 --2- 172.21.15.70:0/3407808052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f34670 0x557a73f50e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:07.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f60813ee700 1 --2- 172.21.15.70:0/3407808052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f33cb0 0x557a73f508d0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f6078004ff0 tx=0x7f60780a8fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c10bda275e125ac7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:07.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f6064ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:07.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f6064ff9700 1 -- 172.21.15.70:0/3407808052 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60780a5bc0 con 0x557a73f33cb0 2024-06-04T05:09:07.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f6064ff9700 1 -- 172.21.15.70:0/3407808052 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8615+0+0 (secure 0 0 0) 0x7f60780a9830 con 0x557a73f33cb0 2024-06-04T05:09:07.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f608dee7b80 1 -- 172.21.15.70:0/3407808052 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6074005070 con 0x557a73f33cb0 2024-06-04T05:09:07.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f6064ff9700 1 -- 172.21.15.70:0/3407808052 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f60780a5490 con 0x557a73f33cb0 2024-06-04T05:09:07.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f608dee7b80 10 client.18681 fetch_fsmap learned FSMap version 36 2024-06-04T05:09:07.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f608dee7b80 1 -- 172.21.15.70:0/3407808052 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x557a73fb82b0 con 0x557a73f33cb0 2024-06-04T05:09:07.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.317+0000 7f6064ff9700 1 -- 172.21.15.70:0/3407808052 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6074005070 con 0x557a73f33cb0 2024-06-04T05:09:07.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.318+0000 7f6064ff9700 1 -- 172.21.15.70:0/3407808052 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f607808b030 con 0x557a73f33cb0 2024-06-04T05:09:07.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.318+0000 7f608dee7b80 10 client.18681 fetch_fsmap finished waiting for FSMap version 36 2024-06-04T05:09:07.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.318+0000 7f608dee7b80 10 client.18681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:07.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.318+0000 7f608dee7b80 20 client.18681 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:07.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.318+0000 7f608dee7b80 1 --2- 172.21.15.70:0/3407808052 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557a73f4d5b0 0x557a73fba870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:07.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.318+0000 7f608dee7b80 4 client.18681 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:09:07.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.318+0000 7f608dee7b80 1 -- 172.21.15.70:0/3407808052 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557a73e894a0 con 0x557a73f4d5b0 2024-06-04T05:09:07.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.318+0000 7f6080bed700 1 --2- 172.21.15.70:0/3407808052 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557a73f4d5b0 0x557a73fba870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:07.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.318+0000 7f6064ff9700 10 client.18681 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:07.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.318+0000 7f6080bed700 1 --2- 172.21.15.70:0/3407808052 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557a73f4d5b0 0x557a73fba870 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=872db007744b0edf server_cookie=c0273a0d9e0f05fe in_seq=0 out_seq=0 2024-06-04T05:09:07.778 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:09:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:09:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:09:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: "c8a8a279-6825-4a7c-94f1-c3913d3c1186": { 2024-06-04T05:09:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:09:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "c8a8a279-6825-4a7c-94f1-c3913d3c1186", 2024-06-04T05:09:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:09:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:09:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:09:07.779 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:09:07.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.318+0000 7f6064ff9700 1 -- 172.21.15.70:0/3407808052 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x557a73e894a0 con 0x557a73f4d5b0 2024-06-04T05:09:07.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.318+0000 7f6064ff9700 10 client.18681 handle_command_reply: tid=0 2024-06-04T05:09:07.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.318+0000 7f608dee7b80 1 client.18681 shutdown 2024-06-04T05:09:07.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f608dee7b80 1 -- 172.21.15.70:0/3407808052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f33cb0 msgr2=0x557a73f508d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:07.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f608dee7b80 1 --2- 172.21.15.70:0/3407808052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f33cb0 0x557a73f508d0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f6078004ff0 tx=0x7f60780a8fe0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f608dee7b80 1 -- 172.21.15.70:0/3407808052 shutdown_connections 2024-06-04T05:09:07.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f608dee7b80 1 --2- 172.21.15.70:0/3407808052 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a73f396c0 0x557a73f51350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f608dee7b80 1 --2- 172.21.15.70:0/3407808052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a73f33cb0 0x557a73f508d0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f608dee7b80 1 --2- 172.21.15.70:0/3407808052 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557a73f4d5b0 0x557a73fba870 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f608dee7b80 1 --2- 172.21.15.70:0/3407808052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a73f34670 0x557a73f50e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f6064ff9700 0 client.18681 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f608dee7b80 1 -- 172.21.15.70:0/3407808052 >> 172.21.15.70:0/3407808052 conn(0x557a73e5b240 msgr2=0x557a73f37980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f608dee7b80 1 -- 172.21.15.70:0/3407808052 shutdown_connections 2024-06-04T05:09:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f608dee7b80 1 -- 172.21.15.70:0/3407808052 wait complete. 2024-06-04T05:09:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f608dee7b80 20 client.18681 trim_cache size 0 max 16384 2024-06-04T05:09:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f6083652700 1 -- 172.21.15.70:0/592629600 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f60680b29f0 msgr2=0x7f60680b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f6083652700 1 --2- 172.21.15.70:0/592629600 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f60680b29f0 0x7f60680b4eb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f607c07ae10 tx=0x7f606c0073d0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f6083652700 1 -- 172.21.15.70:0/592629600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f607c1035d0 msgr2=0x7f607c19f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f6083652700 1 --2- 172.21.15.70:0/592629600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f607c1035d0 0x7f607c19f6b0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f607800d980 tx=0x7f607800dc90 comp rx=0 tx=0).stop 2024-06-04T05:09:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f6083652700 1 -- 172.21.15.70:0/592629600 shutdown_connections 2024-06-04T05:09:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f6083652700 1 --2- 172.21.15.70:0/592629600 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f60680b29f0 0x7f60680b4eb0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f6083652700 1 --2- 172.21.15.70:0/592629600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f607c109670 0x7f607c1998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f6083652700 1 --2- 172.21.15.70:0/592629600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f607c1035d0 0x7f607c19f6b0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f6083652700 1 --2- 172.21.15.70:0/592629600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f607c103f90 0x7f607c19fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:07.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.319+0000 7f6083652700 1 -- 172.21.15.70:0/592629600 >> 172.21.15.70:0/592629600 conn(0x7f607c07a730 msgr2=0x7f607c1022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:07.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.320+0000 7f6083652700 1 -- 172.21.15.70:0/592629600 shutdown_connections 2024-06-04T05:09:07.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:07.320+0000 7f6083652700 1 -- 172.21.15.70:0/592629600 wait complete. 2024-06-04T05:09:08.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 202322 1.76 MB/sec execute 481 sec latency 1426.983 ms 2024-06-04T05:09:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:08 smithi112 ceph-mon[36453]: osdmap e92: 12 total, 12 up, 12 in 2024-06-04T05:09:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:08 smithi112 ceph-mon[36453]: pgmap v372: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-06-04T05:09:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:08 smithi098 ceph-mon[35569]: osdmap e92: 12 total, 12 up, 12 in 2024-06-04T05:09:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:08 smithi098 ceph-mon[35569]: pgmap v372: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-06-04T05:09:09.322 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:08 smithi070 ceph-mon[32631]: osdmap e92: 12 total, 12 up, 12 in 2024-06-04T05:09:09.322 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:08 smithi070 ceph-mon[32631]: pgmap v372: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-06-04T05:09:09.357 INFO:tasks.cephfs.filesystem:scrub status for tag:c8a8a279-6825-4a7c-94f1-c3913d3c1186 - {'path': '/', 'tag': 'c8a8a279-6825-4a7c-94f1-c3913d3c1186', 'options': 'recursive,force'} 2024-06-04T05:09:09.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 202322 1.76 MB/sec execute 482 sec latency 2427.054 ms 2024-06-04T05:09:10.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:09 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:09:10.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:09 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:09:10.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:09 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:09:10.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 202322 1.75 MB/sec execute 483 sec latency 3427.150 ms 2024-06-04T05:09:11.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:10 smithi112 ceph-mon[36453]: pgmap v373: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-06-04T05:09:11.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:10 smithi098 ceph-mon[35569]: pgmap v373: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-06-04T05:09:11.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:10 smithi070 ceph-mon[32631]: pgmap v373: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-06-04T05:09:11.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 202322 1.75 MB/sec execute 484 sec latency 4427.256 ms 2024-06-04T05:09:12.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 202528 1.75 MB/sec execute 485 sec latency 4738.087 ms 2024-06-04T05:09:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:13 smithi112 ceph-mon[36453]: pgmap v374: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-06-04T05:09:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:13 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:09:13.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:13 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:09:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:13 smithi098 ceph-mon[35569]: pgmap v374: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-06-04T05:09:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:13 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:09:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:13 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:09:13.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:13 smithi070 ceph-mon[32631]: pgmap v374: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-06-04T05:09:13.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:13 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:09:13.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:13 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:09:13.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 202842 1.75 MB/sec execute 486 sec latency 17.697 ms 2024-06-04T05:09:14.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:14 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:09:14.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:14 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:09:14.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:14 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:09:14.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 203368 1.75 MB/sec execute 487 sec latency 15.256 ms 2024-06-04T05:09:15.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:15 smithi112 ceph-mon[36453]: pgmap v375: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-06-04T05:09:15.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:15 smithi098 ceph-mon[35569]: pgmap v375: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-06-04T05:09:15.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:15 smithi070 ceph-mon[32631]: pgmap v375: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-06-04T05:09:15.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 203893 1.75 MB/sec execute 488 sec latency 17.280 ms 2024-06-04T05:09:16.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 204541 1.75 MB/sec execute 489 sec latency 17.970 ms 2024-06-04T05:09:17.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.228+0000 7f7565332640 1 -- 172.21.15.70:0/3253486882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f756010d060 msgr2=0x7f756010f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:17.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.228+0000 7f7565332640 1 --2- 172.21.15.70:0/3253486882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f756010d060 0x7f756010f4b0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f7558009ca0 tx=0x7f755802a950 comp rx=0 tx=0).stop 2024-06-04T05:09:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.228+0000 7f7565332640 1 -- 172.21.15.70:0/3253486882 shutdown_connections 2024-06-04T05:09:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.228+0000 7f7565332640 1 --2- 172.21.15.70:0/3253486882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7560104640 0x7f756010cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.228+0000 7f7565332640 1 --2- 172.21.15.70:0/3253486882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f756010d060 0x7f756010f4b0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.228+0000 7f7565332640 1 --2- 172.21.15.70:0/3253486882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7560103d20 0x7f7560104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.228+0000 7f7565332640 1 -- 172.21.15.70:0/3253486882 >> 172.21.15.70:0/3253486882 conn(0x7f7560078ad0 msgr2=0x7f7560078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.228+0000 7f7565332640 1 -- 172.21.15.70:0/3253486882 shutdown_connections 2024-06-04T05:09:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.228+0000 7f7565332640 1 -- 172.21.15.70:0/3253486882 wait complete. 2024-06-04T05:09:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.229+0000 7f7565332640 1 Processor -- start 2024-06-04T05:09:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.229+0000 7f7565332640 1 -- start start 2024-06-04T05:09:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.229+0000 7f7565332640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7560103d20 0x7f7560199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.229+0000 7f7565332640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7560104640 0x7f7560199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.229+0000 7f7565332640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f756010d060 0x7f756019d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.229+0000 7f7565332640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7560112b70 con 0x7f7560103d20 2024-06-04T05:09:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.229+0000 7f7565332640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f75601129f0 con 0x7f7560104640 2024-06-04T05:09:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.229+0000 7f7565332640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7560112cf0 con 0x7f756010d060 2024-06-04T05:09:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.229+0000 7f755effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7560103d20 0x7f7560199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.229+0000 7f755effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7560103d20 0x7f7560199050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38544/0 (socket says 172.21.15.70:38544) 2024-06-04T05:09:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.229+0000 7f755effd640 1 -- 172.21.15.70:0/3181648380 learned_addr learned my addr 172.21.15.70:0/3181648380 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.229+0000 7f755e7fc640 1 --2- 172.21.15.70:0/3181648380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7560104640 0x7f7560199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.229+0000 7f755f7fe640 1 --2- 172.21.15.70:0/3181648380 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f756010d060 0x7f756019d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.230+0000 7f755e7fc640 1 -- 172.21.15.70:0/3181648380 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f756010d060 msgr2=0x7f756019d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.230+0000 7f755e7fc640 1 --2- 172.21.15.70:0/3181648380 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f756010d060 0x7f756019d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.230+0000 7f755e7fc640 1 -- 172.21.15.70:0/3181648380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7560103d20 msgr2=0x7f7560199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.230+0000 7f755e7fc640 1 --2- 172.21.15.70:0/3181648380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7560103d20 0x7f7560199050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.230+0000 7f755e7fc640 1 -- 172.21.15.70:0/3181648380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f756019e040 con 0x7f7560104640 2024-06-04T05:09:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.230+0000 7f755effd640 1 --2- 172.21.15.70:0/3181648380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7560103d20 0x7f7560199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.230+0000 7f755e7fc640 1 --2- 172.21.15.70:0/3181648380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7560104640 0x7f7560199590 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f755400e9f0 tx=0x7f755400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.230+0000 7f7547fff640 1 -- 172.21.15.70:0/3181648380 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f755400ccc0 con 0x7f7560104640 2024-06-04T05:09:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.230+0000 7f7565332640 1 -- 172.21.15.70:0/3181648380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f756019e330 con 0x7f7560104640 2024-06-04T05:09:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.230+0000 7f7547fff640 1 -- 172.21.15.70:0/3181648380 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f75540108c0 con 0x7f7560104640 2024-06-04T05:09:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.230+0000 7f7547fff640 1 -- 172.21.15.70:0/3181648380 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75540187c0 con 0x7f7560104640 2024-06-04T05:09:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.230+0000 7f7565332640 1 -- 172.21.15.70:0/3181648380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75601a5bc0 con 0x7f7560104640 2024-06-04T05:09:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.232+0000 7f7547fff640 1 -- 172.21.15.70:0/3181648380 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7554010430 con 0x7f7560104640 2024-06-04T05:09:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.233+0000 7f756804d740 1 Processor -- start 2024-06-04T05:09:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.233+0000 7f756804d740 1 -- start start 2024-06-04T05:09:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.233+0000 7f756804d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6657820 0x5652f6657c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.233+0000 7f756804d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6658140 0x5652f66624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.233+0000 7f7547fff640 1 --2- 172.21.15.70:0/3181648380 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f75480b2a10 0x7f75480b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.233+0000 7f756804d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6662a00 0x5652f6664df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.233+0000 7f756804d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5652f65c9c30 con 0x5652f6658140 2024-06-04T05:09:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.233+0000 7f756804d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5652f65b97d0 con 0x5652f6662a00 2024-06-04T05:09:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.233+0000 7f756804d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5652f65bd810 con 0x5652f6657820 2024-06-04T05:09:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.233+0000 7f755e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6658140 0x5652f66624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.233+0000 7f755effd640 1 --2- 172.21.15.70:0/3181648380 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f75480b2a10 0x7f75480b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.233+0000 7f755f7fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6657820 0x5652f6657c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.233+0000 7f755f7fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6657820 0x5652f6657c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:45376/0 (socket says 172.21.15.70:45376) 2024-06-04T05:09:17.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.233+0000 7f7547fff640 1 -- 172.21.15.70:0/3181648380 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f75540e28a0 con 0x7f7560104640 2024-06-04T05:09:17.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.233+0000 7f755f7fe640 1 -- 172.21.15.70:0/1999383425 learned_addr learned my addr 172.21.15.70:0/1999383425 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:17.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.234+0000 7f755f7fe640 1 --2- 172.21.15.70:0/1999383425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6662a00 0x5652f6664df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.234+0000 7f755e7fc640 1 -- 172.21.15.70:0/1999383425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6657820 msgr2=0x5652f6657c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:17.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.234+0000 7f755e7fc640 1 --2- 172.21.15.70:0/1999383425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6657820 0x5652f6657c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.234+0000 7f755e7fc640 1 -- 172.21.15.70:0/1999383425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6662a00 msgr2=0x5652f6664df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:17.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.234+0000 7f755e7fc640 1 --2- 172.21.15.70:0/1999383425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6662a00 0x5652f6664df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.234+0000 7f755e7fc640 1 -- 172.21.15.70:0/1999383425 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652f62ce210 con 0x5652f6658140 2024-06-04T05:09:17.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.234+0000 7f755f7fe640 1 --2- 172.21.15.70:0/1999383425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6662a00 0x5652f6664df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:17.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.234+0000 7f755effd640 1 --2- 172.21.15.70:0/3181648380 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f75480b2a10 0x7f75480b4ed0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7550011550 tx=0x7f7550005f90 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:17.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.234+0000 7f755e7fc640 1 --2- 172.21.15.70:0/1999383425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6658140 0x5652f66624c0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f75540dffc0 tx=0x7f755409e480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62872f00a82284f4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:17.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.234+0000 7f75457fa640 1 -- 172.21.15.70:0/1999383425 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75540a9710 con 0x5652f6658140 2024-06-04T05:09:17.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.234+0000 7f75457fa640 1 -- 172.21.15.70:0/1999383425 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f75540a9c00 con 0x5652f6658140 2024-06-04T05:09:17.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.234+0000 7f75457fa640 1 -- 172.21.15.70:0/1999383425 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75540e0240 con 0x5652f6658140 2024-06-04T05:09:17.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.235+0000 7f756804d740 1 -- 172.21.15.70:0/1999383425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6658140 msgr2=0x5652f66624c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:17.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.235+0000 7f756804d740 1 --2- 172.21.15.70:0/1999383425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6658140 0x5652f66624c0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f75540dffc0 tx=0x7f755409e480 comp rx=0 tx=0).stop 2024-06-04T05:09:17.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.235+0000 7f756804d740 1 -- 172.21.15.70:0/1999383425 shutdown_connections 2024-06-04T05:09:17.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.235+0000 7f756804d740 1 --2- 172.21.15.70:0/1999383425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6662a00 0x5652f6664df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.235+0000 7f756804d740 1 --2- 172.21.15.70:0/1999383425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6658140 0x5652f66624c0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.235+0000 7f756804d740 1 --2- 172.21.15.70:0/1999383425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6657820 0x5652f6657c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.235+0000 7f756804d740 1 -- 172.21.15.70:0/1999383425 >> 172.21.15.70:0/1999383425 conn(0x5652f6653d30 msgr2=0x5652f6654110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:17.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.235+0000 7f756804d740 1 -- 172.21.15.70:0/1999383425 shutdown_connections 2024-06-04T05:09:17.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.235+0000 7f756804d740 1 -- 172.21.15.70:0/1999383425 wait complete. 2024-06-04T05:09:17.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.235+0000 7f756804d740 1 -- start start 2024-06-04T05:09:17.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.235+0000 7f756804d740 1 Processor -- start 2024-06-04T05:09:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f756804d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6657820 0x5652f665fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f756804d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6658140 0x5652f665ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f756804d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6662a00 0x5652f66604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f756804d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5652f629f790 con 0x5652f6662a00 2024-06-04T05:09:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f756804d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5652f6575890 con 0x5652f6658140 2024-06-04T05:09:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f756804d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5652f6665630 con 0x5652f6657820 2024-06-04T05:09:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f756804d740 10 client.0 _ll_register_callbacks cb 0x5652f62cda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f755f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6662a00 0x5652f66604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f755effd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6657820 0x5652f665fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f755effd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6657820 0x5652f665fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:45388/0 (socket says 172.21.15.70:45388) 2024-06-04T05:09:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f755f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6662a00 0x5652f66604e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38566/0 (socket says 172.21.15.70:38566) 2024-06-04T05:09:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f755effd640 1 -- 172.21.15.70:0/1132418400 learned_addr learned my addr 172.21.15.70:0/1132418400 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f755e7fc640 1 --2- 172.21.15.70:0/1132418400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6658140 0x5652f665ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f755f7fe640 1 -- 172.21.15.70:0/1132418400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6657820 msgr2=0x5652f665fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f755f7fe640 1 --2- 172.21.15.70:0/1132418400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6657820 0x5652f665fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f755f7fe640 1 -- 172.21.15.70:0/1132418400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6658140 msgr2=0x5652f665ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f755f7fe640 1 --2- 172.21.15.70:0/1132418400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6658140 0x5652f665ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f755f7fe640 1 -- 172.21.15.70:0/1132418400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652f639a620 con 0x5652f6662a00 2024-06-04T05:09:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f755effd640 1 --2- 172.21.15.70:0/1132418400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6657820 0x5652f665fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f755f7fe640 1 --2- 172.21.15.70:0/1132418400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6662a00 0x5652f66604e0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f755802ae70 tx=0x7f7558003e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e18ef8135cf84e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f7542ff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f7542ff5640 1 -- 172.21.15.70:0/1132418400 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7558034040 con 0x5652f6662a00 2024-06-04T05:09:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f7542ff5640 1 -- 172.21.15.70:0/1132418400 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f755802fc70 con 0x5652f6662a00 2024-06-04T05:09:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f756804d740 1 -- 172.21.15.70:0/1132418400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75540a9c00 con 0x5652f6662a00 2024-06-04T05:09:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.236+0000 7f7542ff5640 1 -- 172.21.15.70:0/1132418400 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f755802fe70 con 0x5652f6662a00 2024-06-04T05:09:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.237+0000 7f756804d740 10 client.18693 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.237+0000 7f756804d740 1 -- 172.21.15.70:0/1132418400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5652f66714b0 con 0x5652f6662a00 2024-06-04T05:09:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.237+0000 7f7542ff5640 1 -- 172.21.15.70:0/1132418400 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f75540a9c00 con 0x5652f6662a00 2024-06-04T05:09:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.237+0000 7f7542ff5640 1 -- 172.21.15.70:0/1132418400 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f7558048070 con 0x5652f6662a00 2024-06-04T05:09:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.237+0000 7f756804d740 10 client.18693 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.237+0000 7f756804d740 10 client.18693 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.237+0000 7f756804d740 20 client.18693 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.237+0000 7f756804d740 1 --2- 172.21.15.70:0/1132418400 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5652f66729e0 0x5652f6674dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.237+0000 7f756804d740 4 client.18693 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:09:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.237+0000 7f756804d740 1 -- 172.21.15.70:0/1132418400 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652f65cbb80 con 0x5652f66729e0 2024-06-04T05:09:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.237+0000 7f755effd640 1 --2- 172.21.15.70:0/1132418400 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5652f66729e0 0x5652f6674dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.238+0000 7f755effd640 1 --2- 172.21.15.70:0/1132418400 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5652f66729e0 0x5652f6674dd0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f97ca268e5328ce server_cookie=40af4ecae7ffaefb in_seq=0 out_seq=0 2024-06-04T05:09:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.238+0000 7f7542ff5640 10 client.18693 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.238+0000 7f7542ff5640 1 -- 172.21.15.70:0/1132418400 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5652f65cbb80 con 0x5652f66729e0 2024-06-04T05:09:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.238+0000 7f7542ff5640 10 client.18693 handle_command_reply: tid=0 2024-06-04T05:09:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.238+0000 7f756804d740 1 client.18693 shutdown 2024-06-04T05:09:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.238+0000 7f756804d740 1 -- 172.21.15.70:0/1132418400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6662a00 msgr2=0x5652f66604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.238+0000 7f756804d740 1 --2- 172.21.15.70:0/1132418400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6662a00 0x5652f66604e0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f755802ae70 tx=0x7f7558003e80 comp rx=0 tx=0).stop 2024-06-04T05:09:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.238+0000 7f756804d740 1 -- 172.21.15.70:0/1132418400 shutdown_connections 2024-06-04T05:09:17.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.239+0000 7f756804d740 1 --2- 172.21.15.70:0/1132418400 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5652f66729e0 0x5652f6674dd0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.239+0000 7f756804d740 1 --2- 172.21.15.70:0/1132418400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6658140 0x5652f665ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.239+0000 7f756804d740 1 --2- 172.21.15.70:0/1132418400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6662a00 0x5652f66604e0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.239+0000 7f7542ff5640 0 client.18693 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:17.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.239+0000 7f756804d740 1 --2- 172.21.15.70:0/1132418400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6657820 0x5652f665fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.239+0000 7f756804d740 1 -- 172.21.15.70:0/1132418400 >> 172.21.15.70:0/1132418400 conn(0x5652f6653d30 msgr2=0x5652f6652fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:17.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.239+0000 7f756804d740 1 -- 172.21.15.70:0/1132418400 shutdown_connections 2024-06-04T05:09:17.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.239+0000 7f756804d740 1 -- 172.21.15.70:0/1132418400 wait complete. 2024-06-04T05:09:17.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.239+0000 7f756804d740 20 client.18693 trim_cache size 0 max 16384 2024-06-04T05:09:17.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.255+0000 7f756804d740 1 Processor -- start 2024-06-04T05:09:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.255+0000 7f756804d740 1 -- start start 2024-06-04T05:09:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.255+0000 7f756804d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6657820 0x5652f66598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.255+0000 7f756804d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6658140 0x5652f667a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.255+0000 7f756804d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6662a00 0x5652f667abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.255+0000 7f756804d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f75601ae5a0 con 0x5652f6657820 2024-06-04T05:09:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.256+0000 7f756804d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f75601b0200 con 0x5652f6662a00 2024-06-04T05:09:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.256+0000 7f756804d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f75601aec70 con 0x5652f6658140 2024-06-04T05:09:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.256+0000 7f755effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6657820 0x5652f66598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.256+0000 7f755effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6657820 0x5652f66598e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38574/0 (socket says 172.21.15.70:38574) 2024-06-04T05:09:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.256+0000 7f755effd640 1 -- 172.21.15.70:0/3130928261 learned_addr learned my addr 172.21.15.70:0/3130928261 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.256+0000 7f755e7fc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6658140 0x5652f667a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.256+0000 7f755f7fe640 1 --2- 172.21.15.70:0/3130928261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6662a00 0x5652f667abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.256+0000 7f755effd640 1 -- 172.21.15.70:0/3130928261 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6658140 msgr2=0x5652f667a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.256+0000 7f755effd640 1 --2- 172.21.15.70:0/3130928261 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6658140 0x5652f667a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.256+0000 7f755effd640 1 -- 172.21.15.70:0/3130928261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6662a00 msgr2=0x5652f667abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.256+0000 7f755effd640 1 --2- 172.21.15.70:0/3130928261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6662a00 0x5652f667abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.256+0000 7f755effd640 1 -- 172.21.15.70:0/3130928261 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652f6657f70 con 0x5652f6657820 2024-06-04T05:09:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.256+0000 7f755effd640 1 --2- 172.21.15.70:0/3130928261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6657820 0x5652f66598e0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x5652f6675630 tx=0x7f7550014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b989b479192d720 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.256+0000 7f7544ff9640 1 -- 172.21.15.70:0/3130928261 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7550028460 con 0x5652f6657820 2024-06-04T05:09:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.256+0000 7f7544ff9640 1 -- 172.21.15.70:0/3130928261 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7550028a90 con 0x5652f6657820 2024-06-04T05:09:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.256+0000 7f7544ff9640 1 -- 172.21.15.70:0/3130928261 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f755000eaa0 con 0x5652f6657820 2024-06-04T05:09:17.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.257+0000 7f756804d740 1 -- 172.21.15.70:0/3130928261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6657820 msgr2=0x5652f66598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:17.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.257+0000 7f756804d740 1 --2- 172.21.15.70:0/3130928261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6657820 0x5652f66598e0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x5652f6675630 tx=0x7f7550014000 comp rx=0 tx=0).stop 2024-06-04T05:09:17.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.257+0000 7f756804d740 1 -- 172.21.15.70:0/3130928261 shutdown_connections 2024-06-04T05:09:17.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.257+0000 7f756804d740 1 --2- 172.21.15.70:0/3130928261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6662a00 0x5652f667abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.257+0000 7f756804d740 1 --2- 172.21.15.70:0/3130928261 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6658140 0x5652f667a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.257+0000 7f756804d740 1 --2- 172.21.15.70:0/3130928261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6657820 0x5652f66598e0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.257+0000 7f756804d740 1 -- 172.21.15.70:0/3130928261 >> 172.21.15.70:0/3130928261 conn(0x5652f6653d30 msgr2=0x5652f66591d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.257+0000 7f756804d740 1 -- 172.21.15.70:0/3130928261 shutdown_connections 2024-06-04T05:09:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.257+0000 7f756804d740 1 -- 172.21.15.70:0/3130928261 wait complete. 2024-06-04T05:09:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.257+0000 7f756804d740 1 -- start start 2024-06-04T05:09:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.257+0000 7f756804d740 1 Processor -- start 2024-06-04T05:09:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f756804d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6657820 0x5652f66665a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f756804d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6658140 0x5652f6666ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f756804d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6662a00 0x5652f6667020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f756804d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f75601afad0 con 0x5652f6662a00 2024-06-04T05:09:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f756804d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f75601af3a0 con 0x5652f6657820 2024-06-04T05:09:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f756804d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f75601ac5a0 con 0x5652f6658140 2024-06-04T05:09:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f756804d740 10 client.0 _ll_register_callbacks cb 0x7f7558009970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f755f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6662a00 0x5652f6667020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f755f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6662a00 0x5652f6667020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38578/0 (socket says 172.21.15.70:38578) 2024-06-04T05:09:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f755effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6657820 0x5652f66665a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f755f7fe640 1 -- 172.21.15.70:0/2433534548 learned_addr learned my addr 172.21.15.70:0/2433534548 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f755e7fc640 1 --2- 172.21.15.70:0/2433534548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6658140 0x5652f6666ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f755f7fe640 1 -- 172.21.15.70:0/2433534548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6658140 msgr2=0x5652f6666ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f755f7fe640 1 --2- 172.21.15.70:0/2433534548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6658140 0x5652f6666ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f755f7fe640 1 -- 172.21.15.70:0/2433534548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6657820 msgr2=0x5652f66665a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f755f7fe640 1 --2- 172.21.15.70:0/2433534548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6657820 0x5652f66665a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f755f7fe640 1 -- 172.21.15.70:0/2433534548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652f6675140 con 0x5652f6662a00 2024-06-04T05:09:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f755f7fe640 1 --2- 172.21.15.70:0/2433534548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6662a00 0x5652f6667020 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f7558005e20 tx=0x7f755804a440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07f25fd55351a22 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f7542ff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f7542ff5640 1 -- 172.21.15.70:0/2433534548 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7558034040 con 0x5652f6662a00 2024-06-04T05:09:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f756804d740 1 -- 172.21.15.70:0/2433534548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7550028a90 con 0x5652f6662a00 2024-06-04T05:09:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f7542ff5640 1 -- 172.21.15.70:0/2433534548 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f755804a690 con 0x5652f6662a00 2024-06-04T05:09:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.258+0000 7f7542ff5640 1 -- 172.21.15.70:0/2433534548 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f755802fc80 con 0x5652f6662a00 2024-06-04T05:09:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.259+0000 7f756804d740 10 client.18699 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.259+0000 7f756804d740 1 -- 172.21.15.70:0/2433534548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5652f66dd670 con 0x5652f6662a00 2024-06-04T05:09:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.259+0000 7f7542ff5640 1 -- 172.21.15.70:0/2433534548 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7550028a90 con 0x5652f6662a00 2024-06-04T05:09:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.259+0000 7f7542ff5640 1 -- 172.21.15.70:0/2433534548 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f7558059020 con 0x5652f6662a00 2024-06-04T05:09:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.259+0000 7f756804d740 10 client.18699 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.259+0000 7f756804d740 10 client.18699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.259+0000 7f756804d740 20 client.18699 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.259+0000 7f756804d740 1 --2- 172.21.15.70:0/2433534548 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5652f66dde10 0x5652f66e0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.259+0000 7f756804d740 4 client.18699 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:09:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.259+0000 7f756804d740 1 -- 172.21.15.70:0/2433534548 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5652f65c5360 con 0x5652f66dde10 2024-06-04T05:09:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.259+0000 7f755effd640 1 --2- 172.21.15.70:0/2433534548 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5652f66dde10 0x5652f66e0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.259+0000 7f7542ff5640 10 client.18699 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.259+0000 7f755effd640 1 --2- 172.21.15.70:0/2433534548 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5652f66dde10 0x5652f66e0230 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2f7c899c74ab6c server_cookie=129afe6db0ffe8e3 in_seq=0 out_seq=0 2024-06-04T05:09:17.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.283+0000 7f7542ff5640 1 -- 172.21.15.70:0/2433534548 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5652f65c5360 con 0x5652f66dde10 2024-06-04T05:09:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.284+0000 7f7542ff5640 10 client.18699 handle_command_reply: tid=0 2024-06-04T05:09:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.284+0000 7f756804d740 1 client.18699 shutdown 2024-06-04T05:09:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.284+0000 7f756804d740 1 -- 172.21.15.70:0/2433534548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6662a00 msgr2=0x5652f6667020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.284+0000 7f756804d740 1 --2- 172.21.15.70:0/2433534548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6662a00 0x5652f6667020 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f7558005e20 tx=0x7f755804a440 comp rx=0 tx=0).stop 2024-06-04T05:09:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.284+0000 7f756804d740 1 -- 172.21.15.70:0/2433534548 shutdown_connections 2024-06-04T05:09:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.284+0000 7f756804d740 1 --2- 172.21.15.70:0/2433534548 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5652f66dde10 0x5652f66e0230 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.284+0000 7f756804d740 1 --2- 172.21.15.70:0/2433534548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652f6662a00 0x5652f6667020 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.284+0000 7f756804d740 1 --2- 172.21.15.70:0/2433534548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5652f6658140 0x5652f6666ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.284+0000 7f7542ff5640 0 client.18699 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.284+0000 7f756804d740 1 --2- 172.21.15.70:0/2433534548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652f6657820 0x5652f66665a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.284+0000 7f756804d740 1 -- 172.21.15.70:0/2433534548 >> 172.21.15.70:0/2433534548 conn(0x5652f6653d30 msgr2=0x5652f66602b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.284+0000 7f756804d740 1 -- 172.21.15.70:0/2433534548 shutdown_connections 2024-06-04T05:09:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.284+0000 7f756804d740 1 -- 172.21.15.70:0/2433534548 wait complete. 2024-06-04T05:09:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.284+0000 7f756804d740 20 client.18699 trim_cache size 0 max 16384 2024-06-04T05:09:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:09:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:09:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:09:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.284+0000 7f7565332640 1 -- 172.21.15.70:0/3181648380 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f75480b2a10 msgr2=0x7f75480b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.285+0000 7f7565332640 1 --2- 172.21.15.70:0/3181648380 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f75480b2a10 0x7f75480b4ed0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7550011550 tx=0x7f7550005f90 comp rx=0 tx=0).stop 2024-06-04T05:09:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.285+0000 7f7565332640 1 -- 172.21.15.70:0/3181648380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7560104640 msgr2=0x7f7560199590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.285+0000 7f7565332640 1 --2- 172.21.15.70:0/3181648380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7560104640 0x7f7560199590 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f755400e9f0 tx=0x7f755400eeb0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.285+0000 7f7565332640 1 -- 172.21.15.70:0/3181648380 shutdown_connections 2024-06-04T05:09:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.285+0000 7f7565332640 1 --2- 172.21.15.70:0/3181648380 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f75480b2a10 0x7f75480b4ed0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.285+0000 7f7565332640 1 --2- 172.21.15.70:0/3181648380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7560104640 0x7f7560199590 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.285+0000 7f7565332640 1 --2- 172.21.15.70:0/3181648380 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f756010d060 0x7f756019d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.285+0000 7f7565332640 1 --2- 172.21.15.70:0/3181648380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7560103d20 0x7f7560199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.285+0000 7f7565332640 1 -- 172.21.15.70:0/3181648380 >> 172.21.15.70:0/3181648380 conn(0x7f7560078ad0 msgr2=0x7f75600fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.285+0000 7f7565332640 1 -- 172.21.15.70:0/3181648380 shutdown_connections 2024-06-04T05:09:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:17.285+0000 7f7565332640 1 -- 172.21.15.70:0/3181648380 wait complete. 2024-06-04T05:09:17.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:17 smithi112 ceph-mon[36453]: pgmap v376: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 756 KiB/s wr, 89 op/s 2024-06-04T05:09:17.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:17 smithi098 ceph-mon[35569]: pgmap v376: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 756 KiB/s wr, 89 op/s 2024-06-04T05:09:17.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:17 smithi070 ceph-mon[32631]: pgmap v376: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 756 KiB/s wr, 89 op/s 2024-06-04T05:09:17.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205158 1.76 MB/sec execute 490 sec latency 27.469 ms 2024-06-04T05:09:18.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:18 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:09:18.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:18 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:09:18.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:18 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:09:18.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205578 1.76 MB/sec execute 491 sec latency 16.215 ms 2024-06-04T05:09:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:19 smithi098 ceph-mon[35569]: pgmap v377: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 737 KiB/s wr, 87 op/s 2024-06-04T05:09:19.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:19 smithi070 ceph-mon[32631]: pgmap v377: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 737 KiB/s wr, 87 op/s 2024-06-04T05:09:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:19 smithi112 ceph-mon[36453]: pgmap v377: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 737 KiB/s wr, 87 op/s 2024-06-04T05:09:19.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205802 1.76 MB/sec execute 492 sec latency 735.614 ms 2024-06-04T05:09:20.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205802 1.76 MB/sec execute 493 sec latency 1735.716 ms 2024-06-04T05:09:21.494 INFO:tasks.quiescer.fs.[cephfs]:Total duration (58.8) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-06-04T05:09:21.494 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.4 2024-06-04T05:09:21.494 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:09:21.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:21 smithi098 ceph-mon[35569]: pgmap v378: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 630 KiB/s wr, 74 op/s 2024-06-04T05:09:21.523 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:21 smithi070 ceph-mon[32631]: pgmap v378: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 630 KiB/s wr, 74 op/s 2024-06-04T05:09:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:21 smithi112 ceph-mon[36453]: pgmap v378: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 630 KiB/s wr, 74 op/s 2024-06-04T05:09:21.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205802 1.75 MB/sec execute 494 sec latency 2735.756 ms 2024-06-04T05:09:21.749 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:09:22.739 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205869 1.75 MB/sec execute 495 sec latency 3044.875 ms 2024-06-04T05:09:23.467 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:23 smithi070 ceph-mon[32631]: pgmap v379: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-06-04T05:09:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:23 smithi098 ceph-mon[35569]: pgmap v379: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-06-04T05:09:23.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.566+0000 7fc7279c6700 1 -- 172.21.15.70:0/2980707359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc720102e00 msgr2=0x7fc72010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:23.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.566+0000 7fc7279c6700 1 --2- 172.21.15.70:0/2980707359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc720102e00 0x7fc72010f2f0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fc71800b0a0 tx=0x7fc71800b3b0 comp rx=0 tx=0).stop 2024-06-04T05:09:23.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.566+0000 7fc7279c6700 1 -- 172.21.15.70:0/2980707359 shutdown_connections 2024-06-04T05:09:23.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.566+0000 7fc7279c6700 1 --2- 172.21.15.70:0/2980707359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc720102e00 0x7fc72010f2f0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:23.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.566+0000 7fc7279c6700 1 --2- 172.21.15.70:0/2980707359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc720108520 0x7fc720108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:23.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.566+0000 7fc7279c6700 1 --2- 172.21.15.70:0/2980707359 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc720102440 0x7fc7201028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:23.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.566+0000 7fc7279c6700 1 -- 172.21.15.70:0/2980707359 >> 172.21.15.70:0/2980707359 conn(0x7fc7200793f0 msgr2=0x7fc720079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:23.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.566+0000 7fc7279c6700 1 -- 172.21.15.70:0/2980707359 shutdown_connections 2024-06-04T05:09:23.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.566+0000 7fc7279c6700 1 -- 172.21.15.70:0/2980707359 wait complete. 2024-06-04T05:09:23.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.567+0000 7fc7279c6700 1 Processor -- start 2024-06-04T05:09:23.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.567+0000 7fc7279c6700 1 -- start start 2024-06-04T05:09:23.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.567+0000 7fc7279c6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc720102440 0x7fc72019a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:23.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.567+0000 7fc7279c6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc720102e00 0x7fc72019af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:23.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.567+0000 7fc7279c6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc720108520 0x7fc7201a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:23.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.567+0000 7fc7279c6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc720112550 con 0x7fc720102440 2024-06-04T05:09:23.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.567+0000 7fc7279c6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201123d0 con 0x7fc720108520 2024-06-04T05:09:23.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.567+0000 7fc7279c6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201126d0 con 0x7fc720102e00 2024-06-04T05:09:23.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.567+0000 7fc725762700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc720102440 0x7fc72019a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:23.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.567+0000 7fc725762700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc720102440 0x7fc72019a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38596/0 (socket says 172.21.15.70:38596) 2024-06-04T05:09:23.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.567+0000 7fc725762700 1 -- 172.21.15.70:0/4277678957 learned_addr learned my addr 172.21.15.70:0/4277678957 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:23.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.568+0000 7fc725762700 1 -- 172.21.15.70:0/4277678957 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc720102e00 msgr2=0x7fc72019af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:23.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.568+0000 7fc725762700 1 --2- 172.21.15.70:0/4277678957 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc720102e00 0x7fc72019af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:23.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.568+0000 7fc725762700 1 -- 172.21.15.70:0/4277678957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc720108520 msgr2=0x7fc7201a5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:23.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.568+0000 7fc725762700 1 --2- 172.21.15.70:0/4277678957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc720108520 0x7fc7201a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:23.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.568+0000 7fc725762700 1 -- 172.21.15.70:0/4277678957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7201a5fb0 con 0x7fc720102440 2024-06-04T05:09:23.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.568+0000 7fc725762700 1 --2- 172.21.15.70:0/4277678957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc720102440 0x7fc72019a9f0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00d950 tx=0x7fc71c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:23.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.568+0000 7fc7167fc700 1 -- 172.21.15.70:0/4277678957 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71c016070 con 0x7fc720102440 2024-06-04T05:09:23.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.568+0000 7fc7167fc700 1 -- 172.21.15.70:0/4277678957 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc71c010460 con 0x7fc720102440 2024-06-04T05:09:23.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.568+0000 7fc7279c6700 1 -- 172.21.15.70:0/4277678957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7201a6180 con 0x7fc720102440 2024-06-04T05:09:23.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.568+0000 7fc7167fc700 1 -- 172.21.15.70:0/4277678957 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71c00f620 con 0x7fc720102440 2024-06-04T05:09:23.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.568+0000 7fc7279c6700 1 -- 172.21.15.70:0/4277678957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7201a66c0 con 0x7fc720102440 2024-06-04T05:09:23.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.569+0000 7fc7279c6700 1 -- 172.21.15.70:0/4277678957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc72006a230 con 0x7fc720102440 2024-06-04T05:09:23.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.570+0000 7fc7167fc700 1 -- 172.21.15.70:0/4277678957 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fc71c019070 con 0x7fc720102440 2024-06-04T05:09:23.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.571+0000 7fc7167fc700 1 --2- 172.21.15.70:0/4277678957 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc70c0b29f0 0x7fc70c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:23.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.571+0000 7fc724f61700 1 --2- 172.21.15.70:0/4277678957 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc70c0b29f0 0x7fc70c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:23.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.571+0000 7fc7167fc700 1 -- 172.21.15.70:0/4277678957 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7fc71c0e2260 con 0x7fc720102440 2024-06-04T05:09:23.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.572+0000 7fc724f61700 1 --2- 172.21.15.70:0/4277678957 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc70c0b29f0 0x7fc70c0b4eb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc720079a60 tx=0x7fc710006d20 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:23.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.575+0000 7fc7167fc700 1 -- 172.21.15.70:0/4277678957 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc71c0a9c10 con 0x7fc720102440 2024-06-04T05:09:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:23 smithi112 ceph-mon[36453]: pgmap v379: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-06-04T05:09:23.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205869 1.75 MB/sec execute 496 sec latency 1473.231 ms 2024-06-04T05:09:23.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.830+0000 7fc7279c6700 1 -- 172.21.15.70:0/4277678957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc72002e560 con 0x7fc720102440 2024-06-04T05:09:23.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.830+0000 7fc7167fc700 1 -- 172.21.15.70:0/4277678957 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11938 (secure 0 0 0) 0x7fc71c0a9510 con 0x7fc720102440 2024-06-04T05:09:23.833 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:09:23.833 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:09:17.032462+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24436,24410,24440,24448,24442]},"id":1}]} 2024-06-04T05:09:23.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.832+0000 7fc7279c6700 1 -- 172.21.15.70:0/4277678957 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc70c0b29f0 msgr2=0x7fc70c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:23.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.832+0000 7fc7279c6700 1 --2- 172.21.15.70:0/4277678957 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc70c0b29f0 0x7fc70c0b4eb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc720079a60 tx=0x7fc710006d20 comp rx=0 tx=0).stop 2024-06-04T05:09:23.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.832+0000 7fc7279c6700 1 -- 172.21.15.70:0/4277678957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc720102440 msgr2=0x7fc72019a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:23.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.832+0000 7fc7279c6700 1 --2- 172.21.15.70:0/4277678957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc720102440 0x7fc72019a9f0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00d950 tx=0x7fc71c00dd10 comp rx=0 tx=0).stop 2024-06-04T05:09:23.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.833+0000 7fc7279c6700 1 -- 172.21.15.70:0/4277678957 shutdown_connections 2024-06-04T05:09:23.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.833+0000 7fc7279c6700 1 --2- 172.21.15.70:0/4277678957 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc70c0b29f0 0x7fc70c0b4eb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:23.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.833+0000 7fc7279c6700 1 --2- 172.21.15.70:0/4277678957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc720108520 0x7fc7201a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:23.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.833+0000 7fc7279c6700 1 --2- 172.21.15.70:0/4277678957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc720102440 0x7fc72019a9f0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:23.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.833+0000 7fc7279c6700 1 --2- 172.21.15.70:0/4277678957 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc720102e00 0x7fc72019af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:23.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.833+0000 7fc7279c6700 1 -- 172.21.15.70:0/4277678957 >> 172.21.15.70:0/4277678957 conn(0x7fc7200793f0 msgr2=0x7fc720100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:23.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.833+0000 7fc7279c6700 1 -- 172.21.15.70:0/4277678957 shutdown_connections 2024-06-04T05:09:23.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:23.833+0000 7fc7279c6700 1 -- 172.21.15.70:0/4277678957 wait complete. 2024-06-04T05:09:23.836 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 38 2024-06-04T05:09:24.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:24 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/4277678957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:09:24.557 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:24 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/4277678957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:09:24.586 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 149.4 --await-for 45.0' 2024-06-04T05:09:24.586 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 149.4 --await-for 45.0 2024-06-04T05:09:24.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:24 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/4277678957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:09:24.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205869 1.74 MB/sec execute 497 sec latency 2473.285 ms 2024-06-04T05:09:24.875 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:09:25.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:25 smithi098 ceph-mon[35569]: pgmap v380: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 1016 KiB/s wr, 97 op/s 2024-06-04T05:09:25.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:25 smithi070 ceph-mon[32631]: pgmap v380: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 1016 KiB/s wr, 97 op/s 2024-06-04T05:09:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:25 smithi112 ceph-mon[36453]: pgmap v380: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 1016 KiB/s wr, 97 op/s 2024-06-04T05:09:25.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205869 1.74 MB/sec execute 498 sec latency 3473.335 ms 2024-06-04T05:09:26.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.588+0000 7f79a2f41700 1 -- 172.21.15.70:0/2773659460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f799c102e00 msgr2=0x7f799c10f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:26.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.588+0000 7f79a2f41700 1 --2- 172.21.15.70:0/2773659460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f799c102e00 0x7f799c10f300 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f799400b0a0 tx=0x7f799400b3b0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.588+0000 7f79a2f41700 1 -- 172.21.15.70:0/2773659460 shutdown_connections 2024-06-04T05:09:26.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.588+0000 7f79a2f41700 1 --2- 172.21.15.70:0/2773659460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f799c102e00 0x7f799c10f300 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.588+0000 7f79a2f41700 1 --2- 172.21.15.70:0/2773659460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f799c1084c0 0x7f799c1088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.588+0000 7f79a2f41700 1 --2- 172.21.15.70:0/2773659460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f799c102440 0x7f799c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.588+0000 7f79a2f41700 1 -- 172.21.15.70:0/2773659460 >> 172.21.15.70:0/2773659460 conn(0x7f799c0793e0 msgr2=0x7f799c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:26.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.588+0000 7f79a2f41700 1 -- 172.21.15.70:0/2773659460 shutdown_connections 2024-06-04T05:09:26.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.588+0000 7f79a2f41700 1 -- 172.21.15.70:0/2773659460 wait complete. 2024-06-04T05:09:26.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.589+0000 7f79a2f41700 1 Processor -- start 2024-06-04T05:09:26.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.589+0000 7f79a2f41700 1 -- start start 2024-06-04T05:09:26.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.589+0000 7f79a2f41700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f799c102440 0x7f799c19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.589+0000 7f79a2f41700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f799c102e00 0x7f799c19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.589+0000 7f79a2f41700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f799c1084c0 0x7f799c198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.589+0000 7f79a2f41700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f799c112580 con 0x7f799c1084c0 2024-06-04T05:09:26.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.589+0000 7f79a2f41700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f799c112400 con 0x7f799c102440 2024-06-04T05:09:26.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.589+0000 7f79a2f41700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f799c112700 con 0x7f799c102e00 2024-06-04T05:09:26.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.589+0000 7f79a14de700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f799c1084c0 0x7f799c198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:26.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.589+0000 7f79a14de700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f799c1084c0 0x7f799c198860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45158/0 (socket says 172.21.15.70:45158) 2024-06-04T05:09:26.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.589+0000 7f79a14de700 1 -- 172.21.15.70:0/3366369074 learned_addr learned my addr 172.21.15.70:0/3366369074 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:26.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.590+0000 7f79a14de700 1 -- 172.21.15.70:0/3366369074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f799c102e00 msgr2=0x7f799c19ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:26.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.590+0000 7f79a14de700 1 --2- 172.21.15.70:0/3366369074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f799c102e00 0x7f799c19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.590+0000 7f79a14de700 1 -- 172.21.15.70:0/3366369074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f799c102440 msgr2=0x7f799c19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:26.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.590+0000 7f79a0cdd700 1 --2- 172.21.15.70:0/3366369074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f799c102440 0x7f799c19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:26.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.590+0000 7f79a14de700 1 --2- 172.21.15.70:0/3366369074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f799c102440 0x7f799c19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.590+0000 7f79a14de700 1 -- 172.21.15.70:0/3366369074 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f799c198fc0 con 0x7f799c1084c0 2024-06-04T05:09:26.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.590+0000 7f79a0cdd700 1 --2- 172.21.15.70:0/3366369074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f799c102440 0x7f799c19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:09:26.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.590+0000 7f79a14de700 1 --2- 172.21.15.70:0/3366369074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f799c1084c0 0x7f799c198860 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f79940062a0 tx=0x7f7994004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:26.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.590+0000 7f7999ffb700 1 -- 172.21.15.70:0/3366369074 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7994021070 con 0x7f799c1084c0 2024-06-04T05:09:26.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.590+0000 7f7999ffb700 1 -- 172.21.15.70:0/3366369074 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7994007d00 con 0x7f799c1084c0 2024-06-04T05:09:26.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.590+0000 7f79a2f41700 1 -- 172.21.15.70:0/3366369074 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f799c199250 con 0x7f799c1084c0 2024-06-04T05:09:26.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.590+0000 7f7999ffb700 1 -- 172.21.15.70:0/3366369074 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7994005280 con 0x7f799c1084c0 2024-06-04T05:09:26.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.590+0000 7f79a2f41700 1 -- 172.21.15.70:0/3366369074 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f799c1a5570 con 0x7f799c1084c0 2024-06-04T05:09:26.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.593+0000 7f7999ffb700 1 -- 172.21.15.70:0/3366369074 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7994024070 con 0x7f799c1084c0 2024-06-04T05:09:26.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.594+0000 7f7999ffb700 1 --2- 172.21.15.70:0/3366369074 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f798c0b2b20 0x7f798c0b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.594+0000 7f79a0cdd700 1 --2- 172.21.15.70:0/3366369074 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f798c0b2b20 0x7f798c0b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:26.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.594+0000 7f7999ffb700 1 -- 172.21.15.70:0/3366369074 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f79940e3c20 con 0x7f799c1084c0 2024-06-04T05:09:26.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.594+0000 7f79ad7d6b80 1 Processor -- start 2024-06-04T05:09:26.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.594+0000 7f79ad7d6b80 1 -- start start 2024-06-04T05:09:26.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f79a0cdd700 1 --2- 172.21.15.70:0/3366369074 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f798c0b2b20 0x7f798c0b4fe0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f799c1999a0 tx=0x7f7990008040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:26.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f79ad7d6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5649621449c0 0x564962144da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f79ad7d6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56496213fe90 0x564962140310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f79ad7d6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564962140850 0x56496214c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f79ad7d6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56496209bf00 con 0x564962140850 2024-06-04T05:09:26.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f79ad7d6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56496208baa0 con 0x56496213fe90 2024-06-04T05:09:26.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f79ad7d6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56496208fae0 con 0x5649621449c0 2024-06-04T05:09:26.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f79a14de700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564962140850 0x56496214c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:26.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f799bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56496213fe90 0x564962140310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:26.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f79a14de700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564962140850 0x56496214c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45166/0 (socket says 172.21.15.70:45166) 2024-06-04T05:09:26.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f79a14de700 1 -- 172.21.15.70:0/3096745383 learned_addr learned my addr 172.21.15.70:0/3096745383 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:26.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f79a0cdd700 1 --2- 172.21.15.70:0/3096745383 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5649621449c0 0x564962144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:26.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f79a14de700 1 -- 172.21.15.70:0/3096745383 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5649621449c0 msgr2=0x564962144da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:26.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f79a14de700 1 --2- 172.21.15.70:0/3096745383 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5649621449c0 0x564962144da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f79a14de700 1 -- 172.21.15.70:0/3096745383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56496213fe90 msgr2=0x564962140310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:26.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f79a14de700 1 --2- 172.21.15.70:0/3096745383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56496213fe90 0x564962140310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f79a14de700 1 -- 172.21.15.70:0/3096745383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564962035140 con 0x564962140850 2024-06-04T05:09:26.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.595+0000 7f79a14de700 1 --2- 172.21.15.70:0/3096745383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564962140850 0x56496214c8c0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f7994007f40 tx=0x7f79940aab40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=514fef9368b51524 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:26.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.596+0000 7f7986ffd700 1 -- 172.21.15.70:0/3096745383 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799402b030 con 0x564962140850 2024-06-04T05:09:26.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.596+0000 7f7986ffd700 1 -- 172.21.15.70:0/3096745383 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f79940cd070 con 0x564962140850 2024-06-04T05:09:26.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.596+0000 7f7986ffd700 1 -- 172.21.15.70:0/3096745383 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799401da00 con 0x564962140850 2024-06-04T05:09:26.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.596+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/3096745383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564962140850 msgr2=0x56496214c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:26.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.596+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/3096745383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564962140850 0x56496214c8c0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f7994007f40 tx=0x7f79940aab40 comp rx=0 tx=0).stop 2024-06-04T05:09:26.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.596+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/3096745383 shutdown_connections 2024-06-04T05:09:26.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.596+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/3096745383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564962140850 0x56496214c8c0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.596+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/3096745383 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5649621449c0 0x564962144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.596+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/3096745383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56496213fe90 0x564962140310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.596+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/3096745383 >> 172.21.15.70:0/3096745383 conn(0x5649620693d0 msgr2=0x564962138970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:26.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.596+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/3096745383 shutdown_connections 2024-06-04T05:09:26.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.596+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/3096745383 wait complete. 2024-06-04T05:09:26.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.597+0000 7f79ad7d6b80 1 -- start start 2024-06-04T05:09:26.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.597+0000 7f79ad7d6b80 1 Processor -- start 2024-06-04T05:09:26.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.597+0000 7f79ad7d6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496213fe90 0x56496214c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.597+0000 7f79ad7d6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564962140850 0x564962149910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.597+0000 7f79ad7d6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649621449c0 0x564962149e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.597+0000 7f79ad7d6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564961f92130 con 0x56496213fe90 2024-06-04T05:09:26.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.597+0000 7f79ad7d6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564961ff3f30 con 0x5649621449c0 2024-06-04T05:09:26.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.597+0000 7f79ad7d6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56496209bf00 con 0x564962140850 2024-06-04T05:09:26.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.597+0000 7f79a0cdd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496213fe90 0x56496214c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:26.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.597+0000 7f79ad7d6b80 10 client.0 _ll_register_callbacks cb 0x564962026ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:26.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.597+0000 7f79a0cdd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496213fe90 0x56496214c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45174/0 (socket says 172.21.15.70:45174) 2024-06-04T05:09:26.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.597+0000 7f79a0cdd700 1 -- 172.21.15.70:0/2934506187 learned_addr learned my addr 172.21.15.70:0/2934506187 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:26.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.598+0000 7f79a14de700 1 --2- 172.21.15.70:0/2934506187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649621449c0 0x564962149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:26.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.598+0000 7f799bfff700 1 --2- 172.21.15.70:0/2934506187 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564962140850 0x564962149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:26.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.598+0000 7f79a0cdd700 1 -- 172.21.15.70:0/2934506187 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564962140850 msgr2=0x564962149910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:26.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.598+0000 7f79a0cdd700 1 --2- 172.21.15.70:0/2934506187 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564962140850 0x564962149910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.598+0000 7f79a0cdd700 1 -- 172.21.15.70:0/2934506187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649621449c0 msgr2=0x564962149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:26.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.598+0000 7f79a0cdd700 1 --2- 172.21.15.70:0/2934506187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649621449c0 0x564962149e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.598+0000 7f79a0cdd700 1 -- 172.21.15.70:0/2934506187 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564961fdba60 con 0x56496213fe90 2024-06-04T05:09:26.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.598+0000 7f799bfff700 1 --2- 172.21.15.70:0/2934506187 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564962140850 0x564962149910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:09:26.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.598+0000 7f79a0cdd700 1 --2- 172.21.15.70:0/2934506187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496213fe90 0x56496214c610 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f799000cdc0 tx=0x7f7990004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=917a31f394e46ba7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:26.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.598+0000 7f797bfff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:26.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.598+0000 7f797bfff700 1 -- 172.21.15.70:0/2934506187 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7990023020 con 0x56496213fe90 2024-06-04T05:09:26.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.598+0000 7f797bfff700 1 -- 172.21.15.70:0/2934506187 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f7990013c40 con 0x56496213fe90 2024-06-04T05:09:26.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.598+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/2934506187 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79940cd070 con 0x56496213fe90 2024-06-04T05:09:26.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.598+0000 7f797bfff700 1 -- 172.21.15.70:0/2934506187 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f799002e050 con 0x56496213fe90 2024-06-04T05:09:26.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.598+0000 7f79ad7d6b80 10 client.18717 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:26.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.598+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/2934506187 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5649620458f0 con 0x56496213fe90 2024-06-04T05:09:26.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.599+0000 7f797bfff700 1 -- 172.21.15.70:0/2934506187 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f79940cd070 con 0x56496213fe90 2024-06-04T05:09:26.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.599+0000 7f797bfff700 1 -- 172.21.15.70:0/2934506187 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f79900113d0 con 0x56496213fe90 2024-06-04T05:09:26.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.599+0000 7f79ad7d6b80 10 client.18717 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:26.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.599+0000 7f79ad7d6b80 10 client.18717 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:09:26.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.599+0000 7f79ad7d6b80 20 client.18717 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:26.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.599+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/2934506187 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5649621599d0 0x56496215be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.599+0000 7f79ad7d6b80 4 client.18717 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:09:26.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.599+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/2934506187 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56496209de50 con 0x5649621599d0 2024-06-04T05:09:26.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.599+0000 7f799bfff700 1 --2- 172.21.15.70:0/2934506187 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5649621599d0 0x56496215be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:26.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.600+0000 7f797bfff700 10 client.18717 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:26.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.600+0000 7f799bfff700 1 --2- 172.21.15.70:0/2934506187 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5649621599d0 0x56496215be70 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14cf39de0e83ab2b server_cookie=f38d6700c2991540 in_seq=0 out_seq=0 2024-06-04T05:09:26.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.600+0000 7f797bfff700 1 -- 172.21.15.70:0/2934506187 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56496209de50 con 0x5649621599d0 2024-06-04T05:09:26.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.600+0000 7f797bfff700 10 client.18717 handle_command_reply: tid=0 2024-06-04T05:09:26.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.600+0000 7f79ad7d6b80 1 client.18717 shutdown 2024-06-04T05:09:26.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.600+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/2934506187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496213fe90 msgr2=0x56496214c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:26.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.600+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/2934506187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496213fe90 0x56496214c610 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f799000cdc0 tx=0x7f7990004860 comp rx=0 tx=0).stop 2024-06-04T05:09:26.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.600+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/2934506187 shutdown_connections 2024-06-04T05:09:26.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.600+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/2934506187 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5649621599d0 0x56496215be70 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.600+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/2934506187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649621449c0 0x564962149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.600+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/2934506187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496213fe90 0x56496214c610 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.600+0000 7f797bfff700 0 client.18717 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:26.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.600+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/2934506187 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564962140850 0x564962149910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.600+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/2934506187 >> 172.21.15.70:0/2934506187 conn(0x5649620693d0 msgr2=0x56496213ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:26.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.601+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/2934506187 shutdown_connections 2024-06-04T05:09:26.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.601+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/2934506187 wait complete. 2024-06-04T05:09:26.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.601+0000 7f79ad7d6b80 20 client.18717 trim_cache size 0 max 16384 2024-06-04T05:09:26.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.620+0000 7f79ad7d6b80 1 Processor -- start 2024-06-04T05:09:26.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.620+0000 7f79ad7d6b80 1 -- start start 2024-06-04T05:09:26.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.620+0000 7f79ad7d6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56496213fe90 0x564962132f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79ad7d6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564962140850 0x564962155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79ad7d6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5649621449c0 0x564962161740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79ad7d6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f799c1abaa0 con 0x5649621449c0 2024-06-04T05:09:26.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79ad7d6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f799c1ad700 con 0x564962140850 2024-06-04T05:09:26.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79ad7d6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f799c1ac170 con 0x56496213fe90 2024-06-04T05:09:26.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79a14de700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5649621449c0 0x564962161740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:26.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79a14de700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5649621449c0 0x564962161740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45176/0 (socket says 172.21.15.70:45176) 2024-06-04T05:09:26.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79a14de700 1 -- 172.21.15.70:0/1909181979 learned_addr learned my addr 172.21.15.70:0/1909181979 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:26.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79a0cdd700 1 --2- 172.21.15.70:0/1909181979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56496213fe90 0x564962132f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:26.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f799bfff700 1 --2- 172.21.15.70:0/1909181979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564962140850 0x564962155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:26.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79a14de700 1 -- 172.21.15.70:0/1909181979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56496213fe90 msgr2=0x564962132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:26.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79a14de700 1 --2- 172.21.15.70:0/1909181979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56496213fe90 0x564962132f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79a14de700 1 -- 172.21.15.70:0/1909181979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564962140850 msgr2=0x564962155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:26.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79a14de700 1 --2- 172.21.15.70:0/1909181979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564962140850 0x564962155ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79a14de700 1 -- 172.21.15.70:0/1909181979 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564962147ec0 con 0x5649621449c0 2024-06-04T05:09:26.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79a0cdd700 1 --2- 172.21.15.70:0/1909181979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56496213fe90 0x564962132f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:09:26.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79a14de700 1 --2- 172.21.15.70:0/1909181979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5649621449c0 0x564962161740 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f79940a81d0 tx=0x7f79940060a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4933ae239b0dd272 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:26.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79867fc700 1 -- 172.21.15.70:0/1909181979 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79940ca070 con 0x5649621449c0 2024-06-04T05:09:26.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79867fc700 1 -- 172.21.15.70:0/1909181979 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f79940cd040 con 0x5649621449c0 2024-06-04T05:09:26.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.621+0000 7f79867fc700 1 -- 172.21.15.70:0/1909181979 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799402b040 con 0x5649621449c0 2024-06-04T05:09:26.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.622+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/1909181979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5649621449c0 msgr2=0x564962161740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:26.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.622+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/1909181979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5649621449c0 0x564962161740 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f79940a81d0 tx=0x7f79940060a0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.622+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/1909181979 shutdown_connections 2024-06-04T05:09:26.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.622+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/1909181979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5649621449c0 0x564962161740 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.622+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/1909181979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56496213fe90 0x564962132f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.622+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/1909181979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564962140850 0x564962155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.622+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/1909181979 >> 172.21.15.70:0/1909181979 conn(0x5649620693d0 msgr2=0x564962143a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:26.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.622+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/1909181979 shutdown_connections 2024-06-04T05:09:26.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.622+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/1909181979 wait complete. 2024-06-04T05:09:26.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.622+0000 7f79ad7d6b80 1 -- start start 2024-06-04T05:09:26.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f79ad7d6b80 1 Processor -- start 2024-06-04T05:09:26.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f79ad7d6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56496213fe90 0x56496214e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f79ad7d6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564962140850 0x56496214e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f79ad7d6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5649621449c0 0x5649621c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f79ad7d6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f799c1acfd0 con 0x5649621449c0 2024-06-04T05:09:26.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f79ad7d6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f799c1ac8a0 con 0x564962140850 2024-06-04T05:09:26.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f79ad7d6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f799c1a9a60 con 0x56496213fe90 2024-06-04T05:09:26.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f79ad7d6b80 10 client.0 _ll_register_callbacks cb 0x564962026ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:26.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f79a14de700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5649621449c0 0x5649621c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:26.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f79a14de700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5649621449c0 0x5649621c5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45180/0 (socket says 172.21.15.70:45180) 2024-06-04T05:09:26.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f79a14de700 1 -- 172.21.15.70:0/2371745792 learned_addr learned my addr 172.21.15.70:0/2371745792 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:26.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f79a0cdd700 1 --2- 172.21.15.70:0/2371745792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56496213fe90 0x56496214e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:26.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f799bfff700 1 --2- 172.21.15.70:0/2371745792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564962140850 0x56496214e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:26.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f79a14de700 1 -- 172.21.15.70:0/2371745792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56496213fe90 msgr2=0x56496214e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:26.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f79a14de700 1 --2- 172.21.15.70:0/2371745792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56496213fe90 0x56496214e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f79a14de700 1 -- 172.21.15.70:0/2371745792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564962140850 msgr2=0x56496214e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:26.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f79a14de700 1 --2- 172.21.15.70:0/2371745792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564962140850 0x56496214e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.623+0000 7f79a14de700 1 -- 172.21.15.70:0/2371745792 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649621c6050 con 0x5649621449c0 2024-06-04T05:09:26.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.624+0000 7f79a14de700 1 --2- 172.21.15.70:0/2371745792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5649621449c0 0x5649621c5a10 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f799400b3f0 tx=0x7f79940a7400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719435ebac70d552 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:26.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.624+0000 7f797bfff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:26.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.624+0000 7f797bfff700 1 -- 172.21.15.70:0/2371745792 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79940ca030 con 0x5649621449c0 2024-06-04T05:09:26.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.624+0000 7f797bfff700 1 -- 172.21.15.70:0/2371745792 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f79940a7550 con 0x5649621449c0 2024-06-04T05:09:26.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.624+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/2371745792 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79940cd040 con 0x5649621449c0 2024-06-04T05:09:26.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.624+0000 7f797bfff700 1 -- 172.21.15.70:0/2371745792 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f799402b070 con 0x5649621449c0 2024-06-04T05:09:26.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.624+0000 7f79ad7d6b80 10 client.18723 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:26.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.624+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/2371745792 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5649621c7310 con 0x5649621449c0 2024-06-04T05:09:26.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.624+0000 7f797bfff700 1 -- 172.21.15.70:0/2371745792 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f79940cd040 con 0x5649621449c0 2024-06-04T05:09:26.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.624+0000 7f797bfff700 1 -- 172.21.15.70:0/2371745792 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f799408c070 con 0x5649621449c0 2024-06-04T05:09:26.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.624+0000 7f79ad7d6b80 10 client.18723 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:26.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.624+0000 7f79ad7d6b80 10 client.18723 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:09:26.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.624+0000 7f79ad7d6b80 20 client.18723 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:26.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.624+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/2371745792 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5649621c7540 0x5649621c9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:26.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.624+0000 7f79ad7d6b80 4 client.18723 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.4, "await_for": 45.0}] 2024-06-04T05:09:26.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.624+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/2371745792 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.4, "await_for": 45.0}) -- 0x564962097630 con 0x5649621c7540 2024-06-04T05:09:26.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.625+0000 7f79a0cdd700 1 --2- 172.21.15.70:0/2371745792 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5649621c7540 0x5649621c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:26.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.625+0000 7f79a0cdd700 1 --2- 172.21.15.70:0/2371745792 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5649621c7540 0x5649621c9960 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71b4ab99d20b2565 server_cookie=76742ed2b18d43be in_seq=0 out_seq=0 2024-06-04T05:09:26.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.625+0000 7f797bfff700 10 client.18723 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:26.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.694+0000 7f797bfff700 1 -- 172.21.15.70:0/2371745792 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x564962097630 con 0x5649621c7540 2024-06-04T05:09:26.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.694+0000 7f797bfff700 10 client.18723 handle_command_reply: tid=0 2024-06-04T05:09:26.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.694+0000 7f79ad7d6b80 1 client.18723 shutdown 2024-06-04T05:09:26.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.694+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/2371745792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5649621449c0 msgr2=0x5649621c5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:26.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.694+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/2371745792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5649621449c0 0x5649621c5a10 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f799400b3f0 tx=0x7f79940a7400 comp rx=0 tx=0).stop 2024-06-04T05:09:26.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.694+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/2371745792 shutdown_connections 2024-06-04T05:09:26.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.694+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/2371745792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5649621449c0 0x5649621c5a10 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.694+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/2371745792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56496213fe90 0x56496214e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.694+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/2371745792 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5649621c7540 0x5649621c9960 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.694+0000 7f797bfff700 0 client.18723 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:26.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.694+0000 7f79ad7d6b80 1 --2- 172.21.15.70:0/2371745792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564962140850 0x56496214e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.694+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/2371745792 >> 172.21.15.70:0/2371745792 conn(0x5649620693d0 msgr2=0x56496214b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:26.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.695+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/2371745792 shutdown_connections 2024-06-04T05:09:26.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.695+0000 7f79ad7d6b80 1 -- 172.21.15.70:0/2371745792 wait complete. 2024-06-04T05:09:26.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.695+0000 7f79ad7d6b80 20 client.18723 trim_cache size 0 max 16384 2024-06-04T05:09:26.698 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:09:26.698 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 38, 2024-06-04T05:09:26.698 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:09:26.698 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 38, 2024-06-04T05:09:26.698 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:09:26.698 INFO:teuthology.orchestra.run.smithi070.stdout: "a5e14b41": { 2024-06-04T05:09:26.699 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 38, 2024-06-04T05:09:26.699 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:09:26.699 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:09:26.699 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:09:26.699 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:09:26.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:09:26.699 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:09:26.699 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 149.4, 2024-06-04T05:09:26.699 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:09:26.699 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:09:26.699 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:09:26.699 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:09:26.699 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:09:26.699 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:09:26.699 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:09:26.699 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:09:26.700 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:09:26.700 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:09:26.700 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:09:26.700 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:09:26.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.695+0000 7f79a2f41700 1 -- 172.21.15.70:0/3366369074 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f798c0b2b20 msgr2=0x7f798c0b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:26.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.695+0000 7f79a2f41700 1 --2- 172.21.15.70:0/3366369074 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f798c0b2b20 0x7f798c0b4fe0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f799c1999a0 tx=0x7f7990008040 comp rx=0 tx=0).stop 2024-06-04T05:09:26.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.695+0000 7f79a2f41700 1 -- 172.21.15.70:0/3366369074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f799c1084c0 msgr2=0x7f799c198860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:26.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.695+0000 7f79a2f41700 1 --2- 172.21.15.70:0/3366369074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f799c1084c0 0x7f799c198860 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f79940062a0 tx=0x7f7994004ed0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.696+0000 7f79a2f41700 1 -- 172.21.15.70:0/3366369074 shutdown_connections 2024-06-04T05:09:26.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.696+0000 7f79a2f41700 1 --2- 172.21.15.70:0/3366369074 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f798c0b2b20 0x7f798c0b4fe0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.696+0000 7f79a2f41700 1 --2- 172.21.15.70:0/3366369074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f799c1084c0 0x7f799c198860 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.696+0000 7f79a2f41700 1 --2- 172.21.15.70:0/3366369074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f799c102440 0x7f799c19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.696+0000 7f79a2f41700 1 --2- 172.21.15.70:0/3366369074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f799c102e00 0x7f799c19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:26.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.696+0000 7f79a2f41700 1 -- 172.21.15.70:0/3366369074 >> 172.21.15.70:0/3366369074 conn(0x7f799c0793e0 msgr2=0x7f799c100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:26.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.696+0000 7f79a2f41700 1 -- 172.21.15.70:0/3366369074 shutdown_connections 2024-06-04T05:09:26.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:26.696+0000 7f79a2f41700 1 -- 172.21.15.70:0/3366369074 wait complete. 2024-06-04T05:09:26.738 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.74 MB/sec execute 499 sec latency 4419.609 ms 2024-06-04T05:09:27.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:27 smithi070 ceph-mon[32631]: pgmap v381: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-06-04T05:09:27.428 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'a5e14b41', quiesce took 5.9 seconds. Will release after: 23.5 2024-06-04T05:09:27.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:27 smithi098 ceph-mon[35569]: pgmap v381: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-06-04T05:09:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:27 smithi112 ceph-mon[36453]: pgmap v381: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-06-04T05:09:27.739 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.73 MB/sec execute 500 sec latency 1030.811 ms 2024-06-04T05:09:28.739 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.73 MB/sec execute 501 sec latency 2030.913 ms 2024-06-04T05:09:29.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:29 smithi098 ceph-mon[35569]: pgmap v382: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 563 KiB/s wr, 52 op/s 2024-06-04T05:09:29.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:29 smithi070 ceph-mon[32631]: pgmap v382: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 563 KiB/s wr, 52 op/s 2024-06-04T05:09:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:29 smithi112 ceph-mon[36453]: pgmap v382: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 563 KiB/s wr, 52 op/s 2024-06-04T05:09:29.739 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.73 MB/sec execute 502 sec latency 3030.993 ms 2024-06-04T05:09:30.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.382+0000 7f3602cd1640 1 -- 172.21.15.70:0/752591479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc06bc20 msgr2=0x7f35fc10f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:30.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.382+0000 7f3602cd1640 1 --2- 172.21.15.70:0/752591479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc06bc20 0x7f35fc10f890 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f35f4009ca0 tx=0x7f35f402a8c0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.382+0000 7f3602cd1640 1 -- 172.21.15.70:0/752591479 shutdown_connections 2024-06-04T05:09:30.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.382+0000 7f3602cd1640 1 --2- 172.21.15.70:0/752591479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc06bc20 0x7f35fc10f890 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.382+0000 7f3602cd1640 1 --2- 172.21.15.70:0/752591479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35fc06b280 0x7f35fc06b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.382+0000 7f3602cd1640 1 --2- 172.21.15.70:0/752591479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc108a40 0x7f35fc108e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.382+0000 7f3602cd1640 1 -- 172.21.15.70:0/752591479 >> 172.21.15.70:0/752591479 conn(0x7f35fc077750 msgr2=0x7f35fc077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:30.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.382+0000 7f3602cd1640 1 -- 172.21.15.70:0/752591479 shutdown_connections 2024-06-04T05:09:30.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.382+0000 7f3602cd1640 1 -- 172.21.15.70:0/752591479 wait complete. 2024-06-04T05:09:30.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.382+0000 7f3602cd1640 1 Processor -- start 2024-06-04T05:09:30.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.382+0000 7f3602cd1640 1 -- start start 2024-06-04T05:09:30.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f3602cd1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35fc06b280 0x7f35fc199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f3602cd1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc06bc20 0x7f35fc199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f3602cd1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc108a40 0x7f35fc19d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f3602cd1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc112be0 con 0x7f35fc108a40 2024-06-04T05:09:30.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f3602cd1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc112a60 con 0x7f35fc06bc20 2024-06-04T05:09:30.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f3602cd1640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc112d60 con 0x7f35fc06b280 2024-06-04T05:09:30.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f360126e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc108a40 0x7f35fc19d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f360126e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc108a40 0x7f35fc19d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45188/0 (socket says 172.21.15.70:45188) 2024-06-04T05:09:30.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f360126e640 1 -- 172.21.15.70:0/1512291522 learned_addr learned my addr 172.21.15.70:0/1512291522 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f3600a6d640 1 --2- 172.21.15.70:0/1512291522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35fc06b280 0x7f35fc199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f35fbfff640 1 --2- 172.21.15.70:0/1512291522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc06bc20 0x7f35fc199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f360126e640 1 -- 172.21.15.70:0/1512291522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35fc06b280 msgr2=0x7f35fc199040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f360126e640 1 --2- 172.21.15.70:0/1512291522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35fc06b280 0x7f35fc199040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f360126e640 1 -- 172.21.15.70:0/1512291522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc06bc20 msgr2=0x7f35fc199580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f360126e640 1 --2- 172.21.15.70:0/1512291522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc06bc20 0x7f35fc199580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f360126e640 1 -- 172.21.15.70:0/1512291522 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35fc19e030 con 0x7f35fc108a40 2024-06-04T05:09:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f35fbfff640 1 --2- 172.21.15.70:0/1512291522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc06bc20 0x7f35fc199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:09:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f360126e640 1 --2- 172.21.15.70:0/1512291522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc108a40 0x7f35fc19d900 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f35f4005e30 tx=0x7f35f40093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f35f9ffb640 1 -- 172.21.15.70:0/1512291522 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f4032d10 con 0x7f35fc108a40 2024-06-04T05:09:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f3602cd1640 1 -- 172.21.15.70:0/1512291522 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35fc19e2c0 con 0x7f35fc108a40 2024-06-04T05:09:30.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f35f9ffb640 1 -- 172.21.15.70:0/1512291522 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35f4032eb0 con 0x7f35fc108a40 2024-06-04T05:09:30.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.383+0000 7f35f9ffb640 1 -- 172.21.15.70:0/1512291522 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f4030b20 con 0x7f35fc108a40 2024-06-04T05:09:30.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.384+0000 7f3602cd1640 1 -- 172.21.15.70:0/1512291522 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35fc1a5bb0 con 0x7f35fc108a40 2024-06-04T05:09:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.386+0000 7f35f9ffb640 1 -- 172.21.15.70:0/1512291522 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f35f4030cc0 con 0x7f35fc108a40 2024-06-04T05:09:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.386+0000 7f36059ee740 1 Processor -- start 2024-06-04T05:09:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.386+0000 7f36059ee740 1 -- start start 2024-06-04T05:09:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.386+0000 7f36059ee740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f5883820 0x5634f5883c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.386+0000 7f36059ee740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f5884140 0x5634f588e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.386+0000 7f36059ee740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f588ea00 0x5634f5890df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.386+0000 7f36059ee740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5634f57f5c30 con 0x5634f588ea00 2024-06-04T05:09:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.386+0000 7f36059ee740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5634f57e57d0 con 0x5634f5884140 2024-06-04T05:09:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.386+0000 7f36059ee740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5634f57e9810 con 0x5634f5883820 2024-06-04T05:09:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.387+0000 7f35fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f588ea00 0x5634f5890df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.387+0000 7f35fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f588ea00 0x5634f5890df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45204/0 (socket says 172.21.15.70:45204) 2024-06-04T05:09:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.387+0000 7f35fbfff640 1 -- 172.21.15.70:0/3647719493 learned_addr learned my addr 172.21.15.70:0/3647719493 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.387+0000 7f35fbfff640 1 --2- 172.21.15.70:0/3647719493 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f5883820 0x5634f5883c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.387+0000 7f3600a6d640 1 --2- 172.21.15.70:0/3647719493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f5884140 0x5634f588e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.387+0000 7f35fbfff640 1 -- 172.21.15.70:0/3647719493 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f5883820 msgr2=0x5634f5883c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.387+0000 7f35fbfff640 1 --2- 172.21.15.70:0/3647719493 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f5883820 0x5634f5883c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.387+0000 7f35fbfff640 1 -- 172.21.15.70:0/3647719493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f5884140 msgr2=0x5634f588e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.387+0000 7f35fbfff640 1 --2- 172.21.15.70:0/3647719493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f5884140 0x5634f588e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.387+0000 7f35fbfff640 1 -- 172.21.15.70:0/3647719493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634f54fa210 con 0x5634f588ea00 2024-06-04T05:09:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.387+0000 7f35f9ffb640 1 --2- 172.21.15.70:0/1512291522 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35e40b2a10 0x7f35e40b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.387+0000 7f3600a6d640 1 --2- 172.21.15.70:0/1512291522 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35e40b2a10 0x7f35e40b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.387+0000 7f35fbfff640 1 --2- 172.21.15.70:0/3647719493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f588ea00 0x5634f5890df0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f35f0012820 tx=0x7f35f0009720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b2eb139404af1c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.387+0000 7f35f9ffb640 1 -- 172.21.15.70:0/1512291522 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f35f4103bd0 con 0x7f35fc108a40 2024-06-04T05:09:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.387+0000 7f35e2ffd640 1 -- 172.21.15.70:0/3647719493 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f0012e60 con 0x5634f588ea00 2024-06-04T05:09:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.387+0000 7f35e2ffd640 1 -- 172.21.15.70:0/3647719493 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35f0010c60 con 0x5634f588ea00 2024-06-04T05:09:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.387+0000 7f35e2ffd640 1 -- 172.21.15.70:0/3647719493 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f001eb10 con 0x5634f588ea00 2024-06-04T05:09:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.388+0000 7f3600a6d640 1 --2- 172.21.15.70:0/1512291522 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35e40b2a10 0x7f35e40b4ed0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x5634f5884dc0 tx=0x7f35ec00d490 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.388+0000 7f36059ee740 1 -- 172.21.15.70:0/3647719493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f588ea00 msgr2=0x5634f5890df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.388+0000 7f36059ee740 1 --2- 172.21.15.70:0/3647719493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f588ea00 0x5634f5890df0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f35f0012820 tx=0x7f35f0009720 comp rx=0 tx=0).stop 2024-06-04T05:09:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.388+0000 7f36059ee740 1 -- 172.21.15.70:0/3647719493 shutdown_connections 2024-06-04T05:09:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.388+0000 7f36059ee740 1 --2- 172.21.15.70:0/3647719493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f5884140 0x5634f588e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.388+0000 7f36059ee740 1 --2- 172.21.15.70:0/3647719493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f588ea00 0x5634f5890df0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.388+0000 7f36059ee740 1 --2- 172.21.15.70:0/3647719493 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f5883820 0x5634f5883c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.388+0000 7f36059ee740 1 -- 172.21.15.70:0/3647719493 >> 172.21.15.70:0/3647719493 conn(0x5634f587fd30 msgr2=0x5634f5880110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.388+0000 7f36059ee740 1 -- 172.21.15.70:0/3647719493 shutdown_connections 2024-06-04T05:09:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.388+0000 7f36059ee740 1 -- 172.21.15.70:0/3647719493 wait complete. 2024-06-04T05:09:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.388+0000 7f36059ee740 1 -- start start 2024-06-04T05:09:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f36059ee740 1 Processor -- start 2024-06-04T05:09:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f36059ee740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f5883820 0x5634f588b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f36059ee740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f5884140 0x5634f588bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f36059ee740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f588ea00 0x5634f588c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f36059ee740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5634f54cb790 con 0x5634f5884140 2024-06-04T05:09:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f36059ee740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5634f57a1890 con 0x5634f588ea00 2024-06-04T05:09:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f36059ee740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5634f5891630 con 0x5634f5883820 2024-06-04T05:09:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f36059ee740 10 client.0 _ll_register_callbacks cb 0x5634f54f9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f35fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f5884140 0x5634f588bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f35fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f5884140 0x5634f588bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45218/0 (socket says 172.21.15.70:45218) 2024-06-04T05:09:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f35fbfff640 1 -- 172.21.15.70:0/1065433210 learned_addr learned my addr 172.21.15.70:0/1065433210 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f3600a6d640 1 --2- 172.21.15.70:0/1065433210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f5883820 0x5634f588b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f360126e640 1 --2- 172.21.15.70:0/1065433210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f588ea00 0x5634f588c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f35fbfff640 1 -- 172.21.15.70:0/1065433210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f5883820 msgr2=0x5634f588b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f35fbfff640 1 --2- 172.21.15.70:0/1065433210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f5883820 0x5634f588b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f35fbfff640 1 -- 172.21.15.70:0/1065433210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f588ea00 msgr2=0x5634f588c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f35fbfff640 1 --2- 172.21.15.70:0/1065433210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f588ea00 0x5634f588c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f35fbfff640 1 -- 172.21.15.70:0/1065433210 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634f55c6620 con 0x5634f5884140 2024-06-04T05:09:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.389+0000 7f360126e640 1 --2- 172.21.15.70:0/1065433210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f588ea00 0x5634f588c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:09:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.390+0000 7f35fbfff640 1 --2- 172.21.15.70:0/1065433210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f5884140 0x5634f588bf30 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f35f00127f0 tx=0x7f35f0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1441d62c0ba0c326 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.390+0000 7f35dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.390+0000 7f35dbfff640 1 -- 172.21.15.70:0/1065433210 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f0012e60 con 0x5634f5884140 2024-06-04T05:09:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.390+0000 7f35dbfff640 1 -- 172.21.15.70:0/1065433210 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f35f001eae0 con 0x5634f5884140 2024-06-04T05:09:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.390+0000 7f36059ee740 1 -- 172.21.15.70:0/1065433210 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35f0010c60 con 0x5634f5884140 2024-06-04T05:09:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.390+0000 7f35dbfff640 1 -- 172.21.15.70:0/1065433210 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f35f001e3c0 con 0x5634f5884140 2024-06-04T05:09:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.390+0000 7f36059ee740 10 client.18735 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.390+0000 7f36059ee740 1 -- 172.21.15.70:0/1065433210 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5634f589d420 con 0x5634f5884140 2024-06-04T05:09:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.390+0000 7f35dbfff640 1 -- 172.21.15.70:0/1065433210 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35f0010c60 con 0x5634f5884140 2024-06-04T05:09:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.390+0000 7f35dbfff640 1 -- 172.21.15.70:0/1065433210 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f35f0026030 con 0x5634f5884140 2024-06-04T05:09:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.390+0000 7f36059ee740 10 client.18735 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.390+0000 7f36059ee740 10 client.18735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.390+0000 7f36059ee740 20 client.18735 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.390+0000 7f36059ee740 1 --2- 172.21.15.70:0/1065433210 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5634f589e820 0x5634f58a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.390+0000 7f36059ee740 4 client.18735 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:09:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.390+0000 7f36059ee740 1 -- 172.21.15.70:0/1065433210 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634f57f7b80 con 0x5634f589e820 2024-06-04T05:09:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.391+0000 7f3600a6d640 1 --2- 172.21.15.70:0/1065433210 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5634f589e820 0x5634f58a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.391+0000 7f3600a6d640 1 --2- 172.21.15.70:0/1065433210 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5634f589e820 0x5634f58a0c10 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f36ba07070504ef server_cookie=9c12c22536c7bafa in_seq=0 out_seq=0 2024-06-04T05:09:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.391+0000 7f35dbfff640 10 client.18735 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.392+0000 7f35dbfff640 1 -- 172.21.15.70:0/1065433210 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5634f57f7b80 con 0x5634f589e820 2024-06-04T05:09:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.392+0000 7f35dbfff640 10 client.18735 handle_command_reply: tid=0 2024-06-04T05:09:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.392+0000 7f36059ee740 1 client.18735 shutdown 2024-06-04T05:09:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.392+0000 7f36059ee740 1 -- 172.21.15.70:0/1065433210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f5884140 msgr2=0x5634f588bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.392+0000 7f36059ee740 1 --2- 172.21.15.70:0/1065433210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f5884140 0x5634f588bf30 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f35f00127f0 tx=0x7f35f0005f90 comp rx=0 tx=0).stop 2024-06-04T05:09:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.392+0000 7f36059ee740 1 -- 172.21.15.70:0/1065433210 shutdown_connections 2024-06-04T05:09:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.392+0000 7f36059ee740 1 --2- 172.21.15.70:0/1065433210 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5634f589e820 0x5634f58a0c10 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.392+0000 7f36059ee740 1 --2- 172.21.15.70:0/1065433210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f588ea00 0x5634f588c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.392+0000 7f36059ee740 1 --2- 172.21.15.70:0/1065433210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f5884140 0x5634f588bf30 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.392+0000 7f36059ee740 1 --2- 172.21.15.70:0/1065433210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f5883820 0x5634f588b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.392+0000 7f35dbfff640 0 client.18735 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.392+0000 7f36059ee740 1 -- 172.21.15.70:0/1065433210 >> 172.21.15.70:0/1065433210 conn(0x5634f587fd30 msgr2=0x5634f587efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.392+0000 7f36059ee740 1 -- 172.21.15.70:0/1065433210 shutdown_connections 2024-06-04T05:09:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.392+0000 7f36059ee740 1 -- 172.21.15.70:0/1065433210 wait complete. 2024-06-04T05:09:30.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.392+0000 7f36059ee740 20 client.18735 trim_cache size 0 max 16384 2024-06-04T05:09:30.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.407+0000 7f36059ee740 1 Processor -- start 2024-06-04T05:09:30.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f36059ee740 1 -- start start 2024-06-04T05:09:30.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f36059ee740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f5883820 0x5634f587f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f36059ee740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f5884140 0x5634f58861b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f36059ee740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f588ea00 0x5634f5896c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f36059ee740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc1ae5b0 con 0x5634f5884140 2024-06-04T05:09:30.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f36059ee740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc1b0210 con 0x5634f5883820 2024-06-04T05:09:30.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f36059ee740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc1aec80 con 0x5634f588ea00 2024-06-04T05:09:30.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f35fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f5884140 0x5634f58861b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f35fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f5884140 0x5634f58861b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45228/0 (socket says 172.21.15.70:45228) 2024-06-04T05:09:30.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f35fbfff640 1 -- 172.21.15.70:0/1304982772 learned_addr learned my addr 172.21.15.70:0/1304982772 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:30.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f3600a6d640 1 --2- 172.21.15.70:0/1304982772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f5883820 0x5634f587f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f360126e640 1 --2- 172.21.15.70:0/1304982772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f588ea00 0x5634f5896c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f35fbfff640 1 -- 172.21.15.70:0/1304982772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f588ea00 msgr2=0x5634f5896c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:30.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f35fbfff640 1 --2- 172.21.15.70:0/1304982772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f588ea00 0x5634f5896c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f35fbfff640 1 -- 172.21.15.70:0/1304982772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f5883820 msgr2=0x5634f587f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:30.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f35fbfff640 1 --2- 172.21.15.70:0/1304982772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f5883820 0x5634f587f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f35fbfff640 1 -- 172.21.15.70:0/1304982772 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634f588a9a0 con 0x5634f5884140 2024-06-04T05:09:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f3600a6d640 1 --2- 172.21.15.70:0/1304982772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f5883820 0x5634f587f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:09:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f35fbfff640 1 --2- 172.21.15.70:0/1304982772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f5884140 0x5634f58861b0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f35f00049b0 tx=0x7f35f0004ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40690be8df76b35f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f35e27fc640 1 -- 172.21.15.70:0/1304982772 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f0033030 con 0x5634f5884140 2024-06-04T05:09:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f35e27fc640 1 -- 172.21.15.70:0/1304982772 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35f001e700 con 0x5634f5884140 2024-06-04T05:09:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.408+0000 7f35e27fc640 1 -- 172.21.15.70:0/1304982772 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f0012e60 con 0x5634f5884140 2024-06-04T05:09:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.409+0000 7f36059ee740 1 -- 172.21.15.70:0/1304982772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f5884140 msgr2=0x5634f58861b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.409+0000 7f36059ee740 1 --2- 172.21.15.70:0/1304982772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f5884140 0x5634f58861b0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f35f00049b0 tx=0x7f35f0004ae0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.409+0000 7f36059ee740 1 -- 172.21.15.70:0/1304982772 shutdown_connections 2024-06-04T05:09:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.409+0000 7f36059ee740 1 --2- 172.21.15.70:0/1304982772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f588ea00 0x5634f5896c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.409+0000 7f36059ee740 1 --2- 172.21.15.70:0/1304982772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f5884140 0x5634f58861b0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.409+0000 7f36059ee740 1 --2- 172.21.15.70:0/1304982772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f5883820 0x5634f587f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.409+0000 7f36059ee740 1 -- 172.21.15.70:0/1304982772 >> 172.21.15.70:0/1304982772 conn(0x5634f587fd30 msgr2=0x5634f588c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.409+0000 7f36059ee740 1 -- 172.21.15.70:0/1304982772 shutdown_connections 2024-06-04T05:09:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.409+0000 7f36059ee740 1 -- 172.21.15.70:0/1304982772 wait complete. 2024-06-04T05:09:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.409+0000 7f36059ee740 1 -- start start 2024-06-04T05:09:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.409+0000 7f36059ee740 1 Processor -- start 2024-06-04T05:09:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f36059ee740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f5883820 0x5634f5887c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f36059ee740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f5884140 0x5634f58881a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f36059ee740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f588ea00 0x5634f5888820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f36059ee740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc1afae0 con 0x5634f588ea00 2024-06-04T05:09:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f36059ee740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc1af3b0 con 0x5634f5883820 2024-06-04T05:09:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f36059ee740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc1ac5b0 con 0x5634f5884140 2024-06-04T05:09:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f36059ee740 10 client.0 _ll_register_callbacks cb 0x7f35ec00d360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f360126e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f588ea00 0x5634f5888820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f360126e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f588ea00 0x5634f5888820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45238/0 (socket says 172.21.15.70:45238) 2024-06-04T05:09:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f360126e640 1 -- 172.21.15.70:0/2877573048 learned_addr learned my addr 172.21.15.70:0/2877573048 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f35fbfff640 1 --2- 172.21.15.70:0/2877573048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f5884140 0x5634f58881a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f3600a6d640 1 --2- 172.21.15.70:0/2877573048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f5883820 0x5634f5887c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f360126e640 1 -- 172.21.15.70:0/2877573048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f5884140 msgr2=0x5634f58881a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f360126e640 1 --2- 172.21.15.70:0/2877573048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f5884140 0x5634f58881a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f360126e640 1 -- 172.21.15.70:0/2877573048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f5883820 msgr2=0x5634f5887c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f360126e640 1 --2- 172.21.15.70:0/2877573048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f5883820 0x5634f5887c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f360126e640 1 -- 172.21.15.70:0/2877573048 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634f5889020 con 0x5634f588ea00 2024-06-04T05:09:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f35fbfff640 1 --2- 172.21.15.70:0/2877573048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f5884140 0x5634f58881a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:09:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f360126e640 1 --2- 172.21.15.70:0/2877573048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f588ea00 0x5634f5888820 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f35f40cac90 tx=0x7f35f40c74a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb00df226072eaaa server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.410+0000 7f35dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.411+0000 7f35dbfff640 1 -- 172.21.15.70:0/2877573048 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f40e5070 con 0x5634f588ea00 2024-06-04T05:09:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.411+0000 7f35dbfff640 1 -- 172.21.15.70:0/2877573048 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f35f40cb400 con 0x5634f588ea00 2024-06-04T05:09:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.411+0000 7f36059ee740 1 -- 172.21.15.70:0/2877573048 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35f001e700 con 0x5634f588ea00 2024-06-04T05:09:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.411+0000 7f35dbfff640 1 -- 172.21.15.70:0/2877573048 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f35f40cb690 con 0x5634f588ea00 2024-06-04T05:09:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.411+0000 7f36059ee740 10 client.18741 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.411+0000 7f36059ee740 1 -- 172.21.15.70:0/2877573048 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5634f590d9c0 con 0x5634f588ea00 2024-06-04T05:09:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.411+0000 7f35dbfff640 1 -- 172.21.15.70:0/2877573048 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35f001e700 con 0x5634f588ea00 2024-06-04T05:09:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.411+0000 7f35dbfff640 1 -- 172.21.15.70:0/2877573048 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f35f40ca1f0 con 0x5634f588ea00 2024-06-04T05:09:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.411+0000 7f36059ee740 10 client.18741 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.411+0000 7f36059ee740 10 client.18741 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.411+0000 7f36059ee740 20 client.18741 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.411+0000 7f36059ee740 1 --2- 172.21.15.70:0/2877573048 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5634f590e140 0x5634f5910560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.411+0000 7f36059ee740 4 client.18741 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:09:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.411+0000 7f36059ee740 1 -- 172.21.15.70:0/2877573048 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5634f57f1360 con 0x5634f590e140 2024-06-04T05:09:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.411+0000 7f3600a6d640 1 --2- 172.21.15.70:0/2877573048 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5634f590e140 0x5634f5910560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.412+0000 7f3600a6d640 1 --2- 172.21.15.70:0/2877573048 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5634f590e140 0x5634f5910560 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf1e405f59b2df0c server_cookie=879727ef8b23705b in_seq=0 out_seq=0 2024-06-04T05:09:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.412+0000 7f35dbfff640 10 client.18741 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:30.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.434+0000 7f35dbfff640 1 -- 172.21.15.70:0/2877573048 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5634f57f1360 con 0x5634f590e140 2024-06-04T05:09:30.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.434+0000 7f35dbfff640 10 client.18741 handle_command_reply: tid=0 2024-06-04T05:09:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.434+0000 7f36059ee740 1 client.18741 shutdown 2024-06-04T05:09:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.434+0000 7f36059ee740 1 -- 172.21.15.70:0/2877573048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f588ea00 msgr2=0x5634f5888820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.434+0000 7f36059ee740 1 --2- 172.21.15.70:0/2877573048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f588ea00 0x5634f5888820 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f35f40cac90 tx=0x7f35f40c74a0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.434+0000 7f36059ee740 1 -- 172.21.15.70:0/2877573048 shutdown_connections 2024-06-04T05:09:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.434+0000 7f36059ee740 1 --2- 172.21.15.70:0/2877573048 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5634f590e140 0x5634f5910560 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.434+0000 7f36059ee740 1 --2- 172.21.15.70:0/2877573048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f588ea00 0x5634f5888820 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.434+0000 7f36059ee740 1 --2- 172.21.15.70:0/2877573048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5634f5884140 0x5634f58881a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.434+0000 7f35dbfff640 0 client.18741 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.434+0000 7f36059ee740 1 --2- 172.21.15.70:0/2877573048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5634f5883820 0x5634f5887c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.434+0000 7f36059ee740 1 -- 172.21.15.70:0/2877573048 >> 172.21.15.70:0/2877573048 conn(0x5634f587fd30 msgr2=0x5634f5899680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.434+0000 7f36059ee740 1 -- 172.21.15.70:0/2877573048 shutdown_connections 2024-06-04T05:09:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.434+0000 7f36059ee740 1 -- 172.21.15.70:0/2877573048 wait complete. 2024-06-04T05:09:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.434+0000 7f36059ee740 20 client.18741 trim_cache size 0 max 16384 2024-06-04T05:09:30.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:09:30.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:09:30.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:09:30.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.435+0000 7f3602cd1640 1 -- 172.21.15.70:0/1512291522 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35e40b2a10 msgr2=0x7f35e40b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:30.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.435+0000 7f3602cd1640 1 --2- 172.21.15.70:0/1512291522 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35e40b2a10 0x7f35e40b4ed0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x5634f5884dc0 tx=0x7f35ec00d490 comp rx=0 tx=0).stop 2024-06-04T05:09:30.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.435+0000 7f3602cd1640 1 -- 172.21.15.70:0/1512291522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc108a40 msgr2=0x7f35fc19d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:30.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.435+0000 7f3602cd1640 1 --2- 172.21.15.70:0/1512291522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc108a40 0x7f35fc19d900 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f35f4005e30 tx=0x7f35f40093d0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.435+0000 7f3602cd1640 1 -- 172.21.15.70:0/1512291522 shutdown_connections 2024-06-04T05:09:30.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.435+0000 7f3602cd1640 1 --2- 172.21.15.70:0/1512291522 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35e40b2a10 0x7f35e40b4ed0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.435+0000 7f3602cd1640 1 --2- 172.21.15.70:0/1512291522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc06bc20 0x7f35fc199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.435+0000 7f3602cd1640 1 --2- 172.21.15.70:0/1512291522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc108a40 0x7f35fc19d900 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.435+0000 7f3602cd1640 1 --2- 172.21.15.70:0/1512291522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35fc06b280 0x7f35fc199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:30.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.435+0000 7f3602cd1640 1 -- 172.21.15.70:0/1512291522 >> 172.21.15.70:0/1512291522 conn(0x7f35fc077750 msgr2=0x7f35fc1008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:30.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.435+0000 7f3602cd1640 1 -- 172.21.15.70:0/1512291522 shutdown_connections 2024-06-04T05:09:30.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:30.435+0000 7f3602cd1640 1 -- 172.21.15.70:0/1512291522 wait complete. 2024-06-04T05:09:30.739 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.72 MB/sec execute 503 sec latency 4031.068 ms 2024-06-04T05:09:31.488 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:31 smithi112 ceph-mon[36453]: pgmap v383: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 563 KiB/s wr, 52 op/s 2024-06-04T05:09:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:31 smithi098 ceph-mon[35569]: pgmap v383: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 563 KiB/s wr, 52 op/s 2024-06-04T05:09:31.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:31 smithi070 ceph-mon[32631]: pgmap v383: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 563 KiB/s wr, 52 op/s 2024-06-04T05:09:31.739 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.72 MB/sec execute 504 sec latency 5031.168 ms 2024-06-04T05:09:32.739 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.72 MB/sec execute 505 sec latency 6031.268 ms 2024-06-04T05:09:33.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:33 smithi098 ceph-mon[35569]: pgmap v384: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 596 KiB/s wr, 64 op/s 2024-06-04T05:09:33.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:33 smithi070 ceph-mon[32631]: pgmap v384: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 596 KiB/s wr, 64 op/s 2024-06-04T05:09:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:33 smithi112 ceph-mon[36453]: pgmap v384: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 596 KiB/s wr, 64 op/s 2024-06-04T05:09:33.739 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.71 MB/sec execute 506 sec latency 7031.358 ms 2024-06-04T05:09:34.739 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.71 MB/sec execute 507 sec latency 8031.447 ms 2024-06-04T05:09:35.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:35 smithi098 ceph-mon[35569]: pgmap v385: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 74 KiB/s wr, 23 op/s 2024-06-04T05:09:35.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:35 smithi070 ceph-mon[32631]: pgmap v385: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 74 KiB/s wr, 23 op/s 2024-06-04T05:09:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:35 smithi112 ceph-mon[36453]: pgmap v385: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 74 KiB/s wr, 23 op/s 2024-06-04T05:09:35.739 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.71 MB/sec execute 508 sec latency 9031.550 ms 2024-06-04T05:09:36.739 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.70 MB/sec execute 509 sec latency 10031.643 ms 2024-06-04T05:09:37.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:37 smithi098 ceph-mon[35569]: pgmap v386: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 83 KiB/s wr, 28 op/s 2024-06-04T05:09:37.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:37 smithi070 ceph-mon[32631]: pgmap v386: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 83 KiB/s wr, 28 op/s 2024-06-04T05:09:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:37 smithi112 ceph-mon[36453]: pgmap v386: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 83 KiB/s wr, 28 op/s 2024-06-04T05:09:37.739 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.70 MB/sec execute 510 sec latency 11031.711 ms 2024-06-04T05:09:38.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:38 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:09:38.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:38 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:09:38.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:38 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:09:38.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:38 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:09:38.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:38 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:09:38.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:38 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:09:38.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:38 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:09:38.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:38 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:09:38.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:38 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:09:38.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:38 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:09:38.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:38 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:09:38.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:38 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:09:38.739 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.70 MB/sec execute 511 sec latency 12031.790 ms 2024-06-04T05:09:39.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:09:39.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:39 smithi098 ceph-mon[35569]: pgmap v387: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 42 KiB/s wr, 16 op/s 2024-06-04T05:09:39.510 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:39 smithi070 ceph-mon[32631]: pgmap v387: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 42 KiB/s wr, 16 op/s 2024-06-04T05:09:39.620 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:09:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:39 smithi112 ceph-mon[36453]: pgmap v387: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 42 KiB/s wr, 16 op/s 2024-06-04T05:09:39.739 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.69 MB/sec execute 512 sec latency 13031.836 ms 2024-06-04T05:09:40.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.69 MB/sec execute 513 sec latency 14031.871 ms 2024-06-04T05:09:41.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:41 smithi098 ceph-mon[35569]: pgmap v388: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 42 KiB/s wr, 16 op/s 2024-06-04T05:09:41.570 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:41 smithi070 ceph-mon[32631]: pgmap v388: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 42 KiB/s wr, 16 op/s 2024-06-04T05:09:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:41 smithi112 ceph-mon[36453]: pgmap v388: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 42 KiB/s wr, 16 op/s 2024-06-04T05:09:41.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.69 MB/sec execute 514 sec latency 15031.918 ms 2024-06-04T05:09:41.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.958+0000 7fcc1299a700 1 -- 172.21.15.70:0/4169216788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc0c102470 msgr2=0x7fcc0c1028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:41.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.958+0000 7fcc1299a700 1 --2- 172.21.15.70:0/4169216788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc0c102470 0x7fcc0c1028f0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4009990 tx=0x7fcbf4009ca0 comp rx=0 tx=0).stop 2024-06-04T05:09:41.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.958+0000 7fcc1299a700 1 -- 172.21.15.70:0/4169216788 shutdown_connections 2024-06-04T05:09:41.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.958+0000 7fcc1299a700 1 --2- 172.21.15.70:0/4169216788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc0c102e30 0x7fcc0c10f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:41.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.958+0000 7fcc1299a700 1 --2- 172.21.15.70:0/4169216788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc0c1042f0 0x7fcc0c1046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:41.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.958+0000 7fcc1299a700 1 --2- 172.21.15.70:0/4169216788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc0c102470 0x7fcc0c1028f0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:41.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.958+0000 7fcc1299a700 1 -- 172.21.15.70:0/4169216788 >> 172.21.15.70:0/4169216788 conn(0x7fcc0c0784b0 msgr2=0x7fcc0c0788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:41.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.958+0000 7fcc1299a700 1 -- 172.21.15.70:0/4169216788 shutdown_connections 2024-06-04T05:09:41.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.958+0000 7fcc1299a700 1 -- 172.21.15.70:0/4169216788 wait complete. 2024-06-04T05:09:41.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.959+0000 7fcc1299a700 1 Processor -- start 2024-06-04T05:09:41.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.959+0000 7fcc1299a700 1 -- start start 2024-06-04T05:09:41.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.960+0000 7fcc1299a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc0c102470 0x7fcc0c19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:41.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.960+0000 7fcc1299a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc0c102e30 0x7fcc0c19af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:41.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.960+0000 7fcc1299a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc0c1042f0 0x7fcc0c1a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:41.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.960+0000 7fcc1299a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c1125f0 con 0x7fcc0c102470 2024-06-04T05:09:41.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.960+0000 7fcc1299a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c112470 con 0x7fcc0c1042f0 2024-06-04T05:09:41.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.960+0000 7fcc1299a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c112770 con 0x7fcc0c102e30 2024-06-04T05:09:41.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.960+0000 7fcc0bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc0c102470 0x7fcc0c19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:42.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.960+0000 7fcc0bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc0c102470 0x7fcc0c19a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38464/0 (socket says 172.21.15.70:38464) 2024-06-04T05:09:42.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.960+0000 7fcc0bfff700 1 -- 172.21.15.70:0/184732985 learned_addr learned my addr 172.21.15.70:0/184732985 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:42.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.960+0000 7fcc0b7fe700 1 --2- 172.21.15.70:0/184732985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc0c102e30 0x7fcc0c19af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:42.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.960+0000 7fcc10f37700 1 --2- 172.21.15.70:0/184732985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc0c1042f0 0x7fcc0c1a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:42.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.960+0000 7fcc0bfff700 1 -- 172.21.15.70:0/184732985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc0c102e30 msgr2=0x7fcc0c19af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:42.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.960+0000 7fcc0bfff700 1 --2- 172.21.15.70:0/184732985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc0c102e30 0x7fcc0c19af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.960+0000 7fcc0bfff700 1 -- 172.21.15.70:0/184732985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc0c1042f0 msgr2=0x7fcc0c1a5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:42.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.960+0000 7fcc0bfff700 1 --2- 172.21.15.70:0/184732985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc0c1042f0 0x7fcc0c1a5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.960+0000 7fcc0bfff700 1 -- 172.21.15.70:0/184732985 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0c1a5fb0 con 0x7fcc0c102470 2024-06-04T05:09:42.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.960+0000 7fcc0bfff700 1 --2- 172.21.15.70:0/184732985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc0c102470 0x7fcc0c19a9f0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc00c9b0 tx=0x7fcbfc00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:42.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.961+0000 7fcc097fa700 1 -- 172.21.15.70:0/184732985 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc007b00 con 0x7fcc0c102470 2024-06-04T05:09:42.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.961+0000 7fcc1299a700 1 -- 172.21.15.70:0/184732985 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc0c1a61e0 con 0x7fcc0c102470 2024-06-04T05:09:42.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.961+0000 7fcc097fa700 1 -- 172.21.15.70:0/184732985 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbfc007ca0 con 0x7fcc0c102470 2024-06-04T05:09:42.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.961+0000 7fcc097fa700 1 -- 172.21.15.70:0/184732985 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc018660 con 0x7fcc0c102470 2024-06-04T05:09:42.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.961+0000 7fcc1299a700 1 -- 172.21.15.70:0/184732985 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc0c1a6740 con 0x7fcc0c102470 2024-06-04T05:09:42.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.963+0000 7fcc097fa700 1 -- 172.21.15.70:0/184732985 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fcbfc00f450 con 0x7fcc0c102470 2024-06-04T05:09:42.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.964+0000 7fcc097fa700 1 --2- 172.21.15.70:0/184732985 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcbf80b29f0 0x7fcbf80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:42.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.964+0000 7fcc0b7fe700 1 --2- 172.21.15.70:0/184732985 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcbf80b29f0 0x7fcbf80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:42.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.964+0000 7fcc097fa700 1 -- 172.21.15.70:0/184732985 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7fcbfc0e2630 con 0x7fcc0c102470 2024-06-04T05:09:42.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.964+0000 7fcc0b7fe700 1 --2- 172.21.15.70:0/184732985 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcbf80b29f0 0x7fcbf80b4eb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400b5c0 tx=0x7fcbf4005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:42.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.964+0000 7fcc1d22fb80 1 Processor -- start 2024-06-04T05:09:42.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.964+0000 7fcc1d22fb80 1 -- start start 2024-06-04T05:09:42.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.964+0000 7fcc1d22fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3af1320 0x5636a3af1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:42.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.964+0000 7fcc1d22fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aeb8a0 0x5636a3aebd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:42.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.964+0000 7fcc1d22fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3aec260 0x5636a3af8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:42.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.965+0000 7fcc1d22fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5636a3a49d70 con 0x5636a3aeb8a0 2024-06-04T05:09:42.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.965+0000 7fcc1d22fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5636a3a39910 con 0x5636a3af1320 2024-06-04T05:09:42.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.965+0000 7fcc1d22fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5636a3a3d950 con 0x5636a3aec260 2024-06-04T05:09:42.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.965+0000 7fcc0b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aeb8a0 0x5636a3aebd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:42.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.965+0000 7fcc0b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aeb8a0 0x5636a3aebd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38474/0 (socket says 172.21.15.70:38474) 2024-06-04T05:09:42.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.965+0000 7fcc0b7fe700 1 -- 172.21.15.70:0/2065334902 learned_addr learned my addr 172.21.15.70:0/2065334902 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:42.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.965+0000 7fcc10f37700 1 --2- 172.21.15.70:0/2065334902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3aec260 0x5636a3af8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:42.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.965+0000 7fcc0bfff700 1 --2- 172.21.15.70:0/2065334902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3af1320 0x5636a3af1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:42.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.965+0000 7fcc0b7fe700 1 -- 172.21.15.70:0/2065334902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3aec260 msgr2=0x5636a3af8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:42.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.965+0000 7fcc0b7fe700 1 --2- 172.21.15.70:0/2065334902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3aec260 0x5636a3af8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.965+0000 7fcc0b7fe700 1 -- 172.21.15.70:0/2065334902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3af1320 msgr2=0x5636a3af1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:42.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.965+0000 7fcc0b7fe700 1 --2- 172.21.15.70:0/2065334902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3af1320 0x5636a3af1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.965+0000 7fcc0b7fe700 1 -- 172.21.15.70:0/2065334902 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636a39e2fb0 con 0x5636a3aeb8a0 2024-06-04T05:09:42.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.965+0000 7fcc0b7fe700 1 --2- 172.21.15.70:0/2065334902 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aeb8a0 0x5636a3aebd20 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4015000 tx=0x7fcbf401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89da188073d1a77 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:42.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.965+0000 7fcc027fc700 1 -- 172.21.15.70:0/2065334902 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4034070 con 0x5636a3aeb8a0 2024-06-04T05:09:42.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.965+0000 7fcc027fc700 1 -- 172.21.15.70:0/2065334902 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbf402f900 con 0x5636a3aeb8a0 2024-06-04T05:09:42.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.965+0000 7fcc027fc700 1 -- 172.21.15.70:0/2065334902 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf400f900 con 0x5636a3aeb8a0 2024-06-04T05:09:42.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.966+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/2065334902 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aeb8a0 msgr2=0x5636a3aebd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:42.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.966+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/2065334902 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aeb8a0 0x5636a3aebd20 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4015000 tx=0x7fcbf401f000 comp rx=0 tx=0).stop 2024-06-04T05:09:42.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.966+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/2065334902 shutdown_connections 2024-06-04T05:09:42.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.966+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/2065334902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3aec260 0x5636a3af8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.966+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/2065334902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3af1320 0x5636a3af1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.966+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/2065334902 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aeb8a0 0x5636a3aebd20 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.966+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/2065334902 >> 172.21.15.70:0/2065334902 conn(0x5636a3a17240 msgr2=0x5636a3ae63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:42.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.966+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/2065334902 shutdown_connections 2024-06-04T05:09:42.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.966+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/2065334902 wait complete. 2024-06-04T05:09:42.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.966+0000 7fcc1d22fb80 1 -- start start 2024-06-04T05:09:42.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc1d22fb80 1 Processor -- start 2024-06-04T05:09:42.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc1d22fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3aeb8a0 0x5636a3af7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:42.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc1d22fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aec260 0x5636a3af4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:42.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc1d22fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3af1320 0x5636a3af54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:42.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc1d22fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5636a393ffa0 con 0x5636a3aec260 2024-06-04T05:09:42.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc1d22fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5636a39a1da0 con 0x5636a3aeb8a0 2024-06-04T05:09:42.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc1d22fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5636a3af8680 con 0x5636a3af1320 2024-06-04T05:09:42.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc1d22fb80 10 client.0 _ll_register_callbacks cb 0x5636a39d4950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:42.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc0b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aec260 0x5636a3af4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:42.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc0b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aec260 0x5636a3af4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38482/0 (socket says 172.21.15.70:38482) 2024-06-04T05:09:42.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc0b7fe700 1 -- 172.21.15.70:0/1144019184 learned_addr learned my addr 172.21.15.70:0/1144019184 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:42.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc0bfff700 1 --2- 172.21.15.70:0/1144019184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3aeb8a0 0x5636a3af7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:42.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc10f37700 1 --2- 172.21.15.70:0/1144019184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3af1320 0x5636a3af54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:42.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc0b7fe700 1 -- 172.21.15.70:0/1144019184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3af1320 msgr2=0x5636a3af54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:42.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc0b7fe700 1 --2- 172.21.15.70:0/1144019184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3af1320 0x5636a3af54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc0b7fe700 1 -- 172.21.15.70:0/1144019184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3aeb8a0 msgr2=0x5636a3af7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:42.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc0b7fe700 1 --2- 172.21.15.70:0/1144019184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3aeb8a0 0x5636a3af7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc0b7fe700 1 -- 172.21.15.70:0/1144019184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636a39898d0 con 0x5636a3aec260 2024-06-04T05:09:42.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc10f37700 1 --2- 172.21.15.70:0/1144019184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3af1320 0x5636a3af54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:09:42.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcc0b7fe700 1 --2- 172.21.15.70:0/1144019184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aec260 0x5636a3af4fa0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4000c00 tx=0x7fcbf402f800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b76ec10eb2ea664 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:42.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.967+0000 7fcbeb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:42.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.968+0000 7fcbeb7fe700 1 -- 172.21.15.70:0/1144019184 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4034040 con 0x5636a3aec260 2024-06-04T05:09:42.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.968+0000 7fcbeb7fe700 1 -- 172.21.15.70:0/1144019184 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7fcbf400f500 con 0x5636a3aec260 2024-06-04T05:09:42.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.968+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/1144019184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbf402f900 con 0x5636a3aec260 2024-06-04T05:09:42.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.968+0000 7fcbeb7fe700 1 -- 172.21.15.70:0/1144019184 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fcbf402fde0 con 0x5636a3aec260 2024-06-04T05:09:42.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.968+0000 7fcc1d22fb80 10 client.18753 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:42.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.968+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/1144019184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5636a39f3760 con 0x5636a3aec260 2024-06-04T05:09:42.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.968+0000 7fcbeb7fe700 1 -- 172.21.15.70:0/1144019184 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbf402f900 con 0x5636a3aec260 2024-06-04T05:09:42.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.968+0000 7fcbeb7fe700 1 -- 172.21.15.70:0/1144019184 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7fcbf4042440 con 0x5636a3aec260 2024-06-04T05:09:42.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.968+0000 7fcc1d22fb80 10 client.18753 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:42.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.968+0000 7fcc1d22fb80 10 client.18753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:42.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.968+0000 7fcc1d22fb80 20 client.18753 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:42.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.968+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/1144019184 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5636a3b05040 0x5636a3b07430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:42.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.968+0000 7fcc1d22fb80 4 client.18753 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:09:42.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.968+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/1144019184 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636a3a4bcc0 con 0x5636a3b05040 2024-06-04T05:09:42.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.969+0000 7fcc0bfff700 1 --2- 172.21.15.70:0/1144019184 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5636a3b05040 0x5636a3b07430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:42.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.969+0000 7fcc0bfff700 1 --2- 172.21.15.70:0/1144019184 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5636a3b05040 0x5636a3b07430 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41c84b1982550bf2 server_cookie=1321102572d2309d in_seq=0 out_seq=0 2024-06-04T05:09:42.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.969+0000 7fcbeb7fe700 10 client.18753 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:42.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.970+0000 7fcbeb7fe700 1 -- 172.21.15.70:0/1144019184 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5636a3a4bcc0 con 0x5636a3b05040 2024-06-04T05:09:42.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.970+0000 7fcbeb7fe700 10 client.18753 handle_command_reply: tid=0 2024-06-04T05:09:42.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.970+0000 7fcc1d22fb80 1 client.18753 shutdown 2024-06-04T05:09:42.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.970+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/1144019184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aec260 msgr2=0x5636a3af4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:42.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.970+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/1144019184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aec260 0x5636a3af4fa0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4000c00 tx=0x7fcbf402f800 comp rx=0 tx=0).stop 2024-06-04T05:09:42.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.970+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/1144019184 shutdown_connections 2024-06-04T05:09:42.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.970+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/1144019184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3af1320 0x5636a3af54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.970+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/1144019184 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5636a3b05040 0x5636a3b07430 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.970+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/1144019184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3aeb8a0 0x5636a3af7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.970+0000 7fcbeb7fe700 0 client.18753 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:42.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.970+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/1144019184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aec260 0x5636a3af4fa0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.970+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/1144019184 >> 172.21.15.70:0/1144019184 conn(0x5636a3a17240 msgr2=0x5636a3ae5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:42.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.970+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/1144019184 shutdown_connections 2024-06-04T05:09:42.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.970+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/1144019184 wait complete. 2024-06-04T05:09:42.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.970+0000 7fcc1d22fb80 20 client.18753 trim_cache size 0 max 16384 2024-06-04T05:09:42.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc1d22fb80 1 Processor -- start 2024-06-04T05:09:42.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc1d22fb80 1 -- start start 2024-06-04T05:09:42.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc1d22fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3aeb8a0 0x5636a3aebcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:42.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc1d22fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3aec260 0x5636a3ae6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:42.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc1d22fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3af1320 0x5636a3af1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:42.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc1d22fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c1add30 con 0x5636a3af1320 2024-06-04T05:09:42.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc1d22fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c1af990 con 0x5636a3aeb8a0 2024-06-04T05:09:42.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc1d22fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c1ae400 con 0x5636a3aec260 2024-06-04T05:09:42.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc10f37700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3af1320 0x5636a3af1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:42.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc10f37700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3af1320 0x5636a3af1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38492/0 (socket says 172.21.15.70:38492) 2024-06-04T05:09:42.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc10f37700 1 -- 172.21.15.70:0/694004657 learned_addr learned my addr 172.21.15.70:0/694004657 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:42.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc0bfff700 1 --2- 172.21.15.70:0/694004657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3aeb8a0 0x5636a3aebcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:42.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc0b7fe700 1 --2- 172.21.15.70:0/694004657 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3aec260 0x5636a3ae6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:42.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc10f37700 1 -- 172.21.15.70:0/694004657 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3aec260 msgr2=0x5636a3ae6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:42.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc10f37700 1 --2- 172.21.15.70:0/694004657 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3aec260 0x5636a3ae6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc10f37700 1 -- 172.21.15.70:0/694004657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3aeb8a0 msgr2=0x5636a3aebcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:42.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc10f37700 1 --2- 172.21.15.70:0/694004657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3aeb8a0 0x5636a3aebcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.990+0000 7fcc10f37700 1 -- 172.21.15.70:0/694004657 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636a3ae7df0 con 0x5636a3af1320 2024-06-04T05:09:42.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.991+0000 7fcc10f37700 1 --2- 172.21.15.70:0/694004657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3af1320 0x5636a3af1f80 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fcc04009f50 tx=0x7fcc04009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dfac7b8beb2295a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:42.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.991+0000 7fcc01ffb700 1 -- 172.21.15.70:0/694004657 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0401b070 con 0x5636a3af1320 2024-06-04T05:09:42.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.991+0000 7fcc01ffb700 1 -- 172.21.15.70:0/694004657 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcc04016a80 con 0x5636a3af1320 2024-06-04T05:09:42.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.991+0000 7fcc01ffb700 1 -- 172.21.15.70:0/694004657 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc04015ad0 con 0x5636a3af1320 2024-06-04T05:09:42.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.991+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/694004657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3af1320 msgr2=0x5636a3af1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:42.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.991+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/694004657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3af1320 0x5636a3af1f80 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fcc04009f50 tx=0x7fcc04009f80 comp rx=0 tx=0).stop 2024-06-04T05:09:42.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.991+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/694004657 shutdown_connections 2024-06-04T05:09:42.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.991+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/694004657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3af1320 0x5636a3af1f80 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.991+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/694004657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3aeb8a0 0x5636a3aebcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.991+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/694004657 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3aec260 0x5636a3ae6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.991+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/694004657 >> 172.21.15.70:0/694004657 conn(0x5636a3a17240 msgr2=0x5636a3aeb120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:42.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.991+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/694004657 shutdown_connections 2024-06-04T05:09:42.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.991+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/694004657 wait complete. 2024-06-04T05:09:42.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.992+0000 7fcc1d22fb80 1 -- start start 2024-06-04T05:09:42.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.992+0000 7fcc1d22fb80 1 Processor -- start 2024-06-04T05:09:42.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.992+0000 7fcc1d22fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3aeb8a0 0x5636a3afd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:42.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.992+0000 7fcc1d22fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aec260 0x5636a3afde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:42.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.992+0000 7fcc1d22fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3af1320 0x5636a3af88f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:42.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.992+0000 7fcc1d22fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c1af260 con 0x5636a3aec260 2024-06-04T05:09:42.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.992+0000 7fcc1d22fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c1aeb30 con 0x5636a3aeb8a0 2024-06-04T05:09:42.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.992+0000 7fcc1d22fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c1abcf0 con 0x5636a3af1320 2024-06-04T05:09:42.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.992+0000 7fcc1d22fb80 10 client.0 _ll_register_callbacks cb 0x5636a39d4950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:42.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.992+0000 7fcc0b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aec260 0x5636a3afde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:42.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.992+0000 7fcc0b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aec260 0x5636a3afde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38504/0 (socket says 172.21.15.70:38504) 2024-06-04T05:09:42.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.992+0000 7fcc0b7fe700 1 -- 172.21.15.70:0/1506960578 learned_addr learned my addr 172.21.15.70:0/1506960578 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:42.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.992+0000 7fcc0b7fe700 1 -- 172.21.15.70:0/1506960578 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3af1320 msgr2=0x5636a3af88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:42.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.992+0000 7fcc0b7fe700 1 --2- 172.21.15.70:0/1506960578 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3af1320 0x5636a3af88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.992+0000 7fcc0b7fe700 1 -- 172.21.15.70:0/1506960578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3aeb8a0 msgr2=0x5636a3afd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:42.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.992+0000 7fcc0bfff700 1 --2- 172.21.15.70:0/1506960578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3aeb8a0 0x5636a3afd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:42.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcc0b7fe700 1 --2- 172.21.15.70:0/1506960578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3aeb8a0 0x5636a3afd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcc0b7fe700 1 -- 172.21.15.70:0/1506960578 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636a3b0d580 con 0x5636a3aec260 2024-06-04T05:09:42.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcc0bfff700 1 --2- 172.21.15.70:0/1506960578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3aeb8a0 0x5636a3afd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:09:42.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcc0b7fe700 1 --2- 172.21.15.70:0/1506960578 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aec260 0x5636a3afde80 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4042a70 tx=0x7fcbf400b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6614940c89b6e211 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:42.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcbeb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:42.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcbeb7fe700 1 -- 172.21.15.70:0/1506960578 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4034040 con 0x5636a3aec260 2024-06-04T05:09:42.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcbeb7fe700 1 -- 172.21.15.70:0/1506960578 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7fcbf4042440 con 0x5636a3aec260 2024-06-04T05:09:42.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/1506960578 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc04016a80 con 0x5636a3aec260 2024-06-04T05:09:42.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcbeb7fe700 1 -- 172.21.15.70:0/1506960578 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fcbf4050050 con 0x5636a3aec260 2024-06-04T05:09:42.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcc1d22fb80 10 client.18759 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:42.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/1506960578 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5636a3b6fea0 con 0x5636a3aec260 2024-06-04T05:09:42.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcbeb7fe700 1 -- 172.21.15.70:0/1506960578 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcc04016a80 con 0x5636a3aec260 2024-06-04T05:09:42.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcbeb7fe700 1 -- 172.21.15.70:0/1506960578 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7fcbf403a470 con 0x5636a3aec260 2024-06-04T05:09:42.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcc1d22fb80 10 client.18759 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:42.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcc1d22fb80 10 client.18759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:42.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcc1d22fb80 20 client.18759 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:42.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/1506960578 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5636a3b70660 0x5636a3b72a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:42.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcc1d22fb80 4 client.18759 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:09:42.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.993+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/1506960578 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5636a3a454a0 con 0x5636a3b70660 2024-06-04T05:09:42.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.994+0000 7fcc0bfff700 1 --2- 172.21.15.70:0/1506960578 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5636a3b70660 0x5636a3b72a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:42.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.994+0000 7fcbeb7fe700 10 client.18759 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:42.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.994+0000 7fcc0bfff700 1 --2- 172.21.15.70:0/1506960578 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5636a3b70660 0x5636a3b72a50 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=479b41d22590ccc0 server_cookie=877e776861ab1296 in_seq=0 out_seq=0 2024-06-04T05:09:42.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.994+0000 7fcbeb7fe700 1 -- 172.21.15.70:0/1506960578 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5636a3a454a0 con 0x5636a3b70660 2024-06-04T05:09:42.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.994+0000 7fcbeb7fe700 10 client.18759 handle_command_reply: tid=0 2024-06-04T05:09:42.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.994+0000 7fcc1d22fb80 1 client.18759 shutdown 2024-06-04T05:09:42.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.994+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/1506960578 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aec260 msgr2=0x5636a3afde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:42.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.994+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/1506960578 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aec260 0x5636a3afde80 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4042a70 tx=0x7fcbf400b7a0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.994+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/1506960578 shutdown_connections 2024-06-04T05:09:42.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.994+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/1506960578 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5636a3af1320 0x5636a3af88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.994+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/1506960578 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5636a3b70660 0x5636a3b72a50 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.994+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/1506960578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636a3aeb8a0 0x5636a3afd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.994+0000 7fcbeb7fe700 0 client.18759 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:42.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.994+0000 7fcc1d22fb80 1 --2- 172.21.15.70:0/1506960578 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636a3aec260 0x5636a3afde80 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.994+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/1506960578 >> 172.21.15.70:0/1506960578 conn(0x5636a3a17240 msgr2=0x5636a3aeece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:42.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.995+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/1506960578 shutdown_connections 2024-06-04T05:09:42.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.995+0000 7fcc1d22fb80 1 -- 172.21.15.70:0/1506960578 wait complete. 2024-06-04T05:09:42.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.995+0000 7fcc1d22fb80 20 client.18759 trim_cache size 0 max 16384 2024-06-04T05:09:42.013 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:09:42.013 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:09:42.014 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:09:42.014 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:09:42.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.995+0000 7fcc1299a700 1 -- 172.21.15.70:0/184732985 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcbf80b29f0 msgr2=0x7fcbf80b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:42.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.995+0000 7fcc1299a700 1 --2- 172.21.15.70:0/184732985 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcbf80b29f0 0x7fcbf80b4eb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400b5c0 tx=0x7fcbf4005960 comp rx=0 tx=0).stop 2024-06-04T05:09:42.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.995+0000 7fcc1299a700 1 -- 172.21.15.70:0/184732985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc0c102470 msgr2=0x7fcc0c19a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:42.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.995+0000 7fcc1299a700 1 --2- 172.21.15.70:0/184732985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc0c102470 0x7fcc0c19a9f0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc00c9b0 tx=0x7fcbfc00ccc0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.995+0000 7fcc1299a700 1 -- 172.21.15.70:0/184732985 shutdown_connections 2024-06-04T05:09:42.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.995+0000 7fcc1299a700 1 --2- 172.21.15.70:0/184732985 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcbf80b29f0 0x7fcbf80b4eb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.995+0000 7fcc1299a700 1 --2- 172.21.15.70:0/184732985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc0c1042f0 0x7fcc0c1a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.995+0000 7fcc1299a700 1 --2- 172.21.15.70:0/184732985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc0c102470 0x7fcc0c19a9f0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.995+0000 7fcc1299a700 1 --2- 172.21.15.70:0/184732985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc0c102e30 0x7fcc0c19af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:42.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.995+0000 7fcc1299a700 1 -- 172.21.15.70:0/184732985 >> 172.21.15.70:0/184732985 conn(0x7fcc0c0784b0 msgr2=0x7fcc0c1072c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:42.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.995+0000 7fcc1299a700 1 -- 172.21.15.70:0/184732985 shutdown_connections 2024-06-04T05:09:42.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:41.995+0000 7fcc1299a700 1 -- 172.21.15.70:0/184732985 wait complete. 2024-06-04T05:09:42.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.68 MB/sec execute 515 sec latency 16031.947 ms 2024-06-04T05:09:42.747 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:09:42.748 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:09:42.987 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:09:43.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:43 smithi070 ceph-mon[32631]: pgmap v389: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 46 KiB/s wr, 21 op/s 2024-06-04T05:09:43.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:43 smithi098 ceph-mon[35569]: pgmap v389: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 46 KiB/s wr, 21 op/s 2024-06-04T05:09:43.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.524+0000 7fbf402f8640 1 -- 172.21.15.70:0/980628435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf38102340 msgr2=0x7fbf3810e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:43.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.524+0000 7fbf402f8640 1 --2- 172.21.15.70:0/980628435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf38102340 0x7fbf3810e800 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fbf30009cd0 tx=0x7fbf3002a8e0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.524+0000 7fbf402f8640 1 -- 172.21.15.70:0/980628435 shutdown_connections 2024-06-04T05:09:43.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.524+0000 7fbf402f8640 1 --2- 172.21.15.70:0/980628435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf38101980 0x7fbf38101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.524+0000 7fbf402f8640 1 --2- 172.21.15.70:0/980628435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf38102340 0x7fbf3810e800 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.524+0000 7fbf402f8640 1 --2- 172.21.15.70:0/980628435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf38107940 0x7fbf38107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.524+0000 7fbf402f8640 1 -- 172.21.15.70:0/980628435 >> 172.21.15.70:0/980628435 conn(0x7fbf38078ad0 msgr2=0x7fbf38078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:43.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.524+0000 7fbf402f8640 1 -- 172.21.15.70:0/980628435 shutdown_connections 2024-06-04T05:09:43.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.524+0000 7fbf402f8640 1 -- 172.21.15.70:0/980628435 wait complete. 2024-06-04T05:09:43.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.524+0000 7fbf402f8640 1 Processor -- start 2024-06-04T05:09:43.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.524+0000 7fbf402f8640 1 -- start start 2024-06-04T05:09:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf402f8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf38101980 0x7fbf38197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf402f8640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf38102340 0x7fbf381984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf402f8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf38107940 0x7fbf3819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf402f8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf38111ab0 con 0x7fbf38101980 2024-06-04T05:09:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf402f8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf38111930 con 0x7fbf38107940 2024-06-04T05:09:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf402f8640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbf38111c30 con 0x7fbf38102340 2024-06-04T05:09:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf3e094640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf38101980 0x7fbf38197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf3e094640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf38101980 0x7fbf38197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38524/0 (socket says 172.21.15.70:38524) 2024-06-04T05:09:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf3e094640 1 -- 172.21.15.70:0/1677911228 learned_addr learned my addr 172.21.15.70:0/1677911228 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf3e895640 1 --2- 172.21.15.70:0/1677911228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf38107940 0x7fbf3819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf3d893640 1 --2- 172.21.15.70:0/1677911228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf38102340 0x7fbf381984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf3e094640 1 -- 172.21.15.70:0/1677911228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf38102340 msgr2=0x7fbf381984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf3e094640 1 --2- 172.21.15.70:0/1677911228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf38102340 0x7fbf381984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf3e094640 1 -- 172.21.15.70:0/1677911228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf38107940 msgr2=0x7fbf3819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf3e094640 1 --2- 172.21.15.70:0/1677911228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf38107940 0x7fbf3819c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf3e094640 1 -- 172.21.15.70:0/1677911228 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf3819cf80 con 0x7fbf38101980 2024-06-04T05:09:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf3e094640 1 --2- 172.21.15.70:0/1677911228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf38101980 0x7fbf38197f90 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fbf3400d770 tx=0x7fbf3400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf2f7fe640 1 -- 172.21.15.70:0/1677911228 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf34017070 con 0x7fbf38101980 2024-06-04T05:09:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf2f7fe640 1 -- 172.21.15.70:0/1677911228 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf340054b0 con 0x7fbf38101980 2024-06-04T05:09:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf402f8640 1 -- 172.21.15.70:0/1677911228 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf3819d270 con 0x7fbf38101980 2024-06-04T05:09:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf2f7fe640 1 -- 172.21.15.70:0/1677911228 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf34004c60 con 0x7fbf38101980 2024-06-04T05:09:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.525+0000 7fbf402f8640 1 -- 172.21.15.70:0/1677911228 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf381a4b00 con 0x7fbf38101980 2024-06-04T05:09:43.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.528+0000 7fbf2f7fe640 1 -- 172.21.15.70:0/1677911228 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fbf34004e00 con 0x7fbf38101980 2024-06-04T05:09:43.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.528+0000 7fbf4300f740 1 Processor -- start 2024-06-04T05:09:43.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.528+0000 7fbf4300f740 1 -- start start 2024-06-04T05:09:43.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.528+0000 7fbf4300f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cdddf740 0x55b2cdddfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf4300f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdde0060 0x55b2cddec6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf4300f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cddecc10 0x55b2cddef000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf4300f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b2cdd51c30 con 0x55b2cddecc10 2024-06-04T05:09:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf4300f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b2cdd417d0 con 0x55b2cdddf740 2024-06-04T05:09:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf4300f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b2cdd45810 con 0x55b2cdde0060 2024-06-04T05:09:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf3d893640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cddecc10 0x55b2cddef000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf3d893640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cddecc10 0x55b2cddef000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38526/0 (socket says 172.21.15.70:38526) 2024-06-04T05:09:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf3d893640 1 -- 172.21.15.70:0/4079684986 learned_addr learned my addr 172.21.15.70:0/4079684986 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf2f7fe640 1 --2- 172.21.15.70:0/1677911228 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf240b2a90 0x7fbf240b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf3e895640 1 --2- 172.21.15.70:0/4079684986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cdddf740 0x55b2cdddfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf3e094640 1 --2- 172.21.15.70:0/4079684986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdde0060 0x55b2cddec6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf3e895640 1 --2- 172.21.15.70:0/1677911228 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf240b2a90 0x7fbf240b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf3d893640 1 -- 172.21.15.70:0/4079684986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdde0060 msgr2=0x55b2cddec6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf3d893640 1 --2- 172.21.15.70:0/4079684986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdde0060 0x55b2cddec6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf3d893640 1 -- 172.21.15.70:0/4079684986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cdddf740 msgr2=0x55b2cdddfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf3d893640 1 --2- 172.21.15.70:0/4079684986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cdddf740 0x55b2cdddfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf3d893640 1 -- 172.21.15.70:0/4079684986 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2cda56210 con 0x55b2cddecc10 2024-06-04T05:09:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf3e094640 1 --2- 172.21.15.70:0/4079684986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdde0060 0x55b2cddec6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:09:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf2f7fe640 1 -- 172.21.15.70:0/1677911228 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7fbf340e1fc0 con 0x7fbf38101980 2024-06-04T05:09:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf3d893640 1 --2- 172.21.15.70:0/4079684986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cddecc10 0x55b2cddef000 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800bc80 tx=0x7fbf2800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd9492cbd413a29f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf2cbfe640 1 -- 172.21.15.70:0/4079684986 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf28015070 con 0x55b2cddecc10 2024-06-04T05:09:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf3e895640 1 --2- 172.21.15.70:0/1677911228 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf240b2a90 0x7fbf240b4f50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbf30009430 tx=0x7fbf3002a8e0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf2cbfe640 1 -- 172.21.15.70:0/4079684986 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf2801a8f0 con 0x55b2cddecc10 2024-06-04T05:09:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.529+0000 7fbf2cbfe640 1 -- 172.21.15.70:0/4079684986 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf280117c0 con 0x55b2cddecc10 2024-06-04T05:09:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.530+0000 7fbf4300f740 1 -- 172.21.15.70:0/4079684986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cddecc10 msgr2=0x55b2cddef000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.530+0000 7fbf4300f740 1 --2- 172.21.15.70:0/4079684986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cddecc10 0x55b2cddef000 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800bc80 tx=0x7fbf2800f730 comp rx=0 tx=0).stop 2024-06-04T05:09:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.530+0000 7fbf4300f740 1 -- 172.21.15.70:0/4079684986 shutdown_connections 2024-06-04T05:09:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.530+0000 7fbf4300f740 1 --2- 172.21.15.70:0/4079684986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cddecc10 0x55b2cddef000 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.530+0000 7fbf4300f740 1 --2- 172.21.15.70:0/4079684986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdde0060 0x55b2cddec6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.530+0000 7fbf4300f740 1 --2- 172.21.15.70:0/4079684986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cdddf740 0x55b2cdddfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.530+0000 7fbf4300f740 1 -- 172.21.15.70:0/4079684986 >> 172.21.15.70:0/4079684986 conn(0x55b2cdddbd30 msgr2=0x55b2cdddc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.530+0000 7fbf4300f740 1 -- 172.21.15.70:0/4079684986 shutdown_connections 2024-06-04T05:09:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.530+0000 7fbf4300f740 1 -- 172.21.15.70:0/4079684986 wait complete. 2024-06-04T05:09:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf4300f740 1 -- start start 2024-06-04T05:09:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf4300f740 1 Processor -- start 2024-06-04T05:09:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf4300f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdddf740 0x55b2cdde9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf4300f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cdde0060 0x55b2cddea120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf4300f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cddecc10 0x55b2cddea660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf4300f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b2cda27790 con 0x55b2cddecc10 2024-06-04T05:09:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf4300f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b2cdcfd890 con 0x55b2cdde0060 2024-06-04T05:09:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf4300f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b2cddef840 con 0x55b2cdddf740 2024-06-04T05:09:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf4300f740 10 client.0 _ll_register_callbacks cb 0x55b2cda55a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf3e895640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cddecc10 0x55b2cddea660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf3e895640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cddecc10 0x55b2cddea660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38530/0 (socket says 172.21.15.70:38530) 2024-06-04T05:09:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf3e895640 1 -- 172.21.15.70:0/501449799 learned_addr learned my addr 172.21.15.70:0/501449799 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf3d893640 1 --2- 172.21.15.70:0/501449799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cdde0060 0x55b2cddea120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf3e094640 1 --2- 172.21.15.70:0/501449799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdddf740 0x55b2cdde9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf3e895640 1 -- 172.21.15.70:0/501449799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdddf740 msgr2=0x55b2cdde9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf3e895640 1 --2- 172.21.15.70:0/501449799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdddf740 0x55b2cdde9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf3e895640 1 -- 172.21.15.70:0/501449799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cdde0060 msgr2=0x55b2cddea120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf3e895640 1 --2- 172.21.15.70:0/501449799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cdde0060 0x55b2cddea120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf3e895640 1 -- 172.21.15.70:0/501449799 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2cdb22620 con 0x55b2cddecc10 2024-06-04T05:09:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.531+0000 7fbf3e094640 1 --2- 172.21.15.70:0/501449799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdddf740 0x55b2cdde9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:09:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.532+0000 7fbf3e895640 1 --2- 172.21.15.70:0/501449799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cddecc10 0x55b2cddea660 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fbf30035000 tx=0x7fbf300095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb17d28285dbd92 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.532+0000 7fbf21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.532+0000 7fbf21ffb640 1 -- 172.21.15.70:0/501449799 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf30004690 con 0x55b2cddecc10 2024-06-04T05:09:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.532+0000 7fbf4300f740 1 -- 172.21.15.70:0/501449799 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf2801a8f0 con 0x55b2cddecc10 2024-06-04T05:09:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.532+0000 7fbf21ffb640 1 -- 172.21.15.70:0/501449799 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7fbf300305a0 con 0x55b2cddecc10 2024-06-04T05:09:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.532+0000 7fbf21ffb640 1 -- 172.21.15.70:0/501449799 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fbf3005b050 con 0x55b2cddecc10 2024-06-04T05:09:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.532+0000 7fbf4300f740 10 client.18771 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.532+0000 7fbf4300f740 1 -- 172.21.15.70:0/501449799 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b2cddfb630 con 0x55b2cddecc10 2024-06-04T05:09:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.532+0000 7fbf21ffb640 1 -- 172.21.15.70:0/501449799 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf2801a8f0 con 0x55b2cddecc10 2024-06-04T05:09:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.532+0000 7fbf21ffb640 1 -- 172.21.15.70:0/501449799 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7fbf30030070 con 0x55b2cddecc10 2024-06-04T05:09:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.532+0000 7fbf4300f740 10 client.18771 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.532+0000 7fbf4300f740 10 client.18771 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.532+0000 7fbf4300f740 20 client.18771 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.532+0000 7fbf4300f740 1 --2- 172.21.15.70:0/501449799 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b2cddfcb60 0x55b2cddff000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.532+0000 7fbf4300f740 4 client.18771 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:09:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.532+0000 7fbf4300f740 1 -- 172.21.15.70:0/501449799 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2cdd53b80 con 0x55b2cddfcb60 2024-06-04T05:09:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.533+0000 7fbf3e094640 1 --2- 172.21.15.70:0/501449799 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b2cddfcb60 0x55b2cddff000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.533+0000 7fbf3e094640 1 --2- 172.21.15.70:0/501449799 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b2cddfcb60 0x55b2cddff000 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db148e2258b02eaf server_cookie=66a93b7446593d14 in_seq=0 out_seq=0 2024-06-04T05:09:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.533+0000 7fbf21ffb640 10 client.18771 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.534+0000 7fbf21ffb640 1 -- 172.21.15.70:0/501449799 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b2cdd53b80 con 0x55b2cddfcb60 2024-06-04T05:09:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.534+0000 7fbf21ffb640 10 client.18771 handle_command_reply: tid=0 2024-06-04T05:09:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.534+0000 7fbf4300f740 1 client.18771 shutdown 2024-06-04T05:09:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.534+0000 7fbf4300f740 1 -- 172.21.15.70:0/501449799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cddecc10 msgr2=0x55b2cddea660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.534+0000 7fbf4300f740 1 --2- 172.21.15.70:0/501449799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cddecc10 0x55b2cddea660 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fbf30035000 tx=0x7fbf300095c0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.534+0000 7fbf4300f740 1 -- 172.21.15.70:0/501449799 shutdown_connections 2024-06-04T05:09:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.534+0000 7fbf4300f740 1 --2- 172.21.15.70:0/501449799 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b2cddfcb60 0x55b2cddff000 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.534+0000 7fbf4300f740 1 --2- 172.21.15.70:0/501449799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cdde0060 0x55b2cddea120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.534+0000 7fbf21ffb640 0 client.18771 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.534+0000 7fbf4300f740 1 --2- 172.21.15.70:0/501449799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cddecc10 0x55b2cddea660 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.534+0000 7fbf4300f740 1 --2- 172.21.15.70:0/501449799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdddf740 0x55b2cdde9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.534+0000 7fbf4300f740 1 -- 172.21.15.70:0/501449799 >> 172.21.15.70:0/501449799 conn(0x55b2cdddbd30 msgr2=0x55b2cdde1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.534+0000 7fbf4300f740 1 -- 172.21.15.70:0/501449799 shutdown_connections 2024-06-04T05:09:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.534+0000 7fbf4300f740 1 -- 172.21.15.70:0/501449799 wait complete. 2024-06-04T05:09:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.534+0000 7fbf4300f740 20 client.18771 trim_cache size 0 max 16384 2024-06-04T05:09:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.549+0000 7fbf4300f740 1 Processor -- start 2024-06-04T05:09:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.549+0000 7fbf4300f740 1 -- start start 2024-06-04T05:09:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.549+0000 7fbf4300f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdddf740 0x55b2cddea2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.549+0000 7fbf4300f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cdde0060 0x55b2cde04910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.549+0000 7fbf4300f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cddecc10 0x55b2cde04e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.549+0000 7fbf4300f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf381ac3a0 con 0x55b2cdde0060 2024-06-04T05:09:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.549+0000 7fbf4300f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf381ae000 con 0x55b2cddecc10 2024-06-04T05:09:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.549+0000 7fbf4300f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbf381aca70 con 0x55b2cdddf740 2024-06-04T05:09:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.549+0000 7fbf3d893640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cdde0060 0x55b2cde04910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.549+0000 7fbf3d893640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cdde0060 0x55b2cde04910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38534/0 (socket says 172.21.15.70:38534) 2024-06-04T05:09:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.549+0000 7fbf3d893640 1 -- 172.21.15.70:0/820288141 learned_addr learned my addr 172.21.15.70:0/820288141 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.549+0000 7fbf3d893640 1 -- 172.21.15.70:0/820288141 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdddf740 msgr2=0x55b2cddea2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.549+0000 7fbf3d893640 1 --2- 172.21.15.70:0/820288141 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdddf740 0x55b2cddea2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.549+0000 7fbf3d893640 1 -- 172.21.15.70:0/820288141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cddecc10 msgr2=0x55b2cde04e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.549+0000 7fbf3d893640 1 --2- 172.21.15.70:0/820288141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cddecc10 0x55b2cde04e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.549+0000 7fbf3d893640 1 -- 172.21.15.70:0/820288141 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2cddd63c0 con 0x55b2cdde0060 2024-06-04T05:09:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.550+0000 7fbf3d893640 1 --2- 172.21.15.70:0/820288141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cdde0060 0x55b2cde04910 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800fc30 tx=0x7fbf28005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df7033d50102b23 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.550+0000 7fbf23fff640 1 -- 172.21.15.70:0/820288141 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf28015040 con 0x55b2cdde0060 2024-06-04T05:09:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.550+0000 7fbf23fff640 1 -- 172.21.15.70:0/820288141 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf28005040 con 0x55b2cdde0060 2024-06-04T05:09:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.550+0000 7fbf23fff640 1 -- 172.21.15.70:0/820288141 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf280048e0 con 0x55b2cdde0060 2024-06-04T05:09:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.550+0000 7fbf4300f740 1 -- 172.21.15.70:0/820288141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cdde0060 msgr2=0x55b2cde04910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.550+0000 7fbf4300f740 1 --2- 172.21.15.70:0/820288141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cdde0060 0x55b2cde04910 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800fc30 tx=0x7fbf28005c50 comp rx=0 tx=0).stop 2024-06-04T05:09:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.550+0000 7fbf4300f740 1 -- 172.21.15.70:0/820288141 shutdown_connections 2024-06-04T05:09:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.550+0000 7fbf4300f740 1 --2- 172.21.15.70:0/820288141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cddecc10 0x55b2cde04e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.550+0000 7fbf4300f740 1 --2- 172.21.15.70:0/820288141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cdde0060 0x55b2cde04910 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.550+0000 7fbf4300f740 1 --2- 172.21.15.70:0/820288141 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdddf740 0x55b2cddea2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.550+0000 7fbf4300f740 1 -- 172.21.15.70:0/820288141 >> 172.21.15.70:0/820288141 conn(0x55b2cdddbd30 msgr2=0x55b2cdde9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.550+0000 7fbf4300f740 1 -- 172.21.15.70:0/820288141 shutdown_connections 2024-06-04T05:09:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.550+0000 7fbf4300f740 1 -- 172.21.15.70:0/820288141 wait complete. 2024-06-04T05:09:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf4300f740 1 -- start start 2024-06-04T05:09:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf4300f740 1 Processor -- start 2024-06-04T05:09:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf4300f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdddf740 0x55b2cddf0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf4300f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cdde0060 0x55b2cddf1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf4300f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cddecc10 0x55b2cde667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf4300f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf381ad8d0 con 0x55b2cdde0060 2024-06-04T05:09:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf4300f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf381ad1a0 con 0x55b2cddecc10 2024-06-04T05:09:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf4300f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbf381aa3a0 con 0x55b2cdddf740 2024-06-04T05:09:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf4300f740 10 client.0 _ll_register_callbacks cb 0x7fbf30005f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf3d893640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cdde0060 0x55b2cddf1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf3d893640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cdde0060 0x55b2cddf1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38542/0 (socket says 172.21.15.70:38542) 2024-06-04T05:09:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf3d893640 1 -- 172.21.15.70:0/71217600 learned_addr learned my addr 172.21.15.70:0/71217600 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf3e895640 1 --2- 172.21.15.70:0/71217600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cddecc10 0x55b2cde667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf3e094640 1 --2- 172.21.15.70:0/71217600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdddf740 0x55b2cddf0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf3d893640 1 -- 172.21.15.70:0/71217600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdddf740 msgr2=0x55b2cddf0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf3d893640 1 --2- 172.21.15.70:0/71217600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdddf740 0x55b2cddf0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf3d893640 1 -- 172.21.15.70:0/71217600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cddecc10 msgr2=0x55b2cde667e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf3d893640 1 --2- 172.21.15.70:0/71217600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cddecc10 0x55b2cde667e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf3d893640 1 -- 172.21.15.70:0/71217600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2cde66d20 con 0x55b2cdde0060 2024-06-04T05:09:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.551+0000 7fbf3e094640 1 --2- 172.21.15.70:0/71217600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdddf740 0x55b2cddf0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:09:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.552+0000 7fbf21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.552+0000 7fbf3d893640 1 --2- 172.21.15.70:0/71217600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cdde0060 0x55b2cddf1120 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800b830 tx=0x7fbf2800b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c61c01c99be94d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.552+0000 7fbf21ffb640 1 -- 172.21.15.70:0/71217600 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf28031020 con 0x55b2cdde0060 2024-06-04T05:09:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.552+0000 7fbf21ffb640 1 -- 172.21.15.70:0/71217600 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7fbf28004680 con 0x55b2cdde0060 2024-06-04T05:09:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.552+0000 7fbf4300f740 1 -- 172.21.15.70:0/71217600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf28005040 con 0x55b2cdde0060 2024-06-04T05:09:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.552+0000 7fbf21ffb640 1 -- 172.21.15.70:0/71217600 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fbf2803c050 con 0x55b2cdde0060 2024-06-04T05:09:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.552+0000 7fbf4300f740 10 client.18777 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.552+0000 7fbf4300f740 1 -- 172.21.15.70:0/71217600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b2cde67ef0 con 0x55b2cdde0060 2024-06-04T05:09:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.552+0000 7fbf21ffb640 1 -- 172.21.15.70:0/71217600 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf28005040 con 0x55b2cdde0060 2024-06-04T05:09:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.552+0000 7fbf21ffb640 1 -- 172.21.15.70:0/71217600 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7fbf280117b0 con 0x55b2cdde0060 2024-06-04T05:09:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.552+0000 7fbf4300f740 10 client.18777 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.552+0000 7fbf4300f740 10 client.18777 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.552+0000 7fbf4300f740 20 client.18777 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.552+0000 7fbf4300f740 1 --2- 172.21.15.70:0/71217600 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b2cde683c0 0x55b2cde6a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.553+0000 7fbf4300f740 4 client.18777 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:09:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.553+0000 7fbf4300f740 1 -- 172.21.15.70:0/71217600 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b2cdd4d360 con 0x55b2cde683c0 2024-06-04T05:09:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.553+0000 7fbf3e094640 1 --2- 172.21.15.70:0/71217600 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b2cde683c0 0x55b2cde6a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.553+0000 7fbf3e094640 1 --2- 172.21.15.70:0/71217600 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b2cde683c0 0x55b2cde6a7e0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57b55351a40e6c45 server_cookie=f8aec298d01c5c73 in_seq=0 out_seq=0 2024-06-04T05:09:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.553+0000 7fbf21ffb640 10 client.18777 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:43.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.573+0000 7fbf21ffb640 1 -- 172.21.15.70:0/71217600 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55b2cdd4d360 con 0x55b2cde683c0 2024-06-04T05:09:43.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.573+0000 7fbf21ffb640 10 client.18777 handle_command_reply: tid=0 2024-06-04T05:09:43.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.573+0000 7fbf4300f740 1 client.18777 shutdown 2024-06-04T05:09:43.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.573+0000 7fbf4300f740 1 -- 172.21.15.70:0/71217600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cdde0060 msgr2=0x55b2cddf1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:43.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.573+0000 7fbf4300f740 1 --2- 172.21.15.70:0/71217600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cdde0060 0x55b2cddf1120 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800b830 tx=0x7fbf2800b950 comp rx=0 tx=0).stop 2024-06-04T05:09:43.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.573+0000 7fbf4300f740 1 -- 172.21.15.70:0/71217600 shutdown_connections 2024-06-04T05:09:43.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.573+0000 7fbf4300f740 1 --2- 172.21.15.70:0/71217600 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b2cde683c0 0x55b2cde6a7e0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.573+0000 7fbf4300f740 1 --2- 172.21.15.70:0/71217600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2cddecc10 0x55b2cde667e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.573+0000 7fbf4300f740 1 --2- 172.21.15.70:0/71217600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2cdde0060 0x55b2cddf1120 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.573+0000 7fbf21ffb640 0 client.18777 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:43.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.573+0000 7fbf4300f740 1 --2- 172.21.15.70:0/71217600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b2cdddf740 0x55b2cddf0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.573+0000 7fbf4300f740 1 -- 172.21.15.70:0/71217600 >> 172.21.15.70:0/71217600 conn(0x55b2cdddbd30 msgr2=0x55b2cddea430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:43.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.573+0000 7fbf4300f740 1 -- 172.21.15.70:0/71217600 shutdown_connections 2024-06-04T05:09:43.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.573+0000 7fbf4300f740 1 -- 172.21.15.70:0/71217600 wait complete. 2024-06-04T05:09:43.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.573+0000 7fbf4300f740 20 client.18777 trim_cache size 0 max 16384 2024-06-04T05:09:43.577 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:09:43.577 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:09:43.577 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:09:43.577 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.574+0000 7fbf402f8640 1 -- 172.21.15.70:0/1677911228 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf240b2a90 msgr2=0x7fbf240b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.574+0000 7fbf402f8640 1 --2- 172.21.15.70:0/1677911228 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf240b2a90 0x7fbf240b4f50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbf30009430 tx=0x7fbf3002a8e0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.574+0000 7fbf402f8640 1 -- 172.21.15.70:0/1677911228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf38101980 msgr2=0x7fbf38197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.574+0000 7fbf402f8640 1 --2- 172.21.15.70:0/1677911228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf38101980 0x7fbf38197f90 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fbf3400d770 tx=0x7fbf3400dc30 comp rx=0 tx=0).stop 2024-06-04T05:09:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.574+0000 7fbf402f8640 1 -- 172.21.15.70:0/1677911228 shutdown_connections 2024-06-04T05:09:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.574+0000 7fbf402f8640 1 --2- 172.21.15.70:0/1677911228 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbf240b2a90 0x7fbf240b4f50 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.574+0000 7fbf402f8640 1 --2- 172.21.15.70:0/1677911228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf38107940 0x7fbf3819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.574+0000 7fbf402f8640 1 --2- 172.21.15.70:0/1677911228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf38102340 0x7fbf381984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.574+0000 7fbf402f8640 1 --2- 172.21.15.70:0/1677911228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf38101980 0x7fbf38197f90 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.574+0000 7fbf402f8640 1 -- 172.21.15.70:0/1677911228 >> 172.21.15.70:0/1677911228 conn(0x7fbf38078ad0 msgr2=0x7fbf381045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.574+0000 7fbf402f8640 1 -- 172.21.15.70:0/1677911228 shutdown_connections 2024-06-04T05:09:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:43.574+0000 7fbf402f8640 1 -- 172.21.15.70:0/1677911228 wait complete. 2024-06-04T05:09:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:43 smithi112 ceph-mon[36453]: pgmap v389: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 46 KiB/s wr, 21 op/s 2024-06-04T05:09:43.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.68 MB/sec execute 516 sec latency 17031.974 ms 2024-06-04T05:09:44.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.665+0000 7f07003c2700 1 -- 172.21.15.70:0/4290009514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06f81035c0 msgr2=0x7f06f8103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:44.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.665+0000 7f07003c2700 1 --2- 172.21.15.70:0/4290009514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06f81035c0 0x7f06f8103a40 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f06e8009990 tx=0x7f06e8009ca0 comp rx=0 tx=0).stop 2024-06-04T05:09:44.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.665+0000 7f07003c2700 1 -- 172.21.15.70:0/4290009514 shutdown_connections 2024-06-04T05:09:44.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.665+0000 7f07003c2700 1 --2- 172.21.15.70:0/4290009514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06f8103f80 0x7f06f8110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:44.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.665+0000 7f07003c2700 1 --2- 172.21.15.70:0/4290009514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06f81095d0 0x7f06f81099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:44.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.665+0000 7f07003c2700 1 --2- 172.21.15.70:0/4290009514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06f81035c0 0x7f06f8103a40 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:44.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.665+0000 7f07003c2700 1 -- 172.21.15.70:0/4290009514 >> 172.21.15.70:0/4290009514 conn(0x7f06f807a610 msgr2=0x7f06f807aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:44.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.665+0000 7f07003c2700 1 -- 172.21.15.70:0/4290009514 shutdown_connections 2024-06-04T05:09:44.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.665+0000 7f07003c2700 1 -- 172.21.15.70:0/4290009514 wait complete. 2024-06-04T05:09:44.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.665+0000 7f07003c2700 1 Processor -- start 2024-06-04T05:09:44.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.665+0000 7f07003c2700 1 -- start start 2024-06-04T05:09:44.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f07003c2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06f81035c0 0x7f06f819f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:44.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f07003c2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06f8103f80 0x7f06f819fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:44.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f07003c2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06f81095d0 0x7f06f8199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:44.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f07003c2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f06f81135f0 con 0x7f06f81095d0 2024-06-04T05:09:44.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f07003c2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f06f8113470 con 0x7f06f8103f80 2024-06-04T05:09:44.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f07003c2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f06f8113770 con 0x7f06f81035c0 2024-06-04T05:09:44.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f06fe95f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06f81095d0 0x7f06f8199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:44.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f06fe95f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06f81095d0 0x7f06f8199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38554/0 (socket says 172.21.15.70:38554) 2024-06-04T05:09:44.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f06fe95f700 1 -- 172.21.15.70:0/1528252607 learned_addr learned my addr 172.21.15.70:0/1528252607 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:44.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f06fd95d700 1 --2- 172.21.15.70:0/1528252607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06f8103f80 0x7f06f819fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:44.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f06fe15e700 1 --2- 172.21.15.70:0/1528252607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06f81035c0 0x7f06f819f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:44.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f06fe95f700 1 -- 172.21.15.70:0/1528252607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06f81035c0 msgr2=0x7f06f819f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:44.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f06fe95f700 1 --2- 172.21.15.70:0/1528252607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06f81035c0 0x7f06f819f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f06fe95f700 1 -- 172.21.15.70:0/1528252607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06f8103f80 msgr2=0x7f06f819fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f06fe95f700 1 --2- 172.21.15.70:0/1528252607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06f8103f80 0x7f06f819fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f06fe95f700 1 -- 172.21.15.70:0/1528252607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06f8199ff0 con 0x7f06f81095d0 2024-06-04T05:09:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f06fe95f700 1 --2- 172.21.15.70:0/1528252607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06f81095d0 0x7f06f8199890 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f06f0007ee0 tx=0x7f06f000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f06ef7fe700 1 -- 172.21.15.70:0/1528252607 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f0004bb0 con 0x7f06f81095d0 2024-06-04T05:09:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f06ef7fe700 1 -- 172.21.15.70:0/1528252607 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f06f0004d50 con 0x7f06f81095d0 2024-06-04T05:09:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f06ef7fe700 1 -- 172.21.15.70:0/1528252607 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f00058b0 con 0x7f06f81095d0 2024-06-04T05:09:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.666+0000 7f07003c2700 1 -- 172.21.15.70:0/1528252607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06f819a2e0 con 0x7f06f81095d0 2024-06-04T05:09:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.667+0000 7f07003c2700 1 -- 172.21.15.70:0/1528252607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06f81a65a0 con 0x7f06f81095d0 2024-06-04T05:09:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.667+0000 7f07003c2700 1 -- 172.21.15.70:0/1528252607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06f806a230 con 0x7f06f81095d0 2024-06-04T05:09:44.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.669+0000 7f06ef7fe700 1 -- 172.21.15.70:0/1528252607 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f06f001a020 con 0x7f06f81095d0 2024-06-04T05:09:44.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.669+0000 7f06ef7fe700 1 --2- 172.21.15.70:0/1528252607 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f06e40b2a70 0x7f06e40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:44.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.670+0000 7f06fe15e700 1 --2- 172.21.15.70:0/1528252607 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f06e40b2a70 0x7f06e40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:44.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.670+0000 7f06ef7fe700 1 -- 172.21.15.70:0/1528252607 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f06f00e2310 con 0x7f06f81095d0 2024-06-04T05:09:44.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.670+0000 7f06fe15e700 1 --2- 172.21.15.70:0/1528252607 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f06e40b2a70 0x7f06e40b4f30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f06f819a9d0 tx=0x7f06f400a380 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:44.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.673+0000 7f06ef7fe700 1 -- 172.21.15.70:0/1528252607 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f06f0010070 con 0x7f06f81095d0 2024-06-04T05:09:44.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.68 MB/sec execute 517 sec latency 18032.028 ms 2024-06-04T05:09:44.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.927+0000 7f07003c2700 1 -- 172.21.15.70:0/1528252607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06f802e560 con 0x7f06f81095d0 2024-06-04T05:09:44.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.927+0000 7f06ef7fe700 1 -- 172.21.15.70:0/1528252607 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11938 (secure 0 0 0) 0x7f06f00a9cc0 con 0x7f06f81095d0 2024-06-04T05:09:44.930 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:09:44.930 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:09:17.032462+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24436,24410,24440,24448,24442]},"id":1}]} 2024-06-04T05:09:44.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.930+0000 7f07003c2700 1 -- 172.21.15.70:0/1528252607 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f06e40b2a70 msgr2=0x7f06e40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:44.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.930+0000 7f07003c2700 1 --2- 172.21.15.70:0/1528252607 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f06e40b2a70 0x7f06e40b4f30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f06f819a9d0 tx=0x7f06f400a380 comp rx=0 tx=0).stop 2024-06-04T05:09:44.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.930+0000 7f07003c2700 1 -- 172.21.15.70:0/1528252607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06f81095d0 msgr2=0x7f06f8199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:44.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.930+0000 7f07003c2700 1 --2- 172.21.15.70:0/1528252607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06f81095d0 0x7f06f8199890 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f06f0007ee0 tx=0x7f06f000d3b0 comp rx=0 tx=0).stop 2024-06-04T05:09:44.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.930+0000 7f07003c2700 1 -- 172.21.15.70:0/1528252607 shutdown_connections 2024-06-04T05:09:44.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.930+0000 7f07003c2700 1 --2- 172.21.15.70:0/1528252607 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f06e40b2a70 0x7f06e40b4f30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:44.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.930+0000 7f07003c2700 1 --2- 172.21.15.70:0/1528252607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06f81095d0 0x7f06f8199890 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:44.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.930+0000 7f07003c2700 1 --2- 172.21.15.70:0/1528252607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06f81035c0 0x7f06f819f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:44.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.930+0000 7f07003c2700 1 --2- 172.21.15.70:0/1528252607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06f8103f80 0x7f06f819fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:44.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.930+0000 7f07003c2700 1 -- 172.21.15.70:0/1528252607 >> 172.21.15.70:0/1528252607 conn(0x7f06f807a610 msgr2=0x7f06f81022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:44.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.930+0000 7f07003c2700 1 -- 172.21.15.70:0/1528252607 shutdown_connections 2024-06-04T05:09:44.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:44.930+0000 7f07003c2700 1 -- 172.21.15.70:0/1528252607 wait complete. 2024-06-04T05:09:44.933 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 38 2024-06-04T05:09:45.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:45 smithi098 ceph-mon[35569]: pgmap v390: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-06-04T05:09:45.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:45 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1528252607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:09:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:45 smithi070 ceph-mon[32631]: pgmap v390: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-06-04T05:09:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:45 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1528252607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:09:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:45 smithi112 ceph-mon[36453]: pgmap v390: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-06-04T05:09:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:45 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1528252607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:09:45.735 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:09:45.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.67 MB/sec execute 518 sec latency 19032.062 ms 2024-06-04T05:09:45.981 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:09:46.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.67 MB/sec execute 519 sec latency 20032.141 ms 2024-06-04T05:09:47.447 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:47 smithi070 ceph-mon[32631]: pgmap v391: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 32 KiB/s wr, 17 op/s 2024-06-04T05:09:47.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:47 smithi098 ceph-mon[35569]: pgmap v391: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 32 KiB/s wr, 17 op/s 2024-06-04T05:09:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:47 smithi112 ceph-mon[36453]: pgmap v391: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 32 KiB/s wr, 17 op/s 2024-06-04T05:09:47.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.67 MB/sec execute 520 sec latency 21032.208 ms 2024-06-04T05:09:47.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.789+0000 7f9d9d838700 1 -- 172.21.15.70:0/1671544495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d98108550 msgr2=0x7f9d98108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:47.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.789+0000 7f9d9d838700 1 --2- 172.21.15.70:0/1671544495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d98108550 0x7f9d98108930 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f9d9000b0d0 tx=0x7f9d9000b3e0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.789+0000 7f9d9d838700 1 -- 172.21.15.70:0/1671544495 shutdown_connections 2024-06-04T05:09:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.789+0000 7f9d9d838700 1 --2- 172.21.15.70:0/1671544495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d98102e30 0x7f9d9810f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.789+0000 7f9d9d838700 1 --2- 172.21.15.70:0/1671544495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d98108550 0x7f9d98108930 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.789+0000 7f9d9d838700 1 --2- 172.21.15.70:0/1671544495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d98102470 0x7f9d981028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.789+0000 7f9d9d838700 1 -- 172.21.15.70:0/1671544495 >> 172.21.15.70:0/1671544495 conn(0x7f9d98079480 msgr2=0x7f9d98079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.789+0000 7f9d9d838700 1 -- 172.21.15.70:0/1671544495 shutdown_connections 2024-06-04T05:09:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.789+0000 7f9d9d838700 1 -- 172.21.15.70:0/1671544495 wait complete. 2024-06-04T05:09:47.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.790+0000 7f9d9d838700 1 Processor -- start 2024-06-04T05:09:47.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.790+0000 7f9d9d838700 1 -- start start 2024-06-04T05:09:47.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.790+0000 7f9d9d838700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d98102470 0x7f9d981040b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.790+0000 7f9d9d838700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d98102e30 0x7f9d981045f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.790+0000 7f9d9d838700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d98108550 0x7f9d98104b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.790+0000 7f9d9d838700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d981125d0 con 0x7f9d98102e30 2024-06-04T05:09:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.790+0000 7f9d9d838700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9d98112450 con 0x7f9d98102470 2024-06-04T05:09:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.790+0000 7f9d9d838700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9d98112750 con 0x7f9d98108550 2024-06-04T05:09:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d967fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d98102e30 0x7f9d981045f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d967fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d98102e30 0x7f9d981045f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46156/0 (socket says 172.21.15.70:46156) 2024-06-04T05:09:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d967fc700 1 -- 172.21.15.70:0/2425227592 learned_addr learned my addr 172.21.15.70:0/2425227592 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d96ffd700 1 --2- 172.21.15.70:0/2425227592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d98102470 0x7f9d981040b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d977fe700 1 --2- 172.21.15.70:0/2425227592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d98108550 0x7f9d98104b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d967fc700 1 -- 172.21.15.70:0/2425227592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d98108550 msgr2=0x7f9d98104b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d967fc700 1 --2- 172.21.15.70:0/2425227592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d98108550 0x7f9d98104b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d967fc700 1 -- 172.21.15.70:0/2425227592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d98102470 msgr2=0x7f9d981040b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d967fc700 1 --2- 172.21.15.70:0/2425227592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d98102470 0x7f9d981040b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d967fc700 1 -- 172.21.15.70:0/2425227592 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d981a8200 con 0x7f9d98102e30 2024-06-04T05:09:47.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d96ffd700 1 --2- 172.21.15.70:0/2425227592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d98102470 0x7f9d981040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:09:47.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d977fe700 1 --2- 172.21.15.70:0/2425227592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d98108550 0x7f9d98104b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:09:47.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d967fc700 1 --2- 172.21.15.70:0/2425227592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d98102e30 0x7f9d981045f0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000b530 tx=0x7f9d8000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:47.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d9c836700 1 -- 172.21.15.70:0/2425227592 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d80015020 con 0x7f9d98102e30 2024-06-04T05:09:47.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d9d838700 1 -- 172.21.15.70:0/2425227592 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d981a84f0 con 0x7f9d98102e30 2024-06-04T05:09:47.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d9c836700 1 -- 172.21.15.70:0/2425227592 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d80011cc0 con 0x7f9d98102e30 2024-06-04T05:09:47.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d9c836700 1 -- 172.21.15.70:0/2425227592 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d80010d30 con 0x7f9d98102e30 2024-06-04T05:09:47.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.791+0000 7f9d9d838700 1 -- 172.21.15.70:0/2425227592 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d981a8a80 con 0x7f9d98102e30 2024-06-04T05:09:47.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.794+0000 7f9d9c836700 1 -- 172.21.15.70:0/2425227592 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f9d80018020 con 0x7f9d98102e30 2024-06-04T05:09:47.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.794+0000 7f9d9c836700 1 --2- 172.21.15.70:0/2425227592 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9d840b29f0 0x7f9d840b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.795+0000 7f9d96ffd700 1 --2- 172.21.15.70:0/2425227592 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9d840b29f0 0x7f9d840b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:47.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.795+0000 7f9d9c836700 1 -- 172.21.15.70:0/2425227592 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f9d800e1dc0 con 0x7f9d98102e30 2024-06-04T05:09:47.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.795+0000 7f9da80cdb80 1 Processor -- start 2024-06-04T05:09:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.795+0000 7f9d96ffd700 1 --2- 172.21.15.70:0/2425227592 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9d840b29f0 0x7f9d840b4eb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9d90009e60 tx=0x7f9d9000b040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.795+0000 7f9da80cdb80 1 -- start start 2024-06-04T05:09:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.795+0000 7f9da80cdb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509587320 0x562509587700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.795+0000 7f9da80cdb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625095818a0 0x562509581d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.795+0000 7f9da80cdb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509582260 0x56250958e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.795+0000 7f9da80cdb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5625094dfd70 con 0x562509587320 2024-06-04T05:09:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.795+0000 7f9da80cdb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5625094cf910 con 0x562509582260 2024-06-04T05:09:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.795+0000 7f9da80cdb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5625094d3950 con 0x5625095818a0 2024-06-04T05:09:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.795+0000 7f9d96ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509587320 0x562509587700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.795+0000 7f9d96ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509587320 0x562509587700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46166/0 (socket says 172.21.15.70:46166) 2024-06-04T05:09:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.795+0000 7f9d96ffd700 1 -- 172.21.15.70:0/4169334774 learned_addr learned my addr 172.21.15.70:0/4169334774 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.796+0000 7f9d967fc700 1 --2- 172.21.15.70:0/4169334774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625095818a0 0x562509581d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.796+0000 7f9d96ffd700 1 -- 172.21.15.70:0/4169334774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625095818a0 msgr2=0x562509581d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.796+0000 7f9d96ffd700 1 --2- 172.21.15.70:0/4169334774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625095818a0 0x562509581d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.796+0000 7f9d96ffd700 1 -- 172.21.15.70:0/4169334774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509582260 msgr2=0x56250958e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:47.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.796+0000 7f9d96ffd700 1 --2- 172.21.15.70:0/4169334774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509582260 0x56250958e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.796+0000 7f9d96ffd700 1 -- 172.21.15.70:0/4169334774 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562509478fb0 con 0x562509587320 2024-06-04T05:09:47.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.796+0000 7f9d96ffd700 1 --2- 172.21.15.70:0/4169334774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509587320 0x562509587700 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f9d90004d60 tx=0x7f9d9001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ecfc388f91e6454 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:47.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.796+0000 7f9d8dffb700 1 -- 172.21.15.70:0/4169334774 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d90007d00 con 0x562509587320 2024-06-04T05:09:47.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.796+0000 7f9d8dffb700 1 -- 172.21.15.70:0/4169334774 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d9002e070 con 0x562509587320 2024-06-04T05:09:47.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.796+0000 7f9d8dffb700 1 -- 172.21.15.70:0/4169334774 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d900053c0 con 0x562509587320 2024-06-04T05:09:47.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.797+0000 7f9da80cdb80 1 -- 172.21.15.70:0/4169334774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509587320 msgr2=0x562509587700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:47.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.797+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/4169334774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509587320 0x562509587700 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f9d90004d60 tx=0x7f9d9001c000 comp rx=0 tx=0).stop 2024-06-04T05:09:47.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.797+0000 7f9da80cdb80 1 -- 172.21.15.70:0/4169334774 shutdown_connections 2024-06-04T05:09:47.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.797+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/4169334774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509582260 0x56250958e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.797+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/4169334774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509587320 0x562509587700 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.797+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/4169334774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625095818a0 0x562509581d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.797+0000 7f9da80cdb80 1 -- 172.21.15.70:0/4169334774 >> 172.21.15.70:0/4169334774 conn(0x5625094ad240 msgr2=0x56250957c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:47.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.797+0000 7f9da80cdb80 1 -- 172.21.15.70:0/4169334774 shutdown_connections 2024-06-04T05:09:47.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.797+0000 7f9da80cdb80 1 -- 172.21.15.70:0/4169334774 wait complete. 2024-06-04T05:09:47.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.797+0000 7f9da80cdb80 1 -- start start 2024-06-04T05:09:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9da80cdb80 1 Processor -- start 2024-06-04T05:09:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9da80cdb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625095818a0 0x56250958dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9da80cdb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509582260 0x56250958b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9da80cdb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509587320 0x56250958b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9da80cdb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5625093d5fa0 con 0x562509587320 2024-06-04T05:09:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9da80cdb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562509437da0 con 0x562509582260 2024-06-04T05:09:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9da80cdb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56250958e680 con 0x5625095818a0 2024-06-04T05:09:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9da80cdb80 10 client.0 _ll_register_callbacks cb 0x56250946a950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9d977fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509587320 0x56250958b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9d967fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509582260 0x56250958b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9d977fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509587320 0x56250958b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46168/0 (socket says 172.21.15.70:46168) 2024-06-04T05:09:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9d977fe700 1 -- 172.21.15.70:0/2582098842 learned_addr learned my addr 172.21.15.70:0/2582098842 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9d96ffd700 1 --2- 172.21.15.70:0/2582098842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625095818a0 0x56250958dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:47.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9d977fe700 1 -- 172.21.15.70:0/2582098842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625095818a0 msgr2=0x56250958dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:47.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9d977fe700 1 --2- 172.21.15.70:0/2582098842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625095818a0 0x56250958dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9d977fe700 1 -- 172.21.15.70:0/2582098842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509582260 msgr2=0x56250958b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:47.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9d977fe700 1 --2- 172.21.15.70:0/2582098842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509582260 0x56250958b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9d977fe700 1 -- 172.21.15.70:0/2582098842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56250941f8d0 con 0x562509587320 2024-06-04T05:09:47.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9d977fe700 1 --2- 172.21.15.70:0/2582098842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509587320 0x56250958b5e0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f9d8800d1e0 tx=0x7f9d8800d4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=171563e71e4e6f1a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:47.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.798+0000 7f9d76ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:47.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.799+0000 7f9d76ffd700 1 -- 172.21.15.70:0/2582098842 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8800dd80 con 0x562509587320 2024-06-04T05:09:47.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.799+0000 7f9d76ffd700 1 -- 172.21.15.70:0/2582098842 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f9d88013480 con 0x562509587320 2024-06-04T05:09:47.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.799+0000 7f9da80cdb80 1 -- 172.21.15.70:0/2582098842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d9002e070 con 0x562509587320 2024-06-04T05:09:47.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.799+0000 7f9d76ffd700 1 -- 172.21.15.70:0/2582098842 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f9d88021050 con 0x562509587320 2024-06-04T05:09:47.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.799+0000 7f9da80cdb80 10 client.18795 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:47.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.799+0000 7f9da80cdb80 1 -- 172.21.15.70:0/2582098842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x562509489760 con 0x562509587320 2024-06-04T05:09:47.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.799+0000 7f9d76ffd700 1 -- 172.21.15.70:0/2582098842 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d9002e070 con 0x562509587320 2024-06-04T05:09:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.799+0000 7f9d76ffd700 1 -- 172.21.15.70:0/2582098842 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f9d880126e0 con 0x562509587320 2024-06-04T05:09:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.799+0000 7f9da80cdb80 10 client.18795 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.799+0000 7f9da80cdb80 10 client.18795 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:09:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.799+0000 7f9da80cdb80 20 client.18795 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.799+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/2582098842 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56250959b260 0x56250959d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.799+0000 7f9da80cdb80 4 client.18795 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:09:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.799+0000 7f9da80cdb80 1 -- 172.21.15.70:0/2582098842 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625094e1cc0 con 0x56250959b260 2024-06-04T05:09:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.800+0000 7f9d96ffd700 1 --2- 172.21.15.70:0/2582098842 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56250959b260 0x56250959d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.800+0000 7f9d76ffd700 10 client.18795 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:09:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.800+0000 7f9d96ffd700 1 --2- 172.21.15.70:0/2582098842 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56250959b260 0x56250959d680 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b9d117a9b58fa0ef server_cookie=72493e8f84314174 in_seq=0 out_seq=0 2024-06-04T05:09:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.801+0000 7f9d76ffd700 1 -- 172.21.15.70:0/2582098842 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5625094e1cc0 con 0x56250959b260 2024-06-04T05:09:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.801+0000 7f9d76ffd700 10 client.18795 handle_command_reply: tid=0 2024-06-04T05:09:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.801+0000 7f9da80cdb80 1 client.18795 shutdown 2024-06-04T05:09:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.801+0000 7f9da80cdb80 1 -- 172.21.15.70:0/2582098842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509587320 msgr2=0x56250958b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.801+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/2582098842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509587320 0x56250958b5e0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f9d8800d1e0 tx=0x7f9d8800d4f0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.801+0000 7f9da80cdb80 1 -- 172.21.15.70:0/2582098842 shutdown_connections 2024-06-04T05:09:47.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.801+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/2582098842 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56250959b260 0x56250959d680 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.801+0000 7f9d76ffd700 0 client.18795 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:09:47.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.801+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/2582098842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509587320 0x56250958b5e0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.801+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/2582098842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625095818a0 0x56250958dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.801+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/2582098842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509582260 0x56250958b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.801+0000 7f9da80cdb80 1 -- 172.21.15.70:0/2582098842 >> 172.21.15.70:0/2582098842 conn(0x5625094ad240 msgr2=0x56250957baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:47.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.801+0000 7f9da80cdb80 1 -- 172.21.15.70:0/2582098842 shutdown_connections 2024-06-04T05:09:47.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.801+0000 7f9da80cdb80 1 -- 172.21.15.70:0/2582098842 wait complete. 2024-06-04T05:09:47.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.801+0000 7f9da80cdb80 20 client.18795 trim_cache size 0 max 16384 2024-06-04T05:09:47.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.816+0000 7f9da80cdb80 1 Processor -- start 2024-06-04T05:09:47.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.816+0000 7f9da80cdb80 1 -- start start 2024-06-04T05:09:47.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.816+0000 7f9da80cdb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625095818a0 0x56250957c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.816+0000 7f9da80cdb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509582260 0x562509588010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.816+0000 7f9da80cdb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562509587320 0x562509588550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.816+0000 7f9da80cdb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d981b0170 con 0x5625095818a0 2024-06-04T05:09:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.816+0000 7f9da80cdb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9d981b1dd0 con 0x562509582260 2024-06-04T05:09:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.816+0000 7f9da80cdb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9d981b0840 con 0x562509587320 2024-06-04T05:09:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.816+0000 7f9d96ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625095818a0 0x56250957c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.817+0000 7f9d96ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625095818a0 0x56250957c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46180/0 (socket says 172.21.15.70:46180) 2024-06-04T05:09:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.817+0000 7f9d96ffd700 1 -- 172.21.15.70:0/1917228250 learned_addr learned my addr 172.21.15.70:0/1917228250 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.817+0000 7f9d977fe700 1 --2- 172.21.15.70:0/1917228250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562509587320 0x562509588550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.817+0000 7f9d967fc700 1 --2- 172.21.15.70:0/1917228250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509582260 0x562509588010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.817+0000 7f9d96ffd700 1 -- 172.21.15.70:0/1917228250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562509587320 msgr2=0x562509588550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.817+0000 7f9d96ffd700 1 --2- 172.21.15.70:0/1917228250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562509587320 0x562509588550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.817+0000 7f9d96ffd700 1 -- 172.21.15.70:0/1917228250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509582260 msgr2=0x562509588010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.817+0000 7f9d96ffd700 1 --2- 172.21.15.70:0/1917228250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509582260 0x562509588010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.817+0000 7f9d96ffd700 1 -- 172.21.15.70:0/1917228250 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562509589630 con 0x5625095818a0 2024-06-04T05:09:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.817+0000 7f9d96ffd700 1 --2- 172.21.15.70:0/1917228250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625095818a0 0x56250957c270 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f9d90007b50 tx=0x7f9d900099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581597210294f37d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:47.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.817+0000 7f9d8d7fa700 1 -- 172.21.15.70:0/1917228250 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d90029560 con 0x5625095818a0 2024-06-04T05:09:47.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.817+0000 7f9d8d7fa700 1 -- 172.21.15.70:0/1917228250 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d90040070 con 0x5625095818a0 2024-06-04T05:09:47.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.817+0000 7f9d8d7fa700 1 -- 172.21.15.70:0/1917228250 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9003ba00 con 0x5625095818a0 2024-06-04T05:09:47.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.818+0000 7f9da80cdb80 1 -- 172.21.15.70:0/1917228250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625095818a0 msgr2=0x56250957c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:47.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.818+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/1917228250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625095818a0 0x56250957c270 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f9d90007b50 tx=0x7f9d900099e0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.818+0000 7f9da80cdb80 1 -- 172.21.15.70:0/1917228250 shutdown_connections 2024-06-04T05:09:47.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.818+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/1917228250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562509587320 0x562509588550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.818+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/1917228250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625095818a0 0x56250957c270 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.818+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/1917228250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509582260 0x562509588010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.818+0000 7f9da80cdb80 1 -- 172.21.15.70:0/1917228250 >> 172.21.15.70:0/1917228250 conn(0x5625094ad240 msgr2=0x5625095855e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:47.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.818+0000 7f9da80cdb80 1 -- 172.21.15.70:0/1917228250 shutdown_connections 2024-06-04T05:09:47.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.818+0000 7f9da80cdb80 1 -- 172.21.15.70:0/1917228250 wait complete. 2024-06-04T05:09:47.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.818+0000 7f9da80cdb80 1 -- start start 2024-06-04T05:09:47.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.818+0000 7f9da80cdb80 1 Processor -- start 2024-06-04T05:09:47.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9da80cdb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625095818a0 0x562509593ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9da80cdb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509582260 0x5625095a2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9da80cdb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509587320 0x5625095a2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9da80cdb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d981b16a0 con 0x562509582260 2024-06-04T05:09:47.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9da80cdb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9d981b0f70 con 0x562509587320 2024-06-04T05:09:47.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9da80cdb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9d981ae130 con 0x5625095818a0 2024-06-04T05:09:47.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9da80cdb80 10 client.0 _ll_register_callbacks cb 0x56250946a950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:47.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9d967fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509582260 0x5625095a2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:47.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9d967fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509582260 0x5625095a2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46192/0 (socket says 172.21.15.70:46192) 2024-06-04T05:09:47.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9d967fc700 1 -- 172.21.15.70:0/2734675381 learned_addr learned my addr 172.21.15.70:0/2734675381 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:47.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9d96ffd700 1 --2- 172.21.15.70:0/2734675381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625095818a0 0x562509593ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:47.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9d977fe700 1 --2- 172.21.15.70:0/2734675381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509587320 0x5625095a2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:47.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9d967fc700 1 -- 172.21.15.70:0/2734675381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625095818a0 msgr2=0x562509593ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:47.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9d967fc700 1 --2- 172.21.15.70:0/2734675381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625095818a0 0x562509593ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9d967fc700 1 -- 172.21.15.70:0/2734675381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509587320 msgr2=0x5625095a2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:47.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9d967fc700 1 --2- 172.21.15.70:0/2734675381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509587320 0x5625095a2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9d967fc700 1 -- 172.21.15.70:0/2734675381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625095a3290 con 0x562509582260 2024-06-04T05:09:47.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9d967fc700 1 --2- 172.21.15.70:0/2734675381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509582260 0x5625095a2690 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f9d800df650 tx=0x7f9d800a8d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2f4ce25807e88 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:47.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9d76ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:47.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9d76ffd700 1 -- 172.21.15.70:0/2734675381 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d800c3060 con 0x562509582260 2024-06-04T05:09:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9d76ffd700 1 -- 172.21.15.70:0/2734675381 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f9d800b1b50 con 0x562509582260 2024-06-04T05:09:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.819+0000 7f9da80cdb80 1 -- 172.21.15.70:0/2734675381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d90040070 con 0x562509582260 2024-06-04T05:09:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.820+0000 7f9d76ffd700 1 -- 172.21.15.70:0/2734675381 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f9d800ca050 con 0x562509582260 2024-06-04T05:09:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.820+0000 7f9da80cdb80 10 client.18801 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.820+0000 7f9da80cdb80 1 -- 172.21.15.70:0/2734675381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x562509605d60 con 0x562509582260 2024-06-04T05:09:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.820+0000 7f9d76ffd700 1 -- 172.21.15.70:0/2734675381 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d90040070 con 0x562509582260 2024-06-04T05:09:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.820+0000 7f9d76ffd700 1 -- 172.21.15.70:0/2734675381 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f9d800a83e0 con 0x562509582260 2024-06-04T05:09:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.820+0000 7f9da80cdb80 10 client.18801 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.820+0000 7f9da80cdb80 10 client.18801 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:09:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.820+0000 7f9da80cdb80 20 client.18801 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.820+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/2734675381 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56250959b260 0x562509608490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.820+0000 7f9da80cdb80 4 client.18801 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:09:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.820+0000 7f9da80cdb80 1 -- 172.21.15.70:0/2734675381 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5625094db4a0 con 0x56250959b260 2024-06-04T05:09:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.820+0000 7f9d96ffd700 1 --2- 172.21.15.70:0/2734675381 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56250959b260 0x562509608490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.821+0000 7f9d96ffd700 1 --2- 172.21.15.70:0/2734675381 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56250959b260 0x562509608490 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=65ad51f6150c43a6 server_cookie=e6d90fb4cdcba69a in_seq=0 out_seq=0 2024-06-04T05:09:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.821+0000 7f9d76ffd700 10 client.18801 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:09:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.821+0000 7f9d76ffd700 1 -- 172.21.15.70:0/2734675381 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5625094db4a0 con 0x56250959b260 2024-06-04T05:09:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.821+0000 7f9d76ffd700 10 client.18801 handle_command_reply: tid=0 2024-06-04T05:09:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.821+0000 7f9da80cdb80 1 client.18801 shutdown 2024-06-04T05:09:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.821+0000 7f9da80cdb80 1 -- 172.21.15.70:0/2734675381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509582260 msgr2=0x5625095a2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.821+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/2734675381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509582260 0x5625095a2690 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f9d800df650 tx=0x7f9d800a8d60 comp rx=0 tx=0).stop 2024-06-04T05:09:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.821+0000 7f9da80cdb80 1 -- 172.21.15.70:0/2734675381 shutdown_connections 2024-06-04T05:09:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.821+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/2734675381 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56250959b260 0x562509608490 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.821+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/2734675381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562509587320 0x5625095a2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.821+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/2734675381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625095818a0 0x562509593ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.821+0000 7f9da80cdb80 1 --2- 172.21.15.70:0/2734675381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562509582260 0x5625095a2690 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.821+0000 7f9da80cdb80 1 -- 172.21.15.70:0/2734675381 >> 172.21.15.70:0/2734675381 conn(0x5625094ad240 msgr2=0x562509580890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.821+0000 7f9d76ffd700 0 client.18801 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:09:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.821+0000 7f9da80cdb80 1 -- 172.21.15.70:0/2734675381 shutdown_connections 2024-06-04T05:09:47.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.821+0000 7f9da80cdb80 1 -- 172.21.15.70:0/2734675381 wait complete. 2024-06-04T05:09:47.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.821+0000 7f9da80cdb80 20 client.18801 trim_cache size 0 max 16384 2024-06-04T05:09:47.829 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:09:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.822+0000 7f9d9d838700 1 -- 172.21.15.70:0/2425227592 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9d840b29f0 msgr2=0x7f9d840b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.822+0000 7f9d9d838700 1 --2- 172.21.15.70:0/2425227592 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9d840b29f0 0x7f9d840b4eb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9d90009e60 tx=0x7f9d9000b040 comp rx=0 tx=0).stop 2024-06-04T05:09:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.822+0000 7f9d9d838700 1 -- 172.21.15.70:0/2425227592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d98102e30 msgr2=0x7f9d981045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.822+0000 7f9d9d838700 1 --2- 172.21.15.70:0/2425227592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d98102e30 0x7f9d981045f0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000b530 tx=0x7f9d8000b8f0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.822+0000 7f9d9d838700 1 -- 172.21.15.70:0/2425227592 shutdown_connections 2024-06-04T05:09:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.822+0000 7f9d9d838700 1 --2- 172.21.15.70:0/2425227592 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9d840b29f0 0x7f9d840b4eb0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.822+0000 7f9d9d838700 1 --2- 172.21.15.70:0/2425227592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d98108550 0x7f9d98104b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.822+0000 7f9d9d838700 1 --2- 172.21.15.70:0/2425227592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d98102470 0x7f9d981040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.822+0000 7f9d9d838700 1 --2- 172.21.15.70:0/2425227592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d98102e30 0x7f9d981045f0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.822+0000 7f9d9d838700 1 -- 172.21.15.70:0/2425227592 >> 172.21.15.70:0/2425227592 conn(0x7f9d98079480 msgr2=0x7f9d9806c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.822+0000 7f9d9d838700 1 -- 172.21.15.70:0/2425227592 shutdown_connections 2024-06-04T05:09:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:47.822+0000 7f9d9d838700 1 -- 172.21.15.70:0/2425227592 wait complete. 2024-06-04T05:09:48.500 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:09:48.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.66 MB/sec execute 521 sec latency 22032.235 ms 2024-06-04T05:09:48.779 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:09:49.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:49 smithi098 ceph-mon[35569]: pgmap v392: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-06-04T05:09:49.582 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:49 smithi070 ceph-mon[32631]: pgmap v392: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-06-04T05:09:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:49 smithi112 ceph-mon[36453]: pgmap v392: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-06-04T05:09:49.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.66 MB/sec execute 522 sec latency 23032.274 ms 2024-06-04T05:09:50.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.533+0000 7ffa0afde700 1 -- 172.21.15.70:0/1589940680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa0406b220 msgr2=0x7ffa0406b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:50.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.533+0000 7ffa0afde700 1 --2- 172.21.15.70:0/1589940680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa0406b220 0x7ffa0406b6a0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4009990 tx=0x7ff9f4009ca0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.533+0000 7ffa0afde700 1 -- 172.21.15.70:0/1589940680 shutdown_connections 2024-06-04T05:09:50.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.533+0000 7ffa0afde700 1 --2- 172.21.15.70:0/1589940680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa0406bbe0 0x7ffa041103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.533+0000 7ffa0afde700 1 --2- 172.21.15.70:0/1589940680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa041095b0 0x7ffa04109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.533+0000 7ffa0afde700 1 --2- 172.21.15.70:0/1589940680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa0406b220 0x7ffa0406b6a0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.533+0000 7ffa0afde700 1 -- 172.21.15.70:0/1589940680 >> 172.21.15.70:0/1589940680 conn(0x7ffa040793f0 msgr2=0x7ffa04079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:50.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.533+0000 7ffa0afde700 1 -- 172.21.15.70:0/1589940680 shutdown_connections 2024-06-04T05:09:50.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.533+0000 7ffa0afde700 1 -- 172.21.15.70:0/1589940680 wait complete. 2024-06-04T05:09:50.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.533+0000 7ffa0afde700 1 Processor -- start 2024-06-04T05:09:50.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ffa0afde700 1 -- start start 2024-06-04T05:09:50.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ffa0afde700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa0406b220 0x7ffa0419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ffa0afde700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa0406bbe0 0x7ffa0419fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ffa0afde700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa041095b0 0x7ffa041999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ffa0afde700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041135f0 con 0x7ffa0406bbe0 2024-06-04T05:09:50.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ffa0afde700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa04113470 con 0x7ffa0406b220 2024-06-04T05:09:50.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ffa0afde700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ffa04113770 con 0x7ffa041095b0 2024-06-04T05:09:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ff9fbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa0406bbe0 0x7ffa0419fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ff9fbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa0406bbe0 0x7ffa0419fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46210/0 (socket says 172.21.15.70:46210) 2024-06-04T05:09:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ff9fbfff700 1 -- 172.21.15.70:0/2681688010 learned_addr learned my addr 172.21.15.70:0/2681688010 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ffa08d7a700 1 --2- 172.21.15.70:0/2681688010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa0406b220 0x7ffa0419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ffa0957b700 1 --2- 172.21.15.70:0/2681688010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa041095b0 0x7ffa041999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ff9fbfff700 1 -- 172.21.15.70:0/2681688010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa041095b0 msgr2=0x7ffa041999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ff9fbfff700 1 --2- 172.21.15.70:0/2681688010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa041095b0 0x7ffa041999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ff9fbfff700 1 -- 172.21.15.70:0/2681688010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa0406b220 msgr2=0x7ffa0419f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ff9fbfff700 1 --2- 172.21.15.70:0/2681688010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa0406b220 0x7ffa0419f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ff9fbfff700 1 -- 172.21.15.70:0/2681688010 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa0419a110 con 0x7ffa0406bbe0 2024-06-04T05:09:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.534+0000 7ff9fbfff700 1 --2- 172.21.15.70:0/2681688010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa0406bbe0 0x7ffa0419fd30 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4000c00 tx=0x7ff9f40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.535+0000 7ff9f9ffb700 1 -- 172.21.15.70:0/2681688010 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4027070 con 0x7ffa0406bbe0 2024-06-04T05:09:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.535+0000 7ffa0afde700 1 -- 172.21.15.70:0/2681688010 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa0419a3a0 con 0x7ffa0406bbe0 2024-06-04T05:09:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.535+0000 7ff9f9ffb700 1 -- 172.21.15.70:0/2681688010 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff9f400fb80 con 0x7ffa0406bbe0 2024-06-04T05:09:50.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.535+0000 7ff9f9ffb700 1 -- 172.21.15.70:0/2681688010 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4022d00 con 0x7ffa0406bbe0 2024-06-04T05:09:50.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.535+0000 7ffa0afde700 1 -- 172.21.15.70:0/2681688010 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa041a66c0 con 0x7ffa0406bbe0 2024-06-04T05:09:50.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.537+0000 7ff9f9ffb700 1 -- 172.21.15.70:0/2681688010 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff9f402a070 con 0x7ffa0406bbe0 2024-06-04T05:09:50.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.538+0000 7ff9f9ffb700 1 --2- 172.21.15.70:0/2681688010 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9ec0b29f0 0x7ff9ec0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.538+0000 7ffa08d7a700 1 --2- 172.21.15.70:0/2681688010 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9ec0b29f0 0x7ff9ec0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.538+0000 7ff9f9ffb700 1 -- 172.21.15.70:0/2681688010 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7ff9f40e34d0 con 0x7ffa0406bbe0 2024-06-04T05:09:50.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.539+0000 7ffa08d7a700 1 --2- 172.21.15.70:0/2681688010 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9ec0b29f0 0x7ff9ec0b4eb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009550 tx=0x7ffa000091b0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:50.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.539+0000 7ffa15873b80 1 Processor -- start 2024-06-04T05:09:50.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.539+0000 7ffa15873b80 1 -- start start 2024-06-04T05:09:50.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.539+0000 7ffa15873b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x556766619700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.539+0000 7ffa15873b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567666138a0 0x556766613d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.539+0000 7ffa15873b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556766614260 0x556766620080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.539+0000 7ffa15873b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556766571d70 con 0x556766619320 2024-06-04T05:09:50.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.539+0000 7ffa15873b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556766561910 con 0x556766614260 2024-06-04T05:09:50.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.539+0000 7ffa15873b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556766565950 con 0x5567666138a0 2024-06-04T05:09:50.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.539+0000 7ffa08d7a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x556766619700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.539+0000 7ffa08d7a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x556766619700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46212/0 (socket says 172.21.15.70:46212) 2024-06-04T05:09:50.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.539+0000 7ffa08d7a700 1 -- 172.21.15.70:0/2313949164 learned_addr learned my addr 172.21.15.70:0/2313949164 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:50.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.540+0000 7ff9fbfff700 1 --2- 172.21.15.70:0/2313949164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567666138a0 0x556766613d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.540+0000 7ffa0957b700 1 --2- 172.21.15.70:0/2313949164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556766614260 0x556766620080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.540+0000 7ffa08d7a700 1 -- 172.21.15.70:0/2313949164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567666138a0 msgr2=0x556766613d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:50.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.540+0000 7ffa08d7a700 1 --2- 172.21.15.70:0/2313949164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567666138a0 0x556766613d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.540+0000 7ffa08d7a700 1 -- 172.21.15.70:0/2313949164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556766614260 msgr2=0x556766620080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:50.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.540+0000 7ffa08d7a700 1 --2- 172.21.15.70:0/2313949164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556766614260 0x556766620080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.540+0000 7ffa08d7a700 1 -- 172.21.15.70:0/2313949164 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55676650afb0 con 0x556766619320 2024-06-04T05:09:50.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.540+0000 7ffa0957b700 1 --2- 172.21.15.70:0/2313949164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556766614260 0x556766620080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:09:50.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.540+0000 7ffa08d7a700 1 --2- 172.21.15.70:0/2313949164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x556766619700 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7ffa00004680 tx=0x7ffa00013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ca59cc8e71f197 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:50.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.540+0000 7ff9eaffd700 1 -- 172.21.15.70:0/2313949164 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0001c050 con 0x556766619320 2024-06-04T05:09:50.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.540+0000 7ff9eaffd700 1 -- 172.21.15.70:0/2313949164 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffa00026070 con 0x556766619320 2024-06-04T05:09:50.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.540+0000 7ff9eaffd700 1 -- 172.21.15.70:0/2313949164 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa00015640 con 0x556766619320 2024-06-04T05:09:50.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.541+0000 7ffa15873b80 1 -- 172.21.15.70:0/2313949164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 msgr2=0x556766619700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:50.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.541+0000 7ffa15873b80 1 --2- 172.21.15.70:0/2313949164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x556766619700 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7ffa00004680 tx=0x7ffa00013000 comp rx=0 tx=0).stop 2024-06-04T05:09:50.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.541+0000 7ffa15873b80 1 -- 172.21.15.70:0/2313949164 shutdown_connections 2024-06-04T05:09:50.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.541+0000 7ffa15873b80 1 --2- 172.21.15.70:0/2313949164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556766614260 0x556766620080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.541+0000 7ffa15873b80 1 --2- 172.21.15.70:0/2313949164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x556766619700 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.541+0000 7ffa15873b80 1 --2- 172.21.15.70:0/2313949164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567666138a0 0x556766613d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.541+0000 7ffa15873b80 1 -- 172.21.15.70:0/2313949164 >> 172.21.15.70:0/2313949164 conn(0x55676653f240 msgr2=0x55676660e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:50.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.541+0000 7ffa15873b80 1 -- 172.21.15.70:0/2313949164 shutdown_connections 2024-06-04T05:09:50.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.541+0000 7ffa15873b80 1 -- 172.21.15.70:0/2313949164 wait complete. 2024-06-04T05:09:50.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.542+0000 7ffa15873b80 1 -- start start 2024-06-04T05:09:50.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.542+0000 7ffa15873b80 1 Processor -- start 2024-06-04T05:09:50.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.542+0000 7ffa15873b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567666138a0 0x55676661fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.542+0000 7ffa15873b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556766614260 0x55676661d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.542+0000 7ffa15873b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556766619320 0x55676661d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.542+0000 7ffa15873b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556766467fa0 con 0x5567666138a0 2024-06-04T05:09:50.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.542+0000 7ffa15873b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5567664c9da0 con 0x556766614260 2024-06-04T05:09:50.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.542+0000 7ffa15873b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5567666206e0 con 0x556766619320 2024-06-04T05:09:50.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.542+0000 7ffa08d7a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567666138a0 0x55676661fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.542+0000 7ffa08d7a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567666138a0 0x55676661fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46222/0 (socket says 172.21.15.70:46222) 2024-06-04T05:09:50.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.542+0000 7ffa08d7a700 1 -- 172.21.15.70:0/586408966 learned_addr learned my addr 172.21.15.70:0/586408966 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:50.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.542+0000 7ffa15873b80 10 client.0 _ll_register_callbacks cb 0x5567664fc950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:50.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.542+0000 7ffa0957b700 1 --2- 172.21.15.70:0/586408966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556766619320 0x55676661d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.543+0000 7ff9fbfff700 1 --2- 172.21.15.70:0/586408966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556766614260 0x55676661d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.543+0000 7ffa08d7a700 1 -- 172.21.15.70:0/586408966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556766619320 msgr2=0x55676661d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:50.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.543+0000 7ffa08d7a700 1 --2- 172.21.15.70:0/586408966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556766619320 0x55676661d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.543+0000 7ffa08d7a700 1 -- 172.21.15.70:0/586408966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556766614260 msgr2=0x55676661d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:50.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.543+0000 7ffa08d7a700 1 --2- 172.21.15.70:0/586408966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556766614260 0x55676661d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.543+0000 7ffa08d7a700 1 -- 172.21.15.70:0/586408966 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567664b18d0 con 0x5567666138a0 2024-06-04T05:09:50.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.543+0000 7ffa08d7a700 1 --2- 172.21.15.70:0/586408966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567666138a0 0x55676661fce0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7ffa00021770 tx=0x7ffa000217a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99ea1acd49655153 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:50.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.543+0000 7ff9e3fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:50.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.543+0000 7ff9e3fff700 1 -- 172.21.15.70:0/586408966 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0002f070 con 0x5567666138a0 2024-06-04T05:09:50.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.543+0000 7ffa15873b80 1 -- 172.21.15.70:0/586408966 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa00026070 con 0x5567666138a0 2024-06-04T05:09:50.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.543+0000 7ff9e3fff700 1 -- 172.21.15.70:0/586408966 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7ffa00015640 con 0x5567666138a0 2024-06-04T05:09:50.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.543+0000 7ff9e3fff700 1 -- 172.21.15.70:0/586408966 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ffa00015840 con 0x5567666138a0 2024-06-04T05:09:50.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.543+0000 7ffa15873b80 10 client.18813 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:50.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.543+0000 7ffa15873b80 1 -- 172.21.15.70:0/586408966 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55676651b760 con 0x5567666138a0 2024-06-04T05:09:50.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.543+0000 7ff9e3fff700 1 -- 172.21.15.70:0/586408966 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffa00026070 con 0x5567666138a0 2024-06-04T05:09:50.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.544+0000 7ff9e3fff700 1 -- 172.21.15.70:0/586408966 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7ffa0003a070 con 0x5567666138a0 2024-06-04T05:09:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.544+0000 7ffa15873b80 10 client.18813 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.544+0000 7ffa15873b80 10 client.18813 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:09:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.544+0000 7ffa15873b80 20 client.18813 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.544+0000 7ffa15873b80 1 --2- 172.21.15.70:0/586408966 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55676662d090 0x55676662f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.544+0000 7ffa15873b80 4 client.18813 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:09:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.544+0000 7ffa15873b80 1 -- 172.21.15.70:0/586408966 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556766573cc0 con 0x55676662d090 2024-06-04T05:09:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.544+0000 7ff9fbfff700 1 --2- 172.21.15.70:0/586408966 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55676662d090 0x55676662f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.545+0000 7ff9e3fff700 10 client.18813 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:09:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.545+0000 7ff9fbfff700 1 --2- 172.21.15.70:0/586408966 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55676662d090 0x55676662f480 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f21353bfdd753ff2 server_cookie=68b182a0192955e2 in_seq=0 out_seq=0 2024-06-04T05:09:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.545+0000 7ff9e3fff700 1 -- 172.21.15.70:0/586408966 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556766573cc0 con 0x55676662d090 2024-06-04T05:09:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.545+0000 7ff9e3fff700 10 client.18813 handle_command_reply: tid=0 2024-06-04T05:09:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.545+0000 7ffa15873b80 1 client.18813 shutdown 2024-06-04T05:09:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.545+0000 7ffa15873b80 1 -- 172.21.15.70:0/586408966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567666138a0 msgr2=0x55676661fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.545+0000 7ffa15873b80 1 --2- 172.21.15.70:0/586408966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567666138a0 0x55676661fce0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7ffa00021770 tx=0x7ffa000217a0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.545+0000 7ffa15873b80 1 -- 172.21.15.70:0/586408966 shutdown_connections 2024-06-04T05:09:50.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.545+0000 7ffa15873b80 1 --2- 172.21.15.70:0/586408966 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55676662d090 0x55676662f480 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.545+0000 7ffa15873b80 1 --2- 172.21.15.70:0/586408966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556766619320 0x55676661d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.545+0000 7ffa15873b80 1 --2- 172.21.15.70:0/586408966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567666138a0 0x55676661fce0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.545+0000 7ffa15873b80 1 --2- 172.21.15.70:0/586408966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556766614260 0x55676661d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.546+0000 7ff9e3fff700 0 client.18813 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:09:50.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.546+0000 7ffa15873b80 1 -- 172.21.15.70:0/586408966 >> 172.21.15.70:0/586408966 conn(0x55676653f240 msgr2=0x55676660daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:50.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.546+0000 7ffa15873b80 1 -- 172.21.15.70:0/586408966 shutdown_connections 2024-06-04T05:09:50.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.546+0000 7ffa15873b80 1 -- 172.21.15.70:0/586408966 wait complete. 2024-06-04T05:09:50.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.546+0000 7ffa15873b80 20 client.18813 trim_cache size 0 max 16384 2024-06-04T05:09:50.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa15873b80 1 Processor -- start 2024-06-04T05:09:50.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa15873b80 1 -- start start 2024-06-04T05:09:50.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa15873b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567666138a0 0x55676661b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa15873b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556766614260 0x55676661bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa15873b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x55676661c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa15873b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041adce0 con 0x556766619320 2024-06-04T05:09:50.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa15873b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041af940 con 0x556766614260 2024-06-04T05:09:50.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa15873b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041ae3b0 con 0x5567666138a0 2024-06-04T05:09:50.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa0957b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x55676661c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa0957b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x55676661c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46232/0 (socket says 172.21.15.70:46232) 2024-06-04T05:09:50.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa0957b700 1 -- 172.21.15.70:0/4044941330 learned_addr learned my addr 172.21.15.70:0/4044941330 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:50.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa08d7a700 1 --2- 172.21.15.70:0/4044941330 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567666138a0 0x55676661b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa0957b700 1 -- 172.21.15.70:0/4044941330 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567666138a0 msgr2=0x55676661b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:50.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa0957b700 1 --2- 172.21.15.70:0/4044941330 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567666138a0 0x55676661b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa0957b700 1 -- 172.21.15.70:0/4044941330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556766614260 msgr2=0x55676661bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:50.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ff9fbfff700 1 --2- 172.21.15.70:0/4044941330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556766614260 0x55676661bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa0957b700 1 --2- 172.21.15.70:0/4044941330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556766614260 0x55676661bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa0957b700 1 -- 172.21.15.70:0/4044941330 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556766618f40 con 0x556766619320 2024-06-04T05:09:50.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa08d7a700 1 --2- 172.21.15.70:0/4044941330 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567666138a0 0x55676661b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:09:50.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.562+0000 7ffa0957b700 1 --2- 172.21.15.70:0/4044941330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x55676661c270 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc009870 tx=0x7ff9fc009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f1503b1fcfa7c0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:50.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.563+0000 7ff9ea7fc700 1 -- 172.21.15.70:0/4044941330 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc019070 con 0x556766619320 2024-06-04T05:09:50.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.563+0000 7ff9ea7fc700 1 -- 172.21.15.70:0/4044941330 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff9fc011bf0 con 0x556766619320 2024-06-04T05:09:50.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.563+0000 7ff9ea7fc700 1 -- 172.21.15.70:0/4044941330 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc00f040 con 0x556766619320 2024-06-04T05:09:50.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.563+0000 7ffa15873b80 1 -- 172.21.15.70:0/4044941330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 msgr2=0x55676661c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:50.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.563+0000 7ffa15873b80 1 --2- 172.21.15.70:0/4044941330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x55676661c270 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc009870 tx=0x7ff9fc009b80 comp rx=0 tx=0).stop 2024-06-04T05:09:50.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.563+0000 7ffa15873b80 1 -- 172.21.15.70:0/4044941330 shutdown_connections 2024-06-04T05:09:50.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.563+0000 7ffa15873b80 1 --2- 172.21.15.70:0/4044941330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x55676661c270 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.563+0000 7ffa15873b80 1 --2- 172.21.15.70:0/4044941330 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567666138a0 0x55676661b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.563+0000 7ffa15873b80 1 --2- 172.21.15.70:0/4044941330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556766614260 0x55676661bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.563+0000 7ffa15873b80 1 -- 172.21.15.70:0/4044941330 >> 172.21.15.70:0/4044941330 conn(0x55676653f240 msgr2=0x556766613120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:50.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.563+0000 7ffa15873b80 1 -- 172.21.15.70:0/4044941330 shutdown_connections 2024-06-04T05:09:50.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.563+0000 7ffa15873b80 1 -- 172.21.15.70:0/4044941330 wait complete. 2024-06-04T05:09:50.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.564+0000 7ffa15873b80 1 -- start start 2024-06-04T05:09:50.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.564+0000 7ffa15873b80 1 Processor -- start 2024-06-04T05:09:50.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.564+0000 7ffa15873b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567666138a0 0x556766625d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.564+0000 7ffa15873b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556766614260 0x556766633ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.564+0000 7ffa15873b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x556766634530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.564+0000 7ffa15873b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041af210 con 0x556766619320 2024-06-04T05:09:50.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.564+0000 7ffa15873b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041aeae0 con 0x5567666138a0 2024-06-04T05:09:50.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.564+0000 7ffa15873b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041abca0 con 0x556766614260 2024-06-04T05:09:50.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.564+0000 7ffa15873b80 10 client.0 _ll_register_callbacks cb 0x5567664fc950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:50.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.564+0000 7ffa0957b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x556766634530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.564+0000 7ffa0957b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x556766634530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46246/0 (socket says 172.21.15.70:46246) 2024-06-04T05:09:50.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.564+0000 7ffa0957b700 1 -- 172.21.15.70:0/1819761240 learned_addr learned my addr 172.21.15.70:0/1819761240 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:50.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.564+0000 7ffa08d7a700 1 --2- 172.21.15.70:0/1819761240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567666138a0 0x556766625d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.565+0000 7ff9fbfff700 1 --2- 172.21.15.70:0/1819761240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556766614260 0x556766633ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.565+0000 7ffa0957b700 1 -- 172.21.15.70:0/1819761240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556766614260 msgr2=0x556766633ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:50.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.565+0000 7ffa0957b700 1 --2- 172.21.15.70:0/1819761240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556766614260 0x556766633ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.565+0000 7ffa0957b700 1 -- 172.21.15.70:0/1819761240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567666138a0 msgr2=0x556766625d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:50.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.565+0000 7ffa0957b700 1 --2- 172.21.15.70:0/1819761240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567666138a0 0x556766625d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.565+0000 7ffa0957b700 1 -- 172.21.15.70:0/1819761240 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556766634a70 con 0x556766619320 2024-06-04T05:09:50.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.565+0000 7ff9fbfff700 1 --2- 172.21.15.70:0/1819761240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556766614260 0x556766633ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:09:50.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.565+0000 7ffa08d7a700 1 --2- 172.21.15.70:0/1819761240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567666138a0 0x556766625d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:50.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.565+0000 7ffa0957b700 1 --2- 172.21.15.70:0/1819761240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x556766634530 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc0117f0 tx=0x7ff9fc011820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f71b590e6c7df94c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:50.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.565+0000 7ff9e3fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:50.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.565+0000 7ff9e3fff700 1 -- 172.21.15.70:0/1819761240 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc024070 con 0x556766619320 2024-06-04T05:09:50.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.565+0000 7ff9e3fff700 1 -- 172.21.15.70:0/1819761240 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7ff9fc015ae0 con 0x556766619320 2024-06-04T05:09:50.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.565+0000 7ffa15873b80 1 -- 172.21.15.70:0/1819761240 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9fc011bf0 con 0x556766619320 2024-06-04T05:09:50.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.565+0000 7ff9e3fff700 1 -- 172.21.15.70:0/1819761240 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff9fc019040 con 0x556766619320 2024-06-04T05:09:50.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.565+0000 7ffa15873b80 10 client.18819 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:50.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.565+0000 7ffa15873b80 1 -- 172.21.15.70:0/1819761240 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556766625520 con 0x556766619320 2024-06-04T05:09:50.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.565+0000 7ff9e3fff700 1 -- 172.21.15.70:0/1819761240 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff9fc011bf0 con 0x556766619320 2024-06-04T05:09:50.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.566+0000 7ff9e3fff700 1 -- 172.21.15.70:0/1819761240 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7ff9fc01ec80 con 0x556766619320 2024-06-04T05:09:50.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.566+0000 7ffa15873b80 10 client.18819 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:50.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.566+0000 7ffa15873b80 10 client.18819 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:09:50.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.566+0000 7ffa15873b80 20 client.18819 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:50.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.566+0000 7ffa15873b80 1 --2- 172.21.15.70:0/1819761240 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x556766697b30 0x556766699f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:50.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.566+0000 7ffa15873b80 4 client.18819 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:09:50.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.566+0000 7ffa15873b80 1 -- 172.21.15.70:0/1819761240 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55676656d4a0 con 0x556766697b30 2024-06-04T05:09:50.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.566+0000 7ffa08d7a700 1 --2- 172.21.15.70:0/1819761240 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x556766697b30 0x556766699f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:50.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.566+0000 7ffa08d7a700 1 --2- 172.21.15.70:0/1819761240 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x556766697b30 0x556766699f50 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cca9a734799d9519 server_cookie=e381c6daac490629 in_seq=0 out_seq=0 2024-06-04T05:09:50.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.566+0000 7ff9e3fff700 10 client.18819 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:09:50.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.567+0000 7ff9e3fff700 1 -- 172.21.15.70:0/1819761240 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55676656d4a0 con 0x556766697b30 2024-06-04T05:09:50.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.567+0000 7ff9e3fff700 10 client.18819 handle_command_reply: tid=0 2024-06-04T05:09:50.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.567+0000 7ffa15873b80 1 client.18819 shutdown 2024-06-04T05:09:50.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.567+0000 7ffa15873b80 1 -- 172.21.15.70:0/1819761240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 msgr2=0x556766634530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:50.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.567+0000 7ffa15873b80 1 --2- 172.21.15.70:0/1819761240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x556766634530 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc0117f0 tx=0x7ff9fc011820 comp rx=0 tx=0).stop 2024-06-04T05:09:50.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.567+0000 7ffa15873b80 1 -- 172.21.15.70:0/1819761240 shutdown_connections 2024-06-04T05:09:50.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.567+0000 7ffa15873b80 1 --2- 172.21.15.70:0/1819761240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556766619320 0x556766634530 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.567+0000 7ffa15873b80 1 --2- 172.21.15.70:0/1819761240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567666138a0 0x556766625d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.567+0000 7ffa15873b80 1 --2- 172.21.15.70:0/1819761240 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x556766697b30 0x556766699f50 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.567+0000 7ffa15873b80 1 --2- 172.21.15.70:0/1819761240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556766614260 0x556766633ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.567+0000 7ffa15873b80 1 -- 172.21.15.70:0/1819761240 >> 172.21.15.70:0/1819761240 conn(0x55676653f240 msgr2=0x556766616ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:50.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.567+0000 7ff9e3fff700 0 client.18819 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:09:50.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.567+0000 7ffa15873b80 1 -- 172.21.15.70:0/1819761240 shutdown_connections 2024-06-04T05:09:50.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.567+0000 7ffa15873b80 1 -- 172.21.15.70:0/1819761240 wait complete. 2024-06-04T05:09:50.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.567+0000 7ffa15873b80 20 client.18819 trim_cache size 0 max 16384 2024-06-04T05:09:50.576 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:09:50.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.567+0000 7ffa0afde700 1 -- 172.21.15.70:0/2681688010 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9ec0b29f0 msgr2=0x7ff9ec0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:50.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.568+0000 7ffa0afde700 1 --2- 172.21.15.70:0/2681688010 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9ec0b29f0 0x7ff9ec0b4eb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009550 tx=0x7ffa000091b0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.568+0000 7ffa0afde700 1 -- 172.21.15.70:0/2681688010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa0406bbe0 msgr2=0x7ffa0419fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:50.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.568+0000 7ffa0afde700 1 --2- 172.21.15.70:0/2681688010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa0406bbe0 0x7ffa0419fd30 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4000c00 tx=0x7ff9f40047b0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.568+0000 7ffa0afde700 1 -- 172.21.15.70:0/2681688010 shutdown_connections 2024-06-04T05:09:50.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.568+0000 7ffa0afde700 1 --2- 172.21.15.70:0/2681688010 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9ec0b29f0 0x7ff9ec0b4eb0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.568+0000 7ffa0afde700 1 --2- 172.21.15.70:0/2681688010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa041095b0 0x7ffa041999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.568+0000 7ffa0afde700 1 --2- 172.21.15.70:0/2681688010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa0406b220 0x7ffa0419f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.568+0000 7ffa0afde700 1 --2- 172.21.15.70:0/2681688010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa0406bbe0 0x7ffa0419fd30 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:50.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.568+0000 7ffa0afde700 1 -- 172.21.15.70:0/2681688010 >> 172.21.15.70:0/2681688010 conn(0x7ffa040793f0 msgr2=0x7ffa041012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:50.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.568+0000 7ffa0afde700 1 -- 172.21.15.70:0/2681688010 shutdown_connections 2024-06-04T05:09:50.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:50.568+0000 7ffa0afde700 1 -- 172.21.15.70:0/2681688010 wait complete. 2024-06-04T05:09:50.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.66 MB/sec execute 523 sec latency 24032.342 ms 2024-06-04T05:09:50.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:09:51.453 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:09:51.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:51 smithi098 ceph-mon[35569]: pgmap v393: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-06-04T05:09:51.517 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:09:51.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:51 smithi070 ceph-mon[32631]: pgmap v393: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-06-04T05:09:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:51 smithi112 ceph-mon[36453]: pgmap v393: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-06-04T05:09:51.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.65 MB/sec execute 524 sec latency 25032.388 ms 2024-06-04T05:09:52.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.65 MB/sec execute 525 sec latency 26032.480 ms 2024-06-04T05:09:53.263 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:53 smithi070 ceph-mon[32631]: pgmap v394: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-06-04T05:09:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.366+0000 7f4a69a68700 1 -- 172.21.15.70:0/1194049966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a641035e0 msgr2=0x7f4a64103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.366+0000 7f4a69a68700 1 --2- 172.21.15.70:0/1194049966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a641035e0 0x7f4a64103a60 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f4a580099c0 tx=0x7f4a58009cd0 comp rx=0 tx=0).stop 2024-06-04T05:09:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.367+0000 7f4a69a68700 1 -- 172.21.15.70:0/1194049966 shutdown_connections 2024-06-04T05:09:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.367+0000 7f4a69a68700 1 --2- 172.21.15.70:0/1194049966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4a64103fa0 0x7f4a64110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.367+0000 7f4a69a68700 1 --2- 172.21.15.70:0/1194049966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a64109680 0x7f4a64109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.367+0000 7f4a69a68700 1 --2- 172.21.15.70:0/1194049966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a641035e0 0x7f4a64103a60 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.367+0000 7f4a69a68700 1 -- 172.21.15.70:0/1194049966 >> 172.21.15.70:0/1194049966 conn(0x7f4a6407a6f0 msgr2=0x7f4a6407ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.367+0000 7f4a69a68700 1 -- 172.21.15.70:0/1194049966 shutdown_connections 2024-06-04T05:09:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.367+0000 7f4a69a68700 1 -- 172.21.15.70:0/1194049966 wait complete. 2024-06-04T05:09:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.367+0000 7f4a69a68700 1 Processor -- start 2024-06-04T05:09:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a69a68700 1 -- start start 2024-06-04T05:09:53.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a69a68700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a641035e0 0x7f4a6419f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:53.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a69a68700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a64103fa0 0x7f4a6419fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:53.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a69a68700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4a64109680 0x7f4a641998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:53.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a69a68700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4a64113690 con 0x7f4a64103fa0 2024-06-04T05:09:53.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a69a68700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4a64113510 con 0x7f4a641035e0 2024-06-04T05:09:53.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a69a68700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4a64113810 con 0x7f4a64109680 2024-06-04T05:09:53.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a62ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a64103fa0 0x7f4a6419fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:53.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a62ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a64103fa0 0x7f4a6419fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46260/0 (socket says 172.21.15.70:46260) 2024-06-04T05:09:53.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a62ffd700 1 -- 172.21.15.70:0/1118394784 learned_addr learned my addr 172.21.15.70:0/1118394784 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:53.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a637fe700 1 --2- 172.21.15.70:0/1118394784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a641035e0 0x7f4a6419f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:53.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a63fff700 1 --2- 172.21.15.70:0/1118394784 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4a64109680 0x7f4a641998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:53.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a637fe700 1 -- 172.21.15.70:0/1118394784 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4a64109680 msgr2=0x7f4a641998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:53.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a637fe700 1 --2- 172.21.15.70:0/1118394784 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4a64109680 0x7f4a641998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:53.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a637fe700 1 -- 172.21.15.70:0/1118394784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a64103fa0 msgr2=0x7f4a6419fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:53.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a637fe700 1 --2- 172.21.15.70:0/1118394784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a64103fa0 0x7f4a6419fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:53.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a637fe700 1 -- 172.21.15.70:0/1118394784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a6419a010 con 0x7f4a641035e0 2024-06-04T05:09:53.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a63fff700 1 --2- 172.21.15.70:0/1118394784 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4a64109680 0x7f4a641998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:53.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.368+0000 7f4a62ffd700 1 --2- 172.21.15.70:0/1118394784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a64103fa0 0x7f4a6419fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:53.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.369+0000 7f4a637fe700 1 --2- 172.21.15.70:0/1118394784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a641035e0 0x7f4a6419f6f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4a5400d950 tx=0x7f4a5400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:53.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.369+0000 7f4a60ff9700 1 -- 172.21.15.70:0/1118394784 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a54016070 con 0x7f4a641035e0 2024-06-04T05:09:53.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.369+0000 7f4a60ff9700 1 -- 172.21.15.70:0/1118394784 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4a5400fd20 con 0x7f4a641035e0 2024-06-04T05:09:53.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.369+0000 7f4a69a68700 1 -- 172.21.15.70:0/1118394784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a6419a300 con 0x7f4a641035e0 2024-06-04T05:09:53.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.369+0000 7f4a60ff9700 1 -- 172.21.15.70:0/1118394784 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5400e3b0 con 0x7f4a641035e0 2024-06-04T05:09:53.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.369+0000 7f4a69a68700 1 -- 172.21.15.70:0/1118394784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a641a65c0 con 0x7f4a641035e0 2024-06-04T05:09:53.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.370+0000 7f4a69a68700 1 -- 172.21.15.70:0/1118394784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a64104cf0 con 0x7f4a641035e0 2024-06-04T05:09:53.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.372+0000 7f4a60ff9700 1 -- 172.21.15.70:0/1118394784 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f4a54048d10 con 0x7f4a641035e0 2024-06-04T05:09:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.373+0000 7f4a60ff9700 1 --2- 172.21.15.70:0/1118394784 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4a4c0b29f0 0x7f4a4c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.373+0000 7f4a62ffd700 1 --2- 172.21.15.70:0/1118394784 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4a4c0b29f0 0x7f4a4c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.373+0000 7f4a60ff9700 1 -- 172.21.15.70:0/1118394784 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f4a540e2490 con 0x7f4a641035e0 2024-06-04T05:09:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.373+0000 7f4a62ffd700 1 --2- 172.21.15.70:0/1118394784 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4a4c0b29f0 0x7f4a4c0b4eb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4a6407ad60 tx=0x7f4a5801a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:53.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.376+0000 7f4a60ff9700 1 -- 172.21.15.70:0/1118394784 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4a540a9de0 con 0x7f4a641035e0 2024-06-04T05:09:53.406 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:09:53.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:53 smithi098 ceph-mon[35569]: pgmap v394: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-06-04T05:09:53.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.640+0000 7f4a69a68700 1 -- 172.21.15.70:0/1118394784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a6419ade0 con 0x7f4a641035e0 2024-06-04T05:09:53.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.641+0000 7f4a60ff9700 1 -- 172.21.15.70:0/1118394784 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11938 (secure 0 0 0) 0x7f4a540a96e0 con 0x7f4a641035e0 2024-06-04T05:09:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:09:53.644 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:09:17.032462+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24436,24410,24440,24448,24442]},"id":1}]} 2024-06-04T05:09:53.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.644+0000 7f4a69a68700 1 -- 172.21.15.70:0/1118394784 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4a4c0b29f0 msgr2=0x7f4a4c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:53.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.644+0000 7f4a69a68700 1 --2- 172.21.15.70:0/1118394784 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4a4c0b29f0 0x7f4a4c0b4eb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4a6407ad60 tx=0x7f4a5801a040 comp rx=0 tx=0).stop 2024-06-04T05:09:53.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.644+0000 7f4a69a68700 1 -- 172.21.15.70:0/1118394784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a641035e0 msgr2=0x7f4a6419f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:53.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.644+0000 7f4a69a68700 1 --2- 172.21.15.70:0/1118394784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a641035e0 0x7f4a6419f6f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4a5400d950 tx=0x7f4a5400dc60 comp rx=0 tx=0).stop 2024-06-04T05:09:53.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.644+0000 7f4a69a68700 1 -- 172.21.15.70:0/1118394784 shutdown_connections 2024-06-04T05:09:53.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.644+0000 7f4a69a68700 1 --2- 172.21.15.70:0/1118394784 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4a4c0b29f0 0x7f4a4c0b4eb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:53.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.644+0000 7f4a69a68700 1 --2- 172.21.15.70:0/1118394784 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4a64109680 0x7f4a641998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:53.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.644+0000 7f4a69a68700 1 --2- 172.21.15.70:0/1118394784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a641035e0 0x7f4a6419f6f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:53.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.644+0000 7f4a69a68700 1 --2- 172.21.15.70:0/1118394784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a64103fa0 0x7f4a6419fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:53.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.644+0000 7f4a69a68700 1 -- 172.21.15.70:0/1118394784 >> 172.21.15.70:0/1118394784 conn(0x7f4a6407a6f0 msgr2=0x7f4a641022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:53.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.644+0000 7f4a69a68700 1 -- 172.21.15.70:0/1118394784 shutdown_connections 2024-06-04T05:09:53.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:53.644+0000 7f4a69a68700 1 -- 172.21.15.70:0/1118394784 wait complete. 2024-06-04T05:09:53.648 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 38 2024-06-04T05:09:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:53 smithi112 ceph-mon[36453]: pgmap v394: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-06-04T05:09:53.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.65 MB/sec execute 526 sec latency 27032.542 ms 2024-06-04T05:09:54.424 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id a5e14b41 --release --await' 2024-06-04T05:09:54.424 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id a5e14b41 --release --await 2024-06-04T05:09:54.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:54 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1118394784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:09:54.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:54 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1118394784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:09:54.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.64 MB/sec execute 527 sec latency 28032.575 ms 2024-06-04T05:09:54.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:54 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1118394784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:09:55.337 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:55 smithi070 ceph-mon[32631]: pgmap v395: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-06-04T05:09:55.363 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:09:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:55 smithi112 ceph-mon[36453]: pgmap v395: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-06-04T05:09:55.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.64 MB/sec execute 528 sec latency 29032.634 ms 2024-06-04T05:09:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:55 smithi098 ceph-mon[35569]: pgmap v395: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-06-04T05:09:56.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.665+0000 7f1405958640 1 -- 172.21.15.70:0/1418989859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14001022f0 msgr2=0x7f140010e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:56.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.665+0000 7f1405958640 1 --2- 172.21.15.70:0/1418989859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14001022f0 0x7f140010e800 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f13f8009cd0 tx=0x7f13f802a8e0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.665+0000 7f1405958640 1 -- 172.21.15.70:0/1418989859 shutdown_connections 2024-06-04T05:09:56.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.665+0000 7f1405958640 1 --2- 172.21.15.70:0/1418989859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14001022f0 0x7f140010e800 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.665+0000 7f1405958640 1 --2- 172.21.15.70:0/1418989859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1400101930 0x7f1400101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.665+0000 7f1405958640 1 --2- 172.21.15.70:0/1418989859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1400107940 0x7f1400107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.665+0000 7f1405958640 1 -- 172.21.15.70:0/1418989859 >> 172.21.15.70:0/1418989859 conn(0x7f14000fd420 msgr2=0x7f14000ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:56.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.665+0000 7f1405958640 1 -- 172.21.15.70:0/1418989859 shutdown_connections 2024-06-04T05:09:56.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.665+0000 7f1405958640 1 -- 172.21.15.70:0/1418989859 wait complete. 2024-06-04T05:09:56.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.665+0000 7f1405958640 1 Processor -- start 2024-06-04T05:09:56.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f1405958640 1 -- start start 2024-06-04T05:09:56.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f1405958640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1400101930 0x7f1400199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f1405958640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14001022f0 0x7f1400199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f1405958640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1400107940 0x7f140019dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f1405958640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1400111ad0 con 0x7f1400107940 2024-06-04T05:09:56.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f1405958640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1400111950 con 0x7f14001022f0 2024-06-04T05:09:56.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f1405958640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1400111c50 con 0x7f1400101930 2024-06-04T05:09:56.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f13ff7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1400107940 0x7f140019dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f13ff7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1400107940 0x7f140019dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50854/0 (socket says 172.21.15.70:50854) 2024-06-04T05:09:56.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f13ff7fe640 1 -- 172.21.15.70:0/718040810 learned_addr learned my addr 172.21.15.70:0/718040810 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:56.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f13fe7fc640 1 --2- 172.21.15.70:0/718040810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14001022f0 0x7f1400199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f13feffd640 1 --2- 172.21.15.70:0/718040810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1400101930 0x7f1400199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f13ff7fe640 1 -- 172.21.15.70:0/718040810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1400101930 msgr2=0x7f1400199220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:56.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f13ff7fe640 1 --2- 172.21.15.70:0/718040810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1400101930 0x7f1400199220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f13ff7fe640 1 -- 172.21.15.70:0/718040810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14001022f0 msgr2=0x7f1400199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:56.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f13ff7fe640 1 --2- 172.21.15.70:0/718040810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14001022f0 0x7f1400199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f13ff7fe640 1 -- 172.21.15.70:0/718040810 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f140019e210 con 0x7f1400107940 2024-06-04T05:09:56.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f13feffd640 1 --2- 172.21.15.70:0/718040810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1400101930 0x7f1400199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:09:56.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.666+0000 7f13ff7fe640 1 --2- 172.21.15.70:0/718040810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1400107940 0x7f140019dae0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f13f8005cc0 tx=0x7f13f8003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:56.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.667+0000 7f1404956640 1 -- 172.21.15.70:0/718040810 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f8004240 con 0x7f1400107940 2024-06-04T05:09:56.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.667+0000 7f1404956640 1 -- 172.21.15.70:0/718040810 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f13f8003e10 con 0x7f1400107940 2024-06-04T05:09:56.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.667+0000 7f1405958640 1 -- 172.21.15.70:0/718040810 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f140019e4a0 con 0x7f1400107940 2024-06-04T05:09:56.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.667+0000 7f1404956640 1 -- 172.21.15.70:0/718040810 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f8030da0 con 0x7f1400107940 2024-06-04T05:09:56.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.667+0000 7f1405958640 1 -- 172.21.15.70:0/718040810 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14001a5d90 con 0x7f1400107940 2024-06-04T05:09:56.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.669+0000 7f1404956640 1 -- 172.21.15.70:0/718040810 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f13f80305a0 con 0x7f1400107940 2024-06-04T05:09:56.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.669+0000 7f1408672740 1 Processor -- start 2024-06-04T05:09:56.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.669+0000 7f1408672740 1 -- start start 2024-06-04T05:09:56.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.669+0000 7f1408672740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f3820 0x5575e11f3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.669+0000 7f1408672740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f4140 0x5575e11fe4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.669+0000 7f1408672740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e1200df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.669+0000 7f1408672740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5575e1165c30 con 0x5575e11fea00 2024-06-04T05:09:56.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.669+0000 7f1408672740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5575e11557d0 con 0x5575e11f4140 2024-06-04T05:09:56.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.669+0000 7f1408672740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5575e1159810 con 0x5575e11f3820 2024-06-04T05:09:56.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f1404956640 1 --2- 172.21.15.70:0/718040810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f13e80b2a10 0x7f13e80b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f13fe7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e1200df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f13fe7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e1200df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50864/0 (socket says 172.21.15.70:50864) 2024-06-04T05:09:56.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f13fe7fc640 1 -- 172.21.15.70:0/1301195459 learned_addr learned my addr 172.21.15.70:0/1301195459 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:56.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f13ff7fe640 1 --2- 172.21.15.70:0/718040810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f13e80b2a10 0x7f13e80b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f13fe7fc640 1 --2- 172.21.15.70:0/1301195459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f3820 0x5575e11f3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f13feffd640 1 --2- 172.21.15.70:0/1301195459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f4140 0x5575e11fe4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f13fe7fc640 1 -- 172.21.15.70:0/1301195459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f3820 msgr2=0x5575e11f3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:56.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f13fe7fc640 1 --2- 172.21.15.70:0/1301195459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f3820 0x5575e11f3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f13fe7fc640 1 -- 172.21.15.70:0/1301195459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f4140 msgr2=0x5575e11fe4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:56.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f13fe7fc640 1 --2- 172.21.15.70:0/1301195459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f4140 0x5575e11fe4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f13fe7fc640 1 -- 172.21.15.70:0/1301195459 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575e0e6a210 con 0x5575e11fea00 2024-06-04T05:09:56.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f1404956640 1 -- 172.21.15.70:0/718040810 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f13f802e030 con 0x7f1400107940 2024-06-04T05:09:56.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f13fe7fc640 1 --2- 172.21.15.70:0/1301195459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e1200df0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f13f4005fd0 tx=0x7f13f400e5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01cf24138580ba0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:56.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f13ff7fe640 1 --2- 172.21.15.70:0/718040810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f13e80b2a10 0x7f13e80b4ed0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f13f80ca7c0 tx=0x7f13f8090450 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:56.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f13e5ffb640 1 -- 172.21.15.70:0/1301195459 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f4018030 con 0x5575e11fea00 2024-06-04T05:09:56.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f13e5ffb640 1 -- 172.21.15.70:0/1301195459 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f13f40054b0 con 0x5575e11fea00 2024-06-04T05:09:56.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.670+0000 7f13e5ffb640 1 -- 172.21.15.70:0/1301195459 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f4010640 con 0x5575e11fea00 2024-06-04T05:09:56.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.671+0000 7f1408672740 1 -- 172.21.15.70:0/1301195459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 msgr2=0x5575e1200df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:56.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.671+0000 7f1408672740 1 --2- 172.21.15.70:0/1301195459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e1200df0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f13f4005fd0 tx=0x7f13f400e5b0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.671+0000 7f1408672740 1 -- 172.21.15.70:0/1301195459 shutdown_connections 2024-06-04T05:09:56.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.671+0000 7f1408672740 1 --2- 172.21.15.70:0/1301195459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f4140 0x5575e11fe4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.671+0000 7f1408672740 1 --2- 172.21.15.70:0/1301195459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e1200df0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.671+0000 7f1408672740 1 --2- 172.21.15.70:0/1301195459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f3820 0x5575e11f3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.671+0000 7f1408672740 1 -- 172.21.15.70:0/1301195459 >> 172.21.15.70:0/1301195459 conn(0x5575e11efd30 msgr2=0x5575e11f0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:56.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.671+0000 7f1408672740 1 -- 172.21.15.70:0/1301195459 shutdown_connections 2024-06-04T05:09:56.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.671+0000 7f1408672740 1 -- 172.21.15.70:0/1301195459 wait complete. 2024-06-04T05:09:56.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.671+0000 7f1408672740 1 -- start start 2024-06-04T05:09:56.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f1408672740 1 Processor -- start 2024-06-04T05:09:56.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f1408672740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f3820 0x5575e11fba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f1408672740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f4140 0x5575e11fbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f1408672740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e11fc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f1408672740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5575e0e3b790 con 0x5575e11fea00 2024-06-04T05:09:56.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f1408672740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5575e1111890 con 0x5575e11f4140 2024-06-04T05:09:56.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f1408672740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5575e1201630 con 0x5575e11f3820 2024-06-04T05:09:56.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f13fe7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e11fc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f1408672740 10 client.0 _ll_register_callbacks cb 0x5575e0e69a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:56.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f13fe7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e11fc480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50880/0 (socket says 172.21.15.70:50880) 2024-06-04T05:09:56.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f13fe7fc640 1 -- 172.21.15.70:0/554872278 learned_addr learned my addr 172.21.15.70:0/554872278 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:56.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f13fe7fc640 1 --2- 172.21.15.70:0/554872278 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f3820 0x5575e11fba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f13feffd640 1 --2- 172.21.15.70:0/554872278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f4140 0x5575e11fbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f13fe7fc640 1 -- 172.21.15.70:0/554872278 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f3820 msgr2=0x5575e11fba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:56.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f13fe7fc640 1 --2- 172.21.15.70:0/554872278 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f3820 0x5575e11fba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f13fe7fc640 1 -- 172.21.15.70:0/554872278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f4140 msgr2=0x5575e11fbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:56.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f13fe7fc640 1 --2- 172.21.15.70:0/554872278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f4140 0x5575e11fbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f13fe7fc640 1 -- 172.21.15.70:0/554872278 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575e0f36620 con 0x5575e11fea00 2024-06-04T05:09:56.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f13fe7fc640 1 --2- 172.21.15.70:0/554872278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e11fc480 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f13f4004fd0 tx=0x7f13f40056c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd19cb401c7a356e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:56.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f13deffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:56.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f13deffd640 1 -- 172.21.15.70:0/554872278 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f4022030 con 0x5575e11fea00 2024-06-04T05:09:56.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f13deffd640 1 -- 172.21.15.70:0/554872278 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f13f4004030 con 0x5575e11fea00 2024-06-04T05:09:56.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.672+0000 7f1408672740 1 -- 172.21.15.70:0/554872278 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13f40054b0 con 0x5575e11fea00 2024-06-04T05:09:56.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.673+0000 7f13deffd640 1 -- 172.21.15.70:0/554872278 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f13f4031050 con 0x5575e11fea00 2024-06-04T05:09:56.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.673+0000 7f1408672740 10 client.18837 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:56.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.673+0000 7f1408672740 1 -- 172.21.15.70:0/554872278 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5575e120d400 con 0x5575e11fea00 2024-06-04T05:09:56.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.673+0000 7f13deffd640 1 -- 172.21.15.70:0/554872278 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f13f40054b0 con 0x5575e11fea00 2024-06-04T05:09:56.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.673+0000 7f13deffd640 1 -- 172.21.15.70:0/554872278 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f13f400f040 con 0x5575e11fea00 2024-06-04T05:09:56.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.673+0000 7f1408672740 10 client.18837 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:56.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.673+0000 7f1408672740 10 client.18837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:56.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.673+0000 7f1408672740 20 client.18837 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:56.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.673+0000 7f1408672740 1 --2- 172.21.15.70:0/554872278 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5575e120e860 0x5575e1210c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.673+0000 7f1408672740 4 client.18837 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:09:56.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.673+0000 7f1408672740 1 -- 172.21.15.70:0/554872278 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575e1167b80 con 0x5575e120e860 2024-06-04T05:09:56.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.673+0000 7f13feffd640 1 --2- 172.21.15.70:0/554872278 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5575e120e860 0x5575e1210c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.674+0000 7f13deffd640 10 client.18837 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.674+0000 7f13feffd640 1 --2- 172.21.15.70:0/554872278 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5575e120e860 0x5575e1210c50 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9596c06362306a2 server_cookie=c21ae1f75d6a0dfd in_seq=0 out_seq=0 2024-06-04T05:09:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.674+0000 7f13deffd640 1 -- 172.21.15.70:0/554872278 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5575e1167b80 con 0x5575e120e860 2024-06-04T05:09:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.674+0000 7f13deffd640 10 client.18837 handle_command_reply: tid=0 2024-06-04T05:09:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.674+0000 7f1408672740 1 client.18837 shutdown 2024-06-04T05:09:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.675+0000 7f1408672740 1 -- 172.21.15.70:0/554872278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 msgr2=0x5575e11fc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.675+0000 7f1408672740 1 --2- 172.21.15.70:0/554872278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e11fc480 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f13f4004fd0 tx=0x7f13f40056c0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.675+0000 7f1408672740 1 -- 172.21.15.70:0/554872278 shutdown_connections 2024-06-04T05:09:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.675+0000 7f1408672740 1 --2- 172.21.15.70:0/554872278 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5575e120e860 0x5575e1210c50 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.675+0000 7f1408672740 1 --2- 172.21.15.70:0/554872278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f4140 0x5575e11fbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.675+0000 7f13deffd640 0 client.18837 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.675+0000 7f1408672740 1 --2- 172.21.15.70:0/554872278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e11fc480 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.675+0000 7f1408672740 1 --2- 172.21.15.70:0/554872278 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f3820 0x5575e11fba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.675+0000 7f1408672740 1 -- 172.21.15.70:0/554872278 >> 172.21.15.70:0/554872278 conn(0x5575e11efd30 msgr2=0x5575e11eefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.675+0000 7f1408672740 1 -- 172.21.15.70:0/554872278 shutdown_connections 2024-06-04T05:09:56.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.675+0000 7f1408672740 1 -- 172.21.15.70:0/554872278 wait complete. 2024-06-04T05:09:56.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.675+0000 7f1408672740 20 client.18837 trim_cache size 0 max 16384 2024-06-04T05:09:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.690+0000 7f1408672740 1 Processor -- start 2024-06-04T05:09:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.690+0000 7f1408672740 1 -- start start 2024-06-04T05:09:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.690+0000 7f1408672740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f3820 0x5575e12162b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.690+0000 7f1408672740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f4140 0x5575e12167f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.690+0000 7f1408672740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e11fb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.690+0000 7f1408672740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f14001ae640 con 0x5575e11fea00 2024-06-04T05:09:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.690+0000 7f1408672740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f14001b02a0 con 0x5575e11f4140 2024-06-04T05:09:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.690+0000 7f1408672740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f14001aed10 con 0x5575e11f3820 2024-06-04T05:09:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.690+0000 7f13fe7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e11fb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.690+0000 7f13fe7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e11fb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50896/0 (socket says 172.21.15.70:50896) 2024-06-04T05:09:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.690+0000 7f13fe7fc640 1 -- 172.21.15.70:0/1850379536 learned_addr learned my addr 172.21.15.70:0/1850379536 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.690+0000 7f13fe7fc640 1 -- 172.21.15.70:0/1850379536 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f3820 msgr2=0x5575e12162b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.690+0000 7f13feffd640 1 --2- 172.21.15.70:0/1850379536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f4140 0x5575e12167f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.690+0000 7f13fe7fc640 1 --2- 172.21.15.70:0/1850379536 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f3820 0x5575e12162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.690+0000 7f13fe7fc640 1 -- 172.21.15.70:0/1850379536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f4140 msgr2=0x5575e12167f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.690+0000 7f13fe7fc640 1 --2- 172.21.15.70:0/1850379536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f4140 0x5575e12167f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.690+0000 7f13fe7fc640 1 -- 172.21.15.70:0/1850379536 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575e11f48b0 con 0x5575e11fea00 2024-06-04T05:09:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.691+0000 7f13fe7fc640 1 --2- 172.21.15.70:0/1850379536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e11fb9c0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f13f4005ec0 tx=0x7f13f40058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a78e1fb62da4e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.691+0000 7f13e57fa640 1 -- 172.21.15.70:0/1850379536 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f4022070 con 0x5575e11fea00 2024-06-04T05:09:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.691+0000 7f13e57fa640 1 -- 172.21.15.70:0/1850379536 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f13f4019040 con 0x5575e11fea00 2024-06-04T05:09:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.691+0000 7f13e57fa640 1 -- 172.21.15.70:0/1850379536 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f4018050 con 0x5575e11fea00 2024-06-04T05:09:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.691+0000 7f1408672740 1 -- 172.21.15.70:0/1850379536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 msgr2=0x5575e11fb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.691+0000 7f1408672740 1 --2- 172.21.15.70:0/1850379536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e11fb9c0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f13f4005ec0 tx=0x7f13f40058f0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.691+0000 7f1408672740 1 -- 172.21.15.70:0/1850379536 shutdown_connections 2024-06-04T05:09:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.691+0000 7f1408672740 1 --2- 172.21.15.70:0/1850379536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f4140 0x5575e12167f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.691+0000 7f1408672740 1 --2- 172.21.15.70:0/1850379536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e11fb9c0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.691+0000 7f1408672740 1 --2- 172.21.15.70:0/1850379536 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f3820 0x5575e12162b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.691+0000 7f1408672740 1 -- 172.21.15.70:0/1850379536 >> 172.21.15.70:0/1850379536 conn(0x5575e11efd30 msgr2=0x5575e11eae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.691+0000 7f1408672740 1 -- 172.21.15.70:0/1850379536 shutdown_connections 2024-06-04T05:09:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.691+0000 7f1408672740 1 -- 172.21.15.70:0/1850379536 wait complete. 2024-06-04T05:09:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f1408672740 1 -- start start 2024-06-04T05:09:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f1408672740 1 Processor -- start 2024-06-04T05:09:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f1408672740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f3820 0x5575e1206bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f1408672740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f4140 0x5575e1207130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f1408672740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e1202160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f1408672740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f14001afb70 con 0x5575e11fea00 2024-06-04T05:09:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f1408672740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f14001af440 con 0x5575e11f3820 2024-06-04T05:09:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f1408672740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f14001ac640 con 0x5575e11f4140 2024-06-04T05:09:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f1408672740 10 client.0 _ll_register_callbacks cb 0x7f13f400ef20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f13fe7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e1202160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f13fe7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e1202160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50912/0 (socket says 172.21.15.70:50912) 2024-06-04T05:09:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f13fe7fc640 1 -- 172.21.15.70:0/2562246127 learned_addr learned my addr 172.21.15.70:0/2562246127 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f13feffd640 1 --2- 172.21.15.70:0/2562246127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f4140 0x5575e1207130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f13fe7fc640 1 -- 172.21.15.70:0/2562246127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f4140 msgr2=0x5575e1207130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f13fe7fc640 1 --2- 172.21.15.70:0/2562246127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f4140 0x5575e1207130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f13fe7fc640 1 -- 172.21.15.70:0/2562246127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f3820 msgr2=0x5575e1206bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f13fe7fc640 1 --2- 172.21.15.70:0/2562246127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f3820 0x5575e1206bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.692+0000 7f13fe7fc640 1 -- 172.21.15.70:0/2562246127 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575e1202960 con 0x5575e11fea00 2024-06-04T05:09:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.693+0000 7f13fe7fc640 1 --2- 172.21.15.70:0/2562246127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e1202160 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f13f4004cb0 tx=0x7f13f400e460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b6ae117fe4a3ee server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.693+0000 7f13deffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.693+0000 7f13deffd640 1 -- 172.21.15.70:0/2562246127 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f4018070 con 0x5575e11fea00 2024-06-04T05:09:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.693+0000 7f13deffd640 1 -- 172.21.15.70:0/2562246127 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f13f4004ce0 con 0x5575e11fea00 2024-06-04T05:09:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.693+0000 7f1408672740 1 -- 172.21.15.70:0/2562246127 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13f4019040 con 0x5575e11fea00 2024-06-04T05:09:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.693+0000 7f13deffd640 1 -- 172.21.15.70:0/2562246127 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f13f403a070 con 0x5575e11fea00 2024-06-04T05:09:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.693+0000 7f1408672740 10 client.18843 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.693+0000 7f1408672740 1 -- 172.21.15.70:0/2562246127 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5575e1278e20 con 0x5575e11fea00 2024-06-04T05:09:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.693+0000 7f13deffd640 1 -- 172.21.15.70:0/2562246127 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f13f4019040 con 0x5575e11fea00 2024-06-04T05:09:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.693+0000 7f13deffd640 1 -- 172.21.15.70:0/2562246127 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f13f4034070 con 0x5575e11fea00 2024-06-04T05:09:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.693+0000 7f1408672740 10 client.18843 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.693+0000 7f1408672740 10 client.18843 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:09:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.693+0000 7f1408672740 20 client.18843 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:56.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.693+0000 7f1408672740 1 --2- 172.21.15.70:0/2562246127 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5575e120e860 0x5575e127b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.693+0000 7f1408672740 4 client.18843 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:09:56.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.693+0000 7f1408672740 1 -- 172.21.15.70:0/2562246127 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5575e1161360 con 0x5575e120e860 2024-06-04T05:09:56.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.694+0000 7f13feffd640 1 --2- 172.21.15.70:0/2562246127 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5575e120e860 0x5575e127b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.694+0000 7f13feffd640 1 --2- 172.21.15.70:0/2562246127 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5575e120e860 0x5575e127b4f0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f6ea7b235cf402e server_cookie=766d90a2ccb0f9d7 in_seq=0 out_seq=0 2024-06-04T05:09:56.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.694+0000 7f13deffd640 10 client.18843 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f13deffd640 1 -- 172.21.15.70:0/2562246127 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5575e1161360 con 0x5575e120e860 2024-06-04T05:09:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f13deffd640 10 client.18843 handle_command_reply: tid=0 2024-06-04T05:09:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f1408672740 1 client.18843 shutdown 2024-06-04T05:09:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f1408672740 1 -- 172.21.15.70:0/2562246127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 msgr2=0x5575e1202160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f1408672740 1 --2- 172.21.15.70:0/2562246127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e1202160 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f13f4004cb0 tx=0x7f13f400e460 comp rx=0 tx=0).stop 2024-06-04T05:09:56.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f1408672740 1 -- 172.21.15.70:0/2562246127 shutdown_connections 2024-06-04T05:09:56.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f1408672740 1 --2- 172.21.15.70:0/2562246127 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5575e120e860 0x5575e127b4f0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f1408672740 1 --2- 172.21.15.70:0/2562246127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575e11fea00 0x5575e1202160 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f1408672740 1 --2- 172.21.15.70:0/2562246127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575e11f4140 0x5575e1207130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f1408672740 1 --2- 172.21.15.70:0/2562246127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e11f3820 0x5575e1206bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f13deffd640 0 client.18843 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:56.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f1408672740 1 -- 172.21.15.70:0/2562246127 >> 172.21.15.70:0/2562246127 conn(0x5575e11efd30 msgr2=0x5575e11fb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:56.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f1408672740 1 -- 172.21.15.70:0/2562246127 shutdown_connections 2024-06-04T05:09:56.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f1408672740 1 -- 172.21.15.70:0/2562246127 wait complete. 2024-06-04T05:09:56.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f1408672740 20 client.18843 trim_cache size 0 max 16384 2024-06-04T05:09:56.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:09:56.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:09:56.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:09:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f1405958640 1 -- 172.21.15.70:0/718040810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f13e80b2a10 msgr2=0x7f13e80b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f1405958640 1 --2- 172.21.15.70:0/718040810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f13e80b2a10 0x7f13e80b4ed0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f13f80ca7c0 tx=0x7f13f8090450 comp rx=0 tx=0).stop 2024-06-04T05:09:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f1405958640 1 -- 172.21.15.70:0/718040810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1400107940 msgr2=0x7f140019dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.710+0000 7f1405958640 1 --2- 172.21.15.70:0/718040810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1400107940 0x7f140019dae0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f13f8005cc0 tx=0x7f13f8003c20 comp rx=0 tx=0).stop 2024-06-04T05:09:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.711+0000 7f1405958640 1 -- 172.21.15.70:0/718040810 shutdown_connections 2024-06-04T05:09:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.711+0000 7f1405958640 1 --2- 172.21.15.70:0/718040810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f13e80b2a10 0x7f13e80b4ed0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.711+0000 7f1405958640 1 --2- 172.21.15.70:0/718040810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14001022f0 0x7f1400199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.711+0000 7f1405958640 1 --2- 172.21.15.70:0/718040810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1400107940 0x7f140019dae0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.711+0000 7f1405958640 1 --2- 172.21.15.70:0/718040810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1400101930 0x7f1400199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.711+0000 7f1405958640 1 -- 172.21.15.70:0/718040810 >> 172.21.15.70:0/718040810 conn(0x7f14000fd420 msgr2=0x7f140006b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.711+0000 7f1405958640 1 -- 172.21.15.70:0/718040810 shutdown_connections 2024-06-04T05:09:56.716 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:09:56.711+0000 7f1405958640 1 -- 172.21.15.70:0/718040810 wait complete. 2024-06-04T05:09:56.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.64 MB/sec execute 529 sec latency 30032.695 ms 2024-06-04T05:09:56.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.960+0000 7f0748709700 1 -- 172.21.15.70:0/1349968485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0740101f00 msgr2=0x7f0740102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:56.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.960+0000 7f0748709700 1 --2- 172.21.15.70:0/1349968485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0740101f00 0x7f0740102380 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f0730009990 tx=0x7f0730009ca0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.961+0000 7f0748709700 1 -- 172.21.15.70:0/1349968485 shutdown_connections 2024-06-04T05:09:56.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.961+0000 7f0748709700 1 --2- 172.21.15.70:0/1349968485 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f07401028c0 0x7f0740110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.961+0000 7f0748709700 1 --2- 172.21.15.70:0/1349968485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0740108620 0x7f0740108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.961+0000 7f0748709700 1 --2- 172.21.15.70:0/1349968485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0740101f00 0x7f0740102380 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.961+0000 7f0748709700 1 -- 172.21.15.70:0/1349968485 >> 172.21.15.70:0/1349968485 conn(0x7f0740077780 msgr2=0x7f0740077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:56.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.961+0000 7f0748709700 1 -- 172.21.15.70:0/1349968485 shutdown_connections 2024-06-04T05:09:56.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.961+0000 7f0748709700 1 -- 172.21.15.70:0/1349968485 wait complete. 2024-06-04T05:09:56.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.961+0000 7f0748709700 1 Processor -- start 2024-06-04T05:09:56.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.961+0000 7f0748709700 1 -- start start 2024-06-04T05:09:56.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.961+0000 7f0748709700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0740101f00 0x7f074019f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f0748709700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07401028c0 0x7f074019fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f0748709700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0740108620 0x7f0740199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f0748709700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f07401136e0 con 0x7f0740101f00 2024-06-04T05:09:56.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f0748709700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0740113560 con 0x7f07401028c0 2024-06-04T05:09:56.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f0748709700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0740113860 con 0x7f0740108620 2024-06-04T05:09:56.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f07464a5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0740101f00 0x7f074019f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f07464a5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0740101f00 0x7f074019f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50934/0 (socket says 172.21.15.70:50934) 2024-06-04T05:09:56.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f07464a5700 1 -- 172.21.15.70:0/1133729529 learned_addr learned my addr 172.21.15.70:0/1133729529 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:56.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f0746ca6700 1 --2- 172.21.15.70:0/1133729529 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0740108620 0x7f0740199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f0745ca4700 1 --2- 172.21.15.70:0/1133729529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07401028c0 0x7f074019fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f07464a5700 1 -- 172.21.15.70:0/1133729529 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0740108620 msgr2=0x7f0740199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:56.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f07464a5700 1 --2- 172.21.15.70:0/1133729529 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0740108620 0x7f0740199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f07464a5700 1 -- 172.21.15.70:0/1133729529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07401028c0 msgr2=0x7f074019fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:56.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f07464a5700 1 --2- 172.21.15.70:0/1133729529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07401028c0 0x7f074019fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f07464a5700 1 -- 172.21.15.70:0/1133729529 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f074019a240 con 0x7f0740101f00 2024-06-04T05:09:56.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f0745ca4700 1 --2- 172.21.15.70:0/1133729529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07401028c0 0x7f074019fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:09:56.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f07464a5700 1 --2- 172.21.15.70:0/1133729529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0740101f00 0x7f074019f920 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f073c00b500 tx=0x7f073c00b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f07377fe700 1 -- 172.21.15.70:0/1133729529 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073c015020 con 0x7f0740101f00 2024-06-04T05:09:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f07377fe700 1 -- 172.21.15.70:0/1133729529 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f073c011cc0 con 0x7f0740101f00 2024-06-04T05:09:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f0748709700 1 -- 172.21.15.70:0/1133729529 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f074019a530 con 0x7f0740101f00 2024-06-04T05:09:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f07377fe700 1 -- 172.21.15.70:0/1133729529 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073c010ca0 con 0x7f0740101f00 2024-06-04T05:09:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.962+0000 7f0748709700 1 -- 172.21.15.70:0/1133729529 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07401a67f0 con 0x7f0740101f00 2024-06-04T05:09:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.965+0000 7f07377fe700 1 -- 172.21.15.70:0/1133729529 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f073c018020 con 0x7f0740101f00 2024-06-04T05:09:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.965+0000 7f07377fe700 1 --2- 172.21.15.70:0/1133729529 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f072c0b29f0 0x7f072c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.965+0000 7f0745ca4700 1 --2- 172.21.15.70:0/1133729529 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f072c0b29f0 0x7f072c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.966+0000 7f07377fe700 1 -- 172.21.15.70:0/1133729529 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f073c0e1cc0 con 0x7f0740101f00 2024-06-04T05:09:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.966+0000 7f0752f9eb80 1 Processor -- start 2024-06-04T05:09:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.966+0000 7f0745ca4700 1 --2- 172.21.15.70:0/1133729529 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f072c0b29f0 0x7f072c0b4eb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0730009750 tx=0x7f073001a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.966+0000 7f0752f9eb80 1 -- start start 2024-06-04T05:09:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.966+0000 7f0752f9eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753a9860 0x55fa753a9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.966+0000 7f0752f9eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 0x55fa753ad940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.966+0000 7f0752f9eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753adf60 0x55fa753ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.966+0000 7f0752f9eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fa75308c90 con 0x55fa753adf60 2024-06-04T05:09:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.966+0000 7f0752f9eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fa752f8830 con 0x55fa753a9860 2024-06-04T05:09:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.966+0000 7f0752f9eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fa752fc870 con 0x55fa753aa210 2024-06-04T05:09:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.967+0000 7f0746ca6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753adf60 0x55fa753ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.967+0000 7f0746ca6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753adf60 0x55fa753ae3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50936/0 (socket says 172.21.15.70:50936) 2024-06-04T05:09:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.967+0000 7f0746ca6700 1 -- 172.21.15.70:0/2075327682 learned_addr learned my addr 172.21.15.70:0/2075327682 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.967+0000 7f07464a5700 1 --2- 172.21.15.70:0/2075327682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753a9860 0x55fa753a9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.967+0000 7f0745ca4700 1 --2- 172.21.15.70:0/2075327682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 0x55fa753ad940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.967+0000 7f0746ca6700 1 -- 172.21.15.70:0/2075327682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 msgr2=0x55fa753ad940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.967+0000 7f0746ca6700 1 --2- 172.21.15.70:0/2075327682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 0x55fa753ad940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.967+0000 7f0746ca6700 1 -- 172.21.15.70:0/2075327682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753a9860 msgr2=0x55fa753a9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.967+0000 7f0746ca6700 1 --2- 172.21.15.70:0/2075327682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753a9860 0x55fa753a9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.967+0000 7f0746ca6700 1 -- 172.21.15.70:0/2075327682 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa752488d0 con 0x55fa753adf60 2024-06-04T05:09:56.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.967+0000 7f0746ca6700 1 --2- 172.21.15.70:0/2075327682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753adf60 0x55fa753ae3a0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f073800a430 tx=0x7f0738009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6881f45cbbd1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:56.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.967+0000 7f0734ff9700 1 -- 172.21.15.70:0/2075327682 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0738014620 con 0x55fa753adf60 2024-06-04T05:09:56.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.967+0000 7f0734ff9700 1 -- 172.21.15.70:0/2075327682 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0738019070 con 0x55fa753adf60 2024-06-04T05:09:56.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.967+0000 7f0734ff9700 1 -- 172.21.15.70:0/2075327682 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07380139e0 con 0x55fa753adf60 2024-06-04T05:09:56.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.968+0000 7f0752f9eb80 1 -- 172.21.15.70:0/2075327682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753adf60 msgr2=0x55fa753ae3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:56.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.968+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/2075327682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753adf60 0x55fa753ae3a0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f073800a430 tx=0x7f0738009650 comp rx=0 tx=0).stop 2024-06-04T05:09:56.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.968+0000 7f0752f9eb80 1 -- 172.21.15.70:0/2075327682 shutdown_connections 2024-06-04T05:09:56.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.968+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/2075327682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753adf60 0x55fa753ae3a0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.968+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/2075327682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753a9860 0x55fa753a9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.968+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/2075327682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 0x55fa753ad940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.968+0000 7f0752f9eb80 1 -- 172.21.15.70:0/2075327682 >> 172.21.15.70:0/2075327682 conn(0x55fa752d7080 msgr2=0x55fa753a3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:56.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.968+0000 7f0752f9eb80 1 -- 172.21.15.70:0/2075327682 shutdown_connections 2024-06-04T05:09:56.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.968+0000 7f0752f9eb80 1 -- 172.21.15.70:0/2075327682 wait complete. 2024-06-04T05:09:56.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.968+0000 7f0752f9eb80 1 -- start start 2024-06-04T05:09:56.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f0752f9eb80 1 Processor -- start 2024-06-04T05:09:56.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f0752f9eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753a9860 0x55fa753b5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f0752f9eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 0x55fa753b2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f0752f9eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753adf60 0x55fa753b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f0752f9eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fa751fefa0 con 0x55fa753a9860 2024-06-04T05:09:56.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f0752f9eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fa7529a7c0 con 0x55fa753adf60 2024-06-04T05:09:56.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f0752f9eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fa753b6120 con 0x55fa753aa210 2024-06-04T05:09:56.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f0752f9eb80 10 client.0 _ll_register_callbacks cb 0x55fa75293920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:56.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f07464a5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753a9860 0x55fa753b5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f07464a5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753a9860 0x55fa753b5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50950/0 (socket says 172.21.15.70:50950) 2024-06-04T05:09:56.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f07464a5700 1 -- 172.21.15.70:0/381941271 learned_addr learned my addr 172.21.15.70:0/381941271 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:56.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f0745ca4700 1 --2- 172.21.15.70:0/381941271 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 0x55fa753b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f0746ca6700 1 --2- 172.21.15.70:0/381941271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753adf60 0x55fa753b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f07464a5700 1 -- 172.21.15.70:0/381941271 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 msgr2=0x55fa753b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:56.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f07464a5700 1 --2- 172.21.15.70:0/381941271 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 0x55fa753b2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f07464a5700 1 -- 172.21.15.70:0/381941271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753adf60 msgr2=0x55fa753b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:56.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f07464a5700 1 --2- 172.21.15.70:0/381941271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753adf60 0x55fa753b2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f07464a5700 1 -- 172.21.15.70:0/381941271 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa75297690 con 0x55fa753a9860 2024-06-04T05:09:56.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f07464a5700 1 --2- 172.21.15.70:0/381941271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753a9860 0x55fa753b5340 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f073c0a50b0 tx=0x7f073c010640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d4774c5676a2775 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:56.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f0729ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:56.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.969+0000 7f0729ffb700 1 -- 172.21.15.70:0/381941271 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073c0ad430 con 0x55fa753a9860 2024-06-04T05:09:56.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.970+0000 7f0729ffb700 1 -- 172.21.15.70:0/381941271 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f073c0a5670 con 0x55fa753a9860 2024-06-04T05:09:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.970+0000 7f0752f9eb80 1 -- 172.21.15.70:0/381941271 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0738019070 con 0x55fa753a9860 2024-06-04T05:09:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.970+0000 7f0729ffb700 1 -- 172.21.15.70:0/381941271 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f073c0b1ee0 con 0x55fa753a9860 2024-06-04T05:09:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.970+0000 7f0752f9eb80 10 client.18855 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.970+0000 7f0752f9eb80 1 -- 172.21.15.70:0/381941271 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fa753c07c0 con 0x55fa753a9860 2024-06-04T05:09:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.970+0000 7f0729ffb700 1 -- 172.21.15.70:0/381941271 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0738019070 con 0x55fa753a9860 2024-06-04T05:09:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.970+0000 7f0729ffb700 1 -- 172.21.15.70:0/381941271 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f073c0b1a50 con 0x55fa753a9860 2024-06-04T05:09:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.970+0000 7f0752f9eb80 10 client.18855 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.970+0000 7f0752f9eb80 10 client.18855 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:09:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.970+0000 7f0752f9eb80 20 client.18855 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.970+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/381941271 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55fa753c2d50 0x55fa753c5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.970+0000 7f0752f9eb80 4 client.18855 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:09:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.970+0000 7f0752f9eb80 1 -- 172.21.15.70:0/381941271 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa753759e0 con 0x55fa753c2d50 2024-06-04T05:09:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.970+0000 7f0745ca4700 1 --2- 172.21.15.70:0/381941271 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55fa753c2d50 0x55fa753c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.971+0000 7f0729ffb700 10 client.18855 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:09:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.971+0000 7f0745ca4700 1 --2- 172.21.15.70:0/381941271 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55fa753c2d50 0x55fa753c5190 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6c48d46291bd834a server_cookie=d9867a30b169d4c8 in_seq=0 out_seq=0 2024-06-04T05:09:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.972+0000 7f0729ffb700 1 -- 172.21.15.70:0/381941271 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fa753759e0 con 0x55fa753c2d50 2024-06-04T05:09:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.972+0000 7f0729ffb700 10 client.18855 handle_command_reply: tid=0 2024-06-04T05:09:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.972+0000 7f0752f9eb80 1 client.18855 shutdown 2024-06-04T05:09:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.972+0000 7f0752f9eb80 1 -- 172.21.15.70:0/381941271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753a9860 msgr2=0x55fa753b5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.972+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/381941271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753a9860 0x55fa753b5340 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f073c0a50b0 tx=0x7f073c010640 comp rx=0 tx=0).stop 2024-06-04T05:09:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.972+0000 7f0752f9eb80 1 -- 172.21.15.70:0/381941271 shutdown_connections 2024-06-04T05:09:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.972+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/381941271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753adf60 0x55fa753b2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.972+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/381941271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753a9860 0x55fa753b5340 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.972+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/381941271 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55fa753c2d50 0x55fa753c5190 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.972+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/381941271 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 0x55fa753b2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.972+0000 7f0729ffb700 0 client.18855 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:09:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.972+0000 7f0752f9eb80 1 -- 172.21.15.70:0/381941271 >> 172.21.15.70:0/381941271 conn(0x55fa752d7080 msgr2=0x55fa753af350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:56.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.972+0000 7f0752f9eb80 1 -- 172.21.15.70:0/381941271 shutdown_connections 2024-06-04T05:09:56.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.972+0000 7f0752f9eb80 1 -- 172.21.15.70:0/381941271 wait complete. 2024-06-04T05:09:56.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.972+0000 7f0752f9eb80 20 client.18855 trim_cache size 0 max 16384 2024-06-04T05:09:56.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.987+0000 7f0752f9eb80 1 Processor -- start 2024-06-04T05:09:56.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.987+0000 7f0752f9eb80 1 -- start start 2024-06-04T05:09:56.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.987+0000 7f0752f9eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753a9860 0x55fa753a4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.987+0000 7f0752f9eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 0x55fa753b17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.987+0000 7f0752f9eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753adf60 0x55fa753b4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.987+0000 7f0752f9eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f07401adf70 con 0x55fa753adf60 2024-06-04T05:09:56.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.987+0000 7f0752f9eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f07401afbd0 con 0x55fa753a9860 2024-06-04T05:09:56.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.987+0000 7f0752f9eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f07401ae640 con 0x55fa753aa210 2024-06-04T05:09:56.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.987+0000 7f0746ca6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753adf60 0x55fa753b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.987+0000 7f0746ca6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753adf60 0x55fa753b4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50966/0 (socket says 172.21.15.70:50966) 2024-06-04T05:09:56.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.987+0000 7f0746ca6700 1 -- 172.21.15.70:0/667150827 learned_addr learned my addr 172.21.15.70:0/667150827 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:56.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.988+0000 7f07464a5700 1 --2- 172.21.15.70:0/667150827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753a9860 0x55fa753a4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:56.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.988+0000 7f0746ca6700 1 -- 172.21.15.70:0/667150827 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 msgr2=0x55fa753b17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:56.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.988+0000 7f0746ca6700 1 --2- 172.21.15.70:0/667150827 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 0x55fa753b17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.988+0000 7f0746ca6700 1 -- 172.21.15.70:0/667150827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753a9860 msgr2=0x55fa753a4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:56.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.988+0000 7f0746ca6700 1 --2- 172.21.15.70:0/667150827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753a9860 0x55fa753a4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.988+0000 7f0746ca6700 1 -- 172.21.15.70:0/667150827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa753a4d40 con 0x55fa753adf60 2024-06-04T05:09:56.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.988+0000 7f0746ca6700 1 --2- 172.21.15.70:0/667150827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753adf60 0x55fa753b4580 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f0738004b10 tx=0x7f0738009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b9238684b58418 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:56.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.988+0000 7f072bfff700 1 -- 172.21.15.70:0/667150827 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0738022070 con 0x55fa753adf60 2024-06-04T05:09:56.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.988+0000 7f072bfff700 1 -- 172.21.15.70:0/667150827 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0738019070 con 0x55fa753adf60 2024-06-04T05:09:56.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.988+0000 7f072bfff700 1 -- 172.21.15.70:0/667150827 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0738014aa0 con 0x55fa753adf60 2024-06-04T05:09:56.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 1 -- 172.21.15.70:0/667150827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753adf60 msgr2=0x55fa753b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:56.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/667150827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753adf60 0x55fa753b4580 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f0738004b10 tx=0x7f0738009be0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 1 -- 172.21.15.70:0/667150827 shutdown_connections 2024-06-04T05:09:56.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/667150827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753adf60 0x55fa753b4580 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/667150827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753a9860 0x55fa753a4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/667150827 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 0x55fa753b17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 1 -- 172.21.15.70:0/667150827 >> 172.21.15.70:0/667150827 conn(0x55fa752d7080 msgr2=0x55fa753b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 1 -- 172.21.15.70:0/667150827 shutdown_connections 2024-06-04T05:09:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 1 -- 172.21.15.70:0/667150827 wait complete. 2024-06-04T05:09:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 1 -- start start 2024-06-04T05:09:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 1 Processor -- start 2024-06-04T05:09:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753a9860 0x55fa753ab690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 0x55fa753abbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753adf60 0x55fa753ac110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f07401af4a0 con 0x55fa753a9860 2024-06-04T05:09:57.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f07401aed70 con 0x55fa753adf60 2024-06-04T05:09:57.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f07401abf30 con 0x55fa753aa210 2024-06-04T05:09:57.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f0752f9eb80 10 client.0 _ll_register_callbacks cb 0x55fa75293920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:57.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f07464a5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753a9860 0x55fa753ab690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:57.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f07464a5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753a9860 0x55fa753ab690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50978/0 (socket says 172.21.15.70:50978) 2024-06-04T05:09:57.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.989+0000 7f07464a5700 1 -- 172.21.15.70:0/2312425921 learned_addr learned my addr 172.21.15.70:0/2312425921 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:57.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.990+0000 7f07464a5700 1 -- 172.21.15.70:0/2312425921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 msgr2=0x55fa753abbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:57.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.990+0000 7f07464a5700 1 --2- 172.21.15.70:0/2312425921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 0x55fa753abbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:57.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.990+0000 7f07464a5700 1 -- 172.21.15.70:0/2312425921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753adf60 msgr2=0x55fa753ac110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:57.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.990+0000 7f07464a5700 1 --2- 172.21.15.70:0/2312425921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753adf60 0x55fa753ac110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:57.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.990+0000 7f07464a5700 1 -- 172.21.15.70:0/2312425921 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa7542ca30 con 0x55fa753a9860 2024-06-04T05:09:57.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.990+0000 7f07464a5700 1 --2- 172.21.15.70:0/2312425921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753a9860 0x55fa753ab690 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f073c0e1fe0 tx=0x7f073c004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58dde3d4bbbb3c53 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:57.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.990+0000 7f0729ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:57.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.990+0000 7f0729ffb700 1 -- 172.21.15.70:0/2312425921 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073c0ad430 con 0x55fa753a9860 2024-06-04T05:09:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.990+0000 7f0729ffb700 1 -- 172.21.15.70:0/2312425921 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f073c0a5670 con 0x55fa753a9860 2024-06-04T05:09:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.990+0000 7f0752f9eb80 1 -- 172.21.15.70:0/2312425921 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0738019070 con 0x55fa753a9860 2024-06-04T05:09:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.990+0000 7f0729ffb700 1 -- 172.21.15.70:0/2312425921 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f073c0a84b0 con 0x55fa753a9860 2024-06-04T05:09:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.990+0000 7f0752f9eb80 10 client.18861 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.990+0000 7f0752f9eb80 1 -- 172.21.15.70:0/2312425921 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fa7542dd10 con 0x55fa753a9860 2024-06-04T05:09:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.990+0000 7f0729ffb700 1 -- 172.21.15.70:0/2312425921 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0738019070 con 0x55fa753a9860 2024-06-04T05:09:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.991+0000 7f0729ffb700 1 -- 172.21.15.70:0/2312425921 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f073c0a7d50 con 0x55fa753a9860 2024-06-04T05:09:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.991+0000 7f0752f9eb80 10 client.18861 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.991+0000 7f0752f9eb80 10 client.18861 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:09:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.991+0000 7f0752f9eb80 20 client.18861 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.991+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/2312425921 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55fa7542e450 0x55fa754308d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.991+0000 7f0752f9eb80 4 client.18861 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:09:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.991+0000 7f0752f9eb80 1 -- 172.21.15.70:0/2312425921 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fa7530abe0 con 0x55fa7542e450 2024-06-04T05:09:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.991+0000 7f0745ca4700 1 --2- 172.21.15.70:0/2312425921 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55fa7542e450 0x55fa754308d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.991+0000 7f0745ca4700 1 --2- 172.21.15.70:0/2312425921 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55fa7542e450 0x55fa754308d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=be74e73ce021df2f server_cookie=be10df6a1336e470 in_seq=0 out_seq=0 2024-06-04T05:09:57.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.991+0000 7f0729ffb700 10 client.18861 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:09:57.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.991+0000 7f0729ffb700 1 -- 172.21.15.70:0/2312425921 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fa7530abe0 con 0x55fa7542e450 2024-06-04T05:09:57.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.991+0000 7f0729ffb700 10 client.18861 handle_command_reply: tid=0 2024-06-04T05:09:57.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.991+0000 7f0752f9eb80 1 client.18861 shutdown 2024-06-04T05:09:57.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0752f9eb80 1 -- 172.21.15.70:0/2312425921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753a9860 msgr2=0x55fa753ab690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:57.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/2312425921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753a9860 0x55fa753ab690 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f073c0e1fe0 tx=0x7f073c004860 comp rx=0 tx=0).stop 2024-06-04T05:09:57.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0752f9eb80 1 -- 172.21.15.70:0/2312425921 shutdown_connections 2024-06-04T05:09:57.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/2312425921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa753adf60 0x55fa753ac110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:57.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/2312425921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa753a9860 0x55fa753ab690 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:57.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/2312425921 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55fa7542e450 0x55fa754308d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:57.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0752f9eb80 1 --2- 172.21.15.70:0/2312425921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fa753aa210 0x55fa753abbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:57.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0729ffb700 0 client.18861 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:09:57.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0752f9eb80 1 -- 172.21.15.70:0/2312425921 >> 172.21.15.70:0/2312425921 conn(0x55fa752d7080 msgr2=0x55fa753a8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:57.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0752f9eb80 1 -- 172.21.15.70:0/2312425921 shutdown_connections 2024-06-04T05:09:57.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0752f9eb80 1 -- 172.21.15.70:0/2312425921 wait complete. 2024-06-04T05:09:57.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0752f9eb80 20 client.18861 trim_cache size 0 max 16384 2024-06-04T05:09:57.004 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:09:57.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0748709700 1 -- 172.21.15.70:0/1133729529 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f072c0b29f0 msgr2=0x7f072c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:57.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0748709700 1 --2- 172.21.15.70:0/1133729529 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f072c0b29f0 0x7f072c0b4eb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0730009750 tx=0x7f073001a040 comp rx=0 tx=0).stop 2024-06-04T05:09:57.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0748709700 1 -- 172.21.15.70:0/1133729529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0740101f00 msgr2=0x7f074019f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:57.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0748709700 1 --2- 172.21.15.70:0/1133729529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0740101f00 0x7f074019f920 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f073c00b500 tx=0x7f073c00b810 comp rx=0 tx=0).stop 2024-06-04T05:09:57.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0748709700 1 -- 172.21.15.70:0/1133729529 shutdown_connections 2024-06-04T05:09:57.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0748709700 1 --2- 172.21.15.70:0/1133729529 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f072c0b29f0 0x7f072c0b4eb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:57.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.992+0000 7f0748709700 1 --2- 172.21.15.70:0/1133729529 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0740108620 0x7f0740199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:57.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.993+0000 7f0748709700 1 --2- 172.21.15.70:0/1133729529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0740101f00 0x7f074019f920 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:57.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.993+0000 7f0748709700 1 --2- 172.21.15.70:0/1133729529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07401028c0 0x7f074019fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:57.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.993+0000 7f0748709700 1 -- 172.21.15.70:0/1133729529 >> 172.21.15.70:0/1133729529 conn(0x7f0740077780 msgr2=0x7f0740100ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:57.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.993+0000 7f0748709700 1 -- 172.21.15.70:0/1133729529 shutdown_connections 2024-06-04T05:09:57.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:56.993+0000 7f0748709700 1 -- 172.21.15.70:0/1133729529 wait complete. 2024-06-04T05:09:57.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:57 smithi070 ceph-mon[32631]: pgmap v396: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-06-04T05:09:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:57 smithi112 ceph-mon[36453]: pgmap v396: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-06-04T05:09:57.741 INFO:tasks.workunit.client.0.smithi070.stdout: 1 205871 1.63 MB/sec execute 530 sec latency 31032.798 ms 2024-06-04T05:09:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:57 smithi098 ceph-mon[35569]: pgmap v396: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-06-04T05:09:57.868 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:09:58.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.105+0000 7f6af2427700 1 -- 172.21.15.70:0/3059045014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6aec06bab0 msgr2=0x7f6aec06bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:58.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.105+0000 7f6af2427700 1 --2- 172.21.15.70:0/3059045014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6aec06bab0 0x7f6aec06bf30 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0009990 tx=0x7f6ae0009ca0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.105+0000 7f6af2427700 1 -- 172.21.15.70:0/3059045014 shutdown_connections 2024-06-04T05:09:58.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.105+0000 7f6af2427700 1 --2- 172.21.15.70:0/3059045014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec06c470 0x7f6aec1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.105+0000 7f6af2427700 1 --2- 172.21.15.70:0/3059045014 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6aec1095a0 0x7f6aec109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.105+0000 7f6af2427700 1 --2- 172.21.15.70:0/3059045014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6aec06bab0 0x7f6aec06bf30 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.105+0000 7f6af2427700 1 -- 172.21.15.70:0/3059045014 >> 172.21.15.70:0/3059045014 conn(0x7f6aec0793b0 msgr2=0x7f6aec0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:58.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.105+0000 7f6af2427700 1 -- 172.21.15.70:0/3059045014 shutdown_connections 2024-06-04T05:09:58.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.105+0000 7f6af2427700 1 -- 172.21.15.70:0/3059045014 wait complete. 2024-06-04T05:09:58.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.106+0000 7f6af2427700 1 Processor -- start 2024-06-04T05:09:58.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.106+0000 7f6af2427700 1 -- start start 2024-06-04T05:09:58.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.106+0000 7f6af2427700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6aec06bab0 0x7f6aec19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.106+0000 7f6af2427700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec06c470 0x7f6aec19fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.106+0000 7f6af2427700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6aec1095a0 0x7f6aec199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.106+0000 7f6af2427700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec113580 con 0x7f6aec1095a0 2024-06-04T05:09:58.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.106+0000 7f6af2427700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec113400 con 0x7f6aec06c470 2024-06-04T05:09:58.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.106+0000 7f6af2427700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec113700 con 0x7f6aec06bab0 2024-06-04T05:09:58.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.106+0000 7f6af09c4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6aec1095a0 0x7f6aec199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.106+0000 7f6af09c4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6aec1095a0 0x7f6aec199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51000/0 (socket says 172.21.15.70:51000) 2024-06-04T05:09:58.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.106+0000 7f6af09c4700 1 -- 172.21.15.70:0/741560608 learned_addr learned my addr 172.21.15.70:0/741560608 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:58.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.106+0000 7f6aeb7fe700 1 --2- 172.21.15.70:0/741560608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec06c470 0x7f6aec19fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.107+0000 7f6aebfff700 1 --2- 172.21.15.70:0/741560608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6aec06bab0 0x7f6aec19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.107+0000 7f6af09c4700 1 -- 172.21.15.70:0/741560608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6aec06bab0 msgr2=0x7f6aec19f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:58.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.107+0000 7f6af09c4700 1 --2- 172.21.15.70:0/741560608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6aec06bab0 0x7f6aec19f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.107+0000 7f6af09c4700 1 -- 172.21.15.70:0/741560608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec06c470 msgr2=0x7f6aec19fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:58.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.107+0000 7f6af09c4700 1 --2- 172.21.15.70:0/741560608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec06c470 0x7f6aec19fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.107+0000 7f6af09c4700 1 -- 172.21.15.70:0/741560608 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aec19a080 con 0x7f6aec1095a0 2024-06-04T05:09:58.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.107+0000 7f6af09c4700 1 --2- 172.21.15.70:0/741560608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6aec1095a0 0x7f6aec199920 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4007d70 tx=0x7f6ae400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:58.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.107+0000 7f6ae97fa700 1 -- 172.21.15.70:0/741560608 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae4016020 con 0x7f6aec1095a0 2024-06-04T05:09:58.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.107+0000 7f6af2427700 1 -- 172.21.15.70:0/741560608 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aec19a370 con 0x7f6aec1095a0 2024-06-04T05:09:58.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.107+0000 7f6ae97fa700 1 -- 172.21.15.70:0/741560608 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ae400ac90 con 0x7f6aec1095a0 2024-06-04T05:09:58.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.107+0000 7f6ae97fa700 1 -- 172.21.15.70:0/741560608 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae40115e0 con 0x7f6aec1095a0 2024-06-04T05:09:58.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.107+0000 7f6af2427700 1 -- 172.21.15.70:0/741560608 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aec1a6630 con 0x7f6aec1095a0 2024-06-04T05:09:58.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.110+0000 7f6ae97fa700 1 -- 172.21.15.70:0/741560608 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f6ae4019020 con 0x7f6aec1095a0 2024-06-04T05:09:58.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.110+0000 7f6ae97fa700 1 --2- 172.21.15.70:0/741560608 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ad40b2a70 0x7f6ad40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.111+0000 7f6aebfff700 1 --2- 172.21.15.70:0/741560608 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ad40b2a70 0x7f6ad40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.111+0000 7f6ae97fa700 1 -- 172.21.15.70:0/741560608 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f6ae40e2440 con 0x7f6aec1095a0 2024-06-04T05:09:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.111+0000 7f6aebfff700 1 --2- 172.21.15.70:0/741560608 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ad40b2a70 0x7f6ad40b4f30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6aec19aa60 tx=0x7f6adc00b3c0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:58.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.112+0000 7f6afccbcb80 1 Processor -- start 2024-06-04T05:09:58.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.112+0000 7f6afccbcb80 1 -- start start 2024-06-04T05:09:58.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6afccbcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6e5c0 0x561d0ff6e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6afccbcb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff69cf0 0x561d0ff6a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6afccbcb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff6a6b0 0x561d0ff764e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6afccbcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561d0fec4d90 con 0x561d0ff6e5c0 2024-06-04T05:09:58.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6afccbcb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561d0feb4930 con 0x561d0ff6a6b0 2024-06-04T05:09:58.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6afccbcb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561d0feb8970 con 0x561d0ff69cf0 2024-06-04T05:09:58.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6aebfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6e5c0 0x561d0ff6e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6aebfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6e5c0 0x561d0ff6e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51006/0 (socket says 172.21.15.70:51006) 2024-06-04T05:09:58.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6aebfff700 1 -- 172.21.15.70:0/2735670187 learned_addr learned my addr 172.21.15.70:0/2735670187 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:58.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6aeb7fe700 1 --2- 172.21.15.70:0/2735670187 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff69cf0 0x561d0ff6a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6af09c4700 1 --2- 172.21.15.70:0/2735670187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff6a6b0 0x561d0ff764e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6aebfff700 1 -- 172.21.15.70:0/2735670187 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff69cf0 msgr2=0x561d0ff6a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:58.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6aebfff700 1 --2- 172.21.15.70:0/2735670187 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff69cf0 0x561d0ff6a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6aebfff700 1 -- 172.21.15.70:0/2735670187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff6a6b0 msgr2=0x561d0ff764e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:58.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6aebfff700 1 --2- 172.21.15.70:0/2735670187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff6a6b0 0x561d0ff764e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6aebfff700 1 -- 172.21.15.70:0/2735670187 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d0fe049d0 con 0x561d0ff6e5c0 2024-06-04T05:09:58.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6aeb7fe700 1 --2- 172.21.15.70:0/2735670187 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff69cf0 0x561d0ff6a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:09:58.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.113+0000 7f6aebfff700 1 --2- 172.21.15.70:0/2735670187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6e5c0 0x561d0ff6e9a0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f6adc005e10 tx=0x7f6adc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e88ecfb2cd9421c7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:58.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.114+0000 7f6ad27fc700 1 -- 172.21.15.70:0/2735670187 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc0118d0 con 0x561d0ff6e5c0 2024-06-04T05:09:58.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.114+0000 7f6ad27fc700 1 -- 172.21.15.70:0/2735670187 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6adc01e420 con 0x561d0ff6e5c0 2024-06-04T05:09:58.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.114+0000 7f6ad27fc700 1 -- 172.21.15.70:0/2735670187 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc01c5b0 con 0x561d0ff6e5c0 2024-06-04T05:09:58.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.115+0000 7f6afccbcb80 1 -- 172.21.15.70:0/2735670187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6e5c0 msgr2=0x561d0ff6e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:58.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.115+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/2735670187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6e5c0 0x561d0ff6e9a0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f6adc005e10 tx=0x7f6adc013000 comp rx=0 tx=0).stop 2024-06-04T05:09:58.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.115+0000 7f6afccbcb80 1 -- 172.21.15.70:0/2735670187 shutdown_connections 2024-06-04T05:09:58.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.115+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/2735670187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff6a6b0 0x561d0ff764e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.115+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/2735670187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6e5c0 0x561d0ff6e9a0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.115+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/2735670187 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff69cf0 0x561d0ff6a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.115+0000 7f6afccbcb80 1 -- 172.21.15.70:0/2735670187 >> 172.21.15.70:0/2735670187 conn(0x561d0fe93180 msgr2=0x561d0ff601f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:58.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.115+0000 7f6afccbcb80 1 -- 172.21.15.70:0/2735670187 shutdown_connections 2024-06-04T05:09:58.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.115+0000 7f6afccbcb80 1 -- 172.21.15.70:0/2735670187 wait complete. 2024-06-04T05:09:58.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.115+0000 7f6afccbcb80 1 -- start start 2024-06-04T05:09:58.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.116+0000 7f6afccbcb80 1 Processor -- start 2024-06-04T05:09:58.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.116+0000 7f6afccbcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff69cf0 0x561d0ff760f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.116+0000 7f6afccbcb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff6a6b0 0x561d0ff733f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.116+0000 7f6aebfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff69cf0 0x561d0ff760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.116+0000 7f6afccbcb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff6e5c0 0x561d0ff73930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.116+0000 7f6aebfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff69cf0 0x561d0ff760f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51014/0 (socket says 172.21.15.70:51014) 2024-06-04T05:09:58.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.116+0000 7f6aebfff700 1 -- 172.21.15.70:0/3818751085 learned_addr learned my addr 172.21.15.70:0/3818751085 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:58.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.116+0000 7f6afccbcb80 1 -- 172.21.15.70:0/3818751085 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561d0fdbb0a0 con 0x561d0ff69cf0 2024-06-04T05:09:58.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.116+0000 7f6afccbcb80 1 -- 172.21.15.70:0/3818751085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561d0fe568c0 con 0x561d0ff6e5c0 2024-06-04T05:09:58.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.116+0000 7f6afccbcb80 1 -- 172.21.15.70:0/3818751085 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561d0ff76b40 con 0x561d0ff6a6b0 2024-06-04T05:09:58.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.116+0000 7f6aeb7fe700 1 --2- 172.21.15.70:0/3818751085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff6a6b0 0x561d0ff733f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.117+0000 7f6afccbcb80 10 client.0 _ll_register_callbacks cb 0x561d0fe4fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:58.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.116+0000 7f6af09c4700 1 --2- 172.21.15.70:0/3818751085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff6e5c0 0x561d0ff73930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.117+0000 7f6aebfff700 1 -- 172.21.15.70:0/3818751085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff6a6b0 msgr2=0x561d0ff733f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:58.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.117+0000 7f6aebfff700 1 --2- 172.21.15.70:0/3818751085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff6a6b0 0x561d0ff733f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.117+0000 7f6aebfff700 1 -- 172.21.15.70:0/3818751085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff6e5c0 msgr2=0x561d0ff73930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:58.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.117+0000 7f6aebfff700 1 --2- 172.21.15.70:0/3818751085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff6e5c0 0x561d0ff73930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.117+0000 7f6aebfff700 1 -- 172.21.15.70:0/3818751085 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d0fe53790 con 0x561d0ff69cf0 2024-06-04T05:09:58.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.117+0000 7f6afccbcb80 1 -- 172.21.15.70:0/3818751085 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6adc01e420 con 0x561d0ff69cf0 2024-06-04T05:09:58.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.117+0000 7f6af09c4700 1 --2- 172.21.15.70:0/3818751085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff6e5c0 0x561d0ff73930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:09:58.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.117+0000 7f6aebfff700 1 --2- 172.21.15.70:0/3818751085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff69cf0 0x561d0ff760f0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00b4a0 tx=0x7f6adc01e5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db0a95d3bbd96b3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:58.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.117+0000 7f6acb7fe700 10 client.18873 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:09:58.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.117+0000 7f6acb7fe700 1 -- 172.21.15.70:0/3818751085 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc035060 con 0x561d0ff69cf0 2024-06-04T05:09:58.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.117+0000 7f6acb7fe700 1 -- 172.21.15.70:0/3818751085 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f6adc0118d0 con 0x561d0ff69cf0 2024-06-04T05:09:58.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.117+0000 7f6acb7fe700 1 -- 172.21.15.70:0/3818751085 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6adc011b10 con 0x561d0ff69cf0 2024-06-04T05:09:58.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.117+0000 7f6afccbcb80 10 client.18873 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:58.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.117+0000 7f6afccbcb80 1 -- 172.21.15.70:0/3818751085 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({fsmap=38}) -- 0x561d0ff810c0 con 0x561d0ff69cf0 2024-06-04T05:09:58.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.118+0000 7f6acb7fe700 1 -- 172.21.15.70:0/3818751085 <== mon.0 v2:172.21.15.70:3300/0 4 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f6adc011cb0 con 0x561d0ff69cf0 2024-06-04T05:09:58.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.118+0000 7f6afccbcb80 10 client.18873 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:58.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.118+0000 7f6afccbcb80 10 client.18873 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:09:58.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.118+0000 7f6afccbcb80 20 client.18873 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:58.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.118+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/3818751085 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561d0ff83a00 0x561d0ff85ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.118+0000 7f6afccbcb80 4 client.18873 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:09:58.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.118+0000 7f6afccbcb80 1 -- 172.21.15.70:0/3818751085 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d0ff31ae0 con 0x561d0ff83a00 2024-06-04T05:09:58.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.118+0000 7f6aeb7fe700 1 --2- 172.21.15.70:0/3818751085 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561d0ff83a00 0x561d0ff85ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.119+0000 7f6acb7fe700 10 client.18873 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:58.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.119+0000 7f6aeb7fe700 1 --2- 172.21.15.70:0/3818751085 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561d0ff83a00 0x561d0ff85ec0 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8027545cab8c073c server_cookie=8fd96b5256c69c72 in_seq=0 out_seq=0 2024-06-04T05:09:58.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.119+0000 7f6acb7fe700 1 -- 172.21.15.70:0/3818751085 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561d0ff31ae0 con 0x561d0ff83a00 2024-06-04T05:09:58.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.119+0000 7f6acb7fe700 10 client.18873 handle_command_reply: tid=0 2024-06-04T05:09:58.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.120+0000 7f6afccbcb80 1 client.18873 shutdown 2024-06-04T05:09:58.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.120+0000 7f6afccbcb80 1 -- 172.21.15.70:0/3818751085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff69cf0 msgr2=0x561d0ff760f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:58.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.120+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/3818751085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff69cf0 0x561d0ff760f0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00b4a0 tx=0x7f6adc01e5d0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.120+0000 7f6afccbcb80 1 -- 172.21.15.70:0/3818751085 shutdown_connections 2024-06-04T05:09:58.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.120+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/3818751085 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561d0ff83a00 0x561d0ff85ec0 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.120+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/3818751085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff6e5c0 0x561d0ff73930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.120+0000 7f6acb7fe700 0 client.18873 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:58.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.120+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/3818751085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff69cf0 0x561d0ff760f0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.120+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/3818751085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff6a6b0 0x561d0ff733f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.120+0000 7f6afccbcb80 1 -- 172.21.15.70:0/3818751085 >> 172.21.15.70:0/3818751085 conn(0x561d0fe93180 msgr2=0x561d0ff6fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.120+0000 7f6afccbcb80 1 -- 172.21.15.70:0/3818751085 shutdown_connections 2024-06-04T05:09:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.120+0000 7f6afccbcb80 1 -- 172.21.15.70:0/3818751085 wait complete. 2024-06-04T05:09:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.120+0000 7f6afccbcb80 20 client.18873 trim_cache size 0 max 16384 2024-06-04T05:09:58.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.150+0000 7f6afccbcb80 1 Processor -- start 2024-06-04T05:09:58.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.150+0000 7f6afccbcb80 1 -- start start 2024-06-04T05:09:58.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.150+0000 7f6afccbcb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff69cf0 0x561d0ff6dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.150+0000 7f6afccbcb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff6a6b0 0x561d0ff75c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.150+0000 7f6afccbcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6e5c0 0x561d0ff8b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.150+0000 7f6afccbcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1abc20 con 0x561d0ff6e5c0 2024-06-04T05:09:58.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.150+0000 7f6afccbcb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1ad880 con 0x561d0ff69cf0 2024-06-04T05:09:58.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.150+0000 7f6afccbcb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1ac2f0 con 0x561d0ff6a6b0 2024-06-04T05:09:58.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.151+0000 7f6af09c4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6e5c0 0x561d0ff8b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.151+0000 7f6af09c4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6e5c0 0x561d0ff8b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51026/0 (socket says 172.21.15.70:51026) 2024-06-04T05:09:58.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.151+0000 7f6af09c4700 1 -- 172.21.15.70:0/3596649563 learned_addr learned my addr 172.21.15.70:0/3596649563 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:58.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.151+0000 7f6aeb7fe700 1 --2- 172.21.15.70:0/3596649563 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff6a6b0 0x561d0ff75c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.151+0000 7f6aebfff700 1 --2- 172.21.15.70:0/3596649563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff69cf0 0x561d0ff6dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.151+0000 7f6af09c4700 1 -- 172.21.15.70:0/3596649563 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff6a6b0 msgr2=0x561d0ff75c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:58.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.151+0000 7f6af09c4700 1 --2- 172.21.15.70:0/3596649563 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff6a6b0 0x561d0ff75c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.151+0000 7f6af09c4700 1 -- 172.21.15.70:0/3596649563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff69cf0 msgr2=0x561d0ff6dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:58.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.151+0000 7f6af09c4700 1 --2- 172.21.15.70:0/3596649563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff69cf0 0x561d0ff6dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.151+0000 7f6af09c4700 1 -- 172.21.15.70:0/3596649563 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d0ff71960 con 0x561d0ff6e5c0 2024-06-04T05:09:58.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.151+0000 7f6aeb7fe700 1 --2- 172.21.15.70:0/3596649563 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff6a6b0 0x561d0ff75c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:09:58.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.151+0000 7f6af09c4700 1 --2- 172.21.15.70:0/3596649563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6e5c0 0x561d0ff8b790 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f6ae40a5880 tx=0x7f6ae40a9150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7db0a883bb8b555 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:58.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.151+0000 7f6ad1ffb700 1 -- 172.21.15.70:0/3596649563 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae40a9310 con 0x561d0ff6e5c0 2024-06-04T05:09:58.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.151+0000 7f6ad1ffb700 1 -- 172.21.15.70:0/3596649563 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ae40a9f00 con 0x561d0ff6e5c0 2024-06-04T05:09:58.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.151+0000 7f6ad1ffb700 1 -- 172.21.15.70:0/3596649563 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae40b21d0 con 0x561d0ff6e5c0 2024-06-04T05:09:58.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.152+0000 7f6afccbcb80 1 -- 172.21.15.70:0/3596649563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6e5c0 msgr2=0x561d0ff8b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:58.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.152+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/3596649563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6e5c0 0x561d0ff8b790 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f6ae40a5880 tx=0x7f6ae40a9150 comp rx=0 tx=0).stop 2024-06-04T05:09:58.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.152+0000 7f6afccbcb80 1 -- 172.21.15.70:0/3596649563 shutdown_connections 2024-06-04T05:09:58.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.152+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/3596649563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6e5c0 0x561d0ff8b790 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.152+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/3596649563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff69cf0 0x561d0ff6dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.152+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/3596649563 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff6a6b0 0x561d0ff75c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.152+0000 7f6afccbcb80 1 -- 172.21.15.70:0/3596649563 >> 172.21.15.70:0/3596649563 conn(0x561d0fe93180 msgr2=0x561d0ff65160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:58.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.152+0000 7f6afccbcb80 1 -- 172.21.15.70:0/3596649563 shutdown_connections 2024-06-04T05:09:58.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.152+0000 7f6afccbcb80 1 -- 172.21.15.70:0/3596649563 wait complete. 2024-06-04T05:09:58.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.153+0000 7f6afccbcb80 1 -- start start 2024-06-04T05:09:58.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.153+0000 7f6afccbcb80 1 Processor -- start 2024-06-04T05:09:58.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.153+0000 7f6afccbcb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff69cf0 0x561d0ff7bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.153+0000 7f6afccbcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6a6b0 0x561d0ff7c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.153+0000 7f6afccbcb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff6e5c0 0x561d0ff76d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.153+0000 7f6afccbcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1ad150 con 0x561d0ff6a6b0 2024-06-04T05:09:58.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.153+0000 7f6afccbcb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1aca20 con 0x561d0ff69cf0 2024-06-04T05:09:58.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.153+0000 7f6afccbcb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1a9be0 con 0x561d0ff6e5c0 2024-06-04T05:09:58.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.153+0000 7f6afccbcb80 10 client.0 _ll_register_callbacks cb 0x561d0fe4fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:09:58.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6aeb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6a6b0 0x561d0ff7c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6aeb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6a6b0 0x561d0ff7c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51042/0 (socket says 172.21.15.70:51042) 2024-06-04T05:09:58.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6aeb7fe700 1 -- 172.21.15.70:0/766775752 learned_addr learned my addr 172.21.15.70:0/766775752 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:09:58.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6af09c4700 1 --2- 172.21.15.70:0/766775752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff6e5c0 0x561d0ff76d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6aebfff700 1 --2- 172.21.15.70:0/766775752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff69cf0 0x561d0ff7bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6af09c4700 1 -- 172.21.15.70:0/766775752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff69cf0 msgr2=0x561d0ff7bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:58.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6af09c4700 1 --2- 172.21.15.70:0/766775752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff69cf0 0x561d0ff7bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6af09c4700 1 -- 172.21.15.70:0/766775752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6a6b0 msgr2=0x561d0ff7c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:09:58.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6af09c4700 1 --2- 172.21.15.70:0/766775752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6a6b0 0x561d0ff7c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6af09c4700 1 -- 172.21.15.70:0/766775752 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d0ff77520 con 0x561d0ff6e5c0 2024-06-04T05:09:58.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6aeb7fe700 1 --2- 172.21.15.70:0/766775752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6a6b0 0x561d0ff7c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:09:58.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6af09c4700 1 --2- 172.21.15.70:0/766775752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff6e5c0 0x561d0ff76d50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6ae40a99c0 tx=0x7f6ae40a9c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7b65b214b5bfa5a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:09:58.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6acb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:09:58.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6acb7fe700 1 -- 172.21.15.70:0/766775752 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae40a9800 con 0x561d0ff6e5c0 2024-06-04T05:09:58.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6afccbcb80 1 -- 172.21.15.70:0/766775752 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ae40a9f00 con 0x561d0ff6e5c0 2024-06-04T05:09:58.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6acb7fe700 1 -- 172.21.15.70:0/766775752 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f6ae40b23e0 con 0x561d0ff6e5c0 2024-06-04T05:09:58.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6acb7fe700 1 -- 172.21.15.70:0/766775752 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6ae40b2660 con 0x561d0ff6e5c0 2024-06-04T05:09:58.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.154+0000 7f6afccbcb80 10 client.27943 fetch_fsmap learned FSMap version 38 2024-06-04T05:09:58.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.155+0000 7f6afccbcb80 1 -- 172.21.15.70:0/766775752 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x561d0ffef0d0 con 0x561d0ff6e5c0 2024-06-04T05:09:58.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.155+0000 7f6acb7fe700 1 -- 172.21.15.70:0/766775752 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ae40c6020 con 0x561d0ff6e5c0 2024-06-04T05:09:58.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.155+0000 7f6acb7fe700 1 -- 172.21.15.70:0/766775752 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f6ae40a8b80 con 0x561d0ff6e5c0 2024-06-04T05:09:58.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.155+0000 7f6afccbcb80 10 client.27943 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:09:58.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.155+0000 7f6afccbcb80 10 client.27943 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:09:58.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.155+0000 7f6afccbcb80 20 client.27943 populate_metadata read hostname 'smithi070' 2024-06-04T05:09:58.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.155+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/766775752 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561d0ff83a00 0x561d0fff1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:09:58.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.155+0000 7f6afccbcb80 4 client.27943 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "a5e14b41", "release": true, "await": true}] 2024-06-04T05:09:58.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.155+0000 7f6afccbcb80 1 -- 172.21.15.70:0/766775752 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "a5e14b41", "release": true, "await": true}) -- 0x561d0fec6ce0 con 0x561d0ff83a00 2024-06-04T05:09:58.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.155+0000 7f6aebfff700 1 --2- 172.21.15.70:0/766775752 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561d0ff83a00 0x561d0fff1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:09:58.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.156+0000 7f6aebfff700 1 --2- 172.21.15.70:0/766775752 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561d0ff83a00 0x561d0fff1700 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36057328e781ff90 server_cookie=aa953f79aea241c9 in_seq=0 out_seq=0 2024-06-04T05:09:58.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.156+0000 7f6acb7fe700 10 client.27943 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:58.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.161+0000 7f6acb7fe700 1 -- 172.21.15.70:0/766775752 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x561d0fec6ce0 con 0x561d0ff83a00 2024-06-04T05:09:58.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.161+0000 7f6acb7fe700 10 client.27943 handle_command_reply: tid=0 2024-06-04T05:09:58.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.161+0000 7f6afccbcb80 1 client.27943 shutdown 2024-06-04T05:09:58.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.161+0000 7f6afccbcb80 1 -- 172.21.15.70:0/766775752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff6e5c0 msgr2=0x561d0ff76d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:58.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.161+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/766775752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff6e5c0 0x561d0ff76d50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6ae40a99c0 tx=0x7f6ae40a9c90 comp rx=0 tx=0).stop 2024-06-04T05:09:58.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.161+0000 7f6afccbcb80 1 -- 172.21.15.70:0/766775752 shutdown_connections 2024-06-04T05:09:58.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.161+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/766775752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d0ff6e5c0 0x561d0ff76d50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.161+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/766775752 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561d0ff83a00 0x561d0fff1700 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.161+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/766775752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d0ff69cf0 0x561d0ff7bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.161+0000 7f6acb7fe700 0 client.27943 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:09:58.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.162+0000 7f6afccbcb80 1 --2- 172.21.15.70:0/766775752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0ff6a6b0 0x561d0ff7c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.162+0000 7f6afccbcb80 1 -- 172.21.15.70:0/766775752 >> 172.21.15.70:0/766775752 conn(0x561d0fe93180 msgr2=0x561d0ff65160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:58.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.162+0000 7f6afccbcb80 1 -- 172.21.15.70:0/766775752 shutdown_connections 2024-06-04T05:09:58.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.162+0000 7f6afccbcb80 1 -- 172.21.15.70:0/766775752 wait complete. 2024-06-04T05:09:58.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.162+0000 7f6afccbcb80 20 client.27943 trim_cache size 0 max 16384 2024-06-04T05:09:58.165 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:09:58.165 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 38, 2024-06-04T05:09:58.166 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:09:58.166 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 40, 2024-06-04T05:09:58.166 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:09:58.166 INFO:teuthology.orchestra.run.smithi070.stdout: "a5e14b41": { 2024-06-04T05:09:58.166 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 40, 2024-06-04T05:09:58.166 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:09:58.166 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:09:58.166 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:09:58.166 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:09:58.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:09:58.166 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:09:58.166 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 149.4, 2024-06-04T05:09:58.166 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:09:58.167 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:09:58.167 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:09:58.167 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:09:58.167 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:09:58.167 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:09:58.167 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:09:58.167 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:09:58.167 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:09:58.167 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:09:58.167 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:09:58.167 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:09:58.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.163+0000 7f6af2427700 1 -- 172.21.15.70:0/741560608 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ad40b2a70 msgr2=0x7f6ad40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:58.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.163+0000 7f6af2427700 1 --2- 172.21.15.70:0/741560608 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ad40b2a70 0x7f6ad40b4f30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6aec19aa60 tx=0x7f6adc00b3c0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.163+0000 7f6af2427700 1 -- 172.21.15.70:0/741560608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6aec1095a0 msgr2=0x7f6aec199920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:09:58.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.163+0000 7f6af2427700 1 --2- 172.21.15.70:0/741560608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6aec1095a0 0x7f6aec199920 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4007d70 tx=0x7f6ae400a340 comp rx=0 tx=0).stop 2024-06-04T05:09:58.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.163+0000 7f6af2427700 1 -- 172.21.15.70:0/741560608 shutdown_connections 2024-06-04T05:09:58.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.163+0000 7f6af2427700 1 --2- 172.21.15.70:0/741560608 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ad40b2a70 0x7f6ad40b4f30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.163+0000 7f6af2427700 1 --2- 172.21.15.70:0/741560608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6aec1095a0 0x7f6aec199920 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.163+0000 7f6af2427700 1 --2- 172.21.15.70:0/741560608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6aec06bab0 0x7f6aec19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.163+0000 7f6af2427700 1 --2- 172.21.15.70:0/741560608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec06c470 0x7f6aec19fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:09:58.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.163+0000 7f6af2427700 1 -- 172.21.15.70:0/741560608 >> 172.21.15.70:0/741560608 conn(0x7f6aec0793b0 msgr2=0x7f6aec1012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:09:58.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.163+0000 7f6af2427700 1 -- 172.21.15.70:0/741560608 shutdown_connections 2024-06-04T05:09:58.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:09:58.163+0000 7f6af2427700 1 -- 172.21.15.70:0/741560608 wait complete. 2024-06-04T05:09:58.181 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:09:58.741 INFO:tasks.workunit.client.0.smithi070.stdout: 1 206046 1.63 MB/sec execute 531 sec latency 31460.918 ms 2024-06-04T05:09:58.837 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'a5e14b41', total seconds elapsed: 37.3 2024-06-04T05:09:58.837 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.4 2024-06-04T05:09:58.837 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:09:59.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:09:59 smithi070 ceph-mon[32631]: pgmap v397: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-06-04T05:09:59.538 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:09:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:09:59 smithi112 ceph-mon[36453]: pgmap v397: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-06-04T05:09:59.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 206325 1.63 MB/sec execute 532 sec latency 16.153 ms 2024-06-04T05:09:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:09:59 smithi098 ceph-mon[35569]: pgmap v397: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-06-04T05:10:00.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:00 smithi070 ceph-mon[32631]: overall HEALTH_OK 2024-06-04T05:10:00.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:00 smithi112 ceph-mon[36453]: overall HEALTH_OK 2024-06-04T05:10:00.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 206837 1.63 MB/sec execute 533 sec latency 15.000 ms 2024-06-04T05:10:00.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:00 smithi098 ceph-mon[35569]: overall HEALTH_OK 2024-06-04T05:10:01.350 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:01 smithi070 ceph-mon[32631]: pgmap v398: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-06-04T05:10:01.587 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:01 smithi112 ceph-mon[36453]: pgmap v398: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-06-04T05:10:01.741 INFO:tasks.workunit.client.0.smithi070.stdout: 1 207345 1.63 MB/sec execute 534 sec latency 18.998 ms 2024-06-04T05:10:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:01 smithi098 ceph-mon[35569]: pgmap v398: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-06-04T05:10:01.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.837+0000 7fd38b240700 1 -- 172.21.15.70:0/1675145064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd384102e00 msgr2=0x7fd38410f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:01.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.837+0000 7fd38b240700 1 --2- 172.21.15.70:0/1675145064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd384102e00 0x7fd38410f2f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fd37c00b0a0 tx=0x7fd37c00b3b0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.837+0000 7fd38b240700 1 -- 172.21.15.70:0/1675145064 shutdown_connections 2024-06-04T05:10:01.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.837+0000 7fd38b240700 1 --2- 172.21.15.70:0/1675145064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd384102e00 0x7fd38410f2f0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.837+0000 7fd38b240700 1 --2- 172.21.15.70:0/1675145064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd384108520 0x7fd384108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.837+0000 7fd38b240700 1 --2- 172.21.15.70:0/1675145064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd384102440 0x7fd3841028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.837+0000 7fd38b240700 1 -- 172.21.15.70:0/1675145064 >> 172.21.15.70:0/1675145064 conn(0x7fd3840793f0 msgr2=0x7fd384079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:01.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.837+0000 7fd38b240700 1 -- 172.21.15.70:0/1675145064 shutdown_connections 2024-06-04T05:10:01.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.838+0000 7fd38b240700 1 -- 172.21.15.70:0/1675145064 wait complete. 2024-06-04T05:10:01.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.838+0000 7fd38b240700 1 Processor -- start 2024-06-04T05:10:01.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.838+0000 7fd38b240700 1 -- start start 2024-06-04T05:10:01.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.838+0000 7fd38b240700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd384102440 0x7fd38419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.838+0000 7fd38b240700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd384102e00 0x7fd38419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.838+0000 7fd38b240700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd384108520 0x7fd384199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.839+0000 7fd38b240700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd3841124d0 con 0x7fd384108520 2024-06-04T05:10:01.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.839+0000 7fd38b240700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd384112350 con 0x7fd384102e00 2024-06-04T05:10:01.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.839+0000 7fd38b240700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd384112650 con 0x7fd384102440 2024-06-04T05:10:01.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.839+0000 7fd3897dd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd384108520 0x7fd384199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:01.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.839+0000 7fd3897dd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd384108520 0x7fd384199940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51058/0 (socket says 172.21.15.70:51058) 2024-06-04T05:10:01.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.839+0000 7fd3897dd700 1 -- 172.21.15.70:0/2754235275 learned_addr learned my addr 172.21.15.70:0/2754235275 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:01.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.839+0000 7fd388fdc700 1 --2- 172.21.15.70:0/2754235275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd384102440 0x7fd38419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:01.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.839+0000 7fd383fff700 1 --2- 172.21.15.70:0/2754235275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd384102e00 0x7fd38419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:01.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.839+0000 7fd388fdc700 1 -- 172.21.15.70:0/2754235275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd384102e00 msgr2=0x7fd38419fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:01.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.839+0000 7fd388fdc700 1 --2- 172.21.15.70:0/2754235275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd384102e00 0x7fd38419fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.839+0000 7fd388fdc700 1 -- 172.21.15.70:0/2754235275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd384108520 msgr2=0x7fd384199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:01.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.839+0000 7fd388fdc700 1 --2- 172.21.15.70:0/2754235275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd384108520 0x7fd384199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.839+0000 7fd388fdc700 1 -- 172.21.15.70:0/2754235275 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38419a0a0 con 0x7fd384102440 2024-06-04T05:10:01.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.839+0000 7fd388fdc700 1 --2- 172.21.15.70:0/2754235275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd384102440 0x7fd38419f730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd37800d950 tx=0x7fd37800dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:01.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.839+0000 7fd381ffb700 1 -- 172.21.15.70:0/2754235275 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd378016070 con 0x7fd384102440 2024-06-04T05:10:01.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.840+0000 7fd381ffb700 1 -- 172.21.15.70:0/2754235275 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd37800fd20 con 0x7fd384102440 2024-06-04T05:10:01.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.840+0000 7fd38b240700 1 -- 172.21.15.70:0/2754235275 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd38419a330 con 0x7fd384102440 2024-06-04T05:10:01.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.840+0000 7fd381ffb700 1 -- 172.21.15.70:0/2754235275 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37800e3b0 con 0x7fd384102440 2024-06-04T05:10:01.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.840+0000 7fd38b240700 1 -- 172.21.15.70:0/2754235275 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3841a6600 con 0x7fd384102440 2024-06-04T05:10:01.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.843+0000 7fd381ffb700 1 -- 172.21.15.70:0/2754235275 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fd378019070 con 0x7fd384102440 2024-06-04T05:10:01.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.843+0000 7fd381ffb700 1 --2- 172.21.15.70:0/2754235275 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd3740b2a70 0x7fd3740b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.843+0000 7fd383fff700 1 --2- 172.21.15.70:0/2754235275 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd3740b2a70 0x7fd3740b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:01.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.843+0000 7fd395ad5b80 1 Processor -- start 2024-06-04T05:10:01.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd381ffb700 1 -- 172.21.15.70:0/2754235275 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7fd3780e1f60 con 0x7fd384102440 2024-06-04T05:10:01.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd395ad5b80 1 -- start start 2024-06-04T05:10:01.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd395ad5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d3860 0x555a353d3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd395ad5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 0x555a353d7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd395ad5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d7f60 0x555a353d83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd395ad5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555a35332c90 con 0x555a353d7f60 2024-06-04T05:10:01.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd395ad5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555a35322830 con 0x555a353d3860 2024-06-04T05:10:01.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd395ad5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555a35326870 con 0x555a353d4210 2024-06-04T05:10:01.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd383fff700 1 --2- 172.21.15.70:0/2754235275 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd3740b2a70 0x7fd3740b4f30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd384079ba0 tx=0x7fd370009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:01.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd383fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 0x555a353d7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:01.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd383fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 0x555a353d7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:57412/0 (socket says 172.21.15.70:57412) 2024-06-04T05:10:01.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd3897dd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d7f60 0x555a353d83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:01.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd383fff700 1 -- 172.21.15.70:0/3961451266 learned_addr learned my addr 172.21.15.70:0/3961451266 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:01.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd388fdc700 1 --2- 172.21.15.70:0/3961451266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d3860 0x555a353d3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:01.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd383fff700 1 -- 172.21.15.70:0/3961451266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d3860 msgr2=0x555a353d3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:01.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd383fff700 1 --2- 172.21.15.70:0/3961451266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d3860 0x555a353d3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd383fff700 1 -- 172.21.15.70:0/3961451266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d7f60 msgr2=0x555a353d83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:01.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd383fff700 1 --2- 172.21.15.70:0/3961451266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d7f60 0x555a353d83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd383fff700 1 -- 172.21.15.70:0/3961451266 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a352728d0 con 0x555a353d4210 2024-06-04T05:10:01.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd3897dd700 1 --2- 172.21.15.70:0/3961451266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d7f60 0x555a353d83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:01.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.844+0000 7fd388fdc700 1 --2- 172.21.15.70:0/3961451266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d3860 0x555a353d3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:01.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.845+0000 7fd383fff700 1 --2- 172.21.15.70:0/3961451266 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 0x555a353d7940 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd370011d20 tx=0x7fd370014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ee6150a0b489103 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:01.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.845+0000 7fd36effd700 1 -- 172.21.15.70:0/3961451266 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37000eb20 con 0x555a353d4210 2024-06-04T05:10:01.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.845+0000 7fd36effd700 1 -- 172.21.15.70:0/3961451266 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd37000ecc0 con 0x555a353d4210 2024-06-04T05:10:01.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.845+0000 7fd36effd700 1 -- 172.21.15.70:0/3961451266 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd370025630 con 0x555a353d4210 2024-06-04T05:10:01.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.846+0000 7fd395ad5b80 1 -- 172.21.15.70:0/3961451266 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 msgr2=0x555a353d7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:01.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.846+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/3961451266 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 0x555a353d7940 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd370011d20 tx=0x7fd370014000 comp rx=0 tx=0).stop 2024-06-04T05:10:01.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.846+0000 7fd395ad5b80 1 -- 172.21.15.70:0/3961451266 shutdown_connections 2024-06-04T05:10:01.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.846+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/3961451266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d7f60 0x555a353d83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.846+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/3961451266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d3860 0x555a353d3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.846+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/3961451266 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 0x555a353d7940 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.846+0000 7fd395ad5b80 1 -- 172.21.15.70:0/3961451266 >> 172.21.15.70:0/3961451266 conn(0x555a35301080 msgr2=0x555a353cde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:01.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.846+0000 7fd395ad5b80 1 -- 172.21.15.70:0/3961451266 shutdown_connections 2024-06-04T05:10:01.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.846+0000 7fd395ad5b80 1 -- 172.21.15.70:0/3961451266 wait complete. 2024-06-04T05:10:01.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.846+0000 7fd395ad5b80 1 -- start start 2024-06-04T05:10:01.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.846+0000 7fd395ad5b80 1 Processor -- start 2024-06-04T05:10:01.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.847+0000 7fd395ad5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d3860 0x555a353df420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.847+0000 7fd395ad5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 0x555a353dc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.847+0000 7fd395ad5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d7f60 0x555a353dcc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.847+0000 7fd395ad5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555a35228fa0 con 0x555a353d3860 2024-06-04T05:10:01.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.847+0000 7fd395ad5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555a352c47c0 con 0x555a353d7f60 2024-06-04T05:10:01.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.847+0000 7fd395ad5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555a353e0120 con 0x555a353d4210 2024-06-04T05:10:01.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.847+0000 7fd395ad5b80 10 client.0 _ll_register_callbacks cb 0x555a352bd920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:01.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.847+0000 7fd388fdc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d3860 0x555a353df420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:01.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.847+0000 7fd383fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 0x555a353dc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:01.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.847+0000 7fd383fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 0x555a353dc720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:57414/0 (socket says 172.21.15.70:57414) 2024-06-04T05:10:01.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.847+0000 7fd383fff700 1 -- 172.21.15.70:0/859830251 learned_addr learned my addr 172.21.15.70:0/859830251 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:01.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.847+0000 7fd388fdc700 1 -- 172.21.15.70:0/859830251 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 msgr2=0x555a353dc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:01.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.847+0000 7fd388fdc700 1 --2- 172.21.15.70:0/859830251 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 0x555a353dc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.847+0000 7fd388fdc700 1 -- 172.21.15.70:0/859830251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d7f60 msgr2=0x555a353dcc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:01.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.847+0000 7fd388fdc700 1 --2- 172.21.15.70:0/859830251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d7f60 0x555a353dcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.847+0000 7fd388fdc700 1 -- 172.21.15.70:0/859830251 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a352c1690 con 0x555a353d3860 2024-06-04T05:10:01.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.848+0000 7fd388fdc700 1 --2- 172.21.15.70:0/859830251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d3860 0x555a353df420 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fd3780df680 tx=0x7fd3780a8c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a69456da8f55a2b8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:01.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.848+0000 7fd363fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:01.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.848+0000 7fd363fff700 1 -- 172.21.15.70:0/859830251 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3780c3060 con 0x555a353d3860 2024-06-04T05:10:01.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.848+0000 7fd363fff700 1 -- 172.21.15.70:0/859830251 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7fd3780b1df0 con 0x555a353d3860 2024-06-04T05:10:01.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.848+0000 7fd395ad5b80 1 -- 172.21.15.70:0/859830251 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd37000ecc0 con 0x555a353d3860 2024-06-04T05:10:01.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.848+0000 7fd363fff700 1 -- 172.21.15.70:0/859830251 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd3780ca050 con 0x555a353d3860 2024-06-04T05:10:01.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.848+0000 7fd395ad5b80 10 client.18891 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:01.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.848+0000 7fd395ad5b80 1 -- 172.21.15.70:0/859830251 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x555a353eed10 con 0x555a353d3860 2024-06-04T05:10:01.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.848+0000 7fd363fff700 1 -- 172.21.15.70:0/859830251 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd37000ecc0 con 0x555a353d3860 2024-06-04T05:10:01.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.848+0000 7fd363fff700 1 -- 172.21.15.70:0/859830251 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7fd3780ad170 con 0x555a353d3860 2024-06-04T05:10:01.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.848+0000 7fd395ad5b80 10 client.18891 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:01.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.848+0000 7fd395ad5b80 10 client.18891 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:10:01.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.848+0000 7fd395ad5b80 20 client.18891 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:01.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.848+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/859830251 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x555a353f12f0 0x555a353f3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.848+0000 7fd395ad5b80 4 client.18891 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:10:01.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.848+0000 7fd395ad5b80 1 -- 172.21.15.70:0/859830251 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a3539f9e0 con 0x555a353f12f0 2024-06-04T05:10:01.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.849+0000 7fd383fff700 1 --2- 172.21.15.70:0/859830251 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x555a353f12f0 0x555a353f3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:01.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.849+0000 7fd383fff700 1 --2- 172.21.15.70:0/859830251 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x555a353f12f0 0x555a353f3720 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=de1c25cd1b7e8a39 server_cookie=f1fddfa2dcd696e in_seq=0 out_seq=0 2024-06-04T05:10:01.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.849+0000 7fd363fff700 10 client.18891 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:10:01.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.850+0000 7fd363fff700 1 -- 172.21.15.70:0/859830251 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555a3539f9e0 con 0x555a353f12f0 2024-06-04T05:10:01.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.850+0000 7fd363fff700 10 client.18891 handle_command_reply: tid=0 2024-06-04T05:10:01.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.850+0000 7fd395ad5b80 1 client.18891 shutdown 2024-06-04T05:10:01.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.850+0000 7fd395ad5b80 1 -- 172.21.15.70:0/859830251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d3860 msgr2=0x555a353df420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:01.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.850+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/859830251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d3860 0x555a353df420 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fd3780df680 tx=0x7fd3780a8c80 comp rx=0 tx=0).stop 2024-06-04T05:10:01.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.850+0000 7fd395ad5b80 1 -- 172.21.15.70:0/859830251 shutdown_connections 2024-06-04T05:10:01.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.850+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/859830251 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x555a353f12f0 0x555a353f3720 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.850+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/859830251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d7f60 0x555a353dcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.850+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/859830251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d3860 0x555a353df420 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.850+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/859830251 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 0x555a353dc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.850+0000 7fd363fff700 0 client.18891 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:10:01.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.850+0000 7fd395ad5b80 1 -- 172.21.15.70:0/859830251 >> 172.21.15.70:0/859830251 conn(0x555a35301080 msgr2=0x555a353cd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:01.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.850+0000 7fd395ad5b80 1 -- 172.21.15.70:0/859830251 shutdown_connections 2024-06-04T05:10:01.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.850+0000 7fd395ad5b80 1 -- 172.21.15.70:0/859830251 wait complete. 2024-06-04T05:10:01.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.850+0000 7fd395ad5b80 20 client.18891 trim_cache size 0 max 16384 2024-06-04T05:10:01.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.867+0000 7fd395ad5b80 1 Processor -- start 2024-06-04T05:10:01.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.867+0000 7fd395ad5b80 1 -- start start 2024-06-04T05:10:01.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd395ad5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d3860 0x555a353de660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd395ad5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 0x555a353f8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd395ad5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d7f60 0x555a353f9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd395ad5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd3841abc20 con 0x555a353d3860 2024-06-04T05:10:01.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd395ad5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd3841ad880 con 0x555a353d7f60 2024-06-04T05:10:01.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd395ad5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd3841ac2f0 con 0x555a353d4210 2024-06-04T05:10:01.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd388fdc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d3860 0x555a353de660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:01.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd388fdc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d3860 0x555a353de660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51088/0 (socket says 172.21.15.70:51088) 2024-06-04T05:10:01.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd388fdc700 1 -- 172.21.15.70:0/705836319 learned_addr learned my addr 172.21.15.70:0/705836319 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:01.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd383fff700 1 --2- 172.21.15.70:0/705836319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 0x555a353f8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:01.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd3897dd700 1 --2- 172.21.15.70:0/705836319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d7f60 0x555a353f9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:01.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd388fdc700 1 -- 172.21.15.70:0/705836319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 msgr2=0x555a353f8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:01.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd388fdc700 1 --2- 172.21.15.70:0/705836319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 0x555a353f8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd388fdc700 1 -- 172.21.15.70:0/705836319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d7f60 msgr2=0x555a353f9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:01.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd388fdc700 1 --2- 172.21.15.70:0/705836319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d7f60 0x555a353f9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd388fdc700 1 -- 172.21.15.70:0/705836319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a353dacb0 con 0x555a353d3860 2024-06-04T05:10:01.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd383fff700 1 --2- 172.21.15.70:0/705836319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 0x555a353f8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:01.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.868+0000 7fd388fdc700 1 --2- 172.21.15.70:0/705836319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d3860 0x555a353de660 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fd378010970 tx=0x7fd37800fff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=235a4004e6d9765c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:01.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.869+0000 7fd36e7fc700 1 -- 172.21.15.70:0/705836319 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd378087020 con 0x555a353d3860 2024-06-04T05:10:01.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.869+0000 7fd36e7fc700 1 -- 172.21.15.70:0/705836319 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd3780ad5c0 con 0x555a353d3860 2024-06-04T05:10:01.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.869+0000 7fd36e7fc700 1 -- 172.21.15.70:0/705836319 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd378020040 con 0x555a353d3860 2024-06-04T05:10:01.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.869+0000 7fd395ad5b80 1 -- 172.21.15.70:0/705836319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d3860 msgr2=0x555a353de660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:01.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.869+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/705836319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d3860 0x555a353de660 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fd378010970 tx=0x7fd37800fff0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.869+0000 7fd395ad5b80 1 -- 172.21.15.70:0/705836319 shutdown_connections 2024-06-04T05:10:01.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.869+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/705836319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d7f60 0x555a353f9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.870+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/705836319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d3860 0x555a353de660 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.870+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/705836319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d4210 0x555a353f8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.870+0000 7fd395ad5b80 1 -- 172.21.15.70:0/705836319 >> 172.21.15.70:0/705836319 conn(0x555a35301080 msgr2=0x555a353ddf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:01.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.870+0000 7fd395ad5b80 1 -- 172.21.15.70:0/705836319 shutdown_connections 2024-06-04T05:10:01.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.870+0000 7fd395ad5b80 1 -- 172.21.15.70:0/705836319 wait complete. 2024-06-04T05:10:01.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.870+0000 7fd395ad5b80 1 -- start start 2024-06-04T05:10:01.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.870+0000 7fd395ad5b80 1 Processor -- start 2024-06-04T05:10:01.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.870+0000 7fd395ad5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d3860 0x555a353e5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.870+0000 7fd395ad5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d4210 0x555a353e2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.870+0000 7fd395ad5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d7f60 0x555a353e3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.870+0000 7fd395ad5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd3841ad150 con 0x555a353d4210 2024-06-04T05:10:01.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.870+0000 7fd395ad5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd3841aca20 con 0x555a353d7f60 2024-06-04T05:10:01.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.870+0000 7fd395ad5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd3841a9be0 con 0x555a353d3860 2024-06-04T05:10:01.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.871+0000 7fd395ad5b80 10 client.0 _ll_register_callbacks cb 0x555a352bd920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:01.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.871+0000 7fd383fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d4210 0x555a353e2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:01.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.871+0000 7fd388fdc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d3860 0x555a353e5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:01.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.871+0000 7fd3897dd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d7f60 0x555a353e3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:01.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.871+0000 7fd383fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d4210 0x555a353e2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51100/0 (socket says 172.21.15.70:51100) 2024-06-04T05:10:01.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.871+0000 7fd383fff700 1 -- 172.21.15.70:0/3653555425 learned_addr learned my addr 172.21.15.70:0/3653555425 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:01.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.871+0000 7fd3897dd700 1 -- 172.21.15.70:0/3653555425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d3860 msgr2=0x555a353e5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:01.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.871+0000 7fd3897dd700 1 --2- 172.21.15.70:0/3653555425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d3860 0x555a353e5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.871+0000 7fd3897dd700 1 -- 172.21.15.70:0/3653555425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d4210 msgr2=0x555a353e2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:01.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.871+0000 7fd3897dd700 1 --2- 172.21.15.70:0/3653555425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d4210 0x555a353e2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.871+0000 7fd3897dd700 1 -- 172.21.15.70:0/3653555425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a353e3c60 con 0x555a353d7f60 2024-06-04T05:10:01.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.871+0000 7fd383fff700 1 --2- 172.21.15.70:0/3653555425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d4210 0x555a353e2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:01.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.871+0000 7fd3897dd700 1 --2- 172.21.15.70:0/3653555425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d7f60 0x555a353e3420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd37c009610 tx=0x7fd37c004c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a05f87833dfe917 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:01.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.871+0000 7fd363fff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:10:01.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.871+0000 7fd363fff700 1 -- 172.21.15.70:0/3653555425 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c0076f0 con 0x555a353d7f60 2024-06-04T05:10:01.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.871+0000 7fd395ad5b80 1 -- 172.21.15.70:0/3653555425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3780ad5c0 con 0x555a353d7f60 2024-06-04T05:10:01.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.871+0000 7fd363fff700 1 -- 172.21.15.70:0/3653555425 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7fd37c007890 con 0x555a353d7f60 2024-06-04T05:10:01.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.872+0000 7fd363fff700 1 -- 172.21.15.70:0/3653555425 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd37c007a90 con 0x555a353d7f60 2024-06-04T05:10:01.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.872+0000 7fd395ad5b80 10 client.27842 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:01.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.872+0000 7fd395ad5b80 1 -- 172.21.15.70:0/3653555425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x555a3545bfd0 con 0x555a353d7f60 2024-06-04T05:10:01.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.872+0000 7fd363fff700 1 -- 172.21.15.70:0/3653555425 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd3780ad5c0 con 0x555a353d7f60 2024-06-04T05:10:01.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.872+0000 7fd363fff700 1 -- 172.21.15.70:0/3653555425 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7fd37c011070 con 0x555a353d7f60 2024-06-04T05:10:01.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.872+0000 7fd395ad5b80 10 client.27842 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:01.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.872+0000 7fd395ad5b80 10 client.27842 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:10:01.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.872+0000 7fd395ad5b80 20 client.27842 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:01.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.872+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/3653555425 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x555a3545c6f0 0x555a3545eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:01.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.872+0000 7fd395ad5b80 4 client.27842 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:10:01.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.872+0000 7fd395ad5b80 1 -- 172.21.15.70:0/3653555425 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555a35334be0 con 0x555a3545c6f0 2024-06-04T05:10:01.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.872+0000 7fd388fdc700 1 --2- 172.21.15.70:0/3653555425 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x555a3545c6f0 0x555a3545eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:01.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.872+0000 7fd388fdc700 1 --2- 172.21.15.70:0/3653555425 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x555a3545c6f0 0x555a3545eae0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1ae555f42147536b server_cookie=e4a691c754c2b532 in_seq=0 out_seq=0 2024-06-04T05:10:01.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.872+0000 7fd363fff700 10 client.27842 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:10:01.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.873+0000 7fd363fff700 1 -- 172.21.15.70:0/3653555425 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555a35334be0 con 0x555a3545c6f0 2024-06-04T05:10:01.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.873+0000 7fd363fff700 10 client.27842 handle_command_reply: tid=0 2024-06-04T05:10:01.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.873+0000 7fd395ad5b80 1 client.27842 shutdown 2024-06-04T05:10:01.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.873+0000 7fd395ad5b80 1 -- 172.21.15.70:0/3653555425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d7f60 msgr2=0x555a353e3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:01.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.873+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/3653555425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d7f60 0x555a353e3420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd37c009610 tx=0x7fd37c004c40 comp rx=0 tx=0).stop 2024-06-04T05:10:01.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.873+0000 7fd395ad5b80 1 -- 172.21.15.70:0/3653555425 shutdown_connections 2024-06-04T05:10:01.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.873+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/3653555425 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x555a3545c6f0 0x555a3545eae0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.873+0000 7fd363fff700 0 client.27842 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:10:01.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.873+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/3653555425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a353d7f60 0x555a353e3420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.873+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/3653555425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a353d3860 0x555a353e5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.873+0000 7fd395ad5b80 1 --2- 172.21.15.70:0/3653555425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a353d4210 0x555a353e2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.873+0000 7fd395ad5b80 1 -- 172.21.15.70:0/3653555425 >> 172.21.15.70:0/3653555425 conn(0x555a35301080 msgr2=0x555a353d7140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:01.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.873+0000 7fd395ad5b80 1 -- 172.21.15.70:0/3653555425 shutdown_connections 2024-06-04T05:10:01.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.874+0000 7fd395ad5b80 1 -- 172.21.15.70:0/3653555425 wait complete. 2024-06-04T05:10:01.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.874+0000 7fd395ad5b80 20 client.27842 trim_cache size 0 max 16384 2024-06-04T05:10:01.881 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:10:01.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.874+0000 7fd38b240700 1 -- 172.21.15.70:0/2754235275 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd3740b2a70 msgr2=0x7fd3740b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:01.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.874+0000 7fd38b240700 1 --2- 172.21.15.70:0/2754235275 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd3740b2a70 0x7fd3740b4f30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd384079ba0 tx=0x7fd370009260 comp rx=0 tx=0).stop 2024-06-04T05:10:01.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.874+0000 7fd38b240700 1 -- 172.21.15.70:0/2754235275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd384102440 msgr2=0x7fd38419f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:01.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.874+0000 7fd38b240700 1 --2- 172.21.15.70:0/2754235275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd384102440 0x7fd38419f730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd37800d950 tx=0x7fd37800dc60 comp rx=0 tx=0).stop 2024-06-04T05:10:01.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.875+0000 7fd38b240700 1 -- 172.21.15.70:0/2754235275 shutdown_connections 2024-06-04T05:10:01.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.875+0000 7fd38b240700 1 --2- 172.21.15.70:0/2754235275 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd3740b2a70 0x7fd3740b4f30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.875+0000 7fd38b240700 1 --2- 172.21.15.70:0/2754235275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd384108520 0x7fd384199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.875+0000 7fd38b240700 1 --2- 172.21.15.70:0/2754235275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd384102440 0x7fd38419f730 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.875+0000 7fd38b240700 1 --2- 172.21.15.70:0/2754235275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd384102e00 0x7fd38419fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:01.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.875+0000 7fd38b240700 1 -- 172.21.15.70:0/2754235275 >> 172.21.15.70:0/2754235275 conn(0x7fd3840793f0 msgr2=0x7fd38406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:01.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.875+0000 7fd38b240700 1 -- 172.21.15.70:0/2754235275 shutdown_connections 2024-06-04T05:10:01.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:01.875+0000 7fd38b240700 1 -- 172.21.15.70:0/2754235275 wait complete. 2024-06-04T05:10:02.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.047+0000 7f8406286700 1 -- 172.21.15.70:0/2093720984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8400076090 msgr2=0x7f840010fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:02.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.047+0000 7f8406286700 1 --2- 172.21.15.70:0/2093720984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8400076090 0x7f840010fa00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f83f4009990 tx=0x7f83f4009ca0 comp rx=0 tx=0).stop 2024-06-04T05:10:02.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.047+0000 7f8406286700 1 -- 172.21.15.70:0/2093720984 shutdown_connections 2024-06-04T05:10:02.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.047+0000 7f8406286700 1 --2- 172.21.15.70:0/2093720984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f840010ff40 0x7f8400112330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:02.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.047+0000 7f8406286700 1 --2- 172.21.15.70:0/2093720984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8400075770 0x7f8400075b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:02.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.047+0000 7f8406286700 1 --2- 172.21.15.70:0/2093720984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8400076090 0x7f840010fa00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:02.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.047+0000 7f8406286700 1 -- 172.21.15.70:0/2093720984 >> 172.21.15.70:0/2093720984 conn(0x7f840007a6f0 msgr2=0x7f840007ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:02.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.047+0000 7f8406286700 1 -- 172.21.15.70:0/2093720984 shutdown_connections 2024-06-04T05:10:02.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.047+0000 7f8406286700 1 -- 172.21.15.70:0/2093720984 wait complete. 2024-06-04T05:10:02.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.048+0000 7f8406286700 1 Processor -- start 2024-06-04T05:10:02.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.048+0000 7f8406286700 1 -- start start 2024-06-04T05:10:02.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.048+0000 7f8406286700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8400075770 0x7f840019bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:02.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.048+0000 7f8406286700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8400076090 0x7f840019c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:02.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.048+0000 7f8406286700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f840010ff40 0x7f84001a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:02.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.048+0000 7f8406286700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8400115830 con 0x7f8400076090 2024-06-04T05:10:02.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.048+0000 7f8406286700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f84001156b0 con 0x7f840010ff40 2024-06-04T05:10:02.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.048+0000 7f8406286700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f84001159b0 con 0x7f8400075770 2024-06-04T05:10:02.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.048+0000 7f83ff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8400076090 0x7f840019c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:02.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.048+0000 7f83fffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8400075770 0x7f840019bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:02.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.048+0000 7f83ff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8400076090 0x7f840019c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51122/0 (socket says 172.21.15.70:51122) 2024-06-04T05:10:02.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.048+0000 7f83fffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8400075770 0x7f840019bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:57446/0 (socket says 172.21.15.70:57446) 2024-06-04T05:10:02.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.048+0000 7f83ff7fe700 1 -- 172.21.15.70:0/3033502673 learned_addr learned my addr 172.21.15.70:0/3033502673 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:02.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.048+0000 7f8404823700 1 --2- 172.21.15.70:0/3033502673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f840010ff40 0x7f84001a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:02.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.049+0000 7f83fffff700 1 -- 172.21.15.70:0/3033502673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f840010ff40 msgr2=0x7f84001a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:02.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.049+0000 7f83fffff700 1 --2- 172.21.15.70:0/3033502673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f840010ff40 0x7f84001a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:02.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.049+0000 7f83fffff700 1 -- 172.21.15.70:0/3033502673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8400076090 msgr2=0x7f840019c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:02.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.049+0000 7f83fffff700 1 --2- 172.21.15.70:0/3033502673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8400076090 0x7f840019c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:02.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.049+0000 7f83fffff700 1 -- 172.21.15.70:0/3033502673 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84001a0d70 con 0x7f8400075770 2024-06-04T05:10:02.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.049+0000 7f83ff7fe700 1 --2- 172.21.15.70:0/3033502673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8400076090 0x7f840019c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:02.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.049+0000 7f83fffff700 1 --2- 172.21.15.70:0/3033502673 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8400075770 0x7f840019bd60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f83f000e9f0 tx=0x7f83f000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:02.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.049+0000 7f83fd7fa700 1 -- 172.21.15.70:0/3033502673 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f0004220 con 0x7f8400075770 2024-06-04T05:10:02.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.049+0000 7f83fd7fa700 1 -- 172.21.15.70:0/3033502673 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f83f00043c0 con 0x7f8400075770 2024-06-04T05:10:02.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.049+0000 7f8406286700 1 -- 172.21.15.70:0/3033502673 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84001a1060 con 0x7f8400075770 2024-06-04T05:10:02.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.049+0000 7f83fd7fa700 1 -- 172.21.15.70:0/3033502673 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f000ce30 con 0x7f8400075770 2024-06-04T05:10:02.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.049+0000 7f8406286700 1 -- 172.21.15.70:0/3033502673 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84001a88e0 con 0x7f8400075770 2024-06-04T05:10:02.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.050+0000 7f8406286700 1 -- 172.21.15.70:0/3033502673 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f840006a230 con 0x7f8400075770 2024-06-04T05:10:02.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.053+0000 7f83fd7fa700 1 -- 172.21.15.70:0/3033502673 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f83f0004750 con 0x7f8400075770 2024-06-04T05:10:02.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.053+0000 7f83fd7fa700 1 --2- 172.21.15.70:0/3033502673 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f83e80b2a70 0x7f83e80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:02.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.053+0000 7f83ff7fe700 1 --2- 172.21.15.70:0/3033502673 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f83e80b2a70 0x7f83e80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:02.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.053+0000 7f83fd7fa700 1 -- 172.21.15.70:0/3033502673 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f83f00e2670 con 0x7f8400075770 2024-06-04T05:10:02.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.054+0000 7f83ff7fe700 1 --2- 172.21.15.70:0/3033502673 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f83e80b2a70 0x7f83e80b4f30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8400111ce0 tx=0x7f83f401a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:02.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.056+0000 7f83fd7fa700 1 -- 172.21.15.70:0/3033502673 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f83f00aa0b0 con 0x7f8400075770 2024-06-04T05:10:02.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.354+0000 7f8406286700 1 -- 172.21.15.70:0/3033502673 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f840019d0c0 con 0x7f8400075770 2024-06-04T05:10:02.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.355+0000 7f83fd7fa700 1 -- 172.21.15.70:0/3033502673 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11938 (secure 0 0 0) 0x7f83f00a99b0 con 0x7f8400075770 2024-06-04T05:10:02.357 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:10:02.357 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:09:17.032462+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24436,24410,24440,24448,24442]},"id":1}]} 2024-06-04T05:10:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.357+0000 7f8406286700 1 -- 172.21.15.70:0/3033502673 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f83e80b2a70 msgr2=0x7f83e80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.357+0000 7f8406286700 1 --2- 172.21.15.70:0/3033502673 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f83e80b2a70 0x7f83e80b4f30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8400111ce0 tx=0x7f83f401a040 comp rx=0 tx=0).stop 2024-06-04T05:10:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.357+0000 7f8406286700 1 -- 172.21.15.70:0/3033502673 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8400075770 msgr2=0x7f840019bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.357+0000 7f8406286700 1 --2- 172.21.15.70:0/3033502673 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8400075770 0x7f840019bd60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f83f000e9f0 tx=0x7f83f000ed00 comp rx=0 tx=0).stop 2024-06-04T05:10:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.357+0000 7f8406286700 1 -- 172.21.15.70:0/3033502673 shutdown_connections 2024-06-04T05:10:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.357+0000 7f8406286700 1 --2- 172.21.15.70:0/3033502673 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f83e80b2a70 0x7f83e80b4f30 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.358+0000 7f8406286700 1 --2- 172.21.15.70:0/3033502673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f840010ff40 0x7f84001a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.358+0000 7f8406286700 1 --2- 172.21.15.70:0/3033502673 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8400075770 0x7f840019bd60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.358+0000 7f8406286700 1 --2- 172.21.15.70:0/3033502673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8400076090 0x7f840019c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.358+0000 7f8406286700 1 -- 172.21.15.70:0/3033502673 >> 172.21.15.70:0/3033502673 conn(0x7f840007a6f0 msgr2=0x7f84001007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.358+0000 7f8406286700 1 -- 172.21.15.70:0/3033502673 shutdown_connections 2024-06-04T05:10:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:02.358+0000 7f8406286700 1 -- 172.21.15.70:0/3033502673 wait complete. 2024-06-04T05:10:02.361 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 38 2024-06-04T05:10:02.741 INFO:tasks.workunit.client.0.smithi070.stdout: 1 207913 1.63 MB/sec execute 535 sec latency 19.870 ms 2024-06-04T05:10:02.807 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:10:03.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:03 smithi070 ceph-mon[32631]: pgmap v399: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 462 KiB/s wr, 67 op/s 2024-06-04T05:10:03.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:03 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3033502673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:10:03.513 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 149.4 --await-for 45.0' 2024-06-04T05:10:03.513 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 149.4 --await-for 45.0 2024-06-04T05:10:03.616 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:10:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:03 smithi112 ceph-mon[36453]: pgmap v399: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 462 KiB/s wr, 67 op/s 2024-06-04T05:10:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:03 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3033502673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:10:03.740 INFO:tasks.workunit.client.0.smithi070.stdout: 1 208472 1.64 MB/sec execute 536 sec latency 15.631 ms 2024-06-04T05:10:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:03 smithi098 ceph-mon[35569]: pgmap v399: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 462 KiB/s wr, 67 op/s 2024-06-04T05:10:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:03 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3033502673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:10:04.554 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:04 smithi070 ceph-mon[32631]: osdmap e93: 12 total, 12 up, 12 in 2024-06-04T05:10:04.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:04 smithi112 ceph-mon[36453]: osdmap e93: 12 total, 12 up, 12 in 2024-06-04T05:10:04.741 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209057 1.64 MB/sec execute 537 sec latency 15.079 ms 2024-06-04T05:10:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:04 smithi098 ceph-mon[35569]: osdmap e93: 12 total, 12 up, 12 in 2024-06-04T05:10:04.857 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:10:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.346+0000 7f9c0b44c700 1 -- 172.21.15.70:0/2672091907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c041084b0 msgr2=0x7f9c04108890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.346+0000 7f9c0b44c700 1 --2- 172.21.15.70:0/2672091907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c041084b0 0x7f9c04108890 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009990 tx=0x7f9c00009ca0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.346+0000 7f9c0b44c700 1 -- 172.21.15.70:0/2672091907 shutdown_connections 2024-06-04T05:10:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.346+0000 7f9c0b44c700 1 --2- 172.21.15.70:0/2672091907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c04102e10 0x7f9c0410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.346+0000 7f9c0b44c700 1 --2- 172.21.15.70:0/2672091907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c041084b0 0x7f9c04108890 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.346+0000 7f9c0b44c700 1 --2- 172.21.15.70:0/2672091907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c04102470 0x7f9c041028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.346+0000 7f9c0b44c700 1 -- 172.21.15.70:0/2672091907 >> 172.21.15.70:0/2672091907 conn(0x7f9c040793e0 msgr2=0x7f9c040797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.346+0000 7f9c0b44c700 1 -- 172.21.15.70:0/2672091907 shutdown_connections 2024-06-04T05:10:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.346+0000 7f9c0b44c700 1 -- 172.21.15.70:0/2672091907 wait complete. 2024-06-04T05:10:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.347+0000 7f9c0b44c700 1 Processor -- start 2024-06-04T05:10:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.347+0000 7f9c0b44c700 1 -- start start 2024-06-04T05:10:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.347+0000 7f9c0b44c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c04102470 0x7f9c0419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.347+0000 7f9c0b44c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c04102e10 0x7f9c0419fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.347+0000 7f9c0b44c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c041084b0 0x7f9c04199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.347+0000 7f9c0b44c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9c04112450 con 0x7f9c04102470 2024-06-04T05:10:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.347+0000 7f9c0b44c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9c041122d0 con 0x7f9c04102e10 2024-06-04T05:10:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.347+0000 7f9c0b44c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9c041125d0 con 0x7f9c041084b0 2024-06-04T05:10:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.347+0000 7f9c091e8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c04102470 0x7f9c0419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.347+0000 7f9c091e8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c04102470 0x7f9c0419f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51148/0 (socket says 172.21.15.70:51148) 2024-06-04T05:10:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.347+0000 7f9c091e8700 1 -- 172.21.15.70:0/3595820358 learned_addr learned my addr 172.21.15.70:0/3595820358 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.347+0000 7f9c099e9700 1 --2- 172.21.15.70:0/3595820358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c041084b0 0x7f9c04199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.348+0000 7f9c089e7700 1 --2- 172.21.15.70:0/3595820358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c04102e10 0x7f9c0419fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.348+0000 7f9c091e8700 1 -- 172.21.15.70:0/3595820358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c041084b0 msgr2=0x7f9c04199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.348+0000 7f9c091e8700 1 --2- 172.21.15.70:0/3595820358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c041084b0 0x7f9c04199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.348+0000 7f9c091e8700 1 -- 172.21.15.70:0/3595820358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c04102e10 msgr2=0x7f9c0419fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.348+0000 7f9c091e8700 1 --2- 172.21.15.70:0/3595820358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c04102e10 0x7f9c0419fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.348+0000 7f9c091e8700 1 -- 172.21.15.70:0/3595820358 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c0419a0e0 con 0x7f9c04102470 2024-06-04T05:10:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.348+0000 7f9c099e9700 1 --2- 172.21.15.70:0/3595820358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c041084b0 0x7f9c04199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.348+0000 7f9c089e7700 1 --2- 172.21.15.70:0/3595820358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c04102e10 0x7f9c0419fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:10:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.348+0000 7f9c091e8700 1 --2- 172.21.15.70:0/3595820358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c04102470 0x7f9c0419f770 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f9c00006010 tx=0x7f9c0000bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.348+0000 7f9bfa7fc700 1 -- 172.21.15.70:0/3595820358 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c00027070 con 0x7f9c04102470 2024-06-04T05:10:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.348+0000 7f9c0b44c700 1 -- 172.21.15.70:0/3595820358 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c0419a370 con 0x7f9c04102470 2024-06-04T05:10:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.348+0000 7f9bfa7fc700 1 -- 172.21.15.70:0/3595820358 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c00022470 con 0x7f9c04102470 2024-06-04T05:10:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.348+0000 7f9bfa7fc700 1 -- 172.21.15.70:0/3595820358 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c00017650 con 0x7f9c04102470 2024-06-04T05:10:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.348+0000 7f9c0b44c700 1 -- 172.21.15.70:0/3595820358 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c041a6640 con 0x7f9c04102470 2024-06-04T05:10:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.350+0000 7f9bfa7fc700 1 -- 172.21.15.70:0/3595820358 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f9c00009d10 con 0x7f9c04102470 2024-06-04T05:10:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.351+0000 7f9bfa7fc700 1 --2- 172.21.15.70:0/3595820358 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9bf00b29f0 0x7f9bf00b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.351+0000 7f9c089e7700 1 --2- 172.21.15.70:0/3595820358 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9bf00b29f0 0x7f9bf00b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.352+0000 7f9bfa7fc700 1 -- 172.21.15.70:0/3595820358 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8615+0+0 (secure 0 0 0) 0x7f9c000e3240 con 0x7f9c04102470 2024-06-04T05:10:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.352+0000 7f9c089e7700 1 --2- 172.21.15.70:0/3595820358 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9bf00b29f0 0x7f9bf00b4eb0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4005fd0 tx=0x7f9bf4005ee0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.352+0000 7f9c15ce1b80 1 Processor -- start 2024-06-04T05:10:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.352+0000 7f9c15ce1b80 1 -- start start 2024-06-04T05:10:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.352+0000 7f9c15ce1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc5fc70 0x563d3bc60050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.352+0000 7f9c15ce1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc60620 0x563d3bc63d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.352+0000 7f9c15ce1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc64370 0x563d3bc647b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.352+0000 7f9c15ce1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563d3bbbac90 con 0x563d3bc60620 2024-06-04T05:10:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.352+0000 7f9c15ce1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563d3bbaa830 con 0x563d3bc64370 2024-06-04T05:10:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.352+0000 7f9c15ce1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563d3bbae870 con 0x563d3bc5fc70 2024-06-04T05:10:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.353+0000 7f9c089e7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc60620 0x563d3bc63d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.353+0000 7f9c089e7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc60620 0x563d3bc63d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51152/0 (socket says 172.21.15.70:51152) 2024-06-04T05:10:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.353+0000 7f9c089e7700 1 -- 172.21.15.70:0/1428596629 learned_addr learned my addr 172.21.15.70:0/1428596629 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.353+0000 7f9c099e9700 1 --2- 172.21.15.70:0/1428596629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc64370 0x563d3bc647b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.353+0000 7f9c091e8700 1 --2- 172.21.15.70:0/1428596629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc5fc70 0x563d3bc60050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.353+0000 7f9c089e7700 1 -- 172.21.15.70:0/1428596629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc5fc70 msgr2=0x563d3bc60050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.353+0000 7f9c089e7700 1 --2- 172.21.15.70:0/1428596629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc5fc70 0x563d3bc60050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.353+0000 7f9c089e7700 1 -- 172.21.15.70:0/1428596629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc64370 msgr2=0x563d3bc647b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.353+0000 7f9c089e7700 1 --2- 172.21.15.70:0/1428596629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc64370 0x563d3bc647b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.353+0000 7f9c089e7700 1 -- 172.21.15.70:0/1428596629 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d3bafa8d0 con 0x563d3bc60620 2024-06-04T05:10:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.353+0000 7f9c089e7700 1 --2- 172.21.15.70:0/1428596629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc60620 0x563d3bc63d50 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f9bf400eda0 tx=0x7f9bf4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d2f6a47d3184276 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.353+0000 7f9bef7fe700 1 -- 172.21.15.70:0/1428596629 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf400edd0 con 0x563d3bc60620 2024-06-04T05:10:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.353+0000 7f9bef7fe700 1 -- 172.21.15.70:0/1428596629 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9bf4017030 con 0x563d3bc60620 2024-06-04T05:10:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.353+0000 7f9bef7fe700 1 -- 172.21.15.70:0/1428596629 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf40053a0 con 0x563d3bc60620 2024-06-04T05:10:05.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.354+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/1428596629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc60620 msgr2=0x563d3bc63d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:05.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.354+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/1428596629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc60620 0x563d3bc63d50 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f9bf400eda0 tx=0x7f9bf4009f90 comp rx=0 tx=0).stop 2024-06-04T05:10:05.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.354+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/1428596629 shutdown_connections 2024-06-04T05:10:05.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.354+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/1428596629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc64370 0x563d3bc647b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.354+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/1428596629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc5fc70 0x563d3bc60050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.354+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/1428596629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc60620 0x563d3bc63d50 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.354+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/1428596629 >> 172.21.15.70:0/1428596629 conn(0x563d3bb89080 msgr2=0x563d3bc55e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:05.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.354+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/1428596629 shutdown_connections 2024-06-04T05:10:05.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.354+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/1428596629 wait complete. 2024-06-04T05:10:05.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.354+0000 7f9c15ce1b80 1 -- start start 2024-06-04T05:10:05.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c15ce1b80 1 Processor -- start 2024-06-04T05:10:05.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c15ce1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc5fc70 0x563d3bc6b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c15ce1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc60620 0x563d3bc68a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c15ce1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc64370 0x563d3bc68fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c15ce1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563d3bab0fa0 con 0x563d3bc64370 2024-06-04T05:10:05.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c15ce1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563d3bb4c7c0 con 0x563d3bc5fc70 2024-06-04T05:10:05.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c15ce1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563d3bc6c530 con 0x563d3bc60620 2024-06-04T05:10:05.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c15ce1b80 10 client.0 _ll_register_callbacks cb 0x563d3bb45920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:05.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c099e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc64370 0x563d3bc68fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c099e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc64370 0x563d3bc68fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51168/0 (socket says 172.21.15.70:51168) 2024-06-04T05:10:05.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c099e9700 1 -- 172.21.15.70:0/3454552616 learned_addr learned my addr 172.21.15.70:0/3454552616 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:05.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c091e8700 1 --2- 172.21.15.70:0/3454552616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc5fc70 0x563d3bc6b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c089e7700 1 --2- 172.21.15.70:0/3454552616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc60620 0x563d3bc68a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c099e9700 1 -- 172.21.15.70:0/3454552616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc60620 msgr2=0x563d3bc68a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:05.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c099e9700 1 --2- 172.21.15.70:0/3454552616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc60620 0x563d3bc68a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c099e9700 1 -- 172.21.15.70:0/3454552616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc5fc70 msgr2=0x563d3bc6b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:05.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c099e9700 1 --2- 172.21.15.70:0/3454552616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc5fc70 0x563d3bc6b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.355+0000 7f9c099e9700 1 -- 172.21.15.70:0/3454552616 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d3bb49690 con 0x563d3bc64370 2024-06-04T05:10:05.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.356+0000 7f9c099e9700 1 --2- 172.21.15.70:0/3454552616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc64370 0x563d3bc68fd0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc003d50 tx=0x7f9bfc00e7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaaf553a0f4f13a1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:05.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.356+0000 7f9becff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:05.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.356+0000 7f9becff9700 1 -- 172.21.15.70:0/3454552616 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc0092d0 con 0x563d3bc64370 2024-06-04T05:10:05.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.356+0000 7f9becff9700 1 -- 172.21.15.70:0/3454552616 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8615+0+0 (secure 0 0 0) 0x7f9bfc007500 con 0x563d3bc64370 2024-06-04T05:10:05.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.356+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/3454552616 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bf4017030 con 0x563d3bc64370 2024-06-04T05:10:05.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.356+0000 7f9becff9700 1 -- 172.21.15.70:0/3454552616 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f9bfc007700 con 0x563d3bc64370 2024-06-04T05:10:05.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.356+0000 7f9c15ce1b80 10 client.18915 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:05.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.356+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/3454552616 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x563d3bc76c70 con 0x563d3bc64370 2024-06-04T05:10:05.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.356+0000 7f9becff9700 1 -- 172.21.15.70:0/3454552616 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9bf4017030 con 0x563d3bc64370 2024-06-04T05:10:05.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.356+0000 7f9becff9700 1 -- 172.21.15.70:0/3454552616 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f9bfc026760 con 0x563d3bc64370 2024-06-04T05:10:05.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.356+0000 7f9c15ce1b80 10 client.18915 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:05.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.356+0000 7f9c15ce1b80 10 client.18915 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:10:05.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.356+0000 7f9c15ce1b80 20 client.18915 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:05.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.356+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/3454552616 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563d3bc792b0 0x563d3bc7b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.356+0000 7f9c15ce1b80 4 client.18915 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:10:05.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.356+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/3454552616 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d3bc279e0 con 0x563d3bc792b0 2024-06-04T05:10:05.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.357+0000 7f9c091e8700 1 --2- 172.21.15.70:0/3454552616 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563d3bc792b0 0x563d3bc7b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.357+0000 7f9c091e8700 1 --2- 172.21.15.70:0/3454552616 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563d3bc792b0 0x563d3bc7b6e0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e6fd28e8512ee9 server_cookie=d2c65ab46dc3fe5e in_seq=0 out_seq=0 2024-06-04T05:10:05.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.357+0000 7f9becff9700 10 client.18915 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:05.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.358+0000 7f9becff9700 1 -- 172.21.15.70:0/3454552616 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563d3bc279e0 con 0x563d3bc792b0 2024-06-04T05:10:05.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.358+0000 7f9becff9700 10 client.18915 handle_command_reply: tid=0 2024-06-04T05:10:05.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.358+0000 7f9c15ce1b80 1 client.18915 shutdown 2024-06-04T05:10:05.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.358+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/3454552616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc64370 msgr2=0x563d3bc68fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:05.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.358+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/3454552616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc64370 0x563d3bc68fd0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc003d50 tx=0x7f9bfc00e7d0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.358+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/3454552616 shutdown_connections 2024-06-04T05:10:05.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.358+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/3454552616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc64370 0x563d3bc68fd0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.358+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/3454552616 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563d3bc792b0 0x563d3bc7b6e0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.358+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/3454552616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc5fc70 0x563d3bc6b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.358+0000 7f9becff9700 0 client.18915 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:05.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.358+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/3454552616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc60620 0x563d3bc68a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.358+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/3454552616 >> 172.21.15.70:0/3454552616 conn(0x563d3bb89080 msgr2=0x563d3bc55510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:05.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.358+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/3454552616 shutdown_connections 2024-06-04T05:10:05.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.358+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/3454552616 wait complete. 2024-06-04T05:10:05.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.358+0000 7f9c15ce1b80 20 client.18915 trim_cache size 0 max 16384 2024-06-04T05:10:05.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c15ce1b80 1 Processor -- start 2024-06-04T05:10:05.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c15ce1b80 1 -- start start 2024-06-04T05:10:05.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c15ce1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc5fc70 0x563d3bc68390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c15ce1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc60620 0x563d3bc6a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c15ce1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc64370 0x563d3bc55c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c15ce1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9c041abbc0 con 0x563d3bc64370 2024-06-04T05:10:05.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c15ce1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9c041ad820 con 0x563d3bc60620 2024-06-04T05:10:05.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c15ce1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9c041ac290 con 0x563d3bc5fc70 2024-06-04T05:10:05.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c099e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc64370 0x563d3bc55c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c089e7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc60620 0x563d3bc6a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c099e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc64370 0x563d3bc55c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51174/0 (socket says 172.21.15.70:51174) 2024-06-04T05:10:05.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c099e9700 1 -- 172.21.15.70:0/350335423 learned_addr learned my addr 172.21.15.70:0/350335423 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:05.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c091e8700 1 --2- 172.21.15.70:0/350335423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc5fc70 0x563d3bc68390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c089e7700 1 -- 172.21.15.70:0/350335423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc5fc70 msgr2=0x563d3bc68390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:05.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c089e7700 1 --2- 172.21.15.70:0/350335423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc5fc70 0x563d3bc68390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c089e7700 1 -- 172.21.15.70:0/350335423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc64370 msgr2=0x563d3bc55c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:05.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c089e7700 1 --2- 172.21.15.70:0/350335423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc64370 0x563d3bc55c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c089e7700 1 -- 172.21.15.70:0/350335423 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d3bc640c0 con 0x563d3bc60620 2024-06-04T05:10:05.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.374+0000 7f9c099e9700 1 --2- 172.21.15.70:0/350335423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc64370 0x563d3bc55c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:05.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.375+0000 7f9c089e7700 1 --2- 172.21.15.70:0/350335423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc60620 0x563d3bc6a9d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x563d3bc6a060 tx=0x7f9bf400e730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44ec1707bc08cbf6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:05.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.375+0000 7f9beeffd700 1 -- 172.21.15.70:0/350335423 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf402b070 con 0x563d3bc60620 2024-06-04T05:10:05.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.375+0000 7f9beeffd700 1 -- 172.21.15.70:0/350335423 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9bf4017040 con 0x563d3bc60620 2024-06-04T05:10:05.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.375+0000 7f9beeffd700 1 -- 172.21.15.70:0/350335423 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf40053c0 con 0x563d3bc60620 2024-06-04T05:10:05.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.376+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/350335423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc60620 msgr2=0x563d3bc6a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:05.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.376+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/350335423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc60620 0x563d3bc6a9d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x563d3bc6a060 tx=0x7f9bf400e730 comp rx=0 tx=0).stop 2024-06-04T05:10:05.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.376+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/350335423 shutdown_connections 2024-06-04T05:10:05.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.376+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/350335423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc64370 0x563d3bc55c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.376+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/350335423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc5fc70 0x563d3bc68390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.376+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/350335423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc60620 0x563d3bc6a9d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.376+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/350335423 >> 172.21.15.70:0/350335423 conn(0x563d3bb89080 msgr2=0x563d3bc6a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:05.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.376+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/350335423 shutdown_connections 2024-06-04T05:10:05.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.376+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/350335423 wait complete. 2024-06-04T05:10:05.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.376+0000 7f9c15ce1b80 1 -- start start 2024-06-04T05:10:05.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.376+0000 7f9c15ce1b80 1 Processor -- start 2024-06-04T05:10:05.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.376+0000 7f9c15ce1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc5fc70 0x563d3bc71c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.377+0000 7f9c15ce1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc60620 0x563d3bc6d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.377+0000 7f9c15ce1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc64370 0x563d3bc6d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.377+0000 7f9c15ce1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9c041ad0f0 con 0x563d3bc5fc70 2024-06-04T05:10:05.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.377+0000 7f9c15ce1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9c041ac9c0 con 0x563d3bc64370 2024-06-04T05:10:05.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.377+0000 7f9c15ce1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9c041a9b80 con 0x563d3bc60620 2024-06-04T05:10:05.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.377+0000 7f9c15ce1b80 10 client.0 _ll_register_callbacks cb 0x563d3bb45920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:05.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.377+0000 7f9c091e8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc5fc70 0x563d3bc71c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.377+0000 7f9c091e8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc5fc70 0x563d3bc71c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51186/0 (socket says 172.21.15.70:51186) 2024-06-04T05:10:05.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.377+0000 7f9c091e8700 1 -- 172.21.15.70:0/846584538 learned_addr learned my addr 172.21.15.70:0/846584538 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:05.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.377+0000 7f9c099e9700 1 --2- 172.21.15.70:0/846584538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc64370 0x563d3bc6d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.377+0000 7f9c089e7700 1 --2- 172.21.15.70:0/846584538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc60620 0x563d3bc6d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.377+0000 7f9c091e8700 1 -- 172.21.15.70:0/846584538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc60620 msgr2=0x563d3bc6d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:05.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.377+0000 7f9c091e8700 1 --2- 172.21.15.70:0/846584538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc60620 0x563d3bc6d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.377+0000 7f9c091e8700 1 -- 172.21.15.70:0/846584538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc64370 msgr2=0x563d3bc6d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:05.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.377+0000 7f9c091e8700 1 --2- 172.21.15.70:0/846584538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc64370 0x563d3bc6d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.377+0000 7f9c091e8700 1 -- 172.21.15.70:0/846584538 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d3bc6db80 con 0x563d3bc5fc70 2024-06-04T05:10:05.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.377+0000 7f9c089e7700 1 --2- 172.21.15.70:0/846584538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc60620 0x563d3bc6d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:10:05.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.378+0000 7f9c091e8700 1 --2- 172.21.15.70:0/846584538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc5fc70 0x563d3bc71c80 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f9c00005f50 tx=0x7f9c000aa0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84401ffa0ae87f77 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:05.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.378+0000 7f9becff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:05.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.378+0000 7f9becff9700 1 -- 172.21.15.70:0/846584538 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c00022920 con 0x563d3bc5fc70 2024-06-04T05:10:05.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.378+0000 7f9becff9700 1 -- 172.21.15.70:0/846584538 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8615+0+0 (secure 0 0 0) 0x7f9c000aa7b0 con 0x563d3bc5fc70 2024-06-04T05:10:05.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.378+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/846584538 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bf4017040 con 0x563d3bc5fc70 2024-06-04T05:10:05.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.378+0000 7f9becff9700 1 -- 172.21.15.70:0/846584538 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f9c00022ac0 con 0x563d3bc5fc70 2024-06-04T05:10:05.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.378+0000 7f9c15ce1b80 10 client.18921 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:05.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.378+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/846584538 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x563d3bce3cf0 con 0x563d3bc5fc70 2024-06-04T05:10:05.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.378+0000 7f9becff9700 1 -- 172.21.15.70:0/846584538 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c000aa9b0 con 0x563d3bc5fc70 2024-06-04T05:10:05.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.378+0000 7f9becff9700 1 -- 172.21.15.70:0/846584538 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f9c000cb320 con 0x563d3bc5fc70 2024-06-04T05:10:05.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.378+0000 7f9c15ce1b80 10 client.18921 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:05.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.378+0000 7f9c15ce1b80 10 client.18921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:10:05.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.378+0000 7f9c15ce1b80 20 client.18921 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:05.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.378+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/846584538 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563d3bc792b0 0x563d3bce6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:05.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.378+0000 7f9c15ce1b80 4 client.18921 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:10:05.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.379+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/846584538 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f9bf4017040 con 0x563d3bc792b0 2024-06-04T05:10:05.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.379+0000 7f9c089e7700 1 --2- 172.21.15.70:0/846584538 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563d3bc792b0 0x563d3bce6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:05.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.379+0000 7f9becff9700 10 client.18921 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:05.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.379+0000 7f9c089e7700 1 --2- 172.21.15.70:0/846584538 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563d3bc792b0 0x563d3bce6470 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77860efc266dd0b server_cookie=f8f54177f7692631 in_seq=0 out_seq=0 2024-06-04T05:10:05.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.379+0000 7f9becff9700 1 -- 172.21.15.70:0/846584538 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f9bf4017040 con 0x563d3bc792b0 2024-06-04T05:10:05.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.379+0000 7f9becff9700 10 client.18921 handle_command_reply: tid=0 2024-06-04T05:10:05.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.379+0000 7f9c15ce1b80 1 client.18921 shutdown 2024-06-04T05:10:05.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.379+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/846584538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc5fc70 msgr2=0x563d3bc71c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:05.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.379+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/846584538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc5fc70 0x563d3bc71c80 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f9c00005f50 tx=0x7f9c000aa0e0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.379+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/846584538 shutdown_connections 2024-06-04T05:10:05.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.379+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/846584538 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563d3bc792b0 0x563d3bce6470 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.379+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/846584538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d3bc64370 0x563d3bc6d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.379+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/846584538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d3bc5fc70 0x563d3bc71c80 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.380+0000 7f9becff9700 0 client.18921 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:05.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.380+0000 7f9c15ce1b80 1 --2- 172.21.15.70:0/846584538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d3bc60620 0x563d3bc6d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.380+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/846584538 >> 172.21.15.70:0/846584538 conn(0x563d3bb89080 msgr2=0x563d3bc6ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:05.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.380+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/846584538 shutdown_connections 2024-06-04T05:10:05.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.380+0000 7f9c15ce1b80 1 -- 172.21.15.70:0/846584538 wait complete. 2024-06-04T05:10:05.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.380+0000 7f9c15ce1b80 20 client.18921 trim_cache size 0 max 16384 2024-06-04T05:10:05.388 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:10:05.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.380+0000 7f9c0b44c700 1 -- 172.21.15.70:0/3595820358 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9bf00b29f0 msgr2=0x7f9bf00b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:05.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.380+0000 7f9c0b44c700 1 --2- 172.21.15.70:0/3595820358 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9bf00b29f0 0x7f9bf00b4eb0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4005fd0 tx=0x7f9bf4005ee0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.380+0000 7f9c0b44c700 1 -- 172.21.15.70:0/3595820358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c04102470 msgr2=0x7f9c0419f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:05.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.380+0000 7f9c0b44c700 1 --2- 172.21.15.70:0/3595820358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c04102470 0x7f9c0419f770 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f9c00006010 tx=0x7f9c0000bfc0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.380+0000 7f9c0b44c700 1 -- 172.21.15.70:0/3595820358 shutdown_connections 2024-06-04T05:10:05.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.380+0000 7f9c0b44c700 1 --2- 172.21.15.70:0/3595820358 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9bf00b29f0 0x7f9bf00b4eb0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.380+0000 7f9c0b44c700 1 --2- 172.21.15.70:0/3595820358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c041084b0 0x7f9c04199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.380+0000 7f9c0b44c700 1 --2- 172.21.15.70:0/3595820358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c04102470 0x7f9c0419f770 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.380+0000 7f9c0b44c700 1 --2- 172.21.15.70:0/3595820358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c04102e10 0x7f9c0419fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:05.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.380+0000 7f9c0b44c700 1 -- 172.21.15.70:0/3595820358 >> 172.21.15.70:0/3595820358 conn(0x7f9c040793e0 msgr2=0x7f9c0406c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:05.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.381+0000 7f9c0b44c700 1 -- 172.21.15.70:0/3595820358 shutdown_connections 2024-06-04T05:10:05.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:05.381+0000 7f9c0b44c700 1 -- 172.21.15.70:0/3595820358 wait complete. 2024-06-04T05:10:05.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:05 smithi070 ceph-mon[32631]: pgmap v401: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 555 KiB/s wr, 76 op/s 2024-06-04T05:10:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:05 smithi112 ceph-mon[36453]: pgmap v401: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 555 KiB/s wr, 76 op/s 2024-06-04T05:10:05.741 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209346 1.65 MB/sec execute 538 sec latency 471.650 ms 2024-06-04T05:10:05.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:05 smithi098 ceph-mon[35569]: pgmap v401: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 555 KiB/s wr, 76 op/s 2024-06-04T05:10:06.741 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209346 1.64 MB/sec execute 539 sec latency 1471.760 ms 2024-06-04T05:10:07.386 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:10:07.386 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:10:07.594 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:07 smithi070 ceph-mon[32631]: pgmap v402: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 1.2 MiB/s wr, 129 op/s 2024-06-04T05:10:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:07 smithi112 ceph-mon[36453]: pgmap v402: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 1.2 MiB/s wr, 129 op/s 2024-06-04T05:10:07.741 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209413 1.64 MB/sec execute 540 sec latency 1777.931 ms 2024-06-04T05:10:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:07 smithi098 ceph-mon[35569]: pgmap v402: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 1.2 MiB/s wr, 129 op/s 2024-06-04T05:10:07.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.834+0000 7fdb3973a700 1 -- 172.21.15.70:0/1346714258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb34101b30 msgr2=0x7fdb34101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:07.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.834+0000 7fdb3973a700 1 --2- 172.21.15.70:0/1346714258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb34101b30 0x7fdb34101fb0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fdb28009990 tx=0x7fdb28009ca0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.834+0000 7fdb3973a700 1 -- 172.21.15.70:0/1346714258 shutdown_connections 2024-06-04T05:10:07.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.834+0000 7fdb3973a700 1 --2- 172.21.15.70:0/1346714258 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb341024f0 0x7fdb341103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.834+0000 7fdb3973a700 1 --2- 172.21.15.70:0/1346714258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb341095b0 0x7fdb34109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.834+0000 7fdb3973a700 1 --2- 172.21.15.70:0/1346714258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb34101b30 0x7fdb34101fb0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.834+0000 7fdb3973a700 1 -- 172.21.15.70:0/1346714258 >> 172.21.15.70:0/1346714258 conn(0x7fdb34078ae0 msgr2=0x7fdb340ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:07.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.834+0000 7fdb3973a700 1 -- 172.21.15.70:0/1346714258 shutdown_connections 2024-06-04T05:10:07.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.834+0000 7fdb3973a700 1 -- 172.21.15.70:0/1346714258 wait complete. 2024-06-04T05:10:07.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.835+0000 7fdb3973a700 1 Processor -- start 2024-06-04T05:10:07.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.835+0000 7fdb3973a700 1 -- start start 2024-06-04T05:10:07.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.835+0000 7fdb3973a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb34101b30 0x7fdb3419f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.835+0000 7fdb3973a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb341024f0 0x7fdb3419fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.835+0000 7fdb3973a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb341095b0 0x7fdb341999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.835+0000 7fdb3973a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdb34113670 con 0x7fdb341095b0 2024-06-04T05:10:07.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.835+0000 7fdb3973a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdb341134f0 con 0x7fdb341024f0 2024-06-04T05:10:07.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.835+0000 7fdb3973a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdb341137f0 con 0x7fdb34101b30 2024-06-04T05:10:07.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.835+0000 7fdb337fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb341095b0 0x7fdb341999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.835+0000 7fdb337fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb341095b0 0x7fdb341999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46970/0 (socket says 172.21.15.70:46970) 2024-06-04T05:10:07.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.835+0000 7fdb337fe700 1 -- 172.21.15.70:0/664698720 learned_addr learned my addr 172.21.15.70:0/664698720 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:07.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.835+0000 7fdb32ffd700 1 --2- 172.21.15.70:0/664698720 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb34101b30 0x7fdb3419f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.835+0000 7fdb327fc700 1 --2- 172.21.15.70:0/664698720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb341024f0 0x7fdb3419fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.836+0000 7fdb32ffd700 1 -- 172.21.15.70:0/664698720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb341024f0 msgr2=0x7fdb3419fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:07.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.836+0000 7fdb32ffd700 1 --2- 172.21.15.70:0/664698720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb341024f0 0x7fdb3419fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.836+0000 7fdb32ffd700 1 -- 172.21.15.70:0/664698720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb341095b0 msgr2=0x7fdb341999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:07.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.836+0000 7fdb32ffd700 1 --2- 172.21.15.70:0/664698720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb341095b0 0x7fdb341999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.836+0000 7fdb32ffd700 1 -- 172.21.15.70:0/664698720 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3419a130 con 0x7fdb34101b30 2024-06-04T05:10:07.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.836+0000 7fdb337fe700 1 --2- 172.21.15.70:0/664698720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb341095b0 0x7fdb341999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:07.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.836+0000 7fdb32ffd700 1 --2- 172.21.15.70:0/664698720 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb34101b30 0x7fdb3419f810 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdb2400b450 tx=0x7fdb2400b760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:07.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.836+0000 7fdb1bfff700 1 -- 172.21.15.70:0/664698720 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb24015020 con 0x7fdb34101b30 2024-06-04T05:10:07.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.836+0000 7fdb3973a700 1 -- 172.21.15.70:0/664698720 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb3419a420 con 0x7fdb34101b30 2024-06-04T05:10:07.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.836+0000 7fdb3973a700 1 -- 172.21.15.70:0/664698720 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb34102cc0 con 0x7fdb34101b30 2024-06-04T05:10:07.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.836+0000 7fdb1bfff700 1 -- 172.21.15.70:0/664698720 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdb24011cc0 con 0x7fdb34101b30 2024-06-04T05:10:07.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.836+0000 7fdb1bfff700 1 -- 172.21.15.70:0/664698720 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb24010b70 con 0x7fdb34101b30 2024-06-04T05:10:07.842 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:10:07.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.839+0000 7fdb1bfff700 1 -- 172.21.15.70:0/664698720 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fdb24018020 con 0x7fdb34101b30 2024-06-04T05:10:07.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.840+0000 7fdb1bfff700 1 --2- 172.21.15.70:0/664698720 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdb1c0b2ba0 0x7fdb1c0b5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.840+0000 7fdb327fc700 1 --2- 172.21.15.70:0/664698720 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdb1c0b2ba0 0x7fdb1c0b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.840+0000 7fdb1bfff700 1 -- 172.21.15.70:0/664698720 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8615+0+0 (secure 0 0 0) 0x7fdb240e1ba0 con 0x7fdb34101b30 2024-06-04T05:10:07.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.840+0000 7fdb327fc700 1 --2- 172.21.15.70:0/664698720 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdb1c0b2ba0 0x7fdb1c0b5060 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800b5c0 tx=0x7fdb28005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:07.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.841+0000 7fdb43fcfb80 1 Processor -- start 2024-06-04T05:10:07.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.841+0000 7fdb43fcfb80 1 -- start start 2024-06-04T05:10:07.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.841+0000 7fdb43fcfb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edebb340 0x55d4edebb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.841+0000 7fdb43fcfb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edeb5970 0x55d4edeb5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.841+0000 7fdb43fcfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edeb6330 0x55d4edec2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.841+0000 7fdb43fcfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d4ede14e20 con 0x55d4edeb6330 2024-06-04T05:10:07.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.841+0000 7fdb43fcfb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d4ede049c0 con 0x55d4edebb340 2024-06-04T05:10:07.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.841+0000 7fdb43fcfb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d4ede08a00 con 0x55d4edeb5970 2024-06-04T05:10:07.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.842+0000 7fdb337fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edeb6330 0x55d4edec2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.842+0000 7fdb337fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edeb6330 0x55d4edec2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46974/0 (socket says 172.21.15.70:46974) 2024-06-04T05:10:07.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.842+0000 7fdb337fe700 1 -- 172.21.15.70:0/4292621581 learned_addr learned my addr 172.21.15.70:0/4292621581 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:07.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.842+0000 7fdb327fc700 1 --2- 172.21.15.70:0/4292621581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edeb5970 0x55d4edeb5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.842+0000 7fdb337fe700 1 -- 172.21.15.70:0/4292621581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edeb5970 msgr2=0x55d4edeb5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:07.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.842+0000 7fdb337fe700 1 --2- 172.21.15.70:0/4292621581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edeb5970 0x55d4edeb5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.842+0000 7fdb337fe700 1 -- 172.21.15.70:0/4292621581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edebb340 msgr2=0x55d4edebb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:07.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.842+0000 7fdb32ffd700 1 --2- 172.21.15.70:0/4292621581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edebb340 0x55d4edebb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.842+0000 7fdb337fe700 1 --2- 172.21.15.70:0/4292621581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edebb340 0x55d4edebb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.842+0000 7fdb337fe700 1 -- 172.21.15.70:0/4292621581 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4edd54a60 con 0x55d4edeb6330 2024-06-04T05:10:07.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.842+0000 7fdb327fc700 1 --2- 172.21.15.70:0/4292621581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edeb5970 0x55d4edeb5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:10:07.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.842+0000 7fdb32ffd700 1 --2- 172.21.15.70:0/4292621581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edebb340 0x55d4edebb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:10:07.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.843+0000 7fdb337fe700 1 --2- 172.21.15.70:0/4292621581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edeb6330 0x55d4edec2170 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c010960 tx=0x7fdb2c009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34856313f04298f4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:07.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.843+0000 7fdb197fa700 1 -- 172.21.15.70:0/4292621581 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c01e070 con 0x55d4edeb6330 2024-06-04T05:10:07.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.843+0000 7fdb197fa700 1 -- 172.21.15.70:0/4292621581 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdb2c007a10 con 0x55d4edeb6330 2024-06-04T05:10:07.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.843+0000 7fdb197fa700 1 -- 172.21.15.70:0/4292621581 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c019a10 con 0x55d4edeb6330 2024-06-04T05:10:07.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.844+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/4292621581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edeb6330 msgr2=0x55d4edec2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:07.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.844+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/4292621581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edeb6330 0x55d4edec2170 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c010960 tx=0x7fdb2c009660 comp rx=0 tx=0).stop 2024-06-04T05:10:07.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.844+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/4292621581 shutdown_connections 2024-06-04T05:10:07.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.844+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/4292621581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edeb6330 0x55d4edec2170 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.844+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/4292621581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edebb340 0x55d4edebb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.844+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/4292621581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edeb5970 0x55d4edeb5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.844+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/4292621581 >> 172.21.15.70:0/4292621581 conn(0x55d4edde3210 msgr2=0x55d4edeb0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:07.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.844+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/4292621581 shutdown_connections 2024-06-04T05:10:07.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.844+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/4292621581 wait complete. 2024-06-04T05:10:07.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.844+0000 7fdb43fcfb80 1 -- start start 2024-06-04T05:10:07.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.845+0000 7fdb43fcfb80 1 Processor -- start 2024-06-04T05:10:07.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.845+0000 7fdb43fcfb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb5970 0x55d4edec1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.845+0000 7fdb43fcfb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edeb6330 0x55d4edebf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.845+0000 7fdb43fcfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edebb340 0x55d4edebf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.845+0000 7fdb43fcfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d4edd0b130 con 0x55d4edebb340 2024-06-04T05:10:07.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.845+0000 7fdb43fcfb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d4edda6950 con 0x55d4edeb5970 2024-06-04T05:10:07.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.845+0000 7fdb43fcfb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d4edec2770 con 0x55d4edeb6330 2024-06-04T05:10:07.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.845+0000 7fdb43fcfb80 10 client.0 _ll_register_callbacks cb 0x55d4edd9fab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:07.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.845+0000 7fdb337fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edebb340 0x55d4edebf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.845+0000 7fdb32ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb5970 0x55d4edec1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.845+0000 7fdb337fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edebb340 0x55d4edebf690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46984/0 (socket says 172.21.15.70:46984) 2024-06-04T05:10:07.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.845+0000 7fdb337fe700 1 -- 172.21.15.70:0/1563401056 learned_addr learned my addr 172.21.15.70:0/1563401056 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:07.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.845+0000 7fdb32ffd700 1 --2- 172.21.15.70:0/1563401056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb5970 0x55d4edec1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:35012/0 (socket says 172.21.15.70:35012) 2024-06-04T05:10:07.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.845+0000 7fdb327fc700 1 --2- 172.21.15.70:0/1563401056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edeb6330 0x55d4edebf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.846+0000 7fdb32ffd700 1 -- 172.21.15.70:0/1563401056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edeb6330 msgr2=0x55d4edebf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:07.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.846+0000 7fdb32ffd700 1 --2- 172.21.15.70:0/1563401056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edeb6330 0x55d4edebf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.846+0000 7fdb32ffd700 1 -- 172.21.15.70:0/1563401056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edebb340 msgr2=0x55d4edebf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:07.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.846+0000 7fdb32ffd700 1 --2- 172.21.15.70:0/1563401056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edebb340 0x55d4edebf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.846+0000 7fdb32ffd700 1 -- 172.21.15.70:0/1563401056 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4edda3820 con 0x55d4edeb5970 2024-06-04T05:10:07.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.846+0000 7fdb337fe700 1 --2- 172.21.15.70:0/1563401056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edebb340 0x55d4edebf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:07.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.846+0000 7fdb32ffd700 1 --2- 172.21.15.70:0/1563401056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb5970 0x55d4edec1e50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdb240df340 tx=0x7fdb240df790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c97b6f8a67e1706 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:07.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.846+0000 7fdb16ff5700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:10:07.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.846+0000 7fdb16ff5700 1 -- 172.21.15.70:0/1563401056 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb24010380 con 0x55d4edeb5970 2024-06-04T05:10:07.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.846+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/1563401056 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb2c007a10 con 0x55d4edeb5970 2024-06-04T05:10:07.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.846+0000 7fdb16ff5700 1 -- 172.21.15.70:0/1563401056 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8615+0+0 (secure 0 0 0) 0x7fdb240b1ab0 con 0x55d4edeb5970 2024-06-04T05:10:07.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.846+0000 7fdb16ff5700 1 -- 172.21.15.70:0/1563401056 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fdb240a89e0 con 0x55d4edeb5970 2024-06-04T05:10:07.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.846+0000 7fdb43fcfb80 10 client.27869 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:07.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.846+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/1563401056 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d4edeccee0 con 0x55d4edeb5970 2024-06-04T05:10:07.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.847+0000 7fdb16ff5700 1 -- 172.21.15.70:0/1563401056 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdb2c007a10 con 0x55d4edeb5970 2024-06-04T05:10:07.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.847+0000 7fdb16ff5700 1 -- 172.21.15.70:0/1563401056 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7fdb240a8130 con 0x55d4edeb5970 2024-06-04T05:10:07.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.847+0000 7fdb43fcfb80 10 client.27869 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:07.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.847+0000 7fdb43fcfb80 10 client.27869 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:10:07.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.847+0000 7fdb43fcfb80 20 client.27869 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:07.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.847+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/1563401056 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d4edecf4c0 0x55d4eded1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.847+0000 7fdb43fcfb80 4 client.27869 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:10:07.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.847+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/1563401056 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4ede81b70 con 0x55d4edecf4c0 2024-06-04T05:10:07.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.847+0000 7fdb327fc700 1 --2- 172.21.15.70:0/1563401056 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d4edecf4c0 0x55d4eded1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.848+0000 7fdb16ff5700 10 client.27869 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:07.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.848+0000 7fdb327fc700 1 --2- 172.21.15.70:0/1563401056 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d4edecf4c0 0x55d4eded1910 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5d4a48cffaf132c server_cookie=ecb8854cd990b436 in_seq=0 out_seq=0 2024-06-04T05:10:07.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.848+0000 7fdb16ff5700 1 -- 172.21.15.70:0/1563401056 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d4ede81b70 con 0x55d4edecf4c0 2024-06-04T05:10:07.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.848+0000 7fdb16ff5700 10 client.27869 handle_command_reply: tid=0 2024-06-04T05:10:07.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.848+0000 7fdb43fcfb80 1 client.27869 shutdown 2024-06-04T05:10:07.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.849+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/1563401056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb5970 msgr2=0x55d4edec1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:07.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.849+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/1563401056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb5970 0x55d4edec1e50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdb240df340 tx=0x7fdb240df790 comp rx=0 tx=0).stop 2024-06-04T05:10:07.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.849+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/1563401056 shutdown_connections 2024-06-04T05:10:07.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.849+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/1563401056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edebb340 0x55d4edebf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.849+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/1563401056 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d4edecf4c0 0x55d4eded1910 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.849+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/1563401056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb5970 0x55d4edec1e50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.849+0000 7fdb16ff5700 0 client.27869 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:07.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.849+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/1563401056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edeb6330 0x55d4edebf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.849+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/1563401056 >> 172.21.15.70:0/1563401056 conn(0x55d4edde3210 msgr2=0x55d4edeaad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:07.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.849+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/1563401056 shutdown_connections 2024-06-04T05:10:07.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.849+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/1563401056 wait complete. 2024-06-04T05:10:07.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.849+0000 7fdb43fcfb80 20 client.27869 trim_cache size 0 max 16384 2024-06-04T05:10:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.879+0000 7fdb43fcfb80 1 Processor -- start 2024-06-04T05:10:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.879+0000 7fdb43fcfb80 1 -- start start 2024-06-04T05:10:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.880+0000 7fdb43fcfb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb5970 0x55d4edeb5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.880+0000 7fdb43fcfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edeb6330 0x55d4edecb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.880+0000 7fdb43fcfb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edebb340 0x55d4edec1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.880+0000 7fdb43fcfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdb341abd70 con 0x55d4edeb6330 2024-06-04T05:10:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.880+0000 7fdb43fcfb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdb341ad9d0 con 0x55d4edeb5970 2024-06-04T05:10:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.880+0000 7fdb43fcfb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdb341ac440 con 0x55d4edebb340 2024-06-04T05:10:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.880+0000 7fdb327fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edeb6330 0x55d4edecb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.880+0000 7fdb327fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edeb6330 0x55d4edecb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46992/0 (socket says 172.21.15.70:46992) 2024-06-04T05:10:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.880+0000 7fdb327fc700 1 -- 172.21.15.70:0/1800637745 learned_addr learned my addr 172.21.15.70:0/1800637745 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:07.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.880+0000 7fdb337fe700 1 --2- 172.21.15.70:0/1800637745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edebb340 0x55d4edec1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.880+0000 7fdb32ffd700 1 --2- 172.21.15.70:0/1800637745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb5970 0x55d4edeb5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.880+0000 7fdb327fc700 1 -- 172.21.15.70:0/1800637745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edebb340 msgr2=0x55d4edec1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:07.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.880+0000 7fdb327fc700 1 --2- 172.21.15.70:0/1800637745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edebb340 0x55d4edec1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.880+0000 7fdb327fc700 1 -- 172.21.15.70:0/1800637745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb5970 msgr2=0x55d4edeb5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:07.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.880+0000 7fdb327fc700 1 --2- 172.21.15.70:0/1800637745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb5970 0x55d4edeb5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.880+0000 7fdb327fc700 1 -- 172.21.15.70:0/1800637745 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4edeb2420 con 0x55d4edeb6330 2024-06-04T05:10:07.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.880+0000 7fdb32ffd700 1 --2- 172.21.15.70:0/1800637745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb5970 0x55d4edeb5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:10:07.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.881+0000 7fdb327fc700 1 --2- 172.21.15.70:0/1800637745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edeb6330 0x55d4edecb810 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fdb28009990 tx=0x7fdb28024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f8f7fed4e19424 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:07.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.881+0000 7fdb18ff9700 1 -- 172.21.15.70:0/1800637745 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2800fac0 con 0x55d4edeb6330 2024-06-04T05:10:07.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.881+0000 7fdb18ff9700 1 -- 172.21.15.70:0/1800637745 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdb2800be90 con 0x55d4edeb6330 2024-06-04T05:10:07.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.881+0000 7fdb18ff9700 1 -- 172.21.15.70:0/1800637745 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb28024420 con 0x55d4edeb6330 2024-06-04T05:10:07.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.882+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/1800637745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edeb6330 msgr2=0x55d4edecb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:07.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.882+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/1800637745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edeb6330 0x55d4edecb810 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fdb28009990 tx=0x7fdb28024000 comp rx=0 tx=0).stop 2024-06-04T05:10:07.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.882+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/1800637745 shutdown_connections 2024-06-04T05:10:07.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.882+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/1800637745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edebb340 0x55d4edec1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.882+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/1800637745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb5970 0x55d4edeb5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.882+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/1800637745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edeb6330 0x55d4edecb810 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.882+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/1800637745 >> 172.21.15.70:0/1800637745 conn(0x55d4edde3210 msgr2=0x55d4edeb96b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:07.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.882+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/1800637745 shutdown_connections 2024-06-04T05:10:07.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.882+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/1800637745 wait complete. 2024-06-04T05:10:07.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.882+0000 7fdb43fcfb80 1 -- start start 2024-06-04T05:10:07.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.883+0000 7fdb43fcfb80 1 Processor -- start 2024-06-04T05:10:07.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.883+0000 7fdb43fcfb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edeb5970 0x55d4edec81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.883+0000 7fdb43fcfb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb6330 0x55d4edec49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.883+0000 7fdb43fcfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edebb340 0x55d4edec4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.883+0000 7fdb43fcfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdb341ad2a0 con 0x55d4edebb340 2024-06-04T05:10:07.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.883+0000 7fdb43fcfb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdb341acb70 con 0x55d4edeb6330 2024-06-04T05:10:07.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.883+0000 7fdb43fcfb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdb341a9d30 con 0x55d4edeb5970 2024-06-04T05:10:07.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.883+0000 7fdb43fcfb80 10 client.0 _ll_register_callbacks cb 0x55d4edd9fab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:07.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.883+0000 7fdb337fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edebb340 0x55d4edec4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.883+0000 7fdb337fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edebb340 0x55d4edec4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47002/0 (socket says 172.21.15.70:47002) 2024-06-04T05:10:07.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.883+0000 7fdb337fe700 1 -- 172.21.15.70:0/2412243492 learned_addr learned my addr 172.21.15.70:0/2412243492 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:07.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.883+0000 7fdb327fc700 1 --2- 172.21.15.70:0/2412243492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb6330 0x55d4edec49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.883+0000 7fdb32ffd700 1 --2- 172.21.15.70:0/2412243492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edeb5970 0x55d4edec81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.883+0000 7fdb337fe700 1 -- 172.21.15.70:0/2412243492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edeb5970 msgr2=0x55d4edec81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:07.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.884+0000 7fdb337fe700 1 --2- 172.21.15.70:0/2412243492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edeb5970 0x55d4edec81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.884+0000 7fdb337fe700 1 -- 172.21.15.70:0/2412243492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb6330 msgr2=0x55d4edec49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:07.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.884+0000 7fdb337fe700 1 --2- 172.21.15.70:0/2412243492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb6330 0x55d4edec49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.884+0000 7fdb337fe700 1 -- 172.21.15.70:0/2412243492 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4edec57f0 con 0x55d4edebb340 2024-06-04T05:10:07.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.884+0000 7fdb337fe700 1 --2- 172.21.15.70:0/2412243492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edebb340 0x55d4edec4fe0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c010960 tx=0x7fdb2c019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebeb03f7f4fad1d1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:07.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.884+0000 7fdb16ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:07.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.884+0000 7fdb16ff5700 1 -- 172.21.15.70:0/2412243492 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c028030 con 0x55d4edebb340 2024-06-04T05:10:07.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.884+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/2412243492 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb2800be90 con 0x55d4edebb340 2024-06-04T05:10:07.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.884+0000 7fdb16ff5700 1 -- 172.21.15.70:0/2412243492 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7fdb2c016040 con 0x55d4edebb340 2024-06-04T05:10:07.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.884+0000 7fdb16ff5700 1 -- 172.21.15.70:0/2412243492 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fdb2c01e040 con 0x55d4edebb340 2024-06-04T05:10:07.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.884+0000 7fdb43fcfb80 10 client.18939 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:07.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.884+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/2412243492 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d4edf3b1f0 con 0x55d4edebb340 2024-06-04T05:10:07.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.884+0000 7fdb16ff5700 1 -- 172.21.15.70:0/2412243492 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdb2800be90 con 0x55d4edebb340 2024-06-04T05:10:07.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.884+0000 7fdb16ff5700 1 -- 172.21.15.70:0/2412243492 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7fdb2c0248a0 con 0x55d4edebb340 2024-06-04T05:10:07.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.885+0000 7fdb43fcfb80 10 client.18939 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:07.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.885+0000 7fdb43fcfb80 10 client.18939 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:10:07.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.885+0000 7fdb43fcfb80 20 client.18939 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:07.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.885+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/2412243492 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d4edecf4c0 0x55d4edf3d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:07.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.885+0000 7fdb43fcfb80 4 client.18939 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.4, "await_for": 45.0}] 2024-06-04T05:10:07.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.885+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/2412243492 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.4, "await_for": 45.0}) -- 0x55d4edebadf0 con 0x55d4edecf4c0 2024-06-04T05:10:07.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.885+0000 7fdb32ffd700 1 --2- 172.21.15.70:0/2412243492 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d4edecf4c0 0x55d4edf3d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:07.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.885+0000 7fdb32ffd700 1 --2- 172.21.15.70:0/2412243492 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d4edecf4c0 0x55d4edf3d670 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f85fb914c77e733 server_cookie=92d40460ef894931 in_seq=0 out_seq=0 2024-06-04T05:10:07.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.885+0000 7fdb16ff5700 10 client.18939 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:07.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.990+0000 7fdb16ff5700 1 -- 172.21.15.70:0/2412243492 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55d4edebadf0 con 0x55d4edecf4c0 2024-06-04T05:10:07.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.990+0000 7fdb16ff5700 10 client.18939 handle_command_reply: tid=0 2024-06-04T05:10:07.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.990+0000 7fdb43fcfb80 1 client.18939 shutdown 2024-06-04T05:10:07.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.990+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/2412243492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edebb340 msgr2=0x55d4edec4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:07.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.990+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/2412243492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edebb340 0x55d4edec4fe0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c010960 tx=0x7fdb2c019640 comp rx=0 tx=0).stop 2024-06-04T05:10:07.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.990+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/2412243492 shutdown_connections 2024-06-04T05:10:07.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.990+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/2412243492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4edebb340 0x55d4edec4fe0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.990+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/2412243492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d4edeb5970 0x55d4edec81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.990+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/2412243492 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d4edecf4c0 0x55d4edf3d670 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.990+0000 7fdb16ff5700 0 client.18939 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:07.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.990+0000 7fdb43fcfb80 1 --2- 172.21.15.70:0/2412243492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4edeb6330 0x55d4edec49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.990+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/2412243492 >> 172.21.15.70:0/2412243492 conn(0x55d4edde3210 msgr2=0x55d4edeb8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:07.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.990+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/2412243492 shutdown_connections 2024-06-04T05:10:07.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.990+0000 7fdb43fcfb80 1 -- 172.21.15.70:0/2412243492 wait complete. 2024-06-04T05:10:07.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.990+0000 7fdb43fcfb80 20 client.18939 trim_cache size 0 max 16384 2024-06-04T05:10:07.995 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:10:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 38, 2024-06-04T05:10:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:10:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 42, 2024-06-04T05:10:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:10:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "48e6b43c": { 2024-06-04T05:10:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 42, 2024-06-04T05:10:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:10:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:10:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:10:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:10:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:10:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:10:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 149.4, 2024-06-04T05:10:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:10:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:10:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:10:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:10:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:10:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:10:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:10:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:10:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:10:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:10:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:10:07.997 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:10:07.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.991+0000 7fdb3973a700 1 -- 172.21.15.70:0/664698720 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdb1c0b2ba0 msgr2=0x7fdb1c0b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:07.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.991+0000 7fdb3973a700 1 --2- 172.21.15.70:0/664698720 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdb1c0b2ba0 0x7fdb1c0b5060 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800b5c0 tx=0x7fdb28005960 comp rx=0 tx=0).stop 2024-06-04T05:10:07.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.991+0000 7fdb3973a700 1 -- 172.21.15.70:0/664698720 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb34101b30 msgr2=0x7fdb3419f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:07.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.991+0000 7fdb3973a700 1 --2- 172.21.15.70:0/664698720 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb34101b30 0x7fdb3419f810 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdb2400b450 tx=0x7fdb2400b760 comp rx=0 tx=0).stop 2024-06-04T05:10:07.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.991+0000 7fdb3973a700 1 -- 172.21.15.70:0/664698720 shutdown_connections 2024-06-04T05:10:07.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.991+0000 7fdb3973a700 1 --2- 172.21.15.70:0/664698720 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdb1c0b2ba0 0x7fdb1c0b5060 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.991+0000 7fdb3973a700 1 --2- 172.21.15.70:0/664698720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb341095b0 0x7fdb341999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.991+0000 7fdb3973a700 1 --2- 172.21.15.70:0/664698720 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb34101b30 0x7fdb3419f810 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.991+0000 7fdb3973a700 1 --2- 172.21.15.70:0/664698720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb341024f0 0x7fdb3419fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:07.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.991+0000 7fdb3973a700 1 -- 172.21.15.70:0/664698720 >> 172.21.15.70:0/664698720 conn(0x7fdb34078ae0 msgr2=0x7fdb341002a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:07.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.991+0000 7fdb3973a700 1 -- 172.21.15.70:0/664698720 shutdown_connections 2024-06-04T05:10:07.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:07.991+0000 7fdb3973a700 1 -- 172.21.15.70:0/664698720 wait complete. 2024-06-04T05:10:08.741 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.64 MB/sec execute 541 sec latency 750.706 ms 2024-06-04T05:10:08.902 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '48e6b43c', quiesce took 10.1 seconds. Will release after: 19.299999999999997 2024-06-04T05:10:09.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:08 smithi070 ceph-mon[32631]: osdmap e94: 12 total, 12 up, 12 in 2024-06-04T05:10:09.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:08 smithi070 ceph-mon[32631]: pgmap v404: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-06-04T05:10:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:08 smithi112 ceph-mon[36453]: osdmap e94: 12 total, 12 up, 12 in 2024-06-04T05:10:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:08 smithi112 ceph-mon[36453]: pgmap v404: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-06-04T05:10:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:08 smithi098 ceph-mon[35569]: osdmap e94: 12 total, 12 up, 12 in 2024-06-04T05:10:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:08 smithi098 ceph-mon[35569]: pgmap v404: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-06-04T05:10:09.741 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.63 MB/sec execute 542 sec latency 1750.749 ms 2024-06-04T05:10:09.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.813+0000 7f4e447c8640 1 -- 172.21.15.70:0/3345272309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c101980 msgr2=0x7f4e3c101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.813+0000 7f4e447c8640 1 --2- 172.21.15.70:0/3345272309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c101980 0x7f4e3c101e00 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f4e30009cd0 tx=0x7f4e3002a8e0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.813+0000 7f4e447c8640 1 -- 172.21.15.70:0/3345272309 shutdown_connections 2024-06-04T05:10:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.813+0000 7f4e447c8640 1 --2- 172.21.15.70:0/3345272309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e3c102340 0x7f4e3c10e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.813+0000 7f4e447c8640 1 --2- 172.21.15.70:0/3345272309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c101980 0x7f4e3c101e00 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.813+0000 7f4e447c8640 1 --2- 172.21.15.70:0/3345272309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e3c107940 0x7f4e3c107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.813+0000 7f4e447c8640 1 -- 172.21.15.70:0/3345272309 >> 172.21.15.70:0/3345272309 conn(0x7f4e3c078ad0 msgr2=0x7f4e3c078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.813+0000 7f4e447c8640 1 -- 172.21.15.70:0/3345272309 shutdown_connections 2024-06-04T05:10:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.813+0000 7f4e447c8640 1 -- 172.21.15.70:0/3345272309 wait complete. 2024-06-04T05:10:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e447c8640 1 Processor -- start 2024-06-04T05:10:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e447c8640 1 -- start start 2024-06-04T05:10:09.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e447c8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e3c101980 0x7f4e3c199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e447c8640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e3c102340 0x7f4e3c19a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e447c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c107940 0x7f4e3c1a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e447c8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c111ab0 con 0x7f4e3c107940 2024-06-04T05:10:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e447c8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c111930 con 0x7f4e3c101980 2024-06-04T05:10:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e447c8640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c111c30 con 0x7f4e3c102340 2024-06-04T05:10:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e42d65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c107940 0x7f4e3c1a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e42d65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c107940 0x7f4e3c1a4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47018/0 (socket says 172.21.15.70:47018) 2024-06-04T05:10:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e42d65640 1 -- 172.21.15.70:0/3350953261 learned_addr learned my addr 172.21.15.70:0/3350953261 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e41d63640 1 --2- 172.21.15.70:0/3350953261 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e3c102340 0x7f4e3c19a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e42564640 1 --2- 172.21.15.70:0/3350953261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e3c101980 0x7f4e3c199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e42d65640 1 -- 172.21.15.70:0/3350953261 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e3c102340 msgr2=0x7f4e3c19a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e42d65640 1 --2- 172.21.15.70:0/3350953261 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e3c102340 0x7f4e3c19a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e42d65640 1 -- 172.21.15.70:0/3350953261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e3c101980 msgr2=0x7f4e3c199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e42d65640 1 --2- 172.21.15.70:0/3350953261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e3c101980 0x7f4e3c199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.814+0000 7f4e42d65640 1 -- 172.21.15.70:0/3350953261 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e3c1a5530 con 0x7f4e3c107940 2024-06-04T05:10:09.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.815+0000 7f4e42d65640 1 --2- 172.21.15.70:0/3350953261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c107940 0x7f4e3c1a4ff0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00c970 tx=0x7f4e2c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:09.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.815+0000 7f4e3b7fe640 1 -- 172.21.15.70:0/3350953261 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c007bb0 con 0x7f4e3c107940 2024-06-04T05:10:09.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.815+0000 7f4e447c8640 1 -- 172.21.15.70:0/3350953261 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e3c1a5700 con 0x7f4e3c107940 2024-06-04T05:10:09.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.815+0000 7f4e3b7fe640 1 -- 172.21.15.70:0/3350953261 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e2c007d50 con 0x7f4e3c107940 2024-06-04T05:10:09.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.815+0000 7f4e3b7fe640 1 -- 172.21.15.70:0/3350953261 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c004d50 con 0x7f4e3c107940 2024-06-04T05:10:09.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.815+0000 7f4e447c8640 1 -- 172.21.15.70:0/3350953261 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e3c1a5c20 con 0x7f4e3c107940 2024-06-04T05:10:09.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.817+0000 7f4e475c0740 1 Processor -- start 2024-06-04T05:10:09.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.818+0000 7f4e3b7fe640 1 -- 172.21.15.70:0/3350953261 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f4e2c004f30 con 0x7f4e3c107940 2024-06-04T05:10:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.818+0000 7f4e475c0740 1 -- start start 2024-06-04T05:10:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.818+0000 7f4e475c0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ee700 0x55df873eeae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.818+0000 7f4e475c0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873ef020 0x55df873fd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.818+0000 7f4e475c0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df87400200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.818+0000 7f4e475c0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55df87360c30 con 0x55df873fde10 2024-06-04T05:10:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.818+0000 7f4e475c0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55df873507d0 con 0x55df873ef020 2024-06-04T05:10:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.818+0000 7f4e475c0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55df87354810 con 0x55df873ee700 2024-06-04T05:10:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.818+0000 7f4e41d63640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df87400200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.818+0000 7f4e41d63640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df87400200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47032/0 (socket says 172.21.15.70:47032) 2024-06-04T05:10:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.818+0000 7f4e41d63640 1 -- 172.21.15.70:0/3389304614 learned_addr learned my addr 172.21.15.70:0/3389304614 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.818+0000 7f4e42564640 1 --2- 172.21.15.70:0/3389304614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873ef020 0x55df873fd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.818+0000 7f4e41d63640 1 --2- 172.21.15.70:0/3389304614 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ee700 0x55df873eeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.819+0000 7f4e41d63640 1 -- 172.21.15.70:0/3389304614 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ee700 msgr2=0x55df873eeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.819+0000 7f4e41d63640 1 --2- 172.21.15.70:0/3389304614 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ee700 0x55df873eeae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.819+0000 7f4e41d63640 1 -- 172.21.15.70:0/3389304614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873ef020 msgr2=0x55df873fd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.819+0000 7f4e41d63640 1 --2- 172.21.15.70:0/3389304614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873ef020 0x55df873fd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.819+0000 7f4e41d63640 1 -- 172.21.15.70:0/3389304614 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df87065210 con 0x55df873fde10 2024-06-04T05:10:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.819+0000 7f4e3b7fe640 1 --2- 172.21.15.70:0/3350953261 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e240b2a10 0x7f4e240b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.819+0000 7f4e42564640 1 --2- 172.21.15.70:0/3389304614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873ef020 0x55df873fd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.819+0000 7f4e41d63640 1 --2- 172.21.15.70:0/3389304614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df87400200 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f4e30031840 tx=0x7f4e30009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=222350c0f0887f75 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.819+0000 7f4e42564640 1 --2- 172.21.15.70:0/3350953261 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e240b2a10 0x7f4e240b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.819+0000 7f4e3b7fe640 1 -- 172.21.15.70:0/3350953261 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f4e2c0e23c0 con 0x7f4e3c107940 2024-06-04T05:10:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.819+0000 7f4e38ff9640 1 -- 172.21.15.70:0/3389304614 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e300318a0 con 0x55df873fde10 2024-06-04T05:10:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.819+0000 7f4e38ff9640 1 -- 172.21.15.70:0/3389304614 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e30036040 con 0x55df873fde10 2024-06-04T05:10:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.819+0000 7f4e38ff9640 1 -- 172.21.15.70:0/3389304614 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e300044b0 con 0x55df873fde10 2024-06-04T05:10:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.820+0000 7f4e475c0740 1 -- 172.21.15.70:0/3389304614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 msgr2=0x55df87400200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.820+0000 7f4e475c0740 1 --2- 172.21.15.70:0/3389304614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df87400200 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f4e30031840 tx=0x7f4e30009360 comp rx=0 tx=0).stop 2024-06-04T05:10:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.820+0000 7f4e42564640 1 --2- 172.21.15.70:0/3350953261 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e240b2a10 0x7f4e240b4ed0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4e34005fd0 tx=0x7f4e3400d490 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.820+0000 7f4e475c0740 1 -- 172.21.15.70:0/3389304614 shutdown_connections 2024-06-04T05:10:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.820+0000 7f4e475c0740 1 --2- 172.21.15.70:0/3389304614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873ef020 0x55df873fd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.820+0000 7f4e475c0740 1 --2- 172.21.15.70:0/3389304614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df87400200 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.820+0000 7f4e475c0740 1 --2- 172.21.15.70:0/3389304614 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ee700 0x55df873eeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.820+0000 7f4e475c0740 1 -- 172.21.15.70:0/3389304614 >> 172.21.15.70:0/3389304614 conn(0x55df873ead30 msgr2=0x55df873eb110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.820+0000 7f4e475c0740 1 -- 172.21.15.70:0/3389304614 shutdown_connections 2024-06-04T05:10:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.820+0000 7f4e475c0740 1 -- 172.21.15.70:0/3389304614 wait complete. 2024-06-04T05:10:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e475c0740 1 -- start start 2024-06-04T05:10:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e475c0740 1 Processor -- start 2024-06-04T05:10:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e475c0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873ee700 0x55df873fae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e475c0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ef020 0x55df873fb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e475c0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df873fb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e475c0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55df87036790 con 0x55df873fde10 2024-06-04T05:10:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e475c0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55df8730c890 con 0x55df873ee700 2024-06-04T05:10:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e475c0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55df874009e0 con 0x55df873ef020 2024-06-04T05:10:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e475c0740 10 client.0 _ll_register_callbacks cb 0x55df87064a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e42d65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df873fb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e42d65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df873fb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47038/0 (socket says 172.21.15.70:47038) 2024-06-04T05:10:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e42d65640 1 -- 172.21.15.70:0/3622720434 learned_addr learned my addr 172.21.15.70:0/3622720434 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e41d63640 1 --2- 172.21.15.70:0/3622720434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ef020 0x55df873fb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e42564640 1 --2- 172.21.15.70:0/3622720434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873ee700 0x55df873fae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e42d65640 1 -- 172.21.15.70:0/3622720434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ef020 msgr2=0x55df873fb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e42d65640 1 --2- 172.21.15.70:0/3622720434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ef020 0x55df873fb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e42d65640 1 -- 172.21.15.70:0/3622720434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873ee700 msgr2=0x55df873fae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e42d65640 1 --2- 172.21.15.70:0/3622720434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873ee700 0x55df873fae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e42d65640 1 -- 172.21.15.70:0/3622720434 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df87131620 con 0x55df873fde10 2024-06-04T05:10:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.821+0000 7f4e42564640 1 --2- 172.21.15.70:0/3622720434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873ee700 0x55df873fae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:10:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.822+0000 7f4e42d65640 1 --2- 172.21.15.70:0/3622720434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df873fb880 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c0047e0 tx=0x7f4e2c0a8e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55cfd7d0a985b409 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.822+0000 7f4e21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.822+0000 7f4e21ffb640 1 -- 172.21.15.70:0/3622720434 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c01d040 con 0x55df873fde10 2024-06-04T05:10:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.822+0000 7f4e21ffb640 1 -- 172.21.15.70:0/3622720434 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f4e2c0a5c00 con 0x55df873fde10 2024-06-04T05:10:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.822+0000 7f4e475c0740 1 -- 172.21.15.70:0/3622720434 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e30036040 con 0x55df873fde10 2024-06-04T05:10:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.822+0000 7f4e21ffb640 1 -- 172.21.15.70:0/3622720434 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f4e2c0a9610 con 0x55df873fde10 2024-06-04T05:10:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.822+0000 7f4e475c0740 10 client.18951 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.822+0000 7f4e475c0740 1 -- 172.21.15.70:0/3622720434 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55df8740c870 con 0x55df873fde10 2024-06-04T05:10:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.822+0000 7f4e21ffb640 1 -- 172.21.15.70:0/3622720434 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e30036040 con 0x55df873fde10 2024-06-04T05:10:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.822+0000 7f4e21ffb640 1 -- 172.21.15.70:0/3622720434 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f4e2c0a4f80 con 0x55df873fde10 2024-06-04T05:10:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.822+0000 7f4e475c0740 10 client.18951 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.822+0000 7f4e475c0740 10 client.18951 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:10:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.823+0000 7f4e475c0740 20 client.18951 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.823+0000 7f4e475c0740 1 --2- 172.21.15.70:0/3622720434 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55df8740dda0 0x55df87410190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.823+0000 7f4e475c0740 4 client.18951 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:10:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.823+0000 7f4e475c0740 1 -- 172.21.15.70:0/3622720434 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df87362b80 con 0x55df8740dda0 2024-06-04T05:10:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.823+0000 7f4e42564640 1 --2- 172.21.15.70:0/3622720434 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55df8740dda0 0x55df87410190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.823+0000 7f4e42564640 1 --2- 172.21.15.70:0/3622720434 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55df8740dda0 0x55df87410190 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6ec54d62fbcc045 server_cookie=2b7473dfa6f58cfa in_seq=0 out_seq=0 2024-06-04T05:10:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.823+0000 7f4e21ffb640 10 client.18951 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.824+0000 7f4e21ffb640 1 -- 172.21.15.70:0/3622720434 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55df87362b80 con 0x55df8740dda0 2024-06-04T05:10:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.824+0000 7f4e21ffb640 10 client.18951 handle_command_reply: tid=0 2024-06-04T05:10:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.824+0000 7f4e475c0740 1 client.18951 shutdown 2024-06-04T05:10:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.824+0000 7f4e475c0740 1 -- 172.21.15.70:0/3622720434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 msgr2=0x55df873fb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.824+0000 7f4e475c0740 1 --2- 172.21.15.70:0/3622720434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df873fb880 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c0047e0 tx=0x7f4e2c0a8e40 comp rx=0 tx=0).stop 2024-06-04T05:10:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.824+0000 7f4e475c0740 1 -- 172.21.15.70:0/3622720434 shutdown_connections 2024-06-04T05:10:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.824+0000 7f4e475c0740 1 --2- 172.21.15.70:0/3622720434 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55df8740dda0 0x55df87410190 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.824+0000 7f4e475c0740 1 --2- 172.21.15.70:0/3622720434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df873fb880 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.824+0000 7f4e475c0740 1 --2- 172.21.15.70:0/3622720434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ef020 0x55df873fb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.824+0000 7f4e21ffb640 0 client.18951 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.824+0000 7f4e475c0740 1 --2- 172.21.15.70:0/3622720434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873ee700 0x55df873fae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.824+0000 7f4e475c0740 1 -- 172.21.15.70:0/3622720434 >> 172.21.15.70:0/3622720434 conn(0x55df873ead30 msgr2=0x55df873f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.824+0000 7f4e475c0740 1 -- 172.21.15.70:0/3622720434 shutdown_connections 2024-06-04T05:10:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.824+0000 7f4e475c0740 1 -- 172.21.15.70:0/3622720434 wait complete. 2024-06-04T05:10:09.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.824+0000 7f4e475c0740 20 client.18951 trim_cache size 0 max 16384 2024-06-04T05:10:09.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.848+0000 7f4e475c0740 1 Processor -- start 2024-06-04T05:10:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.848+0000 7f4e475c0740 1 -- start start 2024-06-04T05:10:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.848+0000 7f4e475c0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873ee700 0x55df873f0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.848+0000 7f4e475c0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ef020 0x55df873f0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.848+0000 7f4e475c0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873fde10 0x55df87415a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.848+0000 7f4e475c0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c1ac4a0 con 0x55df873ee700 2024-06-04T05:10:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.848+0000 7f4e475c0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c1ae100 con 0x55df873fde10 2024-06-04T05:10:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.848+0000 7f4e475c0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c1acb70 con 0x55df873ef020 2024-06-04T05:10:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.849+0000 7f4e42564640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873ee700 0x55df873f0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.849+0000 7f4e42564640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873ee700 0x55df873f0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47048/0 (socket says 172.21.15.70:47048) 2024-06-04T05:10:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.849+0000 7f4e42564640 1 -- 172.21.15.70:0/1573416853 learned_addr learned my addr 172.21.15.70:0/1573416853 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.849+0000 7f4e42d65640 1 --2- 172.21.15.70:0/1573416853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873fde10 0x55df87415a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.849+0000 7f4e41d63640 1 --2- 172.21.15.70:0/1573416853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ef020 0x55df873f0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.849+0000 7f4e42564640 1 -- 172.21.15.70:0/1573416853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ef020 msgr2=0x55df873f0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.849+0000 7f4e42564640 1 --2- 172.21.15.70:0/1573416853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ef020 0x55df873f0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.849+0000 7f4e42564640 1 -- 172.21.15.70:0/1573416853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873fde10 msgr2=0x55df87415a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.849+0000 7f4e42564640 1 --2- 172.21.15.70:0/1573416853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873fde10 0x55df87415a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.849+0000 7f4e42564640 1 -- 172.21.15.70:0/1573416853 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df873e53c0 con 0x55df873ee700 2024-06-04T05:10:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.849+0000 7f4e42564640 1 --2- 172.21.15.70:0/1573416853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873ee700 0x55df873f0960 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f4e340039c0 tx=0x7f4e34015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341688dd168d5f9c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.849+0000 7f4e23fff640 1 -- 172.21.15.70:0/1573416853 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e34012450 con 0x55df873ee700 2024-06-04T05:10:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.849+0000 7f4e23fff640 1 -- 172.21.15.70:0/1573416853 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e34012a80 con 0x55df873ee700 2024-06-04T05:10:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.849+0000 7f4e23fff640 1 -- 172.21.15.70:0/1573416853 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3401ba60 con 0x55df873ee700 2024-06-04T05:10:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.850+0000 7f4e475c0740 1 -- 172.21.15.70:0/1573416853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873ee700 msgr2=0x55df873f0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.850+0000 7f4e475c0740 1 --2- 172.21.15.70:0/1573416853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873ee700 0x55df873f0960 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f4e340039c0 tx=0x7f4e34015000 comp rx=0 tx=0).stop 2024-06-04T05:10:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.850+0000 7f4e475c0740 1 -- 172.21.15.70:0/1573416853 shutdown_connections 2024-06-04T05:10:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.850+0000 7f4e475c0740 1 --2- 172.21.15.70:0/1573416853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873fde10 0x55df87415a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.850+0000 7f4e475c0740 1 --2- 172.21.15.70:0/1573416853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ef020 0x55df873f0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.850+0000 7f4e475c0740 1 --2- 172.21.15.70:0/1573416853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873ee700 0x55df873f0960 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.850+0000 7f4e475c0740 1 -- 172.21.15.70:0/1573416853 >> 172.21.15.70:0/1573416853 conn(0x55df873ead30 msgr2=0x55df873f0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.850+0000 7f4e475c0740 1 -- 172.21.15.70:0/1573416853 shutdown_connections 2024-06-04T05:10:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.850+0000 7f4e475c0740 1 -- 172.21.15.70:0/1573416853 wait complete. 2024-06-04T05:10:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.850+0000 7f4e475c0740 1 -- start start 2024-06-04T05:10:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e475c0740 1 Processor -- start 2024-06-04T05:10:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e475c0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873ee700 0x55df87405c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e475c0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ef020 0x55df874061b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e475c0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df87402c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e475c0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c1ad9d0 con 0x55df873fde10 2024-06-04T05:10:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e475c0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c1ad2a0 con 0x55df873ee700 2024-06-04T05:10:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e475c0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c1aa4a0 con 0x55df873ef020 2024-06-04T05:10:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e475c0740 10 client.0 _ll_register_callbacks cb 0x7f4e2c0046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e42d65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df87402c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e42d65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df87402c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47060/0 (socket says 172.21.15.70:47060) 2024-06-04T05:10:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e42d65640 1 -- 172.21.15.70:0/3536797324 learned_addr learned my addr 172.21.15.70:0/3536797324 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e42564640 1 --2- 172.21.15.70:0/3536797324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873ee700 0x55df87405c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e41d63640 1 --2- 172.21.15.70:0/3536797324 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ef020 0x55df874061b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e42d65640 1 -- 172.21.15.70:0/3536797324 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ef020 msgr2=0x55df874061b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e42d65640 1 --2- 172.21.15.70:0/3536797324 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ef020 0x55df874061b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e42d65640 1 -- 172.21.15.70:0/3536797324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873ee700 msgr2=0x55df87405c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e42d65640 1 --2- 172.21.15.70:0/3536797324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873ee700 0x55df87405c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e42d65640 1 -- 172.21.15.70:0/3536797324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df87403280 con 0x55df873fde10 2024-06-04T05:10:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.851+0000 7f4e41d63640 1 --2- 172.21.15.70:0/3536797324 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ef020 0x55df874061b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:10:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.852+0000 7f4e42d65640 1 --2- 172.21.15.70:0/3536797324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df87402c40 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c0a8d10 tx=0x7f4e2c0a8a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fde65a70c4e1ce3d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.852+0000 7f4e21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.852+0000 7f4e21ffb640 1 -- 172.21.15.70:0/3536797324 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c01d070 con 0x55df873fde10 2024-06-04T05:10:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.852+0000 7f4e21ffb640 1 -- 172.21.15.70:0/3536797324 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f4e2c0e1040 con 0x55df873fde10 2024-06-04T05:10:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.852+0000 7f4e475c0740 1 -- 172.21.15.70:0/3536797324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e34012a80 con 0x55df873fde10 2024-06-04T05:10:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.852+0000 7f4e21ffb640 1 -- 172.21.15.70:0/3536797324 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f4e2c085e60 con 0x55df873fde10 2024-06-04T05:10:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.852+0000 7f4e475c0740 10 client.18957 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.852+0000 7f4e475c0740 1 -- 172.21.15.70:0/3536797324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55df87479270 con 0x55df873fde10 2024-06-04T05:10:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.852+0000 7f4e21ffb640 1 -- 172.21.15.70:0/3536797324 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e34012a80 con 0x55df873fde10 2024-06-04T05:10:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.852+0000 7f4e21ffb640 1 -- 172.21.15.70:0/3536797324 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f4e2c0ca300 con 0x55df873fde10 2024-06-04T05:10:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.852+0000 7f4e475c0740 10 client.18957 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.852+0000 7f4e475c0740 10 client.18957 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:10:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.852+0000 7f4e475c0740 20 client.18957 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.852+0000 7f4e475c0740 1 --2- 172.21.15.70:0/3536797324 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55df8740dda0 0x55df8747bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.852+0000 7f4e475c0740 4 client.18957 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:10:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.852+0000 7f4e475c0740 1 -- 172.21.15.70:0/3536797324 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55df8735c360 con 0x55df8740dda0 2024-06-04T05:10:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.853+0000 7f4e42564640 1 --2- 172.21.15.70:0/3536797324 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55df8740dda0 0x55df8747bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.853+0000 7f4e21ffb640 10 client.18957 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.853+0000 7f4e42564640 1 --2- 172.21.15.70:0/3536797324 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55df8740dda0 0x55df8747bab0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f1df29851cd91bf server_cookie=caf624df20082623 in_seq=0 out_seq=0 2024-06-04T05:10:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.876+0000 7f4e21ffb640 1 -- 172.21.15.70:0/3536797324 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55df8735c360 con 0x55df8740dda0 2024-06-04T05:10:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.876+0000 7f4e21ffb640 10 client.18957 handle_command_reply: tid=0 2024-06-04T05:10:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.876+0000 7f4e475c0740 1 client.18957 shutdown 2024-06-04T05:10:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e475c0740 1 -- 172.21.15.70:0/3536797324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 msgr2=0x55df87402c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e475c0740 1 --2- 172.21.15.70:0/3536797324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df87402c40 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c0a8d10 tx=0x7f4e2c0a8a80 comp rx=0 tx=0).stop 2024-06-04T05:10:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e475c0740 1 -- 172.21.15.70:0/3536797324 shutdown_connections 2024-06-04T05:10:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e475c0740 1 --2- 172.21.15.70:0/3536797324 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55df8740dda0 0x55df8747bab0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e475c0740 1 --2- 172.21.15.70:0/3536797324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df873fde10 0x55df87402c40 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e475c0740 1 --2- 172.21.15.70:0/3536797324 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df873ef020 0x55df874061b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e21ffb640 0 client.18957 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e475c0740 1 --2- 172.21.15.70:0/3536797324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55df873ee700 0x55df87405c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e475c0740 1 -- 172.21.15.70:0/3536797324 >> 172.21.15.70:0/3536797324 conn(0x55df873ead30 msgr2=0x55df873f0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e475c0740 1 -- 172.21.15.70:0/3536797324 shutdown_connections 2024-06-04T05:10:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e475c0740 1 -- 172.21.15.70:0/3536797324 wait complete. 2024-06-04T05:10:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e475c0740 20 client.18957 trim_cache size 0 max 16384 2024-06-04T05:10:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:10:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:10:09.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:10:09.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e447c8640 1 -- 172.21.15.70:0/3350953261 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e240b2a10 msgr2=0x7f4e240b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:09.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e447c8640 1 --2- 172.21.15.70:0/3350953261 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e240b2a10 0x7f4e240b4ed0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4e34005fd0 tx=0x7f4e3400d490 comp rx=0 tx=0).stop 2024-06-04T05:10:09.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e447c8640 1 -- 172.21.15.70:0/3350953261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c107940 msgr2=0x7f4e3c1a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:09.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e447c8640 1 --2- 172.21.15.70:0/3350953261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c107940 0x7f4e3c1a4ff0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00c970 tx=0x7f4e2c00ce30 comp rx=0 tx=0).stop 2024-06-04T05:10:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e447c8640 1 -- 172.21.15.70:0/3350953261 shutdown_connections 2024-06-04T05:10:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e447c8640 1 --2- 172.21.15.70:0/3350953261 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e240b2a10 0x7f4e240b4ed0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e447c8640 1 --2- 172.21.15.70:0/3350953261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c107940 0x7f4e3c1a4ff0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e447c8640 1 --2- 172.21.15.70:0/3350953261 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e3c102340 0x7f4e3c19a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e447c8640 1 --2- 172.21.15.70:0/3350953261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e3c101980 0x7f4e3c199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.877+0000 7f4e447c8640 1 -- 172.21.15.70:0/3350953261 >> 172.21.15.70:0/3350953261 conn(0x7f4e3c078ad0 msgr2=0x7f4e3c104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.878+0000 7f4e447c8640 1 -- 172.21.15.70:0/3350953261 shutdown_connections 2024-06-04T05:10:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:09.878+0000 7f4e447c8640 1 -- 172.21.15.70:0/3350953261 wait complete. 2024-06-04T05:10:10.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.334+0000 7f173629e700 1 -- 172.21.15.70:0/2518053991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17301037f0 msgr2=0x7f1730103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:10.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.334+0000 7f173629e700 1 --2- 172.21.15.70:0/2518053991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17301037f0 0x7f1730103bd0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f17200099c0 tx=0x7f1720009cd0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.334+0000 7f173629e700 1 -- 172.21.15.70:0/2518053991 shutdown_connections 2024-06-04T05:10:10.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.334+0000 7f173629e700 1 --2- 172.21.15.70:0/2518053991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17301088f0 0x7f1730110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.334+0000 7f173629e700 1 --2- 172.21.15.70:0/2518053991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17301037f0 0x7f1730103bd0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.334+0000 7f173629e700 1 --2- 172.21.15.70:0/2518053991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17301041a0 0x7f17301081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.334+0000 7f173629e700 1 -- 172.21.15.70:0/2518053991 >> 172.21.15.70:0/2518053991 conn(0x7f173007a6a0 msgr2=0x7f173007aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:10.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.335+0000 7f173629e700 1 -- 172.21.15.70:0/2518053991 shutdown_connections 2024-06-04T05:10:10.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.335+0000 7f173629e700 1 -- 172.21.15.70:0/2518053991 wait complete. 2024-06-04T05:10:10.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.335+0000 7f173629e700 1 Processor -- start 2024-06-04T05:10:10.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.335+0000 7f173629e700 1 -- start start 2024-06-04T05:10:10.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.336+0000 7f173629e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17301037f0 0x7f173019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.336+0000 7f173629e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17301041a0 0x7f173019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.336+0000 7f173629e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17301088f0 0x7f1730199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.336+0000 7f173629e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1730113590 con 0x7f17301041a0 2024-06-04T05:10:10.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.336+0000 7f173629e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1730113410 con 0x7f17301088f0 2024-06-04T05:10:10.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.336+0000 7f173629e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1730113710 con 0x7f17301037f0 2024-06-04T05:10:10.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.336+0000 7f172f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17301041a0 0x7f173019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.336+0000 7f172f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17301041a0 0x7f173019fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47086/0 (socket says 172.21.15.70:47086) 2024-06-04T05:10:10.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.336+0000 7f172f7fe700 1 -- 172.21.15.70:0/1278976382 learned_addr learned my addr 172.21.15.70:0/1278976382 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:10.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.336+0000 7f172ffff700 1 --2- 172.21.15.70:0/1278976382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17301037f0 0x7f173019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.336+0000 7f173483b700 1 --2- 172.21.15.70:0/1278976382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17301088f0 0x7f1730199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.336+0000 7f172f7fe700 1 -- 172.21.15.70:0/1278976382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17301037f0 msgr2=0x7f173019f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:10.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.336+0000 7f172f7fe700 1 --2- 172.21.15.70:0/1278976382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17301037f0 0x7f173019f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.336+0000 7f172f7fe700 1 -- 172.21.15.70:0/1278976382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17301088f0 msgr2=0x7f1730199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:10.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.336+0000 7f172f7fe700 1 --2- 172.21.15.70:0/1278976382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17301088f0 0x7f1730199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.336+0000 7f172f7fe700 1 -- 172.21.15.70:0/1278976382 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f173019a070 con 0x7f17301041a0 2024-06-04T05:10:10.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.336+0000 7f173483b700 1 --2- 172.21.15.70:0/1278976382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17301088f0 0x7f1730199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:10:10.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.337+0000 7f172f7fe700 1 --2- 172.21.15.70:0/1278976382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17301041a0 0x7f173019fc90 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f172400c920 tx=0x7f172400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:10.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.337+0000 7f172d7fa700 1 -- 172.21.15.70:0/1278976382 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1724007b00 con 0x7f17301041a0 2024-06-04T05:10:10.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.337+0000 7f173629e700 1 -- 172.21.15.70:0/1278976382 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f173019a360 con 0x7f17301041a0 2024-06-04T05:10:10.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.337+0000 7f172d7fa700 1 -- 172.21.15.70:0/1278976382 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1724004cc0 con 0x7f17301041a0 2024-06-04T05:10:10.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.337+0000 7f172d7fa700 1 -- 172.21.15.70:0/1278976382 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1724005440 con 0x7f17301041a0 2024-06-04T05:10:10.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.337+0000 7f173629e700 1 -- 172.21.15.70:0/1278976382 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17301a6620 con 0x7f17301041a0 2024-06-04T05:10:10.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.339+0000 7f172d7fa700 1 -- 172.21.15.70:0/1278976382 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f1724004750 con 0x7f17301041a0 2024-06-04T05:10:10.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.340+0000 7f172d7fa700 1 --2- 172.21.15.70:0/1278976382 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f17180b29f0 0x7f17180b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.340+0000 7f172ffff700 1 --2- 172.21.15.70:0/1278976382 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f17180b29f0 0x7f17180b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.340+0000 7f172d7fa700 1 -- 172.21.15.70:0/1278976382 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f17240e2140 con 0x7f17301041a0 2024-06-04T05:10:10.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.340+0000 7f172ffff700 1 --2- 172.21.15.70:0/1278976382 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f17180b29f0 0x7f17180b4eb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1720009990 tx=0x7f172001a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:10.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f1740b33b80 1 Processor -- start 2024-06-04T05:10:10.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f1740b33b80 1 -- start start 2024-06-04T05:10:10.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f1740b33b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5593181343a0 0x559318134780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f1740b33b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55931812e920 0x55931812eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f1740b33b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x55931813b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f1740b33b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55931808cdf0 con 0x55931812f2e0 2024-06-04T05:10:10.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f1740b33b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55931807c990 con 0x55931812e920 2024-06-04T05:10:10.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f1740b33b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5593180809d0 con 0x5593181343a0 2024-06-04T05:10:10.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f173483b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x55931813b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f173483b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x55931813b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47094/0 (socket says 172.21.15.70:47094) 2024-06-04T05:10:10.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f173483b700 1 -- 172.21.15.70:0/1127443104 learned_addr learned my addr 172.21.15.70:0/1127443104 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:10.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f172ffff700 1 --2- 172.21.15.70:0/1127443104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5593181343a0 0x559318134780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f172f7fe700 1 --2- 172.21.15.70:0/1127443104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55931812e920 0x55931812eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f173483b700 1 -- 172.21.15.70:0/1127443104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5593181343a0 msgr2=0x559318134780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:10.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f173483b700 1 --2- 172.21.15.70:0/1127443104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5593181343a0 0x559318134780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f173483b700 1 -- 172.21.15.70:0/1127443104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55931812e920 msgr2=0x55931812eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:10.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f173483b700 1 --2- 172.21.15.70:0/1127443104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55931812e920 0x55931812eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f173483b700 1 -- 172.21.15.70:0/1127443104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559318026030 con 0x55931812f2e0 2024-06-04T05:10:10.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f172f7fe700 1 --2- 172.21.15.70:0/1127443104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55931812e920 0x55931812eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:10:10.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.341+0000 7f172ffff700 1 --2- 172.21.15.70:0/1127443104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5593181343a0 0x559318134780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:10:10.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.342+0000 7f173483b700 1 --2- 172.21.15.70:0/1127443104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x55931813b100 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f17280087c0 tx=0x7f1728008ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba7061f566c3e66 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:10.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.342+0000 7f17167fc700 1 -- 172.21.15.70:0/1127443104 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1728003c00 con 0x55931812f2e0 2024-06-04T05:10:10.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.342+0000 7f17167fc700 1 -- 172.21.15.70:0/1127443104 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1728003da0 con 0x55931812f2e0 2024-06-04T05:10:10.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.342+0000 7f17167fc700 1 -- 172.21.15.70:0/1127443104 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1728012920 con 0x55931812f2e0 2024-06-04T05:10:10.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.343+0000 7f1740b33b80 1 -- 172.21.15.70:0/1127443104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 msgr2=0x55931813b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:10.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.343+0000 7f1740b33b80 1 --2- 172.21.15.70:0/1127443104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x55931813b100 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f17280087c0 tx=0x7f1728008ad0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.343+0000 7f1740b33b80 1 -- 172.21.15.70:0/1127443104 shutdown_connections 2024-06-04T05:10:10.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.343+0000 7f1740b33b80 1 --2- 172.21.15.70:0/1127443104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x55931813b100 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.343+0000 7f1740b33b80 1 --2- 172.21.15.70:0/1127443104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5593181343a0 0x559318134780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.343+0000 7f1740b33b80 1 --2- 172.21.15.70:0/1127443104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55931812e920 0x55931812eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.343+0000 7f1740b33b80 1 -- 172.21.15.70:0/1127443104 >> 172.21.15.70:0/1127443104 conn(0x55931805a2c0 msgr2=0x559318129460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:10.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.343+0000 7f1740b33b80 1 -- 172.21.15.70:0/1127443104 shutdown_connections 2024-06-04T05:10:10.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.343+0000 7f1740b33b80 1 -- 172.21.15.70:0/1127443104 wait complete. 2024-06-04T05:10:10.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.343+0000 7f1740b33b80 1 -- start start 2024-06-04T05:10:10.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.343+0000 7f1740b33b80 1 Processor -- start 2024-06-04T05:10:10.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f1740b33b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55931812e920 0x55931813acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f1740b33b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55931812f2e0 0x559318138020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f1740b33b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5593181343a0 0x559318138560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f1740b33b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559317f83020 con 0x5593181343a0 2024-06-04T05:10:10.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f1740b33b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559317fe4e20 con 0x55931812e920 2024-06-04T05:10:10.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f1740b33b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55931813b760 con 0x55931812f2e0 2024-06-04T05:10:10.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f1740b33b80 10 client.0 _ll_register_callbacks cb 0x5593180179d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:10.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f173483b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5593181343a0 0x559318138560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f173483b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5593181343a0 0x559318138560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47096/0 (socket says 172.21.15.70:47096) 2024-06-04T05:10:10.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f173483b700 1 -- 172.21.15.70:0/1165718148 learned_addr learned my addr 172.21.15.70:0/1165718148 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:10.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f172f7fe700 1 --2- 172.21.15.70:0/1165718148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55931812f2e0 0x559318138020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f172ffff700 1 --2- 172.21.15.70:0/1165718148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55931812e920 0x55931813acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f173483b700 1 -- 172.21.15.70:0/1165718148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55931812f2e0 msgr2=0x559318138020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:10.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f173483b700 1 --2- 172.21.15.70:0/1165718148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55931812f2e0 0x559318138020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f173483b700 1 -- 172.21.15.70:0/1165718148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55931812e920 msgr2=0x55931813acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:10.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f173483b700 1 --2- 172.21.15.70:0/1165718148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55931812e920 0x55931813acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f173483b700 1 -- 172.21.15.70:0/1165718148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559317fcc950 con 0x5593181343a0 2024-06-04T05:10:10.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f173483b700 1 --2- 172.21.15.70:0/1165718148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5593181343a0 0x559318138560 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f1728012540 tx=0x7f1728004230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0c6d8ba1a96fa4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:10.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f170f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:10.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f170f7fe700 1 -- 172.21.15.70:0/1165718148 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1728004710 con 0x5593181343a0 2024-06-04T05:10:10.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f170f7fe700 1 -- 172.21.15.70:0/1165718148 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f1728004ad0 con 0x5593181343a0 2024-06-04T05:10:10.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f1740b33b80 1 -- 172.21.15.70:0/1165718148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1728003da0 con 0x5593181343a0 2024-06-04T05:10:10.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f170f7fe700 1 -- 172.21.15.70:0/1165718148 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f172802b050 con 0x5593181343a0 2024-06-04T05:10:10.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f1740b33b80 10 client.18969 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:10.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.344+0000 7f1740b33b80 1 -- 172.21.15.70:0/1165718148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5593180367e0 con 0x5593181343a0 2024-06-04T05:10:10.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.345+0000 7f170f7fe700 1 -- 172.21.15.70:0/1165718148 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1728003da0 con 0x5593181343a0 2024-06-04T05:10:10.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.345+0000 7f170f7fe700 1 -- 172.21.15.70:0/1165718148 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f1728013070 con 0x5593181343a0 2024-06-04T05:10:10.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.345+0000 7f1740b33b80 10 client.18969 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:10.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.345+0000 7f1740b33b80 10 client.18969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:10:10.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.345+0000 7f1740b33b80 20 client.18969 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:10.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.345+0000 7f1740b33b80 1 --2- 172.21.15.70:0/1165718148 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5593181480e0 0x55931814a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.345+0000 7f1740b33b80 4 client.18969 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:10:10.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.345+0000 7f1740b33b80 1 -- 172.21.15.70:0/1165718148 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55931808ed40 con 0x5593181480e0 2024-06-04T05:10:10.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.345+0000 7f172ffff700 1 --2- 172.21.15.70:0/1165718148 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5593181480e0 0x55931814a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.346+0000 7f170f7fe700 10 client.18969 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:10.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.346+0000 7f172ffff700 1 --2- 172.21.15.70:0/1165718148 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5593181480e0 0x55931814a4d0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3599f888c8b0ace2 server_cookie=23c64eead1b57747 in_seq=0 out_seq=0 2024-06-04T05:10:10.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.346+0000 7f170f7fe700 1 -- 172.21.15.70:0/1165718148 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55931808ed40 con 0x5593181480e0 2024-06-04T05:10:10.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.346+0000 7f170f7fe700 10 client.18969 handle_command_reply: tid=0 2024-06-04T05:10:10.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.346+0000 7f1740b33b80 1 client.18969 shutdown 2024-06-04T05:10:10.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.346+0000 7f1740b33b80 1 -- 172.21.15.70:0/1165718148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5593181343a0 msgr2=0x559318138560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:10.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.346+0000 7f1740b33b80 1 --2- 172.21.15.70:0/1165718148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5593181343a0 0x559318138560 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f1728012540 tx=0x7f1728004230 comp rx=0 tx=0).stop 2024-06-04T05:10:10.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.347+0000 7f1740b33b80 1 -- 172.21.15.70:0/1165718148 shutdown_connections 2024-06-04T05:10:10.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.347+0000 7f1740b33b80 1 --2- 172.21.15.70:0/1165718148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5593181343a0 0x559318138560 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.347+0000 7f1740b33b80 1 --2- 172.21.15.70:0/1165718148 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5593181480e0 0x55931814a4d0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.347+0000 7f1740b33b80 1 --2- 172.21.15.70:0/1165718148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55931812e920 0x55931813acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.347+0000 7f1740b33b80 1 --2- 172.21.15.70:0/1165718148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55931812f2e0 0x559318138020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.347+0000 7f1740b33b80 1 -- 172.21.15.70:0/1165718148 >> 172.21.15.70:0/1165718148 conn(0x55931805a2c0 msgr2=0x559318128b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:10.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.347+0000 7f170f7fe700 0 client.18969 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:10.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.347+0000 7f1740b33b80 1 -- 172.21.15.70:0/1165718148 shutdown_connections 2024-06-04T05:10:10.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.347+0000 7f1740b33b80 1 -- 172.21.15.70:0/1165718148 wait complete. 2024-06-04T05:10:10.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.347+0000 7f1740b33b80 20 client.18969 trim_cache size 0 max 16384 2024-06-04T05:10:10.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.367+0000 7f1740b33b80 1 Processor -- start 2024-06-04T05:10:10.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.367+0000 7f1740b33b80 1 -- start start 2024-06-04T05:10:10.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.367+0000 7f1740b33b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55931812e920 0x559318133810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.367+0000 7f1740b33b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x5593181292f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.367+0000 7f1740b33b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5593181343a0 0x559318135090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.367+0000 7f1740b33b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f17301ade60 con 0x55931812f2e0 2024-06-04T05:10:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.367+0000 7f1740b33b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17301afac0 con 0x55931812e920 2024-06-04T05:10:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.367+0000 7f1740b33b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17301ae530 con 0x5593181343a0 2024-06-04T05:10:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.368+0000 7f172f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x5593181292f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.368+0000 7f172f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x5593181292f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47104/0 (socket says 172.21.15.70:47104) 2024-06-04T05:10:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.368+0000 7f172f7fe700 1 -- 172.21.15.70:0/2736979711 learned_addr learned my addr 172.21.15.70:0/2736979711 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.368+0000 7f172ffff700 1 --2- 172.21.15.70:0/2736979711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55931812e920 0x559318133810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.368+0000 7f173483b700 1 --2- 172.21.15.70:0/2736979711 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5593181343a0 0x559318135090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.368+0000 7f172f7fe700 1 -- 172.21.15.70:0/2736979711 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5593181343a0 msgr2=0x559318135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.368+0000 7f172f7fe700 1 --2- 172.21.15.70:0/2736979711 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5593181343a0 0x559318135090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.368+0000 7f172f7fe700 1 -- 172.21.15.70:0/2736979711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55931812e920 msgr2=0x559318133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.368+0000 7f172f7fe700 1 --2- 172.21.15.70:0/2736979711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55931812e920 0x559318133810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.368+0000 7f172f7fe700 1 -- 172.21.15.70:0/2736979711 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559318136590 con 0x55931812f2e0 2024-06-04T05:10:10.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.368+0000 7f172f7fe700 1 --2- 172.21.15.70:0/2736979711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x5593181292f0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f17240df980 tx=0x7f17240a8e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe3403f219183b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:10.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.368+0000 7f1715ffb700 1 -- 172.21.15.70:0/2736979711 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172401d040 con 0x55931812f2e0 2024-06-04T05:10:10.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.368+0000 7f1715ffb700 1 -- 172.21.15.70:0/2736979711 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f17240a8fa0 con 0x55931812f2e0 2024-06-04T05:10:10.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.368+0000 7f1715ffb700 1 -- 172.21.15.70:0/2736979711 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1724007ca0 con 0x55931812f2e0 2024-06-04T05:10:10.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.369+0000 7f1740b33b80 1 -- 172.21.15.70:0/2736979711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 msgr2=0x5593181292f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:10.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.369+0000 7f1740b33b80 1 --2- 172.21.15.70:0/2736979711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x5593181292f0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f17240df980 tx=0x7f17240a8e60 comp rx=0 tx=0).stop 2024-06-04T05:10:10.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.369+0000 7f1740b33b80 1 -- 172.21.15.70:0/2736979711 shutdown_connections 2024-06-04T05:10:10.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.369+0000 7f1740b33b80 1 --2- 172.21.15.70:0/2736979711 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5593181343a0 0x559318135090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.369+0000 7f1740b33b80 1 --2- 172.21.15.70:0/2736979711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55931812e920 0x559318133810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.369+0000 7f1740b33b80 1 --2- 172.21.15.70:0/2736979711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x5593181292f0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.369+0000 7f1740b33b80 1 -- 172.21.15.70:0/2736979711 >> 172.21.15.70:0/2736979711 conn(0x55931805a2c0 msgr2=0x559318133100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:10.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.369+0000 7f1740b33b80 1 -- 172.21.15.70:0/2736979711 shutdown_connections 2024-06-04T05:10:10.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.369+0000 7f1740b33b80 1 -- 172.21.15.70:0/2736979711 wait complete. 2024-06-04T05:10:10.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.369+0000 7f1740b33b80 1 -- start start 2024-06-04T05:10:10.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f1740b33b80 1 Processor -- start 2024-06-04T05:10:10.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f1740b33b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55931812e920 0x55931814ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f1740b33b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x55931814f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f1740b33b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593181343a0 0x55931814f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f1740b33b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f17301af390 con 0x55931812f2e0 2024-06-04T05:10:10.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f1740b33b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17301aec60 con 0x5593181343a0 2024-06-04T05:10:10.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f1740b33b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17301abe20 con 0x55931812e920 2024-06-04T05:10:10.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f1740b33b80 10 client.0 _ll_register_callbacks cb 0x5593180179d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:10.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f172f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x55931814f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f172f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x55931814f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47118/0 (socket says 172.21.15.70:47118) 2024-06-04T05:10:10.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f172f7fe700 1 -- 172.21.15.70:0/1847293961 learned_addr learned my addr 172.21.15.70:0/1847293961 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:10.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f172ffff700 1 --2- 172.21.15.70:0/1847293961 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55931812e920 0x55931814ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f173483b700 1 --2- 172.21.15.70:0/1847293961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593181343a0 0x55931814f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f172f7fe700 1 -- 172.21.15.70:0/1847293961 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55931812e920 msgr2=0x55931814ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:10.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f172f7fe700 1 --2- 172.21.15.70:0/1847293961 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55931812e920 0x55931814ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f172f7fe700 1 -- 172.21.15.70:0/1847293961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593181343a0 msgr2=0x55931814f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:10.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f172f7fe700 1 --2- 172.21.15.70:0/1847293961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593181343a0 0x55931814f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f172f7fe700 1 -- 172.21.15.70:0/1847293961 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559318150070 con 0x55931812f2e0 2024-06-04T05:10:10.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f173483b700 1 --2- 172.21.15.70:0/1847293961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593181343a0 0x55931814f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:10:10.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f172f7fe700 1 --2- 172.21.15.70:0/1847293961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x55931814f290 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f17240a6700 tx=0x7f17240a9540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a3cf87688d100a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:10.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.370+0000 7f170f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:10.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.371+0000 7f170f7fe700 1 -- 172.21.15.70:0/1847293961 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172401d040 con 0x55931812f2e0 2024-06-04T05:10:10.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.371+0000 7f1740b33b80 1 -- 172.21.15.70:0/1847293961 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17240a8fa0 con 0x55931812f2e0 2024-06-04T05:10:10.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.371+0000 7f170f7fe700 1 -- 172.21.15.70:0/1847293961 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f17240a96f0 con 0x55931812f2e0 2024-06-04T05:10:10.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.371+0000 7f170f7fe700 1 -- 172.21.15.70:0/1847293961 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f17240a98f0 con 0x55931812f2e0 2024-06-04T05:10:10.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.371+0000 7f1740b33b80 10 client.18975 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:10.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.371+0000 7f1740b33b80 1 -- 172.21.15.70:0/1847293961 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5593181b74f0 con 0x55931812f2e0 2024-06-04T05:10:10.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.371+0000 7f170f7fe700 1 -- 172.21.15.70:0/1847293961 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f17240a8fa0 con 0x55931812f2e0 2024-06-04T05:10:10.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.371+0000 7f170f7fe700 1 -- 172.21.15.70:0/1847293961 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f17240b0cc0 con 0x55931812f2e0 2024-06-04T05:10:10.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.371+0000 7f1740b33b80 10 client.18975 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:10.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.371+0000 7f1740b33b80 10 client.18975 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:10:10.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.371+0000 7f1740b33b80 20 client.18975 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:10.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.371+0000 7f1740b33b80 1 --2- 172.21.15.70:0/1847293961 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5593181b7b50 0x5593181b9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:10.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.371+0000 7f1740b33b80 4 client.18975 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:10:10.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.371+0000 7f1740b33b80 1 -- 172.21.15.70:0/1847293961 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559318088520 con 0x5593181b7b50 2024-06-04T05:10:10.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.372+0000 7f172ffff700 1 --2- 172.21.15.70:0/1847293961 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5593181b7b50 0x5593181b9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:10.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.372+0000 7f170f7fe700 10 client.18975 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:10.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.372+0000 7f172ffff700 1 --2- 172.21.15.70:0/1847293961 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5593181b7b50 0x5593181b9f70 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4e83cfea25aa3d2 server_cookie=657fb0a12e4d21e5 in_seq=0 out_seq=0 2024-06-04T05:10:10.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.373+0000 7f170f7fe700 1 -- 172.21.15.70:0/1847293961 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559318088520 con 0x5593181b7b50 2024-06-04T05:10:10.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.373+0000 7f170f7fe700 10 client.18975 handle_command_reply: tid=0 2024-06-04T05:10:10.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.373+0000 7f1740b33b80 1 client.18975 shutdown 2024-06-04T05:10:10.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.373+0000 7f1740b33b80 1 -- 172.21.15.70:0/1847293961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 msgr2=0x55931814f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:10.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.373+0000 7f1740b33b80 1 --2- 172.21.15.70:0/1847293961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x55931814f290 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f17240a6700 tx=0x7f17240a9540 comp rx=0 tx=0).stop 2024-06-04T05:10:10.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.373+0000 7f1740b33b80 1 -- 172.21.15.70:0/1847293961 shutdown_connections 2024-06-04T05:10:10.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.373+0000 7f1740b33b80 1 --2- 172.21.15.70:0/1847293961 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5593181b7b50 0x5593181b9f70 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.373+0000 7f1740b33b80 1 --2- 172.21.15.70:0/1847293961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593181343a0 0x55931814f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.373+0000 7f1740b33b80 1 --2- 172.21.15.70:0/1847293961 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55931812e920 0x55931814ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.373+0000 7f170f7fe700 0 client.18975 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:10.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.373+0000 7f1740b33b80 1 --2- 172.21.15.70:0/1847293961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55931812f2e0 0x55931814f290 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.373+0000 7f1740b33b80 1 -- 172.21.15.70:0/1847293961 >> 172.21.15.70:0/1847293961 conn(0x55931805a2c0 msgr2=0x559318131d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:10.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.373+0000 7f1740b33b80 1 -- 172.21.15.70:0/1847293961 shutdown_connections 2024-06-04T05:10:10.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.373+0000 7f1740b33b80 1 -- 172.21.15.70:0/1847293961 wait complete. 2024-06-04T05:10:10.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.373+0000 7f1740b33b80 20 client.18975 trim_cache size 0 max 16384 2024-06-04T05:10:10.382 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:10:10.382 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:10:10.382 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "09c3fd92-f5fa-407c-8cca-18ae539c4e8f", 2024-06-04T05:10:10.382 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:10:10.382 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:10:10.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.374+0000 7f173629e700 1 -- 172.21.15.70:0/1278976382 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f17180b29f0 msgr2=0x7f17180b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:10.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.374+0000 7f173629e700 1 --2- 172.21.15.70:0/1278976382 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f17180b29f0 0x7f17180b4eb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1720009990 tx=0x7f172001a040 comp rx=0 tx=0).stop 2024-06-04T05:10:10.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.374+0000 7f173629e700 1 -- 172.21.15.70:0/1278976382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17301041a0 msgr2=0x7f173019fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:10.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.374+0000 7f173629e700 1 --2- 172.21.15.70:0/1278976382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17301041a0 0x7f173019fc90 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f172400c920 tx=0x7f172400cc30 comp rx=0 tx=0).stop 2024-06-04T05:10:10.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.374+0000 7f173629e700 1 -- 172.21.15.70:0/1278976382 shutdown_connections 2024-06-04T05:10:10.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.374+0000 7f173629e700 1 --2- 172.21.15.70:0/1278976382 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f17180b29f0 0x7f17180b4eb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.374+0000 7f173629e700 1 --2- 172.21.15.70:0/1278976382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17301088f0 0x7f1730199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.374+0000 7f173629e700 1 --2- 172.21.15.70:0/1278976382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17301037f0 0x7f173019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.374+0000 7f173629e700 1 --2- 172.21.15.70:0/1278976382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17301041a0 0x7f173019fc90 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:10.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.374+0000 7f173629e700 1 -- 172.21.15.70:0/1278976382 >> 172.21.15.70:0/1278976382 conn(0x7f173007a6a0 msgr2=0x7f1730101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:10.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.374+0000 7f173629e700 1 -- 172.21.15.70:0/1278976382 shutdown_connections 2024-06-04T05:10:10.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:10.374+0000 7f173629e700 1 -- 172.21.15.70:0/1278976382 wait complete. 2024-06-04T05:10:10.686 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:10:10 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:10:10.383+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000020c [...15,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/EXCEL/ auth{2=1} v34416 ap=4 f(v2128 m2024-06-04T05:10:03.667148+0000 8=8+0) n(v55 rc2024-06-04T05:10:03.670921+0000 b1283356 9=8+1) old_inodes=6 (inest sync->mix g=2) (ipolicy sync r=1) (iquiesce lock x=1 by request(mds.0:443 nref=3)) caps={15231=pAsLsXs/-@33036} | dirtyscattered=0 request=1 lock=2 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55d6decee100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000020c:[<0x10000000009/EXCEL v14>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x1000000020c:[<0x10000000009/EXCEL v34416>,<0x10000000008/~dmtmp v31874>,<0x10000000007/client0 v1184>,<0x10000000006/clients v1165>,<0x10000000005/tmp v1120>,<0x10000000003/client.0 v1087>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v1101>,<0x10000000001/sv_0 v1073>,<0x10000000000/qa v1066>,<0x1/volumes v1019>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:10:03.650265+0000 7=7+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:10:03.670921+0000 b1283356 9=8+1)","memory_value.dirstat":"f(v2128 m2024-06-04T05:10:03.667148+0000 8=8+0)","memory_value.rstat":"n(v55 rc2024-06-04T05:10:03.670921+0000 b1283356 9=8+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:10:10.687 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:10:10 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:10:10.384+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000208 [...15,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/WORD/ auth{2=1} v34414 ap=4 f(v1344 m2024-06-04T05:10:01.126044+0000 5=5+0) n(v53 rc2024-06-04T05:10:01.126044+0000 b814326 6=5+1) old_inodes=6 (inest sync->mix g=2) (ipolicy sync r=1) (iquiesce lock x=1 by request(mds.0:444 nref=3)) caps={15231=pAsLsXs/-@30236} | dirtyscattered=0 request=1 lock=2 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55d6decef700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000208:[<0x10000000009/WORD v8>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x10000000208:[<0x10000000009/WORD v34414>,<0x10000000008/~dmtmp v31874>,<0x10000000007/client0 v1184>,<0x10000000006/clients v1165>,<0x10000000005/tmp v1120>,<0x10000000003/client.0 v1087>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v1101>,<0x10000000001/sv_0 v1073>,<0x10000000000/qa v1066>,<0x1/volumes v1019>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:10:01.111320+0000 6=6+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:10:01.126044+0000 b814326 6=5+1)","memory_value.dirstat":"f(v1344 m2024-06-04T05:10:01.126044+0000 5=5+0)","memory_value.rstat":"n(v53 rc2024-06-04T05:10:01.126044+0000 b814326 6=5+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:10:10.687 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:10:10 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:10:10.396+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000020f [...15,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ auth{2=1} v34418 ap=4 f(v1568 m2024-06-04T05:10:00.542446+0000 7=7+0) n(v52 rc2024-06-04T05:10:00.542446+0000 b1462582 8=7+1) old_inodes=6 (inest sync->mix g=2) (ipolicy sync r=1) (iquiesce lock x=1 by request(mds.0:449 nref=3)) caps={15231=pAsLsXs/-@30066} | dirtyscattered=0 request=1 lock=2 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55d6de801700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000020f:[<0x10000000009/PWRPNT v234>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x1000000020f:[<0x10000000009/PWRPNT v34418>,<0x10000000008/~dmtmp v31878>,<0x10000000007/client0 v1186>,<0x10000000006/clients v1167>,<0x10000000005/tmp v1122>,<0x10000000003/client.0 v1087>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v1101>,<0x10000000001/sv_0 v1073>,<0x10000000000/qa v1066>,<0x1/volumes v1019>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:10:00.521544+0000 8=8+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:10:00.542446+0000 b1462582 8=7+1)","memory_value.dirstat":"f(v1568 m2024-06-04T05:10:00.542446+0000 7=7+0)","memory_value.rstat":"n(v52 rc2024-06-04T05:10:00.542446+0000 b1462582 8=7+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:10:10.687 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:10:10 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:10:10.397+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000023e [...15,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/ACCESS/ auth{2=1} v34420 ap=3 f(v784 m2024-06-04T05:10:04.930690+0000 4=4+0) n(v54 rc2024-06-04T05:10:04.930690+0000 b363650 5=4+1) old_inodes=6 (inest sync->mix g=2) (ipolicy sync r=1) (iquiesce lock x=1 by request(mds.0:450 nref=3)) caps={15231=pAsLsXs/-@27705} | dirtyscattered=0 request=1 lock=2 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55d6df029080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000023e:[<0x10000000009/ACCESS v112>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x1000000023e:[<0x10000000009/ACCESS v34420>,<0x10000000008/~dmtmp v31878>,<0x10000000007/client0 v1186>,<0x10000000006/clients v1167>,<0x10000000005/tmp v1122>,<0x10000000003/client.0 v1087>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v1101>,<0x10000000001/sv_0 v1073>,<0x10000000000/qa v1066>,<0x1/volumes v1019>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:10:04.919981+0000 5=5+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:10:04.930690+0000 b363650 5=4+1)","memory_value.dirstat":"f(v784 m2024-06-04T05:10:04.930690+0000 4=4+0)","memory_value.rstat":"n(v54 rc2024-06-04T05:10:04.930690+0000 b363650 5=4+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:10:10.741 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.63 MB/sec execute 543 sec latency 2750.781 ms 2024-06-04T05:10:11.063 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:10:11.346 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: pgmap v405: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 877 KiB/s wr, 73 op/s 2024-06-04T05:10:11.346 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:10:11.346 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.346 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:10:11.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:10:11.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:10:11.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:10:11.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: Scrub error on inode 0x1000000020c (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/EXCEL) see mds.i log and `damage ls` output for details 2024-06-04T05:10:11.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: Scrub error on inode 0x10000000208 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/WORD) see mds.i log and `damage ls` output for details 2024-06-04T05:10:11.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:10:11.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:10:11.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.348 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT) see mds.i log and `damage ls` output for details 2024-06-04T05:10:11.348 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: Scrub error on inode 0x1000000023e (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/ACCESS) see mds.i log and `damage ls` output for details 2024-06-04T05:10:11.348 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:11 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:10:11.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: pgmap v405: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 877 KiB/s wr, 73 op/s 2024-06-04T05:10:11.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:10:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:10:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:10:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:10:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:10:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: Scrub error on inode 0x1000000020c (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/EXCEL) see mds.i log and `damage ls` output for details 2024-06-04T05:10:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: Scrub error on inode 0x10000000208 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/WORD) see mds.i log and `damage ls` output for details 2024-06-04T05:10:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:10:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:10:11.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT) see mds.i log and `damage ls` output for details 2024-06-04T05:10:11.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: Scrub error on inode 0x1000000023e (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/ACCESS) see mds.i log and `damage ls` output for details 2024-06-04T05:10:11.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:11 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: pgmap v405: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 877 KiB/s wr, 73 op/s 2024-06-04T05:10:11.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:10:11.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:10:11.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:10:11.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:10:11.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:10:11.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: Scrub error on inode 0x1000000020c (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/EXCEL) see mds.i log and `damage ls` output for details 2024-06-04T05:10:11.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: Scrub error on inode 0x10000000208 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/WORD) see mds.i log and `damage ls` output for details 2024-06-04T05:10:11.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:10:11.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:10:11.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT) see mds.i log and `damage ls` output for details 2024-06-04T05:10:11.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: Scrub error on inode 0x1000000023e (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/ACCESS) see mds.i log and `damage ls` output for details 2024-06-04T05:10:11.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:11 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:10:11.741 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.63 MB/sec execute 544 sec latency 3750.817 ms 2024-06-04T05:10:12.741 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.62 MB/sec execute 545 sec latency 4750.907 ms 2024-06-04T05:10:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:13 smithi098 ceph-mon[35569]: pgmap v406: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 899 KiB/s wr, 107 op/s 2024-06-04T05:10:13.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:13 smithi070 ceph-mon[32631]: pgmap v406: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 899 KiB/s wr, 107 op/s 2024-06-04T05:10:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:13 smithi112 ceph-mon[36453]: pgmap v406: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 899 KiB/s wr, 107 op/s 2024-06-04T05:10:13.742 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.62 MB/sec execute 546 sec latency 5750.942 ms 2024-06-04T05:10:13.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.868+0000 7fcdefd4c700 1 -- 172.21.15.70:0/691871515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcde8102e00 msgr2=0x7fcde810f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:13.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.868+0000 7fcdefd4c700 1 --2- 172.21.15.70:0/691871515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcde8102e00 0x7fcde810f2f0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fcde000b0a0 tx=0x7fcde000b3b0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.868+0000 7fcdefd4c700 1 -- 172.21.15.70:0/691871515 shutdown_connections 2024-06-04T05:10:13.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.868+0000 7fcdefd4c700 1 --2- 172.21.15.70:0/691871515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcde8102e00 0x7fcde810f2f0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.868+0000 7fcdefd4c700 1 --2- 172.21.15.70:0/691871515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcde8108520 0x7fcde8108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.868+0000 7fcdefd4c700 1 --2- 172.21.15.70:0/691871515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcde8102440 0x7fcde81028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.868+0000 7fcdefd4c700 1 -- 172.21.15.70:0/691871515 >> 172.21.15.70:0/691871515 conn(0x7fcde80793f0 msgr2=0x7fcde8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:13.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.868+0000 7fcdefd4c700 1 -- 172.21.15.70:0/691871515 shutdown_connections 2024-06-04T05:10:13.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.868+0000 7fcdefd4c700 1 -- 172.21.15.70:0/691871515 wait complete. 2024-06-04T05:10:13.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcdefd4c700 1 Processor -- start 2024-06-04T05:10:13.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcdefd4c700 1 -- start start 2024-06-04T05:10:13.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcdefd4c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcde8102440 0x7fcde819a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcdefd4c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcde8102e00 0x7fcde819ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcdefd4c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcde8108520 0x7fcde81a59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcdefd4c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcde81124a0 con 0x7fcde8102e00 2024-06-04T05:10:13.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcdefd4c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcde8112320 con 0x7fcde8108520 2024-06-04T05:10:13.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcdefd4c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcde8112620 con 0x7fcde8102440 2024-06-04T05:10:13.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcded2e7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcde8102e00 0x7fcde819ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:13.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcded2e7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcde8102e00 0x7fcde819ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47144/0 (socket says 172.21.15.70:47144) 2024-06-04T05:10:13.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcded2e7700 1 -- 172.21.15.70:0/544463218 learned_addr learned my addr 172.21.15.70:0/544463218 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:13.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcdedae8700 1 --2- 172.21.15.70:0/544463218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcde8102440 0x7fcde819a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:13.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcdee2e9700 1 --2- 172.21.15.70:0/544463218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcde8108520 0x7fcde81a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:13.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcded2e7700 1 -- 172.21.15.70:0/544463218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcde8102440 msgr2=0x7fcde819a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:13.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcded2e7700 1 --2- 172.21.15.70:0/544463218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcde8102440 0x7fcde819a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcded2e7700 1 -- 172.21.15.70:0/544463218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcde8108520 msgr2=0x7fcde81a59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:13.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcded2e7700 1 --2- 172.21.15.70:0/544463218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcde8108520 0x7fcde81a59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.869+0000 7fcded2e7700 1 -- 172.21.15.70:0/544463218 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde8198730 con 0x7fcde8102e00 2024-06-04T05:10:13.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.870+0000 7fcded2e7700 1 --2- 172.21.15.70:0/544463218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcde8102e00 0x7fcde819ae80 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fcdd800d980 tx=0x7fcdd800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:13.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.870+0000 7fcddeffd700 1 -- 172.21.15.70:0/544463218 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8017070 con 0x7fcde8102e00 2024-06-04T05:10:13.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.870+0000 7fcddeffd700 1 -- 172.21.15.70:0/544463218 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcdd8005020 con 0x7fcde8102e00 2024-06-04T05:10:13.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.870+0000 7fcdefd4c700 1 -- 172.21.15.70:0/544463218 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcde8198a20 con 0x7fcde8102e00 2024-06-04T05:10:13.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.870+0000 7fcddeffd700 1 -- 172.21.15.70:0/544463218 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8009ab0 con 0x7fcde8102e00 2024-06-04T05:10:13.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.870+0000 7fcdefd4c700 1 -- 172.21.15.70:0/544463218 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcde8198f10 con 0x7fcde8102e00 2024-06-04T05:10:13.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.873+0000 7fcddeffd700 1 -- 172.21.15.70:0/544463218 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fcdd80051d0 con 0x7fcde8102e00 2024-06-04T05:10:13.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.874+0000 7fcddeffd700 1 --2- 172.21.15.70:0/544463218 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcdd40b29f0 0x7fcdd40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.874+0000 7fcdedae8700 1 --2- 172.21.15.70:0/544463218 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcdd40b29f0 0x7fcdd40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:13.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.874+0000 7fcddeffd700 1 -- 172.21.15.70:0/544463218 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7fcdd80e2550 con 0x7fcde8102e00 2024-06-04T05:10:13.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.874+0000 7fcdfa5e1b80 1 Processor -- start 2024-06-04T05:10:13.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.874+0000 7fcdfa5e1b80 1 -- start start 2024-06-04T05:10:13.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.874+0000 7fcdedae8700 1 --2- 172.21.15.70:0/544463218 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcdd40b29f0 0x7fcdd40b4eb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcde4005950 tx=0x7fcde400a300 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:13.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.874+0000 7fcdfa5e1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dce860 0x55c930dcec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.874+0000 7fcdfa5e1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dcf210 0x55c930dd2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.874+0000 7fcdfa5e1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dd2f60 0x55c930dd33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.874+0000 7fcdfa5e1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c930d2dc90 con 0x55c930dd2f60 2024-06-04T05:10:13.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.874+0000 7fcdfa5e1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c930d1d830 con 0x55c930dce860 2024-06-04T05:10:13.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.874+0000 7fcdfa5e1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c930d21870 con 0x55c930dcf210 2024-06-04T05:10:13.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.875+0000 7fcdee2e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dd2f60 0x55c930dd33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:13.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.875+0000 7fcdee2e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dd2f60 0x55c930dd33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47146/0 (socket says 172.21.15.70:47146) 2024-06-04T05:10:13.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.875+0000 7fcdee2e9700 1 -- 172.21.15.70:0/611392516 learned_addr learned my addr 172.21.15.70:0/611392516 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:13.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.875+0000 7fcded2e7700 1 --2- 172.21.15.70:0/611392516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dcf210 0x55c930dd2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:13.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.875+0000 7fcdedae8700 1 --2- 172.21.15.70:0/611392516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dce860 0x55c930dcec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:13.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.875+0000 7fcdee2e9700 1 -- 172.21.15.70:0/611392516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dcf210 msgr2=0x55c930dd2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:13.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.875+0000 7fcdee2e9700 1 --2- 172.21.15.70:0/611392516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dcf210 0x55c930dd2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.875+0000 7fcdee2e9700 1 -- 172.21.15.70:0/611392516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dce860 msgr2=0x55c930dcec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:13.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.875+0000 7fcdee2e9700 1 --2- 172.21.15.70:0/611392516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dce860 0x55c930dcec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.875+0000 7fcdee2e9700 1 -- 172.21.15.70:0/611392516 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c930c6d8d0 con 0x55c930dd2f60 2024-06-04T05:10:13.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.875+0000 7fcdedae8700 1 --2- 172.21.15.70:0/611392516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dce860 0x55c930dcec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:10:13.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.875+0000 7fcdee2e9700 1 --2- 172.21.15.70:0/611392516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dd2f60 0x55c930dd33a0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009e50 tx=0x7fcde00090d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f755241c101ff69 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:13.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.875+0000 7fcdd3fff700 1 -- 172.21.15.70:0/611392516 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde0004a50 con 0x55c930dd2f60 2024-06-04T05:10:13.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.875+0000 7fcdd3fff700 1 -- 172.21.15.70:0/611392516 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcde0004bf0 con 0x55c930dd2f60 2024-06-04T05:10:13.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.875+0000 7fcdd3fff700 1 -- 172.21.15.70:0/611392516 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde001c6a0 con 0x55c930dd2f60 2024-06-04T05:10:13.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.876+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/611392516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dd2f60 msgr2=0x55c930dd33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:13.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.876+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/611392516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dd2f60 0x55c930dd33a0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009e50 tx=0x7fcde00090d0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.876+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/611392516 shutdown_connections 2024-06-04T05:10:13.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.876+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/611392516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dd2f60 0x55c930dd33a0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.876+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/611392516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dce860 0x55c930dcec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.876+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/611392516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dcf210 0x55c930dd2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.876+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/611392516 >> 172.21.15.70:0/611392516 conn(0x55c930cfc080 msgr2=0x55c930dc8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:13.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.876+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/611392516 shutdown_connections 2024-06-04T05:10:13.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.876+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/611392516 wait complete. 2024-06-04T05:10:13.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.876+0000 7fcdfa5e1b80 1 -- start start 2024-06-04T05:10:13.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcdfa5e1b80 1 Processor -- start 2024-06-04T05:10:13.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcdfa5e1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dce860 0x55c930dda310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcdfa5e1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dcf210 0x55c930dd7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcdfa5e1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dd2f60 0x55c930dd7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcdfa5e1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c930c23fa0 con 0x55c930dcf210 2024-06-04T05:10:13.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcdfa5e1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c930cbf7c0 con 0x55c930dd2f60 2024-06-04T05:10:13.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcdfa5e1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c930ddb0c0 con 0x55c930dce860 2024-06-04T05:10:13.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcdfa5e1b80 10 client.0 _ll_register_callbacks cb 0x55c930cb8920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:13.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcded2e7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dcf210 0x55c930dd7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:13.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcded2e7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dcf210 0x55c930dd7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47150/0 (socket says 172.21.15.70:47150) 2024-06-04T05:10:13.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcded2e7700 1 -- 172.21.15.70:0/3860792034 learned_addr learned my addr 172.21.15.70:0/3860792034 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:13.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcdedae8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dce860 0x55c930dda310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:13.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcdee2e9700 1 --2- 172.21.15.70:0/3860792034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dd2f60 0x55c930dd7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:13.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcded2e7700 1 -- 172.21.15.70:0/3860792034 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dce860 msgr2=0x55c930dda310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:13.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcded2e7700 1 --2- 172.21.15.70:0/3860792034 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dce860 0x55c930dda310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcded2e7700 1 -- 172.21.15.70:0/3860792034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dd2f60 msgr2=0x55c930dd7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:13.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcded2e7700 1 --2- 172.21.15.70:0/3860792034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dd2f60 0x55c930dd7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcded2e7700 1 -- 172.21.15.70:0/3860792034 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c930cbc690 con 0x55c930dcf210 2024-06-04T05:10:13.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.877+0000 7fcdee2e9700 1 --2- 172.21.15.70:0/3860792034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dd2f60 0x55c930dd7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:10:13.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.878+0000 7fcded2e7700 1 --2- 172.21.15.70:0/3860792034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dcf210 0x55c930dd7610 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80a5880 tx=0x7fcdd80a9020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67dae7ec09f680fe server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:13.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.878+0000 7fcdd17fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:13.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.878+0000 7fcdd17fa700 1 -- 172.21.15.70:0/3860792034 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd80c3060 con 0x55c930dcf210 2024-06-04T05:10:13.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.878+0000 7fcdd17fa700 1 -- 172.21.15.70:0/3860792034 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7fcdd80a9a60 con 0x55c930dcf210 2024-06-04T05:10:13.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.878+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/3860792034 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcde0004bf0 con 0x55c930dcf210 2024-06-04T05:10:13.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.878+0000 7fcdd17fa700 1 -- 172.21.15.70:0/3860792034 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fcdd8009e60 con 0x55c930dcf210 2024-06-04T05:10:13.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.878+0000 7fcdfa5e1b80 10 client.18987 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:13.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.878+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/3860792034 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55c930de57c0 con 0x55c930dcf210 2024-06-04T05:10:13.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.878+0000 7fcdd17fa700 1 -- 172.21.15.70:0/3860792034 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcde0004bf0 con 0x55c930dcf210 2024-06-04T05:10:13.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.878+0000 7fcdd17fa700 1 -- 172.21.15.70:0/3860792034 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7fcdd80b21d0 con 0x55c930dcf210 2024-06-04T05:10:13.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.878+0000 7fcdfa5e1b80 10 client.18987 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:13.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.878+0000 7fcdfa5e1b80 10 client.18987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:10:13.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.878+0000 7fcdfa5e1b80 20 client.18987 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:13.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.878+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/3860792034 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c930de7e00 0x55c930dea1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.878+0000 7fcdfa5e1b80 4 client.18987 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:10:13.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.878+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/3860792034 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c930d9a9e0 con 0x55c930de7e00 2024-06-04T05:10:13.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.879+0000 7fcdedae8700 1 --2- 172.21.15.70:0/3860792034 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c930de7e00 0x55c930dea1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:13.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.879+0000 7fcdedae8700 1 --2- 172.21.15.70:0/3860792034 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c930de7e00 0x55c930dea1f0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c956c5e19a5bbbb server_cookie=f57c7ea92ad12771 in_seq=0 out_seq=0 2024-06-04T05:10:13.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.879+0000 7fcdd17fa700 10 client.18987 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:13.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.880+0000 7fcdd17fa700 1 -- 172.21.15.70:0/3860792034 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c930d9a9e0 con 0x55c930de7e00 2024-06-04T05:10:13.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.880+0000 7fcdd17fa700 10 client.18987 handle_command_reply: tid=0 2024-06-04T05:10:13.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.880+0000 7fcdfa5e1b80 1 client.18987 shutdown 2024-06-04T05:10:13.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.880+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/3860792034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dcf210 msgr2=0x55c930dd7610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:13.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.880+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/3860792034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dcf210 0x55c930dd7610 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80a5880 tx=0x7fcdd80a9020 comp rx=0 tx=0).stop 2024-06-04T05:10:13.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.880+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/3860792034 shutdown_connections 2024-06-04T05:10:13.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.880+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/3860792034 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c930de7e00 0x55c930dea1f0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.880+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/3860792034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dd2f60 0x55c930dd7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.880+0000 7fcdd17fa700 0 client.18987 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:13.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.880+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/3860792034 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dce860 0x55c930dda310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.880+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/3860792034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dcf210 0x55c930dd7610 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.880+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/3860792034 >> 172.21.15.70:0/3860792034 conn(0x55c930cfc080 msgr2=0x55c930dd4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:13.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.880+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/3860792034 shutdown_connections 2024-06-04T05:10:13.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.880+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/3860792034 wait complete. 2024-06-04T05:10:13.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.880+0000 7fcdfa5e1b80 20 client.18987 trim_cache size 0 max 16384 2024-06-04T05:10:13.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.900+0000 7fcdfa5e1b80 1 Processor -- start 2024-06-04T05:10:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.900+0000 7fcdfa5e1b80 1 -- start start 2024-06-04T05:10:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdfa5e1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dce860 0x55c930dd7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdfa5e1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dcf210 0x55c930dd9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdfa5e1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dd2f60 0x55c930dc8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdfa5e1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcde81afec0 con 0x55c930dce860 2024-06-04T05:10:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdfa5e1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcde81b1b20 con 0x55c930dcf210 2024-06-04T05:10:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdfa5e1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcde81b0590 con 0x55c930dd2f60 2024-06-04T05:10:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdedae8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dce860 0x55c930dd7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdedae8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dce860 0x55c930dd7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47162/0 (socket says 172.21.15.70:47162) 2024-06-04T05:10:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdedae8700 1 -- 172.21.15.70:0/954573341 learned_addr learned my addr 172.21.15.70:0/954573341 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcded2e7700 1 --2- 172.21.15.70:0/954573341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dcf210 0x55c930dd9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:13.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdedae8700 1 -- 172.21.15.70:0/954573341 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dd2f60 msgr2=0x55c930dc8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:13.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdedae8700 1 --2- 172.21.15.70:0/954573341 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dd2f60 0x55c930dc8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdedae8700 1 -- 172.21.15.70:0/954573341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dcf210 msgr2=0x55c930dd9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:13.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdedae8700 1 --2- 172.21.15.70:0/954573341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dcf210 0x55c930dd9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdedae8700 1 -- 172.21.15.70:0/954573341 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c930dc9d40 con 0x55c930dce860 2024-06-04T05:10:13.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcded2e7700 1 --2- 172.21.15.70:0/954573341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dcf210 0x55c930dd9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:10:13.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdedae8700 1 --2- 172.21.15.70:0/954573341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dce860 0x55c930dd7bc0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fcde401d4e0 tx=0x7fcde4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7524415e69f00138 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:13.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdd37fe700 1 -- 172.21.15.70:0/954573341 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde4016070 con 0x55c930dce860 2024-06-04T05:10:13.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdd37fe700 1 -- 172.21.15.70:0/954573341 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcde402a030 con 0x55c930dce860 2024-06-04T05:10:13.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.901+0000 7fcdd37fe700 1 -- 172.21.15.70:0/954573341 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde4005420 con 0x55c930dce860 2024-06-04T05:10:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.902+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/954573341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dce860 msgr2=0x55c930dd7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.902+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/954573341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dce860 0x55c930dd7bc0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fcde401d4e0 tx=0x7fcde4012000 comp rx=0 tx=0).stop 2024-06-04T05:10:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.902+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/954573341 shutdown_connections 2024-06-04T05:10:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.902+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/954573341 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dd2f60 0x55c930dc8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.902+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/954573341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dce860 0x55c930dd7bc0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.902+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/954573341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dcf210 0x55c930dd9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.902+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/954573341 >> 172.21.15.70:0/954573341 conn(0x55c930cfc080 msgr2=0x55c930dd8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:13.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.902+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/954573341 shutdown_connections 2024-06-04T05:10:13.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.902+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/954573341 wait complete. 2024-06-04T05:10:13.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.902+0000 7fcdfa5e1b80 1 -- start start 2024-06-04T05:10:13.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.902+0000 7fcdfa5e1b80 1 Processor -- start 2024-06-04T05:10:13.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcdfa5e1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dce860 0x55c930dd0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcdfa5e1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dcf210 0x55c930dd0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcdfa5e1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dd2f60 0x55c930dd0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcdfa5e1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcde81b13f0 con 0x55c930dce860 2024-06-04T05:10:13.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcdfa5e1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcde81b0cc0 con 0x55c930dcf210 2024-06-04T05:10:13.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcdfa5e1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcde81ade80 con 0x55c930dd2f60 2024-06-04T05:10:13.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcdfa5e1b80 10 client.0 _ll_register_callbacks cb 0x55c930cb8920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:13.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcdedae8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dce860 0x55c930dd0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:13.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcdedae8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dce860 0x55c930dd0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47168/0 (socket says 172.21.15.70:47168) 2024-06-04T05:10:13.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcdedae8700 1 -- 172.21.15.70:0/2281111117 learned_addr learned my addr 172.21.15.70:0/2281111117 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:13.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcded2e7700 1 --2- 172.21.15.70:0/2281111117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dcf210 0x55c930dd0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:13.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcdedae8700 1 -- 172.21.15.70:0/2281111117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dd2f60 msgr2=0x55c930dd0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:13.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcdee2e9700 1 --2- 172.21.15.70:0/2281111117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dd2f60 0x55c930dd0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:13.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcdedae8700 1 --2- 172.21.15.70:0/2281111117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dd2f60 0x55c930dd0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcdedae8700 1 -- 172.21.15.70:0/2281111117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dcf210 msgr2=0x55c930dd0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:13.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcdedae8700 1 --2- 172.21.15.70:0/2281111117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dcf210 0x55c930dd0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcdedae8700 1 -- 172.21.15.70:0/2281111117 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c930e51970 con 0x55c930dce860 2024-06-04T05:10:13.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcded2e7700 1 --2- 172.21.15.70:0/2281111117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dcf210 0x55c930dd0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:10:13.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.903+0000 7fcdee2e9700 1 --2- 172.21.15.70:0/2281111117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dd2f60 0x55c930dd0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:10:13.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.904+0000 7fcdedae8700 1 --2- 172.21.15.70:0/2281111117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dce860 0x55c930dd0550 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fcde4029040 tx=0x7fcde401de20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ebe1fa02d5fa24 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:13.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.904+0000 7fcdd17fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:13.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.904+0000 7fcdd17fa700 1 -- 172.21.15.70:0/2281111117 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde403d060 con 0x55c930dce860 2024-06-04T05:10:13.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.904+0000 7fcdd17fa700 1 -- 172.21.15.70:0/2281111117 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7fcde4005590 con 0x55c930dce860 2024-06-04T05:10:13.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.904+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/2281111117 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcde402a030 con 0x55c930dce860 2024-06-04T05:10:13.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.904+0000 7fcdd17fa700 1 -- 172.21.15.70:0/2281111117 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fcde4004750 con 0x55c930dce860 2024-06-04T05:10:13.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.904+0000 7fcdfa5e1b80 10 client.18993 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:13.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.904+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/2281111117 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55c930e52bd0 con 0x55c930dce860 2024-06-04T05:10:13.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.904+0000 7fcdd17fa700 1 -- 172.21.15.70:0/2281111117 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcde402a030 con 0x55c930dce860 2024-06-04T05:10:13.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.904+0000 7fcdd17fa700 1 -- 172.21.15.70:0/2281111117 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7fcde40268f0 con 0x55c930dce860 2024-06-04T05:10:13.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.904+0000 7fcdfa5e1b80 10 client.18993 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:13.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.904+0000 7fcdfa5e1b80 10 client.18993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:10:13.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.904+0000 7fcdfa5e1b80 20 client.18993 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:13.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.904+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/2281111117 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c930de7e00 0x55c930e55290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:13.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.904+0000 7fcdfa5e1b80 4 client.18993 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:10:13.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.904+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/2281111117 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c930d2fbe0 con 0x55c930de7e00 2024-06-04T05:10:13.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.904+0000 7fcded2e7700 1 --2- 172.21.15.70:0/2281111117 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c930de7e00 0x55c930e55290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:13.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.905+0000 7fcdd17fa700 10 client.18993 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:13.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.905+0000 7fcded2e7700 1 --2- 172.21.15.70:0/2281111117 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c930de7e00 0x55c930e55290 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ef0668415f3e19a server_cookie=243b98d6f870db9d in_seq=0 out_seq=0 2024-06-04T05:10:13.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.905+0000 7fcdd17fa700 1 -- 172.21.15.70:0/2281111117 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c930d2fbe0 con 0x55c930de7e00 2024-06-04T05:10:13.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.905+0000 7fcdd17fa700 10 client.18993 handle_command_reply: tid=0 2024-06-04T05:10:13.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.905+0000 7fcdfa5e1b80 1 client.18993 shutdown 2024-06-04T05:10:13.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.905+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/2281111117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dce860 msgr2=0x55c930dd0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:13.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.905+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/2281111117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dce860 0x55c930dd0550 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fcde4029040 tx=0x7fcde401de20 comp rx=0 tx=0).stop 2024-06-04T05:10:13.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.905+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/2281111117 shutdown_connections 2024-06-04T05:10:13.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.905+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/2281111117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c930dd2f60 0x55c930dd0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.905+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/2281111117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c930dce860 0x55c930dd0550 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.905+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/2281111117 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c930de7e00 0x55c930e55290 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.905+0000 7fcdfa5e1b80 1 --2- 172.21.15.70:0/2281111117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c930dcf210 0x55c930dd0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.905+0000 7fcdd17fa700 0 client.18993 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:13.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.905+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/2281111117 >> 172.21.15.70:0/2281111117 conn(0x55c930cfc080 msgr2=0x55c930dd2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:13.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.906+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/2281111117 shutdown_connections 2024-06-04T05:10:13.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.906+0000 7fcdfa5e1b80 1 -- 172.21.15.70:0/2281111117 wait complete. 2024-06-04T05:10:13.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.906+0000 7fcdfa5e1b80 20 client.18993 trim_cache size 0 max 16384 2024-06-04T05:10:13.923 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:10:13.924 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:10:13.924 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:10:13.924 INFO:teuthology.orchestra.run.smithi070.stdout: "09c3fd92-f5fa-407c-8cca-18ae539c4e8f": { 2024-06-04T05:10:13.924 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:10:13.924 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "09c3fd92-f5fa-407c-8cca-18ae539c4e8f", 2024-06-04T05:10:13.924 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:10:13.924 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:10:13.924 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:10:13.924 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:10:13.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.906+0000 7fcdefd4c700 1 -- 172.21.15.70:0/544463218 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcdd40b29f0 msgr2=0x7fcdd40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:13.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.906+0000 7fcdefd4c700 1 --2- 172.21.15.70:0/544463218 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcdd40b29f0 0x7fcdd40b4eb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcde4005950 tx=0x7fcde400a300 comp rx=0 tx=0).stop 2024-06-04T05:10:13.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.906+0000 7fcdefd4c700 1 -- 172.21.15.70:0/544463218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcde8102e00 msgr2=0x7fcde819ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:13.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.906+0000 7fcdefd4c700 1 --2- 172.21.15.70:0/544463218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcde8102e00 0x7fcde819ae80 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fcdd800d980 tx=0x7fcdd800dc90 comp rx=0 tx=0).stop 2024-06-04T05:10:13.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.906+0000 7fcdefd4c700 1 -- 172.21.15.70:0/544463218 shutdown_connections 2024-06-04T05:10:13.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.906+0000 7fcdefd4c700 1 --2- 172.21.15.70:0/544463218 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcdd40b29f0 0x7fcdd40b4eb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.906+0000 7fcdefd4c700 1 --2- 172.21.15.70:0/544463218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcde8108520 0x7fcde81a59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.906+0000 7fcdefd4c700 1 --2- 172.21.15.70:0/544463218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcde8102440 0x7fcde819a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.906+0000 7fcdefd4c700 1 --2- 172.21.15.70:0/544463218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcde8102e00 0x7fcde819ae80 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:13.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.906+0000 7fcdefd4c700 1 -- 172.21.15.70:0/544463218 >> 172.21.15.70:0/544463218 conn(0x7fcde80793f0 msgr2=0x7fcde8100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:13.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.906+0000 7fcdefd4c700 1 -- 172.21.15.70:0/544463218 shutdown_connections 2024-06-04T05:10:13.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:13.906+0000 7fcdefd4c700 1 -- 172.21.15.70:0/544463218 wait complete. 2024-06-04T05:10:14.591 INFO:tasks.cephfs.filesystem:scrub status for tag:09c3fd92-f5fa-407c-8cca-18ae539c4e8f - {'path': '/', 'tag': '09c3fd92-f5fa-407c-8cca-18ae539c4e8f', 'options': 'recursive,force'} 2024-06-04T05:10:14.741 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.62 MB/sec execute 547 sec latency 6751.050 ms 2024-06-04T05:10:15.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:15 smithi098 ceph-mon[35569]: pgmap v407: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 797 KiB/s wr, 95 op/s 2024-06-04T05:10:15.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:15 smithi070 ceph-mon[32631]: pgmap v407: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 797 KiB/s wr, 95 op/s 2024-06-04T05:10:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:15 smithi112 ceph-mon[36453]: pgmap v407: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 797 KiB/s wr, 95 op/s 2024-06-04T05:10:15.741 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.62 MB/sec execute 548 sec latency 7751.133 ms 2024-06-04T05:10:16.742 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.61 MB/sec execute 549 sec latency 8751.203 ms 2024-06-04T05:10:17.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:17 smithi098 ceph-mon[35569]: pgmap v408: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 97 KiB/s wr, 42 op/s 2024-06-04T05:10:17.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:17 smithi070 ceph-mon[32631]: pgmap v408: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 97 KiB/s wr, 42 op/s 2024-06-04T05:10:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:17 smithi112 ceph-mon[36453]: pgmap v408: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 97 KiB/s wr, 42 op/s 2024-06-04T05:10:17.741 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.61 MB/sec execute 550 sec latency 9751.324 ms 2024-06-04T05:10:18.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:18 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:10:18.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:18 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:10:18.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:18 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:10:18.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:18 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:10:18.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:18 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:10:18.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:18 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:10:18.741 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.61 MB/sec execute 551 sec latency 10751.371 ms 2024-06-04T05:10:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:19 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:10:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:19 smithi098 ceph-mon[35569]: pgmap v409: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 95 KiB/s wr, 41 op/s 2024-06-04T05:10:19.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:19 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:10:19.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:19 smithi070 ceph-mon[32631]: pgmap v409: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 95 KiB/s wr, 41 op/s 2024-06-04T05:10:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:19 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:10:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:19 smithi112 ceph-mon[36453]: pgmap v409: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 95 KiB/s wr, 41 op/s 2024-06-04T05:10:19.742 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.60 MB/sec execute 552 sec latency 11751.475 ms 2024-06-04T05:10:20.742 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.60 MB/sec execute 553 sec latency 12751.524 ms 2024-06-04T05:10:21.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:21 smithi098 ceph-mon[35569]: pgmap v410: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 81 KiB/s wr, 35 op/s 2024-06-04T05:10:21.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:21 smithi070 ceph-mon[32631]: pgmap v410: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 81 KiB/s wr, 35 op/s 2024-06-04T05:10:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:21 smithi112 ceph-mon[36453]: pgmap v410: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 81 KiB/s wr, 35 op/s 2024-06-04T05:10:21.742 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.60 MB/sec execute 554 sec latency 13751.622 ms 2024-06-04T05:10:22.742 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.60 MB/sec execute 555 sec latency 14751.698 ms 2024-06-04T05:10:22.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.974+0000 7f9d95a72640 1 -- 172.21.15.70:0/133085076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d90101b70 msgr2=0x7f9d90107700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:22.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.974+0000 7f9d95a72640 1 --2- 172.21.15.70:0/133085076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d90101b70 0x7f9d90107700 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009900 tx=0x7f9d8402a410 comp rx=0 tx=0).stop 2024-06-04T05:10:22.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.974+0000 7f9d95a72640 1 -- 172.21.15.70:0/133085076 shutdown_connections 2024-06-04T05:10:22.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.974+0000 7f9d95a72640 1 --2- 172.21.15.70:0/133085076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d90107e30 0x7f9d9010f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:22.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.974+0000 7f9d95a72640 1 --2- 172.21.15.70:0/133085076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d90101b70 0x7f9d90107700 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:22.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.974+0000 7f9d95a72640 1 --2- 172.21.15.70:0/133085076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d901011c0 0x7f9d901015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:22.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.974+0000 7f9d95a72640 1 -- 172.21.15.70:0/133085076 >> 172.21.15.70:0/133085076 conn(0x7f9d90077eb0 msgr2=0x7f9d900ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:22.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.974+0000 7f9d95a72640 1 -- 172.21.15.70:0/133085076 shutdown_connections 2024-06-04T05:10:22.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.974+0000 7f9d95a72640 1 -- 172.21.15.70:0/133085076 wait complete. 2024-06-04T05:10:22.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.975+0000 7f9d95a72640 1 Processor -- start 2024-06-04T05:10:22.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.975+0000 7f9d95a72640 1 -- start start 2024-06-04T05:10:22.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.975+0000 7f9d95a72640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d901011c0 0x7f9d90199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:22.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.975+0000 7f9d95a72640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d90107e30 0x7f9d901996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:22.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.975+0000 7f9d95a72640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d9019da50 0x7f9d9019df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:22.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.975+0000 7f9d95a72640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d90112c10 con 0x7f9d901011c0 2024-06-04T05:10:22.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.975+0000 7f9d95a72640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9d90112a90 con 0x7f9d90107e30 2024-06-04T05:10:22.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.975+0000 7f9d95a72640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9d90112d90 con 0x7f9d9019da50 2024-06-04T05:10:22.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.975+0000 7f9d8f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d901011c0 0x7f9d90199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:22.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.976+0000 7f9d8f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d901011c0 0x7f9d90199190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35658/0 (socket says 172.21.15.70:35658) 2024-06-04T05:10:22.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.976+0000 7f9d8f7fe640 1 -- 172.21.15.70:0/2047918173 learned_addr learned my addr 172.21.15.70:0/2047918173 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:22.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.976+0000 7f9d8ffff640 1 --2- 172.21.15.70:0/2047918173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d9019da50 0x7f9d9019df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:22.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.976+0000 7f9d8effd640 1 --2- 172.21.15.70:0/2047918173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d90107e30 0x7f9d901996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:22.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.976+0000 7f9d8f7fe640 1 -- 172.21.15.70:0/2047918173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d9019da50 msgr2=0x7f9d9019df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:22.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.976+0000 7f9d8f7fe640 1 --2- 172.21.15.70:0/2047918173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d9019da50 0x7f9d9019df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:22.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.976+0000 7f9d8f7fe640 1 -- 172.21.15.70:0/2047918173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d90107e30 msgr2=0x7f9d901996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:22.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.976+0000 7f9d8f7fe640 1 --2- 172.21.15.70:0/2047918173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d90107e30 0x7f9d901996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:22.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.976+0000 7f9d8f7fe640 1 -- 172.21.15.70:0/2047918173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d9019e540 con 0x7f9d901011c0 2024-06-04T05:10:22.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.976+0000 7f9d8effd640 1 --2- 172.21.15.70:0/2047918173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d90107e30 0x7f9d901996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:10:22.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.976+0000 7f9d8f7fe640 1 --2- 172.21.15.70:0/2047918173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d901011c0 0x7f9d90199190 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000b4a0 tx=0x7f9d8000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:22.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.976+0000 7f9d8cff9640 1 -- 172.21.15.70:0/2047918173 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d80014070 con 0x7f9d901011c0 2024-06-04T05:10:22.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.976+0000 7f9d8cff9640 1 -- 172.21.15.70:0/2047918173 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d80010aa0 con 0x7f9d901011c0 2024-06-04T05:10:22.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.976+0000 7f9d95a72640 1 -- 172.21.15.70:0/2047918173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d9019e830 con 0x7f9d901011c0 2024-06-04T05:10:22.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.976+0000 7f9d8cff9640 1 -- 172.21.15.70:0/2047918173 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d80017b70 con 0x7f9d901011c0 2024-06-04T05:10:22.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.976+0000 7f9d95a72640 1 -- 172.21.15.70:0/2047918173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d901a6070 con 0x7f9d901011c0 2024-06-04T05:10:22.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.979+0000 7f9d8cff9640 1 -- 172.21.15.70:0/2047918173 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f9d80017070 con 0x7f9d901011c0 2024-06-04T05:10:22.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.979+0000 7f9d98792740 1 Processor -- start 2024-06-04T05:10:22.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.979+0000 7f9d98792740 1 -- start start 2024-06-04T05:10:22.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.979+0000 7f9d98792740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ec820 0x55b3693ecc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:22.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.979+0000 7f9d98792740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ed140 0x55b3693f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:22.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.979+0000 7f9d98792740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693f7a00 0x55b3693f9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:22.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.979+0000 7f9d98792740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b36935ec30 con 0x55b3693ed140 2024-06-04T05:10:22.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.979+0000 7f9d98792740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b36934e7d0 con 0x55b3693ec820 2024-06-04T05:10:22.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.979+0000 7f9d98792740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b369352810 con 0x55b3693f7a00 2024-06-04T05:10:22.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.979+0000 7f9d8f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ed140 0x55b3693f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:22.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.979+0000 7f9d8f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ed140 0x55b3693f74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35670/0 (socket says 172.21.15.70:35670) 2024-06-04T05:10:22.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.979+0000 7f9d8f7fe640 1 -- 172.21.15.70:0/4121167630 learned_addr learned my addr 172.21.15.70:0/4121167630 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:22.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.980+0000 7f9d8effd640 1 --2- 172.21.15.70:0/4121167630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693f7a00 0x55b3693f9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:22.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.980+0000 7f9d8ffff640 1 --2- 172.21.15.70:0/4121167630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ec820 0x55b3693ecc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:22.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.980+0000 7f9d8cff9640 1 --2- 172.21.15.70:0/2047918173 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9d780b2a90 0x7f9d780b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:22.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.980+0000 7f9d8f7fe640 1 -- 172.21.15.70:0/4121167630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693f7a00 msgr2=0x55b3693f9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:22.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.980+0000 7f9d8f7fe640 1 --2- 172.21.15.70:0/4121167630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693f7a00 0x55b3693f9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:22.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.980+0000 7f9d8f7fe640 1 -- 172.21.15.70:0/4121167630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ec820 msgr2=0x55b3693ecc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:22.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.980+0000 7f9d8f7fe640 1 --2- 172.21.15.70:0/4121167630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ec820 0x55b3693ecc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:22.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.980+0000 7f9d8f7fe640 1 -- 172.21.15.70:0/4121167630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b369063210 con 0x55b3693ed140 2024-06-04T05:10:22.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.980+0000 7f9d8effd640 1 --2- 172.21.15.70:0/4121167630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693f7a00 0x55b3693f9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:10:22.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.980+0000 7f9d8ffff640 1 --2- 172.21.15.70:0/2047918173 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9d780b2a90 0x7f9d780b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:22.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.980+0000 7f9d8f7fe640 1 --2- 172.21.15.70:0/4121167630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ed140 0x55b3693f74c0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f9d800df3d0 tx=0x7f9d80072890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b060d4308bd593 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:22.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.980+0000 7f9d8cff9640 1 -- 172.21.15.70:0/2047918173 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f9d800e1c30 con 0x7f9d901011c0 2024-06-04T05:10:22.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.980+0000 7f9d75ffb640 1 -- 172.21.15.70:0/4121167630 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d800a8760 con 0x55b3693ed140 2024-06-04T05:10:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.980+0000 7f9d75ffb640 1 -- 172.21.15.70:0/4121167630 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d80017360 con 0x55b3693ed140 2024-06-04T05:10:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.980+0000 7f9d75ffb640 1 -- 172.21.15.70:0/4121167630 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d800e5490 con 0x55b3693ed140 2024-06-04T05:10:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.980+0000 7f9d8ffff640 1 --2- 172.21.15.70:0/2047918173 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9d780b2a90 0x7f9d780b4f50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9d88005fd0 tx=0x7f9d88004a40 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.981+0000 7f9d98792740 1 -- 172.21.15.70:0/4121167630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ed140 msgr2=0x55b3693f74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.981+0000 7f9d98792740 1 --2- 172.21.15.70:0/4121167630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ed140 0x55b3693f74c0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f9d800df3d0 tx=0x7f9d80072890 comp rx=0 tx=0).stop 2024-06-04T05:10:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.981+0000 7f9d98792740 1 -- 172.21.15.70:0/4121167630 shutdown_connections 2024-06-04T05:10:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.981+0000 7f9d98792740 1 --2- 172.21.15.70:0/4121167630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693f7a00 0x55b3693f9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.981+0000 7f9d98792740 1 --2- 172.21.15.70:0/4121167630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ed140 0x55b3693f74c0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.981+0000 7f9d98792740 1 --2- 172.21.15.70:0/4121167630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ec820 0x55b3693ecc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.981+0000 7f9d98792740 1 -- 172.21.15.70:0/4121167630 >> 172.21.15.70:0/4121167630 conn(0x55b3693e8d30 msgr2=0x55b3693e9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.981+0000 7f9d98792740 1 -- 172.21.15.70:0/4121167630 shutdown_connections 2024-06-04T05:10:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.981+0000 7f9d98792740 1 -- 172.21.15.70:0/4121167630 wait complete. 2024-06-04T05:10:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.981+0000 7f9d98792740 1 -- start start 2024-06-04T05:10:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d98792740 1 Processor -- start 2024-06-04T05:10:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d98792740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ec820 0x55b369401880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d98792740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693ed140 0x55b369401dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d98792740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693f7a00 0x55b3693fbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d98792740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b369034790 con 0x55b3693f7a00 2024-06-04T05:10:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d98792740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b36930a890 con 0x55b3693ec820 2024-06-04T05:10:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d98792740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b3693fa5d0 con 0x55b3693ed140 2024-06-04T05:10:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d98792740 10 client.0 _ll_register_callbacks cb 0x55b369062a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d8ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693f7a00 0x55b3693fbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d8ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693f7a00 0x55b3693fbbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35678/0 (socket says 172.21.15.70:35678) 2024-06-04T05:10:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d8ffff640 1 -- 172.21.15.70:0/335110964 learned_addr learned my addr 172.21.15.70:0/335110964 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d8f7fe640 1 --2- 172.21.15.70:0/335110964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ec820 0x55b369401880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d8effd640 1 --2- 172.21.15.70:0/335110964 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693ed140 0x55b369401dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d8ffff640 1 -- 172.21.15.70:0/335110964 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693ed140 msgr2=0x55b369401dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d8ffff640 1 --2- 172.21.15.70:0/335110964 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693ed140 0x55b369401dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d8ffff640 1 -- 172.21.15.70:0/335110964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ec820 msgr2=0x55b369401880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d8ffff640 1 --2- 172.21.15.70:0/335110964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ec820 0x55b369401880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d8ffff640 1 -- 172.21.15.70:0/335110964 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b36912f620 con 0x55b3693f7a00 2024-06-04T05:10:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d8f7fe640 1 --2- 172.21.15.70:0/335110964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ec820 0x55b369401880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:10:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.982+0000 7f9d8ffff640 1 --2- 172.21.15.70:0/335110964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693f7a00 0x55b3693fbbf0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f9d88013430 tx=0x7f9d88004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24ff2f20c6e1ea24 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.983+0000 7f9d6effd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.983+0000 7f9d6effd640 1 -- 172.21.15.70:0/335110964 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8801d850 con 0x55b3693f7a00 2024-06-04T05:10:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.983+0000 7f9d6effd640 1 -- 172.21.15.70:0/335110964 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f9d8801d9f0 con 0x55b3693f7a00 2024-06-04T05:10:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.983+0000 7f9d98792740 1 -- 172.21.15.70:0/335110964 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b369360b80 con 0x55b3693f7a00 2024-06-04T05:10:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.983+0000 7f9d6effd640 1 -- 172.21.15.70:0/335110964 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f9d8801dbf0 con 0x55b3693f7a00 2024-06-04T05:10:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.983+0000 7f9d98792740 10 client.19005 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.983+0000 7f9d98792740 1 -- 172.21.15.70:0/335110964 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b36940a7e0 con 0x55b3693f7a00 2024-06-04T05:10:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.983+0000 7f9d6effd640 1 -- 172.21.15.70:0/335110964 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55b369360b80 con 0x55b3693f7a00 2024-06-04T05:10:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.983+0000 7f9d6effd640 1 -- 172.21.15.70:0/335110964 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f9d8801a430 con 0x55b3693f7a00 2024-06-04T05:10:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.983+0000 7f9d98792740 10 client.19005 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.983+0000 7f9d98792740 10 client.19005 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:10:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.983+0000 7f9d98792740 20 client.19005 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.983+0000 7f9d98792740 1 --2- 172.21.15.70:0/335110964 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b36940bd40 0x55b36940e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.983+0000 7f9d98792740 4 client.19005 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:10:22.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.983+0000 7f9d98792740 1 -- 172.21.15.70:0/335110964 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b36935a360 con 0x55b36940bd40 2024-06-04T05:10:22.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.984+0000 7f9d8f7fe640 1 --2- 172.21.15.70:0/335110964 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b36940bd40 0x55b36940e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:22.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.984+0000 7f9d8f7fe640 1 --2- 172.21.15.70:0/335110964 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b36940bd40 0x55b36940e170 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9dd4490840bee365 server_cookie=9efe5fd72095696d in_seq=0 out_seq=0 2024-06-04T05:10:22.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.984+0000 7f9d6effd640 10 client.19005 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.985+0000 7f9d6effd640 1 -- 172.21.15.70:0/335110964 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b36935a360 con 0x55b36940bd40 2024-06-04T05:10:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.985+0000 7f9d6effd640 10 client.19005 handle_command_reply: tid=0 2024-06-04T05:10:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.985+0000 7f9d98792740 1 client.19005 shutdown 2024-06-04T05:10:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.985+0000 7f9d98792740 1 -- 172.21.15.70:0/335110964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693f7a00 msgr2=0x55b3693fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.985+0000 7f9d98792740 1 --2- 172.21.15.70:0/335110964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693f7a00 0x55b3693fbbf0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f9d88013430 tx=0x7f9d88004ab0 comp rx=0 tx=0).stop 2024-06-04T05:10:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.985+0000 7f9d98792740 1 -- 172.21.15.70:0/335110964 shutdown_connections 2024-06-04T05:10:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.985+0000 7f9d98792740 1 --2- 172.21.15.70:0/335110964 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b36940bd40 0x55b36940e170 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.985+0000 7f9d98792740 1 --2- 172.21.15.70:0/335110964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693f7a00 0x55b3693fbbf0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.985+0000 7f9d98792740 1 --2- 172.21.15.70:0/335110964 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693ed140 0x55b369401dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.985+0000 7f9d6effd640 0 client.19005 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.985+0000 7f9d98792740 1 --2- 172.21.15.70:0/335110964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ec820 0x55b369401880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.985+0000 7f9d98792740 1 -- 172.21.15.70:0/335110964 >> 172.21.15.70:0/335110964 conn(0x55b3693e8d30 msgr2=0x55b3693e7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.985+0000 7f9d98792740 1 -- 172.21.15.70:0/335110964 shutdown_connections 2024-06-04T05:10:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.985+0000 7f9d98792740 1 -- 172.21.15.70:0/335110964 wait complete. 2024-06-04T05:10:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:22.985+0000 7f9d98792740 20 client.19005 trim_cache size 0 max 16384 2024-06-04T05:10:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.000+0000 7f9d98792740 1 Processor -- start 2024-06-04T05:10:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.000+0000 7f9d98792740 1 -- start start 2024-06-04T05:10:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.000+0000 7f9d98792740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ec820 0x55b369401cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.000+0000 7f9d98792740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ed140 0x55b369413a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.000+0000 7f9d98792740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693f7a00 0x55b369413fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.000+0000 7f9d98792740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d901ac5f0 con 0x55b3693ec820 2024-06-04T05:10:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.000+0000 7f9d98792740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9d901ae250 con 0x55b3693ed140 2024-06-04T05:10:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.000+0000 7f9d98792740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9d901accc0 con 0x55b3693f7a00 2024-06-04T05:10:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.000+0000 7f9d8f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ec820 0x55b369401cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.000+0000 7f9d8f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ec820 0x55b369401cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35694/0 (socket says 172.21.15.70:35694) 2024-06-04T05:10:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.000+0000 7f9d8f7fe640 1 -- 172.21.15.70:0/2413471586 learned_addr learned my addr 172.21.15.70:0/2413471586 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.000+0000 7f9d8f7fe640 1 -- 172.21.15.70:0/2413471586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693f7a00 msgr2=0x55b369413fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.000+0000 7f9d8f7fe640 1 --2- 172.21.15.70:0/2413471586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693f7a00 0x55b369413fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.000+0000 7f9d8f7fe640 1 -- 172.21.15.70:0/2413471586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ed140 msgr2=0x55b369413a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.000+0000 7f9d8f7fe640 1 --2- 172.21.15.70:0/2413471586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ed140 0x55b369413a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.001+0000 7f9d8f7fe640 1 -- 172.21.15.70:0/2413471586 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3693e46f0 con 0x55b3693ec820 2024-06-04T05:10:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.001+0000 7f9d8f7fe640 1 --2- 172.21.15.70:0/2413471586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ec820 0x55b369401cd0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f9d800df950 tx=0x7f9d80017550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce42d0cdd057bb4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.001+0000 7f9d757fa640 1 -- 172.21.15.70:0/2413471586 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8001e040 con 0x55b3693ec820 2024-06-04T05:10:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.001+0000 7f9d757fa640 1 -- 172.21.15.70:0/2413471586 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d800770c0 con 0x55b3693ec820 2024-06-04T05:10:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.001+0000 7f9d757fa640 1 -- 172.21.15.70:0/2413471586 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d80010c50 con 0x55b3693ec820 2024-06-04T05:10:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.001+0000 7f9d98792740 1 -- 172.21.15.70:0/2413471586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ec820 msgr2=0x55b369401cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.001+0000 7f9d98792740 1 --2- 172.21.15.70:0/2413471586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ec820 0x55b369401cd0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f9d800df950 tx=0x7f9d80017550 comp rx=0 tx=0).stop 2024-06-04T05:10:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.001+0000 7f9d98792740 1 -- 172.21.15.70:0/2413471586 shutdown_connections 2024-06-04T05:10:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.001+0000 7f9d98792740 1 --2- 172.21.15.70:0/2413471586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ed140 0x55b369413a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.001+0000 7f9d98792740 1 --2- 172.21.15.70:0/2413471586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693f7a00 0x55b369413fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.001+0000 7f9d98792740 1 --2- 172.21.15.70:0/2413471586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ec820 0x55b369401cd0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.001+0000 7f9d98792740 1 -- 172.21.15.70:0/2413471586 >> 172.21.15.70:0/2413471586 conn(0x55b3693e8d30 msgr2=0x55b3693fbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.001+0000 7f9d98792740 1 -- 172.21.15.70:0/2413471586 shutdown_connections 2024-06-04T05:10:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.001+0000 7f9d98792740 1 -- 172.21.15.70:0/2413471586 wait complete. 2024-06-04T05:10:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.002+0000 7f9d98792740 1 -- start start 2024-06-04T05:10:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.002+0000 7f9d98792740 1 Processor -- start 2024-06-04T05:10:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.002+0000 7f9d98792740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ec820 0x55b369408270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.002+0000 7f9d98792740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ed140 0x55b369405650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.002+0000 7f9d98792740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693f7a00 0x55b369405b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.002+0000 7f9d98792740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d901adb20 con 0x55b3693ed140 2024-06-04T05:10:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.002+0000 7f9d98792740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9d901ad3f0 con 0x55b3693ec820 2024-06-04T05:10:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.002+0000 7f9d98792740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9d901aa5f0 con 0x55b3693f7a00 2024-06-04T05:10:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.002+0000 7f9d98792740 10 client.0 _ll_register_callbacks cb 0x7f9d88003ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.002+0000 7f9d8effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ed140 0x55b369405650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.002+0000 7f9d8effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ed140 0x55b369405650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35708/0 (socket says 172.21.15.70:35708) 2024-06-04T05:10:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.002+0000 7f9d8effd640 1 -- 172.21.15.70:0/1044380533 learned_addr learned my addr 172.21.15.70:0/1044380533 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d8ffff640 1 --2- 172.21.15.70:0/1044380533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693f7a00 0x55b369405b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d8effd640 1 -- 172.21.15.70:0/1044380533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693f7a00 msgr2=0x55b369405b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d8f7fe640 1 --2- 172.21.15.70:0/1044380533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ec820 0x55b369408270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d8effd640 1 --2- 172.21.15.70:0/1044380533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693f7a00 0x55b369405b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d8effd640 1 -- 172.21.15.70:0/1044380533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ec820 msgr2=0x55b369408270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d8effd640 1 --2- 172.21.15.70:0/1044380533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ec820 0x55b369408270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d8effd640 1 -- 172.21.15.70:0/1044380533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b369406280 con 0x55b3693ed140 2024-06-04T05:10:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d8ffff640 1 --2- 172.21.15.70:0/1044380533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693f7a00 0x55b369405b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:10:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d8effd640 1 --2- 172.21.15.70:0/1044380533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ed140 0x55b369405650 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f9d84005ab0 tx=0x7f9d84031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bea9967fe51e488 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d6effd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d6effd640 1 -- 172.21.15.70:0/1044380533 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d84035070 con 0x55b3693ed140 2024-06-04T05:10:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d6effd640 1 -- 172.21.15.70:0/1044380533 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f9d84004040 con 0x55b3693ed140 2024-06-04T05:10:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d98792740 1 -- 172.21.15.70:0/1044380533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d800770c0 con 0x55b3693ed140 2024-06-04T05:10:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d6effd640 1 -- 172.21.15.70:0/1044380533 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f9d84004240 con 0x55b3693ed140 2024-06-04T05:10:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d98792740 10 client.19011 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d98792740 1 -- 172.21.15.70:0/1044380533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b3694768a0 con 0x55b3693ed140 2024-06-04T05:10:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d6effd640 1 -- 172.21.15.70:0/1044380533 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d800770c0 con 0x55b3693ed140 2024-06-04T05:10:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d6effd640 1 -- 172.21.15.70:0/1044380533 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f9d84004930 con 0x55b3693ed140 2024-06-04T05:10:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d98792740 10 client.19011 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d98792740 10 client.19011 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:10:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d98792740 20 client.19011 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d98792740 1 --2- 172.21.15.70:0/1044380533 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b369477100 0x55b369479520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.003+0000 7f9d98792740 4 client.19011 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:10:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.004+0000 7f9d98792740 1 -- 172.21.15.70:0/1044380533 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b3692d45f0 con 0x55b369477100 2024-06-04T05:10:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.004+0000 7f9d8f7fe640 1 --2- 172.21.15.70:0/1044380533 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b369477100 0x55b369479520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.004+0000 7f9d6effd640 10 client.19011 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.004+0000 7f9d8f7fe640 1 --2- 172.21.15.70:0/1044380533 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b369477100 0x55b369479520 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9192ce7db41c49a4 server_cookie=ed521e6d9f5299b8 in_seq=0 out_seq=0 2024-06-04T05:10:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.023+0000 7f9d6effd640 1 -- 172.21.15.70:0/1044380533 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55b3692d45f0 con 0x55b369477100 2024-06-04T05:10:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.023+0000 7f9d6effd640 10 client.19011 handle_command_reply: tid=0 2024-06-04T05:10:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.023+0000 7f9d98792740 1 client.19011 shutdown 2024-06-04T05:10:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d98792740 1 -- 172.21.15.70:0/1044380533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ed140 msgr2=0x55b369405650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d98792740 1 --2- 172.21.15.70:0/1044380533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ed140 0x55b369405650 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f9d84005ab0 tx=0x7f9d84031630 comp rx=0 tx=0).stop 2024-06-04T05:10:23.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d98792740 1 -- 172.21.15.70:0/1044380533 shutdown_connections 2024-06-04T05:10:23.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d98792740 1 --2- 172.21.15.70:0/1044380533 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b369477100 0x55b369479520 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:23.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d98792740 1 --2- 172.21.15.70:0/1044380533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3693f7a00 0x55b369405b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:23.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d98792740 1 --2- 172.21.15.70:0/1044380533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3693ed140 0x55b369405650 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:23.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d6effd640 0 client.19011 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:23.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d98792740 1 --2- 172.21.15.70:0/1044380533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3693ec820 0x55b369408270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:23.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d98792740 1 -- 172.21.15.70:0/1044380533 >> 172.21.15.70:0/1044380533 conn(0x55b3693e8d30 msgr2=0x55b3693ee1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:23.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d98792740 1 -- 172.21.15.70:0/1044380533 shutdown_connections 2024-06-04T05:10:23.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d98792740 1 -- 172.21.15.70:0/1044380533 wait complete. 2024-06-04T05:10:23.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d98792740 20 client.19011 trim_cache size 0 max 16384 2024-06-04T05:10:23.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:10:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:10:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:10:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d95a72640 1 -- 172.21.15.70:0/2047918173 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9d780b2a90 msgr2=0x7f9d780b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d95a72640 1 --2- 172.21.15.70:0/2047918173 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9d780b2a90 0x7f9d780b4f50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9d88005fd0 tx=0x7f9d88004a40 comp rx=0 tx=0).stop 2024-06-04T05:10:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d95a72640 1 -- 172.21.15.70:0/2047918173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d901011c0 msgr2=0x7f9d90199190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d95a72640 1 --2- 172.21.15.70:0/2047918173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d901011c0 0x7f9d90199190 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000b4a0 tx=0x7f9d8000b960 comp rx=0 tx=0).stop 2024-06-04T05:10:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d95a72640 1 -- 172.21.15.70:0/2047918173 shutdown_connections 2024-06-04T05:10:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d95a72640 1 --2- 172.21.15.70:0/2047918173 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9d780b2a90 0x7f9d780b4f50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.024+0000 7f9d95a72640 1 --2- 172.21.15.70:0/2047918173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d90107e30 0x7f9d901996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.025+0000 7f9d95a72640 1 --2- 172.21.15.70:0/2047918173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d9019da50 0x7f9d9019df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.025+0000 7f9d95a72640 1 --2- 172.21.15.70:0/2047918173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d901011c0 0x7f9d90199190 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.025+0000 7f9d95a72640 1 -- 172.21.15.70:0/2047918173 >> 172.21.15.70:0/2047918173 conn(0x7f9d90077eb0 msgr2=0x7f9d90106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.025+0000 7f9d95a72640 1 -- 172.21.15.70:0/2047918173 shutdown_connections 2024-06-04T05:10:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:23.025+0000 7f9d95a72640 1 -- 172.21.15.70:0/2047918173 wait complete. 2024-06-04T05:10:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:23 smithi098 ceph-mon[35569]: pgmap v411: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 86 KiB/s wr, 39 op/s 2024-06-04T05:10:23.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:23 smithi070 ceph-mon[32631]: pgmap v411: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 86 KiB/s wr, 39 op/s 2024-06-04T05:10:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:23 smithi112 ceph-mon[36453]: pgmap v411: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 86 KiB/s wr, 39 op/s 2024-06-04T05:10:23.742 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.59 MB/sec execute 556 sec latency 15751.798 ms 2024-06-04T05:10:24.742 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.59 MB/sec execute 557 sec latency 16751.892 ms 2024-06-04T05:10:25.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:25 smithi098 ceph-mon[35569]: pgmap v412: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-06-04T05:10:25.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:25 smithi070 ceph-mon[32631]: pgmap v412: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-06-04T05:10:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:25 smithi112 ceph-mon[36453]: pgmap v412: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-06-04T05:10:25.742 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.59 MB/sec execute 558 sec latency 17751.979 ms 2024-06-04T05:10:26.742 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.58 MB/sec execute 559 sec latency 18752.111 ms 2024-06-04T05:10:27.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:27 smithi098 ceph-mon[35569]: pgmap v413: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 17 op/s 2024-06-04T05:10:27.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:27 smithi070 ceph-mon[32631]: pgmap v413: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 17 op/s 2024-06-04T05:10:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:27 smithi112 ceph-mon[36453]: pgmap v413: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 17 op/s 2024-06-04T05:10:27.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.58 MB/sec execute 560 sec latency 19752.226 ms 2024-06-04T05:10:28.203 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:10:28.456 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:10:28.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.58 MB/sec execute 561 sec latency 20752.340 ms 2024-06-04T05:10:29.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:29 smithi098 ceph-mon[35569]: pgmap v414: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 24 KiB/s wr, 12 op/s 2024-06-04T05:10:29.602 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:29 smithi070 ceph-mon[32631]: pgmap v414: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 24 KiB/s wr, 12 op/s 2024-06-04T05:10:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:29 smithi112 ceph-mon[36453]: pgmap v414: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 24 KiB/s wr, 12 op/s 2024-06-04T05:10:29.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.58 MB/sec execute 562 sec latency 21752.382 ms 2024-06-04T05:10:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.173+0000 7ff64afa0700 1 -- 172.21.15.70:0/1658545075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff644102df0 msgr2=0x7ff64410f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.173+0000 7ff64afa0700 1 --2- 172.21.15.70:0/1658545075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff644102df0 0x7ff64410f2e0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00b0a0 tx=0x7ff63c00b3b0 comp rx=0 tx=0).stop 2024-06-04T05:10:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.173+0000 7ff64afa0700 1 -- 172.21.15.70:0/1658545075 shutdown_connections 2024-06-04T05:10:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.173+0000 7ff64afa0700 1 --2- 172.21.15.70:0/1658545075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff644102df0 0x7ff64410f2e0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.173+0000 7ff64afa0700 1 --2- 172.21.15.70:0/1658545075 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff644108510 0x7ff6441088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.173+0000 7ff64afa0700 1 --2- 172.21.15.70:0/1658545075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff644102430 0x7ff6441028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.173+0000 7ff64afa0700 1 -- 172.21.15.70:0/1658545075 >> 172.21.15.70:0/1658545075 conn(0x7ff6440793b0 msgr2=0x7ff6440797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.173+0000 7ff64afa0700 1 -- 172.21.15.70:0/1658545075 shutdown_connections 2024-06-04T05:10:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.173+0000 7ff64afa0700 1 -- 172.21.15.70:0/1658545075 wait complete. 2024-06-04T05:10:30.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff64afa0700 1 Processor -- start 2024-06-04T05:10:30.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff64afa0700 1 -- start start 2024-06-04T05:10:30.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff64afa0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff644102430 0x7ff64419a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:30.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff64afa0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff644102df0 0x7ff64419ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:30.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff64afa0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff644108510 0x7ff6441a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:30.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff64afa0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff644112470 con 0x7ff644102df0 2024-06-04T05:10:30.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff64afa0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff6441122f0 con 0x7ff644108510 2024-06-04T05:10:30.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff64afa0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff6441125f0 con 0x7ff644102430 2024-06-04T05:10:30.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff643fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff644102df0 0x7ff64419ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:30.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff643fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff644102df0 0x7ff64419ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59608/0 (socket says 172.21.15.70:59608) 2024-06-04T05:10:30.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff643fff700 1 -- 172.21.15.70:0/2704003834 learned_addr learned my addr 172.21.15.70:0/2704003834 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:30.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff648d3c700 1 --2- 172.21.15.70:0/2704003834 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff644102430 0x7ff64419a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:30.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff64953d700 1 --2- 172.21.15.70:0/2704003834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff644108510 0x7ff6441a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:30.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff643fff700 1 -- 172.21.15.70:0/2704003834 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff644102430 msgr2=0x7ff64419a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:30.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff643fff700 1 --2- 172.21.15.70:0/2704003834 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff644102430 0x7ff64419a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:30.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff643fff700 1 -- 172.21.15.70:0/2704003834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff644108510 msgr2=0x7ff6441a5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:30.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff643fff700 1 --2- 172.21.15.70:0/2704003834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff644108510 0x7ff6441a5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:30.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff643fff700 1 -- 172.21.15.70:0/2704003834 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6441a5ed0 con 0x7ff644102df0 2024-06-04T05:10:30.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.174+0000 7ff64953d700 1 --2- 172.21.15.70:0/2704003834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff644108510 0x7ff6441a5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:10:30.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.175+0000 7ff643fff700 1 --2- 172.21.15.70:0/2704003834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff644102df0 0x7ff64419ae50 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7ff63000d950 tx=0x7ff63000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:30.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.175+0000 7ff641ffb700 1 -- 172.21.15.70:0/2704003834 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff630016070 con 0x7ff644102df0 2024-06-04T05:10:30.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.175+0000 7ff641ffb700 1 -- 172.21.15.70:0/2704003834 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff630010460 con 0x7ff644102df0 2024-06-04T05:10:30.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.175+0000 7ff641ffb700 1 -- 172.21.15.70:0/2704003834 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63000f620 con 0x7ff644102df0 2024-06-04T05:10:30.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.175+0000 7ff64afa0700 1 -- 172.21.15.70:0/2704003834 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6441a60a0 con 0x7ff644102df0 2024-06-04T05:10:30.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.175+0000 7ff64afa0700 1 -- 172.21.15.70:0/2704003834 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6441a6600 con 0x7ff644102df0 2024-06-04T05:10:30.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.176+0000 7ff64afa0700 1 -- 172.21.15.70:0/2704003834 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff64406a230 con 0x7ff644102df0 2024-06-04T05:10:30.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.177+0000 7ff641ffb700 1 -- 172.21.15.70:0/2704003834 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff630019070 con 0x7ff644102df0 2024-06-04T05:10:30.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.178+0000 7ff641ffb700 1 --2- 172.21.15.70:0/2704003834 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff6340b2a70 0x7ff6340b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:30.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.178+0000 7ff648d3c700 1 --2- 172.21.15.70:0/2704003834 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff6340b2a70 0x7ff6340b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:30.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.178+0000 7ff641ffb700 1 -- 172.21.15.70:0/2704003834 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7ff6300e2140 con 0x7ff644102df0 2024-06-04T05:10:30.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.178+0000 7ff648d3c700 1 --2- 172.21.15.70:0/2704003834 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff6340b2a70 0x7ff6340b4f30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff63800dad0 tx=0x7ff63800d4b0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:30.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.183+0000 7ff641ffb700 1 -- 172.21.15.70:0/2704003834 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff6300a9b90 con 0x7ff644102df0 2024-06-04T05:10:30.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.446+0000 7ff64afa0700 1 -- 172.21.15.70:0/2704003834 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff64402e720 con 0x7ff644102df0 2024-06-04T05:10:30.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.447+0000 7ff641ffb700 1 -- 172.21.15.70:0/2704003834 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11938 (secure 0 0 0) 0x7ff6300a9490 con 0x7ff644102df0 2024-06-04T05:10:30.449 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:10:30.449 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:09:17.032462+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24436,24410,24440,24448,24442]},"id":1}]} 2024-06-04T05:10:30.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.449+0000 7ff64afa0700 1 -- 172.21.15.70:0/2704003834 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff6340b2a70 msgr2=0x7ff6340b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:30.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.449+0000 7ff64afa0700 1 --2- 172.21.15.70:0/2704003834 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff6340b2a70 0x7ff6340b4f30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff63800dad0 tx=0x7ff63800d4b0 comp rx=0 tx=0).stop 2024-06-04T05:10:30.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.449+0000 7ff64afa0700 1 -- 172.21.15.70:0/2704003834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff644102df0 msgr2=0x7ff64419ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:30.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.449+0000 7ff64afa0700 1 --2- 172.21.15.70:0/2704003834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff644102df0 0x7ff64419ae50 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7ff63000d950 tx=0x7ff63000dd10 comp rx=0 tx=0).stop 2024-06-04T05:10:30.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.449+0000 7ff64afa0700 1 -- 172.21.15.70:0/2704003834 shutdown_connections 2024-06-04T05:10:30.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.449+0000 7ff64afa0700 1 --2- 172.21.15.70:0/2704003834 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff6340b2a70 0x7ff6340b4f30 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:30.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.449+0000 7ff64afa0700 1 --2- 172.21.15.70:0/2704003834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff644108510 0x7ff6441a5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:30.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.449+0000 7ff64afa0700 1 --2- 172.21.15.70:0/2704003834 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff644102430 0x7ff64419a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:30.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.449+0000 7ff64afa0700 1 --2- 172.21.15.70:0/2704003834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff644102df0 0x7ff64419ae50 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:30.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.449+0000 7ff64afa0700 1 -- 172.21.15.70:0/2704003834 >> 172.21.15.70:0/2704003834 conn(0x7ff6440793b0 msgr2=0x7ff644100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:30.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.449+0000 7ff64afa0700 1 -- 172.21.15.70:0/2704003834 shutdown_connections 2024-06-04T05:10:30.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:30.449+0000 7ff64afa0700 1 -- 172.21.15.70:0/2704003834 wait complete. 2024-06-04T05:10:30.453 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 38 2024-06-04T05:10:30.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.57 MB/sec execute 563 sec latency 22752.411 ms 2024-06-04T05:10:31.249 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id 48e6b43c --release --await' 2024-06-04T05:10:31.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id 48e6b43c --release --await 2024-06-04T05:10:31.488 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:31 smithi112 ceph-mon[36453]: pgmap v415: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 24 KiB/s wr, 12 op/s 2024-06-04T05:10:31.488 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:31 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2704003834' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:10:31.492 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:10:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:31 smithi098 ceph-mon[35569]: pgmap v415: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 24 KiB/s wr, 12 op/s 2024-06-04T05:10:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:31 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2704003834' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:10:31.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:31 smithi070 ceph-mon[32631]: pgmap v415: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 24 KiB/s wr, 12 op/s 2024-06-04T05:10:31.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:31 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2704003834' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:10:31.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.57 MB/sec execute 564 sec latency 23752.497 ms 2024-06-04T05:10:32.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209415 1.57 MB/sec execute 565 sec latency 24752.580 ms 2024-06-04T05:10:33.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.335+0000 7f37e984f700 1 -- 172.21.15.70:0/2143664939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37e4102440 msgr2=0x7f37e41028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:33.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.335+0000 7f37e984f700 1 --2- 172.21.15.70:0/2143664939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37e4102440 0x7f37e41028c0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f37d8009990 tx=0x7f37d8009ca0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.335+0000 7f37e984f700 1 -- 172.21.15.70:0/2143664939 shutdown_connections 2024-06-04T05:10:33.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.335+0000 7f37e984f700 1 --2- 172.21.15.70:0/2143664939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e4102e00 0x7f37e410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.335+0000 7f37e984f700 1 --2- 172.21.15.70:0/2143664939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e4108520 0x7f37e4108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.335+0000 7f37e984f700 1 --2- 172.21.15.70:0/2143664939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37e4102440 0x7f37e41028c0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.335+0000 7f37e984f700 1 -- 172.21.15.70:0/2143664939 >> 172.21.15.70:0/2143664939 conn(0x7f37e40793f0 msgr2=0x7f37e4079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:33.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.335+0000 7f37e984f700 1 -- 172.21.15.70:0/2143664939 shutdown_connections 2024-06-04T05:10:33.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.335+0000 7f37e984f700 1 -- 172.21.15.70:0/2143664939 wait complete. 2024-06-04T05:10:33.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.335+0000 7f37e984f700 1 Processor -- start 2024-06-04T05:10:33.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.336+0000 7f37e984f700 1 -- start start 2024-06-04T05:10:33.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.336+0000 7f37e984f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37e4102440 0x7f37e419f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.336+0000 7f37e984f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e4102e00 0x7f37e419fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.336+0000 7f37e984f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e4108520 0x7f37e4199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.336+0000 7f37e984f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f37e4112480 con 0x7f37e4102440 2024-06-04T05:10:33.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.336+0000 7f37e984f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37e4112300 con 0x7f37e4102e00 2024-06-04T05:10:33.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.336+0000 7f37e984f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f37e4112600 con 0x7f37e4108520 2024-06-04T05:10:33.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.336+0000 7f37e2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37e4102440 0x7f37e419f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.336+0000 7f37e2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37e4102440 0x7f37e419f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59628/0 (socket says 172.21.15.70:59628) 2024-06-04T05:10:33.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.336+0000 7f37e2ffd700 1 -- 172.21.15.70:0/1239552092 learned_addr learned my addr 172.21.15.70:0/1239552092 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:33.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.336+0000 7f37e27fc700 1 --2- 172.21.15.70:0/1239552092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e4102e00 0x7f37e419fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.336+0000 7f37e37fe700 1 --2- 172.21.15.70:0/1239552092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e4108520 0x7f37e4199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.336+0000 7f37e2ffd700 1 -- 172.21.15.70:0/1239552092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e4108520 msgr2=0x7f37e4199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.336+0000 7f37e2ffd700 1 --2- 172.21.15.70:0/1239552092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e4108520 0x7f37e4199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.336+0000 7f37e2ffd700 1 -- 172.21.15.70:0/1239552092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e4102e00 msgr2=0x7f37e419fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.336+0000 7f37e2ffd700 1 --2- 172.21.15.70:0/1239552092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e4102e00 0x7f37e419fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.336+0000 7f37e2ffd700 1 -- 172.21.15.70:0/1239552092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e419a0e0 con 0x7f37e4102440 2024-06-04T05:10:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.337+0000 7f37e2ffd700 1 --2- 172.21.15.70:0/1239552092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37e4102440 0x7f37e419f7c0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f37d400c9b0 tx=0x7f37d400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.337+0000 7f37e884d700 1 -- 172.21.15.70:0/1239552092 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d4007b00 con 0x7f37e4102440 2024-06-04T05:10:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.337+0000 7f37e884d700 1 -- 172.21.15.70:0/1239552092 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f37d400ce50 con 0x7f37e4102440 2024-06-04T05:10:33.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.337+0000 7f37e884d700 1 -- 172.21.15.70:0/1239552092 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d4018660 con 0x7f37e4102440 2024-06-04T05:10:33.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.337+0000 7f37e984f700 1 -- 172.21.15.70:0/1239552092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e419a3d0 con 0x7f37e4102440 2024-06-04T05:10:33.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.337+0000 7f37e984f700 1 -- 172.21.15.70:0/1239552092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e41a6690 con 0x7f37e4102440 2024-06-04T05:10:33.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.339+0000 7f37e884d700 1 -- 172.21.15.70:0/1239552092 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f37d400f450 con 0x7f37e4102440 2024-06-04T05:10:33.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.340+0000 7f37e884d700 1 --2- 172.21.15.70:0/1239552092 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f37cc0b2a70 0x7f37cc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.340+0000 7f37e27fc700 1 --2- 172.21.15.70:0/1239552092 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f37cc0b2a70 0x7f37cc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.340+0000 7f37e884d700 1 -- 172.21.15.70:0/1239552092 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f37d40e2470 con 0x7f37e4102440 2024-06-04T05:10:33.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.340+0000 7f37e27fc700 1 --2- 172.21.15.70:0/1239552092 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f37cc0b2a70 0x7f37cc0b4f30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f37d80098e0 tx=0x7f37d8005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:33.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.340+0000 7f37f40e4b80 1 Processor -- start 2024-06-04T05:10:33.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.340+0000 7f37f40e4b80 1 -- start start 2024-06-04T05:10:33.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.341+0000 7f37f40e4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e49c720 0x55fc3e49cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.341+0000 7f37f40e4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e497bf0 0x55fc3e498070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.341+0000 7f37f40e4b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e4985b0 0x55fc3e4a4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.341+0000 7f37f40e4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fc3e3f5e70 con 0x55fc3e497bf0 2024-06-04T05:10:33.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.341+0000 7f37f40e4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fc3e3e5a10 con 0x55fc3e49c720 2024-06-04T05:10:33.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.341+0000 7f37f40e4b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fc3e3e9a50 con 0x55fc3e4985b0 2024-06-04T05:10:33.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.341+0000 7f37e27fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e497bf0 0x55fc3e498070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.341+0000 7f37e27fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e497bf0 0x55fc3e498070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59644/0 (socket says 172.21.15.70:59644) 2024-06-04T05:10:33.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.341+0000 7f37e27fc700 1 -- 172.21.15.70:0/3677800261 learned_addr learned my addr 172.21.15.70:0/3677800261 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:33.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.341+0000 7f37e37fe700 1 --2- 172.21.15.70:0/3677800261 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e4985b0 0x55fc3e4a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.341+0000 7f37e2ffd700 1 --2- 172.21.15.70:0/3677800261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e49c720 0x55fc3e49cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:33 smithi070 ceph-mon[32631]: pgmap v416: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 24 KiB/s wr, 17 op/s 2024-06-04T05:10:33.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.341+0000 7f37e27fc700 1 -- 172.21.15.70:0/3677800261 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e4985b0 msgr2=0x55fc3e4a4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:33.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.341+0000 7f37e27fc700 1 --2- 172.21.15.70:0/3677800261 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e4985b0 0x55fc3e4a4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.341+0000 7f37e27fc700 1 -- 172.21.15.70:0/3677800261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e49c720 msgr2=0x55fc3e49cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:33.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.341+0000 7f37e27fc700 1 --2- 172.21.15.70:0/3677800261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e49c720 0x55fc3e49cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.341+0000 7f37e27fc700 1 -- 172.21.15.70:0/3677800261 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc3e38f0b0 con 0x55fc3e497bf0 2024-06-04T05:10:33.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.341+0000 7f37e27fc700 1 --2- 172.21.15.70:0/3677800261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e497bf0 0x55fc3e498070 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f37d8015000 tx=0x7f37d801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82d0e39eb512896 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:33.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.342+0000 7f37c9ffb700 1 -- 172.21.15.70:0/3677800261 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8034070 con 0x55fc3e497bf0 2024-06-04T05:10:33.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.342+0000 7f37c9ffb700 1 -- 172.21.15.70:0/3677800261 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f37d802f900 con 0x55fc3e497bf0 2024-06-04T05:10:33.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.342+0000 7f37c9ffb700 1 -- 172.21.15.70:0/3677800261 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d800f8d0 con 0x55fc3e497bf0 2024-06-04T05:10:33.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.342+0000 7f37f40e4b80 1 -- 172.21.15.70:0/3677800261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e497bf0 msgr2=0x55fc3e498070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:33.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.342+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/3677800261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e497bf0 0x55fc3e498070 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f37d8015000 tx=0x7f37d801f000 comp rx=0 tx=0).stop 2024-06-04T05:10:33.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.342+0000 7f37f40e4b80 1 -- 172.21.15.70:0/3677800261 shutdown_connections 2024-06-04T05:10:33.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.342+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/3677800261 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e4985b0 0x55fc3e4a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.342+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/3677800261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e49c720 0x55fc3e49cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.342+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/3677800261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e497bf0 0x55fc3e498070 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.342+0000 7f37f40e4b80 1 -- 172.21.15.70:0/3677800261 >> 172.21.15.70:0/3677800261 conn(0x55fc3e3c3340 msgr2=0x55fc3e4928e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:33.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.343+0000 7f37f40e4b80 1 -- 172.21.15.70:0/3677800261 shutdown_connections 2024-06-04T05:10:33.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.343+0000 7f37f40e4b80 1 -- 172.21.15.70:0/3677800261 wait complete. 2024-06-04T05:10:33.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.343+0000 7f37f40e4b80 1 -- start start 2024-06-04T05:10:33.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.343+0000 7f37f40e4b80 1 Processor -- start 2024-06-04T05:10:33.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.343+0000 7f37f40e4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e497bf0 0x55fc3e4a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.343+0000 7f37f40e4b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e4985b0 0x55fc3e4a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.343+0000 7f37f40e4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e49c720 0x55fc3e4a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.343+0000 7f37f40e4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fc3e2ec0a0 con 0x55fc3e49c720 2024-06-04T05:10:33.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.343+0000 7f37f40e4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fc3e34dea0 con 0x55fc3e497bf0 2024-06-04T05:10:33.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.343+0000 7f37f40e4b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fc3e4a4c20 con 0x55fc3e4985b0 2024-06-04T05:10:33.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.344+0000 7f37f40e4b80 10 client.0 _ll_register_callbacks cb 0x55fc3e380a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:33.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.344+0000 7f37e37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e49c720 0x55fc3e4a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.344+0000 7f37e37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e49c720 0x55fc3e4a1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59654/0 (socket says 172.21.15.70:59654) 2024-06-04T05:10:33.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.344+0000 7f37e37fe700 1 -- 172.21.15.70:0/342439911 learned_addr learned my addr 172.21.15.70:0/342439911 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:33.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.344+0000 7f37e27fc700 1 --2- 172.21.15.70:0/342439911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e4985b0 0x55fc3e4a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.344+0000 7f37e2ffd700 1 --2- 172.21.15.70:0/342439911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e497bf0 0x55fc3e4a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.344+0000 7f37e37fe700 1 -- 172.21.15.70:0/342439911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e4985b0 msgr2=0x55fc3e4a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:33.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.344+0000 7f37e37fe700 1 --2- 172.21.15.70:0/342439911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e4985b0 0x55fc3e4a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.344+0000 7f37e37fe700 1 -- 172.21.15.70:0/342439911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e497bf0 msgr2=0x55fc3e4a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:33.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.344+0000 7f37e37fe700 1 --2- 172.21.15.70:0/342439911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e497bf0 0x55fc3e4a42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.344+0000 7f37e37fe700 1 -- 172.21.15.70:0/342439911 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc3e3359d0 con 0x55fc3e49c720 2024-06-04T05:10:33.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.344+0000 7f37e37fe700 1 --2- 172.21.15.70:0/342439911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e49c720 0x55fc3e4a1b00 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00ae90 tx=0x7f37dc012510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d5c2125a19866c1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:33.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.344+0000 7f37c2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:33.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.344+0000 7f37c2ffd700 1 -- 172.21.15.70:0/342439911 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc01a070 con 0x55fc3e49c720 2024-06-04T05:10:33.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.344+0000 7f37c2ffd700 1 -- 172.21.15.70:0/342439911 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f37dc012ca0 con 0x55fc3e49c720 2024-06-04T05:10:33.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.344+0000 7f37f40e4b80 1 -- 172.21.15.70:0/342439911 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37d802f900 con 0x55fc3e49c720 2024-06-04T05:10:33.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.345+0000 7f37c2ffd700 1 -- 172.21.15.70:0/342439911 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f37dc026050 con 0x55fc3e49c720 2024-06-04T05:10:33.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.345+0000 7f37f40e4b80 10 client.19029 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:33.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.345+0000 7f37f40e4b80 1 -- 172.21.15.70:0/342439911 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fc3e39f860 con 0x55fc3e49c720 2024-06-04T05:10:33.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.345+0000 7f37c2ffd700 1 -- 172.21.15.70:0/342439911 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f37d802f900 con 0x55fc3e49c720 2024-06-04T05:10:33.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.345+0000 7f37c2ffd700 1 -- 172.21.15.70:0/342439911 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f37dc0083f0 con 0x55fc3e49c720 2024-06-04T05:10:33.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.345+0000 7f37f40e4b80 10 client.19029 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:33.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.345+0000 7f37f40e4b80 10 client.19029 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:10:33.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.345+0000 7f37f40e4b80 20 client.19029 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:33.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.345+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/342439911 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fc3e4b17f0 0x55fc3e4b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.345+0000 7f37f40e4b80 4 client.19029 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:10:33.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.345+0000 7f37f40e4b80 1 -- 172.21.15.70:0/342439911 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc3e3f7dc0 con 0x55fc3e4b17f0 2024-06-04T05:10:33.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.345+0000 7f37e2ffd700 1 --2- 172.21.15.70:0/342439911 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fc3e4b17f0 0x55fc3e4b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.346+0000 7f37e2ffd700 1 --2- 172.21.15.70:0/342439911 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fc3e4b17f0 0x55fc3e4b3be0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24e5d95b48ab0e73 server_cookie=2e9836315bbbb1e7 in_seq=0 out_seq=0 2024-06-04T05:10:33.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.346+0000 7f37c2ffd700 10 client.19029 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:33.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.346+0000 7f37c2ffd700 1 -- 172.21.15.70:0/342439911 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fc3e3f7dc0 con 0x55fc3e4b17f0 2024-06-04T05:10:33.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.346+0000 7f37c2ffd700 10 client.19029 handle_command_reply: tid=0 2024-06-04T05:10:33.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.346+0000 7f37f40e4b80 1 client.19029 shutdown 2024-06-04T05:10:33.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.347+0000 7f37f40e4b80 1 -- 172.21.15.70:0/342439911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e49c720 msgr2=0x55fc3e4a1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:33.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.347+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/342439911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e49c720 0x55fc3e4a1b00 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00ae90 tx=0x7f37dc012510 comp rx=0 tx=0).stop 2024-06-04T05:10:33.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.347+0000 7f37f40e4b80 1 -- 172.21.15.70:0/342439911 shutdown_connections 2024-06-04T05:10:33.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.347+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/342439911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e49c720 0x55fc3e4a1b00 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.347+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/342439911 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fc3e4b17f0 0x55fc3e4b3be0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.347+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/342439911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e497bf0 0x55fc3e4a42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.347+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/342439911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e4985b0 0x55fc3e4a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.347+0000 7f37c2ffd700 0 client.19029 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:33.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.347+0000 7f37f40e4b80 1 -- 172.21.15.70:0/342439911 >> 172.21.15.70:0/342439911 conn(0x55fc3e3c3340 msgr2=0x55fc3e496be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:33.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.347+0000 7f37f40e4b80 1 -- 172.21.15.70:0/342439911 shutdown_connections 2024-06-04T05:10:33.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.347+0000 7f37f40e4b80 1 -- 172.21.15.70:0/342439911 wait complete. 2024-06-04T05:10:33.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.347+0000 7f37f40e4b80 20 client.19029 trim_cache size 0 max 16384 2024-06-04T05:10:33.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.366+0000 7f37f40e4b80 1 Processor -- start 2024-06-04T05:10:33.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37f40e4b80 1 -- start start 2024-06-04T05:10:33.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37f40e4b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e497bf0 0x55fc3e48ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37f40e4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e4985b0 0x55fc3e49b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37f40e4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e49c720 0x55fc3e49bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37f40e4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41adcb0 con 0x55fc3e49c720 2024-06-04T05:10:33.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37f40e4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41af910 con 0x55fc3e4985b0 2024-06-04T05:10:33.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37f40e4b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41ae380 con 0x55fc3e497bf0 2024-06-04T05:10:33.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37e37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e49c720 0x55fc3e49bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37e2ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e497bf0 0x55fc3e48ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37e37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e49c720 0x55fc3e49bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59658/0 (socket says 172.21.15.70:59658) 2024-06-04T05:10:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37e37fe700 1 -- 172.21.15.70:0/1610530212 learned_addr learned my addr 172.21.15.70:0/1610530212 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37e27fc700 1 --2- 172.21.15.70:0/1610530212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e4985b0 0x55fc3e49b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37e2ffd700 1 -- 172.21.15.70:0/1610530212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e4985b0 msgr2=0x55fc3e49b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37e2ffd700 1 --2- 172.21.15.70:0/1610530212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e4985b0 0x55fc3e49b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37e2ffd700 1 -- 172.21.15.70:0/1610530212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e49c720 msgr2=0x55fc3e49bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37e2ffd700 1 --2- 172.21.15.70:0/1610530212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e49c720 0x55fc3e49bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37e2ffd700 1 -- 172.21.15.70:0/1610530212 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc3e49fb30 con 0x55fc3e497bf0 2024-06-04T05:10:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37e37fe700 1 --2- 172.21.15.70:0/1610530212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e49c720 0x55fc3e49bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.367+0000 7f37e2ffd700 1 --2- 172.21.15.70:0/1610530212 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e497bf0 0x55fc3e48ce70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f37d40dfb90 tx=0x7f37d4091040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fee6693b9677f6f2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.368+0000 7f37c97fa700 1 -- 172.21.15.70:0/1610530212 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d40c8070 con 0x55fc3e497bf0 2024-06-04T05:10:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.368+0000 7f37c97fa700 1 -- 172.21.15.70:0/1610530212 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f37d40b0e50 con 0x55fc3e497bf0 2024-06-04T05:10:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.368+0000 7f37c97fa700 1 -- 172.21.15.70:0/1610530212 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d400f740 con 0x55fc3e497bf0 2024-06-04T05:10:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.368+0000 7f37f40e4b80 1 -- 172.21.15.70:0/1610530212 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e497bf0 msgr2=0x55fc3e48ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.368+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/1610530212 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e497bf0 0x55fc3e48ce70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f37d40dfb90 tx=0x7f37d4091040 comp rx=0 tx=0).stop 2024-06-04T05:10:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.368+0000 7f37f40e4b80 1 -- 172.21.15.70:0/1610530212 shutdown_connections 2024-06-04T05:10:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.368+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/1610530212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e49c720 0x55fc3e49bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.368+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/1610530212 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e497bf0 0x55fc3e48ce70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.368+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/1610530212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e4985b0 0x55fc3e49b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.368+0000 7f37f40e4b80 1 -- 172.21.15.70:0/1610530212 >> 172.21.15.70:0/1610530212 conn(0x55fc3e3c3340 msgr2=0x55fc3e4a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.368+0000 7f37f40e4b80 1 -- 172.21.15.70:0/1610530212 shutdown_connections 2024-06-04T05:10:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.368+0000 7f37f40e4b80 1 -- 172.21.15.70:0/1610530212 wait complete. 2024-06-04T05:10:33.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37f40e4b80 1 -- start start 2024-06-04T05:10:33.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37f40e4b80 1 Processor -- start 2024-06-04T05:10:33.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37f40e4b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e497bf0 0x55fc3e4aa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37f40e4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e4985b0 0x55fc3e4a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37f40e4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e49c720 0x55fc3e4a5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37f40e4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41af1e0 con 0x55fc3e4985b0 2024-06-04T05:10:33.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37f40e4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41aeab0 con 0x55fc3e49c720 2024-06-04T05:10:33.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37f40e4b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41abc70 con 0x55fc3e497bf0 2024-06-04T05:10:33.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37f40e4b80 10 client.0 _ll_register_callbacks cb 0x55fc3e380a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37e27fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e4985b0 0x55fc3e4a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37e27fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e4985b0 0x55fc3e4a4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59660/0 (socket says 172.21.15.70:59660) 2024-06-04T05:10:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37e27fc700 1 -- 172.21.15.70:0/4219952604 learned_addr learned my addr 172.21.15.70:0/4219952604 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37e2ffd700 1 --2- 172.21.15.70:0/4219952604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e497bf0 0x55fc3e4aa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37e37fe700 1 --2- 172.21.15.70:0/4219952604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e49c720 0x55fc3e4a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37e27fc700 1 -- 172.21.15.70:0/4219952604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e497bf0 msgr2=0x55fc3e4aa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37e27fc700 1 --2- 172.21.15.70:0/4219952604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e497bf0 0x55fc3e4aa240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37e27fc700 1 -- 172.21.15.70:0/4219952604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e49c720 msgr2=0x55fc3e4a5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37e27fc700 1 --2- 172.21.15.70:0/4219952604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e49c720 0x55fc3e4a5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37e27fc700 1 -- 172.21.15.70:0/4219952604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc3e4a58b0 con 0x55fc3e4985b0 2024-06-04T05:10:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.369+0000 7f37e37fe700 1 --2- 172.21.15.70:0/4219952604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e49c720 0x55fc3e4a5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:10:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.370+0000 7f37e27fc700 1 --2- 172.21.15.70:0/4219952604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e4985b0 0x55fc3e4a4e30 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f37d8009750 tx=0x7f37d802f6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7064733ae4023b96 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.370+0000 7f37c2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.370+0000 7f37c2ffd700 1 -- 172.21.15.70:0/4219952604 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d803e060 con 0x55fc3e4985b0 2024-06-04T05:10:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.370+0000 7f37c2ffd700 1 -- 172.21.15.70:0/4219952604 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f37d802fad0 con 0x55fc3e4985b0 2024-06-04T05:10:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.370+0000 7f37f40e4b80 1 -- 172.21.15.70:0/4219952604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37d40b0e50 con 0x55fc3e4985b0 2024-06-04T05:10:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.370+0000 7f37c2ffd700 1 -- 172.21.15.70:0/4219952604 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f37d8034070 con 0x55fc3e4985b0 2024-06-04T05:10:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.370+0000 7f37f40e4b80 10 client.19035 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.370+0000 7f37f40e4b80 1 -- 172.21.15.70:0/4219952604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fc3e5214e0 con 0x55fc3e4985b0 2024-06-04T05:10:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.370+0000 7f37c2ffd700 1 -- 172.21.15.70:0/4219952604 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f37d40b0e50 con 0x55fc3e4985b0 2024-06-04T05:10:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.370+0000 7f37c2ffd700 1 -- 172.21.15.70:0/4219952604 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f37d8039440 con 0x55fc3e4985b0 2024-06-04T05:10:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.370+0000 7f37f40e4b80 10 client.19035 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.370+0000 7f37f40e4b80 10 client.19035 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:10:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.370+0000 7f37f40e4b80 20 client.19035 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.370+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/4219952604 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fc3e521700 0x55fc3e523af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:33.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.370+0000 7f37f40e4b80 4 client.19035 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "48e6b43c", "release": true, "await": true}] 2024-06-04T05:10:33.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.370+0000 7f37f40e4b80 1 -- 172.21.15.70:0/4219952604 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "48e6b43c", "release": true, "await": true}) -- 0x55fc3e3f15a0 con 0x55fc3e521700 2024-06-04T05:10:33.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.371+0000 7f37e2ffd700 1 --2- 172.21.15.70:0/4219952604 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fc3e521700 0x55fc3e523af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:33.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.371+0000 7f37c2ffd700 10 client.19035 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:33.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.371+0000 7f37e2ffd700 1 --2- 172.21.15.70:0/4219952604 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fc3e521700 0x55fc3e523af0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17f22753d61fb47f server_cookie=54be63faec33b514 in_seq=0 out_seq=0 2024-06-04T05:10:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37c2ffd700 1 -- 172.21.15.70:0/4219952604 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55fc3e3f15a0 con 0x55fc3e521700 2024-06-04T05:10:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37c2ffd700 10 client.19035 handle_command_reply: tid=0 2024-06-04T05:10:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37f40e4b80 1 client.19035 shutdown 2024-06-04T05:10:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37f40e4b80 1 -- 172.21.15.70:0/4219952604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e4985b0 msgr2=0x55fc3e4a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/4219952604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e4985b0 0x55fc3e4a4e30 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f37d8009750 tx=0x7f37d802f6a0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37f40e4b80 1 -- 172.21.15.70:0/4219952604 shutdown_connections 2024-06-04T05:10:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/4219952604 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fc3e521700 0x55fc3e523af0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/4219952604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fc3e49c720 0x55fc3e4a5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/4219952604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc3e497bf0 0x55fc3e4aa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37c2ffd700 0 client.19035 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:33.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37f40e4b80 1 --2- 172.21.15.70:0/4219952604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc3e4985b0 0x55fc3e4a4e30 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37f40e4b80 1 -- 172.21.15.70:0/4219952604 >> 172.21.15.70:0/4219952604 conn(0x55fc3e3c3340 msgr2=0x55fc3e49b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:33.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37f40e4b80 1 -- 172.21.15.70:0/4219952604 shutdown_connections 2024-06-04T05:10:33.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37f40e4b80 1 -- 172.21.15.70:0/4219952604 wait complete. 2024-06-04T05:10:33.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37f40e4b80 20 client.19035 trim_cache size 0 max 16384 2024-06-04T05:10:33.383 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:10:33.384 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 38, 2024-06-04T05:10:33.384 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:10:33.384 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 44, 2024-06-04T05:10:33.384 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:10:33.384 INFO:teuthology.orchestra.run.smithi070.stdout: "48e6b43c": { 2024-06-04T05:10:33.384 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 44, 2024-06-04T05:10:33.384 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:10:33.384 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:10:33.384 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:10:33.384 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:10:33.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:10:33.384 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:10:33.385 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 149.4, 2024-06-04T05:10:33.385 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:10:33.385 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:10:33.385 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:10:33.385 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:10:33.385 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:10:33.385 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:10:33.385 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:10:33.385 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:10:33.386 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:10:33.386 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:10:33.386 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:10:33.386 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:10:33.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37e984f700 1 -- 172.21.15.70:0/1239552092 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f37cc0b2a70 msgr2=0x7f37cc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37e984f700 1 --2- 172.21.15.70:0/1239552092 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f37cc0b2a70 0x7f37cc0b4f30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f37d80098e0 tx=0x7f37d8005960 comp rx=0 tx=0).stop 2024-06-04T05:10:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37e984f700 1 -- 172.21.15.70:0/1239552092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37e4102440 msgr2=0x7f37e419f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.376+0000 7f37e984f700 1 --2- 172.21.15.70:0/1239552092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37e4102440 0x7f37e419f7c0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f37d400c9b0 tx=0x7f37d400ccc0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.377+0000 7f37e984f700 1 -- 172.21.15.70:0/1239552092 shutdown_connections 2024-06-04T05:10:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.377+0000 7f37e984f700 1 --2- 172.21.15.70:0/1239552092 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f37cc0b2a70 0x7f37cc0b4f30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.377+0000 7f37e984f700 1 --2- 172.21.15.70:0/1239552092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e4108520 0x7f37e4199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.377+0000 7f37e984f700 1 --2- 172.21.15.70:0/1239552092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37e4102440 0x7f37e419f7c0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.377+0000 7f37e984f700 1 --2- 172.21.15.70:0/1239552092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e4102e00 0x7f37e419fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.377+0000 7f37e984f700 1 -- 172.21.15.70:0/1239552092 >> 172.21.15.70:0/1239552092 conn(0x7f37e40793f0 msgr2=0x7f37e406b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.377+0000 7f37e984f700 1 -- 172.21.15.70:0/1239552092 shutdown_connections 2024-06-04T05:10:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:33.377+0000 7f37e984f700 1 -- 172.21.15.70:0/1239552092 wait complete. 2024-06-04T05:10:33.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:33 smithi098 ceph-mon[35569]: pgmap v416: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 24 KiB/s wr, 17 op/s 2024-06-04T05:10:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:33 smithi112 ceph-mon[36453]: pgmap v416: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 24 KiB/s wr, 17 op/s 2024-06-04T05:10:33.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209538 1.57 MB/sec execute 566 sec latency 25394.064 ms 2024-06-04T05:10:34.085 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '48e6b43c', total seconds elapsed: 35.2 2024-06-04T05:10:34.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 209833 1.56 MB/sec execute 567 sec latency 15.620 ms 2024-06-04T05:10:35.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:35 smithi098 ceph-mon[35569]: pgmap v417: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-06-04T05:10:35.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:35 smithi070 ceph-mon[32631]: pgmap v417: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-06-04T05:10:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:35 smithi112 ceph-mon[36453]: pgmap v417: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-06-04T05:10:35.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 210328 1.56 MB/sec execute 568 sec latency 14.505 ms 2024-06-04T05:10:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.124+0000 7f4f3e093640 1 -- 172.21.15.70:0/3832090074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f3806bc40 msgr2=0x7f4f3810f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.124+0000 7f4f3e093640 1 --2- 172.21.15.70:0/3832090074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f3806bc40 0x7f4f3810f8f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4f30009cd0 tx=0x7f4f3002a970 comp rx=0 tx=0).stop 2024-06-04T05:10:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.124+0000 7f4f3e093640 1 -- 172.21.15.70:0/3832090074 shutdown_connections 2024-06-04T05:10:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.124+0000 7f4f3e093640 1 --2- 172.21.15.70:0/3832090074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f3806bc40 0x7f4f3810f8f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.124+0000 7f4f3e093640 1 --2- 172.21.15.70:0/3832090074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4f3806b280 0x7f4f3806b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.124+0000 7f4f3e093640 1 --2- 172.21.15.70:0/3832090074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f38108c50 0x7f4f38109030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.124+0000 7f4f3e093640 1 -- 172.21.15.70:0/3832090074 >> 172.21.15.70:0/3832090074 conn(0x7f4f38077750 msgr2=0x7f4f38077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.124+0000 7f4f3e093640 1 -- 172.21.15.70:0/3832090074 shutdown_connections 2024-06-04T05:10:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.124+0000 7f4f3e093640 1 -- 172.21.15.70:0/3832090074 wait complete. 2024-06-04T05:10:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.124+0000 7f4f3e093640 1 Processor -- start 2024-06-04T05:10:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f3e093640 1 -- start start 2024-06-04T05:10:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f3e093640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f3806b280 0x7f4f38199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f3e093640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f3806bc40 0x7f4f38199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f3e093640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4f38108c50 0x7f4f3819d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f3e093640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4f38112c20 con 0x7f4f3806b280 2024-06-04T05:10:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f3e093640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4f38112aa0 con 0x7f4f3806bc40 2024-06-04T05:10:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f3e093640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4f38112da0 con 0x7f4f38108c50 2024-06-04T05:10:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f377fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f3806b280 0x7f4f38199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f377fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f3806b280 0x7f4f38199040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55450/0 (socket says 172.21.15.70:55450) 2024-06-04T05:10:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f377fe640 1 -- 172.21.15.70:0/3722663286 learned_addr learned my addr 172.21.15.70:0/3722663286 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f37fff640 1 --2- 172.21.15.70:0/3722663286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4f38108c50 0x7f4f3819d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f36ffd640 1 --2- 172.21.15.70:0/3722663286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f3806bc40 0x7f4f38199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f377fe640 1 -- 172.21.15.70:0/3722663286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4f38108c50 msgr2=0x7f4f3819d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f377fe640 1 --2- 172.21.15.70:0/3722663286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4f38108c50 0x7f4f3819d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f377fe640 1 -- 172.21.15.70:0/3722663286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f3806bc40 msgr2=0x7f4f38199580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f377fe640 1 --2- 172.21.15.70:0/3722663286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f3806bc40 0x7f4f38199580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f377fe640 1 -- 172.21.15.70:0/3722663286 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3819e030 con 0x7f4f3806b280 2024-06-04T05:10:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.125+0000 7f4f377fe640 1 --2- 172.21.15.70:0/3722663286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f3806b280 0x7f4f38199040 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800e9f0 tx=0x7f4f2800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.126+0000 7f4f34ff9640 1 -- 172.21.15.70:0/3722663286 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2800cd80 con 0x7f4f3806b280 2024-06-04T05:10:36.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.126+0000 7f4f34ff9640 1 -- 172.21.15.70:0/3722663286 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4f280108c0 con 0x7f4f3806b280 2024-06-04T05:10:36.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.126+0000 7f4f3e093640 1 -- 172.21.15.70:0/3722663286 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f3819e320 con 0x7f4f3806b280 2024-06-04T05:10:36.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.126+0000 7f4f34ff9640 1 -- 172.21.15.70:0/3722663286 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f280187c0 con 0x7f4f3806b280 2024-06-04T05:10:36.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.126+0000 7f4f3e093640 1 -- 172.21.15.70:0/3722663286 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f381a5bb0 con 0x7f4f3806b280 2024-06-04T05:10:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.128+0000 7f4f40da9740 1 Processor -- start 2024-06-04T05:10:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.128+0000 7f4f40da9740 1 -- start start 2024-06-04T05:10:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f40da9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4640820 0x55d9c4640c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f34ff9640 1 -- 172.21.15.70:0/3722663286 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f4f28010430 con 0x7f4f3806b280 2024-06-04T05:10:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f40da9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c4641140 0x55d9c464b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f40da9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c464ba00 0x55d9c464ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f40da9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d9c45b2c30 con 0x55d9c4640820 2024-06-04T05:10:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f40da9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d9c45a27d0 con 0x55d9c4641140 2024-06-04T05:10:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f40da9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d9c45a6810 con 0x55d9c464ba00 2024-06-04T05:10:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f37fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4640820 0x55d9c4640c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c4641140 0x55d9c464b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f37fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4640820 0x55d9c4640c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55466/0 (socket says 172.21.15.70:55466) 2024-06-04T05:10:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f37fff640 1 -- 172.21.15.70:0/4184248330 learned_addr learned my addr 172.21.15.70:0/4184248330 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f36ffd640 1 --2- 172.21.15.70:0/4184248330 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c464ba00 0x55d9c464ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f37fff640 1 -- 172.21.15.70:0/4184248330 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c464ba00 msgr2=0x55d9c464ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f37fff640 1 --2- 172.21.15.70:0/4184248330 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c464ba00 0x55d9c464ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f37fff640 1 -- 172.21.15.70:0/4184248330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c4641140 msgr2=0x55d9c464b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f37fff640 1 --2- 172.21.15.70:0/4184248330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c4641140 0x55d9c464b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f37fff640 1 -- 172.21.15.70:0/4184248330 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9c42b7210 con 0x55d9c4640820 2024-06-04T05:10:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f377fe640 1 --2- 172.21.15.70:0/4184248330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c4641140 0x55d9c464b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:10:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.129+0000 7f4f37fff640 1 --2- 172.21.15.70:0/4184248330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4640820 0x55d9c4640c00 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f4f30009cd0 tx=0x7f4f30005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e47745e515b5a2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.130+0000 7f4f34ff9640 1 --2- 172.21.15.70:0/3722663286 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4f200b2a10 0x7f4f200b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.130+0000 7f4f1dffb640 1 -- 172.21.15.70:0/4184248330 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f30048070 con 0x55d9c4640820 2024-06-04T05:10:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.130+0000 7f4f377fe640 1 --2- 172.21.15.70:0/3722663286 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4f200b2a10 0x7f4f200b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.130+0000 7f4f1dffb640 1 -- 172.21.15.70:0/4184248330 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4f30003e00 con 0x55d9c4640820 2024-06-04T05:10:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.130+0000 7f4f1dffb640 1 -- 172.21.15.70:0/4184248330 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f30034070 con 0x55d9c4640820 2024-06-04T05:10:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.130+0000 7f4f34ff9640 1 -- 172.21.15.70:0/3722663286 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f4f280e29b0 con 0x7f4f3806b280 2024-06-04T05:10:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.130+0000 7f4f40da9740 1 -- 172.21.15.70:0/4184248330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4640820 msgr2=0x55d9c4640c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.130+0000 7f4f377fe640 1 --2- 172.21.15.70:0/3722663286 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4f200b2a10 0x7f4f200b4ed0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4f280e0180 tx=0x7f4f280e6230 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.130+0000 7f4f40da9740 1 --2- 172.21.15.70:0/4184248330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4640820 0x55d9c4640c00 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f4f30009cd0 tx=0x7f4f30005ef0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.131+0000 7f4f40da9740 1 -- 172.21.15.70:0/4184248330 shutdown_connections 2024-06-04T05:10:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.131+0000 7f4f40da9740 1 --2- 172.21.15.70:0/4184248330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c4641140 0x55d9c464b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.131+0000 7f4f40da9740 1 --2- 172.21.15.70:0/4184248330 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c464ba00 0x55d9c464ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.131+0000 7f4f40da9740 1 --2- 172.21.15.70:0/4184248330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4640820 0x55d9c4640c00 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.131+0000 7f4f40da9740 1 -- 172.21.15.70:0/4184248330 >> 172.21.15.70:0/4184248330 conn(0x55d9c463cd30 msgr2=0x55d9c463d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.131+0000 7f4f40da9740 1 -- 172.21.15.70:0/4184248330 shutdown_connections 2024-06-04T05:10:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.131+0000 7f4f40da9740 1 -- 172.21.15.70:0/4184248330 wait complete. 2024-06-04T05:10:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.131+0000 7f4f40da9740 1 -- start start 2024-06-04T05:10:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.131+0000 7f4f40da9740 1 Processor -- start 2024-06-04T05:10:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.131+0000 7f4f40da9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c4640820 0x55d9c4648a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.131+0000 7f4f40da9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c4641140 0x55d9c4648fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.131+0000 7f4f40da9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c464ba00 0x55d9c46494e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.131+0000 7f4f40da9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d9c4288790 con 0x55d9c464ba00 2024-06-04T05:10:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.131+0000 7f4f40da9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d9c455e890 con 0x55d9c4641140 2024-06-04T05:10:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.131+0000 7f4f40da9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d9c464e5d0 con 0x55d9c4640820 2024-06-04T05:10:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.131+0000 7f4f40da9740 10 client.0 _ll_register_callbacks cb 0x55d9c42b6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.131+0000 7f4f37fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c464ba00 0x55d9c46494e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.132+0000 7f4f37fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c464ba00 0x55d9c46494e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55478/0 (socket says 172.21.15.70:55478) 2024-06-04T05:10:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.132+0000 7f4f37fff640 1 -- 172.21.15.70:0/1049769766 learned_addr learned my addr 172.21.15.70:0/1049769766 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:36.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.132+0000 7f4f36ffd640 1 --2- 172.21.15.70:0/1049769766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c4641140 0x55d9c4648fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.132+0000 7f4f37fff640 1 --2- 172.21.15.70:0/1049769766 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c4640820 0x55d9c4648a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.132+0000 7f4f37fff640 1 -- 172.21.15.70:0/1049769766 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c4640820 msgr2=0x55d9c4648a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:36.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.132+0000 7f4f37fff640 1 --2- 172.21.15.70:0/1049769766 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c4640820 0x55d9c4648a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.132+0000 7f4f37fff640 1 -- 172.21.15.70:0/1049769766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c4641140 msgr2=0x55d9c4648fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:36.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.132+0000 7f4f37fff640 1 --2- 172.21.15.70:0/1049769766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c4641140 0x55d9c4648fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.132+0000 7f4f37fff640 1 -- 172.21.15.70:0/1049769766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9c4383620 con 0x55d9c464ba00 2024-06-04T05:10:36.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.132+0000 7f4f37fff640 1 --2- 172.21.15.70:0/1049769766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c464ba00 0x55d9c46494e0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f4f3002ae90 tx=0x7f4f3002b0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d00176ece04e213 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:36.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.132+0000 7f4f16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:36.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.132+0000 7f4f16ffd640 1 -- 172.21.15.70:0/1049769766 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f30057060 con 0x55d9c464ba00 2024-06-04T05:10:36.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.132+0000 7f4f40da9740 1 -- 172.21.15.70:0/1049769766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f30003e00 con 0x55d9c464ba00 2024-06-04T05:10:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.132+0000 7f4f16ffd640 1 -- 172.21.15.70:0/1049769766 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f4f3002f600 con 0x55d9c464ba00 2024-06-04T05:10:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.132+0000 7f4f16ffd640 1 -- 172.21.15.70:0/1049769766 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f4f30034030 con 0x55d9c464ba00 2024-06-04T05:10:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.132+0000 7f4f40da9740 10 client.19047 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.132+0000 7f4f40da9740 1 -- 172.21.15.70:0/1049769766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d9c465a3d0 con 0x55d9c464ba00 2024-06-04T05:10:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.133+0000 7f4f16ffd640 1 -- 172.21.15.70:0/1049769766 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4f30003e00 con 0x55d9c464ba00 2024-06-04T05:10:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.133+0000 7f4f16ffd640 1 -- 172.21.15.70:0/1049769766 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f4f3004e070 con 0x55d9c464ba00 2024-06-04T05:10:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.133+0000 7f4f40da9740 10 client.19047 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.133+0000 7f4f40da9740 10 client.19047 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:10:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.133+0000 7f4f40da9740 20 client.19047 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.133+0000 7f4f40da9740 1 --2- 172.21.15.70:0/1049769766 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d9c465b720 0x55d9c465db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.133+0000 7f4f40da9740 4 client.19047 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:10:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.133+0000 7f4f40da9740 1 -- 172.21.15.70:0/1049769766 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9c45b4b80 con 0x55d9c465b720 2024-06-04T05:10:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.133+0000 7f4f36ffd640 1 --2- 172.21.15.70:0/1049769766 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d9c465b720 0x55d9c465db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.133+0000 7f4f36ffd640 1 --2- 172.21.15.70:0/1049769766 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d9c465b720 0x55d9c465db10 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11c3f697e9d563d0 server_cookie=9a74231113863fe3 in_seq=0 out_seq=0 2024-06-04T05:10:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.133+0000 7f4f16ffd640 10 client.19047 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.134+0000 7f4f16ffd640 1 -- 172.21.15.70:0/1049769766 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d9c45b4b80 con 0x55d9c465b720 2024-06-04T05:10:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.134+0000 7f4f16ffd640 10 client.19047 handle_command_reply: tid=0 2024-06-04T05:10:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.134+0000 7f4f40da9740 1 client.19047 shutdown 2024-06-04T05:10:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.134+0000 7f4f40da9740 1 -- 172.21.15.70:0/1049769766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c464ba00 msgr2=0x55d9c46494e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.134+0000 7f4f40da9740 1 --2- 172.21.15.70:0/1049769766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c464ba00 0x55d9c46494e0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f4f3002ae90 tx=0x7f4f3002b0b0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.134+0000 7f4f40da9740 1 -- 172.21.15.70:0/1049769766 shutdown_connections 2024-06-04T05:10:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.134+0000 7f4f40da9740 1 --2- 172.21.15.70:0/1049769766 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d9c465b720 0x55d9c465db10 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.134+0000 7f4f40da9740 1 --2- 172.21.15.70:0/1049769766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c4641140 0x55d9c4648fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.134+0000 7f4f40da9740 1 --2- 172.21.15.70:0/1049769766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c464ba00 0x55d9c46494e0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.134+0000 7f4f40da9740 1 --2- 172.21.15.70:0/1049769766 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c4640820 0x55d9c4648a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.134+0000 7f4f40da9740 1 -- 172.21.15.70:0/1049769766 >> 172.21.15.70:0/1049769766 conn(0x55d9c463cd30 msgr2=0x55d9c463bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:36.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.134+0000 7f4f16ffd640 0 client.19047 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:36.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.134+0000 7f4f40da9740 1 -- 172.21.15.70:0/1049769766 shutdown_connections 2024-06-04T05:10:36.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.134+0000 7f4f40da9740 1 -- 172.21.15.70:0/1049769766 wait complete. 2024-06-04T05:10:36.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.135+0000 7f4f40da9740 20 client.19047 trim_cache size 0 max 16384 2024-06-04T05:10:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f40da9740 1 Processor -- start 2024-06-04T05:10:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f40da9740 1 -- start start 2024-06-04T05:10:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f40da9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c4640820 0x55d9c4642930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f40da9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4641140 0x55d9c4660b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f40da9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c464ba00 0x55d9c4661070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f40da9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4f381ae640 con 0x55d9c4641140 2024-06-04T05:10:36.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f40da9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4f381b02a0 con 0x55d9c464ba00 2024-06-04T05:10:36.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f40da9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4f381aed10 con 0x55d9c4640820 2024-06-04T05:10:36.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f36ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4641140 0x55d9c4660b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f37fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c4640820 0x55d9c4642930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f36ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4641140 0x55d9c4660b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55480/0 (socket says 172.21.15.70:55480) 2024-06-04T05:10:36.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f36ffd640 1 -- 172.21.15.70:0/2104098862 learned_addr learned my addr 172.21.15.70:0/2104098862 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:36.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f37fff640 1 --2- 172.21.15.70:0/2104098862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c464ba00 0x55d9c4661070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f37fff640 1 -- 172.21.15.70:0/2104098862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c464ba00 msgr2=0x55d9c4661070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:36.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f37fff640 1 --2- 172.21.15.70:0/2104098862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c464ba00 0x55d9c4661070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f37fff640 1 -- 172.21.15.70:0/2104098862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4641140 msgr2=0x55d9c4660b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:36.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f37fff640 1 --2- 172.21.15.70:0/2104098862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4641140 0x55d9c4660b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f37fff640 1 -- 172.21.15.70:0/2104098862 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9c46405e0 con 0x55d9c4640820 2024-06-04T05:10:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.150+0000 7f4f36ffd640 1 --2- 172.21.15.70:0/2104098862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4641140 0x55d9c4660b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.151+0000 7f4f37fff640 1 --2- 172.21.15.70:0/2104098862 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c4640820 0x55d9c4642930 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4f300307a0 tx=0x7f4f3002a970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7fedd617af79c32 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.151+0000 7f4f1d7fa640 1 -- 172.21.15.70:0/2104098862 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f30057040 con 0x55d9c4640820 2024-06-04T05:10:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.151+0000 7f4f1d7fa640 1 -- 172.21.15.70:0/2104098862 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4f300309a0 con 0x55d9c4640820 2024-06-04T05:10:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.151+0000 7f4f1d7fa640 1 -- 172.21.15.70:0/2104098862 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f30048040 con 0x55d9c4640820 2024-06-04T05:10:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.152+0000 7f4f40da9740 1 -- 172.21.15.70:0/2104098862 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c4640820 msgr2=0x55d9c4642930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.152+0000 7f4f40da9740 1 --2- 172.21.15.70:0/2104098862 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c4640820 0x55d9c4642930 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4f300307a0 tx=0x7f4f3002a970 comp rx=0 tx=0).stop 2024-06-04T05:10:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.152+0000 7f4f40da9740 1 -- 172.21.15.70:0/2104098862 shutdown_connections 2024-06-04T05:10:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.152+0000 7f4f40da9740 1 --2- 172.21.15.70:0/2104098862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c464ba00 0x55d9c4661070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.152+0000 7f4f40da9740 1 --2- 172.21.15.70:0/2104098862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4641140 0x55d9c4660b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.152+0000 7f4f40da9740 1 --2- 172.21.15.70:0/2104098862 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c4640820 0x55d9c4642930 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.152+0000 7f4f40da9740 1 -- 172.21.15.70:0/2104098862 >> 172.21.15.70:0/2104098862 conn(0x55d9c463cd30 msgr2=0x55d9c4642220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.152+0000 7f4f40da9740 1 -- 172.21.15.70:0/2104098862 shutdown_connections 2024-06-04T05:10:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.152+0000 7f4f40da9740 1 -- 172.21.15.70:0/2104098862 wait complete. 2024-06-04T05:10:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.152+0000 7f4f40da9740 1 -- start start 2024-06-04T05:10:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f40da9740 1 Processor -- start 2024-06-04T05:10:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f40da9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4640820 0x55d9c465e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f40da9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c4641140 0x55d9c465ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f40da9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c464ba00 0x55d9c465f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f40da9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4f381afb70 con 0x55d9c4640820 2024-06-04T05:10:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f40da9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4f381af440 con 0x55d9c4641140 2024-06-04T05:10:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f40da9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4f381ac640 con 0x55d9c464ba00 2024-06-04T05:10:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f40da9740 10 client.0 _ll_register_callbacks cb 0x7f4f30046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f37fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4640820 0x55d9c465e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f37fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4640820 0x55d9c465e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55486/0 (socket says 172.21.15.70:55486) 2024-06-04T05:10:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f37fff640 1 -- 172.21.15.70:0/563661859 learned_addr learned my addr 172.21.15.70:0/563661859 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f36ffd640 1 --2- 172.21.15.70:0/563661859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c4641140 0x55d9c465ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f37fff640 1 --2- 172.21.15.70:0/563661859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c464ba00 0x55d9c465f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f37fff640 1 -- 172.21.15.70:0/563661859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c464ba00 msgr2=0x55d9c465f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f37fff640 1 --2- 172.21.15.70:0/563661859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c464ba00 0x55d9c465f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f37fff640 1 -- 172.21.15.70:0/563661859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c4641140 msgr2=0x55d9c465ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f37fff640 1 --2- 172.21.15.70:0/563661859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c4641140 0x55d9c465ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f37fff640 1 -- 172.21.15.70:0/563661859 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9c46c4f60 con 0x55d9c4640820 2024-06-04T05:10:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.153+0000 7f4f36ffd640 1 --2- 172.21.15.70:0/563661859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c4641140 0x55d9c465ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.154+0000 7f4f37fff640 1 --2- 172.21.15.70:0/563661859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4640820 0x55d9c465e7b0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f4f3002af30 tx=0x7f4f3002b0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6cf25c9d83b19b1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.154+0000 7f4f16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.154+0000 7f4f16ffd640 1 -- 172.21.15.70:0/563661859 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f30057040 con 0x55d9c4640820 2024-06-04T05:10:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.154+0000 7f4f16ffd640 1 -- 172.21.15.70:0/563661859 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f4f30065440 con 0x55d9c4640820 2024-06-04T05:10:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.154+0000 7f4f40da9740 1 -- 172.21.15.70:0/563661859 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f300309a0 con 0x55d9c4640820 2024-06-04T05:10:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.154+0000 7f4f16ffd640 1 -- 172.21.15.70:0/563661859 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f4f30048040 con 0x55d9c4640820 2024-06-04T05:10:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.154+0000 7f4f40da9740 10 client.19053 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.154+0000 7f4f40da9740 1 -- 172.21.15.70:0/563661859 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d9c46c6170 con 0x55d9c4640820 2024-06-04T05:10:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.154+0000 7f4f16ffd640 1 -- 172.21.15.70:0/563661859 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4f300309a0 con 0x55d9c4640820 2024-06-04T05:10:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.154+0000 7f4f16ffd640 1 -- 172.21.15.70:0/563661859 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f4f30063020 con 0x55d9c4640820 2024-06-04T05:10:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.154+0000 7f4f40da9740 10 client.19053 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.154+0000 7f4f40da9740 10 client.19053 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:10:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.154+0000 7f4f40da9740 20 client.19053 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.154+0000 7f4f40da9740 1 --2- 172.21.15.70:0/563661859 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d9c46c6780 0x55d9c46c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.154+0000 7f4f40da9740 4 client.19053 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:10:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.155+0000 7f4f40da9740 1 -- 172.21.15.70:0/563661859 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d9c45ae360 con 0x55d9c46c6780 2024-06-04T05:10:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.155+0000 7f4f36ffd640 1 --2- 172.21.15.70:0/563661859 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d9c46c6780 0x55d9c46c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.155+0000 7f4f36ffd640 1 --2- 172.21.15.70:0/563661859 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d9c46c6780 0x55d9c46c8ba0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba6273a7c1a732f0 server_cookie=ec5fbf4552919f30 in_seq=0 out_seq=0 2024-06-04T05:10:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.155+0000 7f4f16ffd640 10 client.19053 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.175+0000 7f4f16ffd640 1 -- 172.21.15.70:0/563661859 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55d9c45ae360 con 0x55d9c46c6780 2024-06-04T05:10:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.175+0000 7f4f16ffd640 10 client.19053 handle_command_reply: tid=0 2024-06-04T05:10:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.175+0000 7f4f40da9740 1 client.19053 shutdown 2024-06-04T05:10:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.175+0000 7f4f40da9740 1 -- 172.21.15.70:0/563661859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4640820 msgr2=0x55d9c465e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.175+0000 7f4f40da9740 1 --2- 172.21.15.70:0/563661859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4640820 0x55d9c465e7b0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f4f3002af30 tx=0x7f4f3002b0b0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.175+0000 7f4f40da9740 1 -- 172.21.15.70:0/563661859 shutdown_connections 2024-06-04T05:10:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.175+0000 7f4f40da9740 1 --2- 172.21.15.70:0/563661859 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d9c46c6780 0x55d9c46c8ba0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.175+0000 7f4f40da9740 1 --2- 172.21.15.70:0/563661859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9c4641140 0x55d9c465ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.175+0000 7f4f40da9740 1 --2- 172.21.15.70:0/563661859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d9c464ba00 0x55d9c465f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.175+0000 7f4f40da9740 1 --2- 172.21.15.70:0/563661859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9c4640820 0x55d9c465e7b0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.175+0000 7f4f40da9740 1 -- 172.21.15.70:0/563661859 >> 172.21.15.70:0/563661859 conn(0x55d9c463cd30 msgr2=0x55d9c46492b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.175+0000 7f4f16ffd640 0 client.19053 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.176+0000 7f4f40da9740 1 -- 172.21.15.70:0/563661859 shutdown_connections 2024-06-04T05:10:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.176+0000 7f4f40da9740 1 -- 172.21.15.70:0/563661859 wait complete. 2024-06-04T05:10:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.176+0000 7f4f40da9740 20 client.19053 trim_cache size 0 max 16384 2024-06-04T05:10:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:10:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:10:36.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:10:36.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.177+0000 7f4f3e093640 1 -- 172.21.15.70:0/3722663286 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4f200b2a10 msgr2=0x7f4f200b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:36.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.177+0000 7f4f3e093640 1 --2- 172.21.15.70:0/3722663286 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4f200b2a10 0x7f4f200b4ed0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4f280e0180 tx=0x7f4f280e6230 comp rx=0 tx=0).stop 2024-06-04T05:10:36.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.177+0000 7f4f3e093640 1 -- 172.21.15.70:0/3722663286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f3806b280 msgr2=0x7f4f38199040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:36.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.177+0000 7f4f3e093640 1 --2- 172.21.15.70:0/3722663286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f3806b280 0x7f4f38199040 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800e9f0 tx=0x7f4f2800eeb0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.177+0000 7f4f3e093640 1 -- 172.21.15.70:0/3722663286 shutdown_connections 2024-06-04T05:10:36.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.177+0000 7f4f3e093640 1 --2- 172.21.15.70:0/3722663286 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4f200b2a10 0x7f4f200b4ed0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.177+0000 7f4f3e093640 1 --2- 172.21.15.70:0/3722663286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f3806bc40 0x7f4f38199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.177+0000 7f4f3e093640 1 --2- 172.21.15.70:0/3722663286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4f38108c50 0x7f4f3819d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.177+0000 7f4f3e093640 1 --2- 172.21.15.70:0/3722663286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f3806b280 0x7f4f38199040 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:36.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.177+0000 7f4f3e093640 1 -- 172.21.15.70:0/3722663286 >> 172.21.15.70:0/3722663286 conn(0x7f4f38077750 msgr2=0x7f4f38100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:36.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.177+0000 7f4f3e093640 1 -- 172.21.15.70:0/3722663286 shutdown_connections 2024-06-04T05:10:36.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:36.177+0000 7f4f3e093640 1 -- 172.21.15.70:0/3722663286 wait complete. 2024-06-04T05:10:36.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 210860 1.57 MB/sec execute 569 sec latency 19.958 ms 2024-06-04T05:10:37.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:37 smithi098 ceph-mon[35569]: pgmap v418: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 421 KiB/s wr, 51 op/s 2024-06-04T05:10:37.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:37 smithi070 ceph-mon[32631]: pgmap v418: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 421 KiB/s wr, 51 op/s 2024-06-04T05:10:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:37 smithi112 ceph-mon[36453]: pgmap v418: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 421 KiB/s wr, 51 op/s 2024-06-04T05:10:37.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 211464 1.57 MB/sec execute 570 sec latency 18.898 ms 2024-06-04T05:10:38.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 212028 1.57 MB/sec execute 571 sec latency 15.193 ms 2024-06-04T05:10:39.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:39 smithi098 ceph-mon[35569]: pgmap v419: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 402 KiB/s wr, 43 op/s 2024-06-04T05:10:39.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:39 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:10:39.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:39 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:10:39.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:39 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:10:39.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:39 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:10:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:39 smithi070 ceph-mon[32631]: pgmap v419: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 402 KiB/s wr, 43 op/s 2024-06-04T05:10:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:39 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:10:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:39 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:10:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:39 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:10:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:39 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:10:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:39 smithi112 ceph-mon[36453]: pgmap v419: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 402 KiB/s wr, 43 op/s 2024-06-04T05:10:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:39 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:10:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:39 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:10:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:39 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:10:39.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:39 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:10:39.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 212606 1.58 MB/sec execute 572 sec latency 15.519 ms 2024-06-04T05:10:40.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 212892 1.58 MB/sec execute 573 sec latency 543.032 ms 2024-06-04T05:10:41.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:41 smithi098 ceph-mon[35569]: pgmap v420: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 402 KiB/s wr, 43 op/s 2024-06-04T05:10:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:41 smithi070 ceph-mon[32631]: pgmap v420: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 402 KiB/s wr, 43 op/s 2024-06-04T05:10:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:41 smithi112 ceph-mon[36453]: pgmap v420: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 402 KiB/s wr, 43 op/s 2024-06-04T05:10:41.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 212892 1.58 MB/sec execute 574 sec latency 1543.103 ms 2024-06-04T05:10:42.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 212959 1.57 MB/sec execute 575 sec latency 1845.100 ms 2024-06-04T05:10:43.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:43 smithi098 ceph-mon[35569]: pgmap v421: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 985 KiB/s wr, 96 op/s 2024-06-04T05:10:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:43 smithi070 ceph-mon[32631]: pgmap v421: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 985 KiB/s wr, 96 op/s 2024-06-04T05:10:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:43 smithi112 ceph-mon[36453]: pgmap v421: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 985 KiB/s wr, 96 op/s 2024-06-04T05:10:43.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 212959 1.57 MB/sec execute 576 sec latency 1480.424 ms 2024-06-04T05:10:44.592 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:10:44.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 212959 1.57 MB/sec execute 577 sec latency 2480.478 ms 2024-06-04T05:10:44.844 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:10:45.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:45 smithi098 ceph-mon[35569]: pgmap v422: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 985 KiB/s wr, 92 op/s 2024-06-04T05:10:45.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:45 smithi070 ceph-mon[32631]: pgmap v422: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 985 KiB/s wr, 92 op/s 2024-06-04T05:10:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:45 smithi112 ceph-mon[36453]: pgmap v422: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 985 KiB/s wr, 92 op/s 2024-06-04T05:10:45.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 212959 1.57 MB/sec execute 578 sec latency 3480.555 ms 2024-06-04T05:10:46.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.720+0000 7f6822141700 1 -- 172.21.15.70:0/3954195353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c1041a0 msgr2=0x7f681c1081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:46.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.720+0000 7f6822141700 1 --2- 172.21.15.70:0/3954195353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c1041a0 0x7f681c1081c0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f680c0099c0 tx=0x7f680c009cd0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.720+0000 7f6822141700 1 -- 172.21.15.70:0/3954195353 shutdown_connections 2024-06-04T05:10:46.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.720+0000 7f6822141700 1 --2- 172.21.15.70:0/3954195353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f681c1088f0 0x7f681c110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.720+0000 7f6822141700 1 --2- 172.21.15.70:0/3954195353 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f681c1037f0 0x7f681c103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.720+0000 7f6822141700 1 --2- 172.21.15.70:0/3954195353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c1041a0 0x7f681c1081c0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.720+0000 7f6822141700 1 -- 172.21.15.70:0/3954195353 >> 172.21.15.70:0/3954195353 conn(0x7f681c07a6a0 msgr2=0x7f681c07aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:46.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.720+0000 7f6822141700 1 -- 172.21.15.70:0/3954195353 shutdown_connections 2024-06-04T05:10:46.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.720+0000 7f6822141700 1 -- 172.21.15.70:0/3954195353 wait complete. 2024-06-04T05:10:46.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.721+0000 7f6822141700 1 Processor -- start 2024-06-04T05:10:46.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.721+0000 7f6822141700 1 -- start start 2024-06-04T05:10:46.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.721+0000 7f6822141700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c1037f0 0x7f681c075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.721+0000 7f6822141700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f681c1041a0 0x7f681c075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.721+0000 7f6822141700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f681c1088f0 0x7f681c0762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.721+0000 7f6822141700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f681c113560 con 0x7f681c1037f0 2024-06-04T05:10:46.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.721+0000 7f6822141700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f681c1133e0 con 0x7f681c1088f0 2024-06-04T05:10:46.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.721+0000 7f6822141700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f681c1136e0 con 0x7f681c1041a0 2024-06-04T05:10:46.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.721+0000 7f681b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c1037f0 0x7f681c075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:46.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.721+0000 7f681b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c1037f0 0x7f681c075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44694/0 (socket says 172.21.15.70:44694) 2024-06-04T05:10:46.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.721+0000 7f681b7fe700 1 -- 172.21.15.70:0/3036562501 learned_addr learned my addr 172.21.15.70:0/3036562501 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:46.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.722+0000 7f681affd700 1 --2- 172.21.15.70:0/3036562501 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f681c1041a0 0x7f681c075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:46.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.722+0000 7f681bfff700 1 --2- 172.21.15.70:0/3036562501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f681c1088f0 0x7f681c0762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:46.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.722+0000 7f681b7fe700 1 -- 172.21.15.70:0/3036562501 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f681c1041a0 msgr2=0x7f681c075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:46.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.722+0000 7f681b7fe700 1 --2- 172.21.15.70:0/3036562501 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f681c1041a0 0x7f681c075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.722+0000 7f681b7fe700 1 -- 172.21.15.70:0/3036562501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f681c1088f0 msgr2=0x7f681c0762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:46.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.722+0000 7f681b7fe700 1 --2- 172.21.15.70:0/3036562501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f681c1088f0 0x7f681c0762c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.722+0000 7f681b7fe700 1 -- 172.21.15.70:0/3036562501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f681c1a7ed0 con 0x7f681c1037f0 2024-06-04T05:10:46.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.722+0000 7f681affd700 1 --2- 172.21.15.70:0/3036562501 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f681c1041a0 0x7f681c075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:10:46.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.722+0000 7f681b7fe700 1 --2- 172.21.15.70:0/3036562501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c1037f0 0x7f681c075840 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f681000c9b0 tx=0x7f681000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:46.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.722+0000 7f6818ff9700 1 -- 172.21.15.70:0/3036562501 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6810007b00 con 0x7f681c1037f0 2024-06-04T05:10:46.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.722+0000 7f6818ff9700 1 -- 172.21.15.70:0/3036562501 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6810007ca0 con 0x7f681c1037f0 2024-06-04T05:10:46.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.722+0000 7f6822141700 1 -- 172.21.15.70:0/3036562501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f681c1a81c0 con 0x7f681c1037f0 2024-06-04T05:10:46.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.722+0000 7f6818ff9700 1 -- 172.21.15.70:0/3036562501 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6810018660 con 0x7f681c1037f0 2024-06-04T05:10:46.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.722+0000 7f6822141700 1 -- 172.21.15.70:0/3036562501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f681c1a8750 con 0x7f681c1037f0 2024-06-04T05:10:46.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.725+0000 7f6818ff9700 1 -- 172.21.15.70:0/3036562501 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f68100188c0 con 0x7f681c1037f0 2024-06-04T05:10:46.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.725+0000 7f6818ff9700 1 --2- 172.21.15.70:0/3036562501 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68040b2a70 0x7f68040b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.725+0000 7f681affd700 1 --2- 172.21.15.70:0/3036562501 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68040b2a70 0x7f68040b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:46.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f6818ff9700 1 -- 172.21.15.70:0/3036562501 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f68100e26b0 con 0x7f681c1037f0 2024-06-04T05:10:46.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f682c9d6b80 1 Processor -- start 2024-06-04T05:10:46.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f682c9d6b80 1 -- start start 2024-06-04T05:10:46.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f681affd700 1 --2- 172.21.15.70:0/3036562501 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68040b2a70 0x7f68040b4f30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f680c000c00 tx=0x7f680c005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:46.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f682c9d6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db59860 0x55d23db59c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f682c9d6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db5a210 0x55d23db5d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f682c9d6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5df60 0x55d23db5e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f682c9d6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d23dab8c90 con 0x55d23db59860 2024-06-04T05:10:46.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f682c9d6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d23daa8830 con 0x55d23db5a210 2024-06-04T05:10:46.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f682c9d6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d23daac870 con 0x55d23db5df60 2024-06-04T05:10:46.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f681b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db59860 0x55d23db59c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:46.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f681b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db59860 0x55d23db59c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44704/0 (socket says 172.21.15.70:44704) 2024-06-04T05:10:46.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f681b7fe700 1 -- 172.21.15.70:0/1621276535 learned_addr learned my addr 172.21.15.70:0/1621276535 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:46.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f681b7fe700 1 -- 172.21.15.70:0/1621276535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5df60 msgr2=0x55d23db5e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:46.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f681b7fe700 1 --2- 172.21.15.70:0/1621276535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5df60 0x55d23db5e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f681b7fe700 1 -- 172.21.15.70:0/1621276535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db5a210 msgr2=0x55d23db5d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:46.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f681b7fe700 1 --2- 172.21.15.70:0/1621276535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db5a210 0x55d23db5d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.726+0000 7f681b7fe700 1 -- 172.21.15.70:0/1621276535 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d23d9f88d0 con 0x55d23db59860 2024-06-04T05:10:46.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.727+0000 7f681b7fe700 1 --2- 172.21.15.70:0/1621276535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db59860 0x55d23db59c40 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f68100a59e0 tx=0x7f68100a93e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c04f5057ad0d558b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:46.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.727+0000 7f6801ffb700 1 -- 172.21.15.70:0/1621276535 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68100add10 con 0x55d23db59860 2024-06-04T05:10:46.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.727+0000 7f6801ffb700 1 -- 172.21.15.70:0/1621276535 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68100b27c0 con 0x55d23db59860 2024-06-04T05:10:46.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.727+0000 7f6801ffb700 1 -- 172.21.15.70:0/1621276535 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68100a9bc0 con 0x55d23db59860 2024-06-04T05:10:46.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.727+0000 7f682c9d6b80 1 -- 172.21.15.70:0/1621276535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db59860 msgr2=0x55d23db59c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:46.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.727+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/1621276535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db59860 0x55d23db59c40 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f68100a59e0 tx=0x7f68100a93e0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.727+0000 7f682c9d6b80 1 -- 172.21.15.70:0/1621276535 shutdown_connections 2024-06-04T05:10:46.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.727+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/1621276535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5df60 0x55d23db5e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.727+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/1621276535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db59860 0x55d23db59c40 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.727+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/1621276535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db5a210 0x55d23db5d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.727+0000 7f682c9d6b80 1 -- 172.21.15.70:0/1621276535 >> 172.21.15.70:0/1621276535 conn(0x55d23da87080 msgr2=0x55d23db53e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:46.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.728+0000 7f682c9d6b80 1 -- 172.21.15.70:0/1621276535 shutdown_connections 2024-06-04T05:10:46.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.728+0000 7f682c9d6b80 1 -- 172.21.15.70:0/1621276535 wait complete. 2024-06-04T05:10:46.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.728+0000 7f682c9d6b80 1 -- start start 2024-06-04T05:10:46.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.728+0000 7f682c9d6b80 1 Processor -- start 2024-06-04T05:10:46.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.728+0000 7f682c9d6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db59860 0x55d23db65500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.728+0000 7f682c9d6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5a210 0x55d23db62800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.728+0000 7f682c9d6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db5df60 0x55d23db62d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.728+0000 7f682c9d6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d23d9aefa0 con 0x55d23db5df60 2024-06-04T05:10:46.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.728+0000 7f682c9d6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d23da4a7c0 con 0x55d23db59860 2024-06-04T05:10:46.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.728+0000 7f682c9d6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d23db66120 con 0x55d23db5a210 2024-06-04T05:10:46.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.728+0000 7f682c9d6b80 10 client.0 _ll_register_callbacks cb 0x55d23da43920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:46.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f681bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db5df60 0x55d23db62d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:46.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f681bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db5df60 0x55d23db62d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44706/0 (socket says 172.21.15.70:44706) 2024-06-04T05:10:46.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f681bfff700 1 -- 172.21.15.70:0/374484125 learned_addr learned my addr 172.21.15.70:0/374484125 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:46.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f681b7fe700 1 --2- 172.21.15.70:0/374484125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db59860 0x55d23db65500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:46.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f681affd700 1 --2- 172.21.15.70:0/374484125 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5a210 0x55d23db62800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:46.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f681bfff700 1 -- 172.21.15.70:0/374484125 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5a210 msgr2=0x55d23db62800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:46.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f681bfff700 1 --2- 172.21.15.70:0/374484125 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5a210 0x55d23db62800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f681bfff700 1 -- 172.21.15.70:0/374484125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db59860 msgr2=0x55d23db65500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:46.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f681bfff700 1 --2- 172.21.15.70:0/374484125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db59860 0x55d23db65500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f681bfff700 1 -- 172.21.15.70:0/374484125 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d23da47690 con 0x55d23db5df60 2024-06-04T05:10:46.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f681bfff700 1 --2- 172.21.15.70:0/374484125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db5df60 0x55d23db62d40 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f681400d830 tx=0x7f681400db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e37d446adca6b0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:46.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f67faffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:46.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f67faffd700 1 -- 172.21.15.70:0/374484125 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6814016070 con 0x55d23db5df60 2024-06-04T05:10:46.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f67faffd700 1 -- 172.21.15.70:0/374484125 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f6814012800 con 0x55d23db5df60 2024-06-04T05:10:46.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f682c9d6b80 1 -- 172.21.15.70:0/374484125 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68100b27c0 con 0x55d23db5df60 2024-06-04T05:10:46.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f67faffd700 1 -- 172.21.15.70:0/374484125 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6814022050 con 0x55d23db5df60 2024-06-04T05:10:46.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f682c9d6b80 10 client.19065 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:46.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.729+0000 7f682c9d6b80 1 -- 172.21.15.70:0/374484125 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d23db709e0 con 0x55d23db5df60 2024-06-04T05:10:46.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.730+0000 7f67faffd700 1 -- 172.21.15.70:0/374484125 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68100b27c0 con 0x55d23db5df60 2024-06-04T05:10:46.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.730+0000 7f67faffd700 1 -- 172.21.15.70:0/374484125 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f681401aad0 con 0x55d23db5df60 2024-06-04T05:10:46.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.730+0000 7f682c9d6b80 10 client.19065 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:46.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.730+0000 7f682c9d6b80 10 client.19065 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:10:46.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.730+0000 7f682c9d6b80 20 client.19065 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:46.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.730+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/374484125 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d23db72fc0 0x55d23db75420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.730+0000 7f682c9d6b80 4 client.19065 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:10:46.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.730+0000 7f682c9d6b80 1 -- 172.21.15.70:0/374484125 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d23db259e0 con 0x55d23db72fc0 2024-06-04T05:10:46.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.730+0000 7f681b7fe700 1 --2- 172.21.15.70:0/374484125 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d23db72fc0 0x55d23db75420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:46.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.731+0000 7f681b7fe700 1 --2- 172.21.15.70:0/374484125 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d23db72fc0 0x55d23db75420 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10d0e32e40a2c9d8 server_cookie=c2e78a1b529d1537 in_seq=0 out_seq=0 2024-06-04T05:10:46.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.731+0000 7f67faffd700 10 client.19065 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.731+0000 7f67faffd700 1 -- 172.21.15.70:0/374484125 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d23db259e0 con 0x55d23db72fc0 2024-06-04T05:10:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.731+0000 7f67faffd700 10 client.19065 handle_command_reply: tid=0 2024-06-04T05:10:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.731+0000 7f682c9d6b80 1 client.19065 shutdown 2024-06-04T05:10:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.731+0000 7f682c9d6b80 1 -- 172.21.15.70:0/374484125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db5df60 msgr2=0x55d23db62d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.731+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/374484125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db5df60 0x55d23db62d40 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f681400d830 tx=0x7f681400db40 comp rx=0 tx=0).stop 2024-06-04T05:10:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.732+0000 7f682c9d6b80 1 -- 172.21.15.70:0/374484125 shutdown_connections 2024-06-04T05:10:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.732+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/374484125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db5df60 0x55d23db62d40 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.732+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/374484125 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d23db72fc0 0x55d23db75420 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.732+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/374484125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db59860 0x55d23db65500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.732+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/374484125 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5a210 0x55d23db62800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.732+0000 7f682c9d6b80 1 -- 172.21.15.70:0/374484125 >> 172.21.15.70:0/374484125 conn(0x55d23da87080 msgr2=0x55d23db53510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:46.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.732+0000 7f67faffd700 0 client.19065 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:46.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.732+0000 7f682c9d6b80 1 -- 172.21.15.70:0/374484125 shutdown_connections 2024-06-04T05:10:46.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.732+0000 7f682c9d6b80 1 -- 172.21.15.70:0/374484125 wait complete. 2024-06-04T05:10:46.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.732+0000 7f682c9d6b80 20 client.19065 trim_cache size 0 max 16384 2024-06-04T05:10:46.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 212959 1.56 MB/sec execute 579 sec latency 4480.621 ms 2024-06-04T05:10:46.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f682c9d6b80 1 Processor -- start 2024-06-04T05:10:46.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f682c9d6b80 1 -- start start 2024-06-04T05:10:46.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f682c9d6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db59860 0x55d23db61fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f682c9d6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db5a210 0x55d23db53c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f682c9d6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5df60 0x55d23db7acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f682c9d6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f681c1afff0 con 0x55d23db5a210 2024-06-04T05:10:46.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f682c9d6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f681c1b1c50 con 0x55d23db59860 2024-06-04T05:10:46.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f682c9d6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f681c1b06c0 con 0x55d23db5df60 2024-06-04T05:10:46.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f681affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db5a210 0x55d23db53c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:46.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f681affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db5a210 0x55d23db53c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44722/0 (socket says 172.21.15.70:44722) 2024-06-04T05:10:46.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f681affd700 1 -- 172.21.15.70:0/845932829 learned_addr learned my addr 172.21.15.70:0/845932829 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:46.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f681bfff700 1 --2- 172.21.15.70:0/845932829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5df60 0x55d23db7acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:46.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f681affd700 1 -- 172.21.15.70:0/845932829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5df60 msgr2=0x55d23db7acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:46.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f681affd700 1 --2- 172.21.15.70:0/845932829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5df60 0x55d23db7acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f681b7fe700 1 --2- 172.21.15.70:0/845932829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db59860 0x55d23db61fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:46.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f681affd700 1 -- 172.21.15.70:0/845932829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db59860 msgr2=0x55d23db61fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:46.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f681affd700 1 --2- 172.21.15.70:0/845932829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db59860 0x55d23db61fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.752+0000 7f681affd700 1 -- 172.21.15.70:0/845932829 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d23db60d70 con 0x55d23db5a210 2024-06-04T05:10:46.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.753+0000 7f681affd700 1 --2- 172.21.15.70:0/845932829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db5a210 0x55d23db53c90 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f680c0099c0 tx=0x7f680c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def59c0621d9e292 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:46.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.753+0000 7f68017fa700 1 -- 172.21.15.70:0/845932829 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c034070 con 0x55d23db5a210 2024-06-04T05:10:46.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.753+0000 7f68017fa700 1 -- 172.21.15.70:0/845932829 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f680c005670 con 0x55d23db5a210 2024-06-04T05:10:46.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.753+0000 7f68017fa700 1 -- 172.21.15.70:0/845932829 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c00f9b0 con 0x55d23db5a210 2024-06-04T05:10:46.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.753+0000 7f682c9d6b80 1 -- 172.21.15.70:0/845932829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db5a210 msgr2=0x55d23db53c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:46.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.753+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/845932829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db5a210 0x55d23db53c90 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f680c0099c0 tx=0x7f680c01d000 comp rx=0 tx=0).stop 2024-06-04T05:10:46.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.754+0000 7f682c9d6b80 1 -- 172.21.15.70:0/845932829 shutdown_connections 2024-06-04T05:10:46.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.754+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/845932829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5df60 0x55d23db7acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.754+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/845932829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db59860 0x55d23db61fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.754+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/845932829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db5a210 0x55d23db53c90 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.754+0000 7f682c9d6b80 1 -- 172.21.15.70:0/845932829 >> 172.21.15.70:0/845932829 conn(0x55d23da87080 msgr2=0x55d23db648c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:46.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.754+0000 7f682c9d6b80 1 -- 172.21.15.70:0/845932829 shutdown_connections 2024-06-04T05:10:46.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.754+0000 7f682c9d6b80 1 -- 172.21.15.70:0/845932829 wait complete. 2024-06-04T05:10:46.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.754+0000 7f682c9d6b80 1 -- start start 2024-06-04T05:10:46.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.754+0000 7f682c9d6b80 1 Processor -- start 2024-06-04T05:10:46.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.754+0000 7f682c9d6b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db59860 0x55d23db69450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.754+0000 7f682c9d6b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5a210 0x55d23db69990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.754+0000 7f682c9d6b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db5df60 0x55d23dbdcbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.754+0000 7f682c9d6b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f681c1b1520 con 0x55d23db59860 2024-06-04T05:10:46.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.754+0000 7f682c9d6b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f681c1b0df0 con 0x55d23db5df60 2024-06-04T05:10:46.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.754+0000 7f682c9d6b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f681c1adfb0 con 0x55d23db5a210 2024-06-04T05:10:46.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.754+0000 7f682c9d6b80 10 client.0 _ll_register_callbacks cb 0x55d23da43920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:46.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.754+0000 7f681b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db59860 0x55d23db69450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:46.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f681b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db59860 0x55d23db69450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44734/0 (socket says 172.21.15.70:44734) 2024-06-04T05:10:46.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f681b7fe700 1 -- 172.21.15.70:0/2123929532 learned_addr learned my addr 172.21.15.70:0/2123929532 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:46.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f681affd700 1 --2- 172.21.15.70:0/2123929532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5a210 0x55d23db69990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:46.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f681bfff700 1 --2- 172.21.15.70:0/2123929532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db5df60 0x55d23dbdcbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:46.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f681b7fe700 1 -- 172.21.15.70:0/2123929532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5a210 msgr2=0x55d23db69990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:46.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f681b7fe700 1 --2- 172.21.15.70:0/2123929532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5a210 0x55d23db69990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f681b7fe700 1 -- 172.21.15.70:0/2123929532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db5df60 msgr2=0x55d23dbdcbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:46.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f681b7fe700 1 --2- 172.21.15.70:0/2123929532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db5df60 0x55d23dbdcbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f681b7fe700 1 -- 172.21.15.70:0/2123929532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d23db6a270 con 0x55d23db59860 2024-06-04T05:10:46.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f681bfff700 1 --2- 172.21.15.70:0/2123929532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db5df60 0x55d23dbdcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:10:46.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f681b7fe700 1 --2- 172.21.15.70:0/2123929532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db59860 0x55d23db69450 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f68100a6bd0 tx=0x7f68100a9860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf701aed343222b6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:46.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f67faffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:46.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f67faffd700 1 -- 172.21.15.70:0/2123929532 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68100b27c0 con 0x55d23db59860 2024-06-04T05:10:46.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f67faffd700 1 -- 172.21.15.70:0/2123929532 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f68100cc0b0 con 0x55d23db59860 2024-06-04T05:10:46.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f682c9d6b80 1 -- 172.21.15.70:0/2123929532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f680c005670 con 0x55d23db59860 2024-06-04T05:10:46.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f67faffd700 1 -- 172.21.15.70:0/2123929532 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f68100cc2b0 con 0x55d23db59860 2024-06-04T05:10:46.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f682c9d6b80 10 client.19071 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:46.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f682c9d6b80 1 -- 172.21.15.70:0/2123929532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d23dbde230 con 0x55d23db59860 2024-06-04T05:10:46.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.755+0000 7f67faffd700 1 -- 172.21.15.70:0/2123929532 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f680c005670 con 0x55d23db59860 2024-06-04T05:10:46.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.756+0000 7f67faffd700 1 -- 172.21.15.70:0/2123929532 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f68100b2330 con 0x55d23db59860 2024-06-04T05:10:46.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.756+0000 7f682c9d6b80 10 client.19071 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:46.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.756+0000 7f682c9d6b80 10 client.19071 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:10:46.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.756+0000 7f682c9d6b80 20 client.19071 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:46.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.756+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/2123929532 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d23dbde510 0x55d23dbe0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:46.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.756+0000 7f682c9d6b80 4 client.19071 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:10:46.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.756+0000 7f682c9d6b80 1 -- 172.21.15.70:0/2123929532 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d23dababe0 con 0x55d23dbde510 2024-06-04T05:10:46.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.756+0000 7f681affd700 1 --2- 172.21.15.70:0/2123929532 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d23dbde510 0x55d23dbe0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:46.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.756+0000 7f67faffd700 10 client.19071 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:46.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.756+0000 7f681affd700 1 --2- 172.21.15.70:0/2123929532 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d23dbde510 0x55d23dbe0900 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47b9a5003b1a022b server_cookie=1736eaf15019d7c6 in_seq=0 out_seq=0 2024-06-04T05:10:46.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.756+0000 7f67faffd700 1 -- 172.21.15.70:0/2123929532 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d23dababe0 con 0x55d23dbde510 2024-06-04T05:10:46.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.756+0000 7f67faffd700 10 client.19071 handle_command_reply: tid=0 2024-06-04T05:10:46.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.756+0000 7f682c9d6b80 1 client.19071 shutdown 2024-06-04T05:10:46.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f682c9d6b80 1 -- 172.21.15.70:0/2123929532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db59860 msgr2=0x55d23db69450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:46.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/2123929532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db59860 0x55d23db69450 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f68100a6bd0 tx=0x7f68100a9860 comp rx=0 tx=0).stop 2024-06-04T05:10:46.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f682c9d6b80 1 -- 172.21.15.70:0/2123929532 shutdown_connections 2024-06-04T05:10:46.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/2123929532 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d23dbde510 0x55d23dbe0900 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/2123929532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d23db5df60 0x55d23dbdcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/2123929532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d23db59860 0x55d23db69450 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f682c9d6b80 1 --2- 172.21.15.70:0/2123929532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d23db5a210 0x55d23db69990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f682c9d6b80 1 -- 172.21.15.70:0/2123929532 >> 172.21.15.70:0/2123929532 conn(0x55d23da87080 msgr2=0x55d23db64030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:46.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f67faffd700 0 client.19071 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:46.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f682c9d6b80 1 -- 172.21.15.70:0/2123929532 shutdown_connections 2024-06-04T05:10:46.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f682c9d6b80 1 -- 172.21.15.70:0/2123929532 wait complete. 2024-06-04T05:10:46.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f682c9d6b80 20 client.19071 trim_cache size 0 max 16384 2024-06-04T05:10:46.766 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:10:46.766 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:10:46.766 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:10:46.767 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:10:46.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f6822141700 1 -- 172.21.15.70:0/3036562501 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68040b2a70 msgr2=0x7f68040b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:46.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f6822141700 1 --2- 172.21.15.70:0/3036562501 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68040b2a70 0x7f68040b4f30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f680c000c00 tx=0x7f680c005fb0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f6822141700 1 -- 172.21.15.70:0/3036562501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c1037f0 msgr2=0x7f681c075840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:46.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f6822141700 1 --2- 172.21.15.70:0/3036562501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c1037f0 0x7f681c075840 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f681000c9b0 tx=0x7f681000cd70 comp rx=0 tx=0).stop 2024-06-04T05:10:46.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f6822141700 1 -- 172.21.15.70:0/3036562501 shutdown_connections 2024-06-04T05:10:46.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f6822141700 1 --2- 172.21.15.70:0/3036562501 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68040b2a70 0x7f68040b4f30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f6822141700 1 --2- 172.21.15.70:0/3036562501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f681c1088f0 0x7f681c0762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f6822141700 1 --2- 172.21.15.70:0/3036562501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c1037f0 0x7f681c075840 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f6822141700 1 --2- 172.21.15.70:0/3036562501 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f681c1041a0 0x7f681c075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:46.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.757+0000 7f6822141700 1 -- 172.21.15.70:0/3036562501 >> 172.21.15.70:0/3036562501 conn(0x7f681c07a6a0 msgr2=0x7f681c101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:46.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.758+0000 7f6822141700 1 -- 172.21.15.70:0/3036562501 shutdown_connections 2024-06-04T05:10:46.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:46.758+0000 7f6822141700 1 -- 172.21.15.70:0/3036562501 wait complete. 2024-06-04T05:10:47.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:47 smithi070 ceph-mon[32631]: pgmap v423: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1023 KiB/s wr, 104 op/s 2024-06-04T05:10:47.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:47 smithi098 ceph-mon[35569]: pgmap v423: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1023 KiB/s wr, 104 op/s 2024-06-04T05:10:47.545 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:10:47.546 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:10:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:47 smithi112 ceph-mon[36453]: pgmap v423: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1023 KiB/s wr, 104 op/s 2024-06-04T05:10:47.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 213159 1.56 MB/sec execute 580 sec latency 4786.053 ms 2024-06-04T05:10:47.806 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:10:48.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 213435 1.56 MB/sec execute 581 sec latency 14.733 ms 2024-06-04T05:10:49.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.276+0000 7f25accda640 1 -- 172.21.15.70:0/721272464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a8101980 msgr2=0x7f25a8101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:49.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.276+0000 7f25accda640 1 --2- 172.21.15.70:0/721272464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a8101980 0x7f25a8101e00 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f259c009930 tx=0x7f259c02a420 comp rx=0 tx=0).stop 2024-06-04T05:10:49.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.276+0000 7f25accda640 1 -- 172.21.15.70:0/721272464 shutdown_connections 2024-06-04T05:10:49.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.276+0000 7f25accda640 1 --2- 172.21.15.70:0/721272464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25a8102340 0x7f25a810e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.276+0000 7f25accda640 1 --2- 172.21.15.70:0/721272464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a8101980 0x7f25a8101e00 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.276+0000 7f25accda640 1 --2- 172.21.15.70:0/721272464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a8107940 0x7f25a8107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.276+0000 7f25accda640 1 -- 172.21.15.70:0/721272464 >> 172.21.15.70:0/721272464 conn(0x7f25a8078ad0 msgr2=0x7f25a8078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:49.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.276+0000 7f25accda640 1 -- 172.21.15.70:0/721272464 shutdown_connections 2024-06-04T05:10:49.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.276+0000 7f25accda640 1 -- 172.21.15.70:0/721272464 wait complete. 2024-06-04T05:10:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.277+0000 7f25accda640 1 Processor -- start 2024-06-04T05:10:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.277+0000 7f25accda640 1 -- start start 2024-06-04T05:10:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.278+0000 7f25accda640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a8101980 0x7f25a8198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.278+0000 7f25accda640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25a8102340 0x7f25a8198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.278+0000 7f25accda640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a8107940 0x7f25a819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.278+0000 7f25accda640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f25a8111ab0 con 0x7f25a8107940 2024-06-04T05:10:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.278+0000 7f25accda640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f25a8111930 con 0x7f25a8101980 2024-06-04T05:10:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.278+0000 7f25accda640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f25a8111c30 con 0x7f25a8102340 2024-06-04T05:10:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.278+0000 7f25a659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a8101980 0x7f25a8198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.278+0000 7f25a659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a8101980 0x7f25a8198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:44456/0 (socket says 172.21.15.70:44456) 2024-06-04T05:10:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.278+0000 7f25a5d9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25a8102340 0x7f25a8198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.278+0000 7f25a659c640 1 -- 172.21.15.70:0/2233200171 learned_addr learned my addr 172.21.15.70:0/2233200171 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.278+0000 7f25a6d9d640 1 --2- 172.21.15.70:0/2233200171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a8107940 0x7f25a819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.278+0000 7f25a659c640 1 -- 172.21.15.70:0/2233200171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25a8102340 msgr2=0x7f25a8198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.278+0000 7f25a659c640 1 --2- 172.21.15.70:0/2233200171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25a8102340 0x7f25a8198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.278+0000 7f25a659c640 1 -- 172.21.15.70:0/2233200171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a8107940 msgr2=0x7f25a819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.278+0000 7f25a659c640 1 --2- 172.21.15.70:0/2233200171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a8107940 0x7f25a819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.278+0000 7f25a659c640 1 -- 172.21.15.70:0/2233200171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a819d000 con 0x7f25a8101980 2024-06-04T05:10:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.278+0000 7f25a659c640 1 --2- 172.21.15.70:0/2233200171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a8101980 0x7f25a8198010 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f259800e9f0 tx=0x7f259800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.279+0000 7f25977fe640 1 -- 172.21.15.70:0/2233200171 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259800cd80 con 0x7f25a8101980 2024-06-04T05:10:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.279+0000 7f25977fe640 1 -- 172.21.15.70:0/2233200171 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f25980108c0 con 0x7f25a8101980 2024-06-04T05:10:49.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.279+0000 7f25accda640 1 -- 172.21.15.70:0/2233200171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a819d2f0 con 0x7f25a8101980 2024-06-04T05:10:49.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.279+0000 7f25977fe640 1 -- 172.21.15.70:0/2233200171 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25980187c0 con 0x7f25a8101980 2024-06-04T05:10:49.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.279+0000 7f25accda640 1 -- 172.21.15.70:0/2233200171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a806a100 con 0x7f25a8101980 2024-06-04T05:10:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.282+0000 7f25af9fb740 1 Processor -- start 2024-06-04T05:10:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.282+0000 7f25af9fb740 1 -- start start 2024-06-04T05:10:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.282+0000 7f25af9fb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaeb6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.282+0000 7f25977fe640 1 -- 172.21.15.70:0/2233200171 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f2598010430 con 0x7f25a8101980 2024-06-04T05:10:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.282+0000 7f25af9fb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaeb7140 0x559deaec14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.282+0000 7f25af9fb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaec1a00 0x559deaec3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.282+0000 7f25af9fb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559deae28c30 con 0x559deaeb6820 2024-06-04T05:10:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.282+0000 7f25af9fb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559deae187d0 con 0x559deaeb7140 2024-06-04T05:10:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.282+0000 7f25af9fb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559deae1c810 con 0x559deaec1a00 2024-06-04T05:10:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.282+0000 7f25a6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaeb6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.282+0000 7f25a6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaeb6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44762/0 (socket says 172.21.15.70:44762) 2024-06-04T05:10:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.282+0000 7f25a6d9d640 1 -- 172.21.15.70:0/2719608104 learned_addr learned my addr 172.21.15.70:0/2719608104 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.283+0000 7f25a659c640 1 --2- 172.21.15.70:0/2719608104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaeb7140 0x559deaec14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.283+0000 7f25a5d9b640 1 --2- 172.21.15.70:0/2719608104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaec1a00 0x559deaec3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.283+0000 7f25a6d9d640 1 -- 172.21.15.70:0/2719608104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaec1a00 msgr2=0x559deaec3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.283+0000 7f25a6d9d640 1 --2- 172.21.15.70:0/2719608104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaec1a00 0x559deaec3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.283+0000 7f25a6d9d640 1 -- 172.21.15.70:0/2719608104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaeb7140 msgr2=0x559deaec14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.283+0000 7f25a6d9d640 1 --2- 172.21.15.70:0/2719608104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaeb7140 0x559deaec14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.283+0000 7f25a6d9d640 1 -- 172.21.15.70:0/2719608104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559deab2d210 con 0x559deaeb6820 2024-06-04T05:10:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.283+0000 7f25a5d9b640 1 --2- 172.21.15.70:0/2719608104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaec1a00 0x559deaec3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.283+0000 7f25a6d9d640 1 --2- 172.21.15.70:0/2719608104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaeb6c00 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f25a00039f0 tx=0x7f25a0003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=861a8a5cd87ba956 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.283+0000 7f2594ff9640 1 -- 172.21.15.70:0/2719608104 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a001a470 con 0x559deaeb6820 2024-06-04T05:10:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.283+0000 7f2594ff9640 1 -- 172.21.15.70:0/2719608104 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f25a0015070 con 0x559deaeb6820 2024-06-04T05:10:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.283+0000 7f2594ff9640 1 -- 172.21.15.70:0/2719608104 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0010730 con 0x559deaeb6820 2024-06-04T05:10:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.283+0000 7f25977fe640 1 --2- 172.21.15.70:0/2233200171 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f25900b2a90 0x7f25900b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.284+0000 7f25a659c640 1 --2- 172.21.15.70:0/2233200171 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f25900b2a90 0x7f25900b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.284+0000 7f25977fe640 1 -- 172.21.15.70:0/2233200171 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f25980e28c0 con 0x7f25a8101980 2024-06-04T05:10:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.284+0000 7f25af9fb740 1 -- 172.21.15.70:0/2719608104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 msgr2=0x559deaeb6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.284+0000 7f25af9fb740 1 --2- 172.21.15.70:0/2719608104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaeb6c00 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f25a00039f0 tx=0x7f25a0003eb0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.284+0000 7f25a659c640 1 --2- 172.21.15.70:0/2233200171 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f25900b2a90 0x7f25900b4f50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f25980e0090 tx=0x7f25980e8550 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.284+0000 7f25af9fb740 1 -- 172.21.15.70:0/2719608104 shutdown_connections 2024-06-04T05:10:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.285+0000 7f25af9fb740 1 --2- 172.21.15.70:0/2719608104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaeb7140 0x559deaec14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.285+0000 7f25af9fb740 1 --2- 172.21.15.70:0/2719608104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaec1a00 0x559deaec3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.285+0000 7f25af9fb740 1 --2- 172.21.15.70:0/2719608104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaeb6c00 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.285+0000 7f25af9fb740 1 -- 172.21.15.70:0/2719608104 >> 172.21.15.70:0/2719608104 conn(0x559deaeb2d30 msgr2=0x559deaeb3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.285+0000 7f25af9fb740 1 -- 172.21.15.70:0/2719608104 shutdown_connections 2024-06-04T05:10:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.285+0000 7f25af9fb740 1 -- 172.21.15.70:0/2719608104 wait complete. 2024-06-04T05:10:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.285+0000 7f25af9fb740 1 -- start start 2024-06-04T05:10:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.285+0000 7f25af9fb740 1 Processor -- start 2024-06-04T05:10:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.285+0000 7f25af9fb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaeb6820 0x559deaebe9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.285+0000 7f25af9fb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb7140 0x559deaebef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.285+0000 7f25af9fb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaec1a00 0x559deaebf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.285+0000 7f25af9fb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559deaafe790 con 0x559deaeb7140 2024-06-04T05:10:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.285+0000 7f25af9fb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559deadd4890 con 0x559deaeb6820 2024-06-04T05:10:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.285+0000 7f25af9fb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559deaec4630 con 0x559deaec1a00 2024-06-04T05:10:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.285+0000 7f25af9fb740 10 client.0 _ll_register_callbacks cb 0x559deab2ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f25a5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb7140 0x559deaebef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f25a6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaeb6820 0x559deaebe9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f25a5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb7140 0x559deaebef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44772/0 (socket says 172.21.15.70:44772) 2024-06-04T05:10:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f25a5d9b640 1 -- 172.21.15.70:0/2482346772 learned_addr learned my addr 172.21.15.70:0/2482346772 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f25a6d9d640 1 --2- 172.21.15.70:0/2482346772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaec1a00 0x559deaebf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f25a6d9d640 1 -- 172.21.15.70:0/2482346772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaec1a00 msgr2=0x559deaebf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f25a6d9d640 1 --2- 172.21.15.70:0/2482346772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaec1a00 0x559deaebf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f25a6d9d640 1 -- 172.21.15.70:0/2482346772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb7140 msgr2=0x559deaebef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f25a6d9d640 1 --2- 172.21.15.70:0/2482346772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb7140 0x559deaebef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f25a6d9d640 1 -- 172.21.15.70:0/2482346772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559deabf9620 con 0x559deaeb6820 2024-06-04T05:10:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f25a6d9d640 1 --2- 172.21.15.70:0/2482346772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaeb6820 0x559deaebe9f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f25a0009fd0 tx=0x7f25a00096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac9273fa53b1741e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f258dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:10:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f258dffb640 1 -- 172.21.15.70:0/2482346772 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0028960 con 0x559deaeb6820 2024-06-04T05:10:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f25af9fb740 1 -- 172.21.15.70:0/2482346772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25a0015070 con 0x559deaeb6820 2024-06-04T05:10:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f258dffb640 1 -- 172.21.15.70:0/2482346772 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f25a0028b00 con 0x559deaeb6820 2024-06-04T05:10:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f258dffb640 1 -- 172.21.15.70:0/2482346772 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f25a003a050 con 0x559deaeb6820 2024-06-04T05:10:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f25af9fb740 10 client.27971 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.286+0000 7f25af9fb740 1 -- 172.21.15.70:0/2482346772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559deaed03e0 con 0x559deaeb6820 2024-06-04T05:10:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.287+0000 7f258dffb640 1 -- 172.21.15.70:0/2482346772 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f25a0015070 con 0x559deaeb6820 2024-06-04T05:10:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.287+0000 7f258dffb640 1 -- 172.21.15.70:0/2482346772 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f25a0011440 con 0x559deaeb6820 2024-06-04T05:10:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.287+0000 7f25af9fb740 10 client.27971 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.287+0000 7f25af9fb740 10 client.27971 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:10:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.287+0000 7f25af9fb740 20 client.27971 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.287+0000 7f25af9fb740 1 --2- 172.21.15.70:0/2482346772 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559deaed17e0 0x559deaed3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.287+0000 7f25af9fb740 4 client.27971 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:10:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.287+0000 7f25af9fb740 1 -- 172.21.15.70:0/2482346772 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559deae2ab80 con 0x559deaed17e0 2024-06-04T05:10:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.287+0000 7f25a5d9b640 1 --2- 172.21.15.70:0/2482346772 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559deaed17e0 0x559deaed3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.287+0000 7f25a5d9b640 1 --2- 172.21.15.70:0/2482346772 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559deaed17e0 0x559deaed3bd0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7b324e2a5ab4a2e server_cookie=57f03f31dbca3b41 in_seq=0 out_seq=0 2024-06-04T05:10:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.287+0000 7f258dffb640 10 client.27971 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.288+0000 7f258dffb640 1 -- 172.21.15.70:0/2482346772 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559deae2ab80 con 0x559deaed17e0 2024-06-04T05:10:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.288+0000 7f258dffb640 10 client.27971 handle_command_reply: tid=0 2024-06-04T05:10:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.288+0000 7f25af9fb740 1 client.27971 shutdown 2024-06-04T05:10:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.288+0000 7f25af9fb740 1 -- 172.21.15.70:0/2482346772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaeb6820 msgr2=0x559deaebe9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.288+0000 7f25af9fb740 1 --2- 172.21.15.70:0/2482346772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaeb6820 0x559deaebe9f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f25a0009fd0 tx=0x7f25a00096a0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.289+0000 7f25af9fb740 1 -- 172.21.15.70:0/2482346772 shutdown_connections 2024-06-04T05:10:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.289+0000 7f25af9fb740 1 --2- 172.21.15.70:0/2482346772 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559deaed17e0 0x559deaed3bd0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.289+0000 7f258dffb640 0 client.27971 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.289+0000 7f25af9fb740 1 --2- 172.21.15.70:0/2482346772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaec1a00 0x559deaebf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.289+0000 7f25af9fb740 1 --2- 172.21.15.70:0/2482346772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb7140 0x559deaebef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.289+0000 7f25af9fb740 1 --2- 172.21.15.70:0/2482346772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaeb6820 0x559deaebe9f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.289+0000 7f25af9fb740 1 -- 172.21.15.70:0/2482346772 >> 172.21.15.70:0/2482346772 conn(0x559deaeb2d30 msgr2=0x559deaeb1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.289+0000 7f25af9fb740 1 -- 172.21.15.70:0/2482346772 shutdown_connections 2024-06-04T05:10:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.289+0000 7f25af9fb740 1 -- 172.21.15.70:0/2482346772 wait complete. 2024-06-04T05:10:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.289+0000 7f25af9fb740 20 client.27971 trim_cache size 0 max 16384 2024-06-04T05:10:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.306+0000 7f25af9fb740 1 Processor -- start 2024-06-04T05:10:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.306+0000 7f25af9fb740 1 -- start start 2024-06-04T05:10:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.306+0000 7f25af9fb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaebf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.306+0000 7f25af9fb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaeb7140 0x559deaed9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.306+0000 7f25af9fb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaec1a00 0x559deaed9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.306+0000 7f25af9fb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f25a81ac550 con 0x559deaeb6820 2024-06-04T05:10:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.306+0000 7f25af9fb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f25a81ae1b0 con 0x559deaec1a00 2024-06-04T05:10:49.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.306+0000 7f25af9fb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f25a81acc20 con 0x559deaeb7140 2024-06-04T05:10:49.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.306+0000 7f25a6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaebf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.306+0000 7f25a6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaebf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44778/0 (socket says 172.21.15.70:44778) 2024-06-04T05:10:49.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.306+0000 7f25a6d9d640 1 -- 172.21.15.70:0/311819226 learned_addr learned my addr 172.21.15.70:0/311819226 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:49.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.306+0000 7f25a5d9b640 1 --2- 172.21.15.70:0/311819226 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaeb7140 0x559deaed9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.306+0000 7f25a6d9d640 1 --2- 172.21.15.70:0/311819226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaec1a00 0x559deaed9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.307+0000 7f25a6d9d640 1 -- 172.21.15.70:0/311819226 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaeb7140 msgr2=0x559deaed9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:49.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.307+0000 7f25a6d9d640 1 --2- 172.21.15.70:0/311819226 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaeb7140 0x559deaed9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.307+0000 7f25a6d9d640 1 -- 172.21.15.70:0/311819226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaec1a00 msgr2=0x559deaed9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:49.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.307+0000 7f25a6d9d640 1 --2- 172.21.15.70:0/311819226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaec1a00 0x559deaed9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.307+0000 7f25a6d9d640 1 -- 172.21.15.70:0/311819226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559deaeba590 con 0x559deaeb6820 2024-06-04T05:10:49.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.307+0000 7f25a6d9d640 1 --2- 172.21.15.70:0/311819226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaebf290 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f25a0005aa0 tx=0x7f25a0005a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc61826399ec4f74 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:49.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.307+0000 7f258ffff640 1 -- 172.21.15.70:0/311819226 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0011de0 con 0x559deaeb6820 2024-06-04T05:10:49.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.307+0000 7f258ffff640 1 -- 172.21.15.70:0/311819226 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f25a0015040 con 0x559deaeb6820 2024-06-04T05:10:49.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.307+0000 7f258ffff640 1 -- 172.21.15.70:0/311819226 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a002a370 con 0x559deaeb6820 2024-06-04T05:10:49.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.308+0000 7f25af9fb740 1 -- 172.21.15.70:0/311819226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 msgr2=0x559deaebf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:49.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.308+0000 7f25af9fb740 1 --2- 172.21.15.70:0/311819226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaebf290 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f25a0005aa0 tx=0x7f25a0005a10 comp rx=0 tx=0).stop 2024-06-04T05:10:49.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.308+0000 7f25af9fb740 1 -- 172.21.15.70:0/311819226 shutdown_connections 2024-06-04T05:10:49.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.308+0000 7f25af9fb740 1 --2- 172.21.15.70:0/311819226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaec1a00 0x559deaed9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.308+0000 7f25af9fb740 1 --2- 172.21.15.70:0/311819226 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaeb7140 0x559deaed9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.308+0000 7f25af9fb740 1 --2- 172.21.15.70:0/311819226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaebf290 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.308+0000 7f25af9fb740 1 -- 172.21.15.70:0/311819226 >> 172.21.15.70:0/311819226 conn(0x559deaeb2d30 msgr2=0x559deaeb8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:49.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.308+0000 7f25af9fb740 1 -- 172.21.15.70:0/311819226 shutdown_connections 2024-06-04T05:10:49.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.308+0000 7f25af9fb740 1 -- 172.21.15.70:0/311819226 wait complete. 2024-06-04T05:10:49.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.308+0000 7f25af9fb740 1 -- start start 2024-06-04T05:10:49.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.308+0000 7f25af9fb740 1 Processor -- start 2024-06-04T05:10:49.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.308+0000 7f25af9fb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaec9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.308+0000 7f25af9fb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaeb7140 0x559deaec5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.308+0000 7f25af9fb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaec1a00 0x559deaec5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25af9fb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f25a81ada80 con 0x559deaeb6820 2024-06-04T05:10:49.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25af9fb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f25a81ad350 con 0x559deaec1a00 2024-06-04T05:10:49.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25af9fb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f25a81aa550 con 0x559deaeb7140 2024-06-04T05:10:49.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25af9fb740 10 client.0 _ll_register_callbacks cb 0x7f25980e8420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:49.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25a6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaec9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25a6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaec9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44788/0 (socket says 172.21.15.70:44788) 2024-06-04T05:10:49.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25a6d9d640 1 -- 172.21.15.70:0/2088922972 learned_addr learned my addr 172.21.15.70:0/2088922972 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:49.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25a5d9b640 1 --2- 172.21.15.70:0/2088922972 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaeb7140 0x559deaec5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25a6d9d640 1 --2- 172.21.15.70:0/2088922972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaec1a00 0x559deaec5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25a6d9d640 1 -- 172.21.15.70:0/2088922972 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaeb7140 msgr2=0x559deaec5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:49.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25a6d9d640 1 --2- 172.21.15.70:0/2088922972 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaeb7140 0x559deaec5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25a6d9d640 1 -- 172.21.15.70:0/2088922972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaec1a00 msgr2=0x559deaec5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:49.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25a6d9d640 1 --2- 172.21.15.70:0/2088922972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaec1a00 0x559deaec5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25a6d9d640 1 -- 172.21.15.70:0/2088922972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559deaec6040 con 0x559deaeb6820 2024-06-04T05:10:49.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25a6d9d640 1 --2- 172.21.15.70:0/2088922972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaec9f90 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f25a0002230 tx=0x7f25a0009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff26dc86a451db8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:49.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f258dffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:49.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f258dffb640 1 -- 172.21.15.70:0/2088922972 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0051060 con 0x559deaeb6820 2024-06-04T05:10:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f258dffb640 1 -- 172.21.15.70:0/2088922972 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f25a0011440 con 0x559deaeb6820 2024-06-04T05:10:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25af9fb740 1 -- 172.21.15.70:0/2088922972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25a0015040 con 0x559deaeb6820 2024-06-04T05:10:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f258dffb640 1 -- 172.21.15.70:0/2088922972 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f25a0011660 con 0x559deaeb6820 2024-06-04T05:10:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25af9fb740 10 client.19089 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.309+0000 7f25af9fb740 1 -- 172.21.15.70:0/2088922972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559deaf3c110 con 0x559deaeb6820 2024-06-04T05:10:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.310+0000 7f258dffb640 1 -- 172.21.15.70:0/2088922972 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f25a0015040 con 0x559deaeb6820 2024-06-04T05:10:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.310+0000 7f258dffb640 1 -- 172.21.15.70:0/2088922972 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f25a002a560 con 0x559deaeb6820 2024-06-04T05:10:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.310+0000 7f25af9fb740 10 client.19089 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.310+0000 7f25af9fb740 10 client.19089 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:10:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.310+0000 7f25af9fb740 20 client.19089 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.310+0000 7f25af9fb740 1 --2- 172.21.15.70:0/2088922972 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559deaed17e0 0x559deaf3e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.310+0000 7f25af9fb740 4 client.19089 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:10:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.310+0000 7f25af9fb740 1 -- 172.21.15.70:0/2088922972 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559deae24360 con 0x559deaed17e0 2024-06-04T05:10:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.310+0000 7f25a5d9b640 1 --2- 172.21.15.70:0/2088922972 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559deaed17e0 0x559deaf3e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.310+0000 7f258dffb640 10 client.19089 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:49.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.310+0000 7f25a5d9b640 1 --2- 172.21.15.70:0/2088922972 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559deaed17e0 0x559deaf3e700 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f5f729f89407b17 server_cookie=65bae2b28a088fb1 in_seq=0 out_seq=0 2024-06-04T05:10:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.334+0000 7f258dffb640 1 -- 172.21.15.70:0/2088922972 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x559deae24360 con 0x559deaed17e0 2024-06-04T05:10:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.334+0000 7f258dffb640 10 client.19089 handle_command_reply: tid=0 2024-06-04T05:10:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.334+0000 7f25af9fb740 1 client.19089 shutdown 2024-06-04T05:10:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.334+0000 7f25af9fb740 1 -- 172.21.15.70:0/2088922972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 msgr2=0x559deaec9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.334+0000 7f25af9fb740 1 --2- 172.21.15.70:0/2088922972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaec9f90 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f25a0002230 tx=0x7f25a0009eb0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.334+0000 7f25af9fb740 1 -- 172.21.15.70:0/2088922972 shutdown_connections 2024-06-04T05:10:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.334+0000 7f25af9fb740 1 --2- 172.21.15.70:0/2088922972 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559deaed17e0 0x559deaf3e700 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.334+0000 7f25af9fb740 1 --2- 172.21.15.70:0/2088922972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559deaec1a00 0x559deaec5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.334+0000 7f25af9fb740 1 --2- 172.21.15.70:0/2088922972 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559deaeb7140 0x559deaec5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.334+0000 7f258dffb640 0 client.19089 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:10:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.334+0000 7f25af9fb740 1 --2- 172.21.15.70:0/2088922972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deaeb6820 0x559deaec9f90 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.334+0000 7f25af9fb740 1 -- 172.21.15.70:0/2088922972 >> 172.21.15.70:0/2088922972 conn(0x559deaeb2d30 msgr2=0x559deaeb1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.334+0000 7f25af9fb740 1 -- 172.21.15.70:0/2088922972 shutdown_connections 2024-06-04T05:10:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.334+0000 7f25af9fb740 1 -- 172.21.15.70:0/2088922972 wait complete. 2024-06-04T05:10:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.335+0000 7f25af9fb740 20 client.19089 trim_cache size 0 max 16384 2024-06-04T05:10:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:10:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:10:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:10:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.335+0000 7f25accda640 1 -- 172.21.15.70:0/2233200171 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f25900b2a90 msgr2=0x7f25900b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.335+0000 7f25accda640 1 --2- 172.21.15.70:0/2233200171 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f25900b2a90 0x7f25900b4f50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f25980e0090 tx=0x7f25980e8550 comp rx=0 tx=0).stop 2024-06-04T05:10:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.335+0000 7f25accda640 1 -- 172.21.15.70:0/2233200171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a8101980 msgr2=0x7f25a8198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.335+0000 7f25accda640 1 --2- 172.21.15.70:0/2233200171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a8101980 0x7f25a8198010 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f259800e9f0 tx=0x7f259800eeb0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.335+0000 7f25accda640 1 -- 172.21.15.70:0/2233200171 shutdown_connections 2024-06-04T05:10:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.335+0000 7f25accda640 1 --2- 172.21.15.70:0/2233200171 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f25900b2a90 0x7f25900b4f50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.335+0000 7f25accda640 1 --2- 172.21.15.70:0/2233200171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a8107940 0x7f25a819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.335+0000 7f25accda640 1 --2- 172.21.15.70:0/2233200171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25a8102340 0x7f25a8198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.335+0000 7f25accda640 1 --2- 172.21.15.70:0/2233200171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a8101980 0x7f25a8198010 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.335+0000 7f25accda640 1 -- 172.21.15.70:0/2233200171 >> 172.21.15.70:0/2233200171 conn(0x7f25a8078ad0 msgr2=0x7f25a81045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.335+0000 7f25accda640 1 -- 172.21.15.70:0/2233200171 shutdown_connections 2024-06-04T05:10:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:10:49.335+0000 7f25accda640 1 -- 172.21.15.70:0/2233200171 wait complete. 2024-06-04T05:10:49.353 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:49 smithi070 ceph-mon[32631]: pgmap v424: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 622 KiB/s wr, 65 op/s 2024-06-04T05:10:49.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:49 smithi098 ceph-mon[35569]: pgmap v424: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 622 KiB/s wr, 65 op/s 2024-06-04T05:10:49.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.608+0000 7f64cece6700 1 -- 172.21.15.70:0/2629536937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64c8108520 msgr2=0x7f64c8108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:49.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.608+0000 7f64cece6700 1 --2- 172.21.15.70:0/2629536937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64c8108520 0x7f64c8108900 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f64bc0098f0 tx=0x7f64bc009c00 comp rx=0 tx=0).stop 2024-06-04T05:10:49.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.608+0000 7f64cece6700 1 -- 172.21.15.70:0/2629536937 shutdown_connections 2024-06-04T05:10:49.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.608+0000 7f64cece6700 1 --2- 172.21.15.70:0/2629536937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64c8102e00 0x7f64c810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.608+0000 7f64cece6700 1 --2- 172.21.15.70:0/2629536937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64c8108520 0x7f64c8108900 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.608+0000 7f64cece6700 1 --2- 172.21.15.70:0/2629536937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64c8102440 0x7f64c81028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.608+0000 7f64cece6700 1 -- 172.21.15.70:0/2629536937 >> 172.21.15.70:0/2629536937 conn(0x7f64c80793f0 msgr2=0x7f64c8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:49.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.609+0000 7f64cece6700 1 -- 172.21.15.70:0/2629536937 shutdown_connections 2024-06-04T05:10:49.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.609+0000 7f64cece6700 1 -- 172.21.15.70:0/2629536937 wait complete. 2024-06-04T05:10:49.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.609+0000 7f64cece6700 1 Processor -- start 2024-06-04T05:10:49.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.609+0000 7f64cece6700 1 -- start start 2024-06-04T05:10:49.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64cece6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64c8102440 0x7f64c819be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64cece6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64c8102e00 0x7f64c819c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64cece6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64c8108520 0x7f64c81a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64cca82700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64c8102440 0x7f64c819be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64cca82700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64c8102440 0x7f64c819be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44812/0 (socket says 172.21.15.70:44812) 2024-06-04T05:10:49.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64cca82700 1 -- 172.21.15.70:0/512840194 learned_addr learned my addr 172.21.15.70:0/512840194 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:49.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64c7fff700 1 --2- 172.21.15.70:0/512840194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64c8102e00 0x7f64c819c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64cece6700 1 -- 172.21.15.70:0/512840194 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f64c8115810 con 0x7f64c8102440 2024-06-04T05:10:49.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64cece6700 1 -- 172.21.15.70:0/512840194 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f64c8115690 con 0x7f64c8102e00 2024-06-04T05:10:49.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64cd283700 1 --2- 172.21.15.70:0/512840194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64c8108520 0x7f64c81a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64cece6700 1 -- 172.21.15.70:0/512840194 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f64c8115990 con 0x7f64c8108520 2024-06-04T05:10:49.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64cd283700 1 -- 172.21.15.70:0/512840194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64c8102e00 msgr2=0x7f64c819c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:49.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64cd283700 1 --2- 172.21.15.70:0/512840194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64c8102e00 0x7f64c819c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64cd283700 1 -- 172.21.15.70:0/512840194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64c8102440 msgr2=0x7f64c819be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:49.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64cd283700 1 --2- 172.21.15.70:0/512840194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64c8102440 0x7f64c819be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64cd283700 1 -- 172.21.15.70:0/512840194 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64c81a0e20 con 0x7f64c8108520 2024-06-04T05:10:49.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64c7fff700 1 --2- 172.21.15.70:0/512840194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64c8102e00 0x7f64c819c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:49.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64cca82700 1 --2- 172.21.15.70:0/512840194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64c8102440 0x7f64c819be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:49.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.610+0000 7f64cd283700 1 --2- 172.21.15.70:0/512840194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64c8108520 0x7f64c81a06e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f64c0009cd0 tx=0x7f64c000c620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:49.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.611+0000 7f64c5ffb700 1 -- 172.21.15.70:0/512840194 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c000cd10 con 0x7f64c8108520 2024-06-04T05:10:49.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.611+0000 7f64c5ffb700 1 -- 172.21.15.70:0/512840194 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f64c0003a00 con 0x7f64c8108520 2024-06-04T05:10:49.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.611+0000 7f64cece6700 1 -- 172.21.15.70:0/512840194 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64c81a10b0 con 0x7f64c8108520 2024-06-04T05:10:49.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.611+0000 7f64c5ffb700 1 -- 172.21.15.70:0/512840194 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c00043e0 con 0x7f64c8108520 2024-06-04T05:10:49.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.611+0000 7f64cece6700 1 -- 172.21.15.70:0/512840194 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64c81a8990 con 0x7f64c8108520 2024-06-04T05:10:49.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.612+0000 7f64cece6700 1 -- 172.21.15.70:0/512840194 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64c810a9b0 con 0x7f64c8108520 2024-06-04T05:10:49.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.613+0000 7f64c5ffb700 1 -- 172.21.15.70:0/512840194 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f64c001a020 con 0x7f64c8108520 2024-06-04T05:10:49.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.614+0000 7f64c5ffb700 1 --2- 172.21.15.70:0/512840194 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f64b80b29e0 0x7f64b80b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:49.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.614+0000 7f64cca82700 1 --2- 172.21.15.70:0/512840194 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f64b80b29e0 0x7f64b80b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:49.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.614+0000 7f64c5ffb700 1 -- 172.21.15.70:0/512840194 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f64c00e2460 con 0x7f64c8108520 2024-06-04T05:10:49.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.614+0000 7f64cca82700 1 --2- 172.21.15.70:0/512840194 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f64b80b29e0 0x7f64b80b4ea0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f64bc009fd0 tx=0x7f64bc019040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:49.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.618+0000 7f64c5ffb700 1 -- 172.21.15.70:0/512840194 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f64c0010040 con 0x7f64c8108520 2024-06-04T05:10:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:49 smithi112 ceph-mon[36453]: pgmap v424: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 622 KiB/s wr, 65 op/s 2024-06-04T05:10:49.743 INFO:tasks.workunit.client.0.smithi070.stdout: 1 213983 1.56 MB/sec execute 582 sec latency 24.222 ms 2024-06-04T05:10:49.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.920+0000 7f64cece6700 1 -- 172.21.15.70:0/512840194 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64c806a230 con 0x7f64c8108520 2024-06-04T05:10:49.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.921+0000 7f64c5ffb700 1 -- 172.21.15.70:0/512840194 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11938 (secure 0 0 0) 0x7f64c00a9db0 con 0x7f64c8108520 2024-06-04T05:10:49.923 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:10:49.923 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:09:17.032462+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24436,24410,24440,24448,24442]},"id":1}]} 2024-06-04T05:10:49.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.924+0000 7f64cece6700 1 -- 172.21.15.70:0/512840194 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f64b80b29e0 msgr2=0x7f64b80b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:49.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.924+0000 7f64cece6700 1 --2- 172.21.15.70:0/512840194 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f64b80b29e0 0x7f64b80b4ea0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f64bc009fd0 tx=0x7f64bc019040 comp rx=0 tx=0).stop 2024-06-04T05:10:49.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.924+0000 7f64cece6700 1 -- 172.21.15.70:0/512840194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64c8108520 msgr2=0x7f64c81a06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:49.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.924+0000 7f64cece6700 1 --2- 172.21.15.70:0/512840194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64c8108520 0x7f64c81a06e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f64c0009cd0 tx=0x7f64c000c620 comp rx=0 tx=0).stop 2024-06-04T05:10:49.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.924+0000 7f64cece6700 1 -- 172.21.15.70:0/512840194 shutdown_connections 2024-06-04T05:10:49.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.924+0000 7f64cece6700 1 --2- 172.21.15.70:0/512840194 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f64b80b29e0 0x7f64b80b4ea0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.924+0000 7f64cece6700 1 --2- 172.21.15.70:0/512840194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64c8108520 0x7f64c81a06e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.924+0000 7f64cece6700 1 --2- 172.21.15.70:0/512840194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64c8102440 0x7f64c819be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.924+0000 7f64cece6700 1 --2- 172.21.15.70:0/512840194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64c8102e00 0x7f64c819c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:49.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.924+0000 7f64cece6700 1 -- 172.21.15.70:0/512840194 >> 172.21.15.70:0/512840194 conn(0x7f64c80793f0 msgr2=0x7f64c81012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:49.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.924+0000 7f64cece6700 1 -- 172.21.15.70:0/512840194 shutdown_connections 2024-06-04T05:10:49.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:49.924+0000 7f64cece6700 1 -- 172.21.15.70:0/512840194 wait complete. 2024-06-04T05:10:49.927 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 38 2024-06-04T05:10:50.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:50 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/512840194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:10:50.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:50 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/512840194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:10:50.684 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:10:50.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:50 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/512840194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:10:50.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 214506 1.56 MB/sec execute 583 sec latency 23.540 ms 2024-06-04T05:10:50.950 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:10:51.332 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:51 smithi070 ceph-mon[32631]: pgmap v425: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 622 KiB/s wr, 65 op/s 2024-06-04T05:10:51.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:51 smithi098 ceph-mon[35569]: pgmap v425: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 622 KiB/s wr, 65 op/s 2024-06-04T05:10:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:51 smithi112 ceph-mon[36453]: pgmap v425: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 622 KiB/s wr, 65 op/s 2024-06-04T05:10:51.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 215179 1.56 MB/sec execute 584 sec latency 20.238 ms 2024-06-04T05:10:52.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 215765 1.57 MB/sec execute 585 sec latency 23.867 ms 2024-06-04T05:10:52.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.806+0000 7f3565e9a700 1 -- 172.21.15.70:0/2022510044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3560102e30 msgr2=0x7f356010f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:52.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.806+0000 7f3565e9a700 1 --2- 172.21.15.70:0/2022510044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3560102e30 0x7f356010f320 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3558009f90 tx=0x7f355800b330 comp rx=0 tx=0).stop 2024-06-04T05:10:52.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.806+0000 7f3565e9a700 1 -- 172.21.15.70:0/2022510044 shutdown_connections 2024-06-04T05:10:52.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.806+0000 7f3565e9a700 1 --2- 172.21.15.70:0/2022510044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3560102e30 0x7f356010f320 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.806+0000 7f3565e9a700 1 --2- 172.21.15.70:0/2022510044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3560108550 0x7f3560108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.806+0000 7f3565e9a700 1 --2- 172.21.15.70:0/2022510044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3560102470 0x7f35601028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.806+0000 7f3565e9a700 1 -- 172.21.15.70:0/2022510044 >> 172.21.15.70:0/2022510044 conn(0x7f3560079480 msgr2=0x7f3560079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:52.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.806+0000 7f3565e9a700 1 -- 172.21.15.70:0/2022510044 shutdown_connections 2024-06-04T05:10:52.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.806+0000 7f3565e9a700 1 -- 172.21.15.70:0/2022510044 wait complete. 2024-06-04T05:10:52.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.806+0000 7f3565e9a700 1 Processor -- start 2024-06-04T05:10:52.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.807+0000 7f3565e9a700 1 -- start start 2024-06-04T05:10:52.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.807+0000 7f3565e9a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3560102470 0x7f356019f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.807+0000 7f3565e9a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3560102e30 0x7f356019fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.807+0000 7f3565e9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3560108550 0x7f3560199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.807+0000 7f3565e9a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35601124f0 con 0x7f3560102e30 2024-06-04T05:10:52.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.807+0000 7f355effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3560102e30 0x7f356019fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.807+0000 7f355f7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3560102470 0x7f356019f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.807+0000 7f355f7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3560102470 0x7f356019f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:35286/0 (socket says 172.21.15.70:35286) 2024-06-04T05:10:52.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.807+0000 7f355effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3560102e30 0x7f356019fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44828/0 (socket says 172.21.15.70:44828) 2024-06-04T05:10:52.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.807+0000 7f355f7fe700 1 -- 172.21.15.70:0/1896781269 learned_addr learned my addr 172.21.15.70:0/1896781269 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:52.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.807+0000 7f3565e9a700 1 -- 172.21.15.70:0/1896781269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3560112370 con 0x7f3560108550 2024-06-04T05:10:52.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.807+0000 7f3565e9a700 1 -- 172.21.15.70:0/1896781269 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3560112670 con 0x7f3560102470 2024-06-04T05:10:52.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.808+0000 7f355ffff700 1 --2- 172.21.15.70:0/1896781269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3560108550 0x7f3560199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.808+0000 7f355f7fe700 1 -- 172.21.15.70:0/1896781269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3560108550 msgr2=0x7f3560199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:52.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.808+0000 7f355f7fe700 1 --2- 172.21.15.70:0/1896781269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3560108550 0x7f3560199900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.808+0000 7f355f7fe700 1 -- 172.21.15.70:0/1896781269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3560102e30 msgr2=0x7f356019fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:52.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.808+0000 7f355f7fe700 1 --2- 172.21.15.70:0/1896781269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3560102e30 0x7f356019fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.808+0000 7f355f7fe700 1 -- 172.21.15.70:0/1896781269 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f356019a060 con 0x7f3560102470 2024-06-04T05:10:52.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.808+0000 7f355f7fe700 1 --2- 172.21.15.70:0/1896781269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3560102470 0x7f356019f740 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f355000d980 tx=0x7f355000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:52.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.808+0000 7f355cff9700 1 -- 172.21.15.70:0/1896781269 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3550017070 con 0x7f3560102470 2024-06-04T05:10:52.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.808+0000 7f355cff9700 1 -- 172.21.15.70:0/1896781269 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3550005020 con 0x7f3560102470 2024-06-04T05:10:52.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.808+0000 7f3565e9a700 1 -- 172.21.15.70:0/1896781269 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f356019a350 con 0x7f3560102470 2024-06-04T05:10:52.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.808+0000 7f355cff9700 1 -- 172.21.15.70:0/1896781269 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355000f980 con 0x7f3560102470 2024-06-04T05:10:52.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.808+0000 7f3565e9a700 1 -- 172.21.15.70:0/1896781269 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35601a6610 con 0x7f3560102470 2024-06-04T05:10:52.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.811+0000 7f355cff9700 1 -- 172.21.15.70:0/1896781269 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3550004a30 con 0x7f3560102470 2024-06-04T05:10:52.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.811+0000 7f355cff9700 1 --2- 172.21.15.70:0/1896781269 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35480b2a70 0x7f35480b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.812+0000 7f355effd700 1 --2- 172.21.15.70:0/1896781269 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35480b2a70 0x7f35480b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.812+0000 7f357072fb80 1 Processor -- start 2024-06-04T05:10:52.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.812+0000 7f355cff9700 1 -- 172.21.15.70:0/1896781269 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f35500e23f0 con 0x7f3560102470 2024-06-04T05:10:52.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.812+0000 7f357072fb80 1 -- start start 2024-06-04T05:10:52.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.812+0000 7f357072fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd2c70 0x561fc5dd3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.812+0000 7f357072fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd3620 0x561fc5dd6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.812+0000 7f357072fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd7370 0x561fc5dd77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.812+0000 7f357072fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561fc5d2dc90 con 0x561fc5dd7370 2024-06-04T05:10:52.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.812+0000 7f357072fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561fc5d1d830 con 0x561fc5dd3620 2024-06-04T05:10:52.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.812+0000 7f357072fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561fc5d21870 con 0x561fc5dd2c70 2024-06-04T05:10:52.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.812+0000 7f355effd700 1 --2- 172.21.15.70:0/1896781269 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35480b2a70 0x7f35480b4f30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3554009750 tx=0x7f3554009310 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:52.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.812+0000 7f355ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd7370 0x561fc5dd77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.812+0000 7f355ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd7370 0x561fc5dd77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44832/0 (socket says 172.21.15.70:44832) 2024-06-04T05:10:52.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.812+0000 7f355ffff700 1 -- 172.21.15.70:0/757892133 learned_addr learned my addr 172.21.15.70:0/757892133 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:52.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.812+0000 7f355f7fe700 1 --2- 172.21.15.70:0/757892133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd2c70 0x561fc5dd3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.812+0000 7f355effd700 1 --2- 172.21.15.70:0/757892133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd3620 0x561fc5dd6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.813+0000 7f355f7fe700 1 -- 172.21.15.70:0/757892133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd3620 msgr2=0x561fc5dd6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:52.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.813+0000 7f355f7fe700 1 --2- 172.21.15.70:0/757892133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd3620 0x561fc5dd6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.813+0000 7f355f7fe700 1 -- 172.21.15.70:0/757892133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd7370 msgr2=0x561fc5dd77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:52.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.813+0000 7f355f7fe700 1 --2- 172.21.15.70:0/757892133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd7370 0x561fc5dd77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.813+0000 7f355f7fe700 1 -- 172.21.15.70:0/757892133 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fc5c6d8d0 con 0x561fc5dd2c70 2024-06-04T05:10:52.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.813+0000 7f355ffff700 1 --2- 172.21.15.70:0/757892133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd7370 0x561fc5dd77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:52.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.813+0000 7f355f7fe700 1 --2- 172.21.15.70:0/757892133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd2c70 0x561fc5dd3050 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f35500dfab0 tx=0x7f35500a9150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4aaf147f2e50172 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:52.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.813+0000 7f3545ffb700 1 -- 172.21.15.70:0/757892133 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35500a9640 con 0x561fc5dd2c70 2024-06-04T05:10:52.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.813+0000 7f3545ffb700 1 -- 172.21.15.70:0/757892133 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35500ad9f0 con 0x561fc5dd2c70 2024-06-04T05:10:52.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.813+0000 7f3545ffb700 1 -- 172.21.15.70:0/757892133 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35500a9b00 con 0x561fc5dd2c70 2024-06-04T05:10:52.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.814+0000 7f357072fb80 1 -- 172.21.15.70:0/757892133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd2c70 msgr2=0x561fc5dd3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:52.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.814+0000 7f357072fb80 1 --2- 172.21.15.70:0/757892133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd2c70 0x561fc5dd3050 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f35500dfab0 tx=0x7f35500a9150 comp rx=0 tx=0).stop 2024-06-04T05:10:52.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.814+0000 7f357072fb80 1 -- 172.21.15.70:0/757892133 shutdown_connections 2024-06-04T05:10:52.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.814+0000 7f357072fb80 1 --2- 172.21.15.70:0/757892133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd7370 0x561fc5dd77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.814+0000 7f357072fb80 1 --2- 172.21.15.70:0/757892133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd2c70 0x561fc5dd3050 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.814+0000 7f357072fb80 1 --2- 172.21.15.70:0/757892133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd3620 0x561fc5dd6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.814+0000 7f357072fb80 1 -- 172.21.15.70:0/757892133 >> 172.21.15.70:0/757892133 conn(0x561fc5cfc080 msgr2=0x561fc5dc8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:52.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.814+0000 7f357072fb80 1 -- 172.21.15.70:0/757892133 shutdown_connections 2024-06-04T05:10:52.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.814+0000 7f357072fb80 1 -- 172.21.15.70:0/757892133 wait complete. 2024-06-04T05:10:52.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.815+0000 7f357072fb80 1 -- start start 2024-06-04T05:10:52.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.815+0000 7f357072fb80 1 Processor -- start 2024-06-04T05:10:52.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.815+0000 7f357072fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd2c70 0x561fc5dde820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.815+0000 7f357072fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd3620 0x561fc5ddbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.815+0000 7f357072fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd7370 0x561fc5ddc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.815+0000 7f357072fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561fc5c23fa0 con 0x561fc5dd3620 2024-06-04T05:10:52.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.815+0000 7f357072fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561fc5cbf7c0 con 0x561fc5dd2c70 2024-06-04T05:10:52.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.815+0000 7f357072fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561fc5ddf530 con 0x561fc5dd7370 2024-06-04T05:10:52.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.815+0000 7f357072fb80 10 client.0 _ll_register_callbacks cb 0x561fc5cb8920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:52.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.815+0000 7f355effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd3620 0x561fc5ddbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.815+0000 7f355ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd7370 0x561fc5ddc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.815+0000 7f355effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd3620 0x561fc5ddbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44846/0 (socket says 172.21.15.70:44846) 2024-06-04T05:10:52.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.815+0000 7f355effd700 1 -- 172.21.15.70:0/1789573997 learned_addr learned my addr 172.21.15.70:0/1789573997 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:52.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.815+0000 7f355f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd2c70 0x561fc5dde820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.815+0000 7f355ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd7370 0x561fc5ddc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:35292/0 (socket says 172.21.15.70:35292) 2024-06-04T05:10:52.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.816+0000 7f355effd700 1 -- 172.21.15.70:0/1789573997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd7370 msgr2=0x561fc5ddc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:52.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.816+0000 7f355effd700 1 --2- 172.21.15.70:0/1789573997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd7370 0x561fc5ddc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.816+0000 7f355effd700 1 -- 172.21.15.70:0/1789573997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd2c70 msgr2=0x561fc5dde820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:52.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.816+0000 7f355effd700 1 --2- 172.21.15.70:0/1789573997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd2c70 0x561fc5dde820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.816+0000 7f355effd700 1 -- 172.21.15.70:0/1789573997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fc5cbc690 con 0x561fc5dd3620 2024-06-04T05:10:52.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.816+0000 7f355f7fe700 1 --2- 172.21.15.70:0/1789573997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd2c70 0x561fc5dde820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:52.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.816+0000 7f355effd700 1 --2- 172.21.15.70:0/1789573997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd3620 0x561fc5ddbb20 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f355401dc10 tx=0x7f35540047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=602d6fa68d462487 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:52.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.816+0000 7f353effd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:52.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.816+0000 7f353effd700 1 -- 172.21.15.70:0/1789573997 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355402b060 con 0x561fc5dd3620 2024-06-04T05:10:52.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.816+0000 7f353effd700 1 -- 172.21.15.70:0/1789573997 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f355401c9f0 con 0x561fc5dd3620 2024-06-04T05:10:52.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.816+0000 7f357072fb80 1 -- 172.21.15.70:0/1789573997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35500ad9f0 con 0x561fc5dd3620 2024-06-04T05:10:52.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.816+0000 7f353effd700 1 -- 172.21.15.70:0/1789573997 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f355401cbf0 con 0x561fc5dd3620 2024-06-04T05:10:52.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.816+0000 7f357072fb80 10 client.19107 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:52.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.817+0000 7f357072fb80 1 -- 172.21.15.70:0/1789573997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x561fc5de9d40 con 0x561fc5dd3620 2024-06-04T05:10:52.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.817+0000 7f353effd700 1 -- 172.21.15.70:0/1789573997 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f355401cd90 con 0x561fc5dd3620 2024-06-04T05:10:52.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.817+0000 7f353effd700 1 -- 172.21.15.70:0/1789573997 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f355402e500 con 0x561fc5dd3620 2024-06-04T05:10:52.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.817+0000 7f357072fb80 10 client.19107 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:52.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.817+0000 7f357072fb80 10 client.19107 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:10:52.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.817+0000 7f357072fb80 20 client.19107 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:52.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.817+0000 7f357072fb80 1 --2- 172.21.15.70:0/1789573997 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x561fc5dec380 0x561fc5dee770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.817+0000 7f357072fb80 4 client.19107 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:10:52.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.817+0000 7f357072fb80 1 -- 172.21.15.70:0/1789573997 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f35500ad9f0 con 0x561fc5dec380 2024-06-04T05:10:52.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.817+0000 7f355f7fe700 1 --2- 172.21.15.70:0/1789573997 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x561fc5dec380 0x561fc5dee770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.818+0000 7f353effd700 10 client.19107 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:10:52.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.818+0000 7f355f7fe700 1 --2- 172.21.15.70:0/1789573997 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x561fc5dec380 0x561fc5dee770 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3966de723b91a2c7 server_cookie=b0f35a812daaeefc in_seq=0 out_seq=0 2024-06-04T05:10:52.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.818+0000 7f353effd700 1 -- 172.21.15.70:0/1789573997 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f35500ad9f0 con 0x561fc5dec380 2024-06-04T05:10:52.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.818+0000 7f353effd700 10 client.19107 handle_command_reply: tid=0 2024-06-04T05:10:52.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.818+0000 7f357072fb80 1 client.19107 shutdown 2024-06-04T05:10:52.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.819+0000 7f357072fb80 1 -- 172.21.15.70:0/1789573997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd3620 msgr2=0x561fc5ddbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:52.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.819+0000 7f357072fb80 1 --2- 172.21.15.70:0/1789573997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd3620 0x561fc5ddbb20 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f355401dc10 tx=0x7f35540047c0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.819+0000 7f357072fb80 1 -- 172.21.15.70:0/1789573997 shutdown_connections 2024-06-04T05:10:52.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.819+0000 7f357072fb80 1 --2- 172.21.15.70:0/1789573997 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x561fc5dec380 0x561fc5dee770 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.819+0000 7f357072fb80 1 --2- 172.21.15.70:0/1789573997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd7370 0x561fc5ddc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.819+0000 7f357072fb80 1 --2- 172.21.15.70:0/1789573997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd2c70 0x561fc5dde820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.819+0000 7f357072fb80 1 --2- 172.21.15.70:0/1789573997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd3620 0x561fc5ddbb20 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.819+0000 7f357072fb80 1 -- 172.21.15.70:0/1789573997 >> 172.21.15.70:0/1789573997 conn(0x561fc5cfc080 msgr2=0x561fc5dc8510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:52.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.819+0000 7f353effd700 0 client.19107 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:10:52.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.819+0000 7f357072fb80 1 -- 172.21.15.70:0/1789573997 shutdown_connections 2024-06-04T05:10:52.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.819+0000 7f357072fb80 1 -- 172.21.15.70:0/1789573997 wait complete. 2024-06-04T05:10:52.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.819+0000 7f357072fb80 20 client.19107 trim_cache size 0 max 16384 2024-06-04T05:10:52.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.836+0000 7f357072fb80 1 Processor -- start 2024-06-04T05:10:52.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f357072fb80 1 -- start start 2024-06-04T05:10:52.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f357072fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd2c70 0x561fc5ddc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f357072fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd3620 0x561fc5df4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f357072fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd7370 0x561fc5df45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f357072fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35601abbc0 con 0x561fc5dd2c70 2024-06-04T05:10:52.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f357072fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35601ad820 con 0x561fc5dd7370 2024-06-04T05:10:52.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f357072fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f35601ac290 con 0x561fc5dd3620 2024-06-04T05:10:52.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f355f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd2c70 0x561fc5ddc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f355effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd3620 0x561fc5df4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f355f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd2c70 0x561fc5ddc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44850/0 (socket says 172.21.15.70:44850) 2024-06-04T05:10:52.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f355f7fe700 1 -- 172.21.15.70:0/1992954811 learned_addr learned my addr 172.21.15.70:0/1992954811 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:52.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f355ffff700 1 --2- 172.21.15.70:0/1992954811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd7370 0x561fc5df45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f355effd700 1 -- 172.21.15.70:0/1992954811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd7370 msgr2=0x561fc5df45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:52.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f355effd700 1 --2- 172.21.15.70:0/1992954811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd7370 0x561fc5df45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f355effd700 1 -- 172.21.15.70:0/1992954811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd2c70 msgr2=0x561fc5ddc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:52.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f355effd700 1 --2- 172.21.15.70:0/1992954811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd2c70 0x561fc5ddc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f355effd700 1 -- 172.21.15.70:0/1992954811 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fc5dc9d40 con 0x561fc5dd3620 2024-06-04T05:10:52.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f355ffff700 1 --2- 172.21.15.70:0/1992954811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd7370 0x561fc5df45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:10:52.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.837+0000 7f355f7fe700 1 --2- 172.21.15.70:0/1992954811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd2c70 0x561fc5ddc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:52.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.838+0000 7f355effd700 1 --2- 172.21.15.70:0/1992954811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd3620 0x561fc5df4060 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3554009600 tx=0x7f3554004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39a3cf977c991f8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:52.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.838+0000 7f35457fa700 1 -- 172.21.15.70:0/1992954811 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355403e070 con 0x561fc5dd3620 2024-06-04T05:10:52.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.838+0000 7f35457fa700 1 -- 172.21.15.70:0/1992954811 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f355401c9f0 con 0x561fc5dd3620 2024-06-04T05:10:52.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.838+0000 7f35457fa700 1 -- 172.21.15.70:0/1992954811 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35540276e0 con 0x561fc5dd3620 2024-06-04T05:10:52.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.839+0000 7f357072fb80 1 -- 172.21.15.70:0/1992954811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd3620 msgr2=0x561fc5df4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:52.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.839+0000 7f357072fb80 1 --2- 172.21.15.70:0/1992954811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd3620 0x561fc5df4060 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3554009600 tx=0x7f3554004750 comp rx=0 tx=0).stop 2024-06-04T05:10:52.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.839+0000 7f357072fb80 1 -- 172.21.15.70:0/1992954811 shutdown_connections 2024-06-04T05:10:52.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.839+0000 7f357072fb80 1 --2- 172.21.15.70:0/1992954811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd7370 0x561fc5df45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.839+0000 7f357072fb80 1 --2- 172.21.15.70:0/1992954811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd2c70 0x561fc5ddc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.839+0000 7f357072fb80 1 --2- 172.21.15.70:0/1992954811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd3620 0x561fc5df4060 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.839+0000 7f357072fb80 1 -- 172.21.15.70:0/1992954811 >> 172.21.15.70:0/1992954811 conn(0x561fc5cfc080 msgr2=0x561fc5ddbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:52.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.839+0000 7f357072fb80 1 -- 172.21.15.70:0/1992954811 shutdown_connections 2024-06-04T05:10:52.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.839+0000 7f357072fb80 1 -- 172.21.15.70:0/1992954811 wait complete. 2024-06-04T05:10:52.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.839+0000 7f357072fb80 1 -- start start 2024-06-04T05:10:52.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.839+0000 7f357072fb80 1 Processor -- start 2024-06-04T05:10:52.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f357072fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd2c70 0x561fc5de17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f357072fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd3620 0x561fc5de1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f357072fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd7370 0x561fc5de2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f357072fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35601ad0f0 con 0x561fc5dd3620 2024-06-04T05:10:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f357072fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35601ac9c0 con 0x561fc5dd2c70 2024-06-04T05:10:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f357072fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f35601a9ba0 con 0x561fc5dd7370 2024-06-04T05:10:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f355effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd3620 0x561fc5de1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f355effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd3620 0x561fc5de1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44862/0 (socket says 172.21.15.70:44862) 2024-06-04T05:10:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f355effd700 1 -- 172.21.15.70:0/3761783020 learned_addr learned my addr 172.21.15.70:0/3761783020 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f355ffff700 1 --2- 172.21.15.70:0/3761783020 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd7370 0x561fc5de2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f355f7fe700 1 --2- 172.21.15.70:0/3761783020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd2c70 0x561fc5de17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f357072fb80 10 client.0 _ll_register_callbacks cb 0x561fc5cb8920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f355ffff700 1 -- 172.21.15.70:0/3761783020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd2c70 msgr2=0x561fc5de17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f355ffff700 1 --2- 172.21.15.70:0/3761783020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd2c70 0x561fc5de17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f355ffff700 1 -- 172.21.15.70:0/3761783020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd3620 msgr2=0x561fc5de1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f355ffff700 1 --2- 172.21.15.70:0/3761783020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd3620 0x561fc5de1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f355ffff700 1 -- 172.21.15.70:0/3761783020 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fc5e55210 con 0x561fc5dd7370 2024-06-04T05:10:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.840+0000 7f355effd700 1 --2- 172.21.15.70:0/3761783020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd3620 0x561fc5de1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.841+0000 7f355ffff700 1 --2- 172.21.15.70:0/3761783020 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd7370 0x561fc5de2290 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f355800b6c0 tx=0x7f3558003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8795a791c6043b2d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.841+0000 7f353effd700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:10:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.841+0000 7f353effd700 1 -- 172.21.15.70:0/3761783020 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3558032060 con 0x561fc5dd7370 2024-06-04T05:10:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.841+0000 7f353effd700 1 -- 172.21.15.70:0/3761783020 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f3558007420 con 0x561fc5dd7370 2024-06-04T05:10:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.841+0000 7f357072fb80 1 -- 172.21.15.70:0/3761783020 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f355401c9f0 con 0x561fc5dd7370 2024-06-04T05:10:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.841+0000 7f353effd700 1 -- 172.21.15.70:0/3761783020 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3558007660 con 0x561fc5dd7370 2024-06-04T05:10:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.841+0000 7f357072fb80 10 client.28132 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.841+0000 7f357072fb80 1 -- 172.21.15.70:0/3761783020 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x561fc5dce790 con 0x561fc5dd7370 2024-06-04T05:10:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.841+0000 7f353effd700 1 -- 172.21.15.70:0/3761783020 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3558007c90 con 0x561fc5dd7370 2024-06-04T05:10:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.841+0000 7f353effd700 1 -- 172.21.15.70:0/3761783020 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f3558011070 con 0x561fc5dd7370 2024-06-04T05:10:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.841+0000 7f357072fb80 10 client.28132 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:52.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.842+0000 7f357072fb80 10 client.28132 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:10:52.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.842+0000 7f357072fb80 20 client.28132 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:52.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.842+0000 7f357072fb80 1 --2- 172.21.15.70:0/3761783020 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x561fc5dcef00 0x561fc5dd12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:52.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.842+0000 7f357072fb80 4 client.28132 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:10:52.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.842+0000 7f357072fb80 1 -- 172.21.15.70:0/3761783020 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f355401c9f0 con 0x561fc5dcef00 2024-06-04T05:10:52.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.842+0000 7f355f7fe700 1 --2- 172.21.15.70:0/3761783020 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x561fc5dcef00 0x561fc5dd12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:52.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.842+0000 7f353effd700 10 client.28132 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:10:52.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.842+0000 7f355f7fe700 1 --2- 172.21.15.70:0/3761783020 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x561fc5dcef00 0x561fc5dd12f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=46418abfd8f6be0 server_cookie=19e0857f5f37318d in_seq=0 out_seq=0 2024-06-04T05:10:52.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.842+0000 7f353effd700 1 -- 172.21.15.70:0/3761783020 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f355401c9f0 con 0x561fc5dcef00 2024-06-04T05:10:52.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.842+0000 7f353effd700 10 client.28132 handle_command_reply: tid=0 2024-06-04T05:10:52.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.842+0000 7f357072fb80 1 client.28132 shutdown 2024-06-04T05:10:52.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.842+0000 7f357072fb80 1 -- 172.21.15.70:0/3761783020 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd7370 msgr2=0x561fc5de2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:52.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.842+0000 7f357072fb80 1 --2- 172.21.15.70:0/3761783020 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd7370 0x561fc5de2290 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f355800b6c0 tx=0x7f3558003f90 comp rx=0 tx=0).stop 2024-06-04T05:10:52.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.842+0000 7f357072fb80 1 -- 172.21.15.70:0/3761783020 shutdown_connections 2024-06-04T05:10:52.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.842+0000 7f357072fb80 1 --2- 172.21.15.70:0/3761783020 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x561fc5dcef00 0x561fc5dd12f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.842+0000 7f357072fb80 1 --2- 172.21.15.70:0/3761783020 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561fc5dd7370 0x561fc5de2290 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.843+0000 7f357072fb80 1 --2- 172.21.15.70:0/3761783020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fc5dd2c70 0x561fc5de17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.843+0000 7f357072fb80 1 --2- 172.21.15.70:0/3761783020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fc5dd3620 0x561fc5de1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.843+0000 7f353effd700 0 client.28132 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:10:52.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.843+0000 7f357072fb80 1 -- 172.21.15.70:0/3761783020 >> 172.21.15.70:0/3761783020 conn(0x561fc5cfc080 msgr2=0x561fc5dd5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:52.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.843+0000 7f357072fb80 1 -- 172.21.15.70:0/3761783020 shutdown_connections 2024-06-04T05:10:52.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.843+0000 7f357072fb80 1 -- 172.21.15.70:0/3761783020 wait complete. 2024-06-04T05:10:52.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.843+0000 7f357072fb80 20 client.28132 trim_cache size 0 max 16384 2024-06-04T05:10:52.851 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:10:52.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.844+0000 7f3565e9a700 1 -- 172.21.15.70:0/1896781269 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35480b2a70 msgr2=0x7f35480b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:52.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.844+0000 7f3565e9a700 1 --2- 172.21.15.70:0/1896781269 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35480b2a70 0x7f35480b4f30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3554009750 tx=0x7f3554009310 comp rx=0 tx=0).stop 2024-06-04T05:10:52.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.844+0000 7f3565e9a700 1 -- 172.21.15.70:0/1896781269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3560102470 msgr2=0x7f356019f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:52.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.844+0000 7f3565e9a700 1 --2- 172.21.15.70:0/1896781269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3560102470 0x7f356019f740 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f355000d980 tx=0x7f355000dc90 comp rx=0 tx=0).stop 2024-06-04T05:10:52.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.844+0000 7f3565e9a700 1 -- 172.21.15.70:0/1896781269 shutdown_connections 2024-06-04T05:10:52.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.844+0000 7f3565e9a700 1 --2- 172.21.15.70:0/1896781269 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35480b2a70 0x7f35480b4f30 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.844+0000 7f3565e9a700 1 --2- 172.21.15.70:0/1896781269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3560108550 0x7f3560199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.844+0000 7f3565e9a700 1 --2- 172.21.15.70:0/1896781269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3560102470 0x7f356019f740 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.844+0000 7f3565e9a700 1 --2- 172.21.15.70:0/1896781269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3560102e30 0x7f356019fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:52.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.844+0000 7f3565e9a700 1 -- 172.21.15.70:0/1896781269 >> 172.21.15.70:0/1896781269 conn(0x7f3560079480 msgr2=0x7f356006c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:52.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.844+0000 7f3565e9a700 1 -- 172.21.15.70:0/1896781269 shutdown_connections 2024-06-04T05:10:52.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:52.844+0000 7f3565e9a700 1 -- 172.21.15.70:0/1896781269 wait complete. 2024-06-04T05:10:53.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:53 smithi098 ceph-mon[35569]: pgmap v426: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-06-04T05:10:53.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:53 smithi070 ceph-mon[32631]: pgmap v426: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-06-04T05:10:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:53 smithi112 ceph-mon[36453]: pgmap v426: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-06-04T05:10:53.698 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:10:53.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 216244 1.57 MB/sec execute 586 sec latency 15.411 ms 2024-06-04T05:10:53.968 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:10:54.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 216491 1.57 MB/sec execute 587 sec latency 647.059 ms 2024-06-04T05:10:55.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:55 smithi070 ceph-mon[32631]: pgmap v427: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 533 KiB/s wr, 70 op/s 2024-06-04T05:10:55.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:55 smithi098 ceph-mon[35569]: pgmap v427: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 533 KiB/s wr, 70 op/s 2024-06-04T05:10:55.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.655+0000 7f35c9c35700 1 -- 172.21.15.70:0/1220762180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35c4108600 msgr2=0x7f35c41089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:55.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.655+0000 7f35c9c35700 1 --2- 172.21.15.70:0/1220762180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35c4108600 0x7f35c41089e0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009990 tx=0x7f35b4009ca0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.655+0000 7f35c9c35700 1 -- 172.21.15.70:0/1220762180 shutdown_connections 2024-06-04T05:10:55.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.655+0000 7f35c9c35700 1 --2- 172.21.15.70:0/1220762180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c41028a0 0x7f35c4110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.656+0000 7f35c9c35700 1 --2- 172.21.15.70:0/1220762180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35c4108600 0x7f35c41089e0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.656+0000 7f35c9c35700 1 --2- 172.21.15.70:0/1220762180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35c4101ee0 0x7f35c4102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.656+0000 7f35c9c35700 1 -- 172.21.15.70:0/1220762180 >> 172.21.15.70:0/1220762180 conn(0x7f35c4077760 msgr2=0x7f35c4077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:55.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.656+0000 7f35c9c35700 1 -- 172.21.15.70:0/1220762180 shutdown_connections 2024-06-04T05:10:55.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.656+0000 7f35c9c35700 1 -- 172.21.15.70:0/1220762180 wait complete. 2024-06-04T05:10:55.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.656+0000 7f35c9c35700 1 Processor -- start 2024-06-04T05:10:55.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.656+0000 7f35c9c35700 1 -- start start 2024-06-04T05:10:55.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c9c35700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35c4101ee0 0x7f35c41030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c9c35700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35c41028a0 0x7f35c4103600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c9c35700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c4108600 0x7f35c4103b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c9c35700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35c4113640 con 0x7f35c41028a0 2024-06-04T05:10:55.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c9c35700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35c41134c0 con 0x7f35c4108600 2024-06-04T05:10:55.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c9c35700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f35c41137c0 con 0x7f35c4101ee0 2024-06-04T05:10:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35c41028a0 0x7f35c4103600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35c41028a0 0x7f35c4103600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44890/0 (socket says 172.21.15.70:44890) 2024-06-04T05:10:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c2ffd700 1 -- 172.21.15.70:0/3714649910 learned_addr learned my addr 172.21.15.70:0/3714649910 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c37fe700 1 --2- 172.21.15.70:0/3714649910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35c4101ee0 0x7f35c41030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c3fff700 1 --2- 172.21.15.70:0/3714649910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c4108600 0x7f35c4103b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c2ffd700 1 -- 172.21.15.70:0/3714649910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35c4101ee0 msgr2=0x7f35c41030c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c2ffd700 1 --2- 172.21.15.70:0/3714649910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35c4101ee0 0x7f35c41030c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c2ffd700 1 -- 172.21.15.70:0/3714649910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c4108600 msgr2=0x7f35c4103b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c2ffd700 1 --2- 172.21.15.70:0/3714649910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c4108600 0x7f35c4103b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c2ffd700 1 -- 172.21.15.70:0/3714649910 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c41a80e0 con 0x7f35c41028a0 2024-06-04T05:10:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c3fff700 1 --2- 172.21.15.70:0/3714649910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c4108600 0x7f35c4103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:10:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c2ffd700 1 --2- 172.21.15.70:0/3714649910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35c41028a0 0x7f35c4103600 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f35b800c920 tx=0x7f35b800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c0ff9700 1 -- 172.21.15.70:0/3714649910 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b8007b00 con 0x7f35c41028a0 2024-06-04T05:10:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c9c35700 1 -- 172.21.15.70:0/3714649910 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35c41a83d0 con 0x7f35c41028a0 2024-06-04T05:10:55.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c0ff9700 1 -- 172.21.15.70:0/3714649910 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35b8004cc0 con 0x7f35c41028a0 2024-06-04T05:10:55.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.657+0000 7f35c0ff9700 1 -- 172.21.15.70:0/3714649910 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b80054f0 con 0x7f35c41028a0 2024-06-04T05:10:55.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.658+0000 7f35c9c35700 1 -- 172.21.15.70:0/3714649910 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35c41a8960 con 0x7f35c41028a0 2024-06-04T05:10:55.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.660+0000 7f35c0ff9700 1 -- 172.21.15.70:0/3714649910 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f35b8007ca0 con 0x7f35c41028a0 2024-06-04T05:10:55.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.660+0000 7f35c0ff9700 1 --2- 172.21.15.70:0/3714649910 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35ac0b2a70 0x7f35ac0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.661+0000 7f35c37fe700 1 --2- 172.21.15.70:0/3714649910 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35ac0b2a70 0x7f35ac0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.661+0000 7f35c0ff9700 1 -- 172.21.15.70:0/3714649910 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f35b80e2220 con 0x7f35c41028a0 2024-06-04T05:10:55.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.661+0000 7f35c37fe700 1 --2- 172.21.15.70:0/3714649910 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35ac0b2a70 0x7f35ac0b4f30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f35b4005850 tx=0x7f35b401a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:55.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.661+0000 7f35d44cab80 1 Processor -- start 2024-06-04T05:10:55.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.661+0000 7f35d44cab80 1 -- start start 2024-06-04T05:10:55.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.661+0000 7f35d44cab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d85860 0x555e76d85c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.661+0000 7f35d44cab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 0x555e76d89940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.661+0000 7f35d44cab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d89f60 0x555e76d8a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.661+0000 7f35d44cab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555e76ce4c90 con 0x555e76d86210 2024-06-04T05:10:55.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.661+0000 7f35d44cab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555e76cd4830 con 0x555e76d89f60 2024-06-04T05:10:55.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.661+0000 7f35d44cab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555e76cd8870 con 0x555e76d85860 2024-06-04T05:10:55.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.662+0000 7f35c2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 0x555e76d89940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.662+0000 7f35c2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 0x555e76d89940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44896/0 (socket says 172.21.15.70:44896) 2024-06-04T05:10:55.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.662+0000 7f35c2ffd700 1 -- 172.21.15.70:0/3713841398 learned_addr learned my addr 172.21.15.70:0/3713841398 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:55.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.662+0000 7f35c3fff700 1 --2- 172.21.15.70:0/3713841398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d89f60 0x555e76d8a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.662+0000 7f35c37fe700 1 --2- 172.21.15.70:0/3713841398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d85860 0x555e76d85c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.662+0000 7f35c2ffd700 1 -- 172.21.15.70:0/3713841398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d85860 msgr2=0x555e76d85c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:55.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.662+0000 7f35c2ffd700 1 --2- 172.21.15.70:0/3713841398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d85860 0x555e76d85c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.662+0000 7f35c2ffd700 1 -- 172.21.15.70:0/3713841398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d89f60 msgr2=0x555e76d8a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:55.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.662+0000 7f35c2ffd700 1 --2- 172.21.15.70:0/3713841398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d89f60 0x555e76d8a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.662+0000 7f35c2ffd700 1 -- 172.21.15.70:0/3713841398 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e76c248d0 con 0x555e76d86210 2024-06-04T05:10:55.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.662+0000 7f35c2ffd700 1 --2- 172.21.15.70:0/3713841398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 0x555e76d89940 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f35b80a9460 tx=0x7f35b80a9060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880aadbfca018ef1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:55.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.662+0000 7f35a9ffb700 1 -- 172.21.15.70:0/3713841398 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b801d040 con 0x555e76d86210 2024-06-04T05:10:55.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.662+0000 7f35a9ffb700 1 -- 172.21.15.70:0/3713841398 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35b80a94d0 con 0x555e76d86210 2024-06-04T05:10:55.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.662+0000 7f35a9ffb700 1 -- 172.21.15.70:0/3713841398 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b80b1ea0 con 0x555e76d86210 2024-06-04T05:10:55.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.663+0000 7f35d44cab80 1 -- 172.21.15.70:0/3713841398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 msgr2=0x555e76d89940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:55.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.663+0000 7f35d44cab80 1 --2- 172.21.15.70:0/3713841398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 0x555e76d89940 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f35b80a9460 tx=0x7f35b80a9060 comp rx=0 tx=0).stop 2024-06-04T05:10:55.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.663+0000 7f35d44cab80 1 -- 172.21.15.70:0/3713841398 shutdown_connections 2024-06-04T05:10:55.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.663+0000 7f35d44cab80 1 --2- 172.21.15.70:0/3713841398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d89f60 0x555e76d8a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.663+0000 7f35d44cab80 1 --2- 172.21.15.70:0/3713841398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d85860 0x555e76d85c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.663+0000 7f35d44cab80 1 --2- 172.21.15.70:0/3713841398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 0x555e76d89940 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.663+0000 7f35d44cab80 1 -- 172.21.15.70:0/3713841398 >> 172.21.15.70:0/3713841398 conn(0x555e76cb3080 msgr2=0x555e76d7fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:55.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.663+0000 7f35d44cab80 1 -- 172.21.15.70:0/3713841398 shutdown_connections 2024-06-04T05:10:55.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.663+0000 7f35d44cab80 1 -- 172.21.15.70:0/3713841398 wait complete. 2024-06-04T05:10:55.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.663+0000 7f35d44cab80 1 -- start start 2024-06-04T05:10:55.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35d44cab80 1 Processor -- start 2024-06-04T05:10:55.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35d44cab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d85860 0x555e76d91450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35d44cab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 0x555e76d8e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35d44cab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d89f60 0x555e76d8ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35d44cab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555e76bdafa0 con 0x555e76d86210 2024-06-04T05:10:55.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35d44cab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555e76c767c0 con 0x555e76d89f60 2024-06-04T05:10:55.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35d44cab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555e76d92120 con 0x555e76d85860 2024-06-04T05:10:55.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35d44cab80 10 client.0 _ll_register_callbacks cb 0x555e76c6f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:55.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35c37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d85860 0x555e76d91450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35c2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 0x555e76d8e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35c37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d85860 0x555e76d91450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:35338/0 (socket says 172.21.15.70:35338) 2024-06-04T05:10:55.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35c37fe700 1 -- 172.21.15.70:0/4163903867 learned_addr learned my addr 172.21.15.70:0/4163903867 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:55.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35c3fff700 1 --2- 172.21.15.70:0/4163903867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d89f60 0x555e76d8ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35c37fe700 1 -- 172.21.15.70:0/4163903867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d89f60 msgr2=0x555e76d8ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:55.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35c37fe700 1 --2- 172.21.15.70:0/4163903867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d89f60 0x555e76d8ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35c37fe700 1 -- 172.21.15.70:0/4163903867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 msgr2=0x555e76d8e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:55.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35c37fe700 1 --2- 172.21.15.70:0/4163903867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 0x555e76d8e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35c37fe700 1 -- 172.21.15.70:0/4163903867 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e76c73690 con 0x555e76d85860 2024-06-04T05:10:55.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.664+0000 7f35c2ffd700 1 --2- 172.21.15.70:0/4163903867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 0x555e76d8e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:10:55.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.665+0000 7f35c37fe700 1 --2- 172.21.15.70:0/4163903867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d85860 0x555e76d91450 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f35b4015000 tx=0x7f35b4009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92cf32c176e01291 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:55.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.665+0000 7f35a2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:10:55.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.665+0000 7f35a2ffd700 1 -- 172.21.15.70:0/4163903867 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b402f850 con 0x555e76d85860 2024-06-04T05:10:55.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.665+0000 7f35a2ffd700 1 -- 172.21.15.70:0/4163903867 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f35b402f9f0 con 0x555e76d85860 2024-06-04T05:10:55.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.665+0000 7f35d44cab80 1 -- 172.21.15.70:0/4163903867 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35b80a94d0 con 0x555e76d85860 2024-06-04T05:10:55.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.665+0000 7f35a2ffd700 1 -- 172.21.15.70:0/4163903867 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f35b403d050 con 0x555e76d85860 2024-06-04T05:10:55.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.665+0000 7f35d44cab80 10 client.28141 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:55.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.665+0000 7f35d44cab80 1 -- 172.21.15.70:0/4163903867 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x555e76d9c8d0 con 0x555e76d85860 2024-06-04T05:10:55.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.665+0000 7f35a2ffd700 1 -- 172.21.15.70:0/4163903867 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35b80a94d0 con 0x555e76d85860 2024-06-04T05:10:55.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.666+0000 7f35a2ffd700 1 -- 172.21.15.70:0/4163903867 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f35b40239f0 con 0x555e76d85860 2024-06-04T05:10:55.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.666+0000 7f35d44cab80 10 client.28141 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:55.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.666+0000 7f35d44cab80 10 client.28141 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:10:55.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.666+0000 7f35d44cab80 20 client.28141 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:55.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.666+0000 7f35d44cab80 1 --2- 172.21.15.70:0/4163903867 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x555e76d9ef60 0x555e76da1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.666+0000 7f35d44cab80 4 client.28141 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:10:55.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.666+0000 7f35d44cab80 1 -- 172.21.15.70:0/4163903867 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e76d519e0 con 0x555e76d9ef60 2024-06-04T05:10:55.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.666+0000 7f35c2ffd700 1 --2- 172.21.15.70:0/4163903867 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x555e76d9ef60 0x555e76da1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.667+0000 7f35a2ffd700 10 client.28141 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:10:55.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.667+0000 7f35c2ffd700 1 --2- 172.21.15.70:0/4163903867 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x555e76d9ef60 0x555e76da1350 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=64ab6d5ed286a852 server_cookie=e0a7dfc4d80bf4e0 in_seq=0 out_seq=0 2024-06-04T05:10:55.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.667+0000 7f35a2ffd700 1 -- 172.21.15.70:0/4163903867 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555e76d519e0 con 0x555e76d9ef60 2024-06-04T05:10:55.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.667+0000 7f35a2ffd700 10 client.28141 handle_command_reply: tid=0 2024-06-04T05:10:55.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.667+0000 7f35d44cab80 1 client.28141 shutdown 2024-06-04T05:10:55.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.668+0000 7f35d44cab80 1 -- 172.21.15.70:0/4163903867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d85860 msgr2=0x555e76d91450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:55.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.668+0000 7f35d44cab80 1 --2- 172.21.15.70:0/4163903867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d85860 0x555e76d91450 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f35b4015000 tx=0x7f35b4009310 comp rx=0 tx=0).stop 2024-06-04T05:10:55.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.668+0000 7f35d44cab80 1 -- 172.21.15.70:0/4163903867 shutdown_connections 2024-06-04T05:10:55.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.668+0000 7f35d44cab80 1 --2- 172.21.15.70:0/4163903867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d89f60 0x555e76d8ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.668+0000 7f35d44cab80 1 --2- 172.21.15.70:0/4163903867 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x555e76d9ef60 0x555e76da1350 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.668+0000 7f35d44cab80 1 --2- 172.21.15.70:0/4163903867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d85860 0x555e76d91450 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.668+0000 7f35d44cab80 1 --2- 172.21.15.70:0/4163903867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 0x555e76d8e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.668+0000 7f35a2ffd700 0 client.28141 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:10:55.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.668+0000 7f35d44cab80 1 -- 172.21.15.70:0/4163903867 >> 172.21.15.70:0/4163903867 conn(0x555e76cb3080 msgr2=0x555e76d7f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:55.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.668+0000 7f35d44cab80 1 -- 172.21.15.70:0/4163903867 shutdown_connections 2024-06-04T05:10:55.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.668+0000 7f35d44cab80 1 -- 172.21.15.70:0/4163903867 wait complete. 2024-06-04T05:10:55.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.668+0000 7f35d44cab80 20 client.28141 trim_cache size 0 max 16384 2024-06-04T05:10:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35d44cab80 1 Processor -- start 2024-06-04T05:10:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35d44cab80 1 -- start start 2024-06-04T05:10:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35d44cab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d85860 0x555e76d9b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35d44cab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d86210 0x555e76d90690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35d44cab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d89f60 0x555e76d7fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35d44cab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35c41b0100 con 0x555e76d89f60 2024-06-04T05:10:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35d44cab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35c41b1d60 con 0x555e76d85860 2024-06-04T05:10:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35d44cab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f35c41b07d0 con 0x555e76d86210 2024-06-04T05:10:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35c3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d89f60 0x555e76d7fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35c3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d89f60 0x555e76d7fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44914/0 (socket says 172.21.15.70:44914) 2024-06-04T05:10:55.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35c3fff700 1 -- 172.21.15.70:0/3606913430 learned_addr learned my addr 172.21.15.70:0/3606913430 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:55.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35c37fe700 1 --2- 172.21.15.70:0/3606913430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d85860 0x555e76d9b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35c2ffd700 1 --2- 172.21.15.70:0/3606913430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d86210 0x555e76d90690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35c3fff700 1 -- 172.21.15.70:0/3606913430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d86210 msgr2=0x555e76d90690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:55.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35c3fff700 1 --2- 172.21.15.70:0/3606913430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d86210 0x555e76d90690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35c3fff700 1 -- 172.21.15.70:0/3606913430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d85860 msgr2=0x555e76d9b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:55.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35c3fff700 1 --2- 172.21.15.70:0/3606913430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d85860 0x555e76d9b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35c3fff700 1 -- 172.21.15.70:0/3606913430 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e76da19b0 con 0x555e76d89f60 2024-06-04T05:10:55.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35c2ffd700 1 --2- 172.21.15.70:0/3606913430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d86210 0x555e76d90690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:10:55.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.683+0000 7f35c3fff700 1 --2- 172.21.15.70:0/3606913430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d89f60 0x555e76d7fc90 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f35bc00c770 tx=0x7f35bc009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18e442367fdd6270 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:55.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.684+0000 7f35a97fa700 1 -- 172.21.15.70:0/3606913430 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc01c020 con 0x555e76d89f60 2024-06-04T05:10:55.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.684+0000 7f35a97fa700 1 -- 172.21.15.70:0/3606913430 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35bc0188e0 con 0x555e76d89f60 2024-06-04T05:10:55.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.684+0000 7f35a97fa700 1 -- 172.21.15.70:0/3606913430 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc00a660 con 0x555e76d89f60 2024-06-04T05:10:55.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:55 smithi112 ceph-mon[36453]: pgmap v427: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 533 KiB/s wr, 70 op/s 2024-06-04T05:10:55.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.684+0000 7f35d44cab80 1 -- 172.21.15.70:0/3606913430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d89f60 msgr2=0x555e76d7fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:55.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.684+0000 7f35d44cab80 1 --2- 172.21.15.70:0/3606913430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d89f60 0x555e76d7fc90 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f35bc00c770 tx=0x7f35bc009850 comp rx=0 tx=0).stop 2024-06-04T05:10:55.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.684+0000 7f35d44cab80 1 -- 172.21.15.70:0/3606913430 shutdown_connections 2024-06-04T05:10:55.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.684+0000 7f35d44cab80 1 --2- 172.21.15.70:0/3606913430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d89f60 0x555e76d7fc90 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.684+0000 7f35d44cab80 1 --2- 172.21.15.70:0/3606913430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d85860 0x555e76d9b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.684+0000 7f35d44cab80 1 --2- 172.21.15.70:0/3606913430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d86210 0x555e76d90690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.684+0000 7f35d44cab80 1 -- 172.21.15.70:0/3606913430 >> 172.21.15.70:0/3606913430 conn(0x555e76cb3080 msgr2=0x555e76d8ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:55.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.684+0000 7f35d44cab80 1 -- 172.21.15.70:0/3606913430 shutdown_connections 2024-06-04T05:10:55.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.684+0000 7f35d44cab80 1 -- 172.21.15.70:0/3606913430 wait complete. 2024-06-04T05:10:55.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.685+0000 7f35d44cab80 1 -- start start 2024-06-04T05:10:55.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.685+0000 7f35d44cab80 1 Processor -- start 2024-06-04T05:10:55.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.685+0000 7f35d44cab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d85860 0x555e76d97a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.685+0000 7f35d44cab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 0x555e76d94e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.685+0000 7f35d44cab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d89f60 0x555e76d95380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.685+0000 7f35d44cab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35c41b1630 con 0x555e76d86210 2024-06-04T05:10:55.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.685+0000 7f35d44cab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35c41b0f00 con 0x555e76d89f60 2024-06-04T05:10:55.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.685+0000 7f35d44cab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f35c41ae0c0 con 0x555e76d85860 2024-06-04T05:10:55.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.685+0000 7f35c2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 0x555e76d94e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.685+0000 7f35d44cab80 10 client.0 _ll_register_callbacks cb 0x555e76c6f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:55.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.685+0000 7f35c2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 0x555e76d94e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44922/0 (socket says 172.21.15.70:44922) 2024-06-04T05:10:55.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.685+0000 7f35c2ffd700 1 -- 172.21.15.70:0/2968512053 learned_addr learned my addr 172.21.15.70:0/2968512053 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:55.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.685+0000 7f35c3fff700 1 --2- 172.21.15.70:0/2968512053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d89f60 0x555e76d95380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35c37fe700 1 --2- 172.21.15.70:0/2968512053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d85860 0x555e76d97a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35c2ffd700 1 -- 172.21.15.70:0/2968512053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d85860 msgr2=0x555e76d97a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:55.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35c2ffd700 1 --2- 172.21.15.70:0/2968512053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d85860 0x555e76d97a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35c2ffd700 1 -- 172.21.15.70:0/2968512053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d89f60 msgr2=0x555e76d95380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:55.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35c2ffd700 1 --2- 172.21.15.70:0/2968512053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d89f60 0x555e76d95380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35c2ffd700 1 -- 172.21.15.70:0/2968512053 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e76d95bc0 con 0x555e76d86210 2024-06-04T05:10:55.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35c3fff700 1 --2- 172.21.15.70:0/2968512053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d89f60 0x555e76d95380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:10:55.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35c2ffd700 1 --2- 172.21.15.70:0/2968512053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 0x555e76d94e40 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f35b80a6740 tx=0x7f35b8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6362f1ccdc5738cd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:55.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35a2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:55.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35a2ffd700 1 -- 172.21.15.70:0/2968512053 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b80c8060 con 0x555e76d86210 2024-06-04T05:10:55.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35a2ffd700 1 -- 172.21.15.70:0/2968512053 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f35b80b20b0 con 0x555e76d86210 2024-06-04T05:10:55.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35d44cab80 1 -- 172.21.15.70:0/2968512053 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35bc0188e0 con 0x555e76d86210 2024-06-04T05:10:55.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35a2ffd700 1 -- 172.21.15.70:0/2968512053 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f35b8088050 con 0x555e76d86210 2024-06-04T05:10:55.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35d44cab80 10 client.19131 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:55.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35d44cab80 1 -- 172.21.15.70:0/2968512053 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x555e76e09a80 con 0x555e76d86210 2024-06-04T05:10:55.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35a2ffd700 1 -- 172.21.15.70:0/2968512053 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35bc0188e0 con 0x555e76d86210 2024-06-04T05:10:55.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35a2ffd700 1 -- 172.21.15.70:0/2968512053 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f35b80ca3d0 con 0x555e76d86210 2024-06-04T05:10:55.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35d44cab80 10 client.19131 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:55.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35d44cab80 10 client.19131 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:10:55.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35d44cab80 20 client.19131 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:55.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35d44cab80 1 --2- 172.21.15.70:0/2968512053 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x555e76e09d70 0x555e76e0c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:55.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35d44cab80 4 client.19131 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:10:55.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.686+0000 7f35d44cab80 1 -- 172.21.15.70:0/2968512053 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555e76ce6be0 con 0x555e76e09d70 2024-06-04T05:10:55.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.687+0000 7f35c37fe700 1 --2- 172.21.15.70:0/2968512053 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x555e76e09d70 0x555e76e0c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:55.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.687+0000 7f35c37fe700 1 --2- 172.21.15.70:0/2968512053 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x555e76e09d70 0x555e76e0c160 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ee4034b2b98a599e server_cookie=5bceb1704e8d40ca in_seq=0 out_seq=0 2024-06-04T05:10:55.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.687+0000 7f35a2ffd700 10 client.19131 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:10:55.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.687+0000 7f35a2ffd700 1 -- 172.21.15.70:0/2968512053 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555e76ce6be0 con 0x555e76e09d70 2024-06-04T05:10:55.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.687+0000 7f35a2ffd700 10 client.19131 handle_command_reply: tid=0 2024-06-04T05:10:55.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.687+0000 7f35d44cab80 1 client.19131 shutdown 2024-06-04T05:10:55.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.687+0000 7f35d44cab80 1 -- 172.21.15.70:0/2968512053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 msgr2=0x555e76d94e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:55.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.687+0000 7f35d44cab80 1 --2- 172.21.15.70:0/2968512053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 0x555e76d94e40 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f35b80a6740 tx=0x7f35b8000bc0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.687+0000 7f35d44cab80 1 -- 172.21.15.70:0/2968512053 shutdown_connections 2024-06-04T05:10:55.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.687+0000 7f35d44cab80 1 --2- 172.21.15.70:0/2968512053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e76d89f60 0x555e76d95380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.687+0000 7f35d44cab80 1 --2- 172.21.15.70:0/2968512053 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x555e76e09d70 0x555e76e0c160 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.687+0000 7f35a2ffd700 0 client.19131 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:10:55.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.687+0000 7f35d44cab80 1 --2- 172.21.15.70:0/2968512053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555e76d85860 0x555e76d97a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.687+0000 7f35d44cab80 1 --2- 172.21.15.70:0/2968512053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555e76d86210 0x555e76d94e40 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.687+0000 7f35d44cab80 1 -- 172.21.15.70:0/2968512053 >> 172.21.15.70:0/2968512053 conn(0x555e76cb3080 msgr2=0x555e76d89140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:55.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.688+0000 7f35d44cab80 1 -- 172.21.15.70:0/2968512053 shutdown_connections 2024-06-04T05:10:55.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.688+0000 7f35d44cab80 1 -- 172.21.15.70:0/2968512053 wait complete. 2024-06-04T05:10:55.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.688+0000 7f35d44cab80 20 client.19131 trim_cache size 0 max 16384 2024-06-04T05:10:55.696 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:10:55.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.688+0000 7f35c9c35700 1 -- 172.21.15.70:0/3714649910 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35ac0b2a70 msgr2=0x7f35ac0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:55.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.688+0000 7f35c9c35700 1 --2- 172.21.15.70:0/3714649910 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35ac0b2a70 0x7f35ac0b4f30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f35b4005850 tx=0x7f35b401a040 comp rx=0 tx=0).stop 2024-06-04T05:10:55.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.688+0000 7f35c9c35700 1 -- 172.21.15.70:0/3714649910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35c41028a0 msgr2=0x7f35c4103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:55.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.688+0000 7f35c9c35700 1 --2- 172.21.15.70:0/3714649910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35c41028a0 0x7f35c4103600 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f35b800c920 tx=0x7f35b800cce0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.688+0000 7f35c9c35700 1 -- 172.21.15.70:0/3714649910 shutdown_connections 2024-06-04T05:10:55.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.688+0000 7f35c9c35700 1 --2- 172.21.15.70:0/3714649910 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f35ac0b2a70 0x7f35ac0b4f30 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.688+0000 7f35c9c35700 1 --2- 172.21.15.70:0/3714649910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c4108600 0x7f35c4103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.688+0000 7f35c9c35700 1 --2- 172.21.15.70:0/3714649910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35c4101ee0 0x7f35c41030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.688+0000 7f35c9c35700 1 --2- 172.21.15.70:0/3714649910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35c41028a0 0x7f35c4103600 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:55.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.688+0000 7f35c9c35700 1 -- 172.21.15.70:0/3714649910 >> 172.21.15.70:0/3714649910 conn(0x7f35c4077760 msgr2=0x7f35c4100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:55.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.688+0000 7f35c9c35700 1 -- 172.21.15.70:0/3714649910 shutdown_connections 2024-06-04T05:10:55.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:55.688+0000 7f35c9c35700 1 -- 172.21.15.70:0/3714649910 wait complete. 2024-06-04T05:10:55.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 216491 1.57 MB/sec execute 588 sec latency 1647.098 ms 2024-06-04T05:10:56.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:10:56.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 216491 1.57 MB/sec execute 589 sec latency 2647.128 ms 2024-06-04T05:10:56.770 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:10:57.485 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:57 smithi070 ceph-mon[32631]: pgmap v428: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-06-04T05:10:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:57 smithi112 ceph-mon[36453]: pgmap v428: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-06-04T05:10:57.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 216558 1.57 MB/sec execute 590 sec latency 2949.260 ms 2024-06-04T05:10:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:57 smithi098 ceph-mon[35569]: pgmap v428: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-06-04T05:10:58.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.342+0000 7f8bf0c95700 1 -- 172.21.15.70:0/663498148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bec06bab0 msgr2=0x7f8bec06bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:58.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.342+0000 7f8bf0c95700 1 --2- 172.21.15.70:0/663498148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bec06bab0 0x7f8bec06bf30 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009990 tx=0x7f8be0009ca0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.342+0000 7f8bf0c95700 1 -- 172.21.15.70:0/663498148 shutdown_connections 2024-06-04T05:10:58.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.342+0000 7f8bf0c95700 1 --2- 172.21.15.70:0/663498148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bec06c470 0x7f8bec1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.342+0000 7f8bf0c95700 1 --2- 172.21.15.70:0/663498148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bec1095a0 0x7f8bec109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.342+0000 7f8bf0c95700 1 --2- 172.21.15.70:0/663498148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bec06bab0 0x7f8bec06bf30 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.342+0000 7f8bf0c95700 1 -- 172.21.15.70:0/663498148 >> 172.21.15.70:0/663498148 conn(0x7f8bec0793b0 msgr2=0x7f8bec0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:58.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.342+0000 7f8bf0c95700 1 -- 172.21.15.70:0/663498148 shutdown_connections 2024-06-04T05:10:58.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.342+0000 7f8bf0c95700 1 -- 172.21.15.70:0/663498148 wait complete. 2024-06-04T05:10:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.343+0000 7f8bf0c95700 1 Processor -- start 2024-06-04T05:10:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.343+0000 7f8bf0c95700 1 -- start start 2024-06-04T05:10:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.343+0000 7f8bf0c95700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bec06bab0 0x7f8bec19f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.343+0000 7f8bf0c95700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bec06c470 0x7f8bec19fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.343+0000 7f8bf0c95700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bec1095a0 0x7f8bec199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.343+0000 7f8bf0c95700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8bec113580 con 0x7f8bec1095a0 2024-06-04T05:10:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.343+0000 7f8bf0c95700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8bec113400 con 0x7f8bec06bab0 2024-06-04T05:10:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.343+0000 7f8bf0c95700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8bec113700 con 0x7f8bec06c470 2024-06-04T05:10:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.343+0000 7f8bead9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bec1095a0 0x7f8bec199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.343+0000 7f8bead9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bec1095a0 0x7f8bec199980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57494/0 (socket says 172.21.15.70:57494) 2024-06-04T05:10:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.343+0000 7f8bead9d700 1 -- 172.21.15.70:0/155646730 learned_addr learned my addr 172.21.15.70:0/155646730 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.343+0000 7f8bea59c700 1 --2- 172.21.15.70:0/155646730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bec06bab0 0x7f8bec19f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.344+0000 7f8be9d9b700 1 --2- 172.21.15.70:0/155646730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bec06c470 0x7f8bec19fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.344+0000 7f8bead9d700 1 -- 172.21.15.70:0/155646730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bec06c470 msgr2=0x7f8bec19fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.344+0000 7f8bead9d700 1 --2- 172.21.15.70:0/155646730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bec06c470 0x7f8bec19fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.344+0000 7f8bead9d700 1 -- 172.21.15.70:0/155646730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bec06bab0 msgr2=0x7f8bec19f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.344+0000 7f8bead9d700 1 --2- 172.21.15.70:0/155646730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bec06bab0 0x7f8bec19f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.344+0000 7f8bead9d700 1 -- 172.21.15.70:0/155646730 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bec19a0e0 con 0x7f8bec1095a0 2024-06-04T05:10:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.344+0000 7f8be9d9b700 1 --2- 172.21.15.70:0/155646730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bec06c470 0x7f8bec19fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:10:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.344+0000 7f8bead9d700 1 --2- 172.21.15.70:0/155646730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bec1095a0 0x7f8bec199980 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f8be400ada0 tx=0x7f8be400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.344+0000 7f8bdb7fe700 1 -- 172.21.15.70:0/155646730 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be4017020 con 0x7f8bec1095a0 2024-06-04T05:10:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.344+0000 7f8bdb7fe700 1 -- 172.21.15.70:0/155646730 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8be400ce10 con 0x7f8bec1095a0 2024-06-04T05:10:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.344+0000 7f8bf0c95700 1 -- 172.21.15.70:0/155646730 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bec19a3d0 con 0x7f8bec1095a0 2024-06-04T05:10:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.344+0000 7f8bdb7fe700 1 -- 172.21.15.70:0/155646730 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be4005280 con 0x7f8bec1095a0 2024-06-04T05:10:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.344+0000 7f8bf0c95700 1 -- 172.21.15.70:0/155646730 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bec1a6690 con 0x7f8bec1095a0 2024-06-04T05:10:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.346+0000 7f8bdb7fe700 1 -- 172.21.15.70:0/155646730 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f8be400c980 con 0x7f8bec1095a0 2024-06-04T05:10:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.347+0000 7f8bdb7fe700 1 --2- 172.21.15.70:0/155646730 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8bd40b29f0 0x7f8bd40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.347+0000 7f8bea59c700 1 --2- 172.21.15.70:0/155646730 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8bd40b29f0 0x7f8bd40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.347+0000 7f8bdb7fe700 1 -- 172.21.15.70:0/155646730 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f8be40e1e00 con 0x7f8bec1095a0 2024-06-04T05:10:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.348+0000 7f8bfb52ab80 1 Processor -- start 2024-06-04T05:10:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.348+0000 7f8bea59c700 1 --2- 172.21.15.70:0/155646730 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8bd40b29f0 0x7f8bd40b4eb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8bec19aac0 tx=0x7f8bdc00b410 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.348+0000 7f8bfb52ab80 1 -- start start 2024-06-04T05:10:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.348+0000 7f8bfb52ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba4bc70 0x5615fba4c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.348+0000 7f8bfb52ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba4c620 0x5615fba4fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.348+0000 7f8bfb52ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba50370 0x5615fba507b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.348+0000 7f8bfb52ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5615fb9a6c90 con 0x5615fba50370 2024-06-04T05:10:58.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.348+0000 7f8bfb52ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5615fb996830 con 0x5615fba4bc70 2024-06-04T05:10:58.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.348+0000 7f8bfb52ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5615fb99a870 con 0x5615fba4c620 2024-06-04T05:10:58.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.348+0000 7f8bead9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba50370 0x5615fba507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.348+0000 7f8bead9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba50370 0x5615fba507b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57508/0 (socket says 172.21.15.70:57508) 2024-06-04T05:10:58.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.348+0000 7f8bead9d700 1 -- 172.21.15.70:0/3010362008 learned_addr learned my addr 172.21.15.70:0/3010362008 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:58.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.348+0000 7f8bea59c700 1 --2- 172.21.15.70:0/3010362008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba4bc70 0x5615fba4c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.348+0000 7f8be9d9b700 1 --2- 172.21.15.70:0/3010362008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba4c620 0x5615fba4fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.349+0000 7f8bead9d700 1 -- 172.21.15.70:0/3010362008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba4c620 msgr2=0x5615fba4fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.349+0000 7f8bead9d700 1 --2- 172.21.15.70:0/3010362008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba4c620 0x5615fba4fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.349+0000 7f8bead9d700 1 -- 172.21.15.70:0/3010362008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba4bc70 msgr2=0x5615fba4c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.349+0000 7f8bead9d700 1 --2- 172.21.15.70:0/3010362008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba4bc70 0x5615fba4c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.349+0000 7f8bead9d700 1 -- 172.21.15.70:0/3010362008 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615fb8e68d0 con 0x5615fba50370 2024-06-04T05:10:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.349+0000 7f8bead9d700 1 --2- 172.21.15.70:0/3010362008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba50370 0x5615fba507b0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f8be4004d70 tx=0x7f8be400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3e3115c71adf32 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.349+0000 7f8bd8ff9700 1 -- 172.21.15.70:0/3010362008 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be40a8bf0 con 0x5615fba50370 2024-06-04T05:10:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.349+0000 7f8bd8ff9700 1 -- 172.21.15.70:0/3010362008 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8be40a97e0 con 0x5615fba50370 2024-06-04T05:10:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.349+0000 7f8bd8ff9700 1 -- 172.21.15.70:0/3010362008 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be40a9340 con 0x5615fba50370 2024-06-04T05:10:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.350+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/3010362008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba50370 msgr2=0x5615fba507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.350+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/3010362008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba50370 0x5615fba507b0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f8be4004d70 tx=0x7f8be400cd70 comp rx=0 tx=0).stop 2024-06-04T05:10:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.350+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/3010362008 shutdown_connections 2024-06-04T05:10:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.350+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/3010362008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba50370 0x5615fba507b0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.350+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/3010362008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba4bc70 0x5615fba4c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.350+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/3010362008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba4c620 0x5615fba4fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.350+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/3010362008 >> 172.21.15.70:0/3010362008 conn(0x5615fb975080 msgr2=0x5615fba41e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.350+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/3010362008 shutdown_connections 2024-06-04T05:10:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.350+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/3010362008 wait complete. 2024-06-04T05:10:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.350+0000 7f8bfb52ab80 1 -- start start 2024-06-04T05:10:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.350+0000 7f8bfb52ab80 1 Processor -- start 2024-06-04T05:10:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.350+0000 7f8bfb52ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba4bc70 0x5615fba577e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8bfb52ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba4c620 0x5615fba54ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8bfb52ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba50370 0x5615fba55020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8bfb52ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5615fb89cfa0 con 0x5615fba4c620 2024-06-04T05:10:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8bfb52ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5615fb9387c0 con 0x5615fba50370 2024-06-04T05:10:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8bfb52ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5615fba58530 con 0x5615fba4bc70 2024-06-04T05:10:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8bfb52ab80 10 client.0 _ll_register_callbacks cb 0x5615fb931920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8be9d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba4c620 0x5615fba54ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8be9d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba4c620 0x5615fba54ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57514/0 (socket says 172.21.15.70:57514) 2024-06-04T05:10:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8be9d9b700 1 -- 172.21.15.70:0/4056257400 learned_addr learned my addr 172.21.15.70:0/4056257400 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8bea59c700 1 --2- 172.21.15.70:0/4056257400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba4bc70 0x5615fba577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8bead9d700 1 --2- 172.21.15.70:0/4056257400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba50370 0x5615fba55020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8be9d9b700 1 -- 172.21.15.70:0/4056257400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba4bc70 msgr2=0x5615fba577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8be9d9b700 1 --2- 172.21.15.70:0/4056257400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba4bc70 0x5615fba577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8be9d9b700 1 -- 172.21.15.70:0/4056257400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba50370 msgr2=0x5615fba55020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8be9d9b700 1 --2- 172.21.15.70:0/4056257400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba50370 0x5615fba55020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8be9d9b700 1 -- 172.21.15.70:0/4056257400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615fb935690 con 0x5615fba4c620 2024-06-04T05:10:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8be9d9b700 1 --2- 172.21.15.70:0/4056257400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba4c620 0x5615fba54ae0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009990 tx=0x7f8be0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c565689bf3dae8d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8bd1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8bd1ffb700 1 -- 172.21.15.70:0/4056257400 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be0003bd0 con 0x5615fba4c620 2024-06-04T05:10:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8bd1ffb700 1 -- 172.21.15.70:0/4056257400 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f8be0003d70 con 0x5615fba4c620 2024-06-04T05:10:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.351+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/4056257400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8be40a97e0 con 0x5615fba4c620 2024-06-04T05:10:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.352+0000 7f8bd1ffb700 1 -- 172.21.15.70:0/4056257400 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8be0031050 con 0x5615fba4c620 2024-06-04T05:10:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.352+0000 7f8bfb52ab80 10 client.19143 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:58.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.352+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/4056257400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5615fba62cb0 con 0x5615fba4c620 2024-06-04T05:10:58.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.352+0000 7f8bd1ffb700 1 -- 172.21.15.70:0/4056257400 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8be40a97e0 con 0x5615fba4c620 2024-06-04T05:10:58.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.352+0000 7f8bd1ffb700 1 -- 172.21.15.70:0/4056257400 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f8be0023590 con 0x5615fba4c620 2024-06-04T05:10:58.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.352+0000 7f8bfb52ab80 10 client.19143 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:58.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.352+0000 7f8bfb52ab80 10 client.19143 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:10:58.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.352+0000 7f8bfb52ab80 20 client.19143 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:58.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.352+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/4056257400 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5615fba65390 0x5615fba67780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.352+0000 7f8bfb52ab80 4 client.19143 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:10:58.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.352+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/4056257400 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615fba139e0 con 0x5615fba65390 2024-06-04T05:10:58.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.352+0000 7f8bea59c700 1 --2- 172.21.15.70:0/4056257400 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5615fba65390 0x5615fba67780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.353+0000 7f8bd1ffb700 10 client.19143 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:10:58.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.353+0000 7f8bea59c700 1 --2- 172.21.15.70:0/4056257400 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5615fba65390 0x5615fba67780 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=42f101802d703a43 server_cookie=2a0687057cfb867f in_seq=0 out_seq=0 2024-06-04T05:10:58.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.353+0000 7f8bd1ffb700 1 -- 172.21.15.70:0/4056257400 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5615fba139e0 con 0x5615fba65390 2024-06-04T05:10:58.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.354+0000 7f8bd1ffb700 10 client.19143 handle_command_reply: tid=0 2024-06-04T05:10:58.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.354+0000 7f8bfb52ab80 1 client.19143 shutdown 2024-06-04T05:10:58.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.354+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/4056257400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba4c620 msgr2=0x5615fba54ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:58.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.354+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/4056257400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba4c620 0x5615fba54ae0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009990 tx=0x7f8be0005800 comp rx=0 tx=0).stop 2024-06-04T05:10:58.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.354+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/4056257400 shutdown_connections 2024-06-04T05:10:58.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.354+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/4056257400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba50370 0x5615fba55020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.354+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/4056257400 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5615fba65390 0x5615fba67780 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.354+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/4056257400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba4bc70 0x5615fba577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.354+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/4056257400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba4c620 0x5615fba54ae0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.354+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/4056257400 >> 172.21.15.70:0/4056257400 conn(0x5615fb975080 msgr2=0x5615fba51760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.354+0000 7f8bd1ffb700 0 client.19143 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:10:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.354+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/4056257400 shutdown_connections 2024-06-04T05:10:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.354+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/4056257400 wait complete. 2024-06-04T05:10:58.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.354+0000 7f8bfb52ab80 20 client.19143 trim_cache size 0 max 16384 2024-06-04T05:10:58.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8bfb52ab80 1 Processor -- start 2024-06-04T05:10:58.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8bfb52ab80 1 -- start start 2024-06-04T05:10:58.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8bfb52ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba4bc70 0x5615fba42530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8bfb52ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba4c620 0x5615fba6d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8bfb52ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba50370 0x5615fba6d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8bfb52ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8bec1abbe0 con 0x5615fba50370 2024-06-04T05:10:58.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8bfb52ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8bec1ad840 con 0x5615fba4bc70 2024-06-04T05:10:58.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8bfb52ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8bec1ac2b0 con 0x5615fba4c620 2024-06-04T05:10:58.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8bead9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba50370 0x5615fba6d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8bead9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba50370 0x5615fba6d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57524/0 (socket says 172.21.15.70:57524) 2024-06-04T05:10:58.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8bead9d700 1 -- 172.21.15.70:0/2837532909 learned_addr learned my addr 172.21.15.70:0/2837532909 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:58.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8bea59c700 1 --2- 172.21.15.70:0/2837532909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba4bc70 0x5615fba42530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8be9d9b700 1 --2- 172.21.15.70:0/2837532909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba4c620 0x5615fba6d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8bead9d700 1 -- 172.21.15.70:0/2837532909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba4c620 msgr2=0x5615fba6d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:58.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8bead9d700 1 --2- 172.21.15.70:0/2837532909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba4c620 0x5615fba6d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8bead9d700 1 -- 172.21.15.70:0/2837532909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba4bc70 msgr2=0x5615fba42530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:58.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8bead9d700 1 --2- 172.21.15.70:0/2837532909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba4bc70 0x5615fba42530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.369+0000 7f8bead9d700 1 -- 172.21.15.70:0/2837532909 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615fba53070 con 0x5615fba50370 2024-06-04T05:10:58.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.370+0000 7f8bead9d700 1 --2- 172.21.15.70:0/2837532909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba50370 0x5615fba6d590 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f8be400ad70 tx=0x7f8be4004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a94f8f45a752b573 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:58.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.370+0000 7f8bd3fff700 1 -- 172.21.15.70:0/2837532909 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be40a97e0 con 0x5615fba50370 2024-06-04T05:10:58.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.370+0000 7f8bd3fff700 1 -- 172.21.15.70:0/2837532909 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8be40ad490 con 0x5615fba50370 2024-06-04T05:10:58.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.370+0000 7f8bd3fff700 1 -- 172.21.15.70:0/2837532909 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be40a95d0 con 0x5615fba50370 2024-06-04T05:10:58.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.370+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/2837532909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba50370 msgr2=0x5615fba6d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:58.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.370+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/2837532909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba50370 0x5615fba6d590 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f8be400ad70 tx=0x7f8be4004d00 comp rx=0 tx=0).stop 2024-06-04T05:10:58.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/2837532909 shutdown_connections 2024-06-04T05:10:58.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/2837532909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba50370 0x5615fba6d590 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/2837532909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba4bc70 0x5615fba42530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/2837532909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba4c620 0x5615fba6d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/2837532909 >> 172.21.15.70:0/2837532909 conn(0x5615fb975080 msgr2=0x5615fba41e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:58.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/2837532909 shutdown_connections 2024-06-04T05:10:58.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/2837532909 wait complete. 2024-06-04T05:10:58.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bfb52ab80 1 -- start start 2024-06-04T05:10:58.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bfb52ab80 1 Processor -- start 2024-06-04T05:10:58.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bfb52ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba4bc70 0x5615fba5d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bfb52ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba4c620 0x5615fba5dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bfb52ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba50370 0x5615fba4d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bfb52ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8bec1ad110 con 0x5615fba4bc70 2024-06-04T05:10:58.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bfb52ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8bec1ac9e0 con 0x5615fba4c620 2024-06-04T05:10:58.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bfb52ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8bec1a9bc0 con 0x5615fba50370 2024-06-04T05:10:58.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bfb52ab80 10 client.0 _ll_register_callbacks cb 0x5615fb931920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:10:58.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bea59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba4bc70 0x5615fba5d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bea59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba4bc70 0x5615fba5d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57530/0 (socket says 172.21.15.70:57530) 2024-06-04T05:10:58.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.371+0000 7f8bea59c700 1 -- 172.21.15.70:0/929311216 learned_addr learned my addr 172.21.15.70:0/929311216 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:10:58.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.372+0000 7f8be9d9b700 1 --2- 172.21.15.70:0/929311216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba4c620 0x5615fba5dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.372+0000 7f8bead9d700 1 --2- 172.21.15.70:0/929311216 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba50370 0x5615fba4d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.372+0000 7f8bea59c700 1 -- 172.21.15.70:0/929311216 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba50370 msgr2=0x5615fba4d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:58.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.372+0000 7f8bea59c700 1 --2- 172.21.15.70:0/929311216 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba50370 0x5615fba4d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.372+0000 7f8bea59c700 1 -- 172.21.15.70:0/929311216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba4c620 msgr2=0x5615fba5dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:10:58.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.372+0000 7f8bea59c700 1 --2- 172.21.15.70:0/929311216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba4c620 0x5615fba5dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.372+0000 7f8bea59c700 1 -- 172.21.15.70:0/929311216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615fba4d930 con 0x5615fba4bc70 2024-06-04T05:10:58.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.372+0000 7f8bea59c700 1 --2- 172.21.15.70:0/929311216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba4bc70 0x5615fba5d9a0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc0114c0 tx=0x7f8bdc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34d628fc5ce1c7fc server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:10:58.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.372+0000 7f8bd1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:10:58.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.372+0000 7f8bd1ffb700 1 -- 172.21.15.70:0/929311216 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc011e50 con 0x5615fba4bc70 2024-06-04T05:10:58.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.372+0000 7f8bd1ffb700 1 -- 172.21.15.70:0/929311216 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f8bdc02a380 con 0x5615fba4bc70 2024-06-04T05:10:58.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.372+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/929311216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5615fb9a8be0 con 0x5615fba4bc70 2024-06-04T05:10:58.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.372+0000 7f8bd1ffb700 1 -- 172.21.15.70:0/929311216 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8bdc02a580 con 0x5615fba4bc70 2024-06-04T05:10:58.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.372+0000 7f8bfb52ab80 10 client.19149 fetch_fsmap learned FSMap version 38 2024-06-04T05:10:58.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.372+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/929311216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5615fbacf920 con 0x5615fba4bc70 2024-06-04T05:10:58.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.373+0000 7f8bd1ffb700 1 -- 172.21.15.70:0/929311216 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x5615fb9a8be0 con 0x5615fba4bc70 2024-06-04T05:10:58.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.373+0000 7f8bd1ffb700 1 -- 172.21.15.70:0/929311216 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f8bdc01f910 con 0x5615fba4bc70 2024-06-04T05:10:58.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.373+0000 7f8bfb52ab80 10 client.19149 fetch_fsmap finished waiting for FSMap version 38 2024-06-04T05:10:58.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.373+0000 7f8bfb52ab80 10 client.19149 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:10:58.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.373+0000 7f8bfb52ab80 20 client.19149 populate_metadata read hostname 'smithi070' 2024-06-04T05:10:58.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.373+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/929311216 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5615fbad0080 0x5615fbad24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:10:58.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.373+0000 7f8bfb52ab80 4 client.19149 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:10:58.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.373+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/929311216 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5615fb9a23c0 con 0x5615fbad0080 2024-06-04T05:10:58.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.373+0000 7f8be9d9b700 1 --2- 172.21.15.70:0/929311216 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5615fbad0080 0x5615fbad24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:10:58.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.374+0000 7f8be9d9b700 1 --2- 172.21.15.70:0/929311216 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5615fbad0080 0x5615fbad24a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b14242fa25700d47 server_cookie=57a785da0fec9f86 in_seq=0 out_seq=0 2024-06-04T05:10:58.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.374+0000 7f8bd1ffb700 10 client.19149 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:10:58.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.374+0000 7f8bd1ffb700 1 -- 172.21.15.70:0/929311216 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5615fb9a23c0 con 0x5615fbad0080 2024-06-04T05:10:58.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.374+0000 7f8bd1ffb700 10 client.19149 handle_command_reply: tid=0 2024-06-04T05:10:58.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.374+0000 7f8bfb52ab80 1 client.19149 shutdown 2024-06-04T05:10:58.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.374+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/929311216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba4bc70 msgr2=0x5615fba5d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:58.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.374+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/929311216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba4bc70 0x5615fba5d9a0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc0114c0 tx=0x7f8bdc0047c0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.374+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/929311216 shutdown_connections 2024-06-04T05:10:58.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.374+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/929311216 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5615fbad0080 0x5615fbad24a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.374+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/929311216 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fba50370 0x5615fba4d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.374+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/929311216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fba4bc70 0x5615fba5d9a0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.374+0000 7f8bfb52ab80 1 --2- 172.21.15.70:0/929311216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fba4c620 0x5615fba5dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.374+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/929311216 >> 172.21.15.70:0/929311216 conn(0x5615fb975080 msgr2=0x5615fba46670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:58.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.374+0000 7f8bd1ffb700 0 client.19149 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:10:58.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.374+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/929311216 shutdown_connections 2024-06-04T05:10:58.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.374+0000 7f8bfb52ab80 1 -- 172.21.15.70:0/929311216 wait complete. 2024-06-04T05:10:58.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.374+0000 7f8bfb52ab80 20 client.19149 trim_cache size 0 max 16384 2024-06-04T05:10:58.385 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:10:58.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.375+0000 7f8bf0c95700 1 -- 172.21.15.70:0/155646730 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8bd40b29f0 msgr2=0x7f8bd40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.375+0000 7f8bf0c95700 1 --2- 172.21.15.70:0/155646730 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8bd40b29f0 0x7f8bd40b4eb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8bec19aac0 tx=0x7f8bdc00b410 comp rx=0 tx=0).stop 2024-06-04T05:10:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.375+0000 7f8bf0c95700 1 -- 172.21.15.70:0/155646730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bec1095a0 msgr2=0x7f8bec199980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:10:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.375+0000 7f8bf0c95700 1 --2- 172.21.15.70:0/155646730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bec1095a0 0x7f8bec199980 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f8be400ada0 tx=0x7f8be400c330 comp rx=0 tx=0).stop 2024-06-04T05:10:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.375+0000 7f8bf0c95700 1 -- 172.21.15.70:0/155646730 shutdown_connections 2024-06-04T05:10:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.375+0000 7f8bf0c95700 1 --2- 172.21.15.70:0/155646730 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8bd40b29f0 0x7f8bd40b4eb0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.375+0000 7f8bf0c95700 1 --2- 172.21.15.70:0/155646730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bec1095a0 0x7f8bec199980 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.375+0000 7f8bf0c95700 1 --2- 172.21.15.70:0/155646730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bec06bab0 0x7f8bec19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.375+0000 7f8bf0c95700 1 --2- 172.21.15.70:0/155646730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bec06c470 0x7f8bec19fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:10:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.375+0000 7f8bf0c95700 1 -- 172.21.15.70:0/155646730 >> 172.21.15.70:0/155646730 conn(0x7f8bec0793b0 msgr2=0x7f8bec101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:10:58.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.375+0000 7f8bf0c95700 1 -- 172.21.15.70:0/155646730 shutdown_connections 2024-06-04T05:10:58.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:10:58.375+0000 7f8bf0c95700 1 -- 172.21.15.70:0/155646730 wait complete. 2024-06-04T05:10:58.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 216558 1.56 MB/sec execute 591 sec latency 1469.533 ms 2024-06-04T05:10:59.175 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:10:59.414 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:10:59.537 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:10:59 smithi070 ceph-mon[32631]: pgmap v429: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1005 KiB/s wr, 97 op/s 2024-06-04T05:10:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:10:59 smithi112 ceph-mon[36453]: pgmap v429: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1005 KiB/s wr, 97 op/s 2024-06-04T05:10:59.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 216558 1.56 MB/sec execute 592 sec latency 2469.619 ms 2024-06-04T05:10:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:10:59 smithi098 ceph-mon[35569]: pgmap v429: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1005 KiB/s wr, 97 op/s 2024-06-04T05:11:00.584 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:00 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:00.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:00 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:00.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 216558 1.56 MB/sec execute 593 sec latency 3469.737 ms 2024-06-04T05:11:00.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:00 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:01.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.155+0000 7f6ac994e700 1 -- 172.21.15.70:0/2384108086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4102430 msgr2=0x7f6ac41028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:01.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.155+0000 7f6ac994e700 1 --2- 172.21.15.70:0/2384108086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4102430 0x7f6ac41028b0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f6aac009990 tx=0x7f6aac009ca0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.155+0000 7f6ac994e700 1 -- 172.21.15.70:0/2384108086 shutdown_connections 2024-06-04T05:11:01.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.155+0000 7f6ac994e700 1 --2- 172.21.15.70:0/2384108086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ac4102df0 0x7f6ac410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.155+0000 7f6ac994e700 1 --2- 172.21.15.70:0/2384108086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ac4108510 0x7f6ac41088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.155+0000 7f6ac994e700 1 --2- 172.21.15.70:0/2384108086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4102430 0x7f6ac41028b0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.155+0000 7f6ac994e700 1 -- 172.21.15.70:0/2384108086 >> 172.21.15.70:0/2384108086 conn(0x7f6ac40793b0 msgr2=0x7f6ac40797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:01.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.155+0000 7f6ac994e700 1 -- 172.21.15.70:0/2384108086 shutdown_connections 2024-06-04T05:11:01.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.155+0000 7f6ac994e700 1 -- 172.21.15.70:0/2384108086 wait complete. 2024-06-04T05:11:01.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.155+0000 7f6ac994e700 1 Processor -- start 2024-06-04T05:11:01.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac994e700 1 -- start start 2024-06-04T05:11:01.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac994e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ac4102430 0x7f6ac4104070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac994e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4102df0 0x7f6ac41045b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac994e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ac4108510 0x7f6ac4104af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac994e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac4112490 con 0x7f6ac4102df0 2024-06-04T05:11:01.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac994e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac4112310 con 0x7f6ac4108510 2024-06-04T05:11:01.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac994e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac4112610 con 0x7f6ac4102430 2024-06-04T05:11:01.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac27fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4102df0 0x7f6ac41045b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac27fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4102df0 0x7f6ac41045b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57556/0 (socket says 172.21.15.70:57556) 2024-06-04T05:11:01.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac27fc700 1 -- 172.21.15.70:0/2010435662 learned_addr learned my addr 172.21.15.70:0/2010435662 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:01.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac2ffd700 1 --2- 172.21.15.70:0/2010435662 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ac4102430 0x7f6ac4104070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac37fe700 1 --2- 172.21.15.70:0/2010435662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ac4108510 0x7f6ac4104af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac27fc700 1 -- 172.21.15.70:0/2010435662 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ac4102430 msgr2=0x7f6ac4104070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:01.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac27fc700 1 --2- 172.21.15.70:0/2010435662 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ac4102430 0x7f6ac4104070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac27fc700 1 -- 172.21.15.70:0/2010435662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ac4108510 msgr2=0x7f6ac4104af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:01.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac27fc700 1 --2- 172.21.15.70:0/2010435662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ac4108510 0x7f6ac4104af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac27fc700 1 -- 172.21.15.70:0/2010435662 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac41a8120 con 0x7f6ac4102df0 2024-06-04T05:11:01.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.156+0000 7f6ac27fc700 1 --2- 172.21.15.70:0/2010435662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4102df0 0x7f6ac41045b0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f6aac000c00 tx=0x7f6aac0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:01.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.157+0000 7f6ac894c700 1 -- 172.21.15.70:0/2010435662 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aac027070 con 0x7f6ac4102df0 2024-06-04T05:11:01.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.157+0000 7f6ac894c700 1 -- 172.21.15.70:0/2010435662 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6aac00fd00 con 0x7f6ac4102df0 2024-06-04T05:11:01.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.157+0000 7f6ac994e700 1 -- 172.21.15.70:0/2010435662 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac41a83b0 con 0x7f6ac4102df0 2024-06-04T05:11:01.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.157+0000 7f6ac894c700 1 -- 172.21.15.70:0/2010435662 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aac022e20 con 0x7f6ac4102df0 2024-06-04T05:11:01.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.157+0000 7f6ac994e700 1 -- 172.21.15.70:0/2010435662 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ac41a8800 con 0x7f6ac4102df0 2024-06-04T05:11:01.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.160+0000 7f6ac894c700 1 -- 172.21.15.70:0/2010435662 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f6aac02a070 con 0x7f6ac4102df0 2024-06-04T05:11:01.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.161+0000 7f6ac894c700 1 --2- 172.21.15.70:0/2010435662 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ab00b29f0 0x7f6ab00b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.161+0000 7f6ac2ffd700 1 --2- 172.21.15.70:0/2010435662 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ab00b29f0 0x7f6ab00b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.161+0000 7f6ac894c700 1 -- 172.21.15.70:0/2010435662 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f6aac0e3470 con 0x7f6ac4102df0 2024-06-04T05:11:01.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.161+0000 7f6ad41e3b80 1 Processor -- start 2024-06-04T05:11:01.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.161+0000 7f6ad41e3b80 1 -- start start 2024-06-04T05:11:01.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.161+0000 7f6ac2ffd700 1 --2- 172.21.15.70:0/2010435662 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ab00b29f0 0x7f6ab00b4eb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4007fd0 tx=0x7f6ab400c040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:01.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.161+0000 7f6ad41e3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604ce9ddc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.161+0000 7f6ad41e3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9de210 0x5604ce9e1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.161+0000 7f6ad41e3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9e1f60 0x5604ce9e23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.161+0000 7f6ad41e3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5604ce93cc90 con 0x5604ce9dd860 2024-06-04T05:11:01.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.161+0000 7f6ad41e3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5604ce92c830 con 0x5604ce9de210 2024-06-04T05:11:01.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.161+0000 7f6ad41e3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5604ce930870 con 0x5604ce9e1f60 2024-06-04T05:11:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.161+0000 7f6ac2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604ce9ddc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.161+0000 7f6ac2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604ce9ddc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57566/0 (socket says 172.21.15.70:57566) 2024-06-04T05:11:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.161+0000 7f6ac2ffd700 1 -- 172.21.15.70:0/2664208617 learned_addr learned my addr 172.21.15.70:0/2664208617 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.162+0000 7f6ac37fe700 1 --2- 172.21.15.70:0/2664208617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9e1f60 0x5604ce9e23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.162+0000 7f6ac27fc700 1 --2- 172.21.15.70:0/2664208617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9de210 0x5604ce9e1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.162+0000 7f6ac2ffd700 1 -- 172.21.15.70:0/2664208617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9e1f60 msgr2=0x5604ce9e23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.162+0000 7f6ac2ffd700 1 --2- 172.21.15.70:0/2664208617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9e1f60 0x5604ce9e23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.162+0000 7f6ac2ffd700 1 -- 172.21.15.70:0/2664208617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9de210 msgr2=0x5604ce9e1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.162+0000 7f6ac2ffd700 1 --2- 172.21.15.70:0/2664208617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9de210 0x5604ce9e1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.162+0000 7f6ac2ffd700 1 -- 172.21.15.70:0/2664208617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604ce87c8d0 con 0x5604ce9dd860 2024-06-04T05:11:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.162+0000 7f6ac37fe700 1 --2- 172.21.15.70:0/2664208617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9e1f60 0x5604ce9e23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:11:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.162+0000 7f6ac27fc700 1 --2- 172.21.15.70:0/2664208617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9de210 0x5604ce9e1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:11:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.162+0000 7f6ac2ffd700 1 --2- 172.21.15.70:0/2664208617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604ce9ddc40 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f6ab40167a0 tx=0x7f6ab4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d10658772b1dde4b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:01.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.162+0000 7f6ab9ffb700 1 -- 172.21.15.70:0/2664208617 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab401c040 con 0x5604ce9dd860 2024-06-04T05:11:01.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.162+0000 7f6ab9ffb700 1 -- 172.21.15.70:0/2664208617 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ab4025070 con 0x5604ce9dd860 2024-06-04T05:11:01.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.162+0000 7f6ab9ffb700 1 -- 172.21.15.70:0/2664208617 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4014890 con 0x5604ce9dd860 2024-06-04T05:11:01.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.163+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2664208617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 msgr2=0x5604ce9ddc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:01.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.163+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/2664208617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604ce9ddc40 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f6ab40167a0 tx=0x7f6ab4012000 comp rx=0 tx=0).stop 2024-06-04T05:11:01.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.163+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2664208617 shutdown_connections 2024-06-04T05:11:01.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.163+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/2664208617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9e1f60 0x5604ce9e23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.163+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/2664208617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604ce9ddc40 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.163+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/2664208617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9de210 0x5604ce9e1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.163+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2664208617 >> 172.21.15.70:0/2664208617 conn(0x5604ce90b080 msgr2=0x5604ce9d7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:01.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.163+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2664208617 shutdown_connections 2024-06-04T05:11:01.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.163+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2664208617 wait complete. 2024-06-04T05:11:01.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.163+0000 7f6ad41e3b80 1 -- start start 2024-06-04T05:11:01.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.163+0000 7f6ad41e3b80 1 Processor -- start 2024-06-04T05:11:01.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.164+0000 7f6ad41e3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604ce9e93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.164+0000 7f6ad41e3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9de210 0x5604ce9e66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.164+0000 7f6ad41e3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9e1f60 0x5604ce9e6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.164+0000 7f6ac2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604ce9e93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.164+0000 7f6ad41e3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5604ce832fa0 con 0x5604ce9dd860 2024-06-04T05:11:01.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.164+0000 7f6ad41e3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5604ce8ce7c0 con 0x5604ce9de210 2024-06-04T05:11:01.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.164+0000 7f6ad41e3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5604ce9ea120 con 0x5604ce9e1f60 2024-06-04T05:11:01.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.164+0000 7f6ad41e3b80 10 client.0 _ll_register_callbacks cb 0x5604ce8c7920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:01.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.164+0000 7f6ac2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604ce9e93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57570/0 (socket says 172.21.15.70:57570) 2024-06-04T05:11:01.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.164+0000 7f6ac2ffd700 1 -- 172.21.15.70:0/2525983908 learned_addr learned my addr 172.21.15.70:0/2525983908 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:01.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.164+0000 7f6ac27fc700 1 --2- 172.21.15.70:0/2525983908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9de210 0x5604ce9e66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.164+0000 7f6ac37fe700 1 --2- 172.21.15.70:0/2525983908 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9e1f60 0x5604ce9e6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.164+0000 7f6ac2ffd700 1 -- 172.21.15.70:0/2525983908 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9e1f60 msgr2=0x5604ce9e6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:01.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.164+0000 7f6ac2ffd700 1 --2- 172.21.15.70:0/2525983908 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9e1f60 0x5604ce9e6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.164+0000 7f6ac2ffd700 1 -- 172.21.15.70:0/2525983908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9de210 msgr2=0x5604ce9e66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:01.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.164+0000 7f6ac2ffd700 1 --2- 172.21.15.70:0/2525983908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9de210 0x5604ce9e66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.164+0000 7f6ac2ffd700 1 -- 172.21.15.70:0/2525983908 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604ce8cb690 con 0x5604ce9dd860 2024-06-04T05:11:01.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.165+0000 7f6ac2ffd700 1 --2- 172.21.15.70:0/2525983908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604ce9e93f0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4016770 tx=0x7f6ab4005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3a70243c48ddac server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:01.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.165+0000 7f6aa2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:11:01.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.165+0000 7f6aa2ffd700 1 -- 172.21.15.70:0/2525983908 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4014a60 con 0x5604ce9dd860 2024-06-04T05:11:01.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.165+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2525983908 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ab4025070 con 0x5604ce9dd860 2024-06-04T05:11:01.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.165+0000 7f6aa2ffd700 1 -- 172.21.15.70:0/2525983908 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f6ab4014c00 con 0x5604ce9dd860 2024-06-04T05:11:01.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.165+0000 7f6aa2ffd700 1 -- 172.21.15.70:0/2525983908 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f6ab4014e00 con 0x5604ce9dd860 2024-06-04T05:11:01.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.165+0000 7f6ad41e3b80 10 client.19161 fetch_fsmap learned FSMap version 39 2024-06-04T05:11:01.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.165+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2525983908 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5604ce9f4860 con 0x5604ce9dd860 2024-06-04T05:11:01.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.165+0000 7f6aa2ffd700 1 -- 172.21.15.70:0/2525983908 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ab4025070 con 0x5604ce9dd860 2024-06-04T05:11:01.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.165+0000 7f6aa2ffd700 1 -- 172.21.15.70:0/2525983908 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 39) ==== 8374+0+0 (secure 0 0 0) 0x7f6ab401f030 con 0x5604ce9dd860 2024-06-04T05:11:01.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.165+0000 7f6ad41e3b80 10 client.19161 fetch_fsmap finished waiting for FSMap version 39 2024-06-04T05:11:01.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.165+0000 7f6ad41e3b80 10 client.19161 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:11:01.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.165+0000 7f6ad41e3b80 20 client.19161 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:01.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.165+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/2525983908 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5604ce9f7170 0x5604ce9f95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.165+0000 7f6ad41e3b80 4 client.19161 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:11:01.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.165+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2525983908 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604ce9a99e0 con 0x5604ce9f7170 2024-06-04T05:11:01.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.166+0000 7f6ac27fc700 1 --2- 172.21.15.70:0/2525983908 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5604ce9f7170 0x5604ce9f95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.166+0000 7f6aa2ffd700 10 client.19161 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:11:01.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.166+0000 7f6ac27fc700 1 --2- 172.21.15.70:0/2525983908 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5604ce9f7170 0x5604ce9f95b0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9ff91f21b8f3d577 server_cookie=111b849a86cfb75c in_seq=0 out_seq=0 2024-06-04T05:11:01.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.167+0000 7f6aa2ffd700 1 -- 172.21.15.70:0/2525983908 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5604ce9a99e0 con 0x5604ce9f7170 2024-06-04T05:11:01.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.167+0000 7f6aa2ffd700 10 client.19161 handle_command_reply: tid=0 2024-06-04T05:11:01.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.167+0000 7f6ad41e3b80 1 client.19161 shutdown 2024-06-04T05:11:01.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.167+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2525983908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 msgr2=0x5604ce9e93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:01.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.167+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/2525983908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604ce9e93f0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4016770 tx=0x7f6ab4005b50 comp rx=0 tx=0).stop 2024-06-04T05:11:01.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.167+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2525983908 shutdown_connections 2024-06-04T05:11:01.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.167+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/2525983908 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9e1f60 0x5604ce9e6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.167+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/2525983908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604ce9e93f0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.167+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/2525983908 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5604ce9f7170 0x5604ce9f95b0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.167+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/2525983908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9de210 0x5604ce9e66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.167+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2525983908 >> 172.21.15.70:0/2525983908 conn(0x5604ce90b080 msgr2=0x5604ce9d7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:01.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.167+0000 7f6aa2ffd700 0 client.19161 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:11:01.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.167+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2525983908 shutdown_connections 2024-06-04T05:11:01.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.167+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2525983908 wait complete. 2024-06-04T05:11:01.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.167+0000 7f6ad41e3b80 20 client.19161 trim_cache size 0 max 16384 2024-06-04T05:11:01.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.182+0000 7f6ad41e3b80 1 Processor -- start 2024-06-04T05:11:01.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.182+0000 7f6ad41e3b80 1 -- start start 2024-06-04T05:11:01.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.182+0000 7f6ad41e3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9dd860 0x5604ce9dd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.182+0000 7f6ad41e3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9de210 0x5604ce9d7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.182+0000 7f6ad41e3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9e1f60 0x5604ce9d81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.182+0000 7f6ad41e3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac41aff90 con 0x5604ce9e1f60 2024-06-04T05:11:01.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.182+0000 7f6ad41e3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac41b1bc0 con 0x5604ce9dd860 2024-06-04T05:11:01.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.182+0000 7f6ad41e3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac41b0660 con 0x5604ce9de210 2024-06-04T05:11:01.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.182+0000 7f6ac37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9e1f60 0x5604ce9d81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.182+0000 7f6ac37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9e1f60 0x5604ce9d81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57574/0 (socket says 172.21.15.70:57574) 2024-06-04T05:11:01.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.182+0000 7f6ac37fe700 1 -- 172.21.15.70:0/3887285980 learned_addr learned my addr 172.21.15.70:0/3887285980 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:01.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.183+0000 7f6ac2ffd700 1 --2- 172.21.15.70:0/3887285980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9dd860 0x5604ce9dd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.183+0000 7f6ac27fc700 1 --2- 172.21.15.70:0/3887285980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9de210 0x5604ce9d7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.183+0000 7f6ac37fe700 1 -- 172.21.15.70:0/3887285980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9de210 msgr2=0x5604ce9d7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:01.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.183+0000 7f6ac37fe700 1 --2- 172.21.15.70:0/3887285980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9de210 0x5604ce9d7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.183+0000 7f6ac37fe700 1 -- 172.21.15.70:0/3887285980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9dd860 msgr2=0x5604ce9dd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:01.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.183+0000 7f6ac37fe700 1 --2- 172.21.15.70:0/3887285980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9dd860 0x5604ce9dd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.183+0000 7f6ac37fe700 1 -- 172.21.15.70:0/3887285980 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604ce9e4c80 con 0x5604ce9e1f60 2024-06-04T05:11:01.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.183+0000 7f6ac27fc700 1 --2- 172.21.15.70:0/3887285980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9de210 0x5604ce9d7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:11:01.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.183+0000 7f6ac37fe700 1 --2- 172.21.15.70:0/3887285980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9e1f60 0x5604ce9d81d0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00db80 tx=0x7f6abc00de90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3d5cf684516033 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:01.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.183+0000 7f6ab97fa700 1 -- 172.21.15.70:0/3887285980 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6abc01e070 con 0x5604ce9e1f60 2024-06-04T05:11:01.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.183+0000 7f6ab97fa700 1 -- 172.21.15.70:0/3887285980 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6abc010bd0 con 0x5604ce9e1f60 2024-06-04T05:11:01.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.183+0000 7f6ab97fa700 1 -- 172.21.15.70:0/3887285980 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6abc00f680 con 0x5604ce9e1f60 2024-06-04T05:11:01.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.184+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/3887285980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9e1f60 msgr2=0x5604ce9d81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:01.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.184+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/3887285980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9e1f60 0x5604ce9d81d0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00db80 tx=0x7f6abc00de90 comp rx=0 tx=0).stop 2024-06-04T05:11:01.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.184+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/3887285980 shutdown_connections 2024-06-04T05:11:01.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.184+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/3887285980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9e1f60 0x5604ce9d81d0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.184+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/3887285980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9dd860 0x5604ce9dd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.184+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/3887285980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9de210 0x5604ce9d7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.184+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/3887285980 >> 172.21.15.70:0/3887285980 conn(0x5604ce90b080 msgr2=0x5604ce9e87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:01.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.184+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/3887285980 shutdown_connections 2024-06-04T05:11:01.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.184+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/3887285980 wait complete. 2024-06-04T05:11:01.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.184+0000 7f6ad41e3b80 1 -- start start 2024-06-04T05:11:01.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ad41e3b80 1 Processor -- start 2024-06-04T05:11:01.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ad41e3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604cea59040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ad41e3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9de210 0x5604cea59580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ad41e3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9e1f60 0x5604ce9fe390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ad41e3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac41b1490 con 0x5604ce9dd860 2024-06-04T05:11:01.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ad41e3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac41b0d60 con 0x5604ce9de210 2024-06-04T05:11:01.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ad41e3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac41adf50 con 0x5604ce9e1f60 2024-06-04T05:11:01.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ad41e3b80 10 client.0 _ll_register_callbacks cb 0x5604ce8c7920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:01.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ac2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604cea59040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ac2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604cea59040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57586/0 (socket says 172.21.15.70:57586) 2024-06-04T05:11:01.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ac2ffd700 1 -- 172.21.15.70:0/2250682398 learned_addr learned my addr 172.21.15.70:0/2250682398 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:01.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ac37fe700 1 --2- 172.21.15.70:0/2250682398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9e1f60 0x5604ce9fe390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ac27fc700 1 --2- 172.21.15.70:0/2250682398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9de210 0x5604cea59580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ac2ffd700 1 -- 172.21.15.70:0/2250682398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9e1f60 msgr2=0x5604ce9fe390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:01.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ac2ffd700 1 --2- 172.21.15.70:0/2250682398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9e1f60 0x5604ce9fe390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ac2ffd700 1 -- 172.21.15.70:0/2250682398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9de210 msgr2=0x5604cea59580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:01.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ac2ffd700 1 --2- 172.21.15.70:0/2250682398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9de210 0x5604cea59580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ac2ffd700 1 -- 172.21.15.70:0/2250682398 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604ce9feb80 con 0x5604ce9dd860 2024-06-04T05:11:01.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.185+0000 7f6ac27fc700 1 --2- 172.21.15.70:0/2250682398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9de210 0x5604cea59580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:11:01.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.186+0000 7f6ac2ffd700 1 --2- 172.21.15.70:0/2250682398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604cea59040 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4016740 tx=0x7f6ab4014b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d628a254b5fb1e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:01.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.186+0000 7f6aa2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:11:01.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.186+0000 7f6aa2ffd700 1 -- 172.21.15.70:0/2250682398 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab403e070 con 0x5604ce9dd860 2024-06-04T05:11:01.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.186+0000 7f6aa2ffd700 1 -- 172.21.15.70:0/2250682398 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f6ab4032b20 con 0x5604ce9dd860 2024-06-04T05:11:01.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.186+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2250682398 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6abc010bd0 con 0x5604ce9dd860 2024-06-04T05:11:01.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.186+0000 7f6aa2ffd700 1 -- 172.21.15.70:0/2250682398 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f6ab404a050 con 0x5604ce9dd860 2024-06-04T05:11:01.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.186+0000 7f6ad41e3b80 10 client.19167 fetch_fsmap learned FSMap version 39 2024-06-04T05:11:01.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.186+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2250682398 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5604cea61770 con 0x5604ce9dd860 2024-06-04T05:11:01.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.186+0000 7f6aa2ffd700 1 -- 172.21.15.70:0/2250682398 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6abc010bd0 con 0x5604ce9dd860 2024-06-04T05:11:01.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.186+0000 7f6aa2ffd700 1 -- 172.21.15.70:0/2250682398 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 39) ==== 8374+0+0 (secure 0 0 0) 0x7f6ab401f070 con 0x5604ce9dd860 2024-06-04T05:11:01.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.186+0000 7f6ad41e3b80 10 client.19167 fetch_fsmap finished waiting for FSMap version 39 2024-06-04T05:11:01.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.186+0000 7f6ad41e3b80 10 client.19167 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:11:01.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.186+0000 7f6ad41e3b80 20 client.19167 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:01.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.186+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/2250682398 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5604cea61ee0 0x5604cea64300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:01.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.186+0000 7f6ad41e3b80 4 client.19167 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:11:01.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.186+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2250682398 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5604ce93ebe0 con 0x5604cea61ee0 2024-06-04T05:11:01.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.187+0000 7f6ac27fc700 1 --2- 172.21.15.70:0/2250682398 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5604cea61ee0 0x5604cea64300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:01.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.187+0000 7f6ac27fc700 1 --2- 172.21.15.70:0/2250682398 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5604cea61ee0 0x5604cea64300 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7c77e56d5511222 server_cookie=b525bb432bbe7304 in_seq=0 out_seq=0 2024-06-04T05:11:01.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.187+0000 7f6aa2ffd700 10 client.19167 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:11:01.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.187+0000 7f6aa2ffd700 1 -- 172.21.15.70:0/2250682398 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5604ce93ebe0 con 0x5604cea61ee0 2024-06-04T05:11:01.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.187+0000 7f6aa2ffd700 10 client.19167 handle_command_reply: tid=0 2024-06-04T05:11:01.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.187+0000 7f6ad41e3b80 1 client.19167 shutdown 2024-06-04T05:11:01.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.187+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2250682398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 msgr2=0x5604cea59040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:01.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.187+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/2250682398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604cea59040 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4016740 tx=0x7f6ab4014b10 comp rx=0 tx=0).stop 2024-06-04T05:11:01.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.187+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2250682398 shutdown_connections 2024-06-04T05:11:01.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.187+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/2250682398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604ce9e1f60 0x5604ce9fe390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.187+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/2250682398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604ce9dd860 0x5604cea59040 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.187+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/2250682398 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5604cea61ee0 0x5604cea64300 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.187+0000 7f6ad41e3b80 1 --2- 172.21.15.70:0/2250682398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ce9de210 0x5604cea59580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.187+0000 7f6aa2ffd700 0 client.19167 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:11:01.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.187+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2250682398 >> 172.21.15.70:0/2250682398 conn(0x5604ce90b080 msgr2=0x5604ce9e7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:01.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.188+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2250682398 shutdown_connections 2024-06-04T05:11:01.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.188+0000 7f6ad41e3b80 1 -- 172.21.15.70:0/2250682398 wait complete. 2024-06-04T05:11:01.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.188+0000 7f6ad41e3b80 20 client.19167 trim_cache size 0 max 16384 2024-06-04T05:11:01.197 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:11:01.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.188+0000 7f6ac994e700 1 -- 172.21.15.70:0/2010435662 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ab00b29f0 msgr2=0x7f6ab00b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:01.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.188+0000 7f6ac994e700 1 --2- 172.21.15.70:0/2010435662 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ab00b29f0 0x7f6ab00b4eb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4007fd0 tx=0x7f6ab400c040 comp rx=0 tx=0).stop 2024-06-04T05:11:01.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.188+0000 7f6ac994e700 1 -- 172.21.15.70:0/2010435662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4102df0 msgr2=0x7f6ac41045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:01.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.188+0000 7f6ac994e700 1 --2- 172.21.15.70:0/2010435662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4102df0 0x7f6ac41045b0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f6aac000c00 tx=0x7f6aac0047b0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.188+0000 7f6ac994e700 1 -- 172.21.15.70:0/2010435662 shutdown_connections 2024-06-04T05:11:01.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.188+0000 7f6ac994e700 1 --2- 172.21.15.70:0/2010435662 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ab00b29f0 0x7f6ab00b4eb0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.188+0000 7f6ac994e700 1 --2- 172.21.15.70:0/2010435662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ac4108510 0x7f6ac4104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.188+0000 7f6ac994e700 1 --2- 172.21.15.70:0/2010435662 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ac4102430 0x7f6ac4104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.188+0000 7f6ac994e700 1 --2- 172.21.15.70:0/2010435662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4102df0 0x7f6ac41045b0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:01.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.188+0000 7f6ac994e700 1 -- 172.21.15.70:0/2010435662 >> 172.21.15.70:0/2010435662 conn(0x7f6ac40793b0 msgr2=0x7f6ac406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:01.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.189+0000 7f6ac994e700 1 -- 172.21.15.70:0/2010435662 shutdown_connections 2024-06-04T05:11:01.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:01.189+0000 7f6ac994e700 1 -- 172.21.15.70:0/2010435662 wait complete. 2024-06-04T05:11:01.480 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:01 smithi070 ceph-mon[32631]: pgmap v430: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1005 KiB/s wr, 97 op/s 2024-06-04T05:11:01.587 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:01 smithi112 ceph-mon[36453]: pgmap v430: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1005 KiB/s wr, 97 op/s 2024-06-04T05:11:01.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 216558 1.56 MB/sec execute 594 sec latency 4469.789 ms 2024-06-04T05:11:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:01 smithi098 ceph-mon[35569]: pgmap v430: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1005 KiB/s wr, 97 op/s 2024-06-04T05:11:02.044 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:11:02.342 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:11:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.435+0000 7f605eada640 1 -- 172.21.15.70:0/90400058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6058102360 msgr2=0x7f605810e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.435+0000 7f605eada640 1 --2- 172.21.15.70:0/90400058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6058102360 0x7f605810e820 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f6050009cd0 tx=0x7f605002a8e0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.435+0000 7f605eada640 1 -- 172.21.15.70:0/90400058 shutdown_connections 2024-06-04T05:11:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.435+0000 7f605eada640 1 --2- 172.21.15.70:0/90400058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60581019a0 0x7f6058101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.435+0000 7f605eada640 1 --2- 172.21.15.70:0/90400058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6058102360 0x7f605810e820 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.435+0000 7f605eada640 1 --2- 172.21.15.70:0/90400058 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6058107960 0x7f6058107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.435+0000 7f605eada640 1 -- 172.21.15.70:0/90400058 >> 172.21.15.70:0/90400058 conn(0x7f6058078aa0 msgr2=0x7f6058078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.435+0000 7f605eada640 1 -- 172.21.15.70:0/90400058 shutdown_connections 2024-06-04T05:11:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.435+0000 7f605eada640 1 -- 172.21.15.70:0/90400058 wait complete. 2024-06-04T05:11:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f605eada640 1 Processor -- start 2024-06-04T05:11:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f605eada640 1 -- start start 2024-06-04T05:11:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f605eada640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60581019a0 0x7f605819b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f605eada640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6058102360 0x7f605819b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f605eada640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6058107960 0x7f605819fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f605eada640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6058112c90 con 0x7f6058107960 2024-06-04T05:11:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f605eada640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f605806bb80 con 0x7f60581019a0 2024-06-04T05:11:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f605eada640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6058112e10 con 0x7f6058102360 2024-06-04T05:11:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f605d077640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6058107960 0x7f605819fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f605d077640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6058107960 0x7f605819fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57604/0 (socket says 172.21.15.70:57604) 2024-06-04T05:11:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f605d077640 1 -- 172.21.15.70:0/2520100161 learned_addr learned my addr 172.21.15.70:0/2520100161 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f6057fff640 1 --2- 172.21.15.70:0/2520100161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6058102360 0x7f605819b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f605c876640 1 --2- 172.21.15.70:0/2520100161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60581019a0 0x7f605819b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f605d077640 1 -- 172.21.15.70:0/2520100161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6058102360 msgr2=0x7f605819b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f605d077640 1 --2- 172.21.15.70:0/2520100161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6058102360 0x7f605819b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f605d077640 1 -- 172.21.15.70:0/2520100161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60581019a0 msgr2=0x7f605819b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f605d077640 1 --2- 172.21.15.70:0/2520100161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60581019a0 0x7f605819b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.436+0000 7f605d077640 1 -- 172.21.15.70:0/2520100161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60581a02a0 con 0x7f6058107960 2024-06-04T05:11:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.437+0000 7f605d077640 1 --2- 172.21.15.70:0/2520100161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6058107960 0x7f605819fb70 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f6050005cc0 tx=0x7f6050003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.437+0000 7f6055ffb640 1 -- 172.21.15.70:0/2520100161 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6050004240 con 0x7f6058107960 2024-06-04T05:11:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.437+0000 7f6055ffb640 1 -- 172.21.15.70:0/2520100161 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6050003e10 con 0x7f6058107960 2024-06-04T05:11:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.437+0000 7f605eada640 1 -- 172.21.15.70:0/2520100161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60581a0530 con 0x7f6058107960 2024-06-04T05:11:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.437+0000 7f6055ffb640 1 -- 172.21.15.70:0/2520100161 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6050030c30 con 0x7f6058107960 2024-06-04T05:11:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.437+0000 7f605eada640 1 -- 172.21.15.70:0/2520100161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60581a7e20 con 0x7f6058107960 2024-06-04T05:11:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.440+0000 7f6055ffb640 1 -- 172.21.15.70:0/2520100161 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f60500043e0 con 0x7f6058107960 2024-06-04T05:11:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.440+0000 7f60617f9740 1 Processor -- start 2024-06-04T05:11:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.440+0000 7f60617f9740 1 -- start start 2024-06-04T05:11:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.440+0000 7f60617f9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63a1820 0x557df63a1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f60617f9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63ac4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f60617f9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63aca00 0x557df63aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f60617f9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557df6313c30 con 0x557df63a2140 2024-06-04T05:11:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f60617f9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557df63037d0 con 0x557df63a1820 2024-06-04T05:11:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f60617f9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557df6307810 con 0x557df63aca00 2024-06-04T05:11:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f605c876640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63ac4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f6055ffb640 1 --2- 172.21.15.70:0/2520100161 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f60480b2a10 0x7f60480b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f6057fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63a1820 0x557df63a1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f605c876640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63ac4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57608/0 (socket says 172.21.15.70:57608) 2024-06-04T05:11:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f605c876640 1 -- 172.21.15.70:0/1004396270 learned_addr learned my addr 172.21.15.70:0/1004396270 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f6057fff640 1 --2- 172.21.15.70:0/1004396270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63aca00 0x557df63aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f605d077640 1 --2- 172.21.15.70:0/2520100161 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f60480b2a10 0x7f60480b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f605c876640 1 -- 172.21.15.70:0/1004396270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63aca00 msgr2=0x557df63aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f605c876640 1 --2- 172.21.15.70:0/1004396270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63aca00 0x557df63aedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f605c876640 1 -- 172.21.15.70:0/1004396270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63a1820 msgr2=0x557df63a1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f605c876640 1 --2- 172.21.15.70:0/1004396270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63a1820 0x557df63a1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f605c876640 1 -- 172.21.15.70:0/1004396270 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557df6018210 con 0x557df63a2140 2024-06-04T05:11:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f6055ffb640 1 -- 172.21.15.70:0/2520100161 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f605002e030 con 0x7f6058107960 2024-06-04T05:11:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f605c876640 1 --2- 172.21.15.70:0/1004396270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63ac4c0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f604c009ca0 tx=0x7f604c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcca0c8676d447d0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f6042ffd640 1 -- 172.21.15.70:0/1004396270 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604c01e070 con 0x557df63a2140 2024-06-04T05:11:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f6042ffd640 1 -- 172.21.15.70:0/1004396270 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f604c019ac0 con 0x557df63a2140 2024-06-04T05:11:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f6042ffd640 1 -- 172.21.15.70:0/1004396270 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604c00e860 con 0x557df63a2140 2024-06-04T05:11:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.441+0000 7f605d077640 1 --2- 172.21.15.70:0/2520100161 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f60480b2a10 0x7f60480b4ed0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6050100d40 tx=0x7f6050097150 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.442+0000 7f60617f9740 1 -- 172.21.15.70:0/1004396270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 msgr2=0x557df63ac4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.442+0000 7f60617f9740 1 --2- 172.21.15.70:0/1004396270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63ac4c0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f604c009ca0 tx=0x7f604c011810 comp rx=0 tx=0).stop 2024-06-04T05:11:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.442+0000 7f60617f9740 1 -- 172.21.15.70:0/1004396270 shutdown_connections 2024-06-04T05:11:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.442+0000 7f60617f9740 1 --2- 172.21.15.70:0/1004396270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63aca00 0x557df63aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.442+0000 7f60617f9740 1 --2- 172.21.15.70:0/1004396270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63ac4c0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.442+0000 7f60617f9740 1 --2- 172.21.15.70:0/1004396270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63a1820 0x557df63a1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.442+0000 7f60617f9740 1 -- 172.21.15.70:0/1004396270 >> 172.21.15.70:0/1004396270 conn(0x557df639dd30 msgr2=0x557df639e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.442+0000 7f60617f9740 1 -- 172.21.15.70:0/1004396270 shutdown_connections 2024-06-04T05:11:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.442+0000 7f60617f9740 1 -- 172.21.15.70:0/1004396270 wait complete. 2024-06-04T05:11:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.443+0000 7f60617f9740 1 -- start start 2024-06-04T05:11:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.443+0000 7f60617f9740 1 Processor -- start 2024-06-04T05:11:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.443+0000 7f60617f9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63a1820 0x557df63a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.443+0000 7f60617f9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63a2140 0x557df63a9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.443+0000 7f60617f9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63aca00 0x557df63aa440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.443+0000 7f60617f9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557df5fe9790 con 0x557df63aca00 2024-06-04T05:11:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.443+0000 7f60617f9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557df62bf890 con 0x557df63a1820 2024-06-04T05:11:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.443+0000 7f60617f9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557df63af630 con 0x557df63a2140 2024-06-04T05:11:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.443+0000 7f605c876640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63a2140 0x557df63a9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.443+0000 7f6057fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63aca00 0x557df63aa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.443+0000 7f605c876640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63a2140 0x557df63a9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:43482/0 (socket says 172.21.15.70:43482) 2024-06-04T05:11:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.443+0000 7f605c876640 1 -- 172.21.15.70:0/4065575331 learned_addr learned my addr 172.21.15.70:0/4065575331 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.443+0000 7f60617f9740 10 client.0 _ll_register_callbacks cb 0x557df6017a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.443+0000 7f6057fff640 1 --2- 172.21.15.70:0/4065575331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63a1820 0x557df63a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.444+0000 7f605c876640 1 -- 172.21.15.70:0/4065575331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63a1820 msgr2=0x557df63a99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.444+0000 7f605c876640 1 --2- 172.21.15.70:0/4065575331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63a1820 0x557df63a99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.444+0000 7f605c876640 1 -- 172.21.15.70:0/4065575331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63aca00 msgr2=0x557df63aa440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.444+0000 7f605c876640 1 --2- 172.21.15.70:0/4065575331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63aca00 0x557df63aa440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.444+0000 7f605c876640 1 -- 172.21.15.70:0/4065575331 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557df60e4620 con 0x557df63a2140 2024-06-04T05:11:02.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.444+0000 7f6057fff640 1 --2- 172.21.15.70:0/4065575331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63a1820 0x557df63a99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:11:02.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.444+0000 7f605c876640 1 --2- 172.21.15.70:0/4065575331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63a2140 0x557df63a9f00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f604c009870 tx=0x7f604c019660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df2ac1a3e3765acf server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:02.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.444+0000 7f6037fff640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:11:02.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.444+0000 7f6037fff640 1 -- 172.21.15.70:0/4065575331 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604c027030 con 0x557df63a2140 2024-06-04T05:11:02.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.444+0000 7f60617f9740 1 -- 172.21.15.70:0/4065575331 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f604c019ac0 con 0x557df63a2140 2024-06-04T05:11:02.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.444+0000 7f6037fff640 1 -- 172.21.15.70:0/4065575331 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f604c004020 con 0x557df63a2140 2024-06-04T05:11:02.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.444+0000 7f6037fff640 1 -- 172.21.15.70:0/4065575331 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f604c031050 con 0x557df63a2140 2024-06-04T05:11:02.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.444+0000 7f60617f9740 10 client.28180 fetch_fsmap learned FSMap version 39 2024-06-04T05:11:02.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.444+0000 7f60617f9740 1 -- 172.21.15.70:0/4065575331 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x557df63bb3e0 con 0x557df63a2140 2024-06-04T05:11:02.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.445+0000 7f6037fff640 1 -- 172.21.15.70:0/4065575331 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f604c019ac0 con 0x557df63a2140 2024-06-04T05:11:02.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.445+0000 7f6037fff640 1 -- 172.21.15.70:0/4065575331 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 39) ==== 8374+0+0 (secure 0 0 0) 0x7f604c0046c0 con 0x557df63a2140 2024-06-04T05:11:02.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.445+0000 7f60617f9740 10 client.28180 fetch_fsmap finished waiting for FSMap version 39 2024-06-04T05:11:02.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.445+0000 7f60617f9740 10 client.28180 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:02.452 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.445+0000 7f60617f9740 20 client.28180 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.445+0000 7f60617f9740 1 --2- 172.21.15.70:0/4065575331 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557df63bc840 0x557df63bec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.445+0000 7f60617f9740 4 client.28180 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:11:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.445+0000 7f60617f9740 1 -- 172.21.15.70:0/4065575331 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557df6315b80 con 0x557df63bc840 2024-06-04T05:11:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.445+0000 7f6057fff640 1 --2- 172.21.15.70:0/4065575331 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557df63bc840 0x557df63bec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.446+0000 7f6037fff640 10 client.28180 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.446+0000 7f6057fff640 1 --2- 172.21.15.70:0/4065575331 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557df63bc840 0x557df63bec30 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1fa3399dabe30cd server_cookie=efce4eb663edec5f in_seq=0 out_seq=0 2024-06-04T05:11:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.446+0000 7f6037fff640 1 -- 172.21.15.70:0/4065575331 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557df6315b80 con 0x557df63bc840 2024-06-04T05:11:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.446+0000 7f6037fff640 10 client.28180 handle_command_reply: tid=0 2024-06-04T05:11:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.446+0000 7f60617f9740 1 client.28180 shutdown 2024-06-04T05:11:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.446+0000 7f60617f9740 1 -- 172.21.15.70:0/4065575331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63a2140 msgr2=0x557df63a9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.446+0000 7f60617f9740 1 --2- 172.21.15.70:0/4065575331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63a2140 0x557df63a9f00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f604c009870 tx=0x7f604c019660 comp rx=0 tx=0).stop 2024-06-04T05:11:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.447+0000 7f60617f9740 1 -- 172.21.15.70:0/4065575331 shutdown_connections 2024-06-04T05:11:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.447+0000 7f60617f9740 1 --2- 172.21.15.70:0/4065575331 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557df63bc840 0x557df63bec30 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.447+0000 7f60617f9740 1 --2- 172.21.15.70:0/4065575331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63aca00 0x557df63aa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.447+0000 7f6037fff640 0 client.28180 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.447+0000 7f60617f9740 1 --2- 172.21.15.70:0/4065575331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63a2140 0x557df63a9f00 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.447+0000 7f60617f9740 1 --2- 172.21.15.70:0/4065575331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63a1820 0x557df63a99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.447+0000 7f60617f9740 1 -- 172.21.15.70:0/4065575331 >> 172.21.15.70:0/4065575331 conn(0x557df639dd30 msgr2=0x557df639cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.447+0000 7f60617f9740 1 -- 172.21.15.70:0/4065575331 shutdown_connections 2024-06-04T05:11:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.447+0000 7f60617f9740 1 -- 172.21.15.70:0/4065575331 wait complete. 2024-06-04T05:11:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.447+0000 7f60617f9740 20 client.28180 trim_cache size 0 max 16384 2024-06-04T05:11:02.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f60617f9740 1 Processor -- start 2024-06-04T05:11:02.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f60617f9740 1 -- start start 2024-06-04T05:11:02.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f60617f9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63a1820 0x557df63a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f60617f9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63c4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f60617f9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63aca00 0x557df63c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f60617f9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f60581b0790 con 0x557df63a2140 2024-06-04T05:11:02.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f60617f9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f60581b23c0 con 0x557df63a1820 2024-06-04T05:11:02.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f60617f9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f60581b0e60 con 0x557df63aca00 2024-06-04T05:11:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f605c876640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f605c876640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63c4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57624/0 (socket says 172.21.15.70:57624) 2024-06-04T05:11:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f605c876640 1 -- 172.21.15.70:0/3482397914 learned_addr learned my addr 172.21.15.70:0/3482397914 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f6057fff640 1 --2- 172.21.15.70:0/3482397914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63aca00 0x557df63c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f6057fff640 1 --2- 172.21.15.70:0/3482397914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63a1820 0x557df63a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f605c876640 1 -- 172.21.15.70:0/3482397914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63aca00 msgr2=0x557df63c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f605c876640 1 --2- 172.21.15.70:0/3482397914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63aca00 0x557df63c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f605c876640 1 -- 172.21.15.70:0/3482397914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63a1820 msgr2=0x557df63a38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f605c876640 1 --2- 172.21.15.70:0/3482397914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63a1820 0x557df63a38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.465+0000 7f605c876640 1 -- 172.21.15.70:0/3482397914 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557df63a5590 con 0x557df63a2140 2024-06-04T05:11:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.466+0000 7f605c876640 1 --2- 172.21.15.70:0/3482397914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63c4440 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f604c009f50 tx=0x7f604c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792483d434059d6f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.466+0000 7f60427fc640 1 -- 172.21.15.70:0/3482397914 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604c027040 con 0x557df63a2140 2024-06-04T05:11:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.466+0000 7f60427fc640 1 -- 172.21.15.70:0/3482397914 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f604c01f040 con 0x557df63a2140 2024-06-04T05:11:02.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.466+0000 7f60427fc640 1 -- 172.21.15.70:0/3482397914 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604c01e070 con 0x557df63a2140 2024-06-04T05:11:02.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.467+0000 7f60617f9740 1 -- 172.21.15.70:0/3482397914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 msgr2=0x557df63c4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:02.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.467+0000 7f60617f9740 1 --2- 172.21.15.70:0/3482397914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63c4440 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f604c009f50 tx=0x7f604c011810 comp rx=0 tx=0).stop 2024-06-04T05:11:02.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.467+0000 7f60617f9740 1 -- 172.21.15.70:0/3482397914 shutdown_connections 2024-06-04T05:11:02.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.467+0000 7f60617f9740 1 --2- 172.21.15.70:0/3482397914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63aca00 0x557df63c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.467+0000 7f60617f9740 1 --2- 172.21.15.70:0/3482397914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63c4440 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.467+0000 7f60617f9740 1 --2- 172.21.15.70:0/3482397914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63a1820 0x557df63a38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.467+0000 7f60617f9740 1 -- 172.21.15.70:0/3482397914 >> 172.21.15.70:0/3482397914 conn(0x557df639dd30 msgr2=0x557df63a31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:02.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.467+0000 7f60617f9740 1 -- 172.21.15.70:0/3482397914 shutdown_connections 2024-06-04T05:11:02.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.467+0000 7f60617f9740 1 -- 172.21.15.70:0/3482397914 wait complete. 2024-06-04T05:11:02.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.467+0000 7f60617f9740 1 -- start start 2024-06-04T05:11:02.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.467+0000 7f60617f9740 1 Processor -- start 2024-06-04T05:11:02.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f60617f9740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63a1820 0x557df63b0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f60617f9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f60617f9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63aca00 0x557df63b10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f60617f9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f60581b1c90 con 0x557df63a2140 2024-06-04T05:11:02.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f60617f9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f60581b1560 con 0x557df63aca00 2024-06-04T05:11:02.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f60617f9740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f60581ae790 con 0x557df63a1820 2024-06-04T05:11:02.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f60617f9740 10 client.0 _ll_register_callbacks cb 0x7f604c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:02.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f605c876640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f605c876640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57632/0 (socket says 172.21.15.70:57632) 2024-06-04T05:11:02.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f605c876640 1 -- 172.21.15.70:0/2039127977 learned_addr learned my addr 172.21.15.70:0/2039127977 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:02.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f6057fff640 1 --2- 172.21.15.70:0/2039127977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63a1820 0x557df63b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f6057fff640 1 --2- 172.21.15.70:0/2039127977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63aca00 0x557df63b10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f605c876640 1 -- 172.21.15.70:0/2039127977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63a1820 msgr2=0x557df63b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f605c876640 1 --2- 172.21.15.70:0/2039127977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63a1820 0x557df63b0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f605c876640 1 -- 172.21.15.70:0/2039127977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63aca00 msgr2=0x557df63b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f605c876640 1 --2- 172.21.15.70:0/2039127977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63aca00 0x557df63b10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f605c876640 1 -- 172.21.15.70:0/2039127977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557df6426390 con 0x557df63a2140 2024-06-04T05:11:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f6057fff640 1 --2- 172.21.15.70:0/2039127977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63aca00 0x557df63b10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:11:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f605c876640 1 --2- 172.21.15.70:0/2039127977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63b0bb0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f604c01c040 tx=0x7f604c009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b66dabaa5a8ea71a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.468+0000 7f6037fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:11:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.469+0000 7f6037fff640 1 -- 172.21.15.70:0/2039127977 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604c03a020 con 0x557df63a2140 2024-06-04T05:11:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.469+0000 7f60617f9740 1 -- 172.21.15.70:0/2039127977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f604c01f040 con 0x557df63a2140 2024-06-04T05:11:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.469+0000 7f6037fff640 1 -- 172.21.15.70:0/2039127977 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f604c019be0 con 0x557df63a2140 2024-06-04T05:11:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.469+0000 7f6037fff640 1 -- 172.21.15.70:0/2039127977 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f604c01e040 con 0x557df63a2140 2024-06-04T05:11:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.469+0000 7f60617f9740 10 client.19185 fetch_fsmap learned FSMap version 39 2024-06-04T05:11:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.469+0000 7f60617f9740 1 -- 172.21.15.70:0/2039127977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x557df6427680 con 0x557df63a2140 2024-06-04T05:11:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.469+0000 7f6037fff640 1 -- 172.21.15.70:0/2039127977 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f604c01f040 con 0x557df63a2140 2024-06-04T05:11:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.469+0000 7f6037fff640 1 -- 172.21.15.70:0/2039127977 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 39) ==== 8374+0+0 (secure 0 0 0) 0x7f604c003790 con 0x557df63a2140 2024-06-04T05:11:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.469+0000 7f60617f9740 10 client.19185 fetch_fsmap finished waiting for FSMap version 39 2024-06-04T05:11:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.469+0000 7f60617f9740 10 client.19185 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.469+0000 7f60617f9740 20 client.19185 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.469+0000 7f60617f9740 1 --2- 172.21.15.70:0/2039127977 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557df6427df0 0x557df642a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.469+0000 7f60617f9740 4 client.19185 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:11:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.469+0000 7f60617f9740 1 -- 172.21.15.70:0/2039127977 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557df630f360 con 0x557df6427df0 2024-06-04T05:11:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.470+0000 7f6057fff640 1 --2- 172.21.15.70:0/2039127977 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557df6427df0 0x557df642a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.470+0000 7f6037fff640 10 client.19185 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.470+0000 7f6057fff640 1 --2- 172.21.15.70:0/2039127977 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557df6427df0 0x557df642a210 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d76422b480b7d31a server_cookie=e05de52be76d395a in_seq=0 out_seq=0 2024-06-04T05:11:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.497+0000 7f6037fff640 1 -- 172.21.15.70:0/2039127977 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x557df630f360 con 0x557df6427df0 2024-06-04T05:11:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.497+0000 7f6037fff640 10 client.19185 handle_command_reply: tid=0 2024-06-04T05:11:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.497+0000 7f60617f9740 1 client.19185 shutdown 2024-06-04T05:11:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.497+0000 7f60617f9740 1 -- 172.21.15.70:0/2039127977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 msgr2=0x557df63b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.497+0000 7f60617f9740 1 --2- 172.21.15.70:0/2039127977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63b0bb0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f604c01c040 tx=0x7f604c009c30 comp rx=0 tx=0).stop 2024-06-04T05:11:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.498+0000 7f60617f9740 1 -- 172.21.15.70:0/2039127977 shutdown_connections 2024-06-04T05:11:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.498+0000 7f60617f9740 1 --2- 172.21.15.70:0/2039127977 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557df6427df0 0x557df642a210 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.498+0000 7f60617f9740 1 --2- 172.21.15.70:0/2039127977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557df63aca00 0x557df63b10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.498+0000 7f6037fff640 0 client.19185 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.498+0000 7f60617f9740 1 --2- 172.21.15.70:0/2039127977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557df63a2140 0x557df63b0bb0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.498+0000 7f60617f9740 1 --2- 172.21.15.70:0/2039127977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557df63a1820 0x557df63b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.498+0000 7f60617f9740 1 -- 172.21.15.70:0/2039127977 >> 172.21.15.70:0/2039127977 conn(0x557df639dd30 msgr2=0x557df63a9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.498+0000 7f60617f9740 1 -- 172.21.15.70:0/2039127977 shutdown_connections 2024-06-04T05:11:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.498+0000 7f60617f9740 1 -- 172.21.15.70:0/2039127977 wait complete. 2024-06-04T05:11:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.498+0000 7f60617f9740 20 client.19185 trim_cache size 0 max 16384 2024-06-04T05:11:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:11:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:11:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:11:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.499+0000 7f605eada640 1 -- 172.21.15.70:0/2520100161 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f60480b2a10 msgr2=0x7f60480b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.499+0000 7f605eada640 1 --2- 172.21.15.70:0/2520100161 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f60480b2a10 0x7f60480b4ed0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6050100d40 tx=0x7f6050097150 comp rx=0 tx=0).stop 2024-06-04T05:11:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.499+0000 7f605eada640 1 -- 172.21.15.70:0/2520100161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6058107960 msgr2=0x7f605819fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.499+0000 7f605eada640 1 --2- 172.21.15.70:0/2520100161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6058107960 0x7f605819fb70 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f6050005cc0 tx=0x7f6050003c20 comp rx=0 tx=0).stop 2024-06-04T05:11:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.499+0000 7f605eada640 1 -- 172.21.15.70:0/2520100161 shutdown_connections 2024-06-04T05:11:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.499+0000 7f605eada640 1 --2- 172.21.15.70:0/2520100161 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f60480b2a10 0x7f60480b4ed0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.499+0000 7f605eada640 1 --2- 172.21.15.70:0/2520100161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6058107960 0x7f605819fb70 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.499+0000 7f605eada640 1 --2- 172.21.15.70:0/2520100161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6058102360 0x7f605819b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.499+0000 7f605eada640 1 --2- 172.21.15.70:0/2520100161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60581019a0 0x7f605819b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.499+0000 7f605eada640 1 -- 172.21.15.70:0/2520100161 >> 172.21.15.70:0/2520100161 conn(0x7f6058078aa0 msgr2=0x7f605810cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.499+0000 7f605eada640 1 -- 172.21.15.70:0/2520100161 shutdown_connections 2024-06-04T05:11:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:02.499+0000 7f605eada640 1 -- 172.21.15.70:0/2520100161 wait complete. 2024-06-04T05:11:02.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 216743 1.55 MB/sec execute 595 sec latency 4775.244 ms 2024-06-04T05:11:03.585 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:03 smithi070 ceph-mon[32631]: pgmap v431: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-06-04T05:11:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:03 smithi112 ceph-mon[36453]: pgmap v431: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-06-04T05:11:03.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 217010 1.55 MB/sec execute 596 sec latency 19.220 ms 2024-06-04T05:11:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:03 smithi098 ceph-mon[35569]: pgmap v431: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-06-04T05:11:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.168+0000 7faef90ce700 1 -- 172.21.15.70:0/130264899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faef406c670 msgr2=0x7faef41103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.168+0000 7faef90ce700 1 --2- 172.21.15.70:0/130264899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faef406c670 0x7faef41103d0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7faeec00b0a0 tx=0x7faeec00b3b0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.168+0000 7faef90ce700 1 -- 172.21.15.70:0/130264899 shutdown_connections 2024-06-04T05:11:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.168+0000 7faef90ce700 1 --2- 172.21.15.70:0/130264899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faef406c670 0x7faef41103d0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.168+0000 7faef90ce700 1 --2- 172.21.15.70:0/130264899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef41095b0 0x7faef4109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.168+0000 7faef90ce700 1 --2- 172.21.15.70:0/130264899 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faef406bcb0 0x7faef406c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.168+0000 7faef90ce700 1 -- 172.21.15.70:0/130264899 >> 172.21.15.70:0/130264899 conn(0x7faef40793f0 msgr2=0x7faef4079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:04.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.168+0000 7faef90ce700 1 -- 172.21.15.70:0/130264899 shutdown_connections 2024-06-04T05:11:04.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.168+0000 7faef90ce700 1 -- 172.21.15.70:0/130264899 wait complete. 2024-06-04T05:11:04.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.169+0000 7faef90ce700 1 Processor -- start 2024-06-04T05:11:04.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.169+0000 7faef90ce700 1 -- start start 2024-06-04T05:11:04.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.169+0000 7faef90ce700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faef406bcb0 0x7faef419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef90ce700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faef406c670 0x7faef419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef90ce700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef41095b0 0x7faef41998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef90ce700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faef4113570 con 0x7faef406bcb0 2024-06-04T05:11:04.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef90ce700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faef41133f0 con 0x7faef41095b0 2024-06-04T05:11:04.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef90ce700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7faef41136f0 con 0x7faef406c670 2024-06-04T05:11:04.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faef406bcb0 0x7faef419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef359e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef41095b0 0x7faef41998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faef406c670 0x7faef419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef359e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef41095b0 0x7faef41998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45604/0 (socket says 172.21.15.70:45604) 2024-06-04T05:11:04.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faef406c670 0x7faef419fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:43524/0 (socket says 172.21.15.70:43524) 2024-06-04T05:11:04.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef359e700 1 -- 172.21.15.70:0/336573552 learned_addr learned my addr 172.21.15.70:0/336573552 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:04.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef359e700 1 -- 172.21.15.70:0/336573552 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faef406c670 msgr2=0x7faef419fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:04.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef359e700 1 --2- 172.21.15.70:0/336573552 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faef406c670 0x7faef419fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef359e700 1 -- 172.21.15.70:0/336573552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faef406bcb0 msgr2=0x7faef419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:04.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef359e700 1 --2- 172.21.15.70:0/336573552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faef406bcb0 0x7faef419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef359e700 1 -- 172.21.15.70:0/336573552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef419a050 con 0x7faef41095b0 2024-06-04T05:11:04.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef259c700 1 --2- 172.21.15.70:0/336573552 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faef406c670 0x7faef419fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:04.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faef359e700 1 --2- 172.21.15.70:0/336573552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef41095b0 0x7faef41998f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faeec00b780 tx=0x7faeec004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:04.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.170+0000 7faedbfff700 1 -- 172.21.15.70:0/336573552 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeec021070 con 0x7faef41095b0 2024-06-04T05:11:04.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.171+0000 7faedbfff700 1 -- 172.21.15.70:0/336573552 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faeec007b80 con 0x7faef41095b0 2024-06-04T05:11:04.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.171+0000 7faedbfff700 1 -- 172.21.15.70:0/336573552 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeec0052a0 con 0x7faef41095b0 2024-06-04T05:11:04.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.171+0000 7faef90ce700 1 -- 172.21.15.70:0/336573552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faef419a2e0 con 0x7faef41095b0 2024-06-04T05:11:04.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.171+0000 7faef90ce700 1 -- 172.21.15.70:0/336573552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faef41a6600 con 0x7faef41095b0 2024-06-04T05:11:04.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.173+0000 7faedbfff700 1 -- 172.21.15.70:0/336573552 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7faeec024070 con 0x7faef41095b0 2024-06-04T05:11:04.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.174+0000 7faedbfff700 1 --2- 172.21.15.70:0/336573552 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7faedc0b29f0 0x7faedc0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.174+0000 7faef2d9d700 1 --2- 172.21.15.70:0/336573552 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7faedc0b29f0 0x7faedc0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.174+0000 7faedbfff700 1 -- 172.21.15.70:0/336573552 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8615+0+0 (secure 0 0 0) 0x7faeec00f020 con 0x7faef41095b0 2024-06-04T05:11:04.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.174+0000 7faef2d9d700 1 --2- 172.21.15.70:0/336573552 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7faedc0b29f0 0x7faedc0b4eb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7faef419aa30 tx=0x7faee4009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:04.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faf03963b80 1 Processor -- start 2024-06-04T05:11:04.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faf03963b80 1 -- start start 2024-06-04T05:11:04.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faf03963b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab238610 0x558bab2389f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faf03963b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238fc0 0x558bab232cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faf03963b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab233730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faf03963b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558bab18cd70 con 0x558bab2332d0 2024-06-04T05:11:04.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faf03963b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558bab17c910 con 0x558bab238fc0 2024-06-04T05:11:04.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faf03963b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558bab180950 con 0x558bab238610 2024-06-04T05:11:04.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faef359e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab233730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faef359e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab233730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57666/0 (socket says 172.21.15.70:57666) 2024-06-04T05:11:04.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faef359e700 1 -- 172.21.15.70:0/164228286 learned_addr learned my addr 172.21.15.70:0/164228286 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:04.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faef259c700 1 --2- 172.21.15.70:0/164228286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238fc0 0x558bab232cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faef2d9d700 1 --2- 172.21.15.70:0/164228286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab238610 0x558bab2389f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faef359e700 1 -- 172.21.15.70:0/164228286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab238610 msgr2=0x558bab2389f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:04.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faef359e700 1 --2- 172.21.15.70:0/164228286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab238610 0x558bab2389f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faef359e700 1 -- 172.21.15.70:0/164228286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238fc0 msgr2=0x558bab232cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:04.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faef359e700 1 --2- 172.21.15.70:0/164228286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238fc0 0x558bab232cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faef359e700 1 -- 172.21.15.70:0/164228286 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bab125fb0 con 0x558bab2332d0 2024-06-04T05:11:04.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.175+0000 7faef259c700 1 --2- 172.21.15.70:0/164228286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238fc0 0x558bab232cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:04.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.176+0000 7faef359e700 1 --2- 172.21.15.70:0/164228286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab233730 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7faeec0e12c0 tx=0x7faeec009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c545093359cdefa server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:04.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.176+0000 7faed97fa700 1 -- 172.21.15.70:0/164228286 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeec0aae80 con 0x558bab2332d0 2024-06-04T05:11:04.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.176+0000 7faed97fa700 1 -- 172.21.15.70:0/164228286 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faeec0af230 con 0x558bab2332d0 2024-06-04T05:11:04.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.176+0000 7faed97fa700 1 -- 172.21.15.70:0/164228286 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeec0ab0e0 con 0x558bab2332d0 2024-06-04T05:11:04.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.177+0000 7faf03963b80 1 -- 172.21.15.70:0/164228286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 msgr2=0x558bab233730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:04.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.177+0000 7faf03963b80 1 --2- 172.21.15.70:0/164228286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab233730 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7faeec0e12c0 tx=0x7faeec009c50 comp rx=0 tx=0).stop 2024-06-04T05:11:04.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.177+0000 7faf03963b80 1 -- 172.21.15.70:0/164228286 shutdown_connections 2024-06-04T05:11:04.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.177+0000 7faf03963b80 1 --2- 172.21.15.70:0/164228286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab233730 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.177+0000 7faf03963b80 1 --2- 172.21.15.70:0/164228286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab238610 0x558bab2389f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.177+0000 7faf03963b80 1 --2- 172.21.15.70:0/164228286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238fc0 0x558bab232cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.177+0000 7faf03963b80 1 -- 172.21.15.70:0/164228286 >> 172.21.15.70:0/164228286 conn(0x558bab15a240 msgr2=0x558bab2293e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:04.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.177+0000 7faf03963b80 1 -- 172.21.15.70:0/164228286 shutdown_connections 2024-06-04T05:11:04.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.177+0000 7faf03963b80 1 -- 172.21.15.70:0/164228286 wait complete. 2024-06-04T05:11:04.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.177+0000 7faf03963b80 1 -- start start 2024-06-04T05:11:04.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.177+0000 7faf03963b80 1 Processor -- start 2024-06-04T05:11:04.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faf03963b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab23ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faf03963b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab238610 0x558bab23c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faf03963b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238fc0 0x558bab23c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faf03963b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558bab082fa0 con 0x558bab2332d0 2024-06-04T05:11:04.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faf03963b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558bab0e4da0 con 0x558bab238fc0 2024-06-04T05:11:04.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faf03963b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558bab23f860 con 0x558bab238610 2024-06-04T05:11:04.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faef2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab23ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faef2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab23ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57670/0 (socket says 172.21.15.70:57670) 2024-06-04T05:11:04.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faef2d9d700 1 -- 172.21.15.70:0/383414858 learned_addr learned my addr 172.21.15.70:0/383414858 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:04.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faef259c700 1 --2- 172.21.15.70:0/383414858 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab238610 0x558bab23c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faf03963b80 10 client.0 _ll_register_callbacks cb 0x558bab117950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:04.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faef359e700 1 --2- 172.21.15.70:0/383414858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238fc0 0x558bab23c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faef259c700 1 -- 172.21.15.70:0/383414858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238fc0 msgr2=0x558bab23c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:04.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faef259c700 1 --2- 172.21.15.70:0/383414858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238fc0 0x558bab23c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faef259c700 1 -- 172.21.15.70:0/383414858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 msgr2=0x558bab23ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:04.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faef259c700 1 --2- 172.21.15.70:0/383414858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab23ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faef259c700 1 -- 172.21.15.70:0/383414858 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bab0cc8d0 con 0x558bab238610 2024-06-04T05:11:04.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faef2d9d700 1 --2- 172.21.15.70:0/383414858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab23ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:04.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faef259c700 1 --2- 172.21.15.70:0/383414858 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab238610 0x558bab23c130 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x558bab22aea0 tx=0x7faee8011cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1aead9c9f57f7c6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:04.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faed6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:11:04.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.178+0000 7faed6ff5700 1 -- 172.21.15.70:0/383414858 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee8010900 con 0x558bab238610 2024-06-04T05:11:04.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.179+0000 7faed6ff5700 1 -- 172.21.15.70:0/383414858 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8615+0+0 (secure 0 0 0) 0x7faee8010aa0 con 0x558bab238610 2024-06-04T05:11:04.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.179+0000 7faf03963b80 1 -- 172.21.15.70:0/383414858 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faeec0af230 con 0x558bab238610 2024-06-04T05:11:04.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.179+0000 7faed6ff5700 1 -- 172.21.15.70:0/383414858 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7faee8010ca0 con 0x558bab238610 2024-06-04T05:11:04.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.179+0000 7faf03963b80 10 client.28198 fetch_fsmap learned FSMap version 39 2024-06-04T05:11:04.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.179+0000 7faf03963b80 1 -- 172.21.15.70:0/383414858 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x558bab136760 con 0x558bab238610 2024-06-04T05:11:04.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.179+0000 7faed6ff5700 1 -- 172.21.15.70:0/383414858 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faee8010e40 con 0x558bab238610 2024-06-04T05:11:04.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.179+0000 7faed6ff5700 1 -- 172.21.15.70:0/383414858 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 39) ==== 8374+0+0 (secure 0 0 0) 0x7faee801b470 con 0x558bab238610 2024-06-04T05:11:04.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.179+0000 7faf03963b80 10 client.28198 fetch_fsmap finished waiting for FSMap version 39 2024-06-04T05:11:04.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.179+0000 7faf03963b80 10 client.28198 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:04.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.179+0000 7faf03963b80 20 client.28198 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:04.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.179+0000 7faf03963b80 1 --2- 172.21.15.70:0/383414858 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558bab24c820 0x558bab24eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.179+0000 7faf03963b80 4 client.28198 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:11:04.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.179+0000 7faf03963b80 1 -- 172.21.15.70:0/383414858 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7faeec0af230 con 0x558bab24c820 2024-06-04T05:11:04.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.179+0000 7faef2d9d700 1 --2- 172.21.15.70:0/383414858 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558bab24c820 0x558bab24eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.180+0000 7faef2d9d700 1 --2- 172.21.15.70:0/383414858 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558bab24c820 0x558bab24eca0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efbee1f1facfee02 server_cookie=252393a5f0d74961 in_seq=0 out_seq=0 2024-06-04T05:11:04.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.180+0000 7faed6ff5700 10 client.28198 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:04.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.180+0000 7faed6ff5700 1 -- 172.21.15.70:0/383414858 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7faeec0af230 con 0x558bab24c820 2024-06-04T05:11:04.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.180+0000 7faed6ff5700 10 client.28198 handle_command_reply: tid=0 2024-06-04T05:11:04.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.180+0000 7faf03963b80 1 client.28198 shutdown 2024-06-04T05:11:04.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.181+0000 7faf03963b80 1 -- 172.21.15.70:0/383414858 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab238610 msgr2=0x558bab23c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:04.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.181+0000 7faf03963b80 1 --2- 172.21.15.70:0/383414858 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab238610 0x558bab23c130 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x558bab22aea0 tx=0x7faee8011cb0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.181+0000 7faf03963b80 1 -- 172.21.15.70:0/383414858 shutdown_connections 2024-06-04T05:11:04.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.181+0000 7faf03963b80 1 --2- 172.21.15.70:0/383414858 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558bab24c820 0x558bab24eca0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.181+0000 7faf03963b80 1 --2- 172.21.15.70:0/383414858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238fc0 0x558bab23c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.181+0000 7faf03963b80 1 --2- 172.21.15.70:0/383414858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab23ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.181+0000 7faed6ff5700 0 client.28198 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:04.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.181+0000 7faf03963b80 1 --2- 172.21.15.70:0/383414858 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab238610 0x558bab23c130 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.181+0000 7faf03963b80 1 -- 172.21.15.70:0/383414858 >> 172.21.15.70:0/383414858 conn(0x558bab15a240 msgr2=0x558bab228af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:04.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.181+0000 7faf03963b80 1 -- 172.21.15.70:0/383414858 shutdown_connections 2024-06-04T05:11:04.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.181+0000 7faf03963b80 1 -- 172.21.15.70:0/383414858 wait complete. 2024-06-04T05:11:04.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.181+0000 7faf03963b80 20 client.28198 trim_cache size 0 max 16384 2024-06-04T05:11:04.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.197+0000 7faf03963b80 1 Processor -- start 2024-06-04T05:11:04.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.197+0000 7faf03963b80 1 -- start start 2024-06-04T05:11:04.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faf03963b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab2488c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faf03963b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238610 0x558bab236870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faf03963b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab238fc0 0x558bab24ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faf03963b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faef41abc50 con 0x558bab2332d0 2024-06-04T05:11:04.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faf03963b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faef41ad880 con 0x558bab238610 2024-06-04T05:11:04.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faf03963b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7faef41ac320 con 0x558bab238fc0 2024-06-04T05:11:04.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faef2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab2488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faef2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab2488c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57678/0 (socket says 172.21.15.70:57678) 2024-06-04T05:11:04.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faef2d9d700 1 -- 172.21.15.70:0/1674652812 learned_addr learned my addr 172.21.15.70:0/1674652812 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:04.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faef259c700 1 --2- 172.21.15.70:0/1674652812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238610 0x558bab236870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faef359e700 1 --2- 172.21.15.70:0/1674652812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab238fc0 0x558bab24ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faef2d9d700 1 -- 172.21.15.70:0/1674652812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab238fc0 msgr2=0x558bab24ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:04.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faef2d9d700 1 --2- 172.21.15.70:0/1674652812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab238fc0 0x558bab24ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faef2d9d700 1 -- 172.21.15.70:0/1674652812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238610 msgr2=0x558bab236870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:04.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faef2d9d700 1 --2- 172.21.15.70:0/1674652812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238610 0x558bab236870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faef2d9d700 1 -- 172.21.15.70:0/1674652812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bab24f320 con 0x558bab2332d0 2024-06-04T05:11:04.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faef259c700 1 --2- 172.21.15.70:0/1674652812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238610 0x558bab236870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:04.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faef2d9d700 1 --2- 172.21.15.70:0/1674652812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab2488c0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7faee4005930 tx=0x7faee4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd8e47da9a208d5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:04.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.198+0000 7faed8ff9700 1 -- 172.21.15.70:0/1674652812 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee4004460 con 0x558bab2332d0 2024-06-04T05:11:04.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.199+0000 7faed8ff9700 1 -- 172.21.15.70:0/1674652812 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faee4004cc0 con 0x558bab2332d0 2024-06-04T05:11:04.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.199+0000 7faed8ff9700 1 -- 172.21.15.70:0/1674652812 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee4005540 con 0x558bab2332d0 2024-06-04T05:11:04.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.199+0000 7faf03963b80 1 -- 172.21.15.70:0/1674652812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 msgr2=0x558bab2488c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:04.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.199+0000 7faf03963b80 1 --2- 172.21.15.70:0/1674652812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab2488c0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7faee4005930 tx=0x7faee4014000 comp rx=0 tx=0).stop 2024-06-04T05:11:04.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.199+0000 7faf03963b80 1 -- 172.21.15.70:0/1674652812 shutdown_connections 2024-06-04T05:11:04.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.199+0000 7faf03963b80 1 --2- 172.21.15.70:0/1674652812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab238fc0 0x558bab24ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.199+0000 7faf03963b80 1 --2- 172.21.15.70:0/1674652812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab2332d0 0x558bab2488c0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.199+0000 7faf03963b80 1 --2- 172.21.15.70:0/1674652812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238610 0x558bab236870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.199+0000 7faf03963b80 1 -- 172.21.15.70:0/1674652812 >> 172.21.15.70:0/1674652812 conn(0x558bab15a240 msgr2=0x558bab236160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:04.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.199+0000 7faf03963b80 1 -- 172.21.15.70:0/1674652812 shutdown_connections 2024-06-04T05:11:04.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.199+0000 7faf03963b80 1 -- 172.21.15.70:0/1674652812 wait complete. 2024-06-04T05:11:04.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.200+0000 7faf03963b80 1 -- start start 2024-06-04T05:11:04.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.200+0000 7faf03963b80 1 Processor -- start 2024-06-04T05:11:04.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.200+0000 7faf03963b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab2332d0 0x558bab244ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.200+0000 7faf03963b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab238610 0x558bab244ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.200+0000 7faf03963b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238fc0 0x558bab2535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.200+0000 7faf03963b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faef41ad150 con 0x558bab238610 2024-06-04T05:11:04.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.200+0000 7faf03963b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faef41aca20 con 0x558bab238fc0 2024-06-04T05:11:04.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.200+0000 7faf03963b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7faef41a9c10 con 0x558bab2332d0 2024-06-04T05:11:04.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.200+0000 7faf03963b80 10 client.0 _ll_register_callbacks cb 0x558bab117950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:04.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.200+0000 7faef259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab238610 0x558bab244ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.200+0000 7faef259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab238610 0x558bab244ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57692/0 (socket says 172.21.15.70:57692) 2024-06-04T05:11:04.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.200+0000 7faef259c700 1 -- 172.21.15.70:0/43003770 learned_addr learned my addr 172.21.15.70:0/43003770 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:04.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.200+0000 7faef2d9d700 1 --2- 172.21.15.70:0/43003770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab2332d0 0x558bab244ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.201+0000 7faef359e700 1 --2- 172.21.15.70:0/43003770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238fc0 0x558bab2535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.201+0000 7faef259c700 1 -- 172.21.15.70:0/43003770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab2332d0 msgr2=0x558bab244ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:04.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.201+0000 7faef259c700 1 --2- 172.21.15.70:0/43003770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab2332d0 0x558bab244ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.201+0000 7faef259c700 1 -- 172.21.15.70:0/43003770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238fc0 msgr2=0x558bab2535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:04.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.201+0000 7faef259c700 1 --2- 172.21.15.70:0/43003770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238fc0 0x558bab2535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.201+0000 7faef259c700 1 -- 172.21.15.70:0/43003770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bab238c40 con 0x558bab238610 2024-06-04T05:11:04.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.201+0000 7faef359e700 1 --2- 172.21.15.70:0/43003770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238fc0 0x558bab2535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:11:04.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.201+0000 7faef259c700 1 --2- 172.21.15.70:0/43003770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab238610 0x558bab244ff0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7faee80115e0 tx=0x7faee801b750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191acc887869545c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:04.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.201+0000 7faed6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:11:04.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.201+0000 7faed6ff5700 1 -- 172.21.15.70:0/43003770 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee8032060 con 0x558bab238610 2024-06-04T05:11:04.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.201+0000 7faed6ff5700 1 -- 172.21.15.70:0/43003770 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8615+0+0 (secure 0 0 0) 0x7faee80050b0 con 0x558bab238610 2024-06-04T05:11:04.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.201+0000 7faf03963b80 1 -- 172.21.15.70:0/43003770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faee4004cc0 con 0x558bab238610 2024-06-04T05:11:04.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.201+0000 7faed6ff5700 1 -- 172.21.15.70:0/43003770 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7faee8027040 con 0x558bab238610 2024-06-04T05:11:04.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.201+0000 7faf03963b80 10 client.19203 fetch_fsmap learned FSMap version 39 2024-06-04T05:11:04.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.201+0000 7faf03963b80 1 -- 172.21.15.70:0/43003770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x558bab2b6940 con 0x558bab238610 2024-06-04T05:11:04.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.201+0000 7faed6ff5700 1 -- 172.21.15.70:0/43003770 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faee4004cc0 con 0x558bab238610 2024-06-04T05:11:04.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faed6ff5700 1 -- 172.21.15.70:0/43003770 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 39) ==== 8374+0+0 (secure 0 0 0) 0x7faee801bce0 con 0x558bab238610 2024-06-04T05:11:04.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faf03963b80 10 client.19203 fetch_fsmap finished waiting for FSMap version 39 2024-06-04T05:11:04.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faf03963b80 10 client.19203 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:04.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faf03963b80 20 client.19203 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:04.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faf03963b80 1 --2- 172.21.15.70:0/43003770 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558bab2b7090 0x558bab2b94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:04.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faf03963b80 4 client.19203 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:11:04.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faf03963b80 1 -- 172.21.15.70:0/43003770 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558bab18ecc0 con 0x558bab2b7090 2024-06-04T05:11:04.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faef2d9d700 1 --2- 172.21.15.70:0/43003770 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558bab2b7090 0x558bab2b94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:04.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faef2d9d700 1 --2- 172.21.15.70:0/43003770 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558bab2b7090 0x558bab2b94b0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16378412d4a23eef server_cookie=eacd5648f2860f16 in_seq=0 out_seq=0 2024-06-04T05:11:04.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faed6ff5700 10 client.19203 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:04.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faed6ff5700 1 -- 172.21.15.70:0/43003770 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558bab18ecc0 con 0x558bab2b7090 2024-06-04T05:11:04.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faed6ff5700 10 client.19203 handle_command_reply: tid=0 2024-06-04T05:11:04.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faf03963b80 1 client.19203 shutdown 2024-06-04T05:11:04.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faf03963b80 1 -- 172.21.15.70:0/43003770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab238610 msgr2=0x558bab244ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:04.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faf03963b80 1 --2- 172.21.15.70:0/43003770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab238610 0x558bab244ff0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7faee80115e0 tx=0x7faee801b750 comp rx=0 tx=0).stop 2024-06-04T05:11:04.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faf03963b80 1 -- 172.21.15.70:0/43003770 shutdown_connections 2024-06-04T05:11:04.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faf03963b80 1 --2- 172.21.15.70:0/43003770 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558bab2b7090 0x558bab2b94b0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.202+0000 7faf03963b80 1 --2- 172.21.15.70:0/43003770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bab238fc0 0x558bab2535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.203+0000 7faf03963b80 1 --2- 172.21.15.70:0/43003770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558bab2332d0 0x558bab244ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.203+0000 7faf03963b80 1 --2- 172.21.15.70:0/43003770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bab238610 0x558bab244ff0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.203+0000 7faf03963b80 1 -- 172.21.15.70:0/43003770 >> 172.21.15.70:0/43003770 conn(0x558bab15a240 msgr2=0x558bab22e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:04.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.203+0000 7faed6ff5700 0 client.19203 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:04.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.203+0000 7faf03963b80 1 -- 172.21.15.70:0/43003770 shutdown_connections 2024-06-04T05:11:04.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.203+0000 7faf03963b80 1 -- 172.21.15.70:0/43003770 wait complete. 2024-06-04T05:11:04.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.203+0000 7faf03963b80 20 client.19203 trim_cache size 0 max 16384 2024-06-04T05:11:04.213 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:11:04.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.203+0000 7faef90ce700 1 -- 172.21.15.70:0/336573552 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7faedc0b29f0 msgr2=0x7faedc0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:04.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.203+0000 7faef90ce700 1 --2- 172.21.15.70:0/336573552 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7faedc0b29f0 0x7faedc0b4eb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7faef419aa30 tx=0x7faee4009260 comp rx=0 tx=0).stop 2024-06-04T05:11:04.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.203+0000 7faef90ce700 1 -- 172.21.15.70:0/336573552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef41095b0 msgr2=0x7faef41998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:04.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.203+0000 7faef90ce700 1 --2- 172.21.15.70:0/336573552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef41095b0 0x7faef41998f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faeec00b780 tx=0x7faeec004d00 comp rx=0 tx=0).stop 2024-06-04T05:11:04.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.203+0000 7faef90ce700 1 -- 172.21.15.70:0/336573552 shutdown_connections 2024-06-04T05:11:04.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.203+0000 7faef90ce700 1 --2- 172.21.15.70:0/336573552 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7faedc0b29f0 0x7faedc0b4eb0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.203+0000 7faef90ce700 1 --2- 172.21.15.70:0/336573552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef41095b0 0x7faef41998f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.203+0000 7faef90ce700 1 --2- 172.21.15.70:0/336573552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faef406bcb0 0x7faef419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.203+0000 7faef90ce700 1 --2- 172.21.15.70:0/336573552 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faef406c670 0x7faef419fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:04.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.203+0000 7faef90ce700 1 -- 172.21.15.70:0/336573552 >> 172.21.15.70:0/336573552 conn(0x7faef40793f0 msgr2=0x7faef4101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:04.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.204+0000 7faef90ce700 1 -- 172.21.15.70:0/336573552 shutdown_connections 2024-06-04T05:11:04.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:04.204+0000 7faef90ce700 1 -- 172.21.15.70:0/336573552 wait complete. 2024-06-04T05:11:04.601 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:04 smithi070 ceph-mon[32631]: osdmap e95: 12 total, 12 up, 12 in 2024-06-04T05:11:04.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:04 smithi112 ceph-mon[36453]: osdmap e95: 12 total, 12 up, 12 in 2024-06-04T05:11:04.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 217481 1.55 MB/sec execute 597 sec latency 33.586 ms 2024-06-04T05:11:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:04 smithi098 ceph-mon[35569]: osdmap e95: 12 total, 12 up, 12 in 2024-06-04T05:11:05.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:05 smithi070 ceph-mon[32631]: pgmap v433: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 656 KiB/s wr, 62 op/s 2024-06-04T05:11:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:05 smithi112 ceph-mon[36453]: pgmap v433: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 656 KiB/s wr, 62 op/s 2024-06-04T05:11:05.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 218017 1.56 MB/sec execute 598 sec latency 19.232 ms 2024-06-04T05:11:05.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:05 smithi098 ceph-mon[35569]: pgmap v433: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 656 KiB/s wr, 62 op/s 2024-06-04T05:11:06.010 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:11:06.010 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:11:06.276 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:11:06.745 INFO:tasks.workunit.client.0.smithi070.stdout: 1 218601 1.55 MB/sec execute 599 sec latency 22.284 ms 2024-06-04T05:11:07.590 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:07 smithi070 ceph-mon[32631]: pgmap v434: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 699 KiB/s wr, 104 op/s 2024-06-04T05:11:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:07 smithi112 ceph-mon[36453]: pgmap v434: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 699 KiB/s wr, 104 op/s 2024-06-04T05:11:07.744 INFO:tasks.workunit.client.0.smithi070.stdout: 1 cleanup 600 sec 2024-06-04T05:11:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:07 smithi098 ceph-mon[35569]: pgmap v434: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 699 KiB/s wr, 104 op/s 2024-06-04T05:11:07.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.994+0000 7f7e1ac14700 1 -- 172.21.15.70:0/2065303973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e14108510 msgr2=0x7f7e141088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:07.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.994+0000 7f7e1ac14700 1 --2- 172.21.15.70:0/2065303973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e14108510 0x7f7e141088f0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f7e08009990 tx=0x7f7e08009ca0 comp rx=0 tx=0).stop 2024-06-04T05:11:07.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.994+0000 7f7e1ac14700 1 -- 172.21.15.70:0/2065303973 shutdown_connections 2024-06-04T05:11:07.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.994+0000 7f7e1ac14700 1 --2- 172.21.15.70:0/2065303973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e14102df0 0x7f7e1410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:07.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.994+0000 7f7e1ac14700 1 --2- 172.21.15.70:0/2065303973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e14108510 0x7f7e141088f0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:07.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.994+0000 7f7e1ac14700 1 --2- 172.21.15.70:0/2065303973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e14102430 0x7f7e141028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:07.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.994+0000 7f7e1ac14700 1 -- 172.21.15.70:0/2065303973 >> 172.21.15.70:0/2065303973 conn(0x7f7e140793b0 msgr2=0x7f7e140797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:07.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.994+0000 7f7e1ac14700 1 -- 172.21.15.70:0/2065303973 shutdown_connections 2024-06-04T05:11:07.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.994+0000 7f7e1ac14700 1 -- 172.21.15.70:0/2065303973 wait complete. 2024-06-04T05:11:07.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.995+0000 7f7e1ac14700 1 Processor -- start 2024-06-04T05:11:07.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.995+0000 7f7e1ac14700 1 -- start start 2024-06-04T05:11:07.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.995+0000 7f7e1ac14700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e14102430 0x7f7e1419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:07.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.995+0000 7f7e1ac14700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e14102df0 0x7f7e1419fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:07.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.995+0000 7f7e1ac14700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e14108510 0x7f7e14199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:07.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.995+0000 7f7e1ac14700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7e14112490 con 0x7f7e14102df0 2024-06-04T05:11:07.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.995+0000 7f7e1ac14700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e14112310 con 0x7f7e14102430 2024-06-04T05:11:08.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.995+0000 7f7e1ac14700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7e14112610 con 0x7f7e14108510 2024-06-04T05:11:08.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.995+0000 7f7e13fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e14102df0 0x7f7e1419fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:08.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.995+0000 7f7e189b0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e14102430 0x7f7e1419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:08.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.996+0000 7f7e191b1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e14108510 0x7f7e14199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:08.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.996+0000 7f7e189b0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e14102430 0x7f7e1419f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46326/0 (socket says 172.21.15.70:46326) 2024-06-04T05:11:08.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.996+0000 7f7e189b0700 1 -- 172.21.15.70:0/820804046 learned_addr learned my addr 172.21.15.70:0/820804046 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:08.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.996+0000 7f7e189b0700 1 -- 172.21.15.70:0/820804046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e14108510 msgr2=0x7f7e14199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:08.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.996+0000 7f7e189b0700 1 --2- 172.21.15.70:0/820804046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e14108510 0x7f7e14199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.996+0000 7f7e189b0700 1 -- 172.21.15.70:0/820804046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e14102df0 msgr2=0x7f7e1419fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:08.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.996+0000 7f7e189b0700 1 --2- 172.21.15.70:0/820804046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e14102df0 0x7f7e1419fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.996+0000 7f7e189b0700 1 -- 172.21.15.70:0/820804046 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e1419a160 con 0x7f7e14102430 2024-06-04T05:11:08.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.996+0000 7f7e191b1700 1 --2- 172.21.15.70:0/820804046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e14108510 0x7f7e14199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:08.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.996+0000 7f7e189b0700 1 --2- 172.21.15.70:0/820804046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e14102430 0x7f7e1419f7f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7e08006010 tx=0x7f7e080047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:08.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.996+0000 7f7e11ffb700 1 -- 172.21.15.70:0/820804046 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e08027070 con 0x7f7e14102430 2024-06-04T05:11:08.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.996+0000 7f7e1ac14700 1 -- 172.21.15.70:0/820804046 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e1419a3f0 con 0x7f7e14102430 2024-06-04T05:11:08.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.996+0000 7f7e11ffb700 1 -- 172.21.15.70:0/820804046 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e08022470 con 0x7f7e14102430 2024-06-04T05:11:08.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.996+0000 7f7e11ffb700 1 -- 172.21.15.70:0/820804046 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e08017650 con 0x7f7e14102430 2024-06-04T05:11:08.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.996+0000 7f7e1ac14700 1 -- 172.21.15.70:0/820804046 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e141a66c0 con 0x7f7e14102430 2024-06-04T05:11:08.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:07.999+0000 7f7e11ffb700 1 -- 172.21.15.70:0/820804046 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7e08009d10 con 0x7f7e14102430 2024-06-04T05:11:08.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.000+0000 7f7e11ffb700 1 --2- 172.21.15.70:0/820804046 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7e040b29f0 0x7f7e040b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:08.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.000+0000 7f7e13fff700 1 --2- 172.21.15.70:0/820804046 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7e040b29f0 0x7f7e040b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:08.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.000+0000 7f7e11ffb700 1 -- 172.21.15.70:0/820804046 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f7e080e30c0 con 0x7f7e14102430 2024-06-04T05:11:08.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.000+0000 7f7e254a9b80 1 Processor -- start 2024-06-04T05:11:08.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.000+0000 7f7e254a9b80 1 -- start start 2024-06-04T05:11:08.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.000+0000 7f7e13fff700 1 --2- 172.21.15.70:0/820804046 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7e040b29f0 0x7f7e040b4eb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7e14079a20 tx=0x7f7e00009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:08.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e254a9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b9690 0x55661b1b9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:08.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e254a9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1ba040 0x55661b1b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:08.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e254a9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1b4350 0x55661b1b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:08.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e254a9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55661b10ddf0 con 0x55661b1b4350 2024-06-04T05:11:08.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e254a9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55661b0fd990 con 0x55661b1b9690 2024-06-04T05:11:08.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e254a9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55661b1019d0 con 0x55661b1ba040 2024-06-04T05:11:08.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e191b1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1b4350 0x55661b1b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:08.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e189b0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b9690 0x55661b1b9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:08.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e191b1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1b4350 0x55661b1b47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57634/0 (socket says 172.21.15.70:57634) 2024-06-04T05:11:08.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e13fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1ba040 0x55661b1b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:08.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e189b0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b9690 0x55661b1b9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46330/0 (socket says 172.21.15.70:46330) 2024-06-04T05:11:08.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e191b1700 1 -- 172.21.15.70:0/4005657701 learned_addr learned my addr 172.21.15.70:0/4005657701 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:08.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e189b0700 1 -- 172.21.15.70:0/4005657701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1ba040 msgr2=0x55661b1b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:08.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e189b0700 1 --2- 172.21.15.70:0/4005657701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1ba040 0x55661b1b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e189b0700 1 -- 172.21.15.70:0/4005657701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1b4350 msgr2=0x55661b1b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:08.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e189b0700 1 --2- 172.21.15.70:0/4005657701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1b4350 0x55661b1b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e189b0700 1 -- 172.21.15.70:0/4005657701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55661b0a7030 con 0x55661b1b9690 2024-06-04T05:11:08.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7e189b0700 1 --2- 172.21.15.70:0/4005657701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b9690 0x55661b1b9a70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7e080e07e0 tx=0x7f7e080a9e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ab739e39b96d560 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:08.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.001+0000 7f7dfeffd700 1 -- 172.21.15.70:0/4005657701 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e080ae7b0 con 0x55661b1b9690 2024-06-04T05:11:08.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.002+0000 7f7dfeffd700 1 -- 172.21.15.70:0/4005657701 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e080b31d0 con 0x55661b1b9690 2024-06-04T05:11:08.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.002+0000 7f7dfeffd700 1 -- 172.21.15.70:0/4005657701 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e080b2d40 con 0x55661b1b9690 2024-06-04T05:11:08.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.002+0000 7f7e254a9b80 1 -- 172.21.15.70:0/4005657701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b9690 msgr2=0x55661b1b9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:08.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.002+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/4005657701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b9690 0x55661b1b9a70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7e080e07e0 tx=0x7f7e080a9e60 comp rx=0 tx=0).stop 2024-06-04T05:11:08.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.002+0000 7f7e254a9b80 1 -- 172.21.15.70:0/4005657701 shutdown_connections 2024-06-04T05:11:08.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.002+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/4005657701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1b4350 0x55661b1b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.002+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/4005657701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b9690 0x55661b1b9a70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.002+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/4005657701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1ba040 0x55661b1b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.002+0000 7f7e254a9b80 1 -- 172.21.15.70:0/4005657701 >> 172.21.15.70:0/4005657701 conn(0x55661b0db2c0 msgr2=0x55661b1aa460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:08.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.002+0000 7f7e254a9b80 1 -- 172.21.15.70:0/4005657701 shutdown_connections 2024-06-04T05:11:08.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.002+0000 7f7e254a9b80 1 -- 172.21.15.70:0/4005657701 wait complete. 2024-06-04T05:11:08.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.003+0000 7f7e254a9b80 1 -- start start 2024-06-04T05:11:08.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.003+0000 7f7e254a9b80 1 Processor -- start 2024-06-04T05:11:08.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.003+0000 7f7e254a9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b4350 0x55661b1bff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:08.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.003+0000 7f7e254a9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b9690 0x55661b1bd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:08.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.003+0000 7f7e254a9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1ba040 0x55661b1bd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:08.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7e189b0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b4350 0x55661b1bff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:08.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7e189b0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b4350 0x55661b1bff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46338/0 (socket says 172.21.15.70:46338) 2024-06-04T05:11:08.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7e189b0700 1 -- 172.21.15.70:0/3077432625 learned_addr learned my addr 172.21.15.70:0/3077432625 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:08.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7e254a9b80 1 -- 172.21.15.70:0/3077432625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55661b004020 con 0x55661b1ba040 2024-06-04T05:11:08.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7e254a9b80 1 -- 172.21.15.70:0/3077432625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55661b065e20 con 0x55661b1b4350 2024-06-04T05:11:08.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7e254a9b80 1 -- 172.21.15.70:0/3077432625 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55661b1c08e0 con 0x55661b1b9690 2024-06-04T05:11:08.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7e191b1700 1 --2- 172.21.15.70:0/3077432625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1ba040 0x55661b1bd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:08.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7e254a9b80 10 client.0 _ll_register_callbacks cb 0x55661b0989d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:08.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7e13fff700 1 --2- 172.21.15.70:0/3077432625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b9690 0x55661b1bd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:08.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7e191b1700 1 -- 172.21.15.70:0/3077432625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b9690 msgr2=0x55661b1bd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:08.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7e191b1700 1 --2- 172.21.15.70:0/3077432625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b9690 0x55661b1bd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7e191b1700 1 -- 172.21.15.70:0/3077432625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b4350 msgr2=0x55661b1bff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:08.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7e191b1700 1 --2- 172.21.15.70:0/3077432625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b4350 0x55661b1bff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7e191b1700 1 -- 172.21.15.70:0/3077432625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55661b04d950 con 0x55661b1ba040 2024-06-04T05:11:08.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7e189b0700 1 --2- 172.21.15.70:0/3077432625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b4350 0x55661b1bff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:08.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7e191b1700 1 --2- 172.21.15.70:0/3077432625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1ba040 0x55661b1bd770 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c004770 tx=0x7f7e0c003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=271a39c43924aa10 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:08.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7df3fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:11:08.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7df3fff700 1 -- 172.21.15.70:0/3077432625 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c00ec70 con 0x55661b1ba040 2024-06-04T05:11:08.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7e254a9b80 1 -- 172.21.15.70:0/3077432625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e080b31d0 con 0x55661b1ba040 2024-06-04T05:11:08.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.004+0000 7f7df3fff700 1 -- 172.21.15.70:0/3077432625 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f7e0c00ee10 con 0x55661b1ba040 2024-06-04T05:11:08.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.005+0000 7f7df3fff700 1 -- 172.21.15.70:0/3077432625 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7e0c0093a0 con 0x55661b1ba040 2024-06-04T05:11:08.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.005+0000 7f7e254a9b80 10 client.19215 fetch_fsmap learned FSMap version 39 2024-06-04T05:11:08.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.005+0000 7f7e254a9b80 1 -- 172.21.15.70:0/3077432625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55661b0b77e0 con 0x55661b1ba040 2024-06-04T05:11:08.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.005+0000 7f7df3fff700 1 -- 172.21.15.70:0/3077432625 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e080b31d0 con 0x55661b1ba040 2024-06-04T05:11:08.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.005+0000 7f7df3fff700 1 -- 172.21.15.70:0/3077432625 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 39) ==== 8374+0+0 (secure 0 0 0) 0x7f7e0c025070 con 0x55661b1ba040 2024-06-04T05:11:08.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.005+0000 7f7e254a9b80 10 client.19215 fetch_fsmap finished waiting for FSMap version 39 2024-06-04T05:11:08.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.005+0000 7f7e254a9b80 10 client.19215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:08.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.005+0000 7f7e254a9b80 20 client.19215 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:08.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.005+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/3077432625 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55661b1cd590 0x55661b1cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:08.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.005+0000 7f7e254a9b80 4 client.19215 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:11:08.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.005+0000 7f7e254a9b80 1 -- 172.21.15.70:0/3077432625 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55661b10fd40 con 0x55661b1cd590 2024-06-04T05:11:08.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.005+0000 7f7e189b0700 1 --2- 172.21.15.70:0/3077432625 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55661b1cd590 0x55661b1cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:08.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.006+0000 7f7df3fff700 10 client.19215 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:08.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.006+0000 7f7e189b0700 1 --2- 172.21.15.70:0/3077432625 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55661b1cd590 0x55661b1cf9c0 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9272dc731c05b90 server_cookie=a116ba6db1fa9e94 in_seq=0 out_seq=0 2024-06-04T05:11:08.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.007+0000 7f7df3fff700 1 -- 172.21.15.70:0/3077432625 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55661b10fd40 con 0x55661b1cd590 2024-06-04T05:11:08.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.007+0000 7f7df3fff700 10 client.19215 handle_command_reply: tid=0 2024-06-04T05:11:08.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.007+0000 7f7e254a9b80 1 client.19215 shutdown 2024-06-04T05:11:08.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.007+0000 7f7e254a9b80 1 -- 172.21.15.70:0/3077432625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1ba040 msgr2=0x55661b1bd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:08.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.007+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/3077432625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1ba040 0x55661b1bd770 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c004770 tx=0x7f7e0c003f80 comp rx=0 tx=0).stop 2024-06-04T05:11:08.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.007+0000 7f7e254a9b80 1 -- 172.21.15.70:0/3077432625 shutdown_connections 2024-06-04T05:11:08.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.007+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/3077432625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1ba040 0x55661b1bd770 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.007+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/3077432625 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55661b1cd590 0x55661b1cf9c0 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.007+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/3077432625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b4350 0x55661b1bff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.007+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/3077432625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b9690 0x55661b1bd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.007+0000 7f7e254a9b80 1 -- 172.21.15.70:0/3077432625 >> 172.21.15.70:0/3077432625 conn(0x55661b0db2c0 msgr2=0x55661b1a9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:08.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.007+0000 7f7df3fff700 0 client.19215 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:08.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.007+0000 7f7e254a9b80 1 -- 172.21.15.70:0/3077432625 shutdown_connections 2024-06-04T05:11:08.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.007+0000 7f7e254a9b80 1 -- 172.21.15.70:0/3077432625 wait complete. 2024-06-04T05:11:08.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.007+0000 7f7e254a9b80 20 client.19215 trim_cache size 0 max 16384 2024-06-04T05:11:08.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.030+0000 7f7e254a9b80 1 Processor -- start 2024-06-04T05:11:08.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.030+0000 7f7e254a9b80 1 -- start start 2024-06-04T05:11:08.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.030+0000 7f7e254a9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b4350 0x55661b1c9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:08.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e254a9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1b9690 0x55661b1b89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:08.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e254a9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1ba040 0x55661b1aa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:08.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e254a9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7e141abb10 con 0x55661b1b9690 2024-06-04T05:11:08.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e254a9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e141ad740 con 0x55661b1ba040 2024-06-04T05:11:08.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e254a9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7e141ac1e0 con 0x55661b1b4350 2024-06-04T05:11:08.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e189b0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b4350 0x55661b1c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:08.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e191b1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1ba040 0x55661b1aa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:08.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e189b0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b4350 0x55661b1c9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:50294/0 (socket says 172.21.15.70:50294) 2024-06-04T05:11:08.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e191b1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1ba040 0x55661b1aa2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46354/0 (socket says 172.21.15.70:46354) 2024-06-04T05:11:08.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e189b0700 1 -- 172.21.15.70:0/3512040333 learned_addr learned my addr 172.21.15.70:0/3512040333 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:08.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e13fff700 1 --2- 172.21.15.70:0/3512040333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1b9690 0x55661b1b89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:08.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e189b0700 1 -- 172.21.15.70:0/3512040333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1ba040 msgr2=0x55661b1aa2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:08.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e189b0700 1 --2- 172.21.15.70:0/3512040333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1ba040 0x55661b1aa2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e189b0700 1 -- 172.21.15.70:0/3512040333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1b9690 msgr2=0x55661b1b89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:08.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e189b0700 1 --2- 172.21.15.70:0/3512040333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1b9690 0x55661b1b89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e189b0700 1 -- 172.21.15.70:0/3512040333 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55661b1d0040 con 0x55661b1b4350 2024-06-04T05:11:08.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e191b1700 1 --2- 172.21.15.70:0/3512040333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1ba040 0x55661b1aa2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:08.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e13fff700 1 --2- 172.21.15.70:0/3512040333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1b9690 0x55661b1b89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:08.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.031+0000 7f7e189b0700 1 --2- 172.21.15.70:0/3512040333 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b4350 0x55661b1c9940 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7e080aa390 tx=0x7f7e08009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3427a3005bdf1871 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:08.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.032+0000 7f7dfe7fc700 1 -- 172.21.15.70:0/3512040333 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e080aa660 con 0x55661b1b4350 2024-06-04T05:11:08.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.032+0000 7f7dfe7fc700 1 -- 172.21.15.70:0/3512040333 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e080aa800 con 0x55661b1b4350 2024-06-04T05:11:08.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.032+0000 7f7dfe7fc700 1 -- 172.21.15.70:0/3512040333 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e080a4b20 con 0x55661b1b4350 2024-06-04T05:11:08.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.032+0000 7f7e254a9b80 1 -- 172.21.15.70:0/3512040333 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b4350 msgr2=0x55661b1c9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:08.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.032+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/3512040333 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b4350 0x55661b1c9940 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7e080aa390 tx=0x7f7e08009310 comp rx=0 tx=0).stop 2024-06-04T05:11:08.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.033+0000 7f7e254a9b80 1 -- 172.21.15.70:0/3512040333 shutdown_connections 2024-06-04T05:11:08.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.033+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/3512040333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1ba040 0x55661b1aa2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.033+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/3512040333 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b4350 0x55661b1c9940 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.033+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/3512040333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1b9690 0x55661b1b89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.033+0000 7f7e254a9b80 1 -- 172.21.15.70:0/3512040333 >> 172.21.15.70:0/3512040333 conn(0x55661b0db2c0 msgr2=0x55661b1af1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:08.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.033+0000 7f7e254a9b80 1 -- 172.21.15.70:0/3512040333 shutdown_connections 2024-06-04T05:11:08.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.033+0000 7f7e254a9b80 1 -- 172.21.15.70:0/3512040333 wait complete. 2024-06-04T05:11:08.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.033+0000 7f7e254a9b80 1 -- start start 2024-06-04T05:11:08.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.033+0000 7f7e254a9b80 1 Processor -- start 2024-06-04T05:11:08.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e254a9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b4350 0x55661b1b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:08.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e254a9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b9690 0x55661b1b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:08.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e254a9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1ba040 0x55661b1b2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:08.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e254a9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7e141ad010 con 0x55661b1ba040 2024-06-04T05:11:08.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e254a9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e141ac8e0 con 0x55661b1b9690 2024-06-04T05:11:08.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e254a9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7e141a9ad0 con 0x55661b1b4350 2024-06-04T05:11:08.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e254a9b80 10 client.0 _ll_register_callbacks cb 0x55661b0989d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:08.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e191b1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1ba040 0x55661b1b2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:08.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e189b0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b4350 0x55661b1b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:08.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e191b1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1ba040 0x55661b1b2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57656/0 (socket says 172.21.15.70:57656) 2024-06-04T05:11:08.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e191b1700 1 -- 172.21.15.70:0/239660949 learned_addr learned my addr 172.21.15.70:0/239660949 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:08.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e189b0700 1 -- 172.21.15.70:0/239660949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b9690 msgr2=0x55661b1b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:08.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e189b0700 1 --2- 172.21.15.70:0/239660949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b9690 0x55661b1b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e189b0700 1 -- 172.21.15.70:0/239660949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1ba040 msgr2=0x55661b1b2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:08.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e189b0700 1 --2- 172.21.15.70:0/239660949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1ba040 0x55661b1b2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e189b0700 1 -- 172.21.15.70:0/239660949 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55661b1b2890 con 0x55661b1b4350 2024-06-04T05:11:08.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e191b1700 1 --2- 172.21.15.70:0/239660949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1ba040 0x55661b1b2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:08.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e189b0700 1 --2- 172.21.15.70:0/239660949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b4350 0x55661b1b36e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7e08022c70 tx=0x7f7e08005ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eadbb87c52f2e6f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:08.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7df3fff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:11:08.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7df3fff700 1 -- 172.21.15.70:0/239660949 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e080c4060 con 0x55661b1b4350 2024-06-04T05:11:08.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7e254a9b80 1 -- 172.21.15.70:0/239660949 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e080aa800 con 0x55661b1b4350 2024-06-04T05:11:08.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.034+0000 7f7df3fff700 1 -- 172.21.15.70:0/239660949 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f7e080a9660 con 0x55661b1b4350 2024-06-04T05:11:08.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.035+0000 7f7df3fff700 1 -- 172.21.15.70:0/239660949 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7e080aa660 con 0x55661b1b4350 2024-06-04T05:11:08.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.035+0000 7f7e254a9b80 10 client.28222 fetch_fsmap learned FSMap version 39 2024-06-04T05:11:08.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.035+0000 7f7e254a9b80 1 -- 172.21.15.70:0/239660949 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55661b1c5c60 con 0x55661b1b4350 2024-06-04T05:11:08.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.035+0000 7f7df3fff700 1 -- 172.21.15.70:0/239660949 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e080aa800 con 0x55661b1b4350 2024-06-04T05:11:08.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.035+0000 7f7df3fff700 1 -- 172.21.15.70:0/239660949 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 39) ==== 8374+0+0 (secure 0 0 0) 0x7f7e080a5d70 con 0x55661b1b4350 2024-06-04T05:11:08.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.035+0000 7f7e254a9b80 10 client.28222 fetch_fsmap finished waiting for FSMap version 39 2024-06-04T05:11:08.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.035+0000 7f7e254a9b80 10 client.28222 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:08.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.035+0000 7f7e254a9b80 20 client.28222 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:08.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.035+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/239660949 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55661b1c5ef0 0x55661b1c6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:08.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.035+0000 7f7e254a9b80 4 client.28222 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:11:08.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.035+0000 7f7e254a9b80 1 -- 172.21.15.70:0/239660949 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55661b109520 con 0x55661b1c5ef0 2024-06-04T05:11:08.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.035+0000 7f7e13fff700 1 --2- 172.21.15.70:0/239660949 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55661b1c5ef0 0x55661b1c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:08.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.036+0000 7f7df3fff700 10 client.28222 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:08.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.036+0000 7f7e13fff700 1 --2- 172.21.15.70:0/239660949 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55661b1c5ef0 0x55661b1c6300 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64ab5a6648cc02fd server_cookie=6263000e99a71cdb in_seq=0 out_seq=0 2024-06-04T05:11:08.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.036+0000 7f7df3fff700 1 -- 172.21.15.70:0/239660949 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55661b109520 con 0x55661b1c5ef0 2024-06-04T05:11:08.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.036+0000 7f7df3fff700 10 client.28222 handle_command_reply: tid=0 2024-06-04T05:11:08.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.036+0000 7f7e254a9b80 1 client.28222 shutdown 2024-06-04T05:11:08.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.037+0000 7f7e254a9b80 1 -- 172.21.15.70:0/239660949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b4350 msgr2=0x55661b1b36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:08.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.037+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/239660949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b4350 0x55661b1b36e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7e08022c70 tx=0x7f7e08005ad0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.037+0000 7f7e254a9b80 1 -- 172.21.15.70:0/239660949 shutdown_connections 2024-06-04T05:11:08.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.037+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/239660949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55661b1ba040 0x55661b1b2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.037+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/239660949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55661b1b4350 0x55661b1b36e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.037+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/239660949 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55661b1c5ef0 0x55661b1c6300 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.037+0000 7f7e254a9b80 1 --2- 172.21.15.70:0/239660949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55661b1b9690 0x55661b1b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.037+0000 7f7e254a9b80 1 -- 172.21.15.70:0/239660949 >> 172.21.15.70:0/239660949 conn(0x55661b0db2c0 msgr2=0x55661b1bf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:08.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.037+0000 7f7df3fff700 0 client.28222 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:08.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.037+0000 7f7e254a9b80 1 -- 172.21.15.70:0/239660949 shutdown_connections 2024-06-04T05:11:08.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.037+0000 7f7e254a9b80 1 -- 172.21.15.70:0/239660949 wait complete. 2024-06-04T05:11:08.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.037+0000 7f7e254a9b80 20 client.28222 trim_cache size 0 max 16384 2024-06-04T05:11:08.044 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:11:08.044 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:11:08.044 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "9a775736-8d58-480c-ad11-104e479cc479", 2024-06-04T05:11:08.044 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:11:08.044 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:11:08.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.037+0000 7f7e1ac14700 1 -- 172.21.15.70:0/820804046 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7e040b29f0 msgr2=0x7f7e040b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:08.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.038+0000 7f7e1ac14700 1 --2- 172.21.15.70:0/820804046 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7e040b29f0 0x7f7e040b4eb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7e14079a20 tx=0x7f7e00009260 comp rx=0 tx=0).stop 2024-06-04T05:11:08.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.038+0000 7f7e1ac14700 1 -- 172.21.15.70:0/820804046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e14102430 msgr2=0x7f7e1419f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:08.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.038+0000 7f7e1ac14700 1 --2- 172.21.15.70:0/820804046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e14102430 0x7f7e1419f7f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7e08006010 tx=0x7f7e080047b0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.038+0000 7f7e1ac14700 1 -- 172.21.15.70:0/820804046 shutdown_connections 2024-06-04T05:11:08.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.038+0000 7f7e1ac14700 1 --2- 172.21.15.70:0/820804046 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7e040b29f0 0x7f7e040b4eb0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.038+0000 7f7e1ac14700 1 --2- 172.21.15.70:0/820804046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e14108510 0x7f7e14199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.038+0000 7f7e1ac14700 1 --2- 172.21.15.70:0/820804046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e14102430 0x7f7e1419f7f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.038+0000 7f7e1ac14700 1 --2- 172.21.15.70:0/820804046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e14102df0 0x7f7e1419fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:08.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.038+0000 7f7e1ac14700 1 -- 172.21.15.70:0/820804046 >> 172.21.15.70:0/820804046 conn(0x7f7e140793b0 msgr2=0x7f7e1406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:08.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.038+0000 7f7e1ac14700 1 -- 172.21.15.70:0/820804046 shutdown_connections 2024-06-04T05:11:08.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:08.038+0000 7f7e1ac14700 1 -- 172.21.15.70:0/820804046 wait complete. 2024-06-04T05:11:08.694 INFO:tasks.workunit.client.0.smithi070.stdout: 0 cleanup 601 sec 2024-06-04T05:11:08.695 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-06-04T05:11:08.695 INFO:tasks.workunit.client.0.smithi070.stdout: Operation Count AvgLat MaxLat 2024-06-04T05:11:08.695 INFO:tasks.workunit.client.0.smithi070.stdout: ---------------------------------------- 2024-06-04T05:11:08.695 INFO:tasks.workunit.client.0.smithi070.stdout: NTCreateX 30054 8.734 32430.771 2024-06-04T05:11:08.695 INFO:tasks.workunit.client.0.smithi070.stdout: Close 22068 0.043 0.565 2024-06-04T05:11:08.695 INFO:tasks.workunit.client.0.smithi070.stdout: Rename 1273 56.669 4785.734 2024-06-04T05:11:08.695 INFO:tasks.workunit.client.0.smithi070.stdout: Unlink 6074 3.957 47.467 2024-06-04T05:11:08.695 INFO:tasks.workunit.client.0.smithi070.stdout: Qpathinfo 27258 3.094 972.749 2024-06-04T05:11:08.695 INFO:tasks.workunit.client.0.smithi070.stdout: Qfileinfo 4746 0.068 0.349 2024-06-04T05:11:08.695 INFO:tasks.workunit.client.0.smithi070.stdout: Qfsinfo 4995 0.302 3.184 2024-06-04T05:11:08.695 INFO:tasks.workunit.client.0.smithi070.stdout: Sfileinfo 2450 13.985 21587.208 2024-06-04T05:11:08.695 INFO:tasks.workunit.client.0.smithi070.stdout: Find 10532 10.167 92.182 2024-06-04T05:11:08.696 INFO:tasks.workunit.client.0.smithi070.stdout: WriteX 14839 1.359 17785.428 2024-06-04T05:11:08.696 INFO:tasks.workunit.client.0.smithi070.stdout: ReadX 47096 0.004 1.304 2024-06-04T05:11:08.696 INFO:tasks.workunit.client.0.smithi070.stdout: LockX 96 0.644 1.316 2024-06-04T05:11:08.696 INFO:tasks.workunit.client.0.smithi070.stdout: UnlockX 96 0.545 1.103 2024-06-04T05:11:08.696 INFO:tasks.workunit.client.0.smithi070.stdout: Flush 2106 2.450 53.019 2024-06-04T05:11:08.696 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-06-04T05:11:08.696 INFO:tasks.workunit.client.0.smithi070.stdout:Throughput 1.5542 MB/sec 1 clients 1 procs max_latency=32430.776 ms 2024-06-04T05:11:08.696 INFO:tasks.workunit.client.0.smithi070.stderr:+ dbench 10 2024-06-04T05:11:08.697 INFO:tasks.workunit.client.0.smithi070.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-06-04T05:11:08.697 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-06-04T05:11:08.697 INFO:tasks.workunit.client.0.smithi070.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-06-04T05:11:08.697 INFO:tasks.workunit.client.0.smithi070.stdout:8 of 10 processes prepared for launch 0 sec 2024-06-04T05:11:08.796 INFO:tasks.workunit.client.0.smithi070.stdout:10 of 10 processes prepared for launch 0 sec 2024-06-04T05:11:08.796 INFO:tasks.workunit.client.0.smithi070.stdout:releasing clients 2024-06-04T05:11:08.850 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:11:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:08 smithi070 ceph-mon[32631]: osdmap e96: 12 total, 12 up, 12 in 2024-06-04T05:11:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:08 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:11:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:08 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:08 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:11:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:08 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:08 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:11:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:08 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:11:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:08 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:08 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:08 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:11:09.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:08 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:11:09.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:08 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:08 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:11:09.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:08 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:08 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:08 smithi070 ceph-mon[32631]: pgmap v436: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 873 KiB/s wr, 130 op/s 2024-06-04T05:11:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:08 smithi112 ceph-mon[36453]: osdmap e96: 12 total, 12 up, 12 in 2024-06-04T05:11:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:08 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:11:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:08 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:08 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:11:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:08 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:08 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:11:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:08 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:11:09.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:08 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:08 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:08 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:11:09.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:08 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:11:09.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:08 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:08 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:11:09.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:08 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:08 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:08 smithi112 ceph-mon[36453]: pgmap v436: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 873 KiB/s wr, 130 op/s 2024-06-04T05:11:09.203 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:11:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:08 smithi098 ceph-mon[35569]: osdmap e96: 12 total, 12 up, 12 in 2024-06-04T05:11:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:08 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:11:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:08 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:08 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:11:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:08 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:08 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:11:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:08 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:11:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:08 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:08 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:08 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:11:09.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:08 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:11:09.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:08 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:08 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:11:09.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:08 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:08 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:09.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:08 smithi098 ceph-mon[35569]: pgmap v436: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 873 KiB/s wr, 130 op/s 2024-06-04T05:11:09.796 INFO:tasks.workunit.client.0.smithi070.stdout: 10 274 102.88 MB/sec warmup 1 sec latency 71.489 ms 2024-06-04T05:11:10.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:09 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:11:10.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:09 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:10.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:09 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:11:10.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:09 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:11:10.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:09 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:10.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:09 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:11:10.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:09 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:11:10.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:09 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:10.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:09 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:11:10.796 INFO:tasks.workunit.client.0.smithi070.stdout: 10 506 89.13 MB/sec warmup 2 sec latency 1047.419 ms 2024-06-04T05:11:11.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:10 smithi070 ceph-mon[32631]: pgmap v437: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 820 KiB/s wr, 110 op/s 2024-06-04T05:11:11.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:10 smithi112 ceph-mon[36453]: pgmap v437: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 820 KiB/s wr, 110 op/s 2024-06-04T05:11:11.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:10 smithi098 ceph-mon[35569]: pgmap v437: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 820 KiB/s wr, 110 op/s 2024-06-04T05:11:11.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.300+0000 7efced212700 1 -- 172.21.15.70:0/748110636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efce8108520 msgr2=0x7efce8108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:11.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.300+0000 7efced212700 1 --2- 172.21.15.70:0/748110636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efce8108520 0x7efce8108900 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7efcd8009990 tx=0x7efcd8009ca0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.301+0000 7efced212700 1 -- 172.21.15.70:0/748110636 shutdown_connections 2024-06-04T05:11:11.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.301+0000 7efced212700 1 --2- 172.21.15.70:0/748110636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efce8102e00 0x7efce810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.301+0000 7efced212700 1 --2- 172.21.15.70:0/748110636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efce8108520 0x7efce8108900 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.301+0000 7efced212700 1 --2- 172.21.15.70:0/748110636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efce8102440 0x7efce81028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.301+0000 7efced212700 1 -- 172.21.15.70:0/748110636 >> 172.21.15.70:0/748110636 conn(0x7efce80793f0 msgr2=0x7efce8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:11.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.301+0000 7efced212700 1 -- 172.21.15.70:0/748110636 shutdown_connections 2024-06-04T05:11:11.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.301+0000 7efced212700 1 -- 172.21.15.70:0/748110636 wait complete. 2024-06-04T05:11:11.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.302+0000 7efced212700 1 Processor -- start 2024-06-04T05:11:11.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.302+0000 7efced212700 1 -- start start 2024-06-04T05:11:11.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.302+0000 7efced212700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efce8102440 0x7efce819f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.302+0000 7efced212700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efce8102e00 0x7efce819fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.302+0000 7efced212700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efce8108520 0x7efce8199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.302+0000 7efced212700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efce81124f0 con 0x7efce8102440 2024-06-04T05:11:11.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efced212700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efce8112370 con 0x7efce8102e00 2024-06-04T05:11:11.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efced212700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efce8112670 con 0x7efce8108520 2024-06-04T05:11:11.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efce6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efce8102440 0x7efce819f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efcde59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efce8102e00 0x7efce819fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efce6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efce8102440 0x7efce819f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57668/0 (socket says 172.21.15.70:57668) 2024-06-04T05:11:11.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efce6d9d700 1 -- 172.21.15.70:0/803770086 learned_addr learned my addr 172.21.15.70:0/803770086 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:11.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efce759e700 1 --2- 172.21.15.70:0/803770086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efce8108520 0x7efce8199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efce6d9d700 1 -- 172.21.15.70:0/803770086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efce8108520 msgr2=0x7efce8199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:11.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efce6d9d700 1 --2- 172.21.15.70:0/803770086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efce8108520 0x7efce8199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efce6d9d700 1 -- 172.21.15.70:0/803770086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efce8102e00 msgr2=0x7efce819fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:11.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efce6d9d700 1 --2- 172.21.15.70:0/803770086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efce8102e00 0x7efce819fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efce6d9d700 1 -- 172.21.15.70:0/803770086 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce819a1a0 con 0x7efce8102440 2024-06-04T05:11:11.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efcde59c700 1 --2- 172.21.15.70:0/803770086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efce8102e00 0x7efce819fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:11.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efce6d9d700 1 --2- 172.21.15.70:0/803770086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efce8102440 0x7efce819f880 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7efcd80098e0 tx=0x7efcd80057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:11.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efce4d99700 1 -- 172.21.15.70:0/803770086 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd8027070 con 0x7efce8102440 2024-06-04T05:11:11.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efced212700 1 -- 172.21.15.70:0/803770086 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efce819a430 con 0x7efce8102440 2024-06-04T05:11:11.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efce4d99700 1 -- 172.21.15.70:0/803770086 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efcd8022470 con 0x7efce8102440 2024-06-04T05:11:11.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.303+0000 7efce4d99700 1 -- 172.21.15.70:0/803770086 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd8017650 con 0x7efce8102440 2024-06-04T05:11:11.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.304+0000 7efced212700 1 -- 172.21.15.70:0/803770086 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efce8103fb0 con 0x7efce8102440 2024-06-04T05:11:11.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.306+0000 7efce4d99700 1 -- 172.21.15.70:0/803770086 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7efcd802a070 con 0x7efce8102440 2024-06-04T05:11:11.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.307+0000 7efce4d99700 1 --2- 172.21.15.70:0/803770086 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efcd40b29f0 0x7efcd40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.307+0000 7efcde59c700 1 --2- 172.21.15.70:0/803770086 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efcd40b29f0 0x7efcd40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.307+0000 7efce4d99700 1 -- 172.21.15.70:0/803770086 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7efcd80e3320 con 0x7efce8102440 2024-06-04T05:11:11.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.307+0000 7efcde59c700 1 --2- 172.21.15.70:0/803770086 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efcd40b29f0 0x7efcd40b4eb0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efce8079a90 tx=0x7efcd0012040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:11.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.308+0000 7efcf7aa7b80 1 Processor -- start 2024-06-04T05:11:11.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.308+0000 7efcf7aa7b80 1 -- start start 2024-06-04T05:11:11.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.308+0000 7efcf7aa7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f0860 0x558ccc7f0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.309+0000 7efcf7aa7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f1210 0x558ccc7f4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.309+0000 7efcf7aa7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f4f60 0x558ccc7f53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.309+0000 7efcf7aa7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558ccc74fc90 con 0x558ccc7f1210 2024-06-04T05:11:11.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.309+0000 7efcf7aa7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558ccc73f830 con 0x558ccc7f0860 2024-06-04T05:11:11.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.309+0000 7efcf7aa7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558ccc743870 con 0x558ccc7f4f60 2024-06-04T05:11:11.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.309+0000 7efcde59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f1210 0x558ccc7f4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.309+0000 7efce6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f0860 0x558ccc7f0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.309+0000 7efcde59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f1210 0x558ccc7f4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57684/0 (socket says 172.21.15.70:57684) 2024-06-04T05:11:11.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.309+0000 7efce759e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f4f60 0x558ccc7f53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.309+0000 7efcde59c700 1 -- 172.21.15.70:0/1913372117 learned_addr learned my addr 172.21.15.70:0/1913372117 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:11.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.309+0000 7efce6d9d700 1 -- 172.21.15.70:0/1913372117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f4f60 msgr2=0x558ccc7f53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:11.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.309+0000 7efce6d9d700 1 --2- 172.21.15.70:0/1913372117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f4f60 0x558ccc7f53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.309+0000 7efce6d9d700 1 -- 172.21.15.70:0/1913372117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f1210 msgr2=0x558ccc7f4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:11.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.309+0000 7efce6d9d700 1 --2- 172.21.15.70:0/1913372117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f1210 0x558ccc7f4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.309+0000 7efce6d9d700 1 -- 172.21.15.70:0/1913372117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ccc68f8d0 con 0x558ccc7f0860 2024-06-04T05:11:11.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.309+0000 7efce759e700 1 --2- 172.21.15.70:0/1913372117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f4f60 0x558ccc7f53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:11.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.309+0000 7efce6d9d700 1 --2- 172.21.15.70:0/1913372117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f0860 0x558ccc7f0c40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7efcd80e0a40 tx=0x7efcd80aa170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab4a800a2310ddc8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:11.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.310+0000 7efcdd59a700 1 -- 172.21.15.70:0/1913372117 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd80aea10 con 0x558ccc7f0860 2024-06-04T05:11:11.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.310+0000 7efcdd59a700 1 -- 172.21.15.70:0/1913372117 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efcd80b3430 con 0x558ccc7f0860 2024-06-04T05:11:11.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.310+0000 7efcdd59a700 1 -- 172.21.15.70:0/1913372117 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd80aea10 con 0x558ccc7f0860 2024-06-04T05:11:11.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.311+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/1913372117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f0860 msgr2=0x558ccc7f0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:11.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.311+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/1913372117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f0860 0x558ccc7f0c40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7efcd80e0a40 tx=0x7efcd80aa170 comp rx=0 tx=0).stop 2024-06-04T05:11:11.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.311+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/1913372117 shutdown_connections 2024-06-04T05:11:11.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.311+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/1913372117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f4f60 0x558ccc7f53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.311+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/1913372117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f0860 0x558ccc7f0c40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.311+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/1913372117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f1210 0x558ccc7f4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.311+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/1913372117 >> 172.21.15.70:0/1913372117 conn(0x558ccc71e080 msgr2=0x558ccc7eae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:11.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.311+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/1913372117 shutdown_connections 2024-06-04T05:11:11.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.311+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/1913372117 wait complete. 2024-06-04T05:11:11.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.311+0000 7efcf7aa7b80 1 -- start start 2024-06-04T05:11:11.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.311+0000 7efcf7aa7b80 1 Processor -- start 2024-06-04T05:11:11.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efcf7aa7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f0860 0x558ccc7fc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efcf7aa7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f1210 0x558ccc7f9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efcf7aa7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f4f60 0x558ccc7f9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efcf7aa7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558ccc645fa0 con 0x558ccc7f4f60 2024-06-04T05:11:11.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efcf7aa7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558ccc6e17c0 con 0x558ccc7f0860 2024-06-04T05:11:11.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efcf7aa7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558ccc7fcee0 con 0x558ccc7f1210 2024-06-04T05:11:11.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efcf7aa7b80 10 client.0 _ll_register_callbacks cb 0x558ccc6da920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:11.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efce759e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f4f60 0x558ccc7f9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efcde59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f1210 0x558ccc7f9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efce759e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f4f60 0x558ccc7f9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57690/0 (socket says 172.21.15.70:57690) 2024-06-04T05:11:11.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efce759e700 1 -- 172.21.15.70:0/2819916046 learned_addr learned my addr 172.21.15.70:0/2819916046 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:11.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efce6d9d700 1 --2- 172.21.15.70:0/2819916046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f0860 0x558ccc7fc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efcde59c700 1 -- 172.21.15.70:0/2819916046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f0860 msgr2=0x558ccc7fc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:11.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efcde59c700 1 --2- 172.21.15.70:0/2819916046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f0860 0x558ccc7fc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efcde59c700 1 -- 172.21.15.70:0/2819916046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f4f60 msgr2=0x558ccc7f9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:11.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efcde59c700 1 --2- 172.21.15.70:0/2819916046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f4f60 0x558ccc7f9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efcde59c700 1 -- 172.21.15.70:0/2819916046 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ccc6de690 con 0x558ccc7f1210 2024-06-04T05:11:11.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efcde59c700 1 --2- 172.21.15.70:0/2819916046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f1210 0x558ccc7f9740 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7efcd0012010 tx=0x7efcd00047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8012e32eafed163 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:11.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efcce7fc700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:11:11.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.312+0000 7efcce7fc700 1 -- 172.21.15.70:0/2819916046 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd001a9d0 con 0x558ccc7f1210 2024-06-04T05:11:11.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.313+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/2819916046 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcd80b3430 con 0x558ccc7f1210 2024-06-04T05:11:11.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.313+0000 7efcce7fc700 1 -- 172.21.15.70:0/2819916046 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7efcd001ab70 con 0x558ccc7f1210 2024-06-04T05:11:11.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.313+0000 7efcce7fc700 1 -- 172.21.15.70:0/2819916046 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7efcd0036050 con 0x558ccc7f1210 2024-06-04T05:11:11.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.313+0000 7efcf7aa7b80 10 client.28234 fetch_fsmap learned FSMap version 39 2024-06-04T05:11:11.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.313+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/2819916046 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x558ccc807660 con 0x558ccc7f1210 2024-06-04T05:11:11.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.313+0000 7efcce7fc700 1 -- 172.21.15.70:0/2819916046 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efcd80b3430 con 0x558ccc7f1210 2024-06-04T05:11:11.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.313+0000 7efcce7fc700 1 -- 172.21.15.70:0/2819916046 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 39) ==== 8374+0+0 (secure 0 0 0) 0x7efcd001c3f0 con 0x558ccc7f1210 2024-06-04T05:11:11.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.313+0000 7efcf7aa7b80 10 client.28234 fetch_fsmap finished waiting for FSMap version 39 2024-06-04T05:11:11.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.313+0000 7efcf7aa7b80 10 client.28234 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:11.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.313+0000 7efcf7aa7b80 20 client.28234 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:11.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.313+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/2819916046 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558ccc809cf0 0x558ccc80c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.313+0000 7efcf7aa7b80 4 client.28234 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:11:11.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.313+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/2819916046 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ccc7bc9e0 con 0x558ccc809cf0 2024-06-04T05:11:11.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.313+0000 7efce6d9d700 1 --2- 172.21.15.70:0/2819916046 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558ccc809cf0 0x558ccc80c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.314+0000 7efcce7fc700 10 client.28234 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:11.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.314+0000 7efce6d9d700 1 --2- 172.21.15.70:0/2819916046 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558ccc809cf0 0x558ccc80c110 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71156f34a81bbeea server_cookie=b6fdfe995eb6edc4 in_seq=0 out_seq=0 2024-06-04T05:11:11.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.316+0000 7efcce7fc700 1 -- 172.21.15.70:0/2819916046 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558ccc7bc9e0 con 0x558ccc809cf0 2024-06-04T05:11:11.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.316+0000 7efcce7fc700 10 client.28234 handle_command_reply: tid=0 2024-06-04T05:11:11.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.316+0000 7efcf7aa7b80 1 client.28234 shutdown 2024-06-04T05:11:11.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.316+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/2819916046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f1210 msgr2=0x558ccc7f9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:11.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.316+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/2819916046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f1210 0x558ccc7f9740 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7efcd0012010 tx=0x7efcd00047c0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.316+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/2819916046 shutdown_connections 2024-06-04T05:11:11.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.316+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/2819916046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f4f60 0x558ccc7f9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.316+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/2819916046 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558ccc809cf0 0x558ccc80c110 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.316+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/2819916046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f0860 0x558ccc7fc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.316+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/2819916046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f1210 0x558ccc7f9740 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.316+0000 7efcce7fc700 0 client.28234 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:11.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.316+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/2819916046 >> 172.21.15.70:0/2819916046 conn(0x558ccc71e080 msgr2=0x558ccc7ea510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:11.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.316+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/2819916046 shutdown_connections 2024-06-04T05:11:11.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.316+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/2819916046 wait complete. 2024-06-04T05:11:11.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.316+0000 7efcf7aa7b80 20 client.28234 trim_cache size 0 max 16384 2024-06-04T05:11:11.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.341+0000 7efcf7aa7b80 1 Processor -- start 2024-06-04T05:11:11.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.341+0000 7efcf7aa7b80 1 -- start start 2024-06-04T05:11:11.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.341+0000 7efcf7aa7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f0860 0x558ccc7eac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.341+0000 7efcf7aa7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f1210 0x558ccc805640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efcf7aa7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f4f60 0x558ccc805b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efcf7aa7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efce81abba0 con 0x558ccc7f0860 2024-06-04T05:11:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efcf7aa7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efce81ad7d0 con 0x558ccc7f4f60 2024-06-04T05:11:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efcf7aa7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efce81ac270 con 0x558ccc7f1210 2024-06-04T05:11:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efce6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f0860 0x558ccc7eac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efce6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f0860 0x558ccc7eac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57704/0 (socket says 172.21.15.70:57704) 2024-06-04T05:11:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efce6d9d700 1 -- 172.21.15.70:0/502421358 learned_addr learned my addr 172.21.15.70:0/502421358 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efcde59c700 1 --2- 172.21.15.70:0/502421358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f1210 0x558ccc805640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efce759e700 1 --2- 172.21.15.70:0/502421358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f4f60 0x558ccc805b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efce6d9d700 1 -- 172.21.15.70:0/502421358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f1210 msgr2=0x558ccc805640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efce6d9d700 1 --2- 172.21.15.70:0/502421358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f1210 0x558ccc805640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efce6d9d700 1 -- 172.21.15.70:0/502421358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f4f60 msgr2=0x558ccc805b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efce6d9d700 1 --2- 172.21.15.70:0/502421358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f4f60 0x558ccc805b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efce6d9d700 1 -- 172.21.15.70:0/502421358 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ccc80c770 con 0x558ccc7f0860 2024-06-04T05:11:11.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efce6d9d700 1 --2- 172.21.15.70:0/502421358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f0860 0x558ccc7eac90 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7efcd80058b0 tx=0x7efcd8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91807791cc54a6ff server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:11.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efcdcd99700 1 -- 172.21.15.70:0/502421358 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd80094b0 con 0x558ccc7f0860 2024-06-04T05:11:11.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efcdcd99700 1 -- 172.21.15.70:0/502421358 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efcd80a68f0 con 0x558ccc7f0860 2024-06-04T05:11:11.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.342+0000 7efcdcd99700 1 -- 172.21.15.70:0/502421358 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd80aa8c0 con 0x558ccc7f0860 2024-06-04T05:11:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.343+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/502421358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f0860 msgr2=0x558ccc7eac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.343+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/502421358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f0860 0x558ccc7eac90 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7efcd80058b0 tx=0x7efcd8009310 comp rx=0 tx=0).stop 2024-06-04T05:11:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.344+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/502421358 shutdown_connections 2024-06-04T05:11:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.344+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/502421358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f4f60 0x558ccc805b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.344+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/502421358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f0860 0x558ccc7eac90 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.344+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/502421358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f1210 0x558ccc805640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.344+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/502421358 >> 172.21.15.70:0/502421358 conn(0x558ccc71e080 msgr2=0x558ccc804f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.344+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/502421358 shutdown_connections 2024-06-04T05:11:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.344+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/502421358 wait complete. 2024-06-04T05:11:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.344+0000 7efcf7aa7b80 1 -- start start 2024-06-04T05:11:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.344+0000 7efcf7aa7b80 1 Processor -- start 2024-06-04T05:11:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efcf7aa7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f0860 0x558ccc8689f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efcf7aa7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f1210 0x558ccc868f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efcf7aa7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f4f60 0x558ccc873890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efcf7aa7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efce81ad0a0 con 0x558ccc7f0860 2024-06-04T05:11:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efcf7aa7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efce81ac970 con 0x558ccc7f1210 2024-06-04T05:11:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efcf7aa7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efce81a9b80 con 0x558ccc7f4f60 2024-06-04T05:11:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efcf7aa7b80 10 client.0 _ll_register_callbacks cb 0x558ccc6da920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efcde59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f1210 0x558ccc868f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efce6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f0860 0x558ccc8689f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efce6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f0860 0x558ccc8689f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57706/0 (socket says 172.21.15.70:57706) 2024-06-04T05:11:11.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efce6d9d700 1 -- 172.21.15.70:0/620210975 learned_addr learned my addr 172.21.15.70:0/620210975 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:11.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efce759e700 1 --2- 172.21.15.70:0/620210975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f4f60 0x558ccc873890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efcde59c700 1 -- 172.21.15.70:0/620210975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f4f60 msgr2=0x558ccc873890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:11.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efcde59c700 1 --2- 172.21.15.70:0/620210975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f4f60 0x558ccc873890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efcde59c700 1 -- 172.21.15.70:0/620210975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f0860 msgr2=0x558ccc8689f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:11.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efcde59c700 1 --2- 172.21.15.70:0/620210975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f0860 0x558ccc8689f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efcde59c700 1 -- 172.21.15.70:0/620210975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ccc873dd0 con 0x558ccc7f1210 2024-06-04T05:11:11.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efce6d9d700 1 --2- 172.21.15.70:0/620210975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f0860 0x558ccc8689f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:11.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.345+0000 7efce759e700 1 --2- 172.21.15.70:0/620210975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f4f60 0x558ccc873890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:11:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.346+0000 7efcde59c700 1 --2- 172.21.15.70:0/620210975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f1210 0x558ccc868f30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efcd001c7c0 tx=0x7efcd001c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6d027c6fc89b0f2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.346+0000 7efcce7fc700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:11:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.346+0000 7efcce7fc700 1 -- 172.21.15.70:0/620210975 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd0010230 con 0x558ccc7f1210 2024-06-04T05:11:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.346+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/620210975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcd80a68f0 con 0x558ccc7f1210 2024-06-04T05:11:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.346+0000 7efcce7fc700 1 -- 172.21.15.70:0/620210975 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7efcd00103d0 con 0x558ccc7f1210 2024-06-04T05:11:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.346+0000 7efcce7fc700 1 -- 172.21.15.70:0/620210975 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7efcd001a7d0 con 0x558ccc7f1210 2024-06-04T05:11:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.346+0000 7efcf7aa7b80 10 client.28103 fetch_fsmap learned FSMap version 39 2024-06-04T05:11:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.346+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/620210975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x558ccc87d920 con 0x558ccc7f1210 2024-06-04T05:11:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.347+0000 7efcce7fc700 1 -- 172.21.15.70:0/620210975 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efcd80a68f0 con 0x558ccc7f1210 2024-06-04T05:11:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.347+0000 7efcce7fc700 1 -- 172.21.15.70:0/620210975 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 39) ==== 8374+0+0 (secure 0 0 0) 0x7efcd0028670 con 0x558ccc7f1210 2024-06-04T05:11:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.347+0000 7efcf7aa7b80 10 client.28103 fetch_fsmap finished waiting for FSMap version 39 2024-06-04T05:11:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.347+0000 7efcf7aa7b80 10 client.28103 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.347+0000 7efcf7aa7b80 20 client.28103 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.347+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/620210975 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558ccc87e160 0x558ccc880580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.347+0000 7efcf7aa7b80 4 client.28103 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:11:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.347+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/620210975 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558ccc751be0 con 0x558ccc87e160 2024-06-04T05:11:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.347+0000 7efce6d9d700 1 --2- 172.21.15.70:0/620210975 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558ccc87e160 0x558ccc880580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.348+0000 7efce6d9d700 1 --2- 172.21.15.70:0/620210975 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558ccc87e160 0x558ccc880580 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b62d135a39f1afb8 server_cookie=613a3ab366e03cb7 in_seq=0 out_seq=0 2024-06-04T05:11:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.348+0000 7efcce7fc700 10 client.28103 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.349+0000 7efcce7fc700 1 -- 172.21.15.70:0/620210975 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x558ccc751be0 con 0x558ccc87e160 2024-06-04T05:11:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.349+0000 7efcce7fc700 10 client.28103 handle_command_reply: tid=0 2024-06-04T05:11:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.349+0000 7efcf7aa7b80 1 client.28103 shutdown 2024-06-04T05:11:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.349+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/620210975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f1210 msgr2=0x558ccc868f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.349+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/620210975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f1210 0x558ccc868f30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efcd001c7c0 tx=0x7efcd001c680 comp rx=0 tx=0).stop 2024-06-04T05:11:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.349+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/620210975 shutdown_connections 2024-06-04T05:11:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.349+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/620210975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558ccc7f4f60 0x558ccc873890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.350+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/620210975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ccc7f0860 0x558ccc8689f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.350+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/620210975 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558ccc87e160 0x558ccc880580 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.350+0000 7efcf7aa7b80 1 --2- 172.21.15.70:0/620210975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ccc7f1210 0x558ccc868f30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.350+0000 7efcce7fc700 0 client.28103 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:11.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.350+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/620210975 >> 172.21.15.70:0/620210975 conn(0x558ccc71e080 msgr2=0x558ccc8057c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:11.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.350+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/620210975 shutdown_connections 2024-06-04T05:11:11.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.350+0000 7efcf7aa7b80 1 -- 172.21.15.70:0/620210975 wait complete. 2024-06-04T05:11:11.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.350+0000 7efcf7aa7b80 20 client.28103 trim_cache size 0 max 16384 2024-06-04T05:11:11.354 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:11:11.354 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "scrub active (1 inodes in the stack)", 2024-06-04T05:11:11.354 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:11:11.354 INFO:teuthology.orchestra.run.smithi070.stdout: "9a775736-8d58-480c-ad11-104e479cc479": { 2024-06-04T05:11:11.355 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:11:11.355 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "9a775736-8d58-480c-ad11-104e479cc479", 2024-06-04T05:11:11.355 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:11:11.355 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:11:11.355 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:11:11.355 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:11:11.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.351+0000 7efced212700 1 -- 172.21.15.70:0/803770086 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efcd40b29f0 msgr2=0x7efcd40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:11.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.351+0000 7efced212700 1 --2- 172.21.15.70:0/803770086 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efcd40b29f0 0x7efcd40b4eb0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efce8079a90 tx=0x7efcd0012040 comp rx=0 tx=0).stop 2024-06-04T05:11:11.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.351+0000 7efced212700 1 -- 172.21.15.70:0/803770086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efce8102440 msgr2=0x7efce819f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:11.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.351+0000 7efced212700 1 --2- 172.21.15.70:0/803770086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efce8102440 0x7efce819f880 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7efcd80098e0 tx=0x7efcd80057c0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.351+0000 7efced212700 1 -- 172.21.15.70:0/803770086 shutdown_connections 2024-06-04T05:11:11.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.351+0000 7efced212700 1 --2- 172.21.15.70:0/803770086 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efcd40b29f0 0x7efcd40b4eb0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.351+0000 7efced212700 1 --2- 172.21.15.70:0/803770086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efce8108520 0x7efce8199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.351+0000 7efced212700 1 --2- 172.21.15.70:0/803770086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efce8102440 0x7efce819f880 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.351+0000 7efced212700 1 --2- 172.21.15.70:0/803770086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efce8102e00 0x7efce819fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:11.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.351+0000 7efced212700 1 -- 172.21.15.70:0/803770086 >> 172.21.15.70:0/803770086 conn(0x7efce80793f0 msgr2=0x7efce806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:11.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.352+0000 7efced212700 1 -- 172.21.15.70:0/803770086 shutdown_connections 2024-06-04T05:11:11.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:11.352+0000 7efced212700 1 -- 172.21.15.70:0/803770086 wait complete. 2024-06-04T05:11:11.796 INFO:tasks.workunit.client.0.smithi070.stdout: 10 680 66.88 MB/sec warmup 3 sec latency 2047.457 ms 2024-06-04T05:11:12.169 INFO:tasks.cephfs.filesystem:scrub status for tag:9a775736-8d58-480c-ad11-104e479cc479 - {'path': '/', 'tag': '9a775736-8d58-480c-ad11-104e479cc479', 'options': 'recursive,force'} 2024-06-04T05:11:12.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:12 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:11:12.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:12 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:12.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:12 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:11:12.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:12 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:12.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:12 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:11:12.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:12 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:12.796 INFO:tasks.workunit.client.0.smithi070.stdout: 10 810 51.76 MB/sec warmup 4 sec latency 2360.874 ms 2024-06-04T05:11:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:13 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:13 smithi098 ceph-mon[35569]: pgmap v438: 97 pgs: 97 active+clean; 319 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 23 MiB/s wr, 249 op/s 2024-06-04T05:11:13.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:13 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:13.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:13 smithi070 ceph-mon[32631]: pgmap v438: 97 pgs: 97 active+clean; 319 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 23 MiB/s wr, 249 op/s 2024-06-04T05:11:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:13 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:11:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:13 smithi112 ceph-mon[36453]: pgmap v438: 97 pgs: 97 active+clean; 319 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 23 MiB/s wr, 249 op/s 2024-06-04T05:11:13.796 INFO:tasks.workunit.client.0.smithi070.stdout: 10 982 44.17 MB/sec warmup 5 sec latency 192.310 ms 2024-06-04T05:11:14.796 INFO:tasks.workunit.client.0.smithi070.stdout: 10 1180 38.18 MB/sec warmup 6 sec latency 102.218 ms 2024-06-04T05:11:15.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:15 smithi098 ceph-mon[35569]: pgmap v439: 97 pgs: 97 active+clean; 319 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 20 MiB/s wr, 221 op/s 2024-06-04T05:11:15.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:15 smithi070 ceph-mon[32631]: pgmap v439: 97 pgs: 97 active+clean; 319 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 20 MiB/s wr, 221 op/s 2024-06-04T05:11:15.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.625+0000 7fd27cb89640 1 -- 172.21.15.70:0/3881241770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd278103d20 msgr2=0x7fd278104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.625+0000 7fd27cb89640 1 --2- 172.21.15.70:0/3881241770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd278103d20 0x7fd278104100 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fd268009900 tx=0x7fd26802a480 comp rx=0 tx=0).stop 2024-06-04T05:11:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.625+0000 7fd27cb89640 1 -- 172.21.15.70:0/3881241770 shutdown_connections 2024-06-04T05:11:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.625+0000 7fd27cb89640 1 --2- 172.21.15.70:0/3881241770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd27810d060 0x7fd27810f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.625+0000 7fd27cb89640 1 --2- 172.21.15.70:0/3881241770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd278104640 0x7fd27810cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.625+0000 7fd27cb89640 1 --2- 172.21.15.70:0/3881241770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd278103d20 0x7fd278104100 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.625+0000 7fd27cb89640 1 -- 172.21.15.70:0/3881241770 >> 172.21.15.70:0/3881241770 conn(0x7fd278078ad0 msgr2=0x7fd278078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.625+0000 7fd27cb89640 1 -- 172.21.15.70:0/3881241770 shutdown_connections 2024-06-04T05:11:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.625+0000 7fd27cb89640 1 -- 172.21.15.70:0/3881241770 wait complete. 2024-06-04T05:11:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.626+0000 7fd27cb89640 1 Processor -- start 2024-06-04T05:11:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.626+0000 7fd27cb89640 1 -- start start 2024-06-04T05:11:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd27cb89640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd278103d20 0x7fd27810bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd27cb89640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd278104640 0x7fd27810c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd27cb89640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd27810d060 0x7fd2781a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd27cb89640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd278112b30 con 0x7fd27810d060 2024-06-04T05:11:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd27cb89640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd2781129b0 con 0x7fd278104640 2024-06-04T05:11:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd276d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd27810d060 0x7fd2781a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd27cb89640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd278112cb0 con 0x7fd278103d20 2024-06-04T05:11:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd276d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd27810d060 0x7fd2781a3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57734/0 (socket says 172.21.15.70:57734) 2024-06-04T05:11:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd276d9d640 1 -- 172.21.15.70:0/2250532573 learned_addr learned my addr 172.21.15.70:0/2250532573 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd275d9b640 1 --2- 172.21.15.70:0/2250532573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd278104640 0x7fd27810c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd27659c640 1 --2- 172.21.15.70:0/2250532573 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd278103d20 0x7fd27810bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd275d9b640 1 -- 172.21.15.70:0/2250532573 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd278103d20 msgr2=0x7fd27810bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd275d9b640 1 --2- 172.21.15.70:0/2250532573 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd278103d20 0x7fd27810bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd275d9b640 1 -- 172.21.15.70:0/2250532573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd27810d060 msgr2=0x7fd2781a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd275d9b640 1 --2- 172.21.15.70:0/2250532573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd27810d060 0x7fd2781a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd275d9b640 1 -- 172.21.15.70:0/2250532573 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2781a4720 con 0x7fd278104640 2024-06-04T05:11:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd276d9d640 1 --2- 172.21.15.70:0/2250532573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd27810d060 0x7fd2781a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.627+0000 7fd275d9b640 1 --2- 172.21.15.70:0/2250532573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd278104640 0x7fd27810c4c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00e9f0 tx=0x7fd26c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:15.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.628+0000 7fd2677fe640 1 -- 172.21.15.70:0/2250532573 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c00cd80 con 0x7fd278104640 2024-06-04T05:11:15.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.628+0000 7fd27cb89640 1 -- 172.21.15.70:0/2250532573 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2781a4a10 con 0x7fd278104640 2024-06-04T05:11:15.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.628+0000 7fd2677fe640 1 -- 172.21.15.70:0/2250532573 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd26c0035f0 con 0x7fd278104640 2024-06-04T05:11:15.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.628+0000 7fd2677fe640 1 -- 172.21.15.70:0/2250532573 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c010670 con 0x7fd278104640 2024-06-04T05:11:15.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.628+0000 7fd27cb89640 1 -- 172.21.15.70:0/2250532573 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2781ac2a0 con 0x7fd278104640 2024-06-04T05:11:15.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.630+0000 7fd2677fe640 1 -- 172.21.15.70:0/2250532573 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fd26c004000 con 0x7fd278104640 2024-06-04T05:11:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.631+0000 7fd2677fe640 1 --2- 172.21.15.70:0/2250532573 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2600b2a90 0x7fd2600b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.631+0000 7fd27f980740 1 Processor -- start 2024-06-04T05:11:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.631+0000 7fd27659c640 1 --2- 172.21.15.70:0/2250532573 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2600b2a90 0x7fd2600b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.631+0000 7fd2677fe640 1 -- 172.21.15.70:0/2250532573 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fd26c0e2950 con 0x7fd278104640 2024-06-04T05:11:15.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.632+0000 7fd27f980740 1 -- start start 2024-06-04T05:11:15.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.632+0000 7fd27f980740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 0x5608d4806c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.632+0000 7fd27f980740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4807140 0x5608d48114c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.632+0000 7fd27f980740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4811a00 0x5608d4813df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.632+0000 7fd27f980740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5608d4778c30 con 0x5608d4806820 2024-06-04T05:11:15.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.632+0000 7fd27f980740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5608d47687d0 con 0x5608d4811a00 2024-06-04T05:11:15.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.632+0000 7fd27659c640 1 --2- 172.21.15.70:0/2250532573 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2600b2a90 0x7fd2600b4f50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd268000c00 tx=0x7fd268005a20 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:15.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.632+0000 7fd27f980740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5608d476c810 con 0x5608d4807140 2024-06-04T05:11:15.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.632+0000 7fd276d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4811a00 0x5608d4813df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.632+0000 7fd27659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 0x5608d4806c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.632+0000 7fd275d9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4807140 0x5608d48114c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.632+0000 7fd276d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4811a00 0x5608d4813df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46420/0 (socket says 172.21.15.70:46420) 2024-06-04T05:11:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.632+0000 7fd276d9d640 1 -- 172.21.15.70:0/462380308 learned_addr learned my addr 172.21.15.70:0/462380308 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.633+0000 7fd276d9d640 1 -- 172.21.15.70:0/462380308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4807140 msgr2=0x5608d48114c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.633+0000 7fd276d9d640 1 --2- 172.21.15.70:0/462380308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4807140 0x5608d48114c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.633+0000 7fd276d9d640 1 -- 172.21.15.70:0/462380308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 msgr2=0x5608d4806c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.633+0000 7fd276d9d640 1 --2- 172.21.15.70:0/462380308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 0x5608d4806c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.633+0000 7fd276d9d640 1 -- 172.21.15.70:0/462380308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608d447d210 con 0x5608d4811a00 2024-06-04T05:11:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.633+0000 7fd276d9d640 1 --2- 172.21.15.70:0/462380308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4811a00 0x5608d4813df0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd270011850 tx=0x7fd270011d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a294f27eb7278806 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.633+0000 7fd264ff9640 1 -- 172.21.15.70:0/462380308 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd270011330 con 0x5608d4811a00 2024-06-04T05:11:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.633+0000 7fd264ff9640 1 -- 172.21.15.70:0/462380308 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd27000fe30 con 0x5608d4811a00 2024-06-04T05:11:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.633+0000 7fd264ff9640 1 -- 172.21.15.70:0/462380308 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2700193e0 con 0x5608d4811a00 2024-06-04T05:11:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.634+0000 7fd27f980740 1 -- 172.21.15.70:0/462380308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4811a00 msgr2=0x5608d4813df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.634+0000 7fd27f980740 1 --2- 172.21.15.70:0/462380308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4811a00 0x5608d4813df0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd270011850 tx=0x7fd270011d10 comp rx=0 tx=0).stop 2024-06-04T05:11:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.634+0000 7fd27f980740 1 -- 172.21.15.70:0/462380308 shutdown_connections 2024-06-04T05:11:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.634+0000 7fd27f980740 1 --2- 172.21.15.70:0/462380308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4811a00 0x5608d4813df0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.634+0000 7fd27f980740 1 --2- 172.21.15.70:0/462380308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4807140 0x5608d48114c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.634+0000 7fd27f980740 1 --2- 172.21.15.70:0/462380308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 0x5608d4806c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.634+0000 7fd27f980740 1 -- 172.21.15.70:0/462380308 >> 172.21.15.70:0/462380308 conn(0x5608d4802d30 msgr2=0x5608d4803110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.634+0000 7fd27f980740 1 -- 172.21.15.70:0/462380308 shutdown_connections 2024-06-04T05:11:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.634+0000 7fd27f980740 1 -- 172.21.15.70:0/462380308 wait complete. 2024-06-04T05:11:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.634+0000 7fd27f980740 1 -- start start 2024-06-04T05:11:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.634+0000 7fd27f980740 1 Processor -- start 2024-06-04T05:11:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd27f980740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4806820 0x5608d480e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd27f980740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4807140 0x5608d480eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd27f980740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4811a00 0x5608d480f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd27659c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4806820 0x5608d480e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd27659c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4806820 0x5608d480e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:50426/0 (socket says 172.21.15.70:50426) 2024-06-04T05:11:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd276d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4811a00 0x5608d480f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd275d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4807140 0x5608d480eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd275d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4807140 0x5608d480eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46436/0 (socket says 172.21.15.70:46436) 2024-06-04T05:11:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd276d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4811a00 0x5608d480f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57742/0 (socket says 172.21.15.70:57742) 2024-06-04T05:11:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd27659c640 1 -- 172.21.15.70:0/4011943133 learned_addr learned my addr 172.21.15.70:0/4011943133 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd27f980740 1 -- 172.21.15.70:0/4011943133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5608d444e790 con 0x5608d4811a00 2024-06-04T05:11:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd27f980740 1 -- 172.21.15.70:0/4011943133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5608d4724890 con 0x5608d4807140 2024-06-04T05:11:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd27f980740 1 -- 172.21.15.70:0/4011943133 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5608d4814630 con 0x5608d4806820 2024-06-04T05:11:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd275d9b640 1 -- 172.21.15.70:0/4011943133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4806820 msgr2=0x5608d480e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd275d9b640 1 --2- 172.21.15.70:0/4011943133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4806820 0x5608d480e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd275d9b640 1 -- 172.21.15.70:0/4011943133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4811a00 msgr2=0x5608d480f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd275d9b640 1 --2- 172.21.15.70:0/4011943133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4811a00 0x5608d480f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd275d9b640 1 -- 172.21.15.70:0/4011943133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608d4549620 con 0x5608d4807140 2024-06-04T05:11:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd27659c640 1 --2- 172.21.15.70:0/4011943133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4806820 0x5608d480e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.635+0000 7fd276d9d640 1 --2- 172.21.15.70:0/4011943133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4811a00 0x5608d480f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.636+0000 7fd27f980740 10 client.0 _ll_register_callbacks cb 0x5608d447ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.636+0000 7fd27f980740 1 -- 172.21.15.70:0/4011943133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd27000fe30 con 0x5608d4807140 2024-06-04T05:11:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.636+0000 7fd275d9b640 1 --2- 172.21.15.70:0/4011943133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4807140 0x5608d480eeb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd26c0e0040 tx=0x7fd26c003cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15eb813659329564 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.636+0000 7fd25dffb640 10 client.28115 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:11:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.636+0000 7fd25dffb640 1 -- 172.21.15.70:0/4011943133 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c0aa300 con 0x5608d4807140 2024-06-04T05:11:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.636+0000 7fd25dffb640 1 -- 172.21.15.70:0/4011943133 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fd26c0c4280 con 0x5608d4807140 2024-06-04T05:11:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.636+0000 7fd25dffb640 1 -- 172.21.15.70:0/4011943133 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd26c0a9c00 con 0x5608d4807140 2024-06-04T05:11:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.636+0000 7fd27f980740 10 client.28115 fetch_fsmap learned FSMap version 39 2024-06-04T05:11:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.636+0000 7fd27f980740 1 -- 172.21.15.70:0/4011943133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({fsmap=39}) -- 0x5608d481ff50 con 0x5608d4807140 2024-06-04T05:11:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.636+0000 7fd25dffb640 1 -- 172.21.15.70:0/4011943133 <== mon.2 v2:172.21.15.98:3300/0 4 ==== fsmap(e 39) ==== 8374+0+0 (secure 0 0 0) 0x7fd26c0e02f0 con 0x5608d4807140 2024-06-04T05:11:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.636+0000 7fd27f980740 10 client.28115 fetch_fsmap finished waiting for FSMap version 39 2024-06-04T05:11:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.636+0000 7fd27f980740 10 client.28115 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.636+0000 7fd27f980740 20 client.28115 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.636+0000 7fd27f980740 1 --2- 172.21.15.70:0/4011943133 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5608d4821400 0x5608d48238c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.636+0000 7fd27f980740 4 client.28115 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:11:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.636+0000 7fd27f980740 1 -- 172.21.15.70:0/4011943133 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608d477ab80 con 0x5608d4821400 2024-06-04T05:11:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.637+0000 7fd27659c640 1 --2- 172.21.15.70:0/4011943133 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5608d4821400 0x5608d48238c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.637+0000 7fd25dffb640 10 client.28115 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.637+0000 7fd27659c640 1 --2- 172.21.15.70:0/4011943133 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5608d4821400 0x5608d48238c0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccf36f125037f591 server_cookie=db0366668c657e45 in_seq=0 out_seq=0 2024-06-04T05:11:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.638+0000 7fd25dffb640 1 -- 172.21.15.70:0/4011943133 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5608d477ab80 con 0x5608d4821400 2024-06-04T05:11:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.638+0000 7fd25dffb640 10 client.28115 handle_command_reply: tid=0 2024-06-04T05:11:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.638+0000 7fd27f980740 1 client.28115 shutdown 2024-06-04T05:11:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.638+0000 7fd27f980740 1 -- 172.21.15.70:0/4011943133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4807140 msgr2=0x5608d480eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.638+0000 7fd27f980740 1 --2- 172.21.15.70:0/4011943133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4807140 0x5608d480eeb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd26c0e0040 tx=0x7fd26c003cf0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.638+0000 7fd27f980740 1 -- 172.21.15.70:0/4011943133 shutdown_connections 2024-06-04T05:11:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.638+0000 7fd27f980740 1 --2- 172.21.15.70:0/4011943133 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5608d4821400 0x5608d48238c0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.638+0000 7fd27f980740 1 --2- 172.21.15.70:0/4011943133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4807140 0x5608d480eeb0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.638+0000 7fd25dffb640 0 client.28115 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.638+0000 7fd27f980740 1 --2- 172.21.15.70:0/4011943133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4811a00 0x5608d480f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.638+0000 7fd27f980740 1 --2- 172.21.15.70:0/4011943133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4806820 0x5608d480e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.638+0000 7fd27f980740 1 -- 172.21.15.70:0/4011943133 >> 172.21.15.70:0/4011943133 conn(0x5608d4802d30 msgr2=0x5608d4808ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.639+0000 7fd27f980740 1 -- 172.21.15.70:0/4011943133 shutdown_connections 2024-06-04T05:11:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.639+0000 7fd27f980740 1 -- 172.21.15.70:0/4011943133 wait complete. 2024-06-04T05:11:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.639+0000 7fd27f980740 20 client.28115 trim_cache size 0 max 16384 2024-06-04T05:11:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.656+0000 7fd27f980740 1 Processor -- start 2024-06-04T05:11:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.656+0000 7fd27f980740 1 -- start start 2024-06-04T05:11:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.656+0000 7fd27f980740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 0x5608d48091e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.656+0000 7fd27f980740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4807140 0x5608d4801e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.656+0000 7fd27f980740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4811a00 0x5608d4802370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.656+0000 7fd27f980740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd2781b4ce0 con 0x5608d4806820 2024-06-04T05:11:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.656+0000 7fd27f980740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd2781b6910 con 0x5608d4811a00 2024-06-04T05:11:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.656+0000 7fd27f980740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd2781b53b0 con 0x5608d4807140 2024-06-04T05:11:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.656+0000 7fd27659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 0x5608d48091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.656+0000 7fd276d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4811a00 0x5608d4802370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.656+0000 7fd27659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 0x5608d48091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57750/0 (socket says 172.21.15.70:57750) 2024-06-04T05:11:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.656+0000 7fd27659c640 1 -- 172.21.15.70:0/798434593 learned_addr learned my addr 172.21.15.70:0/798434593 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.656+0000 7fd275d9b640 1 --2- 172.21.15.70:0/798434593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4807140 0x5608d4801e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.657+0000 7fd276d9d640 1 -- 172.21.15.70:0/798434593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4807140 msgr2=0x5608d4801e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.657+0000 7fd276d9d640 1 --2- 172.21.15.70:0/798434593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4807140 0x5608d4801e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.657+0000 7fd276d9d640 1 -- 172.21.15.70:0/798434593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 msgr2=0x5608d48091e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.657+0000 7fd276d9d640 1 --2- 172.21.15.70:0/798434593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 0x5608d48091e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.657+0000 7fd276d9d640 1 -- 172.21.15.70:0/798434593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608d47fe6f0 con 0x5608d4811a00 2024-06-04T05:11:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.657+0000 7fd275d9b640 1 --2- 172.21.15.70:0/798434593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4807140 0x5608d4801e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.657+0000 7fd27659c640 1 --2- 172.21.15.70:0/798434593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 0x5608d48091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.657+0000 7fd276d9d640 1 --2- 172.21.15.70:0/798434593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4811a00 0x5608d4802370 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd270011820 tx=0x7fd2700115e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=487e31bb87eea3a8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.657+0000 7fd25ffff640 1 -- 172.21.15.70:0/798434593 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27001f040 con 0x5608d4811a00 2024-06-04T05:11:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.657+0000 7fd25ffff640 1 -- 172.21.15.70:0/798434593 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd27002a070 con 0x5608d4811a00 2024-06-04T05:11:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.657+0000 7fd25ffff640 1 -- 172.21.15.70:0/798434593 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd270005550 con 0x5608d4811a00 2024-06-04T05:11:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.658+0000 7fd27f980740 1 -- 172.21.15.70:0/798434593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4811a00 msgr2=0x5608d4802370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.658+0000 7fd27f980740 1 --2- 172.21.15.70:0/798434593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4811a00 0x5608d4802370 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd270011820 tx=0x7fd2700115e0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.658+0000 7fd27f980740 1 -- 172.21.15.70:0/798434593 shutdown_connections 2024-06-04T05:11:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.658+0000 7fd27f980740 1 --2- 172.21.15.70:0/798434593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4811a00 0x5608d4802370 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.658+0000 7fd27f980740 1 --2- 172.21.15.70:0/798434593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4807140 0x5608d4801e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.658+0000 7fd27f980740 1 --2- 172.21.15.70:0/798434593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 0x5608d48091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.658+0000 7fd27f980740 1 -- 172.21.15.70:0/798434593 >> 172.21.15.70:0/798434593 conn(0x5608d4802d30 msgr2=0x5608d4801720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.658+0000 7fd27f980740 1 -- 172.21.15.70:0/798434593 shutdown_connections 2024-06-04T05:11:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.658+0000 7fd27f980740 1 -- 172.21.15.70:0/798434593 wait complete. 2024-06-04T05:11:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.658+0000 7fd27f980740 1 -- start start 2024-06-04T05:11:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd27f980740 1 Processor -- start 2024-06-04T05:11:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd27f980740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 0x5608d4815590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd27f980740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4807140 0x5608d4815ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd27f980740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4811a00 0x5608d4816010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd27f980740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd2781b61e0 con 0x5608d4806820 2024-06-04T05:11:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd27f980740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd2781b5ab0 con 0x5608d4807140 2024-06-04T05:11:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd27f980740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd2781b2ce0 con 0x5608d4811a00 2024-06-04T05:11:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd27f980740 10 client.0 _ll_register_callbacks cb 0x7fd26c0042f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd27659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 0x5608d4815590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd275d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4807140 0x5608d4815ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd275d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4807140 0x5608d4815ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:46454/0 (socket says 172.21.15.70:46454) 2024-06-04T05:11:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd275d9b640 1 -- 172.21.15.70:0/2876279478 learned_addr learned my addr 172.21.15.70:0/2876279478 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd27659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 0x5608d4815590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57764/0 (socket says 172.21.15.70:57764) 2024-06-04T05:11:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd276d9d640 1 --2- 172.21.15.70:0/2876279478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4811a00 0x5608d4816010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd275d9b640 1 -- 172.21.15.70:0/2876279478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4811a00 msgr2=0x5608d4816010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd275d9b640 1 --2- 172.21.15.70:0/2876279478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4811a00 0x5608d4816010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd275d9b640 1 -- 172.21.15.70:0/2876279478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 msgr2=0x5608d4815590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd275d9b640 1 --2- 172.21.15.70:0/2876279478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 0x5608d4815590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.659+0000 7fd275d9b640 1 -- 172.21.15.70:0/2876279478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608d488b050 con 0x5608d4807140 2024-06-04T05:11:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.660+0000 7fd275d9b640 1 --2- 172.21.15.70:0/2876279478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4807140 0x5608d4815ad0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd26c0a9dc0 tx=0x7fd26c003a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb09e57c014e8d12 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.660+0000 7fd25dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:11:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.660+0000 7fd25dffb640 1 -- 172.21.15.70:0/2876279478 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c0aa300 con 0x5608d4807140 2024-06-04T05:11:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.660+0000 7fd27f980740 1 -- 172.21.15.70:0/2876279478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd27002a070 con 0x5608d4807140 2024-06-04T05:11:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.660+0000 7fd25dffb640 1 -- 172.21.15.70:0/2876279478 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fd26c0a5e90 con 0x5608d4807140 2024-06-04T05:11:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.660+0000 7fd25dffb640 1 -- 172.21.15.70:0/2876279478 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd26c0a8e60 con 0x5608d4807140 2024-06-04T05:11:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.660+0000 7fd27f980740 10 client.28121 fetch_fsmap learned FSMap version 39 2024-06-04T05:11:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.660+0000 7fd27f980740 1 -- 172.21.15.70:0/2876279478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5608d488c2c0 con 0x5608d4807140 2024-06-04T05:11:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.660+0000 7fd25dffb640 1 -- 172.21.15.70:0/2876279478 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd27002a070 con 0x5608d4807140 2024-06-04T05:11:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.660+0000 7fd25dffb640 1 -- 172.21.15.70:0/2876279478 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 39) ==== 8374+0+0 (secure 0 0 0) 0x7fd26c0a88d0 con 0x5608d4807140 2024-06-04T05:11:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.660+0000 7fd27f980740 10 client.28121 fetch_fsmap finished waiting for FSMap version 39 2024-06-04T05:11:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.660+0000 7fd27f980740 10 client.28121 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.660+0000 7fd27f980740 20 client.28121 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.660+0000 7fd27f980740 1 --2- 172.21.15.70:0/2876279478 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5608d4821400 0x5608d488e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.660+0000 7fd27f980740 4 client.28121 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:11:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.660+0000 7fd27f980740 1 -- 172.21.15.70:0/2876279478 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5608d4774360 con 0x5608d4821400 2024-06-04T05:11:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.661+0000 7fd27659c640 1 --2- 172.21.15.70:0/2876279478 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5608d4821400 0x5608d488e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.661+0000 7fd27659c640 1 --2- 172.21.15.70:0/2876279478 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5608d4821400 0x5608d488e870 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=149e48a674ee8a6e server_cookie=9b075e889583d92b in_seq=0 out_seq=0 2024-06-04T05:11:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:15.661+0000 7fd25dffb640 10 client.28121 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:15 smithi112 ceph-mon[36453]: pgmap v439: 97 pgs: 97 active+clean; 319 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 20 MiB/s wr, 221 op/s 2024-06-04T05:11:15.796 INFO:tasks.workunit.client.0.smithi070.stdout: 10 1411 36.60 MB/sec warmup 7 sec latency 135.556 ms 2024-06-04T05:11:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.042+0000 7fd25dffb640 1 -- 172.21.15.70:0/2876279478 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5608d4774360 con 0x5608d4821400 2024-06-04T05:11:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.042+0000 7fd25dffb640 10 client.28121 handle_command_reply: tid=0 2024-06-04T05:11:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.042+0000 7fd27f980740 1 client.28121 shutdown 2024-06-04T05:11:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.043+0000 7fd27f980740 1 -- 172.21.15.70:0/2876279478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4807140 msgr2=0x5608d4815ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.043+0000 7fd27f980740 1 --2- 172.21.15.70:0/2876279478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4807140 0x5608d4815ad0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd26c0a9dc0 tx=0x7fd26c003a30 comp rx=0 tx=0).stop 2024-06-04T05:11:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.043+0000 7fd27f980740 1 -- 172.21.15.70:0/2876279478 shutdown_connections 2024-06-04T05:11:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.043+0000 7fd27f980740 1 --2- 172.21.15.70:0/2876279478 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5608d4821400 0x5608d488e870 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.043+0000 7fd27f980740 1 --2- 172.21.15.70:0/2876279478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608d4807140 0x5608d4815ad0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.043+0000 7fd27f980740 1 --2- 172.21.15.70:0/2876279478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5608d4811a00 0x5608d4816010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.043+0000 7fd27f980740 1 --2- 172.21.15.70:0/2876279478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608d4806820 0x5608d4815590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.043+0000 7fd27f980740 1 -- 172.21.15.70:0/2876279478 >> 172.21.15.70:0/2876279478 conn(0x5608d4802d30 msgr2=0x5608d4801fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.043+0000 7fd25dffb640 0 client.28121 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.043+0000 7fd27f980740 1 -- 172.21.15.70:0/2876279478 shutdown_connections 2024-06-04T05:11:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.043+0000 7fd27f980740 1 -- 172.21.15.70:0/2876279478 wait complete. 2024-06-04T05:11:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.044+0000 7fd27f980740 20 client.28121 trim_cache size 0 max 16384 2024-06-04T05:11:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:11:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:11:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:11:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.049+0000 7fd27cb89640 1 -- 172.21.15.70:0/2250532573 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2600b2a90 msgr2=0x7fd2600b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.049+0000 7fd27cb89640 1 --2- 172.21.15.70:0/2250532573 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2600b2a90 0x7fd2600b4f50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd268000c00 tx=0x7fd268005a20 comp rx=0 tx=0).stop 2024-06-04T05:11:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.049+0000 7fd27cb89640 1 -- 172.21.15.70:0/2250532573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd278104640 msgr2=0x7fd27810c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.049+0000 7fd27cb89640 1 --2- 172.21.15.70:0/2250532573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd278104640 0x7fd27810c4c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00e9f0 tx=0x7fd26c00eeb0 comp rx=0 tx=0).stop 2024-06-04T05:11:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.049+0000 7fd27cb89640 1 -- 172.21.15.70:0/2250532573 shutdown_connections 2024-06-04T05:11:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.049+0000 7fd27cb89640 1 --2- 172.21.15.70:0/2250532573 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2600b2a90 0x7fd2600b4f50 secure :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd268000c00 tx=0x7fd268005a20 comp rx=0 tx=0).stop 2024-06-04T05:11:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.049+0000 7fd27cb89640 1 --2- 172.21.15.70:0/2250532573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd278104640 0x7fd27810c4c0 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00e9f0 tx=0x7fd26c00eeb0 comp rx=0 tx=0).stop 2024-06-04T05:11:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.049+0000 7fd27cb89640 1 --2- 172.21.15.70:0/2250532573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd27810d060 0x7fd2781a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.049+0000 7fd27cb89640 1 --2- 172.21.15.70:0/2250532573 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd278103d20 0x7fd27810bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.049+0000 7fd27cb89640 1 -- 172.21.15.70:0/2250532573 >> 172.21.15.70:0/2250532573 conn(0x7fd278078ad0 msgr2=0x7fd278102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:16.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.052+0000 7fd27cb89640 1 -- 172.21.15.70:0/2250532573 shutdown_connections 2024-06-04T05:11:16.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:16.052+0000 7fd27cb89640 1 -- 172.21.15.70:0/2250532573 wait complete. 2024-06-04T05:11:16.796 INFO:tasks.workunit.client.0.smithi070.stdout: 10 1581 32.30 MB/sec warmup 8 sec latency 191.937 ms 2024-06-04T05:11:17.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:17 smithi098 ceph-mon[35569]: pgmap v440: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 23 MiB/s wr, 360 op/s 2024-06-04T05:11:17.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:17 smithi070 ceph-mon[32631]: pgmap v440: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 23 MiB/s wr, 360 op/s 2024-06-04T05:11:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:17 smithi112 ceph-mon[36453]: pgmap v440: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 23 MiB/s wr, 360 op/s 2024-06-04T05:11:17.796 INFO:tasks.workunit.client.0.smithi070.stdout: 10 1753 28.90 MB/sec warmup 9 sec latency 108.288 ms 2024-06-04T05:11:18.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:18 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:11:18.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:18 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:11:18.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:18 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:11:18.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:18 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:11:18.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:18 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:11:18.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:18 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:11:18.796 INFO:tasks.workunit.client.0.smithi070.stdout: 10 1950 26.21 MB/sec warmup 10 sec latency 96.714 ms 2024-06-04T05:11:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:19 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:11:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:19 smithi098 ceph-mon[35569]: pgmap v441: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 22 MiB/s wr, 350 op/s 2024-06-04T05:11:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:19 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:19.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:19 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:11:19.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:19 smithi070 ceph-mon[32631]: pgmap v441: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 22 MiB/s wr, 350 op/s 2024-06-04T05:11:19.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:19 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:19 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:11:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:19 smithi112 ceph-mon[36453]: pgmap v441: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 22 MiB/s wr, 350 op/s 2024-06-04T05:11:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:19 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:19.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 2157 24.75 MB/sec warmup 11 sec latency 53.607 ms 2024-06-04T05:11:20.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 2333 23.21 MB/sec warmup 12 sec latency 56.812 ms 2024-06-04T05:11:21.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:21 smithi098 ceph-mon[35569]: pgmap v442: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 19 MiB/s wr, 300 op/s 2024-06-04T05:11:21.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:21 smithi070 ceph-mon[32631]: pgmap v442: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 19 MiB/s wr, 300 op/s 2024-06-04T05:11:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:21 smithi112 ceph-mon[36453]: pgmap v442: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 19 MiB/s wr, 300 op/s 2024-06-04T05:11:21.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 2581 23.19 MB/sec warmup 13 sec latency 61.863 ms 2024-06-04T05:11:22.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 2859 22.57 MB/sec warmup 14 sec latency 64.247 ms 2024-06-04T05:11:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:23 smithi098 ceph-mon[35569]: pgmap v443: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 20 MiB/s wr, 407 op/s 2024-06-04T05:11:23.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:23 smithi070 ceph-mon[32631]: pgmap v443: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 20 MiB/s wr, 407 op/s 2024-06-04T05:11:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:23 smithi112 ceph-mon[36453]: pgmap v443: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 20 MiB/s wr, 407 op/s 2024-06-04T05:11:23.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 3074 22.51 MB/sec warmup 15 sec latency 58.835 ms 2024-06-04T05:11:24.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:24 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:24.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:24 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:24.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:24 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:24.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 3289 22.05 MB/sec warmup 16 sec latency 60.757 ms 2024-06-04T05:11:25.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:25 smithi098 ceph-mon[35569]: pgmap v444: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 4.1 MiB/s wr, 296 op/s 2024-06-04T05:11:25.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:25 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:25.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:25 smithi070 ceph-mon[32631]: pgmap v444: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 4.1 MiB/s wr, 296 op/s 2024-06-04T05:11:25.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:25 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:25 smithi112 ceph-mon[36453]: pgmap v444: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 4.1 MiB/s wr, 296 op/s 2024-06-04T05:11:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:25 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:25.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 3456 21.54 MB/sec warmup 17 sec latency 156.722 ms 2024-06-04T05:11:26.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 3656 21.47 MB/sec warmup 18 sec latency 63.354 ms 2024-06-04T05:11:27.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:27 smithi098 ceph-mon[35569]: pgmap v445: 97 pgs: 97 active+clean; 360 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 7.1 MiB/s wr, 399 op/s 2024-06-04T05:11:27.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:27 smithi070 ceph-mon[32631]: pgmap v445: 97 pgs: 97 active+clean; 360 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 7.1 MiB/s wr, 399 op/s 2024-06-04T05:11:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:27 smithi112 ceph-mon[36453]: pgmap v445: 97 pgs: 97 active+clean; 360 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 7.1 MiB/s wr, 399 op/s 2024-06-04T05:11:27.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 3815 20.61 MB/sec warmup 19 sec latency 838.092 ms 2024-06-04T05:11:28.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 3976 19.79 MB/sec warmup 20 sec latency 870.615 ms 2024-06-04T05:11:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.181+0000 7f4e49e12640 1 -- 172.21.15.70:0/2089767829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e44103ce0 msgr2=0x7f4e441040c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.181+0000 7f4e49e12640 1 --2- 172.21.15.70:0/2089767829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e44103ce0 0x7f4e441040c0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009900 tx=0x7f4e3402a510 comp rx=0 tx=0).stop 2024-06-04T05:11:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.181+0000 7f4e49e12640 1 -- 172.21.15.70:0/2089767829 shutdown_connections 2024-06-04T05:11:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.181+0000 7f4e49e12640 1 --2- 172.21.15.70:0/2089767829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e44104600 0x7f4e4410cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.181+0000 7f4e49e12640 1 --2- 172.21.15.70:0/2089767829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e4410d020 0x7f4e4410f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.181+0000 7f4e49e12640 1 --2- 172.21.15.70:0/2089767829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e44103ce0 0x7f4e441040c0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.181+0000 7f4e49e12640 1 -- 172.21.15.70:0/2089767829 >> 172.21.15.70:0/2089767829 conn(0x7f4e44078ad0 msgr2=0x7f4e44078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:29.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.181+0000 7f4e49e12640 1 -- 172.21.15.70:0/2089767829 shutdown_connections 2024-06-04T05:11:29.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.181+0000 7f4e49e12640 1 -- 172.21.15.70:0/2089767829 wait complete. 2024-06-04T05:11:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.181+0000 7f4e49e12640 1 Processor -- start 2024-06-04T05:11:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.182+0000 7f4e49e12640 1 -- start start 2024-06-04T05:11:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.182+0000 7f4e49e12640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e44103ce0 0x7f4e44101930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.182+0000 7f4e49e12640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e44104600 0x7f4e44101e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.182+0000 7f4e49e12640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e4410d020 0x7f4e441023b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.182+0000 7f4e49e12640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e44112b80 con 0x7f4e4410d020 2024-06-04T05:11:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.182+0000 7f4e49e12640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e44112a00 con 0x7f4e44104600 2024-06-04T05:11:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.182+0000 7f4e49e12640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4e44112d00 con 0x7f4e44103ce0 2024-06-04T05:11:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.182+0000 7f4e43fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e4410d020 0x7f4e441023b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.182+0000 7f4e437fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e44103ce0 0x7f4e44101930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.182+0000 7f4e43fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e4410d020 0x7f4e441023b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51412/0 (socket says 172.21.15.70:51412) 2024-06-04T05:11:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.182+0000 7f4e437fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e44103ce0 0x7f4e44101930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:37390/0 (socket says 172.21.15.70:37390) 2024-06-04T05:11:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.182+0000 7f4e43fff640 1 -- 172.21.15.70:0/881070174 learned_addr learned my addr 172.21.15.70:0/881070174 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.182+0000 7f4e42ffd640 1 --2- 172.21.15.70:0/881070174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e44104600 0x7f4e44101e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.182+0000 7f4e437fe640 1 -- 172.21.15.70:0/881070174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e44104600 msgr2=0x7f4e44101e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.182+0000 7f4e437fe640 1 --2- 172.21.15.70:0/881070174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e44104600 0x7f4e44101e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.182+0000 7f4e437fe640 1 -- 172.21.15.70:0/881070174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e4410d020 msgr2=0x7f4e441023b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.182+0000 7f4e437fe640 1 --2- 172.21.15.70:0/881070174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e4410d020 0x7f4e441023b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.183+0000 7f4e437fe640 1 -- 172.21.15.70:0/881070174 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e441a75c0 con 0x7f4e44103ce0 2024-06-04T05:11:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.183+0000 7f4e437fe640 1 --2- 172.21.15.70:0/881070174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e44103ce0 0x7f4e44101930 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4e34005940 tx=0x7f4e34004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.183+0000 7f4e40ff9640 1 -- 172.21.15.70:0/881070174 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3403f070 con 0x7f4e44103ce0 2024-06-04T05:11:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.183+0000 7f4e40ff9640 1 -- 172.21.15.70:0/881070174 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e34003a80 con 0x7f4e44103ce0 2024-06-04T05:11:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.183+0000 7f4e49e12640 1 -- 172.21.15.70:0/881070174 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e441a7850 con 0x7f4e44103ce0 2024-06-04T05:11:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.183+0000 7f4e49e12640 1 -- 172.21.15.70:0/881070174 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e441a7d60 con 0x7f4e44103ce0 2024-06-04T05:11:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.183+0000 7f4e40ff9640 1 -- 172.21.15.70:0/881070174 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e34035070 con 0x7f4e44103ce0 2024-06-04T05:11:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.186+0000 7f4e40ff9640 1 -- 172.21.15.70:0/881070174 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f4e34003c30 con 0x7f4e44103ce0 2024-06-04T05:11:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.187+0000 7f4e4cc05740 1 Processor -- start 2024-06-04T05:11:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.187+0000 7f4e4cc05740 1 -- start start 2024-06-04T05:11:29.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.187+0000 7f4e4cc05740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba30c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.187+0000 7f4e4cc05740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba3b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.187+0000 7f4e4cc05740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 0x55ebbba3ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.187+0000 7f4e4cc05740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ebbb9a2c30 con 0x55ebbba30820 2024-06-04T05:11:29.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.187+0000 7f4e4cc05740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ebbb9927d0 con 0x55ebbba31140 2024-06-04T05:11:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.187+0000 7f4e4cc05740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ebbb996810 con 0x55ebbba3ba00 2024-06-04T05:11:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.187+0000 7f4e43fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba30c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.187+0000 7f4e43fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba30c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51424/0 (socket says 172.21.15.70:51424) 2024-06-04T05:11:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.187+0000 7f4e43fff640 1 -- 172.21.15.70:0/1950864969 learned_addr learned my addr 172.21.15.70:0/1950864969 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.187+0000 7f4e437fe640 1 --2- 172.21.15.70:0/1950864969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba3b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.187+0000 7f4e40ff9640 1 --2- 172.21.15.70:0/881070174 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e2c0b2ac0 0x7f4e2c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.187+0000 7f4e42ffd640 1 --2- 172.21.15.70:0/1950864969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 0x55ebbba3ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.188+0000 7f4e43fff640 1 -- 172.21.15.70:0/1950864969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 msgr2=0x55ebbba3ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.188+0000 7f4e43fff640 1 --2- 172.21.15.70:0/1950864969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 0x55ebbba3ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.188+0000 7f4e43fff640 1 -- 172.21.15.70:0/1950864969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 msgr2=0x55ebbba3b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.188+0000 7f4e40ff9640 1 -- 172.21.15.70:0/881070174 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f4e341026b0 con 0x7f4e44103ce0 2024-06-04T05:11:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.188+0000 7f4e43fff640 1 --2- 172.21.15.70:0/1950864969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba3b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.188+0000 7f4e43fff640 1 -- 172.21.15.70:0/1950864969 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebbb6a7210 con 0x55ebbba30820 2024-06-04T05:11:29.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.188+0000 7f4e43fff640 1 --2- 172.21.15.70:0/881070174 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e2c0b2ac0 0x7f4e2c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.188+0000 7f4e43fff640 1 --2- 172.21.15.70:0/1950864969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba30c00 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c011570 tx=0x7f4e3c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe27314f85a5b9c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:29.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.188+0000 7f4e29ffb640 1 -- 172.21.15.70:0/1950864969 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c017050 con 0x55ebbba30820 2024-06-04T05:11:29.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.188+0000 7f4e43fff640 1 --2- 172.21.15.70:0/881070174 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e2c0b2ac0 0x7f4e2c0b4f80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c023000 tx=0x7f4e3c024000 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:29.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.188+0000 7f4e29ffb640 1 -- 172.21.15.70:0/1950864969 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e3c01bd60 con 0x55ebbba30820 2024-06-04T05:11:29.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.188+0000 7f4e29ffb640 1 -- 172.21.15.70:0/1950864969 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c00ea00 con 0x55ebbba30820 2024-06-04T05:11:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.189+0000 7f4e4cc05740 1 -- 172.21.15.70:0/1950864969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 msgr2=0x55ebbba30c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.189+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/1950864969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba30c00 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c011570 tx=0x7f4e3c011a30 comp rx=0 tx=0).stop 2024-06-04T05:11:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.189+0000 7f4e4cc05740 1 -- 172.21.15.70:0/1950864969 shutdown_connections 2024-06-04T05:11:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.189+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/1950864969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba3b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.189+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/1950864969 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 0x55ebbba3ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.189+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/1950864969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba30c00 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.189+0000 7f4e4cc05740 1 -- 172.21.15.70:0/1950864969 >> 172.21.15.70:0/1950864969 conn(0x55ebbba2cd30 msgr2=0x55ebbba2d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.189+0000 7f4e4cc05740 1 -- 172.21.15.70:0/1950864969 shutdown_connections 2024-06-04T05:11:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.189+0000 7f4e4cc05740 1 -- 172.21.15.70:0/1950864969 wait complete. 2024-06-04T05:11:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.190+0000 7f4e4cc05740 1 -- start start 2024-06-04T05:11:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.190+0000 7f4e4cc05740 1 Processor -- start 2024-06-04T05:11:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.190+0000 7f4e4cc05740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba38930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e4cc05740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba38e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e4cc05740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 0x55ebbba393b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e4cc05740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ebbb678790 con 0x55ebbba30820 2024-06-04T05:11:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e4cc05740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ebbb94e890 con 0x55ebbba31140 2024-06-04T05:11:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e4cc05740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ebbba3e630 con 0x55ebbba3ba00 2024-06-04T05:11:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e437fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba38930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e437fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba38930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51436/0 (socket says 172.21.15.70:51436) 2024-06-04T05:11:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e437fe640 1 -- 172.21.15.70:0/4089228243 learned_addr learned my addr 172.21.15.70:0/4089228243 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e42ffd640 1 --2- 172.21.15.70:0/4089228243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba38e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e4cc05740 10 client.0 _ll_register_callbacks cb 0x55ebbb6a6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e43fff640 1 --2- 172.21.15.70:0/4089228243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 0x55ebbba393b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e437fe640 1 -- 172.21.15.70:0/4089228243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 msgr2=0x55ebbba393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e437fe640 1 --2- 172.21.15.70:0/4089228243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 0x55ebbba393b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e437fe640 1 -- 172.21.15.70:0/4089228243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 msgr2=0x55ebbba38e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e437fe640 1 --2- 172.21.15.70:0/4089228243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba38e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e437fe640 1 -- 172.21.15.70:0/4089228243 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebbb773620 con 0x55ebbba30820 2024-06-04T05:11:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e437fe640 1 --2- 172.21.15.70:0/4089228243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba38930 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f4e34033000 tx=0x7f4e340c5ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e14eb2126932f8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:11:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e22ffd640 1 -- 172.21.15.70:0/4089228243 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e340c5140 con 0x55ebbba30820 2024-06-04T05:11:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e22ffd640 1 -- 172.21.15.70:0/4089228243 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f4e340ffff0 con 0x55ebbba30820 2024-06-04T05:11:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.191+0000 7f4e4cc05740 1 -- 172.21.15.70:0/4089228243 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e3c01bd60 con 0x55ebbba30820 2024-06-04T05:11:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.192+0000 7f4e22ffd640 1 -- 172.21.15.70:0/4089228243 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f4e341001f0 con 0x55ebbba30820 2024-06-04T05:11:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.192+0000 7f4e4cc05740 10 client.19269 fetch_fsmap learned FSMap version 42 2024-06-04T05:11:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.192+0000 7f4e4cc05740 1 -- 172.21.15.70:0/4089228243 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55ebbba4a3c0 con 0x55ebbba30820 2024-06-04T05:11:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.192+0000 7f4e22ffd640 1 -- 172.21.15.70:0/4089228243 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e3c01bd60 con 0x55ebbba30820 2024-06-04T05:11:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.192+0000 7f4e22ffd640 1 -- 172.21.15.70:0/4089228243 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 42) ==== 8386+0+0 (secure 0 0 0) 0x7f4e3404b060 con 0x55ebbba30820 2024-06-04T05:11:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.192+0000 7f4e4cc05740 10 client.19269 fetch_fsmap finished waiting for FSMap version 42 2024-06-04T05:11:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.192+0000 7f4e4cc05740 10 client.19269 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.192+0000 7f4e4cc05740 20 client.19269 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.192+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/4089228243 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ebbba4b8c0 0x55ebbba4dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.192+0000 7f4e4cc05740 4 client.19269 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:11:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.192+0000 7f4e4cc05740 1 -- 172.21.15.70:0/4089228243 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebbb9a4b80 con 0x55ebbba4b8c0 2024-06-04T05:11:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.192+0000 7f4e42ffd640 1 --2- 172.21.15.70:0/4089228243 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ebbba4b8c0 0x55ebbba4dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.193+0000 7f4e22ffd640 10 client.19269 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.193+0000 7f4e42ffd640 1 --2- 172.21.15.70:0/4089228243 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ebbba4b8c0 0x55ebbba4dd60 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cf48c552281c793 server_cookie=185907e418c64e88 in_seq=0 out_seq=0 2024-06-04T05:11:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.193+0000 7f4e22ffd640 1 -- 172.21.15.70:0/4089228243 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ebbb9a4b80 con 0x55ebbba4b8c0 2024-06-04T05:11:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.193+0000 7f4e22ffd640 10 client.19269 handle_command_reply: tid=0 2024-06-04T05:11:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.193+0000 7f4e4cc05740 1 client.19269 shutdown 2024-06-04T05:11:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.194+0000 7f4e4cc05740 1 -- 172.21.15.70:0/4089228243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 msgr2=0x55ebbba38930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.194+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/4089228243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba38930 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f4e34033000 tx=0x7f4e340c5ef0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.194+0000 7f4e4cc05740 1 -- 172.21.15.70:0/4089228243 shutdown_connections 2024-06-04T05:11:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.194+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/4089228243 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ebbba4b8c0 0x55ebbba4dd60 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.194+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/4089228243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba38e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.194+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/4089228243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 0x55ebbba393b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.194+0000 7f4e22ffd640 0 client.19269 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.194+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/4089228243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba38930 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.194+0000 7f4e4cc05740 1 -- 172.21.15.70:0/4089228243 >> 172.21.15.70:0/4089228243 conn(0x55ebbba2cd30 msgr2=0x55ebbba32ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.194+0000 7f4e4cc05740 1 -- 172.21.15.70:0/4089228243 shutdown_connections 2024-06-04T05:11:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.194+0000 7f4e4cc05740 1 -- 172.21.15.70:0/4089228243 wait complete. 2024-06-04T05:11:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.194+0000 7f4e4cc05740 20 client.19269 trim_cache size 0 max 16384 2024-06-04T05:11:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.216+0000 7f4e4cc05740 1 Processor -- start 2024-06-04T05:11:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.216+0000 7f4e4cc05740 1 -- start start 2024-06-04T05:11:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.216+0000 7f4e4cc05740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba391d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.216+0000 7f4e4cc05740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba2be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.216+0000 7f4e4cc05740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 0x55ebbba53670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.216+0000 7f4e4cc05740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3000e860 con 0x55ebbba30820 2024-06-04T05:11:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.216+0000 7f4e4cc05740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e30010520 con 0x55ebbba31140 2024-06-04T05:11:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.216+0000 7f4e4cc05740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3000ef30 con 0x55ebbba3ba00 2024-06-04T05:11:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.216+0000 7f4e437fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.216+0000 7f4e437fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba391d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51448/0 (socket says 172.21.15.70:51448) 2024-06-04T05:11:29.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.216+0000 7f4e437fe640 1 -- 172.21.15.70:0/125892675 learned_addr learned my addr 172.21.15.70:0/125892675 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:29.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.216+0000 7f4e42ffd640 1 --2- 172.21.15.70:0/125892675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba2be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.216+0000 7f4e43fff640 1 --2- 172.21.15.70:0/125892675 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 0x55ebbba53670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.217+0000 7f4e437fe640 1 -- 172.21.15.70:0/125892675 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 msgr2=0x55ebbba53670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:29.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.217+0000 7f4e437fe640 1 --2- 172.21.15.70:0/125892675 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 0x55ebbba53670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.217+0000 7f4e437fe640 1 -- 172.21.15.70:0/125892675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 msgr2=0x55ebbba2be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:29.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.217+0000 7f4e437fe640 1 --2- 172.21.15.70:0/125892675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba2be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.217+0000 7f4e437fe640 1 -- 172.21.15.70:0/125892675 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebbba273c0 con 0x55ebbba30820 2024-06-04T05:11:29.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.217+0000 7f4e42ffd640 1 --2- 172.21.15.70:0/125892675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba2be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:29.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.217+0000 7f4e437fe640 1 --2- 172.21.15.70:0/125892675 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba391d0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f4e340ceda0 tx=0x7f4e340c5890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd1b525dc50f0a67 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:29.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.217+0000 7f4e297fa640 1 -- 172.21.15.70:0/125892675 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e34048070 con 0x55ebbba30820 2024-06-04T05:11:29.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.217+0000 7f4e297fa640 1 -- 172.21.15.70:0/125892675 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e340c94c0 con 0x55ebbba30820 2024-06-04T05:11:29.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.217+0000 7f4e297fa640 1 -- 172.21.15.70:0/125892675 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e340d22d0 con 0x55ebbba30820 2024-06-04T05:11:29.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.218+0000 7f4e4cc05740 1 -- 172.21.15.70:0/125892675 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 msgr2=0x55ebbba391d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:29.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.218+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/125892675 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba391d0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f4e340ceda0 tx=0x7f4e340c5890 comp rx=0 tx=0).stop 2024-06-04T05:11:29.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.218+0000 7f4e4cc05740 1 -- 172.21.15.70:0/125892675 shutdown_connections 2024-06-04T05:11:29.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.218+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/125892675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba2be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.218+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/125892675 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 0x55ebbba53670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.218+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/125892675 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba391d0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.218+0000 7f4e4cc05740 1 -- 172.21.15.70:0/125892675 >> 172.21.15.70:0/125892675 conn(0x55ebbba2cd30 msgr2=0x55ebbba2b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:29.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.218+0000 7f4e4cc05740 1 -- 172.21.15.70:0/125892675 shutdown_connections 2024-06-04T05:11:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.218+0000 7f4e4cc05740 1 -- 172.21.15.70:0/125892675 wait complete. 2024-06-04T05:11:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.219+0000 7f4e4cc05740 1 -- start start 2024-06-04T05:11:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.219+0000 7f4e4cc05740 1 Processor -- start 2024-06-04T05:11:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.219+0000 7f4e4cc05740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba43bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.219+0000 7f4e4cc05740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba440f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.219+0000 7f4e4cc05740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 0x55ebbba3eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.219+0000 7f4e4cc05740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3000fd90 con 0x55ebbba30820 2024-06-04T05:11:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.219+0000 7f4e4cc05740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3000f660 con 0x55ebbba31140 2024-06-04T05:11:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.219+0000 7f4e4cc05740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3000c860 con 0x55ebbba3ba00 2024-06-04T05:11:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.219+0000 7f4e4cc05740 10 client.0 _ll_register_callbacks cb 0x7f4e340c9780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.219+0000 7f4e437fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba43bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.219+0000 7f4e43fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 0x55ebbba3eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.219+0000 7f4e437fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba43bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51452/0 (socket says 172.21.15.70:51452) 2024-06-04T05:11:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.219+0000 7f4e437fe640 1 -- 172.21.15.70:0/4125938374 learned_addr learned my addr 172.21.15.70:0/4125938374 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.219+0000 7f4e42ffd640 1 --2- 172.21.15.70:0/4125938374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.220+0000 7f4e437fe640 1 -- 172.21.15.70:0/4125938374 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 msgr2=0x55ebbba3eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.220+0000 7f4e437fe640 1 --2- 172.21.15.70:0/4125938374 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 0x55ebbba3eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.220+0000 7f4e437fe640 1 -- 172.21.15.70:0/4125938374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 msgr2=0x55ebbba440f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.220+0000 7f4e437fe640 1 --2- 172.21.15.70:0/4125938374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba440f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.220+0000 7f4e437fe640 1 -- 172.21.15.70:0/4125938374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebbba3f820 con 0x55ebbba30820 2024-06-04T05:11:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.220+0000 7f4e43fff640 1 --2- 172.21.15.70:0/4125938374 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 0x55ebbba3eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.220+0000 7f4e42ffd640 1 --2- 172.21.15.70:0/4125938374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.220+0000 7f4e437fe640 1 --2- 172.21.15.70:0/4125938374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba43bb0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f4e340c5a30 tx=0x7f4e340c98b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b833d3cbc4dc8c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.220+0000 7f4e22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:11:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.220+0000 7f4e22ffd640 1 -- 172.21.15.70:0/4125938374 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e34048070 con 0x55ebbba30820 2024-06-04T05:11:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.220+0000 7f4e22ffd640 1 -- 172.21.15.70:0/4125938374 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f4e340eadc0 con 0x55ebbba30820 2024-06-04T05:11:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.220+0000 7f4e4cc05740 1 -- 172.21.15.70:0/4125938374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e340c94c0 con 0x55ebbba30820 2024-06-04T05:11:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.220+0000 7f4e22ffd640 1 -- 172.21.15.70:0/4125938374 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f4e34100200 con 0x55ebbba30820 2024-06-04T05:11:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.220+0000 7f4e4cc05740 10 client.19275 fetch_fsmap learned FSMap version 42 2024-06-04T05:11:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.220+0000 7f4e4cc05740 1 -- 172.21.15.70:0/4125938374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55ebbba3f820 con 0x55ebbba30820 2024-06-04T05:11:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.220+0000 7f4e22ffd640 1 -- 172.21.15.70:0/4125938374 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e340c94c0 con 0x55ebbba30820 2024-06-04T05:11:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.221+0000 7f4e22ffd640 1 -- 172.21.15.70:0/4125938374 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 42) ==== 8386+0+0 (secure 0 0 0) 0x7f4e340c8c10 con 0x55ebbba30820 2024-06-04T05:11:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.221+0000 7f4e4cc05740 10 client.19275 fetch_fsmap finished waiting for FSMap version 42 2024-06-04T05:11:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.221+0000 7f4e4cc05740 10 client.19275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.221+0000 7f4e4cc05740 20 client.19275 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.221+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/4125938374 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ebbbab6530 0x55ebbbab8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.221+0000 7f4e4cc05740 4 client.19275 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:11:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.221+0000 7f4e4cc05740 1 -- 172.21.15.70:0/4125938374 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ebbb99e360 con 0x55ebbbab6530 2024-06-04T05:11:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.221+0000 7f4e42ffd640 1 --2- 172.21.15.70:0/4125938374 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ebbbab6530 0x55ebbbab8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.221+0000 7f4e42ffd640 1 --2- 172.21.15.70:0/4125938374 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ebbbab6530 0x55ebbbab8950 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d5521968acd60fc server_cookie=438f9ab503aea338 in_seq=0 out_seq=0 2024-06-04T05:11:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.221+0000 7f4e22ffd640 10 client.19275 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:29.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.283+0000 7f4e22ffd640 1 -- 172.21.15.70:0/4125938374 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55ebbb99e360 con 0x55ebbbab6530 2024-06-04T05:11:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.283+0000 7f4e22ffd640 10 client.19275 handle_command_reply: tid=0 2024-06-04T05:11:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.283+0000 7f4e4cc05740 1 client.19275 shutdown 2024-06-04T05:11:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.283+0000 7f4e4cc05740 1 -- 172.21.15.70:0/4125938374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 msgr2=0x55ebbba43bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.283+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/4125938374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba43bb0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f4e340c5a30 tx=0x7f4e340c98b0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.283+0000 7f4e4cc05740 1 -- 172.21.15.70:0/4125938374 shutdown_connections 2024-06-04T05:11:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.283+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/4125938374 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ebbbab6530 0x55ebbbab8950 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.283+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/4125938374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebbba31140 0x55ebbba440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.283+0000 7f4e22ffd640 0 client.19275 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.283+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/4125938374 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ebbba3ba00 0x55ebbba3eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.283+0000 7f4e4cc05740 1 --2- 172.21.15.70:0/4125938374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ebbba30820 0x55ebbba43bb0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.283+0000 7f4e4cc05740 1 -- 172.21.15.70:0/4125938374 >> 172.21.15.70:0/4125938374 conn(0x55ebbba2cd30 msgr2=0x55ebbba32ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.283+0000 7f4e4cc05740 1 -- 172.21.15.70:0/4125938374 shutdown_connections 2024-06-04T05:11:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.283+0000 7f4e4cc05740 1 -- 172.21.15.70:0/4125938374 wait complete. 2024-06-04T05:11:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.283+0000 7f4e4cc05740 20 client.19275 trim_cache size 0 max 16384 2024-06-04T05:11:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:11:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:11:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:11:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.284+0000 7f4e49e12640 1 -- 172.21.15.70:0/881070174 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e2c0b2ac0 msgr2=0x7f4e2c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.284+0000 7f4e49e12640 1 --2- 172.21.15.70:0/881070174 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e2c0b2ac0 0x7f4e2c0b4f80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c023000 tx=0x7f4e3c024000 comp rx=0 tx=0).stop 2024-06-04T05:11:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.284+0000 7f4e49e12640 1 -- 172.21.15.70:0/881070174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e44103ce0 msgr2=0x7f4e44101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.284+0000 7f4e49e12640 1 --2- 172.21.15.70:0/881070174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e44103ce0 0x7f4e44101930 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4e34005940 tx=0x7f4e34004040 comp rx=0 tx=0).stop 2024-06-04T05:11:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.284+0000 7f4e49e12640 1 -- 172.21.15.70:0/881070174 shutdown_connections 2024-06-04T05:11:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.284+0000 7f4e49e12640 1 --2- 172.21.15.70:0/881070174 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e2c0b2ac0 0x7f4e2c0b4f80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.284+0000 7f4e49e12640 1 --2- 172.21.15.70:0/881070174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e44104600 0x7f4e44101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.284+0000 7f4e49e12640 1 --2- 172.21.15.70:0/881070174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e4410d020 0x7f4e441023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.284+0000 7f4e49e12640 1 --2- 172.21.15.70:0/881070174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e44103ce0 0x7f4e44101930 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:29.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.284+0000 7f4e49e12640 1 -- 172.21.15.70:0/881070174 >> 172.21.15.70:0/881070174 conn(0x7f4e44078ad0 msgr2=0x7f4e440feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:29.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.285+0000 7f4e49e12640 1 -- 172.21.15.70:0/881070174 shutdown_connections 2024-06-04T05:11:29.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:29.285+0000 7f4e49e12640 1 -- 172.21.15.70:0/881070174 wait complete. 2024-06-04T05:11:29.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:29 smithi098 ceph-mon[35569]: pgmap v446: 97 pgs: 97 active+clean; 360 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 4.4 MiB/s wr, 209 op/s 2024-06-04T05:11:29.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:29 smithi070 ceph-mon[32631]: pgmap v446: 97 pgs: 97 active+clean; 360 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 4.4 MiB/s wr, 209 op/s 2024-06-04T05:11:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:29 smithi112 ceph-mon[36453]: pgmap v446: 97 pgs: 97 active+clean; 360 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 4.4 MiB/s wr, 209 op/s 2024-06-04T05:11:29.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 4125 18.97 MB/sec warmup 21 sec latency 79.095 ms 2024-06-04T05:11:30.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 4306 18.45 MB/sec warmup 22 sec latency 57.417 ms 2024-06-04T05:11:31.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:31 smithi070 ceph-mon[32631]: pgmap v447: 97 pgs: 97 active+clean; 360 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 4.4 MiB/s wr, 209 op/s 2024-06-04T05:11:31.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:31 smithi112 ceph-mon[36453]: pgmap v447: 97 pgs: 97 active+clean; 360 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 4.4 MiB/s wr, 209 op/s 2024-06-04T05:11:31.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:31 smithi098 ceph-mon[35569]: pgmap v447: 97 pgs: 97 active+clean; 360 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 4.4 MiB/s wr, 209 op/s 2024-06-04T05:11:31.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 4500 18.18 MB/sec warmup 23 sec latency 87.399 ms 2024-06-04T05:11:32.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 4701 17.66 MB/sec warmup 24 sec latency 757.539 ms 2024-06-04T05:11:33.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:33 smithi070 ceph-mon[32631]: pgmap v448: 97 pgs: 97 active+clean; 363 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.8 MiB/s wr, 338 op/s 2024-06-04T05:11:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:33 smithi112 ceph-mon[36453]: pgmap v448: 97 pgs: 97 active+clean; 363 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.8 MiB/s wr, 338 op/s 2024-06-04T05:11:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:33 smithi098 ceph-mon[35569]: pgmap v448: 97 pgs: 97 active+clean; 363 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.8 MiB/s wr, 338 op/s 2024-06-04T05:11:33.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 4899 17.51 MB/sec warmup 25 sec latency 797.178 ms 2024-06-04T05:11:34.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 5078 16.91 MB/sec warmup 26 sec latency 59.423 ms 2024-06-04T05:11:35.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:35 smithi070 ceph-mon[32631]: pgmap v449: 97 pgs: 97 active+clean; 363 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.3 MiB/s wr, 231 op/s 2024-06-04T05:11:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:35 smithi112 ceph-mon[36453]: pgmap v449: 97 pgs: 97 active+clean; 363 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.3 MiB/s wr, 231 op/s 2024-06-04T05:11:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:35 smithi098 ceph-mon[35569]: pgmap v449: 97 pgs: 97 active+clean; 363 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.3 MiB/s wr, 231 op/s 2024-06-04T05:11:35.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 5291 16.72 MB/sec warmup 27 sec latency 60.372 ms 2024-06-04T05:11:36.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 5469 16.24 MB/sec warmup 28 sec latency 59.779 ms 2024-06-04T05:11:37.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:37 smithi070 ceph-mon[32631]: pgmap v450: 97 pgs: 97 active+clean; 351 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.9 MiB/s wr, 364 op/s 2024-06-04T05:11:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:37 smithi112 ceph-mon[36453]: pgmap v450: 97 pgs: 97 active+clean; 351 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.9 MiB/s wr, 364 op/s 2024-06-04T05:11:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:37 smithi098 ceph-mon[35569]: pgmap v450: 97 pgs: 97 active+clean; 351 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.9 MiB/s wr, 364 op/s 2024-06-04T05:11:37.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 5671 15.92 MB/sec warmup 29 sec latency 55.364 ms 2024-06-04T05:11:38.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 5888 15.91 MB/sec warmup 30 sec latency 84.380 ms 2024-06-04T05:11:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:39 smithi070 ceph-mon[32631]: pgmap v451: 97 pgs: 97 active+clean; 351 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 262 op/s 2024-06-04T05:11:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:39 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:11:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:39 smithi112 ceph-mon[36453]: pgmap v451: 97 pgs: 97 active+clean; 351 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 262 op/s 2024-06-04T05:11:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:39 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:11:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:39 smithi098 ceph-mon[35569]: pgmap v451: 97 pgs: 97 active+clean; 351 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 262 op/s 2024-06-04T05:11:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:39 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:11:39.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 6115 15.83 MB/sec warmup 31 sec latency 55.350 ms 2024-06-04T05:11:40.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:40 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:11:40.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:40 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:11:40.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:40 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:11:40.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:40 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:11:40.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:40 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:11:40.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:40 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:11:40.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:40 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:11:40.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:40 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:11:40.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:40 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:11:40.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 6284 15.78 MB/sec warmup 32 sec latency 89.405 ms 2024-06-04T05:11:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:41 smithi070 ceph-mon[32631]: pgmap v452: 97 pgs: 97 active+clean; 351 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 262 op/s 2024-06-04T05:11:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:41 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:41 smithi112 ceph-mon[36453]: pgmap v452: 97 pgs: 97 active+clean; 351 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 262 op/s 2024-06-04T05:11:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:41 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:41 smithi098 ceph-mon[35569]: pgmap v452: 97 pgs: 97 active+clean; 351 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 262 op/s 2024-06-04T05:11:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:41 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:41.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 6484 15.74 MB/sec warmup 33 sec latency 67.129 ms 2024-06-04T05:11:42.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:11:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.411+0000 7f2a20d46640 1 -- 172.21.15.70:0/2361319957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c10d020 msgr2=0x7f2a1c10f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.411+0000 7f2a20d46640 1 --2- 172.21.15.70:0/2361319957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c10d020 0x7f2a1c10f4c0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f2a14009ca0 tx=0x7f2a1402a9e0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.411+0000 7f2a20d46640 1 -- 172.21.15.70:0/2361319957 shutdown_connections 2024-06-04T05:11:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.411+0000 7f2a20d46640 1 --2- 172.21.15.70:0/2361319957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a1c104600 0x7f2a1c10cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.411+0000 7f2a20d46640 1 --2- 172.21.15.70:0/2361319957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c10d020 0x7f2a1c10f4c0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.411+0000 7f2a20d46640 1 --2- 172.21.15.70:0/2361319957 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a1c103ce0 0x7f2a1c1040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.411+0000 7f2a20d46640 1 -- 172.21.15.70:0/2361319957 >> 172.21.15.70:0/2361319957 conn(0x7f2a1c078ad0 msgr2=0x7f2a1c078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.411+0000 7f2a20d46640 1 -- 172.21.15.70:0/2361319957 shutdown_connections 2024-06-04T05:11:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.411+0000 7f2a20d46640 1 -- 172.21.15.70:0/2361319957 wait complete. 2024-06-04T05:11:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.412+0000 7f2a20d46640 1 Processor -- start 2024-06-04T05:11:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.412+0000 7f2a20d46640 1 -- start start 2024-06-04T05:11:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.413+0000 7f2a20d46640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c103ce0 0x7f2a1c199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.413+0000 7f2a20d46640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a1c104600 0x7f2a1c1995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.413+0000 7f2a20d46640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a1c10d020 0x7f2a1c19d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.413+0000 7f2a20d46640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c112bd0 con 0x7f2a1c103ce0 2024-06-04T05:11:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.413+0000 7f2a20d46640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c112a50 con 0x7f2a1c10d020 2024-06-04T05:11:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.413+0000 7f2a20d46640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c112d50 con 0x7f2a1c104600 2024-06-04T05:11:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.413+0000 7f2a1a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c103ce0 0x7f2a1c199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.413+0000 7f2a1a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c103ce0 0x7f2a1c199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47744/0 (socket says 172.21.15.70:47744) 2024-06-04T05:11:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.413+0000 7f2a1a59c640 1 -- 172.21.15.70:0/1342842470 learned_addr learned my addr 172.21.15.70:0/1342842470 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.413+0000 7f2a1ad9d640 1 --2- 172.21.15.70:0/1342842470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a1c10d020 0x7f2a1c19d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.413+0000 7f2a11d9b640 1 --2- 172.21.15.70:0/1342842470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a1c104600 0x7f2a1c1995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.413+0000 7f2a1ad9d640 1 -- 172.21.15.70:0/1342842470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a1c104600 msgr2=0x7f2a1c1995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.413+0000 7f2a1ad9d640 1 --2- 172.21.15.70:0/1342842470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a1c104600 0x7f2a1c1995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.413+0000 7f2a1ad9d640 1 -- 172.21.15.70:0/1342842470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c103ce0 msgr2=0x7f2a1c199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.413+0000 7f2a1ad9d640 1 --2- 172.21.15.70:0/1342842470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c103ce0 0x7f2a1c199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.413+0000 7f2a1ad9d640 1 -- 172.21.15.70:0/1342842470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a1c19e080 con 0x7f2a1c10d020 2024-06-04T05:11:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.414+0000 7f2a1ad9d640 1 --2- 172.21.15.70:0/1342842470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a1c10d020 0x7f2a1c19d950 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2a14005e30 tx=0x7f2a14003ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.414+0000 7f2a13fff640 1 -- 172.21.15.70:0/1342842470 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a14046070 con 0x7f2a1c10d020 2024-06-04T05:11:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.414+0000 7f2a13fff640 1 -- 172.21.15.70:0/1342842470 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2a1402ee40 con 0x7f2a1c10d020 2024-06-04T05:11:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.414+0000 7f2a13fff640 1 -- 172.21.15.70:0/1342842470 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a14033070 con 0x7f2a1c10d020 2024-06-04T05:11:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.414+0000 7f2a20d46640 1 -- 172.21.15.70:0/1342842470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a1c19e310 con 0x7f2a1c10d020 2024-06-04T05:11:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.414+0000 7f2a20d46640 1 -- 172.21.15.70:0/1342842470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a1c1a5c00 con 0x7f2a1c10d020 2024-06-04T05:11:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.418+0000 7f2a23a68740 1 Processor -- start 2024-06-04T05:11:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.418+0000 7f2a13fff640 1 -- 172.21.15.70:0/1342842470 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f2a14049070 con 0x7f2a1c10d020 2024-06-04T05:11:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.418+0000 7f2a23a68740 1 -- start start 2024-06-04T05:11:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.418+0000 7f2a23a68740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb691ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.418+0000 7f2a23a68740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb692000 0x562cbb6a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.418+0000 7f2a23a68740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb6a0e10 0x562cbb6a3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.418+0000 7f2a23a68740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562cbb603c30 con 0x562cbb6916e0 2024-06-04T05:11:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.418+0000 7f2a23a68740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562cbb5f37d0 con 0x562cbb692000 2024-06-04T05:11:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.418+0000 7f2a23a68740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562cbb5f7810 con 0x562cbb6a0e10 2024-06-04T05:11:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.419+0000 7f2a1a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb692000 0x562cbb6a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.419+0000 7f2a1a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb692000 0x562cbb6a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:39230/0 (socket says 172.21.15.70:39230) 2024-06-04T05:11:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.419+0000 7f2a1a59c640 1 -- 172.21.15.70:0/468823587 learned_addr learned my addr 172.21.15.70:0/468823587 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.419+0000 7f2a11d9b640 1 --2- 172.21.15.70:0/468823587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb691ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.419+0000 7f2a1a59c640 1 -- 172.21.15.70:0/468823587 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb6a0e10 msgr2=0x562cbb6a3200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-04T05:11:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.419+0000 7f2a1a59c640 1 --2- 172.21.15.70:0/468823587 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb6a0e10 0x562cbb6a3200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.419+0000 7f2a1a59c640 1 -- 172.21.15.70:0/468823587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 msgr2=0x562cbb691ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.419+0000 7f2a1a59c640 1 --2- 172.21.15.70:0/468823587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb691ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.419+0000 7f2a1a59c640 1 -- 172.21.15.70:0/468823587 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562cbb308210 con 0x562cbb692000 2024-06-04T05:11:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.419+0000 7f2a11d9b640 1 --2- 172.21.15.70:0/468823587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb691ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.420+0000 7f2a1a59c640 1 --2- 172.21.15.70:0/468823587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb692000 0x562cbb6a08d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c009dc0 tx=0x7f2a0c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27c219c4d14bac61 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.420+0000 7f2a13fff640 1 --2- 172.21.15.70:0/1342842470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2a080b2a90 0x7f2a080b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.420+0000 7f2a11d9b640 1 --2- 172.21.15.70:0/1342842470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2a080b2a90 0x7f2a080b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.420+0000 7f2a13fff640 1 -- 172.21.15.70:0/1342842470 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f2a14102f10 con 0x7f2a1c10d020 2024-06-04T05:11:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.420+0000 7f2a10d99640 1 -- 172.21.15.70:0/468823587 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c005220 con 0x562cbb692000 2024-06-04T05:11:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.420+0000 7f2a10d99640 1 -- 172.21.15.70:0/468823587 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2a0c004c30 con 0x562cbb692000 2024-06-04T05:11:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.420+0000 7f2a10d99640 1 -- 172.21.15.70:0/468823587 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c019780 con 0x562cbb692000 2024-06-04T05:11:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.421+0000 7f2a23a68740 1 -- 172.21.15.70:0/468823587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb692000 msgr2=0x562cbb6a08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.421+0000 7f2a23a68740 1 --2- 172.21.15.70:0/468823587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb692000 0x562cbb6a08d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c009dc0 tx=0x7f2a0c011810 comp rx=0 tx=0).stop 2024-06-04T05:11:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.421+0000 7f2a23a68740 1 -- 172.21.15.70:0/468823587 shutdown_connections 2024-06-04T05:11:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.421+0000 7f2a23a68740 1 --2- 172.21.15.70:0/468823587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb692000 0x562cbb6a08d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.421+0000 7f2a23a68740 1 --2- 172.21.15.70:0/468823587 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb6a0e10 0x562cbb6a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.421+0000 7f2a23a68740 1 --2- 172.21.15.70:0/468823587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb691ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.421+0000 7f2a23a68740 1 -- 172.21.15.70:0/468823587 >> 172.21.15.70:0/468823587 conn(0x562cbb68dd30 msgr2=0x562cbb68e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.421+0000 7f2a11d9b640 1 --2- 172.21.15.70:0/1342842470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2a080b2a90 0x7f2a080b4f50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c101340 tx=0x7f2a04005f90 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.421+0000 7f2a23a68740 1 -- 172.21.15.70:0/468823587 shutdown_connections 2024-06-04T05:11:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.421+0000 7f2a23a68740 1 -- 172.21.15.70:0/468823587 wait complete. 2024-06-04T05:11:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.422+0000 7f2a23a68740 1 -- start start 2024-06-04T05:11:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.422+0000 7f2a23a68740 1 Processor -- start 2024-06-04T05:11:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.422+0000 7f2a23a68740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb69ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.422+0000 7f2a23a68740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb692000 0x562cbb69e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.422+0000 7f2a23a68740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb6a0e10 0x562cbb69e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.422+0000 7f2a23a68740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562cbb2d9790 con 0x562cbb6916e0 2024-06-04T05:11:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.422+0000 7f2a23a68740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562cbb5af890 con 0x562cbb692000 2024-06-04T05:11:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.422+0000 7f2a23a68740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562cbb6a3a30 con 0x562cbb6a0e10 2024-06-04T05:11:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.422+0000 7f2a23a68740 10 client.0 _ll_register_callbacks cb 0x562cbb307a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.422+0000 7f2a1a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb69ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.422+0000 7f2a1a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb69ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47762/0 (socket says 172.21.15.70:47762) 2024-06-04T05:11:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.422+0000 7f2a1ad9d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb6a0e10 0x562cbb69e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.422+0000 7f2a1a59c640 1 -- 172.21.15.70:0/4239421048 learned_addr learned my addr 172.21.15.70:0/4239421048 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.422+0000 7f2a11d9b640 1 --2- 172.21.15.70:0/4239421048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb692000 0x562cbb69e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.423+0000 7f2a1ad9d640 1 -- 172.21.15.70:0/4239421048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb692000 msgr2=0x562cbb69e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.423+0000 7f2a1ad9d640 1 --2- 172.21.15.70:0/4239421048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb692000 0x562cbb69e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.423+0000 7f2a1ad9d640 1 -- 172.21.15.70:0/4239421048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 msgr2=0x562cbb69ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.423+0000 7f2a1ad9d640 1 --2- 172.21.15.70:0/4239421048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb69ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.423+0000 7f2a1ad9d640 1 -- 172.21.15.70:0/4239421048 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562cbb3d4620 con 0x562cbb6a0e10 2024-06-04T05:11:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.423+0000 7f2a1a59c640 1 --2- 172.21.15.70:0/4239421048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb69ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.423+0000 7f2a1ad9d640 1 --2- 172.21.15.70:0/4239421048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb6a0e10 0x562cbb69e840 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2a14100630 tx=0x7f2a140c9c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e54de382a3f34891 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.423+0000 7f2a01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:11:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.423+0000 7f2a01ffb640 1 -- 172.21.15.70:0/4239421048 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a140e4060 con 0x562cbb6a0e10 2024-06-04T05:11:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.423+0000 7f2a01ffb640 1 -- 172.21.15.70:0/4239421048 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f2a140ce170 con 0x562cbb6a0e10 2024-06-04T05:11:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.423+0000 7f2a23a68740 1 -- 172.21.15.70:0/4239421048 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a0c004c30 con 0x562cbb6a0e10 2024-06-04T05:11:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.424+0000 7f2a01ffb640 1 -- 172.21.15.70:0/4239421048 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f2a140eb050 con 0x562cbb6a0e10 2024-06-04T05:11:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.424+0000 7f2a23a68740 10 client.28282 fetch_fsmap learned FSMap version 43 2024-06-04T05:11:42.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.424+0000 7f2a23a68740 1 -- 172.21.15.70:0/4239421048 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x562cbb6af880 con 0x562cbb6a0e10 2024-06-04T05:11:42.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.424+0000 7f2a01ffb640 1 -- 172.21.15.70:0/4239421048 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2a0c004c30 con 0x562cbb6a0e10 2024-06-04T05:11:42.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.424+0000 7f2a01ffb640 1 -- 172.21.15.70:0/4239421048 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 43) ==== 8390+0+0 (secure 0 0 0) 0x7f2a14004560 con 0x562cbb6a0e10 2024-06-04T05:11:42.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.424+0000 7f2a23a68740 10 client.28282 fetch_fsmap finished waiting for FSMap version 43 2024-06-04T05:11:42.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.424+0000 7f2a23a68740 10 client.28282 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:42.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.424+0000 7f2a23a68740 20 client.28282 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:42.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.424+0000 7f2a23a68740 1 --2- 172.21.15.70:0/4239421048 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562cbb6b0e70 0x562cbb6b3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.424+0000 7f2a23a68740 4 client.28282 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:11:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.424+0000 7f2a23a68740 1 -- 172.21.15.70:0/4239421048 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562cbb605b80 con 0x562cbb6b0e70 2024-06-04T05:11:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.424+0000 7f2a1a59c640 1 --2- 172.21.15.70:0/4239421048 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562cbb6b0e70 0x562cbb6b3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.425+0000 7f2a1a59c640 1 --2- 172.21.15.70:0/4239421048 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562cbb6b0e70 0x562cbb6b3330 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6ebe7f657e79db3 server_cookie=ec8abfabe034307d in_seq=0 out_seq=0 2024-06-04T05:11:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.425+0000 7f2a01ffb640 10 client.28282 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.426+0000 7f2a01ffb640 1 -- 172.21.15.70:0/4239421048 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562cbb605b80 con 0x562cbb6b0e70 2024-06-04T05:11:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.426+0000 7f2a01ffb640 10 client.28282 handle_command_reply: tid=0 2024-06-04T05:11:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.426+0000 7f2a23a68740 1 client.28282 shutdown 2024-06-04T05:11:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.426+0000 7f2a23a68740 1 -- 172.21.15.70:0/4239421048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb6a0e10 msgr2=0x562cbb69e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.426+0000 7f2a23a68740 1 --2- 172.21.15.70:0/4239421048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb6a0e10 0x562cbb69e840 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2a14100630 tx=0x7f2a140c9c40 comp rx=0 tx=0).stop 2024-06-04T05:11:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.426+0000 7f2a23a68740 1 -- 172.21.15.70:0/4239421048 shutdown_connections 2024-06-04T05:11:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.426+0000 7f2a23a68740 1 --2- 172.21.15.70:0/4239421048 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562cbb6b0e70 0x562cbb6b3330 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.426+0000 7f2a23a68740 1 --2- 172.21.15.70:0/4239421048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb692000 0x562cbb69e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.426+0000 7f2a01ffb640 0 client.28282 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.426+0000 7f2a23a68740 1 --2- 172.21.15.70:0/4239421048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb6a0e10 0x562cbb69e840 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.426+0000 7f2a23a68740 1 --2- 172.21.15.70:0/4239421048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb69ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.426+0000 7f2a23a68740 1 -- 172.21.15.70:0/4239421048 >> 172.21.15.70:0/4239421048 conn(0x562cbb68dd30 msgr2=0x562cbb693ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:42.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.426+0000 7f2a23a68740 1 -- 172.21.15.70:0/4239421048 shutdown_connections 2024-06-04T05:11:42.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.426+0000 7f2a23a68740 1 -- 172.21.15.70:0/4239421048 wait complete. 2024-06-04T05:11:42.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.426+0000 7f2a23a68740 20 client.28282 trim_cache size 0 max 16384 2024-06-04T05:11:42.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.452+0000 7f2a23a68740 1 Processor -- start 2024-06-04T05:11:42.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.452+0000 7f2a23a68740 1 -- start start 2024-06-04T05:11:42.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.452+0000 7f2a23a68740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb69d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.452+0000 7f2a23a68740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb692000 0x562cbb6b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.452+0000 7f2a1a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb69d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:42.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.452+0000 7f2a1a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb69d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47770/0 (socket says 172.21.15.70:47770) 2024-06-04T05:11:42.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.452+0000 7f2a23a68740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb6a0e10 0x562cbb6b9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.453+0000 7f2a1a59c640 1 -- 172.21.15.70:0/1776358871 learned_addr learned my addr 172.21.15.70:0/1776358871 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:42.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.453+0000 7f2a11d9b640 1 --2- 172.21.15.70:0/1776358871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb692000 0x562cbb6b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:42.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.453+0000 7f2a23a68740 1 -- 172.21.15.70:0/1776358871 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c1ac380 con 0x562cbb6916e0 2024-06-04T05:11:42.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.453+0000 7f2a1ad9d640 1 --2- 172.21.15.70:0/1776358871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb6a0e10 0x562cbb6b9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:42.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.453+0000 7f2a23a68740 1 -- 172.21.15.70:0/1776358871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c1ae040 con 0x562cbb6a0e10 2024-06-04T05:11:42.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.453+0000 7f2a23a68740 1 -- 172.21.15.70:0/1776358871 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c1aca50 con 0x562cbb692000 2024-06-04T05:11:42.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.453+0000 7f2a11d9b640 1 -- 172.21.15.70:0/1776358871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb6a0e10 msgr2=0x562cbb6b9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:42.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.453+0000 7f2a11d9b640 1 --2- 172.21.15.70:0/1776358871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb6a0e10 0x562cbb6b9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.453+0000 7f2a11d9b640 1 -- 172.21.15.70:0/1776358871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 msgr2=0x562cbb69d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:42.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.453+0000 7f2a11d9b640 1 --2- 172.21.15.70:0/1776358871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb69d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.453+0000 7f2a11d9b640 1 -- 172.21.15.70:0/1776358871 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562cbb691e30 con 0x562cbb692000 2024-06-04T05:11:42.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.453+0000 7f2a1a59c640 1 --2- 172.21.15.70:0/1776358871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb69d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:42.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.453+0000 7f2a11d9b640 1 --2- 172.21.15.70:0/1776358871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb692000 0x562cbb6b8c00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2a04013cc0 tx=0x7f2a0400e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c83035ad096a68d7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:42.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.454+0000 7f2a03fff640 1 -- 172.21.15.70:0/1776358871 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a04004220 con 0x562cbb692000 2024-06-04T05:11:42.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.454+0000 7f2a03fff640 1 -- 172.21.15.70:0/1776358871 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2a040043c0 con 0x562cbb692000 2024-06-04T05:11:42.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.454+0000 7f2a03fff640 1 -- 172.21.15.70:0/1776358871 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0401c6e0 con 0x562cbb692000 2024-06-04T05:11:42.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.455+0000 7f2a23a68740 1 -- 172.21.15.70:0/1776358871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb692000 msgr2=0x562cbb6b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:42.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.455+0000 7f2a23a68740 1 --2- 172.21.15.70:0/1776358871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb692000 0x562cbb6b8c00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2a04013cc0 tx=0x7f2a0400e000 comp rx=0 tx=0).stop 2024-06-04T05:11:42.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.455+0000 7f2a23a68740 1 -- 172.21.15.70:0/1776358871 shutdown_connections 2024-06-04T05:11:42.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.455+0000 7f2a23a68740 1 --2- 172.21.15.70:0/1776358871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb6a0e10 0x562cbb6b9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.455+0000 7f2a23a68740 1 --2- 172.21.15.70:0/1776358871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb692000 0x562cbb6b8c00 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.455+0000 7f2a23a68740 1 --2- 172.21.15.70:0/1776358871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb69d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.455+0000 7f2a23a68740 1 -- 172.21.15.70:0/1776358871 >> 172.21.15.70:0/1776358871 conn(0x562cbb68dd30 msgr2=0x562cbb68cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.455+0000 7f2a23a68740 1 -- 172.21.15.70:0/1776358871 shutdown_connections 2024-06-04T05:11:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.456+0000 7f2a23a68740 1 -- 172.21.15.70:0/1776358871 wait complete. 2024-06-04T05:11:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.456+0000 7f2a23a68740 1 -- start start 2024-06-04T05:11:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.456+0000 7f2a23a68740 1 Processor -- start 2024-06-04T05:11:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.456+0000 7f2a23a68740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb6a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.456+0000 7f2a23a68740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb692000 0x562cbb6a4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.456+0000 7f2a23a68740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb6a0e10 0x562cbb719ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.456+0000 7f2a23a68740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c1ad8b0 con 0x562cbb6916e0 2024-06-04T05:11:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.456+0000 7f2a23a68740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c1ad180 con 0x562cbb6a0e10 2024-06-04T05:11:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.456+0000 7f2a23a68740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c1aa380 con 0x562cbb692000 2024-06-04T05:11:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.456+0000 7f2a23a68740 10 client.0 _ll_register_callbacks cb 0x7f2a1402f850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.457+0000 7f2a1a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb6a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.457+0000 7f2a1ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb6a0e10 0x562cbb719ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.457+0000 7f2a11d9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb692000 0x562cbb6a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.457+0000 7f2a1a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb6a3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47778/0 (socket says 172.21.15.70:47778) 2024-06-04T05:11:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.457+0000 7f2a1a59c640 1 -- 172.21.15.70:0/2769685704 learned_addr learned my addr 172.21.15.70:0/2769685704 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.457+0000 7f2a1ad9d640 1 -- 172.21.15.70:0/2769685704 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb692000 msgr2=0x562cbb6a4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.457+0000 7f2a1ad9d640 1 --2- 172.21.15.70:0/2769685704 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb692000 0x562cbb6a4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.457+0000 7f2a1ad9d640 1 -- 172.21.15.70:0/2769685704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 msgr2=0x562cbb6a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.457+0000 7f2a1ad9d640 1 --2- 172.21.15.70:0/2769685704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb6a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.457+0000 7f2a1ad9d640 1 -- 172.21.15.70:0/2769685704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562cbb71a000 con 0x562cbb6a0e10 2024-06-04T05:11:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.457+0000 7f2a1ad9d640 1 --2- 172.21.15.70:0/2769685704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb6a0e10 0x562cbb719ac0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2a14100c30 tx=0x7f2a140c9860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56047ffe2acd99e6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.457+0000 7f2a01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:11:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.457+0000 7f2a01ffb640 1 -- 172.21.15.70:0/2769685704 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a140a8060 con 0x562cbb6a0e10 2024-06-04T05:11:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.457+0000 7f2a01ffb640 1 -- 172.21.15.70:0/2769685704 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f2a140eb1f0 con 0x562cbb6a0e10 2024-06-04T05:11:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.458+0000 7f2a23a68740 1 -- 172.21.15.70:0/2769685704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a040043c0 con 0x562cbb6a0e10 2024-06-04T05:11:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.458+0000 7f2a01ffb640 1 -- 172.21.15.70:0/2769685704 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f2a140c9f90 con 0x562cbb6a0e10 2024-06-04T05:11:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.458+0000 7f2a23a68740 10 client.28157 fetch_fsmap learned FSMap version 43 2024-06-04T05:11:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.458+0000 7f2a23a68740 1 -- 172.21.15.70:0/2769685704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x562cbb71b220 con 0x562cbb6a0e10 2024-06-04T05:11:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.458+0000 7f2a01ffb640 1 -- 172.21.15.70:0/2769685704 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2a040043c0 con 0x562cbb6a0e10 2024-06-04T05:11:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.458+0000 7f2a01ffb640 1 -- 172.21.15.70:0/2769685704 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 43) ==== 8390+0+0 (secure 0 0 0) 0x7f2a14004b40 con 0x562cbb6a0e10 2024-06-04T05:11:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.458+0000 7f2a23a68740 10 client.28157 fetch_fsmap finished waiting for FSMap version 43 2024-06-04T05:11:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.458+0000 7f2a23a68740 10 client.28157 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.458+0000 7f2a23a68740 20 client.28157 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.458+0000 7f2a23a68740 1 --2- 172.21.15.70:0/2769685704 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562cbb71b760 0x562cbb71db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.458+0000 7f2a23a68740 4 client.28157 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:11:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.458+0000 7f2a23a68740 1 -- 172.21.15.70:0/2769685704 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562cbb5ff360 con 0x562cbb71b760 2024-06-04T05:11:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.458+0000 7f2a1a59c640 1 --2- 172.21.15.70:0/2769685704 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562cbb71b760 0x562cbb71db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:42.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.459+0000 7f2a01ffb640 10 client.28157 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:42.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.459+0000 7f2a1a59c640 1 --2- 172.21.15.70:0/2769685704 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562cbb71b760 0x562cbb71db50 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=920e36617df1fe0b server_cookie=4505bd0363627943 in_seq=0 out_seq=0 2024-06-04T05:11:42.532 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:11:42.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.600+0000 7f2a01ffb640 1 -- 172.21.15.70:0/2769685704 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x562cbb5ff360 con 0x562cbb71b760 2024-06-04T05:11:42.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.600+0000 7f2a01ffb640 10 client.28157 handle_command_reply: tid=0 2024-06-04T05:11:42.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.600+0000 7f2a23a68740 1 client.28157 shutdown 2024-06-04T05:11:42.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.600+0000 7f2a23a68740 1 -- 172.21.15.70:0/2769685704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb6a0e10 msgr2=0x562cbb719ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:42.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.600+0000 7f2a23a68740 1 --2- 172.21.15.70:0/2769685704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb6a0e10 0x562cbb719ac0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2a14100c30 tx=0x7f2a140c9860 comp rx=0 tx=0).stop 2024-06-04T05:11:42.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.601+0000 7f2a23a68740 1 -- 172.21.15.70:0/2769685704 shutdown_connections 2024-06-04T05:11:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.601+0000 7f2a23a68740 1 --2- 172.21.15.70:0/2769685704 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562cbb71b760 0x562cbb71db50 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.601+0000 7f2a23a68740 1 --2- 172.21.15.70:0/2769685704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562cbb6a0e10 0x562cbb719ac0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.601+0000 7f2a01ffb640 0 client.28157 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.601+0000 7f2a23a68740 1 --2- 172.21.15.70:0/2769685704 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562cbb692000 0x562cbb6a4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.601+0000 7f2a23a68740 1 --2- 172.21.15.70:0/2769685704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562cbb6916e0 0x562cbb6a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.601+0000 7f2a23a68740 1 -- 172.21.15.70:0/2769685704 >> 172.21.15.70:0/2769685704 conn(0x562cbb68dd30 msgr2=0x562cbb68c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.601+0000 7f2a23a68740 1 -- 172.21.15.70:0/2769685704 shutdown_connections 2024-06-04T05:11:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.601+0000 7f2a23a68740 1 -- 172.21.15.70:0/2769685704 wait complete. 2024-06-04T05:11:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.601+0000 7f2a23a68740 20 client.28157 trim_cache size 0 max 16384 2024-06-04T05:11:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:11:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:11:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:11:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.602+0000 7f2a20d46640 1 -- 172.21.15.70:0/1342842470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2a080b2a90 msgr2=0x7f2a080b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.602+0000 7f2a20d46640 1 --2- 172.21.15.70:0/1342842470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2a080b2a90 0x7f2a080b4f50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c101340 tx=0x7f2a04005f90 comp rx=0 tx=0).stop 2024-06-04T05:11:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.602+0000 7f2a20d46640 1 -- 172.21.15.70:0/1342842470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a1c10d020 msgr2=0x7f2a1c19d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.602+0000 7f2a20d46640 1 --2- 172.21.15.70:0/1342842470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a1c10d020 0x7f2a1c19d950 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2a14005e30 tx=0x7f2a14003ea0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.602+0000 7f2a20d46640 1 -- 172.21.15.70:0/1342842470 shutdown_connections 2024-06-04T05:11:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.602+0000 7f2a20d46640 1 --2- 172.21.15.70:0/1342842470 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2a080b2a90 0x7f2a080b4f50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.602+0000 7f2a20d46640 1 --2- 172.21.15.70:0/1342842470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a1c10d020 0x7f2a1c19d950 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.602+0000 7f2a20d46640 1 --2- 172.21.15.70:0/1342842470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a1c104600 0x7f2a1c1995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.602+0000 7f2a20d46640 1 --2- 172.21.15.70:0/1342842470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c103ce0 0x7f2a1c199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.602+0000 7f2a20d46640 1 -- 172.21.15.70:0/1342842470 >> 172.21.15.70:0/1342842470 conn(0x7f2a1c078ad0 msgr2=0x7f2a1c0fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.602+0000 7f2a20d46640 1 -- 172.21.15.70:0/1342842470 shutdown_connections 2024-06-04T05:11:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:42.602+0000 7f2a20d46640 1 -- 172.21.15.70:0/1342842470 wait complete. 2024-06-04T05:11:42.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 6719 15.94 MB/sec warmup 34 sec latency 103.378 ms 2024-06-04T05:11:43.546 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:43 smithi098 ceph-mon[35569]: pgmap v453: 97 pgs: 97 active+clean; 366 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.2 MiB/s wr, 384 op/s 2024-06-04T05:11:43.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:43 smithi070 ceph-mon[32631]: pgmap v453: 97 pgs: 97 active+clean; 366 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.2 MiB/s wr, 384 op/s 2024-06-04T05:11:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:43 smithi112 ceph-mon[36453]: pgmap v453: 97 pgs: 97 active+clean; 366 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.2 MiB/s wr, 384 op/s 2024-06-04T05:11:43.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 6903 15.77 MB/sec warmup 35 sec latency 128.142 ms 2024-06-04T05:11:44.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 6992 15.38 MB/sec warmup 36 sec latency 140.852 ms 2024-06-04T05:11:45.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:45 smithi070 ceph-mon[32631]: pgmap v454: 97 pgs: 97 active+clean; 366 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 3.9 MiB/s wr, 256 op/s 2024-06-04T05:11:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:45 smithi112 ceph-mon[36453]: pgmap v454: 97 pgs: 97 active+clean; 366 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 3.9 MiB/s wr, 256 op/s 2024-06-04T05:11:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:45 smithi098 ceph-mon[35569]: pgmap v454: 97 pgs: 97 active+clean; 366 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 3.9 MiB/s wr, 256 op/s 2024-06-04T05:11:45.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 7097 15.19 MB/sec warmup 37 sec latency 225.453 ms 2024-06-04T05:11:46.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:46 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:46.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:46 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:46.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:46 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:11:46.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 7240 15.07 MB/sec warmup 38 sec latency 167.678 ms 2024-06-04T05:11:47.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:47 smithi070 ceph-mon[32631]: pgmap v455: 97 pgs: 97 active+clean; 379 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.2 MiB/s wr, 384 op/s 2024-06-04T05:11:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:47 smithi112 ceph-mon[36453]: pgmap v455: 97 pgs: 97 active+clean; 379 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.2 MiB/s wr, 384 op/s 2024-06-04T05:11:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:47 smithi098 ceph-mon[35569]: pgmap v455: 97 pgs: 97 active+clean; 379 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.2 MiB/s wr, 384 op/s 2024-06-04T05:11:47.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 7383 14.83 MB/sec warmup 39 sec latency 198.523 ms 2024-06-04T05:11:48.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 7520 14.62 MB/sec warmup 40 sec latency 124.977 ms 2024-06-04T05:11:49.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:49 smithi070 ceph-mon[32631]: pgmap v456: 97 pgs: 97 active+clean; 379 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 4.6 MiB/s wr, 250 op/s 2024-06-04T05:11:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:49 smithi112 ceph-mon[36453]: pgmap v456: 97 pgs: 97 active+clean; 379 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 4.6 MiB/s wr, 250 op/s 2024-06-04T05:11:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:49 smithi098 ceph-mon[35569]: pgmap v456: 97 pgs: 97 active+clean; 379 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 4.6 MiB/s wr, 250 op/s 2024-06-04T05:11:49.797 INFO:tasks.workunit.client.0.smithi070.stdout: 10 7655 14.41 MB/sec warmup 41 sec latency 125.361 ms 2024-06-04T05:11:50.798 INFO:tasks.workunit.client.0.smithi070.stdout: 10 7803 14.29 MB/sec warmup 42 sec latency 138.648 ms 2024-06-04T05:11:51.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:51 smithi070 ceph-mon[32631]: pgmap v457: 97 pgs: 97 active+clean; 379 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 4.6 MiB/s wr, 250 op/s 2024-06-04T05:11:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:51 smithi112 ceph-mon[36453]: pgmap v457: 97 pgs: 97 active+clean; 379 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 4.6 MiB/s wr, 250 op/s 2024-06-04T05:11:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:51 smithi098 ceph-mon[35569]: pgmap v457: 97 pgs: 97 active+clean; 379 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 4.6 MiB/s wr, 250 op/s 2024-06-04T05:11:51.798 INFO:tasks.workunit.client.0.smithi070.stdout: 10 7921 14.00 MB/sec warmup 43 sec latency 142.547 ms 2024-06-04T05:11:52.798 INFO:tasks.workunit.client.0.smithi070.stdout: 10 8040 13.77 MB/sec warmup 44 sec latency 199.972 ms 2024-06-04T05:11:53.537 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:53 smithi070 ceph-mon[32631]: pgmap v458: 97 pgs: 97 active+clean; 380 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.4 MiB/s wr, 376 op/s 2024-06-04T05:11:53.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.607+0000 7fe048965700 1 -- 172.21.15.70:0/4099883219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0441024f0 msgr2=0x7fe0441103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:53.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.607+0000 7fe048965700 1 --2- 172.21.15.70:0/4099883219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0441024f0 0x7fe0441103d0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fe03c00b0d0 tx=0x7fe03c00b3e0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.607+0000 7fe048965700 1 -- 172.21.15.70:0/4099883219 shutdown_connections 2024-06-04T05:11:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.607+0000 7fe048965700 1 --2- 172.21.15.70:0/4099883219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0441024f0 0x7fe0441103d0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.607+0000 7fe048965700 1 --2- 172.21.15.70:0/4099883219 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe044109520 0x7fe044109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.607+0000 7fe048965700 1 --2- 172.21.15.70:0/4099883219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe044101b30 0x7fe044101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.607+0000 7fe048965700 1 -- 172.21.15.70:0/4099883219 >> 172.21.15.70:0/4099883219 conn(0x7fe044078ab0 msgr2=0x7fe0440ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.607+0000 7fe048965700 1 -- 172.21.15.70:0/4099883219 shutdown_connections 2024-06-04T05:11:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.607+0000 7fe048965700 1 -- 172.21.15.70:0/4099883219 wait complete. 2024-06-04T05:11:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.608+0000 7fe048965700 1 Processor -- start 2024-06-04T05:11:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.608+0000 7fe048965700 1 -- start start 2024-06-04T05:11:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.609+0000 7fe048965700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe044101b30 0x7fe0441a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.609+0000 7fe048965700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0441024f0 0x7fe0441a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.609+0000 7fe048965700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe044109520 0x7fe04419bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.609+0000 7fe048965700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe044113660 con 0x7fe0441024f0 2024-06-04T05:11:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.609+0000 7fe048965700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe0441134e0 con 0x7fe044101b30 2024-06-04T05:11:53.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.609+0000 7fe048965700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe0441137e0 con 0x7fe044109520 2024-06-04T05:11:53.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.609+0000 7fe042d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe044109520 0x7fe04419bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.609+0000 7fe041d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0441024f0 0x7fe0441a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.609+0000 7fe042d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe044109520 0x7fe04419bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48268/0 (socket says 172.21.15.70:48268) 2024-06-04T05:11:53.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.609+0000 7fe042d9d700 1 -- 172.21.15.70:0/832555805 learned_addr learned my addr 172.21.15.70:0/832555805 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:53.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.609+0000 7fe04259c700 1 --2- 172.21.15.70:0/832555805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe044101b30 0x7fe0441a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.609+0000 7fe042d9d700 1 -- 172.21.15.70:0/832555805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe044101b30 msgr2=0x7fe0441a1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:53.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.609+0000 7fe042d9d700 1 --2- 172.21.15.70:0/832555805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe044101b30 0x7fe0441a1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.609+0000 7fe042d9d700 1 -- 172.21.15.70:0/832555805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0441024f0 msgr2=0x7fe0441a1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:53.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.609+0000 7fe042d9d700 1 --2- 172.21.15.70:0/832555805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0441024f0 0x7fe0441a1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.609+0000 7fe042d9d700 1 -- 172.21.15.70:0/832555805 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe04419c400 con 0x7fe044109520 2024-06-04T05:11:53.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.610+0000 7fe042d9d700 1 --2- 172.21.15.70:0/832555805 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe044109520 0x7fe04419bb20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe03c0079a0 tx=0x7fe03c0079d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:53.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.610+0000 7fe0337fe700 1 -- 172.21.15.70:0/832555805 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c01f070 con 0x7fe044109520 2024-06-04T05:11:53.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.610+0000 7fe0337fe700 1 -- 172.21.15.70:0/832555805 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe03c003c00 con 0x7fe044109520 2024-06-04T05:11:53.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.610+0000 7fe048965700 1 -- 172.21.15.70:0/832555805 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe04419c690 con 0x7fe044109520 2024-06-04T05:11:53.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.610+0000 7fe048965700 1 -- 172.21.15.70:0/832555805 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0441a8800 con 0x7fe044109520 2024-06-04T05:11:53.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.610+0000 7fe0337fe700 1 -- 172.21.15.70:0/832555805 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c01b430 con 0x7fe044109520 2024-06-04T05:11:53.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.614+0000 7fe0531fab80 1 Processor -- start 2024-06-04T05:11:53.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.614+0000 7fe0337fe700 1 -- 172.21.15.70:0/832555805 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fe03c022070 con 0x7fe044109520 2024-06-04T05:11:53.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.615+0000 7fe0531fab80 1 -- start start 2024-06-04T05:11:53.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.615+0000 7fe0531fab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a8094320 0x5610a8094700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.615+0000 7fe0531fab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808e8a0 0x5610a808ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.615+0000 7fe0531fab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808f260 0x5610a809b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.615+0000 7fe0531fab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5610a7fecd70 con 0x5610a808f260 2024-06-04T05:11:53.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.615+0000 7fe0531fab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5610a7fdc910 con 0x5610a8094320 2024-06-04T05:11:53.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.615+0000 7fe0531fab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5610a7fe0950 con 0x5610a808e8a0 2024-06-04T05:11:53.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.615+0000 7fe0337fe700 1 --2- 172.21.15.70:0/832555805 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe02c0b2a70 0x7fe02c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.615+0000 7fe04259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808e8a0 0x5610a808ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.615+0000 7fe041d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808f260 0x5610a809b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.615+0000 7fe04259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808e8a0 0x5610a808ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48274/0 (socket says 172.21.15.70:48274) 2024-06-04T05:11:53.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.615+0000 7fe04259c700 1 -- 172.21.15.70:0/722309437 learned_addr learned my addr 172.21.15.70:0/722309437 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:53.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.615+0000 7fe041d9b700 1 --2- 172.21.15.70:0/722309437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a8094320 0x5610a8094700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.616+0000 7fe0337fe700 1 -- 172.21.15.70:0/832555805 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fe03c0e3360 con 0x7fe044109520 2024-06-04T05:11:53.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.616+0000 7fe042d9d700 1 --2- 172.21.15.70:0/832555805 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe02c0b2a70 0x7fe02c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.616+0000 7fe041d9b700 1 -- 172.21.15.70:0/722309437 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808e8a0 msgr2=0x5610a808ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:53.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.616+0000 7fe041d9b700 1 --2- 172.21.15.70:0/722309437 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808e8a0 0x5610a808ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.616+0000 7fe041d9b700 1 -- 172.21.15.70:0/722309437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a8094320 msgr2=0x5610a8094700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:53.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.616+0000 7fe041d9b700 1 --2- 172.21.15.70:0/722309437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a8094320 0x5610a8094700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.616+0000 7fe041d9b700 1 -- 172.21.15.70:0/722309437 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610a7f85fb0 con 0x5610a808f260 2024-06-04T05:11:53.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.616+0000 7fe041d9b700 1 --2- 172.21.15.70:0/722309437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808f260 0x5610a809b080 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fe04419d270 tx=0x7fe038009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f08eeda8af29fb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:53.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.616+0000 7fe042d9d700 1 --2- 172.21.15.70:0/832555805 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe02c0b2a70 0x7fe02c0b4f30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe03c007a40 tx=0x7fe03c006230 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:53.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.616+0000 7fe030ff9700 1 -- 172.21.15.70:0/722309437 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe038013da0 con 0x5610a808f260 2024-06-04T05:11:53.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.616+0000 7fe030ff9700 1 -- 172.21.15.70:0/722309437 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe038013170 con 0x5610a808f260 2024-06-04T05:11:53.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.616+0000 7fe030ff9700 1 -- 172.21.15.70:0/722309437 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe038005550 con 0x5610a808f260 2024-06-04T05:11:53.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.617+0000 7fe0531fab80 1 -- 172.21.15.70:0/722309437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808f260 msgr2=0x5610a809b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:53.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.617+0000 7fe0531fab80 1 --2- 172.21.15.70:0/722309437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808f260 0x5610a809b080 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fe04419d270 tx=0x7fe038009260 comp rx=0 tx=0).stop 2024-06-04T05:11:53.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.617+0000 7fe0531fab80 1 -- 172.21.15.70:0/722309437 shutdown_connections 2024-06-04T05:11:53.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.617+0000 7fe0531fab80 1 --2- 172.21.15.70:0/722309437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808f260 0x5610a809b080 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.618+0000 7fe0531fab80 1 --2- 172.21.15.70:0/722309437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a8094320 0x5610a8094700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.618+0000 7fe0531fab80 1 --2- 172.21.15.70:0/722309437 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808e8a0 0x5610a808ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.618+0000 7fe0531fab80 1 -- 172.21.15.70:0/722309437 >> 172.21.15.70:0/722309437 conn(0x5610a7fba240 msgr2=0x5610a80893e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:53.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.618+0000 7fe0531fab80 1 -- 172.21.15.70:0/722309437 shutdown_connections 2024-06-04T05:11:53.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.618+0000 7fe0531fab80 1 -- 172.21.15.70:0/722309437 wait complete. 2024-06-04T05:11:53.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.618+0000 7fe0531fab80 1 -- start start 2024-06-04T05:11:53.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.618+0000 7fe0531fab80 1 Processor -- start 2024-06-04T05:11:53.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.619+0000 7fe0531fab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808e8a0 0x5610a809ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.619+0000 7fe0531fab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808f260 0x5610a8098010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.619+0000 7fe0531fab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a8094320 0x5610a8098550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.619+0000 7fe041d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808e8a0 0x5610a809ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.619+0000 7fe0531fab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5610a7ee2fa0 con 0x5610a808e8a0 2024-06-04T05:11:53.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.619+0000 7fe041d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808e8a0 0x5610a809ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50864/0 (socket says 172.21.15.70:50864) 2024-06-04T05:11:53.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.619+0000 7fe0531fab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5610a7f44da0 con 0x5610a8094320 2024-06-04T05:11:53.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.619+0000 7fe041d9b700 1 -- 172.21.15.70:0/2103582466 learned_addr learned my addr 172.21.15.70:0/2103582466 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:53.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.619+0000 7fe0531fab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5610a809b6e0 con 0x5610a808f260 2024-06-04T05:11:53.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.619+0000 7fe0531fab80 10 client.0 _ll_register_callbacks cb 0x5610a7f77950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:53.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.619+0000 7fe04259c700 1 --2- 172.21.15.70:0/2103582466 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808f260 0x5610a8098010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.619+0000 7fe041d9b700 1 --2- 172.21.15.70:0/2103582466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a8094320 0x5610a8098550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.620+0000 7fe041d9b700 1 -- 172.21.15.70:0/2103582466 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808f260 msgr2=0x5610a8098010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:53.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.620+0000 7fe041d9b700 1 --2- 172.21.15.70:0/2103582466 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808f260 0x5610a8098010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.620+0000 7fe041d9b700 1 -- 172.21.15.70:0/2103582466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a8094320 msgr2=0x5610a8098550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:53.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.620+0000 7fe041d9b700 1 --2- 172.21.15.70:0/2103582466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a8094320 0x5610a8098550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.620+0000 7fe041d9b700 1 -- 172.21.15.70:0/2103582466 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610a7f2c8d0 con 0x5610a808e8a0 2024-06-04T05:11:53.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.620+0000 7fe041d9b700 1 --2- 172.21.15.70:0/2103582466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808e8a0 0x5610a809ad10 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fe038005800 tx=0x7fe038001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc27ae8d31e7d3c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:53.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.620+0000 7fe029ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:11:53.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.620+0000 7fe029ffb700 1 -- 172.21.15.70:0/2103582466 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03802a030 con 0x5610a808e8a0 2024-06-04T05:11:53.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.620+0000 7fe029ffb700 1 -- 172.21.15.70:0/2103582466 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fe03801c040 con 0x5610a808e8a0 2024-06-04T05:11:53.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.620+0000 7fe0531fab80 1 -- 172.21.15.70:0/2103582466 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe038013170 con 0x5610a808e8a0 2024-06-04T05:11:53.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.620+0000 7fe029ffb700 1 -- 172.21.15.70:0/2103582466 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fe038034050 con 0x5610a808e8a0 2024-06-04T05:11:53.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.620+0000 7fe0531fab80 10 client.19305 fetch_fsmap learned FSMap version 44 2024-06-04T05:11:53.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.621+0000 7fe0531fab80 1 -- 172.21.15.70:0/2103582466 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5610a7f96760 con 0x5610a808e8a0 2024-06-04T05:11:53.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.621+0000 7fe029ffb700 1 -- 172.21.15.70:0/2103582466 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe038004df0 con 0x5610a808e8a0 2024-06-04T05:11:53.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.621+0000 7fe029ffb700 1 -- 172.21.15.70:0/2103582466 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7fe038020a40 con 0x5610a808e8a0 2024-06-04T05:11:53.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.621+0000 7fe0531fab80 10 client.19305 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:11:53.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.621+0000 7fe0531fab80 10 client.19305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:53.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.621+0000 7fe0531fab80 20 client.19305 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:53.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.621+0000 7fe0531fab80 1 --2- 172.21.15.70:0/2103582466 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5610a80a8520 0x5610a80aa940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.621+0000 7fe0531fab80 4 client.19305 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:11:53.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.621+0000 7fe0531fab80 1 -- 172.21.15.70:0/2103582466 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe038013170 con 0x5610a80a8520 2024-06-04T05:11:53.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.621+0000 7fe04259c700 1 --2- 172.21.15.70:0/2103582466 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5610a80a8520 0x5610a80aa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.622+0000 7fe04259c700 1 --2- 172.21.15.70:0/2103582466 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5610a80a8520 0x5610a80aa940 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3150680866b6e1f server_cookie=8f96ee65bda138ab in_seq=0 out_seq=0 2024-06-04T05:11:53.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.622+0000 7fe029ffb700 10 client.19305 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:53.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.622+0000 7fe029ffb700 1 -- 172.21.15.70:0/2103582466 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fe038013170 con 0x5610a80a8520 2024-06-04T05:11:53.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.622+0000 7fe029ffb700 10 client.19305 handle_command_reply: tid=0 2024-06-04T05:11:53.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.622+0000 7fe0531fab80 1 client.19305 shutdown 2024-06-04T05:11:53.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.623+0000 7fe0531fab80 1 -- 172.21.15.70:0/2103582466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808e8a0 msgr2=0x5610a809ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:53.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.623+0000 7fe0531fab80 1 --2- 172.21.15.70:0/2103582466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808e8a0 0x5610a809ad10 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fe038005800 tx=0x7fe038001e70 comp rx=0 tx=0).stop 2024-06-04T05:11:53.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.623+0000 7fe0531fab80 1 -- 172.21.15.70:0/2103582466 shutdown_connections 2024-06-04T05:11:53.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.623+0000 7fe0531fab80 1 --2- 172.21.15.70:0/2103582466 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5610a80a8520 0x5610a80aa940 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.623+0000 7fe0531fab80 1 --2- 172.21.15.70:0/2103582466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a8094320 0x5610a8098550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.623+0000 7fe0531fab80 1 --2- 172.21.15.70:0/2103582466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808e8a0 0x5610a809ad10 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.623+0000 7fe0531fab80 1 --2- 172.21.15.70:0/2103582466 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808f260 0x5610a8098010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.623+0000 7fe0531fab80 1 -- 172.21.15.70:0/2103582466 >> 172.21.15.70:0/2103582466 conn(0x5610a7fba240 msgr2=0x5610a8088af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:53.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.623+0000 7fe029ffb700 0 client.19305 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:53.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.623+0000 7fe0531fab80 1 -- 172.21.15.70:0/2103582466 shutdown_connections 2024-06-04T05:11:53.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.623+0000 7fe0531fab80 1 -- 172.21.15.70:0/2103582466 wait complete. 2024-06-04T05:11:53.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.623+0000 7fe0531fab80 20 client.19305 trim_cache size 0 max 16384 2024-06-04T05:11:53.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.659+0000 7fe0531fab80 1 Processor -- start 2024-06-04T05:11:53.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.659+0000 7fe0531fab80 1 -- start start 2024-06-04T05:11:53.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.659+0000 7fe0531fab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808e8a0 0x5610a80934b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.659+0000 7fe0531fab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808f260 0x5610a80985a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.659+0000 7fe0531fab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a8094320 0x5610a8089270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.659+0000 7fe0531fab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe02400d6a0 con 0x5610a808f260 2024-06-04T05:11:53.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.659+0000 7fe0531fab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe02400f360 con 0x5610a8094320 2024-06-04T05:11:53.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.659+0000 7fe0531fab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe02400dd70 con 0x5610a808e8a0 2024-06-04T05:11:53.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.659+0000 7fe04259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808f260 0x5610a80985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.659+0000 7fe041d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808e8a0 0x5610a80934b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.659+0000 7fe04259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808f260 0x5610a80985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50870/0 (socket says 172.21.15.70:50870) 2024-06-04T05:11:53.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.659+0000 7fe04259c700 1 -- 172.21.15.70:0/3546940117 learned_addr learned my addr 172.21.15.70:0/3546940117 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:53.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.659+0000 7fe041d9b700 1 --2- 172.21.15.70:0/3546940117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a8094320 0x5610a8089270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.659+0000 7fe041d9b700 1 -- 172.21.15.70:0/3546940117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a8094320 msgr2=0x5610a8089270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:53.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.659+0000 7fe041d9b700 1 --2- 172.21.15.70:0/3546940117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a8094320 0x5610a8089270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.659+0000 7fe041d9b700 1 -- 172.21.15.70:0/3546940117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808f260 msgr2=0x5610a80985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:53.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.660+0000 7fe041d9b700 1 --2- 172.21.15.70:0/3546940117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808f260 0x5610a80985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.660+0000 7fe041d9b700 1 -- 172.21.15.70:0/3546940117 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610a808ab30 con 0x5610a808e8a0 2024-06-04T05:11:53.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.660+0000 7fe04259c700 1 --2- 172.21.15.70:0/3546940117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808f260 0x5610a80985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:53.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.660+0000 7fe041d9b700 1 --2- 172.21.15.70:0/3546940117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808e8a0 0x5610a80934b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe038009b20 tx=0x7fe038023040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4aece72fe2c9b754 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:53.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.660+0000 7fe02bfff700 1 -- 172.21.15.70:0/3546940117 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03802a040 con 0x5610a808e8a0 2024-06-04T05:11:53.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.660+0000 7fe02bfff700 1 -- 172.21.15.70:0/3546940117 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe038013170 con 0x5610a808e8a0 2024-06-04T05:11:53.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.660+0000 7fe02bfff700 1 -- 172.21.15.70:0/3546940117 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe038018040 con 0x5610a808e8a0 2024-06-04T05:11:53.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.661+0000 7fe0531fab80 1 -- 172.21.15.70:0/3546940117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808e8a0 msgr2=0x5610a80934b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:53.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.661+0000 7fe0531fab80 1 --2- 172.21.15.70:0/3546940117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808e8a0 0x5610a80934b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe038009b20 tx=0x7fe038023040 comp rx=0 tx=0).stop 2024-06-04T05:11:53.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.661+0000 7fe0531fab80 1 -- 172.21.15.70:0/3546940117 shutdown_connections 2024-06-04T05:11:53.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.661+0000 7fe0531fab80 1 --2- 172.21.15.70:0/3546940117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a8094320 0x5610a8089270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.661+0000 7fe0531fab80 1 --2- 172.21.15.70:0/3546940117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808e8a0 0x5610a80934b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.661+0000 7fe0531fab80 1 --2- 172.21.15.70:0/3546940117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a808f260 0x5610a80985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.661+0000 7fe0531fab80 1 -- 172.21.15.70:0/3546940117 >> 172.21.15.70:0/3546940117 conn(0x5610a7fba240 msgr2=0x5610a808e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:53.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.661+0000 7fe0531fab80 1 -- 172.21.15.70:0/3546940117 shutdown_connections 2024-06-04T05:11:53.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.661+0000 7fe0531fab80 1 -- 172.21.15.70:0/3546940117 wait complete. 2024-06-04T05:11:53.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.661+0000 7fe0531fab80 1 -- start start 2024-06-04T05:11:53.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe0531fab80 1 Processor -- start 2024-06-04T05:11:53.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe0531fab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808e8a0 0x5610a80a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe0531fab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a808f260 0x5610a80a1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe0531fab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a8094320 0x5610a80afa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe0531fab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe02400ebd0 con 0x5610a8094320 2024-06-04T05:11:53.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe0531fab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe02400e4a0 con 0x5610a808f260 2024-06-04T05:11:53.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe0531fab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe02400b660 con 0x5610a808e8a0 2024-06-04T05:11:53.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe0531fab80 10 client.0 _ll_register_callbacks cb 0x5610a7f77950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:53.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe041d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a8094320 0x5610a80afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe041d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a8094320 0x5610a80afa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50886/0 (socket says 172.21.15.70:50886) 2024-06-04T05:11:53.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe04259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a808f260 0x5610a80a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe041d9b700 1 -- 172.21.15.70:0/1814645132 learned_addr learned my addr 172.21.15.70:0/1814645132 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:53.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe041d9b700 1 --2- 172.21.15.70:0/1814645132 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808e8a0 0x5610a80a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe041d9b700 1 -- 172.21.15.70:0/1814645132 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808e8a0 msgr2=0x5610a80a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:53.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe041d9b700 1 --2- 172.21.15.70:0/1814645132 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808e8a0 0x5610a80a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe041d9b700 1 -- 172.21.15.70:0/1814645132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a808f260 msgr2=0x5610a80a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:53.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe041d9b700 1 --2- 172.21.15.70:0/1814645132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a808f260 0x5610a80a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.662+0000 7fe041d9b700 1 -- 172.21.15.70:0/1814645132 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610a80b0200 con 0x5610a8094320 2024-06-04T05:11:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.663+0000 7fe04259c700 1 --2- 172.21.15.70:0/1814645132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a808f260 0x5610a80a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.663+0000 7fe041d9b700 1 --2- 172.21.15.70:0/1814645132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a8094320 0x5610a80afa00 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x5610a8089ac0 tx=0x7fe038001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f6e036101b365f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.663+0000 7fe029ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:11:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.663+0000 7fe029ffb700 1 -- 172.21.15.70:0/1814645132 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03802a070 con 0x5610a8094320 2024-06-04T05:11:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.663+0000 7fe029ffb700 1 -- 172.21.15.70:0/1814645132 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fe03801c040 con 0x5610a8094320 2024-06-04T05:11:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.663+0000 7fe0531fab80 1 -- 172.21.15.70:0/1814645132 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe038013170 con 0x5610a8094320 2024-06-04T05:11:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.664+0000 7fe029ffb700 1 -- 172.21.15.70:0/1814645132 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fe038005520 con 0x5610a8094320 2024-06-04T05:11:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.664+0000 7fe0531fab80 10 client.19311 fetch_fsmap learned FSMap version 44 2024-06-04T05:11:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.664+0000 7fe0531fab80 1 -- 172.21.15.70:0/1814645132 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5610a80a0650 con 0x5610a8094320 2024-06-04T05:11:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.664+0000 7fe029ffb700 1 -- 172.21.15.70:0/1814645132 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe038009bd0 con 0x5610a8094320 2024-06-04T05:11:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.664+0000 7fe029ffb700 1 -- 172.21.15.70:0/1814645132 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7fe038020460 con 0x5610a8094320 2024-06-04T05:11:53.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.664+0000 7fe0531fab80 10 client.19311 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:11:53.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.664+0000 7fe0531fab80 10 client.19311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:53.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.664+0000 7fe0531fab80 20 client.19311 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:53.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.664+0000 7fe0531fab80 1 --2- 172.21.15.70:0/1814645132 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5610a81133c0 0x5610a81157e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:53.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.664+0000 7fe0531fab80 4 client.19311 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:11:53.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.664+0000 7fe0531fab80 1 -- 172.21.15.70:0/1814645132 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fe038013170 con 0x5610a81133c0 2024-06-04T05:11:53.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.665+0000 7fe04259c700 1 --2- 172.21.15.70:0/1814645132 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5610a81133c0 0x5610a81157e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:53.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.665+0000 7fe04259c700 1 --2- 172.21.15.70:0/1814645132 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5610a81133c0 0x5610a81157e0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5054319c4df1e83 server_cookie=58ace17d195d86b0 in_seq=0 out_seq=0 2024-06-04T05:11:53.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.665+0000 7fe029ffb700 10 client.19311 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.668+0000 7fe029ffb700 1 -- 172.21.15.70:0/1814645132 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fe038013170 con 0x5610a81133c0 2024-06-04T05:11:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.668+0000 7fe029ffb700 10 client.19311 handle_command_reply: tid=0 2024-06-04T05:11:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.668+0000 7fe0531fab80 1 client.19311 shutdown 2024-06-04T05:11:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.668+0000 7fe0531fab80 1 -- 172.21.15.70:0/1814645132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a8094320 msgr2=0x5610a80afa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.668+0000 7fe0531fab80 1 --2- 172.21.15.70:0/1814645132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a8094320 0x5610a80afa00 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x5610a8089ac0 tx=0x7fe038001e70 comp rx=0 tx=0).stop 2024-06-04T05:11:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.668+0000 7fe0531fab80 1 -- 172.21.15.70:0/1814645132 shutdown_connections 2024-06-04T05:11:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.668+0000 7fe0531fab80 1 --2- 172.21.15.70:0/1814645132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610a8094320 0x5610a80afa00 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.668+0000 7fe0531fab80 1 --2- 172.21.15.70:0/1814645132 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5610a808e8a0 0x5610a80a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.668+0000 7fe0531fab80 1 --2- 172.21.15.70:0/1814645132 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5610a81133c0 0x5610a81157e0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.668+0000 7fe0531fab80 1 --2- 172.21.15.70:0/1814645132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610a808f260 0x5610a80a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.668+0000 7fe029ffb700 0 client.19311 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.668+0000 7fe0531fab80 1 -- 172.21.15.70:0/1814645132 >> 172.21.15.70:0/1814645132 conn(0x5610a7fba240 msgr2=0x5610a8091ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.668+0000 7fe0531fab80 1 -- 172.21.15.70:0/1814645132 shutdown_connections 2024-06-04T05:11:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.668+0000 7fe0531fab80 1 -- 172.21.15.70:0/1814645132 wait complete. 2024-06-04T05:11:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.668+0000 7fe0531fab80 20 client.19311 trim_cache size 0 max 16384 2024-06-04T05:11:53.673 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:11:53.673 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:11:53.673 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:11:53.673 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:11:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.669+0000 7fe048965700 1 -- 172.21.15.70:0/832555805 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe02c0b2a70 msgr2=0x7fe02c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.669+0000 7fe048965700 1 --2- 172.21.15.70:0/832555805 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe02c0b2a70 0x7fe02c0b4f30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe03c007a40 tx=0x7fe03c006230 comp rx=0 tx=0).stop 2024-06-04T05:11:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.669+0000 7fe048965700 1 -- 172.21.15.70:0/832555805 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe044109520 msgr2=0x7fe04419bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.669+0000 7fe048965700 1 --2- 172.21.15.70:0/832555805 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe044109520 0x7fe04419bb20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe03c0079a0 tx=0x7fe03c0079d0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.669+0000 7fe048965700 1 -- 172.21.15.70:0/832555805 shutdown_connections 2024-06-04T05:11:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.669+0000 7fe048965700 1 --2- 172.21.15.70:0/832555805 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe02c0b2a70 0x7fe02c0b4f30 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.669+0000 7fe048965700 1 --2- 172.21.15.70:0/832555805 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe044109520 0x7fe04419bb20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.669+0000 7fe048965700 1 --2- 172.21.15.70:0/832555805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe044101b30 0x7fe0441a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.669+0000 7fe048965700 1 --2- 172.21.15.70:0/832555805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0441024f0 0x7fe0441a1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.669+0000 7fe048965700 1 -- 172.21.15.70:0/832555805 >> 172.21.15.70:0/832555805 conn(0x7fe044078ab0 msgr2=0x7fe044100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.669+0000 7fe048965700 1 -- 172.21.15.70:0/832555805 shutdown_connections 2024-06-04T05:11:53.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:53.669+0000 7fe048965700 1 -- 172.21.15.70:0/832555805 wait complete. 2024-06-04T05:11:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:53 smithi112 ceph-mon[36453]: pgmap v458: 97 pgs: 97 active+clean; 380 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.4 MiB/s wr, 376 op/s 2024-06-04T05:11:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:53 smithi098 ceph-mon[35569]: pgmap v458: 97 pgs: 97 active+clean; 380 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.4 MiB/s wr, 376 op/s 2024-06-04T05:11:53.798 INFO:tasks.workunit.client.0.smithi070.stdout: 10 8221 13.65 MB/sec warmup 45 sec latency 83.805 ms 2024-06-04T05:11:54.521 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:11:54.521 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:11:54.798 INFO:tasks.workunit.client.0.smithi070.stdout: 10 8388 13.48 MB/sec warmup 46 sec latency 60.188 ms 2024-06-04T05:11:54.876 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:11:55.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:55 smithi070 ceph-mon[32631]: pgmap v459: 97 pgs: 97 active+clean; 380 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 4.1 MiB/s wr, 254 op/s 2024-06-04T05:11:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:55 smithi112 ceph-mon[36453]: pgmap v459: 97 pgs: 97 active+clean; 380 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 4.1 MiB/s wr, 254 op/s 2024-06-04T05:11:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.730+0000 7fa8d4ef8640 1 -- 172.21.15.70:0/3575082397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8d0107940 msgr2=0x7fa8d0107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.730+0000 7fa8d4ef8640 1 --2- 172.21.15.70:0/3575082397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8d0107940 0x7fa8d0107d20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009900 tx=0x7fa8c002a410 comp rx=0 tx=0).stop 2024-06-04T05:11:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.730+0000 7fa8d4ef8640 1 -- 172.21.15.70:0/3575082397 shutdown_connections 2024-06-04T05:11:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.730+0000 7fa8d4ef8640 1 --2- 172.21.15.70:0/3575082397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8d0102340 0x7fa8d010e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.730+0000 7fa8d4ef8640 1 --2- 172.21.15.70:0/3575082397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8d0101980 0x7fa8d0101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.730+0000 7fa8d4ef8640 1 --2- 172.21.15.70:0/3575082397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8d0107940 0x7fa8d0107d20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.730+0000 7fa8d4ef8640 1 -- 172.21.15.70:0/3575082397 >> 172.21.15.70:0/3575082397 conn(0x7fa8d0078ad0 msgr2=0x7fa8d0078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.730+0000 7fa8d4ef8640 1 -- 172.21.15.70:0/3575082397 shutdown_connections 2024-06-04T05:11:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.731+0000 7fa8d4ef8640 1 -- 172.21.15.70:0/3575082397 wait complete. 2024-06-04T05:11:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.731+0000 7fa8d4ef8640 1 Processor -- start 2024-06-04T05:11:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.731+0000 7fa8d4ef8640 1 -- start start 2024-06-04T05:11:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8d4ef8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8d0101980 0x7fa8d019b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8d4ef8640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8d0102340 0x7fa8d019b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8d4ef8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8d0107940 0x7fa8d019fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8d4ef8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d0114d80 con 0x7fa8d0107940 2024-06-04T05:11:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8d4ef8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d0114c00 con 0x7fa8d0101980 2024-06-04T05:11:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8d4ef8640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d0114f00 con 0x7fa8d0102340 2024-06-04T05:11:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8ce59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8d0101980 0x7fa8d019b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8cdd9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8d0102340 0x7fa8d019b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8ced9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8d0107940 0x7fa8d019fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8cdd9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8d0102340 0x7fa8d019b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48328/0 (socket says 172.21.15.70:48328) 2024-06-04T05:11:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8cdd9b640 1 -- 172.21.15.70:0/4134646399 learned_addr learned my addr 172.21.15.70:0/4134646399 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8cdd9b640 1 -- 172.21.15.70:0/4134646399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8d0101980 msgr2=0x7fa8d019b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8cdd9b640 1 --2- 172.21.15.70:0/4134646399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8d0101980 0x7fa8d019b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8cdd9b640 1 -- 172.21.15.70:0/4134646399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8d0107940 msgr2=0x7fa8d019fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8cdd9b640 1 --2- 172.21.15.70:0/4134646399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8d0107940 0x7fa8d019fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8cdd9b640 1 -- 172.21.15.70:0/4134646399 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8d01a0270 con 0x7fa8d0102340 2024-06-04T05:11:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8ced9d640 1 --2- 172.21.15.70:0/4134646399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8d0107940 0x7fa8d019fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.732+0000 7fa8ce59c640 1 --2- 172.21.15.70:0/4134646399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8d0101980 0x7fa8d019b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:11:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.733+0000 7fa8cdd9b640 1 --2- 172.21.15.70:0/4134646399 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8d0102340 0x7fa8d019b7c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa8c400e9f0 tx=0x7fa8c400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.733+0000 7fa8bf7fe640 1 -- 172.21.15.70:0/4134646399 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c400cd80 con 0x7fa8d0102340 2024-06-04T05:11:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.733+0000 7fa8bf7fe640 1 -- 172.21.15.70:0/4134646399 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa8c40108c0 con 0x7fa8d0102340 2024-06-04T05:11:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.733+0000 7fa8d4ef8640 1 -- 172.21.15.70:0/4134646399 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8d01a0560 con 0x7fa8d0102340 2024-06-04T05:11:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.733+0000 7fa8d4ef8640 1 -- 172.21.15.70:0/4134646399 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8d01a7df0 con 0x7fa8d0102340 2024-06-04T05:11:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.733+0000 7fa8bf7fe640 1 -- 172.21.15.70:0/4134646399 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c40187c0 con 0x7fa8d0102340 2024-06-04T05:11:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.736+0000 7fa8d7c0d740 1 Processor -- start 2024-06-04T05:11:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.736+0000 7fa8bf7fe640 1 -- 172.21.15.70:0/4134646399 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fa8c4018a40 con 0x7fa8d0102340 2024-06-04T05:11:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.736+0000 7fa8d7c0d740 1 -- start start 2024-06-04T05:11:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.736+0000 7fa8d7c0d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56533178c820 0x56533178cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.736+0000 7fa8d7c0d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178d140 0x5653317974c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.736+0000 7fa8d7c0d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565331797a00 0x565331799df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.737+0000 7fa8d7c0d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5653316fec30 con 0x565331797a00 2024-06-04T05:11:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.737+0000 7fa8d7c0d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5653316ee7d0 con 0x56533178c820 2024-06-04T05:11:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.737+0000 7fa8d7c0d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5653316f2810 con 0x56533178d140 2024-06-04T05:11:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.737+0000 7fa8cdd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565331797a00 0x565331799df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.737+0000 7fa8cdd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565331797a00 0x565331799df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50928/0 (socket says 172.21.15.70:50928) 2024-06-04T05:11:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.737+0000 7fa8cdd9b640 1 -- 172.21.15.70:0/945883384 learned_addr learned my addr 172.21.15.70:0/945883384 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.737+0000 7fa8ced9d640 1 --2- 172.21.15.70:0/945883384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56533178c820 0x56533178cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.737+0000 7fa8ce59c640 1 --2- 172.21.15.70:0/945883384 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178d140 0x5653317974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.737+0000 7fa8ced9d640 1 -- 172.21.15.70:0/945883384 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178d140 msgr2=0x5653317974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.737+0000 7fa8ced9d640 1 --2- 172.21.15.70:0/945883384 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178d140 0x5653317974c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.737+0000 7fa8ced9d640 1 -- 172.21.15.70:0/945883384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565331797a00 msgr2=0x565331799df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.737+0000 7fa8ced9d640 1 --2- 172.21.15.70:0/945883384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565331797a00 0x565331799df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.737+0000 7fa8ced9d640 1 -- 172.21.15.70:0/945883384 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565331403210 con 0x56533178c820 2024-06-04T05:11:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.737+0000 7fa8cdd9b640 1 --2- 172.21.15.70:0/945883384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565331797a00 0x565331799df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.737+0000 7fa8bf7fe640 1 --2- 172.21.15.70:0/4134646399 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa8b80b2a80 0x7fa8b80b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.737+0000 7fa8ce59c640 1 --2- 172.21.15.70:0/945883384 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178d140 0x5653317974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:11:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.737+0000 7fa8ced9d640 1 --2- 172.21.15.70:0/945883384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56533178c820 0x56533178cc00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8005fd0 tx=0x7fa8c8011bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d61ad9ffa177d3a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.738+0000 7fa8ce59c640 1 --2- 172.21.15.70:0/4134646399 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa8b80b2a80 0x7fa8b80b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.738+0000 7fa8bcff9640 1 -- 172.21.15.70:0/945883384 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8004d20 con 0x56533178c820 2024-06-04T05:11:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.738+0000 7fa8bf7fe640 1 -- 172.21.15.70:0/4134646399 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fa8c40e2990 con 0x7fa8d0102340 2024-06-04T05:11:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.738+0000 7fa8bcff9640 1 -- 172.21.15.70:0/945883384 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa8c800f9a0 con 0x56533178c820 2024-06-04T05:11:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.738+0000 7fa8bcff9640 1 -- 172.21.15.70:0/945883384 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c80043d0 con 0x56533178c820 2024-06-04T05:11:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.738+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/945883384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56533178c820 msgr2=0x56533178cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.738+0000 7fa8ce59c640 1 --2- 172.21.15.70:0/4134646399 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa8b80b2a80 0x7fa8b80b4f40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0000c00 tx=0x7fa8c0005de0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.738+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/945883384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56533178c820 0x56533178cc00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8005fd0 tx=0x7fa8c8011bf0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.739+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/945883384 shutdown_connections 2024-06-04T05:11:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.739+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/945883384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565331797a00 0x565331799df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.739+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/945883384 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178d140 0x5653317974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.739+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/945883384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56533178c820 0x56533178cc00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.739+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/945883384 >> 172.21.15.70:0/945883384 conn(0x565331788d30 msgr2=0x565331789110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.739+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/945883384 shutdown_connections 2024-06-04T05:11:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.739+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/945883384 wait complete. 2024-06-04T05:11:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.739+0000 7fa8d7c0d740 1 -- start start 2024-06-04T05:11:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.739+0000 7fa8d7c0d740 1 Processor -- start 2024-06-04T05:11:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.739+0000 7fa8d7c0d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178c820 0x5653317949d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.739+0000 7fa8d7c0d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56533178d140 0x565331794f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.739+0000 7fa8d7c0d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565331797a00 0x565331795450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8cdd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56533178d140 0x565331794f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8d7c0d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5653313d4790 con 0x56533178d140 2024-06-04T05:11:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8d7c0d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5653316aa890 con 0x565331797a00 2024-06-04T05:11:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8ce59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178c820 0x5653317949d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8ce59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178c820 0x5653317949d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48344/0 (socket says 172.21.15.70:48344) 2024-06-04T05:11:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8ce59c640 1 -- 172.21.15.70:0/3885769258 learned_addr learned my addr 172.21.15.70:0/3885769258 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8d7c0d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56533179a630 con 0x56533178c820 2024-06-04T05:11:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8d7c0d740 10 client.0 _ll_register_callbacks cb 0x565331402a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8ced9d640 1 --2- 172.21.15.70:0/3885769258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565331797a00 0x565331795450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8ce59c640 1 -- 172.21.15.70:0/3885769258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565331797a00 msgr2=0x565331795450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8ce59c640 1 --2- 172.21.15.70:0/3885769258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565331797a00 0x565331795450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8ce59c640 1 -- 172.21.15.70:0/3885769258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56533178d140 msgr2=0x565331794f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8ce59c640 1 --2- 172.21.15.70:0/3885769258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56533178d140 0x565331794f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8ce59c640 1 -- 172.21.15.70:0/3885769258 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653314cf620 con 0x56533178c820 2024-06-04T05:11:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8ced9d640 1 --2- 172.21.15.70:0/3885769258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565331797a00 0x565331795450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8ce59c640 1 --2- 172.21.15.70:0/3885769258 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178c820 0x5653317949d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0005490 tx=0x7fa8c002a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c8884111b0c8713 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:11:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8b5ffb640 1 -- 172.21.15.70:0/3885769258 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c004c020 con 0x56533178c820 2024-06-04T05:11:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.740+0000 7fa8b5ffb640 1 -- 172.21.15.70:0/3885769258 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fa8c0004af0 con 0x56533178c820 2024-06-04T05:11:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.741+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/3885769258 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8c800f9a0 con 0x56533178c820 2024-06-04T05:11:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.741+0000 7fa8b5ffb640 1 -- 172.21.15.70:0/3885769258 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fa8c0057050 con 0x56533178c820 2024-06-04T05:11:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.741+0000 7fa8d7c0d740 10 client.28315 fetch_fsmap learned FSMap version 44 2024-06-04T05:11:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.741+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/3885769258 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5653317a6340 con 0x56533178c820 2024-06-04T05:11:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.741+0000 7fa8b5ffb640 1 -- 172.21.15.70:0/3885769258 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa8c800f9a0 con 0x56533178c820 2024-06-04T05:11:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.741+0000 7fa8b5ffb640 1 -- 172.21.15.70:0/3885769258 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7fa8c0003c80 con 0x56533178c820 2024-06-04T05:11:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.741+0000 7fa8d7c0d740 10 client.28315 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:11:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.741+0000 7fa8d7c0d740 10 client.28315 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.741+0000 7fa8d7c0d740 20 client.28315 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.741+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/3885769258 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5653317a7940 0x5653317a9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.741+0000 7fa8d7c0d740 4 client.28315 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:11:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.741+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/3885769258 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565331700b80 con 0x5653317a7940 2024-06-04T05:11:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.741+0000 7fa8cdd9b640 1 --2- 172.21.15.70:0/3885769258 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5653317a7940 0x5653317a9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.742+0000 7fa8b5ffb640 10 client.28315 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.742+0000 7fa8cdd9b640 1 --2- 172.21.15.70:0/3885769258 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5653317a7940 0x5653317a9dc0 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9d2cfc497446819 server_cookie=a6d660aa6d71e91c in_seq=0 out_seq=0 2024-06-04T05:11:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.742+0000 7fa8b5ffb640 1 -- 172.21.15.70:0/3885769258 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x565331700b80 con 0x5653317a7940 2024-06-04T05:11:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.742+0000 7fa8b5ffb640 10 client.28315 handle_command_reply: tid=0 2024-06-04T05:11:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.742+0000 7fa8d7c0d740 1 client.28315 shutdown 2024-06-04T05:11:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.743+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/3885769258 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178c820 msgr2=0x5653317949d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.743+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/3885769258 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178c820 0x5653317949d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0005490 tx=0x7fa8c002a410 comp rx=0 tx=0).stop 2024-06-04T05:11:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.743+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/3885769258 shutdown_connections 2024-06-04T05:11:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.743+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/3885769258 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5653317a7940 0x5653317a9dc0 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.743+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/3885769258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565331797a00 0x565331795450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.743+0000 7fa8b5ffb640 0 client.28315 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.743+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/3885769258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56533178d140 0x565331794f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.743+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/3885769258 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178c820 0x5653317949d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.743+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/3885769258 >> 172.21.15.70:0/3885769258 conn(0x565331788d30 msgr2=0x565331787fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.743+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/3885769258 shutdown_connections 2024-06-04T05:11:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.743+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/3885769258 wait complete. 2024-06-04T05:11:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.743+0000 7fa8d7c0d740 20 client.28315 trim_cache size 0 max 16384 2024-06-04T05:11:55.751 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:55 smithi098 ceph-mon[35569]: pgmap v459: 97 pgs: 97 active+clean; 380 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 4.1 MiB/s wr, 254 op/s 2024-06-04T05:11:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.772+0000 7fa8d7c0d740 1 Processor -- start 2024-06-04T05:11:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.772+0000 7fa8d7c0d740 1 -- start start 2024-06-04T05:11:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.772+0000 7fa8d7c0d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56533178c820 0x5653317af690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.772+0000 7fa8d7c0d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178d140 0x5653317afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.772+0000 7fa8d7c0d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565331797a00 0x56533179f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.772+0000 7fa8d7c0d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d01b0950 con 0x56533178c820 2024-06-04T05:11:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.772+0000 7fa8d7c0d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d01b2610 con 0x565331797a00 2024-06-04T05:11:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.772+0000 7fa8d7c0d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d01b1020 con 0x56533178d140 2024-06-04T05:11:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.772+0000 7fa8ce59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56533178c820 0x5653317af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.772+0000 7fa8ce59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56533178c820 0x5653317af690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50958/0 (socket says 172.21.15.70:50958) 2024-06-04T05:11:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.773+0000 7fa8ce59c640 1 -- 172.21.15.70:0/3506869281 learned_addr learned my addr 172.21.15.70:0/3506869281 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.773+0000 7fa8cdd9b640 1 --2- 172.21.15.70:0/3506869281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178d140 0x5653317afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.773+0000 7fa8ced9d640 1 --2- 172.21.15.70:0/3506869281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565331797a00 0x56533179f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.773+0000 7fa8ce59c640 1 -- 172.21.15.70:0/3506869281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178d140 msgr2=0x5653317afbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.773+0000 7fa8ce59c640 1 --2- 172.21.15.70:0/3506869281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178d140 0x5653317afbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.773+0000 7fa8ce59c640 1 -- 172.21.15.70:0/3506869281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565331797a00 msgr2=0x56533179f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.773+0000 7fa8ce59c640 1 --2- 172.21.15.70:0/3506869281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565331797a00 0x56533179f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.773+0000 7fa8ce59c640 1 -- 172.21.15.70:0/3506869281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56533178cf70 con 0x56533178c820 2024-06-04T05:11:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.773+0000 7fa8cdd9b640 1 --2- 172.21.15.70:0/3506869281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178d140 0x5653317afbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.773+0000 7fa8ce59c640 1 --2- 172.21.15.70:0/3506869281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56533178c820 0x5653317af690 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fa8c002a930 tx=0x7fa8c0048840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5a8707af463b34 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.773+0000 7fa8b7fff640 1 -- 172.21.15.70:0/3506869281 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c004c070 con 0x56533178c820 2024-06-04T05:11:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.773+0000 7fa8b7fff640 1 -- 172.21.15.70:0/3506869281 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa8c0004360 con 0x56533178c820 2024-06-04T05:11:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.773+0000 7fa8b7fff640 1 -- 172.21.15.70:0/3506869281 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0035050 con 0x56533178c820 2024-06-04T05:11:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.774+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/3506869281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56533178c820 msgr2=0x5653317af690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.774+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/3506869281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56533178c820 0x5653317af690 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fa8c002a930 tx=0x7fa8c0048840 comp rx=0 tx=0).stop 2024-06-04T05:11:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.774+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/3506869281 shutdown_connections 2024-06-04T05:11:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.774+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/3506869281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565331797a00 0x56533179f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.774+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/3506869281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178d140 0x5653317afbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.774+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/3506869281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56533178c820 0x5653317af690 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.774+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/3506869281 >> 172.21.15.70:0/3506869281 conn(0x565331788d30 msgr2=0x56533178ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.774+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/3506869281 shutdown_connections 2024-06-04T05:11:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.774+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/3506869281 wait complete. 2024-06-04T05:11:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.775+0000 7fa8d7c0d740 1 -- start start 2024-06-04T05:11:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.775+0000 7fa8d7c0d740 1 Processor -- start 2024-06-04T05:11:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.775+0000 7fa8d7c0d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178c820 0x56533179fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.775+0000 7fa8d7c0d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56533178d140 0x5653317a0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.775+0000 7fa8d7c0d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565331797a00 0x56533179d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.775+0000 7fa8d7c0d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d01b1e80 con 0x565331797a00 2024-06-04T05:11:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.775+0000 7fa8d7c0d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d01b1750 con 0x56533178d140 2024-06-04T05:11:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.775+0000 7fa8d7c0d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d01ae950 con 0x56533178c820 2024-06-04T05:11:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.775+0000 7fa8d7c0d740 10 client.0 _ll_register_callbacks cb 0x7fa8c003f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:11:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.775+0000 7fa8ce59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178c820 0x56533179fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.775+0000 7fa8ced9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565331797a00 0x56533179d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.775+0000 7fa8ce59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178c820 0x56533179fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48362/0 (socket says 172.21.15.70:48362) 2024-06-04T05:11:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.775+0000 7fa8ce59c640 1 -- 172.21.15.70:0/1453056719 learned_addr learned my addr 172.21.15.70:0/1453056719 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.775+0000 7fa8cdd9b640 1 --2- 172.21.15.70:0/1453056719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56533178d140 0x5653317a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.776+0000 7fa8ced9d640 1 -- 172.21.15.70:0/1453056719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178c820 msgr2=0x56533179fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.776+0000 7fa8ced9d640 1 --2- 172.21.15.70:0/1453056719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178c820 0x56533179fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.776+0000 7fa8ced9d640 1 -- 172.21.15.70:0/1453056719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56533178d140 msgr2=0x5653317a0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:11:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.776+0000 7fa8ced9d640 1 --2- 172.21.15.70:0/1453056719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56533178d140 0x5653317a0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.776+0000 7fa8ced9d640 1 -- 172.21.15.70:0/1453056719 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56533179d940 con 0x565331797a00 2024-06-04T05:11:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.776+0000 7fa8ced9d640 1 --2- 172.21.15.70:0/1453056719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565331797a00 0x56533179d140 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800f790 tx=0x7fa8c8011530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eebb33a24d8cdb7d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.776+0000 7fa8b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:11:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.776+0000 7fa8b5ffb640 1 -- 172.21.15.70:0/1453056719 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8029020 con 0x565331797a00 2024-06-04T05:11:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.776+0000 7fa8b5ffb640 1 -- 172.21.15.70:0/1453056719 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fa8c8005050 con 0x565331797a00 2024-06-04T05:11:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.776+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/1453056719 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8c0004360 con 0x565331797a00 2024-06-04T05:11:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.776+0000 7fa8b5ffb640 1 -- 172.21.15.70:0/1453056719 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fa8c8034050 con 0x565331797a00 2024-06-04T05:11:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.776+0000 7fa8d7c0d740 10 client.19329 fetch_fsmap learned FSMap version 44 2024-06-04T05:11:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.776+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/1453056719 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5653318121e0 con 0x565331797a00 2024-06-04T05:11:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.776+0000 7fa8b5ffb640 1 -- 172.21.15.70:0/1453056719 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa8c0004360 con 0x565331797a00 2024-06-04T05:11:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.777+0000 7fa8b5ffb640 1 -- 172.21.15.70:0/1453056719 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7fa8c8018070 con 0x565331797a00 2024-06-04T05:11:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.777+0000 7fa8d7c0d740 10 client.19329 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:11:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.777+0000 7fa8d7c0d740 10 client.19329 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:11:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.777+0000 7fa8d7c0d740 20 client.19329 populate_metadata read hostname 'smithi070' 2024-06-04T05:11:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.777+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/1453056719 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5653317a7940 0x5653318148a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.777+0000 7fa8d7c0d740 4 client.19329 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:11:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.777+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/1453056719 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5653316fa360 con 0x5653317a7940 2024-06-04T05:11:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.777+0000 7fa8ce59c640 1 --2- 172.21.15.70:0/1453056719 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5653317a7940 0x5653318148a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.777+0000 7fa8b5ffb640 10 client.19329 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.777+0000 7fa8ce59c640 1 --2- 172.21.15.70:0/1453056719 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5653317a7940 0x5653318148a0 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd5e219e086cf4d1 server_cookie=ff7ae22fbb47ca12 in_seq=0 out_seq=0 2024-06-04T05:11:55.798 INFO:tasks.workunit.client.0.smithi070.stdout: 10 8560 13.33 MB/sec warmup 47 sec latency 73.164 ms 2024-06-04T05:11:55.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.847+0000 7fa8b5ffb640 1 -- 172.21.15.70:0/1453056719 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5653316fa360 con 0x5653317a7940 2024-06-04T05:11:55.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.847+0000 7fa8b5ffb640 10 client.19329 handle_command_reply: tid=0 2024-06-04T05:11:55.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.847+0000 7fa8d7c0d740 1 client.19329 shutdown 2024-06-04T05:11:55.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.847+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/1453056719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565331797a00 msgr2=0x56533179d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:55.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.847+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/1453056719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565331797a00 0x56533179d140 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800f790 tx=0x7fa8c8011530 comp rx=0 tx=0).stop 2024-06-04T05:11:55.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.847+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/1453056719 shutdown_connections 2024-06-04T05:11:55.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.847+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/1453056719 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5653317a7940 0x5653318148a0 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.848+0000 7fa8b5ffb640 0 client.19329 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:11:55.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.848+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/1453056719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56533178d140 0x5653317a0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.848+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/1453056719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565331797a00 0x56533179d140 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.848+0000 7fa8d7c0d740 1 --2- 172.21.15.70:0/1453056719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56533178c820 0x56533179fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.848+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/1453056719 >> 172.21.15.70:0/1453056719 conn(0x565331788d30 msgr2=0x565331787720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:55.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.848+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/1453056719 shutdown_connections 2024-06-04T05:11:55.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.848+0000 7fa8d7c0d740 1 -- 172.21.15.70:0/1453056719 wait complete. 2024-06-04T05:11:55.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.848+0000 7fa8d7c0d740 20 client.19329 trim_cache size 0 max 16384 2024-06-04T05:11:55.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:11:55.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:11:55.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:11:55.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.849+0000 7fa8d4ef8640 1 -- 172.21.15.70:0/4134646399 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa8b80b2a80 msgr2=0x7fa8b80b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:55.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.849+0000 7fa8d4ef8640 1 --2- 172.21.15.70:0/4134646399 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa8b80b2a80 0x7fa8b80b4f40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0000c00 tx=0x7fa8c0005de0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.849+0000 7fa8d4ef8640 1 -- 172.21.15.70:0/4134646399 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8d0102340 msgr2=0x7fa8d019b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:55.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.849+0000 7fa8d4ef8640 1 --2- 172.21.15.70:0/4134646399 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8d0102340 0x7fa8d019b7c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa8c400e9f0 tx=0x7fa8c400eeb0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.849+0000 7fa8d4ef8640 1 -- 172.21.15.70:0/4134646399 shutdown_connections 2024-06-04T05:11:55.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.849+0000 7fa8d4ef8640 1 --2- 172.21.15.70:0/4134646399 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa8b80b2a80 0x7fa8b80b4f40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.849+0000 7fa8d4ef8640 1 --2- 172.21.15.70:0/4134646399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8d0107940 0x7fa8d019fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.849+0000 7fa8d4ef8640 1 --2- 172.21.15.70:0/4134646399 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8d0102340 0x7fa8d019b7c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.849+0000 7fa8d4ef8640 1 --2- 172.21.15.70:0/4134646399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8d0101980 0x7fa8d019b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:55.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.849+0000 7fa8d4ef8640 1 -- 172.21.15.70:0/4134646399 >> 172.21.15.70:0/4134646399 conn(0x7fa8d0078ad0 msgr2=0x7fa8d0104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:55.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.849+0000 7fa8d4ef8640 1 -- 172.21.15.70:0/4134646399 shutdown_connections 2024-06-04T05:11:55.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:11:55.849+0000 7fa8d4ef8640 1 -- 172.21.15.70:0/4134646399 wait complete. 2024-06-04T05:11:56.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.750+0000 7f7a54e09700 1 -- 172.21.15.70:0/3550052748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a50102e10 msgr2=0x7f7a5010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:56.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.750+0000 7f7a54e09700 1 --2- 172.21.15.70:0/3550052748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a50102e10 0x7f7a5010f2f0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f7a4800b0a0 tx=0x7f7a4800b3b0 comp rx=0 tx=0).stop 2024-06-04T05:11:56.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.750+0000 7f7a54e09700 1 -- 172.21.15.70:0/3550052748 shutdown_connections 2024-06-04T05:11:56.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.750+0000 7f7a54e09700 1 --2- 172.21.15.70:0/3550052748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a50102e10 0x7f7a5010f2f0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:56.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.750+0000 7f7a54e09700 1 --2- 172.21.15.70:0/3550052748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a501084b0 0x7f7a50108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:56.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.750+0000 7f7a54e09700 1 --2- 172.21.15.70:0/3550052748 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a50102470 0x7f7a501028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:56.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.750+0000 7f7a54e09700 1 -- 172.21.15.70:0/3550052748 >> 172.21.15.70:0/3550052748 conn(0x7f7a500793e0 msgr2=0x7f7a500797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:56.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.751+0000 7f7a54e09700 1 -- 172.21.15.70:0/3550052748 shutdown_connections 2024-06-04T05:11:56.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.751+0000 7f7a54e09700 1 -- 172.21.15.70:0/3550052748 wait complete. 2024-06-04T05:11:56.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.752+0000 7f7a54e09700 1 Processor -- start 2024-06-04T05:11:56.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.752+0000 7f7a54e09700 1 -- start start 2024-06-04T05:11:56.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.753+0000 7f7a54e09700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a50102470 0x7f7a501178f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:56.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.753+0000 7f7a54e09700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a50102e10 0x7f7a50117e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:56.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.753+0000 7f7a54e09700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a501084b0 0x7f7a50112a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:56.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.753+0000 7f7a54e09700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7a50111370 con 0x7f7a50102470 2024-06-04T05:11:56.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.753+0000 7f7a54e09700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a501111f0 con 0x7f7a50102e10 2024-06-04T05:11:56.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.753+0000 7f7a54e09700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7a501114f0 con 0x7f7a501084b0 2024-06-04T05:11:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.753+0000 7f7a4dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a50102e10 0x7f7a50117e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.753+0000 7f7a4e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a50102470 0x7f7a501178f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.753+0000 7f7a4e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a50102470 0x7f7a501178f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44626/0 (socket says 172.21.15.70:44626) 2024-06-04T05:11:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.753+0000 7f7a4dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a50102e10 0x7f7a50117e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:36268/0 (socket says 172.21.15.70:36268) 2024-06-04T05:11:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.753+0000 7f7a4e59c700 1 -- 172.21.15.70:0/322647740 learned_addr learned my addr 172.21.15.70:0/322647740 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:11:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.753+0000 7f7a4ed9d700 1 --2- 172.21.15.70:0/322647740 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a501084b0 0x7f7a50112a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:56.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.753+0000 7f7a4dd9b700 1 -- 172.21.15.70:0/322647740 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a501084b0 msgr2=0x7f7a50112a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:56.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.754+0000 7f7a4dd9b700 1 --2- 172.21.15.70:0/322647740 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a501084b0 0x7f7a50112a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:56.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.754+0000 7f7a4dd9b700 1 -- 172.21.15.70:0/322647740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a50102470 msgr2=0x7f7a501178f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:56.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.754+0000 7f7a4dd9b700 1 --2- 172.21.15.70:0/322647740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a50102470 0x7f7a501178f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:56.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.754+0000 7f7a4dd9b700 1 -- 172.21.15.70:0/322647740 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a50113360 con 0x7f7a50102e10 2024-06-04T05:11:56.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.754+0000 7f7a4e59c700 1 --2- 172.21.15.70:0/322647740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a50102470 0x7f7a501178f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.754+0000 7f7a4ed9d700 1 --2- 172.21.15.70:0/322647740 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a501084b0 0x7f7a50112a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:11:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.754+0000 7f7a4dd9b700 1 --2- 172.21.15.70:0/322647740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a50102e10 0x7f7a50117e30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400e9f0 tx=0x7f7a4400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.754+0000 7f7a3f7fe700 1 -- 172.21.15.70:0/322647740 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4400c9c0 con 0x7f7a50102e10 2024-06-04T05:11:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.754+0000 7f7a3f7fe700 1 -- 172.21.15.70:0/322647740 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7a44010430 con 0x7f7a50102e10 2024-06-04T05:11:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.754+0000 7f7a3f7fe700 1 -- 172.21.15.70:0/322647740 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a44018680 con 0x7f7a50102e10 2024-06-04T05:11:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.754+0000 7f7a54e09700 1 -- 172.21.15.70:0/322647740 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a501135f0 con 0x7f7a50102e10 2024-06-04T05:11:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.754+0000 7f7a54e09700 1 -- 172.21.15.70:0/322647740 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a5011d7e0 con 0x7f7a50102e10 2024-06-04T05:11:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.756+0000 7f7a54e09700 1 -- 172.21.15.70:0/322647740 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a50103b60 con 0x7f7a50102e10 2024-06-04T05:11:56.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.757+0000 7f7a3f7fe700 1 -- 172.21.15.70:0/322647740 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7a4400cb60 con 0x7f7a50102e10 2024-06-04T05:11:56.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.758+0000 7f7a3f7fe700 1 --2- 172.21.15.70:0/322647740 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7a380b2b40 0x7f7a380b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:11:56.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.759+0000 7f7a4e59c700 1 --2- 172.21.15.70:0/322647740 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7a380b2b40 0x7f7a380b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:11:56.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.759+0000 7f7a3f7fe700 1 -- 172.21.15.70:0/322647740 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f7a440e2660 con 0x7f7a50102e10 2024-06-04T05:11:56.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.759+0000 7f7a4e59c700 1 --2- 172.21.15.70:0/322647740 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7a380b2b40 0x7f7a380b5000 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7a40011550 tx=0x7f7a40009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:11:56.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:56.763+0000 7f7a3f7fe700 1 -- 172.21.15.70:0/322647740 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7a440aa010 con 0x7f7a50102e10 2024-06-04T05:11:56.798 INFO:tasks.workunit.client.0.smithi070.stdout: 10 8725 13.27 MB/sec warmup 48 sec latency 76.952 ms 2024-06-04T05:11:57.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:57.182+0000 7f7a54e09700 1 -- 172.21.15.70:0/322647740 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a5006a260 con 0x7f7a50102e10 2024-06-04T05:11:57.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:57.183+0000 7f7a3f7fe700 1 -- 172.21.15.70:0/322647740 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11949 (secure 0 0 0) 0x7f7a5006a260 con 0x7f7a50102e10 2024-06-04T05:11:57.185 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:11:57.185 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":44,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:11:44.309654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:11:57.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:57.186+0000 7f7a54e09700 1 -- 172.21.15.70:0/322647740 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7a380b2b40 msgr2=0x7f7a380b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:57.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:57.186+0000 7f7a54e09700 1 --2- 172.21.15.70:0/322647740 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7a380b2b40 0x7f7a380b5000 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7a40011550 tx=0x7f7a40009260 comp rx=0 tx=0).stop 2024-06-04T05:11:57.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:57.186+0000 7f7a54e09700 1 -- 172.21.15.70:0/322647740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a50102e10 msgr2=0x7f7a50117e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:11:57.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:57.186+0000 7f7a54e09700 1 --2- 172.21.15.70:0/322647740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a50102e10 0x7f7a50117e30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400e9f0 tx=0x7f7a4400ed00 comp rx=0 tx=0).stop 2024-06-04T05:11:57.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:57.186+0000 7f7a54e09700 1 -- 172.21.15.70:0/322647740 shutdown_connections 2024-06-04T05:11:57.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:57.186+0000 7f7a54e09700 1 --2- 172.21.15.70:0/322647740 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7a380b2b40 0x7f7a380b5000 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:57.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:57.186+0000 7f7a54e09700 1 --2- 172.21.15.70:0/322647740 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a501084b0 0x7f7a50112a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:57.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:57.186+0000 7f7a54e09700 1 --2- 172.21.15.70:0/322647740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a50102470 0x7f7a501178f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:57.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:57.186+0000 7f7a54e09700 1 --2- 172.21.15.70:0/322647740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a50102e10 0x7f7a50117e30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:11:57.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:57.186+0000 7f7a54e09700 1 -- 172.21.15.70:0/322647740 >> 172.21.15.70:0/322647740 conn(0x7f7a500793e0 msgr2=0x7f7a50100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:11:57.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:57.186+0000 7f7a54e09700 1 -- 172.21.15.70:0/322647740 shutdown_connections 2024-06-04T05:11:57.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:11:57.186+0000 7f7a54e09700 1 -- 172.21.15.70:0/322647740 wait complete. 2024-06-04T05:11:57.189 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 44 2024-06-04T05:11:57.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:57 smithi070 ceph-mon[32631]: pgmap v460: 97 pgs: 97 active+clean; 368 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 409 op/s 2024-06-04T05:11:57.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:57 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/322647740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:11:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:57 smithi112 ceph-mon[36453]: pgmap v460: 97 pgs: 97 active+clean; 368 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 409 op/s 2024-06-04T05:11:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:57 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/322647740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:11:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:57 smithi098 ceph-mon[35569]: pgmap v460: 97 pgs: 97 active+clean; 368 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 409 op/s 2024-06-04T05:11:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:57 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/322647740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:11:57.798 INFO:tasks.workunit.client.0.smithi070.stdout: 10 8904 13.19 MB/sec warmup 49 sec latency 58.414 ms 2024-06-04T05:11:58.156 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:11:58.508 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:11:58.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 9116 13.17 MB/sec warmup 50 sec latency 59.341 ms 2024-06-04T05:11:59.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:11:59 smithi070 ceph-mon[32631]: pgmap v461: 97 pgs: 97 active+clean; 368 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 3.2 MiB/s wr, 281 op/s 2024-06-04T05:11:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:11:59 smithi112 ceph-mon[36453]: pgmap v461: 97 pgs: 97 active+clean; 368 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 3.2 MiB/s wr, 281 op/s 2024-06-04T05:11:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:11:59 smithi098 ceph-mon[35569]: pgmap v461: 97 pgs: 97 active+clean; 368 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 3.2 MiB/s wr, 281 op/s 2024-06-04T05:11:59.798 INFO:tasks.workunit.client.0.smithi070.stdout: 10 9337 13.09 MB/sec warmup 51 sec latency 77.870 ms 2024-06-04T05:12:00.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.382+0000 7fc35d0a8700 1 -- 172.21.15.70:0/3488729423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3581095a0 msgr2=0x7fc358109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:00.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.382+0000 7fc35d0a8700 1 --2- 172.21.15.70:0/3488729423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3581095a0 0x7fc358109980 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fc348009990 tx=0x7fc348009ca0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.382+0000 7fc35d0a8700 1 -- 172.21.15.70:0/3488729423 shutdown_connections 2024-06-04T05:12:00.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.382+0000 7fc35d0a8700 1 --2- 172.21.15.70:0/3488729423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc35806c470 0x7fc3581103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.382+0000 7fc35d0a8700 1 --2- 172.21.15.70:0/3488729423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3581095a0 0x7fc358109980 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.382+0000 7fc35d0a8700 1 --2- 172.21.15.70:0/3488729423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc35806bab0 0x7fc35806bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.382+0000 7fc35d0a8700 1 -- 172.21.15.70:0/3488729423 >> 172.21.15.70:0/3488729423 conn(0x7fc3580793b0 msgr2=0x7fc3580797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:00.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.382+0000 7fc35d0a8700 1 -- 172.21.15.70:0/3488729423 shutdown_connections 2024-06-04T05:12:00.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.382+0000 7fc35d0a8700 1 -- 172.21.15.70:0/3488729423 wait complete. 2024-06-04T05:12:00.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.383+0000 7fc35d0a8700 1 Processor -- start 2024-06-04T05:12:00.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.383+0000 7fc35d0a8700 1 -- start start 2024-06-04T05:12:00.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.383+0000 7fc35d0a8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc35806bab0 0x7fc35819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.383+0000 7fc35d0a8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc35806c470 0x7fc35819fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.383+0000 7fc35d0a8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3581095a0 0x7fc358199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.383+0000 7fc35d0a8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc358113560 con 0x7fc3581095a0 2024-06-04T05:12:00.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.383+0000 7fc35d0a8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc3581133e0 con 0x7fc35806bab0 2024-06-04T05:12:00.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.383+0000 7fc35d0a8700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc3581136e0 con 0x7fc35806c470 2024-06-04T05:12:00.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.383+0000 7fc35759e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3581095a0 0x7fc358199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.383+0000 7fc34ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc35806c470 0x7fc35819fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.383+0000 7fc356d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc35806bab0 0x7fc35819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.383+0000 7fc35759e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3581095a0 0x7fc358199940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44632/0 (socket says 172.21.15.70:44632) 2024-06-04T05:12:00.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.383+0000 7fc35759e700 1 -- 172.21.15.70:0/61217822 learned_addr learned my addr 172.21.15.70:0/61217822 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:00.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.383+0000 7fc34ffff700 1 --2- 172.21.15.70:0/61217822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc35806c470 0x7fc35819fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:52204/0 (socket says 172.21.15.70:52204) 2024-06-04T05:12:00.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.384+0000 7fc356d9d700 1 -- 172.21.15.70:0/61217822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc35806c470 msgr2=0x7fc35819fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:00.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.384+0000 7fc356d9d700 1 --2- 172.21.15.70:0/61217822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc35806c470 0x7fc35819fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.384+0000 7fc356d9d700 1 -- 172.21.15.70:0/61217822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3581095a0 msgr2=0x7fc358199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:00.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.384+0000 7fc356d9d700 1 --2- 172.21.15.70:0/61217822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3581095a0 0x7fc358199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.384+0000 7fc356d9d700 1 -- 172.21.15.70:0/61217822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35819a0a0 con 0x7fc35806bab0 2024-06-04T05:12:00.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.384+0000 7fc35759e700 1 --2- 172.21.15.70:0/61217822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3581095a0 0x7fc358199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:00.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.384+0000 7fc356d9d700 1 --2- 172.21.15.70:0/61217822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc35806bab0 0x7fc35819f780 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc348000c00 tx=0x7fc3480047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:00.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.384+0000 7fc354d99700 1 -- 172.21.15.70:0/61217822 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc348026070 con 0x7fc35806bab0 2024-06-04T05:12:00.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.384+0000 7fc35d0a8700 1 -- 172.21.15.70:0/61217822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc35819a330 con 0x7fc35806bab0 2024-06-04T05:12:00.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.384+0000 7fc354d99700 1 -- 172.21.15.70:0/61217822 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc348021410 con 0x7fc35806bab0 2024-06-04T05:12:00.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.384+0000 7fc354d99700 1 -- 172.21.15.70:0/61217822 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc348017650 con 0x7fc35806bab0 2024-06-04T05:12:00.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.384+0000 7fc35d0a8700 1 -- 172.21.15.70:0/61217822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3581a6650 con 0x7fc35806bab0 2024-06-04T05:12:00.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.387+0000 7fc354d99700 1 -- 172.21.15.70:0/61217822 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fc348029070 con 0x7fc35806bab0 2024-06-04T05:12:00.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.388+0000 7fc354d99700 1 --2- 172.21.15.70:0/61217822 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc3440b29f0 0x7fc3440b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.388+0000 7fc34ffff700 1 --2- 172.21.15.70:0/61217822 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc3440b29f0 0x7fc3440b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.388+0000 7fc354d99700 1 -- 172.21.15.70:0/61217822 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fc3480e3420 con 0x7fc35806bab0 2024-06-04T05:12:00.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.389+0000 7fc36793db80 1 Processor -- start 2024-06-04T05:12:00.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.389+0000 7fc36793db80 1 -- start start 2024-06-04T05:12:00.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.389+0000 7fc34ffff700 1 --2- 172.21.15.70:0/61217822 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc3440b29f0 0x7fc3440b4eb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc358079af0 tx=0x7fc340009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:00.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.389+0000 7fc36793db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf6860 0x564187cf6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.389+0000 7fc36793db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cf7210 0x564187cf9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.389+0000 7fc36793db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cfa090 0x564187cfa510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.389+0000 7fc35759e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cfa090 0x564187cfa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.389+0000 7fc34ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cf7210 0x564187cf9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.389+0000 7fc35759e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cfa090 0x564187cfa510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44638/0 (socket says 172.21.15.70:44638) 2024-06-04T05:12:00.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.389+0000 7fc35759e700 1 -- 172.21.15.70:0/4094487746 learned_addr learned my addr 172.21.15.70:0/4094487746 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:00.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.389+0000 7fc356d9d700 1 --2- 172.21.15.70:0/4094487746 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf6860 0x564187cf6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.389+0000 7fc36793db80 1 -- 172.21.15.70:0/4094487746 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564187c55c90 con 0x564187cfa090 2024-06-04T05:12:00.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.389+0000 7fc36793db80 1 -- 172.21.15.70:0/4094487746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564187c45830 con 0x564187cf7210 2024-06-04T05:12:00.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.389+0000 7fc36793db80 1 -- 172.21.15.70:0/4094487746 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564187c49870 con 0x564187cf6860 2024-06-04T05:12:00.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.390+0000 7fc34ffff700 1 -- 172.21.15.70:0/4094487746 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf6860 msgr2=0x564187cf6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:00.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.390+0000 7fc34ffff700 1 --2- 172.21.15.70:0/4094487746 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf6860 0x564187cf6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.390+0000 7fc34ffff700 1 -- 172.21.15.70:0/4094487746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cfa090 msgr2=0x564187cfa510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:00.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.390+0000 7fc34ffff700 1 --2- 172.21.15.70:0/4094487746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cfa090 0x564187cfa510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.390+0000 7fc34ffff700 1 -- 172.21.15.70:0/4094487746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564187b958d0 con 0x564187cf7210 2024-06-04T05:12:00.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.390+0000 7fc35759e700 1 --2- 172.21.15.70:0/4094487746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cfa090 0x564187cfa510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:00.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.390+0000 7fc34ffff700 1 --2- 172.21.15.70:0/4094487746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cf7210 0x564187cf9a70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc340011c90 tx=0x7fc340014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=964abdd0b0bf371f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:00.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.391+0000 7fc34d7fa700 1 -- 172.21.15.70:0/4094487746 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc340004230 con 0x564187cf7210 2024-06-04T05:12:00.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.391+0000 7fc34d7fa700 1 -- 172.21.15.70:0/4094487746 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc3400043d0 con 0x564187cf7210 2024-06-04T05:12:00.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.391+0000 7fc34d7fa700 1 -- 172.21.15.70:0/4094487746 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc340003ca0 con 0x564187cf7210 2024-06-04T05:12:00.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.393+0000 7fc36793db80 1 -- 172.21.15.70:0/4094487746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cf7210 msgr2=0x564187cf9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:00.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.393+0000 7fc36793db80 1 --2- 172.21.15.70:0/4094487746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cf7210 0x564187cf9a70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc340011c90 tx=0x7fc340014000 comp rx=0 tx=0).stop 2024-06-04T05:12:00.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.393+0000 7fc36793db80 1 -- 172.21.15.70:0/4094487746 shutdown_connections 2024-06-04T05:12:00.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.393+0000 7fc36793db80 1 --2- 172.21.15.70:0/4094487746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cfa090 0x564187cfa510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.393+0000 7fc36793db80 1 --2- 172.21.15.70:0/4094487746 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf6860 0x564187cf6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.393+0000 7fc36793db80 1 --2- 172.21.15.70:0/4094487746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cf7210 0x564187cf9a70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.393+0000 7fc36793db80 1 -- 172.21.15.70:0/4094487746 >> 172.21.15.70:0/4094487746 conn(0x564187c24080 msgr2=0x564187cf0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:00.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.393+0000 7fc36793db80 1 -- 172.21.15.70:0/4094487746 shutdown_connections 2024-06-04T05:12:00.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.393+0000 7fc36793db80 1 -- 172.21.15.70:0/4094487746 wait complete. 2024-06-04T05:12:00.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.394+0000 7fc36793db80 1 -- start start 2024-06-04T05:12:00.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.394+0000 7fc36793db80 1 Processor -- start 2024-06-04T05:12:00.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.394+0000 7fc36793db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf6860 0x564187d0ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.394+0000 7fc36793db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf7210 0x564187d07f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.394+0000 7fc36793db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cfa090 0x564187d084d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.394+0000 7fc36793db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564187b4bfa0 con 0x564187cf6860 2024-06-04T05:12:00.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc36793db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564187be77c0 con 0x564187cfa090 2024-06-04T05:12:00.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc36793db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564187d0b9c0 con 0x564187cf7210 2024-06-04T05:12:00.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc356d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf6860 0x564187d0ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc35759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cfa090 0x564187d084d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc356d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf6860 0x564187d0ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44646/0 (socket says 172.21.15.70:44646) 2024-06-04T05:12:00.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc35759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cfa090 0x564187d084d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:36308/0 (socket says 172.21.15.70:36308) 2024-06-04T05:12:00.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc356d9d700 1 -- 172.21.15.70:0/3210708803 learned_addr learned my addr 172.21.15.70:0/3210708803 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:00.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc36793db80 10 client.0 _ll_register_callbacks cb 0x564187be0920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:00.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc34ffff700 1 --2- 172.21.15.70:0/3210708803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf7210 0x564187d07f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc35759e700 1 -- 172.21.15.70:0/3210708803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf7210 msgr2=0x564187d07f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:00.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc35759e700 1 --2- 172.21.15.70:0/3210708803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf7210 0x564187d07f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc35759e700 1 -- 172.21.15.70:0/3210708803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf6860 msgr2=0x564187d0ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:00.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc35759e700 1 --2- 172.21.15.70:0/3210708803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf6860 0x564187d0ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc35759e700 1 -- 172.21.15.70:0/3210708803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564187be4690 con 0x564187cfa090 2024-06-04T05:12:00.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc356d9d700 1 --2- 172.21.15.70:0/3210708803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf6860 0x564187d0ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:00.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc34ffff700 1 --2- 172.21.15.70:0/3210708803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf7210 0x564187d07f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:12:00.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc35759e700 1 --2- 172.21.15.70:0/3210708803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cfa090 0x564187d084d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc350007fd0 tx=0x7fc35000e7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=846b34f2a317b8f8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:00.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc33e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:12:00.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc33e7fc700 1 -- 172.21.15.70:0/3210708803 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc350020610 con 0x564187cfa090 2024-06-04T05:12:00.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc33e7fc700 1 -- 172.21.15.70:0/3210708803 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fc3500207b0 con 0x564187cfa090 2024-06-04T05:12:00.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.395+0000 7fc36793db80 1 -- 172.21.15.70:0/3210708803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3400043d0 con 0x564187cfa090 2024-06-04T05:12:00.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.396+0000 7fc33e7fc700 1 -- 172.21.15.70:0/3210708803 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fc3500209b0 con 0x564187cfa090 2024-06-04T05:12:00.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.396+0000 7fc36793db80 10 client.28211 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:00.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.396+0000 7fc36793db80 1 -- 172.21.15.70:0/3210708803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x564187d16180 con 0x564187cfa090 2024-06-04T05:12:00.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.396+0000 7fc33e7fc700 1 -- 172.21.15.70:0/3210708803 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc3400043d0 con 0x564187cfa090 2024-06-04T05:12:00.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.396+0000 7fc33e7fc700 1 -- 172.21.15.70:0/3210708803 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7fc35001f3c0 con 0x564187cfa090 2024-06-04T05:12:00.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.396+0000 7fc36793db80 10 client.28211 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:00.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.396+0000 7fc36793db80 10 client.28211 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:12:00.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.396+0000 7fc36793db80 20 client.28211 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:00.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.396+0000 7fc36793db80 1 --2- 172.21.15.70:0/3210708803 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x564187d18860 0x564187d1ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.396+0000 7fc36793db80 4 client.28211 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:12:00.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.396+0000 7fc36793db80 1 -- 172.21.15.70:0/3210708803 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564187cc29e0 con 0x564187d18860 2024-06-04T05:12:00.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.396+0000 7fc356d9d700 1 --2- 172.21.15.70:0/3210708803 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x564187d18860 0x564187d1ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.397+0000 7fc356d9d700 1 --2- 172.21.15.70:0/3210708803 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x564187d18860 0x564187d1ac80 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=eba8335e30ebb708 server_cookie=4b10af8e3ca9bca4 in_seq=0 out_seq=0 2024-06-04T05:12:00.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.397+0000 7fc33e7fc700 10 client.28211 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:12:00.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.398+0000 7fc33e7fc700 1 -- 172.21.15.70:0/3210708803 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564187cc29e0 con 0x564187d18860 2024-06-04T05:12:00.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.398+0000 7fc33e7fc700 10 client.28211 handle_command_reply: tid=0 2024-06-04T05:12:00.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.398+0000 7fc36793db80 1 client.28211 shutdown 2024-06-04T05:12:00.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.399+0000 7fc36793db80 1 -- 172.21.15.70:0/3210708803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cfa090 msgr2=0x564187d084d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:00.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.399+0000 7fc36793db80 1 --2- 172.21.15.70:0/3210708803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cfa090 0x564187d084d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc350007fd0 tx=0x7fc35000e7d0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.399+0000 7fc36793db80 1 -- 172.21.15.70:0/3210708803 shutdown_connections 2024-06-04T05:12:00.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.399+0000 7fc36793db80 1 --2- 172.21.15.70:0/3210708803 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x564187d18860 0x564187d1ac80 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.399+0000 7fc33e7fc700 0 client.28211 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:12:00.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.399+0000 7fc36793db80 1 --2- 172.21.15.70:0/3210708803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cfa090 0x564187d084d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.399+0000 7fc36793db80 1 --2- 172.21.15.70:0/3210708803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf6860 0x564187d0ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.399+0000 7fc36793db80 1 --2- 172.21.15.70:0/3210708803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf7210 0x564187d07f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.399+0000 7fc36793db80 1 -- 172.21.15.70:0/3210708803 >> 172.21.15.70:0/3210708803 conn(0x564187c24080 msgr2=0x564187cf0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:00.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.399+0000 7fc36793db80 1 -- 172.21.15.70:0/3210708803 shutdown_connections 2024-06-04T05:12:00.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.399+0000 7fc36793db80 1 -- 172.21.15.70:0/3210708803 wait complete. 2024-06-04T05:12:00.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.399+0000 7fc36793db80 20 client.28211 trim_cache size 0 max 16384 2024-06-04T05:12:00.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.425+0000 7fc36793db80 1 Processor -- start 2024-06-04T05:12:00.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.425+0000 7fc36793db80 1 -- start start 2024-06-04T05:12:00.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.425+0000 7fc36793db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf6860 0x564187d08660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.425+0000 7fc36793db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf7210 0x564187cf0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.425+0000 7fc36793db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cfa090 0x564187cf11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.425+0000 7fc36793db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc3581abb20 con 0x564187cf6860 2024-06-04T05:12:00.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.425+0000 7fc36793db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc3581ad7e0 con 0x564187cfa090 2024-06-04T05:12:00.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.425+0000 7fc36793db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc3581ac1f0 con 0x564187cf7210 2024-06-04T05:12:00.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.426+0000 7fc34ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf7210 0x564187cf0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.426+0000 7fc35759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cfa090 0x564187cf11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.426+0000 7fc34ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf7210 0x564187cf0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:52232/0 (socket says 172.21.15.70:52232) 2024-06-04T05:12:00.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.426+0000 7fc356d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf6860 0x564187d08660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.426+0000 7fc34ffff700 1 -- 172.21.15.70:0/1165703909 learned_addr learned my addr 172.21.15.70:0/1165703909 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:00.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.426+0000 7fc35759e700 1 -- 172.21.15.70:0/1165703909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf7210 msgr2=0x564187cf0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:00.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.426+0000 7fc35759e700 1 --2- 172.21.15.70:0/1165703909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf7210 0x564187cf0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.426+0000 7fc35759e700 1 -- 172.21.15.70:0/1165703909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf6860 msgr2=0x564187d08660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:00.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.426+0000 7fc35759e700 1 --2- 172.21.15.70:0/1165703909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf6860 0x564187d08660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.426+0000 7fc35759e700 1 -- 172.21.15.70:0/1165703909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564187cf1710 con 0x564187cfa090 2024-06-04T05:12:00.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.426+0000 7fc34ffff700 1 --2- 172.21.15.70:0/1165703909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf7210 0x564187cf0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:00.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.426+0000 7fc356d9d700 1 --2- 172.21.15.70:0/1165703909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf6860 0x564187d08660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:00.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.426+0000 7fc35759e700 1 --2- 172.21.15.70:0/1165703909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cfa090 0x564187cf11d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc350000f80 tx=0x7fc350009100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=819d671d83463786 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:00.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.427+0000 7fc34cff9700 1 -- 172.21.15.70:0/1165703909 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc350035070 con 0x564187cfa090 2024-06-04T05:12:00.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.427+0000 7fc34cff9700 1 -- 172.21.15.70:0/1165703909 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc35001f5c0 con 0x564187cfa090 2024-06-04T05:12:00.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.427+0000 7fc34cff9700 1 -- 172.21.15.70:0/1165703909 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc35002c040 con 0x564187cfa090 2024-06-04T05:12:00.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.427+0000 7fc36793db80 1 -- 172.21.15.70:0/1165703909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cfa090 msgr2=0x564187cf11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:00.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.427+0000 7fc36793db80 1 --2- 172.21.15.70:0/1165703909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cfa090 0x564187cf11d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc350000f80 tx=0x7fc350009100 comp rx=0 tx=0).stop 2024-06-04T05:12:00.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.428+0000 7fc36793db80 1 -- 172.21.15.70:0/1165703909 shutdown_connections 2024-06-04T05:12:00.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.428+0000 7fc36793db80 1 --2- 172.21.15.70:0/1165703909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cfa090 0x564187cf11d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.428+0000 7fc36793db80 1 --2- 172.21.15.70:0/1165703909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf6860 0x564187d08660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.428+0000 7fc36793db80 1 --2- 172.21.15.70:0/1165703909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cf7210 0x564187cf0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.428+0000 7fc36793db80 1 -- 172.21.15.70:0/1165703909 >> 172.21.15.70:0/1165703909 conn(0x564187c24080 msgr2=0x564187d07f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:00.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.428+0000 7fc36793db80 1 -- 172.21.15.70:0/1165703909 shutdown_connections 2024-06-04T05:12:00.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.428+0000 7fc36793db80 1 -- 172.21.15.70:0/1165703909 wait complete. 2024-06-04T05:12:00.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.428+0000 7fc36793db80 1 -- start start 2024-06-04T05:12:00.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.428+0000 7fc36793db80 1 Processor -- start 2024-06-04T05:12:00.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.429+0000 7fc36793db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cf6860 0x564187d11200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.429+0000 7fc36793db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf7210 0x564187d0c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.429+0000 7fc356d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cf6860 0x564187d11200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.429+0000 7fc356d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cf6860 0x564187d11200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:36332/0 (socket says 172.21.15.70:36332) 2024-06-04T05:12:00.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.429+0000 7fc36793db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cfa090 0x564187d0cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.429+0000 7fc356d9d700 1 -- 172.21.15.70:0/355366000 learned_addr learned my addr 172.21.15.70:0/355366000 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:00.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.429+0000 7fc36793db80 1 -- 172.21.15.70:0/355366000 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc3581ad050 con 0x564187cf7210 2024-06-04T05:12:00.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.429+0000 7fc34ffff700 1 --2- 172.21.15.70:0/355366000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf7210 0x564187d0c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.429+0000 7fc36793db80 1 -- 172.21.15.70:0/355366000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc3581ac920 con 0x564187cf6860 2024-06-04T05:12:00.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.429+0000 7fc36793db80 1 -- 172.21.15.70:0/355366000 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc3581a9ae0 con 0x564187cfa090 2024-06-04T05:12:00.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.429+0000 7fc36793db80 10 client.0 _ll_register_callbacks cb 0x564187be0920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:00.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.429+0000 7fc35759e700 1 --2- 172.21.15.70:0/355366000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cfa090 0x564187d0cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.429+0000 7fc35759e700 1 -- 172.21.15.70:0/355366000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cf6860 msgr2=0x564187d11200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:00.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.429+0000 7fc35759e700 1 --2- 172.21.15.70:0/355366000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cf6860 0x564187d11200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.429+0000 7fc35759e700 1 -- 172.21.15.70:0/355366000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf7210 msgr2=0x564187d0c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:00.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.429+0000 7fc35759e700 1 --2- 172.21.15.70:0/355366000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf7210 0x564187d0c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.429+0000 7fc35759e700 1 -- 172.21.15.70:0/355366000 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564187d0d1c0 con 0x564187cfa090 2024-06-04T05:12:00.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.430+0000 7fc35759e700 1 --2- 172.21.15.70:0/355366000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cfa090 0x564187d0cac0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x564187d0baf0 tx=0x7fc35001fe10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aff20b3db7ba79f1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:00.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.430+0000 7fc33e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:12:00.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.430+0000 7fc33e7fc700 1 -- 172.21.15.70:0/355366000 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc350035030 con 0x564187cfa090 2024-06-04T05:12:00.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.430+0000 7fc33e7fc700 1 -- 172.21.15.70:0/355366000 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fc350020620 con 0x564187cfa090 2024-06-04T05:12:00.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.430+0000 7fc36793db80 1 -- 172.21.15.70:0/355366000 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564187c57be0 con 0x564187cfa090 2024-06-04T05:12:00.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.430+0000 7fc33e7fc700 1 -- 172.21.15.70:0/355366000 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fc35003f050 con 0x564187cfa090 2024-06-04T05:12:00.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.430+0000 7fc36793db80 10 client.28342 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:00.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.430+0000 7fc36793db80 1 -- 172.21.15.70:0/355366000 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x564187d83330 con 0x564187cfa090 2024-06-04T05:12:00.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.431+0000 7fc33e7fc700 1 -- 172.21.15.70:0/355366000 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x564187c57be0 con 0x564187cfa090 2024-06-04T05:12:00.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.431+0000 7fc33e7fc700 1 -- 172.21.15.70:0/355366000 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7fc350030790 con 0x564187cfa090 2024-06-04T05:12:00.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.431+0000 7fc36793db80 10 client.28342 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.431+0000 7fc36793db80 10 client.28342 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:12:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.431+0000 7fc36793db80 20 client.28342 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.431+0000 7fc36793db80 1 --2- 172.21.15.70:0/355366000 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x564187d18860 0x564187d85a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.431+0000 7fc36793db80 4 client.28342 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:12:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.431+0000 7fc36793db80 1 -- 172.21.15.70:0/355366000 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564187c513c0 con 0x564187d18860 2024-06-04T05:12:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.431+0000 7fc356d9d700 1 --2- 172.21.15.70:0/355366000 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x564187d18860 0x564187d85a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.431+0000 7fc356d9d700 1 --2- 172.21.15.70:0/355366000 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x564187d18860 0x564187d85a50 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1ace949875c15abb server_cookie=28ce0c25bbc7607e in_seq=0 out_seq=0 2024-06-04T05:12:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.431+0000 7fc33e7fc700 10 client.28342 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:12:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.432+0000 7fc33e7fc700 1 -- 172.21.15.70:0/355366000 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564187c513c0 con 0x564187d18860 2024-06-04T05:12:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.432+0000 7fc33e7fc700 10 client.28342 handle_command_reply: tid=0 2024-06-04T05:12:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.432+0000 7fc36793db80 1 client.28342 shutdown 2024-06-04T05:12:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.432+0000 7fc36793db80 1 -- 172.21.15.70:0/355366000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cfa090 msgr2=0x564187d0cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.432+0000 7fc36793db80 1 --2- 172.21.15.70:0/355366000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cfa090 0x564187d0cac0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x564187d0baf0 tx=0x7fc35001fe10 comp rx=0 tx=0).stop 2024-06-04T05:12:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.432+0000 7fc36793db80 1 -- 172.21.15.70:0/355366000 shutdown_connections 2024-06-04T05:12:00.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.432+0000 7fc36793db80 1 --2- 172.21.15.70:0/355366000 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x564187d18860 0x564187d85a50 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.432+0000 7fc33e7fc700 0 client.28342 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:12:00.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.432+0000 7fc36793db80 1 --2- 172.21.15.70:0/355366000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564187cfa090 0x564187d0cac0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.432+0000 7fc36793db80 1 --2- 172.21.15.70:0/355366000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564187cf6860 0x564187d11200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.432+0000 7fc36793db80 1 --2- 172.21.15.70:0/355366000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564187cf7210 0x564187d0c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.432+0000 7fc36793db80 1 -- 172.21.15.70:0/355366000 >> 172.21.15.70:0/355366000 conn(0x564187c24080 msgr2=0x564187cf5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:00.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.432+0000 7fc36793db80 1 -- 172.21.15.70:0/355366000 shutdown_connections 2024-06-04T05:12:00.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.432+0000 7fc36793db80 1 -- 172.21.15.70:0/355366000 wait complete. 2024-06-04T05:12:00.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.432+0000 7fc36793db80 20 client.28342 trim_cache size 0 max 16384 2024-06-04T05:12:00.438 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:12:00.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.433+0000 7fc35d0a8700 1 -- 172.21.15.70:0/61217822 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc3440b29f0 msgr2=0x7fc3440b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:00.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.433+0000 7fc35d0a8700 1 --2- 172.21.15.70:0/61217822 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc3440b29f0 0x7fc3440b4eb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc358079af0 tx=0x7fc340009260 comp rx=0 tx=0).stop 2024-06-04T05:12:00.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.433+0000 7fc35d0a8700 1 -- 172.21.15.70:0/61217822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc35806bab0 msgr2=0x7fc35819f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.433+0000 7fc35d0a8700 1 --2- 172.21.15.70:0/61217822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc35806bab0 0x7fc35819f780 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc348000c00 tx=0x7fc3480047b0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.433+0000 7fc35d0a8700 1 -- 172.21.15.70:0/61217822 shutdown_connections 2024-06-04T05:12:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.433+0000 7fc35d0a8700 1 --2- 172.21.15.70:0/61217822 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc3440b29f0 0x7fc3440b4eb0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.433+0000 7fc35d0a8700 1 --2- 172.21.15.70:0/61217822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3581095a0 0x7fc358199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.433+0000 7fc35d0a8700 1 --2- 172.21.15.70:0/61217822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc35806bab0 0x7fc35819f780 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.433+0000 7fc35d0a8700 1 --2- 172.21.15.70:0/61217822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc35806c470 0x7fc35819fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.433+0000 7fc35d0a8700 1 -- 172.21.15.70:0/61217822 >> 172.21.15.70:0/61217822 conn(0x7fc3580793b0 msgr2=0x7fc3581012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.433+0000 7fc35d0a8700 1 -- 172.21.15.70:0/61217822 shutdown_connections 2024-06-04T05:12:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:00.433+0000 7fc35d0a8700 1 -- 172.21.15.70:0/61217822 wait complete. 2024-06-04T05:12:00.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 9513 13.00 MB/sec warmup 52 sec latency 128.206 ms 2024-06-04T05:12:01.284 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:12:01.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:01 smithi070 ceph-mon[32631]: pgmap v462: 97 pgs: 97 active+clean; 368 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 3.2 MiB/s wr, 281 op/s 2024-06-04T05:12:01.629 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:12:01.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:01 smithi112 ceph-mon[36453]: pgmap v462: 97 pgs: 97 active+clean; 368 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 3.2 MiB/s wr, 281 op/s 2024-06-04T05:12:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:01 smithi098 ceph-mon[35569]: pgmap v462: 97 pgs: 97 active+clean; 368 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 3.2 MiB/s wr, 281 op/s 2024-06-04T05:12:01.798 INFO:tasks.workunit.client.0.smithi070.stdout: 10 9667 12.90 MB/sec warmup 53 sec latency 74.636 ms 2024-06-04T05:12:02.798 INFO:tasks.workunit.client.0.smithi070.stdout: 10 9890 13.05 MB/sec warmup 54 sec latency 67.561 ms 2024-06-04T05:12:03.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:03 smithi070 ceph-mon[32631]: pgmap v463: 97 pgs: 97 active+clean; 378 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 425 op/s 2024-06-04T05:12:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:03 smithi112 ceph-mon[36453]: pgmap v463: 97 pgs: 97 active+clean; 378 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 425 op/s 2024-06-04T05:12:03.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.700+0000 7f9352dc1700 1 -- 172.21.15.70:0/2355401110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934c102440 msgr2=0x7f934c1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:03.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.700+0000 7f9352dc1700 1 --2- 172.21.15.70:0/2355401110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934c102440 0x7f934c1028c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9338009990 tx=0x7f9338009ca0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.700+0000 7f9352dc1700 1 -- 172.21.15.70:0/2355401110 shutdown_connections 2024-06-04T05:12:03.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.700+0000 7f9352dc1700 1 --2- 172.21.15.70:0/2355401110 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f934c102e00 0x7f934c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.700+0000 7f9352dc1700 1 --2- 172.21.15.70:0/2355401110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f934c108520 0x7f934c108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.700+0000 7f9352dc1700 1 --2- 172.21.15.70:0/2355401110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934c102440 0x7f934c1028c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.700+0000 7f9352dc1700 1 -- 172.21.15.70:0/2355401110 >> 172.21.15.70:0/2355401110 conn(0x7f934c0793f0 msgr2=0x7f934c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:03.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.700+0000 7f9352dc1700 1 -- 172.21.15.70:0/2355401110 shutdown_connections 2024-06-04T05:12:03.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.700+0000 7f9352dc1700 1 -- 172.21.15.70:0/2355401110 wait complete. 2024-06-04T05:12:03.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.701+0000 7f9352dc1700 1 Processor -- start 2024-06-04T05:12:03.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.701+0000 7f9352dc1700 1 -- start start 2024-06-04T05:12:03.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.702+0000 7f9352dc1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f934c102440 0x7f934c19e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.702+0000 7f9352dc1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f934c102e00 0x7f934c19ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.702+0000 7f9352dc1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934c108520 0x7f934c1988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.702+0000 7f9352dc1700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f934c112500 con 0x7f934c102e00 2024-06-04T05:12:03.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.702+0000 7f9352dc1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f934c112380 con 0x7f934c108520 2024-06-04T05:12:03.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.702+0000 7f9352dc1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f934c112680 con 0x7f934c102440 2024-06-04T05:12:03.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.702+0000 7f934bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f934c102e00 0x7f934c19ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:03.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.702+0000 7f935135e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934c108520 0x7f934c1988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:03.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.702+0000 7f934bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f934c102e00 0x7f934c19ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44674/0 (socket says 172.21.15.70:44674) 2024-06-04T05:12:03.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.702+0000 7f935135e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934c108520 0x7f934c1988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:36350/0 (socket says 172.21.15.70:36350) 2024-06-04T05:12:03.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.702+0000 7f934bfff700 1 -- 172.21.15.70:0/2524696641 learned_addr learned my addr 172.21.15.70:0/2524696641 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:03.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.703+0000 7f935135e700 1 -- 172.21.15.70:0/2524696641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f934c102440 msgr2=0x7f934c19e720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-04T05:12:03.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.703+0000 7f935135e700 1 --2- 172.21.15.70:0/2524696641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f934c102440 0x7f934c19e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.703+0000 7f935135e700 1 -- 172.21.15.70:0/2524696641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f934c102e00 msgr2=0x7f934c19ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:03.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.703+0000 7f935135e700 1 --2- 172.21.15.70:0/2524696641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f934c102e00 0x7f934c19ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.703+0000 7f935135e700 1 -- 172.21.15.70:0/2524696641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f934c199040 con 0x7f934c108520 2024-06-04T05:12:03.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.703+0000 7f934bfff700 1 --2- 172.21.15.70:0/2524696641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f934c102e00 0x7f934c19ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:03.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.703+0000 7f935135e700 1 --2- 172.21.15.70:0/2524696641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934c108520 0x7f934c1988e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f934400efc0 tx=0x7f934400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:03.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.703+0000 7f9349ffb700 1 -- 172.21.15.70:0/2524696641 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93440092e0 con 0x7f934c108520 2024-06-04T05:12:03.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.703+0000 7f9349ffb700 1 -- 172.21.15.70:0/2524696641 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9344004bb0 con 0x7f934c108520 2024-06-04T05:12:03.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.703+0000 7f9349ffb700 1 -- 172.21.15.70:0/2524696641 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9344004020 con 0x7f934c108520 2024-06-04T05:12:03.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.704+0000 7f9352dc1700 1 -- 172.21.15.70:0/2524696641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f934c1992d0 con 0x7f934c108520 2024-06-04T05:12:03.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.704+0000 7f9352dc1700 1 -- 172.21.15.70:0/2524696641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f934c06b220 con 0x7f934c108520 2024-06-04T05:12:03.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.707+0000 7f9349ffb700 1 -- 172.21.15.70:0/2524696641 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f934401b030 con 0x7f934c108520 2024-06-04T05:12:03.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.708+0000 7f9349ffb700 1 --2- 172.21.15.70:0/2524696641 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f933c0b6f50 0x7f933c0b9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.709+0000 7f935d656b80 1 Processor -- start 2024-06-04T05:12:03.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.709+0000 7f9349ffb700 1 -- 172.21.15.70:0/2524696641 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8615+0+0 (secure 0 0 0) 0x7f93440e20c0 con 0x7f934c108520 2024-06-04T05:12:03.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.709+0000 7f9350b5d700 1 --2- 172.21.15.70:0/2524696641 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f933c0b6f50 0x7f933c0b9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:03.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.709+0000 7f935d656b80 1 -- start start 2024-06-04T05:12:03.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.709+0000 7f935d656b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b240b320 0x55e8b240b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.709+0000 7f935d656b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b24058a0 0x55e8b2405d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.709+0000 7f935d656b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b2412080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.709+0000 7f935d656b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e8b2363d70 con 0x55e8b240b320 2024-06-04T05:12:03.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.709+0000 7f935d656b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e8b2353910 con 0x55e8b24058a0 2024-06-04T05:12:03.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.709+0000 7f935d656b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e8b2357950 con 0x55e8b2406260 2024-06-04T05:12:03.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.709+0000 7f935135e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b2412080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:03.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.709+0000 7f935135e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b2412080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:52270/0 (socket says 172.21.15.70:52270) 2024-06-04T05:12:03.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.709+0000 7f935135e700 1 -- 172.21.15.70:0/3491194985 learned_addr learned my addr 172.21.15.70:0/3491194985 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:03.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.709+0000 7f934bfff700 1 --2- 172.21.15.70:0/3491194985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b24058a0 0x55e8b2405d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:03.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.709+0000 7f9350b5d700 1 --2- 172.21.15.70:0/2524696641 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f933c0b6f50 0x7f933c0b9410 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f934c199a20 tx=0x7f9340006cb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:03.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.710+0000 7f935135e700 1 -- 172.21.15.70:0/3491194985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b24058a0 msgr2=0x55e8b2405d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:03.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.710+0000 7f935135e700 1 --2- 172.21.15.70:0/3491194985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b24058a0 0x55e8b2405d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.710+0000 7f935135e700 1 -- 172.21.15.70:0/3491194985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b240b320 msgr2=0x55e8b240b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:03.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.710+0000 7f9350b5d700 1 --2- 172.21.15.70:0/3491194985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b240b320 0x55e8b240b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:03.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.710+0000 7f935135e700 1 --2- 172.21.15.70:0/3491194985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b240b320 0x55e8b240b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.710+0000 7f935135e700 1 -- 172.21.15.70:0/3491194985 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8b22fcfb0 con 0x55e8b2406260 2024-06-04T05:12:03.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.710+0000 7f9350b5d700 1 --2- 172.21.15.70:0/3491194985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b240b320 0x55e8b240b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:12:03.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.710+0000 7f935135e700 1 --2- 172.21.15.70:0/3491194985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b2412080 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f93440a90d0 tx=0x7f93440a8fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1296bb8e818c7769 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:03.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.710+0000 7f9336ffd700 1 -- 172.21.15.70:0/3491194985 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93440ad780 con 0x55e8b2406260 2024-06-04T05:12:03.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.711+0000 7f9336ffd700 1 -- 172.21.15.70:0/3491194985 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f93440b21a0 con 0x55e8b2406260 2024-06-04T05:12:03.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.711+0000 7f9336ffd700 1 -- 172.21.15.70:0/3491194985 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93440b1fa0 con 0x55e8b2406260 2024-06-04T05:12:03.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.712+0000 7f935d656b80 1 -- 172.21.15.70:0/3491194985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 msgr2=0x55e8b2412080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:03.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.712+0000 7f935d656b80 1 --2- 172.21.15.70:0/3491194985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b2412080 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f93440a90d0 tx=0x7f93440a8fb0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.712+0000 7f935d656b80 1 -- 172.21.15.70:0/3491194985 shutdown_connections 2024-06-04T05:12:03.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.712+0000 7f935d656b80 1 --2- 172.21.15.70:0/3491194985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b2412080 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.712+0000 7f935d656b80 1 --2- 172.21.15.70:0/3491194985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b240b320 0x55e8b240b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.712+0000 7f935d656b80 1 --2- 172.21.15.70:0/3491194985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b24058a0 0x55e8b2405d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.712+0000 7f935d656b80 1 -- 172.21.15.70:0/3491194985 >> 172.21.15.70:0/3491194985 conn(0x55e8b2331240 msgr2=0x55e8b24003e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:03.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.712+0000 7f935d656b80 1 -- 172.21.15.70:0/3491194985 shutdown_connections 2024-06-04T05:12:03.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.713+0000 7f935d656b80 1 -- 172.21.15.70:0/3491194985 wait complete. 2024-06-04T05:12:03.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.713+0000 7f935d656b80 1 -- start start 2024-06-04T05:12:03.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.713+0000 7f935d656b80 1 Processor -- start 2024-06-04T05:12:03.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.714+0000 7f935d656b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b24058a0 0x55e8b2411d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.714+0000 7f935d656b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b240f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.714+0000 7f935d656b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b240b320 0x55e8b240f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.714+0000 7f935d656b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e8b2259fa0 con 0x55e8b24058a0 2024-06-04T05:12:03.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.714+0000 7f935d656b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e8b22bbda0 con 0x55e8b240b320 2024-06-04T05:12:03.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.714+0000 7f935d656b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e8b241af90 con 0x55e8b2406260 2024-06-04T05:12:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.714+0000 7f935d656b80 10 client.0 _ll_register_callbacks cb 0x55e8b22ee950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.714+0000 7f9350b5d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b24058a0 0x55e8b2411d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.714+0000 7f934bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b240f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.714+0000 7f9350b5d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b24058a0 0x55e8b2411d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44688/0 (socket says 172.21.15.70:44688) 2024-06-04T05:12:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.714+0000 7f9350b5d700 1 -- 172.21.15.70:0/2639399435 learned_addr learned my addr 172.21.15.70:0/2639399435 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.714+0000 7f934bfff700 1 -- 172.21.15.70:0/2639399435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b240b320 msgr2=0x55e8b240f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.714+0000 7f934bfff700 1 --2- 172.21.15.70:0/2639399435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b240b320 0x55e8b240f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.714+0000 7f934bfff700 1 -- 172.21.15.70:0/2639399435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b24058a0 msgr2=0x55e8b2411d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.714+0000 7f934bfff700 1 --2- 172.21.15.70:0/2639399435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b24058a0 0x55e8b2411d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.714+0000 7f934bfff700 1 -- 172.21.15.70:0/2639399435 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8b22a38d0 con 0x55e8b2406260 2024-06-04T05:12:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.715+0000 7f934bfff700 1 --2- 172.21.15.70:0/2639399435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b240f0f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9338006010 tx=0x7f93380052e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ced0ad781119f151 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.715+0000 7f932bfff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:12:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.715+0000 7f932bfff700 1 -- 172.21.15.70:0/2639399435 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933803a060 con 0x55e8b2406260 2024-06-04T05:12:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.715+0000 7f935d656b80 1 -- 172.21.15.70:0/2639399435 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93440b21a0 con 0x55e8b2406260 2024-06-04T05:12:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.715+0000 7f932bfff700 1 -- 172.21.15.70:0/2639399435 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8615+0+0 (secure 0 0 0) 0x7f9338003a90 con 0x55e8b2406260 2024-06-04T05:12:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.715+0000 7f932bfff700 1 -- 172.21.15.70:0/2639399435 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f9338003c90 con 0x55e8b2406260 2024-06-04T05:12:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.715+0000 7f935d656b80 10 client.28351 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.715+0000 7f935d656b80 1 -- 172.21.15.70:0/2639399435 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55e8b230d760 con 0x55e8b2406260 2024-06-04T05:12:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.716+0000 7f932bfff700 1 -- 172.21.15.70:0/2639399435 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f93440b21a0 con 0x55e8b2406260 2024-06-04T05:12:03.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.716+0000 7f932bfff700 1 -- 172.21.15.70:0/2639399435 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7f9338035440 con 0x55e8b2406260 2024-06-04T05:12:03.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.716+0000 7f935d656b80 10 client.28351 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:03.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.716+0000 7f935d656b80 10 client.28351 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:12:03.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.716+0000 7f935d656b80 20 client.28351 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:03.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.716+0000 7f935d656b80 1 --2- 172.21.15.70:0/2639399435 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55e8b24149d0 0x55e8b2416e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.716+0000 7f935d656b80 4 client.28351 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:12:03.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.716+0000 7f935d656b80 1 -- 172.21.15.70:0/2639399435 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8b2365cc0 con 0x55e8b24149d0 2024-06-04T05:12:03.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.716+0000 7f9350b5d700 1 --2- 172.21.15.70:0/2639399435 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55e8b24149d0 0x55e8b2416e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:03.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.717+0000 7f932bfff700 10 client.28351 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:12:03.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.717+0000 7f9350b5d700 1 --2- 172.21.15.70:0/2639399435 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55e8b24149d0 0x55e8b2416e50 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=69b8eee331b4eb4e server_cookie=3fa59a9e23fe91e7 in_seq=0 out_seq=0 2024-06-04T05:12:03.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.718+0000 7f932bfff700 1 -- 172.21.15.70:0/2639399435 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e8b2365cc0 con 0x55e8b24149d0 2024-06-04T05:12:03.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.718+0000 7f932bfff700 10 client.28351 handle_command_reply: tid=0 2024-06-04T05:12:03.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.718+0000 7f935d656b80 1 client.28351 shutdown 2024-06-04T05:12:03.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.718+0000 7f935d656b80 1 -- 172.21.15.70:0/2639399435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 msgr2=0x55e8b240f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:03.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.718+0000 7f935d656b80 1 --2- 172.21.15.70:0/2639399435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b240f0f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9338006010 tx=0x7f93380052e0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.718+0000 7f935d656b80 1 -- 172.21.15.70:0/2639399435 shutdown_connections 2024-06-04T05:12:03.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.718+0000 7f935d656b80 1 --2- 172.21.15.70:0/2639399435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b240b320 0x55e8b240f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.718+0000 7f935d656b80 1 --2- 172.21.15.70:0/2639399435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b24058a0 0x55e8b2411d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.718+0000 7f935d656b80 1 --2- 172.21.15.70:0/2639399435 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55e8b24149d0 0x55e8b2416e50 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.718+0000 7f935d656b80 1 --2- 172.21.15.70:0/2639399435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b240f0f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.718+0000 7f932bfff700 0 client.28351 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:12:03.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.718+0000 7f935d656b80 1 -- 172.21.15.70:0/2639399435 >> 172.21.15.70:0/2639399435 conn(0x55e8b2331240 msgr2=0x55e8b23ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:03.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.718+0000 7f935d656b80 1 -- 172.21.15.70:0/2639399435 shutdown_connections 2024-06-04T05:12:03.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.718+0000 7f935d656b80 1 -- 172.21.15.70:0/2639399435 wait complete. 2024-06-04T05:12:03.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.718+0000 7f935d656b80 20 client.28351 trim_cache size 0 max 16384 2024-06-04T05:12:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:03 smithi098 ceph-mon[35569]: pgmap v463: 97 pgs: 97 active+clean; 378 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 425 op/s 2024-06-04T05:12:03.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.749+0000 7f935d656b80 1 Processor -- start 2024-06-04T05:12:03.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.749+0000 7f935d656b80 1 -- start start 2024-06-04T05:12:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.750+0000 7f935d656b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b24058a0 0x55e8b2400270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.750+0000 7f935d656b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b240a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.750+0000 7f935d656b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b240b320 0x55e8b240a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.750+0000 7f935d656b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f934c1abc70 con 0x55e8b240b320 2024-06-04T05:12:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.750+0000 7f935d656b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f934c1ad930 con 0x55e8b24058a0 2024-06-04T05:12:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.750+0000 7f935d656b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f934c1ac340 con 0x55e8b2406260 2024-06-04T05:12:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.750+0000 7f9350b5d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b24058a0 0x55e8b2400270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.750+0000 7f9350b5d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b24058a0 0x55e8b2400270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:36380/0 (socket says 172.21.15.70:36380) 2024-06-04T05:12:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.750+0000 7f9350b5d700 1 -- 172.21.15.70:0/2610452958 learned_addr learned my addr 172.21.15.70:0/2610452958 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.750+0000 7f934bfff700 1 --2- 172.21.15.70:0/2610452958 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b240a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.750+0000 7f935135e700 1 --2- 172.21.15.70:0/2610452958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b240b320 0x55e8b240a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.750+0000 7f9350b5d700 1 -- 172.21.15.70:0/2610452958 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 msgr2=0x55e8b240a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.750+0000 7f9350b5d700 1 --2- 172.21.15.70:0/2610452958 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b240a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.750+0000 7f9350b5d700 1 -- 172.21.15.70:0/2610452958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b240b320 msgr2=0x55e8b240a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:03.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.750+0000 7f9350b5d700 1 --2- 172.21.15.70:0/2610452958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b240b320 0x55e8b240a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.751+0000 7f9350b5d700 1 -- 172.21.15.70:0/2610452958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8b240d660 con 0x55e8b24058a0 2024-06-04T05:12:03.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.751+0000 7f935135e700 1 --2- 172.21.15.70:0/2610452958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b240b320 0x55e8b240a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:03.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.751+0000 7f9350b5d700 1 --2- 172.21.15.70:0/2610452958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b24058a0 0x55e8b2400270 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9340009420 tx=0x7f934000e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e712fecb83e8263 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:03.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.751+0000 7f93367fc700 1 -- 172.21.15.70:0/2610452958 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9340013ab0 con 0x55e8b24058a0 2024-06-04T05:12:03.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.751+0000 7f93367fc700 1 -- 172.21.15.70:0/2610452958 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9340025030 con 0x55e8b24058a0 2024-06-04T05:12:03.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.751+0000 7f93367fc700 1 -- 172.21.15.70:0/2610452958 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9340021730 con 0x55e8b24058a0 2024-06-04T05:12:03.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.752+0000 7f935d656b80 1 -- 172.21.15.70:0/2610452958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b24058a0 msgr2=0x55e8b2400270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:03.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.752+0000 7f935d656b80 1 --2- 172.21.15.70:0/2610452958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b24058a0 0x55e8b2400270 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9340009420 tx=0x7f934000e000 comp rx=0 tx=0).stop 2024-06-04T05:12:03.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.752+0000 7f935d656b80 1 -- 172.21.15.70:0/2610452958 shutdown_connections 2024-06-04T05:12:03.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.752+0000 7f935d656b80 1 --2- 172.21.15.70:0/2610452958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b240b320 0x55e8b240a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.752+0000 7f935d656b80 1 --2- 172.21.15.70:0/2610452958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b24058a0 0x55e8b2400270 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.752+0000 7f935d656b80 1 --2- 172.21.15.70:0/2610452958 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b240a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.752+0000 7f935d656b80 1 -- 172.21.15.70:0/2610452958 >> 172.21.15.70:0/2610452958 conn(0x55e8b2331240 msgr2=0x55e8b240f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:03.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.752+0000 7f935d656b80 1 -- 172.21.15.70:0/2610452958 shutdown_connections 2024-06-04T05:12:03.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.753+0000 7f935d656b80 1 -- 172.21.15.70:0/2610452958 wait complete. 2024-06-04T05:12:03.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.753+0000 7f935d656b80 1 -- start start 2024-06-04T05:12:03.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.753+0000 7f935d656b80 1 Processor -- start 2024-06-04T05:12:03.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f935d656b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b24058a0 0x55e8b242ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f9350b5d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b24058a0 0x55e8b242ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:03.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f9350b5d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b24058a0 0x55e8b242ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44714/0 (socket says 172.21.15.70:44714) 2024-06-04T05:12:03.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f9350b5d700 1 -- 172.21.15.70:0/2298854854 learned_addr learned my addr 172.21.15.70:0/2298854854 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:03.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f935d656b80 1 --2- 172.21.15.70:0/2298854854 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b2419b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f935d656b80 1 --2- 172.21.15.70:0/2298854854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b240b320 0x55e8b241a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f935d656b80 1 -- 172.21.15.70:0/2298854854 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f934c1ad1a0 con 0x55e8b24058a0 2024-06-04T05:12:03.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f935d656b80 1 -- 172.21.15.70:0/2298854854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f934c1aca70 con 0x55e8b240b320 2024-06-04T05:12:03.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f935d656b80 1 -- 172.21.15.70:0/2298854854 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f934c1a9c50 con 0x55e8b2406260 2024-06-04T05:12:03.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f934bfff700 1 --2- 172.21.15.70:0/2298854854 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b2419b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:03.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f935135e700 1 --2- 172.21.15.70:0/2298854854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b240b320 0x55e8b241a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:03.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f935d656b80 10 client.0 _ll_register_callbacks cb 0x55e8b22ee950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:03.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f9350b5d700 1 -- 172.21.15.70:0/2298854854 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 msgr2=0x55e8b2419b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:03.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f9350b5d700 1 --2- 172.21.15.70:0/2298854854 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b2419b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f9350b5d700 1 -- 172.21.15.70:0/2298854854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b240b320 msgr2=0x55e8b241a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:03.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f9350b5d700 1 --2- 172.21.15.70:0/2298854854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b240b320 0x55e8b241a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f9350b5d700 1 -- 172.21.15.70:0/2298854854 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8b241a740 con 0x55e8b24058a0 2024-06-04T05:12:03.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.754+0000 7f934bfff700 1 --2- 172.21.15.70:0/2298854854 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b2419b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:03.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.755+0000 7f9350b5d700 1 --2- 172.21.15.70:0/2298854854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b24058a0 0x55e8b242ac70 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f9340017010 tx=0x7f9340004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e55d04bee8e61b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:03.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.755+0000 7f932bfff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:12:03.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.755+0000 7f932bfff700 1 -- 172.21.15.70:0/2298854854 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9340014ea0 con 0x55e8b24058a0 2024-06-04T05:12:03.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.755+0000 7f935d656b80 1 -- 172.21.15.70:0/2298854854 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9340025030 con 0x55e8b24058a0 2024-06-04T05:12:03.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.755+0000 7f932bfff700 1 -- 172.21.15.70:0/2298854854 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8615+0+0 (secure 0 0 0) 0x7f9340015110 con 0x55e8b24058a0 2024-06-04T05:12:03.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.755+0000 7f932bfff700 1 -- 172.21.15.70:0/2298854854 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f9340015310 con 0x55e8b24058a0 2024-06-04T05:12:03.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.755+0000 7f935d656b80 10 client.19365 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:03.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.755+0000 7f935d656b80 1 -- 172.21.15.70:0/2298854854 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55e8b24895b0 con 0x55e8b24058a0 2024-06-04T05:12:03.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.756+0000 7f932bfff700 1 -- 172.21.15.70:0/2298854854 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9340025030 con 0x55e8b24058a0 2024-06-04T05:12:03.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.756+0000 7f932bfff700 1 -- 172.21.15.70:0/2298854854 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7f93400214d0 con 0x55e8b24058a0 2024-06-04T05:12:03.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.756+0000 7f935d656b80 10 client.19365 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:03.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.756+0000 7f935d656b80 10 client.19365 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:12:03.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.756+0000 7f935d656b80 20 client.19365 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:03.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.756+0000 7f935d656b80 1 --2- 172.21.15.70:0/2298854854 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55e8b24899f0 0x55e8b248be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:03.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.756+0000 7f935d656b80 4 client.19365 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:12:03.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.756+0000 7f935d656b80 1 -- 172.21.15.70:0/2298854854 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e8b235f4a0 con 0x55e8b24899f0 2024-06-04T05:12:03.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.756+0000 7f934bfff700 1 --2- 172.21.15.70:0/2298854854 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55e8b24899f0 0x55e8b248be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:03.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.756+0000 7f932bfff700 10 client.19365 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:12:03.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.756+0000 7f934bfff700 1 --2- 172.21.15.70:0/2298854854 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55e8b24899f0 0x55e8b248be10 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=289b7bcd4d4c5d35 server_cookie=ff5f43d91145c175 in_seq=0 out_seq=0 2024-06-04T05:12:03.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.757+0000 7f932bfff700 1 -- 172.21.15.70:0/2298854854 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e8b235f4a0 con 0x55e8b24899f0 2024-06-04T05:12:03.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.757+0000 7f932bfff700 10 client.19365 handle_command_reply: tid=0 2024-06-04T05:12:03.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.758+0000 7f935d656b80 1 client.19365 shutdown 2024-06-04T05:12:03.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.758+0000 7f935d656b80 1 -- 172.21.15.70:0/2298854854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b24058a0 msgr2=0x55e8b242ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:03.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.758+0000 7f935d656b80 1 --2- 172.21.15.70:0/2298854854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b24058a0 0x55e8b242ac70 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f9340017010 tx=0x7f9340004860 comp rx=0 tx=0).stop 2024-06-04T05:12:03.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.758+0000 7f935d656b80 1 -- 172.21.15.70:0/2298854854 shutdown_connections 2024-06-04T05:12:03.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.758+0000 7f935d656b80 1 --2- 172.21.15.70:0/2298854854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e8b240b320 0x55e8b241a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.758+0000 7f935d656b80 1 --2- 172.21.15.70:0/2298854854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e8b24058a0 0x55e8b242ac70 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.758+0000 7f935d656b80 1 --2- 172.21.15.70:0/2298854854 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55e8b24899f0 0x55e8b248be10 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.758+0000 7f935d656b80 1 --2- 172.21.15.70:0/2298854854 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e8b2406260 0x55e8b2419b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.758+0000 7f935d656b80 1 -- 172.21.15.70:0/2298854854 >> 172.21.15.70:0/2298854854 conn(0x55e8b2331240 msgr2=0x55e8b240a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:03.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.758+0000 7f932bfff700 0 client.19365 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:12:03.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.758+0000 7f935d656b80 1 -- 172.21.15.70:0/2298854854 shutdown_connections 2024-06-04T05:12:03.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.758+0000 7f935d656b80 1 -- 172.21.15.70:0/2298854854 wait complete. 2024-06-04T05:12:03.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.759+0000 7f935d656b80 20 client.19365 trim_cache size 0 max 16384 2024-06-04T05:12:03.764 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:12:03.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.759+0000 7f9352dc1700 1 -- 172.21.15.70:0/2524696641 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f933c0b6f50 msgr2=0x7f933c0b9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:03.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.759+0000 7f9352dc1700 1 --2- 172.21.15.70:0/2524696641 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f933c0b6f50 0x7f933c0b9410 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f934c199a20 tx=0x7f9340006cb0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.759+0000 7f9352dc1700 1 -- 172.21.15.70:0/2524696641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934c108520 msgr2=0x7f934c1988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:03.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.759+0000 7f9352dc1700 1 --2- 172.21.15.70:0/2524696641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934c108520 0x7f934c1988e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f934400efc0 tx=0x7f934400c3a0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.759+0000 7f9352dc1700 1 -- 172.21.15.70:0/2524696641 shutdown_connections 2024-06-04T05:12:03.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.759+0000 7f9352dc1700 1 --2- 172.21.15.70:0/2524696641 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f933c0b6f50 0x7f933c0b9410 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.759+0000 7f9352dc1700 1 --2- 172.21.15.70:0/2524696641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934c108520 0x7f934c1988e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.759+0000 7f9352dc1700 1 --2- 172.21.15.70:0/2524696641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f934c102440 0x7f934c19e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.759+0000 7f9352dc1700 1 --2- 172.21.15.70:0/2524696641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f934c102e00 0x7f934c19ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:03.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.759+0000 7f9352dc1700 1 -- 172.21.15.70:0/2524696641 >> 172.21.15.70:0/2524696641 conn(0x7f934c0793f0 msgr2=0x7f934c100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:03.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.760+0000 7f9352dc1700 1 -- 172.21.15.70:0/2524696641 shutdown_connections 2024-06-04T05:12:03.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:03.760+0000 7f9352dc1700 1 -- 172.21.15.70:0/2524696641 wait complete. 2024-06-04T05:12:03.798 INFO:tasks.workunit.client.0.smithi070.stdout: 10 10080 13.04 MB/sec warmup 55 sec latency 62.680 ms 2024-06-04T05:12:04.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:04 smithi070 ceph-mon[32631]: osdmap e97: 12 total, 12 up, 12 in 2024-06-04T05:12:04.544 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:12:04.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:04 smithi112 ceph-mon[36453]: osdmap e97: 12 total, 12 up, 12 in 2024-06-04T05:12:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:04 smithi098 ceph-mon[35569]: osdmap e97: 12 total, 12 up, 12 in 2024-06-04T05:12:04.798 INFO:tasks.workunit.client.0.smithi070.stdout: 10 10260 13.02 MB/sec warmup 56 sec latency 55.794 ms 2024-06-04T05:12:04.899 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:12:05.586 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:05 smithi070 ceph-mon[32631]: pgmap v465: 97 pgs: 97 active+clean; 378 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 358 op/s 2024-06-04T05:12:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:05 smithi112 ceph-mon[36453]: pgmap v465: 97 pgs: 97 active+clean; 378 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 358 op/s 2024-06-04T05:12:05.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:05 smithi098 ceph-mon[35569]: pgmap v465: 97 pgs: 97 active+clean; 378 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 358 op/s 2024-06-04T05:12:05.798 INFO:tasks.workunit.client.0.smithi070.stdout: 10 10416 12.96 MB/sec warmup 57 sec latency 102.046 ms 2024-06-04T05:12:06.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 10591 12.91 MB/sec warmup 58 sec latency 73.450 ms 2024-06-04T05:12:06.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.922+0000 7f7d7e62b700 1 -- 172.21.15.70:0/1556742625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d781095d0 msgr2=0x7f7d781099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:06.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.922+0000 7f7d7e62b700 1 --2- 172.21.15.70:0/1556742625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d781095d0 0x7f7d781099b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7d680098f0 tx=0x7f7d68009c00 comp rx=0 tx=0).stop 2024-06-04T05:12:06.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.922+0000 7f7d7e62b700 1 -- 172.21.15.70:0/1556742625 shutdown_connections 2024-06-04T05:12:06.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.922+0000 7f7d7e62b700 1 --2- 172.21.15.70:0/1556742625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d78103f80 0x7f7d78110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.922+0000 7f7d7e62b700 1 --2- 172.21.15.70:0/1556742625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d781095d0 0x7f7d781099b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.922+0000 7f7d7e62b700 1 --2- 172.21.15.70:0/1556742625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d781035c0 0x7f7d78103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.922+0000 7f7d7e62b700 1 -- 172.21.15.70:0/1556742625 >> 172.21.15.70:0/1556742625 conn(0x7f7d7807a610 msgr2=0x7f7d7807aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:06.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.922+0000 7f7d7e62b700 1 -- 172.21.15.70:0/1556742625 shutdown_connections 2024-06-04T05:12:06.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.922+0000 7f7d7e62b700 1 -- 172.21.15.70:0/1556742625 wait complete. 2024-06-04T05:12:06.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.923+0000 7f7d7e62b700 1 Processor -- start 2024-06-04T05:12:06.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.923+0000 7f7d7e62b700 1 -- start start 2024-06-04T05:12:06.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.923+0000 7f7d7e62b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d781035c0 0x7f7d78075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.923+0000 7f7d7e62b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d78103f80 0x7f7d78075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.923+0000 7f7d7e62b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d781095d0 0x7f7d780762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.923+0000 7f7d7e62b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781135d0 con 0x7f7d781095d0 2024-06-04T05:12:06.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.923+0000 7f7d7e62b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d78113450 con 0x7f7d78103f80 2024-06-04T05:12:06.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.923+0000 7f7d7e62b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d78113750 con 0x7f7d781035c0 2024-06-04T05:12:06.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.923+0000 7f7d777fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d78103f80 0x7f7d78075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:06.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.923+0000 7f7d77fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d781035c0 0x7f7d78075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:06.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.923+0000 7f7d77fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d781035c0 0x7f7d78075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:42364/0 (socket says 172.21.15.70:42364) 2024-06-04T05:12:06.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.923+0000 7f7d777fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d78103f80 0x7f7d78075d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:38296/0 (socket says 172.21.15.70:38296) 2024-06-04T05:12:06.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.923+0000 7f7d77fff700 1 -- 172.21.15.70:0/1862749949 learned_addr learned my addr 172.21.15.70:0/1862749949 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:06.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.924+0000 7f7d77fff700 1 -- 172.21.15.70:0/1862749949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d78103f80 msgr2=0x7f7d78075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:06.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.924+0000 7f7d77fff700 1 --2- 172.21.15.70:0/1862749949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d78103f80 0x7f7d78075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.924+0000 7f7d77fff700 1 -- 172.21.15.70:0/1862749949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d781095d0 msgr2=0x7f7d780762c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-04T05:12:06.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.924+0000 7f7d77fff700 1 --2- 172.21.15.70:0/1862749949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d781095d0 0x7f7d780762c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.924+0000 7f7d77fff700 1 -- 172.21.15.70:0/1862749949 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d781a7e70 con 0x7f7d781035c0 2024-06-04T05:12:06.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.924+0000 7f7d77fff700 1 --2- 172.21.15.70:0/1862749949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d781035c0 0x7f7d78075840 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800b840 tx=0x7f7d6800b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:06.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.924+0000 7f7d757fa700 1 -- 172.21.15.70:0/1862749949 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d68026070 con 0x7f7d781035c0 2024-06-04T05:12:06.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.924+0000 7f7d757fa700 1 -- 172.21.15.70:0/1862749949 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d680058e0 con 0x7f7d781035c0 2024-06-04T05:12:06.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.924+0000 7f7d757fa700 1 -- 172.21.15.70:0/1862749949 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d68004b70 con 0x7f7d781035c0 2024-06-04T05:12:06.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.924+0000 7f7d7e62b700 1 -- 172.21.15.70:0/1862749949 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d781a8100 con 0x7f7d781035c0 2024-06-04T05:12:06.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.924+0000 7f7d7e62b700 1 -- 172.21.15.70:0/1862749949 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d781a84d0 con 0x7f7d781035c0 2024-06-04T05:12:06.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.927+0000 7f7d757fa700 1 -- 172.21.15.70:0/1862749949 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7d68029070 con 0x7f7d781035c0 2024-06-04T05:12:06.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.928+0000 7f7d757fa700 1 --2- 172.21.15.70:0/1862749949 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d600b2a70 0x7f7d600b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.928+0000 7f7d777fe700 1 --2- 172.21.15.70:0/1862749949 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d600b2a70 0x7f7d600b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:06.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.928+0000 7f7d757fa700 1 -- 172.21.15.70:0/1862749949 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8615+0+0 (secure 0 0 0) 0x7f7d680e34d0 con 0x7f7d781035c0 2024-06-04T05:12:06.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.929+0000 7f7d777fe700 1 --2- 172.21.15.70:0/1862749949 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d600b2a70 0x7f7d600b4f30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c009750 tx=0x7f7d6c009310 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:06.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.929+0000 7f7d88ec0b80 1 Processor -- start 2024-06-04T05:12:06.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.929+0000 7f7d88ec0b80 1 -- start start 2024-06-04T05:12:06.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.929+0000 7f7d88ec0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b5320 0x55c8dc5b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.929+0000 7f7d88ec0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.929+0000 7f7d88ec0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5b0260 0x55c8dc5bc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.929+0000 7f7d88ec0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c8dc50dd70 con 0x55c8dc5b0260 2024-06-04T05:12:06.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.929+0000 7f7d88ec0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c8dc4fd910 con 0x55c8dc5af8a0 2024-06-04T05:12:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.930+0000 7f7d88ec0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c8dc501950 con 0x55c8dc5b5320 2024-06-04T05:12:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.930+0000 7f7d777fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.930+0000 7f7d77fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b5320 0x55c8dc5b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.930+0000 7f7d777fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5afd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:38310/0 (socket says 172.21.15.70:38310) 2024-06-04T05:12:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.930+0000 7f7d777fe700 1 -- 172.21.15.70:0/2710942475 learned_addr learned my addr 172.21.15.70:0/2710942475 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.930+0000 7f7d7cbc8700 1 --2- 172.21.15.70:0/2710942475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5b0260 0x55c8dc5bc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.930+0000 7f7d777fe700 1 -- 172.21.15.70:0/2710942475 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b5320 msgr2=0x55c8dc5b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.930+0000 7f7d777fe700 1 --2- 172.21.15.70:0/2710942475 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b5320 0x55c8dc5b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.930+0000 7f7d777fe700 1 -- 172.21.15.70:0/2710942475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5b0260 msgr2=0x55c8dc5bc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.930+0000 7f7d777fe700 1 --2- 172.21.15.70:0/2710942475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5b0260 0x55c8dc5bc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.930+0000 7f7d777fe700 1 -- 172.21.15.70:0/2710942475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8dc4a6fb0 con 0x55c8dc5af8a0 2024-06-04T05:12:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.930+0000 7f7d777fe700 1 --2- 172.21.15.70:0/2710942475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5afd20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00e520 tx=0x7f7d6c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f62f8d197f17fae server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.930+0000 7f7d5e7fc700 1 -- 172.21.15.70:0/2710942475 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c026430 con 0x55c8dc5af8a0 2024-06-04T05:12:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.930+0000 7f7d5e7fc700 1 -- 172.21.15.70:0/2710942475 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d6c01c8a0 con 0x55c8dc5af8a0 2024-06-04T05:12:06.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.930+0000 7f7d5e7fc700 1 -- 172.21.15.70:0/2710942475 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c01d7d0 con 0x55c8dc5af8a0 2024-06-04T05:12:06.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.931+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/2710942475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5af8a0 msgr2=0x55c8dc5afd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:06.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.931+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/2710942475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5afd20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00e520 tx=0x7f7d6c014000 comp rx=0 tx=0).stop 2024-06-04T05:12:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.931+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/2710942475 shutdown_connections 2024-06-04T05:12:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.931+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/2710942475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5b0260 0x55c8dc5bc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.931+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/2710942475 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b5320 0x55c8dc5b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.931+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/2710942475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5afd20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.931+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/2710942475 >> 172.21.15.70:0/2710942475 conn(0x55c8dc4db240 msgr2=0x55c8dc5aa3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.932+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/2710942475 shutdown_connections 2024-06-04T05:12:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.932+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/2710942475 wait complete. 2024-06-04T05:12:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.932+0000 7f7d88ec0b80 1 -- start start 2024-06-04T05:12:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.932+0000 7f7d88ec0b80 1 Processor -- start 2024-06-04T05:12:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d88ec0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d88ec0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b0260 0x55c8dc5b9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d88ec0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b5320 0x55c8dc5b9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d88ec0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c8dc403fa0 con 0x55c8dc5af8a0 2024-06-04T05:12:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d88ec0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c8dc465da0 con 0x55c8dc5b0260 2024-06-04T05:12:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d88ec0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c8dc5bc6e0 con 0x55c8dc5b5320 2024-06-04T05:12:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d88ec0b80 10 client.0 _ll_register_callbacks cb 0x55c8dc498950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d77fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d7cbc8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b5320 0x55c8dc5b9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d77fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5bbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46130/0 (socket says 172.21.15.70:46130) 2024-06-04T05:12:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d7cbc8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b5320 0x55c8dc5b9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:42374/0 (socket says 172.21.15.70:42374) 2024-06-04T05:12:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d77fff700 1 -- 172.21.15.70:0/1268640818 learned_addr learned my addr 172.21.15.70:0/1268640818 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d777fe700 1 --2- 172.21.15.70:0/1268640818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b0260 0x55c8dc5b9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d7cbc8700 1 -- 172.21.15.70:0/1268640818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b0260 msgr2=0x55c8dc5b9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d7cbc8700 1 --2- 172.21.15.70:0/1268640818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b0260 0x55c8dc5b9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d7cbc8700 1 -- 172.21.15.70:0/1268640818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 msgr2=0x55c8dc5bbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d7cbc8700 1 --2- 172.21.15.70:0/1268640818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5bbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d7cbc8700 1 -- 172.21.15.70:0/1268640818 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8dc44d8d0 con 0x55c8dc5b5320 2024-06-04T05:12:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.933+0000 7f7d77fff700 1 --2- 172.21.15.70:0/1268640818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:06.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.934+0000 7f7d777fe700 1 --2- 172.21.15.70:0/1268640818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b0260 0x55c8dc5b9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:06.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.934+0000 7f7d7cbc8700 1 --2- 172.21.15.70:0/1268640818 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b5320 0x55c8dc5b9570 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7d70011b90 tx=0x7f7d70011ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5286e563e261d628 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:06.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.934+0000 7f7d577fe700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:12:06.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.934+0000 7f7d577fe700 1 -- 172.21.15.70:0/1268640818 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7001a070 con 0x55c8dc5b5320 2024-06-04T05:12:06.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.934+0000 7f7d577fe700 1 -- 172.21.15.70:0/1268640818 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8615+0+0 (secure 0 0 0) 0x7f7d70004bb0 con 0x55c8dc5b5320 2024-06-04T05:12:06.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.934+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/1268640818 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d6c01c8a0 con 0x55c8dc5b5320 2024-06-04T05:12:06.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.934+0000 7f7d577fe700 1 -- 172.21.15.70:0/1268640818 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f7d70023050 con 0x55c8dc5b5320 2024-06-04T05:12:06.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.934+0000 7f7d88ec0b80 10 client.28369 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:06.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.934+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/1268640818 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55c8dc4b7760 con 0x55c8dc5b5320 2024-06-04T05:12:06.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.934+0000 7f7d577fe700 1 -- 172.21.15.70:0/1268640818 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d6c01c8a0 con 0x55c8dc5b5320 2024-06-04T05:12:06.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.935+0000 7f7d577fe700 1 -- 172.21.15.70:0/1268640818 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7f7d7001d020 con 0x55c8dc5b5320 2024-06-04T05:12:06.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.935+0000 7f7d88ec0b80 10 client.28369 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:06.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.935+0000 7f7d88ec0b80 10 client.28369 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:12:06.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.935+0000 7f7d88ec0b80 20 client.28369 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:06.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.935+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/1268640818 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c8dc5c90c0 0x55c8dc5cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.935+0000 7f7d88ec0b80 4 client.28369 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:12:06.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.935+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/1268640818 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8dc50fcc0 con 0x55c8dc5c90c0 2024-06-04T05:12:06.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.935+0000 7f7d77fff700 1 --2- 172.21.15.70:0/1268640818 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c8dc5c90c0 0x55c8dc5cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:06.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.935+0000 7f7d577fe700 10 client.28369 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:12:06.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.936+0000 7f7d77fff700 1 --2- 172.21.15.70:0/1268640818 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c8dc5c90c0 0x55c8dc5cb4e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9cb9963b45884c01 server_cookie=cd6fb3ba0b4584fb in_seq=0 out_seq=0 2024-06-04T05:12:06.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.936+0000 7f7d577fe700 1 -- 172.21.15.70:0/1268640818 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c8dc50fcc0 con 0x55c8dc5c90c0 2024-06-04T05:12:06.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.936+0000 7f7d577fe700 10 client.28369 handle_command_reply: tid=0 2024-06-04T05:12:06.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.936+0000 7f7d88ec0b80 1 client.28369 shutdown 2024-06-04T05:12:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.937+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/1268640818 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b5320 msgr2=0x55c8dc5b9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.937+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/1268640818 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b5320 0x55c8dc5b9570 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7d70011b90 tx=0x7f7d70011ea0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.937+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/1268640818 shutdown_connections 2024-06-04T05:12:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.937+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/1268640818 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c8dc5c90c0 0x55c8dc5cb4e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.937+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/1268640818 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b5320 0x55c8dc5b9570 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.937+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/1268640818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.937+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/1268640818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b0260 0x55c8dc5b9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.937+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/1268640818 >> 172.21.15.70:0/1268640818 conn(0x55c8dc4db240 msgr2=0x55c8dc5a9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.937+0000 7f7d577fe700 0 client.28369 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:12:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.938+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/1268640818 shutdown_connections 2024-06-04T05:12:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.938+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/1268640818 wait complete. 2024-06-04T05:12:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.938+0000 7f7d88ec0b80 20 client.28369 trim_cache size 0 max 16384 2024-06-04T05:12:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.966+0000 7f7d88ec0b80 1 Processor -- start 2024-06-04T05:12:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.966+0000 7f7d88ec0b80 1 -- start start 2024-06-04T05:12:06.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.966+0000 7f7d88ec0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5b9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.966+0000 7f7d88ec0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b0260 0x55c8dc5b3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.966+0000 7f7d88ec0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b5320 0x55c8dc5aa270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.966+0000 7f7d88ec0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781afe30 con 0x55c8dc5af8a0 2024-06-04T05:12:06.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.966+0000 7f7d88ec0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781b1af0 con 0x55c8dc5b5320 2024-06-04T05:12:06.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.966+0000 7f7d88ec0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781b0500 con 0x55c8dc5b0260 2024-06-04T05:12:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.966+0000 7f7d77fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5b9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.966+0000 7f7d77fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5b9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46134/0 (socket says 172.21.15.70:46134) 2024-06-04T05:12:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.966+0000 7f7d77fff700 1 -- 172.21.15.70:0/3651969207 learned_addr learned my addr 172.21.15.70:0/3651969207 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.967+0000 7f7d7cbc8700 1 --2- 172.21.15.70:0/3651969207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b5320 0x55c8dc5aa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.967+0000 7f7d777fe700 1 --2- 172.21.15.70:0/3651969207 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b0260 0x55c8dc5b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.967+0000 7f7d77fff700 1 -- 172.21.15.70:0/3651969207 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b0260 msgr2=0x55c8dc5b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.967+0000 7f7d77fff700 1 --2- 172.21.15.70:0/3651969207 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b0260 0x55c8dc5b3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.967+0000 7f7d77fff700 1 -- 172.21.15.70:0/3651969207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b5320 msgr2=0x55c8dc5aa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.967+0000 7f7d77fff700 1 --2- 172.21.15.70:0/3651969207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b5320 0x55c8dc5aa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.967+0000 7f7d77fff700 1 -- 172.21.15.70:0/3651969207 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8dc5bbce0 con 0x55c8dc5af8a0 2024-06-04T05:12:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.967+0000 7f7d777fe700 1 --2- 172.21.15.70:0/3651969207 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b0260 0x55c8dc5b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:12:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.967+0000 7f7d77fff700 1 --2- 172.21.15.70:0/3651969207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5b9660 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f7d680094b0 tx=0x7f7d68000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812b6ab38e29026d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:06.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.968+0000 7f7d5dffb700 1 -- 172.21.15.70:0/3651969207 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d680b1e60 con 0x55c8dc5af8a0 2024-06-04T05:12:06.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.968+0000 7f7d5dffb700 1 -- 172.21.15.70:0/3651969207 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d680b10b0 con 0x55c8dc5af8a0 2024-06-04T05:12:06.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.968+0000 7f7d5dffb700 1 -- 172.21.15.70:0/3651969207 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d680b3590 con 0x55c8dc5af8a0 2024-06-04T05:12:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.968+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/3651969207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 msgr2=0x55c8dc5b9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.968+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/3651969207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5b9660 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f7d680094b0 tx=0x7f7d68000bc0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.969+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/3651969207 shutdown_connections 2024-06-04T05:12:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.969+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/3651969207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b5320 0x55c8dc5aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.969+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/3651969207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5b9660 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.969+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/3651969207 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b0260 0x55c8dc5b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.969+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/3651969207 >> 172.21.15.70:0/3651969207 conn(0x55c8dc4db240 msgr2=0x55c8dc5b2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.969+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/3651969207 shutdown_connections 2024-06-04T05:12:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.969+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/3651969207 wait complete. 2024-06-04T05:12:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.969+0000 7f7d88ec0b80 1 -- start start 2024-06-04T05:12:06.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.969+0000 7f7d88ec0b80 1 Processor -- start 2024-06-04T05:12:06.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.970+0000 7f7d88ec0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.970+0000 7f7d88ec0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b0260 0x55c8dc5c1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.970+0000 7f7d88ec0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b5320 0x55c8dc5cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.970+0000 7f7d88ec0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781b1360 con 0x55c8dc5af8a0 2024-06-04T05:12:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.970+0000 7f7d88ec0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781b0c30 con 0x55c8dc5b5320 2024-06-04T05:12:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.970+0000 7f7d88ec0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781addf0 con 0x55c8dc5b0260 2024-06-04T05:12:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.970+0000 7f7d88ec0b80 10 client.0 _ll_register_callbacks cb 0x55c8dc498950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.970+0000 7f7d777fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b0260 0x55c8dc5c1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.970+0000 7f7d7cbc8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b5320 0x55c8dc5cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.970+0000 7f7d7cbc8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b5320 0x55c8dc5cfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:38336/0 (socket says 172.21.15.70:38336) 2024-06-04T05:12:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.970+0000 7f7d7cbc8700 1 -- 172.21.15.70:0/3298946780 learned_addr learned my addr 172.21.15.70:0/3298946780 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.970+0000 7f7d77fff700 1 --2- 172.21.15.70:0/3298946780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.971+0000 7f7d7cbc8700 1 -- 172.21.15.70:0/3298946780 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b0260 msgr2=0x55c8dc5c1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.971+0000 7f7d7cbc8700 1 --2- 172.21.15.70:0/3298946780 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b0260 0x55c8dc5c1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.971+0000 7f7d7cbc8700 1 -- 172.21.15.70:0/3298946780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 msgr2=0x55c8dc5c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.971+0000 7f7d7cbc8700 1 --2- 172.21.15.70:0/3298946780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5c1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.971+0000 7f7d7cbc8700 1 -- 172.21.15.70:0/3298946780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8dc5d0390 con 0x55c8dc5b5320 2024-06-04T05:12:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.971+0000 7f7d7cbc8700 1 --2- 172.21.15.70:0/3298946780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b5320 0x55c8dc5cfe20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000f840 tx=0x7f7d7000f870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=594d30605012bf98 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.971+0000 7f7d577fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:12:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.971+0000 7f7d577fe700 1 -- 172.21.15.70:0/3298946780 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7001a040 con 0x55c8dc5b5320 2024-06-04T05:12:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.971+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/3298946780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c8dc5094a0 con 0x55c8dc5b5320 2024-06-04T05:12:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.971+0000 7f7d577fe700 1 -- 172.21.15.70:0/3298946780 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8615+0+0 (secure 0 0 0) 0x7f7d70004970 con 0x55c8dc5b5320 2024-06-04T05:12:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.971+0000 7f7d577fe700 1 -- 172.21.15.70:0/3298946780 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f7d70004b70 con 0x55c8dc5b5320 2024-06-04T05:12:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.971+0000 7f7d88ec0b80 10 client.28250 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.971+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/3298946780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55c8dc63bc20 con 0x55c8dc5b5320 2024-06-04T05:12:06.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.972+0000 7f7d577fe700 1 -- 172.21.15.70:0/3298946780 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55c8dc5094a0 con 0x55c8dc5b5320 2024-06-04T05:12:06.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.972+0000 7f7d577fe700 1 -- 172.21.15.70:0/3298946780 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7f7d7001d040 con 0x55c8dc5b5320 2024-06-04T05:12:06.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.972+0000 7f7d88ec0b80 10 client.28250 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:06.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.972+0000 7f7d88ec0b80 10 client.28250 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:12:06.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.972+0000 7f7d88ec0b80 20 client.28250 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:06.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.972+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/3298946780 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c8dc63c410 0x55c8dc63e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:06.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.972+0000 7f7d88ec0b80 4 client.28250 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:12:06.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.972+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/3298946780 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c8dc380250 con 0x55c8dc63c410 2024-06-04T05:12:06.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.972+0000 7f7d77fff700 1 --2- 172.21.15.70:0/3298946780 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c8dc63c410 0x55c8dc63e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:06.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.973+0000 7f7d577fe700 10 client.28250 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:12:06.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.973+0000 7f7d77fff700 1 --2- 172.21.15.70:0/3298946780 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c8dc63c410 0x55c8dc63e830 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c35d9f1ae17de60b server_cookie=293742d3b12c3936 in_seq=0 out_seq=0 2024-06-04T05:12:06.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.973+0000 7f7d577fe700 1 -- 172.21.15.70:0/3298946780 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c8dc380250 con 0x55c8dc63c410 2024-06-04T05:12:06.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.973+0000 7f7d577fe700 10 client.28250 handle_command_reply: tid=0 2024-06-04T05:12:06.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.973+0000 7f7d88ec0b80 1 client.28250 shutdown 2024-06-04T05:12:06.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.973+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/3298946780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b5320 msgr2=0x55c8dc5cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.973+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/3298946780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b5320 0x55c8dc5cfe20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000f840 tx=0x7f7d7000f870 comp rx=0 tx=0).stop 2024-06-04T05:12:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.973+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/3298946780 shutdown_connections 2024-06-04T05:12:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.973+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/3298946780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8dc5b5320 0x55c8dc5cfe20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.973+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/3298946780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8dc5af8a0 0x55c8dc5c1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.973+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/3298946780 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c8dc63c410 0x55c8dc63e830 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.973+0000 7f7d88ec0b80 1 --2- 172.21.15.70:0/3298946780 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8dc5b0260 0x55c8dc5c1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.974+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/3298946780 >> 172.21.15.70:0/3298946780 conn(0x55c8dc4db240 msgr2=0x55c8dc5ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.974+0000 7f7d577fe700 0 client.28250 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:12:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.974+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/3298946780 shutdown_connections 2024-06-04T05:12:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.974+0000 7f7d88ec0b80 1 -- 172.21.15.70:0/3298946780 wait complete. 2024-06-04T05:12:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.974+0000 7f7d88ec0b80 20 client.28250 trim_cache size 0 max 16384 2024-06-04T05:12:06.979 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:12:06.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.975+0000 7f7d7e62b700 1 -- 172.21.15.70:0/1862749949 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d600b2a70 msgr2=0x7f7d600b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:06.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.975+0000 7f7d7e62b700 1 --2- 172.21.15.70:0/1862749949 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d600b2a70 0x7f7d600b4f30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c009750 tx=0x7f7d6c009310 comp rx=0 tx=0).stop 2024-06-04T05:12:06.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.975+0000 7f7d7e62b700 1 -- 172.21.15.70:0/1862749949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d781035c0 msgr2=0x7f7d78075840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:06.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.975+0000 7f7d7e62b700 1 --2- 172.21.15.70:0/1862749949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d781035c0 0x7f7d78075840 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800b840 tx=0x7f7d6800b870 comp rx=0 tx=0).stop 2024-06-04T05:12:06.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.975+0000 7f7d7e62b700 1 -- 172.21.15.70:0/1862749949 shutdown_connections 2024-06-04T05:12:06.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.975+0000 7f7d7e62b700 1 --2- 172.21.15.70:0/1862749949 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7d600b2a70 0x7f7d600b4f30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.975+0000 7f7d7e62b700 1 --2- 172.21.15.70:0/1862749949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d781095d0 0x7f7d780762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.975+0000 7f7d7e62b700 1 --2- 172.21.15.70:0/1862749949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d781035c0 0x7f7d78075840 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.975+0000 7f7d7e62b700 1 --2- 172.21.15.70:0/1862749949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d78103f80 0x7f7d78075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:06.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.975+0000 7f7d7e62b700 1 -- 172.21.15.70:0/1862749949 >> 172.21.15.70:0/1862749949 conn(0x7f7d7807a610 msgr2=0x7f7d781021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:06.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.975+0000 7f7d7e62b700 1 -- 172.21.15.70:0/1862749949 shutdown_connections 2024-06-04T05:12:06.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:06.975+0000 7f7d7e62b700 1 -- 172.21.15.70:0/1862749949 wait complete. 2024-06-04T05:12:07.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:07 smithi070 ceph-mon[32631]: pgmap v466: 97 pgs: 97 active+clean; 399 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.0 MiB/s wr, 298 op/s 2024-06-04T05:12:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:07 smithi112 ceph-mon[36453]: pgmap v466: 97 pgs: 97 active+clean; 399 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.0 MiB/s wr, 298 op/s 2024-06-04T05:12:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:07 smithi098 ceph-mon[35569]: pgmap v466: 97 pgs: 97 active+clean; 399 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.0 MiB/s wr, 298 op/s 2024-06-04T05:12:07.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 10781 12.82 MB/sec warmup 59 sec latency 85.284 ms 2024-06-04T05:12:07.879 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:12:08.233 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:12:08.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 10925 12.74 MB/sec warmup 60 sec latency 66.382 ms 2024-06-04T05:12:08.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.987+0000 7fa31a03f640 1 -- 172.21.15.70:0/3482879610 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa314109c80 msgr2=0x7fa31410dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:08.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.987+0000 7fa31a03f640 1 --2- 172.21.15.70:0/3482879610 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa314109c80 0x7fa31410dcc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa308009900 tx=0x7fa30802a410 comp rx=0 tx=0).stop 2024-06-04T05:12:08.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.987+0000 7fa31a03f640 1 -- 172.21.15.70:0/3482879610 shutdown_connections 2024-06-04T05:12:08.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.987+0000 7fa31a03f640 1 --2- 172.21.15.70:0/3482879610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa31410e3a0 0x7fa314115f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:08.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.987+0000 7fa31a03f640 1 --2- 172.21.15.70:0/3482879610 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa314109c80 0x7fa31410dcc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:08.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.987+0000 7fa31a03f640 1 --2- 172.21.15.70:0/3482879610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3141092d0 0x7fa3141096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:08.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.987+0000 7fa31a03f640 1 -- 172.21.15.70:0/3482879610 >> 172.21.15.70:0/3482879610 conn(0x7fa314077000 msgr2=0x7fa314079420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:08.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.987+0000 7fa31a03f640 1 -- 172.21.15.70:0/3482879610 shutdown_connections 2024-06-04T05:12:08.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.987+0000 7fa31a03f640 1 -- 172.21.15.70:0/3482879610 wait complete. 2024-06-04T05:12:08.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.988+0000 7fa31a03f640 1 Processor -- start 2024-06-04T05:12:08.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.989+0000 7fa31a03f640 1 -- start start 2024-06-04T05:12:08.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.989+0000 7fa31a03f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3141092d0 0x7fa31419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:08.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.989+0000 7fa31a03f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa31410e3a0 0x7fa31419fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:08.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.989+0000 7fa31a03f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3141a3ff0 0x7fa3141a44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:08.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.989+0000 7fa3137fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3141092d0 0x7fa31419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.989+0000 7fa3137fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3141092d0 0x7fa31419f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46166/0 (socket says 172.21.15.70:46166) 2024-06-04T05:12:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.990+0000 7fa312ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa31410e3a0 0x7fa31419fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.990+0000 7fa312ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa31410e3a0 0x7fa31419fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:38346/0 (socket says 172.21.15.70:38346) 2024-06-04T05:12:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.990+0000 7fa312ffd640 1 -- 172.21.15.70:0/3417343169 learned_addr learned my addr 172.21.15.70:0/3417343169 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.990+0000 7fa31a03f640 1 -- 172.21.15.70:0/3417343169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa3141191c0 con 0x7fa3141092d0 2024-06-04T05:12:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.990+0000 7fa313fff640 1 --2- 172.21.15.70:0/3417343169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3141a3ff0 0x7fa3141a44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.990+0000 7fa31a03f640 1 -- 172.21.15.70:0/3417343169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa314119040 con 0x7fa31410e3a0 2024-06-04T05:12:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.990+0000 7fa31a03f640 1 -- 172.21.15.70:0/3417343169 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa314119340 con 0x7fa3141a3ff0 2024-06-04T05:12:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.990+0000 7fa3137fe640 1 -- 172.21.15.70:0/3417343169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3141a3ff0 msgr2=0x7fa3141a44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.990+0000 7fa3137fe640 1 --2- 172.21.15.70:0/3417343169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3141a3ff0 0x7fa3141a44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.990+0000 7fa3137fe640 1 -- 172.21.15.70:0/3417343169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa31410e3a0 msgr2=0x7fa31419fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.990+0000 7fa3137fe640 1 --2- 172.21.15.70:0/3417343169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa31410e3a0 0x7fa31419fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.990+0000 7fa3137fe640 1 -- 172.21.15.70:0/3417343169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3141a4ae0 con 0x7fa3141092d0 2024-06-04T05:12:08.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.990+0000 7fa312ffd640 1 --2- 172.21.15.70:0/3417343169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa31410e3a0 0x7fa31419fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:08.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.991+0000 7fa3137fe640 1 --2- 172.21.15.70:0/3417343169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3141092d0 0x7fa31419f780 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fa30400d6d0 tx=0x7fa30400dad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:08.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.991+0000 7fa310ff9640 1 -- 172.21.15.70:0/3417343169 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa304017070 con 0x7fa3141092d0 2024-06-04T05:12:08.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.991+0000 7fa310ff9640 1 -- 172.21.15.70:0/3417343169 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa3040047a0 con 0x7fa3141092d0 2024-06-04T05:12:08.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.991+0000 7fa31a03f640 1 -- 172.21.15.70:0/3417343169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3141a4dd0 con 0x7fa3141092d0 2024-06-04T05:12:08.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.991+0000 7fa31a03f640 1 -- 172.21.15.70:0/3417343169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3141ac610 con 0x7fa3141092d0 2024-06-04T05:12:08.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.991+0000 7fa310ff9640 1 -- 172.21.15.70:0/3417343169 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa304009a40 con 0x7fa3141092d0 2024-06-04T05:12:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.994+0000 7fa310ff9640 1 -- 172.21.15.70:0/3417343169 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fa304009be0 con 0x7fa3141092d0 2024-06-04T05:12:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.994+0000 7fa31ce31740 1 Processor -- start 2024-06-04T05:12:08.997 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.995+0000 7fa31ce31740 1 -- start start 2024-06-04T05:12:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.995+0000 7fa31ce31740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.995+0000 7fa31ce31740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df1140 0x555ea0dfb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.995+0000 7fa31ce31740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0dfba00 0x555ea0dfddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.995+0000 7fa31ce31740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555ea0d62c30 con 0x555ea0df0820 2024-06-04T05:12:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.995+0000 7fa31ce31740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555ea0d527d0 con 0x555ea0dfba00 2024-06-04T05:12:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.995+0000 7fa31ce31740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555ea0d56810 con 0x555ea0df1140 2024-06-04T05:12:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.995+0000 7fa313fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.995+0000 7fa313fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46168/0 (socket says 172.21.15.70:46168) 2024-06-04T05:12:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.995+0000 7fa313fff640 1 -- 172.21.15.70:0/1440426786 learned_addr learned my addr 172.21.15.70:0/1440426786 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.995+0000 7fa312ffd640 1 --2- 172.21.15.70:0/1440426786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0dfba00 0x555ea0dfddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.995+0000 7fa3137fe640 1 --2- 172.21.15.70:0/1440426786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df1140 0x555ea0dfb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.996+0000 7fa313fff640 1 -- 172.21.15.70:0/1440426786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df1140 msgr2=0x555ea0dfb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.996+0000 7fa313fff640 1 --2- 172.21.15.70:0/1440426786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df1140 0x555ea0dfb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.996+0000 7fa313fff640 1 -- 172.21.15.70:0/1440426786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0dfba00 msgr2=0x555ea0dfddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.996+0000 7fa313fff640 1 --2- 172.21.15.70:0/1440426786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0dfba00 0x555ea0dfddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.996+0000 7fa313fff640 1 -- 172.21.15.70:0/1440426786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ea0a67210 con 0x555ea0df0820 2024-06-04T05:12:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.996+0000 7fa312ffd640 1 --2- 172.21.15.70:0/1440426786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0dfba00 0x555ea0dfddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:12:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.996+0000 7fa3137fe640 1 --2- 172.21.15.70:0/1440426786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df1140 0x555ea0dfb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:12:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.996+0000 7fa310ff9640 1 --2- 172.21.15.70:0/3417343169 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa2fc0b2a90 0x7fa2fc0b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.996+0000 7fa313fff640 1 --2- 172.21.15.70:0/1440426786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df0c00 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009f60 tx=0x7fa30c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b560350da8f805a3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.996+0000 7fa310ff9640 1 -- 172.21.15.70:0/3417343169 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fa3040e1e10 con 0x7fa3141092d0 2024-06-04T05:12:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.996+0000 7fa3137fe640 1 --2- 172.21.15.70:0/3417343169 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa2fc0b2a90 0x7fa2fc0b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.997+0000 7fa2f9ffb640 1 -- 172.21.15.70:0/1440426786 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c01e070 con 0x555ea0df0820 2024-06-04T05:12:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.997+0000 7fa2f9ffb640 1 -- 172.21.15.70:0/1440426786 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa30c003c30 con 0x555ea0df0820 2024-06-04T05:12:09.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.997+0000 7fa2f9ffb640 1 -- 172.21.15.70:0/1440426786 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c00f840 con 0x555ea0df0820 2024-06-04T05:12:09.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.998+0000 7fa31ce31740 1 -- 172.21.15.70:0/1440426786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 msgr2=0x555ea0df0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:09.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.998+0000 7fa31ce31740 1 --2- 172.21.15.70:0/1440426786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df0c00 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009f60 tx=0x7fa30c009f90 comp rx=0 tx=0).stop 2024-06-04T05:12:09.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.998+0000 7fa3137fe640 1 --2- 172.21.15.70:0/3417343169 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa2fc0b2a90 0x7fa2fc0b4f50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa3040df5e0 tx=0x7fa304004bc0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:09.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.998+0000 7fa31ce31740 1 -- 172.21.15.70:0/1440426786 shutdown_connections 2024-06-04T05:12:09.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.998+0000 7fa31ce31740 1 --2- 172.21.15.70:0/1440426786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0dfba00 0x555ea0dfddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.998+0000 7fa31ce31740 1 --2- 172.21.15.70:0/1440426786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df1140 0x555ea0dfb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.998+0000 7fa31ce31740 1 --2- 172.21.15.70:0/1440426786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df0c00 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.998+0000 7fa31ce31740 1 -- 172.21.15.70:0/1440426786 >> 172.21.15.70:0/1440426786 conn(0x555ea0decd30 msgr2=0x555ea0ded110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.998+0000 7fa31ce31740 1 -- 172.21.15.70:0/1440426786 shutdown_connections 2024-06-04T05:12:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.998+0000 7fa31ce31740 1 -- 172.21.15.70:0/1440426786 wait complete. 2024-06-04T05:12:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.998+0000 7fa31ce31740 1 -- start start 2024-06-04T05:12:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.998+0000 7fa31ce31740 1 Processor -- start 2024-06-04T05:12:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa31ce31740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa31ce31740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df1140 0x555ea0df8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa31ce31740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0dfba00 0x555ea0df9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa313fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa313fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46180/0 (socket says 172.21.15.70:46180) 2024-06-04T05:12:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa313fff640 1 -- 172.21.15.70:0/430042728 learned_addr learned my addr 172.21.15.70:0/430042728 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa31ce31740 1 -- 172.21.15.70:0/430042728 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555ea0a38790 con 0x555ea0df0820 2024-06-04T05:12:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa31ce31740 1 -- 172.21.15.70:0/430042728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555ea0d0e890 con 0x555ea0dfba00 2024-06-04T05:12:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa31ce31740 1 -- 172.21.15.70:0/430042728 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555ea0dfe630 con 0x555ea0df1140 2024-06-04T05:12:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa312ffd640 1 --2- 172.21.15.70:0/430042728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df1140 0x555ea0df8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa31ce31740 10 client.0 _ll_register_callbacks cb 0x555ea0a66a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa313fff640 1 --2- 172.21.15.70:0/430042728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0dfba00 0x555ea0df9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa313fff640 1 -- 172.21.15.70:0/430042728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df1140 msgr2=0x555ea0df8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa313fff640 1 --2- 172.21.15.70:0/430042728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df1140 0x555ea0df8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa313fff640 1 -- 172.21.15.70:0/430042728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0dfba00 msgr2=0x555ea0df9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa313fff640 1 --2- 172.21.15.70:0/430042728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0dfba00 0x555ea0df9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa313fff640 1 -- 172.21.15.70:0/430042728 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ea0b33620 con 0x555ea0df0820 2024-06-04T05:12:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:08.999+0000 7fa312ffd640 1 --2- 172.21.15.70:0/430042728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df1140 0x555ea0df8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.000+0000 7fa313fff640 1 --2- 172.21.15.70:0/430042728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df89f0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009b80 tx=0x7fa30c009af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddbdd5533d0cd894 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.000+0000 7fa2f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:12:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.000+0000 7fa2f2ffd640 1 -- 172.21.15.70:0/430042728 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c02e060 con 0x555ea0df0820 2024-06-04T05:12:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.000+0000 7fa31ce31740 1 -- 172.21.15.70:0/430042728 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa30c003c30 con 0x555ea0df0820 2024-06-04T05:12:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.000+0000 7fa2f2ffd640 1 -- 172.21.15.70:0/430042728 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fa30c004480 con 0x555ea0df0820 2024-06-04T05:12:09.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.000+0000 7fa2f2ffd640 1 -- 172.21.15.70:0/430042728 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fa30c01e040 con 0x555ea0df0820 2024-06-04T05:12:09.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.000+0000 7fa31ce31740 10 client.19392 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:09.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.000+0000 7fa31ce31740 1 -- 172.21.15.70:0/430042728 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x555ea0e0a3c0 con 0x555ea0df0820 2024-06-04T05:12:09.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.001+0000 7fa2f2ffd640 1 -- 172.21.15.70:0/430042728 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa30c003c30 con 0x555ea0df0820 2024-06-04T05:12:09.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.002+0000 7fa2f2ffd640 1 -- 172.21.15.70:0/430042728 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7fa30c026440 con 0x555ea0df0820 2024-06-04T05:12:09.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.002+0000 7fa31ce31740 10 client.19392 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:09.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.002+0000 7fa31ce31740 10 client.19392 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:12:09.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.002+0000 7fa31ce31740 20 client.19392 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:09.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.002+0000 7fa31ce31740 1 --2- 172.21.15.70:0/430042728 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555ea0e0b820 0x555ea0e0dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:09.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.002+0000 7fa31ce31740 4 client.19392 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:12:09.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.002+0000 7fa31ce31740 1 -- 172.21.15.70:0/430042728 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ea0d64b80 con 0x555ea0e0b820 2024-06-04T05:12:09.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.002+0000 7fa312ffd640 1 --2- 172.21.15.70:0/430042728 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555ea0e0b820 0x555ea0e0dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:09.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.002+0000 7fa2f2ffd640 10 client.19392 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:09.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.002+0000 7fa312ffd640 1 --2- 172.21.15.70:0/430042728 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555ea0e0b820 0x555ea0e0dc10 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af1ab68849235dc3 server_cookie=e46ef5a9cfcc7bf2 in_seq=0 out_seq=0 2024-06-04T05:12:09.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.003+0000 7fa2f2ffd640 1 -- 172.21.15.70:0/430042728 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555ea0d64b80 con 0x555ea0e0b820 2024-06-04T05:12:09.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.003+0000 7fa2f2ffd640 10 client.19392 handle_command_reply: tid=0 2024-06-04T05:12:09.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.003+0000 7fa31ce31740 1 client.19392 shutdown 2024-06-04T05:12:09.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.003+0000 7fa31ce31740 1 -- 172.21.15.70:0/430042728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 msgr2=0x555ea0df89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:09.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.003+0000 7fa31ce31740 1 --2- 172.21.15.70:0/430042728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df89f0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009b80 tx=0x7fa30c009af0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.003+0000 7fa31ce31740 1 -- 172.21.15.70:0/430042728 shutdown_connections 2024-06-04T05:12:09.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.003+0000 7fa31ce31740 1 --2- 172.21.15.70:0/430042728 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555ea0e0b820 0x555ea0e0dc10 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.004+0000 7fa31ce31740 1 --2- 172.21.15.70:0/430042728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0dfba00 0x555ea0df9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.004+0000 7fa2f2ffd640 0 client.19392 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:09.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.004+0000 7fa31ce31740 1 --2- 172.21.15.70:0/430042728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df1140 0x555ea0df8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.004+0000 7fa31ce31740 1 --2- 172.21.15.70:0/430042728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df89f0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.004+0000 7fa31ce31740 1 -- 172.21.15.70:0/430042728 >> 172.21.15.70:0/430042728 conn(0x555ea0decd30 msgr2=0x555ea0debfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:09.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.004+0000 7fa31ce31740 1 -- 172.21.15.70:0/430042728 shutdown_connections 2024-06-04T05:12:09.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.004+0000 7fa31ce31740 1 -- 172.21.15.70:0/430042728 wait complete. 2024-06-04T05:12:09.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.004+0000 7fa31ce31740 20 client.19392 trim_cache size 0 max 16384 2024-06-04T05:12:09.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.031+0000 7fa31ce31740 1 Processor -- start 2024-06-04T05:12:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.031+0000 7fa31ce31740 1 -- start start 2024-06-04T05:12:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.031+0000 7fa31ce31740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.031+0000 7fa31ce31740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0df1140 0x555ea0e133f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.031+0000 7fa313fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.031+0000 7fa313fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46190/0 (socket says 172.21.15.70:46190) 2024-06-04T05:12:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.031+0000 7fa313fff640 1 -- 172.21.15.70:0/2521670555 learned_addr learned my addr 172.21.15.70:0/2521670555 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.031+0000 7fa31ce31740 1 --2- 172.21.15.70:0/2521670555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0dfba00 0x555ea0e13930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.031+0000 7fa31ce31740 1 -- 172.21.15.70:0/2521670555 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa3141b4d60 con 0x555ea0df0820 2024-06-04T05:12:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.031+0000 7fa31ce31740 1 -- 172.21.15.70:0/2521670555 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa3141b6a20 con 0x555ea0df1140 2024-06-04T05:12:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.031+0000 7fa31ce31740 1 -- 172.21.15.70:0/2521670555 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa3141b5430 con 0x555ea0dfba00 2024-06-04T05:12:09.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.031+0000 7fa312ffd640 1 --2- 172.21.15.70:0/2521670555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0df1140 0x555ea0e133f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:09.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.032+0000 7fa313fff640 1 --2- 172.21.15.70:0/2521670555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0dfba00 0x555ea0e13930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:09.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.032+0000 7fa313fff640 1 -- 172.21.15.70:0/2521670555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0dfba00 msgr2=0x555ea0e13930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:09.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.032+0000 7fa313fff640 1 --2- 172.21.15.70:0/2521670555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0dfba00 0x555ea0e13930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.032+0000 7fa313fff640 1 -- 172.21.15.70:0/2521670555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0df1140 msgr2=0x555ea0e133f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:09.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.032+0000 7fa313fff640 1 --2- 172.21.15.70:0/2521670555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0df1140 0x555ea0e133f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.032+0000 7fa313fff640 1 -- 172.21.15.70:0/2521670555 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ea0df4590 con 0x555ea0df0820 2024-06-04T05:12:09.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.032+0000 7fa313fff640 1 --2- 172.21.15.70:0/2521670555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df9290 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fa30c005d70 tx=0x7fa30c009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d88c9c900cc6603d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:09.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.032+0000 7fa2f97fa640 1 -- 172.21.15.70:0/2521670555 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c02e070 con 0x555ea0df0820 2024-06-04T05:12:09.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.032+0000 7fa2f97fa640 1 -- 172.21.15.70:0/2521670555 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa30c011d60 con 0x555ea0df0820 2024-06-04T05:12:09.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.032+0000 7fa2f97fa640 1 -- 172.21.15.70:0/2521670555 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c00f480 con 0x555ea0df0820 2024-06-04T05:12:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.033+0000 7fa31ce31740 1 -- 172.21.15.70:0/2521670555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 msgr2=0x555ea0df9290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.033+0000 7fa31ce31740 1 --2- 172.21.15.70:0/2521670555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df9290 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fa30c005d70 tx=0x7fa30c009900 comp rx=0 tx=0).stop 2024-06-04T05:12:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.033+0000 7fa31ce31740 1 -- 172.21.15.70:0/2521670555 shutdown_connections 2024-06-04T05:12:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.033+0000 7fa31ce31740 1 --2- 172.21.15.70:0/2521670555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0df1140 0x555ea0e133f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.033+0000 7fa31ce31740 1 --2- 172.21.15.70:0/2521670555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0dfba00 0x555ea0e13930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.033+0000 7fa31ce31740 1 --2- 172.21.15.70:0/2521670555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0df0820 0x555ea0df9290 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.033+0000 7fa31ce31740 1 -- 172.21.15.70:0/2521670555 >> 172.21.15.70:0/2521670555 conn(0x555ea0decd30 msgr2=0x555ea0de7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.033+0000 7fa31ce31740 1 -- 172.21.15.70:0/2521670555 shutdown_connections 2024-06-04T05:12:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.034+0000 7fa31ce31740 1 -- 172.21.15.70:0/2521670555 wait complete. 2024-06-04T05:12:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.034+0000 7fa31ce31740 1 -- start start 2024-06-04T05:12:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.034+0000 7fa31ce31740 1 Processor -- start 2024-06-04T05:12:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.035+0000 7fa31ce31740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df0820 0x555ea0dfee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.035+0000 7fa31ce31740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0df1140 0x555ea0dff3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.035+0000 7fa31ce31740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0dfba00 0x555ea0e742b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.035+0000 7fa31ce31740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa3141b6290 con 0x555ea0dfba00 2024-06-04T05:12:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.035+0000 7fa31ce31740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa3141b5b60 con 0x555ea0df1140 2024-06-04T05:12:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.035+0000 7fa31ce31740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa3141b2d60 con 0x555ea0df0820 2024-06-04T05:12:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.035+0000 7fa31ce31740 10 client.0 _ll_register_callbacks cb 0x7fa30c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.035+0000 7fa312ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0df1140 0x555ea0dff3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.035+0000 7fa313fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df0820 0x555ea0dfee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.035+0000 7fa313fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df0820 0x555ea0dfee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:42446/0 (socket says 172.21.15.70:42446) 2024-06-04T05:12:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.035+0000 7fa313fff640 1 -- 172.21.15.70:0/1292846891 learned_addr learned my addr 172.21.15.70:0/1292846891 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.035+0000 7fa313fff640 1 --2- 172.21.15.70:0/1292846891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0dfba00 0x555ea0e742b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.036+0000 7fa312ffd640 1 -- 172.21.15.70:0/1292846891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df0820 msgr2=0x555ea0dfee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.036+0000 7fa312ffd640 1 --2- 172.21.15.70:0/1292846891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df0820 0x555ea0dfee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.036+0000 7fa312ffd640 1 -- 172.21.15.70:0/1292846891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0dfba00 msgr2=0x555ea0e742b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.036+0000 7fa312ffd640 1 --2- 172.21.15.70:0/1292846891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0dfba00 0x555ea0e742b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.036+0000 7fa312ffd640 1 -- 172.21.15.70:0/1292846891 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ea0e747f0 con 0x555ea0df1140 2024-06-04T05:12:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.036+0000 7fa312ffd640 1 --2- 172.21.15.70:0/1292846891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0df1140 0x555ea0dff3b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa308000c00 tx=0x7fa308003a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acd1b245c2d9d70c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.036+0000 7fa2f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:12:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.036+0000 7fa2f2ffd640 1 -- 172.21.15.70:0/1292846891 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30804a070 con 0x555ea0df1140 2024-06-04T05:12:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.036+0000 7fa31ce31740 1 -- 172.21.15.70:0/1292846891 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa30c011d60 con 0x555ea0df1140 2024-06-04T05:12:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.036+0000 7fa2f2ffd640 1 -- 172.21.15.70:0/1292846891 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fa308046290 con 0x555ea0df1140 2024-06-04T05:12:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.036+0000 7fa2f2ffd640 1 -- 172.21.15.70:0/1292846891 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fa308035040 con 0x555ea0df1140 2024-06-04T05:12:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.036+0000 7fa31ce31740 10 client.28268 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.036+0000 7fa31ce31740 1 -- 172.21.15.70:0/1292846891 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x555ea0e75b90 con 0x555ea0df1140 2024-06-04T05:12:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.037+0000 7fa2f2ffd640 1 -- 172.21.15.70:0/1292846891 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa30c011d60 con 0x555ea0df1140 2024-06-04T05:12:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.037+0000 7fa2f2ffd640 1 -- 172.21.15.70:0/1292846891 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7fa308056070 con 0x555ea0df1140 2024-06-04T05:12:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.037+0000 7fa31ce31740 10 client.28268 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.037+0000 7fa31ce31740 10 client.28268 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:12:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.037+0000 7fa31ce31740 20 client.28268 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:09.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.037+0000 7fa31ce31740 1 --2- 172.21.15.70:0/1292846891 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555ea0e0b820 0x555ea0e783f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:09.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.037+0000 7fa31ce31740 4 client.28268 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:12:09.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.037+0000 7fa31ce31740 1 -- 172.21.15.70:0/1292846891 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555ea0d5e360 con 0x555ea0e0b820 2024-06-04T05:12:09.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.037+0000 7fa313fff640 1 --2- 172.21.15.70:0/1292846891 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555ea0e0b820 0x555ea0e783f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:09.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.038+0000 7fa313fff640 1 --2- 172.21.15.70:0/1292846891 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555ea0e0b820 0x555ea0e783f0 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd4f7eb5a6d5361c server_cookie=9bd3d8b27322667b in_seq=0 out_seq=0 2024-06-04T05:12:09.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.038+0000 7fa2f2ffd640 10 client.28268 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:09.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:08 smithi070 ceph-mon[32631]: osdmap e98: 12 total, 12 up, 12 in 2024-06-04T05:12:09.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:08 smithi070 ceph-mon[32631]: pgmap v468: 97 pgs: 97 active+clean; 399 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.3 MiB/s wr, 373 op/s 2024-06-04T05:12:09.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.113+0000 7fa2f2ffd640 1 -- 172.21.15.70:0/1292846891 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x555ea0d5e360 con 0x555ea0e0b820 2024-06-04T05:12:09.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.113+0000 7fa2f2ffd640 10 client.28268 handle_command_reply: tid=0 2024-06-04T05:12:09.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.113+0000 7fa31ce31740 1 client.28268 shutdown 2024-06-04T05:12:09.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.113+0000 7fa31ce31740 1 -- 172.21.15.70:0/1292846891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0df1140 msgr2=0x555ea0dff3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.113+0000 7fa31ce31740 1 --2- 172.21.15.70:0/1292846891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0df1140 0x555ea0dff3b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa308000c00 tx=0x7fa308003a60 comp rx=0 tx=0).stop 2024-06-04T05:12:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.113+0000 7fa31ce31740 1 -- 172.21.15.70:0/1292846891 shutdown_connections 2024-06-04T05:12:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.113+0000 7fa31ce31740 1 --2- 172.21.15.70:0/1292846891 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x555ea0e0b820 0x555ea0e783f0 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.113+0000 7fa31ce31740 1 --2- 172.21.15.70:0/1292846891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ea0df1140 0x555ea0dff3b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.113+0000 7fa31ce31740 1 --2- 172.21.15.70:0/1292846891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ea0dfba00 0x555ea0e742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.113+0000 7fa31ce31740 1 --2- 172.21.15.70:0/1292846891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555ea0df0820 0x555ea0dfee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.113+0000 7fa31ce31740 1 -- 172.21.15.70:0/1292846891 >> 172.21.15.70:0/1292846891 conn(0x555ea0decd30 msgr2=0x555ea0df21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.113+0000 7fa2f2ffd640 0 client.28268 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.113+0000 7fa31ce31740 1 -- 172.21.15.70:0/1292846891 shutdown_connections 2024-06-04T05:12:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.113+0000 7fa31ce31740 1 -- 172.21.15.70:0/1292846891 wait complete. 2024-06-04T05:12:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.113+0000 7fa31ce31740 20 client.28268 trim_cache size 0 max 16384 2024-06-04T05:12:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:12:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:12:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:12:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.114+0000 7fa31a03f640 1 -- 172.21.15.70:0/3417343169 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa2fc0b2a90 msgr2=0x7fa2fc0b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.114+0000 7fa31a03f640 1 --2- 172.21.15.70:0/3417343169 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa2fc0b2a90 0x7fa2fc0b4f50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa3040df5e0 tx=0x7fa304004bc0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.114+0000 7fa31a03f640 1 -- 172.21.15.70:0/3417343169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3141092d0 msgr2=0x7fa31419f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.114+0000 7fa31a03f640 1 --2- 172.21.15.70:0/3417343169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3141092d0 0x7fa31419f780 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fa30400d6d0 tx=0x7fa30400dad0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.114+0000 7fa31a03f640 1 -- 172.21.15.70:0/3417343169 shutdown_connections 2024-06-04T05:12:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.114+0000 7fa31a03f640 1 --2- 172.21.15.70:0/3417343169 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa2fc0b2a90 0x7fa2fc0b4f50 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.114+0000 7fa31a03f640 1 --2- 172.21.15.70:0/3417343169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa31410e3a0 0x7fa31419fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.114+0000 7fa31a03f640 1 --2- 172.21.15.70:0/3417343169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3141a3ff0 0x7fa3141a44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.114+0000 7fa31a03f640 1 --2- 172.21.15.70:0/3417343169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3141092d0 0x7fa31419f780 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.114+0000 7fa31a03f640 1 -- 172.21.15.70:0/3417343169 >> 172.21.15.70:0/3417343169 conn(0x7fa314077000 msgr2=0x7fa314078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.114+0000 7fa31a03f640 1 -- 172.21.15.70:0/3417343169 shutdown_connections 2024-06-04T05:12:09.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:09.114+0000 7fa31a03f640 1 -- 172.21.15.70:0/3417343169 wait complete. 2024-06-04T05:12:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:08 smithi112 ceph-mon[36453]: osdmap e98: 12 total, 12 up, 12 in 2024-06-04T05:12:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:08 smithi112 ceph-mon[36453]: pgmap v468: 97 pgs: 97 active+clean; 399 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.3 MiB/s wr, 373 op/s 2024-06-04T05:12:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:08 smithi098 ceph-mon[35569]: osdmap e98: 12 total, 12 up, 12 in 2024-06-04T05:12:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:08 smithi098 ceph-mon[35569]: pgmap v468: 97 pgs: 97 active+clean; 399 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.3 MiB/s wr, 373 op/s 2024-06-04T05:12:09.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 11061 12.60 MB/sec warmup 61 sec latency 90.116 ms 2024-06-04T05:12:10.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.639+0000 7f5694ec7700 1 -- 172.21.15.70:0/2117557084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5690103600 msgr2=0x7f569010f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:10.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.639+0000 7f5694ec7700 1 --2- 172.21.15.70:0/2117557084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5690103600 0x7f569010f2e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5688009f90 tx=0x7f568800b330 comp rx=0 tx=0).stop 2024-06-04T05:12:10.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.639+0000 7f5694ec7700 1 -- 172.21.15.70:0/2117557084 shutdown_connections 2024-06-04T05:12:10.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.639+0000 7f5694ec7700 1 --2- 172.21.15.70:0/2117557084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5690103600 0x7f569010f2e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.639+0000 7f5694ec7700 1 --2- 172.21.15.70:0/2117557084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56901084c0 0x7f56901088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.639+0000 7f5694ec7700 1 --2- 172.21.15.70:0/2117557084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5690102c40 0x7f56901030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.639+0000 7f5694ec7700 1 -- 172.21.15.70:0/2117557084 >> 172.21.15.70:0/2117557084 conn(0x7f5690078ab0 msgr2=0x7f56900ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:10.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.639+0000 7f5694ec7700 1 -- 172.21.15.70:0/2117557084 shutdown_connections 2024-06-04T05:12:10.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.639+0000 7f5694ec7700 1 -- 172.21.15.70:0/2117557084 wait complete. 2024-06-04T05:12:10.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.640+0000 7f5694ec7700 1 Processor -- start 2024-06-04T05:12:10.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.640+0000 7f5694ec7700 1 -- start start 2024-06-04T05:12:10.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.641+0000 7f5694ec7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5690102c40 0x7f56901a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.641+0000 7f5694ec7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5690103600 0x7f56901a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.641+0000 7f5694ec7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56901084c0 0x7f569019bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.641+0000 7f5694ec7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5690111450 con 0x7f56901084c0 2024-06-04T05:12:10.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.641+0000 7f5694ec7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f56901112d0 con 0x7f5690103600 2024-06-04T05:12:10.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.641+0000 7f5694ec7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f56901115d0 con 0x7f5690102c40 2024-06-04T05:12:10.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.641+0000 7f568e59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5690102c40 0x7f56901a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.641+0000 7f568e59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5690102c40 0x7f56901a1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:42452/0 (socket says 172.21.15.70:42452) 2024-06-04T05:12:10.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.641+0000 7f568e59c700 1 -- 172.21.15.70:0/404936197 learned_addr learned my addr 172.21.15.70:0/404936197 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:10.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.641+0000 7f568dd9b700 1 --2- 172.21.15.70:0/404936197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5690103600 0x7f56901a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.641+0000 7f568ed9d700 1 --2- 172.21.15.70:0/404936197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56901084c0 0x7f569019bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.642+0000 7f568dd9b700 1 -- 172.21.15.70:0/404936197 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5690102c40 msgr2=0x7f56901a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:10.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.642+0000 7f568dd9b700 1 --2- 172.21.15.70:0/404936197 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5690102c40 0x7f56901a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.642+0000 7f568dd9b700 1 -- 172.21.15.70:0/404936197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56901084c0 msgr2=0x7f569019bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:10.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.642+0000 7f568dd9b700 1 --2- 172.21.15.70:0/404936197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56901084c0 0x7f569019bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.642+0000 7f568dd9b700 1 -- 172.21.15.70:0/404936197 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f569019c260 con 0x7f5690103600 2024-06-04T05:12:10.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.642+0000 7f568ed9d700 1 --2- 172.21.15.70:0/404936197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56901084c0 0x7f569019bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:12:10.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.642+0000 7f568dd9b700 1 --2- 172.21.15.70:0/404936197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5690103600 0x7f56901a1e80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f568400e9f0 tx=0x7f568400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:10.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.642+0000 7f567f7fe700 1 -- 172.21.15.70:0/404936197 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568400cbd0 con 0x7f5690103600 2024-06-04T05:12:10.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.642+0000 7f567f7fe700 1 -- 172.21.15.70:0/404936197 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f56840044b0 con 0x7f5690103600 2024-06-04T05:12:10.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.642+0000 7f5694ec7700 1 -- 172.21.15.70:0/404936197 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f569019c550 con 0x7f5690103600 2024-06-04T05:12:10.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.643+0000 7f567f7fe700 1 -- 172.21.15.70:0/404936197 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5684005020 con 0x7f5690103600 2024-06-04T05:12:10.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.643+0000 7f5694ec7700 1 -- 172.21.15.70:0/404936197 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56901a8810 con 0x7f5690103600 2024-06-04T05:12:10.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.646+0000 7f567f7fe700 1 -- 172.21.15.70:0/404936197 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f56840051c0 con 0x7f5690103600 2024-06-04T05:12:10.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.647+0000 7f567f7fe700 1 --2- 172.21.15.70:0/404936197 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56780b2a70 0x7f56780b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.647+0000 7f568e59c700 1 --2- 172.21.15.70:0/404936197 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56780b2a70 0x7f56780b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.647+0000 7f567f7fe700 1 -- 172.21.15.70:0/404936197 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f56840e2530 con 0x7f5690103600 2024-06-04T05:12:10.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.648+0000 7f568e59c700 1 --2- 172.21.15.70:0/404936197 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56780b2a70 0x7f56780b4f30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5680009750 tx=0x7f5680009310 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:10.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.648+0000 7f569f75cb80 1 Processor -- start 2024-06-04T05:12:10.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.649+0000 7f569f75cb80 1 -- start start 2024-06-04T05:12:10.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.649+0000 7f569f75cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add2a860 0x55c3add2ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.649+0000 7f569f75cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2b210 0x55c3add2e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.649+0000 7f569f75cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2ef60 0x55c3add2f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.649+0000 7f569f75cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c3adc89c90 con 0x55c3add2a860 2024-06-04T05:12:10.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.649+0000 7f568dd9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2b210 0x55c3add2e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.649+0000 7f568dd9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2b210 0x55c3add2e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:42464/0 (socket says 172.21.15.70:42464) 2024-06-04T05:12:10.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.649+0000 7f568dd9b700 1 -- 172.21.15.70:0/2905535312 learned_addr learned my addr 172.21.15.70:0/2905535312 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:10.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.649+0000 7f568ed9d700 1 --2- 172.21.15.70:0/2905535312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2ef60 0x55c3add2f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.649+0000 7f568e59c700 1 --2- 172.21.15.70:0/2905535312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add2a860 0x55c3add2ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.649+0000 7f569f75cb80 1 -- 172.21.15.70:0/2905535312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c3adc79830 con 0x55c3add2ef60 2024-06-04T05:12:10.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.649+0000 7f569f75cb80 1 -- 172.21.15.70:0/2905535312 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c3adc7d870 con 0x55c3add2b210 2024-06-04T05:12:10.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.650+0000 7f568dd9b700 1 -- 172.21.15.70:0/2905535312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2ef60 msgr2=0x55c3add2f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:10.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.650+0000 7f568dd9b700 1 --2- 172.21.15.70:0/2905535312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2ef60 0x55c3add2f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.650+0000 7f568dd9b700 1 -- 172.21.15.70:0/2905535312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add2a860 msgr2=0x55c3add2ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:10.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.650+0000 7f568dd9b700 1 --2- 172.21.15.70:0/2905535312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add2a860 0x55c3add2ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.650+0000 7f568dd9b700 1 -- 172.21.15.70:0/2905535312 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3adbc98d0 con 0x55c3add2b210 2024-06-04T05:12:10.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.650+0000 7f568dd9b700 1 --2- 172.21.15.70:0/2905535312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2b210 0x55c3add2e940 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f56840a9770 tx=0x7f56840a9250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fab372accb91804d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:10.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.650+0000 7f567cff9700 1 -- 172.21.15.70:0/2905535312 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568401c070 con 0x55c3add2b210 2024-06-04T05:12:10.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.650+0000 7f567cff9700 1 -- 172.21.15.70:0/2905535312 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f56840a92f0 con 0x55c3add2b210 2024-06-04T05:12:10.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.650+0000 7f567cff9700 1 -- 172.21.15.70:0/2905535312 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56840adb90 con 0x55c3add2b210 2024-06-04T05:12:10.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.651+0000 7f569f75cb80 1 -- 172.21.15.70:0/2905535312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2b210 msgr2=0x55c3add2e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:10.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.651+0000 7f569f75cb80 1 --2- 172.21.15.70:0/2905535312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2b210 0x55c3add2e940 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f56840a9770 tx=0x7f56840a9250 comp rx=0 tx=0).stop 2024-06-04T05:12:10.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.651+0000 7f569f75cb80 1 -- 172.21.15.70:0/2905535312 shutdown_connections 2024-06-04T05:12:10.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.651+0000 7f569f75cb80 1 --2- 172.21.15.70:0/2905535312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2ef60 0x55c3add2f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.651+0000 7f569f75cb80 1 --2- 172.21.15.70:0/2905535312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add2a860 0x55c3add2ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.651+0000 7f569f75cb80 1 --2- 172.21.15.70:0/2905535312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2b210 0x55c3add2e940 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.651+0000 7f569f75cb80 1 -- 172.21.15.70:0/2905535312 >> 172.21.15.70:0/2905535312 conn(0x55c3adc58080 msgr2=0x55c3add24e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:10.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.651+0000 7f569f75cb80 1 -- 172.21.15.70:0/2905535312 shutdown_connections 2024-06-04T05:12:10.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.652+0000 7f569f75cb80 1 -- 172.21.15.70:0/2905535312 wait complete. 2024-06-04T05:12:10.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.652+0000 7f569f75cb80 1 -- start start 2024-06-04T05:12:10.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.652+0000 7f569f75cb80 1 Processor -- start 2024-06-04T05:12:10.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.652+0000 7f569f75cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2a860 0x55c3add363e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f569f75cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add2b210 0x55c3add336e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f569f75cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2ef60 0x55c3add33c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f569f75cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c3adb7ffa0 con 0x55c3add2b210 2024-06-04T05:12:10.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f569f75cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c3adc1b7c0 con 0x55c3add2ef60 2024-06-04T05:12:10.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f569f75cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c3add37120 con 0x55c3add2a860 2024-06-04T05:12:10.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f569f75cb80 10 client.0 _ll_register_callbacks cb 0x55c3adc14920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:10.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f568e59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2a860 0x55c3add363e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f568e59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2a860 0x55c3add363e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:42466/0 (socket says 172.21.15.70:42466) 2024-06-04T05:12:10.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f568e59c700 1 -- 172.21.15.70:0/132275868 learned_addr learned my addr 172.21.15.70:0/132275868 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:10.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f568dd9b700 1 --2- 172.21.15.70:0/132275868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add2b210 0x55c3add336e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f568ed9d700 1 --2- 172.21.15.70:0/132275868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2ef60 0x55c3add33c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f568e59c700 1 -- 172.21.15.70:0/132275868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2ef60 msgr2=0x55c3add33c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:10.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f568e59c700 1 --2- 172.21.15.70:0/132275868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2ef60 0x55c3add33c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f568e59c700 1 -- 172.21.15.70:0/132275868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add2b210 msgr2=0x55c3add336e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:10.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f568e59c700 1 --2- 172.21.15.70:0/132275868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add2b210 0x55c3add336e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f568e59c700 1 -- 172.21.15.70:0/132275868 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3adc18690 con 0x55c3add2a860 2024-06-04T05:12:10.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f568ed9d700 1 --2- 172.21.15.70:0/132275868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2ef60 0x55c3add33c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:12:10.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f568e59c700 1 --2- 172.21.15.70:0/132275868 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2a860 0x55c3add363e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f568000ecd0 tx=0x7f56800047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90e313699494f928 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:10.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f5675ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:12:10.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.653+0000 7f5675ffb700 1 -- 172.21.15.70:0/132275868 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568002b060 con 0x55c3add2a860 2024-06-04T05:12:10.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.654+0000 7f5675ffb700 1 -- 172.21.15.70:0/132275868 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f568001dc60 con 0x55c3add2a860 2024-06-04T05:12:10.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.654+0000 7f569f75cb80 1 -- 172.21.15.70:0/132275868 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56840a92f0 con 0x55c3add2a860 2024-06-04T05:12:10.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.654+0000 7f5675ffb700 1 -- 172.21.15.70:0/132275868 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f568001de60 con 0x55c3add2a860 2024-06-04T05:12:10.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.654+0000 7f569f75cb80 10 client.28399 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:10.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.654+0000 7f569f75cb80 1 -- 172.21.15.70:0/132275868 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55c3add4a0c0 con 0x55c3add2a860 2024-06-04T05:12:10.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.654+0000 7f5675ffb700 1 -- 172.21.15.70:0/132275868 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f56840a92f0 con 0x55c3add2a860 2024-06-04T05:12:10.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.655+0000 7f5675ffb700 1 -- 172.21.15.70:0/132275868 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7f568001c490 con 0x55c3add2a860 2024-06-04T05:12:10.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.656+0000 7f569f75cb80 10 client.28399 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:10.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.656+0000 7f569f75cb80 10 client.28399 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:12:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.656+0000 7f569f75cb80 20 client.28399 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.656+0000 7f569f75cb80 1 --2- 172.21.15.70:0/132275868 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55c3add4c6a0 0x55c3add4eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.656+0000 7f569f75cb80 4 client.28399 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:12:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.656+0000 7f569f75cb80 1 -- 172.21.15.70:0/132275868 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3adcf69e0 con 0x55c3add4c6a0 2024-06-04T05:12:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.656+0000 7f568dd9b700 1 --2- 172.21.15.70:0/132275868 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55c3add4c6a0 0x55c3add4eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.656+0000 7f5675ffb700 10 client.28399 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:12:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.656+0000 7f568dd9b700 1 --2- 172.21.15.70:0/132275868 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55c3add4c6a0 0x55c3add4eab0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7f1bd48ffbd5be7 server_cookie=dfcc869ada03afdb in_seq=0 out_seq=0 2024-06-04T05:12:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.657+0000 7f5675ffb700 1 -- 172.21.15.70:0/132275868 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c3adcf69e0 con 0x55c3add4c6a0 2024-06-04T05:12:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.657+0000 7f5675ffb700 10 client.28399 handle_command_reply: tid=0 2024-06-04T05:12:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.657+0000 7f569f75cb80 1 client.28399 shutdown 2024-06-04T05:12:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.657+0000 7f569f75cb80 1 -- 172.21.15.70:0/132275868 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2a860 msgr2=0x55c3add363e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.657+0000 7f569f75cb80 1 --2- 172.21.15.70:0/132275868 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2a860 0x55c3add363e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f568000ecd0 tx=0x7f56800047c0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.657+0000 7f569f75cb80 1 -- 172.21.15.70:0/132275868 shutdown_connections 2024-06-04T05:12:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.657+0000 7f569f75cb80 1 --2- 172.21.15.70:0/132275868 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55c3add4c6a0 0x55c3add4eab0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.657+0000 7f5675ffb700 0 client.28399 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:12:10.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.657+0000 7f569f75cb80 1 --2- 172.21.15.70:0/132275868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2ef60 0x55c3add33c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.657+0000 7f569f75cb80 1 --2- 172.21.15.70:0/132275868 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2a860 0x55c3add363e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.657+0000 7f569f75cb80 1 --2- 172.21.15.70:0/132275868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add2b210 0x55c3add336e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.657+0000 7f569f75cb80 1 -- 172.21.15.70:0/132275868 >> 172.21.15.70:0/132275868 conn(0x55c3adc58080 msgr2=0x55c3add30350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:10.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.657+0000 7f569f75cb80 1 -- 172.21.15.70:0/132275868 shutdown_connections 2024-06-04T05:12:10.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.657+0000 7f569f75cb80 1 -- 172.21.15.70:0/132275868 wait complete. 2024-06-04T05:12:10.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.658+0000 7f569f75cb80 20 client.28399 trim_cache size 0 max 16384 2024-06-04T05:12:10.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.683+0000 7f569f75cb80 1 Processor -- start 2024-06-04T05:12:10.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.684+0000 7f569f75cb80 1 -- start start 2024-06-04T05:12:10.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.684+0000 7f569f75cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add2a860 0x55c3add54380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.684+0000 7f569f75cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2b210 0x55c3add548c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.684+0000 7f569f75cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2ef60 0x55c3add34f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.684+0000 7f568e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add2a860 0x55c3add54380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.684+0000 7f568e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add2a860 0x55c3add54380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46228/0 (socket says 172.21.15.70:46228) 2024-06-04T05:12:10.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.684+0000 7f568e59c700 1 -- 172.21.15.70:0/2845707366 learned_addr learned my addr 172.21.15.70:0/2845707366 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:10.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.684+0000 7f569f75cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f56901abc70 con 0x55c3add2a860 2024-06-04T05:12:10.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.684+0000 7f569f75cb80 1 -- 172.21.15.70:0/2845707366 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f56901ad930 con 0x55c3add2b210 2024-06-04T05:12:10.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.684+0000 7f569f75cb80 1 -- 172.21.15.70:0/2845707366 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f56901ac340 con 0x55c3add2ef60 2024-06-04T05:12:10.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.684+0000 7f568dd9b700 1 --2- 172.21.15.70:0/2845707366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2b210 0x55c3add548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.684+0000 7f568ed9d700 1 --2- 172.21.15.70:0/2845707366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2ef60 0x55c3add34f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.685+0000 7f568e59c700 1 -- 172.21.15.70:0/2845707366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2ef60 msgr2=0x55c3add34f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:10.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.685+0000 7f568e59c700 1 --2- 172.21.15.70:0/2845707366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2ef60 0x55c3add34f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.685+0000 7f568e59c700 1 -- 172.21.15.70:0/2845707366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2b210 msgr2=0x55c3add548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:10.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.685+0000 7f568e59c700 1 --2- 172.21.15.70:0/2845707366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2b210 0x55c3add548c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.685+0000 7f568e59c700 1 -- 172.21.15.70:0/2845707366 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3add345c0 con 0x55c3add2a860 2024-06-04T05:12:10.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.685+0000 7f568e59c700 1 --2- 172.21.15.70:0/2845707366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add2a860 0x55c3add54380 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f5680009480 tx=0x7f5680011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd8edbd22602554 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:10.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.685+0000 7f5677fff700 1 -- 172.21.15.70:0/2845707366 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5680035080 con 0x55c3add2a860 2024-06-04T05:12:10.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.685+0000 7f5677fff700 1 -- 172.21.15.70:0/2845707366 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f568001dc60 con 0x55c3add2a860 2024-06-04T05:12:10.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.685+0000 7f5677fff700 1 -- 172.21.15.70:0/2845707366 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5680030b10 con 0x55c3add2a860 2024-06-04T05:12:10.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.686+0000 7f569f75cb80 1 -- 172.21.15.70:0/2845707366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add2a860 msgr2=0x55c3add54380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:10.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.686+0000 7f569f75cb80 1 --2- 172.21.15.70:0/2845707366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add2a860 0x55c3add54380 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f5680009480 tx=0x7f5680011ab0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.686+0000 7f569f75cb80 1 -- 172.21.15.70:0/2845707366 shutdown_connections 2024-06-04T05:12:10.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.686+0000 7f569f75cb80 1 --2- 172.21.15.70:0/2845707366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2ef60 0x55c3add34f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.686+0000 7f569f75cb80 1 --2- 172.21.15.70:0/2845707366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add2a860 0x55c3add54380 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.686+0000 7f569f75cb80 1 --2- 172.21.15.70:0/2845707366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2b210 0x55c3add548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.686+0000 7f569f75cb80 1 -- 172.21.15.70:0/2845707366 >> 172.21.15.70:0/2845707366 conn(0x55c3adc58080 msgr2=0x55c3add29f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:10.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.686+0000 7f569f75cb80 1 -- 172.21.15.70:0/2845707366 shutdown_connections 2024-06-04T05:12:10.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.687+0000 7f569f75cb80 1 -- 172.21.15.70:0/2845707366 wait complete. 2024-06-04T05:12:10.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.687+0000 7f569f75cb80 1 -- start start 2024-06-04T05:12:10.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.687+0000 7f569f75cb80 1 Processor -- start 2024-06-04T05:12:10.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.687+0000 7f569f75cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2b210 0x55c3add45230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.687+0000 7f569f75cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2ef60 0x55c3add42580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f569f75cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add4c6a0 0x55c3add42ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f569f75cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f56901ad1a0 con 0x55c3add4c6a0 2024-06-04T05:12:10.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f569f75cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f56901aca70 con 0x55c3add2ef60 2024-06-04T05:12:10.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f569f75cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f56901a9cb0 con 0x55c3add2b210 2024-06-04T05:12:10.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f569f75cb80 10 client.0 _ll_register_callbacks cb 0x55c3adc14920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:10.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f568ed9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add4c6a0 0x55c3add42ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f568dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2ef60 0x55c3add42580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f568ed9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add4c6a0 0x55c3add42ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46234/0 (socket says 172.21.15.70:46234) 2024-06-04T05:12:10.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f568dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2ef60 0x55c3add42580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:38446/0 (socket says 172.21.15.70:38446) 2024-06-04T05:12:10.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f568ed9d700 1 -- 172.21.15.70:0/2393084409 learned_addr learned my addr 172.21.15.70:0/2393084409 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:10.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f568e59c700 1 --2- 172.21.15.70:0/2393084409 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2b210 0x55c3add45230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f568ed9d700 1 -- 172.21.15.70:0/2393084409 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2b210 msgr2=0x55c3add45230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:10.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f568ed9d700 1 --2- 172.21.15.70:0/2393084409 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2b210 0x55c3add45230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f568ed9d700 1 -- 172.21.15.70:0/2393084409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2ef60 msgr2=0x55c3add42580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:10.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f568ed9d700 1 --2- 172.21.15.70:0/2393084409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2ef60 0x55c3add42580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f568ed9d700 1 -- 172.21.15.70:0/2393084409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3add431c0 con 0x55c3add4c6a0 2024-06-04T05:12:10.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f568ed9d700 1 --2- 172.21.15.70:0/2393084409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add4c6a0 0x55c3add42ac0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f568800b9f0 tx=0x7f568800ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e20f87314ab4c72 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:10.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.688+0000 7f5675ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:12:10.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.689+0000 7f5675ffb700 1 -- 172.21.15.70:0/2393084409 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568802b060 con 0x55c3add4c6a0 2024-06-04T05:12:10.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.689+0000 7f5675ffb700 1 -- 172.21.15.70:0/2393084409 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f5688007420 con 0x55c3add4c6a0 2024-06-04T05:12:10.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.689+0000 7f569f75cb80 1 -- 172.21.15.70:0/2393084409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f568001dc60 con 0x55c3add4c6a0 2024-06-04T05:12:10.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.689+0000 7f5675ffb700 1 -- 172.21.15.70:0/2393084409 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f5688007660 con 0x55c3add4c6a0 2024-06-04T05:12:10.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.689+0000 7f569f75cb80 10 client.19416 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:10.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.689+0000 7f569f75cb80 1 -- 172.21.15.70:0/2393084409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55c3addb7190 con 0x55c3add4c6a0 2024-06-04T05:12:10.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.689+0000 7f5675ffb700 1 -- 172.21.15.70:0/2393084409 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f568001dc60 con 0x55c3add4c6a0 2024-06-04T05:12:10.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.690+0000 7f5675ffb700 1 -- 172.21.15.70:0/2393084409 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7f5688025070 con 0x55c3add4c6a0 2024-06-04T05:12:10.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.690+0000 7f569f75cb80 10 client.19416 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:10.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.690+0000 7f569f75cb80 10 client.19416 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:12:10.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.690+0000 7f569f75cb80 20 client.19416 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:10.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.690+0000 7f569f75cb80 1 --2- 172.21.15.70:0/2393084409 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55c3addb7960 0x55c3addb9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:10.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.690+0000 7f569f75cb80 4 client.19416 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:12:10.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.690+0000 7f569f75cb80 1 -- 172.21.15.70:0/2393084409 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c3adc8bbe0 con 0x55c3addb7960 2024-06-04T05:12:10.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.690+0000 7f568e59c700 1 --2- 172.21.15.70:0/2393084409 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55c3addb7960 0x55c3addb9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:10.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.691+0000 7f5675ffb700 10 client.19416 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:12:10.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.691+0000 7f568e59c700 1 --2- 172.21.15.70:0/2393084409 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55c3addb7960 0x55c3addb9d80 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9513b64d2dfebf5d server_cookie=c44c2731b61502a6 in_seq=0 out_seq=0 2024-06-04T05:12:10.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.691+0000 7f5675ffb700 1 -- 172.21.15.70:0/2393084409 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c3adc8bbe0 con 0x55c3addb7960 2024-06-04T05:12:10.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.691+0000 7f5675ffb700 10 client.19416 handle_command_reply: tid=0 2024-06-04T05:12:10.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.691+0000 7f569f75cb80 1 client.19416 shutdown 2024-06-04T05:12:10.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.691+0000 7f569f75cb80 1 -- 172.21.15.70:0/2393084409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add4c6a0 msgr2=0x55c3add42ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:10.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.691+0000 7f569f75cb80 1 --2- 172.21.15.70:0/2393084409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add4c6a0 0x55c3add42ac0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f568800b9f0 tx=0x7f568800ba20 comp rx=0 tx=0).stop 2024-06-04T05:12:10.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.691+0000 7f569f75cb80 1 -- 172.21.15.70:0/2393084409 shutdown_connections 2024-06-04T05:12:10.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.691+0000 7f569f75cb80 1 --2- 172.21.15.70:0/2393084409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3add4c6a0 0x55c3add42ac0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.691+0000 7f569f75cb80 1 --2- 172.21.15.70:0/2393084409 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c3add2b210 0x55c3add45230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.691+0000 7f569f75cb80 1 --2- 172.21.15.70:0/2393084409 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55c3addb7960 0x55c3addb9d80 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.691+0000 7f5675ffb700 0 client.19416 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:12:10.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.692+0000 7f569f75cb80 1 --2- 172.21.15.70:0/2393084409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c3add2ef60 0x55c3add42580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.692+0000 7f569f75cb80 1 -- 172.21.15.70:0/2393084409 >> 172.21.15.70:0/2393084409 conn(0x55c3adc58080 msgr2=0x55c3add2dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:10.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.692+0000 7f569f75cb80 1 -- 172.21.15.70:0/2393084409 shutdown_connections 2024-06-04T05:12:10.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.692+0000 7f569f75cb80 1 -- 172.21.15.70:0/2393084409 wait complete. 2024-06-04T05:12:10.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.692+0000 7f569f75cb80 20 client.19416 trim_cache size 0 max 16384 2024-06-04T05:12:10.697 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:12:10.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.692+0000 7f5694ec7700 1 -- 172.21.15.70:0/404936197 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56780b2a70 msgr2=0x7f56780b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:10.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.692+0000 7f5694ec7700 1 --2- 172.21.15.70:0/404936197 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56780b2a70 0x7f56780b4f30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5680009750 tx=0x7f5680009310 comp rx=0 tx=0).stop 2024-06-04T05:12:10.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.692+0000 7f5694ec7700 1 -- 172.21.15.70:0/404936197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5690103600 msgr2=0x7f56901a1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:10.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.692+0000 7f5694ec7700 1 --2- 172.21.15.70:0/404936197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5690103600 0x7f56901a1e80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f568400e9f0 tx=0x7f568400ed00 comp rx=0 tx=0).stop 2024-06-04T05:12:10.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.693+0000 7f5694ec7700 1 -- 172.21.15.70:0/404936197 shutdown_connections 2024-06-04T05:12:10.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.693+0000 7f5694ec7700 1 --2- 172.21.15.70:0/404936197 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56780b2a70 0x7f56780b4f30 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.693+0000 7f5694ec7700 1 --2- 172.21.15.70:0/404936197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56901084c0 0x7f569019bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.693+0000 7f5694ec7700 1 --2- 172.21.15.70:0/404936197 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5690102c40 0x7f56901a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.693+0000 7f5694ec7700 1 --2- 172.21.15.70:0/404936197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5690103600 0x7f56901a1e80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:10.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.693+0000 7f5694ec7700 1 -- 172.21.15.70:0/404936197 >> 172.21.15.70:0/404936197 conn(0x7f5690078ab0 msgr2=0x7f5690100210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:10.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.693+0000 7f5694ec7700 1 -- 172.21.15.70:0/404936197 shutdown_connections 2024-06-04T05:12:10.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:10.693+0000 7f5694ec7700 1 -- 172.21.15.70:0/404936197 wait complete. 2024-06-04T05:12:10.801 INFO:tasks.workunit.client.0.smithi070.stdout: 10 11248 12.67 MB/sec warmup 62 sec latency 71.703 ms 2024-06-04T05:12:11.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:11 smithi070 ceph-mon[32631]: pgmap v469: 97 pgs: 97 active+clean; 399 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 3.6 MiB/s wr, 158 op/s 2024-06-04T05:12:11.486 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:12:11.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:11 smithi098 ceph-mon[35569]: pgmap v469: 97 pgs: 97 active+clean; 399 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 3.6 MiB/s wr, 158 op/s 2024-06-04T05:12:11.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:11 smithi112 ceph-mon[36453]: pgmap v469: 97 pgs: 97 active+clean; 399 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 3.6 MiB/s wr, 158 op/s 2024-06-04T05:12:11.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 11425 12.59 MB/sec warmup 63 sec latency 68.538 ms 2024-06-04T05:12:11.887 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:12:12.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 11594 12.48 MB/sec warmup 64 sec latency 68.107 ms 2024-06-04T05:12:13.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:13 smithi070 ceph-mon[32631]: pgmap v470: 97 pgs: 97 active+clean; 407 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.3 MiB/s wr, 341 op/s 2024-06-04T05:12:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:13 smithi098 ceph-mon[35569]: pgmap v470: 97 pgs: 97 active+clean; 407 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.3 MiB/s wr, 341 op/s 2024-06-04T05:12:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:13 smithi112 ceph-mon[36453]: pgmap v470: 97 pgs: 97 active+clean; 407 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.3 MiB/s wr, 341 op/s 2024-06-04T05:12:13.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 11762 12.38 MB/sec warmup 65 sec latency 97.868 ms 2024-06-04T05:12:13.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.964+0000 7fe496396700 1 -- 172.21.15.70:0/3880206066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe490108460 msgr2=0x7fe49006bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:13.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.964+0000 7fe496396700 1 --2- 172.21.15.70:0/3880206066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe490108460 0x7fe49006bae0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe488009f90 tx=0x7fe48800b330 comp rx=0 tx=0).stop 2024-06-04T05:12:13.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.964+0000 7fe496396700 1 -- 172.21.15.70:0/3880206066 shutdown_connections 2024-06-04T05:12:13.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.964+0000 7fe496396700 1 --2- 172.21.15.70:0/3880206066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe490105f70 0x7fe490106400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:13.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.964+0000 7fe496396700 1 --2- 172.21.15.70:0/3880206066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe490108460 0x7fe49006bae0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:13.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.964+0000 7fe496396700 1 --2- 172.21.15.70:0/3880206066 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe49006c020 0x7fe49006c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:13.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.964+0000 7fe496396700 1 -- 172.21.15.70:0/3880206066 >> 172.21.15.70:0/3880206066 conn(0x7fe4900793b0 msgr2=0x7fe4900797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:13.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.964+0000 7fe496396700 1 -- 172.21.15.70:0/3880206066 shutdown_connections 2024-06-04T05:12:13.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.964+0000 7fe496396700 1 -- 172.21.15.70:0/3880206066 wait complete. 2024-06-04T05:12:13.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.965+0000 7fe496396700 1 Processor -- start 2024-06-04T05:12:13.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.965+0000 7fe496396700 1 -- start start 2024-06-04T05:12:13.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.966+0000 7fe496396700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe49006c020 0x7fe490074c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:13.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.966+0000 7fe496396700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe490105f70 0x7fe490075190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:13.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.966+0000 7fe496396700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe490108460 0x7fe49006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:13.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.966+0000 7fe496396700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe490119bf0 con 0x7fe490108460 2024-06-04T05:12:13.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.966+0000 7fe496396700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe490119a70 con 0x7fe490105f70 2024-06-04T05:12:13.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.966+0000 7fe496396700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe490119d70 con 0x7fe49006c020 2024-06-04T05:12:13.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.966+0000 7fe494933700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe490108460 0x7fe49006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:13.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.966+0000 7fe494933700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe490108460 0x7fe49006de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46256/0 (socket says 172.21.15.70:46256) 2024-06-04T05:12:13.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.966+0000 7fe494933700 1 -- 172.21.15.70:0/1418694440 learned_addr learned my addr 172.21.15.70:0/1418694440 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:13.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.966+0000 7fe48f7fe700 1 --2- 172.21.15.70:0/1418694440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe490105f70 0x7fe490075190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:13.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.966+0000 7fe48ffff700 1 --2- 172.21.15.70:0/1418694440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe49006c020 0x7fe490074c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:13.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.967+0000 7fe48ffff700 1 -- 172.21.15.70:0/1418694440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe490105f70 msgr2=0x7fe490075190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:13.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.967+0000 7fe48ffff700 1 --2- 172.21.15.70:0/1418694440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe490105f70 0x7fe490075190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:13.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.967+0000 7fe48ffff700 1 -- 172.21.15.70:0/1418694440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe490108460 msgr2=0x7fe49006de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:13.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.967+0000 7fe48ffff700 1 --2- 172.21.15.70:0/1418694440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe490108460 0x7fe49006de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:13.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.967+0000 7fe48ffff700 1 -- 172.21.15.70:0/1418694440 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe49006e750 con 0x7fe49006c020 2024-06-04T05:12:13.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.967+0000 7fe494933700 1 --2- 172.21.15.70:0/1418694440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe490108460 0x7fe49006de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:13.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.967+0000 7fe48f7fe700 1 --2- 172.21.15.70:0/1418694440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe490105f70 0x7fe490075190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:13.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.967+0000 7fe48ffff700 1 --2- 172.21.15.70:0/1418694440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe49006c020 0x7fe490074c50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe48800b6c0 tx=0x7fe48800bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:13.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.968+0000 7fe48d7fa700 1 -- 172.21.15.70:0/1418694440 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe488031030 con 0x7fe49006c020 2024-06-04T05:12:13.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.968+0000 7fe48d7fa700 1 -- 172.21.15.70:0/1418694440 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe48802d450 con 0x7fe49006c020 2024-06-04T05:12:13.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.968+0000 7fe48d7fa700 1 -- 172.21.15.70:0/1418694440 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4880052a0 con 0x7fe49006c020 2024-06-04T05:12:13.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.968+0000 7fe496396700 1 -- 172.21.15.70:0/1418694440 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe49006e9e0 con 0x7fe49006c020 2024-06-04T05:12:13.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.968+0000 7fe496396700 1 -- 172.21.15.70:0/1418694440 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe490072950 con 0x7fe49006c020 2024-06-04T05:12:13.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.972+0000 7fe48d7fa700 1 -- 172.21.15.70:0/1418694440 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fe488034030 con 0x7fe49006c020 2024-06-04T05:12:13.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.973+0000 7fe4a0c2bb80 1 Processor -- start 2024-06-04T05:12:13.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.973+0000 7fe48d7fa700 1 --2- 172.21.15.70:0/1418694440 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe4780b2ab0 0x7fe4780b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:13.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.973+0000 7fe4a0c2bb80 1 -- start start 2024-06-04T05:12:13.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.973+0000 7fe494933700 1 --2- 172.21.15.70:0/1418694440 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe4780b2ab0 0x7fe4780b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:13.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.973+0000 7fe4a0c2bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f5133320 0x5643f5133700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:13.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.973+0000 7fe4a0c2bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f512d8a0 0x5643f512dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:13.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.973+0000 7fe4a0c2bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512e260 0x5643f513a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:13.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.973+0000 7fe4a0c2bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5643f508bd70 con 0x5643f512e260 2024-06-04T05:12:13.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.973+0000 7fe4a0c2bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5643f507b910 con 0x5643f5133320 2024-06-04T05:12:13.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.973+0000 7fe4a0c2bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5643f507f950 con 0x5643f512d8a0 2024-06-04T05:12:13.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.973+0000 7fe494933700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512e260 0x5643f513a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:13.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.973+0000 7fe494933700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512e260 0x5643f513a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46262/0 (socket says 172.21.15.70:46262) 2024-06-04T05:12:13.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.973+0000 7fe48f7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f512d8a0 0x5643f512dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:13.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.973+0000 7fe494933700 1 -- 172.21.15.70:0/2974447040 learned_addr learned my addr 172.21.15.70:0/2974447040 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:13.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.974+0000 7fe48f7fe700 1 -- 172.21.15.70:0/2974447040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f5133320 msgr2=0x5643f5133700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:13.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.974+0000 7fe48ffff700 1 --2- 172.21.15.70:0/2974447040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f5133320 0x5643f5133700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:13.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.974+0000 7fe48d7fa700 1 -- 172.21.15.70:0/1418694440 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fe48800f020 con 0x7fe49006c020 2024-06-04T05:12:13.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.974+0000 7fe48f7fe700 1 --2- 172.21.15.70:0/2974447040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f5133320 0x5643f5133700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:13.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.974+0000 7fe48f7fe700 1 -- 172.21.15.70:0/2974447040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512e260 msgr2=0x5643f513a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:13.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.974+0000 7fe48f7fe700 1 --2- 172.21.15.70:0/2974447040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512e260 0x5643f513a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:13.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.974+0000 7fe48f7fe700 1 -- 172.21.15.70:0/2974447040 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643f5024fb0 con 0x5643f512d8a0 2024-06-04T05:12:13.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.974+0000 7fe48f7fe700 1 --2- 172.21.15.70:0/2974447040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f512d8a0 0x5643f512dd20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe49006f530 tx=0x7fe4840117a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6123a17ccff10f5f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:13.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.974+0000 7fe494933700 1 --2- 172.21.15.70:0/1418694440 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe4780b2ab0 0x7fe4780b4f70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe480012e30 tx=0x7fe48000bc70 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:13.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.974+0000 7fe48ffff700 1 --2- 172.21.15.70:0/2974447040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f5133320 0x5643f5133700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:12:13.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.975+0000 7fe4767fc700 1 -- 172.21.15.70:0/2974447040 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4840112f0 con 0x5643f512d8a0 2024-06-04T05:12:13.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.975+0000 7fe4767fc700 1 -- 172.21.15.70:0/2974447040 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe4840044c0 con 0x5643f512d8a0 2024-06-04T05:12:13.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.975+0000 7fe4767fc700 1 -- 172.21.15.70:0/2974447040 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe484010690 con 0x5643f512d8a0 2024-06-04T05:12:13.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.975+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/2974447040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f512d8a0 msgr2=0x5643f512dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:13.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.976+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/2974447040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f512d8a0 0x5643f512dd20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe49006f530 tx=0x7fe4840117a0 comp rx=0 tx=0).stop 2024-06-04T05:12:13.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.976+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/2974447040 shutdown_connections 2024-06-04T05:12:13.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.976+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/2974447040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512e260 0x5643f513a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:13.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.976+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/2974447040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f5133320 0x5643f5133700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:13.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.976+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/2974447040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f512d8a0 0x5643f512dd20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:13.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.976+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/2974447040 >> 172.21.15.70:0/2974447040 conn(0x5643f5059240 msgr2=0x5643f51283e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:13.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.976+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/2974447040 shutdown_connections 2024-06-04T05:12:13.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.976+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/2974447040 wait complete. 2024-06-04T05:12:13.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.976+0000 7fe4a0c2bb80 1 -- start start 2024-06-04T05:12:13.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.976+0000 7fe4a0c2bb80 1 Processor -- start 2024-06-04T05:12:13.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe4a0c2bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512d8a0 0x5643f5139d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:13.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe4a0c2bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f512e260 0x5643f5137040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:13.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe4a0c2bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f5133320 0x5643f5137580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:13.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe4a0c2bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5643f4f81fa0 con 0x5643f512d8a0 2024-06-04T05:12:13.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe4a0c2bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5643f4fe3da0 con 0x5643f512e260 2024-06-04T05:12:13.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe4a0c2bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5643f513a6e0 con 0x5643f5133320 2024-06-04T05:12:13.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe4a0c2bb80 10 client.0 _ll_register_callbacks cb 0x5643f5016950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:13.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe48f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f512e260 0x5643f5137040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:13.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe48ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512d8a0 0x5643f5139d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:13.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe48f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f512e260 0x5643f5137040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:38490/0 (socket says 172.21.15.70:38490) 2024-06-04T05:12:13.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe48f7fe700 1 -- 172.21.15.70:0/977555322 learned_addr learned my addr 172.21.15.70:0/977555322 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:13.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe494933700 1 --2- 172.21.15.70:0/977555322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f5133320 0x5643f5137580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:13.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe48ffff700 1 -- 172.21.15.70:0/977555322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f5133320 msgr2=0x5643f5137580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:13.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe48ffff700 1 --2- 172.21.15.70:0/977555322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f5133320 0x5643f5137580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:13.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe48ffff700 1 -- 172.21.15.70:0/977555322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f512e260 msgr2=0x5643f5137040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:13.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe48ffff700 1 --2- 172.21.15.70:0/977555322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f512e260 0x5643f5137040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:13.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe48ffff700 1 -- 172.21.15.70:0/977555322 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643f4fcb8d0 con 0x5643f512d8a0 2024-06-04T05:12:13.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.977+0000 7fe48f7fe700 1 --2- 172.21.15.70:0/977555322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f512e260 0x5643f5137040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:13.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.978+0000 7fe48ffff700 1 --2- 172.21.15.70:0/977555322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512d8a0 0x5643f5139d40 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fe4880f1d70 tx=0x7fe48809b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee1677a5d1bc4e57 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:13.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.978+0000 7fe46f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:12:13.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.978+0000 7fe46f7fe700 1 -- 172.21.15.70:0/977555322 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4880bb4a0 con 0x5643f512d8a0 2024-06-04T05:12:13.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.978+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/977555322 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4840044c0 con 0x5643f512d8a0 2024-06-04T05:12:13.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.978+0000 7fe46f7fe700 1 -- 172.21.15.70:0/977555322 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fe48809b490 con 0x5643f512d8a0 2024-06-04T05:12:13.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.978+0000 7fe46f7fe700 1 -- 172.21.15.70:0/977555322 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fe4880092e0 con 0x5643f512d8a0 2024-06-04T05:12:13.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.978+0000 7fe4a0c2bb80 10 client.19428 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:13.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.978+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/977555322 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5643f5035760 con 0x5643f512d8a0 2024-06-04T05:12:13.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.978+0000 7fe46f7fe700 1 -- 172.21.15.70:0/977555322 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe4840044c0 con 0x5643f512d8a0 2024-06-04T05:12:13.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.979+0000 7fe46f7fe700 1 -- 172.21.15.70:0/977555322 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7fe488011070 con 0x5643f512d8a0 2024-06-04T05:12:13.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.979+0000 7fe4a0c2bb80 10 client.19428 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:13.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.979+0000 7fe4a0c2bb80 10 client.19428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:12:13.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.979+0000 7fe4a0c2bb80 20 client.19428 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:13.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.979+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/977555322 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5643f51471b0 0x5643f5149610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:13.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.979+0000 7fe4a0c2bb80 4 client.19428 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:12:13.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.979+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/977555322 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643f508dcc0 con 0x5643f51471b0 2024-06-04T05:12:13.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.979+0000 7fe48f7fe700 1 --2- 172.21.15.70:0/977555322 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5643f51471b0 0x5643f5149610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.979+0000 7fe48f7fe700 1 --2- 172.21.15.70:0/977555322 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5643f51471b0 0x5643f5149610 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd9b503a48f5f017 server_cookie=4c057819c60b1872 in_seq=0 out_seq=0 2024-06-04T05:12:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.979+0000 7fe46f7fe700 10 client.19428 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.980+0000 7fe46f7fe700 1 -- 172.21.15.70:0/977555322 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5643f508dcc0 con 0x5643f51471b0 2024-06-04T05:12:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.980+0000 7fe46f7fe700 10 client.19428 handle_command_reply: tid=0 2024-06-04T05:12:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.980+0000 7fe4a0c2bb80 1 client.19428 shutdown 2024-06-04T05:12:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.981+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/977555322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512d8a0 msgr2=0x5643f5139d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.981+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/977555322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512d8a0 0x5643f5139d40 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fe4880f1d70 tx=0x7fe48809b740 comp rx=0 tx=0).stop 2024-06-04T05:12:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.981+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/977555322 shutdown_connections 2024-06-04T05:12:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.981+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/977555322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f5133320 0x5643f5137580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:13.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.981+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/977555322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512d8a0 0x5643f5139d40 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:13.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.981+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/977555322 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5643f51471b0 0x5643f5149610 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:13.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.981+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/977555322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f512e260 0x5643f5137040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:13.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.981+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/977555322 >> 172.21.15.70:0/977555322 conn(0x5643f5059240 msgr2=0x5643f5127af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:13.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.981+0000 7fe46f7fe700 0 client.19428 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:13.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.981+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/977555322 shutdown_connections 2024-06-04T05:12:13.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.981+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/977555322 wait complete. 2024-06-04T05:12:13.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:13.981+0000 7fe4a0c2bb80 20 client.19428 trim_cache size 0 max 16384 2024-06-04T05:12:14.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.003+0000 7fe4a0c2bb80 1 Processor -- start 2024-06-04T05:12:14.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.003+0000 7fe4a0c2bb80 1 -- start start 2024-06-04T05:12:14.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.003+0000 7fe4a0c2bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f512d8a0 0x5643f5135fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:14.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.003+0000 7fe4a0c2bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f512e260 0x5643f5128270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:14.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.003+0000 7fe4a0c2bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f5133320 0x5643f512cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:14.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.003+0000 7fe4a0c2bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe4901105e0 con 0x5643f5133320 2024-06-04T05:12:14.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.003+0000 7fe4a0c2bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe4901b6100 con 0x5643f512e260 2024-06-04T05:12:14.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.003+0000 7fe4a0c2bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe490110cb0 con 0x5643f512d8a0 2024-06-04T05:12:14.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.003+0000 7fe494933700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f5133320 0x5643f512cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:14.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.003+0000 7fe494933700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f5133320 0x5643f512cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46292/0 (socket says 172.21.15.70:46292) 2024-06-04T05:12:14.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.003+0000 7fe494933700 1 -- 172.21.15.70:0/3145644663 learned_addr learned my addr 172.21.15.70:0/3145644663 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:14.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.003+0000 7fe48f7fe700 1 --2- 172.21.15.70:0/3145644663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f512e260 0x5643f5128270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:14.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.004+0000 7fe494933700 1 -- 172.21.15.70:0/3145644663 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f512d8a0 msgr2=0x5643f5135fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:12:14.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.004+0000 7fe494933700 1 --2- 172.21.15.70:0/3145644663 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f512d8a0 0x5643f5135fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:14.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.004+0000 7fe494933700 1 -- 172.21.15.70:0/3145644663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f512e260 msgr2=0x5643f5128270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:14.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.004+0000 7fe494933700 1 --2- 172.21.15.70:0/3145644663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f512e260 0x5643f5128270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:14.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.004+0000 7fe494933700 1 -- 172.21.15.70:0/3145644663 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643f5128eb0 con 0x5643f5133320 2024-06-04T05:12:14.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.004+0000 7fe494933700 1 --2- 172.21.15.70:0/3145644663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f5133320 0x5643f512cfa0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fe4800127c0 tx=0x7fe48001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b731ea7d80132b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:14.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.004+0000 7fe475ffb700 1 -- 172.21.15.70:0/3145644663 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48002e070 con 0x5643f5133320 2024-06-04T05:12:14.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.004+0000 7fe475ffb700 1 -- 172.21.15.70:0/3145644663 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe4800208c0 con 0x5643f5133320 2024-06-04T05:12:14.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.004+0000 7fe475ffb700 1 -- 172.21.15.70:0/3145644663 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe480010430 con 0x5643f5133320 2024-06-04T05:12:14.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.005+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/3145644663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f5133320 msgr2=0x5643f512cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:14.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.005+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/3145644663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f5133320 0x5643f512cfa0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fe4800127c0 tx=0x7fe48001f000 comp rx=0 tx=0).stop 2024-06-04T05:12:14.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.005+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/3145644663 shutdown_connections 2024-06-04T05:12:14.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.005+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/3145644663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f5133320 0x5643f512cfa0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:14.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.005+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/3145644663 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f512d8a0 0x5643f5135fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:14.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.005+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/3145644663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f512e260 0x5643f5128270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:14.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.005+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/3145644663 >> 172.21.15.70:0/3145644663 conn(0x5643f5059240 msgr2=0x5643f512c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:14.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.005+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/3145644663 shutdown_connections 2024-06-04T05:12:14.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.006+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/3145644663 wait complete. 2024-06-04T05:12:14.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.006+0000 7fe4a0c2bb80 1 -- start start 2024-06-04T05:12:14.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.006+0000 7fe4a0c2bb80 1 Processor -- start 2024-06-04T05:12:14.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe4a0c2bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f512d8a0 0x5643f514e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe4a0c2bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512e260 0x5643f514e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe4a0c2bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f5133320 0x5643f513e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe4a0c2bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe490111b10 con 0x5643f512e260 2024-06-04T05:12:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe4a0c2bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe4901113e0 con 0x5643f5133320 2024-06-04T05:12:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe4a0c2bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe49010e5a0 con 0x5643f512d8a0 2024-06-04T05:12:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe48f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512e260 0x5643f514e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe48f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512e260 0x5643f514e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46302/0 (socket says 172.21.15.70:46302) 2024-06-04T05:12:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe48f7fe700 1 -- 172.21.15.70:0/3930723468 learned_addr learned my addr 172.21.15.70:0/3930723468 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe4a0c2bb80 10 client.0 _ll_register_callbacks cb 0x5643f5016950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe494933700 1 --2- 172.21.15.70:0/3930723468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f5133320 0x5643f513e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe48f7fe700 1 -- 172.21.15.70:0/3930723468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f512d8a0 msgr2=0x5643f514e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:12:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe48f7fe700 1 --2- 172.21.15.70:0/3930723468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f512d8a0 0x5643f514e2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe48f7fe700 1 -- 172.21.15.70:0/3930723468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f5133320 msgr2=0x5643f513e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe48f7fe700 1 --2- 172.21.15.70:0/3930723468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f5133320 0x5643f513e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:14.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe48f7fe700 1 -- 172.21.15.70:0/3930723468 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643f513eef0 con 0x5643f512e260 2024-06-04T05:12:14.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe494933700 1 --2- 172.21.15.70:0/3930723468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f5133320 0x5643f513e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:12:14.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe48f7fe700 1 --2- 172.21.15.70:0/3930723468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512e260 0x5643f514e830 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fe484009f80 tx=0x7fe4840050f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e818e9e7d70b309 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:14.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.007+0000 7fe46f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:12:14.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.008+0000 7fe46f7fe700 1 -- 172.21.15.70:0/3930723468 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe484005520 con 0x5643f512e260 2024-06-04T05:12:14.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.008+0000 7fe46f7fe700 1 -- 172.21.15.70:0/3930723468 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fe484022040 con 0x5643f512e260 2024-06-04T05:12:14.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.008+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/3930723468 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4800208c0 con 0x5643f512e260 2024-06-04T05:12:14.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.008+0000 7fe46f7fe700 1 -- 172.21.15.70:0/3930723468 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fe484037050 con 0x5643f512e260 2024-06-04T05:12:14.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.008+0000 7fe4a0c2bb80 10 client.19434 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:14.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.008+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/3930723468 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5643f5140160 con 0x5643f512e260 2024-06-04T05:12:14.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.008+0000 7fe46f7fe700 1 -- 172.21.15.70:0/3930723468 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe4800208c0 con 0x5643f512e260 2024-06-04T05:12:14.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.008+0000 7fe46f7fe700 1 -- 172.21.15.70:0/3930723468 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7fe48401a600 con 0x5643f512e260 2024-06-04T05:12:14.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.009+0000 7fe4a0c2bb80 10 client.19434 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:14.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.009+0000 7fe4a0c2bb80 10 client.19434 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:12:14.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.009+0000 7fe4a0c2bb80 20 client.19434 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:14.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.009+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/3930723468 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5643f51b1df0 0x5643f51b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:14.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.009+0000 7fe4a0c2bb80 4 client.19434 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:12:14.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.009+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/3930723468 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5643f50874a0 con 0x5643f51b1df0 2024-06-04T05:12:14.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.009+0000 7fe48ffff700 1 --2- 172.21.15.70:0/3930723468 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5643f51b1df0 0x5643f51b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:14.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.009+0000 7fe46f7fe700 10 client.19434 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:14.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.009+0000 7fe48ffff700 1 --2- 172.21.15.70:0/3930723468 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5643f51b1df0 0x5643f51b41e0 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8f1a9bc9fb99288 server_cookie=376695598214b47f in_seq=0 out_seq=0 2024-06-04T05:12:14.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.009+0000 7fe46f7fe700 1 -- 172.21.15.70:0/3930723468 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5643f50874a0 con 0x5643f51b1df0 2024-06-04T05:12:14.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.009+0000 7fe46f7fe700 10 client.19434 handle_command_reply: tid=0 2024-06-04T05:12:14.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.010+0000 7fe4a0c2bb80 1 client.19434 shutdown 2024-06-04T05:12:14.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.010+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/3930723468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512e260 msgr2=0x5643f514e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:14.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.010+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/3930723468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512e260 0x5643f514e830 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fe484009f80 tx=0x7fe4840050f0 comp rx=0 tx=0).stop 2024-06-04T05:12:14.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.010+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/3930723468 shutdown_connections 2024-06-04T05:12:14.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.010+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/3930723468 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5643f51b1df0 0x5643f51b41e0 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:14.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.010+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/3930723468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643f5133320 0x5643f513e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:14.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.010+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/3930723468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643f512d8a0 0x5643f514e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:14.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.010+0000 7fe4a0c2bb80 1 --2- 172.21.15.70:0/3930723468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f512e260 0x5643f514e830 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:14.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.010+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/3930723468 >> 172.21.15.70:0/3930723468 conn(0x5643f5059240 msgr2=0x5643f5139100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:14.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.010+0000 7fe46f7fe700 0 client.19434 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:14.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.010+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/3930723468 shutdown_connections 2024-06-04T05:12:14.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.010+0000 7fe4a0c2bb80 1 -- 172.21.15.70:0/3930723468 wait complete. 2024-06-04T05:12:14.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.010+0000 7fe4a0c2bb80 20 client.19434 trim_cache size 0 max 16384 2024-06-04T05:12:14.016 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:12:14.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.011+0000 7fe496396700 1 -- 172.21.15.70:0/1418694440 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe4780b2ab0 msgr2=0x7fe4780b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:14.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.011+0000 7fe496396700 1 --2- 172.21.15.70:0/1418694440 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe4780b2ab0 0x7fe4780b4f70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe480012e30 tx=0x7fe48000bc70 comp rx=0 tx=0).stop 2024-06-04T05:12:14.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.011+0000 7fe496396700 1 -- 172.21.15.70:0/1418694440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe49006c020 msgr2=0x7fe490074c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:14.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.011+0000 7fe496396700 1 --2- 172.21.15.70:0/1418694440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe49006c020 0x7fe490074c50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe48800b6c0 tx=0x7fe48800bf90 comp rx=0 tx=0).stop 2024-06-04T05:12:14.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.011+0000 7fe496396700 1 -- 172.21.15.70:0/1418694440 shutdown_connections 2024-06-04T05:12:14.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.011+0000 7fe496396700 1 --2- 172.21.15.70:0/1418694440 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe4780b2ab0 0x7fe4780b4f70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:14.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.011+0000 7fe496396700 1 --2- 172.21.15.70:0/1418694440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe490108460 0x7fe49006de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:14.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.011+0000 7fe496396700 1 --2- 172.21.15.70:0/1418694440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe49006c020 0x7fe490074c50 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:14.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.011+0000 7fe496396700 1 --2- 172.21.15.70:0/1418694440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe490105f70 0x7fe490075190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:14.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.011+0000 7fe496396700 1 -- 172.21.15.70:0/1418694440 >> 172.21.15.70:0/1418694440 conn(0x7fe4900793b0 msgr2=0x7fe490101420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:14.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.011+0000 7fe496396700 1 -- 172.21.15.70:0/1418694440 shutdown_connections 2024-06-04T05:12:14.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:14.011+0000 7fe496396700 1 -- 172.21.15.70:0/1418694440 wait complete. 2024-06-04T05:12:14.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 11944 12.33 MB/sec warmup 66 sec latency 67.418 ms 2024-06-04T05:12:15.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:15 smithi070 ceph-mon[32631]: pgmap v471: 97 pgs: 97 active+clean; 407 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 302 op/s 2024-06-04T05:12:15.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:15 smithi098 ceph-mon[35569]: pgmap v471: 97 pgs: 97 active+clean; 407 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 302 op/s 2024-06-04T05:12:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:15 smithi112 ceph-mon[36453]: pgmap v471: 97 pgs: 97 active+clean; 407 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 302 op/s 2024-06-04T05:12:15.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 12095 12.24 MB/sec warmup 67 sec latency 116.173 ms 2024-06-04T05:12:16.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 12244 12.16 MB/sec warmup 68 sec latency 119.056 ms 2024-06-04T05:12:17.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:17 smithi098 ceph-mon[35569]: pgmap v472: 97 pgs: 97 active+clean; 407 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 345 op/s 2024-06-04T05:12:17.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:17 smithi070 ceph-mon[32631]: pgmap v472: 97 pgs: 97 active+clean; 407 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 345 op/s 2024-06-04T05:12:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:17 smithi112 ceph-mon[36453]: pgmap v472: 97 pgs: 97 active+clean; 407 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 345 op/s 2024-06-04T05:12:17.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 12429 12.11 MB/sec warmup 69 sec latency 123.877 ms 2024-06-04T05:12:18.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 12549 12.01 MB/sec warmup 70 sec latency 162.841 ms 2024-06-04T05:12:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:19 smithi098 ceph-mon[35569]: pgmap v473: 97 pgs: 97 active+clean; 407 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 336 op/s 2024-06-04T05:12:19.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:19 smithi070 ceph-mon[32631]: pgmap v473: 97 pgs: 97 active+clean; 407 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 336 op/s 2024-06-04T05:12:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:19 smithi112 ceph-mon[36453]: pgmap v473: 97 pgs: 97 active+clean; 407 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 336 op/s 2024-06-04T05:12:19.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 12730 11.99 MB/sec warmup 71 sec latency 117.048 ms 2024-06-04T05:12:20.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 12915 11.98 MB/sec warmup 72 sec latency 72.961 ms 2024-06-04T05:12:21.136 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:12:21.136 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:12:21.489 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:12:21.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:21 smithi098 ceph-mon[35569]: pgmap v474: 97 pgs: 97 active+clean; 407 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 288 op/s 2024-06-04T05:12:21.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:21 smithi070 ceph-mon[32631]: pgmap v474: 97 pgs: 97 active+clean; 407 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 288 op/s 2024-06-04T05:12:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:21 smithi112 ceph-mon[36453]: pgmap v474: 97 pgs: 97 active+clean; 407 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 288 op/s 2024-06-04T05:12:21.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 13114 12.01 MB/sec warmup 73 sec latency 55.180 ms 2024-06-04T05:12:22.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.228+0000 7f0e725c1640 1 -- 172.21.15.70:0/3322158511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e6c102a30 msgr2=0x7f0e6c102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:22.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.228+0000 7f0e725c1640 1 --2- 172.21.15.70:0/3322158511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e6c102a30 0x7f0e6c102eb0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f0e60009930 tx=0x7f0e6002a420 comp rx=0 tx=0).stop 2024-06-04T05:12:22.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.228+0000 7f0e725c1640 1 -- 172.21.15.70:0/3322158511 shutdown_connections 2024-06-04T05:12:22.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.228+0000 7f0e725c1640 1 --2- 172.21.15.70:0/3322158511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e6c1033f0 0x7f0e6c10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.228+0000 7f0e725c1640 1 --2- 172.21.15.70:0/3322158511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e6c102a30 0x7f0e6c102eb0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.228+0000 7f0e725c1640 1 --2- 172.21.15.70:0/3322158511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e6c108a40 0x7f0e6c108e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.228+0000 7f0e725c1640 1 -- 172.21.15.70:0/3322158511 >> 172.21.15.70:0/3322158511 conn(0x7f0e6c0fe500 msgr2=0x7f0e6c100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:22.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.228+0000 7f0e725c1640 1 -- 172.21.15.70:0/3322158511 shutdown_connections 2024-06-04T05:12:22.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.228+0000 7f0e725c1640 1 -- 172.21.15.70:0/3322158511 wait complete. 2024-06-04T05:12:22.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e725c1640 1 Processor -- start 2024-06-04T05:12:22.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e725c1640 1 -- start start 2024-06-04T05:12:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e725c1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e6c102a30 0x7f0e6c199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e725c1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e6c1033f0 0x7f0e6c199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e725c1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e6c108a40 0x7f0e6c19d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e725c1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c112bb0 con 0x7f0e6c102a30 2024-06-04T05:12:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e725c1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c112a30 con 0x7f0e6c1033f0 2024-06-04T05:12:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e725c1640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c112d30 con 0x7f0e6c108a40 2024-06-04T05:12:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e6bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e6c102a30 0x7f0e6c199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e6bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e6c102a30 0x7f0e6c199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35888/0 (socket says 172.21.15.70:35888) 2024-06-04T05:12:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e6bfff640 1 -- 172.21.15.70:0/2286437769 learned_addr learned my addr 172.21.15.70:0/2286437769 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e6b7fe640 1 --2- 172.21.15.70:0/2286437769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e6c1033f0 0x7f0e6c199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e70b5e640 1 --2- 172.21.15.70:0/2286437769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e6c108a40 0x7f0e6c19d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e6bfff640 1 -- 172.21.15.70:0/2286437769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e6c108a40 msgr2=0x7f0e6c19d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e6bfff640 1 --2- 172.21.15.70:0/2286437769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e6c108a40 0x7f0e6c19d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e6bfff640 1 -- 172.21.15.70:0/2286437769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e6c1033f0 msgr2=0x7f0e6c199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e6bfff640 1 --2- 172.21.15.70:0/2286437769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e6c1033f0 0x7f0e6c199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.229+0000 7f0e6bfff640 1 -- 172.21.15.70:0/2286437769 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e6c19e120 con 0x7f0e6c102a30 2024-06-04T05:12:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.230+0000 7f0e6bfff640 1 --2- 172.21.15.70:0/2286437769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e6c102a30 0x7f0e6c199130 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00e9f0 tx=0x7f0e5c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.230+0000 7f0e697fa640 1 -- 172.21.15.70:0/2286437769 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c00cd80 con 0x7f0e6c102a30 2024-06-04T05:12:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.230+0000 7f0e697fa640 1 -- 172.21.15.70:0/2286437769 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0e5c0108c0 con 0x7f0e6c102a30 2024-06-04T05:12:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.230+0000 7f0e725c1640 1 -- 172.21.15.70:0/2286437769 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e6c19e410 con 0x7f0e6c102a30 2024-06-04T05:12:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.230+0000 7f0e697fa640 1 -- 172.21.15.70:0/2286437769 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c0187c0 con 0x7f0e6c102a30 2024-06-04T05:12:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.230+0000 7f0e725c1640 1 -- 172.21.15.70:0/2286437769 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e6c1a5ca0 con 0x7f0e6c102a30 2024-06-04T05:12:22.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.232+0000 7f0e697fa640 1 -- 172.21.15.70:0/2286437769 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f0e5c010430 con 0x7f0e6c102a30 2024-06-04T05:12:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.232+0000 7f0e753b8740 1 Processor -- start 2024-06-04T05:12:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.232+0000 7f0e753b8740 1 -- start start 2024-06-04T05:12:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.232+0000 7f0e753b8740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c6820 0x561e879c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.232+0000 7f0e753b8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879c7140 0x561e879d14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.232+0000 7f0e753b8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879d1a00 0x561e879d3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.232+0000 7f0e753b8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561e87938c30 con 0x561e879c7140 2024-06-04T05:12:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.232+0000 7f0e753b8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561e879287d0 con 0x561e879d1a00 2024-06-04T05:12:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.232+0000 7f0e753b8740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561e8792c810 con 0x561e879c6820 2024-06-04T05:12:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e6bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879c7140 0x561e879d14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e6bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879c7140 0x561e879d14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35898/0 (socket says 172.21.15.70:35898) 2024-06-04T05:12:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e70b5e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c6820 0x561e879c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e6bfff640 1 -- 172.21.15.70:0/3001287032 learned_addr learned my addr 172.21.15.70:0/3001287032 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e6b7fe640 1 --2- 172.21.15.70:0/3001287032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879d1a00 0x561e879d3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e70b5e640 1 -- 172.21.15.70:0/3001287032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879d1a00 msgr2=0x561e879d3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e70b5e640 1 --2- 172.21.15.70:0/3001287032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879d1a00 0x561e879d3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e70b5e640 1 -- 172.21.15.70:0/3001287032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879c7140 msgr2=0x561e879d14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e70b5e640 1 --2- 172.21.15.70:0/3001287032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879c7140 0x561e879d14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e70b5e640 1 -- 172.21.15.70:0/3001287032 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e8763d210 con 0x561e879c6820 2024-06-04T05:12:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e6bfff640 1 --2- 172.21.15.70:0/3001287032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879c7140 0x561e879d14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e697fa640 1 --2- 172.21.15.70:0/2286437769 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0e540b2a10 0x7f0e540b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e70b5e640 1 --2- 172.21.15.70:0/3001287032 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c6820 0x561e879c6c00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0e64009f00 tx=0x7f0e64011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a7029fb49e89471 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e6bfff640 1 --2- 172.21.15.70:0/2286437769 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0e540b2a10 0x7f0e540b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e697fa640 1 -- 172.21.15.70:0/2286437769 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f0e5c0e2970 con 0x7f0e6c102a30 2024-06-04T05:12:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e527fc640 1 -- 172.21.15.70:0/3001287032 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e64004d20 con 0x561e879c6820 2024-06-04T05:12:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e527fc640 1 -- 172.21.15.70:0/3001287032 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0e64004030 con 0x561e879c6820 2024-06-04T05:12:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.233+0000 7f0e527fc640 1 -- 172.21.15.70:0/3001287032 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e64005240 con 0x561e879c6820 2024-06-04T05:12:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.234+0000 7f0e753b8740 1 -- 172.21.15.70:0/3001287032 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c6820 msgr2=0x561e879c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.234+0000 7f0e6bfff640 1 --2- 172.21.15.70:0/2286437769 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0e540b2a10 0x7f0e540b4ed0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c0a6f40 tx=0x7f0e5c0e8530 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.234+0000 7f0e753b8740 1 --2- 172.21.15.70:0/3001287032 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c6820 0x561e879c6c00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0e64009f00 tx=0x7f0e64011810 comp rx=0 tx=0).stop 2024-06-04T05:12:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.234+0000 7f0e753b8740 1 -- 172.21.15.70:0/3001287032 shutdown_connections 2024-06-04T05:12:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.234+0000 7f0e753b8740 1 --2- 172.21.15.70:0/3001287032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879d1a00 0x561e879d3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.234+0000 7f0e753b8740 1 --2- 172.21.15.70:0/3001287032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879c7140 0x561e879d14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.234+0000 7f0e753b8740 1 --2- 172.21.15.70:0/3001287032 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c6820 0x561e879c6c00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.234+0000 7f0e753b8740 1 -- 172.21.15.70:0/3001287032 >> 172.21.15.70:0/3001287032 conn(0x561e879c2d30 msgr2=0x561e879c3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.234+0000 7f0e753b8740 1 -- 172.21.15.70:0/3001287032 shutdown_connections 2024-06-04T05:12:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.234+0000 7f0e753b8740 1 -- 172.21.15.70:0/3001287032 wait complete. 2024-06-04T05:12:22.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.234+0000 7f0e753b8740 1 -- start start 2024-06-04T05:12:22.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e753b8740 1 Processor -- start 2024-06-04T05:12:22.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e753b8740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c6820 0x561e879ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e753b8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879c7140 0x561e879ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e753b8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879d1a00 0x561e879cf410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e753b8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561e8760e790 con 0x561e879d1a00 2024-06-04T05:12:22.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e753b8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561e878e4890 con 0x561e879c7140 2024-06-04T05:12:22.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e753b8740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561e879d4630 con 0x561e879c6820 2024-06-04T05:12:22.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e753b8740 10 client.0 _ll_register_callbacks cb 0x561e8763ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:22.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e70b5e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879d1a00 0x561e879cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e70b5e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879d1a00 0x561e879cf410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35906/0 (socket says 172.21.15.70:35906) 2024-06-04T05:12:22.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e70b5e640 1 -- 172.21.15.70:0/3322842331 learned_addr learned my addr 172.21.15.70:0/3322842331 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:22.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e70b5e640 1 --2- 172.21.15.70:0/3322842331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c6820 0x561e879ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e6b7fe640 1 --2- 172.21.15.70:0/3322842331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879c7140 0x561e879ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e70b5e640 1 -- 172.21.15.70:0/3322842331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c6820 msgr2=0x561e879ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:22.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e70b5e640 1 --2- 172.21.15.70:0/3322842331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c6820 0x561e879ce990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e70b5e640 1 -- 172.21.15.70:0/3322842331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879c7140 msgr2=0x561e879ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:22.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e70b5e640 1 --2- 172.21.15.70:0/3322842331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879c7140 0x561e879ceed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.235+0000 7f0e70b5e640 1 -- 172.21.15.70:0/3322842331 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e87709620 con 0x561e879d1a00 2024-06-04T05:12:22.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.236+0000 7f0e70b5e640 1 --2- 172.21.15.70:0/3322842331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879d1a00 0x561e879cf410 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f0e64004700 tx=0x7f0e64004870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52231aee34eeeda5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:22.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.236+0000 7f0e4b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:12:22.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.236+0000 7f0e4b7fe640 1 -- 172.21.15.70:0/3322842331 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6401e040 con 0x561e879d1a00 2024-06-04T05:12:22.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.236+0000 7f0e4b7fe640 1 -- 172.21.15.70:0/3322842331 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f0e64004a40 con 0x561e879d1a00 2024-06-04T05:12:22.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.236+0000 7f0e753b8740 1 -- 172.21.15.70:0/3322842331 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e64004030 con 0x561e879d1a00 2024-06-04T05:12:22.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.236+0000 7f0e4b7fe640 1 -- 172.21.15.70:0/3322842331 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f0e64004c40 con 0x561e879d1a00 2024-06-04T05:12:22.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.236+0000 7f0e753b8740 10 client.19446 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:22.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.236+0000 7f0e753b8740 1 -- 172.21.15.70:0/3322842331 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x561e879e0380 con 0x561e879d1a00 2024-06-04T05:12:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.236+0000 7f0e4b7fe640 1 -- 172.21.15.70:0/3322842331 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0e64004030 con 0x561e879d1a00 2024-06-04T05:12:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.236+0000 7f0e4b7fe640 1 -- 172.21.15.70:0/3322842331 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7f0e64024070 con 0x561e879d1a00 2024-06-04T05:12:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.236+0000 7f0e753b8740 10 client.19446 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.236+0000 7f0e753b8740 10 client.19446 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:12:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.236+0000 7f0e753b8740 20 client.19446 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.236+0000 7f0e753b8740 1 --2- 172.21.15.70:0/3322842331 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561e879e17e0 0x561e879e3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.236+0000 7f0e753b8740 4 client.19446 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:12:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.236+0000 7f0e753b8740 1 -- 172.21.15.70:0/3322842331 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e8793ab80 con 0x561e879e17e0 2024-06-04T05:12:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.237+0000 7f0e6b7fe640 1 --2- 172.21.15.70:0/3322842331 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561e879e17e0 0x561e879e3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.237+0000 7f0e4b7fe640 10 client.19446 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.237+0000 7f0e6b7fe640 1 --2- 172.21.15.70:0/3322842331 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561e879e17e0 0x561e879e3bd0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4d5d691929f30d4 server_cookie=ffdb572c3306d0f2 in_seq=0 out_seq=0 2024-06-04T05:12:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.238+0000 7f0e4b7fe640 1 -- 172.21.15.70:0/3322842331 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561e8793ab80 con 0x561e879e17e0 2024-06-04T05:12:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.238+0000 7f0e4b7fe640 10 client.19446 handle_command_reply: tid=0 2024-06-04T05:12:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.238+0000 7f0e753b8740 1 client.19446 shutdown 2024-06-04T05:12:22.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.238+0000 7f0e753b8740 1 -- 172.21.15.70:0/3322842331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879d1a00 msgr2=0x561e879cf410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:22.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.238+0000 7f0e753b8740 1 --2- 172.21.15.70:0/3322842331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879d1a00 0x561e879cf410 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f0e64004700 tx=0x7f0e64004870 comp rx=0 tx=0).stop 2024-06-04T05:12:22.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.238+0000 7f0e753b8740 1 -- 172.21.15.70:0/3322842331 shutdown_connections 2024-06-04T05:12:22.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.238+0000 7f0e753b8740 1 --2- 172.21.15.70:0/3322842331 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561e879e17e0 0x561e879e3bd0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.238+0000 7f0e753b8740 1 --2- 172.21.15.70:0/3322842331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879c7140 0x561e879ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.238+0000 7f0e753b8740 1 --2- 172.21.15.70:0/3322842331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879d1a00 0x561e879cf410 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.238+0000 7f0e4b7fe640 0 client.19446 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:22.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.238+0000 7f0e753b8740 1 --2- 172.21.15.70:0/3322842331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c6820 0x561e879ce990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.238+0000 7f0e753b8740 1 -- 172.21.15.70:0/3322842331 >> 172.21.15.70:0/3322842331 conn(0x561e879c2d30 msgr2=0x561e879c1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:22.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.238+0000 7f0e753b8740 1 -- 172.21.15.70:0/3322842331 shutdown_connections 2024-06-04T05:12:22.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.238+0000 7f0e753b8740 1 -- 172.21.15.70:0/3322842331 wait complete. 2024-06-04T05:12:22.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.238+0000 7f0e753b8740 20 client.19446 trim_cache size 0 max 16384 2024-06-04T05:12:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.253+0000 7f0e753b8740 1 Processor -- start 2024-06-04T05:12:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.253+0000 7f0e753b8740 1 -- start start 2024-06-04T05:12:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.253+0000 7f0e753b8740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c6820 0x561e879e93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.253+0000 7f0e753b8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879c7140 0x561e879e98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.253+0000 7f0e753b8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879d1a00 0x561e879cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.253+0000 7f0e753b8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c1ae6c0 con 0x561e879c7140 2024-06-04T05:12:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.253+0000 7f0e753b8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c1b0380 con 0x561e879d1a00 2024-06-04T05:12:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.253+0000 7f0e753b8740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c1aed90 con 0x561e879c6820 2024-06-04T05:12:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.253+0000 7f0e6b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879c7140 0x561e879e98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.253+0000 7f0e6b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879c7140 0x561e879e98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35908/0 (socket says 172.21.15.70:35908) 2024-06-04T05:12:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.253+0000 7f0e6b7fe640 1 -- 172.21.15.70:0/392223228 learned_addr learned my addr 172.21.15.70:0/392223228 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.253+0000 7f0e70b5e640 1 --2- 172.21.15.70:0/392223228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c6820 0x561e879e93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.253+0000 7f0e70b5e640 1 --2- 172.21.15.70:0/392223228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879d1a00 0x561e879cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.254+0000 7f0e6b7fe640 1 -- 172.21.15.70:0/392223228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c6820 msgr2=0x561e879e93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.254+0000 7f0e6b7fe640 1 --2- 172.21.15.70:0/392223228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c6820 0x561e879e93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.254+0000 7f0e6b7fe640 1 -- 172.21.15.70:0/392223228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879d1a00 msgr2=0x561e879cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.254+0000 7f0e6b7fe640 1 --2- 172.21.15.70:0/392223228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879d1a00 0x561e879cbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.254+0000 7f0e6b7fe640 1 -- 172.21.15.70:0/392223228 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e879c7680 con 0x561e879c7140 2024-06-04T05:12:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.254+0000 7f0e70b5e640 1 --2- 172.21.15.70:0/392223228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879d1a00 0x561e879cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:12:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.254+0000 7f0e6b7fe640 1 --2- 172.21.15.70:0/392223228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879c7140 0x561e879e98f0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f0e60005ec0 tx=0x7f0e60004330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5638e79c0b3190d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.254+0000 7f0e51ffb640 1 -- 172.21.15.70:0/392223228 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e60035040 con 0x561e879c7140 2024-06-04T05:12:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.254+0000 7f0e51ffb640 1 -- 172.21.15.70:0/392223228 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0e6004b070 con 0x561e879c7140 2024-06-04T05:12:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.254+0000 7f0e51ffb640 1 -- 172.21.15.70:0/392223228 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e60046bc0 con 0x561e879c7140 2024-06-04T05:12:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.254+0000 7f0e753b8740 1 -- 172.21.15.70:0/392223228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879c7140 msgr2=0x561e879e98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.254+0000 7f0e753b8740 1 --2- 172.21.15.70:0/392223228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879c7140 0x561e879e98f0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f0e60005ec0 tx=0x7f0e60004330 comp rx=0 tx=0).stop 2024-06-04T05:12:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.255+0000 7f0e753b8740 1 -- 172.21.15.70:0/392223228 shutdown_connections 2024-06-04T05:12:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.255+0000 7f0e753b8740 1 --2- 172.21.15.70:0/392223228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879d1a00 0x561e879cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.255+0000 7f0e753b8740 1 --2- 172.21.15.70:0/392223228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879c7140 0x561e879e98f0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.255+0000 7f0e753b8740 1 --2- 172.21.15.70:0/392223228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c6820 0x561e879e93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.255+0000 7f0e753b8740 1 -- 172.21.15.70:0/392223228 >> 172.21.15.70:0/392223228 conn(0x561e879c2d30 msgr2=0x561e879cf1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.255+0000 7f0e753b8740 1 -- 172.21.15.70:0/392223228 shutdown_connections 2024-06-04T05:12:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.255+0000 7f0e753b8740 1 -- 172.21.15.70:0/392223228 wait complete. 2024-06-04T05:12:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.255+0000 7f0e753b8740 1 -- start start 2024-06-04T05:12:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.255+0000 7f0e753b8740 1 Processor -- start 2024-06-04T05:12:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.255+0000 7f0e753b8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879c6820 0x561e879d5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.255+0000 7f0e753b8740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c7140 0x561e879d5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.255+0000 7f0e753b8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879d1a00 0x561e879d6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.255+0000 7f0e753b8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c1afbf0 con 0x561e879d1a00 2024-06-04T05:12:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.255+0000 7f0e753b8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c1af4c0 con 0x561e879c6820 2024-06-04T05:12:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.255+0000 7f0e753b8740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c1ac6c0 con 0x561e879c7140 2024-06-04T05:12:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.255+0000 7f0e753b8740 10 client.0 _ll_register_callbacks cb 0x7f0e5c0e8400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.255+0000 7f0e70b5e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879d1a00 0x561e879d6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e6b7fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c7140 0x561e879d5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e6b7fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c7140 0x561e879d5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:42948/0 (socket says 172.21.15.70:42948) 2024-06-04T05:12:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e6b7fe640 1 -- 172.21.15.70:0/3790025349 learned_addr learned my addr 172.21.15.70:0/3790025349 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e70b5e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879d1a00 0x561e879d6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35916/0 (socket says 172.21.15.70:35916) 2024-06-04T05:12:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e70b5e640 1 --2- 172.21.15.70:0/3790025349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879c6820 0x561e879d5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e6b7fe640 1 -- 172.21.15.70:0/3790025349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879c6820 msgr2=0x561e879d5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e6b7fe640 1 --2- 172.21.15.70:0/3790025349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879c6820 0x561e879d5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e6b7fe640 1 -- 172.21.15.70:0/3790025349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879d1a00 msgr2=0x561e879d6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e6b7fe640 1 --2- 172.21.15.70:0/3790025349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879d1a00 0x561e879d6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e6b7fe640 1 -- 172.21.15.70:0/3790025349 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e87a4b480 con 0x561e879c7140 2024-06-04T05:12:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e70b5e640 1 --2- 172.21.15.70:0/3790025349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879d1a00 0x561e879d6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e6b7fe640 1 --2- 172.21.15.70:0/3790025349 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c7140 0x561e879d5cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0e60009850 tx=0x7f0e60004590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa53891d7b5276ee server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:22.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e4b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:12:22.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e4b7fe640 1 -- 172.21.15.70:0/3790025349 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e60035040 con 0x561e879c7140 2024-06-04T05:12:22.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e4b7fe640 1 -- 172.21.15.70:0/3790025349 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f0e60046a40 con 0x561e879c7140 2024-06-04T05:12:22.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e753b8740 1 -- 172.21.15.70:0/3790025349 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e6004b070 con 0x561e879c7140 2024-06-04T05:12:22.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e4b7fe640 1 -- 172.21.15.70:0/3790025349 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f0e60046c40 con 0x561e879c7140 2024-06-04T05:12:22.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e753b8740 10 client.28435 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:22.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.256+0000 7f0e753b8740 1 -- 172.21.15.70:0/3790025349 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x561e87a4c750 con 0x561e879c7140 2024-06-04T05:12:22.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.257+0000 7f0e4b7fe640 1 -- 172.21.15.70:0/3790025349 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0e6004b070 con 0x561e879c7140 2024-06-04T05:12:22.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.257+0000 7f0e4b7fe640 1 -- 172.21.15.70:0/3790025349 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7f0e60061070 con 0x561e879c7140 2024-06-04T05:12:22.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.257+0000 7f0e753b8740 10 client.28435 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:22.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.257+0000 7f0e753b8740 10 client.28435 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:12:22.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.257+0000 7f0e753b8740 20 client.28435 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:22.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.257+0000 7f0e753b8740 1 --2- 172.21.15.70:0/3790025349 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561e87a4ce20 0x561e87a4f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:22.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.257+0000 7f0e753b8740 4 client.28435 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:12:22.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.257+0000 7f0e753b8740 1 -- 172.21.15.70:0/3790025349 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561e87934360 con 0x561e87a4ce20 2024-06-04T05:12:22.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.257+0000 7f0e70b5e640 1 --2- 172.21.15.70:0/3790025349 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561e87a4ce20 0x561e87a4f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:22.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.257+0000 7f0e70b5e640 1 --2- 172.21.15.70:0/3790025349 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561e87a4ce20 0x561e87a4f240 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dabdbfdbd40a7a75 server_cookie=f998ec76e739118f in_seq=0 out_seq=0 2024-06-04T05:12:22.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.257+0000 7f0e4b7fe640 10 client.28435 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.332+0000 7f0e4b7fe640 1 -- 172.21.15.70:0/3790025349 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x561e87934360 con 0x561e87a4ce20 2024-06-04T05:12:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.332+0000 7f0e4b7fe640 10 client.28435 handle_command_reply: tid=0 2024-06-04T05:12:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.332+0000 7f0e753b8740 1 client.28435 shutdown 2024-06-04T05:12:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.332+0000 7f0e753b8740 1 -- 172.21.15.70:0/3790025349 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c7140 msgr2=0x561e879d5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.332+0000 7f0e753b8740 1 --2- 172.21.15.70:0/3790025349 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c7140 0x561e879d5cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0e60009850 tx=0x7f0e60004590 comp rx=0 tx=0).stop 2024-06-04T05:12:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.332+0000 7f0e753b8740 1 -- 172.21.15.70:0/3790025349 shutdown_connections 2024-06-04T05:12:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.332+0000 7f0e753b8740 1 --2- 172.21.15.70:0/3790025349 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561e87a4ce20 0x561e87a4f240 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.332+0000 7f0e753b8740 1 --2- 172.21.15.70:0/3790025349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e879d1a00 0x561e879d6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.332+0000 7f0e753b8740 1 --2- 172.21.15.70:0/3790025349 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561e879c7140 0x561e879d5cc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.332+0000 7f0e4b7fe640 0 client.28435 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.332+0000 7f0e753b8740 1 --2- 172.21.15.70:0/3790025349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e879c6820 0x561e879d5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.332+0000 7f0e753b8740 1 -- 172.21.15.70:0/3790025349 >> 172.21.15.70:0/3790025349 conn(0x561e879c2d30 msgr2=0x561e879ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.332+0000 7f0e753b8740 1 -- 172.21.15.70:0/3790025349 shutdown_connections 2024-06-04T05:12:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.332+0000 7f0e753b8740 1 -- 172.21.15.70:0/3790025349 wait complete. 2024-06-04T05:12:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.333+0000 7f0e753b8740 20 client.28435 trim_cache size 0 max 16384 2024-06-04T05:12:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:12:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:12:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:12:22.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.333+0000 7f0e725c1640 1 -- 172.21.15.70:0/2286437769 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0e540b2a10 msgr2=0x7f0e540b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:22.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.333+0000 7f0e725c1640 1 --2- 172.21.15.70:0/2286437769 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0e540b2a10 0x7f0e540b4ed0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c0a6f40 tx=0x7f0e5c0e8530 comp rx=0 tx=0).stop 2024-06-04T05:12:22.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.333+0000 7f0e725c1640 1 -- 172.21.15.70:0/2286437769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e6c102a30 msgr2=0x7f0e6c199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:22.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.333+0000 7f0e725c1640 1 --2- 172.21.15.70:0/2286437769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e6c102a30 0x7f0e6c199130 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00e9f0 tx=0x7f0e5c00eeb0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.333+0000 7f0e725c1640 1 -- 172.21.15.70:0/2286437769 shutdown_connections 2024-06-04T05:12:22.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.333+0000 7f0e725c1640 1 --2- 172.21.15.70:0/2286437769 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0e540b2a10 0x7f0e540b4ed0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.333+0000 7f0e725c1640 1 --2- 172.21.15.70:0/2286437769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e6c1033f0 0x7f0e6c199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.333+0000 7f0e725c1640 1 --2- 172.21.15.70:0/2286437769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e6c108a40 0x7f0e6c19d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.333+0000 7f0e725c1640 1 --2- 172.21.15.70:0/2286437769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e6c102a30 0x7f0e6c199130 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:22.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.333+0000 7f0e725c1640 1 -- 172.21.15.70:0/2286437769 >> 172.21.15.70:0/2286437769 conn(0x7f0e6c0fe500 msgr2=0x7f0e6c0fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:22.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.334+0000 7f0e725c1640 1 -- 172.21.15.70:0/2286437769 shutdown_connections 2024-06-04T05:12:22.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:22.334+0000 7f0e725c1640 1 -- 172.21.15.70:0/2286437769 wait complete. 2024-06-04T05:12:22.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 13278 12.04 MB/sec warmup 74 sec latency 137.887 ms 2024-06-04T05:12:23.285 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 32.6 2024-06-04T05:12:23.286 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:12:23.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:23 smithi098 ceph-mon[35569]: pgmap v475: 97 pgs: 97 active+clean; 415 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.7 MiB/s wr, 404 op/s 2024-06-04T05:12:23.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:23 smithi070 ceph-mon[32631]: pgmap v475: 97 pgs: 97 active+clean; 415 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.7 MiB/s wr, 404 op/s 2024-06-04T05:12:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:23 smithi112 ceph-mon[36453]: pgmap v475: 97 pgs: 97 active+clean; 415 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.7 MiB/s wr, 404 op/s 2024-06-04T05:12:23.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 13410 12.00 MB/sec warmup 75 sec latency 157.427 ms 2024-06-04T05:12:24.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 13547 11.90 MB/sec warmup 76 sec latency 150.537 ms 2024-06-04T05:12:25.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:25 smithi098 ceph-mon[35569]: pgmap v476: 97 pgs: 97 active+clean; 415 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 258 op/s 2024-06-04T05:12:25.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:25 smithi070 ceph-mon[32631]: pgmap v476: 97 pgs: 97 active+clean; 415 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 258 op/s 2024-06-04T05:12:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:25 smithi112 ceph-mon[36453]: pgmap v476: 97 pgs: 97 active+clean; 415 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 258 op/s 2024-06-04T05:12:25.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 13698 11.92 MB/sec warmup 77 sec latency 137.404 ms 2024-06-04T05:12:26.800 INFO:tasks.workunit.client.0.smithi070.stdout: 10 13840 11.81 MB/sec warmup 78 sec latency 132.699 ms 2024-06-04T05:12:27.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:27 smithi098 ceph-mon[35569]: pgmap v477: 97 pgs: 97 active+clean; 421 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.4 MiB/s wr, 381 op/s 2024-06-04T05:12:27.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:27 smithi070 ceph-mon[32631]: pgmap v477: 97 pgs: 97 active+clean; 421 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.4 MiB/s wr, 381 op/s 2024-06-04T05:12:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:27 smithi112 ceph-mon[36453]: pgmap v477: 97 pgs: 97 active+clean; 421 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.4 MiB/s wr, 381 op/s 2024-06-04T05:12:27.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 13988 11.75 MB/sec warmup 79 sec latency 137.896 ms 2024-06-04T05:12:28.151 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:12:28.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 14145 11.66 MB/sec warmup 80 sec latency 90.771 ms 2024-06-04T05:12:29.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.400+0000 7f735aad9700 1 -- 172.21.15.70:0/2517399981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7354108730 msgr2=0x7f7354108b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:29.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.400+0000 7f735aad9700 1 --2- 172.21.15.70:0/2517399981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7354108730 0x7f7354108b10 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f73480099c0 tx=0x7f7348009cd0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.400+0000 7f735aad9700 1 -- 172.21.15.70:0/2517399981 shutdown_connections 2024-06-04T05:12:29.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.400+0000 7f735aad9700 1 --2- 172.21.15.70:0/2517399981 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f73541039d0 0x7f735410f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.400+0000 7f735aad9700 1 --2- 172.21.15.70:0/2517399981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7354108730 0x7f7354108b10 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.400+0000 7f735aad9700 1 --2- 172.21.15.70:0/2517399981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7354103010 0x7f7354103490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.400+0000 7f735aad9700 1 -- 172.21.15.70:0/2517399981 >> 172.21.15.70:0/2517399981 conn(0x7f7354077780 msgr2=0x7f7354077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:29.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.400+0000 7f735aad9700 1 -- 172.21.15.70:0/2517399981 shutdown_connections 2024-06-04T05:12:29.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.401+0000 7f735aad9700 1 -- 172.21.15.70:0/2517399981 wait complete. 2024-06-04T05:12:29.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.401+0000 7f735aad9700 1 Processor -- start 2024-06-04T05:12:29.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.402+0000 7f735aad9700 1 -- start start 2024-06-04T05:12:29.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.402+0000 7f735aad9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7354103010 0x7f735419bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.402+0000 7f735aad9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73541039d0 0x7f735419c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.402+0000 7f735aad9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7354108730 0x7f73541a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.402+0000 7f735aad9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f73541159d0 con 0x7f73541039d0 2024-06-04T05:12:29.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.402+0000 7f735aad9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7354115850 con 0x7f7354103010 2024-06-04T05:12:29.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.402+0000 7f735aad9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7354115b50 con 0x7f7354108730 2024-06-04T05:12:29.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.402+0000 7f7353fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73541039d0 0x7f735419c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.402+0000 7f7353fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73541039d0 0x7f735419c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54624/0 (socket says 172.21.15.70:54624) 2024-06-04T05:12:29.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.402+0000 7f7353fff700 1 -- 172.21.15.70:0/914005514 learned_addr learned my addr 172.21.15.70:0/914005514 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:29.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.402+0000 7f7358875700 1 --2- 172.21.15.70:0/914005514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7354103010 0x7f735419bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.402+0000 7f7359076700 1 --2- 172.21.15.70:0/914005514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7354108730 0x7f73541a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.402+0000 7f7353fff700 1 -- 172.21.15.70:0/914005514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7354108730 msgr2=0x7f73541a0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:29.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.402+0000 7f7353fff700 1 --2- 172.21.15.70:0/914005514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7354108730 0x7f73541a0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.403+0000 7f7353fff700 1 -- 172.21.15.70:0/914005514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7354103010 msgr2=0x7f735419bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:29.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.403+0000 7f7353fff700 1 --2- 172.21.15.70:0/914005514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7354103010 0x7f735419bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.403+0000 7f7353fff700 1 -- 172.21.15.70:0/914005514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73541a0e80 con 0x7f73541039d0 2024-06-04T05:12:29.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.403+0000 7f7359076700 1 --2- 172.21.15.70:0/914005514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7354108730 0x7f73541a0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:12:29.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.403+0000 7f7353fff700 1 --2- 172.21.15.70:0/914005514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73541039d0 0x7f735419c400 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f734000eb30 tx=0x7f734000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:29.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.403+0000 7f7351ffb700 1 -- 172.21.15.70:0/914005514 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734000cc90 con 0x7f73541039d0 2024-06-04T05:12:29.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.403+0000 7f7351ffb700 1 -- 172.21.15.70:0/914005514 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f734000ce30 con 0x7f73541039d0 2024-06-04T05:12:29.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.403+0000 7f735aad9700 1 -- 172.21.15.70:0/914005514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73541a1170 con 0x7f73541039d0 2024-06-04T05:12:29.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.403+0000 7f7351ffb700 1 -- 172.21.15.70:0/914005514 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7340003e30 con 0x7f73541039d0 2024-06-04T05:12:29.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.403+0000 7f735aad9700 1 -- 172.21.15.70:0/914005514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73541a89f0 con 0x7f73541039d0 2024-06-04T05:12:29.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.406+0000 7f7351ffb700 1 -- 172.21.15.70:0/914005514 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7340004000 con 0x7f73541039d0 2024-06-04T05:12:29.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.407+0000 7f7351ffb700 1 --2- 172.21.15.70:0/914005514 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73440b29e0 0x7f73440b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.407+0000 7f7358875700 1 --2- 172.21.15.70:0/914005514 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73440b29e0 0x7f73440b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.407+0000 7f7351ffb700 1 -- 172.21.15.70:0/914005514 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f73400e2960 con 0x7f73541039d0 2024-06-04T05:12:29.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.408+0000 7f7358875700 1 --2- 172.21.15.70:0/914005514 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73440b29e0 0x7f73440b4ea0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7348003760 tx=0x7f7348005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:29.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.408+0000 7f736536eb80 1 Processor -- start 2024-06-04T05:12:29.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.408+0000 7f736536eb80 1 -- start start 2024-06-04T05:12:29.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f736536eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 0x561ecda4ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f736536eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4f290 0x561ecda529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f736536eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecda53420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f736536eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561ecd9add10 con 0x561ecda4f290 2024-06-04T05:12:29.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f736536eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561ecd99d8b0 con 0x561ecda4e8e0 2024-06-04T05:12:29.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f736536eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561ecd9a18f0 con 0x561ecda52fe0 2024-06-04T05:12:29.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f7358875700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 0x561ecda4ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f7359076700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecda53420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f7359076700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecda53420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:52544/0 (socket says 172.21.15.70:52544) 2024-06-04T05:12:29.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f7358875700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 0x561ecda4ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45428/0 (socket says 172.21.15.70:45428) 2024-06-04T05:12:29.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f7359076700 1 -- 172.21.15.70:0/3431390242 learned_addr learned my addr 172.21.15.70:0/3431390242 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:29.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f7353fff700 1 --2- 172.21.15.70:0/3431390242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4f290 0x561ecda529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f7359076700 1 -- 172.21.15.70:0/3431390242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 msgr2=0x561ecda4ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:29.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f7359076700 1 --2- 172.21.15.70:0/3431390242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 0x561ecda4ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f7359076700 1 -- 172.21.15.70:0/3431390242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4f290 msgr2=0x561ecda529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:29.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f7359076700 1 --2- 172.21.15.70:0/3431390242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4f290 0x561ecda529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f7359076700 1 -- 172.21.15.70:0/3431390242 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ecd8ed950 con 0x561ecda52fe0 2024-06-04T05:12:29.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f7353fff700 1 --2- 172.21.15.70:0/3431390242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4f290 0x561ecda529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:29.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.409+0000 7f7359076700 1 --2- 172.21.15.70:0/3431390242 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecda53420 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f734c0093d0 tx=0x7f734c009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3da52b7056f9e28d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:29.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.410+0000 7f733effd700 1 -- 172.21.15.70:0/3431390242 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734c007e60 con 0x561ecda52fe0 2024-06-04T05:12:29.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.410+0000 7f733effd700 1 -- 172.21.15.70:0/3431390242 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f734c019070 con 0x561ecda52fe0 2024-06-04T05:12:29.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.410+0000 7f733effd700 1 -- 172.21.15.70:0/3431390242 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734c013970 con 0x561ecda52fe0 2024-06-04T05:12:29.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.411+0000 7f736536eb80 1 -- 172.21.15.70:0/3431390242 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 msgr2=0x561ecda53420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:29.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.411+0000 7f736536eb80 1 --2- 172.21.15.70:0/3431390242 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecda53420 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f734c0093d0 tx=0x7f734c009920 comp rx=0 tx=0).stop 2024-06-04T05:12:29.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.411+0000 7f736536eb80 1 -- 172.21.15.70:0/3431390242 shutdown_connections 2024-06-04T05:12:29.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.411+0000 7f736536eb80 1 --2- 172.21.15.70:0/3431390242 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecda53420 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.411+0000 7f736536eb80 1 --2- 172.21.15.70:0/3431390242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 0x561ecda4ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.411+0000 7f736536eb80 1 --2- 172.21.15.70:0/3431390242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4f290 0x561ecda529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.411+0000 7f736536eb80 1 -- 172.21.15.70:0/3431390242 >> 172.21.15.70:0/3431390242 conn(0x561ecd97c100 msgr2=0x561ecda48f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:29.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.411+0000 7f736536eb80 1 -- 172.21.15.70:0/3431390242 shutdown_connections 2024-06-04T05:12:29.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.411+0000 7f736536eb80 1 -- 172.21.15.70:0/3431390242 wait complete. 2024-06-04T05:12:29.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.411+0000 7f736536eb80 1 -- start start 2024-06-04T05:12:29.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.411+0000 7f736536eb80 1 Processor -- start 2024-06-04T05:12:29.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f736536eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 0x561ecda5a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f736536eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4f290 0x561ecda57730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f736536eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecda57c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f736536eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561ecd8a4020 con 0x561ecda4f290 2024-06-04T05:12:29.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f736536eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561ecd93f840 con 0x561ecda4e8e0 2024-06-04T05:12:29.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f736536eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561ecda5b1a0 con 0x561ecda52fe0 2024-06-04T05:12:29.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f7358875700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 0x561ecda5a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f736536eb80 10 client.0 _ll_register_callbacks cb 0x561ecd9389a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:29.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f7358875700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 0x561ecda5a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45430/0 (socket says 172.21.15.70:45430) 2024-06-04T05:12:29.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f7358875700 1 -- 172.21.15.70:0/153855382 learned_addr learned my addr 172.21.15.70:0/153855382 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:29.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f7359076700 1 --2- 172.21.15.70:0/153855382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecda57c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f7353fff700 1 --2- 172.21.15.70:0/153855382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4f290 0x561ecda57730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f7358875700 1 -- 172.21.15.70:0/153855382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 msgr2=0x561ecda57c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f7358875700 1 --2- 172.21.15.70:0/153855382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecda57c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f7358875700 1 -- 172.21.15.70:0/153855382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4f290 msgr2=0x561ecda57730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f7358875700 1 --2- 172.21.15.70:0/153855382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4f290 0x561ecda57730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f7358875700 1 -- 172.21.15.70:0/153855382 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ecd93c710 con 0x561ecda4e8e0 2024-06-04T05:12:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.412+0000 7f7359076700 1 --2- 172.21.15.70:0/153855382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecda57c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.413+0000 7f7358875700 1 --2- 172.21.15.70:0/153855382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 0x561ecda5a430 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f73480099c0 tx=0x7f7348009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7d1768752366369 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.413+0000 7f7333fff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:12:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.413+0000 7f7333fff700 1 -- 172.21.15.70:0/153855382 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734800b9a0 con 0x561ecda4e8e0 2024-06-04T05:12:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.413+0000 7f7333fff700 1 -- 172.21.15.70:0/153855382 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f734800bb40 con 0x561ecda4e8e0 2024-06-04T05:12:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.413+0000 7f736536eb80 1 -- 172.21.15.70:0/153855382 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f734c019070 con 0x561ecda4e8e0 2024-06-04T05:12:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.413+0000 7f7333fff700 1 -- 172.21.15.70:0/153855382 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f734803d050 con 0x561ecda4e8e0 2024-06-04T05:12:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.413+0000 7f736536eb80 10 client.28331 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.413+0000 7f736536eb80 1 -- 172.21.15.70:0/153855382 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x561ecda658a0 con 0x561ecda4e8e0 2024-06-04T05:12:29.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.413+0000 7f7333fff700 1 -- 172.21.15.70:0/153855382 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f734c019070 con 0x561ecda4e8e0 2024-06-04T05:12:29.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.414+0000 7f7333fff700 1 -- 172.21.15.70:0/153855382 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7f7348010470 con 0x561ecda4e8e0 2024-06-04T05:12:29.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.414+0000 7f736536eb80 10 client.28331 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:29.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.414+0000 7f736536eb80 10 client.28331 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:12:29.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.414+0000 7f736536eb80 20 client.28331 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:29.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.414+0000 7f736536eb80 1 --2- 172.21.15.70:0/153855382 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561ecda68340 0x561ecda6a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.414+0000 7f736536eb80 4 client.28331 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:12:29.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.414+0000 7f736536eb80 1 -- 172.21.15.70:0/153855382 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ecda1aa60 con 0x561ecda68340 2024-06-04T05:12:29.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.414+0000 7f7353fff700 1 --2- 172.21.15.70:0/153855382 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561ecda68340 0x561ecda6a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.414+0000 7f7333fff700 10 client.28331 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:29.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.414+0000 7f7353fff700 1 --2- 172.21.15.70:0/153855382 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561ecda68340 0x561ecda6a7a0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c10cbd0a526e1966 server_cookie=416119a396315d6c in_seq=0 out_seq=0 2024-06-04T05:12:29.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.415+0000 7f7333fff700 1 -- 172.21.15.70:0/153855382 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561ecda1aa60 con 0x561ecda68340 2024-06-04T05:12:29.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.415+0000 7f7333fff700 10 client.28331 handle_command_reply: tid=0 2024-06-04T05:12:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.415+0000 7f736536eb80 1 client.28331 shutdown 2024-06-04T05:12:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.415+0000 7f736536eb80 1 -- 172.21.15.70:0/153855382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 msgr2=0x561ecda5a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.415+0000 7f736536eb80 1 --2- 172.21.15.70:0/153855382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 0x561ecda5a430 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f73480099c0 tx=0x7f7348009cd0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.415+0000 7f736536eb80 1 -- 172.21.15.70:0/153855382 shutdown_connections 2024-06-04T05:12:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.415+0000 7f736536eb80 1 --2- 172.21.15.70:0/153855382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecda57c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.415+0000 7f736536eb80 1 --2- 172.21.15.70:0/153855382 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561ecda68340 0x561ecda6a7a0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.415+0000 7f736536eb80 1 --2- 172.21.15.70:0/153855382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 0x561ecda5a430 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.415+0000 7f736536eb80 1 --2- 172.21.15.70:0/153855382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4f290 0x561ecda57730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.415+0000 7f736536eb80 1 -- 172.21.15.70:0/153855382 >> 172.21.15.70:0/153855382 conn(0x561ecd97c100 msgr2=0x561ecda48590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.415+0000 7f7333fff700 0 client.28331 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.416+0000 7f736536eb80 1 -- 172.21.15.70:0/153855382 shutdown_connections 2024-06-04T05:12:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.416+0000 7f736536eb80 1 -- 172.21.15.70:0/153855382 wait complete. 2024-06-04T05:12:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.416+0000 7f736536eb80 20 client.28331 trim_cache size 0 max 16384 2024-06-04T05:12:29.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.447+0000 7f736536eb80 1 Processor -- start 2024-06-04T05:12:29.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.447+0000 7f736536eb80 1 -- start start 2024-06-04T05:12:29.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.447+0000 7f736536eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4e8e0 0x561ecda48d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.447+0000 7f736536eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4f290 0x561ecda700b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.447+0000 7f736536eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecda705f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.447+0000 7f736536eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f73541b0060 con 0x561ecda4e8e0 2024-06-04T05:12:29.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.447+0000 7f736536eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f73541b1d20 con 0x561ecda4f290 2024-06-04T05:12:29.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.447+0000 7f736536eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f73541b0730 con 0x561ecda52fe0 2024-06-04T05:12:29.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.447+0000 7f7358875700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4e8e0 0x561ecda48d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.447+0000 7f7358875700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4e8e0 0x561ecda48d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54662/0 (socket says 172.21.15.70:54662) 2024-06-04T05:12:29.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.448+0000 7f7358875700 1 -- 172.21.15.70:0/2048523270 learned_addr learned my addr 172.21.15.70:0/2048523270 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:29.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.448+0000 7f7359076700 1 --2- 172.21.15.70:0/2048523270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecda705f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.448+0000 7f7353fff700 1 --2- 172.21.15.70:0/2048523270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4f290 0x561ecda700b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.448+0000 7f7358875700 1 -- 172.21.15.70:0/2048523270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 msgr2=0x561ecda705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:29.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.448+0000 7f7358875700 1 --2- 172.21.15.70:0/2048523270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecda705f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.448+0000 7f7358875700 1 -- 172.21.15.70:0/2048523270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4f290 msgr2=0x561ecda700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:29.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.448+0000 7f7358875700 1 --2- 172.21.15.70:0/2048523270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4f290 0x561ecda700b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.448+0000 7f7358875700 1 -- 172.21.15.70:0/2048523270 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ecda585e0 con 0x561ecda4e8e0 2024-06-04T05:12:29.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.448+0000 7f7358875700 1 --2- 172.21.15.70:0/2048523270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4e8e0 0x561ecda48d10 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f7348000c00 tx=0x7f7348010d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e391aca0537581d0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:29.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.448+0000 7f733e7fc700 1 -- 172.21.15.70:0/2048523270 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734800b9a0 con 0x561ecda4e8e0 2024-06-04T05:12:29.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.448+0000 7f733e7fc700 1 -- 172.21.15.70:0/2048523270 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7348034070 con 0x561ecda4e8e0 2024-06-04T05:12:29.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.448+0000 7f733e7fc700 1 -- 172.21.15.70:0/2048523270 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7348023b10 con 0x561ecda4e8e0 2024-06-04T05:12:29.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.449+0000 7f736536eb80 1 -- 172.21.15.70:0/2048523270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4e8e0 msgr2=0x561ecda48d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:29.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.449+0000 7f736536eb80 1 --2- 172.21.15.70:0/2048523270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4e8e0 0x561ecda48d10 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f7348000c00 tx=0x7f7348010d00 comp rx=0 tx=0).stop 2024-06-04T05:12:29.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.449+0000 7f736536eb80 1 -- 172.21.15.70:0/2048523270 shutdown_connections 2024-06-04T05:12:29.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.449+0000 7f736536eb80 1 --2- 172.21.15.70:0/2048523270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecda705f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.449+0000 7f736536eb80 1 --2- 172.21.15.70:0/2048523270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4e8e0 0x561ecda48d10 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.449+0000 7f736536eb80 1 --2- 172.21.15.70:0/2048523270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4f290 0x561ecda700b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.449+0000 7f736536eb80 1 -- 172.21.15.70:0/2048523270 >> 172.21.15.70:0/2048523270 conn(0x561ecd97c100 msgr2=0x561ecda597f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:29.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.449+0000 7f736536eb80 1 -- 172.21.15.70:0/2048523270 shutdown_connections 2024-06-04T05:12:29.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.449+0000 7f736536eb80 1 -- 172.21.15.70:0/2048523270 wait complete. 2024-06-04T05:12:29.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.450+0000 7f736536eb80 1 -- start start 2024-06-04T05:12:29.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.450+0000 7f736536eb80 1 Processor -- start 2024-06-04T05:12:29.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.450+0000 7f736536eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 0x561ecda5e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.450+0000 7f736536eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4f290 0x561ecda5eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.450+0000 7f736536eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecdad1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.450+0000 7f736536eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f73541b1590 con 0x561ecda4f290 2024-06-04T05:12:29.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.450+0000 7f736536eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f73541b0e60 con 0x561ecda4e8e0 2024-06-04T05:12:29.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.450+0000 7f736536eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f73541ae020 con 0x561ecda52fe0 2024-06-04T05:12:29.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.450+0000 7f7359076700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecdad1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.450+0000 7f7358875700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 0x561ecda5e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.451+0000 7f7358875700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 0x561ecda5e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45448/0 (socket says 172.21.15.70:45448) 2024-06-04T05:12:29.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.451+0000 7f7358875700 1 -- 172.21.15.70:0/2021130876 learned_addr learned my addr 172.21.15.70:0/2021130876 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:29.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.451+0000 7f7353fff700 1 --2- 172.21.15.70:0/2021130876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4f290 0x561ecda5eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.451+0000 7f736536eb80 10 client.0 _ll_register_callbacks cb 0x561ecd9389a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:29.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.451+0000 7f7358875700 1 -- 172.21.15.70:0/2021130876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 msgr2=0x561ecdad1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:29.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.451+0000 7f7358875700 1 --2- 172.21.15.70:0/2021130876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecdad1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.451+0000 7f7358875700 1 -- 172.21.15.70:0/2021130876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4f290 msgr2=0x561ecda5eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:29.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.451+0000 7f7358875700 1 --2- 172.21.15.70:0/2021130876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4f290 0x561ecda5eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.451+0000 7f7358875700 1 -- 172.21.15.70:0/2021130876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ecdad24d0 con 0x561ecda4e8e0 2024-06-04T05:12:29.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.451+0000 7f7353fff700 1 --2- 172.21.15.70:0/2021130876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4f290 0x561ecda5eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:29.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.451+0000 7f7358875700 1 --2- 172.21.15.70:0/2021130876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 0x561ecda5e600 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7348005aa0 tx=0x7f7348005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee88ddd4fd17e562 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:29.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.451+0000 7f7333fff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:12:29.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.452+0000 7f7333fff700 1 -- 172.21.15.70:0/2021130876 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734804e060 con 0x561ecda4e8e0 2024-06-04T05:12:29.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.452+0000 7f736536eb80 1 -- 172.21.15.70:0/2021130876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7348034070 con 0x561ecda4e8e0 2024-06-04T05:12:29.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.452+0000 7f7333fff700 1 -- 172.21.15.70:0/2021130876 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f734800bdb0 con 0x561ecda4e8e0 2024-06-04T05:12:29.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.452+0000 7f7333fff700 1 -- 172.21.15.70:0/2021130876 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f7348023460 con 0x561ecda4e8e0 2024-06-04T05:12:29.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.452+0000 7f736536eb80 10 client.28337 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:29.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.452+0000 7f736536eb80 1 -- 172.21.15.70:0/2021130876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x561ecdad3700 con 0x561ecda4e8e0 2024-06-04T05:12:29.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.452+0000 7f7333fff700 1 -- 172.21.15.70:0/2021130876 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7348034070 con 0x561ecda4e8e0 2024-06-04T05:12:29.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.452+0000 7f7333fff700 1 -- 172.21.15.70:0/2021130876 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7f73480394e0 con 0x561ecda4e8e0 2024-06-04T05:12:29.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.452+0000 7f736536eb80 10 client.28337 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:29.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.452+0000 7f736536eb80 10 client.28337 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:12:29.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.452+0000 7f736536eb80 20 client.28337 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:29.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.452+0000 7f736536eb80 1 --2- 172.21.15.70:0/2021130876 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561ecdad3db0 0x561ecdad61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:29.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.452+0000 7f736536eb80 4 client.28337 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:12:29.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.452+0000 7f736536eb80 1 -- 172.21.15.70:0/2021130876 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561ecd9afc60 con 0x561ecdad3db0 2024-06-04T05:12:29.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.453+0000 7f7353fff700 1 --2- 172.21.15.70:0/2021130876 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561ecdad3db0 0x561ecdad61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:29.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.453+0000 7f7333fff700 10 client.28337 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:29.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.453+0000 7f7353fff700 1 --2- 172.21.15.70:0/2021130876 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561ecdad3db0 0x561ecdad61d0 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58241d9a7597ff2b server_cookie=bf3f139c0f47921a in_seq=0 out_seq=0 2024-06-04T05:12:29.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.454+0000 7f7333fff700 1 -- 172.21.15.70:0/2021130876 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561ecd9afc60 con 0x561ecdad3db0 2024-06-04T05:12:29.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.454+0000 7f7333fff700 10 client.28337 handle_command_reply: tid=0 2024-06-04T05:12:29.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.454+0000 7f736536eb80 1 client.28337 shutdown 2024-06-04T05:12:29.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.454+0000 7f736536eb80 1 -- 172.21.15.70:0/2021130876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 msgr2=0x561ecda5e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:29.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.454+0000 7f736536eb80 1 --2- 172.21.15.70:0/2021130876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 0x561ecda5e600 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7348005aa0 tx=0x7f7348005f10 comp rx=0 tx=0).stop 2024-06-04T05:12:29.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.454+0000 7f736536eb80 1 -- 172.21.15.70:0/2021130876 shutdown_connections 2024-06-04T05:12:29.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.454+0000 7f736536eb80 1 --2- 172.21.15.70:0/2021130876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561ecda52fe0 0x561ecdad1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.454+0000 7f736536eb80 1 --2- 172.21.15.70:0/2021130876 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561ecdad3db0 0x561ecdad61d0 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.454+0000 7f7333fff700 0 client.28337 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:29.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.454+0000 7f736536eb80 1 --2- 172.21.15.70:0/2021130876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ecda4e8e0 0x561ecda5e600 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.454+0000 7f736536eb80 1 --2- 172.21.15.70:0/2021130876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecda4f290 0x561ecda5eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.454+0000 7f736536eb80 1 -- 172.21.15.70:0/2021130876 >> 172.21.15.70:0/2021130876 conn(0x561ecd97c100 msgr2=0x561ecda51680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:29.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.454+0000 7f736536eb80 1 -- 172.21.15.70:0/2021130876 shutdown_connections 2024-06-04T05:12:29.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.454+0000 7f736536eb80 1 -- 172.21.15.70:0/2021130876 wait complete. 2024-06-04T05:12:29.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.454+0000 7f736536eb80 20 client.28337 trim_cache size 0 max 16384 2024-06-04T05:12:29.461 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:12:29.461 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:12:29.461 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "39b8a1c3-5482-495c-86fd-d8bfe78f3810", 2024-06-04T05:12:29.461 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:12:29.462 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:12:29.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.455+0000 7f735aad9700 1 -- 172.21.15.70:0/914005514 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73440b29e0 msgr2=0x7f73440b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:29.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.455+0000 7f735aad9700 1 --2- 172.21.15.70:0/914005514 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73440b29e0 0x7f73440b4ea0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7348003760 tx=0x7f7348005fb0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.455+0000 7f735aad9700 1 -- 172.21.15.70:0/914005514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73541039d0 msgr2=0x7f735419c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:29.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.455+0000 7f735aad9700 1 --2- 172.21.15.70:0/914005514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73541039d0 0x7f735419c400 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f734000eb30 tx=0x7f734000ee40 comp rx=0 tx=0).stop 2024-06-04T05:12:29.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.455+0000 7f735aad9700 1 -- 172.21.15.70:0/914005514 shutdown_connections 2024-06-04T05:12:29.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.455+0000 7f735aad9700 1 --2- 172.21.15.70:0/914005514 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73440b29e0 0x7f73440b4ea0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.455+0000 7f735aad9700 1 --2- 172.21.15.70:0/914005514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7354108730 0x7f73541a0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.455+0000 7f735aad9700 1 --2- 172.21.15.70:0/914005514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7354103010 0x7f735419bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.455+0000 7f735aad9700 1 --2- 172.21.15.70:0/914005514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73541039d0 0x7f735419c400 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:29.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.455+0000 7f735aad9700 1 -- 172.21.15.70:0/914005514 >> 172.21.15.70:0/914005514 conn(0x7f7354077780 msgr2=0x7f73540ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:29.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.456+0000 7f735aad9700 1 -- 172.21.15.70:0/914005514 shutdown_connections 2024-06-04T05:12:29.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:29.456+0000 7f735aad9700 1 -- 172.21.15.70:0/914005514 wait complete. 2024-06-04T05:12:29.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:29 smithi098 ceph-mon[35569]: pgmap v478: 97 pgs: 97 active+clean; 421 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 3.9 MiB/s wr, 239 op/s 2024-06-04T05:12:29.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:29 smithi070 ceph-mon[32631]: pgmap v478: 97 pgs: 97 active+clean; 421 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 3.9 MiB/s wr, 239 op/s 2024-06-04T05:12:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:29 smithi112 ceph-mon[36453]: pgmap v478: 97 pgs: 97 active+clean; 421 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 3.9 MiB/s wr, 239 op/s 2024-06-04T05:12:29.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 14293 11.61 MB/sec warmup 81 sec latency 72.689 ms 2024-06-04T05:12:30.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:12:30.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:12:30.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:12:30.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:12:30.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:12:30.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:12:30.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:12:30.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:12:30.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:12:30.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:12:30.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:12:30.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:12:30.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:12:30.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:12:30.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:12:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:12:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:12:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:12:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:12:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:12:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:12:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:12:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:12:30.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:12:30.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:12:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:12:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:12:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:12:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:12:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:12:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:12:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:12:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:12:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:12:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:12:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:12:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:30.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 14475 11.60 MB/sec warmup 82 sec latency 95.093 ms 2024-06-04T05:12:30.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.814+0000 7f638ef0e700 1 -- 172.21.15.70:0/1389993939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6388108520 msgr2=0x7f6388108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:30.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.814+0000 7f638ef0e700 1 --2- 172.21.15.70:0/1389993939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6388108520 0x7f6388108900 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f637c009990 tx=0x7f637c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:12:30.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.814+0000 7f638ef0e700 1 -- 172.21.15.70:0/1389993939 shutdown_connections 2024-06-04T05:12:30.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.814+0000 7f638ef0e700 1 --2- 172.21.15.70:0/1389993939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6388102e00 0x7f638810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:30.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.814+0000 7f638ef0e700 1 --2- 172.21.15.70:0/1389993939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6388108520 0x7f6388108900 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:30.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.814+0000 7f638ef0e700 1 --2- 172.21.15.70:0/1389993939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6388102440 0x7f63881028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:30.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.814+0000 7f638ef0e700 1 -- 172.21.15.70:0/1389993939 >> 172.21.15.70:0/1389993939 conn(0x7f63880793f0 msgr2=0x7f6388079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:30.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.814+0000 7f638ef0e700 1 -- 172.21.15.70:0/1389993939 shutdown_connections 2024-06-04T05:12:30.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.814+0000 7f638ef0e700 1 -- 172.21.15.70:0/1389993939 wait complete. 2024-06-04T05:12:30.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.815+0000 7f638ef0e700 1 Processor -- start 2024-06-04T05:12:30.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.815+0000 7f638ef0e700 1 -- start start 2024-06-04T05:12:30.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.815+0000 7f638ef0e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6388102440 0x7f638819f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:30.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.815+0000 7f638ef0e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6388102e00 0x7f638819fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:30.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.815+0000 7f638ef0e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6388108520 0x7f63881997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:30.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.815+0000 7f638ef0e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f63881124f0 con 0x7f6388102440 2024-06-04T05:12:30.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.815+0000 7f638ef0e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6388112370 con 0x7f6388102e00 2024-06-04T05:12:30.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.815+0000 7f638ef0e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6388112670 con 0x7f6388108520 2024-06-04T05:12:30.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.816+0000 7f638ccaa700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6388102440 0x7f638819f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:30.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.816+0000 7f6387fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6388102e00 0x7f638819fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:30.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.816+0000 7f6387fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6388102e00 0x7f638819fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45454/0 (socket says 172.21.15.70:45454) 2024-06-04T05:12:30.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.816+0000 7f6387fff700 1 -- 172.21.15.70:0/653627824 learned_addr learned my addr 172.21.15.70:0/653627824 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:30.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.816+0000 7f638d4ab700 1 --2- 172.21.15.70:0/653627824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6388108520 0x7f63881997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:30.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.816+0000 7f638d4ab700 1 -- 172.21.15.70:0/653627824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6388102e00 msgr2=0x7f638819fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:30.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.816+0000 7f638d4ab700 1 --2- 172.21.15.70:0/653627824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6388102e00 0x7f638819fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:30.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.816+0000 7f638d4ab700 1 -- 172.21.15.70:0/653627824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6388102440 msgr2=0x7f638819f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:30.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.816+0000 7f638d4ab700 1 --2- 172.21.15.70:0/653627824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6388102440 0x7f638819f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:30.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.816+0000 7f638d4ab700 1 -- 172.21.15.70:0/653627824 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6388199f90 con 0x7f6388108520 2024-06-04T05:12:30.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.816+0000 7f638ccaa700 1 --2- 172.21.15.70:0/653627824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6388102440 0x7f638819f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:30.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.816+0000 7f638d4ab700 1 --2- 172.21.15.70:0/653627824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6388108520 0x7f63881997b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f638000efc0 tx=0x7f638000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:30.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.816+0000 7f6385ffb700 1 -- 172.21.15.70:0/653627824 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63800092e0 con 0x7f6388108520 2024-06-04T05:12:30.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.817+0000 7f6385ffb700 1 -- 172.21.15.70:0/653627824 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6380004bb0 con 0x7f6388108520 2024-06-04T05:12:30.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.817+0000 7f6385ffb700 1 -- 172.21.15.70:0/653627824 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6380004020 con 0x7f6388108520 2024-06-04T05:12:30.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.817+0000 7f638ef0e700 1 -- 172.21.15.70:0/653627824 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f638819a220 con 0x7f6388108520 2024-06-04T05:12:30.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.817+0000 7f638ef0e700 1 -- 172.21.15.70:0/653627824 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63881092c0 con 0x7f6388108520 2024-06-04T05:12:30.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.818+0000 7f638ef0e700 1 -- 172.21.15.70:0/653627824 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6388103b60 con 0x7f6388108520 2024-06-04T05:12:30.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.820+0000 7f6385ffb700 1 -- 172.21.15.70:0/653627824 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f638001b030 con 0x7f6388108520 2024-06-04T05:12:30.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.821+0000 7f6385ffb700 1 --2- 172.21.15.70:0/653627824 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f63780b29f0 0x7f63780b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:30.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.821+0000 7f638ccaa700 1 --2- 172.21.15.70:0/653627824 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f63780b29f0 0x7f63780b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:30.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.821+0000 7f6385ffb700 1 -- 172.21.15.70:0/653627824 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f63800e2130 con 0x7f6388108520 2024-06-04T05:12:30.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.822+0000 7f638ccaa700 1 --2- 172.21.15.70:0/653627824 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f63780b29f0 0x7f63780b4eb0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f637c009350 tx=0x7f637c01a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:30.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:30.825+0000 7f6385ffb700 1 -- 172.21.15.70:0/653627824 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6380022020 con 0x7f6388108520 2024-06-04T05:12:31.262 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:12:31.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:31.258+0000 7f638ef0e700 1 -- 172.21.15.70:0/653627824 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f638810a6a0 con 0x7f6388108520 2024-06-04T05:12:31.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:31.259+0000 7f6385ffb700 1 -- 172.21.15.70:0/653627824 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11949 (secure 0 0 0) 0x7f6380010040 con 0x7f6388108520 2024-06-04T05:12:31.263 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:12:31.263 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":44,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:11:44.309654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:12:31.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:31.264+0000 7f638ef0e700 1 -- 172.21.15.70:0/653627824 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f63780b29f0 msgr2=0x7f63780b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:31.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:31.264+0000 7f638ef0e700 1 --2- 172.21.15.70:0/653627824 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f63780b29f0 0x7f63780b4eb0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f637c009350 tx=0x7f637c01a040 comp rx=0 tx=0).stop 2024-06-04T05:12:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:31.264+0000 7f638ef0e700 1 -- 172.21.15.70:0/653627824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6388108520 msgr2=0x7f63881997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:31.264+0000 7f638ef0e700 1 --2- 172.21.15.70:0/653627824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6388108520 0x7f63881997b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f638000efc0 tx=0x7f638000c3a0 comp rx=0 tx=0).stop 2024-06-04T05:12:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:31.265+0000 7f638ef0e700 1 -- 172.21.15.70:0/653627824 shutdown_connections 2024-06-04T05:12:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:31.265+0000 7f638ef0e700 1 --2- 172.21.15.70:0/653627824 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f63780b29f0 0x7f63780b4eb0 secure :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f637c009350 tx=0x7f637c01a040 comp rx=0 tx=0).stop 2024-06-04T05:12:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:31.265+0000 7f638ef0e700 1 --2- 172.21.15.70:0/653627824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6388108520 0x7f63881997b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:31.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:31.265+0000 7f638ef0e700 1 --2- 172.21.15.70:0/653627824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6388102440 0x7f638819f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:31.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:31.265+0000 7f638ef0e700 1 --2- 172.21.15.70:0/653627824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6388102e00 0x7f638819fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:31.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:31.265+0000 7f638ef0e700 1 -- 172.21.15.70:0/653627824 >> 172.21.15.70:0/653627824 conn(0x7f63880793f0 msgr2=0x7f63880ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:31.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:31.265+0000 7f638ef0e700 1 -- 172.21.15.70:0/653627824 shutdown_connections 2024-06-04T05:12:31.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:31.265+0000 7f638ef0e700 1 -- 172.21.15.70:0/653627824 wait complete. 2024-06-04T05:12:31.268 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 44 2024-06-04T05:12:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:31 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:31 smithi098 ceph-mon[35569]: pgmap v479: 97 pgs: 97 active+clean; 421 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 3.9 MiB/s wr, 239 op/s 2024-06-04T05:12:31.587 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:31 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:31.587 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:31 smithi112 ceph-mon[36453]: pgmap v479: 97 pgs: 97 active+clean; 421 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 3.9 MiB/s wr, 239 op/s 2024-06-04T05:12:31.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:31 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:12:31.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:31 smithi070 ceph-mon[32631]: pgmap v479: 97 pgs: 97 active+clean; 421 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 3.9 MiB/s wr, 239 op/s 2024-06-04T05:12:31.799 INFO:tasks.workunit.client.0.smithi070.stdout: 10 14659 11.59 MB/sec warmup 83 sec latency 62.409 ms 2024-06-04T05:12:32.209 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 152.6 --await-for 45.0' 2024-06-04T05:12:32.209 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 152.6 --await-for 45.0 2024-06-04T05:12:32.277 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:12:32.436 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:32 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/653627824' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:12:32.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:32 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/653627824' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:12:32.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:32 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/653627824' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:12:32.800 INFO:tasks.workunit.client.0.smithi070.stdout: 10 14836 11.56 MB/sec warmup 84 sec latency 60.768 ms 2024-06-04T05:12:33.578 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:33 smithi070 ceph-mon[32631]: pgmap v480: 97 pgs: 97 active+clean; 424 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.8 MiB/s wr, 402 op/s 2024-06-04T05:12:33.585 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:12:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:33 smithi112 ceph-mon[36453]: pgmap v480: 97 pgs: 97 active+clean; 424 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.8 MiB/s wr, 402 op/s 2024-06-04T05:12:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:33 smithi098 ceph-mon[35569]: pgmap v480: 97 pgs: 97 active+clean; 424 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.8 MiB/s wr, 402 op/s 2024-06-04T05:12:33.800 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15041 11.56 MB/sec warmup 85 sec latency 62.256 ms 2024-06-04T05:12:34.800 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15214 11.50 MB/sec warmup 86 sec latency 77.719 ms 2024-06-04T05:12:34.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.896+0000 7f0150b5b700 1 -- 172.21.15.70:0/2399812063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014c102810 msgr2=0x7f014c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:34.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.896+0000 7f0150b5b700 1 --2- 172.21.15.70:0/2399812063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014c102810 0x7f014c106810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0140009990 tx=0x7f0140009ca0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.896+0000 7f0150b5b700 1 -- 172.21.15.70:0/2399812063 shutdown_connections 2024-06-04T05:12:34.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.896+0000 7f0150b5b700 1 --2- 172.21.15.70:0/2399812063 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f014c106ef0 0x7f014c10eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.896+0000 7f0150b5b700 1 --2- 172.21.15.70:0/2399812063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014c101e60 0x7f014c102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.896+0000 7f0150b5b700 1 --2- 172.21.15.70:0/2399812063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014c102810 0x7f014c106810 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.896+0000 7f0150b5b700 1 -- 172.21.15.70:0/2399812063 >> 172.21.15.70:0/2399812063 conn(0x7f014c078ab0 msgr2=0x7f014c0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:34.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.896+0000 7f0150b5b700 1 -- 172.21.15.70:0/2399812063 shutdown_connections 2024-06-04T05:12:34.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.896+0000 7f0150b5b700 1 -- 172.21.15.70:0/2399812063 wait complete. 2024-06-04T05:12:34.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.897+0000 7f0150b5b700 1 Processor -- start 2024-06-04T05:12:34.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.897+0000 7f0150b5b700 1 -- start start 2024-06-04T05:12:34.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f0150b5b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014c101e60 0x7f014c19bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f0150b5b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f014c102810 0x7f014c19c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f0150b5b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014c106ef0 0x7f014c1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f0150b5b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f014c111c20 con 0x7f014c101e60 2024-06-04T05:12:34.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f0150b5b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f014c111aa0 con 0x7f014c106ef0 2024-06-04T05:12:34.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f0150b5b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f014c111da0 con 0x7f014c102810 2024-06-04T05:12:34.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f014a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014c101e60 0x7f014c19bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f014a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014c101e60 0x7f014c19bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54694/0 (socket says 172.21.15.70:54694) 2024-06-04T05:12:34.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f014a59c700 1 -- 172.21.15.70:0/1201370508 learned_addr learned my addr 172.21.15.70:0/1201370508 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:34.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f014ad9d700 1 --2- 172.21.15.70:0/1201370508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014c106ef0 0x7f014c1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f0149d9b700 1 --2- 172.21.15.70:0/1201370508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f014c102810 0x7f014c19c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f014a59c700 1 -- 172.21.15.70:0/1201370508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f014c102810 msgr2=0x7f014c19c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:34.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f014a59c700 1 --2- 172.21.15.70:0/1201370508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f014c102810 0x7f014c19c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f014a59c700 1 -- 172.21.15.70:0/1201370508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014c106ef0 msgr2=0x7f014c1a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:34.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f014a59c700 1 --2- 172.21.15.70:0/1201370508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014c106ef0 0x7f014c1a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f014a59c700 1 -- 172.21.15.70:0/1201370508 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014c1a0da0 con 0x7f014c101e60 2024-06-04T05:12:34.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f014ad9d700 1 --2- 172.21.15.70:0/1201370508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014c106ef0 0x7f014c1a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:34.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.898+0000 7f014a59c700 1 --2- 172.21.15.70:0/1201370508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014c101e60 0x7f014c19bd90 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f013c00e9a0 tx=0x7f013c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:34.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.899+0000 7f013b7fe700 1 -- 172.21.15.70:0/1201370508 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c00ca10 con 0x7f014c101e60 2024-06-04T05:12:34.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.899+0000 7f013b7fe700 1 -- 172.21.15.70:0/1201370508 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f013c00cbb0 con 0x7f014c101e60 2024-06-04T05:12:34.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.899+0000 7f0150b5b700 1 -- 172.21.15.70:0/1201370508 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f014c1a1090 con 0x7f014c101e60 2024-06-04T05:12:34.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.899+0000 7f013b7fe700 1 -- 172.21.15.70:0/1201370508 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c018680 con 0x7f014c101e60 2024-06-04T05:12:34.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.899+0000 7f0150b5b700 1 -- 172.21.15.70:0/1201370508 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f014c1a8910 con 0x7f014c101e60 2024-06-04T05:12:34.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.902+0000 7f013b7fe700 1 -- 172.21.15.70:0/1201370508 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f013c010430 con 0x7f014c101e60 2024-06-04T05:12:34.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.903+0000 7f013b7fe700 1 --2- 172.21.15.70:0/1201370508 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01340b2a70 0x7f01340b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.903+0000 7f0149d9b700 1 --2- 172.21.15.70:0/1201370508 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01340b2a70 0x7f01340b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.903+0000 7f013b7fe700 1 -- 172.21.15.70:0/1201370508 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f013c0e2740 con 0x7f014c101e60 2024-06-04T05:12:34.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.904+0000 7f0149d9b700 1 --2- 172.21.15.70:0/1201370508 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01340b2a70 0x7f01340b4f30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f014000b2e0 tx=0x7f0140005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:34.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.904+0000 7f015b3f0b80 1 Processor -- start 2024-06-04T05:12:34.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f015b3f0b80 1 -- start start 2024-06-04T05:12:34.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f015b3f0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55734860 0x557d55734c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f015b3f0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55735210 0x557d55738940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f015b3f0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55738f60 0x557d557393a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f015b3f0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557d55693c90 con 0x557d55735210 2024-06-04T05:12:34.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f015b3f0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557d55683830 con 0x557d55734860 2024-06-04T05:12:34.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f015b3f0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557d55687870 con 0x557d55738f60 2024-06-04T05:12:34.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f014a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55734860 0x557d55734c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f014ad9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55738f60 0x557d557393a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f014ad9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55738f60 0x557d557393a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:52634/0 (socket says 172.21.15.70:52634) 2024-06-04T05:12:34.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f014a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55734860 0x557d55734c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45486/0 (socket says 172.21.15.70:45486) 2024-06-04T05:12:34.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f014ad9d700 1 -- 172.21.15.70:0/2321933829 learned_addr learned my addr 172.21.15.70:0/2321933829 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:34.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f0149d9b700 1 --2- 172.21.15.70:0/2321933829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55735210 0x557d55738940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f014ad9d700 1 -- 172.21.15.70:0/2321933829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55734860 msgr2=0x557d55734c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:34.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f014ad9d700 1 --2- 172.21.15.70:0/2321933829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55734860 0x557d55734c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f014ad9d700 1 -- 172.21.15.70:0/2321933829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55735210 msgr2=0x557d55738940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:34.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f014ad9d700 1 --2- 172.21.15.70:0/2321933829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55735210 0x557d55738940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.905+0000 7f014ad9d700 1 -- 172.21.15.70:0/2321933829 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d555d38d0 con 0x557d55738f60 2024-06-04T05:12:34.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.906+0000 7f014ad9d700 1 --2- 172.21.15.70:0/2321933829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55738f60 0x557d557393a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0144003eb0 tx=0x7f0144009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a4ce7ca34d965bb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:34.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.906+0000 7f0138ff9700 1 -- 172.21.15.70:0/2321933829 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0144019020 con 0x557d55738f60 2024-06-04T05:12:34.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.906+0000 7f0138ff9700 1 -- 172.21.15.70:0/2321933829 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f01440092d0 con 0x557d55738f60 2024-06-04T05:12:34.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.906+0000 7f0138ff9700 1 -- 172.21.15.70:0/2321933829 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014400e820 con 0x557d55738f60 2024-06-04T05:12:34.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.907+0000 7f015b3f0b80 1 -- 172.21.15.70:0/2321933829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55738f60 msgr2=0x557d557393a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:34.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.907+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/2321933829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55738f60 0x557d557393a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0144003eb0 tx=0x7f0144009920 comp rx=0 tx=0).stop 2024-06-04T05:12:34.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.907+0000 7f015b3f0b80 1 -- 172.21.15.70:0/2321933829 shutdown_connections 2024-06-04T05:12:34.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.907+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/2321933829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55738f60 0x557d557393a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.907+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/2321933829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55734860 0x557d55734c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.907+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/2321933829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55735210 0x557d55738940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.907+0000 7f015b3f0b80 1 -- 172.21.15.70:0/2321933829 >> 172.21.15.70:0/2321933829 conn(0x557d55662080 msgr2=0x557d5572ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:34.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.907+0000 7f015b3f0b80 1 -- 172.21.15.70:0/2321933829 shutdown_connections 2024-06-04T05:12:34.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.907+0000 7f015b3f0b80 1 -- 172.21.15.70:0/2321933829 wait complete. 2024-06-04T05:12:34.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.908+0000 7f015b3f0b80 1 -- start start 2024-06-04T05:12:34.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.908+0000 7f015b3f0b80 1 Processor -- start 2024-06-04T05:12:34.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.908+0000 7f015b3f0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55734860 0x557d557402d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.908+0000 7f015b3f0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55735210 0x557d5573d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.908+0000 7f015b3f0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55738f60 0x557d5573db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.908+0000 7f015b3f0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557d55589fa0 con 0x557d55734860 2024-06-04T05:12:34.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.908+0000 7f015b3f0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557d556257c0 con 0x557d55735210 2024-06-04T05:12:34.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.908+0000 7f015b3f0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557d557411c0 con 0x557d55738f60 2024-06-04T05:12:34.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.908+0000 7f015b3f0b80 10 client.0 _ll_register_callbacks cb 0x557d5561e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:34.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.908+0000 7f0149d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55735210 0x557d5573d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.908+0000 7f0149d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55735210 0x557d5573d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45502/0 (socket says 172.21.15.70:45502) 2024-06-04T05:12:34.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.908+0000 7f0149d9b700 1 -- 172.21.15.70:0/3051657952 learned_addr learned my addr 172.21.15.70:0/3051657952 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:34.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.908+0000 7f014ad9d700 1 --2- 172.21.15.70:0/3051657952 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55738f60 0x557d5573db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.909+0000 7f014a59c700 1 --2- 172.21.15.70:0/3051657952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55734860 0x557d557402d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.909+0000 7f014ad9d700 1 -- 172.21.15.70:0/3051657952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55735210 msgr2=0x557d5573d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:34.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.909+0000 7f014ad9d700 1 --2- 172.21.15.70:0/3051657952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55735210 0x557d5573d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.909+0000 7f014ad9d700 1 -- 172.21.15.70:0/3051657952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55734860 msgr2=0x557d557402d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:34.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.909+0000 7f014ad9d700 1 --2- 172.21.15.70:0/3051657952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55734860 0x557d557402d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.909+0000 7f014ad9d700 1 -- 172.21.15.70:0/3051657952 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d55622690 con 0x557d55738f60 2024-06-04T05:12:34.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.909+0000 7f014ad9d700 1 --2- 172.21.15.70:0/3051657952 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55738f60 0x557d5573db10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0144003eb0 tx=0x7f0144007730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82af5eb8842c1c6d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:34.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.909+0000 7f0131ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:12:34.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.909+0000 7f0131ffb700 1 -- 172.21.15.70:0/3051657952 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0144019070 con 0x557d55738f60 2024-06-04T05:12:34.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.909+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3051657952 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01440092d0 con 0x557d55738f60 2024-06-04T05:12:34.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.909+0000 7f0131ffb700 1 -- 172.21.15.70:0/3051657952 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f014400e4f0 con 0x557d55738f60 2024-06-04T05:12:34.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.910+0000 7f0131ffb700 1 -- 172.21.15.70:0/3051657952 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f0144007d60 con 0x557d55738f60 2024-06-04T05:12:34.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.910+0000 7f015b3f0b80 10 client.28468 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:34.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.910+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3051657952 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x557d55753e10 con 0x557d55738f60 2024-06-04T05:12:34.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.910+0000 7f0131ffb700 1 -- 172.21.15.70:0/3051657952 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f01440092d0 con 0x557d55738f60 2024-06-04T05:12:34.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.910+0000 7f0131ffb700 1 -- 172.21.15.70:0/3051657952 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7f0144015750 con 0x557d55738f60 2024-06-04T05:12:34.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.910+0000 7f015b3f0b80 10 client.28468 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:34.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.910+0000 7f015b3f0b80 10 client.28468 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:12:34.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.910+0000 7f015b3f0b80 20 client.28468 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:34.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.910+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/3051657952 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557d557566f0 0x557d55758b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.910+0000 7f015b3f0b80 4 client.28468 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:12:34.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.910+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3051657952 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d557009e0 con 0x557d557566f0 2024-06-04T05:12:34.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.911+0000 7f014a59c700 1 --2- 172.21.15.70:0/3051657952 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557d557566f0 0x557d55758b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.911+0000 7f0131ffb700 10 client.28468 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:34.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.911+0000 7f014a59c700 1 --2- 172.21.15.70:0/3051657952 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557d557566f0 0x557d55758b10 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3644027e7cf02f07 server_cookie=85e4b630bd59d067 in_seq=0 out_seq=0 2024-06-04T05:12:34.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.913+0000 7f0131ffb700 1 -- 172.21.15.70:0/3051657952 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557d557009e0 con 0x557d557566f0 2024-06-04T05:12:34.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.913+0000 7f0131ffb700 10 client.28468 handle_command_reply: tid=0 2024-06-04T05:12:34.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.913+0000 7f015b3f0b80 1 client.28468 shutdown 2024-06-04T05:12:34.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.913+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3051657952 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55738f60 msgr2=0x557d5573db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:34.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.913+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/3051657952 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55738f60 0x557d5573db10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0144003eb0 tx=0x7f0144007730 comp rx=0 tx=0).stop 2024-06-04T05:12:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.913+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3051657952 shutdown_connections 2024-06-04T05:12:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.913+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/3051657952 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55738f60 0x557d5573db10 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.913+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/3051657952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55734860 0x557d557402d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.913+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/3051657952 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557d557566f0 0x557d55758b10 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.913+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/3051657952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55735210 0x557d5573d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.913+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3051657952 >> 172.21.15.70:0/3051657952 conn(0x557d55662080 msgr2=0x557d5573a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.914+0000 7f0131ffb700 0 client.28468 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.914+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3051657952 shutdown_connections 2024-06-04T05:12:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.914+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3051657952 wait complete. 2024-06-04T05:12:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.914+0000 7f015b3f0b80 20 client.28468 trim_cache size 0 max 16384 2024-06-04T05:12:34.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.948+0000 7f015b3f0b80 1 Processor -- start 2024-06-04T05:12:34.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.948+0000 7f015b3f0b80 1 -- start start 2024-06-04T05:12:34.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.948+0000 7f015b3f0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55662080 0x557d5573c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.948+0000 7f015b3f0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55734860 0x557d5573fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.948+0000 7f015b3f0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55735210 0x557d5575e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.949+0000 7f015b3f0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f014c1abda0 con 0x557d55734860 2024-06-04T05:12:34.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.949+0000 7f015b3f0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f014c1ada60 con 0x557d55735210 2024-06-04T05:12:34.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.949+0000 7f015b3f0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f014c1ac470 con 0x557d55662080 2024-06-04T05:12:34.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.949+0000 7f014ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55735210 0x557d5575e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.949+0000 7f014ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55735210 0x557d5575e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45518/0 (socket says 172.21.15.70:45518) 2024-06-04T05:12:34.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.949+0000 7f0149d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55734860 0x557d5573fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.949+0000 7f014ad9d700 1 -- 172.21.15.70:0/3684900297 learned_addr learned my addr 172.21.15.70:0/3684900297 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:34.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.949+0000 7f014a59c700 1 --2- 172.21.15.70:0/3684900297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55662080 0x557d5573c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.949+0000 7f014ad9d700 1 -- 172.21.15.70:0/3684900297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55662080 msgr2=0x557d5573c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:34.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.949+0000 7f014ad9d700 1 --2- 172.21.15.70:0/3684900297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55662080 0x557d5573c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.949+0000 7f014ad9d700 1 -- 172.21.15.70:0/3684900297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55734860 msgr2=0x557d5573fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:34.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.949+0000 7f014ad9d700 1 --2- 172.21.15.70:0/3684900297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55734860 0x557d5573fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.949+0000 7f014ad9d700 1 -- 172.21.15.70:0/3684900297 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d5572f940 con 0x557d55735210 2024-06-04T05:12:34.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.949+0000 7f014ad9d700 1 --2- 172.21.15.70:0/3684900297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55735210 0x557d5575e3e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0144007cf0 tx=0x7f0144009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54b3772dd1de5ac7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:34.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.949+0000 7f0133fff700 1 -- 172.21.15.70:0/3684900297 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0144034070 con 0x557d55735210 2024-06-04T05:12:34.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.950+0000 7f0133fff700 1 -- 172.21.15.70:0/3684900297 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0144022030 con 0x557d55735210 2024-06-04T05:12:34.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.950+0000 7f0133fff700 1 -- 172.21.15.70:0/3684900297 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0144019040 con 0x557d55735210 2024-06-04T05:12:34.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.950+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3684900297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55735210 msgr2=0x557d5575e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:34.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.950+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/3684900297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55735210 0x557d5575e3e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0144007cf0 tx=0x7f0144009920 comp rx=0 tx=0).stop 2024-06-04T05:12:34.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.951+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3684900297 shutdown_connections 2024-06-04T05:12:34.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.951+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/3684900297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55735210 0x557d5575e3e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.951+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/3684900297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55662080 0x557d5573c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.951+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/3684900297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55734860 0x557d5573fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.951+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3684900297 >> 172.21.15.70:0/3684900297 conn(0x557d5572f4f0 msgr2=0x557d55733f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:34.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.951+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3684900297 shutdown_connections 2024-06-04T05:12:34.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.951+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3684900297 wait complete. 2024-06-04T05:12:34.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.951+0000 7f015b3f0b80 1 -- start start 2024-06-04T05:12:34.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.951+0000 7f015b3f0b80 1 Processor -- start 2024-06-04T05:12:34.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.951+0000 7f015b3f0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55662080 0x557d5574f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.951+0000 7f015b3f0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55734860 0x557d55735f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.951+0000 7f015b3f0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55735210 0x557d55736460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.951+0000 7f015b3f0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f014c1ad2d0 con 0x557d55735210 2024-06-04T05:12:34.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.951+0000 7f015b3f0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f014c1acba0 con 0x557d55734860 2024-06-04T05:12:34.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.951+0000 7f015b3f0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f014c1a9d90 con 0x557d55662080 2024-06-04T05:12:34.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.951+0000 7f015b3f0b80 10 client.0 _ll_register_callbacks cb 0x557d5561e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:34.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.952+0000 7f0149d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55734860 0x557d55735f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.952+0000 7f0149d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55734860 0x557d55735f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45530/0 (socket says 172.21.15.70:45530) 2024-06-04T05:12:34.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.952+0000 7f0149d9b700 1 -- 172.21.15.70:0/3945311436 learned_addr learned my addr 172.21.15.70:0/3945311436 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:34.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.952+0000 7f014a59c700 1 --2- 172.21.15.70:0/3945311436 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55662080 0x557d5574f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.952+0000 7f014ad9d700 1 --2- 172.21.15.70:0/3945311436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55735210 0x557d55736460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.952+0000 7f0149d9b700 1 -- 172.21.15.70:0/3945311436 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55662080 msgr2=0x557d5574f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:34.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.952+0000 7f0149d9b700 1 --2- 172.21.15.70:0/3945311436 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55662080 0x557d5574f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.952+0000 7f0149d9b700 1 -- 172.21.15.70:0/3945311436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55735210 msgr2=0x557d55736460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:34.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.952+0000 7f0149d9b700 1 --2- 172.21.15.70:0/3945311436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55735210 0x557d55736460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.952+0000 7f0149d9b700 1 -- 172.21.15.70:0/3945311436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d557369d0 con 0x557d55734860 2024-06-04T05:12:34.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.952+0000 7f014a59c700 1 --2- 172.21.15.70:0/3945311436 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55662080 0x557d5574f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:34.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.952+0000 7f0149d9b700 1 --2- 172.21.15.70:0/3945311436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55734860 0x557d55735f20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0140009990 tx=0x7f0140009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1087be813db966e5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:34.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.953+0000 7f0131ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:12:34.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.953+0000 7f0131ffb700 1 -- 172.21.15.70:0/3945311436 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0140025680 con 0x557d55734860 2024-06-04T05:12:34.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.953+0000 7f0131ffb700 1 -- 172.21.15.70:0/3945311436 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f0140025820 con 0x557d55734860 2024-06-04T05:12:34.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.953+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3945311436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0144022030 con 0x557d55734860 2024-06-04T05:12:34.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.953+0000 7f0131ffb700 1 -- 172.21.15.70:0/3945311436 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f0140025a20 con 0x557d55734860 2024-06-04T05:12:34.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.953+0000 7f015b3f0b80 10 client.28361 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:34.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.953+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3945311436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x557d557c1070 con 0x557d55734860 2024-06-04T05:12:34.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.953+0000 7f0131ffb700 1 -- 172.21.15.70:0/3945311436 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0140025bc0 con 0x557d55734860 2024-06-04T05:12:34.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.953+0000 7f0131ffb700 1 -- 172.21.15.70:0/3945311436 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7f0140021070 con 0x557d55734860 2024-06-04T05:12:34.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.954+0000 7f015b3f0b80 10 client.28361 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:34.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.954+0000 7f015b3f0b80 10 client.28361 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:12:34.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.954+0000 7f015b3f0b80 20 client.28361 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:34.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.954+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/3945311436 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557d55738f60 0x557d557c38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:34.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.954+0000 7f015b3f0b80 4 client.28361 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:12:34.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.954+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3945311436 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f0144022030 con 0x557d55738f60 2024-06-04T05:12:34.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.954+0000 7f014a59c700 1 --2- 172.21.15.70:0/3945311436 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557d55738f60 0x557d557c38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:34.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.955+0000 7f014a59c700 1 --2- 172.21.15.70:0/3945311436 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557d55738f60 0x557d557c38e0 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9255aac42359888 server_cookie=33ad34b5920ad545 in_seq=0 out_seq=0 2024-06-04T05:12:34.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.955+0000 7f0131ffb700 10 client.28361 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:34.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.956+0000 7f0131ffb700 1 -- 172.21.15.70:0/3945311436 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f0144022030 con 0x557d55738f60 2024-06-04T05:12:34.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.956+0000 7f0131ffb700 10 client.28361 handle_command_reply: tid=0 2024-06-04T05:12:34.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.956+0000 7f015b3f0b80 1 client.28361 shutdown 2024-06-04T05:12:34.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.956+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3945311436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55734860 msgr2=0x557d55735f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:34.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.956+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/3945311436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55734860 0x557d55735f20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0140009990 tx=0x7f0140009310 comp rx=0 tx=0).stop 2024-06-04T05:12:34.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.957+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3945311436 shutdown_connections 2024-06-04T05:12:34.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.957+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/3945311436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d55735210 0x557d55736460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.957+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/3945311436 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557d55662080 0x557d5574f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.957+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/3945311436 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557d55738f60 0x557d557c38e0 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.957+0000 7f015b3f0b80 1 --2- 172.21.15.70:0/3945311436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557d55734860 0x557d55735f20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.957+0000 7f0131ffb700 0 client.28361 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:34.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.957+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3945311436 >> 172.21.15.70:0/3945311436 conn(0x557d5572f4f0 msgr2=0x557d5573cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:34.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.957+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3945311436 shutdown_connections 2024-06-04T05:12:34.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.958+0000 7f015b3f0b80 1 -- 172.21.15.70:0/3945311436 wait complete. 2024-06-04T05:12:34.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.958+0000 7f015b3f0b80 20 client.28361 trim_cache size 0 max 16384 2024-06-04T05:12:34.961 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:12:34.961 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:12:34.961 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:12:34.961 INFO:teuthology.orchestra.run.smithi070.stdout: "39b8a1c3-5482-495c-86fd-d8bfe78f3810": { 2024-06-04T05:12:34.962 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:12:34.962 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "39b8a1c3-5482-495c-86fd-d8bfe78f3810", 2024-06-04T05:12:34.962 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:12:34.962 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:12:34.962 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:12:34.962 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:12:34.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.959+0000 7f0150b5b700 1 -- 172.21.15.70:0/1201370508 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01340b2a70 msgr2=0x7f01340b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:34.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.959+0000 7f0150b5b700 1 --2- 172.21.15.70:0/1201370508 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01340b2a70 0x7f01340b4f30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f014000b2e0 tx=0x7f0140005fb0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.959+0000 7f0150b5b700 1 -- 172.21.15.70:0/1201370508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014c101e60 msgr2=0x7f014c19bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:34.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.959+0000 7f0150b5b700 1 --2- 172.21.15.70:0/1201370508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014c101e60 0x7f014c19bd90 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f013c00e9a0 tx=0x7f013c00ecb0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.959+0000 7f0150b5b700 1 -- 172.21.15.70:0/1201370508 shutdown_connections 2024-06-04T05:12:34.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.959+0000 7f0150b5b700 1 --2- 172.21.15.70:0/1201370508 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01340b2a70 0x7f01340b4f30 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.959+0000 7f0150b5b700 1 --2- 172.21.15.70:0/1201370508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014c106ef0 0x7f014c1a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.959+0000 7f0150b5b700 1 --2- 172.21.15.70:0/1201370508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014c101e60 0x7f014c19bd90 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.959+0000 7f0150b5b700 1 --2- 172.21.15.70:0/1201370508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f014c102810 0x7f014c19c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:34.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.959+0000 7f0150b5b700 1 -- 172.21.15.70:0/1201370508 >> 172.21.15.70:0/1201370508 conn(0x7f014c078ab0 msgr2=0x7f014c104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:34.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.959+0000 7f0150b5b700 1 -- 172.21.15.70:0/1201370508 shutdown_connections 2024-06-04T05:12:34.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:34.959+0000 7f0150b5b700 1 -- 172.21.15.70:0/1201370508 wait complete. 2024-06-04T05:12:35.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:35 smithi070 ceph-mon[32631]: pgmap v481: 97 pgs: 97 active+clean; 424 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 286 op/s 2024-06-04T05:12:35.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.464+0000 7fbb3d912640 1 -- 172.21.15.70:0/3065498444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb38104580 msgr2=0x7fbb3810ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:35.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.464+0000 7fbb3d912640 1 --2- 172.21.15.70:0/3065498444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb38104580 0x7fbb3810ca60 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009900 tx=0x7fbb2c02a480 comp rx=0 tx=0).stop 2024-06-04T05:12:35.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.464+0000 7fbb3d912640 1 -- 172.21.15.70:0/3065498444 shutdown_connections 2024-06-04T05:12:35.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.464+0000 7fbb3d912640 1 --2- 172.21.15.70:0/3065498444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb3810d030 0x7fbb3810f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.464+0000 7fbb3d912640 1 --2- 172.21.15.70:0/3065498444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb38104580 0x7fbb3810ca60 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.464+0000 7fbb3d912640 1 --2- 172.21.15.70:0/3065498444 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb38103c60 0x7fbb38104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.464+0000 7fbb3d912640 1 -- 172.21.15.70:0/3065498444 >> 172.21.15.70:0/3065498444 conn(0x7fbb380fd420 msgr2=0x7fbb380ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:35.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.464+0000 7fbb3d912640 1 -- 172.21.15.70:0/3065498444 shutdown_connections 2024-06-04T05:12:35.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.464+0000 7fbb3d912640 1 -- 172.21.15.70:0/3065498444 wait complete. 2024-06-04T05:12:35.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.465+0000 7fbb3d912640 1 Processor -- start 2024-06-04T05:12:35.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.465+0000 7fbb3d912640 1 -- start start 2024-06-04T05:12:35.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.465+0000 7fbb3d912640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb38103c60 0x7fbb381018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.465+0000 7fbb3d912640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb38104580 0x7fbb38101df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.465+0000 7fbb3d912640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb3810d030 0x7fbb38102330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.465+0000 7fbb3d912640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbb38112b70 con 0x7fbb3810d030 2024-06-04T05:12:35.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.465+0000 7fbb3d912640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381129f0 con 0x7fbb38104580 2024-06-04T05:12:35.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.465+0000 7fbb3d912640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbb38112cf0 con 0x7fbb38103c60 2024-06-04T05:12:35.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.466+0000 7fbb367fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb38104580 0x7fbb38101df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.466+0000 7fbb36ffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb38103c60 0x7fbb381018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.466+0000 7fbb367fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb38104580 0x7fbb38101df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45538/0 (socket says 172.21.15.70:45538) 2024-06-04T05:12:35.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.466+0000 7fbb367fc640 1 -- 172.21.15.70:0/2004377678 learned_addr learned my addr 172.21.15.70:0/2004377678 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:35.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.466+0000 7fbb377fe640 1 --2- 172.21.15.70:0/2004377678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb3810d030 0x7fbb38102330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.466+0000 7fbb36ffd640 1 -- 172.21.15.70:0/2004377678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb38104580 msgr2=0x7fbb38101df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:35.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.466+0000 7fbb36ffd640 1 --2- 172.21.15.70:0/2004377678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb38104580 0x7fbb38101df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.466+0000 7fbb36ffd640 1 -- 172.21.15.70:0/2004377678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb3810d030 msgr2=0x7fbb38102330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:35.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.466+0000 7fbb36ffd640 1 --2- 172.21.15.70:0/2004377678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb3810d030 0x7fbb38102330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.466+0000 7fbb36ffd640 1 -- 172.21.15.70:0/2004377678 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb381aba70 con 0x7fbb38103c60 2024-06-04T05:12:35.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.466+0000 7fbb367fc640 1 --2- 172.21.15.70:0/2004377678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb38104580 0x7fbb38101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:35.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.466+0000 7fbb377fe640 1 --2- 172.21.15.70:0/2004377678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb3810d030 0x7fbb38102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:35.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.466+0000 7fbb36ffd640 1 --2- 172.21.15.70:0/2004377678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb38103c60 0x7fbb381018b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800e9f0 tx=0x7fbb2800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:35.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.466+0000 7fbb3c910640 1 -- 172.21.15.70:0/2004377678 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2800ccc0 con 0x7fbb38103c60 2024-06-04T05:12:35.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.466+0000 7fbb3d912640 1 -- 172.21.15.70:0/2004377678 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb381abca0 con 0x7fbb38103c60 2024-06-04T05:12:35.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.466+0000 7fbb3c910640 1 -- 172.21.15.70:0/2004377678 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbb280108c0 con 0x7fbb38103c60 2024-06-04T05:12:35.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.467+0000 7fbb3c910640 1 -- 172.21.15.70:0/2004377678 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb280187c0 con 0x7fbb38103c60 2024-06-04T05:12:35.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.467+0000 7fbb3d912640 1 -- 172.21.15.70:0/2004377678 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb381ac1c0 con 0x7fbb38103c60 2024-06-04T05:12:35.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.469+0000 7fbb3c910640 1 -- 172.21.15.70:0/2004377678 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fbb28010430 con 0x7fbb38103c60 2024-06-04T05:12:35.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.469+0000 7fbb40620740 1 Processor -- start 2024-06-04T05:12:35.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.469+0000 7fbb40620740 1 -- start start 2024-06-04T05:12:35.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb40620740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901d820 0x55a9e901dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb40620740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e901e140 0x55a9e90284c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb40620740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e9028a00 0x55a9e902adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb40620740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a9e8f8fc30 con 0x55a9e901d820 2024-06-04T05:12:35.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb40620740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a9e8f7f7d0 con 0x55a9e901e140 2024-06-04T05:12:35.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb40620740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a9e8f83810 con 0x55a9e9028a00 2024-06-04T05:12:35.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb377fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901d820 0x55a9e901dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb377fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901d820 0x55a9e901dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54762/0 (socket says 172.21.15.70:54762) 2024-06-04T05:12:35.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb377fe640 1 -- 172.21.15.70:0/251694798 learned_addr learned my addr 172.21.15.70:0/251694798 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:35.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb36ffd640 1 --2- 172.21.15.70:0/251694798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e901e140 0x55a9e90284c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb367fc640 1 --2- 172.21.15.70:0/251694798 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e9028a00 0x55a9e902adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb377fe640 1 -- 172.21.15.70:0/251694798 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e9028a00 msgr2=0x55a9e902adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:35.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb377fe640 1 --2- 172.21.15.70:0/251694798 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e9028a00 0x55a9e902adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb377fe640 1 -- 172.21.15.70:0/251694798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e901e140 msgr2=0x55a9e90284c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:35.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb377fe640 1 --2- 172.21.15.70:0/251694798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e901e140 0x55a9e90284c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb377fe640 1 -- 172.21.15.70:0/251694798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9e8c94210 con 0x55a9e901d820 2024-06-04T05:12:35.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb367fc640 1 --2- 172.21.15.70:0/251694798 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e9028a00 0x55a9e902adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:35.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb3c910640 1 --2- 172.21.15.70:0/2004377678 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbb200b2a90 0x7fbb200b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.470+0000 7fbb377fe640 1 --2- 172.21.15.70:0/251694798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901d820 0x55a9e901dc00 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fbb30011790 tx=0x7fbb30011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70885a164d8549ae server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:35.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.471+0000 7fbb1dffb640 1 -- 172.21.15.70:0/251694798 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb30003c30 con 0x55a9e901d820 2024-06-04T05:12:35.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.471+0000 7fbb36ffd640 1 --2- 172.21.15.70:0/2004377678 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbb200b2a90 0x7fbb200b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.471+0000 7fbb3c910640 1 -- 172.21.15.70:0/2004377678 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fbb280e2980 con 0x7fbb38103c60 2024-06-04T05:12:35.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.471+0000 7fbb1dffb640 1 -- 172.21.15.70:0/251694798 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbb3000fe30 con 0x55a9e901d820 2024-06-04T05:12:35.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.471+0000 7fbb1dffb640 1 -- 172.21.15.70:0/251694798 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb300047e0 con 0x55a9e901d820 2024-06-04T05:12:35.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.472+0000 7fbb40620740 1 -- 172.21.15.70:0/251694798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901d820 msgr2=0x55a9e901dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:35.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.472+0000 7fbb40620740 1 --2- 172.21.15.70:0/251694798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901d820 0x55a9e901dc00 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fbb30011790 tx=0x7fbb30011c50 comp rx=0 tx=0).stop 2024-06-04T05:12:35.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.472+0000 7fbb36ffd640 1 --2- 172.21.15.70:0/2004377678 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbb200b2a90 0x7fbb200b4f50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55a9e901edc0 tx=0x7fbb280e8530 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:35.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.472+0000 7fbb40620740 1 -- 172.21.15.70:0/251694798 shutdown_connections 2024-06-04T05:12:35.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.472+0000 7fbb40620740 1 --2- 172.21.15.70:0/251694798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e901e140 0x55a9e90284c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.472+0000 7fbb40620740 1 --2- 172.21.15.70:0/251694798 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e9028a00 0x55a9e902adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.472+0000 7fbb40620740 1 --2- 172.21.15.70:0/251694798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901d820 0x55a9e901dc00 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.472+0000 7fbb40620740 1 -- 172.21.15.70:0/251694798 >> 172.21.15.70:0/251694798 conn(0x55a9e9019d30 msgr2=0x55a9e901a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:35.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.472+0000 7fbb40620740 1 -- 172.21.15.70:0/251694798 shutdown_connections 2024-06-04T05:12:35.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.472+0000 7fbb40620740 1 -- 172.21.15.70:0/251694798 wait complete. 2024-06-04T05:12:35.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.473+0000 7fbb40620740 1 -- start start 2024-06-04T05:12:35.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.473+0000 7fbb40620740 1 Processor -- start 2024-06-04T05:12:35.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.473+0000 7fbb40620740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e901d820 0x55a9e90259f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.473+0000 7fbb40620740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901e140 0x55a9e9025f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.473+0000 7fbb40620740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e9028a00 0x55a9e9026470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.473+0000 7fbb40620740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a9e8c65790 con 0x55a9e901e140 2024-06-04T05:12:35.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.473+0000 7fbb40620740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a9e8f3b890 con 0x55a9e9028a00 2024-06-04T05:12:35.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.473+0000 7fbb40620740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a9e902b630 con 0x55a9e901d820 2024-06-04T05:12:35.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.473+0000 7fbb40620740 10 client.0 _ll_register_callbacks cb 0x55a9e8c93a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:35.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.474+0000 7fbb377fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e901d820 0x55a9e90259f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.474+0000 7fbb367fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901e140 0x55a9e9025f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.474+0000 7fbb377fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e901d820 0x55a9e90259f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:52694/0 (socket says 172.21.15.70:52694) 2024-06-04T05:12:35.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.474+0000 7fbb377fe640 1 -- 172.21.15.70:0/2442023717 learned_addr learned my addr 172.21.15.70:0/2442023717 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:35.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.474+0000 7fbb377fe640 1 --2- 172.21.15.70:0/2442023717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e9028a00 0x55a9e9026470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.474+0000 7fbb377fe640 1 -- 172.21.15.70:0/2442023717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e9028a00 msgr2=0x55a9e9026470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:35.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.474+0000 7fbb377fe640 1 --2- 172.21.15.70:0/2442023717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e9028a00 0x55a9e9026470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.474+0000 7fbb377fe640 1 -- 172.21.15.70:0/2442023717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901e140 msgr2=0x55a9e9025f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:35.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.474+0000 7fbb377fe640 1 --2- 172.21.15.70:0/2442023717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901e140 0x55a9e9025f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.474+0000 7fbb377fe640 1 -- 172.21.15.70:0/2442023717 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9e8d60620 con 0x55a9e901d820 2024-06-04T05:12:35.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.474+0000 7fbb377fe640 1 --2- 172.21.15.70:0/2442023717 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e901d820 0x55a9e90259f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbb30009fd0 tx=0x7fbb30005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=937ec5a481cab1e3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:35.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.474+0000 7fbb16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:12:35.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.474+0000 7fbb16ffd640 1 -- 172.21.15.70:0/2442023717 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb30011260 con 0x55a9e901d820 2024-06-04T05:12:35.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.474+0000 7fbb40620740 1 -- 172.21.15.70:0/2442023717 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb3000fe30 con 0x55a9e901d820 2024-06-04T05:12:35.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.474+0000 7fbb16ffd640 1 -- 172.21.15.70:0/2442023717 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fbb30004a40 con 0x55a9e901d820 2024-06-04T05:12:35.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.475+0000 7fbb16ffd640 1 -- 172.21.15.70:0/2442023717 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fbb30004c40 con 0x55a9e901d820 2024-06-04T05:12:35.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.475+0000 7fbb40620740 10 client.28486 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:35.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.475+0000 7fbb40620740 1 -- 172.21.15.70:0/2442023717 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55a9e90373e0 con 0x55a9e901d820 2024-06-04T05:12:35.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.475+0000 7fbb16ffd640 1 -- 172.21.15.70:0/2442023717 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbb3000fe30 con 0x55a9e901d820 2024-06-04T05:12:35.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.475+0000 7fbb16ffd640 1 -- 172.21.15.70:0/2442023717 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7fbb30018070 con 0x55a9e901d820 2024-06-04T05:12:35.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.475+0000 7fbb40620740 10 client.28486 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:35.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.475+0000 7fbb40620740 10 client.28486 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:12:35.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.476+0000 7fbb40620740 20 client.28486 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:35.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.476+0000 7fbb40620740 1 --2- 172.21.15.70:0/2442023717 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a9e9038850 0x55a9e903ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.476+0000 7fbb40620740 4 client.28486 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:12:35.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.476+0000 7fbb367fc640 1 --2- 172.21.15.70:0/2442023717 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a9e9038850 0x55a9e903ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.476+0000 7fbb40620740 1 -- 172.21.15.70:0/2442023717 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9e8f91b80 con 0x55a9e9038850 2024-06-04T05:12:35.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.476+0000 7fbb16ffd640 10 client.28486 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:35.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.476+0000 7fbb367fc640 1 --2- 172.21.15.70:0/2442023717 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a9e9038850 0x55a9e903ac40 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74ae65e8f8917048 server_cookie=b2bddb9fd24720c9 in_seq=0 out_seq=0 2024-06-04T05:12:35.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.477+0000 7fbb16ffd640 1 -- 172.21.15.70:0/2442023717 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a9e8f91b80 con 0x55a9e9038850 2024-06-04T05:12:35.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.477+0000 7fbb16ffd640 10 client.28486 handle_command_reply: tid=0 2024-06-04T05:12:35.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.477+0000 7fbb40620740 1 client.28486 shutdown 2024-06-04T05:12:35.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.477+0000 7fbb40620740 1 -- 172.21.15.70:0/2442023717 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e901d820 msgr2=0x55a9e90259f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:35.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.477+0000 7fbb40620740 1 --2- 172.21.15.70:0/2442023717 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e901d820 0x55a9e90259f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbb30009fd0 tx=0x7fbb30005bd0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.477+0000 7fbb40620740 1 -- 172.21.15.70:0/2442023717 shutdown_connections 2024-06-04T05:12:35.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.477+0000 7fbb40620740 1 --2- 172.21.15.70:0/2442023717 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a9e9038850 0x55a9e903ac40 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.477+0000 7fbb40620740 1 --2- 172.21.15.70:0/2442023717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e9028a00 0x55a9e9026470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.477+0000 7fbb16ffd640 0 client.28486 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:35.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.477+0000 7fbb40620740 1 --2- 172.21.15.70:0/2442023717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901e140 0x55a9e9025f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.477+0000 7fbb40620740 1 --2- 172.21.15.70:0/2442023717 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e901d820 0x55a9e90259f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.478+0000 7fbb40620740 1 -- 172.21.15.70:0/2442023717 >> 172.21.15.70:0/2442023717 conn(0x55a9e9019d30 msgr2=0x55a9e9018fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:35.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.478+0000 7fbb40620740 1 -- 172.21.15.70:0/2442023717 shutdown_connections 2024-06-04T05:12:35.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.478+0000 7fbb40620740 1 -- 172.21.15.70:0/2442023717 wait complete. 2024-06-04T05:12:35.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.478+0000 7fbb40620740 20 client.28486 trim_cache size 0 max 16384 2024-06-04T05:12:35.506 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.502+0000 7fbb40620740 1 Processor -- start 2024-06-04T05:12:35.506 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb40620740 1 -- start start 2024-06-04T05:12:35.506 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb40620740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e901d820 0x55a9e901fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.506 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb40620740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901e140 0x55a9e9018e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.506 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb40620740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e9028a00 0x55a9e9019370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.506 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb40620740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381b4d00 con 0x55a9e901e140 2024-06-04T05:12:35.506 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb40620740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381b69c0 con 0x55a9e9028a00 2024-06-04T05:12:35.506 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb40620740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381b53d0 con 0x55a9e901d820 2024-06-04T05:12:35.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb367fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901e140 0x55a9e9018e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb367fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901e140 0x55a9e9018e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54778/0 (socket says 172.21.15.70:54778) 2024-06-04T05:12:35.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb367fc640 1 -- 172.21.15.70:0/434780380 learned_addr learned my addr 172.21.15.70:0/434780380 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:35.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb377fe640 1 --2- 172.21.15.70:0/434780380 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e901d820 0x55a9e901fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb377fe640 1 --2- 172.21.15.70:0/434780380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e9028a00 0x55a9e9019370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb367fc640 1 -- 172.21.15.70:0/434780380 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e901d820 msgr2=0x55a9e901fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:35.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb367fc640 1 --2- 172.21.15.70:0/434780380 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e901d820 0x55a9e901fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb367fc640 1 -- 172.21.15.70:0/434780380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e9028a00 msgr2=0x55a9e9019370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:35.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb367fc640 1 --2- 172.21.15.70:0/434780380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e9028a00 0x55a9e9019370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb367fc640 1 -- 172.21.15.70:0/434780380 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9e903afb0 con 0x55a9e901e140 2024-06-04T05:12:35.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.503+0000 7fbb367fc640 1 --2- 172.21.15.70:0/434780380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901e140 0x55a9e9018e30 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009230 tx=0x7fbb2c004260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7835ae3c3948e1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:35.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.504+0000 7fbb1d7fa640 1 -- 172.21.15.70:0/434780380 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c035040 con 0x55a9e901e140 2024-06-04T05:12:35.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.504+0000 7fbb1d7fa640 1 -- 172.21.15.70:0/434780380 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbb2c0044d0 con 0x55a9e901e140 2024-06-04T05:12:35.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.504+0000 7fbb1d7fa640 1 -- 172.21.15.70:0/434780380 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c003a60 con 0x55a9e901e140 2024-06-04T05:12:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.505+0000 7fbb40620740 1 -- 172.21.15.70:0/434780380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901e140 msgr2=0x55a9e9018e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.505+0000 7fbb40620740 1 --2- 172.21.15.70:0/434780380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901e140 0x55a9e9018e30 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009230 tx=0x7fbb2c004260 comp rx=0 tx=0).stop 2024-06-04T05:12:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.505+0000 7fbb40620740 1 -- 172.21.15.70:0/434780380 shutdown_connections 2024-06-04T05:12:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.505+0000 7fbb40620740 1 --2- 172.21.15.70:0/434780380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e9028a00 0x55a9e9019370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.505+0000 7fbb40620740 1 --2- 172.21.15.70:0/434780380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901e140 0x55a9e9018e30 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.505+0000 7fbb40620740 1 --2- 172.21.15.70:0/434780380 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e901d820 0x55a9e901fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.505+0000 7fbb40620740 1 -- 172.21.15.70:0/434780380 >> 172.21.15.70:0/434780380 conn(0x55a9e9019d30 msgr2=0x55a9e9018720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.505+0000 7fbb40620740 1 -- 172.21.15.70:0/434780380 shutdown_connections 2024-06-04T05:12:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.505+0000 7fbb40620740 1 -- 172.21.15.70:0/434780380 wait complete. 2024-06-04T05:12:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.505+0000 7fbb40620740 1 -- start start 2024-06-04T05:12:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb40620740 1 Processor -- start 2024-06-04T05:12:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb40620740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901d820 0x55a9e9030f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb40620740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e901e140 0x55a9e902c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb40620740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e9028a00 0x55a9e902c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb40620740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381b6230 con 0x55a9e901d820 2024-06-04T05:12:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb40620740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381b5b00 con 0x55a9e9028a00 2024-06-04T05:12:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb40620740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381b2d00 con 0x55a9e901e140 2024-06-04T05:12:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb377fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901d820 0x55a9e9030f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb377fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901d820 0x55a9e9030f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54792/0 (socket says 172.21.15.70:54792) 2024-06-04T05:12:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb377fe640 1 -- 172.21.15.70:0/3297470742 learned_addr learned my addr 172.21.15.70:0/3297470742 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb40620740 10 client.0 _ll_register_callbacks cb 0x7fbb280e8400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb367fc640 1 --2- 172.21.15.70:0/3297470742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e901e140 0x55a9e902c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb377fe640 1 --2- 172.21.15.70:0/3297470742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e9028a00 0x55a9e902c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb377fe640 1 -- 172.21.15.70:0/3297470742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e901e140 msgr2=0x55a9e902c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb377fe640 1 --2- 172.21.15.70:0/3297470742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e901e140 0x55a9e902c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb377fe640 1 -- 172.21.15.70:0/3297470742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e9028a00 msgr2=0x55a9e902c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb377fe640 1 --2- 172.21.15.70:0/3297470742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e9028a00 0x55a9e902c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.506+0000 7fbb377fe640 1 -- 172.21.15.70:0/3297470742 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9e902cfd0 con 0x55a9e901d820 2024-06-04T05:12:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.507+0000 7fbb377fe640 1 --2- 172.21.15.70:0/3297470742 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901d820 0x55a9e9030f20 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fbb30005fe0 tx=0x7fbb30005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0fa9ccbe6b1804a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.507+0000 7fbb16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:12:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.507+0000 7fbb16ffd640 1 -- 172.21.15.70:0/3297470742 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb300055d0 con 0x55a9e901d820 2024-06-04T05:12:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.507+0000 7fbb40620740 1 -- 172.21.15.70:0/3297470742 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb2c0044d0 con 0x55a9e901d820 2024-06-04T05:12:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.507+0000 7fbb16ffd640 1 -- 172.21.15.70:0/3297470742 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fbb300042d0 con 0x55a9e901d820 2024-06-04T05:12:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.507+0000 7fbb16ffd640 1 -- 172.21.15.70:0/3297470742 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fbb3000f790 con 0x55a9e901d820 2024-06-04T05:12:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.507+0000 7fbb40620740 10 client.19512 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.507+0000 7fbb40620740 1 -- 172.21.15.70:0/3297470742 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55a9e90a3280 con 0x55a9e901d820 2024-06-04T05:12:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.507+0000 7fbb16ffd640 1 -- 172.21.15.70:0/3297470742 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbb2c0044d0 con 0x55a9e901d820 2024-06-04T05:12:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.508+0000 7fbb16ffd640 1 -- 172.21.15.70:0/3297470742 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7fbb30018040 con 0x55a9e901d820 2024-06-04T05:12:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.508+0000 7fbb40620740 10 client.19512 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.508+0000 7fbb40620740 10 client.19512 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:12:35.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.508+0000 7fbb40620740 20 client.19512 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:35.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.508+0000 7fbb40620740 1 --2- 172.21.15.70:0/3297470742 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a9e90a39a0 0x55a9e90a5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:35.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.508+0000 7fbb40620740 4 client.19512 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:12:35.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.508+0000 7fbb40620740 1 -- 172.21.15.70:0/3297470742 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a9e8f8b360 con 0x55a9e90a39a0 2024-06-04T05:12:35.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.508+0000 7fbb367fc640 1 --2- 172.21.15.70:0/3297470742 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a9e90a39a0 0x55a9e90a5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:35.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.508+0000 7fbb367fc640 1 --2- 172.21.15.70:0/3297470742 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a9e90a39a0 0x55a9e90a5dc0 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc2d82c56f164fed server_cookie=4aa2f38bf5bc5da0 in_seq=0 out_seq=0 2024-06-04T05:12:35.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.508+0000 7fbb16ffd640 10 client.19512 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.573+0000 7fbb16ffd640 1 -- 172.21.15.70:0/3297470742 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55a9e8f8b360 con 0x55a9e90a39a0 2024-06-04T05:12:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.573+0000 7fbb16ffd640 10 client.19512 handle_command_reply: tid=0 2024-06-04T05:12:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.573+0000 7fbb40620740 1 client.19512 shutdown 2024-06-04T05:12:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.573+0000 7fbb40620740 1 -- 172.21.15.70:0/3297470742 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901d820 msgr2=0x55a9e9030f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.573+0000 7fbb40620740 1 --2- 172.21.15.70:0/3297470742 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901d820 0x55a9e9030f20 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fbb30005fe0 tx=0x7fbb30005bd0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.573+0000 7fbb40620740 1 -- 172.21.15.70:0/3297470742 shutdown_connections 2024-06-04T05:12:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.573+0000 7fbb40620740 1 --2- 172.21.15.70:0/3297470742 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a9e90a39a0 0x55a9e90a5dc0 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.573+0000 7fbb40620740 1 --2- 172.21.15.70:0/3297470742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9e9028a00 0x55a9e902c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.573+0000 7fbb16ffd640 0 client.19512 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.573+0000 7fbb40620740 1 --2- 172.21.15.70:0/3297470742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a9e901e140 0x55a9e902c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.573+0000 7fbb40620740 1 --2- 172.21.15.70:0/3297470742 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a9e901d820 0x55a9e9030f20 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.573+0000 7fbb40620740 1 -- 172.21.15.70:0/3297470742 >> 172.21.15.70:0/3297470742 conn(0x55a9e9019d30 msgr2=0x55a9e901f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.573+0000 7fbb40620740 1 -- 172.21.15.70:0/3297470742 shutdown_connections 2024-06-04T05:12:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.573+0000 7fbb40620740 1 -- 172.21.15.70:0/3297470742 wait complete. 2024-06-04T05:12:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.574+0000 7fbb40620740 20 client.19512 trim_cache size 0 max 16384 2024-06-04T05:12:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:12:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:12:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:12:35.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.574+0000 7fbb3d912640 1 -- 172.21.15.70:0/2004377678 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbb200b2a90 msgr2=0x7fbb200b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:35.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.574+0000 7fbb3d912640 1 --2- 172.21.15.70:0/2004377678 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbb200b2a90 0x7fbb200b4f50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55a9e901edc0 tx=0x7fbb280e8530 comp rx=0 tx=0).stop 2024-06-04T05:12:35.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.574+0000 7fbb3d912640 1 -- 172.21.15.70:0/2004377678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb38103c60 msgr2=0x7fbb381018b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:35.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.575+0000 7fbb3d912640 1 --2- 172.21.15.70:0/2004377678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb38103c60 0x7fbb381018b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800e9f0 tx=0x7fbb2800eeb0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.575+0000 7fbb3d912640 1 -- 172.21.15.70:0/2004377678 shutdown_connections 2024-06-04T05:12:35.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.575+0000 7fbb3d912640 1 --2- 172.21.15.70:0/2004377678 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbb200b2a90 0x7fbb200b4f50 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.575+0000 7fbb3d912640 1 --2- 172.21.15.70:0/2004377678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb38104580 0x7fbb38101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.575+0000 7fbb3d912640 1 --2- 172.21.15.70:0/2004377678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb3810d030 0x7fbb38102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.575+0000 7fbb3d912640 1 --2- 172.21.15.70:0/2004377678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb38103c60 0x7fbb381018b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:35.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.575+0000 7fbb3d912640 1 -- 172.21.15.70:0/2004377678 >> 172.21.15.70:0/2004377678 conn(0x7fbb380fd420 msgr2=0x7fbb380fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:35.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.575+0000 7fbb3d912640 1 -- 172.21.15.70:0/2004377678 shutdown_connections 2024-06-04T05:12:35.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:35.575+0000 7fbb3d912640 1 -- 172.21.15.70:0/2004377678 wait complete. 2024-06-04T05:12:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:35 smithi112 ceph-mon[36453]: pgmap v481: 97 pgs: 97 active+clean; 424 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 286 op/s 2024-06-04T05:12:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:35 smithi098 ceph-mon[35569]: pgmap v481: 97 pgs: 97 active+clean; 424 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 286 op/s 2024-06-04T05:12:35.800 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15378 11.45 MB/sec warmup 87 sec latency 83.472 ms 2024-06-04T05:12:36.437 INFO:tasks.cephfs.filesystem:scrub status for tag:39b8a1c3-5482-495c-86fd-d8bfe78f3810 - {'path': '/', 'tag': '39b8a1c3-5482-495c-86fd-d8bfe78f3810', 'options': 'recursive,force'} 2024-06-04T05:12:36.800 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15560 11.46 MB/sec warmup 88 sec latency 53.589 ms 2024-06-04T05:12:37.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.050+0000 7f8a9a36b700 1 -- 172.21.15.70:0/3587567458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a94102ae0 msgr2=0x7f8a94102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:37.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.050+0000 7f8a9a36b700 1 --2- 172.21.15.70:0/3587567458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a94102ae0 0x7f8a94102ec0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8a840099c0 tx=0x7f8a84009cd0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.050+0000 7f8a9a36b700 1 -- 172.21.15.70:0/3587567458 shutdown_connections 2024-06-04T05:12:37.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.050+0000 7f8a9a36b700 1 --2- 172.21.15.70:0/3587567458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a94107a30 0x7f8a9410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.050+0000 7f8a9a36b700 1 --2- 172.21.15.70:0/3587567458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a94102ae0 0x7f8a94102ec0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.050+0000 7f8a9a36b700 1 --2- 172.21.15.70:0/3587567458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a94107070 0x7f8a941074f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.050+0000 7f8a9a36b700 1 -- 172.21.15.70:0/3587567458 >> 172.21.15.70:0/3587567458 conn(0x7f8a940793f0 msgr2=0x7f8a94079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:37.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.050+0000 7f8a9a36b700 1 -- 172.21.15.70:0/3587567458 shutdown_connections 2024-06-04T05:12:37.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.050+0000 7f8a9a36b700 1 -- 172.21.15.70:0/3587567458 wait complete. 2024-06-04T05:12:37.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.050+0000 7f8a9a36b700 1 Processor -- start 2024-06-04T05:12:37.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.051+0000 7f8a9a36b700 1 -- start start 2024-06-04T05:12:37.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.051+0000 7f8a9a36b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a94102ae0 0x7f8a9419d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.051+0000 7f8a9a36b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a94107070 0x7f8a9419db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.051+0000 7f8a9a36b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a94107a30 0x7f8a941976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.051+0000 7f8a9a36b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a94111420 con 0x7f8a94107070 2024-06-04T05:12:37.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.051+0000 7f8a9a36b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a941112a0 con 0x7f8a94102ae0 2024-06-04T05:12:37.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.051+0000 7f8a9a36b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8a941115a0 con 0x7f8a94107a30 2024-06-04T05:12:37.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.051+0000 7f8a937fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a94107070 0x7f8a9419db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.051+0000 7f8a937fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a94107070 0x7f8a9419db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35644/0 (socket says 172.21.15.70:35644) 2024-06-04T05:12:37.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.051+0000 7f8a937fe700 1 -- 172.21.15.70:0/3718776751 learned_addr learned my addr 172.21.15.70:0/3718776751 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:37.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.051+0000 7f8a93fff700 1 --2- 172.21.15.70:0/3718776751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a94102ae0 0x7f8a9419d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.051+0000 7f8a98908700 1 --2- 172.21.15.70:0/3718776751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a94107a30 0x7f8a941976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.052+0000 7f8a93fff700 1 -- 172.21.15.70:0/3718776751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a94107a30 msgr2=0x7f8a941976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:37.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.052+0000 7f8a93fff700 1 --2- 172.21.15.70:0/3718776751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a94107a30 0x7f8a941976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.052+0000 7f8a93fff700 1 -- 172.21.15.70:0/3718776751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a94107070 msgr2=0x7f8a9419db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:37.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.052+0000 7f8a93fff700 1 --2- 172.21.15.70:0/3718776751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a94107070 0x7f8a9419db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.052+0000 7f8a93fff700 1 -- 172.21.15.70:0/3718776751 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a94197fc0 con 0x7f8a94102ae0 2024-06-04T05:12:37.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.052+0000 7f8a98908700 1 --2- 172.21.15.70:0/3718776751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a94107a30 0x7f8a941976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:12:37.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.052+0000 7f8a93fff700 1 --2- 172.21.15.70:0/3718776751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a94102ae0 0x7f8a9419d610 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8a840098e0 tx=0x7f8a840045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:37.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.052+0000 7f8a917fa700 1 -- 172.21.15.70:0/3718776751 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a84009e60 con 0x7f8a94102ae0 2024-06-04T05:12:37.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.052+0000 7f8a9a36b700 1 -- 172.21.15.70:0/3718776751 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a94198250 con 0x7f8a94102ae0 2024-06-04T05:12:37.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.052+0000 7f8a917fa700 1 -- 172.21.15.70:0/3718776751 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a840047c0 con 0x7f8a94102ae0 2024-06-04T05:12:37.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.053+0000 7f8a917fa700 1 -- 172.21.15.70:0/3718776751 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a84003e60 con 0x7f8a94102ae0 2024-06-04T05:12:37.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.053+0000 7f8a9a36b700 1 -- 172.21.15.70:0/3718776751 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a941a43c0 con 0x7f8a94102ae0 2024-06-04T05:12:37.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.056+0000 7f8a917fa700 1 -- 172.21.15.70:0/3718776751 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f8a84028060 con 0x7f8a94102ae0 2024-06-04T05:12:37.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.057+0000 7f8a917fa700 1 --2- 172.21.15.70:0/3718776751 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8a7c0b6ff0 0x7f8a7c0b94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.057+0000 7f8a937fe700 1 --2- 172.21.15.70:0/3718776751 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8a7c0b6ff0 0x7f8a7c0b94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.057+0000 7f8a917fa700 1 -- 172.21.15.70:0/3718776751 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f8a840e2c70 con 0x7f8a94102ae0 2024-06-04T05:12:37.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.058+0000 7f8aa4c00b80 1 Processor -- start 2024-06-04T05:12:37.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.058+0000 7f8aa4c00b80 1 -- start start 2024-06-04T05:12:37.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.058+0000 7f8aa4c00b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0f7b0 0x55aa44d0fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.058+0000 7f8aa4c00b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d0ac80 0x55aa44d0b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.058+0000 7f8aa4c00b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0b640 0x55aa44d176b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.059+0000 7f8aa4c00b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55aa44c68f00 con 0x55aa44d0b640 2024-06-04T05:12:37.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.059+0000 7f8aa4c00b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55aa44c58aa0 con 0x55aa44d0ac80 2024-06-04T05:12:37.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.059+0000 7f8aa4c00b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55aa44c5cae0 con 0x55aa44d0f7b0 2024-06-04T05:12:37.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.059+0000 7f8a937fe700 1 --2- 172.21.15.70:0/3718776751 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8a7c0b6ff0 0x7f8a7c0b94b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8a94198ea0 tx=0x7f8a8800b500 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:37.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.059+0000 7f8a937fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d0ac80 0x55aa44d0b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.059+0000 7f8a937fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d0ac80 0x55aa44d0b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:37530/0 (socket says 172.21.15.70:37530) 2024-06-04T05:12:37.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.059+0000 7f8a937fe700 1 -- 172.21.15.70:0/605080576 learned_addr learned my addr 172.21.15.70:0/605080576 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:37.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.059+0000 7f8a98908700 1 --2- 172.21.15.70:0/605080576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0b640 0x55aa44d176b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.059+0000 7f8a93fff700 1 --2- 172.21.15.70:0/605080576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0f7b0 0x55aa44d0fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.059+0000 7f8a98908700 1 -- 172.21.15.70:0/605080576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0f7b0 msgr2=0x55aa44d0fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:37.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.059+0000 7f8a98908700 1 --2- 172.21.15.70:0/605080576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0f7b0 0x55aa44d0fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.059+0000 7f8a98908700 1 -- 172.21.15.70:0/605080576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d0ac80 msgr2=0x55aa44d0b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:37.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.059+0000 7f8a98908700 1 --2- 172.21.15.70:0/605080576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d0ac80 0x55aa44d0b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.059+0000 7f8a98908700 1 -- 172.21.15.70:0/605080576 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa44c02140 con 0x55aa44d0b640 2024-06-04T05:12:37.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.060+0000 7f8a98908700 1 --2- 172.21.15.70:0/605080576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0b640 0x55aa44d176b0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0093a0 tx=0x7f8a8c009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efeb7473d6aacdc4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:37.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.060+0000 7f8a7a7fc700 1 -- 172.21.15.70:0/605080576 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c019030 con 0x55aa44d0b640 2024-06-04T05:12:37.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.060+0000 7f8a7a7fc700 1 -- 172.21.15.70:0/605080576 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a8c004bb0 con 0x55aa44d0b640 2024-06-04T05:12:37.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.060+0000 7f8a7a7fc700 1 -- 172.21.15.70:0/605080576 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c005020 con 0x55aa44d0b640 2024-06-04T05:12:37.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.061+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/605080576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0b640 msgr2=0x55aa44d176b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:37.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.061+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/605080576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0b640 0x55aa44d176b0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0093a0 tx=0x7f8a8c009920 comp rx=0 tx=0).stop 2024-06-04T05:12:37.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.061+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/605080576 shutdown_connections 2024-06-04T05:12:37.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.061+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/605080576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0b640 0x55aa44d176b0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.061+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/605080576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0f7b0 0x55aa44d0fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.061+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/605080576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d0ac80 0x55aa44d0b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.061+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/605080576 >> 172.21.15.70:0/605080576 conn(0x55aa44c363d0 msgr2=0x55aa44d05970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:37.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.061+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/605080576 shutdown_connections 2024-06-04T05:12:37.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.061+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/605080576 wait complete. 2024-06-04T05:12:37.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.061+0000 7f8aa4c00b80 1 -- start start 2024-06-04T05:12:37.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.062+0000 7f8aa4c00b80 1 Processor -- start 2024-06-04T05:12:37.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.062+0000 7f8aa4c00b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0ac80 0x55aa44d17320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.062+0000 7f8aa4c00b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d0b640 0x55aa44d14620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.062+0000 7f8aa4c00b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 0x55aa44d14b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.062+0000 7f8aa4c00b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55aa44b5f130 con 0x55aa44d0f7b0 2024-06-04T05:12:37.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.062+0000 7f8aa4c00b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55aa44bc0f30 con 0x55aa44d0b640 2024-06-04T05:12:37.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.062+0000 7f8aa4c00b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55aa44d17d10 con 0x55aa44d0ac80 2024-06-04T05:12:37.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.062+0000 7f8aa4c00b80 10 client.0 _ll_register_callbacks cb 0x55aa44bf3ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:37.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.063+0000 7f8a93fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0ac80 0x55aa44d17320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.063+0000 7f8a93fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0ac80 0x55aa44d17320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:55426/0 (socket says 172.21.15.70:55426) 2024-06-04T05:12:37.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.063+0000 7f8a93fff700 1 -- 172.21.15.70:0/2300439570 learned_addr learned my addr 172.21.15.70:0/2300439570 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:37.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.063+0000 7f8a98908700 1 --2- 172.21.15.70:0/2300439570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 0x55aa44d14b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.063+0000 7f8a937fe700 1 --2- 172.21.15.70:0/2300439570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d0b640 0x55aa44d14620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.063+0000 7f8a98908700 1 -- 172.21.15.70:0/2300439570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0ac80 msgr2=0x55aa44d17320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:37.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.063+0000 7f8a98908700 1 --2- 172.21.15.70:0/2300439570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0ac80 0x55aa44d17320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.063+0000 7f8a98908700 1 -- 172.21.15.70:0/2300439570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d0b640 msgr2=0x55aa44d14620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:37.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.063+0000 7f8a98908700 1 --2- 172.21.15.70:0/2300439570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d0b640 0x55aa44d14620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.063+0000 7f8a98908700 1 -- 172.21.15.70:0/2300439570 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa44ba8a60 con 0x55aa44d0f7b0 2024-06-04T05:12:37.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.063+0000 7f8a93fff700 1 --2- 172.21.15.70:0/2300439570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0ac80 0x55aa44d17320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:37.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.063+0000 7f8a98908700 1 --2- 172.21.15.70:0/2300439570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 0x55aa44d14b60 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009d20 tx=0x7f8a8c015670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=103a429280e88825 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:37.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.063+0000 7f8a737fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:12:37.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.063+0000 7f8a737fe700 1 -- 172.21.15.70:0/2300439570 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c01f860 con 0x55aa44d0f7b0 2024-06-04T05:12:37.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.064+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/2300439570 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a8c004bb0 con 0x55aa44d0f7b0 2024-06-04T05:12:37.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.064+0000 7f8a737fe700 1 -- 172.21.15.70:0/2300439570 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f8a8c01fa00 con 0x55aa44d0f7b0 2024-06-04T05:12:37.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.064+0000 7f8a737fe700 1 -- 172.21.15.70:0/2300439570 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f8a8c015920 con 0x55aa44d0f7b0 2024-06-04T05:12:37.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.064+0000 7f8aa4c00b80 10 client.19524 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:37.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.064+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/2300439570 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55aa44c128f0 con 0x55aa44d0f7b0 2024-06-04T05:12:37.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.064+0000 7f8a737fe700 1 -- 172.21.15.70:0/2300439570 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a8c004bb0 con 0x55aa44d0f7b0 2024-06-04T05:12:37.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.064+0000 7f8a737fe700 1 -- 172.21.15.70:0/2300439570 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7f8a8c030070 con 0x55aa44d0f7b0 2024-06-04T05:12:37.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.064+0000 7f8aa4c00b80 10 client.19524 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:37.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.064+0000 7f8aa4c00b80 10 client.19524 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:12:37.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.064+0000 7f8aa4c00b80 20 client.19524 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:37.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.064+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/2300439570 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55aa44d247a0 0x55aa44d26b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.064+0000 7f8aa4c00b80 4 client.19524 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:12:37.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.064+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/2300439570 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa44c6ae50 con 0x55aa44d247a0 2024-06-04T05:12:37.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.065+0000 7f8a93fff700 1 --2- 172.21.15.70:0/2300439570 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55aa44d247a0 0x55aa44d26b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.065+0000 7f8a93fff700 1 --2- 172.21.15.70:0/2300439570 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55aa44d247a0 0x55aa44d26b90 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2da821983c0731d server_cookie=d2ab2bc3fce35b3d in_seq=0 out_seq=0 2024-06-04T05:12:37.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.065+0000 7f8a737fe700 10 client.19524 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:37.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.066+0000 7f8a737fe700 1 -- 172.21.15.70:0/2300439570 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55aa44c6ae50 con 0x55aa44d247a0 2024-06-04T05:12:37.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.066+0000 7f8a737fe700 10 client.19524 handle_command_reply: tid=0 2024-06-04T05:12:37.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.066+0000 7f8aa4c00b80 1 client.19524 shutdown 2024-06-04T05:12:37.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.066+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/2300439570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 msgr2=0x55aa44d14b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:37.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.066+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/2300439570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 0x55aa44d14b60 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009d20 tx=0x7f8a8c015670 comp rx=0 tx=0).stop 2024-06-04T05:12:37.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.066+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/2300439570 shutdown_connections 2024-06-04T05:12:37.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.066+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/2300439570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 0x55aa44d14b60 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.066+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/2300439570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0ac80 0x55aa44d17320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.066+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/2300439570 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55aa44d247a0 0x55aa44d26b90 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.066+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/2300439570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d0b640 0x55aa44d14620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.066+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/2300439570 >> 172.21.15.70:0/2300439570 conn(0x55aa44c363d0 msgr2=0x55aa44d09c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:37.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.066+0000 7f8a737fe700 0 client.19524 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:37.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.066+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/2300439570 shutdown_connections 2024-06-04T05:12:37.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.066+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/2300439570 wait complete. 2024-06-04T05:12:37.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.067+0000 7f8aa4c00b80 20 client.19524 trim_cache size 0 max 16384 2024-06-04T05:12:37.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.098+0000 7f8aa4c00b80 1 Processor -- start 2024-06-04T05:12:37.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.098+0000 7f8aa4c00b80 1 -- start start 2024-06-04T05:12:37.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.098+0000 7f8aa4c00b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d0ac80 0x55aa44cfff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.098+0000 7f8aa4c00b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0b640 0x55aa44d0ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.098+0000 7f8aa4c00b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 0x55aa44d09c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.098+0000 7f8aa4c00b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a941abae0 con 0x55aa44d0f7b0 2024-06-04T05:12:37.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.098+0000 7f8aa4c00b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a941ad7a0 con 0x55aa44d0ac80 2024-06-04T05:12:37.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.098+0000 7f8aa4c00b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8a941ac1b0 con 0x55aa44d0b640 2024-06-04T05:12:37.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.098+0000 7f8a937fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0b640 0x55aa44d0ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.098+0000 7f8a93fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d0ac80 0x55aa44cfff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.098+0000 7f8a937fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0b640 0x55aa44d0ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:55436/0 (socket says 172.21.15.70:55436) 2024-06-04T05:12:37.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.098+0000 7f8a937fe700 1 -- 172.21.15.70:0/313476488 learned_addr learned my addr 172.21.15.70:0/313476488 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:37.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.098+0000 7f8a98908700 1 --2- 172.21.15.70:0/313476488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 0x55aa44d09c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.098+0000 7f8a93fff700 1 -- 172.21.15.70:0/313476488 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0b640 msgr2=0x55aa44d0ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:37.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.098+0000 7f8a93fff700 1 --2- 172.21.15.70:0/313476488 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0b640 0x55aa44d0ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.098+0000 7f8a93fff700 1 -- 172.21.15.70:0/313476488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 msgr2=0x55aa44d09c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:37.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.099+0000 7f8a93fff700 1 --2- 172.21.15.70:0/313476488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 0x55aa44d09c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.099+0000 7f8a93fff700 1 -- 172.21.15.70:0/313476488 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa44d0ff00 con 0x55aa44d0ac80 2024-06-04T05:12:37.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.099+0000 7f8a98908700 1 --2- 172.21.15.70:0/313476488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 0x55aa44d09c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:37.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.099+0000 7f8a93fff700 1 --2- 172.21.15.70:0/313476488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d0ac80 0x55aa44cfff00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8a84005000 tx=0x7f8a840a9e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7f88f0b9f0fa9d8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:37.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.099+0000 7f8a79ffb700 1 -- 172.21.15.70:0/313476488 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a840cc050 con 0x55aa44d0ac80 2024-06-04T05:12:37.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.099+0000 7f8a79ffb700 1 -- 172.21.15.70:0/313476488 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a840b2d20 con 0x55aa44d0ac80 2024-06-04T05:12:37.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.099+0000 7f8a79ffb700 1 -- 172.21.15.70:0/313476488 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a840aa410 con 0x55aa44d0ac80 2024-06-04T05:12:37.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.100+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/313476488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d0ac80 msgr2=0x55aa44cfff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:37.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.100+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/313476488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d0ac80 0x55aa44cfff00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8a84005000 tx=0x7f8a840a9e10 comp rx=0 tx=0).stop 2024-06-04T05:12:37.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.100+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/313476488 shutdown_connections 2024-06-04T05:12:37.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.100+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/313476488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 0x55aa44d09c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.100+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/313476488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d0ac80 0x55aa44cfff00 secure :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8a84005000 tx=0x7f8a840a9e10 comp rx=0 tx=0).stop 2024-06-04T05:12:37.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.100+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/313476488 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0b640 0x55aa44d0ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.100+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/313476488 >> 172.21.15.70:0/313476488 conn(0x55aa44c363d0 msgr2=0x55aa44d0e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:37.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.100+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/313476488 shutdown_connections 2024-06-04T05:12:37.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.100+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/313476488 wait complete. 2024-06-04T05:12:37.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.100+0000 7f8aa4c00b80 1 -- start start 2024-06-04T05:12:37.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.101+0000 7f8aa4c00b80 1 Processor -- start 2024-06-04T05:12:37.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.101+0000 7f8aa4c00b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0b640 0x55aa44d1af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.101+0000 7f8aa4c00b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 0x55aa44d1b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.101+0000 7f8aa4c00b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d1ba50 0x55aa44d8e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.101+0000 7f8aa4c00b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a941ad010 con 0x55aa44d0f7b0 2024-06-04T05:12:37.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.101+0000 7f8aa4c00b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a941ac8e0 con 0x55aa44d1ba50 2024-06-04T05:12:37.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.101+0000 7f8aa4c00b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8a941a9aa0 con 0x55aa44d0b640 2024-06-04T05:12:37.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.101+0000 7f8aa4c00b80 10 client.0 _ll_register_callbacks cb 0x55aa44bf3ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:37.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.101+0000 7f8a937fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 0x55aa44d1b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.101+0000 7f8a93fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0b640 0x55aa44d1af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.101+0000 7f8a937fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 0x55aa44d1b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35680/0 (socket says 172.21.15.70:35680) 2024-06-04T05:12:37.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.101+0000 7f8a937fe700 1 -- 172.21.15.70:0/2490553284 learned_addr learned my addr 172.21.15.70:0/2490553284 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:37.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.101+0000 7f8a93fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0b640 0x55aa44d1af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:55448/0 (socket says 172.21.15.70:55448) 2024-06-04T05:12:37.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.101+0000 7f8a98908700 1 --2- 172.21.15.70:0/2490553284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d1ba50 0x55aa44d8e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.102+0000 7f8a93fff700 1 -- 172.21.15.70:0/2490553284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d1ba50 msgr2=0x55aa44d8e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:37.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.102+0000 7f8a93fff700 1 --2- 172.21.15.70:0/2490553284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d1ba50 0x55aa44d8e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.102+0000 7f8a93fff700 1 -- 172.21.15.70:0/2490553284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 msgr2=0x55aa44d1b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:37.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.102+0000 7f8a93fff700 1 --2- 172.21.15.70:0/2490553284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 0x55aa44d1b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.102+0000 7f8a93fff700 1 -- 172.21.15.70:0/2490553284 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa44d8e850 con 0x55aa44d0b640 2024-06-04T05:12:37.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.102+0000 7f8a93fff700 1 --2- 172.21.15.70:0/2490553284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0b640 0x55aa44d1af80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8a8400b910 tx=0x7f8a840094b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bb0c1f7b45bc0cf server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:37.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.102+0000 7f8a737fe700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:12:37.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.102+0000 7f8a737fe700 1 -- 172.21.15.70:0/2490553284 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a840cc040 con 0x55aa44d0b640 2024-06-04T05:12:37.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.102+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/2490553284 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a840b2d20 con 0x55aa44d0b640 2024-06-04T05:12:37.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.102+0000 7f8a737fe700 1 -- 172.21.15.70:0/2490553284 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f8a840aa1b0 con 0x55aa44d0b640 2024-06-04T05:12:37.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.103+0000 7f8a737fe700 1 -- 172.21.15.70:0/2490553284 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f8a840205c0 con 0x55aa44d0b640 2024-06-04T05:12:37.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.103+0000 7f8aa4c00b80 10 client.28510 fetch_fsmap learned FSMap version 44 2024-06-04T05:12:37.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.103+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/2490553284 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55aa44d8fb10 con 0x55aa44d0b640 2024-06-04T05:12:37.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.103+0000 7f8a737fe700 1 -- 172.21.15.70:0/2490553284 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a840b2d20 con 0x55aa44d0b640 2024-06-04T05:12:37.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.103+0000 7f8a737fe700 1 -- 172.21.15.70:0/2490553284 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 44) ==== 8402+0+0 (secure 0 0 0) 0x7f8a840ad040 con 0x55aa44d0b640 2024-06-04T05:12:37.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.103+0000 7f8aa4c00b80 10 client.28510 fetch_fsmap finished waiting for FSMap version 44 2024-06-04T05:12:37.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.103+0000 7f8aa4c00b80 10 client.28510 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:12:37.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.103+0000 7f8aa4c00b80 20 client.28510 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:37.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.103+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/2490553284 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55aa44d90220 0x55aa44d92640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:37.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.103+0000 7f8aa4c00b80 4 client.28510 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 152.6, "await_for": 45.0}] 2024-06-04T05:12:37.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.103+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/2490553284 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 152.6, "await_for": 45.0}) -- 0x55aa44c64630 con 0x55aa44d90220 2024-06-04T05:12:37.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.103+0000 7f8a937fe700 1 --2- 172.21.15.70:0/2490553284 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55aa44d90220 0x55aa44d92640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:37.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.104+0000 7f8a937fe700 1 --2- 172.21.15.70:0/2490553284 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55aa44d90220 0x55aa44d92640 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdaf82e224b9936 server_cookie=a060d767ea3aec1d in_seq=0 out_seq=0 2024-06-04T05:12:37.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.104+0000 7f8a737fe700 10 client.28510 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:37.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.441+0000 7f8a737fe700 1 -- 172.21.15.70:0/2490553284 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55aa44c64630 con 0x55aa44d90220 2024-06-04T05:12:37.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.441+0000 7f8a737fe700 10 client.28510 handle_command_reply: tid=0 2024-06-04T05:12:37.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.441+0000 7f8aa4c00b80 1 client.28510 shutdown 2024-06-04T05:12:37.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.442+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/2490553284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0b640 msgr2=0x55aa44d1af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:37.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.442+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/2490553284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0b640 0x55aa44d1af80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8a8400b910 tx=0x7f8a840094b0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.442+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/2490553284 shutdown_connections 2024-06-04T05:12:37.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.442+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/2490553284 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55aa44d90220 0x55aa44d92640 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.442+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/2490553284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aa44d1ba50 0x55aa44d8e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.442+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/2490553284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aa44d0b640 0x55aa44d1af80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.442+0000 7f8aa4c00b80 1 --2- 172.21.15.70:0/2490553284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa44d0f7b0 0x55aa44d1b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.442+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/2490553284 >> 172.21.15.70:0/2490553284 conn(0x55aa44c363d0 msgr2=0x55aa44d09c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:37.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.442+0000 7f8a737fe700 0 client.28510 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:37.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.442+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/2490553284 shutdown_connections 2024-06-04T05:12:37.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.442+0000 7f8aa4c00b80 1 -- 172.21.15.70:0/2490553284 wait complete. 2024-06-04T05:12:37.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.442+0000 7f8aa4c00b80 20 client.28510 trim_cache size 0 max 16384 2024-06-04T05:12:37.446 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:12:37.446 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 44, 2024-06-04T05:12:37.446 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:12:37.446 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 46, 2024-06-04T05:12:37.446 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:12:37.446 INFO:teuthology.orchestra.run.smithi070.stdout: "e648773": { 2024-06-04T05:12:37.446 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 46, 2024-06-04T05:12:37.446 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:12:37.447 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:12:37.447 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:12:37.447 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:12:37.448 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:12:37.448 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:12:37.448 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 152.6, 2024-06-04T05:12:37.448 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:12:37.448 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:12:37.448 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:12:37.448 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:12:37.448 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:12:37.448 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:12:37.448 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:12:37.448 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:12:37.448 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:12:37.448 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:12:37.448 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:12:37.449 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:12:37.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.443+0000 7f8a9a36b700 1 -- 172.21.15.70:0/3718776751 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8a7c0b6ff0 msgr2=0x7f8a7c0b94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:37.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.443+0000 7f8a9a36b700 1 --2- 172.21.15.70:0/3718776751 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8a7c0b6ff0 0x7f8a7c0b94b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8a94198ea0 tx=0x7f8a8800b500 comp rx=0 tx=0).stop 2024-06-04T05:12:37.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.443+0000 7f8a9a36b700 1 -- 172.21.15.70:0/3718776751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a94102ae0 msgr2=0x7f8a9419d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:37.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.443+0000 7f8a9a36b700 1 --2- 172.21.15.70:0/3718776751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a94102ae0 0x7f8a9419d610 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8a840098e0 tx=0x7f8a840045e0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.443+0000 7f8a9a36b700 1 -- 172.21.15.70:0/3718776751 shutdown_connections 2024-06-04T05:12:37.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.443+0000 7f8a9a36b700 1 --2- 172.21.15.70:0/3718776751 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8a7c0b6ff0 0x7f8a7c0b94b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.443+0000 7f8a9a36b700 1 --2- 172.21.15.70:0/3718776751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a94107a30 0x7f8a941976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.443+0000 7f8a9a36b700 1 --2- 172.21.15.70:0/3718776751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a94102ae0 0x7f8a9419d610 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.443+0000 7f8a9a36b700 1 --2- 172.21.15.70:0/3718776751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a94107070 0x7f8a9419db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:37.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.443+0000 7f8a9a36b700 1 -- 172.21.15.70:0/3718776751 >> 172.21.15.70:0/3718776751 conn(0x7f8a940793f0 msgr2=0x7f8a94100970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:37.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.443+0000 7f8a9a36b700 1 -- 172.21.15.70:0/3718776751 shutdown_connections 2024-06-04T05:12:37.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:37.443+0000 7f8a9a36b700 1 -- 172.21.15.70:0/3718776751 wait complete. 2024-06-04T05:12:37.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:37 smithi070 ceph-mon[32631]: pgmap v482: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.8 MiB/s wr, 435 op/s 2024-06-04T05:12:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:37 smithi112 ceph-mon[36453]: pgmap v482: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.8 MiB/s wr, 435 op/s 2024-06-04T05:12:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:37 smithi098 ceph-mon[35569]: pgmap v482: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.8 MiB/s wr, 435 op/s 2024-06-04T05:12:37.800 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 11.37 MB/sec warmup 89 sec latency 616.567 ms 2024-06-04T05:12:38.788 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'e648773', quiesce took 15.5 seconds. Will release after: 17.1 2024-06-04T05:12:38.800 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 11.24 MB/sec warmup 90 sec latency 1616.614 ms 2024-06-04T05:12:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:39 smithi070 ceph-mon[32631]: pgmap v483: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 311 op/s 2024-06-04T05:12:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:39 smithi112 ceph-mon[36453]: pgmap v483: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 311 op/s 2024-06-04T05:12:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:39 smithi098 ceph-mon[35569]: pgmap v483: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 311 op/s 2024-06-04T05:12:39.800 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 11.12 MB/sec warmup 91 sec latency 2616.696 ms 2024-06-04T05:12:40.800 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 11.00 MB/sec warmup 92 sec latency 3616.754 ms 2024-06-04T05:12:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:41 smithi070 ceph-mon[32631]: pgmap v484: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 311 op/s 2024-06-04T05:12:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:41 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:12:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:41 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:12:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:41 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:12:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:41 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:12:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:41 smithi112 ceph-mon[36453]: pgmap v484: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 311 op/s 2024-06-04T05:12:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:41 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:12:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:41 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:12:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:41 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:12:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:41 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:12:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:41 smithi098 ceph-mon[35569]: pgmap v484: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 311 op/s 2024-06-04T05:12:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:41 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:12:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:41 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:12:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:41 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:12:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:41 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:12:41.800 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 10.88 MB/sec warmup 93 sec latency 4616.853 ms 2024-06-04T05:12:42.800 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 10.77 MB/sec warmup 94 sec latency 5616.968 ms 2024-06-04T05:12:43.547 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:43 smithi098 ceph-mon[35569]: pgmap v485: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.2 MiB/s wr, 385 op/s 2024-06-04T05:12:43.547 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:43 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:12:43.548 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:43 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:12:43.548 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:43 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:12:43.548 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:43 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:12:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:43 smithi070 ceph-mon[32631]: pgmap v485: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.2 MiB/s wr, 385 op/s 2024-06-04T05:12:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:43 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:12:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:43 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:12:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:43 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:12:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:43 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:12:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:43 smithi112 ceph-mon[36453]: pgmap v485: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.2 MiB/s wr, 385 op/s 2024-06-04T05:12:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:43 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:12:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:43 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:12:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:43 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:12:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:43 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:12:43.800 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 10.65 MB/sec warmup 95 sec latency 6617.087 ms 2024-06-04T05:12:44.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:44 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:12:44.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:44 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:12:44.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:44 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:12:44.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:44 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:12:44.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:44 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:12:44.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:44 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:12:44.800 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 10.54 MB/sec warmup 96 sec latency 7617.171 ms 2024-06-04T05:12:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:45 smithi070 ceph-mon[32631]: pgmap v486: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 2.3 MiB/s wr, 223 op/s 2024-06-04T05:12:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:45 smithi112 ceph-mon[36453]: pgmap v486: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 2.3 MiB/s wr, 223 op/s 2024-06-04T05:12:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:45 smithi098 ceph-mon[35569]: pgmap v486: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 2.3 MiB/s wr, 223 op/s 2024-06-04T05:12:45.800 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 10.43 MB/sec warmup 97 sec latency 8617.329 ms 2024-06-04T05:12:46.801 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 10.33 MB/sec warmup 98 sec latency 9617.408 ms 2024-06-04T05:12:47.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:47 smithi070 ceph-mon[32631]: pgmap v487: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 2.5 MiB/s wr, 232 op/s 2024-06-04T05:12:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:47 smithi112 ceph-mon[36453]: pgmap v487: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 2.5 MiB/s wr, 232 op/s 2024-06-04T05:12:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:47 smithi098 ceph-mon[35569]: pgmap v487: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 2.5 MiB/s wr, 232 op/s 2024-06-04T05:12:47.801 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 10.22 MB/sec warmup 99 sec latency 10617.519 ms 2024-06-04T05:12:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.675+0000 7f65ed4d1640 1 -- 172.21.15.70:0/1708795430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e8102340 msgr2=0x7f65e810e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.675+0000 7f65ed4d1640 1 --2- 172.21.15.70:0/1708795430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e8102340 0x7f65e810e800 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f65d8009930 tx=0x7f65d802a420 comp rx=0 tx=0).stop 2024-06-04T05:12:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.676+0000 7f65ed4d1640 1 -- 172.21.15.70:0/1708795430 shutdown_connections 2024-06-04T05:12:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.676+0000 7f65ed4d1640 1 --2- 172.21.15.70:0/1708795430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e8102340 0x7f65e810e800 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.676+0000 7f65ed4d1640 1 --2- 172.21.15.70:0/1708795430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65e8101980 0x7f65e8101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.676+0000 7f65ed4d1640 1 --2- 172.21.15.70:0/1708795430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65e8107940 0x7f65e8107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.676+0000 7f65ed4d1640 1 -- 172.21.15.70:0/1708795430 >> 172.21.15.70:0/1708795430 conn(0x7f65e8078ad0 msgr2=0x7f65e8078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.676+0000 7f65ed4d1640 1 -- 172.21.15.70:0/1708795430 shutdown_connections 2024-06-04T05:12:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.676+0000 7f65ed4d1640 1 -- 172.21.15.70:0/1708795430 wait complete. 2024-06-04T05:12:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.676+0000 7f65ed4d1640 1 Processor -- start 2024-06-04T05:12:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.676+0000 7f65ed4d1640 1 -- start start 2024-06-04T05:12:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65ed4d1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65e8101980 0x7f65e810f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65ed4d1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e8102340 0x7f65e810f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65ed4d1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65e8107940 0x7f65e810fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65ed4d1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65e8112bb0 con 0x7f65e8102340 2024-06-04T05:12:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65ed4d1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f65e8112a30 con 0x7f65e8101980 2024-06-04T05:12:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65ed4d1640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f65e8112d30 con 0x7f65e8107940 2024-06-04T05:12:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65e67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e8102340 0x7f65e810f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65e67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e8102340 0x7f65e810f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57456/0 (socket says 172.21.15.70:57456) 2024-06-04T05:12:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65e67fc640 1 -- 172.21.15.70:0/1969067302 learned_addr learned my addr 172.21.15.70:0/1969067302 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65e6ffd640 1 --2- 172.21.15.70:0/1969067302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65e8101980 0x7f65e810f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65e77fe640 1 --2- 172.21.15.70:0/1969067302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65e8107940 0x7f65e810fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65e67fc640 1 -- 172.21.15.70:0/1969067302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65e8107940 msgr2=0x7f65e810fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65e67fc640 1 --2- 172.21.15.70:0/1969067302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65e8107940 0x7f65e810fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65e67fc640 1 -- 172.21.15.70:0/1969067302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65e8101980 msgr2=0x7f65e810f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65e67fc640 1 --2- 172.21.15.70:0/1969067302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65e8101980 0x7f65e810f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65e67fc640 1 -- 172.21.15.70:0/1969067302 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e81a7370 con 0x7f65e8102340 2024-06-04T05:12:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65e67fc640 1 --2- 172.21.15.70:0/1969067302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e8102340 0x7f65e810f7d0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f65d000d770 tx=0x7f65d000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.677+0000 7f65dffff640 1 -- 172.21.15.70:0/1969067302 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d0017070 con 0x7f65e8102340 2024-06-04T05:12:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.678+0000 7f65dffff640 1 -- 172.21.15.70:0/1969067302 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65d00054b0 con 0x7f65e8102340 2024-06-04T05:12:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.678+0000 7f65dffff640 1 -- 172.21.15.70:0/1969067302 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d0004c60 con 0x7f65e8102340 2024-06-04T05:12:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.678+0000 7f65ed4d1640 1 -- 172.21.15.70:0/1969067302 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65e81a7660 con 0x7f65e8102340 2024-06-04T05:12:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.678+0000 7f65ed4d1640 1 -- 172.21.15.70:0/1969067302 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65e81a7b60 con 0x7f65e8102340 2024-06-04T05:12:48.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.680+0000 7f65dffff640 1 -- 172.21.15.70:0/1969067302 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f65d0004e00 con 0x7f65e8102340 2024-06-04T05:12:48.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.680+0000 7f65f01ed740 1 Processor -- start 2024-06-04T05:12:48.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65f01ed740 1 -- start start 2024-06-04T05:12:48.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65f01ed740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966112f820 0x55966112fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65f01ed740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559661130140 0x55966113a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65f01ed740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55966113aa00 0x55966113cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65f01ed740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5596610a1c30 con 0x55966112f820 2024-06-04T05:12:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65f01ed740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5596610917d0 con 0x559661130140 2024-06-04T05:12:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65f01ed740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559661095810 con 0x55966113aa00 2024-06-04T05:12:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65e77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966112f820 0x55966112fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65e77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966112f820 0x55966112fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57462/0 (socket says 172.21.15.70:57462) 2024-06-04T05:12:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65e77fe640 1 -- 172.21.15.70:0/604914755 learned_addr learned my addr 172.21.15.70:0/604914755 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65e6ffd640 1 --2- 172.21.15.70:0/604914755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559661130140 0x55966113a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65dffff640 1 --2- 172.21.15.70:0/1969067302 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65d40b2a40 0x7f65d40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65e67fc640 1 --2- 172.21.15.70:0/604914755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55966113aa00 0x55966113cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65e77fe640 1 -- 172.21.15.70:0/604914755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55966113aa00 msgr2=0x55966113cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65e77fe640 1 --2- 172.21.15.70:0/604914755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55966113aa00 0x55966113cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65e77fe640 1 -- 172.21.15.70:0/604914755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559661130140 msgr2=0x55966113a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65e77fe640 1 --2- 172.21.15.70:0/604914755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559661130140 0x55966113a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.681+0000 7f65e77fe640 1 -- 172.21.15.70:0/604914755 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559660da6210 con 0x55966112f820 2024-06-04T05:12:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.682+0000 7f65e77fe640 1 --2- 172.21.15.70:0/1969067302 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65d40b2a40 0x7f65d40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.682+0000 7f65dffff640 1 -- 172.21.15.70:0/1969067302 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f65d00e2060 con 0x7f65e8102340 2024-06-04T05:12:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.682+0000 7f65e77fe640 1 --2- 172.21.15.70:0/604914755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966112f820 0x55966112fc00 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f65d8005ab0 tx=0x7f65d8032cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb0d9686061ac8a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.682+0000 7f65dd7fa640 1 -- 172.21.15.70:0/604914755 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d8035040 con 0x55966112f820 2024-06-04T05:12:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.682+0000 7f65dd7fa640 1 -- 172.21.15.70:0/604914755 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65d80054d0 con 0x55966112f820 2024-06-04T05:12:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.682+0000 7f65dd7fa640 1 -- 172.21.15.70:0/604914755 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d8004b30 con 0x55966112f820 2024-06-04T05:12:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.683+0000 7f65f01ed740 1 -- 172.21.15.70:0/604914755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966112f820 msgr2=0x55966112fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.683+0000 7f65e77fe640 1 --2- 172.21.15.70:0/1969067302 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65d40b2a40 0x7f65d40b4f00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f65d8048000 tx=0x7f65d8049000 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.683+0000 7f65f01ed740 1 --2- 172.21.15.70:0/604914755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966112f820 0x55966112fc00 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f65d8005ab0 tx=0x7f65d8032cb0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.683+0000 7f65f01ed740 1 -- 172.21.15.70:0/604914755 shutdown_connections 2024-06-04T05:12:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.683+0000 7f65f01ed740 1 --2- 172.21.15.70:0/604914755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559661130140 0x55966113a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.683+0000 7f65f01ed740 1 --2- 172.21.15.70:0/604914755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55966113aa00 0x55966113cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.683+0000 7f65f01ed740 1 --2- 172.21.15.70:0/604914755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966112f820 0x55966112fc00 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.683+0000 7f65f01ed740 1 -- 172.21.15.70:0/604914755 >> 172.21.15.70:0/604914755 conn(0x55966112bd30 msgr2=0x55966112c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:48.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.683+0000 7f65f01ed740 1 -- 172.21.15.70:0/604914755 shutdown_connections 2024-06-04T05:12:48.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.683+0000 7f65f01ed740 1 -- 172.21.15.70:0/604914755 wait complete. 2024-06-04T05:12:48.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.683+0000 7f65f01ed740 1 -- start start 2024-06-04T05:12:48.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.683+0000 7f65f01ed740 1 Processor -- start 2024-06-04T05:12:48.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65f01ed740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966112f820 0x5596611379c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65f01ed740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559661130140 0x559661137f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65f01ed740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55966113aa00 0x559661138440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65f01ed740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559660d77790 con 0x55966112f820 2024-06-04T05:12:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65f01ed740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55966104d890 con 0x559661130140 2024-06-04T05:12:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65f01ed740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55966113d5d0 con 0x55966113aa00 2024-06-04T05:12:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65f01ed740 10 client.0 _ll_register_callbacks cb 0x559660da5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65e6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966112f820 0x5596611379c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65e6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966112f820 0x5596611379c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57472/0 (socket says 172.21.15.70:57472) 2024-06-04T05:12:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65e6ffd640 1 -- 172.21.15.70:0/3380736217 learned_addr learned my addr 172.21.15.70:0/3380736217 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65e67fc640 1 --2- 172.21.15.70:0/3380736217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559661130140 0x559661137f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65e77fe640 1 --2- 172.21.15.70:0/3380736217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55966113aa00 0x559661138440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65e6ffd640 1 -- 172.21.15.70:0/3380736217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55966113aa00 msgr2=0x559661138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65e6ffd640 1 --2- 172.21.15.70:0/3380736217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55966113aa00 0x559661138440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65e6ffd640 1 -- 172.21.15.70:0/3380736217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559661130140 msgr2=0x559661137f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65e6ffd640 1 --2- 172.21.15.70:0/3380736217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559661130140 0x559661137f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65e6ffd640 1 -- 172.21.15.70:0/3380736217 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559660e72620 con 0x55966112f820 2024-06-04T05:12:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65e6ffd640 1 --2- 172.21.15.70:0/3380736217 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966112f820 0x5596611379c0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f65e0010b70 tx=0x7f65e0010f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe2a50cc9f98191b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65ce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:12:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65ce7fc640 1 -- 172.21.15.70:0/3380736217 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e000fe30 con 0x55966112f820 2024-06-04T05:12:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.684+0000 7f65ce7fc640 1 -- 172.21.15.70:0/3380736217 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f65e0004b20 con 0x55966112f820 2024-06-04T05:12:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.685+0000 7f65f01ed740 1 -- 172.21.15.70:0/3380736217 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65d80054d0 con 0x55966112f820 2024-06-04T05:12:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.685+0000 7f65ce7fc640 1 -- 172.21.15.70:0/3380736217 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f65e0004d20 con 0x55966112f820 2024-06-04T05:12:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.685+0000 7f65f01ed740 10 client.19542 fetch_fsmap learned FSMap version 45 2024-06-04T05:12:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.685+0000 7f65f01ed740 1 -- 172.21.15.70:0/3380736217 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5596611493b0 con 0x55966112f820 2024-06-04T05:12:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.685+0000 7f65ce7fc640 1 -- 172.21.15.70:0/3380736217 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65d80054d0 con 0x55966112f820 2024-06-04T05:12:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.685+0000 7f65ce7fc640 1 -- 172.21.15.70:0/3380736217 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 45) ==== 8398+0+0 (secure 0 0 0) 0x7f65e000e3b0 con 0x55966112f820 2024-06-04T05:12:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.685+0000 7f65f01ed740 10 client.19542 fetch_fsmap finished waiting for FSMap version 45 2024-06-04T05:12:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.685+0000 7f65f01ed740 10 client.19542 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:12:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.685+0000 7f65f01ed740 20 client.19542 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.685+0000 7f65f01ed740 1 --2- 172.21.15.70:0/3380736217 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55966114a930 0x55966114cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.685+0000 7f65f01ed740 4 client.19542 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:12:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.685+0000 7f65f01ed740 1 -- 172.21.15.70:0/3380736217 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596610a3b80 con 0x55966114a930 2024-06-04T05:12:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.685+0000 7f65e67fc640 1 --2- 172.21.15.70:0/3380736217 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55966114a930 0x55966114cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.686+0000 7f65ce7fc640 10 client.19542 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.686+0000 7f65e67fc640 1 --2- 172.21.15.70:0/3380736217 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55966114a930 0x55966114cdc0 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eef6ed256ec1152 server_cookie=49da18b473b0d407 in_seq=0 out_seq=0 2024-06-04T05:12:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.686+0000 7f65ce7fc640 1 -- 172.21.15.70:0/3380736217 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5596610a3b80 con 0x55966114a930 2024-06-04T05:12:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.686+0000 7f65ce7fc640 10 client.19542 handle_command_reply: tid=0 2024-06-04T05:12:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.687+0000 7f65f01ed740 1 client.19542 shutdown 2024-06-04T05:12:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.687+0000 7f65f01ed740 1 -- 172.21.15.70:0/3380736217 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966112f820 msgr2=0x5596611379c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.687+0000 7f65f01ed740 1 --2- 172.21.15.70:0/3380736217 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966112f820 0x5596611379c0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f65e0010b70 tx=0x7f65e0010f70 comp rx=0 tx=0).stop 2024-06-04T05:12:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.687+0000 7f65f01ed740 1 -- 172.21.15.70:0/3380736217 shutdown_connections 2024-06-04T05:12:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.687+0000 7f65f01ed740 1 --2- 172.21.15.70:0/3380736217 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55966114a930 0x55966114cdc0 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.687+0000 7f65f01ed740 1 --2- 172.21.15.70:0/3380736217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559661130140 0x559661137f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.687+0000 7f65f01ed740 1 --2- 172.21.15.70:0/3380736217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55966113aa00 0x559661138440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.687+0000 7f65f01ed740 1 --2- 172.21.15.70:0/3380736217 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966112f820 0x5596611379c0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.687+0000 7f65f01ed740 1 -- 172.21.15.70:0/3380736217 >> 172.21.15.70:0/3380736217 conn(0x55966112bd30 msgr2=0x55966112afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.687+0000 7f65ce7fc640 0 client.19542 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.687+0000 7f65f01ed740 1 -- 172.21.15.70:0/3380736217 shutdown_connections 2024-06-04T05:12:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.687+0000 7f65f01ed740 1 -- 172.21.15.70:0/3380736217 wait complete. 2024-06-04T05:12:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.687+0000 7f65f01ed740 20 client.19542 trim_cache size 0 max 16384 2024-06-04T05:12:48.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.702+0000 7f65f01ed740 1 Processor -- start 2024-06-04T05:12:48.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.702+0000 7f65f01ed740 1 -- start start 2024-06-04T05:12:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65f01ed740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55966112f820 0x559661136290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65f01ed740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559661130140 0x559661137f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65f01ed740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55966113aa00 0x559661138450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65f01ed740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65e81b0680 con 0x559661130140 2024-06-04T05:12:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65f01ed740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f65e81b2310 con 0x55966113aa00 2024-06-04T05:12:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65f01ed740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f65e81b0d50 con 0x55966112f820 2024-06-04T05:12:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65e67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559661130140 0x559661137f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65e67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559661130140 0x559661137f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57488/0 (socket says 172.21.15.70:57488) 2024-06-04T05:12:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65e67fc640 1 -- 172.21.15.70:0/3195928002 learned_addr learned my addr 172.21.15.70:0/3195928002 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65e67fc640 1 -- 172.21.15.70:0/3195928002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55966112f820 msgr2=0x559661136290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65e67fc640 1 --2- 172.21.15.70:0/3195928002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55966112f820 0x559661136290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65e67fc640 1 -- 172.21.15.70:0/3195928002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55966113aa00 msgr2=0x559661138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65e67fc640 1 --2- 172.21.15.70:0/3195928002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55966113aa00 0x559661138450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65e67fc640 1 -- 172.21.15.70:0/3195928002 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559661136970 con 0x559661130140 2024-06-04T05:12:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65e67fc640 1 --2- 172.21.15.70:0/3195928002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559661130140 0x559661137f10 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f65d00df7e0 tx=0x7f65d00c5040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa56fd48e0d1728 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65dcff9640 1 -- 172.21.15.70:0/3195928002 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d00c3070 con 0x559661130140 2024-06-04T05:12:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65dcff9640 1 -- 172.21.15.70:0/3195928002 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65d00a9310 con 0x559661130140 2024-06-04T05:12:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.703+0000 7f65dcff9640 1 -- 172.21.15.70:0/3195928002 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d0020040 con 0x559661130140 2024-06-04T05:12:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.704+0000 7f65f01ed740 1 -- 172.21.15.70:0/3195928002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559661130140 msgr2=0x559661137f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.704+0000 7f65f01ed740 1 --2- 172.21.15.70:0/3195928002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559661130140 0x559661137f10 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f65d00df7e0 tx=0x7f65d00c5040 comp rx=0 tx=0).stop 2024-06-04T05:12:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.704+0000 7f65f01ed740 1 -- 172.21.15.70:0/3195928002 shutdown_connections 2024-06-04T05:12:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.704+0000 7f65f01ed740 1 --2- 172.21.15.70:0/3195928002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55966113aa00 0x559661138450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.704+0000 7f65f01ed740 1 --2- 172.21.15.70:0/3195928002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559661130140 0x559661137f10 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.704+0000 7f65f01ed740 1 --2- 172.21.15.70:0/3195928002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55966112f820 0x559661136290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.704+0000 7f65f01ed740 1 -- 172.21.15.70:0/3195928002 >> 172.21.15.70:0/3195928002 conn(0x55966112bd30 msgr2=0x559661131a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.704+0000 7f65f01ed740 1 -- 172.21.15.70:0/3195928002 shutdown_connections 2024-06-04T05:12:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.704+0000 7f65f01ed740 1 -- 172.21.15.70:0/3195928002 wait complete. 2024-06-04T05:12:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.704+0000 7f65f01ed740 1 -- start start 2024-06-04T05:12:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.704+0000 7f65f01ed740 1 Processor -- start 2024-06-04T05:12:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65f01ed740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55966112f820 0x559661142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65f01ed740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559661130140 0x559661143030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65f01ed740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966113aa00 0x559661134360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65f01ed740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65e81b1b80 con 0x55966113aa00 2024-06-04T05:12:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65f01ed740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f65e81b1480 con 0x55966112f820 2024-06-04T05:12:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65f01ed740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f65e81ae680 con 0x559661130140 2024-06-04T05:12:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65f01ed740 10 client.0 _ll_register_callbacks cb 0x7f65e000aa70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:12:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65e77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966113aa00 0x559661134360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65e6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55966112f820 0x559661142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65e77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966113aa00 0x559661134360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57500/0 (socket says 172.21.15.70:57500) 2024-06-04T05:12:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65e77fe640 1 -- 172.21.15.70:0/707078249 learned_addr learned my addr 172.21.15.70:0/707078249 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65e67fc640 1 --2- 172.21.15.70:0/707078249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559661130140 0x559661143030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65e77fe640 1 -- 172.21.15.70:0/707078249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559661130140 msgr2=0x559661143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65e77fe640 1 --2- 172.21.15.70:0/707078249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559661130140 0x559661143030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65e77fe640 1 -- 172.21.15.70:0/707078249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55966112f820 msgr2=0x559661142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:12:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65e77fe640 1 --2- 172.21.15.70:0/707078249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55966112f820 0x559661142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65e77fe640 1 -- 172.21.15.70:0/707078249 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559661134b30 con 0x55966113aa00 2024-06-04T05:12:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65e67fc640 1 --2- 172.21.15.70:0/707078249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559661130140 0x559661143030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:12:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65e6ffd640 1 --2- 172.21.15.70:0/707078249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55966112f820 0x559661142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:12:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65e77fe640 1 --2- 172.21.15.70:0/707078249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966113aa00 0x559661134360 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f65d8005ab0 tx=0x7f65d80318e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f394487f4a3b23 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65ce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:12:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.705+0000 7f65ce7fc640 1 -- 172.21.15.70:0/707078249 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d8060060 con 0x55966113aa00 2024-06-04T05:12:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.706+0000 7f65ce7fc640 1 -- 172.21.15.70:0/707078249 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f65d8004ad0 con 0x55966113aa00 2024-06-04T05:12:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.706+0000 7f65f01ed740 1 -- 172.21.15.70:0/707078249 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65d00a9310 con 0x55966113aa00 2024-06-04T05:12:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.706+0000 7f65ce7fc640 1 -- 172.21.15.70:0/707078249 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f65d8035070 con 0x55966113aa00 2024-06-04T05:12:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.706+0000 7f65f01ed740 10 client.19548 fetch_fsmap learned FSMap version 45 2024-06-04T05:12:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.706+0000 7f65f01ed740 1 -- 172.21.15.70:0/707078249 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5596611b5130 con 0x55966113aa00 2024-06-04T05:12:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.706+0000 7f65ce7fc640 1 -- 172.21.15.70:0/707078249 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65d00a9310 con 0x55966113aa00 2024-06-04T05:12:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.706+0000 7f65ce7fc640 1 -- 172.21.15.70:0/707078249 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 45) ==== 8398+0+0 (secure 0 0 0) 0x7f65d8045070 con 0x55966113aa00 2024-06-04T05:12:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.706+0000 7f65f01ed740 10 client.19548 fetch_fsmap finished waiting for FSMap version 45 2024-06-04T05:12:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.706+0000 7f65f01ed740 10 client.19548 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:12:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.706+0000 7f65f01ed740 20 client.19548 populate_metadata read hostname 'smithi070' 2024-06-04T05:12:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.706+0000 7f65f01ed740 1 --2- 172.21.15.70:0/707078249 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5596611b5520 0x5596611b7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.706+0000 7f65f01ed740 4 client.19548 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:12:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.706+0000 7f65f01ed740 1 -- 172.21.15.70:0/707078249 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55966109d360 con 0x5596611b5520 2024-06-04T05:12:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.706+0000 7f65e6ffd640 1 --2- 172.21.15.70:0/707078249 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5596611b5520 0x5596611b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.707+0000 7f65ce7fc640 10 client.19548 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.707+0000 7f65e6ffd640 1 --2- 172.21.15.70:0/707078249 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5596611b5520 0x5596611b7910 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdf8ae552b1bfbdd server_cookie=34bc9cc3c2be560d in_seq=0 out_seq=0 2024-06-04T05:12:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.740+0000 7f65ce7fc640 1 -- 172.21.15.70:0/707078249 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55966109d360 con 0x5596611b5520 2024-06-04T05:12:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.740+0000 7f65ce7fc640 10 client.19548 handle_command_reply: tid=0 2024-06-04T05:12:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.740+0000 7f65f01ed740 1 client.19548 shutdown 2024-06-04T05:12:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.740+0000 7f65f01ed740 1 -- 172.21.15.70:0/707078249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966113aa00 msgr2=0x559661134360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.740+0000 7f65f01ed740 1 --2- 172.21.15.70:0/707078249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966113aa00 0x559661134360 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f65d8005ab0 tx=0x7f65d80318e0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.741+0000 7f65f01ed740 1 -- 172.21.15.70:0/707078249 shutdown_connections 2024-06-04T05:12:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.741+0000 7f65f01ed740 1 --2- 172.21.15.70:0/707078249 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5596611b5520 0x5596611b7910 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.741+0000 7f65ce7fc640 0 client.19548 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:12:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.741+0000 7f65f01ed740 1 --2- 172.21.15.70:0/707078249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55966113aa00 0x559661134360 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.741+0000 7f65f01ed740 1 --2- 172.21.15.70:0/707078249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559661130140 0x559661143030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.741+0000 7f65f01ed740 1 --2- 172.21.15.70:0/707078249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55966112f820 0x559661142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.741+0000 7f65f01ed740 1 -- 172.21.15.70:0/707078249 >> 172.21.15.70:0/707078249 conn(0x55966112bd30 msgr2=0x559661131a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.741+0000 7f65f01ed740 1 -- 172.21.15.70:0/707078249 shutdown_connections 2024-06-04T05:12:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.741+0000 7f65f01ed740 1 -- 172.21.15.70:0/707078249 wait complete. 2024-06-04T05:12:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.741+0000 7f65f01ed740 20 client.19548 trim_cache size 0 max 16384 2024-06-04T05:12:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:12:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:12:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:12:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.741+0000 7f65ed4d1640 1 -- 172.21.15.70:0/1969067302 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65d40b2a40 msgr2=0x7f65d40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.741+0000 7f65ed4d1640 1 --2- 172.21.15.70:0/1969067302 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65d40b2a40 0x7f65d40b4f00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f65d8048000 tx=0x7f65d8049000 comp rx=0 tx=0).stop 2024-06-04T05:12:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.741+0000 7f65ed4d1640 1 -- 172.21.15.70:0/1969067302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e8102340 msgr2=0x7f65e810f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.741+0000 7f65ed4d1640 1 --2- 172.21.15.70:0/1969067302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e8102340 0x7f65e810f7d0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f65d000d770 tx=0x7f65d000dc30 comp rx=0 tx=0).stop 2024-06-04T05:12:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.742+0000 7f65ed4d1640 1 -- 172.21.15.70:0/1969067302 shutdown_connections 2024-06-04T05:12:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.742+0000 7f65ed4d1640 1 --2- 172.21.15.70:0/1969067302 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65d40b2a40 0x7f65d40b4f00 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.742+0000 7f65ed4d1640 1 --2- 172.21.15.70:0/1969067302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65e8107940 0x7f65e810fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.742+0000 7f65ed4d1640 1 --2- 172.21.15.70:0/1969067302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e8102340 0x7f65e810f7d0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.742+0000 7f65ed4d1640 1 --2- 172.21.15.70:0/1969067302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65e8101980 0x7f65e810f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.742+0000 7f65ed4d1640 1 -- 172.21.15.70:0/1969067302 >> 172.21.15.70:0/1969067302 conn(0x7f65e8078ad0 msgr2=0x7f65e8104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.742+0000 7f65ed4d1640 1 -- 172.21.15.70:0/1969067302 shutdown_connections 2024-06-04T05:12:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:12:48.742+0000 7f65ed4d1640 1 -- 172.21.15.70:0/1969067302 wait complete. 2024-06-04T05:12:48.801 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 10.12 MB/sec warmup 100 sec latency 11617.574 ms 2024-06-04T05:12:49.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:49 smithi070 ceph-mon[32631]: pgmap v488: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 699 KiB/s wr, 83 op/s 2024-06-04T05:12:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:49 smithi112 ceph-mon[36453]: pgmap v488: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 699 KiB/s wr, 83 op/s 2024-06-04T05:12:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:49 smithi098 ceph-mon[35569]: pgmap v488: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 699 KiB/s wr, 83 op/s 2024-06-04T05:12:49.801 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 10.02 MB/sec warmup 101 sec latency 12617.696 ms 2024-06-04T05:12:50.801 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 9.92 MB/sec warmup 102 sec latency 13617.744 ms 2024-06-04T05:12:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:51 smithi070 ceph-mon[32631]: pgmap v489: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 699 KiB/s wr, 83 op/s 2024-06-04T05:12:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:51 smithi112 ceph-mon[36453]: pgmap v489: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 699 KiB/s wr, 83 op/s 2024-06-04T05:12:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:51 smithi098 ceph-mon[35569]: pgmap v489: 97 pgs: 97 active+clean; 427 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 699 KiB/s wr, 83 op/s 2024-06-04T05:12:51.801 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 9.82 MB/sec warmup 103 sec latency 14617.815 ms 2024-06-04T05:12:52.801 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 9.73 MB/sec warmup 104 sec latency 15617.876 ms 2024-06-04T05:12:53.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:53 smithi070 ceph-mon[32631]: pgmap v490: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 727 KiB/s wr, 98 op/s 2024-06-04T05:12:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:53 smithi112 ceph-mon[36453]: pgmap v490: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 727 KiB/s wr, 98 op/s 2024-06-04T05:12:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:53 smithi098 ceph-mon[35569]: pgmap v490: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 727 KiB/s wr, 98 op/s 2024-06-04T05:12:53.801 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 9.64 MB/sec warmup 105 sec latency 16617.964 ms 2024-06-04T05:12:54.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 9.55 MB/sec warmup 106 sec latency 17618.058 ms 2024-06-04T05:12:55.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:55 smithi070 ceph-mon[32631]: pgmap v491: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 272 KiB/s wr, 24 op/s 2024-06-04T05:12:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:55 smithi112 ceph-mon[36453]: pgmap v491: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 272 KiB/s wr, 24 op/s 2024-06-04T05:12:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:55 smithi098 ceph-mon[35569]: pgmap v491: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 272 KiB/s wr, 24 op/s 2024-06-04T05:12:55.801 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 9.46 MB/sec warmup 107 sec latency 18618.158 ms 2024-06-04T05:12:55.888 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:12:56.145 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:12:56.801 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 9.37 MB/sec warmup 108 sec latency 19618.233 ms 2024-06-04T05:12:57.567 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:57 smithi070 ceph-mon[32631]: pgmap v492: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 276 KiB/s wr, 29 op/s 2024-06-04T05:12:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:57 smithi112 ceph-mon[36453]: pgmap v492: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 276 KiB/s wr, 29 op/s 2024-06-04T05:12:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:57 smithi098 ceph-mon[35569]: pgmap v492: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 276 KiB/s wr, 29 op/s 2024-06-04T05:12:57.801 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 9.28 MB/sec warmup 109 sec latency 20618.306 ms 2024-06-04T05:12:58.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.128+0000 7f27b091b700 1 -- 172.21.15.70:0/3610887693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27ac103600 msgr2=0x7f27ac103a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:58.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.128+0000 7f27b091b700 1 --2- 172.21.15.70:0/3610887693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27ac103600 0x7f27ac103a80 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f27a00099c0 tx=0x7f27a0009cd0 comp rx=0 tx=0).stop 2024-06-04T05:12:58.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.128+0000 7f27b091b700 1 -- 172.21.15.70:0/3610887693 shutdown_connections 2024-06-04T05:12:58.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.128+0000 7f27b091b700 1 --2- 172.21.15.70:0/3610887693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27ac103fc0 0x7f27ac1103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:58.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.128+0000 7f27b091b700 1 --2- 172.21.15.70:0/3610887693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27ac1095e0 0x7f27ac1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:58.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.128+0000 7f27b091b700 1 --2- 172.21.15.70:0/3610887693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27ac103600 0x7f27ac103a80 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:58.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.128+0000 7f27b091b700 1 -- 172.21.15.70:0/3610887693 >> 172.21.15.70:0/3610887693 conn(0x7f27ac07a700 msgr2=0x7f27ac07ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:58.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.128+0000 7f27b091b700 1 -- 172.21.15.70:0/3610887693 shutdown_connections 2024-06-04T05:12:58.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.128+0000 7f27b091b700 1 -- 172.21.15.70:0/3610887693 wait complete. 2024-06-04T05:12:58.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.129+0000 7f27b091b700 1 Processor -- start 2024-06-04T05:12:58.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.129+0000 7f27b091b700 1 -- start start 2024-06-04T05:12:58.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.129+0000 7f27b091b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27ac103600 0x7f27ac19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:58.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.129+0000 7f27b091b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27ac103fc0 0x7f27ac19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:58.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.129+0000 7f27b091b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27ac1095e0 0x7f27ac1998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:58.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.129+0000 7f27b091b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f27ac1135f0 con 0x7f27ac103fc0 2024-06-04T05:12:58.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.129+0000 7f27b091b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f27ac113470 con 0x7f27ac1095e0 2024-06-04T05:12:58.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.129+0000 7f27b091b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f27ac113770 con 0x7f27ac103600 2024-06-04T05:12:58.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.129+0000 7f27a9d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27ac103fc0 0x7f27ac19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:58.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.129+0000 7f27a9d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27ac103fc0 0x7f27ac19fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51034/0 (socket says 172.21.15.70:51034) 2024-06-04T05:12:58.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.129+0000 7f27a9d9b700 1 -- 172.21.15.70:0/312916915 learned_addr learned my addr 172.21.15.70:0/312916915 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:12:58.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.129+0000 7f27aad9d700 1 --2- 172.21.15.70:0/312916915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27ac1095e0 0x7f27ac1998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:58.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.129+0000 7f27aa59c700 1 --2- 172.21.15.70:0/312916915 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27ac103600 0x7f27ac19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:58.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.130+0000 7f27a9d9b700 1 -- 172.21.15.70:0/312916915 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27ac103600 msgr2=0x7f27ac19f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:58.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.130+0000 7f27a9d9b700 1 --2- 172.21.15.70:0/312916915 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27ac103600 0x7f27ac19f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:58.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.130+0000 7f27a9d9b700 1 -- 172.21.15.70:0/312916915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27ac1095e0 msgr2=0x7f27ac1998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:58.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.130+0000 7f27a9d9b700 1 --2- 172.21.15.70:0/312916915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27ac1095e0 0x7f27ac1998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:58.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.130+0000 7f27a9d9b700 1 -- 172.21.15.70:0/312916915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27ac19a050 con 0x7f27ac103fc0 2024-06-04T05:12:58.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.130+0000 7f27aa59c700 1 --2- 172.21.15.70:0/312916915 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27ac103600 0x7f27ac19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:12:58.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.130+0000 7f27a9d9b700 1 --2- 172.21.15.70:0/312916915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27ac103fc0 0x7f27ac19fc70 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f27a0000c00 tx=0x7f27a00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:58.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.130+0000 7f279b7fe700 1 -- 172.21.15.70:0/312916915 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a000f8a0 con 0x7f27ac103fc0 2024-06-04T05:12:58.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.130+0000 7f279b7fe700 1 -- 172.21.15.70:0/312916915 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f27a0004830 con 0x7f27ac103fc0 2024-06-04T05:12:58.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.130+0000 7f27b091b700 1 -- 172.21.15.70:0/312916915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27ac19a2e0 con 0x7f27ac103fc0 2024-06-04T05:12:58.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.130+0000 7f279b7fe700 1 -- 172.21.15.70:0/312916915 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a0005410 con 0x7f27ac103fc0 2024-06-04T05:12:58.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.130+0000 7f27b091b700 1 -- 172.21.15.70:0/312916915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27ac1a6600 con 0x7f27ac103fc0 2024-06-04T05:12:58.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.131+0000 7f27b091b700 1 -- 172.21.15.70:0/312916915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27ac06a230 con 0x7f27ac103fc0 2024-06-04T05:12:58.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.133+0000 7f279b7fe700 1 -- 172.21.15.70:0/312916915 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f27a0029060 con 0x7f27ac103fc0 2024-06-04T05:12:58.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.134+0000 7f279b7fe700 1 --2- 172.21.15.70:0/312916915 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f27940b2a70 0x7f27940b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:12:58.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.134+0000 7f27aa59c700 1 --2- 172.21.15.70:0/312916915 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f27940b2a70 0x7f27940b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:12:58.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.134+0000 7f279b7fe700 1 -- 172.21.15.70:0/312916915 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f27a00e3300 con 0x7f27ac103fc0 2024-06-04T05:12:58.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.134+0000 7f27aa59c700 1 --2- 172.21.15.70:0/312916915 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f27940b2a70 0x7f27940b4f30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f279c005fd0 tx=0x7f279c005d60 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:12:58.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.137+0000 7f279b7fe700 1 -- 172.21.15.70:0/312916915 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f27a0013070 con 0x7f27ac103fc0 2024-06-04T05:12:58.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.391+0000 7f27b091b700 1 -- 172.21.15.70:0/312916915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27ac02e560 con 0x7f27ac103fc0 2024-06-04T05:12:58.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.392+0000 7f279b7fe700 1 -- 172.21.15.70:0/312916915 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11947 (secure 0 0 0) 0x7f27a00aace0 con 0x7f27ac103fc0 2024-06-04T05:12:58.395 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:12:58.395 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":45,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:12:42.042199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T05:12:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.394+0000 7f27b091b700 1 -- 172.21.15.70:0/312916915 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f27940b2a70 msgr2=0x7f27940b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.394+0000 7f27b091b700 1 --2- 172.21.15.70:0/312916915 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f27940b2a70 0x7f27940b4f30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f279c005fd0 tx=0x7f279c005d60 comp rx=0 tx=0).stop 2024-06-04T05:12:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.394+0000 7f27b091b700 1 -- 172.21.15.70:0/312916915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27ac103fc0 msgr2=0x7f27ac19fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:12:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.394+0000 7f27b091b700 1 --2- 172.21.15.70:0/312916915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27ac103fc0 0x7f27ac19fc70 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f27a0000c00 tx=0x7f27a00047b0 comp rx=0 tx=0).stop 2024-06-04T05:12:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.394+0000 7f27b091b700 1 -- 172.21.15.70:0/312916915 shutdown_connections 2024-06-04T05:12:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.394+0000 7f27b091b700 1 --2- 172.21.15.70:0/312916915 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f27940b2a70 0x7f27940b4f30 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.394+0000 7f27b091b700 1 --2- 172.21.15.70:0/312916915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27ac1095e0 0x7f27ac1998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.394+0000 7f27b091b700 1 --2- 172.21.15.70:0/312916915 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27ac103600 0x7f27ac19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.394+0000 7f27b091b700 1 --2- 172.21.15.70:0/312916915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27ac103fc0 0x7f27ac19fc70 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:12:58.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.394+0000 7f27b091b700 1 -- 172.21.15.70:0/312916915 >> 172.21.15.70:0/312916915 conn(0x7f27ac07a700 msgr2=0x7f27ac1023a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:12:58.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.395+0000 7f27b091b700 1 -- 172.21.15.70:0/312916915 shutdown_connections 2024-06-04T05:12:58.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:12:58.395+0000 7f27b091b700 1 -- 172.21.15.70:0/312916915 wait complete. 2024-06-04T05:12:58.399 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 45 2024-06-04T05:12:58.801 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 9.20 MB/sec warmup 110 sec latency 21618.394 ms 2024-06-04T05:12:59.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:59 smithi070 ceph-mon[32631]: pgmap v493: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 32 KiB/s wr, 20 op/s 2024-06-04T05:12:59.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:12:59 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/312916915' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:12:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:59 smithi112 ceph-mon[36453]: pgmap v493: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 32 KiB/s wr, 20 op/s 2024-06-04T05:12:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:12:59 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/312916915' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:12:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:59 smithi098 ceph-mon[35569]: pgmap v493: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 32 KiB/s wr, 20 op/s 2024-06-04T05:12:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:12:59 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/312916915' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:12:59.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 9.12 MB/sec warmup 111 sec latency 22618.514 ms 2024-06-04T05:13:00.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 9.03 MB/sec warmup 112 sec latency 23618.562 ms 2024-06-04T05:13:01.488 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:01 smithi112 ceph-mon[36453]: pgmap v494: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 32 KiB/s wr, 20 op/s 2024-06-04T05:13:01.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:01 smithi070 ceph-mon[32631]: pgmap v494: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 32 KiB/s wr, 20 op/s 2024-06-04T05:13:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:01 smithi098 ceph-mon[35569]: pgmap v494: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 32 KiB/s wr, 20 op/s 2024-06-04T05:13:01.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 8.95 MB/sec warmup 113 sec latency 24618.597 ms 2024-06-04T05:13:01.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.839+0000 7f951172b640 1 -- 172.21.15.70:0/4203123669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f950c102340 msgr2=0x7f950c10e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:01.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.839+0000 7f951172b640 1 --2- 172.21.15.70:0/4203123669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f950c102340 0x7f950c10e800 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f9504009cd0 tx=0x7f950402a8e0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.839+0000 7f951172b640 1 -- 172.21.15.70:0/4203123669 shutdown_connections 2024-06-04T05:13:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.839+0000 7f951172b640 1 --2- 172.21.15.70:0/4203123669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f950c102340 0x7f950c10e800 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.839+0000 7f951172b640 1 --2- 172.21.15.70:0/4203123669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f950c101980 0x7f950c101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.839+0000 7f951172b640 1 --2- 172.21.15.70:0/4203123669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f950c107940 0x7f950c107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.839+0000 7f951172b640 1 -- 172.21.15.70:0/4203123669 >> 172.21.15.70:0/4203123669 conn(0x7f950c078ad0 msgr2=0x7f950c078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.839+0000 7f951172b640 1 -- 172.21.15.70:0/4203123669 shutdown_connections 2024-06-04T05:13:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.839+0000 7f951172b640 1 -- 172.21.15.70:0/4203123669 wait complete. 2024-06-04T05:13:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.840+0000 7f951172b640 1 Processor -- start 2024-06-04T05:13:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.840+0000 7f951172b640 1 -- start start 2024-06-04T05:13:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.840+0000 7f951172b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f950c101980 0x7f950c199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.840+0000 7f951172b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f950c102340 0x7f950c1996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.840+0000 7f951172b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f950c107940 0x7f950c19da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.840+0000 7f951172b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f950c111ab0 con 0x7f950c107940 2024-06-04T05:13:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.840+0000 7f951172b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f950c111930 con 0x7f950c101980 2024-06-04T05:13:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.840+0000 7f951172b640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f950c111c30 con 0x7f950c102340 2024-06-04T05:13:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.840+0000 7f950b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f950c107940 0x7f950c19da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.840+0000 7f950b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f950c107940 0x7f950c19da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51058/0 (socket says 172.21.15.70:51058) 2024-06-04T05:13:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.840+0000 7f950b7fe640 1 -- 172.21.15.70:0/1875825316 learned_addr learned my addr 172.21.15.70:0/1875825316 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.841+0000 7f9503fff640 1 --2- 172.21.15.70:0/1875825316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f950c102340 0x7f950c1996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.841+0000 7f950b7fe640 1 -- 172.21.15.70:0/1875825316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f950c102340 msgr2=0x7f950c1996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.841+0000 7f950affd640 1 --2- 172.21.15.70:0/1875825316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f950c101980 0x7f950c199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.841+0000 7f950b7fe640 1 --2- 172.21.15.70:0/1875825316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f950c102340 0x7f950c1996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.841+0000 7f950b7fe640 1 -- 172.21.15.70:0/1875825316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f950c101980 msgr2=0x7f950c199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.841+0000 7f950b7fe640 1 --2- 172.21.15.70:0/1875825316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f950c101980 0x7f950c199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.841+0000 7f950b7fe640 1 -- 172.21.15.70:0/1875825316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f950c19e170 con 0x7f950c107940 2024-06-04T05:13:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.841+0000 7f9503fff640 1 --2- 172.21.15.70:0/1875825316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f950c102340 0x7f950c1996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:13:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.841+0000 7f950b7fe640 1 --2- 172.21.15.70:0/1875825316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f950c107940 0x7f950c19da40 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f9504006010 tx=0x7f9504003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.841+0000 7f9508ff9640 1 -- 172.21.15.70:0/1875825316 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9504004450 con 0x7f950c107940 2024-06-04T05:13:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.841+0000 7f9508ff9640 1 -- 172.21.15.70:0/1875825316 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9504003e10 con 0x7f950c107940 2024-06-04T05:13:01.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.841+0000 7f951172b640 1 -- 172.21.15.70:0/1875825316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f950c19e400 con 0x7f950c107940 2024-06-04T05:13:01.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.841+0000 7f9508ff9640 1 -- 172.21.15.70:0/1875825316 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95040466e0 con 0x7f950c107940 2024-06-04T05:13:01.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.841+0000 7f951172b640 1 -- 172.21.15.70:0/1875825316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f950c1a5cf0 con 0x7f950c107940 2024-06-04T05:13:01.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.844+0000 7f9508ff9640 1 -- 172.21.15.70:0/1875825316 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f95040305a0 con 0x7f950c107940 2024-06-04T05:13:01.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.844+0000 7f9514449740 1 Processor -- start 2024-06-04T05:13:01.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.844+0000 7f9514449740 1 -- start start 2024-06-04T05:13:01.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.844+0000 7f9514449740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f986820 0x55f47f986c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.844+0000 7f9514449740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 0x55f47f9914c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.844+0000 7f9514449740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f991a00 0x55f47f993df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.844+0000 7f9514449740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f47f8f8c30 con 0x55f47f991a00 2024-06-04T05:13:01.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.844+0000 7f9514449740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f47f8e87d0 con 0x55f47f986820 2024-06-04T05:13:01.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.844+0000 7f9514449740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f47f8ec810 con 0x55f47f987140 2024-06-04T05:13:01.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f9503fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f991a00 0x55f47f993df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:01.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f9503fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f991a00 0x55f47f993df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51072/0 (socket says 172.21.15.70:51072) 2024-06-04T05:13:01.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f9508ff9640 1 --2- 172.21.15.70:0/1875825316 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f94f80b2a10 0x7f94f80b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f9503fff640 1 -- 172.21.15.70:0/2231090767 learned_addr learned my addr 172.21.15.70:0/2231090767 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:01.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f950affd640 1 --2- 172.21.15.70:0/2231090767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 0x55f47f9914c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:01.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f950b7fe640 1 --2- 172.21.15.70:0/1875825316 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f94f80b2a10 0x7f94f80b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f9503fff640 1 --2- 172.21.15.70:0/2231090767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f986820 0x55f47f986c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f9503fff640 1 -- 172.21.15.70:0/2231090767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 msgr2=0x55f47f9914c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f9503fff640 1 --2- 172.21.15.70:0/2231090767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 0x55f47f9914c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f9503fff640 1 -- 172.21.15.70:0/2231090767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f986820 msgr2=0x55f47f986c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f9503fff640 1 --2- 172.21.15.70:0/2231090767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f986820 0x55f47f986c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f9503fff640 1 -- 172.21.15.70:0/2231090767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f47f5fd210 con 0x55f47f991a00 2024-06-04T05:13:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f9508ff9640 1 -- 172.21.15.70:0/1875825316 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f950402e030 con 0x7f950c107940 2024-06-04T05:13:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f9503fff640 1 --2- 172.21.15.70:0/2231090767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f991a00 0x55f47f993df0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f94f400cf90 tx=0x7f94f400c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78151e6304a93509 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f950b7fe640 1 --2- 172.21.15.70:0/1875825316 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f94f80b2a10 0x7f94f80b4ed0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f95040cb320 tx=0x7f9504097040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f95017fa640 1 -- 172.21.15.70:0/2231090767 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f401b070 con 0x55f47f991a00 2024-06-04T05:13:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f95017fa640 1 -- 172.21.15.70:0/2231090767 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94f4016880 con 0x55f47f991a00 2024-06-04T05:13:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.845+0000 7f95017fa640 1 -- 172.21.15.70:0/2231090767 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f400f040 con 0x55f47f991a00 2024-06-04T05:13:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.846+0000 7f9514449740 1 -- 172.21.15.70:0/2231090767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f991a00 msgr2=0x55f47f993df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.846+0000 7f9514449740 1 --2- 172.21.15.70:0/2231090767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f991a00 0x55f47f993df0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f94f400cf90 tx=0x7f94f400c410 comp rx=0 tx=0).stop 2024-06-04T05:13:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.846+0000 7f9514449740 1 -- 172.21.15.70:0/2231090767 shutdown_connections 2024-06-04T05:13:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.846+0000 7f9514449740 1 --2- 172.21.15.70:0/2231090767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f991a00 0x55f47f993df0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.846+0000 7f9514449740 1 --2- 172.21.15.70:0/2231090767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 0x55f47f9914c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.846+0000 7f9514449740 1 --2- 172.21.15.70:0/2231090767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f986820 0x55f47f986c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.846+0000 7f9514449740 1 -- 172.21.15.70:0/2231090767 >> 172.21.15.70:0/2231090767 conn(0x55f47f982d30 msgr2=0x55f47f983110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.846+0000 7f9514449740 1 -- 172.21.15.70:0/2231090767 shutdown_connections 2024-06-04T05:13:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.846+0000 7f9514449740 1 -- 172.21.15.70:0/2231090767 wait complete. 2024-06-04T05:13:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f9514449740 1 -- start start 2024-06-04T05:13:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f9514449740 1 Processor -- start 2024-06-04T05:13:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f9514449740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f986820 0x55f47f98ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f9514449740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 0x55f47f98ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f9514449740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f991a00 0x55f47f98f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f9514449740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f47f5ce790 con 0x55f47f986820 2024-06-04T05:13:01.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f9514449740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f47f8a4890 con 0x55f47f991a00 2024-06-04T05:13:01.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f9514449740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f47f994630 con 0x55f47f987140 2024-06-04T05:13:01.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f9514449740 10 client.0 _ll_register_callbacks cb 0x55f47f5fca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:01.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f9503fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f986820 0x55f47f98ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:01.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f9503fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f986820 0x55f47f98ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51076/0 (socket says 172.21.15.70:51076) 2024-06-04T05:13:01.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f9503fff640 1 -- 172.21.15.70:0/1098049139 learned_addr learned my addr 172.21.15.70:0/1098049139 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:01.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f950affd640 1 --2- 172.21.15.70:0/1098049139 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 0x55f47f98ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:01.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f9503fff640 1 -- 172.21.15.70:0/1098049139 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 msgr2=0x55f47f98ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:01.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f9503fff640 1 --2- 172.21.15.70:0/1098049139 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 0x55f47f98ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f9503fff640 1 -- 172.21.15.70:0/1098049139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f991a00 msgr2=0x55f47f98f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:01.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f9503fff640 1 --2- 172.21.15.70:0/1098049139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f991a00 0x55f47f98f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f9503fff640 1 -- 172.21.15.70:0/1098049139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f47f6c9620 con 0x55f47f986820 2024-06-04T05:13:01.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.847+0000 7f950affd640 1 --2- 172.21.15.70:0/1098049139 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 0x55f47f98ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:13:01.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.848+0000 7f9503fff640 1 --2- 172.21.15.70:0/1098049139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f986820 0x55f47f98ea20 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f94f4004700 tx=0x7f94f4006d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b70e3b29b266611a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.848+0000 7f94f27fc640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:13:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.848+0000 7f94f27fc640 1 -- 172.21.15.70:0/1098049139 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4027060 con 0x55f47f986820 2024-06-04T05:13:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.848+0000 7f94f27fc640 1 -- 172.21.15.70:0/1098049139 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f94f4016c30 con 0x55f47f986820 2024-06-04T05:13:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.848+0000 7f9514449740 1 -- 172.21.15.70:0/1098049139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94f4016880 con 0x55f47f986820 2024-06-04T05:13:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.848+0000 7f94f27fc640 1 -- 172.21.15.70:0/1098049139 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f94f401b070 con 0x55f47f986820 2024-06-04T05:13:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.848+0000 7f9514449740 10 client.19566 fetch_fsmap learned FSMap version 45 2024-06-04T05:13:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.848+0000 7f9514449740 1 -- 172.21.15.70:0/1098049139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55f47f9a0450 con 0x55f47f986820 2024-06-04T05:13:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.848+0000 7f94f27fc640 1 -- 172.21.15.70:0/1098049139 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94f4016880 con 0x55f47f986820 2024-06-04T05:13:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.848+0000 7f94f27fc640 1 -- 172.21.15.70:0/1098049139 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 45) ==== 8398+0+0 (secure 0 0 0) 0x7f94f4015070 con 0x55f47f986820 2024-06-04T05:13:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.848+0000 7f9514449740 10 client.19566 fetch_fsmap finished waiting for FSMap version 45 2024-06-04T05:13:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.848+0000 7f9514449740 10 client.19566 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.848+0000 7f9514449740 20 client.19566 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.848+0000 7f9514449740 1 --2- 172.21.15.70:0/1098049139 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f47f9a1800 0x55f47f9a3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.848+0000 7f9514449740 4 client.19566 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:13:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.848+0000 7f9514449740 1 -- 172.21.15.70:0/1098049139 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f47f8fab80 con 0x55f47f9a1800 2024-06-04T05:13:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.849+0000 7f950affd640 1 --2- 172.21.15.70:0/1098049139 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f47f9a1800 0x55f47f9a3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:01.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.849+0000 7f94f27fc640 10 client.19566 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:01.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.849+0000 7f950affd640 1 --2- 172.21.15.70:0/1098049139 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f47f9a1800 0x55f47f9a3bf0 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71f5f9474fab624f server_cookie=5e1616f8802806a in_seq=0 out_seq=0 2024-06-04T05:13:01.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.850+0000 7f94f27fc640 1 -- 172.21.15.70:0/1098049139 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f47f8fab80 con 0x55f47f9a1800 2024-06-04T05:13:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.850+0000 7f94f27fc640 10 client.19566 handle_command_reply: tid=0 2024-06-04T05:13:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.850+0000 7f9514449740 1 client.19566 shutdown 2024-06-04T05:13:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.850+0000 7f9514449740 1 -- 172.21.15.70:0/1098049139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f986820 msgr2=0x55f47f98ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.850+0000 7f9514449740 1 --2- 172.21.15.70:0/1098049139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f986820 0x55f47f98ea20 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f94f4004700 tx=0x7f94f4006d00 comp rx=0 tx=0).stop 2024-06-04T05:13:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.850+0000 7f9514449740 1 -- 172.21.15.70:0/1098049139 shutdown_connections 2024-06-04T05:13:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.850+0000 7f9514449740 1 --2- 172.21.15.70:0/1098049139 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f47f9a1800 0x55f47f9a3bf0 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.850+0000 7f9514449740 1 --2- 172.21.15.70:0/1098049139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f991a00 0x55f47f98f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.850+0000 7f9514449740 1 --2- 172.21.15.70:0/1098049139 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 0x55f47f98ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.850+0000 7f94f27fc640 0 client.19566 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.850+0000 7f9514449740 1 --2- 172.21.15.70:0/1098049139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f986820 0x55f47f98ea20 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.850+0000 7f9514449740 1 -- 172.21.15.70:0/1098049139 >> 172.21.15.70:0/1098049139 conn(0x55f47f982d30 msgr2=0x55f47f981fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.850+0000 7f9514449740 1 -- 172.21.15.70:0/1098049139 shutdown_connections 2024-06-04T05:13:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.850+0000 7f9514449740 1 -- 172.21.15.70:0/1098049139 wait complete. 2024-06-04T05:13:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.850+0000 7f9514449740 20 client.19566 trim_cache size 0 max 16384 2024-06-04T05:13:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.866+0000 7f9514449740 1 Processor -- start 2024-06-04T05:13:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.866+0000 7f9514449740 1 -- start start 2024-06-04T05:13:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.866+0000 7f9514449740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f986820 0x55f47f9891e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.866+0000 7f9514449740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 0x55f47f98f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.866+0000 7f9514449740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f991a00 0x55f47f9a9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.866+0000 7f9514449740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f950c1ae550 con 0x55f47f991a00 2024-06-04T05:13:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.866+0000 7f9514449740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f950c1b01e0 con 0x55f47f986820 2024-06-04T05:13:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.866+0000 7f9514449740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f950c1aec20 con 0x55f47f987140 2024-06-04T05:13:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.866+0000 7f9503fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f991a00 0x55f47f9a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.866+0000 7f9503fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f991a00 0x55f47f9a9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51092/0 (socket says 172.21.15.70:51092) 2024-06-04T05:13:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.866+0000 7f9503fff640 1 -- 172.21.15.70:0/3612168890 learned_addr learned my addr 172.21.15.70:0/3612168890 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.866+0000 7f9503fff640 1 --2- 172.21.15.70:0/3612168890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f986820 0x55f47f9891e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.866+0000 7f9503fff640 1 -- 172.21.15.70:0/3612168890 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 msgr2=0x55f47f98f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.866+0000 7f9503fff640 1 --2- 172.21.15.70:0/3612168890 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 0x55f47f98f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.866+0000 7f9503fff640 1 -- 172.21.15.70:0/3612168890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f986820 msgr2=0x55f47f9891e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.866+0000 7f9503fff640 1 --2- 172.21.15.70:0/3612168890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f986820 0x55f47f9891e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.866+0000 7f9503fff640 1 -- 172.21.15.70:0/3612168890 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f47f97e4d0 con 0x55f47f991a00 2024-06-04T05:13:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.867+0000 7f9503fff640 1 --2- 172.21.15.70:0/3612168890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f991a00 0x55f47f9a9450 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f94f4016810 tx=0x7f94f400eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809e7323c2bac23c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.867+0000 7f9500ff9640 1 -- 172.21.15.70:0/3612168890 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4027040 con 0x55f47f991a00 2024-06-04T05:13:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.867+0000 7f9500ff9640 1 -- 172.21.15.70:0/3612168890 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94f40226f0 con 0x55f47f991a00 2024-06-04T05:13:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.867+0000 7f9500ff9640 1 -- 172.21.15.70:0/3612168890 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f402b440 con 0x55f47f991a00 2024-06-04T05:13:01.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.868+0000 7f9514449740 1 -- 172.21.15.70:0/3612168890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f991a00 msgr2=0x55f47f9a9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:01.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.868+0000 7f9514449740 1 --2- 172.21.15.70:0/3612168890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f991a00 0x55f47f9a9450 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f94f4016810 tx=0x7f94f400eb10 comp rx=0 tx=0).stop 2024-06-04T05:13:01.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.868+0000 7f9514449740 1 -- 172.21.15.70:0/3612168890 shutdown_connections 2024-06-04T05:13:01.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.868+0000 7f9514449740 1 --2- 172.21.15.70:0/3612168890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f991a00 0x55f47f9a9450 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.868+0000 7f9514449740 1 --2- 172.21.15.70:0/3612168890 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 0x55f47f98f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.868+0000 7f9514449740 1 --2- 172.21.15.70:0/3612168890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f986820 0x55f47f9891e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.868+0000 7f9514449740 1 -- 172.21.15.70:0/3612168890 >> 172.21.15.70:0/3612168890 conn(0x55f47f982d30 msgr2=0x55f47f99cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:01.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.868+0000 7f9514449740 1 -- 172.21.15.70:0/3612168890 shutdown_connections 2024-06-04T05:13:01.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.868+0000 7f9514449740 1 -- 172.21.15.70:0/3612168890 wait complete. 2024-06-04T05:13:01.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.868+0000 7f9514449740 1 -- start start 2024-06-04T05:13:01.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.868+0000 7f9514449740 1 Processor -- start 2024-06-04T05:13:01.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9514449740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f986820 0x55f47f999ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9514449740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 0x55f47f98b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9514449740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f991a00 0x55f47f98bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9514449740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f950c1afa50 con 0x55f47f986820 2024-06-04T05:13:01.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9514449740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f950c1af350 con 0x55f47f991a00 2024-06-04T05:13:01.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9514449740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f950c1ac550 con 0x55f47f987140 2024-06-04T05:13:01.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9514449740 10 client.0 _ll_register_callbacks cb 0x7f94f4019070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:01.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9503fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f986820 0x55f47f999ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:01.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9503fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f986820 0x55f47f999ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51106/0 (socket says 172.21.15.70:51106) 2024-06-04T05:13:01.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9503fff640 1 -- 172.21.15.70:0/1994806686 learned_addr learned my addr 172.21.15.70:0/1994806686 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:01.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f950affd640 1 --2- 172.21.15.70:0/1994806686 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 0x55f47f98b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:01.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9503fff640 1 --2- 172.21.15.70:0/1994806686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f991a00 0x55f47f98bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:01.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9503fff640 1 -- 172.21.15.70:0/1994806686 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 msgr2=0x55f47f98b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:01.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9503fff640 1 --2- 172.21.15.70:0/1994806686 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 0x55f47f98b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9503fff640 1 -- 172.21.15.70:0/1994806686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f991a00 msgr2=0x55f47f98bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:01.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9503fff640 1 --2- 172.21.15.70:0/1994806686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f991a00 0x55f47f98bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9503fff640 1 -- 172.21.15.70:0/1994806686 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f47f98c3b0 con 0x55f47f986820 2024-06-04T05:13:01.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f950affd640 1 --2- 172.21.15.70:0/1994806686 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 0x55f47f98b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:13:01.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9503fff640 1 --2- 172.21.15.70:0/1994806686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f986820 0x55f47f999ea0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f94f4022450 tx=0x7f94f400c5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62763a91f6275172 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:01.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f94f27fc640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:13:01.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f94f27fc640 1 -- 172.21.15.70:0/1994806686 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4045060 con 0x55f47f986820 2024-06-04T05:13:01.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f94f27fc640 1 -- 172.21.15.70:0/1994806686 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f94f4022d60 con 0x55f47f986820 2024-06-04T05:13:01.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.869+0000 7f9514449740 1 -- 172.21.15.70:0/1994806686 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94f40226f0 con 0x55f47f986820 2024-06-04T05:13:01.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.870+0000 7f94f27fc640 1 -- 172.21.15.70:0/1994806686 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f94f4027050 con 0x55f47f986820 2024-06-04T05:13:01.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.870+0000 7f9514449740 10 client.19572 fetch_fsmap learned FSMap version 45 2024-06-04T05:13:01.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.870+0000 7f9514449740 1 -- 172.21.15.70:0/1994806686 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55f47fa0c270 con 0x55f47f986820 2024-06-04T05:13:01.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.870+0000 7f94f27fc640 1 -- 172.21.15.70:0/1994806686 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94f40226f0 con 0x55f47f986820 2024-06-04T05:13:01.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.870+0000 7f94f27fc640 1 -- 172.21.15.70:0/1994806686 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 45) ==== 8398+0+0 (secure 0 0 0) 0x7f94f4015040 con 0x55f47f986820 2024-06-04T05:13:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.870+0000 7f9514449740 10 client.19572 fetch_fsmap finished waiting for FSMap version 45 2024-06-04T05:13:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.870+0000 7f9514449740 10 client.19572 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.870+0000 7f9514449740 20 client.19572 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.870+0000 7f9514449740 1 --2- 172.21.15.70:0/1994806686 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f47fa0c880 0x55f47fa0eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.870+0000 7f9514449740 4 client.19572 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:13:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.870+0000 7f9514449740 1 -- 172.21.15.70:0/1994806686 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f47f8f4360 con 0x55f47fa0c880 2024-06-04T05:13:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.870+0000 7f950affd640 1 --2- 172.21.15.70:0/1994806686 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f47fa0c880 0x55f47fa0eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.871+0000 7f950affd640 1 --2- 172.21.15.70:0/1994806686 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f47fa0c880 0x55f47fa0eca0 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67000a3d4b41de87 server_cookie=5504512b48ca16e1 in_seq=0 out_seq=0 2024-06-04T05:13:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.871+0000 7f94f27fc640 10 client.19572 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:01.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.889+0000 7f94f27fc640 1 -- 172.21.15.70:0/1994806686 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f47f8f4360 con 0x55f47fa0c880 2024-06-04T05:13:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.889+0000 7f94f27fc640 10 client.19572 handle_command_reply: tid=0 2024-06-04T05:13:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.889+0000 7f9514449740 1 client.19572 shutdown 2024-06-04T05:13:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.889+0000 7f9514449740 1 -- 172.21.15.70:0/1994806686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f986820 msgr2=0x55f47f999ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.889+0000 7f9514449740 1 --2- 172.21.15.70:0/1994806686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f986820 0x55f47f999ea0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f94f4022450 tx=0x7f94f400c5a0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.889+0000 7f9514449740 1 -- 172.21.15.70:0/1994806686 shutdown_connections 2024-06-04T05:13:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.889+0000 7f9514449740 1 --2- 172.21.15.70:0/1994806686 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f47fa0c880 0x55f47fa0eca0 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.889+0000 7f9514449740 1 --2- 172.21.15.70:0/1994806686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f47f991a00 0x55f47f98bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.889+0000 7f9514449740 1 --2- 172.21.15.70:0/1994806686 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f47f987140 0x55f47f98b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.889+0000 7f9514449740 1 --2- 172.21.15.70:0/1994806686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f47f986820 0x55f47f999ea0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.889+0000 7f94f27fc640 0 client.19572 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.889+0000 7f9514449740 1 -- 172.21.15.70:0/1994806686 >> 172.21.15.70:0/1994806686 conn(0x55f47f982d30 msgr2=0x55f47f990250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.889+0000 7f9514449740 1 -- 172.21.15.70:0/1994806686 shutdown_connections 2024-06-04T05:13:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.889+0000 7f9514449740 1 -- 172.21.15.70:0/1994806686 wait complete. 2024-06-04T05:13:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.889+0000 7f9514449740 20 client.19572 trim_cache size 0 max 16384 2024-06-04T05:13:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:13:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:13:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:13:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.890+0000 7f951172b640 1 -- 172.21.15.70:0/1875825316 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f94f80b2a10 msgr2=0x7f94f80b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.890+0000 7f951172b640 1 --2- 172.21.15.70:0/1875825316 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f94f80b2a10 0x7f94f80b4ed0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f95040cb320 tx=0x7f9504097040 comp rx=0 tx=0).stop 2024-06-04T05:13:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.890+0000 7f951172b640 1 -- 172.21.15.70:0/1875825316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f950c107940 msgr2=0x7f950c19da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.890+0000 7f951172b640 1 --2- 172.21.15.70:0/1875825316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f950c107940 0x7f950c19da40 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f9504006010 tx=0x7f9504003c20 comp rx=0 tx=0).stop 2024-06-04T05:13:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.890+0000 7f951172b640 1 -- 172.21.15.70:0/1875825316 shutdown_connections 2024-06-04T05:13:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.890+0000 7f951172b640 1 --2- 172.21.15.70:0/1875825316 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f94f80b2a10 0x7f94f80b4ed0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.890+0000 7f951172b640 1 --2- 172.21.15.70:0/1875825316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f950c107940 0x7f950c19da40 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.890+0000 7f951172b640 1 --2- 172.21.15.70:0/1875825316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f950c102340 0x7f950c1996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.890+0000 7f951172b640 1 --2- 172.21.15.70:0/1875825316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f950c101980 0x7f950c199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.890+0000 7f951172b640 1 -- 172.21.15.70:0/1875825316 >> 172.21.15.70:0/1875825316 conn(0x7f950c078ad0 msgr2=0x7f950c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.890+0000 7f951172b640 1 -- 172.21.15.70:0/1875825316 shutdown_connections 2024-06-04T05:13:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:01.890+0000 7f951172b640 1 -- 172.21.15.70:0/1875825316 wait complete. 2024-06-04T05:13:02.180 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id e648773 --release --await' 2024-06-04T05:13:02.180 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id e648773 --release --await 2024-06-04T05:13:02.423 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:13:02.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 8.88 MB/sec warmup 114 sec latency 25618.711 ms 2024-06-04T05:13:03.563 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:03 smithi070 ceph-mon[32631]: pgmap v495: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 80 KiB/s wr, 42 op/s 2024-06-04T05:13:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:03 smithi112 ceph-mon[36453]: pgmap v495: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 80 KiB/s wr, 42 op/s 2024-06-04T05:13:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:03 smithi098 ceph-mon[35569]: pgmap v495: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 80 KiB/s wr, 42 op/s 2024-06-04T05:13:03.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15632 8.80 MB/sec warmup 115 sec latency 26618.768 ms 2024-06-04T05:13:04.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.255+0000 7fa02fd2b700 1 -- 172.21.15.70:0/1598275266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0281027f0 msgr2=0x7fa028106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:04.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.255+0000 7fa02fd2b700 1 --2- 172.21.15.70:0/1598275266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0281027f0 0x7fa028106840 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fa01800aad0 tx=0x7fa01800ade0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.255+0000 7fa02fd2b700 1 -- 172.21.15.70:0/1598275266 shutdown_connections 2024-06-04T05:13:04.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.255+0000 7fa02fd2b700 1 --2- 172.21.15.70:0/1598275266 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa028106f20 0x7fa02810eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.255+0000 7fa02fd2b700 1 --2- 172.21.15.70:0/1598275266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa028101e40 0x7fa028102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.255+0000 7fa02fd2b700 1 --2- 172.21.15.70:0/1598275266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0281027f0 0x7fa028106840 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.255+0000 7fa02fd2b700 1 -- 172.21.15.70:0/1598275266 >> 172.21.15.70:0/1598275266 conn(0x7fa028078a40 msgr2=0x7fa0280ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:04.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.255+0000 7fa02fd2b700 1 -- 172.21.15.70:0/1598275266 shutdown_connections 2024-06-04T05:13:04.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.255+0000 7fa02fd2b700 1 -- 172.21.15.70:0/1598275266 wait complete. 2024-06-04T05:13:04.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.255+0000 7fa02fd2b700 1 Processor -- start 2024-06-04T05:13:04.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.255+0000 7fa02fd2b700 1 -- start start 2024-06-04T05:13:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.255+0000 7fa02fd2b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa028101e40 0x7fa02819f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02fd2b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0281027f0 0x7fa02819fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02fd2b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa028106f20 0x7fa028199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02fd2b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa028111430 con 0x7fa028106f20 2024-06-04T05:13:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02fd2b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa0281112b0 con 0x7fa0281027f0 2024-06-04T05:13:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02fd2b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa0281115b0 con 0x7fa028101e40 2024-06-04T05:13:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02e2c8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa028106f20 0x7fa028199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02e2c8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa028106f20 0x7fa028199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51122/0 (socket says 172.21.15.70:51122) 2024-06-04T05:13:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02e2c8700 1 -- 172.21.15.70:0/1597315364 learned_addr learned my addr 172.21.15.70:0/1597315364 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02d2c6700 1 --2- 172.21.15.70:0/1597315364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0281027f0 0x7fa02819fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02dac7700 1 --2- 172.21.15.70:0/1597315364 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa028101e40 0x7fa02819f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02e2c8700 1 -- 172.21.15.70:0/1597315364 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa028101e40 msgr2=0x7fa02819f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02e2c8700 1 --2- 172.21.15.70:0/1597315364 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa028101e40 0x7fa02819f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02e2c8700 1 -- 172.21.15.70:0/1597315364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0281027f0 msgr2=0x7fa02819fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02e2c8700 1 --2- 172.21.15.70:0/1597315364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0281027f0 0x7fa02819fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02e2c8700 1 -- 172.21.15.70:0/1597315364 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa02819a210 con 0x7fa028106f20 2024-06-04T05:13:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02d2c6700 1 --2- 172.21.15.70:0/1597315364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0281027f0 0x7fa02819fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:13:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02e2c8700 1 --2- 172.21.15.70:0/1597315364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa028106f20 0x7fa028199ab0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fa02000b9e0 tx=0x7fa02000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa01effd700 1 -- 172.21.15.70:0/1597315364 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa020019070 con 0x7fa028106f20 2024-06-04T05:13:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa01effd700 1 -- 172.21.15.70:0/1597315364 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa02000cde0 con 0x7fa028106f20 2024-06-04T05:13:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02fd2b700 1 -- 172.21.15.70:0/1597315364 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa02819a500 con 0x7fa028106f20 2024-06-04T05:13:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa01effd700 1 -- 172.21.15.70:0/1597315364 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa020007370 con 0x7fa028106f20 2024-06-04T05:13:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.256+0000 7fa02fd2b700 1 -- 172.21.15.70:0/1597315364 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0281a67c0 con 0x7fa028106f20 2024-06-04T05:13:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.259+0000 7fa01effd700 1 -- 172.21.15.70:0/1597315364 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fa02001c070 con 0x7fa028106f20 2024-06-04T05:13:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.259+0000 7fa01effd700 1 --2- 172.21.15.70:0/1597315364 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa0140b29f0 0x7fa0140b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.260+0000 7fa02dac7700 1 --2- 172.21.15.70:0/1597315364 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa0140b29f0 0x7fa0140b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.260+0000 7fa01effd700 1 -- 172.21.15.70:0/1597315364 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8615+0+0 (secure 0 0 0) 0x7fa0200e1bd0 con 0x7fa028106f20 2024-06-04T05:13:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.260+0000 7fa03a5c0b80 1 Processor -- start 2024-06-04T05:13:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.260+0000 7fa02dac7700 1 --2- 172.21.15.70:0/1597315364 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa0140b29f0 0x7fa0140b4eb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa024005ea0 tx=0x7fa024005e30 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.260+0000 7fa03a5c0b80 1 -- start start 2024-06-04T05:13:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.260+0000 7fa03a5c0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711692b0 0x55db71169690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.260+0000 7fa03a5c0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711638e0 0x55db71163d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.260+0000 7fa03a5c0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db711700e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.260+0000 7fa03a5c0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55db710c2d90 con 0x55db711642a0 2024-06-04T05:13:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.260+0000 7fa03a5c0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55db710b2930 con 0x55db711692b0 2024-06-04T05:13:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.260+0000 7fa03a5c0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55db710b6970 con 0x55db711638e0 2024-06-04T05:13:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.260+0000 7fa02e2c8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db711700e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.261+0000 7fa02e2c8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db711700e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51126/0 (socket says 172.21.15.70:51126) 2024-06-04T05:13:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.261+0000 7fa02e2c8700 1 -- 172.21.15.70:0/1635711949 learned_addr learned my addr 172.21.15.70:0/1635711949 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.261+0000 7fa02d2c6700 1 --2- 172.21.15.70:0/1635711949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711638e0 0x55db71163d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.261+0000 7fa02dac7700 1 --2- 172.21.15.70:0/1635711949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711692b0 0x55db71169690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.261+0000 7fa02e2c8700 1 -- 172.21.15.70:0/1635711949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711638e0 msgr2=0x55db71163d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.261+0000 7fa02e2c8700 1 --2- 172.21.15.70:0/1635711949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711638e0 0x55db71163d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.261+0000 7fa02e2c8700 1 -- 172.21.15.70:0/1635711949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711692b0 msgr2=0x55db71169690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.261+0000 7fa02e2c8700 1 --2- 172.21.15.70:0/1635711949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711692b0 0x55db71169690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.261+0000 7fa02e2c8700 1 -- 172.21.15.70:0/1635711949 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db710029d0 con 0x55db711642a0 2024-06-04T05:13:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.261+0000 7fa02dac7700 1 --2- 172.21.15.70:0/1635711949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711692b0 0x55db71169690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:13:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.261+0000 7fa02e2c8700 1 --2- 172.21.15.70:0/1635711949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db711700e0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fa0200a8e50 tx=0x7fa020009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303486d3d02e2a07 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.261+0000 7fa013fff700 1 -- 172.21.15.70:0/1635711949 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0200cb070 con 0x55db711642a0 2024-06-04T05:13:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.261+0000 7fa013fff700 1 -- 172.21.15.70:0/1635711949 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa0200ad270 con 0x55db711642a0 2024-06-04T05:13:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.261+0000 7fa013fff700 1 -- 172.21.15.70:0/1635711949 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0200b1960 con 0x55db711642a0 2024-06-04T05:13:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.262+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/1635711949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 msgr2=0x55db711700e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.262+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/1635711949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db711700e0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fa0200a8e50 tx=0x7fa020009f90 comp rx=0 tx=0).stop 2024-06-04T05:13:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.262+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/1635711949 shutdown_connections 2024-06-04T05:13:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.262+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/1635711949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db711700e0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.262+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/1635711949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711692b0 0x55db71169690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.262+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/1635711949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711638e0 0x55db71163d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.262+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/1635711949 >> 172.21.15.70:0/1635711949 conn(0x55db71091180 msgr2=0x55db7115e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.262+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/1635711949 shutdown_connections 2024-06-04T05:13:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.262+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/1635711949 wait complete. 2024-06-04T05:13:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.262+0000 7fa03a5c0b80 1 -- start start 2024-06-04T05:13:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.262+0000 7fa03a5c0b80 1 Processor -- start 2024-06-04T05:13:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.263+0000 7fa03a5c0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711638e0 0x55db7116fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.263+0000 7fa03a5c0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db7116d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.263+0000 7fa03a5c0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711692b0 0x55db7116d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.263+0000 7fa03a5c0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55db70fb90a0 con 0x55db711642a0 2024-06-04T05:13:04.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.263+0000 7fa03a5c0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55db710548c0 con 0x55db711638e0 2024-06-04T05:13:04.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.263+0000 7fa03a5c0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55db71170680 con 0x55db711692b0 2024-06-04T05:13:04.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.263+0000 7fa03a5c0b80 10 client.0 _ll_register_callbacks cb 0x55db7104da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:04.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.263+0000 7fa02d2c6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db7116d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.263+0000 7fa02d2c6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db7116d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51138/0 (socket says 172.21.15.70:51138) 2024-06-04T05:13:04.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.263+0000 7fa02d2c6700 1 -- 172.21.15.70:0/3144741016 learned_addr learned my addr 172.21.15.70:0/3144741016 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:04.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.263+0000 7fa02dac7700 1 --2- 172.21.15.70:0/3144741016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711638e0 0x55db7116fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.263+0000 7fa02e2c8700 1 --2- 172.21.15.70:0/3144741016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711692b0 0x55db7116d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.263+0000 7fa02d2c6700 1 -- 172.21.15.70:0/3144741016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711692b0 msgr2=0x55db7116d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:04.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.263+0000 7fa02d2c6700 1 --2- 172.21.15.70:0/3144741016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711692b0 0x55db7116d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.263+0000 7fa02d2c6700 1 -- 172.21.15.70:0/3144741016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711638e0 msgr2=0x55db7116fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:04.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.263+0000 7fa02d2c6700 1 --2- 172.21.15.70:0/3144741016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711638e0 0x55db7116fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.263+0000 7fa02d2c6700 1 -- 172.21.15.70:0/3144741016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db71051790 con 0x55db711642a0 2024-06-04T05:13:04.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.264+0000 7fa02d2c6700 1 --2- 172.21.15.70:0/3144741016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db7116d0f0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fa01800aad0 tx=0x7fa018003db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a0050f4cdb7f90 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:04.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.264+0000 7fa0117fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:13:04.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.264+0000 7fa0117fa700 1 -- 172.21.15.70:0/3144741016 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01800cd80 con 0x55db711642a0 2024-06-04T05:13:04.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.264+0000 7fa0117fa700 1 -- 172.21.15.70:0/3144741016 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8615+0+0 (secure 0 0 0) 0x7fa018003680 con 0x55db711642a0 2024-06-04T05:13:04.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.264+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/3144741016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0200ad270 con 0x55db711642a0 2024-06-04T05:13:04.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.264+0000 7fa0117fa700 1 -- 172.21.15.70:0/3144741016 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fa018030050 con 0x55db711642a0 2024-06-04T05:13:04.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.264+0000 7fa03a5c0b80 10 client.19584 fetch_fsmap learned FSMap version 45 2024-06-04T05:13:04.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.264+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/3144741016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55db7117ad70 con 0x55db711642a0 2024-06-04T05:13:04.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.264+0000 7fa0117fa700 1 -- 172.21.15.70:0/3144741016 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa0200ad270 con 0x55db711642a0 2024-06-04T05:13:04.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.264+0000 7fa0117fa700 1 -- 172.21.15.70:0/3144741016 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 45) ==== 8398+0+0 (secure 0 0 0) 0x7fa01800b040 con 0x55db711642a0 2024-06-04T05:13:04.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.264+0000 7fa03a5c0b80 10 client.19584 fetch_fsmap finished waiting for FSMap version 45 2024-06-04T05:13:04.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.264+0000 7fa03a5c0b80 10 client.19584 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:13:04.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.264+0000 7fa03a5c0b80 20 client.19584 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:04.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.264+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/3144741016 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55db7117d400 0x55db7117f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.264+0000 7fa03a5c0b80 4 client.19584 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:13:04.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.264+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/3144741016 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db7112fae0 con 0x55db7117d400 2024-06-04T05:13:04.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.265+0000 7fa02dac7700 1 --2- 172.21.15.70:0/3144741016 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55db7117d400 0x55db7117f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.265+0000 7fa02dac7700 1 --2- 172.21.15.70:0/3144741016 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55db7117d400 0x55db7117f840 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c2925939220abdb server_cookie=ef71d6d477ddb28 in_seq=0 out_seq=0 2024-06-04T05:13:04.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.265+0000 7fa0117fa700 10 client.19584 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:04.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.266+0000 7fa0117fa700 1 -- 172.21.15.70:0/3144741016 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55db7112fae0 con 0x55db7117d400 2024-06-04T05:13:04.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.266+0000 7fa0117fa700 10 client.19584 handle_command_reply: tid=0 2024-06-04T05:13:04.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.266+0000 7fa03a5c0b80 1 client.19584 shutdown 2024-06-04T05:13:04.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.266+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/3144741016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 msgr2=0x55db7116d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:04.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.266+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/3144741016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db7116d0f0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fa01800aad0 tx=0x7fa018003db0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.266+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/3144741016 shutdown_connections 2024-06-04T05:13:04.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.266+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/3144741016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711692b0 0x55db7116d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.266+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/3144741016 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55db7117d400 0x55db7117f840 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.266+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/3144741016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711638e0 0x55db7116fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.266+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/3144741016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db7116d0f0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.266+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/3144741016 >> 172.21.15.70:0/3144741016 conn(0x55db71091180 msgr2=0x55db71158ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.266+0000 7fa0117fa700 0 client.19584 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.266+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/3144741016 shutdown_connections 2024-06-04T05:13:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.266+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/3144741016 wait complete. 2024-06-04T05:13:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.266+0000 7fa03a5c0b80 20 client.19584 trim_cache size 0 max 16384 2024-06-04T05:13:04.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.286+0000 7fa03a5c0b80 1 Processor -- start 2024-06-04T05:13:04.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.286+0000 7fa03a5c0b80 1 -- start start 2024-06-04T05:13:04.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.286+0000 7fa03a5c0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711638e0 0x55db71168970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.286+0000 7fa03a5c0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db7116f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.286+0000 7fa03a5c0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711692b0 0x55db71185110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.286+0000 7fa03a5c0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa0281addd0 con 0x55db711642a0 2024-06-04T05:13:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.286+0000 7fa03a5c0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa0281afa60 con 0x55db711638e0 2024-06-04T05:13:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.286+0000 7fa03a5c0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa0281ae4a0 con 0x55db711692b0 2024-06-04T05:13:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.286+0000 7fa02d2c6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db7116f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.287+0000 7fa02d2c6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db7116f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51146/0 (socket says 172.21.15.70:51146) 2024-06-04T05:13:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.287+0000 7fa02d2c6700 1 -- 172.21.15.70:0/2846229489 learned_addr learned my addr 172.21.15.70:0/2846229489 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.287+0000 7fa02dac7700 1 --2- 172.21.15.70:0/2846229489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711638e0 0x55db71168970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.287+0000 7fa02d2c6700 1 -- 172.21.15.70:0/2846229489 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711692b0 msgr2=0x55db71185110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.287+0000 7fa02e2c8700 1 --2- 172.21.15.70:0/2846229489 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711692b0 0x55db71185110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.287+0000 7fa02d2c6700 1 --2- 172.21.15.70:0/2846229489 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711692b0 0x55db71185110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.287+0000 7fa02d2c6700 1 -- 172.21.15.70:0/2846229489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711638e0 msgr2=0x55db71168970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.287+0000 7fa02d2c6700 1 --2- 172.21.15.70:0/2846229489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711638e0 0x55db71168970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.287+0000 7fa02d2c6700 1 -- 172.21.15.70:0/2846229489 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db7115a9f0 con 0x55db711642a0 2024-06-04T05:13:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.287+0000 7fa02dac7700 1 --2- 172.21.15.70:0/2846229489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711638e0 0x55db71168970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:13:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.287+0000 7fa02e2c8700 1 --2- 172.21.15.70:0/2846229489 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711692b0 0x55db71185110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:13:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.287+0000 7fa02d2c6700 1 --2- 172.21.15.70:0/2846229489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db7116f930 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fa01800c690 tx=0x7fa018005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ad1f6a7121ee84 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.287+0000 7fa0137fe700 1 -- 172.21.15.70:0/2846229489 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01800c0f0 con 0x55db711642a0 2024-06-04T05:13:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.287+0000 7fa0137fe700 1 -- 172.21.15.70:0/2846229489 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa018027070 con 0x55db711642a0 2024-06-04T05:13:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.287+0000 7fa0137fe700 1 -- 172.21.15.70:0/2846229489 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa018022e00 con 0x55db711642a0 2024-06-04T05:13:04.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.288+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/2846229489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 msgr2=0x55db7116f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:04.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.288+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/2846229489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db7116f930 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fa01800c690 tx=0x7fa018005f00 comp rx=0 tx=0).stop 2024-06-04T05:13:04.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.288+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/2846229489 shutdown_connections 2024-06-04T05:13:04.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.288+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/2846229489 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711692b0 0x55db71185110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.288+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/2846229489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711638e0 0x55db71168970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.288+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/2846229489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711642a0 0x55db7116f930 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.288+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/2846229489 >> 172.21.15.70:0/2846229489 conn(0x55db71091180 msgr2=0x55db71158ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:04.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.288+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/2846229489 shutdown_connections 2024-06-04T05:13:04.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.288+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/2846229489 wait complete. 2024-06-04T05:13:04.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.288+0000 7fa03a5c0b80 1 -- start start 2024-06-04T05:13:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.288+0000 7fa03a5c0b80 1 Processor -- start 2024-06-04T05:13:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa03a5c0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711638e0 0x55db711730b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa03a5c0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711642a0 0x55db711735f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa03a5c0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711692b0 0x55db711e1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa03a5c0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa0281af2d0 con 0x55db711638e0 2024-06-04T05:13:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa03a5c0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa0281aebd0 con 0x55db711642a0 2024-06-04T05:13:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa03a5c0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa0281abd90 con 0x55db711692b0 2024-06-04T05:13:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa03a5c0b80 10 client.0 _ll_register_callbacks cb 0x55db7104da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa02dac7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711638e0 0x55db711730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa02dac7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711638e0 0x55db711730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51158/0 (socket says 172.21.15.70:51158) 2024-06-04T05:13:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa02dac7700 1 -- 172.21.15.70:0/3314035103 learned_addr learned my addr 172.21.15.70:0/3314035103 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa02d2c6700 1 --2- 172.21.15.70:0/3314035103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711642a0 0x55db711735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa02e2c8700 1 --2- 172.21.15.70:0/3314035103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711692b0 0x55db711e1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa02dac7700 1 -- 172.21.15.70:0/3314035103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711692b0 msgr2=0x55db711e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa02dac7700 1 --2- 172.21.15.70:0/3314035103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711692b0 0x55db711e1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa02dac7700 1 -- 172.21.15.70:0/3314035103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711642a0 msgr2=0x55db711735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa02dac7700 1 --2- 172.21.15.70:0/3314035103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711642a0 0x55db711735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa02dac7700 1 -- 172.21.15.70:0/3314035103 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db711e2640 con 0x55db711638e0 2024-06-04T05:13:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa02e2c8700 1 --2- 172.21.15.70:0/3314035103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711692b0 0x55db711e1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:13:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa02dac7700 1 --2- 172.21.15.70:0/3314035103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711638e0 0x55db711730b0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x55db711800b0 tx=0x7fa0240047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d654c018d2cfbe91 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa0117fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:13:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.289+0000 7fa0117fa700 1 -- 172.21.15.70:0/3314035103 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02402a030 con 0x55db711638e0 2024-06-04T05:13:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.290+0000 7fa0117fa700 1 -- 172.21.15.70:0/3314035103 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8615+0+0 (secure 0 0 0) 0x7fa024010d00 con 0x55db711638e0 2024-06-04T05:13:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.290+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/3314035103 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa018027070 con 0x55db711638e0 2024-06-04T05:13:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.290+0000 7fa0117fa700 1 -- 172.21.15.70:0/3314035103 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fa02400eae0 con 0x55db711638e0 2024-06-04T05:13:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.290+0000 7fa03a5c0b80 10 client.19590 fetch_fsmap learned FSMap version 45 2024-06-04T05:13:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.290+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/3314035103 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55db711e9940 con 0x55db711638e0 2024-06-04T05:13:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.290+0000 7fa0117fa700 1 -- 172.21.15.70:0/3314035103 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa018027070 con 0x55db711638e0 2024-06-04T05:13:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.290+0000 7fa0117fa700 1 -- 172.21.15.70:0/3314035103 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 45) ==== 8398+0+0 (secure 0 0 0) 0x7fa02401d440 con 0x55db711638e0 2024-06-04T05:13:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.290+0000 7fa03a5c0b80 10 client.19590 fetch_fsmap finished waiting for FSMap version 45 2024-06-04T05:13:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.290+0000 7fa03a5c0b80 10 client.19590 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:13:04.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.290+0000 7fa03a5c0b80 20 client.19590 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:04.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.290+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/3314035103 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55db711ea110 0x55db711ec550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:04.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.290+0000 7fa03a5c0b80 4 client.19590 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "e648773", "release": true, "await": true}] 2024-06-04T05:13:04.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.290+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/3314035103 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "e648773", "release": true, "await": true}) -- 0x55db710c4ce0 con 0x55db711ea110 2024-06-04T05:13:04.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.290+0000 7fa02d2c6700 1 --2- 172.21.15.70:0/3314035103 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55db711ea110 0x55db711ec550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:04.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.291+0000 7fa0117fa700 10 client.19590 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:04.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.291+0000 7fa02d2c6700 1 --2- 172.21.15.70:0/3314035103 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55db711ea110 0x55db711ec550 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c532a2e1ea1102f server_cookie=4c2ea9cf66639939 in_seq=0 out_seq=0 2024-06-04T05:13:04.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.323+0000 7fa0117fa700 1 -- 172.21.15.70:0/3314035103 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55db710c4ce0 con 0x55db711ea110 2024-06-04T05:13:04.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.323+0000 7fa0117fa700 10 client.19590 handle_command_reply: tid=0 2024-06-04T05:13:04.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.323+0000 7fa03a5c0b80 1 client.19590 shutdown 2024-06-04T05:13:04.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.323+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/3314035103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711638e0 msgr2=0x55db711730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:04.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.323+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/3314035103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711638e0 0x55db711730b0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x55db711800b0 tx=0x7fa0240047c0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.323+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/3314035103 shutdown_connections 2024-06-04T05:13:04.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.323+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/3314035103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55db711692b0 0x55db711e1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.323+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/3314035103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55db711638e0 0x55db711730b0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.323+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/3314035103 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55db711ea110 0x55db711ec550 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.323+0000 7fa03a5c0b80 1 --2- 172.21.15.70:0/3314035103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55db711642a0 0x55db711735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.323+0000 7fa0117fa700 0 client.19590 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:04.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.323+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/3314035103 >> 172.21.15.70:0/3314035103 conn(0x55db71091180 msgr2=0x55db71158410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:04.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.323+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/3314035103 shutdown_connections 2024-06-04T05:13:04.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.324+0000 7fa03a5c0b80 1 -- 172.21.15.70:0/3314035103 wait complete. 2024-06-04T05:13:04.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.324+0000 7fa03a5c0b80 20 client.19590 trim_cache size 0 max 16384 2024-06-04T05:13:04.327 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:13:04.327 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 45, 2024-06-04T05:13:04.327 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:13:04.327 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 48, 2024-06-04T05:13:04.327 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:13:04.327 INFO:teuthology.orchestra.run.smithi070.stdout: "e648773": { 2024-06-04T05:13:04.327 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 48, 2024-06-04T05:13:04.327 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:13:04.327 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:13:04.328 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:13:04.328 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:13:04.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:13:04.328 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:13:04.328 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 152.6, 2024-06-04T05:13:04.328 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:13:04.328 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:13:04.328 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:13:04.328 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:13:04.328 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:13:04.328 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:13:04.328 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:13:04.328 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:13:04.328 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:13:04.328 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:13:04.329 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:13:04.329 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:13:04.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.324+0000 7fa02fd2b700 1 -- 172.21.15.70:0/1597315364 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa0140b29f0 msgr2=0x7fa0140b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:04.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.324+0000 7fa02fd2b700 1 --2- 172.21.15.70:0/1597315364 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa0140b29f0 0x7fa0140b4eb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa024005ea0 tx=0x7fa024005e30 comp rx=0 tx=0).stop 2024-06-04T05:13:04.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.324+0000 7fa02fd2b700 1 -- 172.21.15.70:0/1597315364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa028106f20 msgr2=0x7fa028199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:04.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.324+0000 7fa02fd2b700 1 --2- 172.21.15.70:0/1597315364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa028106f20 0x7fa028199ab0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fa02000b9e0 tx=0x7fa02000bda0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.325+0000 7fa02fd2b700 1 -- 172.21.15.70:0/1597315364 shutdown_connections 2024-06-04T05:13:04.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.325+0000 7fa02fd2b700 1 --2- 172.21.15.70:0/1597315364 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa0140b29f0 0x7fa0140b4eb0 secure :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa024005ea0 tx=0x7fa024005e30 comp rx=0 tx=0).stop 2024-06-04T05:13:04.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.325+0000 7fa02fd2b700 1 --2- 172.21.15.70:0/1597315364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa028106f20 0x7fa028199ab0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.325+0000 7fa02fd2b700 1 --2- 172.21.15.70:0/1597315364 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa028101e40 0x7fa02819f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.325+0000 7fa02fd2b700 1 --2- 172.21.15.70:0/1597315364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0281027f0 0x7fa02819fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:04.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.325+0000 7fa02fd2b700 1 -- 172.21.15.70:0/1597315364 >> 172.21.15.70:0/1597315364 conn(0x7fa028078a40 msgr2=0x7fa028105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:04.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.325+0000 7fa02fd2b700 1 -- 172.21.15.70:0/1597315364 shutdown_connections 2024-06-04T05:13:04.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:04.325+0000 7fa02fd2b700 1 -- 172.21.15.70:0/1597315364 wait complete. 2024-06-04T05:13:04.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:04 smithi070 ceph-mon[32631]: osdmap e99: 12 total, 12 up, 12 in 2024-06-04T05:13:04.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:04 smithi112 ceph-mon[36453]: osdmap e99: 12 total, 12 up, 12 in 2024-06-04T05:13:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:04 smithi098 ceph-mon[35569]: osdmap e99: 12 total, 12 up, 12 in 2024-06-04T05:13:04.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15710 8.74 MB/sec warmup 116 sec latency 27155.573 ms 2024-06-04T05:13:05.402 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'e648773', total seconds elapsed: 42.1 2024-06-04T05:13:05.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:05 smithi070 ceph-mon[32631]: pgmap v497: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 62 KiB/s wr, 32 op/s 2024-06-04T05:13:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:05 smithi112 ceph-mon[36453]: pgmap v497: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 62 KiB/s wr, 32 op/s 2024-06-04T05:13:05.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:05 smithi098 ceph-mon[35569]: pgmap v497: 97 pgs: 97 active+clean; 423 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 62 KiB/s wr, 32 op/s 2024-06-04T05:13:05.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 15927 8.81 MB/sec warmup 117 sec latency 46.720 ms 2024-06-04T05:13:06.437 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:13:06.793 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:13:06.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 16116 8.81 MB/sec warmup 118 sec latency 54.984 ms 2024-06-04T05:13:07.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:07 smithi070 ceph-mon[32631]: pgmap v498: 97 pgs: 97 active+clean; 430 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.1 MiB/s wr, 87 op/s 2024-06-04T05:13:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:07 smithi112 ceph-mon[36453]: pgmap v498: 97 pgs: 97 active+clean; 430 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.1 MiB/s wr, 87 op/s 2024-06-04T05:13:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:07 smithi098 ceph-mon[35569]: pgmap v498: 97 pgs: 97 active+clean; 430 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.1 MiB/s wr, 87 op/s 2024-06-04T05:13:07.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 16282 8.81 MB/sec warmup 119 sec latency 55.822 ms 2024-06-04T05:13:08.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.710+0000 7f637adb0700 1 -- 172.21.15.70:0/1610182453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6374102430 msgr2=0x7f63741028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:08.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.710+0000 7f637adb0700 1 --2- 172.21.15.70:0/1610182453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6374102430 0x7f63741028b0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f6360009990 tx=0x7f6360009ca0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.710+0000 7f637adb0700 1 -- 172.21.15.70:0/1610182453 shutdown_connections 2024-06-04T05:13:08.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.710+0000 7f637adb0700 1 --2- 172.21.15.70:0/1610182453 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6374102df0 0x7f637410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.710+0000 7f637adb0700 1 --2- 172.21.15.70:0/1610182453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6374108510 0x7f63741088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.710+0000 7f637adb0700 1 --2- 172.21.15.70:0/1610182453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6374102430 0x7f63741028b0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.710+0000 7f637adb0700 1 -- 172.21.15.70:0/1610182453 >> 172.21.15.70:0/1610182453 conn(0x7f63740793b0 msgr2=0x7f63740797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:08.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.710+0000 7f637adb0700 1 -- 172.21.15.70:0/1610182453 shutdown_connections 2024-06-04T05:13:08.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.710+0000 7f637adb0700 1 -- 172.21.15.70:0/1610182453 wait complete. 2024-06-04T05:13:08.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.711+0000 7f637adb0700 1 Processor -- start 2024-06-04T05:13:08.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.711+0000 7f637adb0700 1 -- start start 2024-06-04T05:13:08.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.711+0000 7f637adb0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6374102430 0x7f637419f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.711+0000 7f637adb0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6374102df0 0x7f637419fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.711+0000 7f637adb0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6374108510 0x7f6374199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.712+0000 7f637adb0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f63741124f0 con 0x7f6374102430 2024-06-04T05:13:08.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.712+0000 7f637adb0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6374112370 con 0x7f6374108510 2024-06-04T05:13:08.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.712+0000 7f637adb0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6374112670 con 0x7f6374102df0 2024-06-04T05:13:08.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.712+0000 7f6378b4c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6374102430 0x7f637419f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.712+0000 7f6373fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6374102df0 0x7f637419fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.712+0000 7f6378b4c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6374102430 0x7f637419f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43078/0 (socket says 172.21.15.70:43078) 2024-06-04T05:13:08.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.712+0000 7f6378b4c700 1 -- 172.21.15.70:0/1986503940 learned_addr learned my addr 172.21.15.70:0/1986503940 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:08.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.712+0000 7f637934d700 1 --2- 172.21.15.70:0/1986503940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6374108510 0x7f6374199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.712+0000 7f637934d700 1 -- 172.21.15.70:0/1986503940 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6374102df0 msgr2=0x7f637419fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:08.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.712+0000 7f637934d700 1 --2- 172.21.15.70:0/1986503940 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6374102df0 0x7f637419fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.712+0000 7f637934d700 1 -- 172.21.15.70:0/1986503940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6374102430 msgr2=0x7f637419f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:08.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.712+0000 7f637934d700 1 --2- 172.21.15.70:0/1986503940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6374102430 0x7f637419f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.712+0000 7f637934d700 1 -- 172.21.15.70:0/1986503940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637419a1a0 con 0x7f6374108510 2024-06-04T05:13:08.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.712+0000 7f6378b4c700 1 --2- 172.21.15.70:0/1986503940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6374102430 0x7f637419f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:08.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.712+0000 7f637934d700 1 --2- 172.21.15.70:0/1986503940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6374108510 0x7f6374199a40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f636c00ff40 tx=0x7f636c00ff70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:08.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.712+0000 7f6371ffb700 1 -- 172.21.15.70:0/1986503940 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c00d4e0 con 0x7f6374108510 2024-06-04T05:13:08.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.713+0000 7f6371ffb700 1 -- 172.21.15.70:0/1986503940 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f636c010d10 con 0x7f6374108510 2024-06-04T05:13:08.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.713+0000 7f637adb0700 1 -- 172.21.15.70:0/1986503940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f637419a430 con 0x7f6374108510 2024-06-04T05:13:08.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.713+0000 7f637adb0700 1 -- 172.21.15.70:0/1986503940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6374103fa0 con 0x7f6374108510 2024-06-04T05:13:08.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.713+0000 7f6371ffb700 1 -- 172.21.15.70:0/1986503940 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c0183f0 con 0x7f6374108510 2024-06-04T05:13:08.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.716+0000 7f6371ffb700 1 -- 172.21.15.70:0/1986503940 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f636c00b750 con 0x7f6374108510 2024-06-04T05:13:08.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.717+0000 7f6371ffb700 1 --2- 172.21.15.70:0/1986503940 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f63640b2a70 0x7f63640b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.717+0000 7f6378b4c700 1 --2- 172.21.15.70:0/1986503940 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f63640b2a70 0x7f63640b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.717+0000 7f6371ffb700 1 -- 172.21.15.70:0/1986503940 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f636c0e5a90 con 0x7f6374108510 2024-06-04T05:13:08.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.718+0000 7f6378b4c700 1 --2- 172.21.15.70:0/1986503940 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f63640b2a70 0x7f63640b4f30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f637419ab80 tx=0x7f6368009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:08.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.718+0000 7f6385645b80 1 Processor -- start 2024-06-04T05:13:08.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.718+0000 7f6385645b80 1 -- start start 2024-06-04T05:13:08.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.719+0000 7f6385645b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d0860 0x561aa61d0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.719+0000 7f6385645b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d1210 0x561aa61d4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.719+0000 7f6385645b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d4f60 0x561aa61d53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.719+0000 7f6385645b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561aa612fc90 con 0x561aa61d1210 2024-06-04T05:13:08.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.719+0000 7f6385645b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561aa611f830 con 0x561aa61d4f60 2024-06-04T05:13:08.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.719+0000 7f6373fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d1210 0x561aa61d4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.719+0000 7f6373fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d1210 0x561aa61d4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43080/0 (socket says 172.21.15.70:43080) 2024-06-04T05:13:08.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.719+0000 7f6373fff700 1 -- 172.21.15.70:0/3831535963 learned_addr learned my addr 172.21.15.70:0/3831535963 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:08.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.719+0000 7f6385645b80 1 -- 172.21.15.70:0/3831535963 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561aa6123870 con 0x561aa61d0860 2024-06-04T05:13:08.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.719+0000 7f6378b4c700 1 --2- 172.21.15.70:0/3831535963 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d0860 0x561aa61d0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.719+0000 7f637934d700 1 --2- 172.21.15.70:0/3831535963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d4f60 0x561aa61d53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.720+0000 7f6378b4c700 1 -- 172.21.15.70:0/3831535963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d4f60 msgr2=0x561aa61d53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:08.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.720+0000 7f6378b4c700 1 --2- 172.21.15.70:0/3831535963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d4f60 0x561aa61d53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.720+0000 7f6378b4c700 1 -- 172.21.15.70:0/3831535963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d1210 msgr2=0x561aa61d4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:08.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.720+0000 7f6378b4c700 1 --2- 172.21.15.70:0/3831535963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d1210 0x561aa61d4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.720+0000 7f6378b4c700 1 -- 172.21.15.70:0/3831535963 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561aa606f8d0 con 0x561aa61d0860 2024-06-04T05:13:08.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.720+0000 7f6373fff700 1 --2- 172.21.15.70:0/3831535963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d1210 0x561aa61d4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:08.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.720+0000 7f6378b4c700 1 --2- 172.21.15.70:0/3831535963 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d0860 0x561aa61d0c40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f63680042e0 tx=0x7f6368014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c65a381180ea1d8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:08.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.720+0000 7f635effd700 1 -- 172.21.15.70:0/3831535963 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6368003d10 con 0x561aa61d0860 2024-06-04T05:13:08.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.721+0000 7f635effd700 1 -- 172.21.15.70:0/3831535963 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f636801e8b0 con 0x561aa61d0860 2024-06-04T05:13:08.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.721+0000 7f635effd700 1 -- 172.21.15.70:0/3831535963 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63680266f0 con 0x561aa61d0860 2024-06-04T05:13:08.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.722+0000 7f6385645b80 1 -- 172.21.15.70:0/3831535963 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d0860 msgr2=0x561aa61d0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:08.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.722+0000 7f6385645b80 1 --2- 172.21.15.70:0/3831535963 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d0860 0x561aa61d0c40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f63680042e0 tx=0x7f6368014000 comp rx=0 tx=0).stop 2024-06-04T05:13:08.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.722+0000 7f6385645b80 1 -- 172.21.15.70:0/3831535963 shutdown_connections 2024-06-04T05:13:08.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.722+0000 7f6385645b80 1 --2- 172.21.15.70:0/3831535963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d4f60 0x561aa61d53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.722+0000 7f6385645b80 1 --2- 172.21.15.70:0/3831535963 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d0860 0x561aa61d0c40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.722+0000 7f6385645b80 1 --2- 172.21.15.70:0/3831535963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d1210 0x561aa61d4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.722+0000 7f6385645b80 1 -- 172.21.15.70:0/3831535963 >> 172.21.15.70:0/3831535963 conn(0x561aa60fe080 msgr2=0x561aa61cae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:08.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.722+0000 7f6385645b80 1 -- 172.21.15.70:0/3831535963 shutdown_connections 2024-06-04T05:13:08.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.722+0000 7f6385645b80 1 -- 172.21.15.70:0/3831535963 wait complete. 2024-06-04T05:13:08.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.722+0000 7f6385645b80 1 -- start start 2024-06-04T05:13:08.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.723+0000 7f6385645b80 1 Processor -- start 2024-06-04T05:13:08.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.723+0000 7f6385645b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d0860 0x561aa61dc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.723+0000 7f6385645b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d1210 0x561aa61d9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.724+0000 7f6385645b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d4f60 0x561aa61d9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.724+0000 7f6385645b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561aa6025fa0 con 0x561aa61d4f60 2024-06-04T05:13:08.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.724+0000 7f6385645b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561aa60c17c0 con 0x561aa61d1210 2024-06-04T05:13:08.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.724+0000 7f6385645b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561aa61dd170 con 0x561aa61d0860 2024-06-04T05:13:08.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.724+0000 7f6378b4c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d0860 0x561aa61dc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.724+0000 7f6385645b80 10 client.0 _ll_register_callbacks cb 0x561aa60ba920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:08.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.724+0000 7f6378b4c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d0860 0x561aa61dc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41912/0 (socket says 172.21.15.70:41912) 2024-06-04T05:13:08.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.724+0000 7f6378b4c700 1 -- 172.21.15.70:0/3296891326 learned_addr learned my addr 172.21.15.70:0/3296891326 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:08.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.724+0000 7f637934d700 1 --2- 172.21.15.70:0/3296891326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d4f60 0x561aa61d9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.724+0000 7f6373fff700 1 --2- 172.21.15.70:0/3296891326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d1210 0x561aa61d9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.724+0000 7f6378b4c700 1 -- 172.21.15.70:0/3296891326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d1210 msgr2=0x561aa61d9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:08.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.724+0000 7f6378b4c700 1 --2- 172.21.15.70:0/3296891326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d1210 0x561aa61d9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.724+0000 7f6378b4c700 1 -- 172.21.15.70:0/3296891326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d4f60 msgr2=0x561aa61d9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:08.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.724+0000 7f6378b4c700 1 --2- 172.21.15.70:0/3296891326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d4f60 0x561aa61d9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.724+0000 7f6378b4c700 1 -- 172.21.15.70:0/3296891326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561aa60be690 con 0x561aa61d0860 2024-06-04T05:13:08.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.724+0000 7f6378b4c700 1 --2- 172.21.15.70:0/3296891326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d0860 0x561aa61dc440 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f636801e820 tx=0x7f63680057d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c76de89a63cfa937 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:08.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.724+0000 7f6353fff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:13:08.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.725+0000 7f6353fff700 1 -- 172.21.15.70:0/3296891326 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6368005230 con 0x561aa61d0860 2024-06-04T05:13:08.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.725+0000 7f6353fff700 1 -- 172.21.15.70:0/3296891326 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f63680053d0 con 0x561aa61d0860 2024-06-04T05:13:08.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.725+0000 7f6385645b80 1 -- 172.21.15.70:0/3296891326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f636801e8b0 con 0x561aa61d0860 2024-06-04T05:13:08.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.725+0000 7f6353fff700 1 -- 172.21.15.70:0/3296891326 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f63680046b0 con 0x561aa61d0860 2024-06-04T05:13:08.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.725+0000 7f6385645b80 10 client.28552 fetch_fsmap learned FSMap version 45 2024-06-04T05:13:08.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.725+0000 7f6385645b80 1 -- 172.21.15.70:0/3296891326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x561aa61e7750 con 0x561aa61d0860 2024-06-04T05:13:08.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.725+0000 7f6353fff700 1 -- 172.21.15.70:0/3296891326 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f636801e8b0 con 0x561aa61d0860 2024-06-04T05:13:08.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.726+0000 7f6353fff700 1 -- 172.21.15.70:0/3296891326 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 45) ==== 8398+0+0 (secure 0 0 0) 0x7f636802f730 con 0x561aa61d0860 2024-06-04T05:13:08.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.726+0000 7f6385645b80 10 client.28552 fetch_fsmap finished waiting for FSMap version 45 2024-06-04T05:13:08.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.726+0000 7f6385645b80 10 client.28552 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:08.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.726+0000 7f6385645b80 20 client.28552 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:08.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.726+0000 7f6385645b80 1 --2- 172.21.15.70:0/3296891326 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561aa61e9c20 0x561aa61ec010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.726+0000 7f6385645b80 4 client.28552 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:13:08.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.726+0000 7f6385645b80 1 -- 172.21.15.70:0/3296891326 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561aa619c9e0 con 0x561aa61e9c20 2024-06-04T05:13:08.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.726+0000 7f6373fff700 1 --2- 172.21.15.70:0/3296891326 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561aa61e9c20 0x561aa61ec010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.727+0000 7f6353fff700 10 client.28552 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:08.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.727+0000 7f6373fff700 1 --2- 172.21.15.70:0/3296891326 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561aa61e9c20 0x561aa61ec010 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccc301be7a465f17 server_cookie=9bf98584c57612fb in_seq=0 out_seq=0 2024-06-04T05:13:08.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.728+0000 7f6353fff700 1 -- 172.21.15.70:0/3296891326 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561aa619c9e0 con 0x561aa61e9c20 2024-06-04T05:13:08.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.728+0000 7f6353fff700 10 client.28552 handle_command_reply: tid=0 2024-06-04T05:13:08.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.728+0000 7f6385645b80 1 client.28552 shutdown 2024-06-04T05:13:08.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.728+0000 7f6385645b80 1 -- 172.21.15.70:0/3296891326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d0860 msgr2=0x561aa61dc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:08.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.728+0000 7f6385645b80 1 --2- 172.21.15.70:0/3296891326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d0860 0x561aa61dc440 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f636801e820 tx=0x7f63680057d0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.728+0000 7f6385645b80 1 -- 172.21.15.70:0/3296891326 shutdown_connections 2024-06-04T05:13:08.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.728+0000 7f6385645b80 1 --2- 172.21.15.70:0/3296891326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d4f60 0x561aa61d9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.728+0000 7f6385645b80 1 --2- 172.21.15.70:0/3296891326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d0860 0x561aa61dc440 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.728+0000 7f6385645b80 1 --2- 172.21.15.70:0/3296891326 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561aa61e9c20 0x561aa61ec010 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.728+0000 7f6385645b80 1 --2- 172.21.15.70:0/3296891326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d1210 0x561aa61d9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.728+0000 7f6385645b80 1 -- 172.21.15.70:0/3296891326 >> 172.21.15.70:0/3296891326 conn(0x561aa60fe080 msgr2=0x561aa61ca510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:08.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.728+0000 7f6353fff700 0 client.28552 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:08.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.728+0000 7f6385645b80 1 -- 172.21.15.70:0/3296891326 shutdown_connections 2024-06-04T05:13:08.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.728+0000 7f6385645b80 1 -- 172.21.15.70:0/3296891326 wait complete. 2024-06-04T05:13:08.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.728+0000 7f6385645b80 20 client.28552 trim_cache size 0 max 16384 2024-06-04T05:13:08.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.765+0000 7f6385645b80 1 Processor -- start 2024-06-04T05:13:08.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.765+0000 7f6385645b80 1 -- start start 2024-06-04T05:13:08.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.765+0000 7f6385645b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d0860 0x561aa61cac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.765+0000 7f6385645b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d1210 0x561aa61f1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.765+0000 7f6385645b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d4f60 0x561aa61f1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.765+0000 7f6385645b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f63741abd20 con 0x561aa61d4f60 2024-06-04T05:13:08.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.765+0000 7f6385645b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63741ad9b0 con 0x561aa61d0860 2024-06-04T05:13:08.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.765+0000 7f6385645b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f63741ac3f0 con 0x561aa61d1210 2024-06-04T05:13:08.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.765+0000 7f637934d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d4f60 0x561aa61f1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.765+0000 7f637934d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d4f60 0x561aa61f1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43092/0 (socket says 172.21.15.70:43092) 2024-06-04T05:13:08.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.765+0000 7f6373fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d1210 0x561aa61f1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.765+0000 7f637934d700 1 -- 172.21.15.70:0/2818747925 learned_addr learned my addr 172.21.15.70:0/2818747925 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:08.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.766+0000 7f6378b4c700 1 --2- 172.21.15.70:0/2818747925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d0860 0x561aa61cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.766+0000 7f6373fff700 1 -- 172.21.15.70:0/2818747925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d0860 msgr2=0x561aa61cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:08.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.766+0000 7f6373fff700 1 --2- 172.21.15.70:0/2818747925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d0860 0x561aa61cac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.766+0000 7f6373fff700 1 -- 172.21.15.70:0/2818747925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d4f60 msgr2=0x561aa61f1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:08.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.766+0000 7f6373fff700 1 --2- 172.21.15.70:0/2818747925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d4f60 0x561aa61f1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.766+0000 7f6373fff700 1 -- 172.21.15.70:0/2818747925 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561aa61da680 con 0x561aa61d1210 2024-06-04T05:13:08.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.766+0000 7f6378b4c700 1 --2- 172.21.15.70:0/2818747925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d0860 0x561aa61cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:13:08.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.766+0000 7f6373fff700 1 --2- 172.21.15.70:0/2818747925 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d1210 0x561aa61f1670 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6360000c00 tx=0x7f63600054a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=506e8983d053a542 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:08.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.766+0000 7f635e7fc700 1 -- 172.21.15.70:0/2818747925 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63600039e0 con 0x561aa61d1210 2024-06-04T05:13:08.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.766+0000 7f635e7fc700 1 -- 172.21.15.70:0/2818747925 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6360003b80 con 0x561aa61d1210 2024-06-04T05:13:08.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.766+0000 7f635e7fc700 1 -- 172.21.15.70:0/2818747925 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636000be10 con 0x561aa61d1210 2024-06-04T05:13:08.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.767+0000 7f6385645b80 1 -- 172.21.15.70:0/2818747925 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d1210 msgr2=0x561aa61f1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:08.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.767+0000 7f6385645b80 1 --2- 172.21.15.70:0/2818747925 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d1210 0x561aa61f1670 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6360000c00 tx=0x7f63600054a0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.767+0000 7f6385645b80 1 -- 172.21.15.70:0/2818747925 shutdown_connections 2024-06-04T05:13:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.767+0000 7f6385645b80 1 --2- 172.21.15.70:0/2818747925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d4f60 0x561aa61f1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.767+0000 7f6385645b80 1 --2- 172.21.15.70:0/2818747925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d0860 0x561aa61cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.767+0000 7f6385645b80 1 --2- 172.21.15.70:0/2818747925 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d1210 0x561aa61f1670 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.767+0000 7f6385645b80 1 -- 172.21.15.70:0/2818747925 >> 172.21.15.70:0/2818747925 conn(0x561aa60fe080 msgr2=0x561aa61cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.768+0000 7f6385645b80 1 -- 172.21.15.70:0/2818747925 shutdown_connections 2024-06-04T05:13:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.768+0000 7f6385645b80 1 -- 172.21.15.70:0/2818747925 wait complete. 2024-06-04T05:13:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.768+0000 7f6385645b80 1 -- start start 2024-06-04T05:13:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.768+0000 7f6385645b80 1 Processor -- start 2024-06-04T05:13:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f6385645b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d0860 0x561aa61e18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f6385645b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d1210 0x561aa61e1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f6385645b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d4f60 0x561aa6252550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f6378b4c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d0860 0x561aa61e18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f6373fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d1210 0x561aa61e1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f6378b4c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d0860 0x561aa61e18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:42900/0 (socket says 172.21.15.70:42900) 2024-06-04T05:13:08.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f6373fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d1210 0x561aa61e1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43098/0 (socket says 172.21.15.70:43098) 2024-06-04T05:13:08.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f6385645b80 1 -- 172.21.15.70:0/3746387896 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f63741ad220 con 0x561aa61d1210 2024-06-04T05:13:08.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f6378b4c700 1 -- 172.21.15.70:0/3746387896 learned_addr learned my addr 172.21.15.70:0/3746387896 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:08.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f6385645b80 1 -- 172.21.15.70:0/3746387896 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63741acb20 con 0x561aa61d0860 2024-06-04T05:13:08.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f6385645b80 1 -- 172.21.15.70:0/3746387896 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f63741a9ce0 con 0x561aa61d4f60 2024-06-04T05:13:08.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f6385645b80 10 client.0 _ll_register_callbacks cb 0x561aa60ba920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:08.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f637934d700 1 --2- 172.21.15.70:0/3746387896 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d4f60 0x561aa6252550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f637934d700 1 -- 172.21.15.70:0/3746387896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d0860 msgr2=0x561aa61e18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:08.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f637934d700 1 --2- 172.21.15.70:0/3746387896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d0860 0x561aa61e18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f637934d700 1 -- 172.21.15.70:0/3746387896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d1210 msgr2=0x561aa61e1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:08.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f637934d700 1 --2- 172.21.15.70:0/3746387896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d1210 0x561aa61e1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.769+0000 7f637934d700 1 -- 172.21.15.70:0/3746387896 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561aa6252ac0 con 0x561aa61d4f60 2024-06-04T05:13:08.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.770+0000 7f637934d700 1 --2- 172.21.15.70:0/3746387896 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d4f60 0x561aa6252550 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f636c0aa0c0 tx=0x7f636c004be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee32f617eead3168 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:08.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.770+0000 7f6353fff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:13:08.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.770+0000 7f6353fff700 1 -- 172.21.15.70:0/3746387896 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c0ad550 con 0x561aa61d4f60 2024-06-04T05:13:08.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.770+0000 7f6385645b80 1 -- 172.21.15.70:0/3746387896 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6360003b80 con 0x561aa61d4f60 2024-06-04T05:13:08.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.770+0000 7f6353fff700 1 -- 172.21.15.70:0/3746387896 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f636c0a9440 con 0x561aa61d4f60 2024-06-04T05:13:08.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.770+0000 7f6353fff700 1 -- 172.21.15.70:0/3746387896 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f636c0ace50 con 0x561aa61d4f60 2024-06-04T05:13:08.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.770+0000 7f6385645b80 10 client.28558 fetch_fsmap learned FSMap version 45 2024-06-04T05:13:08.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.770+0000 7f6385645b80 1 -- 172.21.15.70:0/3746387896 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x561aa6253dd0 con 0x561aa61d4f60 2024-06-04T05:13:08.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.770+0000 7f6353fff700 1 -- 172.21.15.70:0/3746387896 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6360003b80 con 0x561aa61d4f60 2024-06-04T05:13:08.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.770+0000 7f6353fff700 1 -- 172.21.15.70:0/3746387896 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 45) ==== 8398+0+0 (secure 0 0 0) 0x7f636c0c4050 con 0x561aa61d4f60 2024-06-04T05:13:08.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.770+0000 7f6385645b80 10 client.28558 fetch_fsmap finished waiting for FSMap version 45 2024-06-04T05:13:08.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.770+0000 7f6385645b80 10 client.28558 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:08.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.770+0000 7f6385645b80 20 client.28558 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:08.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.771+0000 7f6385645b80 1 --2- 172.21.15.70:0/3746387896 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561aa6254540 0x561aa6256960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.771+0000 7f6385645b80 4 client.28558 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:13:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.771+0000 7f6385645b80 1 -- 172.21.15.70:0/3746387896 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561aa6131be0 con 0x561aa6254540 2024-06-04T05:13:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.771+0000 7f6378b4c700 1 --2- 172.21.15.70:0/3746387896 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561aa6254540 0x561aa6256960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.771+0000 7f6353fff700 10 client.28558 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.771+0000 7f6378b4c700 1 --2- 172.21.15.70:0/3746387896 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561aa6254540 0x561aa6256960 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b75d487f3d12fca2 server_cookie=b6a85c32f3db884b in_seq=0 out_seq=0 2024-06-04T05:13:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.771+0000 7f6353fff700 1 -- 172.21.15.70:0/3746387896 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561aa6131be0 con 0x561aa6254540 2024-06-04T05:13:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.771+0000 7f6353fff700 10 client.28558 handle_command_reply: tid=0 2024-06-04T05:13:08.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.772+0000 7f6385645b80 1 client.28558 shutdown 2024-06-04T05:13:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.772+0000 7f6385645b80 1 -- 172.21.15.70:0/3746387896 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d4f60 msgr2=0x561aa6252550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.772+0000 7f6385645b80 1 --2- 172.21.15.70:0/3746387896 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d4f60 0x561aa6252550 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f636c0aa0c0 tx=0x7f636c004be0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.772+0000 7f6385645b80 1 -- 172.21.15.70:0/3746387896 shutdown_connections 2024-06-04T05:13:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.772+0000 7f6385645b80 1 --2- 172.21.15.70:0/3746387896 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561aa61d4f60 0x561aa6252550 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.772+0000 7f6385645b80 1 --2- 172.21.15.70:0/3746387896 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561aa6254540 0x561aa6256960 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.772+0000 7f6385645b80 1 --2- 172.21.15.70:0/3746387896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561aa61d0860 0x561aa61e18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.772+0000 7f6385645b80 1 --2- 172.21.15.70:0/3746387896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561aa61d1210 0x561aa61e1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.772+0000 7f6353fff700 0 client.28558 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:08.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.772+0000 7f6385645b80 1 -- 172.21.15.70:0/3746387896 >> 172.21.15.70:0/3746387896 conn(0x561aa60fe080 msgr2=0x561aa61d38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:08.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.772+0000 7f6385645b80 1 -- 172.21.15.70:0/3746387896 shutdown_connections 2024-06-04T05:13:08.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.772+0000 7f6385645b80 1 -- 172.21.15.70:0/3746387896 wait complete. 2024-06-04T05:13:08.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.772+0000 7f6385645b80 20 client.28558 trim_cache size 0 max 16384 2024-06-04T05:13:08.779 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:13:08.779 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:13:08.779 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:13:08.779 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:13:08.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.773+0000 7f637adb0700 1 -- 172.21.15.70:0/1986503940 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f63640b2a70 msgr2=0x7f63640b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:08.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.773+0000 7f637adb0700 1 --2- 172.21.15.70:0/1986503940 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f63640b2a70 0x7f63640b4f30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f637419ab80 tx=0x7f6368009260 comp rx=0 tx=0).stop 2024-06-04T05:13:08.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.773+0000 7f637adb0700 1 -- 172.21.15.70:0/1986503940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6374108510 msgr2=0x7f6374199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:08.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.773+0000 7f637adb0700 1 --2- 172.21.15.70:0/1986503940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6374108510 0x7f6374199a40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f636c00ff40 tx=0x7f636c00ff70 comp rx=0 tx=0).stop 2024-06-04T05:13:08.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.773+0000 7f637adb0700 1 -- 172.21.15.70:0/1986503940 shutdown_connections 2024-06-04T05:13:08.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.773+0000 7f637adb0700 1 --2- 172.21.15.70:0/1986503940 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f63640b2a70 0x7f63640b4f30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.773+0000 7f637adb0700 1 --2- 172.21.15.70:0/1986503940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6374108510 0x7f6374199a40 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.773+0000 7f637adb0700 1 --2- 172.21.15.70:0/1986503940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6374102430 0x7f637419f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.773+0000 7f637adb0700 1 --2- 172.21.15.70:0/1986503940 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6374102df0 0x7f637419fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:08.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.773+0000 7f637adb0700 1 -- 172.21.15.70:0/1986503940 >> 172.21.15.70:0/1986503940 conn(0x7f63740793b0 msgr2=0x7f637406bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:08.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.773+0000 7f637adb0700 1 -- 172.21.15.70:0/1986503940 shutdown_connections 2024-06-04T05:13:08.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:08.773+0000 7f637adb0700 1 -- 172.21.15.70:0/1986503940 wait complete. 2024-06-04T05:13:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:08 smithi112 ceph-mon[36453]: osdmap e100: 12 total, 12 up, 12 in 2024-06-04T05:13:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:08 smithi112 ceph-mon[36453]: pgmap v500: 97 pgs: 97 active+clean; 430 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-06-04T05:13:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:08 smithi098 ceph-mon[35569]: osdmap e100: 12 total, 12 up, 12 in 2024-06-04T05:13:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:08 smithi098 ceph-mon[35569]: pgmap v500: 97 pgs: 97 active+clean; 430 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-06-04T05:13:09.261 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:08 smithi070 ceph-mon[32631]: osdmap e100: 12 total, 12 up, 12 in 2024-06-04T05:13:09.261 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:08 smithi070 ceph-mon[32631]: pgmap v500: 97 pgs: 97 active+clean; 430 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-06-04T05:13:09.742 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:13:09.743 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:13:09.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 16667 11.16 MB/sec execute 1 sec latency 93.845 ms 2024-06-04T05:13:10.102 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:13:10.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 16826 9.72 MB/sec execute 2 sec latency 59.658 ms 2024-06-04T05:13:11.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:11 smithi098 ceph-mon[35569]: pgmap v501: 97 pgs: 97 active+clean; 430 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.3 MiB/s wr, 76 op/s 2024-06-04T05:13:11.597 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:11 smithi070 ceph-mon[32631]: pgmap v501: 97 pgs: 97 active+clean; 430 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.3 MiB/s wr, 76 op/s 2024-06-04T05:13:11.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:11 smithi112 ceph-mon[36453]: pgmap v501: 97 pgs: 97 active+clean; 430 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.3 MiB/s wr, 76 op/s 2024-06-04T05:13:11.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 17004 10.79 MB/sec execute 3 sec latency 57.075 ms 2024-06-04T05:13:12.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.034+0000 7f5a51fc7700 1 -- 172.21.15.70:0/2701256646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a4c1024f0 msgr2=0x7f5a4c1103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:12.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.034+0000 7f5a51fc7700 1 --2- 172.21.15.70:0/2701256646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a4c1024f0 0x7f5a4c1103d0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400b0d0 tx=0x7f5a4400b3e0 comp rx=0 tx=0).stop 2024-06-04T05:13:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.034+0000 7f5a51fc7700 1 -- 172.21.15.70:0/2701256646 shutdown_connections 2024-06-04T05:13:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.034+0000 7f5a51fc7700 1 --2- 172.21.15.70:0/2701256646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a4c1024f0 0x7f5a4c1103d0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.034+0000 7f5a51fc7700 1 --2- 172.21.15.70:0/2701256646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a4c109520 0x7f5a4c109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.034+0000 7f5a51fc7700 1 --2- 172.21.15.70:0/2701256646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a4c101b30 0x7f5a4c101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.034+0000 7f5a51fc7700 1 -- 172.21.15.70:0/2701256646 >> 172.21.15.70:0/2701256646 conn(0x7f5a4c078ab0 msgr2=0x7f5a4c0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.034+0000 7f5a51fc7700 1 -- 172.21.15.70:0/2701256646 shutdown_connections 2024-06-04T05:13:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.034+0000 7f5a51fc7700 1 -- 172.21.15.70:0/2701256646 wait complete. 2024-06-04T05:13:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.035+0000 7f5a51fc7700 1 Processor -- start 2024-06-04T05:13:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.035+0000 7f5a51fc7700 1 -- start start 2024-06-04T05:13:12.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.035+0000 7f5a51fc7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a4c101b30 0x7f5a4c1a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:12.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.035+0000 7f5a51fc7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a4c1024f0 0x7f5a4c1a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:12.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.035+0000 7f5a51fc7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a4c109520 0x7f5a4c10f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:12.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.035+0000 7f5a51fc7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4c113610 con 0x7f5a4c1024f0 2024-06-04T05:13:12.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.035+0000 7f5a51fc7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4c113490 con 0x7f5a4c109520 2024-06-04T05:13:12.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.035+0000 7f5a51fc7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4c113790 con 0x7f5a4c101b30 2024-06-04T05:13:12.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.035+0000 7f5a4b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a4c101b30 0x7f5a4c1a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:12.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.035+0000 7f5a4bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a4c109520 0x7f5a4c10f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:12.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.035+0000 7f5a4b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a4c101b30 0x7f5a4c1a4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41956/0 (socket says 172.21.15.70:41956) 2024-06-04T05:13:12.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.035+0000 7f5a4b7fe700 1 -- 172.21.15.70:0/1080204850 learned_addr learned my addr 172.21.15.70:0/1080204850 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:12.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.035+0000 7f5a4affd700 1 --2- 172.21.15.70:0/1080204850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a4c1024f0 0x7f5a4c1a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:12.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.036+0000 7f5a4b7fe700 1 -- 172.21.15.70:0/1080204850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a4c109520 msgr2=0x7f5a4c10f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:12.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.036+0000 7f5a4b7fe700 1 --2- 172.21.15.70:0/1080204850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a4c109520 0x7f5a4c10f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:12.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.036+0000 7f5a4b7fe700 1 -- 172.21.15.70:0/1080204850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a4c1024f0 msgr2=0x7f5a4c1a5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:12.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.036+0000 7f5a4b7fe700 1 --2- 172.21.15.70:0/1080204850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a4c1024f0 0x7f5a4c1a5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:12.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.036+0000 7f5a4b7fe700 1 -- 172.21.15.70:0/1080204850 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a4c10fd10 con 0x7f5a4c101b30 2024-06-04T05:13:12.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.036+0000 7f5a4b7fe700 1 --2- 172.21.15.70:0/1080204850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a4c101b30 0x7f5a4c1a4cf0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00e9c0 tx=0x7f5a3c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:12.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.036+0000 7f5a48ff9700 1 -- 172.21.15.70:0/1080204850 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c00cbd0 con 0x7f5a4c101b30 2024-06-04T05:13:12.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.036+0000 7f5a48ff9700 1 -- 172.21.15.70:0/1080204850 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a3c00cd70 con 0x7f5a4c101b30 2024-06-04T05:13:12.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.036+0000 7f5a48ff9700 1 -- 172.21.15.70:0/1080204850 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c005540 con 0x7f5a4c101b30 2024-06-04T05:13:12.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.036+0000 7f5a51fc7700 1 -- 172.21.15.70:0/1080204850 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a4c110000 con 0x7f5a4c101b30 2024-06-04T05:13:12.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.036+0000 7f5a51fc7700 1 -- 172.21.15.70:0/1080204850 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a4c1acc20 con 0x7f5a4c101b30 2024-06-04T05:13:12.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.037+0000 7f5a51fc7700 1 -- 172.21.15.70:0/1080204850 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a4c104e50 con 0x7f5a4c101b30 2024-06-04T05:13:12.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.039+0000 7f5a48ff9700 1 -- 172.21.15.70:0/1080204850 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f5a3c0479e0 con 0x7f5a4c101b30 2024-06-04T05:13:12.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.040+0000 7f5a48ff9700 1 --2- 172.21.15.70:0/1080204850 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5a340b29f0 0x7f5a340b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:12.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.041+0000 7f5a4affd700 1 --2- 172.21.15.70:0/1080204850 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5a340b29f0 0x7f5a340b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:12.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.041+0000 7f5a48ff9700 1 -- 172.21.15.70:0/1080204850 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f5a3c0e28b0 con 0x7f5a4c101b30 2024-06-04T05:13:12.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.041+0000 7f5a4affd700 1 --2- 172.21.15.70:0/1080204850 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5a340b29f0 0x7f5a340b4eb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5a40005fd0 tx=0x7f5a40005f60 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:12.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.045+0000 7f5a48ff9700 1 -- 172.21.15.70:0/1080204850 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5a3c0aa200 con 0x7f5a4c101b30 2024-06-04T05:13:12.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.483+0000 7f5a51fc7700 1 -- 172.21.15.70:0/1080204850 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a4c06a230 con 0x7f5a4c101b30 2024-06-04T05:13:12.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.484+0000 7f5a48ff9700 1 -- 172.21.15.70:0/1080204850 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11947 (secure 0 0 0) 0x7f5a4c06a230 con 0x7f5a4c101b30 2024-06-04T05:13:12.486 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:13:12.487 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":45,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:12:42.042199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T05:13:12.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.487+0000 7f5a51fc7700 1 -- 172.21.15.70:0/1080204850 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5a340b29f0 msgr2=0x7f5a340b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:12.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.487+0000 7f5a51fc7700 1 --2- 172.21.15.70:0/1080204850 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5a340b29f0 0x7f5a340b4eb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5a40005fd0 tx=0x7f5a40005f60 comp rx=0 tx=0).stop 2024-06-04T05:13:12.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.487+0000 7f5a51fc7700 1 -- 172.21.15.70:0/1080204850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a4c101b30 msgr2=0x7f5a4c1a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:12.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.487+0000 7f5a51fc7700 1 --2- 172.21.15.70:0/1080204850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a4c101b30 0x7f5a4c1a4cf0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00e9c0 tx=0x7f5a3c00ecd0 comp rx=0 tx=0).stop 2024-06-04T05:13:12.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.487+0000 7f5a51fc7700 1 -- 172.21.15.70:0/1080204850 shutdown_connections 2024-06-04T05:13:12.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.487+0000 7f5a51fc7700 1 --2- 172.21.15.70:0/1080204850 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5a340b29f0 0x7f5a340b4eb0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:12.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.487+0000 7f5a51fc7700 1 --2- 172.21.15.70:0/1080204850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a4c109520 0x7f5a4c10f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:12.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.487+0000 7f5a51fc7700 1 --2- 172.21.15.70:0/1080204850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a4c101b30 0x7f5a4c1a4cf0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:12.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.487+0000 7f5a51fc7700 1 --2- 172.21.15.70:0/1080204850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a4c1024f0 0x7f5a4c1a5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:12.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.487+0000 7f5a51fc7700 1 -- 172.21.15.70:0/1080204850 >> 172.21.15.70:0/1080204850 conn(0x7f5a4c078ab0 msgr2=0x7f5a4c0ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.487+0000 7f5a51fc7700 1 -- 172.21.15.70:0/1080204850 shutdown_connections 2024-06-04T05:13:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:12.487+0000 7f5a51fc7700 1 -- 172.21.15.70:0/1080204850 wait complete. 2024-06-04T05:13:12.494 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 45 2024-06-04T05:13:12.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 17169 10.49 MB/sec execute 4 sec latency 62.410 ms 2024-06-04T05:13:13.223 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:13:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:13 smithi098 ceph-mon[35569]: pgmap v502: 97 pgs: 97 active+clean; 451 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.2 MiB/s wr, 242 op/s 2024-06-04T05:13:13.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:13 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1080204850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:13:13.577 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:13:13.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:13 smithi070 ceph-mon[32631]: pgmap v502: 97 pgs: 97 active+clean; 451 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.2 MiB/s wr, 242 op/s 2024-06-04T05:13:13.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:13 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1080204850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:13:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:13 smithi112 ceph-mon[36453]: pgmap v502: 97 pgs: 97 active+clean; 451 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.2 MiB/s wr, 242 op/s 2024-06-04T05:13:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:13 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1080204850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:13:13.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 17338 10.76 MB/sec execute 5 sec latency 70.466 ms 2024-06-04T05:13:14.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 17506 9.56 MB/sec execute 6 sec latency 71.325 ms 2024-06-04T05:13:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.023+0000 7fbf13859640 1 -- 172.21.15.70:0/331982909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf040af320 msgr2=0x7fbf040af7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.023+0000 7fbf13859640 1 --2- 172.21.15.70:0/331982909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf040af320 0x7fbf040af7d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c068a30 tx=0x7fbf0c08ec60 comp rx=0 tx=0).stop 2024-06-04T05:13:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.023+0000 7fbf13859640 1 -- 172.21.15.70:0/331982909 shutdown_connections 2024-06-04T05:13:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.023+0000 7fbf13859640 1 --2- 172.21.15.70:0/331982909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf040af320 0x7fbf040af7d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.023+0000 7fbf13859640 1 --2- 172.21.15.70:0/331982909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf040a5990 0x7fbf040aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.023+0000 7fbf13859640 1 --2- 172.21.15.70:0/331982909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf040a5070 0x7fbf040a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.023+0000 7fbf13859640 1 -- 172.21.15.70:0/331982909 >> 172.21.15.70:0/331982909 conn(0x7fbf0401aaa0 msgr2=0x7fbf0401aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.023+0000 7fbf13859640 1 -- 172.21.15.70:0/331982909 shutdown_connections 2024-06-04T05:13:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.023+0000 7fbf13859640 1 -- 172.21.15.70:0/331982909 wait complete. 2024-06-04T05:13:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.024+0000 7fbf13859640 1 Processor -- start 2024-06-04T05:13:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.024+0000 7fbf13859640 1 -- start start 2024-06-04T05:13:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.024+0000 7fbf13859640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf040a5070 0x7fbf041496d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.024+0000 7fbf13859640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf040a5990 0x7fbf04149c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.024+0000 7fbf13859640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf040af320 0x7fbf04143840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.024+0000 7fbf13859640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf040bd3a0 con 0x7fbf040a5990 2024-06-04T05:13:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.024+0000 7fbf13859640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf040bd220 con 0x7fbf040a5070 2024-06-04T05:13:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.024+0000 7fbf13859640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbf040bd520 con 0x7fbf040af320 2024-06-04T05:13:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.024+0000 7fbf10df4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf040a5990 0x7fbf04149c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.024+0000 7fbf115f5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf040a5070 0x7fbf041496d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.024+0000 7fbf11df6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf040af320 0x7fbf04143840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.024+0000 7fbf10df4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf040a5990 0x7fbf04149c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43128/0 (socket says 172.21.15.70:43128) 2024-06-04T05:13:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.024+0000 7fbf10df4640 1 -- 172.21.15.70:0/2400331587 learned_addr learned my addr 172.21.15.70:0/2400331587 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.025+0000 7fbf115f5640 1 -- 172.21.15.70:0/2400331587 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf040af320 msgr2=0x7fbf04143840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.025+0000 7fbf115f5640 1 --2- 172.21.15.70:0/2400331587 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf040af320 0x7fbf04143840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.025+0000 7fbf115f5640 1 -- 172.21.15.70:0/2400331587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf040a5990 msgr2=0x7fbf04149c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.025+0000 7fbf115f5640 1 --2- 172.21.15.70:0/2400331587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf040a5990 0x7fbf04149c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.025+0000 7fbf115f5640 1 -- 172.21.15.70:0/2400331587 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf04144010 con 0x7fbf040a5070 2024-06-04T05:13:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.025+0000 7fbf10df4640 1 --2- 172.21.15.70:0/2400331587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf040a5990 0x7fbf04149c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.025+0000 7fbf115f5640 1 --2- 172.21.15.70:0/2400331587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf040a5070 0x7fbf041496d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbf0800e790 tx=0x7fbf0800ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.025+0000 7fbf027fc640 1 -- 172.21.15.70:0/2400331587 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0800eea0 con 0x7fbf040a5070 2024-06-04T05:13:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.025+0000 7fbf13859640 1 -- 172.21.15.70:0/2400331587 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf04144300 con 0x7fbf040a5070 2024-06-04T05:13:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.025+0000 7fbf027fc640 1 -- 172.21.15.70:0/2400331587 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf0800cc40 con 0x7fbf040a5070 2024-06-04T05:13:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.025+0000 7fbf027fc640 1 -- 172.21.15.70:0/2400331587 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf08018680 con 0x7fbf040a5070 2024-06-04T05:13:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.026+0000 7fbf13859640 1 -- 172.21.15.70:0/2400331587 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf04150520 con 0x7fbf040a5070 2024-06-04T05:13:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.030+0000 7fbf027fc640 1 -- 172.21.15.70:0/2400331587 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fbf08010430 con 0x7fbf040a5070 2024-06-04T05:13:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.031+0000 7fbf16573740 1 Processor -- start 2024-06-04T05:13:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.031+0000 7fbf16573740 1 -- start start 2024-06-04T05:13:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.031+0000 7fbf16573740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaea00 0x55869deaede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.031+0000 7fbf16573740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaf320 0x55869deb96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.031+0000 7fbf16573740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deb9be0 0x55869debbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.031+0000 7fbf16573740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55869de20c30 con 0x55869deb9be0 2024-06-04T05:13:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.031+0000 7fbf16573740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55869de107d0 con 0x55869deaf320 2024-06-04T05:13:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.031+0000 7fbf16573740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55869de14810 con 0x55869deaea00 2024-06-04T05:13:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.031+0000 7fbf027fc640 1 --2- 172.21.15.70:0/2400331587 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbef80b2990 0x7fbef80b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.031+0000 7fbf115f5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaf320 0x55869deb96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.031+0000 7fbf10df4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deb9be0 0x55869debbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.031+0000 7fbf115f5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaf320 0x55869deb96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:42928/0 (socket says 172.21.15.70:42928) 2024-06-04T05:13:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.031+0000 7fbf115f5640 1 -- 172.21.15.70:0/463260816 learned_addr learned my addr 172.21.15.70:0/463260816 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.031+0000 7fbf11df6640 1 --2- 172.21.15.70:0/2400331587 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbef80b2990 0x7fbef80b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.032+0000 7fbf11df6640 1 --2- 172.21.15.70:0/463260816 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaea00 0x55869deaede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.032+0000 7fbf027fc640 1 -- 172.21.15.70:0/2400331587 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fbf080e27d0 con 0x7fbf040a5070 2024-06-04T05:13:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.032+0000 7fbf115f5640 1 -- 172.21.15.70:0/463260816 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaea00 msgr2=0x55869deaede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.032+0000 7fbf115f5640 1 --2- 172.21.15.70:0/463260816 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaea00 0x55869deaede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.032+0000 7fbf115f5640 1 -- 172.21.15.70:0/463260816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deb9be0 msgr2=0x55869debbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.032+0000 7fbf115f5640 1 --2- 172.21.15.70:0/463260816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deb9be0 0x55869debbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.032+0000 7fbf115f5640 1 -- 172.21.15.70:0/463260816 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55869db25210 con 0x55869deaf320 2024-06-04T05:13:15.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.032+0000 7fbf10df4640 1 --2- 172.21.15.70:0/463260816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deb9be0 0x55869debbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:15.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.032+0000 7fbf11df6640 1 --2- 172.21.15.70:0/2400331587 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbef80b2990 0x7fbef80b4e50 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0520e0 tx=0x7fbf0c068ba0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:15.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.032+0000 7fbf115f5640 1 --2- 172.21.15.70:0/463260816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaf320 0x55869deb96a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbf080a5bb0 tx=0x7fbf08077700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e25b626231dd236 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:15.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.032+0000 7fbef77fe640 1 -- 172.21.15.70:0/463260816 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf08026040 con 0x55869deaf320 2024-06-04T05:13:15.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.032+0000 7fbef77fe640 1 -- 172.21.15.70:0/463260816 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf080ade30 con 0x55869deaf320 2024-06-04T05:13:15.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.032+0000 7fbef77fe640 1 -- 172.21.15.70:0/463260816 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf080a5dc0 con 0x55869deaf320 2024-06-04T05:13:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.033+0000 7fbf16573740 1 -- 172.21.15.70:0/463260816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaf320 msgr2=0x55869deb96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.033+0000 7fbf16573740 1 --2- 172.21.15.70:0/463260816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaf320 0x55869deb96a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbf080a5bb0 tx=0x7fbf08077700 comp rx=0 tx=0).stop 2024-06-04T05:13:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.033+0000 7fbf16573740 1 -- 172.21.15.70:0/463260816 shutdown_connections 2024-06-04T05:13:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.033+0000 7fbf16573740 1 --2- 172.21.15.70:0/463260816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaf320 0x55869deb96a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.033+0000 7fbf16573740 1 --2- 172.21.15.70:0/463260816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deb9be0 0x55869debbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.033+0000 7fbf16573740 1 --2- 172.21.15.70:0/463260816 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaea00 0x55869deaede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.033+0000 7fbf16573740 1 -- 172.21.15.70:0/463260816 >> 172.21.15.70:0/463260816 conn(0x55869deaaf10 msgr2=0x55869deab2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.034+0000 7fbf16573740 1 -- 172.21.15.70:0/463260816 shutdown_connections 2024-06-04T05:13:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.034+0000 7fbf16573740 1 -- 172.21.15.70:0/463260816 wait complete. 2024-06-04T05:13:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.034+0000 7fbf16573740 1 -- start start 2024-06-04T05:13:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.034+0000 7fbf16573740 1 Processor -- start 2024-06-04T05:13:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.035+0000 7fbf16573740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaea00 0x55869deb6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.035+0000 7fbf16573740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deaf320 0x55869deb7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.035+0000 7fbf16573740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deb9be0 0x55869deb75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.035+0000 7fbf16573740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55869daf6790 con 0x55869deaf320 2024-06-04T05:13:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.035+0000 7fbf16573740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55869ddcc890 con 0x55869deb9be0 2024-06-04T05:13:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.035+0000 7fbf16573740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55869debc750 con 0x55869deaea00 2024-06-04T05:13:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.035+0000 7fbf16573740 10 client.0 _ll_register_callbacks cb 0x55869db24a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.035+0000 7fbf115f5640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaea00 0x55869deb6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.035+0000 7fbf115f5640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaea00 0x55869deb6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41992/0 (socket says 172.21.15.70:41992) 2024-06-04T05:13:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.035+0000 7fbf115f5640 1 -- 172.21.15.70:0/1965255297 learned_addr learned my addr 172.21.15.70:0/1965255297 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.035+0000 7fbf10df4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deaf320 0x55869deb7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.035+0000 7fbf11df6640 1 --2- 172.21.15.70:0/1965255297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deb9be0 0x55869deb75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.035+0000 7fbf115f5640 1 -- 172.21.15.70:0/1965255297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deb9be0 msgr2=0x55869deb75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.035+0000 7fbf115f5640 1 --2- 172.21.15.70:0/1965255297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deb9be0 0x55869deb75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.035+0000 7fbf115f5640 1 -- 172.21.15.70:0/1965255297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deaf320 msgr2=0x55869deb7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.035+0000 7fbf115f5640 1 --2- 172.21.15.70:0/1965255297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deaf320 0x55869deb7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.035+0000 7fbf115f5640 1 -- 172.21.15.70:0/1965255297 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55869dbf1620 con 0x55869deaea00 2024-06-04T05:13:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.036+0000 7fbf11df6640 1 --2- 172.21.15.70:0/1965255297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deb9be0 0x55869deb75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.036+0000 7fbf115f5640 1 --2- 172.21.15.70:0/1965255297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaea00 0x55869deb6b30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbf080a95f0 tx=0x7fbf080a9900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4d3171e49dc10fd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.036+0000 7fbef4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:13:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.036+0000 7fbef4ff9640 1 -- 172.21.15.70:0/1965255297 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf080b2450 con 0x55869deaea00 2024-06-04T05:13:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.036+0000 7fbef4ff9640 1 -- 172.21.15.70:0/1965255297 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fbf080b25f0 con 0x55869deaea00 2024-06-04T05:13:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.036+0000 7fbf16573740 1 -- 172.21.15.70:0/1965255297 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf080ade30 con 0x55869deaea00 2024-06-04T05:13:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.036+0000 7fbef4ff9640 1 -- 172.21.15.70:0/1965255297 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fbf08010720 con 0x55869deaea00 2024-06-04T05:13:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.036+0000 7fbf16573740 10 client.28576 fetch_fsmap learned FSMap version 45 2024-06-04T05:13:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.036+0000 7fbf16573740 1 -- 172.21.15.70:0/1965255297 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55869dec84c0 con 0x55869deaea00 2024-06-04T05:13:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.037+0000 7fbef4ff9640 1 -- 172.21.15.70:0/1965255297 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf080108c0 con 0x55869deaea00 2024-06-04T05:13:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.037+0000 7fbef4ff9640 1 -- 172.21.15.70:0/1965255297 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 45) ==== 8398+0+0 (secure 0 0 0) 0x7fbf080a5660 con 0x55869deaea00 2024-06-04T05:13:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.037+0000 7fbf16573740 10 client.28576 fetch_fsmap finished waiting for FSMap version 45 2024-06-04T05:13:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.037+0000 7fbf16573740 10 client.28576 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.037+0000 7fbf16573740 20 client.28576 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.037+0000 7fbf16573740 1 --2- 172.21.15.70:0/1965255297 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55869dec9810 0x55869decbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.037+0000 7fbf16573740 4 client.28576 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:13:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.037+0000 7fbf16573740 1 -- 172.21.15.70:0/1965255297 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbf080ade30 con 0x55869dec9810 2024-06-04T05:13:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.037+0000 7fbf10df4640 1 --2- 172.21.15.70:0/1965255297 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55869dec9810 0x55869decbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.038+0000 7fbef4ff9640 10 client.28576 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.038+0000 7fbf10df4640 1 --2- 172.21.15.70:0/1965255297 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55869dec9810 0x55869decbc00 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c41e01d80ba91b9 server_cookie=7f418839d1a8d98b in_seq=0 out_seq=0 2024-06-04T05:13:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.039+0000 7fbef4ff9640 1 -- 172.21.15.70:0/1965255297 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fbf080ade30 con 0x55869dec9810 2024-06-04T05:13:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.039+0000 7fbef4ff9640 10 client.28576 handle_command_reply: tid=0 2024-06-04T05:13:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.039+0000 7fbf16573740 1 client.28576 shutdown 2024-06-04T05:13:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.039+0000 7fbf16573740 1 -- 172.21.15.70:0/1965255297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaea00 msgr2=0x55869deb6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.039+0000 7fbf16573740 1 --2- 172.21.15.70:0/1965255297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaea00 0x55869deb6b30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbf080a95f0 tx=0x7fbf080a9900 comp rx=0 tx=0).stop 2024-06-04T05:13:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.039+0000 7fbf16573740 1 -- 172.21.15.70:0/1965255297 shutdown_connections 2024-06-04T05:13:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.039+0000 7fbf16573740 1 --2- 172.21.15.70:0/1965255297 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55869dec9810 0x55869decbc00 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.039+0000 7fbf16573740 1 --2- 172.21.15.70:0/1965255297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deb9be0 0x55869deb75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.039+0000 7fbf16573740 1 --2- 172.21.15.70:0/1965255297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deaf320 0x55869deb7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.039+0000 7fbf16573740 1 --2- 172.21.15.70:0/1965255297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaea00 0x55869deb6b30 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.039+0000 7fbf16573740 1 -- 172.21.15.70:0/1965255297 >> 172.21.15.70:0/1965255297 conn(0x55869deaaf10 msgr2=0x55869deb0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.039+0000 7fbef4ff9640 0 client.28576 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.039+0000 7fbf16573740 1 -- 172.21.15.70:0/1965255297 shutdown_connections 2024-06-04T05:13:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.039+0000 7fbf16573740 1 -- 172.21.15.70:0/1965255297 wait complete. 2024-06-04T05:13:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.040+0000 7fbf16573740 20 client.28576 trim_cache size 0 max 16384 2024-06-04T05:13:15.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf16573740 1 Processor -- start 2024-06-04T05:13:15.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf16573740 1 -- start start 2024-06-04T05:13:15.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf16573740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaea00 0x55869deb13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf16573740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deaf320 0x55869deb73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf16573740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deb9be0 0x55869ded1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf16573740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf04158fa0 con 0x55869deaf320 2024-06-04T05:13:15.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf16573740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0415ac30 con 0x55869deaea00 2024-06-04T05:13:15.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf16573740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbf04159670 con 0x55869deb9be0 2024-06-04T05:13:15.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf10df4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deaf320 0x55869deb73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf11df6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deb9be0 0x55869ded1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf11df6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deb9be0 0x55869ded1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41996/0 (socket says 172.21.15.70:41996) 2024-06-04T05:13:15.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf11df6640 1 -- 172.21.15.70:0/2854279571 learned_addr learned my addr 172.21.15.70:0/2854279571 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:15.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf10df4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deaf320 0x55869deb73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43152/0 (socket says 172.21.15.70:43152) 2024-06-04T05:13:15.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf115f5640 1 --2- 172.21.15.70:0/2854279571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaea00 0x55869deb13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf115f5640 1 -- 172.21.15.70:0/2854279571 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deb9be0 msgr2=0x55869ded1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:15.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf115f5640 1 --2- 172.21.15.70:0/2854279571 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deb9be0 0x55869ded1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf115f5640 1 -- 172.21.15.70:0/2854279571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deaf320 msgr2=0x55869deb73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:15.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf115f5640 1 --2- 172.21.15.70:0/2854279571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deaf320 0x55869deb73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf115f5640 1 -- 172.21.15.70:0/2854279571 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55869deb5080 con 0x55869deaea00 2024-06-04T05:13:15.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.062+0000 7fbf11df6640 1 --2- 172.21.15.70:0/2854279571 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deb9be0 0x55869ded1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:15.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.063+0000 7fbf115f5640 1 --2- 172.21.15.70:0/2854279571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaea00 0x55869deb13c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbf080a6da0 tx=0x7fbf080a5390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff044ff36c2c8e9b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:15.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.063+0000 7fbef6ffd640 1 -- 172.21.15.70:0/2854279571 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf080b28e0 con 0x55869deaea00 2024-06-04T05:13:15.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.063+0000 7fbef6ffd640 1 -- 172.21.15.70:0/2854279571 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf080108c0 con 0x55869deaea00 2024-06-04T05:13:15.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.063+0000 7fbef6ffd640 1 -- 172.21.15.70:0/2854279571 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf080a54a0 con 0x55869deaea00 2024-06-04T05:13:15.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.064+0000 7fbf16573740 1 -- 172.21.15.70:0/2854279571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaea00 msgr2=0x55869deb13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.064+0000 7fbf16573740 1 --2- 172.21.15.70:0/2854279571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaea00 0x55869deb13c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbf080a6da0 tx=0x7fbf080a5390 comp rx=0 tx=0).stop 2024-06-04T05:13:15.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.064+0000 7fbf16573740 1 -- 172.21.15.70:0/2854279571 shutdown_connections 2024-06-04T05:13:15.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.064+0000 7fbf16573740 1 --2- 172.21.15.70:0/2854279571 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deb9be0 0x55869ded1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.064+0000 7fbf16573740 1 --2- 172.21.15.70:0/2854279571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deaf320 0x55869deb73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.064+0000 7fbf16573740 1 --2- 172.21.15.70:0/2854279571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaea00 0x55869deb13c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.064+0000 7fbf16573740 1 -- 172.21.15.70:0/2854279571 >> 172.21.15.70:0/2854279571 conn(0x55869deaaf10 msgr2=0x55869deb5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:15.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.064+0000 7fbf16573740 1 -- 172.21.15.70:0/2854279571 shutdown_connections 2024-06-04T05:13:15.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.064+0000 7fbf16573740 1 -- 172.21.15.70:0/2854279571 wait complete. 2024-06-04T05:13:15.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.064+0000 7fbf16573740 1 -- start start 2024-06-04T05:13:15.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.065+0000 7fbf16573740 1 Processor -- start 2024-06-04T05:13:15.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.065+0000 7fbf16573740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaea00 0x55869debc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.065+0000 7fbf16573740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaf320 0x55869debced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.065+0000 7fbf16573740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deb9be0 0x55869debd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.065+0000 7fbf16573740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0415a4a0 con 0x55869deb9be0 2024-06-04T05:13:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.065+0000 7fbf16573740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf04159da0 con 0x55869deaea00 2024-06-04T05:13:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.065+0000 7fbf16573740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbf04156fa0 con 0x55869deaf320 2024-06-04T05:13:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.065+0000 7fbf115f5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaea00 0x55869debc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.065+0000 7fbf10df4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaf320 0x55869debced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.065+0000 7fbf16573740 10 client.0 _ll_register_callbacks cb 0x7fbefc009550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.065+0000 7fbf10df4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaf320 0x55869debced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:42000/0 (socket says 172.21.15.70:42000) 2024-06-04T05:13:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.065+0000 7fbf10df4640 1 -- 172.21.15.70:0/1220839460 learned_addr learned my addr 172.21.15.70:0/1220839460 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.065+0000 7fbf11df6640 1 --2- 172.21.15.70:0/1220839460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deb9be0 0x55869debd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.066+0000 7fbf10df4640 1 -- 172.21.15.70:0/1220839460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaea00 msgr2=0x55869debc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.066+0000 7fbf10df4640 1 --2- 172.21.15.70:0/1220839460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaea00 0x55869debc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.066+0000 7fbf10df4640 1 -- 172.21.15.70:0/1220839460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deb9be0 msgr2=0x55869debd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.066+0000 7fbf10df4640 1 --2- 172.21.15.70:0/1220839460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deb9be0 0x55869debd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.066+0000 7fbf10df4640 1 -- 172.21.15.70:0/1220839460 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55869df32430 con 0x55869deaf320 2024-06-04T05:13:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.066+0000 7fbf115f5640 1 --2- 172.21.15.70:0/1220839460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaea00 0x55869debc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.066+0000 7fbf11df6640 1 --2- 172.21.15.70:0/1220839460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deb9be0 0x55869debd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.066+0000 7fbf10df4640 1 --2- 172.21.15.70:0/1220839460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaf320 0x55869debced0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbefc009520 tx=0x7fbefc012c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4767197465d54904 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.066+0000 7fbef4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:13:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.066+0000 7fbef4ff9640 1 -- 172.21.15.70:0/1220839460 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc010a80 con 0x55869deaf320 2024-06-04T05:13:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.066+0000 7fbef4ff9640 1 -- 172.21.15.70:0/1220839460 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fbefc010c20 con 0x55869deaf320 2024-06-04T05:13:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.066+0000 7fbf16573740 1 -- 172.21.15.70:0/1220839460 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf080108c0 con 0x55869deaf320 2024-06-04T05:13:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.066+0000 7fbef4ff9640 1 -- 172.21.15.70:0/1220839460 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fbefc010e20 con 0x55869deaf320 2024-06-04T05:13:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.066+0000 7fbf16573740 10 client.28582 fetch_fsmap learned FSMap version 45 2024-06-04T05:13:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.066+0000 7fbf16573740 1 -- 172.21.15.70:0/1220839460 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55869df337a0 con 0x55869deaf320 2024-06-04T05:13:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.067+0000 7fbef4ff9640 1 -- 172.21.15.70:0/1220839460 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf080108c0 con 0x55869deaf320 2024-06-04T05:13:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.067+0000 7fbef4ff9640 1 -- 172.21.15.70:0/1220839460 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 45) ==== 8398+0+0 (secure 0 0 0) 0x7fbefc027420 con 0x55869deaf320 2024-06-04T05:13:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.067+0000 7fbf16573740 10 client.28582 fetch_fsmap finished waiting for FSMap version 45 2024-06-04T05:13:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.067+0000 7fbf16573740 10 client.28582 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.067+0000 7fbf16573740 20 client.28582 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.067+0000 7fbf16573740 1 --2- 172.21.15.70:0/1220839460 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55869df34010 0x55869df36430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.067+0000 7fbf16573740 4 client.28582 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:13:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.067+0000 7fbf16573740 1 -- 172.21.15.70:0/1220839460 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55869de22b80 con 0x55869df34010 2024-06-04T05:13:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.067+0000 7fbf115f5640 1 --2- 172.21.15.70:0/1220839460 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55869df34010 0x55869df36430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.067+0000 7fbef4ff9640 10 client.28582 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.068+0000 7fbf115f5640 1 --2- 172.21.15.70:0/1220839460 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55869df34010 0x55869df36430 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b251e18972344ca server_cookie=997c62fa849d34e4 in_seq=0 out_seq=0 2024-06-04T05:13:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.158+0000 7fbef4ff9640 1 -- 172.21.15.70:0/1220839460 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55869de22b80 con 0x55869df34010 2024-06-04T05:13:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.158+0000 7fbef4ff9640 10 client.28582 handle_command_reply: tid=0 2024-06-04T05:13:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.158+0000 7fbf16573740 1 client.28582 shutdown 2024-06-04T05:13:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.158+0000 7fbf16573740 1 -- 172.21.15.70:0/1220839460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaf320 msgr2=0x55869debced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.158+0000 7fbf16573740 1 --2- 172.21.15.70:0/1220839460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaf320 0x55869debced0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbefc009520 tx=0x7fbefc012c00 comp rx=0 tx=0).stop 2024-06-04T05:13:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.158+0000 7fbf16573740 1 -- 172.21.15.70:0/1220839460 shutdown_connections 2024-06-04T05:13:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.158+0000 7fbf16573740 1 --2- 172.21.15.70:0/1220839460 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55869df34010 0x55869df36430 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.158+0000 7fbf16573740 1 --2- 172.21.15.70:0/1220839460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55869deb9be0 0x55869debd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.158+0000 7fbf16573740 1 --2- 172.21.15.70:0/1220839460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55869deaf320 0x55869debced0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.158+0000 7fbef4ff9640 0 client.28582 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.158+0000 7fbf16573740 1 --2- 172.21.15.70:0/1220839460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55869deaea00 0x55869debc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.158+0000 7fbf16573740 1 -- 172.21.15.70:0/1220839460 >> 172.21.15.70:0/1220839460 conn(0x55869deaaf10 msgr2=0x55869dea9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.158+0000 7fbf16573740 1 -- 172.21.15.70:0/1220839460 shutdown_connections 2024-06-04T05:13:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.158+0000 7fbf16573740 1 -- 172.21.15.70:0/1220839460 wait complete. 2024-06-04T05:13:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.158+0000 7fbf16573740 20 client.28582 trim_cache size 0 max 16384 2024-06-04T05:13:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:13:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:13:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:13:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.159+0000 7fbf13859640 1 -- 172.21.15.70:0/2400331587 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbef80b2990 msgr2=0x7fbef80b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.159+0000 7fbf13859640 1 --2- 172.21.15.70:0/2400331587 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbef80b2990 0x7fbef80b4e50 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0520e0 tx=0x7fbf0c068ba0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.159+0000 7fbf13859640 1 -- 172.21.15.70:0/2400331587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf040a5070 msgr2=0x7fbf041496d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.159+0000 7fbf13859640 1 --2- 172.21.15.70:0/2400331587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf040a5070 0x7fbf041496d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbf0800e790 tx=0x7fbf0800ec50 comp rx=0 tx=0).stop 2024-06-04T05:13:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.159+0000 7fbf13859640 1 -- 172.21.15.70:0/2400331587 shutdown_connections 2024-06-04T05:13:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.159+0000 7fbf13859640 1 --2- 172.21.15.70:0/2400331587 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbef80b2990 0x7fbef80b4e50 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.159+0000 7fbf13859640 1 --2- 172.21.15.70:0/2400331587 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf040af320 0x7fbf04143840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.159+0000 7fbf13859640 1 --2- 172.21.15.70:0/2400331587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf040a5990 0x7fbf04149c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.159+0000 7fbf13859640 1 --2- 172.21.15.70:0/2400331587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf040a5070 0x7fbf041496d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.159+0000 7fbf13859640 1 -- 172.21.15.70:0/2400331587 >> 172.21.15.70:0/2400331587 conn(0x7fbf0401aaa0 msgr2=0x7fbf040acd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.159+0000 7fbf13859640 1 -- 172.21.15.70:0/2400331587 shutdown_connections 2024-06-04T05:13:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:15.159+0000 7fbf13859640 1 -- 172.21.15.70:0/2400331587 wait complete. 2024-06-04T05:13:15.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:15 smithi098 ceph-mon[35569]: pgmap v503: 97 pgs: 97 active+clean; 451 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 3.8 MiB/s wr, 215 op/s 2024-06-04T05:13:15.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.516+0000 7fbd41919700 1 -- 172.21.15.70:0/1650954688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd3c101b30 msgr2=0x7fbd3c101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.516+0000 7fbd41919700 1 --2- 172.21.15.70:0/1650954688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd3c101b30 0x7fbd3c101fb0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fbd30009990 tx=0x7fbd30009ca0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.516+0000 7fbd41919700 1 -- 172.21.15.70:0/1650954688 shutdown_connections 2024-06-04T05:13:15.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.516+0000 7fbd41919700 1 --2- 172.21.15.70:0/1650954688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd3c1025d0 0x7fbd3c1122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.516+0000 7fbd41919700 1 --2- 172.21.15.70:0/1650954688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd3c104450 0x7fbd3c104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.516+0000 7fbd41919700 1 --2- 172.21.15.70:0/1650954688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd3c101b30 0x7fbd3c101fb0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.516+0000 7fbd41919700 1 -- 172.21.15.70:0/1650954688 >> 172.21.15.70:0/1650954688 conn(0x7fbd3c078ab0 msgr2=0x7fbd3c0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:15.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.516+0000 7fbd41919700 1 -- 172.21.15.70:0/1650954688 shutdown_connections 2024-06-04T05:13:15.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.516+0000 7fbd41919700 1 -- 172.21.15.70:0/1650954688 wait complete. 2024-06-04T05:13:15.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.517+0000 7fbd41919700 1 Processor -- start 2024-06-04T05:13:15.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:15 smithi070 ceph-mon[32631]: pgmap v503: 97 pgs: 97 active+clean; 451 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 3.8 MiB/s wr, 215 op/s 2024-06-04T05:13:15.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.517+0000 7fbd41919700 1 -- start start 2024-06-04T05:13:15.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.517+0000 7fbd41919700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd3c101b30 0x7fbd3c19bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.517+0000 7fbd41919700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd3c1025d0 0x7fbd3c19c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.517+0000 7fbd41919700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd3c104450 0x7fbd3c1a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.517+0000 7fbd41919700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3c1157c0 con 0x7fbd3c104450 2024-06-04T05:13:15.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.517+0000 7fbd41919700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3c115640 con 0x7fbd3c1025d0 2024-06-04T05:13:15.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.517+0000 7fbd41919700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3c115940 con 0x7fbd3c101b30 2024-06-04T05:13:15.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.518+0000 7fbd3b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd3c104450 0x7fbd3c1a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.518+0000 7fbd3b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd3c104450 0x7fbd3c1a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43162/0 (socket says 172.21.15.70:43162) 2024-06-04T05:13:15.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.518+0000 7fbd3b7fe700 1 -- 172.21.15.70:0/3101889628 learned_addr learned my addr 172.21.15.70:0/3101889628 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:15.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.518+0000 7fbd3affd700 1 --2- 172.21.15.70:0/3101889628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd3c101b30 0x7fbd3c19bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.518+0000 7fbd3a7fc700 1 --2- 172.21.15.70:0/3101889628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd3c1025d0 0x7fbd3c19c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.518+0000 7fbd3affd700 1 -- 172.21.15.70:0/3101889628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd3c1025d0 msgr2=0x7fbd3c19c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.518+0000 7fbd3affd700 1 --2- 172.21.15.70:0/3101889628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd3c1025d0 0x7fbd3c19c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.518+0000 7fbd3affd700 1 -- 172.21.15.70:0/3101889628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd3c104450 msgr2=0x7fbd3c1a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.518+0000 7fbd3affd700 1 --2- 172.21.15.70:0/3101889628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd3c104450 0x7fbd3c1a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.518+0000 7fbd3affd700 1 -- 172.21.15.70:0/3101889628 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd3c1a0d80 con 0x7fbd3c101b30 2024-06-04T05:13:15.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.518+0000 7fbd3b7fe700 1 --2- 172.21.15.70:0/3101889628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd3c104450 0x7fbd3c1a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:15.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.518+0000 7fbd3affd700 1 --2- 172.21.15.70:0/3101889628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd3c101b30 0x7fbd3c19bd70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c00e9f0 tx=0x7fbd2c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:15.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.519+0000 7fbd40917700 1 -- 172.21.15.70:0/3101889628 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c004220 con 0x7fbd3c101b30 2024-06-04T05:13:15.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.519+0000 7fbd40917700 1 -- 172.21.15.70:0/3101889628 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd2c0043c0 con 0x7fbd3c101b30 2024-06-04T05:13:15.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.519+0000 7fbd41919700 1 -- 172.21.15.70:0/3101889628 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd3c1a1070 con 0x7fbd3c101b30 2024-06-04T05:13:15.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.519+0000 7fbd40917700 1 -- 172.21.15.70:0/3101889628 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c00ce30 con 0x7fbd3c101b30 2024-06-04T05:13:15.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.519+0000 7fbd41919700 1 -- 172.21.15.70:0/3101889628 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd3c1a88f0 con 0x7fbd3c101b30 2024-06-04T05:13:15.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.522+0000 7fbd40917700 1 -- 172.21.15.70:0/3101889628 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fbd2c004750 con 0x7fbd3c101b30 2024-06-04T05:13:15.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.522+0000 7fbd40917700 1 --2- 172.21.15.70:0/3101889628 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd240b2a70 0x7fbd240b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.523+0000 7fbd3a7fc700 1 --2- 172.21.15.70:0/3101889628 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd240b2a70 0x7fbd240b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.523+0000 7fbd40917700 1 -- 172.21.15.70:0/3101889628 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fbd2c0e2540 con 0x7fbd3c101b30 2024-06-04T05:13:15.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.523+0000 7fbd3a7fc700 1 --2- 172.21.15.70:0/3101889628 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd240b2a70 0x7fbd240b4f30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c101200 tx=0x7fbd30005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:15.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.523+0000 7fbd4c1aeb80 1 Processor -- start 2024-06-04T05:13:15.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.524+0000 7fbd4c1aeb80 1 -- start start 2024-06-04T05:13:15.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.524+0000 7fbd4c1aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc953860 0x55c4fc953c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.524+0000 7fbd4c1aeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 0x55c4fc957940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.524+0000 7fbd4c1aeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 0x55c4fc9583a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.524+0000 7fbd4c1aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c4fc8b2c90 con 0x55c4fc953860 2024-06-04T05:13:15.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.524+0000 7fbd3a7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 0x55c4fc957940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.524+0000 7fbd3b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 0x55c4fc9583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.524+0000 7fbd3a7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 0x55c4fc957940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:42048/0 (socket says 172.21.15.70:42048) 2024-06-04T05:13:15.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.524+0000 7fbd3a7fc700 1 -- 172.21.15.70:0/714462114 learned_addr learned my addr 172.21.15.70:0/714462114 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:15.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.524+0000 7fbd3affd700 1 --2- 172.21.15.70:0/714462114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc953860 0x55c4fc953c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.524+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/714462114 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c4fc8a2830 con 0x55c4fc957f60 2024-06-04T05:13:15.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.524+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/714462114 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c4fc8a6870 con 0x55c4fc954210 2024-06-04T05:13:15.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.524+0000 7fbd3b7fe700 1 -- 172.21.15.70:0/714462114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 msgr2=0x55c4fc957940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:15.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.524+0000 7fbd3b7fe700 1 --2- 172.21.15.70:0/714462114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 0x55c4fc957940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.525+0000 7fbd3b7fe700 1 -- 172.21.15.70:0/714462114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc953860 msgr2=0x55c4fc953c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:15.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.525+0000 7fbd3b7fe700 1 --2- 172.21.15.70:0/714462114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc953860 0x55c4fc953c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.525+0000 7fbd3b7fe700 1 -- 172.21.15.70:0/714462114 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4fc7f28d0 con 0x55c4fc957f60 2024-06-04T05:13:15.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.525+0000 7fbd3b7fe700 1 --2- 172.21.15.70:0/714462114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 0x55c4fc9583a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbd34009410 tx=0x7fbd34009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a724317fd5cd45fd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:15.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.525+0000 7fbd21ffb700 1 -- 172.21.15.70:0/714462114 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd34019070 con 0x55c4fc957f60 2024-06-04T05:13:15.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.525+0000 7fbd21ffb700 1 -- 172.21.15.70:0/714462114 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd34004db0 con 0x55c4fc957f60 2024-06-04T05:13:15.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.525+0000 7fbd21ffb700 1 -- 172.21.15.70:0/714462114 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd340053e0 con 0x55c4fc957f60 2024-06-04T05:13:15.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.526+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/714462114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 msgr2=0x55c4fc9583a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.526+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/714462114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 0x55c4fc9583a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbd34009410 tx=0x7fbd34009920 comp rx=0 tx=0).stop 2024-06-04T05:13:15.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.526+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/714462114 shutdown_connections 2024-06-04T05:13:15.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.526+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/714462114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 0x55c4fc9583a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.526+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/714462114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc953860 0x55c4fc953c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.526+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/714462114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 0x55c4fc957940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.526+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/714462114 >> 172.21.15.70:0/714462114 conn(0x55c4fc881080 msgr2=0x55c4fc94de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:15.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.526+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/714462114 shutdown_connections 2024-06-04T05:13:15.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.526+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/714462114 wait complete. 2024-06-04T05:13:15.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.527+0000 7fbd4c1aeb80 1 -- start start 2024-06-04T05:13:15.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.527+0000 7fbd4c1aeb80 1 Processor -- start 2024-06-04T05:13:15.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.528+0000 7fbd4c1aeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc953860 0x55c4fc95f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.528+0000 7fbd4c1aeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 0x55c4fc95c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.528+0000 7fbd4c1aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc957f60 0x55c4fc95cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.528+0000 7fbd3affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc953860 0x55c4fc95f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.528+0000 7fbd4c1aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c4fc7a8fa0 con 0x55c4fc957f60 2024-06-04T05:13:15.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.528+0000 7fbd4c1aeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c4fc8447c0 con 0x55c4fc953860 2024-06-04T05:13:15.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.528+0000 7fbd4c1aeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c4fc960120 con 0x55c4fc954210 2024-06-04T05:13:15.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.528+0000 7fbd3a7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 0x55c4fc95c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.528+0000 7fbd3a7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 0x55c4fc95c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:42052/0 (socket says 172.21.15.70:42052) 2024-06-04T05:13:15.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.528+0000 7fbd3a7fc700 1 -- 172.21.15.70:0/572605160 learned_addr learned my addr 172.21.15.70:0/572605160 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:15.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.528+0000 7fbd4c1aeb80 10 client.0 _ll_register_callbacks cb 0x55c4fc83d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:15.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.528+0000 7fbd3b7fe700 1 --2- 172.21.15.70:0/572605160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc957f60 0x55c4fc95cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.529+0000 7fbd3b7fe700 1 -- 172.21.15.70:0/572605160 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 msgr2=0x55c4fc95c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:15.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.529+0000 7fbd3b7fe700 1 --2- 172.21.15.70:0/572605160 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 0x55c4fc95c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.529+0000 7fbd3b7fe700 1 -- 172.21.15.70:0/572605160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc953860 msgr2=0x55c4fc95f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:15.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.529+0000 7fbd3b7fe700 1 --2- 172.21.15.70:0/572605160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc953860 0x55c4fc95f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.529+0000 7fbd3b7fe700 1 -- 172.21.15.70:0/572605160 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4fc841690 con 0x55c4fc957f60 2024-06-04T05:13:15.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.529+0000 7fbd3a7fc700 1 --2- 172.21.15.70:0/572605160 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 0x55c4fc95c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:15.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.529+0000 7fbd3b7fe700 1 --2- 172.21.15.70:0/572605160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc957f60 0x55c4fc95cbe0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fbd3400b010 tx=0x7fbd34003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0c493a1109ab7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:15.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.530+0000 7fbd1affd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:13:15.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.530+0000 7fbd1affd700 1 -- 172.21.15.70:0/572605160 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd34019040 con 0x55c4fc957f60 2024-06-04T05:13:15.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.530+0000 7fbd1affd700 1 -- 172.21.15.70:0/572605160 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fbd34004450 con 0x55c4fc957f60 2024-06-04T05:13:15.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.530+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/572605160 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd34004db0 con 0x55c4fc957f60 2024-06-04T05:13:15.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.531+0000 7fbd1affd700 1 -- 172.21.15.70:0/572605160 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fbd34004650 con 0x55c4fc957f60 2024-06-04T05:13:15.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.531+0000 7fbd4c1aeb80 10 client.19644 fetch_fsmap learned FSMap version 45 2024-06-04T05:13:15.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.531+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/572605160 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55c4fc96a7f0 con 0x55c4fc957f60 2024-06-04T05:13:15.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.531+0000 7fbd1affd700 1 -- 172.21.15.70:0/572605160 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd34004db0 con 0x55c4fc957f60 2024-06-04T05:13:15.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.531+0000 7fbd1affd700 1 -- 172.21.15.70:0/572605160 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 45) ==== 8398+0+0 (secure 0 0 0) 0x7fbd34025070 con 0x55c4fc957f60 2024-06-04T05:13:15.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.531+0000 7fbd4c1aeb80 10 client.19644 fetch_fsmap finished waiting for FSMap version 45 2024-06-04T05:13:15.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.532+0000 7fbd4c1aeb80 10 client.19644 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:13:15.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.532+0000 7fbd4c1aeb80 20 client.19644 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:15.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.532+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/572605160 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55c4fc96ccc0 0x55c4fc96f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.532+0000 7fbd4c1aeb80 4 client.19644 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:13:15.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.532+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/572605160 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4fc91f9e0 con 0x55c4fc96ccc0 2024-06-04T05:13:15.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.532+0000 7fbd3affd700 1 --2- 172.21.15.70:0/572605160 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55c4fc96ccc0 0x55c4fc96f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.533+0000 7fbd1affd700 10 client.19644 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:13:15.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.533+0000 7fbd3affd700 1 --2- 172.21.15.70:0/572605160 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55c4fc96ccc0 0x55c4fc96f0b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=46af113104685a44 server_cookie=c52cc168e5ebbc72 in_seq=0 out_seq=0 2024-06-04T05:13:15.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.534+0000 7fbd1affd700 1 -- 172.21.15.70:0/572605160 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c4fc91f9e0 con 0x55c4fc96ccc0 2024-06-04T05:13:15.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.534+0000 7fbd1affd700 10 client.19644 handle_command_reply: tid=0 2024-06-04T05:13:15.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.534+0000 7fbd4c1aeb80 1 client.19644 shutdown 2024-06-04T05:13:15.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.534+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/572605160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc957f60 msgr2=0x55c4fc95cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.534+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/572605160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc957f60 0x55c4fc95cbe0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fbd3400b010 tx=0x7fbd34003f80 comp rx=0 tx=0).stop 2024-06-04T05:13:15.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.534+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/572605160 shutdown_connections 2024-06-04T05:13:15.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.534+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/572605160 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55c4fc96ccc0 0x55c4fc96f0b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.534+0000 7fbd1affd700 0 client.19644 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:13:15.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.534+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/572605160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc957f60 0x55c4fc95cbe0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.534+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/572605160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc953860 0x55c4fc95f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.534+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/572605160 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 0x55c4fc95c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.534+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/572605160 >> 172.21.15.70:0/572605160 conn(0x55c4fc881080 msgr2=0x55c4fc959350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:15.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.534+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/572605160 shutdown_connections 2024-06-04T05:13:15.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.534+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/572605160 wait complete. 2024-06-04T05:13:15.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.534+0000 7fbd4c1aeb80 20 client.19644 trim_cache size 0 max 16384 2024-06-04T05:13:15.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.554+0000 7fbd4c1aeb80 1 Processor -- start 2024-06-04T05:13:15.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.554+0000 7fbd4c1aeb80 1 -- start start 2024-06-04T05:13:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd4c1aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc953860 0x55c4fc94de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd4c1aeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 0x55c4fc94e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd4c1aeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 0x55c4fc974820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd4c1aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3c1b0180 con 0x55c4fc953860 2024-06-04T05:13:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd4c1aeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3c1b1e10 con 0x55c4fc957f60 2024-06-04T05:13:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd4c1aeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3c1b0850 con 0x55c4fc954210 2024-06-04T05:13:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd3affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc953860 0x55c4fc94de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd3affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc953860 0x55c4fc94de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43182/0 (socket says 172.21.15.70:43182) 2024-06-04T05:13:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd3affd700 1 -- 172.21.15.70:0/2315542977 learned_addr learned my addr 172.21.15.70:0/2315542977 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd3a7fc700 1 --2- 172.21.15.70:0/2315542977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 0x55c4fc94e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd3b7fe700 1 --2- 172.21.15.70:0/2315542977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 0x55c4fc974820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd3affd700 1 -- 172.21.15.70:0/2315542977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 msgr2=0x55c4fc94e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd3affd700 1 --2- 172.21.15.70:0/2315542977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 0x55c4fc94e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd3affd700 1 -- 172.21.15.70:0/2315542977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 msgr2=0x55c4fc974820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:15.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd3affd700 1 --2- 172.21.15.70:0/2315542977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 0x55c4fc974820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd3affd700 1 -- 172.21.15.70:0/2315542977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4fc960000 con 0x55c4fc953860 2024-06-04T05:13:15.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd3a7fc700 1 --2- 172.21.15.70:0/2315542977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 0x55c4fc94e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:15.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.555+0000 7fbd3b7fe700 1 --2- 172.21.15.70:0/2315542977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 0x55c4fc974820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:15.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.556+0000 7fbd3affd700 1 --2- 172.21.15.70:0/2315542977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc953860 0x55c4fc94de20 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c0a9780 tx=0x7fbd2c0cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1816f29fb4ca5c9d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.556+0000 7fbd217fa700 1 -- 172.21.15.70:0/2315542977 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c0c2070 con 0x55c4fc953860 2024-06-04T05:13:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.556+0000 7fbd217fa700 1 -- 172.21.15.70:0/2315542977 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd2c0c6020 con 0x55c4fc953860 2024-06-04T05:13:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.556+0000 7fbd217fa700 1 -- 172.21.15.70:0/2315542977 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c01c070 con 0x55c4fc953860 2024-06-04T05:13:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.557+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/2315542977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc953860 msgr2=0x55c4fc94de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.557+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/2315542977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc953860 0x55c4fc94de20 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c0a9780 tx=0x7fbd2c0cc040 comp rx=0 tx=0).stop 2024-06-04T05:13:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.557+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/2315542977 shutdown_connections 2024-06-04T05:13:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.557+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/2315542977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 0x55c4fc974820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.557+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/2315542977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc953860 0x55c4fc94de20 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.557+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/2315542977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954210 0x55c4fc94e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.557+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/2315542977 >> 172.21.15.70:0/2315542977 conn(0x55c4fc881080 msgr2=0x55c4fc95afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.557+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/2315542977 shutdown_connections 2024-06-04T05:13:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.557+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/2315542977 wait complete. 2024-06-04T05:13:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.557+0000 7fbd4c1aeb80 1 -- start start 2024-06-04T05:13:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd4c1aeb80 1 Processor -- start 2024-06-04T05:13:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd4c1aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc954210 0x55c4fc9656d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd4c1aeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 0x55c4fc965c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd4c1aeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954f90 0x55c4fc955410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd4c1aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3c1b1680 con 0x55c4fc954210 2024-06-04T05:13:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd4c1aeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3c1b0f80 con 0x55c4fc957f60 2024-06-04T05:13:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd4c1aeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3c1ae140 con 0x55c4fc954f90 2024-06-04T05:13:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd4c1aeb80 10 client.0 _ll_register_callbacks cb 0x55c4fc83d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd3affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc954210 0x55c4fc9656d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd3a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 0x55c4fc965c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd3b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954f90 0x55c4fc955410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd3a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 0x55c4fc965c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:43012/0 (socket says 172.21.15.70:43012) 2024-06-04T05:13:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd3a7fc700 1 -- 172.21.15.70:0/3514310158 learned_addr learned my addr 172.21.15.70:0/3514310158 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd3a7fc700 1 -- 172.21.15.70:0/3514310158 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954f90 msgr2=0x55c4fc955410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd3a7fc700 1 --2- 172.21.15.70:0/3514310158 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954f90 0x55c4fc955410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd3a7fc700 1 -- 172.21.15.70:0/3514310158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc954210 msgr2=0x55c4fc9656d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:15.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd3a7fc700 1 --2- 172.21.15.70:0/3514310158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc954210 0x55c4fc9656d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd3a7fc700 1 -- 172.21.15.70:0/3514310158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4fc955c50 con 0x55c4fc957f60 2024-06-04T05:13:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.558+0000 7fbd3b7fe700 1 --2- 172.21.15.70:0/3514310158 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954f90 0x55c4fc955410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.559+0000 7fbd3a7fc700 1 --2- 172.21.15.70:0/3514310158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 0x55c4fc965c10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbd30000c00 tx=0x7fbd3001f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f8d1f92f5f61ad server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.559+0000 7fbd1affd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:13:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.559+0000 7fbd1affd700 1 -- 172.21.15.70:0/3514310158 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3000f930 con 0x55c4fc957f60 2024-06-04T05:13:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.559+0000 7fbd1affd700 1 -- 172.21.15.70:0/3514310158 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fbd3000fad0 con 0x55c4fc957f60 2024-06-04T05:13:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.559+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/3514310158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd2c0c6020 con 0x55c4fc957f60 2024-06-04T05:13:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.559+0000 7fbd1affd700 1 -- 172.21.15.70:0/3514310158 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fbd3000fcd0 con 0x55c4fc957f60 2024-06-04T05:13:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.559+0000 7fbd4c1aeb80 10 client.28490 fetch_fsmap learned FSMap version 45 2024-06-04T05:13:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.559+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/3514310158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55c4fc9d76d0 con 0x55c4fc957f60 2024-06-04T05:13:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.559+0000 7fbd1affd700 1 -- 172.21.15.70:0/3514310158 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd2c0c6020 con 0x55c4fc957f60 2024-06-04T05:13:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.560+0000 7fbd1affd700 1 -- 172.21.15.70:0/3514310158 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 45) ==== 8398+0+0 (secure 0 0 0) 0x7fbd30021070 con 0x55c4fc957f60 2024-06-04T05:13:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.560+0000 7fbd4c1aeb80 10 client.28490 fetch_fsmap finished waiting for FSMap version 45 2024-06-04T05:13:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.560+0000 7fbd4c1aeb80 10 client.28490 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:13:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.560+0000 7fbd4c1aeb80 20 client.28490 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.560+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/3514310158 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55c4fc9d7e60 0x55c4fc9da280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.560+0000 7fbd4c1aeb80 4 client.28490 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:13:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.560+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/3514310158 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c4fc8b4be0 con 0x55c4fc9d7e60 2024-06-04T05:13:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.560+0000 7fbd3affd700 1 --2- 172.21.15.70:0/3514310158 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55c4fc9d7e60 0x55c4fc9da280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.560+0000 7fbd3affd700 1 --2- 172.21.15.70:0/3514310158 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55c4fc9d7e60 0x55c4fc9da280 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7b87b00a2450097e server_cookie=df28891cbfe4eec7 in_seq=0 out_seq=0 2024-06-04T05:13:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.560+0000 7fbd1affd700 10 client.28490 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:13:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.561+0000 7fbd1affd700 1 -- 172.21.15.70:0/3514310158 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c4fc8b4be0 con 0x55c4fc9d7e60 2024-06-04T05:13:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.561+0000 7fbd1affd700 10 client.28490 handle_command_reply: tid=0 2024-06-04T05:13:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.561+0000 7fbd4c1aeb80 1 client.28490 shutdown 2024-06-04T05:13:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.561+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/3514310158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 msgr2=0x55c4fc965c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.561+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/3514310158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 0x55c4fc965c10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbd30000c00 tx=0x7fbd3001f000 comp rx=0 tx=0).stop 2024-06-04T05:13:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.561+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/3514310158 shutdown_connections 2024-06-04T05:13:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.561+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/3514310158 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55c4fc9d7e60 0x55c4fc9da280 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.561+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/3514310158 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c4fc954f90 0x55c4fc955410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.561+0000 7fbd1affd700 0 client.28490 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:13:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.561+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/3514310158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4fc954210 0x55c4fc9656d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.561+0000 7fbd4c1aeb80 1 --2- 172.21.15.70:0/3514310158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c4fc957f60 0x55c4fc965c10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.561+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/3514310158 >> 172.21.15.70:0/3514310158 conn(0x55c4fc881080 msgr2=0x55c4fc95ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.561+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/3514310158 shutdown_connections 2024-06-04T05:13:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.561+0000 7fbd4c1aeb80 1 -- 172.21.15.70:0/3514310158 wait complete. 2024-06-04T05:13:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.562+0000 7fbd4c1aeb80 20 client.28490 trim_cache size 0 max 16384 2024-06-04T05:13:15.568 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:13:15.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.562+0000 7fbd41919700 1 -- 172.21.15.70:0/3101889628 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd240b2a70 msgr2=0x7fbd240b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.562+0000 7fbd41919700 1 --2- 172.21.15.70:0/3101889628 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd240b2a70 0x7fbd240b4f30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c101200 tx=0x7fbd30005fb0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.562+0000 7fbd41919700 1 -- 172.21.15.70:0/3101889628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd3c101b30 msgr2=0x7fbd3c19bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:15.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.562+0000 7fbd41919700 1 --2- 172.21.15.70:0/3101889628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd3c101b30 0x7fbd3c19bd70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c00e9f0 tx=0x7fbd2c00ed00 comp rx=0 tx=0).stop 2024-06-04T05:13:15.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.562+0000 7fbd41919700 1 -- 172.21.15.70:0/3101889628 shutdown_connections 2024-06-04T05:13:15.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.562+0000 7fbd41919700 1 --2- 172.21.15.70:0/3101889628 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd240b2a70 0x7fbd240b4f30 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.562+0000 7fbd41919700 1 --2- 172.21.15.70:0/3101889628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd3c104450 0x7fbd3c1a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.562+0000 7fbd41919700 1 --2- 172.21.15.70:0/3101889628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd3c101b30 0x7fbd3c19bd70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.562+0000 7fbd41919700 1 --2- 172.21.15.70:0/3101889628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd3c1025d0 0x7fbd3c19c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:15.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.562+0000 7fbd41919700 1 -- 172.21.15.70:0/3101889628 >> 172.21.15.70:0/3101889628 conn(0x7fbd3c078ab0 msgr2=0x7fbd3c0ff520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:15.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.562+0000 7fbd41919700 1 -- 172.21.15.70:0/3101889628 shutdown_connections 2024-06-04T05:13:15.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:15.562+0000 7fbd41919700 1 -- 172.21.15.70:0/3101889628 wait complete. 2024-06-04T05:13:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:15 smithi112 ceph-mon[36453]: pgmap v503: 97 pgs: 97 active+clean; 451 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 3.8 MiB/s wr, 215 op/s 2024-06-04T05:13:15.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 17658 8.82 MB/sec execute 7 sec latency 96.784 ms 2024-06-04T05:13:16.345 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:13:16.730 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:13:16.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 17822 8.05 MB/sec execute 8 sec latency 62.483 ms 2024-06-04T05:13:17.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:17 smithi098 ceph-mon[35569]: pgmap v504: 97 pgs: 97 active+clean; 461 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.3 MiB/s wr, 331 op/s 2024-06-04T05:13:17.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:17 smithi070 ceph-mon[32631]: pgmap v504: 97 pgs: 97 active+clean; 461 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.3 MiB/s wr, 331 op/s 2024-06-04T05:13:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:17 smithi112 ceph-mon[36453]: pgmap v504: 97 pgs: 97 active+clean; 461 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.3 MiB/s wr, 331 op/s 2024-06-04T05:13:17.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 18016 8.55 MB/sec execute 9 sec latency 58.989 ms 2024-06-04T05:13:18.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.543+0000 7fe4045b0700 1 -- 172.21.15.70:0/174681941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3fc06b220 msgr2=0x7fe3fc06b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:18.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.543+0000 7fe4045b0700 1 --2- 172.21.15.70:0/174681941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3fc06b220 0x7fe3fc06b6a0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec009990 tx=0x7fe3ec009ca0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.543+0000 7fe4045b0700 1 -- 172.21.15.70:0/174681941 shutdown_connections 2024-06-04T05:13:18.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.543+0000 7fe4045b0700 1 --2- 172.21.15.70:0/174681941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3fc06bbe0 0x7fe3fc1103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.543+0000 7fe4045b0700 1 --2- 172.21.15.70:0/174681941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3fc1095b0 0x7fe3fc109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.543+0000 7fe4045b0700 1 --2- 172.21.15.70:0/174681941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3fc06b220 0x7fe3fc06b6a0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.543+0000 7fe4045b0700 1 -- 172.21.15.70:0/174681941 >> 172.21.15.70:0/174681941 conn(0x7fe3fc0793f0 msgr2=0x7fe3fc079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:18.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.544+0000 7fe4045b0700 1 -- 172.21.15.70:0/174681941 shutdown_connections 2024-06-04T05:13:18.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.544+0000 7fe4045b0700 1 -- 172.21.15.70:0/174681941 wait complete. 2024-06-04T05:13:18.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.545+0000 7fe4045b0700 1 Processor -- start 2024-06-04T05:13:18.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.545+0000 7fe4045b0700 1 -- start start 2024-06-04T05:13:18.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.545+0000 7fe4045b0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3fc06b220 0x7fe3fc10fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.545+0000 7fe4045b0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3fc06bbe0 0x7fe3fc110350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.545+0000 7fe4045b0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3fc1095b0 0x7fe3fc10d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.545+0000 7fe4045b0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc103980 con 0x7fe3fc1095b0 2024-06-04T05:13:18.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.545+0000 7fe4045b0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc103800 con 0x7fe3fc06b220 2024-06-04T05:13:18.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.545+0000 7fe4045b0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc103b00 con 0x7fe3fc06bbe0 2024-06-04T05:13:18.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.545+0000 7fe40234c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3fc06b220 0x7fe3fc10fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.545+0000 7fe40234c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3fc06b220 0x7fe3fc10fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:41232/0 (socket says 172.21.15.70:41232) 2024-06-04T05:13:18.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.545+0000 7fe40234c700 1 -- 172.21.15.70:0/1096313791 learned_addr learned my addr 172.21.15.70:0/1096313791 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:18.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.545+0000 7fe401b4b700 1 --2- 172.21.15.70:0/1096313791 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3fc06bbe0 0x7fe3fc110350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.545+0000 7fe402b4d700 1 --2- 172.21.15.70:0/1096313791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3fc1095b0 0x7fe3fc10d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.546+0000 7fe40234c700 1 -- 172.21.15.70:0/1096313791 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3fc06bbe0 msgr2=0x7fe3fc110350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:18.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.546+0000 7fe40234c700 1 --2- 172.21.15.70:0/1096313791 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3fc06bbe0 0x7fe3fc110350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.546+0000 7fe40234c700 1 -- 172.21.15.70:0/1096313791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3fc1095b0 msgr2=0x7fe3fc10d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:18.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.546+0000 7fe40234c700 1 --2- 172.21.15.70:0/1096313791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3fc1095b0 0x7fe3fc10d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.546+0000 7fe40234c700 1 -- 172.21.15.70:0/1096313791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3fc10d9a0 con 0x7fe3fc06b220 2024-06-04T05:13:18.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.546+0000 7fe401b4b700 1 --2- 172.21.15.70:0/1096313791 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3fc06bbe0 0x7fe3fc110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:18.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.546+0000 7fe40234c700 1 --2- 172.21.15.70:0/1096313791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3fc06b220 0x7fe3fc10fe10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe3f800e9a0 tx=0x7fe3f800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:18.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.546+0000 7fe3f37fe700 1 -- 172.21.15.70:0/1096313791 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f800ca10 con 0x7fe3fc06b220 2024-06-04T05:13:18.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.546+0000 7fe3f37fe700 1 -- 172.21.15.70:0/1096313791 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe3f800cbb0 con 0x7fe3fc06b220 2024-06-04T05:13:18.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.546+0000 7fe3f37fe700 1 -- 172.21.15.70:0/1096313791 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8018680 con 0x7fe3fc06b220 2024-06-04T05:13:18.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.546+0000 7fe4045b0700 1 -- 172.21.15.70:0/1096313791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3fc10dc90 con 0x7fe3fc06b220 2024-06-04T05:13:18.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.546+0000 7fe4045b0700 1 -- 172.21.15.70:0/1096313791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3fc1acbb0 con 0x7fe3fc06b220 2024-06-04T05:13:18.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.550+0000 7fe3f37fe700 1 -- 172.21.15.70:0/1096313791 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fe3f8010430 con 0x7fe3fc06b220 2024-06-04T05:13:18.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.551+0000 7fe3f37fe700 1 --2- 172.21.15.70:0/1096313791 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe3e80b2af0 0x7fe3e80b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.551+0000 7fe3f37fe700 1 -- 172.21.15.70:0/1096313791 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fe3f80e26c0 con 0x7fe3fc06b220 2024-06-04T05:13:18.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.551+0000 7fe401b4b700 1 --2- 172.21.15.70:0/1096313791 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe3e80b2af0 0x7fe3e80b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.552+0000 7fe40ee45b80 1 Processor -- start 2024-06-04T05:13:18.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.552+0000 7fe401b4b700 1 --2- 172.21.15.70:0/1096313791 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe3e80b2af0 0x7fe3e80b4fb0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc10e710 tx=0x7fe3ec01a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:18.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.552+0000 7fe40ee45b80 1 -- start start 2024-06-04T05:13:18.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.552+0000 7fe40ee45b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310704320 0x564310704700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.552+0000 7fe40ee45b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643106fe8a0 0x5643106fed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.552+0000 7fe40ee45b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106ff260 0x56431070b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.552+0000 7fe40ee45b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56431065cd70 con 0x5643106fe8a0 2024-06-04T05:13:18.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.552+0000 7fe40ee45b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56431064c910 con 0x5643106ff260 2024-06-04T05:13:18.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.552+0000 7fe40ee45b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564310650950 con 0x564310704320 2024-06-04T05:13:18.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.553+0000 7fe40234c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310704320 0x564310704700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.553+0000 7fe40234c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310704320 0x564310704700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:57796/0 (socket says 172.21.15.70:57796) 2024-06-04T05:13:18.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.553+0000 7fe40234c700 1 -- 172.21.15.70:0/1583343985 learned_addr learned my addr 172.21.15.70:0/1583343985 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:18.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.553+0000 7fe402b4d700 1 --2- 172.21.15.70:0/1583343985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106ff260 0x56431070b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.553+0000 7fe401b4b700 1 --2- 172.21.15.70:0/1583343985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643106fe8a0 0x5643106fed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.553+0000 7fe40234c700 1 -- 172.21.15.70:0/1583343985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106ff260 msgr2=0x56431070b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:18.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.553+0000 7fe40234c700 1 --2- 172.21.15.70:0/1583343985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106ff260 0x56431070b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.553+0000 7fe40234c700 1 -- 172.21.15.70:0/1583343985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643106fe8a0 msgr2=0x5643106fed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:18.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.553+0000 7fe40234c700 1 --2- 172.21.15.70:0/1583343985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643106fe8a0 0x5643106fed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.553+0000 7fe40234c700 1 -- 172.21.15.70:0/1583343985 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643105f5fb0 con 0x564310704320 2024-06-04T05:13:18.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.553+0000 7fe402b4d700 1 --2- 172.21.15.70:0/1583343985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106ff260 0x56431070b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:13:18.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.553+0000 7fe40234c700 1 --2- 172.21.15.70:0/1583343985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310704320 0x564310704700 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe3f80a5990 tx=0x7fe3f80a9510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75664367440f3316 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:18.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.553+0000 7fe3f0ff9700 1 -- 172.21.15.70:0/1583343985 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f80adcc0 con 0x564310704320 2024-06-04T05:13:18.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.553+0000 7fe3f0ff9700 1 -- 172.21.15.70:0/1583343985 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe3f8010e50 con 0x564310704320 2024-06-04T05:13:18.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.553+0000 7fe3f0ff9700 1 -- 172.21.15.70:0/1583343985 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f80b22e0 con 0x564310704320 2024-06-04T05:13:18.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.554+0000 7fe40ee45b80 1 -- 172.21.15.70:0/1583343985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310704320 msgr2=0x564310704700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:18.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.554+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/1583343985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310704320 0x564310704700 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe3f80a5990 tx=0x7fe3f80a9510 comp rx=0 tx=0).stop 2024-06-04T05:13:18.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.554+0000 7fe40ee45b80 1 -- 172.21.15.70:0/1583343985 shutdown_connections 2024-06-04T05:13:18.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.554+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/1583343985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106ff260 0x56431070b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.554+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/1583343985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310704320 0x564310704700 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.554+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/1583343985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643106fe8a0 0x5643106fed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.554+0000 7fe40ee45b80 1 -- 172.21.15.70:0/1583343985 >> 172.21.15.70:0/1583343985 conn(0x56431062a240 msgr2=0x5643106f93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:18.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.554+0000 7fe40ee45b80 1 -- 172.21.15.70:0/1583343985 shutdown_connections 2024-06-04T05:13:18.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.554+0000 7fe40ee45b80 1 -- 172.21.15.70:0/1583343985 wait complete. 2024-06-04T05:13:18.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.555+0000 7fe40ee45b80 1 -- start start 2024-06-04T05:13:18.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.555+0000 7fe40ee45b80 1 Processor -- start 2024-06-04T05:13:18.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.555+0000 7fe40ee45b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643106fe8a0 0x56431070ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.555+0000 7fe40ee45b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106ff260 0x564310708090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.555+0000 7fe40ee45b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310704320 0x5643107085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.555+0000 7fe40ee45b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564310552fa0 con 0x564310704320 2024-06-04T05:13:18.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.555+0000 7fe40ee45b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5643105b4da0 con 0x5643106ff260 2024-06-04T05:13:18.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.555+0000 7fe40ee45b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56431070b6e0 con 0x5643106fe8a0 2024-06-04T05:13:18.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.555+0000 7fe40ee45b80 10 client.0 _ll_register_callbacks cb 0x5643105e7950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:18.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.555+0000 7fe401b4b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106ff260 0x564310708090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.555+0000 7fe401b4b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106ff260 0x564310708090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:41246/0 (socket says 172.21.15.70:41246) 2024-06-04T05:13:18.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.555+0000 7fe401b4b700 1 -- 172.21.15.70:0/350734316 learned_addr learned my addr 172.21.15.70:0/350734316 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:18.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.556+0000 7fe40234c700 1 --2- 172.21.15.70:0/350734316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643106fe8a0 0x56431070ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.556+0000 7fe402b4d700 1 --2- 172.21.15.70:0/350734316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310704320 0x5643107085d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.556+0000 7fe401b4b700 1 -- 172.21.15.70:0/350734316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643106fe8a0 msgr2=0x56431070ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:18.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.556+0000 7fe401b4b700 1 --2- 172.21.15.70:0/350734316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643106fe8a0 0x56431070ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.556+0000 7fe401b4b700 1 -- 172.21.15.70:0/350734316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310704320 msgr2=0x5643107085d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:18.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.556+0000 7fe401b4b700 1 --2- 172.21.15.70:0/350734316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310704320 0x5643107085d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.556+0000 7fe401b4b700 1 -- 172.21.15.70:0/350734316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56431059c8d0 con 0x5643106ff260 2024-06-04T05:13:18.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.556+0000 7fe40234c700 1 --2- 172.21.15.70:0/350734316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643106fe8a0 0x56431070ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:18.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.556+0000 7fe401b4b700 1 --2- 172.21.15.70:0/350734316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106ff260 0x564310708090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec015000 tx=0x7fe3ec009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d5f50a4e3d37721 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:18.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.556+0000 7fe3e5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:13:18.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.556+0000 7fe3e5ffb700 1 -- 172.21.15.70:0/350734316 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec004cb0 con 0x5643106ff260 2024-06-04T05:13:18.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.556+0000 7fe3e5ffb700 1 -- 172.21.15.70:0/350734316 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fe3ec026430 con 0x5643106ff260 2024-06-04T05:13:18.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.556+0000 7fe40ee45b80 1 -- 172.21.15.70:0/350734316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3f8010e50 con 0x5643106ff260 2024-06-04T05:13:18.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.557+0000 7fe3e5ffb700 1 -- 172.21.15.70:0/350734316 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fe3ec004e50 con 0x5643106ff260 2024-06-04T05:13:18.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.557+0000 7fe40ee45b80 10 client.28502 fetch_fsmap learned FSMap version 45 2024-06-04T05:13:18.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.557+0000 7fe40ee45b80 1 -- 172.21.15.70:0/350734316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x564310606760 con 0x5643106ff260 2024-06-04T05:13:18.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.557+0000 7fe3e5ffb700 1 -- 172.21.15.70:0/350734316 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe3f8010e50 con 0x5643106ff260 2024-06-04T05:13:18.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.557+0000 7fe3e5ffb700 1 -- 172.21.15.70:0/350734316 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 45) ==== 8398+0+0 (secure 0 0 0) 0x7fe3ec039710 con 0x5643106ff260 2024-06-04T05:13:18.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.557+0000 7fe40ee45b80 10 client.28502 fetch_fsmap finished waiting for FSMap version 45 2024-06-04T05:13:18.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.557+0000 7fe40ee45b80 10 client.28502 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:13:18.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.557+0000 7fe40ee45b80 20 client.28502 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:18.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.557+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/350734316 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5643107182b0 0x56431071a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.557+0000 7fe40ee45b80 4 client.28502 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:13:18.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.557+0000 7fe40ee45b80 1 -- 172.21.15.70:0/350734316 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56431065ecc0 con 0x5643107182b0 2024-06-04T05:13:18.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.557+0000 7fe40234c700 1 --2- 172.21.15.70:0/350734316 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5643107182b0 0x56431071a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.558+0000 7fe3e5ffb700 10 client.28502 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:13:18.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.558+0000 7fe40234c700 1 --2- 172.21.15.70:0/350734316 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5643107182b0 0x56431071a730 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8d5a64921d9a24bd server_cookie=1eb941faae50e4b9 in_seq=0 out_seq=0 2024-06-04T05:13:18.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.559+0000 7fe3e5ffb700 1 -- 172.21.15.70:0/350734316 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56431065ecc0 con 0x5643107182b0 2024-06-04T05:13:18.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.559+0000 7fe3e5ffb700 10 client.28502 handle_command_reply: tid=0 2024-06-04T05:13:18.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.559+0000 7fe40ee45b80 1 client.28502 shutdown 2024-06-04T05:13:18.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.559+0000 7fe40ee45b80 1 -- 172.21.15.70:0/350734316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106ff260 msgr2=0x564310708090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:18.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.559+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/350734316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106ff260 0x564310708090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec015000 tx=0x7fe3ec009310 comp rx=0 tx=0).stop 2024-06-04T05:13:18.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.559+0000 7fe40ee45b80 1 -- 172.21.15.70:0/350734316 shutdown_connections 2024-06-04T05:13:18.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.559+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/350734316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564310704320 0x5643107085d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.559+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/350734316 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5643107182b0 0x56431071a730 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.559+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/350734316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643106fe8a0 0x56431070ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.559+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/350734316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106ff260 0x564310708090 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.559+0000 7fe40ee45b80 1 -- 172.21.15.70:0/350734316 >> 172.21.15.70:0/350734316 conn(0x56431062a240 msgr2=0x5643106f8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:18.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.559+0000 7fe3e5ffb700 0 client.28502 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:13:18.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.559+0000 7fe40ee45b80 1 -- 172.21.15.70:0/350734316 shutdown_connections 2024-06-04T05:13:18.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.559+0000 7fe40ee45b80 1 -- 172.21.15.70:0/350734316 wait complete. 2024-06-04T05:13:18.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.559+0000 7fe40ee45b80 20 client.28502 trim_cache size 0 max 16384 2024-06-04T05:13:18.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.582+0000 7fe40ee45b80 1 Processor -- start 2024-06-04T05:13:18.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.582+0000 7fe40ee45b80 1 -- start start 2024-06-04T05:13:18.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.582+0000 7fe40ee45b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106fe8a0 0x564310703830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.582+0000 7fe40ee45b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643106ff260 0x564310707860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.582+0000 7fe40ee45b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310704320 0x5643106f9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.582+0000 7fe40ee45b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc1b4480 con 0x5643106ff260 2024-06-04T05:13:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.582+0000 7fe40234c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106fe8a0 0x564310703830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.582+0000 7fe40234c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106fe8a0 0x564310703830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:41260/0 (socket says 172.21.15.70:41260) 2024-06-04T05:13:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.582+0000 7fe40234c700 1 -- 172.21.15.70:0/1731004203 learned_addr learned my addr 172.21.15.70:0/1731004203 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.582+0000 7fe40ee45b80 1 -- 172.21.15.70:0/1731004203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc1b6110 con 0x5643106fe8a0 2024-06-04T05:13:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.582+0000 7fe40ee45b80 1 -- 172.21.15.70:0/1731004203 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc1b4b50 con 0x564310704320 2024-06-04T05:13:18.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.582+0000 7fe402b4d700 1 --2- 172.21.15.70:0/1731004203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310704320 0x5643106f9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.582+0000 7fe401b4b700 1 --2- 172.21.15.70:0/1731004203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643106ff260 0x564310707860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.583+0000 7fe402b4d700 1 -- 172.21.15.70:0/1731004203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106fe8a0 msgr2=0x564310703830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:18.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.583+0000 7fe402b4d700 1 --2- 172.21.15.70:0/1731004203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106fe8a0 0x564310703830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.583+0000 7fe402b4d700 1 -- 172.21.15.70:0/1731004203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643106ff260 msgr2=0x564310707860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:18.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.583+0000 7fe402b4d700 1 --2- 172.21.15.70:0/1731004203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643106ff260 0x564310707860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.583+0000 7fe402b4d700 1 -- 172.21.15.70:0/1731004203 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643106fb380 con 0x564310704320 2024-06-04T05:13:18.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.583+0000 7fe402b4d700 1 --2- 172.21.15.70:0/1731004203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310704320 0x5643106f9270 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4007530 tx=0x7fe3f400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e0f3e309d3b7e10 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.583+0000 7fe3e7fff700 1 -- 172.21.15.70:0/1731004203 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4003c00 con 0x564310704320 2024-06-04T05:13:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.583+0000 7fe3e7fff700 1 -- 172.21.15.70:0/1731004203 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe3f401b070 con 0x564310704320 2024-06-04T05:13:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.583+0000 7fe3e7fff700 1 -- 172.21.15.70:0/1731004203 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4017670 con 0x564310704320 2024-06-04T05:13:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.584+0000 7fe40ee45b80 1 -- 172.21.15.70:0/1731004203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310704320 msgr2=0x5643106f9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.584+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/1731004203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310704320 0x5643106f9270 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4007530 tx=0x7fe3f400b040 comp rx=0 tx=0).stop 2024-06-04T05:13:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.584+0000 7fe40ee45b80 1 -- 172.21.15.70:0/1731004203 shutdown_connections 2024-06-04T05:13:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.584+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/1731004203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564310704320 0x5643106f9270 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.584+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/1731004203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5643106fe8a0 0x564310703830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.584+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/1731004203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643106ff260 0x564310707860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.584+0000 7fe40ee45b80 1 -- 172.21.15.70:0/1731004203 >> 172.21.15.70:0/1731004203 conn(0x56431062a240 msgr2=0x5643106fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.585+0000 7fe40ee45b80 1 -- 172.21.15.70:0/1731004203 shutdown_connections 2024-06-04T05:13:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.585+0000 7fe40ee45b80 1 -- 172.21.15.70:0/1731004203 wait complete. 2024-06-04T05:13:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.585+0000 7fe40ee45b80 1 -- start start 2024-06-04T05:13:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.585+0000 7fe40ee45b80 1 Processor -- start 2024-06-04T05:13:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.585+0000 7fe40ee45b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643106fe8a0 0x56431071f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.585+0000 7fe40ee45b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643106ff260 0x56431071fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.585+0000 7fe40ee45b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310704320 0x56431070f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.585+0000 7fe40ee45b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc1b5980 con 0x5643106fe8a0 2024-06-04T05:13:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.585+0000 7fe40ee45b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc1b5280 con 0x564310704320 2024-06-04T05:13:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.585+0000 7fe40ee45b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc1b2440 con 0x5643106ff260 2024-06-04T05:13:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.585+0000 7fe40ee45b80 10 client.0 _ll_register_callbacks cb 0x5643105e7950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.586+0000 7fe402b4d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310704320 0x56431070f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.586+0000 7fe401b4b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643106ff260 0x56431071fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.586+0000 7fe40234c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643106fe8a0 0x56431071f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.586+0000 7fe402b4d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310704320 0x56431070f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:41272/0 (socket says 172.21.15.70:41272) 2024-06-04T05:13:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.586+0000 7fe402b4d700 1 -- 172.21.15.70:0/37712089 learned_addr learned my addr 172.21.15.70:0/37712089 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.586+0000 7fe401b4b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643106ff260 0x56431071fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:57824/0 (socket says 172.21.15.70:57824) 2024-06-04T05:13:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.586+0000 7fe402b4d700 1 -- 172.21.15.70:0/37712089 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643106ff260 msgr2=0x56431071fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:18.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.586+0000 7fe402b4d700 1 --2- 172.21.15.70:0/37712089 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643106ff260 0x56431071fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.586+0000 7fe402b4d700 1 -- 172.21.15.70:0/37712089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643106fe8a0 msgr2=0x56431071f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:18.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.586+0000 7fe402b4d700 1 --2- 172.21.15.70:0/37712089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643106fe8a0 0x56431071f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.586+0000 7fe402b4d700 1 -- 172.21.15.70:0/37712089 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56431070fe70 con 0x564310704320 2024-06-04T05:13:18.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.586+0000 7fe402b4d700 1 --2- 172.21.15.70:0/37712089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310704320 0x56431070f930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4020d70 tx=0x7fe3f4020da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e01d8ab0e686208a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:18.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.586+0000 7fe3e5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:13:18.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.586+0000 7fe3e5ffb700 1 -- 172.21.15.70:0/37712089 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4005050 con 0x564310704320 2024-06-04T05:13:18.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.587+0000 7fe3e5ffb700 1 -- 172.21.15.70:0/37712089 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fe3f40051f0 con 0x564310704320 2024-06-04T05:13:18.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.587+0000 7fe40ee45b80 1 -- 172.21.15.70:0/37712089 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3f401b070 con 0x564310704320 2024-06-04T05:13:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.587+0000 7fe3e5ffb700 1 -- 172.21.15.70:0/37712089 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fe3f40053f0 con 0x564310704320 2024-06-04T05:13:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.587+0000 7fe40ee45b80 10 client.28508 fetch_fsmap learned FSMap version 45 2024-06-04T05:13:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.587+0000 7fe40ee45b80 1 -- 172.21.15.70:0/37712089 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5643107110e0 con 0x564310704320 2024-06-04T05:13:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.587+0000 7fe3e5ffb700 1 -- 172.21.15.70:0/37712089 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe3f4005a20 con 0x564310704320 2024-06-04T05:13:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.587+0000 7fe3e5ffb700 1 -- 172.21.15.70:0/37712089 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 45) ==== 8398+0+0 (secure 0 0 0) 0x7fe3f4027070 con 0x564310704320 2024-06-04T05:13:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.588+0000 7fe40ee45b80 10 client.28508 fetch_fsmap finished waiting for FSMap version 45 2024-06-04T05:13:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.588+0000 7fe40ee45b80 10 client.28508 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:13:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.588+0000 7fe40ee45b80 20 client.28508 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.588+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/37712089 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x564310782f20 0x564310785340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.588+0000 7fe40ee45b80 4 client.28508 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:13:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.588+0000 7fe40ee45b80 1 -- 172.21.15.70:0/37712089 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fe3f401b070 con 0x564310782f20 2024-06-04T05:13:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.588+0000 7fe40234c700 1 --2- 172.21.15.70:0/37712089 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x564310782f20 0x564310785340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.588+0000 7fe3e5ffb700 10 client.28508 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:13:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.588+0000 7fe40234c700 1 --2- 172.21.15.70:0/37712089 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x564310782f20 0x564310785340 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=91f30b4d8f0c4e27 server_cookie=155d7004fd554803 in_seq=0 out_seq=0 2024-06-04T05:13:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.589+0000 7fe3e5ffb700 1 -- 172.21.15.70:0/37712089 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fe3f401b070 con 0x564310782f20 2024-06-04T05:13:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.589+0000 7fe3e5ffb700 10 client.28508 handle_command_reply: tid=0 2024-06-04T05:13:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.589+0000 7fe40ee45b80 1 client.28508 shutdown 2024-06-04T05:13:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.589+0000 7fe40ee45b80 1 -- 172.21.15.70:0/37712089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310704320 msgr2=0x56431070f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.589+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/37712089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310704320 0x56431070f930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4020d70 tx=0x7fe3f4020da0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.589+0000 7fe40ee45b80 1 -- 172.21.15.70:0/37712089 shutdown_connections 2024-06-04T05:13:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.589+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/37712089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564310704320 0x56431070f930 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.589+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/37712089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643106fe8a0 0x56431071f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.589+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/37712089 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x564310782f20 0x564310785340 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.589+0000 7fe40ee45b80 1 --2- 172.21.15.70:0/37712089 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643106ff260 0x56431071fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.589+0000 7fe3e5ffb700 0 client.28508 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:13:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.590+0000 7fe40ee45b80 1 -- 172.21.15.70:0/37712089 >> 172.21.15.70:0/37712089 conn(0x56431062a240 msgr2=0x5643106fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.590+0000 7fe40ee45b80 1 -- 172.21.15.70:0/37712089 shutdown_connections 2024-06-04T05:13:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.590+0000 7fe40ee45b80 1 -- 172.21.15.70:0/37712089 wait complete. 2024-06-04T05:13:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.590+0000 7fe40ee45b80 20 client.28508 trim_cache size 0 max 16384 2024-06-04T05:13:18.596 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:13:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.591+0000 7fe4045b0700 1 -- 172.21.15.70:0/1096313791 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe3e80b2af0 msgr2=0x7fe3e80b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.591+0000 7fe4045b0700 1 --2- 172.21.15.70:0/1096313791 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe3e80b2af0 0x7fe3e80b4fb0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc10e710 tx=0x7fe3ec01a040 comp rx=0 tx=0).stop 2024-06-04T05:13:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.591+0000 7fe4045b0700 1 -- 172.21.15.70:0/1096313791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3fc06b220 msgr2=0x7fe3fc10fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.591+0000 7fe4045b0700 1 --2- 172.21.15.70:0/1096313791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3fc06b220 0x7fe3fc10fe10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe3f800e9a0 tx=0x7fe3f800ecb0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.591+0000 7fe4045b0700 1 -- 172.21.15.70:0/1096313791 shutdown_connections 2024-06-04T05:13:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.591+0000 7fe4045b0700 1 --2- 172.21.15.70:0/1096313791 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe3e80b2af0 0x7fe3e80b4fb0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.591+0000 7fe4045b0700 1 --2- 172.21.15.70:0/1096313791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3fc1095b0 0x7fe3fc10d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.591+0000 7fe4045b0700 1 --2- 172.21.15.70:0/1096313791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3fc06b220 0x7fe3fc10fe10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.591+0000 7fe4045b0700 1 --2- 172.21.15.70:0/1096313791 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3fc06bbe0 0x7fe3fc110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:18.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.591+0000 7fe4045b0700 1 -- 172.21.15.70:0/1096313791 >> 172.21.15.70:0/1096313791 conn(0x7fe3fc0793f0 msgr2=0x7fe3fc107100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:18.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.591+0000 7fe4045b0700 1 -- 172.21.15.70:0/1096313791 shutdown_connections 2024-06-04T05:13:18.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:18.591+0000 7fe4045b0700 1 -- 172.21.15.70:0/1096313791 wait complete. 2024-06-04T05:13:18.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 18204 8.65 MB/sec execute 10 sec latency 120.972 ms 2024-06-04T05:13:19.374 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:13:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:19 smithi098 ceph-mon[35569]: pgmap v505: 97 pgs: 97 active+clean; 461 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.1 MiB/s wr, 322 op/s 2024-06-04T05:13:19.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:19 smithi070 ceph-mon[32631]: pgmap v505: 97 pgs: 97 active+clean; 461 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.1 MiB/s wr, 322 op/s 2024-06-04T05:13:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:19 smithi112 ceph-mon[36453]: pgmap v505: 97 pgs: 97 active+clean; 461 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.1 MiB/s wr, 322 op/s 2024-06-04T05:13:19.703 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:13:19.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 18378 8.69 MB/sec execute 11 sec latency 54.122 ms 2024-06-04T05:13:20.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 18534 8.27 MB/sec execute 12 sec latency 122.965 ms 2024-06-04T05:13:21.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:21 smithi098 ceph-mon[35569]: pgmap v506: 97 pgs: 97 active+clean; 461 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 275 op/s 2024-06-04T05:13:21.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:21 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:21.538 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:21 smithi070 ceph-mon[32631]: pgmap v506: 97 pgs: 97 active+clean; 461 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 275 op/s 2024-06-04T05:13:21.538 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:21 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:21 smithi112 ceph-mon[36453]: pgmap v506: 97 pgs: 97 active+clean; 461 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 275 op/s 2024-06-04T05:13:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:21 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:21.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 18763 8.80 MB/sec execute 13 sec latency 54.581 ms 2024-06-04T05:13:21.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.852+0000 7f2933f40700 1 -- 172.21.15.70:0/2007153703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f292c108510 msgr2=0x7f292c1088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:21.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.852+0000 7f2933f40700 1 --2- 172.21.15.70:0/2007153703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f292c108510 0x7f292c1088f0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f2928009990 tx=0x7f2928009ca0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.852+0000 7f2933f40700 1 -- 172.21.15.70:0/2007153703 shutdown_connections 2024-06-04T05:13:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.852+0000 7f2933f40700 1 --2- 172.21.15.70:0/2007153703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f292c102df0 0x7f292c10f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.852+0000 7f2933f40700 1 --2- 172.21.15.70:0/2007153703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f292c108510 0x7f292c1088f0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.852+0000 7f2933f40700 1 --2- 172.21.15.70:0/2007153703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f292c102430 0x7f292c1028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.852+0000 7f2933f40700 1 -- 172.21.15.70:0/2007153703 >> 172.21.15.70:0/2007153703 conn(0x7f292c0793b0 msgr2=0x7f292c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.852+0000 7f2933f40700 1 -- 172.21.15.70:0/2007153703 shutdown_connections 2024-06-04T05:13:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.852+0000 7f2933f40700 1 -- 172.21.15.70:0/2007153703 wait complete. 2024-06-04T05:13:21.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.853+0000 7f2933f40700 1 Processor -- start 2024-06-04T05:13:21.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.853+0000 7f2933f40700 1 -- start start 2024-06-04T05:13:21.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.854+0000 7f2933f40700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f292c102430 0x7f292c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.854+0000 7f2933f40700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f292c102df0 0x7f292c19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.854+0000 7f2933f40700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f292c108510 0x7f292c198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.854+0000 7f2933f40700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f292c112470 con 0x7f292c108510 2024-06-04T05:13:21.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.854+0000 7f2933f40700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f292c1122f0 con 0x7f292c102df0 2024-06-04T05:13:21.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.854+0000 7f2933f40700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f292c1125f0 con 0x7f292c102430 2024-06-04T05:13:21.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.854+0000 7f29314db700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f292c102df0 0x7f292c19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.854+0000 7f29314db700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f292c102df0 0x7f292c19ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:41296/0 (socket says 172.21.15.70:41296) 2024-06-04T05:13:21.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.854+0000 7f29314db700 1 -- 172.21.15.70:0/2541446116 learned_addr learned my addr 172.21.15.70:0/2541446116 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:21.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.854+0000 7f2931cdc700 1 --2- 172.21.15.70:0/2541446116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f292c102430 0x7f292c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.854+0000 7f29324dd700 1 --2- 172.21.15.70:0/2541446116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f292c108510 0x7f292c198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.854+0000 7f29314db700 1 -- 172.21.15.70:0/2541446116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f292c102430 msgr2=0x7f292c19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:21.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.854+0000 7f29314db700 1 --2- 172.21.15.70:0/2541446116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f292c102430 0x7f292c19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.854+0000 7f29314db700 1 -- 172.21.15.70:0/2541446116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f292c108510 msgr2=0x7f292c198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:21.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.854+0000 7f29314db700 1 --2- 172.21.15.70:0/2541446116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f292c108510 0x7f292c198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.854+0000 7f29314db700 1 -- 172.21.15.70:0/2541446116 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f292c199030 con 0x7f292c102df0 2024-06-04T05:13:21.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.855+0000 7f29314db700 1 --2- 172.21.15.70:0/2541446116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f292c102df0 0x7f292c19ec10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f291c00e9a0 tx=0x7f291c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:21.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.855+0000 7f2922ffd700 1 -- 172.21.15.70:0/2541446116 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f291c00cbd0 con 0x7f292c102df0 2024-06-04T05:13:21.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.855+0000 7f2922ffd700 1 -- 172.21.15.70:0/2541446116 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f291c00cd70 con 0x7f292c102df0 2024-06-04T05:13:21.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.855+0000 7f2922ffd700 1 -- 172.21.15.70:0/2541446116 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f291c018680 con 0x7f292c102df0 2024-06-04T05:13:21.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.855+0000 7f2933f40700 1 -- 172.21.15.70:0/2541446116 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f292c199320 con 0x7f292c102df0 2024-06-04T05:13:21.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.855+0000 7f2933f40700 1 -- 172.21.15.70:0/2541446116 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f292c06bab0 con 0x7f292c102df0 2024-06-04T05:13:21.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.859+0000 7f2922ffd700 1 -- 172.21.15.70:0/2541446116 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f291c010430 con 0x7f292c102df0 2024-06-04T05:13:21.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.860+0000 7f293e7d5b80 1 Processor -- start 2024-06-04T05:13:21.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.860+0000 7f293e7d5b80 1 -- start start 2024-06-04T05:13:21.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f293e7d5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 0x55b9ae1cde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f293e7d5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1ce420 0x55b9ae1c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f293e7d5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f293e7d5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b9ae11dd70 con 0x55b9ae1c86d0 2024-06-04T05:13:21.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f293e7d5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b9ae10d910 con 0x55b9ae1ce420 2024-06-04T05:13:21.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f293e7d5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b9ae111950 con 0x55b9ae1cda70 2024-06-04T05:13:21.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f2922ffd700 1 --2- 172.21.15.70:0/2541446116 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f29180b2a70 0x7f29180b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f29324dd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 0x55b9ae1cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f29324dd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 0x55b9ae1cde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:57856/0 (socket says 172.21.15.70:57856) 2024-06-04T05:13:21.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f29324dd700 1 -- 172.21.15.70:0/3109239857 learned_addr learned my addr 172.21.15.70:0/3109239857 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:21.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f29314db700 1 --2- 172.21.15.70:0/3109239857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f2931cdc700 1 --2- 172.21.15.70:0/3109239857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1ce420 0x55b9ae1c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f29324dd700 1 -- 172.21.15.70:0/3109239857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1ce420 msgr2=0x55b9ae1c80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:21.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f29324dd700 1 --2- 172.21.15.70:0/3109239857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1ce420 0x55b9ae1c80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f29324dd700 1 -- 172.21.15.70:0/3109239857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1c86d0 msgr2=0x55b9ae1c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:21.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f29324dd700 1 --2- 172.21.15.70:0/3109239857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1c8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f29324dd700 1 -- 172.21.15.70:0/3109239857 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9ae0b6fb0 con 0x55b9ae1cda70 2024-06-04T05:13:21.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f29324dd700 1 --2- 172.21.15.70:0/2541446116 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f29180b2a70 0x7f29180b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.861+0000 7f2922ffd700 1 -- 172.21.15.70:0/2541446116 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f291c0e2840 con 0x7f292c102df0 2024-06-04T05:13:21.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.862+0000 7f29324dd700 1 --2- 172.21.15.70:0/3109239857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 0x55b9ae1cde50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2924009710 tx=0x7f29240077e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b1ba10ff96922ad server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:21.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.862+0000 7f29324dd700 1 --2- 172.21.15.70:0/2541446116 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f29180b2a70 0x7f29180b4f30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2924020000 tx=0x7f292401d000 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:21.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.862+0000 7f2917fff700 1 -- 172.21.15.70:0/3109239857 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2924012040 con 0x55b9ae1cda70 2024-06-04T05:13:21.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.862+0000 7f2917fff700 1 -- 172.21.15.70:0/3109239857 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2924018d40 con 0x55b9ae1cda70 2024-06-04T05:13:21.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.862+0000 7f2917fff700 1 -- 172.21.15.70:0/3109239857 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29240173d0 con 0x55b9ae1cda70 2024-06-04T05:13:21.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.863+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3109239857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 msgr2=0x55b9ae1cde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:21.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.863+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/3109239857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 0x55b9ae1cde50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2924009710 tx=0x7f29240077e0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.863+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3109239857 shutdown_connections 2024-06-04T05:13:21.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.863+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/3109239857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1c8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.863+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/3109239857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 0x55b9ae1cde50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.863+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/3109239857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1ce420 0x55b9ae1c80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.863+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3109239857 >> 172.21.15.70:0/3109239857 conn(0x55b9ae0eb240 msgr2=0x55b9ae1ba3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:21.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.863+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3109239857 shutdown_connections 2024-06-04T05:13:21.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.863+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3109239857 wait complete. 2024-06-04T05:13:21.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.864+0000 7f293e7d5b80 1 -- start start 2024-06-04T05:13:21.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.864+0000 7f293e7d5b80 1 Processor -- start 2024-06-04T05:13:21.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f293e7d5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1d4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f293e7d5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 0x55b9ae1d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f293e7d5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1ce420 0x55b9ae1d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f2931cdc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1d4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f29314db700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 0x55b9ae1d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f29314db700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 0x55b9ae1d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:57860/0 (socket says 172.21.15.70:57860) 2024-06-04T05:13:21.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f29314db700 1 -- 172.21.15.70:0/4048611120 learned_addr learned my addr 172.21.15.70:0/4048611120 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:21.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f293e7d5b80 1 -- 172.21.15.70:0/4048611120 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b9ae013fa0 con 0x55b9ae1ce420 2024-06-04T05:13:21.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f293e7d5b80 1 -- 172.21.15.70:0/4048611120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b9ae075da0 con 0x55b9ae1c86d0 2024-06-04T05:13:21.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f293e7d5b80 1 -- 172.21.15.70:0/4048611120 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b9ae1d4f40 con 0x55b9ae1cda70 2024-06-04T05:13:21.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f29324dd700 1 --2- 172.21.15.70:0/4048611120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1ce420 0x55b9ae1d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f293e7d5b80 10 client.0 _ll_register_callbacks cb 0x55b9ae0a8950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:21.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f2931cdc700 1 -- 172.21.15.70:0/4048611120 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 msgr2=0x55b9ae1d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:21.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f2931cdc700 1 --2- 172.21.15.70:0/4048611120 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 0x55b9ae1d1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f2931cdc700 1 -- 172.21.15.70:0/4048611120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1ce420 msgr2=0x55b9ae1d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:21.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f2931cdc700 1 --2- 172.21.15.70:0/4048611120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1ce420 0x55b9ae1d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f2931cdc700 1 -- 172.21.15.70:0/4048611120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9ae05d8d0 con 0x55b9ae1c86d0 2024-06-04T05:13:21.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.865+0000 7f29324dd700 1 --2- 172.21.15.70:0/4048611120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1ce420 0x55b9ae1d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:21.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.866+0000 7f2931cdc700 1 --2- 172.21.15.70:0/4048611120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1d4220 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f292800b650 tx=0x7f2928005800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff5474ba06290574 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:21.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.866+0000 7f29157fa700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:13:21.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.866+0000 7f29157fa700 1 -- 172.21.15.70:0/4048611120 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2928029060 con 0x55b9ae1c86d0 2024-06-04T05:13:21.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.866+0000 7f29157fa700 1 -- 172.21.15.70:0/4048611120 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f292800b6b0 con 0x55b9ae1c86d0 2024-06-04T05:13:21.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.866+0000 7f293e7d5b80 1 -- 172.21.15.70:0/4048611120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2924018d40 con 0x55b9ae1c86d0 2024-06-04T05:13:21.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.867+0000 7f29157fa700 1 -- 172.21.15.70:0/4048611120 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f29280095c0 con 0x55b9ae1c86d0 2024-06-04T05:13:21.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.867+0000 7f293e7d5b80 10 client.28520 fetch_fsmap learned FSMap version 46 2024-06-04T05:13:21.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.867+0000 7f293e7d5b80 1 -- 172.21.15.70:0/4048611120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55b9ae0c7760 con 0x55b9ae1c86d0 2024-06-04T05:13:21.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.867+0000 7f29157fa700 1 -- 172.21.15.70:0/4048611120 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2924018d40 con 0x55b9ae1c86d0 2024-06-04T05:13:21.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.867+0000 7f29157fa700 1 -- 172.21.15.70:0/4048611120 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 46) ==== 8402+0+0 (secure 0 0 0) 0x7f292800f750 con 0x55b9ae1c86d0 2024-06-04T05:13:21.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.867+0000 7f293e7d5b80 10 client.28520 fetch_fsmap finished waiting for FSMap version 46 2024-06-04T05:13:21.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.867+0000 7f293e7d5b80 10 client.28520 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:13:21.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.867+0000 7f293e7d5b80 20 client.28520 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.867+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/4048611120 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55b9ae1e1970 0x55b9ae1e3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.867+0000 7f293e7d5b80 4 client.28520 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:13:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.867+0000 7f293e7d5b80 1 -- 172.21.15.70:0/4048611120 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9ae11fcc0 con 0x55b9ae1e1970 2024-06-04T05:13:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.867+0000 7f29314db700 1 --2- 172.21.15.70:0/4048611120 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55b9ae1e1970 0x55b9ae1e3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.868+0000 7f29314db700 1 --2- 172.21.15.70:0/4048611120 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55b9ae1e1970 0x55b9ae1e3d60 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d51000b146fe72d9 server_cookie=4177b5262e4e18bc in_seq=0 out_seq=0 2024-06-04T05:13:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.868+0000 7f29157fa700 10 client.28520 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:13:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.869+0000 7f29157fa700 1 -- 172.21.15.70:0/4048611120 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b9ae11fcc0 con 0x55b9ae1e1970 2024-06-04T05:13:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.869+0000 7f29157fa700 10 client.28520 handle_command_reply: tid=0 2024-06-04T05:13:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.869+0000 7f293e7d5b80 1 client.28520 shutdown 2024-06-04T05:13:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.869+0000 7f293e7d5b80 1 -- 172.21.15.70:0/4048611120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1c86d0 msgr2=0x55b9ae1d4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.869+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/4048611120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1d4220 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f292800b650 tx=0x7f2928005800 comp rx=0 tx=0).stop 2024-06-04T05:13:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.869+0000 7f293e7d5b80 1 -- 172.21.15.70:0/4048611120 shutdown_connections 2024-06-04T05:13:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.869+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/4048611120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1ce420 0x55b9ae1d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.869+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/4048611120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1d4220 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.869+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/4048611120 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55b9ae1e1970 0x55b9ae1e3d60 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.869+0000 7f29157fa700 0 client.28520 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:13:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.869+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/4048611120 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 0x55b9ae1d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.869+0000 7f293e7d5b80 1 -- 172.21.15.70:0/4048611120 >> 172.21.15.70:0/4048611120 conn(0x55b9ae0eb240 msgr2=0x55b9ae1b9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.870+0000 7f293e7d5b80 1 -- 172.21.15.70:0/4048611120 shutdown_connections 2024-06-04T05:13:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.870+0000 7f293e7d5b80 1 -- 172.21.15.70:0/4048611120 wait complete. 2024-06-04T05:13:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.870+0000 7f293e7d5b80 20 client.28520 trim_cache size 0 max 16384 2024-06-04T05:13:21.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.895+0000 7f293e7d5b80 1 Processor -- start 2024-06-04T05:13:21.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.895+0000 7f293e7d5b80 1 -- start start 2024-06-04T05:13:21.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.895+0000 7f293e7d5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1ddfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.895+0000 7f293e7d5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 0x55b9ae1cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.895+0000 7f293e7d5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1ce420 0x55b9ae1ba270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.895+0000 7f2931cdc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1ddfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.895+0000 7f2931cdc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1ddfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49248/0 (socket says 172.21.15.70:49248) 2024-06-04T05:13:21.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.895+0000 7f2931cdc700 1 -- 172.21.15.70:0/3083602910 learned_addr learned my addr 172.21.15.70:0/3083602910 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:21.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.895+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3083602910 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f292c1abcb0 con 0x55b9ae1c86d0 2024-06-04T05:13:21.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.895+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3083602910 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f292c1ad970 con 0x55b9ae1ce420 2024-06-04T05:13:21.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.895+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3083602910 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f292c1ac3b0 con 0x55b9ae1cda70 2024-06-04T05:13:21.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.895+0000 7f29314db700 1 --2- 172.21.15.70:0/3083602910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 0x55b9ae1cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.895+0000 7f29324dd700 1 --2- 172.21.15.70:0/3083602910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1ce420 0x55b9ae1ba270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.896+0000 7f2931cdc700 1 -- 172.21.15.70:0/3083602910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 msgr2=0x55b9ae1cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:21.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.896+0000 7f2931cdc700 1 --2- 172.21.15.70:0/3083602910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 0x55b9ae1cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.896+0000 7f2931cdc700 1 -- 172.21.15.70:0/3083602910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1ce420 msgr2=0x55b9ae1ba270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:21.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.896+0000 7f2931cdc700 1 --2- 172.21.15.70:0/3083602910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1ce420 0x55b9ae1ba270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.896+0000 7f2931cdc700 1 -- 172.21.15.70:0/3083602910 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9ae1e43c0 con 0x55b9ae1c86d0 2024-06-04T05:13:21.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.896+0000 7f2931cdc700 1 --2- 172.21.15.70:0/3083602910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1ddfe0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f2928009990 tx=0x7f2928010fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aafc35f47c8672c1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:21.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.896+0000 7f29177fe700 1 -- 172.21.15.70:0/3083602910 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292803a070 con 0x55b9ae1c86d0 2024-06-04T05:13:21.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.896+0000 7f29177fe700 1 -- 172.21.15.70:0/3083602910 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f292800f660 con 0x55b9ae1c86d0 2024-06-04T05:13:21.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.897+0000 7f29177fe700 1 -- 172.21.15.70:0/3083602910 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2928029070 con 0x55b9ae1c86d0 2024-06-04T05:13:21.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.897+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3083602910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1c86d0 msgr2=0x55b9ae1ddfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:21.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.897+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/3083602910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1ddfe0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f2928009990 tx=0x7f2928010fb0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.897+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3083602910 shutdown_connections 2024-06-04T05:13:21.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.897+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/3083602910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1ce420 0x55b9ae1ba270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.898+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/3083602910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1ddfe0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.898+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/3083602910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1cda70 0x55b9ae1cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.898+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3083602910 >> 172.21.15.70:0/3083602910 conn(0x55b9ae0eb240 msgr2=0x55b9ae1bf120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:21.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.898+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3083602910 shutdown_connections 2024-06-04T05:13:21.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.898+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3083602910 wait complete. 2024-06-04T05:13:21.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.898+0000 7f293e7d5b80 1 -- start start 2024-06-04T05:13:21.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.898+0000 7f293e7d5b80 1 Processor -- start 2024-06-04T05:13:21.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.899+0000 7f293e7d5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1da1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.899+0000 7f293e7d5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1cda70 0x55b9ae1da710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.899+0000 7f293e7d5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1ce420 0x55b9ae1d71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.899+0000 7f2931cdc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1da1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.900+0000 7f293e7d5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f292c1ad1e0 con 0x55b9ae1ce420 2024-06-04T05:13:21.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.900+0000 7f293e7d5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f292c1acae0 con 0x55b9ae1cda70 2024-06-04T05:13:21.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.900+0000 7f293e7d5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f292c1a9c70 con 0x55b9ae1c86d0 2024-06-04T05:13:21.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.900+0000 7f2931cdc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1da1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:57868/0 (socket says 172.21.15.70:57868) 2024-06-04T05:13:21.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.900+0000 7f2931cdc700 1 -- 172.21.15.70:0/3233588386 learned_addr learned my addr 172.21.15.70:0/3233588386 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:21.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.900+0000 7f293e7d5b80 10 client.0 _ll_register_callbacks cb 0x55b9ae0a8950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:21.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.900+0000 7f29314db700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1cda70 0x55b9ae1da710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.900+0000 7f29324dd700 1 --2- 172.21.15.70:0/3233588386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1ce420 0x55b9ae1d71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.900+0000 7f2931cdc700 1 -- 172.21.15.70:0/3233588386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1cda70 msgr2=0x55b9ae1da710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:21.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.900+0000 7f2931cdc700 1 --2- 172.21.15.70:0/3233588386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1cda70 0x55b9ae1da710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.900+0000 7f2931cdc700 1 -- 172.21.15.70:0/3233588386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1ce420 msgr2=0x55b9ae1d71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:21.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.900+0000 7f2931cdc700 1 --2- 172.21.15.70:0/3233588386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1ce420 0x55b9ae1d71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.900+0000 7f2931cdc700 1 -- 172.21.15.70:0/3233588386 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9ae1d7790 con 0x55b9ae1c86d0 2024-06-04T05:13:21.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.900+0000 7f29324dd700 1 --2- 172.21.15.70:0/3233588386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1ce420 0x55b9ae1d71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:13:21.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.900+0000 7f2931cdc700 1 --2- 172.21.15.70:0/3233588386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1da1d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2928006010 tx=0x7f292800bde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72431b025e39d3ac server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:21.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.900+0000 7f29157fa700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:13:21.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.901+0000 7f29157fa700 1 -- 172.21.15.70:0/3233588386 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2928044060 con 0x55b9ae1c86d0 2024-06-04T05:13:21.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.901+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3233588386 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f292800f660 con 0x55b9ae1c86d0 2024-06-04T05:13:21.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.901+0000 7f29157fa700 1 -- 172.21.15.70:0/3233588386 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f292800f810 con 0x55b9ae1c86d0 2024-06-04T05:13:21.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.901+0000 7f29157fa700 1 -- 172.21.15.70:0/3233588386 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f2928029070 con 0x55b9ae1c86d0 2024-06-04T05:13:21.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.901+0000 7f293e7d5b80 10 client.28636 fetch_fsmap learned FSMap version 46 2024-06-04T05:13:21.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.901+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3233588386 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55b9ae24cdb0 con 0x55b9ae1c86d0 2024-06-04T05:13:21.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.901+0000 7f29157fa700 1 -- 172.21.15.70:0/3233588386 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f292800f660 con 0x55b9ae1c86d0 2024-06-04T05:13:21.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.901+0000 7f29157fa700 1 -- 172.21.15.70:0/3233588386 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 46) ==== 8402+0+0 (secure 0 0 0) 0x7f2928024480 con 0x55b9ae1c86d0 2024-06-04T05:13:21.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.901+0000 7f293e7d5b80 10 client.28636 fetch_fsmap finished waiting for FSMap version 46 2024-06-04T05:13:21.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.901+0000 7f293e7d5b80 10 client.28636 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:13:21.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.901+0000 7f293e7d5b80 20 client.28636 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:21.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.901+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/3233588386 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55b9ae24d370 0x55b9ae24f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:21.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.901+0000 7f293e7d5b80 4 client.28636 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:13:21.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.901+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3233588386 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b9ae1194a0 con 0x55b9ae24d370 2024-06-04T05:13:21.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.902+0000 7f29314db700 1 --2- 172.21.15.70:0/3233588386 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55b9ae24d370 0x55b9ae24f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:21.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.902+0000 7f29314db700 1 --2- 172.21.15.70:0/3233588386 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55b9ae24d370 0x55b9ae24f790 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d1d1855bddd2084e server_cookie=80d29c3802d41fdf in_seq=0 out_seq=0 2024-06-04T05:13:21.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.902+0000 7f29157fa700 10 client.28636 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:13:21.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.902+0000 7f29157fa700 1 -- 172.21.15.70:0/3233588386 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b9ae1194a0 con 0x55b9ae24d370 2024-06-04T05:13:21.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.902+0000 7f29157fa700 10 client.28636 handle_command_reply: tid=0 2024-06-04T05:13:21.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.902+0000 7f293e7d5b80 1 client.28636 shutdown 2024-06-04T05:13:21.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.903+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3233588386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1c86d0 msgr2=0x55b9ae1da1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:21.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.903+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/3233588386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1da1d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2928006010 tx=0x7f292800bde0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.903+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3233588386 shutdown_connections 2024-06-04T05:13:21.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.903+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/3233588386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ae1ce420 0x55b9ae1d71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.903+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/3233588386 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55b9ae24d370 0x55b9ae24f790 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.903+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/3233588386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ae1c86d0 0x55b9ae1da1d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.903+0000 7f293e7d5b80 1 --2- 172.21.15.70:0/3233588386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ae1cda70 0x55b9ae1da710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.903+0000 7f29157fa700 0 client.28636 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:13:21.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.903+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3233588386 >> 172.21.15.70:0/3233588386 conn(0x55b9ae0eb240 msgr2=0x55b9ae1bf120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:21.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.903+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3233588386 shutdown_connections 2024-06-04T05:13:21.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.903+0000 7f293e7d5b80 1 -- 172.21.15.70:0/3233588386 wait complete. 2024-06-04T05:13:21.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.903+0000 7f293e7d5b80 20 client.28636 trim_cache size 0 max 16384 2024-06-04T05:13:21.910 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:13:21.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.904+0000 7f2933f40700 1 -- 172.21.15.70:0/2541446116 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f29180b2a70 msgr2=0x7f29180b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:21.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.904+0000 7f2933f40700 1 --2- 172.21.15.70:0/2541446116 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f29180b2a70 0x7f29180b4f30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2924020000 tx=0x7f292401d000 comp rx=0 tx=0).stop 2024-06-04T05:13:21.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.904+0000 7f2933f40700 1 -- 172.21.15.70:0/2541446116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f292c102df0 msgr2=0x7f292c19ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:21.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.904+0000 7f2933f40700 1 --2- 172.21.15.70:0/2541446116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f292c102df0 0x7f292c19ec10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f291c00e9a0 tx=0x7f291c00ecb0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.904+0000 7f2933f40700 1 -- 172.21.15.70:0/2541446116 shutdown_connections 2024-06-04T05:13:21.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.904+0000 7f2933f40700 1 --2- 172.21.15.70:0/2541446116 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f29180b2a70 0x7f29180b4f30 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.904+0000 7f2933f40700 1 --2- 172.21.15.70:0/2541446116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f292c108510 0x7f292c198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.904+0000 7f2933f40700 1 --2- 172.21.15.70:0/2541446116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f292c102430 0x7f292c19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.904+0000 7f2933f40700 1 --2- 172.21.15.70:0/2541446116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f292c102df0 0x7f292c19ec10 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:21.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.904+0000 7f2933f40700 1 -- 172.21.15.70:0/2541446116 >> 172.21.15.70:0/2541446116 conn(0x7f292c0793b0 msgr2=0x7f292c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:21.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.904+0000 7f2933f40700 1 -- 172.21.15.70:0/2541446116 shutdown_connections 2024-06-04T05:13:21.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:21.904+0000 7f2933f40700 1 -- 172.21.15.70:0/2541446116 wait complete. 2024-06-04T05:13:22.653 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:13:22.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 18973 8.88 MB/sec execute 14 sec latency 54.974 ms 2024-06-04T05:13:22.998 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:13:23.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:23 smithi070 ceph-mon[32631]: pgmap v507: 97 pgs: 97 active+clean; 460 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 414 op/s 2024-06-04T05:13:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:23 smithi112 ceph-mon[36453]: pgmap v507: 97 pgs: 97 active+clean; 460 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 414 op/s 2024-06-04T05:13:23.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:23 smithi098 ceph-mon[35569]: pgmap v507: 97 pgs: 97 active+clean; 460 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 414 op/s 2024-06-04T05:13:23.802 INFO:tasks.workunit.client.0.smithi070.stdout: 10 19173 9.13 MB/sec execute 15 sec latency 55.117 ms 2024-06-04T05:13:24.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 19366 9.03 MB/sec execute 16 sec latency 95.478 ms 2024-06-04T05:13:25.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.168+0000 7f65be254700 1 -- 172.21.15.70:0/3400168947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65b806b220 msgr2=0x7f65b806b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:25.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.168+0000 7f65be254700 1 --2- 172.21.15.70:0/3400168947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65b806b220 0x7f65b806b6a0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f65ac009990 tx=0x7f65ac009ca0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.168+0000 7f65be254700 1 -- 172.21.15.70:0/3400168947 shutdown_connections 2024-06-04T05:13:25.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.168+0000 7f65be254700 1 --2- 172.21.15.70:0/3400168947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65b806bbe0 0x7f65b81103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.168+0000 7f65be254700 1 --2- 172.21.15.70:0/3400168947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65b81095b0 0x7f65b8109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.168+0000 7f65be254700 1 --2- 172.21.15.70:0/3400168947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65b806b220 0x7f65b806b6a0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.168+0000 7f65be254700 1 -- 172.21.15.70:0/3400168947 >> 172.21.15.70:0/3400168947 conn(0x7f65b80793f0 msgr2=0x7f65b8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:25.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.169+0000 7f65be254700 1 -- 172.21.15.70:0/3400168947 shutdown_connections 2024-06-04T05:13:25.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.169+0000 7f65be254700 1 -- 172.21.15.70:0/3400168947 wait complete. 2024-06-04T05:13:25.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.169+0000 7f65be254700 1 Processor -- start 2024-06-04T05:13:25.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65be254700 1 -- start start 2024-06-04T05:13:25.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65be254700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65b806b220 0x7f65b8102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65be254700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65b806bbe0 0x7f65b8102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65be254700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65b81095b0 0x7f65b8102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65be254700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65b8113570 con 0x7f65b806b220 2024-06-04T05:13:25.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65be254700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f65b81133f0 con 0x7f65b81095b0 2024-06-04T05:13:25.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65be254700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f65b81136f0 con 0x7f65b806bbe0 2024-06-04T05:13:25.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65b77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65b806b220 0x7f65b8102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65b6ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65b806bbe0 0x7f65b8102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65b6ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65b806bbe0 0x7f65b8102a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:57882/0 (socket says 172.21.15.70:57882) 2024-06-04T05:13:25.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65b6ffd700 1 -- 172.21.15.70:0/2210904676 learned_addr learned my addr 172.21.15.70:0/2210904676 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:25.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65b7fff700 1 --2- 172.21.15.70:0/2210904676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65b81095b0 0x7f65b8102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65b6ffd700 1 -- 172.21.15.70:0/2210904676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65b81095b0 msgr2=0x7f65b8102f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:25.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65b6ffd700 1 --2- 172.21.15.70:0/2210904676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65b81095b0 0x7f65b8102f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65b6ffd700 1 -- 172.21.15.70:0/2210904676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65b806b220 msgr2=0x7f65b8102510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:25.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65b6ffd700 1 --2- 172.21.15.70:0/2210904676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65b806b220 0x7f65b8102510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65b6ffd700 1 -- 172.21.15.70:0/2210904676 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b81a8070 con 0x7f65b806bbe0 2024-06-04T05:13:25.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.170+0000 7f65b7fff700 1 --2- 172.21.15.70:0/2210904676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65b81095b0 0x7f65b8102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:25.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.171+0000 7f65b6ffd700 1 --2- 172.21.15.70:0/2210904676 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65b806bbe0 0x7f65b8102a50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f65ac006010 tx=0x7f65ac0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:25.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.171+0000 7f65b4ff9700 1 -- 172.21.15.70:0/2210904676 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac027070 con 0x7f65b806bbe0 2024-06-04T05:13:25.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.171+0000 7f65b4ff9700 1 -- 172.21.15.70:0/2210904676 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65ac022470 con 0x7f65b806bbe0 2024-06-04T05:13:25.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.171+0000 7f65b4ff9700 1 -- 172.21.15.70:0/2210904676 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac017650 con 0x7f65b806bbe0 2024-06-04T05:13:25.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.171+0000 7f65be254700 1 -- 172.21.15.70:0/2210904676 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65b81a8300 con 0x7f65b806bbe0 2024-06-04T05:13:25.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.171+0000 7f65be254700 1 -- 172.21.15.70:0/2210904676 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65b81a86a0 con 0x7f65b806bbe0 2024-06-04T05:13:25.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.175+0000 7f65b4ff9700 1 -- 172.21.15.70:0/2210904676 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f65ac009d10 con 0x7f65b806bbe0 2024-06-04T05:13:25.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.176+0000 7f65c8ae9b80 1 Processor -- start 2024-06-04T05:13:25.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.176+0000 7f65c8ae9b80 1 -- start start 2024-06-04T05:13:25.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.176+0000 7f65b4ff9700 1 --2- 172.21.15.70:0/2210904676 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65a00b2a70 0x7f65a00b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.176+0000 7f65c8ae9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 0x55bdba932c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.176+0000 7f65b7fff700 1 --2- 172.21.15.70:0/2210904676 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65a00b2a70 0x7f65a00b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.176+0000 7f65b4ff9700 1 -- 172.21.15.70:0/2210904676 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f65ac0e30a0 con 0x7f65b806bbe0 2024-06-04T05:13:25.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.176+0000 7f65b77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 0x55bdba932c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.176+0000 7f65b77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 0x55bdba932c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49294/0 (socket says 172.21.15.70:49294) 2024-06-04T05:13:25.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.176+0000 7f65b77fe700 1 -- 172.21.15.70:0/2786191761 learned_addr learned my addr 172.21.15.70:0/2786191761 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:25.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.176+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/2786191761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba933210 0x55bdba936940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.176+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/2786191761 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba936f60 0x55bdba9373a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.176+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2786191761 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bdba891c90 con 0x55bdba932860 2024-06-04T05:13:25.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.176+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2786191761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bdba881830 con 0x55bdba933210 2024-06-04T05:13:25.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.176+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2786191761 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bdba885870 con 0x55bdba936f60 2024-06-04T05:13:25.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.176+0000 7f65b6ffd700 1 --2- 172.21.15.70:0/2786191761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba933210 0x55bdba936940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.176+0000 7f65b7fff700 1 --2- 172.21.15.70:0/2210904676 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65a00b2a70 0x7f65a00b4f30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f65b0003e30 tx=0x7f65b000b040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:25.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.177+0000 7f65b7fff700 1 --2- 172.21.15.70:0/2786191761 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba936f60 0x55bdba9373a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.177+0000 7f65b6ffd700 1 -- 172.21.15.70:0/2786191761 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba936f60 msgr2=0x55bdba9373a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:25.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.177+0000 7f65b6ffd700 1 --2- 172.21.15.70:0/2786191761 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba936f60 0x55bdba9373a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.177+0000 7f65b6ffd700 1 -- 172.21.15.70:0/2786191761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 msgr2=0x55bdba932c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:25.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.177+0000 7f65b6ffd700 1 --2- 172.21.15.70:0/2786191761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 0x55bdba932c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.177+0000 7f65b6ffd700 1 -- 172.21.15.70:0/2786191761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdba7d18d0 con 0x55bdba933210 2024-06-04T05:13:25.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.177+0000 7f65b77fe700 1 --2- 172.21.15.70:0/2786191761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 0x55bdba932c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:25.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.177+0000 7f65b6ffd700 1 --2- 172.21.15.70:0/2786191761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba933210 0x55bdba936940 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f65ac0e0760 tx=0x7f65ac0a9de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b160eb69de84356 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:25.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.177+0000 7f659dffb700 1 -- 172.21.15.70:0/2786191761 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac0ae730 con 0x55bdba933210 2024-06-04T05:13:25.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.177+0000 7f659dffb700 1 -- 172.21.15.70:0/2786191761 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65ac0b3150 con 0x55bdba933210 2024-06-04T05:13:25.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.177+0000 7f659dffb700 1 -- 172.21.15.70:0/2786191761 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac0aa870 con 0x55bdba933210 2024-06-04T05:13:25.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.178+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2786191761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba933210 msgr2=0x55bdba936940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:25.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.178+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/2786191761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba933210 0x55bdba936940 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f65ac0e0760 tx=0x7f65ac0a9de0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.178+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2786191761 shutdown_connections 2024-06-04T05:13:25.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.178+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/2786191761 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba936f60 0x55bdba9373a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.178+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/2786191761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 0x55bdba932c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.178+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/2786191761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba933210 0x55bdba936940 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.178+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2786191761 >> 172.21.15.70:0/2786191761 conn(0x55bdba860080 msgr2=0x55bdba92ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:25.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.178+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2786191761 shutdown_connections 2024-06-04T05:13:25.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.178+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2786191761 wait complete. 2024-06-04T05:13:25.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.179+0000 7f65c8ae9b80 1 -- start start 2024-06-04T05:13:25.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.179+0000 7f65c8ae9b80 1 Processor -- start 2024-06-04T05:13:25.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.179+0000 7f65c8ae9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 0x55bdba93e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.179+0000 7f65c8ae9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba933210 0x55bdba93b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.179+0000 7f65c8ae9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba936f60 0x55bdba93bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.180+0000 7f65c8ae9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bdba787fa0 con 0x55bdba932860 2024-06-04T05:13:25.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.180+0000 7f65c8ae9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bdba8237c0 con 0x55bdba936f60 2024-06-04T05:13:25.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.180+0000 7f65b7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba936f60 0x55bdba93bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.180+0000 7f65b7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba936f60 0x55bdba93bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:41372/0 (socket says 172.21.15.70:41372) 2024-06-04T05:13:25.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.180+0000 7f65b7fff700 1 -- 172.21.15.70:0/2577599173 learned_addr learned my addr 172.21.15.70:0/2577599173 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:25.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.180+0000 7f65b6ffd700 1 --2- 172.21.15.70:0/2577599173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba933210 0x55bdba93b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.180+0000 7f65c8ae9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bdba93eee0 con 0x55bdba933210 2024-06-04T05:13:25.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.180+0000 7f65c8ae9b80 10 client.0 _ll_register_callbacks cb 0x55bdba81c920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:25.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.180+0000 7f65b77fe700 1 --2- 172.21.15.70:0/2577599173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 0x55bdba93e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.180+0000 7f65b7fff700 1 -- 172.21.15.70:0/2577599173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba933210 msgr2=0x55bdba93b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:25.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.180+0000 7f65b7fff700 1 --2- 172.21.15.70:0/2577599173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba933210 0x55bdba93b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.180+0000 7f65b7fff700 1 -- 172.21.15.70:0/2577599173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 msgr2=0x55bdba93e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:25.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.180+0000 7f65b7fff700 1 --2- 172.21.15.70:0/2577599173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 0x55bdba93e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.180+0000 7f65b7fff700 1 -- 172.21.15.70:0/2577599173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdba820690 con 0x55bdba936f60 2024-06-04T05:13:25.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.180+0000 7f65b77fe700 1 --2- 172.21.15.70:0/2577599173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 0x55bdba93e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:13:25.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.180+0000 7f65b7fff700 1 --2- 172.21.15.70:0/2577599173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba936f60 0x55bdba93bd30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f65b000eef0 tx=0x7f65b0004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e38d1a4afb4174a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:25.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.180+0000 7f6596ffd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:13:25.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.181+0000 7f6596ffd700 1 -- 172.21.15.70:0/2577599173 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b0027030 con 0x55bdba936f60 2024-06-04T05:13:25.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.181+0000 7f6596ffd700 1 -- 172.21.15.70:0/2577599173 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f65b0009020 con 0x55bdba936f60 2024-06-04T05:13:25.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.181+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2577599173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65ac0b3150 con 0x55bdba936f60 2024-06-04T05:13:25.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.181+0000 7f6596ffd700 1 -- 172.21.15.70:0/2577599173 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f65b0033050 con 0x55bdba936f60 2024-06-04T05:13:25.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.181+0000 7f65c8ae9b80 10 client.28538 fetch_fsmap learned FSMap version 46 2024-06-04T05:13:25.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.181+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2577599173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55bdba949690 con 0x55bdba936f60 2024-06-04T05:13:25.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.181+0000 7f6596ffd700 1 -- 172.21.15.70:0/2577599173 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65ac0b3150 con 0x55bdba936f60 2024-06-04T05:13:25.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.182+0000 7f6596ffd700 1 -- 172.21.15.70:0/2577599173 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 46) ==== 8402+0+0 (secure 0 0 0) 0x7f65b0020050 con 0x55bdba936f60 2024-06-04T05:13:25.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.182+0000 7f65c8ae9b80 10 client.28538 fetch_fsmap finished waiting for FSMap version 46 2024-06-04T05:13:25.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.182+0000 7f65c8ae9b80 10 client.28538 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:13:25.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.182+0000 7f65c8ae9b80 20 client.28538 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:25.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.182+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/2577599173 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55bdba94bc20 0x55bdba94e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.182+0000 7f65c8ae9b80 4 client.28538 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:13:25.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.182+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2577599173 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdba8fe9e0 con 0x55bdba94bc20 2024-06-04T05:13:25.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.182+0000 7f65b77fe700 1 --2- 172.21.15.70:0/2577599173 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55bdba94bc20 0x55bdba94e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.183+0000 7f65b77fe700 1 --2- 172.21.15.70:0/2577599173 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55bdba94bc20 0x55bdba94e0a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9acac3040069d57a server_cookie=5b2e268cf912afe2 in_seq=0 out_seq=0 2024-06-04T05:13:25.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.183+0000 7f6596ffd700 10 client.28538 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:13:25.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.183+0000 7f6596ffd700 1 -- 172.21.15.70:0/2577599173 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bdba8fe9e0 con 0x55bdba94bc20 2024-06-04T05:13:25.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.183+0000 7f6596ffd700 10 client.28538 handle_command_reply: tid=0 2024-06-04T05:13:25.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.184+0000 7f65c8ae9b80 1 client.28538 shutdown 2024-06-04T05:13:25.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.184+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2577599173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba936f60 msgr2=0x55bdba93bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:25.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.184+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/2577599173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba936f60 0x55bdba93bd30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f65b000eef0 tx=0x7f65b0004ad0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.184+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2577599173 shutdown_connections 2024-06-04T05:13:25.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.184+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/2577599173 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55bdba94bc20 0x55bdba94e0a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.184+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/2577599173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba936f60 0x55bdba93bd30 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.184+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/2577599173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 0x55bdba93e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.184+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/2577599173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba933210 0x55bdba93b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.184+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2577599173 >> 172.21.15.70:0/2577599173 conn(0x55bdba860080 msgr2=0x55bdba938350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:25.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.184+0000 7f6596ffd700 0 client.28538 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:13:25.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.184+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2577599173 shutdown_connections 2024-06-04T05:13:25.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.184+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2577599173 wait complete. 2024-06-04T05:13:25.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.184+0000 7f65c8ae9b80 20 client.28538 trim_cache size 0 max 16384 2024-06-04T05:13:25.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.207+0000 7f65c8ae9b80 1 Processor -- start 2024-06-04T05:13:25.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.207+0000 7f65c8ae9b80 1 -- start start 2024-06-04T05:13:25.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.207+0000 7f65c8ae9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 0x55bdba94e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.207+0000 7f65c8ae9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba933210 0x55bdba947f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.207+0000 7f65c8ae9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba936f60 0x55bdba9539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.208+0000 7f65c8ae9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65b81affa0 con 0x55bdba932860 2024-06-04T05:13:25.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.208+0000 7f65c8ae9b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f65b81b1c60 con 0x55bdba933210 2024-06-04T05:13:25.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.208+0000 7f65c8ae9b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f65b81b06a0 con 0x55bdba936f60 2024-06-04T05:13:25.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.208+0000 7f65b77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 0x55bdba94e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.208+0000 7f65b77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 0x55bdba94e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49316/0 (socket says 172.21.15.70:49316) 2024-06-04T05:13:25.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.208+0000 7f65b77fe700 1 -- 172.21.15.70:0/2824110519 learned_addr learned my addr 172.21.15.70:0/2824110519 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:25.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.208+0000 7f65b6ffd700 1 --2- 172.21.15.70:0/2824110519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba933210 0x55bdba947f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.208+0000 7f65b7fff700 1 --2- 172.21.15.70:0/2824110519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba936f60 0x55bdba9539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.208+0000 7f65b77fe700 1 -- 172.21.15.70:0/2824110519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba936f60 msgr2=0x55bdba9539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:25.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.208+0000 7f65b77fe700 1 --2- 172.21.15.70:0/2824110519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba936f60 0x55bdba9539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.208+0000 7f65b77fe700 1 -- 172.21.15.70:0/2824110519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba933210 msgr2=0x55bdba947f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:25.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.208+0000 7f65b77fe700 1 --2- 172.21.15.70:0/2824110519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba933210 0x55bdba947f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.208+0000 7f65b77fe700 1 -- 172.21.15.70:0/2824110519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdba93c040 con 0x55bdba932860 2024-06-04T05:13:25.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.208+0000 7f65b6ffd700 1 --2- 172.21.15.70:0/2824110519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba933210 0x55bdba947f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:25.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.208+0000 7f65b77fe700 1 --2- 172.21.15.70:0/2824110519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 0x55bdba94e090 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f65a8005c20 tx=0x7f65a80045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59d37ace2efa8dee server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:25.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.208+0000 7f659d7fa700 1 -- 172.21.15.70:0/2824110519 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8003cf0 con 0x55bdba932860 2024-06-04T05:13:25.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.208+0000 7f659d7fa700 1 -- 172.21.15.70:0/2824110519 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65a8004cc0 con 0x55bdba932860 2024-06-04T05:13:25.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.209+0000 7f659d7fa700 1 -- 172.21.15.70:0/2824110519 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a80056e0 con 0x55bdba932860 2024-06-04T05:13:25.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.209+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2824110519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 msgr2=0x55bdba94e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:25.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.209+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/2824110519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 0x55bdba94e090 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f65a8005c20 tx=0x7f65a80045b0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.209+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2824110519 shutdown_connections 2024-06-04T05:13:25.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.209+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/2824110519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba936f60 0x55bdba9539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.209+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/2824110519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba932860 0x55bdba94e090 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.209+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/2824110519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba933210 0x55bdba947f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.209+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2824110519 >> 172.21.15.70:0/2824110519 conn(0x55bdba860080 msgr2=0x55bdba93a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:25.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.210+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2824110519 shutdown_connections 2024-06-04T05:13:25.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.210+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/2824110519 wait complete. 2024-06-04T05:13:25.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.210+0000 7f65c8ae9b80 1 -- start start 2024-06-04T05:13:25.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.210+0000 7f65c8ae9b80 1 Processor -- start 2024-06-04T05:13:25.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65c8ae9b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba932860 0x55bdba9440e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65c8ae9b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba933210 0x55bdba944620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65c8ae9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba936f60 0x55bdba9416e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65b7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba936f60 0x55bdba9416e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65b7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba936f60 0x55bdba9416e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49330/0 (socket says 172.21.15.70:49330) 2024-06-04T05:13:25.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65b7fff700 1 -- 172.21.15.70:0/144577802 learned_addr learned my addr 172.21.15.70:0/144577802 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:25.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/144577802 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65b81b14d0 con 0x55bdba936f60 2024-06-04T05:13:25.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/144577802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f65b81b0dd0 con 0x55bdba933210 2024-06-04T05:13:25.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/144577802 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f65b81adf60 con 0x55bdba932860 2024-06-04T05:13:25.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65b6ffd700 1 --2- 172.21.15.70:0/144577802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba933210 0x55bdba944620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65b77fe700 1 --2- 172.21.15.70:0/144577802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba932860 0x55bdba9440e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65c8ae9b80 10 client.0 _ll_register_callbacks cb 0x55bdba81c920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:25.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65b7fff700 1 -- 172.21.15.70:0/144577802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba932860 msgr2=0x55bdba9440e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:25.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65b7fff700 1 --2- 172.21.15.70:0/144577802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba932860 0x55bdba9440e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65b7fff700 1 -- 172.21.15.70:0/144577802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba933210 msgr2=0x55bdba944620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:25.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65b7fff700 1 --2- 172.21.15.70:0/144577802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba933210 0x55bdba944620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65b7fff700 1 -- 172.21.15.70:0/144577802 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdba941e10 con 0x55bdba936f60 2024-06-04T05:13:25.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.211+0000 7f65b77fe700 1 --2- 172.21.15.70:0/144577802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba932860 0x55bdba9440e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:13:25.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.212+0000 7f65b7fff700 1 --2- 172.21.15.70:0/144577802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba936f60 0x55bdba9416e0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f65b0005b90 tx=0x7f65b0017ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea45b237627f108 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:25.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.212+0000 7f6596ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:13:25.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.212+0000 7f6596ffd700 1 -- 172.21.15.70:0/144577802 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b0027070 con 0x55bdba936f60 2024-06-04T05:13:25.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.212+0000 7f6596ffd700 1 -- 172.21.15.70:0/144577802 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f65b0004b40 con 0x55bdba936f60 2024-06-04T05:13:25.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.212+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/144577802 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65a8004cc0 con 0x55bdba936f60 2024-06-04T05:13:25.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.212+0000 7f6596ffd700 1 -- 172.21.15.70:0/144577802 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f65b00093b0 con 0x55bdba936f60 2024-06-04T05:13:25.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.212+0000 7f65c8ae9b80 10 client.19701 fetch_fsmap learned FSMap version 46 2024-06-04T05:13:25.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.212+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/144577802 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55bdba9bea90 con 0x55bdba936f60 2024-06-04T05:13:25.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.212+0000 7f6596ffd700 1 -- 172.21.15.70:0/144577802 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65a8004cc0 con 0x55bdba936f60 2024-06-04T05:13:25.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.213+0000 7f6596ffd700 1 -- 172.21.15.70:0/144577802 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 46) ==== 8402+0+0 (secure 0 0 0) 0x7f65b0020040 con 0x55bdba936f60 2024-06-04T05:13:25.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.213+0000 7f65c8ae9b80 10 client.19701 fetch_fsmap finished waiting for FSMap version 46 2024-06-04T05:13:25.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.213+0000 7f65c8ae9b80 10 client.19701 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:13:25.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.213+0000 7f65c8ae9b80 20 client.19701 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:25.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.213+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/144577802 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55bdba9bf250 0x55bdba9c1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:25.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.213+0000 7f65c8ae9b80 4 client.19701 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:13:25.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.213+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/144577802 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bdba893be0 con 0x55bdba9bf250 2024-06-04T05:13:25.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.213+0000 7f65b77fe700 1 --2- 172.21.15.70:0/144577802 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55bdba9bf250 0x55bdba9c1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:25.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.214+0000 7f65b77fe700 1 --2- 172.21.15.70:0/144577802 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55bdba9bf250 0x55bdba9c1670 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9df3265794ea8b00 server_cookie=125373e6006f4bf in_seq=0 out_seq=0 2024-06-04T05:13:25.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.214+0000 7f6596ffd700 10 client.19701 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:13:25.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.214+0000 7f6596ffd700 1 -- 172.21.15.70:0/144577802 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bdba893be0 con 0x55bdba9bf250 2024-06-04T05:13:25.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.214+0000 7f6596ffd700 10 client.19701 handle_command_reply: tid=0 2024-06-04T05:13:25.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.214+0000 7f65c8ae9b80 1 client.19701 shutdown 2024-06-04T05:13:25.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.214+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/144577802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba936f60 msgr2=0x55bdba9416e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:25.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.215+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/144577802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba936f60 0x55bdba9416e0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f65b0005b90 tx=0x7f65b0017ab0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.215+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/144577802 shutdown_connections 2024-06-04T05:13:25.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.215+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/144577802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdba936f60 0x55bdba9416e0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.215+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/144577802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdba932860 0x55bdba9440e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.215+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/144577802 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55bdba9bf250 0x55bdba9c1670 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.215+0000 7f65c8ae9b80 1 --2- 172.21.15.70:0/144577802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdba933210 0x55bdba944620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.215+0000 7f6596ffd700 0 client.19701 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:13:25.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.215+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/144577802 >> 172.21.15.70:0/144577802 conn(0x55bdba860080 msgr2=0x55bdba931670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:25.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.215+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/144577802 shutdown_connections 2024-06-04T05:13:25.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.215+0000 7f65c8ae9b80 1 -- 172.21.15.70:0/144577802 wait complete. 2024-06-04T05:13:25.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.215+0000 7f65c8ae9b80 20 client.19701 trim_cache size 0 max 16384 2024-06-04T05:13:25.221 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:13:25.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.216+0000 7f65be254700 1 -- 172.21.15.70:0/2210904676 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65a00b2a70 msgr2=0x7f65a00b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:25.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.216+0000 7f65be254700 1 --2- 172.21.15.70:0/2210904676 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65a00b2a70 0x7f65a00b4f30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f65b0003e30 tx=0x7f65b000b040 comp rx=0 tx=0).stop 2024-06-04T05:13:25.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.216+0000 7f65be254700 1 -- 172.21.15.70:0/2210904676 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65b806bbe0 msgr2=0x7f65b8102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:25.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.216+0000 7f65be254700 1 --2- 172.21.15.70:0/2210904676 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65b806bbe0 0x7f65b8102a50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f65ac006010 tx=0x7f65ac0047b0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.216+0000 7f65be254700 1 -- 172.21.15.70:0/2210904676 shutdown_connections 2024-06-04T05:13:25.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.216+0000 7f65be254700 1 --2- 172.21.15.70:0/2210904676 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f65a00b2a70 0x7f65a00b4f30 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.216+0000 7f65be254700 1 --2- 172.21.15.70:0/2210904676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65b81095b0 0x7f65b8102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.216+0000 7f65be254700 1 --2- 172.21.15.70:0/2210904676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65b806b220 0x7f65b8102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.216+0000 7f65be254700 1 --2- 172.21.15.70:0/2210904676 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65b806bbe0 0x7f65b8102a50 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:25.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.216+0000 7f65be254700 1 -- 172.21.15.70:0/2210904676 >> 172.21.15.70:0/2210904676 conn(0x7f65b80793f0 msgr2=0x7f65b81012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:25.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.216+0000 7f65be254700 1 -- 172.21.15.70:0/2210904676 shutdown_connections 2024-06-04T05:13:25.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:25.216+0000 7f65be254700 1 -- 172.21.15.70:0/2210904676 wait complete. 2024-06-04T05:13:25.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:25 smithi070 ceph-mon[32631]: pgmap v508: 97 pgs: 97 active+clean; 460 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 286 op/s 2024-06-04T05:13:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:25 smithi112 ceph-mon[36453]: pgmap v508: 97 pgs: 97 active+clean; 460 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 286 op/s 2024-06-04T05:13:25.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:25 smithi098 ceph-mon[35569]: pgmap v508: 97 pgs: 97 active+clean; 460 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 286 op/s 2024-06-04T05:13:25.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 19514 9.48 MB/sec execute 17 sec latency 252.164 ms 2024-06-04T05:13:26.142 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:13:26.477 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:13:26.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:26 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:26.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:26 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:26.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:26 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:26.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 19702 9.68 MB/sec execute 18 sec latency 60.014 ms 2024-06-04T05:13:27.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:27 smithi070 ceph-mon[32631]: pgmap v509: 97 pgs: 97 active+clean; 473 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.1 MiB/s wr, 401 op/s 2024-06-04T05:13:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:27 smithi112 ceph-mon[36453]: pgmap v509: 97 pgs: 97 active+clean; 473 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.1 MiB/s wr, 401 op/s 2024-06-04T05:13:27.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:27 smithi098 ceph-mon[35569]: pgmap v509: 97 pgs: 97 active+clean; 473 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.1 MiB/s wr, 401 op/s 2024-06-04T05:13:27.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 19898 9.95 MB/sec execute 19 sec latency 107.768 ms 2024-06-04T05:13:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.288+0000 7f8d2701d640 1 -- 172.21.15.70:0/2215737445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d20107970 msgr2=0x7f8d20107d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.288+0000 7f8d2701d640 1 --2- 172.21.15.70:0/2215737445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d20107970 0x7f8d20107d50 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f8d14009930 tx=0x7f8d1402c650 comp rx=0 tx=0).stop 2024-06-04T05:13:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.288+0000 7f8d2701d640 1 -- 172.21.15.70:0/2215737445 shutdown_connections 2024-06-04T05:13:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.288+0000 7f8d2701d640 1 --2- 172.21.15.70:0/2215737445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d20102370 0x7f8d2010e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.288+0000 7f8d2701d640 1 --2- 172.21.15.70:0/2215737445 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d201019b0 0x7f8d20101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.288+0000 7f8d2701d640 1 --2- 172.21.15.70:0/2215737445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d20107970 0x7f8d20107d50 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.288+0000 7f8d2701d640 1 -- 172.21.15.70:0/2215737445 >> 172.21.15.70:0/2215737445 conn(0x7f8d20078aa0 msgr2=0x7f8d20078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.288+0000 7f8d2701d640 1 -- 172.21.15.70:0/2215737445 shutdown_connections 2024-06-04T05:13:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.288+0000 7f8d2701d640 1 -- 172.21.15.70:0/2215737445 wait complete. 2024-06-04T05:13:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.288+0000 7f8d2701d640 1 Processor -- start 2024-06-04T05:13:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d2701d640 1 -- start start 2024-06-04T05:13:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d2701d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d201019b0 0x7f8d20196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d2701d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d20102370 0x7f8d201973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d2701d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d20107970 0x7f8d2019b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d2701d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8d201109d0 con 0x7f8d201019b0 2024-06-04T05:13:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d2701d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8d20110850 con 0x7f8d20102370 2024-06-04T05:13:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d2701d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8d20110b50 con 0x7f8d20107970 2024-06-04T05:13:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d24db9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d201019b0 0x7f8d20196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d24db9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d201019b0 0x7f8d20196eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45442/0 (socket says 172.21.15.70:45442) 2024-06-04T05:13:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d24db9640 1 -- 172.21.15.70:0/3210902061 learned_addr learned my addr 172.21.15.70:0/3210902061 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/3210902061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d20102370 0x7f8d201973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d255ba640 1 --2- 172.21.15.70:0/3210902061 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d20107970 0x7f8d2019b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d1ffff640 1 -- 172.21.15.70:0/3210902061 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d20107970 msgr2=0x7f8d2019b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/3210902061 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d20107970 0x7f8d2019b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d1ffff640 1 -- 172.21.15.70:0/3210902061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d201019b0 msgr2=0x7f8d20196eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/3210902061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d201019b0 0x7f8d20196eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d1ffff640 1 -- 172.21.15.70:0/3210902061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d2019be50 con 0x7f8d20102370 2024-06-04T05:13:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.289+0000 7f8d24db9640 1 --2- 172.21.15.70:0/3210902061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d201019b0 0x7f8d20196eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.290+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/3210902061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d20102370 0x7f8d201973f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c00d6d0 tx=0x7f8d0c00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.290+0000 7f8d1dffb640 1 -- 172.21.15.70:0/3210902061 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c017070 con 0x7f8d20102370 2024-06-04T05:13:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.290+0000 7f8d2701d640 1 -- 172.21.15.70:0/3210902061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d2019c140 con 0x7f8d20102370 2024-06-04T05:13:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.290+0000 7f8d1dffb640 1 -- 172.21.15.70:0/3210902061 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8d0c009820 con 0x7f8d20102370 2024-06-04T05:13:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.290+0000 7f8d1dffb640 1 -- 172.21.15.70:0/3210902061 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c004bc0 con 0x7f8d20102370 2024-06-04T05:13:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.290+0000 7f8d2701d640 1 -- 172.21.15.70:0/3210902061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d201a39d0 con 0x7f8d20102370 2024-06-04T05:13:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.293+0000 7f8d29e0b740 1 Processor -- start 2024-06-04T05:13:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.293+0000 7f8d1dffb640 1 -- 172.21.15.70:0/3210902061 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f8d0c005220 con 0x7f8d20102370 2024-06-04T05:13:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.293+0000 7f8d29e0b740 1 -- start start 2024-06-04T05:13:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.293+0000 7f8d29e0b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d7e820 0x562b58d7ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.293+0000 7f8d29e0b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 0x562b58d894c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.294+0000 7f8d29e0b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d89a00 0x562b58d8bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.294+0000 7f8d29e0b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562b58cf0c30 con 0x562b58d89a00 2024-06-04T05:13:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.294+0000 7f8d29e0b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562b58ce07d0 con 0x562b58d7f140 2024-06-04T05:13:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.294+0000 7f8d29e0b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562b58ce4810 con 0x562b58d7e820 2024-06-04T05:13:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.294+0000 7f8d255ba640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d7e820 0x562b58d7ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.294+0000 7f8d255ba640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d7e820 0x562b58d7ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:56102/0 (socket says 172.21.15.70:56102) 2024-06-04T05:13:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.294+0000 7f8d255ba640 1 -- 172.21.15.70:0/196762641 learned_addr learned my addr 172.21.15.70:0/196762641 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.294+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/196762641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d89a00 0x562b58d8bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.294+0000 7f8d24db9640 1 --2- 172.21.15.70:0/196762641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 0x562b58d894c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.294+0000 7f8d255ba640 1 -- 172.21.15.70:0/196762641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 msgr2=0x562b58d894c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.294+0000 7f8d255ba640 1 --2- 172.21.15.70:0/196762641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 0x562b58d894c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.294+0000 7f8d255ba640 1 -- 172.21.15.70:0/196762641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d89a00 msgr2=0x562b58d8bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.294+0000 7f8d255ba640 1 --2- 172.21.15.70:0/196762641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d89a00 0x562b58d8bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.294+0000 7f8d255ba640 1 -- 172.21.15.70:0/196762641 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b589f5210 con 0x562b58d7e820 2024-06-04T05:13:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.294+0000 7f8d1dffb640 1 --2- 172.21.15.70:0/3210902061 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8d100b2a10 0x7f8d100b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.294+0000 7f8d255ba640 1 --2- 172.21.15.70:0/196762641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d7e820 0x562b58d7ec00 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8d18009f50 tx=0x7f8d180118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14a2a79bb2c99dea server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.295+0000 7f8d24db9640 1 --2- 172.21.15.70:0/3210902061 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8d100b2a10 0x7f8d100b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.295+0000 7f8d1dffb640 1 -- 172.21.15.70:0/3210902061 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f8d0c0e2110 con 0x7f8d20102370 2024-06-04T05:13:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.295+0000 7f8d0affd640 1 -- 172.21.15.70:0/196762641 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d18004d20 con 0x562b58d7e820 2024-06-04T05:13:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.295+0000 7f8d0affd640 1 -- 172.21.15.70:0/196762641 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8d18004050 con 0x562b58d7e820 2024-06-04T05:13:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.295+0000 7f8d0affd640 1 -- 172.21.15.70:0/196762641 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d18005240 con 0x562b58d7e820 2024-06-04T05:13:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.296+0000 7f8d29e0b740 1 -- 172.21.15.70:0/196762641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d7e820 msgr2=0x562b58d7ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.296+0000 7f8d24db9640 1 --2- 172.21.15.70:0/3210902061 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8d100b2a10 0x7f8d100b4ed0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x562b58d7fdc0 tx=0x7f8d140058d0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.296+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/196762641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d7e820 0x562b58d7ec00 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8d18009f50 tx=0x7f8d180118d0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.296+0000 7f8d29e0b740 1 -- 172.21.15.70:0/196762641 shutdown_connections 2024-06-04T05:13:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.296+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/196762641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 0x562b58d894c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.296+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/196762641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d89a00 0x562b58d8bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.296+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/196762641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d7e820 0x562b58d7ec00 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.296+0000 7f8d29e0b740 1 -- 172.21.15.70:0/196762641 >> 172.21.15.70:0/196762641 conn(0x562b58d7ad30 msgr2=0x562b58d7b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.296+0000 7f8d29e0b740 1 -- 172.21.15.70:0/196762641 shutdown_connections 2024-06-04T05:13:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.296+0000 7f8d29e0b740 1 -- 172.21.15.70:0/196762641 wait complete. 2024-06-04T05:13:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.296+0000 7f8d29e0b740 1 -- start start 2024-06-04T05:13:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.296+0000 7f8d29e0b740 1 Processor -- start 2024-06-04T05:13:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.297+0000 7f8d29e0b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7e820 0x562b58d86a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.297+0000 7f8d29e0b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 0x562b58d86f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.297+0000 7f8d29e0b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d89a00 0x562b58d87480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.297+0000 7f8d29e0b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562b589c6790 con 0x562b58d7e820 2024-06-04T05:13:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.297+0000 7f8d29e0b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562b58c9c890 con 0x562b58d7f140 2024-06-04T05:13:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.297+0000 7f8d29e0b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562b58d8c630 con 0x562b58d89a00 2024-06-04T05:13:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.297+0000 7f8d29e0b740 10 client.0 _ll_register_callbacks cb 0x562b589f4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.297+0000 7f8d1ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 0x562b58d86f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.297+0000 7f8d24db9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7e820 0x562b58d86a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.297+0000 7f8d255ba640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d89a00 0x562b58d87480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.297+0000 7f8d1ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 0x562b58d86f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:34874/0 (socket says 172.21.15.70:34874) 2024-06-04T05:13:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.297+0000 7f8d1ffff640 1 -- 172.21.15.70:0/2609447591 learned_addr learned my addr 172.21.15.70:0/2609447591 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.297+0000 7f8d1ffff640 1 -- 172.21.15.70:0/2609447591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d89a00 msgr2=0x562b58d87480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.297+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/2609447591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d89a00 0x562b58d87480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.297+0000 7f8d1ffff640 1 -- 172.21.15.70:0/2609447591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7e820 msgr2=0x562b58d86a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.297+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/2609447591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7e820 0x562b58d86a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.298+0000 7f8d1ffff640 1 -- 172.21.15.70:0/2609447591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b58ac1620 con 0x562b58d7f140 2024-06-04T05:13:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.298+0000 7f8d24db9640 1 --2- 172.21.15.70:0/2609447591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7e820 0x562b58d86a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.298+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/2609447591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 0x562b58d86f40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c00d6a0 tx=0x7f8d0c0a5350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23a62028507e029 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.298+0000 7f8cfffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:13:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.298+0000 7f8cfffff640 1 -- 172.21.15.70:0/2609447591 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c020070 con 0x562b58d7f140 2024-06-04T05:13:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.298+0000 7f8cfffff640 1 -- 172.21.15.70:0/2609447591 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f8d0c0099d0 con 0x562b58d7f140 2024-06-04T05:13:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.298+0000 7f8d29e0b740 1 -- 172.21.15.70:0/2609447591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d18004050 con 0x562b58d7f140 2024-06-04T05:13:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.298+0000 7f8cfffff640 1 -- 172.21.15.70:0/2609447591 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f8d0c0a8f20 con 0x562b58d7f140 2024-06-04T05:13:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.298+0000 7f8d29e0b740 10 client.28556 fetch_fsmap learned FSMap version 47 2024-06-04T05:13:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.298+0000 7f8d29e0b740 1 -- 172.21.15.70:0/2609447591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x562b58d9c880 con 0x562b58d7f140 2024-06-04T05:13:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.299+0000 7f8cfffff640 1 -- 172.21.15.70:0/2609447591 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8d0c0a9410 con 0x562b58d7f140 2024-06-04T05:13:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.299+0000 7f8cfffff640 1 -- 172.21.15.70:0/2609447591 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 47) ==== 8406+0+0 (secure 0 0 0) 0x7f8d0c0dfa50 con 0x562b58d7f140 2024-06-04T05:13:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.299+0000 7f8d29e0b740 10 client.28556 fetch_fsmap finished waiting for FSMap version 47 2024-06-04T05:13:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.299+0000 7f8d29e0b740 10 client.28556 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.299+0000 7f8d29e0b740 20 client.28556 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.299+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/2609447591 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562b58d9dde0 0x562b58da01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.299+0000 7f8d29e0b740 4 client.28556 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:13:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.299+0000 7f8d29e0b740 1 -- 172.21.15.70:0/2609447591 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8d18004050 con 0x562b58d9dde0 2024-06-04T05:13:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.299+0000 7f8d24db9640 1 --2- 172.21.15.70:0/2609447591 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562b58d9dde0 0x562b58da01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.300+0000 7f8d24db9640 1 --2- 172.21.15.70:0/2609447591 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562b58d9dde0 0x562b58da01f0 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c86976b5663e8f07 server_cookie=a47e1aa11416cb75 in_seq=0 out_seq=0 2024-06-04T05:13:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.300+0000 7f8cfffff640 10 client.28556 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.300+0000 7f8cfffff640 1 -- 172.21.15.70:0/2609447591 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f8d18004050 con 0x562b58d9dde0 2024-06-04T05:13:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.300+0000 7f8cfffff640 10 client.28556 handle_command_reply: tid=0 2024-06-04T05:13:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.300+0000 7f8d29e0b740 1 client.28556 shutdown 2024-06-04T05:13:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.301+0000 7f8d29e0b740 1 -- 172.21.15.70:0/2609447591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 msgr2=0x562b58d86f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.301+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/2609447591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 0x562b58d86f40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c00d6a0 tx=0x7f8d0c0a5350 comp rx=0 tx=0).stop 2024-06-04T05:13:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.301+0000 7f8d29e0b740 1 -- 172.21.15.70:0/2609447591 shutdown_connections 2024-06-04T05:13:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.301+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/2609447591 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562b58d9dde0 0x562b58da01f0 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.301+0000 7f8cfffff640 0 client.28556 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.301+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/2609447591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 0x562b58d86f40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.301+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/2609447591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d89a00 0x562b58d87480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.301+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/2609447591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7e820 0x562b58d86a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.301+0000 7f8d29e0b740 1 -- 172.21.15.70:0/2609447591 >> 172.21.15.70:0/2609447591 conn(0x562b58d7ad30 msgr2=0x562b58d79fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.301+0000 7f8d29e0b740 1 -- 172.21.15.70:0/2609447591 shutdown_connections 2024-06-04T05:13:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.301+0000 7f8d29e0b740 1 -- 172.21.15.70:0/2609447591 wait complete. 2024-06-04T05:13:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.301+0000 7f8d29e0b740 20 client.28556 trim_cache size 0 max 16384 2024-06-04T05:13:28.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.320+0000 7f8d29e0b740 1 Processor -- start 2024-06-04T05:13:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.320+0000 7f8d29e0b740 1 -- start start 2024-06-04T05:13:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d29e0b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7e820 0x562b58d811e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d29e0b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7f140 0x562b58d94d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d29e0b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d89a00 0x562b58d952d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d29e0b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8d201ac460 con 0x562b58d7f140 2024-06-04T05:13:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d29e0b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8d201ae150 con 0x562b58d7e820 2024-06-04T05:13:28.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d29e0b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8d201acb90 con 0x562b58d89a00 2024-06-04T05:13:28.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d1ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7f140 0x562b58d94d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d1ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7f140 0x562b58d94d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45470/0 (socket says 172.21.15.70:45470) 2024-06-04T05:13:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d1ffff640 1 -- 172.21.15.70:0/986491788 learned_addr learned my addr 172.21.15.70:0/986491788 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d255ba640 1 --2- 172.21.15.70:0/986491788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d89a00 0x562b58d952d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d24db9640 1 --2- 172.21.15.70:0/986491788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7e820 0x562b58d811e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d1ffff640 1 -- 172.21.15.70:0/986491788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d89a00 msgr2=0x562b58d952d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/986491788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d89a00 0x562b58d952d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d1ffff640 1 -- 172.21.15.70:0/986491788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7e820 msgr2=0x562b58d811e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/986491788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7e820 0x562b58d811e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d1ffff640 1 -- 172.21.15.70:0/986491788 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b58d84f70 con 0x562b58d7f140 2024-06-04T05:13:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d255ba640 1 --2- 172.21.15.70:0/986491788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d89a00 0x562b58d952d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.321+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/986491788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7f140 0x562b58d94d90 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c0a6680 tx=0x7f8d0c0e23f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a2044da90c5286 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.322+0000 7f8d0a7fc640 1 -- 172.21.15.70:0/986491788 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c0c2070 con 0x562b58d7f140 2024-06-04T05:13:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.322+0000 7f8d0a7fc640 1 -- 172.21.15.70:0/986491788 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8d0c0b21c0 con 0x562b58d7f140 2024-06-04T05:13:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.322+0000 7f8d0a7fc640 1 -- 172.21.15.70:0/986491788 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c020030 con 0x562b58d7f140 2024-06-04T05:13:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.322+0000 7f8d29e0b740 1 -- 172.21.15.70:0/986491788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7f140 msgr2=0x562b58d94d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:28.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.322+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/986491788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7f140 0x562b58d94d90 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c0a6680 tx=0x7f8d0c0e23f0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.323+0000 7f8d29e0b740 1 -- 172.21.15.70:0/986491788 shutdown_connections 2024-06-04T05:13:28.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.323+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/986491788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d89a00 0x562b58d952d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.323+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/986491788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7f140 0x562b58d94d90 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.323+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/986491788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7e820 0x562b58d811e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.323+0000 7f8d29e0b740 1 -- 172.21.15.70:0/986491788 >> 172.21.15.70:0/986491788 conn(0x562b58d7ad30 msgr2=0x562b58d94680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:28.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.323+0000 7f8d29e0b740 1 -- 172.21.15.70:0/986491788 shutdown_connections 2024-06-04T05:13:28.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.323+0000 7f8d29e0b740 1 -- 172.21.15.70:0/986491788 wait complete. 2024-06-04T05:13:28.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.323+0000 7f8d29e0b740 1 -- start start 2024-06-04T05:13:28.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.323+0000 7f8d29e0b740 1 Processor -- start 2024-06-04T05:13:28.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.323+0000 7f8d29e0b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7e820 0x562b58d83ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.323+0000 7f8d29e0b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 0x562b58d840e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.323+0000 7f8d29e0b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d89a00 0x562b58d84620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.323+0000 7f8d29e0b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8d201ad9c0 con 0x562b58d7e820 2024-06-04T05:13:28.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.323+0000 7f8d29e0b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8d201ad2c0 con 0x562b58d7f140 2024-06-04T05:13:28.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.323+0000 7f8d29e0b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8d201aa460 con 0x562b58d89a00 2024-06-04T05:13:28.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8d29e0b740 10 client.0 _ll_register_callbacks cb 0x7f8d0c004f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:28.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8d24db9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7e820 0x562b58d83ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8d1ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 0x562b58d840e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8d24db9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7e820 0x562b58d83ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45484/0 (socket says 172.21.15.70:45484) 2024-06-04T05:13:28.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8d24db9640 1 -- 172.21.15.70:0/633247693 learned_addr learned my addr 172.21.15.70:0/633247693 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8d255ba640 1 --2- 172.21.15.70:0/633247693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d89a00 0x562b58d84620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8d1ffff640 1 -- 172.21.15.70:0/633247693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d89a00 msgr2=0x562b58d84620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/633247693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d89a00 0x562b58d84620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8d1ffff640 1 -- 172.21.15.70:0/633247693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7e820 msgr2=0x562b58d83ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/633247693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7e820 0x562b58d83ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8d1ffff640 1 -- 172.21.15.70:0/633247693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b58e07b60 con 0x562b58d7f140 2024-06-04T05:13:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8d24db9640 1 --2- 172.21.15.70:0/633247693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7e820 0x562b58d83ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8d255ba640 1 --2- 172.21.15.70:0/633247693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d89a00 0x562b58d84620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/633247693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 0x562b58d840e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x562b58d7bf80 tx=0x7f8d0c0dff90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cebacfc5da55c5c7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8cfffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:13:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8cfffff640 1 -- 172.21.15.70:0/633247693 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c0c2070 con 0x562b58d7f140 2024-06-04T05:13:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8d29e0b740 1 -- 172.21.15.70:0/633247693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d0c0b21c0 con 0x562b58d7f140 2024-06-04T05:13:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.324+0000 7f8cfffff640 1 -- 172.21.15.70:0/633247693 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f8d0c0a56a0 con 0x562b58d7f140 2024-06-04T05:13:28.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.325+0000 7f8cfffff640 1 -- 172.21.15.70:0/633247693 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f8d0c020070 con 0x562b58d7f140 2024-06-04T05:13:28.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.325+0000 7f8d29e0b740 10 client.28562 fetch_fsmap learned FSMap version 47 2024-06-04T05:13:28.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.325+0000 7f8d29e0b740 1 -- 172.21.15.70:0/633247693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x562b58e08e30 con 0x562b58d7f140 2024-06-04T05:13:28.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.325+0000 7f8cfffff640 1 -- 172.21.15.70:0/633247693 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8d0c0b21c0 con 0x562b58d7f140 2024-06-04T05:13:28.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.325+0000 7f8cfffff640 1 -- 172.21.15.70:0/633247693 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 47) ==== 8406+0+0 (secure 0 0 0) 0x7f8d0c0a4d80 con 0x562b58d7f140 2024-06-04T05:13:28.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.325+0000 7f8d29e0b740 10 client.28562 fetch_fsmap finished waiting for FSMap version 47 2024-06-04T05:13:28.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.325+0000 7f8d29e0b740 10 client.28562 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:28.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.325+0000 7f8d29e0b740 20 client.28562 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:28.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.325+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/633247693 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562b58d9dde0 0x562b58e0b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:28.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.325+0000 7f8d29e0b740 4 client.28562 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:13:28.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.325+0000 7f8d29e0b740 1 -- 172.21.15.70:0/633247693 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562b58cf2b80 con 0x562b58d9dde0 2024-06-04T05:13:28.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.325+0000 7f8d24db9640 1 --2- 172.21.15.70:0/633247693 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562b58d9dde0 0x562b58e0b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:28.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.326+0000 7f8cfffff640 10 client.28562 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:28.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.326+0000 7f8d24db9640 1 --2- 172.21.15.70:0/633247693 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562b58d9dde0 0x562b58e0b360 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2916e45897b3b8a server_cookie=510910418ff19127 in_seq=0 out_seq=0 2024-06-04T05:13:28.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.428+0000 7f8cfffff640 1 -- 172.21.15.70:0/633247693 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x562b58cf2b80 con 0x562b58d9dde0 2024-06-04T05:13:28.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.428+0000 7f8cfffff640 10 client.28562 handle_command_reply: tid=0 2024-06-04T05:13:28.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.428+0000 7f8d29e0b740 1 client.28562 shutdown 2024-06-04T05:13:28.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.428+0000 7f8d29e0b740 1 -- 172.21.15.70:0/633247693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 msgr2=0x562b58d840e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:28.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.428+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/633247693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 0x562b58d840e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x562b58d7bf80 tx=0x7f8d0c0dff90 comp rx=0 tx=0).stop 2024-06-04T05:13:28.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.429+0000 7f8d29e0b740 1 -- 172.21.15.70:0/633247693 shutdown_connections 2024-06-04T05:13:28.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.429+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/633247693 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562b58d9dde0 0x562b58e0b360 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.429+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/633247693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b58d7f140 0x562b58d840e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.429+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/633247693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562b58d89a00 0x562b58d84620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.429+0000 7f8cfffff640 0 client.28562 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.429+0000 7f8d29e0b740 1 --2- 172.21.15.70:0/633247693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562b58d7e820 0x562b58d83ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.429+0000 7f8d29e0b740 1 -- 172.21.15.70:0/633247693 >> 172.21.15.70:0/633247693 conn(0x562b58d7ad30 msgr2=0x562b58d94f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.429+0000 7f8d29e0b740 1 -- 172.21.15.70:0/633247693 shutdown_connections 2024-06-04T05:13:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.429+0000 7f8d29e0b740 1 -- 172.21.15.70:0/633247693 wait complete. 2024-06-04T05:13:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.429+0000 7f8d29e0b740 20 client.28562 trim_cache size 0 max 16384 2024-06-04T05:13:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:13:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:13:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:13:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.430+0000 7f8d2701d640 1 -- 172.21.15.70:0/3210902061 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8d100b2a10 msgr2=0x7f8d100b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.430+0000 7f8d2701d640 1 --2- 172.21.15.70:0/3210902061 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8d100b2a10 0x7f8d100b4ed0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x562b58d7fdc0 tx=0x7f8d140058d0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.430+0000 7f8d2701d640 1 -- 172.21.15.70:0/3210902061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d20102370 msgr2=0x7f8d201973f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.430+0000 7f8d2701d640 1 --2- 172.21.15.70:0/3210902061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d20102370 0x7f8d201973f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c00d6d0 tx=0x7f8d0c00db90 comp rx=0 tx=0).stop 2024-06-04T05:13:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.430+0000 7f8d2701d640 1 -- 172.21.15.70:0/3210902061 shutdown_connections 2024-06-04T05:13:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.430+0000 7f8d2701d640 1 --2- 172.21.15.70:0/3210902061 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8d100b2a10 0x7f8d100b4ed0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.430+0000 7f8d2701d640 1 --2- 172.21.15.70:0/3210902061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d20102370 0x7f8d201973f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.430+0000 7f8d2701d640 1 --2- 172.21.15.70:0/3210902061 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d20107970 0x7f8d2019b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.430+0000 7f8d2701d640 1 --2- 172.21.15.70:0/3210902061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d201019b0 0x7f8d20196eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.430+0000 7f8d2701d640 1 -- 172.21.15.70:0/3210902061 >> 172.21.15.70:0/3210902061 conn(0x7f8d20078aa0 msgr2=0x7f8d20104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.430+0000 7f8d2701d640 1 -- 172.21.15.70:0/3210902061 shutdown_connections 2024-06-04T05:13:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:28.430+0000 7f8d2701d640 1 -- 172.21.15.70:0/3210902061 wait complete. 2024-06-04T05:13:28.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 20047 9.77 MB/sec execute 20 sec latency 119.756 ms 2024-06-04T05:13:28.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.995+0000 7fe1e2b76700 1 -- 172.21.15.70:0/554874115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc109610 msgr2=0x7fe1dc1099f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:28.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.995+0000 7fe1e2b76700 1 --2- 172.21.15.70:0/554874115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc109610 0x7fe1dc1099f0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc0099c0 tx=0x7fe1cc009cd0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.995+0000 7fe1e2b76700 1 -- 172.21.15.70:0/554874115 shutdown_connections 2024-06-04T05:13:28.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.995+0000 7fe1e2b76700 1 --2- 172.21.15.70:0/554874115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1dc102450 0x7fe1dc1103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.995+0000 7fe1e2b76700 1 --2- 172.21.15.70:0/554874115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc109610 0x7fe1dc1099f0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.995+0000 7fe1e2b76700 1 --2- 172.21.15.70:0/554874115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1dc101a90 0x7fe1dc101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:28.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.995+0000 7fe1e2b76700 1 -- 172.21.15.70:0/554874115 >> 172.21.15.70:0/554874115 conn(0x7fe1dc078a40 msgr2=0x7fe1dc0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:28.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.995+0000 7fe1e2b76700 1 -- 172.21.15.70:0/554874115 shutdown_connections 2024-06-04T05:13:28.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.995+0000 7fe1e2b76700 1 -- 172.21.15.70:0/554874115 wait complete. 2024-06-04T05:13:28.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.996+0000 7fe1e2b76700 1 Processor -- start 2024-06-04T05:13:29.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.996+0000 7fe1e2b76700 1 -- start start 2024-06-04T05:13:29.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.997+0000 7fe1e2b76700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1dc101a90 0x7fe1dc10d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.997+0000 7fe1e2b76700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc102450 0x7fe1dc10d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.997+0000 7fe1dbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc102450 0x7fe1dc10d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:29.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.997+0000 7fe1dbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc102450 0x7fe1dc10d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45508/0 (socket says 172.21.15.70:45508) 2024-06-04T05:13:29.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.997+0000 7fe1dbfff700 1 -- 172.21.15.70:0/3525762327 learned_addr learned my addr 172.21.15.70:0/3525762327 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:29.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.997+0000 7fe1e2b76700 1 --2- 172.21.15.70:0/3525762327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1dc109610 0x7fe1dc1a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.997+0000 7fe1e2b76700 1 -- 172.21.15.70:0/3525762327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc104180 con 0x7fe1dc102450 2024-06-04T05:13:29.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.997+0000 7fe1e2b76700 1 -- 172.21.15.70:0/3525762327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc104000 con 0x7fe1dc101a90 2024-06-04T05:13:29.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.997+0000 7fe1e2b76700 1 -- 172.21.15.70:0/3525762327 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc104300 con 0x7fe1dc109610 2024-06-04T05:13:29.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.997+0000 7fe1e1113700 1 --2- 172.21.15.70:0/3525762327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1dc109610 0x7fe1dc1a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:29.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.997+0000 7fe1e0912700 1 --2- 172.21.15.70:0/3525762327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1dc101a90 0x7fe1dc10d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:29.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.998+0000 7fe1dbfff700 1 -- 172.21.15.70:0/3525762327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1dc109610 msgr2=0x7fe1dc1a49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:29.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.998+0000 7fe1dbfff700 1 --2- 172.21.15.70:0/3525762327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1dc109610 0x7fe1dc1a49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.998+0000 7fe1dbfff700 1 -- 172.21.15.70:0/3525762327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1dc101a90 msgr2=0x7fe1dc10d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:29.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.998+0000 7fe1dbfff700 1 --2- 172.21.15.70:0/3525762327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1dc101a90 0x7fe1dc10d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.998+0000 7fe1dbfff700 1 -- 172.21.15.70:0/3525762327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1dc1a5120 con 0x7fe1dc102450 2024-06-04T05:13:29.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.998+0000 7fe1e0912700 1 --2- 172.21.15.70:0/3525762327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1dc101a90 0x7fe1dc10d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:29.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.998+0000 7fe1dbfff700 1 --2- 172.21.15.70:0/3525762327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc102450 0x7fe1dc10d5a0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000eb30 tx=0x7fe1d000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:29.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.999+0000 7fe1d9ffb700 1 -- 172.21.15.70:0/3525762327 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d000cbd0 con 0x7fe1dc102450 2024-06-04T05:13:29.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.999+0000 7fe1d9ffb700 1 -- 172.21.15.70:0/3525762327 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1d00044b0 con 0x7fe1dc102450 2024-06-04T05:13:29.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.999+0000 7fe1e2b76700 1 -- 172.21.15.70:0/3525762327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1dc1a5410 con 0x7fe1dc102450 2024-06-04T05:13:29.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.999+0000 7fe1d9ffb700 1 -- 172.21.15.70:0/3525762327 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0003ce0 con 0x7fe1dc102450 2024-06-04T05:13:29.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:28.999+0000 7fe1e2b76700 1 -- 172.21.15.70:0/3525762327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1dc1acc90 con 0x7fe1dc102450 2024-06-04T05:13:29.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.002+0000 7fe1d9ffb700 1 -- 172.21.15.70:0/3525762327 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fe1d0003f40 con 0x7fe1dc102450 2024-06-04T05:13:29.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.003+0000 7fe1d9ffb700 1 --2- 172.21.15.70:0/3525762327 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1c40b2b80 0x7fe1c40b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.004+0000 7fe1e0912700 1 --2- 172.21.15.70:0/3525762327 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1c40b2b80 0x7fe1c40b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:29.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.004+0000 7fe1d9ffb700 1 -- 172.21.15.70:0/3525762327 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fe1d00e2880 con 0x7fe1dc102450 2024-06-04T05:13:29.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.004+0000 7fe1e0912700 1 --2- 172.21.15.70:0/3525762327 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1c40b2b80 0x7fe1c40b5040 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc006010 tx=0x7fe1cc009e60 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:29.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.005+0000 7fe1ed40bb80 1 Processor -- start 2024-06-04T05:13:29.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.005+0000 7fe1ed40bb80 1 -- start start 2024-06-04T05:13:29.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.005+0000 7fe1ed40bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a4860 0x55e6c51a4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.005+0000 7fe1ed40bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a5210 0x55e6c51a8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.005+0000 7fe1ed40bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 0x55e6c51a93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.005+0000 7fe1ed40bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e6c5103c90 con 0x55e6c51a8f60 2024-06-04T05:13:29.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.005+0000 7fe1ed40bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e6c50f3830 con 0x55e6c51a4860 2024-06-04T05:13:29.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.005+0000 7fe1ed40bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e6c50f7870 con 0x55e6c51a5210 2024-06-04T05:13:29.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.006+0000 7fe1e0912700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a4860 0x55e6c51a4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:29.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.006+0000 7fe1e0912700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a4860 0x55e6c51a4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:34920/0 (socket says 172.21.15.70:34920) 2024-06-04T05:13:29.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.006+0000 7fe1e0912700 1 -- 172.21.15.70:0/2858151255 learned_addr learned my addr 172.21.15.70:0/2858151255 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:29.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.006+0000 7fe1dbfff700 1 --2- 172.21.15.70:0/2858151255 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a5210 0x55e6c51a8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:29.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.006+0000 7fe1e1113700 1 --2- 172.21.15.70:0/2858151255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 0x55e6c51a93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:29.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.006+0000 7fe1dbfff700 1 -- 172.21.15.70:0/2858151255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a4860 msgr2=0x55e6c51a4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:29.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.006+0000 7fe1dbfff700 1 --2- 172.21.15.70:0/2858151255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a4860 0x55e6c51a4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.006+0000 7fe1dbfff700 1 -- 172.21.15.70:0/2858151255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 msgr2=0x55e6c51a93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:29.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.006+0000 7fe1dbfff700 1 --2- 172.21.15.70:0/2858151255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 0x55e6c51a93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.006+0000 7fe1dbfff700 1 -- 172.21.15.70:0/2858151255 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6c50438d0 con 0x55e6c51a5210 2024-06-04T05:13:29.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.006+0000 7fe1e1113700 1 --2- 172.21.15.70:0/2858151255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 0x55e6c51a93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:29.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.007+0000 7fe1dbfff700 1 --2- 172.21.15.70:0/2858151255 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a5210 0x55e6c51a8940 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00dffa0 tx=0x7fe1d00a5b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd8f894f3ec67e0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:29.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.007+0000 7fe1c2ffd700 1 -- 172.21.15.70:0/2858151255 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d00aae60 con 0x55e6c51a5210 2024-06-04T05:13:29.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.007+0000 7fe1c2ffd700 1 -- 172.21.15.70:0/2858151255 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1d00b22e0 con 0x55e6c51a5210 2024-06-04T05:13:29.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.007+0000 7fe1c2ffd700 1 -- 172.21.15.70:0/2858151255 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d00a9b30 con 0x55e6c51a5210 2024-06-04T05:13:29.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.008+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2858151255 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a5210 msgr2=0x55e6c51a8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:29.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.008+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/2858151255 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a5210 0x55e6c51a8940 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00dffa0 tx=0x7fe1d00a5b10 comp rx=0 tx=0).stop 2024-06-04T05:13:29.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.008+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2858151255 shutdown_connections 2024-06-04T05:13:29.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.008+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/2858151255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 0x55e6c51a93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.008+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/2858151255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a4860 0x55e6c51a4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.008+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/2858151255 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a5210 0x55e6c51a8940 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.008+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2858151255 >> 172.21.15.70:0/2858151255 conn(0x55e6c50d2080 msgr2=0x55e6c519ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:29.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.008+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2858151255 shutdown_connections 2024-06-04T05:13:29.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.008+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2858151255 wait complete. 2024-06-04T05:13:29.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.009+0000 7fe1ed40bb80 1 -- start start 2024-06-04T05:13:29.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.009+0000 7fe1ed40bb80 1 Processor -- start 2024-06-04T05:13:29.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.010+0000 7fe1ed40bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a4860 0x55e6c51b0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.010+0000 7fe1ed40bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a5210 0x55e6c51ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.010+0000 7fe1ed40bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 0x55e6c51adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.010+0000 7fe1ed40bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e6c4ff9fa0 con 0x55e6c51a8f60 2024-06-04T05:13:29.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.010+0000 7fe1ed40bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e6c50957c0 con 0x55e6c51a5210 2024-06-04T05:13:29.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.010+0000 7fe1ed40bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e6c51b1120 con 0x55e6c51a4860 2024-06-04T05:13:29.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.010+0000 7fe1e0912700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a4860 0x55e6c51b0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:29.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.010+0000 7fe1e0912700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a4860 0x55e6c51b0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:56192/0 (socket says 172.21.15.70:56192) 2024-06-04T05:13:29.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.010+0000 7fe1e0912700 1 -- 172.21.15.70:0/1919833822 learned_addr learned my addr 172.21.15.70:0/1919833822 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:29.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.010+0000 7fe1ed40bb80 10 client.0 _ll_register_callbacks cb 0x55e6c508e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:29.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.010+0000 7fe1e0912700 1 -- 172.21.15.70:0/1919833822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a5210 msgr2=0x55e6c51ad770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-04T05:13:29.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.010+0000 7fe1e0912700 1 --2- 172.21.15.70:0/1919833822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a5210 0x55e6c51ad770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.010+0000 7fe1e0912700 1 -- 172.21.15.70:0/1919833822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 msgr2=0x55e6c51adcb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-04T05:13:29.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.011+0000 7fe1e1113700 1 --2- 172.21.15.70:0/1919833822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 0x55e6c51adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:29.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.011+0000 7fe1e0912700 1 --2- 172.21.15.70:0/1919833822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 0x55e6c51adcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.011+0000 7fe1e0912700 1 -- 172.21.15.70:0/1919833822 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6c5092690 con 0x55e6c51a4860 2024-06-04T05:13:29.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.011+0000 7fe1e1113700 1 --2- 172.21.15.70:0/1919833822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 0x55e6c51adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:13:29.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.012+0000 7fe1e0912700 1 --2- 172.21.15.70:0/1919833822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a4860 0x55e6c51b0470 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc000c00 tx=0x7fe1cc020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=153a2ebe68bd55fe server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:29.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.012+0000 7fe1bbfff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:13:29.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.012+0000 7fe1bbfff700 1 -- 172.21.15.70:0/1919833822 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc032630 con 0x55e6c51a4860 2024-06-04T05:13:29.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.012+0000 7fe1bbfff700 1 -- 172.21.15.70:0/1919833822 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fe1cc024410 con 0x55e6c51a4860 2024-06-04T05:13:29.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.012+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/1919833822 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1d00b22e0 con 0x55e6c51a4860 2024-06-04T05:13:29.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.013+0000 7fe1bbfff700 1 -- 172.21.15.70:0/1919833822 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fe1cc003e10 con 0x55e6c51a4860 2024-06-04T05:13:29.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.013+0000 7fe1ed40bb80 10 client.28684 fetch_fsmap learned FSMap version 47 2024-06-04T05:13:29.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.013+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/1919833822 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55e6c51bb6c0 con 0x55e6c51a4860 2024-06-04T05:13:29.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.013+0000 7fe1bbfff700 1 -- 172.21.15.70:0/1919833822 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1d00b22e0 con 0x55e6c51a4860 2024-06-04T05:13:29.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.014+0000 7fe1bbfff700 1 -- 172.21.15.70:0/1919833822 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 47) ==== 8406+0+0 (secure 0 0 0) 0x7fe1cc0249b0 con 0x55e6c51a4860 2024-06-04T05:13:29.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.014+0000 7fe1ed40bb80 10 client.28684 fetch_fsmap finished waiting for FSMap version 47 2024-06-04T05:13:29.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.014+0000 7fe1ed40bb80 10 client.28684 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:29.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.014+0000 7fe1ed40bb80 20 client.28684 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:29.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.014+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/1919833822 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e6c51bdda0 0x55e6c51c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.014+0000 7fe1ed40bb80 4 client.28684 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:13:29.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.014+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/1919833822 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6c51709e0 con 0x55e6c51bdda0 2024-06-04T05:13:29.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.014+0000 7fe1dbfff700 1 --2- 172.21.15.70:0/1919833822 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e6c51bdda0 0x55e6c51c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:29.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.014+0000 7fe1bbfff700 10 client.28684 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:29.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.015+0000 7fe1dbfff700 1 --2- 172.21.15.70:0/1919833822 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e6c51bdda0 0x55e6c51c01b0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ece7884af5b571c server_cookie=f517b7c5fbe93ec6 in_seq=0 out_seq=0 2024-06-04T05:13:29.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.016+0000 7fe1bbfff700 1 -- 172.21.15.70:0/1919833822 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e6c51709e0 con 0x55e6c51bdda0 2024-06-04T05:13:29.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.016+0000 7fe1bbfff700 10 client.28684 handle_command_reply: tid=0 2024-06-04T05:13:29.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.016+0000 7fe1ed40bb80 1 client.28684 shutdown 2024-06-04T05:13:29.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.016+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/1919833822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a4860 msgr2=0x55e6c51b0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:29.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.016+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/1919833822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a4860 0x55e6c51b0470 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc000c00 tx=0x7fe1cc020000 comp rx=0 tx=0).stop 2024-06-04T05:13:29.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.016+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/1919833822 shutdown_connections 2024-06-04T05:13:29.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.016+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/1919833822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 0x55e6c51adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.016+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/1919833822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a4860 0x55e6c51b0470 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.016+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/1919833822 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e6c51bdda0 0x55e6c51c01b0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.016+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/1919833822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a5210 0x55e6c51ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.016+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/1919833822 >> 172.21.15.70:0/1919833822 conn(0x55e6c50d2080 msgr2=0x55e6c519e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:29.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.016+0000 7fe1bbfff700 0 client.28684 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:29.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.016+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/1919833822 shutdown_connections 2024-06-04T05:13:29.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.016+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/1919833822 wait complete. 2024-06-04T05:13:29.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.017+0000 7fe1ed40bb80 20 client.28684 trim_cache size 0 max 16384 2024-06-04T05:13:29.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.037+0000 7fe1ed40bb80 1 Processor -- start 2024-06-04T05:13:29.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.037+0000 7fe1ed40bb80 1 -- start start 2024-06-04T05:13:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.038+0000 7fe1ed40bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a4860 0x55e6c51adce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.038+0000 7fe1ed40bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a5210 0x55e6c51af6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.038+0000 7fe1ed40bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a8f60 0x55e6c519ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.038+0000 7fe1ed40bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc1b4450 con 0x55e6c51a5210 2024-06-04T05:13:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.038+0000 7fe1ed40bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc1b6140 con 0x55e6c51a4860 2024-06-04T05:13:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.038+0000 7fe1ed40bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc1b4b80 con 0x55e6c51a8f60 2024-06-04T05:13:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.038+0000 7fe1e0912700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a4860 0x55e6c51adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.038+0000 7fe1dbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a5210 0x55e6c51af6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.038+0000 7fe1dbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a5210 0x55e6c51af6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45540/0 (socket says 172.21.15.70:45540) 2024-06-04T05:13:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.038+0000 7fe1dbfff700 1 -- 172.21.15.70:0/2838057550 learned_addr learned my addr 172.21.15.70:0/2838057550 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.038+0000 7fe1dbfff700 1 -- 172.21.15.70:0/2838057550 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a8f60 msgr2=0x55e6c519ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.038+0000 7fe1dbfff700 1 --2- 172.21.15.70:0/2838057550 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a8f60 0x55e6c519ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.038+0000 7fe1dbfff700 1 -- 172.21.15.70:0/2838057550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a4860 msgr2=0x55e6c51adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.038+0000 7fe1dbfff700 1 --2- 172.21.15.70:0/2838057550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a4860 0x55e6c51adce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.038+0000 7fe1dbfff700 1 -- 172.21.15.70:0/2838057550 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6c51ae220 con 0x55e6c51a5210 2024-06-04T05:13:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.039+0000 7fe1dbfff700 1 --2- 172.21.15.70:0/2838057550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a5210 0x55e6c51af6b0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00037d0 tx=0x7fe1d0004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4a3ff822336773 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:29.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.039+0000 7fe1c27fc700 1 -- 172.21.15.70:0/2838057550 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d00aae60 con 0x55e6c51a5210 2024-06-04T05:13:29.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.039+0000 7fe1c27fc700 1 -- 172.21.15.70:0/2838057550 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1d00bc070 con 0x55e6c51a5210 2024-06-04T05:13:29.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.039+0000 7fe1c27fc700 1 -- 172.21.15.70:0/2838057550 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d00adee0 con 0x55e6c51a5210 2024-06-04T05:13:29.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.040+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2838057550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a5210 msgr2=0x55e6c51af6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:29.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.040+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/2838057550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a5210 0x55e6c51af6b0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00037d0 tx=0x7fe1d0004440 comp rx=0 tx=0).stop 2024-06-04T05:13:29.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.040+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2838057550 shutdown_connections 2024-06-04T05:13:29.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.040+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/2838057550 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a8f60 0x55e6c519ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.040+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/2838057550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a4860 0x55e6c51adce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.040+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/2838057550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a5210 0x55e6c51af6b0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.040+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2838057550 >> 172.21.15.70:0/2838057550 conn(0x55e6c50d2080 msgr2=0x55e6c51aefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:29.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.041+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2838057550 shutdown_connections 2024-06-04T05:13:29.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.041+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2838057550 wait complete. 2024-06-04T05:13:29.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.041+0000 7fe1ed40bb80 1 -- start start 2024-06-04T05:13:29.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.041+0000 7fe1ed40bb80 1 Processor -- start 2024-06-04T05:13:29.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1ed40bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a4860 0x55e6c521fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1ed40bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a5210 0x55e6c5220230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1ed40bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 0x55e6c51b3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1ed40bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc1b59b0 con 0x55e6c51a8f60 2024-06-04T05:13:29.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1ed40bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc1b52b0 con 0x55e6c51a5210 2024-06-04T05:13:29.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1ed40bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc1b2410 con 0x55e6c51a4860 2024-06-04T05:13:29.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1ed40bb80 10 client.0 _ll_register_callbacks cb 0x55e6c508e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:29.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1e0912700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a4860 0x55e6c521fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:29.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1dbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a5210 0x55e6c5220230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:29.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1e1113700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 0x55e6c51b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:29.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1dbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a5210 0x55e6c5220230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:34952/0 (socket says 172.21.15.70:34952) 2024-06-04T05:13:29.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1e0912700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a4860 0x55e6c521fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:56210/0 (socket says 172.21.15.70:56210) 2024-06-04T05:13:29.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1dbfff700 1 -- 172.21.15.70:0/2451818139 learned_addr learned my addr 172.21.15.70:0/2451818139 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:29.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1e1113700 1 -- 172.21.15.70:0/2451818139 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a4860 msgr2=0x55e6c521fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:29.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1e1113700 1 --2- 172.21.15.70:0/2451818139 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a4860 0x55e6c521fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1e1113700 1 -- 172.21.15.70:0/2451818139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a5210 msgr2=0x55e6c5220230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:29.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1e1113700 1 --2- 172.21.15.70:0/2451818139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a5210 0x55e6c5220230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1e1113700 1 -- 172.21.15.70:0/2451818139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6c51b4160 con 0x55e6c51a8f60 2024-06-04T05:13:29.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1dbfff700 1 --2- 172.21.15.70:0/2451818139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a5210 0x55e6c5220230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:29.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.042+0000 7fe1e0912700 1 --2- 172.21.15.70:0/2451818139 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a4860 0x55e6c521fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:29.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.043+0000 7fe1e1113700 1 --2- 172.21.15.70:0/2451818139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 0x55e6c51b3920 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x55e6c51aea10 tx=0x7fe1d4007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7abff8d9b811796 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:29.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.043+0000 7fe1bbfff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:13:29.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.043+0000 7fe1bbfff700 1 -- 172.21.15.70:0/2451818139 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4003e10 con 0x55e6c51a8f60 2024-06-04T05:13:29.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.043+0000 7fe1bbfff700 1 -- 172.21.15.70:0/2451818139 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fe1d4004bb0 con 0x55e6c51a8f60 2024-06-04T05:13:29.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.043+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2451818139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1d00bc070 con 0x55e6c51a8f60 2024-06-04T05:13:29.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.043+0000 7fe1bbfff700 1 -- 172.21.15.70:0/2451818139 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fe1d4004db0 con 0x55e6c51a8f60 2024-06-04T05:13:29.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.043+0000 7fe1ed40bb80 10 client.19734 fetch_fsmap learned FSMap version 47 2024-06-04T05:13:29.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.043+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2451818139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55e6c5228550 con 0x55e6c51a8f60 2024-06-04T05:13:29.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.044+0000 7fe1bbfff700 1 -- 172.21.15.70:0/2451818139 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1d00bc070 con 0x55e6c51a8f60 2024-06-04T05:13:29.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.044+0000 7fe1bbfff700 1 -- 172.21.15.70:0/2451818139 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 47) ==== 8406+0+0 (secure 0 0 0) 0x7fe1d40043a0 con 0x55e6c51a8f60 2024-06-04T05:13:29.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.044+0000 7fe1ed40bb80 10 client.19734 fetch_fsmap finished waiting for FSMap version 47 2024-06-04T05:13:29.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.044+0000 7fe1ed40bb80 10 client.19734 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:29.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.044+0000 7fe1ed40bb80 20 client.19734 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:29.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.044+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/2451818139 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e6c5228d70 0x55e6c522b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:29.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.044+0000 7fe1ed40bb80 4 client.19734 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:13:29.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.044+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2451818139 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e6c5105be0 con 0x55e6c5228d70 2024-06-04T05:13:29.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.044+0000 7fe1e0912700 1 --2- 172.21.15.70:0/2451818139 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e6c5228d70 0x55e6c522b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:29.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.045+0000 7fe1bbfff700 10 client.19734 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:29.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.045+0000 7fe1e0912700 1 --2- 172.21.15.70:0/2451818139 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e6c5228d70 0x55e6c522b190 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=626308711df3f0d4 server_cookie=5c418236b0932e21 in_seq=0 out_seq=0 2024-06-04T05:13:29.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.045+0000 7fe1bbfff700 1 -- 172.21.15.70:0/2451818139 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e6c5105be0 con 0x55e6c5228d70 2024-06-04T05:13:29.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.045+0000 7fe1bbfff700 10 client.19734 handle_command_reply: tid=0 2024-06-04T05:13:29.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.045+0000 7fe1ed40bb80 1 client.19734 shutdown 2024-06-04T05:13:29.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.045+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2451818139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 msgr2=0x55e6c51b3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:29.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.045+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/2451818139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 0x55e6c51b3920 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x55e6c51aea10 tx=0x7fe1d4007f60 comp rx=0 tx=0).stop 2024-06-04T05:13:29.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.045+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2451818139 shutdown_connections 2024-06-04T05:13:29.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.045+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/2451818139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6c51a8f60 0x55e6c51b3920 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.045+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/2451818139 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e6c51a4860 0x55e6c521fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.045+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/2451818139 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e6c5228d70 0x55e6c522b190 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.045+0000 7fe1ed40bb80 1 --2- 172.21.15.70:0/2451818139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6c51a5210 0x55e6c5220230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.045+0000 7fe1bbfff700 0 client.19734 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:29.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.045+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2451818139 >> 172.21.15.70:0/2451818139 conn(0x55e6c50d2080 msgr2=0x55e6c51a8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:29.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.046+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2451818139 shutdown_connections 2024-06-04T05:13:29.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.046+0000 7fe1ed40bb80 1 -- 172.21.15.70:0/2451818139 wait complete. 2024-06-04T05:13:29.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.046+0000 7fe1ed40bb80 20 client.19734 trim_cache size 0 max 16384 2024-06-04T05:13:29.051 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:13:29.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.047+0000 7fe1e2b76700 1 -- 172.21.15.70:0/3525762327 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1c40b2b80 msgr2=0x7fe1c40b5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:29.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.047+0000 7fe1e2b76700 1 --2- 172.21.15.70:0/3525762327 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1c40b2b80 0x7fe1c40b5040 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc006010 tx=0x7fe1cc009e60 comp rx=0 tx=0).stop 2024-06-04T05:13:29.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.047+0000 7fe1e2b76700 1 -- 172.21.15.70:0/3525762327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc102450 msgr2=0x7fe1dc10d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:29.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.047+0000 7fe1e2b76700 1 --2- 172.21.15.70:0/3525762327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc102450 0x7fe1dc10d5a0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000eb30 tx=0x7fe1d000ee40 comp rx=0 tx=0).stop 2024-06-04T05:13:29.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.047+0000 7fe1e2b76700 1 -- 172.21.15.70:0/3525762327 shutdown_connections 2024-06-04T05:13:29.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.047+0000 7fe1e2b76700 1 --2- 172.21.15.70:0/3525762327 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1c40b2b80 0x7fe1c40b5040 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.047+0000 7fe1e2b76700 1 --2- 172.21.15.70:0/3525762327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1dc109610 0x7fe1dc1a49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.047+0000 7fe1e2b76700 1 --2- 172.21.15.70:0/3525762327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1dc101a90 0x7fe1dc10d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.047+0000 7fe1e2b76700 1 --2- 172.21.15.70:0/3525762327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc102450 0x7fe1dc10d5a0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.047+0000 7fe1e2b76700 1 -- 172.21.15.70:0/3525762327 >> 172.21.15.70:0/3525762327 conn(0x7fe1dc078a40 msgr2=0x7fe1dc112050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.047+0000 7fe1e2b76700 1 -- 172.21.15.70:0/3525762327 shutdown_connections 2024-06-04T05:13:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:29.047+0000 7fe1e2b76700 1 -- 172.21.15.70:0/3525762327 wait complete. 2024-06-04T05:13:29.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:29 smithi070 ceph-mon[32631]: pgmap v510: 97 pgs: 97 active+clean; 473 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 254 op/s 2024-06-04T05:13:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:29 smithi112 ceph-mon[36453]: pgmap v510: 97 pgs: 97 active+clean; 473 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 254 op/s 2024-06-04T05:13:29.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:29 smithi098 ceph-mon[35569]: pgmap v510: 97 pgs: 97 active+clean; 473 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 254 op/s 2024-06-04T05:13:29.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 20219 9.95 MB/sec execute 21 sec latency 94.895 ms 2024-06-04T05:13:30.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 20397 9.93 MB/sec execute 22 sec latency 49.197 ms 2024-06-04T05:13:30.887 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:13:30.887 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:13:31.230 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:13:31.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:31 smithi070 ceph-mon[32631]: pgmap v511: 97 pgs: 97 active+clean; 473 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 254 op/s 2024-06-04T05:13:31.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:31 smithi112 ceph-mon[36453]: pgmap v511: 97 pgs: 97 active+clean; 473 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 254 op/s 2024-06-04T05:13:31.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:31 smithi098 ceph-mon[35569]: pgmap v511: 97 pgs: 97 active+clean; 473 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 254 op/s 2024-06-04T05:13:31.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 20534 9.62 MB/sec execute 23 sec latency 207.141 ms 2024-06-04T05:13:32.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 20688 9.46 MB/sec execute 24 sec latency 75.198 ms 2024-06-04T05:13:33.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:33 smithi070 ceph-mon[32631]: pgmap v512: 97 pgs: 97 active+clean; 486 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.6 MiB/s wr, 403 op/s 2024-06-04T05:13:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:33 smithi112 ceph-mon[36453]: pgmap v512: 97 pgs: 97 active+clean; 486 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.6 MiB/s wr, 403 op/s 2024-06-04T05:13:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:33 smithi098 ceph-mon[35569]: pgmap v512: 97 pgs: 97 active+clean; 486 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.6 MiB/s wr, 403 op/s 2024-06-04T05:13:33.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 20851 9.39 MB/sec execute 25 sec latency 143.094 ms 2024-06-04T05:13:34.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 21007 9.32 MB/sec execute 26 sec latency 162.586 ms 2024-06-04T05:13:35.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:35 smithi070 ceph-mon[32631]: pgmap v513: 97 pgs: 97 active+clean; 486 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.9 MiB/s wr, 264 op/s 2024-06-04T05:13:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:35 smithi112 ceph-mon[36453]: pgmap v513: 97 pgs: 97 active+clean; 486 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.9 MiB/s wr, 264 op/s 2024-06-04T05:13:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:35 smithi098 ceph-mon[35569]: pgmap v513: 97 pgs: 97 active+clean; 486 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.9 MiB/s wr, 264 op/s 2024-06-04T05:13:35.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 21151 9.19 MB/sec execute 27 sec latency 128.348 ms 2024-06-04T05:13:36.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:36 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:36.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:36 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:36.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:36 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:36.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 21282 9.18 MB/sec execute 28 sec latency 151.191 ms 2024-06-04T05:13:37.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:37 smithi070 ceph-mon[32631]: pgmap v514: 97 pgs: 97 active+clean; 483 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.8 MiB/s wr, 404 op/s 2024-06-04T05:13:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:37 smithi112 ceph-mon[36453]: pgmap v514: 97 pgs: 97 active+clean; 483 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.8 MiB/s wr, 404 op/s 2024-06-04T05:13:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:37 smithi098 ceph-mon[35569]: pgmap v514: 97 pgs: 97 active+clean; 483 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.8 MiB/s wr, 404 op/s 2024-06-04T05:13:37.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 21431 9.11 MB/sec execute 29 sec latency 142.031 ms 2024-06-04T05:13:38.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 21606 9.17 MB/sec execute 30 sec latency 151.132 ms 2024-06-04T05:13:39.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:39 smithi070 ceph-mon[32631]: pgmap v515: 97 pgs: 97 active+clean; 483 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 288 op/s 2024-06-04T05:13:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:39 smithi112 ceph-mon[36453]: pgmap v515: 97 pgs: 97 active+clean; 483 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 288 op/s 2024-06-04T05:13:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:39 smithi098 ceph-mon[35569]: pgmap v515: 97 pgs: 97 active+clean; 483 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 288 op/s 2024-06-04T05:13:39.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 21753 9.05 MB/sec execute 31 sec latency 151.334 ms 2024-06-04T05:13:40.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.305+0000 7fd37033b700 1 -- 172.21.15.70:0/3862565081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd368108920 msgr2=0x7fd3681104a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:40.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.305+0000 7fd37033b700 1 --2- 172.21.15.70:0/3862565081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd368108920 0x7fd3681104a0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fd36000b0d0 tx=0x7fd36000b3e0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.305+0000 7fd37033b700 1 -- 172.21.15.70:0/3862565081 shutdown_connections 2024-06-04T05:13:40.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.305+0000 7fd37033b700 1 --2- 172.21.15.70:0/3862565081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd368108920 0x7fd3681104a0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.305+0000 7fd37033b700 1 --2- 172.21.15.70:0/3862565081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd368103850 0x7fd368103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.305+0000 7fd37033b700 1 --2- 172.21.15.70:0/3862565081 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd368104200 0x7fd3681081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.305+0000 7fd37033b700 1 -- 172.21.15.70:0/3862565081 >> 172.21.15.70:0/3862565081 conn(0x7fd36807a5d0 msgr2=0x7fd36807a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:40.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.305+0000 7fd37033b700 1 -- 172.21.15.70:0/3862565081 shutdown_connections 2024-06-04T05:13:40.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.305+0000 7fd37033b700 1 -- 172.21.15.70:0/3862565081 wait complete. 2024-06-04T05:13:40.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.306+0000 7fd37033b700 1 Processor -- start 2024-06-04T05:13:40.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.306+0000 7fd37033b700 1 -- start start 2024-06-04T05:13:40.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.306+0000 7fd37033b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd368103850 0x7fd36819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.306+0000 7fd37033b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd368104200 0x7fd36819fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.306+0000 7fd37033b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd368108920 0x7fd368199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.306+0000 7fd37033b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd3681135c0 con 0x7fd368103850 2024-06-04T05:13:40.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.306+0000 7fd37033b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd368113440 con 0x7fd368104200 2024-06-04T05:13:40.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.306+0000 7fd37033b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd368113740 con 0x7fd368108920 2024-06-04T05:13:40.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.306+0000 7fd36e0d7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd368103850 0x7fd36819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.306+0000 7fd36e0d7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd368103850 0x7fd36819f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56808/0 (socket says 172.21.15.70:56808) 2024-06-04T05:13:40.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.306+0000 7fd36e0d7700 1 -- 172.21.15.70:0/2612695344 learned_addr learned my addr 172.21.15.70:0/2612695344 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:40.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.306+0000 7fd36d8d6700 1 --2- 172.21.15.70:0/2612695344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd368104200 0x7fd36819fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.306+0000 7fd36e8d8700 1 --2- 172.21.15.70:0/2612695344 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd368108920 0x7fd368199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.307+0000 7fd36e0d7700 1 -- 172.21.15.70:0/2612695344 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd368108920 msgr2=0x7fd368199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:40.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.307+0000 7fd36e0d7700 1 --2- 172.21.15.70:0/2612695344 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd368108920 0x7fd368199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.307+0000 7fd36e0d7700 1 -- 172.21.15.70:0/2612695344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd368104200 msgr2=0x7fd36819fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:40.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.307+0000 7fd36e0d7700 1 --2- 172.21.15.70:0/2612695344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd368104200 0x7fd36819fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.307+0000 7fd36e0d7700 1 -- 172.21.15.70:0/2612695344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd368199fe0 con 0x7fd368103850 2024-06-04T05:13:40.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.307+0000 7fd36d8d6700 1 --2- 172.21.15.70:0/2612695344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd368104200 0x7fd36819fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:13:40.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.307+0000 7fd36e8d8700 1 --2- 172.21.15.70:0/2612695344 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd368108920 0x7fd368199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:13:40.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.307+0000 7fd36e0d7700 1 --2- 172.21.15.70:0/2612695344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd368103850 0x7fd36819f6c0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fd36400d980 tx=0x7fd36400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:40.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.307+0000 7fd35f7fe700 1 -- 172.21.15.70:0/2612695344 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd364016070 con 0x7fd368103850 2024-06-04T05:13:40.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.307+0000 7fd35f7fe700 1 -- 172.21.15.70:0/2612695344 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd364004020 con 0x7fd368103850 2024-06-04T05:13:40.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.307+0000 7fd35f7fe700 1 -- 172.21.15.70:0/2612695344 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd364004bb0 con 0x7fd368103850 2024-06-04T05:13:40.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.307+0000 7fd37033b700 1 -- 172.21.15.70:0/2612695344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd36819a2d0 con 0x7fd368103850 2024-06-04T05:13:40.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.307+0000 7fd37033b700 1 -- 172.21.15.70:0/2612695344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3681a6590 con 0x7fd368103850 2024-06-04T05:13:40.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.309+0000 7fd35f7fe700 1 -- 172.21.15.70:0/2612695344 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fd36400bb60 con 0x7fd368103850 2024-06-04T05:13:40.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.310+0000 7fd35f7fe700 1 --2- 172.21.15.70:0/2612695344 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd3540b29f0 0x7fd3540b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.310+0000 7fd35f7fe700 1 -- 172.21.15.70:0/2612695344 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fd3640e2540 con 0x7fd368103850 2024-06-04T05:13:40.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.310+0000 7fd36d8d6700 1 --2- 172.21.15.70:0/2612695344 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd3540b29f0 0x7fd3540b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.311+0000 7fd36d8d6700 1 --2- 172.21.15.70:0/2612695344 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd3540b29f0 0x7fd3540b4eb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd358005fd0 tx=0x7fd358005d90 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:40.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.311+0000 7fd37abd0b80 1 Processor -- start 2024-06-04T05:13:40.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.311+0000 7fd37abd0b80 1 -- start start 2024-06-04T05:13:40.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.311+0000 7fd37abd0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97bd8e0 0x55dde97bdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.311+0000 7fd37abd0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97be290 0x55dde97c19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.311+0000 7fd37abd0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97c1fe0 0x55dde97c2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.311+0000 7fd37abd0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dde971cd10 con 0x55dde97bd8e0 2024-06-04T05:13:40.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.311+0000 7fd37abd0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dde970c8b0 con 0x55dde97be290 2024-06-04T05:13:40.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.311+0000 7fd37abd0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55dde97108f0 con 0x55dde97c1fe0 2024-06-04T05:13:40.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.311+0000 7fd36e0d7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97bd8e0 0x55dde97bdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.311+0000 7fd36e0d7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97bd8e0 0x55dde97bdcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56814/0 (socket says 172.21.15.70:56814) 2024-06-04T05:13:40.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.311+0000 7fd36e0d7700 1 -- 172.21.15.70:0/1297939688 learned_addr learned my addr 172.21.15.70:0/1297939688 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:40.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.311+0000 7fd36d8d6700 1 --2- 172.21.15.70:0/1297939688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97be290 0x55dde97c19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.311+0000 7fd36e8d8700 1 --2- 172.21.15.70:0/1297939688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97c1fe0 0x55dde97c2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.312+0000 7fd36e0d7700 1 -- 172.21.15.70:0/1297939688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97c1fe0 msgr2=0x55dde97c2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:40.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.312+0000 7fd36e0d7700 1 --2- 172.21.15.70:0/1297939688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97c1fe0 0x55dde97c2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.312+0000 7fd36e0d7700 1 -- 172.21.15.70:0/1297939688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97be290 msgr2=0x55dde97c19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:40.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.312+0000 7fd36e0d7700 1 --2- 172.21.15.70:0/1297939688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97be290 0x55dde97c19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.312+0000 7fd36e0d7700 1 -- 172.21.15.70:0/1297939688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dde965c950 con 0x55dde97bd8e0 2024-06-04T05:13:40.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.312+0000 7fd36e8d8700 1 --2- 172.21.15.70:0/1297939688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97c1fe0 0x55dde97c2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:13:40.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.312+0000 7fd36e0d7700 1 --2- 172.21.15.70:0/1297939688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97bd8e0 0x55dde97bdcc0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fd3640a5870 tx=0x7fd364004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64987e921cdafba2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:40.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.312+0000 7fd35cd70700 1 -- 172.21.15.70:0/1297939688 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3640a97f0 con 0x55dde97bd8e0 2024-06-04T05:13:40.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.312+0000 7fd35cd70700 1 -- 172.21.15.70:0/1297939688 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd3640adba0 con 0x55dde97bd8e0 2024-06-04T05:13:40.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.312+0000 7fd35cd70700 1 -- 172.21.15.70:0/1297939688 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3640b23c0 con 0x55dde97bd8e0 2024-06-04T05:13:40.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.313+0000 7fd37abd0b80 1 -- 172.21.15.70:0/1297939688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97bd8e0 msgr2=0x55dde97bdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:40.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.313+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/1297939688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97bd8e0 0x55dde97bdcc0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fd3640a5870 tx=0x7fd364004600 comp rx=0 tx=0).stop 2024-06-04T05:13:40.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.313+0000 7fd37abd0b80 1 -- 172.21.15.70:0/1297939688 shutdown_connections 2024-06-04T05:13:40.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.313+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/1297939688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97c1fe0 0x55dde97c2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.313+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/1297939688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97bd8e0 0x55dde97bdcc0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.313+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/1297939688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97be290 0x55dde97c19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.313+0000 7fd37abd0b80 1 -- 172.21.15.70:0/1297939688 >> 172.21.15.70:0/1297939688 conn(0x55dde96eb100 msgr2=0x55dde97b7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:40.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.313+0000 7fd37abd0b80 1 -- 172.21.15.70:0/1297939688 shutdown_connections 2024-06-04T05:13:40.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.313+0000 7fd37abd0b80 1 -- 172.21.15.70:0/1297939688 wait complete. 2024-06-04T05:13:40.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.313+0000 7fd37abd0b80 1 -- start start 2024-06-04T05:13:40.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.313+0000 7fd37abd0b80 1 Processor -- start 2024-06-04T05:13:40.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.313+0000 7fd37abd0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97bd8e0 0x55dde97c9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.313+0000 7fd37abd0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97be290 0x55dde97c6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.313+0000 7fd37abd0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97c1fe0 0x55dde97c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.313+0000 7fd37abd0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dde9613020 con 0x55dde97be290 2024-06-04T05:13:40.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.313+0000 7fd37abd0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dde96ae840 con 0x55dde97bd8e0 2024-06-04T05:13:40.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.313+0000 7fd37abd0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55dde97ca140 con 0x55dde97c1fe0 2024-06-04T05:13:40.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd37abd0b80 10 client.0 _ll_register_callbacks cb 0x55dde96a79a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:40.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd36d8d6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97be290 0x55dde97c6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd36d8d6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97be290 0x55dde97c6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56828/0 (socket says 172.21.15.70:56828) 2024-06-04T05:13:40.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd36d8d6700 1 -- 172.21.15.70:0/1877211956 learned_addr learned my addr 172.21.15.70:0/1877211956 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:40.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd36e0d7700 1 --2- 172.21.15.70:0/1877211956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97bd8e0 0x55dde97c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd36e8d8700 1 --2- 172.21.15.70:0/1877211956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97c1fe0 0x55dde97c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd36d8d6700 1 -- 172.21.15.70:0/1877211956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97c1fe0 msgr2=0x55dde97c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:40.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd36d8d6700 1 --2- 172.21.15.70:0/1877211956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97c1fe0 0x55dde97c6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd36d8d6700 1 -- 172.21.15.70:0/1877211956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97bd8e0 msgr2=0x55dde97c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:40.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd36d8d6700 1 --2- 172.21.15.70:0/1877211956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97bd8e0 0x55dde97c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd36d8d6700 1 -- 172.21.15.70:0/1877211956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dde96ab710 con 0x55dde97be290 2024-06-04T05:13:40.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd36d8d6700 1 --2- 172.21.15.70:0/1877211956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97be290 0x55dde97c6850 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fd358010760 tx=0x7fd358004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b239e88526c6ceb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:40.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd351ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:13:40.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd351ffb700 1 -- 172.21.15.70:0/1877211956 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd358004a30 con 0x55dde97be290 2024-06-04T05:13:40.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd351ffb700 1 -- 172.21.15.70:0/1877211956 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fd358004bd0 con 0x55dde97be290 2024-06-04T05:13:40.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd37abd0b80 1 -- 172.21.15.70:0/1877211956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3640adba0 con 0x55dde97be290 2024-06-04T05:13:40.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd351ffb700 1 -- 172.21.15.70:0/1877211956 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fd358004dd0 con 0x55dde97be290 2024-06-04T05:13:40.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd37abd0b80 10 client.19746 fetch_fsmap learned FSMap version 48 2024-06-04T05:13:40.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.314+0000 7fd37abd0b80 1 -- 172.21.15.70:0/1877211956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55dde97d49d0 con 0x55dde97be290 2024-06-04T05:13:40.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.315+0000 7fd351ffb700 1 -- 172.21.15.70:0/1877211956 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd3640adba0 con 0x55dde97be290 2024-06-04T05:13:40.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.315+0000 7fd351ffb700 1 -- 172.21.15.70:0/1877211956 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 48) ==== 8398+0+0 (secure 0 0 0) 0x7fd35801e9c0 con 0x55dde97be290 2024-06-04T05:13:40.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.315+0000 7fd37abd0b80 10 client.19746 fetch_fsmap finished waiting for FSMap version 48 2024-06-04T05:13:40.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.315+0000 7fd37abd0b80 10 client.19746 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:40.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.315+0000 7fd37abd0b80 20 client.19746 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:40.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.315+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/1877211956 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dde97d7010 0x55dde97d9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.315+0000 7fd37abd0b80 4 client.19746 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:13:40.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.315+0000 7fd37abd0b80 1 -- 172.21.15.70:0/1877211956 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dde9789a60 con 0x55dde97d7010 2024-06-04T05:13:40.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.315+0000 7fd36e0d7700 1 --2- 172.21.15.70:0/1877211956 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dde97d7010 0x55dde97d9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.315+0000 7fd351ffb700 10 client.19746 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:40.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.315+0000 7fd36e0d7700 1 --2- 172.21.15.70:0/1877211956 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dde97d7010 0x55dde97d9400 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cde1c5898b5bd9eb server_cookie=f37a32316905a2b1 in_seq=0 out_seq=0 2024-06-04T05:13:40.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.316+0000 7fd351ffb700 1 -- 172.21.15.70:0/1877211956 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55dde9789a60 con 0x55dde97d7010 2024-06-04T05:13:40.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.316+0000 7fd351ffb700 10 client.19746 handle_command_reply: tid=0 2024-06-04T05:13:40.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.316+0000 7fd37abd0b80 1 client.19746 shutdown 2024-06-04T05:13:40.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.316+0000 7fd37abd0b80 1 -- 172.21.15.70:0/1877211956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97be290 msgr2=0x55dde97c6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:40.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.316+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/1877211956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97be290 0x55dde97c6850 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fd358010760 tx=0x7fd358004750 comp rx=0 tx=0).stop 2024-06-04T05:13:40.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.316+0000 7fd37abd0b80 1 -- 172.21.15.70:0/1877211956 shutdown_connections 2024-06-04T05:13:40.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.316+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/1877211956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97c1fe0 0x55dde97c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.316+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/1877211956 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dde97d7010 0x55dde97d9400 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.316+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/1877211956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97bd8e0 0x55dde97c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.316+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/1877211956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97be290 0x55dde97c6850 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.316+0000 7fd351ffb700 0 client.19746 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:40.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.316+0000 7fd37abd0b80 1 -- 172.21.15.70:0/1877211956 >> 172.21.15.70:0/1877211956 conn(0x55dde96eb100 msgr2=0x55dde97b7590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:40.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.316+0000 7fd37abd0b80 1 -- 172.21.15.70:0/1877211956 shutdown_connections 2024-06-04T05:13:40.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.316+0000 7fd37abd0b80 1 -- 172.21.15.70:0/1877211956 wait complete. 2024-06-04T05:13:40.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.316+0000 7fd37abd0b80 20 client.19746 trim_cache size 0 max 16384 2024-06-04T05:13:40.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.337+0000 7fd37abd0b80 1 Processor -- start 2024-06-04T05:13:40.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.337+0000 7fd37abd0b80 1 -- start start 2024-06-04T05:13:40.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.337+0000 7fd37abd0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97bd8e0 0x55dde97d31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.337+0000 7fd37abd0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97be290 0x55dde97c1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.337+0000 7fd37abd0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97c1fe0 0x55dde97dec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.337+0000 7fd37abd0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd3681adde0 con 0x55dde97c1fe0 2024-06-04T05:13:40.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.337+0000 7fd37abd0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd3681afad0 con 0x55dde97be290 2024-06-04T05:13:40.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.337+0000 7fd37abd0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd3681ae510 con 0x55dde97bd8e0 2024-06-04T05:13:40.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.337+0000 7fd36e8d8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97c1fe0 0x55dde97dec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.337+0000 7fd36e8d8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97c1fe0 0x55dde97dec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56838/0 (socket says 172.21.15.70:56838) 2024-06-04T05:13:40.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.337+0000 7fd36e8d8700 1 -- 172.21.15.70:0/3800359053 learned_addr learned my addr 172.21.15.70:0/3800359053 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:40.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.338+0000 7fd36e0d7700 1 --2- 172.21.15.70:0/3800359053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97bd8e0 0x55dde97d31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.338+0000 7fd36d8d6700 1 --2- 172.21.15.70:0/3800359053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97be290 0x55dde97c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.338+0000 7fd36e8d8700 1 -- 172.21.15.70:0/3800359053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97bd8e0 msgr2=0x55dde97d31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:40.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.338+0000 7fd36e8d8700 1 --2- 172.21.15.70:0/3800359053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97bd8e0 0x55dde97d31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.338+0000 7fd36e8d8700 1 -- 172.21.15.70:0/3800359053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97be290 msgr2=0x55dde97c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:40.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.338+0000 7fd36e8d8700 1 --2- 172.21.15.70:0/3800359053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97be290 0x55dde97c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.338+0000 7fd36e8d8700 1 -- 172.21.15.70:0/3800359053 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dde97c6620 con 0x55dde97c1fe0 2024-06-04T05:13:40.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.338+0000 7fd36d8d6700 1 --2- 172.21.15.70:0/3800359053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97be290 0x55dde97c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:13:40.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.338+0000 7fd36e8d8700 1 --2- 172.21.15.70:0/3800359053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97c1fe0 0x55dde97dec80 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fd360009290 tx=0x7fd3600092c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e47138bef8c7e0ac server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:40.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.338+0000 7fd353fff700 1 -- 172.21.15.70:0/3800359053 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd360004c50 con 0x55dde97c1fe0 2024-06-04T05:13:40.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.338+0000 7fd353fff700 1 -- 172.21.15.70:0/3800359053 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd360004df0 con 0x55dde97c1fe0 2024-06-04T05:13:40.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.338+0000 7fd353fff700 1 -- 172.21.15.70:0/3800359053 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3600278b0 con 0x55dde97c1fe0 2024-06-04T05:13:40.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.339+0000 7fd37abd0b80 1 -- 172.21.15.70:0/3800359053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97c1fe0 msgr2=0x55dde97dec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:40.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.339+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/3800359053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97c1fe0 0x55dde97dec80 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fd360009290 tx=0x7fd3600092c0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.339+0000 7fd37abd0b80 1 -- 172.21.15.70:0/3800359053 shutdown_connections 2024-06-04T05:13:40.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.339+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/3800359053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97c1fe0 0x55dde97dec80 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.339+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/3800359053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97bd8e0 0x55dde97d31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.339+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/3800359053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97be290 0x55dde97c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.339+0000 7fd37abd0b80 1 -- 172.21.15.70:0/3800359053 >> 172.21.15.70:0/3800359053 conn(0x55dde96eb100 msgr2=0x55dde97c0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:40.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.339+0000 7fd37abd0b80 1 -- 172.21.15.70:0/3800359053 shutdown_connections 2024-06-04T05:13:40.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.339+0000 7fd37abd0b80 1 -- 172.21.15.70:0/3800359053 wait complete. 2024-06-04T05:13:40.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.339+0000 7fd37abd0b80 1 -- start start 2024-06-04T05:13:40.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.339+0000 7fd37abd0b80 1 Processor -- start 2024-06-04T05:13:40.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd37abd0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97bd8e0 0x55dde97ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd37abd0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97be290 0x55dde97ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd37abd0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97c1fe0 0x55dde983bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd37abd0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd3681af340 con 0x55dde97c1fe0 2024-06-04T05:13:40.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd37abd0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd3681aec40 con 0x55dde97be290 2024-06-04T05:13:40.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd37abd0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd3681abda0 con 0x55dde97bd8e0 2024-06-04T05:13:40.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd37abd0b80 10 client.0 _ll_register_callbacks cb 0x55dde96a79a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:40.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd36e8d8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97c1fe0 0x55dde983bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd36e0d7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97bd8e0 0x55dde97ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd36e8d8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97c1fe0 0x55dde983bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56850/0 (socket says 172.21.15.70:56850) 2024-06-04T05:13:40.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd36e8d8700 1 -- 172.21.15.70:0/3552010507 learned_addr learned my addr 172.21.15.70:0/3552010507 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:40.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd36d8d6700 1 --2- 172.21.15.70:0/3552010507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97be290 0x55dde97ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd36e8d8700 1 -- 172.21.15.70:0/3552010507 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97bd8e0 msgr2=0x55dde97ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:40.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd36e8d8700 1 --2- 172.21.15.70:0/3552010507 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97bd8e0 0x55dde97ce9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd36e8d8700 1 -- 172.21.15.70:0/3552010507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97be290 msgr2=0x55dde97ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:40.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd36e8d8700 1 --2- 172.21.15.70:0/3552010507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97be290 0x55dde97ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd36e8d8700 1 -- 172.21.15.70:0/3552010507 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dde983c180 con 0x55dde97c1fe0 2024-06-04T05:13:40.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd36e0d7700 1 --2- 172.21.15.70:0/3552010507 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97bd8e0 0x55dde97ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:40.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd36e8d8700 1 --2- 172.21.15.70:0/3552010507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97c1fe0 0x55dde983bb40 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fd360009260 tx=0x7fd3600074a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=493091fe82c0c0bd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:40.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd351ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:13:40.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd351ffb700 1 -- 172.21.15.70:0/3552010507 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3600079d0 con 0x55dde97c1fe0 2024-06-04T05:13:40.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd37abd0b80 1 -- 172.21.15.70:0/3552010507 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd360004df0 con 0x55dde97c1fe0 2024-06-04T05:13:40.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd351ffb700 1 -- 172.21.15.70:0/3552010507 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fd360007b70 con 0x55dde97c1fe0 2024-06-04T05:13:40.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.340+0000 7fd351ffb700 1 -- 172.21.15.70:0/3552010507 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fd360027510 con 0x55dde97c1fe0 2024-06-04T05:13:40.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.341+0000 7fd37abd0b80 10 client.19752 fetch_fsmap learned FSMap version 48 2024-06-04T05:13:40.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.341+0000 7fd37abd0b80 1 -- 172.21.15.70:0/3552010507 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55dde98434c0 con 0x55dde97c1fe0 2024-06-04T05:13:40.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.341+0000 7fd351ffb700 1 -- 172.21.15.70:0/3552010507 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd360004df0 con 0x55dde97c1fe0 2024-06-04T05:13:40.255 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:13:40.255 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:13:40.255 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "89905956-13a2-4e20-8138-06079142a8d9", 2024-06-04T05:13:40.255 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:13:40.255 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:13:40.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.341+0000 7fd351ffb700 1 -- 172.21.15.70:0/3552010507 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 48) ==== 8398+0+0 (secure 0 0 0) 0x7fd360037070 con 0x55dde97c1fe0 2024-06-04T05:13:40.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.341+0000 7fd37abd0b80 10 client.19752 fetch_fsmap finished waiting for FSMap version 48 2024-06-04T05:13:40.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.341+0000 7fd37abd0b80 10 client.19752 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:40.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.341+0000 7fd37abd0b80 20 client.19752 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:40.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.341+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/3552010507 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dde9843b20 0x55dde9845f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:40.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.341+0000 7fd37abd0b80 4 client.19752 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:13:40.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.341+0000 7fd37abd0b80 1 -- 172.21.15.70:0/3552010507 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55dde971ec60 con 0x55dde9843b20 2024-06-04T05:13:40.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.341+0000 7fd36e0d7700 1 --2- 172.21.15.70:0/3552010507 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dde9843b20 0x55dde9845f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:40.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.342+0000 7fd351ffb700 10 client.19752 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:40.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.342+0000 7fd36e0d7700 1 --2- 172.21.15.70:0/3552010507 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dde9843b20 0x55dde9845f40 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f091a0560f31a0c server_cookie=f99b8bbebf7cf462 in_seq=0 out_seq=0 2024-06-04T05:13:40.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.343+0000 7fd351ffb700 1 -- 172.21.15.70:0/3552010507 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dde971ec60 con 0x55dde9843b20 2024-06-04T05:13:40.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.343+0000 7fd351ffb700 10 client.19752 handle_command_reply: tid=0 2024-06-04T05:13:40.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.343+0000 7fd37abd0b80 1 client.19752 shutdown 2024-06-04T05:13:40.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.343+0000 7fd37abd0b80 1 -- 172.21.15.70:0/3552010507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97c1fe0 msgr2=0x55dde983bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:40.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.343+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/3552010507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97c1fe0 0x55dde983bb40 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fd360009260 tx=0x7fd3600074a0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.343+0000 7fd37abd0b80 1 -- 172.21.15.70:0/3552010507 shutdown_connections 2024-06-04T05:13:40.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.343+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/3552010507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dde97c1fe0 0x55dde983bb40 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.343+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/3552010507 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dde97bd8e0 0x55dde97ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.343+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/3552010507 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dde9843b20 0x55dde9845f40 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.343+0000 7fd37abd0b80 1 --2- 172.21.15.70:0/3552010507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dde97be290 0x55dde97ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.343+0000 7fd37abd0b80 1 -- 172.21.15.70:0/3552010507 >> 172.21.15.70:0/3552010507 conn(0x55dde96eb100 msgr2=0x55dde97c11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.343+0000 7fd351ffb700 0 client.19752 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.344+0000 7fd37abd0b80 1 -- 172.21.15.70:0/3552010507 shutdown_connections 2024-06-04T05:13:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.344+0000 7fd37abd0b80 1 -- 172.21.15.70:0/3552010507 wait complete. 2024-06-04T05:13:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.344+0000 7fd37abd0b80 20 client.19752 trim_cache size 0 max 16384 2024-06-04T05:13:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.345+0000 7fd37033b700 1 -- 172.21.15.70:0/2612695344 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd3540b29f0 msgr2=0x7fd3540b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.345+0000 7fd37033b700 1 --2- 172.21.15.70:0/2612695344 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd3540b29f0 0x7fd3540b4eb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd358005fd0 tx=0x7fd358005d90 comp rx=0 tx=0).stop 2024-06-04T05:13:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.345+0000 7fd37033b700 1 -- 172.21.15.70:0/2612695344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd368103850 msgr2=0x7fd36819f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.345+0000 7fd37033b700 1 --2- 172.21.15.70:0/2612695344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd368103850 0x7fd36819f6c0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fd36400d980 tx=0x7fd36400dd40 comp rx=0 tx=0).stop 2024-06-04T05:13:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.345+0000 7fd37033b700 1 -- 172.21.15.70:0/2612695344 shutdown_connections 2024-06-04T05:13:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.345+0000 7fd37033b700 1 --2- 172.21.15.70:0/2612695344 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd3540b29f0 0x7fd3540b4eb0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.345+0000 7fd37033b700 1 --2- 172.21.15.70:0/2612695344 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd368108920 0x7fd368199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.345+0000 7fd37033b700 1 --2- 172.21.15.70:0/2612695344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd368103850 0x7fd36819f6c0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.345+0000 7fd37033b700 1 --2- 172.21.15.70:0/2612695344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd368104200 0x7fd36819fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:40.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.345+0000 7fd37033b700 1 -- 172.21.15.70:0/2612695344 >> 172.21.15.70:0/2612695344 conn(0x7fd36807a5d0 msgr2=0x7fd3681018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:40.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.346+0000 7fd37033b700 1 -- 172.21.15.70:0/2612695344 shutdown_connections 2024-06-04T05:13:40.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:39.346+0000 7fd37033b700 1 -- 172.21.15.70:0/2612695344 wait complete. 2024-06-04T05:13:40.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:13:40.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:13:40.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:13:40.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:13:40.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:13:40.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:13:40.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:13:40.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:13:40.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:13:40.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:13:40.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:13:40.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:13:40.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:13:40.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:13:40.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:40 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:13:40.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:13:40.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:13:40.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:13:40.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:13:40.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:13:40.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:13:40.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:13:40.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:13:40.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:13:40.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:13:40.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:13:40.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:13:40.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:13:40.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:40 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:13:40.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:13:40.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:13:40.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:13:40.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:13:40.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:13:40.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:13:40.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:13:40.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:13:40.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:13:40.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:13:40.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:13:40.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:13:40.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:13:40.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:40 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:13:40.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 21901 8.85 MB/sec execute 32 sec latency 144.748 ms 2024-06-04T05:13:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.556+0000 7fc89fb2c640 1 -- 172.21.15.70:0/2930762622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc898106960 msgr2=0x7fc898106d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.556+0000 7fc89fb2c640 1 --2- 172.21.15.70:0/2930762622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc898106960 0x7fc898106d40 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fc894009900 tx=0x7fc89402a410 comp rx=0 tx=0).stop 2024-06-04T05:13:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.557+0000 7fc89fb2c640 1 -- 172.21.15.70:0/2930762622 shutdown_connections 2024-06-04T05:13:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.557+0000 7fc89fb2c640 1 --2- 172.21.15.70:0/2930762622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc898102080 0x7fc898102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.557+0000 7fc89fb2c640 1 --2- 172.21.15.70:0/2930762622 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc898102a40 0x7fc89810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.557+0000 7fc89fb2c640 1 --2- 172.21.15.70:0/2930762622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc898106960 0x7fc898106d40 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.557+0000 7fc89fb2c640 1 -- 172.21.15.70:0/2930762622 >> 172.21.15.70:0/2930762622 conn(0x7fc898077ee0 msgr2=0x7fc8980ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.557+0000 7fc89fb2c640 1 -- 172.21.15.70:0/2930762622 shutdown_connections 2024-06-04T05:13:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.557+0000 7fc89fb2c640 1 -- 172.21.15.70:0/2930762622 wait complete. 2024-06-04T05:13:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.558+0000 7fc89fb2c640 1 Processor -- start 2024-06-04T05:13:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.558+0000 7fc89fb2c640 1 -- start start 2024-06-04T05:13:41.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.559+0000 7fc89fb2c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc898102080 0x7fc898199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.559+0000 7fc89fb2c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc898102a40 0x7fc898199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.559+0000 7fc89fb2c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc898106960 0x7fc89819d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.559+0000 7fc89fb2c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc898112ba0 con 0x7fc898102080 2024-06-04T05:13:41.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.559+0000 7fc89fb2c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc898112a20 con 0x7fc898102a40 2024-06-04T05:13:41.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.559+0000 7fc89d8c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc898102080 0x7fc898199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.559+0000 7fc89fb2c640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc898112d20 con 0x7fc898106960 2024-06-04T05:13:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.559+0000 7fc89d8c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc898102080 0x7fc898199020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56874/0 (socket says 172.21.15.70:56874) 2024-06-04T05:13:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.559+0000 7fc89d8c8640 1 -- 172.21.15.70:0/3639975913 learned_addr learned my addr 172.21.15.70:0/3639975913 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.559+0000 7fc89e0c9640 1 --2- 172.21.15.70:0/3639975913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc898106960 0x7fc89819d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.559+0000 7fc89d0c7640 1 --2- 172.21.15.70:0/3639975913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc898102a40 0x7fc898199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.559+0000 7fc89d8c8640 1 -- 172.21.15.70:0/3639975913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc898106960 msgr2=0x7fc89819d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.559+0000 7fc89d8c8640 1 --2- 172.21.15.70:0/3639975913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc898106960 0x7fc89819d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.559+0000 7fc89d8c8640 1 -- 172.21.15.70:0/3639975913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc898102a40 msgr2=0x7fc898199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.559+0000 7fc89d8c8640 1 --2- 172.21.15.70:0/3639975913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc898102a40 0x7fc898199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.559+0000 7fc89d8c8640 1 -- 172.21.15.70:0/3639975913 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc89819e010 con 0x7fc898102080 2024-06-04T05:13:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.559+0000 7fc89d0c7640 1 --2- 172.21.15.70:0/3639975913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc898102a40 0x7fc898199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.560+0000 7fc89d8c8640 1 --2- 172.21.15.70:0/3639975913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc898102080 0x7fc898199020 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fc894006010 tx=0x7fc894032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.560+0000 7fc88effd640 1 -- 172.21.15.70:0/3639975913 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc89403f030 con 0x7fc898102080 2024-06-04T05:13:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.560+0000 7fc88effd640 1 -- 172.21.15.70:0/3639975913 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc894004450 con 0x7fc898102080 2024-06-04T05:13:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.560+0000 7fc88effd640 1 -- 172.21.15.70:0/3639975913 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc894035070 con 0x7fc898102080 2024-06-04T05:13:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.560+0000 7fc89fb2c640 1 -- 172.21.15.70:0/3639975913 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc89819e2a0 con 0x7fc898102080 2024-06-04T05:13:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.560+0000 7fc89fb2c640 1 -- 172.21.15.70:0/3639975913 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc89810f1b0 con 0x7fc898102080 2024-06-04T05:13:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.564+0000 7fc88effd640 1 -- 172.21.15.70:0/3639975913 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fc894042030 con 0x7fc898102080 2024-06-04T05:13:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.564+0000 7fc8a2845740 1 Processor -- start 2024-06-04T05:13:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.564+0000 7fc8a2845740 1 -- start start 2024-06-04T05:13:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.564+0000 7fc8a2845740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 0x55c81661eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.564+0000 7fc8a2845740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c81661f080 0x55c816631cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.564+0000 7fc8a2845740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c816632210 0x55c816634600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.564+0000 7fc8a2845740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c816590c30 con 0x55c816632210 2024-06-04T05:13:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.564+0000 7fc8a2845740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c8165807d0 con 0x55c81661f080 2024-06-04T05:13:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.564+0000 7fc8a2845740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c816584810 con 0x55c81661e760 2024-06-04T05:13:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.564+0000 7fc89e0c9640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 0x55c81661eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.564+0000 7fc89d8c8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c81661f080 0x55c816631cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.564+0000 7fc89e0c9640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 0x55c81661eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:38546/0 (socket says 172.21.15.70:38546) 2024-06-04T05:13:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.564+0000 7fc89e0c9640 1 -- 172.21.15.70:0/4122585728 learned_addr learned my addr 172.21.15.70:0/4122585728 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.564+0000 7fc89d0c7640 1 --2- 172.21.15.70:0/4122585728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c816632210 0x55c816634600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.565+0000 7fc89e0c9640 1 -- 172.21.15.70:0/4122585728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c81661f080 msgr2=0x55c816631cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.565+0000 7fc89e0c9640 1 --2- 172.21.15.70:0/4122585728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c81661f080 0x55c816631cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.565+0000 7fc89e0c9640 1 -- 172.21.15.70:0/4122585728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c816632210 msgr2=0x55c816634600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.565+0000 7fc89e0c9640 1 --2- 172.21.15.70:0/4122585728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c816632210 0x55c816634600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.565+0000 7fc89e0c9640 1 -- 172.21.15.70:0/4122585728 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c816295210 con 0x55c81661e760 2024-06-04T05:13:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.565+0000 7fc89d0c7640 1 --2- 172.21.15.70:0/4122585728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c816632210 0x55c816634600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:13:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.565+0000 7fc89e0c9640 1 --2- 172.21.15.70:0/4122585728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 0x55c81661eb40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc890009fd0 tx=0x7fc890009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70f74aae32b45c6a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.565+0000 7fc883fff640 1 -- 172.21.15.70:0/4122585728 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc890004d20 con 0x55c81661e760 2024-06-04T05:13:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.565+0000 7fc883fff640 1 -- 172.21.15.70:0/4122585728 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc890003e00 con 0x55c81661e760 2024-06-04T05:13:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.566+0000 7fc88effd640 1 --2- 172.21.15.70:0/3639975913 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc8840b2a90 0x7fc8840b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.566+0000 7fc883fff640 1 -- 172.21.15.70:0/4122585728 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc890005020 con 0x55c81661e760 2024-06-04T05:13:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.566+0000 7fc89d8c8640 1 --2- 172.21.15.70:0/3639975913 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc8840b2a90 0x7fc8840b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.566+0000 7fc88effd640 1 -- 172.21.15.70:0/3639975913 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fc894102cb0 con 0x7fc898102080 2024-06-04T05:13:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.566+0000 7fc8a2845740 1 -- 172.21.15.70:0/4122585728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 msgr2=0x55c81661eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.566+0000 7fc8a2845740 1 --2- 172.21.15.70:0/4122585728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 0x55c81661eb40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc890009fd0 tx=0x7fc890009500 comp rx=0 tx=0).stop 2024-06-04T05:13:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.567+0000 7fc8a2845740 1 -- 172.21.15.70:0/4122585728 shutdown_connections 2024-06-04T05:13:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.567+0000 7fc8a2845740 1 --2- 172.21.15.70:0/4122585728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c81661f080 0x55c816631cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.567+0000 7fc8a2845740 1 --2- 172.21.15.70:0/4122585728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c816632210 0x55c816634600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.567+0000 7fc8a2845740 1 --2- 172.21.15.70:0/4122585728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 0x55c81661eb40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.567+0000 7fc8a2845740 1 -- 172.21.15.70:0/4122585728 >> 172.21.15.70:0/4122585728 conn(0x55c81661ad30 msgr2=0x55c81661b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.567+0000 7fc89d8c8640 1 --2- 172.21.15.70:0/3639975913 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc8840b2a90 0x7fc8840b4f50 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc8940c9f40 tx=0x7fc894106760 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.567+0000 7fc8a2845740 1 -- 172.21.15.70:0/4122585728 shutdown_connections 2024-06-04T05:13:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.567+0000 7fc8a2845740 1 -- 172.21.15.70:0/4122585728 wait complete. 2024-06-04T05:13:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.567+0000 7fc8a2845740 1 -- start start 2024-06-04T05:13:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.567+0000 7fc8a2845740 1 Processor -- start 2024-06-04T05:13:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.568+0000 7fc8a2845740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 0x55c81662f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.568+0000 7fc8a2845740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 0x55c81662f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.568+0000 7fc8a2845740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c816632210 0x55c81662fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.568+0000 7fc8a2845740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c816266790 con 0x55c81661f080 2024-06-04T05:13:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.568+0000 7fc8a2845740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c81653c890 con 0x55c816632210 2024-06-04T05:13:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.568+0000 7fc8a2845740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c816634e80 con 0x55c81661e760 2024-06-04T05:13:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.568+0000 7fc8a2845740 10 client.0 _ll_register_callbacks cb 0x55c816294a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.568+0000 7fc89e0c9640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 0x55c81662f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.568+0000 7fc89e0c9640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 0x55c81662f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:38558/0 (socket says 172.21.15.70:38558) 2024-06-04T05:13:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.568+0000 7fc89e0c9640 1 -- 172.21.15.70:0/4231459927 learned_addr learned my addr 172.21.15.70:0/4231459927 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.568+0000 7fc89d0c7640 1 --2- 172.21.15.70:0/4231459927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 0x55c81662f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.569+0000 7fc89e0c9640 1 --2- 172.21.15.70:0/4231459927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c816632210 0x55c81662fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.569+0000 7fc89e0c9640 1 -- 172.21.15.70:0/4231459927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c816632210 msgr2=0x55c81662fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.569+0000 7fc89e0c9640 1 --2- 172.21.15.70:0/4231459927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c816632210 0x55c81662fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.569+0000 7fc89e0c9640 1 -- 172.21.15.70:0/4231459927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 msgr2=0x55c81662f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.569+0000 7fc89e0c9640 1 --2- 172.21.15.70:0/4231459927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 0x55c81662f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.569+0000 7fc89e0c9640 1 -- 172.21.15.70:0/4231459927 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c816361620 con 0x55c81661e760 2024-06-04T05:13:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.569+0000 7fc89d0c7640 1 --2- 172.21.15.70:0/4231459927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 0x55c81662f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.569+0000 7fc89e0c9640 1 --2- 172.21.15.70:0/4231459927 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 0x55c81662f120 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc890009de0 tx=0x7fc890009be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4b53906f894de2b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.569+0000 7fc8817fa640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:13:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.569+0000 7fc8817fa640 1 -- 172.21.15.70:0/4231459927 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc890005040 con 0x55c81661e760 2024-06-04T05:13:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.569+0000 7fc8a2845740 1 -- 172.21.15.70:0/4231459927 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc890003e00 con 0x55c81661e760 2024-06-04T05:13:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.569+0000 7fc8817fa640 1 -- 172.21.15.70:0/4231459927 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fc8900051e0 con 0x55c81661e760 2024-06-04T05:13:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.570+0000 7fc8817fa640 1 -- 172.21.15.70:0/4231459927 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fc8900053e0 con 0x55c81661e760 2024-06-04T05:13:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.570+0000 7fc8a2845740 10 client.28714 fetch_fsmap learned FSMap version 48 2024-06-04T05:13:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.570+0000 7fc8a2845740 1 -- 172.21.15.70:0/4231459927 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55c816640a70 con 0x55c81661e760 2024-06-04T05:13:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.570+0000 7fc8817fa640 1 -- 172.21.15.70:0/4231459927 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc890003e00 con 0x55c81661e760 2024-06-04T05:13:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.570+0000 7fc8817fa640 1 -- 172.21.15.70:0/4231459927 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 48) ==== 8398+0+0 (secure 0 0 0) 0x7fc8900197b0 con 0x55c81661e760 2024-06-04T05:13:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.570+0000 7fc8a2845740 10 client.28714 fetch_fsmap finished waiting for FSMap version 48 2024-06-04T05:13:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.570+0000 7fc8a2845740 10 client.28714 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.570+0000 7fc8a2845740 20 client.28714 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.570+0000 7fc8a2845740 1 --2- 172.21.15.70:0/4231459927 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c816641ee0 0x55c8166442d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.570+0000 7fc8a2845740 4 client.28714 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:13:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.570+0000 7fc8a2845740 1 -- 172.21.15.70:0/4231459927 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c816592b80 con 0x55c816641ee0 2024-06-04T05:13:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.570+0000 7fc89d0c7640 1 --2- 172.21.15.70:0/4231459927 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c816641ee0 0x55c8166442d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.571+0000 7fc8817fa640 10 client.28714 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.571+0000 7fc89d0c7640 1 --2- 172.21.15.70:0/4231459927 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c816641ee0 0x55c8166442d0 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc3ee9da1bdd36f2 server_cookie=297e86eebd9fba4d in_seq=0 out_seq=0 2024-06-04T05:13:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.571+0000 7fc8817fa640 1 -- 172.21.15.70:0/4231459927 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c816592b80 con 0x55c816641ee0 2024-06-04T05:13:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.571+0000 7fc8817fa640 10 client.28714 handle_command_reply: tid=0 2024-06-04T05:13:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.571+0000 7fc8a2845740 1 client.28714 shutdown 2024-06-04T05:13:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.572+0000 7fc8a2845740 1 -- 172.21.15.70:0/4231459927 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 msgr2=0x55c81662f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.572+0000 7fc8a2845740 1 --2- 172.21.15.70:0/4231459927 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 0x55c81662f120 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc890009de0 tx=0x7fc890009be0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.572+0000 7fc8a2845740 1 -- 172.21.15.70:0/4231459927 shutdown_connections 2024-06-04T05:13:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.572+0000 7fc8a2845740 1 --2- 172.21.15.70:0/4231459927 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c816641ee0 0x55c8166442d0 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.572+0000 7fc8a2845740 1 --2- 172.21.15.70:0/4231459927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c816632210 0x55c81662fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.572+0000 7fc8817fa640 0 client.28714 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.572+0000 7fc8a2845740 1 --2- 172.21.15.70:0/4231459927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 0x55c81662f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.572+0000 7fc8a2845740 1 --2- 172.21.15.70:0/4231459927 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 0x55c81662f120 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.572+0000 7fc8a2845740 1 -- 172.21.15.70:0/4231459927 >> 172.21.15.70:0/4231459927 conn(0x55c81661ad30 msgr2=0x55c816620ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:41.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.572+0000 7fc8a2845740 1 -- 172.21.15.70:0/4231459927 shutdown_connections 2024-06-04T05:13:41.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.572+0000 7fc8a2845740 1 -- 172.21.15.70:0/4231459927 wait complete. 2024-06-04T05:13:41.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.572+0000 7fc8a2845740 20 client.28714 trim_cache size 0 max 16384 2024-06-04T05:13:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.592+0000 7fc8a2845740 1 Processor -- start 2024-06-04T05:13:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.592+0000 7fc8a2845740 1 -- start start 2024-06-04T05:13:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc8a2845740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c81661e760 0x55c816649930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc8a2845740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 0x55c816649e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc8a2845740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c816632210 0x55c81662c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc8a2845740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc8981ac400 con 0x55c81661f080 2024-06-04T05:13:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc8a2845740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc8981ae0f0 con 0x55c81661e760 2024-06-04T05:13:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc8a2845740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc8981acb30 con 0x55c816632210 2024-06-04T05:13:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc89d0c7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 0x55c816649e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc89d0c7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 0x55c816649e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56896/0 (socket says 172.21.15.70:56896) 2024-06-04T05:13:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc89d0c7640 1 -- 172.21.15.70:0/2495648616 learned_addr learned my addr 172.21.15.70:0/2495648616 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc89e0c9640 1 --2- 172.21.15.70:0/2495648616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c81661e760 0x55c816649930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc89e0c9640 1 --2- 172.21.15.70:0/2495648616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c816632210 0x55c81662c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc89d0c7640 1 -- 172.21.15.70:0/2495648616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c816632210 msgr2=0x55c81662c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc89d0c7640 1 --2- 172.21.15.70:0/2495648616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c816632210 0x55c81662c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc89d0c7640 1 -- 172.21.15.70:0/2495648616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c81661e760 msgr2=0x55c816649930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc89d0c7640 1 --2- 172.21.15.70:0/2495648616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c81661e760 0x55c816649930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc89d0c7640 1 -- 172.21.15.70:0/2495648616 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c816630080 con 0x55c81661f080 2024-06-04T05:13:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc89e0c9640 1 --2- 172.21.15.70:0/2495648616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c81661e760 0x55c816649930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc89d0c7640 1 --2- 172.21.15.70:0/2495648616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 0x55c816649e70 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fc888012780 tx=0x7fc888005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7879a45f130b0b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc8837fe640 1 -- 172.21.15.70:0/2495648616 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc888004230 con 0x55c81661f080 2024-06-04T05:13:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc8837fe640 1 -- 172.21.15.70:0/2495648616 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc8880043d0 con 0x55c81661f080 2024-06-04T05:13:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.593+0000 7fc8837fe640 1 -- 172.21.15.70:0/2495648616 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc888010800 con 0x55c81661f080 2024-06-04T05:13:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.594+0000 7fc8a2845740 1 -- 172.21.15.70:0/2495648616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 msgr2=0x55c816649e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.594+0000 7fc8a2845740 1 --2- 172.21.15.70:0/2495648616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 0x55c816649e70 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fc888012780 tx=0x7fc888005f90 comp rx=0 tx=0).stop 2024-06-04T05:13:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.594+0000 7fc8a2845740 1 -- 172.21.15.70:0/2495648616 shutdown_connections 2024-06-04T05:13:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.594+0000 7fc8a2845740 1 --2- 172.21.15.70:0/2495648616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c816632210 0x55c81662c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.594+0000 7fc8a2845740 1 --2- 172.21.15.70:0/2495648616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 0x55c816649e70 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.594+0000 7fc8a2845740 1 --2- 172.21.15.70:0/2495648616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c81661e760 0x55c816649930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.594+0000 7fc8a2845740 1 -- 172.21.15.70:0/2495648616 >> 172.21.15.70:0/2495648616 conn(0x55c81661ad30 msgr2=0x55c81662f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.594+0000 7fc8a2845740 1 -- 172.21.15.70:0/2495648616 shutdown_connections 2024-06-04T05:13:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.594+0000 7fc8a2845740 1 -- 172.21.15.70:0/2495648616 wait complete. 2024-06-04T05:13:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc8a2845740 1 -- start start 2024-06-04T05:13:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc8a2845740 1 Processor -- start 2024-06-04T05:13:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc8a2845740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 0x55c81662c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc8a2845740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 0x55c81662c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc8a2845740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c816632210 0x55c81662cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc8a2845740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc8981ad960 con 0x55c81661f080 2024-06-04T05:13:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc8a2845740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc8981ad260 con 0x55c816632210 2024-06-04T05:13:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc8a2845740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc8981aa400 con 0x55c81661e760 2024-06-04T05:13:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc8a2845740 10 client.0 _ll_register_callbacks cb 0x7fc894106630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc89d0c7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 0x55c81662c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc89e0c9640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 0x55c81662c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc89d0c7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 0x55c81662c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56912/0 (socket says 172.21.15.70:56912) 2024-06-04T05:13:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc89d0c7640 1 -- 172.21.15.70:0/668839781 learned_addr learned my addr 172.21.15.70:0/668839781 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc89e0c9640 1 --2- 172.21.15.70:0/668839781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c816632210 0x55c81662cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc89e0c9640 1 -- 172.21.15.70:0/668839781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c816632210 msgr2=0x55c81662cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc89e0c9640 1 --2- 172.21.15.70:0/668839781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c816632210 0x55c81662cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc89e0c9640 1 -- 172.21.15.70:0/668839781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 msgr2=0x55c81662c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc89e0c9640 1 --2- 172.21.15.70:0/668839781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 0x55c81662c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.595+0000 7fc89e0c9640 1 -- 172.21.15.70:0/668839781 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8166ab800 con 0x55c81661e760 2024-06-04T05:13:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.596+0000 7fc89e0c9640 1 --2- 172.21.15.70:0/668839781 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 0x55c81662c090 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc890004500 tx=0x7fc890004760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af89db6cc924f5c2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.596+0000 7fc8817fa640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:13:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.596+0000 7fc8817fa640 1 -- 172.21.15.70:0/668839781 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc890005580 con 0x55c81661e760 2024-06-04T05:13:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.596+0000 7fc8817fa640 1 -- 172.21.15.70:0/668839781 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7fc89001ae00 con 0x55c81661e760 2024-06-04T05:13:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.596+0000 7fc8a2845740 1 -- 172.21.15.70:0/668839781 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8880043d0 con 0x55c81661e760 2024-06-04T05:13:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.596+0000 7fc8817fa640 1 -- 172.21.15.70:0/668839781 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fc890005720 con 0x55c81661e760 2024-06-04T05:13:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.596+0000 7fc8a2845740 10 client.28720 fetch_fsmap learned FSMap version 48 2024-06-04T05:13:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.596+0000 7fc8a2845740 1 -- 172.21.15.70:0/668839781 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55c8166aca90 con 0x55c81661e760 2024-06-04T05:13:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.596+0000 7fc8817fa640 1 -- 172.21.15.70:0/668839781 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc8880043d0 con 0x55c81661e760 2024-06-04T05:13:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.596+0000 7fc8817fa640 1 -- 172.21.15.70:0/668839781 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 48) ==== 8398+0+0 (secure 0 0 0) 0x7fc890010690 con 0x55c81661e760 2024-06-04T05:13:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.596+0000 7fc8a2845740 10 client.28720 fetch_fsmap finished waiting for FSMap version 48 2024-06-04T05:13:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.596+0000 7fc8a2845740 10 client.28720 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.596+0000 7fc8a2845740 20 client.28720 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.596+0000 7fc8a2845740 1 --2- 172.21.15.70:0/668839781 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c8166ad1b0 0x55c8166af5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.596+0000 7fc8a2845740 4 client.28720 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:13:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.596+0000 7fc8a2845740 1 -- 172.21.15.70:0/668839781 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c81658c360 con 0x55c8166ad1b0 2024-06-04T05:13:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.597+0000 7fc89d0c7640 1 --2- 172.21.15.70:0/668839781 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c8166ad1b0 0x55c8166af5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.597+0000 7fc8817fa640 10 client.28720 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.597+0000 7fc89d0c7640 1 --2- 172.21.15.70:0/668839781 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c8166ad1b0 0x55c8166af5d0 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bfbddae193fd54e server_cookie=13783b242704da07 in_seq=0 out_seq=0 2024-06-04T05:13:41.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:41 smithi070 ceph-mon[32631]: pgmap v516: 97 pgs: 97 active+clean; 483 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 288 op/s 2024-06-04T05:13:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:41 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:13:41.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.665+0000 7fc8817fa640 1 -- 172.21.15.70:0/668839781 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55c81658c360 con 0x55c8166ad1b0 2024-06-04T05:13:41.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.665+0000 7fc8817fa640 10 client.28720 handle_command_reply: tid=0 2024-06-04T05:13:41.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.665+0000 7fc8a2845740 1 client.28720 shutdown 2024-06-04T05:13:41.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.665+0000 7fc8a2845740 1 -- 172.21.15.70:0/668839781 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 msgr2=0x55c81662c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:41.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.665+0000 7fc8a2845740 1 --2- 172.21.15.70:0/668839781 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 0x55c81662c090 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc890004500 tx=0x7fc890004760 comp rx=0 tx=0).stop 2024-06-04T05:13:41.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.665+0000 7fc8a2845740 1 -- 172.21.15.70:0/668839781 shutdown_connections 2024-06-04T05:13:41.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.665+0000 7fc8a2845740 1 --2- 172.21.15.70:0/668839781 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c8166ad1b0 0x55c8166af5d0 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.665+0000 7fc8817fa640 0 client.28720 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:41.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.665+0000 7fc8a2845740 1 --2- 172.21.15.70:0/668839781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c816632210 0x55c81662cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.665+0000 7fc8a2845740 1 --2- 172.21.15.70:0/668839781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c81661f080 0x55c81662c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.665+0000 7fc8a2845740 1 --2- 172.21.15.70:0/668839781 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c81661e760 0x55c81662c090 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.665+0000 7fc8a2845740 1 -- 172.21.15.70:0/668839781 >> 172.21.15.70:0/668839781 conn(0x55c81661ad30 msgr2=0x55c81662ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:41.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.665+0000 7fc8a2845740 1 -- 172.21.15.70:0/668839781 shutdown_connections 2024-06-04T05:13:41.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.665+0000 7fc8a2845740 1 -- 172.21.15.70:0/668839781 wait complete. 2024-06-04T05:13:41.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.666+0000 7fc8a2845740 20 client.28720 trim_cache size 0 max 16384 2024-06-04T05:13:41.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:13:41.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:13:41.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:13:41.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.666+0000 7fc89fb2c640 1 -- 172.21.15.70:0/3639975913 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc8840b2a90 msgr2=0x7fc8840b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:41.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.666+0000 7fc89fb2c640 1 --2- 172.21.15.70:0/3639975913 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc8840b2a90 0x7fc8840b4f50 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc8940c9f40 tx=0x7fc894106760 comp rx=0 tx=0).stop 2024-06-04T05:13:41.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.666+0000 7fc89fb2c640 1 -- 172.21.15.70:0/3639975913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc898102080 msgr2=0x7fc898199020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:41.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.666+0000 7fc89fb2c640 1 --2- 172.21.15.70:0/3639975913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc898102080 0x7fc898199020 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fc894006010 tx=0x7fc894032f90 comp rx=0 tx=0).stop 2024-06-04T05:13:41.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.666+0000 7fc89fb2c640 1 -- 172.21.15.70:0/3639975913 shutdown_connections 2024-06-04T05:13:41.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.666+0000 7fc89fb2c640 1 --2- 172.21.15.70:0/3639975913 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc8840b2a90 0x7fc8840b4f50 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.666+0000 7fc89fb2c640 1 --2- 172.21.15.70:0/3639975913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc898102a40 0x7fc898199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.666+0000 7fc89fb2c640 1 --2- 172.21.15.70:0/3639975913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc898106960 0x7fc89819d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:41.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.666+0000 7fc89fb2c640 1 --2- 172.21.15.70:0/3639975913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc898102080 0x7fc898199020 secure :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fc894006010 tx=0x7fc894032f90 comp rx=0 tx=0).stop 2024-06-04T05:13:41.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.666+0000 7fc89fb2c640 1 -- 172.21.15.70:0/3639975913 >> 172.21.15.70:0/3639975913 conn(0x7fc898077ee0 msgr2=0x7fc898104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:41.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.667+0000 7fc89fb2c640 1 -- 172.21.15.70:0/3639975913 shutdown_connections 2024-06-04T05:13:41.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:41.667+0000 7fc89fb2c640 1 -- 172.21.15.70:0/3639975913 wait complete. 2024-06-04T05:13:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:41 smithi112 ceph-mon[36453]: pgmap v516: 97 pgs: 97 active+clean; 483 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 288 op/s 2024-06-04T05:13:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:41 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:13:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:41 smithi098 ceph-mon[35569]: pgmap v516: 97 pgs: 97 active+clean; 483 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 288 op/s 2024-06-04T05:13:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:41 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:13:41.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 22001 8.64 MB/sec execute 33 sec latency 234.231 ms 2024-06-04T05:13:42.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 22137 8.49 MB/sec execute 34 sec latency 201.832 ms 2024-06-04T05:13:43.547 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:43 smithi098 ceph-mon[35569]: pgmap v517: 97 pgs: 97 active+clean; 479 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.6 MiB/s wr, 442 op/s 2024-06-04T05:13:43.547 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:43 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:13:43.547 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:43 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:13:43.547 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:43 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:13:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:43 smithi070 ceph-mon[32631]: pgmap v517: 97 pgs: 97 active+clean; 479 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.6 MiB/s wr, 442 op/s 2024-06-04T05:13:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:43 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:13:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:43 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:13:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:43 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:13:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:43 smithi112 ceph-mon[36453]: pgmap v517: 97 pgs: 97 active+clean; 479 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.6 MiB/s wr, 442 op/s 2024-06-04T05:13:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:43 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:13:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:43 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:13:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:43 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:13:43.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 22305 8.46 MB/sec execute 35 sec latency 136.200 ms 2024-06-04T05:13:44.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 22494 8.61 MB/sec execute 36 sec latency 85.026 ms 2024-06-04T05:13:45.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:45 smithi070 ceph-mon[32631]: pgmap v518: 97 pgs: 97 active+clean; 479 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 292 op/s 2024-06-04T05:13:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:45 smithi112 ceph-mon[36453]: pgmap v518: 97 pgs: 97 active+clean; 479 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 292 op/s 2024-06-04T05:13:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:45 smithi098 ceph-mon[35569]: pgmap v518: 97 pgs: 97 active+clean; 479 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 292 op/s 2024-06-04T05:13:45.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 22661 8.59 MB/sec execute 37 sec latency 141.139 ms 2024-06-04T05:13:46.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 22859 8.71 MB/sec execute 38 sec latency 203.015 ms 2024-06-04T05:13:47.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:47 smithi070 ceph-mon[32631]: pgmap v519: 97 pgs: 97 active+clean; 489 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 420 op/s 2024-06-04T05:13:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:47 smithi112 ceph-mon[36453]: pgmap v519: 97 pgs: 97 active+clean; 489 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 420 op/s 2024-06-04T05:13:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:47 smithi098 ceph-mon[35569]: pgmap v519: 97 pgs: 97 active+clean; 489 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 420 op/s 2024-06-04T05:13:47.803 INFO:tasks.workunit.client.0.smithi070.stdout: 10 23026 8.84 MB/sec execute 39 sec latency 128.056 ms 2024-06-04T05:13:48.052 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:13:48.404 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:13:48.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 23212 8.97 MB/sec execute 40 sec latency 75.032 ms 2024-06-04T05:13:49.562 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:49 smithi070 ceph-mon[32631]: pgmap v520: 97 pgs: 97 active+clean; 489 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 3.0 MiB/s wr, 281 op/s 2024-06-04T05:13:49.562 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:49 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:49 smithi112 ceph-mon[36453]: pgmap v520: 97 pgs: 97 active+clean; 489 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 3.0 MiB/s wr, 281 op/s 2024-06-04T05:13:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:49 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:49 smithi098 ceph-mon[35569]: pgmap v520: 97 pgs: 97 active+clean; 489 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 3.0 MiB/s wr, 281 op/s 2024-06-04T05:13:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:49 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:49.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 23433 9.18 MB/sec execute 41 sec latency 55.727 ms 2024-06-04T05:13:50.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.467+0000 7f78f5662700 1 -- 172.21.15.70:0/2138742045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78f0108920 msgr2=0x7f78f01104a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:50.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.467+0000 7f78f5662700 1 --2- 172.21.15.70:0/2138742045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78f0108920 0x7f78f01104a0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f78e800b0d0 tx=0x7f78e800b3e0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.467+0000 7f78f5662700 1 -- 172.21.15.70:0/2138742045 shutdown_connections 2024-06-04T05:13:50.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.467+0000 7f78f5662700 1 --2- 172.21.15.70:0/2138742045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78f0108920 0x7f78f01104a0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.467+0000 7f78f5662700 1 --2- 172.21.15.70:0/2138742045 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78f0103850 0x7f78f0103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.467+0000 7f78f5662700 1 --2- 172.21.15.70:0/2138742045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f0104200 0x7f78f01081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.467+0000 7f78f5662700 1 -- 172.21.15.70:0/2138742045 >> 172.21.15.70:0/2138742045 conn(0x7f78f007a5d0 msgr2=0x7f78f007a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:50.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.467+0000 7f78f5662700 1 -- 172.21.15.70:0/2138742045 shutdown_connections 2024-06-04T05:13:50.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.467+0000 7f78f5662700 1 -- 172.21.15.70:0/2138742045 wait complete. 2024-06-04T05:13:50.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.468+0000 7f78f5662700 1 Processor -- start 2024-06-04T05:13:50.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.468+0000 7f78f5662700 1 -- start start 2024-06-04T05:13:50.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.468+0000 7f78f5662700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f0103850 0x7f78f010f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.468+0000 7f78f5662700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78f0104200 0x7f78f010fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.468+0000 7f78f5662700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78f0108920 0x7f78f01a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.468+0000 7f78f5662700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f78f0113620 con 0x7f78f0104200 2024-06-04T05:13:50.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.468+0000 7f78f5662700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78f01134a0 con 0x7f78f0103850 2024-06-04T05:13:50.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.468+0000 7f78f5662700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f78f01137a0 con 0x7f78f0108920 2024-06-04T05:13:50.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.468+0000 7f78eeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f0103850 0x7f78f010f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:50.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.469+0000 7f78eeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f0103850 0x7f78f010f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:39892/0 (socket says 172.21.15.70:39892) 2024-06-04T05:13:50.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.469+0000 7f78eeffd700 1 -- 172.21.15.70:0/3061913846 learned_addr learned my addr 172.21.15.70:0/3061913846 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:50.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.469+0000 7f78ee7fc700 1 --2- 172.21.15.70:0/3061913846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78f0104200 0x7f78f010fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:50.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.469+0000 7f78eeffd700 1 -- 172.21.15.70:0/3061913846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78f0108920 msgr2=0x7f78f01a4930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-04T05:13:50.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.469+0000 7f78eeffd700 1 --2- 172.21.15.70:0/3061913846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78f0108920 0x7f78f01a4930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.469+0000 7f78eeffd700 1 -- 172.21.15.70:0/3061913846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78f0104200 msgr2=0x7f78f010fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:50.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.469+0000 7f78eeffd700 1 --2- 172.21.15.70:0/3061913846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78f0104200 0x7f78f010fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.469+0000 7f78eeffd700 1 -- 172.21.15.70:0/3061913846 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f01a5070 con 0x7f78f0103850 2024-06-04T05:13:50.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.469+0000 7f78ee7fc700 1 --2- 172.21.15.70:0/3061913846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78f0104200 0x7f78f010fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:50.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.469+0000 7f78eeffd700 1 --2- 172.21.15.70:0/3061913846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f0103850 0x7f78f010f740 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f78e000e9c0 tx=0x7f78e000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:50.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.469+0000 7f78d7fff700 1 -- 172.21.15.70:0/3061913846 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e000cbd0 con 0x7f78f0103850 2024-06-04T05:13:50.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.470+0000 7f78f5662700 1 -- 172.21.15.70:0/3061913846 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f01a5360 con 0x7f78f0103850 2024-06-04T05:13:50.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.470+0000 7f78d7fff700 1 -- 172.21.15.70:0/3061913846 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f78e000cd70 con 0x7f78f0103850 2024-06-04T05:13:50.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.470+0000 7f78d7fff700 1 -- 172.21.15.70:0/3061913846 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e0005540 con 0x7f78f0103850 2024-06-04T05:13:50.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.470+0000 7f78f5662700 1 -- 172.21.15.70:0/3061913846 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f01acbe0 con 0x7f78f0103850 2024-06-04T05:13:50.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.473+0000 7f78d7fff700 1 -- 172.21.15.70:0/3061913846 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f78e0004750 con 0x7f78f0103850 2024-06-04T05:13:50.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.474+0000 7f78d7fff700 1 --2- 172.21.15.70:0/3061913846 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f78d80b2a70 0x7f78d80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.474+0000 7f78d7fff700 1 -- 172.21.15.70:0/3061913846 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f78e00e28f0 con 0x7f78f0103850 2024-06-04T05:13:50.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.474+0000 7f78ee7fc700 1 --2- 172.21.15.70:0/3061913846 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f78d80b2a70 0x7f78d80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:50.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.474+0000 7f78ffef7b80 1 Processor -- start 2024-06-04T05:13:50.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.474+0000 7f78ffef7b80 1 -- start start 2024-06-04T05:13:50.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.475+0000 7f78ffef7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf1249320 0x557bf1249700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.475+0000 7f78ffef7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 0x557bf1243d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.475+0000 7f78ffef7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1244260 0x557bf1250080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.475+0000 7f78ffef7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557bf11a1d70 con 0x557bf1249320 2024-06-04T05:13:50.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.475+0000 7f78ffef7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557bf1191910 con 0x557bf12438a0 2024-06-04T05:13:50.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.475+0000 7f78ffef7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557bf1195950 con 0x557bf1244260 2024-06-04T05:13:50.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.475+0000 7f78eeffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf1249320 0x557bf1249700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:50.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.475+0000 7f78eeffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf1249320 0x557bf1249700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44150/0 (socket says 172.21.15.70:44150) 2024-06-04T05:13:50.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.475+0000 7f78eeffd700 1 -- 172.21.15.70:0/3605144804 learned_addr learned my addr 172.21.15.70:0/3605144804 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:50.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.475+0000 7f78ef7fe700 1 --2- 172.21.15.70:0/3605144804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1244260 0x557bf1250080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:50.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.475+0000 7f78ee7fc700 1 --2- 172.21.15.70:0/3605144804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 0x557bf1243d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:50.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.475+0000 7f78eeffd700 1 -- 172.21.15.70:0/3605144804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1244260 msgr2=0x557bf1250080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:50.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.475+0000 7f78eeffd700 1 --2- 172.21.15.70:0/3605144804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1244260 0x557bf1250080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.475+0000 7f78eeffd700 1 -- 172.21.15.70:0/3605144804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 msgr2=0x557bf1243d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:50.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.475+0000 7f78eeffd700 1 --2- 172.21.15.70:0/3605144804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 0x557bf1243d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.475+0000 7f78eeffd700 1 -- 172.21.15.70:0/3605144804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bf113afb0 con 0x557bf1249320 2024-06-04T05:13:50.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.476+0000 7f78ee7fc700 1 --2- 172.21.15.70:0/3061913846 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f78d80b2a70 0x7f78d80b4f30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f78f0077860 tx=0x7f78e4009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:50.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.476+0000 7f78eeffd700 1 --2- 172.21.15.70:0/3605144804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf1249320 0x557bf1249700 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f78e00058b0 tx=0x7f78e00a9560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf895fdd49d1743 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:50.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.476+0000 7f78d57fa700 1 -- 172.21.15.70:0/3605144804 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e00b22f0 con 0x557bf1249320 2024-06-04T05:13:50.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.476+0000 7f78d57fa700 1 -- 172.21.15.70:0/3605144804 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f78e00aa240 con 0x557bf1249320 2024-06-04T05:13:50.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.476+0000 7f78d57fa700 1 -- 172.21.15.70:0/3605144804 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e00aa000 con 0x557bf1249320 2024-06-04T05:13:50.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.477+0000 7f78ffef7b80 1 -- 172.21.15.70:0/3605144804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf1249320 msgr2=0x557bf1249700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:50.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.477+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/3605144804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf1249320 0x557bf1249700 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f78e00058b0 tx=0x7f78e00a9560 comp rx=0 tx=0).stop 2024-06-04T05:13:50.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.477+0000 7f78ffef7b80 1 -- 172.21.15.70:0/3605144804 shutdown_connections 2024-06-04T05:13:50.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.477+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/3605144804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1244260 0x557bf1250080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.477+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/3605144804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf1249320 0x557bf1249700 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.477+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/3605144804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 0x557bf1243d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.477+0000 7f78ffef7b80 1 -- 172.21.15.70:0/3605144804 >> 172.21.15.70:0/3605144804 conn(0x557bf116f240 msgr2=0x557bf123e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:50.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.477+0000 7f78ffef7b80 1 -- 172.21.15.70:0/3605144804 shutdown_connections 2024-06-04T05:13:50.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.477+0000 7f78ffef7b80 1 -- 172.21.15.70:0/3605144804 wait complete. 2024-06-04T05:13:50.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.478+0000 7f78ffef7b80 1 -- start start 2024-06-04T05:13:50.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.478+0000 7f78ffef7b80 1 Processor -- start 2024-06-04T05:13:50.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.478+0000 7f78ffef7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 0x557bf124fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.478+0000 7f78ffef7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1244260 0x557bf124d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.478+0000 7f78ffef7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf1249320 0x557bf124d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.478+0000 7f78ffef7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557bf1097fa0 con 0x557bf1249320 2024-06-04T05:13:50.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.478+0000 7f78ffef7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557bf10f9da0 con 0x557bf12438a0 2024-06-04T05:13:50.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.478+0000 7f78ee7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1244260 0x557bf124d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:50.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.478+0000 7f78eeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 0x557bf124fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:50.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.478+0000 7f78ee7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1244260 0x557bf124d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:40128/0 (socket says 172.21.15.70:40128) 2024-06-04T05:13:50.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.478+0000 7f78ee7fc700 1 -- 172.21.15.70:0/2402832454 learned_addr learned my addr 172.21.15.70:0/2402832454 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:50.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.479+0000 7f78ef7fe700 1 --2- 172.21.15.70:0/2402832454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf1249320 0x557bf124d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:50.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.479+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2402832454 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557bf12506e0 con 0x557bf1244260 2024-06-04T05:13:50.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.479+0000 7f78ffef7b80 10 client.0 _ll_register_callbacks cb 0x557bf112c950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:50.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.479+0000 7f78eeffd700 1 -- 172.21.15.70:0/2402832454 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1244260 msgr2=0x557bf124d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:50.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.479+0000 7f78eeffd700 1 --2- 172.21.15.70:0/2402832454 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1244260 0x557bf124d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.479+0000 7f78eeffd700 1 -- 172.21.15.70:0/2402832454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf1249320 msgr2=0x557bf124d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:50.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.479+0000 7f78eeffd700 1 --2- 172.21.15.70:0/2402832454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf1249320 0x557bf124d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.479+0000 7f78eeffd700 1 -- 172.21.15.70:0/2402832454 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bf10e18d0 con 0x557bf12438a0 2024-06-04T05:13:50.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.479+0000 7f78ef7fe700 1 --2- 172.21.15.70:0/2402832454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf1249320 0x557bf124d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:50.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.479+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2402832454 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78e00aa240 con 0x557bf12438a0 2024-06-04T05:13:50.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.479+0000 7f78eeffd700 1 --2- 172.21.15.70:0/2402832454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 0x557bf124fd40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x557bf12447c0 tx=0x7f78e00a9780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=527655b0640f71b4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:50.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.479+0000 7f78d2ff5700 10 client.28619 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:13:50.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.479+0000 7f78d2ff5700 1 -- 172.21.15.70:0/2402832454 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e00c8070 con 0x557bf12438a0 2024-06-04T05:13:50.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.479+0000 7f78d2ff5700 1 -- 172.21.15.70:0/2402832454 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f78e00b2720 con 0x557bf12438a0 2024-06-04T05:13:50.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.479+0000 7f78d2ff5700 1 -- 172.21.15.70:0/2402832454 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f78e00b22f0 con 0x557bf12438a0 2024-06-04T05:13:50.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.480+0000 7f78ffef7b80 10 client.28619 fetch_fsmap learned FSMap version 49 2024-06-04T05:13:50.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.480+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2402832454 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({fsmap=49}) -- 0x557bf114b760 con 0x557bf12438a0 2024-06-04T05:13:50.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.480+0000 7f78d2ff5700 1 -- 172.21.15.70:0/2402832454 <== mon.2 v2:172.21.15.98:3300/0 4 ==== fsmap(e 49) ==== 8394+0+0 (secure 0 0 0) 0x7f78e0087060 con 0x557bf12438a0 2024-06-04T05:13:50.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.480+0000 7f78ffef7b80 10 client.28619 fetch_fsmap finished waiting for FSMap version 49 2024-06-04T05:13:50.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.480+0000 7f78ffef7b80 10 client.28619 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:50.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.480+0000 7f78ffef7b80 20 client.28619 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:50.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.480+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/2402832454 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557bf125cfb0 0x557bf125f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.480+0000 7f78ffef7b80 4 client.28619 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:13:50.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.480+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2402832454 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bf11a3cc0 con 0x557bf125cfb0 2024-06-04T05:13:50.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.480+0000 7f78ee7fc700 1 --2- 172.21.15.70:0/2402832454 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557bf125cfb0 0x557bf125f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:50.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.481+0000 7f78d2ff5700 10 client.28619 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:50.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.481+0000 7f78ee7fc700 1 --2- 172.21.15.70:0/2402832454 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557bf125cfb0 0x557bf125f3a0 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64b597d809331bfc server_cookie=e74447aaca6cb63 in_seq=0 out_seq=0 2024-06-04T05:13:50.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.482+0000 7f78d2ff5700 1 -- 172.21.15.70:0/2402832454 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557bf11a3cc0 con 0x557bf125cfb0 2024-06-04T05:13:50.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.482+0000 7f78d2ff5700 10 client.28619 handle_command_reply: tid=0 2024-06-04T05:13:50.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.482+0000 7f78ffef7b80 1 client.28619 shutdown 2024-06-04T05:13:50.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.482+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2402832454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 msgr2=0x557bf124fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:50.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.482+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/2402832454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 0x557bf124fd40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x557bf12447c0 tx=0x7f78e00a9780 comp rx=0 tx=0).stop 2024-06-04T05:13:50.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.482+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2402832454 shutdown_connections 2024-06-04T05:13:50.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.482+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/2402832454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf1249320 0x557bf124d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.482+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/2402832454 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557bf125cfb0 0x557bf125f3a0 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.482+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/2402832454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 0x557bf124fd40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.482+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/2402832454 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1244260 0x557bf124d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.482+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2402832454 >> 172.21.15.70:0/2402832454 conn(0x557bf116f240 msgr2=0x557bf123daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:50.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.482+0000 7f78d2ff5700 0 client.28619 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:50.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.482+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2402832454 shutdown_connections 2024-06-04T05:13:50.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.482+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2402832454 wait complete. 2024-06-04T05:13:50.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.482+0000 7f78ffef7b80 20 client.28619 trim_cache size 0 max 16384 2024-06-04T05:13:50.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.512+0000 7f78ffef7b80 1 Processor -- start 2024-06-04T05:13:50.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.512+0000 7f78ffef7b80 1 -- start start 2024-06-04T05:13:50.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78ffef7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf12438a0 0x557bf123e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78ffef7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf1244260 0x557bf124ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78ffef7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1249320 0x557bf124a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78ffef7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f78f01b4350 con 0x557bf12438a0 2024-06-04T05:13:50.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78ffef7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78f01b6010 con 0x557bf1244260 2024-06-04T05:13:50.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78ffef7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f78f01b4a80 con 0x557bf1249320 2024-06-04T05:13:50.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78eeffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf12438a0 0x557bf123e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:50.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78ee7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf1244260 0x557bf124ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:50.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78ee7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf1244260 0x557bf124ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:39928/0 (socket says 172.21.15.70:39928) 2024-06-04T05:13:50.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78ee7fc700 1 -- 172.21.15.70:0/2687087291 learned_addr learned my addr 172.21.15.70:0/2687087291 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:50.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78eeffd700 1 --2- 172.21.15.70:0/2687087291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf12438a0 0x557bf123e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44176/0 (socket says 172.21.15.70:44176) 2024-06-04T05:13:50.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78ef7fe700 1 --2- 172.21.15.70:0/2687087291 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1249320 0x557bf124a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:50.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78ee7fc700 1 -- 172.21.15.70:0/2687087291 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1249320 msgr2=0x557bf124a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:50.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78ee7fc700 1 --2- 172.21.15.70:0/2687087291 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1249320 0x557bf124a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78ee7fc700 1 -- 172.21.15.70:0/2687087291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf12438a0 msgr2=0x557bf123e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:50.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78ee7fc700 1 --2- 172.21.15.70:0/2687087291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf12438a0 0x557bf123e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78ee7fc700 1 -- 172.21.15.70:0/2687087291 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bf1249a70 con 0x557bf1244260 2024-06-04T05:13:50.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.513+0000 7f78eeffd700 1 --2- 172.21.15.70:0/2687087291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf12438a0 0x557bf123e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:50.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.514+0000 7f78ee7fc700 1 --2- 172.21.15.70:0/2687087291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf1244260 0x557bf124ef80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009f10 tx=0x7f78e4018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c5ad190ab7e25a7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:50.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.514+0000 7f78d4ff9700 1 -- 172.21.15.70:0/2687087291 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e4003d40 con 0x557bf1244260 2024-06-04T05:13:50.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.514+0000 7f78d4ff9700 1 -- 172.21.15.70:0/2687087291 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f78e401e8e0 con 0x557bf1244260 2024-06-04T05:13:50.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.514+0000 7f78d4ff9700 1 -- 172.21.15.70:0/2687087291 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e4029660 con 0x557bf1244260 2024-06-04T05:13:50.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.515+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2687087291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf1244260 msgr2=0x557bf124ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:50.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.515+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/2687087291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf1244260 0x557bf124ef80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009f10 tx=0x7f78e4018000 comp rx=0 tx=0).stop 2024-06-04T05:13:50.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.515+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2687087291 shutdown_connections 2024-06-04T05:13:50.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.515+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/2687087291 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1249320 0x557bf124a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.515+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/2687087291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf12438a0 0x557bf123e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.515+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/2687087291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf1244260 0x557bf124ef80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.515+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2687087291 >> 172.21.15.70:0/2687087291 conn(0x557bf116f240 msgr2=0x557bf124e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:50.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.515+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2687087291 shutdown_connections 2024-06-04T05:13:50.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.515+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2687087291 wait complete. 2024-06-04T05:13:50.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.516+0000 7f78ffef7b80 1 -- start start 2024-06-04T05:13:50.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.516+0000 7f78ffef7b80 1 Processor -- start 2024-06-04T05:13:50.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.516+0000 7f78ffef7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 0x557bf1262960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.516+0000 7f78ffef7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1244260 0x557bf125dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.516+0000 7f78ffef7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf1249320 0x557bf125e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.516+0000 7f78ffef7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f78f01b58b0 con 0x557bf1249320 2024-06-04T05:13:50.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.516+0000 7f78ffef7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78f01b51b0 con 0x557bf12438a0 2024-06-04T05:13:50.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.516+0000 7f78ffef7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f78f01b2310 con 0x557bf1244260 2024-06-04T05:13:50.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.516+0000 7f78eeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 0x557bf1262960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:50.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.516+0000 7f78ffef7b80 10 client.0 _ll_register_callbacks cb 0x557bf112c950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:50.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.516+0000 7f78eeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 0x557bf1262960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:39940/0 (socket says 172.21.15.70:39940) 2024-06-04T05:13:50.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.516+0000 7f78eeffd700 1 -- 172.21.15.70:0/2971574630 learned_addr learned my addr 172.21.15.70:0/2971574630 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:50.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.516+0000 7f78ee7fc700 1 --2- 172.21.15.70:0/2971574630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1244260 0x557bf125dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:50.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.516+0000 7f78eeffd700 1 -- 172.21.15.70:0/2971574630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1244260 msgr2=0x557bf125dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:50.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.516+0000 7f78eeffd700 1 --2- 172.21.15.70:0/2971574630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1244260 0x557bf125dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.517+0000 7f78eeffd700 1 -- 172.21.15.70:0/2971574630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf1249320 msgr2=0x557bf125e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:50.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.517+0000 7f78eeffd700 1 --2- 172.21.15.70:0/2971574630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf1249320 0x557bf125e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.517+0000 7f78eeffd700 1 -- 172.21.15.70:0/2971574630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bf125e8f0 con 0x557bf12438a0 2024-06-04T05:13:50.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.517+0000 7f78eeffd700 1 --2- 172.21.15.70:0/2971574630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 0x557bf1262960 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f78e00a5bc0 tx=0x7f78e00a5a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=609bb96c9ce7ca34 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:50.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.517+0000 7f78d2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:13:50.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.517+0000 7f78d2ff5700 1 -- 172.21.15.70:0/2971574630 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e00b29f0 con 0x557bf12438a0 2024-06-04T05:13:50.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.517+0000 7f78d2ff5700 1 -- 172.21.15.70:0/2971574630 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f78e00a9da0 con 0x557bf12438a0 2024-06-04T05:13:50.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.517+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2971574630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78e401e8e0 con 0x557bf12438a0 2024-06-04T05:13:50.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.518+0000 7f78d2ff5700 1 -- 172.21.15.70:0/2971574630 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f78e00b22f0 con 0x557bf12438a0 2024-06-04T05:13:50.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.518+0000 7f78ffef7b80 10 client.28625 fetch_fsmap learned FSMap version 49 2024-06-04T05:13:50.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.518+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2971574630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x557bf125e8f0 con 0x557bf12438a0 2024-06-04T05:13:50.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.518+0000 7f78d2ff5700 1 -- 172.21.15.70:0/2971574630 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f78e00aa240 con 0x557bf12438a0 2024-06-04T05:13:50.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.518+0000 7f78d2ff5700 1 -- 172.21.15.70:0/2971574630 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 49) ==== 8394+0+0 (secure 0 0 0) 0x7f78e00a54b0 con 0x557bf12438a0 2024-06-04T05:13:50.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.518+0000 7f78ffef7b80 10 client.28625 fetch_fsmap finished waiting for FSMap version 49 2024-06-04T05:13:50.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.518+0000 7f78ffef7b80 10 client.28625 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:50.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.518+0000 7f78ffef7b80 20 client.28625 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:50.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.518+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/2971574630 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557bf12c7a20 0x557bf12c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:50.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.518+0000 7f78ffef7b80 4 client.28625 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:13:50.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.518+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2971574630 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f78e401e8e0 con 0x557bf12c7a20 2024-06-04T05:13:50.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.518+0000 7f78ee7fc700 1 --2- 172.21.15.70:0/2971574630 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557bf12c7a20 0x557bf12c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:50.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.519+0000 7f78d2ff5700 10 client.28625 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:50.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.519+0000 7f78ee7fc700 1 --2- 172.21.15.70:0/2971574630 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557bf12c7a20 0x557bf12c9e40 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4238b3a228511d5a server_cookie=bd69a1584e1bbc90 in_seq=0 out_seq=0 2024-06-04T05:13:50.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.519+0000 7f78d2ff5700 1 -- 172.21.15.70:0/2971574630 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f78e401e8e0 con 0x557bf12c7a20 2024-06-04T05:13:50.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.519+0000 7f78d2ff5700 10 client.28625 handle_command_reply: tid=0 2024-06-04T05:13:50.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.519+0000 7f78ffef7b80 1 client.28625 shutdown 2024-06-04T05:13:50.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.519+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2971574630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 msgr2=0x557bf1262960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:50.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.519+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/2971574630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 0x557bf1262960 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f78e00a5bc0 tx=0x7f78e00a5a00 comp rx=0 tx=0).stop 2024-06-04T05:13:50.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.519+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2971574630 shutdown_connections 2024-06-04T05:13:50.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.519+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/2971574630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557bf1249320 0x557bf125e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.519+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/2971574630 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557bf12c7a20 0x557bf12c9e40 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.519+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/2971574630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557bf12438a0 0x557bf1262960 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.519+0000 7f78d2ff5700 0 client.28625 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:50.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.520+0000 7f78ffef7b80 1 --2- 172.21.15.70:0/2971574630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557bf1244260 0x557bf125dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.520+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2971574630 >> 172.21.15.70:0/2971574630 conn(0x557bf116f240 msgr2=0x557bf1246ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:50.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.520+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2971574630 shutdown_connections 2024-06-04T05:13:50.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.520+0000 7f78ffef7b80 1 -- 172.21.15.70:0/2971574630 wait complete. 2024-06-04T05:13:50.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.520+0000 7f78ffef7b80 20 client.28625 trim_cache size 0 max 16384 2024-06-04T05:13:50.527 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:13:50.527 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:13:50.527 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:13:50.527 INFO:teuthology.orchestra.run.smithi070.stdout: "89905956-13a2-4e20-8138-06079142a8d9": { 2024-06-04T05:13:50.527 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:13:50.527 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "89905956-13a2-4e20-8138-06079142a8d9", 2024-06-04T05:13:50.527 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:13:50.527 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:13:50.527 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:13:50.528 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:13:50.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.521+0000 7f78f5662700 1 -- 172.21.15.70:0/3061913846 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f78d80b2a70 msgr2=0x7f78d80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:50.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.521+0000 7f78f5662700 1 --2- 172.21.15.70:0/3061913846 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f78d80b2a70 0x7f78d80b4f30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f78f0077860 tx=0x7f78e4009260 comp rx=0 tx=0).stop 2024-06-04T05:13:50.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.521+0000 7f78f5662700 1 -- 172.21.15.70:0/3061913846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f0103850 msgr2=0x7f78f010f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:50.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.521+0000 7f78f5662700 1 --2- 172.21.15.70:0/3061913846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f0103850 0x7f78f010f740 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f78e000e9c0 tx=0x7f78e000ecd0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.521+0000 7f78f5662700 1 -- 172.21.15.70:0/3061913846 shutdown_connections 2024-06-04T05:13:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.521+0000 7f78f5662700 1 --2- 172.21.15.70:0/3061913846 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f78d80b2a70 0x7f78d80b4f30 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.521+0000 7f78f5662700 1 --2- 172.21.15.70:0/3061913846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78f0108920 0x7f78f01a4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.521+0000 7f78f5662700 1 --2- 172.21.15.70:0/3061913846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f0103850 0x7f78f010f740 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.521+0000 7f78f5662700 1 --2- 172.21.15.70:0/3061913846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78f0104200 0x7f78f010fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.521+0000 7f78f5662700 1 -- 172.21.15.70:0/3061913846 >> 172.21.15.70:0/3061913846 conn(0x7f78f007a5d0 msgr2=0x7f78f0101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.521+0000 7f78f5662700 1 -- 172.21.15.70:0/3061913846 shutdown_connections 2024-06-04T05:13:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:13:50.521+0000 7f78f5662700 1 -- 172.21.15.70:0/3061913846 wait complete. 2024-06-04T05:13:50.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 23589 9.28 MB/sec execute 42 sec latency 455.533 ms 2024-06-04T05:13:51.377 INFO:tasks.cephfs.filesystem:scrub status for tag:89905956-13a2-4e20-8138-06079142a8d9 - {'path': '/', 'tag': '89905956-13a2-4e20-8138-06079142a8d9', 'options': 'recursive,force'} 2024-06-04T05:13:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:51 smithi070 ceph-mon[32631]: pgmap v521: 97 pgs: 97 active+clean; 489 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 3.0 MiB/s wr, 281 op/s 2024-06-04T05:13:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:51 smithi112 ceph-mon[36453]: pgmap v521: 97 pgs: 97 active+clean; 489 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 3.0 MiB/s wr, 281 op/s 2024-06-04T05:13:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:51 smithi098 ceph-mon[35569]: pgmap v521: 97 pgs: 97 active+clean; 489 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 3.0 MiB/s wr, 281 op/s 2024-06-04T05:13:51.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 23758 9.28 MB/sec execute 43 sec latency 536.238 ms 2024-06-04T05:13:52.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 23923 9.25 MB/sec execute 44 sec latency 46.312 ms 2024-06-04T05:13:53.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:53 smithi070 ceph-mon[32631]: pgmap v522: 97 pgs: 97 active+clean; 503 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.3 MiB/s wr, 383 op/s 2024-06-04T05:13:53.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:53 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:13:53.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:53 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:13:53.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:53 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:13:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:53 smithi112 ceph-mon[36453]: pgmap v522: 97 pgs: 97 active+clean; 503 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.3 MiB/s wr, 383 op/s 2024-06-04T05:13:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:53 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:13:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:53 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:13:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:53 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:13:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:53 smithi098 ceph-mon[35569]: pgmap v522: 97 pgs: 97 active+clean; 503 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.3 MiB/s wr, 383 op/s 2024-06-04T05:13:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:53 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:13:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:53 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:13:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:53 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:13:53.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 24085 9.23 MB/sec execute 45 sec latency 62.733 ms 2024-06-04T05:13:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:54 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:13:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:54 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:13:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:54 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:54.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:54 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:13:54.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:54 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:13:54.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:54 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:54.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:54 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:13:54.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:54 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:13:54.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:54 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.797+0000 7f55fe485640 1 -- 172.21.15.70:0/2775834680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8102340 msgr2=0x7f55f810e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.797+0000 7f55fe485640 1 --2- 172.21.15.70:0/2775834680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8102340 0x7f55f810e800 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009cd0 tx=0x7f55f002a8e0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.797+0000 7f55fe485640 1 -- 172.21.15.70:0/2775834680 shutdown_connections 2024-06-04T05:13:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.797+0000 7f55fe485640 1 --2- 172.21.15.70:0/2775834680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f8101980 0x7f55f8101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.797+0000 7f55fe485640 1 --2- 172.21.15.70:0/2775834680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8102340 0x7f55f810e800 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.797+0000 7f55fe485640 1 --2- 172.21.15.70:0/2775834680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55f8107940 0x7f55f8107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.797+0000 7f55fe485640 1 -- 172.21.15.70:0/2775834680 >> 172.21.15.70:0/2775834680 conn(0x7f55f8078ad0 msgr2=0x7f55f8078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.797+0000 7f55fe485640 1 -- 172.21.15.70:0/2775834680 shutdown_connections 2024-06-04T05:13:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.797+0000 7f55fe485640 1 -- 172.21.15.70:0/2775834680 wait complete. 2024-06-04T05:13:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.798+0000 7f55fe485640 1 Processor -- start 2024-06-04T05:13:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.798+0000 7f55fe485640 1 -- start start 2024-06-04T05:13:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.798+0000 7f55fe485640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8101980 0x7f55f8196c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.798+0000 7f55fe485640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55f8102340 0x7f55f81971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.798+0000 7f55fe485640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f8107940 0x7f55f81976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.798+0000 7f55fe485640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f55f81109a0 con 0x7f55f8101980 2024-06-04T05:13:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.798+0000 7f55fe485640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f55f8110820 con 0x7f55f8107940 2024-06-04T05:13:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.798+0000 7f55fe485640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f55f8110b20 con 0x7f55f8102340 2024-06-04T05:13:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.798+0000 7f55f7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8101980 0x7f55f8196c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.798+0000 7f55f77fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55f8102340 0x7f55f81971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.798+0000 7f55f7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8101980 0x7f55f8196c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44206/0 (socket says 172.21.15.70:44206) 2024-06-04T05:13:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.798+0000 7f55f7fff640 1 -- 172.21.15.70:0/2497679064 learned_addr learned my addr 172.21.15.70:0/2497679064 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.798+0000 7f55fca22640 1 --2- 172.21.15.70:0/2497679064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f8107940 0x7f55f81976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.799+0000 7f55f77fe640 1 -- 172.21.15.70:0/2497679064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f8107940 msgr2=0x7f55f81976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.799+0000 7f55f77fe640 1 --2- 172.21.15.70:0/2497679064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f8107940 0x7f55f81976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.799+0000 7f55f77fe640 1 -- 172.21.15.70:0/2497679064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8101980 msgr2=0x7f55f8196c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.799+0000 7f55f77fe640 1 --2- 172.21.15.70:0/2497679064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8101980 0x7f55f8196c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.799+0000 7f55f77fe640 1 -- 172.21.15.70:0/2497679064 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55f81ab8e0 con 0x7f55f8102340 2024-06-04T05:13:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.799+0000 7f55f7fff640 1 --2- 172.21.15.70:0/2497679064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8101980 0x7f55f8196c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.799+0000 7f55f77fe640 1 --2- 172.21.15.70:0/2497679064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55f8102340 0x7f55f81971a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f55ec009800 tx=0x7f55ec009cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.799+0000 7f55f57fa640 1 -- 172.21.15.70:0/2497679064 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ec004020 con 0x7f55f8102340 2024-06-04T05:13:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.799+0000 7f55f57fa640 1 -- 172.21.15.70:0/2497679064 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f55ec00ed40 con 0x7f55f8102340 2024-06-04T05:13:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.799+0000 7f55fe485640 1 -- 172.21.15.70:0/2497679064 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55f81abbd0 con 0x7f55f8102340 2024-06-04T05:13:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.799+0000 7f55f57fa640 1 -- 172.21.15.70:0/2497679064 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ec003940 con 0x7f55f8102340 2024-06-04T05:13:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.799+0000 7f55fe485640 1 -- 172.21.15.70:0/2497679064 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55f81ac170 con 0x7f55f8102340 2024-06-04T05:13:54.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 24251 9.11 MB/sec execute 46 sec latency 49.790 ms 2024-06-04T05:13:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f560119e740 1 Processor -- start 2024-06-04T05:13:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f560119e740 1 -- start start 2024-06-04T05:13:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f560119e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a83820 0x5575f4a83c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f560119e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a84140 0x5575f4a8e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f560119e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a8ea00 0x5575f4a90df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f560119e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5575f49f5c30 con 0x5575f4a84140 2024-06-04T05:13:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f560119e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5575f49e57d0 con 0x5575f4a83820 2024-06-04T05:13:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f560119e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5575f49e9810 con 0x5575f4a8ea00 2024-06-04T05:13:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f55f7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a84140 0x5575f4a8e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f55fca22640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a83820 0x5575f4a83c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f55fca22640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a83820 0x5575f4a83c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:39972/0 (socket says 172.21.15.70:39972) 2024-06-04T05:13:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f55f7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a84140 0x5575f4a8e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44214/0 (socket says 172.21.15.70:44214) 2024-06-04T05:13:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f55fca22640 1 -- 172.21.15.70:0/1118228626 learned_addr learned my addr 172.21.15.70:0/1118228626 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f55f57fa640 1 -- 172.21.15.70:0/2497679064 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f55ec01b020 con 0x7f55f8102340 2024-06-04T05:13:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f55fca22640 1 -- 172.21.15.70:0/1118228626 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a8ea00 msgr2=0x5575f4a90df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-04T05:13:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f55fca22640 1 --2- 172.21.15.70:0/1118228626 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a8ea00 0x5575f4a90df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f55fca22640 1 -- 172.21.15.70:0/1118228626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a84140 msgr2=0x5575f4a8e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f55fca22640 1 --2- 172.21.15.70:0/1118228626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a84140 0x5575f4a8e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f55fca22640 1 -- 172.21.15.70:0/1118228626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575f46fa210 con 0x5575f4a83820 2024-06-04T05:13:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.803+0000 7f55f7fff640 1 --2- 172.21.15.70:0/1118228626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a84140 0x5575f4a8e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.804+0000 7f55fca22640 1 --2- 172.21.15.70:0/1118228626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a83820 0x5575f4a83c00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009cd0 tx=0x7f55f0009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e87b60755609816c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:54.808 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.804+0000 7f55de7fc640 1 -- 172.21.15.70:0/1118228626 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f0048070 con 0x5575f4a83820 2024-06-04T05:13:54.808 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.804+0000 7f55de7fc640 1 -- 172.21.15.70:0/1118228626 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f55f0036070 con 0x5575f4a83820 2024-06-04T05:13:54.808 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.804+0000 7f55de7fc640 1 -- 172.21.15.70:0/1118228626 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f00042b0 con 0x5575f4a83820 2024-06-04T05:13:54.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.804+0000 7f55f57fa640 1 --2- 172.21.15.70:0/2497679064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f55e00b2a90 0x7f55e00b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.804+0000 7f55f7fff640 1 --2- 172.21.15.70:0/2497679064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f55e00b2a90 0x7f55e00b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:54.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.804+0000 7f55f57fa640 1 -- 172.21.15.70:0/2497679064 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f55ec0e1b40 con 0x7f55f8102340 2024-06-04T05:13:54.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.805+0000 7f560119e740 1 -- 172.21.15.70:0/1118228626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a83820 msgr2=0x5575f4a83c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:54.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.805+0000 7f560119e740 1 --2- 172.21.15.70:0/1118228626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a83820 0x5575f4a83c00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009cd0 tx=0x7f55f0009a20 comp rx=0 tx=0).stop 2024-06-04T05:13:54.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.805+0000 7f560119e740 1 -- 172.21.15.70:0/1118228626 shutdown_connections 2024-06-04T05:13:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.805+0000 7f560119e740 1 --2- 172.21.15.70:0/1118228626 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a8ea00 0x5575f4a90df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.805+0000 7f560119e740 1 --2- 172.21.15.70:0/1118228626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a84140 0x5575f4a8e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.805+0000 7f560119e740 1 --2- 172.21.15.70:0/1118228626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a83820 0x5575f4a83c00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.805+0000 7f560119e740 1 -- 172.21.15.70:0/1118228626 >> 172.21.15.70:0/1118228626 conn(0x5575f4a7fd30 msgr2=0x5575f4a80110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.805+0000 7f55f7fff640 1 --2- 172.21.15.70:0/2497679064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f55e00b2a90 0x7f55e00b4f50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f55e8009fd0 tx=0x7f55e8005f90 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.805+0000 7f560119e740 1 -- 172.21.15.70:0/1118228626 shutdown_connections 2024-06-04T05:13:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.805+0000 7f560119e740 1 -- 172.21.15.70:0/1118228626 wait complete. 2024-06-04T05:13:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.805+0000 7f560119e740 1 -- start start 2024-06-04T05:13:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.806+0000 7f560119e740 1 Processor -- start 2024-06-04T05:13:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.806+0000 7f560119e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a83820 0x5575f4a8baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.806+0000 7f560119e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a84140 0x5575f4a8bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.806+0000 7f560119e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a8ea00 0x5575f4a8c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.806+0000 7f560119e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5575f46cb790 con 0x5575f4a83820 2024-06-04T05:13:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.806+0000 7f560119e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5575f49a1890 con 0x5575f4a84140 2024-06-04T05:13:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.806+0000 7f560119e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5575f4a915d0 con 0x5575f4a8ea00 2024-06-04T05:13:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.806+0000 7f55f77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a84140 0x5575f4a8bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.806+0000 7f55fca22640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a8ea00 0x5575f4a8c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.806+0000 7f55fca22640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a8ea00 0x5575f4a8c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:40178/0 (socket says 172.21.15.70:40178) 2024-06-04T05:13:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.806+0000 7f55fca22640 1 -- 172.21.15.70:0/2870022880 learned_addr learned my addr 172.21.15.70:0/2870022880 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.806+0000 7f55f7fff640 1 --2- 172.21.15.70:0/2870022880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a83820 0x5575f4a8baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.807+0000 7f560119e740 10 client.0 _ll_register_callbacks cb 0x5575f46f9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.807+0000 7f55f77fe640 1 -- 172.21.15.70:0/2870022880 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a8ea00 msgr2=0x5575f4a8c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.807+0000 7f55f77fe640 1 --2- 172.21.15.70:0/2870022880 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a8ea00 0x5575f4a8c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.807+0000 7f55f77fe640 1 -- 172.21.15.70:0/2870022880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a83820 msgr2=0x5575f4a8baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.807+0000 7f55f77fe640 1 --2- 172.21.15.70:0/2870022880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a83820 0x5575f4a8baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.807+0000 7f55f77fe640 1 -- 172.21.15.70:0/2870022880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575f47c6620 con 0x5575f4a84140 2024-06-04T05:13:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.807+0000 7f55fca22640 1 --2- 172.21.15.70:0/2870022880 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a8ea00 0x5575f4a8c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:13:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.807+0000 7f55f77fe640 1 --2- 172.21.15.70:0/2870022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a84140 0x5575f4a8bfe0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f55ec0df260 tx=0x7f55ec0a5300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0146b2bb8c0ee77 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.807+0000 7f55d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:13:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.807+0000 7f55d77fe640 1 -- 172.21.15.70:0/2870022880 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ec00b660 con 0x5575f4a84140 2024-06-04T05:13:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.808+0000 7f55d77fe640 1 -- 172.21.15.70:0/2870022880 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f55ec0c3470 con 0x5575f4a84140 2024-06-04T05:13:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.808+0000 7f560119e740 1 -- 172.21.15.70:0/2870022880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55f0036070 con 0x5575f4a84140 2024-06-04T05:13:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.808+0000 7f55d77fe640 1 -- 172.21.15.70:0/2870022880 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f55ec0a8900 con 0x5575f4a84140 2024-06-04T05:13:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.808+0000 7f560119e740 10 client.28637 fetch_fsmap learned FSMap version 50 2024-06-04T05:13:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.808+0000 7f560119e740 1 -- 172.21.15.70:0/2870022880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x5575f4a9d450 con 0x5575f4a84140 2024-06-04T05:13:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.808+0000 7f55d77fe640 1 -- 172.21.15.70:0/2870022880 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f55f0036070 con 0x5575f4a84140 2024-06-04T05:13:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.808+0000 7f55d77fe640 1 -- 172.21.15.70:0/2870022880 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 50) ==== 8386+0+0 (secure 0 0 0) 0x7f55ec0c1070 con 0x5575f4a84140 2024-06-04T05:13:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.808+0000 7f560119e740 10 client.28637 fetch_fsmap finished waiting for FSMap version 50 2024-06-04T05:13:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.808+0000 7f560119e740 10 client.28637 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.808+0000 7f560119e740 20 client.28637 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.808+0000 7f560119e740 1 --2- 172.21.15.70:0/2870022880 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5575f4a9ea00 0x5575f4aa0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.808+0000 7f560119e740 4 client.28637 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:13:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.808+0000 7f560119e740 1 -- 172.21.15.70:0/2870022880 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575f49f7b80 con 0x5575f4a9ea00 2024-06-04T05:13:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.808+0000 7f55f7fff640 1 --2- 172.21.15.70:0/2870022880 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5575f4a9ea00 0x5575f4aa0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.809+0000 7f55f7fff640 1 --2- 172.21.15.70:0/2870022880 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5575f4a9ea00 0x5575f4aa0e20 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc22d68b55dec8b5 server_cookie=1f327339d58c5e6b in_seq=0 out_seq=0 2024-06-04T05:13:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.809+0000 7f55d77fe640 10 client.28637 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.810+0000 7f55d77fe640 1 -- 172.21.15.70:0/2870022880 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5575f49f7b80 con 0x5575f4a9ea00 2024-06-04T05:13:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.810+0000 7f55d77fe640 10 client.28637 handle_command_reply: tid=0 2024-06-04T05:13:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.810+0000 7f560119e740 1 client.28637 shutdown 2024-06-04T05:13:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.810+0000 7f560119e740 1 -- 172.21.15.70:0/2870022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a84140 msgr2=0x5575f4a8bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.810+0000 7f560119e740 1 --2- 172.21.15.70:0/2870022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a84140 0x5575f4a8bfe0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f55ec0df260 tx=0x7f55ec0a5300 comp rx=0 tx=0).stop 2024-06-04T05:13:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.810+0000 7f560119e740 1 -- 172.21.15.70:0/2870022880 shutdown_connections 2024-06-04T05:13:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.810+0000 7f560119e740 1 --2- 172.21.15.70:0/2870022880 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5575f4a9ea00 0x5575f4aa0e20 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.810+0000 7f560119e740 1 --2- 172.21.15.70:0/2870022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a84140 0x5575f4a8bfe0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.810+0000 7f560119e740 1 --2- 172.21.15.70:0/2870022880 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a8ea00 0x5575f4a8c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.810+0000 7f560119e740 1 --2- 172.21.15.70:0/2870022880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a83820 0x5575f4a8baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.810+0000 7f560119e740 1 -- 172.21.15.70:0/2870022880 >> 172.21.15.70:0/2870022880 conn(0x5575f4a7fd30 msgr2=0x5575f4a7efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.810+0000 7f55d77fe640 0 client.28637 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.811+0000 7f560119e740 1 -- 172.21.15.70:0/2870022880 shutdown_connections 2024-06-04T05:13:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.811+0000 7f560119e740 1 -- 172.21.15.70:0/2870022880 wait complete. 2024-06-04T05:13:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.811+0000 7f560119e740 20 client.28637 trim_cache size 0 max 16384 2024-06-04T05:13:54.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.834+0000 7f560119e740 1 Processor -- start 2024-06-04T05:13:54.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.834+0000 7f560119e740 1 -- start start 2024-06-04T05:13:54.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.834+0000 7f560119e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a83820 0x5575f4a7f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.834+0000 7f560119e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a84140 0x5575f4a8c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.834+0000 7f560119e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a8ea00 0x5575f4aa66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.834+0000 7f560119e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f55f81b4b90 con 0x5575f4a83820 2024-06-04T05:13:54.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.834+0000 7f560119e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f55f81b67f0 con 0x5575f4a8ea00 2024-06-04T05:13:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.834+0000 7f560119e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f55f81b52c0 con 0x5575f4a84140 2024-06-04T05:13:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.834+0000 7f55f7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a83820 0x5575f4a7f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.834+0000 7f55f7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a83820 0x5575f4a7f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44230/0 (socket says 172.21.15.70:44230) 2024-06-04T05:13:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.834+0000 7f55f7fff640 1 -- 172.21.15.70:0/4090257578 learned_addr learned my addr 172.21.15.70:0/4090257578 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.834+0000 7f55fca22640 1 --2- 172.21.15.70:0/4090257578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a8ea00 0x5575f4aa66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.834+0000 7f55f77fe640 1 --2- 172.21.15.70:0/4090257578 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a84140 0x5575f4a8c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.835+0000 7f55f7fff640 1 -- 172.21.15.70:0/4090257578 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a84140 msgr2=0x5575f4a8c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.835+0000 7f55f7fff640 1 --2- 172.21.15.70:0/4090257578 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a84140 0x5575f4a8c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.835+0000 7f55f7fff640 1 -- 172.21.15.70:0/4090257578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a8ea00 msgr2=0x5575f4aa66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.835+0000 7f55f7fff640 1 --2- 172.21.15.70:0/4090257578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a8ea00 0x5575f4aa66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.835+0000 7f55f7fff640 1 -- 172.21.15.70:0/4090257578 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575f4a7b6f0 con 0x5575f4a83820 2024-06-04T05:13:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.835+0000 7f55f7fff640 1 --2- 172.21.15.70:0/4090257578 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a83820 0x5575f4a7f730 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f55e8012ef0 tx=0x7f55e8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8043b593b9b72da server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.835+0000 7f55ddffb640 1 -- 172.21.15.70:0/4090257578 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e8004030 con 0x5575f4a83820 2024-06-04T05:13:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.835+0000 7f55ddffb640 1 -- 172.21.15.70:0/4090257578 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f55e8005020 con 0x5575f4a83820 2024-06-04T05:13:54.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.835+0000 7f55ddffb640 1 -- 172.21.15.70:0/4090257578 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e8004b60 con 0x5575f4a83820 2024-06-04T05:13:54.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.836+0000 7f560119e740 1 -- 172.21.15.70:0/4090257578 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a83820 msgr2=0x5575f4a7f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:54.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.836+0000 7f560119e740 1 --2- 172.21.15.70:0/4090257578 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a83820 0x5575f4a7f730 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f55e8012ef0 tx=0x7f55e8019000 comp rx=0 tx=0).stop 2024-06-04T05:13:54.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.836+0000 7f560119e740 1 -- 172.21.15.70:0/4090257578 shutdown_connections 2024-06-04T05:13:54.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.836+0000 7f560119e740 1 --2- 172.21.15.70:0/4090257578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a8ea00 0x5575f4aa66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.836+0000 7f560119e740 1 --2- 172.21.15.70:0/4090257578 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a84140 0x5575f4a8c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.836+0000 7f560119e740 1 --2- 172.21.15.70:0/4090257578 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a83820 0x5575f4a7f730 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.836+0000 7f560119e740 1 -- 172.21.15.70:0/4090257578 >> 172.21.15.70:0/4090257578 conn(0x5575f4a7fd30 msgr2=0x5575f4a85ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:54.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.837+0000 7f560119e740 1 -- 172.21.15.70:0/4090257578 shutdown_connections 2024-06-04T05:13:54.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.837+0000 7f560119e740 1 -- 172.21.15.70:0/4090257578 wait complete. 2024-06-04T05:13:54.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.837+0000 7f560119e740 1 -- start start 2024-06-04T05:13:54.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.837+0000 7f560119e740 1 Processor -- start 2024-06-04T05:13:54.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f560119e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a84140 0x5575f4a925f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f560119e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a8ea00 0x5575f4a92b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f560119e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a93070 0x5575f4b085c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f560119e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f55f81b60f0 con 0x5575f4a84140 2024-06-04T05:13:54.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f560119e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f55f81b59f0 con 0x5575f4a93070 2024-06-04T05:13:54.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f560119e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f55f81b2b70 con 0x5575f4a8ea00 2024-06-04T05:13:54.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f560119e740 10 client.0 _ll_register_callbacks cb 0x7f55ec004370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:13:54.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f55f7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a84140 0x5575f4a925f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:54.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f55f77fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a8ea00 0x5575f4a92b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:54.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f55f7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a84140 0x5575f4a925f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44236/0 (socket says 172.21.15.70:44236) 2024-06-04T05:13:54.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f55f7fff640 1 -- 172.21.15.70:0/1709674011 learned_addr learned my addr 172.21.15.70:0/1709674011 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:13:54.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f55fca22640 1 --2- 172.21.15.70:0/1709674011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a93070 0x5575f4b085c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f55f7fff640 1 -- 172.21.15.70:0/1709674011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a8ea00 msgr2=0x5575f4a92b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f55f7fff640 1 --2- 172.21.15.70:0/1709674011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a8ea00 0x5575f4a92b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f55f7fff640 1 -- 172.21.15.70:0/1709674011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a93070 msgr2=0x5575f4b085c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:13:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f55f7fff640 1 --2- 172.21.15.70:0/1709674011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a93070 0x5575f4b085c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f55f7fff640 1 -- 172.21.15.70:0/1709674011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575f4b08b00 con 0x5575f4a84140 2024-06-04T05:13:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f55f7fff640 1 --2- 172.21.15.70:0/1709674011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a84140 0x5575f4a925f0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f55e802b000 tx=0x7f55e8009df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae13494e301bee2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:13:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.838+0000 7f55d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:13:54.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.839+0000 7f55d77fe640 1 -- 172.21.15.70:0/1709674011 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e80054d0 con 0x5575f4a84140 2024-06-04T05:13:54.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.839+0000 7f55d77fe640 1 -- 172.21.15.70:0/1709674011 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f55e8004030 con 0x5575f4a84140 2024-06-04T05:13:54.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.839+0000 7f560119e740 1 -- 172.21.15.70:0/1709674011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55e8005020 con 0x5575f4a84140 2024-06-04T05:13:54.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.839+0000 7f55d77fe640 1 -- 172.21.15.70:0/1709674011 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f55e8044050 con 0x5575f4a84140 2024-06-04T05:13:54.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.839+0000 7f560119e740 10 client.19803 fetch_fsmap learned FSMap version 50 2024-06-04T05:13:54.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.839+0000 7f560119e740 1 -- 172.21.15.70:0/1709674011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x5575f4b09d70 con 0x5575f4a84140 2024-06-04T05:13:54.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.839+0000 7f55d77fe640 1 -- 172.21.15.70:0/1709674011 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f55e8005020 con 0x5575f4a84140 2024-06-04T05:13:54.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.839+0000 7f55d77fe640 1 -- 172.21.15.70:0/1709674011 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 50) ==== 8386+0+0 (secure 0 0 0) 0x7f55e8032040 con 0x5575f4a84140 2024-06-04T05:13:54.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.839+0000 7f560119e740 10 client.19803 fetch_fsmap finished waiting for FSMap version 50 2024-06-04T05:13:54.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.839+0000 7f560119e740 10 client.19803 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:13:54.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.839+0000 7f560119e740 20 client.19803 populate_metadata read hostname 'smithi070' 2024-06-04T05:13:54.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.839+0000 7f560119e740 1 --2- 172.21.15.70:0/1709674011 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5575f4b0a4d0 0x5575f4b0c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:13:54.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.839+0000 7f560119e740 4 client.19803 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:13:54.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.839+0000 7f560119e740 1 -- 172.21.15.70:0/1709674011 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5575f49f1360 con 0x5575f4b0a4d0 2024-06-04T05:13:54.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.840+0000 7f55f77fe640 1 --2- 172.21.15.70:0/1709674011 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5575f4b0a4d0 0x5575f4b0c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:13:54.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.840+0000 7f55d77fe640 10 client.19803 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:54.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.840+0000 7f55f77fe640 1 --2- 172.21.15.70:0/1709674011 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5575f4b0a4d0 0x5575f4b0c8f0 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f30ada7ccf188fc server_cookie=3d9315421065ef2f in_seq=0 out_seq=0 2024-06-04T05:13:54.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.910+0000 7f55d77fe640 1 -- 172.21.15.70:0/1709674011 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5575f49f1360 con 0x5575f4b0a4d0 2024-06-04T05:13:54.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.910+0000 7f55d77fe640 10 client.19803 handle_command_reply: tid=0 2024-06-04T05:13:54.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.910+0000 7f560119e740 1 client.19803 shutdown 2024-06-04T05:13:54.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.910+0000 7f560119e740 1 -- 172.21.15.70:0/1709674011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a84140 msgr2=0x5575f4a925f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:54.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.910+0000 7f560119e740 1 --2- 172.21.15.70:0/1709674011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a84140 0x5575f4a925f0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f55e802b000 tx=0x7f55e8009df0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.910+0000 7f560119e740 1 -- 172.21.15.70:0/1709674011 shutdown_connections 2024-06-04T05:13:54.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.910+0000 7f560119e740 1 --2- 172.21.15.70:0/1709674011 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5575f4b0a4d0 0x5575f4b0c8f0 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.910+0000 7f560119e740 1 --2- 172.21.15.70:0/1709674011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575f4a93070 0x5575f4b085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.910+0000 7f55d77fe640 0 client.19803 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:13:54.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.910+0000 7f560119e740 1 --2- 172.21.15.70:0/1709674011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5575f4a8ea00 0x5575f4a92b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.910+0000 7f560119e740 1 --2- 172.21.15.70:0/1709674011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f4a84140 0x5575f4a925f0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.910+0000 7f560119e740 1 -- 172.21.15.70:0/1709674011 >> 172.21.15.70:0/1709674011 conn(0x5575f4a7fd30 msgr2=0x5575f4a99620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:54.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.910+0000 7f560119e740 1 -- 172.21.15.70:0/1709674011 shutdown_connections 2024-06-04T05:13:54.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.910+0000 7f560119e740 1 -- 172.21.15.70:0/1709674011 wait complete. 2024-06-04T05:13:54.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.911+0000 7f560119e740 20 client.19803 trim_cache size 0 max 16384 2024-06-04T05:13:54.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:13:54.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:13:54.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:13:54.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.911+0000 7f55fe485640 1 -- 172.21.15.70:0/2497679064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f55e00b2a90 msgr2=0x7f55e00b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:54.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.911+0000 7f55fe485640 1 --2- 172.21.15.70:0/2497679064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f55e00b2a90 0x7f55e00b4f50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f55e8009fd0 tx=0x7f55e8005f90 comp rx=0 tx=0).stop 2024-06-04T05:13:54.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.911+0000 7f55fe485640 1 -- 172.21.15.70:0/2497679064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55f8102340 msgr2=0x7f55f81971a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:13:54.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.911+0000 7f55fe485640 1 --2- 172.21.15.70:0/2497679064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55f8102340 0x7f55f81971a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f55ec009800 tx=0x7f55ec009cc0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.911+0000 7f55fe485640 1 -- 172.21.15.70:0/2497679064 shutdown_connections 2024-06-04T05:13:54.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.911+0000 7f55fe485640 1 --2- 172.21.15.70:0/2497679064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f55e00b2a90 0x7f55e00b4f50 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.911+0000 7f55fe485640 1 --2- 172.21.15.70:0/2497679064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f8107940 0x7f55f81976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.911+0000 7f55fe485640 1 --2- 172.21.15.70:0/2497679064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55f8102340 0x7f55f81971a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.911+0000 7f55fe485640 1 --2- 172.21.15.70:0/2497679064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8101980 0x7f55f8196c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:13:54.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.911+0000 7f55fe485640 1 -- 172.21.15.70:0/2497679064 >> 172.21.15.70:0/2497679064 conn(0x7f55f8078ad0 msgr2=0x7f55f8104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:13:54.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.912+0000 7f55fe485640 1 -- 172.21.15.70:0/2497679064 shutdown_connections 2024-06-04T05:13:54.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:13:54.912+0000 7f55fe485640 1 -- 172.21.15.70:0/2497679064 wait complete. 2024-06-04T05:13:55.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:55 smithi070 ceph-mon[32631]: pgmap v523: 97 pgs: 97 active+clean; 503 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 4.3 MiB/s wr, 230 op/s 2024-06-04T05:13:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:55 smithi112 ceph-mon[36453]: pgmap v523: 97 pgs: 97 active+clean; 503 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 4.3 MiB/s wr, 230 op/s 2024-06-04T05:13:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:55 smithi098 ceph-mon[35569]: pgmap v523: 97 pgs: 97 active+clean; 503 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 4.3 MiB/s wr, 230 op/s 2024-06-04T05:13:55.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 24430 9.11 MB/sec execute 47 sec latency 93.828 ms 2024-06-04T05:13:56.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 24615 9.14 MB/sec execute 48 sec latency 64.735 ms 2024-06-04T05:13:57.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:57 smithi070 ceph-mon[32631]: pgmap v524: 97 pgs: 97 active+clean; 506 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.9 MiB/s wr, 405 op/s 2024-06-04T05:13:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:57 smithi112 ceph-mon[36453]: pgmap v524: 97 pgs: 97 active+clean; 506 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.9 MiB/s wr, 405 op/s 2024-06-04T05:13:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:57 smithi098 ceph-mon[35569]: pgmap v524: 97 pgs: 97 active+clean; 506 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.9 MiB/s wr, 405 op/s 2024-06-04T05:13:57.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 24783 9.10 MB/sec execute 49 sec latency 52.930 ms 2024-06-04T05:13:58.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:58 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:58.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:58 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:58.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:58 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:58.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 24974 9.08 MB/sec execute 50 sec latency 58.589 ms 2024-06-04T05:13:59.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:59 smithi070 ceph-mon[32631]: pgmap v525: 97 pgs: 97 active+clean; 506 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 278 op/s 2024-06-04T05:13:59.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:13:59 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:59 smithi112 ceph-mon[36453]: pgmap v525: 97 pgs: 97 active+clean; 506 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 278 op/s 2024-06-04T05:13:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:13:59 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:59 smithi098 ceph-mon[35569]: pgmap v525: 97 pgs: 97 active+clean; 506 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 278 op/s 2024-06-04T05:13:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:13:59 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:13:59.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 25176 9.08 MB/sec execute 51 sec latency 50.416 ms 2024-06-04T05:14:00.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 25349 9.00 MB/sec execute 52 sec latency 94.744 ms 2024-06-04T05:14:01.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:01 smithi070 ceph-mon[32631]: pgmap v526: 97 pgs: 97 active+clean; 506 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 278 op/s 2024-06-04T05:14:01.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:01 smithi112 ceph-mon[36453]: pgmap v526: 97 pgs: 97 active+clean; 506 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 278 op/s 2024-06-04T05:14:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:01 smithi098 ceph-mon[35569]: pgmap v526: 97 pgs: 97 active+clean; 506 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 278 op/s 2024-06-04T05:14:01.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 25545 9.03 MB/sec execute 53 sec latency 53.933 ms 2024-06-04T05:14:02.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 25723 9.00 MB/sec execute 54 sec latency 53.086 ms 2024-06-04T05:14:03.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:03 smithi070 ceph-mon[32631]: pgmap v527: 97 pgs: 97 active+clean; 499 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.6 MiB/s wr, 448 op/s 2024-06-04T05:14:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:03 smithi112 ceph-mon[36453]: pgmap v527: 97 pgs: 97 active+clean; 499 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.6 MiB/s wr, 448 op/s 2024-06-04T05:14:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:03 smithi098 ceph-mon[35569]: pgmap v527: 97 pgs: 97 active+clean; 499 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.6 MiB/s wr, 448 op/s 2024-06-04T05:14:03.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 25929 9.08 MB/sec execute 55 sec latency 66.329 ms 2024-06-04T05:14:04.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:04 smithi070 ceph-mon[32631]: osdmap e101: 12 total, 12 up, 12 in 2024-06-04T05:14:04.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:04 smithi112 ceph-mon[36453]: osdmap e101: 12 total, 12 up, 12 in 2024-06-04T05:14:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:04 smithi098 ceph-mon[35569]: osdmap e101: 12 total, 12 up, 12 in 2024-06-04T05:14:04.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 26160 9.10 MB/sec execute 56 sec latency 49.933 ms 2024-06-04T05:14:05.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:05 smithi070 ceph-mon[32631]: pgmap v529: 97 pgs: 97 active+clean; 499 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.2 MiB/s wr, 414 op/s 2024-06-04T05:14:05.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:05 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:14:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:05 smithi112 ceph-mon[36453]: pgmap v529: 97 pgs: 97 active+clean; 499 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.2 MiB/s wr, 414 op/s 2024-06-04T05:14:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:05 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:14:05.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:05 smithi098 ceph-mon[35569]: pgmap v529: 97 pgs: 97 active+clean; 499 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.2 MiB/s wr, 414 op/s 2024-06-04T05:14:05.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:05 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:14:05.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 26346 9.10 MB/sec execute 57 sec latency 87.948 ms 2024-06-04T05:14:05.945 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-06-04T05:14:05.946 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:14:06.289 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:14:06.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 26533 9.18 MB/sec execute 58 sec latency 157.533 ms 2024-06-04T05:14:07.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:07 smithi070 ceph-mon[32631]: pgmap v530: 97 pgs: 97 active+clean; 521 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 351 op/s 2024-06-04T05:14:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:07 smithi112 ceph-mon[36453]: pgmap v530: 97 pgs: 97 active+clean; 521 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 351 op/s 2024-06-04T05:14:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:07 smithi098 ceph-mon[35569]: pgmap v530: 97 pgs: 97 active+clean; 521 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 351 op/s 2024-06-04T05:14:07.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 26744 9.34 MB/sec execute 59 sec latency 55.995 ms 2024-06-04T05:14:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.047+0000 7f16bb57f640 1 -- 172.21.15.70:0/200916331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b4101910 msgr2=0x7f16b41066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.047+0000 7f16bb57f640 1 --2- 172.21.15.70:0/200916331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b4101910 0x7f16b41066a0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f16ac009ca0 tx=0x7f16ac02a8b0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.047+0000 7f16bb57f640 1 -- 172.21.15.70:0/200916331 shutdown_connections 2024-06-04T05:14:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.047+0000 7f16bb57f640 1 --2- 172.21.15.70:0/200916331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4100f50 0x7f16b41013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.047+0000 7f16bb57f640 1 --2- 172.21.15.70:0/200916331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b4101910 0x7f16b41066a0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.047+0000 7f16bb57f640 1 --2- 172.21.15.70:0/200916331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16b4107a40 0x7f16b4107e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.047+0000 7f16bb57f640 1 -- 172.21.15.70:0/200916331 >> 172.21.15.70:0/200916331 conn(0x7f16b4077ee0 msgr2=0x7f16b40ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.047+0000 7f16bb57f640 1 -- 172.21.15.70:0/200916331 shutdown_connections 2024-06-04T05:14:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.047+0000 7f16bb57f640 1 -- 172.21.15.70:0/200916331 wait complete. 2024-06-04T05:14:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.049+0000 7f16bb57f640 1 Processor -- start 2024-06-04T05:14:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.049+0000 7f16bb57f640 1 -- start start 2024-06-04T05:14:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.049+0000 7f16bb57f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16b4100f50 0x7f16b419f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.049+0000 7f16bb57f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b4101910 0x7f16b419fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.049+0000 7f16bb57f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4107a40 0x7f16b41a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.049+0000 7f16bb57f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f16b41191e0 con 0x7f16b4101910 2024-06-04T05:14:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.049+0000 7f16bb57f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f16b4119060 con 0x7f16b4107a40 2024-06-04T05:14:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.049+0000 7f16bb57f640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f16b4119360 con 0x7f16b4100f50 2024-06-04T05:14:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.049+0000 7f16b9b1c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4107a40 0x7f16b41a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.049+0000 7f16b8b1a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b4101910 0x7f16b419fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.049+0000 7f16b9b1c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4107a40 0x7f16b41a3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59696/0 (socket says 172.21.15.70:59696) 2024-06-04T05:14:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.049+0000 7f16b9b1c640 1 -- 172.21.15.70:0/470408537 learned_addr learned my addr 172.21.15.70:0/470408537 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.049+0000 7f16b931b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16b4100f50 0x7f16b419f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.050+0000 7f16b9b1c640 1 -- 172.21.15.70:0/470408537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16b4100f50 msgr2=0x7f16b419f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.050+0000 7f16b9b1c640 1 --2- 172.21.15.70:0/470408537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16b4100f50 0x7f16b419f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.050+0000 7f16b9b1c640 1 -- 172.21.15.70:0/470408537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b4101910 msgr2=0x7f16b419fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.051+0000 7f16b9b1c640 1 --2- 172.21.15.70:0/470408537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b4101910 0x7f16b419fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.051+0000 7f16b9b1c640 1 -- 172.21.15.70:0/470408537 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b41a45e0 con 0x7f16b4107a40 2024-06-04T05:14:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.051+0000 7f16b8b1a640 1 --2- 172.21.15.70:0/470408537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b4101910 0x7f16b419fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.052+0000 7f16b9b1c640 1 --2- 172.21.15.70:0/470408537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4107a40 0x7f16b41a3ee0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f16ac031030 tx=0x7f16ac003970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.052+0000 7f16aa7fc640 1 -- 172.21.15.70:0/470408537 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac004040 con 0x7f16b4107a40 2024-06-04T05:14:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.052+0000 7f16aa7fc640 1 -- 172.21.15.70:0/470408537 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f16ac040070 con 0x7f16b4107a40 2024-06-04T05:14:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.052+0000 7f16aa7fc640 1 -- 172.21.15.70:0/470408537 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac045970 con 0x7f16b4107a40 2024-06-04T05:14:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.053+0000 7f16bb57f640 1 -- 172.21.15.70:0/470408537 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b41a4870 con 0x7f16b4107a40 2024-06-04T05:14:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.053+0000 7f16bb57f640 1 -- 172.21.15.70:0/470408537 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b4108680 con 0x7f16b4107a40 2024-06-04T05:14:08.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.056+0000 7f16aa7fc640 1 -- 172.21.15.70:0/470408537 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f16ac03c410 con 0x7f16b4107a40 2024-06-04T05:14:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.056+0000 7f16be371740 1 Processor -- start 2024-06-04T05:14:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.056+0000 7f16be371740 1 -- start start 2024-06-04T05:14:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.057+0000 7f16be371740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f34820 0x55fbc2f34c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.057+0000 7f16be371740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f35140 0x55fbc2f3f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.057+0000 7f16be371740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f41df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.057+0000 7f16be371740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fbc2ea6c30 con 0x55fbc2f3fa00 2024-06-04T05:14:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.057+0000 7f16be371740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fbc2e967d0 con 0x55fbc2f35140 2024-06-04T05:14:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.057+0000 7f16be371740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fbc2e9a810 con 0x55fbc2f34820 2024-06-04T05:14:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.057+0000 7f16b8b1a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f34820 0x55fbc2f34c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.057+0000 7f16b8b1a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f34820 0x55fbc2f34c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:44906/0 (socket says 172.21.15.70:44906) 2024-06-04T05:14:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.057+0000 7f16b931b640 1 --2- 172.21.15.70:0/1606656433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f35140 0x55fbc2f3f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.057+0000 7f16b8b1a640 1 -- 172.21.15.70:0/1606656433 learned_addr learned my addr 172.21.15.70:0/1606656433 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.057+0000 7f16b8b1a640 1 --2- 172.21.15.70:0/1606656433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f41df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.057+0000 7f16b931b640 1 -- 172.21.15.70:0/1606656433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f34820 msgr2=0x55fbc2f34c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.057+0000 7f16b931b640 1 --2- 172.21.15.70:0/1606656433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f34820 0x55fbc2f34c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.057+0000 7f16b931b640 1 -- 172.21.15.70:0/1606656433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 msgr2=0x55fbc2f41df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.057+0000 7f16b931b640 1 --2- 172.21.15.70:0/1606656433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f41df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.057+0000 7f16b931b640 1 -- 172.21.15.70:0/1606656433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbc2bab210 con 0x55fbc2f35140 2024-06-04T05:14:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.057+0000 7f16aa7fc640 1 --2- 172.21.15.70:0/470408537 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f16a00b2ba0 0x7f16a00b5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.058+0000 7f16b8b1a640 1 --2- 172.21.15.70:0/470408537 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f16a00b2ba0 0x7f16a00b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.058+0000 7f16b931b640 1 --2- 172.21.15.70:0/1606656433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f35140 0x55fbc2f3f4c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009b40 tx=0x7f16b00113e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c243c592d5b9ce4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.058+0000 7f16aa7fc640 1 -- 172.21.15.70:0/470408537 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f16ac103670 con 0x7f16b4107a40 2024-06-04T05:14:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.058+0000 7f169f7fe640 1 -- 172.21.15.70:0/1606656433 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b0011d90 con 0x55fbc2f35140 2024-06-04T05:14:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.058+0000 7f169f7fe640 1 -- 172.21.15.70:0/1606656433 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f16b00128e0 con 0x55fbc2f35140 2024-06-04T05:14:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.058+0000 7f169f7fe640 1 -- 172.21.15.70:0/1606656433 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b000f840 con 0x55fbc2f35140 2024-06-04T05:14:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.059+0000 7f16b8b1a640 1 --2- 172.21.15.70:0/470408537 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f16a00b2ba0 0x7f16a00b5060 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x55fbc2f32150 tx=0x7f16a4008040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.059+0000 7f16be371740 1 -- 172.21.15.70:0/1606656433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f35140 msgr2=0x55fbc2f3f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.059+0000 7f16be371740 1 --2- 172.21.15.70:0/1606656433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f35140 0x55fbc2f3f4c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009b40 tx=0x7f16b00113e0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.059+0000 7f16be371740 1 -- 172.21.15.70:0/1606656433 shutdown_connections 2024-06-04T05:14:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.059+0000 7f16be371740 1 --2- 172.21.15.70:0/1606656433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f35140 0x55fbc2f3f4c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.059+0000 7f16be371740 1 --2- 172.21.15.70:0/1606656433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f41df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.059+0000 7f16be371740 1 --2- 172.21.15.70:0/1606656433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f34820 0x55fbc2f34c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.059+0000 7f16be371740 1 -- 172.21.15.70:0/1606656433 >> 172.21.15.70:0/1606656433 conn(0x55fbc2f30d30 msgr2=0x55fbc2f31110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.059+0000 7f16be371740 1 -- 172.21.15.70:0/1606656433 shutdown_connections 2024-06-04T05:14:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.059+0000 7f16be371740 1 -- 172.21.15.70:0/1606656433 wait complete. 2024-06-04T05:14:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.060+0000 7f16be371740 1 -- start start 2024-06-04T05:14:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.060+0000 7f16be371740 1 Processor -- start 2024-06-04T05:14:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.060+0000 7f16be371740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f34820 0x55fbc2f3c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.060+0000 7f16be371740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f35140 0x55fbc2f3cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.060+0000 7f16be371740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f3d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.060+0000 7f16be371740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fbc2b7c790 con 0x55fbc2f3fa00 2024-06-04T05:14:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.060+0000 7f16be371740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fbc2e52890 con 0x55fbc2f34820 2024-06-04T05:14:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.060+0000 7f16be371740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fbc2f42630 con 0x55fbc2f35140 2024-06-04T05:14:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.060+0000 7f16be371740 10 client.0 _ll_register_callbacks cb 0x55fbc2baaa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.060+0000 7f16b9b1c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f3d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.060+0000 7f16b9b1c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f3d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60348/0 (socket says 172.21.15.70:60348) 2024-06-04T05:14:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.060+0000 7f16b9b1c640 1 -- 172.21.15.70:0/2582300075 learned_addr learned my addr 172.21.15.70:0/2582300075 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.060+0000 7f16b8b1a640 1 --2- 172.21.15.70:0/2582300075 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f35140 0x55fbc2f3cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.060+0000 7f16b931b640 1 --2- 172.21.15.70:0/2582300075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f34820 0x55fbc2f3c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.061+0000 7f16b9b1c640 1 -- 172.21.15.70:0/2582300075 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f35140 msgr2=0x55fbc2f3cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.061+0000 7f16b9b1c640 1 --2- 172.21.15.70:0/2582300075 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f35140 0x55fbc2f3cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.061+0000 7f16b9b1c640 1 -- 172.21.15.70:0/2582300075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f34820 msgr2=0x55fbc2f3c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.061+0000 7f16b9b1c640 1 --2- 172.21.15.70:0/2582300075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f34820 0x55fbc2f3c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.061+0000 7f16b9b1c640 1 -- 172.21.15.70:0/2582300075 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbc2c77620 con 0x55fbc2f3fa00 2024-06-04T05:14:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.061+0000 7f16b9b1c640 1 --2- 172.21.15.70:0/2582300075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f3d450 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f16ac005e30 tx=0x7f16ac0ca260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c44ed23e94621b6b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.061+0000 7f169cff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:14:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.061+0000 7f169cff9640 1 -- 172.21.15.70:0/2582300075 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac0caa30 con 0x55fbc2f3fa00 2024-06-04T05:14:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.061+0000 7f169cff9640 1 -- 172.21.15.70:0/2582300075 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f16ac0caea0 con 0x55fbc2f3fa00 2024-06-04T05:14:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.061+0000 7f16be371740 1 -- 172.21.15.70:0/2582300075 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16b00128e0 con 0x55fbc2f3fa00 2024-06-04T05:14:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.061+0000 7f169cff9640 1 -- 172.21.15.70:0/2582300075 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f16ac0cede0 con 0x55fbc2f3fa00 2024-06-04T05:14:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.061+0000 7f16be371740 10 client.19815 fetch_fsmap learned FSMap version 53 2024-06-04T05:14:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.061+0000 7f16be371740 1 -- 172.21.15.70:0/2582300075 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55fbc2f4e3d0 con 0x55fbc2f3fa00 2024-06-04T05:14:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.062+0000 7f169cff9640 1 -- 172.21.15.70:0/2582300075 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f16b00128e0 con 0x55fbc2f3fa00 2024-06-04T05:14:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.062+0000 7f169cff9640 1 -- 172.21.15.70:0/2582300075 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 53) ==== 8374+0+0 (secure 0 0 0) 0x7f16ac0ebce0 con 0x55fbc2f3fa00 2024-06-04T05:14:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.062+0000 7f16be371740 10 client.19815 fetch_fsmap finished waiting for FSMap version 53 2024-06-04T05:14:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.062+0000 7f16be371740 10 client.19815 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:14:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.062+0000 7f16be371740 20 client.19815 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.062+0000 7f16be371740 1 --2- 172.21.15.70:0/2582300075 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fbc2f4f830 0x55fbc2f51c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.062+0000 7f16be371740 4 client.19815 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:14:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.062+0000 7f16be371740 1 -- 172.21.15.70:0/2582300075 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbc2ea8b80 con 0x55fbc2f4f830 2024-06-04T05:14:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.062+0000 7f16b931b640 1 --2- 172.21.15.70:0/2582300075 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fbc2f4f830 0x55fbc2f51c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.063+0000 7f16b931b640 1 --2- 172.21.15.70:0/2582300075 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fbc2f4f830 0x55fbc2f51c80 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cdd2f5b272bf29f server_cookie=75bd8b9587399abb in_seq=0 out_seq=0 2024-06-04T05:14:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.063+0000 7f169cff9640 10 client.19815 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.064+0000 7f169cff9640 1 -- 172.21.15.70:0/2582300075 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fbc2ea8b80 con 0x55fbc2f4f830 2024-06-04T05:14:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.064+0000 7f169cff9640 10 client.19815 handle_command_reply: tid=0 2024-06-04T05:14:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.064+0000 7f16be371740 1 client.19815 shutdown 2024-06-04T05:14:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.064+0000 7f16be371740 1 -- 172.21.15.70:0/2582300075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 msgr2=0x55fbc2f3d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.064+0000 7f16be371740 1 --2- 172.21.15.70:0/2582300075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f3d450 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f16ac005e30 tx=0x7f16ac0ca260 comp rx=0 tx=0).stop 2024-06-04T05:14:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.064+0000 7f16be371740 1 -- 172.21.15.70:0/2582300075 shutdown_connections 2024-06-04T05:14:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.064+0000 7f16be371740 1 --2- 172.21.15.70:0/2582300075 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fbc2f4f830 0x55fbc2f51c80 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.064+0000 7f16be371740 1 --2- 172.21.15.70:0/2582300075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f3d450 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.064+0000 7f16be371740 1 --2- 172.21.15.70:0/2582300075 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f35140 0x55fbc2f3cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.064+0000 7f16be371740 1 --2- 172.21.15.70:0/2582300075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f34820 0x55fbc2f3c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.064+0000 7f16be371740 1 -- 172.21.15.70:0/2582300075 >> 172.21.15.70:0/2582300075 conn(0x55fbc2f30d30 msgr2=0x55fbc2f2ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.064+0000 7f169cff9640 0 client.19815 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.064+0000 7f16be371740 1 -- 172.21.15.70:0/2582300075 shutdown_connections 2024-06-04T05:14:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.064+0000 7f16be371740 1 -- 172.21.15.70:0/2582300075 wait complete. 2024-06-04T05:14:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.065+0000 7f16be371740 20 client.19815 trim_cache size 0 max 16384 2024-06-04T05:14:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.090+0000 7f16be371740 1 Processor -- start 2024-06-04T05:14:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.090+0000 7f16be371740 1 -- start start 2024-06-04T05:14:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.091+0000 7f16be371740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f34820 0x55fbc2f3d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.091+0000 7f16be371740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f35140 0x55fbc2f371e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.091+0000 7f16be371740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f4ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.091+0000 7f16be371740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f16b41bd3b0 con 0x55fbc2f3fa00 2024-06-04T05:14:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.091+0000 7f16be371740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f16b41befb0 con 0x55fbc2f35140 2024-06-04T05:14:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.091+0000 7f16b8b1a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f35140 0x55fbc2f371e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.091+0000 7f16b8b1a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f35140 0x55fbc2f371e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59730/0 (socket says 172.21.15.70:59730) 2024-06-04T05:14:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.091+0000 7f16b8b1a640 1 -- 172.21.15.70:0/3952558842 learned_addr learned my addr 172.21.15.70:0/3952558842 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.091+0000 7f16b931b640 1 --2- 172.21.15.70:0/3952558842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f34820 0x55fbc2f3d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.091+0000 7f16b9b1c640 1 --2- 172.21.15.70:0/3952558842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f4ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.091+0000 7f16be371740 1 -- 172.21.15.70:0/3952558842 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f16b41bdae0 con 0x55fbc2f34820 2024-06-04T05:14:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.092+0000 7f16b9b1c640 1 -- 172.21.15.70:0/3952558842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f34820 msgr2=0x55fbc2f3d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.092+0000 7f16b9b1c640 1 --2- 172.21.15.70:0/3952558842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f34820 0x55fbc2f3d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.092+0000 7f16b9b1c640 1 -- 172.21.15.70:0/3952558842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f35140 msgr2=0x55fbc2f371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.092+0000 7f16b9b1c640 1 --2- 172.21.15.70:0/3952558842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f35140 0x55fbc2f371e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.092+0000 7f16b9b1c640 1 -- 172.21.15.70:0/3952558842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbc2f2c6f0 con 0x55fbc2f3fa00 2024-06-04T05:14:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.092+0000 7f16b8b1a640 1 --2- 172.21.15.70:0/3952558842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f35140 0x55fbc2f371e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.092+0000 7f16b9b1c640 1 --2- 172.21.15.70:0/3952558842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f4ad90 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f16ac0c7ca0 tx=0x7f16ac02a8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b73280f04c972020 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.092+0000 7f169effd640 1 -- 172.21.15.70:0/3952558842 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac0caa30 con 0x55fbc2f3fa00 2024-06-04T05:14:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.092+0000 7f169effd640 1 -- 172.21.15.70:0/3952558842 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f16ac036040 con 0x55fbc2f3fa00 2024-06-04T05:14:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.092+0000 7f169effd640 1 -- 172.21.15.70:0/3952558842 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac0d3890 con 0x55fbc2f3fa00 2024-06-04T05:14:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.093+0000 7f16be371740 1 -- 172.21.15.70:0/3952558842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 msgr2=0x55fbc2f4ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.094+0000 7f16be371740 1 --2- 172.21.15.70:0/3952558842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f4ad90 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f16ac0c7ca0 tx=0x7f16ac02a8b0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.094+0000 7f16be371740 1 -- 172.21.15.70:0/3952558842 shutdown_connections 2024-06-04T05:14:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.094+0000 7f16be371740 1 --2- 172.21.15.70:0/3952558842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f35140 0x55fbc2f371e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.094+0000 7f16be371740 1 --2- 172.21.15.70:0/3952558842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f4ad90 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.094+0000 7f16be371740 1 --2- 172.21.15.70:0/3952558842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f34820 0x55fbc2f3d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.094+0000 7f16be371740 1 -- 172.21.15.70:0/3952558842 >> 172.21.15.70:0/3952558842 conn(0x55fbc2f30d30 msgr2=0x55fbc2f4a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.094+0000 7f16be371740 1 -- 172.21.15.70:0/3952558842 shutdown_connections 2024-06-04T05:14:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.094+0000 7f16be371740 1 -- 172.21.15.70:0/3952558842 wait complete. 2024-06-04T05:14:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.095+0000 7f16be371740 1 -- start start 2024-06-04T05:14:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.095+0000 7f16be371740 1 Processor -- start 2024-06-04T05:14:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.095+0000 7f16be371740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f34820 0x55fbc2f43780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.095+0000 7f16be371740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f35140 0x55fbc2f43cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.095+0000 7f16be371740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f44200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.095+0000 7f16be371740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f16b41be8e0 con 0x55fbc2f35140 2024-06-04T05:14:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.095+0000 7f16be371740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f16b41be210 con 0x55fbc2f34820 2024-06-04T05:14:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.095+0000 7f16be371740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f16b40507e0 con 0x55fbc2f3fa00 2024-06-04T05:14:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.096+0000 7f16be371740 10 client.0 _ll_register_callbacks cb 0x7f16ac004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.096+0000 7f16b931b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f34820 0x55fbc2f43780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.096+0000 7f16b931b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f34820 0x55fbc2f43780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59736/0 (socket says 172.21.15.70:59736) 2024-06-04T05:14:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.096+0000 7f16b931b640 1 -- 172.21.15.70:0/2131546934 learned_addr learned my addr 172.21.15.70:0/2131546934 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.096+0000 7f16b9b1c640 1 --2- 172.21.15.70:0/2131546934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f44200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.096+0000 7f16b931b640 1 -- 172.21.15.70:0/2131546934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f3fa00 msgr2=0x55fbc2f44200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.096+0000 7f16b931b640 1 --2- 172.21.15.70:0/2131546934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f44200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.096+0000 7f16b931b640 1 -- 172.21.15.70:0/2131546934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f35140 msgr2=0x55fbc2f43cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.096+0000 7f16b931b640 1 --2- 172.21.15.70:0/2131546934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f35140 0x55fbc2f43cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.096+0000 7f16b931b640 1 -- 172.21.15.70:0/2131546934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbc2fb9660 con 0x55fbc2f34820 2024-06-04T05:14:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.097+0000 7f16b931b640 1 --2- 172.21.15.70:0/2131546934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f34820 0x55fbc2f43780 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009b40 tx=0x7f16b00052e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ce7039c91a1c847 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.097+0000 7f169cff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:14:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.097+0000 7f169cff9640 1 -- 172.21.15.70:0/2131546934 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b00055e0 con 0x55fbc2f34820 2024-06-04T05:14:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.097+0000 7f16be371740 1 -- 172.21.15.70:0/2131546934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16ac036040 con 0x55fbc2f34820 2024-06-04T05:14:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.097+0000 7f169cff9640 1 -- 172.21.15.70:0/2131546934 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f16b00035f0 con 0x55fbc2f34820 2024-06-04T05:14:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.098+0000 7f169cff9640 1 -- 172.21.15.70:0/2131546934 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f16b003b050 con 0x55fbc2f34820 2024-06-04T05:14:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.098+0000 7f16be371740 10 client.28658 fetch_fsmap learned FSMap version 53 2024-06-04T05:14:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.098+0000 7f16be371740 1 -- 172.21.15.70:0/2131546934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55fbc2fba930 con 0x55fbc2f34820 2024-06-04T05:14:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.098+0000 7f169cff9640 1 -- 172.21.15.70:0/2131546934 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f16ac036040 con 0x55fbc2f34820 2024-06-04T05:14:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.098+0000 7f169cff9640 1 -- 172.21.15.70:0/2131546934 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 53) ==== 8374+0+0 (secure 0 0 0) 0x7f16b0012800 con 0x55fbc2f34820 2024-06-04T05:14:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.098+0000 7f16be371740 10 client.28658 fetch_fsmap finished waiting for FSMap version 53 2024-06-04T05:14:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.098+0000 7f16be371740 10 client.28658 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:14:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.099+0000 7f16be371740 20 client.28658 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.099+0000 7f16be371740 1 --2- 172.21.15.70:0/2131546934 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fbc2fbadd0 0x55fbc2fbd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.099+0000 7f16be371740 4 client.28658 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:14:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.099+0000 7f16be371740 1 -- 172.21.15.70:0/2131546934 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fbc2ea2360 con 0x55fbc2fbadd0 2024-06-04T05:14:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.099+0000 7f16b8b1a640 1 --2- 172.21.15.70:0/2131546934 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fbc2fbadd0 0x55fbc2fbd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.100+0000 7f169cff9640 10 client.28658 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.100+0000 7f16b8b1a640 1 --2- 172.21.15.70:0/2131546934 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fbc2fbadd0 0x55fbc2fbd1c0 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf539b9716cc51bd server_cookie=a149085efe6a4ba in_seq=0 out_seq=0 2024-06-04T05:14:08.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.143+0000 7f7a01398700 1 -- 172.21.15.70:0/3106376543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79fc10a540 msgr2=0x7f79fc10a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:08.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.143+0000 7f7a01398700 1 --2- 172.21.15.70:0/3106376543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79fc10a540 0x7f79fc10a9c0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f79e4009990 tx=0x7f79e4009ca0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.144+0000 7f7a01398700 1 -- 172.21.15.70:0/3106376543 shutdown_connections 2024-06-04T05:14:08.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.144+0000 7f7a01398700 1 --2- 172.21.15.70:0/3106376543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79fc10af00 0x7f79fc116e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.144+0000 7f7a01398700 1 --2- 172.21.15.70:0/3106376543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79fc10ef90 0x7f79fc10f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.144+0000 7f7a01398700 1 --2- 172.21.15.70:0/3106376543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79fc10a540 0x7f79fc10a9c0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.144+0000 7f7a01398700 1 -- 172.21.15.70:0/3106376543 >> 172.21.15.70:0/3106376543 conn(0x7f79fc06ea90 msgr2=0x7f79fc06eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:08.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.145+0000 7f7a01398700 1 -- 172.21.15.70:0/3106376543 shutdown_connections 2024-06-04T05:14:08.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.145+0000 7f7a01398700 1 -- 172.21.15.70:0/3106376543 wait complete. 2024-06-04T05:14:08.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.146+0000 7f7a01398700 1 Processor -- start 2024-06-04T05:14:08.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.147+0000 7f7a01398700 1 -- start start 2024-06-04T05:14:08.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.148+0000 7f7a01398700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79fc10a540 0x7f79fc1a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.148+0000 7f7a01398700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79fc10af00 0x7f79fc1a28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.148+0000 7f7a01398700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79fc10ef90 0x7f79fc1a6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.148+0000 7f7a01398700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc11a070 con 0x7f79fc10a540 2024-06-04T05:14:08.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.148+0000 7f7a01398700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc119ef0 con 0x7f79fc10af00 2024-06-04T05:14:08.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.148+0000 7f7a01398700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc11a1f0 con 0x7f79fc10ef90 2024-06-04T05:14:08.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.149+0000 7f79fb7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79fc10ef90 0x7f79fc1a6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.149+0000 7f79fb7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79fc10ef90 0x7f79fc1a6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:44944/0 (socket says 172.21.15.70:44944) 2024-06-04T05:14:08.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.149+0000 7f79fb7fe700 1 -- 172.21.15.70:0/2933017112 learned_addr learned my addr 172.21.15.70:0/2933017112 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:08.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.149+0000 7f79f3fff700 1 --2- 172.21.15.70:0/2933017112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79fc10af00 0x7f79fc1a28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.149+0000 7f79fb7fe700 1 -- 172.21.15.70:0/2933017112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79fc10af00 msgr2=0x7f79fc1a28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:08.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.149+0000 7f79fb7fe700 1 --2- 172.21.15.70:0/2933017112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79fc10af00 0x7f79fc1a28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.149+0000 7f79fb7fe700 1 -- 172.21.15.70:0/2933017112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79fc10a540 msgr2=0x7f79fc1a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:08.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.149+0000 7f79fb7fe700 1 --2- 172.21.15.70:0/2933017112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79fc10a540 0x7f79fc1a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.149+0000 7f79fb7fe700 1 -- 172.21.15.70:0/2933017112 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79fc1a7390 con 0x7f79fc10ef90 2024-06-04T05:14:08.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.150+0000 7f79fb7fe700 1 --2- 172.21.15.70:0/2933017112 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79fc10ef90 0x7f79fc1a6c50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f79f400b9e0 tx=0x7f79f400bda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:08.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.150+0000 7f79f8ff9700 1 -- 172.21.15.70:0/2933017112 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f4019070 con 0x7f79fc10ef90 2024-06-04T05:14:08.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.150+0000 7f7a01398700 1 -- 172.21.15.70:0/2933017112 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79fc1a7680 con 0x7f79fc10ef90 2024-06-04T05:14:08.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.150+0000 7f79f8ff9700 1 -- 172.21.15.70:0/2933017112 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f79f400ce40 con 0x7f79fc10ef90 2024-06-04T05:14:08.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.150+0000 7f79f8ff9700 1 -- 172.21.15.70:0/2933017112 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f4012340 con 0x7f79fc10ef90 2024-06-04T05:14:08.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.150+0000 7f7a01398700 1 -- 172.21.15.70:0/2933017112 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79fc1aef00 con 0x7f79fc10ef90 2024-06-04T05:14:08.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.154+0000 7f7a01398700 1 -- 172.21.15.70:0/2933017112 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79fc06a230 con 0x7f79fc10ef90 2024-06-04T05:14:08.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.157+0000 7f79f8ff9700 1 -- 172.21.15.70:0/2933017112 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f79f401c070 con 0x7f79fc10ef90 2024-06-04T05:14:08.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.158+0000 7f79f8ff9700 1 --2- 172.21.15.70:0/2933017112 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f79e80b2af0 0x7f79e80b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:08.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.158+0000 7f79f8ff9700 1 -- 172.21.15.70:0/2933017112 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f79f40e1a20 con 0x7f79fc10ef90 2024-06-04T05:14:08.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.159+0000 7f79faffd700 1 --2- 172.21.15.70:0/2933017112 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f79e80b2af0 0x7f79e80b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:08.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.161+0000 7f79faffd700 1 --2- 172.21.15.70:0/2933017112 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f79e80b2af0 0x7f79e80b4fb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f79ec005950 tx=0x7f79ec0058e0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:08.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.166+0000 7f79f8ff9700 1 -- 172.21.15.70:0/2933017112 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f79f40a9480 con 0x7f79fc10ef90 2024-06-04T05:14:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.175+0000 7f169cff9640 1 -- 172.21.15.70:0/2131546934 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55fbc2ea2360 con 0x55fbc2fbadd0 2024-06-04T05:14:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.175+0000 7f169cff9640 10 client.28658 handle_command_reply: tid=0 2024-06-04T05:14:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.175+0000 7f16be371740 1 client.28658 shutdown 2024-06-04T05:14:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.176+0000 7f16be371740 1 -- 172.21.15.70:0/2131546934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f34820 msgr2=0x55fbc2f43780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.176+0000 7f16be371740 1 --2- 172.21.15.70:0/2131546934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f34820 0x55fbc2f43780 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009b40 tx=0x7f16b00052e0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.176+0000 7f16be371740 1 -- 172.21.15.70:0/2131546934 shutdown_connections 2024-06-04T05:14:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.176+0000 7f16be371740 1 --2- 172.21.15.70:0/2131546934 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fbc2fbadd0 0x55fbc2fbd1c0 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.176+0000 7f16be371740 1 --2- 172.21.15.70:0/2131546934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbc2f3fa00 0x55fbc2f44200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.176+0000 7f169cff9640 0 client.28658 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.176+0000 7f16be371740 1 --2- 172.21.15.70:0/2131546934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbc2f35140 0x55fbc2f43cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.176+0000 7f16be371740 1 --2- 172.21.15.70:0/2131546934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbc2f34820 0x55fbc2f43780 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.176+0000 7f16be371740 1 -- 172.21.15.70:0/2131546934 >> 172.21.15.70:0/2131546934 conn(0x55fbc2f30d30 msgr2=0x55fbc2f4af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.176+0000 7f16be371740 1 -- 172.21.15.70:0/2131546934 shutdown_connections 2024-06-04T05:14:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.176+0000 7f16be371740 1 -- 172.21.15.70:0/2131546934 wait complete. 2024-06-04T05:14:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.177+0000 7f16be371740 20 client.28658 trim_cache size 0 max 16384 2024-06-04T05:14:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:14:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:14:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:14:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.177+0000 7f16bb57f640 1 -- 172.21.15.70:0/470408537 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f16a00b2ba0 msgr2=0x7f16a00b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.177+0000 7f16bb57f640 1 --2- 172.21.15.70:0/470408537 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f16a00b2ba0 0x7f16a00b5060 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x55fbc2f32150 tx=0x7f16a4008040 comp rx=0 tx=0).stop 2024-06-04T05:14:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.177+0000 7f16bb57f640 1 -- 172.21.15.70:0/470408537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4107a40 msgr2=0x7f16b41a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.177+0000 7f16bb57f640 1 --2- 172.21.15.70:0/470408537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4107a40 0x7f16b41a3ee0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f16ac031030 tx=0x7f16ac003970 comp rx=0 tx=0).stop 2024-06-04T05:14:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.177+0000 7f16bb57f640 1 -- 172.21.15.70:0/470408537 shutdown_connections 2024-06-04T05:14:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.177+0000 7f16bb57f640 1 --2- 172.21.15.70:0/470408537 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f16a00b2ba0 0x7f16a00b5060 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.177+0000 7f16bb57f640 1 --2- 172.21.15.70:0/470408537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4107a40 0x7f16b41a3ee0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.177+0000 7f16bb57f640 1 --2- 172.21.15.70:0/470408537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b4101910 0x7f16b419fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.177+0000 7f16bb57f640 1 --2- 172.21.15.70:0/470408537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16b4100f50 0x7f16b419f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.177+0000 7f16bb57f640 1 -- 172.21.15.70:0/470408537 >> 172.21.15.70:0/470408537 conn(0x7f16b4077ee0 msgr2=0x7f16b40fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.178+0000 7f16bb57f640 1 -- 172.21.15.70:0/470408537 shutdown_connections 2024-06-04T05:14:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:08.178+0000 7f16bb57f640 1 -- 172.21.15.70:0/470408537 wait complete. 2024-06-04T05:14:08.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.593+0000 7f7a01398700 1 -- 172.21.15.70:0/2933017112 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79fc1af1a0 con 0x7f79fc10ef90 2024-06-04T05:14:08.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.594+0000 7f79f8ff9700 1 -- 172.21.15.70:0/2933017112 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11936 (secure 0 0 0) 0x7f79f40a8d80 con 0x7f79fc10ef90 2024-06-04T05:14:08.598 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:14:08.598 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":53,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:14:04.316709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T05:14:08.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.598+0000 7f7a01398700 1 -- 172.21.15.70:0/2933017112 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f79e80b2af0 msgr2=0x7f79e80b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:08.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.598+0000 7f7a01398700 1 --2- 172.21.15.70:0/2933017112 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f79e80b2af0 0x7f79e80b4fb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f79ec005950 tx=0x7f79ec0058e0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.598+0000 7f7a01398700 1 -- 172.21.15.70:0/2933017112 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79fc10ef90 msgr2=0x7f79fc1a6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:08.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.598+0000 7f7a01398700 1 --2- 172.21.15.70:0/2933017112 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79fc10ef90 0x7f79fc1a6c50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f79f400b9e0 tx=0x7f79f400bda0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.599+0000 7f7a01398700 1 -- 172.21.15.70:0/2933017112 shutdown_connections 2024-06-04T05:14:08.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.599+0000 7f7a01398700 1 --2- 172.21.15.70:0/2933017112 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f79e80b2af0 0x7f79e80b4fb0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.599+0000 7f7a01398700 1 --2- 172.21.15.70:0/2933017112 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79fc10ef90 0x7f79fc1a6c50 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.599+0000 7f7a01398700 1 --2- 172.21.15.70:0/2933017112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79fc10a540 0x7f79fc1a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.599+0000 7f7a01398700 1 --2- 172.21.15.70:0/2933017112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79fc10af00 0x7f79fc1a28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:08.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.599+0000 7f7a01398700 1 -- 172.21.15.70:0/2933017112 >> 172.21.15.70:0/2933017112 conn(0x7f79fc06ea90 msgr2=0x7f79fc0740d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:08.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.599+0000 7f7a01398700 1 -- 172.21.15.70:0/2933017112 shutdown_connections 2024-06-04T05:14:08.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:08.599+0000 7f7a01398700 1 -- 172.21.15.70:0/2933017112 wait complete. 2024-06-04T05:14:08.603 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 53 2024-06-04T05:14:08.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 26883 9.34 MB/sec execute 60 sec latency 104.217 ms 2024-06-04T05:14:09.235 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:08 smithi070 ceph-mon[32631]: osdmap e102: 12 total, 12 up, 12 in 2024-06-04T05:14:09.236 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:08 smithi070 ceph-mon[32631]: pgmap v532: 97 pgs: 97 active+clean; 521 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.7 MiB/s wr, 439 op/s 2024-06-04T05:14:09.236 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:08 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2933017112' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:14:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:08 smithi098 ceph-mon[35569]: osdmap e102: 12 total, 12 up, 12 in 2024-06-04T05:14:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:08 smithi098 ceph-mon[35569]: pgmap v532: 97 pgs: 97 active+clean; 521 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.7 MiB/s wr, 439 op/s 2024-06-04T05:14:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:08 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2933017112' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:14:09.271 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 130 --await-for 45.0' 2024-06-04T05:14:09.271 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 130 --await-for 45.0 2024-06-04T05:14:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:08 smithi112 ceph-mon[36453]: osdmap e102: 12 total, 12 up, 12 in 2024-06-04T05:14:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:08 smithi112 ceph-mon[36453]: pgmap v532: 97 pgs: 97 active+clean; 521 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.7 MiB/s wr, 439 op/s 2024-06-04T05:14:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:08 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2933017112' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:14:09.617 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:14:09.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 27036 9.35 MB/sec execute 61 sec latency 57.141 ms 2024-06-04T05:14:10.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 27211 9.42 MB/sec execute 62 sec latency 125.805 ms 2024-06-04T05:14:11.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:11 smithi098 ceph-mon[35569]: pgmap v533: 97 pgs: 97 active+clean; 521 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 3.2 MiB/s wr, 183 op/s 2024-06-04T05:14:11.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:11 smithi070 ceph-mon[32631]: pgmap v533: 97 pgs: 97 active+clean; 521 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 3.2 MiB/s wr, 183 op/s 2024-06-04T05:14:11.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:11 smithi112 ceph-mon[36453]: pgmap v533: 97 pgs: 97 active+clean; 521 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 3.2 MiB/s wr, 183 op/s 2024-06-04T05:14:11.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 27336 9.34 MB/sec execute 63 sec latency 164.113 ms 2024-06-04T05:14:12.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 27466 9.27 MB/sec execute 64 sec latency 222.623 ms 2024-06-04T05:14:13.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:13 smithi070 ceph-mon[32631]: pgmap v534: 97 pgs: 97 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.6 MiB/s wr, 345 op/s 2024-06-04T05:14:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:13 smithi112 ceph-mon[36453]: pgmap v534: 97 pgs: 97 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.6 MiB/s wr, 345 op/s 2024-06-04T05:14:13.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:13 smithi098 ceph-mon[35569]: pgmap v534: 97 pgs: 97 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.6 MiB/s wr, 345 op/s 2024-06-04T05:14:13.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 27589 9.21 MB/sec execute 65 sec latency 301.315 ms 2024-06-04T05:14:14.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 27724 9.17 MB/sec execute 66 sec latency 112.439 ms 2024-06-04T05:14:15.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:15 smithi070 ceph-mon[32631]: pgmap v535: 97 pgs: 97 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.8 MiB/s wr, 305 op/s 2024-06-04T05:14:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:15 smithi112 ceph-mon[36453]: pgmap v535: 97 pgs: 97 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.8 MiB/s wr, 305 op/s 2024-06-04T05:14:15.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:15 smithi098 ceph-mon[35569]: pgmap v535: 97 pgs: 97 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.8 MiB/s wr, 305 op/s 2024-06-04T05:14:15.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 27880 9.15 MB/sec execute 67 sec latency 109.931 ms 2024-06-04T05:14:16.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:16 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:14:16.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:16 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:14:16.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:16 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:14:16.804 INFO:tasks.workunit.client.0.smithi070.stdout: 10 28056 9.26 MB/sec execute 68 sec latency 138.339 ms 2024-06-04T05:14:17.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:17 smithi070 ceph-mon[32631]: pgmap v536: 97 pgs: 97 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.6 MiB/s wr, 295 op/s 2024-06-04T05:14:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:17 smithi112 ceph-mon[36453]: pgmap v536: 97 pgs: 97 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.6 MiB/s wr, 295 op/s 2024-06-04T05:14:17.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:17 smithi098 ceph-mon[35569]: pgmap v536: 97 pgs: 97 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.6 MiB/s wr, 295 op/s 2024-06-04T05:14:17.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 28200 9.17 MB/sec execute 69 sec latency 152.725 ms 2024-06-04T05:14:18.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 28338 9.13 MB/sec execute 70 sec latency 183.125 ms 2024-06-04T05:14:19.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:19 smithi070 ceph-mon[32631]: pgmap v537: 97 pgs: 97 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.5 MiB/s wr, 288 op/s 2024-06-04T05:14:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:19 smithi112 ceph-mon[36453]: pgmap v537: 97 pgs: 97 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.5 MiB/s wr, 288 op/s 2024-06-04T05:14:19.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:19 smithi098 ceph-mon[35569]: pgmap v537: 97 pgs: 97 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.5 MiB/s wr, 288 op/s 2024-06-04T05:14:19.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 28471 9.08 MB/sec execute 71 sec latency 128.734 ms 2024-06-04T05:14:20.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:20 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:14:20.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:20 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:14:20.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:20 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:14:20.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 28624 8.99 MB/sec execute 72 sec latency 148.394 ms 2024-06-04T05:14:21.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.308+0000 7f524af40640 1 -- 172.21.15.70:0/1789075585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5244102340 msgr2=0x7f524410e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:21.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.308+0000 7f524af40640 1 --2- 172.21.15.70:0/1789075585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5244102340 0x7f524410e800 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f523c009c10 tx=0x7f523c02a6e0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.308+0000 7f524af40640 1 -- 172.21.15.70:0/1789075585 shutdown_connections 2024-06-04T05:14:21.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.308+0000 7f524af40640 1 --2- 172.21.15.70:0/1789075585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5244101980 0x7f5244101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.308+0000 7f524af40640 1 --2- 172.21.15.70:0/1789075585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5244102340 0x7f524410e800 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.308+0000 7f524af40640 1 --2- 172.21.15.70:0/1789075585 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5244107940 0x7f5244107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.308+0000 7f524af40640 1 -- 172.21.15.70:0/1789075585 >> 172.21.15.70:0/1789075585 conn(0x7f5244078ad0 msgr2=0x7f5244078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:21.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.308+0000 7f524af40640 1 -- 172.21.15.70:0/1789075585 shutdown_connections 2024-06-04T05:14:21.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.308+0000 7f524af40640 1 -- 172.21.15.70:0/1789075585 wait complete. 2024-06-04T05:14:21.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f524af40640 1 Processor -- start 2024-06-04T05:14:21.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f524af40640 1 -- start start 2024-06-04T05:14:21.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f524af40640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5244101980 0x7f524410f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f524af40640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5244102340 0x7f524410f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f524af40640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5244107940 0x7f524410fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f524af40640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5244112bb0 con 0x7f5244107940 2024-06-04T05:14:21.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f524af40640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5244112a30 con 0x7f5244102340 2024-06-04T05:14:21.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f524af40640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5244112d30 con 0x7f5244101980 2024-06-04T05:14:21.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f52494dd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5244107940 0x7f524410fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f5248cdc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5244101980 0x7f524410f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f52494dd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5244107940 0x7f524410fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51598/0 (socket says 172.21.15.70:51598) 2024-06-04T05:14:21.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f5248cdc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5244101980 0x7f524410f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48864/0 (socket says 172.21.15.70:48864) 2024-06-04T05:14:21.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f52494dd640 1 -- 172.21.15.70:0/2181921664 learned_addr learned my addr 172.21.15.70:0/2181921664 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:21.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f5243fff640 1 --2- 172.21.15.70:0/2181921664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5244102340 0x7f524410f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f52494dd640 1 -- 172.21.15.70:0/2181921664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5244101980 msgr2=0x7f524410f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:21.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f52494dd640 1 --2- 172.21.15.70:0/2181921664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5244101980 0x7f524410f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f52494dd640 1 -- 172.21.15.70:0/2181921664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5244102340 msgr2=0x7f524410f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:21.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f52494dd640 1 --2- 172.21.15.70:0/2181921664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5244102340 0x7f524410f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.309+0000 7f52494dd640 1 -- 172.21.15.70:0/2181921664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52441a7610 con 0x7f5244107940 2024-06-04T05:14:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.310+0000 7f52494dd640 1 --2- 172.21.15.70:0/2181921664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5244107940 0x7f524410fd10 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f523c004af0 tx=0x7f523c030b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.310+0000 7f5241ffb640 1 -- 172.21.15.70:0/2181921664 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c041020 con 0x7f5244107940 2024-06-04T05:14:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.310+0000 7f524af40640 1 -- 172.21.15.70:0/2181921664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52441a78a0 con 0x7f5244107940 2024-06-04T05:14:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.310+0000 7f5241ffb640 1 -- 172.21.15.70:0/2181921664 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f523c037040 con 0x7f5244107940 2024-06-04T05:14:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.310+0000 7f5241ffb640 1 -- 172.21.15.70:0/2181921664 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c036070 con 0x7f5244107940 2024-06-04T05:14:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.310+0000 7f524af40640 1 -- 172.21.15.70:0/2181921664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52441a7de0 con 0x7f5244107940 2024-06-04T05:14:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.313+0000 7f524dc56740 1 Processor -- start 2024-06-04T05:14:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f5241ffb640 1 -- 172.21.15.70:0/2181921664 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f523c044020 con 0x7f5244107940 2024-06-04T05:14:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f524dc56740 1 -- start start 2024-06-04T05:14:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f524dc56740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55901352e820 0x55901352ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f524dc56740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352f140 0x5590135394c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f524dc56740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 0x55901353bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f524dc56740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5590134a0c30 con 0x559013539a00 2024-06-04T05:14:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f524dc56740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5590134907d0 con 0x55901352f140 2024-06-04T05:14:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f524dc56740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559013494810 con 0x55901352e820 2024-06-04T05:14:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f5248cdc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352f140 0x5590135394c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f5243fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55901352e820 0x55901352ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f5248cdc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352f140 0x5590135394c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:48804/0 (socket says 172.21.15.70:48804) 2024-06-04T05:14:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f5248cdc640 1 -- 172.21.15.70:0/3689535974 learned_addr learned my addr 172.21.15.70:0/3689535974 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f5243fff640 1 --2- 172.21.15.70:0/3689535974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 0x55901353bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f5248cdc640 1 -- 172.21.15.70:0/3689535974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55901352e820 msgr2=0x55901352ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f5248cdc640 1 --2- 172.21.15.70:0/3689535974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55901352e820 0x55901352ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f5248cdc640 1 -- 172.21.15.70:0/3689535974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 msgr2=0x55901353bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f5248cdc640 1 --2- 172.21.15.70:0/3689535974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 0x55901353bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.314+0000 7f5248cdc640 1 -- 172.21.15.70:0/3689535974 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590131a5210 con 0x55901352f140 2024-06-04T05:14:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.315+0000 7f5241ffb640 1 --2- 172.21.15.70:0/2181921664 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52340b2a90 0x7f52340b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.315+0000 7f5241ffb640 1 -- 172.21.15.70:0/2181921664 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f523c102e10 con 0x7f5244107940 2024-06-04T05:14:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.315+0000 7f5243fff640 1 --2- 172.21.15.70:0/2181921664 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52340b2a90 0x7f52340b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.315+0000 7f5248cdc640 1 --2- 172.21.15.70:0/3689535974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352f140 0x5590135394c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5238009c50 tx=0x7f5238011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d973d6132226bc1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.315+0000 7f522effd640 1 -- 172.21.15.70:0/3689535974 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523801e070 con 0x55901352f140 2024-06-04T05:14:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.315+0000 7f522effd640 1 -- 172.21.15.70:0/3689535974 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5238019ac0 con 0x55901352f140 2024-06-04T05:14:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.315+0000 7f522effd640 1 -- 172.21.15.70:0/3689535974 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523800e8b0 con 0x55901352f140 2024-06-04T05:14:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.316+0000 7f524dc56740 1 -- 172.21.15.70:0/3689535974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352f140 msgr2=0x5590135394c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.316+0000 7f5243fff640 1 --2- 172.21.15.70:0/2181921664 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52340b2a90 0x7f52340b4f50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x55901352c150 tx=0x7f5230013040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.316+0000 7f524dc56740 1 --2- 172.21.15.70:0/3689535974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352f140 0x5590135394c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5238009c50 tx=0x7f5238011810 comp rx=0 tx=0).stop 2024-06-04T05:14:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.316+0000 7f524dc56740 1 -- 172.21.15.70:0/3689535974 shutdown_connections 2024-06-04T05:14:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.316+0000 7f524dc56740 1 --2- 172.21.15.70:0/3689535974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352f140 0x5590135394c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.316+0000 7f524dc56740 1 --2- 172.21.15.70:0/3689535974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 0x55901353bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.316+0000 7f524dc56740 1 --2- 172.21.15.70:0/3689535974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55901352e820 0x55901352ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.316+0000 7f524dc56740 1 -- 172.21.15.70:0/3689535974 >> 172.21.15.70:0/3689535974 conn(0x55901352ad30 msgr2=0x55901352b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.316+0000 7f524dc56740 1 -- 172.21.15.70:0/3689535974 shutdown_connections 2024-06-04T05:14:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.316+0000 7f524dc56740 1 -- 172.21.15.70:0/3689535974 wait complete. 2024-06-04T05:14:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.317+0000 7f524dc56740 1 -- start start 2024-06-04T05:14:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.317+0000 7f524dc56740 1 Processor -- start 2024-06-04T05:14:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.317+0000 7f524dc56740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352e820 0x559013536a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.317+0000 7f524dc56740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55901352f140 0x559013536f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.317+0000 7f524dc56740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 0x5590135374c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.317+0000 7f524dc56740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559013176790 con 0x559013539a00 2024-06-04T05:14:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.317+0000 7f524dc56740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55901344c890 con 0x55901352e820 2024-06-04T05:14:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.317+0000 7f524dc56740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55901353c630 con 0x55901352f140 2024-06-04T05:14:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.317+0000 7f524dc56740 10 client.0 _ll_register_callbacks cb 0x5590131a4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.318+0000 7f5248cdc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352e820 0x559013536a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.318+0000 7f5248cdc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352e820 0x559013536a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:48820/0 (socket says 172.21.15.70:48820) 2024-06-04T05:14:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.318+0000 7f5248cdc640 1 -- 172.21.15.70:0/3926421560 learned_addr learned my addr 172.21.15.70:0/3926421560 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.318+0000 7f5243fff640 1 --2- 172.21.15.70:0/3926421560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55901352f140 0x559013536f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.318+0000 7f52494dd640 1 --2- 172.21.15.70:0/3926421560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 0x5590135374c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.318+0000 7f5243fff640 1 -- 172.21.15.70:0/3926421560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352e820 msgr2=0x559013536a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.318+0000 7f5243fff640 1 --2- 172.21.15.70:0/3926421560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352e820 0x559013536a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.318+0000 7f5243fff640 1 -- 172.21.15.70:0/3926421560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 msgr2=0x5590135374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.318+0000 7f5243fff640 1 --2- 172.21.15.70:0/3926421560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 0x5590135374c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.318+0000 7f5243fff640 1 -- 172.21.15.70:0/3926421560 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559013271620 con 0x55901352f140 2024-06-04T05:14:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.318+0000 7f52494dd640 1 --2- 172.21.15.70:0/3926421560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 0x5590135374c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.319+0000 7f5243fff640 1 --2- 172.21.15.70:0/3926421560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55901352f140 0x559013536f80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5230011fd0 tx=0x7f5230004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d528679bcab6323b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.319+0000 7f5223fff640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:14:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.319+0000 7f5223fff640 1 -- 172.21.15.70:0/3926421560 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5230021d30 con 0x55901352f140 2024-06-04T05:14:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.319+0000 7f524dc56740 1 -- 172.21.15.70:0/3926421560 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5238019ac0 con 0x55901352f140 2024-06-04T05:14:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.319+0000 7f5223fff640 1 -- 172.21.15.70:0/3926421560 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f523001a070 con 0x55901352f140 2024-06-04T05:14:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.319+0000 7f5223fff640 1 -- 172.21.15.70:0/3926421560 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f523001f440 con 0x55901352f140 2024-06-04T05:14:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.319+0000 7f524dc56740 10 client.28783 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.319+0000 7f524dc56740 1 -- 172.21.15.70:0/3926421560 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x559013550d40 con 0x55901352f140 2024-06-04T05:14:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.319+0000 7f5223fff640 1 -- 172.21.15.70:0/3926421560 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5238019ac0 con 0x55901352f140 2024-06-04T05:14:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.319+0000 7f5223fff640 1 -- 172.21.15.70:0/3926421560 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f523002b3f0 con 0x55901352f140 2024-06-04T05:14:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.320+0000 7f524dc56740 10 client.28783 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.320+0000 7f524dc56740 10 client.28783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:14:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.320+0000 7f524dc56740 20 client.28783 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.320+0000 7f524dc56740 1 --2- 172.21.15.70:0/3926421560 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5590135522a0 0x559013554690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.320+0000 7f524dc56740 4 client.28783 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:14:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.320+0000 7f524dc56740 1 -- 172.21.15.70:0/3926421560 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590134a2b80 con 0x5590135522a0 2024-06-04T05:14:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.320+0000 7f5248cdc640 1 --2- 172.21.15.70:0/3926421560 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5590135522a0 0x559013554690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.320+0000 7f5223fff640 10 client.28783 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.320+0000 7f5248cdc640 1 --2- 172.21.15.70:0/3926421560 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5590135522a0 0x559013554690 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4edc4d31876387b server_cookie=a44c4365848b218f in_seq=0 out_seq=0 2024-06-04T05:14:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.321+0000 7f5223fff640 1 -- 172.21.15.70:0/3926421560 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5590134a2b80 con 0x5590135522a0 2024-06-04T05:14:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.321+0000 7f5223fff640 10 client.28783 handle_command_reply: tid=0 2024-06-04T05:14:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.321+0000 7f524dc56740 1 client.28783 shutdown 2024-06-04T05:14:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.322+0000 7f524dc56740 1 -- 172.21.15.70:0/3926421560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55901352f140 msgr2=0x559013536f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.322+0000 7f524dc56740 1 --2- 172.21.15.70:0/3926421560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55901352f140 0x559013536f80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5230011fd0 tx=0x7f5230004730 comp rx=0 tx=0).stop 2024-06-04T05:14:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.322+0000 7f524dc56740 1 -- 172.21.15.70:0/3926421560 shutdown_connections 2024-06-04T05:14:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.322+0000 7f524dc56740 1 --2- 172.21.15.70:0/3926421560 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5590135522a0 0x559013554690 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.322+0000 7f524dc56740 1 --2- 172.21.15.70:0/3926421560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 0x5590135374c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.322+0000 7f524dc56740 1 --2- 172.21.15.70:0/3926421560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55901352f140 0x559013536f80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.322+0000 7f524dc56740 1 --2- 172.21.15.70:0/3926421560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352e820 0x559013536a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.322+0000 7f524dc56740 1 -- 172.21.15.70:0/3926421560 >> 172.21.15.70:0/3926421560 conn(0x55901352ad30 msgr2=0x559013529fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.322+0000 7f5223fff640 0 client.28783 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.322+0000 7f524dc56740 1 -- 172.21.15.70:0/3926421560 shutdown_connections 2024-06-04T05:14:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.322+0000 7f524dc56740 1 -- 172.21.15.70:0/3926421560 wait complete. 2024-06-04T05:14:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.322+0000 7f524dc56740 20 client.28783 trim_cache size 0 max 16384 2024-06-04T05:14:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.347+0000 7f524dc56740 1 Processor -- start 2024-06-04T05:14:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.347+0000 7f524dc56740 1 -- start start 2024-06-04T05:14:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.347+0000 7f524dc56740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55901352e820 0x55901352a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.347+0000 7f524dc56740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 0x559013525eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.347+0000 7f524dc56740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559013539280 0x559013559ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.347+0000 7f524dc56740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f52441ac400 con 0x559013539a00 2024-06-04T05:14:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.347+0000 7f524dc56740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f52441adfa0 con 0x559013539280 2024-06-04T05:14:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.347+0000 7f524dc56740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f52441acb30 con 0x55901352e820 2024-06-04T05:14:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.347+0000 7f5243fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 0x559013525eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.347+0000 7f5243fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 0x559013525eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51622/0 (socket says 172.21.15.70:51622) 2024-06-04T05:14:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.347+0000 7f5243fff640 1 -- 172.21.15.70:0/3385164866 learned_addr learned my addr 172.21.15.70:0/3385164866 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.347+0000 7f52494dd640 1 --2- 172.21.15.70:0/3385164866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559013539280 0x559013559ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.347+0000 7f5248cdc640 1 --2- 172.21.15.70:0/3385164866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55901352e820 0x55901352a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.348+0000 7f5243fff640 1 -- 172.21.15.70:0/3385164866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55901352e820 msgr2=0x55901352a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.348+0000 7f5243fff640 1 --2- 172.21.15.70:0/3385164866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55901352e820 0x55901352a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.348+0000 7f5243fff640 1 -- 172.21.15.70:0/3385164866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559013539280 msgr2=0x559013559ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.348+0000 7f5243fff640 1 --2- 172.21.15.70:0/3385164866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559013539280 0x559013559ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.348+0000 7f5243fff640 1 -- 172.21.15.70:0/3385164866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590135359f0 con 0x559013539a00 2024-06-04T05:14:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.348+0000 7f5243fff640 1 --2- 172.21.15.70:0/3385164866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 0x559013525eb0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f523002b9f0 tx=0x7f523000ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e84250a7127485a6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.348+0000 7f522e7fc640 1 -- 172.21.15.70:0/3385164866 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523000f790 con 0x559013539a00 2024-06-04T05:14:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.348+0000 7f522e7fc640 1 -- 172.21.15.70:0/3385164866 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f523002b3f0 con 0x559013539a00 2024-06-04T05:14:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.348+0000 7f522e7fc640 1 -- 172.21.15.70:0/3385164866 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523001fcd0 con 0x559013539a00 2024-06-04T05:14:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.349+0000 7f524dc56740 1 -- 172.21.15.70:0/3385164866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 msgr2=0x559013525eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.349+0000 7f524dc56740 1 --2- 172.21.15.70:0/3385164866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 0x559013525eb0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f523002b9f0 tx=0x7f523000ffd0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.349+0000 7f524dc56740 1 -- 172.21.15.70:0/3385164866 shutdown_connections 2024-06-04T05:14:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.349+0000 7f524dc56740 1 --2- 172.21.15.70:0/3385164866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559013539280 0x559013559ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.349+0000 7f524dc56740 1 --2- 172.21.15.70:0/3385164866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539a00 0x559013525eb0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.349+0000 7f524dc56740 1 --2- 172.21.15.70:0/3385164866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55901352e820 0x55901352a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.349+0000 7f524dc56740 1 -- 172.21.15.70:0/3385164866 >> 172.21.15.70:0/3385164866 conn(0x55901352ad30 msgr2=0x559013537290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.349+0000 7f524dc56740 1 -- 172.21.15.70:0/3385164866 shutdown_connections 2024-06-04T05:14:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.350+0000 7f524dc56740 1 -- 172.21.15.70:0/3385164866 wait complete. 2024-06-04T05:14:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.350+0000 7f524dc56740 1 -- start start 2024-06-04T05:14:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.350+0000 7f524dc56740 1 Processor -- start 2024-06-04T05:14:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f524dc56740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352e820 0x559013545ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f524dc56740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539280 0x5590135463e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f524dc56740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559013539a00 0x559013546920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f524dc56740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f52441ad8d0 con 0x559013539280 2024-06-04T05:14:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f524dc56740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f52441ad200 con 0x55901352e820 2024-06-04T05:14:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f524dc56740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f52441aa420 con 0x559013539a00 2024-06-04T05:14:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f524dc56740 10 client.0 _ll_register_callbacks cb 0x7f5230009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f5243fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539280 0x5590135463e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f52494dd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559013539a00 0x559013546920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f52494dd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559013539a00 0x559013546920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48888/0 (socket says 172.21.15.70:48888) 2024-06-04T05:14:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f52494dd640 1 -- 172.21.15.70:0/359813736 learned_addr learned my addr 172.21.15.70:0/359813736 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f5248cdc640 1 --2- 172.21.15.70:0/359813736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352e820 0x559013545ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f5243fff640 1 -- 172.21.15.70:0/359813736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559013539a00 msgr2=0x559013546920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f5243fff640 1 --2- 172.21.15.70:0/359813736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559013539a00 0x559013546920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f5243fff640 1 -- 172.21.15.70:0/359813736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352e820 msgr2=0x559013545ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f5243fff640 1 --2- 172.21.15.70:0/359813736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352e820 0x559013545ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f5243fff640 1 -- 172.21.15.70:0/359813736 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590135bbe10 con 0x559013539280 2024-06-04T05:14:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f5248cdc640 1 --2- 172.21.15.70:0/359813736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352e820 0x559013545ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f5243fff640 1 --2- 172.21.15.70:0/359813736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539280 0x5590135463e0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x5590135269d0 tx=0x7f52300218b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6167dd7ece5f6e39 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.351+0000 7f5223fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:14:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.352+0000 7f5223fff640 1 -- 172.21.15.70:0/359813736 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523002ba20 con 0x559013539280 2024-06-04T05:14:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.352+0000 7f524dc56740 1 -- 172.21.15.70:0/359813736 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f523002b3f0 con 0x559013539280 2024-06-04T05:14:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.352+0000 7f5223fff640 1 -- 172.21.15.70:0/359813736 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f5230009040 con 0x559013539280 2024-06-04T05:14:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.352+0000 7f5223fff640 1 -- 172.21.15.70:0/359813736 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f523002bc50 con 0x559013539280 2024-06-04T05:14:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.352+0000 7f524dc56740 10 client.19839 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.352+0000 7f524dc56740 1 -- 172.21.15.70:0/359813736 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5590135bd120 con 0x559013539280 2024-06-04T05:14:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.352+0000 7f5223fff640 1 -- 172.21.15.70:0/359813736 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f523002b3f0 con 0x559013539280 2024-06-04T05:14:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.352+0000 7f5223fff640 1 -- 172.21.15.70:0/359813736 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f523004d710 con 0x559013539280 2024-06-04T05:14:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.352+0000 7f524dc56740 10 client.19839 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.352+0000 7f524dc56740 10 client.19839 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:14:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.352+0000 7f524dc56740 20 client.19839 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.352+0000 7f524dc56740 1 --2- 172.21.15.70:0/359813736 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5590135bd7b0 0x5590135bfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.352+0000 7f524dc56740 4 client.19839 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:14:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.352+0000 7f524dc56740 1 -- 172.21.15.70:0/359813736 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55901349c360 con 0x5590135bd7b0 2024-06-04T05:14:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.353+0000 7f5248cdc640 1 --2- 172.21.15.70:0/359813736 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5590135bd7b0 0x5590135bfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.353+0000 7f5223fff640 10 client.19839 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:21.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.353+0000 7f5248cdc640 1 --2- 172.21.15.70:0/359813736 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5590135bd7b0 0x5590135bfbd0 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e02c10dd5961aef6 server_cookie=b7a47a9bc04c0ea8 in_seq=0 out_seq=0 2024-06-04T05:14:21.378 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:14:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.454+0000 7f5223fff640 1 -- 172.21.15.70:0/359813736 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55901349c360 con 0x5590135bd7b0 2024-06-04T05:14:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.454+0000 7f5223fff640 10 client.19839 handle_command_reply: tid=0 2024-06-04T05:14:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.454+0000 7f524dc56740 1 client.19839 shutdown 2024-06-04T05:14:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.454+0000 7f524dc56740 1 -- 172.21.15.70:0/359813736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539280 msgr2=0x5590135463e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.454+0000 7f524dc56740 1 --2- 172.21.15.70:0/359813736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539280 0x5590135463e0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x5590135269d0 tx=0x7f52300218b0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.454+0000 7f524dc56740 1 -- 172.21.15.70:0/359813736 shutdown_connections 2024-06-04T05:14:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.454+0000 7f524dc56740 1 --2- 172.21.15.70:0/359813736 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5590135bd7b0 0x5590135bfbd0 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.454+0000 7f524dc56740 1 --2- 172.21.15.70:0/359813736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559013539a00 0x559013546920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.454+0000 7f524dc56740 1 --2- 172.21.15.70:0/359813736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559013539280 0x5590135463e0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.454+0000 7f5223fff640 0 client.19839 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.454+0000 7f524dc56740 1 --2- 172.21.15.70:0/359813736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55901352e820 0x559013545ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.454+0000 7f524dc56740 1 -- 172.21.15.70:0/359813736 >> 172.21.15.70:0/359813736 conn(0x55901352ad30 msgr2=0x559013538b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.454+0000 7f524dc56740 1 -- 172.21.15.70:0/359813736 shutdown_connections 2024-06-04T05:14:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.455+0000 7f524dc56740 1 -- 172.21.15.70:0/359813736 wait complete. 2024-06-04T05:14:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.455+0000 7f524dc56740 20 client.19839 trim_cache size 0 max 16384 2024-06-04T05:14:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:14:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:14:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:14:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.455+0000 7f524af40640 1 -- 172.21.15.70:0/2181921664 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52340b2a90 msgr2=0x7f52340b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.455+0000 7f524af40640 1 --2- 172.21.15.70:0/2181921664 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52340b2a90 0x7f52340b4f50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x55901352c150 tx=0x7f5230013040 comp rx=0 tx=0).stop 2024-06-04T05:14:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.455+0000 7f524af40640 1 -- 172.21.15.70:0/2181921664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5244107940 msgr2=0x7f524410fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.455+0000 7f524af40640 1 --2- 172.21.15.70:0/2181921664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5244107940 0x7f524410fd10 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f523c004af0 tx=0x7f523c030b80 comp rx=0 tx=0).stop 2024-06-04T05:14:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.456+0000 7f524af40640 1 -- 172.21.15.70:0/2181921664 shutdown_connections 2024-06-04T05:14:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.456+0000 7f524af40640 1 --2- 172.21.15.70:0/2181921664 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52340b2a90 0x7f52340b4f50 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.456+0000 7f524af40640 1 --2- 172.21.15.70:0/2181921664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5244102340 0x7f524410f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.456+0000 7f524af40640 1 --2- 172.21.15.70:0/2181921664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5244107940 0x7f524410fd10 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.456+0000 7f524af40640 1 --2- 172.21.15.70:0/2181921664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5244101980 0x7f524410f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.456+0000 7f524af40640 1 -- 172.21.15.70:0/2181921664 >> 172.21.15.70:0/2181921664 conn(0x7f5244078ad0 msgr2=0x7f52441045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.456+0000 7f524af40640 1 -- 172.21.15.70:0/2181921664 shutdown_connections 2024-06-04T05:14:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:21.456+0000 7f524af40640 1 -- 172.21.15.70:0/2181921664 wait complete. 2024-06-04T05:14:21.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:21 smithi070 ceph-mon[32631]: pgmap v538: 97 pgs: 97 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 246 op/s 2024-06-04T05:14:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:21 smithi112 ceph-mon[36453]: pgmap v538: 97 pgs: 97 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 246 op/s 2024-06-04T05:14:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:21 smithi098 ceph-mon[35569]: pgmap v538: 97 pgs: 97 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 246 op/s 2024-06-04T05:14:21.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 28752 8.97 MB/sec execute 73 sec latency 151.309 ms 2024-06-04T05:14:22.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 28916 8.95 MB/sec execute 74 sec latency 91.050 ms 2024-06-04T05:14:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.809+0000 7fd64afbe700 1 -- 172.21.15.70:0/884225058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd644101e10 msgr2=0x7fd6441021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.809+0000 7fd64afbe700 1 --2- 172.21.15.70:0/884225058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd644101e10 0x7fd6441021f0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fd6340099c0 tx=0x7fd634009cd0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.809+0000 7fd64afbe700 1 -- 172.21.15.70:0/884225058 shutdown_connections 2024-06-04T05:14:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.809+0000 7fd64afbe700 1 --2- 172.21.15.70:0/884225058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd644106ef0 0x7fd64410eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.809+0000 7fd64afbe700 1 --2- 172.21.15.70:0/884225058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd644101e10 0x7fd6441021f0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.809+0000 7fd64afbe700 1 --2- 172.21.15.70:0/884225058 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6441027c0 0x7fd644106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.809+0000 7fd64afbe700 1 -- 172.21.15.70:0/884225058 >> 172.21.15.70:0/884225058 conn(0x7fd644078ae0 msgr2=0x7fd6440ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.809+0000 7fd64afbe700 1 -- 172.21.15.70:0/884225058 shutdown_connections 2024-06-04T05:14:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.809+0000 7fd64afbe700 1 -- 172.21.15.70:0/884225058 wait complete. 2024-06-04T05:14:22.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.810+0000 7fd64afbe700 1 Processor -- start 2024-06-04T05:14:22.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.810+0000 7fd64afbe700 1 -- start start 2024-06-04T05:14:22.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.811+0000 7fd64afbe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6441027c0 0x7fd64419bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.811+0000 7fd64afbe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd644106ef0 0x7fd64419c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.811+0000 7fd64afbe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6441a0870 0x7fd6441a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.811+0000 7fd64afbe700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd644111460 con 0x7fd6441a0870 2024-06-04T05:14:22.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.811+0000 7fd64afbe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd6441112e0 con 0x7fd644106ef0 2024-06-04T05:14:22.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.811+0000 7fd64afbe700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd6441115e0 con 0x7fd6441027c0 2024-06-04T05:14:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.811+0000 7fd64955b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6441a0870 0x7fd6441a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.811+0000 7fd648d5a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6441027c0 0x7fd64419bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.811+0000 7fd64955b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6441a0870 0x7fd6441a0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51648/0 (socket says 172.21.15.70:51648) 2024-06-04T05:14:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.811+0000 7fd64955b700 1 -- 172.21.15.70:0/3027450590 learned_addr learned my addr 172.21.15.70:0/3027450590 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.811+0000 7fd643fff700 1 --2- 172.21.15.70:0/3027450590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd644106ef0 0x7fd64419c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.811+0000 7fd648d5a700 1 -- 172.21.15.70:0/3027450590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd644106ef0 msgr2=0x7fd64419c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.811+0000 7fd648d5a700 1 --2- 172.21.15.70:0/3027450590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd644106ef0 0x7fd64419c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.811+0000 7fd648d5a700 1 -- 172.21.15.70:0/3027450590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6441a0870 msgr2=0x7fd6441a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.811+0000 7fd648d5a700 1 --2- 172.21.15.70:0/3027450590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6441a0870 0x7fd6441a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.811+0000 7fd648d5a700 1 -- 172.21.15.70:0/3027450590 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6441a1390 con 0x7fd6441027c0 2024-06-04T05:14:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.811+0000 7fd648d5a700 1 --2- 172.21.15.70:0/3027450590 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6441027c0 0x7fd64419bff0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd634008000 tx=0x7fd6340156e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.812+0000 7fd641ffb700 1 -- 172.21.15.70:0/3027450590 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63401a070 con 0x7fd6441027c0 2024-06-04T05:14:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.812+0000 7fd641ffb700 1 -- 172.21.15.70:0/3027450590 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd634004c90 con 0x7fd6441027c0 2024-06-04T05:14:22.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.812+0000 7fd641ffb700 1 -- 172.21.15.70:0/3027450590 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd634005310 con 0x7fd6441027c0 2024-06-04T05:14:22.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.812+0000 7fd64afbe700 1 -- 172.21.15.70:0/3027450590 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6441a1620 con 0x7fd6441027c0 2024-06-04T05:14:22.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.812+0000 7fd64afbe700 1 -- 172.21.15.70:0/3027450590 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6441a8cf0 con 0x7fd6441027c0 2024-06-04T05:14:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.816+0000 7fd641ffb700 1 -- 172.21.15.70:0/3027450590 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fd63400c040 con 0x7fd6441027c0 2024-06-04T05:14:22.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.817+0000 7fd641ffb700 1 --2- 172.21.15.70:0/3027450590 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd62c0b2be0 0x7fd62c0b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.817+0000 7fd643fff700 1 --2- 172.21.15.70:0/3027450590 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd62c0b2be0 0x7fd62c0b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.817+0000 7fd641ffb700 1 -- 172.21.15.70:0/3027450590 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fd6340e2330 con 0x7fd6441027c0 2024-06-04T05:14:22.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.818+0000 7fd643fff700 1 --2- 172.21.15.70:0/3027450590 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd62c0b2be0 0x7fd62c0b50a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd64410ef10 tx=0x7fd638004040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:22.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.818+0000 7fd655853b80 1 Processor -- start 2024-06-04T05:14:22.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.818+0000 7fd655853b80 1 -- start start 2024-06-04T05:14:22.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.818+0000 7fd655853b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f89b0 0x5580736f8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.818+0000 7fd655853b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f0c60 0x5580736f10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.818+0000 7fd655853b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f1620 0x5580737114c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.818+0000 7fd648d5a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f89b0 0x5580736f8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.819+0000 7fd655853b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558073651f00 con 0x5580736f89b0 2024-06-04T05:14:22.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.819+0000 7fd648d5a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f89b0 0x5580736f8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51652/0 (socket says 172.21.15.70:51652) 2024-06-04T05:14:22.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.819+0000 7fd655853b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558073641aa0 con 0x5580736f1620 2024-06-04T05:14:22.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.819+0000 7fd648d5a700 1 -- 172.21.15.70:0/104959930 learned_addr learned my addr 172.21.15.70:0/104959930 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:22.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.819+0000 7fd655853b80 1 -- 172.21.15.70:0/104959930 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558073645ae0 con 0x5580736f0c60 2024-06-04T05:14:22.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.819+0000 7fd643fff700 1 --2- 172.21.15.70:0/104959930 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f0c60 0x5580736f10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.819+0000 7fd64955b700 1 --2- 172.21.15.70:0/104959930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f1620 0x5580737114c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.819+0000 7fd64955b700 1 -- 172.21.15.70:0/104959930 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f0c60 msgr2=0x5580736f10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:22.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.819+0000 7fd64955b700 1 --2- 172.21.15.70:0/104959930 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f0c60 0x5580736f10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.819+0000 7fd64955b700 1 -- 172.21.15.70:0/104959930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f89b0 msgr2=0x5580736f8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:22.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.819+0000 7fd64955b700 1 --2- 172.21.15.70:0/104959930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f89b0 0x5580736f8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.819+0000 7fd64955b700 1 -- 172.21.15.70:0/104959930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580735eb140 con 0x5580736f1620 2024-06-04T05:14:22.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.819+0000 7fd648d5a700 1 --2- 172.21.15.70:0/104959930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f89b0 0x5580736f8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:22.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.819+0000 7fd64955b700 1 --2- 172.21.15.70:0/104959930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f1620 0x5580737114c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd63c00e650 tx=0x7fd63c009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40160230c97cbbda server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:22.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.820+0000 7fd62affd700 1 -- 172.21.15.70:0/104959930 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63c019030 con 0x5580736f1620 2024-06-04T05:14:22.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.820+0000 7fd62affd700 1 -- 172.21.15.70:0/104959930 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd63c0092e0 con 0x5580736f1620 2024-06-04T05:14:22.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.820+0000 7fd62affd700 1 -- 172.21.15.70:0/104959930 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63c004750 con 0x5580736f1620 2024-06-04T05:14:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.821+0000 7fd655853b80 1 -- 172.21.15.70:0/104959930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f1620 msgr2=0x5580737114c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.821+0000 7fd655853b80 1 --2- 172.21.15.70:0/104959930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f1620 0x5580737114c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd63c00e650 tx=0x7fd63c009920 comp rx=0 tx=0).stop 2024-06-04T05:14:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.821+0000 7fd655853b80 1 -- 172.21.15.70:0/104959930 shutdown_connections 2024-06-04T05:14:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.821+0000 7fd655853b80 1 --2- 172.21.15.70:0/104959930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f1620 0x5580737114c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.821+0000 7fd655853b80 1 --2- 172.21.15.70:0/104959930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f89b0 0x5580736f8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.821+0000 7fd655853b80 1 --2- 172.21.15.70:0/104959930 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f0c60 0x5580736f10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.821+0000 7fd655853b80 1 -- 172.21.15.70:0/104959930 >> 172.21.15.70:0/104959930 conn(0x55807361f3d0 msgr2=0x5580736ee970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.821+0000 7fd655853b80 1 -- 172.21.15.70:0/104959930 shutdown_connections 2024-06-04T05:14:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.821+0000 7fd655853b80 1 -- 172.21.15.70:0/104959930 wait complete. 2024-06-04T05:14:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.822+0000 7fd655853b80 1 -- start start 2024-06-04T05:14:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.822+0000 7fd655853b80 1 Processor -- start 2024-06-04T05:14:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd655853b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f0c60 0x558073711110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd655853b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f1620 0x55807370e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd648d5a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f0c60 0x558073711110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd655853b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f89b0 0x55807370e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd648d5a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f0c60 0x558073711110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51656/0 (socket says 172.21.15.70:51656) 2024-06-04T05:14:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd648d5a700 1 -- 172.21.15.70:0/459577589 learned_addr learned my addr 172.21.15.70:0/459577589 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd655853b80 1 -- 172.21.15.70:0/459577589 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558073548130 con 0x5580736f0c60 2024-06-04T05:14:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd655853b80 1 -- 172.21.15.70:0/459577589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5580735a9f30 con 0x5580736f1620 2024-06-04T05:14:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd655853b80 1 -- 172.21.15.70:0/459577589 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558073711d50 con 0x5580736f89b0 2024-06-04T05:14:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd643fff700 1 --2- 172.21.15.70:0/459577589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f1620 0x55807370e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd64955b700 1 --2- 172.21.15.70:0/459577589 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f89b0 0x55807370e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd655853b80 10 client.0 _ll_register_callbacks cb 0x5580735dcae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd643fff700 1 -- 172.21.15.70:0/459577589 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f89b0 msgr2=0x55807370e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd643fff700 1 --2- 172.21.15.70:0/459577589 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f89b0 0x55807370e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd643fff700 1 -- 172.21.15.70:0/459577589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f0c60 msgr2=0x558073711110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd643fff700 1 --2- 172.21.15.70:0/459577589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f0c60 0x558073711110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd643fff700 1 -- 172.21.15.70:0/459577589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558073591a60 con 0x5580736f1620 2024-06-04T05:14:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.823+0000 7fd64955b700 1 --2- 172.21.15.70:0/459577589 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f89b0 0x55807370e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.824+0000 7fd643fff700 1 --2- 172.21.15.70:0/459577589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f1620 0x55807370e460 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd644075dd0 tx=0x7fd6380047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48225ad86e6c5cac server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.824+0000 7fd623fff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:14:22.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.824+0000 7fd623fff700 1 -- 172.21.15.70:0/459577589 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd638027430 con 0x5580736f1620 2024-06-04T05:14:22.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.824+0000 7fd623fff700 1 -- 172.21.15.70:0/459577589 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fd638015030 con 0x5580736f1620 2024-06-04T05:14:22.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.824+0000 7fd655853b80 1 -- 172.21.15.70:0/459577589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd63c0092e0 con 0x5580736f1620 2024-06-04T05:14:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.824+0000 7fd623fff700 1 -- 172.21.15.70:0/459577589 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fd638004830 con 0x5580736f1620 2024-06-04T05:14:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.824+0000 7fd655853b80 10 client.28694 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.824+0000 7fd655853b80 1 -- 172.21.15.70:0/459577589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5580735fb8f0 con 0x5580736f1620 2024-06-04T05:14:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.824+0000 7fd623fff700 1 -- 172.21.15.70:0/459577589 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd63c0092e0 con 0x5580736f1620 2024-06-04T05:14:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.825+0000 7fd623fff700 1 -- 172.21.15.70:0/459577589 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7fd63800b040 con 0x5580736f1620 2024-06-04T05:14:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.825+0000 7fd655853b80 10 client.28694 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.825+0000 7fd655853b80 10 client.28694 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:14:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.825+0000 7fd655853b80 20 client.28694 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.825+0000 7fd655853b80 1 --2- 172.21.15.70:0/459577589 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55807371e8b0 0x558073720ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.825+0000 7fd655853b80 4 client.28694 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:14:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.825+0000 7fd655853b80 1 -- 172.21.15.70:0/459577589 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558073653e50 con 0x55807371e8b0 2024-06-04T05:14:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.825+0000 7fd648d5a700 1 --2- 172.21.15.70:0/459577589 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55807371e8b0 0x558073720ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.826+0000 7fd623fff700 10 client.28694 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.826+0000 7fd648d5a700 1 --2- 172.21.15.70:0/459577589 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55807371e8b0 0x558073720ca0 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e09c66bef5cce718 server_cookie=afa55a9ee32e5052 in_seq=0 out_seq=0 2024-06-04T05:14:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.828+0000 7fd623fff700 1 -- 172.21.15.70:0/459577589 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558073653e50 con 0x55807371e8b0 2024-06-04T05:14:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.828+0000 7fd623fff700 10 client.28694 handle_command_reply: tid=0 2024-06-04T05:14:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:14:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.828+0000 7fd655853b80 1 client.28694 shutdown 2024-06-04T05:14:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.828+0000 7fd655853b80 1 -- 172.21.15.70:0/459577589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f1620 msgr2=0x55807370e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.828+0000 7fd655853b80 1 --2- 172.21.15.70:0/459577589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f1620 0x55807370e460 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd644075dd0 tx=0x7fd6380047c0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.829+0000 7fd655853b80 1 -- 172.21.15.70:0/459577589 shutdown_connections 2024-06-04T05:14:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.829+0000 7fd655853b80 1 --2- 172.21.15.70:0/459577589 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f89b0 0x55807370e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.829+0000 7fd655853b80 1 --2- 172.21.15.70:0/459577589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f0c60 0x558073711110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.829+0000 7fd655853b80 1 --2- 172.21.15.70:0/459577589 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55807371e8b0 0x558073720ca0 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.829+0000 7fd655853b80 1 --2- 172.21.15.70:0/459577589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f1620 0x55807370e460 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.829+0000 7fd655853b80 1 -- 172.21.15.70:0/459577589 >> 172.21.15.70:0/459577589 conn(0x55807361f3d0 msgr2=0x5580736fb450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.829+0000 7fd623fff700 0 client.28694 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.829+0000 7fd655853b80 1 -- 172.21.15.70:0/459577589 shutdown_connections 2024-06-04T05:14:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.829+0000 7fd655853b80 1 -- 172.21.15.70:0/459577589 wait complete. 2024-06-04T05:14:22.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.829+0000 7fd655853b80 20 client.28694 trim_cache size 0 max 16384 2024-06-04T05:14:22.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.861+0000 7fd655853b80 1 Processor -- start 2024-06-04T05:14:22.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.861+0000 7fd655853b80 1 -- start start 2024-06-04T05:14:22.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.861+0000 7fd655853b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f0c60 0x5580736f7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.861+0000 7fd655853b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f1620 0x5580736e8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.862+0000 7fd655853b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f89b0 0x5580736f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.862+0000 7fd655853b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6441ac070 con 0x5580736f1620 2024-06-04T05:14:22.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.862+0000 7fd643fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f1620 0x5580736e8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.862+0000 7fd643fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f1620 0x5580736e8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51670/0 (socket says 172.21.15.70:51670) 2024-06-04T05:14:22.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.862+0000 7fd643fff700 1 -- 172.21.15.70:0/3075056890 learned_addr learned my addr 172.21.15.70:0/3075056890 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:22.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.862+0000 7fd648d5a700 1 --2- 172.21.15.70:0/3075056890 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f0c60 0x5580736f7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.862+0000 7fd64955b700 1 --2- 172.21.15.70:0/3075056890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f89b0 0x5580736f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.862+0000 7fd655853b80 1 -- 172.21.15.70:0/3075056890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd6441adc10 con 0x5580736f89b0 2024-06-04T05:14:22.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.862+0000 7fd655853b80 1 -- 172.21.15.70:0/3075056890 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd6441ac7a0 con 0x5580736f0c60 2024-06-04T05:14:22.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.863+0000 7fd643fff700 1 -- 172.21.15.70:0/3075056890 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f0c60 msgr2=0x5580736f7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:22.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.863+0000 7fd643fff700 1 --2- 172.21.15.70:0/3075056890 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f0c60 0x5580736f7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.863+0000 7fd643fff700 1 -- 172.21.15.70:0/3075056890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f89b0 msgr2=0x5580736f6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:22.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.863+0000 7fd643fff700 1 --2- 172.21.15.70:0/3075056890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f89b0 0x5580736f6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.863+0000 7fd643fff700 1 -- 172.21.15.70:0/3075056890 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558073711480 con 0x5580736f1620 2024-06-04T05:14:22.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.863+0000 7fd643fff700 1 --2- 172.21.15.70:0/3075056890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f1620 0x5580736e8f00 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fd638004790 tx=0x7fd63800ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=331f6c79db988101 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:22.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.863+0000 7fd648d5a700 1 --2- 172.21.15.70:0/3075056890 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f0c60 0x5580736f7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:14:22.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.863+0000 7fd64955b700 1 --2- 172.21.15.70:0/3075056890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f89b0 0x5580736f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:14:22.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.863+0000 7fd62a7fc700 1 -- 172.21.15.70:0/3075056890 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6380275b0 con 0x5580736f1620 2024-06-04T05:14:22.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.863+0000 7fd62a7fc700 1 -- 172.21.15.70:0/3075056890 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd638027be0 con 0x5580736f1620 2024-06-04T05:14:22.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.863+0000 7fd62a7fc700 1 -- 172.21.15.70:0/3075056890 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63801cc00 con 0x5580736f1620 2024-06-04T05:14:22.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.864+0000 7fd655853b80 1 -- 172.21.15.70:0/3075056890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f1620 msgr2=0x5580736e8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:22.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.864+0000 7fd655853b80 1 --2- 172.21.15.70:0/3075056890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f1620 0x5580736e8f00 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fd638004790 tx=0x7fd63800ea90 comp rx=0 tx=0).stop 2024-06-04T05:14:22.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.864+0000 7fd655853b80 1 -- 172.21.15.70:0/3075056890 shutdown_connections 2024-06-04T05:14:22.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.864+0000 7fd655853b80 1 --2- 172.21.15.70:0/3075056890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f89b0 0x5580736f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.864+0000 7fd655853b80 1 --2- 172.21.15.70:0/3075056890 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f0c60 0x5580736f7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.864+0000 7fd655853b80 1 --2- 172.21.15.70:0/3075056890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f1620 0x5580736e8f00 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.864+0000 7fd655853b80 1 -- 172.21.15.70:0/3075056890 >> 172.21.15.70:0/3075056890 conn(0x55807361f3d0 msgr2=0x55807370e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:22.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.865+0000 7fd655853b80 1 -- 172.21.15.70:0/3075056890 shutdown_connections 2024-06-04T05:14:22.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.865+0000 7fd655853b80 1 -- 172.21.15.70:0/3075056890 wait complete. 2024-06-04T05:14:22.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.865+0000 7fd655853b80 1 -- start start 2024-06-04T05:14:22.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.865+0000 7fd655853b80 1 Processor -- start 2024-06-04T05:14:22.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd655853b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f0c60 0x558073716fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd655853b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f1620 0x558073717520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd655853b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f89b0 0x558073713fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd655853b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6441ad540 con 0x5580736f1620 2024-06-04T05:14:22.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd655853b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd6441ace70 con 0x5580736f0c60 2024-06-04T05:14:22.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd655853b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd6441aa130 con 0x5580736f89b0 2024-06-04T05:14:22.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd655853b80 10 client.0 _ll_register_callbacks cb 0x5580735dcae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:22.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd643fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f1620 0x558073717520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd648d5a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f0c60 0x558073716fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd643fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f1620 0x558073717520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51684/0 (socket says 172.21.15.70:51684) 2024-06-04T05:14:22.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd64955b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f89b0 0x558073713fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd643fff700 1 -- 172.21.15.70:0/1819497600 learned_addr learned my addr 172.21.15.70:0/1819497600 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:22.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd648d5a700 1 --2- 172.21.15.70:0/1819497600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f0c60 0x558073716fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:48894/0 (socket says 172.21.15.70:48894) 2024-06-04T05:14:22.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd643fff700 1 -- 172.21.15.70:0/1819497600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f89b0 msgr2=0x558073713fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:22.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd643fff700 1 --2- 172.21.15.70:0/1819497600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f89b0 0x558073713fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd643fff700 1 -- 172.21.15.70:0/1819497600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f0c60 msgr2=0x558073716fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:22.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd643fff700 1 --2- 172.21.15.70:0/1819497600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f0c60 0x558073716fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:22.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.866+0000 7fd643fff700 1 -- 172.21.15.70:0/1819497600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558073714520 con 0x5580736f1620 2024-06-04T05:14:22.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.867+0000 7fd643fff700 1 --2- 172.21.15.70:0/1819497600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f1620 0x558073717520 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fd638005790 tx=0x7fd63801c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b98c1fdc583cb4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:22.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.867+0000 7fd623fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:14:22.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.867+0000 7fd623fff700 1 -- 172.21.15.70:0/1819497600 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63803e060 con 0x5580736f1620 2024-06-04T05:14:22.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.867+0000 7fd623fff700 1 -- 172.21.15.70:0/1819497600 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fd638027840 con 0x5580736f1620 2024-06-04T05:14:22.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.867+0000 7fd655853b80 1 -- 172.21.15.70:0/1819497600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd638027be0 con 0x5580736f1620 2024-06-04T05:14:22.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.867+0000 7fd623fff700 1 -- 172.21.15.70:0/1819497600 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fd638027a40 con 0x5580736f1620 2024-06-04T05:14:22.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.868+0000 7fd655853b80 10 client.19857 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:22.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.868+0000 7fd655853b80 1 -- 172.21.15.70:0/1819497600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x558073789cb0 con 0x5580736f1620 2024-06-04T05:14:22.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.868+0000 7fd623fff700 1 -- 172.21.15.70:0/1819497600 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd638027be0 con 0x5580736f1620 2024-06-04T05:14:22.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.868+0000 7fd623fff700 1 -- 172.21.15.70:0/1819497600 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7fd638030910 con 0x5580736f1620 2024-06-04T05:14:22.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.868+0000 7fd655853b80 10 client.19857 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:22.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.868+0000 7fd655853b80 10 client.19857 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:14:22.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.868+0000 7fd655853b80 20 client.19857 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:22.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.868+0000 7fd655853b80 1 --2- 172.21.15.70:0/1819497600 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55807378a290 0x55807378c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:22.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.868+0000 7fd655853b80 4 client.19857 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}] 2024-06-04T05:14:22.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.869+0000 7fd655853b80 1 -- 172.21.15.70:0/1819497600 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}) -- 0x55807364d630 con 0x55807378a290 2024-06-04T05:14:22.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.869+0000 7fd648d5a700 1 --2- 172.21.15.70:0/1819497600 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55807378a290 0x55807378c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:22.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.869+0000 7fd648d5a700 1 --2- 172.21.15.70:0/1819497600 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55807378a290 0x55807378c6b0 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73796ed604d34a8d server_cookie=ab4c67b89d2c21bb in_seq=0 out_seq=0 2024-06-04T05:14:22.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:22.869+0000 7fd623fff700 10 client.19857 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.175+0000 7fd623fff700 1 -- 172.21.15.70:0/1819497600 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55807364d630 con 0x55807378a290 2024-06-04T05:14:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.175+0000 7fd623fff700 10 client.19857 handle_command_reply: tid=0 2024-06-04T05:14:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.175+0000 7fd655853b80 1 client.19857 shutdown 2024-06-04T05:14:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.175+0000 7fd655853b80 1 -- 172.21.15.70:0/1819497600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f1620 msgr2=0x558073717520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.175+0000 7fd655853b80 1 --2- 172.21.15.70:0/1819497600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f1620 0x558073717520 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fd638005790 tx=0x7fd63801c650 comp rx=0 tx=0).stop 2024-06-04T05:14:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.175+0000 7fd655853b80 1 -- 172.21.15.70:0/1819497600 shutdown_connections 2024-06-04T05:14:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.175+0000 7fd655853b80 1 --2- 172.21.15.70:0/1819497600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580736f89b0 0x558073713fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.175+0000 7fd655853b80 1 --2- 172.21.15.70:0/1819497600 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55807378a290 0x55807378c6b0 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.175+0000 7fd655853b80 1 --2- 172.21.15.70:0/1819497600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5580736f0c60 0x558073716fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.175+0000 7fd655853b80 1 --2- 172.21.15.70:0/1819497600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580736f1620 0x558073717520 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.175+0000 7fd655853b80 1 -- 172.21.15.70:0/1819497600 >> 172.21.15.70:0/1819497600 conn(0x55807361f3d0 msgr2=0x55807370dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.175+0000 7fd623fff700 0 client.19857 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.175+0000 7fd655853b80 1 -- 172.21.15.70:0/1819497600 shutdown_connections 2024-06-04T05:14:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.175+0000 7fd655853b80 1 -- 172.21.15.70:0/1819497600 wait complete. 2024-06-04T05:14:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.175+0000 7fd655853b80 20 client.19857 trim_cache size 0 max 16384 2024-06-04T05:14:23.179 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:14:23.179 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 55, 2024-06-04T05:14:23.179 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:14:23.179 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 50, 2024-06-04T05:14:23.179 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:14:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "f2bf973b": { 2024-06-04T05:14:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 50, 2024-06-04T05:14:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:14:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:14:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:14:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:14:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:14:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:14:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 130.0, 2024-06-04T05:14:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:14:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:14:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:14:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:14:23.180 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:14:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:14:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:14:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:14:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:14:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:14:23.181 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:14:23.181 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:14:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.176+0000 7fd64afbe700 1 -- 172.21.15.70:0/3027450590 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd62c0b2be0 msgr2=0x7fd62c0b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.176+0000 7fd64afbe700 1 --2- 172.21.15.70:0/3027450590 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd62c0b2be0 0x7fd62c0b50a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd64410ef10 tx=0x7fd638004040 comp rx=0 tx=0).stop 2024-06-04T05:14:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.176+0000 7fd64afbe700 1 -- 172.21.15.70:0/3027450590 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6441027c0 msgr2=0x7fd64419bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.176+0000 7fd64afbe700 1 --2- 172.21.15.70:0/3027450590 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6441027c0 0x7fd64419bff0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd634008000 tx=0x7fd6340156e0 comp rx=0 tx=0).stop 2024-06-04T05:14:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.176+0000 7fd64afbe700 1 -- 172.21.15.70:0/3027450590 shutdown_connections 2024-06-04T05:14:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.176+0000 7fd64afbe700 1 --2- 172.21.15.70:0/3027450590 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd62c0b2be0 0x7fd62c0b50a0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.176+0000 7fd64afbe700 1 --2- 172.21.15.70:0/3027450590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6441a0870 0x7fd6441a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.176+0000 7fd64afbe700 1 --2- 172.21.15.70:0/3027450590 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6441027c0 0x7fd64419bff0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.176+0000 7fd64afbe700 1 --2- 172.21.15.70:0/3027450590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd644106ef0 0x7fd64419c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.176+0000 7fd64afbe700 1 -- 172.21.15.70:0/3027450590 >> 172.21.15.70:0/3027450590 conn(0x7fd644078ae0 msgr2=0x7fd6440fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.177+0000 7fd64afbe700 1 -- 172.21.15.70:0/3027450590 shutdown_connections 2024-06-04T05:14:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:23.177+0000 7fd64afbe700 1 -- 172.21.15.70:0/3027450590 wait complete. 2024-06-04T05:14:23.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:23 smithi070 ceph-mon[32631]: pgmap v539: 97 pgs: 97 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.1 MiB/s wr, 374 op/s 2024-06-04T05:14:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:23 smithi112 ceph-mon[36453]: pgmap v539: 97 pgs: 97 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.1 MiB/s wr, 374 op/s 2024-06-04T05:14:23.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:23 smithi098 ceph-mon[35569]: pgmap v539: 97 pgs: 97 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.1 MiB/s wr, 374 op/s 2024-06-04T05:14:23.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 28936 8.84 MB/sec execute 75 sec latency 914.137 ms 2024-06-04T05:14:24.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 28936 8.72 MB/sec execute 76 sec latency 1914.196 ms 2024-06-04T05:14:25.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.026+0000 7f4d1650d700 1 -- 172.21.15.70:0/3874996531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d10101cf0 msgr2=0x7f4d101020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:25.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.026+0000 7f4d1650d700 1 --2- 172.21.15.70:0/3874996531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d10101cf0 0x7f4d101020d0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f4d000099c0 tx=0x7f4d00009cd0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.026+0000 7f4d1650d700 1 -- 172.21.15.70:0/3874996531 shutdown_connections 2024-06-04T05:14:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.026+0000 7f4d1650d700 1 --2- 172.21.15.70:0/3874996531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d10108930 0x7f4d101104b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.026+0000 7f4d1650d700 1 --2- 172.21.15.70:0/3874996531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d10101cf0 0x7f4d101020d0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.026+0000 7f4d1650d700 1 --2- 172.21.15.70:0/3874996531 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4d101026a0 0x7f4d10108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.026+0000 7f4d1650d700 1 -- 172.21.15.70:0/3874996531 >> 172.21.15.70:0/3874996531 conn(0x7f4d10078a40 msgr2=0x7f4d100ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.026+0000 7f4d1650d700 1 -- 172.21.15.70:0/3874996531 shutdown_connections 2024-06-04T05:14:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.026+0000 7f4d1650d700 1 -- 172.21.15.70:0/3874996531 wait complete. 2024-06-04T05:14:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d1650d700 1 Processor -- start 2024-06-04T05:14:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d1650d700 1 -- start start 2024-06-04T05:14:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d1650d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d10101cf0 0x7f4d1019f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d1650d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4d101026a0 0x7f4d1019fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d1650d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d10108930 0x7f4d10199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d1650d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4d101136a0 con 0x7f4d10101cf0 2024-06-04T05:14:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d1650d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d10113520 con 0x7f4d10108930 2024-06-04T05:14:25.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d1650d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4d10113820 con 0x7f4d101026a0 2024-06-04T05:14:25.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d0ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d10101cf0 0x7f4d1019f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:25.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d0ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d10101cf0 0x7f4d1019f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51698/0 (socket says 172.21.15.70:51698) 2024-06-04T05:14:25.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d0ffff700 1 -- 172.21.15.70:0/523185425 learned_addr learned my addr 172.21.15.70:0/523185425 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:25.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d07fff700 1 --2- 172.21.15.70:0/523185425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4d101026a0 0x7f4d1019fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:25.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d14aaa700 1 --2- 172.21.15.70:0/523185425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d10108930 0x7f4d10199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:25.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d0ffff700 1 -- 172.21.15.70:0/523185425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4d101026a0 msgr2=0x7f4d1019fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:25.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d0ffff700 1 --2- 172.21.15.70:0/523185425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4d101026a0 0x7f4d1019fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d0ffff700 1 -- 172.21.15.70:0/523185425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d10108930 msgr2=0x7f4d10199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:25.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d0ffff700 1 --2- 172.21.15.70:0/523185425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d10108930 0x7f4d10199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d0ffff700 1 -- 172.21.15.70:0/523185425 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d1019a1f0 con 0x7f4d10101cf0 2024-06-04T05:14:25.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d07fff700 1 --2- 172.21.15.70:0/523185425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4d101026a0 0x7f4d1019fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:14:25.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.027+0000 7f4d14aaa700 1 --2- 172.21.15.70:0/523185425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d10108930 0x7f4d10199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:14:25.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.028+0000 7f4d0ffff700 1 --2- 172.21.15.70:0/523185425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d10101cf0 0x7f4d1019f880 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f4d00000c00 tx=0x7f4d0000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:25.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.028+0000 7f4d0dffb700 1 -- 172.21.15.70:0/523185425 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d00026070 con 0x7f4d10101cf0 2024-06-04T05:14:25.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.028+0000 7f4d1650d700 1 -- 172.21.15.70:0/523185425 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d1019a480 con 0x7f4d10101cf0 2024-06-04T05:14:25.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.028+0000 7f4d0dffb700 1 -- 172.21.15.70:0/523185425 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4d000054c0 con 0x7f4d10101cf0 2024-06-04T05:14:25.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.028+0000 7f4d0dffb700 1 -- 172.21.15.70:0/523185425 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d00004c20 con 0x7f4d10101cf0 2024-06-04T05:14:25.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.028+0000 7f4d1650d700 1 -- 172.21.15.70:0/523185425 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d101a6750 con 0x7f4d10101cf0 2024-06-04T05:14:25.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.030+0000 7f4d0dffb700 1 -- 172.21.15.70:0/523185425 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f4d00029070 con 0x7f4d10101cf0 2024-06-04T05:14:25.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.031+0000 7f4d0dffb700 1 --2- 172.21.15.70:0/523185425 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4cfc0b29f0 0x7f4cfc0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.031+0000 7f4d07fff700 1 --2- 172.21.15.70:0/523185425 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4cfc0b29f0 0x7f4cfc0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:25.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.031+0000 7f4d0dffb700 1 -- 172.21.15.70:0/523185425 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f4d000e3260 con 0x7f4d10101cf0 2024-06-04T05:14:25.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.031+0000 7f4d07fff700 1 --2- 172.21.15.70:0/523185425 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4cfc0b29f0 0x7f4cfc0b4eb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4d10107490 tx=0x7f4cf8008040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:25.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d20da2b80 1 Processor -- start 2024-06-04T05:14:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d20da2b80 1 -- start start 2024-06-04T05:14:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d20da2b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739945e320 0x55739945e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d20da2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5573994588a0 0x557399458d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d20da2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557399459260 0x557399465080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d20da2b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5573993b6d70 con 0x55739945e320 2024-06-04T05:14:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d20da2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5573993a6910 con 0x557399459260 2024-06-04T05:14:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d20da2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5573993aa950 con 0x5573994588a0 2024-06-04T05:14:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d0ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739945e320 0x55739945e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d0ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739945e320 0x55739945e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51708/0 (socket says 172.21.15.70:51708) 2024-06-04T05:14:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d0ffff700 1 -- 172.21.15.70:0/157294747 learned_addr learned my addr 172.21.15.70:0/157294747 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d07fff700 1 --2- 172.21.15.70:0/157294747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5573994588a0 0x557399458d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:25.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d14aaa700 1 --2- 172.21.15.70:0/157294747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557399459260 0x557399465080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:25.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d0ffff700 1 -- 172.21.15.70:0/157294747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5573994588a0 msgr2=0x557399458d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:25.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d0ffff700 1 --2- 172.21.15.70:0/157294747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5573994588a0 0x557399458d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d0ffff700 1 -- 172.21.15.70:0/157294747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557399459260 msgr2=0x557399465080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:25.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d0ffff700 1 --2- 172.21.15.70:0/157294747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557399459260 0x557399465080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d0ffff700 1 -- 172.21.15.70:0/157294747 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55739934ffb0 con 0x55739945e320 2024-06-04T05:14:25.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d14aaa700 1 --2- 172.21.15.70:0/157294747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557399459260 0x557399465080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:14:25.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.032+0000 7f4d0ffff700 1 --2- 172.21.15.70:0/157294747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739945e320 0x55739945e700 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f4d000aa530 tx=0x7f4d000a9ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d416027b91a5543c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:25.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.033+0000 7f4d067fc700 1 -- 172.21.15.70:0/157294747 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d000e3050 con 0x55739945e320 2024-06-04T05:14:25.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.033+0000 7f4d067fc700 1 -- 172.21.15.70:0/157294747 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4d000aaca0 con 0x55739945e320 2024-06-04T05:14:25.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.033+0000 7f4d067fc700 1 -- 172.21.15.70:0/157294747 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d000aa800 con 0x55739945e320 2024-06-04T05:14:25.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.033+0000 7f4d20da2b80 1 -- 172.21.15.70:0/157294747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739945e320 msgr2=0x55739945e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:25.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.033+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/157294747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739945e320 0x55739945e700 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f4d000aa530 tx=0x7f4d000a9ff0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.033+0000 7f4d20da2b80 1 -- 172.21.15.70:0/157294747 shutdown_connections 2024-06-04T05:14:25.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.033+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/157294747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557399459260 0x557399465080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.033+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/157294747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739945e320 0x55739945e700 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.033+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/157294747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5573994588a0 0x557399458d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.033+0000 7f4d20da2b80 1 -- 172.21.15.70:0/157294747 >> 172.21.15.70:0/157294747 conn(0x557399384240 msgr2=0x5573994533e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:25.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.033+0000 7f4d20da2b80 1 -- 172.21.15.70:0/157294747 shutdown_connections 2024-06-04T05:14:25.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.033+0000 7f4d20da2b80 1 -- 172.21.15.70:0/157294747 wait complete. 2024-06-04T05:14:25.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.034+0000 7f4d20da2b80 1 -- start start 2024-06-04T05:14:25.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.034+0000 7f4d20da2b80 1 Processor -- start 2024-06-04T05:14:25.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.034+0000 7f4d20da2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5573994588a0 0x557399464da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.034+0000 7f4d20da2b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 0x5573994620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.034+0000 7f4d20da2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55739945e320 0x5573994625e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.034+0000 7f4d20da2b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5573992acfa0 con 0x557399459260 2024-06-04T05:14:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.034+0000 7f4d20da2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55739930eda0 con 0x55739945e320 2024-06-04T05:14:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.034+0000 7f4d20da2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5573993b6d70 con 0x5573994588a0 2024-06-04T05:14:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.034+0000 7f4d20da2b80 10 client.0 _ll_register_callbacks cb 0x557399341950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.034+0000 7f4d07fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 0x5573994620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.034+0000 7f4d07fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 0x5573994620a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51722/0 (socket says 172.21.15.70:51722) 2024-06-04T05:14:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.034+0000 7f4d07fff700 1 -- 172.21.15.70:0/1335804988 learned_addr learned my addr 172.21.15.70:0/1335804988 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.035+0000 7f4d0ffff700 1 --2- 172.21.15.70:0/1335804988 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5573994588a0 0x557399464da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.035+0000 7f4d07fff700 1 -- 172.21.15.70:0/1335804988 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5573994588a0 msgr2=0x557399464da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.035+0000 7f4d07fff700 1 --2- 172.21.15.70:0/1335804988 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5573994588a0 0x557399464da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.035+0000 7f4d07fff700 1 -- 172.21.15.70:0/1335804988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55739945e320 msgr2=0x5573994625e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:25.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.035+0000 7f4d14aaa700 1 --2- 172.21.15.70:0/1335804988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55739945e320 0x5573994625e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:25.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.035+0000 7f4d07fff700 1 --2- 172.21.15.70:0/1335804988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55739945e320 0x5573994625e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.035+0000 7f4d07fff700 1 -- 172.21.15.70:0/1335804988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573992f68d0 con 0x557399459260 2024-06-04T05:14:25.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.035+0000 7f4d14aaa700 1 --2- 172.21.15.70:0/1335804988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55739945e320 0x5573994625e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:14:25.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.035+0000 7f4d07fff700 1 --2- 172.21.15.70:0/1335804988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 0x5573994620a0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f4cf800cd80 tx=0x7f4cf8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec8d411ea9346e7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:25.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.035+0000 7f4cef7fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:14:25.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.035+0000 7f4cef7fe700 1 -- 172.21.15.70:0/1335804988 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf8013c40 con 0x557399459260 2024-06-04T05:14:25.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.035+0000 7f4cef7fe700 1 -- 172.21.15.70:0/1335804988 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f4cf8013de0 con 0x557399459260 2024-06-04T05:14:25.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.035+0000 7f4d20da2b80 1 -- 172.21.15.70:0/1335804988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d000aaca0 con 0x557399459260 2024-06-04T05:14:25.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.035+0000 7f4cef7fe700 1 -- 172.21.15.70:0/1335804988 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f4cf802d050 con 0x557399459260 2024-06-04T05:14:25.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.035+0000 7f4d20da2b80 10 client.19869 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:25.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.035+0000 7f4d20da2b80 1 -- 172.21.15.70:0/1335804988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x557399360760 con 0x557399459260 2024-06-04T05:14:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.035+0000 7f4cef7fe700 1 -- 172.21.15.70:0/1335804988 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4d000aaca0 con 0x557399459260 2024-06-04T05:14:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.036+0000 7f4cef7fe700 1 -- 172.21.15.70:0/1335804988 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f4cf80113d0 con 0x557399459260 2024-06-04T05:14:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.036+0000 7f4d20da2b80 10 client.19869 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.036+0000 7f4d20da2b80 10 client.19869 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:14:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.036+0000 7f4d20da2b80 20 client.19869 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.036+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/1335804988 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557399471fe0 0x557399474420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.036+0000 7f4d20da2b80 4 client.19869 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:14:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.036+0000 7f4d20da2b80 1 -- 172.21.15.70:0/1335804988 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573993b8cc0 con 0x557399471fe0 2024-06-04T05:14:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.036+0000 7f4d0ffff700 1 --2- 172.21.15.70:0/1335804988 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557399471fe0 0x557399474420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.037+0000 7f4cef7fe700 10 client.19869 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.037+0000 7f4d0ffff700 1 --2- 172.21.15.70:0/1335804988 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557399471fe0 0x557399474420 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bd9c99c1f1fcb5f server_cookie=e2a5311d0be5c400 in_seq=0 out_seq=0 2024-06-04T05:14:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.037+0000 7f4cef7fe700 1 -- 172.21.15.70:0/1335804988 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5573993b8cc0 con 0x557399471fe0 2024-06-04T05:14:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.037+0000 7f4cef7fe700 10 client.19869 handle_command_reply: tid=0 2024-06-04T05:14:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.037+0000 7f4d20da2b80 1 client.19869 shutdown 2024-06-04T05:14:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.037+0000 7f4d20da2b80 1 -- 172.21.15.70:0/1335804988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 msgr2=0x5573994620a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.037+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/1335804988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 0x5573994620a0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f4cf800cd80 tx=0x7f4cf8004750 comp rx=0 tx=0).stop 2024-06-04T05:14:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.038+0000 7f4d20da2b80 1 -- 172.21.15.70:0/1335804988 shutdown_connections 2024-06-04T05:14:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.038+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/1335804988 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557399471fe0 0x557399474420 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.038+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/1335804988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55739945e320 0x5573994625e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.038+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/1335804988 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5573994588a0 0x557399464da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.038+0000 7f4cef7fe700 0 client.19869 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:25.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.038+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/1335804988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 0x5573994620a0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.038+0000 7f4d20da2b80 1 -- 172.21.15.70:0/1335804988 >> 172.21.15.70:0/1335804988 conn(0x557399384240 msgr2=0x557399452af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:25.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.038+0000 7f4d20da2b80 1 -- 172.21.15.70:0/1335804988 shutdown_connections 2024-06-04T05:14:25.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.038+0000 7f4d20da2b80 1 -- 172.21.15.70:0/1335804988 wait complete. 2024-06-04T05:14:25.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.038+0000 7f4d20da2b80 20 client.19869 trim_cache size 0 max 16384 2024-06-04T05:14:25.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.059+0000 7f4d20da2b80 1 Processor -- start 2024-06-04T05:14:25.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.059+0000 7f4d20da2b80 1 -- start start 2024-06-04T05:14:25.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.059+0000 7f4d20da2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5573994588a0 0x557399453270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.059+0000 7f4d20da2b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 0x557399460f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.059+0000 7f4d20da2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55739945e320 0x55739945ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.059+0000 7f4d20da2b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4d101addd0 con 0x557399459260 2024-06-04T05:14:25.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.059+0000 7f4d20da2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d101af970 con 0x5573994588a0 2024-06-04T05:14:25.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.059+0000 7f4d20da2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4d101ae500 con 0x55739945e320 2024-06-04T05:14:25.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.059+0000 7f4d07fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 0x557399460f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:25.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.059+0000 7f4d07fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 0x557399460f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51724/0 (socket says 172.21.15.70:51724) 2024-06-04T05:14:25.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.059+0000 7f4d07fff700 1 -- 172.21.15.70:0/3895192946 learned_addr learned my addr 172.21.15.70:0/3895192946 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:25.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.060+0000 7f4d0ffff700 1 --2- 172.21.15.70:0/3895192946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5573994588a0 0x557399453270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:25.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.060+0000 7f4d14aaa700 1 --2- 172.21.15.70:0/3895192946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55739945e320 0x55739945ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:25.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.060+0000 7f4d07fff700 1 -- 172.21.15.70:0/3895192946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55739945e320 msgr2=0x55739945ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:25.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.060+0000 7f4d07fff700 1 --2- 172.21.15.70:0/3895192946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55739945e320 0x55739945ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.060+0000 7f4d07fff700 1 -- 172.21.15.70:0/3895192946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5573994588a0 msgr2=0x557399453270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:25.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.060+0000 7f4d07fff700 1 --2- 172.21.15.70:0/3895192946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5573994588a0 0x557399453270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.060+0000 7f4d07fff700 1 -- 172.21.15.70:0/3895192946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55739945dcd0 con 0x557399459260 2024-06-04T05:14:25.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.060+0000 7f4d14aaa700 1 --2- 172.21.15.70:0/3895192946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55739945e320 0x55739945ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:14:25.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.060+0000 7f4d07fff700 1 --2- 172.21.15.70:0/3895192946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 0x557399460f90 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8000c00 tx=0x7f4cf8007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1933e45ad715a85 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:25.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.060+0000 7f4d05ffb700 1 -- 172.21.15.70:0/3895192946 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf8035020 con 0x557399459260 2024-06-04T05:14:25.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.060+0000 7f4d05ffb700 1 -- 172.21.15.70:0/3895192946 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4cf8024070 con 0x557399459260 2024-06-04T05:14:25.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.060+0000 7f4d05ffb700 1 -- 172.21.15.70:0/3895192946 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf8031690 con 0x557399459260 2024-06-04T05:14:25.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.061+0000 7f4d20da2b80 1 -- 172.21.15.70:0/3895192946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 msgr2=0x557399460f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:25.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.061+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/3895192946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 0x557399460f90 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8000c00 tx=0x7f4cf8007de0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.061+0000 7f4d20da2b80 1 -- 172.21.15.70:0/3895192946 shutdown_connections 2024-06-04T05:14:25.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.061+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/3895192946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55739945e320 0x55739945ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.061+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/3895192946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5573994588a0 0x557399453270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.061+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/3895192946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 0x557399460f90 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.061+0000 7f4d20da2b80 1 -- 172.21.15.70:0/3895192946 >> 172.21.15.70:0/3895192946 conn(0x557399384240 msgr2=0x557399460880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:25.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.061+0000 7f4d20da2b80 1 -- 172.21.15.70:0/3895192946 shutdown_connections 2024-06-04T05:14:25.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.061+0000 7f4d20da2b80 1 -- 172.21.15.70:0/3895192946 wait complete. 2024-06-04T05:14:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.062+0000 7f4d20da2b80 1 -- start start 2024-06-04T05:14:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.062+0000 7f4d20da2b80 1 Processor -- start 2024-06-04T05:14:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.062+0000 7f4d20da2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5573994588a0 0x55739946a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.062+0000 7f4d20da2b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 0x55739946adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.062+0000 7f4d20da2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55739945e320 0x557399465650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.063+0000 7f4d20da2b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4d101af2a0 con 0x557399459260 2024-06-04T05:14:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.063+0000 7f4d20da2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d101aebd0 con 0x55739945e320 2024-06-04T05:14:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.063+0000 7f4d20da2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4d101abd90 con 0x5573994588a0 2024-06-04T05:14:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.063+0000 7f4d20da2b80 10 client.0 _ll_register_callbacks cb 0x557399341950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.063+0000 7f4d14aaa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55739945e320 0x557399465650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:25.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.063+0000 7f4d14aaa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55739945e320 0x557399465650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:48940/0 (socket says 172.21.15.70:48940) 2024-06-04T05:14:25.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.063+0000 7f4d14aaa700 1 -- 172.21.15.70:0/4191306922 learned_addr learned my addr 172.21.15.70:0/4191306922 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:25.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.063+0000 7f4d14aaa700 1 -- 172.21.15.70:0/4191306922 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5573994588a0 msgr2=0x55739946a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:25.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.063+0000 7f4d14aaa700 1 --2- 172.21.15.70:0/4191306922 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5573994588a0 0x55739946a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.063+0000 7f4d14aaa700 1 -- 172.21.15.70:0/4191306922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 msgr2=0x55739946adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:25.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.063+0000 7f4d14aaa700 1 --2- 172.21.15.70:0/4191306922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 0x55739946adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.063+0000 7f4d14aaa700 1 -- 172.21.15.70:0/4191306922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557399465e20 con 0x55739945e320 2024-06-04T05:14:25.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.064+0000 7f4d14aaa700 1 --2- 172.21.15.70:0/4191306922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55739945e320 0x557399465650 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800dac0 tx=0x7f4d0800de80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24568977495c49f8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:25.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.064+0000 7f4cef7fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:14:25.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.064+0000 7f4cef7fe700 1 -- 172.21.15.70:0/4191306922 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0800a950 con 0x55739945e320 2024-06-04T05:14:25.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.064+0000 7f4cef7fe700 1 -- 172.21.15.70:0/4191306922 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f4d0800aaf0 con 0x55739945e320 2024-06-04T05:14:25.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.064+0000 7f4d20da2b80 1 -- 172.21.15.70:0/4191306922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cf8024070 con 0x55739945e320 2024-06-04T05:14:25.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.064+0000 7f4cef7fe700 1 -- 172.21.15.70:0/4191306922 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f4d0800acf0 con 0x55739945e320 2024-06-04T05:14:25.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.064+0000 7f4d20da2b80 10 client.28706 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:25.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.064+0000 7f4d20da2b80 1 -- 172.21.15.70:0/4191306922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5573994e1a70 con 0x55739945e320 2024-06-04T05:14:25.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.065+0000 7f4cef7fe700 1 -- 172.21.15.70:0/4191306922 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4d0801f8c0 con 0x55739945e320 2024-06-04T05:14:25.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.065+0000 7f4cef7fe700 1 -- 172.21.15.70:0/4191306922 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f4d0800b410 con 0x55739945e320 2024-06-04T05:14:25.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.065+0000 7f4d20da2b80 10 client.28706 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:25.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.065+0000 7f4d20da2b80 10 client.28706 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:14:25.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.065+0000 7f4d20da2b80 20 client.28706 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:25.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.065+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/4191306922 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5573994e21e0 0x5573994e4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:25.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.065+0000 7f4d20da2b80 4 client.28706 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:14:25.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.065+0000 7f4d20da2b80 1 -- 172.21.15.70:0/4191306922 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f4cf8024070 con 0x5573994e21e0 2024-06-04T05:14:25.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.065+0000 7f4d0ffff700 1 --2- 172.21.15.70:0/4191306922 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5573994e21e0 0x5573994e4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:25.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.066+0000 7f4d0ffff700 1 --2- 172.21.15.70:0/4191306922 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5573994e21e0 0x5573994e4660 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df625943e61f0f48 server_cookie=95f4734f6fad68d7 in_seq=0 out_seq=0 2024-06-04T05:14:25.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.066+0000 7f4cef7fe700 10 client.28706 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:25.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.066+0000 7f4cef7fe700 1 -- 172.21.15.70:0/4191306922 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f4cf8024070 con 0x5573994e21e0 2024-06-04T05:14:25.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.066+0000 7f4cef7fe700 10 client.28706 handle_command_reply: tid=0 2024-06-04T05:14:25.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.066+0000 7f4d20da2b80 1 client.28706 shutdown 2024-06-04T05:14:25.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.067+0000 7f4d20da2b80 1 -- 172.21.15.70:0/4191306922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55739945e320 msgr2=0x557399465650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:25.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.067+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/4191306922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55739945e320 0x557399465650 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800dac0 tx=0x7f4d0800de80 comp rx=0 tx=0).stop 2024-06-04T05:14:25.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.067+0000 7f4d20da2b80 1 -- 172.21.15.70:0/4191306922 shutdown_connections 2024-06-04T05:14:25.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.067+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/4191306922 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5573994e21e0 0x5573994e4660 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.067+0000 7f4cef7fe700 0 client.28706 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:25.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.067+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/4191306922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55739945e320 0x557399465650 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.067+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/4191306922 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5573994588a0 0x55739946a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.067+0000 7f4d20da2b80 1 --2- 172.21.15.70:0/4191306922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557399459260 0x55739946adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.067+0000 7f4d20da2b80 1 -- 172.21.15.70:0/4191306922 >> 172.21.15.70:0/4191306922 conn(0x557399384240 msgr2=0x557399461110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:25.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.067+0000 7f4d20da2b80 1 -- 172.21.15.70:0/4191306922 shutdown_connections 2024-06-04T05:14:25.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.067+0000 7f4d20da2b80 1 -- 172.21.15.70:0/4191306922 wait complete. 2024-06-04T05:14:25.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.067+0000 7f4d20da2b80 20 client.28706 trim_cache size 0 max 16384 2024-06-04T05:14:25.074 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:14:25.074 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:14:25.074 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:14:25.074 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:14:25.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.068+0000 7f4d1650d700 1 -- 172.21.15.70:0/523185425 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4cfc0b29f0 msgr2=0x7f4cfc0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:25.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.068+0000 7f4d1650d700 1 --2- 172.21.15.70:0/523185425 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4cfc0b29f0 0x7f4cfc0b4eb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4d10107490 tx=0x7f4cf8008040 comp rx=0 tx=0).stop 2024-06-04T05:14:25.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.068+0000 7f4d1650d700 1 -- 172.21.15.70:0/523185425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d10101cf0 msgr2=0x7f4d1019f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:25.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.068+0000 7f4d1650d700 1 --2- 172.21.15.70:0/523185425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d10101cf0 0x7f4d1019f880 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f4d00000c00 tx=0x7f4d0000f740 comp rx=0 tx=0).stop 2024-06-04T05:14:25.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.068+0000 7f4d1650d700 1 -- 172.21.15.70:0/523185425 shutdown_connections 2024-06-04T05:14:25.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.068+0000 7f4d1650d700 1 --2- 172.21.15.70:0/523185425 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4cfc0b29f0 0x7f4cfc0b4eb0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.068+0000 7f4d1650d700 1 --2- 172.21.15.70:0/523185425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d10108930 0x7f4d10199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.068+0000 7f4d1650d700 1 --2- 172.21.15.70:0/523185425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d10101cf0 0x7f4d1019f880 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.068+0000 7f4d1650d700 1 --2- 172.21.15.70:0/523185425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4d101026a0 0x7f4d1019fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:25.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.068+0000 7f4d1650d700 1 -- 172.21.15.70:0/523185425 >> 172.21.15.70:0/523185425 conn(0x7f4d10078a40 msgr2=0x7f4d100ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:25.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.068+0000 7f4d1650d700 1 -- 172.21.15.70:0/523185425 shutdown_connections 2024-06-04T05:14:25.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:25.068+0000 7f4d1650d700 1 -- 172.21.15.70:0/523185425 wait complete. 2024-06-04T05:14:25.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:25 smithi070 ceph-mon[32631]: pgmap v540: 97 pgs: 97 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 242 op/s 2024-06-04T05:14:25.655 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'f2bf973b', quiesce took 19.7 seconds. Will release after: -9.7 2024-06-04T05:14:25.655 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:14:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:25 smithi112 ceph-mon[36453]: pgmap v540: 97 pgs: 97 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 242 op/s 2024-06-04T05:14:25.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:25 smithi098 ceph-mon[35569]: pgmap v540: 97 pgs: 97 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 242 op/s 2024-06-04T05:14:25.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 28936 8.61 MB/sec execute 77 sec latency 2914.263 ms 2024-06-04T05:14:26.400 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:14:26.401 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:14:26.440 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:14:26.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 28936 8.50 MB/sec execute 78 sec latency 3914.305 ms 2024-06-04T05:14:27.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:27 smithi070 ceph-mon[32631]: pgmap v541: 97 pgs: 97 active+clean; 539 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 344 op/s 2024-06-04T05:14:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:27 smithi112 ceph-mon[36453]: pgmap v541: 97 pgs: 97 active+clean; 539 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 344 op/s 2024-06-04T05:14:27.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:27 smithi098 ceph-mon[35569]: pgmap v541: 97 pgs: 97 active+clean; 539 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 344 op/s 2024-06-04T05:14:27.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 28936 8.39 MB/sec execute 79 sec latency 4914.430 ms 2024-06-04T05:14:28.175 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:14:28.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.177+0000 7f33e0f77700 1 -- 172.21.15.70:0/3215981549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33dc1095a0 msgr2=0x7f33dc109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:28.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.177+0000 7f33e0f77700 1 --2- 172.21.15.70:0/3215981549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33dc1095a0 0x7f33dc109980 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009990 tx=0x7f33cc009ca0 comp rx=0 tx=0).stop 2024-06-04T05:14:28.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.177+0000 7f33e0f77700 1 -- 172.21.15.70:0/3215981549 shutdown_connections 2024-06-04T05:14:28.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.177+0000 7f33e0f77700 1 --2- 172.21.15.70:0/3215981549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33dc06c470 0x7f33dc1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:28.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.177+0000 7f33e0f77700 1 --2- 172.21.15.70:0/3215981549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33dc1095a0 0x7f33dc109980 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:28.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.177+0000 7f33e0f77700 1 --2- 172.21.15.70:0/3215981549 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33dc06bab0 0x7f33dc06bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:28.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.177+0000 7f33e0f77700 1 -- 172.21.15.70:0/3215981549 >> 172.21.15.70:0/3215981549 conn(0x7f33dc0793b0 msgr2=0x7f33dc0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:28.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.177+0000 7f33e0f77700 1 -- 172.21.15.70:0/3215981549 shutdown_connections 2024-06-04T05:14:28.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.177+0000 7f33e0f77700 1 -- 172.21.15.70:0/3215981549 wait complete. 2024-06-04T05:14:28.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.178+0000 7f33e0f77700 1 Processor -- start 2024-06-04T05:14:28.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.178+0000 7f33e0f77700 1 -- start start 2024-06-04T05:14:28.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.178+0000 7f33e0f77700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33dc06bab0 0x7f33dc102500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:28.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.178+0000 7f33e0f77700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33dc06c470 0x7f33dc102a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:28.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.178+0000 7f33e0f77700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33dc1095a0 0x7f33dc102f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:28.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.178+0000 7f33e0f77700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc113580 con 0x7f33dc06bab0 2024-06-04T05:14:28.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.178+0000 7f33e0f77700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc113400 con 0x7f33dc1095a0 2024-06-04T05:14:28.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.178+0000 7f33e0f77700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc113700 con 0x7f33dc06c470 2024-06-04T05:14:28.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.179+0000 7f33da59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33dc06bab0 0x7f33dc102500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:28.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.179+0000 7f33da59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33dc06bab0 0x7f33dc102500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52352/0 (socket says 172.21.15.70:52352) 2024-06-04T05:14:28.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.179+0000 7f33da59c700 1 -- 172.21.15.70:0/2515448971 learned_addr learned my addr 172.21.15.70:0/2515448971 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:28.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.179+0000 7f33dad9d700 1 --2- 172.21.15.70:0/2515448971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33dc1095a0 0x7f33dc102f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:28.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.179+0000 7f33d9d9b700 1 --2- 172.21.15.70:0/2515448971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33dc06c470 0x7f33dc102a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:28.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.179+0000 7f33dad9d700 1 -- 172.21.15.70:0/2515448971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33dc06c470 msgr2=0x7f33dc102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:28.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.179+0000 7f33dad9d700 1 --2- 172.21.15.70:0/2515448971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33dc06c470 0x7f33dc102a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:28.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.179+0000 7f33dad9d700 1 -- 172.21.15.70:0/2515448971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33dc06bab0 msgr2=0x7f33dc102500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:28.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.179+0000 7f33dad9d700 1 --2- 172.21.15.70:0/2515448971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33dc06bab0 0x7f33dc102500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:28.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.179+0000 7f33dad9d700 1 -- 172.21.15.70:0/2515448971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33dc1a8060 con 0x7f33dc1095a0 2024-06-04T05:14:28.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.179+0000 7f33da59c700 1 --2- 172.21.15.70:0/2515448971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33dc06bab0 0x7f33dc102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:28.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.179+0000 7f33dad9d700 1 --2- 172.21.15.70:0/2515448971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33dc1095a0 0x7f33dc102f80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f33d400efc0 tx=0x7f33d400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:28.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.179+0000 7f33cb7fe700 1 -- 172.21.15.70:0/2515448971 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d40092e0 con 0x7f33dc1095a0 2024-06-04T05:14:28.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.179+0000 7f33e0f77700 1 -- 172.21.15.70:0/2515448971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33dc1a82f0 con 0x7f33dc1095a0 2024-06-04T05:14:28.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.179+0000 7f33e0f77700 1 -- 172.21.15.70:0/2515448971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33dc1a8830 con 0x7f33dc1095a0 2024-06-04T05:14:28.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.180+0000 7f33cb7fe700 1 -- 172.21.15.70:0/2515448971 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f33d4004bb0 con 0x7f33dc1095a0 2024-06-04T05:14:28.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.180+0000 7f33cb7fe700 1 -- 172.21.15.70:0/2515448971 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d4004020 con 0x7f33dc1095a0 2024-06-04T05:14:28.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.180+0000 7f33e0f77700 1 -- 172.21.15.70:0/2515448971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33dc10ba80 con 0x7f33dc1095a0 2024-06-04T05:14:28.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.182+0000 7f33cb7fe700 1 -- 172.21.15.70:0/2515448971 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f33d401b030 con 0x7f33dc1095a0 2024-06-04T05:14:28.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.183+0000 7f33cb7fe700 1 --2- 172.21.15.70:0/2515448971 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33c40b29f0 0x7f33c40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:28.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.183+0000 7f33da59c700 1 --2- 172.21.15.70:0/2515448971 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33c40b29f0 0x7f33c40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:28.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.183+0000 7f33cb7fe700 1 -- 172.21.15.70:0/2515448971 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f33d40e2ec0 con 0x7f33dc1095a0 2024-06-04T05:14:28.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.183+0000 7f33da59c700 1 --2- 172.21.15.70:0/2515448971 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33c40b29f0 0x7f33c40b4eb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009750 tx=0x7f33cc01a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:28.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.187+0000 7f33cb7fe700 1 -- 172.21.15.70:0/2515448971 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f33d4022020 con 0x7f33dc1095a0 2024-06-04T05:14:28.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.452+0000 7f33e0f77700 1 -- 172.21.15.70:0/2515448971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33dc10b460 con 0x7f33dc1095a0 2024-06-04T05:14:28.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.453+0000 7f33cb7fe700 1 -- 172.21.15.70:0/2515448971 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+11932 (secure 0 0 0) 0x7f33d4010040 con 0x7f33dc1095a0 2024-06-04T05:14:28.456 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:14:28.456 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":55,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:14:18.668019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:14:28.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.456+0000 7f33e0f77700 1 -- 172.21.15.70:0/2515448971 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33c40b29f0 msgr2=0x7f33c40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:28.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.456+0000 7f33e0f77700 1 --2- 172.21.15.70:0/2515448971 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33c40b29f0 0x7f33c40b4eb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009750 tx=0x7f33cc01a040 comp rx=0 tx=0).stop 2024-06-04T05:14:28.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.456+0000 7f33e0f77700 1 -- 172.21.15.70:0/2515448971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33dc1095a0 msgr2=0x7f33dc102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:28.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.456+0000 7f33e0f77700 1 --2- 172.21.15.70:0/2515448971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33dc1095a0 0x7f33dc102f80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f33d400efc0 tx=0x7f33d400c3a0 comp rx=0 tx=0).stop 2024-06-04T05:14:28.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.456+0000 7f33e0f77700 1 -- 172.21.15.70:0/2515448971 shutdown_connections 2024-06-04T05:14:28.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.456+0000 7f33e0f77700 1 --2- 172.21.15.70:0/2515448971 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33c40b29f0 0x7f33c40b4eb0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:28.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.456+0000 7f33e0f77700 1 --2- 172.21.15.70:0/2515448971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33dc1095a0 0x7f33dc102f80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:28.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.456+0000 7f33e0f77700 1 --2- 172.21.15.70:0/2515448971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33dc06bab0 0x7f33dc102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:28.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.456+0000 7f33e0f77700 1 --2- 172.21.15.70:0/2515448971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33dc06c470 0x7f33dc102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:28.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.456+0000 7f33e0f77700 1 -- 172.21.15.70:0/2515448971 >> 172.21.15.70:0/2515448971 conn(0x7f33dc0793b0 msgr2=0x7f33dc101330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:28.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.456+0000 7f33e0f77700 1 -- 172.21.15.70:0/2515448971 shutdown_connections 2024-06-04T05:14:28.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:28.456+0000 7f33e0f77700 1 -- 172.21.15.70:0/2515448971 wait complete. 2024-06-04T05:14:28.459 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 55 2024-06-04T05:14:28.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 28936 8.29 MB/sec execute 80 sec latency 5914.466 ms 2024-06-04T05:14:29.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:29 smithi070 ceph-mon[32631]: pgmap v542: 97 pgs: 97 active+clean; 539 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 2.2 MiB/s wr, 230 op/s 2024-06-04T05:14:29.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:29 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2515448971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:14:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:29 smithi112 ceph-mon[36453]: pgmap v542: 97 pgs: 97 active+clean; 539 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 2.2 MiB/s wr, 230 op/s 2024-06-04T05:14:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:29 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2515448971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:14:29.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:29 smithi098 ceph-mon[35569]: pgmap v542: 97 pgs: 97 active+clean; 539 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 2.2 MiB/s wr, 230 op/s 2024-06-04T05:14:29.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:29 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2515448971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:14:29.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 28936 8.18 MB/sec execute 81 sec latency 6914.550 ms 2024-06-04T05:14:30.045 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id f2bf973b --release --await' 2024-06-04T05:14:30.046 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id f2bf973b --release --await 2024-06-04T05:14:30.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.538+0000 7fe93c70b700 1 -- 172.21.15.70:0/4181286683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe934108510 msgr2=0x7fe9341088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:30.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.538+0000 7fe93c70b700 1 --2- 172.21.15.70:0/4181286683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe934108510 0x7fe9341088f0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fe930009990 tx=0x7fe930009ca0 comp rx=0 tx=0).stop 2024-06-04T05:14:30.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.538+0000 7fe93c70b700 1 -- 172.21.15.70:0/4181286683 shutdown_connections 2024-06-04T05:14:30.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.538+0000 7fe93c70b700 1 --2- 172.21.15.70:0/4181286683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe934102df0 0x7fe93410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:30.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.538+0000 7fe93c70b700 1 --2- 172.21.15.70:0/4181286683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe934108510 0x7fe9341088f0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:30.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.538+0000 7fe93c70b700 1 --2- 172.21.15.70:0/4181286683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe934102430 0x7fe9341028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:30.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.538+0000 7fe93c70b700 1 -- 172.21.15.70:0/4181286683 >> 172.21.15.70:0/4181286683 conn(0x7fe9340793b0 msgr2=0x7fe9340797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:30.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.538+0000 7fe93c70b700 1 -- 172.21.15.70:0/4181286683 shutdown_connections 2024-06-04T05:14:30.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.538+0000 7fe93c70b700 1 -- 172.21.15.70:0/4181286683 wait complete. 2024-06-04T05:14:30.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.538+0000 7fe93c70b700 1 Processor -- start 2024-06-04T05:14:30.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.538+0000 7fe93c70b700 1 -- start start 2024-06-04T05:14:30.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.539+0000 7fe93c70b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe934102430 0x7fe93419a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:30.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.539+0000 7fe93c70b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe934102df0 0x7fe93419ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:30.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.539+0000 7fe93c70b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe934108510 0x7fe9341a59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:30.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.539+0000 7fe93c70b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe934112470 con 0x7fe934102430 2024-06-04T05:14:30.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.539+0000 7fe93c70b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe9341122f0 con 0x7fe934102df0 2024-06-04T05:14:30.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.539+0000 7fe93c70b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe9341125f0 con 0x7fe934108510 2024-06-04T05:14:30.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.539+0000 7fe93a4a7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe934102430 0x7fe93419a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:30.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.539+0000 7fe93a4a7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe934102430 0x7fe93419a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52382/0 (socket says 172.21.15.70:52382) 2024-06-04T05:14:30.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.539+0000 7fe93a4a7700 1 -- 172.21.15.70:0/4163313640 learned_addr learned my addr 172.21.15.70:0/4163313640 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:30.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.539+0000 7fe939ca6700 1 --2- 172.21.15.70:0/4163313640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe934102df0 0x7fe93419ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:30.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.539+0000 7fe93aca8700 1 --2- 172.21.15.70:0/4163313640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe934108510 0x7fe9341a59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:30.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.539+0000 7fe939ca6700 1 -- 172.21.15.70:0/4163313640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe934108510 msgr2=0x7fe9341a59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:30.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.539+0000 7fe939ca6700 1 --2- 172.21.15.70:0/4163313640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe934108510 0x7fe9341a59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:30.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.539+0000 7fe939ca6700 1 -- 172.21.15.70:0/4163313640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe934102430 msgr2=0x7fe93419a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:30.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.539+0000 7fe939ca6700 1 --2- 172.21.15.70:0/4163313640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe934102430 0x7fe93419a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:30.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.539+0000 7fe939ca6700 1 -- 172.21.15.70:0/4163313640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9341a5ef0 con 0x7fe934102df0 2024-06-04T05:14:30.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.539+0000 7fe93a4a7700 1 --2- 172.21.15.70:0/4163313640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe934102430 0x7fe93419a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:30.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.540+0000 7fe939ca6700 1 --2- 172.21.15.70:0/4163313640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe934102df0 0x7fe93419ae70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe92400d750 tx=0x7fe92400da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:30.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.540+0000 7fe92b7fe700 1 -- 172.21.15.70:0/4163313640 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe924014030 con 0x7fe934102df0 2024-06-04T05:14:30.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.540+0000 7fe92b7fe700 1 -- 172.21.15.70:0/4163313640 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe9240044b0 con 0x7fe934102df0 2024-06-04T05:14:30.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.540+0000 7fe93c70b700 1 -- 172.21.15.70:0/4163313640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe934198780 con 0x7fe934102df0 2024-06-04T05:14:30.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.540+0000 7fe92b7fe700 1 -- 172.21.15.70:0/4163313640 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe924005260 con 0x7fe934102df0 2024-06-04T05:14:30.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.540+0000 7fe93c70b700 1 -- 172.21.15.70:0/4163313640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe934198d10 con 0x7fe934102df0 2024-06-04T05:14:30.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.541+0000 7fe93c70b700 1 -- 172.21.15.70:0/4163313640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe93410a9a0 con 0x7fe934102df0 2024-06-04T05:14:30.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.543+0000 7fe92b7fe700 1 -- 172.21.15.70:0/4163313640 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fe9240049e0 con 0x7fe934102df0 2024-06-04T05:14:30.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.543+0000 7fe92b7fe700 1 --2- 172.21.15.70:0/4163313640 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9200b2a70 0x7fe9200b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:30.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.543+0000 7fe93a4a7700 1 --2- 172.21.15.70:0/4163313640 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9200b2a70 0x7fe9200b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:30.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.543+0000 7fe92b7fe700 1 -- 172.21.15.70:0/4163313640 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fe9240e21a0 con 0x7fe934102df0 2024-06-04T05:14:30.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.544+0000 7fe93a4a7700 1 --2- 172.21.15.70:0/4163313640 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9200b2a70 0x7fe9200b4f30 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe930009750 tx=0x7fe93001a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:30.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.547+0000 7fe92b7fe700 1 -- 172.21.15.70:0/4163313640 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe9240a9b50 con 0x7fe934102df0 2024-06-04T05:14:30.566 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:14:30.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 28936 8.08 MB/sec execute 82 sec latency 7914.616 ms 2024-06-04T05:14:30.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.804+0000 7fe93c70b700 1 -- 172.21.15.70:0/4163313640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe93406a230 con 0x7fe934102df0 2024-06-04T05:14:30.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.805+0000 7fe92b7fe700 1 -- 172.21.15.70:0/4163313640 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+11932 (secure 0 0 0) 0x7fe9240a9450 con 0x7fe934102df0 2024-06-04T05:14:30.807 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:14:30.807 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":55,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:14:18.668019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:14:30.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.807+0000 7fe93c70b700 1 -- 172.21.15.70:0/4163313640 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9200b2a70 msgr2=0x7fe9200b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:30.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.807+0000 7fe93c70b700 1 --2- 172.21.15.70:0/4163313640 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9200b2a70 0x7fe9200b4f30 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe930009750 tx=0x7fe93001a040 comp rx=0 tx=0).stop 2024-06-04T05:14:30.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.807+0000 7fe93c70b700 1 -- 172.21.15.70:0/4163313640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe934102df0 msgr2=0x7fe93419ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:30.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.807+0000 7fe93c70b700 1 --2- 172.21.15.70:0/4163313640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe934102df0 0x7fe93419ae70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe92400d750 tx=0x7fe92400da60 comp rx=0 tx=0).stop 2024-06-04T05:14:30.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.807+0000 7fe93c70b700 1 -- 172.21.15.70:0/4163313640 shutdown_connections 2024-06-04T05:14:30.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.807+0000 7fe93c70b700 1 --2- 172.21.15.70:0/4163313640 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9200b2a70 0x7fe9200b4f30 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:30.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.807+0000 7fe93c70b700 1 --2- 172.21.15.70:0/4163313640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe934108510 0x7fe9341a59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:30.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.807+0000 7fe93c70b700 1 --2- 172.21.15.70:0/4163313640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe934102430 0x7fe93419a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:30.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.807+0000 7fe93c70b700 1 --2- 172.21.15.70:0/4163313640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe934102df0 0x7fe93419ae70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:30.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.807+0000 7fe93c70b700 1 -- 172.21.15.70:0/4163313640 >> 172.21.15.70:0/4163313640 conn(0x7fe9340793b0 msgr2=0x7fe934101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:30.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.807+0000 7fe93c70b700 1 -- 172.21.15.70:0/4163313640 shutdown_connections 2024-06-04T05:14:30.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:30.807+0000 7fe93c70b700 1 -- 172.21.15.70:0/4163313640 wait complete. 2024-06-04T05:14:30.811 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 55 2024-06-04T05:14:31.599 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:31 smithi070 ceph-mon[32631]: pgmap v543: 97 pgs: 97 active+clean; 539 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 2.2 MiB/s wr, 230 op/s 2024-06-04T05:14:31.599 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:31 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/4163313640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:14:31.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:31 smithi112 ceph-mon[36453]: pgmap v543: 97 pgs: 97 active+clean; 539 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 2.2 MiB/s wr, 230 op/s 2024-06-04T05:14:31.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:31 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/4163313640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:14:31.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:31 smithi098 ceph-mon[35569]: pgmap v543: 97 pgs: 97 active+clean; 539 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 2.2 MiB/s wr, 230 op/s 2024-06-04T05:14:31.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:31 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/4163313640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:14:31.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 28936 7.99 MB/sec execute 83 sec latency 8914.714 ms 2024-06-04T05:14:32.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:14:32.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 28936 7.89 MB/sec execute 84 sec latency 9914.761 ms 2024-06-04T05:14:32.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.903+0000 7fb9483b6700 1 -- 172.21.15.70:0/3298974346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb940102e30 msgr2=0x7fb94010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:32.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.903+0000 7fb9483b6700 1 --2- 172.21.15.70:0/3298974346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb940102e30 0x7fb94010f2f0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fb93800b0a0 tx=0x7fb93800b3b0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.903+0000 7fb9483b6700 1 -- 172.21.15.70:0/3298974346 shutdown_connections 2024-06-04T05:14:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.903+0000 7fb9483b6700 1 --2- 172.21.15.70:0/3298974346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb940102e30 0x7fb94010f2f0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.903+0000 7fb9483b6700 1 --2- 172.21.15.70:0/3298974346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9401084b0 0x7fb940108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.903+0000 7fb9483b6700 1 --2- 172.21.15.70:0/3298974346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb940102470 0x7fb9401028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.903+0000 7fb9483b6700 1 -- 172.21.15.70:0/3298974346 >> 172.21.15.70:0/3298974346 conn(0x7fb940079410 msgr2=0x7fb940079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.903+0000 7fb9483b6700 1 -- 172.21.15.70:0/3298974346 shutdown_connections 2024-06-04T05:14:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.903+0000 7fb9483b6700 1 -- 172.21.15.70:0/3298974346 wait complete. 2024-06-04T05:14:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.904+0000 7fb9483b6700 1 Processor -- start 2024-06-04T05:14:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.904+0000 7fb9483b6700 1 -- start start 2024-06-04T05:14:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb9483b6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb940102470 0x7fb94019aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb9483b6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb940102e30 0x7fb94019afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb9483b6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9401084b0 0x7fb9401a5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb9483b6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb940112570 con 0x7fb9401084b0 2024-06-04T05:14:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb9483b6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9401123f0 con 0x7fb940102470 2024-06-04T05:14:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb9483b6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb9401126f0 con 0x7fb940102e30 2024-06-04T05:14:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb946953700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9401084b0 0x7fb9401a5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb946953700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9401084b0 0x7fb9401a5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52386/0 (socket says 172.21.15.70:52386) 2024-06-04T05:14:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb946953700 1 -- 172.21.15.70:0/763780616 learned_addr learned my addr 172.21.15.70:0/763780616 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb945951700 1 --2- 172.21.15.70:0/763780616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb940102e30 0x7fb94019afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb946152700 1 --2- 172.21.15.70:0/763780616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb940102470 0x7fb94019aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb946953700 1 -- 172.21.15.70:0/763780616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb940102e30 msgr2=0x7fb94019afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb946953700 1 --2- 172.21.15.70:0/763780616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb940102e30 0x7fb94019afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb946953700 1 -- 172.21.15.70:0/763780616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb940102470 msgr2=0x7fb94019aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb946953700 1 --2- 172.21.15.70:0/763780616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb940102470 0x7fb94019aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb946953700 1 -- 172.21.15.70:0/763780616 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9401a6050 con 0x7fb9401084b0 2024-06-04T05:14:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb946152700 1 --2- 172.21.15.70:0/763780616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb940102470 0x7fb94019aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:14:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb945951700 1 --2- 172.21.15.70:0/763780616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb940102e30 0x7fb94019afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:14:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb946953700 1 --2- 172.21.15.70:0/763780616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9401084b0 0x7fb9401a5b10 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fb9380062a0 tx=0x7fb938004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb9377fe700 1 -- 172.21.15.70:0/763780616 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb938021070 con 0x7fb9401084b0 2024-06-04T05:14:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.905+0000 7fb9483b6700 1 -- 172.21.15.70:0/763780616 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9401a6220 con 0x7fb9401084b0 2024-06-04T05:14:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.906+0000 7fb9377fe700 1 -- 172.21.15.70:0/763780616 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb938007d00 con 0x7fb9401084b0 2024-06-04T05:14:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.906+0000 7fb9377fe700 1 -- 172.21.15.70:0/763780616 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb938005280 con 0x7fb9401084b0 2024-06-04T05:14:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.906+0000 7fb9483b6700 1 -- 172.21.15.70:0/763780616 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb940198880 con 0x7fb9401084b0 2024-06-04T05:14:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.908+0000 7fb9377fe700 1 -- 172.21.15.70:0/763780616 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fb938024070 con 0x7fb9401084b0 2024-06-04T05:14:32.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.909+0000 7fb9377fe700 1 --2- 172.21.15.70:0/763780616 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb92c0b6e00 0x7fb92c0b92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.909+0000 7fb946152700 1 --2- 172.21.15.70:0/763780616 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb92c0b6e00 0x7fb92c0b92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.910+0000 7fb9377fe700 1 -- 172.21.15.70:0/763780616 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fb93800f020 con 0x7fb9401084b0 2024-06-04T05:14:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.910+0000 7fb946152700 1 --2- 172.21.15.70:0/763780616 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb92c0b6e00 0x7fb92c0b92c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0099d0 tx=0x7fb93c008040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.910+0000 7fb952c4bb80 1 Processor -- start 2024-06-04T05:14:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.910+0000 7fb952c4bb80 1 -- start start 2024-06-04T05:14:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.910+0000 7fb952c4bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad40834720 0x55ad40834b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.910+0000 7fb952c4bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad4082fbf0 0x55ad40830070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.910+0000 7fb952c4bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad408305b0 0x55ad4083c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.910+0000 7fb952c4bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ad4078de70 con 0x55ad40834720 2024-06-04T05:14:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.910+0000 7fb952c4bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ad4077da10 con 0x55ad408305b0 2024-06-04T05:14:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.910+0000 7fb952c4bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ad40781a50 con 0x55ad4082fbf0 2024-06-04T05:14:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.910+0000 7fb946152700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad40834720 0x55ad40834b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.910+0000 7fb946152700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad40834720 0x55ad40834b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52400/0 (socket says 172.21.15.70:52400) 2024-06-04T05:14:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.911+0000 7fb946152700 1 -- 172.21.15.70:0/535328479 learned_addr learned my addr 172.21.15.70:0/535328479 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.911+0000 7fb946953700 1 --2- 172.21.15.70:0/535328479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad408305b0 0x55ad4083c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.911+0000 7fb945951700 1 --2- 172.21.15.70:0/535328479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad4082fbf0 0x55ad40830070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.911+0000 7fb946152700 1 -- 172.21.15.70:0/535328479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad4082fbf0 msgr2=0x55ad40830070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.911+0000 7fb946152700 1 --2- 172.21.15.70:0/535328479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad4082fbf0 0x55ad40830070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.911+0000 7fb946152700 1 -- 172.21.15.70:0/535328479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad408305b0 msgr2=0x55ad4083c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.911+0000 7fb946152700 1 --2- 172.21.15.70:0/535328479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad408305b0 0x55ad4083c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.911+0000 7fb946152700 1 -- 172.21.15.70:0/535328479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad407270b0 con 0x55ad40834720 2024-06-04T05:14:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.911+0000 7fb945951700 1 --2- 172.21.15.70:0/535328479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad4082fbf0 0x55ad40830070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:14:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.911+0000 7fb946152700 1 --2- 172.21.15.70:0/535328479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad40834720 0x55ad40834b00 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00d7c0 tx=0x7fb93c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a89fecc8146783f8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.911+0000 7fb934d70700 1 -- 172.21.15.70:0/535328479 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c024070 con 0x55ad40834720 2024-06-04T05:14:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.911+0000 7fb934d70700 1 -- 172.21.15.70:0/535328479 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb93c014ad0 con 0x55ad40834720 2024-06-04T05:14:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.911+0000 7fb934d70700 1 -- 172.21.15.70:0/535328479 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c01fa30 con 0x55ad40834720 2024-06-04T05:14:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.912+0000 7fb952c4bb80 1 -- 172.21.15.70:0/535328479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad40834720 msgr2=0x55ad40834b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.912+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/535328479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad40834720 0x55ad40834b00 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00d7c0 tx=0x7fb93c010000 comp rx=0 tx=0).stop 2024-06-04T05:14:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.912+0000 7fb952c4bb80 1 -- 172.21.15.70:0/535328479 shutdown_connections 2024-06-04T05:14:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.912+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/535328479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad408305b0 0x55ad4083c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.912+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/535328479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad40834720 0x55ad40834b00 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.912+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/535328479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad4082fbf0 0x55ad40830070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.912+0000 7fb952c4bb80 1 -- 172.21.15.70:0/535328479 >> 172.21.15.70:0/535328479 conn(0x55ad4075b340 msgr2=0x55ad4082a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.912+0000 7fb952c4bb80 1 -- 172.21.15.70:0/535328479 shutdown_connections 2024-06-04T05:14:32.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.912+0000 7fb952c4bb80 1 -- 172.21.15.70:0/535328479 wait complete. 2024-06-04T05:14:32.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.913+0000 7fb952c4bb80 1 -- start start 2024-06-04T05:14:32.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.913+0000 7fb952c4bb80 1 Processor -- start 2024-06-04T05:14:32.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.913+0000 7fb952c4bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad4082fbf0 0x55ad4083c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.913+0000 7fb952c4bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad408305b0 0x55ad408395c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.913+0000 7fb952c4bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad40834720 0x55ad40839b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.913+0000 7fb952c4bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ad406840a0 con 0x55ad408305b0 2024-06-04T05:14:32.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.913+0000 7fb952c4bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ad406e5ea0 con 0x55ad4082fbf0 2024-06-04T05:14:32.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.913+0000 7fb945951700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad408305b0 0x55ad408395c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.913+0000 7fb945951700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad408305b0 0x55ad408395c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52416/0 (socket says 172.21.15.70:52416) 2024-06-04T05:14:32.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.913+0000 7fb945951700 1 -- 172.21.15.70:0/641923370 learned_addr learned my addr 172.21.15.70:0/641923370 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:32.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.913+0000 7fb952c4bb80 1 -- 172.21.15.70:0/641923370 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ad4083cc80 con 0x55ad40834720 2024-06-04T05:14:32.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.914+0000 7fb952c4bb80 10 client.0 _ll_register_callbacks cb 0x55ad40718a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:32.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.914+0000 7fb946152700 1 --2- 172.21.15.70:0/641923370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad4082fbf0 0x55ad4083c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.914+0000 7fb946953700 1 --2- 172.21.15.70:0/641923370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad40834720 0x55ad40839b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.914+0000 7fb945951700 1 -- 172.21.15.70:0/641923370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad40834720 msgr2=0x55ad40839b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:32.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.914+0000 7fb945951700 1 --2- 172.21.15.70:0/641923370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad40834720 0x55ad40839b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.914+0000 7fb945951700 1 -- 172.21.15.70:0/641923370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad4082fbf0 msgr2=0x55ad4083c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:32.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.914+0000 7fb945951700 1 --2- 172.21.15.70:0/641923370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad4082fbf0 0x55ad4083c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.914+0000 7fb945951700 1 -- 172.21.15.70:0/641923370 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad406cd9d0 con 0x55ad408305b0 2024-06-04T05:14:32.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.914+0000 7fb945951700 1 --2- 172.21.15.70:0/641923370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad408305b0 0x55ad408395c0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fb93000e4a0 tx=0x7fb93000e7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2558e8b4cac84b48 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:32.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.914+0000 7fb929ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:14:32.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.914+0000 7fb929ffb700 1 -- 172.21.15.70:0/641923370 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb930017020 con 0x55ad408305b0 2024-06-04T05:14:32.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.914+0000 7fb929ffb700 1 -- 172.21.15.70:0/641923370 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fb930013410 con 0x55ad408305b0 2024-06-04T05:14:32.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.914+0000 7fb952c4bb80 1 -- 172.21.15.70:0/641923370 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb93c014ad0 con 0x55ad408305b0 2024-06-04T05:14:32.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.914+0000 7fb929ffb700 1 -- 172.21.15.70:0/641923370 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fb930022050 con 0x55ad408305b0 2024-06-04T05:14:32.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.914+0000 7fb952c4bb80 10 client.19896 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:32.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.914+0000 7fb952c4bb80 1 -- 172.21.15.70:0/641923370 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55ad40737860 con 0x55ad408305b0 2024-06-04T05:14:32.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.915+0000 7fb929ffb700 1 -- 172.21.15.70:0/641923370 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb93c014ad0 con 0x55ad408305b0 2024-06-04T05:14:32.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.915+0000 7fb929ffb700 1 -- 172.21.15.70:0/641923370 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7fb930012050 con 0x55ad408305b0 2024-06-04T05:14:32.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.915+0000 7fb952c4bb80 10 client.19896 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:32.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.915+0000 7fb952c4bb80 10 client.19896 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:14:32.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.915+0000 7fb952c4bb80 20 client.19896 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:32.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.915+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/641923370 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ad408497b0 0x55ad4084bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.915+0000 7fb952c4bb80 4 client.19896 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:14:32.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.915+0000 7fb952c4bb80 1 -- 172.21.15.70:0/641923370 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad4078fdc0 con 0x55ad408497b0 2024-06-04T05:14:32.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.915+0000 7fb946152700 1 --2- 172.21.15.70:0/641923370 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ad408497b0 0x55ad4084bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.916+0000 7fb946152700 1 --2- 172.21.15.70:0/641923370 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ad408497b0 0x55ad4084bc00 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e4abae09f3c9fc5 server_cookie=d35a00f426848bd0 in_seq=0 out_seq=0 2024-06-04T05:14:32.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.916+0000 7fb929ffb700 10 client.19896 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:32.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.916+0000 7fb929ffb700 1 -- 172.21.15.70:0/641923370 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ad4078fdc0 con 0x55ad408497b0 2024-06-04T05:14:32.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.916+0000 7fb929ffb700 10 client.19896 handle_command_reply: tid=0 2024-06-04T05:14:32.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.916+0000 7fb952c4bb80 1 client.19896 shutdown 2024-06-04T05:14:32.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.917+0000 7fb952c4bb80 1 -- 172.21.15.70:0/641923370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad408305b0 msgr2=0x55ad408395c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:32.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.917+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/641923370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad408305b0 0x55ad408395c0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fb93000e4a0 tx=0x7fb93000e7b0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.917+0000 7fb952c4bb80 1 -- 172.21.15.70:0/641923370 shutdown_connections 2024-06-04T05:14:32.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.917+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/641923370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad40834720 0x55ad40839b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.917+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/641923370 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ad408497b0 0x55ad4084bc00 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.917+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/641923370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad4082fbf0 0x55ad4083c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.917+0000 7fb929ffb700 0 client.19896 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:32.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.917+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/641923370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad408305b0 0x55ad408395c0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.917+0000 7fb952c4bb80 1 -- 172.21.15.70:0/641923370 >> 172.21.15.70:0/641923370 conn(0x55ad4075b340 msgr2=0x55ad4082ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:32.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.917+0000 7fb952c4bb80 1 -- 172.21.15.70:0/641923370 shutdown_connections 2024-06-04T05:14:32.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.917+0000 7fb952c4bb80 1 -- 172.21.15.70:0/641923370 wait complete. 2024-06-04T05:14:32.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.917+0000 7fb952c4bb80 20 client.19896 trim_cache size 0 max 16384 2024-06-04T05:14:32.940 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:14:32.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.938+0000 7fb952c4bb80 1 Processor -- start 2024-06-04T05:14:32.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.938+0000 7fb952c4bb80 1 -- start start 2024-06-04T05:14:32.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.938+0000 7fb952c4bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad4082fbf0 0x55ad40838220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.939+0000 7fb952c4bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad408305b0 0x55ad40845ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.939+0000 7fb952c4bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad40834720 0x55ad40839bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.939+0000 7fb952c4bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9401abc80 con 0x55ad4082fbf0 2024-06-04T05:14:32.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.939+0000 7fb952c4bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9401ad820 con 0x55ad408305b0 2024-06-04T05:14:32.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.939+0000 7fb952c4bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb9401ac3b0 con 0x55ad40834720 2024-06-04T05:14:32.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.939+0000 7fb946152700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad4082fbf0 0x55ad40838220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.939+0000 7fb946152700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad4082fbf0 0x55ad40838220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52432/0 (socket says 172.21.15.70:52432) 2024-06-04T05:14:32.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.939+0000 7fb946152700 1 -- 172.21.15.70:0/4274571236 learned_addr learned my addr 172.21.15.70:0/4274571236 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:32.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.939+0000 7fb945951700 1 --2- 172.21.15.70:0/4274571236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad408305b0 0x55ad40845ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.939+0000 7fb946953700 1 --2- 172.21.15.70:0/4274571236 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad40834720 0x55ad40839bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.939+0000 7fb946152700 1 -- 172.21.15.70:0/4274571236 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad40834720 msgr2=0x55ad40839bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:32.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.939+0000 7fb946152700 1 --2- 172.21.15.70:0/4274571236 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad40834720 0x55ad40839bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.939+0000 7fb946152700 1 -- 172.21.15.70:0/4274571236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad408305b0 msgr2=0x55ad40845ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:32.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.939+0000 7fb946152700 1 --2- 172.21.15.70:0/4274571236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad408305b0 0x55ad40845ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.939+0000 7fb946152700 1 -- 172.21.15.70:0/4274571236 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad40837b30 con 0x55ad4082fbf0 2024-06-04T05:14:32.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.939+0000 7fb946152700 1 --2- 172.21.15.70:0/4274571236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad4082fbf0 0x55ad40838220 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00ddb0 tx=0x7fb93c0148a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a5f818edeeccc0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:32.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.939+0000 7fb92bfff700 1 -- 172.21.15.70:0/4274571236 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c024040 con 0x55ad4082fbf0 2024-06-04T05:14:32.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.940+0000 7fb92bfff700 1 -- 172.21.15.70:0/4274571236 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb93c02f070 con 0x55ad4082fbf0 2024-06-04T05:14:32.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.940+0000 7fb92bfff700 1 -- 172.21.15.70:0/4274571236 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c030bf0 con 0x55ad4082fbf0 2024-06-04T05:14:32.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.940+0000 7fb952c4bb80 1 -- 172.21.15.70:0/4274571236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad4082fbf0 msgr2=0x55ad40838220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:32.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.940+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/4274571236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad4082fbf0 0x55ad40838220 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00ddb0 tx=0x7fb93c0148a0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.940+0000 7fb952c4bb80 1 -- 172.21.15.70:0/4274571236 shutdown_connections 2024-06-04T05:14:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.940+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/4274571236 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad40834720 0x55ad40839bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.940+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/4274571236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad4082fbf0 0x55ad40838220 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.940+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/4274571236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad408305b0 0x55ad40845ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.940+0000 7fb952c4bb80 1 -- 172.21.15.70:0/4274571236 >> 172.21.15.70:0/4274571236 conn(0x55ad4075b340 msgr2=0x55ad408336d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.940+0000 7fb952c4bb80 1 -- 172.21.15.70:0/4274571236 shutdown_connections 2024-06-04T05:14:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.940+0000 7fb952c4bb80 1 -- 172.21.15.70:0/4274571236 wait complete. 2024-06-04T05:14:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.941+0000 7fb952c4bb80 1 -- start start 2024-06-04T05:14:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.941+0000 7fb952c4bb80 1 Processor -- start 2024-06-04T05:14:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.941+0000 7fb952c4bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad4082fbf0 0x55ad408420c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.941+0000 7fb952c4bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad408305b0 0x55ad40842600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.941+0000 7fb952c4bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad40834720 0x55ad4083ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.941+0000 7fb952c4bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9401ad150 con 0x55ad408305b0 2024-06-04T05:14:32.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.941+0000 7fb952c4bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9401aca80 con 0x55ad4082fbf0 2024-06-04T05:14:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.941+0000 7fb952c4bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb9401a9c60 con 0x55ad40834720 2024-06-04T05:14:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.941+0000 7fb952c4bb80 10 client.0 _ll_register_callbacks cb 0x55ad40718a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.941+0000 7fb945951700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad408305b0 0x55ad40842600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.941+0000 7fb945951700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad408305b0 0x55ad40842600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52438/0 (socket says 172.21.15.70:52438) 2024-06-04T05:14:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.941+0000 7fb945951700 1 -- 172.21.15.70:0/2005140409 learned_addr learned my addr 172.21.15.70:0/2005140409 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.941+0000 7fb946152700 1 --2- 172.21.15.70:0/2005140409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad4082fbf0 0x55ad408420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.941+0000 7fb946953700 1 --2- 172.21.15.70:0/2005140409 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad40834720 0x55ad4083ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.941+0000 7fb945951700 1 -- 172.21.15.70:0/2005140409 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad40834720 msgr2=0x55ad4083ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.941+0000 7fb945951700 1 --2- 172.21.15.70:0/2005140409 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad40834720 0x55ad4083ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.942+0000 7fb945951700 1 -- 172.21.15.70:0/2005140409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad4082fbf0 msgr2=0x55ad408420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.942+0000 7fb945951700 1 --2- 172.21.15.70:0/2005140409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad4082fbf0 0x55ad408420c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.942+0000 7fb945951700 1 -- 172.21.15.70:0/2005140409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad4083d5e0 con 0x55ad408305b0 2024-06-04T05:14:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.942+0000 7fb946152700 1 --2- 172.21.15.70:0/2005140409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad4082fbf0 0x55ad408420c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:32.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.942+0000 7fb945951700 1 --2- 172.21.15.70:0/2005140409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad408305b0 0x55ad40842600 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fb930009ad0 tx=0x7fb93000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0ecfaedc9161a0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.942+0000 7fb929ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:14:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.942+0000 7fb929ffb700 1 -- 172.21.15.70:0/2005140409 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93002a070 con 0x55ad408305b0 2024-06-04T05:14:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.942+0000 7fb929ffb700 1 -- 172.21.15.70:0/2005140409 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fb93001bd20 con 0x55ad408305b0 2024-06-04T05:14:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.942+0000 7fb952c4bb80 1 -- 172.21.15.70:0/2005140409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb93c02f070 con 0x55ad408305b0 2024-06-04T05:14:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.942+0000 7fb929ffb700 1 -- 172.21.15.70:0/2005140409 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fb930036050 con 0x55ad408305b0 2024-06-04T05:14:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.942+0000 7fb952c4bb80 10 client.19902 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.942+0000 7fb952c4bb80 1 -- 172.21.15.70:0/2005140409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55ad408b4ec0 con 0x55ad408305b0 2024-06-04T05:14:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.942+0000 7fb929ffb700 1 -- 172.21.15.70:0/2005140409 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb93c02f070 con 0x55ad408305b0 2024-06-04T05:14:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.943+0000 7fb929ffb700 1 -- 172.21.15.70:0/2005140409 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7fb930012040 con 0x55ad408305b0 2024-06-04T05:14:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.943+0000 7fb952c4bb80 10 client.19902 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.943+0000 7fb952c4bb80 10 client.19902 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:14:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.943+0000 7fb952c4bb80 20 client.19902 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.943+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/2005140409 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ad408497b0 0x55ad408b7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.943+0000 7fb952c4bb80 4 client.19902 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "f2bf973b", "release": true, "await": true}] 2024-06-04T05:14:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.943+0000 7fb952c4bb80 1 -- 172.21.15.70:0/2005140409 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "f2bf973b", "release": true, "await": true}) -- 0x55ad407895a0 con 0x55ad408497b0 2024-06-04T05:14:32.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.943+0000 7fb946152700 1 --2- 172.21.15.70:0/2005140409 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ad408497b0 0x55ad408b7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:32.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.943+0000 7fb929ffb700 10 client.19902 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:32.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.943+0000 7fb946152700 1 --2- 172.21.15.70:0/2005140409 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ad408497b0 0x55ad408b7610 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70138b1565d3efec server_cookie=51a3e7def9097dcd in_seq=0 out_seq=0 2024-06-04T05:14:32.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.973+0000 7fb929ffb700 1 -- 172.21.15.70:0/2005140409 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55ad407895a0 con 0x55ad408497b0 2024-06-04T05:14:32.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.973+0000 7fb929ffb700 10 client.19902 handle_command_reply: tid=0 2024-06-04T05:14:32.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.973+0000 7fb952c4bb80 1 client.19902 shutdown 2024-06-04T05:14:32.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.973+0000 7fb952c4bb80 1 -- 172.21.15.70:0/2005140409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad408305b0 msgr2=0x55ad40842600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:32.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.973+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/2005140409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad408305b0 0x55ad40842600 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fb930009ad0 tx=0x7fb93000edc0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.973+0000 7fb952c4bb80 1 -- 172.21.15.70:0/2005140409 shutdown_connections 2024-06-04T05:14:32.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.973+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/2005140409 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ad40834720 0x55ad4083ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.973+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/2005140409 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ad408497b0 0x55ad408b7610 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.973+0000 7fb929ffb700 0 client.19902 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:32.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.973+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/2005140409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad4082fbf0 0x55ad408420c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.973+0000 7fb952c4bb80 1 --2- 172.21.15.70:0/2005140409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad408305b0 0x55ad40842600 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.973+0000 7fb952c4bb80 1 -- 172.21.15.70:0/2005140409 >> 172.21.15.70:0/2005140409 conn(0x55ad4075b340 msgr2=0x55ad408353c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:32.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.973+0000 7fb952c4bb80 1 -- 172.21.15.70:0/2005140409 shutdown_connections 2024-06-04T05:14:32.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.973+0000 7fb952c4bb80 1 -- 172.21.15.70:0/2005140409 wait complete. 2024-06-04T05:14:32.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.973+0000 7fb952c4bb80 20 client.19902 trim_cache size 0 max 16384 2024-06-04T05:14:32.977 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:14:32.977 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 55, 2024-06-04T05:14:32.977 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:14:32.977 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 52, 2024-06-04T05:14:32.977 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:14:32.977 INFO:teuthology.orchestra.run.smithi070.stdout: "f2bf973b": { 2024-06-04T05:14:32.977 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 52, 2024-06-04T05:14:32.978 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:14:32.978 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:14:32.978 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:14:32.978 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:14:32.978 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:14:32.978 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:14:32.978 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 130.0, 2024-06-04T05:14:32.978 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:14:32.978 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:14:32.978 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:14:32.978 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:14:32.978 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:14:32.978 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:14:32.978 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:14:32.979 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:14:32.979 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:14:32.979 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:14:32.979 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:14:32.979 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:14:32.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.974+0000 7fb9483b6700 1 -- 172.21.15.70:0/763780616 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb92c0b6e00 msgr2=0x7fb92c0b92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:32.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.974+0000 7fb9483b6700 1 --2- 172.21.15.70:0/763780616 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb92c0b6e00 0x7fb92c0b92c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0099d0 tx=0x7fb93c008040 comp rx=0 tx=0).stop 2024-06-04T05:14:32.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.974+0000 7fb9483b6700 1 -- 172.21.15.70:0/763780616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9401084b0 msgr2=0x7fb9401a5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:32.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.974+0000 7fb9483b6700 1 --2- 172.21.15.70:0/763780616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9401084b0 0x7fb9401a5b10 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fb9380062a0 tx=0x7fb938004ed0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.974+0000 7fb9483b6700 1 -- 172.21.15.70:0/763780616 shutdown_connections 2024-06-04T05:14:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.974+0000 7fb9483b6700 1 --2- 172.21.15.70:0/763780616 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb92c0b6e00 0x7fb92c0b92c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.974+0000 7fb9483b6700 1 --2- 172.21.15.70:0/763780616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9401084b0 0x7fb9401a5b10 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.974+0000 7fb9483b6700 1 --2- 172.21.15.70:0/763780616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb940102470 0x7fb94019aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.974+0000 7fb9483b6700 1 --2- 172.21.15.70:0/763780616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb940102e30 0x7fb94019afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.974+0000 7fb9483b6700 1 -- 172.21.15.70:0/763780616 >> 172.21.15.70:0/763780616 conn(0x7fb940079410 msgr2=0x7fb9401005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.975+0000 7fb9483b6700 1 -- 172.21.15.70:0/763780616 shutdown_connections 2024-06-04T05:14:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:32.975+0000 7fb9483b6700 1 -- 172.21.15.70:0/763780616 wait complete. 2024-06-04T05:14:33.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:33 smithi070 ceph-mon[32631]: pgmap v544: 97 pgs: 97 active+clean; 539 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 2.3 MiB/s wr, 239 op/s 2024-06-04T05:14:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:33 smithi112 ceph-mon[36453]: pgmap v544: 97 pgs: 97 active+clean; 539 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 2.3 MiB/s wr, 239 op/s 2024-06-04T05:14:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:33 smithi098 ceph-mon[35569]: pgmap v544: 97 pgs: 97 active+clean; 539 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 2.3 MiB/s wr, 239 op/s 2024-06-04T05:14:33.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 29088 7.86 MB/sec execute 85 sec latency 10089.538 ms 2024-06-04T05:14:33.995 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'f2bf973b', total seconds elapsed: 28.1 2024-06-04T05:14:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.585+0000 7f1e0e609640 1 -- 172.21.15.70:0/4078052666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e081033f0 msgr2=0x7f1e0810f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.585+0000 7f1e0e609640 1 --2- 172.21.15.70:0/4078052666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e081033f0 0x7f1e0810f900 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f1df8009930 tx=0x7f1df802a440 comp rx=0 tx=0).stop 2024-06-04T05:14:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.585+0000 7f1e0e609640 1 -- 172.21.15.70:0/4078052666 shutdown_connections 2024-06-04T05:14:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.585+0000 7f1e0e609640 1 --2- 172.21.15.70:0/4078052666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e08102a30 0x7f1e08102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.585+0000 7f1e0e609640 1 --2- 172.21.15.70:0/4078052666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e081033f0 0x7f1e0810f900 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.585+0000 7f1e0e609640 1 --2- 172.21.15.70:0/4078052666 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e08108a40 0x7f1e08108e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.585+0000 7f1e0e609640 1 -- 172.21.15.70:0/4078052666 >> 172.21.15.70:0/4078052666 conn(0x7f1e08079ee0 msgr2=0x7f1e0807a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.585+0000 7f1e0e609640 1 -- 172.21.15.70:0/4078052666 shutdown_connections 2024-06-04T05:14:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.585+0000 7f1e0e609640 1 -- 172.21.15.70:0/4078052666 wait complete. 2024-06-04T05:14:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.586+0000 7f1e0e609640 1 Processor -- start 2024-06-04T05:14:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.586+0000 7f1e0e609640 1 -- start start 2024-06-04T05:14:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.586+0000 7f1e0e609640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e08102a30 0x7f1e08074d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.586+0000 7f1e0e609640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e081033f0 0x7f1e08075290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.586+0000 7f1e0e609640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e08108a40 0x7f1e080757d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.586+0000 7f1e0e609640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1e08112bb0 con 0x7f1e08108a40 2024-06-04T05:14:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.586+0000 7f1e0e609640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e08112a30 con 0x7f1e081033f0 2024-06-04T05:14:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.586+0000 7f1e0e609640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1e08112d30 con 0x7f1e08102a30 2024-06-04T05:14:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.586+0000 7f1e07fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e08102a30 0x7f1e08074d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.586+0000 7f1e0cba6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e08108a40 0x7f1e080757d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.586+0000 7f1e07fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e08102a30 0x7f1e08074d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:60946/0 (socket says 172.21.15.70:60946) 2024-06-04T05:14:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.586+0000 7f1e07fff640 1 -- 172.21.15.70:0/618153003 learned_addr learned my addr 172.21.15.70:0/618153003 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.587+0000 7f1e077fe640 1 --2- 172.21.15.70:0/618153003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e081033f0 0x7f1e08075290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.587+0000 7f1e0cba6640 1 -- 172.21.15.70:0/618153003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e08102a30 msgr2=0x7f1e08074d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.587+0000 7f1e0cba6640 1 --2- 172.21.15.70:0/618153003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e08102a30 0x7f1e08074d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.587+0000 7f1e0cba6640 1 -- 172.21.15.70:0/618153003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e081033f0 msgr2=0x7f1e08075290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.587+0000 7f1e0cba6640 1 --2- 172.21.15.70:0/618153003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e081033f0 0x7f1e08075290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.587+0000 7f1e0cba6640 1 -- 172.21.15.70:0/618153003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e081a7550 con 0x7f1e08108a40 2024-06-04T05:14:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.587+0000 7f1e0cba6640 1 --2- 172.21.15.70:0/618153003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e08108a40 0x7f1e080757d0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f1df802a960 tx=0x7f1df80318f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.587+0000 7f1e057fa640 1 -- 172.21.15.70:0/618153003 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df803f030 con 0x7f1e08108a40 2024-06-04T05:14:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.587+0000 7f1e0e609640 1 -- 172.21.15.70:0/618153003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e081a77e0 con 0x7f1e08108a40 2024-06-04T05:14:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.587+0000 7f1e057fa640 1 -- 172.21.15.70:0/618153003 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1df8004350 con 0x7f1e08108a40 2024-06-04T05:14:34.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.587+0000 7f1e057fa640 1 -- 172.21.15.70:0/618153003 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df8035070 con 0x7f1e08108a40 2024-06-04T05:14:34.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.587+0000 7f1e0e609640 1 -- 172.21.15.70:0/618153003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e081a7cf0 con 0x7f1e08108a40 2024-06-04T05:14:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.591+0000 7f1e113fe740 1 Processor -- start 2024-06-04T05:14:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.591+0000 7f1e057fa640 1 -- 172.21.15.70:0/618153003 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f1df8042030 con 0x7f1e08108a40 2024-06-04T05:14:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.591+0000 7f1e113fe740 1 -- start start 2024-06-04T05:14:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.591+0000 7f1e113fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654284820 0x557654284c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.591+0000 7f1e113fe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557654285140 0x55765428f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.591+0000 7f1e113fe740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55765428fa00 0x557654291df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.591+0000 7f1e113fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5576541f6c30 con 0x557654284820 2024-06-04T05:14:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.591+0000 7f1e113fe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5576541e67d0 con 0x557654285140 2024-06-04T05:14:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.591+0000 7f1e113fe740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5576541ea810 con 0x55765428fa00 2024-06-04T05:14:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.591+0000 7f1e077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654284820 0x557654284c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.591+0000 7f1e07fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557654285140 0x55765428f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.592+0000 7f1e077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654284820 0x557654284c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52458/0 (socket says 172.21.15.70:52458) 2024-06-04T05:14:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.592+0000 7f1e07fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557654285140 0x55765428f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55006/0 (socket says 172.21.15.70:55006) 2024-06-04T05:14:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.592+0000 7f1e077fe640 1 -- 172.21.15.70:0/1819377136 learned_addr learned my addr 172.21.15.70:0/1819377136 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.592+0000 7f1e077fe640 1 --2- 172.21.15.70:0/1819377136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55765428fa00 0x557654291df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.592+0000 7f1e07fff640 1 -- 172.21.15.70:0/1819377136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55765428fa00 msgr2=0x557654291df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.592+0000 7f1e07fff640 1 --2- 172.21.15.70:0/1819377136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55765428fa00 0x557654291df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.592+0000 7f1e07fff640 1 -- 172.21.15.70:0/1819377136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654284820 msgr2=0x557654284c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.592+0000 7f1e07fff640 1 --2- 172.21.15.70:0/1819377136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654284820 0x557654284c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.592+0000 7f1e07fff640 1 -- 172.21.15.70:0/1819377136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557653efb210 con 0x557654285140 2024-06-04T05:14:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.592+0000 7f1e077fe640 1 --2- 172.21.15.70:0/1819377136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654284820 0x557654284c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.592+0000 7f1e057fa640 1 --2- 172.21.15.70:0/618153003 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1df00b2ac0 0x7f1df00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.592+0000 7f1e077fe640 1 --2- 172.21.15.70:0/618153003 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1df00b2ac0 0x7f1df00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.592+0000 7f1e07fff640 1 --2- 172.21.15.70:0/1819377136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557654285140 0x55765428f4c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1e00009f70 tx=0x7f1e00011870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39b3ba2426353a8a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.592+0000 7f1e057fa640 1 -- 172.21.15.70:0/618153003 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f1df8102960 con 0x7f1e08108a40 2024-06-04T05:14:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.592+0000 7f1dee7fc640 1 -- 172.21.15.70:0/1819377136 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e00003b70 con 0x557654285140 2024-06-04T05:14:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.592+0000 7f1dee7fc640 1 -- 172.21.15.70:0/1819377136 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e00003d10 con 0x557654285140 2024-06-04T05:14:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.592+0000 7f1dee7fc640 1 -- 172.21.15.70:0/1819377136 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e000128b0 con 0x557654285140 2024-06-04T05:14:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.593+0000 7f1e113fe740 1 -- 172.21.15.70:0/1819377136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557654285140 msgr2=0x55765428f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.593+0000 7f1e113fe740 1 --2- 172.21.15.70:0/1819377136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557654285140 0x55765428f4c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1e00009f70 tx=0x7f1e00011870 comp rx=0 tx=0).stop 2024-06-04T05:14:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.593+0000 7f1e077fe640 1 --2- 172.21.15.70:0/618153003 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1df00b2ac0 0x7f1df00b4f80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x557654282150 tx=0x7f1dfc0096e0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.593+0000 7f1e113fe740 1 -- 172.21.15.70:0/1819377136 shutdown_connections 2024-06-04T05:14:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.593+0000 7f1e113fe740 1 --2- 172.21.15.70:0/1819377136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557654285140 0x55765428f4c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.593+0000 7f1e113fe740 1 --2- 172.21.15.70:0/1819377136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55765428fa00 0x557654291df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.593+0000 7f1e113fe740 1 --2- 172.21.15.70:0/1819377136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654284820 0x557654284c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.593+0000 7f1e113fe740 1 -- 172.21.15.70:0/1819377136 >> 172.21.15.70:0/1819377136 conn(0x557654280d30 msgr2=0x557654281110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.593+0000 7f1e113fe740 1 -- 172.21.15.70:0/1819377136 shutdown_connections 2024-06-04T05:14:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.593+0000 7f1e113fe740 1 -- 172.21.15.70:0/1819377136 wait complete. 2024-06-04T05:14:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.594+0000 7f1e113fe740 1 -- start start 2024-06-04T05:14:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.594+0000 7f1e113fe740 1 Processor -- start 2024-06-04T05:14:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.594+0000 7f1e113fe740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557654284820 0x55765428c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.594+0000 7f1e113fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654285140 0x55765428ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.594+0000 7f1e113fe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55765428fa00 0x55765428d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.594+0000 7f1e113fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557653ecc790 con 0x557654285140 2024-06-04T05:14:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.594+0000 7f1e113fe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5576541a2890 con 0x55765428fa00 2024-06-04T05:14:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.594+0000 7f1e113fe740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557654292630 con 0x557654284820 2024-06-04T05:14:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.594+0000 7f1e113fe740 10 client.0 _ll_register_callbacks cb 0x557653efaa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.594+0000 7f1e077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654285140 0x55765428ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.594+0000 7f1e07fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557654284820 0x55765428c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.595+0000 7f1e077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654285140 0x55765428ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52462/0 (socket says 172.21.15.70:52462) 2024-06-04T05:14:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.595+0000 7f1e07fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557654284820 0x55765428c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:60962/0 (socket says 172.21.15.70:60962) 2024-06-04T05:14:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.595+0000 7f1e077fe640 1 -- 172.21.15.70:0/3763077545 learned_addr learned my addr 172.21.15.70:0/3763077545 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.595+0000 7f1e0cba6640 1 --2- 172.21.15.70:0/3763077545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55765428fa00 0x55765428d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.595+0000 7f1e077fe640 1 -- 172.21.15.70:0/3763077545 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557654284820 msgr2=0x55765428c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.595+0000 7f1e077fe640 1 --2- 172.21.15.70:0/3763077545 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557654284820 0x55765428c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.595+0000 7f1e077fe640 1 -- 172.21.15.70:0/3763077545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55765428fa00 msgr2=0x55765428d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.595+0000 7f1e077fe640 1 --2- 172.21.15.70:0/3763077545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55765428fa00 0x55765428d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.595+0000 7f1e077fe640 1 -- 172.21.15.70:0/3763077545 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557653fc7620 con 0x557654285140 2024-06-04T05:14:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.595+0000 7f1e077fe640 1 --2- 172.21.15.70:0/3763077545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654285140 0x55765428ce90 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc00ce50 tx=0x7f1dfc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ea3a4289884688 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.595+0000 7f1de77fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:14:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.595+0000 7f1de77fe640 1 -- 172.21.15.70:0/3763077545 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc02e060 con 0x557654285140 2024-06-04T05:14:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.595+0000 7f1de77fe640 1 -- 172.21.15.70:0/3763077545 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f1dfc00f9d0 con 0x557654285140 2024-06-04T05:14:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.595+0000 7f1e113fe740 1 -- 172.21.15.70:0/3763077545 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e00003d10 con 0x557654285140 2024-06-04T05:14:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.596+0000 7f1de77fe640 1 -- 172.21.15.70:0/3763077545 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f1dfc00fbd0 con 0x557654285140 2024-06-04T05:14:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.596+0000 7f1e113fe740 10 client.19914 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.596+0000 7f1e113fe740 1 -- 172.21.15.70:0/3763077545 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55765429e330 con 0x557654285140 2024-06-04T05:14:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.596+0000 7f1de77fe640 1 -- 172.21.15.70:0/3763077545 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e00003d10 con 0x557654285140 2024-06-04T05:14:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.596+0000 7f1de77fe640 1 -- 172.21.15.70:0/3763077545 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f1dfc028030 con 0x557654285140 2024-06-04T05:14:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.596+0000 7f1e113fe740 10 client.19914 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.596+0000 7f1e113fe740 10 client.19914 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:14:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.596+0000 7f1e113fe740 20 client.19914 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.596+0000 7f1e113fe740 1 --2- 172.21.15.70:0/3763077545 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55765429f840 0x5576542a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.596+0000 7f1e113fe740 4 client.19914 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:14:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.596+0000 7f1e113fe740 1 -- 172.21.15.70:0/3763077545 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576541f8b80 con 0x55765429f840 2024-06-04T05:14:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.596+0000 7f1e07fff640 1 --2- 172.21.15.70:0/3763077545 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55765429f840 0x5576542a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.597+0000 7f1de77fe640 10 client.19914 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.597+0000 7f1e07fff640 1 --2- 172.21.15.70:0/3763077545 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55765429f840 0x5576542a1c60 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d59583cbbface88b server_cookie=f341ce2efd8b723e in_seq=0 out_seq=0 2024-06-04T05:14:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.598+0000 7f1de77fe640 1 -- 172.21.15.70:0/3763077545 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5576541f8b80 con 0x55765429f840 2024-06-04T05:14:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.598+0000 7f1de77fe640 10 client.19914 handle_command_reply: tid=0 2024-06-04T05:14:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.598+0000 7f1e113fe740 1 client.19914 shutdown 2024-06-04T05:14:34.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.598+0000 7f1e113fe740 1 -- 172.21.15.70:0/3763077545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654285140 msgr2=0x55765428ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:34.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.598+0000 7f1e113fe740 1 --2- 172.21.15.70:0/3763077545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654285140 0x55765428ce90 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc00ce50 tx=0x7f1dfc005f90 comp rx=0 tx=0).stop 2024-06-04T05:14:34.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.598+0000 7f1e113fe740 1 -- 172.21.15.70:0/3763077545 shutdown_connections 2024-06-04T05:14:34.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.598+0000 7f1e113fe740 1 --2- 172.21.15.70:0/3763077545 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55765429f840 0x5576542a1c60 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.598+0000 7f1e113fe740 1 --2- 172.21.15.70:0/3763077545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55765428fa00 0x55765428d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.598+0000 7f1e113fe740 1 --2- 172.21.15.70:0/3763077545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654285140 0x55765428ce90 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.598+0000 7f1e113fe740 1 --2- 172.21.15.70:0/3763077545 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557654284820 0x55765428c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.598+0000 7f1e113fe740 1 -- 172.21.15.70:0/3763077545 >> 172.21.15.70:0/3763077545 conn(0x557654280d30 msgr2=0x55765427ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:34.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.598+0000 7f1de77fe640 0 client.19914 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:34.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.598+0000 7f1e113fe740 1 -- 172.21.15.70:0/3763077545 shutdown_connections 2024-06-04T05:14:34.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.598+0000 7f1e113fe740 1 -- 172.21.15.70:0/3763077545 wait complete. 2024-06-04T05:14:34.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.598+0000 7f1e113fe740 20 client.19914 trim_cache size 0 max 16384 2024-06-04T05:14:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.625+0000 7f1e113fe740 1 Processor -- start 2024-06-04T05:14:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.625+0000 7f1e113fe740 1 -- start start 2024-06-04T05:14:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.626+0000 7f1e113fe740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557654280d30 0x55765428d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.626+0000 7f1e113fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654284820 0x5576542a7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.626+0000 7f1e113fe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55765428fa00 0x5576542a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.626+0000 7f1e113fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1e081b08e0 con 0x557654284820 2024-06-04T05:14:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.626+0000 7f1e113fe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e081b2480 con 0x55765428fa00 2024-06-04T05:14:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.626+0000 7f1e113fe740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1e081b1010 con 0x557654280d30 2024-06-04T05:14:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.626+0000 7f1e077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654284820 0x5576542a7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.626+0000 7f1e077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654284820 0x5576542a7530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52472/0 (socket says 172.21.15.70:52472) 2024-06-04T05:14:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.626+0000 7f1e077fe640 1 -- 172.21.15.70:0/1696033357 learned_addr learned my addr 172.21.15.70:0/1696033357 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:34.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.626+0000 7f1e0cba6640 1 --2- 172.21.15.70:0/1696033357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55765428fa00 0x5576542a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.626+0000 7f1e07fff640 1 --2- 172.21.15.70:0/1696033357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557654280d30 0x55765428d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.626+0000 7f1e077fe640 1 -- 172.21.15.70:0/1696033357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557654280d30 msgr2=0x55765428d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:34.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.626+0000 7f1e077fe640 1 --2- 172.21.15.70:0/1696033357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557654280d30 0x55765428d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.626+0000 7f1e077fe640 1 -- 172.21.15.70:0/1696033357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55765428fa00 msgr2=0x5576542a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:34.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.626+0000 7f1e077fe640 1 --2- 172.21.15.70:0/1696033357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55765428fa00 0x5576542a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.626+0000 7f1e077fe640 1 -- 172.21.15.70:0/1696033357 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55765427c4d0 con 0x557654284820 2024-06-04T05:14:34.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.627+0000 7f1e077fe640 1 --2- 172.21.15.70:0/1696033357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654284820 0x5576542a7530 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0099e0 tx=0x7f1dfc00ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e8d303cd4ad4fe server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:34.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.627+0000 7f1dedffb640 1 -- 172.21.15.70:0/1696033357 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc02e040 con 0x557654284820 2024-06-04T05:14:34.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.627+0000 7f1dedffb640 1 -- 172.21.15.70:0/1696033357 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1dfc0054c0 con 0x557654284820 2024-06-04T05:14:34.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.627+0000 7f1dedffb640 1 -- 172.21.15.70:0/1696033357 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc004990 con 0x557654284820 2024-06-04T05:14:34.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.628+0000 7f1e113fe740 1 -- 172.21.15.70:0/1696033357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654284820 msgr2=0x5576542a7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.628+0000 7f1e113fe740 1 --2- 172.21.15.70:0/1696033357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654284820 0x5576542a7530 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0099e0 tx=0x7f1dfc00ff00 comp rx=0 tx=0).stop 2024-06-04T05:14:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.628+0000 7f1e113fe740 1 -- 172.21.15.70:0/1696033357 shutdown_connections 2024-06-04T05:14:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.628+0000 7f1e113fe740 1 --2- 172.21.15.70:0/1696033357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55765428fa00 0x5576542a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.628+0000 7f1e113fe740 1 --2- 172.21.15.70:0/1696033357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654284820 0x5576542a7530 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.628+0000 7f1e113fe740 1 --2- 172.21.15.70:0/1696033357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557654280d30 0x55765428d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.628+0000 7f1e113fe740 1 -- 172.21.15.70:0/1696033357 >> 172.21.15.70:0/1696033357 conn(0x557654288320 msgr2=0x55765428c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.628+0000 7f1e113fe740 1 -- 172.21.15.70:0/1696033357 shutdown_connections 2024-06-04T05:14:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.628+0000 7f1e113fe740 1 -- 172.21.15.70:0/1696033357 wait complete. 2024-06-04T05:14:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.628+0000 7f1e113fe740 1 -- start start 2024-06-04T05:14:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.628+0000 7f1e113fe740 1 Processor -- start 2024-06-04T05:14:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e113fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654280d30 0x557654297e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e113fe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557654284820 0x557654293210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e113fe740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55765428fa00 0x557654293750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e113fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1e081b1db0 con 0x557654280d30 2024-06-04T05:14:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e113fe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e081b16e0 con 0x557654284820 2024-06-04T05:14:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e113fe740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1e081ae8e0 con 0x55765428fa00 2024-06-04T05:14:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e07fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654280d30 0x557654297e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e113fe740 10 client.0 _ll_register_callbacks cb 0x7f1dfc00cb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e07fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654280d30 0x557654297e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52482/0 (socket says 172.21.15.70:52482) 2024-06-04T05:14:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e07fff640 1 -- 172.21.15.70:0/1269236502 learned_addr learned my addr 172.21.15.70:0/1269236502 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e077fe640 1 --2- 172.21.15.70:0/1269236502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557654284820 0x557654293210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e0cba6640 1 --2- 172.21.15.70:0/1269236502 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55765428fa00 0x557654293750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e07fff640 1 -- 172.21.15.70:0/1269236502 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55765428fa00 msgr2=0x557654293750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e07fff640 1 --2- 172.21.15.70:0/1269236502 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55765428fa00 0x557654293750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e07fff640 1 -- 172.21.15.70:0/1269236502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557654284820 msgr2=0x557654293210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e07fff640 1 --2- 172.21.15.70:0/1269236502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557654284820 0x557654293210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e07fff640 1 -- 172.21.15.70:0/1269236502 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557654293e40 con 0x557654280d30 2024-06-04T05:14:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.629+0000 7f1e077fe640 1 --2- 172.21.15.70:0/1269236502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557654284820 0x557654293210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.630+0000 7f1e07fff640 1 --2- 172.21.15.70:0/1269236502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654280d30 0x557654297e50 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f1e00003a10 tx=0x7f1e000097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b91a300de874ae server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.630+0000 7f1de77fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:14:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.630+0000 7f1de77fe640 1 -- 172.21.15.70:0/1269236502 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0001f040 con 0x557654280d30 2024-06-04T05:14:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.630+0000 7f1de77fe640 1 -- 172.21.15.70:0/1269236502 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f1e00004d00 con 0x557654280d30 2024-06-04T05:14:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.630+0000 7f1e113fe740 1 -- 172.21.15.70:0/1269236502 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dfc0054c0 con 0x557654280d30 2024-06-04T05:14:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.630+0000 7f1de77fe640 1 -- 172.21.15.70:0/1269236502 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f1e000125a0 con 0x557654280d30 2024-06-04T05:14:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.630+0000 7f1e113fe740 10 client.19920 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.630+0000 7f1e113fe740 1 -- 172.21.15.70:0/1269236502 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55765430a310 con 0x557654280d30 2024-06-04T05:14:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.630+0000 7f1de77fe640 1 -- 172.21.15.70:0/1269236502 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1dfc0054c0 con 0x557654280d30 2024-06-04T05:14:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.631+0000 7f1de77fe640 1 -- 172.21.15.70:0/1269236502 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f1e0000f040 con 0x557654280d30 2024-06-04T05:14:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.631+0000 7f1e113fe740 10 client.19920 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.631+0000 7f1e113fe740 10 client.19920 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:14:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.631+0000 7f1e113fe740 20 client.19920 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.631+0000 7f1e113fe740 1 --2- 172.21.15.70:0/1269236502 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55765430a760 0x55765430cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.631+0000 7f1e113fe740 4 client.19920 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:14:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.631+0000 7f1e113fe740 1 -- 172.21.15.70:0/1269236502 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5576541f2360 con 0x55765430a760 2024-06-04T05:14:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.631+0000 7f1e077fe640 1 --2- 172.21.15.70:0/1269236502 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55765430a760 0x55765430cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.631+0000 7f1de77fe640 10 client.19920 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.631+0000 7f1e077fe640 1 --2- 172.21.15.70:0/1269236502 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55765430a760 0x55765430cb50 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db6e2c984eae5f27 server_cookie=f8a8fbb359a96d96 in_seq=0 out_seq=0 2024-06-04T05:14:34.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.693+0000 7f1de77fe640 1 -- 172.21.15.70:0/1269236502 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5576541f2360 con 0x55765430a760 2024-06-04T05:14:34.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.693+0000 7f1de77fe640 10 client.19920 handle_command_reply: tid=0 2024-06-04T05:14:34.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.694+0000 7f1e113fe740 1 client.19920 shutdown 2024-06-04T05:14:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.694+0000 7f1e113fe740 1 -- 172.21.15.70:0/1269236502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654280d30 msgr2=0x557654297e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.694+0000 7f1e113fe740 1 --2- 172.21.15.70:0/1269236502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654280d30 0x557654297e50 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f1e00003a10 tx=0x7f1e000097b0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.694+0000 7f1e113fe740 1 -- 172.21.15.70:0/1269236502 shutdown_connections 2024-06-04T05:14:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.694+0000 7f1e113fe740 1 --2- 172.21.15.70:0/1269236502 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55765430a760 0x55765430cb50 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.694+0000 7f1e113fe740 1 --2- 172.21.15.70:0/1269236502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557654284820 0x557654293210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.694+0000 7f1de77fe640 0 client.19920 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.694+0000 7f1e113fe740 1 --2- 172.21.15.70:0/1269236502 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55765428fa00 0x557654293750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.694+0000 7f1e113fe740 1 --2- 172.21.15.70:0/1269236502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557654280d30 0x557654297e50 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.694+0000 7f1e113fe740 1 -- 172.21.15.70:0/1269236502 >> 172.21.15.70:0/1269236502 conn(0x557654288320 msgr2=0x55765428ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.694+0000 7f1e113fe740 1 -- 172.21.15.70:0/1269236502 shutdown_connections 2024-06-04T05:14:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.694+0000 7f1e113fe740 1 -- 172.21.15.70:0/1269236502 wait complete. 2024-06-04T05:14:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.694+0000 7f1e113fe740 20 client.19920 trim_cache size 0 max 16384 2024-06-04T05:14:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:14:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:14:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:14:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.695+0000 7f1e0e609640 1 -- 172.21.15.70:0/618153003 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1df00b2ac0 msgr2=0x7f1df00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.695+0000 7f1e0e609640 1 --2- 172.21.15.70:0/618153003 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1df00b2ac0 0x7f1df00b4f80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x557654282150 tx=0x7f1dfc0096e0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.695+0000 7f1e0e609640 1 -- 172.21.15.70:0/618153003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e08108a40 msgr2=0x7f1e080757d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.695+0000 7f1e0e609640 1 --2- 172.21.15.70:0/618153003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e08108a40 0x7f1e080757d0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f1df802a960 tx=0x7f1df80318f0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.696+0000 7f1e0e609640 1 -- 172.21.15.70:0/618153003 shutdown_connections 2024-06-04T05:14:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.696+0000 7f1e0e609640 1 --2- 172.21.15.70:0/618153003 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1df00b2ac0 0x7f1df00b4f80 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.696+0000 7f1e0e609640 1 --2- 172.21.15.70:0/618153003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e081033f0 0x7f1e08075290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.696+0000 7f1e0e609640 1 --2- 172.21.15.70:0/618153003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e08108a40 0x7f1e080757d0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.696+0000 7f1e0e609640 1 --2- 172.21.15.70:0/618153003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e08102a30 0x7f1e08074d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.696+0000 7f1e0e609640 1 -- 172.21.15.70:0/618153003 >> 172.21.15.70:0/618153003 conn(0x7f1e08079ee0 msgr2=0x7f1e08100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.697+0000 7f1e0e609640 1 -- 172.21.15.70:0/618153003 shutdown_connections 2024-06-04T05:14:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:34.697+0000 7f1e0e609640 1 -- 172.21.15.70:0/618153003 wait complete. 2024-06-04T05:14:34.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 29283 7.92 MB/sec execute 86 sec latency 69.534 ms 2024-06-04T05:14:35.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:35 smithi070 ceph-mon[32631]: pgmap v545: 97 pgs: 97 active+clean; 539 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 820 KiB/s wr, 110 op/s 2024-06-04T05:14:35.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.645+0000 7fdfea54c700 1 -- 172.21.15.70:0/3355524392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfe4108520 msgr2=0x7fdfe4108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:35.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.645+0000 7fdfea54c700 1 --2- 172.21.15.70:0/3355524392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfe4108520 0x7fdfe4108900 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4009990 tx=0x7fdfd4009ca0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.645+0000 7fdfea54c700 1 -- 172.21.15.70:0/3355524392 shutdown_connections 2024-06-04T05:14:35.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.645+0000 7fdfea54c700 1 --2- 172.21.15.70:0/3355524392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4102e00 0x7fdfe410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.645+0000 7fdfea54c700 1 --2- 172.21.15.70:0/3355524392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfe4108520 0x7fdfe4108900 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.645+0000 7fdfea54c700 1 --2- 172.21.15.70:0/3355524392 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfe4102440 0x7fdfe41028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.646+0000 7fdfea54c700 1 -- 172.21.15.70:0/3355524392 >> 172.21.15.70:0/3355524392 conn(0x7fdfe40793f0 msgr2=0x7fdfe4079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:35.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.646+0000 7fdfea54c700 1 -- 172.21.15.70:0/3355524392 shutdown_connections 2024-06-04T05:14:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.646+0000 7fdfea54c700 1 -- 172.21.15.70:0/3355524392 wait complete. 2024-06-04T05:14:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.646+0000 7fdfea54c700 1 Processor -- start 2024-06-04T05:14:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.646+0000 7fdfea54c700 1 -- start start 2024-06-04T05:14:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.647+0000 7fdfea54c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4102440 0x7fdfe4109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.647+0000 7fdfea54c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfe4102e00 0x7fdfe41098d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.647+0000 7fdfea54c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfe4108520 0x7fdfe4109e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.647+0000 7fdfea54c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe41124a0 con 0x7fdfe4102e00 2024-06-04T05:14:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.647+0000 7fdfea54c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe4112320 con 0x7fdfe4102440 2024-06-04T05:14:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.647+0000 7fdfea54c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe4112620 con 0x7fdfe4108520 2024-06-04T05:14:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.647+0000 7fdfe37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfe4102e00 0x7fdfe41098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.647+0000 7fdfe3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4102440 0x7fdfe4109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.647+0000 7fdfe8ae9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfe4108520 0x7fdfe4109e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.647+0000 7fdfe3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4102440 0x7fdfe4109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55042/0 (socket says 172.21.15.70:55042) 2024-06-04T05:14:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.647+0000 7fdfe3fff700 1 -- 172.21.15.70:0/179609109 learned_addr learned my addr 172.21.15.70:0/179609109 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.647+0000 7fdfe3fff700 1 -- 172.21.15.70:0/179609109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfe4108520 msgr2=0x7fdfe4109e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.647+0000 7fdfe3fff700 1 --2- 172.21.15.70:0/179609109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfe4108520 0x7fdfe4109e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.647+0000 7fdfe3fff700 1 -- 172.21.15.70:0/179609109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfe4102e00 msgr2=0x7fdfe41098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.647+0000 7fdfe3fff700 1 --2- 172.21.15.70:0/179609109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfe4102e00 0x7fdfe41098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.648+0000 7fdfe3fff700 1 -- 172.21.15.70:0/179609109 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe41a8100 con 0x7fdfe4102440 2024-06-04T05:14:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.648+0000 7fdfe3fff700 1 --2- 172.21.15.70:0/179609109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4102440 0x7fdfe4109390 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4006010 tx=0x7fdfd40047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.648+0000 7fdfe17fa700 1 -- 172.21.15.70:0/179609109 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4027070 con 0x7fdfe4102440 2024-06-04T05:14:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.648+0000 7fdfea54c700 1 -- 172.21.15.70:0/179609109 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfe41a8390 con 0x7fdfe4102440 2024-06-04T05:14:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.648+0000 7fdfe17fa700 1 -- 172.21.15.70:0/179609109 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdfd4022470 con 0x7fdfe4102440 2024-06-04T05:14:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.648+0000 7fdfe17fa700 1 -- 172.21.15.70:0/179609109 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4017650 con 0x7fdfe4102440 2024-06-04T05:14:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.648+0000 7fdfea54c700 1 -- 172.21.15.70:0/179609109 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfe41a8840 con 0x7fdfe4102440 2024-06-04T05:14:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.651+0000 7fdfe17fa700 1 -- 172.21.15.70:0/179609109 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fdfd4009d10 con 0x7fdfe4102440 2024-06-04T05:14:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.652+0000 7fdfe17fa700 1 --2- 172.21.15.70:0/179609109 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdfcc0b2a70 0x7fdfcc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.652+0000 7fdfe17fa700 1 -- 172.21.15.70:0/179609109 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fdfd40e30a0 con 0x7fdfe4102440 2024-06-04T05:14:35.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.652+0000 7fdfe37fe700 1 --2- 172.21.15.70:0/179609109 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdfcc0b2a70 0x7fdfcc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.653+0000 7fdff4de1b80 1 Processor -- start 2024-06-04T05:14:35.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.653+0000 7fdff4de1b80 1 -- start start 2024-06-04T05:14:35.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.653+0000 7fdfe37fe700 1 --2- 172.21.15.70:0/179609109 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdfcc0b2a70 0x7fdfcc0b4f30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40ff860 tx=0x7fdfd8009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:35.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.653+0000 7fdff4de1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 0x56530e0eae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.653+0000 7fdff4de1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0eb420 0x56530e0e50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.653+0000 7fdff4de1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0e56d0 0x56530e0e5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.653+0000 7fdff4de1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56530e03ad70 con 0x56530e0eb420 2024-06-04T05:14:35.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.653+0000 7fdff4de1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56530e02a910 con 0x56530e0eaa70 2024-06-04T05:14:35.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.653+0000 7fdff4de1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56530e02e950 con 0x56530e0e56d0 2024-06-04T05:14:35.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.653+0000 7fdfe37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0eb420 0x56530e0e50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.653+0000 7fdfe37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0eb420 0x56530e0e50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52508/0 (socket says 172.21.15.70:52508) 2024-06-04T05:14:35.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.653+0000 7fdfe37fe700 1 -- 172.21.15.70:0/1995772149 learned_addr learned my addr 172.21.15.70:0/1995772149 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:35.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.653+0000 7fdfe3fff700 1 --2- 172.21.15.70:0/1995772149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 0x56530e0eae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.653+0000 7fdfe8ae9700 1 --2- 172.21.15.70:0/1995772149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0e56d0 0x56530e0e5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.654+0000 7fdfe3fff700 1 -- 172.21.15.70:0/1995772149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0e56d0 msgr2=0x56530e0e5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:35.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.654+0000 7fdfe3fff700 1 --2- 172.21.15.70:0/1995772149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0e56d0 0x56530e0e5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.654+0000 7fdfe3fff700 1 -- 172.21.15.70:0/1995772149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0eb420 msgr2=0x56530e0e50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:35.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.654+0000 7fdfe3fff700 1 --2- 172.21.15.70:0/1995772149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0eb420 0x56530e0e50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.654+0000 7fdfe3fff700 1 -- 172.21.15.70:0/1995772149 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56530dfd3fb0 con 0x56530e0eaa70 2024-06-04T05:14:35.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.654+0000 7fdfe37fe700 1 --2- 172.21.15.70:0/1995772149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0eb420 0x56530e0e50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:35.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.654+0000 7fdfe3fff700 1 --2- 172.21.15.70:0/1995772149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 0x56530e0eae50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdfd40e0760 tx=0x7fdfd40a9bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb6eba8ab77a8f2a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:35.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.654+0000 7fdfca7fc700 1 -- 172.21.15.70:0/1995772149 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd40aaa80 con 0x56530e0eaa70 2024-06-04T05:14:35.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.654+0000 7fdfca7fc700 1 -- 172.21.15.70:0/1995772149 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdfd40aa380 con 0x56530e0eaa70 2024-06-04T05:14:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.654+0000 7fdfca7fc700 1 -- 172.21.15.70:0/1995772149 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd40aa840 con 0x56530e0eaa70 2024-06-04T05:14:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.655+0000 7fdff4de1b80 1 -- 172.21.15.70:0/1995772149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 msgr2=0x56530e0eae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.655+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/1995772149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 0x56530e0eae50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdfd40e0760 tx=0x7fdfd40a9bc0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.655+0000 7fdff4de1b80 1 -- 172.21.15.70:0/1995772149 shutdown_connections 2024-06-04T05:14:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.655+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/1995772149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0e56d0 0x56530e0e5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.655+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/1995772149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 0x56530e0eae50 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.655+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/1995772149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0eb420 0x56530e0e50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.655+0000 7fdff4de1b80 1 -- 172.21.15.70:0/1995772149 >> 172.21.15.70:0/1995772149 conn(0x56530e008240 msgr2=0x56530e0d73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.655+0000 7fdff4de1b80 1 -- 172.21.15.70:0/1995772149 shutdown_connections 2024-06-04T05:14:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.656+0000 7fdff4de1b80 1 -- 172.21.15.70:0/1995772149 wait complete. 2024-06-04T05:14:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.656+0000 7fdff4de1b80 1 -- start start 2024-06-04T05:14:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.656+0000 7fdff4de1b80 1 Processor -- start 2024-06-04T05:14:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.657+0000 7fdff4de1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0e56d0 0x56530e0f11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.657+0000 7fdff4de1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 0x56530e0ee4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.657+0000 7fdff4de1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0eb420 0x56530e0eea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.657+0000 7fdff4de1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56530df30fa0 con 0x56530e0eb420 2024-06-04T05:14:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.657+0000 7fdfe37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 0x56530e0ee4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.657+0000 7fdfe37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 0x56530e0ee4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55060/0 (socket says 172.21.15.70:55060) 2024-06-04T05:14:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.657+0000 7fdfe37fe700 1 -- 172.21.15.70:0/2494964160 learned_addr learned my addr 172.21.15.70:0/2494964160 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.657+0000 7fdff4de1b80 1 -- 172.21.15.70:0/2494964160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56530df92da0 con 0x56530e0eaa70 2024-06-04T05:14:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.657+0000 7fdff4de1b80 1 -- 172.21.15.70:0/2494964160 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56530e0f1c60 con 0x56530e0e56d0 2024-06-04T05:14:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.658+0000 7fdfe8ae9700 1 --2- 172.21.15.70:0/2494964160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0eb420 0x56530e0eea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.658+0000 7fdfe3fff700 1 --2- 172.21.15.70:0/2494964160 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0e56d0 0x56530e0f11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.658+0000 7fdff4de1b80 10 client.0 _ll_register_callbacks cb 0x56530dfc5950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.658+0000 7fdfe37fe700 1 -- 172.21.15.70:0/2494964160 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0e56d0 msgr2=0x56530e0f11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.658+0000 7fdfe37fe700 1 --2- 172.21.15.70:0/2494964160 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0e56d0 0x56530e0f11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.658+0000 7fdfe37fe700 1 -- 172.21.15.70:0/2494964160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0eb420 msgr2=0x56530e0eea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.658+0000 7fdfe37fe700 1 --2- 172.21.15.70:0/2494964160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0eb420 0x56530e0eea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.658+0000 7fdfe37fe700 1 -- 172.21.15.70:0/2494964160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56530df7a8d0 con 0x56530e0eaa70 2024-06-04T05:14:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.658+0000 7fdfe8ae9700 1 --2- 172.21.15.70:0/2494964160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0eb420 0x56530e0eea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:14:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.658+0000 7fdfe3fff700 1 --2- 172.21.15.70:0/2494964160 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0e56d0 0x56530e0f11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:14:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.658+0000 7fdfe37fe700 1 --2- 172.21.15.70:0/2494964160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 0x56530e0ee4f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8004410 tx=0x7fdfd80047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9725218530e567ac server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.658+0000 7fdfc37fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:14:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.658+0000 7fdfc37fe700 1 -- 172.21.15.70:0/2494964160 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd801e420 con 0x56530e0eaa70 2024-06-04T05:14:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.658+0000 7fdfc37fe700 1 -- 172.21.15.70:0/2494964160 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fdfd801e5c0 con 0x56530e0eaa70 2024-06-04T05:14:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.658+0000 7fdff4de1b80 1 -- 172.21.15.70:0/2494964160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfd40aa380 con 0x56530e0eaa70 2024-06-04T05:14:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.659+0000 7fdfc37fe700 1 -- 172.21.15.70:0/2494964160 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fdfd801e7c0 con 0x56530e0eaa70 2024-06-04T05:14:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.659+0000 7fdff4de1b80 10 client.28763 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.659+0000 7fdff4de1b80 1 -- 172.21.15.70:0/2494964160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x56530dfe4760 con 0x56530e0eaa70 2024-06-04T05:14:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.659+0000 7fdfc37fe700 1 -- 172.21.15.70:0/2494964160 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdfd801edf0 con 0x56530e0eaa70 2024-06-04T05:14:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.660+0000 7fdfc37fe700 1 -- 172.21.15.70:0/2494964160 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7fdfd802e020 con 0x56530e0eaa70 2024-06-04T05:14:35.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.660+0000 7fdff4de1b80 10 client.28763 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:35.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.660+0000 7fdff4de1b80 10 client.28763 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:14:35.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.660+0000 7fdff4de1b80 20 client.28763 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:35.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.660+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/2494964160 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56530e0fe8c0 0x56530e100cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.660+0000 7fdff4de1b80 4 client.28763 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:14:35.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.660+0000 7fdff4de1b80 1 -- 172.21.15.70:0/2494964160 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fdfd40aa380 con 0x56530e0fe8c0 2024-06-04T05:14:35.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.660+0000 7fdfe3fff700 1 --2- 172.21.15.70:0/2494964160 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56530e0fe8c0 0x56530e100cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.661+0000 7fdfe3fff700 1 --2- 172.21.15.70:0/2494964160 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56530e0fe8c0 0x56530e100cb0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4509c6357541f961 server_cookie=956bb5f4cf415ca2 in_seq=0 out_seq=0 2024-06-04T05:14:35.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.661+0000 7fdfc37fe700 10 client.28763 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:14:35.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.661+0000 7fdfc37fe700 1 -- 172.21.15.70:0/2494964160 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fdfd40aa380 con 0x56530e0fe8c0 2024-06-04T05:14:35.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.661+0000 7fdfc37fe700 10 client.28763 handle_command_reply: tid=0 2024-06-04T05:14:35.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.661+0000 7fdff4de1b80 1 client.28763 shutdown 2024-06-04T05:14:35.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.662+0000 7fdff4de1b80 1 -- 172.21.15.70:0/2494964160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 msgr2=0x56530e0ee4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:35.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.662+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/2494964160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 0x56530e0ee4f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8004410 tx=0x7fdfd80047c0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.662+0000 7fdff4de1b80 1 -- 172.21.15.70:0/2494964160 shutdown_connections 2024-06-04T05:14:35.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.662+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/2494964160 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56530e0fe8c0 0x56530e100cb0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.662+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/2494964160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0eb420 0x56530e0eea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.662+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/2494964160 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0e56d0 0x56530e0f11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.662+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/2494964160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 0x56530e0ee4f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.662+0000 7fdfc37fe700 0 client.28763 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:14:35.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.662+0000 7fdff4de1b80 1 -- 172.21.15.70:0/2494964160 >> 172.21.15.70:0/2494964160 conn(0x56530e008240 msgr2=0x56530e0d6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:35.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.662+0000 7fdff4de1b80 1 -- 172.21.15.70:0/2494964160 shutdown_connections 2024-06-04T05:14:35.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.662+0000 7fdff4de1b80 1 -- 172.21.15.70:0/2494964160 wait complete. 2024-06-04T05:14:35.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.663+0000 7fdff4de1b80 20 client.28763 trim_cache size 0 max 16384 2024-06-04T05:14:35.692 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:35 smithi112 ceph-mon[36453]: pgmap v545: 97 pgs: 97 active+clean; 539 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 820 KiB/s wr, 110 op/s 2024-06-04T05:14:35.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.688+0000 7fdff4de1b80 1 Processor -- start 2024-06-04T05:14:35.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.688+0000 7fdff4de1b80 1 -- start start 2024-06-04T05:14:35.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.689+0000 7fdff4de1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0e56d0 0x56530e0ed470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.689+0000 7fdff4de1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 0x56530e0ec090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.689+0000 7fdff4de1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0eb420 0x56530e0eea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.689+0000 7fdff4de1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe41abb70 con 0x56530e0e56d0 2024-06-04T05:14:35.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.689+0000 7fdff4de1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe41ad710 con 0x56530e0eaa70 2024-06-04T05:14:35.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.689+0000 7fdff4de1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe41ac2a0 con 0x56530e0eb420 2024-06-04T05:14:35.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.689+0000 7fdfe3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0e56d0 0x56530e0ed470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.689+0000 7fdfe3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0e56d0 0x56530e0ed470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52522/0 (socket says 172.21.15.70:52522) 2024-06-04T05:14:35.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.689+0000 7fdfe3fff700 1 -- 172.21.15.70:0/866264683 learned_addr learned my addr 172.21.15.70:0/866264683 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:35.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.689+0000 7fdfe37fe700 1 --2- 172.21.15.70:0/866264683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 0x56530e0ec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.689+0000 7fdfe8ae9700 1 --2- 172.21.15.70:0/866264683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0eb420 0x56530e0eea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.689+0000 7fdfe37fe700 1 -- 172.21.15.70:0/866264683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0eb420 msgr2=0x56530e0eea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:35.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.689+0000 7fdfe37fe700 1 --2- 172.21.15.70:0/866264683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0eb420 0x56530e0eea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.689+0000 7fdfe37fe700 1 -- 172.21.15.70:0/866264683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0e56d0 msgr2=0x56530e0ed470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:35.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.689+0000 7fdfe37fe700 1 --2- 172.21.15.70:0/866264683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0e56d0 0x56530e0ed470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.689+0000 7fdfe37fe700 1 -- 172.21.15.70:0/866264683 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56530e0e5300 con 0x56530e0eaa70 2024-06-04T05:14:35.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.689+0000 7fdfe37fe700 1 --2- 172.21.15.70:0/866264683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 0x56530e0ec090 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdfd80094f0 tx=0x7fdfd8004720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9df146c21f2dcd6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:35.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.690+0000 7fdfc9ffb700 1 -- 172.21.15.70:0/866264683 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd801e620 con 0x56530e0eaa70 2024-06-04T05:14:35.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.690+0000 7fdfc9ffb700 1 -- 172.21.15.70:0/866264683 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdfd801edf0 con 0x56530e0eaa70 2024-06-04T05:14:35.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.690+0000 7fdfc9ffb700 1 -- 172.21.15.70:0/866264683 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8030410 con 0x56530e0eaa70 2024-06-04T05:14:35.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.691+0000 7fdff4de1b80 1 -- 172.21.15.70:0/866264683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 msgr2=0x56530e0ec090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:35.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.691+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/866264683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 0x56530e0ec090 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdfd80094f0 tx=0x7fdfd8004720 comp rx=0 tx=0).stop 2024-06-04T05:14:35.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.691+0000 7fdff4de1b80 1 -- 172.21.15.70:0/866264683 shutdown_connections 2024-06-04T05:14:35.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.691+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/866264683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0eb420 0x56530e0eea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.691+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/866264683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0e56d0 0x56530e0ed470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.691+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/866264683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eaa70 0x56530e0ec090 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.691+0000 7fdff4de1b80 1 -- 172.21.15.70:0/866264683 >> 172.21.15.70:0/866264683 conn(0x56530e008240 msgr2=0x56530e0e9650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:35.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.691+0000 7fdff4de1b80 1 -- 172.21.15.70:0/866264683 shutdown_connections 2024-06-04T05:14:35.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.691+0000 7fdff4de1b80 1 -- 172.21.15.70:0/866264683 wait complete. 2024-06-04T05:14:35.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.691+0000 7fdff4de1b80 1 -- start start 2024-06-04T05:14:35.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.691+0000 7fdff4de1b80 1 Processor -- start 2024-06-04T05:14:35.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.692+0000 7fdff4de1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0e56d0 0x56530e0f74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.692+0000 7fdff4de1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0eaa70 0x56530e105d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.692+0000 7fdff4de1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eb420 0x56530e1062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.692+0000 7fdff4de1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe41ad040 con 0x56530e0e56d0 2024-06-04T05:14:35.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.692+0000 7fdff4de1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe41ac970 con 0x56530e0eb420 2024-06-04T05:14:35.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.692+0000 7fdff4de1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe419dbf0 con 0x56530e0eaa70 2024-06-04T05:14:35.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.692+0000 7fdfe3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0e56d0 0x56530e0f74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.692+0000 7fdfe3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0e56d0 0x56530e0f74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52534/0 (socket says 172.21.15.70:52534) 2024-06-04T05:14:35.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.692+0000 7fdfe3fff700 1 -- 172.21.15.70:0/699905329 learned_addr learned my addr 172.21.15.70:0/699905329 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:35.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.692+0000 7fdfe8ae9700 1 --2- 172.21.15.70:0/699905329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eb420 0x56530e1062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.692+0000 7fdff4de1b80 10 client.0 _ll_register_callbacks cb 0x56530dfc5950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.692+0000 7fdfe8ae9700 1 -- 172.21.15.70:0/699905329 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0eaa70 msgr2=0x56530e105d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.692+0000 7fdfe37fe700 1 --2- 172.21.15.70:0/699905329 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0eaa70 0x56530e105d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.692+0000 7fdfe8ae9700 1 --2- 172.21.15.70:0/699905329 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0eaa70 0x56530e105d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.692+0000 7fdfe8ae9700 1 -- 172.21.15.70:0/699905329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0e56d0 msgr2=0x56530e0f74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.692+0000 7fdfe8ae9700 1 --2- 172.21.15.70:0/699905329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0e56d0 0x56530e0f74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.692+0000 7fdfe8ae9700 1 -- 172.21.15.70:0/699905329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56530e106ac0 con 0x56530e0eb420 2024-06-04T05:14:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.693+0000 7fdfe3fff700 1 --2- 172.21.15.70:0/699905329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0e56d0 0x56530e0f74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.693+0000 7fdfe37fe700 1 --2- 172.21.15.70:0/699905329 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0eaa70 0x56530e105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:14:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.693+0000 7fdfe8ae9700 1 --2- 172.21.15.70:0/699905329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eb420 0x56530e1062b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc007ca0 tx=0x7fdfdc018420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caa3f68dfa934f65 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.693+0000 7fdfc37fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:14:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.693+0000 7fdfc37fe700 1 -- 172.21.15.70:0/699905329 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc003a00 con 0x56530e0eb420 2024-06-04T05:14:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.693+0000 7fdff4de1b80 1 -- 172.21.15.70:0/699905329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfd801edf0 con 0x56530e0eb420 2024-06-04T05:14:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.693+0000 7fdfc37fe700 1 -- 172.21.15.70:0/699905329 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fdfdc003ba0 con 0x56530e0eb420 2024-06-04T05:14:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.693+0000 7fdfc37fe700 1 -- 172.21.15.70:0/699905329 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fdfdc003da0 con 0x56530e0eb420 2024-06-04T05:14:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.693+0000 7fdff4de1b80 10 client.28769 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.693+0000 7fdff4de1b80 1 -- 172.21.15.70:0/699905329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x56530e169410 con 0x56530e0eb420 2024-06-04T05:14:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.693+0000 7fdfc37fe700 1 -- 172.21.15.70:0/699905329 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdfd801edf0 con 0x56530e0eb420 2024-06-04T05:14:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.694+0000 7fdfc37fe700 1 -- 172.21.15.70:0/699905329 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7fdfdc027070 con 0x56530e0eb420 2024-06-04T05:14:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.694+0000 7fdff4de1b80 10 client.28769 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.694+0000 7fdff4de1b80 10 client.28769 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:14:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.694+0000 7fdff4de1b80 20 client.28769 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.694+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/699905329 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56530e169b90 0x56530e16bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.694+0000 7fdff4de1b80 4 client.28769 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:14:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.694+0000 7fdff4de1b80 1 -- 172.21.15.70:0/699905329 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56530e03ccc0 con 0x56530e169b90 2024-06-04T05:14:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.694+0000 7fdfe3fff700 1 --2- 172.21.15.70:0/699905329 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56530e169b90 0x56530e16bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.694+0000 7fdfe3fff700 1 --2- 172.21.15.70:0/699905329 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56530e169b90 0x56530e16bfb0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e7d01893febe96b3 server_cookie=3c1ae7e22470afed in_seq=0 out_seq=0 2024-06-04T05:14:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.694+0000 7fdfc37fe700 10 client.28769 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:14:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.695+0000 7fdfc37fe700 1 -- 172.21.15.70:0/699905329 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56530e03ccc0 con 0x56530e169b90 2024-06-04T05:14:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.695+0000 7fdfc37fe700 10 client.28769 handle_command_reply: tid=0 2024-06-04T05:14:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.695+0000 7fdff4de1b80 1 client.28769 shutdown 2024-06-04T05:14:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.695+0000 7fdff4de1b80 1 -- 172.21.15.70:0/699905329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eb420 msgr2=0x56530e1062b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.695+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/699905329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eb420 0x56530e1062b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc007ca0 tx=0x7fdfdc018420 comp rx=0 tx=0).stop 2024-06-04T05:14:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.695+0000 7fdff4de1b80 1 -- 172.21.15.70:0/699905329 shutdown_connections 2024-06-04T05:14:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.695+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/699905329 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x56530e169b90 0x56530e16bfb0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.695+0000 7fdfc37fe700 0 client.28769 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:14:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.695+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/699905329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56530e0eb420 0x56530e1062b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.695+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/699905329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56530e0e56d0 0x56530e0f74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.695+0000 7fdff4de1b80 1 --2- 172.21.15.70:0/699905329 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56530e0eaa70 0x56530e105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.695+0000 7fdff4de1b80 1 -- 172.21.15.70:0/699905329 >> 172.21.15.70:0/699905329 conn(0x56530e008240 msgr2=0x56530e0db890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.695+0000 7fdff4de1b80 1 -- 172.21.15.70:0/699905329 shutdown_connections 2024-06-04T05:14:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.695+0000 7fdff4de1b80 1 -- 172.21.15.70:0/699905329 wait complete. 2024-06-04T05:14:35.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.695+0000 7fdff4de1b80 20 client.28769 trim_cache size 0 max 16384 2024-06-04T05:14:35.703 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:14:35.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.696+0000 7fdfea54c700 1 -- 172.21.15.70:0/179609109 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdfcc0b2a70 msgr2=0x7fdfcc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:35.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.696+0000 7fdfea54c700 1 --2- 172.21.15.70:0/179609109 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdfcc0b2a70 0x7fdfcc0b4f30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40ff860 tx=0x7fdfd8009260 comp rx=0 tx=0).stop 2024-06-04T05:14:35.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.696+0000 7fdfea54c700 1 -- 172.21.15.70:0/179609109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4102440 msgr2=0x7fdfe4109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:35.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.696+0000 7fdfea54c700 1 --2- 172.21.15.70:0/179609109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4102440 0x7fdfe4109390 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4006010 tx=0x7fdfd40047b0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.696+0000 7fdfea54c700 1 -- 172.21.15.70:0/179609109 shutdown_connections 2024-06-04T05:14:35.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.696+0000 7fdfea54c700 1 --2- 172.21.15.70:0/179609109 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdfcc0b2a70 0x7fdfcc0b4f30 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.696+0000 7fdfea54c700 1 --2- 172.21.15.70:0/179609109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfe4108520 0x7fdfe4109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.696+0000 7fdfea54c700 1 --2- 172.21.15.70:0/179609109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4102440 0x7fdfe4109390 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.696+0000 7fdfea54c700 1 --2- 172.21.15.70:0/179609109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfe4102e00 0x7fdfe41098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:35.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.696+0000 7fdfea54c700 1 -- 172.21.15.70:0/179609109 >> 172.21.15.70:0/179609109 conn(0x7fdfe40793f0 msgr2=0x7fdfe4100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:35.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.696+0000 7fdfea54c700 1 -- 172.21.15.70:0/179609109 shutdown_connections 2024-06-04T05:14:35.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:35.696+0000 7fdfea54c700 1 -- 172.21.15.70:0/179609109 wait complete. 2024-06-04T05:14:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:35 smithi098 ceph-mon[35569]: pgmap v545: 97 pgs: 97 active+clean; 539 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 820 KiB/s wr, 110 op/s 2024-06-04T05:14:35.805 INFO:tasks.workunit.client.0.smithi070.stdout: 10 29491 7.94 MB/sec execute 87 sec latency 106.431 ms 2024-06-04T05:14:36.436 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:14:36.787 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:14:36.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 29688 8.00 MB/sec execute 88 sec latency 61.770 ms 2024-06-04T05:14:37.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:37 smithi070 ceph-mon[32631]: pgmap v546: 97 pgs: 97 active+clean; 547 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 2.3 MiB/s wr, 202 op/s 2024-06-04T05:14:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:37 smithi112 ceph-mon[36453]: pgmap v546: 97 pgs: 97 active+clean; 547 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 2.3 MiB/s wr, 202 op/s 2024-06-04T05:14:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:37 smithi098 ceph-mon[35569]: pgmap v546: 97 pgs: 97 active+clean; 547 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 2.3 MiB/s wr, 202 op/s 2024-06-04T05:14:37.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 29859 8.04 MB/sec execute 89 sec latency 55.219 ms 2024-06-04T05:14:38.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.652+0000 7fae98f68700 1 -- 172.21.15.70:0/655854559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae94102470 msgr2=0x7fae941028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:38.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.652+0000 7fae98f68700 1 --2- 172.21.15.70:0/655854559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae94102470 0x7fae941028f0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fae880099c0 tx=0x7fae88009cd0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.652+0000 7fae98f68700 1 -- 172.21.15.70:0/655854559 shutdown_connections 2024-06-04T05:14:38.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.652+0000 7fae98f68700 1 --2- 172.21.15.70:0/655854559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae94102e30 0x7fae9410f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.652+0000 7fae98f68700 1 --2- 172.21.15.70:0/655854559 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fae94108550 0x7fae94108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.652+0000 7fae98f68700 1 --2- 172.21.15.70:0/655854559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae94102470 0x7fae941028f0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.652+0000 7fae98f68700 1 -- 172.21.15.70:0/655854559 >> 172.21.15.70:0/655854559 conn(0x7fae94079480 msgr2=0x7fae94079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:38.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.652+0000 7fae98f68700 1 -- 172.21.15.70:0/655854559 shutdown_connections 2024-06-04T05:14:38.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.653+0000 7fae98f68700 1 -- 172.21.15.70:0/655854559 wait complete. 2024-06-04T05:14:38.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.653+0000 7fae98f68700 1 Processor -- start 2024-06-04T05:14:38.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.653+0000 7fae98f68700 1 -- start start 2024-06-04T05:14:38.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.654+0000 7fae98f68700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fae94102470 0x7fae9419bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.654+0000 7fae98f68700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae94102e30 0x7fae9419c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.654+0000 7fae98f68700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae94108550 0x7fae941a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.654+0000 7fae98f68700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fae941157f0 con 0x7fae94102e30 2024-06-04T05:14:38.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.654+0000 7fae98f68700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fae94115670 con 0x7fae94108550 2024-06-04T05:14:38.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.654+0000 7fae98f68700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fae94115970 con 0x7fae94102470 2024-06-04T05:14:38.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.654+0000 7fae9259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fae94102470 0x7fae9419bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.654+0000 7fae9259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fae94102470 0x7fae9419bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:49440/0 (socket says 172.21.15.70:49440) 2024-06-04T05:14:38.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.654+0000 7fae9259c700 1 -- 172.21.15.70:0/3935574094 learned_addr learned my addr 172.21.15.70:0/3935574094 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:38.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.654+0000 7fae92d9d700 1 --2- 172.21.15.70:0/3935574094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae94108550 0x7fae941a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.654+0000 7fae91d9b700 1 --2- 172.21.15.70:0/3935574094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae94102e30 0x7fae9419c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.654+0000 7fae9259c700 1 -- 172.21.15.70:0/3935574094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae94108550 msgr2=0x7fae941a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:38.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.654+0000 7fae9259c700 1 --2- 172.21.15.70:0/3935574094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae94108550 0x7fae941a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.654+0000 7fae9259c700 1 -- 172.21.15.70:0/3935574094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae94102e30 msgr2=0x7fae9419c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:38.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.654+0000 7fae9259c700 1 --2- 172.21.15.70:0/3935574094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae94102e30 0x7fae9419c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.654+0000 7fae9259c700 1 -- 172.21.15.70:0/3935574094 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae941a0d90 con 0x7fae94102470 2024-06-04T05:14:38.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.654+0000 7fae9259c700 1 --2- 172.21.15.70:0/3935574094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fae94102470 0x7fae9419bd80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fae8400e9a0 tx=0x7fae8400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:38.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.655+0000 7fae837fe700 1 -- 172.21.15.70:0/3935574094 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8400cbd0 con 0x7fae94102470 2024-06-04T05:14:38.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.655+0000 7fae837fe700 1 -- 172.21.15.70:0/3935574094 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fae8400cd70 con 0x7fae94102470 2024-06-04T05:14:38.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.655+0000 7fae837fe700 1 -- 172.21.15.70:0/3935574094 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae84018680 con 0x7fae94102470 2024-06-04T05:14:38.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.655+0000 7fae98f68700 1 -- 172.21.15.70:0/3935574094 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae941a1080 con 0x7fae94102470 2024-06-04T05:14:38.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.655+0000 7fae98f68700 1 -- 172.21.15.70:0/3935574094 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae941a1480 con 0x7fae94102470 2024-06-04T05:14:38.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.657+0000 7fae837fe700 1 -- 172.21.15.70:0/3935574094 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fae84010430 con 0x7fae94102470 2024-06-04T05:14:38.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.658+0000 7fae837fe700 1 --2- 172.21.15.70:0/3935574094 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fae7c0b29f0 0x7fae7c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.659+0000 7fae837fe700 1 -- 172.21.15.70:0/3935574094 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fae840e2840 con 0x7fae94102470 2024-06-04T05:14:38.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.659+0000 7fae91d9b700 1 --2- 172.21.15.70:0/3935574094 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fae7c0b29f0 0x7fae7c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.659+0000 7fae91d9b700 1 --2- 172.21.15.70:0/3935574094 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fae7c0b29f0 0x7fae7c0b4eb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fae8800d790 tx=0x7fae88005fd0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:38.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.660+0000 7faea37fdb80 1 Processor -- start 2024-06-04T05:14:38.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.660+0000 7faea37fdb80 1 -- start start 2024-06-04T05:14:38.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.660+0000 7faea37fdb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 0x561d9b5ea700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.660+0000 7faea37fdb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 0x561d9b5e4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.660+0000 7faea37fdb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e5260 0x561d9b5f1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.660+0000 7fae9259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 0x561d9b5ea700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.661+0000 7fae9259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 0x561d9b5ea700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:49444/0 (socket says 172.21.15.70:49444) 2024-06-04T05:14:38.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.661+0000 7fae9259c700 1 -- 172.21.15.70:0/2253699918 learned_addr learned my addr 172.21.15.70:0/2253699918 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:38.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.661+0000 7faea37fdb80 1 -- 172.21.15.70:0/2253699918 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561d9b542d70 con 0x561d9b5e5260 2024-06-04T05:14:38.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.661+0000 7faea37fdb80 1 -- 172.21.15.70:0/2253699918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561d9b532910 con 0x561d9b5e48a0 2024-06-04T05:14:38.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.661+0000 7faea37fdb80 1 -- 172.21.15.70:0/2253699918 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561d9b536950 con 0x561d9b5ea320 2024-06-04T05:14:38.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.661+0000 7fae91d9b700 1 --2- 172.21.15.70:0/2253699918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 0x561d9b5e4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.661+0000 7fae92d9d700 1 --2- 172.21.15.70:0/2253699918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e5260 0x561d9b5f1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.661+0000 7fae91d9b700 1 -- 172.21.15.70:0/2253699918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 msgr2=0x561d9b5ea700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:38.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.661+0000 7fae91d9b700 1 --2- 172.21.15.70:0/2253699918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 0x561d9b5ea700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.661+0000 7fae91d9b700 1 -- 172.21.15.70:0/2253699918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e5260 msgr2=0x561d9b5f1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:38.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.661+0000 7fae91d9b700 1 --2- 172.21.15.70:0/2253699918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e5260 0x561d9b5f1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.661+0000 7fae91d9b700 1 -- 172.21.15.70:0/2253699918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d9b4dbfb0 con 0x561d9b5e48a0 2024-06-04T05:14:38.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.661+0000 7fae9259c700 1 --2- 172.21.15.70:0/2253699918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 0x561d9b5ea700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:38.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.661+0000 7fae91d9b700 1 --2- 172.21.15.70:0/2253699918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 0x561d9b5e4d20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fae88018040 tx=0x7fae8801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b35aa6ddbb9a9b56 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:38.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.661+0000 7fae80ff9700 1 -- 172.21.15.70:0/2253699918 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8801a040 con 0x561d9b5e48a0 2024-06-04T05:14:38.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.662+0000 7fae80ff9700 1 -- 172.21.15.70:0/2253699918 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fae880054b0 con 0x561d9b5e48a0 2024-06-04T05:14:38.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.662+0000 7fae80ff9700 1 -- 172.21.15.70:0/2253699918 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae88015450 con 0x561d9b5e48a0 2024-06-04T05:14:38.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.663+0000 7faea37fdb80 1 -- 172.21.15.70:0/2253699918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 msgr2=0x561d9b5e4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:38.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.663+0000 7faea37fdb80 1 --2- 172.21.15.70:0/2253699918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 0x561d9b5e4d20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fae88018040 tx=0x7fae8801b000 comp rx=0 tx=0).stop 2024-06-04T05:14:38.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.663+0000 7faea37fdb80 1 -- 172.21.15.70:0/2253699918 shutdown_connections 2024-06-04T05:14:38.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.663+0000 7faea37fdb80 1 --2- 172.21.15.70:0/2253699918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e5260 0x561d9b5f1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.663+0000 7faea37fdb80 1 --2- 172.21.15.70:0/2253699918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 0x561d9b5ea700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.663+0000 7faea37fdb80 1 --2- 172.21.15.70:0/2253699918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 0x561d9b5e4d20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.663+0000 7faea37fdb80 1 -- 172.21.15.70:0/2253699918 >> 172.21.15.70:0/2253699918 conn(0x561d9b510240 msgr2=0x561d9b5df3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:38.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.663+0000 7faea37fdb80 1 -- 172.21.15.70:0/2253699918 shutdown_connections 2024-06-04T05:14:38.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.663+0000 7faea37fdb80 1 -- 172.21.15.70:0/2253699918 wait complete. 2024-06-04T05:14:38.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.663+0000 7faea37fdb80 1 -- start start 2024-06-04T05:14:38.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.663+0000 7faea37fdb80 1 Processor -- start 2024-06-04T05:14:38.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7faea37fdb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e48a0 0x561d9b5f0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7faea37fdb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e5260 0x561d9b5edfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7faea37fdb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 0x561d9b5ee510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7faea37fdb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561d9b438fa0 con 0x561d9b5e48a0 2024-06-04T05:14:38.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7faea37fdb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561d9b49ada0 con 0x561d9b5e5260 2024-06-04T05:14:38.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7faea37fdb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561d9b5f15c0 con 0x561d9b5ea320 2024-06-04T05:14:38.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7fae9259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e48a0 0x561d9b5f0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7faea37fdb80 10 client.0 _ll_register_callbacks cb 0x561d9b4cd950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:38.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7fae9259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e48a0 0x561d9b5f0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50070/0 (socket says 172.21.15.70:50070) 2024-06-04T05:14:38.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7fae9259c700 1 -- 172.21.15.70:0/2871679846 learned_addr learned my addr 172.21.15.70:0/2871679846 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:38.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7fae91d9b700 1 --2- 172.21.15.70:0/2871679846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e5260 0x561d9b5edfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7fae92d9d700 1 --2- 172.21.15.70:0/2871679846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 0x561d9b5ee510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7fae9259c700 1 -- 172.21.15.70:0/2871679846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 msgr2=0x561d9b5ee510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:38.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7fae9259c700 1 --2- 172.21.15.70:0/2871679846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 0x561d9b5ee510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7fae9259c700 1 -- 172.21.15.70:0/2871679846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e5260 msgr2=0x561d9b5edfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:38.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7fae9259c700 1 --2- 172.21.15.70:0/2871679846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e5260 0x561d9b5edfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7fae9259c700 1 -- 172.21.15.70:0/2871679846 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d9b4828d0 con 0x561d9b5e48a0 2024-06-04T05:14:38.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.664+0000 7fae92d9d700 1 --2- 172.21.15.70:0/2871679846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 0x561d9b5ee510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:14:38.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.665+0000 7fae9259c700 1 --2- 172.21.15.70:0/2871679846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e48a0 0x561d9b5f0cd0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fae84009860 tx=0x7fae840b27a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6aa774e088a7f7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:38.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.665+0000 7fae79ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:14:38.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.665+0000 7fae79ffb700 1 -- 172.21.15.70:0/2871679846 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae840a9fd0 con 0x561d9b5e48a0 2024-06-04T05:14:38.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.665+0000 7fae79ffb700 1 -- 172.21.15.70:0/2871679846 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fae840a8f60 con 0x561d9b5e48a0 2024-06-04T05:14:38.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.665+0000 7faea37fdb80 1 -- 172.21.15.70:0/2871679846 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae880054b0 con 0x561d9b5e48a0 2024-06-04T05:14:38.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.665+0000 7fae79ffb700 1 -- 172.21.15.70:0/2871679846 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fae840b2290 con 0x561d9b5e48a0 2024-06-04T05:14:38.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.665+0000 7faea37fdb80 10 client.19950 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:38.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.665+0000 7faea37fdb80 1 -- 172.21.15.70:0/2871679846 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x561d9b4ec760 con 0x561d9b5e48a0 2024-06-04T05:14:38.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.665+0000 7fae79ffb700 1 -- 172.21.15.70:0/2871679846 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fae880054b0 con 0x561d9b5e48a0 2024-06-04T05:14:38.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.666+0000 7fae79ffb700 1 -- 172.21.15.70:0/2871679846 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7fae8406f2c0 con 0x561d9b5e48a0 2024-06-04T05:14:38.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.666+0000 7faea37fdb80 10 client.19950 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:38.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.666+0000 7faea37fdb80 10 client.19950 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:14:38.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.666+0000 7faea37fdb80 20 client.19950 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:38.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.666+0000 7faea37fdb80 1 --2- 172.21.15.70:0/2871679846 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x561d9b606750 0x561d9b608b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.666+0000 7faea37fdb80 4 client.19950 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:14:38.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.666+0000 7faea37fdb80 1 -- 172.21.15.70:0/2871679846 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d9b544cc0 con 0x561d9b606750 2024-06-04T05:14:38.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.666+0000 7fae91d9b700 1 --2- 172.21.15.70:0/2871679846 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x561d9b606750 0x561d9b608b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.666+0000 7fae91d9b700 1 --2- 172.21.15.70:0/2871679846 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x561d9b606750 0x561d9b608b40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bb4bd348710bf9c4 server_cookie=178ae31ce05dd72f in_seq=0 out_seq=0 2024-06-04T05:14:38.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.666+0000 7fae79ffb700 10 client.19950 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:14:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.667+0000 7fae79ffb700 1 -- 172.21.15.70:0/2871679846 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561d9b544cc0 con 0x561d9b606750 2024-06-04T05:14:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.667+0000 7fae79ffb700 10 client.19950 handle_command_reply: tid=0 2024-06-04T05:14:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.668+0000 7faea37fdb80 1 client.19950 shutdown 2024-06-04T05:14:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.668+0000 7faea37fdb80 1 -- 172.21.15.70:0/2871679846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e48a0 msgr2=0x561d9b5f0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.668+0000 7faea37fdb80 1 --2- 172.21.15.70:0/2871679846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e48a0 0x561d9b5f0cd0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fae84009860 tx=0x7fae840b27a0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.668+0000 7faea37fdb80 1 -- 172.21.15.70:0/2871679846 shutdown_connections 2024-06-04T05:14:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.668+0000 7faea37fdb80 1 --2- 172.21.15.70:0/2871679846 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x561d9b606750 0x561d9b608b40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.668+0000 7faea37fdb80 1 --2- 172.21.15.70:0/2871679846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 0x561d9b5ee510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.668+0000 7faea37fdb80 1 --2- 172.21.15.70:0/2871679846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e48a0 0x561d9b5f0cd0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.668+0000 7faea37fdb80 1 --2- 172.21.15.70:0/2871679846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e5260 0x561d9b5edfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.668+0000 7faea37fdb80 1 -- 172.21.15.70:0/2871679846 >> 172.21.15.70:0/2871679846 conn(0x561d9b510240 msgr2=0x561d9b5deaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.668+0000 7fae79ffb700 0 client.19950 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:14:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.668+0000 7faea37fdb80 1 -- 172.21.15.70:0/2871679846 shutdown_connections 2024-06-04T05:14:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.668+0000 7faea37fdb80 1 -- 172.21.15.70:0/2871679846 wait complete. 2024-06-04T05:14:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.668+0000 7faea37fdb80 20 client.19950 trim_cache size 0 max 16384 2024-06-04T05:14:38.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.694+0000 7faea37fdb80 1 Processor -- start 2024-06-04T05:14:38.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.694+0000 7faea37fdb80 1 -- start start 2024-06-04T05:14:38.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.694+0000 7faea37fdb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 0x561d9b5e4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.694+0000 7faea37fdb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e5260 0x561d9b5df270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.694+0000 7faea37fdb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 0x561d9b5eb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.694+0000 7faea37fdb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fae941abd00 con 0x561d9b5e5260 2024-06-04T05:14:38.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.694+0000 7faea37fdb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fae941ad8a0 con 0x561d9b5e48a0 2024-06-04T05:14:38.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.694+0000 7faea37fdb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fae941ac430 con 0x561d9b5ea320 2024-06-04T05:14:38.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.694+0000 7fae91d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e5260 0x561d9b5df270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.694+0000 7fae91d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e5260 0x561d9b5df270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50076/0 (socket says 172.21.15.70:50076) 2024-06-04T05:14:38.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.694+0000 7fae91d9b700 1 -- 172.21.15.70:0/4270434767 learned_addr learned my addr 172.21.15.70:0/4270434767 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:38.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.694+0000 7fae9259c700 1 --2- 172.21.15.70:0/4270434767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 0x561d9b5e4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.694+0000 7fae92d9d700 1 --2- 172.21.15.70:0/4270434767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 0x561d9b5eb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.695+0000 7fae91d9b700 1 -- 172.21.15.70:0/4270434767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 msgr2=0x561d9b5eb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:38.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.695+0000 7fae91d9b700 1 --2- 172.21.15.70:0/4270434767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 0x561d9b5eb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.695+0000 7fae91d9b700 1 -- 172.21.15.70:0/4270434767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 msgr2=0x561d9b5e4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:38.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.695+0000 7fae91d9b700 1 --2- 172.21.15.70:0/4270434767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 0x561d9b5e4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.695+0000 7fae91d9b700 1 -- 172.21.15.70:0/4270434767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d9b5e9f40 con 0x561d9b5e5260 2024-06-04T05:14:38.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.695+0000 7fae9259c700 1 --2- 172.21.15.70:0/4270434767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 0x561d9b5e4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:38.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.695+0000 7fae92d9d700 1 --2- 172.21.15.70:0/4270434767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 0x561d9b5eb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:38.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.695+0000 7fae91d9b700 1 --2- 172.21.15.70:0/4270434767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e5260 0x561d9b5df270 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fae880099c0 tx=0x7fae88000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=353deca513ff909b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:38.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.695+0000 7fae7bfff700 1 -- 172.21.15.70:0/4270434767 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8801a040 con 0x561d9b5e5260 2024-06-04T05:14:38.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.695+0000 7fae7bfff700 1 -- 172.21.15.70:0/4270434767 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fae88038030 con 0x561d9b5e5260 2024-06-04T05:14:38.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.695+0000 7fae7bfff700 1 -- 172.21.15.70:0/4270434767 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae880157b0 con 0x561d9b5e5260 2024-06-04T05:14:38.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.696+0000 7faea37fdb80 1 -- 172.21.15.70:0/4270434767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e5260 msgr2=0x561d9b5df270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:38.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.696+0000 7faea37fdb80 1 --2- 172.21.15.70:0/4270434767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e5260 0x561d9b5df270 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fae880099c0 tx=0x7fae88000bc0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.697+0000 7faea37fdb80 1 -- 172.21.15.70:0/4270434767 shutdown_connections 2024-06-04T05:14:38.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.697+0000 7faea37fdb80 1 --2- 172.21.15.70:0/4270434767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5ea320 0x561d9b5eb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.697+0000 7faea37fdb80 1 --2- 172.21.15.70:0/4270434767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 0x561d9b5e4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.697+0000 7faea37fdb80 1 --2- 172.21.15.70:0/4270434767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5e5260 0x561d9b5df270 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.697+0000 7faea37fdb80 1 -- 172.21.15.70:0/4270434767 >> 172.21.15.70:0/4270434767 conn(0x561d9b510240 msgr2=0x561d9b5e9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:38.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.697+0000 7faea37fdb80 1 -- 172.21.15.70:0/4270434767 shutdown_connections 2024-06-04T05:14:38.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.697+0000 7faea37fdb80 1 -- 172.21.15.70:0/4270434767 wait complete. 2024-06-04T05:14:38.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.698+0000 7faea37fdb80 1 -- start start 2024-06-04T05:14:38.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.698+0000 7faea37fdb80 1 Processor -- start 2024-06-04T05:14:38.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.699+0000 7faea37fdb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 0x561d9b5ff6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.699+0000 7faea37fdb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5e5260 0x561d9b60dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.699+0000 7faea37fdb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5ea320 0x561d9b60e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.699+0000 7faea37fdb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fae941ad1d0 con 0x561d9b5ea320 2024-06-04T05:14:38.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.699+0000 7fae9259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 0x561d9b5ff6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.699+0000 7faea37fdb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fae941acb00 con 0x561d9b5e48a0 2024-06-04T05:14:38.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.699+0000 7faea37fdb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fae941a9cc0 con 0x561d9b5e5260 2024-06-04T05:14:38.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.699+0000 7fae9259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 0x561d9b5ff6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:44278/0 (socket says 172.21.15.70:44278) 2024-06-04T05:14:38.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.699+0000 7fae9259c700 1 -- 172.21.15.70:0/4170888902 learned_addr learned my addr 172.21.15.70:0/4170888902 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:38.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.699+0000 7fae91d9b700 1 --2- 172.21.15.70:0/4170888902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5e5260 0x561d9b60dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.699+0000 7faea37fdb80 10 client.0 _ll_register_callbacks cb 0x561d9b4cd950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:38.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.699+0000 7fae92d9d700 1 --2- 172.21.15.70:0/4170888902 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5ea320 0x561d9b60e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.700+0000 7fae91d9b700 1 -- 172.21.15.70:0/4170888902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 msgr2=0x561d9b5ff6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:38.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.700+0000 7fae91d9b700 1 --2- 172.21.15.70:0/4170888902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 0x561d9b5ff6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.700+0000 7fae91d9b700 1 -- 172.21.15.70:0/4170888902 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5ea320 msgr2=0x561d9b60e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:38.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.700+0000 7fae91d9b700 1 --2- 172.21.15.70:0/4170888902 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5ea320 0x561d9b60e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.700+0000 7fae91d9b700 1 -- 172.21.15.70:0/4170888902 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d9b60e970 con 0x561d9b5e5260 2024-06-04T05:14:38.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.700+0000 7fae92d9d700 1 --2- 172.21.15.70:0/4170888902 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5ea320 0x561d9b60e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:38.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.700+0000 7fae91d9b700 1 --2- 172.21.15.70:0/4170888902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5e5260 0x561d9b60dbf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fae8800eb60 tx=0x7fae88009590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5372a35948bb6091 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:38.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.700+0000 7fae79ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:14:38.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.700+0000 7fae79ffb700 1 -- 172.21.15.70:0/4170888902 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae88044060 con 0x561d9b5e5260 2024-06-04T05:14:38.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.700+0000 7fae79ffb700 1 -- 172.21.15.70:0/4170888902 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fae880045d0 con 0x561d9b5e5260 2024-06-04T05:14:38.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.700+0000 7faea37fdb80 1 -- 172.21.15.70:0/4170888902 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae88038030 con 0x561d9b5e5260 2024-06-04T05:14:38.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.701+0000 7fae79ffb700 1 -- 172.21.15.70:0/4170888902 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fae8801a050 con 0x561d9b5e5260 2024-06-04T05:14:38.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.701+0000 7faea37fdb80 10 client.28888 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:38.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.701+0000 7faea37fdb80 1 -- 172.21.15.70:0/4170888902 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x561d9b671290 con 0x561d9b5e5260 2024-06-04T05:14:38.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.701+0000 7fae79ffb700 1 -- 172.21.15.70:0/4170888902 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fae88004e80 con 0x561d9b5e5260 2024-06-04T05:14:38.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.701+0000 7fae79ffb700 1 -- 172.21.15.70:0/4170888902 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7fae88015c20 con 0x561d9b5e5260 2024-06-04T05:14:38.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.701+0000 7faea37fdb80 10 client.28888 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:38.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.702+0000 7faea37fdb80 10 client.28888 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:14:38.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.702+0000 7faea37fdb80 20 client.28888 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:38.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.702+0000 7faea37fdb80 1 --2- 172.21.15.70:0/4170888902 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x561d9b671560 0x561d9b673950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:38.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.702+0000 7faea37fdb80 4 client.28888 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:14:38.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.702+0000 7faea37fdb80 1 -- 172.21.15.70:0/4170888902 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fae88038030 con 0x561d9b671560 2024-06-04T05:14:38.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.702+0000 7fae9259c700 1 --2- 172.21.15.70:0/4170888902 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x561d9b671560 0x561d9b673950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:38.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.702+0000 7fae9259c700 1 --2- 172.21.15.70:0/4170888902 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x561d9b671560 0x561d9b673950 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d0a55b03ed62c1aa server_cookie=4132c8b0d5619dfe in_seq=0 out_seq=0 2024-06-04T05:14:38.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.702+0000 7fae79ffb700 10 client.28888 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:14:38.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.703+0000 7fae79ffb700 1 -- 172.21.15.70:0/4170888902 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fae88038030 con 0x561d9b671560 2024-06-04T05:14:38.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.703+0000 7fae79ffb700 10 client.28888 handle_command_reply: tid=0 2024-06-04T05:14:38.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.703+0000 7faea37fdb80 1 client.28888 shutdown 2024-06-04T05:14:38.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.703+0000 7faea37fdb80 1 -- 172.21.15.70:0/4170888902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5e5260 msgr2=0x561d9b60dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:38.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.703+0000 7faea37fdb80 1 --2- 172.21.15.70:0/4170888902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5e5260 0x561d9b60dbf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fae8800eb60 tx=0x7fae88009590 comp rx=0 tx=0).stop 2024-06-04T05:14:38.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.703+0000 7faea37fdb80 1 -- 172.21.15.70:0/4170888902 shutdown_connections 2024-06-04T05:14:38.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.703+0000 7faea37fdb80 1 --2- 172.21.15.70:0/4170888902 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d9b5ea320 0x561d9b60e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.703+0000 7faea37fdb80 1 --2- 172.21.15.70:0/4170888902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d9b5e48a0 0x561d9b5ff6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.703+0000 7faea37fdb80 1 --2- 172.21.15.70:0/4170888902 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x561d9b671560 0x561d9b673950 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.703+0000 7fae79ffb700 0 client.28888 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:14:38.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.703+0000 7faea37fdb80 1 --2- 172.21.15.70:0/4170888902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d9b5e5260 0x561d9b60dbf0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.703+0000 7faea37fdb80 1 -- 172.21.15.70:0/4170888902 >> 172.21.15.70:0/4170888902 conn(0x561d9b510240 msgr2=0x561d9b5e4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:38.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.703+0000 7faea37fdb80 1 -- 172.21.15.70:0/4170888902 shutdown_connections 2024-06-04T05:14:38.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.703+0000 7faea37fdb80 1 -- 172.21.15.70:0/4170888902 wait complete. 2024-06-04T05:14:38.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.703+0000 7faea37fdb80 20 client.28888 trim_cache size 0 max 16384 2024-06-04T05:14:38.708 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:14:38.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.704+0000 7fae98f68700 1 -- 172.21.15.70:0/3935574094 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fae7c0b29f0 msgr2=0x7fae7c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:38.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.704+0000 7fae98f68700 1 --2- 172.21.15.70:0/3935574094 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fae7c0b29f0 0x7fae7c0b4eb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fae8800d790 tx=0x7fae88005fd0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.704+0000 7fae98f68700 1 -- 172.21.15.70:0/3935574094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fae94102470 msgr2=0x7fae9419bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:38.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.704+0000 7fae98f68700 1 --2- 172.21.15.70:0/3935574094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fae94102470 0x7fae9419bd80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fae8400e9a0 tx=0x7fae8400ecb0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.704+0000 7fae98f68700 1 -- 172.21.15.70:0/3935574094 shutdown_connections 2024-06-04T05:14:38.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.704+0000 7fae98f68700 1 --2- 172.21.15.70:0/3935574094 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fae7c0b29f0 0x7fae7c0b4eb0 secure :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fae8800d790 tx=0x7fae88005fd0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.704+0000 7fae98f68700 1 --2- 172.21.15.70:0/3935574094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae94108550 0x7fae941a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.704+0000 7fae98f68700 1 --2- 172.21.15.70:0/3935574094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fae94102470 0x7fae9419bd80 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.704+0000 7fae98f68700 1 --2- 172.21.15.70:0/3935574094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae94102e30 0x7fae9419c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:38.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.704+0000 7fae98f68700 1 -- 172.21.15.70:0/3935574094 >> 172.21.15.70:0/3935574094 conn(0x7fae94079480 msgr2=0x7fae94100640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:38.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.705+0000 7fae98f68700 1 -- 172.21.15.70:0/3935574094 shutdown_connections 2024-06-04T05:14:38.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:38.705+0000 7fae98f68700 1 -- 172.21.15.70:0/3935574094 wait complete. 2024-06-04T05:14:38.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 30055 8.11 MB/sec execute 90 sec latency 65.728 ms 2024-06-04T05:14:39.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:39 smithi070 ceph-mon[32631]: pgmap v547: 97 pgs: 97 active+clean; 547 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 1.5 MiB/s wr, 100 op/s 2024-06-04T05:14:39.451 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:14:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:39 smithi112 ceph-mon[36453]: pgmap v547: 97 pgs: 97 active+clean; 547 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 1.5 MiB/s wr, 100 op/s 2024-06-04T05:14:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:39 smithi098 ceph-mon[35569]: pgmap v547: 97 pgs: 97 active+clean; 547 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 1.5 MiB/s wr, 100 op/s 2024-06-04T05:14:39.780 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:14:39.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 30243 8.18 MB/sec execute 91 sec latency 59.842 ms 2024-06-04T05:14:40.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 30420 8.17 MB/sec execute 92 sec latency 58.275 ms 2024-06-04T05:14:41.565 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:41 smithi070 ceph-mon[32631]: pgmap v548: 97 pgs: 97 active+clean; 547 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 1.5 MiB/s wr, 100 op/s 2024-06-04T05:14:41.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.560+0000 7fdc928f0700 1 -- 172.21.15.70:0/1750131363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc8c1035c0 msgr2=0x7fdc8c103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:41.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.560+0000 7fdc928f0700 1 --2- 172.21.15.70:0/1750131363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc8c1035c0 0x7fdc8c103a40 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009990 tx=0x7fdc80009ca0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.560+0000 7fdc928f0700 1 -- 172.21.15.70:0/1750131363 shutdown_connections 2024-06-04T05:14:41.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.560+0000 7fdc928f0700 1 --2- 172.21.15.70:0/1750131363 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc8c103f80 0x7fdc8c110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.560+0000 7fdc928f0700 1 --2- 172.21.15.70:0/1750131363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc8c1095d0 0x7fdc8c1099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.560+0000 7fdc928f0700 1 --2- 172.21.15.70:0/1750131363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc8c1035c0 0x7fdc8c103a40 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.560+0000 7fdc928f0700 1 -- 172.21.15.70:0/1750131363 >> 172.21.15.70:0/1750131363 conn(0x7fdc8c07a610 msgr2=0x7fdc8c07aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:41.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.560+0000 7fdc928f0700 1 -- 172.21.15.70:0/1750131363 shutdown_connections 2024-06-04T05:14:41.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.560+0000 7fdc928f0700 1 -- 172.21.15.70:0/1750131363 wait complete. 2024-06-04T05:14:41.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.561+0000 7fdc928f0700 1 Processor -- start 2024-06-04T05:14:41.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.561+0000 7fdc928f0700 1 -- start start 2024-06-04T05:14:41.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.562+0000 7fdc928f0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc8c1035c0 0x7fdc8c19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.562+0000 7fdc928f0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc8c103f80 0x7fdc8c19fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.562+0000 7fdc928f0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc8c1095d0 0x7fdc8c1996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.562+0000 7fdc928f0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8c1135d0 con 0x7fdc8c1095d0 2024-06-04T05:14:41.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.562+0000 7fdc928f0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8c113450 con 0x7fdc8c103f80 2024-06-04T05:14:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.562+0000 7fdc928f0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8c113750 con 0x7fdc8c1035c0 2024-06-04T05:14:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.563+0000 7fdc90e8d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc8c1095d0 0x7fdc8c1996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.563+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc8c1035c0 0x7fdc8c19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.563+0000 7fdc90e8d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc8c1095d0 0x7fdc8c1996b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50100/0 (socket says 172.21.15.70:50100) 2024-06-04T05:14:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.563+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc8c1035c0 0x7fdc8c19f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:49490/0 (socket says 172.21.15.70:49490) 2024-06-04T05:14:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.563+0000 7fdc90e8d700 1 -- 172.21.15.70:0/2435268796 learned_addr learned my addr 172.21.15.70:0/2435268796 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.563+0000 7fdc8b7fe700 1 --2- 172.21.15.70:0/2435268796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc8c103f80 0x7fdc8c19fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.563+0000 7fdc8bfff700 1 -- 172.21.15.70:0/2435268796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc8c103f80 msgr2=0x7fdc8c19fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.563+0000 7fdc8bfff700 1 --2- 172.21.15.70:0/2435268796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc8c103f80 0x7fdc8c19fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.563+0000 7fdc8bfff700 1 -- 172.21.15.70:0/2435268796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc8c1095d0 msgr2=0x7fdc8c1996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.563+0000 7fdc8bfff700 1 --2- 172.21.15.70:0/2435268796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc8c1095d0 0x7fdc8c1996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.563+0000 7fdc8bfff700 1 -- 172.21.15.70:0/2435268796 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc8c199f90 con 0x7fdc8c1035c0 2024-06-04T05:14:41.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.563+0000 7fdc90e8d700 1 --2- 172.21.15.70:0/2435268796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc8c1095d0 0x7fdc8c1996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:14:41.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.563+0000 7fdc8bfff700 1 --2- 172.21.15.70:0/2435268796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc8c1035c0 0x7fdc8c19f5e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00e9f0 tx=0x7fdc7c00edb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:41.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.564+0000 7fdc897fa700 1 -- 172.21.15.70:0/2435268796 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c00cbd0 con 0x7fdc8c1035c0 2024-06-04T05:14:41.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.564+0000 7fdc897fa700 1 -- 172.21.15.70:0/2435268796 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc7c00cd70 con 0x7fdc8c1035c0 2024-06-04T05:14:41.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.564+0000 7fdc897fa700 1 -- 172.21.15.70:0/2435268796 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c018680 con 0x7fdc8c1035c0 2024-06-04T05:14:41.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.564+0000 7fdc928f0700 1 -- 172.21.15.70:0/2435268796 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc8c19a280 con 0x7fdc8c1035c0 2024-06-04T05:14:41.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.564+0000 7fdc928f0700 1 -- 172.21.15.70:0/2435268796 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc8c1a6560 con 0x7fdc8c1035c0 2024-06-04T05:14:41.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.567+0000 7fdc897fa700 1 -- 172.21.15.70:0/2435268796 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fdc7c010430 con 0x7fdc8c1035c0 2024-06-04T05:14:41.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.568+0000 7fdc897fa700 1 --2- 172.21.15.70:0/2435268796 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdc740b29f0 0x7fdc740b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.568+0000 7fdc897fa700 1 -- 172.21.15.70:0/2435268796 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fdc7c0e28f0 con 0x7fdc8c1035c0 2024-06-04T05:14:41.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.568+0000 7fdc8b7fe700 1 --2- 172.21.15.70:0/2435268796 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdc740b29f0 0x7fdc740b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.569+0000 7fdc8b7fe700 1 --2- 172.21.15.70:0/2435268796 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdc740b29f0 0x7fdc740b4eb0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fdc8000b2e0 tx=0x7fdc8001a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:41.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.569+0000 7fdc9d185b80 1 Processor -- start 2024-06-04T05:14:41.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.570+0000 7fdc9d185b80 1 -- start start 2024-06-04T05:14:41.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.570+0000 7fdc9d185b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed7416c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.570+0000 7fdc9d185b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed7419a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.570+0000 7fdc9d185b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed741a090 0x564ed741a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.570+0000 7fdc9d185b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564ed7375c90 con 0x564ed7417210 2024-06-04T05:14:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.570+0000 7fdc9d185b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564ed7365830 con 0x564ed741a090 2024-06-04T05:14:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.570+0000 7fdc9d185b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564ed7369870 con 0x564ed7416860 2024-06-04T05:14:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.570+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed7416c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.570+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed7416c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:49504/0 (socket says 172.21.15.70:49504) 2024-06-04T05:14:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.570+0000 7fdc8bfff700 1 -- 172.21.15.70:0/649945421 learned_addr learned my addr 172.21.15.70:0/649945421 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.570+0000 7fdc8b7fe700 1 --2- 172.21.15.70:0/649945421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed7419a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.570+0000 7fdc90e8d700 1 --2- 172.21.15.70:0/649945421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed741a090 0x564ed741a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.571+0000 7fdc8bfff700 1 -- 172.21.15.70:0/649945421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed741a090 msgr2=0x564ed741a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.571+0000 7fdc8bfff700 1 --2- 172.21.15.70:0/649945421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed741a090 0x564ed741a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.571+0000 7fdc8bfff700 1 -- 172.21.15.70:0/649945421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 msgr2=0x564ed7419a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.571+0000 7fdc8bfff700 1 --2- 172.21.15.70:0/649945421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed7419a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.571+0000 7fdc8bfff700 1 -- 172.21.15.70:0/649945421 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ed72b58d0 con 0x564ed7416860 2024-06-04T05:14:41.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.571+0000 7fdc90e8d700 1 --2- 172.21.15.70:0/649945421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed741a090 0x564ed741a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:41.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.571+0000 7fdc8bfff700 1 --2- 172.21.15.70:0/649945421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed7416c40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0a9ad0 tx=0x7fdc7c0a9330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e85f46dbecd0d016 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:41.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.571+0000 7fdc727fc700 1 -- 172.21.15.70:0/649945421 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c0aa240 con 0x564ed7416860 2024-06-04T05:14:41.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.571+0000 7fdc727fc700 1 -- 172.21.15.70:0/649945421 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc7c0a9b40 con 0x564ed7416860 2024-06-04T05:14:41.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.572+0000 7fdc727fc700 1 -- 172.21.15.70:0/649945421 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c0b2510 con 0x564ed7416860 2024-06-04T05:14:41.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.572+0000 7fdc9d185b80 1 -- 172.21.15.70:0/649945421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 msgr2=0x564ed7416c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:41.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.572+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/649945421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed7416c40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0a9ad0 tx=0x7fdc7c0a9330 comp rx=0 tx=0).stop 2024-06-04T05:14:41.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.572+0000 7fdc9d185b80 1 -- 172.21.15.70:0/649945421 shutdown_connections 2024-06-04T05:14:41.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.572+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/649945421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed741a090 0x564ed741a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.572+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/649945421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed7416c40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.572+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/649945421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed7419a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.572+0000 7fdc9d185b80 1 -- 172.21.15.70:0/649945421 >> 172.21.15.70:0/649945421 conn(0x564ed7344080 msgr2=0x564ed7410e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:41.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.572+0000 7fdc9d185b80 1 -- 172.21.15.70:0/649945421 shutdown_connections 2024-06-04T05:14:41.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.572+0000 7fdc9d185b80 1 -- 172.21.15.70:0/649945421 wait complete. 2024-06-04T05:14:41.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.573+0000 7fdc9d185b80 1 -- start start 2024-06-04T05:14:41.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.573+0000 7fdc9d185b80 1 Processor -- start 2024-06-04T05:14:41.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.574+0000 7fdc9d185b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed742ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.574+0000 7fdc9d185b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed7427f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.574+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed742ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.574+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed742ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:49510/0 (socket says 172.21.15.70:49510) 2024-06-04T05:14:41.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.574+0000 7fdc8b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed7427f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.574+0000 7fdc8b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed7427f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50122/0 (socket says 172.21.15.70:50122) 2024-06-04T05:14:41.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.574+0000 7fdc9d185b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed741a090 0x564ed7428440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.574+0000 7fdc8b7fe700 1 -- 172.21.15.70:0/952287925 learned_addr learned my addr 172.21.15.70:0/952287925 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:41.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.574+0000 7fdc9d185b80 1 -- 172.21.15.70:0/952287925 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564ed726bfa0 con 0x564ed7417210 2024-06-04T05:14:41.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.574+0000 7fdc9d185b80 1 -- 172.21.15.70:0/952287925 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564ed73077c0 con 0x564ed741a090 2024-06-04T05:14:41.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.574+0000 7fdc9d185b80 1 -- 172.21.15.70:0/952287925 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564ed742ba60 con 0x564ed7416860 2024-06-04T05:14:41.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.574+0000 7fdc9d185b80 10 client.0 _ll_register_callbacks cb 0x564ed7300920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:41.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.574+0000 7fdc90e8d700 1 --2- 172.21.15.70:0/952287925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed741a090 0x564ed7428440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.575+0000 7fdc8bfff700 1 -- 172.21.15.70:0/952287925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed741a090 msgr2=0x564ed7428440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:41.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.575+0000 7fdc8bfff700 1 --2- 172.21.15.70:0/952287925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed741a090 0x564ed7428440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.575+0000 7fdc8bfff700 1 -- 172.21.15.70:0/952287925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 msgr2=0x564ed7427f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:41.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.575+0000 7fdc8bfff700 1 --2- 172.21.15.70:0/952287925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed7427f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.575+0000 7fdc8bfff700 1 -- 172.21.15.70:0/952287925 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ed7304690 con 0x564ed7416860 2024-06-04T05:14:41.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.575+0000 7fdc90e8d700 1 --2- 172.21.15.70:0/952287925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed741a090 0x564ed7428440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:14:41.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.575+0000 7fdc8b7fe700 1 --2- 172.21.15.70:0/952287925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed7427f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:41.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.575+0000 7fdc8bfff700 1 --2- 172.21.15.70:0/952287925 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed742ac00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0aa1a0 tx=0x7fdc7c010950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72f5b04ef43f2200 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:41.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.575+0000 7fdc6b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:14:41.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.575+0000 7fdc6b7fe700 1 -- 172.21.15.70:0/952287925 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c06d070 con 0x564ed7416860 2024-06-04T05:14:41.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.575+0000 7fdc9d185b80 1 -- 172.21.15.70:0/952287925 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc7c0a9b40 con 0x564ed7416860 2024-06-04T05:14:41.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.575+0000 7fdc6b7fe700 1 -- 172.21.15.70:0/952287925 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fdc7c0a8fb0 con 0x564ed7416860 2024-06-04T05:14:41.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.575+0000 7fdc6b7fe700 1 -- 172.21.15.70:0/952287925 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fdc7c0aa240 con 0x564ed7416860 2024-06-04T05:14:41.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.575+0000 7fdc9d185b80 10 client.28900 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:41.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.575+0000 7fdc9d185b80 1 -- 172.21.15.70:0/952287925 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x564ed7435fd0 con 0x564ed7416860 2024-06-04T05:14:41.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.576+0000 7fdc6b7fe700 1 -- 172.21.15.70:0/952287925 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc7c0a9b40 con 0x564ed7416860 2024-06-04T05:14:41.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.576+0000 7fdc6b7fe700 1 -- 172.21.15.70:0/952287925 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7fdc7c087050 con 0x564ed7416860 2024-06-04T05:14:41.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.576+0000 7fdc9d185b80 10 client.28900 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:41.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.576+0000 7fdc9d185b80 10 client.28900 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:14:41.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.576+0000 7fdc9d185b80 20 client.28900 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:41.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.576+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/952287925 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x564ed74385e0 0x564ed743aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.576+0000 7fdc9d185b80 4 client.28900 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:14:41.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.576+0000 7fdc9d185b80 1 -- 172.21.15.70:0/952287925 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ed73e29e0 con 0x564ed74385e0 2024-06-04T05:14:41.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.577+0000 7fdc8b7fe700 1 --2- 172.21.15.70:0/952287925 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x564ed74385e0 0x564ed743aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.577+0000 7fdc8b7fe700 1 --2- 172.21.15.70:0/952287925 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x564ed74385e0 0x564ed743aa40 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2b7aa5f5d4618b02 server_cookie=935db8a90e983e8e in_seq=0 out_seq=0 2024-06-04T05:14:41.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.577+0000 7fdc6b7fe700 10 client.28900 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:14:41.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.578+0000 7fdc6b7fe700 1 -- 172.21.15.70:0/952287925 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564ed73e29e0 con 0x564ed74385e0 2024-06-04T05:14:41.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.578+0000 7fdc6b7fe700 10 client.28900 handle_command_reply: tid=0 2024-06-04T05:14:41.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.578+0000 7fdc9d185b80 1 client.28900 shutdown 2024-06-04T05:14:41.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.578+0000 7fdc9d185b80 1 -- 172.21.15.70:0/952287925 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 msgr2=0x564ed742ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:41.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.578+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/952287925 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed742ac00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0aa1a0 tx=0x7fdc7c010950 comp rx=0 tx=0).stop 2024-06-04T05:14:41.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.578+0000 7fdc9d185b80 1 -- 172.21.15.70:0/952287925 shutdown_connections 2024-06-04T05:14:41.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.578+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/952287925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed741a090 0x564ed7428440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.578+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/952287925 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x564ed74385e0 0x564ed743aa40 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.578+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/952287925 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed742ac00 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.578+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/952287925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed7427f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.578+0000 7fdc6b7fe700 0 client.28900 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:14:41.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.578+0000 7fdc9d185b80 1 -- 172.21.15.70:0/952287925 >> 172.21.15.70:0/952287925 conn(0x564ed7344080 msgr2=0x564ed7410510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:41.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.578+0000 7fdc9d185b80 1 -- 172.21.15.70:0/952287925 shutdown_connections 2024-06-04T05:14:41.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.578+0000 7fdc9d185b80 1 -- 172.21.15.70:0/952287925 wait complete. 2024-06-04T05:14:41.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.578+0000 7fdc9d185b80 20 client.28900 trim_cache size 0 max 16384 2024-06-04T05:14:41.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.602+0000 7fdc9d185b80 1 Processor -- start 2024-06-04T05:14:41.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.602+0000 7fdc9d185b80 1 -- start start 2024-06-04T05:14:41.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.602+0000 7fdc9d185b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed7344080 0x564ed7426e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.602+0000 7fdc9d185b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed7410c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.602+0000 7fdc9d185b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed74111d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.603+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed7344080 0x564ed7426e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.603+0000 7fdc90e8d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed74111d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.603+0000 7fdc90e8d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed74111d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50138/0 (socket says 172.21.15.70:50138) 2024-06-04T05:14:41.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.603+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed7344080 0x564ed7426e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:44322/0 (socket says 172.21.15.70:44322) 2024-06-04T05:14:41.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.603+0000 7fdc90e8d700 1 -- 172.21.15.70:0/1222337924 learned_addr learned my addr 172.21.15.70:0/1222337924 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:41.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.603+0000 7fdc9d185b80 1 -- 172.21.15.70:0/1222337924 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8c1aba30 con 0x564ed7417210 2024-06-04T05:14:41.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.603+0000 7fdc9d185b80 1 -- 172.21.15.70:0/1222337924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8c1ad5d0 con 0x564ed7344080 2024-06-04T05:14:41.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.603+0000 7fdc8b7fe700 1 --2- 172.21.15.70:0/1222337924 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed7410c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.603+0000 7fdc9d185b80 1 -- 172.21.15.70:0/1222337924 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8c1ac160 con 0x564ed7416860 2024-06-04T05:14:41.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.603+0000 7fdc90e8d700 1 -- 172.21.15.70:0/1222337924 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 msgr2=0x564ed7410c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:41.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.603+0000 7fdc90e8d700 1 --2- 172.21.15.70:0/1222337924 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed7410c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.603+0000 7fdc90e8d700 1 -- 172.21.15.70:0/1222337924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed7344080 msgr2=0x564ed7426e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:41.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.603+0000 7fdc90e8d700 1 --2- 172.21.15.70:0/1222337924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed7344080 0x564ed7426e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.603+0000 7fdc90e8d700 1 -- 172.21.15.70:0/1222337924 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ed7411d40 con 0x564ed7417210 2024-06-04T05:14:41.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.603+0000 7fdc8bfff700 1 --2- 172.21.15.70:0/1222337924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed7344080 0x564ed7426e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:41.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.603+0000 7fdc90e8d700 1 --2- 172.21.15.70:0/1222337924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed74111d0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fdc84007550 tx=0x7fdc84007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb577341e7619d1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:41.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.603+0000 7fdc71ffb700 1 -- 172.21.15.70:0/1222337924 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8401b020 con 0x564ed7417210 2024-06-04T05:14:41.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.604+0000 7fdc71ffb700 1 -- 172.21.15.70:0/1222337924 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc84017900 con 0x564ed7417210 2024-06-04T05:14:41.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.604+0000 7fdc71ffb700 1 -- 172.21.15.70:0/1222337924 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc84016950 con 0x564ed7417210 2024-06-04T05:14:41.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.604+0000 7fdc9d185b80 1 -- 172.21.15.70:0/1222337924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 msgr2=0x564ed74111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:41.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.604+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/1222337924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed74111d0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fdc84007550 tx=0x7fdc84007860 comp rx=0 tx=0).stop 2024-06-04T05:14:41.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.604+0000 7fdc9d185b80 1 -- 172.21.15.70:0/1222337924 shutdown_connections 2024-06-04T05:14:41.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.604+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/1222337924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed74111d0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.604+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/1222337924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed7344080 0x564ed7426e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.604+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/1222337924 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed7410c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.604+0000 7fdc9d185b80 1 -- 172.21.15.70:0/1222337924 >> 172.21.15.70:0/1222337924 conn(0x564ed740db00 msgr2=0x564ed7415f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:41.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.605+0000 7fdc9d185b80 1 -- 172.21.15.70:0/1222337924 shutdown_connections 2024-06-04T05:14:41.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.605+0000 7fdc9d185b80 1 -- 172.21.15.70:0/1222337924 wait complete. 2024-06-04T05:14:41.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.605+0000 7fdc9d185b80 1 -- start start 2024-06-04T05:14:41.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.605+0000 7fdc9d185b80 1 Processor -- start 2024-06-04T05:14:41.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc9d185b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed7344080 0x564ed7430d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc9d185b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed7431280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc9d185b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed74244f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc8b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed7431280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc90e8d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed74244f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed7344080 0x564ed7430d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc90e8d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed74244f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50150/0 (socket says 172.21.15.70:50150) 2024-06-04T05:14:41.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed7344080 0x564ed7430d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:44334/0 (socket says 172.21.15.70:44334) 2024-06-04T05:14:41.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc8b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed7431280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:49540/0 (socket says 172.21.15.70:49540) 2024-06-04T05:14:41.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc90e8d700 1 -- 172.21.15.70:0/776521724 learned_addr learned my addr 172.21.15.70:0/776521724 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:41.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc9d185b80 1 -- 172.21.15.70:0/776521724 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8c1acf00 con 0x564ed7417210 2024-06-04T05:14:41.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc9d185b80 1 -- 172.21.15.70:0/776521724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8c1ac830 con 0x564ed7344080 2024-06-04T05:14:41.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc9d185b80 1 -- 172.21.15.70:0/776521724 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8c1a99f0 con 0x564ed7416860 2024-06-04T05:14:41.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc9d185b80 10 client.0 _ll_register_callbacks cb 0x564ed7300920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:41.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc8bfff700 1 -- 172.21.15.70:0/776521724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 msgr2=0x564ed7431280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:41.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc8bfff700 1 --2- 172.21.15.70:0/776521724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed7431280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc8bfff700 1 -- 172.21.15.70:0/776521724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 msgr2=0x564ed74244f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:41.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc8bfff700 1 --2- 172.21.15.70:0/776521724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed74244f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.606+0000 7fdc8bfff700 1 -- 172.21.15.70:0/776521724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ed7424a30 con 0x564ed7344080 2024-06-04T05:14:41.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.607+0000 7fdc8bfff700 1 --2- 172.21.15.70:0/776521724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed7344080 0x564ed7430d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0a9420 tx=0x7fdc7c06f2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be2017256121b18d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:41.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.607+0000 7fdc6b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:14:41.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.607+0000 7fdc6b7fe700 1 -- 172.21.15.70:0/776521724 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c06d070 con 0x564ed7344080 2024-06-04T05:14:41.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.607+0000 7fdc9d185b80 1 -- 172.21.15.70:0/776521724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc84017900 con 0x564ed7344080 2024-06-04T05:14:41.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.607+0000 7fdc6b7fe700 1 -- 172.21.15.70:0/776521724 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fdc7c06f3b0 con 0x564ed7344080 2024-06-04T05:14:41.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.607+0000 7fdc6b7fe700 1 -- 172.21.15.70:0/776521724 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fdc7c0a9b40 con 0x564ed7344080 2024-06-04T05:14:41.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.607+0000 7fdc9d185b80 10 client.28802 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:41.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.607+0000 7fdc9d185b80 1 -- 172.21.15.70:0/776521724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x564ed74a2960 con 0x564ed7344080 2024-06-04T05:14:41.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.607+0000 7fdc6b7fe700 1 -- 172.21.15.70:0/776521724 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc84017900 con 0x564ed7344080 2024-06-04T05:14:41.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.607+0000 7fdc6b7fe700 1 -- 172.21.15.70:0/776521724 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7fdc7c087030 con 0x564ed7344080 2024-06-04T05:14:41.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.607+0000 7fdc9d185b80 10 client.28802 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:41.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.607+0000 7fdc9d185b80 10 client.28802 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:14:41.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.607+0000 7fdc9d185b80 20 client.28802 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:41.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.607+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/776521724 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x564ed741a090 0x564ed74a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:41.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.608+0000 7fdc9d185b80 4 client.28802 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:14:41.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.608+0000 7fdc9d185b80 1 -- 172.21.15.70:0/776521724 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564ed7377be0 con 0x564ed741a090 2024-06-04T05:14:41.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.608+0000 7fdc8b7fe700 1 --2- 172.21.15.70:0/776521724 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x564ed741a090 0x564ed74a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:41.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.608+0000 7fdc8b7fe700 1 --2- 172.21.15.70:0/776521724 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x564ed741a090 0x564ed74a4dd0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3e43d318b4a19365 server_cookie=c52eb1b26470ce33 in_seq=0 out_seq=0 2024-06-04T05:14:41.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.608+0000 7fdc6b7fe700 10 client.28802 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:14:41.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.608+0000 7fdc6b7fe700 1 -- 172.21.15.70:0/776521724 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564ed7377be0 con 0x564ed741a090 2024-06-04T05:14:41.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.608+0000 7fdc6b7fe700 10 client.28802 handle_command_reply: tid=0 2024-06-04T05:14:41.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.608+0000 7fdc9d185b80 1 client.28802 shutdown 2024-06-04T05:14:41.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.609+0000 7fdc9d185b80 1 -- 172.21.15.70:0/776521724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed7344080 msgr2=0x564ed7430d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:41.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.609+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/776521724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed7344080 0x564ed7430d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0a9420 tx=0x7fdc7c06f2e0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.609+0000 7fdc9d185b80 1 -- 172.21.15.70:0/776521724 shutdown_connections 2024-06-04T05:14:41.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.609+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/776521724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ed7417210 0x564ed74244f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.609+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/776521724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ed7344080 0x564ed7430d40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.609+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/776521724 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x564ed741a090 0x564ed74a4dd0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.609+0000 7fdc9d185b80 1 --2- 172.21.15.70:0/776521724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564ed7416860 0x564ed7431280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.609+0000 7fdc6b7fe700 0 client.28802 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:14:41.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.609+0000 7fdc9d185b80 1 -- 172.21.15.70:0/776521724 >> 172.21.15.70:0/776521724 conn(0x564ed740db00 msgr2=0x564ed7429730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:41.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.609+0000 7fdc9d185b80 1 -- 172.21.15.70:0/776521724 shutdown_connections 2024-06-04T05:14:41.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.609+0000 7fdc9d185b80 1 -- 172.21.15.70:0/776521724 wait complete. 2024-06-04T05:14:41.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.609+0000 7fdc9d185b80 20 client.28802 trim_cache size 0 max 16384 2024-06-04T05:14:41.617 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:14:41.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.610+0000 7fdc928f0700 1 -- 172.21.15.70:0/2435268796 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdc740b29f0 msgr2=0x7fdc740b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:41.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.610+0000 7fdc928f0700 1 --2- 172.21.15.70:0/2435268796 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdc740b29f0 0x7fdc740b4eb0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fdc8000b2e0 tx=0x7fdc8001a040 comp rx=0 tx=0).stop 2024-06-04T05:14:41.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.610+0000 7fdc928f0700 1 -- 172.21.15.70:0/2435268796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc8c1035c0 msgr2=0x7fdc8c19f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:41.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.610+0000 7fdc928f0700 1 --2- 172.21.15.70:0/2435268796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc8c1035c0 0x7fdc8c19f5e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00e9f0 tx=0x7fdc7c00edb0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.610+0000 7fdc928f0700 1 -- 172.21.15.70:0/2435268796 shutdown_connections 2024-06-04T05:14:41.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.610+0000 7fdc928f0700 1 --2- 172.21.15.70:0/2435268796 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdc740b29f0 0x7fdc740b4eb0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.610+0000 7fdc928f0700 1 --2- 172.21.15.70:0/2435268796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc8c1095d0 0x7fdc8c1996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.610+0000 7fdc928f0700 1 --2- 172.21.15.70:0/2435268796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdc8c1035c0 0x7fdc8c19f5e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.610+0000 7fdc928f0700 1 --2- 172.21.15.70:0/2435268796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc8c103f80 0x7fdc8c19fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:41.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.610+0000 7fdc928f0700 1 -- 172.21.15.70:0/2435268796 >> 172.21.15.70:0/2435268796 conn(0x7fdc8c07a610 msgr2=0x7fdc8c101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:41.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.610+0000 7fdc928f0700 1 -- 172.21.15.70:0/2435268796 shutdown_connections 2024-06-04T05:14:41.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:41.610+0000 7fdc928f0700 1 -- 172.21.15.70:0/2435268796 wait complete. 2024-06-04T05:14:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:41 smithi112 ceph-mon[36453]: pgmap v548: 97 pgs: 97 active+clean; 547 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 1.5 MiB/s wr, 100 op/s 2024-06-04T05:14:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:41 smithi098 ceph-mon[35569]: pgmap v548: 97 pgs: 97 active+clean; 547 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 1.5 MiB/s wr, 100 op/s 2024-06-04T05:14:41.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 30590 8.17 MB/sec execute 93 sec latency 97.120 ms 2024-06-04T05:14:42.446 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:14:42.796 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:14:42.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 30727 8.13 MB/sec execute 94 sec latency 70.797 ms 2024-06-04T05:14:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:43 smithi070 ceph-mon[32631]: pgmap v549: 97 pgs: 97 active+clean; 560 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 3.9 MiB/s wr, 219 op/s 2024-06-04T05:14:43.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:43 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:14:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:43 smithi112 ceph-mon[36453]: pgmap v549: 97 pgs: 97 active+clean; 560 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 3.9 MiB/s wr, 219 op/s 2024-06-04T05:14:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:43 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:14:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:43 smithi098 ceph-mon[35569]: pgmap v549: 97 pgs: 97 active+clean; 560 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 3.9 MiB/s wr, 219 op/s 2024-06-04T05:14:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:43 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:14:43.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 30889 8.14 MB/sec execute 95 sec latency 71.984 ms 2024-06-04T05:14:44.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 31061 8.15 MB/sec execute 96 sec latency 86.847 ms 2024-06-04T05:14:45.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.012+0000 7fcbf11f2700 1 -- 172.21.15.70:0/3615124103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec102440 msgr2=0x7fcbec1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:45.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.012+0000 7fcbf11f2700 1 --2- 172.21.15.70:0/3615124103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec102440 0x7fcbec1028c0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0009990 tx=0x7fcbe0009ca0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.012+0000 7fcbf11f2700 1 -- 172.21.15.70:0/3615124103 shutdown_connections 2024-06-04T05:14:45.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.012+0000 7fcbf11f2700 1 --2- 172.21.15.70:0/3615124103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbec102e00 0x7fcbec10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.012+0000 7fcbf11f2700 1 --2- 172.21.15.70:0/3615124103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcbec108520 0x7fcbec108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.012+0000 7fcbf11f2700 1 --2- 172.21.15.70:0/3615124103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec102440 0x7fcbec1028c0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.012+0000 7fcbf11f2700 1 -- 172.21.15.70:0/3615124103 >> 172.21.15.70:0/3615124103 conn(0x7fcbec0793f0 msgr2=0x7fcbec079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:45.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.012+0000 7fcbf11f2700 1 -- 172.21.15.70:0/3615124103 shutdown_connections 2024-06-04T05:14:45.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.012+0000 7fcbf11f2700 1 -- 172.21.15.70:0/3615124103 wait complete. 2024-06-04T05:14:45.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.013+0000 7fcbf11f2700 1 Processor -- start 2024-06-04T05:14:45.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.013+0000 7fcbf11f2700 1 -- start start 2024-06-04T05:14:45.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.014+0000 7fcbf11f2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbec102440 0x7fcbec10f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.014+0000 7fcbf11f2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcbec102e00 0x7fcbec10b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.014+0000 7fcbf11f2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec108520 0x7fcbec10b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.014+0000 7fcbf11f2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec1113f0 con 0x7fcbec108520 2024-06-04T05:14:45.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.014+0000 7fcbf11f2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec111270 con 0x7fcbec102440 2024-06-04T05:14:45.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.014+0000 7fcbf11f2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec111570 con 0x7fcbec102e00 2024-06-04T05:14:45.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.014+0000 7fcbead9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbec102440 0x7fcbec10f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.014+0000 7fcbead9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbec102440 0x7fcbec10f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:44352/0 (socket says 172.21.15.70:44352) 2024-06-04T05:14:45.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.014+0000 7fcbead9d700 1 -- 172.21.15.70:0/1415523694 learned_addr learned my addr 172.21.15.70:0/1415523694 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:45.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.014+0000 7fcbeb59e700 1 --2- 172.21.15.70:0/1415523694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec108520 0x7fcbec10b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.014+0000 7fcbea59c700 1 --2- 172.21.15.70:0/1415523694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcbec102e00 0x7fcbec10b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.015+0000 7fcbeb59e700 1 -- 172.21.15.70:0/1415523694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcbec102e00 msgr2=0x7fcbec10b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:45.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.015+0000 7fcbeb59e700 1 --2- 172.21.15.70:0/1415523694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcbec102e00 0x7fcbec10b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.015+0000 7fcbeb59e700 1 -- 172.21.15.70:0/1415523694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbec102440 msgr2=0x7fcbec10f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:45.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.015+0000 7fcbeb59e700 1 --2- 172.21.15.70:0/1415523694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbec102440 0x7fcbec10f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.015+0000 7fcbeb59e700 1 -- 172.21.15.70:0/1415523694 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbec1ac580 con 0x7fcbec108520 2024-06-04T05:14:45.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.015+0000 7fcbead9d700 1 --2- 172.21.15.70:0/1415523694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbec102440 0x7fcbec10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:14:45.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.015+0000 7fcbeb59e700 1 --2- 172.21.15.70:0/1415523694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec108520 0x7fcbec10b860 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400efc0 tx=0x7fcbe400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:45.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.015+0000 7fcbd3fff700 1 -- 172.21.15.70:0/1415523694 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe40092e0 con 0x7fcbec108520 2024-06-04T05:14:45.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.015+0000 7fcbf11f2700 1 -- 172.21.15.70:0/1415523694 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbec1ac750 con 0x7fcbec108520 2024-06-04T05:14:45.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.015+0000 7fcbd3fff700 1 -- 172.21.15.70:0/1415523694 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbe4004bb0 con 0x7fcbec108520 2024-06-04T05:14:45.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.015+0000 7fcbd3fff700 1 -- 172.21.15.70:0/1415523694 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4004020 con 0x7fcbec108520 2024-06-04T05:14:45.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.016+0000 7fcbf11f2700 1 -- 172.21.15.70:0/1415523694 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbec1acba0 con 0x7fcbec108520 2024-06-04T05:14:45.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.020+0000 7fcbd3fff700 1 -- 172.21.15.70:0/1415523694 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fcbe401b030 con 0x7fcbec108520 2024-06-04T05:14:45.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.021+0000 7fcbfba87b80 1 Processor -- start 2024-06-04T05:14:45.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.021+0000 7fcbfba87b80 1 -- start start 2024-06-04T05:14:45.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.021+0000 7fcbd3fff700 1 --2- 172.21.15.70:0/1415523694 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcbd40b2a70 0x7fcbd40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.021+0000 7fcbfba87b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e6320 0x561c887e6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.021+0000 7fcbfba87b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e08a0 0x561c887e0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.021+0000 7fcbfba87b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e1260 0x561c887ed080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.021+0000 7fcbfba87b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561c8873ed70 con 0x561c887e1260 2024-06-04T05:14:45.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.021+0000 7fcbfba87b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561c8872e910 con 0x561c887e6320 2024-06-04T05:14:45.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.021+0000 7fcbfba87b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561c88732950 con 0x561c887e08a0 2024-06-04T05:14:45.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.021+0000 7fcbeb59e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e1260 0x561c887ed080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.021+0000 7fcbead9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e6320 0x561c887e6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.021+0000 7fcbea59c700 1 --2- 172.21.15.70:0/1415523694 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcbd40b2a70 0x7fcbd40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.021+0000 7fcbead9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e6320 0x561c887e6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:44362/0 (socket says 172.21.15.70:44362) 2024-06-04T05:14:45.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.021+0000 7fcbead9d700 1 -- 172.21.15.70:0/3059856021 learned_addr learned my addr 172.21.15.70:0/3059856021 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:45.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.021+0000 7fcbea59c700 1 --2- 172.21.15.70:0/3059856021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e08a0 0x561c887e0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.021+0000 7fcbd3fff700 1 -- 172.21.15.70:0/1415523694 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fcbe40e2010 con 0x7fcbec108520 2024-06-04T05:14:45.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.022+0000 7fcbeb59e700 1 -- 172.21.15.70:0/3059856021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e08a0 msgr2=0x561c887e0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:45.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.022+0000 7fcbeb59e700 1 --2- 172.21.15.70:0/3059856021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e08a0 0x561c887e0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.022+0000 7fcbeb59e700 1 -- 172.21.15.70:0/3059856021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e6320 msgr2=0x561c887e6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:45.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.022+0000 7fcbeb59e700 1 --2- 172.21.15.70:0/3059856021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e6320 0x561c887e6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.022+0000 7fcbeb59e700 1 -- 172.21.15.70:0/3059856021 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c886d7fb0 con 0x561c887e1260 2024-06-04T05:14:45.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.022+0000 7fcbead9d700 1 --2- 172.21.15.70:0/3059856021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e6320 0x561c887e6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:45.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.022+0000 7fcbeb59e700 1 --2- 172.21.15.70:0/3059856021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e1260 0x561c887ed080 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40df730 tx=0x7fcbe401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a9e865fd0c1ab5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:45.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.022+0000 7fcbea59c700 1 --2- 172.21.15.70:0/1415523694 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcbd40b2a70 0x7fcbd40b4f30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcbec10cac0 tx=0x7fcbe00052b0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:45.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.022+0000 7fcbd17fa700 1 -- 172.21.15.70:0/3059856021 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe40c3bb0 con 0x561c887e1260 2024-06-04T05:14:45.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.022+0000 7fcbd17fa700 1 -- 172.21.15.70:0/3059856021 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbe40a8e60 con 0x561c887e1260 2024-06-04T05:14:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.022+0000 7fcbd17fa700 1 -- 172.21.15.70:0/3059856021 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe40b2120 con 0x561c887e1260 2024-06-04T05:14:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.023+0000 7fcbfba87b80 1 -- 172.21.15.70:0/3059856021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e1260 msgr2=0x561c887ed080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.023+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/3059856021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e1260 0x561c887ed080 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40df730 tx=0x7fcbe401c000 comp rx=0 tx=0).stop 2024-06-04T05:14:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.023+0000 7fcbfba87b80 1 -- 172.21.15.70:0/3059856021 shutdown_connections 2024-06-04T05:14:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.023+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/3059856021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e1260 0x561c887ed080 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.023+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/3059856021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e6320 0x561c887e6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.023+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/3059856021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e08a0 0x561c887e0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.023+0000 7fcbfba87b80 1 -- 172.21.15.70:0/3059856021 >> 172.21.15.70:0/3059856021 conn(0x561c8870c240 msgr2=0x561c887db3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.023+0000 7fcbfba87b80 1 -- 172.21.15.70:0/3059856021 shutdown_connections 2024-06-04T05:14:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.023+0000 7fcbfba87b80 1 -- 172.21.15.70:0/3059856021 wait complete. 2024-06-04T05:14:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.024+0000 7fcbfba87b80 1 -- start start 2024-06-04T05:14:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.024+0000 7fcbfba87b80 1 Processor -- start 2024-06-04T05:14:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.024+0000 7fcbfba87b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e08a0 0x561c887ecc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.024+0000 7fcbfba87b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e1260 0x561c887e9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.024+0000 7fcbfba87b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e6320 0x561c887ea460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.024+0000 7fcbfba87b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561c88634fa0 con 0x561c887e6320 2024-06-04T05:14:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.024+0000 7fcbfba87b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561c88696da0 con 0x561c887e08a0 2024-06-04T05:14:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.024+0000 7fcbfba87b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561c887ed620 con 0x561c887e1260 2024-06-04T05:14:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.024+0000 7fcbfba87b80 10 client.0 _ll_register_callbacks cb 0x561c886c9950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.024+0000 7fcbead9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e08a0 0x561c887ecc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.024+0000 7fcbea59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e1260 0x561c887e9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.024+0000 7fcbea59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e1260 0x561c887e9f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:49574/0 (socket says 172.21.15.70:49574) 2024-06-04T05:14:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.024+0000 7fcbea59c700 1 -- 172.21.15.70:0/2377064069 learned_addr learned my addr 172.21.15.70:0/2377064069 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.025+0000 7fcbeb59e700 1 --2- 172.21.15.70:0/2377064069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e6320 0x561c887ea460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.025+0000 7fcbea59c700 1 -- 172.21.15.70:0/2377064069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e08a0 msgr2=0x561c887ecc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.025+0000 7fcbea59c700 1 --2- 172.21.15.70:0/2377064069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e08a0 0x561c887ecc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.025+0000 7fcbea59c700 1 -- 172.21.15.70:0/2377064069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e6320 msgr2=0x561c887ea460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.025+0000 7fcbea59c700 1 --2- 172.21.15.70:0/2377064069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e6320 0x561c887ea460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.025+0000 7fcbea59c700 1 -- 172.21.15.70:0/2377064069 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c8867e8d0 con 0x561c887e1260 2024-06-04T05:14:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.025+0000 7fcbead9d700 1 --2- 172.21.15.70:0/2377064069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e08a0 0x561c887ecc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:14:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.025+0000 7fcbea59c700 1 --2- 172.21.15.70:0/2377064069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e1260 0x561c887e9f20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0015000 tx=0x7fcbe0009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32ebd32de53c4cd4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.026+0000 7fcbceff5700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:14:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.026+0000 7fcbceff5700 1 -- 172.21.15.70:0/2377064069 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0021040 con 0x561c887e1260 2024-06-04T05:14:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.026+0000 7fcbfba87b80 1 -- 172.21.15.70:0/2377064069 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbe40a8e60 con 0x561c887e1260 2024-06-04T05:14:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.026+0000 7fcbceff5700 1 -- 172.21.15.70:0/2377064069 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fcbe0003bd0 con 0x561c887e1260 2024-06-04T05:14:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.026+0000 7fcbceff5700 1 -- 172.21.15.70:0/2377064069 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fcbe0003dd0 con 0x561c887e1260 2024-06-04T05:14:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.026+0000 7fcbfba87b80 10 client.28918 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.027+0000 7fcbfba87b80 1 -- 172.21.15.70:0/2377064069 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x561c886e8760 con 0x561c887e1260 2024-06-04T05:14:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.027+0000 7fcbceff5700 1 -- 172.21.15.70:0/2377064069 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbe40a8e60 con 0x561c887e1260 2024-06-04T05:14:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.027+0000 7fcbceff5700 1 -- 172.21.15.70:0/2377064069 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7fcbe0031070 con 0x561c887e1260 2024-06-04T05:14:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.027+0000 7fcbfba87b80 10 client.28918 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.027+0000 7fcbfba87b80 10 client.28918 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:14:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.027+0000 7fcbfba87b80 20 client.28918 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.027+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/2377064069 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x561c887fa0a0 0x561c887fc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.027+0000 7fcbfba87b80 4 client.28918 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:14:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.027+0000 7fcbfba87b80 1 -- 172.21.15.70:0/2377064069 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c88740cc0 con 0x561c887fa0a0 2024-06-04T05:14:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.028+0000 7fcbead9d700 1 --2- 172.21.15.70:0/2377064069 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x561c887fa0a0 0x561c887fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.028+0000 7fcbead9d700 1 --2- 172.21.15.70:0/2377064069 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x561c887fa0a0 0x561c887fc510 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d074c8e18490dbff server_cookie=6834e0272dfe7368 in_seq=0 out_seq=0 2024-06-04T05:14:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.028+0000 7fcbceff5700 10 client.28918 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:14:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.029+0000 7fcbceff5700 1 -- 172.21.15.70:0/2377064069 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561c88740cc0 con 0x561c887fa0a0 2024-06-04T05:14:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.029+0000 7fcbceff5700 10 client.28918 handle_command_reply: tid=0 2024-06-04T05:14:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.029+0000 7fcbfba87b80 1 client.28918 shutdown 2024-06-04T05:14:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.029+0000 7fcbfba87b80 1 -- 172.21.15.70:0/2377064069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e1260 msgr2=0x561c887e9f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.029+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/2377064069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e1260 0x561c887e9f20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0015000 tx=0x7fcbe0009ca0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.029+0000 7fcbfba87b80 1 -- 172.21.15.70:0/2377064069 shutdown_connections 2024-06-04T05:14:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.029+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/2377064069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e6320 0x561c887ea460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.029+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/2377064069 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x561c887fa0a0 0x561c887fc510 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.029+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/2377064069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e08a0 0x561c887ecc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.029+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/2377064069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e1260 0x561c887e9f20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.029+0000 7fcbceff5700 0 client.28918 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:14:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.029+0000 7fcbfba87b80 1 -- 172.21.15.70:0/2377064069 >> 172.21.15.70:0/2377064069 conn(0x561c8870c240 msgr2=0x561c887daaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.030+0000 7fcbfba87b80 1 -- 172.21.15.70:0/2377064069 shutdown_connections 2024-06-04T05:14:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.030+0000 7fcbfba87b80 1 -- 172.21.15.70:0/2377064069 wait complete. 2024-06-04T05:14:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.030+0000 7fcbfba87b80 20 client.28918 trim_cache size 0 max 16384 2024-06-04T05:14:45.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.049+0000 7fcbfba87b80 1 Processor -- start 2024-06-04T05:14:45.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.049+0000 7fcbfba87b80 1 -- start start 2024-06-04T05:14:45.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.049+0000 7fcbfba87b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e08a0 0x561c887e5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.049+0000 7fcbfba87b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e1260 0x561c887db270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.049+0000 7fcbfba87b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e6320 0x561c88801e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.049+0000 7fcbfba87b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd800d710 con 0x561c887e08a0 2024-06-04T05:14:45.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.049+0000 7fcbfba87b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd800f2b0 con 0x561c887e1260 2024-06-04T05:14:45.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.049+0000 7fcbfba87b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd800de40 con 0x561c887e6320 2024-06-04T05:14:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.049+0000 7fcbead9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e08a0 0x561c887e5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.049+0000 7fcbead9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e08a0 0x561c887e5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50172/0 (socket says 172.21.15.70:50172) 2024-06-04T05:14:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.049+0000 7fcbead9d700 1 -- 172.21.15.70:0/2139444714 learned_addr learned my addr 172.21.15.70:0/2139444714 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.049+0000 7fcbeb59e700 1 --2- 172.21.15.70:0/2139444714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e6320 0x561c88801e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.049+0000 7fcbea59c700 1 --2- 172.21.15.70:0/2139444714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e1260 0x561c887db270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.050+0000 7fcbead9d700 1 -- 172.21.15.70:0/2139444714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e6320 msgr2=0x561c88801e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.050+0000 7fcbead9d700 1 --2- 172.21.15.70:0/2139444714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e6320 0x561c88801e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.050+0000 7fcbead9d700 1 -- 172.21.15.70:0/2139444714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e1260 msgr2=0x561c887db270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.050+0000 7fcbead9d700 1 --2- 172.21.15.70:0/2139444714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e1260 0x561c887db270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.050+0000 7fcbead9d700 1 -- 172.21.15.70:0/2139444714 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c887ec860 con 0x561c887e08a0 2024-06-04T05:14:45.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.050+0000 7fcbead9d700 1 --2- 172.21.15.70:0/2139444714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e08a0 0x561c887e5780 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc004790 tx=0x7fcbdc0045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a367635724316b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.050+0000 7fcbd0ff9700 1 -- 172.21.15.70:0/2139444714 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc023070 con 0x561c887e08a0 2024-06-04T05:14:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.050+0000 7fcbd0ff9700 1 -- 172.21.15.70:0/2139444714 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbdc003e40 con 0x561c887e08a0 2024-06-04T05:14:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.050+0000 7fcbd0ff9700 1 -- 172.21.15.70:0/2139444714 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc01c780 con 0x561c887e08a0 2024-06-04T05:14:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.051+0000 7fcbfba87b80 1 -- 172.21.15.70:0/2139444714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e08a0 msgr2=0x561c887e5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.051+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/2139444714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e08a0 0x561c887e5780 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc004790 tx=0x7fcbdc0045f0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.051+0000 7fcbfba87b80 1 -- 172.21.15.70:0/2139444714 shutdown_connections 2024-06-04T05:14:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.051+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/2139444714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e6320 0x561c88801e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.051+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/2139444714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e08a0 0x561c887e5780 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.051+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/2139444714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e1260 0x561c887db270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.051+0000 7fcbfba87b80 1 -- 172.21.15.70:0/2139444714 >> 172.21.15.70:0/2139444714 conn(0x561c8870c240 msgr2=0x561c887e5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.051+0000 7fcbfba87b80 1 -- 172.21.15.70:0/2139444714 shutdown_connections 2024-06-04T05:14:45.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.051+0000 7fcbfba87b80 1 -- 172.21.15.70:0/2139444714 wait complete. 2024-06-04T05:14:45.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.052+0000 7fcbfba87b80 1 -- start start 2024-06-04T05:14:45.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.052+0000 7fcbfba87b80 1 Processor -- start 2024-06-04T05:14:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.052+0000 7fcbfba87b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e08a0 0x561c887f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.052+0000 7fcbfba87b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e1260 0x561c887f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.052+0000 7fcbfba87b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e6320 0x561c88800f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.052+0000 7fcbfba87b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd800ebe0 con 0x561c887e08a0 2024-06-04T05:14:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.052+0000 7fcbfba87b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd800e510 con 0x561c887e6320 2024-06-04T05:14:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.052+0000 7fcbfba87b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd800b6d0 con 0x561c887e1260 2024-06-04T05:14:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.052+0000 7fcbfba87b80 10 client.0 _ll_register_callbacks cb 0x561c886c9950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.052+0000 7fcbead9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e08a0 0x561c887f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.052+0000 7fcbea59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e1260 0x561c887f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.052+0000 7fcbead9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e08a0 0x561c887f28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50184/0 (socket says 172.21.15.70:50184) 2024-06-04T05:14:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.052+0000 7fcbea59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e1260 0x561c887f2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:49588/0 (socket says 172.21.15.70:49588) 2024-06-04T05:14:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.052+0000 7fcbeb59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e6320 0x561c88800f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.052+0000 7fcbead9d700 1 -- 172.21.15.70:0/228386789 learned_addr learned my addr 172.21.15.70:0/228386789 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.053+0000 7fcbeb59e700 1 -- 172.21.15.70:0/228386789 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e1260 msgr2=0x561c887f2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.053+0000 7fcbeb59e700 1 --2- 172.21.15.70:0/228386789 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e1260 0x561c887f2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.053+0000 7fcbeb59e700 1 -- 172.21.15.70:0/228386789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e08a0 msgr2=0x561c887f28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.053+0000 7fcbeb59e700 1 --2- 172.21.15.70:0/228386789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e08a0 0x561c887f28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.053+0000 7fcbeb59e700 1 -- 172.21.15.70:0/228386789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c88801610 con 0x561c887e6320 2024-06-04T05:14:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.053+0000 7fcbead9d700 1 --2- 172.21.15.70:0/228386789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e08a0 0x561c887f28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.053+0000 7fcbeb59e700 1 --2- 172.21.15.70:0/228386789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e6320 0x561c88800f90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40a92e0 tx=0x7fcbe40a9860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dac0a0724dcc6979 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.053+0000 7fcbceff5700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:14:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.053+0000 7fcbceff5700 1 -- 172.21.15.70:0/228386789 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe40b2120 con 0x561c887e6320 2024-06-04T05:14:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.053+0000 7fcbceff5700 1 -- 172.21.15.70:0/228386789 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7fcbe40b1ea0 con 0x561c887e6320 2024-06-04T05:14:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.053+0000 7fcbfba87b80 1 -- 172.21.15.70:0/228386789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbdc003e40 con 0x561c887e6320 2024-06-04T05:14:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.053+0000 7fcbceff5700 1 -- 172.21.15.70:0/228386789 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fcbe40c3050 con 0x561c887e6320 2024-06-04T05:14:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.053+0000 7fcbfba87b80 10 client.28817 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.053+0000 7fcbfba87b80 1 -- 172.21.15.70:0/228386789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x561c88864650 con 0x561c887e6320 2024-06-04T05:14:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.054+0000 7fcbceff5700 1 -- 172.21.15.70:0/228386789 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbdc003e40 con 0x561c887e6320 2024-06-04T05:14:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.054+0000 7fcbceff5700 1 -- 172.21.15.70:0/228386789 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7fcbe40a4cc0 con 0x561c887e6320 2024-06-04T05:14:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.054+0000 7fcbfba87b80 10 client.28817 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.054+0000 7fcbfba87b80 10 client.28817 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:14:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.054+0000 7fcbfba87b80 20 client.28817 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.054+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/228386789 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x561c88864940 0x561c88866d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.054+0000 7fcbfba87b80 4 client.28817 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:14:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.054+0000 7fcbfba87b80 1 -- 172.21.15.70:0/228386789 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561c8873a4a0 con 0x561c88864940 2024-06-04T05:14:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.054+0000 7fcbead9d700 1 --2- 172.21.15.70:0/228386789 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x561c88864940 0x561c88866d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.055+0000 7fcbead9d700 1 --2- 172.21.15.70:0/228386789 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x561c88864940 0x561c88866d30 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=516979e301ddbeb9 server_cookie=b7d59597ef4ccfb2 in_seq=0 out_seq=0 2024-06-04T05:14:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.055+0000 7fcbceff5700 10 client.28817 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:14:45.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.055+0000 7fcbceff5700 1 -- 172.21.15.70:0/228386789 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561c8873a4a0 con 0x561c88864940 2024-06-04T05:14:45.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.055+0000 7fcbceff5700 10 client.28817 handle_command_reply: tid=0 2024-06-04T05:14:45.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.055+0000 7fcbfba87b80 1 client.28817 shutdown 2024-06-04T05:14:45.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.055+0000 7fcbfba87b80 1 -- 172.21.15.70:0/228386789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e6320 msgr2=0x561c88800f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:45.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.055+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/228386789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e6320 0x561c88800f90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40a92e0 tx=0x7fcbe40a9860 comp rx=0 tx=0).stop 2024-06-04T05:14:45.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.056+0000 7fcbfba87b80 1 -- 172.21.15.70:0/228386789 shutdown_connections 2024-06-04T05:14:45.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.056+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/228386789 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x561c88864940 0x561c88866d30 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.056+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/228386789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c887e6320 0x561c88800f90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.056+0000 7fcbceff5700 0 client.28817 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:14:45.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.056+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/228386789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c887e08a0 0x561c887f28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.056+0000 7fcbfba87b80 1 --2- 172.21.15.70:0/228386789 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c887e1260 0x561c887f2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.056+0000 7fcbfba87b80 1 -- 172.21.15.70:0/228386789 >> 172.21.15.70:0/228386789 conn(0x561c8870c240 msgr2=0x561c887e5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:45.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.056+0000 7fcbfba87b80 1 -- 172.21.15.70:0/228386789 shutdown_connections 2024-06-04T05:14:45.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.056+0000 7fcbfba87b80 1 -- 172.21.15.70:0/228386789 wait complete. 2024-06-04T05:14:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.056+0000 7fcbfba87b80 20 client.28817 trim_cache size 0 max 16384 2024-06-04T05:14:45.063 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:14:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.059+0000 7fcbf11f2700 1 -- 172.21.15.70:0/1415523694 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcbd40b2a70 msgr2=0x7fcbd40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.059+0000 7fcbf11f2700 1 --2- 172.21.15.70:0/1415523694 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcbd40b2a70 0x7fcbd40b4f30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcbec10cac0 tx=0x7fcbe00052b0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.059+0000 7fcbf11f2700 1 -- 172.21.15.70:0/1415523694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec108520 msgr2=0x7fcbec10b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.059+0000 7fcbf11f2700 1 --2- 172.21.15.70:0/1415523694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec108520 0x7fcbec10b860 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400efc0 tx=0x7fcbe400c3a0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.059+0000 7fcbf11f2700 1 -- 172.21.15.70:0/1415523694 shutdown_connections 2024-06-04T05:14:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.059+0000 7fcbf11f2700 1 --2- 172.21.15.70:0/1415523694 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcbd40b2a70 0x7fcbd40b4f30 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.059+0000 7fcbf11f2700 1 --2- 172.21.15.70:0/1415523694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbec108520 0x7fcbec10b860 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.059+0000 7fcbf11f2700 1 --2- 172.21.15.70:0/1415523694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbec102440 0x7fcbec10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.059+0000 7fcbf11f2700 1 --2- 172.21.15.70:0/1415523694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcbec102e00 0x7fcbec10b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.059+0000 7fcbf11f2700 1 -- 172.21.15.70:0/1415523694 >> 172.21.15.70:0/1415523694 conn(0x7fcbec0793f0 msgr2=0x7fcbec100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.059+0000 7fcbf11f2700 1 -- 172.21.15.70:0/1415523694 shutdown_connections 2024-06-04T05:14:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:45.059+0000 7fcbf11f2700 1 -- 172.21.15.70:0/1415523694 wait complete. 2024-06-04T05:14:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:45 smithi070 ceph-mon[32631]: pgmap v550: 97 pgs: 97 active+clean; 560 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 3.9 MiB/s wr, 210 op/s 2024-06-04T05:14:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:45 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:14:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:45 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:14:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:45 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:14:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:45 smithi112 ceph-mon[36453]: pgmap v550: 97 pgs: 97 active+clean; 560 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 3.9 MiB/s wr, 210 op/s 2024-06-04T05:14:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:45 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:14:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:45 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:14:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:45 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:14:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:45 smithi098 ceph-mon[35569]: pgmap v550: 97 pgs: 97 active+clean; 560 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 3.9 MiB/s wr, 210 op/s 2024-06-04T05:14:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:45 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:14:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:45 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:14:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:45 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:14:45.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 31246 8.19 MB/sec execute 97 sec latency 111.916 ms 2024-06-04T05:14:46.089 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:14:46.431 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:14:46.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 31432 8.23 MB/sec execute 98 sec latency 59.861 ms 2024-06-04T05:14:47.599 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:47 smithi070 ceph-mon[32631]: pgmap v551: 97 pgs: 97 active+clean; 567 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.2 MiB/s wr, 335 op/s 2024-06-04T05:14:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:47 smithi112 ceph-mon[36453]: pgmap v551: 97 pgs: 97 active+clean; 567 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.2 MiB/s wr, 335 op/s 2024-06-04T05:14:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:47 smithi098 ceph-mon[35569]: pgmap v551: 97 pgs: 97 active+clean; 567 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.2 MiB/s wr, 335 op/s 2024-06-04T05:14:47.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 31622 8.23 MB/sec execute 99 sec latency 53.484 ms 2024-06-04T05:14:47.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.835+0000 7f4af0209640 1 -- 172.21.15.70:0/3227805573 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ae810f270 msgr2=0x7f4ae81116c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:47.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.835+0000 7f4af0209640 1 --2- 172.21.15.70:0/3227805573 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ae810f270 0x7f4ae81116c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0009900 tx=0x7f4ae002a480 comp rx=0 tx=0).stop 2024-06-04T05:14:47.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.836+0000 7f4af0209640 1 -- 172.21.15.70:0/3227805573 shutdown_connections 2024-06-04T05:14:47.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.836+0000 7f4af0209640 1 --2- 172.21.15.70:0/3227805573 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ae810f270 0x7f4ae81116c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.836+0000 7f4af0209640 1 --2- 172.21.15.70:0/3227805573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ae80757a0 0x7f4ae810ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.836+0000 7f4af0209640 1 --2- 172.21.15.70:0/3227805573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae8074e80 0x7f4ae8075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.836+0000 7f4af0209640 1 -- 172.21.15.70:0/3227805573 >> 172.21.15.70:0/3227805573 conn(0x7f4ae80fe500 msgr2=0x7f4ae8100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.836+0000 7f4af0209640 1 -- 172.21.15.70:0/3227805573 shutdown_connections 2024-06-04T05:14:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.836+0000 7f4af0209640 1 -- 172.21.15.70:0/3227805573 wait complete. 2024-06-04T05:14:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.836+0000 7f4af0209640 1 Processor -- start 2024-06-04T05:14:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.836+0000 7f4af0209640 1 -- start start 2024-06-04T05:14:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.837+0000 7f4af0209640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae8074e80 0x7f4ae819b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.837+0000 7f4af0209640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ae80757a0 0x7f4ae819b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.837+0000 7f4af0209640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ae810f270 0x7f4ae819fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.837+0000 7f4af0209640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae8114d70 con 0x7f4ae80757a0 2024-06-04T05:14:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.837+0000 7f4af0209640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae8114bf0 con 0x7f4ae8074e80 2024-06-04T05:14:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.837+0000 7f4af0209640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae8114ef0 con 0x7f4ae810f270 2024-06-04T05:14:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.837+0000 7f4aedfa5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae8074e80 0x7f4ae819b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.837+0000 7f4aed7a4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ae80757a0 0x7f4ae819b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.837+0000 7f4aedfa5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae8074e80 0x7f4ae819b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:48816/0 (socket says 172.21.15.70:48816) 2024-06-04T05:14:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.837+0000 7f4aedfa5640 1 -- 172.21.15.70:0/3035544635 learned_addr learned my addr 172.21.15.70:0/3035544635 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.837+0000 7f4aee7a6640 1 --2- 172.21.15.70:0/3035544635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ae810f270 0x7f4ae819fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.837+0000 7f4aedfa5640 1 -- 172.21.15.70:0/3035544635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ae810f270 msgr2=0x7f4ae819fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.837+0000 7f4aedfa5640 1 --2- 172.21.15.70:0/3035544635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ae810f270 0x7f4ae819fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.837+0000 7f4aedfa5640 1 -- 172.21.15.70:0/3035544635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ae80757a0 msgr2=0x7f4ae819b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.837+0000 7f4aedfa5640 1 --2- 172.21.15.70:0/3035544635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ae80757a0 0x7f4ae819b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.837+0000 7f4aedfa5640 1 -- 172.21.15.70:0/3035544635 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae81a02d0 con 0x7f4ae8074e80 2024-06-04T05:14:47.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.837+0000 7f4aedfa5640 1 --2- 172.21.15.70:0/3035544635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae8074e80 0x7f4ae819b330 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4ae400eb90 tx=0x7f4ae400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:47.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.838+0000 7f4adaffd640 1 -- 172.21.15.70:0/3035544635 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae4004d20 con 0x7f4ae8074e80 2024-06-04T05:14:47.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.838+0000 7f4adaffd640 1 -- 172.21.15.70:0/3035544635 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4ae400ee90 con 0x7f4ae8074e80 2024-06-04T05:14:47.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.838+0000 7f4af0209640 1 -- 172.21.15.70:0/3035544635 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ae81a05c0 con 0x7f4ae8074e80 2024-06-04T05:14:47.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.838+0000 7f4adaffd640 1 -- 172.21.15.70:0/3035544635 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae4010690 con 0x7f4ae8074e80 2024-06-04T05:14:47.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.838+0000 7f4af0209640 1 -- 172.21.15.70:0/3035544635 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ae81a7e50 con 0x7f4ae8074e80 2024-06-04T05:14:47.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.840+0000 7f4af3001740 1 Processor -- start 2024-06-04T05:14:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.841+0000 7f4af3001740 1 -- start start 2024-06-04T05:14:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.841+0000 7f4adaffd640 1 -- 172.21.15.70:0/3035544635 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f4ae4005020 con 0x7f4ae8074e80 2024-06-04T05:14:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.841+0000 7f4af3001740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d00820 0x55e046d00c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.841+0000 7f4af3001740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d01140 0x55e046d0b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.841+0000 7f4aee7a6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d00820 0x55e046d00c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.841+0000 7f4aee7a6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d00820 0x55e046d00c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43318/0 (socket says 172.21.15.70:43318) 2024-06-04T05:14:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.841+0000 7f4af3001740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d0ba00 0x55e046d0ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.841+0000 7f4aee7a6640 1 -- 172.21.15.70:0/3363271058 learned_addr learned my addr 172.21.15.70:0/3363271058 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.841+0000 7f4af3001740 1 -- 172.21.15.70:0/3363271058 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e046c72c30 con 0x55e046d00820 2024-06-04T05:14:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.841+0000 7f4af3001740 1 -- 172.21.15.70:0/3363271058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e046c627d0 con 0x55e046d01140 2024-06-04T05:14:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.841+0000 7f4af3001740 1 -- 172.21.15.70:0/3363271058 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e046c66810 con 0x55e046d0ba00 2024-06-04T05:14:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.841+0000 7f4aedfa5640 1 --2- 172.21.15.70:0/3363271058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d01140 0x55e046d0b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.841+0000 7f4aed7a4640 1 --2- 172.21.15.70:0/3363271058 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d0ba00 0x55e046d0ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.842+0000 7f4aee7a6640 1 -- 172.21.15.70:0/3363271058 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d0ba00 msgr2=0x55e046d0ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:47.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.842+0000 7f4aee7a6640 1 --2- 172.21.15.70:0/3363271058 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d0ba00 0x55e046d0ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.842+0000 7f4aee7a6640 1 -- 172.21.15.70:0/3363271058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d01140 msgr2=0x55e046d0b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:47.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.842+0000 7f4aee7a6640 1 --2- 172.21.15.70:0/3363271058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d01140 0x55e046d0b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.842+0000 7f4aee7a6640 1 -- 172.21.15.70:0/3363271058 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e046977210 con 0x55e046d00820 2024-06-04T05:14:47.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.842+0000 7f4aed7a4640 1 --2- 172.21.15.70:0/3363271058 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d0ba00 0x55e046d0ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:14:47.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.842+0000 7f4aee7a6640 1 --2- 172.21.15.70:0/3363271058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d00820 0x55e046d00c00 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0009900 tx=0x7f4ae0031b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=940faf00d3187cb5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:47.848 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.842+0000 7f4adaffd640 1 --2- 172.21.15.70:0/3035544635 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4ad40b2a90 0x7f4ad40b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.848 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.842+0000 7f4ad3fff640 1 -- 172.21.15.70:0/3363271058 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae0035030 con 0x55e046d00820 2024-06-04T05:14:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.842+0000 7f4ad3fff640 1 -- 172.21.15.70:0/3363271058 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4ae0040070 con 0x55e046d00820 2024-06-04T05:14:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.842+0000 7f4ad3fff640 1 -- 172.21.15.70:0/3363271058 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae0003850 con 0x55e046d00820 2024-06-04T05:14:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.842+0000 7f4aedfa5640 1 --2- 172.21.15.70:0/3035544635 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4ad40b2a90 0x7f4ad40b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.843+0000 7f4adaffd640 1 -- 172.21.15.70:0/3035544635 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f4ae40e2c30 con 0x7f4ae8074e80 2024-06-04T05:14:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.843+0000 7f4af3001740 1 -- 172.21.15.70:0/3363271058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d00820 msgr2=0x55e046d00c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.843+0000 7f4af3001740 1 --2- 172.21.15.70:0/3363271058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d00820 0x55e046d00c00 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0009900 tx=0x7f4ae0031b20 comp rx=0 tx=0).stop 2024-06-04T05:14:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.843+0000 7f4aedfa5640 1 --2- 172.21.15.70:0/3035544635 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4ad40b2a90 0x7f4ad40b4f50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x55e046d01dc0 tx=0x7f4ae40e66e0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.843+0000 7f4af3001740 1 -- 172.21.15.70:0/3363271058 shutdown_connections 2024-06-04T05:14:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.843+0000 7f4af3001740 1 --2- 172.21.15.70:0/3363271058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d01140 0x55e046d0b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.843+0000 7f4af3001740 1 --2- 172.21.15.70:0/3363271058 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d0ba00 0x55e046d0ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.843+0000 7f4af3001740 1 --2- 172.21.15.70:0/3363271058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d00820 0x55e046d00c00 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.843+0000 7f4af3001740 1 -- 172.21.15.70:0/3363271058 >> 172.21.15.70:0/3363271058 conn(0x55e046cfcd30 msgr2=0x55e046cfd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.843+0000 7f4af3001740 1 -- 172.21.15.70:0/3363271058 shutdown_connections 2024-06-04T05:14:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.843+0000 7f4af3001740 1 -- 172.21.15.70:0/3363271058 wait complete. 2024-06-04T05:14:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.844+0000 7f4af3001740 1 -- start start 2024-06-04T05:14:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.844+0000 7f4af3001740 1 Processor -- start 2024-06-04T05:14:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.844+0000 7f4af3001740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d00820 0x55e046d08a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.844+0000 7f4af3001740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d01140 0x55e046d08f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.844+0000 7f4af3001740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d0ba00 0x55e046d094b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.844+0000 7f4af3001740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e046948790 con 0x55e046d01140 2024-06-04T05:14:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.844+0000 7f4af3001740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e046c1e890 con 0x55e046d00820 2024-06-04T05:14:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.844+0000 7f4af3001740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e046d0e5d0 con 0x55e046d0ba00 2024-06-04T05:14:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.844+0000 7f4aee7a6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d00820 0x55e046d08a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.844+0000 7f4af3001740 10 client.0 _ll_register_callbacks cb 0x55e046976a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.844+0000 7f4aee7a6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d00820 0x55e046d08a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:48830/0 (socket says 172.21.15.70:48830) 2024-06-04T05:14:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.844+0000 7f4aee7a6640 1 -- 172.21.15.70:0/2357210653 learned_addr learned my addr 172.21.15.70:0/2357210653 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.845+0000 7f4aee7a6640 1 --2- 172.21.15.70:0/2357210653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d0ba00 0x55e046d094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.845+0000 7f4aed7a4640 1 --2- 172.21.15.70:0/2357210653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d01140 0x55e046d08f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.845+0000 7f4aee7a6640 1 -- 172.21.15.70:0/2357210653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d0ba00 msgr2=0x55e046d094b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.845+0000 7f4aee7a6640 1 --2- 172.21.15.70:0/2357210653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d0ba00 0x55e046d094b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.845+0000 7f4aee7a6640 1 -- 172.21.15.70:0/2357210653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d01140 msgr2=0x55e046d08f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.845+0000 7f4aee7a6640 1 --2- 172.21.15.70:0/2357210653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d01140 0x55e046d08f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.845+0000 7f4aee7a6640 1 -- 172.21.15.70:0/2357210653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e046a43620 con 0x55e046d00820 2024-06-04T05:14:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.845+0000 7f4aee7a6640 1 --2- 172.21.15.70:0/2357210653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d00820 0x55e046d08a30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0004530 tx=0x7f4ae0004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea15f7af681c297b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.845+0000 7f4ad17fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:14:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.845+0000 7f4ad17fa640 1 -- 172.21.15.70:0/2357210653 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae0035040 con 0x55e046d00820 2024-06-04T05:14:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.845+0000 7f4ad17fa640 1 -- 172.21.15.70:0/2357210653 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f4ae0031610 con 0x55e046d00820 2024-06-04T05:14:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.845+0000 7f4af3001740 1 -- 172.21.15.70:0/2357210653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ae0040070 con 0x55e046d00820 2024-06-04T05:14:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.846+0000 7f4ad17fa640 1 -- 172.21.15.70:0/2357210653 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f4ae0031810 con 0x55e046d00820 2024-06-04T05:14:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.846+0000 7f4af3001740 10 client.28829 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.846+0000 7f4af3001740 1 -- 172.21.15.70:0/2357210653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55e046d1a3d0 con 0x55e046d00820 2024-06-04T05:14:47.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.846+0000 7f4ad17fa640 1 -- 172.21.15.70:0/2357210653 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4ae0040070 con 0x55e046d00820 2024-06-04T05:14:47.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.846+0000 7f4ad17fa640 1 -- 172.21.15.70:0/2357210653 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f4ae003c640 con 0x55e046d00820 2024-06-04T05:14:47.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.846+0000 7f4af3001740 10 client.28829 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:47.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.846+0000 7f4af3001740 10 client.28829 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:14:47.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.846+0000 7f4af3001740 20 client.28829 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:47.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.846+0000 7f4af3001740 1 --2- 172.21.15.70:0/2357210653 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e046d1b720 0x55e046d1db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.846+0000 7f4af3001740 4 client.28829 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:14:47.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.846+0000 7f4af3001740 1 -- 172.21.15.70:0/2357210653 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e046c74b80 con 0x55e046d1b720 2024-06-04T05:14:47.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.847+0000 7f4aed7a4640 1 --2- 172.21.15.70:0/2357210653 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e046d1b720 0x55e046d1db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.847+0000 7f4ad17fa640 10 client.28829 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:47.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.847+0000 7f4aed7a4640 1 --2- 172.21.15.70:0/2357210653 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e046d1b720 0x55e046d1db10 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88cbf8526b0a5736 server_cookie=68a46dd863bf3daa in_seq=0 out_seq=0 2024-06-04T05:14:47.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.848+0000 7f4ad17fa640 1 -- 172.21.15.70:0/2357210653 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e046c74b80 con 0x55e046d1b720 2024-06-04T05:14:47.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.848+0000 7f4ad17fa640 10 client.28829 handle_command_reply: tid=0 2024-06-04T05:14:47.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.848+0000 7f4af3001740 1 client.28829 shutdown 2024-06-04T05:14:47.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.848+0000 7f4af3001740 1 -- 172.21.15.70:0/2357210653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d00820 msgr2=0x55e046d08a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.848+0000 7f4af3001740 1 --2- 172.21.15.70:0/2357210653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d00820 0x55e046d08a30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0004530 tx=0x7f4ae0004430 comp rx=0 tx=0).stop 2024-06-04T05:14:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.848+0000 7f4af3001740 1 -- 172.21.15.70:0/2357210653 shutdown_connections 2024-06-04T05:14:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.848+0000 7f4af3001740 1 --2- 172.21.15.70:0/2357210653 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e046d1b720 0x55e046d1db10 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.848+0000 7f4ad17fa640 0 client.28829 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.848+0000 7f4af3001740 1 --2- 172.21.15.70:0/2357210653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d0ba00 0x55e046d094b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.848+0000 7f4af3001740 1 --2- 172.21.15.70:0/2357210653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d01140 0x55e046d08f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.848+0000 7f4af3001740 1 --2- 172.21.15.70:0/2357210653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d00820 0x55e046d08a30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.848+0000 7f4af3001740 1 -- 172.21.15.70:0/2357210653 >> 172.21.15.70:0/2357210653 conn(0x55e046cfcd30 msgr2=0x55e046d02ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.848+0000 7f4af3001740 1 -- 172.21.15.70:0/2357210653 shutdown_connections 2024-06-04T05:14:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.848+0000 7f4af3001740 1 -- 172.21.15.70:0/2357210653 wait complete. 2024-06-04T05:14:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.849+0000 7f4af3001740 20 client.28829 trim_cache size 0 max 16384 2024-06-04T05:14:47.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.871+0000 7f4af3001740 1 Processor -- start 2024-06-04T05:14:47.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.871+0000 7f4af3001740 1 -- start start 2024-06-04T05:14:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.871+0000 7f4af3001740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d00820 0x55e046cfc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.871+0000 7f4af3001740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d01140 0x55e046d092d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.871+0000 7f4af3001740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d0ba00 0x55e046d0aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.871+0000 7f4af3001740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae81b0770 con 0x55e046d0ba00 2024-06-04T05:14:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.871+0000 7f4af3001740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae81b2310 con 0x55e046d00820 2024-06-04T05:14:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.871+0000 7f4af3001740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae81b0ea0 con 0x55e046d01140 2024-06-04T05:14:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.872+0000 7f4aee7a6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d0ba00 0x55e046d0aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.872+0000 7f4aee7a6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d0ba00 0x55e046d0aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43338/0 (socket says 172.21.15.70:43338) 2024-06-04T05:14:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.872+0000 7f4aee7a6640 1 -- 172.21.15.70:0/3977020565 learned_addr learned my addr 172.21.15.70:0/3977020565 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.872+0000 7f4aed7a4640 1 --2- 172.21.15.70:0/3977020565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d01140 0x55e046d092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.872+0000 7f4aee7a6640 1 --2- 172.21.15.70:0/3977020565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d00820 0x55e046cfc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.872+0000 7f4aee7a6640 1 -- 172.21.15.70:0/3977020565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d01140 msgr2=0x55e046d092d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.872+0000 7f4aee7a6640 1 --2- 172.21.15.70:0/3977020565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d01140 0x55e046d092d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.872+0000 7f4aee7a6640 1 -- 172.21.15.70:0/3977020565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d00820 msgr2=0x55e046cfc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.872+0000 7f4aee7a6640 1 --2- 172.21.15.70:0/3977020565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d00820 0x55e046cfc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.872+0000 7f4aee7a6640 1 -- 172.21.15.70:0/3977020565 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e046d1de80 con 0x55e046d0ba00 2024-06-04T05:14:47.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.872+0000 7f4aee7a6640 1 --2- 172.21.15.70:0/3977020565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d0ba00 0x55e046d0aaf0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0004400 tx=0x7f4ae002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a0c8deff44333bf server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:47.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.872+0000 7f4ad37fe640 1 -- 172.21.15.70:0/3977020565 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae0035070 con 0x55e046d0ba00 2024-06-04T05:14:47.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.872+0000 7f4ad37fe640 1 -- 172.21.15.70:0/3977020565 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4ae0040040 con 0x55e046d0ba00 2024-06-04T05:14:47.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.872+0000 7f4ad37fe640 1 -- 172.21.15.70:0/3977020565 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae0003890 con 0x55e046d0ba00 2024-06-04T05:14:47.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.873+0000 7f4af3001740 1 -- 172.21.15.70:0/3977020565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d0ba00 msgr2=0x55e046d0aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:47.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.873+0000 7f4af3001740 1 --2- 172.21.15.70:0/3977020565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d0ba00 0x55e046d0aaf0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0004400 tx=0x7f4ae002a480 comp rx=0 tx=0).stop 2024-06-04T05:14:47.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.873+0000 7f4af3001740 1 -- 172.21.15.70:0/3977020565 shutdown_connections 2024-06-04T05:14:47.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.873+0000 7f4af3001740 1 --2- 172.21.15.70:0/3977020565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d0ba00 0x55e046d0aaf0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.873+0000 7f4af3001740 1 --2- 172.21.15.70:0/3977020565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d01140 0x55e046d092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.873+0000 7f4af3001740 1 --2- 172.21.15.70:0/3977020565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d00820 0x55e046cfc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.873+0000 7f4af3001740 1 -- 172.21.15.70:0/3977020565 >> 172.21.15.70:0/3977020565 conn(0x55e046cfcd30 msgr2=0x55e046d170f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:47.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.874+0000 7f4af3001740 1 -- 172.21.15.70:0/3977020565 shutdown_connections 2024-06-04T05:14:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.874+0000 7f4af3001740 1 -- 172.21.15.70:0/3977020565 wait complete. 2024-06-04T05:14:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.874+0000 7f4af3001740 1 -- start start 2024-06-04T05:14:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.874+0000 7f4af3001740 1 Processor -- start 2024-06-04T05:14:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.874+0000 7f4af3001740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d00820 0x55e046d05910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.874+0000 7f4af3001740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d01140 0x55e046d05e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4af3001740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d0ba00 0x55e046d06390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4af3001740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae81b1c40 con 0x55e046d00820 2024-06-04T05:14:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4af3001740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae81b1570 con 0x55e046d01140 2024-06-04T05:14:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4af3001740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae81ae770 con 0x55e046d0ba00 2024-06-04T05:14:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4af3001740 10 client.0 _ll_register_callbacks cb 0x7f4ae40e65b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4aee7a6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d00820 0x55e046d05910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4aee7a6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d00820 0x55e046d05910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43354/0 (socket says 172.21.15.70:43354) 2024-06-04T05:14:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4aee7a6640 1 -- 172.21.15.70:0/2322373744 learned_addr learned my addr 172.21.15.70:0/2322373744 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4aed7a4640 1 --2- 172.21.15.70:0/2322373744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d01140 0x55e046d05e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4aee7a6640 1 --2- 172.21.15.70:0/2322373744 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d0ba00 0x55e046d06390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4aee7a6640 1 -- 172.21.15.70:0/2322373744 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d0ba00 msgr2=0x55e046d06390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4aee7a6640 1 --2- 172.21.15.70:0/2322373744 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d0ba00 0x55e046d06390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4aee7a6640 1 -- 172.21.15.70:0/2322373744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d01140 msgr2=0x55e046d05e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4aee7a6640 1 --2- 172.21.15.70:0/2322373744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d01140 0x55e046d05e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4aee7a6640 1 -- 172.21.15.70:0/2322373744 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e046d84fb0 con 0x55e046d00820 2024-06-04T05:14:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4aee7a6640 1 --2- 172.21.15.70:0/2322373744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d00820 0x55e046d05910 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f4ae003ca30 tx=0x7f4ae0031e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b753ba3d56841784 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4ad17fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:14:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4ad17fa640 1 -- 172.21.15.70:0/2322373744 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae0035070 con 0x55e046d00820 2024-06-04T05:14:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4ad17fa640 1 -- 172.21.15.70:0/2322373744 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f4ae0003f30 con 0x55e046d00820 2024-06-04T05:14:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.875+0000 7f4af3001740 1 -- 172.21.15.70:0/2322373744 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ae0040040 con 0x55e046d00820 2024-06-04T05:14:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.876+0000 7f4ad17fa640 1 -- 172.21.15.70:0/2322373744 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f4ae0031b20 con 0x55e046d00820 2024-06-04T05:14:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.876+0000 7f4af3001740 10 client.20010 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.876+0000 7f4af3001740 1 -- 172.21.15.70:0/2322373744 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55e046d86160 con 0x55e046d00820 2024-06-04T05:14:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.876+0000 7f4ad17fa640 1 -- 172.21.15.70:0/2322373744 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4ae0003d20 con 0x55e046d00820 2024-06-04T05:14:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.876+0000 7f4ad17fa640 1 -- 172.21.15.70:0/2322373744 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f4ae0071890 con 0x55e046d00820 2024-06-04T05:14:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.876+0000 7f4af3001740 10 client.20010 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.876+0000 7f4af3001740 10 client.20010 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:14:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.876+0000 7f4af3001740 20 client.20010 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.876+0000 7f4af3001740 1 --2- 172.21.15.70:0/2322373744 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e046d1b720 0x55e046d88640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.876+0000 7f4af3001740 4 client.20010 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:14:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.876+0000 7f4af3001740 1 -- 172.21.15.70:0/2322373744 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f4ae0040040 con 0x55e046d1b720 2024-06-04T05:14:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.876+0000 7f4aed7a4640 1 --2- 172.21.15.70:0/2322373744 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e046d1b720 0x55e046d88640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.877+0000 7f4ad17fa640 10 client.20010 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.877+0000 7f4aed7a4640 1 --2- 172.21.15.70:0/2322373744 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e046d1b720 0x55e046d88640 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a88b916e23d04fd server_cookie=63cebd6824b0a6af in_seq=0 out_seq=0 2024-06-04T05:14:47.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.950+0000 7f4ad17fa640 1 -- 172.21.15.70:0/2322373744 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x7f4ae0040040 con 0x55e046d1b720 2024-06-04T05:14:47.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.950+0000 7f4ad17fa640 10 client.20010 handle_command_reply: tid=0 2024-06-04T05:14:47.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.950+0000 7f4af3001740 1 client.20010 shutdown 2024-06-04T05:14:47.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.951+0000 7f4af3001740 1 -- 172.21.15.70:0/2322373744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d00820 msgr2=0x55e046d05910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:47.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.951+0000 7f4af3001740 1 --2- 172.21.15.70:0/2322373744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d00820 0x55e046d05910 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f4ae003ca30 tx=0x7f4ae0031e50 comp rx=0 tx=0).stop 2024-06-04T05:14:47.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.951+0000 7f4af3001740 1 -- 172.21.15.70:0/2322373744 shutdown_connections 2024-06-04T05:14:47.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.951+0000 7f4af3001740 1 --2- 172.21.15.70:0/2322373744 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e046d1b720 0x55e046d88640 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.951+0000 7f4af3001740 1 --2- 172.21.15.70:0/2322373744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e046d01140 0x55e046d05e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.951+0000 7f4af3001740 1 --2- 172.21.15.70:0/2322373744 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e046d0ba00 0x55e046d06390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.951+0000 7f4af3001740 1 --2- 172.21.15.70:0/2322373744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e046d00820 0x55e046d05910 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.951+0000 7f4af3001740 1 -- 172.21.15.70:0/2322373744 >> 172.21.15.70:0/2322373744 conn(0x55e046cfcd30 msgr2=0x55e046d0aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:47.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.951+0000 7f4ad17fa640 0 client.20010 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:47.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.951+0000 7f4af3001740 1 -- 172.21.15.70:0/2322373744 shutdown_connections 2024-06-04T05:14:47.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.951+0000 7f4af3001740 1 -- 172.21.15.70:0/2322373744 wait complete. 2024-06-04T05:14:47.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.951+0000 7f4af3001740 20 client.20010 trim_cache size 0 max 16384 2024-06-04T05:14:47.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:14:47.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:14:47.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:14:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.952+0000 7f4af0209640 1 -- 172.21.15.70:0/3035544635 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4ad40b2a90 msgr2=0x7f4ad40b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.952+0000 7f4af0209640 1 --2- 172.21.15.70:0/3035544635 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4ad40b2a90 0x7f4ad40b4f50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x55e046d01dc0 tx=0x7f4ae40e66e0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.952+0000 7f4af0209640 1 -- 172.21.15.70:0/3035544635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae8074e80 msgr2=0x7f4ae819b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.952+0000 7f4af0209640 1 --2- 172.21.15.70:0/3035544635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae8074e80 0x7f4ae819b330 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4ae400eb90 tx=0x7f4ae400c6a0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.953+0000 7f4af0209640 1 -- 172.21.15.70:0/3035544635 shutdown_connections 2024-06-04T05:14:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.953+0000 7f4af0209640 1 --2- 172.21.15.70:0/3035544635 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4ad40b2a90 0x7f4ad40b4f50 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.953+0000 7f4af0209640 1 --2- 172.21.15.70:0/3035544635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ae810f270 0x7f4ae819fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.953+0000 7f4af0209640 1 --2- 172.21.15.70:0/3035544635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ae80757a0 0x7f4ae819b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.953+0000 7f4af0209640 1 --2- 172.21.15.70:0/3035544635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae8074e80 0x7f4ae819b330 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:47.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.953+0000 7f4af0209640 1 -- 172.21.15.70:0/3035544635 >> 172.21.15.70:0/3035544635 conn(0x7f4ae80fe500 msgr2=0x7f4ae80ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:47.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.953+0000 7f4af0209640 1 -- 172.21.15.70:0/3035544635 shutdown_connections 2024-06-04T05:14:47.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:14:47.953+0000 7f4af0209640 1 -- 172.21.15.70:0/3035544635 wait complete. 2024-06-04T05:14:48.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 31793 8.20 MB/sec execute 100 sec latency 82.213 ms 2024-06-04T05:14:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.962+0000 7ff93b6bf700 1 -- 172.21.15.70:0/592600550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff934103600 msgr2=0x7ff93410f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.962+0000 7ff93b6bf700 1 --2- 172.21.15.70:0/592600550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff934103600 0x7ff93410f2e0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7ff92c00b0a0 tx=0x7ff92c00b3b0 comp rx=0 tx=0).stop 2024-06-04T05:14:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.962+0000 7ff93b6bf700 1 -- 172.21.15.70:0/592600550 shutdown_connections 2024-06-04T05:14:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.962+0000 7ff93b6bf700 1 --2- 172.21.15.70:0/592600550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff934103600 0x7ff93410f2e0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.962+0000 7ff93b6bf700 1 --2- 172.21.15.70:0/592600550 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9341084c0 0x7ff9341088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.962+0000 7ff93b6bf700 1 --2- 172.21.15.70:0/592600550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff934102c40 0x7ff9341030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.962+0000 7ff93b6bf700 1 -- 172.21.15.70:0/592600550 >> 172.21.15.70:0/592600550 conn(0x7ff934078ab0 msgr2=0x7ff9340ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.962+0000 7ff93b6bf700 1 -- 172.21.15.70:0/592600550 shutdown_connections 2024-06-04T05:14:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.962+0000 7ff93b6bf700 1 -- 172.21.15.70:0/592600550 wait complete. 2024-06-04T05:14:48.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.963+0000 7ff93b6bf700 1 Processor -- start 2024-06-04T05:14:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.963+0000 7ff93b6bf700 1 -- start start 2024-06-04T05:14:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.963+0000 7ff93b6bf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff934102c40 0x7ff93419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.963+0000 7ff93b6bf700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff934103600 0x7ff93419ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.963+0000 7ff93b6bf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9341084c0 0x7ff9341a77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.963+0000 7ff93b6bf700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff934111410 con 0x7ff934102c40 2024-06-04T05:14:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.963+0000 7ff93b6bf700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff934102a20 con 0x7ff9341084c0 2024-06-04T05:14:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.963+0000 7ff93b6bf700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff934111590 con 0x7ff934103600 2024-06-04T05:14:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.963+0000 7ff93945b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff934102c40 0x7ff93419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.963+0000 7ff93945b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff934102c40 0x7ff93419c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43376/0 (socket says 172.21.15.70:43376) 2024-06-04T05:14:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.963+0000 7ff93945b700 1 -- 172.21.15.70:0/1536000562 learned_addr learned my addr 172.21.15.70:0/1536000562 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.963+0000 7ff939c5c700 1 --2- 172.21.15.70:0/1536000562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9341084c0 0x7ff9341a77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.963+0000 7ff938c5a700 1 --2- 172.21.15.70:0/1536000562 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff934103600 0x7ff93419ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.964+0000 7ff93945b700 1 -- 172.21.15.70:0/1536000562 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff934103600 msgr2=0x7ff93419ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.964+0000 7ff93945b700 1 --2- 172.21.15.70:0/1536000562 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff934103600 0x7ff93419ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.964+0000 7ff93945b700 1 -- 172.21.15.70:0/1536000562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9341084c0 msgr2=0x7ff9341a77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.964+0000 7ff93945b700 1 --2- 172.21.15.70:0/1536000562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9341084c0 0x7ff9341a77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.964+0000 7ff93945b700 1 -- 172.21.15.70:0/1536000562 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9341a7f80 con 0x7ff934102c40 2024-06-04T05:14:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.964+0000 7ff939c5c700 1 --2- 172.21.15.70:0/1536000562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9341084c0 0x7ff9341a77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.964+0000 7ff93945b700 1 --2- 172.21.15.70:0/1536000562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff934102c40 0x7ff93419c910 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7ff93000e970 tx=0x7ff93000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.964+0000 7ff92a7fc700 1 -- 172.21.15.70:0/1536000562 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93000cbd0 con 0x7ff934102c40 2024-06-04T05:14:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.964+0000 7ff92a7fc700 1 -- 172.21.15.70:0/1536000562 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff9300044b0 con 0x7ff934102c40 2024-06-04T05:14:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.964+0000 7ff93b6bf700 1 -- 172.21.15.70:0/1536000562 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9341a8210 con 0x7ff934102c40 2024-06-04T05:14:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.964+0000 7ff92a7fc700 1 -- 172.21.15.70:0/1536000562 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff930003c90 con 0x7ff934102c40 2024-06-04T05:14:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.965+0000 7ff93b6bf700 1 -- 172.21.15.70:0/1536000562 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9341a87a0 con 0x7ff934102c40 2024-06-04T05:14:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.968+0000 7ff92a7fc700 1 -- 172.21.15.70:0/1536000562 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff930003e50 con 0x7ff934102c40 2024-06-04T05:14:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.968+0000 7ff92a7fc700 1 --2- 172.21.15.70:0/1536000562 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9200b2a70 0x7ff9200b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.969+0000 7ff92a7fc700 1 -- 172.21.15.70:0/1536000562 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7ff9300e2930 con 0x7ff934102c40 2024-06-04T05:14:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.969+0000 7ff938c5a700 1 --2- 172.21.15.70:0/1536000562 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9200b2a70 0x7ff9200b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.969+0000 7ff938c5a700 1 --2- 172.21.15.70:0/1536000562 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9200b2a70 0x7ff9200b4f30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff934108c00 tx=0x7ff924009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:48.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.970+0000 7ff945f54b80 1 Processor -- start 2024-06-04T05:14:48.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.970+0000 7ff945f54b80 1 -- start start 2024-06-04T05:14:48.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.970+0000 7ff945f54b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf161b320 0x55ddf161b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:48.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.970+0000 7ff945f54b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf16158a0 0x55ddf1615d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:48.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.970+0000 7ff945f54b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf1616260 0x55ddf1622080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:48.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.971+0000 7ff93945b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf161b320 0x55ddf161b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:48.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.971+0000 7ff945f54b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ddf1573d70 con 0x55ddf1616260 2024-06-04T05:14:48.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.971+0000 7ff945f54b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ddf1563910 con 0x55ddf161b320 2024-06-04T05:14:48.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.971+0000 7ff945f54b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ddf1567950 con 0x55ddf16158a0 2024-06-04T05:14:48.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.971+0000 7ff938c5a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf16158a0 0x55ddf1615d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:48.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.971+0000 7ff938c5a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf16158a0 0x55ddf1615d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:37648/0 (socket says 172.21.15.70:37648) 2024-06-04T05:14:48.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.971+0000 7ff938c5a700 1 -- 172.21.15.70:0/3655667708 learned_addr learned my addr 172.21.15.70:0/3655667708 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:48.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.971+0000 7ff939c5c700 1 --2- 172.21.15.70:0/3655667708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf1616260 0x55ddf1622080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:48.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.971+0000 7ff939c5c700 1 -- 172.21.15.70:0/3655667708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf16158a0 msgr2=0x55ddf1615d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:48.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.971+0000 7ff939c5c700 1 --2- 172.21.15.70:0/3655667708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf16158a0 0x55ddf1615d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:48.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.971+0000 7ff939c5c700 1 -- 172.21.15.70:0/3655667708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf161b320 msgr2=0x55ddf161b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:48.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.971+0000 7ff939c5c700 1 --2- 172.21.15.70:0/3655667708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf161b320 0x55ddf161b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:48.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.971+0000 7ff939c5c700 1 -- 172.21.15.70:0/3655667708 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddf150cfb0 con 0x55ddf1616260 2024-06-04T05:14:48.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.971+0000 7ff939c5c700 1 --2- 172.21.15.70:0/3655667708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf1616260 0x55ddf1622080 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7ff92c00b070 tx=0x7ff92c009d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92500f63c3070088 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:48.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.971+0000 7ff91f7fe700 1 -- 172.21.15.70:0/3655667708 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92c021070 con 0x55ddf1616260 2024-06-04T05:14:48.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.972+0000 7ff91f7fe700 1 -- 172.21.15.70:0/3655667708 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff92c0074e0 con 0x55ddf1616260 2024-06-04T05:14:48.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.972+0000 7ff91f7fe700 1 -- 172.21.15.70:0/3655667708 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92c0052a0 con 0x55ddf1616260 2024-06-04T05:14:48.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.972+0000 7ff945f54b80 1 -- 172.21.15.70:0/3655667708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf1616260 msgr2=0x55ddf1622080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:48.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.972+0000 7ff945f54b80 1 --2- 172.21.15.70:0/3655667708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf1616260 0x55ddf1622080 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7ff92c00b070 tx=0x7ff92c009d90 comp rx=0 tx=0).stop 2024-06-04T05:14:48.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.973+0000 7ff945f54b80 1 -- 172.21.15.70:0/3655667708 shutdown_connections 2024-06-04T05:14:48.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.973+0000 7ff945f54b80 1 --2- 172.21.15.70:0/3655667708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf1616260 0x55ddf1622080 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:48.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.973+0000 7ff945f54b80 1 --2- 172.21.15.70:0/3655667708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf161b320 0x55ddf161b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:48.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.973+0000 7ff945f54b80 1 --2- 172.21.15.70:0/3655667708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf16158a0 0x55ddf1615d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:48.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.973+0000 7ff945f54b80 1 -- 172.21.15.70:0/3655667708 >> 172.21.15.70:0/3655667708 conn(0x55ddf1541240 msgr2=0x55ddf16103e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:48.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.973+0000 7ff945f54b80 1 -- 172.21.15.70:0/3655667708 shutdown_connections 2024-06-04T05:14:48.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.973+0000 7ff945f54b80 1 -- 172.21.15.70:0/3655667708 wait complete. 2024-06-04T05:14:48.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.973+0000 7ff945f54b80 1 -- start start 2024-06-04T05:14:48.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.973+0000 7ff945f54b80 1 Processor -- start 2024-06-04T05:14:48.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff945f54b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf16158a0 0x55ddf1621cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:48.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff945f54b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf1616260 0x55ddf161efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:48.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff945f54b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf161b320 0x55ddf161f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:48.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff945f54b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ddf1469fa0 con 0x55ddf161b320 2024-06-04T05:14:48.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff945f54b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ddf14cbda0 con 0x55ddf1616260 2024-06-04T05:14:48.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff945f54b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ddf16226d0 con 0x55ddf16158a0 2024-06-04T05:14:48.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff945f54b80 10 client.0 _ll_register_callbacks cb 0x55ddf14fe950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:48.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff939c5c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf161b320 0x55ddf161f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:48.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff93945b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf16158a0 0x55ddf1621cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:48.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff939c5c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf161b320 0x55ddf161f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43390/0 (socket says 172.21.15.70:43390) 2024-06-04T05:14:48.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff93945b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf16158a0 0x55ddf1621cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:37654/0 (socket says 172.21.15.70:37654) 2024-06-04T05:14:48.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff939c5c700 1 -- 172.21.15.70:0/540378813 learned_addr learned my addr 172.21.15.70:0/540378813 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:48.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff939c5c700 1 -- 172.21.15.70:0/540378813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf16158a0 msgr2=0x55ddf1621cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:48.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff939c5c700 1 --2- 172.21.15.70:0/540378813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf16158a0 0x55ddf1621cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:48.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff939c5c700 1 -- 172.21.15.70:0/540378813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf1616260 msgr2=0x55ddf161efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:48.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff939c5c700 1 --2- 172.21.15.70:0/540378813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf1616260 0x55ddf161efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:48.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff939c5c700 1 -- 172.21.15.70:0/540378813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddf14b38d0 con 0x55ddf161b320 2024-06-04T05:14:48.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff939c5c700 1 --2- 172.21.15.70:0/540378813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf161b320 0x55ddf161f510 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7ff92c00d010 tx=0x7ff92c003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a208510117df2aa server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:48.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.974+0000 7ff91cff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:14:48.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.975+0000 7ff91cff9700 1 -- 172.21.15.70:0/540378813 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92c02b020 con 0x55ddf161b320 2024-06-04T05:14:48.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.975+0000 7ff945f54b80 1 -- 172.21.15.70:0/540378813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff92c0074e0 con 0x55ddf161b320 2024-06-04T05:14:48.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.975+0000 7ff91cff9700 1 -- 172.21.15.70:0/540378813 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7ff92c007aa0 con 0x55ddf161b320 2024-06-04T05:14:48.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.975+0000 7ff91cff9700 1 -- 172.21.15.70:0/540378813 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7ff92c036050 con 0x55ddf161b320 2024-06-04T05:14:48.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.975+0000 7ff945f54b80 10 client.20022 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:48.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.975+0000 7ff945f54b80 1 -- 172.21.15.70:0/540378813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55ddf151d760 con 0x55ddf161b320 2024-06-04T05:14:48.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.975+0000 7ff91cff9700 1 -- 172.21.15.70:0/540378813 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff92c0074e0 con 0x55ddf161b320 2024-06-04T05:14:48.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.975+0000 7ff91cff9700 1 -- 172.21.15.70:0/540378813 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7ff92c011070 con 0x55ddf161b320 2024-06-04T05:14:48.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.975+0000 7ff945f54b80 10 client.20022 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:48.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.976+0000 7ff945f54b80 10 client.20022 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:14:48.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.976+0000 7ff945f54b80 20 client.20022 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:48.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.976+0000 7ff945f54b80 1 --2- 172.21.15.70:0/540378813 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ddf162f050 0x55ddf1631440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:48.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.976+0000 7ff945f54b80 4 client.20022 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:14:48.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.976+0000 7ff945f54b80 1 -- 172.21.15.70:0/540378813 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddf1575cc0 con 0x55ddf162f050 2024-06-04T05:14:48.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.976+0000 7ff93945b700 1 --2- 172.21.15.70:0/540378813 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ddf162f050 0x55ddf1631440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:48.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.976+0000 7ff91cff9700 10 client.20022 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:48.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.976+0000 7ff93945b700 1 --2- 172.21.15.70:0/540378813 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ddf162f050 0x55ddf1631440 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=502474b252426be7 server_cookie=640de3ad87874787 in_seq=0 out_seq=0 2024-06-04T05:14:48.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.977+0000 7ff91cff9700 1 -- 172.21.15.70:0/540378813 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ddf1575cc0 con 0x55ddf162f050 2024-06-04T05:14:48.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.977+0000 7ff91cff9700 10 client.20022 handle_command_reply: tid=0 2024-06-04T05:14:48.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.977+0000 7ff945f54b80 1 client.20022 shutdown 2024-06-04T05:14:48.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.977+0000 7ff945f54b80 1 -- 172.21.15.70:0/540378813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf161b320 msgr2=0x55ddf161f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:48.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.977+0000 7ff945f54b80 1 --2- 172.21.15.70:0/540378813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf161b320 0x55ddf161f510 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7ff92c00d010 tx=0x7ff92c003d70 comp rx=0 tx=0).stop 2024-06-04T05:14:48.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.977+0000 7ff945f54b80 1 -- 172.21.15.70:0/540378813 shutdown_connections 2024-06-04T05:14:48.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.977+0000 7ff945f54b80 1 --2- 172.21.15.70:0/540378813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf161b320 0x55ddf161f510 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:48.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.977+0000 7ff945f54b80 1 --2- 172.21.15.70:0/540378813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf16158a0 0x55ddf1621cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:48.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.977+0000 7ff945f54b80 1 --2- 172.21.15.70:0/540378813 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ddf162f050 0x55ddf1631440 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:48.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.977+0000 7ff91cff9700 0 client.20022 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:48.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.977+0000 7ff945f54b80 1 --2- 172.21.15.70:0/540378813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf1616260 0x55ddf161efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:48.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.977+0000 7ff945f54b80 1 -- 172.21.15.70:0/540378813 >> 172.21.15.70:0/540378813 conn(0x55ddf1541240 msgr2=0x55ddf160faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:48.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.977+0000 7ff945f54b80 1 -- 172.21.15.70:0/540378813 shutdown_connections 2024-06-04T05:14:48.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.977+0000 7ff945f54b80 1 -- 172.21.15.70:0/540378813 wait complete. 2024-06-04T05:14:48.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:48.977+0000 7ff945f54b80 20 client.20022 trim_cache size 0 max 16384 2024-06-04T05:14:49.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.002+0000 7ff945f54b80 1 Processor -- start 2024-06-04T05:14:49.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.002+0000 7ff945f54b80 1 -- start start 2024-06-04T05:14:49.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.002+0000 7ff945f54b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf16158a0 0x55ddf1619460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:49.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.002+0000 7ff945f54b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf1616260 0x55ddf1610270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:49.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.002+0000 7ff945f54b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf161b320 0x55ddf1634460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:49.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.002+0000 7ff945f54b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9341abb40 con 0x55ddf161b320 2024-06-04T05:14:49.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.002+0000 7ff945f54b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff9341ad6e0 con 0x55ddf16158a0 2024-06-04T05:14:49.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.002+0000 7ff945f54b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff9341ac270 con 0x55ddf1616260 2024-06-04T05:14:49.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.002+0000 7ff93945b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf16158a0 0x55ddf1619460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:49.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.002+0000 7ff939c5c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf161b320 0x55ddf1634460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:49.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.002+0000 7ff93945b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf16158a0 0x55ddf1619460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:48906/0 (socket says 172.21.15.70:48906) 2024-06-04T05:14:49.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.002+0000 7ff93945b700 1 -- 172.21.15.70:0/700526357 learned_addr learned my addr 172.21.15.70:0/700526357 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:49.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.002+0000 7ff938c5a700 1 --2- 172.21.15.70:0/700526357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf1616260 0x55ddf1610270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:49.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.003+0000 7ff93945b700 1 -- 172.21.15.70:0/700526357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf1616260 msgr2=0x55ddf1610270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:49.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.003+0000 7ff93945b700 1 --2- 172.21.15.70:0/700526357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf1616260 0x55ddf1610270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:49.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.003+0000 7ff93945b700 1 -- 172.21.15.70:0/700526357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf161b320 msgr2=0x55ddf1634460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:49.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.003+0000 7ff93945b700 1 --2- 172.21.15.70:0/700526357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf161b320 0x55ddf1634460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:49.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.003+0000 7ff93945b700 1 -- 172.21.15.70:0/700526357 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddf161baf0 con 0x55ddf16158a0 2024-06-04T05:14:49.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.003+0000 7ff939c5c700 1 --2- 172.21.15.70:0/700526357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf161b320 0x55ddf1634460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:49.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.003+0000 7ff938c5a700 1 --2- 172.21.15.70:0/700526357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf1616260 0x55ddf1610270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:14:49.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.003+0000 7ff93945b700 1 --2- 172.21.15.70:0/700526357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf16158a0 0x55ddf1619460 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff930003730 tx=0x7ff9300a5a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b2489766ae9ba06 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:49.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.003+0000 7ff91effd700 1 -- 172.21.15.70:0/700526357 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9300a9690 con 0x55ddf16158a0 2024-06-04T05:14:49.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.003+0000 7ff91effd700 1 -- 172.21.15.70:0/700526357 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff9300aa280 con 0x55ddf16158a0 2024-06-04T05:14:49.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.003+0000 7ff91effd700 1 -- 172.21.15.70:0/700526357 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93000ce00 con 0x55ddf16158a0 2024-06-04T05:14:49.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.004+0000 7ff945f54b80 1 -- 172.21.15.70:0/700526357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf16158a0 msgr2=0x55ddf1619460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:49.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.004+0000 7ff945f54b80 1 --2- 172.21.15.70:0/700526357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf16158a0 0x55ddf1619460 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff930003730 tx=0x7ff9300a5a40 comp rx=0 tx=0).stop 2024-06-04T05:14:49.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.004+0000 7ff945f54b80 1 -- 172.21.15.70:0/700526357 shutdown_connections 2024-06-04T05:14:49.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.004+0000 7ff945f54b80 1 --2- 172.21.15.70:0/700526357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf161b320 0x55ddf1634460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:49.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.004+0000 7ff945f54b80 1 --2- 172.21.15.70:0/700526357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf16158a0 0x55ddf1619460 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:49.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.004+0000 7ff945f54b80 1 --2- 172.21.15.70:0/700526357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf1616260 0x55ddf1610270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:49.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.004+0000 7ff945f54b80 1 -- 172.21.15.70:0/700526357 >> 172.21.15.70:0/700526357 conn(0x55ddf1541240 msgr2=0x55ddf1618d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:49.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.004+0000 7ff945f54b80 1 -- 172.21.15.70:0/700526357 shutdown_connections 2024-06-04T05:14:49.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.004+0000 7ff945f54b80 1 -- 172.21.15.70:0/700526357 wait complete. 2024-06-04T05:14:49.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.005+0000 7ff945f54b80 1 -- start start 2024-06-04T05:14:49.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.005+0000 7ff945f54b80 1 Processor -- start 2024-06-04T05:14:49.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.005+0000 7ff945f54b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf16158a0 0x55ddf1634ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:49.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.005+0000 7ff945f54b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf1616260 0x55ddf1631e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:49.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.005+0000 7ff945f54b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf161b320 0x55ddf1632390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:49.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.005+0000 7ff945f54b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9341ad010 con 0x55ddf16158a0 2024-06-04T05:14:49.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.005+0000 7ff945f54b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff9341ac940 con 0x55ddf161b320 2024-06-04T05:14:49.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.005+0000 7ff945f54b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff934117850 con 0x55ddf1616260 2024-06-04T05:14:49.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.005+0000 7ff945f54b80 10 client.0 _ll_register_callbacks cb 0x55ddf14fe950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:49.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.005+0000 7ff93945b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf16158a0 0x55ddf1634ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:49.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.006+0000 7ff938c5a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf1616260 0x55ddf1631e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:49.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.006+0000 7ff93945b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf16158a0 0x55ddf1634ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43400/0 (socket says 172.21.15.70:43400) 2024-06-04T05:14:49.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.006+0000 7ff93945b700 1 -- 172.21.15.70:0/2380750549 learned_addr learned my addr 172.21.15.70:0/2380750549 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:49.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.006+0000 7ff939c5c700 1 --2- 172.21.15.70:0/2380750549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf161b320 0x55ddf1632390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:49.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.006+0000 7ff93945b700 1 -- 172.21.15.70:0/2380750549 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf1616260 msgr2=0x55ddf1631e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:49.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.006+0000 7ff93945b700 1 --2- 172.21.15.70:0/2380750549 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf1616260 0x55ddf1631e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.006+0000 7ff93945b700 1 -- 172.21.15.70:0/2380750549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf161b320 msgr2=0x55ddf1632390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.006+0000 7ff93945b700 1 --2- 172.21.15.70:0/2380750549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf161b320 0x55ddf1632390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.006+0000 7ff93945b700 1 -- 172.21.15.70:0/2380750549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddf1632bd0 con 0x55ddf16158a0 2024-06-04T05:14:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.006+0000 7ff93945b700 1 --2- 172.21.15.70:0/2380750549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf16158a0 0x55ddf1634ab0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7ff930000c00 tx=0x7ff9300a5440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0bcecf643e47a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.006+0000 7ff91cff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:14:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.006+0000 7ff91cff9700 1 -- 172.21.15.70:0/2380750549 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9300c9060 con 0x55ddf16158a0 2024-06-04T05:14:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.006+0000 7ff945f54b80 1 -- 172.21.15.70:0/2380750549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9300aa280 con 0x55ddf16158a0 2024-06-04T05:14:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.006+0000 7ff91cff9700 1 -- 172.21.15.70:0/2380750549 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7ff930004140 con 0x55ddf16158a0 2024-06-04T05:14:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.007+0000 7ff91cff9700 1 -- 172.21.15.70:0/2380750549 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7ff9300adf30 con 0x55ddf16158a0 2024-06-04T05:14:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.007+0000 7ff945f54b80 10 client.20028 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.007+0000 7ff945f54b80 1 -- 172.21.15.70:0/2380750549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55ddf1699840 con 0x55ddf16158a0 2024-06-04T05:14:49.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.007+0000 7ff91cff9700 1 -- 172.21.15.70:0/2380750549 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff9300aa280 con 0x55ddf16158a0 2024-06-04T05:14:49.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.007+0000 7ff91cff9700 1 -- 172.21.15.70:0/2380750549 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7ff930016330 con 0x55ddf16158a0 2024-06-04T05:14:49.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.007+0000 7ff945f54b80 10 client.20028 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:49.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.007+0000 7ff945f54b80 10 client.20028 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:14:49.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.007+0000 7ff945f54b80 20 client.20028 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:49.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.007+0000 7ff945f54b80 1 --2- 172.21.15.70:0/2380750549 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ddf1699df0 0x55ddf169c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:49.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.007+0000 7ff945f54b80 4 client.20028 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:14:49.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.007+0000 7ff945f54b80 1 -- 172.21.15.70:0/2380750549 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ddf156f4a0 con 0x55ddf1699df0 2024-06-04T05:14:49.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.007+0000 7ff938c5a700 1 --2- 172.21.15.70:0/2380750549 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ddf1699df0 0x55ddf169c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:49.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.008+0000 7ff938c5a700 1 --2- 172.21.15.70:0/2380750549 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ddf1699df0 0x55ddf169c1e0 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca23e9aecef937c6 server_cookie=86cead143a51999a in_seq=0 out_seq=0 2024-06-04T05:14:49.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.008+0000 7ff91cff9700 10 client.20028 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:49.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.009+0000 7ff91cff9700 1 -- 172.21.15.70:0/2380750549 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ddf156f4a0 con 0x55ddf1699df0 2024-06-04T05:14:49.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.009+0000 7ff91cff9700 10 client.20028 handle_command_reply: tid=0 2024-06-04T05:14:49.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.009+0000 7ff945f54b80 1 client.20028 shutdown 2024-06-04T05:14:49.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.009+0000 7ff945f54b80 1 -- 172.21.15.70:0/2380750549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf16158a0 msgr2=0x55ddf1634ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:49.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.009+0000 7ff945f54b80 1 --2- 172.21.15.70:0/2380750549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf16158a0 0x55ddf1634ab0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7ff930000c00 tx=0x7ff9300a5440 comp rx=0 tx=0).stop 2024-06-04T05:14:49.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.009+0000 7ff945f54b80 1 -- 172.21.15.70:0/2380750549 shutdown_connections 2024-06-04T05:14:49.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.009+0000 7ff945f54b80 1 --2- 172.21.15.70:0/2380750549 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ddf1699df0 0x55ddf169c1e0 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:49.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.009+0000 7ff945f54b80 1 --2- 172.21.15.70:0/2380750549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddf161b320 0x55ddf1632390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:49.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.009+0000 7ff945f54b80 1 --2- 172.21.15.70:0/2380750549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddf16158a0 0x55ddf1634ab0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:49.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.009+0000 7ff945f54b80 1 --2- 172.21.15.70:0/2380750549 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddf1616260 0x55ddf1631e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:49.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.009+0000 7ff945f54b80 1 -- 172.21.15.70:0/2380750549 >> 172.21.15.70:0/2380750549 conn(0x55ddf1541240 msgr2=0x55ddf1614890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:49.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.009+0000 7ff91cff9700 0 client.20028 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:49.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.009+0000 7ff945f54b80 1 -- 172.21.15.70:0/2380750549 shutdown_connections 2024-06-04T05:14:49.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.009+0000 7ff945f54b80 1 -- 172.21.15.70:0/2380750549 wait complete. 2024-06-04T05:14:49.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.010+0000 7ff945f54b80 20 client.20028 trim_cache size 0 max 16384 2024-06-04T05:14:49.015 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:14:49.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.010+0000 7ff93b6bf700 1 -- 172.21.15.70:0/1536000562 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9200b2a70 msgr2=0x7ff9200b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:49.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.010+0000 7ff93b6bf700 1 --2- 172.21.15.70:0/1536000562 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9200b2a70 0x7ff9200b4f30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff934108c00 tx=0x7ff924009260 comp rx=0 tx=0).stop 2024-06-04T05:14:49.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.010+0000 7ff93b6bf700 1 -- 172.21.15.70:0/1536000562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff934102c40 msgr2=0x7ff93419c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:49.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.010+0000 7ff93b6bf700 1 --2- 172.21.15.70:0/1536000562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff934102c40 0x7ff93419c910 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7ff93000e970 tx=0x7ff93000ec80 comp rx=0 tx=0).stop 2024-06-04T05:14:49.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.010+0000 7ff93b6bf700 1 -- 172.21.15.70:0/1536000562 shutdown_connections 2024-06-04T05:14:49.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.010+0000 7ff93b6bf700 1 --2- 172.21.15.70:0/1536000562 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff9200b2a70 0x7ff9200b4f30 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:49.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.010+0000 7ff93b6bf700 1 --2- 172.21.15.70:0/1536000562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9341084c0 0x7ff9341a77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:49.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.010+0000 7ff93b6bf700 1 --2- 172.21.15.70:0/1536000562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff934102c40 0x7ff93419c910 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:49.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.010+0000 7ff93b6bf700 1 --2- 172.21.15.70:0/1536000562 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff934103600 0x7ff93419ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:49.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.010+0000 7ff93b6bf700 1 -- 172.21.15.70:0/1536000562 >> 172.21.15.70:0/1536000562 conn(0x7ff934078ab0 msgr2=0x7ff934100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:49.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.011+0000 7ff93b6bf700 1 -- 172.21.15.70:0/1536000562 shutdown_connections 2024-06-04T05:14:49.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:49.011+0000 7ff93b6bf700 1 -- 172.21.15.70:0/1536000562 wait complete. 2024-06-04T05:14:49.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:49 smithi070 ceph-mon[32631]: pgmap v552: 97 pgs: 97 active+clean; 567 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.7 MiB/s wr, 243 op/s 2024-06-04T05:14:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:49 smithi112 ceph-mon[36453]: pgmap v552: 97 pgs: 97 active+clean; 567 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.7 MiB/s wr, 243 op/s 2024-06-04T05:14:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:49 smithi098 ceph-mon[35569]: pgmap v552: 97 pgs: 97 active+clean; 567 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.7 MiB/s wr, 243 op/s 2024-06-04T05:14:49.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 31972 8.21 MB/sec execute 101 sec latency 239.913 ms 2024-06-04T05:14:50.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 32152 8.20 MB/sec execute 102 sec latency 55.296 ms 2024-06-04T05:14:50.991 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:14:50.992 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:14:51.341 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:14:51.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:51 smithi070 ceph-mon[32631]: pgmap v553: 97 pgs: 97 active+clean; 567 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.7 MiB/s wr, 243 op/s 2024-06-04T05:14:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:51 smithi112 ceph-mon[36453]: pgmap v553: 97 pgs: 97 active+clean; 567 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.7 MiB/s wr, 243 op/s 2024-06-04T05:14:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:51 smithi098 ceph-mon[35569]: pgmap v553: 97 pgs: 97 active+clean; 567 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.7 MiB/s wr, 243 op/s 2024-06-04T05:14:51.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 32335 8.25 MB/sec execute 103 sec latency 49.842 ms 2024-06-04T05:14:52.568 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-06-04T05:14:52.568 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:14:52.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 32526 8.26 MB/sec execute 104 sec latency 60.276 ms 2024-06-04T05:14:53.265 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:14:53.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.273+0000 7f384718a700 1 -- 172.21.15.70:0/2621720879 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3840102e30 msgr2=0x7f384010f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:53.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.273+0000 7f384718a700 1 --2- 172.21.15.70:0/2621720879 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3840102e30 0x7f384010f320 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3838009f90 tx=0x7f383800b330 comp rx=0 tx=0).stop 2024-06-04T05:14:53.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.273+0000 7f384718a700 1 -- 172.21.15.70:0/2621720879 shutdown_connections 2024-06-04T05:14:53.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.273+0000 7f384718a700 1 --2- 172.21.15.70:0/2621720879 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3840102e30 0x7f384010f320 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.273+0000 7f384718a700 1 --2- 172.21.15.70:0/2621720879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3840108550 0x7f3840108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.273+0000 7f384718a700 1 --2- 172.21.15.70:0/2621720879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3840102470 0x7f38401028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.273+0000 7f384718a700 1 -- 172.21.15.70:0/2621720879 >> 172.21.15.70:0/2621720879 conn(0x7f3840079480 msgr2=0x7f3840079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:53.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.273+0000 7f384718a700 1 -- 172.21.15.70:0/2621720879 shutdown_connections 2024-06-04T05:14:53.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.273+0000 7f384718a700 1 -- 172.21.15.70:0/2621720879 wait complete. 2024-06-04T05:14:53.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.274+0000 7f384718a700 1 Processor -- start 2024-06-04T05:14:53.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.274+0000 7f384718a700 1 -- start start 2024-06-04T05:14:53.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.274+0000 7f384718a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3840102470 0x7f384019bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.275+0000 7f384718a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3840102e30 0x7f384019c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.275+0000 7f384718a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3840108550 0x7f38401a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.275+0000 7f384718a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3840111450 con 0x7f3840102e30 2024-06-04T05:14:53.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.275+0000 7f384718a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f38401112d0 con 0x7f3840102470 2024-06-04T05:14:53.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.275+0000 7f384718a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f38401115d0 con 0x7f3840108550 2024-06-04T05:14:53.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.275+0000 7f3844f26700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3840102470 0x7f384019bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.275+0000 7f383ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3840102e30 0x7f384019c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.275+0000 7f3844f26700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3840102470 0x7f384019bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:48938/0 (socket says 172.21.15.70:48938) 2024-06-04T05:14:53.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.275+0000 7f3844f26700 1 -- 172.21.15.70:0/366322446 learned_addr learned my addr 172.21.15.70:0/366322446 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:53.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.275+0000 7f3845727700 1 --2- 172.21.15.70:0/366322446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3840108550 0x7f38401a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.275+0000 7f3844f26700 1 -- 172.21.15.70:0/366322446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3840108550 msgr2=0x7f38401a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:53.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.275+0000 7f3844f26700 1 --2- 172.21.15.70:0/366322446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3840108550 0x7f38401a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.275+0000 7f3844f26700 1 -- 172.21.15.70:0/366322446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3840102e30 msgr2=0x7f384019c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:53.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.275+0000 7f3844f26700 1 --2- 172.21.15.70:0/366322446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3840102e30 0x7f384019c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.275+0000 7f3844f26700 1 -- 172.21.15.70:0/366322446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38401a0d80 con 0x7f3840102470 2024-06-04T05:14:53.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.275+0000 7f3844f26700 1 --2- 172.21.15.70:0/366322446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3840102470 0x7f384019bd70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f383400e920 tx=0x7f383400ec30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:53.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.276+0000 7f383dffb700 1 -- 172.21.15.70:0/366322446 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383400c9c0 con 0x7f3840102470 2024-06-04T05:14:53.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.276+0000 7f384718a700 1 -- 172.21.15.70:0/366322446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38401a1070 con 0x7f3840102470 2024-06-04T05:14:53.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.276+0000 7f383dffb700 1 -- 172.21.15.70:0/366322446 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3834010430 con 0x7f3840102470 2024-06-04T05:14:53.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.276+0000 7f383dffb700 1 -- 172.21.15.70:0/366322446 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3834018680 con 0x7f3840102470 2024-06-04T05:14:53.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.276+0000 7f384718a700 1 -- 172.21.15.70:0/366322446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38401a88f0 con 0x7f3840102470 2024-06-04T05:14:53.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.279+0000 7f383dffb700 1 -- 172.21.15.70:0/366322446 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3834018820 con 0x7f3840102470 2024-06-04T05:14:53.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.280+0000 7f3851a1fb80 1 Processor -- start 2024-06-04T05:14:53.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.280+0000 7f3851a1fb80 1 -- start start 2024-06-04T05:14:53.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.280+0000 7f3851a1fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead4b00 0x5590dead4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.280+0000 7f3851a1fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 0x5590dead8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.280+0000 7f3851a1fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead95f0 0x5590dead5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.280+0000 7f3851a1fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5590dea32df0 con 0x5590dead4b00 2024-06-04T05:14:53.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.280+0000 7f3851a1fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5590dea22990 con 0x5590dead95f0 2024-06-04T05:14:53.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.280+0000 7f3851a1fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5590dea269d0 con 0x5590dead54b0 2024-06-04T05:14:53.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.281+0000 7f383dffb700 1 --2- 172.21.15.70:0/366322446 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38280b2a20 0x7f38280b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.281+0000 7f3844f26700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 0x5590dead8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.281+0000 7f3845727700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead4b00 0x5590dead4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.281+0000 7f3844f26700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 0x5590dead8fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:37710/0 (socket says 172.21.15.70:37710) 2024-06-04T05:14:53.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.281+0000 7f3844f26700 1 -- 172.21.15.70:0/4056699731 learned_addr learned my addr 172.21.15.70:0/4056699731 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:53.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.281+0000 7f383ffff700 1 --2- 172.21.15.70:0/4056699731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead95f0 0x5590dead5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.281+0000 7f3844f26700 1 -- 172.21.15.70:0/4056699731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead95f0 msgr2=0x5590dead5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:53.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.281+0000 7f3844f26700 1 --2- 172.21.15.70:0/4056699731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead95f0 0x5590dead5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.281+0000 7f3844f26700 1 -- 172.21.15.70:0/4056699731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead4b00 msgr2=0x5590dead4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:53.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.281+0000 7f3844f26700 1 --2- 172.21.15.70:0/4056699731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead4b00 0x5590dead4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.281+0000 7f3844f26700 1 -- 172.21.15.70:0/4056699731 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590de9cc030 con 0x5590dead54b0 2024-06-04T05:14:53.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.281+0000 7f3844f26700 1 --2- 172.21.15.70:0/4056699731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 0x5590dead8fd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f383406c760 tx=0x7f383406c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6b39d56997176b0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:53.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.281+0000 7f3826ffd700 1 -- 172.21.15.70:0/4056699731 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38340a3e80 con 0x5590dead54b0 2024-06-04T05:14:53.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.281+0000 7f3845727700 1 --2- 172.21.15.70:0/4056699731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead4b00 0x5590dead4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:14:53.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.281+0000 7f3826ffd700 1 -- 172.21.15.70:0/4056699731 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f38340a99e0 con 0x5590dead54b0 2024-06-04T05:14:53.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.281+0000 7f3826ffd700 1 -- 172.21.15.70:0/4056699731 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38340a9ea0 con 0x5590dead54b0 2024-06-04T05:14:53.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.281+0000 7f3845727700 1 --2- 172.21.15.70:0/366322446 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38280b2a20 0x7f38280b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.282+0000 7f383dffb700 1 -- 172.21.15.70:0/366322446 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f38340e2730 con 0x7f3840102470 2024-06-04T05:14:53.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.282+0000 7f3851a1fb80 1 -- 172.21.15.70:0/4056699731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 msgr2=0x5590dead8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:53.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.282+0000 7f3845727700 1 --2- 172.21.15.70:0/366322446 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38280b2a20 0x7f38280b4ee0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3838009da0 tx=0x7f3838009b20 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:53.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.282+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/4056699731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 0x5590dead8fd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f383406c760 tx=0x7f383406c000 comp rx=0 tx=0).stop 2024-06-04T05:14:53.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.282+0000 7f3851a1fb80 1 -- 172.21.15.70:0/4056699731 shutdown_connections 2024-06-04T05:14:53.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.282+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/4056699731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead95f0 0x5590dead5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.282+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/4056699731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead4b00 0x5590dead4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.282+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/4056699731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 0x5590dead8fd0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.282+0000 7f3851a1fb80 1 -- 172.21.15.70:0/4056699731 >> 172.21.15.70:0/4056699731 conn(0x5590dea002c0 msgr2=0x5590deacf510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:53.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.283+0000 7f3851a1fb80 1 -- 172.21.15.70:0/4056699731 shutdown_connections 2024-06-04T05:14:53.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.283+0000 7f3851a1fb80 1 -- 172.21.15.70:0/4056699731 wait complete. 2024-06-04T05:14:53.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.283+0000 7f3851a1fb80 1 -- start start 2024-06-04T05:14:53.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.283+0000 7f3851a1fb80 1 Processor -- start 2024-06-04T05:14:53.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f3851a1fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead4b00 0x5590deaf5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f3851a1fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 0x5590deaf5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f3851a1fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead95f0 0x5590deaef520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f3851a1fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5590de929020 con 0x5590dead95f0 2024-06-04T05:14:53.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f3851a1fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5590de98ae20 con 0x5590dead4b00 2024-06-04T05:14:53.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f3851a1fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5590dead6500 con 0x5590dead54b0 2024-06-04T05:14:53.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f3851a1fb80 10 client.0 _ll_register_callbacks cb 0x5590de9bd9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:53.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f3844f26700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead4b00 0x5590deaf5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f383ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 0x5590deaf5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f3844f26700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead4b00 0x5590deaf5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:48944/0 (socket says 172.21.15.70:48944) 2024-06-04T05:14:53.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f3844f26700 1 -- 172.21.15.70:0/528102838 learned_addr learned my addr 172.21.15.70:0/528102838 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:53.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f3845727700 1 --2- 172.21.15.70:0/528102838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead95f0 0x5590deaef520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f383ffff700 1 -- 172.21.15.70:0/528102838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead4b00 msgr2=0x5590deaf5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:53.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f383ffff700 1 --2- 172.21.15.70:0/528102838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead4b00 0x5590deaf5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f383ffff700 1 -- 172.21.15.70:0/528102838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead95f0 msgr2=0x5590deaef520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:53.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f383ffff700 1 --2- 172.21.15.70:0/528102838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead95f0 0x5590deaef520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f383ffff700 1 -- 172.21.15.70:0/528102838 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590de972950 con 0x5590dead54b0 2024-06-04T05:14:53.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f383ffff700 1 --2- 172.21.15.70:0/528102838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 0x5590deaf5690 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3830005fd0 tx=0x7f38300129b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d67e13098432c9c0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:53.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.284+0000 7f381ffff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:14:53.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.285+0000 7f381ffff700 1 -- 172.21.15.70:0/528102838 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3830004050 con 0x5590dead54b0 2024-06-04T05:14:53.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.285+0000 7f381ffff700 1 -- 172.21.15.70:0/528102838 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f38300041f0 con 0x5590dead54b0 2024-06-04T05:14:53.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.285+0000 7f3851a1fb80 1 -- 172.21.15.70:0/528102838 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38340a99e0 con 0x5590dead54b0 2024-06-04T05:14:53.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.285+0000 7f381ffff700 1 -- 172.21.15.70:0/528102838 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f3830004430 con 0x5590dead54b0 2024-06-04T05:14:53.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.285+0000 7f3851a1fb80 10 client.28972 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:53.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.285+0000 7f3851a1fb80 1 -- 172.21.15.70:0/528102838 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5590de9dc7e0 con 0x5590dead54b0 2024-06-04T05:14:53.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.285+0000 7f381ffff700 1 -- 172.21.15.70:0/528102838 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f38340a99e0 con 0x5590dead54b0 2024-06-04T05:14:53.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.286+0000 7f381ffff700 1 -- 172.21.15.70:0/528102838 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f3830023070 con 0x5590dead54b0 2024-06-04T05:14:53.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.286+0000 7f3851a1fb80 10 client.28972 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:53.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.286+0000 7f3851a1fb80 10 client.28972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:14:53.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.286+0000 7f3851a1fb80 20 client.28972 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:53.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.286+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/528102838 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5590deaff0e0 0x5590deb014d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.286+0000 7f3851a1fb80 4 client.28972 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:14:53.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.286+0000 7f3851a1fb80 1 -- 172.21.15.70:0/528102838 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590dea34d40 con 0x5590deaff0e0 2024-06-04T05:14:53.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.286+0000 7f3844f26700 1 --2- 172.21.15.70:0/528102838 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5590deaff0e0 0x5590deb014d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.286+0000 7f381ffff700 10 client.28972 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:53.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.286+0000 7f3844f26700 1 --2- 172.21.15.70:0/528102838 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5590deaff0e0 0x5590deb014d0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6af13776591d61b2 server_cookie=1a015ac5b5d7eb90 in_seq=0 out_seq=0 2024-06-04T05:14:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.288+0000 7f381ffff700 1 -- 172.21.15.70:0/528102838 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5590dea34d40 con 0x5590deaff0e0 2024-06-04T05:14:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.288+0000 7f381ffff700 10 client.28972 handle_command_reply: tid=0 2024-06-04T05:14:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.288+0000 7f3851a1fb80 1 client.28972 shutdown 2024-06-04T05:14:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.288+0000 7f3851a1fb80 1 -- 172.21.15.70:0/528102838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 msgr2=0x5590deaf5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.288+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/528102838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 0x5590deaf5690 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3830005fd0 tx=0x7f38300129b0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.288+0000 7f3851a1fb80 1 -- 172.21.15.70:0/528102838 shutdown_connections 2024-06-04T05:14:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.288+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/528102838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead95f0 0x5590deaef520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.288+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/528102838 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5590deaff0e0 0x5590deb014d0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.288+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/528102838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead4b00 0x5590deaf5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.288+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/528102838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 0x5590deaf5690 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.288+0000 7f3851a1fb80 1 -- 172.21.15.70:0/528102838 >> 172.21.15.70:0/528102838 conn(0x5590dea002c0 msgr2=0x5590deada940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.288+0000 7f381ffff700 0 client.28972 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.289+0000 7f3851a1fb80 1 -- 172.21.15.70:0/528102838 shutdown_connections 2024-06-04T05:14:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.289+0000 7f3851a1fb80 1 -- 172.21.15.70:0/528102838 wait complete. 2024-06-04T05:14:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.289+0000 7f3851a1fb80 20 client.28972 trim_cache size 0 max 16384 2024-06-04T05:14:53.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f3851a1fb80 1 Processor -- start 2024-06-04T05:14:53.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f3851a1fb80 1 -- start start 2024-06-04T05:14:53.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f3851a1fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead4b00 0x5590deadc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f3851a1fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead54b0 0x5590deadd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f3851a1fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead95f0 0x5590deada5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f3851a1fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f38401abd50 con 0x5590dead4b00 2024-06-04T05:14:53.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f3851a1fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f38401ad8f0 con 0x5590dead54b0 2024-06-04T05:14:53.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f3851a1fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f38401ac480 con 0x5590dead95f0 2024-06-04T05:14:53.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f3844f26700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead4b00 0x5590deadc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f3844f26700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead4b00 0x5590deadc110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43430/0 (socket says 172.21.15.70:43430) 2024-06-04T05:14:53.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f383ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead54b0 0x5590deadd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f3844f26700 1 -- 172.21.15.70:0/1128437171 learned_addr learned my addr 172.21.15.70:0/1128437171 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:53.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f3845727700 1 --2- 172.21.15.70:0/1128437171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead95f0 0x5590deada5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f383ffff700 1 -- 172.21.15.70:0/1128437171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead95f0 msgr2=0x5590deada5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:53.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f383ffff700 1 --2- 172.21.15.70:0/1128437171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead95f0 0x5590deada5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f383ffff700 1 -- 172.21.15.70:0/1128437171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead4b00 msgr2=0x5590deadc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:53.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f383ffff700 1 --2- 172.21.15.70:0/1128437171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead4b00 0x5590deadc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.323+0000 7f383ffff700 1 -- 172.21.15.70:0/1128437171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590dead0090 con 0x5590dead54b0 2024-06-04T05:14:53.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.324+0000 7f383ffff700 1 --2- 172.21.15.70:0/1128437171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead54b0 0x5590deadd470 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f38300107c0 tx=0x7f3830012660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc576df6ae8f0360 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:53.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.324+0000 7f38267fc700 1 -- 172.21.15.70:0/1128437171 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3830029040 con 0x5590dead54b0 2024-06-04T05:14:53.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.324+0000 7f38267fc700 1 -- 172.21.15.70:0/1128437171 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3830009310 con 0x5590dead54b0 2024-06-04T05:14:53.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.324+0000 7f38267fc700 1 -- 172.21.15.70:0/1128437171 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3830025dd0 con 0x5590dead54b0 2024-06-04T05:14:53.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.325+0000 7f3851a1fb80 1 -- 172.21.15.70:0/1128437171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead54b0 msgr2=0x5590deadd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:53.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.325+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/1128437171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead54b0 0x5590deadd470 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f38300107c0 tx=0x7f3830012660 comp rx=0 tx=0).stop 2024-06-04T05:14:53.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.325+0000 7f3851a1fb80 1 -- 172.21.15.70:0/1128437171 shutdown_connections 2024-06-04T05:14:53.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.325+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/1128437171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead95f0 0x5590deada5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.325+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/1128437171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead4b00 0x5590deadc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.325+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/1128437171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead54b0 0x5590deadd470 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.325+0000 7f3851a1fb80 1 -- 172.21.15.70:0/1128437171 >> 172.21.15.70:0/1128437171 conn(0x5590dea002c0 msgr2=0x5590dead9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:53.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.325+0000 7f3851a1fb80 1 -- 172.21.15.70:0/1128437171 shutdown_connections 2024-06-04T05:14:53.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.325+0000 7f3851a1fb80 1 -- 172.21.15.70:0/1128437171 wait complete. 2024-06-04T05:14:53.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.325+0000 7f3851a1fb80 1 -- start start 2024-06-04T05:14:53.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.326+0000 7f3851a1fb80 1 Processor -- start 2024-06-04T05:14:53.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.326+0000 7f3851a1fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead4b00 0x5590deafa330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.326+0000 7f3851a1fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 0x5590deafa870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.326+0000 7f3851a1fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead95f0 0x5590deafadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.326+0000 7f3851a1fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f38401ad220 con 0x5590dead4b00 2024-06-04T05:14:53.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.326+0000 7f3851a1fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f38401acb50 con 0x5590dead95f0 2024-06-04T05:14:53.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.326+0000 7f3851a1fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f38401a9d10 con 0x5590dead54b0 2024-06-04T05:14:53.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.326+0000 7f3851a1fb80 10 client.0 _ll_register_callbacks cb 0x5590de9bd9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:14:53.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.326+0000 7f383ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 0x5590deafa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.326+0000 7f3845727700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead95f0 0x5590deafadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.326+0000 7f3845727700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead95f0 0x5590deafadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:48960/0 (socket says 172.21.15.70:48960) 2024-06-04T05:14:53.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.326+0000 7f3845727700 1 -- 172.21.15.70:0/842415316 learned_addr learned my addr 172.21.15.70:0/842415316 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:14:53.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.327+0000 7f3844f26700 1 --2- 172.21.15.70:0/842415316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead4b00 0x5590deafa330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.327+0000 7f3845727700 1 -- 172.21.15.70:0/842415316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 msgr2=0x5590deafa870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:53.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.327+0000 7f3845727700 1 --2- 172.21.15.70:0/842415316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 0x5590deafa870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.327+0000 7f3845727700 1 -- 172.21.15.70:0/842415316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead4b00 msgr2=0x5590deafa330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:14:53.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.327+0000 7f3845727700 1 --2- 172.21.15.70:0/842415316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead4b00 0x5590deafa330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.327+0000 7f3845727700 1 -- 172.21.15.70:0/842415316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590deb69c90 con 0x5590dead95f0 2024-06-04T05:14:53.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.327+0000 7f383ffff700 1 --2- 172.21.15.70:0/842415316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 0x5590deafa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:14:53.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.327+0000 7f3845727700 1 --2- 172.21.15.70:0/842415316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead95f0 0x5590deafadb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3838017000 tx=0x7f3838009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d43d12ad9a3998d8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:14:53.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.327+0000 7f381ffff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:14:53.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.327+0000 7f381ffff700 1 -- 172.21.15.70:0/842415316 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3838015040 con 0x5590dead95f0 2024-06-04T05:14:53.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.327+0000 7f381ffff700 1 -- 172.21.15.70:0/842415316 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f3838007a40 con 0x5590dead95f0 2024-06-04T05:14:53.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.328+0000 7f3851a1fb80 1 -- 172.21.15.70:0/842415316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3830009310 con 0x5590dead95f0 2024-06-04T05:14:53.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.328+0000 7f381ffff700 1 -- 172.21.15.70:0/842415316 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f3838007c40 con 0x5590dead95f0 2024-06-04T05:14:53.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.328+0000 7f3851a1fb80 10 client.28865 fetch_fsmap learned FSMap version 55 2024-06-04T05:14:53.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.328+0000 7f3851a1fb80 1 -- 172.21.15.70:0/842415316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5590deb6afd0 con 0x5590dead95f0 2024-06-04T05:14:53.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.328+0000 7f381ffff700 1 -- 172.21.15.70:0/842415316 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3830009310 con 0x5590dead95f0 2024-06-04T05:14:53.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.328+0000 7f381ffff700 1 -- 172.21.15.70:0/842415316 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f3838011460 con 0x5590dead95f0 2024-06-04T05:14:53.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.328+0000 7f3851a1fb80 10 client.28865 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:14:53.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.328+0000 7f3851a1fb80 10 client.28865 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:14:53.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.328+0000 7f3851a1fb80 20 client.28865 populate_metadata read hostname 'smithi070' 2024-06-04T05:14:53.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.328+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/842415316 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5590deb6b780 0x5590deb6dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:14:53.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.328+0000 7f3851a1fb80 4 client.28865 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:14:53.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.328+0000 7f3851a1fb80 1 -- 172.21.15.70:0/842415316 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5590dea2e520 con 0x5590deb6b780 2024-06-04T05:14:53.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.328+0000 7f3844f26700 1 --2- 172.21.15.70:0/842415316 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5590deb6b780 0x5590deb6dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:14:53.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.329+0000 7f381ffff700 10 client.28865 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:53.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.329+0000 7f3844f26700 1 --2- 172.21.15.70:0/842415316 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5590deb6b780 0x5590deb6dba0 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=500ab0709b61eab8 server_cookie=5baf256220c39a9e in_seq=0 out_seq=0 2024-06-04T05:14:53.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.330+0000 7f381ffff700 1 -- 172.21.15.70:0/842415316 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5590dea2e520 con 0x5590deb6b780 2024-06-04T05:14:53.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.330+0000 7f381ffff700 10 client.28865 handle_command_reply: tid=0 2024-06-04T05:14:53.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.330+0000 7f3851a1fb80 1 client.28865 shutdown 2024-06-04T05:14:53.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.331+0000 7f3851a1fb80 1 -- 172.21.15.70:0/842415316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead95f0 msgr2=0x5590deafadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:53.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.331+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/842415316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead95f0 0x5590deafadb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3838017000 tx=0x7f3838009920 comp rx=0 tx=0).stop 2024-06-04T05:14:53.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.331+0000 7f3851a1fb80 1 -- 172.21.15.70:0/842415316 shutdown_connections 2024-06-04T05:14:53.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.331+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/842415316 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5590deb6b780 0x5590deb6dba0 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.331+0000 7f381ffff700 0 client.28865 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:14:53.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.331+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/842415316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590dead95f0 0x5590deafadb0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.332+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/842415316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590dead4b00 0x5590deafa330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.332+0000 7f3851a1fb80 1 --2- 172.21.15.70:0/842415316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5590dead54b0 0x5590deafa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.332+0000 7f3851a1fb80 1 -- 172.21.15.70:0/842415316 >> 172.21.15.70:0/842415316 conn(0x5590dea002c0 msgr2=0x5590deada6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:53.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.332+0000 7f3851a1fb80 1 -- 172.21.15.70:0/842415316 shutdown_connections 2024-06-04T05:14:53.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.333+0000 7f3851a1fb80 1 -- 172.21.15.70:0/842415316 wait complete. 2024-06-04T05:14:53.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.333+0000 7f3851a1fb80 20 client.28865 trim_cache size 0 max 16384 2024-06-04T05:14:53.337 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:14:53.337 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:14:53.337 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "2663ccac-9153-4397-baaa-f68acdec68d5", 2024-06-04T05:14:53.337 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:14:53.337 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:14:53.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.334+0000 7f384718a700 1 -- 172.21.15.70:0/366322446 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38280b2a20 msgr2=0x7f38280b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:53.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.334+0000 7f384718a700 1 --2- 172.21.15.70:0/366322446 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38280b2a20 0x7f38280b4ee0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3838009da0 tx=0x7f3838009b20 comp rx=0 tx=0).stop 2024-06-04T05:14:53.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.334+0000 7f384718a700 1 -- 172.21.15.70:0/366322446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3840102470 msgr2=0x7f384019bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:14:53.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.334+0000 7f384718a700 1 --2- 172.21.15.70:0/366322446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3840102470 0x7f384019bd70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f383400e920 tx=0x7f383400ec30 comp rx=0 tx=0).stop 2024-06-04T05:14:53.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.334+0000 7f384718a700 1 -- 172.21.15.70:0/366322446 shutdown_connections 2024-06-04T05:14:53.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.334+0000 7f384718a700 1 --2- 172.21.15.70:0/366322446 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38280b2a20 0x7f38280b4ee0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.334+0000 7f384718a700 1 --2- 172.21.15.70:0/366322446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3840108550 0x7f38401a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.334+0000 7f384718a700 1 --2- 172.21.15.70:0/366322446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3840102470 0x7f384019bd70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.334+0000 7f384718a700 1 --2- 172.21.15.70:0/366322446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3840102e30 0x7f384019c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:14:53.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.334+0000 7f384718a700 1 -- 172.21.15.70:0/366322446 >> 172.21.15.70:0/366322446 conn(0x7f3840079480 msgr2=0x7f38401003e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:14:53.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.334+0000 7f384718a700 1 -- 172.21.15.70:0/366322446 shutdown_connections 2024-06-04T05:14:53.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:14:53.334+0000 7f384718a700 1 -- 172.21.15.70:0/366322446 wait complete. 2024-06-04T05:14:53.400 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:53 smithi070 ceph-mon[32631]: pgmap v554: 97 pgs: 97 active+clean; 563 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.5 MiB/s wr, 406 op/s 2024-06-04T05:14:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:53 smithi112 ceph-mon[36453]: pgmap v554: 97 pgs: 97 active+clean; 563 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.5 MiB/s wr, 406 op/s 2024-06-04T05:14:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:53 smithi098 ceph-mon[35569]: pgmap v554: 97 pgs: 97 active+clean; 563 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.5 MiB/s wr, 406 op/s 2024-06-04T05:14:53.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 32712 8.32 MB/sec execute 105 sec latency 308.614 ms 2024-06-04T05:14:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:54 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:14:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:54 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:54 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:14:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:54 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:14:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:54 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:14:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:54 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:14:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:54 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:14:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:54 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:54 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:54 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:54.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:54 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:54.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:54 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:14:54.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:54 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:54.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:54 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:14:54.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:54 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:14:54.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:54 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:14:54.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:54 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:14:54.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:54 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:14:54.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:54 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:54.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:54 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:54.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:54 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:54.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:54 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:54.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:54 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:14:54.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:54 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:54.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:54 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:14:54.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:54 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:14:54.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:54 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:14:54.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:54 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:14:54.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:54 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:14:54.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:54 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:54.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:54 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:54.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:54 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:54.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:54 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:54.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 32887 8.30 MB/sec execute 106 sec latency 100.724 ms 2024-06-04T05:14:55.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:55 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:55.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:55 smithi070 ceph-mon[32631]: pgmap v555: 97 pgs: 97 active+clean; 563 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 287 op/s 2024-06-04T05:14:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:55 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:55 smithi112 ceph-mon[36453]: pgmap v555: 97 pgs: 97 active+clean; 563 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 287 op/s 2024-06-04T05:14:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:55 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:14:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:55 smithi098 ceph-mon[35569]: pgmap v555: 97 pgs: 97 active+clean; 563 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 287 op/s 2024-06-04T05:14:55.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 33021 8.28 MB/sec execute 107 sec latency 126.630 ms 2024-06-04T05:14:56.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 33186 8.30 MB/sec execute 108 sec latency 128.040 ms 2024-06-04T05:14:57.579 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:57 smithi070 ceph-mon[32631]: pgmap v556: 97 pgs: 97 active+clean; 580 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 442 op/s 2024-06-04T05:14:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:57 smithi112 ceph-mon[36453]: pgmap v556: 97 pgs: 97 active+clean; 580 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 442 op/s 2024-06-04T05:14:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:57 smithi098 ceph-mon[35569]: pgmap v556: 97 pgs: 97 active+clean; 580 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 442 op/s 2024-06-04T05:14:57.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 33363 8.33 MB/sec execute 109 sec latency 155.419 ms 2024-06-04T05:14:58.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 33492 8.32 MB/sec execute 110 sec latency 146.382 ms 2024-06-04T05:14:59.520 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:14:59.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:14:59 smithi070 ceph-mon[32631]: pgmap v557: 97 pgs: 97 active+clean; 580 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 318 op/s 2024-06-04T05:14:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:14:59 smithi112 ceph-mon[36453]: pgmap v557: 97 pgs: 97 active+clean; 580 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 318 op/s 2024-06-04T05:14:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:14:59 smithi098 ceph-mon[35569]: pgmap v557: 97 pgs: 97 active+clean; 580 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 318 op/s 2024-06-04T05:14:59.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 33635 8.33 MB/sec execute 111 sec latency 141.152 ms 2024-06-04T05:15:00.807 INFO:tasks.workunit.client.0.smithi070.stdout: 10 33788 8.35 MB/sec execute 112 sec latency 130.422 ms 2024-06-04T05:15:01.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.076+0000 7f81e9d04640 1 -- 172.21.15.70:0/2932987410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81e4103d20 msgr2=0x7f81e4104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:01.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.076+0000 7f81e9d04640 1 --2- 172.21.15.70:0/2932987410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81e4103d20 0x7f81e4104100 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f81d4009900 tx=0x7f81d402a480 comp rx=0 tx=0).stop 2024-06-04T05:15:01.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.076+0000 7f81e9d04640 1 -- 172.21.15.70:0/2932987410 shutdown_connections 2024-06-04T05:15:01.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.076+0000 7f81e9d04640 1 --2- 172.21.15.70:0/2932987410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e410d060 0x7f81e410f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.076+0000 7f81e9d04640 1 --2- 172.21.15.70:0/2932987410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81e4104640 0x7f81e410cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.076+0000 7f81e9d04640 1 --2- 172.21.15.70:0/2932987410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81e4103d20 0x7f81e4104100 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.076+0000 7f81e9d04640 1 -- 172.21.15.70:0/2932987410 >> 172.21.15.70:0/2932987410 conn(0x7f81e4078ad0 msgr2=0x7f81e4078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:01.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.076+0000 7f81e9d04640 1 -- 172.21.15.70:0/2932987410 shutdown_connections 2024-06-04T05:15:01.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.076+0000 7f81e9d04640 1 -- 172.21.15.70:0/2932987410 wait complete. 2024-06-04T05:15:01.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.076+0000 7f81e9d04640 1 Processor -- start 2024-06-04T05:15:01.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e9d04640 1 -- start start 2024-06-04T05:15:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e9d04640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81e4103d20 0x7f81e4199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e9d04640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e4104640 0x7f81e41996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e9d04640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81e410d060 0x7f81e419d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e9d04640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f81e4112bc0 con 0x7f81e410d060 2024-06-04T05:15:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e9d04640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f81e4112a40 con 0x7f81e4104640 2024-06-04T05:15:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e9d04640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f81e4112d40 con 0x7f81e4103d20 2024-06-04T05:15:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81e410d060 0x7f81e419d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e37fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81e4103d20 0x7f81e4199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e4104640 0x7f81e41996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e37fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81e4103d20 0x7f81e4199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:47408/0 (socket says 172.21.15.70:47408) 2024-06-04T05:15:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e37fe640 1 -- 172.21.15.70:0/3468975249 learned_addr learned my addr 172.21.15.70:0/3468975249 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e3fff640 1 -- 172.21.15.70:0/3468975249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81e4103d20 msgr2=0x7f81e4199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e3fff640 1 --2- 172.21.15.70:0/3468975249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81e4103d20 0x7f81e4199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e3fff640 1 -- 172.21.15.70:0/3468975249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e4104640 msgr2=0x7f81e41996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e3fff640 1 --2- 172.21.15.70:0/3468975249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e4104640 0x7f81e41996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e3fff640 1 -- 172.21.15.70:0/3468975249 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e419e100 con 0x7f81e410d060 2024-06-04T05:15:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e2ffd640 1 --2- 172.21.15.70:0/3468975249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e4104640 0x7f81e41996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e3fff640 1 --2- 172.21.15.70:0/3468975249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81e410d060 0x7f81e419d9d0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f81dc009fd0 tx=0x7f81dc00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.077+0000 7f81e0ff9640 1 -- 172.21.15.70:0/3468975249 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc00ce20 con 0x7f81e410d060 2024-06-04T05:15:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.078+0000 7f81e0ff9640 1 -- 172.21.15.70:0/3468975249 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f81dc005020 con 0x7f81e410d060 2024-06-04T05:15:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.078+0000 7f81e9d04640 1 -- 172.21.15.70:0/3468975249 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e419e3f0 con 0x7f81e410d060 2024-06-04T05:15:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.078+0000 7f81e0ff9640 1 -- 172.21.15.70:0/3468975249 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc010690 con 0x7f81e410d060 2024-06-04T05:15:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.078+0000 7f81e9d04640 1 -- 172.21.15.70:0/3468975249 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e41a5c80 con 0x7f81e410d060 2024-06-04T05:15:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.080+0000 7f81e0ff9640 1 -- 172.21.15.70:0/3468975249 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f81dc004a40 con 0x7f81e410d060 2024-06-04T05:15:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.080+0000 7f81eca1b740 1 Processor -- start 2024-06-04T05:15:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.080+0000 7f81eca1b740 1 -- start start 2024-06-04T05:15:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81eca1b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973b820 0x55d6d973bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81eca1b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d973c140 0x55d6d97464c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81eca1b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 0x55d6d9748df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81eca1b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d6d96adc30 con 0x55d6d9746a00 2024-06-04T05:15:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81eca1b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d6d969d7d0 con 0x55d6d973c140 2024-06-04T05:15:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81eca1b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d6d96a1810 con 0x55d6d973b820 2024-06-04T05:15:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81e2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 0x55d6d9748df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81e37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d973c140 0x55d6d97464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81e2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 0x55d6d9748df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36810/0 (socket says 172.21.15.70:36810) 2024-06-04T05:15:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81e2ffd640 1 -- 172.21.15.70:0/1703795191 learned_addr learned my addr 172.21.15.70:0/1703795191 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81e0ff9640 1 --2- 172.21.15.70:0/3468975249 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81cc0b2a10 0x7f81cc0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81e2ffd640 1 -- 172.21.15.70:0/1703795191 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973b820 msgr2=0x55d6d973bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81e2ffd640 1 --2- 172.21.15.70:0/1703795191 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973b820 0x55d6d973bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81e2ffd640 1 -- 172.21.15.70:0/1703795191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d973c140 msgr2=0x55d6d97464c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81e2ffd640 1 --2- 172.21.15.70:0/1703795191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d973c140 0x55d6d97464c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81e2ffd640 1 -- 172.21.15.70:0/1703795191 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6d93b2210 con 0x55d6d9746a00 2024-06-04T05:15:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81e3fff640 1 --2- 172.21.15.70:0/3468975249 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81cc0b2a10 0x7f81cc0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81e0ff9640 1 -- 172.21.15.70:0/3468975249 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f81dc0e2cc0 con 0x7f81e410d060 2024-06-04T05:15:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81e2ffd640 1 --2- 172.21.15.70:0/1703795191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 0x55d6d9748df0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f81d8011570 tx=0x7f81d8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e76998ddef957d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.081+0000 7f81c9ffb640 1 -- 172.21.15.70:0/1703795191 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d801f070 con 0x55d6d9746a00 2024-06-04T05:15:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.082+0000 7f81e3fff640 1 --2- 172.21.15.70:0/3468975249 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81cc0b2a10 0x7f81cc0b4ed0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0a9f00 tx=0x7f81dc077b40 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.082+0000 7f81c9ffb640 1 -- 172.21.15.70:0/1703795191 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f81d800fc50 con 0x55d6d9746a00 2024-06-04T05:15:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.082+0000 7f81c9ffb640 1 -- 172.21.15.70:0/1703795191 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d801adb0 con 0x55d6d9746a00 2024-06-04T05:15:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.082+0000 7f81eca1b740 1 -- 172.21.15.70:0/1703795191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 msgr2=0x55d6d9748df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.082+0000 7f81eca1b740 1 --2- 172.21.15.70:0/1703795191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 0x55d6d9748df0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f81d8011570 tx=0x7f81d8011a30 comp rx=0 tx=0).stop 2024-06-04T05:15:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.082+0000 7f81eca1b740 1 -- 172.21.15.70:0/1703795191 shutdown_connections 2024-06-04T05:15:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.082+0000 7f81eca1b740 1 --2- 172.21.15.70:0/1703795191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d973c140 0x55d6d97464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.082+0000 7f81eca1b740 1 --2- 172.21.15.70:0/1703795191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 0x55d6d9748df0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.082+0000 7f81eca1b740 1 --2- 172.21.15.70:0/1703795191 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973b820 0x55d6d973bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.082+0000 7f81eca1b740 1 -- 172.21.15.70:0/1703795191 >> 172.21.15.70:0/1703795191 conn(0x55d6d9737d30 msgr2=0x55d6d9738110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.082+0000 7f81eca1b740 1 -- 172.21.15.70:0/1703795191 shutdown_connections 2024-06-04T05:15:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.082+0000 7f81eca1b740 1 -- 172.21.15.70:0/1703795191 wait complete. 2024-06-04T05:15:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.083+0000 7f81eca1b740 1 -- start start 2024-06-04T05:15:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.083+0000 7f81eca1b740 1 Processor -- start 2024-06-04T05:15:01.090 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.083+0000 7f81eca1b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973b820 0x55d6d9743a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.083+0000 7f81eca1b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d973c140 0x55d6d9743fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.083+0000 7f81eca1b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 0x55d6d9744510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.083+0000 7f81eca1b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d6d9383790 con 0x55d6d9746a00 2024-06-04T05:15:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.083+0000 7f81eca1b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d6d9659890 con 0x55d6d973c140 2024-06-04T05:15:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.083+0000 7f81eca1b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d6d9749630 con 0x55d6d973b820 2024-06-04T05:15:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.083+0000 7f81eca1b740 10 client.0 _ll_register_callbacks cb 0x55d6d93b1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.083+0000 7f81e2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 0x55d6d9744510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.083+0000 7f81e2ffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973b820 0x55d6d9743a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.083+0000 7f81e2ffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973b820 0x55d6d9743a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:47418/0 (socket says 172.21.15.70:47418) 2024-06-04T05:15:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.083+0000 7f81e2ffd640 1 -- 172.21.15.70:0/4258647361 learned_addr learned my addr 172.21.15.70:0/4258647361 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.083+0000 7f81e37fe640 1 --2- 172.21.15.70:0/4258647361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d973c140 0x55d6d9743fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.084+0000 7f81e37fe640 1 -- 172.21.15.70:0/4258647361 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973b820 msgr2=0x55d6d9743a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.084+0000 7f81e37fe640 1 --2- 172.21.15.70:0/4258647361 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973b820 0x55d6d9743a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.084+0000 7f81e37fe640 1 -- 172.21.15.70:0/4258647361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 msgr2=0x55d6d9744510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.084+0000 7f81e37fe640 1 --2- 172.21.15.70:0/4258647361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 0x55d6d9744510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.084+0000 7f81e37fe640 1 -- 172.21.15.70:0/4258647361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6d947e620 con 0x55d6d973c140 2024-06-04T05:15:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.084+0000 7f81e37fe640 1 --2- 172.21.15.70:0/4258647361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d973c140 0x55d6d9743fd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f81d4000c00 tx=0x7f81d40044f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f326c10a9dfff815 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.084+0000 7f81c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:15:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.085+0000 7f81c2ffd640 1 -- 172.21.15.70:0/4258647361 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d4035040 con 0x55d6d973c140 2024-06-04T05:15:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.085+0000 7f81c2ffd640 1 -- 172.21.15.70:0/4258647361 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f81d4005460 con 0x55d6d973c140 2024-06-04T05:15:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.085+0000 7f81eca1b740 1 -- 172.21.15.70:0/4258647361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81d800fc50 con 0x55d6d973c140 2024-06-04T05:15:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.085+0000 7f81c2ffd640 1 -- 172.21.15.70:0/4258647361 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f81d4041860 con 0x55d6d973c140 2024-06-04T05:15:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.085+0000 7f81eca1b740 10 client.28877 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.085+0000 7f81eca1b740 1 -- 172.21.15.70:0/4258647361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55d6d9755470 con 0x55d6d973c140 2024-06-04T05:15:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.085+0000 7f81c2ffd640 1 -- 172.21.15.70:0/4258647361 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f81d800fc50 con 0x55d6d973c140 2024-06-04T05:15:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.085+0000 7f81c2ffd640 1 -- 172.21.15.70:0/4258647361 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f81d40035f0 con 0x55d6d973c140 2024-06-04T05:15:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.085+0000 7f81eca1b740 10 client.28877 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:01.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.085+0000 7f81eca1b740 10 client.28877 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:01.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.085+0000 7f81eca1b740 20 client.28877 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:01.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.085+0000 7f81eca1b740 1 --2- 172.21.15.70:0/4258647361 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d6d9756930 0x55d6d9758d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.086+0000 7f81eca1b740 4 client.28877 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:15:01.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.086+0000 7f81eca1b740 1 -- 172.21.15.70:0/4258647361 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6d96afb80 con 0x55d6d9756930 2024-06-04T05:15:01.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.086+0000 7f81e2ffd640 1 --2- 172.21.15.70:0/4258647361 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d6d9756930 0x55d6d9758d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:01.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.086+0000 7f81c2ffd640 10 client.28877 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:01.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.086+0000 7f81e2ffd640 1 --2- 172.21.15.70:0/4258647361 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d6d9756930 0x55d6d9758d20 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ad7051349c1d5ea server_cookie=54b4a46fec5e35dc in_seq=0 out_seq=0 2024-06-04T05:15:01.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.087+0000 7f81c2ffd640 1 -- 172.21.15.70:0/4258647361 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d6d96afb80 con 0x55d6d9756930 2024-06-04T05:15:01.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.087+0000 7f81c2ffd640 10 client.28877 handle_command_reply: tid=0 2024-06-04T05:15:01.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.087+0000 7f81eca1b740 1 client.28877 shutdown 2024-06-04T05:15:01.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.087+0000 7f81eca1b740 1 -- 172.21.15.70:0/4258647361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d973c140 msgr2=0x55d6d9743fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:01.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.087+0000 7f81eca1b740 1 --2- 172.21.15.70:0/4258647361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d973c140 0x55d6d9743fd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f81d4000c00 tx=0x7f81d40044f0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.087+0000 7f81eca1b740 1 -- 172.21.15.70:0/4258647361 shutdown_connections 2024-06-04T05:15:01.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.087+0000 7f81eca1b740 1 --2- 172.21.15.70:0/4258647361 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d6d9756930 0x55d6d9758d20 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.087+0000 7f81eca1b740 1 --2- 172.21.15.70:0/4258647361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d973c140 0x55d6d9743fd0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.087+0000 7f81eca1b740 1 --2- 172.21.15.70:0/4258647361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 0x55d6d9744510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.087+0000 7f81eca1b740 1 --2- 172.21.15.70:0/4258647361 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973b820 0x55d6d9743a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.087+0000 7f81eca1b740 1 -- 172.21.15.70:0/4258647361 >> 172.21.15.70:0/4258647361 conn(0x55d6d9737d30 msgr2=0x55d6d9736fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.087+0000 7f81c2ffd640 0 client.28877 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.087+0000 7f81eca1b740 1 -- 172.21.15.70:0/4258647361 shutdown_connections 2024-06-04T05:15:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.087+0000 7f81eca1b740 1 -- 172.21.15.70:0/4258647361 wait complete. 2024-06-04T05:15:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.087+0000 7f81eca1b740 20 client.28877 trim_cache size 0 max 16384 2024-06-04T05:15:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.102+0000 7f81eca1b740 1 Processor -- start 2024-06-04T05:15:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.102+0000 7f81eca1b740 1 -- start start 2024-06-04T05:15:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81eca1b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973b820 0x55d6d9737730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81eca1b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d973c140 0x55d6d9744160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81eca1b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d9746a00 0x55d6d97446a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81eca1b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f81e41ac560 con 0x55d6d973c140 2024-06-04T05:15:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81eca1b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f81e41ae100 con 0x55d6d9746a00 2024-06-04T05:15:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81eca1b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f81e41acc90 con 0x55d6d973b820 2024-06-04T05:15:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81e37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d973c140 0x55d6d9744160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81e37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d973c140 0x55d6d9744160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36820/0 (socket says 172.21.15.70:36820) 2024-06-04T05:15:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81e37fe640 1 -- 172.21.15.70:0/73434770 learned_addr learned my addr 172.21.15.70:0/73434770 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81e2ffd640 1 --2- 172.21.15.70:0/73434770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973b820 0x55d6d9737730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81e2ffd640 1 --2- 172.21.15.70:0/73434770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d9746a00 0x55d6d97446a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81e37fe640 1 -- 172.21.15.70:0/73434770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973b820 msgr2=0x55d6d9737730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81e37fe640 1 --2- 172.21.15.70:0/73434770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973b820 0x55d6d9737730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81e37fe640 1 -- 172.21.15.70:0/73434770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d9746a00 msgr2=0x55d6d97446a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81e37fe640 1 --2- 172.21.15.70:0/73434770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d9746a00 0x55d6d97446a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81e37fe640 1 -- 172.21.15.70:0/73434770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6d9742a60 con 0x55d6d973c140 2024-06-04T05:15:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81e37fe640 1 --2- 172.21.15.70:0/73434770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d973c140 0x55d6d9744160 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f81d402a9a0 tx=0x7f81d4003e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24bf42167e47a43 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81c97fa640 1 -- 172.21.15.70:0/73434770 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d4035030 con 0x55d6d973c140 2024-06-04T05:15:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81c97fa640 1 -- 172.21.15.70:0/73434770 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f81d4045070 con 0x55d6d973c140 2024-06-04T05:15:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.103+0000 7f81c97fa640 1 -- 172.21.15.70:0/73434770 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d4055070 con 0x55d6d973c140 2024-06-04T05:15:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.104+0000 7f81eca1b740 1 -- 172.21.15.70:0/73434770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d973c140 msgr2=0x55d6d9744160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.104+0000 7f81eca1b740 1 --2- 172.21.15.70:0/73434770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d973c140 0x55d6d9744160 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f81d402a9a0 tx=0x7f81d4003e10 comp rx=0 tx=0).stop 2024-06-04T05:15:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.104+0000 7f81eca1b740 1 -- 172.21.15.70:0/73434770 shutdown_connections 2024-06-04T05:15:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.104+0000 7f81eca1b740 1 --2- 172.21.15.70:0/73434770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d9746a00 0x55d6d97446a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.104+0000 7f81eca1b740 1 --2- 172.21.15.70:0/73434770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d973c140 0x55d6d9744160 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.104+0000 7f81eca1b740 1 --2- 172.21.15.70:0/73434770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973b820 0x55d6d9737730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.104+0000 7f81eca1b740 1 -- 172.21.15.70:0/73434770 >> 172.21.15.70:0/73434770 conn(0x55d6d9737d30 msgr2=0x55d6d9743a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.104+0000 7f81eca1b740 1 -- 172.21.15.70:0/73434770 shutdown_connections 2024-06-04T05:15:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.104+0000 7f81eca1b740 1 -- 172.21.15.70:0/73434770 wait complete. 2024-06-04T05:15:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.104+0000 7f81eca1b740 1 -- start start 2024-06-04T05:15:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.105+0000 7f81eca1b740 1 Processor -- start 2024-06-04T05:15:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.105+0000 7f81eca1b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d973b820 0x55d6d974ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.105+0000 7f81eca1b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973c140 0x55d6d9740660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.105+0000 7f81eca1b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 0x55d6d9740ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.105+0000 7f81eca1b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f81e41ada30 con 0x55d6d9746a00 2024-06-04T05:15:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.105+0000 7f81eca1b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f81e41ad360 con 0x55d6d973b820 2024-06-04T05:15:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.105+0000 7f81eca1b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f81e41aa560 con 0x55d6d973c140 2024-06-04T05:15:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.105+0000 7f81eca1b740 10 client.0 _ll_register_callbacks cb 0x7f81d4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.106+0000 7f81e2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 0x55d6d9740ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.106+0000 7f81e2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 0x55d6d9740ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36836/0 (socket says 172.21.15.70:36836) 2024-06-04T05:15:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.106+0000 7f81e2ffd640 1 -- 172.21.15.70:0/1052513821 learned_addr learned my addr 172.21.15.70:0/1052513821 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.106+0000 7f81e2ffd640 1 --2- 172.21.15.70:0/1052513821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d973b820 0x55d6d974ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.106+0000 7f81e37fe640 1 --2- 172.21.15.70:0/1052513821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973c140 0x55d6d9740660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.106+0000 7f81e37fe640 1 -- 172.21.15.70:0/1052513821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d973b820 msgr2=0x55d6d974ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.106+0000 7f81e37fe640 1 --2- 172.21.15.70:0/1052513821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d973b820 0x55d6d974ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.106+0000 7f81e37fe640 1 -- 172.21.15.70:0/1052513821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 msgr2=0x55d6d9740ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.106+0000 7f81e37fe640 1 --2- 172.21.15.70:0/1052513821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 0x55d6d9740ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.106+0000 7f81e37fe640 1 -- 172.21.15.70:0/1052513821 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6d9741240 con 0x55d6d973c140 2024-06-04T05:15:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.106+0000 7f81e2ffd640 1 --2- 172.21.15.70:0/1052513821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 0x55d6d9740ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.106+0000 7f81e37fe640 1 --2- 172.21.15.70:0/1052513821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973c140 0x55d6d9740660 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55d6d970c000 tx=0x7f81d4041e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b329d5feb29bdc08 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.106+0000 7f81c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:15:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.107+0000 7f81c2ffd640 1 -- 172.21.15.70:0/1052513821 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d4055030 con 0x55d6d973c140 2024-06-04T05:15:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.107+0000 7f81c2ffd640 1 -- 172.21.15.70:0/1052513821 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f81d40046c0 con 0x55d6d973c140 2024-06-04T05:15:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.107+0000 7f81eca1b740 1 -- 172.21.15.70:0/1052513821 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81d4045070 con 0x55d6d973c140 2024-06-04T05:15:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.107+0000 7f81c2ffd640 1 -- 172.21.15.70:0/1052513821 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f81d4005850 con 0x55d6d973c140 2024-06-04T05:15:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.107+0000 7f81eca1b740 10 client.28993 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.107+0000 7f81eca1b740 1 -- 172.21.15.70:0/1052513821 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55d6d97c1390 con 0x55d6d973c140 2024-06-04T05:15:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.107+0000 7f81c2ffd640 1 -- 172.21.15.70:0/1052513821 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f81d4045070 con 0x55d6d973c140 2024-06-04T05:15:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.107+0000 7f81c2ffd640 1 -- 172.21.15.70:0/1052513821 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f81d4061440 con 0x55d6d973c140 2024-06-04T05:15:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.107+0000 7f81eca1b740 10 client.28993 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.108+0000 7f81eca1b740 10 client.28993 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.108+0000 7f81eca1b740 20 client.28993 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.108+0000 7f81eca1b740 1 --2- 172.21.15.70:0/1052513821 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d6d97c1940 0x55d6d97c3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.108+0000 7f81eca1b740 4 client.28993 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:15:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.108+0000 7f81eca1b740 1 -- 172.21.15.70:0/1052513821 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d6d96a9360 con 0x55d6d97c1940 2024-06-04T05:15:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.108+0000 7f81e2ffd640 1 --2- 172.21.15.70:0/1052513821 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d6d97c1940 0x55d6d97c3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.108+0000 7f81c2ffd640 10 client.28993 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.108+0000 7f81e2ffd640 1 --2- 172.21.15.70:0/1052513821 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d6d97c1940 0x55d6d97c3d30 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6161db193968cc5c server_cookie=791b0499d52be6fc in_seq=0 out_seq=0 2024-06-04T05:15:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.203+0000 7f81c2ffd640 1 -- 172.21.15.70:0/1052513821 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55d6d96a9360 con 0x55d6d97c1940 2024-06-04T05:15:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.203+0000 7f81c2ffd640 10 client.28993 handle_command_reply: tid=0 2024-06-04T05:15:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.203+0000 7f81eca1b740 1 client.28993 shutdown 2024-06-04T05:15:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.203+0000 7f81eca1b740 1 -- 172.21.15.70:0/1052513821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973c140 msgr2=0x55d6d9740660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.203+0000 7f81eca1b740 1 --2- 172.21.15.70:0/1052513821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973c140 0x55d6d9740660 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55d6d970c000 tx=0x7f81d4041e80 comp rx=0 tx=0).stop 2024-06-04T05:15:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.203+0000 7f81eca1b740 1 -- 172.21.15.70:0/1052513821 shutdown_connections 2024-06-04T05:15:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.203+0000 7f81eca1b740 1 --2- 172.21.15.70:0/1052513821 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d6d97c1940 0x55d6d97c3d30 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.203+0000 7f81eca1b740 1 --2- 172.21.15.70:0/1052513821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6d9746a00 0x55d6d9740ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.203+0000 7f81eca1b740 1 --2- 172.21.15.70:0/1052513821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6d973c140 0x55d6d9740660 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.203+0000 7f81c2ffd640 0 client.28993 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.203+0000 7f81eca1b740 1 --2- 172.21.15.70:0/1052513821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6d973b820 0x55d6d974ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.203+0000 7f81eca1b740 1 -- 172.21.15.70:0/1052513821 >> 172.21.15.70:0/1052513821 conn(0x55d6d9737d30 msgr2=0x55d6d97442c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.203+0000 7f81eca1b740 1 -- 172.21.15.70:0/1052513821 shutdown_connections 2024-06-04T05:15:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.203+0000 7f81eca1b740 1 -- 172.21.15.70:0/1052513821 wait complete. 2024-06-04T05:15:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.203+0000 7f81eca1b740 20 client.28993 trim_cache size 0 max 16384 2024-06-04T05:15:01.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:15:01.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:15:01.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:15:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.204+0000 7f81e9d04640 1 -- 172.21.15.70:0/3468975249 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81cc0b2a10 msgr2=0x7f81cc0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.204+0000 7f81e9d04640 1 --2- 172.21.15.70:0/3468975249 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81cc0b2a10 0x7f81cc0b4ed0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0a9f00 tx=0x7f81dc077b40 comp rx=0 tx=0).stop 2024-06-04T05:15:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.204+0000 7f81e9d04640 1 -- 172.21.15.70:0/3468975249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81e410d060 msgr2=0x7f81e419d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.204+0000 7f81e9d04640 1 --2- 172.21.15.70:0/3468975249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81e410d060 0x7f81e419d9d0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f81dc009fd0 tx=0x7f81dc00ef10 comp rx=0 tx=0).stop 2024-06-04T05:15:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.204+0000 7f81e9d04640 1 -- 172.21.15.70:0/3468975249 shutdown_connections 2024-06-04T05:15:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.204+0000 7f81e9d04640 1 --2- 172.21.15.70:0/3468975249 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81cc0b2a10 0x7f81cc0b4ed0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.204+0000 7f81e9d04640 1 --2- 172.21.15.70:0/3468975249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e4104640 0x7f81e41996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.204+0000 7f81e9d04640 1 --2- 172.21.15.70:0/3468975249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81e410d060 0x7f81e419d9d0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.204+0000 7f81e9d04640 1 --2- 172.21.15.70:0/3468975249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81e4103d20 0x7f81e4199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.204+0000 7f81e9d04640 1 -- 172.21.15.70:0/3468975249 >> 172.21.15.70:0/3468975249 conn(0x7f81e4078ad0 msgr2=0x7f81e40feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.204+0000 7f81e9d04640 1 -- 172.21.15.70:0/3468975249 shutdown_connections 2024-06-04T05:15:01.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:01.204+0000 7f81e9d04640 1 -- 172.21.15.70:0/3468975249 wait complete. 2024-06-04T05:15:01.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:01 smithi070 ceph-mon[32631]: pgmap v558: 97 pgs: 97 active+clean; 580 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 318 op/s 2024-06-04T05:15:01.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:01 smithi112 ceph-mon[36453]: pgmap v558: 97 pgs: 97 active+clean; 580 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 318 op/s 2024-06-04T05:15:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:01 smithi098 ceph-mon[35569]: pgmap v558: 97 pgs: 97 active+clean; 580 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 318 op/s 2024-06-04T05:15:01.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 33926 8.37 MB/sec execute 113 sec latency 148.220 ms 2024-06-04T05:15:02.807 INFO:tasks.workunit.client.0.smithi070.stdout: 10 34053 8.33 MB/sec execute 114 sec latency 155.901 ms 2024-06-04T05:15:03.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:03 smithi070 ceph-mon[32631]: pgmap v559: 97 pgs: 97 active+clean; 586 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.4 MiB/s wr, 451 op/s 2024-06-04T05:15:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:03 smithi112 ceph-mon[36453]: pgmap v559: 97 pgs: 97 active+clean; 586 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.4 MiB/s wr, 451 op/s 2024-06-04T05:15:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:03 smithi098 ceph-mon[35569]: pgmap v559: 97 pgs: 97 active+clean; 586 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.4 MiB/s wr, 451 op/s 2024-06-04T05:15:03.807 INFO:tasks.workunit.client.0.smithi070.stdout: 10 34187 8.28 MB/sec execute 115 sec latency 166.307 ms 2024-06-04T05:15:04.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:04 smithi070 ceph-mon[32631]: osdmap e103: 12 total, 12 up, 12 in 2024-06-04T05:15:04.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:04 smithi112 ceph-mon[36453]: osdmap e103: 12 total, 12 up, 12 in 2024-06-04T05:15:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:04 smithi098 ceph-mon[35569]: osdmap e103: 12 total, 12 up, 12 in 2024-06-04T05:15:04.807 INFO:tasks.workunit.client.0.smithi070.stdout: 10 34330 8.27 MB/sec execute 116 sec latency 140.805 ms 2024-06-04T05:15:05.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:05 smithi070 ceph-mon[32631]: pgmap v561: 97 pgs: 97 active+clean; 586 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 346 op/s 2024-06-04T05:15:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:05 smithi112 ceph-mon[36453]: pgmap v561: 97 pgs: 97 active+clean; 586 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 346 op/s 2024-06-04T05:15:05.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:05 smithi098 ceph-mon[35569]: pgmap v561: 97 pgs: 97 active+clean; 586 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 346 op/s 2024-06-04T05:15:05.806 INFO:tasks.workunit.client.0.smithi070.stdout: 10 34466 8.23 MB/sec execute 117 sec latency 142.261 ms 2024-06-04T05:15:06.289 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:15:06.807 INFO:tasks.workunit.client.0.smithi070.stdout: 10 34645 8.24 MB/sec execute 118 sec latency 71.104 ms 2024-06-04T05:15:07.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:07 smithi070 ceph-mon[32631]: pgmap v562: 97 pgs: 97 active+clean; 599 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.6 MiB/s wr, 317 op/s 2024-06-04T05:15:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:07 smithi112 ceph-mon[36453]: pgmap v562: 97 pgs: 97 active+clean; 599 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.6 MiB/s wr, 317 op/s 2024-06-04T05:15:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:07 smithi098 ceph-mon[35569]: pgmap v562: 97 pgs: 97 active+clean; 599 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.6 MiB/s wr, 317 op/s 2024-06-04T05:15:07.807 INFO:tasks.workunit.client.0.smithi070.stdout: 10 34828 8.26 MB/sec execute 119 sec latency 61.813 ms 2024-06-04T05:15:08.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:08 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:15:08.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:08 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:15:08.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:08 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:15:08.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:08 smithi070 ceph-mon[32631]: osdmap e104: 12 total, 12 up, 12 in 2024-06-04T05:15:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:08 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:15:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:08 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:15:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:08 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:15:08.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:08 smithi112 ceph-mon[36453]: osdmap e104: 12 total, 12 up, 12 in 2024-06-04T05:15:08.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:08 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:15:08.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:08 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:15:08.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:08 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:15:08.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:08 smithi098 ceph-mon[35569]: osdmap e104: 12 total, 12 up, 12 in 2024-06-04T05:15:08.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.752+0000 7f1eacdb7700 1 -- 172.21.15.70:0/1660229858 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ea81024f0 msgr2=0x7f1ea81103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:08.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.752+0000 7f1eacdb7700 1 --2- 172.21.15.70:0/1660229858 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ea81024f0 0x7f1ea81103d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0009f90 tx=0x7f1ea000b330 comp rx=0 tx=0).stop 2024-06-04T05:15:08.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.752+0000 7f1eacdb7700 1 -- 172.21.15.70:0/1660229858 shutdown_connections 2024-06-04T05:15:08.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.752+0000 7f1eacdb7700 1 --2- 172.21.15.70:0/1660229858 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ea81024f0 0x7f1ea81103d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:08.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.752+0000 7f1eacdb7700 1 --2- 172.21.15.70:0/1660229858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ea8109520 0x7f1ea8109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:08.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.752+0000 7f1eacdb7700 1 --2- 172.21.15.70:0/1660229858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ea8101b30 0x7f1ea8101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:08.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.752+0000 7f1eacdb7700 1 -- 172.21.15.70:0/1660229858 >> 172.21.15.70:0/1660229858 conn(0x7f1ea8078ab0 msgr2=0x7f1ea80ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:08.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.752+0000 7f1eacdb7700 1 -- 172.21.15.70:0/1660229858 shutdown_connections 2024-06-04T05:15:08.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.752+0000 7f1eacdb7700 1 -- 172.21.15.70:0/1660229858 wait complete. 2024-06-04T05:15:08.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.753+0000 7f1eacdb7700 1 Processor -- start 2024-06-04T05:15:08.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.753+0000 7f1eacdb7700 1 -- start start 2024-06-04T05:15:08.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1eacdb7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ea8101b30 0x7f1ea819f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:08.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1eacdb7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ea81024f0 0x7f1ea819fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:08.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1eacdb7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ea8109520 0x7f1ea81998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:08.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1eacdb7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea81135d0 con 0x7f1ea8101b30 2024-06-04T05:15:08.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1ea659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ea8101b30 0x7f1ea819f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:08.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1eacdb7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea8113450 con 0x7f1ea8109520 2024-06-04T05:15:08.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1eacdb7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea8113750 con 0x7f1ea81024f0 2024-06-04T05:15:08.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1ea659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ea8101b30 0x7f1ea819f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51860/0 (socket says 172.21.15.70:51860) 2024-06-04T05:15:08.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1ea659c700 1 -- 172.21.15.70:0/2611020230 learned_addr learned my addr 172.21.15.70:0/2611020230 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:08.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1ea5d9b700 1 --2- 172.21.15.70:0/2611020230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ea81024f0 0x7f1ea819fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:08.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1ea6d9d700 1 --2- 172.21.15.70:0/2611020230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ea8109520 0x7f1ea81998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:08.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1ea659c700 1 -- 172.21.15.70:0/2611020230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ea81024f0 msgr2=0x7f1ea819fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:08.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1ea659c700 1 --2- 172.21.15.70:0/2611020230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ea81024f0 0x7f1ea819fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:08.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1ea659c700 1 -- 172.21.15.70:0/2611020230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ea8109520 msgr2=0x7f1ea81998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:08.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1ea659c700 1 --2- 172.21.15.70:0/2611020230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ea8109520 0x7f1ea81998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:08.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1ea659c700 1 -- 172.21.15.70:0/2611020230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea819a050 con 0x7f1ea8101b30 2024-06-04T05:15:08.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1ea5d9b700 1 --2- 172.21.15.70:0/2611020230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ea81024f0 0x7f1ea819fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:08.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.754+0000 7f1ea659c700 1 --2- 172.21.15.70:0/2611020230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ea8101b30 0x7f1ea819f730 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f1e9800e9c0 tx=0x7f1e9800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:08.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.755+0000 7f1e977fe700 1 -- 172.21.15.70:0/2611020230 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e98004a30 con 0x7f1ea8101b30 2024-06-04T05:15:08.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.755+0000 7f1eacdb7700 1 -- 172.21.15.70:0/2611020230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea819a340 con 0x7f1ea8101b30 2024-06-04T05:15:08.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.755+0000 7f1e977fe700 1 -- 172.21.15.70:0/2611020230 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e9800cc50 con 0x7f1ea8101b30 2024-06-04T05:15:08.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.755+0000 7f1eacdb7700 1 -- 172.21.15.70:0/2611020230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea8102c40 con 0x7f1ea8101b30 2024-06-04T05:15:08.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.755+0000 7f1e977fe700 1 -- 172.21.15.70:0/2611020230 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e98005420 con 0x7f1ea8101b30 2024-06-04T05:15:08.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.756+0000 7f1eacdb7700 1 -- 172.21.15.70:0/2611020230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ea806a230 con 0x7f1ea8101b30 2024-06-04T05:15:08.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.758+0000 7f1e977fe700 1 -- 172.21.15.70:0/2611020230 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f1e980055c0 con 0x7f1ea8101b30 2024-06-04T05:15:08.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.758+0000 7f1e977fe700 1 --2- 172.21.15.70:0/2611020230 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1e900b29f0 0x7f1e900b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:08.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.758+0000 7f1ea5d9b700 1 --2- 172.21.15.70:0/2611020230 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1e900b29f0 0x7f1e900b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:08.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.759+0000 7f1e977fe700 1 -- 172.21.15.70:0/2611020230 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f1e980e28a0 con 0x7f1ea8101b30 2024-06-04T05:15:08.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.759+0000 7f1ea5d9b700 1 --2- 172.21.15.70:0/2611020230 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1e900b29f0 0x7f1e900b4eb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1ea80ffc00 tx=0x7f1e9c009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:08.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.763+0000 7f1e977fe700 1 -- 172.21.15.70:0/2611020230 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1e98054df0 con 0x7f1ea8101b30 2024-06-04T05:15:08.807 INFO:tasks.workunit.client.0.smithi070.stdout: 10 34986 8.26 MB/sec execute 120 sec latency 83.112 ms 2024-06-04T05:15:08.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.984+0000 7f817655a700 1 -- 172.21.15.70:0/2096950375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8170103fa0 msgr2=0x7f8170110450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:08.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.984+0000 7f817655a700 1 --2- 172.21.15.70:0/2096950375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8170103fa0 0x7f8170110450 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8168009f90 tx=0x7f816800b330 comp rx=0 tx=0).stop 2024-06-04T05:15:08.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.985+0000 7f817655a700 1 -- 172.21.15.70:0/2096950375 shutdown_connections 2024-06-04T05:15:08.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.985+0000 7f817655a700 1 --2- 172.21.15.70:0/2096950375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8170103fa0 0x7f8170110450 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:08.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.985+0000 7f817655a700 1 --2- 172.21.15.70:0/2096950375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8170109680 0x7f8170109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:08.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.985+0000 7f817655a700 1 --2- 172.21.15.70:0/2096950375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81701035e0 0x7f8170103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:08.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.985+0000 7f817655a700 1 -- 172.21.15.70:0/2096950375 >> 172.21.15.70:0/2096950375 conn(0x7f817007a6f0 msgr2=0x7f817007ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:08.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.985+0000 7f817655a700 1 -- 172.21.15.70:0/2096950375 shutdown_connections 2024-06-04T05:15:08.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.985+0000 7f817655a700 1 -- 172.21.15.70:0/2096950375 wait complete. 2024-06-04T05:15:08.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.986+0000 7f817655a700 1 Processor -- start 2024-06-04T05:15:08.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.986+0000 7f817655a700 1 -- start start 2024-06-04T05:15:08.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.987+0000 7f817655a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81701035e0 0x7f81701a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:08.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.987+0000 7f817655a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8170103fa0 0x7f81701a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:08.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.987+0000 7f817655a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8170109680 0x7f817019bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:08.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.987+0000 7f817655a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8170077140 con 0x7f8170103fa0 2024-06-04T05:15:08.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.987+0000 7f817655a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8170076fc0 con 0x7f81701035e0 2024-06-04T05:15:08.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.987+0000 7f817655a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f81700772c0 con 0x7f8170109680 2024-06-04T05:15:08.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.987+0000 7f816ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81701035e0 0x7f81701a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:08.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.987+0000 7f8174af7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8170109680 0x7f817019bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:08.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.987+0000 7f816ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81701035e0 0x7f81701a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45594/0 (socket says 172.21.15.70:45594) 2024-06-04T05:15:08.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.987+0000 7f816ffff700 1 -- 172.21.15.70:0/2566755826 learned_addr learned my addr 172.21.15.70:0/2566755826 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:08.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.987+0000 7f816f7fe700 1 --2- 172.21.15.70:0/2566755826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8170103fa0 0x7f81701a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:08.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.987+0000 7f8174af7700 1 -- 172.21.15.70:0/2566755826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81701035e0 msgr2=0x7f81701a1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:08.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.987+0000 7f8174af7700 1 --2- 172.21.15.70:0/2566755826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81701035e0 0x7f81701a1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:08.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.987+0000 7f8174af7700 1 -- 172.21.15.70:0/2566755826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8170103fa0 msgr2=0x7f81701a1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:08.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.987+0000 7f8174af7700 1 --2- 172.21.15.70:0/2566755826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8170103fa0 0x7f81701a1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:08.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.987+0000 7f8174af7700 1 -- 172.21.15.70:0/2566755826 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817019c440 con 0x7f8170109680 2024-06-04T05:15:08.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.988+0000 7f8174af7700 1 --2- 172.21.15.70:0/2566755826 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8170109680 0x7f817019bb60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f816800b6c0 tx=0x7f816800cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:08.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.988+0000 7f816d7fa700 1 -- 172.21.15.70:0/2566755826 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8168021070 con 0x7f8170109680 2024-06-04T05:15:08.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.988+0000 7f816d7fa700 1 -- 172.21.15.70:0/2566755826 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f816801d450 con 0x7f8170109680 2024-06-04T05:15:08.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.988+0000 7f817655a700 1 -- 172.21.15.70:0/2566755826 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f817019c6d0 con 0x7f8170109680 2024-06-04T05:15:08.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.988+0000 7f817655a700 1 -- 172.21.15.70:0/2566755826 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81701a89c0 con 0x7f8170109680 2024-06-04T05:15:08.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.988+0000 7f816d7fa700 1 -- 172.21.15.70:0/2566755826 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8168007d40 con 0x7f8170109680 2024-06-04T05:15:08.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.992+0000 7f816d7fa700 1 -- 172.21.15.70:0/2566755826 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f8168024070 con 0x7f8170109680 2024-06-04T05:15:08.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.992+0000 7f8180defb80 1 Processor -- start 2024-06-04T05:15:08.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.992+0000 7f8180defb80 1 -- start start 2024-06-04T05:15:08.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.992+0000 7f8180defb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f70ab30 0x56244f70af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:08.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.992+0000 7f8180defb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f7050b0 0x56244f705530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:08.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.992+0000 7f816f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f70ab30 0x56244f70af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:08.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.992+0000 7f816f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f70ab30 0x56244f70af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45608/0 (socket says 172.21.15.70:45608) 2024-06-04T05:15:08.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.993+0000 7f8180defb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f705a70 0x56244f711890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:08.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.993+0000 7f8180defb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56244f65ad70 con 0x56244f705a70 2024-06-04T05:15:08.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.993+0000 7f8180defb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56244f64a910 con 0x56244f70ab30 2024-06-04T05:15:08.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.993+0000 7f816f7fe700 1 -- 172.21.15.70:0/1874267279 learned_addr learned my addr 172.21.15.70:0/1874267279 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:08.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.993+0000 7f8180defb80 1 -- 172.21.15.70:0/1874267279 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56244f64e950 con 0x56244f7050b0 2024-06-04T05:15:08.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.993+0000 7f816ffff700 1 --2- 172.21.15.70:0/1874267279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f7050b0 0x56244f705530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:08.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.993+0000 7f816f7fe700 1 --2- 172.21.15.70:0/1874267279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f705a70 0x56244f711890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:08.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.993+0000 7f816f7fe700 1 -- 172.21.15.70:0/1874267279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f7050b0 msgr2=0x56244f705530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:08.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.993+0000 7f816f7fe700 1 --2- 172.21.15.70:0/1874267279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f7050b0 0x56244f705530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:08.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.993+0000 7f816f7fe700 1 -- 172.21.15.70:0/1874267279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f705a70 msgr2=0x56244f711890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:08.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.993+0000 7f816f7fe700 1 --2- 172.21.15.70:0/1874267279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f705a70 0x56244f711890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:08.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.993+0000 7f816f7fe700 1 -- 172.21.15.70:0/1874267279 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56244f5f3fb0 con 0x56244f70ab30 2024-06-04T05:15:08.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.993+0000 7f816d7fa700 1 --2- 172.21.15.70:0/2566755826 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81580b2a70 0x7f81580b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:08.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.993+0000 7f816f7fe700 1 --2- 172.21.15.70:0/1874267279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f70ab30 0x56244f70af10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f81640136c0 tx=0x7f81640098c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b80f915aabe45f4c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:08.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.994+0000 7f816d7fa700 1 -- 172.21.15.70:0/2566755826 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f816800f020 con 0x7f8170109680 2024-06-04T05:15:08.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.994+0000 7f81567fc700 1 -- 172.21.15.70:0/1874267279 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8164004050 con 0x56244f70ab30 2024-06-04T05:15:08.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.994+0000 7f81567fc700 1 -- 172.21.15.70:0/1874267279 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8164003680 con 0x56244f70ab30 2024-06-04T05:15:08.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.994+0000 7f816ffff700 1 --2- 172.21.15.70:0/2566755826 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81580b2a70 0x7f81580b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:08.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.994+0000 7f81567fc700 1 -- 172.21.15.70:0/1874267279 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8164010690 con 0x56244f70ab30 2024-06-04T05:15:08.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.995+0000 7f816ffff700 1 --2- 172.21.15.70:0/2566755826 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81580b2a70 0x7f81580b4f30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8160005950 tx=0x7f8160009310 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:08.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.995+0000 7f8180defb80 1 -- 172.21.15.70:0/1874267279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f70ab30 msgr2=0x56244f70af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:08.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.995+0000 7f8180defb80 1 --2- 172.21.15.70:0/1874267279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f70ab30 0x56244f70af10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f81640136c0 tx=0x7f81640098c0 comp rx=0 tx=0).stop 2024-06-04T05:15:08.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.995+0000 7f8180defb80 1 -- 172.21.15.70:0/1874267279 shutdown_connections 2024-06-04T05:15:08.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.995+0000 7f8180defb80 1 --2- 172.21.15.70:0/1874267279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f705a70 0x56244f711890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:08.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.995+0000 7f8180defb80 1 --2- 172.21.15.70:0/1874267279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f70ab30 0x56244f70af10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:08.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.995+0000 7f8180defb80 1 --2- 172.21.15.70:0/1874267279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f7050b0 0x56244f705530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:08.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.995+0000 7f8180defb80 1 -- 172.21.15.70:0/1874267279 >> 172.21.15.70:0/1874267279 conn(0x56244f628240 msgr2=0x56244f6f73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:08.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.995+0000 7f8180defb80 1 -- 172.21.15.70:0/1874267279 shutdown_connections 2024-06-04T05:15:08.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.995+0000 7f8180defb80 1 -- 172.21.15.70:0/1874267279 wait complete. 2024-06-04T05:15:08.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.995+0000 7f8180defb80 1 -- start start 2024-06-04T05:15:08.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.996+0000 7f8180defb80 1 Processor -- start 2024-06-04T05:15:08.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.996+0000 7f8180defb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f7050b0 0x56244f7114f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:09.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.996+0000 7f8180defb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f705a70 0x56244f70e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:09.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.996+0000 7f816ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f7050b0 0x56244f7114f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:09.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.996+0000 7f816ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f7050b0 0x56244f7114f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51874/0 (socket says 172.21.15.70:51874) 2024-06-04T05:15:09.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.996+0000 7f816ffff700 1 -- 172.21.15.70:0/2016843591 learned_addr learned my addr 172.21.15.70:0/2016843591 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:09.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.996+0000 7f8180defb80 1 --2- 172.21.15.70:0/2016843591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f70ab30 0x56244f70ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:09.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.996+0000 7f8180defb80 1 -- 172.21.15.70:0/2016843591 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56244f550fa0 con 0x56244f7050b0 2024-06-04T05:15:09.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.996+0000 7f8180defb80 1 -- 172.21.15.70:0/2016843591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56244f5b2da0 con 0x56244f70ab30 2024-06-04T05:15:09.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.996+0000 7f8180defb80 1 -- 172.21.15.70:0/2016843591 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56244f711ef0 con 0x56244f705a70 2024-06-04T05:15:09.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.996+0000 7f8180defb80 10 client.0 _ll_register_callbacks cb 0x56244f5e5950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:09.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.996+0000 7f816f7fe700 1 --2- 172.21.15.70:0/2016843591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f705a70 0x56244f70e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:09.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.997+0000 7f8174af7700 1 --2- 172.21.15.70:0/2016843591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f70ab30 0x56244f70ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:09.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.997+0000 7f816ffff700 1 -- 172.21.15.70:0/2016843591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f705a70 msgr2=0x56244f70e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:09.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.997+0000 7f816ffff700 1 --2- 172.21.15.70:0/2016843591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f705a70 0x56244f70e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.997+0000 7f816ffff700 1 -- 172.21.15.70:0/2016843591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f70ab30 msgr2=0x56244f70ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:09.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.997+0000 7f816ffff700 1 --2- 172.21.15.70:0/2016843591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f70ab30 0x56244f70ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.997+0000 7f816ffff700 1 -- 172.21.15.70:0/2016843591 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56244f59a8d0 con 0x56244f7050b0 2024-06-04T05:15:09.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.997+0000 7f816ffff700 1 --2- 172.21.15.70:0/2016843591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f7050b0 0x56244f7114f0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f816001ea00 tx=0x7f81600047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca187190ed69d16 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:09.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.997+0000 7f814f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:15:09.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.997+0000 7f814f7fe700 1 -- 172.21.15.70:0/2016843591 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816002e060 con 0x56244f7050b0 2024-06-04T05:15:09.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.997+0000 7f814f7fe700 1 -- 172.21.15.70:0/2016843591 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f8160018030 con 0x56244f7050b0 2024-06-04T05:15:09.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.997+0000 7f8180defb80 1 -- 172.21.15.70:0/2016843591 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8164003680 con 0x56244f7050b0 2024-06-04T05:15:09.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.998+0000 7f814f7fe700 1 -- 172.21.15.70:0/2016843591 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f8160012c50 con 0x56244f7050b0 2024-06-04T05:15:09.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.998+0000 7f8180defb80 10 client.20079 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:09.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.998+0000 7f8180defb80 1 -- 172.21.15.70:0/2016843591 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x56244f604760 con 0x56244f7050b0 2024-06-04T05:15:09.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.998+0000 7f814f7fe700 1 -- 172.21.15.70:0/2016843591 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8160012df0 con 0x56244f7050b0 2024-06-04T05:15:09.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.998+0000 7f814f7fe700 1 -- 172.21.15.70:0/2016843591 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f8160029820 con 0x56244f7050b0 2024-06-04T05:15:09.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.999+0000 7f8180defb80 10 client.20079 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:09.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.999+0000 7f8180defb80 10 client.20079 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:09.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.999+0000 7f8180defb80 20 client.20079 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:09.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.999+0000 7f8180defb80 1 --2- 172.21.15.70:0/2016843591 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56244f71e960 0x56244f720d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:09.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.999+0000 7f8180defb80 4 client.20079 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:15:09.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.999+0000 7f8180defb80 1 -- 172.21.15.70:0/2016843591 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8164003680 con 0x56244f71e960 2024-06-04T05:15:09.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:08.999+0000 7f816f7fe700 1 --2- 172.21.15.70:0/2016843591 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56244f71e960 0x56244f720d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:09.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.000+0000 7f814f7fe700 10 client.20079 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:09.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.000+0000 7f816f7fe700 1 --2- 172.21.15.70:0/2016843591 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56244f71e960 0x56244f720d90 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90dad9595c7e0080 server_cookie=66619a458a29d9da in_seq=0 out_seq=0 2024-06-04T05:15:09.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.001+0000 7f814f7fe700 1 -- 172.21.15.70:0/2016843591 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f8164003680 con 0x56244f71e960 2024-06-04T05:15:09.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.001+0000 7f814f7fe700 10 client.20079 handle_command_reply: tid=0 2024-06-04T05:15:09.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.001+0000 7f8180defb80 1 client.20079 shutdown 2024-06-04T05:15:09.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.001+0000 7f8180defb80 1 -- 172.21.15.70:0/2016843591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f7050b0 msgr2=0x56244f7114f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:09.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.001+0000 7f8180defb80 1 --2- 172.21.15.70:0/2016843591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f7050b0 0x56244f7114f0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f816001ea00 tx=0x7f81600047c0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.001+0000 7f8180defb80 1 -- 172.21.15.70:0/2016843591 shutdown_connections 2024-06-04T05:15:09.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.001+0000 7f8180defb80 1 --2- 172.21.15.70:0/2016843591 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56244f71e960 0x56244f720d90 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.001+0000 7f8180defb80 1 --2- 172.21.15.70:0/2016843591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f70ab30 0x56244f70ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.001+0000 7f814f7fe700 0 client.20079 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:09.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.001+0000 7f8180defb80 1 --2- 172.21.15.70:0/2016843591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f7050b0 0x56244f7114f0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.001+0000 7f8180defb80 1 --2- 172.21.15.70:0/2016843591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f705a70 0x56244f70e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.001+0000 7f8180defb80 1 -- 172.21.15.70:0/2016843591 >> 172.21.15.70:0/2016843591 conn(0x56244f628240 msgr2=0x56244f6f6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:09.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.002+0000 7f8180defb80 1 -- 172.21.15.70:0/2016843591 shutdown_connections 2024-06-04T05:15:09.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.002+0000 7f8180defb80 1 -- 172.21.15.70:0/2016843591 wait complete. 2024-06-04T05:15:09.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.002+0000 7f8180defb80 20 client.20079 trim_cache size 0 max 16384 2024-06-04T05:15:09.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.038+0000 7f8180defb80 1 Processor -- start 2024-06-04T05:15:09.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.038+0000 7f8180defb80 1 -- start start 2024-06-04T05:15:09.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.039+0000 7f8180defb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f7050b0 0x56244f71afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:09.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.039+0000 7f8180defb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f705a70 0x56244f70edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:09.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.039+0000 7f816ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f7050b0 0x56244f71afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:09.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.039+0000 7f816ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f7050b0 0x56244f71afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:34774/0 (socket says 172.21.15.70:34774) 2024-06-04T05:15:09.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.040+0000 7f816ffff700 1 -- 172.21.15.70:0/3948126343 learned_addr learned my addr 172.21.15.70:0/3948126343 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:09.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.040+0000 7f8180defb80 1 --2- 172.21.15.70:0/3948126343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f70ab30 0x56244f708df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:09.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.040+0000 7f816f7fe700 1 --2- 172.21.15.70:0/3948126343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f705a70 0x56244f70edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:09.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.040+0000 7f8180defb80 1 -- 172.21.15.70:0/3948126343 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f815c00d6a0 con 0x56244f705a70 2024-06-04T05:15:09.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.040+0000 7f8180defb80 1 -- 172.21.15.70:0/3948126343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f815c00f240 con 0x56244f70ab30 2024-06-04T05:15:09.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.040+0000 7f8180defb80 1 -- 172.21.15.70:0/3948126343 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f815c00ddd0 con 0x56244f7050b0 2024-06-04T05:15:09.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.040+0000 7f8174af7700 1 --2- 172.21.15.70:0/3948126343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f70ab30 0x56244f708df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:09.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.040+0000 7f816ffff700 1 -- 172.21.15.70:0/3948126343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f70ab30 msgr2=0x56244f708df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:09.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.040+0000 7f816ffff700 1 --2- 172.21.15.70:0/3948126343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f70ab30 0x56244f708df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.040+0000 7f816ffff700 1 -- 172.21.15.70:0/3948126343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f705a70 msgr2=0x56244f70edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:09.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.040+0000 7f816ffff700 1 --2- 172.21.15.70:0/3948126343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f705a70 0x56244f70edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.040+0000 7f816ffff700 1 -- 172.21.15.70:0/3948126343 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56244f711cf0 con 0x56244f7050b0 2024-06-04T05:15:09.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.041+0000 7f816ffff700 1 --2- 172.21.15.70:0/3948126343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f7050b0 0x56244f71afe0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8160012b40 tx=0x7f8160031570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fc88a674ecc6144 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:09.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.041+0000 7f8155ffb700 1 -- 172.21.15.70:0/3948126343 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160041070 con 0x56244f7050b0 2024-06-04T05:15:09.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.041+0000 7f8155ffb700 1 -- 172.21.15.70:0/3948126343 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8160012df0 con 0x56244f7050b0 2024-06-04T05:15:09.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.041+0000 7f8155ffb700 1 -- 172.21.15.70:0/3948126343 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816000fce0 con 0x56244f7050b0 2024-06-04T05:15:09.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.042+0000 7f8180defb80 1 -- 172.21.15.70:0/3948126343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f7050b0 msgr2=0x56244f71afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:09.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.042+0000 7f8180defb80 1 --2- 172.21.15.70:0/3948126343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f7050b0 0x56244f71afe0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8160012b40 tx=0x7f8160031570 comp rx=0 tx=0).stop 2024-06-04T05:15:09.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.042+0000 7f8180defb80 1 -- 172.21.15.70:0/3948126343 shutdown_connections 2024-06-04T05:15:09.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.042+0000 7f8180defb80 1 --2- 172.21.15.70:0/3948126343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f70ab30 0x56244f708df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.042+0000 7f8180defb80 1 --2- 172.21.15.70:0/3948126343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f7050b0 0x56244f71afe0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.042+0000 7f8180defb80 1 --2- 172.21.15.70:0/3948126343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f705a70 0x56244f70edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.042+0000 7f8180defb80 1 -- 172.21.15.70:0/3948126343 >> 172.21.15.70:0/3948126343 conn(0x56244f628240 msgr2=0x56244f710020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:09.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.042+0000 7f8180defb80 1 -- 172.21.15.70:0/3948126343 shutdown_connections 2024-06-04T05:15:09.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.042+0000 7f8180defb80 1 -- 172.21.15.70:0/3948126343 wait complete. 2024-06-04T05:15:09.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.043+0000 7f8180defb80 1 -- start start 2024-06-04T05:15:09.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.043+0000 7f8180defb80 1 Processor -- start 2024-06-04T05:15:09.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.043+0000 7f8180defb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f7050b0 0x56244f7171d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:09.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.043+0000 7f8180defb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f705a70 0x56244f717710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:09.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.043+0000 7f816ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f7050b0 0x56244f7171d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:09.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.043+0000 7f8180defb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f70ab30 0x56244f7141a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:09.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.043+0000 7f816ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f7050b0 0x56244f7171d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45642/0 (socket says 172.21.15.70:45642) 2024-06-04T05:15:09.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.044+0000 7f816f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f705a70 0x56244f717710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:09.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.044+0000 7f816ffff700 1 -- 172.21.15.70:0/2973318141 learned_addr learned my addr 172.21.15.70:0/2973318141 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:09.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.044+0000 7f8180defb80 1 -- 172.21.15.70:0/2973318141 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f815c00eb70 con 0x56244f705a70 2024-06-04T05:15:09.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.044+0000 7f8180defb80 1 -- 172.21.15.70:0/2973318141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f815c00e4a0 con 0x56244f7050b0 2024-06-04T05:15:09.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.044+0000 7f8180defb80 1 -- 172.21.15.70:0/2973318141 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f815c00b660 con 0x56244f70ab30 2024-06-04T05:15:09.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.044+0000 7f8180defb80 10 client.0 _ll_register_callbacks cb 0x56244f5e5950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:09.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.044+0000 7f8174af7700 1 --2- 172.21.15.70:0/2973318141 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f70ab30 0x56244f7141a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:09.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.044+0000 7f816ffff700 1 -- 172.21.15.70:0/2973318141 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f70ab30 msgr2=0x56244f7141a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:09.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.044+0000 7f816ffff700 1 --2- 172.21.15.70:0/2973318141 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f70ab30 0x56244f7141a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.044+0000 7f816ffff700 1 -- 172.21.15.70:0/2973318141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f705a70 msgr2=0x56244f717710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:09.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.044+0000 7f816ffff700 1 --2- 172.21.15.70:0/2973318141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f705a70 0x56244f717710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.044+0000 7f816ffff700 1 -- 172.21.15.70:0/2973318141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56244f7147f0 con 0x56244f7050b0 2024-06-04T05:15:09.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.044+0000 7f8174af7700 1 --2- 172.21.15.70:0/2973318141 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f70ab30 0x56244f7141a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:09.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.044+0000 7f816f7fe700 1 --2- 172.21.15.70:0/2973318141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f705a70 0x56244f717710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:09.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.045+0000 7f816ffff700 1 --2- 172.21.15.70:0/2973318141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f7050b0 0x56244f7171d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8160029c40 tx=0x7f8160029970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fde10dca0d207297 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:09.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.045+0000 7f814f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:15:09.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.045+0000 7f814f7fe700 1 -- 172.21.15.70:0/2973318141 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160041070 con 0x56244f7050b0 2024-06-04T05:15:09.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.045+0000 7f8180defb80 1 -- 172.21.15.70:0/2973318141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8160012df0 con 0x56244f7050b0 2024-06-04T05:15:09.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.045+0000 7f814f7fe700 1 -- 172.21.15.70:0/2973318141 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f8160005250 con 0x56244f7050b0 2024-06-04T05:15:09.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.045+0000 7f814f7fe700 1 -- 172.21.15.70:0/2973318141 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f816002e040 con 0x56244f7050b0 2024-06-04T05:15:09.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.045+0000 7f8180defb80 10 client.28901 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:09.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.045+0000 7f8180defb80 1 -- 172.21.15.70:0/2973318141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x56244f789eb0 con 0x56244f7050b0 2024-06-04T05:15:09.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.045+0000 7f814f7fe700 1 -- 172.21.15.70:0/2973318141 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8160012df0 con 0x56244f7050b0 2024-06-04T05:15:09.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.046+0000 7f814f7fe700 1 -- 172.21.15.70:0/2973318141 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f816004c070 con 0x56244f7050b0 2024-06-04T05:15:09.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.046+0000 7f8180defb80 10 client.28901 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:09.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.046+0000 7f8180defb80 10 client.28901 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:09.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.046+0000 7f8180defb80 20 client.28901 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:09.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.046+0000 7f8180defb80 1 --2- 172.21.15.70:0/2973318141 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56244f71e960 0x56244f78c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:09.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.046+0000 7f8180defb80 4 client.28901 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:15:09.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.046+0000 7f8180defb80 1 -- 172.21.15.70:0/2973318141 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56244f65ccc0 con 0x56244f71e960 2024-06-04T05:15:09.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.046+0000 7f816f7fe700 1 --2- 172.21.15.70:0/2973318141 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56244f71e960 0x56244f78c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:09.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.046+0000 7f816f7fe700 1 --2- 172.21.15.70:0/2973318141 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56244f71e960 0x56244f78c490 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc1ef8d65a4d135f server_cookie=33384488995660cb in_seq=0 out_seq=0 2024-06-04T05:15:09.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.046+0000 7f814f7fe700 10 client.28901 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:09.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.046+0000 7f814f7fe700 1 -- 172.21.15.70:0/2973318141 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56244f65ccc0 con 0x56244f71e960 2024-06-04T05:15:09.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.046+0000 7f814f7fe700 10 client.28901 handle_command_reply: tid=0 2024-06-04T05:15:09.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.046+0000 7f8180defb80 1 client.28901 shutdown 2024-06-04T05:15:09.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.047+0000 7f8180defb80 1 -- 172.21.15.70:0/2973318141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f7050b0 msgr2=0x56244f7171d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:09.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.047+0000 7f8180defb80 1 --2- 172.21.15.70:0/2973318141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f7050b0 0x56244f7171d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8160029c40 tx=0x7f8160029970 comp rx=0 tx=0).stop 2024-06-04T05:15:09.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.047+0000 7f8180defb80 1 -- 172.21.15.70:0/2973318141 shutdown_connections 2024-06-04T05:15:09.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.047+0000 7f8180defb80 1 --2- 172.21.15.70:0/2973318141 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56244f70ab30 0x56244f7141a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.047+0000 7f8180defb80 1 --2- 172.21.15.70:0/2973318141 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56244f71e960 0x56244f78c490 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.053 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:15:09.053 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:15:09.053 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:15:09.053 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:15:09.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.047+0000 7f8180defb80 1 --2- 172.21.15.70:0/2973318141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56244f7050b0 0x56244f7171d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.047+0000 7f8180defb80 1 --2- 172.21.15.70:0/2973318141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56244f705a70 0x56244f717710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.047+0000 7f8180defb80 1 -- 172.21.15.70:0/2973318141 >> 172.21.15.70:0/2973318141 conn(0x56244f628240 msgr2=0x56244f704930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:09.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.047+0000 7f814f7fe700 0 client.28901 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:09.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.047+0000 7f8180defb80 1 -- 172.21.15.70:0/2973318141 shutdown_connections 2024-06-04T05:15:09.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.047+0000 7f8180defb80 1 -- 172.21.15.70:0/2973318141 wait complete. 2024-06-04T05:15:09.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.047+0000 7f8180defb80 20 client.28901 trim_cache size 0 max 16384 2024-06-04T05:15:09.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.048+0000 7f817655a700 1 -- 172.21.15.70:0/2566755826 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81580b2a70 msgr2=0x7f81580b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:09.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.048+0000 7f817655a700 1 --2- 172.21.15.70:0/2566755826 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81580b2a70 0x7f81580b4f30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8160005950 tx=0x7f8160009310 comp rx=0 tx=0).stop 2024-06-04T05:15:09.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.048+0000 7f817655a700 1 -- 172.21.15.70:0/2566755826 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8170109680 msgr2=0x7f817019bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:09.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.048+0000 7f817655a700 1 --2- 172.21.15.70:0/2566755826 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8170109680 0x7f817019bb60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f816800b6c0 tx=0x7f816800cc30 comp rx=0 tx=0).stop 2024-06-04T05:15:09.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.048+0000 7f817655a700 1 -- 172.21.15.70:0/2566755826 shutdown_connections 2024-06-04T05:15:09.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.048+0000 7f817655a700 1 --2- 172.21.15.70:0/2566755826 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81580b2a70 0x7f81580b4f30 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.048+0000 7f817655a700 1 --2- 172.21.15.70:0/2566755826 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8170109680 0x7f817019bb60 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.048+0000 7f817655a700 1 --2- 172.21.15.70:0/2566755826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81701035e0 0x7f81701a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.048+0000 7f817655a700 1 --2- 172.21.15.70:0/2566755826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8170103fa0 0x7f81701a1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.048+0000 7f817655a700 1 -- 172.21.15.70:0/2566755826 >> 172.21.15.70:0/2566755826 conn(0x7f817007a6f0 msgr2=0x7f81701020b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:09.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.048+0000 7f817655a700 1 -- 172.21.15.70:0/2566755826 shutdown_connections 2024-06-04T05:15:09.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.048+0000 7f817655a700 1 -- 172.21.15.70:0/2566755826 wait complete. 2024-06-04T05:15:09.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.214+0000 7f1eacdb7700 1 -- 172.21.15.70:0/2611020230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ea802e600 con 0x7f1ea8101b30 2024-06-04T05:15:09.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.215+0000 7f1e977fe700 1 -- 172.21.15.70:0/2611020230 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+11932 (secure 0 0 0) 0x7f1e980aa250 con 0x7f1ea8101b30 2024-06-04T05:15:09.217 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:15:09.217 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":55,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:14:18.668019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:15:09.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.218+0000 7f1eacdb7700 1 -- 172.21.15.70:0/2611020230 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1e900b29f0 msgr2=0x7f1e900b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:09.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.218+0000 7f1eacdb7700 1 --2- 172.21.15.70:0/2611020230 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1e900b29f0 0x7f1e900b4eb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1ea80ffc00 tx=0x7f1e9c009260 comp rx=0 tx=0).stop 2024-06-04T05:15:09.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.218+0000 7f1eacdb7700 1 -- 172.21.15.70:0/2611020230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ea8101b30 msgr2=0x7f1ea819f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:09.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.218+0000 7f1eacdb7700 1 --2- 172.21.15.70:0/2611020230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ea8101b30 0x7f1ea819f730 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f1e9800e9c0 tx=0x7f1e9800ecd0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.218+0000 7f1eacdb7700 1 -- 172.21.15.70:0/2611020230 shutdown_connections 2024-06-04T05:15:09.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.218+0000 7f1eacdb7700 1 --2- 172.21.15.70:0/2611020230 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1e900b29f0 0x7f1e900b4eb0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.218+0000 7f1eacdb7700 1 --2- 172.21.15.70:0/2611020230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ea8109520 0x7f1ea81998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.218+0000 7f1eacdb7700 1 --2- 172.21.15.70:0/2611020230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ea8101b30 0x7f1ea819f730 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.218+0000 7f1eacdb7700 1 --2- 172.21.15.70:0/2611020230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ea81024f0 0x7f1ea819fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:09.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.218+0000 7f1eacdb7700 1 -- 172.21.15.70:0/2611020230 >> 172.21.15.70:0/2611020230 conn(0x7f1ea8078ab0 msgr2=0x7f1ea8100400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:09.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.218+0000 7f1eacdb7700 1 -- 172.21.15.70:0/2611020230 shutdown_connections 2024-06-04T05:15:09.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:09.218+0000 7f1eacdb7700 1 -- 172.21.15.70:0/2611020230 wait complete. 2024-06-04T05:15:09.221 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 55 2024-06-04T05:15:09.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:09 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:15:09.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:09 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:15:09.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:09 smithi070 ceph-mon[32631]: pgmap v564: 97 pgs: 97 active+clean; 599 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.8 MiB/s wr, 396 op/s 2024-06-04T05:15:09.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:09 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2611020230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:15:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:09 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:15:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:09 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:15:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:09 smithi112 ceph-mon[36453]: pgmap v564: 97 pgs: 97 active+clean; 599 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.8 MiB/s wr, 396 op/s 2024-06-04T05:15:09.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:09 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2611020230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:15:09.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:09 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:15:09.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:09 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:15:09.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:09 smithi098 ceph-mon[35569]: pgmap v564: 97 pgs: 97 active+clean; 599 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.8 MiB/s wr, 396 op/s 2024-06-04T05:15:09.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:09 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2611020230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:15:09.807 INFO:tasks.workunit.client.0.smithi070.stdout: 10 35173 8.27 MB/sec execute 121 sec latency 66.687 ms 2024-06-04T05:15:10.105 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:15:10.105 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:15:10.665 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 130 --await-for 45.0' 2024-06-04T05:15:10.666 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 130 --await-for 45.0 2024-06-04T05:15:10.792 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:15:10.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 35374 8.28 MB/sec execute 122 sec latency 146.334 ms 2024-06-04T05:15:11.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:11 smithi070 ceph-mon[32631]: pgmap v565: 97 pgs: 97 active+clean; 599 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 196 op/s 2024-06-04T05:15:11.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:11 smithi112 ceph-mon[36453]: pgmap v565: 97 pgs: 97 active+clean; 599 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 196 op/s 2024-06-04T05:15:11.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:11 smithi098 ceph-mon[35569]: pgmap v565: 97 pgs: 97 active+clean; 599 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 196 op/s 2024-06-04T05:15:11.807 INFO:tasks.workunit.client.0.smithi070.stdout: 10 35562 8.28 MB/sec execute 123 sec latency 93.320 ms 2024-06-04T05:15:12.050 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:15:12.815 INFO:tasks.workunit.client.0.smithi070.stdout: 10 35719 8.28 MB/sec execute 124 sec latency 101.935 ms 2024-06-04T05:15:13.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.147+0000 7f20a067d700 1 -- 172.21.15.70:0/310709391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2098103f80 msgr2=0x7f2098110410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:13.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.147+0000 7f20a067d700 1 --2- 172.21.15.70:0/310709391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2098103f80 0x7f2098110410 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f209000b0a0 tx=0x7f209000b3b0 comp rx=0 tx=0).stop 2024-06-04T05:15:13.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.149+0000 7f20a067d700 1 -- 172.21.15.70:0/310709391 shutdown_connections 2024-06-04T05:15:13.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.149+0000 7f20a067d700 1 --2- 172.21.15.70:0/310709391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2098103f80 0x7f2098110410 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:13.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.149+0000 7f20a067d700 1 --2- 172.21.15.70:0/310709391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20981095d0 0x7f20981099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:13.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.149+0000 7f20a067d700 1 --2- 172.21.15.70:0/310709391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20981035c0 0x7f2098103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:13.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.149+0000 7f20a067d700 1 -- 172.21.15.70:0/310709391 >> 172.21.15.70:0/310709391 conn(0x7f209807a610 msgr2=0x7f209807aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:13.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.149+0000 7f20a067d700 1 -- 172.21.15.70:0/310709391 shutdown_connections 2024-06-04T05:15:13.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.150+0000 7f20a067d700 1 -- 172.21.15.70:0/310709391 wait complete. 2024-06-04T05:15:13.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.151+0000 7f20a067d700 1 Processor -- start 2024-06-04T05:15:13.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.151+0000 7f20a067d700 1 -- start start 2024-06-04T05:15:13.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.151+0000 7f20a067d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20981035c0 0x7f2098074c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:13.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.151+0000 7f20a067d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2098103f80 0x7f20980751d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:13.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.151+0000 7f20a067d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20981095d0 0x7f209806df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:13.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.151+0000 7f20a067d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2098113630 con 0x7f20981035c0 2024-06-04T05:15:13.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.151+0000 7f20a067d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f20981134b0 con 0x7f20981095d0 2024-06-04T05:15:13.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.151+0000 7f20a067d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f20981137b0 con 0x7f2098103f80 2024-06-04T05:15:13.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.151+0000 7f209ec1a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20981095d0 0x7f209806df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:13.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.151+0000 7f209ec1a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20981095d0 0x7f209806df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45654/0 (socket says 172.21.15.70:45654) 2024-06-04T05:15:13.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.151+0000 7f209ec1a700 1 -- 172.21.15.70:0/1969463802 learned_addr learned my addr 172.21.15.70:0/1969463802 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:13.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.152+0000 7f209e419700 1 --2- 172.21.15.70:0/1969463802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20981035c0 0x7f2098074c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:13.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.152+0000 7f209ec1a700 1 -- 172.21.15.70:0/1969463802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2098103f80 msgr2=0x7f20980751d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-04T05:15:13.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.152+0000 7f209ec1a700 1 --2- 172.21.15.70:0/1969463802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2098103f80 0x7f20980751d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:13.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.152+0000 7f209ec1a700 1 -- 172.21.15.70:0/1969463802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20981035c0 msgr2=0x7f2098074c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:13.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.152+0000 7f209ec1a700 1 --2- 172.21.15.70:0/1969463802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20981035c0 0x7f2098074c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:13.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.152+0000 7f209ec1a700 1 -- 172.21.15.70:0/1969463802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f209806e870 con 0x7f20981095d0 2024-06-04T05:15:13.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.152+0000 7f209ec1a700 1 --2- 172.21.15.70:0/1969463802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20981095d0 0x7f209806df70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2090004d00 tx=0x7f2090004d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:13.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.152+0000 7f208f7fe700 1 -- 172.21.15.70:0/1969463802 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2090021070 con 0x7f20981095d0 2024-06-04T05:15:13.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.152+0000 7f208f7fe700 1 -- 172.21.15.70:0/1969463802 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2090007b80 con 0x7f20981095d0 2024-06-04T05:15:13.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.153+0000 7f208f7fe700 1 -- 172.21.15.70:0/1969463802 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20900052a0 con 0x7f20981095d0 2024-06-04T05:15:13.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.153+0000 7f20a067d700 1 -- 172.21.15.70:0/1969463802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f209806eaa0 con 0x7f20981095d0 2024-06-04T05:15:13.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.153+0000 7f20a067d700 1 -- 172.21.15.70:0/1969463802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2098072990 con 0x7f20981095d0 2024-06-04T05:15:13.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.155+0000 7f20a067d700 1 -- 172.21.15.70:0/1969463802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2098104cb0 con 0x7f20981095d0 2024-06-04T05:15:13.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.156+0000 7f208f7fe700 1 -- 172.21.15.70:0/1969463802 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f2090024070 con 0x7f20981095d0 2024-06-04T05:15:13.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.157+0000 7f208f7fe700 1 --2- 172.21.15.70:0/1969463802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f20840b2bc0 0x7f20840b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:13.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.157+0000 7f208f7fe700 1 -- 172.21.15.70:0/1969463802 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f20900e2b60 con 0x7f20981095d0 2024-06-04T05:15:13.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.157+0000 7f209e419700 1 --2- 172.21.15.70:0/1969463802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f20840b2bc0 0x7f20840b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:13.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.158+0000 7f209e419700 1 --2- 172.21.15.70:0/1969463802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f20840b2bc0 0x7f20840b5080 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2088005950 tx=0x7f2088017040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:13.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.163+0000 7f208f7fe700 1 -- 172.21.15.70:0/1969463802 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f20900aa540 con 0x7f20981095d0 2024-06-04T05:15:13.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.575+0000 7f20a067d700 1 -- 172.21.15.70:0/1969463802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20980008d0 con 0x7f20981095d0 2024-06-04T05:15:13.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.576+0000 7f208f7fe700 1 -- 172.21.15.70:0/1969463802 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+11932 (secure 0 0 0) 0x7f20900a9e40 con 0x7f20981095d0 2024-06-04T05:15:13.579 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:15:13.579 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":55,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:14:18.668019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:15:13.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.579+0000 7f20a067d700 1 -- 172.21.15.70:0/1969463802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f20840b2bc0 msgr2=0x7f20840b5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:13.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.579+0000 7f20a067d700 1 --2- 172.21.15.70:0/1969463802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f20840b2bc0 0x7f20840b5080 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2088005950 tx=0x7f2088017040 comp rx=0 tx=0).stop 2024-06-04T05:15:13.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.579+0000 7f20a067d700 1 -- 172.21.15.70:0/1969463802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20981095d0 msgr2=0x7f209806df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:13.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.579+0000 7f20a067d700 1 --2- 172.21.15.70:0/1969463802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20981095d0 0x7f209806df70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2090004d00 tx=0x7f2090004d30 comp rx=0 tx=0).stop 2024-06-04T05:15:13.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.580+0000 7f20a067d700 1 -- 172.21.15.70:0/1969463802 shutdown_connections 2024-06-04T05:15:13.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.580+0000 7f20a067d700 1 --2- 172.21.15.70:0/1969463802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f20840b2bc0 0x7f20840b5080 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:13.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.580+0000 7f20a067d700 1 --2- 172.21.15.70:0/1969463802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20981095d0 0x7f209806df70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:13.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.580+0000 7f20a067d700 1 --2- 172.21.15.70:0/1969463802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20981035c0 0x7f2098074c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:13.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.580+0000 7f20a067d700 1 --2- 172.21.15.70:0/1969463802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2098103f80 0x7f20980751d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:13.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.580+0000 7f20a067d700 1 -- 172.21.15.70:0/1969463802 >> 172.21.15.70:0/1969463802 conn(0x7f209807a610 msgr2=0x7f2098102280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:13.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.580+0000 7f20a067d700 1 -- 172.21.15.70:0/1969463802 shutdown_connections 2024-06-04T05:15:13.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:13.580+0000 7f20a067d700 1 -- 172.21.15.70:0/1969463802 wait complete. 2024-06-04T05:15:13.583 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 55 2024-06-04T05:15:13.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:13 smithi070 ceph-mon[32631]: pgmap v566: 97 pgs: 97 active+clean; 615 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 363 op/s 2024-06-04T05:15:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:13 smithi112 ceph-mon[36453]: pgmap v566: 97 pgs: 97 active+clean; 615 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 363 op/s 2024-06-04T05:15:13.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:13 smithi098 ceph-mon[35569]: pgmap v566: 97 pgs: 97 active+clean; 615 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 363 op/s 2024-06-04T05:15:13.815 INFO:tasks.workunit.client.0.smithi070.stdout: 10 35890 8.30 MB/sec execute 125 sec latency 114.369 ms 2024-06-04T05:15:14.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.329+0000 7f68c1420640 1 -- 172.21.15.70:0/1084392326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc103420 msgr2=0x7f68bc10f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:14.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.329+0000 7f68c1420640 1 --2- 172.21.15.70:0/1084392326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc103420 0x7f68bc10f930 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f68b4009ca0 tx=0x7f68b402a8c0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.329+0000 7f68c1420640 1 -- 172.21.15.70:0/1084392326 shutdown_connections 2024-06-04T05:15:14.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.329+0000 7f68c1420640 1 --2- 172.21.15.70:0/1084392326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc103420 0x7f68bc10f930 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.329+0000 7f68c1420640 1 --2- 172.21.15.70:0/1084392326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68bc102a60 0x7f68bc102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.329+0000 7f68c1420640 1 --2- 172.21.15.70:0/1084392326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc108a70 0x7f68bc108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.329+0000 7f68c1420640 1 -- 172.21.15.70:0/1084392326 >> 172.21.15.70:0/1084392326 conn(0x7f68bc0fe510 msgr2=0x7f68bc100930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:14.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.329+0000 7f68c1420640 1 -- 172.21.15.70:0/1084392326 shutdown_connections 2024-06-04T05:15:14.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.329+0000 7f68c1420640 1 -- 172.21.15.70:0/1084392326 wait complete. 2024-06-04T05:15:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.330+0000 7f68c1420640 1 Processor -- start 2024-06-04T05:15:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.330+0000 7f68c1420640 1 -- start start 2024-06-04T05:15:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.330+0000 7f68c1420640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc102a60 0x7f68bc074d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.330+0000 7f68c1420640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc103420 0x7f68bc075290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.330+0000 7f68c1420640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68bc108a70 0x7f68bc0757d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.330+0000 7f68c1420640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc112c00 con 0x7f68bc103420 2024-06-04T05:15:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.330+0000 7f68c1420640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc112a80 con 0x7f68bc102a60 2024-06-04T05:15:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.330+0000 7f68c1420640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc112d80 con 0x7f68bc108a70 2024-06-04T05:15:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.330+0000 7f68ba7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc103420 0x7f68bc075290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.330+0000 7f68ba7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc103420 0x7f68bc075290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51922/0 (socket says 172.21.15.70:51922) 2024-06-04T05:15:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.330+0000 7f68ba7fc640 1 -- 172.21.15.70:0/1301880911 learned_addr learned my addr 172.21.15.70:0/1301880911 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.331+0000 7f68baffd640 1 --2- 172.21.15.70:0/1301880911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc102a60 0x7f68bc074d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.331+0000 7f68bb7fe640 1 --2- 172.21.15.70:0/1301880911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68bc108a70 0x7f68bc0757d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.331+0000 7f68baffd640 1 -- 172.21.15.70:0/1301880911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68bc108a70 msgr2=0x7f68bc0757d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.331+0000 7f68baffd640 1 --2- 172.21.15.70:0/1301880911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68bc108a70 0x7f68bc0757d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.331+0000 7f68baffd640 1 -- 172.21.15.70:0/1301880911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc103420 msgr2=0x7f68bc075290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.331+0000 7f68baffd640 1 --2- 172.21.15.70:0/1301880911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc103420 0x7f68bc075290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.331+0000 7f68baffd640 1 -- 172.21.15.70:0/1301880911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68bc1a7580 con 0x7f68bc102a60 2024-06-04T05:15:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.331+0000 7f68ba7fc640 1 --2- 172.21.15.70:0/1301880911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc103420 0x7f68bc075290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.331+0000 7f68baffd640 1 --2- 172.21.15.70:0/1301880911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc102a60 0x7f68bc074d50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f68ac00e9f0 tx=0x7f68ac00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.331+0000 7f68a3fff640 1 -- 172.21.15.70:0/1301880911 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ac00cd80 con 0x7f68bc102a60 2024-06-04T05:15:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.332+0000 7f68a3fff640 1 -- 172.21.15.70:0/1301880911 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68ac0035f0 con 0x7f68bc102a60 2024-06-04T05:15:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.332+0000 7f68a3fff640 1 -- 172.21.15.70:0/1301880911 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ac010670 con 0x7f68bc102a60 2024-06-04T05:15:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.332+0000 7f68c1420640 1 -- 172.21.15.70:0/1301880911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68bc1a7870 con 0x7f68bc102a60 2024-06-04T05:15:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.332+0000 7f68c1420640 1 -- 172.21.15.70:0/1301880911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68bc1a7d70 con 0x7f68bc102a60 2024-06-04T05:15:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.335+0000 7f68c4215740 1 Processor -- start 2024-06-04T05:15:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.335+0000 7f68c4215740 1 -- start start 2024-06-04T05:15:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.335+0000 7f68a3fff640 1 -- 172.21.15.70:0/1301880911 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f68ac004000 con 0x7f68bc102a60 2024-06-04T05:15:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.335+0000 7f68c4215740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c1820 0x563cc28c1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.335+0000 7f68c4215740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28c2140 0x563cc28cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.335+0000 7f68c4215740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28cca00 0x563cc28cedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.335+0000 7f68c4215740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563cc2833c30 con 0x563cc28c1820 2024-06-04T05:15:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.335+0000 7f68c4215740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563cc28237d0 con 0x563cc28cca00 2024-06-04T05:15:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.335+0000 7f68c4215740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563cc2827810 con 0x563cc28c2140 2024-06-04T05:15:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.335+0000 7f68bb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c1820 0x563cc28c1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.335+0000 7f68baffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28c2140 0x563cc28cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.335+0000 7f68bb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c1820 0x563cc28c1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51926/0 (socket says 172.21.15.70:51926) 2024-06-04T05:15:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.335+0000 7f68bb7fe640 1 -- 172.21.15.70:0/763691700 learned_addr learned my addr 172.21.15.70:0/763691700 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.335+0000 7f68ba7fc640 1 --2- 172.21.15.70:0/763691700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28cca00 0x563cc28cedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.336+0000 7f68ba7fc640 1 -- 172.21.15.70:0/763691700 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28c2140 msgr2=0x563cc28cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.336+0000 7f68ba7fc640 1 --2- 172.21.15.70:0/763691700 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28c2140 0x563cc28cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.336+0000 7f68ba7fc640 1 -- 172.21.15.70:0/763691700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c1820 msgr2=0x563cc28c1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.336+0000 7f68ba7fc640 1 --2- 172.21.15.70:0/763691700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c1820 0x563cc28c1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.336+0000 7f68ba7fc640 1 -- 172.21.15.70:0/763691700 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cc2538210 con 0x563cc28cca00 2024-06-04T05:15:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.336+0000 7f68bb7fe640 1 --2- 172.21.15.70:0/763691700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c1820 0x563cc28c1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.336+0000 7f68ba7fc640 1 --2- 172.21.15.70:0/763691700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28cca00 0x563cc28cedf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f68b0005fd0 tx=0x7f68b00109e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5778cab818a82182 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.336+0000 7f68a3fff640 1 --2- 172.21.15.70:0/1301880911 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68a40b2a90 0x7f68a40b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.336+0000 7f68a17fa640 1 -- 172.21.15.70:0/763691700 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b0011820 con 0x563cc28cca00 2024-06-04T05:15:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.336+0000 7f68a17fa640 1 -- 172.21.15.70:0/763691700 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68b0011e50 con 0x563cc28cca00 2024-06-04T05:15:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.336+0000 7f68a17fa640 1 -- 172.21.15.70:0/763691700 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b000edc0 con 0x563cc28cca00 2024-06-04T05:15:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.336+0000 7f68bb7fe640 1 --2- 172.21.15.70:0/1301880911 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68a40b2a90 0x7f68a40b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.336+0000 7f68a3fff640 1 -- 172.21.15.70:0/1301880911 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f68ac0e2800 con 0x7f68bc102a60 2024-06-04T05:15:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.337+0000 7f68bb7fe640 1 --2- 172.21.15.70:0/1301880911 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68a40b2a90 0x7f68a40b4f50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f68b4004af0 tx=0x7f68b4004a80 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.337+0000 7f68c4215740 1 -- 172.21.15.70:0/763691700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28cca00 msgr2=0x563cc28cedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.337+0000 7f68c4215740 1 --2- 172.21.15.70:0/763691700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28cca00 0x563cc28cedf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f68b0005fd0 tx=0x7f68b00109e0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.337+0000 7f68c4215740 1 -- 172.21.15.70:0/763691700 shutdown_connections 2024-06-04T05:15:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.337+0000 7f68c4215740 1 --2- 172.21.15.70:0/763691700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28cca00 0x563cc28cedf0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.337+0000 7f68c4215740 1 --2- 172.21.15.70:0/763691700 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28c2140 0x563cc28cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.337+0000 7f68c4215740 1 --2- 172.21.15.70:0/763691700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c1820 0x563cc28c1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.337+0000 7f68c4215740 1 -- 172.21.15.70:0/763691700 >> 172.21.15.70:0/763691700 conn(0x563cc28bdd30 msgr2=0x563cc28be110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.337+0000 7f68c4215740 1 -- 172.21.15.70:0/763691700 shutdown_connections 2024-06-04T05:15:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.338+0000 7f68c4215740 1 -- 172.21.15.70:0/763691700 wait complete. 2024-06-04T05:15:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.338+0000 7f68c4215740 1 -- start start 2024-06-04T05:15:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.338+0000 7f68c4215740 1 Processor -- start 2024-06-04T05:15:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.338+0000 7f68c4215740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28c1820 0x563cc28c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.338+0000 7f68c4215740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28c2140 0x563cc28c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.338+0000 7f68c4215740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28cca00 0x563cc28ca430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.338+0000 7f68c4215740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563cc2509790 con 0x563cc28cca00 2024-06-04T05:15:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.338+0000 7f68c4215740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563cc27df890 con 0x563cc28c1820 2024-06-04T05:15:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.338+0000 7f68c4215740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563cc28cf630 con 0x563cc28c2140 2024-06-04T05:15:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.338+0000 7f68c4215740 10 client.0 _ll_register_callbacks cb 0x563cc2537a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.338+0000 7f68bb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28cca00 0x563cc28ca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.339+0000 7f68ba7fc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28c2140 0x563cc28c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.339+0000 7f68bb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28cca00 0x563cc28ca430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51934/0 (socket says 172.21.15.70:51934) 2024-06-04T05:15:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.339+0000 7f68bb7fe640 1 -- 172.21.15.70:0/3274830998 learned_addr learned my addr 172.21.15.70:0/3274830998 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.339+0000 7f68baffd640 1 --2- 172.21.15.70:0/3274830998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28c1820 0x563cc28c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.339+0000 7f68bb7fe640 1 -- 172.21.15.70:0/3274830998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28c2140 msgr2=0x563cc28c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.339+0000 7f68bb7fe640 1 --2- 172.21.15.70:0/3274830998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28c2140 0x563cc28c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.339+0000 7f68bb7fe640 1 -- 172.21.15.70:0/3274830998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28c1820 msgr2=0x563cc28c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.339+0000 7f68bb7fe640 1 --2- 172.21.15.70:0/3274830998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28c1820 0x563cc28c99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.339+0000 7f68bb7fe640 1 -- 172.21.15.70:0/3274830998 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cc2604620 con 0x563cc28cca00 2024-06-04T05:15:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.339+0000 7f68ba7fc640 1 --2- 172.21.15.70:0/3274830998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28c2140 0x563cc28c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.339+0000 7f68baffd640 1 --2- 172.21.15.70:0/3274830998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28c1820 0x563cc28c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:15:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.339+0000 7f68bb7fe640 1 --2- 172.21.15.70:0/3274830998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28cca00 0x563cc28ca430 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f68b4003f50 tx=0x7f68b402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336ce72c17c70ec4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.339+0000 7f689eff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:15:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.339+0000 7f689eff5640 1 -- 172.21.15.70:0/3274830998 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b4031e70 con 0x563cc28cca00 2024-06-04T05:15:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.339+0000 7f689eff5640 1 -- 172.21.15.70:0/3274830998 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f68b4004250 con 0x563cc28cca00 2024-06-04T05:15:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.339+0000 7f68c4215740 1 -- 172.21.15.70:0/3274830998 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68b0011e50 con 0x563cc28cca00 2024-06-04T05:15:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.340+0000 7f689eff5640 1 -- 172.21.15.70:0/3274830998 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f68b4004450 con 0x563cc28cca00 2024-06-04T05:15:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.340+0000 7f68c4215740 10 client.20103 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.340+0000 7f68c4215740 1 -- 172.21.15.70:0/3274830998 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x563cc28db3e0 con 0x563cc28cca00 2024-06-04T05:15:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.340+0000 7f689eff5640 1 -- 172.21.15.70:0/3274830998 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68b0011e50 con 0x563cc28cca00 2024-06-04T05:15:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.340+0000 7f689eff5640 1 -- 172.21.15.70:0/3274830998 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f68b4051020 con 0x563cc28cca00 2024-06-04T05:15:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.340+0000 7f68c4215740 10 client.20103 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.340+0000 7f68c4215740 10 client.20103 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.340+0000 7f68c4215740 20 client.20103 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.340+0000 7f68c4215740 1 --2- 172.21.15.70:0/3274830998 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563cc28dc940 0x563cc28dede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.340+0000 7f68c4215740 4 client.20103 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:15:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.340+0000 7f68c4215740 1 -- 172.21.15.70:0/3274830998 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563cc2835b80 con 0x563cc28dc940 2024-06-04T05:15:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.340+0000 7f68baffd640 1 --2- 172.21.15.70:0/3274830998 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563cc28dc940 0x563cc28dede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.341+0000 7f689eff5640 10 client.20103 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.341+0000 7f68baffd640 1 --2- 172.21.15.70:0/3274830998 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563cc28dc940 0x563cc28dede0 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96db8284ef402fbb server_cookie=adbcad8c455b8dde in_seq=0 out_seq=0 2024-06-04T05:15:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.342+0000 7f689eff5640 1 -- 172.21.15.70:0/3274830998 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563cc2835b80 con 0x563cc28dc940 2024-06-04T05:15:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.342+0000 7f689eff5640 10 client.20103 handle_command_reply: tid=0 2024-06-04T05:15:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.342+0000 7f68c4215740 1 client.20103 shutdown 2024-06-04T05:15:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.342+0000 7f68c4215740 1 -- 172.21.15.70:0/3274830998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28cca00 msgr2=0x563cc28ca430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.342+0000 7f68c4215740 1 --2- 172.21.15.70:0/3274830998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28cca00 0x563cc28ca430 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f68b4003f50 tx=0x7f68b402a8c0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.342+0000 7f68c4215740 1 -- 172.21.15.70:0/3274830998 shutdown_connections 2024-06-04T05:15:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.342+0000 7f68c4215740 1 --2- 172.21.15.70:0/3274830998 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563cc28dc940 0x563cc28dede0 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.342+0000 7f68c4215740 1 --2- 172.21.15.70:0/3274830998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28cca00 0x563cc28ca430 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.342+0000 7f68c4215740 1 --2- 172.21.15.70:0/3274830998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28c2140 0x563cc28c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.342+0000 7f68c4215740 1 --2- 172.21.15.70:0/3274830998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28c1820 0x563cc28c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.342+0000 7f689eff5640 0 client.20103 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.342+0000 7f68c4215740 1 -- 172.21.15.70:0/3274830998 >> 172.21.15.70:0/3274830998 conn(0x563cc28bdd30 msgr2=0x563cc28c3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.342+0000 7f68c4215740 1 -- 172.21.15.70:0/3274830998 shutdown_connections 2024-06-04T05:15:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.342+0000 7f68c4215740 1 -- 172.21.15.70:0/3274830998 wait complete. 2024-06-04T05:15:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.342+0000 7f68c4215740 20 client.20103 trim_cache size 0 max 16384 2024-06-04T05:15:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.367+0000 7f68c4215740 1 Processor -- start 2024-06-04T05:15:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.367+0000 7f68c4215740 1 -- start start 2024-06-04T05:15:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.367+0000 7f68c4215740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28c1820 0x563cc28c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.367+0000 7f68c4215740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c2140 0x563cc28e46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.367+0000 7f68c4215740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28cca00 0x563cc28e4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.367+0000 7f68c4215740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc1b0940 con 0x563cc28c2140 2024-06-04T05:15:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.367+0000 7f68c4215740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc1b24e0 con 0x563cc28cca00 2024-06-04T05:15:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.367+0000 7f68c4215740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc1b1070 con 0x563cc28c1820 2024-06-04T05:15:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.367+0000 7f68ba7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c2140 0x563cc28e46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.367+0000 7f68ba7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c2140 0x563cc28e46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51938/0 (socket says 172.21.15.70:51938) 2024-06-04T05:15:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.367+0000 7f68ba7fc640 1 -- 172.21.15.70:0/3886060553 learned_addr learned my addr 172.21.15.70:0/3886060553 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.367+0000 7f68baffd640 1 --2- 172.21.15.70:0/3886060553 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28c1820 0x563cc28c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.367+0000 7f68bb7fe640 1 --2- 172.21.15.70:0/3886060553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28cca00 0x563cc28e4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.367+0000 7f68baffd640 1 -- 172.21.15.70:0/3886060553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28cca00 msgr2=0x563cc28e4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.367+0000 7f68baffd640 1 --2- 172.21.15.70:0/3886060553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28cca00 0x563cc28e4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.368+0000 7f68baffd640 1 -- 172.21.15.70:0/3886060553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c2140 msgr2=0x563cc28e46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.368+0000 7f68baffd640 1 --2- 172.21.15.70:0/3886060553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c2140 0x563cc28e46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.368+0000 7f68baffd640 1 -- 172.21.15.70:0/3886060553 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cc28b9850 con 0x563cc28c1820 2024-06-04T05:15:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.368+0000 7f68ba7fc640 1 --2- 172.21.15.70:0/3886060553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c2140 0x563cc28e46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.368+0000 7f68baffd640 1 --2- 172.21.15.70:0/3886060553 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28c1820 0x563cc28c41e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0dff20 tx=0x7f68ac0e4d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ae63c1eb752b236 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.368+0000 7f68a0ff9640 1 -- 172.21.15.70:0/3886060553 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ac08ae00 con 0x563cc28c1820 2024-06-04T05:15:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.368+0000 7f68a0ff9640 1 -- 172.21.15.70:0/3886060553 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x563cc2835b80 con 0x563cc28c1820 2024-06-04T05:15:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.368+0000 7f68a0ff9640 1 -- 172.21.15.70:0/3886060553 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ac0e5e60 con 0x563cc28c1820 2024-06-04T05:15:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.369+0000 7f68c4215740 1 -- 172.21.15.70:0/3886060553 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28c1820 msgr2=0x563cc28c41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.369+0000 7f68c4215740 1 --2- 172.21.15.70:0/3886060553 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28c1820 0x563cc28c41e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0dff20 tx=0x7f68ac0e4d10 comp rx=0 tx=0).stop 2024-06-04T05:15:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.369+0000 7f68c4215740 1 -- 172.21.15.70:0/3886060553 shutdown_connections 2024-06-04T05:15:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.369+0000 7f68c4215740 1 --2- 172.21.15.70:0/3886060553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28cca00 0x563cc28e4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.369+0000 7f68c4215740 1 --2- 172.21.15.70:0/3886060553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c2140 0x563cc28e46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.369+0000 7f68c4215740 1 --2- 172.21.15.70:0/3886060553 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28c1820 0x563cc28c41e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.369+0000 7f68c4215740 1 -- 172.21.15.70:0/3886060553 >> 172.21.15.70:0/3886060553 conn(0x563cc28bdd30 msgr2=0x563cc28ca200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.369+0000 7f68c4215740 1 -- 172.21.15.70:0/3886060553 shutdown_connections 2024-06-04T05:15:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.369+0000 7f68c4215740 1 -- 172.21.15.70:0/3886060553 wait complete. 2024-06-04T05:15:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.369+0000 7f68c4215740 1 -- start start 2024-06-04T05:15:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.370+0000 7f68c4215740 1 Processor -- start 2024-06-04T05:15:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.370+0000 7f68c4215740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c1820 0x563cc28d4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.370+0000 7f68c4215740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28c2140 0x563cc28d4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.370+0000 7f68c4215740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28cca00 0x563cc28d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.370+0000 7f68c4215740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc1b1e10 con 0x563cc28c1820 2024-06-04T05:15:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.370+0000 7f68c4215740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc1b1740 con 0x563cc28c2140 2024-06-04T05:15:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.370+0000 7f68c4215740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc1ae940 con 0x563cc28cca00 2024-06-04T05:15:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.370+0000 7f68c4215740 10 client.0 _ll_register_callbacks cb 0x7f68b4003de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.370+0000 7f68baffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c1820 0x563cc28d4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.370+0000 7f68baffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c1820 0x563cc28d4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51942/0 (socket says 172.21.15.70:51942) 2024-06-04T05:15:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.370+0000 7f68baffd640 1 -- 172.21.15.70:0/2473278674 learned_addr learned my addr 172.21.15.70:0/2473278674 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.370+0000 7f68ba7fc640 1 --2- 172.21.15.70:0/2473278674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28c2140 0x563cc28d4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.370+0000 7f68bb7fe640 1 --2- 172.21.15.70:0/2473278674 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28cca00 0x563cc28d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.371+0000 7f68baffd640 1 -- 172.21.15.70:0/2473278674 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28cca00 msgr2=0x563cc28d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.371+0000 7f68baffd640 1 --2- 172.21.15.70:0/2473278674 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28cca00 0x563cc28d1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.371+0000 7f68baffd640 1 -- 172.21.15.70:0/2473278674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28c2140 msgr2=0x563cc28d4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.371+0000 7f68baffd640 1 --2- 172.21.15.70:0/2473278674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28c2140 0x563cc28d4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.371+0000 7f68baffd640 1 -- 172.21.15.70:0/2473278674 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cc28d1d90 con 0x563cc28c1820 2024-06-04T05:15:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.371+0000 7f68bb7fe640 1 --2- 172.21.15.70:0/2473278674 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28cca00 0x563cc28d1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:15:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.371+0000 7f68ba7fc640 1 --2- 172.21.15.70:0/2473278674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28c2140 0x563cc28d4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.371+0000 7f68baffd640 1 --2- 172.21.15.70:0/2473278674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c1820 0x563cc28d4880 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0dffd0 tx=0x7f68ac0e83b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af412a6c018e482d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.371+0000 7f689eff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:15:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.371+0000 7f689eff5640 1 -- 172.21.15.70:0/2473278674 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ac0b2250 con 0x563cc28c1820 2024-06-04T05:15:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.371+0000 7f689eff5640 1 -- 172.21.15.70:0/2473278674 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f68ac0a60a0 con 0x563cc28c1820 2024-06-04T05:15:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.371+0000 7f68c4215740 1 -- 172.21.15.70:0/2473278674 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563cc2835b80 con 0x563cc28c1820 2024-06-04T05:15:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.372+0000 7f689eff5640 1 -- 172.21.15.70:0/2473278674 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f68ac0aa1b0 con 0x563cc28c1820 2024-06-04T05:15:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.372+0000 7f68c4215740 10 client.20109 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:14.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.372+0000 7f68c4215740 1 -- 172.21.15.70:0/2473278674 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x563cc2947bd0 con 0x563cc28c1820 2024-06-04T05:15:14.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.372+0000 7f689eff5640 1 -- 172.21.15.70:0/2473278674 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68ac0a9ab0 con 0x563cc28c1820 2024-06-04T05:15:14.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.372+0000 7f689eff5640 1 -- 172.21.15.70:0/2473278674 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f68ac0e01a0 con 0x563cc28c1820 2024-06-04T05:15:14.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.373+0000 7f68c4215740 10 client.20109 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:14.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.373+0000 7f68c4215740 10 client.20109 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:14.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.373+0000 7f68c4215740 20 client.20109 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:14.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.373+0000 7f68c4215740 1 --2- 172.21.15.70:0/2473278674 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563cc2948180 0x563cc294a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:14.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.373+0000 7f68c4215740 4 client.20109 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:15:14.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.373+0000 7f68c4215740 1 -- 172.21.15.70:0/2473278674 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563cc2835b80 con 0x563cc2948180 2024-06-04T05:15:14.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.373+0000 7f68ba7fc640 1 --2- 172.21.15.70:0/2473278674 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563cc2948180 0x563cc294a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:14.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.373+0000 7f68ba7fc640 1 --2- 172.21.15.70:0/2473278674 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563cc2948180 0x563cc294a570 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e261685d48acdf7 server_cookie=8b74b4df884d01b2 in_seq=0 out_seq=0 2024-06-04T05:15:14.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.373+0000 7f689eff5640 10 client.20109 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.463+0000 7f689eff5640 1 -- 172.21.15.70:0/2473278674 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x563cc2835b80 con 0x563cc2948180 2024-06-04T05:15:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.463+0000 7f689eff5640 10 client.20109 handle_command_reply: tid=0 2024-06-04T05:15:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.463+0000 7f68c4215740 1 client.20109 shutdown 2024-06-04T05:15:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.464+0000 7f68c4215740 1 -- 172.21.15.70:0/2473278674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c1820 msgr2=0x563cc28d4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.464+0000 7f68c4215740 1 --2- 172.21.15.70:0/2473278674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c1820 0x563cc28d4880 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0dffd0 tx=0x7f68ac0e83b0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.464+0000 7f68c4215740 1 -- 172.21.15.70:0/2473278674 shutdown_connections 2024-06-04T05:15:14.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.465+0000 7f68c4215740 1 --2- 172.21.15.70:0/2473278674 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563cc2948180 0x563cc294a570 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.465+0000 7f68c4215740 1 --2- 172.21.15.70:0/2473278674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc28c2140 0x563cc28d4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.465+0000 7f68c4215740 1 --2- 172.21.15.70:0/2473278674 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563cc28cca00 0x563cc28d1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.465+0000 7f68c4215740 1 --2- 172.21.15.70:0/2473278674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563cc28c1820 0x563cc28d4880 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.465+0000 7f68c4215740 1 -- 172.21.15.70:0/2473278674 >> 172.21.15.70:0/2473278674 conn(0x563cc28bdd30 msgr2=0x563cc28bc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.466+0000 7f689eff5640 0 client.20109 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.466+0000 7f68c4215740 1 -- 172.21.15.70:0/2473278674 shutdown_connections 2024-06-04T05:15:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.466+0000 7f68c4215740 1 -- 172.21.15.70:0/2473278674 wait complete. 2024-06-04T05:15:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.466+0000 7f68c4215740 20 client.20109 trim_cache size 0 max 16384 2024-06-04T05:15:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:15:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:15:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:15:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.467+0000 7f68c1420640 1 -- 172.21.15.70:0/1301880911 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68a40b2a90 msgr2=0x7f68a40b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.467+0000 7f68c1420640 1 --2- 172.21.15.70:0/1301880911 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68a40b2a90 0x7f68a40b4f50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f68b4004af0 tx=0x7f68b4004a80 comp rx=0 tx=0).stop 2024-06-04T05:15:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.467+0000 7f68c1420640 1 -- 172.21.15.70:0/1301880911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc102a60 msgr2=0x7f68bc074d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.467+0000 7f68c1420640 1 --2- 172.21.15.70:0/1301880911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc102a60 0x7f68bc074d50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f68ac00e9f0 tx=0x7f68ac00eeb0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.467+0000 7f68c1420640 1 -- 172.21.15.70:0/1301880911 shutdown_connections 2024-06-04T05:15:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.467+0000 7f68c1420640 1 --2- 172.21.15.70:0/1301880911 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68a40b2a90 0x7f68a40b4f50 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.467+0000 7f68c1420640 1 --2- 172.21.15.70:0/1301880911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68bc108a70 0x7f68bc0757d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.467+0000 7f68c1420640 1 --2- 172.21.15.70:0/1301880911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68bc103420 0x7f68bc075290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.467+0000 7f68c1420640 1 --2- 172.21.15.70:0/1301880911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc102a60 0x7f68bc074d50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.467+0000 7f68c1420640 1 -- 172.21.15.70:0/1301880911 >> 172.21.15.70:0/1301880911 conn(0x7f68bc0fe510 msgr2=0x7f68bc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.467+0000 7f68c1420640 1 -- 172.21.15.70:0/1301880911 shutdown_connections 2024-06-04T05:15:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:14.467+0000 7f68c1420640 1 -- 172.21.15.70:0/1301880911 wait complete. 2024-06-04T05:15:14.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:14 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1969463802' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:15:14.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:14 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1969463802' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:15:14.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:14 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1969463802' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:15:14.815 INFO:tasks.workunit.client.0.smithi070.stdout: 10 36085 8.34 MB/sec execute 126 sec latency 101.988 ms 2024-06-04T05:15:14.927 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:15:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.213+0000 7f7e52d39700 1 -- 172.21.15.70:0/2553877754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e4c102ae0 msgr2=0x7f7e4c102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.213+0000 7f7e52d39700 1 --2- 172.21.15.70:0/2553877754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e4c102ae0 0x7f7e4c102ec0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7e400131e0 tx=0x7f7e40011f90 comp rx=0 tx=0).stop 2024-06-04T05:15:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.213+0000 7f7e52d39700 1 -- 172.21.15.70:0/2553877754 shutdown_connections 2024-06-04T05:15:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.213+0000 7f7e52d39700 1 --2- 172.21.15.70:0/2553877754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e4c107a30 0x7f7e4c06bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.213+0000 7f7e52d39700 1 --2- 172.21.15.70:0/2553877754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e4c102ae0 0x7f7e4c102ec0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.213+0000 7f7e52d39700 1 --2- 172.21.15.70:0/2553877754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e4c107070 0x7f7e4c1074f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.213+0000 7f7e52d39700 1 -- 172.21.15.70:0/2553877754 >> 172.21.15.70:0/2553877754 conn(0x7f7e4c0793f0 msgr2=0x7f7e4c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.214+0000 7f7e52d39700 1 -- 172.21.15.70:0/2553877754 shutdown_connections 2024-06-04T05:15:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.214+0000 7f7e52d39700 1 -- 172.21.15.70:0/2553877754 wait complete. 2024-06-04T05:15:15.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.215+0000 7f7e52d39700 1 Processor -- start 2024-06-04T05:15:15.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.215+0000 7f7e52d39700 1 -- start start 2024-06-04T05:15:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.215+0000 7f7e52d39700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e4c102ae0 0x7f7e4c19d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.215+0000 7f7e52d39700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e4c107070 0x7f7e4c19db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.215+0000 7f7e52d39700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e4c107a30 0x7f7e4c197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.215+0000 7f7e52d39700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4c111500 con 0x7f7e4c107070 2024-06-04T05:15:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.215+0000 7f7e52d39700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4c10b2d0 con 0x7f7e4c102ae0 2024-06-04T05:15:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.215+0000 7f7e52d39700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4c111680 con 0x7f7e4c107a30 2024-06-04T05:15:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.216+0000 7f7e512d6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e4c107a30 0x7f7e4c197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.216+0000 7f7e512d6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e4c107a30 0x7f7e4c197780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:34896/0 (socket says 172.21.15.70:34896) 2024-06-04T05:15:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.216+0000 7f7e512d6700 1 -- 172.21.15.70:0/1334534455 learned_addr learned my addr 172.21.15.70:0/1334534455 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.216+0000 7f7e50ad5700 1 --2- 172.21.15.70:0/1334534455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e4c102ae0 0x7f7e4c19d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.216+0000 7f7e50ad5700 1 -- 172.21.15.70:0/1334534455 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e4c107a30 msgr2=0x7f7e4c197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.216+0000 7f7e50ad5700 1 --2- 172.21.15.70:0/1334534455 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e4c107a30 0x7f7e4c197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.216+0000 7f7e50ad5700 1 -- 172.21.15.70:0/1334534455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e4c107070 msgr2=0x7f7e4c19db40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-04T05:15:15.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.216+0000 7f7e50ad5700 1 --2- 172.21.15.70:0/1334534455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e4c107070 0x7f7e4c19db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.216+0000 7f7e50ad5700 1 -- 172.21.15.70:0/1334534455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e4c197f60 con 0x7f7e4c102ae0 2024-06-04T05:15:15.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.217+0000 7f7e50ad5700 1 --2- 172.21.15.70:0/1334534455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e4c102ae0 0x7f7e4c19d600 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7e400131b0 tx=0x7f7e40011380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:15.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.217+0000 7f7e49ffb700 1 -- 172.21.15.70:0/1334534455 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4001f040 con 0x7f7e4c102ae0 2024-06-04T05:15:15.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.217+0000 7f7e49ffb700 1 -- 172.21.15.70:0/1334534455 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e4002a070 con 0x7f7e4c102ae0 2024-06-04T05:15:15.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.217+0000 7f7e49ffb700 1 -- 172.21.15.70:0/1334534455 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e40025e60 con 0x7f7e4c102ae0 2024-06-04T05:15:15.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.217+0000 7f7e52d39700 1 -- 172.21.15.70:0/1334534455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e4c1981f0 con 0x7f7e4c102ae0 2024-06-04T05:15:15.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.218+0000 7f7e52d39700 1 -- 172.21.15.70:0/1334534455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e4c1a4460 con 0x7f7e4c102ae0 2024-06-04T05:15:15.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.221+0000 7f7e49ffb700 1 -- 172.21.15.70:0/1334534455 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7e40025450 con 0x7f7e4c102ae0 2024-06-04T05:15:15.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.223+0000 7f7e49ffb700 1 --2- 172.21.15.70:0/1334534455 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7e3c0b2b20 0x7f7e3c0b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.223+0000 7f7e49ffb700 1 -- 172.21.15.70:0/1334534455 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f7e40017020 con 0x7f7e4c102ae0 2024-06-04T05:15:15.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.223+0000 7f7e5d5ceb80 1 Processor -- start 2024-06-04T05:15:15.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.223+0000 7f7e5d5ceb80 1 -- start start 2024-06-04T05:15:15.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.223+0000 7f7e4bfff700 1 --2- 172.21.15.70:0/1334534455 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7e3c0b2b20 0x7f7e3c0b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:15.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.223+0000 7f7e5d5ceb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d87b0 0x55d6119d8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.223+0000 7f7e5d5ceb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d3c80 0x55d6119d4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e5d5ceb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 0x55d6119e06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e5d5ceb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d611931f00 con 0x55d6119d3c80 2024-06-04T05:15:15.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e5d5ceb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d611921aa0 con 0x55d6119d87b0 2024-06-04T05:15:15.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e50ad5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d87b0 0x55d6119d8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:15.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e5d5ceb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d611925ae0 con 0x55d6119d4640 2024-06-04T05:15:15.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e50ad5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d87b0 0x55d6119d8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45726/0 (socket says 172.21.15.70:45726) 2024-06-04T05:15:15.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e50ad5700 1 -- 172.21.15.70:0/2154599020 learned_addr learned my addr 172.21.15.70:0/2154599020 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:15.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e4bfff700 1 --2- 172.21.15.70:0/2154599020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d3c80 0x55d6119d4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:15.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e512d6700 1 --2- 172.21.15.70:0/2154599020 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 0x55d6119e06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:15.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e4bfff700 1 --2- 172.21.15.70:0/1334534455 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7e3c0b2b20 0x7f7e3c0b4fe0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c079cf0 tx=0x7f7e38009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:15.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e50ad5700 1 -- 172.21.15.70:0/2154599020 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 msgr2=0x55d6119e06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:15.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e50ad5700 1 --2- 172.21.15.70:0/2154599020 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 0x55d6119e06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e50ad5700 1 -- 172.21.15.70:0/2154599020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d3c80 msgr2=0x55d6119d4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:15.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e50ad5700 1 --2- 172.21.15.70:0/2154599020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d3c80 0x55d6119d4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e50ad5700 1 -- 172.21.15.70:0/2154599020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6118cb140 con 0x55d6119d87b0 2024-06-04T05:15:15.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e512d6700 1 --2- 172.21.15.70:0/2154599020 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 0x55d6119e06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:15:15.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e50ad5700 1 --2- 172.21.15.70:0/2154599020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d87b0 0x55d6119d8b90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7e4001d040 tx=0x7f7e400b1b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d30634b72cc84ad8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:15.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e36ffd700 1 -- 172.21.15.70:0/2154599020 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e400b2350 con 0x55d6119d87b0 2024-06-04T05:15:15.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.224+0000 7f7e36ffd700 1 -- 172.21.15.70:0/2154599020 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e400b6700 con 0x55d6119d87b0 2024-06-04T05:15:15.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.225+0000 7f7e36ffd700 1 -- 172.21.15.70:0/2154599020 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e40019c40 con 0x55d6119d87b0 2024-06-04T05:15:15.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.225+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/2154599020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d87b0 msgr2=0x55d6119d8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:15.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.225+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/2154599020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d87b0 0x55d6119d8b90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7e4001d040 tx=0x7f7e400b1b90 comp rx=0 tx=0).stop 2024-06-04T05:15:15.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.226+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/2154599020 shutdown_connections 2024-06-04T05:15:15.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.226+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/2154599020 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 0x55d6119e06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.226+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/2154599020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d87b0 0x55d6119d8b90 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.226+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/2154599020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d3c80 0x55d6119d4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.226+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/2154599020 >> 172.21.15.70:0/2154599020 conn(0x55d6118ff3d0 msgr2=0x55d6119ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:15.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.226+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/2154599020 shutdown_connections 2024-06-04T05:15:15.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.226+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/2154599020 wait complete. 2024-06-04T05:15:15.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.226+0000 7f7e5d5ceb80 1 -- start start 2024-06-04T05:15:15.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.226+0000 7f7e5d5ceb80 1 Processor -- start 2024-06-04T05:15:15.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.227+0000 7f7e5d5ceb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d3c80 0x55d6119e0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.227+0000 7f7e5d5ceb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 0x55d6119dd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.227+0000 7f7e5d5ceb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d87b0 0x55d6119ddcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.227+0000 7f7e5d5ceb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d611828130 con 0x55d6119d3c80 2024-06-04T05:15:15.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.227+0000 7f7e5d5ceb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d611889f30 con 0x55d6119d87b0 2024-06-04T05:15:15.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.227+0000 7f7e5d5ceb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d611921aa0 con 0x55d6119d4640 2024-06-04T05:15:15.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.227+0000 7f7e5d5ceb80 10 client.0 _ll_register_callbacks cb 0x55d6118bcae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:15.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.227+0000 7f7e50ad5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d3c80 0x55d6119e0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:15.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.227+0000 7f7e50ad5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d3c80 0x55d6119e0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51968/0 (socket says 172.21.15.70:51968) 2024-06-04T05:15:15.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.227+0000 7f7e50ad5700 1 -- 172.21.15.70:0/278589480 learned_addr learned my addr 172.21.15.70:0/278589480 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:15.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.227+0000 7f7e512d6700 1 --2- 172.21.15.70:0/278589480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d87b0 0x55d6119ddcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:15.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.228+0000 7f7e50ad5700 1 -- 172.21.15.70:0/278589480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 msgr2=0x55d6119dd770 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:15:15.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.228+0000 7f7e50ad5700 1 --2- 172.21.15.70:0/278589480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 0x55d6119dd770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.228+0000 7f7e50ad5700 1 -- 172.21.15.70:0/278589480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d87b0 msgr2=0x55d6119ddcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:15.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.228+0000 7f7e50ad5700 1 --2- 172.21.15.70:0/278589480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d87b0 0x55d6119ddcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.228+0000 7f7e50ad5700 1 -- 172.21.15.70:0/278589480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d611871a60 con 0x55d6119d3c80 2024-06-04T05:15:15.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.228+0000 7f7e50ad5700 1 --2- 172.21.15.70:0/278589480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d3c80 0x55d6119e0470 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f7e400e8d30 tx=0x7f7e400bafd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=605df3616eb4a821 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:15.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.228+0000 7f7e2bfff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:15:15.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.228+0000 7f7e2bfff700 1 -- 172.21.15.70:0/278589480 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e400b2350 con 0x55d6119d3c80 2024-06-04T05:15:15.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.228+0000 7f7e2bfff700 1 -- 172.21.15.70:0/278589480 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f7e400b27c0 con 0x55d6119d3c80 2024-06-04T05:15:15.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.228+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/278589480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e400b6700 con 0x55d6119d3c80 2024-06-04T05:15:15.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.228+0000 7f7e2bfff700 1 -- 172.21.15.70:0/278589480 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f7e400bb120 con 0x55d6119d3c80 2024-06-04T05:15:15.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.228+0000 7f7e5d5ceb80 10 client.20115 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:15.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.228+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/278589480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55d6118db8f0 con 0x55d6119d3c80 2024-06-04T05:15:15.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.229+0000 7f7e2bfff700 1 -- 172.21.15.70:0/278589480 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e400b6700 con 0x55d6119d3c80 2024-06-04T05:15:15.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.229+0000 7f7e2bfff700 1 -- 172.21.15.70:0/278589480 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f7e40019070 con 0x55d6119d3c80 2024-06-04T05:15:15.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.229+0000 7f7e5d5ceb80 10 client.20115 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:15.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.229+0000 7f7e5d5ceb80 10 client.20115 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:15:15.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.229+0000 7f7e5d5ceb80 20 client.20115 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:15.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.229+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/278589480 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d6119ed570 0x55d6119ef960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.229+0000 7f7e5d5ceb80 4 client.20115 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:15:15.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.229+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/278589480 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d611933e50 con 0x55d6119ed570 2024-06-04T05:15:15.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.229+0000 7f7e4bfff700 1 --2- 172.21.15.70:0/278589480 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d6119ed570 0x55d6119ef960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:15.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.230+0000 7f7e2bfff700 10 client.20115 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:15.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.230+0000 7f7e4bfff700 1 --2- 172.21.15.70:0/278589480 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d6119ed570 0x55d6119ef960 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e92e458814f872a2 server_cookie=f71a7d3a7bc0f203 in_seq=0 out_seq=0 2024-06-04T05:15:15.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.231+0000 7f7e2bfff700 1 -- 172.21.15.70:0/278589480 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d611933e50 con 0x55d6119ed570 2024-06-04T05:15:15.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.231+0000 7f7e2bfff700 10 client.20115 handle_command_reply: tid=0 2024-06-04T05:15:15.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.231+0000 7f7e5d5ceb80 1 client.20115 shutdown 2024-06-04T05:15:15.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.231+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/278589480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d3c80 msgr2=0x55d6119e0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:15.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.231+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/278589480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d3c80 0x55d6119e0470 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f7e400e8d30 tx=0x7f7e400bafd0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.231+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/278589480 shutdown_connections 2024-06-04T05:15:15.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.231+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/278589480 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d6119ed570 0x55d6119ef960 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.231+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/278589480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d87b0 0x55d6119ddcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.231+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/278589480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d3c80 0x55d6119e0470 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.231+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/278589480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 0x55d6119dd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.231+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/278589480 >> 172.21.15.70:0/278589480 conn(0x55d6118ff3d0 msgr2=0x55d6119d2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:15.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.231+0000 7f7e2bfff700 0 client.20115 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:15.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.231+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/278589480 shutdown_connections 2024-06-04T05:15:15.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.232+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/278589480 wait complete. 2024-06-04T05:15:15.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.232+0000 7f7e5d5ceb80 20 client.20115 trim_cache size 0 max 16384 2024-06-04T05:15:15.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.262+0000 7f7e5d5ceb80 1 Processor -- start 2024-06-04T05:15:15.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.262+0000 7f7e5d5ceb80 1 -- start start 2024-06-04T05:15:15.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.262+0000 7f7e5d5ceb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d3c80 0x55d6119d8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.262+0000 7f7e5d5ceb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 0x55d6119c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.263+0000 7f7e5d5ceb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d87b0 0x55d6119e9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.263+0000 7f7e5d5ceb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4c1abb70 con 0x55d6119d87b0 2024-06-04T05:15:15.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.263+0000 7f7e5d5ceb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4c1ad710 con 0x55d6119d3c80 2024-06-04T05:15:15.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.263+0000 7f7e5d5ceb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4c1ac2a0 con 0x55d6119d4640 2024-06-04T05:15:15.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.263+0000 7f7e50ad5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d3c80 0x55d6119d8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:15.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.263+0000 7f7e512d6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d87b0 0x55d6119e9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:15.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.263+0000 7f7e50ad5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d3c80 0x55d6119d8060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45750/0 (socket says 172.21.15.70:45750) 2024-06-04T05:15:15.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.263+0000 7f7e50ad5700 1 -- 172.21.15.70:0/2613642766 learned_addr learned my addr 172.21.15.70:0/2613642766 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:15.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.263+0000 7f7e4bfff700 1 --2- 172.21.15.70:0/2613642766 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 0x55d6119c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:15.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.263+0000 7f7e50ad5700 1 -- 172.21.15.70:0/2613642766 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 msgr2=0x55d6119c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:15.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.263+0000 7f7e50ad5700 1 --2- 172.21.15.70:0/2613642766 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 0x55d6119c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.263+0000 7f7e50ad5700 1 -- 172.21.15.70:0/2613642766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d87b0 msgr2=0x55d6119e9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:15.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.263+0000 7f7e50ad5700 1 --2- 172.21.15.70:0/2613642766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d87b0 0x55d6119e9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.263+0000 7f7e50ad5700 1 -- 172.21.15.70:0/2613642766 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6119dffb0 con 0x55d6119d3c80 2024-06-04T05:15:15.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.263+0000 7f7e4bfff700 1 --2- 172.21.15.70:0/2613642766 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 0x55d6119c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:15:15.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.263+0000 7f7e50ad5700 1 --2- 172.21.15.70:0/2613642766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d3c80 0x55d6119d8060 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7e400e8cb0 tx=0x7f7e40005ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11330b5b17912c99 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:15.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.264+0000 7f7e367fc700 1 -- 172.21.15.70:0/2613642766 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e400b1ac0 con 0x55d6119d3c80 2024-06-04T05:15:15.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.264+0000 7f7e367fc700 1 -- 172.21.15.70:0/2613642766 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e400b2350 con 0x55d6119d3c80 2024-06-04T05:15:15.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.264+0000 7f7e367fc700 1 -- 172.21.15.70:0/2613642766 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e40019c40 con 0x55d6119d3c80 2024-06-04T05:15:15.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.265+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/2613642766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d3c80 msgr2=0x55d6119d8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:15.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.265+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/2613642766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d3c80 0x55d6119d8060 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7e400e8cb0 tx=0x7f7e40005ad0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.265+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/2613642766 shutdown_connections 2024-06-04T05:15:15.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.265+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/2613642766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d87b0 0x55d6119e9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.265+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/2613642766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d3c80 0x55d6119d8060 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.265+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/2613642766 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 0x55d6119c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.265+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/2613642766 >> 172.21.15.70:0/2613642766 conn(0x55d6118ff3d0 msgr2=0x55d6119d32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:15.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.265+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/2613642766 shutdown_connections 2024-06-04T05:15:15.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.265+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/2613642766 wait complete. 2024-06-04T05:15:15.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.266+0000 7f7e5d5ceb80 1 -- start start 2024-06-04T05:15:15.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.266+0000 7f7e5d5ceb80 1 Processor -- start 2024-06-04T05:15:15.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.266+0000 7f7e5d5ceb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d3c80 0x55d6119e5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.266+0000 7f7e5d5ceb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 0x55d6119e6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.266+0000 7f7e5d5ceb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d87b0 0x55d6119e2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.266+0000 7f7e5d5ceb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4c1ad040 con 0x55d6119d87b0 2024-06-04T05:15:15.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.266+0000 7f7e5d5ceb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4c1ac970 con 0x55d6119d3c80 2024-06-04T05:15:15.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.266+0000 7f7e5d5ceb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4c1a9b30 con 0x55d6119d4640 2024-06-04T05:15:15.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.266+0000 7f7e5d5ceb80 10 client.0 _ll_register_callbacks cb 0x55d6118bcae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:15.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.266+0000 7f7e50ad5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d3c80 0x55d6119e5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:15.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.266+0000 7f7e50ad5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d3c80 0x55d6119e5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45764/0 (socket says 172.21.15.70:45764) 2024-06-04T05:15:15.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.266+0000 7f7e50ad5700 1 -- 172.21.15.70:0/1707114620 learned_addr learned my addr 172.21.15.70:0/1707114620 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:15.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.267+0000 7f7e512d6700 1 --2- 172.21.15.70:0/1707114620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d87b0 0x55d6119e2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:15.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.267+0000 7f7e50ad5700 1 -- 172.21.15.70:0/1707114620 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 msgr2=0x55d6119e6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:15.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.267+0000 7f7e50ad5700 1 --2- 172.21.15.70:0/1707114620 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 0x55d6119e6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.267+0000 7f7e50ad5700 1 -- 172.21.15.70:0/1707114620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d87b0 msgr2=0x55d6119e2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:15.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.267+0000 7f7e50ad5700 1 --2- 172.21.15.70:0/1707114620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d87b0 0x55d6119e2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.267+0000 7f7e50ad5700 1 -- 172.21.15.70:0/1707114620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6119e3200 con 0x55d6119d3c80 2024-06-04T05:15:15.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.267+0000 7f7e50ad5700 1 --2- 172.21.15.70:0/1707114620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d3c80 0x55d6119e5cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7e400e8d30 tx=0x7f7e40005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd920e8e29ae4266 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:15.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.267+0000 7f7e2bfff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:15:15.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.267+0000 7f7e2bfff700 1 -- 172.21.15.70:0/1707114620 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e400b1ac0 con 0x55d6119d3c80 2024-06-04T05:15:15.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.267+0000 7f7e2bfff700 1 -- 172.21.15.70:0/1707114620 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f7e40019c40 con 0x55d6119d3c80 2024-06-04T05:15:15.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.267+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/1707114620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e400b2350 con 0x55d6119d3c80 2024-06-04T05:15:15.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.267+0000 7f7e2bfff700 1 -- 172.21.15.70:0/1707114620 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f7e400b6700 con 0x55d6119d3c80 2024-06-04T05:15:15.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.267+0000 7f7e5d5ceb80 10 client.28940 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:15.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.267+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/1707114620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55d6119e3200 con 0x55d6119d3c80 2024-06-04T05:15:15.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.268+0000 7f7e2bfff700 1 -- 172.21.15.70:0/1707114620 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e400b2350 con 0x55d6119d3c80 2024-06-04T05:15:15.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.268+0000 7f7e2bfff700 1 -- 172.21.15.70:0/1707114620 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f7e40019070 con 0x55d6119d3c80 2024-06-04T05:15:15.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.268+0000 7f7e5d5ceb80 10 client.28940 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:15.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.268+0000 7f7e5d5ceb80 10 client.28940 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:15:15.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.268+0000 7f7e5d5ceb80 20 client.28940 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:15.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.268+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/1707114620 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d611a589a0 0x55d611a5ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:15.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.268+0000 7f7e5d5ceb80 4 client.28940 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}] 2024-06-04T05:15:15.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.268+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/1707114620 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}) -- 0x55d61192d630 con 0x55d611a589a0 2024-06-04T05:15:15.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.268+0000 7f7e4bfff700 1 --2- 172.21.15.70:0/1707114620 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d611a589a0 0x55d611a5ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:15.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.269+0000 7f7e2bfff700 10 client.28940 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:15.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.269+0000 7f7e4bfff700 1 --2- 172.21.15.70:0/1707114620 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d611a589a0 0x55d611a5ad90 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b74bd6204a81501a server_cookie=27e267706261243b in_seq=0 out_seq=0 2024-06-04T05:15:15.296 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:15:15.579 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:15 smithi070 ceph-mon[32631]: pgmap v567: 97 pgs: 97 active+clean; 615 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 322 op/s 2024-06-04T05:15:15.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.600+0000 7f7e2bfff700 1 -- 172.21.15.70:0/1707114620 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55d61192d630 con 0x55d611a589a0 2024-06-04T05:15:15.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.600+0000 7f7e2bfff700 10 client.28940 handle_command_reply: tid=0 2024-06-04T05:15:15.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.600+0000 7f7e5d5ceb80 1 client.28940 shutdown 2024-06-04T05:15:15.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.600+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/1707114620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d3c80 msgr2=0x55d6119e5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:15.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.600+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/1707114620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d3c80 0x55d6119e5cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7e400e8d30 tx=0x7f7e40005fd0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.600+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/1707114620 shutdown_connections 2024-06-04T05:15:15.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.600+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/1707114620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6119d87b0 0x55d6119e2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.600+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/1707114620 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d611a589a0 0x55d611a5ad90 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.600+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/1707114620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6119d3c80 0x55d6119e5cc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.600+0000 7f7e5d5ceb80 1 --2- 172.21.15.70:0/1707114620 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6119d4640 0x55d6119e6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.600+0000 7f7e2bfff700 0 client.28940 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:15.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.600+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/1707114620 >> 172.21.15.70:0/1707114620 conn(0x55d6118ff3d0 msgr2=0x55d6119df830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:15.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.600+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/1707114620 shutdown_connections 2024-06-04T05:15:15.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.600+0000 7f7e5d5ceb80 1 -- 172.21.15.70:0/1707114620 wait complete. 2024-06-04T05:15:15.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.600+0000 7f7e5d5ceb80 20 client.28940 trim_cache size 0 max 16384 2024-06-04T05:15:15.605 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:15:15.605 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 55, 2024-06-04T05:15:15.605 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:15:15.605 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 54, 2024-06-04T05:15:15.605 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:15:15.605 INFO:teuthology.orchestra.run.smithi070.stdout: "77470b0": { 2024-06-04T05:15:15.605 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 54, 2024-06-04T05:15:15.605 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:15:15.605 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:15:15.605 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:15:15.606 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:15:15.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:15:15.606 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:15:15.606 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 130.0, 2024-06-04T05:15:15.606 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:15:15.606 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:15:15.606 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:15:15.606 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:15:15.606 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:15:15.606 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:15:15.606 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:15:15.606 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:15:15.606 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:15:15.607 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:15:15.607 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:15:15.607 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:15:15.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.601+0000 7f7e52d39700 1 -- 172.21.15.70:0/1334534455 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7e3c0b2b20 msgr2=0x7f7e3c0b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:15.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.601+0000 7f7e52d39700 1 --2- 172.21.15.70:0/1334534455 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7e3c0b2b20 0x7f7e3c0b4fe0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c079cf0 tx=0x7f7e38009260 comp rx=0 tx=0).stop 2024-06-04T05:15:15.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.601+0000 7f7e52d39700 1 -- 172.21.15.70:0/1334534455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e4c102ae0 msgr2=0x7f7e4c19d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:15.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.601+0000 7f7e52d39700 1 --2- 172.21.15.70:0/1334534455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e4c102ae0 0x7f7e4c19d600 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7e400131b0 tx=0x7f7e40011380 comp rx=0 tx=0).stop 2024-06-04T05:15:15.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.601+0000 7f7e52d39700 1 -- 172.21.15.70:0/1334534455 shutdown_connections 2024-06-04T05:15:15.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.601+0000 7f7e52d39700 1 --2- 172.21.15.70:0/1334534455 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7e3c0b2b20 0x7f7e3c0b4fe0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.601+0000 7f7e52d39700 1 --2- 172.21.15.70:0/1334534455 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e4c107a30 0x7f7e4c197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.601+0000 7f7e52d39700 1 --2- 172.21.15.70:0/1334534455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e4c102ae0 0x7f7e4c19d600 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.601+0000 7f7e52d39700 1 --2- 172.21.15.70:0/1334534455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e4c107070 0x7f7e4c19db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:15.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.601+0000 7f7e52d39700 1 -- 172.21.15.70:0/1334534455 >> 172.21.15.70:0/1334534455 conn(0x7f7e4c0793f0 msgr2=0x7f7e4c106050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:15.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.601+0000 7f7e52d39700 1 -- 172.21.15.70:0/1334534455 shutdown_connections 2024-06-04T05:15:15.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:15.601+0000 7f7e52d39700 1 -- 172.21.15.70:0/1334534455 wait complete. 2024-06-04T05:15:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:15 smithi112 ceph-mon[36453]: pgmap v567: 97 pgs: 97 active+clean; 615 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 322 op/s 2024-06-04T05:15:15.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:15 smithi098 ceph-mon[35569]: pgmap v567: 97 pgs: 97 active+clean; 615 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 322 op/s 2024-06-04T05:15:15.815 INFO:tasks.workunit.client.0.smithi070.stdout: 10 36180 8.35 MB/sec execute 127 sec latency 530.021 ms 2024-06-04T05:15:16.816 INFO:tasks.workunit.client.0.smithi070.stdout: 10 36180 8.29 MB/sec execute 128 sec latency 1530.123 ms 2024-06-04T05:15:17.552 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:17 smithi070 ceph-mon[32631]: pgmap v568: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.2 MiB/s wr, 354 op/s 2024-06-04T05:15:17.577 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '77470b0', quiesce took 25.0 seconds. Will release after: -15.0 2024-06-04T05:15:17.578 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:15:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:17 smithi112 ceph-mon[36453]: pgmap v568: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.2 MiB/s wr, 354 op/s 2024-06-04T05:15:17.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:17 smithi098 ceph-mon[35569]: pgmap v568: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.2 MiB/s wr, 354 op/s 2024-06-04T05:15:17.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.792+0000 7fef7d3d8700 1 -- 172.21.15.70:0/375891260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef78102140 msgr2=0x7fef78102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:17.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.792+0000 7fef7d3d8700 1 --2- 172.21.15.70:0/375891260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef78102140 0x7fef78102520 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fef68009990 tx=0x7fef68009ca0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.792+0000 7fef7d3d8700 1 -- 172.21.15.70:0/375891260 shutdown_connections 2024-06-04T05:15:17.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.792+0000 7fef7d3d8700 1 --2- 172.21.15.70:0/375891260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78106fe0 0x7fef7810eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.792+0000 7fef7d3d8700 1 --2- 172.21.15.70:0/375891260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef78102140 0x7fef78102520 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.792+0000 7fef7d3d8700 1 --2- 172.21.15.70:0/375891260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef78102af0 0x7fef781068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.792+0000 7fef7d3d8700 1 -- 172.21.15.70:0/375891260 >> 172.21.15.70:0/375891260 conn(0x7fef78077760 msgr2=0x7fef78077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:17.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.792+0000 7fef7d3d8700 1 -- 172.21.15.70:0/375891260 shutdown_connections 2024-06-04T05:15:17.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.792+0000 7fef7d3d8700 1 -- 172.21.15.70:0/375891260 wait complete. 2024-06-04T05:15:17.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.793+0000 7fef7d3d8700 1 Processor -- start 2024-06-04T05:15:17.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.793+0000 7fef7d3d8700 1 -- start start 2024-06-04T05:15:17.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.793+0000 7fef7d3d8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef78102140 0x7fef7819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.793+0000 7fef7d3d8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78102af0 0x7fef7819fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.793+0000 7fef7d3d8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef78106fe0 0x7fef781999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.793+0000 7fef7d3d8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fef78112550 con 0x7fef78102140 2024-06-04T05:15:17.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.793+0000 7fef7d3d8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef781123d0 con 0x7fef78102af0 2024-06-04T05:15:17.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.793+0000 7fef7d3d8700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fef781126d0 con 0x7fef78106fe0 2024-06-04T05:15:17.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.793+0000 7fef76ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef78102140 0x7fef7819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.794+0000 7fef76ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef78102140 0x7fef7819f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35664/0 (socket says 172.21.15.70:35664) 2024-06-04T05:15:17.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.794+0000 7fef76ffd700 1 -- 172.21.15.70:0/3864886064 learned_addr learned my addr 172.21.15.70:0/3864886064 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:17.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.794+0000 7fef767fc700 1 --2- 172.21.15.70:0/3864886064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78102af0 0x7fef7819fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.794+0000 7fef777fe700 1 --2- 172.21.15.70:0/3864886064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef78106fe0 0x7fef781999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.794+0000 7fef76ffd700 1 -- 172.21.15.70:0/3864886064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef78106fe0 msgr2=0x7fef781999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:17.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.794+0000 7fef76ffd700 1 --2- 172.21.15.70:0/3864886064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef78106fe0 0x7fef781999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.794+0000 7fef76ffd700 1 -- 172.21.15.70:0/3864886064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78102af0 msgr2=0x7fef7819fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:17.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.794+0000 7fef76ffd700 1 --2- 172.21.15.70:0/3864886064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78102af0 0x7fef7819fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.794+0000 7fef76ffd700 1 -- 172.21.15.70:0/3864886064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef7819a120 con 0x7fef78102140 2024-06-04T05:15:17.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.794+0000 7fef767fc700 1 --2- 172.21.15.70:0/3864886064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78102af0 0x7fef7819fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:15:17.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.794+0000 7fef777fe700 1 --2- 172.21.15.70:0/3864886064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef78106fe0 0x7fef781999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:15:17.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.794+0000 7fef76ffd700 1 --2- 172.21.15.70:0/3864886064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef78102140 0x7fef7819f7b0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fef680057c0 tx=0x7fef680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:17.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.794+0000 7fef5ffff700 1 -- 172.21.15.70:0/3864886064 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef68027070 con 0x7fef78102140 2024-06-04T05:15:17.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.794+0000 7fef5ffff700 1 -- 172.21.15.70:0/3864886064 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef68003ff0 con 0x7fef78102140 2024-06-04T05:15:17.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.794+0000 7fef7d3d8700 1 -- 172.21.15.70:0/3864886064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef7819a3b0 con 0x7fef78102140 2024-06-04T05:15:17.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.794+0000 7fef5ffff700 1 -- 172.21.15.70:0/3864886064 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef68022d60 con 0x7fef78102140 2024-06-04T05:15:17.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.794+0000 7fef7d3d8700 1 -- 172.21.15.70:0/3864886064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef781a6680 con 0x7fef78102140 2024-06-04T05:15:17.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.797+0000 7fef5ffff700 1 -- 172.21.15.70:0/3864886064 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fef6802a070 con 0x7fef78102140 2024-06-04T05:15:17.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.798+0000 7fef5ffff700 1 --2- 172.21.15.70:0/3864886064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fef600b29f0 0x7fef600b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.798+0000 7fef767fc700 1 --2- 172.21.15.70:0/3864886064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fef600b29f0 0x7fef600b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.798+0000 7fef5ffff700 1 -- 172.21.15.70:0/3864886064 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7fef680e35c0 con 0x7fef78102140 2024-06-04T05:15:17.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.798+0000 7fef767fc700 1 --2- 172.21.15.70:0/3864886064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fef600b29f0 0x7fef600b4eb0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fef78106b00 tx=0x7fef6c008040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:17.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.798+0000 7fef87c6db80 1 Processor -- start 2024-06-04T05:15:17.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.798+0000 7fef87c6db80 1 -- start start 2024-06-04T05:15:17.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef87c6db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b892860 0x556d8b892c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef87c6db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b893210 0x556d8b896940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef87c6db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b896f60 0x556d8b8973a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef87c6db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556d8b7f1c90 con 0x556d8b896f60 2024-06-04T05:15:17.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef87c6db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556d8b7e1830 con 0x556d8b892860 2024-06-04T05:15:17.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef87c6db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556d8b7e5870 con 0x556d8b893210 2024-06-04T05:15:17.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef777fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b896f60 0x556d8b8973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef777fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b896f60 0x556d8b8973a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35676/0 (socket says 172.21.15.70:35676) 2024-06-04T05:15:17.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef777fe700 1 -- 172.21.15.70:0/517034270 learned_addr learned my addr 172.21.15.70:0/517034270 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:17.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef76ffd700 1 --2- 172.21.15.70:0/517034270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b892860 0x556d8b892c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef767fc700 1 --2- 172.21.15.70:0/517034270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b893210 0x556d8b896940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef777fe700 1 -- 172.21.15.70:0/517034270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b893210 msgr2=0x556d8b896940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:17.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef777fe700 1 --2- 172.21.15.70:0/517034270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b893210 0x556d8b896940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef777fe700 1 -- 172.21.15.70:0/517034270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b892860 msgr2=0x556d8b892c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:17.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef777fe700 1 --2- 172.21.15.70:0/517034270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b892860 0x556d8b892c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef777fe700 1 -- 172.21.15.70:0/517034270 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d8b7318d0 con 0x556d8b896f60 2024-06-04T05:15:17.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef777fe700 1 --2- 172.21.15.70:0/517034270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b896f60 0x556d8b8973a0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fef7000e260 tx=0x7fef7000e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd5d340c8e96d441 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:17.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef5d7fa700 1 -- 172.21.15.70:0/517034270 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7000ee00 con 0x556d8b896f60 2024-06-04T05:15:17.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef5d7fa700 1 -- 172.21.15.70:0/517034270 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef70004bb0 con 0x556d8b896f60 2024-06-04T05:15:17.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.799+0000 7fef5d7fa700 1 -- 172.21.15.70:0/517034270 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef70014e60 con 0x556d8b896f60 2024-06-04T05:15:17.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.800+0000 7fef87c6db80 1 -- 172.21.15.70:0/517034270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b896f60 msgr2=0x556d8b8973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:17.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.800+0000 7fef87c6db80 1 --2- 172.21.15.70:0/517034270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b896f60 0x556d8b8973a0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fef7000e260 tx=0x7fef7000e570 comp rx=0 tx=0).stop 2024-06-04T05:15:17.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.800+0000 7fef87c6db80 1 -- 172.21.15.70:0/517034270 shutdown_connections 2024-06-04T05:15:17.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.800+0000 7fef87c6db80 1 --2- 172.21.15.70:0/517034270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b896f60 0x556d8b8973a0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.800+0000 7fef87c6db80 1 --2- 172.21.15.70:0/517034270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b892860 0x556d8b892c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.800+0000 7fef87c6db80 1 --2- 172.21.15.70:0/517034270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b893210 0x556d8b896940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.800+0000 7fef87c6db80 1 -- 172.21.15.70:0/517034270 >> 172.21.15.70:0/517034270 conn(0x556d8b7c0080 msgr2=0x556d8b88ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:17.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.800+0000 7fef87c6db80 1 -- 172.21.15.70:0/517034270 shutdown_connections 2024-06-04T05:15:17.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.800+0000 7fef87c6db80 1 -- 172.21.15.70:0/517034270 wait complete. 2024-06-04T05:15:17.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.801+0000 7fef87c6db80 1 -- start start 2024-06-04T05:15:17.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.801+0000 7fef87c6db80 1 Processor -- start 2024-06-04T05:15:17.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.801+0000 7fef87c6db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b892860 0x556d8b89e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.801+0000 7fef87c6db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b893210 0x556d8b89b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.801+0000 7fef87c6db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b896f60 0x556d8b89bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.801+0000 7fef87c6db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556d8b6e7fa0 con 0x556d8b892860 2024-06-04T05:15:17.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.801+0000 7fef87c6db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556d8b7837c0 con 0x556d8b893210 2024-06-04T05:15:17.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.801+0000 7fef87c6db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556d8b89f120 con 0x556d8b896f60 2024-06-04T05:15:17.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.801+0000 7fef87c6db80 10 client.0 _ll_register_callbacks cb 0x556d8b77c920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:17.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.801+0000 7fef76ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b892860 0x556d8b89e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.801+0000 7fef76ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b892860 0x556d8b89e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35680/0 (socket says 172.21.15.70:35680) 2024-06-04T05:15:17.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.801+0000 7fef76ffd700 1 -- 172.21.15.70:0/1937906084 learned_addr learned my addr 172.21.15.70:0/1937906084 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:17.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.801+0000 7fef767fc700 1 --2- 172.21.15.70:0/1937906084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b893210 0x556d8b89b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.801+0000 7fef777fe700 1 --2- 172.21.15.70:0/1937906084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b896f60 0x556d8b89bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.802+0000 7fef76ffd700 1 -- 172.21.15.70:0/1937906084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b896f60 msgr2=0x556d8b89bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:17.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.802+0000 7fef76ffd700 1 --2- 172.21.15.70:0/1937906084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b896f60 0x556d8b89bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.802+0000 7fef76ffd700 1 -- 172.21.15.70:0/1937906084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b893210 msgr2=0x556d8b89b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:17.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.802+0000 7fef76ffd700 1 --2- 172.21.15.70:0/1937906084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b893210 0x556d8b89b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.802+0000 7fef76ffd700 1 -- 172.21.15.70:0/1937906084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d8b780690 con 0x556d8b892860 2024-06-04T05:15:17.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.802+0000 7fef777fe700 1 --2- 172.21.15.70:0/1937906084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b896f60 0x556d8b89bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:15:17.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.802+0000 7fef76ffd700 1 --2- 172.21.15.70:0/1937906084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b892860 0x556d8b89e3e0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fef680e0ce0 tx=0x7fef680227e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd200bfab1891d8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:17.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.802+0000 7fef5aff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:15:17.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.802+0000 7fef5aff5700 1 -- 172.21.15.70:0/1937906084 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef680aa3b0 con 0x556d8b892860 2024-06-04T05:15:17.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.802+0000 7fef5aff5700 1 -- 172.21.15.70:0/1937906084 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7fef680aab00 con 0x556d8b892860 2024-06-04T05:15:17.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.802+0000 7fef87c6db80 1 -- 172.21.15.70:0/1937906084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef70004bb0 con 0x556d8b892860 2024-06-04T05:15:17.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.802+0000 7fef5aff5700 1 -- 172.21.15.70:0/1937906084 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fef680aafa0 con 0x556d8b892860 2024-06-04T05:15:17.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.802+0000 7fef87c6db80 10 client.20133 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:17.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.802+0000 7fef87c6db80 1 -- 172.21.15.70:0/1937906084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x556d8b8a9850 con 0x556d8b892860 2024-06-04T05:15:17.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.802+0000 7fef5aff5700 1 -- 172.21.15.70:0/1937906084 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef70004bb0 con 0x556d8b892860 2024-06-04T05:15:17.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.802+0000 7fef5aff5700 1 -- 172.21.15.70:0/1937906084 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7fef680abbd0 con 0x556d8b892860 2024-06-04T05:15:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.803+0000 7fef87c6db80 10 client.20133 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.803+0000 7fef87c6db80 10 client.20133 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:15:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.803+0000 7fef87c6db80 20 client.20133 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.803+0000 7fef87c6db80 1 --2- 172.21.15.70:0/1937906084 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x556d8b8abd70 0x556d8b8ae180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.803+0000 7fef87c6db80 4 client.20133 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:15:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.803+0000 7fef87c6db80 1 -- 172.21.15.70:0/1937906084 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d8b85e9e0 con 0x556d8b8abd70 2024-06-04T05:15:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.803+0000 7fef767fc700 1 --2- 172.21.15.70:0/1937906084 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x556d8b8abd70 0x556d8b8ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.803+0000 7fef5aff5700 10 client.20133 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:15:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.803+0000 7fef767fc700 1 --2- 172.21.15.70:0/1937906084 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x556d8b8abd70 0x556d8b8ae180 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9fd064f8d5eb0a73 server_cookie=7a65763d4cd76c2 in_seq=0 out_seq=0 2024-06-04T05:15:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.804+0000 7fef5aff5700 1 -- 172.21.15.70:0/1937906084 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556d8b85e9e0 con 0x556d8b8abd70 2024-06-04T05:15:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.804+0000 7fef5aff5700 10 client.20133 handle_command_reply: tid=0 2024-06-04T05:15:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.804+0000 7fef87c6db80 1 client.20133 shutdown 2024-06-04T05:15:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.804+0000 7fef87c6db80 1 -- 172.21.15.70:0/1937906084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b892860 msgr2=0x556d8b89e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.804+0000 7fef87c6db80 1 --2- 172.21.15.70:0/1937906084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b892860 0x556d8b89e3e0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fef680e0ce0 tx=0x7fef680227e0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.804+0000 7fef87c6db80 1 -- 172.21.15.70:0/1937906084 shutdown_connections 2024-06-04T05:15:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.804+0000 7fef87c6db80 1 --2- 172.21.15.70:0/1937906084 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x556d8b8abd70 0x556d8b8ae180 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.804+0000 7fef87c6db80 1 --2- 172.21.15.70:0/1937906084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b896f60 0x556d8b89bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.804+0000 7fef87c6db80 1 --2- 172.21.15.70:0/1937906084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b892860 0x556d8b89e3e0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.804+0000 7fef87c6db80 1 --2- 172.21.15.70:0/1937906084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b893210 0x556d8b89b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.804+0000 7fef87c6db80 1 -- 172.21.15.70:0/1937906084 >> 172.21.15.70:0/1937906084 conn(0x556d8b7c0080 msgr2=0x556d8b898350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.804+0000 7fef5aff5700 0 client.20133 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:15:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.804+0000 7fef87c6db80 1 -- 172.21.15.70:0/1937906084 shutdown_connections 2024-06-04T05:15:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.804+0000 7fef87c6db80 1 -- 172.21.15.70:0/1937906084 wait complete. 2024-06-04T05:15:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.804+0000 7fef87c6db80 20 client.20133 trim_cache size 0 max 16384 2024-06-04T05:15:17.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 36180 8.22 MB/sec execute 129 sec latency 2530.180 ms 2024-06-04T05:15:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.820+0000 7fef87c6db80 1 Processor -- start 2024-06-04T05:15:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.820+0000 7fef87c6db80 1 -- start start 2024-06-04T05:15:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.820+0000 7fef87c6db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b893210 0x556d8b89a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.820+0000 7fef87c6db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b896f60 0x556d8b88d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.820+0000 7fef87c6db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b8a81c0 0x556d8b89d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.820+0000 7fef87c6db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fef781ade20 con 0x556d8b896f60 2024-06-04T05:15:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.820+0000 7fef87c6db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef781af9c0 con 0x556d8b8a81c0 2024-06-04T05:15:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.820+0000 7fef87c6db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fef781ae550 con 0x556d8b893210 2024-06-04T05:15:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.820+0000 7fef767fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b896f60 0x556d8b88d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.820+0000 7fef767fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b896f60 0x556d8b88d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35692/0 (socket says 172.21.15.70:35692) 2024-06-04T05:15:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.820+0000 7fef767fc700 1 -- 172.21.15.70:0/2744857393 learned_addr learned my addr 172.21.15.70:0/2744857393 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.821+0000 7fef777fe700 1 --2- 172.21.15.70:0/2744857393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b8a81c0 0x556d8b89d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.821+0000 7fef76ffd700 1 --2- 172.21.15.70:0/2744857393 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b893210 0x556d8b89a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.821+0000 7fef767fc700 1 -- 172.21.15.70:0/2744857393 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b893210 msgr2=0x556d8b89a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.821+0000 7fef767fc700 1 --2- 172.21.15.70:0/2744857393 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b893210 0x556d8b89a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.821+0000 7fef767fc700 1 -- 172.21.15.70:0/2744857393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b8a81c0 msgr2=0x556d8b89d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.821+0000 7fef767fc700 1 --2- 172.21.15.70:0/2744857393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b8a81c0 0x556d8b89d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.821+0000 7fef767fc700 1 -- 172.21.15.70:0/2744857393 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d8b8ae4a0 con 0x556d8b896f60 2024-06-04T05:15:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.821+0000 7fef767fc700 1 --2- 172.21.15.70:0/2744857393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b896f60 0x556d8b88d030 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fef6c008010 tx=0x7fef6c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb70f4377bcb571 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.821+0000 7fef5cff9700 1 -- 172.21.15.70:0/2744857393 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c032070 con 0x556d8b896f60 2024-06-04T05:15:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.821+0000 7fef5cff9700 1 -- 172.21.15.70:0/2744857393 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef6c00d850 con 0x556d8b896f60 2024-06-04T05:15:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.821+0000 7fef5cff9700 1 -- 172.21.15.70:0/2744857393 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c02d880 con 0x556d8b896f60 2024-06-04T05:15:17.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.822+0000 7fef87c6db80 1 -- 172.21.15.70:0/2744857393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b896f60 msgr2=0x556d8b88d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:17.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.822+0000 7fef87c6db80 1 --2- 172.21.15.70:0/2744857393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b896f60 0x556d8b88d030 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fef6c008010 tx=0x7fef6c013000 comp rx=0 tx=0).stop 2024-06-04T05:15:17.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.822+0000 7fef87c6db80 1 -- 172.21.15.70:0/2744857393 shutdown_connections 2024-06-04T05:15:17.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.822+0000 7fef87c6db80 1 --2- 172.21.15.70:0/2744857393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b8a81c0 0x556d8b89d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.822+0000 7fef87c6db80 1 --2- 172.21.15.70:0/2744857393 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b893210 0x556d8b89a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.822+0000 7fef87c6db80 1 --2- 172.21.15.70:0/2744857393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b896f60 0x556d8b88d030 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.822+0000 7fef87c6db80 1 -- 172.21.15.70:0/2744857393 >> 172.21.15.70:0/2744857393 conn(0x556d8b7c0080 msgr2=0x556d8b89cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:17.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.822+0000 7fef87c6db80 1 -- 172.21.15.70:0/2744857393 shutdown_connections 2024-06-04T05:15:17.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.822+0000 7fef87c6db80 1 -- 172.21.15.70:0/2744857393 wait complete. 2024-06-04T05:15:17.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.822+0000 7fef87c6db80 1 -- start start 2024-06-04T05:15:17.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.822+0000 7fef87c6db80 1 Processor -- start 2024-06-04T05:15:17.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef87c6db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b893210 0x556d8b894640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef87c6db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b896f60 0x556d8b894b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef87c6db80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b8a81c0 0x556d8b8950c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef87c6db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fef781af2f0 con 0x556d8b893210 2024-06-04T05:15:17.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef87c6db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef781aec20 con 0x556d8b896f60 2024-06-04T05:15:17.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef87c6db80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fef781abde0 con 0x556d8b8a81c0 2024-06-04T05:15:17.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef76ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b893210 0x556d8b894640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef87c6db80 10 client.0 _ll_register_callbacks cb 0x556d8b77c920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:17.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef777fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b8a81c0 0x556d8b8950c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef76ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b893210 0x556d8b894640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35696/0 (socket says 172.21.15.70:35696) 2024-06-04T05:15:17.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef76ffd700 1 -- 172.21.15.70:0/1133756885 learned_addr learned my addr 172.21.15.70:0/1133756885 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:17.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef767fc700 1 --2- 172.21.15.70:0/1133756885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b896f60 0x556d8b894b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef76ffd700 1 -- 172.21.15.70:0/1133756885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b8a81c0 msgr2=0x556d8b8950c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:17.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef76ffd700 1 --2- 172.21.15.70:0/1133756885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b8a81c0 0x556d8b8950c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef76ffd700 1 -- 172.21.15.70:0/1133756885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b896f60 msgr2=0x556d8b894b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:17.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef76ffd700 1 --2- 172.21.15.70:0/1133756885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b896f60 0x556d8b894b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef76ffd700 1 -- 172.21.15.70:0/1133756885 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d8b9159b0 con 0x556d8b893210 2024-06-04T05:15:17.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef76ffd700 1 --2- 172.21.15.70:0/1133756885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b893210 0x556d8b894640 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fef680a6920 tx=0x7fef680aa8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=901e2108139f03af server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:17.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.823+0000 7fef5aff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:15:17.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.824+0000 7fef5aff5700 1 -- 172.21.15.70:0/1133756885 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef680aa3b0 con 0x556d8b893210 2024-06-04T05:15:17.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.824+0000 7fef5aff5700 1 -- 172.21.15.70:0/1133756885 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7fef680aab00 con 0x556d8b893210 2024-06-04T05:15:17.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.824+0000 7fef87c6db80 1 -- 172.21.15.70:0/1133756885 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef6c00d850 con 0x556d8b893210 2024-06-04T05:15:17.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.824+0000 7fef5aff5700 1 -- 172.21.15.70:0/1133756885 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fef680aec50 con 0x556d8b893210 2024-06-04T05:15:17.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.824+0000 7fef87c6db80 10 client.20139 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.824+0000 7fef87c6db80 1 -- 172.21.15.70:0/1133756885 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x556d8b9159b0 con 0x556d8b893210 2024-06-04T05:15:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.824+0000 7fef5aff5700 1 -- 172.21.15.70:0/1133756885 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef6c00d850 con 0x556d8b893210 2024-06-04T05:15:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.824+0000 7fef5aff5700 1 -- 172.21.15.70:0/1133756885 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7fef680a9b00 con 0x556d8b893210 2024-06-04T05:15:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.824+0000 7fef87c6db80 10 client.20139 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.824+0000 7fef87c6db80 10 client.20139 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:15:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.824+0000 7fef87c6db80 20 client.20139 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.824+0000 7fef87c6db80 1 --2- 172.21.15.70:0/1133756885 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x556d8b9170e0 0x556d8b919500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:17.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.824+0000 7fef87c6db80 4 client.20139 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:15:17.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.824+0000 7fef87c6db80 1 -- 172.21.15.70:0/1133756885 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556d8b7f3be0 con 0x556d8b9170e0 2024-06-04T05:15:17.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.825+0000 7fef767fc700 1 --2- 172.21.15.70:0/1133756885 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x556d8b9170e0 0x556d8b919500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:17.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.825+0000 7fef5aff5700 10 client.20139 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:15:17.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.825+0000 7fef767fc700 1 --2- 172.21.15.70:0/1133756885 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x556d8b9170e0 0x556d8b919500 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3043c4c721496b54 server_cookie=fcfaff6c91467224 in_seq=0 out_seq=0 2024-06-04T05:15:17.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.825+0000 7fef5aff5700 1 -- 172.21.15.70:0/1133756885 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556d8b7f3be0 con 0x556d8b9170e0 2024-06-04T05:15:17.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.825+0000 7fef5aff5700 10 client.20139 handle_command_reply: tid=0 2024-06-04T05:15:17.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.825+0000 7fef87c6db80 1 client.20139 shutdown 2024-06-04T05:15:17.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.825+0000 7fef87c6db80 1 -- 172.21.15.70:0/1133756885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b893210 msgr2=0x556d8b894640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:17.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.825+0000 7fef87c6db80 1 --2- 172.21.15.70:0/1133756885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b893210 0x556d8b894640 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fef680a6920 tx=0x7fef680aa8a0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.825+0000 7fef87c6db80 1 -- 172.21.15.70:0/1133756885 shutdown_connections 2024-06-04T05:15:17.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.826+0000 7fef87c6db80 1 --2- 172.21.15.70:0/1133756885 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x556d8b9170e0 0x556d8b919500 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.826+0000 7fef87c6db80 1 --2- 172.21.15.70:0/1133756885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556d8b8a81c0 0x556d8b8950c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.826+0000 7fef87c6db80 1 --2- 172.21.15.70:0/1133756885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d8b893210 0x556d8b894640 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.826+0000 7fef5aff5700 0 client.20139 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:15:17.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.826+0000 7fef87c6db80 1 --2- 172.21.15.70:0/1133756885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d8b896f60 0x556d8b894b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.826+0000 7fef87c6db80 1 -- 172.21.15.70:0/1133756885 >> 172.21.15.70:0/1133756885 conn(0x556d8b7c0080 msgr2=0x556d8b891670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:17.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.826+0000 7fef87c6db80 1 -- 172.21.15.70:0/1133756885 shutdown_connections 2024-06-04T05:15:17.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.826+0000 7fef87c6db80 1 -- 172.21.15.70:0/1133756885 wait complete. 2024-06-04T05:15:17.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.826+0000 7fef87c6db80 20 client.20139 trim_cache size 0 max 16384 2024-06-04T05:15:17.837 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:15:17.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.826+0000 7fef7d3d8700 1 -- 172.21.15.70:0/3864886064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fef600b29f0 msgr2=0x7fef600b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:17.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.826+0000 7fef7d3d8700 1 --2- 172.21.15.70:0/3864886064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fef600b29f0 0x7fef600b4eb0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fef78106b00 tx=0x7fef6c008040 comp rx=0 tx=0).stop 2024-06-04T05:15:17.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.826+0000 7fef7d3d8700 1 -- 172.21.15.70:0/3864886064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef78102140 msgr2=0x7fef7819f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:17.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.826+0000 7fef7d3d8700 1 --2- 172.21.15.70:0/3864886064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef78102140 0x7fef7819f7b0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fef680057c0 tx=0x7fef680057f0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.827+0000 7fef7d3d8700 1 -- 172.21.15.70:0/3864886064 shutdown_connections 2024-06-04T05:15:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.827+0000 7fef7d3d8700 1 --2- 172.21.15.70:0/3864886064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fef600b29f0 0x7fef600b4eb0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.827+0000 7fef7d3d8700 1 --2- 172.21.15.70:0/3864886064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef78106fe0 0x7fef781999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.827+0000 7fef7d3d8700 1 --2- 172.21.15.70:0/3864886064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef78102140 0x7fef7819f7b0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.827+0000 7fef7d3d8700 1 --2- 172.21.15.70:0/3864886064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78102af0 0x7fef7819fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.827+0000 7fef7d3d8700 1 -- 172.21.15.70:0/3864886064 >> 172.21.15.70:0/3864886064 conn(0x7fef78077760 msgr2=0x7fef7810f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.827+0000 7fef7d3d8700 1 -- 172.21.15.70:0/3864886064 shutdown_connections 2024-06-04T05:15:17.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:17.827+0000 7fef7d3d8700 1 -- 172.21.15.70:0/3864886064 wait complete. 2024-06-04T05:15:18.100 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:15:18.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:15:18.816 INFO:tasks.workunit.client.0.smithi070.stdout: 10 36180 8.16 MB/sec execute 130 sec latency 3530.210 ms 2024-06-04T05:15:19.600 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:19 smithi070 ceph-mon[32631]: pgmap v569: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 346 op/s 2024-06-04T05:15:19.621 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:15:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:19 smithi112 ceph-mon[36453]: pgmap v569: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 346 op/s 2024-06-04T05:15:19.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:19 smithi098 ceph-mon[35569]: pgmap v569: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 346 op/s 2024-06-04T05:15:19.816 INFO:tasks.workunit.client.0.smithi070.stdout: 10 36180 8.10 MB/sec execute 131 sec latency 4530.307 ms 2024-06-04T05:15:20.816 INFO:tasks.workunit.client.0.smithi070.stdout: 10 36180 8.04 MB/sec execute 132 sec latency 5530.397 ms 2024-06-04T05:15:20.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.959+0000 7fd2eab89700 1 -- 172.21.15.70:0/161981455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2e406c1b0 msgr2=0x7fd2e406c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.959+0000 7fd2eab89700 1 --2- 172.21.15.70:0/161981455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2e406c1b0 0x7fd2e406c630 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0009990 tx=0x7fd2d0009ca0 comp rx=0 tx=0).stop 2024-06-04T05:15:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.959+0000 7fd2eab89700 1 -- 172.21.15.70:0/161981455 shutdown_connections 2024-06-04T05:15:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.959+0000 7fd2eab89700 1 --2- 172.21.15.70:0/161981455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e4105f70 0x7fd2e4106420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.959+0000 7fd2eab89700 1 --2- 172.21.15.70:0/161981455 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2e4108460 0x7fd2e406bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.959+0000 7fd2eab89700 1 --2- 172.21.15.70:0/161981455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2e406c1b0 0x7fd2e406c630 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.959+0000 7fd2eab89700 1 -- 172.21.15.70:0/161981455 >> 172.21.15.70:0/161981455 conn(0x7fd2e40793f0 msgr2=0x7fd2e4079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.959+0000 7fd2eab89700 1 -- 172.21.15.70:0/161981455 shutdown_connections 2024-06-04T05:15:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.959+0000 7fd2eab89700 1 -- 172.21.15.70:0/161981455 wait complete. 2024-06-04T05:15:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.959+0000 7fd2eab89700 1 Processor -- start 2024-06-04T05:15:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.959+0000 7fd2eab89700 1 -- start start 2024-06-04T05:15:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2eab89700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2e406c1b0 0x7fd2e41a0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2eab89700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2e4105f70 0x7fd2e41a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2eab89700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e4108460 0x7fd2e41a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2eab89700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e4119bc0 con 0x7fd2e4105f70 2024-06-04T05:15:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2eab89700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e4119a40 con 0x7fd2e4108460 2024-06-04T05:15:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2eab89700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e4119d40 con 0x7fd2e406c1b0 2024-06-04T05:15:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2e3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2e4105f70 0x7fd2e41a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2e3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2e4105f70 0x7fd2e41a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35706/0 (socket says 172.21.15.70:35706) 2024-06-04T05:15:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2e3fff700 1 -- 172.21.15.70:0/1967998613 learned_addr learned my addr 172.21.15.70:0/1967998613 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2e9126700 1 --2- 172.21.15.70:0/1967998613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e4108460 0x7fd2e41a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2e8925700 1 --2- 172.21.15.70:0/1967998613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2e406c1b0 0x7fd2e41a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2e3fff700 1 -- 172.21.15.70:0/1967998613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2e406c1b0 msgr2=0x7fd2e41a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2e3fff700 1 --2- 172.21.15.70:0/1967998613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2e406c1b0 0x7fd2e41a0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2e3fff700 1 -- 172.21.15.70:0/1967998613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e4108460 msgr2=0x7fd2e41a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2e3fff700 1 --2- 172.21.15.70:0/1967998613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e4108460 0x7fd2e41a4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2e3fff700 1 -- 172.21.15.70:0/1967998613 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e41a52a0 con 0x7fd2e4105f70 2024-06-04T05:15:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2e3fff700 1 --2- 172.21.15.70:0/1967998613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2e4105f70 0x7fd2e41a07d0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0000c00 tx=0x7fd2d00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.960+0000 7fd2e1ffb700 1 -- 172.21.15.70:0/1967998613 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0027070 con 0x7fd2e4105f70 2024-06-04T05:15:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.961+0000 7fd2eab89700 1 -- 172.21.15.70:0/1967998613 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2e41a5530 con 0x7fd2e4105f70 2024-06-04T05:15:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.961+0000 7fd2e1ffb700 1 -- 172.21.15.70:0/1967998613 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd2d000fb00 con 0x7fd2e4105f70 2024-06-04T05:15:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.961+0000 7fd2e1ffb700 1 -- 172.21.15.70:0/1967998613 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0022c40 con 0x7fd2e4105f70 2024-06-04T05:15:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.961+0000 7fd2eab89700 1 -- 172.21.15.70:0/1967998613 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2e41ace10 con 0x7fd2e4105f70 2024-06-04T05:15:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.962+0000 7fd2eab89700 1 -- 172.21.15.70:0/1967998613 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2e4102a50 con 0x7fd2e4105f70 2024-06-04T05:15:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.963+0000 7fd2e1ffb700 1 -- 172.21.15.70:0/1967998613 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fd2d002a070 con 0x7fd2e4105f70 2024-06-04T05:15:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.964+0000 7fd2e1ffb700 1 --2- 172.21.15.70:0/1967998613 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2d40b29f0 0x7fd2d40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.964+0000 7fd2e8925700 1 --2- 172.21.15.70:0/1967998613 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2d40b29f0 0x7fd2d40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.964+0000 7fd2e1ffb700 1 -- 172.21.15.70:0/1967998613 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7fd2d00e3950 con 0x7fd2e4105f70 2024-06-04T05:15:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.964+0000 7fd2e8925700 1 --2- 172.21.15.70:0/1967998613 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2d40b29f0 0x7fd2d40b4eb0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80045b0 tx=0x7fd2d800b3c0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:20.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:20.967+0000 7fd2e1ffb700 1 -- 172.21.15.70:0/1967998613 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd2d0031070 con 0x7fd2e4105f70 2024-06-04T05:15:21.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:21.226+0000 7fd2eab89700 1 -- 172.21.15.70:0/1967998613 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2e41a15f0 con 0x7fd2e4105f70 2024-06-04T05:15:21.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:21.227+0000 7fd2e1ffb700 1 -- 172.21.15.70:0/1967998613 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+11932 (secure 0 0 0) 0x7fd2d0013070 con 0x7fd2e4105f70 2024-06-04T05:15:21.229 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:15:21.229 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":55,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:14:18.668019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:15:21.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:21.229+0000 7fd2eab89700 1 -- 172.21.15.70:0/1967998613 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2d40b29f0 msgr2=0x7fd2d40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:21.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:21.229+0000 7fd2eab89700 1 --2- 172.21.15.70:0/1967998613 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2d40b29f0 0x7fd2d40b4eb0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80045b0 tx=0x7fd2d800b3c0 comp rx=0 tx=0).stop 2024-06-04T05:15:21.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:21.229+0000 7fd2eab89700 1 -- 172.21.15.70:0/1967998613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2e4105f70 msgr2=0x7fd2e41a07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:21.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:21.229+0000 7fd2eab89700 1 --2- 172.21.15.70:0/1967998613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2e4105f70 0x7fd2e41a07d0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0000c00 tx=0x7fd2d00047b0 comp rx=0 tx=0).stop 2024-06-04T05:15:21.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:21.229+0000 7fd2eab89700 1 -- 172.21.15.70:0/1967998613 shutdown_connections 2024-06-04T05:15:21.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:21.229+0000 7fd2eab89700 1 --2- 172.21.15.70:0/1967998613 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd2d40b29f0 0x7fd2d40b4eb0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:21.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:21.229+0000 7fd2eab89700 1 --2- 172.21.15.70:0/1967998613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e4108460 0x7fd2e41a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:21.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:21.229+0000 7fd2eab89700 1 --2- 172.21.15.70:0/1967998613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2e406c1b0 0x7fd2e41a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:21.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:21.229+0000 7fd2eab89700 1 --2- 172.21.15.70:0/1967998613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2e4105f70 0x7fd2e41a07d0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:21.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:21.229+0000 7fd2eab89700 1 -- 172.21.15.70:0/1967998613 >> 172.21.15.70:0/1967998613 conn(0x7fd2e40793f0 msgr2=0x7fd2e4102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:21.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:21.229+0000 7fd2eab89700 1 -- 172.21.15.70:0/1967998613 shutdown_connections 2024-06-04T05:15:21.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:21.229+0000 7fd2eab89700 1 -- 172.21.15.70:0/1967998613 wait complete. 2024-06-04T05:15:21.233 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 55 2024-06-04T05:15:21.446 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:21 smithi070 ceph-mon[32631]: pgmap v570: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.3 MiB/s wr, 295 op/s 2024-06-04T05:15:21.446 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:21 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1967998613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:15:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:21 smithi112 ceph-mon[36453]: pgmap v570: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.3 MiB/s wr, 295 op/s 2024-06-04T05:15:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:21 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1967998613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:15:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:21 smithi098 ceph-mon[35569]: pgmap v570: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.3 MiB/s wr, 295 op/s 2024-06-04T05:15:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:21 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1967998613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:15:21.816 INFO:tasks.workunit.client.0.smithi070.stdout: 10 36180 7.97 MB/sec execute 133 sec latency 6530.489 ms 2024-06-04T05:15:22.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.304+0000 7f09479b7700 1 -- 172.21.15.70:0/2800469284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09401037f0 msgr2=0x7f0940103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:22.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.304+0000 7f09479b7700 1 --2- 172.21.15.70:0/2800469284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09401037f0 0x7f0940103bd0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f093c0099c0 tx=0x7f093c009cd0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.304+0000 7f09479b7700 1 -- 172.21.15.70:0/2800469284 shutdown_connections 2024-06-04T05:15:22.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.304+0000 7f09479b7700 1 --2- 172.21.15.70:0/2800469284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09401088f0 0x7f0940110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.304+0000 7f09479b7700 1 --2- 172.21.15.70:0/2800469284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09401037f0 0x7f0940103bd0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.304+0000 7f09479b7700 1 --2- 172.21.15.70:0/2800469284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09401041a0 0x7f09401081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.305+0000 7f09479b7700 1 -- 172.21.15.70:0/2800469284 >> 172.21.15.70:0/2800469284 conn(0x7f094007a6a0 msgr2=0x7f094007aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:22.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.305+0000 7f09479b7700 1 -- 172.21.15.70:0/2800469284 shutdown_connections 2024-06-04T05:15:22.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.305+0000 7f09479b7700 1 -- 172.21.15.70:0/2800469284 wait complete. 2024-06-04T05:15:22.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.305+0000 7f09479b7700 1 Processor -- start 2024-06-04T05:15:22.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.305+0000 7f09479b7700 1 -- start start 2024-06-04T05:15:22.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.305+0000 7f09479b7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09401037f0 0x7f094019f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.305+0000 7f09479b7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09401041a0 0x7f094019fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.305+0000 7f09479b7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09401088f0 0x7f09401999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.305+0000 7f09479b7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f09401135e0 con 0x7f09401041a0 2024-06-04T05:15:22.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f09479b7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0940113460 con 0x7f09401088f0 2024-06-04T05:15:22.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f09479b7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0940113760 con 0x7f09401037f0 2024-06-04T05:15:22.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f0944f52700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09401041a0 0x7f094019fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:22.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f0944f52700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09401041a0 0x7f094019fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35726/0 (socket says 172.21.15.70:35726) 2024-06-04T05:15:22.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f0944f52700 1 -- 172.21.15.70:0/2266968565 learned_addr learned my addr 172.21.15.70:0/2266968565 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:22.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f0945753700 1 --2- 172.21.15.70:0/2266968565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09401037f0 0x7f094019f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:22.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f0945f54700 1 --2- 172.21.15.70:0/2266968565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09401088f0 0x7f09401999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:22.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f0944f52700 1 -- 172.21.15.70:0/2266968565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09401037f0 msgr2=0x7f094019f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:22.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f0944f52700 1 --2- 172.21.15.70:0/2266968565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09401037f0 0x7f094019f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f0944f52700 1 -- 172.21.15.70:0/2266968565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09401088f0 msgr2=0x7f09401999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:22.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f0944f52700 1 --2- 172.21.15.70:0/2266968565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09401088f0 0x7f09401999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f0944f52700 1 -- 172.21.15.70:0/2266968565 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f094019a100 con 0x7f09401041a0 2024-06-04T05:15:22.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f0945753700 1 --2- 172.21.15.70:0/2266968565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09401037f0 0x7f094019f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:15:22.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f0945f54700 1 --2- 172.21.15.70:0/2266968565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09401088f0 0x7f09401999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:15:22.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f0944f52700 1 --2- 172.21.15.70:0/2266968565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09401041a0 0x7f094019fd20 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f093000b530 tx=0x7f093000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:22.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f09367fc700 1 -- 172.21.15.70:0/2266968565 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0930015020 con 0x7f09401041a0 2024-06-04T05:15:22.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f09367fc700 1 -- 172.21.15.70:0/2266968565 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0930011cc0 con 0x7f09401041a0 2024-06-04T05:15:22.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f09479b7700 1 -- 172.21.15.70:0/2266968565 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f094019a3f0 con 0x7f09401041a0 2024-06-04T05:15:22.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f09367fc700 1 -- 172.21.15.70:0/2266968565 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0930010d30 con 0x7f09401041a0 2024-06-04T05:15:22.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.306+0000 7f09479b7700 1 -- 172.21.15.70:0/2266968565 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09401a66b0 con 0x7f09401041a0 2024-06-04T05:15:22.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.309+0000 7f09367fc700 1 -- 172.21.15.70:0/2266968565 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f0930018020 con 0x7f09401041a0 2024-06-04T05:15:22.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.310+0000 7f09367fc700 1 --2- 172.21.15.70:0/2266968565 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f092c0b29f0 0x7f092c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.310+0000 7f0945753700 1 --2- 172.21.15.70:0/2266968565 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f092c0b29f0 0x7f092c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:22.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.310+0000 7f09367fc700 1 -- 172.21.15.70:0/2266968565 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f09300e1dd0 con 0x7f09401041a0 2024-06-04T05:15:22.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.310+0000 7f095224cb80 1 Processor -- start 2024-06-04T05:15:22.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.310+0000 7f095224cb80 1 -- start start 2024-06-04T05:15:22.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.310+0000 7f0945753700 1 --2- 172.21.15.70:0/2266968565 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f092c0b29f0 0x7f092c0b4eb0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f093c006010 tx=0x7f093c005e80 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:22.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f095224cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc105b320 0x559dc105b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f095224cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 0x559dc1055d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f095224cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc1056260 0x559dc1062080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f095224cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559dc0fb3d70 con 0x559dc1056260 2024-06-04T05:15:22.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f095224cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559dc0fa3910 con 0x559dc105b320 2024-06-04T05:15:22.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f095224cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559dc0fa7950 con 0x559dc10558a0 2024-06-04T05:15:22.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f0945f54700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc1056260 0x559dc1062080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:22.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f0945f54700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc1056260 0x559dc1062080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35738/0 (socket says 172.21.15.70:35738) 2024-06-04T05:15:22.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f0945f54700 1 -- 172.21.15.70:0/2928575935 learned_addr learned my addr 172.21.15.70:0/2928575935 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:22.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f0945753700 1 --2- 172.21.15.70:0/2928575935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc105b320 0x559dc105b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:22.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f0944f52700 1 --2- 172.21.15.70:0/2928575935 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 0x559dc1055d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:22.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f0945f54700 1 -- 172.21.15.70:0/2928575935 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 msgr2=0x559dc1055d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:22.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f0945f54700 1 --2- 172.21.15.70:0/2928575935 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 0x559dc1055d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f0945f54700 1 -- 172.21.15.70:0/2928575935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc105b320 msgr2=0x559dc105b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:22.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f0945f54700 1 --2- 172.21.15.70:0/2928575935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc105b320 0x559dc105b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f0945f54700 1 -- 172.21.15.70:0/2928575935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dc0f4cfb0 con 0x559dc1056260 2024-06-04T05:15:22.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f0945f54700 1 --2- 172.21.15.70:0/2928575935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc1056260 0x559dc1062080 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f09380084f0 tx=0x7f0938008800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e34170e32db285dc server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:22.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f092b7fe700 1 -- 172.21.15.70:0/2928575935 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938018070 con 0x559dc1056260 2024-06-04T05:15:22.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f092b7fe700 1 -- 172.21.15.70:0/2928575935 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0938011ba0 con 0x559dc1056260 2024-06-04T05:15:22.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.311+0000 7f092b7fe700 1 -- 172.21.15.70:0/2928575935 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938013b10 con 0x559dc1056260 2024-06-04T05:15:22.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.312+0000 7f095224cb80 1 -- 172.21.15.70:0/2928575935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc1056260 msgr2=0x559dc1062080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:22.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.312+0000 7f095224cb80 1 --2- 172.21.15.70:0/2928575935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc1056260 0x559dc1062080 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f09380084f0 tx=0x7f0938008800 comp rx=0 tx=0).stop 2024-06-04T05:15:22.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.312+0000 7f095224cb80 1 -- 172.21.15.70:0/2928575935 shutdown_connections 2024-06-04T05:15:22.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.312+0000 7f095224cb80 1 --2- 172.21.15.70:0/2928575935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc1056260 0x559dc1062080 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.312+0000 7f095224cb80 1 --2- 172.21.15.70:0/2928575935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc105b320 0x559dc105b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.312+0000 7f095224cb80 1 --2- 172.21.15.70:0/2928575935 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 0x559dc1055d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.312+0000 7f095224cb80 1 -- 172.21.15.70:0/2928575935 >> 172.21.15.70:0/2928575935 conn(0x559dc0f81240 msgr2=0x559dc10503e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:22.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.312+0000 7f095224cb80 1 -- 172.21.15.70:0/2928575935 shutdown_connections 2024-06-04T05:15:22.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.312+0000 7f095224cb80 1 -- 172.21.15.70:0/2928575935 wait complete. 2024-06-04T05:15:22.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.313+0000 7f095224cb80 1 -- start start 2024-06-04T05:15:22.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.313+0000 7f095224cb80 1 Processor -- start 2024-06-04T05:15:22.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.313+0000 7f095224cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 0x559dc1061d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.313+0000 7f095224cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc1056260 0x559dc105f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.313+0000 7f095224cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc105b320 0x559dc105f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.313+0000 7f095224cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559dc0ea9fa0 con 0x559dc1056260 2024-06-04T05:15:22.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.313+0000 7f095224cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559dc0f0bda0 con 0x559dc105b320 2024-06-04T05:15:22.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.313+0000 7f095224cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559dc10626e0 con 0x559dc10558a0 2024-06-04T05:15:22.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.313+0000 7f095224cb80 10 client.0 _ll_register_callbacks cb 0x559dc0f3e950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:22.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.313+0000 7f0944f52700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc1056260 0x559dc105f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:22.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.313+0000 7f0944f52700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc1056260 0x559dc105f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35742/0 (socket says 172.21.15.70:35742) 2024-06-04T05:15:22.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.313+0000 7f0944f52700 1 -- 172.21.15.70:0/3064641805 learned_addr learned my addr 172.21.15.70:0/3064641805 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:22.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.313+0000 7f0945f54700 1 --2- 172.21.15.70:0/3064641805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc105b320 0x559dc105f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:22.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f0945753700 1 --2- 172.21.15.70:0/3064641805 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 0x559dc1061d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:22.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f0944f52700 1 -- 172.21.15.70:0/3064641805 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 msgr2=0x559dc1061d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:22.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f0944f52700 1 --2- 172.21.15.70:0/3064641805 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 0x559dc1061d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f0944f52700 1 -- 172.21.15.70:0/3064641805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc105b320 msgr2=0x559dc105f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:22.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f0944f52700 1 --2- 172.21.15.70:0/3064641805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc105b320 0x559dc105f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f0944f52700 1 -- 172.21.15.70:0/3064641805 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dc0ef38d0 con 0x559dc1056260 2024-06-04T05:15:22.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f0945f54700 1 --2- 172.21.15.70:0/3064641805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc105b320 0x559dc105f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:15:22.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f0944f52700 1 --2- 172.21.15.70:0/3064641805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc1056260 0x559dc105f030 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f09300a5210 tx=0x7f0930010f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ea19918ec221be server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:22.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f0928ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:15:22.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f0928ff9700 1 -- 172.21.15.70:0/3064641805 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09300a9190 con 0x559dc1056260 2024-06-04T05:15:22.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f0928ff9700 1 -- 172.21.15.70:0/3064641805 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f09300b1b60 con 0x559dc1056260 2024-06-04T05:15:22.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f095224cb80 1 -- 172.21.15.70:0/3064641805 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0938011ba0 con 0x559dc1056260 2024-06-04T05:15:22.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f0928ff9700 1 -- 172.21.15.70:0/3064641805 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f09300ad540 con 0x559dc1056260 2024-06-04T05:15:22.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f095224cb80 10 client.20157 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:22.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f095224cb80 1 -- 172.21.15.70:0/3064641805 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x559dc0f5d760 con 0x559dc1056260 2024-06-04T05:15:22.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f0928ff9700 1 -- 172.21.15.70:0/3064641805 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0938011ba0 con 0x559dc1056260 2024-06-04T05:15:22.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f0928ff9700 1 -- 172.21.15.70:0/3064641805 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f09300a8580 con 0x559dc1056260 2024-06-04T05:15:22.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f095224cb80 10 client.20157 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:22.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.314+0000 7f095224cb80 10 client.20157 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:15:22.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.315+0000 7f095224cb80 20 client.20157 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:22.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.315+0000 7f095224cb80 1 --2- 172.21.15.70:0/3064641805 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x559dc106f190 0x559dc10715d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.315+0000 7f095224cb80 4 client.20157 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:15:22.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.315+0000 7f095224cb80 1 -- 172.21.15.70:0/3064641805 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559dc0fb5cc0 con 0x559dc106f190 2024-06-04T05:15:22.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.315+0000 7f0945753700 1 --2- 172.21.15.70:0/3064641805 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x559dc106f190 0x559dc10715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:22.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.315+0000 7f0945753700 1 --2- 172.21.15.70:0/3064641805 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x559dc106f190 0x559dc10715d0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=75378944cd4dbd6d server_cookie=7bc13276a2ad4296 in_seq=0 out_seq=0 2024-06-04T05:15:22.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.315+0000 7f0928ff9700 10 client.20157 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:15:22.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.316+0000 7f0928ff9700 1 -- 172.21.15.70:0/3064641805 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559dc0fb5cc0 con 0x559dc106f190 2024-06-04T05:15:22.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.316+0000 7f0928ff9700 10 client.20157 handle_command_reply: tid=0 2024-06-04T05:15:22.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.316+0000 7f095224cb80 1 client.20157 shutdown 2024-06-04T05:15:22.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.316+0000 7f095224cb80 1 -- 172.21.15.70:0/3064641805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc1056260 msgr2=0x559dc105f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:22.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.316+0000 7f095224cb80 1 --2- 172.21.15.70:0/3064641805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc1056260 0x559dc105f030 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f09300a5210 tx=0x7f0930010f80 comp rx=0 tx=0).stop 2024-06-04T05:15:22.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.316+0000 7f095224cb80 1 -- 172.21.15.70:0/3064641805 shutdown_connections 2024-06-04T05:15:22.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.316+0000 7f095224cb80 1 --2- 172.21.15.70:0/3064641805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc105b320 0x559dc105f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.316+0000 7f095224cb80 1 --2- 172.21.15.70:0/3064641805 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x559dc106f190 0x559dc10715d0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.316+0000 7f0928ff9700 0 client.20157 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:15:22.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.316+0000 7f095224cb80 1 --2- 172.21.15.70:0/3064641805 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 0x559dc1061d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.316+0000 7f095224cb80 1 --2- 172.21.15.70:0/3064641805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc1056260 0x559dc105f030 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.316+0000 7f095224cb80 1 -- 172.21.15.70:0/3064641805 >> 172.21.15.70:0/3064641805 conn(0x559dc0f81240 msgr2=0x559dc104faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:22.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.316+0000 7f095224cb80 1 -- 172.21.15.70:0/3064641805 shutdown_connections 2024-06-04T05:15:22.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.316+0000 7f095224cb80 1 -- 172.21.15.70:0/3064641805 wait complete. 2024-06-04T05:15:22.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.316+0000 7f095224cb80 20 client.20157 trim_cache size 0 max 16384 2024-06-04T05:15:22.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.331+0000 7f095224cb80 1 Processor -- start 2024-06-04T05:15:22.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.331+0000 7f095224cb80 1 -- start start 2024-06-04T05:15:22.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.331+0000 7f095224cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 0x559dc1050270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f095224cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc1056260 0x559dc1060f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f095224cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc105b320 0x559dc105e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f095224cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f09401adea0 con 0x559dc105b320 2024-06-04T05:15:22.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f095224cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09401afa40 con 0x559dc1056260 2024-06-04T05:15:22.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f095224cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f09401ae5d0 con 0x559dc10558a0 2024-06-04T05:15:22.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f0945f54700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc105b320 0x559dc105e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:22.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f0945f54700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc105b320 0x559dc105e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35744/0 (socket says 172.21.15.70:35744) 2024-06-04T05:15:22.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f0945f54700 1 -- 172.21.15.70:0/2984375288 learned_addr learned my addr 172.21.15.70:0/2984375288 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:22.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f0945753700 1 --2- 172.21.15.70:0/2984375288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 0x559dc1050270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:22.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f0945f54700 1 -- 172.21.15.70:0/2984375288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 msgr2=0x559dc1050270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:22.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f0945f54700 1 --2- 172.21.15.70:0/2984375288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 0x559dc1050270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f0945f54700 1 -- 172.21.15.70:0/2984375288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc1056260 msgr2=0x559dc1060f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:22.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f0945f54700 1 --2- 172.21.15.70:0/2984375288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc1056260 0x559dc1060f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f0945f54700 1 -- 172.21.15.70:0/2984375288 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dc10620a0 con 0x559dc105b320 2024-06-04T05:15:22.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f0945f54700 1 --2- 172.21.15.70:0/2984375288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc105b320 0x559dc105e8e0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f09380084f0 tx=0x7f09380076b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7baed6d23899e8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:22.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f092affd700 1 -- 172.21.15.70:0/2984375288 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938022070 con 0x559dc105b320 2024-06-04T05:15:22.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f092affd700 1 -- 172.21.15.70:0/2984375288 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0938013bb0 con 0x559dc105b320 2024-06-04T05:15:22.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.332+0000 7f092affd700 1 -- 172.21.15.70:0/2984375288 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938018040 con 0x559dc105b320 2024-06-04T05:15:22.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.333+0000 7f095224cb80 1 -- 172.21.15.70:0/2984375288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc105b320 msgr2=0x559dc105e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:22.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.333+0000 7f095224cb80 1 --2- 172.21.15.70:0/2984375288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc105b320 0x559dc105e8e0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f09380084f0 tx=0x7f09380076b0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.333+0000 7f095224cb80 1 -- 172.21.15.70:0/2984375288 shutdown_connections 2024-06-04T05:15:22.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.333+0000 7f095224cb80 1 --2- 172.21.15.70:0/2984375288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc105b320 0x559dc105e8e0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.333+0000 7f095224cb80 1 --2- 172.21.15.70:0/2984375288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 0x559dc1050270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.333+0000 7f095224cb80 1 --2- 172.21.15.70:0/2984375288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc1056260 0x559dc1060f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.333+0000 7f095224cb80 1 -- 172.21.15.70:0/2984375288 >> 172.21.15.70:0/2984375288 conn(0x559dc0f81240 msgr2=0x559dc1060860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:22.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.333+0000 7f095224cb80 1 -- 172.21.15.70:0/2984375288 shutdown_connections 2024-06-04T05:15:22.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.333+0000 7f095224cb80 1 -- 172.21.15.70:0/2984375288 wait complete. 2024-06-04T05:15:22.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.333+0000 7f095224cb80 1 -- start start 2024-06-04T05:15:22.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.333+0000 7f095224cb80 1 Processor -- start 2024-06-04T05:15:22.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f095224cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 0x559dc1056cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f095224cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc1056260 0x559dc1057210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f095224cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc105b320 0x559dc1057750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f095224cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f09401af370 con 0x559dc105b320 2024-06-04T05:15:22.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f095224cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09401aeca0 con 0x559dc1056260 2024-06-04T05:15:22.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f095224cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f09401abe60 con 0x559dc10558a0 2024-06-04T05:15:22.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f095224cb80 10 client.0 _ll_register_callbacks cb 0x559dc0f3e950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:22.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f0945f54700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc105b320 0x559dc1057750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:22.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f0945753700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 0x559dc1056cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:22.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f0945f54700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc105b320 0x559dc1057750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35758/0 (socket says 172.21.15.70:35758) 2024-06-04T05:15:22.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f0945f54700 1 -- 172.21.15.70:0/1087707227 learned_addr learned my addr 172.21.15.70:0/1087707227 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:22.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f0944f52700 1 --2- 172.21.15.70:0/1087707227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc1056260 0x559dc1057210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:22.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f0945f54700 1 -- 172.21.15.70:0/1087707227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 msgr2=0x559dc1056cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:22.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f0945f54700 1 --2- 172.21.15.70:0/1087707227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 0x559dc1056cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f0945f54700 1 -- 172.21.15.70:0/1087707227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc1056260 msgr2=0x559dc1057210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:22.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f0945f54700 1 --2- 172.21.15.70:0/1087707227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc1056260 0x559dc1057210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f0945f54700 1 -- 172.21.15.70:0/1087707227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dc10dcdf0 con 0x559dc105b320 2024-06-04T05:15:22.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f0944f52700 1 --2- 172.21.15.70:0/1087707227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc1056260 0x559dc1057210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:15:22.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f0945f54700 1 --2- 172.21.15.70:0/1087707227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc105b320 0x559dc1057750 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f0938008430 tx=0x7f09380136b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74de3dbb820ef58 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:22.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.334+0000 7f0928ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:15:22.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.335+0000 7f0928ff9700 1 -- 172.21.15.70:0/1087707227 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938022070 con 0x559dc105b320 2024-06-04T05:15:22.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.335+0000 7f0928ff9700 1 -- 172.21.15.70:0/1087707227 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f0938011710 con 0x559dc105b320 2024-06-04T05:15:22.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.335+0000 7f095224cb80 1 -- 172.21.15.70:0/1087707227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0938013bb0 con 0x559dc105b320 2024-06-04T05:15:22.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.335+0000 7f0928ff9700 1 -- 172.21.15.70:0/1087707227 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f0938018040 con 0x559dc105b320 2024-06-04T05:15:22.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.335+0000 7f095224cb80 10 client.20163 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:22.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.335+0000 7f095224cb80 1 -- 172.21.15.70:0/1087707227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x559dc10de0d0 con 0x559dc105b320 2024-06-04T05:15:22.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.335+0000 7f0928ff9700 1 -- 172.21.15.70:0/1087707227 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0938013bb0 con 0x559dc105b320 2024-06-04T05:15:22.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.335+0000 7f0928ff9700 1 -- 172.21.15.70:0/1087707227 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f09380277f0 con 0x559dc105b320 2024-06-04T05:15:22.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.335+0000 7f095224cb80 10 client.20163 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:22.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.335+0000 7f095224cb80 10 client.20163 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:15:22.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.335+0000 7f095224cb80 20 client.20163 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:22.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.335+0000 7f095224cb80 1 --2- 172.21.15.70:0/1087707227 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x559dc106f190 0x559dc10e04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:22.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.335+0000 7f095224cb80 4 client.20163 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:15:22.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.335+0000 7f095224cb80 1 -- 172.21.15.70:0/1087707227 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559dc0faf4a0 con 0x559dc106f190 2024-06-04T05:15:22.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.335+0000 7f0945753700 1 --2- 172.21.15.70:0/1087707227 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x559dc106f190 0x559dc10e04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:22.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.336+0000 7f0928ff9700 10 client.20163 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:15:22.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.336+0000 7f0945753700 1 --2- 172.21.15.70:0/1087707227 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x559dc106f190 0x559dc10e04e0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6276472c7c9e059b server_cookie=b70a66273b4a86dd in_seq=0 out_seq=0 2024-06-04T05:15:22.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.336+0000 7f0928ff9700 1 -- 172.21.15.70:0/1087707227 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559dc0faf4a0 con 0x559dc106f190 2024-06-04T05:15:22.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.336+0000 7f0928ff9700 10 client.20163 handle_command_reply: tid=0 2024-06-04T05:15:22.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.336+0000 7f095224cb80 1 client.20163 shutdown 2024-06-04T05:15:22.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.336+0000 7f095224cb80 1 -- 172.21.15.70:0/1087707227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc105b320 msgr2=0x559dc1057750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:22.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.336+0000 7f095224cb80 1 --2- 172.21.15.70:0/1087707227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc105b320 0x559dc1057750 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f0938008430 tx=0x7f09380136b0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.336+0000 7f095224cb80 1 -- 172.21.15.70:0/1087707227 shutdown_connections 2024-06-04T05:15:22.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.336+0000 7f095224cb80 1 --2- 172.21.15.70:0/1087707227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559dc105b320 0x559dc1057750 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.336+0000 7f095224cb80 1 --2- 172.21.15.70:0/1087707227 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x559dc106f190 0x559dc10e04e0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.336+0000 7f095224cb80 1 --2- 172.21.15.70:0/1087707227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559dc10558a0 0x559dc1056cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.336+0000 7f095224cb80 1 --2- 172.21.15.70:0/1087707227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dc1056260 0x559dc1057210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.336+0000 7f0928ff9700 0 client.20163 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:15:22.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.336+0000 7f095224cb80 1 -- 172.21.15.70:0/1087707227 >> 172.21.15.70:0/1087707227 conn(0x559dc0f81240 msgr2=0x559dc10610f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:22.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.336+0000 7f095224cb80 1 -- 172.21.15.70:0/1087707227 shutdown_connections 2024-06-04T05:15:22.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.336+0000 7f095224cb80 1 -- 172.21.15.70:0/1087707227 wait complete. 2024-06-04T05:15:22.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.337+0000 7f095224cb80 20 client.20163 trim_cache size 0 max 16384 2024-06-04T05:15:22.344 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:15:22.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.337+0000 7f09479b7700 1 -- 172.21.15.70:0/2266968565 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f092c0b29f0 msgr2=0x7f092c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:22.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.337+0000 7f09479b7700 1 --2- 172.21.15.70:0/2266968565 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f092c0b29f0 0x7f092c0b4eb0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f093c006010 tx=0x7f093c005e80 comp rx=0 tx=0).stop 2024-06-04T05:15:22.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.337+0000 7f09479b7700 1 -- 172.21.15.70:0/2266968565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09401041a0 msgr2=0x7f094019fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:22.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.337+0000 7f09479b7700 1 --2- 172.21.15.70:0/2266968565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09401041a0 0x7f094019fd20 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f093000b530 tx=0x7f093000b8f0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.337+0000 7f09479b7700 1 -- 172.21.15.70:0/2266968565 shutdown_connections 2024-06-04T05:15:22.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.337+0000 7f09479b7700 1 --2- 172.21.15.70:0/2266968565 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f092c0b29f0 0x7f092c0b4eb0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.337+0000 7f09479b7700 1 --2- 172.21.15.70:0/2266968565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09401088f0 0x7f09401999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.337+0000 7f09479b7700 1 --2- 172.21.15.70:0/2266968565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09401037f0 0x7f094019f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.337+0000 7f09479b7700 1 --2- 172.21.15.70:0/2266968565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09401041a0 0x7f094019fd20 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:22.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.337+0000 7f09479b7700 1 -- 172.21.15.70:0/2266968565 >> 172.21.15.70:0/2266968565 conn(0x7f094007a6a0 msgr2=0x7f0940109a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:22.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.337+0000 7f09479b7700 1 -- 172.21.15.70:0/2266968565 shutdown_connections 2024-06-04T05:15:22.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:22.337+0000 7f09479b7700 1 -- 172.21.15.70:0/2266968565 wait complete. 2024-06-04T05:15:22.816 INFO:tasks.workunit.client.0.smithi070.stdout: 10 36180 7.92 MB/sec execute 134 sec latency 7530.573 ms 2024-06-04T05:15:22.967 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id 77470b0 --release --await' 2024-06-04T05:15:22.967 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id 77470b0 --release --await 2024-06-04T05:15:23.471 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:23 smithi070 ceph-mon[32631]: pgmap v571: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 310 op/s 2024-06-04T05:15:23.472 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:15:23.499 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:15:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:23 smithi112 ceph-mon[36453]: pgmap v571: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 310 op/s 2024-06-04T05:15:23.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:23 smithi098 ceph-mon[35569]: pgmap v571: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 310 op/s 2024-06-04T05:15:23.816 INFO:tasks.workunit.client.0.smithi070.stdout: 10 36180 7.86 MB/sec execute 135 sec latency 8530.668 ms 2024-06-04T05:15:24.194 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:15:24.816 INFO:tasks.workunit.client.0.smithi070.stdout: 10 36180 7.80 MB/sec execute 136 sec latency 9530.741 ms 2024-06-04T05:15:25.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:25 smithi070 ceph-mon[32631]: pgmap v572: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 2.5 MiB/s wr, 173 op/s 2024-06-04T05:15:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:25 smithi112 ceph-mon[36453]: pgmap v572: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 2.5 MiB/s wr, 173 op/s 2024-06-04T05:15:25.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:25 smithi098 ceph-mon[35569]: pgmap v572: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 2.5 MiB/s wr, 173 op/s 2024-06-04T05:15:25.816 INFO:tasks.workunit.client.0.smithi070.stdout: 10 36180 7.74 MB/sec execute 137 sec latency 10530.799 ms 2024-06-04T05:15:26.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.314+0000 7fad5fc19700 1 -- 172.21.15.70:0/3008671764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad58108920 msgr2=0x7fad581104a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:26.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.314+0000 7fad5fc19700 1 --2- 172.21.15.70:0/3008671764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad58108920 0x7fad581104a0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fad480099c0 tx=0x7fad48009cd0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.314+0000 7fad5fc19700 1 -- 172.21.15.70:0/3008671764 shutdown_connections 2024-06-04T05:15:26.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.314+0000 7fad5fc19700 1 --2- 172.21.15.70:0/3008671764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad58108920 0x7fad581104a0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.314+0000 7fad5fc19700 1 --2- 172.21.15.70:0/3008671764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad58103850 0x7fad58103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.314+0000 7fad5fc19700 1 --2- 172.21.15.70:0/3008671764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad58104200 0x7fad581081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.314+0000 7fad5fc19700 1 -- 172.21.15.70:0/3008671764 >> 172.21.15.70:0/3008671764 conn(0x7fad5807a5d0 msgr2=0x7fad5807a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:26.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.314+0000 7fad5fc19700 1 -- 172.21.15.70:0/3008671764 shutdown_connections 2024-06-04T05:15:26.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.314+0000 7fad5fc19700 1 -- 172.21.15.70:0/3008671764 wait complete. 2024-06-04T05:15:26.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.315+0000 7fad5fc19700 1 Processor -- start 2024-06-04T05:15:26.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.315+0000 7fad5fc19700 1 -- start start 2024-06-04T05:15:26.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.315+0000 7fad5fc19700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad58103850 0x7fad5819f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.315+0000 7fad5fc19700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad58104200 0x7fad5819fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.315+0000 7fad5fc19700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad58108920 0x7fad581998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.315+0000 7fad5fc19700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fad581135c0 con 0x7fad58103850 2024-06-04T05:15:26.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.315+0000 7fad5fc19700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad58113440 con 0x7fad58108920 2024-06-04T05:15:26.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.315+0000 7fad5fc19700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fad58113740 con 0x7fad58104200 2024-06-04T05:15:26.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.315+0000 7fad5d9b5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad58103850 0x7fad5819f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.315+0000 7fad5d9b5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad58103850 0x7fad5819f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34476/0 (socket says 172.21.15.70:34476) 2024-06-04T05:15:26.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.315+0000 7fad5d9b5700 1 -- 172.21.15.70:0/3278421163 learned_addr learned my addr 172.21.15.70:0/3278421163 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:26.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.316+0000 7fad5d1b4700 1 --2- 172.21.15.70:0/3278421163 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad58104200 0x7fad5819fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.316+0000 7fad5e1b6700 1 --2- 172.21.15.70:0/3278421163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad58108920 0x7fad581998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.316+0000 7fad5d9b5700 1 -- 172.21.15.70:0/3278421163 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad58104200 msgr2=0x7fad5819fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:26.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.316+0000 7fad5d9b5700 1 --2- 172.21.15.70:0/3278421163 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad58104200 0x7fad5819fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.316+0000 7fad5d9b5700 1 -- 172.21.15.70:0/3278421163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad58108920 msgr2=0x7fad581998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:26.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.316+0000 7fad5d9b5700 1 --2- 172.21.15.70:0/3278421163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad58108920 0x7fad581998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.316+0000 7fad5d9b5700 1 -- 172.21.15.70:0/3278421163 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad5819a020 con 0x7fad58103850 2024-06-04T05:15:26.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.316+0000 7fad5e1b6700 1 --2- 172.21.15.70:0/3278421163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad58108920 0x7fad581998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:15:26.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.316+0000 7fad5d9b5700 1 --2- 172.21.15.70:0/3278421163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad58103850 0x7fad5819f6b0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fad5000ef10 tx=0x7fad5000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:26.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.316+0000 7fad56ffd700 1 -- 172.21.15.70:0/3278421163 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5000ca70 con 0x7fad58103850 2024-06-04T05:15:26.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.316+0000 7fad56ffd700 1 -- 172.21.15.70:0/3278421163 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad50019070 con 0x7fad58103850 2024-06-04T05:15:26.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.316+0000 7fad56ffd700 1 -- 172.21.15.70:0/3278421163 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad50014480 con 0x7fad58103850 2024-06-04T05:15:26.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.316+0000 7fad5fc19700 1 -- 172.21.15.70:0/3278421163 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad5819a2b0 con 0x7fad58103850 2024-06-04T05:15:26.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.316+0000 7fad5fc19700 1 -- 172.21.15.70:0/3278421163 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad581a6580 con 0x7fad58103850 2024-06-04T05:15:26.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.318+0000 7fad56ffd700 1 -- 172.21.15.70:0/3278421163 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fad50012050 con 0x7fad58103850 2024-06-04T05:15:26.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.319+0000 7fad56ffd700 1 --2- 172.21.15.70:0/3278421163 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fad400b2a70 0x7fad400b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.319+0000 7fad5d1b4700 1 --2- 172.21.15.70:0/3278421163 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fad400b2a70 0x7fad400b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.319+0000 7fad56ffd700 1 -- 172.21.15.70:0/3278421163 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7fad500e2660 con 0x7fad58103850 2024-06-04T05:15:26.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.320+0000 7fad5d1b4700 1 --2- 172.21.15.70:0/3278421163 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fad400b2a70 0x7fad400b4f30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fad581030c0 tx=0x7fad4c00a300 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:26.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.320+0000 7fad6a4aeb80 1 Processor -- start 2024-06-04T05:15:26.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.320+0000 7fad6a4aeb80 1 -- start start 2024-06-04T05:15:26.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.320+0000 7fad6a4aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299f2b0 0x56011299f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.320+0000 7fad6a4aeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5601129998e0 0x560112999d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.320+0000 7fad6a4aeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56011299a2a0 0x5601129a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.320+0000 7fad6a4aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5601128f8d90 con 0x56011299f2b0 2024-06-04T05:15:26.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.320+0000 7fad6a4aeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5601128e8930 con 0x56011299a2a0 2024-06-04T05:15:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.320+0000 7fad6a4aeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5601128ec970 con 0x5601129998e0 2024-06-04T05:15:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.320+0000 7fad5d9b5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299f2b0 0x56011299f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.320+0000 7fad5d9b5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299f2b0 0x56011299f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34490/0 (socket says 172.21.15.70:34490) 2024-06-04T05:15:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.320+0000 7fad5d9b5700 1 -- 172.21.15.70:0/2878832705 learned_addr learned my addr 172.21.15.70:0/2878832705 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.321+0000 7fad5d9b5700 1 -- 172.21.15.70:0/2878832705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5601129998e0 msgr2=0x560112999d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.321+0000 7fad5d9b5700 1 --2- 172.21.15.70:0/2878832705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5601129998e0 0x560112999d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.321+0000 7fad5d9b5700 1 -- 172.21.15.70:0/2878832705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56011299a2a0 msgr2=0x5601129a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.321+0000 7fad5d9b5700 1 --2- 172.21.15.70:0/2878832705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56011299a2a0 0x5601129a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.321+0000 7fad5d9b5700 1 -- 172.21.15.70:0/2878832705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601128389d0 con 0x56011299f2b0 2024-06-04T05:15:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.321+0000 7fad5d9b5700 1 --2- 172.21.15.70:0/2878832705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299f2b0 0x56011299f690 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fad500a98a0 tx=0x7fad500a9080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e37691deeceec145 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.321+0000 7fad3ffff700 1 -- 172.21.15.70:0/2878832705 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad500b20b0 con 0x56011299f2b0 2024-06-04T05:15:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.321+0000 7fad3ffff700 1 -- 172.21.15.70:0/2878832705 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad500a9420 con 0x56011299f2b0 2024-06-04T05:15:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.321+0000 7fad3ffff700 1 -- 172.21.15.70:0/2878832705 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad500b2770 con 0x56011299f2b0 2024-06-04T05:15:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.322+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/2878832705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299f2b0 msgr2=0x56011299f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.322+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/2878832705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299f2b0 0x56011299f690 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fad500a98a0 tx=0x7fad500a9080 comp rx=0 tx=0).stop 2024-06-04T05:15:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.322+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/2878832705 shutdown_connections 2024-06-04T05:15:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.322+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/2878832705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56011299a2a0 0x5601129a60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.322+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/2878832705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299f2b0 0x56011299f690 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.322+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/2878832705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5601129998e0 0x560112999d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.322+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/2878832705 >> 172.21.15.70:0/2878832705 conn(0x5601128c7180 msgr2=0x5601129941f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.322+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/2878832705 shutdown_connections 2024-06-04T05:15:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.322+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/2878832705 wait complete. 2024-06-04T05:15:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.322+0000 7fad6a4aeb80 1 -- start start 2024-06-04T05:15:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.322+0000 7fad6a4aeb80 1 Processor -- start 2024-06-04T05:15:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad6a4aeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601129998e0 0x5601129a5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad6a4aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299a2a0 0x5601129a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad6a4aeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56011299f2b0 0x5601129a3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad6a4aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5601127ef0a0 con 0x56011299a2a0 2024-06-04T05:15:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad6a4aeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56011288a8c0 con 0x5601129998e0 2024-06-04T05:15:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad6a4aeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5601129a6680 con 0x56011299f2b0 2024-06-04T05:15:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad6a4aeb80 10 client.0 _ll_register_callbacks cb 0x560112883a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad5d1b4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299a2a0 0x5601129a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad5d1b4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299a2a0 0x5601129a31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34492/0 (socket says 172.21.15.70:34492) 2024-06-04T05:15:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad5d1b4700 1 -- 172.21.15.70:0/61524412 learned_addr learned my addr 172.21.15.70:0/61524412 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad5d9b5700 1 --2- 172.21.15.70:0/61524412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601129998e0 0x5601129a5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad5e1b6700 1 --2- 172.21.15.70:0/61524412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56011299f2b0 0x5601129a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad5d1b4700 1 -- 172.21.15.70:0/61524412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56011299f2b0 msgr2=0x5601129a3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad5d1b4700 1 --2- 172.21.15.70:0/61524412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56011299f2b0 0x5601129a3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad5d1b4700 1 -- 172.21.15.70:0/61524412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601129998e0 msgr2=0x5601129a5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad5d1b4700 1 --2- 172.21.15.70:0/61524412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601129998e0 0x5601129a5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad5d1b4700 1 -- 172.21.15.70:0/61524412 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560112887790 con 0x56011299a2a0 2024-06-04T05:15:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad5d1b4700 1 --2- 172.21.15.70:0/61524412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299a2a0 0x5601129a31d0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0109e0 tx=0x7fad4c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9125c91acad7533c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.323+0000 7fad3d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:15:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.324+0000 7fad3d7fa700 1 -- 172.21.15.70:0/61524412 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4c016070 con 0x56011299a2a0 2024-06-04T05:15:26.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.324+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/61524412 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad500a9420 con 0x56011299a2a0 2024-06-04T05:15:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.324+0000 7fad3d7fa700 1 -- 172.21.15.70:0/61524412 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7fad4c01b420 con 0x56011299a2a0 2024-06-04T05:15:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.324+0000 7fad3d7fa700 1 -- 172.21.15.70:0/61524412 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fad4c010e20 con 0x56011299a2a0 2024-06-04T05:15:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.324+0000 7fad6a4aeb80 10 client.20175 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.324+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/61524412 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5601129b0fc0 con 0x56011299a2a0 2024-06-04T05:15:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.324+0000 7fad3d7fa700 1 -- 172.21.15.70:0/61524412 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad500a9420 con 0x56011299a2a0 2024-06-04T05:15:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.324+0000 7fad3d7fa700 1 -- 172.21.15.70:0/61524412 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7fad4c0258f0 con 0x56011299a2a0 2024-06-04T05:15:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.324+0000 7fad6a4aeb80 10 client.20175 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.324+0000 7fad6a4aeb80 10 client.20175 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:15:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.324+0000 7fad6a4aeb80 20 client.20175 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.324+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/61524412 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5601129b35a0 0x5601129b5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.324+0000 7fad6a4aeb80 4 client.20175 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:15:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.324+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/61524412 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560112965ae0 con 0x5601129b35a0 2024-06-04T05:15:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.325+0000 7fad5d9b5700 1 --2- 172.21.15.70:0/61524412 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5601129b35a0 0x5601129b5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.325+0000 7fad3d7fa700 10 client.20175 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.325+0000 7fad5d9b5700 1 --2- 172.21.15.70:0/61524412 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5601129b35a0 0x5601129b5990 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce24721abcefef19 server_cookie=ccaf7a4ff653ad92 in_seq=0 out_seq=0 2024-06-04T05:15:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.326+0000 7fad3d7fa700 1 -- 172.21.15.70:0/61524412 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560112965ae0 con 0x5601129b35a0 2024-06-04T05:15:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.326+0000 7fad3d7fa700 10 client.20175 handle_command_reply: tid=0 2024-06-04T05:15:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.326+0000 7fad6a4aeb80 1 client.20175 shutdown 2024-06-04T05:15:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.326+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/61524412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299a2a0 msgr2=0x5601129a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.326+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/61524412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299a2a0 0x5601129a31d0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0109e0 tx=0x7fad4c004750 comp rx=0 tx=0).stop 2024-06-04T05:15:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.326+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/61524412 shutdown_connections 2024-06-04T05:15:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.326+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/61524412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56011299f2b0 0x5601129a3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.326+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/61524412 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5601129b35a0 0x5601129b5990 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.326+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/61524412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601129998e0 0x5601129a5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.326+0000 7fad3d7fa700 0 client.20175 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.326+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/61524412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299a2a0 0x5601129a31d0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.326+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/61524412 >> 172.21.15.70:0/61524412 conn(0x5601128c7180 msgr2=0x56011298eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.326+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/61524412 shutdown_connections 2024-06-04T05:15:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.326+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/61524412 wait complete. 2024-06-04T05:15:26.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.326+0000 7fad6a4aeb80 20 client.20175 trim_cache size 0 max 16384 2024-06-04T05:15:26.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.349+0000 7fad6a4aeb80 1 Processor -- start 2024-06-04T05:15:26.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.349+0000 7fad6a4aeb80 1 -- start start 2024-06-04T05:15:26.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad6a4aeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5601128c7180 0x5601129af720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad6a4aeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601129998e0 0x5601129a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad6a4aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299a2a0 0x5601129bb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad6a4aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fad581addf0 con 0x56011299a2a0 2024-06-04T05:15:26.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad6a4aeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad581af990 con 0x5601129998e0 2024-06-04T05:15:26.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad6a4aeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fad581ae520 con 0x5601128c7180 2024-06-04T05:15:26.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad5e1b6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299a2a0 0x5601129bb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad5e1b6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299a2a0 0x5601129bb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34500/0 (socket says 172.21.15.70:34500) 2024-06-04T05:15:26.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad5e1b6700 1 -- 172.21.15.70:0/3622752822 learned_addr learned my addr 172.21.15.70:0/3622752822 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:26.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad5d9b5700 1 --2- 172.21.15.70:0/3622752822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5601128c7180 0x5601129af720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad5d1b4700 1 --2- 172.21.15.70:0/3622752822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601129998e0 0x5601129a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad5e1b6700 1 -- 172.21.15.70:0/3622752822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5601128c7180 msgr2=0x5601129af720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad5e1b6700 1 --2- 172.21.15.70:0/3622752822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5601128c7180 0x5601129af720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad5e1b6700 1 -- 172.21.15.70:0/3622752822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601129998e0 msgr2=0x5601129a5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad5e1b6700 1 --2- 172.21.15.70:0/3622752822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601129998e0 0x5601129a5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad5e1b6700 1 -- 172.21.15.70:0/3622752822 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601129a1740 con 0x56011299a2a0 2024-06-04T05:15:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad5d1b4700 1 --2- 172.21.15.70:0/3622752822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601129998e0 0x5601129a5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:15:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.350+0000 7fad5e1b6700 1 --2- 172.21.15.70:0/3622752822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299a2a0 0x5601129bb260 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fad48005990 tx=0x7fad480092b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b725993b2e667a7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.351+0000 7fad3f7fe700 1 -- 172.21.15.70:0/3622752822 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad480052b0 con 0x56011299a2a0 2024-06-04T05:15:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.351+0000 7fad3f7fe700 1 -- 172.21.15.70:0/3622752822 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad48027070 con 0x56011299a2a0 2024-06-04T05:15:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.351+0000 7fad3f7fe700 1 -- 172.21.15.70:0/3622752822 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4800b9f0 con 0x56011299a2a0 2024-06-04T05:15:26.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.351+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/3622752822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299a2a0 msgr2=0x5601129bb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:26.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.351+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/3622752822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299a2a0 0x5601129bb260 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fad48005990 tx=0x7fad480092b0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.351+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/3622752822 shutdown_connections 2024-06-04T05:15:26.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.351+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/3622752822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56011299a2a0 0x5601129bb260 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.351+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/3622752822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5601128c7180 0x5601129af720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.351+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/3622752822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601129998e0 0x5601129a5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.351+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/3622752822 >> 172.21.15.70:0/3622752822 conn(0x56011299e660 msgr2=0x56011298eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:26.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.352+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/3622752822 shutdown_connections 2024-06-04T05:15:26.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.352+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/3622752822 wait complete. 2024-06-04T05:15:26.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.352+0000 7fad6a4aeb80 1 -- start start 2024-06-04T05:15:26.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.352+0000 7fad6a4aeb80 1 Processor -- start 2024-06-04T05:15:26.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.352+0000 7fad6a4aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128c7180 0x5601129a8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.352+0000 7fad6a4aeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601129998e0 0x5601129a8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.352+0000 7fad6a4aeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56011299a2a0 0x560112a1e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.352+0000 7fad6a4aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fad581af2c0 con 0x5601128c7180 2024-06-04T05:15:26.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.352+0000 7fad6a4aeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad581aebf0 con 0x5601129998e0 2024-06-04T05:15:26.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.352+0000 7fad6a4aeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fad581abdb0 con 0x56011299a2a0 2024-06-04T05:15:26.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad6a4aeb80 10 client.0 _ll_register_callbacks cb 0x560112883a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:26.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad5d9b5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128c7180 0x5601129a8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad5d9b5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128c7180 0x5601129a8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34504/0 (socket says 172.21.15.70:34504) 2024-06-04T05:15:26.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad5d9b5700 1 -- 172.21.15.70:0/3608825682 learned_addr learned my addr 172.21.15.70:0/3608825682 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:26.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad5d1b4700 1 --2- 172.21.15.70:0/3608825682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601129998e0 0x5601129a8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad5e1b6700 1 --2- 172.21.15.70:0/3608825682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56011299a2a0 0x560112a1e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad5d9b5700 1 -- 172.21.15.70:0/3608825682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56011299a2a0 msgr2=0x560112a1e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:26.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad5d9b5700 1 --2- 172.21.15.70:0/3608825682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56011299a2a0 0x560112a1e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad5d9b5700 1 -- 172.21.15.70:0/3608825682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601129998e0 msgr2=0x5601129a8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:26.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad5d9b5700 1 --2- 172.21.15.70:0/3608825682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601129998e0 0x5601129a8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad5d9b5700 1 -- 172.21.15.70:0/3608825682 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560112a1e680 con 0x5601128c7180 2024-06-04T05:15:26.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad5e1b6700 1 --2- 172.21.15.70:0/3608825682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56011299a2a0 0x560112a1e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:15:26.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad5d1b4700 1 --2- 172.21.15.70:0/3608825682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601129998e0 0x5601129a8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:15:26.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad5d9b5700 1 --2- 172.21.15.70:0/3608825682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128c7180 0x5601129a8a60 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fad500a98a0 tx=0x7fad50009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a187fe9bc89bde54 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:26.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad3d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:15:26.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad3d7fa700 1 -- 172.21.15.70:0/3608825682 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad500a9420 con 0x5601128c7180 2024-06-04T05:15:26.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad3d7fa700 1 -- 172.21.15.70:0/3608825682 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7fad500a9b70 con 0x5601128c7180 2024-06-04T05:15:26.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/3608825682 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad48027070 con 0x5601128c7180 2024-06-04T05:15:26.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad3d7fa700 1 -- 172.21.15.70:0/3608825682 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fad500aa010 con 0x5601128c7180 2024-06-04T05:15:26.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad6a4aeb80 10 client.20181 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:26.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.353+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/3608825682 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x560112a1f840 con 0x5601128c7180 2024-06-04T05:15:26.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.354+0000 7fad3d7fa700 1 -- 172.21.15.70:0/3608825682 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad48027070 con 0x5601128c7180 2024-06-04T05:15:26.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.354+0000 7fad3d7fa700 1 -- 172.21.15.70:0/3608825682 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7fad500b22e0 con 0x5601128c7180 2024-06-04T05:15:26.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.354+0000 7fad6a4aeb80 10 client.20181 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:26.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.354+0000 7fad6a4aeb80 10 client.20181 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:15:26.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.354+0000 7fad6a4aeb80 20 client.20181 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:26.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.354+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/3608825682 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56011299f2b0 0x560112a21a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.354+0000 7fad6a4aeb80 4 client.20181 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "77470b0", "release": true, "await": true}] 2024-06-04T05:15:26.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.354+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/3608825682 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "77470b0", "release": true, "await": true}) -- 0x5601128face0 con 0x56011299f2b0 2024-06-04T05:15:26.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.354+0000 7fad5d1b4700 1 --2- 172.21.15.70:0/3608825682 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56011299f2b0 0x560112a21a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.354+0000 7fad5d1b4700 1 --2- 172.21.15.70:0/3608825682 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56011299f2b0 0x560112a21a80 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ea217b62cfe53dd server_cookie=945c2111078fa5a3 in_seq=0 out_seq=0 2024-06-04T05:15:26.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.354+0000 7fad3d7fa700 10 client.20181 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:26.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.382+0000 7fad3d7fa700 1 -- 172.21.15.70:0/3608825682 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5601128face0 con 0x56011299f2b0 2024-06-04T05:15:26.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.382+0000 7fad3d7fa700 10 client.20181 handle_command_reply: tid=0 2024-06-04T05:15:26.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.382+0000 7fad6a4aeb80 1 client.20181 shutdown 2024-06-04T05:15:26.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.383+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/3608825682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128c7180 msgr2=0x5601129a8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:26.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.383+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/3608825682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128c7180 0x5601129a8a60 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fad500a98a0 tx=0x7fad50009eb0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.383+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/3608825682 shutdown_connections 2024-06-04T05:15:26.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.383+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/3608825682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56011299a2a0 0x560112a1e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.383+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/3608825682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128c7180 0x5601129a8a60 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.383+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/3608825682 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56011299f2b0 0x560112a21a80 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.383+0000 7fad6a4aeb80 1 --2- 172.21.15.70:0/3608825682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601129998e0 0x5601129a8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.383+0000 7fad3d7fa700 0 client.20181 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:26.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.383+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/3608825682 >> 172.21.15.70:0/3608825682 conn(0x56011299e660 msgr2=0x56011298e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:26.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.383+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/3608825682 shutdown_connections 2024-06-04T05:15:26.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.383+0000 7fad6a4aeb80 1 -- 172.21.15.70:0/3608825682 wait complete. 2024-06-04T05:15:26.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.383+0000 7fad6a4aeb80 20 client.20181 trim_cache size 0 max 16384 2024-06-04T05:15:26.386 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:15:26.387 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 55, 2024-06-04T05:15:26.387 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:15:26.387 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 56, 2024-06-04T05:15:26.387 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:15:26.387 INFO:teuthology.orchestra.run.smithi070.stdout: "77470b0": { 2024-06-04T05:15:26.387 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 56, 2024-06-04T05:15:26.387 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:15:26.387 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:15:26.387 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:15:26.387 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:15:26.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:15:26.387 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:15:26.387 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 130.0, 2024-06-04T05:15:26.387 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:15:26.388 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:15:26.388 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:15:26.388 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:15:26.388 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:15:26.388 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:15:26.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:15:26.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:15:26.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:15:26.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:15:26.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:15:26.388 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:15:26.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.384+0000 7fad5fc19700 1 -- 172.21.15.70:0/3278421163 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fad400b2a70 msgr2=0x7fad400b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:26.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.384+0000 7fad5fc19700 1 --2- 172.21.15.70:0/3278421163 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fad400b2a70 0x7fad400b4f30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fad581030c0 tx=0x7fad4c00a300 comp rx=0 tx=0).stop 2024-06-04T05:15:26.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.384+0000 7fad5fc19700 1 -- 172.21.15.70:0/3278421163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad58103850 msgr2=0x7fad5819f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:26.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.384+0000 7fad5fc19700 1 --2- 172.21.15.70:0/3278421163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad58103850 0x7fad5819f6b0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fad5000ef10 tx=0x7fad5000c3a0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.384+0000 7fad5fc19700 1 -- 172.21.15.70:0/3278421163 shutdown_connections 2024-06-04T05:15:26.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.384+0000 7fad5fc19700 1 --2- 172.21.15.70:0/3278421163 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fad400b2a70 0x7fad400b4f30 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.384+0000 7fad5fc19700 1 --2- 172.21.15.70:0/3278421163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad58108920 0x7fad581998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.384+0000 7fad5fc19700 1 --2- 172.21.15.70:0/3278421163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad58103850 0x7fad5819f6b0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.384+0000 7fad5fc19700 1 --2- 172.21.15.70:0/3278421163 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad58104200 0x7fad5819fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.384+0000 7fad5fc19700 1 -- 172.21.15.70:0/3278421163 >> 172.21.15.70:0/3278421163 conn(0x7fad5807a5d0 msgr2=0x7fad581017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:26.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.384+0000 7fad5fc19700 1 -- 172.21.15.70:0/3278421163 shutdown_connections 2024-06-04T05:15:26.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.384+0000 7fad5fc19700 1 -- 172.21.15.70:0/3278421163 wait complete. 2024-06-04T05:15:26.816 INFO:tasks.workunit.client.0.smithi070.stdout: 10 36277 7.72 MB/sec execute 138 sec latency 11085.318 ms 2024-06-04T05:15:26.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.969+0000 7f30c4b5a700 1 -- 172.21.15.70:0/4087349756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30c01084c0 msgr2=0x7f30c01088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:26.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.969+0000 7f30c4b5a700 1 --2- 172.21.15.70:0/4087349756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30c01084c0 0x7f30c01088a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f30b40098f0 tx=0x7f30b4009c00 comp rx=0 tx=0).stop 2024-06-04T05:15:26.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.970+0000 7f30c4b5a700 1 -- 172.21.15.70:0/4087349756 shutdown_connections 2024-06-04T05:15:26.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.970+0000 7f30c4b5a700 1 --2- 172.21.15.70:0/4087349756 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30c0103600 0x7f30c010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.970+0000 7f30c4b5a700 1 --2- 172.21.15.70:0/4087349756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30c01084c0 0x7f30c01088a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.970+0000 7f30c4b5a700 1 --2- 172.21.15.70:0/4087349756 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30c0102c40 0x7f30c01030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.970+0000 7f30c4b5a700 1 -- 172.21.15.70:0/4087349756 >> 172.21.15.70:0/4087349756 conn(0x7f30c0078ab0 msgr2=0x7f30c00ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:26.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.970+0000 7f30c4b5a700 1 -- 172.21.15.70:0/4087349756 shutdown_connections 2024-06-04T05:15:26.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.970+0000 7f30c4b5a700 1 -- 172.21.15.70:0/4087349756 wait complete. 2024-06-04T05:15:26.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.971+0000 7f30c4b5a700 1 Processor -- start 2024-06-04T05:15:26.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.971+0000 7f30c4b5a700 1 -- start start 2024-06-04T05:15:26.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.971+0000 7f30c4b5a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30c0102c40 0x7f30c019e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.971+0000 7f30c4b5a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30c0103600 0x7f30c019ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.971+0000 7f30c4b5a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30c01084c0 0x7f30c01988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.971+0000 7f30be59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30c0102c40 0x7f30c019e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.971+0000 7f30c4b5a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f30c0112480 con 0x7f30c0103600 2024-06-04T05:15:26.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.971+0000 7f30c4b5a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30c0112300 con 0x7f30c0102c40 2024-06-04T05:15:26.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.971+0000 7f30be59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30c0102c40 0x7f30c019e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:43254/0 (socket says 172.21.15.70:43254) 2024-06-04T05:15:26.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.971+0000 7f30c4b5a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f30c0112600 con 0x7f30c01084c0 2024-06-04T05:15:26.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.971+0000 7f30be59c700 1 -- 172.21.15.70:0/513047987 learned_addr learned my addr 172.21.15.70:0/513047987 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:26.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.971+0000 7f30bdd9b700 1 --2- 172.21.15.70:0/513047987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30c0103600 0x7f30c019ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.971+0000 7f30bed9d700 1 --2- 172.21.15.70:0/513047987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30c01084c0 0x7f30c01988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.972+0000 7f30be59c700 1 -- 172.21.15.70:0/513047987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30c01084c0 msgr2=0x7f30c01988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:26.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.972+0000 7f30be59c700 1 --2- 172.21.15.70:0/513047987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30c01084c0 0x7f30c01988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.972+0000 7f30be59c700 1 -- 172.21.15.70:0/513047987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30c0103600 msgr2=0x7f30c019ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:26.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.972+0000 7f30be59c700 1 --2- 172.21.15.70:0/513047987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30c0103600 0x7f30c019ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.972+0000 7f30be59c700 1 -- 172.21.15.70:0/513047987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30c0199040 con 0x7f30c0102c40 2024-06-04T05:15:26.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.972+0000 7f30bdd9b700 1 --2- 172.21.15.70:0/513047987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30c0103600 0x7f30c019ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:26.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.972+0000 7f30be59c700 1 --2- 172.21.15.70:0/513047987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30c0102c40 0x7f30c019e720 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009230 tx=0x7f30b400fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:26.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.972+0000 7f30af7fe700 1 -- 172.21.15.70:0/513047987 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b400b660 con 0x7f30c0102c40 2024-06-04T05:15:26.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.972+0000 7f30af7fe700 1 -- 172.21.15.70:0/513047987 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f30b4005040 con 0x7f30c0102c40 2024-06-04T05:15:26.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.972+0000 7f30c4b5a700 1 -- 172.21.15.70:0/513047987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30c01992d0 con 0x7f30c0102c40 2024-06-04T05:15:26.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.973+0000 7f30af7fe700 1 -- 172.21.15.70:0/513047987 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b40038c0 con 0x7f30c0102c40 2024-06-04T05:15:26.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.973+0000 7f30c4b5a700 1 -- 172.21.15.70:0/513047987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30c0101b30 con 0x7f30c0102c40 2024-06-04T05:15:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.976+0000 7f30af7fe700 1 -- 172.21.15.70:0/513047987 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f30b40054f0 con 0x7f30c0102c40 2024-06-04T05:15:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.977+0000 7f30af7fe700 1 --2- 172.21.15.70:0/513047987 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30a80b2a70 0x7f30a80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.977+0000 7f30bdd9b700 1 --2- 172.21.15.70:0/513047987 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30a80b2a70 0x7f30a80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.977+0000 7f30cf3efb80 1 Processor -- start 2024-06-04T05:15:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.977+0000 7f30af7fe700 1 -- 172.21.15.70:0/513047987 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f30b40e3630 con 0x7f30c0102c40 2024-06-04T05:15:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.978+0000 7f30bdd9b700 1 --2- 172.21.15.70:0/513047987 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30a80b2a70 0x7f30a80b4f30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f30c0107d00 tx=0x7f30b0009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.978+0000 7f30cf3efb80 1 -- start start 2024-06-04T05:15:26.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.978+0000 7f30cf3efb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf9320 0x562a3caf9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.978+0000 7f30cf3efb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3caf3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.978+0000 7f30cf3efb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf4260 0x562a3cb00080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.978+0000 7f30cf3efb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562a3ca51d70 con 0x562a3caf4260 2024-06-04T05:15:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.978+0000 7f30cf3efb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562a3ca41910 con 0x562a3caf9320 2024-06-04T05:15:26.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.978+0000 7f30cf3efb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562a3ca45950 con 0x562a3caf38a0 2024-06-04T05:15:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.978+0000 7f30bdd9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3caf3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.978+0000 7f30bed9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf4260 0x562a3cb00080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.978+0000 7f30be59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf9320 0x562a3caf9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.978+0000 7f30bdd9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3caf3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41292/0 (socket says 172.21.15.70:41292) 2024-06-04T05:15:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.978+0000 7f30bdd9b700 1 -- 172.21.15.70:0/2786115071 learned_addr learned my addr 172.21.15.70:0/2786115071 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.979+0000 7f30bdd9b700 1 -- 172.21.15.70:0/2786115071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf9320 msgr2=0x562a3caf9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.979+0000 7f30bdd9b700 1 --2- 172.21.15.70:0/2786115071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf9320 0x562a3caf9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.979+0000 7f30bdd9b700 1 -- 172.21.15.70:0/2786115071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf4260 msgr2=0x562a3cb00080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.979+0000 7f30bdd9b700 1 --2- 172.21.15.70:0/2786115071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf4260 0x562a3cb00080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.979+0000 7f30bdd9b700 1 -- 172.21.15.70:0/2786115071 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a3c9eafb0 con 0x562a3caf38a0 2024-06-04T05:15:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.979+0000 7f30bed9d700 1 --2- 172.21.15.70:0/2786115071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf4260 0x562a3cb00080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.979+0000 7f30bdd9b700 1 --2- 172.21.15.70:0/2786115071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3caf3d20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f30b00043c0 tx=0x7f30b0014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e75b3fab91f8e986 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.979+0000 7f30acff9700 1 -- 172.21.15.70:0/2786115071 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b0003d10 con 0x562a3caf38a0 2024-06-04T05:15:26.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.979+0000 7f30acff9700 1 -- 172.21.15.70:0/2786115071 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f30b0026890 con 0x562a3caf38a0 2024-06-04T05:15:26.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.979+0000 7f30acff9700 1 -- 172.21.15.70:0/2786115071 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b001e770 con 0x562a3caf38a0 2024-06-04T05:15:26.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.980+0000 7f30cf3efb80 1 -- 172.21.15.70:0/2786115071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 msgr2=0x562a3caf3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:26.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.980+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/2786115071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3caf3d20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f30b00043c0 tx=0x7f30b0014000 comp rx=0 tx=0).stop 2024-06-04T05:15:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.980+0000 7f30cf3efb80 1 -- 172.21.15.70:0/2786115071 shutdown_connections 2024-06-04T05:15:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.980+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/2786115071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf4260 0x562a3cb00080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.980+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/2786115071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf9320 0x562a3caf9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.980+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/2786115071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3caf3d20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.980+0000 7f30cf3efb80 1 -- 172.21.15.70:0/2786115071 >> 172.21.15.70:0/2786115071 conn(0x562a3ca1f240 msgr2=0x562a3caee3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.981+0000 7f30cf3efb80 1 -- 172.21.15.70:0/2786115071 shutdown_connections 2024-06-04T05:15:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.981+0000 7f30cf3efb80 1 -- 172.21.15.70:0/2786115071 wait complete. 2024-06-04T05:15:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.981+0000 7f30cf3efb80 1 -- start start 2024-06-04T05:15:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.981+0000 7f30cf3efb80 1 Processor -- start 2024-06-04T05:15:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30cf3efb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3caffd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30cf3efb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf4260 0x562a3cafd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30cf3efb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf9320 0x562a3cafd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30cf3efb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562a3c947fa0 con 0x562a3caf9320 2024-06-04T05:15:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30cf3efb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562a3c9a9da0 con 0x562a3caf4260 2024-06-04T05:15:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30cf3efb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562a3cb006e0 con 0x562a3caf38a0 2024-06-04T05:15:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30cf3efb80 10 client.0 _ll_register_callbacks cb 0x562a3c9dc950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30bed9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf9320 0x562a3cafd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30be59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3caffd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30be59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3caffd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41302/0 (socket says 172.21.15.70:41302) 2024-06-04T05:15:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30be59c700 1 -- 172.21.15.70:0/1826542862 learned_addr learned my addr 172.21.15.70:0/1826542862 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30bdd9b700 1 --2- 172.21.15.70:0/1826542862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf4260 0x562a3cafd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30bdd9b700 1 -- 172.21.15.70:0/1826542862 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 msgr2=0x562a3caffd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30bdd9b700 1 --2- 172.21.15.70:0/1826542862 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3caffd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30bdd9b700 1 -- 172.21.15.70:0/1826542862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf9320 msgr2=0x562a3cafd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30bdd9b700 1 --2- 172.21.15.70:0/1826542862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf9320 0x562a3cafd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30bdd9b700 1 -- 172.21.15.70:0/1826542862 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a3c9918d0 con 0x562a3caf4260 2024-06-04T05:15:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30bed9d700 1 --2- 172.21.15.70:0/1826542862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf9320 0x562a3cafd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.982+0000 7f30bdd9b700 1 --2- 172.21.15.70:0/1826542862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf4260 0x562a3cafd060 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f30b00043c0 tx=0x7f30b0003d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c054d5e881ff247b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.983+0000 7f30a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:15:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.983+0000 7f30a5ffb700 1 -- 172.21.15.70:0/1826542862 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b0026e50 con 0x562a3caf4260 2024-06-04T05:15:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.983+0000 7f30a5ffb700 1 -- 172.21.15.70:0/1826542862 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f30b0005030 con 0x562a3caf4260 2024-06-04T05:15:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.983+0000 7f30cf3efb80 1 -- 172.21.15.70:0/1826542862 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30b0026890 con 0x562a3caf4260 2024-06-04T05:15:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.983+0000 7f30a5ffb700 1 -- 172.21.15.70:0/1826542862 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f30b0005230 con 0x562a3caf4260 2024-06-04T05:15:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.983+0000 7f30cf3efb80 10 client.28994 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.983+0000 7f30cf3efb80 1 -- 172.21.15.70:0/1826542862 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x562a3c9fb760 con 0x562a3caf4260 2024-06-04T05:15:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.983+0000 7f30a5ffb700 1 -- 172.21.15.70:0/1826542862 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f30b0026890 con 0x562a3caf4260 2024-06-04T05:15:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.983+0000 7f30a5ffb700 1 -- 172.21.15.70:0/1826542862 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f30b0037430 con 0x562a3caf4260 2024-06-04T05:15:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.984+0000 7f30cf3efb80 10 client.28994 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.984+0000 7f30cf3efb80 10 client.28994 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:15:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.984+0000 7f30cf3efb80 20 client.28994 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.984+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/1826542862 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x562a3cb15900 0x562a3cb17cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.984+0000 7f30cf3efb80 4 client.28994 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:15:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.984+0000 7f30cf3efb80 1 -- 172.21.15.70:0/1826542862 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a3ca53cc0 con 0x562a3cb15900 2024-06-04T05:15:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.984+0000 7f30be59c700 1 --2- 172.21.15.70:0/1826542862 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x562a3cb15900 0x562a3cb17cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.984+0000 7f30a5ffb700 10 client.28994 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:15:26.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.984+0000 7f30be59c700 1 --2- 172.21.15.70:0/1826542862 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x562a3cb15900 0x562a3cb17cf0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7258f9659558c560 server_cookie=42ca906df67754e8 in_seq=0 out_seq=0 2024-06-04T05:15:26.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.985+0000 7f30a5ffb700 1 -- 172.21.15.70:0/1826542862 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562a3ca53cc0 con 0x562a3cb15900 2024-06-04T05:15:26.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.985+0000 7f30a5ffb700 10 client.28994 handle_command_reply: tid=0 2024-06-04T05:15:26.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.985+0000 7f30cf3efb80 1 client.28994 shutdown 2024-06-04T05:15:26.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.985+0000 7f30cf3efb80 1 -- 172.21.15.70:0/1826542862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf4260 msgr2=0x562a3cafd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:26.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.985+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/1826542862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf4260 0x562a3cafd060 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f30b00043c0 tx=0x7f30b0003d40 comp rx=0 tx=0).stop 2024-06-04T05:15:26.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.985+0000 7f30cf3efb80 1 -- 172.21.15.70:0/1826542862 shutdown_connections 2024-06-04T05:15:26.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.985+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/1826542862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf9320 0x562a3cafd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.985+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/1826542862 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x562a3cb15900 0x562a3cb17cf0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.985+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/1826542862 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3caffd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.985+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/1826542862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf4260 0x562a3cafd060 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:26.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.985+0000 7f30a5ffb700 0 client.28994 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:15:26.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.985+0000 7f30cf3efb80 1 -- 172.21.15.70:0/1826542862 >> 172.21.15.70:0/1826542862 conn(0x562a3ca1f240 msgr2=0x562a3caedaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:26.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.985+0000 7f30cf3efb80 1 -- 172.21.15.70:0/1826542862 shutdown_connections 2024-06-04T05:15:26.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.985+0000 7f30cf3efb80 1 -- 172.21.15.70:0/1826542862 wait complete. 2024-06-04T05:15:26.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:26.985+0000 7f30cf3efb80 20 client.28994 trim_cache size 0 max 16384 2024-06-04T05:15:27.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.008+0000 7f30cf3efb80 1 Processor -- start 2024-06-04T05:15:27.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.008+0000 7f30cf3efb80 1 -- start start 2024-06-04T05:15:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.008+0000 7f30cf3efb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3caf87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.008+0000 7f30cf3efb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf4260 0x562a3caee270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.008+0000 7f30cf3efb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf9320 0x562a3cafa010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.008+0000 7f30cf3efb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f30c0073500 con 0x562a3caf9320 2024-06-04T05:15:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.008+0000 7f30cf3efb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30c00750a0 con 0x562a3caf4260 2024-06-04T05:15:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.008+0000 7f30cf3efb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f30c0073c30 con 0x562a3caf38a0 2024-06-04T05:15:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.008+0000 7f30bed9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf9320 0x562a3cafa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.008+0000 7f30bdd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf4260 0x562a3caee270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.008+0000 7f30bed9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf9320 0x562a3cafa010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34550/0 (socket says 172.21.15.70:34550) 2024-06-04T05:15:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.008+0000 7f30bed9d700 1 -- 172.21.15.70:0/871993509 learned_addr learned my addr 172.21.15.70:0/871993509 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.008+0000 7f30be59c700 1 --2- 172.21.15.70:0/871993509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3caf87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.009+0000 7f30bdd9b700 1 -- 172.21.15.70:0/871993509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 msgr2=0x562a3caf87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.009+0000 7f30bdd9b700 1 --2- 172.21.15.70:0/871993509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3caf87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.009+0000 7f30bdd9b700 1 -- 172.21.15.70:0/871993509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf9320 msgr2=0x562a3cafa010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.009+0000 7f30bdd9b700 1 --2- 172.21.15.70:0/871993509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf9320 0x562a3cafa010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.009+0000 7f30bdd9b700 1 -- 172.21.15.70:0/871993509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a3cafb5d0 con 0x562a3caf4260 2024-06-04T05:15:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.009+0000 7f30bed9d700 1 --2- 172.21.15.70:0/871993509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf9320 0x562a3cafa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.009+0000 7f30bdd9b700 1 --2- 172.21.15.70:0/871993509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf4260 0x562a3caee270 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f30b0003960 tx=0x7f30b0004050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8b641a62849f84a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.009+0000 7f30a7fff700 1 -- 172.21.15.70:0/871993509 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b00053d0 con 0x562a3caf4260 2024-06-04T05:15:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.009+0000 7f30a7fff700 1 -- 172.21.15.70:0/871993509 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f30b0005570 con 0x562a3caf4260 2024-06-04T05:15:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.009+0000 7f30a7fff700 1 -- 172.21.15.70:0/871993509 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b0004d00 con 0x562a3caf4260 2024-06-04T05:15:27.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.010+0000 7f30cf3efb80 1 -- 172.21.15.70:0/871993509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf4260 msgr2=0x562a3caee270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:27.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.010+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/871993509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf4260 0x562a3caee270 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f30b0003960 tx=0x7f30b0004050 comp rx=0 tx=0).stop 2024-06-04T05:15:27.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.010+0000 7f30cf3efb80 1 -- 172.21.15.70:0/871993509 shutdown_connections 2024-06-04T05:15:27.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.010+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/871993509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf9320 0x562a3cafa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.010+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/871993509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3caf87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.010+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/871993509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf4260 0x562a3caee270 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.010+0000 7f30cf3efb80 1 -- 172.21.15.70:0/871993509 >> 172.21.15.70:0/871993509 conn(0x562a3ca1f240 msgr2=0x562a3caf3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:27.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.010+0000 7f30cf3efb80 1 -- 172.21.15.70:0/871993509 shutdown_connections 2024-06-04T05:15:27.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.010+0000 7f30cf3efb80 1 -- 172.21.15.70:0/871993509 wait complete. 2024-06-04T05:15:27.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.010+0000 7f30cf3efb80 1 -- start start 2024-06-04T05:15:27.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30cf3efb80 1 Processor -- start 2024-06-04T05:15:27.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30cf3efb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3cb0e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30cf3efb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf4260 0x562a3cb1c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30cf3efb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf9320 0x562a3cb1ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30cf3efb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f30c00749d0 con 0x562a3caf4260 2024-06-04T05:15:27.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30cf3efb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30c0074300 con 0x562a3caf9320 2024-06-04T05:15:27.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30cf3efb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f30c00714c0 con 0x562a3caf38a0 2024-06-04T05:15:27.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30cf3efb80 10 client.0 _ll_register_callbacks cb 0x562a3c9dc950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:27.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30bdd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf4260 0x562a3cb1c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30bdd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf4260 0x562a3cb1c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34566/0 (socket says 172.21.15.70:34566) 2024-06-04T05:15:27.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30be59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3cb0e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30bdd9b700 1 -- 172.21.15.70:0/1362943657 learned_addr learned my addr 172.21.15.70:0/1362943657 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:27.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30bed9d700 1 --2- 172.21.15.70:0/1362943657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf9320 0x562a3cb1ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30be59c700 1 -- 172.21.15.70:0/1362943657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf9320 msgr2=0x562a3cb1ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:27.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30be59c700 1 --2- 172.21.15.70:0/1362943657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf9320 0x562a3cb1ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30be59c700 1 -- 172.21.15.70:0/1362943657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf4260 msgr2=0x562a3cb1c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:27.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30be59c700 1 --2- 172.21.15.70:0/1362943657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf4260 0x562a3cb1c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30be59c700 1 -- 172.21.15.70:0/1362943657 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a3cb1d4c0 con 0x562a3caf38a0 2024-06-04T05:15:27.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.011+0000 7f30bdd9b700 1 --2- 172.21.15.70:0/1362943657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf4260 0x562a3cb1c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:27.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.012+0000 7f30be59c700 1 --2- 172.21.15.70:0/1362943657 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3cb0e730 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f30b40a6990 tx=0x7f30b40b1040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe7864e4aa72305b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:27.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.012+0000 7f30a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:15:27.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.012+0000 7f30a5ffb700 1 -- 172.21.15.70:0/1362943657 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b40aa910 con 0x562a3caf38a0 2024-06-04T05:15:27.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.012+0000 7f30cf3efb80 1 -- 172.21.15.70:0/1362943657 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30b0005570 con 0x562a3caf38a0 2024-06-04T05:15:27.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.012+0000 7f30a5ffb700 1 -- 172.21.15.70:0/1362943657 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f30b406a0b0 con 0x562a3caf38a0 2024-06-04T05:15:27.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.012+0000 7f30a5ffb700 1 -- 172.21.15.70:0/1362943657 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f30b40c6070 con 0x562a3caf38a0 2024-06-04T05:15:27.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.012+0000 7f30cf3efb80 10 client.29104 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:27.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.012+0000 7f30cf3efb80 1 -- 172.21.15.70:0/1362943657 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x562a3cb80160 con 0x562a3caf38a0 2024-06-04T05:15:27.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.012+0000 7f30a5ffb700 1 -- 172.21.15.70:0/1362943657 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f30b0005570 con 0x562a3caf38a0 2024-06-04T05:15:27.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.013+0000 7f30a5ffb700 1 -- 172.21.15.70:0/1362943657 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f30b40a6ba0 con 0x562a3caf38a0 2024-06-04T05:15:27.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.013+0000 7f30cf3efb80 10 client.29104 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:27.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.013+0000 7f30cf3efb80 10 client.29104 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:15:27.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.013+0000 7f30cf3efb80 20 client.29104 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:27.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.013+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/1362943657 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x562a3cb806b0 0x562a3cb82ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.013+0000 7f30cf3efb80 4 client.29104 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:15:27.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.013+0000 7f30cf3efb80 1 -- 172.21.15.70:0/1362943657 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562a3ca4d4a0 con 0x562a3cb806b0 2024-06-04T05:15:27.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.013+0000 7f30bdd9b700 1 --2- 172.21.15.70:0/1362943657 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x562a3cb806b0 0x562a3cb82ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.013+0000 7f30a5ffb700 10 client.29104 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:15:27.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.013+0000 7f30bdd9b700 1 --2- 172.21.15.70:0/1362943657 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x562a3cb806b0 0x562a3cb82ad0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5abf40d099751319 server_cookie=2947730efe889cfa in_seq=0 out_seq=0 2024-06-04T05:15:27.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.014+0000 7f30a5ffb700 1 -- 172.21.15.70:0/1362943657 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562a3ca4d4a0 con 0x562a3cb806b0 2024-06-04T05:15:27.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.014+0000 7f30a5ffb700 10 client.29104 handle_command_reply: tid=0 2024-06-04T05:15:27.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.014+0000 7f30cf3efb80 1 client.29104 shutdown 2024-06-04T05:15:27.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.014+0000 7f30cf3efb80 1 -- 172.21.15.70:0/1362943657 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 msgr2=0x562a3cb0e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:27.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.014+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/1362943657 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3cb0e730 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f30b40a6990 tx=0x7f30b40b1040 comp rx=0 tx=0).stop 2024-06-04T05:15:27.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.014+0000 7f30cf3efb80 1 -- 172.21.15.70:0/1362943657 shutdown_connections 2024-06-04T05:15:27.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.014+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/1362943657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a3caf9320 0x562a3cb1ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.014+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/1362943657 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x562a3cb806b0 0x562a3cb82ad0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.014+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/1362943657 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a3caf38a0 0x562a3cb0e730 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.014+0000 7f30cf3efb80 1 --2- 172.21.15.70:0/1362943657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a3caf4260 0x562a3cb1c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.014+0000 7f30cf3efb80 1 -- 172.21.15.70:0/1362943657 >> 172.21.15.70:0/1362943657 conn(0x562a3ca1f240 msgr2=0x562a3caf6ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:27.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.014+0000 7f30a5ffb700 0 client.29104 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:15:27.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.014+0000 7f30cf3efb80 1 -- 172.21.15.70:0/1362943657 shutdown_connections 2024-06-04T05:15:27.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.014+0000 7f30cf3efb80 1 -- 172.21.15.70:0/1362943657 wait complete. 2024-06-04T05:15:27.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.014+0000 7f30cf3efb80 20 client.29104 trim_cache size 0 max 16384 2024-06-04T05:15:27.021 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:15:27.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.015+0000 7f30c4b5a700 1 -- 172.21.15.70:0/513047987 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30a80b2a70 msgr2=0x7f30a80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:27.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.015+0000 7f30c4b5a700 1 --2- 172.21.15.70:0/513047987 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30a80b2a70 0x7f30a80b4f30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f30c0107d00 tx=0x7f30b0009260 comp rx=0 tx=0).stop 2024-06-04T05:15:27.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.015+0000 7f30c4b5a700 1 -- 172.21.15.70:0/513047987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30c0102c40 msgr2=0x7f30c019e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:27.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.015+0000 7f30c4b5a700 1 --2- 172.21.15.70:0/513047987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30c0102c40 0x7f30c019e720 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009230 tx=0x7f30b400fa00 comp rx=0 tx=0).stop 2024-06-04T05:15:27.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.015+0000 7f30c4b5a700 1 -- 172.21.15.70:0/513047987 shutdown_connections 2024-06-04T05:15:27.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.015+0000 7f30c4b5a700 1 --2- 172.21.15.70:0/513047987 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f30a80b2a70 0x7f30a80b4f30 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.015+0000 7f30c4b5a700 1 --2- 172.21.15.70:0/513047987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30c01084c0 0x7f30c01988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.015+0000 7f30c4b5a700 1 --2- 172.21.15.70:0/513047987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30c0102c40 0x7f30c019e720 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.015+0000 7f30c4b5a700 1 --2- 172.21.15.70:0/513047987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30c0103600 0x7f30c019ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.015+0000 7f30c4b5a700 1 -- 172.21.15.70:0/513047987 >> 172.21.15.70:0/513047987 conn(0x7f30c0078ab0 msgr2=0x7f30c0100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:27.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.015+0000 7f30c4b5a700 1 -- 172.21.15.70:0/513047987 shutdown_connections 2024-06-04T05:15:27.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:27.015+0000 7f30c4b5a700 1 -- 172.21.15.70:0/513047987 wait complete. 2024-06-04T05:15:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.593+0000 7f0467ee1640 1 -- 172.21.15.70:0/2149731050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0460107940 msgr2=0x7f0460107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.593+0000 7f0467ee1640 1 --2- 172.21.15.70:0/2149731050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0460107940 0x7f0460107d20 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f0458009cd0 tx=0x7f045802a8e0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.593+0000 7f0467ee1640 1 -- 172.21.15.70:0/2149731050 shutdown_connections 2024-06-04T05:15:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.593+0000 7f0467ee1640 1 --2- 172.21.15.70:0/2149731050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0460102340 0x7f046010e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.593+0000 7f0467ee1640 1 --2- 172.21.15.70:0/2149731050 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0460101980 0x7f0460101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.593+0000 7f0467ee1640 1 --2- 172.21.15.70:0/2149731050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0460107940 0x7f0460107d20 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.593+0000 7f0467ee1640 1 -- 172.21.15.70:0/2149731050 >> 172.21.15.70:0/2149731050 conn(0x7f0460078ad0 msgr2=0x7f0460078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.593+0000 7f0467ee1640 1 -- 172.21.15.70:0/2149731050 shutdown_connections 2024-06-04T05:15:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.593+0000 7f0467ee1640 1 -- 172.21.15.70:0/2149731050 wait complete. 2024-06-04T05:15:27.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.594+0000 7f0467ee1640 1 Processor -- start 2024-06-04T05:15:27.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.594+0000 7f0467ee1640 1 -- start start 2024-06-04T05:15:27.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.594+0000 7f0467ee1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0460101980 0x7f046019b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.594+0000 7f0467ee1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0460102340 0x7f046019b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f0467ee1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0460107940 0x7f046019fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f0467ee1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0460114d30 con 0x7f0460102340 2024-06-04T05:15:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f0467ee1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0460114bb0 con 0x7f0460101980 2024-06-04T05:15:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f0467ee1640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0460114eb0 con 0x7f0460107940 2024-06-04T05:15:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f046547c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0460102340 0x7f046019b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f046547c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0460102340 0x7f046019b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34576/0 (socket says 172.21.15.70:34576) 2024-06-04T05:15:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f046547c640 1 -- 172.21.15.70:0/2864393700 learned_addr learned my addr 172.21.15.70:0/2864393700 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f0465c7d640 1 --2- 172.21.15.70:0/2864393700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0460101980 0x7f046019b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f046647e640 1 --2- 172.21.15.70:0/2864393700 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0460107940 0x7f046019fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f0465c7d640 1 -- 172.21.15.70:0/2864393700 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0460107940 msgr2=0x7f046019fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f0465c7d640 1 --2- 172.21.15.70:0/2864393700 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0460107940 0x7f046019fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f0465c7d640 1 -- 172.21.15.70:0/2864393700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0460102340 msgr2=0x7f046019b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f0465c7d640 1 --2- 172.21.15.70:0/2864393700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0460102340 0x7f046019b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f0465c7d640 1 -- 172.21.15.70:0/2864393700 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04601a02a0 con 0x7f0460101980 2024-06-04T05:15:27.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f046547c640 1 --2- 172.21.15.70:0/2864393700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0460102340 0x7f046019b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:27.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f0465c7d640 1 --2- 172.21.15.70:0/2864393700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0460101980 0x7f046019b2b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0458000f80 tx=0x7f0458003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:27.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f0456ffd640 1 -- 172.21.15.70:0/2864393700 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0458004040 con 0x7f0460101980 2024-06-04T05:15:27.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.595+0000 7f0467ee1640 1 -- 172.21.15.70:0/2864393700 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04601a0530 con 0x7f0460101980 2024-06-04T05:15:27.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.596+0000 7f0456ffd640 1 -- 172.21.15.70:0/2864393700 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0458004670 con 0x7f0460101980 2024-06-04T05:15:27.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.596+0000 7f0456ffd640 1 -- 172.21.15.70:0/2864393700 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0458030bf0 con 0x7f0460101980 2024-06-04T05:15:27.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.596+0000 7f0467ee1640 1 -- 172.21.15.70:0/2864393700 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04601a7e20 con 0x7f0460101980 2024-06-04T05:15:27.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.599+0000 7f0456ffd640 1 -- 172.21.15.70:0/2864393700 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f04580041e0 con 0x7f0460101980 2024-06-04T05:15:27.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.599+0000 7f046ac02740 1 Processor -- start 2024-06-04T05:15:27.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.599+0000 7f046ac02740 1 -- start start 2024-06-04T05:15:27.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f046ac02740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da1f9820 0x55e1da1f9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f046ac02740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1fa140 0x55e1da2044c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f046ac02740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 0x55e1da206df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f046ac02740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e1da16bc30 con 0x55e1da204a00 2024-06-04T05:15:27.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f046ac02740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e1da15b7d0 con 0x55e1da1fa140 2024-06-04T05:15:27.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f046ac02740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e1da15f810 con 0x55e1da1f9820 2024-06-04T05:15:27.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f0465c7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1fa140 0x55e1da2044c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f0465c7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1fa140 0x55e1da2044c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:43310/0 (socket says 172.21.15.70:43310) 2024-06-04T05:15:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f0465c7d640 1 -- 172.21.15.70:0/2902005699 learned_addr learned my addr 172.21.15.70:0/2902005699 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f046647e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da1f9820 0x55e1da1f9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f046547c640 1 --2- 172.21.15.70:0/2902005699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 0x55e1da206df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f0456ffd640 1 --2- 172.21.15.70:0/2864393700 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f044c0b2a10 0x7f044c0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f046647e640 1 -- 172.21.15.70:0/2902005699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1fa140 msgr2=0x55e1da2044c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f046647e640 1 --2- 172.21.15.70:0/2902005699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1fa140 0x55e1da2044c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f046647e640 1 -- 172.21.15.70:0/2902005699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 msgr2=0x55e1da206df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f046647e640 1 --2- 172.21.15.70:0/2902005699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 0x55e1da206df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f046647e640 1 -- 172.21.15.70:0/2902005699 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1d9e70210 con 0x55e1da1f9820 2024-06-04T05:15:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f046547c640 1 --2- 172.21.15.70:0/2902005699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 0x55e1da206df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.600+0000 7f046647e640 1 --2- 172.21.15.70:0/2864393700 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f044c0b2a10 0x7f044c0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.601+0000 7f0456ffd640 1 -- 172.21.15.70:0/2864393700 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f045802e030 con 0x7f0460101980 2024-06-04T05:15:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.601+0000 7f046647e640 1 --2- 172.21.15.70:0/2902005699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da1f9820 0x55e1da1f9c00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0450009c20 tx=0x7f0450011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f175f8212a2dbc6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:27.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.601+0000 7f044bfff640 1 -- 172.21.15.70:0/2902005699 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0450017070 con 0x55e1da1f9820 2024-06-04T05:15:27.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.601+0000 7f044bfff640 1 -- 172.21.15.70:0/2902005699 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f04500268c0 con 0x55e1da1f9820 2024-06-04T05:15:27.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.601+0000 7f044bfff640 1 -- 172.21.15.70:0/2902005699 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045001c650 con 0x55e1da1f9820 2024-06-04T05:15:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.602+0000 7f046ac02740 1 -- 172.21.15.70:0/2902005699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da1f9820 msgr2=0x55e1da1f9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.602+0000 7f046ac02740 1 --2- 172.21.15.70:0/2902005699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da1f9820 0x55e1da1f9c00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0450009c20 tx=0x7f0450011810 comp rx=0 tx=0).stop 2024-06-04T05:15:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.602+0000 7f046647e640 1 --2- 172.21.15.70:0/2864393700 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f044c0b2a10 0x7f044c0b4ed0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0450021000 tx=0x7f0450022000 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.602+0000 7f046ac02740 1 -- 172.21.15.70:0/2902005699 shutdown_connections 2024-06-04T05:15:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.602+0000 7f046ac02740 1 --2- 172.21.15.70:0/2902005699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1fa140 0x55e1da2044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.602+0000 7f046ac02740 1 --2- 172.21.15.70:0/2902005699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 0x55e1da206df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.602+0000 7f046ac02740 1 --2- 172.21.15.70:0/2902005699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da1f9820 0x55e1da1f9c00 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.602+0000 7f046ac02740 1 -- 172.21.15.70:0/2902005699 >> 172.21.15.70:0/2902005699 conn(0x55e1da1f5d30 msgr2=0x55e1da1f6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.602+0000 7f046ac02740 1 -- 172.21.15.70:0/2902005699 shutdown_connections 2024-06-04T05:15:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.602+0000 7f046ac02740 1 -- 172.21.15.70:0/2902005699 wait complete. 2024-06-04T05:15:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.603+0000 7f046ac02740 1 -- start start 2024-06-04T05:15:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.603+0000 7f046ac02740 1 Processor -- start 2024-06-04T05:15:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.604+0000 7f046ac02740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1f9820 0x55e1da2019b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.604+0000 7f046ac02740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da1fa140 0x55e1da201ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.604+0000 7f046ac02740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 0x55e1da202430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.604+0000 7f0465c7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1f9820 0x55e1da2019b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.604+0000 7f046ac02740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e1d9e41790 con 0x55e1da204a00 2024-06-04T05:15:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.604+0000 7f046ac02740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e1da117890 con 0x55e1da1f9820 2024-06-04T05:15:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.604+0000 7f046ac02740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e1da207630 con 0x55e1da1fa140 2024-06-04T05:15:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.604+0000 7f0465c7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1f9820 0x55e1da2019b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:43320/0 (socket says 172.21.15.70:43320) 2024-06-04T05:15:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.604+0000 7f0465c7d640 1 -- 172.21.15.70:0/1441145534 learned_addr learned my addr 172.21.15.70:0/1441145534 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.604+0000 7f046ac02740 10 client.0 _ll_register_callbacks cb 0x55e1d9e6fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.604+0000 7f046647e640 1 --2- 172.21.15.70:0/1441145534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 0x55e1da202430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.604+0000 7f046547c640 1 --2- 172.21.15.70:0/1441145534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da1fa140 0x55e1da201ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.605+0000 7f0465c7d640 1 -- 172.21.15.70:0/1441145534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da1fa140 msgr2=0x55e1da201ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.605+0000 7f0465c7d640 1 --2- 172.21.15.70:0/1441145534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da1fa140 0x55e1da201ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.605+0000 7f0465c7d640 1 -- 172.21.15.70:0/1441145534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 msgr2=0x55e1da202430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.605+0000 7f0465c7d640 1 --2- 172.21.15.70:0/1441145534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 0x55e1da202430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.605+0000 7f0465c7d640 1 -- 172.21.15.70:0/1441145534 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1d9f3c620 con 0x55e1da1f9820 2024-06-04T05:15:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.605+0000 7f046647e640 1 --2- 172.21.15.70:0/1441145534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 0x55e1da202430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:15:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.605+0000 7f0465c7d640 1 --2- 172.21.15.70:0/1441145534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1f9820 0x55e1da2019b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f04580c79a0 tx=0x7f0458030d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95500c8a9f90e43c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.605+0000 7f04497fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:15:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.605+0000 7f04497fa640 1 -- 172.21.15.70:0/1441145534 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04580c65f0 con 0x55e1da1f9820 2024-06-04T05:15:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.605+0000 7f046ac02740 1 -- 172.21.15.70:0/1441145534 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04500268c0 con 0x55e1da1f9820 2024-06-04T05:15:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.605+0000 7f04497fa640 1 -- 172.21.15.70:0/1441145534 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f0458100d10 con 0x55e1da1f9820 2024-06-04T05:15:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.605+0000 7f04497fa640 1 -- 172.21.15.70:0/1441145534 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f04580044d0 con 0x55e1da1f9820 2024-06-04T05:15:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.605+0000 7f046ac02740 10 client.29012 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.605+0000 7f046ac02740 1 -- 172.21.15.70:0/1441145534 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55e1da21bbf0 con 0x55e1da1f9820 2024-06-04T05:15:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.605+0000 7f04497fa640 1 -- 172.21.15.70:0/1441145534 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f04500268c0 con 0x55e1da1f9820 2024-06-04T05:15:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.606+0000 7f04497fa640 1 -- 172.21.15.70:0/1441145534 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f0458030070 con 0x55e1da1f9820 2024-06-04T05:15:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.606+0000 7f046ac02740 10 client.29012 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.606+0000 7f046ac02740 10 client.29012 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.606+0000 7f046ac02740 20 client.29012 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.606+0000 7f046ac02740 1 --2- 172.21.15.70:0/1441145534 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e1da21d0a0 0x55e1da21f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.606+0000 7f046ac02740 4 client.29012 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:15:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.606+0000 7f046ac02740 1 -- 172.21.15.70:0/1441145534 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1da16db80 con 0x55e1da21d0a0 2024-06-04T05:15:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.606+0000 7f046547c640 1 --2- 172.21.15.70:0/1441145534 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e1da21d0a0 0x55e1da21f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.612 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:27 smithi070 ceph-mon[32631]: pgmap v573: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 2.5 MiB/s wr, 180 op/s 2024-06-04T05:15:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.606+0000 7f046547c640 1 --2- 172.21.15.70:0/1441145534 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e1da21d0a0 0x55e1da21f4b0 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c57bae7de00e03d5 server_cookie=6e134d3c321bc07f in_seq=0 out_seq=0 2024-06-04T05:15:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.607+0000 7f04497fa640 10 client.29012 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.607+0000 7f04497fa640 1 -- 172.21.15.70:0/1441145534 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e1da16db80 con 0x55e1da21d0a0 2024-06-04T05:15:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.607+0000 7f04497fa640 10 client.29012 handle_command_reply: tid=0 2024-06-04T05:15:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.607+0000 7f046ac02740 1 client.29012 shutdown 2024-06-04T05:15:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.607+0000 7f046ac02740 1 -- 172.21.15.70:0/1441145534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1f9820 msgr2=0x55e1da2019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.607+0000 7f046ac02740 1 --2- 172.21.15.70:0/1441145534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1f9820 0x55e1da2019b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f04580c79a0 tx=0x7f0458030d90 comp rx=0 tx=0).stop 2024-06-04T05:15:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.608+0000 7f046ac02740 1 -- 172.21.15.70:0/1441145534 shutdown_connections 2024-06-04T05:15:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.608+0000 7f046ac02740 1 --2- 172.21.15.70:0/1441145534 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e1da21d0a0 0x55e1da21f4b0 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.608+0000 7f046ac02740 1 --2- 172.21.15.70:0/1441145534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 0x55e1da202430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.608+0000 7f04497fa640 0 client.29012 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.608+0000 7f046ac02740 1 --2- 172.21.15.70:0/1441145534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da1fa140 0x55e1da201ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.608+0000 7f046ac02740 1 --2- 172.21.15.70:0/1441145534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1f9820 0x55e1da2019b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.608+0000 7f046ac02740 1 -- 172.21.15.70:0/1441145534 >> 172.21.15.70:0/1441145534 conn(0x55e1da1f5d30 msgr2=0x55e1da1f4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.608+0000 7f046ac02740 1 -- 172.21.15.70:0/1441145534 shutdown_connections 2024-06-04T05:15:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.608+0000 7f046ac02740 1 -- 172.21.15.70:0/1441145534 wait complete. 2024-06-04T05:15:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.608+0000 7f046ac02740 20 client.29012 trim_cache size 0 max 16384 2024-06-04T05:15:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.631+0000 7f046ac02740 1 Processor -- start 2024-06-04T05:15:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.631+0000 7f046ac02740 1 -- start start 2024-06-04T05:15:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f046ac02740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1f9820 0x55e1da202250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f046ac02740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da1fa140 0x55e1da224dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f046ac02740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da204a00 0x55e1da225300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f046ac02740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f04601ac370 con 0x55e1da1fa140 2024-06-04T05:15:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f046ac02740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f04601adf10 con 0x55e1da1f9820 2024-06-04T05:15:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f046ac02740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f04601acaa0 con 0x55e1da204a00 2024-06-04T05:15:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f046547c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da1fa140 0x55e1da224dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f046547c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da1fa140 0x55e1da224dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34602/0 (socket says 172.21.15.70:34602) 2024-06-04T05:15:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f046547c640 1 -- 172.21.15.70:0/2703179310 learned_addr learned my addr 172.21.15.70:0/2703179310 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f0465c7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1f9820 0x55e1da202250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f046647e640 1 --2- 172.21.15.70:0/2703179310 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da204a00 0x55e1da225300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f0465c7d640 1 -- 172.21.15.70:0/2703179310 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da204a00 msgr2=0x55e1da225300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f0465c7d640 1 --2- 172.21.15.70:0/2703179310 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da204a00 0x55e1da225300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f0465c7d640 1 -- 172.21.15.70:0/2703179310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da1fa140 msgr2=0x55e1da224dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f0465c7d640 1 --2- 172.21.15.70:0/2703179310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da1fa140 0x55e1da224dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f0465c7d640 1 -- 172.21.15.70:0/2703179310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1da1fd590 con 0x55e1da1f9820 2024-06-04T05:15:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f046547c640 1 --2- 172.21.15.70:0/2703179310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da1fa140 0x55e1da224dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:27.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.632+0000 7f0465c7d640 1 --2- 172.21.15.70:0/2703179310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1f9820 0x55e1da202250 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0458005e30 tx=0x7f04580095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cdbdf74dae91e6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:27.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.633+0000 7f044b7fe640 1 -- 172.21.15.70:0/2703179310 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04580c9ea0 con 0x55e1da1f9820 2024-06-04T05:15:27.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.633+0000 7f044b7fe640 1 -- 172.21.15.70:0/2703179310 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0458036070 con 0x55e1da1f9820 2024-06-04T05:15:27.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.633+0000 7f044b7fe640 1 -- 172.21.15.70:0/2703179310 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04580ca080 con 0x55e1da1f9820 2024-06-04T05:15:27.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.634+0000 7f046ac02740 1 -- 172.21.15.70:0/2703179310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1f9820 msgr2=0x55e1da202250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:27.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.634+0000 7f046ac02740 1 --2- 172.21.15.70:0/2703179310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1f9820 0x55e1da202250 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0458005e30 tx=0x7f04580095c0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.634+0000 7f046ac02740 1 -- 172.21.15.70:0/2703179310 shutdown_connections 2024-06-04T05:15:27.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.634+0000 7f046ac02740 1 --2- 172.21.15.70:0/2703179310 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da204a00 0x55e1da225300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.634+0000 7f046ac02740 1 --2- 172.21.15.70:0/2703179310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da1fa140 0x55e1da224dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.634+0000 7f046ac02740 1 --2- 172.21.15.70:0/2703179310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1f9820 0x55e1da202250 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.634+0000 7f046ac02740 1 -- 172.21.15.70:0/2703179310 >> 172.21.15.70:0/2703179310 conn(0x55e1da1f5d30 msgr2=0x55e1da1fba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.634+0000 7f046ac02740 1 -- 172.21.15.70:0/2703179310 shutdown_connections 2024-06-04T05:15:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.634+0000 7f046ac02740 1 -- 172.21.15.70:0/2703179310 wait complete. 2024-06-04T05:15:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.634+0000 7f046ac02740 1 -- start start 2024-06-04T05:15:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.634+0000 7f046ac02740 1 Processor -- start 2024-06-04T05:15:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046ac02740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da1f9820 0x55e1da20d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046ac02740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1fa140 0x55e1da1fe8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046ac02740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 0x55e1da1fef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046ac02740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f04601ad840 con 0x55e1da204a00 2024-06-04T05:15:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046ac02740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f04601ad170 con 0x55e1da1fa140 2024-06-04T05:15:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046ac02740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f04601aa370 con 0x55e1da1f9820 2024-06-04T05:15:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046ac02740 10 client.0 _ll_register_callbacks cb 0x7f0458009360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046647e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 0x55e1da1fef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046547c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1fa140 0x55e1da1fe8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046547c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1fa140 0x55e1da1fe8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:43348/0 (socket says 172.21.15.70:43348) 2024-06-04T05:15:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046647e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 0x55e1da1fef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34614/0 (socket says 172.21.15.70:34614) 2024-06-04T05:15:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046547c640 1 -- 172.21.15.70:0/4235453997 learned_addr learned my addr 172.21.15.70:0/4235453997 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f0465c7d640 1 --2- 172.21.15.70:0/4235453997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da1f9820 0x55e1da20d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046547c640 1 -- 172.21.15.70:0/4235453997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da1f9820 msgr2=0x55e1da20d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046547c640 1 --2- 172.21.15.70:0/4235453997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da1f9820 0x55e1da20d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046547c640 1 -- 172.21.15.70:0/4235453997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 msgr2=0x55e1da1fef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046547c640 1 --2- 172.21.15.70:0/4235453997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 0x55e1da1fef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046547c640 1 -- 172.21.15.70:0/4235453997 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1da286c90 con 0x55e1da1fa140 2024-06-04T05:15:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046647e640 1 --2- 172.21.15.70:0/4235453997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 0x55e1da1fef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f046547c640 1 --2- 172.21.15.70:0/4235453997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1fa140 0x55e1da1fe8a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f045c00b3c0 tx=0x7f045c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=276e6c9ee7816eb5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.635+0000 7f04497fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:15:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.636+0000 7f04497fa640 1 -- 172.21.15.70:0/4235453997 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045c018cc0 con 0x55e1da1fa140 2024-06-04T05:15:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.636+0000 7f046ac02740 1 -- 172.21.15.70:0/4235453997 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0458036070 con 0x55e1da1fa140 2024-06-04T05:15:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.636+0000 7f04497fa640 1 -- 172.21.15.70:0/4235453997 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f045c027410 con 0x55e1da1fa140 2024-06-04T05:15:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.636+0000 7f04497fa640 1 -- 172.21.15.70:0/4235453997 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f045c018e60 con 0x55e1da1fa140 2024-06-04T05:15:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.636+0000 7f046ac02740 10 client.29018 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.636+0000 7f046ac02740 1 -- 172.21.15.70:0/4235453997 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55e1da287ed0 con 0x55e1da1fa140 2024-06-04T05:15:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.636+0000 7f04497fa640 1 -- 172.21.15.70:0/4235453997 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0458036070 con 0x55e1da1fa140 2024-06-04T05:15:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.636+0000 7f04497fa640 1 -- 172.21.15.70:0/4235453997 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f045c02e070 con 0x55e1da1fa140 2024-06-04T05:15:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.636+0000 7f046ac02740 10 client.29018 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.636+0000 7f046ac02740 10 client.29018 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.636+0000 7f046ac02740 20 client.29018 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.636+0000 7f046ac02740 1 --2- 172.21.15.70:0/4235453997 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e1da21d0a0 0x55e1da28a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.636+0000 7f046ac02740 4 client.29018 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:15:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.636+0000 7f046ac02740 1 -- 172.21.15.70:0/4235453997 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e1da167360 con 0x55e1da21d0a0 2024-06-04T05:15:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.637+0000 7f0465c7d640 1 --2- 172.21.15.70:0/4235453997 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e1da21d0a0 0x55e1da28a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.637+0000 7f04497fa640 10 client.29018 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.637+0000 7f0465c7d640 1 --2- 172.21.15.70:0/4235453997 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e1da21d0a0 0x55e1da28a670 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62b346a144f87b06 server_cookie=11c6626da7b7d6b in_seq=0 out_seq=0 2024-06-04T05:15:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:27 smithi112 ceph-mon[36453]: pgmap v573: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 2.5 MiB/s wr, 180 op/s 2024-06-04T05:15:27.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.721+0000 7f04497fa640 1 -- 172.21.15.70:0/4235453997 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55e1da167360 con 0x55e1da21d0a0 2024-06-04T05:15:27.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.721+0000 7f04497fa640 10 client.29018 handle_command_reply: tid=0 2024-06-04T05:15:27.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.721+0000 7f046ac02740 1 client.29018 shutdown 2024-06-04T05:15:27.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.722+0000 7f046ac02740 1 -- 172.21.15.70:0/4235453997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1fa140 msgr2=0x55e1da1fe8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:27.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.722+0000 7f046ac02740 1 --2- 172.21.15.70:0/4235453997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1fa140 0x55e1da1fe8a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f045c00b3c0 tx=0x7f045c005f90 comp rx=0 tx=0).stop 2024-06-04T05:15:27.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.722+0000 7f046ac02740 1 -- 172.21.15.70:0/4235453997 shutdown_connections 2024-06-04T05:15:27.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.722+0000 7f046ac02740 1 --2- 172.21.15.70:0/4235453997 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e1da21d0a0 0x55e1da28a670 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.722+0000 7f046ac02740 1 --2- 172.21.15.70:0/4235453997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1da1fa140 0x55e1da1fe8a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.722+0000 7f04497fa640 0 client.29018 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:27.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.722+0000 7f046ac02740 1 --2- 172.21.15.70:0/4235453997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1da204a00 0x55e1da1fef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.722+0000 7f046ac02740 1 --2- 172.21.15.70:0/4235453997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1da1f9820 0x55e1da20d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.722+0000 7f046ac02740 1 -- 172.21.15.70:0/4235453997 >> 172.21.15.70:0/4235453997 conn(0x55e1da1f5d30 msgr2=0x55e1da20f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:27.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.722+0000 7f046ac02740 1 -- 172.21.15.70:0/4235453997 shutdown_connections 2024-06-04T05:15:27.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.722+0000 7f046ac02740 1 -- 172.21.15.70:0/4235453997 wait complete. 2024-06-04T05:15:27.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.722+0000 7f046ac02740 20 client.29018 trim_cache size 0 max 16384 2024-06-04T05:15:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:15:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:15:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:15:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.723+0000 7f0467ee1640 1 -- 172.21.15.70:0/2864393700 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f044c0b2a10 msgr2=0x7f044c0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.723+0000 7f0467ee1640 1 --2- 172.21.15.70:0/2864393700 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f044c0b2a10 0x7f044c0b4ed0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0450021000 tx=0x7f0450022000 comp rx=0 tx=0).stop 2024-06-04T05:15:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.723+0000 7f0467ee1640 1 -- 172.21.15.70:0/2864393700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0460101980 msgr2=0x7f046019b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:27.727 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.723+0000 7f0467ee1640 1 --2- 172.21.15.70:0/2864393700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0460101980 0x7f046019b2b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0458000f80 tx=0x7f0458003ba0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.727 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.723+0000 7f0467ee1640 1 -- 172.21.15.70:0/2864393700 shutdown_connections 2024-06-04T05:15:27.727 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.723+0000 7f0467ee1640 1 --2- 172.21.15.70:0/2864393700 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f044c0b2a10 0x7f044c0b4ed0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.727 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.723+0000 7f0467ee1640 1 --2- 172.21.15.70:0/2864393700 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0460107940 0x7f046019fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.727 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.723+0000 7f0467ee1640 1 --2- 172.21.15.70:0/2864393700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0460102340 0x7f046019b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.727 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.723+0000 7f0467ee1640 1 --2- 172.21.15.70:0/2864393700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0460101980 0x7f046019b2b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:27.727 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.723+0000 7f0467ee1640 1 -- 172.21.15.70:0/2864393700 >> 172.21.15.70:0/2864393700 conn(0x7f0460078ad0 msgr2=0x7f04601045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:27.727 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.723+0000 7f0467ee1640 1 -- 172.21.15.70:0/2864393700 shutdown_connections 2024-06-04T05:15:27.727 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:27.723+0000 7f0467ee1640 1 -- 172.21.15.70:0/2864393700 wait complete. 2024-06-04T05:15:27.742 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '77470b0', total seconds elapsed: 35.2 2024-06-04T05:15:27.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:27 smithi098 ceph-mon[35569]: pgmap v573: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 2.5 MiB/s wr, 180 op/s 2024-06-04T05:15:27.816 INFO:tasks.workunit.client.0.smithi070.stdout: 10 36435 7.74 MB/sec execute 139 sec latency 68.888 ms 2024-06-04T05:15:28.549 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:15:28.816 INFO:tasks.workunit.client.0.smithi070.stdout: 10 36619 7.77 MB/sec execute 140 sec latency 50.722 ms 2024-06-04T05:15:28.897 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:15:29.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:29 smithi070 ceph-mon[32631]: pgmap v574: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 113 KiB/s wr, 22 op/s 2024-06-04T05:15:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:29 smithi112 ceph-mon[36453]: pgmap v574: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 113 KiB/s wr, 22 op/s 2024-06-04T05:15:29.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:29 smithi098 ceph-mon[35569]: pgmap v574: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 113 KiB/s wr, 22 op/s 2024-06-04T05:15:29.816 INFO:tasks.workunit.client.0.smithi070.stdout: 10 36796 7.79 MB/sec execute 141 sec latency 55.737 ms 2024-06-04T05:15:30.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.706+0000 7f01f7a4d700 1 -- 172.21.15.70:0/3214507558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01f0102df0 msgr2=0x7f01f010f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:30.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.706+0000 7f01f7a4d700 1 --2- 172.21.15.70:0/3214507558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01f0102df0 0x7f01f010f2e0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f01e800b0a0 tx=0x7f01e800b3b0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.706+0000 7f01f7a4d700 1 -- 172.21.15.70:0/3214507558 shutdown_connections 2024-06-04T05:15:30.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.706+0000 7f01f7a4d700 1 --2- 172.21.15.70:0/3214507558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01f0102df0 0x7f01f010f2e0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.706+0000 7f01f7a4d700 1 --2- 172.21.15.70:0/3214507558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01f0108510 0x7f01f01088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.706+0000 7f01f7a4d700 1 --2- 172.21.15.70:0/3214507558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01f0102430 0x7f01f01028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.706+0000 7f01f7a4d700 1 -- 172.21.15.70:0/3214507558 >> 172.21.15.70:0/3214507558 conn(0x7f01f00793b0 msgr2=0x7f01f00797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:30.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.706+0000 7f01f7a4d700 1 -- 172.21.15.70:0/3214507558 shutdown_connections 2024-06-04T05:15:30.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.706+0000 7f01f7a4d700 1 -- 172.21.15.70:0/3214507558 wait complete. 2024-06-04T05:15:30.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.707+0000 7f01f7a4d700 1 Processor -- start 2024-06-04T05:15:30.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.707+0000 7f01f7a4d700 1 -- start start 2024-06-04T05:15:30.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.707+0000 7f01f7a4d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01f0102430 0x7f01f01a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f7a4d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01f0102df0 0x7f01f01a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f7a4d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01f0108510 0x7f01f019bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f7a4d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f01f01113f0 con 0x7f01f0102430 2024-06-04T05:15:30.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f7a4d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f01f0111270 con 0x7f01f0102df0 2024-06-04T05:15:30.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f57e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01f0102430 0x7f01f01a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f7a4d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f01f0111570 con 0x7f01f0108510 2024-06-04T05:15:30.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f57e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01f0102430 0x7f01f01a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34640/0 (socket says 172.21.15.70:34640) 2024-06-04T05:15:30.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f57e9700 1 -- 172.21.15.70:0/3708445760 learned_addr learned my addr 172.21.15.70:0/3708445760 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:30.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f4fe8700 1 --2- 172.21.15.70:0/3708445760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01f0102df0 0x7f01f01a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f5fea700 1 --2- 172.21.15.70:0/3708445760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01f0108510 0x7f01f019bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f57e9700 1 -- 172.21.15.70:0/3708445760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01f0108510 msgr2=0x7f01f019bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:30.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f57e9700 1 --2- 172.21.15.70:0/3708445760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01f0108510 0x7f01f019bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f57e9700 1 -- 172.21.15.70:0/3708445760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01f0102df0 msgr2=0x7f01f01a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:30.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f57e9700 1 --2- 172.21.15.70:0/3708445760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01f0102df0 0x7f01f01a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f57e9700 1 -- 172.21.15.70:0/3708445760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f019c240 con 0x7f01f0102430 2024-06-04T05:15:30.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f5fea700 1 --2- 172.21.15.70:0/3708445760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01f0108510 0x7f01f019bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:30.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f4fe8700 1 --2- 172.21.15.70:0/3708445760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01f0102df0 0x7f01f01a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:30.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.708+0000 7f01f57e9700 1 --2- 172.21.15.70:0/3708445760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01f0102430 0x7f01f01a1920 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f01ec00e9a0 tx=0x7f01ec00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:30.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.709+0000 7f01e67fc700 1 -- 172.21.15.70:0/3708445760 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec00ca10 con 0x7f01f0102430 2024-06-04T05:15:30.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.709+0000 7f01e67fc700 1 -- 172.21.15.70:0/3708445760 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f01ec00cbb0 con 0x7f01f0102430 2024-06-04T05:15:30.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.709+0000 7f01f7a4d700 1 -- 172.21.15.70:0/3708445760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01f019c4d0 con 0x7f01f0102430 2024-06-04T05:15:30.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.709+0000 7f01f7a4d700 1 -- 172.21.15.70:0/3708445760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01f01a87f0 con 0x7f01f0102430 2024-06-04T05:15:30.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.709+0000 7f01e67fc700 1 -- 172.21.15.70:0/3708445760 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec018680 con 0x7f01f0102430 2024-06-04T05:15:30.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.712+0000 7f01e67fc700 1 -- 172.21.15.70:0/3708445760 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f01ec010430 con 0x7f01f0102430 2024-06-04T05:15:30.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.713+0000 7f02022e2b80 1 Processor -- start 2024-06-04T05:15:30.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.713+0000 7f01e67fc700 1 --2- 172.21.15.70:0/3708445760 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01dc0b2a70 0x7f01dc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f02022e2b80 1 -- start start 2024-06-04T05:15:30.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f01e67fc700 1 -- 172.21.15.70:0/3708445760 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f01ec0e2620 con 0x7f01f0102430 2024-06-04T05:15:30.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f02022e2b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8d7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f02022e2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8d20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f02022e2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8d2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f02022e2b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ca7d827d70 con 0x55ca7d8d7a70 2024-06-04T05:15:30.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f02022e2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ca7d817910 con 0x55ca7d8d8420 2024-06-04T05:15:30.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f02022e2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ca7d81b950 con 0x55ca7d8d26d0 2024-06-04T05:15:30.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f01f5fea700 1 --2- 172.21.15.70:0/3708445760 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01dc0b2a70 0x7f01dc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f01f4fe8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8d20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f01f57e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8d7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f01f57e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8d7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34644/0 (socket says 172.21.15.70:34644) 2024-06-04T05:15:30.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f01f5fea700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8d2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f01f57e9700 1 -- 172.21.15.70:0/3601097275 learned_addr learned my addr 172.21.15.70:0/3601097275 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:30.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f01f57e9700 1 -- 172.21.15.70:0/3601097275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d26d0 msgr2=0x55ca7d8d2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:30.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f01f57e9700 1 --2- 172.21.15.70:0/3601097275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8d2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f01f57e9700 1 -- 172.21.15.70:0/3601097275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 msgr2=0x55ca7d8d20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:30.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f01f57e9700 1 --2- 172.21.15.70:0/3601097275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8d20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.714+0000 7f01f57e9700 1 -- 172.21.15.70:0/3601097275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca7d7c0fb0 con 0x55ca7d8d7a70 2024-06-04T05:15:30.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.715+0000 7f01f5fea700 1 --2- 172.21.15.70:0/3601097275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8d2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:30.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.715+0000 7f01f5fea700 1 --2- 172.21.15.70:0/3708445760 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01dc0b2a70 0x7f01dc0b4f30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f01e8009e60 tx=0x7f01e8009590 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:30.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.715+0000 7f01f57e9700 1 --2- 172.21.15.70:0/3601097275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8d7e50 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f01ec0a9860 tx=0x7f01ec0a9170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f58e8a376bf2288f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:30.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.715+0000 7f01db7fe700 1 -- 172.21.15.70:0/3601097275 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec0adc80 con 0x55ca7d8d7a70 2024-06-04T05:15:30.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.715+0000 7f01db7fe700 1 -- 172.21.15.70:0/3601097275 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f01ec010e50 con 0x55ca7d8d7a70 2024-06-04T05:15:30.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.716+0000 7f01db7fe700 1 -- 172.21.15.70:0/3601097275 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec0a9d90 con 0x55ca7d8d7a70 2024-06-04T05:15:30.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.716+0000 7f02022e2b80 1 -- 172.21.15.70:0/3601097275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d7a70 msgr2=0x55ca7d8d7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:30.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.716+0000 7f02022e2b80 1 --2- 172.21.15.70:0/3601097275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8d7e50 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f01ec0a9860 tx=0x7f01ec0a9170 comp rx=0 tx=0).stop 2024-06-04T05:15:30.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.716+0000 7f02022e2b80 1 -- 172.21.15.70:0/3601097275 shutdown_connections 2024-06-04T05:15:30.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.716+0000 7f02022e2b80 1 --2- 172.21.15.70:0/3601097275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8d2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.716+0000 7f02022e2b80 1 --2- 172.21.15.70:0/3601097275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8d7e50 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.716+0000 7f02022e2b80 1 --2- 172.21.15.70:0/3601097275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8d20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.716+0000 7f02022e2b80 1 -- 172.21.15.70:0/3601097275 >> 172.21.15.70:0/3601097275 conn(0x55ca7d7f5240 msgr2=0x55ca7d8c43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:30.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.716+0000 7f02022e2b80 1 -- 172.21.15.70:0/3601097275 shutdown_connections 2024-06-04T05:15:30.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.716+0000 7f02022e2b80 1 -- 172.21.15.70:0/3601097275 wait complete. 2024-06-04T05:15:30.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.717+0000 7f02022e2b80 1 -- start start 2024-06-04T05:15:30.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.717+0000 7f02022e2b80 1 Processor -- start 2024-06-04T05:15:30.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.717+0000 7f02022e2b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8de150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.717+0000 7f02022e2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8db4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.717+0000 7f01f57e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8de150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.717+0000 7f01f57e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8de150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34660/0 (socket says 172.21.15.70:34660) 2024-06-04T05:15:30.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.717+0000 7f02022e2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8db9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.717+0000 7f02022e2b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ca7d71dfa0 con 0x55ca7d8d26d0 2024-06-04T05:15:30.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.717+0000 7f02022e2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ca7d77fda0 con 0x55ca7d8d8420 2024-06-04T05:15:30.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.717+0000 7f02022e2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ca7d8deee0 con 0x55ca7d8d7a70 2024-06-04T05:15:30.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.718+0000 7f02022e2b80 10 client.0 _ll_register_callbacks cb 0x55ca7d7b2950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:30.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.718+0000 7f01f4fe8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8db4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.718+0000 7f01f4fe8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8db4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41400/0 (socket says 172.21.15.70:41400) 2024-06-04T05:15:30.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.718+0000 7f01f4fe8700 1 -- 172.21.15.70:0/1883450272 learned_addr learned my addr 172.21.15.70:0/1883450272 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:30.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.718+0000 7f01f5fea700 1 --2- 172.21.15.70:0/1883450272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8db9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.718+0000 7f01f4fe8700 1 -- 172.21.15.70:0/1883450272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 msgr2=0x55ca7d8db9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:30.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.718+0000 7f01f4fe8700 1 --2- 172.21.15.70:0/1883450272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8db9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.718+0000 7f01f4fe8700 1 -- 172.21.15.70:0/1883450272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 msgr2=0x55ca7d8de150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:30.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.718+0000 7f01f4fe8700 1 --2- 172.21.15.70:0/1883450272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8de150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.718+0000 7f01f4fe8700 1 -- 172.21.15.70:0/1883450272 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca7d7678d0 con 0x55ca7d8d7a70 2024-06-04T05:15:30.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.718+0000 7f01f4fe8700 1 --2- 172.21.15.70:0/1883450272 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8db4a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f01e000b3c0 tx=0x7f01e0009aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ade3b541c9b5b8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:30.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.718+0000 7f01d8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:15:30.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.718+0000 7f01d8ff9700 1 -- 172.21.15.70:0/1883450272 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e001b030 con 0x55ca7d8d7a70 2024-06-04T05:15:30.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.718+0000 7f02022e2b80 1 -- 172.21.15.70:0/1883450272 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01ec010e50 con 0x55ca7d8d7a70 2024-06-04T05:15:30.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.718+0000 7f01d8ff9700 1 -- 172.21.15.70:0/1883450272 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f01e0004830 con 0x55ca7d8d7a70 2024-06-04T05:15:30.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.719+0000 7f01d8ff9700 1 -- 172.21.15.70:0/1883450272 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f01e0025050 con 0x55ca7d8d7a70 2024-06-04T05:15:30.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.719+0000 7f02022e2b80 10 client.29134 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:30.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.719+0000 7f02022e2b80 1 -- 172.21.15.70:0/1883450272 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55ca7d7d1760 con 0x55ca7d8d7a70 2024-06-04T05:15:30.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.719+0000 7f01d8ff9700 1 -- 172.21.15.70:0/1883450272 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f01ec010e50 con 0x55ca7d8d7a70 2024-06-04T05:15:30.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.719+0000 7f01d8ff9700 1 -- 172.21.15.70:0/1883450272 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f01e0020890 con 0x55ca7d8d7a70 2024-06-04T05:15:30.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.719+0000 7f02022e2b80 10 client.29134 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:30.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.719+0000 7f02022e2b80 10 client.29134 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:15:30.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.719+0000 7f02022e2b80 20 client.29134 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:30.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.719+0000 7f02022e2b80 1 --2- 172.21.15.70:0/1883450272 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ca7d8ebee0 0x55ca7d8ee360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.719+0000 7f02022e2b80 4 client.29134 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:15:30.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.719+0000 7f02022e2b80 1 -- 172.21.15.70:0/1883450272 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca7d829cc0 con 0x55ca7d8ebee0 2024-06-04T05:15:30.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.719+0000 7f01f57e9700 1 --2- 172.21.15.70:0/1883450272 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ca7d8ebee0 0x55ca7d8ee360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.720+0000 7f01d8ff9700 10 client.29134 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:15:30.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.720+0000 7f01f57e9700 1 --2- 172.21.15.70:0/1883450272 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ca7d8ebee0 0x55ca7d8ee360 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=25a1b3bf9fb99dc1 server_cookie=91fc9d07f762dd9a in_seq=0 out_seq=0 2024-06-04T05:15:30.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.720+0000 7f01d8ff9700 1 -- 172.21.15.70:0/1883450272 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ca7d829cc0 con 0x55ca7d8ebee0 2024-06-04T05:15:30.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.721+0000 7f01d8ff9700 10 client.29134 handle_command_reply: tid=0 2024-06-04T05:15:30.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.721+0000 7f02022e2b80 1 client.29134 shutdown 2024-06-04T05:15:30.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.721+0000 7f02022e2b80 1 -- 172.21.15.70:0/1883450272 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 msgr2=0x55ca7d8db4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:30.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.721+0000 7f02022e2b80 1 --2- 172.21.15.70:0/1883450272 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8db4a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f01e000b3c0 tx=0x7f01e0009aa0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.721+0000 7f02022e2b80 1 -- 172.21.15.70:0/1883450272 shutdown_connections 2024-06-04T05:15:30.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.721+0000 7f02022e2b80 1 --2- 172.21.15.70:0/1883450272 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ca7d8ebee0 0x55ca7d8ee360 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.721+0000 7f02022e2b80 1 --2- 172.21.15.70:0/1883450272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8db9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.721+0000 7f02022e2b80 1 --2- 172.21.15.70:0/1883450272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8de150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.721+0000 7f02022e2b80 1 --2- 172.21.15.70:0/1883450272 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8db4a0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.721+0000 7f02022e2b80 1 -- 172.21.15.70:0/1883450272 >> 172.21.15.70:0/1883450272 conn(0x55ca7d7f5240 msgr2=0x55ca7d8c3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:30.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.721+0000 7f01d8ff9700 0 client.29134 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:15:30.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.721+0000 7f02022e2b80 1 -- 172.21.15.70:0/1883450272 shutdown_connections 2024-06-04T05:15:30.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.721+0000 7f02022e2b80 1 -- 172.21.15.70:0/1883450272 wait complete. 2024-06-04T05:15:30.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.721+0000 7f02022e2b80 20 client.29134 trim_cache size 0 max 16384 2024-06-04T05:15:30.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.743+0000 7f02022e2b80 1 Processor -- start 2024-06-04T05:15:30.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.743+0000 7f02022e2b80 1 -- start start 2024-06-04T05:15:30.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.743+0000 7f02022e2b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8c4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.743+0000 7f02022e2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8d9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.744+0000 7f02022e2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8da2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.744+0000 7f02022e2b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f01d400d6a0 con 0x55ca7d8d26d0 2024-06-04T05:15:30.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.744+0000 7f02022e2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f01d400f240 con 0x55ca7d8d8420 2024-06-04T05:15:30.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.744+0000 7f02022e2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f01d400ddd0 con 0x55ca7d8d7a70 2024-06-04T05:15:30.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.744+0000 7f01f57e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.744+0000 7f01f57e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8c4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34676/0 (socket says 172.21.15.70:34676) 2024-06-04T05:15:30.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.744+0000 7f01f57e9700 1 -- 172.21.15.70:0/2269524609 learned_addr learned my addr 172.21.15.70:0/2269524609 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:30.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.744+0000 7f01f4fe8700 1 --2- 172.21.15.70:0/2269524609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8d9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.744+0000 7f01f5fea700 1 --2- 172.21.15.70:0/2269524609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8da2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.744+0000 7f01f57e9700 1 -- 172.21.15.70:0/2269524609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 msgr2=0x55ca7d8d9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:30.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.744+0000 7f01f57e9700 1 --2- 172.21.15.70:0/2269524609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8d9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.744+0000 7f01f57e9700 1 -- 172.21.15.70:0/2269524609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 msgr2=0x55ca7d8da2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:30.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.744+0000 7f01f57e9700 1 --2- 172.21.15.70:0/2269524609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8da2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.744+0000 7f01f57e9700 1 -- 172.21.15.70:0/2269524609 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca7d8bc610 con 0x55ca7d8d26d0 2024-06-04T05:15:30.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.744+0000 7f01f4fe8700 1 --2- 172.21.15.70:0/2269524609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8d9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:30.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.744+0000 7f01f57e9700 1 --2- 172.21.15.70:0/2269524609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8c4270 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f01ec0a9860 tx=0x7f01ec0a9170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6626be99dabd55e5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:30.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.745+0000 7f01daffd700 1 -- 172.21.15.70:0/2269524609 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec0adc80 con 0x55ca7d8d26d0 2024-06-04T05:15:30.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.745+0000 7f01daffd700 1 -- 172.21.15.70:0/2269524609 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f01ec010e50 con 0x55ca7d8d26d0 2024-06-04T05:15:30.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.745+0000 7f01daffd700 1 -- 172.21.15.70:0/2269524609 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec0a9d70 con 0x55ca7d8d26d0 2024-06-04T05:15:30.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.745+0000 7f02022e2b80 1 -- 172.21.15.70:0/2269524609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 msgr2=0x55ca7d8c4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:30.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.745+0000 7f02022e2b80 1 --2- 172.21.15.70:0/2269524609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8c4270 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f01ec0a9860 tx=0x7f01ec0a9170 comp rx=0 tx=0).stop 2024-06-04T05:15:30.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.746+0000 7f02022e2b80 1 -- 172.21.15.70:0/2269524609 shutdown_connections 2024-06-04T05:15:30.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.746+0000 7f02022e2b80 1 --2- 172.21.15.70:0/2269524609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8da2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.746+0000 7f02022e2b80 1 --2- 172.21.15.70:0/2269524609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8c4270 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.746+0000 7f02022e2b80 1 --2- 172.21.15.70:0/2269524609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8d9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.746+0000 7f02022e2b80 1 -- 172.21.15.70:0/2269524609 >> 172.21.15.70:0/2269524609 conn(0x55ca7d7f5240 msgr2=0x55ca7d8db460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:30.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.746+0000 7f02022e2b80 1 -- 172.21.15.70:0/2269524609 shutdown_connections 2024-06-04T05:15:30.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.746+0000 7f02022e2b80 1 -- 172.21.15.70:0/2269524609 wait complete. 2024-06-04T05:15:30.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.747+0000 7f02022e2b80 1 -- start start 2024-06-04T05:15:30.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.747+0000 7f02022e2b80 1 Processor -- start 2024-06-04T05:15:30.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.747+0000 7f02022e2b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8e41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.747+0000 7f02022e2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8e4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.747+0000 7f02022e2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.747+0000 7f02022e2b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f01d400eb70 con 0x55ca7d8d26d0 2024-06-04T05:15:30.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.747+0000 7f02022e2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f01d400e4a0 con 0x55ca7d8d8420 2024-06-04T05:15:30.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.747+0000 7f02022e2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f01d400b660 con 0x55ca7d8d7a70 2024-06-04T05:15:30.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.747+0000 7f02022e2b80 10 client.0 _ll_register_callbacks cb 0x55ca7d7b2950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:30.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.747+0000 7f01f5fea700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.747+0000 7f01f4fe8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8e4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.748+0000 7f01f5fea700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8f2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:43416/0 (socket says 172.21.15.70:43416) 2024-06-04T05:15:30.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.748+0000 7f01f5fea700 1 -- 172.21.15.70:0/1840283073 learned_addr learned my addr 172.21.15.70:0/1840283073 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:30.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.748+0000 7f01f57e9700 1 --2- 172.21.15.70:0/1840283073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8e41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.748+0000 7f01f5fea700 1 -- 172.21.15.70:0/1840283073 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 msgr2=0x55ca7d8e4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:30.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.748+0000 7f01f5fea700 1 --2- 172.21.15.70:0/1840283073 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8e4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.748+0000 7f01f5fea700 1 -- 172.21.15.70:0/1840283073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 msgr2=0x55ca7d8e41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:30.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.748+0000 7f01f5fea700 1 --2- 172.21.15.70:0/1840283073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8e41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.748+0000 7f01f5fea700 1 -- 172.21.15.70:0/1840283073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca7d8f31a0 con 0x55ca7d8d8420 2024-06-04T05:15:30.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.748+0000 7f01f4fe8700 1 --2- 172.21.15.70:0/1840283073 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8e4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:30.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.748+0000 7f01d8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:15:30.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.748+0000 7f01f5fea700 1 --2- 172.21.15.70:0/1840283073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8f2c60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f01e800b0a0 tx=0x7f01e800b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0b2edc303663565 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:30.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.748+0000 7f01d8ff9700 1 -- 172.21.15.70:0/1840283073 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e802a660 con 0x55ca7d8d8420 2024-06-04T05:15:30.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.749+0000 7f01d8ff9700 1 -- 172.21.15.70:0/1840283073 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f01e802a800 con 0x55ca7d8d8420 2024-06-04T05:15:30.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.749+0000 7f02022e2b80 1 -- 172.21.15.70:0/1840283073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01ec010e50 con 0x55ca7d8d8420 2024-06-04T05:15:30.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.749+0000 7f01d8ff9700 1 -- 172.21.15.70:0/1840283073 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f01e802aa00 con 0x55ca7d8d8420 2024-06-04T05:15:30.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.749+0000 7f02022e2b80 10 client.29033 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:30.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.749+0000 7f02022e2b80 1 -- 172.21.15.70:0/1840283073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55ca7d9562c0 con 0x55ca7d8d8420 2024-06-04T05:15:30.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.749+0000 7f01d8ff9700 1 -- 172.21.15.70:0/1840283073 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f01e802aba0 con 0x55ca7d8d8420 2024-06-04T05:15:30.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.750+0000 7f01d8ff9700 1 -- 172.21.15.70:0/1840283073 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f01e8011070 con 0x55ca7d8d8420 2024-06-04T05:15:30.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.750+0000 7f02022e2b80 10 client.29033 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:30.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.750+0000 7f02022e2b80 10 client.29033 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:15:30.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.750+0000 7f02022e2b80 20 client.29033 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:30.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.750+0000 7f02022e2b80 1 --2- 172.21.15.70:0/1840283073 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ca7d9569d0 0x55ca7d958df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:30.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.750+0000 7f02022e2b80 4 client.29033 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:15:30.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.750+0000 7f02022e2b80 1 -- 172.21.15.70:0/1840283073 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f01ec010e50 con 0x55ca7d9569d0 2024-06-04T05:15:30.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.750+0000 7f01f57e9700 1 --2- 172.21.15.70:0/1840283073 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ca7d9569d0 0x55ca7d958df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:30.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.751+0000 7f01f57e9700 1 --2- 172.21.15.70:0/1840283073 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ca7d9569d0 0x55ca7d958df0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=66096c20d10f93cd server_cookie=e1855fbe708fa062 in_seq=0 out_seq=0 2024-06-04T05:15:30.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.751+0000 7f01d8ff9700 10 client.29033 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:15:30.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.751+0000 7f01d8ff9700 1 -- 172.21.15.70:0/1840283073 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f01ec010e50 con 0x55ca7d9569d0 2024-06-04T05:15:30.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.751+0000 7f01d8ff9700 10 client.29033 handle_command_reply: tid=0 2024-06-04T05:15:30.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.751+0000 7f02022e2b80 1 client.29033 shutdown 2024-06-04T05:15:30.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.751+0000 7f02022e2b80 1 -- 172.21.15.70:0/1840283073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 msgr2=0x55ca7d8f2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:30.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.751+0000 7f02022e2b80 1 --2- 172.21.15.70:0/1840283073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8f2c60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f01e800b0a0 tx=0x7f01e800b3b0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.751+0000 7f02022e2b80 1 -- 172.21.15.70:0/1840283073 shutdown_connections 2024-06-04T05:15:30.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.751+0000 7f02022e2b80 1 --2- 172.21.15.70:0/1840283073 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ca7d9569d0 0x55ca7d958df0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.751+0000 7f02022e2b80 1 --2- 172.21.15.70:0/1840283073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca7d8d8420 0x55ca7d8f2c60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.751+0000 7f02022e2b80 1 --2- 172.21.15.70:0/1840283073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca7d8d26d0 0x55ca7d8e41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.751+0000 7f02022e2b80 1 --2- 172.21.15.70:0/1840283073 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ca7d8d7a70 0x55ca7d8e4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.752+0000 7f02022e2b80 1 -- 172.21.15.70:0/1840283073 >> 172.21.15.70:0/1840283073 conn(0x55ca7d7f5240 msgr2=0x55ca7d8d6590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:30.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.752+0000 7f01d8ff9700 0 client.29033 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:15:30.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.752+0000 7f02022e2b80 1 -- 172.21.15.70:0/1840283073 shutdown_connections 2024-06-04T05:15:30.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.752+0000 7f02022e2b80 1 -- 172.21.15.70:0/1840283073 wait complete. 2024-06-04T05:15:30.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.752+0000 7f02022e2b80 20 client.29033 trim_cache size 0 max 16384 2024-06-04T05:15:30.756 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:15:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.753+0000 7f01f7a4d700 1 -- 172.21.15.70:0/3708445760 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01dc0b2a70 msgr2=0x7f01dc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.753+0000 7f01f7a4d700 1 --2- 172.21.15.70:0/3708445760 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01dc0b2a70 0x7f01dc0b4f30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f01e8009e60 tx=0x7f01e8009590 comp rx=0 tx=0).stop 2024-06-04T05:15:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.753+0000 7f01f7a4d700 1 -- 172.21.15.70:0/3708445760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01f0102430 msgr2=0x7f01f01a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.753+0000 7f01f7a4d700 1 --2- 172.21.15.70:0/3708445760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01f0102430 0x7f01f01a1920 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f01ec00e9a0 tx=0x7f01ec00ecb0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.753+0000 7f01f7a4d700 1 -- 172.21.15.70:0/3708445760 shutdown_connections 2024-06-04T05:15:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.753+0000 7f01f7a4d700 1 --2- 172.21.15.70:0/3708445760 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01dc0b2a70 0x7f01dc0b4f30 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.753+0000 7f01f7a4d700 1 --2- 172.21.15.70:0/3708445760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01f0108510 0x7f01f019bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.753+0000 7f01f7a4d700 1 --2- 172.21.15.70:0/3708445760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01f0102430 0x7f01f01a1920 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.753+0000 7f01f7a4d700 1 --2- 172.21.15.70:0/3708445760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01f0102df0 0x7f01f01a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:30.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.753+0000 7f01f7a4d700 1 -- 172.21.15.70:0/3708445760 >> 172.21.15.70:0/3708445760 conn(0x7f01f00793b0 msgr2=0x7f01f01005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:30.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.753+0000 7f01f7a4d700 1 -- 172.21.15.70:0/3708445760 shutdown_connections 2024-06-04T05:15:30.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:30.753+0000 7f01f7a4d700 1 -- 172.21.15.70:0/3708445760 wait complete. 2024-06-04T05:15:30.816 INFO:tasks.workunit.client.0.smithi070.stdout: 10 36947 7.80 MB/sec execute 142 sec latency 91.415 ms 2024-06-04T05:15:31.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:31 smithi070 ceph-mon[32631]: pgmap v575: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 113 KiB/s wr, 22 op/s 2024-06-04T05:15:31.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:31 smithi112 ceph-mon[36453]: pgmap v575: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 113 KiB/s wr, 22 op/s 2024-06-04T05:15:31.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:31 smithi098 ceph-mon[35569]: pgmap v575: 97 pgs: 97 active+clean; 625 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 113 KiB/s wr, 22 op/s 2024-06-04T05:15:31.568 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:15:31.816 INFO:tasks.workunit.client.0.smithi070.stdout: 10 37106 7.81 MB/sec execute 143 sec latency 59.289 ms 2024-06-04T05:15:31.927 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:15:32.816 INFO:tasks.workunit.client.0.smithi070.stdout: 10 37277 7.79 MB/sec execute 144 sec latency 56.874 ms 2024-06-04T05:15:33.387 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:33 smithi070 ceph-mon[32631]: pgmap v576: 97 pgs: 97 active+clean; 640 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 2.2 MiB/s wr, 147 op/s 2024-06-04T05:15:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:33 smithi112 ceph-mon[36453]: pgmap v576: 97 pgs: 97 active+clean; 640 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 2.2 MiB/s wr, 147 op/s 2024-06-04T05:15:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:33 smithi098 ceph-mon[35569]: pgmap v576: 97 pgs: 97 active+clean; 640 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 2.2 MiB/s wr, 147 op/s 2024-06-04T05:15:33.816 INFO:tasks.workunit.client.0.smithi070.stdout: 10 37433 7.78 MB/sec execute 145 sec latency 69.047 ms 2024-06-04T05:15:34.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.038+0000 7f08dab03700 1 -- 172.21.15.70:0/1433587726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08d41088f0 msgr2=0x7f08d4110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:34.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.038+0000 7f08dab03700 1 --2- 172.21.15.70:0/1433587726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08d41088f0 0x7f08d4110470 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f08cc00b0d0 tx=0x7f08cc00b3e0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.038+0000 7f08dab03700 1 -- 172.21.15.70:0/1433587726 shutdown_connections 2024-06-04T05:15:34.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.038+0000 7f08dab03700 1 --2- 172.21.15.70:0/1433587726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08d41088f0 0x7f08d4110470 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.038+0000 7f08dab03700 1 --2- 172.21.15.70:0/1433587726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d41037f0 0x7f08d4103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.038+0000 7f08dab03700 1 --2- 172.21.15.70:0/1433587726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08d41041a0 0x7f08d41081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.038+0000 7f08dab03700 1 -- 172.21.15.70:0/1433587726 >> 172.21.15.70:0/1433587726 conn(0x7f08d407a6a0 msgr2=0x7f08d407aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:34.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.038+0000 7f08dab03700 1 -- 172.21.15.70:0/1433587726 shutdown_connections 2024-06-04T05:15:34.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.038+0000 7f08dab03700 1 -- 172.21.15.70:0/1433587726 wait complete. 2024-06-04T05:15:34.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.039+0000 7f08dab03700 1 Processor -- start 2024-06-04T05:15:34.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.039+0000 7f08dab03700 1 -- start start 2024-06-04T05:15:34.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.039+0000 7f08dab03700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08d41037f0 0x7f08d4075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.039+0000 7f08dab03700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d41041a0 0x7f08d4075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.039+0000 7f08dab03700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08d41088f0 0x7f08d40762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.039+0000 7f08dab03700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08d41135c0 con 0x7f08d41037f0 2024-06-04T05:15:34.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.039+0000 7f08dab03700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08d4113440 con 0x7f08d41041a0 2024-06-04T05:15:34.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.039+0000 7f08dab03700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f08d4113740 con 0x7f08d41088f0 2024-06-04T05:15:34.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.039+0000 7f08d889f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08d41037f0 0x7f08d4075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:34.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.040+0000 7f08d3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d41041a0 0x7f08d4075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:34.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.040+0000 7f08d889f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08d41037f0 0x7f08d4075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34692/0 (socket says 172.21.15.70:34692) 2024-06-04T05:15:34.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.040+0000 7f08d889f700 1 -- 172.21.15.70:0/2861004518 learned_addr learned my addr 172.21.15.70:0/2861004518 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:34.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.040+0000 7f08d3fff700 1 -- 172.21.15.70:0/2861004518 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08d41088f0 msgr2=0x7f08d40762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:34.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.040+0000 7f08d3fff700 1 --2- 172.21.15.70:0/2861004518 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08d41088f0 0x7f08d40762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.040+0000 7f08d3fff700 1 -- 172.21.15.70:0/2861004518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08d41037f0 msgr2=0x7f08d4075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:34.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.040+0000 7f08d3fff700 1 --2- 172.21.15.70:0/2861004518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08d41037f0 0x7f08d4075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.040+0000 7f08d3fff700 1 -- 172.21.15.70:0/2861004518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d41a7fc0 con 0x7f08d41041a0 2024-06-04T05:15:34.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.040+0000 7f08d889f700 1 --2- 172.21.15.70:0/2861004518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08d41037f0 0x7f08d4075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:34.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.040+0000 7f08d3fff700 1 --2- 172.21.15.70:0/2861004518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d41041a0 0x7f08d4075d80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f08c000d950 tx=0x7f08c000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:34.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.040+0000 7f08d1ffb700 1 -- 172.21.15.70:0/2861004518 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0017070 con 0x7f08d41041a0 2024-06-04T05:15:34.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.040+0000 7f08dab03700 1 -- 172.21.15.70:0/2861004518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08d41a82b0 con 0x7f08d41041a0 2024-06-04T05:15:34.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.041+0000 7f08d1ffb700 1 -- 172.21.15.70:0/2861004518 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f08c0009dc0 con 0x7f08d41041a0 2024-06-04T05:15:34.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.041+0000 7f08d1ffb700 1 -- 172.21.15.70:0/2861004518 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0010cf0 con 0x7f08d41041a0 2024-06-04T05:15:34.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.041+0000 7f08dab03700 1 -- 172.21.15.70:0/2861004518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08d41a8840 con 0x7f08d41041a0 2024-06-04T05:15:34.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.044+0000 7f08d1ffb700 1 -- 172.21.15.70:0/2861004518 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f08c001a070 con 0x7f08d41041a0 2024-06-04T05:15:34.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.045+0000 7f08d1ffb700 1 --2- 172.21.15.70:0/2861004518 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08c40b2a70 0x7f08c40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.045+0000 7f08d889f700 1 --2- 172.21.15.70:0/2861004518 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08c40b2a70 0x7f08c40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:34.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.045+0000 7f08d1ffb700 1 -- 172.21.15.70:0/2861004518 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f08c00e1e90 con 0x7f08d41041a0 2024-06-04T05:15:34.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.046+0000 7f08d889f700 1 --2- 172.21.15.70:0/2861004518 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08c40b2a70 0x7f08c40b4f30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f08c80104e0 tx=0x7f08c8007480 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:34.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.046+0000 7f08e5398b80 1 Processor -- start 2024-06-04T05:15:34.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.046+0000 7f08e5398b80 1 -- start start 2024-06-04T05:15:34.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.046+0000 7f08e5398b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d2860 0x564cc73d2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.047+0000 7f08e5398b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 0x564cc73d6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.047+0000 7f08e5398b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73d6f60 0x564cc73d73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.047+0000 7f08d889f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d2860 0x564cc73d2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:34.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.047+0000 7f08e5398b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564cc7331c90 con 0x564cc73d6f60 2024-06-04T05:15:34.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.047+0000 7f08e5398b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564cc7321830 con 0x564cc73d3210 2024-06-04T05:15:34.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.047+0000 7f08e5398b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564cc7325870 con 0x564cc73d2860 2024-06-04T05:15:34.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.047+0000 7f08d889f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d2860 0x564cc73d2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41420/0 (socket says 172.21.15.70:41420) 2024-06-04T05:15:34.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.047+0000 7f08d90a0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73d6f60 0x564cc73d73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:34.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.047+0000 7f08d889f700 1 -- 172.21.15.70:0/2397006886 learned_addr learned my addr 172.21.15.70:0/2397006886 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:34.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.047+0000 7f08d3fff700 1 --2- 172.21.15.70:0/2397006886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 0x564cc73d6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:34.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.047+0000 7f08d3fff700 1 -- 172.21.15.70:0/2397006886 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d2860 msgr2=0x564cc73d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:34.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.047+0000 7f08d3fff700 1 --2- 172.21.15.70:0/2397006886 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d2860 0x564cc73d2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.047+0000 7f08d3fff700 1 -- 172.21.15.70:0/2397006886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73d6f60 msgr2=0x564cc73d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:34.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.047+0000 7f08d3fff700 1 --2- 172.21.15.70:0/2397006886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73d6f60 0x564cc73d73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.047+0000 7f08d3fff700 1 -- 172.21.15.70:0/2397006886 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cc72718d0 con 0x564cc73d3210 2024-06-04T05:15:34.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.047+0000 7f08d90a0700 1 --2- 172.21.15.70:0/2397006886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73d6f60 0x564cc73d73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:34.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.047+0000 7f08d3fff700 1 --2- 172.21.15.70:0/2397006886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 0x564cc73d6940 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f08c00df5b0 tx=0x7f08c00a8bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d5a506ef344cf51 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:34.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.048+0000 7f08beffd700 1 -- 172.21.15.70:0/2397006886 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0021070 con 0x564cc73d3210 2024-06-04T05:15:34.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.048+0000 7f08beffd700 1 -- 172.21.15.70:0/2397006886 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f08c00a9140 con 0x564cc73d3210 2024-06-04T05:15:34.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.048+0000 7f08beffd700 1 -- 172.21.15.70:0/2397006886 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c00a9600 con 0x564cc73d3210 2024-06-04T05:15:34.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.049+0000 7f08e5398b80 1 -- 172.21.15.70:0/2397006886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 msgr2=0x564cc73d6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:34.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.049+0000 7f08e5398b80 1 --2- 172.21.15.70:0/2397006886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 0x564cc73d6940 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f08c00df5b0 tx=0x7f08c00a8bb0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.049+0000 7f08e5398b80 1 -- 172.21.15.70:0/2397006886 shutdown_connections 2024-06-04T05:15:34.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.049+0000 7f08e5398b80 1 --2- 172.21.15.70:0/2397006886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73d6f60 0x564cc73d73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.049+0000 7f08e5398b80 1 --2- 172.21.15.70:0/2397006886 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d2860 0x564cc73d2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.049+0000 7f08e5398b80 1 --2- 172.21.15.70:0/2397006886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 0x564cc73d6940 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.049+0000 7f08e5398b80 1 -- 172.21.15.70:0/2397006886 >> 172.21.15.70:0/2397006886 conn(0x564cc7300080 msgr2=0x564cc73cce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:34.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.049+0000 7f08e5398b80 1 -- 172.21.15.70:0/2397006886 shutdown_connections 2024-06-04T05:15:34.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.049+0000 7f08e5398b80 1 -- 172.21.15.70:0/2397006886 wait complete. 2024-06-04T05:15:34.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.049+0000 7f08e5398b80 1 -- start start 2024-06-04T05:15:34.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.049+0000 7f08e5398b80 1 Processor -- start 2024-06-04T05:15:34.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08e5398b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73d2860 0x564cc73de520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08e5398b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d3210 0x564cc73db820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08e5398b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d6f60 0x564cc73dbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08e5398b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564cc7227fa0 con 0x564cc73d2860 2024-06-04T05:15:34.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08e5398b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564cc72c37c0 con 0x564cc73d6f60 2024-06-04T05:15:34.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08e5398b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564cc73df120 con 0x564cc73d3210 2024-06-04T05:15:34.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08e5398b80 10 client.0 _ll_register_callbacks cb 0x564cc72bc920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:34.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08d889f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73d2860 0x564cc73de520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:34.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08d3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d3210 0x564cc73db820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:34.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08d889f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73d2860 0x564cc73de520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34714/0 (socket says 172.21.15.70:34714) 2024-06-04T05:15:34.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08d3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d3210 0x564cc73db820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41434/0 (socket says 172.21.15.70:41434) 2024-06-04T05:15:34.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08d889f700 1 -- 172.21.15.70:0/1763661262 learned_addr learned my addr 172.21.15.70:0/1763661262 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:34.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08d90a0700 1 --2- 172.21.15.70:0/1763661262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d6f60 0x564cc73dbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:34.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08d889f700 1 -- 172.21.15.70:0/1763661262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d3210 msgr2=0x564cc73db820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:34.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08d889f700 1 --2- 172.21.15.70:0/1763661262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d3210 0x564cc73db820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08d889f700 1 -- 172.21.15.70:0/1763661262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d6f60 msgr2=0x564cc73dbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:34.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08d889f700 1 --2- 172.21.15.70:0/1763661262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d6f60 0x564cc73dbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08d889f700 1 -- 172.21.15.70:0/1763661262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cc72c0690 con 0x564cc73d2860 2024-06-04T05:15:34.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.050+0000 7f08d3fff700 1 --2- 172.21.15.70:0/1763661262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d3210 0x564cc73db820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:34.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.051+0000 7f08d889f700 1 --2- 172.21.15.70:0/1763661262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73d2860 0x564cc73de520 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f08c80119b0 tx=0x7f08c80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e0bb67d6dc5bbb8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:34.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.051+0000 7f08b3fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:15:34.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.051+0000 7f08b3fff700 1 -- 172.21.15.70:0/1763661262 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c801f840 con 0x564cc73d2860 2024-06-04T05:15:34.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.051+0000 7f08b3fff700 1 -- 172.21.15.70:0/1763661262 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f08c801f9e0 con 0x564cc73d2860 2024-06-04T05:15:34.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.051+0000 7f08e5398b80 1 -- 172.21.15.70:0/1763661262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08c00a9140 con 0x564cc73d2860 2024-06-04T05:15:34.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.051+0000 7f08b3fff700 1 -- 172.21.15.70:0/1763661262 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f08c801fc20 con 0x564cc73d2860 2024-06-04T05:15:34.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.051+0000 7f08e5398b80 10 client.20247 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:34.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.051+0000 7f08e5398b80 1 -- 172.21.15.70:0/1763661262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x564cc73f2220 con 0x564cc73d2860 2024-06-04T05:15:34.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.052+0000 7f08b3fff700 1 -- 172.21.15.70:0/1763661262 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f08c00a9140 con 0x564cc73d2860 2024-06-04T05:15:34.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.052+0000 7f08b3fff700 1 -- 172.21.15.70:0/1763661262 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f08c802f070 con 0x564cc73d2860 2024-06-04T05:15:34.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.052+0000 7f08e5398b80 10 client.20247 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:34.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.052+0000 7f08e5398b80 10 client.20247 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:34.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.052+0000 7f08e5398b80 20 client.20247 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:34.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.052+0000 7f08e5398b80 1 --2- 172.21.15.70:0/1763661262 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564cc73f47b0 0x564cc73f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.052+0000 7f08e5398b80 4 client.20247 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:15:34.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.052+0000 7f08e5398b80 1 -- 172.21.15.70:0/1763661262 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cc739e9e0 con 0x564cc73f47b0 2024-06-04T05:15:34.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.052+0000 7f08d3fff700 1 --2- 172.21.15.70:0/1763661262 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564cc73f47b0 0x564cc73f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:34.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.053+0000 7f08b3fff700 10 client.20247 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:34.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.053+0000 7f08d3fff700 1 --2- 172.21.15.70:0/1763661262 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564cc73f47b0 0x564cc73f6be0 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a463586e5b0e0cf server_cookie=6c6cafc2f55d4ed4 in_seq=0 out_seq=0 2024-06-04T05:15:34.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.054+0000 7f08b3fff700 1 -- 172.21.15.70:0/1763661262 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564cc739e9e0 con 0x564cc73f47b0 2024-06-04T05:15:34.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.054+0000 7f08b3fff700 10 client.20247 handle_command_reply: tid=0 2024-06-04T05:15:34.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.054+0000 7f08e5398b80 1 client.20247 shutdown 2024-06-04T05:15:34.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.055+0000 7f08e5398b80 1 -- 172.21.15.70:0/1763661262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73d2860 msgr2=0x564cc73de520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:34.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.055+0000 7f08e5398b80 1 --2- 172.21.15.70:0/1763661262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73d2860 0x564cc73de520 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f08c80119b0 tx=0x7f08c80047c0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.055+0000 7f08e5398b80 1 -- 172.21.15.70:0/1763661262 shutdown_connections 2024-06-04T05:15:34.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.055+0000 7f08e5398b80 1 --2- 172.21.15.70:0/1763661262 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564cc73f47b0 0x564cc73f6be0 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.055+0000 7f08e5398b80 1 --2- 172.21.15.70:0/1763661262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d6f60 0x564cc73dbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.055+0000 7f08e5398b80 1 --2- 172.21.15.70:0/1763661262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73d2860 0x564cc73de520 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.055+0000 7f08e5398b80 1 --2- 172.21.15.70:0/1763661262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d3210 0x564cc73db820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.055+0000 7f08e5398b80 1 -- 172.21.15.70:0/1763661262 >> 172.21.15.70:0/1763661262 conn(0x564cc7300080 msgr2=0x564cc73cc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:34.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.055+0000 7f08b3fff700 0 client.20247 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:34.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.055+0000 7f08e5398b80 1 -- 172.21.15.70:0/1763661262 shutdown_connections 2024-06-04T05:15:34.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.055+0000 7f08e5398b80 1 -- 172.21.15.70:0/1763661262 wait complete. 2024-06-04T05:15:34.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.055+0000 7f08e5398b80 20 client.20247 trim_cache size 0 max 16384 2024-06-04T05:15:34.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.085+0000 7f08e5398b80 1 Processor -- start 2024-06-04T05:15:34.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.085+0000 7f08e5398b80 1 -- start start 2024-06-04T05:15:34.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.085+0000 7f08e5398b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 0x564cc73f0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.085+0000 7f08e5398b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d6f60 0x564cc73dd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.085+0000 7f08e5398b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73ddca0 0x564cc73ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.085+0000 7f08e5398b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08d41abd60 con 0x564cc73ddca0 2024-06-04T05:15:34.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.085+0000 7f08e5398b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08d41ad900 con 0x564cc73d3210 2024-06-04T05:15:34.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.085+0000 7f08e5398b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f08d41ac490 con 0x564cc73d6f60 2024-06-04T05:15:34.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.085+0000 7f08d3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d6f60 0x564cc73dd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:34.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.085+0000 7f08d90a0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73ddca0 0x564cc73ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:34.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.085+0000 7f08d3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d6f60 0x564cc73dd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41440/0 (socket says 172.21.15.70:41440) 2024-06-04T05:15:34.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.085+0000 7f08d90a0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73ddca0 0x564cc73ccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34728/0 (socket says 172.21.15.70:34728) 2024-06-04T05:15:34.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.085+0000 7f08d3fff700 1 -- 172.21.15.70:0/1055906641 learned_addr learned my addr 172.21.15.70:0/1055906641 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:34.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.085+0000 7f08d889f700 1 --2- 172.21.15.70:0/1055906641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 0x564cc73f0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:34.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.086+0000 7f08d889f700 1 -- 172.21.15.70:0/1055906641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d6f60 msgr2=0x564cc73dd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:34.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.086+0000 7f08d889f700 1 --2- 172.21.15.70:0/1055906641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d6f60 0x564cc73dd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.086+0000 7f08d889f700 1 -- 172.21.15.70:0/1055906641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73ddca0 msgr2=0x564cc73ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:34.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.086+0000 7f08d889f700 1 --2- 172.21.15.70:0/1055906641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73ddca0 0x564cc73ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.086+0000 7f08d889f700 1 -- 172.21.15.70:0/1055906641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cc73ce8e0 con 0x564cc73d3210 2024-06-04T05:15:34.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.086+0000 7f08d889f700 1 --2- 172.21.15.70:0/1055906641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 0x564cc73f0a50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f08c8031ec0 tx=0x7f08c8010ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d1b2e5852930e97 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:34.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.086+0000 7f08be7fc700 1 -- 172.21.15.70:0/1055906641 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c801f840 con 0x564cc73d3210 2024-06-04T05:15:34.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.086+0000 7f08be7fc700 1 -- 172.21.15.70:0/1055906641 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f08c8023070 con 0x564cc73d3210 2024-06-04T05:15:34.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.086+0000 7f08be7fc700 1 -- 172.21.15.70:0/1055906641 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c801e800 con 0x564cc73d3210 2024-06-04T05:15:34.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.087+0000 7f08e5398b80 1 -- 172.21.15.70:0/1055906641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 msgr2=0x564cc73f0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:34.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.087+0000 7f08e5398b80 1 --2- 172.21.15.70:0/1055906641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 0x564cc73f0a50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f08c8031ec0 tx=0x7f08c8010ef0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.087+0000 7f08e5398b80 1 -- 172.21.15.70:0/1055906641 shutdown_connections 2024-06-04T05:15:34.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.087+0000 7f08e5398b80 1 --2- 172.21.15.70:0/1055906641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73ddca0 0x564cc73ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.087+0000 7f08e5398b80 1 --2- 172.21.15.70:0/1055906641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 0x564cc73f0a50 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.087+0000 7f08e5398b80 1 --2- 172.21.15.70:0/1055906641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d6f60 0x564cc73dd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.087+0000 7f08e5398b80 1 -- 172.21.15.70:0/1055906641 >> 172.21.15.70:0/1055906641 conn(0x564cc7300080 msgr2=0x564cc73dd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:34.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.087+0000 7f08e5398b80 1 -- 172.21.15.70:0/1055906641 shutdown_connections 2024-06-04T05:15:34.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.087+0000 7f08e5398b80 1 -- 172.21.15.70:0/1055906641 wait complete. 2024-06-04T05:15:34.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.088+0000 7f08e5398b80 1 -- start start 2024-06-04T05:15:34.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.088+0000 7f08e5398b80 1 Processor -- start 2024-06-04T05:15:34.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.088+0000 7f08e5398b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 0x564cc73ecfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.088+0000 7f08e5398b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d6f60 0x564cc73ea320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.088+0000 7f08e5398b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73ddca0 0x564cc73ea860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.088+0000 7f08e5398b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08d41ad230 con 0x564cc73ddca0 2024-06-04T05:15:34.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.088+0000 7f08e5398b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08d41acb60 con 0x564cc73d3210 2024-06-04T05:15:34.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.088+0000 7f08e5398b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f08d41a9d70 con 0x564cc73d6f60 2024-06-04T05:15:34.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.089+0000 7f08d889f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 0x564cc73ecfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:34.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.089+0000 7f08e5398b80 10 client.0 _ll_register_callbacks cb 0x7f08c8011580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:34.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.089+0000 7f08d3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d6f60 0x564cc73ea320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:34.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.089+0000 7f08d3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d6f60 0x564cc73ea320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41444/0 (socket says 172.21.15.70:41444) 2024-06-04T05:15:34.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.089+0000 7f08d3fff700 1 -- 172.21.15.70:0/363933280 learned_addr learned my addr 172.21.15.70:0/363933280 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:34.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.089+0000 7f08d90a0700 1 --2- 172.21.15.70:0/363933280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73ddca0 0x564cc73ea860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:34.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.089+0000 7f08d889f700 1 -- 172.21.15.70:0/363933280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d6f60 msgr2=0x564cc73ea320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:34.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.089+0000 7f08d889f700 1 --2- 172.21.15.70:0/363933280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d6f60 0x564cc73ea320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.089+0000 7f08d889f700 1 -- 172.21.15.70:0/363933280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73ddca0 msgr2=0x564cc73ea860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:34.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.089+0000 7f08d889f700 1 --2- 172.21.15.70:0/363933280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73ddca0 0x564cc73ea860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.089+0000 7f08d889f700 1 -- 172.21.15.70:0/363933280 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cc73d2e90 con 0x564cc73d3210 2024-06-04T05:15:34.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.089+0000 7f08d889f700 1 --2- 172.21.15.70:0/363933280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 0x564cc73ecfd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f08c801e450 tx=0x7f08c8031e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b2980afd6da02dc server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:34.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.089+0000 7f08b3fff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:15:34.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.089+0000 7f08b3fff700 1 -- 172.21.15.70:0/363933280 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c801fa50 con 0x564cc73d3210 2024-06-04T05:15:34.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.089+0000 7f08e5398b80 1 -- 172.21.15.70:0/363933280 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08c8023070 con 0x564cc73d3210 2024-06-04T05:15:34.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.089+0000 7f08b3fff700 1 -- 172.21.15.70:0/363933280 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f08c8005040 con 0x564cc73d3210 2024-06-04T05:15:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.090+0000 7f08b3fff700 1 -- 172.21.15.70:0/363933280 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f08c8031810 con 0x564cc73d3210 2024-06-04T05:15:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.090+0000 7f08e5398b80 10 client.29051 fetch_fsmap learned FSMap version 55 2024-06-04T05:15:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.090+0000 7f08e5398b80 1 -- 172.21.15.70:0/363933280 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x564cc745eed0 con 0x564cc73d3210 2024-06-04T05:15:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.090+0000 7f08b3fff700 1 -- 172.21.15.70:0/363933280 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f08c8023070 con 0x564cc73d3210 2024-06-04T05:15:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.090+0000 7f08b3fff700 1 -- 172.21.15.70:0/363933280 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f08c802f040 con 0x564cc73d3210 2024-06-04T05:15:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.090+0000 7f08e5398b80 10 client.29051 fetch_fsmap finished waiting for FSMap version 55 2024-06-04T05:15:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.090+0000 7f08e5398b80 10 client.29051 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.090+0000 7f08e5398b80 20 client.29051 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.090+0000 7f08e5398b80 1 --2- 172.21.15.70:0/363933280 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564cc745f540 0x564cc7461960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.090+0000 7f08e5398b80 4 client.29051 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:15:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.090+0000 7f08e5398b80 1 -- 172.21.15.70:0/363933280 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564cc7333be0 con 0x564cc745f540 2024-06-04T05:15:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.090+0000 7f08d3fff700 1 --2- 172.21.15.70:0/363933280 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564cc745f540 0x564cc7461960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.091+0000 7f08b3fff700 10 client.29051 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.091+0000 7f08d3fff700 1 --2- 172.21.15.70:0/363933280 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564cc745f540 0x564cc7461960 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1db37288e97b96b8 server_cookie=36554a09d98b3fec in_seq=0 out_seq=0 2024-06-04T05:15:34.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.091+0000 7f08b3fff700 1 -- 172.21.15.70:0/363933280 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564cc7333be0 con 0x564cc745f540 2024-06-04T05:15:34.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.091+0000 7f08b3fff700 10 client.29051 handle_command_reply: tid=0 2024-06-04T05:15:34.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.091+0000 7f08e5398b80 1 client.29051 shutdown 2024-06-04T05:15:34.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.091+0000 7f08e5398b80 1 -- 172.21.15.70:0/363933280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 msgr2=0x564cc73ecfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:34.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.091+0000 7f08e5398b80 1 --2- 172.21.15.70:0/363933280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 0x564cc73ecfd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f08c801e450 tx=0x7f08c8031e00 comp rx=0 tx=0).stop 2024-06-04T05:15:34.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.091+0000 7f08e5398b80 1 -- 172.21.15.70:0/363933280 shutdown_connections 2024-06-04T05:15:34.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.091+0000 7f08e5398b80 1 --2- 172.21.15.70:0/363933280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cc73ddca0 0x564cc73ea860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.091+0000 7f08e5398b80 1 --2- 172.21.15.70:0/363933280 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564cc745f540 0x564cc7461960 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.091+0000 7f08e5398b80 1 --2- 172.21.15.70:0/363933280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564cc73d3210 0x564cc73ecfd0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.091+0000 7f08e5398b80 1 --2- 172.21.15.70:0/363933280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564cc73d6f60 0x564cc73ea320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.091+0000 7f08b3fff700 0 client.29051 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:34.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.091+0000 7f08e5398b80 1 -- 172.21.15.70:0/363933280 >> 172.21.15.70:0/363933280 conn(0x564cc7300080 msgr2=0x564cc73d58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:34.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.092+0000 7f08e5398b80 1 -- 172.21.15.70:0/363933280 shutdown_connections 2024-06-04T05:15:34.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.092+0000 7f08e5398b80 1 -- 172.21.15.70:0/363933280 wait complete. 2024-06-04T05:15:34.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.092+0000 7f08e5398b80 20 client.29051 trim_cache size 0 max 16384 2024-06-04T05:15:34.099 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:15:34.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.092+0000 7f08dab03700 1 -- 172.21.15.70:0/2861004518 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08c40b2a70 msgr2=0x7f08c40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:34.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.092+0000 7f08dab03700 1 --2- 172.21.15.70:0/2861004518 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08c40b2a70 0x7f08c40b4f30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f08c80104e0 tx=0x7f08c8007480 comp rx=0 tx=0).stop 2024-06-04T05:15:34.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.092+0000 7f08dab03700 1 -- 172.21.15.70:0/2861004518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d41041a0 msgr2=0x7f08d4075d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:34.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.092+0000 7f08dab03700 1 --2- 172.21.15.70:0/2861004518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d41041a0 0x7f08d4075d80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f08c000d950 tx=0x7f08c000dc60 comp rx=0 tx=0).stop 2024-06-04T05:15:34.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.093+0000 7f08dab03700 1 -- 172.21.15.70:0/2861004518 shutdown_connections 2024-06-04T05:15:34.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.093+0000 7f08dab03700 1 --2- 172.21.15.70:0/2861004518 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08c40b2a70 0x7f08c40b4f30 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.093+0000 7f08dab03700 1 --2- 172.21.15.70:0/2861004518 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08d41088f0 0x7f08d40762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.093+0000 7f08dab03700 1 --2- 172.21.15.70:0/2861004518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08d41037f0 0x7f08d4075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.093+0000 7f08dab03700 1 --2- 172.21.15.70:0/2861004518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d41041a0 0x7f08d4075d80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:34.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.093+0000 7f08dab03700 1 -- 172.21.15.70:0/2861004518 >> 172.21.15.70:0/2861004518 conn(0x7f08d407a6a0 msgr2=0x7f08d41017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:34.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.093+0000 7f08dab03700 1 -- 172.21.15.70:0/2861004518 shutdown_connections 2024-06-04T05:15:34.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:34.093+0000 7f08dab03700 1 -- 172.21.15.70:0/2861004518 wait complete. 2024-06-04T05:15:34.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 37599 7.78 MB/sec execute 146 sec latency 85.365 ms 2024-06-04T05:15:35.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:35 smithi070 ceph-mon[32631]: pgmap v577: 97 pgs: 97 active+clean; 640 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 2.1 MiB/s wr, 132 op/s 2024-06-04T05:15:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:35 smithi112 ceph-mon[36453]: pgmap v577: 97 pgs: 97 active+clean; 640 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 2.1 MiB/s wr, 132 op/s 2024-06-04T05:15:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:35 smithi098 ceph-mon[35569]: pgmap v577: 97 pgs: 97 active+clean; 640 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 2.1 MiB/s wr, 132 op/s 2024-06-04T05:15:35.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 37797 7.81 MB/sec execute 147 sec latency 49.129 ms 2024-06-04T05:15:35.846 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:15:35.846 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:15:36.185 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:15:36.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 37969 7.80 MB/sec execute 148 sec latency 60.682 ms 2024-06-04T05:15:37.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:37 smithi070 ceph-mon[32631]: pgmap v578: 97 pgs: 97 active+clean; 641 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.7 MiB/s wr, 277 op/s 2024-06-04T05:15:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:37 smithi112 ceph-mon[36453]: pgmap v578: 97 pgs: 97 active+clean; 641 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.7 MiB/s wr, 277 op/s 2024-06-04T05:15:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:37 smithi098 ceph-mon[35569]: pgmap v578: 97 pgs: 97 active+clean; 641 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.7 MiB/s wr, 277 op/s 2024-06-04T05:15:37.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 38129 7.82 MB/sec execute 149 sec latency 109.016 ms 2024-06-04T05:15:38.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 38238 7.79 MB/sec execute 150 sec latency 147.194 ms 2024-06-04T05:15:39.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:39 smithi070 ceph-mon[32631]: pgmap v579: 97 pgs: 97 active+clean; 641 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 270 op/s 2024-06-04T05:15:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:39 smithi112 ceph-mon[36453]: pgmap v579: 97 pgs: 97 active+clean; 641 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 270 op/s 2024-06-04T05:15:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:39 smithi098 ceph-mon[35569]: pgmap v579: 97 pgs: 97 active+clean; 641 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 270 op/s 2024-06-04T05:15:39.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 38390 7.79 MB/sec execute 151 sec latency 115.056 ms 2024-06-04T05:15:40.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 38545 7.77 MB/sec execute 152 sec latency 107.980 ms 2024-06-04T05:15:40.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.852+0000 7fcd0b228640 1 -- 172.21.15.70:0/4166891598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd04101980 msgr2=0x7fcd04101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:40.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.852+0000 7fcd0b228640 1 --2- 172.21.15.70:0/4166891598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd04101980 0x7fcd04101e00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fccf0009930 tx=0x7fccf002a420 comp rx=0 tx=0).stop 2024-06-04T05:15:40.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.852+0000 7fcd0b228640 1 -- 172.21.15.70:0/4166891598 shutdown_connections 2024-06-04T05:15:40.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.852+0000 7fcd0b228640 1 --2- 172.21.15.70:0/4166891598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd04102340 0x7fcd0410e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.852+0000 7fcd0b228640 1 --2- 172.21.15.70:0/4166891598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd04101980 0x7fcd04101e00 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.852+0000 7fcd0b228640 1 --2- 172.21.15.70:0/4166891598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd04107940 0x7fcd04107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.852+0000 7fcd0b228640 1 -- 172.21.15.70:0/4166891598 >> 172.21.15.70:0/4166891598 conn(0x7fcd04078ad0 msgr2=0x7fcd04078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:40.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.852+0000 7fcd0b228640 1 -- 172.21.15.70:0/4166891598 shutdown_connections 2024-06-04T05:15:40.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.852+0000 7fcd0b228640 1 -- 172.21.15.70:0/4166891598 wait complete. 2024-06-04T05:15:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.852+0000 7fcd0b228640 1 Processor -- start 2024-06-04T05:15:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd0b228640 1 -- start start 2024-06-04T05:15:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd0b228640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd04101980 0x7fcd04196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd0b228640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd04102340 0x7fcd04197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd0b228640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd04107940 0x7fcd0419b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd0b228640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcd041109a0 con 0x7fcd04101980 2024-06-04T05:15:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd0b228640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd04110820 con 0x7fcd04102340 2024-06-04T05:15:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd0b228640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcd04110b20 con 0x7fcd04107940 2024-06-04T05:15:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd03fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd04102340 0x7fcd04197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd08fc4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd04101980 0x7fcd04196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd08fc4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd04101980 0x7fcd04196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56066/0 (socket says 172.21.15.70:56066) 2024-06-04T05:15:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd03fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd04102340 0x7fcd04197440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:47860/0 (socket says 172.21.15.70:47860) 2024-06-04T05:15:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd08fc4640 1 -- 172.21.15.70:0/3035949810 learned_addr learned my addr 172.21.15.70:0/3035949810 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd097c5640 1 --2- 172.21.15.70:0/3035949810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd04107940 0x7fcd0419b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd08fc4640 1 -- 172.21.15.70:0/3035949810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd04107940 msgr2=0x7fcd0419b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd08fc4640 1 --2- 172.21.15.70:0/3035949810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd04107940 0x7fcd0419b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd08fc4640 1 -- 172.21.15.70:0/3035949810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd04102340 msgr2=0x7fcd04197440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd08fc4640 1 --2- 172.21.15.70:0/3035949810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd04102340 0x7fcd04197440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.853+0000 7fcd08fc4640 1 -- 172.21.15.70:0/3035949810 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd0419bef0 con 0x7fcd04101980 2024-06-04T05:15:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.854+0000 7fcd08fc4640 1 --2- 172.21.15.70:0/3035949810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd04101980 0x7fcd04196f00 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fccf800d950 tx=0x7fccf800dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.854+0000 7fcd01ffb640 1 -- 172.21.15.70:0/3035949810 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf8017030 con 0x7fcd04101980 2024-06-04T05:15:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.854+0000 7fcd0b228640 1 -- 172.21.15.70:0/3035949810 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd0419c1e0 con 0x7fcd04101980 2024-06-04T05:15:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.854+0000 7fcd01ffb640 1 -- 172.21.15.70:0/3035949810 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fccf8004020 con 0x7fcd04101980 2024-06-04T05:15:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.854+0000 7fcd01ffb640 1 -- 172.21.15.70:0/3035949810 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf80054b0 con 0x7fcd04101980 2024-06-04T05:15:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.854+0000 7fcd0b228640 1 -- 172.21.15.70:0/3035949810 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd041a3a70 con 0x7fcd04101980 2024-06-04T05:15:40.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.858+0000 7fcd01ffb640 1 -- 172.21.15.70:0/3035949810 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fccf8009a30 con 0x7fcd04101980 2024-06-04T05:15:40.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd01ffb640 1 --2- 172.21.15.70:0/3035949810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fccf40b2a90 0x7fccf40b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd0e017740 1 Processor -- start 2024-06-04T05:15:40.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd0e017740 1 -- start start 2024-06-04T05:15:40.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd03fff640 1 --2- 172.21.15.70:0/3035949810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fccf40b2a90 0x7fccf40b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd01ffb640 1 -- 172.21.15.70:0/3035949810 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7fccf80e1e80 con 0x7fcd04101980 2024-06-04T05:15:40.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd0e017740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a3f820 0x5585b8a3fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd0e017740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a40140 0x5585b8a4a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd0e017740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a4aa00 0x5585b8a4cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd0e017740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5585b89b1c30 con 0x5585b8a4aa00 2024-06-04T05:15:40.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd0e017740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5585b89a17d0 con 0x5585b8a40140 2024-06-04T05:15:40.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd0e017740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5585b89a5810 con 0x5585b8a3f820 2024-06-04T05:15:40.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd03fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a40140 0x5585b8a4a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd03fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a40140 0x5585b8a4a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:47872/0 (socket says 172.21.15.70:47872) 2024-06-04T05:15:40.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd03fff640 1 -- 172.21.15.70:0/1661632318 learned_addr learned my addr 172.21.15.70:0/1661632318 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:40.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd08fc4640 1 --2- 172.21.15.70:0/1661632318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a3f820 0x5585b8a3fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd097c5640 1 --2- 172.21.15.70:0/1661632318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a4aa00 0x5585b8a4cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd08fc4640 1 -- 172.21.15.70:0/1661632318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a40140 msgr2=0x5585b8a4a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:40.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd08fc4640 1 --2- 172.21.15.70:0/1661632318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a40140 0x5585b8a4a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd08fc4640 1 -- 172.21.15.70:0/1661632318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a4aa00 msgr2=0x5585b8a4cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:40.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd08fc4640 1 --2- 172.21.15.70:0/1661632318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a4aa00 0x5585b8a4cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd03fff640 1 --2- 172.21.15.70:0/3035949810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fccf40b2a90 0x7fccf40b4f50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fccf00057c0 tx=0x7fccf0031de0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:40.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.860+0000 7fcd08fc4640 1 -- 172.21.15.70:0/1661632318 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585b86b6210 con 0x5585b8a3f820 2024-06-04T05:15:40.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.861+0000 7fcd08fc4640 1 --2- 172.21.15.70:0/1661632318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a3f820 0x5585b8a3fc00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fccf80df540 tx=0x7fccf80a8950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23be5a583d41cd66 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:40.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.861+0000 7fcceeffd640 1 -- 172.21.15.70:0/1661632318 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf80a90d0 con 0x5585b8a3f820 2024-06-04T05:15:40.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.861+0000 7fcceeffd640 1 -- 172.21.15.70:0/1661632318 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fccf80ad480 con 0x5585b8a3f820 2024-06-04T05:15:40.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.861+0000 7fcceeffd640 1 -- 172.21.15.70:0/1661632318 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf80df7c0 con 0x5585b8a3f820 2024-06-04T05:15:40.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.862+0000 7fcd0e017740 1 -- 172.21.15.70:0/1661632318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a3f820 msgr2=0x5585b8a3fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:40.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.862+0000 7fcd0e017740 1 --2- 172.21.15.70:0/1661632318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a3f820 0x5585b8a3fc00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fccf80df540 tx=0x7fccf80a8950 comp rx=0 tx=0).stop 2024-06-04T05:15:40.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.862+0000 7fcd0e017740 1 -- 172.21.15.70:0/1661632318 shutdown_connections 2024-06-04T05:15:40.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.862+0000 7fcd0e017740 1 --2- 172.21.15.70:0/1661632318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a40140 0x5585b8a4a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.862+0000 7fcd0e017740 1 --2- 172.21.15.70:0/1661632318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a4aa00 0x5585b8a4cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.862+0000 7fcd0e017740 1 --2- 172.21.15.70:0/1661632318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a3f820 0x5585b8a3fc00 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.862+0000 7fcd0e017740 1 -- 172.21.15.70:0/1661632318 >> 172.21.15.70:0/1661632318 conn(0x5585b8a3bd30 msgr2=0x5585b8a3c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:40.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.862+0000 7fcd0e017740 1 -- 172.21.15.70:0/1661632318 shutdown_connections 2024-06-04T05:15:40.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.862+0000 7fcd0e017740 1 -- 172.21.15.70:0/1661632318 wait complete. 2024-06-04T05:15:40.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.862+0000 7fcd0e017740 1 -- start start 2024-06-04T05:15:40.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.863+0000 7fcd0e017740 1 Processor -- start 2024-06-04T05:15:40.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.863+0000 7fcd0e017740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a3f820 0x5585b8a47b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.863+0000 7fcd0e017740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a40140 0x5585b8a48040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.863+0000 7fcd0e017740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a4aa00 0x5585b8a48580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.863+0000 7fcd0e017740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5585b8687790 con 0x5585b8a3f820 2024-06-04T05:15:40.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.863+0000 7fcd0e017740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5585b895d890 con 0x5585b8a4aa00 2024-06-04T05:15:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.863+0000 7fcd0e017740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5585b8a4d630 con 0x5585b8a40140 2024-06-04T05:15:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.863+0000 7fcd0e017740 10 client.0 _ll_register_callbacks cb 0x5585b86b5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.863+0000 7fcd08fc4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a3f820 0x5585b8a47b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.863+0000 7fcd08fc4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a3f820 0x5585b8a47b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56072/0 (socket says 172.21.15.70:56072) 2024-06-04T05:15:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.863+0000 7fcd08fc4640 1 -- 172.21.15.70:0/2698237165 learned_addr learned my addr 172.21.15.70:0/2698237165 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.863+0000 7fcd03fff640 1 --2- 172.21.15.70:0/2698237165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a40140 0x5585b8a48040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.863+0000 7fcd097c5640 1 --2- 172.21.15.70:0/2698237165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a4aa00 0x5585b8a48580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.864+0000 7fcd08fc4640 1 -- 172.21.15.70:0/2698237165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a40140 msgr2=0x5585b8a48040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.864+0000 7fcd08fc4640 1 --2- 172.21.15.70:0/2698237165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a40140 0x5585b8a48040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.864+0000 7fcd08fc4640 1 -- 172.21.15.70:0/2698237165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a4aa00 msgr2=0x5585b8a48580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.864+0000 7fcd08fc4640 1 --2- 172.21.15.70:0/2698237165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a4aa00 0x5585b8a48580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.864+0000 7fcd08fc4640 1 -- 172.21.15.70:0/2698237165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585b8782620 con 0x5585b8a3f820 2024-06-04T05:15:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.864+0000 7fcd097c5640 1 --2- 172.21.15.70:0/2698237165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a4aa00 0x5585b8a48580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:15:40.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.864+0000 7fcd03fff640 1 --2- 172.21.15.70:0/2698237165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a40140 0x5585b8a48040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:40.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.864+0000 7fcd08fc4640 1 --2- 172.21.15.70:0/2698237165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a3f820 0x5585b8a47b00 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fccf80a8dd0 tx=0x7fccf80a9910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7eb9f543ca715c6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:40.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.864+0000 7fcce3fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:15:40.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.864+0000 7fcce3fff640 1 -- 172.21.15.70:0/2698237165 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf80c2020 con 0x5585b8a3f820 2024-06-04T05:15:40.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.864+0000 7fcd0e017740 1 -- 172.21.15.70:0/2698237165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccf80ad480 con 0x5585b8a3f820 2024-06-04T05:15:40.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.864+0000 7fcce3fff640 1 -- 172.21.15.70:0/2698237165 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7fccf8009d20 con 0x5585b8a3f820 2024-06-04T05:15:40.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.865+0000 7fcce3fff640 1 -- 172.21.15.70:0/2698237165 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7fccf80a5360 con 0x5585b8a3f820 2024-06-04T05:15:40.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.865+0000 7fcd0e017740 10 client.20265 fetch_fsmap learned FSMap version 56 2024-06-04T05:15:40.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.865+0000 7fcd0e017740 1 -- 172.21.15.70:0/2698237165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x5585b8a59500 con 0x5585b8a3f820 2024-06-04T05:15:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.865+0000 7fcce3fff640 1 -- 172.21.15.70:0/2698237165 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fccf80ad480 con 0x5585b8a3f820 2024-06-04T05:15:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.866+0000 7fcce3fff640 1 -- 172.21.15.70:0/2698237165 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7fccf80a7da0 con 0x5585b8a3f820 2024-06-04T05:15:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.866+0000 7fcd0e017740 10 client.20265 fetch_fsmap finished waiting for FSMap version 56 2024-06-04T05:15:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.866+0000 7fcd0e017740 10 client.20265 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.866+0000 7fcd0e017740 20 client.20265 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.866+0000 7fcd0e017740 1 --2- 172.21.15.70:0/2698237165 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5585b8a5ab20 0x5585b8a5cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.866+0000 7fcd0e017740 4 client.20265 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:15:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.866+0000 7fcd0e017740 1 -- 172.21.15.70:0/2698237165 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585b89b3b80 con 0x5585b8a5ab20 2024-06-04T05:15:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.866+0000 7fcd03fff640 1 --2- 172.21.15.70:0/2698237165 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5585b8a5ab20 0x5585b8a5cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.866+0000 7fcce3fff640 10 client.20265 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.866+0000 7fcd03fff640 1 --2- 172.21.15.70:0/2698237165 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5585b8a5ab20 0x5585b8a5cf50 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c38d5e275e0e8559 server_cookie=52f616276879f99d in_seq=0 out_seq=0 2024-06-04T05:15:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.867+0000 7fcce3fff640 1 -- 172.21.15.70:0/2698237165 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5585b89b3b80 con 0x5585b8a5ab20 2024-06-04T05:15:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.867+0000 7fcce3fff640 10 client.20265 handle_command_reply: tid=0 2024-06-04T05:15:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.867+0000 7fcd0e017740 1 client.20265 shutdown 2024-06-04T05:15:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.868+0000 7fcd0e017740 1 -- 172.21.15.70:0/2698237165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a3f820 msgr2=0x5585b8a47b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.868+0000 7fcd0e017740 1 --2- 172.21.15.70:0/2698237165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a3f820 0x5585b8a47b00 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fccf80a8dd0 tx=0x7fccf80a9910 comp rx=0 tx=0).stop 2024-06-04T05:15:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.868+0000 7fcd0e017740 1 -- 172.21.15.70:0/2698237165 shutdown_connections 2024-06-04T05:15:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.868+0000 7fcd0e017740 1 --2- 172.21.15.70:0/2698237165 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5585b8a5ab20 0x5585b8a5cf50 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.868+0000 7fcd0e017740 1 --2- 172.21.15.70:0/2698237165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a4aa00 0x5585b8a48580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.868+0000 7fcd0e017740 1 --2- 172.21.15.70:0/2698237165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a40140 0x5585b8a48040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.868+0000 7fcd0e017740 1 --2- 172.21.15.70:0/2698237165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a3f820 0x5585b8a47b00 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.868+0000 7fcd0e017740 1 -- 172.21.15.70:0/2698237165 >> 172.21.15.70:0/2698237165 conn(0x5585b8a3bd30 msgr2=0x5585b8a3afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.868+0000 7fcce3fff640 0 client.20265 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.868+0000 7fcd0e017740 1 -- 172.21.15.70:0/2698237165 shutdown_connections 2024-06-04T05:15:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.868+0000 7fcd0e017740 1 -- 172.21.15.70:0/2698237165 wait complete. 2024-06-04T05:15:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.868+0000 7fcd0e017740 20 client.20265 trim_cache size 0 max 16384 2024-06-04T05:15:40.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.894+0000 7fcd0e017740 1 Processor -- start 2024-06-04T05:15:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.894+0000 7fcd0e017740 1 -- start start 2024-06-04T05:15:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.895+0000 7fcd0e017740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a40140 0x5585b8a3b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.895+0000 7fcd0e017740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a4aa00 0x5585b8a481d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.895+0000 7fcd0e017740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a4a340 0x5585b8a62830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.895+0000 7fcd0e017740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcd041ac540 con 0x5585b8a4aa00 2024-06-04T05:15:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.895+0000 7fcd0e017740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd041ae0b0 con 0x5585b8a40140 2024-06-04T05:15:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.895+0000 7fcd0e017740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcd041acc40 con 0x5585b8a4a340 2024-06-04T05:15:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.895+0000 7fcd08fc4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a40140 0x5585b8a3b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.895+0000 7fcd08fc4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a40140 0x5585b8a3b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:47888/0 (socket says 172.21.15.70:47888) 2024-06-04T05:15:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.895+0000 7fcd08fc4640 1 -- 172.21.15.70:0/2376790861 learned_addr learned my addr 172.21.15.70:0/2376790861 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.895+0000 7fcd097c5640 1 --2- 172.21.15.70:0/2376790861 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a4a340 0x5585b8a62830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.895+0000 7fcd03fff640 1 --2- 172.21.15.70:0/2376790861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a4aa00 0x5585b8a481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.895+0000 7fcd08fc4640 1 -- 172.21.15.70:0/2376790861 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a4a340 msgr2=0x5585b8a62830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.895+0000 7fcd08fc4640 1 --2- 172.21.15.70:0/2376790861 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a4a340 0x5585b8a62830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.895+0000 7fcd08fc4640 1 -- 172.21.15.70:0/2376790861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a4aa00 msgr2=0x5585b8a481d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.895+0000 7fcd08fc4640 1 --2- 172.21.15.70:0/2376790861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a4aa00 0x5585b8a481d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.895+0000 7fcd08fc4640 1 -- 172.21.15.70:0/2376790861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585b8a46ab0 con 0x5585b8a40140 2024-06-04T05:15:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.895+0000 7fcd03fff640 1 --2- 172.21.15.70:0/2376790861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a4aa00 0x5585b8a481d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:15:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.896+0000 7fcd08fc4640 1 --2- 172.21.15.70:0/2376790861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a40140 0x5585b8a3b730 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fccf80a9980 tx=0x7fccf80df620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46681f43a219e2d4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.896+0000 7fccee7fc640 1 -- 172.21.15.70:0/2376790861 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf80c2070 con 0x5585b8a40140 2024-06-04T05:15:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.896+0000 7fccee7fc640 1 -- 172.21.15.70:0/2376790861 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fccf80ad480 con 0x5585b8a40140 2024-06-04T05:15:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.896+0000 7fccee7fc640 1 -- 172.21.15.70:0/2376790861 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf80b1f30 con 0x5585b8a40140 2024-06-04T05:15:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.897+0000 7fcd0e017740 1 -- 172.21.15.70:0/2376790861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a40140 msgr2=0x5585b8a3b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.897+0000 7fcd0e017740 1 --2- 172.21.15.70:0/2376790861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a40140 0x5585b8a3b730 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fccf80a9980 tx=0x7fccf80df620 comp rx=0 tx=0).stop 2024-06-04T05:15:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.897+0000 7fcd0e017740 1 -- 172.21.15.70:0/2376790861 shutdown_connections 2024-06-04T05:15:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.897+0000 7fcd0e017740 1 --2- 172.21.15.70:0/2376790861 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a4a340 0x5585b8a62830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.897+0000 7fcd0e017740 1 --2- 172.21.15.70:0/2376790861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a4aa00 0x5585b8a481d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.897+0000 7fcd0e017740 1 --2- 172.21.15.70:0/2376790861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a40140 0x5585b8a3b730 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.897+0000 7fcd0e017740 1 -- 172.21.15.70:0/2376790861 >> 172.21.15.70:0/2376790861 conn(0x5585b8a3bd30 msgr2=0x5585b8a47ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.897+0000 7fcd0e017740 1 -- 172.21.15.70:0/2376790861 shutdown_connections 2024-06-04T05:15:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.897+0000 7fcd0e017740 1 -- 172.21.15.70:0/2376790861 wait complete. 2024-06-04T05:15:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.898+0000 7fcd0e017740 1 -- start start 2024-06-04T05:15:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.898+0000 7fcd0e017740 1 Processor -- start 2024-06-04T05:15:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.898+0000 7fcd0e017740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a40140 0x5585b8a53110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.898+0000 7fcd0e017740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a4a340 0x5585b8a4e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.898+0000 7fcd0e017740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a4aa00 0x5585b8a4eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.898+0000 7fcd0e017740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcd041ad9e0 con 0x5585b8a4a340 2024-06-04T05:15:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.898+0000 7fcd0e017740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd041ad310 con 0x5585b8a4aa00 2024-06-04T05:15:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.898+0000 7fcd0e017740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcd041aa540 con 0x5585b8a40140 2024-06-04T05:15:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.899+0000 7fcd0e017740 10 client.0 _ll_register_callbacks cb 0x7fccfc009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.899+0000 7fcd08fc4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a40140 0x5585b8a53110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.899+0000 7fcd08fc4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a40140 0x5585b8a53110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:58164/0 (socket says 172.21.15.70:58164) 2024-06-04T05:15:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.899+0000 7fcd08fc4640 1 -- 172.21.15.70:0/1632221176 learned_addr learned my addr 172.21.15.70:0/1632221176 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.899+0000 7fcd097c5640 1 --2- 172.21.15.70:0/1632221176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a4aa00 0x5585b8a4eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.899+0000 7fcd03fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a4a340 0x5585b8a4e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.899+0000 7fcd08fc4640 1 -- 172.21.15.70:0/1632221176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a4aa00 msgr2=0x5585b8a4eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.899+0000 7fcd08fc4640 1 --2- 172.21.15.70:0/1632221176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a4aa00 0x5585b8a4eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.899+0000 7fcd08fc4640 1 -- 172.21.15.70:0/1632221176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a4a340 msgr2=0x5585b8a4e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.899+0000 7fcd08fc4640 1 --2- 172.21.15.70:0/1632221176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a4a340 0x5585b8a4e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.899+0000 7fcd08fc4640 1 -- 172.21.15.70:0/1632221176 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585b8ac4700 con 0x5585b8a40140 2024-06-04T05:15:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.899+0000 7fcd097c5640 1 --2- 172.21.15.70:0/1632221176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a4aa00 0x5585b8a4eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.899+0000 7fcd08fc4640 1 --2- 172.21.15.70:0/1632221176 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a40140 0x5585b8a53110 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fccf80a6340 tx=0x7fccf80a9910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1105295b8fa81a0f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.899+0000 7fcce3fff640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:15:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.899+0000 7fcce3fff640 1 -- 172.21.15.70:0/1632221176 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf80c2070 con 0x5585b8a40140 2024-06-04T05:15:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.899+0000 7fcce3fff640 1 -- 172.21.15.70:0/1632221176 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7fccf80a56c0 con 0x5585b8a40140 2024-06-04T05:15:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.899+0000 7fcd0e017740 1 -- 172.21.15.70:0/1632221176 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccf80ad480 con 0x5585b8a40140 2024-06-04T05:15:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.900+0000 7fcce3fff640 1 -- 172.21.15.70:0/1632221176 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7fccf80a90d0 con 0x5585b8a40140 2024-06-04T05:15:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.900+0000 7fcd0e017740 10 client.29173 fetch_fsmap learned FSMap version 56 2024-06-04T05:15:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.900+0000 7fcd0e017740 1 -- 172.21.15.70:0/1632221176 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x5585b8ac4700 con 0x5585b8a40140 2024-06-04T05:15:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.900+0000 7fcce3fff640 1 -- 172.21.15.70:0/1632221176 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fccf80ad480 con 0x5585b8a40140 2024-06-04T05:15:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.900+0000 7fcce3fff640 1 -- 172.21.15.70:0/1632221176 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7fccf80a4a40 con 0x5585b8a40140 2024-06-04T05:15:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.900+0000 7fcd0e017740 10 client.29173 fetch_fsmap finished waiting for FSMap version 56 2024-06-04T05:15:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.900+0000 7fcd0e017740 10 client.29173 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.900+0000 7fcd0e017740 20 client.29173 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.900+0000 7fcd0e017740 1 --2- 172.21.15.70:0/1632221176 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5585b8ac5a70 0x5585b8ac7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.900+0000 7fcd0e017740 4 client.29173 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:15:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.900+0000 7fcd0e017740 1 -- 172.21.15.70:0/1632221176 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5585b89ad360 con 0x5585b8ac5a70 2024-06-04T05:15:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.900+0000 7fcd03fff640 1 --2- 172.21.15.70:0/1632221176 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5585b8ac5a70 0x5585b8ac7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.901+0000 7fcce3fff640 10 client.29173 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:40.901+0000 7fcd03fff640 1 --2- 172.21.15.70:0/1632221176 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5585b8ac5a70 0x5585b8ac7e60 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48afe11a8308f251 server_cookie=fd6c7a77c40bed1 in_seq=0 out_seq=0 2024-06-04T05:15:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.026+0000 7fcce3fff640 1 -- 172.21.15.70:0/1632221176 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5585b89ad360 con 0x5585b8ac5a70 2024-06-04T05:15:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.026+0000 7fcce3fff640 10 client.29173 handle_command_reply: tid=0 2024-06-04T05:15:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.026+0000 7fcd0e017740 1 client.29173 shutdown 2024-06-04T05:15:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.026+0000 7fcd0e017740 1 -- 172.21.15.70:0/1632221176 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a40140 msgr2=0x5585b8a53110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.026+0000 7fcd0e017740 1 --2- 172.21.15.70:0/1632221176 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a40140 0x5585b8a53110 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fccf80a6340 tx=0x7fccf80a9910 comp rx=0 tx=0).stop 2024-06-04T05:15:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.026+0000 7fcd0e017740 1 -- 172.21.15.70:0/1632221176 shutdown_connections 2024-06-04T05:15:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.026+0000 7fcd0e017740 1 --2- 172.21.15.70:0/1632221176 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5585b8ac5a70 0x5585b8ac7e60 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.026+0000 7fcd0e017740 1 --2- 172.21.15.70:0/1632221176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585b8a4aa00 0x5585b8a4eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.026+0000 7fcd0e017740 1 --2- 172.21.15.70:0/1632221176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585b8a4a340 0x5585b8a4e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.026+0000 7fcd0e017740 1 --2- 172.21.15.70:0/1632221176 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5585b8a40140 0x5585b8a53110 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.026+0000 7fcce3fff640 0 client.29173 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.026+0000 7fcd0e017740 1 -- 172.21.15.70:0/1632221176 >> 172.21.15.70:0/1632221176 conn(0x5585b8a3bd30 msgr2=0x5585b8a49bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.026+0000 7fcd0e017740 1 -- 172.21.15.70:0/1632221176 shutdown_connections 2024-06-04T05:15:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.026+0000 7fcd0e017740 1 -- 172.21.15.70:0/1632221176 wait complete. 2024-06-04T05:15:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.026+0000 7fcd0e017740 20 client.29173 trim_cache size 0 max 16384 2024-06-04T05:15:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:15:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:15:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:15:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.027+0000 7fcd0b228640 1 -- 172.21.15.70:0/3035949810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fccf40b2a90 msgr2=0x7fccf40b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.027+0000 7fcd0b228640 1 --2- 172.21.15.70:0/3035949810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fccf40b2a90 0x7fccf40b4f50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fccf00057c0 tx=0x7fccf0031de0 comp rx=0 tx=0).stop 2024-06-04T05:15:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.027+0000 7fcd0b228640 1 -- 172.21.15.70:0/3035949810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd04101980 msgr2=0x7fcd04196f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.027+0000 7fcd0b228640 1 --2- 172.21.15.70:0/3035949810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd04101980 0x7fcd04196f00 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fccf800d950 tx=0x7fccf800dd50 comp rx=0 tx=0).stop 2024-06-04T05:15:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.027+0000 7fcd0b228640 1 -- 172.21.15.70:0/3035949810 shutdown_connections 2024-06-04T05:15:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.027+0000 7fcd0b228640 1 --2- 172.21.15.70:0/3035949810 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fccf40b2a90 0x7fccf40b4f50 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.027+0000 7fcd0b228640 1 --2- 172.21.15.70:0/3035949810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd04102340 0x7fcd04197440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.027+0000 7fcd0b228640 1 --2- 172.21.15.70:0/3035949810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd04107940 0x7fcd0419b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:41.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.027+0000 7fcd0b228640 1 --2- 172.21.15.70:0/3035949810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd04101980 0x7fcd04196f00 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:41.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.027+0000 7fcd0b228640 1 -- 172.21.15.70:0/3035949810 >> 172.21.15.70:0/3035949810 conn(0x7fcd04078ad0 msgr2=0x7fcd041045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:41.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.028+0000 7fcd0b228640 1 -- 172.21.15.70:0/3035949810 shutdown_connections 2024-06-04T05:15:41.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:41.028+0000 7fcd0b228640 1 -- 172.21.15.70:0/3035949810 wait complete. 2024-06-04T05:15:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:41 smithi070 ceph-mon[32631]: pgmap v580: 97 pgs: 97 active+clean; 641 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 270 op/s 2024-06-04T05:15:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:41 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:15:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:41 smithi112 ceph-mon[36453]: pgmap v580: 97 pgs: 97 active+clean; 641 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 270 op/s 2024-06-04T05:15:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:41 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:15:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:41 smithi098 ceph-mon[35569]: pgmap v580: 97 pgs: 97 active+clean; 641 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 270 op/s 2024-06-04T05:15:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:41 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:15:41.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 38688 7.76 MB/sec execute 153 sec latency 141.265 ms 2024-06-04T05:15:42.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 38812 7.74 MB/sec execute 154 sec latency 223.740 ms 2024-06-04T05:15:43.547 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:43 smithi098 ceph-mon[35569]: pgmap v581: 97 pgs: 97 active+clean; 639 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.9 MiB/s wr, 412 op/s 2024-06-04T05:15:43.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:43 smithi070 ceph-mon[32631]: pgmap v581: 97 pgs: 97 active+clean; 639 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.9 MiB/s wr, 412 op/s 2024-06-04T05:15:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:43 smithi112 ceph-mon[36453]: pgmap v581: 97 pgs: 97 active+clean; 639 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.9 MiB/s wr, 412 op/s 2024-06-04T05:15:43.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 38928 7.71 MB/sec execute 155 sec latency 238.063 ms 2024-06-04T05:15:44.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 39075 7.73 MB/sec execute 156 sec latency 129.509 ms 2024-06-04T05:15:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:45 smithi070 ceph-mon[32631]: pgmap v582: 97 pgs: 97 active+clean; 639 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 286 op/s 2024-06-04T05:15:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:45 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:15:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:45 smithi112 ceph-mon[36453]: pgmap v582: 97 pgs: 97 active+clean; 639 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 286 op/s 2024-06-04T05:15:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:45 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:15:45.686 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:15:45 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:15:45.503+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000020f [...1f,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ auth{2=1,4=1} fragtree_t(*^2 00*^1 01*^1 10*^1) v46690 ap=2 f(v1847 m2024-06-04T05:15:14.601651+0000 7=7+0) n(v73 rc2024-06-04T05:15:14.601651+0000 b1462582 8=7+1) old_inodes=6 (inest sync->mix g=2,4) caps={15231=pAsLsXsFs/-@37018} | dirtyscattered=0 request=0 lock=0 importing=0 dirfrag=7 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 0x55d6de801700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000020f:[<0x10000000009/PWRPNT v234>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x1000000020f:[<0x10000000009/PWRPNT v46690>,<0x10000000008/~dmtmp v35277>,<0x10000000007/client0 v2548>,<0x10000000006/clients v2101>,<0x10000000005/tmp v1651>,<0x10000000003/client.0 v1596>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v1592>,<0x10000000001/sv_0 v1562>,<0x10000000000/qa v1538>,<0x1/volumes v1487>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:15:14.515063+0000 8=8+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:15:14.601651+0000 b1462582 8=7+1)","memory_value.dirstat":"f(v1847 m2024-06-04T05:15:14.601651+0000 7=7+0)","memory_value.rstat":"n(v73 rc2024-06-04T05:15:14.601651+0000 b1462582 8=7+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:15:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:45 smithi098 ceph-mon[35569]: pgmap v582: 97 pgs: 97 active+clean; 639 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 286 op/s 2024-06-04T05:15:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:45 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:15:45.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 39187 7.71 MB/sec execute 157 sec latency 167.944 ms 2024-06-04T05:15:46.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.294+0000 7f8a27fff700 1 -- 172.21.15.70:0/258062020 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a20009480 con 0x7f8a2810fff0 2024-06-04T05:15:46.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.294+0000 7f8a2f706700 1 -- 172.21.15.70:0/258062020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a2810fff0 msgr2=0x7f8a281123e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:46.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.294+0000 7f8a2f706700 1 --2- 172.21.15.70:0/258062020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a2810fff0 0x7f8a281123e0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f8a2000b0a0 tx=0x7f8a2000b3b0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.295+0000 7f8a2f706700 1 -- 172.21.15.70:0/258062020 shutdown_connections 2024-06-04T05:15:46.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.295+0000 7f8a2f706700 1 --2- 172.21.15.70:0/258062020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a2810fff0 0x7f8a281123e0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.295+0000 7f8a2f706700 1 --2- 172.21.15.70:0/258062020 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a2806b250 0x7f8a2806b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.295+0000 7f8a2f706700 1 --2- 172.21.15.70:0/258062020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a2806bb70 0x7f8a2810fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.295+0000 7f8a2f706700 1 -- 172.21.15.70:0/258062020 >> 172.21.15.70:0/258062020 conn(0x7f8a280793f0 msgr2=0x7f8a28079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:46.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.295+0000 7f8a2f706700 1 -- 172.21.15.70:0/258062020 shutdown_connections 2024-06-04T05:15:46.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.295+0000 7f8a2f706700 1 -- 172.21.15.70:0/258062020 wait complete. 2024-06-04T05:15:46.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.296+0000 7f8a2f706700 1 Processor -- start 2024-06-04T05:15:46.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.296+0000 7f8a2f706700 1 -- start start 2024-06-04T05:15:46.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.296+0000 7f8a2f706700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a2806b250 0x7f8a281a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.296+0000 7f8a2f706700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a2806bb70 0x7f8a281a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.296+0000 7f8a2f706700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a2810fff0 0x7f8a2819bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.296+0000 7f8a2f706700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a281157b0 con 0x7f8a2806bb70 2024-06-04T05:15:46.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.296+0000 7f8a2f706700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a28115630 con 0x7f8a2806b250 2024-06-04T05:15:46.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.296+0000 7f8a2f706700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8a28115930 con 0x7f8a2810fff0 2024-06-04T05:15:46.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.296+0000 7f8a2d4a2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a2806b250 0x7f8a281a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:46.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.296+0000 7f8a2d4a2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a2806b250 0x7f8a281a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:47908/0 (socket says 172.21.15.70:47908) 2024-06-04T05:15:46.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.296+0000 7f8a2d4a2700 1 -- 172.21.15.70:0/407864270 learned_addr learned my addr 172.21.15.70:0/407864270 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:46.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.296+0000 7f8a2dca3700 1 --2- 172.21.15.70:0/407864270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a2810fff0 0x7f8a2819bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:46.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.296+0000 7f8a2cca1700 1 --2- 172.21.15.70:0/407864270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a2806bb70 0x7f8a281a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:46.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.296+0000 7f8a2d4a2700 1 -- 172.21.15.70:0/407864270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a2810fff0 msgr2=0x7f8a2819bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:46.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.296+0000 7f8a2d4a2700 1 --2- 172.21.15.70:0/407864270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a2810fff0 0x7f8a2819bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.297+0000 7f8a2d4a2700 1 -- 172.21.15.70:0/407864270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a2806bb70 msgr2=0x7f8a281a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:46.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.297+0000 7f8a2d4a2700 1 --2- 172.21.15.70:0/407864270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a2806bb70 0x7f8a281a1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.297+0000 7f8a2d4a2700 1 -- 172.21.15.70:0/407864270 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a2819c270 con 0x7f8a2806b250 2024-06-04T05:15:46.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.297+0000 7f8a2d4a2700 1 --2- 172.21.15.70:0/407864270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a2806b250 0x7f8a281a1900 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00e9a0 tx=0x7f8a1c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:46.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.297+0000 7f8a267fc700 1 -- 172.21.15.70:0/407864270 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c00cbd0 con 0x7f8a2806b250 2024-06-04T05:15:46.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.297+0000 7f8a267fc700 1 -- 172.21.15.70:0/407864270 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a1c00cd70 con 0x7f8a2806b250 2024-06-04T05:15:46.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.297+0000 7f8a267fc700 1 -- 172.21.15.70:0/407864270 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c018680 con 0x7f8a2806b250 2024-06-04T05:15:46.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.297+0000 7f8a2f706700 1 -- 172.21.15.70:0/407864270 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a2819c500 con 0x7f8a2806b250 2024-06-04T05:15:46.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.297+0000 7f8a2f706700 1 -- 172.21.15.70:0/407864270 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a2819c930 con 0x7f8a2806b250 2024-06-04T05:15:46.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.301+0000 7f8a267fc700 1 -- 172.21.15.70:0/407864270 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f8a1c018820 con 0x7f8a2806b250 2024-06-04T05:15:46.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.301+0000 7f8a267fc700 1 --2- 172.21.15.70:0/407864270 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8a100b2a70 0x7f8a100b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.301+0000 7f8a267fc700 1 -- 172.21.15.70:0/407864270 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f8a1c0e2d00 con 0x7f8a2806b250 2024-06-04T05:15:46.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.301+0000 7f8a2cca1700 1 --2- 172.21.15.70:0/407864270 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8a100b2a70 0x7f8a100b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:46.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.302+0000 7f8a2cca1700 1 --2- 172.21.15.70:0/407864270 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8a100b2a70 0x7f8a100b4f30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8a28103a70 tx=0x7f8a18009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:46.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.302+0000 7f8a39f9bb80 1 Processor -- start 2024-06-04T05:15:46.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.302+0000 7f8a39f9bb80 1 -- start start 2024-06-04T05:15:46.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.302+0000 7f8a39f9bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308c2bb0 0x55ac308c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.302+0000 7f8a39f9bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308bd130 0x55ac308bd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a39f9bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bdaf0 0x55ac308c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a39f9bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ac30812df0 con 0x55ac308bd130 2024-06-04T05:15:46.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a39f9bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ac30802990 con 0x55ac308bdaf0 2024-06-04T05:15:46.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a39f9bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ac308069d0 con 0x55ac308c2bb0 2024-06-04T05:15:46.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a2d4a2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308c2bb0 0x55ac308c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:46.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a2d4a2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308c2bb0 0x55ac308c2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:58178/0 (socket says 172.21.15.70:58178) 2024-06-04T05:15:46.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a2d4a2700 1 -- 172.21.15.70:0/3888884937 learned_addr learned my addr 172.21.15.70:0/3888884937 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:46.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a2dca3700 1 --2- 172.21.15.70:0/3888884937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bdaf0 0x55ac308c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:46.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a2cca1700 1 --2- 172.21.15.70:0/3888884937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308bd130 0x55ac308bd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:46.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a2d4a2700 1 -- 172.21.15.70:0/3888884937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bdaf0 msgr2=0x55ac308c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:46.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a2d4a2700 1 --2- 172.21.15.70:0/3888884937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bdaf0 0x55ac308c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a2d4a2700 1 -- 172.21.15.70:0/3888884937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308bd130 msgr2=0x55ac308bd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:46.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a2d4a2700 1 --2- 172.21.15.70:0/3888884937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308bd130 0x55ac308bd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a2d4a2700 1 -- 172.21.15.70:0/3888884937 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac307ac030 con 0x55ac308c2bb0 2024-06-04T05:15:46.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a2d4a2700 1 --2- 172.21.15.70:0/3888884937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308c2bb0 0x55ac308c2f90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c0a6030 tx=0x7f8a1c0a97a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae3e571cc49b83a8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:46.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a0f7fe700 1 -- 172.21.15.70:0/3888884937 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c0a9fb0 con 0x55ac308c2bb0 2024-06-04T05:15:46.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a0f7fe700 1 -- 172.21.15.70:0/3888884937 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a1c0ae360 con 0x55ac308c2bb0 2024-06-04T05:15:46.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.303+0000 7f8a0f7fe700 1 -- 172.21.15.70:0/3888884937 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c0a9210 con 0x55ac308c2bb0 2024-06-04T05:15:46.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.304+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/3888884937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308c2bb0 msgr2=0x55ac308c2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:46.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.304+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/3888884937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308c2bb0 0x55ac308c2f90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c0a6030 tx=0x7f8a1c0a97a0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.304+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/3888884937 shutdown_connections 2024-06-04T05:15:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.304+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/3888884937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bdaf0 0x55ac308c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.304+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/3888884937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308c2bb0 0x55ac308c2f90 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.304+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/3888884937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308bd130 0x55ac308bd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.304+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/3888884937 >> 172.21.15.70:0/3888884937 conn(0x55ac307e02c0 msgr2=0x55ac308af460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.304+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/3888884937 shutdown_connections 2024-06-04T05:15:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.304+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/3888884937 wait complete. 2024-06-04T05:15:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.305+0000 7f8a39f9bb80 1 -- start start 2024-06-04T05:15:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.305+0000 7f8a39f9bb80 1 Processor -- start 2024-06-04T05:15:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.305+0000 7f8a39f9bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bd130 0x55ac308c9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.305+0000 7f8a39f9bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308bdaf0 0x55ac308c6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.305+0000 7f8a39f9bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308c2bb0 0x55ac308c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a39f9bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ac30709020 con 0x55ac308c2bb0 2024-06-04T05:15:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a39f9bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ac3076ae20 con 0x55ac308bd130 2024-06-04T05:15:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a39f9bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ac308c9f70 con 0x55ac308bdaf0 2024-06-04T05:15:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a2d4a2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bd130 0x55ac308c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:46.154 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:15:46.154 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:15:46.154 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "50dfa3b3-6f24-4ea4-811e-dbdafd6e928a", 2024-06-04T05:15:46.154 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:15:46.154 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:15:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a2d4a2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bd130 0x55ac308c9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:47936/0 (socket says 172.21.15.70:47936) 2024-06-04T05:15:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a2d4a2700 1 -- 172.21.15.70:0/3295271849 learned_addr learned my addr 172.21.15.70:0/3295271849 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a39f9bb80 10 client.0 _ll_register_callbacks cb 0x55ac3079d9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a2cca1700 1 --2- 172.21.15.70:0/3295271849 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308bdaf0 0x55ac308c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a2dca3700 1 --2- 172.21.15.70:0/3295271849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308c2bb0 0x55ac308c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a2d4a2700 1 -- 172.21.15.70:0/3295271849 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308bdaf0 msgr2=0x55ac308c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a2d4a2700 1 --2- 172.21.15.70:0/3295271849 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308bdaf0 0x55ac308c6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a2d4a2700 1 -- 172.21.15.70:0/3295271849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308c2bb0 msgr2=0x55ac308c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a2d4a2700 1 --2- 172.21.15.70:0/3295271849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308c2bb0 0x55ac308c6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a2d4a2700 1 -- 172.21.15.70:0/3295271849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac30752950 con 0x55ac308bd130 2024-06-04T05:15:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a2d4a2700 1 --2- 172.21.15.70:0/3295271849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bd130 0x55ac308c9660 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c0e09a0 tx=0x7f8a1c0b2c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6446c268b1f15acf server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a0cff9700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:15:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a0cff9700 1 -- 172.21.15.70:0/3295271849 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c0a9fb0 con 0x55ac308bd130 2024-06-04T05:15:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a0cff9700 1 -- 172.21.15.70:0/3295271849 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f8a1c0a65a0 con 0x55ac308bd130 2024-06-04T05:15:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.306+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/3295271849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a1c0ae360 con 0x55ac308bd130 2024-06-04T05:15:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.307+0000 7f8a0cff9700 1 -- 172.21.15.70:0/3295271849 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f8a1c0b2e10 con 0x55ac308bd130 2024-06-04T05:15:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.307+0000 7f8a39f9bb80 10 client.29081 fetch_fsmap learned FSMap version 56 2024-06-04T05:15:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.307+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/3295271849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x55ac307bc7e0 con 0x55ac308bd130 2024-06-04T05:15:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.307+0000 7f8a0cff9700 1 -- 172.21.15.70:0/3295271849 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a1c0ae360 con 0x55ac308bd130 2024-06-04T05:15:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.307+0000 7f8a0cff9700 1 -- 172.21.15.70:0/3295271849 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7f8a1c0a5a40 con 0x55ac308bd130 2024-06-04T05:15:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.307+0000 7f8a39f9bb80 10 client.29081 fetch_fsmap finished waiting for FSMap version 56 2024-06-04T05:15:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.307+0000 7f8a39f9bb80 10 client.29081 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.307+0000 7f8a39f9bb80 20 client.29081 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:46.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.307+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/3295271849 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ac308d67f0 0x55ac308d8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.307+0000 7f8a39f9bb80 4 client.29081 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:15:46.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.307+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/3295271849 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac30814d40 con 0x55ac308d67f0 2024-06-04T05:15:46.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.307+0000 7f8a2cca1700 1 --2- 172.21.15.70:0/3295271849 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ac308d67f0 0x55ac308d8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:46.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.308+0000 7f8a0cff9700 10 client.29081 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:46.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.308+0000 7f8a2cca1700 1 --2- 172.21.15.70:0/3295271849 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ac308d67f0 0x55ac308d8be0 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f510c07669581344 server_cookie=f362bc4760e6cef5 in_seq=0 out_seq=0 2024-06-04T05:15:46.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.309+0000 7f8a0cff9700 1 -- 172.21.15.70:0/3295271849 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ac30814d40 con 0x55ac308d67f0 2024-06-04T05:15:46.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.309+0000 7f8a0cff9700 10 client.29081 handle_command_reply: tid=0 2024-06-04T05:15:46.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.309+0000 7f8a39f9bb80 1 client.29081 shutdown 2024-06-04T05:15:46.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.309+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/3295271849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bd130 msgr2=0x55ac308c9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:46.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.309+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/3295271849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bd130 0x55ac308c9660 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c0e09a0 tx=0x7f8a1c0b2c30 comp rx=0 tx=0).stop 2024-06-04T05:15:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.309+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/3295271849 shutdown_connections 2024-06-04T05:15:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.309+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/3295271849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308c2bb0 0x55ac308c6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.309+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/3295271849 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ac308d67f0 0x55ac308d8be0 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.309+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/3295271849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bd130 0x55ac308c9660 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.309+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/3295271849 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308bdaf0 0x55ac308c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.309+0000 7f8a0cff9700 0 client.29081 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.309+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/3295271849 >> 172.21.15.70:0/3295271849 conn(0x55ac307e02c0 msgr2=0x55ac308aeb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.309+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/3295271849 shutdown_connections 2024-06-04T05:15:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.309+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/3295271849 wait complete. 2024-06-04T05:15:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.309+0000 7f8a39f9bb80 20 client.29081 trim_cache size 0 max 16384 2024-06-04T05:15:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.342+0000 7f8a39f9bb80 1 Processor -- start 2024-06-04T05:15:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.342+0000 7f8a39f9bb80 1 -- start start 2024-06-04T05:15:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.342+0000 7f8a39f9bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308bd130 0x55ac308af2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.343+0000 7f8a2d4a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308bd130 0x55ac308af2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.343+0000 7f8a39f9bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bdaf0 0x55ac308c0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.343+0000 7f8a2d4a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308bd130 0x55ac308af2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56148/0 (socket says 172.21.15.70:56148) 2024-06-04T05:15:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.343+0000 7f8a2d4a2700 1 -- 172.21.15.70:0/1873055302 learned_addr learned my addr 172.21.15.70:0/1873055302 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.343+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/1873055302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308c2bb0 0x55ac308dbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.343+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/1873055302 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a281abba0 con 0x55ac308bd130 2024-06-04T05:15:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.343+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/1873055302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a281ad710 con 0x55ac308bdaf0 2024-06-04T05:15:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.343+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/1873055302 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8a281ac2a0 con 0x55ac308c2bb0 2024-06-04T05:15:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.343+0000 7f8a2dca3700 1 --2- 172.21.15.70:0/1873055302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308c2bb0 0x55ac308dbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.343+0000 7f8a2d4a2700 1 -- 172.21.15.70:0/1873055302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308c2bb0 msgr2=0x55ac308dbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.343+0000 7f8a2d4a2700 1 --2- 172.21.15.70:0/1873055302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308c2bb0 0x55ac308dbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.343+0000 7f8a2d4a2700 1 -- 172.21.15.70:0/1873055302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bdaf0 msgr2=0x55ac308c0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.343+0000 7f8a2d4a2700 1 --2- 172.21.15.70:0/1873055302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bdaf0 0x55ac308c0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.343+0000 7f8a2d4a2700 1 -- 172.21.15.70:0/1873055302 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac308c4ef0 con 0x55ac308bd130 2024-06-04T05:15:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.343+0000 7f8a2d4a2700 1 --2- 172.21.15.70:0/1873055302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308bd130 0x55ac308af2f0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c0a6030 tx=0x7f8a1c0a97a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=714ddad2977cb4d0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.343+0000 7f8a0effd700 1 -- 172.21.15.70:0/1873055302 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c0a5e70 con 0x55ac308bd130 2024-06-04T05:15:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.344+0000 7f8a0effd700 1 -- 172.21.15.70:0/1873055302 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a1c0a9fb0 con 0x55ac308bd130 2024-06-04T05:15:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.344+0000 7f8a0effd700 1 -- 172.21.15.70:0/1873055302 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c0a9210 con 0x55ac308bd130 2024-06-04T05:15:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.344+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/1873055302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308bd130 msgr2=0x55ac308af2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.344+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/1873055302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308bd130 0x55ac308af2f0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c0a6030 tx=0x7f8a1c0a97a0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.345+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/1873055302 shutdown_connections 2024-06-04T05:15:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.345+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/1873055302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308c2bb0 0x55ac308dbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.345+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/1873055302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308bd130 0x55ac308af2f0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.345+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/1873055302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bdaf0 0x55ac308c0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.345+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/1873055302 >> 172.21.15.70:0/1873055302 conn(0x55ac307e02c0 msgr2=0x55ac308c05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.345+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/1873055302 shutdown_connections 2024-06-04T05:15:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.345+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/1873055302 wait complete. 2024-06-04T05:15:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.345+0000 7f8a39f9bb80 1 -- start start 2024-06-04T05:15:46.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.345+0000 7f8a39f9bb80 1 Processor -- start 2024-06-04T05:15:46.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.346+0000 7f8a39f9bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308bd130 0x55ac308da0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.346+0000 7f8a39f9bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bdaf0 0x55ac308da5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.346+0000 7f8a39f9bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308c2bb0 0x55ac30940030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.346+0000 7f8a39f9bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a281ad040 con 0x55ac308c2bb0 2024-06-04T05:15:46.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.346+0000 7f8a39f9bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a281ac970 con 0x55ac308bdaf0 2024-06-04T05:15:46.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.346+0000 7f8a39f9bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8a281a9b60 con 0x55ac308bd130 2024-06-04T05:15:46.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.346+0000 7f8a39f9bb80 10 client.0 _ll_register_callbacks cb 0x55ac308c26b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:46.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.346+0000 7f8a2d4a2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308bd130 0x55ac308da0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:46.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.346+0000 7f8a2d4a2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308bd130 0x55ac308da0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:58196/0 (socket says 172.21.15.70:58196) 2024-06-04T05:15:46.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.346+0000 7f8a2cca1700 1 --2- 172.21.15.70:0/812502062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bdaf0 0x55ac308da5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:46.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.346+0000 7f8a2d4a2700 1 -- 172.21.15.70:0/812502062 learned_addr learned my addr 172.21.15.70:0/812502062 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:46.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.346+0000 7f8a2dca3700 1 --2- 172.21.15.70:0/812502062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308c2bb0 0x55ac30940030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.347+0000 7f8a2cca1700 1 -- 172.21.15.70:0/812502062 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308bd130 msgr2=0x55ac308da0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.347+0000 7f8a2cca1700 1 --2- 172.21.15.70:0/812502062 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308bd130 0x55ac308da0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.347+0000 7f8a2cca1700 1 -- 172.21.15.70:0/812502062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308c2bb0 msgr2=0x55ac30940030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.347+0000 7f8a2cca1700 1 --2- 172.21.15.70:0/812502062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308c2bb0 0x55ac30940030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.347+0000 7f8a2cca1700 1 -- 172.21.15.70:0/812502062 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac309405c0 con 0x55ac308bdaf0 2024-06-04T05:15:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.347+0000 7f8a2cca1700 1 --2- 172.21.15.70:0/812502062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bdaf0 0x55ac308da5f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55ac308ca080 tx=0x7f8a18004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3aad1b750b178e4e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.347+0000 7f8a0cff9700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:15:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.347+0000 7f8a0cff9700 1 -- 172.21.15.70:0/812502062 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1802e060 con 0x55ac308bdaf0 2024-06-04T05:15:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.347+0000 7f8a0cff9700 1 -- 172.21.15.70:0/812502062 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f8a18011650 con 0x55ac308bdaf0 2024-06-04T05:15:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.347+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/812502062 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a1c0a9fb0 con 0x55ac308bdaf0 2024-06-04T05:15:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.348+0000 7f8a0cff9700 1 -- 172.21.15.70:0/812502062 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f8a18011850 con 0x55ac308bdaf0 2024-06-04T05:15:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.348+0000 7f8a39f9bb80 10 client.29084 fetch_fsmap learned FSMap version 56 2024-06-04T05:15:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.348+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/812502062 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x55ac30941990 con 0x55ac308bdaf0 2024-06-04T05:15:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.348+0000 7f8a0cff9700 1 -- 172.21.15.70:0/812502062 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a1c0a9fb0 con 0x55ac308bdaf0 2024-06-04T05:15:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.348+0000 7f8a0cff9700 1 -- 172.21.15.70:0/812502062 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7f8a1801eab0 con 0x55ac308bdaf0 2024-06-04T05:15:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.348+0000 7f8a39f9bb80 10 client.29084 fetch_fsmap finished waiting for FSMap version 56 2024-06-04T05:15:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.348+0000 7f8a39f9bb80 10 client.29084 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.348+0000 7f8a39f9bb80 20 client.29084 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.349+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/812502062 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ac30942000 0x55ac30944420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.349+0000 7f8a39f9bb80 4 client.29084 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:15:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.349+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/812502062 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ac3080e520 con 0x55ac30942000 2024-06-04T05:15:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.349+0000 7f8a2d4a2700 1 --2- 172.21.15.70:0/812502062 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ac30942000 0x55ac30944420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.349+0000 7f8a0cff9700 10 client.29084 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.349+0000 7f8a2d4a2700 1 --2- 172.21.15.70:0/812502062 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ac30942000 0x55ac30944420 crc :-1 s=READY pgs=400 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96f97191c93af09c server_cookie=5827a64fc5926426 in_seq=0 out_seq=0 2024-06-04T05:15:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.350+0000 7f8a0cff9700 1 -- 172.21.15.70:0/812502062 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ac3080e520 con 0x55ac30942000 2024-06-04T05:15:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.350+0000 7f8a0cff9700 10 client.29084 handle_command_reply: tid=0 2024-06-04T05:15:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.350+0000 7f8a39f9bb80 1 client.29084 shutdown 2024-06-04T05:15:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.350+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/812502062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bdaf0 msgr2=0x55ac308da5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.350+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/812502062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bdaf0 0x55ac308da5f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55ac308ca080 tx=0x7f8a18004750 comp rx=0 tx=0).stop 2024-06-04T05:15:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.350+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/812502062 shutdown_connections 2024-06-04T05:15:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.350+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/812502062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ac308c2bb0 0x55ac30940030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.350+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/812502062 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ac308bd130 0x55ac308da0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.350+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/812502062 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ac30942000 0x55ac30944420 crc :-1 s=READY pgs=400 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.350+0000 7f8a39f9bb80 1 --2- 172.21.15.70:0/812502062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac308bdaf0 0x55ac308da5f0 secure :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55ac308ca080 tx=0x7f8a18004750 comp rx=0 tx=0).stop 2024-06-04T05:15:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.350+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/812502062 >> 172.21.15.70:0/812502062 conn(0x55ac307e02c0 msgr2=0x55ac308c0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.350+0000 7f8a0cff9700 0 client.29084 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.351+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/812502062 shutdown_connections 2024-06-04T05:15:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.351+0000 7f8a39f9bb80 1 -- 172.21.15.70:0/812502062 wait complete. 2024-06-04T05:15:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.351+0000 7f8a39f9bb80 20 client.29084 trim_cache size 0 max 16384 2024-06-04T05:15:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.352+0000 7f8a2f706700 1 -- 172.21.15.70:0/407864270 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8a100b2a70 msgr2=0x7f8a100b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.352+0000 7f8a2f706700 1 --2- 172.21.15.70:0/407864270 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8a100b2a70 0x7f8a100b4f30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8a28103a70 tx=0x7f8a18009260 comp rx=0 tx=0).stop 2024-06-04T05:15:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.352+0000 7f8a2f706700 1 -- 172.21.15.70:0/407864270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a2806b250 msgr2=0x7f8a281a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.352+0000 7f8a2f706700 1 --2- 172.21.15.70:0/407864270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a2806b250 0x7f8a281a1900 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00e9a0 tx=0x7f8a1c00ecb0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.353+0000 7f8a2f706700 1 -- 172.21.15.70:0/407864270 shutdown_connections 2024-06-04T05:15:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.353+0000 7f8a2f706700 1 --2- 172.21.15.70:0/407864270 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8a100b2a70 0x7f8a100b4f30 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.353+0000 7f8a2f706700 1 --2- 172.21.15.70:0/407864270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a2810fff0 0x7f8a2819bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.353+0000 7f8a2f706700 1 --2- 172.21.15.70:0/407864270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a2806b250 0x7f8a281a1900 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.353+0000 7f8a2f706700 1 --2- 172.21.15.70:0/407864270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a2806bb70 0x7f8a281a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.353+0000 7f8a2f706700 1 -- 172.21.15.70:0/407864270 >> 172.21.15.70:0/407864270 conn(0x7f8a280793f0 msgr2=0x7f8a28111500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.355+0000 7f8a2f706700 1 -- 172.21.15.70:0/407864270 shutdown_connections 2024-06-04T05:15:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:45.356+0000 7f8a2f706700 1 -- 172.21.15.70:0/407864270 wait complete. 2024-06-04T05:15:46.315 INFO:tasks.quiescer.fs.[cephfs]:Total duration (53.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-06-04T05:15:46.315 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 26.85 2024-06-04T05:15:46.315 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:15:46.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:15:46.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:15:46.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:15:46.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:15:46.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:15:46.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:15:46.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:15:46.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:15:46.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:15:46.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:15:46.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:15:46.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:15:46.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:15:46.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:15:46.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT) see mds.i log and `damage ls` output for details 2024-06-04T05:15:46.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:46 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:15:46.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:15:46.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:15:46.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:15:46.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:15:46.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:15:46.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:15:46.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:15:46.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:15:46.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:15:46.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:15:46.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:15:46.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:15:46.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:15:46.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT) see mds.i log and `damage ls` output for details 2024-06-04T05:15:46.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:46 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:15:46.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:15:46.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:15:46.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:15:46.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:15:46.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:15:46.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:15:46.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:15:46.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:15:46.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:15:46.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:15:46.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:15:46.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:15:46.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:15:46.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT) see mds.i log and `damage ls` output for details 2024-06-04T05:15:46.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:46 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:15:46.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 39353 7.73 MB/sec execute 158 sec latency 119.194 ms 2024-06-04T05:15:47.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:47 smithi070 ceph-mon[32631]: pgmap v583: 97 pgs: 97 active+clean; 648 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.4 MiB/s wr, 418 op/s 2024-06-04T05:15:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:47 smithi112 ceph-mon[36453]: pgmap v583: 97 pgs: 97 active+clean; 648 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.4 MiB/s wr, 418 op/s 2024-06-04T05:15:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:47 smithi098 ceph-mon[35569]: pgmap v583: 97 pgs: 97 active+clean; 648 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.4 MiB/s wr, 418 op/s 2024-06-04T05:15:47.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 39507 7.76 MB/sec execute 159 sec latency 209.993 ms 2024-06-04T05:15:48.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 39684 7.80 MB/sec execute 160 sec latency 118.472 ms 2024-06-04T05:15:49.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:49 smithi070 ceph-mon[32631]: pgmap v584: 97 pgs: 97 active+clean; 648 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 273 op/s 2024-06-04T05:15:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:49 smithi112 ceph-mon[36453]: pgmap v584: 97 pgs: 97 active+clean; 648 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 273 op/s 2024-06-04T05:15:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:49 smithi098 ceph-mon[35569]: pgmap v584: 97 pgs: 97 active+clean; 648 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 273 op/s 2024-06-04T05:15:49.786 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:15:49.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 39854 7.81 MB/sec execute 161 sec latency 135.024 ms 2024-06-04T05:15:49.824 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:15:50.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:50 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:15:50.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:50 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:15:50.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:50 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:15:50.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:50 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:15:50.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:50 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:15:50.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:50 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:15:50.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:50 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:15:50.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:50 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:15:50.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:50 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:15:50.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 40033 7.85 MB/sec execute 162 sec latency 63.017 ms 2024-06-04T05:15:51.202 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:15:51.559 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:51 smithi070 ceph-mon[32631]: pgmap v585: 97 pgs: 97 active+clean; 648 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 273 op/s 2024-06-04T05:15:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:51 smithi112 ceph-mon[36453]: pgmap v585: 97 pgs: 97 active+clean; 648 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 273 op/s 2024-06-04T05:15:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:51 smithi098 ceph-mon[35569]: pgmap v585: 97 pgs: 97 active+clean; 648 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 273 op/s 2024-06-04T05:15:51.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 40187 7.86 MB/sec execute 163 sec latency 93.264 ms 2024-06-04T05:15:52.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.837+0000 7f8c622f0700 1 -- 172.21.15.70:0/2630561531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c103840 msgr2=0x7f8c5c103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:52.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.837+0000 7f8c622f0700 1 --2- 172.21.15.70:0/2630561531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c103840 0x7f8c5c103cc0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f8c50009990 tx=0x7f8c50009ca0 comp rx=0 tx=0).stop 2024-06-04T05:15:52.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.837+0000 7f8c622f0700 1 -- 172.21.15.70:0/2630561531 shutdown_connections 2024-06-04T05:15:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.837+0000 7f8c622f0700 1 --2- 172.21.15.70:0/2630561531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c5c104200 0x7f8c5c1104b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.837+0000 7f8c622f0700 1 --2- 172.21.15.70:0/2630561531 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c5c1085a0 0x7f8c5c108980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.837+0000 7f8c622f0700 1 --2- 172.21.15.70:0/2630561531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c103840 0x7f8c5c103cc0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.837+0000 7f8c622f0700 1 -- 172.21.15.70:0/2630561531 >> 172.21.15.70:0/2630561531 conn(0x7f8c5c07b440 msgr2=0x7f8c5c07b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.837+0000 7f8c622f0700 1 -- 172.21.15.70:0/2630561531 shutdown_connections 2024-06-04T05:15:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.837+0000 7f8c622f0700 1 -- 172.21.15.70:0/2630561531 wait complete. 2024-06-04T05:15:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.838+0000 7f8c622f0700 1 Processor -- start 2024-06-04T05:15:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.838+0000 7f8c622f0700 1 -- start start 2024-06-04T05:15:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.839+0000 7f8c622f0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c5c103840 0x7f8c5c075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.839+0000 7f8c622f0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c104200 0x7f8c5c075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.839+0000 7f8c622f0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c5c1085a0 0x7f8c5c0762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.839+0000 7f8c5bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c5c103840 0x7f8c5c075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.839+0000 7f8c622f0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5c113630 con 0x7f8c5c104200 2024-06-04T05:15:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.839+0000 7f8c622f0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5c1134b0 con 0x7f8c5c103840 2024-06-04T05:15:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.839+0000 7f8c622f0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5c1137b0 con 0x7f8c5c1085a0 2024-06-04T05:15:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.839+0000 7f8c5bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c5c103840 0x7f8c5c075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33646/0 (socket says 172.21.15.70:33646) 2024-06-04T05:15:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.839+0000 7f8c5bfff700 1 -- 172.21.15.70:0/3257717496 learned_addr learned my addr 172.21.15.70:0/3257717496 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.839+0000 7f8c6088d700 1 --2- 172.21.15.70:0/3257717496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c5c1085a0 0x7f8c5c0762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.839+0000 7f8c5b7fe700 1 --2- 172.21.15.70:0/3257717496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c104200 0x7f8c5c075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.840+0000 7f8c6088d700 1 -- 172.21.15.70:0/3257717496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c5c103840 msgr2=0x7f8c5c075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.840+0000 7f8c6088d700 1 --2- 172.21.15.70:0/3257717496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c5c103840 0x7f8c5c075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.840+0000 7f8c6088d700 1 -- 172.21.15.70:0/3257717496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c104200 msgr2=0x7f8c5c075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.840+0000 7f8c6088d700 1 --2- 172.21.15.70:0/3257717496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c104200 0x7f8c5c075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.840+0000 7f8c6088d700 1 -- 172.21.15.70:0/3257717496 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c5c1ac4b0 con 0x7f8c5c1085a0 2024-06-04T05:15:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.840+0000 7f8c6088d700 1 --2- 172.21.15.70:0/3257717496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c5c1085a0 0x7f8c5c0762c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400efb0 tx=0x7f8c5400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.840+0000 7f8c597fa700 1 -- 172.21.15.70:0/3257717496 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c54003a00 con 0x7f8c5c1085a0 2024-06-04T05:15:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.840+0000 7f8c622f0700 1 -- 172.21.15.70:0/3257717496 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c5c1ac740 con 0x7f8c5c1085a0 2024-06-04T05:15:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.840+0000 7f8c597fa700 1 -- 172.21.15.70:0/3257717496 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8c5400ce30 con 0x7f8c5c1085a0 2024-06-04T05:15:52.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.840+0000 7f8c597fa700 1 -- 172.21.15.70:0/3257717496 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c540046b0 con 0x7f8c5c1085a0 2024-06-04T05:15:52.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.840+0000 7f8c622f0700 1 -- 172.21.15.70:0/3257717496 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c5c1acc50 con 0x7f8c5c1085a0 2024-06-04T05:15:52.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.842+0000 7f8c622f0700 1 -- 172.21.15.70:0/3257717496 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c5c10bb70 con 0x7f8c5c1085a0 2024-06-04T05:15:52.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.843+0000 7f8c597fa700 1 -- 172.21.15.70:0/3257717496 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f8c54012050 con 0x7f8c5c1085a0 2024-06-04T05:15:52.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.844+0000 7f8c597fa700 1 --2- 172.21.15.70:0/3257717496 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8c440b2a70 0x7f8c440b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:52.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.845+0000 7f8c5bfff700 1 --2- 172.21.15.70:0/3257717496 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8c440b2a70 0x7f8c440b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:52.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.845+0000 7f8c597fa700 1 -- 172.21.15.70:0/3257717496 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f8c540e2060 con 0x7f8c5c1085a0 2024-06-04T05:15:52.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.845+0000 7f8c5bfff700 1 --2- 172.21.15.70:0/3257717496 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8c440b2a70 0x7f8c440b4f30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c009ce0 tx=0x7f8c4c009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:52.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:51.851+0000 7f8c597fa700 1 -- 172.21.15.70:0/3257717496 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8c54022020 con 0x7f8c5c1085a0 2024-06-04T05:15:52.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:52.283+0000 7f8c622f0700 1 -- 172.21.15.70:0/3257717496 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c5c076d70 con 0x7f8c5c1085a0 2024-06-04T05:15:52.287 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:15:52.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:52.284+0000 7f8c597fa700 1 -- 172.21.15.70:0/3257717496 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+11930 (secure 0 0 0) 0x7f8c54010040 con 0x7f8c5c1085a0 2024-06-04T05:15:52.288 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":56,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:15:40.005565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:15:52.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:52.287+0000 7f8c622f0700 1 -- 172.21.15.70:0/3257717496 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8c440b2a70 msgr2=0x7f8c440b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:52.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:52.287+0000 7f8c622f0700 1 --2- 172.21.15.70:0/3257717496 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8c440b2a70 0x7f8c440b4f30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c009ce0 tx=0x7f8c4c009260 comp rx=0 tx=0).stop 2024-06-04T05:15:52.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:52.287+0000 7f8c622f0700 1 -- 172.21.15.70:0/3257717496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c5c1085a0 msgr2=0x7f8c5c0762c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:52.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:52.287+0000 7f8c622f0700 1 --2- 172.21.15.70:0/3257717496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c5c1085a0 0x7f8c5c0762c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400efb0 tx=0x7f8c5400c3a0 comp rx=0 tx=0).stop 2024-06-04T05:15:52.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:52.288+0000 7f8c622f0700 1 -- 172.21.15.70:0/3257717496 shutdown_connections 2024-06-04T05:15:52.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:52.288+0000 7f8c622f0700 1 --2- 172.21.15.70:0/3257717496 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8c440b2a70 0x7f8c440b4f30 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:52.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:52.288+0000 7f8c622f0700 1 --2- 172.21.15.70:0/3257717496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c5c1085a0 0x7f8c5c0762c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:52.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:52.288+0000 7f8c622f0700 1 --2- 172.21.15.70:0/3257717496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c5c103840 0x7f8c5c075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:52.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:52.288+0000 7f8c622f0700 1 --2- 172.21.15.70:0/3257717496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c104200 0x7f8c5c075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:52.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:52.288+0000 7f8c622f0700 1 -- 172.21.15.70:0/3257717496 >> 172.21.15.70:0/3257717496 conn(0x7f8c5c07b440 msgr2=0x7f8c5c101990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:52.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:52.288+0000 7f8c622f0700 1 -- 172.21.15.70:0/3257717496 shutdown_connections 2024-06-04T05:15:52.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:52.288+0000 7f8c622f0700 1 -- 172.21.15.70:0/3257717496 wait complete. 2024-06-04T05:15:52.292 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 56 2024-06-04T05:15:52.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:52 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3257717496' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:15:52.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:52 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3257717496' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:15:52.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:52 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3257717496' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:15:52.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 40332 7.88 MB/sec execute 164 sec latency 71.884 ms 2024-06-04T05:15:53.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:53 smithi070 ceph-mon[32631]: pgmap v586: 97 pgs: 97 active+clean; 664 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.5 MiB/s wr, 378 op/s 2024-06-04T05:15:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:53 smithi112 ceph-mon[36453]: pgmap v586: 97 pgs: 97 active+clean; 664 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.5 MiB/s wr, 378 op/s 2024-06-04T05:15:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:53 smithi098 ceph-mon[35569]: pgmap v586: 97 pgs: 97 active+clean; 664 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.5 MiB/s wr, 378 op/s 2024-06-04T05:15:53.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 40504 7.88 MB/sec execute 165 sec latency 98.609 ms 2024-06-04T05:15:53.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.819+0000 7fca1c8b9700 1 -- 172.21.15.70:0/2487102282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca18101b30 msgr2=0x7fca18101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:53.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.819+0000 7fca1c8b9700 1 --2- 172.21.15.70:0/2487102282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca18101b30 0x7fca18101fb0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fca0800b0d0 tx=0x7fca0800b3e0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.820+0000 7fca1c8b9700 1 -- 172.21.15.70:0/2487102282 shutdown_connections 2024-06-04T05:15:53.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.820+0000 7fca1c8b9700 1 --2- 172.21.15.70:0/2487102282 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca181026e0 0x7fca18116ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.820+0000 7fca1c8b9700 1 --2- 172.21.15.70:0/2487102282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca18104450 0x7fca18104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.820+0000 7fca1c8b9700 1 --2- 172.21.15.70:0/2487102282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca18101b30 0x7fca18101fb0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.820+0000 7fca1c8b9700 1 -- 172.21.15.70:0/2487102282 >> 172.21.15.70:0/2487102282 conn(0x7fca18078ab0 msgr2=0x7fca180ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:53.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.820+0000 7fca1c8b9700 1 -- 172.21.15.70:0/2487102282 shutdown_connections 2024-06-04T05:15:53.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.820+0000 7fca1c8b9700 1 -- 172.21.15.70:0/2487102282 wait complete. 2024-06-04T05:15:53.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.821+0000 7fca1c8b9700 1 Processor -- start 2024-06-04T05:15:53.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.821+0000 7fca1c8b9700 1 -- start start 2024-06-04T05:15:53.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.821+0000 7fca1c8b9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca18101b30 0x7fca18074d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.821+0000 7fca1c8b9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca181026e0 0x7fca1806de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.821+0000 7fca1c8b9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca18104450 0x7fca1806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.821+0000 7fca1c8b9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fca18119c60 con 0x7fca18101b30 2024-06-04T05:15:53.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.821+0000 7fca1c8b9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fca18119ae0 con 0x7fca181026e0 2024-06-04T05:15:53.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.821+0000 7fca1c8b9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fca18119de0 con 0x7fca18104450 2024-06-04T05:15:53.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.821+0000 7fca1659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca18101b30 0x7fca18074d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:53.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.821+0000 7fca15d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca181026e0 0x7fca1806de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:53.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.822+0000 7fca16d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca18104450 0x7fca1806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:53.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.822+0000 7fca1659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca18101b30 0x7fca18074d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35568/0 (socket says 172.21.15.70:35568) 2024-06-04T05:15:53.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.822+0000 7fca16d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca18104450 0x7fca1806e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:53560/0 (socket says 172.21.15.70:53560) 2024-06-04T05:15:53.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.822+0000 7fca1659c700 1 -- 172.21.15.70:0/1264825898 learned_addr learned my addr 172.21.15.70:0/1264825898 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:53.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.822+0000 7fca1659c700 1 -- 172.21.15.70:0/1264825898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca18104450 msgr2=0x7fca1806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:53.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.822+0000 7fca1659c700 1 --2- 172.21.15.70:0/1264825898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca18104450 0x7fca1806e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.822+0000 7fca1659c700 1 -- 172.21.15.70:0/1264825898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca181026e0 msgr2=0x7fca1806de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:53.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.822+0000 7fca1659c700 1 --2- 172.21.15.70:0/1264825898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca181026e0 0x7fca1806de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.822+0000 7fca1659c700 1 -- 172.21.15.70:0/1264825898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1806e990 con 0x7fca18101b30 2024-06-04T05:15:53.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.822+0000 7fca1659c700 1 --2- 172.21.15.70:0/1264825898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca18101b30 0x7fca18074d30 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fca1000e9a0 tx=0x7fca1000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:53.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.822+0000 7fca0f7fe700 1 -- 172.21.15.70:0/1264825898 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1000cbd0 con 0x7fca18101b30 2024-06-04T05:15:53.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.822+0000 7fca0f7fe700 1 -- 172.21.15.70:0/1264825898 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca1000cd70 con 0x7fca18101b30 2024-06-04T05:15:53.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.822+0000 7fca0f7fe700 1 -- 172.21.15.70:0/1264825898 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca10018680 con 0x7fca18101b30 2024-06-04T05:15:53.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.822+0000 7fca1c8b9700 1 -- 172.21.15.70:0/1264825898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca18072860 con 0x7fca18101b30 2024-06-04T05:15:53.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.823+0000 7fca1c8b9700 1 -- 172.21.15.70:0/1264825898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca18072da0 con 0x7fca18101b30 2024-06-04T05:15:53.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.826+0000 7fca0f7fe700 1 -- 172.21.15.70:0/1264825898 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fca10018820 con 0x7fca18101b30 2024-06-04T05:15:53.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.827+0000 7fca0f7fe700 1 --2- 172.21.15.70:0/1264825898 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fca040b2b40 0x7fca040b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.827+0000 7fca0f7fe700 1 -- 172.21.15.70:0/1264825898 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7fca100e2830 con 0x7fca18101b30 2024-06-04T05:15:53.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.827+0000 7fca15d9b700 1 --2- 172.21.15.70:0/1264825898 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fca040b2b40 0x7fca040b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:53.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.827+0000 7fca2714eb80 1 Processor -- start 2024-06-04T05:15:53.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.827+0000 7fca2714eb80 1 -- start start 2024-06-04T05:15:53.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca15d9b700 1 --2- 172.21.15.70:0/1264825898 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fca040b2b40 0x7fca040b5000 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fca1806f580 tx=0x7fca0800b040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:53.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca2714eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e2860 0x55cbfe8e2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca2714eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e3210 0x55cbfe8e6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca2714eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e6f60 0x55cbfe8e73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca2714eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cbfe841c90 con 0x55cbfe8e2860 2024-06-04T05:15:53.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca2714eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cbfe831830 con 0x55cbfe8e3210 2024-06-04T05:15:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca2714eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cbfe835870 con 0x55cbfe8e6f60 2024-06-04T05:15:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca15d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e3210 0x55cbfe8e6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca15d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e3210 0x55cbfe8e6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33680/0 (socket says 172.21.15.70:33680) 2024-06-04T05:15:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca15d9b700 1 -- 172.21.15.70:0/1915250309 learned_addr learned my addr 172.21.15.70:0/1915250309 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca1659c700 1 --2- 172.21.15.70:0/1915250309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e2860 0x55cbfe8e2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca16d9d700 1 --2- 172.21.15.70:0/1915250309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e6f60 0x55cbfe8e73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca15d9b700 1 -- 172.21.15.70:0/1915250309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e6f60 msgr2=0x55cbfe8e73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca15d9b700 1 --2- 172.21.15.70:0/1915250309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e6f60 0x55cbfe8e73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca15d9b700 1 -- 172.21.15.70:0/1915250309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e2860 msgr2=0x55cbfe8e2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca15d9b700 1 --2- 172.21.15.70:0/1915250309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e2860 0x55cbfe8e2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca15d9b700 1 -- 172.21.15.70:0/1915250309 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbfe7818d0 con 0x55cbfe8e3210 2024-06-04T05:15:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.828+0000 7fca16d9d700 1 --2- 172.21.15.70:0/1915250309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e6f60 0x55cbfe8e73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:15:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.829+0000 7fca15d9b700 1 --2- 172.21.15.70:0/1915250309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e3210 0x55cbfe8e6940 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fca08015000 tx=0x7fca0801c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6176b7bf6c59866d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.829+0000 7fca0cff9700 1 -- 172.21.15.70:0/1915250309 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca08022070 con 0x55cbfe8e3210 2024-06-04T05:15:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.829+0000 7fca0cff9700 1 -- 172.21.15.70:0/1915250309 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca0802e070 con 0x55cbfe8e3210 2024-06-04T05:15:53.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.829+0000 7fca0cff9700 1 -- 172.21.15.70:0/1915250309 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca08029b20 con 0x55cbfe8e3210 2024-06-04T05:15:53.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.830+0000 7fca2714eb80 1 -- 172.21.15.70:0/1915250309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e3210 msgr2=0x55cbfe8e6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:53.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.830+0000 7fca2714eb80 1 --2- 172.21.15.70:0/1915250309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e3210 0x55cbfe8e6940 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fca08015000 tx=0x7fca0801c000 comp rx=0 tx=0).stop 2024-06-04T05:15:53.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.830+0000 7fca2714eb80 1 -- 172.21.15.70:0/1915250309 shutdown_connections 2024-06-04T05:15:53.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.830+0000 7fca2714eb80 1 --2- 172.21.15.70:0/1915250309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e6f60 0x55cbfe8e73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.830+0000 7fca2714eb80 1 --2- 172.21.15.70:0/1915250309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e2860 0x55cbfe8e2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.830+0000 7fca2714eb80 1 --2- 172.21.15.70:0/1915250309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e3210 0x55cbfe8e6940 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.830+0000 7fca2714eb80 1 -- 172.21.15.70:0/1915250309 >> 172.21.15.70:0/1915250309 conn(0x55cbfe810080 msgr2=0x55cbfe8dce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:53.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.830+0000 7fca2714eb80 1 -- 172.21.15.70:0/1915250309 shutdown_connections 2024-06-04T05:15:53.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.830+0000 7fca2714eb80 1 -- 172.21.15.70:0/1915250309 wait complete. 2024-06-04T05:15:53.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.831+0000 7fca2714eb80 1 -- start start 2024-06-04T05:15:53.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.831+0000 7fca2714eb80 1 Processor -- start 2024-06-04T05:15:53.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.831+0000 7fca2714eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e2860 0x55cbfe8ee330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.831+0000 7fca2714eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e3210 0x55cbfe8eb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.831+0000 7fca2714eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e6f60 0x55cbfe8ebb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.831+0000 7fca2714eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cbfe737fa0 con 0x55cbfe8e2860 2024-06-04T05:15:53.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.831+0000 7fca2714eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cbfe7d37c0 con 0x55cbfe8e3210 2024-06-04T05:15:53.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.831+0000 7fca2714eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cbfe8ef160 con 0x55cbfe8e6f60 2024-06-04T05:15:53.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.831+0000 7fca2714eb80 10 client.0 _ll_register_callbacks cb 0x55cbfe7cc920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:53.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.831+0000 7fca1659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e2860 0x55cbfe8ee330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:53.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.831+0000 7fca1659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e2860 0x55cbfe8ee330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35578/0 (socket says 172.21.15.70:35578) 2024-06-04T05:15:53.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.831+0000 7fca1659c700 1 -- 172.21.15.70:0/187314053 learned_addr learned my addr 172.21.15.70:0/187314053 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:53.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.831+0000 7fca16d9d700 1 --2- 172.21.15.70:0/187314053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e6f60 0x55cbfe8ebb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:53.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.832+0000 7fca15d9b700 1 --2- 172.21.15.70:0/187314053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e3210 0x55cbfe8eb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:53.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.832+0000 7fca16d9d700 1 -- 172.21.15.70:0/187314053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e3210 msgr2=0x55cbfe8eb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:53.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.832+0000 7fca16d9d700 1 --2- 172.21.15.70:0/187314053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e3210 0x55cbfe8eb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.832+0000 7fca16d9d700 1 -- 172.21.15.70:0/187314053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e2860 msgr2=0x55cbfe8ee330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:53.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.832+0000 7fca16d9d700 1 --2- 172.21.15.70:0/187314053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e2860 0x55cbfe8ee330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.832+0000 7fca16d9d700 1 -- 172.21.15.70:0/187314053 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbfe7d0690 con 0x55cbfe8e6f60 2024-06-04T05:15:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.832+0000 7fca15d9b700 1 --2- 172.21.15.70:0/187314053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e3210 0x55cbfe8eb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:15:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.832+0000 7fca16d9d700 1 --2- 172.21.15.70:0/187314053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e6f60 0x55cbfe8ebb70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fca18102b10 tx=0x7fca000137b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dd6ae62db4c680f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.832+0000 7fc9fdffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:15:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.832+0000 7fc9fdffb700 1 -- 172.21.15.70:0/187314053 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca00004230 con 0x55cbfe8e6f60 2024-06-04T05:15:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.832+0000 7fca2714eb80 1 -- 172.21.15.70:0/187314053 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca0802e070 con 0x55cbfe8e6f60 2024-06-04T05:15:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.832+0000 7fc9fdffb700 1 -- 172.21.15.70:0/187314053 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7fca000043d0 con 0x55cbfe8e6f60 2024-06-04T05:15:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.833+0000 7fc9fdffb700 1 -- 172.21.15.70:0/187314053 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7fca00003680 con 0x55cbfe8e6f60 2024-06-04T05:15:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.833+0000 7fca2714eb80 10 client.29209 fetch_fsmap learned FSMap version 56 2024-06-04T05:15:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.833+0000 7fca2714eb80 1 -- 172.21.15.70:0/187314053 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x55cbfe90a8f0 con 0x55cbfe8e6f60 2024-06-04T05:15:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.833+0000 7fc9fdffb700 1 -- 172.21.15.70:0/187314053 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca0802e070 con 0x55cbfe8e6f60 2024-06-04T05:15:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.833+0000 7fc9fdffb700 1 -- 172.21.15.70:0/187314053 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7fca00023070 con 0x55cbfe8e6f60 2024-06-04T05:15:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.833+0000 7fca2714eb80 10 client.29209 fetch_fsmap finished waiting for FSMap version 56 2024-06-04T05:15:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.833+0000 7fca2714eb80 10 client.29209 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.833+0000 7fca2714eb80 20 client.29209 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:53.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.833+0000 7fca2714eb80 1 --2- 172.21.15.70:0/187314053 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cbfe90ced0 0x55cbfe90f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.833+0000 7fca2714eb80 4 client.29209 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:15:53.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.833+0000 7fca2714eb80 1 -- 172.21.15.70:0/187314053 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbfe8ae9e0 con 0x55cbfe90ced0 2024-06-04T05:15:53.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.833+0000 7fca1659c700 1 --2- 172.21.15.70:0/187314053 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cbfe90ced0 0x55cbfe90f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:53.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.834+0000 7fca1659c700 1 --2- 172.21.15.70:0/187314053 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cbfe90ced0 0x55cbfe90f2f0 crc :-1 s=READY pgs=401 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8dc4e540bc76146 server_cookie=c4954ff43b1652aa in_seq=0 out_seq=0 2024-06-04T05:15:53.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.834+0000 7fc9fdffb700 10 client.29209 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:53.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.834+0000 7fc9fdffb700 1 -- 172.21.15.70:0/187314053 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cbfe8ae9e0 con 0x55cbfe90ced0 2024-06-04T05:15:53.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.834+0000 7fc9fdffb700 10 client.29209 handle_command_reply: tid=0 2024-06-04T05:15:53.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.834+0000 7fca2714eb80 1 client.29209 shutdown 2024-06-04T05:15:53.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.835+0000 7fca2714eb80 1 -- 172.21.15.70:0/187314053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e6f60 msgr2=0x55cbfe8ebb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:53.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.835+0000 7fca2714eb80 1 --2- 172.21.15.70:0/187314053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e6f60 0x55cbfe8ebb70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fca18102b10 tx=0x7fca000137b0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.835+0000 7fca2714eb80 1 -- 172.21.15.70:0/187314053 shutdown_connections 2024-06-04T05:15:53.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.835+0000 7fca2714eb80 1 --2- 172.21.15.70:0/187314053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e6f60 0x55cbfe8ebb70 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.835+0000 7fca2714eb80 1 --2- 172.21.15.70:0/187314053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e2860 0x55cbfe8ee330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.835+0000 7fca2714eb80 1 --2- 172.21.15.70:0/187314053 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cbfe90ced0 0x55cbfe90f2f0 crc :-1 s=READY pgs=401 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.835+0000 7fca2714eb80 1 --2- 172.21.15.70:0/187314053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e3210 0x55cbfe8eb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.835+0000 7fc9fdffb700 0 client.29209 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:53.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.835+0000 7fca2714eb80 1 -- 172.21.15.70:0/187314053 >> 172.21.15.70:0/187314053 conn(0x55cbfe810080 msgr2=0x55cbfe8dc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:53.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.835+0000 7fca2714eb80 1 -- 172.21.15.70:0/187314053 shutdown_connections 2024-06-04T05:15:53.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.835+0000 7fca2714eb80 1 -- 172.21.15.70:0/187314053 wait complete. 2024-06-04T05:15:53.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.835+0000 7fca2714eb80 20 client.29209 trim_cache size 0 max 16384 2024-06-04T05:15:53.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.873+0000 7fca2714eb80 1 Processor -- start 2024-06-04T05:15:53.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.873+0000 7fca2714eb80 1 -- start start 2024-06-04T05:15:53.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.873+0000 7fca2714eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e2860 0x55cbfe8e48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.874+0000 7fca2714eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e3210 0x55cbfe8e4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.874+0000 7fca2714eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e6f60 0x55cbfe914b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.874+0000 7fca1659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e2860 0x55cbfe8e48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:53.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.874+0000 7fca1659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e2860 0x55cbfe8e48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33692/0 (socket says 172.21.15.70:33692) 2024-06-04T05:15:53.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.874+0000 7fca15d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e3210 0x55cbfe8e4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:53.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.874+0000 7fca1659c700 1 -- 172.21.15.70:0/1220360490 learned_addr learned my addr 172.21.15.70:0/1220360490 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:53.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.874+0000 7fca2714eb80 1 -- 172.21.15.70:0/1220360490 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fca18114e10 con 0x55cbfe8e6f60 2024-06-04T05:15:53.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.874+0000 7fca2714eb80 1 -- 172.21.15.70:0/1220360490 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fca181b6170 con 0x55cbfe8e2860 2024-06-04T05:15:53.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.874+0000 7fca2714eb80 1 -- 172.21.15.70:0/1220360490 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fca18115510 con 0x55cbfe8e3210 2024-06-04T05:15:53.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.874+0000 7fca15d9b700 1 -- 172.21.15.70:0/1220360490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e2860 msgr2=0x55cbfe8e48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:53.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.874+0000 7fca15d9b700 1 --2- 172.21.15.70:0/1220360490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e2860 0x55cbfe8e48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.874+0000 7fca15d9b700 1 -- 172.21.15.70:0/1220360490 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e6f60 msgr2=0x55cbfe914b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:53.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.874+0000 7fca15d9b700 1 --2- 172.21.15.70:0/1220360490 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e6f60 0x55cbfe914b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.874+0000 7fca15d9b700 1 -- 172.21.15.70:0/1220360490 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbfe90f950 con 0x55cbfe8e3210 2024-06-04T05:15:53.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.874+0000 7fca1659c700 1 --2- 172.21.15.70:0/1220360490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e2860 0x55cbfe8e48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:53.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.875+0000 7fca15d9b700 1 --2- 172.21.15.70:0/1220360490 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e3210 0x55cbfe8e4df0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x55cbfe8ecc00 tx=0x7fca0800b3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b1a868148813e84 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:53.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.875+0000 7fc9fffff700 1 -- 172.21.15.70:0/1220360490 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca08022070 con 0x55cbfe8e3210 2024-06-04T05:15:53.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.875+0000 7fc9fffff700 1 -- 172.21.15.70:0/1220360490 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca0802e040 con 0x55cbfe8e3210 2024-06-04T05:15:53.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.875+0000 7fc9fffff700 1 -- 172.21.15.70:0/1220360490 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0801c6a0 con 0x55cbfe8e3210 2024-06-04T05:15:53.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.876+0000 7fca2714eb80 1 -- 172.21.15.70:0/1220360490 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e3210 msgr2=0x55cbfe8e4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:53.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.876+0000 7fca2714eb80 1 --2- 172.21.15.70:0/1220360490 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e3210 0x55cbfe8e4df0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x55cbfe8ecc00 tx=0x7fca0800b3e0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.876+0000 7fca2714eb80 1 -- 172.21.15.70:0/1220360490 shutdown_connections 2024-06-04T05:15:53.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.876+0000 7fca2714eb80 1 --2- 172.21.15.70:0/1220360490 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e6f60 0x55cbfe914b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.876+0000 7fca2714eb80 1 --2- 172.21.15.70:0/1220360490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e2860 0x55cbfe8e48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.876+0000 7fca2714eb80 1 --2- 172.21.15.70:0/1220360490 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e3210 0x55cbfe8e4df0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.877+0000 7fca2714eb80 1 -- 172.21.15.70:0/1220360490 >> 172.21.15.70:0/1220360490 conn(0x55cbfe810080 msgr2=0x55cbfe8e1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:53.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.877+0000 7fca2714eb80 1 -- 172.21.15.70:0/1220360490 shutdown_connections 2024-06-04T05:15:53.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.877+0000 7fca2714eb80 1 -- 172.21.15.70:0/1220360490 wait complete. 2024-06-04T05:15:53.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.878+0000 7fca2714eb80 1 -- start start 2024-06-04T05:15:53.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.878+0000 7fca2714eb80 1 Processor -- start 2024-06-04T05:15:53.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.878+0000 7fca2714eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e2860 0x55cbfe8f08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.878+0000 7fca2714eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e3210 0x55cbfe8f0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca1659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e2860 0x55cbfe8f08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:53.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca1659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e2860 0x55cbfe8f08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:53596/0 (socket says 172.21.15.70:53596) 2024-06-04T05:15:53.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca1659c700 1 -- 172.21.15.70:0/3569255517 learned_addr learned my addr 172.21.15.70:0/3569255517 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:53.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca2714eb80 1 --2- 172.21.15.70:0/3569255517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e6f60 0x55cbfe976a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca2714eb80 1 -- 172.21.15.70:0/3569255517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fca181162b0 con 0x55cbfe8e3210 2024-06-04T05:15:53.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca2714eb80 1 -- 172.21.15.70:0/3569255517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fca18115be0 con 0x55cbfe8e6f60 2024-06-04T05:15:53.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca2714eb80 1 -- 172.21.15.70:0/3569255517 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fca18112dd0 con 0x55cbfe8e2860 2024-06-04T05:15:53.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca15d9b700 1 --2- 172.21.15.70:0/3569255517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e3210 0x55cbfe8f0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:53.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca16d9d700 1 --2- 172.21.15.70:0/3569255517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e6f60 0x55cbfe976a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:53.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca2714eb80 10 client.0 _ll_register_callbacks cb 0x55cbfe7cc920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:53.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca1659c700 1 -- 172.21.15.70:0/3569255517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e6f60 msgr2=0x55cbfe976a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:53.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca1659c700 1 --2- 172.21.15.70:0/3569255517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e6f60 0x55cbfe976a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca1659c700 1 -- 172.21.15.70:0/3569255517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e3210 msgr2=0x55cbfe8f0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:53.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca1659c700 1 --2- 172.21.15.70:0/3569255517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e3210 0x55cbfe8f0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca1659c700 1 -- 172.21.15.70:0/3569255517 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbfe976fe0 con 0x55cbfe8e2860 2024-06-04T05:15:53.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca2714eb80 1 -- 172.21.15.70:0/3569255517 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca0802e040 con 0x55cbfe8e2860 2024-06-04T05:15:53.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca16d9d700 1 --2- 172.21.15.70:0/3569255517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e6f60 0x55cbfe976a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:15:53.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fca1659c700 1 --2- 172.21.15.70:0/3569255517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e2860 0x55cbfe8f08d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fca100a9a70 tx=0x7fca100cc040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87d21a964658c838 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:53.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.879+0000 7fc9fdffb700 10 client.29215 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:15:53.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.880+0000 7fc9fdffb700 1 -- 172.21.15.70:0/3569255517 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca100aa1e0 con 0x55cbfe8e2860 2024-06-04T05:15:53.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.880+0000 7fc9fdffb700 1 -- 172.21.15.70:0/3569255517 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7fca100a9f50 con 0x55cbfe8e2860 2024-06-04T05:15:53.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.880+0000 7fc9fdffb700 1 -- 172.21.15.70:0/3569255517 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7fca100a9ae0 con 0x55cbfe8e2860 2024-06-04T05:15:53.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.880+0000 7fca2714eb80 10 client.29215 fetch_fsmap learned FSMap version 56 2024-06-04T05:15:53.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.880+0000 7fca2714eb80 1 -- 172.21.15.70:0/3569255517 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({fsmap=56}) -- 0x55cbfe978220 con 0x55cbfe8e2860 2024-06-04T05:15:53.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.880+0000 7fc9fdffb700 1 -- 172.21.15.70:0/3569255517 <== mon.1 v2:172.21.15.112:3300/0 4 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7fca100b24b0 con 0x55cbfe8e2860 2024-06-04T05:15:53.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.880+0000 7fca2714eb80 10 client.29215 fetch_fsmap finished waiting for FSMap version 56 2024-06-04T05:15:53.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.880+0000 7fca2714eb80 10 client.29215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:53.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.880+0000 7fca2714eb80 20 client.29215 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:53.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.880+0000 7fca2714eb80 1 --2- 172.21.15.70:0/3569255517 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cbfe978840 0x55cbfe97ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:53.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.880+0000 7fca2714eb80 4 client.29215 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:15:53.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.880+0000 7fca2714eb80 1 -- 172.21.15.70:0/3569255517 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cbfe843be0 con 0x55cbfe978840 2024-06-04T05:15:53.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.881+0000 7fca15d9b700 1 --2- 172.21.15.70:0/3569255517 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cbfe978840 0x55cbfe97ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:53.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.881+0000 7fca15d9b700 1 --2- 172.21.15.70:0/3569255517 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cbfe978840 0x55cbfe97ac60 crc :-1 s=READY pgs=402 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46d62cef92e8dafd server_cookie=d3d528f44f965df4 in_seq=0 out_seq=0 2024-06-04T05:15:53.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.881+0000 7fc9fdffb700 10 client.29215 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:53.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.881+0000 7fc9fdffb700 1 -- 172.21.15.70:0/3569255517 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55cbfe843be0 con 0x55cbfe978840 2024-06-04T05:15:53.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.881+0000 7fc9fdffb700 10 client.29215 handle_command_reply: tid=0 2024-06-04T05:15:53.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.881+0000 7fca2714eb80 1 client.29215 shutdown 2024-06-04T05:15:53.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.882+0000 7fca2714eb80 1 -- 172.21.15.70:0/3569255517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e2860 msgr2=0x55cbfe8f08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:53.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.882+0000 7fca2714eb80 1 --2- 172.21.15.70:0/3569255517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e2860 0x55cbfe8f08d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fca100a9a70 tx=0x7fca100cc040 comp rx=0 tx=0).stop 2024-06-04T05:15:53.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.882+0000 7fca2714eb80 1 -- 172.21.15.70:0/3569255517 shutdown_connections 2024-06-04T05:15:53.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.882+0000 7fca2714eb80 1 --2- 172.21.15.70:0/3569255517 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cbfe978840 0x55cbfe97ac60 crc :-1 s=READY pgs=402 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.882+0000 7fca2714eb80 1 --2- 172.21.15.70:0/3569255517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbfe8e6f60 0x55cbfe976a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.882+0000 7fc9fdffb700 0 client.29215 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:53.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.882+0000 7fca2714eb80 1 --2- 172.21.15.70:0/3569255517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cbfe8e2860 0x55cbfe8f08d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.882+0000 7fca2714eb80 1 --2- 172.21.15.70:0/3569255517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cbfe8e3210 0x55cbfe8f0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.882+0000 7fca2714eb80 1 -- 172.21.15.70:0/3569255517 >> 172.21.15.70:0/3569255517 conn(0x55cbfe810080 msgr2=0x55cbfe8e58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:53.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.882+0000 7fca2714eb80 1 -- 172.21.15.70:0/3569255517 shutdown_connections 2024-06-04T05:15:53.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.882+0000 7fca2714eb80 1 -- 172.21.15.70:0/3569255517 wait complete. 2024-06-04T05:15:53.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.882+0000 7fca2714eb80 20 client.29215 trim_cache size 0 max 16384 2024-06-04T05:15:53.888 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:15:53.888 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:15:53.888 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:15:53.888 INFO:teuthology.orchestra.run.smithi070.stdout: "50dfa3b3-6f24-4ea4-811e-dbdafd6e928a": { 2024-06-04T05:15:53.888 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:15:53.888 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "50dfa3b3-6f24-4ea4-811e-dbdafd6e928a", 2024-06-04T05:15:53.888 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:15:53.889 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:15:53.889 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:15:53.889 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:15:53.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.883+0000 7fca1c8b9700 1 -- 172.21.15.70:0/1264825898 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fca040b2b40 msgr2=0x7fca040b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:53.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.883+0000 7fca1c8b9700 1 --2- 172.21.15.70:0/1264825898 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fca040b2b40 0x7fca040b5000 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fca1806f580 tx=0x7fca0800b040 comp rx=0 tx=0).stop 2024-06-04T05:15:53.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.883+0000 7fca1c8b9700 1 -- 172.21.15.70:0/1264825898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca18101b30 msgr2=0x7fca18074d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:53.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.883+0000 7fca1c8b9700 1 --2- 172.21.15.70:0/1264825898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca18101b30 0x7fca18074d30 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fca1000e9a0 tx=0x7fca1000ed60 comp rx=0 tx=0).stop 2024-06-04T05:15:53.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.883+0000 7fca1c8b9700 1 -- 172.21.15.70:0/1264825898 shutdown_connections 2024-06-04T05:15:53.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.883+0000 7fca1c8b9700 1 --2- 172.21.15.70:0/1264825898 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fca040b2b40 0x7fca040b5000 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.883+0000 7fca1c8b9700 1 --2- 172.21.15.70:0/1264825898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca18104450 0x7fca1806e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.883+0000 7fca1c8b9700 1 --2- 172.21.15.70:0/1264825898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca18101b30 0x7fca18074d30 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.883+0000 7fca1c8b9700 1 --2- 172.21.15.70:0/1264825898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca181026e0 0x7fca1806de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:53.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.883+0000 7fca1c8b9700 1 -- 172.21.15.70:0/1264825898 >> 172.21.15.70:0/1264825898 conn(0x7fca18078ab0 msgr2=0x7fca180ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:53.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.883+0000 7fca1c8b9700 1 -- 172.21.15.70:0/1264825898 shutdown_connections 2024-06-04T05:15:53.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:53.884+0000 7fca1c8b9700 1 -- 172.21.15.70:0/1264825898 wait complete. 2024-06-04T05:15:54.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.217+0000 7f74aa30a640 1 -- 172.21.15.70:0/1241373078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74a4107940 msgr2=0x7f74a4107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:54.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.217+0000 7f74aa30a640 1 --2- 172.21.15.70:0/1241373078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74a4107940 0x7f74a4107d20 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f7494009930 tx=0x7f749402a420 comp rx=0 tx=0).stop 2024-06-04T05:15:54.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.217+0000 7f74aa30a640 1 -- 172.21.15.70:0/1241373078 shutdown_connections 2024-06-04T05:15:54.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.217+0000 7f74aa30a640 1 --2- 172.21.15.70:0/1241373078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74a4102340 0x7f74a410e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.217+0000 7f74aa30a640 1 --2- 172.21.15.70:0/1241373078 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74a4101980 0x7f74a4101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.217+0000 7f74aa30a640 1 --2- 172.21.15.70:0/1241373078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74a4107940 0x7f74a4107d20 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.217+0000 7f74aa30a640 1 -- 172.21.15.70:0/1241373078 >> 172.21.15.70:0/1241373078 conn(0x7f74a4078ad0 msgr2=0x7f74a4078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:54.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.217+0000 7f74aa30a640 1 -- 172.21.15.70:0/1241373078 shutdown_connections 2024-06-04T05:15:54.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.217+0000 7f74aa30a640 1 -- 172.21.15.70:0/1241373078 wait complete. 2024-06-04T05:15:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.218+0000 7f74aa30a640 1 Processor -- start 2024-06-04T05:15:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.218+0000 7f74aa30a640 1 -- start start 2024-06-04T05:15:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.218+0000 7f74aa30a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74a4101980 0x7f74a419b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.218+0000 7f74aa30a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74a4102340 0x7f74a419b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.218+0000 7f74aa30a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74a4107940 0x7f74a419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.218+0000 7f74aa30a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f74a4111b00 con 0x7f74a4107940 2024-06-04T05:15:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.218+0000 7f74aa30a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f74a4111980 con 0x7f74a4102340 2024-06-04T05:15:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.218+0000 7f74aa30a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f74a4111c80 con 0x7f74a4101980 2024-06-04T05:15:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.218+0000 7f74a3fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74a4101980 0x7f74a419b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.219+0000 7f74a88a7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74a4107940 0x7f74a419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.219+0000 7f74a88a7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74a4107940 0x7f74a419fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35614/0 (socket says 172.21.15.70:35614) 2024-06-04T05:15:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.219+0000 7f74a88a7640 1 -- 172.21.15.70:0/2742750784 learned_addr learned my addr 172.21.15.70:0/2742750784 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.219+0000 7f74a37fe640 1 --2- 172.21.15.70:0/2742750784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74a4102340 0x7f74a419b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.219+0000 7f74a88a7640 1 -- 172.21.15.70:0/2742750784 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74a4101980 msgr2=0x7f74a419b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.219+0000 7f74a88a7640 1 --2- 172.21.15.70:0/2742750784 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74a4101980 0x7f74a419b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.219+0000 7f74a88a7640 1 -- 172.21.15.70:0/2742750784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74a4102340 msgr2=0x7f74a419b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.219+0000 7f74a88a7640 1 --2- 172.21.15.70:0/2742750784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74a4102340 0x7f74a419b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.219+0000 7f74a88a7640 1 -- 172.21.15.70:0/2742750784 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a41a0300 con 0x7f74a4107940 2024-06-04T05:15:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.219+0000 7f74a88a7640 1 --2- 172.21.15.70:0/2742750784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74a4107940 0x7f74a419fbd0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f749c00eb10 tx=0x7f749c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.219+0000 7f74a17fa640 1 -- 172.21.15.70:0/2742750784 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c004b20 con 0x7f74a4107940 2024-06-04T05:15:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.219+0000 7f74aa30a640 1 -- 172.21.15.70:0/2742750784 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74a41a05f0 con 0x7f74a4107940 2024-06-04T05:15:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.220+0000 7f74a17fa640 1 -- 172.21.15.70:0/2742750784 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f749c004cc0 con 0x7f74a4107940 2024-06-04T05:15:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.220+0000 7f74a17fa640 1 -- 172.21.15.70:0/2742750784 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c010690 con 0x7f74a4107940 2024-06-04T05:15:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.220+0000 7f74aa30a640 1 -- 172.21.15.70:0/2742750784 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74a4108790 con 0x7f74a4107940 2024-06-04T05:15:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.223+0000 7f74ad01e740 1 Processor -- start 2024-06-04T05:15:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.224+0000 7f74a17fa640 1 -- 172.21.15.70:0/2742750784 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f749c005020 con 0x7f74a4107940 2024-06-04T05:15:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.224+0000 7f74ad01e740 1 -- start start 2024-06-04T05:15:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.224+0000 7f74ad01e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad549820 0x55b9ad549c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74ad01e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad54a140 0x55b9ad5544c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74ad01e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad554a00 0x55b9ad556df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74ad01e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b9ad4bbc30 con 0x55b9ad554a00 2024-06-04T05:15:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74ad01e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b9ad4ab7d0 con 0x55b9ad549820 2024-06-04T05:15:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74ad01e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b9ad4af810 con 0x55b9ad54a140 2024-06-04T05:15:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74a37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad549820 0x55b9ad549c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74a37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad549820 0x55b9ad549c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33738/0 (socket says 172.21.15.70:33738) 2024-06-04T05:15:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74a37fe640 1 -- 172.21.15.70:0/838646706 learned_addr learned my addr 172.21.15.70:0/838646706 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74a3fff640 1 --2- 172.21.15.70:0/838646706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad54a140 0x55b9ad5544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74a37fe640 1 --2- 172.21.15.70:0/838646706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad554a00 0x55b9ad556df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74a3fff640 1 -- 172.21.15.70:0/838646706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad549820 msgr2=0x55b9ad549c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74a3fff640 1 --2- 172.21.15.70:0/838646706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad549820 0x55b9ad549c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74a3fff640 1 -- 172.21.15.70:0/838646706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad554a00 msgr2=0x55b9ad556df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74a3fff640 1 --2- 172.21.15.70:0/838646706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad554a00 0x55b9ad556df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74a17fa640 1 --2- 172.21.15.70:0/2742750784 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f748c0b2a90 0x7f748c0b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74a3fff640 1 -- 172.21.15.70:0/838646706 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9ad1c0210 con 0x55b9ad54a140 2024-06-04T05:15:54.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74a88a7640 1 --2- 172.21.15.70:0/2742750784 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f748c0b2a90 0x7f748c0b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.225+0000 7f74a3fff640 1 --2- 172.21.15.70:0/838646706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad54a140 0x55b9ad5544c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f749402a940 tx=0x7f74940040e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=611bdaa252efc4ae server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:54.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.226+0000 7f748a7fc640 1 -- 172.21.15.70:0/838646706 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494035030 con 0x55b9ad54a140 2024-06-04T05:15:54.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.226+0000 7f748a7fc640 1 -- 172.21.15.70:0/838646706 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7494040070 con 0x55b9ad54a140 2024-06-04T05:15:54.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.226+0000 7f748a7fc640 1 -- 172.21.15.70:0/838646706 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74940457f0 con 0x55b9ad54a140 2024-06-04T05:15:54.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.227+0000 7f74ad01e740 1 -- 172.21.15.70:0/838646706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad54a140 msgr2=0x55b9ad5544c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:54.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.227+0000 7f74ad01e740 1 --2- 172.21.15.70:0/838646706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad54a140 0x55b9ad5544c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f749402a940 tx=0x7f74940040e0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.227+0000 7f74a17fa640 1 -- 172.21.15.70:0/2742750784 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f749c0e29f0 con 0x7f74a4107940 2024-06-04T05:15:54.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.227+0000 7f74ad01e740 1 -- 172.21.15.70:0/838646706 shutdown_connections 2024-06-04T05:15:54.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.227+0000 7f74ad01e740 1 --2- 172.21.15.70:0/838646706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad554a00 0x55b9ad556df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.227+0000 7f74ad01e740 1 --2- 172.21.15.70:0/838646706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad54a140 0x55b9ad5544c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.227+0000 7f74ad01e740 1 --2- 172.21.15.70:0/838646706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad549820 0x55b9ad549c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.227+0000 7f74ad01e740 1 -- 172.21.15.70:0/838646706 >> 172.21.15.70:0/838646706 conn(0x55b9ad545d30 msgr2=0x55b9ad546110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:54.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.227+0000 7f74ad01e740 1 -- 172.21.15.70:0/838646706 shutdown_connections 2024-06-04T05:15:54.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.227+0000 7f74a88a7640 1 --2- 172.21.15.70:0/2742750784 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f748c0b2a90 0x7f748c0b4f50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f749c0a9c30 tx=0x7f749c0a5c40 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:54.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.227+0000 7f74ad01e740 1 -- 172.21.15.70:0/838646706 wait complete. 2024-06-04T05:15:54.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.227+0000 7f74ad01e740 1 -- start start 2024-06-04T05:15:54.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.228+0000 7f74ad01e740 1 Processor -- start 2024-06-04T05:15:54.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.228+0000 7f74ad01e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad549820 0x55b9ad5519a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.228+0000 7f74ad01e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad54a140 0x55b9ad551ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.228+0000 7f74ad01e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad554a00 0x55b9ad552420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.228+0000 7f74ad01e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b9ad191790 con 0x55b9ad554a00 2024-06-04T05:15:54.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.228+0000 7f74ad01e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b9ad467890 con 0x55b9ad54a140 2024-06-04T05:15:54.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.228+0000 7f74ad01e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b9ad5575d0 con 0x55b9ad549820 2024-06-04T05:15:54.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.228+0000 7f74ad01e740 10 client.0 _ll_register_callbacks cb 0x55b9ad1bfa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:54.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.228+0000 7f74a37fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad549820 0x55b9ad5519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.228+0000 7f74a37fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad549820 0x55b9ad5519a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:53636/0 (socket says 172.21.15.70:53636) 2024-06-04T05:15:54.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.228+0000 7f74a37fe640 1 -- 172.21.15.70:0/707002618 learned_addr learned my addr 172.21.15.70:0/707002618 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:54.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.228+0000 7f74a37fe640 1 --2- 172.21.15.70:0/707002618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad554a00 0x55b9ad552420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.229+0000 7f74a3fff640 1 --2- 172.21.15.70:0/707002618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad54a140 0x55b9ad551ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.229+0000 7f74a37fe640 1 -- 172.21.15.70:0/707002618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad54a140 msgr2=0x55b9ad551ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:54.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.229+0000 7f74a37fe640 1 --2- 172.21.15.70:0/707002618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad54a140 0x55b9ad551ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.229+0000 7f74a37fe640 1 -- 172.21.15.70:0/707002618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad554a00 msgr2=0x55b9ad552420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:54.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.229+0000 7f74a37fe640 1 --2- 172.21.15.70:0/707002618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad554a00 0x55b9ad552420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.229+0000 7f74a37fe640 1 -- 172.21.15.70:0/707002618 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9ad28c620 con 0x55b9ad549820 2024-06-04T05:15:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.229+0000 7f74a3fff640 1 --2- 172.21.15.70:0/707002618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad54a140 0x55b9ad551ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:15:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.229+0000 7f74a37fe640 1 --2- 172.21.15.70:0/707002618 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad549820 0x55b9ad5519a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7498027d90 tx=0x7f7498004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7e95745111f5960 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.229+0000 7f74837fe640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:15:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.229+0000 7f74837fe640 1 -- 172.21.15.70:0/707002618 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74980272d0 con 0x55b9ad549820 2024-06-04T05:15:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.229+0000 7f74837fe640 1 -- 172.21.15.70:0/707002618 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f7498004020 con 0x55b9ad549820 2024-06-04T05:15:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.229+0000 7f74ad01e740 1 -- 172.21.15.70:0/707002618 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7494040070 con 0x55b9ad549820 2024-06-04T05:15:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.230+0000 7f74837fe640 1 -- 172.21.15.70:0/707002618 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f7498024e70 con 0x55b9ad549820 2024-06-04T05:15:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.230+0000 7f74ad01e740 10 client.29227 fetch_fsmap learned FSMap version 56 2024-06-04T05:15:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.230+0000 7f74ad01e740 1 -- 172.21.15.70:0/707002618 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x55b9ad5631a0 con 0x55b9ad549820 2024-06-04T05:15:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.230+0000 7f74837fe640 1 -- 172.21.15.70:0/707002618 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f74980054b0 con 0x55b9ad549820 2024-06-04T05:15:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.230+0000 7f74837fe640 1 -- 172.21.15.70:0/707002618 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7f74980048c0 con 0x55b9ad549820 2024-06-04T05:15:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.230+0000 7f74ad01e740 10 client.29227 fetch_fsmap finished waiting for FSMap version 56 2024-06-04T05:15:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.230+0000 7f74ad01e740 10 client.29227 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.230+0000 7f74ad01e740 20 client.29227 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.230+0000 7f74ad01e740 1 --2- 172.21.15.70:0/707002618 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b9ad564700 0x55b9ad566b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.230+0000 7f74ad01e740 4 client.29227 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:15:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.230+0000 7f74ad01e740 1 -- 172.21.15.70:0/707002618 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7494040070 con 0x55b9ad564700 2024-06-04T05:15:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.230+0000 7f74a3fff640 1 --2- 172.21.15.70:0/707002618 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b9ad564700 0x55b9ad566b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.231+0000 7f74837fe640 10 client.29227 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.231+0000 7f74a3fff640 1 --2- 172.21.15.70:0/707002618 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b9ad564700 0x55b9ad566b80 crc :-1 s=READY pgs=403 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c7a985e5769bbdd server_cookie=c1a1b501b8616e0a in_seq=0 out_seq=0 2024-06-04T05:15:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.232+0000 7f74837fe640 1 -- 172.21.15.70:0/707002618 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f7494040070 con 0x55b9ad564700 2024-06-04T05:15:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.232+0000 7f74837fe640 10 client.29227 handle_command_reply: tid=0 2024-06-04T05:15:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.232+0000 7f74ad01e740 1 client.29227 shutdown 2024-06-04T05:15:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.232+0000 7f74ad01e740 1 -- 172.21.15.70:0/707002618 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad549820 msgr2=0x55b9ad5519a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.232+0000 7f74ad01e740 1 --2- 172.21.15.70:0/707002618 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad549820 0x55b9ad5519a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7498027d90 tx=0x7f7498004730 comp rx=0 tx=0).stop 2024-06-04T05:15:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.232+0000 7f74ad01e740 1 -- 172.21.15.70:0/707002618 shutdown_connections 2024-06-04T05:15:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.232+0000 7f74ad01e740 1 --2- 172.21.15.70:0/707002618 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b9ad564700 0x55b9ad566b80 crc :-1 s=READY pgs=403 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.232+0000 7f74ad01e740 1 --2- 172.21.15.70:0/707002618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad54a140 0x55b9ad551ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.232+0000 7f74ad01e740 1 --2- 172.21.15.70:0/707002618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad554a00 0x55b9ad552420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.232+0000 7f74ad01e740 1 --2- 172.21.15.70:0/707002618 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad549820 0x55b9ad5519a0 secure :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7498027d90 tx=0x7f7498004730 comp rx=0 tx=0).stop 2024-06-04T05:15:54.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.232+0000 7f74ad01e740 1 -- 172.21.15.70:0/707002618 >> 172.21.15.70:0/707002618 conn(0x55b9ad545d30 msgr2=0x55b9ad544fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:54.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.232+0000 7f74837fe640 0 client.29227 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:54.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.232+0000 7f74ad01e740 1 -- 172.21.15.70:0/707002618 shutdown_connections 2024-06-04T05:15:54.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.232+0000 7f74ad01e740 1 -- 172.21.15.70:0/707002618 wait complete. 2024-06-04T05:15:54.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.232+0000 7f74ad01e740 20 client.29227 trim_cache size 0 max 16384 2024-06-04T05:15:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.255+0000 7f74ad01e740 1 Processor -- start 2024-06-04T05:15:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.255+0000 7f74ad01e740 1 -- start start 2024-06-04T05:15:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.255+0000 7f74ad01e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad54a140 0x55b9ad545730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.255+0000 7f74ad01e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad554a00 0x55b9ad552270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.255+0000 7f74ad01e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad56c450 0x55b9ad56c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.255+0000 7f74ad01e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f749000e890 con 0x55b9ad554a00 2024-06-04T05:15:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.255+0000 7f74ad01e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7490010400 con 0x55b9ad56c450 2024-06-04T05:15:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.255+0000 7f74ad01e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f749000ef90 con 0x55b9ad54a140 2024-06-04T05:15:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.255+0000 7f74a37fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad54a140 0x55b9ad545730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.255+0000 7f74a37fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad54a140 0x55b9ad545730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:53652/0 (socket says 172.21.15.70:53652) 2024-06-04T05:15:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.255+0000 7f74a37fe640 1 -- 172.21.15.70:0/3077322743 learned_addr learned my addr 172.21.15.70:0/3077322743 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.255+0000 7f74a37fe640 1 --2- 172.21.15.70:0/3077322743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad56c450 0x55b9ad56c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.255+0000 7f74a3fff640 1 --2- 172.21.15.70:0/3077322743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad554a00 0x55b9ad552270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.255+0000 7f74a37fe640 1 -- 172.21.15.70:0/3077322743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad56c450 msgr2=0x55b9ad56c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.255+0000 7f74a37fe640 1 --2- 172.21.15.70:0/3077322743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad56c450 0x55b9ad56c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.255+0000 7f74a37fe640 1 -- 172.21.15.70:0/3077322743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad554a00 msgr2=0x55b9ad552270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.256+0000 7f74a37fe640 1 --2- 172.21.15.70:0/3077322743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad554a00 0x55b9ad552270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.256+0000 7f74a37fe640 1 -- 172.21.15.70:0/3077322743 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9ad557020 con 0x55b9ad54a140 2024-06-04T05:15:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.256+0000 7f74a37fe640 1 --2- 172.21.15.70:0/3077322743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad54a140 0x55b9ad545730 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7498004700 tx=0x7f7498000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e05a2746fc2c2624 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.256+0000 7f7489ffb640 1 -- 172.21.15.70:0/3077322743 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74980272d0 con 0x55b9ad54a140 2024-06-04T05:15:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.256+0000 7f7489ffb640 1 -- 172.21.15.70:0/3077322743 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7498004020 con 0x55b9ad54a140 2024-06-04T05:15:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.256+0000 7f7489ffb640 1 -- 172.21.15.70:0/3077322743 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7498005130 con 0x55b9ad54a140 2024-06-04T05:15:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.257+0000 7f74ad01e740 1 -- 172.21.15.70:0/3077322743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad54a140 msgr2=0x55b9ad545730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.257+0000 7f74ad01e740 1 --2- 172.21.15.70:0/3077322743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad54a140 0x55b9ad545730 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7498004700 tx=0x7f7498000bc0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.257+0000 7f74ad01e740 1 -- 172.21.15.70:0/3077322743 shutdown_connections 2024-06-04T05:15:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.257+0000 7f74ad01e740 1 --2- 172.21.15.70:0/3077322743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad56c450 0x55b9ad56c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.257+0000 7f74ad01e740 1 --2- 172.21.15.70:0/3077322743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad554a00 0x55b9ad552270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.257+0000 7f74ad01e740 1 --2- 172.21.15.70:0/3077322743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad54a140 0x55b9ad545730 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.257+0000 7f74ad01e740 1 -- 172.21.15.70:0/3077322743 >> 172.21.15.70:0/3077322743 conn(0x55b9ad545d30 msgr2=0x55b9ad5531d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.257+0000 7f74ad01e740 1 -- 172.21.15.70:0/3077322743 shutdown_connections 2024-06-04T05:15:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.257+0000 7f74ad01e740 1 -- 172.21.15.70:0/3077322743 wait complete. 2024-06-04T05:15:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.257+0000 7f74ad01e740 1 -- start start 2024-06-04T05:15:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.257+0000 7f74ad01e740 1 Processor -- start 2024-06-04T05:15:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74ad01e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad54a140 0x55b9ad5588e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74ad01e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad554a00 0x55b9ad558e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74ad01e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad56c450 0x55b9ad5ce320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74ad01e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f749000fd30 con 0x55b9ad54a140 2024-06-04T05:15:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74ad01e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f749000f660 con 0x55b9ad56c450 2024-06-04T05:15:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74ad01e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f749000c890 con 0x55b9ad554a00 2024-06-04T05:15:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74ad01e740 10 client.0 _ll_register_callbacks cb 0x7f7498014bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74a37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad54a140 0x55b9ad5588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74a37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad54a140 0x55b9ad5588e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35636/0 (socket says 172.21.15.70:35636) 2024-06-04T05:15:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74a37fe640 1 -- 172.21.15.70:0/3554846866 learned_addr learned my addr 172.21.15.70:0/3554846866 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74a37fe640 1 --2- 172.21.15.70:0/3554846866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad56c450 0x55b9ad5ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74a3fff640 1 --2- 172.21.15.70:0/3554846866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad554a00 0x55b9ad558e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74a37fe640 1 -- 172.21.15.70:0/3554846866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad554a00 msgr2=0x55b9ad558e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74a37fe640 1 --2- 172.21.15.70:0/3554846866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad554a00 0x55b9ad558e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74a37fe640 1 -- 172.21.15.70:0/3554846866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad56c450 msgr2=0x55b9ad5ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74a37fe640 1 --2- 172.21.15.70:0/3554846866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad56c450 0x55b9ad5ce320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74a37fe640 1 -- 172.21.15.70:0/3554846866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9ad5596f0 con 0x55b9ad54a140 2024-06-04T05:15:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74a3fff640 1 --2- 172.21.15.70:0/3554846866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad554a00 0x55b9ad558e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:15:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.258+0000 7f74a37fe640 1 --2- 172.21.15.70:0/3554846866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad54a140 0x55b9ad5588e0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f7498016040 tx=0x7f7498027860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=325a0d4822d465c2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.259+0000 7f74837fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:15:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.259+0000 7f74837fe640 1 -- 172.21.15.70:0/3554846866 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7498045040 con 0x55b9ad54a140 2024-06-04T05:15:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.259+0000 7f74ad01e740 1 -- 172.21.15.70:0/3554846866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7498004020 con 0x55b9ad54a140 2024-06-04T05:15:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.259+0000 7f74837fe640 1 -- 172.21.15.70:0/3554846866 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f7498004930 con 0x55b9ad54a140 2024-06-04T05:15:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.259+0000 7f74837fe640 1 -- 172.21.15.70:0/3554846866 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f74980035f0 con 0x55b9ad54a140 2024-06-04T05:15:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.259+0000 7f74ad01e740 10 client.20325 fetch_fsmap learned FSMap version 56 2024-06-04T05:15:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.259+0000 7f74ad01e740 1 -- 172.21.15.70:0/3554846866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x55b9ad5cf970 con 0x55b9ad54a140 2024-06-04T05:15:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.259+0000 7f74837fe640 1 -- 172.21.15.70:0/3554846866 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7498004020 con 0x55b9ad54a140 2024-06-04T05:15:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.259+0000 7f74837fe640 1 -- 172.21.15.70:0/3554846866 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7f749804b070 con 0x55b9ad54a140 2024-06-04T05:15:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.260+0000 7f74ad01e740 10 client.20325 fetch_fsmap finished waiting for FSMap version 56 2024-06-04T05:15:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.260+0000 7f74ad01e740 10 client.20325 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:15:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.260+0000 7f74ad01e740 20 client.20325 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.260+0000 7f74ad01e740 1 --2- 172.21.15.70:0/3554846866 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b9ad5d0090 0x55b9ad5d2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.260+0000 7f74ad01e740 4 client.20325 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:15:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.260+0000 7f74ad01e740 1 -- 172.21.15.70:0/3554846866 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b9ad4bdb80 con 0x55b9ad5d0090 2024-06-04T05:15:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.260+0000 7f74a3fff640 1 --2- 172.21.15.70:0/3554846866 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b9ad5d0090 0x55b9ad5d2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.260+0000 7f74837fe640 10 client.20325 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.260+0000 7f74a3fff640 1 --2- 172.21.15.70:0/3554846866 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b9ad5d0090 0x55b9ad5d2480 crc :-1 s=READY pgs=404 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e3c4bec18485888 server_cookie=31a772f18a39848a in_seq=0 out_seq=0 2024-06-04T05:15:54.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.365+0000 7f74837fe640 1 -- 172.21.15.70:0/3554846866 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55b9ad4bdb80 con 0x55b9ad5d0090 2024-06-04T05:15:54.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.365+0000 7f74837fe640 10 client.20325 handle_command_reply: tid=0 2024-06-04T05:15:54.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.365+0000 7f74ad01e740 1 client.20325 shutdown 2024-06-04T05:15:54.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.365+0000 7f74ad01e740 1 -- 172.21.15.70:0/3554846866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad54a140 msgr2=0x55b9ad5588e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:54.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.365+0000 7f74ad01e740 1 --2- 172.21.15.70:0/3554846866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad54a140 0x55b9ad5588e0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f7498016040 tx=0x7f7498027860 comp rx=0 tx=0).stop 2024-06-04T05:15:54.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.365+0000 7f74ad01e740 1 -- 172.21.15.70:0/3554846866 shutdown_connections 2024-06-04T05:15:54.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.365+0000 7f74ad01e740 1 --2- 172.21.15.70:0/3554846866 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b9ad5d0090 0x55b9ad5d2480 crc :-1 s=READY pgs=404 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.365+0000 7f74837fe640 0 client.20325 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:54.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.365+0000 7f74ad01e740 1 --2- 172.21.15.70:0/3554846866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9ad56c450 0x55b9ad5ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.365+0000 7f74ad01e740 1 --2- 172.21.15.70:0/3554846866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b9ad554a00 0x55b9ad558e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.365+0000 7f74ad01e740 1 --2- 172.21.15.70:0/3554846866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b9ad54a140 0x55b9ad5588e0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.365+0000 7f74ad01e740 1 -- 172.21.15.70:0/3554846866 >> 172.21.15.70:0/3554846866 conn(0x55b9ad545d30 msgr2=0x55b9ad56b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:54.369 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.365+0000 7f74ad01e740 1 -- 172.21.15.70:0/3554846866 shutdown_connections 2024-06-04T05:15:54.369 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.365+0000 7f74ad01e740 1 -- 172.21.15.70:0/3554846866 wait complete. 2024-06-04T05:15:54.369 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.365+0000 7f74ad01e740 20 client.20325 trim_cache size 0 max 16384 2024-06-04T05:15:54.369 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:15:54.369 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:15:54.369 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:15:54.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.366+0000 7f74aa30a640 1 -- 172.21.15.70:0/2742750784 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f748c0b2a90 msgr2=0x7f748c0b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:54.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.366+0000 7f74aa30a640 1 --2- 172.21.15.70:0/2742750784 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f748c0b2a90 0x7f748c0b4f50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f749c0a9c30 tx=0x7f749c0a5c40 comp rx=0 tx=0).stop 2024-06-04T05:15:54.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.366+0000 7f74aa30a640 1 -- 172.21.15.70:0/2742750784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74a4107940 msgr2=0x7f74a419fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:54.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.366+0000 7f74aa30a640 1 --2- 172.21.15.70:0/2742750784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74a4107940 0x7f74a419fbd0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f749c00eb10 tx=0x7f749c00c2d0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.366+0000 7f74aa30a640 1 -- 172.21.15.70:0/2742750784 shutdown_connections 2024-06-04T05:15:54.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.366+0000 7f74aa30a640 1 --2- 172.21.15.70:0/2742750784 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f748c0b2a90 0x7f748c0b4f50 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.366+0000 7f74aa30a640 1 --2- 172.21.15.70:0/2742750784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74a4102340 0x7f74a419b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.366+0000 7f74aa30a640 1 --2- 172.21.15.70:0/2742750784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74a4107940 0x7f74a419fbd0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.366+0000 7f74aa30a640 1 --2- 172.21.15.70:0/2742750784 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74a4101980 0x7f74a419b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.366+0000 7f74aa30a640 1 -- 172.21.15.70:0/2742750784 >> 172.21.15.70:0/2742750784 conn(0x7f74a4078ad0 msgr2=0x7f74a406a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.366+0000 7f74aa30a640 1 -- 172.21.15.70:0/2742750784 shutdown_connections 2024-06-04T05:15:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:15:54.366+0000 7f74aa30a640 1 -- 172.21.15.70:0/2742750784 wait complete. 2024-06-04T05:15:54.663 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 146.85 --await-for 45.0' 2024-06-04T05:15:54.664 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 146.85 --await-for 45.0 2024-06-04T05:15:54.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 40676 7.88 MB/sec execute 166 sec latency 122.908 ms 2024-06-04T05:15:55.462 INFO:tasks.cephfs.filesystem:scrub status for tag:50dfa3b3-6f24-4ea4-811e-dbdafd6e928a - {'path': '/', 'tag': '50dfa3b3-6f24-4ea4-811e-dbdafd6e928a', 'options': 'recursive,force'} 2024-06-04T05:15:55.512 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:15:55.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:55 smithi070 ceph-mon[32631]: pgmap v587: 97 pgs: 97 active+clean; 664 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.3 MiB/s wr, 237 op/s 2024-06-04T05:15:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:55 smithi112 ceph-mon[36453]: pgmap v587: 97 pgs: 97 active+clean; 664 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.3 MiB/s wr, 237 op/s 2024-06-04T05:15:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:55 smithi098 ceph-mon[35569]: pgmap v587: 97 pgs: 97 active+clean; 664 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.3 MiB/s wr, 237 op/s 2024-06-04T05:15:55.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 40811 7.86 MB/sec execute 167 sec latency 75.038 ms 2024-06-04T05:15:56.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:56 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:15:56.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:56 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:15:56.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:56 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:15:56.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 40966 7.85 MB/sec execute 168 sec latency 57.110 ms 2024-06-04T05:15:57.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.366+0000 7f341576c700 1 -- 172.21.15.70:0/1158489567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34101095d0 msgr2=0x7f34101099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:57.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.366+0000 7f341576c700 1 --2- 172.21.15.70:0/1158489567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34101095d0 0x7f34101099b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f34000098f0 tx=0x7f3400009c00 comp rx=0 tx=0).stop 2024-06-04T05:15:57.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.366+0000 7f341576c700 1 -- 172.21.15.70:0/1158489567 shutdown_connections 2024-06-04T05:15:57.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.366+0000 7f341576c700 1 --2- 172.21.15.70:0/1158489567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f341006c4d0 0x7f34101103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.366+0000 7f341576c700 1 --2- 172.21.15.70:0/1158489567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34101095d0 0x7f34101099b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.366+0000 7f341576c700 1 --2- 172.21.15.70:0/1158489567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f341006bb10 0x7f341006bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.366+0000 7f341576c700 1 -- 172.21.15.70:0/1158489567 >> 172.21.15.70:0/1158489567 conn(0x7f3410079480 msgr2=0x7f3410079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:57.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.366+0000 7f341576c700 1 -- 172.21.15.70:0/1158489567 shutdown_connections 2024-06-04T05:15:57.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.366+0000 7f341576c700 1 -- 172.21.15.70:0/1158489567 wait complete. 2024-06-04T05:15:57.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.367+0000 7f341576c700 1 Processor -- start 2024-06-04T05:15:57.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.367+0000 7f341576c700 1 -- start start 2024-06-04T05:15:57.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.367+0000 7f341576c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f341006bb10 0x7f34101a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.368+0000 7f341576c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f341006c4d0 0x7f34101a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.368+0000 7f341576c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34101095d0 0x7f341019bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.368+0000 7f341576c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f34101135b0 con 0x7f34101095d0 2024-06-04T05:15:57.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.368+0000 7f341576c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3410113430 con 0x7f341006c4d0 2024-06-04T05:15:57.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.368+0000 7f340effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f341006bb10 0x7f34101a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.368+0000 7f340e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f341006c4d0 0x7f34101a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.368+0000 7f340f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34101095d0 0x7f341019bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.368+0000 7f340e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f341006c4d0 0x7f34101a1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33762/0 (socket says 172.21.15.70:33762) 2024-06-04T05:15:57.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.368+0000 7f340e7fc700 1 -- 172.21.15.70:0/2049010847 learned_addr learned my addr 172.21.15.70:0/2049010847 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:57.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.368+0000 7f341576c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3410113730 con 0x7f341006bb10 2024-06-04T05:15:57.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.368+0000 7f340e7fc700 1 -- 172.21.15.70:0/2049010847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f341006bb10 msgr2=0x7f34101a1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:57.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.368+0000 7f340e7fc700 1 --2- 172.21.15.70:0/2049010847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f341006bb10 0x7f34101a1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.368+0000 7f340e7fc700 1 -- 172.21.15.70:0/2049010847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34101095d0 msgr2=0x7f341019bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:57.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.368+0000 7f340e7fc700 1 --2- 172.21.15.70:0/2049010847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34101095d0 0x7f341019bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.368+0000 7f340e7fc700 1 -- 172.21.15.70:0/2049010847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341019c400 con 0x7f341006c4d0 2024-06-04T05:15:57.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.369+0000 7f340f7fe700 1 --2- 172.21.15.70:0/2049010847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34101095d0 0x7f341019bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:57.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.369+0000 7f340e7fc700 1 --2- 172.21.15.70:0/2049010847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f341006c4d0 0x7f34101a1f90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f340400e9c0 tx=0x7f340400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:57.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.369+0000 7f33f7fff700 1 -- 172.21.15.70:0/2049010847 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340400cbd0 con 0x7f341006c4d0 2024-06-04T05:15:57.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.369+0000 7f33f7fff700 1 -- 172.21.15.70:0/2049010847 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f340400ee60 con 0x7f341006c4d0 2024-06-04T05:15:57.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.370+0000 7f33f7fff700 1 -- 172.21.15.70:0/2049010847 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34040054b0 con 0x7f341006c4d0 2024-06-04T05:15:57.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.370+0000 7f341576c700 1 -- 172.21.15.70:0/2049010847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f341019c6f0 con 0x7f341006c4d0 2024-06-04T05:15:57.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.370+0000 7f341576c700 1 -- 172.21.15.70:0/2049010847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34101a89d0 con 0x7f341006c4d0 2024-06-04T05:15:57.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.373+0000 7f33f7fff700 1 -- 172.21.15.70:0/2049010847 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3404004750 con 0x7f341006c4d0 2024-06-04T05:15:57.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.374+0000 7f33f7fff700 1 --2- 172.21.15.70:0/2049010847 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33f80b2ba0 0x7f33f80b5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.374+0000 7f33f7fff700 1 -- 172.21.15.70:0/2049010847 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f34040e2a50 con 0x7f341006c4d0 2024-06-04T05:15:57.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.374+0000 7f3420001b80 1 Processor -- start 2024-06-04T05:15:57.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.375+0000 7f3420001b80 1 -- start start 2024-06-04T05:15:57.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.375+0000 7f340effd700 1 --2- 172.21.15.70:0/2049010847 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33f80b2ba0 0x7f33f80b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.375+0000 7f3420001b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c383663350 0x55c383663730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.375+0000 7f340effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c383663350 0x55c383663730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.375+0000 7f340effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c383663350 0x55c383663730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54776/0 (socket says 172.21.15.70:54776) 2024-06-04T05:15:57.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.375+0000 7f340effd700 1 -- 172.21.15.70:0/3583819831 learned_addr learned my addr 172.21.15.70:0/3583819831 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:57.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.375+0000 7f3420001b80 1 --2- 172.21.15.70:0/3583819831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365d980 0x55c38365de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.375+0000 7f3420001b80 1 --2- 172.21.15.70:0/3583819831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38365e340 0x55c38366a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.375+0000 7f3420001b80 1 -- 172.21.15.70:0/3583819831 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c3835bce30 con 0x55c383663350 2024-06-04T05:15:57.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.376+0000 7f3420001b80 1 -- 172.21.15.70:0/3583819831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c3835ac9d0 con 0x55c38365e340 2024-06-04T05:15:57.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.376+0000 7f3420001b80 1 -- 172.21.15.70:0/3583819831 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c3835b0a10 con 0x55c38365d980 2024-06-04T05:15:57.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.376+0000 7f340e7fc700 1 --2- 172.21.15.70:0/3583819831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365d980 0x55c38365de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.376+0000 7f340f7fe700 1 --2- 172.21.15.70:0/3583819831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38365e340 0x55c38366a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.376+0000 7f340effd700 1 --2- 172.21.15.70:0/2049010847 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33f80b2ba0 0x7f33f80b5060 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3400009810 tx=0x7f3400005dc0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:57.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.376+0000 7f340e7fc700 1 -- 172.21.15.70:0/3583819831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38365e340 msgr2=0x55c38366a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:57.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.376+0000 7f340e7fc700 1 --2- 172.21.15.70:0/3583819831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38365e340 0x55c38366a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.376+0000 7f340e7fc700 1 -- 172.21.15.70:0/3583819831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c383663350 msgr2=0x55c383663730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:57.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.376+0000 7f340e7fc700 1 --2- 172.21.15.70:0/3583819831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c383663350 0x55c383663730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.376+0000 7f340e7fc700 1 -- 172.21.15.70:0/3583819831 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3834fca70 con 0x55c38365d980 2024-06-04T05:15:57.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.376+0000 7f340e7fc700 1 --2- 172.21.15.70:0/3583819831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365d980 0x55c38365de00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f34040e0110 tx=0x7f34040a9510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4d7e73ec1c0dbe7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:57.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.376+0000 7f33f57fa700 1 -- 172.21.15.70:0/3583819831 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34040b2450 con 0x55c38365d980 2024-06-04T05:15:57.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.377+0000 7f33f57fa700 1 -- 172.21.15.70:0/3583819831 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f34040aa3a0 con 0x55c38365d980 2024-06-04T05:15:57.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.377+0000 7f33f57fa700 1 -- 172.21.15.70:0/3583819831 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34040a9f00 con 0x55c38365d980 2024-06-04T05:15:57.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.378+0000 7f3420001b80 1 -- 172.21.15.70:0/3583819831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365d980 msgr2=0x55c38365de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:57.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.378+0000 7f3420001b80 1 --2- 172.21.15.70:0/3583819831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365d980 0x55c38365de00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f34040e0110 tx=0x7f34040a9510 comp rx=0 tx=0).stop 2024-06-04T05:15:57.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.378+0000 7f3420001b80 1 -- 172.21.15.70:0/3583819831 shutdown_connections 2024-06-04T05:15:57.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.378+0000 7f3420001b80 1 --2- 172.21.15.70:0/3583819831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38365e340 0x55c38366a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.378+0000 7f3420001b80 1 --2- 172.21.15.70:0/3583819831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c383663350 0x55c383663730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.378+0000 7f3420001b80 1 --2- 172.21.15.70:0/3583819831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365d980 0x55c38365de00 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.378+0000 7f3420001b80 1 -- 172.21.15.70:0/3583819831 >> 172.21.15.70:0/3583819831 conn(0x55c38358b220 msgr2=0x55c383658290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:57.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.378+0000 7f3420001b80 1 -- 172.21.15.70:0/3583819831 shutdown_connections 2024-06-04T05:15:57.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.378+0000 7f3420001b80 1 -- 172.21.15.70:0/3583819831 wait complete. 2024-06-04T05:15:57.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.378+0000 7f3420001b80 1 -- start start 2024-06-04T05:15:57.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.378+0000 7f3420001b80 1 Processor -- start 2024-06-04T05:15:57.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.379+0000 7f3420001b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 0x55c383669e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.379+0000 7f3420001b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365e340 0x55c383667140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.379+0000 7f3420001b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c383663350 0x55c383667680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.379+0000 7f3420001b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c3834b3140 con 0x55c38365d980 2024-06-04T05:15:57.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.379+0000 7f3420001b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c38354e960 con 0x55c383663350 2024-06-04T05:15:57.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.379+0000 7f3420001b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c38366a7e0 con 0x55c38365e340 2024-06-04T05:15:57.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.379+0000 7f3420001b80 10 client.0 _ll_register_callbacks cb 0x55c383547ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:57.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.379+0000 7f340effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 0x55c383669e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.379+0000 7f340e7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365e340 0x55c383667140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.379+0000 7f340effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 0x55c383669e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54784/0 (socket says 172.21.15.70:54784) 2024-06-04T05:15:57.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.379+0000 7f340effd700 1 -- 172.21.15.70:0/3324763724 learned_addr learned my addr 172.21.15.70:0/3324763724 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:57.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.379+0000 7f340f7fe700 1 --2- 172.21.15.70:0/3324763724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c383663350 0x55c383667680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.379+0000 7f340e7fc700 1 -- 172.21.15.70:0/3324763724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c383663350 msgr2=0x55c383667680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:57.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.379+0000 7f340e7fc700 1 --2- 172.21.15.70:0/3324763724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c383663350 0x55c383667680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.379+0000 7f340e7fc700 1 -- 172.21.15.70:0/3324763724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 msgr2=0x55c383669e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:57.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.379+0000 7f340e7fc700 1 --2- 172.21.15.70:0/3324763724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 0x55c383669e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.379+0000 7f340e7fc700 1 -- 172.21.15.70:0/3324763724 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c38354b830 con 0x55c38365e340 2024-06-04T05:15:57.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.380+0000 7f340e7fc700 1 --2- 172.21.15.70:0/3324763724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365e340 0x55c383667140 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f34040aafd0 tx=0x7f340400cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ac6a423c36bf6a7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:57.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.380+0000 7f33f2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:15:57.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.380+0000 7f33f2ff5700 1 -- 172.21.15.70:0/3324763724 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340408b060 con 0x55c38365e340 2024-06-04T05:15:57.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.380+0000 7f33f2ff5700 1 -- 172.21.15.70:0/3324763724 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f34040c4330 con 0x55c38365e340 2024-06-04T05:15:57.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.380+0000 7f3420001b80 1 -- 172.21.15.70:0/3324763724 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34040aa3a0 con 0x55c38365e340 2024-06-04T05:15:57.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.380+0000 7f33f2ff5700 1 -- 172.21.15.70:0/3324763724 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f34040a9ca0 con 0x55c38365e340 2024-06-04T05:15:57.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.380+0000 7f3420001b80 10 client.29245 fetch_fsmap learned FSMap version 57 2024-06-04T05:15:57.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.380+0000 7f3420001b80 1 -- 172.21.15.70:0/3324763724 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55c383674ed0 con 0x55c38365e340 2024-06-04T05:15:57.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.380+0000 7f33f2ff5700 1 -- 172.21.15.70:0/3324763724 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f34040aa3a0 con 0x55c38365e340 2024-06-04T05:15:57.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.380+0000 7f33f2ff5700 1 -- 172.21.15.70:0/3324763724 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f34040a8f00 con 0x55c38365e340 2024-06-04T05:15:57.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.381+0000 7f3420001b80 10 client.29245 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:15:57.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.381+0000 7f3420001b80 10 client.29245 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:15:57.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.381+0000 7f3420001b80 20 client.29245 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:57.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.381+0000 7f3420001b80 1 --2- 172.21.15.70:0/3324763724 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c383677360 0x55c383679770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.381+0000 7f3420001b80 4 client.29245 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:15:57.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.381+0000 7f3420001b80 1 -- 172.21.15.70:0/3324763724 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c383629b80 con 0x55c383677360 2024-06-04T05:15:57.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.381+0000 7f340effd700 1 --2- 172.21.15.70:0/3324763724 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c383677360 0x55c383679770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.382+0000 7f33f2ff5700 10 client.29245 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:57.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.382+0000 7f340effd700 1 --2- 172.21.15.70:0/3324763724 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c383677360 0x55c383679770 crc :-1 s=READY pgs=405 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a1f737b2531fdd6 server_cookie=2f1276fe8e3b54a6 in_seq=0 out_seq=0 2024-06-04T05:15:57.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.383+0000 7f33f2ff5700 1 -- 172.21.15.70:0/3324763724 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c383629b80 con 0x55c383677360 2024-06-04T05:15:57.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.383+0000 7f33f2ff5700 10 client.29245 handle_command_reply: tid=0 2024-06-04T05:15:57.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.383+0000 7f3420001b80 1 client.29245 shutdown 2024-06-04T05:15:57.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.383+0000 7f3420001b80 1 -- 172.21.15.70:0/3324763724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365e340 msgr2=0x55c383667140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:57.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.383+0000 7f3420001b80 1 --2- 172.21.15.70:0/3324763724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365e340 0x55c383667140 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f34040aafd0 tx=0x7f340400cf90 comp rx=0 tx=0).stop 2024-06-04T05:15:57.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.383+0000 7f3420001b80 1 -- 172.21.15.70:0/3324763724 shutdown_connections 2024-06-04T05:15:57.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.383+0000 7f3420001b80 1 --2- 172.21.15.70:0/3324763724 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c383677360 0x55c383679770 crc :-1 s=READY pgs=405 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.383+0000 7f3420001b80 1 --2- 172.21.15.70:0/3324763724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c383663350 0x55c383667680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.383+0000 7f3420001b80 1 --2- 172.21.15.70:0/3324763724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 0x55c383669e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.383+0000 7f33f2ff5700 0 client.29245 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:57.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.383+0000 7f3420001b80 1 --2- 172.21.15.70:0/3324763724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365e340 0x55c383667140 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.383+0000 7f3420001b80 1 -- 172.21.15.70:0/3324763724 >> 172.21.15.70:0/3324763724 conn(0x55c38358b220 msgr2=0x55c383652d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:57.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.383+0000 7f3420001b80 1 -- 172.21.15.70:0/3324763724 shutdown_connections 2024-06-04T05:15:57.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.383+0000 7f3420001b80 1 -- 172.21.15.70:0/3324763724 wait complete. 2024-06-04T05:15:57.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.383+0000 7f3420001b80 20 client.29245 trim_cache size 0 max 16384 2024-06-04T05:15:57.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.416+0000 7f3420001b80 1 Processor -- start 2024-06-04T05:15:57.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.416+0000 7f3420001b80 1 -- start start 2024-06-04T05:15:57.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.416+0000 7f3420001b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 0x55c383669080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.416+0000 7f3420001b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365e340 0x55c38365fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.416+0000 7f3420001b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c383663350 0x55c383660370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.416+0000 7f3420001b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f34101abc00 con 0x55c38365d980 2024-06-04T05:15:57.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.416+0000 7f3420001b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f34101ad740 con 0x55c383663350 2024-06-04T05:15:57.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.416+0000 7f3420001b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f34101ac2d0 con 0x55c38365e340 2024-06-04T05:15:57.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.416+0000 7f340effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 0x55c383669080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.416+0000 7f340effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 0x55c383669080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54798/0 (socket says 172.21.15.70:54798) 2024-06-04T05:15:57.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.416+0000 7f340e7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365e340 0x55c38365fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.416+0000 7f340effd700 1 -- 172.21.15.70:0/3536286706 learned_addr learned my addr 172.21.15.70:0/3536286706 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:57.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.416+0000 7f340f7fe700 1 --2- 172.21.15.70:0/3536286706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c383663350 0x55c383660370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.417+0000 7f340effd700 1 -- 172.21.15.70:0/3536286706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365e340 msgr2=0x55c38365fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:57.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.417+0000 7f340effd700 1 --2- 172.21.15.70:0/3536286706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365e340 0x55c38365fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.417+0000 7f340effd700 1 -- 172.21.15.70:0/3536286706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c383663350 msgr2=0x55c383660370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:57.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.417+0000 7f340effd700 1 --2- 172.21.15.70:0/3536286706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c383663350 0x55c383660370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.417+0000 7f340effd700 1 -- 172.21.15.70:0/3536286706 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3836587d0 con 0x55c38365d980 2024-06-04T05:15:57.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.417+0000 7f340effd700 1 --2- 172.21.15.70:0/3536286706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 0x55c383669080 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f34000098f0 tx=0x7f3400026000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba5835cd0141216 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:57.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.417+0000 7f33f4ff9700 1 -- 172.21.15.70:0/3536286706 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340002f940 con 0x55c38365d980 2024-06-04T05:15:57.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.417+0000 7f33f4ff9700 1 -- 172.21.15.70:0/3536286706 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f340003f070 con 0x55c38365d980 2024-06-04T05:15:57.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.417+0000 7f33f4ff9700 1 -- 172.21.15.70:0/3536286706 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34000038e0 con 0x55c38365d980 2024-06-04T05:15:57.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.418+0000 7f3420001b80 1 -- 172.21.15.70:0/3536286706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 msgr2=0x55c383669080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:57.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.418+0000 7f3420001b80 1 --2- 172.21.15.70:0/3536286706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 0x55c383669080 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f34000098f0 tx=0x7f3400026000 comp rx=0 tx=0).stop 2024-06-04T05:15:57.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.418+0000 7f3420001b80 1 -- 172.21.15.70:0/3536286706 shutdown_connections 2024-06-04T05:15:57.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.418+0000 7f3420001b80 1 --2- 172.21.15.70:0/3536286706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c383663350 0x55c383660370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.418+0000 7f3420001b80 1 --2- 172.21.15.70:0/3536286706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 0x55c383669080 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.418+0000 7f3420001b80 1 --2- 172.21.15.70:0/3536286706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365e340 0x55c38365fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.418+0000 7f3420001b80 1 -- 172.21.15.70:0/3536286706 >> 172.21.15.70:0/3536286706 conn(0x55c38358b220 msgr2=0x55c383668970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:57.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.418+0000 7f3420001b80 1 -- 172.21.15.70:0/3536286706 shutdown_connections 2024-06-04T05:15:57.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.418+0000 7f3420001b80 1 -- 172.21.15.70:0/3536286706 wait complete. 2024-06-04T05:15:57.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.419+0000 7f3420001b80 1 -- start start 2024-06-04T05:15:57.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.419+0000 7f3420001b80 1 Processor -- start 2024-06-04T05:15:57.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.419+0000 7f3420001b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 0x55c38366fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.419+0000 7f3420001b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365e340 0x55c38366ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.419+0000 7f3420001b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c383663350 0x55c38367e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.419+0000 7f3420001b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f34101ad070 con 0x55c38365d980 2024-06-04T05:15:57.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.419+0000 7f3420001b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f34101ac9a0 con 0x55c383663350 2024-06-04T05:15:57.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.420+0000 7f3420001b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f341010e550 con 0x55c38365e340 2024-06-04T05:15:57.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.420+0000 7f340effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 0x55c38366fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.420+0000 7f340effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 0x55c38366fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54802/0 (socket says 172.21.15.70:54802) 2024-06-04T05:15:57.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.420+0000 7f340effd700 1 -- 172.21.15.70:0/1180008327 learned_addr learned my addr 172.21.15.70:0/1180008327 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:15:57.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.420+0000 7f340f7fe700 1 --2- 172.21.15.70:0/1180008327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c383663350 0x55c38367e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.420+0000 7f340e7fc700 1 --2- 172.21.15.70:0/1180008327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365e340 0x55c38366ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.420+0000 7f3420001b80 10 client.0 _ll_register_callbacks cb 0x55c383547ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:15:57.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.420+0000 7f340effd700 1 -- 172.21.15.70:0/1180008327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365e340 msgr2=0x55c38366ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:57.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.420+0000 7f340effd700 1 --2- 172.21.15.70:0/1180008327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365e340 0x55c38366ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.420+0000 7f340effd700 1 -- 172.21.15.70:0/1180008327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c383663350 msgr2=0x55c38367e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:15:57.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.420+0000 7f340effd700 1 --2- 172.21.15.70:0/1180008327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c383663350 0x55c38367e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.420+0000 7f340effd700 1 -- 172.21.15.70:0/1180008327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c38367e6b0 con 0x55c38365d980 2024-06-04T05:15:57.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.420+0000 7f340f7fe700 1 --2- 172.21.15.70:0/1180008327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c383663350 0x55c38367e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:57.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.420+0000 7f340e7fc700 1 --2- 172.21.15.70:0/1180008327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365e340 0x55c38366ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:15:57.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.421+0000 7f340effd700 1 --2- 172.21.15.70:0/1180008327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 0x55c38366fa30 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f340000fe00 tx=0x7f340000b360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b14ebd168bab944 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:15:57.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.421+0000 7f33f2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:15:57.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.421+0000 7f33f2ff5700 1 -- 172.21.15.70:0/1180008327 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3400047020 con 0x55c38365d980 2024-06-04T05:15:57.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.421+0000 7f33f2ff5700 1 -- 172.21.15.70:0/1180008327 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8571+0+0 (secure 0 0 0) 0x7f3400005280 con 0x55c38365d980 2024-06-04T05:15:57.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.421+0000 7f3420001b80 1 -- 172.21.15.70:0/1180008327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f340003f070 con 0x55c38365d980 2024-06-04T05:15:57.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.421+0000 7f33f2ff5700 1 -- 172.21.15.70:0/1180008327 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f340002f940 con 0x55c38365d980 2024-06-04T05:15:57.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.421+0000 7f3420001b80 10 client.20343 fetch_fsmap learned FSMap version 57 2024-06-04T05:15:57.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.421+0000 7f3420001b80 1 -- 172.21.15.70:0/1180008327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55c38367f8f0 con 0x55c38365d980 2024-06-04T05:15:57.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.421+0000 7f33f2ff5700 1 -- 172.21.15.70:0/1180008327 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f340003f070 con 0x55c38365d980 2024-06-04T05:15:57.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.422+0000 7f33f2ff5700 1 -- 172.21.15.70:0/1180008327 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f3400003c20 con 0x55c38365d980 2024-06-04T05:15:57.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.422+0000 7f3420001b80 10 client.20343 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:15:57.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.422+0000 7f3420001b80 10 client.20343 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:15:57.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.422+0000 7f3420001b80 20 client.20343 populate_metadata read hostname 'smithi070' 2024-06-04T05:15:57.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.422+0000 7f3420001b80 1 --2- 172.21.15.70:0/1180008327 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c383677360 0x55c3836e51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:15:57.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.422+0000 7f3420001b80 4 client.20343 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 146.85, "await_for": 45.0}] 2024-06-04T05:15:57.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.422+0000 7f3420001b80 1 -- 172.21.15.70:0/1180008327 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 146.85, "await_for": 45.0}) -- 0x55c3835bed80 con 0x55c383677360 2024-06-04T05:15:57.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.422+0000 7f340e7fc700 1 --2- 172.21.15.70:0/1180008327 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c383677360 0x55c3836e51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:15:57.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.423+0000 7f33f2ff5700 10 client.20343 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:57.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.423+0000 7f340e7fc700 1 --2- 172.21.15.70:0/1180008327 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c383677360 0x55c3836e51b0 crc :-1 s=READY pgs=406 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9efb298f7ae0e099 server_cookie=d86ca9fdaa698be9 in_seq=0 out_seq=0 2024-06-04T05:15:57.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:57 smithi070 ceph-mon[32631]: pgmap v588: 97 pgs: 97 active+clean; 670 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.5 MiB/s wr, 385 op/s 2024-06-04T05:15:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:57 smithi112 ceph-mon[36453]: pgmap v588: 97 pgs: 97 active+clean; 670 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.5 MiB/s wr, 385 op/s 2024-06-04T05:15:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:57 smithi098 ceph-mon[35569]: pgmap v588: 97 pgs: 97 active+clean; 670 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.5 MiB/s wr, 385 op/s 2024-06-04T05:15:57.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.751+0000 7f33f2ff5700 1 -- 172.21.15.70:0/1180008327 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55c3835bed80 con 0x55c383677360 2024-06-04T05:15:57.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.751+0000 7f33f2ff5700 10 client.20343 handle_command_reply: tid=0 2024-06-04T05:15:57.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.751+0000 7f3420001b80 1 client.20343 shutdown 2024-06-04T05:15:57.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.751+0000 7f3420001b80 1 -- 172.21.15.70:0/1180008327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 msgr2=0x55c38366fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:57.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.751+0000 7f3420001b80 1 --2- 172.21.15.70:0/1180008327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 0x55c38366fa30 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f340000fe00 tx=0x7f340000b360 comp rx=0 tx=0).stop 2024-06-04T05:15:57.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.751+0000 7f3420001b80 1 -- 172.21.15.70:0/1180008327 shutdown_connections 2024-06-04T05:15:57.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.751+0000 7f3420001b80 1 --2- 172.21.15.70:0/1180008327 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c383677360 0x55c3836e51b0 crc :-1 s=READY pgs=406 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.751+0000 7f3420001b80 1 --2- 172.21.15.70:0/1180008327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c383663350 0x55c38367e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.751+0000 7f3420001b80 1 --2- 172.21.15.70:0/1180008327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38365d980 0x55c38366fa30 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.751+0000 7f33f2ff5700 0 client.20343 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:15:57.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.751+0000 7f3420001b80 1 --2- 172.21.15.70:0/1180008327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38365e340 0x55c38366ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.751+0000 7f3420001b80 1 -- 172.21.15.70:0/1180008327 >> 172.21.15.70:0/1180008327 conn(0x55c38358b220 msgr2=0x55c38365d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:57.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.752+0000 7f3420001b80 1 -- 172.21.15.70:0/1180008327 shutdown_connections 2024-06-04T05:15:57.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.752+0000 7f3420001b80 1 -- 172.21.15.70:0/1180008327 wait complete. 2024-06-04T05:15:57.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.752+0000 7f3420001b80 20 client.20343 trim_cache size 0 max 16384 2024-06-04T05:15:57.755 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:15:57.755 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 57, 2024-06-04T05:15:57.756 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:15:57.756 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 58, 2024-06-04T05:15:57.756 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:15:57.756 INFO:teuthology.orchestra.run.smithi070.stdout: "724664c3": { 2024-06-04T05:15:57.756 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 58, 2024-06-04T05:15:57.756 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:15:57.756 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:15:57.756 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:15:57.756 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:15:57.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:15:57.756 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:15:57.756 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 146.8, 2024-06-04T05:15:57.756 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:15:57.757 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:15:57.757 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:15:57.757 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:15:57.757 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:15:57.757 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:15:57.757 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:15:57.757 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:15:57.757 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:15:57.757 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:15:57.757 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:15:57.757 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:15:57.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.752+0000 7f341576c700 1 -- 172.21.15.70:0/2049010847 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33f80b2ba0 msgr2=0x7f33f80b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:57.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.752+0000 7f341576c700 1 --2- 172.21.15.70:0/2049010847 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33f80b2ba0 0x7f33f80b5060 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3400009810 tx=0x7f3400005dc0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.752+0000 7f341576c700 1 -- 172.21.15.70:0/2049010847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f341006c4d0 msgr2=0x7f34101a1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:15:57.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.752+0000 7f341576c700 1 --2- 172.21.15.70:0/2049010847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f341006c4d0 0x7f34101a1f90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f340400e9c0 tx=0x7f340400ecd0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.752+0000 7f341576c700 1 -- 172.21.15.70:0/2049010847 shutdown_connections 2024-06-04T05:15:57.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.752+0000 7f341576c700 1 --2- 172.21.15.70:0/2049010847 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f33f80b2ba0 0x7f33f80b5060 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.752+0000 7f341576c700 1 --2- 172.21.15.70:0/2049010847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34101095d0 0x7f341019bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.752+0000 7f341576c700 1 --2- 172.21.15.70:0/2049010847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f341006bb10 0x7f34101a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.752+0000 7f341576c700 1 --2- 172.21.15.70:0/2049010847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f341006c4d0 0x7f34101a1f90 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:15:57.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.752+0000 7f341576c700 1 -- 172.21.15.70:0/2049010847 >> 172.21.15.70:0/2049010847 conn(0x7f3410079480 msgr2=0x7f34101010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:15:57.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.753+0000 7f341576c700 1 -- 172.21.15.70:0/2049010847 shutdown_connections 2024-06-04T05:15:57.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:15:57.753+0000 7f341576c700 1 -- 172.21.15.70:0/2049010847 wait complete. 2024-06-04T05:15:57.817 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.86 MB/sec execute 169 sec latency 320.285 ms 2024-06-04T05:15:58.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:58 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:15:58.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:58 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:15:58.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:58 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:15:58.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:58 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:15:58.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:58 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:15:58.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:58 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:15:58.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:58 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:15:58.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:58 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:15:58.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:58 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:15:58.818 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.81 MB/sec execute 170 sec latency 1295.740 ms 2024-06-04T05:15:58.855 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '724664c3', quiesce took 12.5 seconds. Will release after: 14.350000000000001 2024-06-04T05:15:59.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:59 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:15:59.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:59 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:15:59.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:15:59 smithi070 ceph-mon[32631]: pgmap v589: 97 pgs: 97 active+clean; 670 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 253 op/s 2024-06-04T05:15:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:59 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:15:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:59 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:15:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:15:59 smithi112 ceph-mon[36453]: pgmap v589: 97 pgs: 97 active+clean; 670 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 253 op/s 2024-06-04T05:15:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:59 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:15:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:59 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:15:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:15:59 smithi098 ceph-mon[35569]: pgmap v589: 97 pgs: 97 active+clean; 670 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 253 op/s 2024-06-04T05:15:59.818 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.77 MB/sec execute 171 sec latency 2295.863 ms 2024-06-04T05:16:00.818 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.72 MB/sec execute 172 sec latency 3295.910 ms 2024-06-04T05:16:01.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:01 smithi070 ceph-mon[32631]: pgmap v590: 97 pgs: 97 active+clean; 670 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 253 op/s 2024-06-04T05:16:01.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:01 smithi112 ceph-mon[36453]: pgmap v590: 97 pgs: 97 active+clean; 670 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 253 op/s 2024-06-04T05:16:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:01 smithi098 ceph-mon[35569]: pgmap v590: 97 pgs: 97 active+clean; 670 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 253 op/s 2024-06-04T05:16:01.818 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.68 MB/sec execute 173 sec latency 4296.012 ms 2024-06-04T05:16:02.818 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.63 MB/sec execute 174 sec latency 5296.125 ms 2024-06-04T05:16:03.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:03 smithi070 ceph-mon[32631]: pgmap v591: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.5 MiB/s wr, 348 op/s 2024-06-04T05:16:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:03 smithi112 ceph-mon[36453]: pgmap v591: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.5 MiB/s wr, 348 op/s 2024-06-04T05:16:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:03 smithi098 ceph-mon[35569]: pgmap v591: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.5 MiB/s wr, 348 op/s 2024-06-04T05:16:03.818 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.59 MB/sec execute 175 sec latency 6296.231 ms 2024-06-04T05:16:04.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:04 smithi070 ceph-mon[32631]: osdmap e105: 12 total, 12 up, 12 in 2024-06-04T05:16:04.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:04 smithi112 ceph-mon[36453]: osdmap e105: 12 total, 12 up, 12 in 2024-06-04T05:16:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:04 smithi098 ceph-mon[35569]: osdmap e105: 12 total, 12 up, 12 in 2024-06-04T05:16:04.818 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.55 MB/sec execute 176 sec latency 7296.309 ms 2024-06-04T05:16:05.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:05 smithi070 ceph-mon[32631]: pgmap v593: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 292 op/s 2024-06-04T05:16:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:05 smithi112 ceph-mon[36453]: pgmap v593: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 292 op/s 2024-06-04T05:16:05.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:05 smithi098 ceph-mon[35569]: pgmap v593: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 292 op/s 2024-06-04T05:16:05.818 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.50 MB/sec execute 177 sec latency 8296.427 ms 2024-06-04T05:16:06.818 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.46 MB/sec execute 178 sec latency 9296.542 ms 2024-06-04T05:16:07.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.468+0000 7fecf6540640 1 -- 172.21.15.70:0/3860619508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf0103420 msgr2=0x7fecf010f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:07.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.468+0000 7fecf6540640 1 --2- 172.21.15.70:0/3860619508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf0103420 0x7fecf010f930 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fece8009cd0 tx=0x7fece802a8e0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.468+0000 7fecf6540640 1 -- 172.21.15.70:0/3860619508 shutdown_connections 2024-06-04T05:16:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.468+0000 7fecf6540640 1 --2- 172.21.15.70:0/3860619508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf0103420 0x7fecf010f930 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.468+0000 7fecf6540640 1 --2- 172.21.15.70:0/3860619508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecf0102a60 0x7fecf0102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.468+0000 7fecf6540640 1 --2- 172.21.15.70:0/3860619508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecf0108a70 0x7fecf0108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.468+0000 7fecf6540640 1 -- 172.21.15.70:0/3860619508 >> 172.21.15.70:0/3860619508 conn(0x7fecf0079ee0 msgr2=0x7fecf007a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.468+0000 7fecf6540640 1 -- 172.21.15.70:0/3860619508 shutdown_connections 2024-06-04T05:16:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.468+0000 7fecf6540640 1 -- 172.21.15.70:0/3860619508 wait complete. 2024-06-04T05:16:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.468+0000 7fecf6540640 1 Processor -- start 2024-06-04T05:16:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.468+0000 7fecf6540640 1 -- start start 2024-06-04T05:16:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.469+0000 7fecf6540640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecf0102a60 0x7fecf0074d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.469+0000 7fecf6540640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf0103420 0x7fecf0075290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.469+0000 7fecf6540640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecf0108a70 0x7fecf00757d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.469+0000 7fecf6540640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fecf0112be0 con 0x7fecf0103420 2024-06-04T05:16:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.469+0000 7fecf6540640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fecf0112a60 con 0x7fecf0108a70 2024-06-04T05:16:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.469+0000 7fecf6540640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fecf0112d60 con 0x7fecf0102a60 2024-06-04T05:16:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.469+0000 7fecef7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf0103420 0x7fecf0075290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.469+0000 7fecef7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf0103420 0x7fecf0075290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42978/0 (socket says 172.21.15.70:42978) 2024-06-04T05:16:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.469+0000 7fecef7fe640 1 -- 172.21.15.70:0/2560550786 learned_addr learned my addr 172.21.15.70:0/2560550786 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.469+0000 7feceffff640 1 --2- 172.21.15.70:0/2560550786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecf0102a60 0x7fecf0074d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.469+0000 7fecf4add640 1 --2- 172.21.15.70:0/2560550786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecf0108a70 0x7fecf00757d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.469+0000 7fecef7fe640 1 -- 172.21.15.70:0/2560550786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecf0102a60 msgr2=0x7fecf0074d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.469+0000 7fecef7fe640 1 --2- 172.21.15.70:0/2560550786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecf0102a60 0x7fecf0074d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.469+0000 7fecef7fe640 1 -- 172.21.15.70:0/2560550786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecf0108a70 msgr2=0x7fecf00757d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.469+0000 7fecef7fe640 1 --2- 172.21.15.70:0/2560550786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecf0108a70 0x7fecf00757d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.469+0000 7fecef7fe640 1 -- 172.21.15.70:0/2560550786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf01a75a0 con 0x7fecf0103420 2024-06-04T05:16:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.469+0000 7fecef7fe640 1 --2- 172.21.15.70:0/2560550786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf0103420 0x7fecf0075290 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fece400d950 tx=0x7fece400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.470+0000 7feced7fa640 1 -- 172.21.15.70:0/2560550786 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4017070 con 0x7fecf0103420 2024-06-04T05:16:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.470+0000 7feced7fa640 1 -- 172.21.15.70:0/2560550786 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fece4009c30 con 0x7fecf0103420 2024-06-04T05:16:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.470+0000 7feced7fa640 1 -- 172.21.15.70:0/2560550786 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece400f620 con 0x7fecf0103420 2024-06-04T05:16:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.470+0000 7fecf6540640 1 -- 172.21.15.70:0/2560550786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecf01a7890 con 0x7fecf0103420 2024-06-04T05:16:07.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.470+0000 7fecf6540640 1 -- 172.21.15.70:0/2560550786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecf01a7de0 con 0x7fecf0103420 2024-06-04T05:16:07.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.472+0000 7feced7fa640 1 -- 172.21.15.70:0/2560550786 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fece4010460 con 0x7fecf0103420 2024-06-04T05:16:07.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.472+0000 7fecf925c740 1 Processor -- start 2024-06-04T05:16:07.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.472+0000 7fecf925c740 1 -- start start 2024-06-04T05:16:07.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7fecf925c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de63b820 0x55c8de63bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7fecf925c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de63c140 0x55c8de6464c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7fecf925c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de646a00 0x55c8de648df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7fecf925c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c8de5adc30 con 0x55c8de646a00 2024-06-04T05:16:07.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7fecf925c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c8de59d7d0 con 0x55c8de63c140 2024-06-04T05:16:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7fecf925c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c8de5a1810 con 0x55c8de63b820 2024-06-04T05:16:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7fecef7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de646a00 0x55c8de648df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7fecf4add640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de63b820 0x55c8de63bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7fecef7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de646a00 0x55c8de648df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42982/0 (socket says 172.21.15.70:42982) 2024-06-04T05:16:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7fecef7fe640 1 -- 172.21.15.70:0/2413946044 learned_addr learned my addr 172.21.15.70:0/2413946044 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7feceffff640 1 --2- 172.21.15.70:0/2413946044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de63c140 0x55c8de6464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7feced7fa640 1 --2- 172.21.15.70:0/2560550786 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fecd80b2a10 0x7fecd80b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7fecef7fe640 1 -- 172.21.15.70:0/2413946044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de63b820 msgr2=0x55c8de63bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7fecef7fe640 1 --2- 172.21.15.70:0/2413946044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de63b820 0x55c8de63bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7fecef7fe640 1 -- 172.21.15.70:0/2413946044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de63c140 msgr2=0x55c8de6464c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7fecef7fe640 1 --2- 172.21.15.70:0/2413946044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de63c140 0x55c8de6464c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7fecef7fe640 1 -- 172.21.15.70:0/2413946044 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8de2b2210 con 0x55c8de646a00 2024-06-04T05:16:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7fecf4add640 1 --2- 172.21.15.70:0/2560550786 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fecd80b2a10 0x7fecd80b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.473+0000 7fecef7fe640 1 --2- 172.21.15.70:0/2413946044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de646a00 0x55c8de648df0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fece40df860 tx=0x7fece40bab30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d3fe8f928b16447 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.474+0000 7feced7fa640 1 -- 172.21.15.70:0/2560550786 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8615+0+0 (secure 0 0 0) 0x7fece40e2160 con 0x7fecf0103420 2024-06-04T05:16:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.474+0000 7fecd67fc640 1 -- 172.21.15.70:0/2413946044 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4020070 con 0x55c8de646a00 2024-06-04T05:16:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.474+0000 7fecd67fc640 1 -- 172.21.15.70:0/2413946044 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fece40a9040 con 0x55c8de646a00 2024-06-04T05:16:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.474+0000 7fecd67fc640 1 -- 172.21.15.70:0/2413946044 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece40a8ca0 con 0x55c8de646a00 2024-06-04T05:16:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.474+0000 7fecf925c740 1 -- 172.21.15.70:0/2413946044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de646a00 msgr2=0x55c8de648df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.474+0000 7fecf4add640 1 --2- 172.21.15.70:0/2560550786 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fecd80b2a10 0x7fecd80b4ed0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fece8004ab0 tx=0x7fece80095c0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.474+0000 7fecf925c740 1 --2- 172.21.15.70:0/2413946044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de646a00 0x55c8de648df0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fece40df860 tx=0x7fece40bab30 comp rx=0 tx=0).stop 2024-06-04T05:16:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.474+0000 7fecf925c740 1 -- 172.21.15.70:0/2413946044 shutdown_connections 2024-06-04T05:16:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.474+0000 7fecf925c740 1 --2- 172.21.15.70:0/2413946044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de63c140 0x55c8de6464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.474+0000 7fecf925c740 1 --2- 172.21.15.70:0/2413946044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de646a00 0x55c8de648df0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.474+0000 7fecf925c740 1 --2- 172.21.15.70:0/2413946044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de63b820 0x55c8de63bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.474+0000 7fecf925c740 1 -- 172.21.15.70:0/2413946044 >> 172.21.15.70:0/2413946044 conn(0x55c8de637d30 msgr2=0x55c8de638110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.475+0000 7fecf925c740 1 -- 172.21.15.70:0/2413946044 shutdown_connections 2024-06-04T05:16:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.475+0000 7fecf925c740 1 -- 172.21.15.70:0/2413946044 wait complete. 2024-06-04T05:16:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.475+0000 7fecf925c740 1 -- start start 2024-06-04T05:16:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.475+0000 7fecf925c740 1 Processor -- start 2024-06-04T05:16:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.475+0000 7fecf925c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de63b820 0x55c8de643a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.475+0000 7fecf925c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63c140 0x55c8de643fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.475+0000 7fecf925c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de646a00 0x55c8de6444e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.475+0000 7fecf925c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c8de283790 con 0x55c8de63c140 2024-06-04T05:16:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.475+0000 7fecf925c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c8de559890 con 0x55c8de646a00 2024-06-04T05:16:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.475+0000 7fecf925c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c8de649570 con 0x55c8de63b820 2024-06-04T05:16:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7fecf925c740 10 client.0 _ll_register_callbacks cb 0x55c8de2b1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7fecef7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63c140 0x55c8de643fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7fecef7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63c140 0x55c8de643fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42992/0 (socket says 172.21.15.70:42992) 2024-06-04T05:16:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7fecef7fe640 1 -- 172.21.15.70:0/1727465748 learned_addr learned my addr 172.21.15.70:0/1727465748 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7fecf4add640 1 --2- 172.21.15.70:0/1727465748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de646a00 0x55c8de6444e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7feceffff640 1 --2- 172.21.15.70:0/1727465748 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de63b820 0x55c8de643a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7fecef7fe640 1 -- 172.21.15.70:0/1727465748 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de63b820 msgr2=0x55c8de643a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7fecef7fe640 1 --2- 172.21.15.70:0/1727465748 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de63b820 0x55c8de643a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7fecef7fe640 1 -- 172.21.15.70:0/1727465748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de646a00 msgr2=0x55c8de6444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7fecef7fe640 1 --2- 172.21.15.70:0/1727465748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de646a00 0x55c8de6444e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7fecef7fe640 1 -- 172.21.15.70:0/1727465748 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8de37e620 con 0x55c8de63c140 2024-06-04T05:16:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7feceffff640 1 --2- 172.21.15.70:0/1727465748 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de63b820 0x55c8de643a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:16:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7fecef7fe640 1 --2- 172.21.15.70:0/1727465748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63c140 0x55c8de643fa0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fece40a94c0 tx=0x7fece40df3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9673412c7938f4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7feccf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7feccf7fe640 1 -- 172.21.15.70:0/1727465748 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4020030 con 0x55c8de63c140 2024-06-04T05:16:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7feccf7fe640 1 -- 172.21.15.70:0/1727465748 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8615+0+0 (secure 0 0 0) 0x7fece40dfb70 con 0x55c8de63c140 2024-06-04T05:16:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7fecf925c740 1 -- 172.21.15.70:0/1727465748 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fece40a9040 con 0x55c8de63c140 2024-06-04T05:16:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7feccf7fe640 1 -- 172.21.15.70:0/1727465748 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fece40a8ca0 con 0x55c8de63c140 2024-06-04T05:16:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7fecf925c740 10 client.20355 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.476+0000 7fecf925c740 1 -- 172.21.15.70:0/1727465748 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55c8de6553a0 con 0x55c8de63c140 2024-06-04T05:16:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.477+0000 7feccf7fe640 1 -- 172.21.15.70:0/1727465748 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fece40a9040 con 0x55c8de63c140 2024-06-04T05:16:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.477+0000 7feccf7fe640 1 -- 172.21.15.70:0/1727465748 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fece40a57c0 con 0x55c8de63c140 2024-06-04T05:16:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.477+0000 7fecf925c740 10 client.20355 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.477+0000 7fecf925c740 10 client.20355 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:16:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.477+0000 7fecf925c740 20 client.20355 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.477+0000 7fecf925c740 1 --2- 172.21.15.70:0/1727465748 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c8de6566f0 0x55c8de658ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.477+0000 7fecf925c740 4 client.20355 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:16:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.477+0000 7fecf925c740 1 -- 172.21.15.70:0/1727465748 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8de5afb80 con 0x55c8de6566f0 2024-06-04T05:16:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.477+0000 7feceffff640 1 --2- 172.21.15.70:0/1727465748 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c8de6566f0 0x55c8de658ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.478+0000 7feccf7fe640 10 client.20355 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.478+0000 7feceffff640 1 --2- 172.21.15.70:0/1727465748 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c8de6566f0 0x55c8de658ae0 crc :-1 s=READY pgs=407 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0f12b1c6ec8be1 server_cookie=2032faa365008db in_seq=0 out_seq=0 2024-06-04T05:16:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.478+0000 7feccf7fe640 1 -- 172.21.15.70:0/1727465748 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c8de5afb80 con 0x55c8de6566f0 2024-06-04T05:16:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.478+0000 7feccf7fe640 10 client.20355 handle_command_reply: tid=0 2024-06-04T05:16:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.478+0000 7fecf925c740 1 client.20355 shutdown 2024-06-04T05:16:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.478+0000 7fecf925c740 1 -- 172.21.15.70:0/1727465748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63c140 msgr2=0x55c8de643fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.478+0000 7fecf925c740 1 --2- 172.21.15.70:0/1727465748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63c140 0x55c8de643fa0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fece40a94c0 tx=0x7fece40df3e0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.478+0000 7fecf925c740 1 -- 172.21.15.70:0/1727465748 shutdown_connections 2024-06-04T05:16:07.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.478+0000 7fecf925c740 1 --2- 172.21.15.70:0/1727465748 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c8de6566f0 0x55c8de658ae0 crc :-1 s=READY pgs=407 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.478+0000 7fecf925c740 1 --2- 172.21.15.70:0/1727465748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de646a00 0x55c8de6444e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.478+0000 7fecf925c740 1 --2- 172.21.15.70:0/1727465748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63c140 0x55c8de643fa0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.478+0000 7fecf925c740 1 --2- 172.21.15.70:0/1727465748 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de63b820 0x55c8de643a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.478+0000 7feccf7fe640 0 client.20355 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:07.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.479+0000 7fecf925c740 1 -- 172.21.15.70:0/1727465748 >> 172.21.15.70:0/1727465748 conn(0x55c8de637d30 msgr2=0x55c8de636fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:07.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.479+0000 7fecf925c740 1 -- 172.21.15.70:0/1727465748 shutdown_connections 2024-06-04T05:16:07.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.479+0000 7fecf925c740 1 -- 172.21.15.70:0/1727465748 wait complete. 2024-06-04T05:16:07.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.479+0000 7fecf925c740 20 client.20355 trim_cache size 0 max 16384 2024-06-04T05:16:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7fecf925c740 1 Processor -- start 2024-06-04T05:16:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7fecf925c740 1 -- start start 2024-06-04T05:16:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7fecf925c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63b820 0x55c8de6442b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7fecf925c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de63c140 0x55c8de63e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7fecf925c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de646a00 0x55c8de651cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7fecf925c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fecf01b0890 con 0x55c8de63b820 2024-06-04T05:16:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7fecf925c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fecf01b23d0 con 0x55c8de646a00 2024-06-04T05:16:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7fecf925c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fecf01b0f60 con 0x55c8de63c140 2024-06-04T05:16:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7feceffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63b820 0x55c8de6442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7feceffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63b820 0x55c8de6442b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42994/0 (socket says 172.21.15.70:42994) 2024-06-04T05:16:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7feceffff640 1 -- 172.21.15.70:0/688048322 learned_addr learned my addr 172.21.15.70:0/688048322 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7fecef7fe640 1 --2- 172.21.15.70:0/688048322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de63c140 0x55c8de63e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7feceffff640 1 -- 172.21.15.70:0/688048322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de63c140 msgr2=0x55c8de63e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7feceffff640 1 --2- 172.21.15.70:0/688048322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de63c140 0x55c8de63e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7feceffff640 1 -- 172.21.15.70:0/688048322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de646a00 msgr2=0x55c8de651cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7fecf4add640 1 --2- 172.21.15.70:0/688048322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de646a00 0x55c8de651cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7feceffff640 1 --2- 172.21.15.70:0/688048322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de646a00 0x55c8de651cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.494+0000 7feceffff640 1 -- 172.21.15.70:0/688048322 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8de641fd0 con 0x55c8de63b820 2024-06-04T05:16:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.495+0000 7feceffff640 1 --2- 172.21.15.70:0/688048322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63b820 0x55c8de6442b0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fece0000c00 tx=0x7fece000bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf23bdf92aab729 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.495+0000 7fecd5ffb640 1 -- 172.21.15.70:0/688048322 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece0003d60 con 0x55c8de63b820 2024-06-04T05:16:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.495+0000 7fecd5ffb640 1 -- 172.21.15.70:0/688048322 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fece00054b0 con 0x55c8de63b820 2024-06-04T05:16:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.495+0000 7fecd5ffb640 1 -- 172.21.15.70:0/688048322 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece001d430 con 0x55c8de63b820 2024-06-04T05:16:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.495+0000 7fecf925c740 1 -- 172.21.15.70:0/688048322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63b820 msgr2=0x55c8de6442b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.495+0000 7fecf925c740 1 --2- 172.21.15.70:0/688048322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63b820 0x55c8de6442b0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fece0000c00 tx=0x7fece000bd90 comp rx=0 tx=0).stop 2024-06-04T05:16:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.495+0000 7fecf925c740 1 -- 172.21.15.70:0/688048322 shutdown_connections 2024-06-04T05:16:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.495+0000 7fecf925c740 1 --2- 172.21.15.70:0/688048322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de646a00 0x55c8de651cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.495+0000 7fecf925c740 1 --2- 172.21.15.70:0/688048322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de63c140 0x55c8de63e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.495+0000 7fecf925c740 1 --2- 172.21.15.70:0/688048322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63b820 0x55c8de6442b0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.495+0000 7fecf925c740 1 -- 172.21.15.70:0/688048322 >> 172.21.15.70:0/688048322 conn(0x55c8de637d30 msgr2=0x55c8de6515c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.496+0000 7fecf925c740 1 -- 172.21.15.70:0/688048322 shutdown_connections 2024-06-04T05:16:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.496+0000 7fecf925c740 1 -- 172.21.15.70:0/688048322 wait complete. 2024-06-04T05:16:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.496+0000 7fecf925c740 1 -- start start 2024-06-04T05:16:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.496+0000 7fecf925c740 1 Processor -- start 2024-06-04T05:16:07.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.496+0000 7fecf925c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de63b820 0x55c8de64e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.496+0000 7fecf925c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63c140 0x55c8de64ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.496+0000 7fecf925c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de646a00 0x55c8de64b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.496+0000 7fecf925c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fecf01b1d00 con 0x55c8de63c140 2024-06-04T05:16:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.496+0000 7fecf925c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fecf01b1630 con 0x55c8de63b820 2024-06-04T05:16:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.496+0000 7fecf925c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fecf01ae890 con 0x55c8de646a00 2024-06-04T05:16:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.496+0000 7fecf925c740 10 client.0 _ll_register_callbacks cb 0x7fece0004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.496+0000 7fecef7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63c140 0x55c8de64ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.496+0000 7fecef7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63c140 0x55c8de64ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43008/0 (socket says 172.21.15.70:43008) 2024-06-04T05:16:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.496+0000 7fecef7fe640 1 -- 172.21.15.70:0/1067149135 learned_addr learned my addr 172.21.15.70:0/1067149135 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7fecf4add640 1 --2- 172.21.15.70:0/1067149135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de646a00 0x55c8de64b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7feceffff640 1 --2- 172.21.15.70:0/1067149135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de63b820 0x55c8de64e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7fecef7fe640 1 -- 172.21.15.70:0/1067149135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de646a00 msgr2=0x55c8de64b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7fecef7fe640 1 --2- 172.21.15.70:0/1067149135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de646a00 0x55c8de64b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7fecef7fe640 1 -- 172.21.15.70:0/1067149135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de63b820 msgr2=0x55c8de64e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7fecef7fe640 1 --2- 172.21.15.70:0/1067149135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de63b820 0x55c8de64e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7fecef7fe640 1 -- 172.21.15.70:0/1067149135 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8de64bcd0 con 0x55c8de63c140 2024-06-04T05:16:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7fecef7fe640 1 --2- 172.21.15.70:0/1067149135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63c140 0x55c8de64ed00 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fece400bfd0 tx=0x7fece40e2540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e577ac3e91bf8314 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7feccf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7feccf7fe640 1 -- 172.21.15.70:0/1067149135 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4020050 con 0x55c8de63c140 2024-06-04T05:16:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7feccf7fe640 1 -- 172.21.15.70:0/1067149135 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8615+0+0 (secure 0 0 0) 0x7fece40b1e50 con 0x55c8de63c140 2024-06-04T05:16:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7fecf925c740 1 -- 172.21.15.70:0/1067149135 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fece00054b0 con 0x55c8de63c140 2024-06-04T05:16:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7feccf7fe640 1 -- 172.21.15.70:0/1067149135 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fece40a8e80 con 0x55c8de63c140 2024-06-04T05:16:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7fecf925c740 10 client.20361 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7fecf925c740 1 -- 172.21.15.70:0/1067149135 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55c8de64bcd0 con 0x55c8de63c140 2024-06-04T05:16:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7feccf7fe640 1 -- 172.21.15.70:0/1067149135 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fece00054b0 con 0x55c8de63c140 2024-06-04T05:16:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7feccf7fe640 1 -- 172.21.15.70:0/1067149135 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fece40a9790 con 0x55c8de63c140 2024-06-04T05:16:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.497+0000 7fecf925c740 10 client.20361 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.498+0000 7fecf925c740 10 client.20361 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:16:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.498+0000 7fecf925c740 20 client.20361 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.498+0000 7fecf925c740 1 --2- 172.21.15.70:0/1067149135 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c8de6c1a40 0x55c8de6c3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.498+0000 7fecf925c740 4 client.20361 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:16:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.498+0000 7fecf925c740 1 -- 172.21.15.70:0/1067149135 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c8de5a9360 con 0x55c8de6c1a40 2024-06-04T05:16:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.498+0000 7feceffff640 1 --2- 172.21.15.70:0/1067149135 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c8de6c1a40 0x55c8de6c3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.498+0000 7feceffff640 1 --2- 172.21.15.70:0/1067149135 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c8de6c1a40 0x55c8de6c3e60 crc :-1 s=READY pgs=408 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e368f3dc71399a server_cookie=9d20bc872bf24263 in_seq=0 out_seq=0 2024-06-04T05:16:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.498+0000 7feccf7fe640 10 client.20361 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.549+0000 7feccf7fe640 1 -- 172.21.15.70:0/1067149135 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55c8de5a9360 con 0x55c8de6c1a40 2024-06-04T05:16:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.549+0000 7feccf7fe640 10 client.20361 handle_command_reply: tid=0 2024-06-04T05:16:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.550+0000 7fecf925c740 1 client.20361 shutdown 2024-06-04T05:16:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.550+0000 7fecf925c740 1 -- 172.21.15.70:0/1067149135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63c140 msgr2=0x55c8de64ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.550+0000 7fecf925c740 1 --2- 172.21.15.70:0/1067149135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63c140 0x55c8de64ed00 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fece400bfd0 tx=0x7fece40e2540 comp rx=0 tx=0).stop 2024-06-04T05:16:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.550+0000 7fecf925c740 1 -- 172.21.15.70:0/1067149135 shutdown_connections 2024-06-04T05:16:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.550+0000 7fecf925c740 1 --2- 172.21.15.70:0/1067149135 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c8de6c1a40 0x55c8de6c3e60 crc :-1 s=READY pgs=408 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.550+0000 7fecf925c740 1 --2- 172.21.15.70:0/1067149135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c8de646a00 0x55c8de64b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.550+0000 7feccf7fe640 0 client.20361 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.550+0000 7fecf925c740 1 --2- 172.21.15.70:0/1067149135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8de63c140 0x55c8de64ed00 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.550+0000 7fecf925c740 1 --2- 172.21.15.70:0/1067149135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8de63b820 0x55c8de64e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.550+0000 7fecf925c740 1 -- 172.21.15.70:0/1067149135 >> 172.21.15.70:0/1067149135 conn(0x55c8de637d30 msgr2=0x55c8de644130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.550+0000 7fecf925c740 1 -- 172.21.15.70:0/1067149135 shutdown_connections 2024-06-04T05:16:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.550+0000 7fecf925c740 1 -- 172.21.15.70:0/1067149135 wait complete. 2024-06-04T05:16:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.550+0000 7fecf925c740 20 client.20361 trim_cache size 0 max 16384 2024-06-04T05:16:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:16:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:16:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:16:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.551+0000 7fecf6540640 1 -- 172.21.15.70:0/2560550786 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fecd80b2a10 msgr2=0x7fecd80b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.551+0000 7fecf6540640 1 --2- 172.21.15.70:0/2560550786 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fecd80b2a10 0x7fecd80b4ed0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fece8004ab0 tx=0x7fece80095c0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.551+0000 7fecf6540640 1 -- 172.21.15.70:0/2560550786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf0103420 msgr2=0x7fecf0075290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.551+0000 7fecf6540640 1 --2- 172.21.15.70:0/2560550786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf0103420 0x7fecf0075290 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fece400d950 tx=0x7fece400de10 comp rx=0 tx=0).stop 2024-06-04T05:16:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.551+0000 7fecf6540640 1 -- 172.21.15.70:0/2560550786 shutdown_connections 2024-06-04T05:16:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.551+0000 7fecf6540640 1 --2- 172.21.15.70:0/2560550786 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fecd80b2a10 0x7fecd80b4ed0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.551+0000 7fecf6540640 1 --2- 172.21.15.70:0/2560550786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecf0108a70 0x7fecf00757d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.551+0000 7fecf6540640 1 --2- 172.21.15.70:0/2560550786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf0103420 0x7fecf0075290 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.551+0000 7fecf6540640 1 --2- 172.21.15.70:0/2560550786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecf0102a60 0x7fecf0074d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.551+0000 7fecf6540640 1 -- 172.21.15.70:0/2560550786 >> 172.21.15.70:0/2560550786 conn(0x7fecf0079ee0 msgr2=0x7fecf0100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.551+0000 7fecf6540640 1 -- 172.21.15.70:0/2560550786 shutdown_connections 2024-06-04T05:16:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:07.551+0000 7fecf6540640 1 -- 172.21.15.70:0/2560550786 wait complete. 2024-06-04T05:16:07.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:07 smithi070 ceph-mon[32631]: pgmap v594: 97 pgs: 97 active+clean; 664 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 1.0 MiB/s wr, 127 op/s 2024-06-04T05:16:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:07 smithi112 ceph-mon[36453]: pgmap v594: 97 pgs: 97 active+clean; 664 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 1.0 MiB/s wr, 127 op/s 2024-06-04T05:16:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:07 smithi098 ceph-mon[35569]: pgmap v594: 97 pgs: 97 active+clean; 664 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 1.0 MiB/s wr, 127 op/s 2024-06-04T05:16:07.818 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.42 MB/sec execute 179 sec latency 10296.665 ms 2024-06-04T05:16:08.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.38 MB/sec execute 180 sec latency 11296.766 ms 2024-06-04T05:16:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:08 smithi098 ceph-mon[35569]: osdmap e106: 12 total, 12 up, 12 in 2024-06-04T05:16:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:08 smithi098 ceph-mon[35569]: pgmap v596: 97 pgs: 97 active+clean; 664 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 1.3 MiB/s wr, 159 op/s 2024-06-04T05:16:09.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:08 smithi070 ceph-mon[32631]: osdmap e106: 12 total, 12 up, 12 in 2024-06-04T05:16:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:08 smithi070 ceph-mon[32631]: pgmap v596: 97 pgs: 97 active+clean; 664 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 1.3 MiB/s wr, 159 op/s 2024-06-04T05:16:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:08 smithi112 ceph-mon[36453]: osdmap e106: 12 total, 12 up, 12 in 2024-06-04T05:16:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:08 smithi112 ceph-mon[36453]: pgmap v596: 97 pgs: 97 active+clean; 664 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 1.3 MiB/s wr, 159 op/s 2024-06-04T05:16:09.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.34 MB/sec execute 181 sec latency 12296.901 ms 2024-06-04T05:16:10.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.30 MB/sec execute 182 sec latency 13296.990 ms 2024-06-04T05:16:11.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:11 smithi070 ceph-mon[32631]: pgmap v597: 97 pgs: 97 active+clean; 664 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 272 KiB/s wr, 16 op/s 2024-06-04T05:16:11.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:11 smithi112 ceph-mon[36453]: pgmap v597: 97 pgs: 97 active+clean; 664 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 272 KiB/s wr, 16 op/s 2024-06-04T05:16:11.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:11 smithi098 ceph-mon[35569]: pgmap v597: 97 pgs: 97 active+clean; 664 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 272 KiB/s wr, 16 op/s 2024-06-04T05:16:11.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.26 MB/sec execute 183 sec latency 14297.025 ms 2024-06-04T05:16:12.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.22 MB/sec execute 184 sec latency 15297.078 ms 2024-06-04T05:16:13.206 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:16:13.477 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:16:13.547 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:13 smithi098 ceph-mon[35569]: pgmap v598: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 281 KiB/s wr, 37 op/s 2024-06-04T05:16:13.599 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:13 smithi070 ceph-mon[32631]: pgmap v598: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 281 KiB/s wr, 37 op/s 2024-06-04T05:16:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:13 smithi112 ceph-mon[36453]: pgmap v598: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 281 KiB/s wr, 37 op/s 2024-06-04T05:16:13.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.18 MB/sec execute 185 sec latency 16297.139 ms 2024-06-04T05:16:14.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.14 MB/sec execute 186 sec latency 17297.223 ms 2024-06-04T05:16:15.438 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:15 smithi070 ceph-mon[32631]: pgmap v599: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 250 KiB/s wr, 33 op/s 2024-06-04T05:16:15.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.671+0000 7f4e40e79700 1 -- 172.21.15.70:0/2100448785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c06c470 msgr2=0x7f4e3c1103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:15.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.671+0000 7f4e40e79700 1 --2- 172.21.15.70:0/2100448785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c06c470 0x7f4e3c1103c0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400b0a0 tx=0x7f4e3400b3b0 comp rx=0 tx=0).stop 2024-06-04T05:16:15.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.671+0000 7f4e40e79700 1 -- 172.21.15.70:0/2100448785 shutdown_connections 2024-06-04T05:16:15.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.671+0000 7f4e40e79700 1 --2- 172.21.15.70:0/2100448785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c06c470 0x7f4e3c1103c0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:15.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.671+0000 7f4e40e79700 1 --2- 172.21.15.70:0/2100448785 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e3c1095a0 0x7f4e3c109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:15.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.671+0000 7f4e40e79700 1 --2- 172.21.15.70:0/2100448785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e3c06bab0 0x7f4e3c06bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:15.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.671+0000 7f4e40e79700 1 -- 172.21.15.70:0/2100448785 >> 172.21.15.70:0/2100448785 conn(0x7f4e3c0793b0 msgr2=0x7f4e3c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:15.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.671+0000 7f4e40e79700 1 -- 172.21.15.70:0/2100448785 shutdown_connections 2024-06-04T05:16:15.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.671+0000 7f4e40e79700 1 -- 172.21.15.70:0/2100448785 wait complete. 2024-06-04T05:16:15.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.672+0000 7f4e40e79700 1 Processor -- start 2024-06-04T05:16:15.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.672+0000 7f4e40e79700 1 -- start start 2024-06-04T05:16:15.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.672+0000 7f4e40e79700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e3c06bab0 0x7f4e3c19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:15.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.672+0000 7f4e40e79700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e3c06c470 0x7f4e3c19fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:15.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.672+0000 7f4e40e79700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c1095a0 0x7f4e3c1998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:15.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e3ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c1095a0 0x7f4e3c1998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:15.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e3ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c1095a0 0x7f4e3c1998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43032/0 (socket says 172.21.15.70:43032) 2024-06-04T05:16:15.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e40e79700 1 -- 172.21.15.70:0/1358138521 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c113560 con 0x7f4e3c1095a0 2024-06-04T05:16:15.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e3ad9d700 1 -- 172.21.15.70:0/1358138521 learned_addr learned my addr 172.21.15.70:0/1358138521 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:15.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e40e79700 1 -- 172.21.15.70:0/1358138521 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c1133e0 con 0x7f4e3c06bab0 2024-06-04T05:16:15.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e40e79700 1 -- 172.21.15.70:0/1358138521 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c1136e0 con 0x7f4e3c06c470 2024-06-04T05:16:15.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e39d9b700 1 --2- 172.21.15.70:0/1358138521 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e3c06c470 0x7f4e3c19fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:15.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e3a59c700 1 --2- 172.21.15.70:0/1358138521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e3c06bab0 0x7f4e3c19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:15.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e3ad9d700 1 -- 172.21.15.70:0/1358138521 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e3c06c470 msgr2=0x7f4e3c19fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:15.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e3ad9d700 1 --2- 172.21.15.70:0/1358138521 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e3c06c470 0x7f4e3c19fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:15.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e3ad9d700 1 -- 172.21.15.70:0/1358138521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e3c06bab0 msgr2=0x7f4e3c19f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:15.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e3ad9d700 1 --2- 172.21.15.70:0/1358138521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e3c06bab0 0x7f4e3c19f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:15.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e3ad9d700 1 -- 172.21.15.70:0/1358138521 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e3c19a020 con 0x7f4e3c1095a0 2024-06-04T05:16:15.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e3a59c700 1 --2- 172.21.15.70:0/1358138521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e3c06bab0 0x7f4e3c19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:16:15.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e3ad9d700 1 --2- 172.21.15.70:0/1358138521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c1095a0 0x7f4e3c1998c0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400b830 tx=0x7f4e34004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:15.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e2b7fe700 1 -- 172.21.15.70:0/1358138521 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e34021070 con 0x7f4e3c1095a0 2024-06-04T05:16:15.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e40e79700 1 -- 172.21.15.70:0/1358138521 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e3c19a2b0 con 0x7f4e3c1095a0 2024-06-04T05:16:15.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e2b7fe700 1 -- 172.21.15.70:0/1358138521 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e34007b80 con 0x7f4e3c1095a0 2024-06-04T05:16:15.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e2b7fe700 1 -- 172.21.15.70:0/1358138521 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e340052a0 con 0x7f4e3c1095a0 2024-06-04T05:16:15.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.673+0000 7f4e40e79700 1 -- 172.21.15.70:0/1358138521 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e3c1a65d0 con 0x7f4e3c1095a0 2024-06-04T05:16:15.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.674+0000 7f4e40e79700 1 -- 172.21.15.70:0/1358138521 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e3c06a230 con 0x7f4e3c1095a0 2024-06-04T05:16:15.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.676+0000 7f4e2b7fe700 1 -- 172.21.15.70:0/1358138521 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f4e34024070 con 0x7f4e3c1095a0 2024-06-04T05:16:15.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.677+0000 7f4e2b7fe700 1 --2- 172.21.15.70:0/1358138521 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e240b29f0 0x7f4e240b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:15.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.677+0000 7f4e3a59c700 1 --2- 172.21.15.70:0/1358138521 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e240b29f0 0x7f4e240b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:15.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.677+0000 7f4e2b7fe700 1 -- 172.21.15.70:0/1358138521 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f4e3400f020 con 0x7f4e3c1095a0 2024-06-04T05:16:15.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.677+0000 7f4e3a59c700 1 --2- 172.21.15.70:0/1358138521 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e240b29f0 0x7f4e240b4eb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c19aa00 tx=0x7f4e2c005d80 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:15.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.681+0000 7f4e2b7fe700 1 -- 172.21.15.70:0/1358138521 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4e3402b020 con 0x7f4e3c1095a0 2024-06-04T05:16:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:15 smithi112 ceph-mon[36453]: pgmap v599: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 250 KiB/s wr, 33 op/s 2024-06-04T05:16:15.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:15 smithi098 ceph-mon[35569]: pgmap v599: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 250 KiB/s wr, 33 op/s 2024-06-04T05:16:15.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.10 MB/sec execute 187 sec latency 18297.273 ms 2024-06-04T05:16:15.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.938+0000 7f4e40e79700 1 -- 172.21.15.70:0/1358138521 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e3c19adf0 con 0x7f4e3c1095a0 2024-06-04T05:16:15.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.939+0000 7f4e2b7fe700 1 -- 172.21.15.70:0/1358138521 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+11929 (secure 0 0 0) 0x7f4e340abab0 con 0x7f4e3c1095a0 2024-06-04T05:16:15.941 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:16:15.941 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":57,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:15:55.006036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T05:16:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.941+0000 7f4e40e79700 1 -- 172.21.15.70:0/1358138521 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e240b29f0 msgr2=0x7f4e240b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.941+0000 7f4e40e79700 1 --2- 172.21.15.70:0/1358138521 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e240b29f0 0x7f4e240b4eb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c19aa00 tx=0x7f4e2c005d80 comp rx=0 tx=0).stop 2024-06-04T05:16:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.941+0000 7f4e40e79700 1 -- 172.21.15.70:0/1358138521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c1095a0 msgr2=0x7f4e3c1998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.941+0000 7f4e40e79700 1 --2- 172.21.15.70:0/1358138521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c1095a0 0x7f4e3c1998c0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400b830 tx=0x7f4e34004d00 comp rx=0 tx=0).stop 2024-06-04T05:16:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.941+0000 7f4e40e79700 1 -- 172.21.15.70:0/1358138521 shutdown_connections 2024-06-04T05:16:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.941+0000 7f4e40e79700 1 --2- 172.21.15.70:0/1358138521 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f4e240b29f0 0x7f4e240b4eb0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.941+0000 7f4e40e79700 1 --2- 172.21.15.70:0/1358138521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e3c1095a0 0x7f4e3c1998c0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.941+0000 7f4e40e79700 1 --2- 172.21.15.70:0/1358138521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e3c06bab0 0x7f4e3c19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.941+0000 7f4e40e79700 1 --2- 172.21.15.70:0/1358138521 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e3c06c470 0x7f4e3c19fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.941+0000 7f4e40e79700 1 -- 172.21.15.70:0/1358138521 >> 172.21.15.70:0/1358138521 conn(0x7f4e3c0793b0 msgr2=0x7f4e3c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.941+0000 7f4e40e79700 1 -- 172.21.15.70:0/1358138521 shutdown_connections 2024-06-04T05:16:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:15.941+0000 7f4e40e79700 1 -- 172.21.15.70:0/1358138521 wait complete. 2024-06-04T05:16:15.945 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 57 2024-06-04T05:16:16.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:16 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1358138521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:16:16.680 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id 724664c3 --release --await' 2024-06-04T05:16:16.681 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id 724664c3 --release --await 2024-06-04T05:16:16.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:16 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1358138521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:16:16.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:16 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1358138521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:16:16.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.07 MB/sec execute 188 sec latency 19297.314 ms 2024-06-04T05:16:16.921 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:16:17.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:17 smithi070 ceph-mon[32631]: pgmap v600: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 44 KiB/s wr, 27 op/s 2024-06-04T05:16:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:17 smithi112 ceph-mon[36453]: pgmap v600: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 44 KiB/s wr, 27 op/s 2024-06-04T05:16:17.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:17 smithi098 ceph-mon[35569]: pgmap v600: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 44 KiB/s wr, 27 op/s 2024-06-04T05:16:17.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41080 7.03 MB/sec execute 189 sec latency 20297.358 ms 2024-06-04T05:16:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.588+0000 7fd2b8197700 1 -- 172.21.15.70:0/161878806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2b0102e10 msgr2=0x7fd2b010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.588+0000 7fd2b8197700 1 --2- 172.21.15.70:0/161878806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2b0102e10 0x7fd2b010f2f0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fd2a800b0a0 tx=0x7fd2a800b3b0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.588+0000 7fd2b8197700 1 -- 172.21.15.70:0/161878806 shutdown_connections 2024-06-04T05:16:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.588+0000 7fd2b8197700 1 --2- 172.21.15.70:0/161878806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2b0102e10 0x7fd2b010f2f0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.588+0000 7fd2b8197700 1 --2- 172.21.15.70:0/161878806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2b01084b0 0x7fd2b0108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.588+0000 7fd2b8197700 1 --2- 172.21.15.70:0/161878806 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2b0102470 0x7fd2b01028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.588+0000 7fd2b8197700 1 -- 172.21.15.70:0/161878806 >> 172.21.15.70:0/161878806 conn(0x7fd2b00793e0 msgr2=0x7fd2b00797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.588+0000 7fd2b8197700 1 -- 172.21.15.70:0/161878806 shutdown_connections 2024-06-04T05:16:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.588+0000 7fd2b8197700 1 -- 172.21.15.70:0/161878806 wait complete. 2024-06-04T05:16:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.589+0000 7fd2b8197700 1 Processor -- start 2024-06-04T05:16:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.589+0000 7fd2b8197700 1 -- start start 2024-06-04T05:16:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.589+0000 7fd2b8197700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2b0102470 0x7fd2b019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.589+0000 7fd2b8197700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2b0102e10 0x7fd2b019ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.589+0000 7fd2b8197700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2b01084b0 0x7fd2b01a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.589+0000 7fd2b8197700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b0112510 con 0x7fd2b0102470 2024-06-04T05:16:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.589+0000 7fd2b8197700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b0112390 con 0x7fd2b0102e10 2024-06-04T05:16:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.589+0000 7fd2b8197700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b0112690 con 0x7fd2b01084b0 2024-06-04T05:16:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.589+0000 7fd2b5f33700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2b0102470 0x7fd2b019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.589+0000 7fd2b5f33700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2b0102470 0x7fd2b019a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37824/0 (socket says 172.21.15.70:37824) 2024-06-04T05:16:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.589+0000 7fd2b5f33700 1 -- 172.21.15.70:0/2019768741 learned_addr learned my addr 172.21.15.70:0/2019768741 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.589+0000 7fd2b5732700 1 --2- 172.21.15.70:0/2019768741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2b0102e10 0x7fd2b019ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.590+0000 7fd2b6734700 1 --2- 172.21.15.70:0/2019768741 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2b01084b0 0x7fd2b01a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.590+0000 7fd2b5f33700 1 -- 172.21.15.70:0/2019768741 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2b01084b0 msgr2=0x7fd2b01a5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.590+0000 7fd2b5f33700 1 --2- 172.21.15.70:0/2019768741 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2b01084b0 0x7fd2b01a5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.590+0000 7fd2b5f33700 1 -- 172.21.15.70:0/2019768741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2b0102e10 msgr2=0x7fd2b019ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.590+0000 7fd2b5f33700 1 --2- 172.21.15.70:0/2019768741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2b0102e10 0x7fd2b019ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.590+0000 7fd2b5f33700 1 -- 172.21.15.70:0/2019768741 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b01a5ed0 con 0x7fd2b0102470 2024-06-04T05:16:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.590+0000 7fd2b5f33700 1 --2- 172.21.15.70:0/2019768741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2b0102470 0x7fd2b019a910 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00c9b0 tx=0x7fd2ac00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.590+0000 7fd2a6ffd700 1 -- 172.21.15.70:0/2019768741 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac007b00 con 0x7fd2b0102470 2024-06-04T05:16:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.590+0000 7fd2a6ffd700 1 -- 172.21.15.70:0/2019768741 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd2ac00ce50 con 0x7fd2b0102470 2024-06-04T05:16:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.591+0000 7fd2a6ffd700 1 -- 172.21.15.70:0/2019768741 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac018660 con 0x7fd2b0102470 2024-06-04T05:16:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.591+0000 7fd2b8197700 1 -- 172.21.15.70:0/2019768741 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2b01a60a0 con 0x7fd2b0102470 2024-06-04T05:16:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.591+0000 7fd2b8197700 1 -- 172.21.15.70:0/2019768741 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2b01a65b0 con 0x7fd2b0102470 2024-06-04T05:16:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.593+0000 7fd2a6ffd700 1 -- 172.21.15.70:0/2019768741 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fd2ac00f450 con 0x7fd2b0102470 2024-06-04T05:16:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.594+0000 7fd2a6ffd700 1 --2- 172.21.15.70:0/2019768741 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd29c0b29f0 0x7fd29c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.594+0000 7fd2b5732700 1 --2- 172.21.15.70:0/2019768741 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd29c0b29f0 0x7fd29c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.594+0000 7fd2a6ffd700 1 -- 172.21.15.70:0/2019768741 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7fd2ac0e25c0 con 0x7fd2b0102470 2024-06-04T05:16:18.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.595+0000 7fd2c2a2cb80 1 Processor -- start 2024-06-04T05:16:18.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.595+0000 7fd2b5732700 1 --2- 172.21.15.70:0/2019768741 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd29c0b29f0 0x7fd29c0b4eb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd2b00799e0 tx=0x7fd2a0009190 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:18.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.595+0000 7fd2c2a2cb80 1 -- start start 2024-06-04T05:16:18.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.595+0000 7fd2c2a2cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a47720 0x55cc07a47b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.595+0000 7fd2c2a2cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a42bf0 0x55cc07a43070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.595+0000 7fd2c2a2cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a435b0 0x55cc07a4f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.595+0000 7fd2c2a2cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cc079a0e70 con 0x55cc07a435b0 2024-06-04T05:16:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.595+0000 7fd2c2a2cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cc07990a10 con 0x55cc07a42bf0 2024-06-04T05:16:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.595+0000 7fd2c2a2cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cc07994a50 con 0x55cc07a47720 2024-06-04T05:16:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.595+0000 7fd2b6734700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a435b0 0x55cc07a4f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.595+0000 7fd2b6734700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a435b0 0x55cc07a4f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37832/0 (socket says 172.21.15.70:37832) 2024-06-04T05:16:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.595+0000 7fd2b6734700 1 -- 172.21.15.70:0/3702231344 learned_addr learned my addr 172.21.15.70:0/3702231344 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.595+0000 7fd2b5732700 1 --2- 172.21.15.70:0/3702231344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a42bf0 0x55cc07a43070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.595+0000 7fd2b5f33700 1 --2- 172.21.15.70:0/3702231344 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a47720 0x55cc07a47b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.596+0000 7fd2b6734700 1 -- 172.21.15.70:0/3702231344 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a47720 msgr2=0x55cc07a47b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.596+0000 7fd2b6734700 1 --2- 172.21.15.70:0/3702231344 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a47720 0x55cc07a47b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.596+0000 7fd2b6734700 1 -- 172.21.15.70:0/3702231344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a42bf0 msgr2=0x55cc07a43070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.596+0000 7fd2b6734700 1 --2- 172.21.15.70:0/3702231344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a42bf0 0x55cc07a43070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.596+0000 7fd2b6734700 1 -- 172.21.15.70:0/3702231344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc0793a0b0 con 0x55cc07a435b0 2024-06-04T05:16:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.596+0000 7fd2b6734700 1 --2- 172.21.15.70:0/3702231344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a435b0 0x55cc07a4f620 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fd2a80093c0 tx=0x7fd2a80093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0f21882489f103 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:18.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.596+0000 7fd29bfff700 1 -- 172.21.15.70:0/3702231344 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8020070 con 0x55cc07a435b0 2024-06-04T05:16:18.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.596+0000 7fd29bfff700 1 -- 172.21.15.70:0/3702231344 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd2a80074e0 con 0x55cc07a435b0 2024-06-04T05:16:18.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.596+0000 7fd29bfff700 1 -- 172.21.15.70:0/3702231344 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8005260 con 0x55cc07a435b0 2024-06-04T05:16:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.597+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/3702231344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a435b0 msgr2=0x55cc07a4f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.597+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/3702231344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a435b0 0x55cc07a4f620 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fd2a80093c0 tx=0x7fd2a80093f0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.597+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/3702231344 shutdown_connections 2024-06-04T05:16:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.597+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/3702231344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a435b0 0x55cc07a4f620 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.597+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/3702231344 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a47720 0x55cc07a47b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.597+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/3702231344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a42bf0 0x55cc07a43070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.597+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/3702231344 >> 172.21.15.70:0/3702231344 conn(0x55cc0796e340 msgr2=0x55cc07a3d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.597+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/3702231344 shutdown_connections 2024-06-04T05:16:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.597+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/3702231344 wait complete. 2024-06-04T05:16:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.597+0000 7fd2c2a2cb80 1 -- start start 2024-06-04T05:16:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.597+0000 7fd2c2a2cb80 1 Processor -- start 2024-06-04T05:16:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2c2a2cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a42bf0 0x55cc07a4f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2c2a2cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a435b0 0x55cc07a4c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2c2a2cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a47720 0x55cc07a4cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2c2a2cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cc078970a0 con 0x55cc07a42bf0 2024-06-04T05:16:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2c2a2cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cc078f8ea0 con 0x55cc07a435b0 2024-06-04T05:16:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2c2a2cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cc07a4fc20 con 0x55cc07a47720 2024-06-04T05:16:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2c2a2cb80 10 client.0 _ll_register_callbacks cb 0x55cc0792ba50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2b5f33700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a42bf0 0x55cc07a4f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2b5f33700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a42bf0 0x55cc07a4f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37842/0 (socket says 172.21.15.70:37842) 2024-06-04T05:16:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2b5f33700 1 -- 172.21.15.70:0/2699075419 learned_addr learned my addr 172.21.15.70:0/2699075419 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2b5732700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a435b0 0x55cc07a4c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2b6734700 1 --2- 172.21.15.70:0/2699075419 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a47720 0x55cc07a4cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2b5f33700 1 -- 172.21.15.70:0/2699075419 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a47720 msgr2=0x55cc07a4cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2b5f33700 1 --2- 172.21.15.70:0/2699075419 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a47720 0x55cc07a4cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2b5f33700 1 -- 172.21.15.70:0/2699075419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a435b0 msgr2=0x55cc07a4c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2b5f33700 1 --2- 172.21.15.70:0/2699075419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a435b0 0x55cc07a4c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2b5f33700 1 -- 172.21.15.70:0/2699075419 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc078e09d0 con 0x55cc07a42bf0 2024-06-04T05:16:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2b5732700 1 --2- 172.21.15.70:0/2699075419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a435b0 0x55cc07a4c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:16:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2b5f33700 1 --2- 172.21.15.70:0/2699075419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a42bf0 0x55cc07a4f2a0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00afd0 tx=0x7fd2ac0a5db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a3a885348aee1aa server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2997fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2997fa700 1 -- 172.21.15.70:0/2699075419 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac00de20 con 0x55cc07a42bf0 2024-06-04T05:16:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.598+0000 7fd2997fa700 1 -- 172.21.15.70:0/2699075419 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7fd2ac007ca0 con 0x55cc07a42bf0 2024-06-04T05:16:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.599+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/2699075419 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2a80074e0 con 0x55cc07a42bf0 2024-06-04T05:16:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.599+0000 7fd2997fa700 1 -- 172.21.15.70:0/2699075419 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fd2ac0a9f70 con 0x55cc07a42bf0 2024-06-04T05:16:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.599+0000 7fd2c2a2cb80 10 client.20379 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.599+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/2699075419 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55cc0794a860 con 0x55cc07a42bf0 2024-06-04T05:16:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.599+0000 7fd2997fa700 1 -- 172.21.15.70:0/2699075419 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd2a80074e0 con 0x55cc07a42bf0 2024-06-04T05:16:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.599+0000 7fd2997fa700 1 -- 172.21.15.70:0/2699075419 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fd2ac0a9ad0 con 0x55cc07a42bf0 2024-06-04T05:16:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.599+0000 7fd2c2a2cb80 10 client.20379 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.599+0000 7fd2c2a2cb80 10 client.20379 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:16:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.599+0000 7fd2c2a2cb80 20 client.20379 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.599+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/2699075419 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cc07a60bf0 0x55cc07a62fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.599+0000 7fd2c2a2cb80 4 client.20379 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:16:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.599+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/2699075419 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc079a2dc0 con 0x55cc07a60bf0 2024-06-04T05:16:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.599+0000 7fd2b5732700 1 --2- 172.21.15.70:0/2699075419 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cc07a60bf0 0x55cc07a62fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.600+0000 7fd2997fa700 10 client.20379 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.600+0000 7fd2b5732700 1 --2- 172.21.15.70:0/2699075419 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cc07a60bf0 0x55cc07a62fe0 crc :-1 s=READY pgs=409 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99f9c06dcfac5348 server_cookie=87b88788a8b45db7 in_seq=0 out_seq=0 2024-06-04T05:16:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.600+0000 7fd2997fa700 1 -- 172.21.15.70:0/2699075419 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cc079a2dc0 con 0x55cc07a60bf0 2024-06-04T05:16:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.600+0000 7fd2997fa700 10 client.20379 handle_command_reply: tid=0 2024-06-04T05:16:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.600+0000 7fd2c2a2cb80 1 client.20379 shutdown 2024-06-04T05:16:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.601+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/2699075419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a42bf0 msgr2=0x55cc07a4f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.601+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/2699075419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a42bf0 0x55cc07a4f2a0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00afd0 tx=0x7fd2ac0a5db0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.601+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/2699075419 shutdown_connections 2024-06-04T05:16:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.601+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/2699075419 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a47720 0x55cc07a4cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.601+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/2699075419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a42bf0 0x55cc07a4f2a0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.601+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/2699075419 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cc07a60bf0 0x55cc07a62fe0 crc :-1 s=READY pgs=409 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.601+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/2699075419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a435b0 0x55cc07a4c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.601+0000 7fd2997fa700 0 client.20379 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.601+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/2699075419 >> 172.21.15.70:0/2699075419 conn(0x55cc0796e340 msgr2=0x55cc07a41be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.601+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/2699075419 shutdown_connections 2024-06-04T05:16:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.601+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/2699075419 wait complete. 2024-06-04T05:16:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.601+0000 7fd2c2a2cb80 20 client.20379 trim_cache size 0 max 16384 2024-06-04T05:16:18.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.620+0000 7fd2c2a2cb80 1 Processor -- start 2024-06-04T05:16:18.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.620+0000 7fd2c2a2cb80 1 -- start start 2024-06-04T05:16:18.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd2c2a2cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a42bf0 0x55cc07a37e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd2c2a2cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a435b0 0x55cc07a4e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd2c2a2cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a47720 0x55cc07a688b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd2c2a2cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b01abb70 con 0x55cc07a47720 2024-06-04T05:16:18.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd2c2a2cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b01ad6b0 con 0x55cc07a42bf0 2024-06-04T05:16:18.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd2c2a2cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b01ac240 con 0x55cc07a435b0 2024-06-04T05:16:18.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd2b6734700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a47720 0x55cc07a688b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd2b6734700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a47720 0x55cc07a688b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37846/0 (socket says 172.21.15.70:37846) 2024-06-04T05:16:18.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd2b6734700 1 -- 172.21.15.70:0/50927148 learned_addr learned my addr 172.21.15.70:0/50927148 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:18.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd2b5f33700 1 --2- 172.21.15.70:0/50927148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a42bf0 0x55cc07a37e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd2b5732700 1 --2- 172.21.15.70:0/50927148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a435b0 0x55cc07a4e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd2b6734700 1 -- 172.21.15.70:0/50927148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a435b0 msgr2=0x55cc07a4e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:18.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd2b6734700 1 --2- 172.21.15.70:0/50927148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a435b0 0x55cc07a4e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd2b6734700 1 -- 172.21.15.70:0/50927148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a42bf0 msgr2=0x55cc07a37e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:18.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd2b6734700 1 --2- 172.21.15.70:0/50927148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a42bf0 0x55cc07a37e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd2b6734700 1 -- 172.21.15.70:0/50927148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc07a63350 con 0x55cc07a47720 2024-06-04T05:16:18.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd2b6734700 1 --2- 172.21.15.70:0/50927148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a47720 0x55cc07a688b0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fd2a80093c0 tx=0x7fd2a8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4fd5e9718a7f733 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:18.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.621+0000 7fd29b7fe700 1 -- 172.21.15.70:0/50927148 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a802c070 con 0x55cc07a47720 2024-06-04T05:16:18.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.622+0000 7fd29b7fe700 1 -- 172.21.15.70:0/50927148 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd2a8003c00 con 0x55cc07a47720 2024-06-04T05:16:18.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.622+0000 7fd29b7fe700 1 -- 172.21.15.70:0/50927148 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8020040 con 0x55cc07a47720 2024-06-04T05:16:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.622+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/50927148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a47720 msgr2=0x55cc07a688b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.622+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/50927148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a47720 0x55cc07a688b0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fd2a80093c0 tx=0x7fd2a8009780 comp rx=0 tx=0).stop 2024-06-04T05:16:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.622+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/50927148 shutdown_connections 2024-06-04T05:16:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.622+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/50927148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a47720 0x55cc07a688b0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.622+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/50927148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a42bf0 0x55cc07a37e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.622+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/50927148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a435b0 0x55cc07a4e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.622+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/50927148 >> 172.21.15.70:0/50927148 conn(0x55cc0796e340 msgr2=0x55cc07a4ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.622+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/50927148 shutdown_connections 2024-06-04T05:16:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.622+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/50927148 wait complete. 2024-06-04T05:16:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.623+0000 7fd2c2a2cb80 1 -- start start 2024-06-04T05:16:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.623+0000 7fd2c2a2cb80 1 Processor -- start 2024-06-04T05:16:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.623+0000 7fd2c2a2cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a42bf0 0x55cc07a53370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.623+0000 7fd2c2a2cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a435b0 0x55cc07a538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.623+0000 7fd2c2a2cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a47720 0x55cc07aca790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.623+0000 7fd2c2a2cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b01acfe0 con 0x55cc07a435b0 2024-06-04T05:16:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.623+0000 7fd2c2a2cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b01ac910 con 0x55cc07a47720 2024-06-04T05:16:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.623+0000 7fd2c2a2cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b01a9b30 con 0x55cc07a42bf0 2024-06-04T05:16:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.623+0000 7fd2c2a2cb80 10 client.0 _ll_register_callbacks cb 0x55cc0792ba50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.623+0000 7fd2b5732700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a435b0 0x55cc07a538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.623+0000 7fd2b5732700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a435b0 0x55cc07a538b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37862/0 (socket says 172.21.15.70:37862) 2024-06-04T05:16:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.623+0000 7fd2b5732700 1 -- 172.21.15.70:0/3257648779 learned_addr learned my addr 172.21.15.70:0/3257648779 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.623+0000 7fd2b5f33700 1 --2- 172.21.15.70:0/3257648779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a42bf0 0x55cc07a53370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.623+0000 7fd2b6734700 1 --2- 172.21.15.70:0/3257648779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a47720 0x55cc07aca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.624+0000 7fd2b5732700 1 -- 172.21.15.70:0/3257648779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a42bf0 msgr2=0x55cc07a53370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.624+0000 7fd2b5732700 1 --2- 172.21.15.70:0/3257648779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a42bf0 0x55cc07a53370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.624+0000 7fd2b5732700 1 -- 172.21.15.70:0/3257648779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a47720 msgr2=0x55cc07aca790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.624+0000 7fd2b5732700 1 --2- 172.21.15.70:0/3257648779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a47720 0x55cc07aca790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.624+0000 7fd2b5732700 1 -- 172.21.15.70:0/3257648779 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc07acad50 con 0x55cc07a435b0 2024-06-04T05:16:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.624+0000 7fd2b6734700 1 --2- 172.21.15.70:0/3257648779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a47720 0x55cc07aca790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:16:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.624+0000 7fd2b5732700 1 --2- 172.21.15.70:0/3257648779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a435b0 0x55cc07a538b0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fd2a00097f0 tx=0x7fd2a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ee0a7ae62f021 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.624+0000 7fd2997fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.624+0000 7fd2997fa700 1 -- 172.21.15.70:0/3257648779 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a001b040 con 0x55cc07a435b0 2024-06-04T05:16:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.624+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/3257648779 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2a8003c00 con 0x55cc07a435b0 2024-06-04T05:16:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.624+0000 7fd2997fa700 1 -- 172.21.15.70:0/3257648779 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7fd2a0029250 con 0x55cc07a435b0 2024-06-04T05:16:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.624+0000 7fd2997fa700 1 -- 172.21.15.70:0/3257648779 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fd2a0005a50 con 0x55cc07a435b0 2024-06-04T05:16:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.624+0000 7fd2c2a2cb80 10 client.20385 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.624+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/3257648779 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55cc07acc150 con 0x55cc07a435b0 2024-06-04T05:16:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.624+0000 7fd2997fa700 1 -- 172.21.15.70:0/3257648779 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd2a8003c00 con 0x55cc07a435b0 2024-06-04T05:16:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.624+0000 7fd2997fa700 1 -- 172.21.15.70:0/3257648779 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fd2a0038020 con 0x55cc07a435b0 2024-06-04T05:16:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.625+0000 7fd2c2a2cb80 10 client.20385 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.625+0000 7fd2c2a2cb80 10 client.20385 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:16:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.625+0000 7fd2c2a2cb80 20 client.20385 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.625+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/3257648779 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cc07acc870 0x55cc07acec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.625+0000 7fd2c2a2cb80 4 client.20385 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "724664c3", "release": true, "await": true}] 2024-06-04T05:16:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.625+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/3257648779 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "724664c3", "release": true, "await": true}) -- 0x55cc0799c5a0 con 0x55cc07acc870 2024-06-04T05:16:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.625+0000 7fd2b5f33700 1 --2- 172.21.15.70:0/3257648779 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cc07acc870 0x55cc07acec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.625+0000 7fd2997fa700 10 client.20385 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:18.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.625+0000 7fd2b5f33700 1 --2- 172.21.15.70:0/3257648779 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cc07acc870 0x55cc07acec90 crc :-1 s=READY pgs=410 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f1ca5d1c7241eec server_cookie=13009124c87038d7 in_seq=0 out_seq=0 2024-06-04T05:16:18.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.653+0000 7fd2997fa700 1 -- 172.21.15.70:0/3257648779 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55cc0799c5a0 con 0x55cc07acc870 2024-06-04T05:16:18.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.653+0000 7fd2997fa700 10 client.20385 handle_command_reply: tid=0 2024-06-04T05:16:18.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.653+0000 7fd2c2a2cb80 1 client.20385 shutdown 2024-06-04T05:16:18.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.653+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/3257648779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a435b0 msgr2=0x55cc07a538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:18.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.653+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/3257648779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a435b0 0x55cc07a538b0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fd2a00097f0 tx=0x7fd2a0004750 comp rx=0 tx=0).stop 2024-06-04T05:16:18.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.653+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/3257648779 shutdown_connections 2024-06-04T05:16:18.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.653+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/3257648779 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cc07acc870 0x55cc07acec90 crc :-1 s=READY pgs=410 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.653+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/3257648779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc07a47720 0x55cc07aca790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.653+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/3257648779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc07a42bf0 0x55cc07a53370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.653+0000 7fd2997fa700 0 client.20385 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:18.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.653+0000 7fd2c2a2cb80 1 --2- 172.21.15.70:0/3257648779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc07a435b0 0x55cc07a538b0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.653+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/3257648779 >> 172.21.15.70:0/3257648779 conn(0x55cc0796e340 msgr2=0x55cc07a4e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:18.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.653+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/3257648779 shutdown_connections 2024-06-04T05:16:18.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.653+0000 7fd2c2a2cb80 1 -- 172.21.15.70:0/3257648779 wait complete. 2024-06-04T05:16:18.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.654+0000 7fd2c2a2cb80 20 client.20385 trim_cache size 0 max 16384 2024-06-04T05:16:18.657 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:16:18.657 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 57, 2024-06-04T05:16:18.657 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:16:18.657 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 60, 2024-06-04T05:16:18.657 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:16:18.657 INFO:teuthology.orchestra.run.smithi070.stdout: "724664c3": { 2024-06-04T05:16:18.657 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 60, 2024-06-04T05:16:18.657 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:16:18.657 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:16:18.657 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:16:18.657 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:16:18.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:16:18.657 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:16:18.658 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 146.8, 2024-06-04T05:16:18.658 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:16:18.658 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:16:18.658 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:16:18.658 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:16:18.658 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:16:18.658 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:16:18.658 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:16:18.658 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:16:18.658 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:16:18.658 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:16:18.658 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:16:18.658 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:16:18.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.655+0000 7fd2b8197700 1 -- 172.21.15.70:0/2019768741 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd29c0b29f0 msgr2=0x7fd29c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:18.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.655+0000 7fd2b8197700 1 --2- 172.21.15.70:0/2019768741 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd29c0b29f0 0x7fd29c0b4eb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd2b00799e0 tx=0x7fd2a0009190 comp rx=0 tx=0).stop 2024-06-04T05:16:18.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.655+0000 7fd2b8197700 1 -- 172.21.15.70:0/2019768741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2b0102470 msgr2=0x7fd2b019a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:18.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.655+0000 7fd2b8197700 1 --2- 172.21.15.70:0/2019768741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2b0102470 0x7fd2b019a910 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00c9b0 tx=0x7fd2ac00ccc0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.656+0000 7fd2b8197700 1 -- 172.21.15.70:0/2019768741 shutdown_connections 2024-06-04T05:16:18.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.656+0000 7fd2b8197700 1 --2- 172.21.15.70:0/2019768741 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd29c0b29f0 0x7fd29c0b4eb0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.656+0000 7fd2b8197700 1 --2- 172.21.15.70:0/2019768741 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2b01084b0 0x7fd2b01a5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.656+0000 7fd2b8197700 1 --2- 172.21.15.70:0/2019768741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2b0102470 0x7fd2b019a910 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.656+0000 7fd2b8197700 1 --2- 172.21.15.70:0/2019768741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2b0102e10 0x7fd2b019ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:18.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.656+0000 7fd2b8197700 1 -- 172.21.15.70:0/2019768741 >> 172.21.15.70:0/2019768741 conn(0x7fd2b00793e0 msgr2=0x7fd2b0100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:18.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.656+0000 7fd2b8197700 1 -- 172.21.15.70:0/2019768741 shutdown_connections 2024-06-04T05:16:18.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:18.656+0000 7fd2b8197700 1 -- 172.21.15.70:0/2019768741 wait complete. 2024-06-04T05:16:18.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41108 6.99 MB/sec execute 190 sec latency 21151.315 ms 2024-06-04T05:16:19.384 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '724664c3', total seconds elapsed: 33.1 2024-06-04T05:16:19.384 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 26.85 2024-06-04T05:16:19.384 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:16:19.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:19 smithi070 ceph-mon[32631]: pgmap v601: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 43 KiB/s wr, 26 op/s 2024-06-04T05:16:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:19 smithi112 ceph-mon[36453]: pgmap v601: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 43 KiB/s wr, 26 op/s 2024-06-04T05:16:19.730 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:16:19.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:19 smithi098 ceph-mon[35569]: pgmap v601: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 43 KiB/s wr, 26 op/s 2024-06-04T05:16:19.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41274 6.98 MB/sec execute 191 sec latency 49.207 ms 2024-06-04T05:16:20.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41448 7.01 MB/sec execute 192 sec latency 130.614 ms 2024-06-04T05:16:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.278+0000 7f15659da640 1 -- 172.21.15.70:0/3424443186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1560101950 msgr2=0x7f1560101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.278+0000 7f15659da640 1 --2- 172.21.15.70:0/3424443186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1560101950 0x7f1560101dd0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f1548009930 tx=0x7f154802a420 comp rx=0 tx=0).stop 2024-06-04T05:16:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.278+0000 7f15659da640 1 -- 172.21.15.70:0/3424443186 shutdown_connections 2024-06-04T05:16:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.278+0000 7f15659da640 1 --2- 172.21.15.70:0/3424443186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1560102310 0x7f156010e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.278+0000 7f15659da640 1 --2- 172.21.15.70:0/3424443186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1560101950 0x7f1560101dd0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.278+0000 7f15659da640 1 --2- 172.21.15.70:0/3424443186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1560107960 0x7f1560107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.278+0000 7f15659da640 1 -- 172.21.15.70:0/3424443186 >> 172.21.15.70:0/3424443186 conn(0x7f1560078aa0 msgr2=0x7f1560078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.278+0000 7f15659da640 1 -- 172.21.15.70:0/3424443186 shutdown_connections 2024-06-04T05:16:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.278+0000 7f15659da640 1 -- 172.21.15.70:0/3424443186 wait complete. 2024-06-04T05:16:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.279+0000 7f15659da640 1 Processor -- start 2024-06-04T05:16:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.279+0000 7f15659da640 1 -- start start 2024-06-04T05:16:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.280+0000 7f15659da640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1560101950 0x7f1560116f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.280+0000 7f15659da640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1560102310 0x7f15601174a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.280+0000 7f15659da640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1560107960 0x7f1560112110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.280+0000 7f15659da640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1560110a10 con 0x7f1560102310 2024-06-04T05:16:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.280+0000 7f15659da640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1560110890 con 0x7f1560101950 2024-06-04T05:16:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.280+0000 7f15659da640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1560110b90 con 0x7f1560107960 2024-06-04T05:16:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.280+0000 7f155effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1560101950 0x7f1560116f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.280+0000 7f155f7fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1560107960 0x7f1560112110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.280+0000 7f155effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1560101950 0x7f1560116f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:35690/0 (socket says 172.21.15.70:35690) 2024-06-04T05:16:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.280+0000 7f155effd640 1 -- 172.21.15.70:0/1828095974 learned_addr learned my addr 172.21.15.70:0/1828095974 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.280+0000 7f155e7fc640 1 --2- 172.21.15.70:0/1828095974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1560102310 0x7f15601174a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.280+0000 7f155f7fe640 1 -- 172.21.15.70:0/1828095974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1560101950 msgr2=0x7f1560116f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.280+0000 7f155f7fe640 1 --2- 172.21.15.70:0/1828095974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1560101950 0x7f1560116f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.280+0000 7f155f7fe640 1 -- 172.21.15.70:0/1828095974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1560102310 msgr2=0x7f15601174a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.280+0000 7f155f7fe640 1 --2- 172.21.15.70:0/1828095974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1560102310 0x7f15601174a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.280+0000 7f155f7fe640 1 -- 172.21.15.70:0/1828095974 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1560112650 con 0x7f1560107960 2024-06-04T05:16:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.280+0000 7f155e7fc640 1 --2- 172.21.15.70:0/1828095974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1560102310 0x7f15601174a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:16:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.281+0000 7f155f7fe640 1 --2- 172.21.15.70:0/1828095974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1560107960 0x7f1560112110 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1558009fd0 tx=0x7f155800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.281+0000 7f15649d8640 1 -- 172.21.15.70:0/1828095974 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155800ce20 con 0x7f1560107960 2024-06-04T05:16:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.281+0000 7f15659da640 1 -- 172.21.15.70:0/1828095974 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1560112940 con 0x7f1560107960 2024-06-04T05:16:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.281+0000 7f15649d8640 1 -- 172.21.15.70:0/1828095974 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1558005020 con 0x7f1560107960 2024-06-04T05:16:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.281+0000 7f15649d8640 1 -- 172.21.15.70:0/1828095974 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1558010690 con 0x7f1560107960 2024-06-04T05:16:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.281+0000 7f15659da640 1 -- 172.21.15.70:0/1828095974 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1560112eb0 con 0x7f1560107960 2024-06-04T05:16:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.284+0000 7f15649d8640 1 -- 172.21.15.70:0/1828095974 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f1558004a40 con 0x7f1560107960 2024-06-04T05:16:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.284+0000 7f15649d8640 1 --2- 172.21.15.70:0/1828095974 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f154c0b2a60 0x7f154c0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.285+0000 7f155effd640 1 --2- 172.21.15.70:0/1828095974 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f154c0b2a60 0x7f154c0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.285+0000 7f15649d8640 1 -- 172.21.15.70:0/1828095974 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f15580e2d10 con 0x7f1560107960 2024-06-04T05:16:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.285+0000 7f155effd640 1 --2- 172.21.15.70:0/1828095974 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f154c0b2a60 0x7f154c0b4f20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1550009c10 tx=0x7f1550005f90 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:21.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.333+0000 7f15687ce740 1 Processor -- start 2024-06-04T05:16:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.333+0000 7f15687ce740 1 -- start start 2024-06-04T05:16:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f15687ce740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7d5820 0x5581cb7d5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f15687ce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d6140 0x5581cb7e04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f15687ce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7e2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f15687ce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5581cb747c30 con 0x5581cb7e0a00 2024-06-04T05:16:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f15687ce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5581cb7377d0 con 0x5581cb7d6140 2024-06-04T05:16:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f15687ce740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5581cb73b810 con 0x5581cb7d5820 2024-06-04T05:16:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f155f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7e2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f155f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7e2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37886/0 (socket says 172.21.15.70:37886) 2024-06-04T05:16:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f155f7fe640 1 -- 172.21.15.70:0/2345861198 learned_addr learned my addr 172.21.15.70:0/2345861198 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f155e7fc640 1 --2- 172.21.15.70:0/2345861198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d6140 0x5581cb7e04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f155effd640 1 --2- 172.21.15.70:0/2345861198 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7d5820 0x5581cb7d5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f155f7fe640 1 -- 172.21.15.70:0/2345861198 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7d5820 msgr2=0x5581cb7d5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f155f7fe640 1 --2- 172.21.15.70:0/2345861198 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7d5820 0x5581cb7d5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f155f7fe640 1 -- 172.21.15.70:0/2345861198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d6140 msgr2=0x5581cb7e04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f155f7fe640 1 --2- 172.21.15.70:0/2345861198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d6140 0x5581cb7e04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f155f7fe640 1 -- 172.21.15.70:0/2345861198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581cb44c210 con 0x5581cb7e0a00 2024-06-04T05:16:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f155f7fe640 1 --2- 172.21.15.70:0/2345861198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7e2df0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f15580a5fe0 tx=0x7f15580aa720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bfa56279075790b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f1555ffb640 1 -- 172.21.15.70:0/2345861198 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15580b2dc0 con 0x5581cb7e0a00 2024-06-04T05:16:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f1555ffb640 1 -- 172.21.15.70:0/2345861198 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f15580054d0 con 0x5581cb7e0a00 2024-06-04T05:16:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.334+0000 7f1555ffb640 1 -- 172.21.15.70:0/2345861198 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15580b2930 con 0x5581cb7e0a00 2024-06-04T05:16:21.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.335+0000 7f15687ce740 1 -- 172.21.15.70:0/2345861198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 msgr2=0x5581cb7e2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:21.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.335+0000 7f15687ce740 1 --2- 172.21.15.70:0/2345861198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7e2df0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f15580a5fe0 tx=0x7f15580aa720 comp rx=0 tx=0).stop 2024-06-04T05:16:21.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.335+0000 7f15687ce740 1 -- 172.21.15.70:0/2345861198 shutdown_connections 2024-06-04T05:16:21.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.335+0000 7f15687ce740 1 --2- 172.21.15.70:0/2345861198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d6140 0x5581cb7e04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.335+0000 7f15687ce740 1 --2- 172.21.15.70:0/2345861198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7e2df0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.335+0000 7f15687ce740 1 --2- 172.21.15.70:0/2345861198 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7d5820 0x5581cb7d5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.335+0000 7f15687ce740 1 -- 172.21.15.70:0/2345861198 >> 172.21.15.70:0/2345861198 conn(0x5581cb7d1d30 msgr2=0x5581cb7d2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:21.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.335+0000 7f15687ce740 1 -- 172.21.15.70:0/2345861198 shutdown_connections 2024-06-04T05:16:21.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.335+0000 7f15687ce740 1 -- 172.21.15.70:0/2345861198 wait complete. 2024-06-04T05:16:21.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.336+0000 7f15687ce740 1 -- start start 2024-06-04T05:16:21.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.336+0000 7f15687ce740 1 Processor -- start 2024-06-04T05:16:21.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.336+0000 7f15687ce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d5820 0x5581cb7dda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.336+0000 7f15687ce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7d6140 0x5581cb7ddfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f15687ce740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7e0a00 0x5581cb7de500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f15687ce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5581cb41d790 con 0x5581cb7d6140 2024-06-04T05:16:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f15687ce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5581cb6f3890 con 0x5581cb7d5820 2024-06-04T05:16:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f15687ce740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5581cb7e3630 con 0x5581cb7e0a00 2024-06-04T05:16:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f15687ce740 10 client.0 _ll_register_callbacks cb 0x5581cb44ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f155effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d5820 0x5581cb7dda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f155effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d5820 0x5581cb7dda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:35706/0 (socket says 172.21.15.70:35706) 2024-06-04T05:16:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f155effd640 1 -- 172.21.15.70:0/11890200 learned_addr learned my addr 172.21.15.70:0/11890200 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f155e7fc640 1 --2- 172.21.15.70:0/11890200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7d6140 0x5581cb7ddfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f155f7fe640 1 --2- 172.21.15.70:0/11890200 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7e0a00 0x5581cb7de500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f155effd640 1 -- 172.21.15.70:0/11890200 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7e0a00 msgr2=0x5581cb7de500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f155effd640 1 --2- 172.21.15.70:0/11890200 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7e0a00 0x5581cb7de500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f155effd640 1 -- 172.21.15.70:0/11890200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7d6140 msgr2=0x5581cb7ddfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f155effd640 1 --2- 172.21.15.70:0/11890200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7d6140 0x5581cb7ddfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f155effd640 1 -- 172.21.15.70:0/11890200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581cb518620 con 0x5581cb7d5820 2024-06-04T05:16:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f155f7fe640 1 --2- 172.21.15.70:0/11890200 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7e0a00 0x5581cb7de500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:16:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f155effd640 1 --2- 172.21.15.70:0/11890200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d5820 0x5581cb7dda80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1550009750 tx=0x7f1550013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a63380b765c39191 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f153effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:16:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.337+0000 7f153effd640 1 -- 172.21.15.70:0/11890200 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155001b810 con 0x5581cb7d5820 2024-06-04T05:16:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.338+0000 7f153effd640 1 -- 172.21.15.70:0/11890200 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f155001b9b0 con 0x5581cb7d5820 2024-06-04T05:16:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.338+0000 7f15687ce740 1 -- 172.21.15.70:0/11890200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15580054d0 con 0x5581cb7d5820 2024-06-04T05:16:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.338+0000 7f153effd640 1 -- 172.21.15.70:0/11890200 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f155001bbb0 con 0x5581cb7d5820 2024-06-04T05:16:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.338+0000 7f15687ce740 10 client.29186 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.338+0000 7f15687ce740 1 -- 172.21.15.70:0/11890200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x5581cb7ef4d0 con 0x5581cb7d5820 2024-06-04T05:16:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.338+0000 7f153effd640 1 -- 172.21.15.70:0/11890200 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f15580054d0 con 0x5581cb7d5820 2024-06-04T05:16:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.338+0000 7f153effd640 1 -- 172.21.15.70:0/11890200 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f155000e3b0 con 0x5581cb7d5820 2024-06-04T05:16:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.338+0000 7f15687ce740 10 client.29186 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.338+0000 7f15687ce740 10 client.29186 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:16:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.338+0000 7f15687ce740 20 client.29186 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.338+0000 7f15687ce740 1 --2- 172.21.15.70:0/11890200 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5581cb7f0990 0x5581cb7f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.338+0000 7f15687ce740 4 client.29186 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:16:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.338+0000 7f15687ce740 1 -- 172.21.15.70:0/11890200 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581cb749b80 con 0x5581cb7f0990 2024-06-04T05:16:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.339+0000 7f155e7fc640 1 --2- 172.21.15.70:0/11890200 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5581cb7f0990 0x5581cb7f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.339+0000 7f153effd640 10 client.29186 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.339+0000 7f155e7fc640 1 --2- 172.21.15.70:0/11890200 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5581cb7f0990 0x5581cb7f2d80 crc :-1 s=READY pgs=411 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35ab4574a80d6b86 server_cookie=1658b21bd8c828ab in_seq=0 out_seq=0 2024-06-04T05:16:21.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.341+0000 7f153effd640 1 -- 172.21.15.70:0/11890200 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5581cb749b80 con 0x5581cb7f0990 2024-06-04T05:16:21.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.341+0000 7f153effd640 10 client.29186 handle_command_reply: tid=0 2024-06-04T05:16:21.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.341+0000 7f15687ce740 1 client.29186 shutdown 2024-06-04T05:16:21.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.341+0000 7f15687ce740 1 -- 172.21.15.70:0/11890200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d5820 msgr2=0x5581cb7dda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:21.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.341+0000 7f15687ce740 1 --2- 172.21.15.70:0/11890200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d5820 0x5581cb7dda80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1550009750 tx=0x7f1550013040 comp rx=0 tx=0).stop 2024-06-04T05:16:21.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.341+0000 7f15687ce740 1 -- 172.21.15.70:0/11890200 shutdown_connections 2024-06-04T05:16:21.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.341+0000 7f15687ce740 1 --2- 172.21.15.70:0/11890200 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5581cb7f0990 0x5581cb7f2d80 crc :-1 s=READY pgs=411 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.341+0000 7f15687ce740 1 --2- 172.21.15.70:0/11890200 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7e0a00 0x5581cb7de500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.341+0000 7f15687ce740 1 --2- 172.21.15.70:0/11890200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7d6140 0x5581cb7ddfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.341+0000 7f15687ce740 1 --2- 172.21.15.70:0/11890200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d5820 0x5581cb7dda80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.341+0000 7f15687ce740 1 -- 172.21.15.70:0/11890200 >> 172.21.15.70:0/11890200 conn(0x5581cb7d1d30 msgr2=0x5581cb7d0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:21.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.341+0000 7f153effd640 0 client.29186 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:21.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.341+0000 7f15687ce740 1 -- 172.21.15.70:0/11890200 shutdown_connections 2024-06-04T05:16:21.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.341+0000 7f15687ce740 1 -- 172.21.15.70:0/11890200 wait complete. 2024-06-04T05:16:21.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.341+0000 7f15687ce740 20 client.29186 trim_cache size 0 max 16384 2024-06-04T05:16:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f15687ce740 1 Processor -- start 2024-06-04T05:16:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f15687ce740 1 -- start start 2024-06-04T05:16:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f15687ce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d5820 0x5581cb7d81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f15687ce740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7d6140 0x5581cb7de320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f15687ce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7f8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f15687ce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f15601bd470 con 0x5581cb7e0a00 2024-06-04T05:16:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f15687ce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f15601befb0 con 0x5581cb7d5820 2024-06-04T05:16:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f15687ce740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f15601bdb40 con 0x5581cb7d6140 2024-06-04T05:16:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f155f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7f8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f155f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7f8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37908/0 (socket says 172.21.15.70:37908) 2024-06-04T05:16:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f155f7fe640 1 -- 172.21.15.70:0/1864727593 learned_addr learned my addr 172.21.15.70:0/1864727593 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f155e7fc640 1 --2- 172.21.15.70:0/1864727593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7d6140 0x5581cb7de320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f155effd640 1 --2- 172.21.15.70:0/1864727593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d5820 0x5581cb7d81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f155f7fe640 1 -- 172.21.15.70:0/1864727593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7d6140 msgr2=0x5581cb7de320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f155f7fe640 1 --2- 172.21.15.70:0/1864727593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7d6140 0x5581cb7de320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f155f7fe640 1 -- 172.21.15.70:0/1864727593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d5820 msgr2=0x5581cb7d81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f155f7fe640 1 --2- 172.21.15.70:0/1864727593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d5820 0x5581cb7d81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f155f7fe640 1 -- 172.21.15.70:0/1864727593 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581cb7dbfd0 con 0x5581cb7e0a00 2024-06-04T05:16:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.357+0000 7f155f7fe640 1 --2- 172.21.15.70:0/1864727593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7f8640 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f15580aa810 tx=0x7f1558005680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e63d09afca7e81 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.358+0000 7f15557fa640 1 -- 172.21.15.70:0/1864727593 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1558089070 con 0x5581cb7e0a00 2024-06-04T05:16:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.358+0000 7f15557fa640 1 -- 172.21.15.70:0/1864727593 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f15580c8070 con 0x5581cb7e0a00 2024-06-04T05:16:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.358+0000 7f15557fa640 1 -- 172.21.15.70:0/1864727593 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15580b2b90 con 0x5581cb7e0a00 2024-06-04T05:16:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.358+0000 7f15687ce740 1 -- 172.21.15.70:0/1864727593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 msgr2=0x5581cb7f8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.358+0000 7f15687ce740 1 --2- 172.21.15.70:0/1864727593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7f8640 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f15580aa810 tx=0x7f1558005680 comp rx=0 tx=0).stop 2024-06-04T05:16:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.358+0000 7f15687ce740 1 -- 172.21.15.70:0/1864727593 shutdown_connections 2024-06-04T05:16:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.358+0000 7f15687ce740 1 --2- 172.21.15.70:0/1864727593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7f8640 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.358+0000 7f15687ce740 1 --2- 172.21.15.70:0/1864727593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7d6140 0x5581cb7de320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.358+0000 7f15687ce740 1 --2- 172.21.15.70:0/1864727593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d5820 0x5581cb7d81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.358+0000 7f15687ce740 1 -- 172.21.15.70:0/1864727593 >> 172.21.15.70:0/1864727593 conn(0x5581cb7d1d30 msgr2=0x5581cb7ebf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.358+0000 7f15687ce740 1 -- 172.21.15.70:0/1864727593 shutdown_connections 2024-06-04T05:16:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.358+0000 7f15687ce740 1 -- 172.21.15.70:0/1864727593 wait complete. 2024-06-04T05:16:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f15687ce740 1 -- start start 2024-06-04T05:16:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f15687ce740 1 Processor -- start 2024-06-04T05:16:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f15687ce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d5820 0x5581cb7e9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f15687ce740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7d6140 0x5581cb7da8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f15687ce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7daf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f15687ce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f15601be8e0 con 0x5581cb7e0a00 2024-06-04T05:16:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f15687ce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f15601be210 con 0x5581cb7d5820 2024-06-04T05:16:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f15687ce740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f15601bb470 con 0x5581cb7d6140 2024-06-04T05:16:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f15687ce740 10 client.0 _ll_register_callbacks cb 0x7f1550009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f155f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7daf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f155f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7daf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37914/0 (socket says 172.21.15.70:37914) 2024-06-04T05:16:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f155f7fe640 1 -- 172.21.15.70:0/437062625 learned_addr learned my addr 172.21.15.70:0/437062625 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f155effd640 1 --2- 172.21.15.70:0/437062625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d5820 0x5581cb7e9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f155e7fc640 1 --2- 172.21.15.70:0/437062625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7d6140 0x5581cb7da8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f155f7fe640 1 -- 172.21.15.70:0/437062625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7d6140 msgr2=0x5581cb7da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f155f7fe640 1 --2- 172.21.15.70:0/437062625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7d6140 0x5581cb7da8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f155f7fe640 1 -- 172.21.15.70:0/437062625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d5820 msgr2=0x5581cb7e9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f155f7fe640 1 --2- 172.21.15.70:0/437062625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d5820 0x5581cb7e9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.359+0000 7f155f7fe640 1 -- 172.21.15.70:0/437062625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581cb7db720 con 0x5581cb7e0a00 2024-06-04T05:16:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.360+0000 7f155f7fe640 1 --2- 172.21.15.70:0/437062625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7daf20 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f15580a5fe0 tx=0x7f15580a9d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2225c8c734e1e8f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.360+0000 7f153effd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.360+0000 7f153effd640 1 -- 172.21.15.70:0/437062625 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1558089030 con 0x5581cb7e0a00 2024-06-04T05:16:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.360+0000 7f153effd640 1 -- 172.21.15.70:0/437062625 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f15580a6550 con 0x5581cb7e0a00 2024-06-04T05:16:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.360+0000 7f15687ce740 1 -- 172.21.15.70:0/437062625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15580c8070 con 0x5581cb7e0a00 2024-06-04T05:16:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.360+0000 7f153effd640 1 -- 172.21.15.70:0/437062625 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f1558090020 con 0x5581cb7e0a00 2024-06-04T05:16:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.360+0000 7f15687ce740 10 client.20403 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.360+0000 7f15687ce740 1 -- 172.21.15.70:0/437062625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x5581cb85b5f0 con 0x5581cb7e0a00 2024-06-04T05:16:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.360+0000 7f153effd640 1 -- 172.21.15.70:0/437062625 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f15580c8070 con 0x5581cb7e0a00 2024-06-04T05:16:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.360+0000 7f153effd640 1 -- 172.21.15.70:0/437062625 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f15580a58d0 con 0x5581cb7e0a00 2024-06-04T05:16:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.360+0000 7f15687ce740 10 client.20403 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.360+0000 7f15687ce740 10 client.20403 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:16:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.360+0000 7f15687ce740 20 client.20403 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.360+0000 7f15687ce740 1 --2- 172.21.15.70:0/437062625 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5581cb85b7c0 0x5581cb85dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.360+0000 7f15687ce740 4 client.20403 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:16:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.360+0000 7f15687ce740 1 -- 172.21.15.70:0/437062625 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5581cb743360 con 0x5581cb85b7c0 2024-06-04T05:16:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.361+0000 7f155effd640 1 --2- 172.21.15.70:0/437062625 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5581cb85b7c0 0x5581cb85dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.361+0000 7f153effd640 10 client.20403 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.361+0000 7f155effd640 1 --2- 172.21.15.70:0/437062625 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5581cb85b7c0 0x5581cb85dbb0 crc :-1 s=READY pgs=412 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90a1faba322207aa server_cookie=5c1c36b69028dafb in_seq=0 out_seq=0 2024-06-04T05:16:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.460+0000 7f153effd640 1 -- 172.21.15.70:0/437062625 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5581cb743360 con 0x5581cb85b7c0 2024-06-04T05:16:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.460+0000 7f153effd640 10 client.20403 handle_command_reply: tid=0 2024-06-04T05:16:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.460+0000 7f15687ce740 1 client.20403 shutdown 2024-06-04T05:16:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.461+0000 7f15687ce740 1 -- 172.21.15.70:0/437062625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 msgr2=0x5581cb7daf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.461+0000 7f15687ce740 1 --2- 172.21.15.70:0/437062625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7daf20 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f15580a5fe0 tx=0x7f15580a9d30 comp rx=0 tx=0).stop 2024-06-04T05:16:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.461+0000 7f15687ce740 1 -- 172.21.15.70:0/437062625 shutdown_connections 2024-06-04T05:16:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.461+0000 7f15687ce740 1 --2- 172.21.15.70:0/437062625 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5581cb85b7c0 0x5581cb85dbb0 crc :-1 s=READY pgs=412 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.461+0000 7f15687ce740 1 --2- 172.21.15.70:0/437062625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb7e0a00 0x5581cb7daf20 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.461+0000 7f15687ce740 1 --2- 172.21.15.70:0/437062625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb7d6140 0x5581cb7da8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.461+0000 7f15687ce740 1 --2- 172.21.15.70:0/437062625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb7d5820 0x5581cb7e9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.461+0000 7f15687ce740 1 -- 172.21.15.70:0/437062625 >> 172.21.15.70:0/437062625 conn(0x5581cb7d1d30 msgr2=0x5581cb7df2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.461+0000 7f153effd640 0 client.20403 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.461+0000 7f15687ce740 1 -- 172.21.15.70:0/437062625 shutdown_connections 2024-06-04T05:16:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.461+0000 7f15687ce740 1 -- 172.21.15.70:0/437062625 wait complete. 2024-06-04T05:16:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.461+0000 7f15687ce740 20 client.20403 trim_cache size 0 max 16384 2024-06-04T05:16:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:16:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:16:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:16:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.462+0000 7f15659da640 1 -- 172.21.15.70:0/1828095974 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f154c0b2a60 msgr2=0x7f154c0b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.462+0000 7f15659da640 1 --2- 172.21.15.70:0/1828095974 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f154c0b2a60 0x7f154c0b4f20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1550009c10 tx=0x7f1550005f90 comp rx=0 tx=0).stop 2024-06-04T05:16:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.462+0000 7f15659da640 1 -- 172.21.15.70:0/1828095974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1560107960 msgr2=0x7f1560112110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.462+0000 7f15659da640 1 --2- 172.21.15.70:0/1828095974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1560107960 0x7f1560112110 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1558009fd0 tx=0x7f155800ef10 comp rx=0 tx=0).stop 2024-06-04T05:16:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.462+0000 7f15659da640 1 -- 172.21.15.70:0/1828095974 shutdown_connections 2024-06-04T05:16:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.462+0000 7f15659da640 1 --2- 172.21.15.70:0/1828095974 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f154c0b2a60 0x7f154c0b4f20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.462+0000 7f15659da640 1 --2- 172.21.15.70:0/1828095974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1560107960 0x7f1560112110 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.462+0000 7f15659da640 1 --2- 172.21.15.70:0/1828095974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1560102310 0x7f15601174a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.462+0000 7f15659da640 1 --2- 172.21.15.70:0/1828095974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1560101950 0x7f1560116f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.462+0000 7f15659da640 1 -- 172.21.15.70:0/1828095974 >> 172.21.15.70:0/1828095974 conn(0x7f1560078aa0 msgr2=0x7f15601045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.462+0000 7f15659da640 1 -- 172.21.15.70:0/1828095974 shutdown_connections 2024-06-04T05:16:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:21.462+0000 7f15659da640 1 -- 172.21.15.70:0/1828095974 wait complete. 2024-06-04T05:16:21.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:21 smithi070 ceph-mon[32631]: pgmap v602: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 36 KiB/s wr, 22 op/s 2024-06-04T05:16:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:21 smithi112 ceph-mon[36453]: pgmap v602: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 36 KiB/s wr, 22 op/s 2024-06-04T05:16:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:21 smithi098 ceph-mon[35569]: pgmap v602: 97 pgs: 97 active+clean; 665 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 36 KiB/s wr, 22 op/s 2024-06-04T05:16:21.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41606 6.99 MB/sec execute 193 sec latency 85.249 ms 2024-06-04T05:16:22.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41746 6.98 MB/sec execute 194 sec latency 131.734 ms 2024-06-04T05:16:23.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:23 smithi070 ceph-mon[32631]: pgmap v603: 97 pgs: 97 active+clean; 675 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 968 KiB/s wr, 83 op/s 2024-06-04T05:16:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:23 smithi112 ceph-mon[36453]: pgmap v603: 97 pgs: 97 active+clean; 675 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 968 KiB/s wr, 83 op/s 2024-06-04T05:16:23.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:23 smithi098 ceph-mon[35569]: pgmap v603: 97 pgs: 97 active+clean; 675 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 968 KiB/s wr, 83 op/s 2024-06-04T05:16:23.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 41893 6.99 MB/sec execute 195 sec latency 137.679 ms 2024-06-04T05:16:24.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 42027 6.97 MB/sec execute 196 sec latency 127.724 ms 2024-06-04T05:16:25.464 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:16:25.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:25 smithi070 ceph-mon[32631]: pgmap v604: 97 pgs: 97 active+clean; 675 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 941 KiB/s wr, 66 op/s 2024-06-04T05:16:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:25 smithi112 ceph-mon[36453]: pgmap v604: 97 pgs: 97 active+clean; 675 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 941 KiB/s wr, 66 op/s 2024-06-04T05:16:25.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:25 smithi098 ceph-mon[35569]: pgmap v604: 97 pgs: 97 active+clean; 675 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 941 KiB/s wr, 66 op/s 2024-06-04T05:16:25.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 42182 6.99 MB/sec execute 197 sec latency 132.351 ms 2024-06-04T05:16:26.719 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:16:26.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 42380 7.02 MB/sec execute 198 sec latency 59.412 ms 2024-06-04T05:16:27.462 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:27 smithi070 ceph-mon[32631]: pgmap v605: 97 pgs: 97 active+clean; 685 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 210 op/s 2024-06-04T05:16:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:27 smithi112 ceph-mon[36453]: pgmap v605: 97 pgs: 97 active+clean; 685 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 210 op/s 2024-06-04T05:16:27.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:27 smithi098 ceph-mon[35569]: pgmap v605: 97 pgs: 97 active+clean; 685 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 210 op/s 2024-06-04T05:16:27.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 42564 7.02 MB/sec execute 199 sec latency 68.911 ms 2024-06-04T05:16:27.965 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:16:27.965 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":57,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:15:55.006036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T05:16:27.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.285+0000 7fb1ffa31700 1 -- 172.21.15.70:0/799294769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1f8101b30 msgr2=0x7fb1f8101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.285+0000 7fb1ffa31700 1 --2- 172.21.15.70:0/799294769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1f8101b30 0x7fb1f8101fb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80098f0 tx=0x7fb1e8009c00 comp rx=0 tx=0).stop 2024-06-04T05:16:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.285+0000 7fb1ffa31700 1 -- 172.21.15.70:0/799294769 shutdown_connections 2024-06-04T05:16:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.285+0000 7fb1ffa31700 1 --2- 172.21.15.70:0/799294769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1f81024f0 0x7fb1f81103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.285+0000 7fb1ffa31700 1 --2- 172.21.15.70:0/799294769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1f8109520 0x7fb1f8109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.285+0000 7fb1ffa31700 1 --2- 172.21.15.70:0/799294769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1f8101b30 0x7fb1f8101fb0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.285+0000 7fb1ffa31700 1 -- 172.21.15.70:0/799294769 >> 172.21.15.70:0/799294769 conn(0x7fb1f8078ab0 msgr2=0x7fb1f80ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.285+0000 7fb1ffa31700 1 -- 172.21.15.70:0/799294769 shutdown_connections 2024-06-04T05:16:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.285+0000 7fb1ffa31700 1 -- 172.21.15.70:0/799294769 wait complete. 2024-06-04T05:16:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.286+0000 7fb1ffa31700 1 Processor -- start 2024-06-04T05:16:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.286+0000 7fb1ffa31700 1 -- start start 2024-06-04T05:16:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.286+0000 7fb1ffa31700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1f8101b30 0x7fb1f8102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.286+0000 7fb1ffa31700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1f81024f0 0x7fb1f8103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:27.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.287+0000 7fb1ffa31700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1f8109520 0x7fb1f8103790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.287+0000 7fb1ffa31700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f8113660 con 0x7fb1f8101b30 2024-06-04T05:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.287+0000 7fb1ffa31700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f81134e0 con 0x7fb1f8109520 2024-06-04T05:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.287+0000 7fb1ffa31700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f81137e0 con 0x7fb1f81024f0 2024-06-04T05:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.287+0000 7fb1fcfcc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1f81024f0 0x7fb1f8103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.287+0000 7fb1fd7cd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1f8101b30 0x7fb1f8102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.287+0000 7fb1fd7cd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1f8101b30 0x7fb1f8102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37858/0 (socket says 172.21.15.70:37858) 2024-06-04T05:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.287+0000 7fb1fdfce700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1f8109520 0x7fb1f8103790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.287+0000 7fb1fd7cd700 1 -- 172.21.15.70:0/4134803082 learned_addr learned my addr 172.21.15.70:0/4134803082 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.287+0000 7fb1fcfcc700 1 -- 172.21.15.70:0/4134803082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1f8109520 msgr2=0x7fb1f8103790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.287+0000 7fb1fcfcc700 1 --2- 172.21.15.70:0/4134803082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1f8109520 0x7fb1f8103790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.287+0000 7fb1fcfcc700 1 -- 172.21.15.70:0/4134803082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1f8101b30 msgr2=0x7fb1f8102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.287+0000 7fb1fcfcc700 1 --2- 172.21.15.70:0/4134803082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1f8101b30 0x7fb1f8102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.287+0000 7fb1fcfcc700 1 -- 172.21.15.70:0/4134803082 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1f81ac5b0 con 0x7fb1f81024f0 2024-06-04T05:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.287+0000 7fb1fd7cd700 1 --2- 172.21.15.70:0/4134803082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1f8101b30 0x7fb1f8102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:16:27.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.287+0000 7fb1fcfcc700 1 --2- 172.21.15.70:0/4134803082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1f81024f0 0x7fb1f8103250 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8009fd0 tx=0x7fb1e800fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:27.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.288+0000 7fb1ee7fc700 1 -- 172.21.15.70:0/4134803082 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e800b560 con 0x7fb1f81024f0 2024-06-04T05:16:27.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.288+0000 7fb1ee7fc700 1 -- 172.21.15.70:0/4134803082 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb1e800bb90 con 0x7fb1f81024f0 2024-06-04T05:16:27.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.288+0000 7fb1ee7fc700 1 -- 172.21.15.70:0/4134803082 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e80038e0 con 0x7fb1f81024f0 2024-06-04T05:16:27.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.288+0000 7fb1ffa31700 1 -- 172.21.15.70:0/4134803082 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1f81ac7e0 con 0x7fb1f81024f0 2024-06-04T05:16:27.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.288+0000 7fb1ffa31700 1 -- 172.21.15.70:0/4134803082 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1f81acb80 con 0x7fb1f81024f0 2024-06-04T05:16:27.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.289+0000 7fb1ffa31700 1 -- 172.21.15.70:0/4134803082 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1f806a230 con 0x7fb1f81024f0 2024-06-04T05:16:27.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.291+0000 7fb1ee7fc700 1 -- 172.21.15.70:0/4134803082 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fb1e80051b0 con 0x7fb1f81024f0 2024-06-04T05:16:27.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.292+0000 7fb1ee7fc700 1 --2- 172.21.15.70:0/4134803082 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1e40b2bc0 0x7fb1e40b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:27.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.292+0000 7fb1fd7cd700 1 --2- 172.21.15.70:0/4134803082 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1e40b2bc0 0x7fb1e40b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:27.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.292+0000 7fb1ee7fc700 1 -- 172.21.15.70:0/4134803082 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7fb1e80e3370 con 0x7fb1f81024f0 2024-06-04T05:16:27.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.292+0000 7fb1fd7cd700 1 --2- 172.21.15.70:0/4134803082 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1e40b2bc0 0x7fb1e40b5080 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4011550 tx=0x7fb1f4009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:27.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.298+0000 7fb1ee7fc700 1 -- 172.21.15.70:0/4134803082 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb1e8013030 con 0x7fb1f81024f0 2024-06-04T05:16:27.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.719+0000 7fb1ffa31700 1 -- 172.21.15.70:0/4134803082 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1f810b470 con 0x7fb1f81024f0 2024-06-04T05:16:27.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.720+0000 7fb1ee7fc700 1 -- 172.21.15.70:0/4134803082 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+11929 (secure 0 0 0) 0x7fb1e80aad50 con 0x7fb1f81024f0 2024-06-04T05:16:27.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.724+0000 7fb1ffa31700 1 -- 172.21.15.70:0/4134803082 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1e40b2bc0 msgr2=0x7fb1e40b5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:27.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.724+0000 7fb1ffa31700 1 --2- 172.21.15.70:0/4134803082 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1e40b2bc0 0x7fb1e40b5080 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4011550 tx=0x7fb1f4009260 comp rx=0 tx=0).stop 2024-06-04T05:16:27.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.724+0000 7fb1ffa31700 1 -- 172.21.15.70:0/4134803082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1f81024f0 msgr2=0x7fb1f8103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:27.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.724+0000 7fb1ffa31700 1 --2- 172.21.15.70:0/4134803082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1f81024f0 0x7fb1f8103250 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8009fd0 tx=0x7fb1e800fa00 comp rx=0 tx=0).stop 2024-06-04T05:16:27.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.724+0000 7fb1ffa31700 1 -- 172.21.15.70:0/4134803082 shutdown_connections 2024-06-04T05:16:27.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.724+0000 7fb1ffa31700 1 --2- 172.21.15.70:0/4134803082 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1e40b2bc0 0x7fb1e40b5080 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:27.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.724+0000 7fb1ffa31700 1 --2- 172.21.15.70:0/4134803082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1f8109520 0x7fb1f8103790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:27.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.724+0000 7fb1ffa31700 1 --2- 172.21.15.70:0/4134803082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1f8101b30 0x7fb1f8102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:27.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.724+0000 7fb1ffa31700 1 --2- 172.21.15.70:0/4134803082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1f81024f0 0x7fb1f8103250 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:27.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.724+0000 7fb1ffa31700 1 -- 172.21.15.70:0/4134803082 >> 172.21.15.70:0/4134803082 conn(0x7fb1f8078ab0 msgr2=0x7fb1f81004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:27.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.724+0000 7fb1ffa31700 1 -- 172.21.15.70:0/4134803082 shutdown_connections 2024-06-04T05:16:27.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:27.724+0000 7fb1ffa31700 1 -- 172.21.15.70:0/4134803082 wait complete. 2024-06-04T05:16:27.970 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 57 2024-06-04T05:16:28.562 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:28 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/4134803082' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:16:28.595 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 146.85 --await-for 45.0' 2024-06-04T05:16:28.595 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 146.85 --await-for 45.0 2024-06-04T05:16:28.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:28 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/4134803082' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:16:28.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:28 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/4134803082' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:16:28.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 42745 7.04 MB/sec execute 200 sec latency 127.933 ms 2024-06-04T05:16:29.562 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:29 smithi070 ceph-mon[32631]: pgmap v606: 97 pgs: 97 active+clean; 685 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 204 op/s 2024-06-04T05:16:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:29 smithi112 ceph-mon[36453]: pgmap v606: 97 pgs: 97 active+clean; 685 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 204 op/s 2024-06-04T05:16:29.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.718+0000 7f97d0e26700 1 -- 172.21.15.70:0/2507299097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97cc102e00 msgr2=0x7f97cc10f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:29.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.718+0000 7f97d0e26700 1 --2- 172.21.15.70:0/2507299097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97cc102e00 0x7f97cc10f2f0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f97c400b0a0 tx=0x7f97c400b3b0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.719+0000 7f97d0e26700 1 -- 172.21.15.70:0/2507299097 shutdown_connections 2024-06-04T05:16:29.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.719+0000 7f97d0e26700 1 --2- 172.21.15.70:0/2507299097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97cc102e00 0x7f97cc10f2f0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.719+0000 7f97d0e26700 1 --2- 172.21.15.70:0/2507299097 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97cc108520 0x7f97cc108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.719+0000 7f97d0e26700 1 --2- 172.21.15.70:0/2507299097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc102440 0x7f97cc1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.719+0000 7f97d0e26700 1 -- 172.21.15.70:0/2507299097 >> 172.21.15.70:0/2507299097 conn(0x7f97cc0793f0 msgr2=0x7f97cc079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:29.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.719+0000 7f97d0e26700 1 -- 172.21.15.70:0/2507299097 shutdown_connections 2024-06-04T05:16:29.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.719+0000 7f97d0e26700 1 -- 172.21.15.70:0/2507299097 wait complete. 2024-06-04T05:16:29.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.720+0000 7f97d0e26700 1 Processor -- start 2024-06-04T05:16:29.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.720+0000 7f97d0e26700 1 -- start start 2024-06-04T05:16:29.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.720+0000 7f97d0e26700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97cc102440 0x7f97cc19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.720+0000 7f97d0e26700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97cc102e00 0x7f97cc19fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.720+0000 7f97d0e26700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc108520 0x7f97cc199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.720+0000 7f97d0e26700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc1124d0 con 0x7f97cc102440 2024-06-04T05:16:29.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.720+0000 7f97d0e26700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc112350 con 0x7f97cc108520 2024-06-04T05:16:29.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.720+0000 7f97d0e26700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc112650 con 0x7f97cc102e00 2024-06-04T05:16:29.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.720+0000 7f97ca59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97cc102440 0x7f97cc19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:29.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.720+0000 7f97ca59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97cc102440 0x7f97cc19f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37874/0 (socket says 172.21.15.70:37874) 2024-06-04T05:16:29.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.720+0000 7f97ca59c700 1 -- 172.21.15.70:0/1013499867 learned_addr learned my addr 172.21.15.70:0/1013499867 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:29.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.720+0000 7f97c9d9b700 1 --2- 172.21.15.70:0/1013499867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97cc102e00 0x7f97cc19fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:29.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.720+0000 7f97cad9d700 1 --2- 172.21.15.70:0/1013499867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc108520 0x7f97cc199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:29.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.721+0000 7f97ca59c700 1 -- 172.21.15.70:0/1013499867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97cc102e00 msgr2=0x7f97cc19fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:29.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.721+0000 7f97ca59c700 1 --2- 172.21.15.70:0/1013499867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97cc102e00 0x7f97cc19fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.721+0000 7f97ca59c700 1 -- 172.21.15.70:0/1013499867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc108520 msgr2=0x7f97cc199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:29.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.721+0000 7f97ca59c700 1 --2- 172.21.15.70:0/1013499867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc108520 0x7f97cc199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.721+0000 7f97ca59c700 1 -- 172.21.15.70:0/1013499867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97cc19a0d0 con 0x7f97cc102440 2024-06-04T05:16:29.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.721+0000 7f97cad9d700 1 --2- 172.21.15.70:0/1013499867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc108520 0x7f97cc199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.721+0000 7f97ca59c700 1 --2- 172.21.15.70:0/1013499867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97cc102440 0x7f97cc19f760 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00e9a0 tx=0x7f97bc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.721+0000 7f97bb7fe700 1 -- 172.21.15.70:0/1013499867 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc020070 con 0x7f97cc102440 2024-06-04T05:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.721+0000 7f97d0e26700 1 -- 172.21.15.70:0/1013499867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97cc19a360 con 0x7f97cc102440 2024-06-04T05:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.721+0000 7f97bb7fe700 1 -- 172.21.15.70:0/1013499867 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97bc00cc50 con 0x7f97cc102440 2024-06-04T05:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.721+0000 7f97bb7fe700 1 -- 172.21.15.70:0/1013499867 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc01b840 con 0x7f97cc102440 2024-06-04T05:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.721+0000 7f97d0e26700 1 -- 172.21.15.70:0/1013499867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97cc103fb0 con 0x7f97cc102440 2024-06-04T05:16:29.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.725+0000 7f97bb7fe700 1 -- 172.21.15.70:0/1013499867 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f97bc023070 con 0x7f97cc102440 2024-06-04T05:16:29.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.726+0000 7f97bb7fe700 1 --2- 172.21.15.70:0/1013499867 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97b40b29f0 0x7f97b40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.726+0000 7f97bb7fe700 1 -- 172.21.15.70:0/1013499867 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f97bc0e6620 con 0x7f97cc102440 2024-06-04T05:16:29.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.726+0000 7f97c9d9b700 1 --2- 172.21.15.70:0/1013499867 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97b40b29f0 0x7f97b40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:29.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.727+0000 7f97c9d9b700 1 --2- 172.21.15.70:0/1013499867 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97b40b29f0 0x7f97b40b4eb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f97cc079ba0 tx=0x7f97c0009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:29.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.727+0000 7f97db6bbb80 1 Processor -- start 2024-06-04T05:16:29.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.727+0000 7f97db6bbb80 1 -- start start 2024-06-04T05:16:29.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.728+0000 7f97db6bbb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 0x560eb0036700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.728+0000 7f97db6bbb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb00308a0 0x560eb0030d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.728+0000 7f97db6bbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb0031260 0x560eb003d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.728+0000 7f97db6bbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560eaff8ed70 con 0x560eb0031260 2024-06-04T05:16:29.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.728+0000 7f97db6bbb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560eaff7e910 con 0x560eb0036320 2024-06-04T05:16:29.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.728+0000 7f97db6bbb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560eaff82950 con 0x560eb00308a0 2024-06-04T05:16:29.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.728+0000 7f97c9d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb00308a0 0x560eb0030d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:29.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.728+0000 7f97cad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb0031260 0x560eb003d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:29.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.728+0000 7f97c9d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb00308a0 0x560eb0030d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48552/0 (socket says 172.21.15.70:48552) 2024-06-04T05:16:29.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.728+0000 7f97c9d9b700 1 -- 172.21.15.70:0/141183541 learned_addr learned my addr 172.21.15.70:0/141183541 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:29.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.728+0000 7f97ca59c700 1 --2- 172.21.15.70:0/141183541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 0x560eb0036700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:29.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.729+0000 7f97cad9d700 1 -- 172.21.15.70:0/141183541 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb00308a0 msgr2=0x560eb0030d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:29.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.729+0000 7f97cad9d700 1 --2- 172.21.15.70:0/141183541 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb00308a0 0x560eb0030d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.729+0000 7f97cad9d700 1 -- 172.21.15.70:0/141183541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 msgr2=0x560eb0036700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:29.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.729+0000 7f97cad9d700 1 --2- 172.21.15.70:0/141183541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 0x560eb0036700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.729+0000 7f97cad9d700 1 -- 172.21.15.70:0/141183541 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560eaff27fb0 con 0x560eb0031260 2024-06-04T05:16:29.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.729+0000 7f97cad9d700 1 --2- 172.21.15.70:0/141183541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb0031260 0x560eb003d080 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f97c400b070 tx=0x7f97c4009df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21c54ae1b88dbe5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:29.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.729+0000 7f97b8ff9700 1 -- 172.21.15.70:0/141183541 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c4021070 con 0x560eb0031260 2024-06-04T05:16:29.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.729+0000 7f97b8ff9700 1 -- 172.21.15.70:0/141183541 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97c4009460 con 0x560eb0031260 2024-06-04T05:16:29.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.729+0000 7f97b8ff9700 1 -- 172.21.15.70:0/141183541 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c4007af0 con 0x560eb0031260 2024-06-04T05:16:29.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.730+0000 7f97db6bbb80 1 -- 172.21.15.70:0/141183541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb0031260 msgr2=0x560eb003d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:29.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.730+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/141183541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb0031260 0x560eb003d080 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f97c400b070 tx=0x7f97c4009df0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.730+0000 7f97db6bbb80 1 -- 172.21.15.70:0/141183541 shutdown_connections 2024-06-04T05:16:29.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.730+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/141183541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb0031260 0x560eb003d080 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.730+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/141183541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 0x560eb0036700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.730+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/141183541 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb00308a0 0x560eb0030d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.730+0000 7f97db6bbb80 1 -- 172.21.15.70:0/141183541 >> 172.21.15.70:0/141183541 conn(0x560eaff5c240 msgr2=0x560eb002b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:29.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.731+0000 7f97db6bbb80 1 -- 172.21.15.70:0/141183541 shutdown_connections 2024-06-04T05:16:29.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.731+0000 7f97db6bbb80 1 -- 172.21.15.70:0/141183541 wait complete. 2024-06-04T05:16:29.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.731+0000 7f97db6bbb80 1 -- start start 2024-06-04T05:16:29.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.731+0000 7f97db6bbb80 1 Processor -- start 2024-06-04T05:16:29.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.732+0000 7f97db6bbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb00308a0 0x560eb003cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.732+0000 7f97db6bbb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 0x560eb003a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.732+0000 7f97ca59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb00308a0 0x560eb003cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:29.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.732+0000 7f97ca59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb00308a0 0x560eb003cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37892/0 (socket says 172.21.15.70:37892) 2024-06-04T05:16:29.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.732+0000 7f97ca59c700 1 -- 172.21.15.70:0/1019475006 learned_addr learned my addr 172.21.15.70:0/1019475006 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:29.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.732+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/1019475006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 0x560eb003a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.732+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1019475006 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560eafe84fa0 con 0x560eb00308a0 2024-06-04T05:16:29.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.732+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1019475006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560eafee6da0 con 0x560eb0036320 2024-06-04T05:16:29.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.732+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1019475006 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560eb003d6d0 con 0x560eb0031260 2024-06-04T05:16:29.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.732+0000 7f97db6bbb80 10 client.0 _ll_register_callbacks cb 0x560eaff19950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:29.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.732+0000 7f97cad9d700 1 --2- 172.21.15.70:0/1019475006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 0x560eb003a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:29.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.733+0000 7f97c9d9b700 1 --2- 172.21.15.70:0/1019475006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 0x560eb003a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:29.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.733+0000 7f97ca59c700 1 -- 172.21.15.70:0/1019475006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 msgr2=0x560eb003a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:29.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.733+0000 7f97ca59c700 1 --2- 172.21.15.70:0/1019475006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 0x560eb003a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.733+0000 7f97ca59c700 1 -- 172.21.15.70:0/1019475006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 msgr2=0x560eb003a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.733+0000 7f97ca59c700 1 --2- 172.21.15.70:0/1019475006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 0x560eb003a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.733+0000 7f97ca59c700 1 -- 172.21.15.70:0/1019475006 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560eafece8d0 con 0x560eb00308a0 2024-06-04T05:16:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.733+0000 7f97ca59c700 1 --2- 172.21.15.70:0/1019475006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb00308a0 0x560eb003cd00 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00e970 tx=0x7f97bc0aedc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35281c58641a34e8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.733+0000 7f97b1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.733+0000 7f97b1ffb700 1 -- 172.21.15.70:0/1019475006 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc0ae390 con 0x560eb00308a0 2024-06-04T05:16:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.733+0000 7f97b1ffb700 1 -- 172.21.15.70:0/1019475006 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f97bc0d02c0 con 0x560eb00308a0 2024-06-04T05:16:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.733+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1019475006 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97c4009460 con 0x560eb00308a0 2024-06-04T05:16:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.734+0000 7f97b1ffb700 1 -- 172.21.15.70:0/1019475006 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f97bc0aef80 con 0x560eb00308a0 2024-06-04T05:16:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.734+0000 7f97db6bbb80 10 client.20421 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.734+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1019475006 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x560eaff38760 con 0x560eb00308a0 2024-06-04T05:16:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.734+0000 7f97b1ffb700 1 -- 172.21.15.70:0/1019475006 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97c4009460 con 0x560eb00308a0 2024-06-04T05:16:29.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.735+0000 7f97b1ffb700 1 -- 172.21.15.70:0/1019475006 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f97bc00c760 con 0x560eb00308a0 2024-06-04T05:16:29.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.735+0000 7f97db6bbb80 10 client.20421 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:29.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.735+0000 7f97db6bbb80 10 client.20421 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:16:29.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.735+0000 7f97db6bbb80 20 client.20421 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:29.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.735+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/1019475006 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560eb004a190 0x560eb004c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.735+0000 7f97db6bbb80 4 client.20421 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:16:29.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.735+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1019475006 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560eaff90cc0 con 0x560eb004a190 2024-06-04T05:16:29.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.735+0000 7f97c9d9b700 1 --2- 172.21.15.70:0/1019475006 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560eb004a190 0x560eb004c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:29.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.735+0000 7f97c9d9b700 1 --2- 172.21.15.70:0/1019475006 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560eb004a190 0x560eb004c5a0 crc :-1 s=READY pgs=413 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a673411b17a45abb server_cookie=4a1b1c8b4b72b3ca in_seq=0 out_seq=0 2024-06-04T05:16:29.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.735+0000 7f97b1ffb700 10 client.20421 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:29.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.736+0000 7f97b1ffb700 1 -- 172.21.15.70:0/1019475006 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560eaff90cc0 con 0x560eb004a190 2024-06-04T05:16:29.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.736+0000 7f97b1ffb700 10 client.20421 handle_command_reply: tid=0 2024-06-04T05:16:29.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.736+0000 7f97db6bbb80 1 client.20421 shutdown 2024-06-04T05:16:29.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.736+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1019475006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb00308a0 msgr2=0x560eb003cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:29.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.736+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/1019475006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb00308a0 0x560eb003cd00 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00e970 tx=0x7f97bc0aedc0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.736+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1019475006 shutdown_connections 2024-06-04T05:16:29.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.736+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/1019475006 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560eb004a190 0x560eb004c5a0 crc :-1 s=READY pgs=413 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.736+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/1019475006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 0x560eb003a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.736+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/1019475006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb00308a0 0x560eb003cd00 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.736+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/1019475006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 0x560eb003a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.736+0000 7f97b1ffb700 0 client.20421 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:29.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.736+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1019475006 >> 172.21.15.70:0/1019475006 conn(0x560eaff5c240 msgr2=0x560eb002aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:29.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.737+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1019475006 shutdown_connections 2024-06-04T05:16:29.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.737+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1019475006 wait complete. 2024-06-04T05:16:29.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.737+0000 7f97db6bbb80 20 client.20421 trim_cache size 0 max 16384 2024-06-04T05:16:29.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:29 smithi098 ceph-mon[35569]: pgmap v606: 97 pgs: 97 active+clean; 685 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 204 op/s 2024-06-04T05:16:29.746 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:16:29.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.768+0000 7f97db6bbb80 1 Processor -- start 2024-06-04T05:16:29.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.768+0000 7f97db6bbb80 1 -- start start 2024-06-04T05:16:29.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.768+0000 7f97db6bbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb00308a0 0x560eb002b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.768+0000 7f97db6bbb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 0x560eb0038ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.768+0000 7f97db6bbb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 0x560eb0032d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.768+0000 7f97db6bbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc1abb70 con 0x560eb00308a0 2024-06-04T05:16:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.768+0000 7f97db6bbb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc1ad6b0 con 0x560eb0036320 2024-06-04T05:16:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.768+0000 7f97db6bbb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc1ac240 con 0x560eb0031260 2024-06-04T05:16:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.769+0000 7f97c9d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 0x560eb0038ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.769+0000 7f97c9d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 0x560eb0038ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48564/0 (socket says 172.21.15.70:48564) 2024-06-04T05:16:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.769+0000 7f97c9d9b700 1 -- 172.21.15.70:0/1766005113 learned_addr learned my addr 172.21.15.70:0/1766005113 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.769+0000 7f97c9d9b700 1 -- 172.21.15.70:0/1766005113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 msgr2=0x560eb0032d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.769+0000 7f97c9d9b700 1 --2- 172.21.15.70:0/1766005113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 0x560eb0032d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.769+0000 7f97c9d9b700 1 -- 172.21.15.70:0/1766005113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb00308a0 msgr2=0x560eb002b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.769+0000 7f97c9d9b700 1 --2- 172.21.15.70:0/1766005113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb00308a0 0x560eb002b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.769+0000 7f97c9d9b700 1 -- 172.21.15.70:0/1766005113 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560eb0038570 con 0x560eb0031260 2024-06-04T05:16:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.769+0000 7f97c9d9b700 1 --2- 172.21.15.70:0/1766005113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 0x560eb0038ed0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f97c0005bf0 tx=0x7f97c0014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf5efaae0bbee57f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.769+0000 7f97b3fff700 1 -- 172.21.15.70:0/1766005113 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0003880 con 0x560eb0031260 2024-06-04T05:16:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.770+0000 7f97b3fff700 1 -- 172.21.15.70:0/1766005113 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97c0029420 con 0x560eb0031260 2024-06-04T05:16:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.770+0000 7f97b3fff700 1 -- 172.21.15.70:0/1766005113 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c001d670 con 0x560eb0031260 2024-06-04T05:16:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.770+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1766005113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 msgr2=0x560eb0038ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.770+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/1766005113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 0x560eb0038ed0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f97c0005bf0 tx=0x7f97c0014000 comp rx=0 tx=0).stop 2024-06-04T05:16:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.770+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1766005113 shutdown_connections 2024-06-04T05:16:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.770+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/1766005113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 0x560eb0032d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.770+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/1766005113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb00308a0 0x560eb002b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.770+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/1766005113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 0x560eb0038ed0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.771+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1766005113 >> 172.21.15.70:0/1766005113 conn(0x560eaff5c240 msgr2=0x560eb00387c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.771+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1766005113 shutdown_connections 2024-06-04T05:16:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.771+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1766005113 wait complete. 2024-06-04T05:16:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.771+0000 7f97db6bbb80 1 -- start start 2024-06-04T05:16:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.771+0000 7f97db6bbb80 1 Processor -- start 2024-06-04T05:16:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.772+0000 7f97db6bbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb00308a0 0x560eb0042980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.772+0000 7f97db6bbb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 0x560eb0042ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.772+0000 7f97db6bbb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 0x560eb003d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.772+0000 7f97db6bbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc1acfe0 con 0x560eb00308a0 2024-06-04T05:16:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.772+0000 7f97db6bbb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc1ac910 con 0x560eb0036320 2024-06-04T05:16:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.772+0000 7f97c9d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 0x560eb0042ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.772+0000 7f97db6bbb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc1a9b50 con 0x560eb0031260 2024-06-04T05:16:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.772+0000 7f97ca59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb00308a0 0x560eb0042980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.772+0000 7f97c9d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 0x560eb0042ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48576/0 (socket says 172.21.15.70:48576) 2024-06-04T05:16:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.772+0000 7f97c9d9b700 1 -- 172.21.15.70:0/1301360414 learned_addr learned my addr 172.21.15.70:0/1301360414 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.772+0000 7f97db6bbb80 10 client.0 _ll_register_callbacks cb 0x560eaff19950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.772+0000 7f97cad9d700 1 --2- 172.21.15.70:0/1301360414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 0x560eb003d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.773+0000 7f97c9d9b700 1 -- 172.21.15.70:0/1301360414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 msgr2=0x560eb003d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.773+0000 7f97c9d9b700 1 --2- 172.21.15.70:0/1301360414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 0x560eb003d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.773+0000 7f97c9d9b700 1 -- 172.21.15.70:0/1301360414 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb00308a0 msgr2=0x560eb0042980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.773+0000 7f97c9d9b700 1 --2- 172.21.15.70:0/1301360414 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb00308a0 0x560eb0042980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.773+0000 7f97c9d9b700 1 -- 172.21.15.70:0/1301360414 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560eb003de70 con 0x560eb0031260 2024-06-04T05:16:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.773+0000 7f97c9d9b700 1 --2- 172.21.15.70:0/1301360414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 0x560eb0042ec0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f97c0005bf0 tx=0x7f97c0005330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c54a2b8e7bbddd8d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.773+0000 7f97b1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:16:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.773+0000 7f97b1ffb700 1 -- 172.21.15.70:0/1301360414 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0037070 con 0x560eb0031260 2024-06-04T05:16:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.773+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1301360414 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97c0029420 con 0x560eb0031260 2024-06-04T05:16:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.773+0000 7f97b1ffb700 1 -- 172.21.15.70:0/1301360414 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f97c0004950 con 0x560eb0031260 2024-06-04T05:16:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.774+0000 7f97b1ffb700 1 -- 172.21.15.70:0/1301360414 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f97c0004b50 con 0x560eb0031260 2024-06-04T05:16:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.774+0000 7f97db6bbb80 10 client.29329 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.774+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1301360414 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x560eb00bdcc0 con 0x560eb0031260 2024-06-04T05:16:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.774+0000 7f97b1ffb700 1 -- 172.21.15.70:0/1301360414 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97c0029420 con 0x560eb0031260 2024-06-04T05:16:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.774+0000 7f97b1ffb700 1 -- 172.21.15.70:0/1301360414 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f97c0029660 con 0x560eb0031260 2024-06-04T05:16:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.774+0000 7f97db6bbb80 10 client.29329 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.774+0000 7f97db6bbb80 10 client.29329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:16:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.774+0000 7f97db6bbb80 20 client.29329 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.774+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/1301360414 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560eb00be380 0x560eb00c07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.774+0000 7f97db6bbb80 4 client.29329 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:16:29.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.774+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1301360414 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560eaff8a4a0 con 0x560eb00be380 2024-06-04T05:16:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.775+0000 7f97ca59c700 1 --2- 172.21.15.70:0/1301360414 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560eb00be380 0x560eb00c07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.776+0000 7f97b1ffb700 10 client.29329 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.776+0000 7f97ca59c700 1 --2- 172.21.15.70:0/1301360414 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560eb00be380 0x560eb00c07a0 crc :-1 s=READY pgs=414 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37f0d740e9c53878 server_cookie=88f39bbfbf205be0 in_seq=0 out_seq=0 2024-06-04T05:16:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.776+0000 7f97b1ffb700 1 -- 172.21.15.70:0/1301360414 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560eaff8a4a0 con 0x560eb00be380 2024-06-04T05:16:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.776+0000 7f97b1ffb700 10 client.29329 handle_command_reply: tid=0 2024-06-04T05:16:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.776+0000 7f97db6bbb80 1 client.29329 shutdown 2024-06-04T05:16:29.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.777+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1301360414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 msgr2=0x560eb0042ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.777+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/1301360414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 0x560eb0042ec0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f97c0005bf0 tx=0x7f97c0005330 comp rx=0 tx=0).stop 2024-06-04T05:16:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.777+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1301360414 shutdown_connections 2024-06-04T05:16:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.777+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/1301360414 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560eb00be380 0x560eb00c07a0 crc :-1 s=READY pgs=414 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.777+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/1301360414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560eb0036320 0x560eb003d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.777+0000 7f97b1ffb700 0 client.29329 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.777+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/1301360414 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560eb00308a0 0x560eb0042980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.777+0000 7f97db6bbb80 1 --2- 172.21.15.70:0/1301360414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb0031260 0x560eb0042ec0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.777+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1301360414 >> 172.21.15.70:0/1301360414 conn(0x560eaff5c240 msgr2=0x560eb0039050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.777+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1301360414 shutdown_connections 2024-06-04T05:16:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.777+0000 7f97db6bbb80 1 -- 172.21.15.70:0/1301360414 wait complete. 2024-06-04T05:16:29.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.777+0000 7f97db6bbb80 20 client.29329 trim_cache size 0 max 16384 2024-06-04T05:16:29.782 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:16:29.782 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:16:29.782 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:16:29.782 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:16:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.778+0000 7f97d0e26700 1 -- 172.21.15.70:0/1013499867 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97b40b29f0 msgr2=0x7f97b40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.778+0000 7f97d0e26700 1 --2- 172.21.15.70:0/1013499867 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97b40b29f0 0x7f97b40b4eb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f97cc079ba0 tx=0x7f97c0009260 comp rx=0 tx=0).stop 2024-06-04T05:16:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.778+0000 7f97d0e26700 1 -- 172.21.15.70:0/1013499867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97cc102440 msgr2=0x7f97cc19f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.778+0000 7f97d0e26700 1 --2- 172.21.15.70:0/1013499867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97cc102440 0x7f97cc19f760 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00e9a0 tx=0x7f97bc00ecb0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.778+0000 7f97d0e26700 1 -- 172.21.15.70:0/1013499867 shutdown_connections 2024-06-04T05:16:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.778+0000 7f97d0e26700 1 --2- 172.21.15.70:0/1013499867 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97b40b29f0 0x7f97b40b4eb0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.778+0000 7f97d0e26700 1 --2- 172.21.15.70:0/1013499867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc108520 0x7f97cc199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.778+0000 7f97d0e26700 1 --2- 172.21.15.70:0/1013499867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97cc102440 0x7f97cc19f760 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.778+0000 7f97d0e26700 1 --2- 172.21.15.70:0/1013499867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97cc102e00 0x7f97cc19fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.778+0000 7f97d0e26700 1 -- 172.21.15.70:0/1013499867 >> 172.21.15.70:0/1013499867 conn(0x7f97cc0793f0 msgr2=0x7f97cc06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.779+0000 7f97d0e26700 1 -- 172.21.15.70:0/1013499867 shutdown_connections 2024-06-04T05:16:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:29.779+0000 7f97d0e26700 1 -- 172.21.15.70:0/1013499867 wait complete. 2024-06-04T05:16:29.819 INFO:tasks.workunit.client.0.smithi070.stdout: 10 42917 7.04 MB/sec execute 201 sec latency 93.046 ms 2024-06-04T05:16:30.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43104 7.07 MB/sec execute 202 sec latency 97.641 ms 2024-06-04T05:16:31.538 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:31 smithi070 ceph-mon[32631]: pgmap v607: 97 pgs: 97 active+clean; 685 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 204 op/s 2024-06-04T05:16:31.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:31 smithi112 ceph-mon[36453]: pgmap v607: 97 pgs: 97 active+clean; 685 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 204 op/s 2024-06-04T05:16:31.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:31 smithi098 ceph-mon[35569]: pgmap v607: 97 pgs: 97 active+clean; 685 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 204 op/s 2024-06-04T05:16:31.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43289 7.09 MB/sec execute 203 sec latency 118.366 ms 2024-06-04T05:16:31.889 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:16:31.889 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:16:32.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.482+0000 7f2b457c4700 1 -- 172.21.15.70:0/1093746980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b40103890 msgr2=0x7f2b4010fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:32.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.482+0000 7f2b457c4700 1 --2- 172.21.15.70:0/1093746980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b40103890 0x7f2b4010fab0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009990 tx=0x7f2b34009ca0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.482+0000 7f2b457c4700 1 -- 172.21.15.70:0/1093746980 shutdown_connections 2024-06-04T05:16:32.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.482+0000 7f2b457c4700 1 --2- 172.21.15.70:0/1093746980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b4010fff0 0x7f2b401123e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.482+0000 7f2b457c4700 1 --2- 172.21.15.70:0/1093746980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b40102f70 0x7f2b40103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.482+0000 7f2b457c4700 1 --2- 172.21.15.70:0/1093746980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b40103890 0x7f2b4010fab0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.482+0000 7f2b457c4700 1 -- 172.21.15.70:0/1093746980 >> 172.21.15.70:0/1093746980 conn(0x7f2b400793e0 msgr2=0x7f2b400797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:32.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.482+0000 7f2b457c4700 1 -- 172.21.15.70:0/1093746980 shutdown_connections 2024-06-04T05:16:32.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.482+0000 7f2b457c4700 1 -- 172.21.15.70:0/1093746980 wait complete. 2024-06-04T05:16:32.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.484+0000 7f2b457c4700 1 Processor -- start 2024-06-04T05:16:32.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.484+0000 7f2b457c4700 1 -- start start 2024-06-04T05:16:32.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.484+0000 7f2b457c4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b40102f70 0x7f2b4010f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.484+0000 7f2b457c4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b40103890 0x7f2b4010c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.485+0000 7f2b3effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b40102f70 0x7f2b4010f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.485+0000 7f2b457c4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b4010fff0 0x7f2b4010cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.485+0000 7f2b3e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b40103890 0x7f2b4010c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.485+0000 7f2b3effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b40102f70 0x7f2b4010f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:49154/0 (socket says 172.21.15.70:49154) 2024-06-04T05:16:32.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.485+0000 7f2b3effd700 1 -- 172.21.15.70:0/4200324692 learned_addr learned my addr 172.21.15.70:0/4200324692 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:32.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.485+0000 7f2b457c4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2b40115800 con 0x7f2b40103890 2024-06-04T05:16:32.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.485+0000 7f2b457c4700 1 -- 172.21.15.70:0/4200324692 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2b40115680 con 0x7f2b40102f70 2024-06-04T05:16:32.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.485+0000 7f2b457c4700 1 -- 172.21.15.70:0/4200324692 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2b40115980 con 0x7f2b4010fff0 2024-06-04T05:16:32.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.485+0000 7f2b3f7fe700 1 --2- 172.21.15.70:0/4200324692 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b4010fff0 0x7f2b4010cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.485+0000 7f2b3effd700 1 -- 172.21.15.70:0/4200324692 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b4010fff0 msgr2=0x7f2b4010cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:32.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.485+0000 7f2b3effd700 1 --2- 172.21.15.70:0/4200324692 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b4010fff0 0x7f2b4010cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.485+0000 7f2b3effd700 1 -- 172.21.15.70:0/4200324692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b40103890 msgr2=0x7f2b4010c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:32.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.485+0000 7f2b3effd700 1 --2- 172.21.15.70:0/4200324692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b40103890 0x7f2b4010c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.485+0000 7f2b3effd700 1 -- 172.21.15.70:0/4200324692 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4010d560 con 0x7f2b40102f70 2024-06-04T05:16:32.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.485+0000 7f2b3e7fc700 1 --2- 172.21.15.70:0/4200324692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b40103890 0x7f2b4010c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:16:32.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.485+0000 7f2b3effd700 1 --2- 172.21.15.70:0/4200324692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b40102f70 0x7f2b4010f7a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2b3000e9a0 tx=0x7f2b3000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:32.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.486+0000 7f2b27fff700 1 -- 172.21.15.70:0/4200324692 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b3000cbd0 con 0x7f2b40102f70 2024-06-04T05:16:32.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.486+0000 7f2b27fff700 1 -- 172.21.15.70:0/4200324692 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2b3000cd70 con 0x7f2b40102f70 2024-06-04T05:16:32.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.486+0000 7f2b457c4700 1 -- 172.21.15.70:0/4200324692 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b401ac540 con 0x7f2b40102f70 2024-06-04T05:16:32.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.486+0000 7f2b27fff700 1 -- 172.21.15.70:0/4200324692 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b30018680 con 0x7f2b40102f70 2024-06-04T05:16:32.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.486+0000 7f2b457c4700 1 -- 172.21.15.70:0/4200324692 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b401aca70 con 0x7f2b40102f70 2024-06-04T05:16:32.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.489+0000 7f2b27fff700 1 -- 172.21.15.70:0/4200324692 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f2b30018820 con 0x7f2b40102f70 2024-06-04T05:16:32.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.490+0000 7f2b27fff700 1 --2- 172.21.15.70:0/4200324692 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2b280b2ba0 0x7f2b280b5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.490+0000 7f2b27fff700 1 -- 172.21.15.70:0/4200324692 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f2b300e2720 con 0x7f2b40102f70 2024-06-04T05:16:32.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.490+0000 7f2b3e7fc700 1 --2- 172.21.15.70:0/4200324692 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2b280b2ba0 0x7f2b280b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.491+0000 7f2b3e7fc700 1 --2- 172.21.15.70:0/4200324692 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2b280b2ba0 0x7f2b280b5060 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2b3400b2e0 tx=0x7f2b3400b610 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:32.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.492+0000 7f2b50059b80 1 Processor -- start 2024-06-04T05:16:32.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.492+0000 7f2b50059b80 1 -- start start 2024-06-04T05:16:32.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.492+0000 7f2b50059b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97ea350 0x55dcb97ea730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.492+0000 7f2b50059b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97e4980 0x55dcb97e4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.492+0000 7f2b50059b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 0x55dcb97f1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.492+0000 7f2b50059b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dcb9743e30 con 0x55dcb97e4980 2024-06-04T05:16:32.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.492+0000 7f2b3e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97e4980 0x55dcb97e4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.493+0000 7f2b3e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97e4980 0x55dcb97e4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37936/0 (socket says 172.21.15.70:37936) 2024-06-04T05:16:32.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.493+0000 7f2b3e7fc700 1 -- 172.21.15.70:0/4024825689 learned_addr learned my addr 172.21.15.70:0/4024825689 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:32.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.493+0000 7f2b3f7fe700 1 --2- 172.21.15.70:0/4024825689 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 0x55dcb97f1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.493+0000 7f2b50059b80 1 -- 172.21.15.70:0/4024825689 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dcb97339d0 con 0x55dcb97ea350 2024-06-04T05:16:32.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.493+0000 7f2b50059b80 1 -- 172.21.15.70:0/4024825689 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55dcb9737a10 con 0x55dcb97e5340 2024-06-04T05:16:32.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.493+0000 7f2b3effd700 1 --2- 172.21.15.70:0/4024825689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97ea350 0x55dcb97ea730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.493+0000 7f2b3effd700 1 -- 172.21.15.70:0/4024825689 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 msgr2=0x55dcb97f1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:32.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.493+0000 7f2b3effd700 1 --2- 172.21.15.70:0/4024825689 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 0x55dcb97f1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.493+0000 7f2b3effd700 1 -- 172.21.15.70:0/4024825689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97e4980 msgr2=0x55dcb97e4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:32.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.493+0000 7f2b3effd700 1 --2- 172.21.15.70:0/4024825689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97e4980 0x55dcb97e4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.493+0000 7f2b3effd700 1 -- 172.21.15.70:0/4024825689 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcb9683a70 con 0x55dcb97ea350 2024-06-04T05:16:32.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.493+0000 7f2b3e7fc700 1 --2- 172.21.15.70:0/4024825689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97e4980 0x55dcb97e4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:16:32.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.494+0000 7f2b3effd700 1 --2- 172.21.15.70:0/4024825689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97ea350 0x55dcb97ea730 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2b300dfde0 tx=0x7f2b300b2190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c28a008eda54953 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:32.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.494+0000 7f2b257fa700 1 -- 172.21.15.70:0/4024825689 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b300aa070 con 0x55dcb97ea350 2024-06-04T05:16:32.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.494+0000 7f2b257fa700 1 -- 172.21.15.70:0/4024825689 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2b300a9970 con 0x55dcb97ea350 2024-06-04T05:16:32.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.494+0000 7f2b257fa700 1 -- 172.21.15.70:0/4024825689 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b300a9bd0 con 0x55dcb97ea350 2024-06-04T05:16:32.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.495+0000 7f2b50059b80 1 -- 172.21.15.70:0/4024825689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97ea350 msgr2=0x55dcb97ea730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:32.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.495+0000 7f2b50059b80 1 --2- 172.21.15.70:0/4024825689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97ea350 0x55dcb97ea730 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2b300dfde0 tx=0x7f2b300b2190 comp rx=0 tx=0).stop 2024-06-04T05:16:32.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.495+0000 7f2b50059b80 1 -- 172.21.15.70:0/4024825689 shutdown_connections 2024-06-04T05:16:32.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.495+0000 7f2b50059b80 1 --2- 172.21.15.70:0/4024825689 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 0x55dcb97f1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.495+0000 7f2b50059b80 1 --2- 172.21.15.70:0/4024825689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97ea350 0x55dcb97ea730 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.495+0000 7f2b50059b80 1 --2- 172.21.15.70:0/4024825689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97e4980 0x55dcb97e4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.495+0000 7f2b50059b80 1 -- 172.21.15.70:0/4024825689 >> 172.21.15.70:0/4024825689 conn(0x55dcb9712220 msgr2=0x55dcb97df290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:32.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.495+0000 7f2b50059b80 1 -- 172.21.15.70:0/4024825689 shutdown_connections 2024-06-04T05:16:32.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.495+0000 7f2b50059b80 1 -- 172.21.15.70:0/4024825689 wait complete. 2024-06-04T05:16:32.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.496+0000 7f2b50059b80 1 -- start start 2024-06-04T05:16:32.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.496+0000 7f2b50059b80 1 Processor -- start 2024-06-04T05:16:32.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.496+0000 7f2b50059b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 0x55dcb97f0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.497+0000 7f2b50059b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 0x55dcb97ee150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.497+0000 7f2b50059b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 0x55dcb97ee690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.497+0000 7f2b50059b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dcb963a140 con 0x55dcb97ea350 2024-06-04T05:16:32.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.497+0000 7f2b50059b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dcb96d5960 con 0x55dcb97e4980 2024-06-04T05:16:32.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.497+0000 7f2b50059b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55dcb97f16c0 con 0x55dcb97e5340 2024-06-04T05:16:32.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.497+0000 7f2b50059b80 10 client.0 _ll_register_callbacks cb 0x55dcb96ceac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:32.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.497+0000 7f2b3effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 0x55dcb97f0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.497+0000 7f2b3f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 0x55dcb97ee690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.497+0000 7f2b3effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 0x55dcb97f0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:49178/0 (socket says 172.21.15.70:49178) 2024-06-04T05:16:32.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.497+0000 7f2b3effd700 1 -- 172.21.15.70:0/3574894346 learned_addr learned my addr 172.21.15.70:0/3574894346 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:32.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.497+0000 7f2b3e7fc700 1 --2- 172.21.15.70:0/3574894346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 0x55dcb97ee150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.497+0000 7f2b3f7fe700 1 -- 172.21.15.70:0/3574894346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 msgr2=0x55dcb97ee150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:32.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.497+0000 7f2b3f7fe700 1 --2- 172.21.15.70:0/3574894346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 0x55dcb97ee150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.497+0000 7f2b3f7fe700 1 -- 172.21.15.70:0/3574894346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 msgr2=0x55dcb97f0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:32.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.497+0000 7f2b3f7fe700 1 --2- 172.21.15.70:0/3574894346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 0x55dcb97f0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.497+0000 7f2b3f7fe700 1 -- 172.21.15.70:0/3574894346 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcb96d2830 con 0x55dcb97ea350 2024-06-04T05:16:32.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.498+0000 7f2b3f7fe700 1 --2- 172.21.15.70:0/3574894346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 0x55dcb97ee690 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f2b4006b0a0 tx=0x7f2b38012580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15ca768c6730821e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:32.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.498+0000 7f2b22ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:32.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.498+0000 7f2b22ff5700 1 -- 172.21.15.70:0/3574894346 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b38012a60 con 0x55dcb97ea350 2024-06-04T05:16:32.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.498+0000 7f2b22ff5700 1 -- 172.21.15.70:0/3574894346 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f2b38012c00 con 0x55dcb97ea350 2024-06-04T05:16:32.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.498+0000 7f2b50059b80 1 -- 172.21.15.70:0/3574894346 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b300a9970 con 0x55dcb97ea350 2024-06-04T05:16:32.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.498+0000 7f2b22ff5700 1 -- 172.21.15.70:0/3574894346 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f2b38012e20 con 0x55dcb97ea350 2024-06-04T05:16:32.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.498+0000 7f2b50059b80 10 client.20433 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:32.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.498+0000 7f2b50059b80 1 -- 172.21.15.70:0/3574894346 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55dcb97fbea0 con 0x55dcb97ea350 2024-06-04T05:16:32.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.499+0000 7f2b22ff5700 1 -- 172.21.15.70:0/3574894346 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2b300a9970 con 0x55dcb97ea350 2024-06-04T05:16:32.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.499+0000 7f2b22ff5700 1 -- 172.21.15.70:0/3574894346 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f2b38025070 con 0x55dcb97ea350 2024-06-04T05:16:32.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.499+0000 7f2b50059b80 10 client.20433 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:32.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.499+0000 7f2b50059b80 10 client.20433 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:16:32.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.499+0000 7f2b50059b80 20 client.20433 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:32.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.499+0000 7f2b50059b80 1 --2- 172.21.15.70:0/3574894346 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dcb97fe990 0x55dcb9800db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.499+0000 7f2b50059b80 4 client.20433 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:16:32.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.499+0000 7f2b50059b80 1 -- 172.21.15.70:0/3574894346 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcb97b0b80 con 0x55dcb97fe990 2024-06-04T05:16:32.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.500+0000 7f2b3effd700 1 --2- 172.21.15.70:0/3574894346 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dcb97fe990 0x55dcb9800db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.500+0000 7f2b3effd700 1 --2- 172.21.15.70:0/3574894346 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dcb97fe990 0x55dcb9800db0 crc :-1 s=READY pgs=415 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9883ee93193a977a server_cookie=6306465355ce7acf in_seq=0 out_seq=0 2024-06-04T05:16:32.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.500+0000 7f2b22ff5700 10 client.20433 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:32.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.501+0000 7f2b22ff5700 1 -- 172.21.15.70:0/3574894346 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55dcb97b0b80 con 0x55dcb97fe990 2024-06-04T05:16:32.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.501+0000 7f2b22ff5700 10 client.20433 handle_command_reply: tid=0 2024-06-04T05:16:32.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.501+0000 7f2b50059b80 1 client.20433 shutdown 2024-06-04T05:16:32.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.501+0000 7f2b50059b80 1 -- 172.21.15.70:0/3574894346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 msgr2=0x55dcb97ee690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:32.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.501+0000 7f2b50059b80 1 --2- 172.21.15.70:0/3574894346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 0x55dcb97ee690 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f2b4006b0a0 tx=0x7f2b38012580 comp rx=0 tx=0).stop 2024-06-04T05:16:32.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.501+0000 7f2b50059b80 1 -- 172.21.15.70:0/3574894346 shutdown_connections 2024-06-04T05:16:32.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.501+0000 7f2b50059b80 1 --2- 172.21.15.70:0/3574894346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 0x55dcb97ee690 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.501+0000 7f2b50059b80 1 --2- 172.21.15.70:0/3574894346 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dcb97fe990 0x55dcb9800db0 crc :-1 s=READY pgs=415 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.501+0000 7f2b50059b80 1 --2- 172.21.15.70:0/3574894346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 0x55dcb97f0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.501+0000 7f2b50059b80 1 --2- 172.21.15.70:0/3574894346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 0x55dcb97ee150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.501+0000 7f2b50059b80 1 -- 172.21.15.70:0/3574894346 >> 172.21.15.70:0/3574894346 conn(0x55dcb9712220 msgr2=0x55dcb97d9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:32.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.501+0000 7f2b22ff5700 0 client.20433 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:32.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.501+0000 7f2b50059b80 1 -- 172.21.15.70:0/3574894346 shutdown_connections 2024-06-04T05:16:32.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.502+0000 7f2b50059b80 1 -- 172.21.15.70:0/3574894346 wait complete. 2024-06-04T05:16:32.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.502+0000 7f2b50059b80 20 client.20433 trim_cache size 0 max 16384 2024-06-04T05:16:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.532+0000 7f2b50059b80 1 Processor -- start 2024-06-04T05:16:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.532+0000 7f2b50059b80 1 -- start start 2024-06-04T05:16:32.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.533+0000 7f2b50059b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 0x55dcb97e8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.533+0000 7f2b50059b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 0x55dcb97f0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.533+0000 7f2b50059b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 0x55dcb98066a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.533+0000 7f2b3f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 0x55dcb98066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.533+0000 7f2b3f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 0x55dcb98066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37950/0 (socket says 172.21.15.70:37950) 2024-06-04T05:16:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.533+0000 7f2b3e7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 0x55dcb97f0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.533+0000 7f2b3f7fe700 1 -- 172.21.15.70:0/118688697 learned_addr learned my addr 172.21.15.70:0/118688697 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.533+0000 7f2b3effd700 1 --2- 172.21.15.70:0/118688697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 0x55dcb97e8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.533+0000 7f2b50059b80 1 -- 172.21.15.70:0/118688697 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2c00d6a0 con 0x55dcb97ea350 2024-06-04T05:16:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.533+0000 7f2b50059b80 1 -- 172.21.15.70:0/118688697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2c00f1e0 con 0x55dcb97e4980 2024-06-04T05:16:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.533+0000 7f2b50059b80 1 -- 172.21.15.70:0/118688697 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2c00dd70 con 0x55dcb97e5340 2024-06-04T05:16:32.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.534+0000 7f2b3f7fe700 1 -- 172.21.15.70:0/118688697 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 msgr2=0x55dcb97f0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:32.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.534+0000 7f2b3f7fe700 1 --2- 172.21.15.70:0/118688697 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 0x55dcb97f0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.534+0000 7f2b3f7fe700 1 -- 172.21.15.70:0/118688697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 msgr2=0x55dcb97e8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:32.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.534+0000 7f2b3f7fe700 1 --2- 172.21.15.70:0/118688697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 0x55dcb97e8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.534+0000 7f2b3f7fe700 1 -- 172.21.15.70:0/118688697 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcb97d7940 con 0x55dcb97ea350 2024-06-04T05:16:32.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.534+0000 7f2b3effd700 1 --2- 172.21.15.70:0/118688697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 0x55dcb97e8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:16:32.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.534+0000 7f2b3f7fe700 1 --2- 172.21.15.70:0/118688697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 0x55dcb98066a0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f2b38015010 tx=0x7f2b3801e590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f89a74d0e84613 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:32.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.534+0000 7f2b24ff9700 1 -- 172.21.15.70:0/118688697 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b3802b070 con 0x55dcb97ea350 2024-06-04T05:16:32.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.534+0000 7f2b24ff9700 1 -- 172.21.15.70:0/118688697 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2b3802d070 con 0x55dcb97ea350 2024-06-04T05:16:32.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.534+0000 7f2b24ff9700 1 -- 172.21.15.70:0/118688697 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b38003c60 con 0x55dcb97ea350 2024-06-04T05:16:32.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.535+0000 7f2b50059b80 1 -- 172.21.15.70:0/118688697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 msgr2=0x55dcb98066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.535+0000 7f2b50059b80 1 --2- 172.21.15.70:0/118688697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 0x55dcb98066a0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f2b38015010 tx=0x7f2b3801e590 comp rx=0 tx=0).stop 2024-06-04T05:16:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.535+0000 7f2b50059b80 1 -- 172.21.15.70:0/118688697 shutdown_connections 2024-06-04T05:16:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.535+0000 7f2b50059b80 1 --2- 172.21.15.70:0/118688697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 0x55dcb98066a0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.535+0000 7f2b50059b80 1 --2- 172.21.15.70:0/118688697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 0x55dcb97e8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.535+0000 7f2b50059b80 1 --2- 172.21.15.70:0/118688697 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 0x55dcb97f0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.536+0000 7f2b50059b80 1 -- 172.21.15.70:0/118688697 >> 172.21.15.70:0/118688697 conn(0x55dcb9712220 msgr2=0x55dcb97e7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.536+0000 7f2b50059b80 1 -- 172.21.15.70:0/118688697 shutdown_connections 2024-06-04T05:16:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.536+0000 7f2b50059b80 1 -- 172.21.15.70:0/118688697 wait complete. 2024-06-04T05:16:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.536+0000 7f2b50059b80 1 -- start start 2024-06-04T05:16:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.536+0000 7f2b50059b80 1 Processor -- start 2024-06-04T05:16:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.537+0000 7f2b50059b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 0x55dcb97f6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.537+0000 7f2b50059b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 0x55dcb97f6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.537+0000 7f2b50059b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 0x55dcb97f3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.537+0000 7f2b3f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 0x55dcb97f3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.537+0000 7f2b3f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 0x55dcb97f3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37958/0 (socket says 172.21.15.70:37958) 2024-06-04T05:16:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.537+0000 7f2b3f7fe700 1 -- 172.21.15.70:0/3712819425 learned_addr learned my addr 172.21.15.70:0/3712819425 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.537+0000 7f2b3effd700 1 --2- 172.21.15.70:0/3712819425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 0x55dcb97f6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.537+0000 7f2b3e7fc700 1 --2- 172.21.15.70:0/3712819425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 0x55dcb97f6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.537+0000 7f2b50059b80 1 -- 172.21.15.70:0/3712819425 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2c00eb10 con 0x55dcb97ea350 2024-06-04T05:16:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.537+0000 7f2b50059b80 1 -- 172.21.15.70:0/3712819425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2c00e440 con 0x55dcb97e4980 2024-06-04T05:16:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.538+0000 7f2b50059b80 1 -- 172.21.15.70:0/3712819425 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2c00b660 con 0x55dcb97e5340 2024-06-04T05:16:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.538+0000 7f2b50059b80 10 client.0 _ll_register_callbacks cb 0x55dcb96ceac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.538+0000 7f2b3f7fe700 1 -- 172.21.15.70:0/3712819425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 msgr2=0x55dcb97f6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.538+0000 7f2b3f7fe700 1 --2- 172.21.15.70:0/3712819425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 0x55dcb97f6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.538+0000 7f2b3f7fe700 1 -- 172.21.15.70:0/3712819425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 msgr2=0x55dcb97f6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.538+0000 7f2b3f7fe700 1 --2- 172.21.15.70:0/3712819425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 0x55dcb97f6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.538+0000 7f2b3f7fe700 1 -- 172.21.15.70:0/3712819425 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcb97f3fa0 con 0x55dcb97ea350 2024-06-04T05:16:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.538+0000 7f2b3effd700 1 --2- 172.21.15.70:0/3712819425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 0x55dcb97f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:16:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.538+0000 7f2b3f7fe700 1 --2- 172.21.15.70:0/3712819425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 0x55dcb97f3920 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f2b38009fd0 tx=0x7f2b38009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8642369eb153b27 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.538+0000 7f2b22ff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.538+0000 7f2b22ff5700 1 -- 172.21.15.70:0/3712819425 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b38037060 con 0x55dcb97ea350 2024-06-04T05:16:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.538+0000 7f2b50059b80 1 -- 172.21.15.70:0/3712819425 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b3802d070 con 0x55dcb97ea350 2024-06-04T05:16:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.538+0000 7f2b22ff5700 1 -- 172.21.15.70:0/3712819425 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f2b38003a00 con 0x55dcb97ea350 2024-06-04T05:16:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.539+0000 7f2b22ff5700 1 -- 172.21.15.70:0/3712819425 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f2b3802b040 con 0x55dcb97ea350 2024-06-04T05:16:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.539+0000 7f2b50059b80 10 client.20439 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.539+0000 7f2b50059b80 1 -- 172.21.15.70:0/3712819425 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55dcb9869ea0 con 0x55dcb97ea350 2024-06-04T05:16:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.539+0000 7f2b22ff5700 1 -- 172.21.15.70:0/3712819425 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2b3802d070 con 0x55dcb97ea350 2024-06-04T05:16:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.539+0000 7f2b22ff5700 1 -- 172.21.15.70:0/3712819425 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f2b38025040 con 0x55dcb97ea350 2024-06-04T05:16:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.539+0000 7f2b50059b80 10 client.20439 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.539+0000 7f2b50059b80 10 client.20439 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:16:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.539+0000 7f2b50059b80 20 client.20439 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.539+0000 7f2b50059b80 1 --2- 172.21.15.70:0/3712819425 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dcb986a4f0 0x55dcb986c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.539+0000 7f2b50059b80 4 client.20439 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 146.85, "await_for": 45.0}] 2024-06-04T05:16:32.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.539+0000 7f2b50059b80 1 -- 172.21.15.70:0/3712819425 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 146.85, "await_for": 45.0}) -- 0x55dcb9745d80 con 0x55dcb986a4f0 2024-06-04T05:16:32.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.540+0000 7f2b3effd700 1 --2- 172.21.15.70:0/3712819425 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dcb986a4f0 0x55dcb986c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:32.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.540+0000 7f2b3effd700 1 --2- 172.21.15.70:0/3712819425 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dcb986a4f0 0x55dcb986c910 crc :-1 s=READY pgs=416 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2deb732fc2014ca1 server_cookie=49845de87c5270b7 in_seq=0 out_seq=0 2024-06-04T05:16:32.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.540+0000 7f2b22ff5700 10 client.20439 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:32.548 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:16:32.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43426 7.11 MB/sec execute 204 sec latency 270.900 ms 2024-06-04T05:16:32.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.927+0000 7f2b22ff5700 1 -- 172.21.15.70:0/3712819425 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55dcb9745d80 con 0x55dcb986a4f0 2024-06-04T05:16:32.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.927+0000 7f2b22ff5700 10 client.20439 handle_command_reply: tid=0 2024-06-04T05:16:32.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.927+0000 7f2b50059b80 1 client.20439 shutdown 2024-06-04T05:16:32.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.928+0000 7f2b50059b80 1 -- 172.21.15.70:0/3712819425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 msgr2=0x55dcb97f3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:32.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.928+0000 7f2b50059b80 1 --2- 172.21.15.70:0/3712819425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 0x55dcb97f3920 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f2b38009fd0 tx=0x7f2b38009510 comp rx=0 tx=0).stop 2024-06-04T05:16:32.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.928+0000 7f2b50059b80 1 -- 172.21.15.70:0/3712819425 shutdown_connections 2024-06-04T05:16:32.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.928+0000 7f2b50059b80 1 --2- 172.21.15.70:0/3712819425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcb97ea350 0x55dcb97f3920 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.928+0000 7f2b50059b80 1 --2- 172.21.15.70:0/3712819425 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55dcb986a4f0 0x55dcb986c910 crc :-1 s=READY pgs=416 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.928+0000 7f2b22ff5700 0 client.20439 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:32.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.928+0000 7f2b50059b80 1 --2- 172.21.15.70:0/3712819425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcb97e4980 0x55dcb97f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.928+0000 7f2b50059b80 1 --2- 172.21.15.70:0/3712819425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dcb97e5340 0x55dcb97f6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.928+0000 7f2b50059b80 1 -- 172.21.15.70:0/3712819425 >> 172.21.15.70:0/3712819425 conn(0x55dcb9712220 msgr2=0x55dcb97e86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:32.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.928+0000 7f2b50059b80 1 -- 172.21.15.70:0/3712819425 shutdown_connections 2024-06-04T05:16:32.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.928+0000 7f2b50059b80 1 -- 172.21.15.70:0/3712819425 wait complete. 2024-06-04T05:16:32.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.928+0000 7f2b50059b80 20 client.20439 trim_cache size 0 max 16384 2024-06-04T05:16:32.931 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:16:32.932 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 57, 2024-06-04T05:16:32.932 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:16:32.932 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 62, 2024-06-04T05:16:32.932 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:16:32.932 INFO:teuthology.orchestra.run.smithi070.stdout: "49a0c915": { 2024-06-04T05:16:32.932 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 62, 2024-06-04T05:16:32.932 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:16:32.932 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:16:32.932 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:16:32.932 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:16:32.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:16:32.932 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:16:32.932 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 146.8, 2024-06-04T05:16:32.933 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:16:32.933 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:16:32.933 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:16:32.933 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:16:32.933 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:16:32.933 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:16:32.933 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:16:32.933 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:16:32.933 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:16:32.933 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:16:32.933 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:16:32.933 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:16:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.929+0000 7f2b457c4700 1 -- 172.21.15.70:0/4200324692 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2b280b2ba0 msgr2=0x7f2b280b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.929+0000 7f2b457c4700 1 --2- 172.21.15.70:0/4200324692 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2b280b2ba0 0x7f2b280b5060 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2b3400b2e0 tx=0x7f2b3400b610 comp rx=0 tx=0).stop 2024-06-04T05:16:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.929+0000 7f2b457c4700 1 -- 172.21.15.70:0/4200324692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b40102f70 msgr2=0x7f2b4010f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.929+0000 7f2b457c4700 1 --2- 172.21.15.70:0/4200324692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b40102f70 0x7f2b4010f7a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2b3000e9a0 tx=0x7f2b3000ed60 comp rx=0 tx=0).stop 2024-06-04T05:16:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.929+0000 7f2b457c4700 1 -- 172.21.15.70:0/4200324692 shutdown_connections 2024-06-04T05:16:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.929+0000 7f2b457c4700 1 --2- 172.21.15.70:0/4200324692 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2b280b2ba0 0x7f2b280b5060 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.929+0000 7f2b457c4700 1 --2- 172.21.15.70:0/4200324692 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b4010fff0 0x7f2b4010cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.929+0000 7f2b457c4700 1 --2- 172.21.15.70:0/4200324692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b40102f70 0x7f2b4010f7a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.929+0000 7f2b457c4700 1 --2- 172.21.15.70:0/4200324692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b40103890 0x7f2b4010c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:32.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.929+0000 7f2b457c4700 1 -- 172.21.15.70:0/4200324692 >> 172.21.15.70:0/4200324692 conn(0x7f2b400793e0 msgr2=0x7f2b40111300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:32.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.929+0000 7f2b457c4700 1 -- 172.21.15.70:0/4200324692 shutdown_connections 2024-06-04T05:16:32.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:32.929+0000 7f2b457c4700 1 -- 172.21.15.70:0/4200324692 wait complete. 2024-06-04T05:16:33.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:33 smithi070 ceph-mon[32631]: pgmap v608: 97 pgs: 97 active+clean; 705 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.6 MiB/s wr, 312 op/s 2024-06-04T05:16:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:33 smithi112 ceph-mon[36453]: pgmap v608: 97 pgs: 97 active+clean; 705 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.6 MiB/s wr, 312 op/s 2024-06-04T05:16:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:33 smithi098 ceph-mon[35569]: pgmap v608: 97 pgs: 97 active+clean; 705 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.6 MiB/s wr, 312 op/s 2024-06-04T05:16:33.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 7.08 MB/sec execute 205 sec latency 1246.156 ms 2024-06-04T05:16:34.426 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '49a0c915', quiesce took 15.0 seconds. Will release after: 11.850000000000001 2024-06-04T05:16:34.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.573+0000 7fb8c189c640 1 -- 172.21.15.70:0/3959264050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8bc101980 msgr2=0x7fb8bc101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.573+0000 7fb8c189c640 1 --2- 172.21.15.70:0/3959264050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8bc101980 0x7fb8bc101e00 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0009930 tx=0x7fb8b002a420 comp rx=0 tx=0).stop 2024-06-04T05:16:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.573+0000 7fb8c189c640 1 -- 172.21.15.70:0/3959264050 shutdown_connections 2024-06-04T05:16:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.573+0000 7fb8c189c640 1 --2- 172.21.15.70:0/3959264050 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8bc102340 0x7fb8bc10e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.573+0000 7fb8c189c640 1 --2- 172.21.15.70:0/3959264050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8bc101980 0x7fb8bc101e00 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.573+0000 7fb8c189c640 1 --2- 172.21.15.70:0/3959264050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8bc107940 0x7fb8bc107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.573+0000 7fb8c189c640 1 -- 172.21.15.70:0/3959264050 >> 172.21.15.70:0/3959264050 conn(0x7fb8bc078ad0 msgr2=0x7fb8bc078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.573+0000 7fb8c189c640 1 -- 172.21.15.70:0/3959264050 shutdown_connections 2024-06-04T05:16:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.573+0000 7fb8c189c640 1 -- 172.21.15.70:0/3959264050 wait complete. 2024-06-04T05:16:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.574+0000 7fb8c189c640 1 Processor -- start 2024-06-04T05:16:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.574+0000 7fb8c189c640 1 -- start start 2024-06-04T05:16:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.574+0000 7fb8c189c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8bc101980 0x7fb8bc1991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.574+0000 7fb8c189c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8bc102340 0x7fb8bc1996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.574+0000 7fb8c189c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8bc107940 0x7fb8bc19da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.574+0000 7fb8c189c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc111ab0 con 0x7fb8bc101980 2024-06-04T05:16:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.574+0000 7fb8c189c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc111930 con 0x7fb8bc102340 2024-06-04T05:16:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.574+0000 7fb8c189c640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc111c30 con 0x7fb8bc107940 2024-06-04T05:16:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.574+0000 7fb8baffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8bc101980 0x7fb8bc1991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.574+0000 7fb8baffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8bc101980 0x7fb8bc1991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37968/0 (socket says 172.21.15.70:37968) 2024-06-04T05:16:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.574+0000 7fb8baffd640 1 -- 172.21.15.70:0/141255023 learned_addr learned my addr 172.21.15.70:0/141255023 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.574+0000 7fb8ba7fc640 1 --2- 172.21.15.70:0/141255023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8bc102340 0x7fb8bc1996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.575+0000 7fb8baffd640 1 -- 172.21.15.70:0/141255023 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8bc107940 msgr2=0x7fb8bc19da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.575+0000 7fb8baffd640 1 --2- 172.21.15.70:0/141255023 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8bc107940 0x7fb8bc19da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.575+0000 7fb8baffd640 1 -- 172.21.15.70:0/141255023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8bc102340 msgr2=0x7fb8bc1996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.575+0000 7fb8baffd640 1 --2- 172.21.15.70:0/141255023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8bc102340 0x7fb8bc1996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.575+0000 7fb8baffd640 1 -- 172.21.15.70:0/141255023 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8bc19e190 con 0x7fb8bc101980 2024-06-04T05:16:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.575+0000 7fb8baffd640 1 --2- 172.21.15.70:0/141255023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8bc101980 0x7fb8bc1991a0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00c890 tx=0x7fb8ac00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.575+0000 7fb8c089a640 1 -- 172.21.15.70:0/141255023 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac004220 con 0x7fb8bc101980 2024-06-04T05:16:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.575+0000 7fb8c089a640 1 -- 172.21.15.70:0/141255023 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb8ac007e40 con 0x7fb8bc101980 2024-06-04T05:16:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.575+0000 7fb8c189c640 1 -- 172.21.15.70:0/141255023 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8bc19e480 con 0x7fb8bc101980 2024-06-04T05:16:34.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.575+0000 7fb8c089a640 1 -- 172.21.15.70:0/141255023 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac004a00 con 0x7fb8bc101980 2024-06-04T05:16:34.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.575+0000 7fb8c189c640 1 -- 172.21.15.70:0/141255023 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8bc1a5d10 con 0x7fb8bc101980 2024-06-04T05:16:34.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.578+0000 7fb8c089a640 1 -- 172.21.15.70:0/141255023 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fb8ac0079b0 con 0x7fb8bc101980 2024-06-04T05:16:34.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.578+0000 7fb8c089a640 1 --2- 172.21.15.70:0/141255023 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb8a40b2a10 0x7fb8a40b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.578+0000 7fb8c089a640 1 -- 172.21.15.70:0/141255023 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7fb8ac0e2390 con 0x7fb8bc101980 2024-06-04T05:16:34.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.578+0000 7fb8ba7fc640 1 --2- 172.21.15.70:0/141255023 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb8a40b2a10 0x7fb8a40b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.579+0000 7fb8ba7fc640 1 --2- 172.21.15.70:0/141255023 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb8a40b2a10 0x7fb8a40b4ed0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0005ec0 tx=0x7fb8b00058d0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8c45b7740 1 Processor -- start 2024-06-04T05:16:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8c45b7740 1 -- start start 2024-06-04T05:16:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8c45b7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb7e820 0x55fd8eb7ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8c45b7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7f140 0x55fd8eb894c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8c45b7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb89a00 0x55fd8eb8bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8c45b7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fd8eaf0c30 con 0x55fd8eb89a00 2024-06-04T05:16:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8c45b7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fd8eae07d0 con 0x55fd8eb7f140 2024-06-04T05:16:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8c45b7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fd8eae4810 con 0x55fd8eb7e820 2024-06-04T05:16:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8bb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb89a00 0x55fd8eb8bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8bb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb89a00 0x55fd8eb8bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37984/0 (socket says 172.21.15.70:37984) 2024-06-04T05:16:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8bb7fe640 1 -- 172.21.15.70:0/1862555252 learned_addr learned my addr 172.21.15.70:0/1862555252 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8ba7fc640 1 --2- 172.21.15.70:0/1862555252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7f140 0x55fd8eb894c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8baffd640 1 --2- 172.21.15.70:0/1862555252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb7e820 0x55fd8eb7ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8bb7fe640 1 -- 172.21.15.70:0/1862555252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb7e820 msgr2=0x55fd8eb7ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8bb7fe640 1 --2- 172.21.15.70:0/1862555252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb7e820 0x55fd8eb7ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8bb7fe640 1 -- 172.21.15.70:0/1862555252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7f140 msgr2=0x55fd8eb894c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8bb7fe640 1 --2- 172.21.15.70:0/1862555252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7f140 0x55fd8eb894c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8bb7fe640 1 -- 172.21.15.70:0/1862555252 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd8e7f5210 con 0x55fd8eb89a00 2024-06-04T05:16:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8bb7fe640 1 --2- 172.21.15.70:0/1862555252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb89a00 0x55fd8eb8bdf0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009d00 tx=0x7fb8b400b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c524eac160cff97f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.585+0000 7fb8a1ffb640 1 -- 172.21.15.70:0/1862555252 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b400bd40 con 0x55fd8eb89a00 2024-06-04T05:16:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.586+0000 7fb8a1ffb640 1 -- 172.21.15.70:0/1862555252 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb8b4017070 con 0x55fd8eb89a00 2024-06-04T05:16:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.586+0000 7fb8a1ffb640 1 -- 172.21.15.70:0/1862555252 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b40115f0 con 0x55fd8eb89a00 2024-06-04T05:16:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.586+0000 7fb8c45b7740 1 -- 172.21.15.70:0/1862555252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb89a00 msgr2=0x55fd8eb8bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.586+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/1862555252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb89a00 0x55fd8eb8bdf0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009d00 tx=0x7fb8b400b420 comp rx=0 tx=0).stop 2024-06-04T05:16:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.586+0000 7fb8c45b7740 1 -- 172.21.15.70:0/1862555252 shutdown_connections 2024-06-04T05:16:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.586+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/1862555252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7f140 0x55fd8eb894c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.586+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/1862555252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb89a00 0x55fd8eb8bdf0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.586+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/1862555252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb7e820 0x55fd8eb7ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.586+0000 7fb8c45b7740 1 -- 172.21.15.70:0/1862555252 >> 172.21.15.70:0/1862555252 conn(0x55fd8eb7ad30 msgr2=0x55fd8eb7b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.586+0000 7fb8c45b7740 1 -- 172.21.15.70:0/1862555252 shutdown_connections 2024-06-04T05:16:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.586+0000 7fb8c45b7740 1 -- 172.21.15.70:0/1862555252 wait complete. 2024-06-04T05:16:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.587+0000 7fb8c45b7740 1 -- start start 2024-06-04T05:16:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.587+0000 7fb8c45b7740 1 Processor -- start 2024-06-04T05:16:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.587+0000 7fb8c45b7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7e820 0x55fd8eb86970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.587+0000 7fb8c45b7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb7f140 0x55fd8eb86eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.587+0000 7fb8c45b7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb89a00 0x55fd8eb873f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.587+0000 7fb8c45b7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fd8e7c6790 con 0x55fd8eb7f140 2024-06-04T05:16:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.587+0000 7fb8c45b7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fd8ea9c890 con 0x55fd8eb7e820 2024-06-04T05:16:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.587+0000 7fb8c45b7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fd8eb8c630 con 0x55fd8eb89a00 2024-06-04T05:16:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.587+0000 7fb8c45b7740 10 client.0 _ll_register_callbacks cb 0x55fd8e7f4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.587+0000 7fb8ba7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb7f140 0x55fd8eb86eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.587+0000 7fb8ba7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb7f140 0x55fd8eb86eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37986/0 (socket says 172.21.15.70:37986) 2024-06-04T05:16:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.587+0000 7fb8ba7fc640 1 -- 172.21.15.70:0/1535529826 learned_addr learned my addr 172.21.15.70:0/1535529826 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.587+0000 7fb8baffd640 1 --2- 172.21.15.70:0/1535529826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7e820 0x55fd8eb86970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb8ba7fc640 1 -- 172.21.15.70:0/1535529826 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb89a00 msgr2=0x55fd8eb873f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb8ba7fc640 1 --2- 172.21.15.70:0/1535529826 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb89a00 0x55fd8eb873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb8ba7fc640 1 -- 172.21.15.70:0/1535529826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7e820 msgr2=0x55fd8eb86970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb8ba7fc640 1 --2- 172.21.15.70:0/1535529826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7e820 0x55fd8eb86970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb8ba7fc640 1 -- 172.21.15.70:0/1535529826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd8e8c1620 con 0x55fd8eb7f140 2024-06-04T05:16:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb8ba7fc640 1 --2- 172.21.15.70:0/1535529826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb7f140 0x55fd8eb86eb0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0006010 tx=0x7fb8b002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43abf27f8ccca35e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb89affd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb89affd640 1 -- 172.21.15.70:0/1535529826 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0035050 con 0x55fd8eb7f140 2024-06-04T05:16:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb89affd640 1 -- 172.21.15.70:0/1535529826 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7fb8b0003970 con 0x55fd8eb7f140 2024-06-04T05:16:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb8c45b7740 1 -- 172.21.15.70:0/1535529826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8b4017070 con 0x55fd8eb7f140 2024-06-04T05:16:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb89affd640 1 -- 172.21.15.70:0/1535529826 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fb8b0003b70 con 0x55fd8eb7f140 2024-06-04T05:16:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb8c45b7740 10 client.20451 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb8c45b7740 1 -- 172.21.15.70:0/1535529826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55fd8eb98360 con 0x55fd8eb7f140 2024-06-04T05:16:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb89affd640 1 -- 172.21.15.70:0/1535529826 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb8b4017070 con 0x55fd8eb7f140 2024-06-04T05:16:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb89affd640 1 -- 172.21.15.70:0/1535529826 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fb8b003c440 con 0x55fd8eb7f140 2024-06-04T05:16:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb8c45b7740 10 client.20451 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb8c45b7740 10 client.20451 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:16:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb8c45b7740 20 client.20451 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/1535529826 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fd8eb99870 0x55fd8eb9bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb8c45b7740 4 client.20451 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:16:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.588+0000 7fb8c45b7740 1 -- 172.21.15.70:0/1535529826 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd8eaf2b80 con 0x55fd8eb99870 2024-06-04T05:16:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.589+0000 7fb8baffd640 1 --2- 172.21.15.70:0/1535529826 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fd8eb99870 0x55fd8eb9bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.589+0000 7fb8baffd640 1 --2- 172.21.15.70:0/1535529826 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fd8eb99870 0x55fd8eb9bc60 crc :-1 s=READY pgs=417 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39e32a4ffa51e9cd server_cookie=ca7b6eb53157ff81 in_seq=0 out_seq=0 2024-06-04T05:16:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.589+0000 7fb89affd640 10 client.20451 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.590+0000 7fb89affd640 1 -- 172.21.15.70:0/1535529826 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fd8eaf2b80 con 0x55fd8eb99870 2024-06-04T05:16:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.590+0000 7fb89affd640 10 client.20451 handle_command_reply: tid=0 2024-06-04T05:16:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.590+0000 7fb8c45b7740 1 client.20451 shutdown 2024-06-04T05:16:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.590+0000 7fb8c45b7740 1 -- 172.21.15.70:0/1535529826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb7f140 msgr2=0x55fd8eb86eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.590+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/1535529826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb7f140 0x55fd8eb86eb0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0006010 tx=0x7fb8b002a420 comp rx=0 tx=0).stop 2024-06-04T05:16:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.590+0000 7fb8c45b7740 1 -- 172.21.15.70:0/1535529826 shutdown_connections 2024-06-04T05:16:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.590+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/1535529826 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fd8eb99870 0x55fd8eb9bc60 crc :-1 s=READY pgs=417 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.590+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/1535529826 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb89a00 0x55fd8eb873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.590+0000 7fb89affd640 0 client.20451 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.590+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/1535529826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb7f140 0x55fd8eb86eb0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.590+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/1535529826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7e820 0x55fd8eb86970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.590+0000 7fb8c45b7740 1 -- 172.21.15.70:0/1535529826 >> 172.21.15.70:0/1535529826 conn(0x55fd8eb7ad30 msgr2=0x55fd8eb79fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.590+0000 7fb8c45b7740 1 -- 172.21.15.70:0/1535529826 shutdown_connections 2024-06-04T05:16:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.590+0000 7fb8c45b7740 1 -- 172.21.15.70:0/1535529826 wait complete. 2024-06-04T05:16:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.590+0000 7fb8c45b7740 20 client.20451 trim_cache size 0 max 16384 2024-06-04T05:16:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.605+0000 7fb8c45b7740 1 Processor -- start 2024-06-04T05:16:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.605+0000 7fb8c45b7740 1 -- start start 2024-06-04T05:16:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.605+0000 7fb8c45b7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb7e820 0x55fd8eba1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.605+0000 7fb8c45b7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7f140 0x55fd8eba1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.605+0000 7fb8c45b7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb89a00 0x55fd8eb83bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.605+0000 7fb8c45b7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1ae710 con 0x55fd8eb7e820 2024-06-04T05:16:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.605+0000 7fb8c45b7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1b0250 con 0x55fd8eb7f140 2024-06-04T05:16:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.605+0000 7fb8c45b7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1aede0 con 0x55fd8eb89a00 2024-06-04T05:16:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.605+0000 7fb8baffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb7e820 0x55fd8eba1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.605+0000 7fb8baffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb7e820 0x55fd8eba1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37994/0 (socket says 172.21.15.70:37994) 2024-06-04T05:16:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.605+0000 7fb8baffd640 1 -- 172.21.15.70:0/3014213781 learned_addr learned my addr 172.21.15.70:0/3014213781 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.605+0000 7fb8baffd640 1 -- 172.21.15.70:0/3014213781 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb89a00 msgr2=0x55fd8eb83bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.605+0000 7fb8baffd640 1 --2- 172.21.15.70:0/3014213781 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb89a00 0x55fd8eb83bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.605+0000 7fb8baffd640 1 -- 172.21.15.70:0/3014213781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7f140 msgr2=0x55fd8eba1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.605+0000 7fb8baffd640 1 --2- 172.21.15.70:0/3014213781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7f140 0x55fd8eba1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.605+0000 7fb8baffd640 1 -- 172.21.15.70:0/3014213781 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd8eb7ab60 con 0x55fd8eb7e820 2024-06-04T05:16:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.606+0000 7fb8baffd640 1 --2- 172.21.15.70:0/3014213781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb7e820 0x55fd8eba1570 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0a9100 tx=0x7fb8ac00a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1214d42062924718 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.606+0000 7fb8a17fa640 1 -- 172.21.15.70:0/3014213781 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac01d040 con 0x55fd8eb7e820 2024-06-04T05:16:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.606+0000 7fb8a17fa640 1 -- 172.21.15.70:0/3014213781 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb8ac0ad9f0 con 0x55fd8eb7e820 2024-06-04T05:16:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.606+0000 7fb8a17fa640 1 -- 172.21.15.70:0/3014213781 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac091070 con 0x55fd8eb7e820 2024-06-04T05:16:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.606+0000 7fb8c45b7740 1 -- 172.21.15.70:0/3014213781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb7e820 msgr2=0x55fd8eba1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.606+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/3014213781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb7e820 0x55fd8eba1570 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0a9100 tx=0x7fb8ac00a3f0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.606+0000 7fb8c45b7740 1 -- 172.21.15.70:0/3014213781 shutdown_connections 2024-06-04T05:16:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.606+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/3014213781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7f140 0x55fd8eba1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.606+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/3014213781 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb89a00 0x55fd8eb83bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.606+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/3014213781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb7e820 0x55fd8eba1570 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.606+0000 7fb8c45b7740 1 -- 172.21.15.70:0/3014213781 >> 172.21.15.70:0/3014213781 conn(0x55fd8eb7ad30 msgr2=0x55fd8eb80a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.606+0000 7fb8c45b7740 1 -- 172.21.15.70:0/3014213781 shutdown_connections 2024-06-04T05:16:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.606+0000 7fb8c45b7740 1 -- 172.21.15.70:0/3014213781 wait complete. 2024-06-04T05:16:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.607+0000 7fb8c45b7740 1 -- start start 2024-06-04T05:16:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.607+0000 7fb8c45b7740 1 Processor -- start 2024-06-04T05:16:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.607+0000 7fb8c45b7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7e820 0x55fd8eb91a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.607+0000 7fb8c45b7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb7f140 0x55fd8eb91fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.607+0000 7fb8c45b7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb89a00 0x55fd8eb8e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.607+0000 7fb8c45b7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1afb80 con 0x55fd8eb89a00 2024-06-04T05:16:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.607+0000 7fb8c45b7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1af4b0 con 0x55fd8eb7e820 2024-06-04T05:16:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.607+0000 7fb8c45b7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1ac710 con 0x55fd8eb7f140 2024-06-04T05:16:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.607+0000 7fb8c45b7740 10 client.0 _ll_register_callbacks cb 0x7fb8b0031400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.607+0000 7fb8bb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb89a00 0x55fd8eb8e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.607+0000 7fb8bb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb89a00 0x55fd8eb8e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38008/0 (socket says 172.21.15.70:38008) 2024-06-04T05:16:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.607+0000 7fb8bb7fe640 1 -- 172.21.15.70:0/456597180 learned_addr learned my addr 172.21.15.70:0/456597180 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.607+0000 7fb8ba7fc640 1 --2- 172.21.15.70:0/456597180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb7f140 0x55fd8eb91fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb8baffd640 1 --2- 172.21.15.70:0/456597180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7e820 0x55fd8eb91a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb8bb7fe640 1 -- 172.21.15.70:0/456597180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb7f140 msgr2=0x55fd8eb91fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb8bb7fe640 1 --2- 172.21.15.70:0/456597180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb7f140 0x55fd8eb91fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb8bb7fe640 1 -- 172.21.15.70:0/456597180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7e820 msgr2=0x55fd8eb91a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb8bb7fe640 1 --2- 172.21.15.70:0/456597180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7e820 0x55fd8eb91a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb8bb7fe640 1 -- 172.21.15.70:0/456597180 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd8eb8ef90 con 0x55fd8eb89a00 2024-06-04T05:16:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb8baffd640 1 --2- 172.21.15.70:0/456597180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7e820 0x55fd8eb91a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:16:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb8bb7fe640 1 --2- 172.21.15.70:0/456597180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb89a00 0x55fd8eb8e850 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fb8b40095b0 tx=0x7fb8b4012660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c892a8b706bdc820 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb89affd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb89affd640 1 -- 172.21.15.70:0/456597180 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4021070 con 0x55fd8eb89a00 2024-06-04T05:16:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb89affd640 1 -- 172.21.15.70:0/456597180 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7fb8b4012be0 con 0x55fd8eb89a00 2024-06-04T05:16:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb8c45b7740 1 -- 172.21.15.70:0/456597180 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8ac0ad9f0 con 0x55fd8eb89a00 2024-06-04T05:16:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb89affd640 1 -- 172.21.15.70:0/456597180 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fb8b4012e60 con 0x55fd8eb89a00 2024-06-04T05:16:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb8c45b7740 10 client.20457 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb8c45b7740 1 -- 172.21.15.70:0/456597180 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55fd8ec04e00 con 0x55fd8eb89a00 2024-06-04T05:16:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb89affd640 1 -- 172.21.15.70:0/456597180 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb8ac0ad9f0 con 0x55fd8eb89a00 2024-06-04T05:16:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb89affd640 1 -- 172.21.15.70:0/456597180 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fb8b4011070 con 0x55fd8eb89a00 2024-06-04T05:16:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb8c45b7740 10 client.20457 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb8c45b7740 10 client.20457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:16:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb8c45b7740 20 client.20457 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/456597180 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fd8ec052c0 0x55fd8ec076e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb8c45b7740 4 client.20457 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:16:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.608+0000 7fb8c45b7740 1 -- 172.21.15.70:0/456597180 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fd8eaec360 con 0x55fd8ec052c0 2024-06-04T05:16:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.609+0000 7fb8baffd640 1 --2- 172.21.15.70:0/456597180 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fd8ec052c0 0x55fd8ec076e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.609+0000 7fb89affd640 10 client.20457 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.609+0000 7fb8baffd640 1 --2- 172.21.15.70:0/456597180 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fd8ec052c0 0x55fd8ec076e0 crc :-1 s=READY pgs=418 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a36e8d3ed03e8465 server_cookie=90faf49753b69a8d in_seq=0 out_seq=0 2024-06-04T05:16:34.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.679+0000 7fb89affd640 1 -- 172.21.15.70:0/456597180 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55fd8eaec360 con 0x55fd8ec052c0 2024-06-04T05:16:34.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.679+0000 7fb89affd640 10 client.20457 handle_command_reply: tid=0 2024-06-04T05:16:34.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.679+0000 7fb8c45b7740 1 client.20457 shutdown 2024-06-04T05:16:34.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c45b7740 1 -- 172.21.15.70:0/456597180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb89a00 msgr2=0x55fd8eb8e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:34.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/456597180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb89a00 0x55fd8eb8e850 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fb8b40095b0 tx=0x7fb8b4012660 comp rx=0 tx=0).stop 2024-06-04T05:16:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c45b7740 1 -- 172.21.15.70:0/456597180 shutdown_connections 2024-06-04T05:16:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/456597180 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fd8ec052c0 0x55fd8ec076e0 crc :-1 s=READY pgs=418 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/456597180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd8eb89a00 0x55fd8eb8e850 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/456597180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fd8eb7f140 0x55fd8eb91fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c45b7740 1 --2- 172.21.15.70:0/456597180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd8eb7e820 0x55fd8eb91a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb89affd640 0 client.20457 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c45b7740 1 -- 172.21.15.70:0/456597180 >> 172.21.15.70:0/456597180 conn(0x55fd8eb7ad30 msgr2=0x55fd8eb86930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c45b7740 1 -- 172.21.15.70:0/456597180 shutdown_connections 2024-06-04T05:16:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c45b7740 1 -- 172.21.15.70:0/456597180 wait complete. 2024-06-04T05:16:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c45b7740 20 client.20457 trim_cache size 0 max 16384 2024-06-04T05:16:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:16:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:16:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:16:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c189c640 1 -- 172.21.15.70:0/141255023 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb8a40b2a10 msgr2=0x7fb8a40b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c189c640 1 --2- 172.21.15.70:0/141255023 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb8a40b2a10 0x7fb8a40b4ed0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0005ec0 tx=0x7fb8b00058d0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c189c640 1 -- 172.21.15.70:0/141255023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8bc101980 msgr2=0x7fb8bc1991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c189c640 1 --2- 172.21.15.70:0/141255023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8bc101980 0x7fb8bc1991a0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00c890 tx=0x7fb8ac00cd50 comp rx=0 tx=0).stop 2024-06-04T05:16:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c189c640 1 -- 172.21.15.70:0/141255023 shutdown_connections 2024-06-04T05:16:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c189c640 1 --2- 172.21.15.70:0/141255023 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb8a40b2a10 0x7fb8a40b4ed0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c189c640 1 --2- 172.21.15.70:0/141255023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8bc102340 0x7fb8bc1996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c189c640 1 --2- 172.21.15.70:0/141255023 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8bc107940 0x7fb8bc19da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c189c640 1 --2- 172.21.15.70:0/141255023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8bc101980 0x7fb8bc1991a0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.680+0000 7fb8c189c640 1 -- 172.21.15.70:0/141255023 >> 172.21.15.70:0/141255023 conn(0x7fb8bc078ad0 msgr2=0x7fb8bc06a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.681+0000 7fb8c189c640 1 -- 172.21.15.70:0/141255023 shutdown_connections 2024-06-04T05:16:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:34.681+0000 7fb8c189c640 1 -- 172.21.15.70:0/141255023 wait complete. 2024-06-04T05:16:34.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 7.05 MB/sec execute 206 sec latency 2246.202 ms 2024-06-04T05:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.937+0000 7f3b31871700 1 -- 172.21.15.70:0/3087909893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b2c1084b0 msgr2=0x7f3b2c108890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.937+0000 7f3b31871700 1 --2- 172.21.15.70:0/3087909893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b2c1084b0 0x7f3b2c108890 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009990 tx=0x7f3b1c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.937+0000 7f3b31871700 1 -- 172.21.15.70:0/3087909893 shutdown_connections 2024-06-04T05:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.937+0000 7f3b31871700 1 --2- 172.21.15.70:0/3087909893 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b2c102e30 0x7f3b2c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.937+0000 7f3b31871700 1 --2- 172.21.15.70:0/3087909893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b2c1084b0 0x7f3b2c108890 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.937+0000 7f3b31871700 1 --2- 172.21.15.70:0/3087909893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b2c102470 0x7f3b2c1028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.937+0000 7f3b31871700 1 -- 172.21.15.70:0/3087909893 >> 172.21.15.70:0/3087909893 conn(0x7f3b2c0793f0 msgr2=0x7f3b2c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.937+0000 7f3b31871700 1 -- 172.21.15.70:0/3087909893 shutdown_connections 2024-06-04T05:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.937+0000 7f3b31871700 1 -- 172.21.15.70:0/3087909893 wait complete. 2024-06-04T05:16:34.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.937+0000 7f3b31871700 1 Processor -- start 2024-06-04T05:16:34.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.938+0000 7f3b31871700 1 -- start start 2024-06-04T05:16:34.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.938+0000 7f3b31871700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b2c102470 0x7f3b2c104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.938+0000 7f3b31871700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b2c102e30 0x7f3b2c1045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.938+0000 7f3b31871700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b2c1084b0 0x7f3b2c104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.938+0000 7f3b31871700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c112570 con 0x7f3b2c102470 2024-06-04T05:16:34.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.938+0000 7f3b31871700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c1123f0 con 0x7f3b2c1084b0 2024-06-04T05:16:34.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.938+0000 7f3b31871700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c1126f0 con 0x7f3b2c102e30 2024-06-04T05:16:34.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.938+0000 7f3b2affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b2c102470 0x7f3b2c104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.938+0000 7f3b2affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b2c102470 0x7f3b2c104080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38032/0 (socket says 172.21.15.70:38032) 2024-06-04T05:16:34.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.938+0000 7f3b2affd700 1 -- 172.21.15.70:0/1164433738 learned_addr learned my addr 172.21.15.70:0/1164433738 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:34.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.938+0000 7f3b2a7fc700 1 --2- 172.21.15.70:0/1164433738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b2c102e30 0x7f3b2c1045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.938+0000 7f3b2b7fe700 1 --2- 172.21.15.70:0/1164433738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b2c1084b0 0x7f3b2c104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.938+0000 7f3b2affd700 1 -- 172.21.15.70:0/1164433738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b2c102e30 msgr2=0x7f3b2c1045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:34.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.938+0000 7f3b2affd700 1 --2- 172.21.15.70:0/1164433738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b2c102e30 0x7f3b2c1045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.938+0000 7f3b2affd700 1 -- 172.21.15.70:0/1164433738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b2c1084b0 msgr2=0x7f3b2c104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:34.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.938+0000 7f3b2affd700 1 --2- 172.21.15.70:0/1164433738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b2c1084b0 0x7f3b2c104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:34.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.938+0000 7f3b2affd700 1 -- 172.21.15.70:0/1164433738 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2c1a81a0 con 0x7f3b2c102470 2024-06-04T05:16:34.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.939+0000 7f3b2affd700 1 --2- 172.21.15.70:0/1164433738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b2c102470 0x7f3b2c104080 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c000c00 tx=0x7f3b1c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:34.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.939+0000 7f3b3086f700 1 -- 172.21.15.70:0/1164433738 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c027070 con 0x7f3b2c102470 2024-06-04T05:16:34.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.939+0000 7f3b3086f700 1 -- 172.21.15.70:0/1164433738 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b1c00fdb0 con 0x7f3b2c102470 2024-06-04T05:16:34.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.939+0000 7f3b31871700 1 -- 172.21.15.70:0/1164433738 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b2c1a8430 con 0x7f3b2c102470 2024-06-04T05:16:34.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.939+0000 7f3b3086f700 1 -- 172.21.15.70:0/1164433738 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c0173f0 con 0x7f3b2c102470 2024-06-04T05:16:34.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.939+0000 7f3b31871700 1 -- 172.21.15.70:0/1164433738 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b2c1a8880 con 0x7f3b2c102470 2024-06-04T05:16:34.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.940+0000 7f3b31871700 1 -- 172.21.15.70:0/1164433738 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b2c06a230 con 0x7f3b2c102470 2024-06-04T05:16:34.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.941+0000 7f3b3086f700 1 -- 172.21.15.70:0/1164433738 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3b1c02a070 con 0x7f3b2c102470 2024-06-04T05:16:34.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.942+0000 7f3b3086f700 1 --2- 172.21.15.70:0/1164433738 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3b140b29f0 0x7f3b140b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:34.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.942+0000 7f3b2a7fc700 1 --2- 172.21.15.70:0/1164433738 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3b140b29f0 0x7f3b140b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:34.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.942+0000 7f3b3086f700 1 -- 172.21.15.70:0/1164433738 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f3b1c0e3c00 con 0x7f3b2c102470 2024-06-04T05:16:34.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.942+0000 7f3b2a7fc700 1 --2- 172.21.15.70:0/1164433738 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3b140b29f0 0x7f3b140b4eb0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3b20006fd0 tx=0x7f3b20008040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:34.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:34.945+0000 7f3b3086f700 1 -- 172.21.15.70:0/1164433738 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3b1c031070 con 0x7f3b2c102470 2024-06-04T05:16:35.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:35.207+0000 7f3b31871700 1 -- 172.21.15.70:0/1164433738 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b2c02e720 con 0x7f3b2c102470 2024-06-04T05:16:35.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:35.207+0000 7f3b3086f700 1 -- 172.21.15.70:0/1164433738 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+11929 (secure 0 0 0) 0x7f3b1c013070 con 0x7f3b2c102470 2024-06-04T05:16:35.210 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:16:35.210 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":57,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:15:55.006036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T05:16:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:35.210+0000 7f3b31871700 1 -- 172.21.15.70:0/1164433738 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3b140b29f0 msgr2=0x7f3b140b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:35.210+0000 7f3b31871700 1 --2- 172.21.15.70:0/1164433738 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3b140b29f0 0x7f3b140b4eb0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3b20006fd0 tx=0x7f3b20008040 comp rx=0 tx=0).stop 2024-06-04T05:16:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:35.210+0000 7f3b31871700 1 -- 172.21.15.70:0/1164433738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b2c102470 msgr2=0x7f3b2c104080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:35.210+0000 7f3b31871700 1 --2- 172.21.15.70:0/1164433738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b2c102470 0x7f3b2c104080 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c000c00 tx=0x7f3b1c0047b0 comp rx=0 tx=0).stop 2024-06-04T05:16:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:35.210+0000 7f3b31871700 1 -- 172.21.15.70:0/1164433738 shutdown_connections 2024-06-04T05:16:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:35.210+0000 7f3b31871700 1 --2- 172.21.15.70:0/1164433738 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3b140b29f0 0x7f3b140b4eb0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:35.210+0000 7f3b31871700 1 --2- 172.21.15.70:0/1164433738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b2c1084b0 0x7f3b2c104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:35.210+0000 7f3b31871700 1 --2- 172.21.15.70:0/1164433738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b2c102470 0x7f3b2c104080 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:35.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:35.210+0000 7f3b31871700 1 --2- 172.21.15.70:0/1164433738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b2c102e30 0x7f3b2c1045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:35.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:35.210+0000 7f3b31871700 1 -- 172.21.15.70:0/1164433738 >> 172.21.15.70:0/1164433738 conn(0x7f3b2c0793f0 msgr2=0x7f3b2c06c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:35.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:35.210+0000 7f3b31871700 1 -- 172.21.15.70:0/1164433738 shutdown_connections 2024-06-04T05:16:35.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:35.210+0000 7f3b31871700 1 -- 172.21.15.70:0/1164433738 wait complete. 2024-06-04T05:16:35.213 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 57 2024-06-04T05:16:35.574 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:35 smithi070 ceph-mon[32631]: pgmap v609: 97 pgs: 97 active+clean; 705 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 252 op/s 2024-06-04T05:16:35.574 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:35 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1164433738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:16:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:35 smithi112 ceph-mon[36453]: pgmap v609: 97 pgs: 97 active+clean; 705 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 252 op/s 2024-06-04T05:16:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:35 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1164433738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:16:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:35 smithi098 ceph-mon[35569]: pgmap v609: 97 pgs: 97 active+clean; 705 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 252 op/s 2024-06-04T05:16:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:35 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1164433738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:16:35.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 7.01 MB/sec execute 207 sec latency 3246.289 ms 2024-06-04T05:16:35.918 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:16:36.165 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:16:36.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.98 MB/sec execute 208 sec latency 4246.400 ms 2024-06-04T05:16:37.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:37 smithi070 ceph-mon[32631]: pgmap v610: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 357 op/s 2024-06-04T05:16:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:37 smithi112 ceph-mon[36453]: pgmap v610: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 357 op/s 2024-06-04T05:16:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:37 smithi098 ceph-mon[35569]: pgmap v610: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 357 op/s 2024-06-04T05:16:37.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.94 MB/sec execute 209 sec latency 5246.472 ms 2024-06-04T05:16:38.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.855+0000 7f6685f43700 1 -- 172.21.15.70:0/3238104476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6680108600 msgr2=0x7f66801089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:38.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.855+0000 7f6685f43700 1 --2- 172.21.15.70:0/3238104476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6680108600 0x7f66801089e0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f6670009990 tx=0x7f6670009ca0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.855+0000 7f6685f43700 1 -- 172.21.15.70:0/3238104476 shutdown_connections 2024-06-04T05:16:38.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.855+0000 7f6685f43700 1 --2- 172.21.15.70:0/3238104476 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66801028a0 0x7f6680110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.855+0000 7f6685f43700 1 --2- 172.21.15.70:0/3238104476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6680108600 0x7f66801089e0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.855+0000 7f6685f43700 1 --2- 172.21.15.70:0/3238104476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6680101ee0 0x7f6680102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.855+0000 7f6685f43700 1 -- 172.21.15.70:0/3238104476 >> 172.21.15.70:0/3238104476 conn(0x7f6680077760 msgr2=0x7f6680077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:38.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.855+0000 7f6685f43700 1 -- 172.21.15.70:0/3238104476 shutdown_connections 2024-06-04T05:16:38.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.855+0000 7f6685f43700 1 -- 172.21.15.70:0/3238104476 wait complete. 2024-06-04T05:16:38.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.856+0000 7f6685f43700 1 Processor -- start 2024-06-04T05:16:38.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.856+0000 7f6685f43700 1 -- start start 2024-06-04T05:16:38.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.856+0000 7f6685f43700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6680101ee0 0x7f66801030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.856+0000 7f6685f43700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66801028a0 0x7f6680103600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.856+0000 7f6685f43700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6680108600 0x7f6680103b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.856+0000 7f6685f43700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6680113640 con 0x7f6680101ee0 2024-06-04T05:16:38.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.856+0000 7f6685f43700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66801134c0 con 0x7f66801028a0 2024-06-04T05:16:38.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.856+0000 7f6685f43700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f66801137c0 con 0x7f6680108600 2024-06-04T05:16:38.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.856+0000 7f667f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6680101ee0 0x7f66801030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:38.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.856+0000 7f667f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6680101ee0 0x7f66801030c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56438/0 (socket says 172.21.15.70:56438) 2024-06-04T05:16:38.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.856+0000 7f667f7fe700 1 -- 172.21.15.70:0/1813156227 learned_addr learned my addr 172.21.15.70:0/1813156227 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:38.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.857+0000 7f667ffff700 1 --2- 172.21.15.70:0/1813156227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6680108600 0x7f6680103b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:38.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.857+0000 7f6676ffd700 1 --2- 172.21.15.70:0/1813156227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66801028a0 0x7f6680103600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:38.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.857+0000 7f667f7fe700 1 -- 172.21.15.70:0/1813156227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6680108600 msgr2=0x7f6680103b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:38.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.857+0000 7f667f7fe700 1 --2- 172.21.15.70:0/1813156227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6680108600 0x7f6680103b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.857+0000 7f667f7fe700 1 -- 172.21.15.70:0/1813156227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66801028a0 msgr2=0x7f6680103600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:38.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.857+0000 7f667f7fe700 1 --2- 172.21.15.70:0/1813156227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66801028a0 0x7f6680103600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.857+0000 7f667f7fe700 1 -- 172.21.15.70:0/1813156227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66801a80c0 con 0x7f6680101ee0 2024-06-04T05:16:38.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.857+0000 7f667ffff700 1 --2- 172.21.15.70:0/1813156227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6680108600 0x7f6680103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:16:38.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.857+0000 7f667f7fe700 1 --2- 172.21.15.70:0/1813156227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6680101ee0 0x7f66801030c0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f6670000c00 tx=0x7f66700047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:38.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.857+0000 7f667d7fa700 1 -- 172.21.15.70:0/1813156227 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6670027070 con 0x7f6680101ee0 2024-06-04T05:16:38.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.857+0000 7f667d7fa700 1 -- 172.21.15.70:0/1813156227 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f667000fdb0 con 0x7f6680101ee0 2024-06-04T05:16:38.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.857+0000 7f6685f43700 1 -- 172.21.15.70:0/1813156227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66801a8350 con 0x7f6680101ee0 2024-06-04T05:16:38.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.857+0000 7f667d7fa700 1 -- 172.21.15.70:0/1813156227 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66700173f0 con 0x7f6680101ee0 2024-06-04T05:16:38.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.857+0000 7f6685f43700 1 -- 172.21.15.70:0/1813156227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66801a8800 con 0x7f6680101ee0 2024-06-04T05:16:38.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.860+0000 7f667d7fa700 1 -- 172.21.15.70:0/1813156227 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f667002a070 con 0x7f6680101ee0 2024-06-04T05:16:38.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.860+0000 7f667d7fa700 1 --2- 172.21.15.70:0/1813156227 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f666c0b2a70 0x7f666c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.861+0000 7f6676ffd700 1 --2- 172.21.15.70:0/1813156227 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f666c0b2a70 0x7f666c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:38.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.861+0000 7f667d7fa700 1 -- 172.21.15.70:0/1813156227 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f66700e3740 con 0x7f6680101ee0 2024-06-04T05:16:38.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.861+0000 7f6676ffd700 1 --2- 172.21.15.70:0/1813156227 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f666c0b2a70 0x7f666c0b4f30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6668005ea0 tx=0x7f6668005e30 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:38.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.861+0000 7f66907d8b80 1 Processor -- start 2024-06-04T05:16:38.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.861+0000 7f66907d8b80 1 -- start start 2024-06-04T05:16:38.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.861+0000 7f66907d8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee790e320 0x55dee790e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f66907d8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7908d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f66907d8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee7909260 0x55dee7915080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f66907d8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dee7866d70 con 0x55dee79088a0 2024-06-04T05:16:38.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f66907d8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dee7856910 con 0x55dee7909260 2024-06-04T05:16:38.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f66907d8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55dee785a950 con 0x55dee790e320 2024-06-04T05:16:38.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f6676ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7908d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:38.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f6676ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7908d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56442/0 (socket says 172.21.15.70:56442) 2024-06-04T05:16:38.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f6676ffd700 1 -- 172.21.15.70:0/999449975 learned_addr learned my addr 172.21.15.70:0/999449975 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:38.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f667f7fe700 1 --2- 172.21.15.70:0/999449975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee790e320 0x55dee790e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:38.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f6676ffd700 1 -- 172.21.15.70:0/999449975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee790e320 msgr2=0x55dee790e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:38.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f6676ffd700 1 --2- 172.21.15.70:0/999449975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee790e320 0x55dee790e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f6676ffd700 1 -- 172.21.15.70:0/999449975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee7909260 msgr2=0x55dee7915080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:38.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f667ffff700 1 --2- 172.21.15.70:0/999449975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee7909260 0x55dee7915080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:38.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f6676ffd700 1 --2- 172.21.15.70:0/999449975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee7909260 0x55dee7915080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f6676ffd700 1 -- 172.21.15.70:0/999449975 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dee77fffb0 con 0x55dee79088a0 2024-06-04T05:16:38.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f667ffff700 1 --2- 172.21.15.70:0/999449975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee7909260 0x55dee7915080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:16:38.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f6676ffd700 1 --2- 172.21.15.70:0/999449975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7908d20 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f666800ec90 tx=0x7f666800af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f9f0277e23c7d8d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:38.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f6675ffb700 1 -- 172.21.15.70:0/999449975 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6668021070 con 0x55dee79088a0 2024-06-04T05:16:38.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f6675ffb700 1 -- 172.21.15.70:0/999449975 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6668010d90 con 0x55dee79088a0 2024-06-04T05:16:38.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.862+0000 7f6675ffb700 1 -- 172.21.15.70:0/999449975 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666800f620 con 0x55dee79088a0 2024-06-04T05:16:38.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.863+0000 7f66907d8b80 1 -- 172.21.15.70:0/999449975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 msgr2=0x55dee7908d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:38.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.863+0000 7f66907d8b80 1 --2- 172.21.15.70:0/999449975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7908d20 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f666800ec90 tx=0x7f666800af90 comp rx=0 tx=0).stop 2024-06-04T05:16:38.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.863+0000 7f66907d8b80 1 -- 172.21.15.70:0/999449975 shutdown_connections 2024-06-04T05:16:38.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.863+0000 7f66907d8b80 1 --2- 172.21.15.70:0/999449975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee7909260 0x55dee7915080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.863+0000 7f66907d8b80 1 --2- 172.21.15.70:0/999449975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee790e320 0x55dee790e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.863+0000 7f66907d8b80 1 --2- 172.21.15.70:0/999449975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7908d20 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.863+0000 7f66907d8b80 1 -- 172.21.15.70:0/999449975 >> 172.21.15.70:0/999449975 conn(0x55dee7834240 msgr2=0x55dee79033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:38.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.863+0000 7f66907d8b80 1 -- 172.21.15.70:0/999449975 shutdown_connections 2024-06-04T05:16:38.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.863+0000 7f66907d8b80 1 -- 172.21.15.70:0/999449975 wait complete. 2024-06-04T05:16:38.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.863+0000 7f66907d8b80 1 -- start start 2024-06-04T05:16:38.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f66907d8b80 1 Processor -- start 2024-06-04T05:16:38.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f66907d8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7914d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f66907d8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee7909260 0x55dee7912040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f66907d8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee790e320 0x55dee7912580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f66907d8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dee775cfa0 con 0x55dee79088a0 2024-06-04T05:16:38.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f66907d8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dee77beda0 con 0x55dee790e320 2024-06-04T05:16:38.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f66907d8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55dee79156e0 con 0x55dee7909260 2024-06-04T05:16:38.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f66907d8b80 10 client.0 _ll_register_callbacks cb 0x55dee77f1950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:38.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f667f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7914d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:38.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f667f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7914d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56458/0 (socket says 172.21.15.70:56458) 2024-06-04T05:16:38.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f667f7fe700 1 -- 172.21.15.70:0/3953148309 learned_addr learned my addr 172.21.15.70:0/3953148309 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:38.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f6676ffd700 1 --2- 172.21.15.70:0/3953148309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee7909260 0x55dee7912040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:38.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f667ffff700 1 --2- 172.21.15.70:0/3953148309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee790e320 0x55dee7912580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:38.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f667f7fe700 1 -- 172.21.15.70:0/3953148309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee7909260 msgr2=0x55dee7912040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:38.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f667f7fe700 1 --2- 172.21.15.70:0/3953148309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee7909260 0x55dee7912040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f667f7fe700 1 -- 172.21.15.70:0/3953148309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee790e320 msgr2=0x55dee7912580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:38.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f667f7fe700 1 --2- 172.21.15.70:0/3953148309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee790e320 0x55dee7912580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.864+0000 7f667f7fe700 1 -- 172.21.15.70:0/3953148309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dee77a68d0 con 0x55dee79088a0 2024-06-04T05:16:38.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.865+0000 7f667f7fe700 1 --2- 172.21.15.70:0/3953148309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7914d40 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f66700e0e60 tx=0x7f6670022990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55862bce616acf6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:38.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.865+0000 7f665effd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:38.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.865+0000 7f665effd700 1 -- 172.21.15.70:0/3953148309 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66700cd2d0 con 0x55dee79088a0 2024-06-04T05:16:38.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.865+0000 7f665effd700 1 -- 172.21.15.70:0/3953148309 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f66700b10e0 con 0x55dee79088a0 2024-06-04T05:16:38.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.865+0000 7f66907d8b80 1 -- 172.21.15.70:0/3953148309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6668010d90 con 0x55dee79088a0 2024-06-04T05:16:38.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.865+0000 7f665effd700 1 -- 172.21.15.70:0/3953148309 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f66700ab120 con 0x55dee79088a0 2024-06-04T05:16:38.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.865+0000 7f66907d8b80 10 client.20475 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:38.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.865+0000 7f66907d8b80 1 -- 172.21.15.70:0/3953148309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55dee7810760 con 0x55dee79088a0 2024-06-04T05:16:38.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.865+0000 7f665effd700 1 -- 172.21.15.70:0/3953148309 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6668010d90 con 0x55dee79088a0 2024-06-04T05:16:38.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.865+0000 7f665effd700 1 -- 172.21.15.70:0/3953148309 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f66700b3360 con 0x55dee79088a0 2024-06-04T05:16:38.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.865+0000 7f66907d8b80 10 client.20475 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:38.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.865+0000 7f66907d8b80 10 client.20475 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:16:38.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.865+0000 7f66907d8b80 20 client.20475 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:38.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.865+0000 7f66907d8b80 1 --2- 172.21.15.70:0/3953148309 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55dee79220e0 0x55dee79244f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.865+0000 7f66907d8b80 4 client.20475 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:16:38.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.865+0000 7f66907d8b80 1 -- 172.21.15.70:0/3953148309 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dee7868cc0 con 0x55dee79220e0 2024-06-04T05:16:38.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.866+0000 7f6676ffd700 1 --2- 172.21.15.70:0/3953148309 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55dee79220e0 0x55dee79244f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:38.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.866+0000 7f6676ffd700 1 --2- 172.21.15.70:0/3953148309 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55dee79220e0 0x55dee79244f0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5641edd0abc2107e server_cookie=5e643a1a12b5c295 in_seq=0 out_seq=0 2024-06-04T05:16:38.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.866+0000 7f665effd700 10 client.20475 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:16:38.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.867+0000 7f665effd700 1 -- 172.21.15.70:0/3953148309 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55dee7868cc0 con 0x55dee79220e0 2024-06-04T05:16:38.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.867+0000 7f665effd700 10 client.20475 handle_command_reply: tid=0 2024-06-04T05:16:38.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.867+0000 7f66907d8b80 1 client.20475 shutdown 2024-06-04T05:16:38.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.867+0000 7f66907d8b80 1 -- 172.21.15.70:0/3953148309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 msgr2=0x55dee7914d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:38.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.867+0000 7f66907d8b80 1 --2- 172.21.15.70:0/3953148309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7914d40 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f66700e0e60 tx=0x7f6670022990 comp rx=0 tx=0).stop 2024-06-04T05:16:38.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.867+0000 7f66907d8b80 1 -- 172.21.15.70:0/3953148309 shutdown_connections 2024-06-04T05:16:38.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.867+0000 7f66907d8b80 1 --2- 172.21.15.70:0/3953148309 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55dee79220e0 0x55dee79244f0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.867+0000 7f66907d8b80 1 --2- 172.21.15.70:0/3953148309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee790e320 0x55dee7912580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.867+0000 7f66907d8b80 1 --2- 172.21.15.70:0/3953148309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7914d40 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.867+0000 7f665effd700 0 client.20475 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:16:38.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.867+0000 7f66907d8b80 1 --2- 172.21.15.70:0/3953148309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee7909260 0x55dee7912040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.867+0000 7f66907d8b80 1 -- 172.21.15.70:0/3953148309 >> 172.21.15.70:0/3953148309 conn(0x55dee7834240 msgr2=0x55dee7902af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:38.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.868+0000 7f66907d8b80 1 -- 172.21.15.70:0/3953148309 shutdown_connections 2024-06-04T05:16:38.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.868+0000 7f66907d8b80 1 -- 172.21.15.70:0/3953148309 wait complete. 2024-06-04T05:16:38.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.868+0000 7f66907d8b80 20 client.20475 trim_cache size 0 max 16384 2024-06-04T05:16:38.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.882+0000 7f66907d8b80 1 Processor -- start 2024-06-04T05:16:38.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.882+0000 7f66907d8b80 1 -- start start 2024-06-04T05:16:38.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f66907d8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7903eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f66907d8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee7909260 0x55dee7903270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f66907d8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee790e320 0x55dee7913f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f66907d8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f66801aff80 con 0x55dee79088a0 2024-06-04T05:16:38.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f66907d8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66801b1ac0 con 0x55dee7909260 2024-06-04T05:16:38.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f66907d8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f66801b0650 con 0x55dee790e320 2024-06-04T05:16:38.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f667f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7903eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:38.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f667f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7903eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56466/0 (socket says 172.21.15.70:56466) 2024-06-04T05:16:38.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f667f7fe700 1 -- 172.21.15.70:0/2995819633 learned_addr learned my addr 172.21.15.70:0/2995819633 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:38.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f6676ffd700 1 --2- 172.21.15.70:0/2995819633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee7909260 0x55dee7903270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:38.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f667f7fe700 1 -- 172.21.15.70:0/2995819633 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee790e320 msgr2=0x55dee7913f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:38.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f667f7fe700 1 --2- 172.21.15.70:0/2995819633 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee790e320 0x55dee7913f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f667f7fe700 1 -- 172.21.15.70:0/2995819633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee7909260 msgr2=0x55dee7903270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:38.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f667f7fe700 1 --2- 172.21.15.70:0/2995819633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee7909260 0x55dee7903270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f667f7fe700 1 -- 172.21.15.70:0/2995819633 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dee79105d0 con 0x55dee79088a0 2024-06-04T05:16:38.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f6676ffd700 1 --2- 172.21.15.70:0/2995819633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee7909260 0x55dee7903270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:16:38.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f667f7fe700 1 --2- 172.21.15.70:0/2995819633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7903eb0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f66700a6aa0 tx=0x7f6670009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8691df1f8b837e4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:38.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f66757fa700 1 -- 172.21.15.70:0/2995819633 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66700a9f60 con 0x55dee79088a0 2024-06-04T05:16:38.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f66757fa700 1 -- 172.21.15.70:0/2995819633 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f66700aaa20 con 0x55dee79088a0 2024-06-04T05:16:38.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.883+0000 7f66757fa700 1 -- 172.21.15.70:0/2995819633 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66700b10f0 con 0x55dee79088a0 2024-06-04T05:16:38.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.884+0000 7f66907d8b80 1 -- 172.21.15.70:0/2995819633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 msgr2=0x55dee7903eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:38.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.884+0000 7f66907d8b80 1 --2- 172.21.15.70:0/2995819633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7903eb0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f66700a6aa0 tx=0x7f6670009ca0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.884+0000 7f66907d8b80 1 -- 172.21.15.70:0/2995819633 shutdown_connections 2024-06-04T05:16:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.884+0000 7f66907d8b80 1 --2- 172.21.15.70:0/2995819633 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee790e320 0x55dee7913f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.884+0000 7f66907d8b80 1 --2- 172.21.15.70:0/2995819633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee79088a0 0x55dee7903eb0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.884+0000 7f66907d8b80 1 --2- 172.21.15.70:0/2995819633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee7909260 0x55dee7903270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.884+0000 7f66907d8b80 1 -- 172.21.15.70:0/2995819633 >> 172.21.15.70:0/2995819633 conn(0x55dee7834240 msgr2=0x55dee7913870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.884+0000 7f66907d8b80 1 -- 172.21.15.70:0/2995819633 shutdown_connections 2024-06-04T05:16:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.884+0000 7f66907d8b80 1 -- 172.21.15.70:0/2995819633 wait complete. 2024-06-04T05:16:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.885+0000 7f66907d8b80 1 -- start start 2024-06-04T05:16:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.885+0000 7f66907d8b80 1 Processor -- start 2024-06-04T05:16:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.885+0000 7f66907d8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee79088a0 0x55dee791b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.885+0000 7f66907d8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee7909260 0x55dee7929670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.885+0000 7f66907d8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee790e320 0x55dee7929bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.885+0000 7f66907d8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f66801b13f0 con 0x55dee790e320 2024-06-04T05:16:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.885+0000 7f66907d8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66801b0d20 con 0x55dee79088a0 2024-06-04T05:16:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.885+0000 7f66907d8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f66801adf40 con 0x55dee7909260 2024-06-04T05:16:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.885+0000 7f66907d8b80 10 client.0 _ll_register_callbacks cb 0x55dee77f1950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.885+0000 7f667ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee790e320 0x55dee7929bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.885+0000 7f667ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee790e320 0x55dee7929bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56470/0 (socket says 172.21.15.70:56470) 2024-06-04T05:16:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.885+0000 7f667ffff700 1 -- 172.21.15.70:0/1641440371 learned_addr learned my addr 172.21.15.70:0/1641440371 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.886+0000 7f6676ffd700 1 --2- 172.21.15.70:0/1641440371 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee7909260 0x55dee7929670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.886+0000 7f667f7fe700 1 --2- 172.21.15.70:0/1641440371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee79088a0 0x55dee791b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.886+0000 7f667ffff700 1 -- 172.21.15.70:0/1641440371 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee7909260 msgr2=0x55dee7929670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.886+0000 7f667ffff700 1 --2- 172.21.15.70:0/1641440371 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee7909260 0x55dee7929670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.886+0000 7f667ffff700 1 -- 172.21.15.70:0/1641440371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee79088a0 msgr2=0x55dee791b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.886+0000 7f667ffff700 1 --2- 172.21.15.70:0/1641440371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee79088a0 0x55dee791b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.886+0000 7f667ffff700 1 -- 172.21.15.70:0/1641440371 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dee792a3d0 con 0x55dee790e320 2024-06-04T05:16:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.886+0000 7f667f7fe700 1 --2- 172.21.15.70:0/1641440371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee79088a0 0x55dee791b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:16:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.886+0000 7f6676ffd700 1 --2- 172.21.15.70:0/1641440371 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee7909260 0x55dee7929670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:16:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.886+0000 7f667ffff700 1 --2- 172.21.15.70:0/1641440371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee790e320 0x55dee7929bb0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f667800aa70 tx=0x7f667800ab50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721e735f91d8f568 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.886+0000 7f665effd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.886+0000 7f665effd700 1 -- 172.21.15.70:0/1641440371 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6678014780 con 0x55dee790e320 2024-06-04T05:16:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.886+0000 7f665effd700 1 -- 172.21.15.70:0/1641440371 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f6678014920 con 0x55dee790e320 2024-06-04T05:16:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.886+0000 7f66907d8b80 1 -- 172.21.15.70:0/1641440371 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66700aaa20 con 0x55dee790e320 2024-06-04T05:16:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.886+0000 7f665effd700 1 -- 172.21.15.70:0/1641440371 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f6678023050 con 0x55dee790e320 2024-06-04T05:16:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.886+0000 7f66907d8b80 10 client.20481 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.886+0000 7f66907d8b80 1 -- 172.21.15.70:0/1641440371 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55dee7995540 con 0x55dee790e320 2024-06-04T05:16:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.887+0000 7f665effd700 1 -- 172.21.15.70:0/1641440371 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f66700aaa20 con 0x55dee790e320 2024-06-04T05:16:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.887+0000 7f665effd700 1 -- 172.21.15.70:0/1641440371 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f6678015c10 con 0x55dee790e320 2024-06-04T05:16:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.887+0000 7f66907d8b80 10 client.20481 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.887+0000 7f66907d8b80 10 client.20481 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:16:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.887+0000 7f66907d8b80 20 client.20481 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.887+0000 7f66907d8b80 1 --2- 172.21.15.70:0/1641440371 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55dee7995cd0 0x55dee79980f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.887+0000 7f66907d8b80 4 client.20481 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:16:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.887+0000 7f66907d8b80 1 -- 172.21.15.70:0/1641440371 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dee78624a0 con 0x55dee7995cd0 2024-06-04T05:16:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.887+0000 7f667f7fe700 1 --2- 172.21.15.70:0/1641440371 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55dee7995cd0 0x55dee79980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.888+0000 7f667f7fe700 1 --2- 172.21.15.70:0/1641440371 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55dee7995cd0 0x55dee79980f0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8b562dae604bda69 server_cookie=b82e4a846d8621ea in_seq=0 out_seq=0 2024-06-04T05:16:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.888+0000 7f665effd700 10 client.20481 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:16:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.888+0000 7f665effd700 1 -- 172.21.15.70:0/1641440371 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dee78624a0 con 0x55dee7995cd0 2024-06-04T05:16:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.888+0000 7f665effd700 10 client.20481 handle_command_reply: tid=0 2024-06-04T05:16:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.888+0000 7f66907d8b80 1 client.20481 shutdown 2024-06-04T05:16:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.888+0000 7f66907d8b80 1 -- 172.21.15.70:0/1641440371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee790e320 msgr2=0x55dee7929bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.888+0000 7f66907d8b80 1 --2- 172.21.15.70:0/1641440371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee790e320 0x55dee7929bb0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f667800aa70 tx=0x7f667800ab50 comp rx=0 tx=0).stop 2024-06-04T05:16:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.888+0000 7f66907d8b80 1 -- 172.21.15.70:0/1641440371 shutdown_connections 2024-06-04T05:16:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.888+0000 7f66907d8b80 1 --2- 172.21.15.70:0/1641440371 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55dee7995cd0 0x55dee79980f0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.888+0000 7f66907d8b80 1 --2- 172.21.15.70:0/1641440371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dee790e320 0x55dee7929bb0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.888+0000 7f66907d8b80 1 --2- 172.21.15.70:0/1641440371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dee79088a0 0x55dee791b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.888+0000 7f66907d8b80 1 --2- 172.21.15.70:0/1641440371 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dee7909260 0x55dee7929670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.888+0000 7f66907d8b80 1 -- 172.21.15.70:0/1641440371 >> 172.21.15.70:0/1641440371 conn(0x55dee7834240 msgr2=0x55dee790bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:38.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.888+0000 7f665effd700 0 client.20481 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:16:38.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.888+0000 7f66907d8b80 1 -- 172.21.15.70:0/1641440371 shutdown_connections 2024-06-04T05:16:38.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.888+0000 7f66907d8b80 1 -- 172.21.15.70:0/1641440371 wait complete. 2024-06-04T05:16:38.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.888+0000 7f66907d8b80 20 client.20481 trim_cache size 0 max 16384 2024-06-04T05:16:38.151 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:16:38.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.889+0000 7f6685f43700 1 -- 172.21.15.70:0/1813156227 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f666c0b2a70 msgr2=0x7f666c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:38.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.889+0000 7f6685f43700 1 --2- 172.21.15.70:0/1813156227 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f666c0b2a70 0x7f666c0b4f30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6668005ea0 tx=0x7f6668005e30 comp rx=0 tx=0).stop 2024-06-04T05:16:38.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.889+0000 7f6685f43700 1 -- 172.21.15.70:0/1813156227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6680101ee0 msgr2=0x7f66801030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:38.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.889+0000 7f6685f43700 1 --2- 172.21.15.70:0/1813156227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6680101ee0 0x7f66801030c0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f6670000c00 tx=0x7f66700047b0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.889+0000 7f6685f43700 1 -- 172.21.15.70:0/1813156227 shutdown_connections 2024-06-04T05:16:38.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.889+0000 7f6685f43700 1 --2- 172.21.15.70:0/1813156227 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f666c0b2a70 0x7f666c0b4f30 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.889+0000 7f6685f43700 1 --2- 172.21.15.70:0/1813156227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6680108600 0x7f6680103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.889+0000 7f6685f43700 1 --2- 172.21.15.70:0/1813156227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6680101ee0 0x7f66801030c0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.889+0000 7f6685f43700 1 --2- 172.21.15.70:0/1813156227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66801028a0 0x7f6680103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:38.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.889+0000 7f6685f43700 1 -- 172.21.15.70:0/1813156227 >> 172.21.15.70:0/1813156227 conn(0x7f6680077760 msgr2=0x7f6680100b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:38.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.889+0000 7f6685f43700 1 -- 172.21.15.70:0/1813156227 shutdown_connections 2024-06-04T05:16:38.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:37.889+0000 7f6685f43700 1 -- 172.21.15.70:0/1813156227 wait complete. 2024-06-04T05:16:38.596 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:16:38.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.91 MB/sec execute 210 sec latency 6246.503 ms 2024-06-04T05:16:38.838 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:16:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:39 smithi070 ceph-mon[32631]: pgmap v611: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 212 op/s 2024-06-04T05:16:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:39 smithi112 ceph-mon[36453]: pgmap v611: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 212 op/s 2024-06-04T05:16:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:39 smithi098 ceph-mon[35569]: pgmap v611: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 212 op/s 2024-06-04T05:16:39.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.88 MB/sec execute 211 sec latency 7246.590 ms 2024-06-04T05:16:40.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.499+0000 7fab47484700 1 -- 172.21.15.70:0/3290485513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab40103560 msgr2=0x7fab4010f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:40.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.499+0000 7fab47484700 1 --2- 172.21.15.70:0/3290485513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab40103560 0x7fab4010f550 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fab3800b0a0 tx=0x7fab3800b3b0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.499+0000 7fab47484700 1 -- 172.21.15.70:0/3290485513 shutdown_connections 2024-06-04T05:16:40.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.499+0000 7fab47484700 1 --2- 172.21.15.70:0/3290485513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab40103560 0x7fab4010f550 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.499+0000 7fab47484700 1 --2- 172.21.15.70:0/3290485513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab40108710 0x7fab40108af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.499+0000 7fab47484700 1 --2- 172.21.15.70:0/3290485513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab40102ba0 0x7fab40103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.499+0000 7fab47484700 1 -- 172.21.15.70:0/3290485513 >> 172.21.15.70:0/3290485513 conn(0x7fab40078a40 msgr2=0x7fab400ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:40.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.499+0000 7fab47484700 1 -- 172.21.15.70:0/3290485513 shutdown_connections 2024-06-04T05:16:40.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.499+0000 7fab47484700 1 -- 172.21.15.70:0/3290485513 wait complete. 2024-06-04T05:16:40.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.499+0000 7fab47484700 1 Processor -- start 2024-06-04T05:16:40.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.500+0000 7fab47484700 1 -- start start 2024-06-04T05:16:40.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.500+0000 7fab47484700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab40102ba0 0x7fab4019f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.500+0000 7fab47484700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab40103560 0x7fab4019fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.500+0000 7fab47484700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab40108710 0x7fab40199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.500+0000 7fab47484700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab40112730 con 0x7fab40102ba0 2024-06-04T05:16:40.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.500+0000 7fab47484700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fab401125b0 con 0x7fab40103560 2024-06-04T05:16:40.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.500+0000 7fab47484700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fab401128b0 con 0x7fab40108710 2024-06-04T05:16:40.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.500+0000 7fab45220700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab40102ba0 0x7fab4019f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.500+0000 7fab45220700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab40102ba0 0x7fab4019f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56484/0 (socket says 172.21.15.70:56484) 2024-06-04T05:16:40.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.500+0000 7fab45220700 1 -- 172.21.15.70:0/3639603105 learned_addr learned my addr 172.21.15.70:0/3639603105 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:40.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.500+0000 7fab44a1f700 1 --2- 172.21.15.70:0/3639603105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab40103560 0x7fab4019fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.500+0000 7fab45a21700 1 --2- 172.21.15.70:0/3639603105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab40108710 0x7fab40199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.500+0000 7fab45220700 1 -- 172.21.15.70:0/3639603105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab40108710 msgr2=0x7fab40199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:40.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.500+0000 7fab45220700 1 --2- 172.21.15.70:0/3639603105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab40108710 0x7fab40199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.500+0000 7fab45220700 1 -- 172.21.15.70:0/3639603105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab40103560 msgr2=0x7fab4019fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:40.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.500+0000 7fab45220700 1 --2- 172.21.15.70:0/3639603105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab40103560 0x7fab4019fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.500+0000 7fab45220700 1 -- 172.21.15.70:0/3639603105 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4019a1e0 con 0x7fab40102ba0 2024-06-04T05:16:40.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.501+0000 7fab45220700 1 --2- 172.21.15.70:0/3639603105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab40102ba0 0x7fab4019f8c0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00d980 tx=0x7fab3c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:40.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.501+0000 7fab367fc700 1 -- 172.21.15.70:0/3639603105 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab3c017070 con 0x7fab40102ba0 2024-06-04T05:16:40.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.501+0000 7fab367fc700 1 -- 172.21.15.70:0/3639603105 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab3c00fd20 con 0x7fab40102ba0 2024-06-04T05:16:40.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.501+0000 7fab47484700 1 -- 172.21.15.70:0/3639603105 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab4019a470 con 0x7fab40102ba0 2024-06-04T05:16:40.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.501+0000 7fab367fc700 1 -- 172.21.15.70:0/3639603105 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab3c00e3b0 con 0x7fab40102ba0 2024-06-04T05:16:40.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.501+0000 7fab47484700 1 -- 172.21.15.70:0/3639603105 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab401a6790 con 0x7fab40102ba0 2024-06-04T05:16:40.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.503+0000 7fab367fc700 1 -- 172.21.15.70:0/3639603105 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fab3c00f890 con 0x7fab40102ba0 2024-06-04T05:16:40.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.504+0000 7fab367fc700 1 --2- 172.21.15.70:0/3639603105 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab2c0b29f0 0x7fab2c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.504+0000 7fab44a1f700 1 --2- 172.21.15.70:0/3639603105 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab2c0b29f0 0x7fab2c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.504+0000 7fab367fc700 1 -- 172.21.15.70:0/3639603105 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7fab3c0e1a70 con 0x7fab40102ba0 2024-06-04T05:16:40.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.504+0000 7fab44a1f700 1 --2- 172.21.15.70:0/3639603105 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab2c0b29f0 0x7fab2c0b4eb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fab40107d20 tx=0x7fab3000a300 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:40.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab51d19b80 1 Processor -- start 2024-06-04T05:16:40.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab51d19b80 1 -- start start 2024-06-04T05:16:40.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab51d19b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3d0320 0x55eb7d3d0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab51d19b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3ca8a0 0x55eb7d3cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab51d19b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3cb260 0x55eb7d3d7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab51d19b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55eb7d328d70 con 0x55eb7d3d0320 2024-06-04T05:16:40.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab51d19b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eb7d318910 con 0x55eb7d3cb260 2024-06-04T05:16:40.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab51d19b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55eb7d31c950 con 0x55eb7d3ca8a0 2024-06-04T05:16:40.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab45220700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3d0320 0x55eb7d3d0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab45220700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3d0320 0x55eb7d3d0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56488/0 (socket says 172.21.15.70:56488) 2024-06-04T05:16:40.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab45220700 1 -- 172.21.15.70:0/2370579432 learned_addr learned my addr 172.21.15.70:0/2370579432 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:40.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab44a1f700 1 --2- 172.21.15.70:0/2370579432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3ca8a0 0x55eb7d3cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab45a21700 1 --2- 172.21.15.70:0/2370579432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3cb260 0x55eb7d3d7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab45220700 1 -- 172.21.15.70:0/2370579432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3ca8a0 msgr2=0x55eb7d3cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:40.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab45220700 1 --2- 172.21.15.70:0/2370579432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3ca8a0 0x55eb7d3cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab45220700 1 -- 172.21.15.70:0/2370579432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3cb260 msgr2=0x55eb7d3d7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:40.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab45220700 1 --2- 172.21.15.70:0/2370579432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3cb260 0x55eb7d3d7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab45220700 1 -- 172.21.15.70:0/2370579432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb7d2c1fb0 con 0x55eb7d3d0320 2024-06-04T05:16:40.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.505+0000 7fab45220700 1 --2- 172.21.15.70:0/2370579432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3d0320 0x55eb7d3d0700 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fab3c0df790 tx=0x7fab3c0a8560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf00d54e0392683 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:40.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.506+0000 7fab2b7fe700 1 -- 172.21.15.70:0/2370579432 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab3c0bf020 con 0x55eb7d3d0320 2024-06-04T05:16:40.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.506+0000 7fab2b7fe700 1 -- 172.21.15.70:0/2370579432 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab3c0a8d20 con 0x55eb7d3d0320 2024-06-04T05:16:40.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.506+0000 7fab2b7fe700 1 -- 172.21.15.70:0/2370579432 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab3c020040 con 0x55eb7d3d0320 2024-06-04T05:16:40.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.506+0000 7fab51d19b80 1 -- 172.21.15.70:0/2370579432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3d0320 msgr2=0x55eb7d3d0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:40.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.506+0000 7fab51d19b80 1 --2- 172.21.15.70:0/2370579432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3d0320 0x55eb7d3d0700 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fab3c0df790 tx=0x7fab3c0a8560 comp rx=0 tx=0).stop 2024-06-04T05:16:40.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.506+0000 7fab51d19b80 1 -- 172.21.15.70:0/2370579432 shutdown_connections 2024-06-04T05:16:40.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.506+0000 7fab51d19b80 1 --2- 172.21.15.70:0/2370579432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3cb260 0x55eb7d3d7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.506+0000 7fab51d19b80 1 --2- 172.21.15.70:0/2370579432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3d0320 0x55eb7d3d0700 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.506+0000 7fab51d19b80 1 --2- 172.21.15.70:0/2370579432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3ca8a0 0x55eb7d3cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.506+0000 7fab51d19b80 1 -- 172.21.15.70:0/2370579432 >> 172.21.15.70:0/2370579432 conn(0x55eb7d2f6240 msgr2=0x55eb7d3c53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:40.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.506+0000 7fab51d19b80 1 -- 172.21.15.70:0/2370579432 shutdown_connections 2024-06-04T05:16:40.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.506+0000 7fab51d19b80 1 -- 172.21.15.70:0/2370579432 wait complete. 2024-06-04T05:16:40.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.507+0000 7fab51d19b80 1 -- start start 2024-06-04T05:16:40.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.507+0000 7fab51d19b80 1 Processor -- start 2024-06-04T05:16:40.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.507+0000 7fab51d19b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3ca8a0 0x55eb7d3d6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.507+0000 7fab51d19b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3cb260 0x55eb7d3d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.507+0000 7fab51d19b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3d0320 0x55eb7d3d4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.507+0000 7fab51d19b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55eb7d21efa0 con 0x55eb7d3cb260 2024-06-04T05:16:40.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.507+0000 7fab51d19b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eb7d280da0 con 0x55eb7d3ca8a0 2024-06-04T05:16:40.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.507+0000 7fab51d19b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55eb7d3d7680 con 0x55eb7d3d0320 2024-06-04T05:16:40.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.507+0000 7fab51d19b80 10 client.0 _ll_register_callbacks cb 0x55eb7d2b3950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:40.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.507+0000 7fab44a1f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3cb260 0x55eb7d3d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.507+0000 7fab44a1f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3cb260 0x55eb7d3d40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56500/0 (socket says 172.21.15.70:56500) 2024-06-04T05:16:40.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.507+0000 7fab44a1f700 1 -- 172.21.15.70:0/4234079546 learned_addr learned my addr 172.21.15.70:0/4234079546 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:40.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.508+0000 7fab45a21700 1 --2- 172.21.15.70:0/4234079546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3d0320 0x55eb7d3d4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.508+0000 7fab45220700 1 --2- 172.21.15.70:0/4234079546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3ca8a0 0x55eb7d3d6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.508+0000 7fab44a1f700 1 -- 172.21.15.70:0/4234079546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3d0320 msgr2=0x55eb7d3d4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:40.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.508+0000 7fab44a1f700 1 --2- 172.21.15.70:0/4234079546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3d0320 0x55eb7d3d4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.508+0000 7fab44a1f700 1 -- 172.21.15.70:0/4234079546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3ca8a0 msgr2=0x55eb7d3d6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:40.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.508+0000 7fab44a1f700 1 --2- 172.21.15.70:0/4234079546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3ca8a0 0x55eb7d3d6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.508+0000 7fab44a1f700 1 -- 172.21.15.70:0/4234079546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb7d2688d0 con 0x55eb7d3cb260 2024-06-04T05:16:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.508+0000 7fab44a1f700 1 --2- 172.21.15.70:0/4234079546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3cb260 0x55eb7d3d40c0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fab3001d9e0 tx=0x7fab300047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9fbeda1cd6fb4db server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.508+0000 7fab28ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.508+0000 7fab28ff9700 1 -- 172.21.15.70:0/4234079546 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab3002a060 con 0x55eb7d3cb260 2024-06-04T05:16:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.508+0000 7fab28ff9700 1 -- 172.21.15.70:0/4234079546 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7fab30010840 con 0x55eb7d3cb260 2024-06-04T05:16:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.508+0000 7fab51d19b80 1 -- 172.21.15.70:0/4234079546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab3c0a8d20 con 0x55eb7d3cb260 2024-06-04T05:16:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.508+0000 7fab28ff9700 1 -- 172.21.15.70:0/4234079546 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fab30016070 con 0x55eb7d3cb260 2024-06-04T05:16:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.508+0000 7fab51d19b80 10 client.20493 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.508+0000 7fab51d19b80 1 -- 172.21.15.70:0/4234079546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55eb7d2d2760 con 0x55eb7d3cb260 2024-06-04T05:16:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.508+0000 7fab28ff9700 1 -- 172.21.15.70:0/4234079546 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab3c0a8d20 con 0x55eb7d3cb260 2024-06-04T05:16:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.509+0000 7fab28ff9700 1 -- 172.21.15.70:0/4234079546 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fab30024030 con 0x55eb7d3cb260 2024-06-04T05:16:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.509+0000 7fab51d19b80 10 client.20493 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.509+0000 7fab51d19b80 10 client.20493 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:16:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.509+0000 7fab51d19b80 20 client.20493 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:40.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.509+0000 7fab51d19b80 1 --2- 172.21.15.70:0/4234079546 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55eb7d3e4270 0x55eb7d3e6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.509+0000 7fab51d19b80 4 client.20493 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:16:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.509+0000 7fab51d19b80 1 -- 172.21.15.70:0/4234079546 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb7d32acc0 con 0x55eb7d3e4270 2024-06-04T05:16:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.509+0000 7fab45220700 1 --2- 172.21.15.70:0/4234079546 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55eb7d3e4270 0x55eb7d3e6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.509+0000 7fab45220700 1 --2- 172.21.15.70:0/4234079546 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55eb7d3e4270 0x55eb7d3e6660 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f652660fde0f7f6d server_cookie=87dcc98740f4aced in_seq=0 out_seq=0 2024-06-04T05:16:40.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.509+0000 7fab28ff9700 10 client.20493 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:16:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.510+0000 7fab28ff9700 1 -- 172.21.15.70:0/4234079546 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55eb7d32acc0 con 0x55eb7d3e4270 2024-06-04T05:16:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.510+0000 7fab28ff9700 10 client.20493 handle_command_reply: tid=0 2024-06-04T05:16:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.510+0000 7fab51d19b80 1 client.20493 shutdown 2024-06-04T05:16:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.510+0000 7fab51d19b80 1 -- 172.21.15.70:0/4234079546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3cb260 msgr2=0x55eb7d3d40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.510+0000 7fab51d19b80 1 --2- 172.21.15.70:0/4234079546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3cb260 0x55eb7d3d40c0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fab3001d9e0 tx=0x7fab300047c0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.510+0000 7fab51d19b80 1 -- 172.21.15.70:0/4234079546 shutdown_connections 2024-06-04T05:16:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.510+0000 7fab51d19b80 1 --2- 172.21.15.70:0/4234079546 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55eb7d3e4270 0x55eb7d3e6660 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.510+0000 7fab51d19b80 1 --2- 172.21.15.70:0/4234079546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3d0320 0x55eb7d3d4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.510+0000 7fab51d19b80 1 --2- 172.21.15.70:0/4234079546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3ca8a0 0x55eb7d3d6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.510+0000 7fab51d19b80 1 --2- 172.21.15.70:0/4234079546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3cb260 0x55eb7d3d40c0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.510+0000 7fab51d19b80 1 -- 172.21.15.70:0/4234079546 >> 172.21.15.70:0/4234079546 conn(0x55eb7d2f6240 msgr2=0x55eb7d3c4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:40.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.510+0000 7fab28ff9700 0 client.20493 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:16:40.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.511+0000 7fab51d19b80 1 -- 172.21.15.70:0/4234079546 shutdown_connections 2024-06-04T05:16:40.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.511+0000 7fab51d19b80 1 -- 172.21.15.70:0/4234079546 wait complete. 2024-06-04T05:16:40.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.511+0000 7fab51d19b80 20 client.20493 trim_cache size 0 max 16384 2024-06-04T05:16:40.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab51d19b80 1 Processor -- start 2024-06-04T05:16:40.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab51d19b80 1 -- start start 2024-06-04T05:16:40.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab51d19b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3ca8a0 0x55eb7d3d3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab51d19b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3cb260 0x55eb7d3cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab51d19b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3d0320 0x55eb7d3c5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab51d19b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab401adde0 con 0x55eb7d3d0320 2024-06-04T05:16:40.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab51d19b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fab401af920 con 0x55eb7d3cb260 2024-06-04T05:16:40.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab51d19b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fab401ae4b0 con 0x55eb7d3ca8a0 2024-06-04T05:16:40.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab45a21700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3d0320 0x55eb7d3c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab45a21700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3d0320 0x55eb7d3c5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56504/0 (socket says 172.21.15.70:56504) 2024-06-04T05:16:40.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab45a21700 1 -- 172.21.15.70:0/521992004 learned_addr learned my addr 172.21.15.70:0/521992004 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:40.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab45220700 1 --2- 172.21.15.70:0/521992004 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3ca8a0 0x55eb7d3d3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab44a1f700 1 --2- 172.21.15.70:0/521992004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3cb260 0x55eb7d3cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab45a21700 1 -- 172.21.15.70:0/521992004 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3ca8a0 msgr2=0x55eb7d3d3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:40.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab45a21700 1 --2- 172.21.15.70:0/521992004 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3ca8a0 0x55eb7d3d3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab45a21700 1 -- 172.21.15.70:0/521992004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3cb260 msgr2=0x55eb7d3cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:40.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab45a21700 1 --2- 172.21.15.70:0/521992004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3cb260 0x55eb7d3cf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab45a21700 1 -- 172.21.15.70:0/521992004 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb7d3cff40 con 0x55eb7d3d0320 2024-06-04T05:16:40.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.526+0000 7fab45a21700 1 --2- 172.21.15.70:0/521992004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3d0320 0x55eb7d3c5270 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fab38009370 tx=0x7fab380093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2163eb9ec874dcdd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:40.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.527+0000 7fab2affd700 1 -- 172.21.15.70:0/521992004 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab38003c00 con 0x55eb7d3d0320 2024-06-04T05:16:40.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.527+0000 7fab2affd700 1 -- 172.21.15.70:0/521992004 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab38003da0 con 0x55eb7d3d0320 2024-06-04T05:16:40.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.527+0000 7fab2affd700 1 -- 172.21.15.70:0/521992004 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab38005470 con 0x55eb7d3d0320 2024-06-04T05:16:40.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.527+0000 7fab51d19b80 1 -- 172.21.15.70:0/521992004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3d0320 msgr2=0x55eb7d3c5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:40.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.527+0000 7fab51d19b80 1 --2- 172.21.15.70:0/521992004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3d0320 0x55eb7d3c5270 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fab38009370 tx=0x7fab380093a0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.527+0000 7fab51d19b80 1 -- 172.21.15.70:0/521992004 shutdown_connections 2024-06-04T05:16:40.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.527+0000 7fab51d19b80 1 --2- 172.21.15.70:0/521992004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3d0320 0x55eb7d3c5270 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.527+0000 7fab51d19b80 1 --2- 172.21.15.70:0/521992004 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3ca8a0 0x55eb7d3d3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.527+0000 7fab51d19b80 1 --2- 172.21.15.70:0/521992004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3cb260 0x55eb7d3cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.527+0000 7fab51d19b80 1 -- 172.21.15.70:0/521992004 >> 172.21.15.70:0/521992004 conn(0x55eb7d2f6240 msgr2=0x55eb7d3cf1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:40.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.527+0000 7fab51d19b80 1 -- 172.21.15.70:0/521992004 shutdown_connections 2024-06-04T05:16:40.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.527+0000 7fab51d19b80 1 -- 172.21.15.70:0/521992004 wait complete. 2024-06-04T05:16:40.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.528+0000 7fab51d19b80 1 -- start start 2024-06-04T05:16:40.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.528+0000 7fab51d19b80 1 Processor -- start 2024-06-04T05:16:40.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.528+0000 7fab51d19b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3ca8a0 0x55eb7d3dcb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.528+0000 7fab51d19b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3cb260 0x55eb7d3dd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.528+0000 7fab51d19b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3d0320 0x55eb7d3eb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.528+0000 7fab51d19b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab401af250 con 0x55eb7d3cb260 2024-06-04T05:16:40.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.528+0000 7fab51d19b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fab401aeb80 con 0x55eb7d3d0320 2024-06-04T05:16:40.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.528+0000 7fab51d19b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fab401abda0 con 0x55eb7d3ca8a0 2024-06-04T05:16:40.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.528+0000 7fab51d19b80 10 client.0 _ll_register_callbacks cb 0x55eb7d2b3950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:40.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.528+0000 7fab44a1f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3cb260 0x55eb7d3dd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.528+0000 7fab44a1f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3cb260 0x55eb7d3dd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56506/0 (socket says 172.21.15.70:56506) 2024-06-04T05:16:40.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.528+0000 7fab44a1f700 1 -- 172.21.15.70:0/3705952396 learned_addr learned my addr 172.21.15.70:0/3705952396 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:40.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.529+0000 7fab45a21700 1 --2- 172.21.15.70:0/3705952396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3d0320 0x55eb7d3eb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.529+0000 7fab44a1f700 1 -- 172.21.15.70:0/3705952396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3ca8a0 msgr2=0x55eb7d3dcb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:40.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.529+0000 7fab45220700 1 --2- 172.21.15.70:0/3705952396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3ca8a0 0x55eb7d3dcb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.529+0000 7fab44a1f700 1 --2- 172.21.15.70:0/3705952396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3ca8a0 0x55eb7d3dcb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.529+0000 7fab44a1f700 1 -- 172.21.15.70:0/3705952396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3d0320 msgr2=0x55eb7d3eb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:40.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.529+0000 7fab44a1f700 1 --2- 172.21.15.70:0/3705952396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3d0320 0x55eb7d3eb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.529+0000 7fab44a1f700 1 -- 172.21.15.70:0/3705952396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb7d3ebbb0 con 0x55eb7d3cb260 2024-06-04T05:16:40.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.529+0000 7fab45220700 1 --2- 172.21.15.70:0/3705952396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3ca8a0 0x55eb7d3dcb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:16:40.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.529+0000 7fab44a1f700 1 --2- 172.21.15.70:0/3705952396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3cb260 0x55eb7d3dd070 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fab30010010 tx=0x7fab300253f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad14696779f20d1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:40.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.529+0000 7fab28ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:40.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.529+0000 7fab28ff9700 1 -- 172.21.15.70:0/3705952396 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab30004830 con 0x55eb7d3cb260 2024-06-04T05:16:40.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.529+0000 7fab28ff9700 1 -- 172.21.15.70:0/3705952396 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7fab300049d0 con 0x55eb7d3cb260 2024-06-04T05:16:40.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.529+0000 7fab51d19b80 1 -- 172.21.15.70:0/3705952396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab38003da0 con 0x55eb7d3cb260 2024-06-04T05:16:40.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.529+0000 7fab28ff9700 1 -- 172.21.15.70:0/3705952396 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fab30043050 con 0x55eb7d3cb260 2024-06-04T05:16:40.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.529+0000 7fab51d19b80 10 client.20499 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:40.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.529+0000 7fab51d19b80 1 -- 172.21.15.70:0/3705952396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55eb7d3dc4c0 con 0x55eb7d3cb260 2024-06-04T05:16:40.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.529+0000 7fab28ff9700 1 -- 172.21.15.70:0/3705952396 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab38003da0 con 0x55eb7d3cb260 2024-06-04T05:16:40.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.530+0000 7fab28ff9700 1 -- 172.21.15.70:0/3705952396 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fab30024040 con 0x55eb7d3cb260 2024-06-04T05:16:40.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.530+0000 7fab51d19b80 10 client.20499 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:40.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.530+0000 7fab51d19b80 10 client.20499 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:16:40.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.530+0000 7fab51d19b80 20 client.20499 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:40.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.530+0000 7fab51d19b80 1 --2- 172.21.15.70:0/3705952396 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55eb7d44f180 0x55eb7d4515a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:40.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.530+0000 7fab51d19b80 4 client.20499 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:16:40.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.530+0000 7fab51d19b80 1 -- 172.21.15.70:0/3705952396 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eb7d3244a0 con 0x55eb7d44f180 2024-06-04T05:16:40.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.530+0000 7fab45220700 1 --2- 172.21.15.70:0/3705952396 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55eb7d44f180 0x55eb7d4515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:40.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.530+0000 7fab28ff9700 10 client.20499 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:16:40.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.530+0000 7fab45220700 1 --2- 172.21.15.70:0/3705952396 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55eb7d44f180 0x55eb7d4515a0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fbe47026b572006 server_cookie=aa39f0a8dae8c7a7 in_seq=0 out_seq=0 2024-06-04T05:16:40.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.531+0000 7fab28ff9700 1 -- 172.21.15.70:0/3705952396 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eb7d3244a0 con 0x55eb7d44f180 2024-06-04T05:16:40.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.531+0000 7fab28ff9700 10 client.20499 handle_command_reply: tid=0 2024-06-04T05:16:40.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.531+0000 7fab51d19b80 1 client.20499 shutdown 2024-06-04T05:16:40.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.531+0000 7fab51d19b80 1 -- 172.21.15.70:0/3705952396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3cb260 msgr2=0x55eb7d3dd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:40.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.531+0000 7fab51d19b80 1 --2- 172.21.15.70:0/3705952396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3cb260 0x55eb7d3dd070 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fab30010010 tx=0x7fab300253f0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.531+0000 7fab51d19b80 1 -- 172.21.15.70:0/3705952396 shutdown_connections 2024-06-04T05:16:40.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.531+0000 7fab51d19b80 1 --2- 172.21.15.70:0/3705952396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb7d3d0320 0x55eb7d3eb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.531+0000 7fab51d19b80 1 --2- 172.21.15.70:0/3705952396 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55eb7d44f180 0x55eb7d4515a0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.531+0000 7fab51d19b80 1 --2- 172.21.15.70:0/3705952396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb7d3ca8a0 0x55eb7d3dcb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.531+0000 7fab51d19b80 1 --2- 172.21.15.70:0/3705952396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb7d3cb260 0x55eb7d3dd070 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.531+0000 7fab51d19b80 1 -- 172.21.15.70:0/3705952396 >> 172.21.15.70:0/3705952396 conn(0x55eb7d2f6240 msgr2=0x55eb7d3c9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:40.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.531+0000 7fab28ff9700 0 client.20499 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:16:40.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.531+0000 7fab51d19b80 1 -- 172.21.15.70:0/3705952396 shutdown_connections 2024-06-04T05:16:40.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.531+0000 7fab51d19b80 1 -- 172.21.15.70:0/3705952396 wait complete. 2024-06-04T05:16:40.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.531+0000 7fab51d19b80 20 client.20499 trim_cache size 0 max 16384 2024-06-04T05:16:40.538 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:16:40.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.531+0000 7fab47484700 1 -- 172.21.15.70:0/3639603105 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab2c0b29f0 msgr2=0x7fab2c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:40.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.531+0000 7fab47484700 1 --2- 172.21.15.70:0/3639603105 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab2c0b29f0 0x7fab2c0b4eb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fab40107d20 tx=0x7fab3000a300 comp rx=0 tx=0).stop 2024-06-04T05:16:40.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.532+0000 7fab47484700 1 -- 172.21.15.70:0/3639603105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab40102ba0 msgr2=0x7fab4019f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:40.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.532+0000 7fab47484700 1 --2- 172.21.15.70:0/3639603105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab40102ba0 0x7fab4019f8c0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00d980 tx=0x7fab3c00dc90 comp rx=0 tx=0).stop 2024-06-04T05:16:40.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.532+0000 7fab47484700 1 -- 172.21.15.70:0/3639603105 shutdown_connections 2024-06-04T05:16:40.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.532+0000 7fab47484700 1 --2- 172.21.15.70:0/3639603105 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fab2c0b29f0 0x7fab2c0b4eb0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.532+0000 7fab47484700 1 --2- 172.21.15.70:0/3639603105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab40108710 0x7fab40199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.532+0000 7fab47484700 1 --2- 172.21.15.70:0/3639603105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab40102ba0 0x7fab4019f8c0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.532+0000 7fab47484700 1 --2- 172.21.15.70:0/3639603105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab40103560 0x7fab4019fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:40.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.532+0000 7fab47484700 1 -- 172.21.15.70:0/3639603105 >> 172.21.15.70:0/3639603105 conn(0x7fab40078a40 msgr2=0x7fab40100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:40.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.532+0000 7fab47484700 1 -- 172.21.15.70:0/3639603105 shutdown_connections 2024-06-04T05:16:40.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:40.532+0000 7fab47484700 1 -- 172.21.15.70:0/3639603105 wait complete. 2024-06-04T05:16:40.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.85 MB/sec execute 212 sec latency 8246.714 ms 2024-06-04T05:16:41.263 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:16:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:16:41.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:41 smithi070 ceph-mon[32631]: pgmap v612: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 212 op/s 2024-06-04T05:16:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:41 smithi112 ceph-mon[36453]: pgmap v612: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 212 op/s 2024-06-04T05:16:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:41 smithi098 ceph-mon[35569]: pgmap v612: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 212 op/s 2024-06-04T05:16:41.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.81 MB/sec execute 213 sec latency 9246.798 ms 2024-06-04T05:16:42.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.78 MB/sec execute 214 sec latency 10246.840 ms 2024-06-04T05:16:43.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.443+0000 7f3bcadd2700 1 -- 172.21.15.70:0/2805507224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bc41085b0 msgr2=0x7f3bc4108990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:43.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.443+0000 7f3bcadd2700 1 --2- 172.21.15.70:0/2805507224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bc41085b0 0x7f3bc4108990 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f3bc000b0d0 tx=0x7f3bc000b3e0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.443+0000 7f3bcadd2700 1 -- 172.21.15.70:0/2805507224 shutdown_connections 2024-06-04T05:16:43.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.443+0000 7f3bcadd2700 1 --2- 172.21.15.70:0/2805507224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bc41028a0 0x7f3bc4110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.443+0000 7f3bcadd2700 1 --2- 172.21.15.70:0/2805507224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bc41085b0 0x7f3bc4108990 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.443+0000 7f3bcadd2700 1 --2- 172.21.15.70:0/2805507224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bc4101ee0 0x7f3bc4102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.443+0000 7f3bcadd2700 1 -- 172.21.15.70:0/2805507224 >> 172.21.15.70:0/2805507224 conn(0x7f3bc40777e0 msgr2=0x7f3bc4077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:43.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.444+0000 7f3bcadd2700 1 -- 172.21.15.70:0/2805507224 shutdown_connections 2024-06-04T05:16:43.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.444+0000 7f3bcadd2700 1 -- 172.21.15.70:0/2805507224 wait complete. 2024-06-04T05:16:43.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.444+0000 7f3bcadd2700 1 Processor -- start 2024-06-04T05:16:43.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.444+0000 7f3bcadd2700 1 -- start start 2024-06-04T05:16:43.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bcadd2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bc4101ee0 0x7f3bc419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bcadd2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bc41028a0 0x7f3bc419fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bcadd2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bc41085b0 0x7f3bc4199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bcadd2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc41135e0 con 0x7f3bc41028a0 2024-06-04T05:16:43.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bcadd2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc4113460 con 0x7f3bc4101ee0 2024-06-04T05:16:43.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bcadd2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc4113760 con 0x7f3bc41085b0 2024-06-04T05:16:43.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bbbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bc41028a0 0x7f3bc419fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:43.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bbbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bc41028a0 0x7f3bc419fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56524/0 (socket says 172.21.15.70:56524) 2024-06-04T05:16:43.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bbbfff700 1 -- 172.21.15.70:0/3530886794 learned_addr learned my addr 172.21.15.70:0/3530886794 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bc936f700 1 --2- 172.21.15.70:0/3530886794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bc41085b0 0x7f3bc4199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bc8b6e700 1 --2- 172.21.15.70:0/3530886794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bc4101ee0 0x7f3bc419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bbbfff700 1 -- 172.21.15.70:0/3530886794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bc41085b0 msgr2=0x7f3bc4199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bbbfff700 1 --2- 172.21.15.70:0/3530886794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bc41085b0 0x7f3bc4199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bbbfff700 1 -- 172.21.15.70:0/3530886794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bc4101ee0 msgr2=0x7f3bc419f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bbbfff700 1 --2- 172.21.15.70:0/3530886794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bc4101ee0 0x7f3bc419f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bbbfff700 1 -- 172.21.15.70:0/3530886794 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc419a080 con 0x7f3bc41028a0 2024-06-04T05:16:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bbbfff700 1 --2- 172.21.15.70:0/3530886794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bc41028a0 0x7f3bc419fc50 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400d980 tx=0x7f3bb400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.445+0000 7f3bb9ffb700 1 -- 172.21.15.70:0/3530886794 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4017070 con 0x7f3bc41028a0 2024-06-04T05:16:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.446+0000 7f3bb9ffb700 1 -- 172.21.15.70:0/3530886794 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3bb4005020 con 0x7f3bc41028a0 2024-06-04T05:16:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.446+0000 7f3bcadd2700 1 -- 172.21.15.70:0/3530886794 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc419a370 con 0x7f3bc41028a0 2024-06-04T05:16:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.446+0000 7f3bb9ffb700 1 -- 172.21.15.70:0/3530886794 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb400f980 con 0x7f3bc41028a0 2024-06-04T05:16:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.446+0000 7f3bcadd2700 1 -- 172.21.15.70:0/3530886794 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc4102ff0 con 0x7f3bc41028a0 2024-06-04T05:16:43.451 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:43 smithi070 ceph-mon[32631]: pgmap v613: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 220 op/s 2024-06-04T05:16:43.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.448+0000 7f3bb9ffb700 1 -- 172.21.15.70:0/3530886794 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3bb4004a30 con 0x7f3bc41028a0 2024-06-04T05:16:43.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.449+0000 7f3bb9ffb700 1 --2- 172.21.15.70:0/3530886794 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3bac0b29f0 0x7f3bac0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.449+0000 7f3bc8b6e700 1 --2- 172.21.15.70:0/3530886794 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3bac0b29f0 0x7f3bac0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:43.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.449+0000 7f3bb9ffb700 1 -- 172.21.15.70:0/3530886794 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f3bb40e24c0 con 0x7f3bc41028a0 2024-06-04T05:16:43.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.450+0000 7f3bc8b6e700 1 --2- 172.21.15.70:0/3530886794 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3bac0b29f0 0x7f3bac0b4eb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009fd0 tx=0x7f3bc000b040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:43.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bd5667b80 1 Processor -- start 2024-06-04T05:16:43.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bd5667b80 1 -- start start 2024-06-04T05:16:43.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bd5667b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a2860 0x55ffbd7a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bd5667b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a3210 0x55ffbd7a6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bd5667b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a6f60 0x55ffbd7a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bd5667b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ffbd701c90 con 0x55ffbd7a3210 2024-06-04T05:16:43.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bd5667b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ffbd6f1830 con 0x55ffbd7a6f60 2024-06-04T05:16:43.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bd5667b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ffbd6f5870 con 0x55ffbd7a2860 2024-06-04T05:16:43.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bbbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a3210 0x55ffbd7a6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bbbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a3210 0x55ffbd7a6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56528/0 (socket says 172.21.15.70:56528) 2024-06-04T05:16:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bbbfff700 1 -- 172.21.15.70:0/1084261452 learned_addr learned my addr 172.21.15.70:0/1084261452 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bc8b6e700 1 --2- 172.21.15.70:0/1084261452 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a2860 0x55ffbd7a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bc936f700 1 --2- 172.21.15.70:0/1084261452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a6f60 0x55ffbd7a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bbbfff700 1 -- 172.21.15.70:0/1084261452 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a2860 msgr2=0x55ffbd7a2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bbbfff700 1 --2- 172.21.15.70:0/1084261452 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a2860 0x55ffbd7a2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bbbfff700 1 -- 172.21.15.70:0/1084261452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a6f60 msgr2=0x55ffbd7a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bbbfff700 1 --2- 172.21.15.70:0/1084261452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a6f60 0x55ffbd7a73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bbbfff700 1 -- 172.21.15.70:0/1084261452 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffbd6418d0 con 0x55ffbd7a3210 2024-06-04T05:16:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bc8b6e700 1 --2- 172.21.15.70:0/1084261452 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a2860 0x55ffbd7a2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:16:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.451+0000 7f3bc936f700 1 --2- 172.21.15.70:0/1084261452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a6f60 0x55ffbd7a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:16:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.452+0000 7f3bbbfff700 1 --2- 172.21.15.70:0/1084261452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a3210 0x55ffbd7a6940 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40a9470 tx=0x7f3bb4004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=560a81ce3f77900c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.452+0000 7f3baaffd700 1 -- 172.21.15.70:0/1084261452 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb40a9770 con 0x55ffbd7a3210 2024-06-04T05:16:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.452+0000 7f3baaffd700 1 -- 172.21.15.70:0/1084261452 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3bb40adb20 con 0x55ffbd7a3210 2024-06-04T05:16:43.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.452+0000 7f3baaffd700 1 -- 172.21.15.70:0/1084261452 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb40a9c30 con 0x55ffbd7a3210 2024-06-04T05:16:43.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.453+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1084261452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a3210 msgr2=0x55ffbd7a6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:43.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.453+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/1084261452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a3210 0x55ffbd7a6940 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40a9470 tx=0x7f3bb4004df0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.453+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1084261452 shutdown_connections 2024-06-04T05:16:43.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.453+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/1084261452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a6f60 0x55ffbd7a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.453+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/1084261452 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a2860 0x55ffbd7a2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.453+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/1084261452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a3210 0x55ffbd7a6940 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.453+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1084261452 >> 172.21.15.70:0/1084261452 conn(0x55ffbd6d0080 msgr2=0x55ffbd79ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:43.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.453+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1084261452 shutdown_connections 2024-06-04T05:16:43.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.453+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1084261452 wait complete. 2024-06-04T05:16:43.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.453+0000 7f3bd5667b80 1 -- start start 2024-06-04T05:16:43.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.453+0000 7f3bd5667b80 1 Processor -- start 2024-06-04T05:16:43.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bd5667b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a2860 0x55ffbd7ae520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bd5667b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a3210 0x55ffbd7ab890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bd5667b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a6f60 0x55ffbd7abdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bd5667b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ffbd5f7fa0 con 0x55ffbd7a3210 2024-06-04T05:16:43.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bd5667b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ffbd6937c0 con 0x55ffbd7a6f60 2024-06-04T05:16:43.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bd5667b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ffbd7af0c0 con 0x55ffbd7a2860 2024-06-04T05:16:43.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bd5667b80 10 client.0 _ll_register_callbacks cb 0x55ffbd68c920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:43.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bbbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a3210 0x55ffbd7ab890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:43.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bc8b6e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a2860 0x55ffbd7ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:43.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bc8b6e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a2860 0x55ffbd7ae520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:33608/0 (socket says 172.21.15.70:33608) 2024-06-04T05:16:43.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bbbfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a3210 0x55ffbd7ab890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56544/0 (socket says 172.21.15.70:56544) 2024-06-04T05:16:43.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bc8b6e700 1 -- 172.21.15.70:0/2334046186 learned_addr learned my addr 172.21.15.70:0/2334046186 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:43.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bc936f700 1 --2- 172.21.15.70:0/2334046186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a6f60 0x55ffbd7abdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:43.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bbbfff700 1 -- 172.21.15.70:0/2334046186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a2860 msgr2=0x55ffbd7ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:43.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bbbfff700 1 --2- 172.21.15.70:0/2334046186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a2860 0x55ffbd7ae520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bbbfff700 1 -- 172.21.15.70:0/2334046186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a6f60 msgr2=0x55ffbd7abdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:43.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bbbfff700 1 --2- 172.21.15.70:0/2334046186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a6f60 0x55ffbd7abdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bbbfff700 1 -- 172.21.15.70:0/2334046186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffbd690690 con 0x55ffbd7a3210 2024-06-04T05:16:43.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bc8b6e700 1 --2- 172.21.15.70:0/2334046186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a2860 0x55ffbd7ae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:16:43.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3bbbfff700 1 --2- 172.21.15.70:0/2334046186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a3210 0x55ffbd7ab890 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40a9230 tx=0x7f3bb4005440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e8a6165a3ff7a0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:43.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.454+0000 7f3ba3fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:43.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.455+0000 7f3ba3fff700 1 -- 172.21.15.70:0/2334046186 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb40a9770 con 0x55ffbd7a3210 2024-06-04T05:16:43.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.455+0000 7f3ba3fff700 1 -- 172.21.15.70:0/2334046186 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f3bb40a9be0 con 0x55ffbd7a3210 2024-06-04T05:16:43.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.455+0000 7f3bd5667b80 1 -- 172.21.15.70:0/2334046186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bb40adb20 con 0x55ffbd7a3210 2024-06-04T05:16:43.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.455+0000 7f3ba3fff700 1 -- 172.21.15.70:0/2334046186 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f3bb40b25d0 con 0x55ffbd7a3210 2024-06-04T05:16:43.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.455+0000 7f3bd5667b80 10 client.20511 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:43.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.455+0000 7f3bd5667b80 1 -- 172.21.15.70:0/2334046186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55ffbd7b99b0 con 0x55ffbd7a3210 2024-06-04T05:16:43.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.455+0000 7f3ba3fff700 1 -- 172.21.15.70:0/2334046186 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3bb40adb20 con 0x55ffbd7a3210 2024-06-04T05:16:43.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.455+0000 7f3ba3fff700 1 -- 172.21.15.70:0/2334046186 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f3bb40bec50 con 0x55ffbd7a3210 2024-06-04T05:16:43.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.455+0000 7f3bd5667b80 10 client.20511 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:43.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.455+0000 7f3bd5667b80 10 client.20511 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:16:43.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.455+0000 7f3bd5667b80 20 client.20511 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:43.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.455+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/2334046186 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55ffbd7bbe80 0x55ffbd7be270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.456+0000 7f3bd5667b80 4 client.20511 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:16:43.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.456+0000 7f3bd5667b80 1 -- 172.21.15.70:0/2334046186 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffbd76e9e0 con 0x55ffbd7bbe80 2024-06-04T05:16:43.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.456+0000 7f3bc8b6e700 1 --2- 172.21.15.70:0/2334046186 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55ffbd7bbe80 0x55ffbd7be270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:43.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.456+0000 7f3bc8b6e700 1 --2- 172.21.15.70:0/2334046186 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55ffbd7bbe80 0x55ffbd7be270 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dec5b6cc11b4f19d server_cookie=1e801abfdeb85c70 in_seq=0 out_seq=0 2024-06-04T05:16:43.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.456+0000 7f3ba3fff700 10 client.20511 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:16:43.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.457+0000 7f3ba3fff700 1 -- 172.21.15.70:0/2334046186 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ffbd76e9e0 con 0x55ffbd7bbe80 2024-06-04T05:16:43.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.457+0000 7f3ba3fff700 10 client.20511 handle_command_reply: tid=0 2024-06-04T05:16:43.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.457+0000 7f3bd5667b80 1 client.20511 shutdown 2024-06-04T05:16:43.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.457+0000 7f3bd5667b80 1 -- 172.21.15.70:0/2334046186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a3210 msgr2=0x55ffbd7ab890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:43.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.457+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/2334046186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a3210 0x55ffbd7ab890 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40a9230 tx=0x7f3bb4005440 comp rx=0 tx=0).stop 2024-06-04T05:16:43.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.457+0000 7f3bd5667b80 1 -- 172.21.15.70:0/2334046186 shutdown_connections 2024-06-04T05:16:43.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.457+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/2334046186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a6f60 0x55ffbd7abdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.457+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/2334046186 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55ffbd7bbe80 0x55ffbd7be270 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.457+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/2334046186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a2860 0x55ffbd7ae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.457+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/2334046186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a3210 0x55ffbd7ab890 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.457+0000 7f3bd5667b80 1 -- 172.21.15.70:0/2334046186 >> 172.21.15.70:0/2334046186 conn(0x55ffbd6d0080 msgr2=0x55ffbd79c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:43.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.457+0000 7f3ba3fff700 0 client.20511 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:16:43.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.457+0000 7f3bd5667b80 1 -- 172.21.15.70:0/2334046186 shutdown_connections 2024-06-04T05:16:43.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.457+0000 7f3bd5667b80 1 -- 172.21.15.70:0/2334046186 wait complete. 2024-06-04T05:16:43.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.457+0000 7f3bd5667b80 20 client.20511 trim_cache size 0 max 16384 2024-06-04T05:16:43.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bd5667b80 1 Processor -- start 2024-06-04T05:16:43.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bd5667b80 1 -- start start 2024-06-04T05:16:43.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bd5667b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a2860 0x55ffbd79cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bd5667b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a3210 0x55ffbd7a8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bd5667b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a6f60 0x55ffbd7a91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bd5667b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc41aff20 con 0x55ffbd7a2860 2024-06-04T05:16:43.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bd5667b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc41b1a60 con 0x55ffbd7a3210 2024-06-04T05:16:43.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bd5667b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc41b05f0 con 0x55ffbd7a6f60 2024-06-04T05:16:43.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bc8b6e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a2860 0x55ffbd79cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:43.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bc8b6e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a2860 0x55ffbd79cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56558/0 (socket says 172.21.15.70:56558) 2024-06-04T05:16:43.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bc8b6e700 1 -- 172.21.15.70:0/1792215194 learned_addr learned my addr 172.21.15.70:0/1792215194 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:43.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bbbfff700 1 --2- 172.21.15.70:0/1792215194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a3210 0x55ffbd7a8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:43.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bc8b6e700 1 -- 172.21.15.70:0/1792215194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a6f60 msgr2=0x55ffbd7a91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:43.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bc8b6e700 1 --2- 172.21.15.70:0/1792215194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a6f60 0x55ffbd7a91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bc8b6e700 1 -- 172.21.15.70:0/1792215194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a3210 msgr2=0x55ffbd7a8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:43.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bc8b6e700 1 --2- 172.21.15.70:0/1792215194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a3210 0x55ffbd7a8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bc8b6e700 1 -- 172.21.15.70:0/1792215194 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffbd7a9e00 con 0x55ffbd7a2860 2024-06-04T05:16:43.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.473+0000 7f3bc8b6e700 1 --2- 172.21.15.70:0/1792215194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a2860 0x55ffbd79cc90 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0006130 tx=0x7f3bc001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bd568b120442a8c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:43.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.474+0000 7f3baa7fc700 1 -- 172.21.15.70:0/1792215194 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc00074e0 con 0x55ffbd7a2860 2024-06-04T05:16:43.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.474+0000 7f3baa7fc700 1 -- 172.21.15.70:0/1792215194 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3bc0007b10 con 0x55ffbd7a2860 2024-06-04T05:16:43.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.474+0000 7f3baa7fc700 1 -- 172.21.15.70:0/1792215194 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc0033970 con 0x55ffbd7a2860 2024-06-04T05:16:43.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.474+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1792215194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a2860 msgr2=0x55ffbd79cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:43.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.474+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/1792215194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a2860 0x55ffbd79cc90 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0006130 tx=0x7f3bc001e000 comp rx=0 tx=0).stop 2024-06-04T05:16:43.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.474+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1792215194 shutdown_connections 2024-06-04T05:16:43.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.474+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/1792215194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a6f60 0x55ffbd7a91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.474+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/1792215194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a2860 0x55ffbd79cc90 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.474+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/1792215194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a3210 0x55ffbd7a8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.474+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1792215194 >> 172.21.15.70:0/1792215194 conn(0x55ffbd6d0080 msgr2=0x55ffbd7ad950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:43.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.474+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1792215194 shutdown_connections 2024-06-04T05:16:43.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.474+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1792215194 wait complete. 2024-06-04T05:16:43.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.475+0000 7f3bd5667b80 1 -- start start 2024-06-04T05:16:43.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.475+0000 7f3bd5667b80 1 Processor -- start 2024-06-04T05:16:43.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.475+0000 7f3bd5667b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a2860 0x55ffbd7bfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.475+0000 7f3bd5667b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a3210 0x55ffbd7c01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.475+0000 7f3bd5667b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a6f60 0x55ffbd7c0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.475+0000 7f3bd5667b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc41b1390 con 0x55ffbd7a2860 2024-06-04T05:16:43.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.475+0000 7f3bd5667b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc41b0cc0 con 0x55ffbd7a6f60 2024-06-04T05:16:43.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.475+0000 7f3bd5667b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc41adee0 con 0x55ffbd7a3210 2024-06-04T05:16:43.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.475+0000 7f3bd5667b80 10 client.0 _ll_register_callbacks cb 0x55ffbd68c920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:43.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.475+0000 7f3bc8b6e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a2860 0x55ffbd7bfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:43.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.475+0000 7f3bc8b6e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a2860 0x55ffbd7bfca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56566/0 (socket says 172.21.15.70:56566) 2024-06-04T05:16:43.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.475+0000 7f3bc8b6e700 1 -- 172.21.15.70:0/1179292285 learned_addr learned my addr 172.21.15.70:0/1179292285 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:43.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.476+0000 7f3bbbfff700 1 --2- 172.21.15.70:0/1179292285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a3210 0x55ffbd7c01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:43.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.476+0000 7f3bc936f700 1 --2- 172.21.15.70:0/1179292285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a6f60 0x55ffbd7c0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:43.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.476+0000 7f3bc8b6e700 1 -- 172.21.15.70:0/1179292285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a3210 msgr2=0x55ffbd7c01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.476+0000 7f3bc8b6e700 1 --2- 172.21.15.70:0/1179292285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a3210 0x55ffbd7c01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.476+0000 7f3bc8b6e700 1 -- 172.21.15.70:0/1179292285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a6f60 msgr2=0x55ffbd7c0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.476+0000 7f3bc8b6e700 1 --2- 172.21.15.70:0/1179292285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a6f60 0x55ffbd7c0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.476+0000 7f3bc8b6e700 1 -- 172.21.15.70:0/1179292285 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffbd824740 con 0x55ffbd7a2860 2024-06-04T05:16:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.476+0000 7f3bc936f700 1 --2- 172.21.15.70:0/1179292285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a6f60 0x55ffbd7c0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:16:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.476+0000 7f3bbbfff700 1 --2- 172.21.15.70:0/1179292285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a3210 0x55ffbd7c01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:16:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.476+0000 7f3bc8b6e700 1 --2- 172.21.15.70:0/1179292285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a2860 0x55ffbd7bfca0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009610 tx=0x7f3bc00051e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0640a60ce6700a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.476+0000 7f3ba3fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.476+0000 7f3ba3fff700 1 -- 172.21.15.70:0/1179292285 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc0044060 con 0x55ffbd7a2860 2024-06-04T05:16:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.476+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1179292285 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bc0007b10 con 0x55ffbd7a2860 2024-06-04T05:16:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.476+0000 7f3ba3fff700 1 -- 172.21.15.70:0/1179292285 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f3bc00074a0 con 0x55ffbd7a2860 2024-06-04T05:16:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.476+0000 7f3ba3fff700 1 -- 172.21.15.70:0/1179292285 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f3bc00076a0 con 0x55ffbd7a2860 2024-06-04T05:16:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.476+0000 7f3bd5667b80 10 client.20517 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:43.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.476+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1179292285 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55ffbd825a40 con 0x55ffbd7a2860 2024-06-04T05:16:43.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.477+0000 7f3ba3fff700 1 -- 172.21.15.70:0/1179292285 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3bc0007b10 con 0x55ffbd7a2860 2024-06-04T05:16:43.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.477+0000 7f3ba3fff700 1 -- 172.21.15.70:0/1179292285 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f3bc0011070 con 0x55ffbd7a2860 2024-06-04T05:16:43.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.477+0000 7f3bd5667b80 10 client.20517 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:43.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.477+0000 7f3bd5667b80 10 client.20517 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:16:43.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.477+0000 7f3bd5667b80 20 client.20517 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:43.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.477+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/1179292285 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55ffbd826060 0x55ffbd828480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:43.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.477+0000 7f3bd5667b80 4 client.20517 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:16:43.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.477+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1179292285 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ffbd703be0 con 0x55ffbd826060 2024-06-04T05:16:43.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.477+0000 7f3bbbfff700 1 --2- 172.21.15.70:0/1179292285 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55ffbd826060 0x55ffbd828480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:43.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.477+0000 7f3ba3fff700 10 client.20517 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:16:43.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.477+0000 7f3bbbfff700 1 --2- 172.21.15.70:0/1179292285 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55ffbd826060 0x55ffbd828480 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c4bb15d89a45745e server_cookie=93375b8e61dcc53e in_seq=0 out_seq=0 2024-06-04T05:16:43.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.478+0000 7f3ba3fff700 1 -- 172.21.15.70:0/1179292285 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ffbd703be0 con 0x55ffbd826060 2024-06-04T05:16:43.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.478+0000 7f3ba3fff700 10 client.20517 handle_command_reply: tid=0 2024-06-04T05:16:43.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.478+0000 7f3bd5667b80 1 client.20517 shutdown 2024-06-04T05:16:43.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.478+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1179292285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a2860 msgr2=0x55ffbd7bfca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:43.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.478+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/1179292285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a2860 0x55ffbd7bfca0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009610 tx=0x7f3bc00051e0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.478+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1179292285 shutdown_connections 2024-06-04T05:16:43.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.478+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/1179292285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffbd7a6f60 0x55ffbd7c0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.478+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/1179292285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ffbd7a2860 0x55ffbd7bfca0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.478+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/1179292285 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55ffbd826060 0x55ffbd828480 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.478+0000 7f3bd5667b80 1 --2- 172.21.15.70:0/1179292285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ffbd7a3210 0x55ffbd7c01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.478+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1179292285 >> 172.21.15.70:0/1179292285 conn(0x55ffbd6d0080 msgr2=0x55ffbd7a58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:43.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.478+0000 7f3ba3fff700 0 client.20517 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:16:43.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.478+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1179292285 shutdown_connections 2024-06-04T05:16:43.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.478+0000 7f3bd5667b80 1 -- 172.21.15.70:0/1179292285 wait complete. 2024-06-04T05:16:43.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.478+0000 7f3bd5667b80 20 client.20517 trim_cache size 0 max 16384 2024-06-04T05:16:43.486 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:16:43.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.479+0000 7f3bcadd2700 1 -- 172.21.15.70:0/3530886794 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3bac0b29f0 msgr2=0x7f3bac0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:43.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.479+0000 7f3bcadd2700 1 --2- 172.21.15.70:0/3530886794 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3bac0b29f0 0x7f3bac0b4eb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009fd0 tx=0x7f3bc000b040 comp rx=0 tx=0).stop 2024-06-04T05:16:43.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.479+0000 7f3bcadd2700 1 -- 172.21.15.70:0/3530886794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bc41028a0 msgr2=0x7f3bc419fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:43.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.479+0000 7f3bcadd2700 1 --2- 172.21.15.70:0/3530886794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bc41028a0 0x7f3bc419fc50 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400d980 tx=0x7f3bb400dc90 comp rx=0 tx=0).stop 2024-06-04T05:16:43.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.479+0000 7f3bcadd2700 1 -- 172.21.15.70:0/3530886794 shutdown_connections 2024-06-04T05:16:43.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.479+0000 7f3bcadd2700 1 --2- 172.21.15.70:0/3530886794 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3bac0b29f0 0x7f3bac0b4eb0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.479+0000 7f3bcadd2700 1 --2- 172.21.15.70:0/3530886794 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bc41085b0 0x7f3bc4199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.479+0000 7f3bcadd2700 1 --2- 172.21.15.70:0/3530886794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bc4101ee0 0x7f3bc419f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.479+0000 7f3bcadd2700 1 --2- 172.21.15.70:0/3530886794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bc41028a0 0x7f3bc419fc50 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:43.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.479+0000 7f3bcadd2700 1 -- 172.21.15.70:0/3530886794 >> 172.21.15.70:0/3530886794 conn(0x7f3bc40777e0 msgr2=0x7f3bc40fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:43.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.479+0000 7f3bcadd2700 1 -- 172.21.15.70:0/3530886794 shutdown_connections 2024-06-04T05:16:43.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:43.479+0000 7f3bcadd2700 1 -- 172.21.15.70:0/3530886794 wait complete. 2024-06-04T05:16:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:43 smithi112 ceph-mon[36453]: pgmap v613: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 220 op/s 2024-06-04T05:16:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:43 smithi098 ceph-mon[35569]: pgmap v613: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 220 op/s 2024-06-04T05:16:43.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.75 MB/sec execute 215 sec latency 11246.882 ms 2024-06-04T05:16:44.199 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:16:44.447 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:16:44.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.72 MB/sec execute 216 sec latency 12246.923 ms 2024-06-04T05:16:45.590 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:45 smithi070 ceph-mon[32631]: pgmap v614: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-06-04T05:16:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:45 smithi112 ceph-mon[36453]: pgmap v614: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-06-04T05:16:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:45 smithi098 ceph-mon[35569]: pgmap v614: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-06-04T05:16:45.820 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.69 MB/sec execute 217 sec latency 13247.011 ms 2024-06-04T05:16:46.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.245+0000 7f3711b80700 1 -- 172.21.15.70:0/3833243028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f370c106f20 msgr2=0x7f370c10eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:46.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.245+0000 7f3711b80700 1 --2- 172.21.15.70:0/3833243028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f370c106f20 0x7f370c10eaf0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f370400b0d0 tx=0x7f370400b3e0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.246+0000 7f3711b80700 1 -- 172.21.15.70:0/3833243028 shutdown_connections 2024-06-04T05:16:46.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.246+0000 7f3711b80700 1 --2- 172.21.15.70:0/3833243028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f370c106f20 0x7f370c10eaf0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.246+0000 7f3711b80700 1 --2- 172.21.15.70:0/3833243028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f370c101e40 0x7f370c102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.246+0000 7f3711b80700 1 --2- 172.21.15.70:0/3833243028 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f370c1027f0 0x7f370c106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.246+0000 7f3711b80700 1 -- 172.21.15.70:0/3833243028 >> 172.21.15.70:0/3833243028 conn(0x7f370c078a40 msgr2=0x7f370c0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:46.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.246+0000 7f3711b80700 1 -- 172.21.15.70:0/3833243028 shutdown_connections 2024-06-04T05:16:46.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.246+0000 7f3711b80700 1 -- 172.21.15.70:0/3833243028 wait complete. 2024-06-04T05:16:46.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.246+0000 7f3711b80700 1 Processor -- start 2024-06-04T05:16:46.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.246+0000 7f3711b80700 1 -- start start 2024-06-04T05:16:46.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.247+0000 7f3711b80700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f370c101e40 0x7f370c19f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.247+0000 7f3711b80700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f370c1027f0 0x7f370c19fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.247+0000 7f3711b80700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f370c106f20 0x7f370c199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.247+0000 7f3711b80700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f370c111450 con 0x7f370c101e40 2024-06-04T05:16:46.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.247+0000 7f3711b80700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f370c1112d0 con 0x7f370c1027f0 2024-06-04T05:16:46.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.247+0000 7f3711b80700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f370c1115d0 con 0x7f370c106f20 2024-06-04T05:16:46.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.247+0000 7f370b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f370c101e40 0x7f370c19f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.247+0000 7f370b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f370c101e40 0x7f370c19f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47090/0 (socket says 172.21.15.70:47090) 2024-06-04T05:16:46.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.247+0000 7f370b7fe700 1 -- 172.21.15.70:0/2483652703 learned_addr learned my addr 172.21.15.70:0/2483652703 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:46.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.247+0000 7f370affd700 1 --2- 172.21.15.70:0/2483652703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f370c1027f0 0x7f370c19fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.247+0000 7f370bfff700 1 --2- 172.21.15.70:0/2483652703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f370c106f20 0x7f370c199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.247+0000 7f370b7fe700 1 -- 172.21.15.70:0/2483652703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f370c106f20 msgr2=0x7f370c199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:46.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.247+0000 7f370b7fe700 1 --2- 172.21.15.70:0/2483652703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f370c106f20 0x7f370c199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.247+0000 7f370b7fe700 1 -- 172.21.15.70:0/2483652703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f370c1027f0 msgr2=0x7f370c19fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:46.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.247+0000 7f370b7fe700 1 --2- 172.21.15.70:0/2483652703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f370c1027f0 0x7f370c19fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.247+0000 7f370b7fe700 1 -- 172.21.15.70:0/2483652703 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370c19a0f0 con 0x7f370c101e40 2024-06-04T05:16:46.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.247+0000 7f370b7fe700 1 --2- 172.21.15.70:0/2483652703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f370c101e40 0x7f370c19f7d0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f36fc00ddc0 tx=0x7f36fc00a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:46.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.248+0000 7f3708ff9700 1 -- 172.21.15.70:0/2483652703 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36fc01b070 con 0x7f370c101e40 2024-06-04T05:16:46.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.248+0000 7f3711b80700 1 -- 172.21.15.70:0/2483652703 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f370c19a3e0 con 0x7f370c101e40 2024-06-04T05:16:46.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.248+0000 7f3708ff9700 1 -- 172.21.15.70:0/2483652703 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f36fc00ac90 con 0x7f370c101e40 2024-06-04T05:16:46.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.248+0000 7f3708ff9700 1 -- 172.21.15.70:0/2483652703 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36fc0166a0 con 0x7f370c101e40 2024-06-04T05:16:46.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.248+0000 7f3711b80700 1 -- 172.21.15.70:0/2483652703 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f370c1a66a0 con 0x7f370c101e40 2024-06-04T05:16:46.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.250+0000 7f3708ff9700 1 -- 172.21.15.70:0/2483652703 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f36fc01e070 con 0x7f370c101e40 2024-06-04T05:16:46.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.251+0000 7f3708ff9700 1 --2- 172.21.15.70:0/2483652703 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f36f40b29f0 0x7f36f40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.251+0000 7f370affd700 1 --2- 172.21.15.70:0/2483652703 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f36f40b29f0 0x7f36f40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.251+0000 7f3708ff9700 1 -- 172.21.15.70:0/2483652703 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f36fc0e2050 con 0x7f370c101e40 2024-06-04T05:16:46.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.251+0000 7f370affd700 1 --2- 172.21.15.70:0/2483652703 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f36f40b29f0 0x7f36f40b4eb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f370c104a70 tx=0x7f3700009190 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:46.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.251+0000 7f371c415b80 1 Processor -- start 2024-06-04T05:16:46.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.251+0000 7f371c415b80 1 -- start start 2024-06-04T05:16:46.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f371c415b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4c860 0x561d05f4cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f371c415b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f4d210 0x561d05f50940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f371c415b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f50f60 0x561d05f513a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f371c415b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561d05eabc90 con 0x561d05f4c860 2024-06-04T05:16:46.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f371c415b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561d05e9b830 con 0x561d05f50f60 2024-06-04T05:16:46.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f371c415b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561d05e9f870 con 0x561d05f4d210 2024-06-04T05:16:46.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f370b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4c860 0x561d05f4cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f370b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4c860 0x561d05f4cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47098/0 (socket says 172.21.15.70:47098) 2024-06-04T05:16:46.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f370b7fe700 1 -- 172.21.15.70:0/600719303 learned_addr learned my addr 172.21.15.70:0/600719303 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:46.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f370bfff700 1 --2- 172.21.15.70:0/600719303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f50f60 0x561d05f513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f370affd700 1 --2- 172.21.15.70:0/600719303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f4d210 0x561d05f50940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f370b7fe700 1 -- 172.21.15.70:0/600719303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f4d210 msgr2=0x561d05f50940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:46.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f370b7fe700 1 --2- 172.21.15.70:0/600719303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f4d210 0x561d05f50940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f370b7fe700 1 -- 172.21.15.70:0/600719303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f50f60 msgr2=0x561d05f513a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:46.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f370b7fe700 1 --2- 172.21.15.70:0/600719303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f50f60 0x561d05f513a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f370b7fe700 1 -- 172.21.15.70:0/600719303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d05deb8d0 con 0x561d05f4c860 2024-06-04T05:16:46.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f370bfff700 1 --2- 172.21.15.70:0/600719303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f50f60 0x561d05f513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:16:46.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f370b7fe700 1 --2- 172.21.15.70:0/600719303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4c860 0x561d05f4cc40 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f36fc0a5490 tx=0x7f36fc0a8b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36198cb4b6a191c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:46.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f36f1ffb700 1 -- 172.21.15.70:0/600719303 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36fc0a9410 con 0x561d05f4c860 2024-06-04T05:16:46.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f36f1ffb700 1 -- 172.21.15.70:0/600719303 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f36fc0ad7c0 con 0x561d05f4c860 2024-06-04T05:16:46.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.252+0000 7f36f1ffb700 1 -- 172.21.15.70:0/600719303 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36fc0a3d90 con 0x561d05f4c860 2024-06-04T05:16:46.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.253+0000 7f371c415b80 1 -- 172.21.15.70:0/600719303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4c860 msgr2=0x561d05f4cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:46.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.253+0000 7f371c415b80 1 --2- 172.21.15.70:0/600719303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4c860 0x561d05f4cc40 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f36fc0a5490 tx=0x7f36fc0a8b80 comp rx=0 tx=0).stop 2024-06-04T05:16:46.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.253+0000 7f371c415b80 1 -- 172.21.15.70:0/600719303 shutdown_connections 2024-06-04T05:16:46.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.253+0000 7f371c415b80 1 --2- 172.21.15.70:0/600719303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f50f60 0x561d05f513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.253+0000 7f371c415b80 1 --2- 172.21.15.70:0/600719303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4c860 0x561d05f4cc40 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.253+0000 7f371c415b80 1 --2- 172.21.15.70:0/600719303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f4d210 0x561d05f50940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.253+0000 7f371c415b80 1 -- 172.21.15.70:0/600719303 >> 172.21.15.70:0/600719303 conn(0x561d05e7a080 msgr2=0x561d05f46e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:46.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.253+0000 7f371c415b80 1 -- 172.21.15.70:0/600719303 shutdown_connections 2024-06-04T05:16:46.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.253+0000 7f371c415b80 1 -- 172.21.15.70:0/600719303 wait complete. 2024-06-04T05:16:46.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.253+0000 7f371c415b80 1 -- start start 2024-06-04T05:16:46.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f371c415b80 1 Processor -- start 2024-06-04T05:16:46.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f371c415b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f4c860 0x561d05f584d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f371c415b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05f557d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f371c415b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f50f60 0x561d05f55d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f371c415b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561d05da1fa0 con 0x561d05f4d210 2024-06-04T05:16:46.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f371c415b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561d05e3d7c0 con 0x561d05f50f60 2024-06-04T05:16:46.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f371c415b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561d05f59120 con 0x561d05f4c860 2024-06-04T05:16:46.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f371c415b80 10 client.0 _ll_register_callbacks cb 0x561d05e36920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:46.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f370affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05f557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f370affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05f557d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47108/0 (socket says 172.21.15.70:47108) 2024-06-04T05:16:46.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f370affd700 1 -- 172.21.15.70:0/2010125373 learned_addr learned my addr 172.21.15.70:0/2010125373 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:46.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f370b7fe700 1 --2- 172.21.15.70:0/2010125373 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f4c860 0x561d05f584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f370bfff700 1 --2- 172.21.15.70:0/2010125373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f50f60 0x561d05f55d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f370affd700 1 -- 172.21.15.70:0/2010125373 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f4c860 msgr2=0x561d05f584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:46.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f370affd700 1 --2- 172.21.15.70:0/2010125373 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f4c860 0x561d05f584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f370affd700 1 -- 172.21.15.70:0/2010125373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f50f60 msgr2=0x561d05f55d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:46.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f370affd700 1 --2- 172.21.15.70:0/2010125373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f50f60 0x561d05f55d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f370affd700 1 -- 172.21.15.70:0/2010125373 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d05e3a690 con 0x561d05f4d210 2024-06-04T05:16:46.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.254+0000 7f370bfff700 1 --2- 172.21.15.70:0/2010125373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f50f60 0x561d05f55d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:16:46.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.255+0000 7f370affd700 1 --2- 172.21.15.70:0/2010125373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05f557d0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f3700015700 tx=0x7f3700004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c74e673ebba5526 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:46.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.255+0000 7f36eaffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:46.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.255+0000 7f36eaffd700 1 -- 172.21.15.70:0/2010125373 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3700025020 con 0x561d05f4d210 2024-06-04T05:16:46.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.255+0000 7f371c415b80 1 -- 172.21.15.70:0/2010125373 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36fc0ad7c0 con 0x561d05f4d210 2024-06-04T05:16:46.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.255+0000 7f36eaffd700 1 -- 172.21.15.70:0/2010125373 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f3700015d20 con 0x561d05f4d210 2024-06-04T05:16:46.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.255+0000 7f36eaffd700 1 -- 172.21.15.70:0/2010125373 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f3700030050 con 0x561d05f4d210 2024-06-04T05:16:46.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.255+0000 7f371c415b80 10 client.20529 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:46.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.255+0000 7f371c415b80 1 -- 172.21.15.70:0/2010125373 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x561d05f639b0 con 0x561d05f4d210 2024-06-04T05:16:46.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.255+0000 7f36eaffd700 1 -- 172.21.15.70:0/2010125373 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f36fc0ad7c0 con 0x561d05f4d210 2024-06-04T05:16:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.255+0000 7f36eaffd700 1 -- 172.21.15.70:0/2010125373 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f3700013450 con 0x561d05f4d210 2024-06-04T05:16:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.255+0000 7f371c415b80 10 client.20529 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.255+0000 7f371c415b80 10 client.20529 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:16:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.255+0000 7f371c415b80 20 client.20529 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.255+0000 7f371c415b80 1 --2- 172.21.15.70:0/2010125373 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x561d05f66090 0x561d05f68480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.255+0000 7f371c415b80 4 client.20529 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:16:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.255+0000 7f371c415b80 1 -- 172.21.15.70:0/2010125373 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d05f189e0 con 0x561d05f66090 2024-06-04T05:16:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.256+0000 7f370b7fe700 1 --2- 172.21.15.70:0/2010125373 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x561d05f66090 0x561d05f68480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.256+0000 7f36eaffd700 10 client.20529 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:16:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.256+0000 7f370b7fe700 1 --2- 172.21.15.70:0/2010125373 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x561d05f66090 0x561d05f68480 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f5b465efd653158b server_cookie=f58dabbc79d8e800 in_seq=0 out_seq=0 2024-06-04T05:16:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.257+0000 7f36eaffd700 1 -- 172.21.15.70:0/2010125373 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561d05f189e0 con 0x561d05f66090 2024-06-04T05:16:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.257+0000 7f36eaffd700 10 client.20529 handle_command_reply: tid=0 2024-06-04T05:16:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.257+0000 7f371c415b80 1 client.20529 shutdown 2024-06-04T05:16:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.257+0000 7f371c415b80 1 -- 172.21.15.70:0/2010125373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 msgr2=0x561d05f557d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.257+0000 7f371c415b80 1 --2- 172.21.15.70:0/2010125373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05f557d0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f3700015700 tx=0x7f3700004860 comp rx=0 tx=0).stop 2024-06-04T05:16:46.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.257+0000 7f371c415b80 1 -- 172.21.15.70:0/2010125373 shutdown_connections 2024-06-04T05:16:46.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.257+0000 7f371c415b80 1 --2- 172.21.15.70:0/2010125373 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x561d05f66090 0x561d05f68480 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.257+0000 7f371c415b80 1 --2- 172.21.15.70:0/2010125373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f50f60 0x561d05f55d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.257+0000 7f371c415b80 1 --2- 172.21.15.70:0/2010125373 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f4c860 0x561d05f584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.257+0000 7f371c415b80 1 --2- 172.21.15.70:0/2010125373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05f557d0 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.257+0000 7f371c415b80 1 -- 172.21.15.70:0/2010125373 >> 172.21.15.70:0/2010125373 conn(0x561d05e7a080 msgr2=0x561d05f46510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:46.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.257+0000 7f36eaffd700 0 client.20529 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:16:46.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.257+0000 7f371c415b80 1 -- 172.21.15.70:0/2010125373 shutdown_connections 2024-06-04T05:16:46.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.257+0000 7f371c415b80 1 -- 172.21.15.70:0/2010125373 wait complete. 2024-06-04T05:16:46.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.257+0000 7f371c415b80 20 client.20529 trim_cache size 0 max 16384 2024-06-04T05:16:46.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.272+0000 7f371c415b80 1 Processor -- start 2024-06-04T05:16:46.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.272+0000 7f371c415b80 1 -- start start 2024-06-04T05:16:46.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f371c415b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f4c860 0x561d05f621c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f371c415b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05f54950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f371c415b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f50f60 0x561d05f57710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f371c415b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f370c1adeb0 con 0x561d05f4d210 2024-06-04T05:16:46.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f371c415b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f370c1af9f0 con 0x561d05f4c860 2024-06-04T05:16:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f371c415b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f370c1ae580 con 0x561d05f50f60 2024-06-04T05:16:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f370affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05f54950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f370affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05f54950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47110/0 (socket says 172.21.15.70:47110) 2024-06-04T05:16:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f370affd700 1 -- 172.21.15.70:0/1303234182 learned_addr learned my addr 172.21.15.70:0/1303234182 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f370bfff700 1 --2- 172.21.15.70:0/1303234182 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f50f60 0x561d05f57710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f370b7fe700 1 --2- 172.21.15.70:0/1303234182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f4c860 0x561d05f621c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f370affd700 1 -- 172.21.15.70:0/1303234182 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f50f60 msgr2=0x561d05f57710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f370affd700 1 --2- 172.21.15.70:0/1303234182 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f50f60 0x561d05f57710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f370affd700 1 -- 172.21.15.70:0/1303234182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f4c860 msgr2=0x561d05f621c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f370affd700 1 --2- 172.21.15.70:0/1303234182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f4c860 0x561d05f621c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f370affd700 1 -- 172.21.15.70:0/1303234182 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d05f53d40 con 0x561d05f4d210 2024-06-04T05:16:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f370b7fe700 1 --2- 172.21.15.70:0/1303234182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f4c860 0x561d05f621c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:16:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f370affd700 1 --2- 172.21.15.70:0/1303234182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05f54950 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f3700013a30 tx=0x7f3700006e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f23e852e0440296 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f36f17fa700 1 -- 172.21.15.70:0/1303234182 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3700025070 con 0x561d05f4d210 2024-06-04T05:16:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f36f17fa700 1 -- 172.21.15.70:0/1303234182 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3700013ab0 con 0x561d05f4d210 2024-06-04T05:16:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.273+0000 7f36f17fa700 1 -- 172.21.15.70:0/1303234182 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370001b040 con 0x561d05f4d210 2024-06-04T05:16:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.274+0000 7f371c415b80 1 -- 172.21.15.70:0/1303234182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 msgr2=0x561d05f54950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.274+0000 7f371c415b80 1 --2- 172.21.15.70:0/1303234182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05f54950 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f3700013a30 tx=0x7f3700006e10 comp rx=0 tx=0).stop 2024-06-04T05:16:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.274+0000 7f371c415b80 1 -- 172.21.15.70:0/1303234182 shutdown_connections 2024-06-04T05:16:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.274+0000 7f371c415b80 1 --2- 172.21.15.70:0/1303234182 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f50f60 0x561d05f57710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.274+0000 7f371c415b80 1 --2- 172.21.15.70:0/1303234182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f4c860 0x561d05f621c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.274+0000 7f371c415b80 1 --2- 172.21.15.70:0/1303234182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05f54950 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.274+0000 7f371c415b80 1 -- 172.21.15.70:0/1303234182 >> 172.21.15.70:0/1303234182 conn(0x561d05e7a080 msgr2=0x561d05f57000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.274+0000 7f371c415b80 1 -- 172.21.15.70:0/1303234182 shutdown_connections 2024-06-04T05:16:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.274+0000 7f371c415b80 1 -- 172.21.15.70:0/1303234182 wait complete. 2024-06-04T05:16:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.274+0000 7f371c415b80 1 -- start start 2024-06-04T05:16:46.279 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:16:46.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.274+0000 7f371c415b80 1 Processor -- start 2024-06-04T05:16:46.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f371c415b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f4c860 0x561d05fc7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f371c415b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05fc84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f371c415b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f50f60 0x561d05f5b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f371c415b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f370c1af320 con 0x561d05f4d210 2024-06-04T05:16:46.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f371c415b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f370c1aec50 con 0x561d05f4c860 2024-06-04T05:16:46.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f371c415b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f370c1abe70 con 0x561d05f50f60 2024-06-04T05:16:46.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f371c415b80 10 client.0 _ll_register_callbacks cb 0x561d05e36920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:46.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f370affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05fc84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f370affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05fc84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47112/0 (socket says 172.21.15.70:47112) 2024-06-04T05:16:46.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f370affd700 1 -- 172.21.15.70:0/3546807068 learned_addr learned my addr 172.21.15.70:0/3546807068 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:46.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f370b7fe700 1 --2- 172.21.15.70:0/3546807068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f4c860 0x561d05fc7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f370bfff700 1 --2- 172.21.15.70:0/3546807068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f50f60 0x561d05f5b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f370affd700 1 -- 172.21.15.70:0/3546807068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f50f60 msgr2=0x561d05f5b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:46.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f370affd700 1 --2- 172.21.15.70:0/3546807068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f50f60 0x561d05f5b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f370affd700 1 -- 172.21.15.70:0/3546807068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f4c860 msgr2=0x561d05fc7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:46.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f370affd700 1 --2- 172.21.15.70:0/3546807068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f4c860 0x561d05fc7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f370affd700 1 -- 172.21.15.70:0/3546807068 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d05f5c1e0 con 0x561d05f4d210 2024-06-04T05:16:46.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f370affd700 1 --2- 172.21.15.70:0/3546807068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05fc84a0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f3700006c70 tx=0x7f3700013450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2ab21f0d7d93930 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:46.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f36eaffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:46.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.275+0000 7f36eaffd700 1 -- 172.21.15.70:0/3546807068 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3700025040 con 0x561d05f4d210 2024-06-04T05:16:46.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.276+0000 7f36eaffd700 1 -- 172.21.15.70:0/3546807068 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f3700005a50 con 0x561d05f4d210 2024-06-04T05:16:46.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.276+0000 7f371c415b80 1 -- 172.21.15.70:0/3546807068 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3700013ab0 con 0x561d05f4d210 2024-06-04T05:16:46.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.276+0000 7f36eaffd700 1 -- 172.21.15.70:0/3546807068 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f3700005c50 con 0x561d05f4d210 2024-06-04T05:16:46.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.276+0000 7f371c415b80 10 client.20535 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:46.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.276+0000 7f371c415b80 1 -- 172.21.15.70:0/3546807068 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x561d05fd06d0 con 0x561d05f4d210 2024-06-04T05:16:46.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.276+0000 7f36eaffd700 1 -- 172.21.15.70:0/3546807068 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3700013ab0 con 0x561d05f4d210 2024-06-04T05:16:46.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.276+0000 7f36eaffd700 1 -- 172.21.15.70:0/3546807068 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f3700006160 con 0x561d05f4d210 2024-06-04T05:16:46.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.276+0000 7f371c415b80 10 client.20535 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:46.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.276+0000 7f371c415b80 10 client.20535 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:16:46.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.276+0000 7f371c415b80 20 client.20535 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:46.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.276+0000 7f371c415b80 1 --2- 172.21.15.70:0/3546807068 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x561d05fd0c90 0x561d05fd30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:46.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.276+0000 7f371c415b80 4 client.20535 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:16:46.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.276+0000 7f371c415b80 1 -- 172.21.15.70:0/3546807068 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561d05eadbe0 con 0x561d05fd0c90 2024-06-04T05:16:46.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.276+0000 7f370b7fe700 1 --2- 172.21.15.70:0/3546807068 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x561d05fd0c90 0x561d05fd30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:46.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.277+0000 7f370b7fe700 1 --2- 172.21.15.70:0/3546807068 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x561d05fd0c90 0x561d05fd30b0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f4fb277c55d1ecea server_cookie=55b58c642d536111 in_seq=0 out_seq=0 2024-06-04T05:16:46.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.277+0000 7f36eaffd700 10 client.20535 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:16:46.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.277+0000 7f36eaffd700 1 -- 172.21.15.70:0/3546807068 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561d05eadbe0 con 0x561d05fd0c90 2024-06-04T05:16:46.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.277+0000 7f36eaffd700 10 client.20535 handle_command_reply: tid=0 2024-06-04T05:16:46.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.277+0000 7f371c415b80 1 client.20535 shutdown 2024-06-04T05:16:46.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.277+0000 7f371c415b80 1 -- 172.21.15.70:0/3546807068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 msgr2=0x561d05fc84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:46.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.277+0000 7f371c415b80 1 --2- 172.21.15.70:0/3546807068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05fc84a0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f3700006c70 tx=0x7f3700013450 comp rx=0 tx=0).stop 2024-06-04T05:16:46.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.277+0000 7f371c415b80 1 -- 172.21.15.70:0/3546807068 shutdown_connections 2024-06-04T05:16:46.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.277+0000 7f371c415b80 1 --2- 172.21.15.70:0/3546807068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561d05f50f60 0x561d05f5b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.277+0000 7f371c415b80 1 --2- 172.21.15.70:0/3546807068 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x561d05fd0c90 0x561d05fd30b0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.277+0000 7f371c415b80 1 --2- 172.21.15.70:0/3546807068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d05f4c860 0x561d05fc7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.277+0000 7f371c415b80 1 --2- 172.21.15.70:0/3546807068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d05f4d210 0x561d05fc84a0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.277+0000 7f371c415b80 1 -- 172.21.15.70:0/3546807068 >> 172.21.15.70:0/3546807068 conn(0x561d05e7a080 msgr2=0x561d05f57890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:46.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.277+0000 7f36eaffd700 0 client.20535 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:16:46.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.277+0000 7f371c415b80 1 -- 172.21.15.70:0/3546807068 shutdown_connections 2024-06-04T05:16:46.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.277+0000 7f371c415b80 1 -- 172.21.15.70:0/3546807068 wait complete. 2024-06-04T05:16:46.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.277+0000 7f371c415b80 20 client.20535 trim_cache size 0 max 16384 2024-06-04T05:16:46.287 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:16:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.278+0000 7f3711b80700 1 -- 172.21.15.70:0/2483652703 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f36f40b29f0 msgr2=0x7f36f40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.278+0000 7f3711b80700 1 --2- 172.21.15.70:0/2483652703 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f36f40b29f0 0x7f36f40b4eb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f370c104a70 tx=0x7f3700009190 comp rx=0 tx=0).stop 2024-06-04T05:16:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.278+0000 7f3711b80700 1 -- 172.21.15.70:0/2483652703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f370c101e40 msgr2=0x7f370c19f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.278+0000 7f3711b80700 1 --2- 172.21.15.70:0/2483652703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f370c101e40 0x7f370c19f7d0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f36fc00ddc0 tx=0x7f36fc00a530 comp rx=0 tx=0).stop 2024-06-04T05:16:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.278+0000 7f3711b80700 1 -- 172.21.15.70:0/2483652703 shutdown_connections 2024-06-04T05:16:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.278+0000 7f3711b80700 1 --2- 172.21.15.70:0/2483652703 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f36f40b29f0 0x7f36f40b4eb0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.278+0000 7f3711b80700 1 --2- 172.21.15.70:0/2483652703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f370c106f20 0x7f370c199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.278+0000 7f3711b80700 1 --2- 172.21.15.70:0/2483652703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f370c101e40 0x7f370c19f7d0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.278+0000 7f3711b80700 1 --2- 172.21.15.70:0/2483652703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f370c1027f0 0x7f370c19fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.278+0000 7f3711b80700 1 -- 172.21.15.70:0/2483652703 >> 172.21.15.70:0/2483652703 conn(0x7f370c078a40 msgr2=0x7f370c100400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.278+0000 7f3711b80700 1 -- 172.21.15.70:0/2483652703 shutdown_connections 2024-06-04T05:16:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:46.278+0000 7f3711b80700 1 -- 172.21.15.70:0/2483652703 wait complete. 2024-06-04T05:16:46.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.66 MB/sec execute 218 sec latency 14247.095 ms 2024-06-04T05:16:47.199 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:16:47.237 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:16:47.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:47 smithi070 ceph-mon[32631]: pgmap v615: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-06-04T05:16:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:47 smithi112 ceph-mon[36453]: pgmap v615: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-06-04T05:16:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:47 smithi098 ceph-mon[35569]: pgmap v615: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-06-04T05:16:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.771+0000 7f93195c1640 1 -- 172.21.15.70:0/3456327443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9314108a70 msgr2=0x7f9314108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.771+0000 7f93195c1640 1 --2- 172.21.15.70:0/3456327443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9314108a70 0x7f9314108e50 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f9304009930 tx=0x7f930402a420 comp rx=0 tx=0).stop 2024-06-04T05:16:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.771+0000 7f93195c1640 1 -- 172.21.15.70:0/3456327443 shutdown_connections 2024-06-04T05:16:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.771+0000 7f93195c1640 1 --2- 172.21.15.70:0/3456327443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9314103420 0x7f931410f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.771+0000 7f93195c1640 1 --2- 172.21.15.70:0/3456327443 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9314102a60 0x7f9314102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.771+0000 7f93195c1640 1 --2- 172.21.15.70:0/3456327443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9314108a70 0x7f9314108e50 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.771+0000 7f93195c1640 1 -- 172.21.15.70:0/3456327443 >> 172.21.15.70:0/3456327443 conn(0x7f93140fe550 msgr2=0x7f9314100970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.771+0000 7f93195c1640 1 -- 172.21.15.70:0/3456327443 shutdown_connections 2024-06-04T05:16:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.771+0000 7f93195c1640 1 -- 172.21.15.70:0/3456327443 wait complete. 2024-06-04T05:16:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.771+0000 7f93195c1640 1 Processor -- start 2024-06-04T05:16:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f93195c1640 1 -- start start 2024-06-04T05:16:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f93195c1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9314102a60 0x7f9314199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f93195c1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9314103420 0x7f93141996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f93195c1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9314108a70 0x7f931419da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f93195c1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9314112be0 con 0x7f9314102a60 2024-06-04T05:16:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f93195c1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9314112a60 con 0x7f9314103420 2024-06-04T05:16:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f93195c1640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9314112d60 con 0x7f9314108a70 2024-06-04T05:16:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f9312ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9314102a60 0x7f9314199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f9312ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9314102a60 0x7f9314199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47130/0 (socket says 172.21.15.70:47130) 2024-06-04T05:16:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f9312ffd640 1 -- 172.21.15.70:0/1911932802 learned_addr learned my addr 172.21.15.70:0/1911932802 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f93127fc640 1 --2- 172.21.15.70:0/1911932802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9314103420 0x7f93141996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f93137fe640 1 --2- 172.21.15.70:0/1911932802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9314108a70 0x7f931419da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f9312ffd640 1 -- 172.21.15.70:0/1911932802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9314108a70 msgr2=0x7f931419da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f9312ffd640 1 --2- 172.21.15.70:0/1911932802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9314108a70 0x7f931419da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f9312ffd640 1 -- 172.21.15.70:0/1911932802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9314103420 msgr2=0x7f93141996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f9312ffd640 1 --2- 172.21.15.70:0/1911932802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9314103420 0x7f93141996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f9312ffd640 1 -- 172.21.15.70:0/1911932802 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f931419e170 con 0x7f9314102a60 2024-06-04T05:16:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f93127fc640 1 --2- 172.21.15.70:0/1911932802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9314103420 0x7f93141996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:16:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.772+0000 7f9312ffd640 1 --2- 172.21.15.70:0/1911932802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9314102a60 0x7f9314199180 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f9304005940 tx=0x7f93040042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.773+0000 7f92fbfff640 1 -- 172.21.15.70:0/1911932802 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930403f030 con 0x7f9314102a60 2024-06-04T05:16:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.773+0000 7f92fbfff640 1 -- 172.21.15.70:0/1911932802 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9304003c90 con 0x7f9314102a60 2024-06-04T05:16:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.773+0000 7f93195c1640 1 -- 172.21.15.70:0/1911932802 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f931419e400 con 0x7f9314102a60 2024-06-04T05:16:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.773+0000 7f92fbfff640 1 -- 172.21.15.70:0/1911932802 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9304035070 con 0x7f9314102a60 2024-06-04T05:16:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.773+0000 7f93195c1640 1 -- 172.21.15.70:0/1911932802 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93141a5cf0 con 0x7f9314102a60 2024-06-04T05:16:47.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.775+0000 7f92fbfff640 1 -- 172.21.15.70:0/1911932802 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f9304042030 con 0x7f9314102a60 2024-06-04T05:16:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.775+0000 7f931c3b4740 1 Processor -- start 2024-06-04T05:16:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.775+0000 7f931c3b4740 1 -- start start 2024-06-04T05:16:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.775+0000 7f931c3b4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a14820 0x5565f6a14c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.775+0000 7f931c3b4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a1f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.775+0000 7f931c3b4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a1fa00 0x5565f6a21df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.775+0000 7f931c3b4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5565f6986c30 con 0x5565f6a15140 2024-06-04T05:16:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.775+0000 7f931c3b4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5565f69767d0 con 0x5565f6a14820 2024-06-04T05:16:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.775+0000 7f931c3b4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5565f697a810 con 0x5565f6a1fa00 2024-06-04T05:16:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.775+0000 7f9312ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a1f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.775+0000 7f9312ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a1f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47132/0 (socket says 172.21.15.70:47132) 2024-06-04T05:16:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.775+0000 7f9312ffd640 1 -- 172.21.15.70:0/1180272754 learned_addr learned my addr 172.21.15.70:0/1180272754 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.776+0000 7f93127fc640 1 --2- 172.21.15.70:0/1180272754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a1fa00 0x5565f6a21df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.776+0000 7f93137fe640 1 --2- 172.21.15.70:0/1180272754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a14820 0x5565f6a14c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.776+0000 7f9312ffd640 1 -- 172.21.15.70:0/1180272754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a1fa00 msgr2=0x5565f6a21df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.776+0000 7f9312ffd640 1 --2- 172.21.15.70:0/1180272754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a1fa00 0x5565f6a21df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.776+0000 7f9312ffd640 1 -- 172.21.15.70:0/1180272754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a14820 msgr2=0x5565f6a14c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.776+0000 7f9312ffd640 1 --2- 172.21.15.70:0/1180272754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a14820 0x5565f6a14c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.776+0000 7f9312ffd640 1 -- 172.21.15.70:0/1180272754 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565f668b210 con 0x5565f6a15140 2024-06-04T05:16:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.776+0000 7f92fbfff640 1 --2- 172.21.15.70:0/1911932802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f92fc0b2a10 0x7f92fc0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.776+0000 7f9312ffd640 1 --2- 172.21.15.70:0/1180272754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a1f4c0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f9304100290 tx=0x7f9304106ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e626d42365caf954 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.776+0000 7f93137fe640 1 --2- 172.21.15.70:0/1911932802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f92fc0b2a10 0x7f92fc0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.776+0000 7f92fbfff640 1 -- 172.21.15.70:0/1911932802 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f9304102c00 con 0x7f9314102a60 2024-06-04T05:16:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.776+0000 7f92f97fa640 1 -- 172.21.15.70:0/1180272754 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93040c5d70 con 0x5565f6a15140 2024-06-04T05:16:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.776+0000 7f92f97fa640 1 -- 172.21.15.70:0/1180272754 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9304003e40 con 0x5565f6a15140 2024-06-04T05:16:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.776+0000 7f92f97fa640 1 -- 172.21.15.70:0/1180272754 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93040c9620 con 0x5565f6a15140 2024-06-04T05:16:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.777+0000 7f931c3b4740 1 -- 172.21.15.70:0/1180272754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 msgr2=0x5565f6a1f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.777+0000 7f93137fe640 1 --2- 172.21.15.70:0/1911932802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f92fc0b2a10 0x7f92fc0b4ed0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f930c00cd40 tx=0x7f930c00c330 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.777+0000 7f931c3b4740 1 --2- 172.21.15.70:0/1180272754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a1f4c0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f9304100290 tx=0x7f9304106ed0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.777+0000 7f931c3b4740 1 -- 172.21.15.70:0/1180272754 shutdown_connections 2024-06-04T05:16:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.777+0000 7f931c3b4740 1 --2- 172.21.15.70:0/1180272754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a1fa00 0x5565f6a21df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.777+0000 7f931c3b4740 1 --2- 172.21.15.70:0/1180272754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a1f4c0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.777+0000 7f931c3b4740 1 --2- 172.21.15.70:0/1180272754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a14820 0x5565f6a14c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.777+0000 7f931c3b4740 1 -- 172.21.15.70:0/1180272754 >> 172.21.15.70:0/1180272754 conn(0x5565f6a10d30 msgr2=0x5565f6a11110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.777+0000 7f931c3b4740 1 -- 172.21.15.70:0/1180272754 shutdown_connections 2024-06-04T05:16:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.777+0000 7f931c3b4740 1 -- 172.21.15.70:0/1180272754 wait complete. 2024-06-04T05:16:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.777+0000 7f931c3b4740 1 -- start start 2024-06-04T05:16:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.777+0000 7f931c3b4740 1 Processor -- start 2024-06-04T05:16:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.777+0000 7f931c3b4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a14820 0x5565f6a1c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.777+0000 7f931c3b4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a1cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.777+0000 7f931c3b4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a1fa00 0x5565f6a1d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.777+0000 7f931c3b4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5565f665c790 con 0x5565f6a15140 2024-06-04T05:16:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f931c3b4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5565f6932890 con 0x5565f6a14820 2024-06-04T05:16:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f931c3b4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5565f6a223f0 con 0x5565f6a1fa00 2024-06-04T05:16:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f931c3b4740 10 client.0 _ll_register_callbacks cb 0x5565f668aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f93127fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a1cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f93127fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a1cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47138/0 (socket says 172.21.15.70:47138) 2024-06-04T05:16:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f93127fc640 1 -- 172.21.15.70:0/3883904289 learned_addr learned my addr 172.21.15.70:0/3883904289 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f9312ffd640 1 --2- 172.21.15.70:0/3883904289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a14820 0x5565f6a1c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f93137fe640 1 --2- 172.21.15.70:0/3883904289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a1fa00 0x5565f6a1d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f93127fc640 1 -- 172.21.15.70:0/3883904289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a1fa00 msgr2=0x5565f6a1d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f93127fc640 1 --2- 172.21.15.70:0/3883904289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a1fa00 0x5565f6a1d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f93127fc640 1 -- 172.21.15.70:0/3883904289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a14820 msgr2=0x5565f6a1c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f93127fc640 1 --2- 172.21.15.70:0/3883904289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a14820 0x5565f6a1c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f93127fc640 1 -- 172.21.15.70:0/3883904289 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565f6757620 con 0x5565f6a15140 2024-06-04T05:16:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f93127fc640 1 --2- 172.21.15.70:0/3883904289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a1cf10 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f930800be90 tx=0x7f93080117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856ee1e44fa74c51 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f92f6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f92f6ff5640 1 -- 172.21.15.70:0/3883904289 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9308005220 con 0x5565f6a15140 2024-06-04T05:16:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f92f6ff5640 1 -- 172.21.15.70:0/3883904289 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f93080053c0 con 0x5565f6a15140 2024-06-04T05:16:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f931c3b4740 1 -- 172.21.15.70:0/3883904289 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5565f6988b80 con 0x5565f6a15140 2024-06-04T05:16:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f92f6ff5640 1 -- 172.21.15.70:0/3883904289 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f93080055c0 con 0x5565f6a15140 2024-06-04T05:16:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f931c3b4740 10 client.20547 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.778+0000 7f931c3b4740 1 -- 172.21.15.70:0/3883904289 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x5565f6a2e1a0 con 0x5565f6a15140 2024-06-04T05:16:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.779+0000 7f92f6ff5640 1 -- 172.21.15.70:0/3883904289 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x5565f6988b80 con 0x5565f6a15140 2024-06-04T05:16:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.779+0000 7f92f6ff5640 1 -- 172.21.15.70:0/3883904289 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f930801b820 con 0x5565f6a15140 2024-06-04T05:16:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.779+0000 7f931c3b4740 10 client.20547 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.779+0000 7f931c3b4740 10 client.20547 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:16:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.779+0000 7f931c3b4740 20 client.20547 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.779+0000 7f931c3b4740 1 --2- 172.21.15.70:0/3883904289 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5565f6a2f660 0x5565f6a31a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.779+0000 7f931c3b4740 4 client.20547 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:16:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.779+0000 7f931c3b4740 1 -- 172.21.15.70:0/3883904289 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565f6982360 con 0x5565f6a2f660 2024-06-04T05:16:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.779+0000 7f9312ffd640 1 --2- 172.21.15.70:0/3883904289 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5565f6a2f660 0x5565f6a31a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.779+0000 7f92f6ff5640 10 client.20547 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.779+0000 7f9312ffd640 1 --2- 172.21.15.70:0/3883904289 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5565f6a2f660 0x5565f6a31a70 crc :-1 s=READY pgs=419 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b77456a50ffa4410 server_cookie=696abd6597c76437 in_seq=0 out_seq=0 2024-06-04T05:16:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.780+0000 7f92f6ff5640 1 -- 172.21.15.70:0/3883904289 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5565f6982360 con 0x5565f6a2f660 2024-06-04T05:16:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.780+0000 7f92f6ff5640 10 client.20547 handle_command_reply: tid=0 2024-06-04T05:16:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.780+0000 7f931c3b4740 1 client.20547 shutdown 2024-06-04T05:16:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.780+0000 7f931c3b4740 1 -- 172.21.15.70:0/3883904289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 msgr2=0x5565f6a1cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.780+0000 7f931c3b4740 1 --2- 172.21.15.70:0/3883904289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a1cf10 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f930800be90 tx=0x7f93080117b0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.780+0000 7f931c3b4740 1 -- 172.21.15.70:0/3883904289 shutdown_connections 2024-06-04T05:16:47.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.780+0000 7f931c3b4740 1 --2- 172.21.15.70:0/3883904289 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5565f6a2f660 0x5565f6a31a70 crc :-1 s=READY pgs=419 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.780+0000 7f931c3b4740 1 --2- 172.21.15.70:0/3883904289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a1fa00 0x5565f6a1d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.780+0000 7f931c3b4740 1 --2- 172.21.15.70:0/3883904289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a1cf10 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.780+0000 7f92f6ff5640 0 client.20547 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:47.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.780+0000 7f931c3b4740 1 --2- 172.21.15.70:0/3883904289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a14820 0x5565f6a1c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.780+0000 7f931c3b4740 1 -- 172.21.15.70:0/3883904289 >> 172.21.15.70:0/3883904289 conn(0x5565f6a10d30 msgr2=0x5565f6a0ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:47.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.780+0000 7f931c3b4740 1 -- 172.21.15.70:0/3883904289 shutdown_connections 2024-06-04T05:16:47.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.780+0000 7f931c3b4740 1 -- 172.21.15.70:0/3883904289 wait complete. 2024-06-04T05:16:47.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.781+0000 7f931c3b4740 20 client.20547 trim_cache size 0 max 16384 2024-06-04T05:16:47.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.795+0000 7f931c3b4740 1 Processor -- start 2024-06-04T05:16:47.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.795+0000 7f931c3b4740 1 -- start start 2024-06-04T05:16:47.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.795+0000 7f931c3b4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a14820 0x5565f6a1d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.795+0000 7f931c3b4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a15140 0x5565f6a37380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f931c3b4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a1fa00 0x5565f6a378c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f931c3b4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f93141ae680 con 0x5565f6a14820 2024-06-04T05:16:47.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f931c3b4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f93141b01c0 con 0x5565f6a1fa00 2024-06-04T05:16:47.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f931c3b4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f93141aed50 con 0x5565f6a15140 2024-06-04T05:16:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f9312ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a14820 0x5565f6a1d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f9312ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a14820 0x5565f6a1d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47150/0 (socket says 172.21.15.70:47150) 2024-06-04T05:16:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f9312ffd640 1 -- 172.21.15.70:0/1378888855 learned_addr learned my addr 172.21.15.70:0/1378888855 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f93127fc640 1 --2- 172.21.15.70:0/1378888855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a15140 0x5565f6a37380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f93137fe640 1 --2- 172.21.15.70:0/1378888855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a1fa00 0x5565f6a378c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f9312ffd640 1 -- 172.21.15.70:0/1378888855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a15140 msgr2=0x5565f6a37380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f9312ffd640 1 --2- 172.21.15.70:0/1378888855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a15140 0x5565f6a37380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f9312ffd640 1 -- 172.21.15.70:0/1378888855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a1fa00 msgr2=0x5565f6a378c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f9312ffd640 1 --2- 172.21.15.70:0/1378888855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a1fa00 0x5565f6a378c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f9312ffd640 1 -- 172.21.15.70:0/1378888855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565f6a170d0 con 0x5565f6a14820 2024-06-04T05:16:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f93127fc640 1 --2- 172.21.15.70:0/1378888855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a15140 0x5565f6a37380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:16:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f9312ffd640 1 --2- 172.21.15.70:0/1378888855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a14820 0x5565f6a1d0a0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f9304100920 tx=0x7f9304009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0d0e3fb410effe2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f92f8ff9640 1 -- 172.21.15.70:0/1378888855 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9304049040 con 0x5565f6a14820 2024-06-04T05:16:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f92f8ff9640 1 -- 172.21.15.70:0/1378888855 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f93040c99c0 con 0x5565f6a14820 2024-06-04T05:16:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.796+0000 7f92f8ff9640 1 -- 172.21.15.70:0/1378888855 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93040c9800 con 0x5565f6a14820 2024-06-04T05:16:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 1 -- 172.21.15.70:0/1378888855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a14820 msgr2=0x5565f6a1d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 1 --2- 172.21.15.70:0/1378888855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a14820 0x5565f6a1d0a0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f9304100920 tx=0x7f9304009260 comp rx=0 tx=0).stop 2024-06-04T05:16:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 1 -- 172.21.15.70:0/1378888855 shutdown_connections 2024-06-04T05:16:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 1 --2- 172.21.15.70:0/1378888855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a1fa00 0x5565f6a378c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 1 --2- 172.21.15.70:0/1378888855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a15140 0x5565f6a37380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 1 --2- 172.21.15.70:0/1378888855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a14820 0x5565f6a1d0a0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 1 -- 172.21.15.70:0/1378888855 >> 172.21.15.70:0/1378888855 conn(0x5565f6a10d30 msgr2=0x5565f6a1c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 1 -- 172.21.15.70:0/1378888855 shutdown_connections 2024-06-04T05:16:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 1 -- 172.21.15.70:0/1378888855 wait complete. 2024-06-04T05:16:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 1 -- start start 2024-06-04T05:16:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 1 Processor -- start 2024-06-04T05:16:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a14820 0x5565f6a236e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a23c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a1fa00 0x5565f6a99250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f93141afaf0 con 0x5565f6a15140 2024-06-04T05:16:47.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f93141af420 con 0x5565f6a1fa00 2024-06-04T05:16:47.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f93141ac680 con 0x5565f6a14820 2024-06-04T05:16:47.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.797+0000 7f931c3b4740 10 client.0 _ll_register_callbacks cb 0x7f930800ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:47.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f93127fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a23c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f93127fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a23c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47156/0 (socket says 172.21.15.70:47156) 2024-06-04T05:16:47.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f93127fc640 1 -- 172.21.15.70:0/42484795 learned_addr learned my addr 172.21.15.70:0/42484795 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:47.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f93137fe640 1 --2- 172.21.15.70:0/42484795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a1fa00 0x5565f6a99250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f93127fc640 1 -- 172.21.15.70:0/42484795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a14820 msgr2=0x5565f6a236e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:47.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f9312ffd640 1 --2- 172.21.15.70:0/42484795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a14820 0x5565f6a236e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f93127fc640 1 --2- 172.21.15.70:0/42484795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a14820 0x5565f6a236e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f93127fc640 1 -- 172.21.15.70:0/42484795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a1fa00 msgr2=0x5565f6a99250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f93127fc640 1 --2- 172.21.15.70:0/42484795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a1fa00 0x5565f6a99250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f93127fc640 1 -- 172.21.15.70:0/42484795 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565f6a244f0 con 0x5565f6a15140 2024-06-04T05:16:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f93137fe640 1 --2- 172.21.15.70:0/42484795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a1fa00 0x5565f6a99250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:16:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f93127fc640 1 --2- 172.21.15.70:0/42484795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a23c20 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x5565f6a1dfa0 tx=0x7f93080124d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a40689d0c2ea27 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f92f6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f92f6ff5640 1 -- 172.21.15.70:0/42484795 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9308027040 con 0x5565f6a15140 2024-06-04T05:16:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f92f6ff5640 1 -- 172.21.15.70:0/42484795 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f930801b7c0 con 0x5565f6a15140 2024-06-04T05:16:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f931c3b4740 1 -- 172.21.15.70:0/42484795 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93040c99c0 con 0x5565f6a15140 2024-06-04T05:16:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f92f6ff5640 1 -- 172.21.15.70:0/42484795 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f930801b9e0 con 0x5565f6a15140 2024-06-04T05:16:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f931c3b4740 10 client.20553 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f931c3b4740 1 -- 172.21.15.70:0/42484795 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x5565f6a9a8a0 con 0x5565f6a15140 2024-06-04T05:16:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.798+0000 7f92f6ff5640 1 -- 172.21.15.70:0/42484795 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f93040c99c0 con 0x5565f6a15140 2024-06-04T05:16:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.799+0000 7f92f6ff5640 1 -- 172.21.15.70:0/42484795 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f9308023410 con 0x5565f6a15140 2024-06-04T05:16:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.799+0000 7f931c3b4740 10 client.20553 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.799+0000 7f931c3b4740 10 client.20553 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:16:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.799+0000 7f931c3b4740 20 client.20553 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.799+0000 7f931c3b4740 1 --2- 172.21.15.70:0/42484795 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5565f6a9b0a0 0x5565f6a9d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.799+0000 7f931c3b4740 4 client.20553 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:16:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.799+0000 7f931c3b4740 1 -- 172.21.15.70:0/42484795 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5565f68fc5f0 con 0x5565f6a9b0a0 2024-06-04T05:16:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.799+0000 7f9312ffd640 1 --2- 172.21.15.70:0/42484795 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5565f6a9b0a0 0x5565f6a9d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.799+0000 7f92f6ff5640 10 client.20553 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.799+0000 7f9312ffd640 1 --2- 172.21.15.70:0/42484795 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5565f6a9b0a0 0x5565f6a9d4c0 crc :-1 s=READY pgs=420 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4656a09ea963910 server_cookie=47bbc0294f9bd707 in_seq=0 out_seq=0 2024-06-04T05:16:47.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.63 MB/sec execute 219 sec latency 15247.142 ms 2024-06-04T05:16:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.836+0000 7f92f6ff5640 1 -- 172.21.15.70:0/42484795 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5565f68fc5f0 con 0x5565f6a9b0a0 2024-06-04T05:16:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.836+0000 7f92f6ff5640 10 client.20553 handle_command_reply: tid=0 2024-06-04T05:16:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.836+0000 7f931c3b4740 1 client.20553 shutdown 2024-06-04T05:16:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.836+0000 7f931c3b4740 1 -- 172.21.15.70:0/42484795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 msgr2=0x5565f6a23c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.836+0000 7f931c3b4740 1 --2- 172.21.15.70:0/42484795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a23c20 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x5565f6a1dfa0 tx=0x7f93080124d0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.836+0000 7f931c3b4740 1 -- 172.21.15.70:0/42484795 shutdown_connections 2024-06-04T05:16:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.836+0000 7f931c3b4740 1 --2- 172.21.15.70:0/42484795 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5565f6a9b0a0 0x5565f6a9d4c0 crc :-1 s=READY pgs=420 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.836+0000 7f931c3b4740 1 --2- 172.21.15.70:0/42484795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5565f6a1fa00 0x5565f6a99250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.836+0000 7f92f6ff5640 0 client.20553 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.836+0000 7f931c3b4740 1 --2- 172.21.15.70:0/42484795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565f6a15140 0x5565f6a23c20 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.836+0000 7f931c3b4740 1 --2- 172.21.15.70:0/42484795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5565f6a14820 0x5565f6a236e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.836+0000 7f931c3b4740 1 -- 172.21.15.70:0/42484795 >> 172.21.15.70:0/42484795 conn(0x5565f6a10d30 msgr2=0x5565f6a1c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.836+0000 7f931c3b4740 1 -- 172.21.15.70:0/42484795 shutdown_connections 2024-06-04T05:16:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.836+0000 7f931c3b4740 1 -- 172.21.15.70:0/42484795 wait complete. 2024-06-04T05:16:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.836+0000 7f931c3b4740 20 client.20553 trim_cache size 0 max 16384 2024-06-04T05:16:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:16:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:16:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:16:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.837+0000 7f93195c1640 1 -- 172.21.15.70:0/1911932802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f92fc0b2a10 msgr2=0x7f92fc0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.837+0000 7f93195c1640 1 --2- 172.21.15.70:0/1911932802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f92fc0b2a10 0x7f92fc0b4ed0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f930c00cd40 tx=0x7f930c00c330 comp rx=0 tx=0).stop 2024-06-04T05:16:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.837+0000 7f93195c1640 1 -- 172.21.15.70:0/1911932802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9314102a60 msgr2=0x7f9314199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.837+0000 7f93195c1640 1 --2- 172.21.15.70:0/1911932802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9314102a60 0x7f9314199180 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f9304005940 tx=0x7f93040042f0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.837+0000 7f93195c1640 1 -- 172.21.15.70:0/1911932802 shutdown_connections 2024-06-04T05:16:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.837+0000 7f93195c1640 1 --2- 172.21.15.70:0/1911932802 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f92fc0b2a10 0x7f92fc0b4ed0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.837+0000 7f93195c1640 1 --2- 172.21.15.70:0/1911932802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9314103420 0x7f93141996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.837+0000 7f93195c1640 1 --2- 172.21.15.70:0/1911932802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9314108a70 0x7f931419da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.837+0000 7f93195c1640 1 --2- 172.21.15.70:0/1911932802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9314102a60 0x7f9314199180 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:47.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.837+0000 7f93195c1640 1 -- 172.21.15.70:0/1911932802 >> 172.21.15.70:0/1911932802 conn(0x7f93140fe550 msgr2=0x7f93140feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:47.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.837+0000 7f93195c1640 1 -- 172.21.15.70:0/1911932802 shutdown_connections 2024-06-04T05:16:47.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:16:47.837+0000 7f93195c1640 1 -- 172.21.15.70:0/1911932802 wait complete. 2024-06-04T05:16:48.542 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:16:48.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.60 MB/sec execute 220 sec latency 16247.169 ms 2024-06-04T05:16:49.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:49 smithi070 ceph-mon[32631]: pgmap v616: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 177 KiB/s wr, 13 op/s 2024-06-04T05:16:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:49 smithi112 ceph-mon[36453]: pgmap v616: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 177 KiB/s wr, 13 op/s 2024-06-04T05:16:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:49 smithi098 ceph-mon[35569]: pgmap v616: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 177 KiB/s wr, 13 op/s 2024-06-04T05:16:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.802+0000 7f8e192c8700 1 -- 172.21.15.70:0/2825293532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e14102e00 msgr2=0x7f8e1410f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.802+0000 7f8e192c8700 1 --2- 172.21.15.70:0/2825293532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e14102e00 0x7f8e1410f2f0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00b0a0 tx=0x7f8e0c00b3b0 comp rx=0 tx=0).stop 2024-06-04T05:16:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.802+0000 7f8e192c8700 1 -- 172.21.15.70:0/2825293532 shutdown_connections 2024-06-04T05:16:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.802+0000 7f8e192c8700 1 --2- 172.21.15.70:0/2825293532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e14102e00 0x7f8e1410f2f0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.802+0000 7f8e192c8700 1 --2- 172.21.15.70:0/2825293532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e14108520 0x7f8e14108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.802+0000 7f8e192c8700 1 --2- 172.21.15.70:0/2825293532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e14102440 0x7f8e141028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.802+0000 7f8e192c8700 1 -- 172.21.15.70:0/2825293532 >> 172.21.15.70:0/2825293532 conn(0x7f8e140793f0 msgr2=0x7f8e14079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.802+0000 7f8e192c8700 1 -- 172.21.15.70:0/2825293532 shutdown_connections 2024-06-04T05:16:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.802+0000 7f8e192c8700 1 -- 172.21.15.70:0/2825293532 wait complete. 2024-06-04T05:16:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.803+0000 7f8e192c8700 1 Processor -- start 2024-06-04T05:16:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.803+0000 7f8e192c8700 1 -- start start 2024-06-04T05:16:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e192c8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e14102440 0x7f8e1419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e192c8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e14102e00 0x7f8e1419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e192c8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e14108520 0x7f8e14198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e192c8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e14112500 con 0x7f8e14102440 2024-06-04T05:16:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e192c8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8e14112380 con 0x7f8e14108520 2024-06-04T05:16:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e192c8700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8e14112680 con 0x7f8e14102e00 2024-06-04T05:16:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e12ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e14102440 0x7f8e1419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e12ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e14102440 0x7f8e1419e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47178/0 (socket says 172.21.15.70:47178) 2024-06-04T05:16:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e12ffd700 1 -- 172.21.15.70:0/2279920064 learned_addr learned my addr 172.21.15.70:0/2279920064 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e137fe700 1 --2- 172.21.15.70:0/2279920064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e14108520 0x7f8e14198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e127fc700 1 --2- 172.21.15.70:0/2279920064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e14102e00 0x7f8e1419ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e12ffd700 1 -- 172.21.15.70:0/2279920064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e14102e00 msgr2=0x7f8e1419ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e12ffd700 1 --2- 172.21.15.70:0/2279920064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e14102e00 0x7f8e1419ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e12ffd700 1 -- 172.21.15.70:0/2279920064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e14108520 msgr2=0x7f8e14198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e12ffd700 1 --2- 172.21.15.70:0/2279920064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e14108520 0x7f8e14198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e12ffd700 1 -- 172.21.15.70:0/2279920064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e14199020 con 0x7f8e14102440 2024-06-04T05:16:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e12ffd700 1 --2- 172.21.15.70:0/2279920064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e14102440 0x7f8e1419e6c0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00d980 tx=0x7f8dfc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e0bfff700 1 -- 172.21.15.70:0/2279920064 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc016070 con 0x7f8e14102440 2024-06-04T05:16:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e192c8700 1 -- 172.21.15.70:0/2279920064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e141992b0 con 0x7f8e14102440 2024-06-04T05:16:49.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e0bfff700 1 -- 172.21.15.70:0/2279920064 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8dfc004830 con 0x7f8e14102440 2024-06-04T05:16:49.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.804+0000 7f8e0bfff700 1 -- 172.21.15.70:0/2279920064 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc0054c0 con 0x7f8e14102440 2024-06-04T05:16:49.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.805+0000 7f8e192c8700 1 -- 172.21.15.70:0/2279920064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e1406b220 con 0x7f8e14102440 2024-06-04T05:16:49.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.805+0000 7f8e192c8700 1 -- 172.21.15.70:0/2279920064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e14103b60 con 0x7f8e14102440 2024-06-04T05:16:49.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.807+0000 7f8e0bfff700 1 -- 172.21.15.70:0/2279920064 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f8dfc004220 con 0x7f8e14102440 2024-06-04T05:16:49.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.808+0000 7f8e0bfff700 1 --2- 172.21.15.70:0/2279920064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8e000b29f0 0x7f8e000b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:49.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.808+0000 7f8e127fc700 1 --2- 172.21.15.70:0/2279920064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8e000b29f0 0x7f8e000b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:49.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.808+0000 7f8e0bfff700 1 -- 172.21.15.70:0/2279920064 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f8dfc0e2910 con 0x7f8e14102440 2024-06-04T05:16:49.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.809+0000 7f8e127fc700 1 --2- 172.21.15.70:0/2279920064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8e000b29f0 0x7f8e000b4eb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8e14079b30 tx=0x7f8e0400f040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:49.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:49.812+0000 7f8e0bfff700 1 -- 172.21.15.70:0/2279920064 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8dfc01f020 con 0x7f8e14102440 2024-06-04T05:16:49.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.57 MB/sec execute 221 sec latency 17247.203 ms 2024-06-04T05:16:50.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:50.069+0000 7f8e192c8700 1 -- 172.21.15.70:0/2279920064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e1406a230 con 0x7f8e14102440 2024-06-04T05:16:50.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:50.069+0000 7f8e0bfff700 1 -- 172.21.15.70:0/2279920064 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+11929 (secure 0 0 0) 0x7f8dfc0aa260 con 0x7f8e14102440 2024-06-04T05:16:50.071 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:16:50.072 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":57,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:15:55.006036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T05:16:50.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:50.072+0000 7f8e192c8700 1 -- 172.21.15.70:0/2279920064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8e000b29f0 msgr2=0x7f8e000b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:50.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:50.072+0000 7f8e192c8700 1 --2- 172.21.15.70:0/2279920064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8e000b29f0 0x7f8e000b4eb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8e14079b30 tx=0x7f8e0400f040 comp rx=0 tx=0).stop 2024-06-04T05:16:50.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:50.072+0000 7f8e192c8700 1 -- 172.21.15.70:0/2279920064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e14102440 msgr2=0x7f8e1419e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:50.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:50.072+0000 7f8e192c8700 1 --2- 172.21.15.70:0/2279920064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e14102440 0x7f8e1419e6c0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00d980 tx=0x7f8dfc00dc90 comp rx=0 tx=0).stop 2024-06-04T05:16:50.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:50.072+0000 7f8e192c8700 1 -- 172.21.15.70:0/2279920064 shutdown_connections 2024-06-04T05:16:50.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:50.072+0000 7f8e192c8700 1 --2- 172.21.15.70:0/2279920064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8e000b29f0 0x7f8e000b4eb0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:50.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:50.072+0000 7f8e192c8700 1 --2- 172.21.15.70:0/2279920064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e14108520 0x7f8e14198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:50.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:50.072+0000 7f8e192c8700 1 --2- 172.21.15.70:0/2279920064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e14102440 0x7f8e1419e6c0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:50.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:50.072+0000 7f8e192c8700 1 --2- 172.21.15.70:0/2279920064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e14102e00 0x7f8e1419ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:50.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:50.072+0000 7f8e192c8700 1 -- 172.21.15.70:0/2279920064 >> 172.21.15.70:0/2279920064 conn(0x7f8e140793f0 msgr2=0x7f8e141005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:50.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:50.072+0000 7f8e192c8700 1 -- 172.21.15.70:0/2279920064 shutdown_connections 2024-06-04T05:16:50.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:50.072+0000 7f8e192c8700 1 -- 172.21.15.70:0/2279920064 wait complete. 2024-06-04T05:16:50.075 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 57 2024-06-04T05:16:50.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:50 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2279920064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:16:50.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:50 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:16:50.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:50 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2279920064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:16:50.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:50 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:16:50.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:50 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2279920064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:16:50.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:50 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:16:50.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.54 MB/sec execute 222 sec latency 18247.242 ms 2024-06-04T05:16:51.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.112+0000 7f556a001700 1 -- 172.21.15.70:0/1797236778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55641024f0 msgr2=0x7f55641103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:51.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.112+0000 7f556a001700 1 --2- 172.21.15.70:0/1797236778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55641024f0 0x7f55641103d0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f555c00b0d0 tx=0x7f555c00b3e0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.112+0000 7f556a001700 1 -- 172.21.15.70:0/1797236778 shutdown_connections 2024-06-04T05:16:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.112+0000 7f556a001700 1 --2- 172.21.15.70:0/1797236778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55641024f0 0x7f55641103d0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.112+0000 7f556a001700 1 --2- 172.21.15.70:0/1797236778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5564109520 0x7f5564109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.112+0000 7f556a001700 1 --2- 172.21.15.70:0/1797236778 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5564101b30 0x7f5564101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.112+0000 7f556a001700 1 -- 172.21.15.70:0/1797236778 >> 172.21.15.70:0/1797236778 conn(0x7f5564078ab0 msgr2=0x7f55640ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.112+0000 7f556a001700 1 -- 172.21.15.70:0/1797236778 shutdown_connections 2024-06-04T05:16:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.112+0000 7f556a001700 1 -- 172.21.15.70:0/1797236778 wait complete. 2024-06-04T05:16:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.113+0000 7f556a001700 1 Processor -- start 2024-06-04T05:16:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.113+0000 7f556a001700 1 -- start start 2024-06-04T05:16:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.113+0000 7f556a001700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5564101b30 0x7f556419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.113+0000 7f556a001700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55641024f0 0x7f556419fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.113+0000 7f556a001700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5564109520 0x7f55641998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.113+0000 7f556a001700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f55641135d0 con 0x7f5564109520 2024-06-04T05:16:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.113+0000 7f556a001700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5564113450 con 0x7f55641024f0 2024-06-04T05:16:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.113+0000 7f556a001700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5564113750 con 0x7f5564101b30 2024-06-04T05:16:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f5563fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5564109520 0x7f55641998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f55637fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5564101b30 0x7f556419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f5562ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55641024f0 0x7f556419fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f5562ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55641024f0 0x7f556419fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:32962/0 (socket says 172.21.15.70:32962) 2024-06-04T05:16:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f55637fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5564101b30 0x7f556419f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:43718/0 (socket says 172.21.15.70:43718) 2024-06-04T05:16:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f5562ffd700 1 -- 172.21.15.70:0/2984561738 learned_addr learned my addr 172.21.15.70:0/2984561738 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f5563fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5564109520 0x7f55641998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47190/0 (socket says 172.21.15.70:47190) 2024-06-04T05:16:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f5563fff700 1 -- 172.21.15.70:0/2984561738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5564101b30 msgr2=0x7f556419f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f5563fff700 1 --2- 172.21.15.70:0/2984561738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5564101b30 0x7f556419f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f5563fff700 1 -- 172.21.15.70:0/2984561738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55641024f0 msgr2=0x7f556419fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f5563fff700 1 --2- 172.21.15.70:0/2984561738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55641024f0 0x7f556419fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f5563fff700 1 -- 172.21.15.70:0/2984561738 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f556419a030 con 0x7f5564109520 2024-06-04T05:16:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f5563fff700 1 --2- 172.21.15.70:0/2984561738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5564109520 0x7f55641998d0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f555c00b0d0 tx=0x7f555c007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f5560ff9700 1 -- 172.21.15.70:0/2984561738 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c01f070 con 0x7f5564109520 2024-06-04T05:16:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f556a001700 1 -- 172.21.15.70:0/2984561738 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f556419a2c0 con 0x7f5564109520 2024-06-04T05:16:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f5560ff9700 1 -- 172.21.15.70:0/2984561738 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f555c005500 con 0x7f5564109520 2024-06-04T05:16:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f5560ff9700 1 -- 172.21.15.70:0/2984561738 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c01b690 con 0x7f5564109520 2024-06-04T05:16:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.114+0000 7f556a001700 1 -- 172.21.15.70:0/2984561738 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55641a65e0 con 0x7f5564109520 2024-06-04T05:16:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.117+0000 7f5560ff9700 1 -- 172.21.15.70:0/2984561738 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f555c022070 con 0x7f5564109520 2024-06-04T05:16:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.118+0000 7f5560ff9700 1 --2- 172.21.15.70:0/2984561738 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f554c0b29f0 0x7f554c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.118+0000 7f55637fe700 1 --2- 172.21.15.70:0/2984561738 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f554c0b29f0 0x7f554c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.118+0000 7f5560ff9700 1 -- 172.21.15.70:0/2984561738 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f555c0e3550 con 0x7f5564109520 2024-06-04T05:16:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.118+0000 7f5574896b80 1 Processor -- start 2024-06-04T05:16:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.118+0000 7f55637fe700 1 --2- 172.21.15.70:0/2984561738 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f554c0b29f0 0x7f554c0b4eb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f556419aa10 tx=0x7f55580073d0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.118+0000 7f5574896b80 1 -- start start 2024-06-04T05:16:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f5574896b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888125860 0x55b888125c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f5574896b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888126210 0x55b888129940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f5574896b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888129f60 0x55b88812a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f5574896b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b888084c90 con 0x55b888125860 2024-06-04T05:16:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f5574896b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b888074830 con 0x55b888126210 2024-06-04T05:16:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f5574896b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b888078870 con 0x55b888129f60 2024-06-04T05:16:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f55637fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888125860 0x55b888125c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f55637fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888125860 0x55b888125c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47200/0 (socket says 172.21.15.70:47200) 2024-06-04T05:16:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f55637fe700 1 -- 172.21.15.70:0/3121506840 learned_addr learned my addr 172.21.15.70:0/3121506840 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f5562ffd700 1 --2- 172.21.15.70:0/3121506840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888126210 0x55b888129940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f5563fff700 1 --2- 172.21.15.70:0/3121506840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888129f60 0x55b88812a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f55637fe700 1 -- 172.21.15.70:0/3121506840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888129f60 msgr2=0x55b88812a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f55637fe700 1 --2- 172.21.15.70:0/3121506840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888129f60 0x55b88812a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f55637fe700 1 -- 172.21.15.70:0/3121506840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888126210 msgr2=0x55b888129940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f55637fe700 1 --2- 172.21.15.70:0/3121506840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888126210 0x55b888129940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f55637fe700 1 -- 172.21.15.70:0/3121506840 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b887fc48d0 con 0x55b888125860 2024-06-04T05:16:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f5562ffd700 1 --2- 172.21.15.70:0/3121506840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888126210 0x55b888129940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:16:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f55637fe700 1 --2- 172.21.15.70:0/3121506840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888125860 0x55b888125c40 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f5558004570 tx=0x7f5558013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d9f3d99acb7d9b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f5549ffb700 1 -- 172.21.15.70:0/3121506840 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5558024070 con 0x55b888125860 2024-06-04T05:16:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f5549ffb700 1 -- 172.21.15.70:0/3121506840 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5558015070 con 0x55b888125860 2024-06-04T05:16:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.119+0000 7f5549ffb700 1 -- 172.21.15.70:0/3121506840 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5558011940 con 0x55b888125860 2024-06-04T05:16:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.120+0000 7f5574896b80 1 -- 172.21.15.70:0/3121506840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888125860 msgr2=0x55b888125c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.120+0000 7f5574896b80 1 --2- 172.21.15.70:0/3121506840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888125860 0x55b888125c40 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f5558004570 tx=0x7f5558013000 comp rx=0 tx=0).stop 2024-06-04T05:16:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.120+0000 7f5574896b80 1 -- 172.21.15.70:0/3121506840 shutdown_connections 2024-06-04T05:16:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.120+0000 7f5574896b80 1 --2- 172.21.15.70:0/3121506840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888129f60 0x55b88812a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.120+0000 7f5574896b80 1 --2- 172.21.15.70:0/3121506840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888125860 0x55b888125c40 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.120+0000 7f5574896b80 1 --2- 172.21.15.70:0/3121506840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888126210 0x55b888129940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.120+0000 7f5574896b80 1 -- 172.21.15.70:0/3121506840 >> 172.21.15.70:0/3121506840 conn(0x55b888053080 msgr2=0x55b88811fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.120+0000 7f5574896b80 1 -- 172.21.15.70:0/3121506840 shutdown_connections 2024-06-04T05:16:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.120+0000 7f5574896b80 1 -- 172.21.15.70:0/3121506840 wait complete. 2024-06-04T05:16:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.120+0000 7f5574896b80 1 -- start start 2024-06-04T05:16:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5574896b80 1 Processor -- start 2024-06-04T05:16:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5574896b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888125860 0x55b888131400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5574896b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888126210 0x55b88812e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5574896b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888129f60 0x55b88812ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5574896b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b887f7afa0 con 0x55b888126210 2024-06-04T05:16:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5574896b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b8880167c0 con 0x55b888125860 2024-06-04T05:16:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5574896b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b888132120 con 0x55b888129f60 2024-06-04T05:16:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5574896b80 10 client.0 _ll_register_callbacks cb 0x55b88800f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5562ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888126210 0x55b88812e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5562ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888126210 0x55b88812e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47210/0 (socket says 172.21.15.70:47210) 2024-06-04T05:16:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5562ffd700 1 -- 172.21.15.70:0/655807294 learned_addr learned my addr 172.21.15.70:0/655807294 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5563fff700 1 --2- 172.21.15.70:0/655807294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888129f60 0x55b88812ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f55637fe700 1 --2- 172.21.15.70:0/655807294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888125860 0x55b888131400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5562ffd700 1 -- 172.21.15.70:0/655807294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888129f60 msgr2=0x55b88812ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5562ffd700 1 --2- 172.21.15.70:0/655807294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888129f60 0x55b88812ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5562ffd700 1 -- 172.21.15.70:0/655807294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888125860 msgr2=0x55b888131400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5562ffd700 1 --2- 172.21.15.70:0/655807294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888125860 0x55b888131400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5562ffd700 1 -- 172.21.15.70:0/655807294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b888013690 con 0x55b888126210 2024-06-04T05:16:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5562ffd700 1 --2- 172.21.15.70:0/655807294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888126210 0x55b88812e700 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f5554011460 tx=0x7f5554011770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d01ada93e74771 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.121+0000 7f5542ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.122+0000 7f5542ffd700 1 -- 172.21.15.70:0/655807294 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5554012860 con 0x55b888126210 2024-06-04T05:16:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.122+0000 7f5542ffd700 1 -- 172.21.15.70:0/655807294 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f5554012a00 con 0x55b888126210 2024-06-04T05:16:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.122+0000 7f5574896b80 1 -- 172.21.15.70:0/655807294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5558015070 con 0x55b888126210 2024-06-04T05:16:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.122+0000 7f5542ffd700 1 -- 172.21.15.70:0/655807294 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f5554012c00 con 0x55b888126210 2024-06-04T05:16:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.122+0000 7f5574896b80 10 client.20571 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.122+0000 7f5574896b80 1 -- 172.21.15.70:0/655807294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55b88813c890 con 0x55b888126210 2024-06-04T05:16:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.122+0000 7f5542ffd700 1 -- 172.21.15.70:0/655807294 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5558015070 con 0x55b888126210 2024-06-04T05:16:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.122+0000 7f5542ffd700 1 -- 172.21.15.70:0/655807294 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f5554023390 con 0x55b888126210 2024-06-04T05:16:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.122+0000 7f5574896b80 10 client.20571 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.122+0000 7f5574896b80 10 client.20571 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:16:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.122+0000 7f5574896b80 20 client.20571 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.122+0000 7f5574896b80 1 --2- 172.21.15.70:0/655807294 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b88813eed0 0x55b8881412c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.122+0000 7f5574896b80 4 client.20571 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:16:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.122+0000 7f5574896b80 1 -- 172.21.15.70:0/655807294 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8880f19e0 con 0x55b88813eed0 2024-06-04T05:16:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.123+0000 7f55637fe700 1 --2- 172.21.15.70:0/655807294 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b88813eed0 0x55b8881412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.123+0000 7f5542ffd700 10 client.20571 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.123+0000 7f55637fe700 1 --2- 172.21.15.70:0/655807294 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b88813eed0 0x55b8881412c0 crc :-1 s=READY pgs=421 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a8415c201514f66 server_cookie=e83387d6fd6c0977 in_seq=0 out_seq=0 2024-06-04T05:16:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.124+0000 7f5542ffd700 1 -- 172.21.15.70:0/655807294 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b8880f19e0 con 0x55b88813eed0 2024-06-04T05:16:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.124+0000 7f5542ffd700 10 client.20571 handle_command_reply: tid=0 2024-06-04T05:16:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.124+0000 7f5574896b80 1 client.20571 shutdown 2024-06-04T05:16:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.124+0000 7f5574896b80 1 -- 172.21.15.70:0/655807294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888126210 msgr2=0x55b88812e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.124+0000 7f5574896b80 1 --2- 172.21.15.70:0/655807294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888126210 0x55b88812e700 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f5554011460 tx=0x7f5554011770 comp rx=0 tx=0).stop 2024-06-04T05:16:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.124+0000 7f5574896b80 1 -- 172.21.15.70:0/655807294 shutdown_connections 2024-06-04T05:16:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.124+0000 7f5574896b80 1 --2- 172.21.15.70:0/655807294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888129f60 0x55b88812ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.124+0000 7f5574896b80 1 --2- 172.21.15.70:0/655807294 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b88813eed0 0x55b8881412c0 crc :-1 s=READY pgs=421 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.124+0000 7f5574896b80 1 --2- 172.21.15.70:0/655807294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888125860 0x55b888131400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.124+0000 7f5574896b80 1 --2- 172.21.15.70:0/655807294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888126210 0x55b88812e700 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.124+0000 7f5542ffd700 0 client.20571 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.124+0000 7f5574896b80 1 -- 172.21.15.70:0/655807294 >> 172.21.15.70:0/655807294 conn(0x55b888053080 msgr2=0x55b88811f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.124+0000 7f5574896b80 1 -- 172.21.15.70:0/655807294 shutdown_connections 2024-06-04T05:16:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.124+0000 7f5574896b80 1 -- 172.21.15.70:0/655807294 wait complete. 2024-06-04T05:16:51.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.124+0000 7f5574896b80 20 client.20571 trim_cache size 0 max 16384 2024-06-04T05:16:51.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.139+0000 7f5574896b80 1 Processor -- start 2024-06-04T05:16:51.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.139+0000 7f5574896b80 1 -- start start 2024-06-04T05:16:51.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f5574896b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888125860 0x55b88813b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f5574896b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888126210 0x55b88812ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f5574896b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888129f60 0x55b88811fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f5574896b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f55641adce0 con 0x55b888126210 2024-06-04T05:16:51.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f5574896b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f55641af820 con 0x55b888125860 2024-06-04T05:16:51.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f5574896b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f55641ae3b0 con 0x55b888129f60 2024-06-04T05:16:51.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f5562ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888126210 0x55b88812ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f5562ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888126210 0x55b88812ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47220/0 (socket says 172.21.15.70:47220) 2024-06-04T05:16:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f5562ffd700 1 -- 172.21.15.70:0/2463359054 learned_addr learned my addr 172.21.15.70:0/2463359054 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f5562ffd700 1 -- 172.21.15.70:0/2463359054 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888129f60 msgr2=0x55b88811fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f5562ffd700 1 --2- 172.21.15.70:0/2463359054 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888129f60 0x55b88811fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f5562ffd700 1 -- 172.21.15.70:0/2463359054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888125860 msgr2=0x55b88813b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f5562ffd700 1 --2- 172.21.15.70:0/2463359054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888125860 0x55b88813b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f5562ffd700 1 -- 172.21.15.70:0/2463359054 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8881418f0 con 0x55b888126210 2024-06-04T05:16:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f5562ffd700 1 --2- 172.21.15.70:0/2463359054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888126210 0x55b88812ed30 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f55540239a0 tx=0x7f555400b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45ce5f234857b751 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f55497fa700 1 -- 172.21.15.70:0/2463359054 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5554028040 con 0x55b888126210 2024-06-04T05:16:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f55497fa700 1 -- 172.21.15.70:0/2463359054 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5554023590 con 0x55b888126210 2024-06-04T05:16:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.140+0000 7f55497fa700 1 -- 172.21.15.70:0/2463359054 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5554005280 con 0x55b888126210 2024-06-04T05:16:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.141+0000 7f5574896b80 1 -- 172.21.15.70:0/2463359054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888126210 msgr2=0x55b88812ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.141+0000 7f5574896b80 1 --2- 172.21.15.70:0/2463359054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888126210 0x55b88812ed30 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f55540239a0 tx=0x7f555400b9d0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.141+0000 7f5574896b80 1 -- 172.21.15.70:0/2463359054 shutdown_connections 2024-06-04T05:16:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.141+0000 7f5574896b80 1 --2- 172.21.15.70:0/2463359054 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888129f60 0x55b88811fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.141+0000 7f5574896b80 1 --2- 172.21.15.70:0/2463359054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888125860 0x55b88813b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.141+0000 7f5574896b80 1 --2- 172.21.15.70:0/2463359054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888126210 0x55b88812ed30 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.141+0000 7f5574896b80 1 -- 172.21.15.70:0/2463359054 >> 172.21.15.70:0/2463359054 conn(0x55b888053080 msgr2=0x55b888124f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.141+0000 7f5574896b80 1 -- 172.21.15.70:0/2463359054 shutdown_connections 2024-06-04T05:16:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.141+0000 7f5574896b80 1 -- 172.21.15.70:0/2463359054 wait complete. 2024-06-04T05:16:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.141+0000 7f5574896b80 1 -- start start 2024-06-04T05:16:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5574896b80 1 Processor -- start 2024-06-04T05:16:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5574896b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888125860 0x55b888135880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5574896b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888126210 0x55b888135dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5574896b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888129f60 0x55b8881a8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5574896b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f55641af150 con 0x55b888129f60 2024-06-04T05:16:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5574896b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f55641aea80 con 0x55b888126210 2024-06-04T05:16:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5574896b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f55641abca0 con 0x55b888125860 2024-06-04T05:16:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5574896b80 10 client.0 _ll_register_callbacks cb 0x55b88800f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5563fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888129f60 0x55b8881a8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5563fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888129f60 0x55b8881a8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47224/0 (socket says 172.21.15.70:47224) 2024-06-04T05:16:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5563fff700 1 -- 172.21.15.70:0/361875940 learned_addr learned my addr 172.21.15.70:0/361875940 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5562ffd700 1 --2- 172.21.15.70:0/361875940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888126210 0x55b888135dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f55637fe700 1 --2- 172.21.15.70:0/361875940 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888125860 0x55b888135880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5563fff700 1 -- 172.21.15.70:0/361875940 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888125860 msgr2=0x55b888135880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5563fff700 1 --2- 172.21.15.70:0/361875940 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888125860 0x55b888135880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5563fff700 1 -- 172.21.15.70:0/361875940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888126210 msgr2=0x55b888135dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5563fff700 1 --2- 172.21.15.70:0/361875940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888126210 0x55b888135dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5563fff700 1 -- 172.21.15.70:0/361875940 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8881a8f60 con 0x55b888129f60 2024-06-04T05:16:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f55637fe700 1 --2- 172.21.15.70:0/361875940 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888125860 0x55b888135880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:16:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5563fff700 1 --2- 172.21.15.70:0/361875940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888129f60 0x55b8881a8a20 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f555c0a68b0 tx=0x7f555c0aa180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d64149a0d79017cd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.142+0000 7f5542ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.143+0000 7f5542ffd700 1 -- 172.21.15.70:0/361875940 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c029070 con 0x55b888129f60 2024-06-04T05:16:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.143+0000 7f5542ffd700 1 -- 172.21.15.70:0/361875940 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f555c0056b0 con 0x55b888129f60 2024-06-04T05:16:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.143+0000 7f5574896b80 1 -- 172.21.15.70:0/361875940 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5554023590 con 0x55b888129f60 2024-06-04T05:16:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.143+0000 7f5542ffd700 1 -- 172.21.15.70:0/361875940 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f555c0aa830 con 0x55b888129f60 2024-06-04T05:16:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.143+0000 7f5574896b80 10 client.20577 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.143+0000 7f5574896b80 1 -- 172.21.15.70:0/361875940 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55b8881aa290 con 0x55b888129f60 2024-06-04T05:16:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.143+0000 7f5542ffd700 1 -- 172.21.15.70:0/361875940 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5554023590 con 0x55b888129f60 2024-06-04T05:16:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.143+0000 7f5542ffd700 1 -- 172.21.15.70:0/361875940 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f555c011070 con 0x55b888129f60 2024-06-04T05:16:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.143+0000 7f5574896b80 10 client.20577 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.143+0000 7f5574896b80 10 client.20577 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:16:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.143+0000 7f5574896b80 20 client.20577 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.143+0000 7f5574896b80 1 --2- 172.21.15.70:0/361875940 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b8881aa940 0x55b8881acd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.144+0000 7f5574896b80 4 client.20577 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:16:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.144+0000 7f5574896b80 1 -- 172.21.15.70:0/361875940 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b888086be0 con 0x55b8881aa940 2024-06-04T05:16:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.144+0000 7f55637fe700 1 --2- 172.21.15.70:0/361875940 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b8881aa940 0x55b8881acd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.144+0000 7f5542ffd700 10 client.20577 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.144+0000 7f55637fe700 1 --2- 172.21.15.70:0/361875940 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b8881aa940 0x55b8881acd30 crc :-1 s=READY pgs=422 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3317e722006ae8b server_cookie=cf33ae3fd71e8387 in_seq=0 out_seq=0 2024-06-04T05:16:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.144+0000 7f5542ffd700 1 -- 172.21.15.70:0/361875940 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b888086be0 con 0x55b8881aa940 2024-06-04T05:16:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.144+0000 7f5542ffd700 10 client.20577 handle_command_reply: tid=0 2024-06-04T05:16:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.144+0000 7f5574896b80 1 client.20577 shutdown 2024-06-04T05:16:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.144+0000 7f5574896b80 1 -- 172.21.15.70:0/361875940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888129f60 msgr2=0x55b8881a8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.144+0000 7f5574896b80 1 --2- 172.21.15.70:0/361875940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888129f60 0x55b8881a8a20 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f555c0a68b0 tx=0x7f555c0aa180 comp rx=0 tx=0).stop 2024-06-04T05:16:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f5574896b80 1 -- 172.21.15.70:0/361875940 shutdown_connections 2024-06-04T05:16:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f5574896b80 1 --2- 172.21.15.70:0/361875940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b888129f60 0x55b8881a8a20 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f5574896b80 1 --2- 172.21.15.70:0/361875940 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b888125860 0x55b888135880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f5574896b80 1 --2- 172.21.15.70:0/361875940 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b8881aa940 0x55b8881acd30 crc :-1 s=READY pgs=422 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.274 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:16:51.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f5574896b80 1 --2- 172.21.15.70:0/361875940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b888126210 0x55b888135dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f5542ffd700 0 client.20577 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:51.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f5574896b80 1 -- 172.21.15.70:0/361875940 >> 172.21.15.70:0/361875940 conn(0x55b888053080 msgr2=0x55b8881288b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:51.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f5574896b80 1 -- 172.21.15.70:0/361875940 shutdown_connections 2024-06-04T05:16:51.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f5574896b80 1 -- 172.21.15.70:0/361875940 wait complete. 2024-06-04T05:16:51.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f5574896b80 20 client.20577 trim_cache size 0 max 16384 2024-06-04T05:16:51.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f556a001700 1 -- 172.21.15.70:0/2984561738 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f554c0b29f0 msgr2=0x7f554c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f556a001700 1 --2- 172.21.15.70:0/2984561738 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f554c0b29f0 0x7f554c0b4eb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f556419aa10 tx=0x7f55580073d0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f556a001700 1 -- 172.21.15.70:0/2984561738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5564109520 msgr2=0x7f55641998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f556a001700 1 --2- 172.21.15.70:0/2984561738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5564109520 0x7f55641998d0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f555c00b0d0 tx=0x7f555c007d90 comp rx=0 tx=0).stop 2024-06-04T05:16:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f556a001700 1 -- 172.21.15.70:0/2984561738 shutdown_connections 2024-06-04T05:16:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f556a001700 1 --2- 172.21.15.70:0/2984561738 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f554c0b29f0 0x7f554c0b4eb0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f556a001700 1 --2- 172.21.15.70:0/2984561738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5564109520 0x7f55641998d0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f556a001700 1 --2- 172.21.15.70:0/2984561738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5564101b30 0x7f556419f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f556a001700 1 --2- 172.21.15.70:0/2984561738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55641024f0 0x7f556419fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.145+0000 7f556a001700 1 -- 172.21.15.70:0/2984561738 >> 172.21.15.70:0/2984561738 conn(0x7f5564078ab0 msgr2=0x7f5564100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.146+0000 7f556a001700 1 -- 172.21.15.70:0/2984561738 shutdown_connections 2024-06-04T05:16:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:51.146+0000 7f556a001700 1 -- 172.21.15.70:0/2984561738 wait complete. 2024-06-04T05:16:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:51 smithi070 ceph-mon[32631]: pgmap v617: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 177 KiB/s wr, 13 op/s 2024-06-04T05:16:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:51 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:16:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:51 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:16:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:51 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:16:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:51 smithi112 ceph-mon[36453]: pgmap v617: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 177 KiB/s wr, 13 op/s 2024-06-04T05:16:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:51 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:16:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:51 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:16:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:51 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:16:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:51 smithi098 ceph-mon[35569]: pgmap v617: 97 pgs: 97 active+clean; 709 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 177 KiB/s wr, 13 op/s 2024-06-04T05:16:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:51 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:16:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:51 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:16:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:51 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:16:51.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.51 MB/sec execute 223 sec latency 19247.294 ms 2024-06-04T05:16:52.649 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id 49a0c915 --release --await' 2024-06-04T05:16:52.650 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id 49a0c915 --release --await 2024-06-04T05:16:52.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.48 MB/sec execute 224 sec latency 20247.357 ms 2024-06-04T05:16:52.897 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:16:53.008 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:16:53.008 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:16:53.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:53 smithi070 ceph-mon[32631]: pgmap v618: 97 pgs: 97 active+clean; 705 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 201 KiB/s wr, 24 op/s 2024-06-04T05:16:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:53 smithi112 ceph-mon[36453]: pgmap v618: 97 pgs: 97 active+clean; 705 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 201 KiB/s wr, 24 op/s 2024-06-04T05:16:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:53 smithi098 ceph-mon[35569]: pgmap v618: 97 pgs: 97 active+clean; 705 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 201 KiB/s wr, 24 op/s 2024-06-04T05:16:53.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43435 6.45 MB/sec execute 225 sec latency 21247.388 ms 2024-06-04T05:16:54.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.671+0000 7fc9b26c4700 1 -- 172.21.15.70:0/4056647429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9ac102a80 msgr2=0x7fc9ac102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:54.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.671+0000 7fc9b26c4700 1 --2- 172.21.15.70:0/4056647429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9ac102a80 0x7fc9ac102e60 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0099c0 tx=0x7fc99c009cd0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.671+0000 7fc9b26c4700 1 -- 172.21.15.70:0/4056647429 shutdown_connections 2024-06-04T05:16:54.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.671+0000 7fc9b26c4700 1 --2- 172.21.15.70:0/4056647429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9ac1077d0 0x7fc9ac10f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.671+0000 7fc9b26c4700 1 --2- 172.21.15.70:0/4056647429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9ac102a80 0x7fc9ac102e60 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.671+0000 7fc9b26c4700 1 --2- 172.21.15.70:0/4056647429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9ac103430 0x7fc9ac1070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.671+0000 7fc9b26c4700 1 -- 172.21.15.70:0/4056647429 >> 172.21.15.70:0/4056647429 conn(0x7fc9ac0793f0 msgr2=0x7fc9ac079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:54.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.671+0000 7fc9b26c4700 1 -- 172.21.15.70:0/4056647429 shutdown_connections 2024-06-04T05:16:54.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.671+0000 7fc9b26c4700 1 -- 172.21.15.70:0/4056647429 wait complete. 2024-06-04T05:16:54.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.671+0000 7fc9b26c4700 1 Processor -- start 2024-06-04T05:16:54.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9b26c4700 1 -- start start 2024-06-04T05:16:54.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9b26c4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9ac102a80 0x7fc9ac104140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9b26c4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9ac103430 0x7fc9ac104680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9b26c4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9ac1077d0 0x7fc9ac104bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9b26c4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac1125c0 con 0x7fc9ac1077d0 2024-06-04T05:16:54.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9b26c4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac112440 con 0x7fc9ac103430 2024-06-04T05:16:54.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9b26c4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac112740 con 0x7fc9ac102a80 2024-06-04T05:16:54.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9b0c61700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9ac1077d0 0x7fc9ac104bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9b0c61700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9ac1077d0 0x7fc9ac104bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47238/0 (socket says 172.21.15.70:47238) 2024-06-04T05:16:54.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9b0c61700 1 -- 172.21.15.70:0/2764065599 learned_addr learned my addr 172.21.15.70:0/2764065599 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:54.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9abfff700 1 --2- 172.21.15.70:0/2764065599 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9ac102a80 0x7fc9ac104140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9ab7fe700 1 --2- 172.21.15.70:0/2764065599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9ac103430 0x7fc9ac104680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9b0c61700 1 -- 172.21.15.70:0/2764065599 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9ac102a80 msgr2=0x7fc9ac104140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:54.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9b0c61700 1 --2- 172.21.15.70:0/2764065599 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9ac102a80 0x7fc9ac104140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9b0c61700 1 -- 172.21.15.70:0/2764065599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9ac103430 msgr2=0x7fc9ac104680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:54.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9b0c61700 1 --2- 172.21.15.70:0/2764065599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9ac103430 0x7fc9ac104680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9b0c61700 1 -- 172.21.15.70:0/2764065599 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9ac1a81d0 con 0x7fc9ac1077d0 2024-06-04T05:16:54.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.672+0000 7fc9ab7fe700 1 --2- 172.21.15.70:0/2764065599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9ac103430 0x7fc9ac104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:16:54.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.673+0000 7fc9b0c61700 1 --2- 172.21.15.70:0/2764065599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9ac1077d0 0x7fc9ac104bc0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400b970 tx=0x7fc9a400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:54.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.673+0000 7fc9a97fa700 1 -- 172.21.15.70:0/2764065599 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4019070 con 0x7fc9ac1077d0 2024-06-04T05:16:54.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.673+0000 7fc9a97fa700 1 -- 172.21.15.70:0/2764065599 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9a400cdc0 con 0x7fc9ac1077d0 2024-06-04T05:16:54.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.673+0000 7fc9b26c4700 1 -- 172.21.15.70:0/2764065599 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9ac1a84c0 con 0x7fc9ac1077d0 2024-06-04T05:16:54.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.673+0000 7fc9a97fa700 1 -- 172.21.15.70:0/2764065599 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4014e30 con 0x7fc9ac1077d0 2024-06-04T05:16:54.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.673+0000 7fc9b26c4700 1 -- 172.21.15.70:0/2764065599 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9ac1a8a50 con 0x7fc9ac1077d0 2024-06-04T05:16:54.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.675+0000 7fc9a97fa700 1 -- 172.21.15.70:0/2764065599 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fc9a401c070 con 0x7fc9ac1077d0 2024-06-04T05:16:54.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.676+0000 7fc9a97fa700 1 --2- 172.21.15.70:0/2764065599 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9940b2a70 0x7fc9940b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.676+0000 7fc9abfff700 1 --2- 172.21.15.70:0/2764065599 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9940b2a70 0x7fc9940b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.676+0000 7fc9a97fa700 1 -- 172.21.15.70:0/2764065599 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7fc9a40e1800 con 0x7fc9ac1077d0 2024-06-04T05:16:54.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.676+0000 7fc9abfff700 1 --2- 172.21.15.70:0/2764065599 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9940b2a70 0x7fc9940b4f30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc99c006010 tx=0x7fc99c009f10 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:54.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.676+0000 7fc9bcf59b80 1 Processor -- start 2024-06-04T05:16:54.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9bcf59b80 1 -- start start 2024-06-04T05:16:54.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9bcf59b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e6842b0 0x55e13e684690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9bcf59b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e67e8e0 0x55e13e67ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9bcf59b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e68b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9bcf59b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e13e5ddd90 con 0x55e13e67f2a0 2024-06-04T05:16:54.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9bcf59b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e13e5cd930 con 0x55e13e67e8e0 2024-06-04T05:16:54.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9bcf59b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e13e5d1970 con 0x55e13e6842b0 2024-06-04T05:16:54.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9b0c61700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e68b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9b0c61700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e68b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47252/0 (socket says 172.21.15.70:47252) 2024-06-04T05:16:54.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9b0c61700 1 -- 172.21.15.70:0/795969793 learned_addr learned my addr 172.21.15.70:0/795969793 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:54.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9abfff700 1 --2- 172.21.15.70:0/795969793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e6842b0 0x55e13e684690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9ab7fe700 1 --2- 172.21.15.70:0/795969793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e67e8e0 0x55e13e67ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9b0c61700 1 -- 172.21.15.70:0/795969793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e6842b0 msgr2=0x55e13e684690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:54.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9b0c61700 1 --2- 172.21.15.70:0/795969793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e6842b0 0x55e13e684690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9b0c61700 1 -- 172.21.15.70:0/795969793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e67e8e0 msgr2=0x55e13e67ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:54.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9b0c61700 1 --2- 172.21.15.70:0/795969793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e67e8e0 0x55e13e67ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9b0c61700 1 -- 172.21.15.70:0/795969793 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e13e51d9d0 con 0x55e13e67f2a0 2024-06-04T05:16:54.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9ab7fe700 1 --2- 172.21.15.70:0/795969793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e67e8e0 0x55e13e67ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:16:54.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9b0c61700 1 --2- 172.21.15.70:0/795969793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e68b0e0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fc9a40df520 tx=0x7fc9a40a8400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86e1f990a04b12 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:54.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.677+0000 7fc9927fc700 1 -- 172.21.15.70:0/795969793 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a40a92c0 con 0x55e13e67f2a0 2024-06-04T05:16:54.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.678+0000 7fc9927fc700 1 -- 172.21.15.70:0/795969793 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9a40a8bc0 con 0x55e13e67f2a0 2024-06-04T05:16:54.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.678+0000 7fc9927fc700 1 -- 172.21.15.70:0/795969793 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a40a9080 con 0x55e13e67f2a0 2024-06-04T05:16:54.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.678+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/795969793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 msgr2=0x55e13e68b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:54.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.678+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/795969793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e68b0e0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fc9a40df520 tx=0x7fc9a40a8400 comp rx=0 tx=0).stop 2024-06-04T05:16:54.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.678+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/795969793 shutdown_connections 2024-06-04T05:16:54.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.678+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/795969793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e68b0e0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.678+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/795969793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e6842b0 0x55e13e684690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.678+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/795969793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e67e8e0 0x55e13e67ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.678+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/795969793 >> 172.21.15.70:0/795969793 conn(0x55e13e5ac180 msgr2=0x55e13e6791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:54.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.678+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/795969793 shutdown_connections 2024-06-04T05:16:54.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.678+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/795969793 wait complete. 2024-06-04T05:16:54.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.679+0000 7fc9bcf59b80 1 -- start start 2024-06-04T05:16:54.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.679+0000 7fc9bcf59b80 1 Processor -- start 2024-06-04T05:16:54.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.679+0000 7fc9bcf59b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e67e8e0 0x55e13e68aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.679+0000 7fc9bcf59b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e67f2a0 0x55e13e6881d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.679+0000 7fc9bcf59b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e6842b0 0x55e13e688710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.679+0000 7fc9bcf59b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e13e4d40a0 con 0x55e13e6842b0 2024-06-04T05:16:54.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.679+0000 7fc9bcf59b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e13e56f8c0 con 0x55e13e67e8e0 2024-06-04T05:16:54.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.679+0000 7fc9bcf59b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e13e68b6e0 con 0x55e13e67f2a0 2024-06-04T05:16:54.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.679+0000 7fc9bcf59b80 10 client.0 _ll_register_callbacks cb 0x55e13e568a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.679+0000 7fc9b0c61700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e6842b0 0x55e13e688710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.679+0000 7fc9b0c61700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e6842b0 0x55e13e688710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47268/0 (socket says 172.21.15.70:47268) 2024-06-04T05:16:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.679+0000 7fc9b0c61700 1 -- 172.21.15.70:0/1038400188 learned_addr learned my addr 172.21.15.70:0/1038400188 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc9ab7fe700 1 --2- 172.21.15.70:0/1038400188 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e67f2a0 0x55e13e6881d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc9abfff700 1 --2- 172.21.15.70:0/1038400188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e67e8e0 0x55e13e68aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc9b0c61700 1 -- 172.21.15.70:0/1038400188 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e67f2a0 msgr2=0x55e13e6881d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc9b0c61700 1 --2- 172.21.15.70:0/1038400188 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e67f2a0 0x55e13e6881d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc9b0c61700 1 -- 172.21.15.70:0/1038400188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e67e8e0 msgr2=0x55e13e68aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc9b0c61700 1 --2- 172.21.15.70:0/1038400188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e67e8e0 0x55e13e68aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc9b0c61700 1 -- 172.21.15.70:0/1038400188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e13e56c790 con 0x55e13e6842b0 2024-06-04T05:16:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc9abfff700 1 --2- 172.21.15.70:0/1038400188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e67e8e0 0x55e13e68aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:16:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc9ab7fe700 1 --2- 172.21.15.70:0/1038400188 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e67f2a0 0x55e13e6881d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:16:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc9b0c61700 1 --2- 172.21.15.70:0/1038400188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e6842b0 0x55e13e688710 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fc9a40a88c0 tx=0x7fc9a4004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf98108032a38e9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc98b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc98b7fe700 1 -- 172.21.15.70:0/1038400188 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a40c7020 con 0x55e13e6842b0 2024-06-04T05:16:54.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/1038400188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9a40a8bc0 con 0x55e13e6842b0 2024-06-04T05:16:54.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc98b7fe700 1 -- 172.21.15.70:0/1038400188 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7fc9a40b17a0 con 0x55e13e6842b0 2024-06-04T05:16:54.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc98b7fe700 1 -- 172.21.15.70:0/1038400188 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fc9a408c050 con 0x55e13e6842b0 2024-06-04T05:16:54.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc9bcf59b80 10 client.20589 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:54.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/1038400188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55e13e695ee0 con 0x55e13e6842b0 2024-06-04T05:16:54.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc98b7fe700 1 -- 172.21.15.70:0/1038400188 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9a40a8bc0 con 0x55e13e6842b0 2024-06-04T05:16:54.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.680+0000 7fc98b7fe700 1 -- 172.21.15.70:0/1038400188 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fc9a40b1040 con 0x55e13e6842b0 2024-06-04T05:16:54.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.681+0000 7fc9bcf59b80 10 client.20589 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:54.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.681+0000 7fc9bcf59b80 10 client.20589 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:16:54.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.681+0000 7fc9bcf59b80 20 client.20589 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:54.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.681+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/1038400188 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e13e6983b0 0x55e13e69a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.681+0000 7fc9bcf59b80 4 client.20589 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:16:54.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.681+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/1038400188 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e13e64aae0 con 0x55e13e6983b0 2024-06-04T05:16:54.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.681+0000 7fc9abfff700 1 --2- 172.21.15.70:0/1038400188 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e13e6983b0 0x55e13e69a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.681+0000 7fc98b7fe700 10 client.20589 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:54.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.681+0000 7fc9abfff700 1 --2- 172.21.15.70:0/1038400188 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e13e6983b0 0x55e13e69a7a0 crc :-1 s=READY pgs=423 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea188dbcbba4303f server_cookie=abc82adc95cdf6a2 in_seq=0 out_seq=0 2024-06-04T05:16:54.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.682+0000 7fc98b7fe700 1 -- 172.21.15.70:0/1038400188 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e13e64aae0 con 0x55e13e6983b0 2024-06-04T05:16:54.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.682+0000 7fc98b7fe700 10 client.20589 handle_command_reply: tid=0 2024-06-04T05:16:54.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.682+0000 7fc9bcf59b80 1 client.20589 shutdown 2024-06-04T05:16:54.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.682+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/1038400188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e6842b0 msgr2=0x55e13e688710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:54.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.682+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/1038400188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e6842b0 0x55e13e688710 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fc9a40a88c0 tx=0x7fc9a4004be0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.682+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/1038400188 shutdown_connections 2024-06-04T05:16:54.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.682+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/1038400188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e6842b0 0x55e13e688710 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.683+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/1038400188 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e13e6983b0 0x55e13e69a7a0 crc :-1 s=READY pgs=423 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.683+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/1038400188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e67e8e0 0x55e13e68aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.683+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/1038400188 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e67f2a0 0x55e13e6881d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.683+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/1038400188 >> 172.21.15.70:0/1038400188 conn(0x55e13e5ac180 msgr2=0x55e13e673ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:54.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.683+0000 7fc98b7fe700 0 client.20589 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:54.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.683+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/1038400188 shutdown_connections 2024-06-04T05:16:54.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.683+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/1038400188 wait complete. 2024-06-04T05:16:54.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.683+0000 7fc9bcf59b80 20 client.20589 trim_cache size 0 max 16384 2024-06-04T05:16:54.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.702+0000 7fc9bcf59b80 1 Processor -- start 2024-06-04T05:16:54.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.702+0000 7fc9bcf59b80 1 -- start start 2024-06-04T05:16:54.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9bcf59b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e67e8e0 0x55e13e673b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9bcf59b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e68aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9bcf59b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e6842b0 0x55e13e693970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9bcf59b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac1b0210 con 0x55e13e67f2a0 2024-06-04T05:16:54.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9bcf59b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac1b1d50 con 0x55e13e6842b0 2024-06-04T05:16:54.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9bcf59b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac1b08e0 con 0x55e13e67e8e0 2024-06-04T05:16:54.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9ab7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e68aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9ab7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e68aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47276/0 (socket says 172.21.15.70:47276) 2024-06-04T05:16:54.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9ab7fe700 1 -- 172.21.15.70:0/187885812 learned_addr learned my addr 172.21.15.70:0/187885812 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:54.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9abfff700 1 --2- 172.21.15.70:0/187885812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e67e8e0 0x55e13e673b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9b0c61700 1 --2- 172.21.15.70:0/187885812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e6842b0 0x55e13e693970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9ab7fe700 1 -- 172.21.15.70:0/187885812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e67e8e0 msgr2=0x55e13e673b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:54.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9ab7fe700 1 --2- 172.21.15.70:0/187885812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e67e8e0 0x55e13e673b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9ab7fe700 1 -- 172.21.15.70:0/187885812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e6842b0 msgr2=0x55e13e693970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:54.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9ab7fe700 1 --2- 172.21.15.70:0/187885812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e6842b0 0x55e13e693970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9ab7fe700 1 -- 172.21.15.70:0/187885812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e13e6848e0 con 0x55e13e67f2a0 2024-06-04T05:16:54.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9abfff700 1 --2- 172.21.15.70:0/187885812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e67e8e0 0x55e13e673b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:16:54.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc9ab7fe700 1 --2- 172.21.15.70:0/187885812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e68aa10 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac06ba80 tx=0x7fc9a000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81789725a23631b0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:54.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.703+0000 7fc991ffb700 1 -- 172.21.15.70:0/187885812 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a001a030 con 0x55e13e67f2a0 2024-06-04T05:16:54.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.704+0000 7fc991ffb700 1 -- 172.21.15.70:0/187885812 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9a0010d50 con 0x55e13e67f2a0 2024-06-04T05:16:54.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.704+0000 7fc991ffb700 1 -- 172.21.15.70:0/187885812 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a000f040 con 0x55e13e67f2a0 2024-06-04T05:16:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.704+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/187885812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 msgr2=0x55e13e68aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.704+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/187885812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e68aa10 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac06ba80 tx=0x7fc9a000ec80 comp rx=0 tx=0).stop 2024-06-04T05:16:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.704+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/187885812 shutdown_connections 2024-06-04T05:16:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.704+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/187885812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e6842b0 0x55e13e693970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.704+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/187885812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e67e8e0 0x55e13e673b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.704+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/187885812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e68aa10 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.704+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/187885812 >> 172.21.15.70:0/187885812 conn(0x55e13e5ac180 msgr2=0x55e13e673410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.704+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/187885812 shutdown_connections 2024-06-04T05:16:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.704+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/187885812 wait complete. 2024-06-04T05:16:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.705+0000 7fc9bcf59b80 1 -- start start 2024-06-04T05:16:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.705+0000 7fc9bcf59b80 1 Processor -- start 2024-06-04T05:16:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.705+0000 7fc9bcf59b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e67e8e0 0x55e13e69d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.705+0000 7fc9bcf59b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e69bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.705+0000 7fc9bcf59b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e6842b0 0x55e13e69c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.705+0000 7fc9bcf59b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac1b1680 con 0x55e13e67f2a0 2024-06-04T05:16:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.705+0000 7fc9bcf59b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac1b0fb0 con 0x55e13e67e8e0 2024-06-04T05:16:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.705+0000 7fc9bcf59b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac1ae1d0 con 0x55e13e6842b0 2024-06-04T05:16:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.705+0000 7fc9bcf59b80 10 client.0 _ll_register_callbacks cb 0x55e13e568a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.705+0000 7fc9ab7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e69bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.705+0000 7fc9abfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e67e8e0 0x55e13e69d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.705+0000 7fc9ab7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e69bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47286/0 (socket says 172.21.15.70:47286) 2024-06-04T05:16:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.705+0000 7fc9ab7fe700 1 -- 172.21.15.70:0/1179915283 learned_addr learned my addr 172.21.15.70:0/1179915283 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.706+0000 7fc9b0c61700 1 --2- 172.21.15.70:0/1179915283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e6842b0 0x55e13e69c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.706+0000 7fc9ab7fe700 1 -- 172.21.15.70:0/1179915283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e6842b0 msgr2=0x55e13e69c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.706+0000 7fc9ab7fe700 1 --2- 172.21.15.70:0/1179915283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e6842b0 0x55e13e69c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.706+0000 7fc9ab7fe700 1 -- 172.21.15.70:0/1179915283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e67e8e0 msgr2=0x55e13e69d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.706+0000 7fc9ab7fe700 1 --2- 172.21.15.70:0/1179915283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e67e8e0 0x55e13e69d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.706+0000 7fc9ab7fe700 1 -- 172.21.15.70:0/1179915283 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e13e69c910 con 0x55e13e67f2a0 2024-06-04T05:16:54.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.706+0000 7fc9ab7fe700 1 --2- 172.21.15.70:0/1179915283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e69bd50 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fc9a00048a0 tx=0x7fc9a0010f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a3e520a39ffd594 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:54.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.706+0000 7fc98b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:54.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.706+0000 7fc98b7fe700 1 -- 172.21.15.70:0/1179915283 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a0015dd0 con 0x55e13e67f2a0 2024-06-04T05:16:54.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.706+0000 7fc98b7fe700 1 -- 172.21.15.70:0/1179915283 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7fc9a001f440 con 0x55e13e67f2a0 2024-06-04T05:16:54.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.706+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/1179915283 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9a0010d50 con 0x55e13e67f2a0 2024-06-04T05:16:54.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.706+0000 7fc98b7fe700 1 -- 172.21.15.70:0/1179915283 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fc9a000e5a0 con 0x55e13e67f2a0 2024-06-04T05:16:54.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.706+0000 7fc9bcf59b80 10 client.20595 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:54.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.706+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/1179915283 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55e13e703af0 con 0x55e13e67f2a0 2024-06-04T05:16:54.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.707+0000 7fc98b7fe700 1 -- 172.21.15.70:0/1179915283 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9a0010d50 con 0x55e13e67f2a0 2024-06-04T05:16:54.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.707+0000 7fc98b7fe700 1 -- 172.21.15.70:0/1179915283 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fc9a0030070 con 0x55e13e67f2a0 2024-06-04T05:16:54.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.707+0000 7fc9bcf59b80 10 client.20595 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:54.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.707+0000 7fc9bcf59b80 10 client.20595 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:16:54.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.707+0000 7fc9bcf59b80 20 client.20595 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:54.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.707+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/1179915283 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e13e704050 0x55e13e706470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:54.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.707+0000 7fc9bcf59b80 4 client.20595 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "49a0c915", "release": true, "await": true}] 2024-06-04T05:16:54.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.707+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/1179915283 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "49a0c915", "release": true, "await": true}) -- 0x55e13e5dfce0 con 0x55e13e704050 2024-06-04T05:16:54.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.707+0000 7fc9abfff700 1 --2- 172.21.15.70:0/1179915283 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e13e704050 0x55e13e706470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:54.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.707+0000 7fc9abfff700 1 --2- 172.21.15.70:0/1179915283 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e13e704050 0x55e13e706470 crc :-1 s=READY pgs=424 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49b3dbd6d77da1b3 server_cookie=31c34e99d3ae3cdc in_seq=0 out_seq=0 2024-06-04T05:16:54.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.707+0000 7fc98b7fe700 10 client.20595 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:54.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.735+0000 7fc98b7fe700 1 -- 172.21.15.70:0/1179915283 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e13e5dfce0 con 0x55e13e704050 2024-06-04T05:16:54.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.735+0000 7fc98b7fe700 10 client.20595 handle_command_reply: tid=0 2024-06-04T05:16:54.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.735+0000 7fc9bcf59b80 1 client.20595 shutdown 2024-06-04T05:16:54.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.735+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/1179915283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 msgr2=0x55e13e69bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:54.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.735+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/1179915283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e69bd50 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fc9a00048a0 tx=0x7fc9a0010f70 comp rx=0 tx=0).stop 2024-06-04T05:16:54.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.736+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/1179915283 shutdown_connections 2024-06-04T05:16:54.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.736+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/1179915283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e13e6842b0 0x55e13e69c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.736+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/1179915283 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e13e704050 0x55e13e706470 crc :-1 s=READY pgs=424 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.736+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/1179915283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e13e67e8e0 0x55e13e69d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.736+0000 7fc9bcf59b80 1 --2- 172.21.15.70:0/1179915283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e13e67f2a0 0x55e13e69bd50 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.736+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/1179915283 >> 172.21.15.70:0/1179915283 conn(0x55e13e5ac180 msgr2=0x55e13e673410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:54.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.736+0000 7fc98b7fe700 0 client.20595 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:54.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.736+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/1179915283 shutdown_connections 2024-06-04T05:16:54.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.736+0000 7fc9bcf59b80 1 -- 172.21.15.70:0/1179915283 wait complete. 2024-06-04T05:16:54.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.736+0000 7fc9bcf59b80 20 client.20595 trim_cache size 0 max 16384 2024-06-04T05:16:54.739 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:16:54.739 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 57, 2024-06-04T05:16:54.739 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:16:54.739 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 64, 2024-06-04T05:16:54.739 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:16:54.740 INFO:teuthology.orchestra.run.smithi070.stdout: "49a0c915": { 2024-06-04T05:16:54.740 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 64, 2024-06-04T05:16:54.740 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:16:54.740 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:16:54.740 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:16:54.740 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:16:54.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:16:54.740 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:16:54.740 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 146.8, 2024-06-04T05:16:54.740 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:16:54.740 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:16:54.740 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:16:54.740 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:16:54.740 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:16:54.740 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:16:54.741 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:16:54.741 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:16:54.741 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:16:54.741 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:16:54.741 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:16:54.741 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:16:54.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.737+0000 7fc9b26c4700 1 -- 172.21.15.70:0/2764065599 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9940b2a70 msgr2=0x7fc9940b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:54.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.737+0000 7fc9b26c4700 1 --2- 172.21.15.70:0/2764065599 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9940b2a70 0x7fc9940b4f30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc99c006010 tx=0x7fc99c009f10 comp rx=0 tx=0).stop 2024-06-04T05:16:54.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.737+0000 7fc9b26c4700 1 -- 172.21.15.70:0/2764065599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9ac1077d0 msgr2=0x7fc9ac104bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:54.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.737+0000 7fc9b26c4700 1 --2- 172.21.15.70:0/2764065599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9ac1077d0 0x7fc9ac104bc0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400b970 tx=0x7fc9a400bc80 comp rx=0 tx=0).stop 2024-06-04T05:16:54.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.737+0000 7fc9b26c4700 1 -- 172.21.15.70:0/2764065599 shutdown_connections 2024-06-04T05:16:54.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.737+0000 7fc9b26c4700 1 --2- 172.21.15.70:0/2764065599 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9940b2a70 0x7fc9940b4f30 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.737+0000 7fc9b26c4700 1 --2- 172.21.15.70:0/2764065599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9ac1077d0 0x7fc9ac104bc0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.737+0000 7fc9b26c4700 1 --2- 172.21.15.70:0/2764065599 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9ac102a80 0x7fc9ac104140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.737+0000 7fc9b26c4700 1 --2- 172.21.15.70:0/2764065599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9ac103430 0x7fc9ac104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:54.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.737+0000 7fc9b26c4700 1 -- 172.21.15.70:0/2764065599 >> 172.21.15.70:0/2764065599 conn(0x7fc9ac0793f0 msgr2=0x7fc9ac06c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:54.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.737+0000 7fc9b26c4700 1 -- 172.21.15.70:0/2764065599 shutdown_connections 2024-06-04T05:16:54.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:54.737+0000 7fc9b26c4700 1 -- 172.21.15.70:0/2764065599 wait complete. 2024-06-04T05:16:54.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43445 6.42 MB/sec execute 226 sec latency 22175.148 ms 2024-06-04T05:16:55.571 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:16:55.575 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '49a0c915', total seconds elapsed: 36.2 2024-06-04T05:16:55.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:55 smithi070 ceph-mon[32631]: pgmap v619: 97 pgs: 97 active+clean; 705 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 36 KiB/s wr, 16 op/s 2024-06-04T05:16:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:55 smithi112 ceph-mon[36453]: pgmap v619: 97 pgs: 97 active+clean; 705 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 36 KiB/s wr, 16 op/s 2024-06-04T05:16:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:55 smithi098 ceph-mon[35569]: pgmap v619: 97 pgs: 97 active+clean; 705 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 36 KiB/s wr, 16 op/s 2024-06-04T05:16:55.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43581 6.42 MB/sec execute 227 sec latency 55.126 ms 2024-06-04T05:16:56.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43780 6.47 MB/sec execute 228 sec latency 58.790 ms 2024-06-04T05:16:57.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.434+0000 7f3b1a32d700 1 -- 172.21.15.70:0/330494538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b14101b30 msgr2=0x7f3b14101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:57.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.434+0000 7f3b1a32d700 1 --2- 172.21.15.70:0/330494538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b14101b30 0x7f3b14101fb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3b080098f0 tx=0x7f3b08009c00 comp rx=0 tx=0).stop 2024-06-04T05:16:57.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.434+0000 7f3b1a32d700 1 -- 172.21.15.70:0/330494538 shutdown_connections 2024-06-04T05:16:57.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.434+0000 7f3b1a32d700 1 --2- 172.21.15.70:0/330494538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b141024f0 0x7f3b141103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.434+0000 7f3b1a32d700 1 --2- 172.21.15.70:0/330494538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b14109520 0x7f3b14109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.434+0000 7f3b1a32d700 1 --2- 172.21.15.70:0/330494538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b14101b30 0x7f3b14101fb0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.434+0000 7f3b1a32d700 1 -- 172.21.15.70:0/330494538 >> 172.21.15.70:0/330494538 conn(0x7f3b14078ab0 msgr2=0x7f3b140ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:57.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.434+0000 7f3b1a32d700 1 -- 172.21.15.70:0/330494538 shutdown_connections 2024-06-04T05:16:57.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.434+0000 7f3b1a32d700 1 -- 172.21.15.70:0/330494538 wait complete. 2024-06-04T05:16:57.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.435+0000 7f3b1a32d700 1 Processor -- start 2024-06-04T05:16:57.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.435+0000 7f3b1a32d700 1 -- start start 2024-06-04T05:16:57.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.436+0000 7f3b1a32d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b14101b30 0x7f3b1419f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.436+0000 7f3b1a32d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b141024f0 0x7f3b1419fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.436+0000 7f3b1a32d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b14109520 0x7f3b14199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.436+0000 7f3b1a32d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3b14113660 con 0x7f3b141024f0 2024-06-04T05:16:57.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.436+0000 7f3b188ca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b14109520 0x7f3b14199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:57.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.436+0000 7f3b137fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b141024f0 0x7f3b1419fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:57.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.436+0000 7f3b137fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b141024f0 0x7f3b1419fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39252/0 (socket says 172.21.15.70:39252) 2024-06-04T05:16:57.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.436+0000 7f3b137fe700 1 -- 172.21.15.70:0/4132911114 learned_addr learned my addr 172.21.15.70:0/4132911114 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:57.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.436+0000 7f3b1a32d700 1 -- 172.21.15.70:0/4132911114 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b141134e0 con 0x7f3b14109520 2024-06-04T05:16:57.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.436+0000 7f3b1a32d700 1 -- 172.21.15.70:0/4132911114 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3b141137e0 con 0x7f3b14101b30 2024-06-04T05:16:57.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.436+0000 7f3b13fff700 1 --2- 172.21.15.70:0/4132911114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b14101b30 0x7f3b1419f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:57.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.436+0000 7f3b13fff700 1 -- 172.21.15.70:0/4132911114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b14109520 msgr2=0x7f3b14199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:57.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.436+0000 7f3b13fff700 1 --2- 172.21.15.70:0/4132911114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b14109520 0x7f3b14199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.436+0000 7f3b13fff700 1 -- 172.21.15.70:0/4132911114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b141024f0 msgr2=0x7f3b1419fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:57.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.436+0000 7f3b13fff700 1 --2- 172.21.15.70:0/4132911114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b141024f0 0x7f3b1419fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.436+0000 7f3b13fff700 1 -- 172.21.15.70:0/4132911114 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b1419a190 con 0x7f3b14101b30 2024-06-04T05:16:57.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.436+0000 7f3b137fe700 1 --2- 172.21.15.70:0/4132911114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b141024f0 0x7f3b1419fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:16:57.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.437+0000 7f3b13fff700 1 --2- 172.21.15.70:0/4132911114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b14101b30 0x7f3b1419f870 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400e9f0 tx=0x7f3b0400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:57.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.437+0000 7f3b117fa700 1 -- 172.21.15.70:0/4132911114 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0400cbd0 con 0x7f3b14101b30 2024-06-04T05:16:57.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.437+0000 7f3b117fa700 1 -- 172.21.15.70:0/4132911114 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b0400cd70 con 0x7f3b14101b30 2024-06-04T05:16:57.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.437+0000 7f3b117fa700 1 -- 172.21.15.70:0/4132911114 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b04018680 con 0x7f3b14101b30 2024-06-04T05:16:57.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.437+0000 7f3b1a32d700 1 -- 172.21.15.70:0/4132911114 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b1419a480 con 0x7f3b14101b30 2024-06-04T05:16:57.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.437+0000 7f3b1a32d700 1 -- 172.21.15.70:0/4132911114 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b14102c40 con 0x7f3b14101b30 2024-06-04T05:16:57.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.440+0000 7f3b117fa700 1 -- 172.21.15.70:0/4132911114 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3b04010430 con 0x7f3b14101b30 2024-06-04T05:16:57.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.441+0000 7f3b24bc2b80 1 Processor -- start 2024-06-04T05:16:57.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.441+0000 7f3b24bc2b80 1 -- start start 2024-06-04T05:16:57.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.441+0000 7f3b117fa700 1 --2- 172.21.15.70:0/4132911114 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3afc0b2af0 0x7f3afc0b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.441+0000 7f3b24bc2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 0x564746a18cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.441+0000 7f3b24bc2b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a19290 0x564746a1c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.441+0000 7f3b24bc2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a1cfe0 0x564746a1d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.441+0000 7f3b24bc2b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564746977d10 con 0x564746a19290 2024-06-04T05:16:57.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.441+0000 7f3b24bc2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5647469678b0 con 0x564746a1cfe0 2024-06-04T05:16:57.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.441+0000 7f3b24bc2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56474696b8f0 con 0x564746a188e0 2024-06-04T05:16:57.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.441+0000 7f3b117fa700 1 -- 172.21.15.70:0/4132911114 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f3b040e2710 con 0x7f3b14101b30 2024-06-04T05:16:57.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.442+0000 7f3b13fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 0x564746a18cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:57.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.442+0000 7f3b13fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 0x564746a18cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:50640/0 (socket says 172.21.15.70:50640) 2024-06-04T05:16:57.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.442+0000 7f3b13fff700 1 -- 172.21.15.70:0/965052302 learned_addr learned my addr 172.21.15.70:0/965052302 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:57.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.442+0000 7f3b188ca700 1 --2- 172.21.15.70:0/4132911114 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3afc0b2af0 0x7f3afc0b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:57.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.442+0000 7f3b137fe700 1 --2- 172.21.15.70:0/965052302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a19290 0x564746a1c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:57.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.442+0000 7f3b13fff700 1 -- 172.21.15.70:0/965052302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a1cfe0 msgr2=0x564746a1d420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-04T05:16:57.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.442+0000 7f3b13fff700 1 --2- 172.21.15.70:0/965052302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a1cfe0 0x564746a1d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.442+0000 7f3b13fff700 1 -- 172.21.15.70:0/965052302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a19290 msgr2=0x564746a1c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:57.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.442+0000 7f3b13fff700 1 --2- 172.21.15.70:0/965052302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a19290 0x564746a1c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.442+0000 7f3b13fff700 1 -- 172.21.15.70:0/965052302 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647468b7950 con 0x564746a188e0 2024-06-04T05:16:57.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.442+0000 7f3b137fe700 1 --2- 172.21.15.70:0/965052302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a19290 0x564746a1c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:16:57.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.442+0000 7f3b13fff700 1 --2- 172.21.15.70:0/965052302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 0x564746a18cc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3b040a59e0 tx=0x7f3b040a9150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c688d80f6a2ab8d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:57.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.442+0000 7f3afa7fc700 1 -- 172.21.15.70:0/965052302 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b040e0260 con 0x564746a188e0 2024-06-04T05:16:57.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.442+0000 7f3afa7fc700 1 -- 172.21.15.70:0/965052302 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b040aa060 con 0x564746a188e0 2024-06-04T05:16:57.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.442+0000 7f3afa7fc700 1 -- 172.21.15.70:0/965052302 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b040add10 con 0x564746a188e0 2024-06-04T05:16:57.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.443+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/965052302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 msgr2=0x564746a18cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:57.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.443+0000 7f3b188ca700 1 --2- 172.21.15.70:0/4132911114 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3afc0b2af0 0x7f3afc0b4fb0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c009fd0 tx=0x7f3b0c00b040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:57.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.443+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/965052302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 0x564746a18cc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3b040a59e0 tx=0x7f3b040a9150 comp rx=0 tx=0).stop 2024-06-04T05:16:57.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.443+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/965052302 shutdown_connections 2024-06-04T05:16:57.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.443+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/965052302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a1cfe0 0x564746a1d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.443+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/965052302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 0x564746a18cc0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.443+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/965052302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a19290 0x564746a1c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.443+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/965052302 >> 172.21.15.70:0/965052302 conn(0x564746946100 msgr2=0x564746a12f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:57.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.444+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/965052302 shutdown_connections 2024-06-04T05:16:57.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.444+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/965052302 wait complete. 2024-06-04T05:16:57.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.444+0000 7f3b24bc2b80 1 -- start start 2024-06-04T05:16:57.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.444+0000 7f3b24bc2b80 1 Processor -- start 2024-06-04T05:16:57.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b24bc2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 0x564746a24430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b24bc2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a19290 0x564746a21730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b24bc2b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a1cfe0 0x564746a21c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b24bc2b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56474686e020 con 0x564746a1cfe0 2024-06-04T05:16:57.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b24bc2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564746909840 con 0x564746a19290 2024-06-04T05:16:57.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b24bc2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564746a250e0 con 0x564746a188e0 2024-06-04T05:16:57.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b24bc2b80 10 client.0 _ll_register_callbacks cb 0x5647469029a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:57.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b13fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 0x564746a24430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:57.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b13fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 0x564746a24430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:50648/0 (socket says 172.21.15.70:50648) 2024-06-04T05:16:57.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b13fff700 1 -- 172.21.15.70:0/2692275945 learned_addr learned my addr 172.21.15.70:0/2692275945 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:57.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b137fe700 1 --2- 172.21.15.70:0/2692275945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a19290 0x564746a21730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:57.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b188ca700 1 --2- 172.21.15.70:0/2692275945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a1cfe0 0x564746a21c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:57.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b137fe700 1 -- 172.21.15.70:0/2692275945 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 msgr2=0x564746a24430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:57.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b137fe700 1 --2- 172.21.15.70:0/2692275945 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 0x564746a24430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b137fe700 1 -- 172.21.15.70:0/2692275945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a1cfe0 msgr2=0x564746a21c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:57.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b137fe700 1 --2- 172.21.15.70:0/2692275945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a1cfe0 0x564746a21c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b137fe700 1 -- 172.21.15.70:0/2692275945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564746906710 con 0x564746a19290 2024-06-04T05:16:57.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b188ca700 1 --2- 172.21.15.70:0/2692275945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a1cfe0 0x564746a21c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:16:57.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3b137fe700 1 --2- 172.21.15.70:0/2692275945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a19290 0x564746a21730 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3b08009810 tx=0x7f3b08010f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f737ddd00cab5dd8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:57.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.445+0000 7f3af37fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:16:57.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.446+0000 7f3af37fe700 1 -- 172.21.15.70:0/2692275945 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b08005450 con 0x564746a19290 2024-06-04T05:16:57.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.446+0000 7f3af37fe700 1 -- 172.21.15.70:0/2692275945 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f3b080055f0 con 0x564746a19290 2024-06-04T05:16:57.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.446+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/2692275945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b040aa060 con 0x564746a19290 2024-06-04T05:16:57.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.446+0000 7f3af37fe700 1 -- 172.21.15.70:0/2692275945 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f3b08003680 con 0x564746a19290 2024-06-04T05:16:57.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.446+0000 7f3b24bc2b80 10 client.29318 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:57.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.446+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/2692275945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x564746a2f750 con 0x564746a19290 2024-06-04T05:16:57.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.446+0000 7f3af37fe700 1 -- 172.21.15.70:0/2692275945 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b040aa060 con 0x564746a19290 2024-06-04T05:16:57.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.447+0000 7f3af37fe700 1 -- 172.21.15.70:0/2692275945 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f3b08029050 con 0x564746a19290 2024-06-04T05:16:57.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.447+0000 7f3b24bc2b80 10 client.29318 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:57.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.447+0000 7f3b24bc2b80 10 client.29318 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:16:57.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.447+0000 7f3b24bc2b80 20 client.29318 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:57.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.447+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/2692275945 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564746a31e30 0x564746a34220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.447+0000 7f3b24bc2b80 4 client.29318 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:16:57.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.447+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/2692275945 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647469e4a60 con 0x564746a31e30 2024-06-04T05:16:57.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.447+0000 7f3b13fff700 1 --2- 172.21.15.70:0/2692275945 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564746a31e30 0x564746a34220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:57.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.448+0000 7f3b13fff700 1 --2- 172.21.15.70:0/2692275945 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564746a31e30 0x564746a34220 crc :-1 s=READY pgs=425 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0755ca4d0589d03 server_cookie=1dc4a8c3bd3574d8 in_seq=0 out_seq=0 2024-06-04T05:16:57.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.448+0000 7f3af37fe700 10 client.29318 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:57.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.448+0000 7f3af37fe700 1 -- 172.21.15.70:0/2692275945 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5647469e4a60 con 0x564746a31e30 2024-06-04T05:16:57.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.448+0000 7f3af37fe700 10 client.29318 handle_command_reply: tid=0 2024-06-04T05:16:57.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.448+0000 7f3b24bc2b80 1 client.29318 shutdown 2024-06-04T05:16:57.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.449+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/2692275945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a19290 msgr2=0x564746a21730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:57.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.449+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/2692275945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a19290 0x564746a21730 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3b08009810 tx=0x7f3b08010f30 comp rx=0 tx=0).stop 2024-06-04T05:16:57.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.449+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/2692275945 shutdown_connections 2024-06-04T05:16:57.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.449+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/2692275945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a1cfe0 0x564746a21c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.449+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/2692275945 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 0x564746a24430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.449+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/2692275945 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564746a31e30 0x564746a34220 crc :-1 s=READY pgs=425 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.449+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/2692275945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a19290 0x564746a21730 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.449+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/2692275945 >> 172.21.15.70:0/2692275945 conn(0x564746946100 msgr2=0x564746a12590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:57.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.449+0000 7f3af37fe700 0 client.29318 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:57.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.449+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/2692275945 shutdown_connections 2024-06-04T05:16:57.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.449+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/2692275945 wait complete. 2024-06-04T05:16:57.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.449+0000 7f3b24bc2b80 20 client.29318 trim_cache size 0 max 16384 2024-06-04T05:16:57.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b24bc2b80 1 Processor -- start 2024-06-04T05:16:57.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b24bc2b80 1 -- start start 2024-06-04T05:16:57.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b24bc2b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a188e0 0x564746a12d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b24bc2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a19290 0x564746a13250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b24bc2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a1cfe0 0x564746a39ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b24bc2b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3b141abcf0 con 0x564746a188e0 2024-06-04T05:16:57.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b24bc2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b141ad830 con 0x564746a1cfe0 2024-06-04T05:16:57.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b24bc2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3b141ac3c0 con 0x564746a19290 2024-06-04T05:16:57.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b13fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a188e0 0x564746a12d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:57.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b137fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a19290 0x564746a13250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:57.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b13fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a188e0 0x564746a12d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39280/0 (socket says 172.21.15.70:39280) 2024-06-04T05:16:57.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b188ca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a1cfe0 0x564746a39ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:57.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b13fff700 1 -- 172.21.15.70:0/3276163964 learned_addr learned my addr 172.21.15.70:0/3276163964 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:57.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b137fe700 1 -- 172.21.15.70:0/3276163964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a1cfe0 msgr2=0x564746a39ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:57.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b137fe700 1 --2- 172.21.15.70:0/3276163964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a1cfe0 0x564746a39ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b137fe700 1 -- 172.21.15.70:0/3276163964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a188e0 msgr2=0x564746a12d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:57.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b137fe700 1 --2- 172.21.15.70:0/3276163964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a188e0 0x564746a12d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.478+0000 7f3b137fe700 1 -- 172.21.15.70:0/3276163964 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564746a34880 con 0x564746a19290 2024-06-04T05:16:57.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.479+0000 7f3b188ca700 1 --2- 172.21.15.70:0/3276163964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a1cfe0 0x564746a39ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:16:57.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.479+0000 7f3b13fff700 1 --2- 172.21.15.70:0/3276163964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a188e0 0x564746a12d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:16:57.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.479+0000 7f3b137fe700 1 --2- 172.21.15.70:0/3276163964 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a19290 0x564746a13250 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3b080098c0 tx=0x7f3b08009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9af9377853a994b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:57.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.479+0000 7f3af9ffb700 1 -- 172.21.15.70:0/3276163964 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b08036020 con 0x564746a19290 2024-06-04T05:16:57.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.479+0000 7f3af9ffb700 1 -- 172.21.15.70:0/3276163964 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b0800b560 con 0x564746a19290 2024-06-04T05:16:57.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.479+0000 7f3af9ffb700 1 -- 172.21.15.70:0/3276163964 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0802e030 con 0x564746a19290 2024-06-04T05:16:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.480+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/3276163964 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a19290 msgr2=0x564746a13250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.480+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/3276163964 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a19290 0x564746a13250 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3b080098c0 tx=0x7f3b08009c00 comp rx=0 tx=0).stop 2024-06-04T05:16:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.480+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/3276163964 shutdown_connections 2024-06-04T05:16:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.480+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/3276163964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a1cfe0 0x564746a39ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.480+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/3276163964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a188e0 0x564746a12d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.480+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/3276163964 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a19290 0x564746a13250 secure :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3b080098c0 tx=0x7f3b08009c00 comp rx=0 tx=0).stop 2024-06-04T05:16:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.480+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/3276163964 >> 172.21.15.70:0/3276163964 conn(0x564746946100 msgr2=0x564746a237f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.481+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/3276163964 shutdown_connections 2024-06-04T05:16:57.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.481+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/3276163964 wait complete. 2024-06-04T05:16:57.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.481+0000 7f3b24bc2b80 1 -- start start 2024-06-04T05:16:57.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.481+0000 7f3b24bc2b80 1 Processor -- start 2024-06-04T05:16:57.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.482+0000 7f3b24bc2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 0x564746a296a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.482+0000 7f3b24bc2b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a1cfe0 0x564746a29be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.482+0000 7f3b24bc2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a9a9b0 0x564746a9cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.482+0000 7f3b24bc2b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3b141ad160 con 0x564746a1cfe0 2024-06-04T05:16:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.482+0000 7f3b24bc2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b141aca90 con 0x564746a9a9b0 2024-06-04T05:16:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.482+0000 7f3b24bc2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3b141a9cb0 con 0x564746a188e0 2024-06-04T05:16:57.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.482+0000 7f3b137fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a1cfe0 0x564746a29be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:57.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.482+0000 7f3b24bc2b80 10 client.0 _ll_register_callbacks cb 0x5647469029a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:16:57.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.482+0000 7f3b137fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a1cfe0 0x564746a29be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39296/0 (socket says 172.21.15.70:39296) 2024-06-04T05:16:57.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.482+0000 7f3b137fe700 1 -- 172.21.15.70:0/2155005701 learned_addr learned my addr 172.21.15.70:0/2155005701 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:16:57.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.482+0000 7f3b13fff700 1 --2- 172.21.15.70:0/2155005701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 0x564746a296a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:57.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.482+0000 7f3b188ca700 1 --2- 172.21.15.70:0/2155005701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a9a9b0 0x564746a9cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:57.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.483+0000 7f3b137fe700 1 -- 172.21.15.70:0/2155005701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 msgr2=0x564746a296a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:57.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.483+0000 7f3b137fe700 1 --2- 172.21.15.70:0/2155005701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 0x564746a296a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.483+0000 7f3b137fe700 1 -- 172.21.15.70:0/2155005701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a9a9b0 msgr2=0x564746a9cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:16:57.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.483+0000 7f3b137fe700 1 --2- 172.21.15.70:0/2155005701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a9a9b0 0x564746a9cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.483+0000 7f3b137fe700 1 -- 172.21.15.70:0/2155005701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564746a9d2e0 con 0x564746a1cfe0 2024-06-04T05:16:57.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.483+0000 7f3b13fff700 1 --2- 172.21.15.70:0/2155005701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 0x564746a296a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:16:57.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.483+0000 7f3b137fe700 1 --2- 172.21.15.70:0/2155005701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a1cfe0 0x564746a29be0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f3b0800ba40 tx=0x7f3b08005b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5d72eaa4d1e817 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:16:57.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.483+0000 7f3af37fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:16:57.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.483+0000 7f3af37fe700 1 -- 172.21.15.70:0/2155005701 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0802e070 con 0x564746a1cfe0 2024-06-04T05:16:57.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.483+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/2155005701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b0800b560 con 0x564746a1cfe0 2024-06-04T05:16:57.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.483+0000 7f3af37fe700 1 -- 172.21.15.70:0/2155005701 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f3b08005660 con 0x564746a1cfe0 2024-06-04T05:16:57.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.483+0000 7f3af37fe700 1 -- 172.21.15.70:0/2155005701 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f3b08003820 con 0x564746a1cfe0 2024-06-04T05:16:57.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.483+0000 7f3b24bc2b80 10 client.20610 fetch_fsmap learned FSMap version 57 2024-06-04T05:16:57.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.483+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/2155005701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x564746a9e5f0 con 0x564746a1cfe0 2024-06-04T05:16:57.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.484+0000 7f3af37fe700 1 -- 172.21.15.70:0/2155005701 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b0800b560 con 0x564746a1cfe0 2024-06-04T05:16:57.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.484+0000 7f3af37fe700 1 -- 172.21.15.70:0/2155005701 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f3b08029070 con 0x564746a1cfe0 2024-06-04T05:16:57.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.484+0000 7f3b24bc2b80 10 client.20610 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:16:57.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.484+0000 7f3b24bc2b80 10 client.20610 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:16:57.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.484+0000 7f3b24bc2b80 20 client.20610 populate_metadata read hostname 'smithi070' 2024-06-04T05:16:57.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.484+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/2155005701 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564746a9eb40 0x564746aa0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:16:57.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.484+0000 7f3b24bc2b80 4 client.20610 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:16:57.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.484+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/2155005701 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564746979c60 con 0x564746a9eb40 2024-06-04T05:16:57.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.484+0000 7f3b13fff700 1 --2- 172.21.15.70:0/2155005701 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564746a9eb40 0x564746aa0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:16:57.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.485+0000 7f3af37fe700 10 client.20610 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:57.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.485+0000 7f3b13fff700 1 --2- 172.21.15.70:0/2155005701 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564746a9eb40 0x564746aa0f60 crc :-1 s=READY pgs=426 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39b35c85f6320f38 server_cookie=dbc8679514f2e3a6 in_seq=0 out_seq=0 2024-06-04T05:16:57.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.486+0000 7f3af37fe700 1 -- 172.21.15.70:0/2155005701 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564746979c60 con 0x564746a9eb40 2024-06-04T05:16:57.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.486+0000 7f3af37fe700 10 client.20610 handle_command_reply: tid=0 2024-06-04T05:16:57.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.486+0000 7f3b24bc2b80 1 client.20610 shutdown 2024-06-04T05:16:57.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.486+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/2155005701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a1cfe0 msgr2=0x564746a29be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:57.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.486+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/2155005701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a1cfe0 0x564746a29be0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f3b0800ba40 tx=0x7f3b08005b40 comp rx=0 tx=0).stop 2024-06-04T05:16:57.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.486+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/2155005701 shutdown_connections 2024-06-04T05:16:57.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.486+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/2155005701 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564746a9eb40 0x564746aa0f60 crc :-1 s=READY pgs=426 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.486+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/2155005701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564746a9a9b0 0x564746a9cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.486+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/2155005701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564746a188e0 0x564746a296a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.486+0000 7f3b24bc2b80 1 --2- 172.21.15.70:0/2155005701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564746a1cfe0 0x564746a29be0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.486+0000 7f3af37fe700 0 client.20610 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:16:57.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.486+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/2155005701 >> 172.21.15.70:0/2155005701 conn(0x564746946100 msgr2=0x564746a20fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:57.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.487+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/2155005701 shutdown_connections 2024-06-04T05:16:57.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.487+0000 7f3b24bc2b80 1 -- 172.21.15.70:0/2155005701 wait complete. 2024-06-04T05:16:57.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.487+0000 7f3b24bc2b80 20 client.20610 trim_cache size 0 max 16384 2024-06-04T05:16:57.492 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:16:57.492 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:16:57.492 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "174d61ee-a27b-41aa-9eab-89f28c49482c", 2024-06-04T05:16:57.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:16:57.492 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:16:57.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.488+0000 7f3b1a32d700 1 -- 172.21.15.70:0/4132911114 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3afc0b2af0 msgr2=0x7f3afc0b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:57.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.488+0000 7f3b1a32d700 1 --2- 172.21.15.70:0/4132911114 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3afc0b2af0 0x7f3afc0b4fb0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c009fd0 tx=0x7f3b0c00b040 comp rx=0 tx=0).stop 2024-06-04T05:16:57.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.488+0000 7f3b1a32d700 1 -- 172.21.15.70:0/4132911114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b14101b30 msgr2=0x7f3b1419f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:16:57.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.488+0000 7f3b1a32d700 1 --2- 172.21.15.70:0/4132911114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b14101b30 0x7f3b1419f870 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400e9f0 tx=0x7f3b0400ed00 comp rx=0 tx=0).stop 2024-06-04T05:16:57.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.488+0000 7f3b1a32d700 1 -- 172.21.15.70:0/4132911114 shutdown_connections 2024-06-04T05:16:57.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.488+0000 7f3b1a32d700 1 --2- 172.21.15.70:0/4132911114 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3afc0b2af0 0x7f3afc0b4fb0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.488+0000 7f3b1a32d700 1 --2- 172.21.15.70:0/4132911114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b14109520 0x7f3b14199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.488+0000 7f3b1a32d700 1 --2- 172.21.15.70:0/4132911114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b14101b30 0x7f3b1419f870 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.488+0000 7f3b1a32d700 1 --2- 172.21.15.70:0/4132911114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b141024f0 0x7f3b1419fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:16:57.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.488+0000 7f3b1a32d700 1 -- 172.21.15.70:0/4132911114 >> 172.21.15.70:0/4132911114 conn(0x7f3b14078ab0 msgr2=0x7f3b14100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:16:57.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.488+0000 7f3b1a32d700 1 -- 172.21.15.70:0/4132911114 shutdown_connections 2024-06-04T05:16:57.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:16:57.488+0000 7f3b1a32d700 1 -- 172.21.15.70:0/4132911114 wait complete. 2024-06-04T05:16:57.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:57 smithi070 ceph-mon[32631]: pgmap v620: 97 pgs: 97 active+clean; 711 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 750 KiB/s wr, 47 op/s 2024-06-04T05:16:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:57 smithi112 ceph-mon[36453]: pgmap v620: 97 pgs: 97 active+clean; 711 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 750 KiB/s wr, 47 op/s 2024-06-04T05:16:57.686 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:16:57 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:16:57.557+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000215 [...21,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL/ auth{1=1} fragtree_t(*^3) v48959 ap=2 f(v1470 m2024-06-04T05:16:30.098045+0000 11=11+0) n(v78 rc2024-06-04T05:16:30.098045+0000 b2861056 12=11+1) old_inodes=6 (inest sync->mix g=1) caps={15231=pAsLsXsFs/-@39095} | dirtyscattered=0 request=0 lock=0 importing=0 dirfrag=8 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 0x55d6deff1b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000215:[<0x10000000009/COREL v50>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x10000000215:[<0x10000000009/COREL v48959>,<0x10000000008/~dmtmp v35498>,<0x10000000007/client0 v2745>,<0x10000000006/clients v2266>,<0x10000000005/tmp v1745>,<0x10000000003/client.0 v1692>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v1688>,<0x10000000001/sv_0 v1656>,<0x10000000000/qa v1628>,<0x1/volumes v1572>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:16:30.081758+0000 12=12+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:16:30.098045+0000 b2861056 12=11+1)","memory_value.dirstat":"f(v1470 m2024-06-04T05:16:30.098045+0000 11=11+0)","memory_value.rstat":"n(v78 rc2024-06-04T05:16:30.098045+0000 b2861056 12=11+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:16:57.686 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:16:57 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:16:57.569+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000020c [...21,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/EXCEL/ auth{2=1,4=1} v48970 ap=2 f(v2668 m2024-06-04T05:16:29.934220+0000 2=2+0) n(v83 rc2024-06-04T05:16:29.934220+0000 b520192 3=2+1) old_inodes=6 (inest sync->mix g=2,4) caps={15231=pAsLsXsFs/-@40841} | dirtyscattered=0 request=0 lock=0 importing=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55d6decee100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000020c:[<0x10000000009/EXCEL v14>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x1000000020c:[<0x10000000009/EXCEL v48970>,<0x10000000008/~dmtmp v35498>,<0x10000000007/client0 v2745>,<0x10000000006/clients v2266>,<0x10000000005/tmp v1745>,<0x10000000003/client.0 v1692>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v1688>,<0x10000000001/sv_0 v1656>,<0x10000000000/qa v1628>,<0x1/volumes v1572>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:16:29.910823+0000 3=3+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:16:29.934220+0000 b520192 3=2+1)","memory_value.dirstat":"f(v2668 m2024-06-04T05:16:29.934220+0000 2=2+0)","memory_value.rstat":"n(v83 rc2024-06-04T05:16:29.934220+0000 b520192 3=2+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:16:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:57 smithi098 ceph-mon[35569]: pgmap v620: 97 pgs: 97 active+clean; 711 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 750 KiB/s wr, 47 op/s 2024-06-04T05:16:57.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 43961 6.48 MB/sec execute 229 sec latency 79.634 ms 2024-06-04T05:16:58.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:58 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:16:58.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:58.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:58 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:16:58.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:58 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:16:58.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:58 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:16:58.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:58 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:16:58.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:58 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:16:58.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:58 smithi070 ceph-mon[32631]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.i log and `damage ls` output for details 2024-06-04T05:16:58.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:58 smithi070 ceph-mon[32631]: Scrub error on inode 0x1000000020c (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/EXCEL) see mds.i log and `damage ls` output for details 2024-06-04T05:16:58.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:58.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:58.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:58.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:58 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:58.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:58 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:16:58.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:58.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:58 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:16:58.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:58 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:16:58.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:58 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:16:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:58 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:16:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:58 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:16:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:58 smithi112 ceph-mon[36453]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.i log and `damage ls` output for details 2024-06-04T05:16:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:58 smithi112 ceph-mon[36453]: Scrub error on inode 0x1000000020c (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/EXCEL) see mds.i log and `damage ls` output for details 2024-06-04T05:16:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:58.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:58 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:58.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:58 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:16:58.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:58.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:58 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:16:58.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:58 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:16:58.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:58 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:16:58.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:58 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:16:58.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:58 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:16:58.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:58 smithi098 ceph-mon[35569]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.i log and `damage ls` output for details 2024-06-04T05:16:58.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:58 smithi098 ceph-mon[35569]: Scrub error on inode 0x1000000020c (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/EXCEL) see mds.i log and `damage ls` output for details 2024-06-04T05:16:58.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:58.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:58.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:58.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:58 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:58.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 44139 6.49 MB/sec execute 230 sec latency 87.900 ms 2024-06-04T05:16:59.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:59 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:59.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:16:59 smithi070 ceph-mon[32631]: pgmap v621: 97 pgs: 97 active+clean; 711 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 738 KiB/s wr, 41 op/s 2024-06-04T05:16:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:59 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:16:59 smithi112 ceph-mon[36453]: pgmap v621: 97 pgs: 97 active+clean; 711 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 738 KiB/s wr, 41 op/s 2024-06-04T05:16:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:59 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:16:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:16:59 smithi098 ceph-mon[35569]: pgmap v621: 97 pgs: 97 active+clean; 711 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 738 KiB/s wr, 41 op/s 2024-06-04T05:16:59.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 44254 6.47 MB/sec execute 231 sec latency 135.205 ms 2024-06-04T05:17:00.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 44371 6.48 MB/sec execute 232 sec latency 117.658 ms 2024-06-04T05:17:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.933+0000 7f089d800640 1 -- 172.21.15.70:0/936173460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0898102890 msgr2=0x7f0898102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.933+0000 7f089d800640 1 --2- 172.21.15.70:0/936173460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0898102890 0x7f0898102d10 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f088c009930 tx=0x7f088c02a420 comp rx=0 tx=0).stop 2024-06-04T05:17:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.933+0000 7f089d800640 1 -- 172.21.15.70:0/936173460 shutdown_connections 2024-06-04T05:17:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.933+0000 7f089d800640 1 --2- 172.21.15.70:0/936173460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0898103250 0x7f089810f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.934+0000 7f089d800640 1 --2- 172.21.15.70:0/936173460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0898102890 0x7f0898102d10 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.934+0000 7f089d800640 1 --2- 172.21.15.70:0/936173460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08981088a0 0x7f0898108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.934+0000 7f089d800640 1 -- 172.21.15.70:0/936173460 >> 172.21.15.70:0/936173460 conn(0x7f08980fe3c0 msgr2=0x7f08981007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.934+0000 7f089d800640 1 -- 172.21.15.70:0/936173460 shutdown_connections 2024-06-04T05:17:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.934+0000 7f089d800640 1 -- 172.21.15.70:0/936173460 wait complete. 2024-06-04T05:17:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.934+0000 7f089d800640 1 Processor -- start 2024-06-04T05:17:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f089d800640 1 -- start start 2024-06-04T05:17:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f089d800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0898102890 0x7f0898074d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f089d800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0898103250 0x7f0898075290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f089d800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08981088a0 0x7f08980757d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f089d800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0898112a10 con 0x7f0898103250 2024-06-04T05:17:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f089d800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0898112890 con 0x7f08981088a0 2024-06-04T05:17:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f089d800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0898112b90 con 0x7f0898102890 2024-06-04T05:17:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f0896ffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0898102890 0x7f0898074d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f08977fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08981088a0 0x7f08980757d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f0896ffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0898102890 0x7f0898074d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:50680/0 (socket says 172.21.15.70:50680) 2024-06-04T05:17:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f08977fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08981088a0 0x7f08980757d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:34848/0 (socket says 172.21.15.70:34848) 2024-06-04T05:17:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f0896ffd640 1 -- 172.21.15.70:0/2401173905 learned_addr learned my addr 172.21.15.70:0/2401173905 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f08967fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0898103250 0x7f0898075290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f08977fe640 1 -- 172.21.15.70:0/2401173905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0898102890 msgr2=0x7f0898074d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f08977fe640 1 --2- 172.21.15.70:0/2401173905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0898102890 0x7f0898074d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f08977fe640 1 -- 172.21.15.70:0/2401173905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0898103250 msgr2=0x7f0898075290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f08977fe640 1 --2- 172.21.15.70:0/2401173905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0898103250 0x7f0898075290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.935+0000 7f08977fe640 1 -- 172.21.15.70:0/2401173905 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08981a7580 con 0x7f08981088a0 2024-06-04T05:17:00.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.936+0000 7f08977fe640 1 --2- 172.21.15.70:0/2401173905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08981088a0 0x7f08980757d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f089000ec50 tx=0x7f089000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:00.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.936+0000 7f087ffff640 1 -- 172.21.15.70:0/2401173905 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0890003b70 con 0x7f08981088a0 2024-06-04T05:17:00.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.936+0000 7f089d800640 1 -- 172.21.15.70:0/2401173905 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08981a7870 con 0x7f08981088a0 2024-06-04T05:17:00.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.936+0000 7f087ffff640 1 -- 172.21.15.70:0/2401173905 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0890003d10 con 0x7f08981088a0 2024-06-04T05:17:00.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.936+0000 7f087ffff640 1 -- 172.21.15.70:0/2401173905 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08900107a0 con 0x7f08981088a0 2024-06-04T05:17:00.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.936+0000 7f089d800640 1 -- 172.21.15.70:0/2401173905 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08981a7d90 con 0x7f08981088a0 2024-06-04T05:17:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.939+0000 7f087ffff640 1 -- 172.21.15.70:0/2401173905 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f0890003fc0 con 0x7f08981088a0 2024-06-04T05:17:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.939+0000 7f08a05f0740 1 Processor -- start 2024-06-04T05:17:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.939+0000 7f08a05f0740 1 -- start start 2024-06-04T05:17:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.939+0000 7f08a05f0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403eda00 0x561c403edde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f08a05f0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403ee320 0x561c403f86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f08a05f0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403f8be0 0x561c403fafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f08a05f0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561c4035fc30 con 0x561c403f8be0 2024-06-04T05:17:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f08a05f0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561c4034f7d0 con 0x561c403ee320 2024-06-04T05:17:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f08a05f0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561c40353810 con 0x561c403eda00 2024-06-04T05:17:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f087ffff640 1 --2- 172.21.15.70:0/2401173905 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08800b2950 0x7f08800b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f08967fc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403eda00 0x561c403edde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f0896ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403ee320 0x561c403f86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f08967fc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403eda00 0x561c403edde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:50686/0 (socket says 172.21.15.70:50686) 2024-06-04T05:17:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f08967fc640 1 -- 172.21.15.70:0/2404614430 learned_addr learned my addr 172.21.15.70:0/2404614430 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f08967fc640 1 --2- 172.21.15.70:0/2404614430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403f8be0 0x561c403fafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f08977fe640 1 --2- 172.21.15.70:0/2401173905 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08800b2950 0x7f08800b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f087ffff640 1 -- 172.21.15.70:0/2401173905 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f08900e2bb0 con 0x7f08981088a0 2024-06-04T05:17:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f08967fc640 1 -- 172.21.15.70:0/2404614430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403ee320 msgr2=0x561c403f86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f08967fc640 1 --2- 172.21.15.70:0/2404614430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403ee320 0x561c403f86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f08967fc640 1 -- 172.21.15.70:0/2404614430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403f8be0 msgr2=0x561c403fafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f08967fc640 1 --2- 172.21.15.70:0/2404614430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403f8be0 0x561c403fafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f08967fc640 1 -- 172.21.15.70:0/2404614430 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c40064210 con 0x561c403eda00 2024-06-04T05:17:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f0896ffd640 1 --2- 172.21.15.70:0/2404614430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403ee320 0x561c403f86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f08967fc640 1 --2- 172.21.15.70:0/2404614430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403eda00 0x561c403edde0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f088c006010 tx=0x7f088c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ede3323b06647c34 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.940+0000 7f08977fe640 1 --2- 172.21.15.70:0/2401173905 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08800b2950 0x7f08800b4e10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f08900e08d0 tx=0x7f089000ce70 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.941+0000 7f087d7fa640 1 -- 172.21.15.70:0/2404614430 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c035040 con 0x561c403eda00 2024-06-04T05:17:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.941+0000 7f087d7fa640 1 -- 172.21.15.70:0/2404614430 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f088c004020 con 0x561c403eda00 2024-06-04T05:17:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.941+0000 7f087d7fa640 1 -- 172.21.15.70:0/2404614430 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c003850 con 0x561c403eda00 2024-06-04T05:17:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.941+0000 7f08a05f0740 1 -- 172.21.15.70:0/2404614430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403eda00 msgr2=0x561c403edde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.941+0000 7f08a05f0740 1 --2- 172.21.15.70:0/2404614430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403eda00 0x561c403edde0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f088c006010 tx=0x7f088c009260 comp rx=0 tx=0).stop 2024-06-04T05:17:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.942+0000 7f08a05f0740 1 -- 172.21.15.70:0/2404614430 shutdown_connections 2024-06-04T05:17:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.942+0000 7f08a05f0740 1 --2- 172.21.15.70:0/2404614430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403ee320 0x561c403f86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.942+0000 7f08a05f0740 1 --2- 172.21.15.70:0/2404614430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403f8be0 0x561c403fafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.942+0000 7f08a05f0740 1 --2- 172.21.15.70:0/2404614430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403eda00 0x561c403edde0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.942+0000 7f08a05f0740 1 -- 172.21.15.70:0/2404614430 >> 172.21.15.70:0/2404614430 conn(0x561c403e9f10 msgr2=0x561c403ea2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.942+0000 7f08a05f0740 1 -- 172.21.15.70:0/2404614430 shutdown_connections 2024-06-04T05:17:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.942+0000 7f08a05f0740 1 -- 172.21.15.70:0/2404614430 wait complete. 2024-06-04T05:17:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.942+0000 7f08a05f0740 1 -- start start 2024-06-04T05:17:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.942+0000 7f08a05f0740 1 Processor -- start 2024-06-04T05:17:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f08a05f0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403eda00 0x561c403f5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f08a05f0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403ee320 0x561c403f61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f08a05f0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403f8be0 0x561c403f66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f08a05f0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561c40035790 con 0x561c403ee320 2024-06-04T05:17:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f0896ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403ee320 0x561c403f61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f08967fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403eda00 0x561c403f5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f0896ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403ee320 0x561c403f61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39330/0 (socket says 172.21.15.70:39330) 2024-06-04T05:17:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f0896ffd640 1 -- 172.21.15.70:0/4125379245 learned_addr learned my addr 172.21.15.70:0/4125379245 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f08a05f0740 1 -- 172.21.15.70:0/4125379245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561c4030b890 con 0x561c403eda00 2024-06-04T05:17:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f08967fc640 1 --2- 172.21.15.70:0/4125379245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403f8be0 0x561c403f66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f08a05f0740 1 -- 172.21.15.70:0/4125379245 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561c403fb7b0 con 0x561c403f8be0 2024-06-04T05:17:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f08a05f0740 10 client.0 _ll_register_callbacks cb 0x561c40063a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f08967fc640 1 -- 172.21.15.70:0/4125379245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403f8be0 msgr2=0x561c403f66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f08967fc640 1 --2- 172.21.15.70:0/4125379245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403f8be0 0x561c403f66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f08967fc640 1 -- 172.21.15.70:0/4125379245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403ee320 msgr2=0x561c403f61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f08967fc640 1 --2- 172.21.15.70:0/4125379245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403ee320 0x561c403f61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f08967fc640 1 -- 172.21.15.70:0/4125379245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c40130620 con 0x561c403eda00 2024-06-04T05:17:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f0896ffd640 1 --2- 172.21.15.70:0/4125379245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403ee320 0x561c403f61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f08967fc640 1 --2- 172.21.15.70:0/4125379245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403eda00 0x561c403f5c70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f088c000c00 tx=0x7f088c032dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa021a293801466d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.943+0000 7f087aff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:17:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.944+0000 7f087aff5640 1 -- 172.21.15.70:0/4125379245 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c035070 con 0x561c403eda00 2024-06-04T05:17:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.944+0000 7f08a05f0740 1 -- 172.21.15.70:0/4125379245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f088c004020 con 0x561c403eda00 2024-06-04T05:17:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.944+0000 7f087aff5640 1 -- 172.21.15.70:0/4125379245 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f088c045440 con 0x561c403eda00 2024-06-04T05:17:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.944+0000 7f087aff5640 1 -- 172.21.15.70:0/4125379245 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f088c06a050 con 0x561c403eda00 2024-06-04T05:17:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.944+0000 7f08a05f0740 10 client.29336 fetch_fsmap learned FSMap version 57 2024-06-04T05:17:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.944+0000 7f08a05f0740 1 -- 172.21.15.70:0/4125379245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x561c40407400 con 0x561c403eda00 2024-06-04T05:17:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.944+0000 7f087aff5640 1 -- 172.21.15.70:0/4125379245 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f088c004020 con 0x561c403eda00 2024-06-04T05:17:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.944+0000 7f087aff5640 1 -- 172.21.15.70:0/4125379245 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f088c003a40 con 0x561c403eda00 2024-06-04T05:17:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.944+0000 7f08a05f0740 10 client.29336 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:17:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.944+0000 7f08a05f0740 10 client.29336 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:17:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.945+0000 7f08a05f0740 20 client.29336 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.945+0000 7f08a05f0740 1 --2- 172.21.15.70:0/4125379245 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561c40408750 0x561c4040ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.945+0000 7f08a05f0740 4 client.29336 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:17:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.945+0000 7f08a05f0740 1 -- 172.21.15.70:0/4125379245 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c40361b80 con 0x561c40408750 2024-06-04T05:17:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.945+0000 7f0896ffd640 1 --2- 172.21.15.70:0/4125379245 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561c40408750 0x561c4040ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.945+0000 7f087aff5640 10 client.29336 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.945+0000 7f0896ffd640 1 --2- 172.21.15.70:0/4125379245 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561c40408750 0x561c4040ab40 crc :-1 s=READY pgs=427 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20444101c0ff3354 server_cookie=d7fecef2d391994a in_seq=0 out_seq=0 2024-06-04T05:17:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.946+0000 7f087aff5640 1 -- 172.21.15.70:0/4125379245 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561c40361b80 con 0x561c40408750 2024-06-04T05:17:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.946+0000 7f087aff5640 10 client.29336 handle_command_reply: tid=0 2024-06-04T05:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.946+0000 7f08a05f0740 1 client.29336 shutdown 2024-06-04T05:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.946+0000 7f08a05f0740 1 -- 172.21.15.70:0/4125379245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403eda00 msgr2=0x561c403f5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.946+0000 7f08a05f0740 1 --2- 172.21.15.70:0/4125379245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403eda00 0x561c403f5c70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f088c000c00 tx=0x7f088c032dd0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.946+0000 7f08a05f0740 1 -- 172.21.15.70:0/4125379245 shutdown_connections 2024-06-04T05:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.947+0000 7f08a05f0740 1 --2- 172.21.15.70:0/4125379245 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561c40408750 0x561c4040ab40 crc :-1 s=READY pgs=427 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.947+0000 7f08a05f0740 1 --2- 172.21.15.70:0/4125379245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403f8be0 0x561c403f66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.947+0000 7f08a05f0740 1 --2- 172.21.15.70:0/4125379245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403ee320 0x561c403f61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.947+0000 7f08a05f0740 1 --2- 172.21.15.70:0/4125379245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403eda00 0x561c403f5c70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.947+0000 7f08a05f0740 1 -- 172.21.15.70:0/4125379245 >> 172.21.15.70:0/4125379245 conn(0x561c403e9f10 msgr2=0x561c403efc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.947+0000 7f087aff5640 0 client.29336 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.947+0000 7f08a05f0740 1 -- 172.21.15.70:0/4125379245 shutdown_connections 2024-06-04T05:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.947+0000 7f08a05f0740 1 -- 172.21.15.70:0/4125379245 wait complete. 2024-06-04T05:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.947+0000 7f08a05f0740 20 client.29336 trim_cache size 0 max 16384 2024-06-04T05:17:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.971+0000 7f08a05f0740 1 Processor -- start 2024-06-04T05:17:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.971+0000 7f08a05f0740 1 -- start start 2024-06-04T05:17:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.971+0000 7f08a05f0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403eda00 0x561c403f4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.971+0000 7f08a05f0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403ee320 0x561c403f6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.971+0000 7f08a05f0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403f8be0 0x561c40404110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.971+0000 7f08a05f0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08981ac380 con 0x561c403f8be0 2024-06-04T05:17:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.971+0000 7f08a05f0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08981adec0 con 0x561c403eda00 2024-06-04T05:17:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.971+0000 7f08a05f0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f08981aca50 con 0x561c403ee320 2024-06-04T05:17:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.971+0000 7f08967fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403f8be0 0x561c40404110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.971+0000 7f08967fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403f8be0 0x561c40404110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39344/0 (socket says 172.21.15.70:39344) 2024-06-04T05:17:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.971+0000 7f08967fc640 1 -- 172.21.15.70:0/2182817084 learned_addr learned my addr 172.21.15.70:0/2182817084 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.971+0000 7f08967fc640 1 --2- 172.21.15.70:0/2182817084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403eda00 0x561c403f4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.971+0000 7f0896ffd640 1 --2- 172.21.15.70:0/2182817084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403ee320 0x561c403f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.972+0000 7f08967fc640 1 -- 172.21.15.70:0/2182817084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403ee320 msgr2=0x561c403f6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.972+0000 7f08967fc640 1 --2- 172.21.15.70:0/2182817084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403ee320 0x561c403f6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.972+0000 7f08967fc640 1 -- 172.21.15.70:0/2182817084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403eda00 msgr2=0x561c403f4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.972+0000 7f08967fc640 1 --2- 172.21.15.70:0/2182817084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403eda00 0x561c403f4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.972+0000 7f08967fc640 1 -- 172.21.15.70:0/2182817084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c403ed7c0 con 0x561c403f8be0 2024-06-04T05:17:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.972+0000 7f08967fc640 1 --2- 172.21.15.70:0/2182817084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403f8be0 0x561c40404110 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f088c0096c0 tx=0x7f088c009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=509bd6710bc928f2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.972+0000 7f087cff9640 1 -- 172.21.15.70:0/2182817084 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c035040 con 0x561c403f8be0 2024-06-04T05:17:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.972+0000 7f087cff9640 1 -- 172.21.15.70:0/2182817084 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f088c0035f0 con 0x561c403f8be0 2024-06-04T05:17:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.972+0000 7f087cff9640 1 -- 172.21.15.70:0/2182817084 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c0454a0 con 0x561c403f8be0 2024-06-04T05:17:00.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.973+0000 7f08a05f0740 1 -- 172.21.15.70:0/2182817084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403f8be0 msgr2=0x561c40404110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.973+0000 7f08a05f0740 1 --2- 172.21.15.70:0/2182817084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403f8be0 0x561c40404110 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f088c0096c0 tx=0x7f088c009520 comp rx=0 tx=0).stop 2024-06-04T05:17:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.973+0000 7f08a05f0740 1 -- 172.21.15.70:0/2182817084 shutdown_connections 2024-06-04T05:17:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.973+0000 7f08a05f0740 1 --2- 172.21.15.70:0/2182817084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403f8be0 0x561c40404110 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.973+0000 7f08a05f0740 1 --2- 172.21.15.70:0/2182817084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403ee320 0x561c403f6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.973+0000 7f08a05f0740 1 --2- 172.21.15.70:0/2182817084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403eda00 0x561c403f4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.973+0000 7f08a05f0740 1 -- 172.21.15.70:0/2182817084 >> 172.21.15.70:0/2182817084 conn(0x561c403e9f10 msgr2=0x561c40403a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.973+0000 7f08a05f0740 1 -- 172.21.15.70:0/2182817084 shutdown_connections 2024-06-04T05:17:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.973+0000 7f08a05f0740 1 -- 172.21.15.70:0/2182817084 wait complete. 2024-06-04T05:17:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.973+0000 7f08a05f0740 1 -- start start 2024-06-04T05:17:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08a05f0740 1 Processor -- start 2024-06-04T05:17:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08a05f0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403eda00 0x561c4040ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08a05f0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403ee320 0x561c4040b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08a05f0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403f8be0 0x561c40470f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08a05f0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08981ad7f0 con 0x561c403eda00 2024-06-04T05:17:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08a05f0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08981ad120 con 0x561c403f8be0 2024-06-04T05:17:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08a05f0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f08981aa380 con 0x561c403ee320 2024-06-04T05:17:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08a05f0740 10 client.0 _ll_register_callbacks cb 0x7f088c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08967fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403eda00 0x561c4040ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08967fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403eda00 0x561c4040ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39360/0 (socket says 172.21.15.70:39360) 2024-06-04T05:17:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08967fc640 1 -- 172.21.15.70:0/3681996174 learned_addr learned my addr 172.21.15.70:0/3681996174 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f0896ffd640 1 --2- 172.21.15.70:0/3681996174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403ee320 0x561c4040b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08967fc640 1 --2- 172.21.15.70:0/3681996174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403f8be0 0x561c40470f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08967fc640 1 -- 172.21.15.70:0/3681996174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403ee320 msgr2=0x561c4040b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08967fc640 1 --2- 172.21.15.70:0/3681996174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403ee320 0x561c4040b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08967fc640 1 -- 172.21.15.70:0/3681996174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403f8be0 msgr2=0x561c40470f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08967fc640 1 --2- 172.21.15.70:0/3681996174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403f8be0 0x561c40470f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08967fc640 1 -- 172.21.15.70:0/3681996174 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c4040ba90 con 0x561c403eda00 2024-06-04T05:17:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f08967fc640 1 --2- 172.21.15.70:0/3681996174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403eda00 0x561c4040ac80 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f088c004720 tx=0x7f088c03d450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d1316c62c2e958b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.974+0000 7f087aff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:17:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.975+0000 7f087aff5640 1 -- 172.21.15.70:0/3681996174 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c035070 con 0x561c403eda00 2024-06-04T05:17:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.975+0000 7f087aff5640 1 -- 172.21.15.70:0/3681996174 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8571+0+0 (secure 0 0 0) 0x7f088c0037a0 con 0x561c403eda00 2024-06-04T05:17:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.975+0000 7f08a05f0740 1 -- 172.21.15.70:0/3681996174 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f088c0035f0 con 0x561c403eda00 2024-06-04T05:17:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.975+0000 7f087aff5640 1 -- 172.21.15.70:0/3681996174 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f088c0039a0 con 0x561c403eda00 2024-06-04T05:17:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.975+0000 7f08a05f0740 10 client.20628 fetch_fsmap learned FSMap version 57 2024-06-04T05:17:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.975+0000 7f08a05f0740 1 -- 172.21.15.70:0/3681996174 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x561c404725a0 con 0x561c403eda00 2024-06-04T05:17:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.975+0000 7f087aff5640 1 -- 172.21.15.70:0/3681996174 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f088c0035f0 con 0x561c403eda00 2024-06-04T05:17:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.975+0000 7f087aff5640 1 -- 172.21.15.70:0/3681996174 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f088c03d970 con 0x561c403eda00 2024-06-04T05:17:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.975+0000 7f08a05f0740 10 client.20628 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:17:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.975+0000 7f08a05f0740 10 client.20628 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:17:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.975+0000 7f08a05f0740 20 client.20628 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.975+0000 7f08a05f0740 1 --2- 172.21.15.70:0/3681996174 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561c40472b30 0x561c40474f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.975+0000 7f08a05f0740 4 client.20628 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:17:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.975+0000 7f08a05f0740 1 -- 172.21.15.70:0/3681996174 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561c4035b360 con 0x561c40472b30 2024-06-04T05:17:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.975+0000 7f0896ffd640 1 --2- 172.21.15.70:0/3681996174 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561c40472b30 0x561c40474f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.976+0000 7f087aff5640 10 client.20628 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:00.976+0000 7f0896ffd640 1 --2- 172.21.15.70:0/3681996174 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561c40472b30 0x561c40474f50 crc :-1 s=READY pgs=428 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc36e88b20e1bb24 server_cookie=e0891d6491b0a501 in_seq=0 out_seq=0 2024-06-04T05:17:01.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.067+0000 7f087aff5640 1 -- 172.21.15.70:0/3681996174 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x561c4035b360 con 0x561c40472b30 2024-06-04T05:17:01.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.067+0000 7f087aff5640 10 client.20628 handle_command_reply: tid=0 2024-06-04T05:17:01.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.067+0000 7f08a05f0740 1 client.20628 shutdown 2024-06-04T05:17:01.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.067+0000 7f08a05f0740 1 -- 172.21.15.70:0/3681996174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403eda00 msgr2=0x561c4040ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:01.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.067+0000 7f08a05f0740 1 --2- 172.21.15.70:0/3681996174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403eda00 0x561c4040ac80 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f088c004720 tx=0x7f088c03d450 comp rx=0 tx=0).stop 2024-06-04T05:17:01.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.067+0000 7f08a05f0740 1 -- 172.21.15.70:0/3681996174 shutdown_connections 2024-06-04T05:17:01.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.067+0000 7f08a05f0740 1 --2- 172.21.15.70:0/3681996174 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561c40472b30 0x561c40474f50 crc :-1 s=READY pgs=428 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:01.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.067+0000 7f08a05f0740 1 --2- 172.21.15.70:0/3681996174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c403f8be0 0x561c40470f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:01.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.067+0000 7f08a05f0740 1 --2- 172.21.15.70:0/3681996174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561c403ee320 0x561c4040b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:01.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.067+0000 7f08a05f0740 1 --2- 172.21.15.70:0/3681996174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c403eda00 0x561c4040ac80 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:01.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.067+0000 7f087aff5640 0 client.20628 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:01.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.067+0000 7f08a05f0740 1 -- 172.21.15.70:0/3681996174 >> 172.21.15.70:0/3681996174 conn(0x561c403e9f10 msgr2=0x561c40404290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:01.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.067+0000 7f08a05f0740 1 -- 172.21.15.70:0/3681996174 shutdown_connections 2024-06-04T05:17:01.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.067+0000 7f08a05f0740 1 -- 172.21.15.70:0/3681996174 wait complete. 2024-06-04T05:17:01.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.067+0000 7f08a05f0740 20 client.20628 trim_cache size 0 max 16384 2024-06-04T05:17:01.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:17:01.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:17:01.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:17:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.068+0000 7f089d800640 1 -- 172.21.15.70:0/2401173905 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08800b2950 msgr2=0x7f08800b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.068+0000 7f089d800640 1 --2- 172.21.15.70:0/2401173905 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08800b2950 0x7f08800b4e10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f08900e08d0 tx=0x7f089000ce70 comp rx=0 tx=0).stop 2024-06-04T05:17:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.068+0000 7f089d800640 1 -- 172.21.15.70:0/2401173905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08981088a0 msgr2=0x7f08980757d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.068+0000 7f089d800640 1 --2- 172.21.15.70:0/2401173905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08981088a0 0x7f08980757d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f089000ec50 tx=0x7f089000c6a0 comp rx=0 tx=0).stop 2024-06-04T05:17:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.068+0000 7f089d800640 1 -- 172.21.15.70:0/2401173905 shutdown_connections 2024-06-04T05:17:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.068+0000 7f089d800640 1 --2- 172.21.15.70:0/2401173905 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08800b2950 0x7f08800b4e10 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.068+0000 7f089d800640 1 --2- 172.21.15.70:0/2401173905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08981088a0 0x7f08980757d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.068+0000 7f089d800640 1 --2- 172.21.15.70:0/2401173905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0898103250 0x7f0898075290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.068+0000 7f089d800640 1 --2- 172.21.15.70:0/2401173905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0898102890 0x7f0898074d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.068+0000 7f089d800640 1 -- 172.21.15.70:0/2401173905 >> 172.21.15.70:0/2401173905 conn(0x7f08980fe3c0 msgr2=0x7f08980feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.068+0000 7f089d800640 1 -- 172.21.15.70:0/2401173905 shutdown_connections 2024-06-04T05:17:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:01.068+0000 7f089d800640 1 -- 172.21.15.70:0/2401173905 wait complete. 2024-06-04T05:17:01.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:01 smithi070 ceph-mon[32631]: pgmap v622: 97 pgs: 97 active+clean; 711 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 738 KiB/s wr, 41 op/s 2024-06-04T05:17:01.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:01 smithi112 ceph-mon[36453]: pgmap v622: 97 pgs: 97 active+clean; 711 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 738 KiB/s wr, 41 op/s 2024-06-04T05:17:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:01 smithi098 ceph-mon[35569]: pgmap v622: 97 pgs: 97 active+clean; 711 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 738 KiB/s wr, 41 op/s 2024-06-04T05:17:01.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 44530 6.49 MB/sec execute 233 sec latency 136.343 ms 2024-06-04T05:17:02.425 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:17:02.817 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:17:02.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 44679 6.49 MB/sec execute 234 sec latency 175.847 ms 2024-06-04T05:17:03.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:03 smithi070 ceph-mon[32631]: pgmap v623: 97 pgs: 97 active+clean; 722 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 2.8 MiB/s wr, 215 op/s 2024-06-04T05:17:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:03 smithi112 ceph-mon[36453]: pgmap v623: 97 pgs: 97 active+clean; 722 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 2.8 MiB/s wr, 215 op/s 2024-06-04T05:17:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:03 smithi098 ceph-mon[35569]: pgmap v623: 97 pgs: 97 active+clean; 722 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 2.8 MiB/s wr, 215 op/s 2024-06-04T05:17:03.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 44828 6.51 MB/sec execute 235 sec latency 126.412 ms 2024-06-04T05:17:04.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:04 smithi070 ceph-mon[32631]: osdmap e107: 12 total, 12 up, 12 in 2024-06-04T05:17:04.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:04 smithi112 ceph-mon[36453]: osdmap e107: 12 total, 12 up, 12 in 2024-06-04T05:17:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:04 smithi098 ceph-mon[35569]: osdmap e107: 12 total, 12 up, 12 in 2024-06-04T05:17:04.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 44973 6.49 MB/sec execute 236 sec latency 59.322 ms 2024-06-04T05:17:05.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.401+0000 7f109de20700 1 -- 172.21.15.70:0/926124370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1098102df0 msgr2=0x7f109810f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:05.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.401+0000 7f109de20700 1 --2- 172.21.15.70:0/926124370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1098102df0 0x7f109810f2e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1090009f90 tx=0x7f109000b330 comp rx=0 tx=0).stop 2024-06-04T05:17:05.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.401+0000 7f109de20700 1 -- 172.21.15.70:0/926124370 shutdown_connections 2024-06-04T05:17:05.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.401+0000 7f109de20700 1 --2- 172.21.15.70:0/926124370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1098102df0 0x7f109810f2e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.401+0000 7f109de20700 1 --2- 172.21.15.70:0/926124370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1098108510 0x7f10981088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.401+0000 7f109de20700 1 --2- 172.21.15.70:0/926124370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1098102430 0x7f10981028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.401+0000 7f109de20700 1 -- 172.21.15.70:0/926124370 >> 172.21.15.70:0/926124370 conn(0x7f10980793b0 msgr2=0x7f10980797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:05.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.401+0000 7f109de20700 1 -- 172.21.15.70:0/926124370 shutdown_connections 2024-06-04T05:17:05.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.401+0000 7f109de20700 1 -- 172.21.15.70:0/926124370 wait complete. 2024-06-04T05:17:05.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.402+0000 7f109de20700 1 Processor -- start 2024-06-04T05:17:05.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.402+0000 7f109de20700 1 -- start start 2024-06-04T05:17:05.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.402+0000 7f109de20700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1098102430 0x7f10981a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.402+0000 7f109de20700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1098102df0 0x7f10981a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.402+0000 7f109de20700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1098108510 0x7f109810bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.403+0000 7f109de20700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1098111400 con 0x7f1098102430 2024-06-04T05:17:05.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.403+0000 7f109de20700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1098111280 con 0x7f1098108510 2024-06-04T05:17:05.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.403+0000 7f109de20700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1098111580 con 0x7f1098102df0 2024-06-04T05:17:05.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.403+0000 7f10977fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1098102430 0x7f10981a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.403+0000 7f10977fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1098102430 0x7f10981a4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39368/0 (socket says 172.21.15.70:39368) 2024-06-04T05:17:05.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.403+0000 7f10977fe700 1 -- 172.21.15.70:0/3596742243 learned_addr learned my addr 172.21.15.70:0/3596742243 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:05.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.403+0000 7f1096ffd700 1 --2- 172.21.15.70:0/3596742243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1098102df0 0x7f10981a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.403+0000 7f1097fff700 1 --2- 172.21.15.70:0/3596742243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1098108510 0x7f109810bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.403+0000 7f1096ffd700 1 -- 172.21.15.70:0/3596742243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1098108510 msgr2=0x7f109810bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:05.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.403+0000 7f1096ffd700 1 --2- 172.21.15.70:0/3596742243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1098108510 0x7f109810bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.403+0000 7f1096ffd700 1 -- 172.21.15.70:0/3596742243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1098102430 msgr2=0x7f10981a4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:05.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.403+0000 7f1096ffd700 1 --2- 172.21.15.70:0/3596742243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1098102430 0x7f10981a4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.403+0000 7f1096ffd700 1 -- 172.21.15.70:0/3596742243 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f109810c5c0 con 0x7f1098102df0 2024-06-04T05:17:05.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.403+0000 7f1096ffd700 1 --2- 172.21.15.70:0/3596742243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1098102df0 0x7f10981a5350 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f108c00eb90 tx=0x7f108c00eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:05.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.403+0000 7f1094ff9700 1 -- 172.21.15.70:0/3596742243 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c00cc90 con 0x7f1098102df0 2024-06-04T05:17:05.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.403+0000 7f1094ff9700 1 -- 172.21.15.70:0/3596742243 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f108c00ce30 con 0x7f1098102df0 2024-06-04T05:17:05.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.404+0000 7f109de20700 1 -- 172.21.15.70:0/3596742243 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f109810c8b0 con 0x7f1098102df0 2024-06-04T05:17:05.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.404+0000 7f109de20700 1 -- 172.21.15.70:0/3596742243 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10981acd10 con 0x7f1098102df0 2024-06-04T05:17:05.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.404+0000 7f1094ff9700 1 -- 172.21.15.70:0/3596742243 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c003e30 con 0x7f1098102df0 2024-06-04T05:17:05.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.406+0000 7f1094ff9700 1 -- 172.21.15.70:0/3596742243 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f108c004000 con 0x7f1098102df0 2024-06-04T05:17:05.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.407+0000 7f1094ff9700 1 --2- 172.21.15.70:0/3596742243 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10800b2a70 0x7f10800b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.407+0000 7f10977fe700 1 --2- 172.21.15.70:0/3596742243 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10800b2a70 0x7f10800b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.407+0000 7f1094ff9700 1 -- 172.21.15.70:0/3596742243 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8615+0+0 (secure 0 0 0) 0x7f108c0e2770 con 0x7f1098102df0 2024-06-04T05:17:05.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.408+0000 7f10977fe700 1 --2- 172.21.15.70:0/3596742243 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10800b2a70 0x7f10800b4f30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1088009ce0 tx=0x7f1088009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:05.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.408+0000 7f10a86b5b80 1 Processor -- start 2024-06-04T05:17:05.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.408+0000 7f10a86b5b80 1 -- start start 2024-06-04T05:17:05.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.409+0000 7f10a86b5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c8286700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.409+0000 7f10a86b5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c82808a0 0x5566c8280d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.409+0000 7f10a86b5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c8281260 0x5566c828d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.409+0000 7f10a86b5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5566c81ded70 con 0x5566c8286320 2024-06-04T05:17:05.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.409+0000 7f10a86b5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5566c81ce910 con 0x5566c82808a0 2024-06-04T05:17:05.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.409+0000 7f10a86b5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5566c81d2950 con 0x5566c8281260 2024-06-04T05:17:05.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.409+0000 7f10977fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c8286700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.409+0000 7f10977fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c8286700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39372/0 (socket says 172.21.15.70:39372) 2024-06-04T05:17:05.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.409+0000 7f10977fe700 1 -- 172.21.15.70:0/1624671262 learned_addr learned my addr 172.21.15.70:0/1624671262 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:05.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.409+0000 7f1096ffd700 1 --2- 172.21.15.70:0/1624671262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c82808a0 0x5566c8280d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.409+0000 7f1097fff700 1 --2- 172.21.15.70:0/1624671262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c8281260 0x5566c828d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.409+0000 7f10977fe700 1 -- 172.21.15.70:0/1624671262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c8281260 msgr2=0x5566c828d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:05.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.409+0000 7f10977fe700 1 --2- 172.21.15.70:0/1624671262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c8281260 0x5566c828d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.409+0000 7f10977fe700 1 -- 172.21.15.70:0/1624671262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c82808a0 msgr2=0x5566c8280d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:05.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.409+0000 7f10977fe700 1 --2- 172.21.15.70:0/1624671262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c82808a0 0x5566c8280d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.409+0000 7f10977fe700 1 -- 172.21.15.70:0/1624671262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566c8177fb0 con 0x5566c8286320 2024-06-04T05:17:05.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.410+0000 7f10977fe700 1 --2- 172.21.15.70:0/1624671262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c8286700 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f108800f550 tx=0x7f1088014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abecf5a2f43427a9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:05.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.410+0000 7f107dffb700 1 -- 172.21.15.70:0/1624671262 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108800fda0 con 0x5566c8286320 2024-06-04T05:17:05.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.410+0000 7f107dffb700 1 -- 172.21.15.70:0/1624671262 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f108801e8b0 con 0x5566c8286320 2024-06-04T05:17:05.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.410+0000 7f107dffb700 1 -- 172.21.15.70:0/1624671262 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10880267d0 con 0x5566c8286320 2024-06-04T05:17:05.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.411+0000 7f10a86b5b80 1 -- 172.21.15.70:0/1624671262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 msgr2=0x5566c8286700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:05.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.411+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/1624671262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c8286700 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f108800f550 tx=0x7f1088014000 comp rx=0 tx=0).stop 2024-06-04T05:17:05.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.411+0000 7f10a86b5b80 1 -- 172.21.15.70:0/1624671262 shutdown_connections 2024-06-04T05:17:05.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.411+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/1624671262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c8281260 0x5566c828d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.411+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/1624671262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c8286700 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.411+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/1624671262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c82808a0 0x5566c8280d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.411+0000 7f10a86b5b80 1 -- 172.21.15.70:0/1624671262 >> 172.21.15.70:0/1624671262 conn(0x5566c81ac240 msgr2=0x5566c827b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:05.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.411+0000 7f10a86b5b80 1 -- 172.21.15.70:0/1624671262 shutdown_connections 2024-06-04T05:17:05.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.411+0000 7f10a86b5b80 1 -- 172.21.15.70:0/1624671262 wait complete. 2024-06-04T05:17:05.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.411+0000 7f10a86b5b80 1 -- start start 2024-06-04T05:17:05.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.412+0000 7f10a86b5b80 1 Processor -- start 2024-06-04T05:17:05.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.412+0000 7f10a86b5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 0x5566c828cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.412+0000 7f10a86b5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 0x5566c8289f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.412+0000 7f10a86b5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c828a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.412+0000 7f10a86b5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5566c80d4fa0 con 0x5566c8286320 2024-06-04T05:17:05.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.412+0000 7f10a86b5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5566c8136da0 con 0x5566c8281260 2024-06-04T05:17:05.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.412+0000 7f10a86b5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5566c828d780 con 0x5566c82808a0 2024-06-04T05:17:05.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.412+0000 7f1096ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 0x5566c8289f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.412+0000 7f10a86b5b80 10 client.0 _ll_register_callbacks cb 0x5566c8169950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:05.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.412+0000 7f1097fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c828a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.412+0000 7f1096ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 0x5566c8289f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:34922/0 (socket says 172.21.15.70:34922) 2024-06-04T05:17:05.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.412+0000 7f1096ffd700 1 -- 172.21.15.70:0/156471468 learned_addr learned my addr 172.21.15.70:0/156471468 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:05.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.412+0000 7f10977fe700 1 --2- 172.21.15.70:0/156471468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 0x5566c828cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.413+0000 7f1096ffd700 1 -- 172.21.15.70:0/156471468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 msgr2=0x5566c828cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:05.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.413+0000 7f1096ffd700 1 --2- 172.21.15.70:0/156471468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 0x5566c828cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.413+0000 7f1096ffd700 1 -- 172.21.15.70:0/156471468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 msgr2=0x5566c828a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:05.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.413+0000 7f1096ffd700 1 --2- 172.21.15.70:0/156471468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c828a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.413+0000 7f1096ffd700 1 -- 172.21.15.70:0/156471468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566c811e8d0 con 0x5566c8281260 2024-06-04T05:17:05.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.413+0000 7f1096ffd700 1 --2- 172.21.15.70:0/156471468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 0x5566c8289f20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f108c005fd0 tx=0x7f108c0a5180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3ab11f879a92769 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:05.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.413+0000 7f1076ffd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:17:05.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.413+0000 7f1076ffd700 1 -- 172.21.15.70:0/156471468 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c0c3060 con 0x5566c8281260 2024-06-04T05:17:05.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.413+0000 7f10a86b5b80 1 -- 172.21.15.70:0/156471468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f108801e8b0 con 0x5566c8281260 2024-06-04T05:17:05.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.413+0000 7f1076ffd700 1 -- 172.21.15.70:0/156471468 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8615+0+0 (secure 0 0 0) 0x7f108c0cc2e0 con 0x5566c8281260 2024-06-04T05:17:05.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.413+0000 7f1076ffd700 1 -- 172.21.15.70:0/156471468 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f108c0b2190 con 0x5566c8281260 2024-06-04T05:17:05.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.413+0000 7f10a86b5b80 10 client.29354 fetch_fsmap learned FSMap version 57 2024-06-04T05:17:05.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.413+0000 7f10a86b5b80 1 -- 172.21.15.70:0/156471468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x5566c8188760 con 0x5566c8281260 2024-06-04T05:17:05.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.413+0000 7f1076ffd700 1 -- 172.21.15.70:0/156471468 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f108801e8b0 con 0x5566c8281260 2024-06-04T05:17:05.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.414+0000 7f1076ffd700 1 -- 172.21.15.70:0/156471468 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f108c0a9c50 con 0x5566c8281260 2024-06-04T05:17:05.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.414+0000 7f10a86b5b80 10 client.29354 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:17:05.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.414+0000 7f10a86b5b80 10 client.29354 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:17:05.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.414+0000 7f10a86b5b80 20 client.29354 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:05.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.414+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/156471468 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5566c829a680 0x5566c829caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.414+0000 7f10a86b5b80 4 client.29354 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:17:05.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.414+0000 7f10a86b5b80 1 -- 172.21.15.70:0/156471468 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566c81e0cc0 con 0x5566c829a680 2024-06-04T05:17:05.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.461+0000 7f10977fe700 1 --2- 172.21.15.70:0/156471468 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5566c829a680 0x5566c829caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.462+0000 7f1076ffd700 10 client.29354 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:05.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.462+0000 7f10977fe700 1 --2- 172.21.15.70:0/156471468 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5566c829a680 0x5566c829caa0 crc :-1 s=READY pgs=429 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5df2b26659a4670 server_cookie=246c3026e3ea4749 in_seq=0 out_seq=0 2024-06-04T05:17:05.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.463+0000 7f1076ffd700 1 -- 172.21.15.70:0/156471468 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5566c81e0cc0 con 0x5566c829a680 2024-06-04T05:17:05.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.463+0000 7f1076ffd700 10 client.29354 handle_command_reply: tid=0 2024-06-04T05:17:05.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.463+0000 7f10a86b5b80 1 client.29354 shutdown 2024-06-04T05:17:05.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.463+0000 7f10a86b5b80 1 -- 172.21.15.70:0/156471468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 msgr2=0x5566c8289f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:05.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.463+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/156471468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 0x5566c8289f20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f108c005fd0 tx=0x7f108c0a5180 comp rx=0 tx=0).stop 2024-06-04T05:17:05.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.463+0000 7f10a86b5b80 1 -- 172.21.15.70:0/156471468 shutdown_connections 2024-06-04T05:17:05.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.463+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/156471468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c828a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.463+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/156471468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 0x5566c828cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.463+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/156471468 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5566c829a680 0x5566c829caa0 crc :-1 s=READY pgs=429 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.463+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/156471468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 0x5566c8289f20 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.463+0000 7f10a86b5b80 1 -- 172.21.15.70:0/156471468 >> 172.21.15.70:0/156471468 conn(0x5566c81ac240 msgr2=0x5566c827aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:05.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.464+0000 7f1076ffd700 0 client.29354 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:05.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.464+0000 7f10a86b5b80 1 -- 172.21.15.70:0/156471468 shutdown_connections 2024-06-04T05:17:05.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.464+0000 7f10a86b5b80 1 -- 172.21.15.70:0/156471468 wait complete. 2024-06-04T05:17:05.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.464+0000 7f10a86b5b80 20 client.29354 trim_cache size 0 max 16384 2024-06-04T05:17:05.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.503+0000 7f10a86b5b80 1 Processor -- start 2024-06-04T05:17:05.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.503+0000 7f10a86b5b80 1 -- start start 2024-06-04T05:17:05.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.503+0000 7f10a86b5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 0x5566c82858e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.503+0000 7f10a86b5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 0x5566c8284460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.506 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:05 smithi070 ceph-mon[32631]: pgmap v625: 97 pgs: 97 active+clean; 722 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 246 op/s 2024-06-04T05:17:05.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.503+0000 7f10a86b5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c827b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.503+0000 7f10a86b5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f108400d770 con 0x5566c8286320 2024-06-04T05:17:05.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.503+0000 7f10a86b5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f108400f2b0 con 0x5566c8281260 2024-06-04T05:17:05.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.503+0000 7f10a86b5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f108400de40 con 0x5566c82808a0 2024-06-04T05:17:05.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.503+0000 7f10977fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 0x5566c82858e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.503+0000 7f1097fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c827b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.503+0000 7f10977fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 0x5566c82858e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:50790/0 (socket says 172.21.15.70:50790) 2024-06-04T05:17:05.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.504+0000 7f10977fe700 1 -- 172.21.15.70:0/2511698256 learned_addr learned my addr 172.21.15.70:0/2511698256 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:05.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.504+0000 7f1096ffd700 1 --2- 172.21.15.70:0/2511698256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 0x5566c8284460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.504+0000 7f1097fff700 1 -- 172.21.15.70:0/2511698256 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 msgr2=0x5566c82858e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:05.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.504+0000 7f1097fff700 1 --2- 172.21.15.70:0/2511698256 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 0x5566c82858e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.504+0000 7f1097fff700 1 -- 172.21.15.70:0/2511698256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 msgr2=0x5566c8284460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:05.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.504+0000 7f1097fff700 1 --2- 172.21.15.70:0/2511698256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 0x5566c8284460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.504+0000 7f1097fff700 1 -- 172.21.15.70:0/2511698256 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566c827d370 con 0x5566c8286320 2024-06-04T05:17:05.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.504+0000 7f1096ffd700 1 --2- 172.21.15.70:0/2511698256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 0x5566c8284460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:05.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.504+0000 7f1097fff700 1 --2- 172.21.15.70:0/2511698256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c827b270 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f10981a5a60 tx=0x7f1090003a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8551969ec9aaf99c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:05.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.504+0000 7f107d7fa700 1 -- 172.21.15.70:0/2511698256 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1090003d70 con 0x5566c8286320 2024-06-04T05:17:05.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.505+0000 7f107d7fa700 1 -- 172.21.15.70:0/2511698256 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1090007420 con 0x5566c8286320 2024-06-04T05:17:05.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.505+0000 7f107d7fa700 1 -- 172.21.15.70:0/2511698256 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10900266e0 con 0x5566c8286320 2024-06-04T05:17:05.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.505+0000 7f10a86b5b80 1 -- 172.21.15.70:0/2511698256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 msgr2=0x5566c827b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:05.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.505+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/2511698256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c827b270 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f10981a5a60 tx=0x7f1090003a00 comp rx=0 tx=0).stop 2024-06-04T05:17:05.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.506+0000 7f10a86b5b80 1 -- 172.21.15.70:0/2511698256 shutdown_connections 2024-06-04T05:17:05.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.506+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/2511698256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c827b270 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.506+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/2511698256 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 0x5566c82858e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.506+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/2511698256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 0x5566c8284460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.506+0000 7f10a86b5b80 1 -- 172.21.15.70:0/2511698256 >> 172.21.15.70:0/2511698256 conn(0x5566c81ac240 msgr2=0x5566c8283d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:05.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.506+0000 7f10a86b5b80 1 -- 172.21.15.70:0/2511698256 shutdown_connections 2024-06-04T05:17:05.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.506+0000 7f10a86b5b80 1 -- 172.21.15.70:0/2511698256 wait complete. 2024-06-04T05:17:05.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.506+0000 7f10a86b5b80 1 -- start start 2024-06-04T05:17:05.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.506+0000 7f10a86b5b80 1 Processor -- start 2024-06-04T05:17:05.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f10a86b5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 0x5566c8292a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f10a86b5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 0x5566c8292fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f10a86b5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c82a1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f10a86b5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f108400ebe0 con 0x5566c8286320 2024-06-04T05:17:05.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f10a86b5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f108400e510 con 0x5566c8281260 2024-06-04T05:17:05.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f10a86b5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f108400b730 con 0x5566c82808a0 2024-06-04T05:17:05.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f1096ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 0x5566c8292fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f10977fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 0x5566c8292a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f10977fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 0x5566c8292a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:50804/0 (socket says 172.21.15.70:50804) 2024-06-04T05:17:05.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f1096ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 0x5566c8292fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:34946/0 (socket says 172.21.15.70:34946) 2024-06-04T05:17:05.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f10977fe700 1 -- 172.21.15.70:0/3157600700 learned_addr learned my addr 172.21.15.70:0/3157600700 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:05.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f1097fff700 1 --2- 172.21.15.70:0/3157600700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c82a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f10a86b5b80 10 client.0 _ll_register_callbacks cb 0x5566c8169950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:05.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f10977fe700 1 -- 172.21.15.70:0/3157600700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 msgr2=0x5566c8292fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:05.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f10977fe700 1 --2- 172.21.15.70:0/3157600700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 0x5566c8292fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f10977fe700 1 -- 172.21.15.70:0/3157600700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 msgr2=0x5566c82a1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:05.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f10977fe700 1 --2- 172.21.15.70:0/3157600700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c82a1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f10977fe700 1 -- 172.21.15.70:0/3157600700 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566c82a18c0 con 0x5566c82808a0 2024-06-04T05:17:05.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.507+0000 7f1097fff700 1 --2- 172.21.15.70:0/3157600700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c82a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:05.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.508+0000 7f10977fe700 1 --2- 172.21.15.70:0/3157600700 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 0x5566c8292a60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1088025000 tx=0x7f1088005cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bab78648e2c6f8d1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:05.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.508+0000 7f1076ffd700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:17:05.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.508+0000 7f1076ffd700 1 -- 172.21.15.70:0/3157600700 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1088005250 con 0x5566c82808a0 2024-06-04T05:17:05.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.508+0000 7f10a86b5b80 1 -- 172.21.15.70:0/3157600700 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1090007420 con 0x5566c82808a0 2024-06-04T05:17:05.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.508+0000 7f1076ffd700 1 -- 172.21.15.70:0/3157600700 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8615+0+0 (secure 0 0 0) 0x7f10880053f0 con 0x5566c82808a0 2024-06-04T05:17:05.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.508+0000 7f1076ffd700 1 -- 172.21.15.70:0/3157600700 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f1088042050 con 0x5566c82808a0 2024-06-04T05:17:05.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.508+0000 7f10a86b5b80 10 client.29512 fetch_fsmap learned FSMap version 57 2024-06-04T05:17:05.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.508+0000 7f10a86b5b80 1 -- 172.21.15.70:0/3157600700 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x5566c8304930 con 0x5566c82808a0 2024-06-04T05:17:05.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.508+0000 7f1076ffd700 1 -- 172.21.15.70:0/3157600700 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1090007420 con 0x5566c82808a0 2024-06-04T05:17:05.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.509+0000 7f1076ffd700 1 -- 172.21.15.70:0/3157600700 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7f108802f040 con 0x5566c82808a0 2024-06-04T05:17:05.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.509+0000 7f10a86b5b80 10 client.29512 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:17:05.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.509+0000 7f10a86b5b80 10 client.29512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:17:05.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.509+0000 7f10a86b5b80 20 client.29512 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:05.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.509+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/3157600700 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5566c8304df0 0x5566c8307210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:05.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.509+0000 7f10a86b5b80 4 client.29512 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:17:05.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.509+0000 7f10a86b5b80 1 -- 172.21.15.70:0/3157600700 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5566c81da4a0 con 0x5566c8304df0 2024-06-04T05:17:05.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.509+0000 7f1096ffd700 1 --2- 172.21.15.70:0/3157600700 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5566c8304df0 0x5566c8307210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:05.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.509+0000 7f1076ffd700 10 client.29512 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:05.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.510+0000 7f1096ffd700 1 --2- 172.21.15.70:0/3157600700 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5566c8304df0 0x5566c8307210 crc :-1 s=READY pgs=430 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bb3384af79e6dcf server_cookie=27d14374180287ac in_seq=0 out_seq=0 2024-06-04T05:17:05.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.511+0000 7f1076ffd700 1 -- 172.21.15.70:0/3157600700 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5566c81da4a0 con 0x5566c8304df0 2024-06-04T05:17:05.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.511+0000 7f1076ffd700 10 client.29512 handle_command_reply: tid=0 2024-06-04T05:17:05.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.511+0000 7f10a86b5b80 1 client.29512 shutdown 2024-06-04T05:17:05.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.511+0000 7f10a86b5b80 1 -- 172.21.15.70:0/3157600700 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 msgr2=0x5566c8292a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:05.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.511+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/3157600700 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 0x5566c8292a60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1088025000 tx=0x7f1088005cb0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.511+0000 7f10a86b5b80 1 -- 172.21.15.70:0/3157600700 shutdown_connections 2024-06-04T05:17:05.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.511+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/3157600700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566c8286320 0x5566c82a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.511+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/3157600700 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566c82808a0 0x5566c8292a60 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.511+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/3157600700 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5566c8304df0 0x5566c8307210 crc :-1 s=READY pgs=430 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.511+0000 7f10a86b5b80 1 --2- 172.21.15.70:0/3157600700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566c8281260 0x5566c8292fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.511+0000 7f10a86b5b80 1 -- 172.21.15.70:0/3157600700 >> 172.21.15.70:0/3157600700 conn(0x5566c81ac240 msgr2=0x5566c8284460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:05.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.511+0000 7f1076ffd700 0 client.29512 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:05.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.511+0000 7f10a86b5b80 1 -- 172.21.15.70:0/3157600700 shutdown_connections 2024-06-04T05:17:05.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.511+0000 7f10a86b5b80 1 -- 172.21.15.70:0/3157600700 wait complete. 2024-06-04T05:17:05.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.511+0000 7f10a86b5b80 20 client.29512 trim_cache size 0 max 16384 2024-06-04T05:17:05.516 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:17:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:17:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:17:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: "174d61ee-a27b-41aa-9eab-89f28c49482c": { 2024-06-04T05:17:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:17:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "174d61ee-a27b-41aa-9eab-89f28c49482c", 2024-06-04T05:17:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:17:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:17:05.517 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:17:05.517 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:17:05.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.512+0000 7f109de20700 1 -- 172.21.15.70:0/3596742243 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10800b2a70 msgr2=0x7f10800b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:05.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.512+0000 7f109de20700 1 --2- 172.21.15.70:0/3596742243 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10800b2a70 0x7f10800b4f30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1088009ce0 tx=0x7f1088009260 comp rx=0 tx=0).stop 2024-06-04T05:17:05.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.512+0000 7f109de20700 1 -- 172.21.15.70:0/3596742243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1098102df0 msgr2=0x7f10981a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:05.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.512+0000 7f109de20700 1 --2- 172.21.15.70:0/3596742243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1098102df0 0x7f10981a5350 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f108c00eb90 tx=0x7f108c00eea0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.512+0000 7f109de20700 1 -- 172.21.15.70:0/3596742243 shutdown_connections 2024-06-04T05:17:05.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.512+0000 7f109de20700 1 --2- 172.21.15.70:0/3596742243 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f10800b2a70 0x7f10800b4f30 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.512+0000 7f109de20700 1 --2- 172.21.15.70:0/3596742243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1098108510 0x7f109810bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.512+0000 7f109de20700 1 --2- 172.21.15.70:0/3596742243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1098102430 0x7f10981a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.512+0000 7f109de20700 1 --2- 172.21.15.70:0/3596742243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1098102df0 0x7f10981a5350 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:05.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.512+0000 7f109de20700 1 -- 172.21.15.70:0/3596742243 >> 172.21.15.70:0/3596742243 conn(0x7f10980793b0 msgr2=0x7f1098105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:05.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.512+0000 7f109de20700 1 -- 172.21.15.70:0/3596742243 shutdown_connections 2024-06-04T05:17:05.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:05.512+0000 7f109de20700 1 -- 172.21.15.70:0/3596742243 wait complete. 2024-06-04T05:17:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:05 smithi112 ceph-mon[36453]: pgmap v625: 97 pgs: 97 active+clean; 722 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 246 op/s 2024-06-04T05:17:05.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:05 smithi098 ceph-mon[35569]: pgmap v625: 97 pgs: 97 active+clean; 722 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 246 op/s 2024-06-04T05:17:05.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 45154 6.52 MB/sec execute 237 sec latency 105.273 ms 2024-06-04T05:17:06.417 INFO:tasks.cephfs.filesystem:scrub status for tag:174d61ee-a27b-41aa-9eab-89f28c49482c - {'path': '/', 'tag': '174d61ee-a27b-41aa-9eab-89f28c49482c', 'options': 'recursive,force'} 2024-06-04T05:17:06.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 45325 6.50 MB/sec execute 238 sec latency 56.521 ms 2024-06-04T05:17:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:07 smithi112 ceph-mon[36453]: pgmap v626: 97 pgs: 97 active+clean; 715 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.9 MiB/s wr, 371 op/s 2024-06-04T05:17:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:07 smithi098 ceph-mon[35569]: pgmap v626: 97 pgs: 97 active+clean; 715 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.9 MiB/s wr, 371 op/s 2024-06-04T05:17:07.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 45489 6.49 MB/sec execute 239 sec latency 52.058 ms 2024-06-04T05:17:07.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:07 smithi070 ceph-mon[32631]: pgmap v626: 97 pgs: 97 active+clean; 715 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.9 MiB/s wr, 371 op/s 2024-06-04T05:17:08.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 45675 6.51 MB/sec execute 240 sec latency 56.633 ms 2024-06-04T05:17:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:08 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:17:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:08 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:17:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:08 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:17:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:08 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:17:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:08 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:17:09.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:08 smithi098 ceph-mon[35569]: osdmap e108: 12 total, 12 up, 12 in 2024-06-04T05:17:09.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:08 smithi098 ceph-mon[35569]: pgmap v628: 97 pgs: 97 active+clean; 715 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.1 MiB/s wr, 464 op/s 2024-06-04T05:17:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:08 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:17:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:08 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:17:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:08 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:17:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:08 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:17:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:08 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:17:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:08 smithi070 ceph-mon[32631]: osdmap e108: 12 total, 12 up, 12 in 2024-06-04T05:17:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:08 smithi070 ceph-mon[32631]: pgmap v628: 97 pgs: 97 active+clean; 715 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.1 MiB/s wr, 464 op/s 2024-06-04T05:17:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:08 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:17:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:08 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:17:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:08 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:17:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:08 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:17:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:08 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:17:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:08 smithi112 ceph-mon[36453]: osdmap e108: 12 total, 12 up, 12 in 2024-06-04T05:17:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:08 smithi112 ceph-mon[36453]: pgmap v628: 97 pgs: 97 active+clean; 715 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.1 MiB/s wr, 464 op/s 2024-06-04T05:17:09.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 45892 6.54 MB/sec execute 241 sec latency 51.224 ms 2024-06-04T05:17:10.821 INFO:tasks.workunit.client.0.smithi070.stdout: 10 46085 6.58 MB/sec execute 242 sec latency 159.419 ms 2024-06-04T05:17:11.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:11 smithi070 ceph-mon[32631]: pgmap v629: 97 pgs: 97 active+clean; 715 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 202 op/s 2024-06-04T05:17:11.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:11 smithi112 ceph-mon[36453]: pgmap v629: 97 pgs: 97 active+clean; 715 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 202 op/s 2024-06-04T05:17:11.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:11 smithi098 ceph-mon[35569]: pgmap v629: 97 pgs: 97 active+clean; 715 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 202 op/s 2024-06-04T05:17:11.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 46313 6.60 MB/sec execute 243 sec latency 56.585 ms 2024-06-04T05:17:12.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 46488 6.63 MB/sec execute 244 sec latency 51.382 ms 2024-06-04T05:17:13.547 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:13 smithi098 ceph-mon[35569]: pgmap v630: 97 pgs: 97 active+clean; 735 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.3 MiB/s wr, 378 op/s 2024-06-04T05:17:13.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:13 smithi070 ceph-mon[32631]: pgmap v630: 97 pgs: 97 active+clean; 735 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.3 MiB/s wr, 378 op/s 2024-06-04T05:17:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:13 smithi112 ceph-mon[36453]: pgmap v630: 97 pgs: 97 active+clean; 735 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.3 MiB/s wr, 378 op/s 2024-06-04T05:17:13.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 46661 6.63 MB/sec execute 245 sec latency 54.965 ms 2024-06-04T05:17:14.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.201+0000 7efe90933640 1 -- 172.21.15.70:0/2527762221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe8c10d020 msgr2=0x7efe8c10f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:14.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.201+0000 7efe90933640 1 --2- 172.21.15.70:0/2527762221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe8c10d020 0x7efe8c10f4c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efe84009be0 tx=0x7efe8402a7e0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.202+0000 7efe90933640 1 -- 172.21.15.70:0/2527762221 shutdown_connections 2024-06-04T05:17:14.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.202+0000 7efe90933640 1 --2- 172.21.15.70:0/2527762221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe8c10d020 0x7efe8c10f4c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.202+0000 7efe90933640 1 --2- 172.21.15.70:0/2527762221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe8c104600 0x7efe8c10cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.202+0000 7efe90933640 1 --2- 172.21.15.70:0/2527762221 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efe8c103ce0 0x7efe8c1040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.202+0000 7efe90933640 1 -- 172.21.15.70:0/2527762221 >> 172.21.15.70:0/2527762221 conn(0x7efe8c078ad0 msgr2=0x7efe8c078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:14.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.202+0000 7efe90933640 1 -- 172.21.15.70:0/2527762221 shutdown_connections 2024-06-04T05:17:14.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.202+0000 7efe90933640 1 -- 172.21.15.70:0/2527762221 wait complete. 2024-06-04T05:17:14.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.202+0000 7efe90933640 1 Processor -- start 2024-06-04T05:17:14.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.203+0000 7efe90933640 1 -- start start 2024-06-04T05:17:14.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.203+0000 7efe90933640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efe8c103ce0 0x7efe8c1990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.203+0000 7efe90933640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe8c104600 0x7efe8c199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.203+0000 7efe90933640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe8c10d020 0x7efe8c19d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.203+0000 7efe90933640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c112b80 con 0x7efe8c10d020 2024-06-04T05:17:14.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.203+0000 7efe90933640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c112a00 con 0x7efe8c104600 2024-06-04T05:17:14.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.203+0000 7efe90933640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c112d00 con 0x7efe8c103ce0 2024-06-04T05:17:14.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.203+0000 7efe8a59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efe8c103ce0 0x7efe8c1990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.203+0000 7efe8ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe8c10d020 0x7efe8c19d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.203+0000 7efe89d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe8c104600 0x7efe8c199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.203+0000 7efe8a59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efe8c103ce0 0x7efe8c1990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39110/0 (socket says 172.21.15.70:39110) 2024-06-04T05:17:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.203+0000 7efe8a59c640 1 -- 172.21.15.70:0/4123012143 learned_addr learned my addr 172.21.15.70:0/4123012143 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.203+0000 7efe89d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe8c104600 0x7efe8c199620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55686/0 (socket says 172.21.15.70:55686) 2024-06-04T05:17:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.203+0000 7efe8ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe8c10d020 0x7efe8c19d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56906/0 (socket says 172.21.15.70:56906) 2024-06-04T05:17:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.204+0000 7efe89d9b640 1 -- 172.21.15.70:0/4123012143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efe8c103ce0 msgr2=0x7efe8c1990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.204+0000 7efe89d9b640 1 --2- 172.21.15.70:0/4123012143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efe8c103ce0 0x7efe8c1990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.204+0000 7efe89d9b640 1 -- 172.21.15.70:0/4123012143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe8c10d020 msgr2=0x7efe8c19d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.204+0000 7efe89d9b640 1 --2- 172.21.15.70:0/4123012143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe8c10d020 0x7efe8c19d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.204+0000 7efe89d9b640 1 -- 172.21.15.70:0/4123012143 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe8c19e080 con 0x7efe8c104600 2024-06-04T05:17:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.204+0000 7efe8a59c640 1 --2- 172.21.15.70:0/4123012143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efe8c103ce0 0x7efe8c1990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.204+0000 7efe8ad9d640 1 --2- 172.21.15.70:0/4123012143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe8c10d020 0x7efe8c19d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:14.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.204+0000 7efe89d9b640 1 --2- 172.21.15.70:0/4123012143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe8c104600 0x7efe8c199620 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efe8000e9f0 tx=0x7efe8000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:14.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.204+0000 7efe7b7fe640 1 -- 172.21.15.70:0/4123012143 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8000cd80 con 0x7efe8c104600 2024-06-04T05:17:14.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.204+0000 7efe7b7fe640 1 -- 172.21.15.70:0/4123012143 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efe800035f0 con 0x7efe8c104600 2024-06-04T05:17:14.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.204+0000 7efe7b7fe640 1 -- 172.21.15.70:0/4123012143 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe80010670 con 0x7efe8c104600 2024-06-04T05:17:14.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.204+0000 7efe90933640 1 -- 172.21.15.70:0/4123012143 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe8c19e370 con 0x7efe8c104600 2024-06-04T05:17:14.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.204+0000 7efe90933640 1 -- 172.21.15.70:0/4123012143 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe8c1a5c00 con 0x7efe8c104600 2024-06-04T05:17:14.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.207+0000 7efe7b7fe640 1 -- 172.21.15.70:0/4123012143 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7efe80004000 con 0x7efe8c104600 2024-06-04T05:17:14.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.207+0000 7efe9364b740 1 Processor -- start 2024-06-04T05:17:14.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.207+0000 7efe9364b740 1 -- start start 2024-06-04T05:17:14.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.208+0000 7efe9364b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc57820 0x5626dbc57c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.208+0000 7efe9364b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc58140 0x5626dbc624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.208+0000 7efe9364b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc62a00 0x5626dbc64df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.208+0000 7efe9364b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5626dbbc9c30 con 0x5626dbc58140 2024-06-04T05:17:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.208+0000 7efe9364b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5626dbbb97d0 con 0x5626dbc62a00 2024-06-04T05:17:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.208+0000 7efe9364b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5626dbbbd810 con 0x5626dbc57820 2024-06-04T05:17:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.208+0000 7efe8ad9d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc57820 0x5626dbc57c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.208+0000 7efe8ad9d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc57820 0x5626dbc57c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39112/0 (socket says 172.21.15.70:39112) 2024-06-04T05:17:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.208+0000 7efe8ad9d640 1 -- 172.21.15.70:0/922071104 learned_addr learned my addr 172.21.15.70:0/922071104 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.208+0000 7efe89d9b640 1 --2- 172.21.15.70:0/922071104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc62a00 0x5626dbc64df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.208+0000 7efe8a59c640 1 --2- 172.21.15.70:0/922071104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc58140 0x5626dbc624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.208+0000 7efe8ad9d640 1 -- 172.21.15.70:0/922071104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc62a00 msgr2=0x5626dbc64df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.208+0000 7efe8ad9d640 1 --2- 172.21.15.70:0/922071104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc62a00 0x5626dbc64df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.208+0000 7efe8ad9d640 1 -- 172.21.15.70:0/922071104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc58140 msgr2=0x5626dbc624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.208+0000 7efe7b7fe640 1 --2- 172.21.15.70:0/4123012143 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efe740b2a90 0x7efe740b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.209+0000 7efe8ad9d640 1 --2- 172.21.15.70:0/922071104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc58140 0x5626dbc624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.209+0000 7efe8ad9d640 1 -- 172.21.15.70:0/922071104 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626db8ce210 con 0x5626dbc57820 2024-06-04T05:17:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.209+0000 7efe89d9b640 1 --2- 172.21.15.70:0/4123012143 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efe740b2a90 0x7efe740b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.209+0000 7efe8a59c640 1 --2- 172.21.15.70:0/922071104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc58140 0x5626dbc624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.209+0000 7efe7b7fe640 1 -- 172.21.15.70:0/4123012143 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7efe800e2840 con 0x7efe8c104600 2024-06-04T05:17:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.209+0000 7efe8ad9d640 1 --2- 172.21.15.70:0/922071104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc57820 0x5626dbc57c00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efe84009be0 tx=0x7efe84005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3e3c3e86d1a14f4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.209+0000 7efe89d9b640 1 --2- 172.21.15.70:0/4123012143 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efe740b2a90 0x7efe740b4f50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7efe800e0010 tx=0x7efe800e8550 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.209+0000 7efe78ff9640 1 -- 172.21.15.70:0/922071104 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8404f070 con 0x5626dbc57820 2024-06-04T05:17:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.209+0000 7efe78ff9640 1 -- 172.21.15.70:0/922071104 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efe84037040 con 0x5626dbc57820 2024-06-04T05:17:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.209+0000 7efe78ff9640 1 -- 172.21.15.70:0/922071104 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe84036040 con 0x5626dbc57820 2024-06-04T05:17:14.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.210+0000 7efe9364b740 1 -- 172.21.15.70:0/922071104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc57820 msgr2=0x5626dbc57c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.210+0000 7efe9364b740 1 --2- 172.21.15.70:0/922071104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc57820 0x5626dbc57c00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efe84009be0 tx=0x7efe84005eb0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.210+0000 7efe9364b740 1 -- 172.21.15.70:0/922071104 shutdown_connections 2024-06-04T05:17:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.210+0000 7efe9364b740 1 --2- 172.21.15.70:0/922071104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc62a00 0x5626dbc64df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.210+0000 7efe9364b740 1 --2- 172.21.15.70:0/922071104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc58140 0x5626dbc624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.210+0000 7efe9364b740 1 --2- 172.21.15.70:0/922071104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc57820 0x5626dbc57c00 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.210+0000 7efe9364b740 1 -- 172.21.15.70:0/922071104 >> 172.21.15.70:0/922071104 conn(0x5626dbc53d30 msgr2=0x5626dbc54110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.210+0000 7efe9364b740 1 -- 172.21.15.70:0/922071104 shutdown_connections 2024-06-04T05:17:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.210+0000 7efe9364b740 1 -- 172.21.15.70:0/922071104 wait complete. 2024-06-04T05:17:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.210+0000 7efe9364b740 1 -- start start 2024-06-04T05:17:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe9364b740 1 Processor -- start 2024-06-04T05:17:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe9364b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc57820 0x5626dbc5ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe9364b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc58140 0x5626dbc5bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe9364b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc62a00 0x5626dbc5c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe9364b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5626db89f790 con 0x5626dbc62a00 2024-06-04T05:17:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe9364b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5626dbb75890 con 0x5626dbc58140 2024-06-04T05:17:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe9364b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5626dbc65510 con 0x5626dbc57820 2024-06-04T05:17:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe8ad9d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc57820 0x5626dbc5ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe8ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc62a00 0x5626dbc5c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe9364b740 10 client.0 _ll_register_callbacks cb 0x5626db8cda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe8ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc62a00 0x5626dbc5c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56928/0 (socket says 172.21.15.70:56928) 2024-06-04T05:17:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe8ad9d640 1 -- 172.21.15.70:0/350260830 learned_addr learned my addr 172.21.15.70:0/350260830 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe8a59c640 1 --2- 172.21.15.70:0/350260830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc58140 0x5626dbc5bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe8ad9d640 1 -- 172.21.15.70:0/350260830 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc57820 msgr2=0x5626dbc5ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe8ad9d640 1 --2- 172.21.15.70:0/350260830 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc57820 0x5626dbc5ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe8ad9d640 1 -- 172.21.15.70:0/350260830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc58140 msgr2=0x5626dbc5bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe8ad9d640 1 --2- 172.21.15.70:0/350260830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc58140 0x5626dbc5bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe8ad9d640 1 -- 172.21.15.70:0/350260830 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626db99a620 con 0x5626dbc62a00 2024-06-04T05:17:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe8a59c640 1 --2- 172.21.15.70:0/350260830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc58140 0x5626dbc5bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.211+0000 7efe8ad9d640 1 --2- 172.21.15.70:0/350260830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc62a00 0x5626dbc5c4b0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7efe8402ad00 tx=0x7efe84030f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d35fd7652550c8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.212+0000 7efe71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:17:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.212+0000 7efe71ffb640 1 -- 172.21.15.70:0/350260830 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8405e060 con 0x5626dbc62a00 2024-06-04T05:17:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.212+0000 7efe71ffb640 1 -- 172.21.15.70:0/350260830 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7efe840042c0 con 0x5626dbc62a00 2024-06-04T05:17:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.212+0000 7efe9364b740 1 -- 172.21.15.70:0/350260830 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe84037040 con 0x5626dbc62a00 2024-06-04T05:17:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.212+0000 7efe71ffb640 1 -- 172.21.15.70:0/350260830 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7efe8404f070 con 0x5626dbc62a00 2024-06-04T05:17:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.212+0000 7efe9364b740 10 client.20658 fetch_fsmap learned FSMap version 57 2024-06-04T05:17:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.212+0000 7efe9364b740 1 -- 172.21.15.70:0/350260830 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x5626dbc71390 con 0x5626dbc62a00 2024-06-04T05:17:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.213+0000 7efe71ffb640 1 -- 172.21.15.70:0/350260830 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efe84037040 con 0x5626dbc62a00 2024-06-04T05:17:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.213+0000 7efe71ffb640 1 -- 172.21.15.70:0/350260830 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7efe84049030 con 0x5626dbc62a00 2024-06-04T05:17:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.213+0000 7efe9364b740 10 client.20658 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:17:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.213+0000 7efe9364b740 10 client.20658 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:17:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.213+0000 7efe9364b740 20 client.20658 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.213+0000 7efe9364b740 1 --2- 172.21.15.70:0/350260830 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5626dbc726e0 0x5626dbc74ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.213+0000 7efe9364b740 4 client.20658 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:17:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.213+0000 7efe9364b740 1 -- 172.21.15.70:0/350260830 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626dbbcbb80 con 0x5626dbc726e0 2024-06-04T05:17:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.213+0000 7efe8a59c640 1 --2- 172.21.15.70:0/350260830 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5626dbc726e0 0x5626dbc74ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.213+0000 7efe71ffb640 10 client.20658 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.214+0000 7efe8a59c640 1 --2- 172.21.15.70:0/350260830 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5626dbc726e0 0x5626dbc74ad0 crc :-1 s=READY pgs=431 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a70438a0f0357a0c server_cookie=906ddfb3793de414 in_seq=0 out_seq=0 2024-06-04T05:17:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.214+0000 7efe71ffb640 1 -- 172.21.15.70:0/350260830 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5626dbbcbb80 con 0x5626dbc726e0 2024-06-04T05:17:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.215+0000 7efe71ffb640 10 client.20658 handle_command_reply: tid=0 2024-06-04T05:17:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.215+0000 7efe9364b740 1 client.20658 shutdown 2024-06-04T05:17:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.215+0000 7efe9364b740 1 -- 172.21.15.70:0/350260830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc62a00 msgr2=0x5626dbc5c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.215+0000 7efe9364b740 1 --2- 172.21.15.70:0/350260830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc62a00 0x5626dbc5c4b0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7efe8402ad00 tx=0x7efe84030f50 comp rx=0 tx=0).stop 2024-06-04T05:17:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.215+0000 7efe9364b740 1 -- 172.21.15.70:0/350260830 shutdown_connections 2024-06-04T05:17:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.215+0000 7efe9364b740 1 --2- 172.21.15.70:0/350260830 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5626dbc726e0 0x5626dbc74ad0 crc :-1 s=READY pgs=431 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.215+0000 7efe9364b740 1 --2- 172.21.15.70:0/350260830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc58140 0x5626dbc5bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.215+0000 7efe9364b740 1 --2- 172.21.15.70:0/350260830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc62a00 0x5626dbc5c4b0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.215+0000 7efe9364b740 1 --2- 172.21.15.70:0/350260830 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc57820 0x5626dbc5ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.215+0000 7efe71ffb640 0 client.20658 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:14.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.215+0000 7efe9364b740 1 -- 172.21.15.70:0/350260830 >> 172.21.15.70:0/350260830 conn(0x5626dbc53d30 msgr2=0x5626dbc52fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:14.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.215+0000 7efe9364b740 1 -- 172.21.15.70:0/350260830 shutdown_connections 2024-06-04T05:17:14.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.215+0000 7efe9364b740 1 -- 172.21.15.70:0/350260830 wait complete. 2024-06-04T05:17:14.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.215+0000 7efe9364b740 20 client.20658 trim_cache size 0 max 16384 2024-06-04T05:17:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.241+0000 7efe9364b740 1 Processor -- start 2024-06-04T05:17:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.241+0000 7efe9364b740 1 -- start start 2024-06-04T05:17:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.241+0000 7efe9364b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc57820 0x5626dbc67f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.241+0000 7efe9364b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc58140 0x5626dbc684b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.241+0000 7efe9364b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc62a00 0x5626dbc6e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.241+0000 7efe9364b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c1ac540 con 0x5626dbc57820 2024-06-04T05:17:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.241+0000 7efe9364b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c1ae080 con 0x5626dbc62a00 2024-06-04T05:17:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.241+0000 7efe9364b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c1acc10 con 0x5626dbc58140 2024-06-04T05:17:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.242+0000 7efe8ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc57820 0x5626dbc67f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.242+0000 7efe8ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc57820 0x5626dbc67f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56934/0 (socket says 172.21.15.70:56934) 2024-06-04T05:17:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.242+0000 7efe8ad9d640 1 -- 172.21.15.70:0/1130485118 learned_addr learned my addr 172.21.15.70:0/1130485118 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.242+0000 7efe8a59c640 1 --2- 172.21.15.70:0/1130485118 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc58140 0x5626dbc684b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.243+0000 7efe8a59c640 1 -- 172.21.15.70:0/1130485118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc62a00 msgr2=0x5626dbc6e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.243+0000 7efe8a59c640 1 --2- 172.21.15.70:0/1130485118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc62a00 0x5626dbc6e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.243+0000 7efe8a59c640 1 -- 172.21.15.70:0/1130485118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc57820 msgr2=0x5626dbc67f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.243+0000 7efe8a59c640 1 --2- 172.21.15.70:0/1130485118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc57820 0x5626dbc67f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.243+0000 7efe8a59c640 1 -- 172.21.15.70:0/1130485118 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626dbc588f0 con 0x5626dbc58140 2024-06-04T05:17:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.243+0000 7efe8a59c640 1 --2- 172.21.15.70:0/1130485118 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc58140 0x5626dbc684b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x5626dbbbd8a0 tx=0x7efe7c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57f8a0f4283d0503 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.244+0000 7efe73fff640 1 -- 172.21.15.70:0/1130485118 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c003ab0 con 0x5626dbc58140 2024-06-04T05:17:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.244+0000 7efe73fff640 1 -- 172.21.15.70:0/1130485118 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efe7c003c50 con 0x5626dbc58140 2024-06-04T05:17:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.244+0000 7efe73fff640 1 -- 172.21.15.70:0/1130485118 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c0263e0 con 0x5626dbc58140 2024-06-04T05:17:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.245+0000 7efe9364b740 1 -- 172.21.15.70:0/1130485118 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc58140 msgr2=0x5626dbc684b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.245+0000 7efe9364b740 1 --2- 172.21.15.70:0/1130485118 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc58140 0x5626dbc684b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x5626dbbbd8a0 tx=0x7efe7c009260 comp rx=0 tx=0).stop 2024-06-04T05:17:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.245+0000 7efe9364b740 1 -- 172.21.15.70:0/1130485118 shutdown_connections 2024-06-04T05:17:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.245+0000 7efe9364b740 1 --2- 172.21.15.70:0/1130485118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc62a00 0x5626dbc6e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.245+0000 7efe9364b740 1 --2- 172.21.15.70:0/1130485118 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc58140 0x5626dbc684b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.245+0000 7efe9364b740 1 --2- 172.21.15.70:0/1130485118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc57820 0x5626dbc67f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.245+0000 7efe9364b740 1 -- 172.21.15.70:0/1130485118 >> 172.21.15.70:0/1130485118 conn(0x5626dbc53d30 msgr2=0x5626dbc5fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.245+0000 7efe9364b740 1 -- 172.21.15.70:0/1130485118 shutdown_connections 2024-06-04T05:17:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.245+0000 7efe9364b740 1 -- 172.21.15.70:0/1130485118 wait complete. 2024-06-04T05:17:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.245+0000 7efe9364b740 1 -- start start 2024-06-04T05:17:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.246+0000 7efe9364b740 1 Processor -- start 2024-06-04T05:17:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.246+0000 7efe9364b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc57820 0x5626dbc6c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.246+0000 7efe9364b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc58140 0x5626dbc6cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.246+0000 7efe9364b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc62a00 0x5626dbc6d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.246+0000 7efe9364b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c1ad9b0 con 0x5626dbc58140 2024-06-04T05:17:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.246+0000 7efe9364b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c1ad2e0 con 0x5626dbc57820 2024-06-04T05:17:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.246+0000 7efe9364b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c1aa540 con 0x5626dbc62a00 2024-06-04T05:17:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.246+0000 7efe9364b740 10 client.0 _ll_register_callbacks cb 0x7efe8404d070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.246+0000 7efe8ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc57820 0x5626dbc6c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.246+0000 7efe8ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc57820 0x5626dbc6c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55710/0 (socket says 172.21.15.70:55710) 2024-06-04T05:17:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.246+0000 7efe8ad9d640 1 -- 172.21.15.70:0/2672770287 learned_addr learned my addr 172.21.15.70:0/2672770287 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.246+0000 7efe8a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc58140 0x5626dbc6cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.246+0000 7efe8ad9d640 1 --2- 172.21.15.70:0/2672770287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc62a00 0x5626dbc6d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.247+0000 7efe8ad9d640 1 -- 172.21.15.70:0/2672770287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc62a00 msgr2=0x5626dbc6d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.247+0000 7efe8ad9d640 1 --2- 172.21.15.70:0/2672770287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc62a00 0x5626dbc6d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.247+0000 7efe8ad9d640 1 -- 172.21.15.70:0/2672770287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc58140 msgr2=0x5626dbc6cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.247+0000 7efe8ad9d640 1 --2- 172.21.15.70:0/2672770287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc58140 0x5626dbc6cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.247+0000 7efe8ad9d640 1 -- 172.21.15.70:0/2672770287 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626dbcecf60 con 0x5626dbc57820 2024-06-04T05:17:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.247+0000 7efe8ad9d640 1 --2- 172.21.15.70:0/2672770287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc57820 0x5626dbc6c7c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efe84004a80 tx=0x7efe84033040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a29392e282d510ed server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.247+0000 7efe71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:17:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.247+0000 7efe71ffb640 1 -- 172.21.15.70:0/2672770287 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8405e040 con 0x5626dbc57820 2024-06-04T05:17:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.247+0000 7efe71ffb640 1 -- 172.21.15.70:0/2672770287 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7efe84030c20 con 0x5626dbc57820 2024-06-04T05:17:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.247+0000 7efe9364b740 1 -- 172.21.15.70:0/2672770287 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe7c003c50 con 0x5626dbc57820 2024-06-04T05:17:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.248+0000 7efe71ffb640 1 -- 172.21.15.70:0/2672770287 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7efe84036040 con 0x5626dbc57820 2024-06-04T05:17:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.248+0000 7efe9364b740 10 client.29375 fetch_fsmap learned FSMap version 57 2024-06-04T05:17:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.248+0000 7efe9364b740 1 -- 172.21.15.70:0/2672770287 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x5626dbcee250 con 0x5626dbc57820 2024-06-04T05:17:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.248+0000 7efe71ffb640 1 -- 172.21.15.70:0/2672770287 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efe7c003c50 con 0x5626dbc57820 2024-06-04T05:17:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.248+0000 7efe71ffb640 1 -- 172.21.15.70:0/2672770287 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7efe84049070 con 0x5626dbc57820 2024-06-04T05:17:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.248+0000 7efe9364b740 10 client.29375 fetch_fsmap finished waiting for FSMap version 57 2024-06-04T05:17:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.248+0000 7efe9364b740 10 client.29375 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:17:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.248+0000 7efe9364b740 20 client.29375 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.248+0000 7efe9364b740 1 --2- 172.21.15.70:0/2672770287 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5626dbcee880 0x5626dbcf0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.248+0000 7efe9364b740 4 client.29375 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:17:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.248+0000 7efe9364b740 1 -- 172.21.15.70:0/2672770287 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5626dbbc5360 con 0x5626dbcee880 2024-06-04T05:17:14.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.248+0000 7efe8a59c640 1 --2- 172.21.15.70:0/2672770287 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5626dbcee880 0x5626dbcf0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:14.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.249+0000 7efe71ffb640 10 client.29375 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:14.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.249+0000 7efe8a59c640 1 --2- 172.21.15.70:0/2672770287 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5626dbcee880 0x5626dbcf0ca0 crc :-1 s=READY pgs=432 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8259c2ca1ca1fda4 server_cookie=b8643e0218b7bce8 in_seq=0 out_seq=0 2024-06-04T05:17:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.368+0000 7efe71ffb640 1 -- 172.21.15.70:0/2672770287 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5626dbbc5360 con 0x5626dbcee880 2024-06-04T05:17:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.368+0000 7efe71ffb640 10 client.29375 handle_command_reply: tid=0 2024-06-04T05:17:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.368+0000 7efe9364b740 1 client.29375 shutdown 2024-06-04T05:17:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.368+0000 7efe9364b740 1 -- 172.21.15.70:0/2672770287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc57820 msgr2=0x5626dbc6c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.368+0000 7efe9364b740 1 --2- 172.21.15.70:0/2672770287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc57820 0x5626dbc6c7c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efe84004a80 tx=0x7efe84033040 comp rx=0 tx=0).stop 2024-06-04T05:17:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.368+0000 7efe9364b740 1 -- 172.21.15.70:0/2672770287 shutdown_connections 2024-06-04T05:17:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.368+0000 7efe9364b740 1 --2- 172.21.15.70:0/2672770287 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5626dbcee880 0x5626dbcf0ca0 crc :-1 s=READY pgs=432 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.368+0000 7efe9364b740 1 --2- 172.21.15.70:0/2672770287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5626dbc62a00 0x5626dbc6d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.368+0000 7efe9364b740 1 --2- 172.21.15.70:0/2672770287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626dbc58140 0x5626dbc6cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.368+0000 7efe71ffb640 0 client.29375 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.368+0000 7efe9364b740 1 --2- 172.21.15.70:0/2672770287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626dbc57820 0x5626dbc6c7c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.369+0000 7efe9364b740 1 -- 172.21.15.70:0/2672770287 >> 172.21.15.70:0/2672770287 conn(0x5626dbc53d30 msgr2=0x5626dbc591d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.369+0000 7efe9364b740 1 -- 172.21.15.70:0/2672770287 shutdown_connections 2024-06-04T05:17:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.369+0000 7efe9364b740 1 -- 172.21.15.70:0/2672770287 wait complete. 2024-06-04T05:17:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.369+0000 7efe9364b740 20 client.29375 trim_cache size 0 max 16384 2024-06-04T05:17:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:17:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:17:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:17:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.370+0000 7efe90933640 1 -- 172.21.15.70:0/4123012143 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efe740b2a90 msgr2=0x7efe740b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.370+0000 7efe90933640 1 --2- 172.21.15.70:0/4123012143 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efe740b2a90 0x7efe740b4f50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7efe800e0010 tx=0x7efe800e8550 comp rx=0 tx=0).stop 2024-06-04T05:17:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.370+0000 7efe90933640 1 -- 172.21.15.70:0/4123012143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe8c104600 msgr2=0x7efe8c199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.370+0000 7efe90933640 1 --2- 172.21.15.70:0/4123012143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe8c104600 0x7efe8c199620 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efe8000e9f0 tx=0x7efe8000eeb0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.370+0000 7efe90933640 1 -- 172.21.15.70:0/4123012143 shutdown_connections 2024-06-04T05:17:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.370+0000 7efe90933640 1 --2- 172.21.15.70:0/4123012143 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efe740b2a90 0x7efe740b4f50 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.370+0000 7efe90933640 1 --2- 172.21.15.70:0/4123012143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe8c104600 0x7efe8c199620 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.370+0000 7efe90933640 1 --2- 172.21.15.70:0/4123012143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe8c10d020 0x7efe8c19d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.370+0000 7efe90933640 1 --2- 172.21.15.70:0/4123012143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efe8c103ce0 0x7efe8c1990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.370+0000 7efe90933640 1 -- 172.21.15.70:0/4123012143 >> 172.21.15.70:0/4123012143 conn(0x7efe8c078ad0 msgr2=0x7efe8c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.370+0000 7efe90933640 1 -- 172.21.15.70:0/4123012143 shutdown_connections 2024-06-04T05:17:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:14.370+0000 7efe90933640 1 -- 172.21.15.70:0/4123012143 wait complete. 2024-06-04T05:17:14.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 46834 6.66 MB/sec execute 246 sec latency 122.125 ms 2024-06-04T05:17:15.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:15 smithi070 ceph-mon[32631]: pgmap v631: 97 pgs: 97 active+clean; 735 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.6 MiB/s wr, 335 op/s 2024-06-04T05:17:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:15 smithi112 ceph-mon[36453]: pgmap v631: 97 pgs: 97 active+clean; 735 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.6 MiB/s wr, 335 op/s 2024-06-04T05:17:15.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:15 smithi098 ceph-mon[35569]: pgmap v631: 97 pgs: 97 active+clean; 735 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.6 MiB/s wr, 335 op/s 2024-06-04T05:17:15.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 46989 6.65 MB/sec execute 247 sec latency 80.967 ms 2024-06-04T05:17:16.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:16 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:17:16.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:16 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:17:16.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:16 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:17:16.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 47193 6.68 MB/sec execute 248 sec latency 152.997 ms 2024-06-04T05:17:17.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:17 smithi070 ceph-mon[32631]: pgmap v632: 97 pgs: 97 active+clean; 746 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.3 MiB/s wr, 357 op/s 2024-06-04T05:17:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:17 smithi112 ceph-mon[36453]: pgmap v632: 97 pgs: 97 active+clean; 746 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.3 MiB/s wr, 357 op/s 2024-06-04T05:17:17.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:17 smithi098 ceph-mon[35569]: pgmap v632: 97 pgs: 97 active+clean; 746 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.3 MiB/s wr, 357 op/s 2024-06-04T05:17:17.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 47384 6.68 MB/sec execute 249 sec latency 54.047 ms 2024-06-04T05:17:18.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 47545 6.69 MB/sec execute 250 sec latency 63.624 ms 2024-06-04T05:17:19.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:19 smithi070 ceph-mon[32631]: pgmap v633: 97 pgs: 97 active+clean; 746 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.1 MiB/s wr, 348 op/s 2024-06-04T05:17:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:19 smithi112 ceph-mon[36453]: pgmap v633: 97 pgs: 97 active+clean; 746 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.1 MiB/s wr, 348 op/s 2024-06-04T05:17:19.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:19 smithi098 ceph-mon[35569]: pgmap v633: 97 pgs: 97 active+clean; 746 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.1 MiB/s wr, 348 op/s 2024-06-04T05:17:19.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 47708 6.68 MB/sec execute 251 sec latency 45.961 ms 2024-06-04T05:17:20.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 47916 6.71 MB/sec execute 252 sec latency 63.618 ms 2024-06-04T05:17:21.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:21 smithi070 ceph-mon[32631]: pgmap v634: 97 pgs: 97 active+clean; 746 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.2 MiB/s wr, 297 op/s 2024-06-04T05:17:21.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:21 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:17:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:21 smithi112 ceph-mon[36453]: pgmap v634: 97 pgs: 97 active+clean; 746 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.2 MiB/s wr, 297 op/s 2024-06-04T05:17:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:21 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:17:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:21 smithi098 ceph-mon[35569]: pgmap v634: 97 pgs: 97 active+clean; 746 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.2 MiB/s wr, 297 op/s 2024-06-04T05:17:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:21 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:17:21.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 48128 6.73 MB/sec execute 253 sec latency 55.688 ms 2024-06-04T05:17:22.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 48338 6.75 MB/sec execute 254 sec latency 58.636 ms 2024-06-04T05:17:23.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:23 smithi070 ceph-mon[32631]: pgmap v635: 97 pgs: 97 active+clean; 755 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.7 MiB/s wr, 432 op/s 2024-06-04T05:17:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:23 smithi112 ceph-mon[36453]: pgmap v635: 97 pgs: 97 active+clean; 755 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.7 MiB/s wr, 432 op/s 2024-06-04T05:17:23.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:23 smithi098 ceph-mon[35569]: pgmap v635: 97 pgs: 97 active+clean; 755 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.7 MiB/s wr, 432 op/s 2024-06-04T05:17:23.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 48499 6.75 MB/sec execute 255 sec latency 159.225 ms 2024-06-04T05:17:24.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 48662 6.74 MB/sec execute 256 sec latency 75.892 ms 2024-06-04T05:17:25.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:25 smithi070 ceph-mon[32631]: pgmap v636: 97 pgs: 97 active+clean; 755 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.0 MiB/s wr, 287 op/s 2024-06-04T05:17:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:25 smithi112 ceph-mon[36453]: pgmap v636: 97 pgs: 97 active+clean; 755 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.0 MiB/s wr, 287 op/s 2024-06-04T05:17:25.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:25 smithi098 ceph-mon[35569]: pgmap v636: 97 pgs: 97 active+clean; 755 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.0 MiB/s wr, 287 op/s 2024-06-04T05:17:25.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 48871 6.77 MB/sec execute 257 sec latency 87.140 ms 2024-06-04T05:17:26.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 49073 6.78 MB/sec execute 258 sec latency 60.492 ms 2024-06-04T05:17:27.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.498+0000 7f68c8319640 1 -- 172.21.15.70:0/4244574055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68c0107940 msgr2=0x7f68c0107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:27.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.498+0000 7f68c8319640 1 --2- 172.21.15.70:0/4244574055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68c0107940 0x7f68c0107d20 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f68bc009930 tx=0x7f68bc02a420 comp rx=0 tx=0).stop 2024-06-04T05:17:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.498+0000 7f68c8319640 1 -- 172.21.15.70:0/4244574055 shutdown_connections 2024-06-04T05:17:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.498+0000 7f68c8319640 1 --2- 172.21.15.70:0/4244574055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68c0102340 0x7f68c010e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.498+0000 7f68c8319640 1 --2- 172.21.15.70:0/4244574055 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68c0101980 0x7f68c0101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.498+0000 7f68c8319640 1 --2- 172.21.15.70:0/4244574055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68c0107940 0x7f68c0107d20 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.498+0000 7f68c8319640 1 -- 172.21.15.70:0/4244574055 >> 172.21.15.70:0/4244574055 conn(0x7f68c0078ad0 msgr2=0x7f68c0078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.498+0000 7f68c8319640 1 -- 172.21.15.70:0/4244574055 shutdown_connections 2024-06-04T05:17:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.498+0000 7f68c8319640 1 -- 172.21.15.70:0/4244574055 wait complete. 2024-06-04T05:17:27.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.499+0000 7f68c8319640 1 Processor -- start 2024-06-04T05:17:27.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.499+0000 7f68c8319640 1 -- start start 2024-06-04T05:17:27.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.499+0000 7f68c8319640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68c0101980 0x7f68c01990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.499+0000 7f68c8319640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68c0102340 0x7f68c01995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.499+0000 7f68c8319640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68c0107940 0x7f68c019d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.499+0000 7f68c8319640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f68c0111a80 con 0x7f68c0101980 2024-06-04T05:17:27.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.499+0000 7f68c8319640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68c0111900 con 0x7f68c0107940 2024-06-04T05:17:27.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.499+0000 7f68c8319640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f68c0111c00 con 0x7f68c0102340 2024-06-04T05:17:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.499+0000 7f68c60b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68c0101980 0x7f68c01990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.499+0000 7f68c60b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68c0101980 0x7f68c01990b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59504/0 (socket says 172.21.15.70:59504) 2024-06-04T05:17:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.499+0000 7f68c60b5640 1 -- 172.21.15.70:0/2977555652 learned_addr learned my addr 172.21.15.70:0/2977555652 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.499+0000 7f68c58b4640 1 --2- 172.21.15.70:0/2977555652 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68c0102340 0x7f68c01995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.499+0000 7f68c68b6640 1 --2- 172.21.15.70:0/2977555652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68c0107940 0x7f68c019d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.500+0000 7f68c58b4640 1 -- 172.21.15.70:0/2977555652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68c0107940 msgr2=0x7f68c019d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.500+0000 7f68c58b4640 1 --2- 172.21.15.70:0/2977555652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68c0107940 0x7f68c019d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.500+0000 7f68c58b4640 1 -- 172.21.15.70:0/2977555652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68c0101980 msgr2=0x7f68c01990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.500+0000 7f68c58b4640 1 --2- 172.21.15.70:0/2977555652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68c0101980 0x7f68c01990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.500+0000 7f68c58b4640 1 -- 172.21.15.70:0/2977555652 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68c019e0a0 con 0x7f68c0102340 2024-06-04T05:17:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.500+0000 7f68c68b6640 1 --2- 172.21.15.70:0/2977555652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68c0107940 0x7f68c019d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.500+0000 7f68c58b4640 1 --2- 172.21.15.70:0/2977555652 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68c0102340 0x7f68c01995f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f68b000e9f0 tx=0x7f68b000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.500+0000 7f68b77fe640 1 -- 172.21.15.70:0/2977555652 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b000ccc0 con 0x7f68c0102340 2024-06-04T05:17:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.500+0000 7f68c8319640 1 -- 172.21.15.70:0/2977555652 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68c019e390 con 0x7f68c0102340 2024-06-04T05:17:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.500+0000 7f68b77fe640 1 -- 172.21.15.70:0/2977555652 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68b00108c0 con 0x7f68c0102340 2024-06-04T05:17:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.500+0000 7f68b77fe640 1 -- 172.21.15.70:0/2977555652 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b00187c0 con 0x7f68c0102340 2024-06-04T05:17:27.505 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.500+0000 7f68c8319640 1 -- 172.21.15.70:0/2977555652 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68c01a5c20 con 0x7f68c0102340 2024-06-04T05:17:27.506 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.504+0000 7f68b77fe640 1 -- 172.21.15.70:0/2977555652 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f68b0010430 con 0x7f68c0102340 2024-06-04T05:17:27.506 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.504+0000 7f68cb038740 1 Processor -- start 2024-06-04T05:17:27.506 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.504+0000 7f68cb038740 1 -- start start 2024-06-04T05:17:27.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.504+0000 7f68cb038740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd250700 0x556dcd250ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.504+0000 7f68cb038740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd251020 0x556dcd25f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.504+0000 7f68cb038740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd25fe10 0x556dcd262200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.504+0000 7f68cb038740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556dcd1c2c30 con 0x556dcd250700 2024-06-04T05:17:27.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.504+0000 7f68cb038740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556dcd1b27d0 con 0x556dcd251020 2024-06-04T05:17:27.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.504+0000 7f68cb038740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556dcd1b6810 con 0x556dcd25fe10 2024-06-04T05:17:27.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.505+0000 7f68c68b6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd250700 0x556dcd250ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.505+0000 7f68c58b4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd25fe10 0x556dcd262200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.505+0000 7f68c68b6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd250700 0x556dcd250ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59518/0 (socket says 172.21.15.70:59518) 2024-06-04T05:17:27.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.505+0000 7f68c68b6640 1 -- 172.21.15.70:0/3271219702 learned_addr learned my addr 172.21.15.70:0/3271219702 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:27.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.505+0000 7f68c60b5640 1 --2- 172.21.15.70:0/3271219702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd251020 0x556dcd25f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.505+0000 7f68c68b6640 1 -- 172.21.15.70:0/3271219702 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd25fe10 msgr2=0x556dcd262200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:27.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.505+0000 7f68c68b6640 1 --2- 172.21.15.70:0/3271219702 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd25fe10 0x556dcd262200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.505+0000 7f68c68b6640 1 -- 172.21.15.70:0/3271219702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd251020 msgr2=0x556dcd25f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:27.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.505+0000 7f68c68b6640 1 --2- 172.21.15.70:0/3271219702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd251020 0x556dcd25f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.505+0000 7f68c68b6640 1 -- 172.21.15.70:0/3271219702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dccec7210 con 0x556dcd250700 2024-06-04T05:17:27.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.505+0000 7f68c60b5640 1 --2- 172.21.15.70:0/3271219702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd251020 0x556dcd25f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:17:27.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.505+0000 7f68c68b6640 1 --2- 172.21.15.70:0/3271219702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd250700 0x556dcd250ae0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f68b80117c0 tx=0x7f68b8011c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a8eb187122cfb6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:27.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.505+0000 7f68b77fe640 1 --2- 172.21.15.70:0/2977555652 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68ac0b2a10 0x7f68ac0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.505+0000 7f68b4bfe640 1 -- 172.21.15.70:0/3271219702 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b8003c30 con 0x556dcd250700 2024-06-04T05:17:27.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.505+0000 7f68b4bfe640 1 -- 172.21.15.70:0/3271219702 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68b800fe30 con 0x556dcd250700 2024-06-04T05:17:27.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.506+0000 7f68c60b5640 1 --2- 172.21.15.70:0/2977555652 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68ac0b2a10 0x7f68ac0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.506+0000 7f68b4bfe640 1 -- 172.21.15.70:0/3271219702 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b80193e0 con 0x556dcd250700 2024-06-04T05:17:27.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.506+0000 7f68b77fe640 1 -- 172.21.15.70:0/2977555652 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7f68b00e28f0 con 0x7f68c0102340 2024-06-04T05:17:27.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.507+0000 7f68cb038740 1 -- 172.21.15.70:0/3271219702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd250700 msgr2=0x556dcd250ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:27.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.507+0000 7f68c60b5640 1 --2- 172.21.15.70:0/2977555652 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68ac0b2a10 0x7f68ac0b4ed0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f68bc0096c0 tx=0x7f68bc0058d0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:27.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.507+0000 7f68cb038740 1 --2- 172.21.15.70:0/3271219702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd250700 0x556dcd250ae0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f68b80117c0 tx=0x7f68b8011c80 comp rx=0 tx=0).stop 2024-06-04T05:17:27.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.507+0000 7f68cb038740 1 -- 172.21.15.70:0/3271219702 shutdown_connections 2024-06-04T05:17:27.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.507+0000 7f68cb038740 1 --2- 172.21.15.70:0/3271219702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd251020 0x556dcd25f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.507+0000 7f68cb038740 1 --2- 172.21.15.70:0/3271219702 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd25fe10 0x556dcd262200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.507+0000 7f68cb038740 1 --2- 172.21.15.70:0/3271219702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd250700 0x556dcd250ae0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.507+0000 7f68cb038740 1 -- 172.21.15.70:0/3271219702 >> 172.21.15.70:0/3271219702 conn(0x556dcd24cd30 msgr2=0x556dcd24d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:27.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.507+0000 7f68cb038740 1 -- 172.21.15.70:0/3271219702 shutdown_connections 2024-06-04T05:17:27.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.507+0000 7f68cb038740 1 -- 172.21.15.70:0/3271219702 wait complete. 2024-06-04T05:17:27.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.507+0000 7f68cb038740 1 -- start start 2024-06-04T05:17:27.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.507+0000 7f68cb038740 1 Processor -- start 2024-06-04T05:17:27.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68cb038740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd250700 0x556dcd25cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68cb038740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd251020 0x556dcd25d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68cb038740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd25fe10 0x556dcd25d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68cb038740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556dcce98790 con 0x556dcd250700 2024-06-04T05:17:27.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68cb038740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556dcd16e890 con 0x556dcd251020 2024-06-04T05:17:27.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68cb038740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556dcd262a40 con 0x556dcd25fe10 2024-06-04T05:17:27.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68cb038740 10 client.0 _ll_register_callbacks cb 0x556dccec6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68c60b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd250700 0x556dcd25cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68c60b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd250700 0x556dcd25cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59534/0 (socket says 172.21.15.70:59534) 2024-06-04T05:17:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68c60b5640 1 -- 172.21.15.70:0/1360719970 learned_addr learned my addr 172.21.15.70:0/1360719970 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68c58b4640 1 --2- 172.21.15.70:0/1360719970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd251020 0x556dcd25d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68c68b6640 1 --2- 172.21.15.70:0/1360719970 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd25fe10 0x556dcd25d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68c58b4640 1 -- 172.21.15.70:0/1360719970 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd25fe10 msgr2=0x556dcd25d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68c58b4640 1 --2- 172.21.15.70:0/1360719970 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd25fe10 0x556dcd25d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68c58b4640 1 -- 172.21.15.70:0/1360719970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd250700 msgr2=0x556dcd25cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68c58b4640 1 --2- 172.21.15.70:0/1360719970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd250700 0x556dcd25cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68c58b4640 1 -- 172.21.15.70:0/1360719970 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dccf93620 con 0x556dcd251020 2024-06-04T05:17:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.508+0000 7f68c60b5640 1 --2- 172.21.15.70:0/1360719970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd250700 0x556dcd25cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.509+0000 7f68c58b4640 1 --2- 172.21.15.70:0/1360719970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd251020 0x556dcd25d2d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f68b00e0010 tx=0x7f68b00a9ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=516e51f8fdd7771a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.509+0000 7f68a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:17:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.509+0000 7f68a9ffb640 1 -- 172.21.15.70:0/1360719970 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b0091050 con 0x556dcd251020 2024-06-04T05:17:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.509+0000 7f68a9ffb640 1 -- 172.21.15.70:0/1360719970 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7f68b00a8eb0 con 0x556dcd251020 2024-06-04T05:17:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.509+0000 7f68cb038740 1 -- 172.21.15.70:0/1360719970 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68b800fe30 con 0x556dcd251020 2024-06-04T05:17:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.509+0000 7f68a9ffb640 1 -- 172.21.15.70:0/1360719970 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f68b00a93c0 con 0x556dcd251020 2024-06-04T05:17:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.509+0000 7f68cb038740 10 client.29387 fetch_fsmap learned FSMap version 59 2024-06-04T05:17:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.509+0000 7f68cb038740 1 -- 172.21.15.70:0/1360719970 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x556dcd26e7f0 con 0x556dcd251020 2024-06-04T05:17:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.509+0000 7f68a9ffb640 1 -- 172.21.15.70:0/1360719970 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68b800fe30 con 0x556dcd251020 2024-06-04T05:17:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.510+0000 7f68a9ffb640 1 -- 172.21.15.70:0/1360719970 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f68b00a8920 con 0x556dcd251020 2024-06-04T05:17:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.510+0000 7f68cb038740 10 client.29387 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:17:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.510+0000 7f68cb038740 10 client.29387 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:17:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.510+0000 7f68cb038740 20 client.29387 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.510+0000 7f68cb038740 1 --2- 172.21.15.70:0/1360719970 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556dcd26fc60 0x556dcd2720c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.510+0000 7f68cb038740 4 client.29387 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:17:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.510+0000 7f68cb038740 1 -- 172.21.15.70:0/1360719970 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556dcd1c4b80 con 0x556dcd26fc60 2024-06-04T05:17:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.510+0000 7f68c60b5640 1 --2- 172.21.15.70:0/1360719970 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556dcd26fc60 0x556dcd2720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.510+0000 7f68a9ffb640 10 client.29387 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.510+0000 7f68c60b5640 1 --2- 172.21.15.70:0/1360719970 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556dcd26fc60 0x556dcd2720c0 crc :-1 s=READY pgs=433 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2296397e5ed8bf32 server_cookie=884261e75b4bd915 in_seq=0 out_seq=0 2024-06-04T05:17:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.511+0000 7f68a9ffb640 1 -- 172.21.15.70:0/1360719970 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556dcd1c4b80 con 0x556dcd26fc60 2024-06-04T05:17:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.511+0000 7f68a9ffb640 10 client.29387 handle_command_reply: tid=0 2024-06-04T05:17:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.511+0000 7f68cb038740 1 client.29387 shutdown 2024-06-04T05:17:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.511+0000 7f68cb038740 1 -- 172.21.15.70:0/1360719970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd251020 msgr2=0x556dcd25d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.511+0000 7f68cb038740 1 --2- 172.21.15.70:0/1360719970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd251020 0x556dcd25d2d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f68b00e0010 tx=0x7f68b00a9ae0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.511+0000 7f68cb038740 1 -- 172.21.15.70:0/1360719970 shutdown_connections 2024-06-04T05:17:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.511+0000 7f68cb038740 1 --2- 172.21.15.70:0/1360719970 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556dcd26fc60 0x556dcd2720c0 crc :-1 s=READY pgs=433 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.512+0000 7f68cb038740 1 --2- 172.21.15.70:0/1360719970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd251020 0x556dcd25d2d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.512+0000 7f68cb038740 1 --2- 172.21.15.70:0/1360719970 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd25fe10 0x556dcd25d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.512+0000 7f68cb038740 1 --2- 172.21.15.70:0/1360719970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd250700 0x556dcd25cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.512+0000 7f68a9ffb640 0 client.29387 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.512+0000 7f68cb038740 1 -- 172.21.15.70:0/1360719970 >> 172.21.15.70:0/1360719970 conn(0x556dcd24cd30 msgr2=0x556dcd252b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.512+0000 7f68cb038740 1 -- 172.21.15.70:0/1360719970 shutdown_connections 2024-06-04T05:17:27.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.512+0000 7f68cb038740 1 -- 172.21.15.70:0/1360719970 wait complete. 2024-06-04T05:17:27.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.512+0000 7f68cb038740 20 client.29387 trim_cache size 0 max 16384 2024-06-04T05:17:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.535+0000 7f68cb038740 1 Processor -- start 2024-06-04T05:17:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.535+0000 7f68cb038740 1 -- start start 2024-06-04T05:17:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.535+0000 7f68cb038740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd250700 0x556dcd25d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.535+0000 7f68cb038740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd251020 0x556dcd277990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.535+0000 7f68cb038740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd25fe10 0x556dcd277ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.535+0000 7f68cb038740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f68c01ac4c0 con 0x556dcd251020 2024-06-04T05:17:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.535+0000 7f68cb038740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68c01ae060 con 0x556dcd250700 2024-06-04T05:17:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.536+0000 7f68cb038740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f68c01acbf0 con 0x556dcd25fe10 2024-06-04T05:17:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.536+0000 7f68c58b4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd251020 0x556dcd277990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.536+0000 7f68c60b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd250700 0x556dcd25d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.536+0000 7f68c58b4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd251020 0x556dcd277990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59550/0 (socket says 172.21.15.70:59550) 2024-06-04T05:17:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.536+0000 7f68c58b4640 1 -- 172.21.15.70:0/3421930636 learned_addr learned my addr 172.21.15.70:0/3421930636 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.536+0000 7f68c68b6640 1 --2- 172.21.15.70:0/3421930636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd25fe10 0x556dcd277ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.536+0000 7f68c58b4640 1 -- 172.21.15.70:0/3421930636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd25fe10 msgr2=0x556dcd277ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.536+0000 7f68c58b4640 1 --2- 172.21.15.70:0/3421930636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd25fe10 0x556dcd277ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.536+0000 7f68c58b4640 1 -- 172.21.15.70:0/3421930636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd250700 msgr2=0x556dcd25d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.536+0000 7f68c58b4640 1 --2- 172.21.15.70:0/3421930636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd250700 0x556dcd25d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.536+0000 7f68c58b4640 1 -- 172.21.15.70:0/3421930636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dcd2486f0 con 0x556dcd251020 2024-06-04T05:17:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.536+0000 7f68c60b5640 1 --2- 172.21.15.70:0/3421930636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd250700 0x556dcd25d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.536+0000 7f68c58b4640 1 --2- 172.21.15.70:0/3421930636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd251020 0x556dcd277990 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f68b000e9c0 tx=0x7f68b00a9b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b25fed4aa64c133 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.536+0000 7f68abfff640 1 -- 172.21.15.70:0/3421930636 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b0091050 con 0x556dcd251020 2024-06-04T05:17:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.536+0000 7f68abfff640 1 -- 172.21.15.70:0/3421930636 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68b00cae50 con 0x556dcd251020 2024-06-04T05:17:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.536+0000 7f68abfff640 1 -- 172.21.15.70:0/3421930636 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b00a53e0 con 0x556dcd251020 2024-06-04T05:17:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.537+0000 7f68cb038740 1 -- 172.21.15.70:0/3421930636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd251020 msgr2=0x556dcd277990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.537+0000 7f68cb038740 1 --2- 172.21.15.70:0/3421930636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd251020 0x556dcd277990 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f68b000e9c0 tx=0x7f68b00a9b50 comp rx=0 tx=0).stop 2024-06-04T05:17:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.538+0000 7f68cb038740 1 -- 172.21.15.70:0/3421930636 shutdown_connections 2024-06-04T05:17:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.538+0000 7f68cb038740 1 --2- 172.21.15.70:0/3421930636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd25fe10 0x556dcd277ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.538+0000 7f68cb038740 1 --2- 172.21.15.70:0/3421930636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd251020 0x556dcd277990 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.538+0000 7f68cb038740 1 --2- 172.21.15.70:0/3421930636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd250700 0x556dcd25d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.538+0000 7f68cb038740 1 -- 172.21.15.70:0/3421930636 >> 172.21.15.70:0/3421930636 conn(0x556dcd24cd30 msgr2=0x556dcd25cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.538+0000 7f68cb038740 1 -- 172.21.15.70:0/3421930636 shutdown_connections 2024-06-04T05:17:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.538+0000 7f68cb038740 1 -- 172.21.15.70:0/3421930636 wait complete. 2024-06-04T05:17:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.538+0000 7f68cb038740 1 -- start start 2024-06-04T05:17:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.538+0000 7f68cb038740 1 Processor -- start 2024-06-04T05:17:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68cb038740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd250700 0x556dcd263b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68cb038740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd251020 0x556dcd2640b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68cb038740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd25fe10 0x556dcd2645f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68cb038740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f68c01ad990 con 0x556dcd25fe10 2024-06-04T05:17:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68cb038740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68c01ad2c0 con 0x556dcd250700 2024-06-04T05:17:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68cb038740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f68c01aa4c0 con 0x556dcd251020 2024-06-04T05:17:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68c68b6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd25fe10 0x556dcd2645f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68c68b6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd25fe10 0x556dcd2645f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59560/0 (socket says 172.21.15.70:59560) 2024-06-04T05:17:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68c60b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd250700 0x556dcd263b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68c68b6640 1 -- 172.21.15.70:0/3204588364 learned_addr learned my addr 172.21.15.70:0/3204588364 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68cb038740 10 client.0 _ll_register_callbacks cb 0x7f68b00e8240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:27.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68c58b4640 1 --2- 172.21.15.70:0/3204588364 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd251020 0x556dcd2640b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68c60b5640 1 -- 172.21.15.70:0/3204588364 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd251020 msgr2=0x556dcd2640b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68c60b5640 1 --2- 172.21.15.70:0/3204588364 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd251020 0x556dcd2640b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68c60b5640 1 -- 172.21.15.70:0/3204588364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd25fe10 msgr2=0x556dcd2645f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68c60b5640 1 --2- 172.21.15.70:0/3204588364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd25fe10 0x556dcd2645f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68c60b5640 1 -- 172.21.15.70:0/3204588364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dcd2d98b0 con 0x556dcd250700 2024-06-04T05:17:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68c60b5640 1 --2- 172.21.15.70:0/3204588364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd250700 0x556dcd263b70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x556dcd270040 tx=0x7f68bc009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36eb01d3a1de7d44 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.539+0000 7f68a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:17:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.540+0000 7f68a9ffb640 1 -- 172.21.15.70:0/3204588364 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68bc035070 con 0x556dcd250700 2024-06-04T05:17:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.540+0000 7f68a9ffb640 1 -- 172.21.15.70:0/3204588364 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7f68bc003ac0 con 0x556dcd250700 2024-06-04T05:17:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.540+0000 7f68cb038740 1 -- 172.21.15.70:0/3204588364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68b00cae50 con 0x556dcd250700 2024-06-04T05:17:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.540+0000 7f68a9ffb640 1 -- 172.21.15.70:0/3204588364 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f68bc003cc0 con 0x556dcd250700 2024-06-04T05:17:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.540+0000 7f68cb038740 10 client.29393 fetch_fsmap learned FSMap version 59 2024-06-04T05:17:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.540+0000 7f68cb038740 1 -- 172.21.15.70:0/3204588364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x556dcd2dabe0 con 0x556dcd250700 2024-06-04T05:17:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.540+0000 7f68a9ffb640 1 -- 172.21.15.70:0/3204588364 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68b00cae50 con 0x556dcd250700 2024-06-04T05:17:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.540+0000 7f68a9ffb640 1 -- 172.21.15.70:0/3204588364 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f68bc043040 con 0x556dcd250700 2024-06-04T05:17:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.540+0000 7f68cb038740 10 client.29393 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:17:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.540+0000 7f68cb038740 10 client.29393 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:17:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.540+0000 7f68cb038740 20 client.29393 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.540+0000 7f68cb038740 1 --2- 172.21.15.70:0/3204588364 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556dcd2db360 0x556dcd2dd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.540+0000 7f68cb038740 4 client.29393 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:17:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.540+0000 7f68cb038740 1 -- 172.21.15.70:0/3204588364 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556dcd1be360 con 0x556dcd2db360 2024-06-04T05:17:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.541+0000 7f68c58b4640 1 --2- 172.21.15.70:0/3204588364 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556dcd2db360 0x556dcd2dd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.541+0000 7f68a9ffb640 10 client.29393 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.541+0000 7f68c58b4640 1 --2- 172.21.15.70:0/3204588364 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556dcd2db360 0x556dcd2dd780 crc :-1 s=READY pgs=434 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=843ee380e89ad7d6 server_cookie=afa55e19e4f742e0 in_seq=0 out_seq=0 2024-06-04T05:17:27.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:27 smithi070 ceph-mon[32631]: pgmap v637: 97 pgs: 97 active+clean; 761 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.1 MiB/s wr, 427 op/s 2024-06-04T05:17:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.679+0000 7f68a9ffb640 1 -- 172.21.15.70:0/3204588364 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x556dcd1be360 con 0x556dcd2db360 2024-06-04T05:17:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.679+0000 7f68a9ffb640 10 client.29393 handle_command_reply: tid=0 2024-06-04T05:17:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.679+0000 7f68cb038740 1 client.29393 shutdown 2024-06-04T05:17:27.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.679+0000 7f68cb038740 1 -- 172.21.15.70:0/3204588364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd250700 msgr2=0x556dcd263b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:27.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.679+0000 7f68cb038740 1 --2- 172.21.15.70:0/3204588364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd250700 0x556dcd263b70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x556dcd270040 tx=0x7f68bc009260 comp rx=0 tx=0).stop 2024-06-04T05:17:27.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.680+0000 7f68cb038740 1 -- 172.21.15.70:0/3204588364 shutdown_connections 2024-06-04T05:17:27.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.680+0000 7f68cb038740 1 --2- 172.21.15.70:0/3204588364 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556dcd2db360 0x556dcd2dd780 crc :-1 s=READY pgs=434 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.680+0000 7f68cb038740 1 --2- 172.21.15.70:0/3204588364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcd25fe10 0x556dcd2645f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.680+0000 7f68cb038740 1 --2- 172.21.15.70:0/3204588364 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556dcd251020 0x556dcd2640b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.680+0000 7f68cb038740 1 --2- 172.21.15.70:0/3204588364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556dcd250700 0x556dcd263b70 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.680+0000 7f68cb038740 1 -- 172.21.15.70:0/3204588364 >> 172.21.15.70:0/3204588364 conn(0x556dcd24cd30 msgr2=0x556dcd25d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:27.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.680+0000 7f68a9ffb640 0 client.29393 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:27.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.680+0000 7f68cb038740 1 -- 172.21.15.70:0/3204588364 shutdown_connections 2024-06-04T05:17:27.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.680+0000 7f68cb038740 1 -- 172.21.15.70:0/3204588364 wait complete. 2024-06-04T05:17:27.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.680+0000 7f68cb038740 20 client.29393 trim_cache size 0 max 16384 2024-06-04T05:17:27.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:17:27.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:17:27.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:17:27.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.681+0000 7f68c8319640 1 -- 172.21.15.70:0/2977555652 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68ac0b2a10 msgr2=0x7f68ac0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:27.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.681+0000 7f68c8319640 1 --2- 172.21.15.70:0/2977555652 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68ac0b2a10 0x7f68ac0b4ed0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f68bc0096c0 tx=0x7f68bc0058d0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.681+0000 7f68c8319640 1 -- 172.21.15.70:0/2977555652 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68c0102340 msgr2=0x7f68c01995f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:27.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.681+0000 7f68c8319640 1 --2- 172.21.15.70:0/2977555652 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68c0102340 0x7f68c01995f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f68b000e9f0 tx=0x7f68b000eeb0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.681+0000 7f68c8319640 1 -- 172.21.15.70:0/2977555652 shutdown_connections 2024-06-04T05:17:27.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.681+0000 7f68c8319640 1 --2- 172.21.15.70:0/2977555652 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f68ac0b2a10 0x7f68ac0b4ed0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.681+0000 7f68c8319640 1 --2- 172.21.15.70:0/2977555652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68c0107940 0x7f68c019d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.681+0000 7f68c8319640 1 --2- 172.21.15.70:0/2977555652 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68c0102340 0x7f68c01995f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.681+0000 7f68c8319640 1 --2- 172.21.15.70:0/2977555652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68c0101980 0x7f68c01990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:27.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.681+0000 7f68c8319640 1 -- 172.21.15.70:0/2977555652 >> 172.21.15.70:0/2977555652 conn(0x7f68c0078ad0 msgr2=0x7f68c006b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:27.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.681+0000 7f68c8319640 1 -- 172.21.15.70:0/2977555652 shutdown_connections 2024-06-04T05:17:27.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:27.681+0000 7f68c8319640 1 -- 172.21.15.70:0/2977555652 wait complete. 2024-06-04T05:17:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:27 smithi112 ceph-mon[36453]: pgmap v637: 97 pgs: 97 active+clean; 761 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.1 MiB/s wr, 427 op/s 2024-06-04T05:17:27.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:27 smithi098 ceph-mon[35569]: pgmap v637: 97 pgs: 97 active+clean; 761 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.1 MiB/s wr, 427 op/s 2024-06-04T05:17:27.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 49246 6.79 MB/sec execute 259 sec latency 88.267 ms 2024-06-04T05:17:28.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 49438 6.82 MB/sec execute 260 sec latency 48.052 ms 2024-06-04T05:17:29.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:29 smithi070 ceph-mon[32631]: pgmap v638: 97 pgs: 97 active+clean; 761 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 274 op/s 2024-06-04T05:17:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:29 smithi112 ceph-mon[36453]: pgmap v638: 97 pgs: 97 active+clean; 761 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 274 op/s 2024-06-04T05:17:29.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:29 smithi098 ceph-mon[35569]: pgmap v638: 97 pgs: 97 active+clean; 761 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 274 op/s 2024-06-04T05:17:29.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 49654 6.85 MB/sec execute 261 sec latency 56.020 ms 2024-06-04T05:17:30.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 49804 6.86 MB/sec execute 262 sec latency 131.286 ms 2024-06-04T05:17:31.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:31 smithi070 ceph-mon[32631]: pgmap v639: 97 pgs: 97 active+clean; 761 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 274 op/s 2024-06-04T05:17:31.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:31 smithi112 ceph-mon[36453]: pgmap v639: 97 pgs: 97 active+clean; 761 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 274 op/s 2024-06-04T05:17:31.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:31 smithi098 ceph-mon[35569]: pgmap v639: 97 pgs: 97 active+clean; 761 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 274 op/s 2024-06-04T05:17:31.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 50005 6.86 MB/sec execute 263 sec latency 91.866 ms 2024-06-04T05:17:32.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 50175 6.88 MB/sec execute 264 sec latency 207.198 ms 2024-06-04T05:17:33.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:33 smithi070 ceph-mon[32631]: pgmap v640: 97 pgs: 97 active+clean; 779 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.5 MiB/s wr, 435 op/s 2024-06-04T05:17:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:33 smithi112 ceph-mon[36453]: pgmap v640: 97 pgs: 97 active+clean; 779 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.5 MiB/s wr, 435 op/s 2024-06-04T05:17:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:33 smithi098 ceph-mon[35569]: pgmap v640: 97 pgs: 97 active+clean; 779 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.5 MiB/s wr, 435 op/s 2024-06-04T05:17:33.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 50332 6.89 MB/sec execute 265 sec latency 48.929 ms 2024-06-04T05:17:34.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 50528 6.91 MB/sec execute 266 sec latency 51.014 ms 2024-06-04T05:17:35.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:35 smithi070 ceph-mon[32631]: pgmap v641: 97 pgs: 97 active+clean; 779 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.0 MiB/s wr, 301 op/s 2024-06-04T05:17:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:35 smithi112 ceph-mon[36453]: pgmap v641: 97 pgs: 97 active+clean; 779 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.0 MiB/s wr, 301 op/s 2024-06-04T05:17:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:35 smithi098 ceph-mon[35569]: pgmap v641: 97 pgs: 97 active+clean; 779 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.0 MiB/s wr, 301 op/s 2024-06-04T05:17:35.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 50708 6.93 MB/sec execute 267 sec latency 130.441 ms 2024-06-04T05:17:36.419 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:17:36.794 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:17:36.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 50856 6.93 MB/sec execute 268 sec latency 92.290 ms 2024-06-04T05:17:37.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:37 smithi070 ceph-mon[32631]: pgmap v642: 97 pgs: 97 active+clean; 788 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.8 MiB/s wr, 435 op/s 2024-06-04T05:17:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:37 smithi112 ceph-mon[36453]: pgmap v642: 97 pgs: 97 active+clean; 788 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.8 MiB/s wr, 435 op/s 2024-06-04T05:17:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:37 smithi098 ceph-mon[35569]: pgmap v642: 97 pgs: 97 active+clean; 788 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.8 MiB/s wr, 435 op/s 2024-06-04T05:17:37.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 51002 6.91 MB/sec execute 269 sec latency 63.321 ms 2024-06-04T05:17:38.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 51164 6.91 MB/sec execute 270 sec latency 243.300 ms 2024-06-04T05:17:39.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.237+0000 7f3e922a0700 1 -- 172.21.15.70:0/1249299016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e8c1095d0 msgr2=0x7f3e8c1099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:39.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.237+0000 7f3e922a0700 1 --2- 172.21.15.70:0/1249299016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e8c1095d0 0x7f3e8c1099b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c009990 tx=0x7f3e7c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.237+0000 7f3e922a0700 1 -- 172.21.15.70:0/1249299016 shutdown_connections 2024-06-04T05:17:39.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.237+0000 7f3e922a0700 1 --2- 172.21.15.70:0/1249299016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e8c103f80 0x7f3e8c110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.237+0000 7f3e922a0700 1 --2- 172.21.15.70:0/1249299016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e8c1095d0 0x7f3e8c1099b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.237+0000 7f3e922a0700 1 --2- 172.21.15.70:0/1249299016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e8c1035c0 0x7f3e8c103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.237+0000 7f3e922a0700 1 -- 172.21.15.70:0/1249299016 >> 172.21.15.70:0/1249299016 conn(0x7f3e8c07a610 msgr2=0x7f3e8c07aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:39.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.237+0000 7f3e922a0700 1 -- 172.21.15.70:0/1249299016 shutdown_connections 2024-06-04T05:17:39.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.238+0000 7f3e922a0700 1 -- 172.21.15.70:0/1249299016 wait complete. 2024-06-04T05:17:39.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.238+0000 7f3e922a0700 1 Processor -- start 2024-06-04T05:17:39.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.238+0000 7f3e922a0700 1 -- start start 2024-06-04T05:17:39.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e922a0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e8c1035c0 0x7f3e8c1a5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e922a0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e8c103f80 0x7f3e8c1a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e922a0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e8c1095d0 0x7f3e8c10f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e922a0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c113630 con 0x7f3e8c103f80 2024-06-04T05:17:39.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e922a0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1134b0 con 0x7f3e8c1035c0 2024-06-04T05:17:39.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e922a0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1137b0 con 0x7f3e8c1095d0 2024-06-04T05:17:39.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e8c1035c0 0x7f3e8c1a5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:39.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e8c1035c0 0x7f3e8c1a5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:37166/0 (socket says 172.21.15.70:37166) 2024-06-04T05:17:39.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e8bfff700 1 -- 172.21.15.70:0/3751681102 learned_addr learned my addr 172.21.15.70:0/3751681102 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:39.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e9083d700 1 --2- 172.21.15.70:0/3751681102 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e8c1095d0 0x7f3e8c10f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:39.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e8bfff700 1 -- 172.21.15.70:0/3751681102 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e8c1095d0 msgr2=0x7f3e8c10f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:39.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e8b7fe700 1 --2- 172.21.15.70:0/3751681102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e8c103f80 0x7f3e8c1a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:39.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e8bfff700 1 --2- 172.21.15.70:0/3751681102 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e8c1095d0 0x7f3e8c10f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e8bfff700 1 -- 172.21.15.70:0/3751681102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e8c103f80 msgr2=0x7f3e8c1a62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:39.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e8bfff700 1 --2- 172.21.15.70:0/3751681102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e8c103f80 0x7f3e8c1a62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e8bfff700 1 -- 172.21.15.70:0/3751681102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8c10fe00 con 0x7f3e8c1035c0 2024-06-04T05:17:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e9083d700 1 --2- 172.21.15.70:0/3751681102 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e8c1095d0 0x7f3e8c10f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:17:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.239+0000 7f3e8b7fe700 1 --2- 172.21.15.70:0/3751681102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e8c103f80 0x7f3e8c1a62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:17:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.240+0000 7f3e8bfff700 1 --2- 172.21.15.70:0/3751681102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e8c1035c0 0x7f3e8c1a5db0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00b650 tx=0x7f3e7c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.240+0000 7f3e897fa700 1 -- 172.21.15.70:0/3751681102 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c027070 con 0x7f3e8c1035c0 2024-06-04T05:17:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.240+0000 7f3e897fa700 1 -- 172.21.15.70:0/3751681102 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e7c022470 con 0x7f3e8c1035c0 2024-06-04T05:17:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.240+0000 7f3e922a0700 1 -- 172.21.15.70:0/3751681102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e8c110090 con 0x7f3e8c1035c0 2024-06-04T05:17:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.240+0000 7f3e897fa700 1 -- 172.21.15.70:0/3751681102 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c017650 con 0x7f3e8c1035c0 2024-06-04T05:17:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.240+0000 7f3e922a0700 1 -- 172.21.15.70:0/3751681102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e8c1acc10 con 0x7f3e8c1035c0 2024-06-04T05:17:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.243+0000 7f3e897fa700 1 -- 172.21.15.70:0/3751681102 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3e7c009d10 con 0x7f3e8c1035c0 2024-06-04T05:17:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.244+0000 7f3e897fa700 1 --2- 172.21.15.70:0/3751681102 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e740b29f0 0x7f3e740b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.244+0000 7f3e8b7fe700 1 --2- 172.21.15.70:0/3751681102 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e740b29f0 0x7f3e740b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.244+0000 7f3e897fa700 1 -- 172.21.15.70:0/3751681102 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7f3e7c0e40f0 con 0x7f3e8c1035c0 2024-06-04T05:17:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.271+0000 7f3e8b7fe700 1 --2- 172.21.15.70:0/3751681102 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e740b29f0 0x7f3e740b4eb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c077860 tx=0x7f3e80009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.274+0000 7f3e9cb35b80 1 Processor -- start 2024-06-04T05:17:39.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.274+0000 7f3e9cb35b80 1 -- start start 2024-06-04T05:17:39.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.274+0000 7f3e9cb35b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563dac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.274+0000 7f3e9cb35b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563db210 0x55e1563de940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.274+0000 7f3e9cb35b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563def60 0x55e1563df3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.274+0000 7f3e9cb35b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e156339c90 con 0x55e1563def60 2024-06-04T05:17:39.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.274+0000 7f3e9cb35b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e156329830 con 0x55e1563da860 2024-06-04T05:17:39.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.274+0000 7f3e9cb35b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e15632d870 con 0x55e1563db210 2024-06-04T05:17:39.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.274+0000 7f3e8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563dac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:39.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.275+0000 7f3e8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563dac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:37176/0 (socket says 172.21.15.70:37176) 2024-06-04T05:17:39.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.275+0000 7f3e8bfff700 1 -- 172.21.15.70:0/3287860799 learned_addr learned my addr 172.21.15.70:0/3287860799 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:39.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.275+0000 7f3e8b7fe700 1 --2- 172.21.15.70:0/3287860799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563db210 0x55e1563de940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:39.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.275+0000 7f3e9083d700 1 --2- 172.21.15.70:0/3287860799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563def60 0x55e1563df3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:39.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.275+0000 7f3e8bfff700 1 -- 172.21.15.70:0/3287860799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563db210 msgr2=0x55e1563de940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:39.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.275+0000 7f3e8bfff700 1 --2- 172.21.15.70:0/3287860799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563db210 0x55e1563de940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.275+0000 7f3e8bfff700 1 -- 172.21.15.70:0/3287860799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563def60 msgr2=0x55e1563df3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:39.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.275+0000 7f3e8bfff700 1 --2- 172.21.15.70:0/3287860799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563def60 0x55e1563df3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.275+0000 7f3e8bfff700 1 -- 172.21.15.70:0/3287860799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1562798d0 con 0x55e1563da860 2024-06-04T05:17:39.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.275+0000 7f3e9083d700 1 --2- 172.21.15.70:0/3287860799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563def60 0x55e1563df3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:17:39.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.275+0000 7f3e8bfff700 1 --2- 172.21.15.70:0/3287860799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563dac40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0a74b0 tx=0x7f3e7c0aac70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af0504c9a8e93f76 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:39.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.275+0000 7f3e727fc700 1 -- 172.21.15.70:0/3287860799 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c0af7e0 con 0x55e1563da860 2024-06-04T05:17:39.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.275+0000 7f3e727fc700 1 -- 172.21.15.70:0/3287860799 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e7c0b4200 con 0x55e1563da860 2024-06-04T05:17:39.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.276+0000 7f3e727fc700 1 -- 172.21.15.70:0/3287860799 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c0b3d70 con 0x55e1563da860 2024-06-04T05:17:39.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.276+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/3287860799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 msgr2=0x55e1563dac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:39.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.276+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/3287860799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563dac40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0a74b0 tx=0x7f3e7c0aac70 comp rx=0 tx=0).stop 2024-06-04T05:17:39.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.276+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/3287860799 shutdown_connections 2024-06-04T05:17:39.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.276+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/3287860799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563def60 0x55e1563df3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.276+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/3287860799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563dac40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.276+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/3287860799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563db210 0x55e1563de940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.276+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/3287860799 >> 172.21.15.70:0/3287860799 conn(0x55e156308080 msgr2=0x55e1563d4e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.277+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/3287860799 shutdown_connections 2024-06-04T05:17:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.277+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/3287860799 wait complete. 2024-06-04T05:17:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.277+0000 7f3e9cb35b80 1 -- start start 2024-06-04T05:17:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.277+0000 7f3e9cb35b80 1 Processor -- start 2024-06-04T05:17:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.278+0000 7f3e9cb35b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563e6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.278+0000 7f3e9cb35b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 0x55e1563e3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.278+0000 7f3e9cb35b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563def60 0x55e1563e3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.278+0000 7f3e9cb35b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e15622ffa0 con 0x55e1563db210 2024-06-04T05:17:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.278+0000 7f3e9cb35b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e1562cb7c0 con 0x55e1563da860 2024-06-04T05:17:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.278+0000 7f3e9cb35b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e1563e6ee0 con 0x55e1563def60 2024-06-04T05:17:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.278+0000 7f3e9cb35b80 10 client.0 _ll_register_callbacks cb 0x55e1562c4920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:39.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.278+0000 7f3e8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563e6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:39.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.278+0000 7f3e8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563e6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:37186/0 (socket says 172.21.15.70:37186) 2024-06-04T05:17:39.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.278+0000 7f3e8bfff700 1 -- 172.21.15.70:0/2826027936 learned_addr learned my addr 172.21.15.70:0/2826027936 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:39.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.278+0000 7f3e8b7fe700 1 --2- 172.21.15.70:0/2826027936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 0x55e1563e3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:39.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.278+0000 7f3e9083d700 1 --2- 172.21.15.70:0/2826027936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563def60 0x55e1563e3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:39.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.279+0000 7f3e8bfff700 1 -- 172.21.15.70:0/2826027936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563def60 msgr2=0x55e1563e3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:39.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.279+0000 7f3e8bfff700 1 --2- 172.21.15.70:0/2826027936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563def60 0x55e1563e3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.279+0000 7f3e8bfff700 1 -- 172.21.15.70:0/2826027936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 msgr2=0x55e1563e3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:39.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.279+0000 7f3e8bfff700 1 --2- 172.21.15.70:0/2826027936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 0x55e1563e3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.279+0000 7f3e8bfff700 1 -- 172.21.15.70:0/2826027936 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1562c8690 con 0x55e1563da860 2024-06-04T05:17:39.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.279+0000 7f3e8b7fe700 1 --2- 172.21.15.70:0/2826027936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 0x55e1563e3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:39.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.279+0000 7f3e8bfff700 1 --2- 172.21.15.70:0/2826027936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563e6410 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0e18c0 tx=0x7f3e7c009710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f2637f7cfb1e05b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:39.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.279+0000 7f3e6b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:17:39.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.279+0000 7f3e6b7fe700 1 -- 172.21.15.70:0/2826027936 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c0af7e0 con 0x55e1563da860 2024-06-04T05:17:39.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.280+0000 7f3e6b7fe700 1 -- 172.21.15.70:0/2826027936 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7f3e7c0a7a20 con 0x55e1563da860 2024-06-04T05:17:39.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.280+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/2826027936 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e7c0b4200 con 0x55e1563da860 2024-06-04T05:17:39.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.280+0000 7f3e6b7fe700 1 -- 172.21.15.70:0/2826027936 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f3e7c0b3f80 con 0x55e1563da860 2024-06-04T05:17:39.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.280+0000 7f3e9cb35b80 10 client.29405 fetch_fsmap learned FSMap version 59 2024-06-04T05:17:39.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.280+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/2826027936 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55e1563f15c0 con 0x55e1563da860 2024-06-04T05:17:39.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.280+0000 7f3e6b7fe700 1 -- 172.21.15.70:0/2826027936 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e7c06ae50 con 0x55e1563da860 2024-06-04T05:17:39.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.280+0000 7f3e6b7fe700 1 -- 172.21.15.70:0/2826027936 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f3e7c00fba0 con 0x55e1563da860 2024-06-04T05:17:39.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.280+0000 7f3e9cb35b80 10 client.29405 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:17:39.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.280+0000 7f3e9cb35b80 10 client.29405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:17:39.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.280+0000 7f3e9cb35b80 20 client.29405 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:39.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.281+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/2826027936 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e1563f3a90 0x55e1563f5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.281+0000 7f3e9cb35b80 4 client.29405 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:17:39.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.281+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/2826027936 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1563a69e0 con 0x55e1563f3a90 2024-06-04T05:17:39.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.281+0000 7f3e8b7fe700 1 --2- 172.21.15.70:0/2826027936 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e1563f3a90 0x55e1563f5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:39.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.281+0000 7f3e6b7fe700 10 client.29405 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:39.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.281+0000 7f3e8b7fe700 1 --2- 172.21.15.70:0/2826027936 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e1563f3a90 0x55e1563f5e80 crc :-1 s=READY pgs=435 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcea51c183957bff server_cookie=c0956e2ca295f465 in_seq=0 out_seq=0 2024-06-04T05:17:39.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.283+0000 7f3e6b7fe700 1 -- 172.21.15.70:0/2826027936 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e1563a69e0 con 0x55e1563f3a90 2024-06-04T05:17:39.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.283+0000 7f3e6b7fe700 10 client.29405 handle_command_reply: tid=0 2024-06-04T05:17:39.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.284+0000 7f3e9cb35b80 1 client.29405 shutdown 2024-06-04T05:17:39.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.284+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/2826027936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 msgr2=0x55e1563e6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:39.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.284+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/2826027936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563e6410 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0e18c0 tx=0x7f3e7c009710 comp rx=0 tx=0).stop 2024-06-04T05:17:39.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.284+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/2826027936 shutdown_connections 2024-06-04T05:17:39.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.284+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/2826027936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563def60 0x55e1563e3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.284+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/2826027936 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e1563f3a90 0x55e1563f5e80 crc :-1 s=READY pgs=435 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.284+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/2826027936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563e6410 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.284+0000 7f3e6b7fe700 0 client.29405 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:39.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.284+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/2826027936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 0x55e1563e3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.284+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/2826027936 >> 172.21.15.70:0/2826027936 conn(0x55e156308080 msgr2=0x55e1563d4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:39.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.284+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/2826027936 shutdown_connections 2024-06-04T05:17:39.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.285+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/2826027936 wait complete. 2024-06-04T05:17:39.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.285+0000 7f3e9cb35b80 20 client.29405 trim_cache size 0 max 16384 2024-06-04T05:17:39.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.316+0000 7f3e9cb35b80 1 Processor -- start 2024-06-04T05:17:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.316+0000 7f3e9cb35b80 1 -- start start 2024-06-04T05:17:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.316+0000 7f3e9cb35b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563ddd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.316+0000 7f3e9cb35b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 0x55e1563de250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.316+0000 7f3e9cb35b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563def60 0x55e1563fb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.316+0000 7f3e9cb35b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1b44a0 con 0x55e1563db210 2024-06-04T05:17:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.316+0000 7f3e9cb35b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1b6040 con 0x55e1563da860 2024-06-04T05:17:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.316+0000 7f3e9cb35b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1b4bd0 con 0x55e1563def60 2024-06-04T05:17:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.317+0000 7f3e8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563ddd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.317+0000 7f3e8b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 0x55e1563de250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.317+0000 7f3e8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563ddd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:37192/0 (socket says 172.21.15.70:37192) 2024-06-04T05:17:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.317+0000 7f3e8bfff700 1 -- 172.21.15.70:0/1196989186 learned_addr learned my addr 172.21.15.70:0/1196989186 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.317+0000 7f3e8b7fe700 1 --2- 172.21.15.70:0/1196989186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 0x55e1563de250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33210/0 (socket says 172.21.15.70:33210) 2024-06-04T05:17:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.317+0000 7f3e8b7fe700 1 -- 172.21.15.70:0/1196989186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563def60 msgr2=0x55e1563fb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.317+0000 7f3e8b7fe700 1 --2- 172.21.15.70:0/1196989186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563def60 0x55e1563fb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.317+0000 7f3e8b7fe700 1 -- 172.21.15.70:0/1196989186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 msgr2=0x55e1563ddd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.317+0000 7f3e8b7fe700 1 --2- 172.21.15.70:0/1196989186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563ddd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.317+0000 7f3e8b7fe700 1 -- 172.21.15.70:0/1196989186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1563eff40 con 0x55e1563db210 2024-06-04T05:17:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.317+0000 7f3e8b7fe700 1 --2- 172.21.15.70:0/1196989186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 0x55e1563de250 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x55e1563f6530 tx=0x7f3e80013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da122052cfcb412b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.318+0000 7f3e71ffb700 1 -- 172.21.15.70:0/1196989186 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e80003960 con 0x55e1563db210 2024-06-04T05:17:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.318+0000 7f3e71ffb700 1 -- 172.21.15.70:0/1196989186 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e80032070 con 0x55e1563db210 2024-06-04T05:17:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.318+0000 7f3e71ffb700 1 -- 172.21.15.70:0/1196989186 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e80012450 con 0x55e1563db210 2024-06-04T05:17:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.318+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/1196989186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 msgr2=0x55e1563de250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.318+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/1196989186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 0x55e1563de250 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x55e1563f6530 tx=0x7f3e80013000 comp rx=0 tx=0).stop 2024-06-04T05:17:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.319+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/1196989186 shutdown_connections 2024-06-04T05:17:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.319+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/1196989186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563def60 0x55e1563fb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.319+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/1196989186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563ddd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.319+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/1196989186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 0x55e1563de250 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.319+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/1196989186 >> 172.21.15.70:0/1196989186 conn(0x55e156308080 msgr2=0x55e1563dd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:39.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.319+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/1196989186 shutdown_connections 2024-06-04T05:17:39.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.319+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/1196989186 wait complete. 2024-06-04T05:17:39.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.319+0000 7f3e9cb35b80 1 -- start start 2024-06-04T05:17:39.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e9cb35b80 1 Processor -- start 2024-06-04T05:17:39.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e9cb35b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563ea130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e9cb35b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 0x55e1563ea670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e9cb35b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563def60 0x55e1563eabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e9cb35b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1b5970 con 0x55e1563db210 2024-06-04T05:17:39.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e9cb35b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1b52a0 con 0x55e1563da860 2024-06-04T05:17:39.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e9cb35b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1b2460 con 0x55e1563def60 2024-06-04T05:17:39.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e8b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 0x55e1563ea670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:39.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563ea130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:39.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563ea130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:37204/0 (socket says 172.21.15.70:37204) 2024-06-04T05:17:39.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e8b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 0x55e1563ea670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33218/0 (socket says 172.21.15.70:33218) 2024-06-04T05:17:39.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e8bfff700 1 -- 172.21.15.70:0/466907461 learned_addr learned my addr 172.21.15.70:0/466907461 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:39.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e9083d700 1 --2- 172.21.15.70:0/466907461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563def60 0x55e1563eabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:39.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e9cb35b80 10 client.0 _ll_register_callbacks cb 0x55e1562c4920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:39.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e8bfff700 1 -- 172.21.15.70:0/466907461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563def60 msgr2=0x55e1563eabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:39.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e8bfff700 1 --2- 172.21.15.70:0/466907461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563def60 0x55e1563eabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e8bfff700 1 -- 172.21.15.70:0/466907461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 msgr2=0x55e1563ea670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:39.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e8bfff700 1 --2- 172.21.15.70:0/466907461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 0x55e1563ea670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e8bfff700 1 -- 172.21.15.70:0/466907461 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e15645d700 con 0x55e1563da860 2024-06-04T05:17:39.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e9083d700 1 --2- 172.21.15.70:0/466907461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563def60 0x55e1563eabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:39.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.320+0000 7f3e8b7fe700 1 --2- 172.21.15.70:0/466907461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 0x55e1563ea670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:39.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.321+0000 7f3e8bfff700 1 --2- 172.21.15.70:0/466907461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563ea130 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0e1e10 tx=0x7f3e7c0aa8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b6e632a0aa8de95 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:39.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.321+0000 7f3e6b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:17:39.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.321+0000 7f3e6b7fe700 1 -- 172.21.15.70:0/466907461 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c0a72f0 con 0x55e1563da860 2024-06-04T05:17:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.321+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/466907461 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e80032070 con 0x55e1563da860 2024-06-04T05:17:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.321+0000 7f3e6b7fe700 1 -- 172.21.15.70:0/466907461 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7f3e7c0a7a20 con 0x55e1563da860 2024-06-04T05:17:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.321+0000 7f3e6b7fe700 1 -- 172.21.15.70:0/466907461 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f3e7c0af7e0 con 0x55e1563da860 2024-06-04T05:17:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.321+0000 7f3e9cb35b80 10 client.29411 fetch_fsmap learned FSMap version 59 2024-06-04T05:17:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.321+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/466907461 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55e15645d700 con 0x55e1563da860 2024-06-04T05:17:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.321+0000 7f3e6b7fe700 1 -- 172.21.15.70:0/466907461 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e80032070 con 0x55e1563da860 2024-06-04T05:17:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.321+0000 7f3e6b7fe700 1 -- 172.21.15.70:0/466907461 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f3e7c0b2ce0 con 0x55e1563da860 2024-06-04T05:17:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.321+0000 7f3e9cb35b80 10 client.29411 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:17:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.321+0000 7f3e9cb35b80 10 client.29411 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:17:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.321+0000 7f3e9cb35b80 20 client.29411 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.322+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/466907461 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e1563f3a90 0x55e156460ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.322+0000 7f3e9cb35b80 4 client.29411 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:17:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.322+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/466907461 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e15633bbe0 con 0x55e1563f3a90 2024-06-04T05:17:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.322+0000 7f3e8b7fe700 1 --2- 172.21.15.70:0/466907461 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e1563f3a90 0x55e156460ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.322+0000 7f3e6b7fe700 10 client.29411 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.322+0000 7f3e8b7fe700 1 --2- 172.21.15.70:0/466907461 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e1563f3a90 0x55e156460ca0 crc :-1 s=READY pgs=436 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13e1927fad2bed66 server_cookie=41bb077934e45dff in_seq=0 out_seq=0 2024-06-04T05:17:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.324+0000 7f3e6b7fe700 1 -- 172.21.15.70:0/466907461 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e15633bbe0 con 0x55e1563f3a90 2024-06-04T05:17:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.324+0000 7f3e6b7fe700 10 client.29411 handle_command_reply: tid=0 2024-06-04T05:17:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.324+0000 7f3e9cb35b80 1 client.29411 shutdown 2024-06-04T05:17:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.325+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/466907461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 msgr2=0x55e1563ea130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.325+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/466907461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563ea130 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0e1e10 tx=0x7f3e7c0aa8c0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.325+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/466907461 shutdown_connections 2024-06-04T05:17:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.325+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/466907461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e1563def60 0x55e1563eabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.325+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/466907461 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e1563f3a90 0x55e156460ca0 crc :-1 s=READY pgs=436 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.325+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/466907461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1563da860 0x55e1563ea130 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.325+0000 7f3e9cb35b80 1 --2- 172.21.15.70:0/466907461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1563db210 0x55e1563ea670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.325+0000 7f3e6b7fe700 0 client.29411 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:39.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.325+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/466907461 >> 172.21.15.70:0/466907461 conn(0x55e156308080 msgr2=0x55e1563ef7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:39.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.325+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/466907461 shutdown_connections 2024-06-04T05:17:39.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.325+0000 7f3e9cb35b80 1 -- 172.21.15.70:0/466907461 wait complete. 2024-06-04T05:17:39.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.325+0000 7f3e9cb35b80 20 client.29411 trim_cache size 0 max 16384 2024-06-04T05:17:39.353 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:17:39.353 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:17:39.353 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:17:39.353 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:17:39.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.326+0000 7f3e922a0700 1 -- 172.21.15.70:0/3751681102 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e740b29f0 msgr2=0x7f3e740b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:39.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.326+0000 7f3e922a0700 1 --2- 172.21.15.70:0/3751681102 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e740b29f0 0x7f3e740b4eb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c077860 tx=0x7f3e80009260 comp rx=0 tx=0).stop 2024-06-04T05:17:39.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.326+0000 7f3e922a0700 1 -- 172.21.15.70:0/3751681102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e8c1035c0 msgr2=0x7f3e8c1a5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:39.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.326+0000 7f3e922a0700 1 --2- 172.21.15.70:0/3751681102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e8c1035c0 0x7f3e8c1a5db0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00b650 tx=0x7f3e7c0047b0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.326+0000 7f3e922a0700 1 -- 172.21.15.70:0/3751681102 shutdown_connections 2024-06-04T05:17:39.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.326+0000 7f3e922a0700 1 --2- 172.21.15.70:0/3751681102 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e740b29f0 0x7f3e740b4eb0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.326+0000 7f3e922a0700 1 --2- 172.21.15.70:0/3751681102 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e8c1095d0 0x7f3e8c10f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.326+0000 7f3e922a0700 1 --2- 172.21.15.70:0/3751681102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e8c1035c0 0x7f3e8c1a5db0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.326+0000 7f3e922a0700 1 --2- 172.21.15.70:0/3751681102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e8c103f80 0x7f3e8c1a62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:39.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.326+0000 7f3e922a0700 1 -- 172.21.15.70:0/3751681102 >> 172.21.15.70:0/3751681102 conn(0x7f3e8c07a610 msgr2=0x7f3e8c102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:39.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.326+0000 7f3e922a0700 1 -- 172.21.15.70:0/3751681102 shutdown_connections 2024-06-04T05:17:39.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:39.326+0000 7f3e922a0700 1 -- 172.21.15.70:0/3751681102 wait complete. 2024-06-04T05:17:39.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:39 smithi070 ceph-mon[32631]: pgmap v643: 97 pgs: 97 active+clean; 788 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 294 op/s 2024-06-04T05:17:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:39 smithi112 ceph-mon[36453]: pgmap v643: 97 pgs: 97 active+clean; 788 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 294 op/s 2024-06-04T05:17:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:39 smithi098 ceph-mon[35569]: pgmap v643: 97 pgs: 97 active+clean; 788 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 294 op/s 2024-06-04T05:17:39.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 51362 6.92 MB/sec execute 271 sec latency 67.200 ms 2024-06-04T05:17:40.260 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:17:40.260 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:17:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:17:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.810+0000 7f6ec1845640 1 -- 172.21.15.70:0/4192079300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ebc105870 msgr2=0x7f6ebc1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.810+0000 7f6ec1845640 1 --2- 172.21.15.70:0/4192079300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ebc105870 0x7f6ebc1098b0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0009900 tx=0x7f6eb002a410 comp rx=0 tx=0).stop 2024-06-04T05:17:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.810+0000 7f6ec1845640 1 -- 172.21.15.70:0/4192079300 shutdown_connections 2024-06-04T05:17:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.810+0000 7f6ec1845640 1 --2- 172.21.15.70:0/4192079300 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ebc109f90 0x7f6ebc111b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.810+0000 7f6ec1845640 1 --2- 172.21.15.70:0/4192079300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ebc105870 0x7f6ebc1098b0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.810+0000 7f6ec1845640 1 --2- 172.21.15.70:0/4192079300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ebc104ec0 0x7f6ebc1052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.810+0000 7f6ec1845640 1 -- 172.21.15.70:0/4192079300 >> 172.21.15.70:0/4192079300 conn(0x7f6ebc100700 msgr2=0x7f6ebc102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.810+0000 7f6ec1845640 1 -- 172.21.15.70:0/4192079300 shutdown_connections 2024-06-04T05:17:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.810+0000 7f6ec1845640 1 -- 172.21.15.70:0/4192079300 wait complete. 2024-06-04T05:17:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.811+0000 7f6ec1845640 1 Processor -- start 2024-06-04T05:17:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.811+0000 7f6ec1845640 1 -- start start 2024-06-04T05:17:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.811+0000 7f6ec1845640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ebc104ec0 0x7f6ebc076f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6ec1845640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ebc109f90 0x7f6ebc0774a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6ec1845640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ebc0779e0 0x7f6ebc1ab920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6ec1845640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ebc114da0 con 0x7f6ebc104ec0 2024-06-04T05:17:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6ec1845640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ebc114c20 con 0x7f6ebc0779e0 2024-06-04T05:17:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6ec1845640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6ebc114f20 con 0x7f6ebc109f90 2024-06-04T05:17:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6ebaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ebc104ec0 0x7f6ebc076f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6ebaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ebc104ec0 0x7f6ebc076f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33228/0 (socket says 172.21.15.70:33228) 2024-06-04T05:17:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6ebaffd640 1 -- 172.21.15.70:0/2773906498 learned_addr learned my addr 172.21.15.70:0/2773906498 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6ebb7fe640 1 --2- 172.21.15.70:0/2773906498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ebc0779e0 0x7f6ebc1ab920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6eba7fc640 1 --2- 172.21.15.70:0/2773906498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ebc109f90 0x7f6ebc0774a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6ebaffd640 1 -- 172.21.15.70:0/2773906498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ebc109f90 msgr2=0x7f6ebc0774a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6ebaffd640 1 --2- 172.21.15.70:0/2773906498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ebc109f90 0x7f6ebc0774a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6ebaffd640 1 -- 172.21.15.70:0/2773906498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ebc0779e0 msgr2=0x7f6ebc1ab920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6ebaffd640 1 --2- 172.21.15.70:0/2773906498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ebc0779e0 0x7f6ebc1ab920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6ebaffd640 1 -- 172.21.15.70:0/2773906498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ebc1abe60 con 0x7f6ebc104ec0 2024-06-04T05:17:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6eba7fc640 1 --2- 172.21.15.70:0/2773906498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ebc109f90 0x7f6ebc0774a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:17:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6ebb7fe640 1 --2- 172.21.15.70:0/2773906498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ebc0779e0 0x7f6ebc1ab920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.812+0000 7f6ebaffd640 1 --2- 172.21.15.70:0/2773906498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ebc104ec0 0x7f6ebc076f60 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f6eac00e9f0 tx=0x7f6eac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.813+0000 7f6ec0843640 1 -- 172.21.15.70:0/2773906498 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eac00cd80 con 0x7f6ebc104ec0 2024-06-04T05:17:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.813+0000 7f6ec0843640 1 -- 172.21.15.70:0/2773906498 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6eac0108c0 con 0x7f6ebc104ec0 2024-06-04T05:17:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.813+0000 7f6ec1845640 1 -- 172.21.15.70:0/2773906498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ebc1ac0f0 con 0x7f6ebc104ec0 2024-06-04T05:17:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.813+0000 7f6ec1845640 1 -- 172.21.15.70:0/2773906498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ebc1ac660 con 0x7f6ebc104ec0 2024-06-04T05:17:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.813+0000 7f6ec0843640 1 -- 172.21.15.70:0/2773906498 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eac0187c0 con 0x7f6ebc104ec0 2024-06-04T05:17:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.816+0000 7f6ec41fd740 1 Processor -- start 2024-06-04T05:17:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.816+0000 7f6ec41fd740 1 -- start start 2024-06-04T05:17:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.816+0000 7f6ec0843640 1 -- 172.21.15.70:0/2773906498 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f6eac018960 con 0x7f6ebc104ec0 2024-06-04T05:17:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6ec41fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 0x55f165f9dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6ec41fd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165f9e140 0x55f165fa84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6ec41fd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165fa8a00 0x55f165faadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6ec41fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f165f0fc30 con 0x55f165f9d820 2024-06-04T05:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6ec41fd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f165eff7d0 con 0x55f165fa8a00 2024-06-04T05:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6ec41fd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f165f03810 con 0x55f165f9e140 2024-06-04T05:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6ebb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 0x55f165f9dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6ebb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 0x55f165f9dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33238/0 (socket says 172.21.15.70:33238) 2024-06-04T05:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6ebb7fe640 1 -- 172.21.15.70:0/3267530703 learned_addr learned my addr 172.21.15.70:0/3267530703 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6eba7fc640 1 --2- 172.21.15.70:0/3267530703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165fa8a00 0x55f165faadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6ebaffd640 1 --2- 172.21.15.70:0/3267530703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165f9e140 0x55f165fa84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6ebb7fe640 1 -- 172.21.15.70:0/3267530703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165f9e140 msgr2=0x55f165fa84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6ebb7fe640 1 --2- 172.21.15.70:0/3267530703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165f9e140 0x55f165fa84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6ebb7fe640 1 -- 172.21.15.70:0/3267530703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165fa8a00 msgr2=0x55f165faadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6ebb7fe640 1 --2- 172.21.15.70:0/3267530703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165fa8a00 0x55f165faadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6ebb7fe640 1 -- 172.21.15.70:0/3267530703 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f165c14210 con 0x55f165f9d820 2024-06-04T05:17:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6ebb7fe640 1 --2- 172.21.15.70:0/3267530703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 0x55f165f9dc00 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4011790 tx=0x7f6eb4011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5d1b1401c76c54 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.817+0000 7f6ec0843640 1 --2- 172.21.15.70:0/2773906498 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ea40b2a90 0x7f6ea40b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.818+0000 7f6ea1ffb640 1 -- 172.21.15.70:0/3267530703 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4003c30 con 0x55f165f9d820 2024-06-04T05:17:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.818+0000 7f6ec0843640 1 -- 172.21.15.70:0/2773906498 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7f6eac0e28e0 con 0x7f6ebc104ec0 2024-06-04T05:17:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.818+0000 7f6ea1ffb640 1 -- 172.21.15.70:0/3267530703 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6eb400fe30 con 0x55f165f9d820 2024-06-04T05:17:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.818+0000 7f6ebaffd640 1 --2- 172.21.15.70:0/2773906498 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ea40b2a90 0x7f6ea40b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.818+0000 7f6ea1ffb640 1 -- 172.21.15.70:0/3267530703 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb40193e0 con 0x55f165f9d820 2024-06-04T05:17:40.824 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.819+0000 7f6ebaffd640 1 --2- 172.21.15.70:0/2773906498 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ea40b2a90 0x7f6ea40b4f50 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0e00b0 tx=0x7f6eac000bc0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:40.824 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.819+0000 7f6ec41fd740 1 -- 172.21.15.70:0/3267530703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 msgr2=0x55f165f9dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:40.824 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.819+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/3267530703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 0x55f165f9dc00 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4011790 tx=0x7f6eb4011c50 comp rx=0 tx=0).stop 2024-06-04T05:17:40.824 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.819+0000 7f6ec41fd740 1 -- 172.21.15.70:0/3267530703 shutdown_connections 2024-06-04T05:17:40.824 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.819+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/3267530703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165fa8a00 0x55f165faadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.824 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.819+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/3267530703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165f9e140 0x55f165fa84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.819+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/3267530703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 0x55f165f9dc00 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.819+0000 7f6ec41fd740 1 -- 172.21.15.70:0/3267530703 >> 172.21.15.70:0/3267530703 conn(0x55f165f99d30 msgr2=0x55f165f9a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.819+0000 7f6ec41fd740 1 -- 172.21.15.70:0/3267530703 shutdown_connections 2024-06-04T05:17:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.819+0000 7f6ec41fd740 1 -- 172.21.15.70:0/3267530703 wait complete. 2024-06-04T05:17:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.819+0000 7f6ec41fd740 1 -- start start 2024-06-04T05:17:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.820+0000 7f6ec41fd740 1 Processor -- start 2024-06-04T05:17:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.820+0000 7f6ec41fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 0x55f165fa5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.820+0000 7f6ec41fd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165f9e140 0x55f165fa5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.826 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.820+0000 7f6ec41fd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165fa8a00 0x55f165fa63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.826 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.820+0000 7f6ec41fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f165be5790 con 0x55f165f9d820 2024-06-04T05:17:40.826 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.820+0000 7f6ec41fd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f165ebb890 con 0x55f165f9e140 2024-06-04T05:17:40.826 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.820+0000 7f6ec41fd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f165fab630 con 0x55f165fa8a00 2024-06-04T05:17:40.826 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.820+0000 7f6ec41fd740 10 client.0 _ll_register_callbacks cb 0x55f165c13a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:40.826 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.820+0000 7f6eba7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165f9e140 0x55f165fa5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:40.826 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.820+0000 7f6eba7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165f9e140 0x55f165fa5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:37230/0 (socket says 172.21.15.70:37230) 2024-06-04T05:17:40.826 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.820+0000 7f6eba7fc640 1 -- 172.21.15.70:0/763779325 learned_addr learned my addr 172.21.15.70:0/763779325 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:40.826 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.820+0000 7f6ebb7fe640 1 --2- 172.21.15.70:0/763779325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 0x55f165fa5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:40.826 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.820+0000 7f6eba7fc640 1 -- 172.21.15.70:0/763779325 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165fa8a00 msgr2=0x55f165fa63e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-04T05:17:40.826 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.820+0000 7f6eba7fc640 1 --2- 172.21.15.70:0/763779325 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165fa8a00 0x55f165fa63e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.826 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.820+0000 7f6eba7fc640 1 -- 172.21.15.70:0/763779325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 msgr2=0x55f165fa5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:40.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 51516 6.92 MB/sec execute 272 sec latency 92.089 ms 2024-06-04T05:17:40.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.821+0000 7f6eba7fc640 1 --2- 172.21.15.70:0/763779325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 0x55f165fa5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.821+0000 7f6eba7fc640 1 -- 172.21.15.70:0/763779325 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f165ce0620 con 0x55f165f9e140 2024-06-04T05:17:40.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.821+0000 7f6ebb7fe640 1 --2- 172.21.15.70:0/763779325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 0x55f165fa5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:40.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.821+0000 7f6eba7fc640 1 --2- 172.21.15.70:0/763779325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165f9e140 0x55f165fa5ea0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc06c120 tx=0x7f6eb0031630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9231841f9929ac4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.821+0000 7f6e9affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.821+0000 7f6e9affd640 1 -- 172.21.15.70:0/763779325 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0035040 con 0x55f165f9e140 2024-06-04T05:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.821+0000 7f6ec41fd740 1 -- 172.21.15.70:0/763779325 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6eb400fe30 con 0x55f165f9e140 2024-06-04T05:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.821+0000 7f6e9affd640 1 -- 172.21.15.70:0/763779325 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7f6eb0004040 con 0x55f165f9e140 2024-06-04T05:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.821+0000 7f6e9affd640 1 -- 172.21.15.70:0/763779325 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f6eb0004240 con 0x55f165f9e140 2024-06-04T05:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.821+0000 7f6ec41fd740 10 client.29423 fetch_fsmap learned FSMap version 59 2024-06-04T05:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.821+0000 7f6ec41fd740 1 -- 172.21.15.70:0/763779325 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55f165fb7370 con 0x55f165f9e140 2024-06-04T05:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.821+0000 7f6e9affd640 1 -- 172.21.15.70:0/763779325 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6eb400fe30 con 0x55f165f9e140 2024-06-04T05:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.822+0000 7f6e9affd640 1 -- 172.21.15.70:0/763779325 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f6eb004c070 con 0x55f165f9e140 2024-06-04T05:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.822+0000 7f6ec41fd740 10 client.29423 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:17:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.822+0000 7f6ec41fd740 10 client.29423 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:17:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.822+0000 7f6ec41fd740 20 client.29423 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.822+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/763779325 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f165fb88a0 0x55f165fbacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.822+0000 7f6ec41fd740 4 client.29423 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:17:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.822+0000 7f6ec41fd740 1 -- 172.21.15.70:0/763779325 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f165f11b80 con 0x55f165fb88a0 2024-06-04T05:17:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.822+0000 7f6ebb7fe640 1 --2- 172.21.15.70:0/763779325 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f165fb88a0 0x55f165fbacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.823+0000 7f6e9affd640 10 client.29423 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.823+0000 7f6ebb7fe640 1 --2- 172.21.15.70:0/763779325 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f165fb88a0 0x55f165fbacf0 crc :-1 s=READY pgs=437 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d501d43582355ea server_cookie=50e29d41d7fb94a5 in_seq=0 out_seq=0 2024-06-04T05:17:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.823+0000 7f6e9affd640 1 -- 172.21.15.70:0/763779325 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f165f11b80 con 0x55f165fb88a0 2024-06-04T05:17:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.823+0000 7f6e9affd640 10 client.29423 handle_command_reply: tid=0 2024-06-04T05:17:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.823+0000 7f6ec41fd740 1 client.29423 shutdown 2024-06-04T05:17:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.823+0000 7f6ec41fd740 1 -- 172.21.15.70:0/763779325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165f9e140 msgr2=0x55f165fa5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.824+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/763779325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165f9e140 0x55f165fa5ea0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc06c120 tx=0x7f6eb0031630 comp rx=0 tx=0).stop 2024-06-04T05:17:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.824+0000 7f6ec41fd740 1 -- 172.21.15.70:0/763779325 shutdown_connections 2024-06-04T05:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.824+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/763779325 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f165fb88a0 0x55f165fbacf0 crc :-1 s=READY pgs=437 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.824+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/763779325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165f9e140 0x55f165fa5ea0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.824+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/763779325 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165fa8a00 0x55f165fa63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.824+0000 7f6e9affd640 0 client.29423 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.824+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/763779325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 0x55f165fa5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.824+0000 7f6ec41fd740 1 -- 172.21.15.70:0/763779325 >> 172.21.15.70:0/763779325 conn(0x55f165f99d30 msgr2=0x55f165f98fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.824+0000 7f6ec41fd740 1 -- 172.21.15.70:0/763779325 shutdown_connections 2024-06-04T05:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.824+0000 7f6ec41fd740 1 -- 172.21.15.70:0/763779325 wait complete. 2024-06-04T05:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.824+0000 7f6ec41fd740 20 client.29423 trim_cache size 0 max 16384 2024-06-04T05:17:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.848+0000 7f6ec41fd740 1 Processor -- start 2024-06-04T05:17:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.848+0000 7f6ec41fd740 1 -- start start 2024-06-04T05:17:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.849+0000 7f6ec41fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 0x55f165f9f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.849+0000 7f6ec41fd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165f9e140 0x55f165fc0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.849+0000 7f6ec41fd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165fa8a00 0x55f165fc0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.849+0000 7f6ec41fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea800e860 con 0x55f165f9d820 2024-06-04T05:17:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.849+0000 7f6ec41fd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea8010400 con 0x55f165fa8a00 2024-06-04T05:17:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.849+0000 7f6ebb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 0x55f165f9f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.849+0000 7f6eba7fc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165f9e140 0x55f165fc0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.849+0000 7f6ebb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 0x55f165f9f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33256/0 (socket says 172.21.15.70:33256) 2024-06-04T05:17:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.849+0000 7f6ebb7fe640 1 -- 172.21.15.70:0/2950761694 learned_addr learned my addr 172.21.15.70:0/2950761694 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.849+0000 7f6ec41fd740 1 -- 172.21.15.70:0/2950761694 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea800ef90 con 0x55f165f9e140 2024-06-04T05:17:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.849+0000 7f6ebb7fe640 1 -- 172.21.15.70:0/2950761694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165f9e140 msgr2=0x55f165fc0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.849+0000 7f6ebb7fe640 1 --2- 172.21.15.70:0/2950761694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165f9e140 0x55f165fc0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.849+0000 7f6ebb7fe640 1 -- 172.21.15.70:0/2950761694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165fa8a00 msgr2=0x55f165fc0b90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:17:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.849+0000 7f6ebb7fe640 1 --2- 172.21.15.70:0/2950761694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165fa8a00 0x55f165fc0b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.849+0000 7f6ebb7fe640 1 -- 172.21.15.70:0/2950761694 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f165fa0790 con 0x55f165f9d820 2024-06-04T05:17:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.849+0000 7f6eba7fc640 1 --2- 172.21.15.70:0/2950761694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165f9e140 0x55f165fc0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.850+0000 7f6ebb7fe640 1 --2- 172.21.15.70:0/2950761694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 0x55f165f9f8e0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4011730 tx=0x7f6eb4009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a12b7985d5cda73 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:40.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.850+0000 7f6ea17fa640 1 -- 172.21.15.70:0/2950761694 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb401f040 con 0x55f165f9d820 2024-06-04T05:17:40.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.850+0000 7f6ea17fa640 1 -- 172.21.15.70:0/2950761694 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6eb401a440 con 0x55f165f9d820 2024-06-04T05:17:40.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.850+0000 7f6ea17fa640 1 -- 172.21.15.70:0/2950761694 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4010690 con 0x55f165f9d820 2024-06-04T05:17:40.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.851+0000 7f6ec41fd740 1 -- 172.21.15.70:0/2950761694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 msgr2=0x55f165f9f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:40.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.851+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/2950761694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 0x55f165f9f8e0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4011730 tx=0x7f6eb4009f30 comp rx=0 tx=0).stop 2024-06-04T05:17:40.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.851+0000 7f6ec41fd740 1 -- 172.21.15.70:0/2950761694 shutdown_connections 2024-06-04T05:17:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.851+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/2950761694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165fa8a00 0x55f165fc0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.852+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/2950761694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165f9e140 0x55f165fc0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.852+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/2950761694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165f9d820 0x55f165f9f8e0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.852+0000 7f6ec41fd740 1 -- 172.21.15.70:0/2950761694 >> 172.21.15.70:0/2950761694 conn(0x55f165f99d30 msgr2=0x55f165f9f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.852+0000 7f6ec41fd740 1 -- 172.21.15.70:0/2950761694 shutdown_connections 2024-06-04T05:17:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.852+0000 7f6ec41fd740 1 -- 172.21.15.70:0/2950761694 wait complete. 2024-06-04T05:17:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.852+0000 7f6ec41fd740 1 -- start start 2024-06-04T05:17:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.852+0000 7f6ec41fd740 1 Processor -- start 2024-06-04T05:17:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.853+0000 7f6ec41fd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165f9d820 0x55f165fb0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.853+0000 7f6ec41fd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165f9e140 0x55f165fac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.853+0000 7f6ec41fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165fa8a00 0x55f165fac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.853+0000 7f6ec41fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea800fd30 con 0x55f165fa8a00 2024-06-04T05:17:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.853+0000 7f6ec41fd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea800f660 con 0x55f165f9d820 2024-06-04T05:17:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.853+0000 7f6ec41fd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea800c860 con 0x55f165f9e140 2024-06-04T05:17:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.853+0000 7f6ec41fd740 10 client.0 _ll_register_callbacks cb 0x7f6eb40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.853+0000 7f6ebb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165f9d820 0x55f165fb0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.853+0000 7f6eba7fc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165f9e140 0x55f165fac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.853+0000 7f6ebb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165f9d820 0x55f165fb0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:37246/0 (socket says 172.21.15.70:37246) 2024-06-04T05:17:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.853+0000 7f6ebb7fe640 1 -- 172.21.15.70:0/1011012683 learned_addr learned my addr 172.21.15.70:0/1011012683 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.853+0000 7f6ebb7fe640 1 --2- 172.21.15.70:0/1011012683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165fa8a00 0x55f165fac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:40.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.854+0000 7f6eba7fc640 1 -- 172.21.15.70:0/1011012683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165f9d820 msgr2=0x55f165fb0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:40.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.854+0000 7f6eba7fc640 1 --2- 172.21.15.70:0/1011012683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165f9d820 0x55f165fb0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.854+0000 7f6eba7fc640 1 -- 172.21.15.70:0/1011012683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165fa8a00 msgr2=0x55f165fac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:40.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.854+0000 7f6eba7fc640 1 --2- 172.21.15.70:0/1011012683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165fa8a00 0x55f165fac8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.854+0000 7f6eba7fc640 1 -- 172.21.15.70:0/1011012683 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f165fad0e0 con 0x55f165f9e140 2024-06-04T05:17:40.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.854+0000 7f6eba7fc640 1 --2- 172.21.15.70:0/1011012683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165f9e140 0x55f165fac3a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0009900 tx=0x7f6eb0004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fbfe4225f370249 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:40.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.854+0000 7f6e9affd640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:17:40.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.854+0000 7f6e9affd640 1 -- 172.21.15.70:0/1011012683 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0035070 con 0x55f165f9e140 2024-06-04T05:17:40.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.854+0000 7f6e9affd640 1 -- 172.21.15.70:0/1011012683 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7f6eb0005200 con 0x55f165f9e140 2024-06-04T05:17:40.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.854+0000 7f6ec41fd740 1 -- 172.21.15.70:0/1011012683 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6eb401a440 con 0x55f165f9e140 2024-06-04T05:17:40.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.854+0000 7f6e9affd640 1 -- 172.21.15.70:0/1011012683 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f6eb0052070 con 0x55f165f9e140 2024-06-04T05:17:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.854+0000 7f6ec41fd740 10 client.29578 fetch_fsmap learned FSMap version 59 2024-06-04T05:17:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.854+0000 7f6ec41fd740 1 -- 172.21.15.70:0/1011012683 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55f166023560 con 0x55f165f9e140 2024-06-04T05:17:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.854+0000 7f6e9affd640 1 -- 172.21.15.70:0/1011012683 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6eb401a440 con 0x55f165f9e140 2024-06-04T05:17:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.855+0000 7f6e9affd640 1 -- 172.21.15.70:0/1011012683 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f6eb004c040 con 0x55f165f9e140 2024-06-04T05:17:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.855+0000 7f6ec41fd740 10 client.29578 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:17:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.855+0000 7f6ec41fd740 10 client.29578 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:17:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.855+0000 7f6ec41fd740 20 client.29578 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.855+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/1011012683 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f165fb88a0 0x55f166025d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.855+0000 7f6ec41fd740 4 client.29578 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:17:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.855+0000 7f6ec41fd740 1 -- 172.21.15.70:0/1011012683 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f165f0b360 con 0x55f165fb88a0 2024-06-04T05:17:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.855+0000 7f6ebb7fe640 1 --2- 172.21.15.70:0/1011012683 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f165fb88a0 0x55f166025d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.855+0000 7f6ebb7fe640 1 --2- 172.21.15.70:0/1011012683 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f165fb88a0 0x55f166025d20 crc :-1 s=READY pgs=438 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d71691d8b92ba0d8 server_cookie=a69b46c59e1c3dad in_seq=0 out_seq=0 2024-06-04T05:17:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.855+0000 7f6e9affd640 10 client.29578 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.981+0000 7f6e9affd640 1 -- 172.21.15.70:0/1011012683 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f165f0b360 con 0x55f165fb88a0 2024-06-04T05:17:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.981+0000 7f6e9affd640 10 client.29578 handle_command_reply: tid=0 2024-06-04T05:17:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.981+0000 7f6ec41fd740 1 client.29578 shutdown 2024-06-04T05:17:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.981+0000 7f6ec41fd740 1 -- 172.21.15.70:0/1011012683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165f9e140 msgr2=0x55f165fac3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.981+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/1011012683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165f9e140 0x55f165fac3a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0009900 tx=0x7f6eb0004610 comp rx=0 tx=0).stop 2024-06-04T05:17:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.981+0000 7f6ec41fd740 1 -- 172.21.15.70:0/1011012683 shutdown_connections 2024-06-04T05:17:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.981+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/1011012683 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f165fb88a0 0x55f166025d20 crc :-1 s=READY pgs=438 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.981+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/1011012683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f165fa8a00 0x55f165fac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.981+0000 7f6e9affd640 0 client.29578 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.981+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/1011012683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f165f9e140 0x55f165fac3a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.981+0000 7f6ec41fd740 1 --2- 172.21.15.70:0/1011012683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f165f9d820 0x55f165fb0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.981+0000 7f6ec41fd740 1 -- 172.21.15.70:0/1011012683 >> 172.21.15.70:0/1011012683 conn(0x55f165f99d30 msgr2=0x55f165f9fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.981+0000 7f6ec41fd740 1 -- 172.21.15.70:0/1011012683 shutdown_connections 2024-06-04T05:17:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.981+0000 7f6ec41fd740 1 -- 172.21.15.70:0/1011012683 wait complete. 2024-06-04T05:17:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.981+0000 7f6ec41fd740 20 client.29578 trim_cache size 0 max 16384 2024-06-04T05:17:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:17:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:17:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:17:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.982+0000 7f6ec1845640 1 -- 172.21.15.70:0/2773906498 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ea40b2a90 msgr2=0x7f6ea40b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.982+0000 7f6ec1845640 1 --2- 172.21.15.70:0/2773906498 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ea40b2a90 0x7f6ea40b4f50 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0e00b0 tx=0x7f6eac000bc0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.982+0000 7f6ec1845640 1 -- 172.21.15.70:0/2773906498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ebc104ec0 msgr2=0x7f6ebc076f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.982+0000 7f6ec1845640 1 --2- 172.21.15.70:0/2773906498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ebc104ec0 0x7f6ebc076f60 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f6eac00e9f0 tx=0x7f6eac00eeb0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.982+0000 7f6ec1845640 1 -- 172.21.15.70:0/2773906498 shutdown_connections 2024-06-04T05:17:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.982+0000 7f6ec1845640 1 --2- 172.21.15.70:0/2773906498 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ea40b2a90 0x7f6ea40b4f50 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.982+0000 7f6ec1845640 1 --2- 172.21.15.70:0/2773906498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ebc0779e0 0x7f6ebc1ab920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.982+0000 7f6ec1845640 1 --2- 172.21.15.70:0/2773906498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6ebc109f90 0x7f6ebc0774a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.982+0000 7f6ec1845640 1 --2- 172.21.15.70:0/2773906498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ebc104ec0 0x7f6ebc076f60 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.982+0000 7f6ec1845640 1 -- 172.21.15.70:0/2773906498 >> 172.21.15.70:0/2773906498 conn(0x7f6ebc100700 msgr2=0x7f6ebc101b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.983+0000 7f6ec1845640 1 -- 172.21.15.70:0/2773906498 shutdown_connections 2024-06-04T05:17:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:40.983+0000 7f6ec1845640 1 -- 172.21.15.70:0/2773906498 wait complete. 2024-06-04T05:17:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:41 smithi070 ceph-mon[32631]: pgmap v644: 97 pgs: 97 active+clean; 788 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 294 op/s 2024-06-04T05:17:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:41 smithi112 ceph-mon[36453]: pgmap v644: 97 pgs: 97 active+clean; 788 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 294 op/s 2024-06-04T05:17:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:41 smithi098 ceph-mon[35569]: pgmap v644: 97 pgs: 97 active+clean; 788 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 294 op/s 2024-06-04T05:17:41.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 51670 6.91 MB/sec execute 273 sec latency 86.936 ms 2024-06-04T05:17:42.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.562+0000 7fcfe5772700 1 -- 172.21.15.70:0/4131526723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe0108d50 msgr2=0x7fcfe01070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:42.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.562+0000 7fcfe5772700 1 --2- 172.21.15.70:0/4131526723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe0108d50 0x7fcfe01070a0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4009990 tx=0x7fcfd4009ca0 comp rx=0 tx=0).stop 2024-06-04T05:17:42.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.562+0000 7fcfe5772700 1 -- 172.21.15.70:0/4131526723 shutdown_connections 2024-06-04T05:17:42.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.562+0000 7fcfe5772700 1 --2- 172.21.15.70:0/4131526723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe0107780 0x7fcfe010f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:42.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.562+0000 7fcfe5772700 1 --2- 172.21.15.70:0/4131526723 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcfe01083a0 0x7fcfe0108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:42.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.562+0000 7fcfe5772700 1 --2- 172.21.15.70:0/4131526723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe0108d50 0x7fcfe01070a0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:42.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.562+0000 7fcfe5772700 1 -- 172.21.15.70:0/4131526723 >> 172.21.15.70:0/4131526723 conn(0x7fcfe0078640 msgr2=0x7fcfe00ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:42.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.562+0000 7fcfe5772700 1 -- 172.21.15.70:0/4131526723 shutdown_connections 2024-06-04T05:17:42.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.562+0000 7fcfe5772700 1 -- 172.21.15.70:0/4131526723 wait complete. 2024-06-04T05:17:42.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.563+0000 7fcfe5772700 1 Processor -- start 2024-06-04T05:17:42.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.563+0000 7fcfe5772700 1 -- start start 2024-06-04T05:17:42.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.563+0000 7fcfe5772700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe0107780 0x7fcfe0103130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:42.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.564+0000 7fcfe5772700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe01083a0 0x7fcfe0103670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:42.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.564+0000 7fcfdeffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe0107780 0x7fcfe0103130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:42.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.564+0000 7fcfe5772700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcfe0108d50 0x7fcfe0103bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:42.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.564+0000 7fcfe5772700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe0112510 con 0x7fcfe0107780 2024-06-04T05:17:42.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.564+0000 7fcfe5772700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe0112390 con 0x7fcfe01083a0 2024-06-04T05:17:42.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.564+0000 7fcfdeffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe0107780 0x7fcfe0103130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33302/0 (socket says 172.21.15.70:33302) 2024-06-04T05:17:42.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.564+0000 7fcfe5772700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe0112690 con 0x7fcfe0108d50 2024-06-04T05:17:42.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.564+0000 7fcfdeffd700 1 -- 172.21.15.70:0/3484380128 learned_addr learned my addr 172.21.15.70:0/3484380128 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:42.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.564+0000 7fcfde7fc700 1 --2- 172.21.15.70:0/3484380128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe01083a0 0x7fcfe0103670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:42.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.564+0000 7fcfdf7fe700 1 --2- 172.21.15.70:0/3484380128 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcfe0108d50 0x7fcfe0103bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:42.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.564+0000 7fcfdeffd700 1 -- 172.21.15.70:0/3484380128 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcfe0108d50 msgr2=0x7fcfe0103bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:42.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.564+0000 7fcfdeffd700 1 --2- 172.21.15.70:0/3484380128 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcfe0108d50 0x7fcfe0103bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:42.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.564+0000 7fcfdeffd700 1 -- 172.21.15.70:0/3484380128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe01083a0 msgr2=0x7fcfe0103670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:42.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.564+0000 7fcfdeffd700 1 --2- 172.21.15.70:0/3484380128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe01083a0 0x7fcfe0103670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:42.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.564+0000 7fcfdeffd700 1 -- 172.21.15.70:0/3484380128 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe01a8100 con 0x7fcfe0107780 2024-06-04T05:17:42.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.564+0000 7fcfdf7fe700 1 --2- 172.21.15.70:0/3484380128 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcfe0108d50 0x7fcfe0103bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:17:42.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.564+0000 7fcfdeffd700 1 --2- 172.21.15.70:0/3484380128 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe0107780 0x7fcfe0103130 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fcfd000e9c0 tx=0x7fcfd000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:42.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.565+0000 7fcfc7fff700 1 -- 172.21.15.70:0/3484380128 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd000cbd0 con 0x7fcfe0107780 2024-06-04T05:17:42.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.565+0000 7fcfc7fff700 1 -- 172.21.15.70:0/3484380128 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcfd000cd70 con 0x7fcfe0107780 2024-06-04T05:17:42.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.565+0000 7fcfe5772700 1 -- 172.21.15.70:0/3484380128 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfe01a83f0 con 0x7fcfe0107780 2024-06-04T05:17:42.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.565+0000 7fcfc7fff700 1 -- 172.21.15.70:0/3484380128 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0005540 con 0x7fcfe0107780 2024-06-04T05:17:42.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.565+0000 7fcfe5772700 1 -- 172.21.15.70:0/3484380128 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfe01a8900 con 0x7fcfe0107780 2024-06-04T05:17:42.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.566+0000 7fcfe5772700 1 -- 172.21.15.70:0/3484380128 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfe0101a40 con 0x7fcfe0107780 2024-06-04T05:17:42.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.568+0000 7fcfc7fff700 1 -- 172.21.15.70:0/3484380128 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fcfd0004750 con 0x7fcfe0107780 2024-06-04T05:17:42.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.569+0000 7fcfc7fff700 1 --2- 172.21.15.70:0/3484380128 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcfc80b2b40 0x7fcfc80b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:42.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.569+0000 7fcfde7fc700 1 --2- 172.21.15.70:0/3484380128 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcfc80b2b40 0x7fcfc80b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:42.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.569+0000 7fcfc7fff700 1 -- 172.21.15.70:0/3484380128 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7fcfd00e27b0 con 0x7fcfe0107780 2024-06-04T05:17:42.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.570+0000 7fcfde7fc700 1 --2- 172.21.15.70:0/3484380128 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcfc80b2b40 0x7fcfc80b5000 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400b2e0 tx=0x7fcfd4005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:42.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.576+0000 7fcfc7fff700 1 -- 172.21.15.70:0/3484380128 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcfd00aa160 con 0x7fcfe0107780 2024-06-04T05:17:42.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 51835 6.93 MB/sec execute 274 sec latency 95.159 ms 2024-06-04T05:17:42.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.997+0000 7fcfe5772700 1 -- 172.21.15.70:0/3484380128 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfe002e560 con 0x7fcfe0107780 2024-06-04T05:17:42.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:42.998+0000 7fcfc7fff700 1 -- 172.21.15.70:0/3484380128 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+11932 (secure 0 0 0) 0x7fcfd00a9a60 con 0x7fcfe0107780 2024-06-04T05:17:43.000 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:17:43.000 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":59,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:17:20.010341+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:17:43.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:43.001+0000 7fcfe5772700 1 -- 172.21.15.70:0/3484380128 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcfc80b2b40 msgr2=0x7fcfc80b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:43.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:43.001+0000 7fcfe5772700 1 --2- 172.21.15.70:0/3484380128 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcfc80b2b40 0x7fcfc80b5000 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400b2e0 tx=0x7fcfd4005fb0 comp rx=0 tx=0).stop 2024-06-04T05:17:43.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:43.001+0000 7fcfe5772700 1 -- 172.21.15.70:0/3484380128 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe0107780 msgr2=0x7fcfe0103130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:43.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:43.001+0000 7fcfe5772700 1 --2- 172.21.15.70:0/3484380128 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe0107780 0x7fcfe0103130 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fcfd000e9c0 tx=0x7fcfd000ecd0 comp rx=0 tx=0).stop 2024-06-04T05:17:43.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:43.001+0000 7fcfe5772700 1 -- 172.21.15.70:0/3484380128 shutdown_connections 2024-06-04T05:17:43.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:43.001+0000 7fcfe5772700 1 --2- 172.21.15.70:0/3484380128 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcfc80b2b40 0x7fcfc80b5000 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:43.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:43.001+0000 7fcfe5772700 1 --2- 172.21.15.70:0/3484380128 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcfe0108d50 0x7fcfe0103bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:43.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:43.001+0000 7fcfe5772700 1 --2- 172.21.15.70:0/3484380128 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe0107780 0x7fcfe0103130 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:43.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:43.001+0000 7fcfe5772700 1 --2- 172.21.15.70:0/3484380128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe01083a0 0x7fcfe0103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:43.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:43.001+0000 7fcfe5772700 1 -- 172.21.15.70:0/3484380128 >> 172.21.15.70:0/3484380128 conn(0x7fcfe0078640 msgr2=0x7fcfe01004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:43.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:43.001+0000 7fcfe5772700 1 -- 172.21.15.70:0/3484380128 shutdown_connections 2024-06-04T05:17:43.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:43.001+0000 7fcfe5772700 1 -- 172.21.15.70:0/3484380128 wait complete. 2024-06-04T05:17:43.004 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 59 2024-06-04T05:17:43.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:43 smithi070 ceph-mon[32631]: pgmap v645: 97 pgs: 97 active+clean; 781 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.6 MiB/s wr, 458 op/s 2024-06-04T05:17:43.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:43 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3484380128' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:17:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:43 smithi112 ceph-mon[36453]: pgmap v645: 97 pgs: 97 active+clean; 781 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.6 MiB/s wr, 458 op/s 2024-06-04T05:17:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:43 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3484380128' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:17:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:43 smithi098 ceph-mon[35569]: pgmap v645: 97 pgs: 97 active+clean; 781 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.6 MiB/s wr, 458 op/s 2024-06-04T05:17:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:43 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3484380128' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:17:43.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 52003 6.92 MB/sec execute 275 sec latency 53.706 ms 2024-06-04T05:17:43.934 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:17:44.301 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:17:44.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 52224 6.95 MB/sec execute 276 sec latency 60.753 ms 2024-06-04T05:17:45.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:45 smithi070 ceph-mon[32631]: pgmap v646: 97 pgs: 97 active+clean; 781 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 297 op/s 2024-06-04T05:17:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:45 smithi112 ceph-mon[36453]: pgmap v646: 97 pgs: 97 active+clean; 781 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 297 op/s 2024-06-04T05:17:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:45 smithi098 ceph-mon[35569]: pgmap v646: 97 pgs: 97 active+clean; 781 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 297 op/s 2024-06-04T05:17:45.822 INFO:tasks.workunit.client.0.smithi070.stdout: 10 52425 6.95 MB/sec execute 277 sec latency 52.656 ms 2024-06-04T05:17:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.151+0000 7f59c4bb2700 1 -- 172.21.15.70:0/4149090951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c01035f0 msgr2=0x7f59c0103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.151+0000 7f59c4bb2700 1 --2- 172.21.15.70:0/4149090951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c01035f0 0x7f59c0103a50 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009990 tx=0x7f59b4009ca0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.151+0000 7f59c4bb2700 1 -- 172.21.15.70:0/4149090951 shutdown_connections 2024-06-04T05:17:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.151+0000 7f59c4bb2700 1 --2- 172.21.15.70:0/4149090951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59c0103f90 0x7f59c0110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.151+0000 7f59c4bb2700 1 --2- 172.21.15.70:0/4149090951 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59c0108500 0x7f59c01088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.151+0000 7f59c4bb2700 1 --2- 172.21.15.70:0/4149090951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c01035f0 0x7f59c0103a50 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.151+0000 7f59c4bb2700 1 -- 172.21.15.70:0/4149090951 >> 172.21.15.70:0/4149090951 conn(0x7f59c007a6a0 msgr2=0x7f59c007aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.151+0000 7f59c4bb2700 1 -- 172.21.15.70:0/4149090951 shutdown_connections 2024-06-04T05:17:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.151+0000 7f59c4bb2700 1 -- 172.21.15.70:0/4149090951 wait complete. 2024-06-04T05:17:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.152+0000 7f59c4bb2700 1 Processor -- start 2024-06-04T05:17:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.152+0000 7f59c4bb2700 1 -- start start 2024-06-04T05:17:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.152+0000 7f59c4bb2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59c01035f0 0x7f59c019f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.152+0000 7f59c4bb2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59c0103f90 0x7f59c019fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.152+0000 7f59c4bb2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c0108500 0x7f59c0199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.152+0000 7f59c4bb2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f59c0113690 con 0x7f59c0108500 2024-06-04T05:17:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.152+0000 7f59c4bb2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59c0113510 con 0x7f59c0103f90 2024-06-04T05:17:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.152+0000 7f59c4bb2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f59c0113810 con 0x7f59c01035f0 2024-06-04T05:17:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.153+0000 7f59be59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59c01035f0 0x7f59c019f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.153+0000 7f59be59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59c01035f0 0x7f59c019f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39474/0 (socket says 172.21.15.70:39474) 2024-06-04T05:17:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.153+0000 7f59be59c700 1 -- 172.21.15.70:0/3869785391 learned_addr learned my addr 172.21.15.70:0/3869785391 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.153+0000 7f59bed9d700 1 --2- 172.21.15.70:0/3869785391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c0108500 0x7f59c0199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.153+0000 7f59bdd9b700 1 --2- 172.21.15.70:0/3869785391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59c0103f90 0x7f59c019fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.153+0000 7f59be59c700 1 -- 172.21.15.70:0/3869785391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59c0103f90 msgr2=0x7f59c019fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.153+0000 7f59be59c700 1 --2- 172.21.15.70:0/3869785391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59c0103f90 0x7f59c019fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.153+0000 7f59be59c700 1 -- 172.21.15.70:0/3869785391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c0108500 msgr2=0x7f59c0199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.153+0000 7f59be59c700 1 --2- 172.21.15.70:0/3869785391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c0108500 0x7f59c0199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.153+0000 7f59be59c700 1 -- 172.21.15.70:0/3869785391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c019a0d0 con 0x7f59c01035f0 2024-06-04T05:17:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.153+0000 7f59be59c700 1 --2- 172.21.15.70:0/3869785391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59c01035f0 0x7f59c019f7b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f59b000e970 tx=0x7f59b000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.153+0000 7f59af7fe700 1 -- 172.21.15.70:0/3869785391 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b0013d00 con 0x7f59c01035f0 2024-06-04T05:17:46.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.153+0000 7f59af7fe700 1 -- 172.21.15.70:0/3869785391 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f59b00044b0 con 0x7f59c01035f0 2024-06-04T05:17:46.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.154+0000 7f59c4bb2700 1 -- 172.21.15.70:0/3869785391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59c019a3c0 con 0x7f59c01035f0 2024-06-04T05:17:46.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.154+0000 7f59af7fe700 1 -- 172.21.15.70:0/3869785391 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b000c4b0 con 0x7f59c01035f0 2024-06-04T05:17:46.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.154+0000 7f59c4bb2700 1 -- 172.21.15.70:0/3869785391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59c0075770 con 0x7f59c01035f0 2024-06-04T05:17:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.157+0000 7f59af7fe700 1 -- 172.21.15.70:0/3869785391 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f59b0022020 con 0x7f59c01035f0 2024-06-04T05:17:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.158+0000 7f59af7fe700 1 --2- 172.21.15.70:0/3869785391 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f59a80b2a70 0x7f59a80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.158+0000 7f59bdd9b700 1 --2- 172.21.15.70:0/3869785391 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f59a80b2a70 0x7f59a80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.158+0000 7f59af7fe700 1 -- 172.21.15.70:0/3869785391 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7f59b00e6490 con 0x7f59c01035f0 2024-06-04T05:17:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.158+0000 7f59cf447b80 1 Processor -- start 2024-06-04T05:17:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.159+0000 7f59cf447b80 1 -- start start 2024-06-04T05:17:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.159+0000 7f59bdd9b700 1 --2- 172.21.15.70:0/3869785391 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f59a80b2a70 0x7f59a80b4f30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f59c007ad80 tx=0x7f59b4005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.159+0000 7f59cf447b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6fdb30 0x55e4fe6fdf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.159+0000 7f59cf447b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f80b0 0x55e4fe6f8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.159+0000 7f59cf447b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6f8a70 0x55e4fe704890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.159+0000 7f59cf447b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e4fe64dd70 con 0x55e4fe6f80b0 2024-06-04T05:17:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.159+0000 7f59cf447b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e4fe63d910 con 0x55e4fe6fdb30 2024-06-04T05:17:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.159+0000 7f59cf447b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e4fe641950 con 0x55e4fe6f8a70 2024-06-04T05:17:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.159+0000 7f59be59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6fdb30 0x55e4fe6fdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.159+0000 7f59bdd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f80b0 0x55e4fe6f8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.159+0000 7f59be59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6fdb30 0x55e4fe6fdf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:51752/0 (socket says 172.21.15.70:51752) 2024-06-04T05:17:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.159+0000 7f59be59c700 1 -- 172.21.15.70:0/1738936723 learned_addr learned my addr 172.21.15.70:0/1738936723 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.159+0000 7f59bed9d700 1 --2- 172.21.15.70:0/1738936723 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6f8a70 0x55e4fe704890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.160+0000 7f59bdd9b700 1 -- 172.21.15.70:0/1738936723 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6f8a70 msgr2=0x55e4fe704890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.160+0000 7f59bdd9b700 1 --2- 172.21.15.70:0/1738936723 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6f8a70 0x55e4fe704890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.160+0000 7f59bdd9b700 1 -- 172.21.15.70:0/1738936723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6fdb30 msgr2=0x55e4fe6fdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.160+0000 7f59bdd9b700 1 --2- 172.21.15.70:0/1738936723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6fdb30 0x55e4fe6fdf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.160+0000 7f59bdd9b700 1 -- 172.21.15.70:0/1738936723 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4fe5e6fb0 con 0x55e4fe6f80b0 2024-06-04T05:17:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.160+0000 7f59bed9d700 1 --2- 172.21.15.70:0/1738936723 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6f8a70 0x55e4fe704890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:46.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.160+0000 7f59be59c700 1 --2- 172.21.15.70:0/1738936723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6fdb30 0x55e4fe6fdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.160+0000 7f59bdd9b700 1 --2- 172.21.15.70:0/1738936723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f80b0 0x55e4fe6f8530 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f59b400b650 tx=0x7f59b401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f898dc0614dd354 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.161+0000 7f59acff9700 1 -- 172.21.15.70:0/1738936723 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b400b6b0 con 0x55e4fe6f80b0 2024-06-04T05:17:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.161+0000 7f59acff9700 1 -- 172.21.15.70:0/1738936723 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f59b400bce0 con 0x55e4fe6f80b0 2024-06-04T05:17:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.161+0000 7f59acff9700 1 -- 172.21.15.70:0/1738936723 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b4025cb0 con 0x55e4fe6f80b0 2024-06-04T05:17:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.161+0000 7f59cf447b80 1 -- 172.21.15.70:0/1738936723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f80b0 msgr2=0x55e4fe6f8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.161+0000 7f59cf447b80 1 --2- 172.21.15.70:0/1738936723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f80b0 0x55e4fe6f8530 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f59b400b650 tx=0x7f59b401f000 comp rx=0 tx=0).stop 2024-06-04T05:17:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.162+0000 7f59cf447b80 1 -- 172.21.15.70:0/1738936723 shutdown_connections 2024-06-04T05:17:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.162+0000 7f59cf447b80 1 --2- 172.21.15.70:0/1738936723 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6f8a70 0x55e4fe704890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.162+0000 7f59cf447b80 1 --2- 172.21.15.70:0/1738936723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6fdb30 0x55e4fe6fdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.162+0000 7f59cf447b80 1 --2- 172.21.15.70:0/1738936723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f80b0 0x55e4fe6f8530 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.162+0000 7f59cf447b80 1 -- 172.21.15.70:0/1738936723 >> 172.21.15.70:0/1738936723 conn(0x55e4fe61b240 msgr2=0x55e4fe6ea3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.162+0000 7f59cf447b80 1 -- 172.21.15.70:0/1738936723 shutdown_connections 2024-06-04T05:17:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.162+0000 7f59cf447b80 1 -- 172.21.15.70:0/1738936723 wait complete. 2024-06-04T05:17:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.162+0000 7f59cf447b80 1 -- start start 2024-06-04T05:17:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.162+0000 7f59cf447b80 1 Processor -- start 2024-06-04T05:17:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59cf447b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6f80b0 0x55e4fe704510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59cf447b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f8a70 0x55e4fe701810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59cf447b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 0x55e4fe701d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59cf447b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e4fe543fa0 con 0x55e4fe6f8a70 2024-06-04T05:17:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59cf447b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e4fe5a5da0 con 0x55e4fe6f80b0 2024-06-04T05:17:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59cf447b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e4fe704f30 con 0x55e4fe6fdb30 2024-06-04T05:17:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59cf447b80 10 client.0 _ll_register_callbacks cb 0x55e4fe5d8950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59be59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6f80b0 0x55e4fe704510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59bed9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 0x55e4fe701d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59bdd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f8a70 0x55e4fe701810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59bed9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 0x55e4fe701d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39480/0 (socket says 172.21.15.70:39480) 2024-06-04T05:17:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59be59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6f80b0 0x55e4fe704510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:51766/0 (socket says 172.21.15.70:51766) 2024-06-04T05:17:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59bed9d700 1 -- 172.21.15.70:0/1330598415 learned_addr learned my addr 172.21.15.70:0/1330598415 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59bed9d700 1 -- 172.21.15.70:0/1330598415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6f80b0 msgr2=0x55e4fe704510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59bed9d700 1 --2- 172.21.15.70:0/1330598415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6f80b0 0x55e4fe704510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59bed9d700 1 -- 172.21.15.70:0/1330598415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f8a70 msgr2=0x55e4fe701810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59bed9d700 1 --2- 172.21.15.70:0/1330598415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f8a70 0x55e4fe701810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.163+0000 7f59bed9d700 1 -- 172.21.15.70:0/1330598415 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4fe58d8d0 con 0x55e4fe6fdb30 2024-06-04T05:17:46.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.164+0000 7f59bed9d700 1 --2- 172.21.15.70:0/1330598415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 0x55e4fe701d50 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f59b8011f20 tx=0x7f59b8014600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c5dbc012a4f88df server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:46.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.164+0000 7f59a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:17:46.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.164+0000 7f59a5ffb700 1 -- 172.21.15.70:0/1330598415 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b8004bb0 con 0x55e4fe6fdb30 2024-06-04T05:17:46.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.164+0000 7f59cf447b80 1 -- 172.21.15.70:0/1330598415 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59b400bce0 con 0x55e4fe6fdb30 2024-06-04T05:17:46.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.164+0000 7f59a5ffb700 1 -- 172.21.15.70:0/1330598415 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7f59b8004d50 con 0x55e4fe6fdb30 2024-06-04T05:17:46.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.164+0000 7f59a5ffb700 1 -- 172.21.15.70:0/1330598415 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f59b8014b90 con 0x55e4fe6fdb30 2024-06-04T05:17:46.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.164+0000 7f59cf447b80 10 client.29596 fetch_fsmap learned FSMap version 59 2024-06-04T05:17:46.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.164+0000 7f59cf447b80 1 -- 172.21.15.70:0/1330598415 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55e4fe5f7760 con 0x55e4fe6fdb30 2024-06-04T05:17:46.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.164+0000 7f59a5ffb700 1 -- 172.21.15.70:0/1330598415 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f59b400bce0 con 0x55e4fe6fdb30 2024-06-04T05:17:46.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.165+0000 7f59a5ffb700 1 -- 172.21.15.70:0/1330598415 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f59b80043c0 con 0x55e4fe6fdb30 2024-06-04T05:17:46.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.165+0000 7f59cf447b80 10 client.29596 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:17:46.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.165+0000 7f59cf447b80 10 client.29596 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:17:46.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.165+0000 7f59cf447b80 20 client.29596 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:46.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.165+0000 7f59cf447b80 1 --2- 172.21.15.70:0/1330598415 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55e4fe711af0 0x55e4fe713ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.165+0000 7f59cf447b80 4 client.29596 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:17:46.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.165+0000 7f59cf447b80 1 -- 172.21.15.70:0/1330598415 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e4fe64fcc0 con 0x55e4fe711af0 2024-06-04T05:17:46.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.165+0000 7f59be59c700 1 --2- 172.21.15.70:0/1330598415 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55e4fe711af0 0x55e4fe713ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.165+0000 7f59be59c700 1 --2- 172.21.15.70:0/1330598415 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55e4fe711af0 0x55e4fe713ee0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c8741d48dc8fee8f server_cookie=7c0f667d3de7a047 in_seq=0 out_seq=0 2024-06-04T05:17:46.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.166+0000 7f59a5ffb700 10 client.29596 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:17:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.166+0000 7f59a5ffb700 1 -- 172.21.15.70:0/1330598415 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e4fe64fcc0 con 0x55e4fe711af0 2024-06-04T05:17:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.166+0000 7f59a5ffb700 10 client.29596 handle_command_reply: tid=0 2024-06-04T05:17:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.166+0000 7f59cf447b80 1 client.29596 shutdown 2024-06-04T05:17:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.167+0000 7f59cf447b80 1 -- 172.21.15.70:0/1330598415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 msgr2=0x55e4fe701d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.167+0000 7f59cf447b80 1 --2- 172.21.15.70:0/1330598415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 0x55e4fe701d50 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f59b8011f20 tx=0x7f59b8014600 comp rx=0 tx=0).stop 2024-06-04T05:17:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.167+0000 7f59cf447b80 1 -- 172.21.15.70:0/1330598415 shutdown_connections 2024-06-04T05:17:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.167+0000 7f59cf447b80 1 --2- 172.21.15.70:0/1330598415 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55e4fe711af0 0x55e4fe713ee0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.167+0000 7f59a5ffb700 0 client.29596 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:17:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.167+0000 7f59cf447b80 1 --2- 172.21.15.70:0/1330598415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 0x55e4fe701d50 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.167+0000 7f59cf447b80 1 --2- 172.21.15.70:0/1330598415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6f80b0 0x55e4fe704510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.167+0000 7f59cf447b80 1 --2- 172.21.15.70:0/1330598415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f8a70 0x55e4fe701810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.167+0000 7f59cf447b80 1 -- 172.21.15.70:0/1330598415 >> 172.21.15.70:0/1330598415 conn(0x55e4fe61b240 msgr2=0x55e4fe6e9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.167+0000 7f59cf447b80 1 -- 172.21.15.70:0/1330598415 shutdown_connections 2024-06-04T05:17:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.167+0000 7f59cf447b80 1 -- 172.21.15.70:0/1330598415 wait complete. 2024-06-04T05:17:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.167+0000 7f59cf447b80 20 client.29596 trim_cache size 0 max 16384 2024-06-04T05:17:46.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.191+0000 7f59cf447b80 1 Processor -- start 2024-06-04T05:17:46.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.191+0000 7f59cf447b80 1 -- start start 2024-06-04T05:17:46.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.192+0000 7f59cf447b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6f80b0 0x55e4fe702460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.192+0000 7f59cf447b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f8a70 0x55e4fe701d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.192+0000 7f59cf447b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 0x55e4fe6f7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.192+0000 7f59cf447b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f59c01abcf0 con 0x55e4fe6f8a70 2024-06-04T05:17:46.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.192+0000 7f59cf447b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59c01ad890 con 0x55e4fe6f80b0 2024-06-04T05:17:46.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.192+0000 7f59cf447b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f59c01ac420 con 0x55e4fe6fdb30 2024-06-04T05:17:46.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.192+0000 7f59bdd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f8a70 0x55e4fe701d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.192+0000 7f59bdd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f8a70 0x55e4fe701d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42412/0 (socket says 172.21.15.70:42412) 2024-06-04T05:17:46.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.192+0000 7f59bdd9b700 1 -- 172.21.15.70:0/692929010 learned_addr learned my addr 172.21.15.70:0/692929010 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:46.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.192+0000 7f59be59c700 1 --2- 172.21.15.70:0/692929010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6f80b0 0x55e4fe702460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.192+0000 7f59bed9d700 1 --2- 172.21.15.70:0/692929010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 0x55e4fe6f7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.192+0000 7f59be59c700 1 -- 172.21.15.70:0/692929010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 msgr2=0x55e4fe6f7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:46.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.192+0000 7f59be59c700 1 --2- 172.21.15.70:0/692929010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 0x55e4fe6f7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.192+0000 7f59be59c700 1 -- 172.21.15.70:0/692929010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f8a70 msgr2=0x55e4fe701d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:46.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.192+0000 7f59be59c700 1 --2- 172.21.15.70:0/692929010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f8a70 0x55e4fe701d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.192+0000 7f59be59c700 1 -- 172.21.15.70:0/692929010 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4fe6ffd80 con 0x55e4fe6f80b0 2024-06-04T05:17:46.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.193+0000 7f59be59c700 1 --2- 172.21.15.70:0/692929010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6f80b0 0x55e4fe702460 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f59b0009230 tx=0x7f59b0015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2e4d509af2d2c1e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:46.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.193+0000 7f59a7fff700 1 -- 172.21.15.70:0/692929010 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b00adf50 con 0x55e4fe6f80b0 2024-06-04T05:17:46.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.193+0000 7f59a7fff700 1 -- 172.21.15.70:0/692929010 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f59b00ad850 con 0x55e4fe6f80b0 2024-06-04T05:17:46.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.193+0000 7f59a7fff700 1 -- 172.21.15.70:0/692929010 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b00adab0 con 0x55e4fe6f80b0 2024-06-04T05:17:46.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.194+0000 7f59cf447b80 1 -- 172.21.15.70:0/692929010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6f80b0 msgr2=0x55e4fe702460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:46.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.194+0000 7f59cf447b80 1 --2- 172.21.15.70:0/692929010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6f80b0 0x55e4fe702460 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f59b0009230 tx=0x7f59b0015040 comp rx=0 tx=0).stop 2024-06-04T05:17:46.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.194+0000 7f59cf447b80 1 -- 172.21.15.70:0/692929010 shutdown_connections 2024-06-04T05:17:46.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.194+0000 7f59cf447b80 1 --2- 172.21.15.70:0/692929010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 0x55e4fe6f7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.194+0000 7f59cf447b80 1 --2- 172.21.15.70:0/692929010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6f80b0 0x55e4fe702460 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.194+0000 7f59cf447b80 1 --2- 172.21.15.70:0/692929010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f8a70 0x55e4fe701d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.194+0000 7f59cf447b80 1 -- 172.21.15.70:0/692929010 >> 172.21.15.70:0/692929010 conn(0x55e4fe61b240 msgr2=0x55e4fe6fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:46.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.194+0000 7f59cf447b80 1 -- 172.21.15.70:0/692929010 shutdown_connections 2024-06-04T05:17:46.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.194+0000 7f59cf447b80 1 -- 172.21.15.70:0/692929010 wait complete. 2024-06-04T05:17:46.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.194+0000 7f59cf447b80 1 -- start start 2024-06-04T05:17:46.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.195+0000 7f59cf447b80 1 Processor -- start 2024-06-04T05:17:46.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.195+0000 7f59cf447b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f80b0 0x55e4fe70a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.195+0000 7f59cf447b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6f8a70 0x55e4fe718da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.195+0000 7f59cf447b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 0x55e4fe7192e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.195+0000 7f59be59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f80b0 0x55e4fe70a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.195+0000 7f59be59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f80b0 0x55e4fe70a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42414/0 (socket says 172.21.15.70:42414) 2024-06-04T05:17:46.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.195+0000 7f59be59c700 1 -- 172.21.15.70:0/1257446690 learned_addr learned my addr 172.21.15.70:0/1257446690 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:46.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.195+0000 7f59cf447b80 1 -- 172.21.15.70:0/1257446690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f59c01ad1c0 con 0x55e4fe6f80b0 2024-06-04T05:17:46.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.195+0000 7f59cf447b80 1 -- 172.21.15.70:0/1257446690 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59c01acaf0 con 0x55e4fe6f8a70 2024-06-04T05:17:46.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.195+0000 7f59cf447b80 1 -- 172.21.15.70:0/1257446690 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f59c01a9cb0 con 0x55e4fe6fdb30 2024-06-04T05:17:46.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.195+0000 7f59cf447b80 10 client.0 _ll_register_callbacks cb 0x55e4fe5d8950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:46.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.195+0000 7f59bed9d700 1 --2- 172.21.15.70:0/1257446690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 0x55e4fe7192e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.195+0000 7f59bdd9b700 1 --2- 172.21.15.70:0/1257446690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6f8a70 0x55e4fe718da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.196+0000 7f59bed9d700 1 -- 172.21.15.70:0/1257446690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6f8a70 msgr2=0x55e4fe718da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:46.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.196+0000 7f59bed9d700 1 --2- 172.21.15.70:0/1257446690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6f8a70 0x55e4fe718da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.196+0000 7f59bed9d700 1 -- 172.21.15.70:0/1257446690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f80b0 msgr2=0x55e4fe70a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.196+0000 7f59bed9d700 1 --2- 172.21.15.70:0/1257446690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f80b0 0x55e4fe70a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.196+0000 7f59bed9d700 1 -- 172.21.15.70:0/1257446690 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4fe719820 con 0x55e4fe6fdb30 2024-06-04T05:17:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.196+0000 7f59be59c700 1 --2- 172.21.15.70:0/1257446690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f80b0 0x55e4fe70a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.196+0000 7f59bed9d700 1 --2- 172.21.15.70:0/1257446690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 0x55e4fe7192e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f59b8012010 tx=0x7f59b8007ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58b01774bee90864 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.196+0000 7f59a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:17:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.196+0000 7f59a5ffb700 1 -- 172.21.15.70:0/1257446690 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b802a070 con 0x55e4fe6fdb30 2024-06-04T05:17:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.196+0000 7f59a5ffb700 1 -- 172.21.15.70:0/1257446690 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7f59b8014e00 con 0x55e4fe6fdb30 2024-06-04T05:17:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.196+0000 7f59cf447b80 1 -- 172.21.15.70:0/1257446690 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59b00ad850 con 0x55e4fe6fdb30 2024-06-04T05:17:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.197+0000 7f59a5ffb700 1 -- 172.21.15.70:0/1257446690 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f59b8004bf0 con 0x55e4fe6fdb30 2024-06-04T05:17:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.197+0000 7f59cf447b80 10 client.29602 fetch_fsmap learned FSMap version 59 2024-06-04T05:17:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.197+0000 7f59cf447b80 1 -- 172.21.15.70:0/1257446690 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55e4fe709f80 con 0x55e4fe6fdb30 2024-06-04T05:17:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.197+0000 7f59a5ffb700 1 -- 172.21.15.70:0/1257446690 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f59b802b040 con 0x55e4fe6fdb30 2024-06-04T05:17:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.197+0000 7f59a5ffb700 1 -- 172.21.15.70:0/1257446690 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f59b8037cc0 con 0x55e4fe6fdb30 2024-06-04T05:17:46.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.197+0000 7f59cf447b80 10 client.29602 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:17:46.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.197+0000 7f59cf447b80 10 client.29602 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:17:46.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.197+0000 7f59cf447b80 20 client.29602 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:46.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.197+0000 7f59cf447b80 1 --2- 172.21.15.70:0/1257446690 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55e4fe711af0 0x55e4fe77eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:46.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.197+0000 7f59cf447b80 4 client.29602 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:17:46.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.197+0000 7f59cf447b80 1 -- 172.21.15.70:0/1257446690 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f59b00ad850 con 0x55e4fe711af0 2024-06-04T05:17:46.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.197+0000 7f59be59c700 1 --2- 172.21.15.70:0/1257446690 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55e4fe711af0 0x55e4fe77eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:46.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.198+0000 7f59a5ffb700 10 client.29602 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:17:46.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.198+0000 7f59be59c700 1 --2- 172.21.15.70:0/1257446690 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55e4fe711af0 0x55e4fe77eab0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=343d6edc897d9a1f server_cookie=252974eaf4d0b659 in_seq=0 out_seq=0 2024-06-04T05:17:46.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.198+0000 7f59a5ffb700 1 -- 172.21.15.70:0/1257446690 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f59b00ad850 con 0x55e4fe711af0 2024-06-04T05:17:46.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.198+0000 7f59a5ffb700 10 client.29602 handle_command_reply: tid=0 2024-06-04T05:17:46.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.198+0000 7f59cf447b80 1 client.29602 shutdown 2024-06-04T05:17:46.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.198+0000 7f59cf447b80 1 -- 172.21.15.70:0/1257446690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 msgr2=0x55e4fe7192e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:46.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.198+0000 7f59cf447b80 1 --2- 172.21.15.70:0/1257446690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 0x55e4fe7192e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f59b8012010 tx=0x7f59b8007ba0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.198+0000 7f59cf447b80 1 -- 172.21.15.70:0/1257446690 shutdown_connections 2024-06-04T05:17:46.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.198+0000 7f59cf447b80 1 --2- 172.21.15.70:0/1257446690 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55e4fe711af0 0x55e4fe77eab0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.198+0000 7f59cf447b80 1 --2- 172.21.15.70:0/1257446690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e4fe6fdb30 0x55e4fe7192e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.198+0000 7f59cf447b80 1 --2- 172.21.15.70:0/1257446690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4fe6f80b0 0x55e4fe70a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.198+0000 7f59a5ffb700 0 client.29602 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:17:46.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.198+0000 7f59cf447b80 1 --2- 172.21.15.70:0/1257446690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4fe6f8a70 0x55e4fe718da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.199+0000 7f59cf447b80 1 -- 172.21.15.70:0/1257446690 >> 172.21.15.70:0/1257446690 conn(0x55e4fe61b240 msgr2=0x55e4fe6f7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:46.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.199+0000 7f59cf447b80 1 -- 172.21.15.70:0/1257446690 shutdown_connections 2024-06-04T05:17:46.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.199+0000 7f59cf447b80 1 -- 172.21.15.70:0/1257446690 wait complete. 2024-06-04T05:17:46.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.199+0000 7f59cf447b80 20 client.29602 trim_cache size 0 max 16384 2024-06-04T05:17:46.204 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:17:46.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.200+0000 7f59c4bb2700 1 -- 172.21.15.70:0/3869785391 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f59a80b2a70 msgr2=0x7f59a80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:46.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.200+0000 7f59c4bb2700 1 --2- 172.21.15.70:0/3869785391 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f59a80b2a70 0x7f59a80b4f30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f59c007ad80 tx=0x7f59b4005fb0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.200+0000 7f59c4bb2700 1 -- 172.21.15.70:0/3869785391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59c01035f0 msgr2=0x7f59c019f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:46.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.200+0000 7f59c4bb2700 1 --2- 172.21.15.70:0/3869785391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59c01035f0 0x7f59c019f7b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f59b000e970 tx=0x7f59b000ec80 comp rx=0 tx=0).stop 2024-06-04T05:17:46.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.200+0000 7f59c4bb2700 1 -- 172.21.15.70:0/3869785391 shutdown_connections 2024-06-04T05:17:46.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.200+0000 7f59c4bb2700 1 --2- 172.21.15.70:0/3869785391 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f59a80b2a70 0x7f59a80b4f30 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.200+0000 7f59c4bb2700 1 --2- 172.21.15.70:0/3869785391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c0108500 0x7f59c0199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.200+0000 7f59c4bb2700 1 --2- 172.21.15.70:0/3869785391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59c01035f0 0x7f59c019f7b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.200+0000 7f59c4bb2700 1 --2- 172.21.15.70:0/3869785391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59c0103f90 0x7f59c019fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:46.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.200+0000 7f59c4bb2700 1 -- 172.21.15.70:0/3869785391 >> 172.21.15.70:0/3869785391 conn(0x7f59c007a6a0 msgr2=0x7f59c01017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:46.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.200+0000 7f59c4bb2700 1 -- 172.21.15.70:0/3869785391 shutdown_connections 2024-06-04T05:17:46.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:46.200+0000 7f59c4bb2700 1 -- 172.21.15.70:0/3869785391 wait complete. 2024-06-04T05:17:46.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 52580 6.95 MB/sec execute 278 sec latency 75.192 ms 2024-06-04T05:17:46.920 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:17:47.276 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:17:47.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:47 smithi070 ceph-mon[32631]: pgmap v647: 97 pgs: 97 active+clean; 793 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.9 MiB/s wr, 424 op/s 2024-06-04T05:17:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:47 smithi112 ceph-mon[36453]: pgmap v647: 97 pgs: 97 active+clean; 793 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.9 MiB/s wr, 424 op/s 2024-06-04T05:17:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:47 smithi098 ceph-mon[35569]: pgmap v647: 97 pgs: 97 active+clean; 793 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.9 MiB/s wr, 424 op/s 2024-06-04T05:17:47.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 52783 6.99 MB/sec execute 279 sec latency 69.428 ms 2024-06-04T05:17:48.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 53011 7.02 MB/sec execute 280 sec latency 55.219 ms 2024-06-04T05:17:49.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.316+0000 7fef3cb8a700 1 -- 172.21.15.70:0/2972425855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef381035f0 msgr2=0x7fef38103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:49.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.316+0000 7fef3cb8a700 1 --2- 172.21.15.70:0/2972425855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef381035f0 0x7fef38103a50 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fef2c009990 tx=0x7fef2c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.317+0000 7fef3cb8a700 1 -- 172.21.15.70:0/2972425855 shutdown_connections 2024-06-04T05:17:49.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.317+0000 7fef3cb8a700 1 --2- 172.21.15.70:0/2972425855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef38103f90 0x7fef38110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.317+0000 7fef3cb8a700 1 --2- 172.21.15.70:0/2972425855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef38108500 0x7fef381088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.317+0000 7fef3cb8a700 1 --2- 172.21.15.70:0/2972425855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef381035f0 0x7fef38103a50 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.317+0000 7fef3cb8a700 1 -- 172.21.15.70:0/2972425855 >> 172.21.15.70:0/2972425855 conn(0x7fef3807a6a0 msgr2=0x7fef3807aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:49.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.317+0000 7fef3cb8a700 1 -- 172.21.15.70:0/2972425855 shutdown_connections 2024-06-04T05:17:49.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.317+0000 7fef3cb8a700 1 -- 172.21.15.70:0/2972425855 wait complete. 2024-06-04T05:17:49.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.318+0000 7fef3cb8a700 1 Processor -- start 2024-06-04T05:17:49.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.318+0000 7fef3cb8a700 1 -- start start 2024-06-04T05:17:49.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.318+0000 7fef3cb8a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef381035f0 0x7fef381a5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.318+0000 7fef3cb8a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef38103f90 0x7fef381a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.318+0000 7fef3cb8a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef38108500 0x7fef3810f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.318+0000 7fef3cb8a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fef38113690 con 0x7fef381035f0 2024-06-04T05:17:49.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.318+0000 7fef3cb8a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef38113510 con 0x7fef38108500 2024-06-04T05:17:49.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.318+0000 7fef3cb8a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fef38113810 con 0x7fef38103f90 2024-06-04T05:17:49.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.318+0000 7fef3659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef381035f0 0x7fef381a5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.318+0000 7fef36d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef38108500 0x7fef3810f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.318+0000 7fef35d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef38103f90 0x7fef381a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.318+0000 7fef36d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef38108500 0x7fef3810f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:51808/0 (socket says 172.21.15.70:51808) 2024-06-04T05:17:49.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.318+0000 7fef3659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef381035f0 0x7fef381a5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42418/0 (socket says 172.21.15.70:42418) 2024-06-04T05:17:49.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.318+0000 7fef36d9d700 1 -- 172.21.15.70:0/1671680653 learned_addr learned my addr 172.21.15.70:0/1671680653 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:49.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.318+0000 7fef36d9d700 1 -- 172.21.15.70:0/1671680653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef38103f90 msgr2=0x7fef381a6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:49.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.318+0000 7fef36d9d700 1 --2- 172.21.15.70:0/1671680653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef38103f90 0x7fef381a6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.319+0000 7fef36d9d700 1 -- 172.21.15.70:0/1671680653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef381035f0 msgr2=0x7fef381a5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:49.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.319+0000 7fef36d9d700 1 --2- 172.21.15.70:0/1671680653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef381035f0 0x7fef381a5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.319+0000 7fef36d9d700 1 -- 172.21.15.70:0/1671680653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef3810fe10 con 0x7fef38108500 2024-06-04T05:17:49.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.319+0000 7fef3659c700 1 --2- 172.21.15.70:0/1671680653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef381035f0 0x7fef381a5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:49.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.319+0000 7fef36d9d700 1 --2- 172.21.15.70:0/1671680653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef38108500 0x7fef3810f640 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fef3000c0f0 tx=0x7fef3000c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:49.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.319+0000 7fef277fe700 1 -- 172.21.15.70:0/1671680653 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef30018020 con 0x7fef38108500 2024-06-04T05:17:49.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.319+0000 7fef3cb8a700 1 -- 172.21.15.70:0/1671680653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef38110100 con 0x7fef38108500 2024-06-04T05:17:49.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.319+0000 7fef3cb8a700 1 -- 172.21.15.70:0/1671680653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef381acc30 con 0x7fef38108500 2024-06-04T05:17:49.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.319+0000 7fef277fe700 1 -- 172.21.15.70:0/1671680653 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef3000ca00 con 0x7fef38108500 2024-06-04T05:17:49.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.319+0000 7fef277fe700 1 -- 172.21.15.70:0/1671680653 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef300045b0 con 0x7fef38108500 2024-06-04T05:17:49.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.322+0000 7fef277fe700 1 -- 172.21.15.70:0/1671680653 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fef30012050 con 0x7fef38108500 2024-06-04T05:17:49.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.323+0000 7fef277fe700 1 --2- 172.21.15.70:0/1671680653 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fef200b2a70 0x7fef200b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.323+0000 7fef3659c700 1 --2- 172.21.15.70:0/1671680653 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fef200b2a70 0x7fef200b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.323+0000 7fef277fe700 1 -- 172.21.15.70:0/1671680653 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7fef300e20a0 con 0x7fef38108500 2024-06-04T05:17:49.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.324+0000 7fef3659c700 1 --2- 172.21.15.70:0/1671680653 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fef200b2a70 0x7fef200b4f30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fef381047d0 tx=0x7fef28009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:49.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.324+0000 7fef4741fb80 1 Processor -- start 2024-06-04T05:17:49.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.324+0000 7fef4741fb80 1 -- start start 2024-06-04T05:17:49.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.324+0000 7fef4741fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3e860 0x55dc4ec3ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.324+0000 7fef4741fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec3f210 0x55dc4ec42940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.324+0000 7fef4741fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec42f60 0x55dc4ec433a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.324+0000 7fef4741fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dc4eb9dc90 con 0x55dc4ec3f210 2024-06-04T05:17:49.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.324+0000 7fef4741fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dc4eb8d830 con 0x55dc4ec42f60 2024-06-04T05:17:49.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.324+0000 7fef4741fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55dc4eb91870 con 0x55dc4ec3e860 2024-06-04T05:17:49.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.325+0000 7fef3659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3e860 0x55dc4ec3ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.325+0000 7fef3659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3e860 0x55dc4ec3ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39526/0 (socket says 172.21.15.70:39526) 2024-06-04T05:17:49.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.325+0000 7fef3659c700 1 -- 172.21.15.70:0/2459909853 learned_addr learned my addr 172.21.15.70:0/2459909853 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:49.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.325+0000 7fef35d9b700 1 --2- 172.21.15.70:0/2459909853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec3f210 0x55dc4ec42940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.325+0000 7fef36d9d700 1 --2- 172.21.15.70:0/2459909853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec42f60 0x55dc4ec433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.325+0000 7fef3659c700 1 -- 172.21.15.70:0/2459909853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec42f60 msgr2=0x55dc4ec433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:49.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.325+0000 7fef3659c700 1 --2- 172.21.15.70:0/2459909853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec42f60 0x55dc4ec433a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.325+0000 7fef3659c700 1 -- 172.21.15.70:0/2459909853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec3f210 msgr2=0x55dc4ec42940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:49.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.325+0000 7fef3659c700 1 --2- 172.21.15.70:0/2459909853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec3f210 0x55dc4ec42940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.325+0000 7fef3659c700 1 -- 172.21.15.70:0/2459909853 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc4eadd8d0 con 0x55dc4ec3e860 2024-06-04T05:17:49.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.325+0000 7fef3659c700 1 --2- 172.21.15.70:0/2459909853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3e860 0x55dc4ec3ec40 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fef2800e720 tx=0x7fef28014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd5840032447bf0c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:49.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.325+0000 7fef24ff9700 1 -- 172.21.15.70:0/2459909853 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef28026600 con 0x55dc4ec3e860 2024-06-04T05:17:49.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.326+0000 7fef24ff9700 1 -- 172.21.15.70:0/2459909853 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef28026c30 con 0x55dc4ec3e860 2024-06-04T05:17:49.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.326+0000 7fef24ff9700 1 -- 172.21.15.70:0/2459909853 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2801ec10 con 0x55dc4ec3e860 2024-06-04T05:17:49.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.326+0000 7fef4741fb80 1 -- 172.21.15.70:0/2459909853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3e860 msgr2=0x55dc4ec3ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:49.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.326+0000 7fef4741fb80 1 --2- 172.21.15.70:0/2459909853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3e860 0x55dc4ec3ec40 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fef2800e720 tx=0x7fef28014000 comp rx=0 tx=0).stop 2024-06-04T05:17:49.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.327+0000 7fef4741fb80 1 -- 172.21.15.70:0/2459909853 shutdown_connections 2024-06-04T05:17:49.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.327+0000 7fef4741fb80 1 --2- 172.21.15.70:0/2459909853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec42f60 0x55dc4ec433a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.327+0000 7fef4741fb80 1 --2- 172.21.15.70:0/2459909853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3e860 0x55dc4ec3ec40 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.327+0000 7fef4741fb80 1 --2- 172.21.15.70:0/2459909853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec3f210 0x55dc4ec42940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.327+0000 7fef4741fb80 1 -- 172.21.15.70:0/2459909853 >> 172.21.15.70:0/2459909853 conn(0x55dc4eb6c080 msgr2=0x55dc4ec38e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:49.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.327+0000 7fef4741fb80 1 -- 172.21.15.70:0/2459909853 shutdown_connections 2024-06-04T05:17:49.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.327+0000 7fef4741fb80 1 -- 172.21.15.70:0/2459909853 wait complete. 2024-06-04T05:17:49.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.327+0000 7fef4741fb80 1 -- start start 2024-06-04T05:17:49.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.327+0000 7fef4741fb80 1 Processor -- start 2024-06-04T05:17:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.328+0000 7fef4741fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 0x55dc4ec4a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.328+0000 7fef4741fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3f210 0x55dc4ec476e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.328+0000 7fef4741fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec42f60 0x55dc4ec47c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.328+0000 7fef4741fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dc4ea93fa0 con 0x55dc4ec42f60 2024-06-04T05:17:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.328+0000 7fef4741fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dc4eb2f7c0 con 0x55dc4ec3e860 2024-06-04T05:17:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.328+0000 7fef4741fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55dc4ec4b120 con 0x55dc4ec3f210 2024-06-04T05:17:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.328+0000 7fef3659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 0x55dc4ec4a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.328+0000 7fef4741fb80 10 client.0 _ll_register_callbacks cb 0x55dc4eb28920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.328+0000 7fef3659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 0x55dc4ec4a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:51822/0 (socket says 172.21.15.70:51822) 2024-06-04T05:17:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.328+0000 7fef3659c700 1 -- 172.21.15.70:0/1578708173 learned_addr learned my addr 172.21.15.70:0/1578708173 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.328+0000 7fef35d9b700 1 --2- 172.21.15.70:0/1578708173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3f210 0x55dc4ec476e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.328+0000 7fef36d9d700 1 --2- 172.21.15.70:0/1578708173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec42f60 0x55dc4ec47c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.329+0000 7fef35d9b700 1 -- 172.21.15.70:0/1578708173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 msgr2=0x55dc4ec4a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.329+0000 7fef35d9b700 1 --2- 172.21.15.70:0/1578708173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 0x55dc4ec4a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.329+0000 7fef35d9b700 1 -- 172.21.15.70:0/1578708173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec42f60 msgr2=0x55dc4ec47c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.329+0000 7fef35d9b700 1 --2- 172.21.15.70:0/1578708173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec42f60 0x55dc4ec47c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.329+0000 7fef35d9b700 1 -- 172.21.15.70:0/1578708173 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc4eb2c690 con 0x55dc4ec3f210 2024-06-04T05:17:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.329+0000 7fef35d9b700 1 --2- 172.21.15.70:0/1578708173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3f210 0x55dc4ec476e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fef2c00b650 tx=0x7fef2c005800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a087208a0cb8b7be server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.329+0000 7fef1dffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:17:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.329+0000 7fef1dffb700 1 -- 172.21.15.70:0/1578708173 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2c029060 con 0x55dc4ec3f210 2024-06-04T05:17:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.329+0000 7fef1dffb700 1 -- 172.21.15.70:0/1578708173 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7fef2c003c60 con 0x55dc4ec3f210 2024-06-04T05:17:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.329+0000 7fef4741fb80 1 -- 172.21.15.70:0/1578708173 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef28026c30 con 0x55dc4ec3f210 2024-06-04T05:17:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.329+0000 7fef1dffb700 1 -- 172.21.15.70:0/1578708173 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fef2c003e60 con 0x55dc4ec3f210 2024-06-04T05:17:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.330+0000 7fef4741fb80 10 client.29614 fetch_fsmap learned FSMap version 59 2024-06-04T05:17:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.330+0000 7fef4741fb80 1 -- 172.21.15.70:0/1578708173 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55dc4ec558e0 con 0x55dc4ec3f210 2024-06-04T05:17:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.330+0000 7fef1dffb700 1 -- 172.21.15.70:0/1578708173 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef28026c30 con 0x55dc4ec3f210 2024-06-04T05:17:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.330+0000 7fef1dffb700 1 -- 172.21.15.70:0/1578708173 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7fef2c00f800 con 0x55dc4ec3f210 2024-06-04T05:17:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.330+0000 7fef4741fb80 10 client.29614 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:17:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.330+0000 7fef4741fb80 10 client.29614 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:17:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.330+0000 7fef4741fb80 20 client.29614 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.330+0000 7fef4741fb80 1 --2- 172.21.15.70:0/1578708173 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55dc4ec57f20 0x55dc4ec5a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.330+0000 7fef4741fb80 4 client.29614 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:17:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.330+0000 7fef4741fb80 1 -- 172.21.15.70:0/1578708173 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc4ec0a9e0 con 0x55dc4ec57f20 2024-06-04T05:17:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.330+0000 7fef3659c700 1 --2- 172.21.15.70:0/1578708173 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55dc4ec57f20 0x55dc4ec5a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.331+0000 7fef1dffb700 10 client.29614 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:17:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.331+0000 7fef3659c700 1 --2- 172.21.15.70:0/1578708173 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55dc4ec57f20 0x55dc4ec5a310 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bc6110941f8d96d9 server_cookie=7c45c8eca1ff233a in_seq=0 out_seq=0 2024-06-04T05:17:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.332+0000 7fef1dffb700 1 -- 172.21.15.70:0/1578708173 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55dc4ec0a9e0 con 0x55dc4ec57f20 2024-06-04T05:17:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.332+0000 7fef1dffb700 10 client.29614 handle_command_reply: tid=0 2024-06-04T05:17:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.332+0000 7fef4741fb80 1 client.29614 shutdown 2024-06-04T05:17:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.332+0000 7fef4741fb80 1 -- 172.21.15.70:0/1578708173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3f210 msgr2=0x55dc4ec476e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.332+0000 7fef4741fb80 1 --2- 172.21.15.70:0/1578708173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3f210 0x55dc4ec476e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fef2c00b650 tx=0x7fef2c005800 comp rx=0 tx=0).stop 2024-06-04T05:17:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.332+0000 7fef4741fb80 1 -- 172.21.15.70:0/1578708173 shutdown_connections 2024-06-04T05:17:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.332+0000 7fef4741fb80 1 --2- 172.21.15.70:0/1578708173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec42f60 0x55dc4ec47c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.332+0000 7fef4741fb80 1 --2- 172.21.15.70:0/1578708173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 0x55dc4ec4a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.332+0000 7fef4741fb80 1 --2- 172.21.15.70:0/1578708173 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55dc4ec57f20 0x55dc4ec5a310 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.332+0000 7fef1dffb700 0 client.29614 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:17:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.332+0000 7fef4741fb80 1 --2- 172.21.15.70:0/1578708173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3f210 0x55dc4ec476e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.332+0000 7fef4741fb80 1 -- 172.21.15.70:0/1578708173 >> 172.21.15.70:0/1578708173 conn(0x55dc4eb6c080 msgr2=0x55dc4ec38510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.332+0000 7fef4741fb80 1 -- 172.21.15.70:0/1578708173 shutdown_connections 2024-06-04T05:17:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.332+0000 7fef4741fb80 1 -- 172.21.15.70:0/1578708173 wait complete. 2024-06-04T05:17:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.332+0000 7fef4741fb80 20 client.29614 trim_cache size 0 max 16384 2024-06-04T05:17:49.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.361+0000 7fef4741fb80 1 Processor -- start 2024-06-04T05:17:49.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.361+0000 7fef4741fb80 1 -- start start 2024-06-04T05:17:49.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.361+0000 7fef4741fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 0x55dc4ec41fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.361+0000 7fef4741fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec3f210 0x55dc4ec38c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.361+0000 7fef4741fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec42f60 0x55dc4ec391d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.362+0000 7fef4741fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fef1800d6d0 con 0x55dc4ec3f210 2024-06-04T05:17:49.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.362+0000 7fef4741fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef1800f270 con 0x55dc4ec3e860 2024-06-04T05:17:49.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.362+0000 7fef4741fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fef1800de00 con 0x55dc4ec42f60 2024-06-04T05:17:49.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.362+0000 7fef36d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec42f60 0x55dc4ec391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.362+0000 7fef36d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec42f60 0x55dc4ec391d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39558/0 (socket says 172.21.15.70:39558) 2024-06-04T05:17:49.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.362+0000 7fef36d9d700 1 -- 172.21.15.70:0/3102644196 learned_addr learned my addr 172.21.15.70:0/3102644196 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:49.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.362+0000 7fef3659c700 1 --2- 172.21.15.70:0/3102644196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 0x55dc4ec41fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.362+0000 7fef35d9b700 1 --2- 172.21.15.70:0/3102644196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec3f210 0x55dc4ec38c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.362+0000 7fef36d9d700 1 -- 172.21.15.70:0/3102644196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 msgr2=0x55dc4ec41fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:49.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.362+0000 7fef36d9d700 1 --2- 172.21.15.70:0/3102644196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 0x55dc4ec41fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.362+0000 7fef36d9d700 1 -- 172.21.15.70:0/3102644196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec3f210 msgr2=0x55dc4ec38c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:49.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.362+0000 7fef36d9d700 1 --2- 172.21.15.70:0/3102644196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec3f210 0x55dc4ec38c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.362+0000 7fef36d9d700 1 -- 172.21.15.70:0/3102644196 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc4ec474b0 con 0x55dc4ec42f60 2024-06-04T05:17:49.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.363+0000 7fef3659c700 1 --2- 172.21.15.70:0/3102644196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 0x55dc4ec41fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:49.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.363+0000 7fef36d9d700 1 --2- 172.21.15.70:0/3102644196 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec42f60 0x55dc4ec391d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fef300a5400 tx=0x7fef300a8fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65fe78aa334f9001 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:49.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.364+0000 7fef1ffff700 1 -- 172.21.15.70:0/3102644196 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef300ad730 con 0x55dc4ec42f60 2024-06-04T05:17:49.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.364+0000 7fef1ffff700 1 -- 172.21.15.70:0/3102644196 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef300b2150 con 0x55dc4ec42f60 2024-06-04T05:17:49.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.364+0000 7fef1ffff700 1 -- 172.21.15.70:0/3102644196 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef300a9870 con 0x55dc4ec42f60 2024-06-04T05:17:49.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.366+0000 7fef4741fb80 1 -- 172.21.15.70:0/3102644196 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec42f60 msgr2=0x55dc4ec391d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:49.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.366+0000 7fef4741fb80 1 --2- 172.21.15.70:0/3102644196 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec42f60 0x55dc4ec391d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fef300a5400 tx=0x7fef300a8fb0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.366+0000 7fef4741fb80 1 -- 172.21.15.70:0/3102644196 shutdown_connections 2024-06-04T05:17:49.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.366+0000 7fef4741fb80 1 --2- 172.21.15.70:0/3102644196 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec42f60 0x55dc4ec391d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.366+0000 7fef4741fb80 1 --2- 172.21.15.70:0/3102644196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 0x55dc4ec41fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.366+0000 7fef4741fb80 1 --2- 172.21.15.70:0/3102644196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec3f210 0x55dc4ec38c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.366+0000 7fef4741fb80 1 -- 172.21.15.70:0/3102644196 >> 172.21.15.70:0/3102644196 conn(0x55dc4eb6c080 msgr2=0x55dc4ec418b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:49.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.366+0000 7fef4741fb80 1 -- 172.21.15.70:0/3102644196 shutdown_connections 2024-06-04T05:17:49.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.366+0000 7fef4741fb80 1 -- 172.21.15.70:0/3102644196 wait complete. 2024-06-04T05:17:49.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.367+0000 7fef4741fb80 1 -- start start 2024-06-04T05:17:49.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.367+0000 7fef4741fb80 1 Processor -- start 2024-06-04T05:17:49.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.367+0000 7fef4741fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 0x55dc4ec50870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.367+0000 7fef4741fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3f210 0x55dc4ec4dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.367+0000 7fef4741fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec42f60 0x55dc4ec4e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.368+0000 7fef4741fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fef1800eba0 con 0x55dc4ec42f60 2024-06-04T05:17:49.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.368+0000 7fef4741fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef1800e4d0 con 0x55dc4ec3e860 2024-06-04T05:17:49.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.368+0000 7fef4741fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fef1800b690 con 0x55dc4ec3f210 2024-06-04T05:17:49.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.368+0000 7fef4741fb80 10 client.0 _ll_register_callbacks cb 0x55dc4eb28920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:49.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.368+0000 7fef3659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 0x55dc4ec50870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.368+0000 7fef3659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 0x55dc4ec50870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:51844/0 (socket says 172.21.15.70:51844) 2024-06-04T05:17:49.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.368+0000 7fef3659c700 1 -- 172.21.15.70:0/2904919470 learned_addr learned my addr 172.21.15.70:0/2904919470 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:49.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.368+0000 7fef36d9d700 1 --2- 172.21.15.70:0/2904919470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec42f60 0x55dc4ec4e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.368+0000 7fef35d9b700 1 --2- 172.21.15.70:0/2904919470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3f210 0x55dc4ec4dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.368+0000 7fef3659c700 1 -- 172.21.15.70:0/2904919470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3f210 msgr2=0x55dc4ec4dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:49.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.368+0000 7fef3659c700 1 --2- 172.21.15.70:0/2904919470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3f210 0x55dc4ec4dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.368+0000 7fef3659c700 1 -- 172.21.15.70:0/2904919470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec42f60 msgr2=0x55dc4ec4e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:49.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.368+0000 7fef3659c700 1 --2- 172.21.15.70:0/2904919470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec42f60 0x55dc4ec4e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.368+0000 7fef3659c700 1 -- 172.21.15.70:0/2904919470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc4ec4e6c0 con 0x55dc4ec3e860 2024-06-04T05:17:49.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.368+0000 7fef36d9d700 1 --2- 172.21.15.70:0/2904919470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec42f60 0x55dc4ec4e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:17:49.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.368+0000 7fef3659c700 1 --2- 172.21.15.70:0/2904919470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 0x55dc4ec50870 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x55dc4ec5ab20 tx=0x7fef2800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b2bf45fd8c42e76 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:49.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.369+0000 7fef1dffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:17:49.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.369+0000 7fef1dffb700 1 -- 172.21.15.70:0/2904919470 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2801ea60 con 0x55dc4ec3e860 2024-06-04T05:17:49.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.369+0000 7fef1dffb700 1 -- 172.21.15.70:0/2904919470 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7fef2801ec00 con 0x55dc4ec3e860 2024-06-04T05:17:49.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.369+0000 7fef4741fb80 1 -- 172.21.15.70:0/2904919470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef300b2150 con 0x55dc4ec3e860 2024-06-04T05:17:49.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.369+0000 7fef1dffb700 1 -- 172.21.15.70:0/2904919470 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fef2801ee00 con 0x55dc4ec3e860 2024-06-04T05:17:49.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.370+0000 7fef4741fb80 10 client.29468 fetch_fsmap learned FSMap version 59 2024-06-04T05:17:49.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.370+0000 7fef4741fb80 1 -- 172.21.15.70:0/2904919470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55dc4ecc2840 con 0x55dc4ec3e860 2024-06-04T05:17:49.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.371+0000 7fef1dffb700 1 -- 172.21.15.70:0/2904919470 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef300b2150 con 0x55dc4ec3e860 2024-06-04T05:17:49.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.371+0000 7fef1dffb700 1 -- 172.21.15.70:0/2904919470 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7fef28003980 con 0x55dc4ec3e860 2024-06-04T05:17:49.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.371+0000 7fef4741fb80 10 client.29468 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:17:49.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.371+0000 7fef4741fb80 10 client.29468 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:17:49.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.371+0000 7fef4741fb80 20 client.29468 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:49.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.371+0000 7fef4741fb80 1 --2- 172.21.15.70:0/2904919470 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55dc4ecc2be0 0x55dc4ecc5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:49.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.371+0000 7fef4741fb80 4 client.29468 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:17:49.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.371+0000 7fef4741fb80 1 -- 172.21.15.70:0/2904919470 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dc4eb9fbe0 con 0x55dc4ecc2be0 2024-06-04T05:17:49.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.371+0000 7fef35d9b700 1 --2- 172.21.15.70:0/2904919470 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55dc4ecc2be0 0x55dc4ecc5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:49.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.372+0000 7fef35d9b700 1 --2- 172.21.15.70:0/2904919470 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55dc4ecc2be0 0x55dc4ecc5000 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a9c482c0a1b90879 server_cookie=4b43674b3f58907a in_seq=0 out_seq=0 2024-06-04T05:17:49.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.372+0000 7fef1dffb700 10 client.29468 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:17:49.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.372+0000 7fef1dffb700 1 -- 172.21.15.70:0/2904919470 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dc4eb9fbe0 con 0x55dc4ecc2be0 2024-06-04T05:17:49.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.372+0000 7fef1dffb700 10 client.29468 handle_command_reply: tid=0 2024-06-04T05:17:49.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.372+0000 7fef4741fb80 1 client.29468 shutdown 2024-06-04T05:17:49.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.372+0000 7fef4741fb80 1 -- 172.21.15.70:0/2904919470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 msgr2=0x55dc4ec50870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:49.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.372+0000 7fef4741fb80 1 --2- 172.21.15.70:0/2904919470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 0x55dc4ec50870 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x55dc4ec5ab20 tx=0x7fef2800eeb0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.372+0000 7fef4741fb80 1 -- 172.21.15.70:0/2904919470 shutdown_connections 2024-06-04T05:17:49.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.372+0000 7fef4741fb80 1 --2- 172.21.15.70:0/2904919470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dc4ec42f60 0x55dc4ec4e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.372+0000 7fef4741fb80 1 --2- 172.21.15.70:0/2904919470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc4ec3e860 0x55dc4ec50870 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.372+0000 7fef4741fb80 1 --2- 172.21.15.70:0/2904919470 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55dc4ecc2be0 0x55dc4ecc5000 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.372+0000 7fef4741fb80 1 --2- 172.21.15.70:0/2904919470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55dc4ec3f210 0x55dc4ec4dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.372+0000 7fef4741fb80 1 -- 172.21.15.70:0/2904919470 >> 172.21.15.70:0/2904919470 conn(0x55dc4eb6c080 msgr2=0x55dc4ec42140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:49.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.372+0000 7fef1dffb700 0 client.29468 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:17:49.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.372+0000 7fef4741fb80 1 -- 172.21.15.70:0/2904919470 shutdown_connections 2024-06-04T05:17:49.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.373+0000 7fef4741fb80 1 -- 172.21.15.70:0/2904919470 wait complete. 2024-06-04T05:17:49.377 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:17:49.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.373+0000 7fef4741fb80 20 client.29468 trim_cache size 0 max 16384 2024-06-04T05:17:49.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.373+0000 7fef3cb8a700 1 -- 172.21.15.70:0/1671680653 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fef200b2a70 msgr2=0x7fef200b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:49.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.373+0000 7fef3cb8a700 1 --2- 172.21.15.70:0/1671680653 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fef200b2a70 0x7fef200b4f30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fef381047d0 tx=0x7fef28009260 comp rx=0 tx=0).stop 2024-06-04T05:17:49.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.373+0000 7fef3cb8a700 1 -- 172.21.15.70:0/1671680653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef38108500 msgr2=0x7fef3810f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:49.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.373+0000 7fef3cb8a700 1 --2- 172.21.15.70:0/1671680653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef38108500 0x7fef3810f640 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fef3000c0f0 tx=0x7fef3000c400 comp rx=0 tx=0).stop 2024-06-04T05:17:49.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.373+0000 7fef3cb8a700 1 -- 172.21.15.70:0/1671680653 shutdown_connections 2024-06-04T05:17:49.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.374+0000 7fef3cb8a700 1 --2- 172.21.15.70:0/1671680653 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fef200b2a70 0x7fef200b4f30 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.374+0000 7fef3cb8a700 1 --2- 172.21.15.70:0/1671680653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef38108500 0x7fef3810f640 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.374+0000 7fef3cb8a700 1 --2- 172.21.15.70:0/1671680653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef381035f0 0x7fef381a5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.374+0000 7fef3cb8a700 1 --2- 172.21.15.70:0/1671680653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef38103f90 0x7fef381a6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:49.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.374+0000 7fef3cb8a700 1 -- 172.21.15.70:0/1671680653 >> 172.21.15.70:0/1671680653 conn(0x7fef3807a6a0 msgr2=0x7fef38101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:49.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.374+0000 7fef3cb8a700 1 -- 172.21.15.70:0/1671680653 shutdown_connections 2024-06-04T05:17:49.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:49.374+0000 7fef3cb8a700 1 -- 172.21.15.70:0/1671680653 wait complete. 2024-06-04T05:17:49.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:49 smithi070 ceph-mon[32631]: pgmap v648: 97 pgs: 97 active+clean; 793 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.0 MiB/s wr, 290 op/s 2024-06-04T05:17:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:49 smithi112 ceph-mon[36453]: pgmap v648: 97 pgs: 97 active+clean; 793 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.0 MiB/s wr, 290 op/s 2024-06-04T05:17:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:49 smithi098 ceph-mon[35569]: pgmap v648: 97 pgs: 97 active+clean; 793 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.0 MiB/s wr, 290 op/s 2024-06-04T05:17:49.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 53155 7.03 MB/sec execute 281 sec latency 103.856 ms 2024-06-04T05:17:50.302 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:17:50.650 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:17:50.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 53356 7.06 MB/sec execute 282 sec latency 61.453 ms 2024-06-04T05:17:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:51 smithi112 ceph-mon[36453]: pgmap v649: 97 pgs: 97 active+clean; 793 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.0 MiB/s wr, 290 op/s 2024-06-04T05:17:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:51 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:17:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:51 smithi098 ceph-mon[35569]: pgmap v649: 97 pgs: 97 active+clean; 793 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.0 MiB/s wr, 290 op/s 2024-06-04T05:17:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:51 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:17:51.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:51 smithi070 ceph-mon[32631]: pgmap v649: 97 pgs: 97 active+clean; 793 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.0 MiB/s wr, 290 op/s 2024-06-04T05:17:51.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:51 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:17:51.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 53543 7.07 MB/sec execute 283 sec latency 63.004 ms 2024-06-04T05:17:52.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 53666 7.07 MB/sec execute 284 sec latency 64.351 ms 2024-06-04T05:17:52.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.858+0000 7f05dfafe700 1 -- 172.21.15.70:0/1481338884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05d8108500 msgr2=0x7f05d81088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:52.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.858+0000 7f05dfafe700 1 --2- 172.21.15.70:0/1481338884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05d8108500 0x7f05d81088e0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f05d4009990 tx=0x7f05d4009ca0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.858+0000 7f05dfafe700 1 -- 172.21.15.70:0/1481338884 shutdown_connections 2024-06-04T05:17:52.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.858+0000 7f05dfafe700 1 --2- 172.21.15.70:0/1481338884 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05d8103f90 0x7f05d8110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.858+0000 7f05dfafe700 1 --2- 172.21.15.70:0/1481338884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05d8108500 0x7f05d81088e0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.858+0000 7f05dfafe700 1 --2- 172.21.15.70:0/1481338884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05d81035f0 0x7f05d8103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.858+0000 7f05dfafe700 1 -- 172.21.15.70:0/1481338884 >> 172.21.15.70:0/1481338884 conn(0x7f05d807a6a0 msgr2=0x7f05d807aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:52.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.860+0000 7f05dfafe700 1 -- 172.21.15.70:0/1481338884 shutdown_connections 2024-06-04T05:17:52.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.860+0000 7f05dfafe700 1 -- 172.21.15.70:0/1481338884 wait complete. 2024-06-04T05:17:52.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.861+0000 7f05dfafe700 1 Processor -- start 2024-06-04T05:17:52.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.861+0000 7f05dfafe700 1 -- start start 2024-06-04T05:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.861+0000 7f05dfafe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05d81035f0 0x7f05d8074ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.861+0000 7f05dfafe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05d8103f90 0x7f05d806df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.861+0000 7f05dfafe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05d806e4b0 0x7f05d806e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.861+0000 7f05dd099700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05d8103f90 0x7f05d806df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.861+0000 7f05dd099700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05d8103f90 0x7f05d806df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42492/0 (socket says 172.21.15.70:42492) 2024-06-04T05:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.861+0000 7f05dd89a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05d81035f0 0x7f05d8074ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.861+0000 7f05dd099700 1 -- 172.21.15.70:0/2558123367 learned_addr learned my addr 172.21.15.70:0/2558123367 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.861+0000 7f05dfafe700 1 -- 172.21.15.70:0/2558123367 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f05d80770c0 con 0x7f05d8103f90 2024-06-04T05:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.861+0000 7f05dfafe700 1 -- 172.21.15.70:0/2558123367 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f05d8076f40 con 0x7f05d81035f0 2024-06-04T05:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.861+0000 7f05dfafe700 1 -- 172.21.15.70:0/2558123367 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f05d8077240 con 0x7f05d806e4b0 2024-06-04T05:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.862+0000 7f05de09b700 1 --2- 172.21.15.70:0/2558123367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05d806e4b0 0x7f05d806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.862+0000 7f05dd099700 1 -- 172.21.15.70:0/2558123367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05d806e4b0 msgr2=0x7f05d806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.862+0000 7f05dd099700 1 --2- 172.21.15.70:0/2558123367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05d806e4b0 0x7f05d806e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.862+0000 7f05dd099700 1 -- 172.21.15.70:0/2558123367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05d81035f0 msgr2=0x7f05d8074ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.862+0000 7f05dd099700 1 --2- 172.21.15.70:0/2558123367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05d81035f0 0x7f05d8074ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.862+0000 7f05dd099700 1 -- 172.21.15.70:0/2558123367 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d8072b10 con 0x7f05d8103f90 2024-06-04T05:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.862+0000 7f05dd099700 1 --2- 172.21.15.70:0/2558123367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05d8103f90 0x7f05d806df70 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f05c800e9a0 tx=0x7f05c800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:52.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.862+0000 7f05ceffd700 1 -- 172.21.15.70:0/2558123367 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c800cbd0 con 0x7f05d8103f90 2024-06-04T05:17:52.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.862+0000 7f05ceffd700 1 -- 172.21.15.70:0/2558123367 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05c800cd70 con 0x7f05d8103f90 2024-06-04T05:17:52.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.862+0000 7f05ceffd700 1 -- 172.21.15.70:0/2558123367 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c8018680 con 0x7f05d8103f90 2024-06-04T05:17:52.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.863+0000 7f05dfafe700 1 -- 172.21.15.70:0/2558123367 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05d8072e00 con 0x7f05d8103f90 2024-06-04T05:17:52.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.863+0000 7f05dfafe700 1 -- 172.21.15.70:0/2558123367 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d80732f0 con 0x7f05d8103f90 2024-06-04T05:17:52.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.866+0000 7f05ceffd700 1 -- 172.21.15.70:0/2558123367 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f05c8018820 con 0x7f05d8103f90 2024-06-04T05:17:52.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.867+0000 7f05ea393b80 1 Processor -- start 2024-06-04T05:17:52.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.867+0000 7f05ea393b80 1 -- start start 2024-06-04T05:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.867+0000 7f05ea393b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb921860 0x55c0eb921c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.867+0000 7f05ea393b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb922210 0x55c0eb925940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.868+0000 7f05ea393b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb925f60 0x55c0eb9263a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.868+0000 7f05ea393b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c0eb880c90 con 0x55c0eb922210 2024-06-04T05:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.868+0000 7f05ea393b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c0eb870830 con 0x55c0eb921860 2024-06-04T05:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.868+0000 7f05ea393b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c0eb874870 con 0x55c0eb925f60 2024-06-04T05:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.868+0000 7f05ceffd700 1 --2- 172.21.15.70:0/2558123367 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f05c40b2a70 0x7f05c40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.868+0000 7f05de09b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb921860 0x55c0eb921c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.868+0000 7f05de09b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb921860 0x55c0eb921c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:51870/0 (socket says 172.21.15.70:51870) 2024-06-04T05:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.868+0000 7f05de09b700 1 -- 172.21.15.70:0/1828853480 learned_addr learned my addr 172.21.15.70:0/1828853480 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.868+0000 7f05dd89a700 1 --2- 172.21.15.70:0/1828853480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb922210 0x55c0eb925940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.868+0000 7f05dd099700 1 --2- 172.21.15.70:0/1828853480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb925f60 0x55c0eb9263a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.868+0000 7f05ceffd700 1 -- 172.21.15.70:0/2558123367 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7f05c80e2870 con 0x7f05d8103f90 2024-06-04T05:17:52.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.868+0000 7f05de09b700 1 --2- 172.21.15.70:0/2558123367 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f05c40b2a70 0x7f05c40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.868+0000 7f05dd89a700 1 -- 172.21.15.70:0/1828853480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb925f60 msgr2=0x55c0eb9263a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:52.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.868+0000 7f05dd89a700 1 --2- 172.21.15.70:0/1828853480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb925f60 0x55c0eb9263a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.868+0000 7f05dd89a700 1 -- 172.21.15.70:0/1828853480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb921860 msgr2=0x55c0eb921c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:52.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.868+0000 7f05dd89a700 1 --2- 172.21.15.70:0/1828853480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb921860 0x55c0eb921c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.868+0000 7f05dd89a700 1 -- 172.21.15.70:0/1828853480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0eb7c08d0 con 0x55c0eb922210 2024-06-04T05:17:52.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.869+0000 7f05dd89a700 1 --2- 172.21.15.70:0/1828853480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb922210 0x55c0eb925940 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f05d81046b0 tx=0x7f05d400b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfdef6878e9dad51 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:52.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.869+0000 7f05de09b700 1 --2- 172.21.15.70:0/2558123367 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f05c40b2a70 0x7f05c40b4f30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f05d00060b0 tx=0x7f05d0009680 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:52.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.869+0000 7f05c3fff700 1 -- 172.21.15.70:0/1828853480 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d4027070 con 0x55c0eb922210 2024-06-04T05:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.869+0000 7f05c3fff700 1 -- 172.21.15.70:0/1828853480 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05d4005620 con 0x55c0eb922210 2024-06-04T05:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.869+0000 7f05c3fff700 1 -- 172.21.15.70:0/1828853480 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d4003a00 con 0x55c0eb922210 2024-06-04T05:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.870+0000 7f05ea393b80 1 -- 172.21.15.70:0/1828853480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb922210 msgr2=0x55c0eb925940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.870+0000 7f05ea393b80 1 --2- 172.21.15.70:0/1828853480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb922210 0x55c0eb925940 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f05d81046b0 tx=0x7f05d400b6b0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.870+0000 7f05ea393b80 1 -- 172.21.15.70:0/1828853480 shutdown_connections 2024-06-04T05:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.870+0000 7f05ea393b80 1 --2- 172.21.15.70:0/1828853480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb925f60 0x55c0eb9263a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.870+0000 7f05ea393b80 1 --2- 172.21.15.70:0/1828853480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb921860 0x55c0eb921c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.870+0000 7f05ea393b80 1 --2- 172.21.15.70:0/1828853480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb922210 0x55c0eb925940 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.870+0000 7f05ea393b80 1 -- 172.21.15.70:0/1828853480 >> 172.21.15.70:0/1828853480 conn(0x55c0eb84f080 msgr2=0x55c0eb91be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.870+0000 7f05ea393b80 1 -- 172.21.15.70:0/1828853480 shutdown_connections 2024-06-04T05:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.870+0000 7f05ea393b80 1 -- 172.21.15.70:0/1828853480 wait complete. 2024-06-04T05:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.870+0000 7f05ea393b80 1 -- start start 2024-06-04T05:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.870+0000 7f05ea393b80 1 Processor -- start 2024-06-04T05:17:52.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05ea393b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb921860 0x55c0eb92d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05ea393b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb922210 0x55c0eb92a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05ea393b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb925f60 0x55c0eb92ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05ea393b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c0eb776fa0 con 0x55c0eb921860 2024-06-04T05:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05ea393b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c0eb8127c0 con 0x55c0eb922210 2024-06-04T05:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05ea393b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c0eb92e0c0 con 0x55c0eb925f60 2024-06-04T05:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05dd89a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb921860 0x55c0eb92d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05dd89a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb921860 0x55c0eb92d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42508/0 (socket says 172.21.15.70:42508) 2024-06-04T05:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05dd89a700 1 -- 172.21.15.70:0/1580745096 learned_addr learned my addr 172.21.15.70:0/1580745096 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05ea393b80 10 client.0 _ll_register_callbacks cb 0x55c0eb80b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05de09b700 1 --2- 172.21.15.70:0/1580745096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb925f60 0x55c0eb92ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05dd099700 1 --2- 172.21.15.70:0/1580745096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb922210 0x55c0eb92a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05dd89a700 1 -- 172.21.15.70:0/1580745096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb925f60 msgr2=0x55c0eb92ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05dd89a700 1 --2- 172.21.15.70:0/1580745096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb925f60 0x55c0eb92ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05dd89a700 1 -- 172.21.15.70:0/1580745096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb922210 msgr2=0x55c0eb92a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05dd89a700 1 --2- 172.21.15.70:0/1580745096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb922210 0x55c0eb92a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05dd89a700 1 -- 172.21.15.70:0/1580745096 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0eb80f690 con 0x55c0eb921860 2024-06-04T05:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.871+0000 7f05dd099700 1 --2- 172.21.15.70:0/1580745096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb922210 0x55c0eb92a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.872+0000 7f05dd89a700 1 --2- 172.21.15.70:0/1580745096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb921860 0x55c0eb92d3c0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f05d4025040 tx=0x7f05d400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456ee1b3cf8d6634 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.872+0000 7f05c17fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.872+0000 7f05c17fa700 1 -- 172.21.15.70:0/1580745096 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d4030060 con 0x55c0eb921860 2024-06-04T05:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.872+0000 7f05c17fa700 1 -- 172.21.15.70:0/1580745096 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7f05d4003680 con 0x55c0eb921860 2024-06-04T05:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.872+0000 7f05ea393b80 1 -- 172.21.15.70:0/1580745096 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05d4005620 con 0x55c0eb921860 2024-06-04T05:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.872+0000 7f05c17fa700 1 -- 172.21.15.70:0/1580745096 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f05d4027040 con 0x55c0eb921860 2024-06-04T05:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.872+0000 7f05ea393b80 10 client.20769 fetch_fsmap learned FSMap version 59 2024-06-04T05:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.872+0000 7f05ea393b80 1 -- 172.21.15.70:0/1580745096 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55c0eb940db0 con 0x55c0eb921860 2024-06-04T05:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.872+0000 7f05c17fa700 1 -- 172.21.15.70:0/1580745096 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05d4003d10 con 0x55c0eb921860 2024-06-04T05:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.872+0000 7f05c17fa700 1 -- 172.21.15.70:0/1580745096 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f05d402a070 con 0x55c0eb921860 2024-06-04T05:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.873+0000 7f05ea393b80 10 client.20769 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.873+0000 7f05ea393b80 10 client.20769 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.873+0000 7f05ea393b80 20 client.20769 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.873+0000 7f05ea393b80 1 --2- 172.21.15.70:0/1580745096 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c0eb943280 0x55c0eb945670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.873+0000 7f05ea393b80 4 client.20769 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.873+0000 7f05ea393b80 1 -- 172.21.15.70:0/1580745096 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f05d4005620 con 0x55c0eb943280 2024-06-04T05:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.873+0000 7f05dd099700 1 --2- 172.21.15.70:0/1580745096 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c0eb943280 0x55c0eb945670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.873+0000 7f05dd099700 1 --2- 172.21.15.70:0/1580745096 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c0eb943280 0x55c0eb945670 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=31ba12061702fe0a server_cookie=a00d7b5c2a086df5 in_seq=0 out_seq=0 2024-06-04T05:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.873+0000 7f05c17fa700 10 client.20769 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.874+0000 7f05c17fa700 1 -- 172.21.15.70:0/1580745096 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f05d4005620 con 0x55c0eb943280 2024-06-04T05:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.874+0000 7f05c17fa700 10 client.20769 handle_command_reply: tid=0 2024-06-04T05:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.874+0000 7f05ea393b80 1 client.20769 shutdown 2024-06-04T05:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.874+0000 7f05ea393b80 1 -- 172.21.15.70:0/1580745096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb921860 msgr2=0x55c0eb92d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.874+0000 7f05ea393b80 1 --2- 172.21.15.70:0/1580745096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb921860 0x55c0eb92d3c0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f05d4025040 tx=0x7f05d400bf90 comp rx=0 tx=0).stop 2024-06-04T05:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.874+0000 7f05ea393b80 1 -- 172.21.15.70:0/1580745096 shutdown_connections 2024-06-04T05:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.874+0000 7f05ea393b80 1 --2- 172.21.15.70:0/1580745096 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c0eb943280 0x55c0eb945670 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.874+0000 7f05ea393b80 1 --2- 172.21.15.70:0/1580745096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb925f60 0x55c0eb92ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.874+0000 7f05ea393b80 1 --2- 172.21.15.70:0/1580745096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb921860 0x55c0eb92d3c0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.874+0000 7f05ea393b80 1 --2- 172.21.15.70:0/1580745096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb922210 0x55c0eb92a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.874+0000 7f05c17fa700 0 client.20769 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:17:52.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.874+0000 7f05ea393b80 1 -- 172.21.15.70:0/1580745096 >> 172.21.15.70:0/1580745096 conn(0x55c0eb84f080 msgr2=0x55c0eb91b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:52.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.875+0000 7f05ea393b80 1 -- 172.21.15.70:0/1580745096 shutdown_connections 2024-06-04T05:17:52.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.875+0000 7f05ea393b80 1 -- 172.21.15.70:0/1580745096 wait complete. 2024-06-04T05:17:52.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.875+0000 7f05ea393b80 20 client.20769 trim_cache size 0 max 16384 2024-06-04T05:17:52.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.898+0000 7f05ea393b80 1 Processor -- start 2024-06-04T05:17:52.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.899+0000 7f05ea393b80 1 -- start start 2024-06-04T05:17:52.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.899+0000 7f05ea393b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb921860 0x55c0eb929640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.899+0000 7f05ea393b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb922210 0x55c0eb91bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.899+0000 7f05ea393b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb925f60 0x55c0eb9238b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.899+0000 7f05ea393b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f05d81b48f0 con 0x55c0eb925f60 2024-06-04T05:17:52.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.899+0000 7f05ea393b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f05d81b6490 con 0x55c0eb921860 2024-06-04T05:17:52.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.899+0000 7f05ea393b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f05d81b5020 con 0x55c0eb922210 2024-06-04T05:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.899+0000 7f05de09b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb925f60 0x55c0eb9238b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.899+0000 7f05dd89a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb921860 0x55c0eb929640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.899+0000 7f05de09b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb925f60 0x55c0eb9238b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42520/0 (socket says 172.21.15.70:42520) 2024-06-04T05:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.899+0000 7f05de09b700 1 -- 172.21.15.70:0/3988404696 learned_addr learned my addr 172.21.15.70:0/3988404696 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.900+0000 7f05dd89a700 1 -- 172.21.15.70:0/3988404696 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb922210 msgr2=0x55c0eb91bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.900+0000 7f05dd099700 1 --2- 172.21.15.70:0/3988404696 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb922210 0x55c0eb91bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.900+0000 7f05dd89a700 1 --2- 172.21.15.70:0/3988404696 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb922210 0x55c0eb91bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.900+0000 7f05dd89a700 1 -- 172.21.15.70:0/3988404696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb925f60 msgr2=0x55c0eb9238b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.900+0000 7f05dd89a700 1 --2- 172.21.15.70:0/3988404696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb925f60 0x55c0eb9238b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.900+0000 7f05dd89a700 1 -- 172.21.15.70:0/3988404696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0eb921fb0 con 0x55c0eb921860 2024-06-04T05:17:52.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.900+0000 7f05dd89a700 1 --2- 172.21.15.70:0/3988404696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb921860 0x55c0eb929640 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f05d40098e0 tx=0x7f05d402b4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e49de088dfa30eb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:52.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.901+0000 7f05c37fe700 1 -- 172.21.15.70:0/3988404696 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d4030070 con 0x55c0eb921860 2024-06-04T05:17:52.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.901+0000 7f05c37fe700 1 -- 172.21.15.70:0/3988404696 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05d400f8f0 con 0x55c0eb921860 2024-06-04T05:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.901+0000 7f05c37fe700 1 -- 172.21.15.70:0/3988404696 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d4003db0 con 0x55c0eb921860 2024-06-04T05:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.902+0000 7f05ea393b80 1 -- 172.21.15.70:0/3988404696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb921860 msgr2=0x55c0eb929640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.902+0000 7f05ea393b80 1 --2- 172.21.15.70:0/3988404696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb921860 0x55c0eb929640 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f05d40098e0 tx=0x7f05d402b4b0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.902+0000 7f05ea393b80 1 -- 172.21.15.70:0/3988404696 shutdown_connections 2024-06-04T05:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.902+0000 7f05ea393b80 1 --2- 172.21.15.70:0/3988404696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb925f60 0x55c0eb9238b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.902+0000 7f05ea393b80 1 --2- 172.21.15.70:0/3988404696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb921860 0x55c0eb929640 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.902+0000 7f05ea393b80 1 --2- 172.21.15.70:0/3988404696 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb922210 0x55c0eb91bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.902+0000 7f05ea393b80 1 -- 172.21.15.70:0/3988404696 >> 172.21.15.70:0/3988404696 conn(0x55c0eb84f080 msgr2=0x55c0eb920f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.902+0000 7f05ea393b80 1 -- 172.21.15.70:0/3988404696 shutdown_connections 2024-06-04T05:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.902+0000 7f05ea393b80 1 -- 172.21.15.70:0/3988404696 wait complete. 2024-06-04T05:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.903+0000 7f05ea393b80 1 -- start start 2024-06-04T05:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.903+0000 7f05ea393b80 1 Processor -- start 2024-06-04T05:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05ea393b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb921860 0x55c0eb9489a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05ea393b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb922210 0x55c0eb948ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05ea393b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb925f60 0x55c0eb945f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05ea393b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f05d81b5dc0 con 0x55c0eb922210 2024-06-04T05:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05ea393b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f05d81b56f0 con 0x55c0eb925f60 2024-06-04T05:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05ea393b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f05d8115950 con 0x55c0eb921860 2024-06-04T05:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05dd89a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb921860 0x55c0eb9489a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05ea393b80 10 client.0 _ll_register_callbacks cb 0x55c0eb80b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05dd89a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb921860 0x55c0eb9489a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39630/0 (socket says 172.21.15.70:39630) 2024-06-04T05:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05dd89a700 1 -- 172.21.15.70:0/3981548310 learned_addr learned my addr 172.21.15.70:0/3981548310 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05de09b700 1 --2- 172.21.15.70:0/3981548310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb925f60 0x55c0eb945f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05dd099700 1 --2- 172.21.15.70:0/3981548310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb922210 0x55c0eb948ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05dd89a700 1 -- 172.21.15.70:0/3981548310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb925f60 msgr2=0x55c0eb945f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05dd89a700 1 --2- 172.21.15.70:0/3981548310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb925f60 0x55c0eb945f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05dd89a700 1 -- 172.21.15.70:0/3981548310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb922210 msgr2=0x55c0eb948ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05dd89a700 1 --2- 172.21.15.70:0/3981548310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb922210 0x55c0eb948ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05dd89a700 1 -- 172.21.15.70:0/3981548310 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0eb946780 con 0x55c0eb921860 2024-06-04T05:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.904+0000 7f05dd099700 1 --2- 172.21.15.70:0/3981548310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb922210 0x55c0eb948ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.905+0000 7f05dd89a700 1 --2- 172.21.15.70:0/3981548310 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb921860 0x55c0eb9489a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f05d4009350 tx=0x7f05d4005790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4db20bc7917fd476 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.905+0000 7f05c17fa700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:17:52.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.905+0000 7f05c17fa700 1 -- 172.21.15.70:0/3981548310 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d4030040 con 0x55c0eb921860 2024-06-04T05:17:52.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.905+0000 7f05c17fa700 1 -- 172.21.15.70:0/3981548310 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7f05d4003a90 con 0x55c0eb921860 2024-06-04T05:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.905+0000 7f05ea393b80 1 -- 172.21.15.70:0/3981548310 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c0eb8ed9e0 con 0x55c0eb921860 2024-06-04T05:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.905+0000 7f05c17fa700 1 -- 172.21.15.70:0/3981548310 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f05d400f710 con 0x55c0eb921860 2024-06-04T05:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.905+0000 7f05ea393b80 10 client.29635 fetch_fsmap learned FSMap version 59 2024-06-04T05:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.905+0000 7f05ea393b80 1 -- 172.21.15.70:0/3981548310 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55c0eb9ad720 con 0x55c0eb921860 2024-06-04T05:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.906+0000 7f05c17fa700 1 -- 172.21.15.70:0/3981548310 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55c0eb8ed9e0 con 0x55c0eb921860 2024-06-04T05:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.906+0000 7f05c17fa700 1 -- 172.21.15.70:0/3981548310 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f05d402a040 con 0x55c0eb921860 2024-06-04T05:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.906+0000 7f05ea393b80 10 client.29635 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.906+0000 7f05ea393b80 10 client.29635 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.906+0000 7f05ea393b80 20 client.29635 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.906+0000 7f05ea393b80 1 --2- 172.21.15.70:0/3981548310 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c0eb9adc10 0x55c0eb9b0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.906+0000 7f05ea393b80 4 client.29635 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.906+0000 7f05ea393b80 1 -- 172.21.15.70:0/3981548310 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c0eb882be0 con 0x55c0eb9adc10 2024-06-04T05:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.906+0000 7f05dd099700 1 --2- 172.21.15.70:0/3981548310 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c0eb9adc10 0x55c0eb9b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.906+0000 7f05c17fa700 10 client.29635 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.906+0000 7f05dd099700 1 --2- 172.21.15.70:0/3981548310 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c0eb9adc10 0x55c0eb9b0030 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1369e49fcc9a4bd1 server_cookie=e4e1dc49bd0706a8 in_seq=0 out_seq=0 2024-06-04T05:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.907+0000 7f05c17fa700 1 -- 172.21.15.70:0/3981548310 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c0eb882be0 con 0x55c0eb9adc10 2024-06-04T05:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.907+0000 7f05c17fa700 10 client.29635 handle_command_reply: tid=0 2024-06-04T05:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.907+0000 7f05ea393b80 1 client.29635 shutdown 2024-06-04T05:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.907+0000 7f05ea393b80 1 -- 172.21.15.70:0/3981548310 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb921860 msgr2=0x55c0eb9489a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.907+0000 7f05ea393b80 1 --2- 172.21.15.70:0/3981548310 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb921860 0x55c0eb9489a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f05d4009350 tx=0x7f05d4005790 comp rx=0 tx=0).stop 2024-06-04T05:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.907+0000 7f05ea393b80 1 -- 172.21.15.70:0/3981548310 shutdown_connections 2024-06-04T05:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.907+0000 7f05ea393b80 1 --2- 172.21.15.70:0/3981548310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0eb925f60 0x55c0eb945f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.907+0000 7f05ea393b80 1 --2- 172.21.15.70:0/3981548310 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55c0eb9adc10 0x55c0eb9b0030 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.907+0000 7f05c17fa700 0 client.29635 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.907+0000 7f05ea393b80 1 --2- 172.21.15.70:0/3981548310 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c0eb921860 0x55c0eb9489a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.907+0000 7f05ea393b80 1 --2- 172.21.15.70:0/3981548310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c0eb922210 0x55c0eb948ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.907+0000 7f05ea393b80 1 -- 172.21.15.70:0/3981548310 >> 172.21.15.70:0/3981548310 conn(0x55c0eb84f080 msgr2=0x55c0eb9248b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.907+0000 7f05ea393b80 1 -- 172.21.15.70:0/3981548310 shutdown_connections 2024-06-04T05:17:52.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.907+0000 7f05ea393b80 1 -- 172.21.15.70:0/3981548310 wait complete. 2024-06-04T05:17:52.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.908+0000 7f05ea393b80 20 client.29635 trim_cache size 0 max 16384 2024-06-04T05:17:52.913 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:17:52.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.909+0000 7f05dfafe700 1 -- 172.21.15.70:0/2558123367 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f05c40b2a70 msgr2=0x7f05c40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.909+0000 7f05dfafe700 1 --2- 172.21.15.70:0/2558123367 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f05c40b2a70 0x7f05c40b4f30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f05d00060b0 tx=0x7f05d0009680 comp rx=0 tx=0).stop 2024-06-04T05:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.909+0000 7f05dfafe700 1 -- 172.21.15.70:0/2558123367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05d8103f90 msgr2=0x7f05d806df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.909+0000 7f05dfafe700 1 --2- 172.21.15.70:0/2558123367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05d8103f90 0x7f05d806df70 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f05c800e9a0 tx=0x7f05c800ecb0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.910+0000 7f05dfafe700 1 -- 172.21.15.70:0/2558123367 shutdown_connections 2024-06-04T05:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.910+0000 7f05dfafe700 1 --2- 172.21.15.70:0/2558123367 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f05c40b2a70 0x7f05c40b4f30 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.910+0000 7f05dfafe700 1 --2- 172.21.15.70:0/2558123367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05d806e4b0 0x7f05d806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.910+0000 7f05dfafe700 1 --2- 172.21.15.70:0/2558123367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05d81035f0 0x7f05d8074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.910+0000 7f05dfafe700 1 --2- 172.21.15.70:0/2558123367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05d8103f90 0x7f05d806df70 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.910+0000 7f05dfafe700 1 -- 172.21.15.70:0/2558123367 >> 172.21.15.70:0/2558123367 conn(0x7f05d807a6a0 msgr2=0x7f05d81017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.910+0000 7f05dfafe700 1 -- 172.21.15.70:0/2558123367 shutdown_connections 2024-06-04T05:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:52.910+0000 7f05dfafe700 1 -- 172.21.15.70:0/2558123367 wait complete. 2024-06-04T05:17:53.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:53 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:53 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:53 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:53 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:53 smithi070 ceph-mon[32631]: pgmap v650: 97 pgs: 97 active+clean; 815 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 7.0 MiB/s wr, 401 op/s 2024-06-04T05:17:53.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:53 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:53 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:53 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:53 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:53 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:53 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:53 smithi112 ceph-mon[36453]: pgmap v650: 97 pgs: 97 active+clean; 815 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 7.0 MiB/s wr, 401 op/s 2024-06-04T05:17:53.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:53 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:53 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:53 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:53 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:53 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:53 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:53 smithi098 ceph-mon[35569]: pgmap v650: 97 pgs: 97 active+clean; 815 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 7.0 MiB/s wr, 401 op/s 2024-06-04T05:17:53.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:53 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:53 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:53.752 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:17:53.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 53800 7.08 MB/sec execute 285 sec latency 59.543 ms 2024-06-04T05:17:54.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.126+0000 7fd1bdb59640 1 -- 172.21.15.70:0/3124769277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b8102a30 msgr2=0x7fd1b8102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:54.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.126+0000 7fd1bdb59640 1 --2- 172.21.15.70:0/3124769277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b8102a30 0x7fd1b8102eb0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0009930 tx=0x7fd1a002a420 comp rx=0 tx=0).stop 2024-06-04T05:17:54.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.127+0000 7fd1bdb59640 1 -- 172.21.15.70:0/3124769277 shutdown_connections 2024-06-04T05:17:54.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.127+0000 7fd1bdb59640 1 --2- 172.21.15.70:0/3124769277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b81033f0 0x7fd1b810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.127+0000 7fd1bdb59640 1 --2- 172.21.15.70:0/3124769277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b8102a30 0x7fd1b8102eb0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.127+0000 7fd1bdb59640 1 --2- 172.21.15.70:0/3124769277 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1b8108a40 0x7fd1b8108e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.127+0000 7fd1bdb59640 1 -- 172.21.15.70:0/3124769277 >> 172.21.15.70:0/3124769277 conn(0x7fd1b80fe500 msgr2=0x7fd1b8100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:54.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.127+0000 7fd1bdb59640 1 -- 172.21.15.70:0/3124769277 shutdown_connections 2024-06-04T05:17:54.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.127+0000 7fd1bdb59640 1 -- 172.21.15.70:0/3124769277 wait complete. 2024-06-04T05:17:54.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.127+0000 7fd1bdb59640 1 Processor -- start 2024-06-04T05:17:54.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.127+0000 7fd1bdb59640 1 -- start start 2024-06-04T05:17:54.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1bdb59640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b8102a30 0x7fd1b8074d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1bdb59640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1b81033f0 0x7fd1b8075290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1bdb59640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8108a40 0x7fd1b80757d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1bdb59640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b8112bb0 con 0x7fd1b8102a30 2024-06-04T05:17:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1bdb59640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b8112a30 con 0x7fd1b8108a40 2024-06-04T05:17:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1bdb59640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b8112d30 con 0x7fd1b81033f0 2024-06-04T05:17:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1b77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b8102a30 0x7fd1b8074d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1b77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b8102a30 0x7fd1b8074d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42540/0 (socket says 172.21.15.70:42540) 2024-06-04T05:17:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1b77fe640 1 -- 172.21.15.70:0/2289024445 learned_addr learned my addr 172.21.15.70:0/2289024445 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1affff640 1 --2- 172.21.15.70:0/2289024445 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1b81033f0 0x7fd1b8075290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1b7fff640 1 --2- 172.21.15.70:0/2289024445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8108a40 0x7fd1b80757d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1b77fe640 1 -- 172.21.15.70:0/2289024445 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1b81033f0 msgr2=0x7fd1b8075290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1b77fe640 1 --2- 172.21.15.70:0/2289024445 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1b81033f0 0x7fd1b8075290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1b77fe640 1 -- 172.21.15.70:0/2289024445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8108a40 msgr2=0x7fd1b80757d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1b77fe640 1 --2- 172.21.15.70:0/2289024445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8108a40 0x7fd1b80757d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1b77fe640 1 -- 172.21.15.70:0/2289024445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b81a7650 con 0x7fd1b8102a30 2024-06-04T05:17:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1affff640 1 --2- 172.21.15.70:0/2289024445 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1b81033f0 0x7fd1b8075290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.128+0000 7fd1b77fe640 1 --2- 172.21.15.70:0/2289024445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b8102a30 0x7fd1b8074d50 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800e9f0 tx=0x7fd1a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.129+0000 7fd1b57fa640 1 -- 172.21.15.70:0/2289024445 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a800cd80 con 0x7fd1b8102a30 2024-06-04T05:17:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.129+0000 7fd1bdb59640 1 -- 172.21.15.70:0/2289024445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1b81a7940 con 0x7fd1b8102a30 2024-06-04T05:17:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.129+0000 7fd1b57fa640 1 -- 172.21.15.70:0/2289024445 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd1a80054b0 con 0x7fd1b8102a30 2024-06-04T05:17:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.129+0000 7fd1bdb59640 1 -- 172.21.15.70:0/2289024445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1b81a7e60 con 0x7fd1b8102a30 2024-06-04T05:17:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.129+0000 7fd1b57fa640 1 -- 172.21.15.70:0/2289024445 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8010430 con 0x7fd1b8102a30 2024-06-04T05:17:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.132+0000 7fd1c0870740 1 Processor -- start 2024-06-04T05:17:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.132+0000 7fd1c0870740 1 -- start start 2024-06-04T05:17:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.132+0000 7fd1b57fa640 1 -- 172.21.15.70:0/2289024445 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fd1a80046c0 con 0x7fd1b8102a30 2024-06-04T05:17:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.132+0000 7fd1c0870740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd661820 0x55decd661c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.133+0000 7fd1c0870740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd662140 0x55decd66c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.133+0000 7fd1c0870740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd66ca00 0x55decd66edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.133+0000 7fd1b7fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd661820 0x55decd661c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.133+0000 7fd1b7fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd661820 0x55decd661c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39662/0 (socket says 172.21.15.70:39662) 2024-06-04T05:17:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.133+0000 7fd1b7fff640 1 -- 172.21.15.70:0/2992413184 learned_addr learned my addr 172.21.15.70:0/2992413184 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.133+0000 7fd1c0870740 1 -- 172.21.15.70:0/2992413184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55decd5d3c30 con 0x55decd66ca00 2024-06-04T05:17:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.133+0000 7fd1c0870740 1 -- 172.21.15.70:0/2992413184 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55decd5c37d0 con 0x55decd662140 2024-06-04T05:17:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.133+0000 7fd1c0870740 1 -- 172.21.15.70:0/2992413184 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55decd5c7810 con 0x55decd661820 2024-06-04T05:17:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.133+0000 7fd1affff640 1 --2- 172.21.15.70:0/2992413184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd66ca00 0x55decd66edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.133+0000 7fd1b77fe640 1 --2- 172.21.15.70:0/2992413184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd662140 0x55decd66c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.133+0000 7fd1affff640 1 -- 172.21.15.70:0/2992413184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd661820 msgr2=0x55decd661c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.133+0000 7fd1affff640 1 --2- 172.21.15.70:0/2992413184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd661820 0x55decd661c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.133+0000 7fd1affff640 1 -- 172.21.15.70:0/2992413184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd662140 msgr2=0x55decd66c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.133+0000 7fd1affff640 1 --2- 172.21.15.70:0/2992413184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd662140 0x55decd66c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.133+0000 7fd1affff640 1 -- 172.21.15.70:0/2992413184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55decd2d8210 con 0x55decd66ca00 2024-06-04T05:17:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.133+0000 7fd1b7fff640 1 --2- 172.21.15.70:0/2992413184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd661820 0x55decd661c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.134+0000 7fd1affff640 1 --2- 172.21.15.70:0/2992413184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd66ca00 0x55decd66edf0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0009900 tx=0x7fd1a0032e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7095785007c772da server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.134+0000 7fd1adffb640 1 -- 172.21.15.70:0/2992413184 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0035070 con 0x55decd66ca00 2024-06-04T05:17:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.134+0000 7fd1adffb640 1 -- 172.21.15.70:0/2992413184 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd1a00044b0 con 0x55decd66ca00 2024-06-04T05:17:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.134+0000 7fd1adffb640 1 -- 172.21.15.70:0/2992413184 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0003c10 con 0x55decd66ca00 2024-06-04T05:17:54.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.134+0000 7fd1b57fa640 1 --2- 172.21.15.70:0/2289024445 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd1a40b2a90 0x7fd1a40b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.135+0000 7fd1b7fff640 1 --2- 172.21.15.70:0/2289024445 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd1a40b2a90 0x7fd1a40b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.135+0000 7fd1b57fa640 1 -- 172.21.15.70:0/2289024445 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7fd1a80e29a0 con 0x7fd1b8102a30 2024-06-04T05:17:54.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.135+0000 7fd1b7fff640 1 --2- 172.21.15.70:0/2289024445 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd1a40b2a90 0x7fd1a40b4f50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0011890 tx=0x7fd1b0009500 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:54.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.136+0000 7fd1c0870740 1 -- 172.21.15.70:0/2992413184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd66ca00 msgr2=0x55decd66edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:54.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.136+0000 7fd1c0870740 1 --2- 172.21.15.70:0/2992413184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd66ca00 0x55decd66edf0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0009900 tx=0x7fd1a0032e30 comp rx=0 tx=0).stop 2024-06-04T05:17:54.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.136+0000 7fd1c0870740 1 -- 172.21.15.70:0/2992413184 shutdown_connections 2024-06-04T05:17:54.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.136+0000 7fd1c0870740 1 --2- 172.21.15.70:0/2992413184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd662140 0x55decd66c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.136+0000 7fd1c0870740 1 --2- 172.21.15.70:0/2992413184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd66ca00 0x55decd66edf0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.136+0000 7fd1c0870740 1 --2- 172.21.15.70:0/2992413184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd661820 0x55decd661c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.136+0000 7fd1c0870740 1 -- 172.21.15.70:0/2992413184 >> 172.21.15.70:0/2992413184 conn(0x55decd65dd30 msgr2=0x55decd65e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:54.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.136+0000 7fd1c0870740 1 -- 172.21.15.70:0/2992413184 shutdown_connections 2024-06-04T05:17:54.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.136+0000 7fd1c0870740 1 -- 172.21.15.70:0/2992413184 wait complete. 2024-06-04T05:17:54.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.136+0000 7fd1c0870740 1 -- start start 2024-06-04T05:17:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.136+0000 7fd1c0870740 1 Processor -- start 2024-06-04T05:17:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.137+0000 7fd1c0870740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd661820 0x55decd669aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.137+0000 7fd1c0870740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd662140 0x55decd669fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.137+0000 7fd1c0870740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd66ca00 0x55decd66a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.137+0000 7fd1c0870740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55decd2a9790 con 0x55decd66ca00 2024-06-04T05:17:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.137+0000 7fd1b77fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd661820 0x55decd669aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.137+0000 7fd1b77fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd661820 0x55decd669aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39670/0 (socket says 172.21.15.70:39670) 2024-06-04T05:17:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.137+0000 7fd1b77fe640 1 -- 172.21.15.70:0/1509477887 learned_addr learned my addr 172.21.15.70:0/1509477887 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.137+0000 7fd1affff640 1 --2- 172.21.15.70:0/1509477887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd662140 0x55decd669fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.137+0000 7fd1c0870740 1 -- 172.21.15.70:0/1509477887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55decd57f890 con 0x55decd662140 2024-06-04T05:17:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.137+0000 7fd1c0870740 1 -- 172.21.15.70:0/1509477887 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55decd66f5d0 con 0x55decd661820 2024-06-04T05:17:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.137+0000 7fd1c0870740 10 client.0 _ll_register_callbacks cb 0x55decd2d7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.138+0000 7fd1b7fff640 1 --2- 172.21.15.70:0/1509477887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd66ca00 0x55decd66a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.138+0000 7fd1affff640 1 -- 172.21.15.70:0/1509477887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd661820 msgr2=0x55decd669aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.138+0000 7fd1affff640 1 --2- 172.21.15.70:0/1509477887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd661820 0x55decd669aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.138+0000 7fd1affff640 1 -- 172.21.15.70:0/1509477887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd66ca00 msgr2=0x55decd66a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.138+0000 7fd1affff640 1 --2- 172.21.15.70:0/1509477887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd66ca00 0x55decd66a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.138+0000 7fd1affff640 1 -- 172.21.15.70:0/1509477887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55decd3a4620 con 0x55decd662140 2024-06-04T05:17:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.138+0000 7fd1b7fff640 1 --2- 172.21.15.70:0/1509477887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd66ca00 0x55decd66a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:17:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.138+0000 7fd1affff640 1 --2- 172.21.15.70:0/1509477887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd662140 0x55decd669fe0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x55decd65f150 tx=0x7fd1a0009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edd1fb2f323f6eaa server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.138+0000 7fd196ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:17:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.138+0000 7fd196ffd640 1 -- 172.21.15.70:0/1509477887 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0035040 con 0x55decd662140 2024-06-04T05:17:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.138+0000 7fd1c0870740 1 -- 172.21.15.70:0/1509477887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1a00044b0 con 0x55decd662140 2024-06-04T05:17:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.138+0000 7fd196ffd640 1 -- 172.21.15.70:0/1509477887 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7fd1a0031680 con 0x55decd662140 2024-06-04T05:17:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.139+0000 7fd196ffd640 1 -- 172.21.15.70:0/1509477887 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fd1a0031880 con 0x55decd662140 2024-06-04T05:17:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.139+0000 7fd1c0870740 10 client.29498 fetch_fsmap learned FSMap version 59 2024-06-04T05:17:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.139+0000 7fd1c0870740 1 -- 172.21.15.70:0/1509477887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55decd67b400 con 0x55decd662140 2024-06-04T05:17:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.139+0000 7fd196ffd640 1 -- 172.21.15.70:0/1509477887 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd1a00044b0 con 0x55decd662140 2024-06-04T05:17:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.139+0000 7fd196ffd640 1 -- 172.21.15.70:0/1509477887 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7fd1a00505e0 con 0x55decd662140 2024-06-04T05:17:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.139+0000 7fd1c0870740 10 client.29498 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:17:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.139+0000 7fd1c0870740 10 client.29498 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:17:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.139+0000 7fd1c0870740 20 client.29498 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.139+0000 7fd1c0870740 1 --2- 172.21.15.70:0/1509477887 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55decd67c750 0x55decd67eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.139+0000 7fd1c0870740 4 client.29498 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:17:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.139+0000 7fd1c0870740 1 -- 172.21.15.70:0/1509477887 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55decd5d5b80 con 0x55decd67c750 2024-06-04T05:17:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.139+0000 7fd1b77fe640 1 --2- 172.21.15.70:0/1509477887 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55decd67c750 0x55decd67eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.140+0000 7fd196ffd640 10 client.29498 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.140+0000 7fd1b77fe640 1 --2- 172.21.15.70:0/1509477887 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55decd67c750 0x55decd67eb40 crc :-1 s=READY pgs=439 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e6858518f5d99ad server_cookie=b51c6ada64e779a0 in_seq=0 out_seq=0 2024-06-04T05:17:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.140+0000 7fd196ffd640 1 -- 172.21.15.70:0/1509477887 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55decd5d5b80 con 0x55decd67c750 2024-06-04T05:17:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.140+0000 7fd196ffd640 10 client.29498 handle_command_reply: tid=0 2024-06-04T05:17:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.141+0000 7fd1c0870740 1 client.29498 shutdown 2024-06-04T05:17:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.141+0000 7fd1c0870740 1 -- 172.21.15.70:0/1509477887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd662140 msgr2=0x55decd669fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.141+0000 7fd1c0870740 1 --2- 172.21.15.70:0/1509477887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd662140 0x55decd669fe0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x55decd65f150 tx=0x7fd1a0009680 comp rx=0 tx=0).stop 2024-06-04T05:17:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.141+0000 7fd1c0870740 1 -- 172.21.15.70:0/1509477887 shutdown_connections 2024-06-04T05:17:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.141+0000 7fd1c0870740 1 --2- 172.21.15.70:0/1509477887 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55decd67c750 0x55decd67eb40 crc :-1 s=READY pgs=439 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.141+0000 7fd196ffd640 0 client.29498 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.141+0000 7fd1c0870740 1 --2- 172.21.15.70:0/1509477887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd662140 0x55decd669fe0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.141+0000 7fd1c0870740 1 --2- 172.21.15.70:0/1509477887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd66ca00 0x55decd66a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.141+0000 7fd1c0870740 1 --2- 172.21.15.70:0/1509477887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd661820 0x55decd669aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.141+0000 7fd1c0870740 1 -- 172.21.15.70:0/1509477887 >> 172.21.15.70:0/1509477887 conn(0x55decd65dd30 msgr2=0x55decd65cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.141+0000 7fd1c0870740 1 -- 172.21.15.70:0/1509477887 shutdown_connections 2024-06-04T05:17:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.141+0000 7fd1c0870740 1 -- 172.21.15.70:0/1509477887 wait complete. 2024-06-04T05:17:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.141+0000 7fd1c0870740 20 client.29498 trim_cache size 0 max 16384 2024-06-04T05:17:54.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.168+0000 7fd1c0870740 1 Processor -- start 2024-06-04T05:17:54.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.168+0000 7fd1c0870740 1 -- start start 2024-06-04T05:17:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.169+0000 7fd1c0870740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd661820 0x55decd65d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.169+0000 7fd1c0870740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd662140 0x55decd6693c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.169+0000 7fd1c0870740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd66ca00 0x55decd6780b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.169+0000 7fd1b77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd661820 0x55decd65d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.169+0000 7fd1b77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd661820 0x55decd65d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42562/0 (socket says 172.21.15.70:42562) 2024-06-04T05:17:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.169+0000 7fd1affff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd662140 0x55decd6693c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.169+0000 7fd1affff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd662140 0x55decd6693c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39682/0 (socket says 172.21.15.70:39682) 2024-06-04T05:17:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.169+0000 7fd1affff640 1 -- 172.21.15.70:0/3525528874 learned_addr learned my addr 172.21.15.70:0/3525528874 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.169+0000 7fd1c0870740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b81b0840 con 0x55decd661820 2024-06-04T05:17:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.169+0000 7fd1c0870740 1 -- 172.21.15.70:0/3525528874 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b81b23e0 con 0x55decd66ca00 2024-06-04T05:17:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.169+0000 7fd1c0870740 1 -- 172.21.15.70:0/3525528874 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b81b0f70 con 0x55decd662140 2024-06-04T05:17:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.169+0000 7fd1b7fff640 1 --2- 172.21.15.70:0/3525528874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd66ca00 0x55decd6780b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.170+0000 7fd1b77fe640 1 -- 172.21.15.70:0/3525528874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd662140 msgr2=0x55decd6693c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.170+0000 7fd1b77fe640 1 --2- 172.21.15.70:0/3525528874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd662140 0x55decd6693c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.170+0000 7fd1b77fe640 1 -- 172.21.15.70:0/3525528874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd66ca00 msgr2=0x55decd6780b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.170+0000 7fd1b77fe640 1 --2- 172.21.15.70:0/3525528874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd66ca00 0x55decd6780b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.170+0000 7fd1b77fe640 1 -- 172.21.15.70:0/3525528874 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55decd6594d0 con 0x55decd661820 2024-06-04T05:17:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.170+0000 7fd1b7fff640 1 --2- 172.21.15.70:0/3525528874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd66ca00 0x55decd6780b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.170+0000 7fd1b77fe640 1 --2- 172.21.15.70:0/3525528874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd661820 0x55decd65d730 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800e9c0 tx=0x7fd1a80e4d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e9d800272421a9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.170+0000 7fd1ad7fa640 1 -- 172.21.15.70:0/3525528874 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a80a5ab0 con 0x55decd661820 2024-06-04T05:17:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.170+0000 7fd1ad7fa640 1 -- 172.21.15.70:0/3525528874 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd1a80e4e50 con 0x55decd661820 2024-06-04T05:17:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.170+0000 7fd1ad7fa640 1 -- 172.21.15.70:0/3525528874 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a80adfa0 con 0x55decd661820 2024-06-04T05:17:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.171+0000 7fd1c0870740 1 -- 172.21.15.70:0/3525528874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd661820 msgr2=0x55decd65d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.171+0000 7fd1c0870740 1 --2- 172.21.15.70:0/3525528874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd661820 0x55decd65d730 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800e9c0 tx=0x7fd1a80e4d70 comp rx=0 tx=0).stop 2024-06-04T05:17:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.171+0000 7fd1c0870740 1 -- 172.21.15.70:0/3525528874 shutdown_connections 2024-06-04T05:17:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.171+0000 7fd1c0870740 1 --2- 172.21.15.70:0/3525528874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd66ca00 0x55decd6780b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.171+0000 7fd1c0870740 1 --2- 172.21.15.70:0/3525528874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd662140 0x55decd6693c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.171+0000 7fd1c0870740 1 --2- 172.21.15.70:0/3525528874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd661820 0x55decd65d730 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.171+0000 7fd1c0870740 1 -- 172.21.15.70:0/3525528874 >> 172.21.15.70:0/3525528874 conn(0x55decd65dd30 msgr2=0x55decd663a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.171+0000 7fd1c0870740 1 -- 172.21.15.70:0/3525528874 shutdown_connections 2024-06-04T05:17:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.171+0000 7fd1c0870740 1 -- 172.21.15.70:0/3525528874 wait complete. 2024-06-04T05:17:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.172+0000 7fd1c0870740 1 -- start start 2024-06-04T05:17:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.172+0000 7fd1c0870740 1 Processor -- start 2024-06-04T05:17:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.172+0000 7fd1c0870740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd661820 0x55decd6df440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.172+0000 7fd1c0870740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd662140 0x55decd6df980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.172+0000 7fd1c0870740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd66ca00 0x55decd67f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.173+0000 7fd1c0870740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b81b1d10 con 0x55decd661820 2024-06-04T05:17:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.173+0000 7fd1c0870740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b81b1640 con 0x55decd662140 2024-06-04T05:17:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.173+0000 7fd1c0870740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b81ae840 con 0x55decd66ca00 2024-06-04T05:17:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.173+0000 7fd1c0870740 10 client.0 _ll_register_callbacks cb 0x7fd1b0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.173+0000 7fd1affff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd662140 0x55decd6df980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.173+0000 7fd1b7fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd66ca00 0x55decd67f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.173+0000 7fd1b7fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd66ca00 0x55decd67f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39690/0 (socket says 172.21.15.70:39690) 2024-06-04T05:17:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.173+0000 7fd1b7fff640 1 -- 172.21.15.70:0/41905824 learned_addr learned my addr 172.21.15.70:0/41905824 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.173+0000 7fd1b77fe640 1 --2- 172.21.15.70:0/41905824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd661820 0x55decd6df440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.173+0000 7fd1b7fff640 1 -- 172.21.15.70:0/41905824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd662140 msgr2=0x55decd6df980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.173+0000 7fd1b7fff640 1 --2- 172.21.15.70:0/41905824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd662140 0x55decd6df980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.173+0000 7fd1b7fff640 1 -- 172.21.15.70:0/41905824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd661820 msgr2=0x55decd6df440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.173+0000 7fd1b7fff640 1 --2- 172.21.15.70:0/41905824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd661820 0x55decd6df440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.173+0000 7fd1b7fff640 1 -- 172.21.15.70:0/41905824 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55decd67f960 con 0x55decd66ca00 2024-06-04T05:17:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.173+0000 7fd1b77fe640 1 --2- 172.21.15.70:0/41905824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd661820 0x55decd6df440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.174+0000 7fd1b7fff640 1 --2- 172.21.15.70:0/41905824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd66ca00 0x55decd67f160 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0003f50 tx=0x7fd1b0004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39fddc772b339fd4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.174+0000 7fd196ffd640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:17:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.174+0000 7fd196ffd640 1 -- 172.21.15.70:0/41905824 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b000fe40 con 0x55decd66ca00 2024-06-04T05:17:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.174+0000 7fd196ffd640 1 -- 172.21.15.70:0/41905824 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7fd1b0004250 con 0x55decd66ca00 2024-06-04T05:17:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.174+0000 7fd1c0870740 1 -- 172.21.15.70:0/41905824 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1a80e4e50 con 0x55decd66ca00 2024-06-04T05:17:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.174+0000 7fd196ffd640 1 -- 172.21.15.70:0/41905824 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fd1b0037050 con 0x55decd66ca00 2024-06-04T05:17:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.174+0000 7fd1c0870740 10 client.29653 fetch_fsmap learned FSMap version 59 2024-06-04T05:17:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.174+0000 7fd1c0870740 1 -- 172.21.15.70:0/41905824 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55decd6e6c10 con 0x55decd66ca00 2024-06-04T05:17:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.174+0000 7fd196ffd640 1 -- 172.21.15.70:0/41905824 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd1a80e4e50 con 0x55decd66ca00 2024-06-04T05:17:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.175+0000 7fd196ffd640 1 -- 172.21.15.70:0/41905824 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7fd1b0029a20 con 0x55decd66ca00 2024-06-04T05:17:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.175+0000 7fd1c0870740 10 client.29653 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:17:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.175+0000 7fd1c0870740 10 client.29653 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:17:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.175+0000 7fd1c0870740 20 client.29653 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.175+0000 7fd1c0870740 1 --2- 172.21.15.70:0/41905824 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55decd67c750 0x55decd6e9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.175+0000 7fd1c0870740 4 client.29653 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:17:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.175+0000 7fd1c0870740 1 -- 172.21.15.70:0/41905824 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55decd5cf360 con 0x55decd67c750 2024-06-04T05:17:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.175+0000 7fd1b77fe640 1 --2- 172.21.15.70:0/41905824 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55decd67c750 0x55decd6e9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.175+0000 7fd1b77fe640 1 --2- 172.21.15.70:0/41905824 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55decd67c750 0x55decd6e9390 crc :-1 s=READY pgs=440 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a7d45f703056449 server_cookie=3acac7f2c2efe4d9 in_seq=0 out_seq=0 2024-06-04T05:17:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.175+0000 7fd196ffd640 10 client.29653 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:54.300 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.298+0000 7fd196ffd640 1 -- 172.21.15.70:0/41905824 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55decd5cf360 con 0x55decd67c750 2024-06-04T05:17:54.300 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.298+0000 7fd196ffd640 10 client.29653 handle_command_reply: tid=0 2024-06-04T05:17:54.300 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.298+0000 7fd1c0870740 1 client.29653 shutdown 2024-06-04T05:17:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.298+0000 7fd1c0870740 1 -- 172.21.15.70:0/41905824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd66ca00 msgr2=0x55decd67f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.298+0000 7fd1c0870740 1 --2- 172.21.15.70:0/41905824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd66ca00 0x55decd67f160 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0003f50 tx=0x7fd1b0004ab0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.298+0000 7fd1c0870740 1 -- 172.21.15.70:0/41905824 shutdown_connections 2024-06-04T05:17:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.298+0000 7fd1c0870740 1 --2- 172.21.15.70:0/41905824 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55decd67c750 0x55decd6e9390 crc :-1 s=READY pgs=440 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.298+0000 7fd1c0870740 1 --2- 172.21.15.70:0/41905824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55decd662140 0x55decd6df980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.298+0000 7fd1c0870740 1 --2- 172.21.15.70:0/41905824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55decd66ca00 0x55decd67f160 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.298+0000 7fd1c0870740 1 --2- 172.21.15.70:0/41905824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55decd661820 0x55decd6df440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.298+0000 7fd1c0870740 1 -- 172.21.15.70:0/41905824 >> 172.21.15.70:0/41905824 conn(0x55decd65dd30 msgr2=0x55decd66a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.298+0000 7fd196ffd640 0 client.29653 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:17:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.299+0000 7fd1c0870740 1 -- 172.21.15.70:0/41905824 shutdown_connections 2024-06-04T05:17:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.299+0000 7fd1c0870740 1 -- 172.21.15.70:0/41905824 wait complete. 2024-06-04T05:17:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.299+0000 7fd1c0870740 20 client.29653 trim_cache size 0 max 16384 2024-06-04T05:17:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:17:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:17:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:17:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.300+0000 7fd1bdb59640 1 -- 172.21.15.70:0/2289024445 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd1a40b2a90 msgr2=0x7fd1a40b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.300+0000 7fd1bdb59640 1 --2- 172.21.15.70:0/2289024445 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd1a40b2a90 0x7fd1a40b4f50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0011890 tx=0x7fd1b0009500 comp rx=0 tx=0).stop 2024-06-04T05:17:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.300+0000 7fd1bdb59640 1 -- 172.21.15.70:0/2289024445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b8102a30 msgr2=0x7fd1b8074d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.300+0000 7fd1bdb59640 1 --2- 172.21.15.70:0/2289024445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b8102a30 0x7fd1b8074d50 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800e9f0 tx=0x7fd1a800edf0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.300+0000 7fd1bdb59640 1 -- 172.21.15.70:0/2289024445 shutdown_connections 2024-06-04T05:17:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.300+0000 7fd1bdb59640 1 --2- 172.21.15.70:0/2289024445 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd1a40b2a90 0x7fd1a40b4f50 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.300+0000 7fd1bdb59640 1 --2- 172.21.15.70:0/2289024445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8108a40 0x7fd1b80757d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.300+0000 7fd1bdb59640 1 --2- 172.21.15.70:0/2289024445 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1b81033f0 0x7fd1b8075290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.300+0000 7fd1bdb59640 1 --2- 172.21.15.70:0/2289024445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b8102a30 0x7fd1b8074d50 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.300+0000 7fd1bdb59640 1 -- 172.21.15.70:0/2289024445 >> 172.21.15.70:0/2289024445 conn(0x7fd1b80fe500 msgr2=0x7fd1b80fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.300+0000 7fd1bdb59640 1 -- 172.21.15.70:0/2289024445 shutdown_connections 2024-06-04T05:17:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:17:54.301+0000 7fd1bdb59640 1 -- 172.21.15.70:0/2289024445 wait complete. 2024-06-04T05:17:54.352 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:17:54.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 53946 7.09 MB/sec execute 286 sec latency 93.622 ms 2024-06-04T05:17:55.174 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:54 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:55.174 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:54 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:55.174 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:54 smithi070 ceph-mon[32631]: pgmap v651: 97 pgs: 97 active+clean; 815 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.1 MiB/s wr, 238 op/s 2024-06-04T05:17:55.174 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:54 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:55.174 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:54 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:55.174 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:54 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:55.174 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:54 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:55.174 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:54 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi098 to 176.1M 2024-06-04T05:17:55.174 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:54 smithi070 ceph-mon[32631]: Unable to set osd_memory_target on smithi098 to 184732979: error parsing value: Value '184732979' is below minimum 939524096 2024-06-04T05:17:55.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:54 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:55.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:54 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:55.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:54 smithi098 ceph-mon[35569]: pgmap v651: 97 pgs: 97 active+clean; 815 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.1 MiB/s wr, 238 op/s 2024-06-04T05:17:55.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:54 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:55.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:54 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:55.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:54 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:55.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:54 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:55.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:54 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi098 to 176.1M 2024-06-04T05:17:55.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:54 smithi098 ceph-mon[35569]: Unable to set osd_memory_target on smithi098 to 184732979: error parsing value: Value '184732979' is below minimum 939524096 2024-06-04T05:17:55.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:54 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:55.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:54 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:55.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:54 smithi112 ceph-mon[36453]: pgmap v651: 97 pgs: 97 active+clean; 815 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.1 MiB/s wr, 238 op/s 2024-06-04T05:17:55.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:54 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:55.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:54 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:55.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:54 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:55.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:54 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:55.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:54 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi098 to 176.1M 2024-06-04T05:17:55.437 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:55 smithi112 ceph-mon[36453]: Unable to set osd_memory_target on smithi098 to 184732979: error parsing value: Value '184732979' is below minimum 939524096 2024-06-04T05:17:55.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 54121 7.09 MB/sec execute 287 sec latency 54.016 ms 2024-06-04T05:17:56.459 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:56 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:56.459 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:56 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:56.577 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:56 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:56.577 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:56 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:56.577 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:56 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:56.577 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:56 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:56.577 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:56 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:56.577 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:56 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:56.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.625+0000 7ff1983dc700 1 -- 172.21.15.70:0/3392951026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff190102430 msgr2=0x7ff1901028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:56.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.625+0000 7ff1983dc700 1 --2- 172.21.15.70:0/3392951026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff190102430 0x7ff1901028b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff180009990 tx=0x7ff180009ca0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.626+0000 7ff1983dc700 1 -- 172.21.15.70:0/3392951026 shutdown_connections 2024-06-04T05:17:56.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.626+0000 7ff1983dc700 1 --2- 172.21.15.70:0/3392951026 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff190102df0 0x7ff19010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.626+0000 7ff1983dc700 1 --2- 172.21.15.70:0/3392951026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff190108510 0x7ff1901088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.626+0000 7ff1983dc700 1 --2- 172.21.15.70:0/3392951026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff190102430 0x7ff1901028b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.626+0000 7ff1983dc700 1 -- 172.21.15.70:0/3392951026 >> 172.21.15.70:0/3392951026 conn(0x7ff1900793b0 msgr2=0x7ff1900797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:56.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.627+0000 7ff1983dc700 1 -- 172.21.15.70:0/3392951026 shutdown_connections 2024-06-04T05:17:56.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.627+0000 7ff1983dc700 1 -- 172.21.15.70:0/3392951026 wait complete. 2024-06-04T05:17:56.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.628+0000 7ff1983dc700 1 Processor -- start 2024-06-04T05:17:56.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.628+0000 7ff1983dc700 1 -- start start 2024-06-04T05:17:56.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.628+0000 7ff1983dc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff190102430 0x7ff190198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.628+0000 7ff1983dc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff190102df0 0x7ff190198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.628+0000 7ff1983dc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff190108510 0x7ff19019d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.628+0000 7ff1983dc700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901124c0 con 0x7ff190102430 2024-06-04T05:17:56.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.628+0000 7ff1983dc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff190112340 con 0x7ff190102df0 2024-06-04T05:17:56.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.628+0000 7ff1983dc700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff190112640 con 0x7ff190108510 2024-06-04T05:17:56.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.628+0000 7ff195977700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff190102df0 0x7ff190198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.628+0000 7ff195977700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff190102df0 0x7ff190198cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:52430/0 (socket says 172.21.15.70:52430) 2024-06-04T05:17:56.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.628+0000 7ff195977700 1 -- 172.21.15.70:0/1463484166 learned_addr learned my addr 172.21.15.70:0/1463484166 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:56.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.629+0000 7ff196979700 1 --2- 172.21.15.70:0/1463484166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff190108510 0x7ff19019d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.629+0000 7ff196178700 1 --2- 172.21.15.70:0/1463484166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff190102430 0x7ff190198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.629+0000 7ff195977700 1 -- 172.21.15.70:0/1463484166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff190108510 msgr2=0x7ff19019d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:56.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.629+0000 7ff195977700 1 --2- 172.21.15.70:0/1463484166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff190108510 0x7ff19019d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.629+0000 7ff195977700 1 -- 172.21.15.70:0/1463484166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff190102430 msgr2=0x7ff190198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:56.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.629+0000 7ff195977700 1 --2- 172.21.15.70:0/1463484166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff190102430 0x7ff190198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.629+0000 7ff195977700 1 -- 172.21.15.70:0/1463484166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19019dd20 con 0x7ff190102df0 2024-06-04T05:17:56.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.629+0000 7ff195977700 1 --2- 172.21.15.70:0/1463484166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff190102df0 0x7ff190198cd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff18000b650 tx=0x7ff1800047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:56.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.629+0000 7ff1877fe700 1 -- 172.21.15.70:0/1463484166 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff180027070 con 0x7ff190102df0 2024-06-04T05:17:56.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.630+0000 7ff1877fe700 1 -- 172.21.15.70:0/1463484166 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff180022470 con 0x7ff190102df0 2024-06-04T05:17:56.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.630+0000 7ff1877fe700 1 -- 172.21.15.70:0/1463484166 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff180017650 con 0x7ff190102df0 2024-06-04T05:17:56.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.630+0000 7ff1983dc700 1 -- 172.21.15.70:0/1463484166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff19019dfb0 con 0x7ff190102df0 2024-06-04T05:17:56.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.630+0000 7ff1983dc700 1 -- 172.21.15.70:0/1463484166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff19019e440 con 0x7ff190102df0 2024-06-04T05:17:56.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.633+0000 7ff1877fe700 1 -- 172.21.15.70:0/1463484166 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff18002a070 con 0x7ff190102df0 2024-06-04T05:17:56.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.634+0000 7ff1877fe700 1 --2- 172.21.15.70:0/1463484166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff17c0b29f0 0x7ff17c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.634+0000 7ff196178700 1 --2- 172.21.15.70:0/1463484166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff17c0b29f0 0x7ff17c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.634+0000 7ff1877fe700 1 -- 172.21.15.70:0/1463484166 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7ff1800e34e0 con 0x7ff190102df0 2024-06-04T05:17:56.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.635+0000 7ff196178700 1 --2- 172.21.15.70:0/1463484166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff17c0b29f0 0x7ff17c0b4eb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff18c009be0 tx=0x7ff18c009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:56.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.635+0000 7ff1a2c71b80 1 Processor -- start 2024-06-04T05:17:56.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.635+0000 7ff1a2c71b80 1 -- start start 2024-06-04T05:17:56.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.635+0000 7ff1a2c71b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b577320 0x56443b577700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.635+0000 7ff1a2c71b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b5718a0 0x56443b571d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.635+0000 7ff1a2c71b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b572260 0x56443b57e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.635+0000 7ff1a2c71b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56443b4cfd70 con 0x56443b572260 2024-06-04T05:17:56.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.635+0000 7ff1a2c71b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56443b4bf910 con 0x56443b5718a0 2024-06-04T05:17:56.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.635+0000 7ff1a2c71b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56443b4c3950 con 0x56443b577320 2024-06-04T05:17:56.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.636+0000 7ff196979700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b572260 0x56443b57e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.636+0000 7ff196979700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b572260 0x56443b57e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41884/0 (socket says 172.21.15.70:41884) 2024-06-04T05:17:56.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.636+0000 7ff196979700 1 -- 172.21.15.70:0/515302834 learned_addr learned my addr 172.21.15.70:0/515302834 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:56.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.636+0000 7ff195977700 1 --2- 172.21.15.70:0/515302834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b5718a0 0x56443b571d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.636+0000 7ff196178700 1 --2- 172.21.15.70:0/515302834 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b577320 0x56443b577700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.636+0000 7ff196979700 1 -- 172.21.15.70:0/515302834 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b577320 msgr2=0x56443b577700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:56.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.636+0000 7ff196979700 1 --2- 172.21.15.70:0/515302834 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b577320 0x56443b577700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.636+0000 7ff196979700 1 -- 172.21.15.70:0/515302834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b5718a0 msgr2=0x56443b571d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:56.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.636+0000 7ff196979700 1 --2- 172.21.15.70:0/515302834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b5718a0 0x56443b571d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.636+0000 7ff196979700 1 -- 172.21.15.70:0/515302834 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56443b468fb0 con 0x56443b572260 2024-06-04T05:17:56.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.636+0000 7ff195977700 1 --2- 172.21.15.70:0/515302834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b5718a0 0x56443b571d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:17:56.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.636+0000 7ff196979700 1 --2- 172.21.15.70:0/515302834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b572260 0x56443b57e080 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7ff1880118e0 tx=0x7ff188009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9741fe4cfe374ff4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.637+0000 7ff184d70700 1 -- 172.21.15.70:0/515302834 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188019030 con 0x56443b572260 2024-06-04T05:17:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.637+0000 7ff184d70700 1 -- 172.21.15.70:0/515302834 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff188009190 con 0x56443b572260 2024-06-04T05:17:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.637+0000 7ff184d70700 1 -- 172.21.15.70:0/515302834 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188004700 con 0x56443b572260 2024-06-04T05:17:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.638+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/515302834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b572260 msgr2=0x56443b57e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.638+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/515302834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b572260 0x56443b57e080 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7ff1880118e0 tx=0x7ff188009920 comp rx=0 tx=0).stop 2024-06-04T05:17:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.638+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/515302834 shutdown_connections 2024-06-04T05:17:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.638+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/515302834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b572260 0x56443b57e080 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.638+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/515302834 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b577320 0x56443b577700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.638+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/515302834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b5718a0 0x56443b571d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.638+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/515302834 >> 172.21.15.70:0/515302834 conn(0x56443b49d240 msgr2=0x56443b56c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.638+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/515302834 shutdown_connections 2024-06-04T05:17:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.638+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/515302834 wait complete. 2024-06-04T05:17:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.639+0000 7ff1a2c71b80 1 -- start start 2024-06-04T05:17:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.639+0000 7ff1a2c71b80 1 Processor -- start 2024-06-04T05:17:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.639+0000 7ff1a2c71b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b5718a0 0x56443b57ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.640+0000 7ff1a2c71b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b572260 0x56443b57b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.640+0000 7ff1a2c71b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b577320 0x56443b57b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.640+0000 7ff196178700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b5718a0 0x56443b57ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.640+0000 7ff196178700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b5718a0 0x56443b57ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41890/0 (socket says 172.21.15.70:41890) 2024-06-04T05:17:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.640+0000 7ff1a2c71b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56443b3c5fa0 con 0x56443b5718a0 2024-06-04T05:17:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.640+0000 7ff196178700 1 -- 172.21.15.70:0/2864977726 learned_addr learned my addr 172.21.15.70:0/2864977726 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.640+0000 7ff196979700 1 --2- 172.21.15.70:0/2864977726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b577320 0x56443b57b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.640+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/2864977726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56443b427da0 con 0x56443b577320 2024-06-04T05:17:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.640+0000 7ff195977700 1 --2- 172.21.15.70:0/2864977726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b572260 0x56443b57b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.640+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/2864977726 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56443b57e6e0 con 0x56443b572260 2024-06-04T05:17:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.640+0000 7ff195977700 1 -- 172.21.15.70:0/2864977726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b577320 msgr2=0x56443b57b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.640+0000 7ff195977700 1 --2- 172.21.15.70:0/2864977726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b577320 0x56443b57b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.640+0000 7ff195977700 1 -- 172.21.15.70:0/2864977726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b5718a0 msgr2=0x56443b57ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.641+0000 7ff195977700 1 --2- 172.21.15.70:0/2864977726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b5718a0 0x56443b57ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.641+0000 7ff195977700 1 -- 172.21.15.70:0/2864977726 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56443b40f8d0 con 0x56443b572260 2024-06-04T05:17:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.641+0000 7ff196178700 1 --2- 172.21.15.70:0/2864977726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b5718a0 0x56443b57ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:17:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.641+0000 7ff1a2c71b80 10 client.0 _ll_register_callbacks cb 0x56443b45a950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.641+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/2864977726 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff188009190 con 0x56443b572260 2024-06-04T05:17:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.641+0000 7ff195977700 1 --2- 172.21.15.70:0/2864977726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b572260 0x56443b57b0d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff1800e0c00 tx=0x7ff1800aa000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a29b78008f5ba5a1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.641+0000 7ff179ffb700 10 client.29665 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:17:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.641+0000 7ff179ffb700 1 -- 172.21.15.70:0/2864977726 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1800cd0b0 con 0x56443b572260 2024-06-04T05:17:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.641+0000 7ff179ffb700 1 -- 172.21.15.70:0/2864977726 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7ff1800bfbe0 con 0x56443b572260 2024-06-04T05:17:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.641+0000 7ff179ffb700 1 -- 172.21.15.70:0/2864977726 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7ff1800bfde0 con 0x56443b572260 2024-06-04T05:17:56.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.641+0000 7ff1a2c71b80 10 client.29665 fetch_fsmap learned FSMap version 59 2024-06-04T05:17:56.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.642+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/2864977726 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({fsmap=59}) -- 0x56443b479760 con 0x56443b572260 2024-06-04T05:17:56.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.642+0000 7ff179ffb700 1 -- 172.21.15.70:0/2864977726 <== mon.1 v2:172.21.15.112:3300/0 4 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7ff1800aaa80 con 0x56443b572260 2024-06-04T05:17:56.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.642+0000 7ff1a2c71b80 10 client.29665 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:17:56.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.642+0000 7ff1a2c71b80 10 client.29665 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:17:56.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.642+0000 7ff1a2c71b80 20 client.29665 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:56.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.642+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/2864977726 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x56443b58add0 0x56443b58d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.642+0000 7ff1a2c71b80 4 client.29665 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:17:56.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.642+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/2864977726 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56443b4d1cc0 con 0x56443b58add0 2024-06-04T05:17:56.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.642+0000 7ff196178700 1 --2- 172.21.15.70:0/2864977726 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x56443b58add0 0x56443b58d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.643+0000 7ff196178700 1 --2- 172.21.15.70:0/2864977726 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x56443b58add0 0x56443b58d290 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e3198c06380d46d1 server_cookie=af418776b1458cb6 in_seq=0 out_seq=0 2024-06-04T05:17:56.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.643+0000 7ff179ffb700 10 client.29665 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:17:56.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.643+0000 7ff179ffb700 1 -- 172.21.15.70:0/2864977726 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56443b4d1cc0 con 0x56443b58add0 2024-06-04T05:17:56.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.643+0000 7ff179ffb700 10 client.29665 handle_command_reply: tid=0 2024-06-04T05:17:56.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.643+0000 7ff1a2c71b80 1 client.29665 shutdown 2024-06-04T05:17:56.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.644+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/2864977726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b572260 msgr2=0x56443b57b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:56.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.644+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/2864977726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b572260 0x56443b57b0d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff1800e0c00 tx=0x7ff1800aa000 comp rx=0 tx=0).stop 2024-06-04T05:17:56.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.644+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/2864977726 shutdown_connections 2024-06-04T05:17:56.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.644+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/2864977726 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x56443b58add0 0x56443b58d290 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.644+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/2864977726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b577320 0x56443b57b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.644+0000 7ff179ffb700 0 client.29665 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:17:56.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.644+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/2864977726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b5718a0 0x56443b57ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.644+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/2864977726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b572260 0x56443b57b0d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.644+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/2864977726 >> 172.21.15.70:0/2864977726 conn(0x56443b49d240 msgr2=0x56443b56baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:56.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.644+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/2864977726 shutdown_connections 2024-06-04T05:17:56.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.644+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/2864977726 wait complete. 2024-06-04T05:17:56.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.644+0000 7ff1a2c71b80 20 client.29665 trim_cache size 0 max 16384 2024-06-04T05:17:56.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.674+0000 7ff1a2c71b80 1 Processor -- start 2024-06-04T05:17:56.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.674+0000 7ff1a2c71b80 1 -- start start 2024-06-04T05:17:56.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.674+0000 7ff1a2c71b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b5718a0 0x56443b571c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.674+0000 7ff1a2c71b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b572260 0x56443b576800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.674+0000 7ff1a2c71b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b577320 0x56443b56c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.674+0000 7ff1a2c71b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901b4310 con 0x56443b5718a0 2024-06-04T05:17:56.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.674+0000 7ff1a2c71b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901b5eb0 con 0x56443b572260 2024-06-04T05:17:56.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.674+0000 7ff1a2c71b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901b4a40 con 0x56443b577320 2024-06-04T05:17:56.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.674+0000 7ff196178700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b5718a0 0x56443b571c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.674+0000 7ff195977700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b572260 0x56443b576800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.674+0000 7ff196178700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b5718a0 0x56443b571c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41896/0 (socket says 172.21.15.70:41896) 2024-06-04T05:17:56.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.675+0000 7ff195977700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b572260 0x56443b576800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:52464/0 (socket says 172.21.15.70:52464) 2024-06-04T05:17:56.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.675+0000 7ff196178700 1 -- 172.21.15.70:0/386054004 learned_addr learned my addr 172.21.15.70:0/386054004 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:56.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.675+0000 7ff196979700 1 --2- 172.21.15.70:0/386054004 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b577320 0x56443b56c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.675+0000 7ff195977700 1 -- 172.21.15.70:0/386054004 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b577320 msgr2=0x56443b56c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:56.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.675+0000 7ff195977700 1 --2- 172.21.15.70:0/386054004 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b577320 0x56443b56c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.675+0000 7ff195977700 1 -- 172.21.15.70:0/386054004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b5718a0 msgr2=0x56443b571c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:56.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.675+0000 7ff195977700 1 --2- 172.21.15.70:0/386054004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b5718a0 0x56443b571c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.675+0000 7ff195977700 1 -- 172.21.15.70:0/386054004 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56443b57e5c0 con 0x56443b572260 2024-06-04T05:17:56.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.675+0000 7ff196979700 1 --2- 172.21.15.70:0/386054004 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b577320 0x56443b56c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:17:56.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.675+0000 7ff195977700 1 --2- 172.21.15.70:0/386054004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b572260 0x56443b576800 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff1800e1200 tx=0x7ff1800a6000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dae3286a835b9f4c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:56.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.675+0000 7ff17bfff700 1 -- 172.21.15.70:0/386054004 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1800aa820 con 0x56443b572260 2024-06-04T05:17:56.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.675+0000 7ff17bfff700 1 -- 172.21.15.70:0/386054004 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff188009190 con 0x56443b572260 2024-06-04T05:17:56.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.675+0000 7ff17bfff700 1 -- 172.21.15.70:0/386054004 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1800b3160 con 0x56443b572260 2024-06-04T05:17:56.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.676+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/386054004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b572260 msgr2=0x56443b576800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:56.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.676+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/386054004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b572260 0x56443b576800 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff1800e1200 tx=0x7ff1800a6000 comp rx=0 tx=0).stop 2024-06-04T05:17:56.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.677+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/386054004 shutdown_connections 2024-06-04T05:17:56.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.677+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/386054004 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b577320 0x56443b56c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.677+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/386054004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b5718a0 0x56443b571c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.677+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/386054004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b572260 0x56443b576800 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.677+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/386054004 >> 172.21.15.70:0/386054004 conn(0x56443b49d240 msgr2=0x56443b5755e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:56.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.677+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/386054004 shutdown_connections 2024-06-04T05:17:56.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.677+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/386054004 wait complete. 2024-06-04T05:17:56.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.677+0000 7ff1a2c71b80 1 -- start start 2024-06-04T05:17:56.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.677+0000 7ff1a2c71b80 1 Processor -- start 2024-06-04T05:17:56.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff1a2c71b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b5718a0 0x56443b583980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff1a2c71b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b572260 0x56443b583ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff1a2c71b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b577320 0x56443b591b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff1a2c71b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901b57e0 con 0x56443b577320 2024-06-04T05:17:56.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff1a2c71b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901b5110 con 0x56443b5718a0 2024-06-04T05:17:56.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff1a2c71b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901b22f0 con 0x56443b572260 2024-06-04T05:17:56.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff1a2c71b80 10 client.0 _ll_register_callbacks cb 0x56443b45a950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:17:56.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff196178700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b5718a0 0x56443b583980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff195977700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b572260 0x56443b583ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff196178700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b5718a0 0x56443b583980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:52476/0 (socket says 172.21.15.70:52476) 2024-06-04T05:17:56.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff196178700 1 -- 172.21.15.70:0/1167094483 learned_addr learned my addr 172.21.15.70:0/1167094483 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:17:56.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff195977700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b572260 0x56443b583ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:32940/0 (socket says 172.21.15.70:32940) 2024-06-04T05:17:56.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff196979700 1 --2- 172.21.15.70:0/1167094483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b577320 0x56443b591b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff196178700 1 -- 172.21.15.70:0/1167094483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b572260 msgr2=0x56443b583ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:56.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff196178700 1 --2- 172.21.15.70:0/1167094483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b572260 0x56443b583ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff196178700 1 -- 172.21.15.70:0/1167094483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b577320 msgr2=0x56443b591b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:17:56.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff196178700 1 --2- 172.21.15.70:0/1167094483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b577320 0x56443b591b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff196178700 1 -- 172.21.15.70:0/1167094483 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56443b584590 con 0x56443b5718a0 2024-06-04T05:17:56.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff196979700 1 --2- 172.21.15.70:0/1167094483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b577320 0x56443b591b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:17:56.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff196178700 1 --2- 172.21.15.70:0/1167094483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b5718a0 0x56443b583980 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff18c004300 tx=0x7ff18c0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddb52eb5cd4a1684 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:17:56.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.678+0000 7ff179ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:17:56.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.679+0000 7ff179ffb700 1 -- 172.21.15.70:0/1167094483 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18c020ac0 con 0x56443b5718a0 2024-06-04T05:17:56.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.679+0000 7ff179ffb700 1 -- 172.21.15.70:0/1167094483 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8571+0+0 (secure 0 0 0) 0x7ff18c020c60 con 0x56443b5718a0 2024-06-04T05:17:56.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.679+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/1167094483 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff188009190 con 0x56443b5718a0 2024-06-04T05:17:56.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.679+0000 7ff179ffb700 1 -- 172.21.15.70:0/1167094483 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7ff18c039050 con 0x56443b5718a0 2024-06-04T05:17:56.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.679+0000 7ff1a2c71b80 10 client.29522 fetch_fsmap learned FSMap version 59 2024-06-04T05:17:56.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.679+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/1167094483 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x56443b5f5130 con 0x56443b5718a0 2024-06-04T05:17:56.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.680+0000 7ff179ffb700 1 -- 172.21.15.70:0/1167094483 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff188009190 con 0x56443b5718a0 2024-06-04T05:17:56.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.680+0000 7ff179ffb700 1 -- 172.21.15.70:0/1167094483 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7ff18c02b5e0 con 0x56443b5718a0 2024-06-04T05:17:56.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.680+0000 7ff1a2c71b80 10 client.29522 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:17:56.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.680+0000 7ff1a2c71b80 10 client.29522 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:17:56.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.680+0000 7ff1a2c71b80 20 client.29522 populate_metadata read hostname 'smithi070' 2024-06-04T05:17:56.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.680+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/1167094483 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x56443b5f5830 0x56443b5f7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:17:56.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.680+0000 7ff1a2c71b80 4 client.29522 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:17:56.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.680+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/1167094483 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56443b4cb4a0 con 0x56443b5f5830 2024-06-04T05:17:56.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.681+0000 7ff195977700 1 --2- 172.21.15.70:0/1167094483 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x56443b5f5830 0x56443b5f7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:17:56.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.681+0000 7ff195977700 1 --2- 172.21.15.70:0/1167094483 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x56443b5f5830 0x56443b5f7c50 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4670f3b9295f6d54 server_cookie=629046c951eefe72 in_seq=0 out_seq=0 2024-06-04T05:17:56.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.681+0000 7ff179ffb700 10 client.29522 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:17:56.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.681+0000 7ff179ffb700 1 -- 172.21.15.70:0/1167094483 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56443b4cb4a0 con 0x56443b5f5830 2024-06-04T05:17:56.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.681+0000 7ff179ffb700 10 client.29522 handle_command_reply: tid=0 2024-06-04T05:17:56.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.681+0000 7ff1a2c71b80 1 client.29522 shutdown 2024-06-04T05:17:56.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.682+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/1167094483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b5718a0 msgr2=0x56443b583980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:56.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.682+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/1167094483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b5718a0 0x56443b583980 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff18c004300 tx=0x7ff18c0047c0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.682+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/1167094483 shutdown_connections 2024-06-04T05:17:56.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.682+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/1167094483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56443b577320 0x56443b591b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.682+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/1167094483 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x56443b5f5830 0x56443b5f7c50 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.682+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/1167094483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443b5718a0 0x56443b583980 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.682+0000 7ff179ffb700 0 client.29522 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:17:56.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.682+0000 7ff1a2c71b80 1 --2- 172.21.15.70:0/1167094483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56443b572260 0x56443b583ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.682+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/1167094483 >> 172.21.15.70:0/1167094483 conn(0x56443b49d240 msgr2=0x56443b570890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:56.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.682+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/1167094483 shutdown_connections 2024-06-04T05:17:56.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.682+0000 7ff1a2c71b80 1 -- 172.21.15.70:0/1167094483 wait complete. 2024-06-04T05:17:56.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.682+0000 7ff1a2c71b80 20 client.29522 trim_cache size 0 max 16384 2024-06-04T05:17:56.687 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:17:56.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.683+0000 7ff1983dc700 1 -- 172.21.15.70:0/1463484166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff17c0b29f0 msgr2=0x7ff17c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:56.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.683+0000 7ff1983dc700 1 --2- 172.21.15.70:0/1463484166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff17c0b29f0 0x7ff17c0b4eb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff18c009be0 tx=0x7ff18c009260 comp rx=0 tx=0).stop 2024-06-04T05:17:56.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.683+0000 7ff1983dc700 1 -- 172.21.15.70:0/1463484166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff190102df0 msgr2=0x7ff190198cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:17:56.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.683+0000 7ff1983dc700 1 --2- 172.21.15.70:0/1463484166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff190102df0 0x7ff190198cd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff18000b650 tx=0x7ff1800047b0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.683+0000 7ff1983dc700 1 -- 172.21.15.70:0/1463484166 shutdown_connections 2024-06-04T05:17:56.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.683+0000 7ff1983dc700 1 --2- 172.21.15.70:0/1463484166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff17c0b29f0 0x7ff17c0b4eb0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.683+0000 7ff1983dc700 1 --2- 172.21.15.70:0/1463484166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff190108510 0x7ff19019d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.683+0000 7ff1983dc700 1 --2- 172.21.15.70:0/1463484166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff190102430 0x7ff190198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.683+0000 7ff1983dc700 1 --2- 172.21.15.70:0/1463484166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff190102df0 0x7ff190198cd0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:17:56.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.683+0000 7ff1983dc700 1 -- 172.21.15.70:0/1463484166 >> 172.21.15.70:0/1463484166 conn(0x7ff1900793b0 msgr2=0x7ff190100640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:17:56.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.683+0000 7ff1983dc700 1 -- 172.21.15.70:0/1463484166 shutdown_connections 2024-06-04T05:17:56.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:17:56.683+0000 7ff1983dc700 1 -- 172.21.15.70:0/1463484166 wait complete. 2024-06-04T05:17:56.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:56 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:56.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:56 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:17:56.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:56 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:56.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:56 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:56.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:56 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:56.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:56 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:56.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 54261 7.08 MB/sec execute 288 sec latency 126.917 ms 2024-06-04T05:17:56.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:56 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:56.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:56 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:56.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:56 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:56.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:56 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:17:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:57 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi112 to 176.1M 2024-06-04T05:17:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:57 smithi098 ceph-mon[35569]: Unable to set osd_memory_target on smithi112 to 184710758: error parsing value: Value '184710758' is below minimum 939524096 2024-06-04T05:17:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:57 smithi098 ceph-mon[35569]: pgmap v652: 97 pgs: 97 active+clean; 826 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.4 MiB/s wr, 398 op/s 2024-06-04T05:17:57.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 54402 7.07 MB/sec execute 289 sec latency 98.390 ms 2024-06-04T05:17:57.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:57 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi112 to 176.1M 2024-06-04T05:17:57.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:57 smithi070 ceph-mon[32631]: Unable to set osd_memory_target on smithi112 to 184710758: error parsing value: Value '184710758' is below minimum 939524096 2024-06-04T05:17:57.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:57 smithi070 ceph-mon[32631]: pgmap v652: 97 pgs: 97 active+clean; 826 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.4 MiB/s wr, 398 op/s 2024-06-04T05:17:57.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:57 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi112 to 176.1M 2024-06-04T05:17:57.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:57 smithi112 ceph-mon[36453]: Unable to set osd_memory_target on smithi112 to 184710758: error parsing value: Value '184710758' is below minimum 939524096 2024-06-04T05:17:57.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:57 smithi112 ceph-mon[36453]: pgmap v652: 97 pgs: 97 active+clean; 826 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.4 MiB/s wr, 398 op/s 2024-06-04T05:17:58.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 54557 7.07 MB/sec execute 290 sec latency 96.471 ms 2024-06-04T05:17:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:17:59 smithi098 ceph-mon[35569]: pgmap v653: 97 pgs: 97 active+clean; 826 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.3 MiB/s wr, 272 op/s 2024-06-04T05:17:59.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 54688 7.06 MB/sec execute 291 sec latency 188.301 ms 2024-06-04T05:17:59.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:17:59 smithi070 ceph-mon[32631]: pgmap v653: 97 pgs: 97 active+clean; 826 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.3 MiB/s wr, 272 op/s 2024-06-04T05:17:59.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:17:59 smithi112 ceph-mon[36453]: pgmap v653: 97 pgs: 97 active+clean; 826 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.3 MiB/s wr, 272 op/s 2024-06-04T05:18:00.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 54843 7.08 MB/sec execute 292 sec latency 107.716 ms 2024-06-04T05:18:01.738 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:01 smithi112 ceph-mon[36453]: pgmap v654: 97 pgs: 97 active+clean; 826 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.3 MiB/s wr, 272 op/s 2024-06-04T05:18:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:01 smithi098 ceph-mon[35569]: pgmap v654: 97 pgs: 97 active+clean; 826 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.3 MiB/s wr, 272 op/s 2024-06-04T05:18:01.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 54983 7.07 MB/sec execute 293 sec latency 133.981 ms 2024-06-04T05:18:01.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:01 smithi070 ceph-mon[32631]: pgmap v654: 97 pgs: 97 active+clean; 826 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.3 MiB/s wr, 272 op/s 2024-06-04T05:18:02.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 55153 7.08 MB/sec execute 294 sec latency 107.224 ms 2024-06-04T05:18:03.500 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:18:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:03 smithi098 ceph-mon[35569]: pgmap v655: 97 pgs: 97 active+clean; 818 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.3 MiB/s wr, 422 op/s 2024-06-04T05:18:03.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 55300 7.07 MB/sec execute 295 sec latency 153.547 ms 2024-06-04T05:18:03.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:03 smithi070 ceph-mon[32631]: pgmap v655: 97 pgs: 97 active+clean; 818 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.3 MiB/s wr, 422 op/s 2024-06-04T05:18:03.889 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:18:03.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:03 smithi112 ceph-mon[36453]: pgmap v655: 97 pgs: 97 active+clean; 818 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.3 MiB/s wr, 422 op/s 2024-06-04T05:18:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:04 smithi098 ceph-mon[35569]: osdmap e109: 12 total, 12 up, 12 in 2024-06-04T05:18:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:04 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:18:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:04 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:18:04.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 55427 7.06 MB/sec execute 296 sec latency 147.467 ms 2024-06-04T05:18:04.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:04 smithi070 ceph-mon[32631]: osdmap e109: 12 total, 12 up, 12 in 2024-06-04T05:18:04.988 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:04 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:18:04.989 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:04 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:18:04.989 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:04 smithi112 ceph-mon[36453]: osdmap e109: 12 total, 12 up, 12 in 2024-06-04T05:18:04.989 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:04 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:18:04.989 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:04 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:18:05.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 55578 7.05 MB/sec execute 297 sec latency 188.895 ms 2024-06-04T05:18:05.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:05 smithi112 ceph-mon[36453]: pgmap v657: 97 pgs: 97 active+clean; 818 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.3 MiB/s wr, 373 op/s 2024-06-04T05:18:05.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:05 smithi098 ceph-mon[35569]: pgmap v657: 97 pgs: 97 active+clean; 818 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.3 MiB/s wr, 373 op/s 2024-06-04T05:18:06.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:05 smithi070 ceph-mon[32631]: pgmap v657: 97 pgs: 97 active+clean; 818 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.3 MiB/s wr, 373 op/s 2024-06-04T05:18:06.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 55723 7.04 MB/sec execute 298 sec latency 119.349 ms 2024-06-04T05:18:07.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.439+0000 7f3282703640 1 -- 172.21.15.70:0/2344630459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f327c102340 msgr2=0x7f327c10e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:07.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.439+0000 7f3282703640 1 --2- 172.21.15.70:0/2344630459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f327c102340 0x7f327c10e800 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3274009cd0 tx=0x7f327402a8e0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.439+0000 7f3282703640 1 -- 172.21.15.70:0/2344630459 shutdown_connections 2024-06-04T05:18:07.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.439+0000 7f3282703640 1 --2- 172.21.15.70:0/2344630459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f327c102340 0x7f327c10e800 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.439+0000 7f3282703640 1 --2- 172.21.15.70:0/2344630459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f327c101980 0x7f327c101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.439+0000 7f3282703640 1 --2- 172.21.15.70:0/2344630459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f327c107940 0x7f327c107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.439+0000 7f3282703640 1 -- 172.21.15.70:0/2344630459 >> 172.21.15.70:0/2344630459 conn(0x7f327c078ad0 msgr2=0x7f327c078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:07.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.439+0000 7f3282703640 1 -- 172.21.15.70:0/2344630459 shutdown_connections 2024-06-04T05:18:07.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.439+0000 7f3282703640 1 -- 172.21.15.70:0/2344630459 wait complete. 2024-06-04T05:18:07.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.440+0000 7f3282703640 1 Processor -- start 2024-06-04T05:18:07.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.440+0000 7f3282703640 1 -- start start 2024-06-04T05:18:07.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.440+0000 7f3282703640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f327c101980 0x7f327c19b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.440+0000 7f3282703640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f327c102340 0x7f327c19b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.440+0000 7f3282703640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f327c107940 0x7f327c19fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.440+0000 7f3282703640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f327c112c20 con 0x7f327c107940 2024-06-04T05:18:07.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.440+0000 7f3282703640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f327c112aa0 con 0x7f327c102340 2024-06-04T05:18:07.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.440+0000 7f3282703640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f327c112da0 con 0x7f327c101980 2024-06-04T05:18:07.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.440+0000 7f327bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f327c101980 0x7f327c19b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.440+0000 7f327bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f327c101980 0x7f327c19b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:40236/0 (socket says 172.21.15.70:40236) 2024-06-04T05:18:07.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.440+0000 7f327bfff640 1 -- 172.21.15.70:0/2339307846 learned_addr learned my addr 172.21.15.70:0/2339307846 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:07.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.440+0000 7f327b7fe640 1 --2- 172.21.15.70:0/2339307846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f327c102340 0x7f327c19b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.440+0000 7f3280ca0640 1 --2- 172.21.15.70:0/2339307846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f327c107940 0x7f327c19fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.441+0000 7f327bfff640 1 -- 172.21.15.70:0/2339307846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f327c102340 msgr2=0x7f327c19b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:07.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.441+0000 7f327bfff640 1 --2- 172.21.15.70:0/2339307846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f327c102340 0x7f327c19b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.441+0000 7f327bfff640 1 -- 172.21.15.70:0/2339307846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f327c107940 msgr2=0x7f327c19fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:07.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.441+0000 7f327bfff640 1 --2- 172.21.15.70:0/2339307846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f327c107940 0x7f327c19fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.441+0000 7f327bfff640 1 -- 172.21.15.70:0/2339307846 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f327c1a0280 con 0x7f327c101980 2024-06-04T05:18:07.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.441+0000 7f327b7fe640 1 --2- 172.21.15.70:0/2339307846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f327c102340 0x7f327c19b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:07.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.441+0000 7f3280ca0640 1 --2- 172.21.15.70:0/2339307846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f327c107940 0x7f327c19fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:07.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.441+0000 7f327bfff640 1 --2- 172.21.15.70:0/2339307846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f327c101980 0x7f327c19b290 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f326c00e9f0 tx=0x7f326c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:07.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.441+0000 7f32797fa640 1 -- 172.21.15.70:0/2339307846 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c00cd80 con 0x7f327c101980 2024-06-04T05:18:07.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.441+0000 7f3282703640 1 -- 172.21.15.70:0/2339307846 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f327c1a0570 con 0x7f327c101980 2024-06-04T05:18:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.441+0000 7f32797fa640 1 -- 172.21.15.70:0/2339307846 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f326c0108c0 con 0x7f327c101980 2024-06-04T05:18:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.441+0000 7f32797fa640 1 -- 172.21.15.70:0/2339307846 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c0187c0 con 0x7f327c101980 2024-06-04T05:18:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.441+0000 7f3282703640 1 -- 172.21.15.70:0/2339307846 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f327c1a7e00 con 0x7f327c101980 2024-06-04T05:18:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.444+0000 7f32797fa640 1 -- 172.21.15.70:0/2339307846 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f326c010430 con 0x7f327c101980 2024-06-04T05:18:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.444+0000 7f3285418740 1 Processor -- start 2024-06-04T05:18:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.444+0000 7f3285418740 1 -- start start 2024-06-04T05:18:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.444+0000 7f3285418740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa07c820 0x55f4aa07cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.444+0000 7f3285418740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 0x55f4aa0874c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.444+0000 7f3285418740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa087a00 0x55f4aa089df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.444+0000 7f3285418740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f4a9feec30 con 0x55f4aa07d140 2024-06-04T05:18:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.444+0000 7f3285418740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f4a9fde7d0 con 0x55f4aa087a00 2024-06-04T05:18:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.444+0000 7f3285418740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f4a9fe2810 con 0x55f4aa07c820 2024-06-04T05:18:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.444+0000 7f327bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 0x55f4aa0874c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.444+0000 7f327bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 0x55f4aa0874c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39234/0 (socket says 172.21.15.70:39234) 2024-06-04T05:18:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.444+0000 7f3280ca0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa07c820 0x55f4aa07cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.444+0000 7f327bfff640 1 -- 172.21.15.70:0/483737079 learned_addr learned my addr 172.21.15.70:0/483737079 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.444+0000 7f32797fa640 1 --2- 172.21.15.70:0/2339307846 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f32640b2a10 0x7f32640b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.444+0000 7f327b7fe640 1 --2- 172.21.15.70:0/483737079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa087a00 0x55f4aa089df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.445+0000 7f3280ca0640 1 --2- 172.21.15.70:0/2339307846 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f32640b2a10 0x7f32640b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.445+0000 7f3280ca0640 1 -- 172.21.15.70:0/483737079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa087a00 msgr2=0x55f4aa089df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.445+0000 7f3280ca0640 1 --2- 172.21.15.70:0/483737079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa087a00 0x55f4aa089df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.445+0000 7f3280ca0640 1 -- 172.21.15.70:0/483737079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 msgr2=0x55f4aa0874c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.445+0000 7f3280ca0640 1 --2- 172.21.15.70:0/483737079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 0x55f4aa0874c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.445+0000 7f3280ca0640 1 -- 172.21.15.70:0/483737079 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4a9cf3210 con 0x55f4aa07c820 2024-06-04T05:18:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.445+0000 7f327bfff640 1 --2- 172.21.15.70:0/483737079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 0x55f4aa0874c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.445+0000 7f32797fa640 1 -- 172.21.15.70:0/2339307846 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8615+0+0 (secure 0 0 0) 0x7f326c0e2960 con 0x7f327c101980 2024-06-04T05:18:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.445+0000 7f3280ca0640 1 --2- 172.21.15.70:0/483737079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa07c820 0x55f4aa07cc00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3274009ca0 tx=0x7f3274031ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b6ecac5ffcb4335 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.445+0000 7f32627fc640 1 -- 172.21.15.70:0/483737079 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3274004790 con 0x55f4aa07c820 2024-06-04T05:18:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.445+0000 7f3280ca0640 1 --2- 172.21.15.70:0/2339307846 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f32640b2a10 0x7f32640b4ed0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3274005e30 tx=0x7f327404c000 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.445+0000 7f32627fc640 1 -- 172.21.15.70:0/483737079 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3274036070 con 0x55f4aa07c820 2024-06-04T05:18:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.445+0000 7f32627fc640 1 -- 172.21.15.70:0/483737079 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327404a690 con 0x55f4aa07c820 2024-06-04T05:18:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.446+0000 7f3285418740 1 -- 172.21.15.70:0/483737079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa07c820 msgr2=0x55f4aa07cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.446+0000 7f3285418740 1 --2- 172.21.15.70:0/483737079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa07c820 0x55f4aa07cc00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3274009ca0 tx=0x7f3274031ca0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.446+0000 7f3285418740 1 -- 172.21.15.70:0/483737079 shutdown_connections 2024-06-04T05:18:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.446+0000 7f3285418740 1 --2- 172.21.15.70:0/483737079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa087a00 0x55f4aa089df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.446+0000 7f3285418740 1 --2- 172.21.15.70:0/483737079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 0x55f4aa0874c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.446+0000 7f3285418740 1 --2- 172.21.15.70:0/483737079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa07c820 0x55f4aa07cc00 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.446+0000 7f3285418740 1 -- 172.21.15.70:0/483737079 >> 172.21.15.70:0/483737079 conn(0x55f4aa078d30 msgr2=0x55f4aa079110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.446+0000 7f3285418740 1 -- 172.21.15.70:0/483737079 shutdown_connections 2024-06-04T05:18:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.446+0000 7f3285418740 1 -- 172.21.15.70:0/483737079 wait complete. 2024-06-04T05:18:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.446+0000 7f3285418740 1 -- start start 2024-06-04T05:18:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.446+0000 7f3285418740 1 Processor -- start 2024-06-04T05:18:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f3285418740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa07c820 0x55f4aa084a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f3285418740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 0x55f4aa084fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f3285418740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa087a00 0x55f4aa0854e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f3285418740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f4a9cc4790 con 0x55f4aa07d140 2024-06-04T05:18:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f3285418740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f4a9f9a890 con 0x55f4aa07c820 2024-06-04T05:18:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f3285418740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f4aa08a630 con 0x55f4aa087a00 2024-06-04T05:18:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f3285418740 10 client.0 _ll_register_callbacks cb 0x55f4a9cf2a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f327bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa07c820 0x55f4aa084a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f327bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa07c820 0x55f4aa084a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55952/0 (socket says 172.21.15.70:55952) 2024-06-04T05:18:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f327bfff640 1 -- 172.21.15.70:0/4036036659 learned_addr learned my addr 172.21.15.70:0/4036036659 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f327b7fe640 1 --2- 172.21.15.70:0/4036036659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 0x55f4aa084fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f3280ca0640 1 --2- 172.21.15.70:0/4036036659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa087a00 0x55f4aa0854e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f327bfff640 1 -- 172.21.15.70:0/4036036659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa087a00 msgr2=0x55f4aa0854e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f327bfff640 1 --2- 172.21.15.70:0/4036036659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa087a00 0x55f4aa0854e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f327bfff640 1 -- 172.21.15.70:0/4036036659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 msgr2=0x55f4aa084fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f327bfff640 1 --2- 172.21.15.70:0/4036036659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 0x55f4aa084fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f327bfff640 1 -- 172.21.15.70:0/4036036659 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4a9dbf620 con 0x55f4aa07c820 2024-06-04T05:18:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f3280ca0640 1 --2- 172.21.15.70:0/4036036659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa087a00 0x55f4aa0854e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:18:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f327bfff640 1 --2- 172.21.15.70:0/4036036659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa07c820 0x55f4aa084a60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f326c0e0100 tx=0x7f326c0056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5ed930ac8ee3c0c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f325b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:18:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.447+0000 7f325b7fe640 1 -- 172.21.15.70:0/4036036659 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c010720 con 0x55f4aa07c820 2024-06-04T05:18:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.448+0000 7f325b7fe640 1 -- 172.21.15.70:0/4036036659 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8615+0+0 (secure 0 0 0) 0x7f326c0b27b0 con 0x55f4aa07c820 2024-06-04T05:18:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.448+0000 7f3285418740 1 -- 172.21.15.70:0/4036036659 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3274036070 con 0x55f4aa07c820 2024-06-04T05:18:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.448+0000 7f325b7fe640 1 -- 172.21.15.70:0/4036036659 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f326c0c6e60 con 0x55f4aa07c820 2024-06-04T05:18:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.448+0000 7f3285418740 10 client.29534 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.448+0000 7f3285418740 1 -- 172.21.15.70:0/4036036659 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55f4aa0964b0 con 0x55f4aa07c820 2024-06-04T05:18:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.448+0000 7f325b7fe640 1 -- 172.21.15.70:0/4036036659 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3274036070 con 0x55f4aa07c820 2024-06-04T05:18:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.448+0000 7f325b7fe640 1 -- 172.21.15.70:0/4036036659 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f326c0a5600 con 0x55f4aa07c820 2024-06-04T05:18:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.448+0000 7f3285418740 10 client.29534 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.448+0000 7f3285418740 10 client.29534 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:18:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.448+0000 7f3285418740 20 client.29534 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.448+0000 7f3285418740 1 --2- 172.21.15.70:0/4036036659 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f4aa097de0 0x55f4aa09a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.448+0000 7f3285418740 4 client.29534 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:18:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.448+0000 7f3285418740 1 -- 172.21.15.70:0/4036036659 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4a9ff0b80 con 0x55f4aa097de0 2024-06-04T05:18:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.448+0000 7f327b7fe640 1 --2- 172.21.15.70:0/4036036659 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f4aa097de0 0x55f4aa09a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.449+0000 7f325b7fe640 10 client.29534 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.449+0000 7f327b7fe640 1 --2- 172.21.15.70:0/4036036659 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f4aa097de0 0x55f4aa09a260 crc :-1 s=READY pgs=441 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56ea344278728279 server_cookie=c4ef86057ed4e896 in_seq=0 out_seq=0 2024-06-04T05:18:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.450+0000 7f325b7fe640 1 -- 172.21.15.70:0/4036036659 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f4a9ff0b80 con 0x55f4aa097de0 2024-06-04T05:18:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.450+0000 7f325b7fe640 10 client.29534 handle_command_reply: tid=0 2024-06-04T05:18:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.450+0000 7f3285418740 1 client.29534 shutdown 2024-06-04T05:18:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.450+0000 7f3285418740 1 -- 172.21.15.70:0/4036036659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa07c820 msgr2=0x55f4aa084a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.450+0000 7f3285418740 1 --2- 172.21.15.70:0/4036036659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa07c820 0x55f4aa084a60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f326c0e0100 tx=0x7f326c0056d0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.450+0000 7f3285418740 1 -- 172.21.15.70:0/4036036659 shutdown_connections 2024-06-04T05:18:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.450+0000 7f3285418740 1 --2- 172.21.15.70:0/4036036659 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f4aa097de0 0x55f4aa09a260 crc :-1 s=READY pgs=441 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.450+0000 7f3285418740 1 --2- 172.21.15.70:0/4036036659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa087a00 0x55f4aa0854e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.450+0000 7f3285418740 1 --2- 172.21.15.70:0/4036036659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 0x55f4aa084fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.450+0000 7f3285418740 1 --2- 172.21.15.70:0/4036036659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa07c820 0x55f4aa084a60 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.450+0000 7f325b7fe640 0 client.29534 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.450+0000 7f3285418740 1 -- 172.21.15.70:0/4036036659 >> 172.21.15.70:0/4036036659 conn(0x55f4aa078d30 msgr2=0x55f4aa077fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.450+0000 7f3285418740 1 -- 172.21.15.70:0/4036036659 shutdown_connections 2024-06-04T05:18:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.450+0000 7f3285418740 1 -- 172.21.15.70:0/4036036659 wait complete. 2024-06-04T05:18:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.450+0000 7f3285418740 20 client.29534 trim_cache size 0 max 16384 2024-06-04T05:18:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.465+0000 7f3285418740 1 Processor -- start 2024-06-04T05:18:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.465+0000 7f3285418740 1 -- start start 2024-06-04T05:18:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f3285418740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa07c820 0x55f4aa078730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f3285418740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 0x55f4aa085300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f3285418740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa087a00 0x55f4aa09fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f3285418740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f327c1b0860 con 0x55f4aa07d140 2024-06-04T05:18:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f3285418740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f327c1b2400 con 0x55f4aa087a00 2024-06-04T05:18:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f3285418740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f327c1b0f90 con 0x55f4aa07c820 2024-06-04T05:18:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f327b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 0x55f4aa085300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f327b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 0x55f4aa085300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39246/0 (socket says 172.21.15.70:39246) 2024-06-04T05:18:07.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f327b7fe640 1 -- 172.21.15.70:0/53634056 learned_addr learned my addr 172.21.15.70:0/53634056 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:07.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f327bfff640 1 --2- 172.21.15.70:0/53634056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa07c820 0x55f4aa078730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f3280ca0640 1 --2- 172.21.15.70:0/53634056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa087a00 0x55f4aa09fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f327b7fe640 1 -- 172.21.15.70:0/53634056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa07c820 msgr2=0x55f4aa078730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:07.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f327b7fe640 1 --2- 172.21.15.70:0/53634056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa07c820 0x55f4aa078730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f327b7fe640 1 -- 172.21.15.70:0/53634056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa087a00 msgr2=0x55f4aa09fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:07.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f327b7fe640 1 --2- 172.21.15.70:0/53634056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa087a00 0x55f4aa09fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f327b7fe640 1 -- 172.21.15.70:0/53634056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4aa0746f0 con 0x55f4aa07d140 2024-06-04T05:18:07.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f327bfff640 1 --2- 172.21.15.70:0/53634056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa07c820 0x55f4aa078730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:07.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.466+0000 7f327b7fe640 1 --2- 172.21.15.70:0/53634056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 0x55f4aa085300 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f3270009fd0 tx=0x7f3270005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e21e0d753d967ce5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:07.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.467+0000 7f3261ffb640 1 -- 172.21.15.70:0/53634056 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3270005220 con 0x55f4aa07d140 2024-06-04T05:18:07.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.467+0000 7f3261ffb640 1 -- 172.21.15.70:0/53634056 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3270004c30 con 0x55f4aa07d140 2024-06-04T05:18:07.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.467+0000 7f3261ffb640 1 -- 172.21.15.70:0/53634056 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32700108c0 con 0x55f4aa07d140 2024-06-04T05:18:07.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.467+0000 7f3285418740 1 -- 172.21.15.70:0/53634056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 msgr2=0x55f4aa085300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:07.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.467+0000 7f3285418740 1 --2- 172.21.15.70:0/53634056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 0x55f4aa085300 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f3270009fd0 tx=0x7f3270005f90 comp rx=0 tx=0).stop 2024-06-04T05:18:07.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.467+0000 7f3285418740 1 -- 172.21.15.70:0/53634056 shutdown_connections 2024-06-04T05:18:07.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.467+0000 7f3285418740 1 --2- 172.21.15.70:0/53634056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa087a00 0x55f4aa09fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.467+0000 7f3285418740 1 --2- 172.21.15.70:0/53634056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07d140 0x55f4aa085300 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.467+0000 7f3285418740 1 --2- 172.21.15.70:0/53634056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa07c820 0x55f4aa078730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.467+0000 7f3285418740 1 -- 172.21.15.70:0/53634056 >> 172.21.15.70:0/53634056 conn(0x55f4aa078d30 msgr2=0x55f4aa07ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:07.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.467+0000 7f3285418740 1 -- 172.21.15.70:0/53634056 shutdown_connections 2024-06-04T05:18:07.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.467+0000 7f3285418740 1 -- 172.21.15.70:0/53634056 wait complete. 2024-06-04T05:18:07.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.468+0000 7f3285418740 1 -- start start 2024-06-04T05:18:07.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.468+0000 7f3285418740 1 Processor -- start 2024-06-04T05:18:07.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.468+0000 7f3285418740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07c820 0x55f4aa0900e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.468+0000 7f3285418740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa07d140 0x55f4aa08b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.468+0000 7f3285418740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa087a00 0x55f4aa08bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.468+0000 7f3285418740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f327c1b1d30 con 0x55f4aa07c820 2024-06-04T05:18:07.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.468+0000 7f3285418740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f327c1b1660 con 0x55f4aa07d140 2024-06-04T05:18:07.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.468+0000 7f3285418740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f327c1ae860 con 0x55f4aa087a00 2024-06-04T05:18:07.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.468+0000 7f3285418740 10 client.0 _ll_register_callbacks cb 0x7f326c00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.468+0000 7f327b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa07d140 0x55f4aa08b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.468+0000 7f327bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07c820 0x55f4aa0900e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.468+0000 7f327b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa07d140 0x55f4aa08b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55968/0 (socket says 172.21.15.70:55968) 2024-06-04T05:18:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.468+0000 7f327b7fe640 1 -- 172.21.15.70:0/733660458 learned_addr learned my addr 172.21.15.70:0/733660458 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.468+0000 7f3280ca0640 1 --2- 172.21.15.70:0/733660458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa087a00 0x55f4aa08bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.469+0000 7f327b7fe640 1 -- 172.21.15.70:0/733660458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa087a00 msgr2=0x55f4aa08bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.469+0000 7f327b7fe640 1 --2- 172.21.15.70:0/733660458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa087a00 0x55f4aa08bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.469+0000 7f327b7fe640 1 -- 172.21.15.70:0/733660458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07c820 msgr2=0x55f4aa0900e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.469+0000 7f327b7fe640 1 --2- 172.21.15.70:0/733660458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07c820 0x55f4aa0900e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.469+0000 7f327b7fe640 1 -- 172.21.15.70:0/733660458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4aa101a00 con 0x55f4aa07d140 2024-06-04T05:18:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.469+0000 7f327b7fe640 1 --2- 172.21.15.70:0/733660458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa07d140 0x55f4aa08b4a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3270009460 tx=0x7f3270004de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b75ffa801ef0b249 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.469+0000 7f325b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:18:07.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.469+0000 7f325b7fe640 1 -- 172.21.15.70:0/733660458 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3270004940 con 0x55f4aa07d140 2024-06-04T05:18:07.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.469+0000 7f3285418740 1 -- 172.21.15.70:0/733660458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3270004c30 con 0x55f4aa07d140 2024-06-04T05:18:07.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.469+0000 7f325b7fe640 1 -- 172.21.15.70:0/733660458 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8615+0+0 (secure 0 0 0) 0x7f3270005020 con 0x55f4aa07d140 2024-06-04T05:18:07.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.469+0000 7f325b7fe640 1 -- 172.21.15.70:0/733660458 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f3270005220 con 0x55f4aa07d140 2024-06-04T05:18:07.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.469+0000 7f3285418740 10 client.29540 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:07.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.469+0000 7f3285418740 1 -- 172.21.15.70:0/733660458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55f4aa102bb0 con 0x55f4aa07d140 2024-06-04T05:18:07.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.469+0000 7f325b7fe640 1 -- 172.21.15.70:0/733660458 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3270004c30 con 0x55f4aa07d140 2024-06-04T05:18:07.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.470+0000 7f325b7fe640 1 -- 172.21.15.70:0/733660458 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f3270028450 con 0x55f4aa07d140 2024-06-04T05:18:07.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.470+0000 7f3285418740 10 client.29540 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:07.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.470+0000 7f3285418740 10 client.29540 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:18:07.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.470+0000 7f3285418740 20 client.29540 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:07.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.470+0000 7f3285418740 1 --2- 172.21.15.70:0/733660458 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f4aa1032e0 0x55f4aa105700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:07.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.470+0000 7f3285418740 4 client.29540 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:18:07.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.470+0000 7f3285418740 1 -- 172.21.15.70:0/733660458 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f4a9fea360 con 0x55f4aa1032e0 2024-06-04T05:18:07.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.470+0000 7f327bfff640 1 --2- 172.21.15.70:0/733660458 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f4aa1032e0 0x55f4aa105700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:07.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.470+0000 7f325b7fe640 10 client.29540 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:07.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.470+0000 7f327bfff640 1 --2- 172.21.15.70:0/733660458 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f4aa1032e0 0x55f4aa105700 crc :-1 s=READY pgs=442 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e577e764b01f500 server_cookie=1561d0123bbff46f in_seq=0 out_seq=0 2024-06-04T05:18:07.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.563+0000 7f325b7fe640 1 -- 172.21.15.70:0/733660458 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f4a9fea360 con 0x55f4aa1032e0 2024-06-04T05:18:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.564+0000 7f325b7fe640 10 client.29540 handle_command_reply: tid=0 2024-06-04T05:18:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.564+0000 7f3285418740 1 client.29540 shutdown 2024-06-04T05:18:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.564+0000 7f3285418740 1 -- 172.21.15.70:0/733660458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa07d140 msgr2=0x55f4aa08b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.564+0000 7f3285418740 1 --2- 172.21.15.70:0/733660458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa07d140 0x55f4aa08b4a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3270009460 tx=0x7f3270004de0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.564+0000 7f3285418740 1 -- 172.21.15.70:0/733660458 shutdown_connections 2024-06-04T05:18:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.564+0000 7f3285418740 1 --2- 172.21.15.70:0/733660458 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f4aa1032e0 0x55f4aa105700 crc :-1 s=READY pgs=442 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.564+0000 7f3285418740 1 --2- 172.21.15.70:0/733660458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4aa07d140 0x55f4aa08b4a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.564+0000 7f3285418740 1 --2- 172.21.15.70:0/733660458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f4aa087a00 0x55f4aa08bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.564+0000 7f325b7fe640 0 client.29540 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.564+0000 7f3285418740 1 --2- 172.21.15.70:0/733660458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4aa07c820 0x55f4aa0900e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.564+0000 7f3285418740 1 -- 172.21.15.70:0/733660458 >> 172.21.15.70:0/733660458 conn(0x55f4aa078d30 msgr2=0x55f4aa092680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.564+0000 7f3285418740 1 -- 172.21.15.70:0/733660458 shutdown_connections 2024-06-04T05:18:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.564+0000 7f3285418740 1 -- 172.21.15.70:0/733660458 wait complete. 2024-06-04T05:18:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.564+0000 7f3285418740 20 client.29540 trim_cache size 0 max 16384 2024-06-04T05:18:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:18:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:18:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:18:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.565+0000 7f3282703640 1 -- 172.21.15.70:0/2339307846 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f32640b2a10 msgr2=0x7f32640b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.565+0000 7f3282703640 1 --2- 172.21.15.70:0/2339307846 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f32640b2a10 0x7f32640b4ed0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3274005e30 tx=0x7f327404c000 comp rx=0 tx=0).stop 2024-06-04T05:18:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.565+0000 7f3282703640 1 -- 172.21.15.70:0/2339307846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f327c101980 msgr2=0x7f327c19b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.565+0000 7f3282703640 1 --2- 172.21.15.70:0/2339307846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f327c101980 0x7f327c19b290 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f326c00e9f0 tx=0x7f326c00eeb0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.565+0000 7f3282703640 1 -- 172.21.15.70:0/2339307846 shutdown_connections 2024-06-04T05:18:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.565+0000 7f3282703640 1 --2- 172.21.15.70:0/2339307846 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f32640b2a10 0x7f32640b4ed0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.565+0000 7f3282703640 1 --2- 172.21.15.70:0/2339307846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f327c102340 0x7f327c19b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.565+0000 7f3282703640 1 --2- 172.21.15.70:0/2339307846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f327c107940 0x7f327c19fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.565+0000 7f3282703640 1 --2- 172.21.15.70:0/2339307846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f327c101980 0x7f327c19b290 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.565+0000 7f3282703640 1 -- 172.21.15.70:0/2339307846 >> 172.21.15.70:0/2339307846 conn(0x7f327c078ad0 msgr2=0x7f327c1042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.565+0000 7f3282703640 1 -- 172.21.15.70:0/2339307846 shutdown_connections 2024-06-04T05:18:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:07.565+0000 7f3282703640 1 -- 172.21.15.70:0/2339307846 wait complete. 2024-06-04T05:18:07.823 INFO:tasks.workunit.client.0.smithi070.stdout: 10 55860 7.04 MB/sec execute 299 sec latency 121.239 ms 2024-06-04T05:18:07.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:07 smithi112 ceph-mon[36453]: pgmap v658: 97 pgs: 97 active+clean; 810 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 307 op/s 2024-06-04T05:18:07.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:07 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:18:07.938 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:07 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:18:07.939 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:07 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:18:07.939 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:07 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:18:07.939 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:07 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:18:07.939 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:07 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:18:07.939 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:07 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:18:07.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:07 smithi098 ceph-mon[35569]: pgmap v658: 97 pgs: 97 active+clean; 810 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 307 op/s 2024-06-04T05:18:07.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:07 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:18:07.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:07 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:18:07.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:07 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:18:07.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:07 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:18:07.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:07 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:18:07.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:07 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:18:07.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:07 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:18:08.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:07 smithi070 ceph-mon[32631]: pgmap v658: 97 pgs: 97 active+clean; 810 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 307 op/s 2024-06-04T05:18:08.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:07 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:18:08.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:07 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:18:08.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:07 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:18:08.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:07 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-04T05:18:08.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:07 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:18:08.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:07 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:18:08.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:07 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:18:08.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 56011 7.05 MB/sec execute 300 sec latency 136.878 ms 2024-06-04T05:18:08.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:08 smithi112 ceph-mon[36453]: Adjusting osd_memory_target on smithi070 to 176.1M 2024-06-04T05:18:08.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:08 smithi112 ceph-mon[36453]: Unable to set osd_memory_target on smithi070 to 184717209: error parsing value: Value '184717209' is below minimum 939524096 2024-06-04T05:18:08.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:08 smithi112 ceph-mon[36453]: osdmap e110: 12 total, 12 up, 12 in 2024-06-04T05:18:08.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:08 smithi098 ceph-mon[35569]: Adjusting osd_memory_target on smithi070 to 176.1M 2024-06-04T05:18:08.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:08 smithi098 ceph-mon[35569]: Unable to set osd_memory_target on smithi070 to 184717209: error parsing value: Value '184717209' is below minimum 939524096 2024-06-04T05:18:08.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:08 smithi098 ceph-mon[35569]: osdmap e110: 12 total, 12 up, 12 in 2024-06-04T05:18:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:08 smithi070 ceph-mon[32631]: Adjusting osd_memory_target on smithi070 to 176.1M 2024-06-04T05:18:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:08 smithi070 ceph-mon[32631]: Unable to set osd_memory_target on smithi070 to 184717209: error parsing value: Value '184717209' is below minimum 939524096 2024-06-04T05:18:09.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:08 smithi070 ceph-mon[32631]: osdmap e110: 12 total, 12 up, 12 in 2024-06-04T05:18:09.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 56202 7.07 MB/sec execute 301 sec latency 137.922 ms 2024-06-04T05:18:09.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:09 smithi112 ceph-mon[36453]: pgmap v660: 97 pgs: 97 active+clean; 810 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 384 op/s 2024-06-04T05:18:09.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:09 smithi098 ceph-mon[35569]: pgmap v660: 97 pgs: 97 active+clean; 810 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 384 op/s 2024-06-04T05:18:10.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:09 smithi070 ceph-mon[32631]: pgmap v660: 97 pgs: 97 active+clean; 810 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 384 op/s 2024-06-04T05:18:10.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 56394 7.10 MB/sec execute 302 sec latency 174.582 ms 2024-06-04T05:18:11.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 56563 7.11 MB/sec execute 303 sec latency 82.772 ms 2024-06-04T05:18:11.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:11 smithi098 ceph-mon[35569]: pgmap v661: 97 pgs: 97 active+clean; 810 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 158 op/s 2024-06-04T05:18:12.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:11 smithi070 ceph-mon[32631]: pgmap v661: 97 pgs: 97 active+clean; 810 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 158 op/s 2024-06-04T05:18:12.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:11 smithi112 ceph-mon[36453]: pgmap v661: 97 pgs: 97 active+clean; 810 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 158 op/s 2024-06-04T05:18:12.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 56787 7.14 MB/sec execute 304 sec latency 63.706 ms 2024-06-04T05:18:12.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.872+0000 7fe194eb1700 1 -- 172.21.15.70:0/504363483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe190102430 msgr2=0x7fe1901028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:12.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.872+0000 7fe194eb1700 1 --2- 172.21.15.70:0/504363483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe190102430 0x7fe1901028b0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fe184009990 tx=0x7fe184009ca0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.872+0000 7fe194eb1700 1 -- 172.21.15.70:0/504363483 shutdown_connections 2024-06-04T05:18:12.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.872+0000 7fe194eb1700 1 --2- 172.21.15.70:0/504363483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe190102df0 0x7fe19010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.872+0000 7fe194eb1700 1 --2- 172.21.15.70:0/504363483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe190108510 0x7fe1901088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.872+0000 7fe194eb1700 1 --2- 172.21.15.70:0/504363483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe190102430 0x7fe1901028b0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.872+0000 7fe194eb1700 1 -- 172.21.15.70:0/504363483 >> 172.21.15.70:0/504363483 conn(0x7fe1900793b0 msgr2=0x7fe1900797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:12.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.873+0000 7fe194eb1700 1 -- 172.21.15.70:0/504363483 shutdown_connections 2024-06-04T05:18:12.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.873+0000 7fe194eb1700 1 -- 172.21.15.70:0/504363483 wait complete. 2024-06-04T05:18:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.874+0000 7fe194eb1700 1 Processor -- start 2024-06-04T05:18:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.874+0000 7fe194eb1700 1 -- start start 2024-06-04T05:18:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.874+0000 7fe194eb1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe190102df0 0x7fe190199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.874+0000 7fe194eb1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe190108510 0x7fe19019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.874+0000 7fe194eb1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe19019e670 0x7fe19019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.874+0000 7fe194eb1700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe190113610 con 0x7fe190102df0 2024-06-04T05:18:12.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.874+0000 7fe194eb1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe190113490 con 0x7fe19019e670 2024-06-04T05:18:12.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.874+0000 7fe194eb1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe190113790 con 0x7fe190108510 2024-06-04T05:18:12.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.874+0000 7fe18ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe19019e670 0x7fe19019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:12.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.874+0000 7fe18ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe19019e670 0x7fe19019eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55988/0 (socket says 172.21.15.70:55988) 2024-06-04T05:18:12.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.874+0000 7fe18e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe190102df0 0x7fe190199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:12.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.874+0000 7fe18ed9d700 1 -- 172.21.15.70:0/1067620759 learned_addr learned my addr 172.21.15.70:0/1067620759 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:12.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.874+0000 7fe18dd9b700 1 --2- 172.21.15.70:0/1067620759 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe190108510 0x7fe19019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:12.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.875+0000 7fe18e59c700 1 -- 172.21.15.70:0/1067620759 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe190108510 msgr2=0x7fe19019a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:12.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.875+0000 7fe18e59c700 1 --2- 172.21.15.70:0/1067620759 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe190108510 0x7fe19019a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.875+0000 7fe18e59c700 1 -- 172.21.15.70:0/1067620759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe19019e670 msgr2=0x7fe19019eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:12.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.875+0000 7fe18e59c700 1 --2- 172.21.15.70:0/1067620759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe19019e670 0x7fe19019eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.875+0000 7fe18e59c700 1 -- 172.21.15.70:0/1067620759 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe19019f190 con 0x7fe190102df0 2024-06-04T05:18:12.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.875+0000 7fe18ed9d700 1 --2- 172.21.15.70:0/1067620759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe19019e670 0x7fe19019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:12.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.875+0000 7fe18e59c700 1 --2- 172.21.15.70:0/1067620759 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe190102df0 0x7fe190199da0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fe18000e9f0 tx=0x7fe18000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:12.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.875+0000 7fe17f7fe700 1 -- 172.21.15.70:0/1067620759 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18000cbd0 con 0x7fe190102df0 2024-06-04T05:18:12.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.875+0000 7fe17f7fe700 1 -- 172.21.15.70:0/1067620759 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1800044b0 con 0x7fe190102df0 2024-06-04T05:18:12.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.875+0000 7fe17f7fe700 1 -- 172.21.15.70:0/1067620759 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe180005020 con 0x7fe190102df0 2024-06-04T05:18:12.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.875+0000 7fe194eb1700 1 -- 172.21.15.70:0/1067620759 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe19019f480 con 0x7fe190102df0 2024-06-04T05:18:12.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.875+0000 7fe194eb1700 1 -- 172.21.15.70:0/1067620759 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1901a6c10 con 0x7fe190102df0 2024-06-04T05:18:12.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.880+0000 7fe17f7fe700 1 -- 172.21.15.70:0/1067620759 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fe1800051c0 con 0x7fe190102df0 2024-06-04T05:18:12.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.880+0000 7fe19f746b80 1 Processor -- start 2024-06-04T05:18:12.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.881+0000 7fe19f746b80 1 -- start start 2024-06-04T05:18:12.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.881+0000 7fe19f746b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da45320 0x56385da45700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.881+0000 7fe19f746b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da3f8a0 0x56385da3fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.881+0000 7fe19f746b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da40260 0x56385da4c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.881+0000 7fe19f746b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56385d99dd70 con 0x56385da45320 2024-06-04T05:18:12.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.881+0000 7fe19f746b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56385d98d910 con 0x56385da40260 2024-06-04T05:18:12.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.881+0000 7fe19f746b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56385d991950 con 0x56385da3f8a0 2024-06-04T05:18:12.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.881+0000 7fe18ed9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da45320 0x56385da45700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:12.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.881+0000 7fe18ed9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da45320 0x56385da45700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39282/0 (socket says 172.21.15.70:39282) 2024-06-04T05:18:12.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.881+0000 7fe18ed9d700 1 -- 172.21.15.70:0/1507618799 learned_addr learned my addr 172.21.15.70:0/1507618799 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:12.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.881+0000 7fe18dd9b700 1 --2- 172.21.15.70:0/1507618799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da40260 0x56385da4c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:12.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.881+0000 7fe18ed9d700 1 -- 172.21.15.70:0/1507618799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da3f8a0 msgr2=0x56385da3fd20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-04T05:18:12.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.881+0000 7fe18ed9d700 1 --2- 172.21.15.70:0/1507618799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da3f8a0 0x56385da3fd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.881+0000 7fe18ed9d700 1 -- 172.21.15.70:0/1507618799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da40260 msgr2=0x56385da4c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:12.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.881+0000 7fe18ed9d700 1 --2- 172.21.15.70:0/1507618799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da40260 0x56385da4c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.881+0000 7fe18ed9d700 1 -- 172.21.15.70:0/1507618799 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56385d936fb0 con 0x56385da45320 2024-06-04T05:18:12.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.882+0000 7fe18ed9d700 1 --2- 172.21.15.70:0/1507618799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da45320 0x56385da45700 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fe188009500 tx=0x7fe188009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a65321ab1e44f43a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:12.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.882+0000 7fe17cff9700 1 -- 172.21.15.70:0/1507618799 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18801b020 con 0x56385da45320 2024-06-04T05:18:12.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.882+0000 7fe17cff9700 1 -- 172.21.15.70:0/1507618799 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe188007e50 con 0x56385da45320 2024-06-04T05:18:12.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.882+0000 7fe17cff9700 1 -- 172.21.15.70:0/1507618799 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe188017c50 con 0x56385da45320 2024-06-04T05:18:12.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.882+0000 7fe17f7fe700 1 --2- 172.21.15.70:0/1067620759 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1780b2af0 0x7fe1780b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.883+0000 7fe18e59c700 1 --2- 172.21.15.70:0/1067620759 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1780b2af0 0x7fe1780b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:12.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.883+0000 7fe19f746b80 1 -- 172.21.15.70:0/1507618799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da45320 msgr2=0x56385da45700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:12.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.883+0000 7fe19f746b80 1 --2- 172.21.15.70:0/1507618799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da45320 0x56385da45700 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fe188009500 tx=0x7fe188009920 comp rx=0 tx=0).stop 2024-06-04T05:18:12.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.883+0000 7fe19f746b80 1 -- 172.21.15.70:0/1507618799 shutdown_connections 2024-06-04T05:18:12.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.883+0000 7fe19f746b80 1 --2- 172.21.15.70:0/1507618799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da40260 0x56385da4c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.883+0000 7fe19f746b80 1 --2- 172.21.15.70:0/1507618799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da45320 0x56385da45700 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.883+0000 7fe19f746b80 1 --2- 172.21.15.70:0/1507618799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da3f8a0 0x56385da3fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.883+0000 7fe19f746b80 1 -- 172.21.15.70:0/1507618799 >> 172.21.15.70:0/1507618799 conn(0x56385d96b240 msgr2=0x56385da3a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:12.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.883+0000 7fe19f746b80 1 -- 172.21.15.70:0/1507618799 shutdown_connections 2024-06-04T05:18:12.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.883+0000 7fe18e59c700 1 --2- 172.21.15.70:0/1067620759 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1780b2af0 0x7fe1780b4fb0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe1800a6930 tx=0x7fe1800a5580 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:12.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.883+0000 7fe19f746b80 1 -- 172.21.15.70:0/1507618799 wait complete. 2024-06-04T05:18:12.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.884+0000 7fe19f746b80 1 -- start start 2024-06-04T05:18:12.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.884+0000 7fe19f746b80 1 Processor -- start 2024-06-04T05:18:12.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.884+0000 7fe19f746b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da3f8a0 0x56385da4bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.884+0000 7fe19f746b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da40260 0x56385da48f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.884+0000 7fe19f746b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da45320 0x56385da494c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.884+0000 7fe18dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da40260 0x56385da48f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:12.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.884+0000 7fe18dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da40260 0x56385da48f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55996/0 (socket says 172.21.15.70:55996) 2024-06-04T05:18:12.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.884+0000 7fe18dd9b700 1 -- 172.21.15.70:0/2263502522 learned_addr learned my addr 172.21.15.70:0/2263502522 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:12.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.884+0000 7fe19f746b80 1 -- 172.21.15.70:0/2263502522 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56385d893fa0 con 0x56385da45320 2024-06-04T05:18:12.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.884+0000 7fe19f746b80 1 -- 172.21.15.70:0/2263502522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56385d8f5da0 con 0x56385da40260 2024-06-04T05:18:12.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.884+0000 7fe19f746b80 1 -- 172.21.15.70:0/2263502522 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56385da4c6e0 con 0x56385da3f8a0 2024-06-04T05:18:12.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.884+0000 7fe18ed9d700 1 --2- 172.21.15.70:0/2263502522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da3f8a0 0x56385da4bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:12.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.884+0000 7fe17f7fe700 1 -- 172.21.15.70:0/1067620759 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7fe1800e2470 con 0x7fe190102df0 2024-06-04T05:18:12.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.885+0000 7fe18ed9d700 1 --2- 172.21.15.70:0/2263502522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da45320 0x56385da494c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:12.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.885+0000 7fe18dd9b700 1 -- 172.21.15.70:0/2263502522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da3f8a0 msgr2=0x56385da4bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:12.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.885+0000 7fe18dd9b700 1 --2- 172.21.15.70:0/2263502522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da3f8a0 0x56385da4bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.885+0000 7fe18dd9b700 1 -- 172.21.15.70:0/2263502522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da45320 msgr2=0x56385da494c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:12.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.885+0000 7fe19f746b80 10 client.0 _ll_register_callbacks cb 0x56385d928950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:12.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.885+0000 7fe18dd9b700 1 --2- 172.21.15.70:0/2263502522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da45320 0x56385da494c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.885+0000 7fe18dd9b700 1 -- 172.21.15.70:0/2263502522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56385d8dd8d0 con 0x56385da40260 2024-06-04T05:18:12.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.885+0000 7fe18ed9d700 1 --2- 172.21.15.70:0/2263502522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da3f8a0 0x56385da4bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:18:12.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.885+0000 7fe19f746b80 1 -- 172.21.15.70:0/2263502522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe188007e50 con 0x56385da40260 2024-06-04T05:18:12.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.885+0000 7fe18dd9b700 1 --2- 172.21.15.70:0/2263502522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da40260 0x56385da48f80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe1901034d0 tx=0x7fe184005900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f787b50afee2c638 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:12.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.885+0000 7fe175ffb700 10 client.29552 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:18:12.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.885+0000 7fe175ffb700 1 -- 172.21.15.70:0/2263502522 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18404c060 con 0x56385da40260 2024-06-04T05:18:12.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.885+0000 7fe175ffb700 1 -- 172.21.15.70:0/2263502522 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7fe18400bbb0 con 0x56385da40260 2024-06-04T05:18:12.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.885+0000 7fe175ffb700 1 -- 172.21.15.70:0/2263502522 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fe18400bdb0 con 0x56385da40260 2024-06-04T05:18:12.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.885+0000 7fe19f746b80 10 client.29552 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:12.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.885+0000 7fe19f746b80 1 -- 172.21.15.70:0/2263502522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({fsmap=59}) -- 0x56385d947760 con 0x56385da40260 2024-06-04T05:18:12.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.886+0000 7fe175ffb700 1 -- 172.21.15.70:0/2263502522 <== mon.2 v2:172.21.15.98:3300/0 4 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7fe184003680 con 0x56385da40260 2024-06-04T05:18:12.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.886+0000 7fe19f746b80 10 client.29552 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:12.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.886+0000 7fe19f746b80 10 client.29552 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:18:12.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.886+0000 7fe19f746b80 20 client.29552 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:12.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.886+0000 7fe19f746b80 1 --2- 172.21.15.70:0/2263502522 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56385da69fa0 0x56385da6c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.886+0000 7fe19f746b80 4 client.29552 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:18:12.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.886+0000 7fe19f746b80 1 -- 172.21.15.70:0/2263502522 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56385d99fcc0 con 0x56385da69fa0 2024-06-04T05:18:12.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.886+0000 7fe18ed9d700 1 --2- 172.21.15.70:0/2263502522 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56385da69fa0 0x56385da6c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:12.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.886+0000 7fe18ed9d700 1 --2- 172.21.15.70:0/2263502522 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56385da69fa0 0x56385da6c440 crc :-1 s=READY pgs=443 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48df260cc0fab675 server_cookie=c5732eda0448740a in_seq=0 out_seq=0 2024-06-04T05:18:12.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.886+0000 7fe175ffb700 10 client.29552 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:12.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.887+0000 7fe175ffb700 1 -- 172.21.15.70:0/2263502522 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56385d99fcc0 con 0x56385da69fa0 2024-06-04T05:18:12.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.887+0000 7fe175ffb700 10 client.29552 handle_command_reply: tid=0 2024-06-04T05:18:12.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.887+0000 7fe19f746b80 1 client.29552 shutdown 2024-06-04T05:18:12.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.887+0000 7fe19f746b80 1 -- 172.21.15.70:0/2263502522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da40260 msgr2=0x56385da48f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:12.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.887+0000 7fe19f746b80 1 --2- 172.21.15.70:0/2263502522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da40260 0x56385da48f80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe1901034d0 tx=0x7fe184005900 comp rx=0 tx=0).stop 2024-06-04T05:18:12.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.888+0000 7fe19f746b80 1 -- 172.21.15.70:0/2263502522 shutdown_connections 2024-06-04T05:18:12.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.888+0000 7fe19f746b80 1 --2- 172.21.15.70:0/2263502522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da45320 0x56385da494c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.888+0000 7fe19f746b80 1 --2- 172.21.15.70:0/2263502522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da3f8a0 0x56385da4bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.888+0000 7fe19f746b80 1 --2- 172.21.15.70:0/2263502522 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56385da69fa0 0x56385da6c440 crc :-1 s=READY pgs=443 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.888+0000 7fe19f746b80 1 --2- 172.21.15.70:0/2263502522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da40260 0x56385da48f80 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.888+0000 7fe175ffb700 0 client.29552 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:12.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.888+0000 7fe19f746b80 1 -- 172.21.15.70:0/2263502522 >> 172.21.15.70:0/2263502522 conn(0x56385d96b240 msgr2=0x56385da39af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:12.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.888+0000 7fe19f746b80 1 -- 172.21.15.70:0/2263502522 shutdown_connections 2024-06-04T05:18:12.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.888+0000 7fe19f746b80 1 -- 172.21.15.70:0/2263502522 wait complete. 2024-06-04T05:18:12.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.888+0000 7fe19f746b80 20 client.29552 trim_cache size 0 max 16384 2024-06-04T05:18:12.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.915+0000 7fe19f746b80 1 Processor -- start 2024-06-04T05:18:12.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.916+0000 7fe19f746b80 1 -- start start 2024-06-04T05:18:12.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.916+0000 7fe19f746b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da3f8a0 0x56385da3fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.916+0000 7fe19f746b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da40260 0x56385da435e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.916+0000 7fe19f746b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da45320 0x56385da3a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.916+0000 7fe19f746b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe17000d590 con 0x56385da40260 2024-06-04T05:18:12.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.916+0000 7fe19f746b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe17000f130 con 0x56385da3f8a0 2024-06-04T05:18:12.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.916+0000 7fe19f746b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe17000dcc0 con 0x56385da45320 2024-06-04T05:18:12.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.916+0000 7fe18ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da3f8a0 0x56385da3fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:12.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.916+0000 7fe18ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da3f8a0 0x56385da3fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:56006/0 (socket says 172.21.15.70:56006) 2024-06-04T05:18:12.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.916+0000 7fe18ed9d700 1 -- 172.21.15.70:0/2391054996 learned_addr learned my addr 172.21.15.70:0/2391054996 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:12.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.916+0000 7fe18ed9d700 1 --2- 172.21.15.70:0/2391054996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da45320 0x56385da3a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:12.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.916+0000 7fe18dd9b700 1 --2- 172.21.15.70:0/2391054996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da40260 0x56385da435e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:12.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.916+0000 7fe18ed9d700 1 -- 172.21.15.70:0/2391054996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da45320 msgr2=0x56385da3a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:12.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.916+0000 7fe18ed9d700 1 --2- 172.21.15.70:0/2391054996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da45320 0x56385da3a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.916+0000 7fe18ed9d700 1 -- 172.21.15.70:0/2391054996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da40260 msgr2=0x56385da435e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:12.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.916+0000 7fe18ed9d700 1 --2- 172.21.15.70:0/2391054996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da40260 0x56385da435e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.916+0000 7fe18ed9d700 1 -- 172.21.15.70:0/2391054996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56385da47510 con 0x56385da3f8a0 2024-06-04T05:18:12.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.917+0000 7fe18ed9d700 1 --2- 172.21.15.70:0/2391054996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da3f8a0 0x56385da3fce0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe188009f00 tx=0x7fe188009e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3aa77f662a9d0698 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:12.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.917+0000 7fe177fff700 1 -- 172.21.15.70:0/2391054996 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18801b040 con 0x56385da3f8a0 2024-06-04T05:18:12.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.917+0000 7fe177fff700 1 -- 172.21.15.70:0/2391054996 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe18800e700 con 0x56385da3f8a0 2024-06-04T05:18:12.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.917+0000 7fe177fff700 1 -- 172.21.15.70:0/2391054996 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe188005280 con 0x56385da3f8a0 2024-06-04T05:18:12.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.918+0000 7fe19f746b80 1 -- 172.21.15.70:0/2391054996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da3f8a0 msgr2=0x56385da3fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:12.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.918+0000 7fe19f746b80 1 --2- 172.21.15.70:0/2391054996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da3f8a0 0x56385da3fce0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe188009f00 tx=0x7fe188009e90 comp rx=0 tx=0).stop 2024-06-04T05:18:12.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.918+0000 7fe19f746b80 1 -- 172.21.15.70:0/2391054996 shutdown_connections 2024-06-04T05:18:12.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.918+0000 7fe19f746b80 1 --2- 172.21.15.70:0/2391054996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da45320 0x56385da3a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.918+0000 7fe19f746b80 1 --2- 172.21.15.70:0/2391054996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da3f8a0 0x56385da3fce0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.918+0000 7fe19f746b80 1 --2- 172.21.15.70:0/2391054996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da40260 0x56385da435e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.918+0000 7fe19f746b80 1 -- 172.21.15.70:0/2391054996 >> 172.21.15.70:0/2391054996 conn(0x56385d96b240 msgr2=0x56385da44720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:12.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.918+0000 7fe19f746b80 1 -- 172.21.15.70:0/2391054996 shutdown_connections 2024-06-04T05:18:12.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.918+0000 7fe19f746b80 1 -- 172.21.15.70:0/2391054996 wait complete. 2024-06-04T05:18:12.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.918+0000 7fe19f746b80 1 -- start start 2024-06-04T05:18:12.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe19f746b80 1 Processor -- start 2024-06-04T05:18:12.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe19f746b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da3f8a0 0x56385da632c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe19f746b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da40260 0x56385da71750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe19f746b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da45320 0x56385da71e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe19f746b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe17000ea60 con 0x56385da40260 2024-06-04T05:18:12.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe19f746b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe17000e390 con 0x56385da3f8a0 2024-06-04T05:18:12.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe19f746b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe17000b550 con 0x56385da45320 2024-06-04T05:18:12.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe19f746b80 10 client.0 _ll_register_callbacks cb 0x56385d928950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:12.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe18ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da3f8a0 0x56385da632c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:12.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe18ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da3f8a0 0x56385da632c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:56014/0 (socket says 172.21.15.70:56014) 2024-06-04T05:18:12.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe18ed9d700 1 -- 172.21.15.70:0/2707427142 learned_addr learned my addr 172.21.15.70:0/2707427142 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:12.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe18ed9d700 1 --2- 172.21.15.70:0/2707427142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da45320 0x56385da71e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:12.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe18dd9b700 1 --2- 172.21.15.70:0/2707427142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da40260 0x56385da71750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:12.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe18ed9d700 1 -- 172.21.15.70:0/2707427142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da45320 msgr2=0x56385da71e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:12.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe18ed9d700 1 --2- 172.21.15.70:0/2707427142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da45320 0x56385da71e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe18ed9d700 1 -- 172.21.15.70:0/2707427142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da40260 msgr2=0x56385da71750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:12.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe18ed9d700 1 --2- 172.21.15.70:0/2707427142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da40260 0x56385da71750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.919+0000 7fe18ed9d700 1 -- 172.21.15.70:0/2707427142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56385dad3c30 con 0x56385da3f8a0 2024-06-04T05:18:12.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.920+0000 7fe18ed9d700 1 --2- 172.21.15.70:0/2707427142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da3f8a0 0x56385da632c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe188009960 tx=0x7fe188006160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8af0c1fd5d1606a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:12.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.920+0000 7fe175ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:18:12.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.920+0000 7fe175ffb700 1 -- 172.21.15.70:0/2707427142 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18801b040 con 0x56385da3f8a0 2024-06-04T05:18:12.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.920+0000 7fe175ffb700 1 -- 172.21.15.70:0/2707427142 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7fe188004c30 con 0x56385da3f8a0 2024-06-04T05:18:12.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.920+0000 7fe19f746b80 1 -- 172.21.15.70:0/2707427142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe18800e700 con 0x56385da3f8a0 2024-06-04T05:18:12.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.920+0000 7fe175ffb700 1 -- 172.21.15.70:0/2707427142 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fe188004e30 con 0x56385da3f8a0 2024-06-04T05:18:12.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.920+0000 7fe19f746b80 10 client.29558 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:12.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.920+0000 7fe19f746b80 1 -- 172.21.15.70:0/2707427142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x56385dad4df0 con 0x56385da3f8a0 2024-06-04T05:18:12.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.921+0000 7fe175ffb700 1 -- 172.21.15.70:0/2707427142 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe18800ed40 con 0x56385da3f8a0 2024-06-04T05:18:12.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.921+0000 7fe175ffb700 1 -- 172.21.15.70:0/2707427142 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7fe1880054d0 con 0x56385da3f8a0 2024-06-04T05:18:12.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.921+0000 7fe19f746b80 10 client.29558 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:12.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.921+0000 7fe19f746b80 10 client.29558 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:18:12.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.921+0000 7fe19f746b80 20 client.29558 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:12.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.921+0000 7fe19f746b80 1 --2- 172.21.15.70:0/2707427142 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56385dad5220 0x56385dad7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:12.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.921+0000 7fe19f746b80 4 client.29558 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:18:12.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.921+0000 7fe19f746b80 1 -- 172.21.15.70:0/2707427142 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fe18800e700 con 0x56385dad5220 2024-06-04T05:18:12.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.921+0000 7fe18dd9b700 1 --2- 172.21.15.70:0/2707427142 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56385dad5220 0x56385dad7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:12.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.922+0000 7fe175ffb700 10 client.29558 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:12.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.922+0000 7fe18dd9b700 1 --2- 172.21.15.70:0/2707427142 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56385dad5220 0x56385dad7640 crc :-1 s=READY pgs=444 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34cb72f0dea1461e server_cookie=f8bb47763bb81d07 in_seq=0 out_seq=0 2024-06-04T05:18:12.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.922+0000 7fe175ffb700 1 -- 172.21.15.70:0/2707427142 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fe18800e700 con 0x56385dad5220 2024-06-04T05:18:12.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.922+0000 7fe175ffb700 10 client.29558 handle_command_reply: tid=0 2024-06-04T05:18:12.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.922+0000 7fe19f746b80 1 client.29558 shutdown 2024-06-04T05:18:12.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.922+0000 7fe19f746b80 1 -- 172.21.15.70:0/2707427142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da3f8a0 msgr2=0x56385da632c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:12.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.922+0000 7fe19f746b80 1 --2- 172.21.15.70:0/2707427142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da3f8a0 0x56385da632c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe188009960 tx=0x7fe188006160 comp rx=0 tx=0).stop 2024-06-04T05:18:12.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.923+0000 7fe19f746b80 1 -- 172.21.15.70:0/2707427142 shutdown_connections 2024-06-04T05:18:12.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.923+0000 7fe19f746b80 1 --2- 172.21.15.70:0/2707427142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56385da45320 0x56385da71e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.923+0000 7fe19f746b80 1 --2- 172.21.15.70:0/2707427142 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56385dad5220 0x56385dad7640 crc :-1 s=READY pgs=444 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.923+0000 7fe19f746b80 1 --2- 172.21.15.70:0/2707427142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385da3f8a0 0x56385da632c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.923+0000 7fe19f746b80 1 --2- 172.21.15.70:0/2707427142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56385da40260 0x56385da71750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.923+0000 7fe19f746b80 1 -- 172.21.15.70:0/2707427142 >> 172.21.15.70:0/2707427142 conn(0x56385d96b240 msgr2=0x56385da3e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:12.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.923+0000 7fe175ffb700 0 client.29558 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:12.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.923+0000 7fe19f746b80 1 -- 172.21.15.70:0/2707427142 shutdown_connections 2024-06-04T05:18:12.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.923+0000 7fe19f746b80 1 -- 172.21.15.70:0/2707427142 wait complete. 2024-06-04T05:18:12.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.923+0000 7fe19f746b80 20 client.29558 trim_cache size 0 max 16384 2024-06-04T05:18:12.928 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:18:12.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.924+0000 7fe194eb1700 1 -- 172.21.15.70:0/1067620759 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1780b2af0 msgr2=0x7fe1780b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:12.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.924+0000 7fe194eb1700 1 --2- 172.21.15.70:0/1067620759 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1780b2af0 0x7fe1780b4fb0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe1800a6930 tx=0x7fe1800a5580 comp rx=0 tx=0).stop 2024-06-04T05:18:12.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.924+0000 7fe194eb1700 1 -- 172.21.15.70:0/1067620759 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe190102df0 msgr2=0x7fe190199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:12.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.924+0000 7fe194eb1700 1 --2- 172.21.15.70:0/1067620759 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe190102df0 0x7fe190199da0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fe18000e9f0 tx=0x7fe18000ed00 comp rx=0 tx=0).stop 2024-06-04T05:18:12.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.924+0000 7fe194eb1700 1 -- 172.21.15.70:0/1067620759 shutdown_connections 2024-06-04T05:18:12.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.924+0000 7fe194eb1700 1 --2- 172.21.15.70:0/1067620759 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe1780b2af0 0x7fe1780b4fb0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.924+0000 7fe194eb1700 1 --2- 172.21.15.70:0/1067620759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe19019e670 0x7fe19019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.924+0000 7fe194eb1700 1 --2- 172.21.15.70:0/1067620759 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe190102df0 0x7fe190199da0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.925+0000 7fe194eb1700 1 --2- 172.21.15.70:0/1067620759 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe190108510 0x7fe19019a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:12.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.925+0000 7fe194eb1700 1 -- 172.21.15.70:0/1067620759 >> 172.21.15.70:0/1067620759 conn(0x7fe1900793b0 msgr2=0x7fe1901006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:12.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.925+0000 7fe194eb1700 1 -- 172.21.15.70:0/1067620759 shutdown_connections 2024-06-04T05:18:12.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:12.925+0000 7fe194eb1700 1 -- 172.21.15.70:0/1067620759 wait complete. 2024-06-04T05:18:13.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 56934 7.15 MB/sec execute 305 sec latency 69.662 ms 2024-06-04T05:18:13.957 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:13 smithi070 ceph-mon[32631]: pgmap v662: 97 pgs: 97 active+clean; 832 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.3 MiB/s wr, 340 op/s 2024-06-04T05:18:13.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:13 smithi098 ceph-mon[35569]: pgmap v662: 97 pgs: 97 active+clean; 832 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.3 MiB/s wr, 340 op/s 2024-06-04T05:18:14.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:13 smithi112 ceph-mon[36453]: pgmap v662: 97 pgs: 97 active+clean; 832 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.3 MiB/s wr, 340 op/s 2024-06-04T05:18:14.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 57075 7.16 MB/sec execute 306 sec latency 59.683 ms 2024-06-04T05:18:14.969 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:18:14.969 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:18:15.321 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:18:15.795 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:15 smithi070 ceph-mon[32631]: pgmap v663: 97 pgs: 97 active+clean; 832 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 299 op/s 2024-06-04T05:18:15.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 57268 7.18 MB/sec execute 307 sec latency 59.140 ms 2024-06-04T05:18:15.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:15 smithi098 ceph-mon[35569]: pgmap v663: 97 pgs: 97 active+clean; 832 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 299 op/s 2024-06-04T05:18:16.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:15 smithi112 ceph-mon[36453]: pgmap v663: 97 pgs: 97 active+clean; 832 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 299 op/s 2024-06-04T05:18:16.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 57439 7.20 MB/sec execute 308 sec latency 58.224 ms 2024-06-04T05:18:17.686 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:18:17 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:18:17.362+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000020f [...25,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ auth{2=1,4=1} fragtree_t(*^2 00*^1 01*^1 10*^1) v52909 ap=2 f(v1922 m2024-06-04T05:17:58.723852+0000 7=7+0) n(v83 rc2024-06-04T05:17:58.723852+0000 b1462582 8=7+1) old_inodes=6 (inest sync->mix g=2,4) caps={15231=pAsLsXsFs/-@39967} | dirtyscattered=0 request=0 lock=0 importing=0 dirfrag=7 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 0x55d6de801700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000020f:[<0x10000000009/PWRPNT v234>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x1000000020f:[<0x10000000009/PWRPNT v52909>,<0x10000000008/~dmtmp v35887>,<0x10000000007/client0 v3068>,<0x10000000006/clients v2528>,<0x10000000005/tmp v1886>,<0x10000000003/client.0 v1828>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v1820>,<0x10000000001/sv_0 v1787>,<0x10000000000/qa v1758>,<0x1/volumes v1700>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:17:58.580313+0000 8=8+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:17:58.723852+0000 b1462582 8=7+1)","memory_value.dirstat":"f(v1922 m2024-06-04T05:17:58.723852+0000 7=7+0)","memory_value.rstat":"n(v83 rc2024-06-04T05:17:58.723852+0000 b1462582 8=7+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:18:17.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 57595 7.20 MB/sec execute 309 sec latency 164.297 ms 2024-06-04T05:18:17.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:17 smithi098 ceph-mon[35569]: pgmap v664: 97 pgs: 97 active+clean; 850 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.3 MiB/s wr, 299 op/s 2024-06-04T05:18:17.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:17 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:18:17.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:17 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:18.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.221+0000 7fbcf9258700 1 -- 172.21.15.70:0/2449479300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf4108510 msgr2=0x7fbcf41088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:18.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.221+0000 7fbcf9258700 1 --2- 172.21.15.70:0/2449479300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf4108510 0x7fbcf41088f0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009990 tx=0x7fbce4009ca0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.221+0000 7fbcf9258700 1 -- 172.21.15.70:0/2449479300 shutdown_connections 2024-06-04T05:18:18.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.221+0000 7fbcf9258700 1 --2- 172.21.15.70:0/2449479300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcf4102df0 0x7fbcf410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.221+0000 7fbcf9258700 1 --2- 172.21.15.70:0/2449479300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf4108510 0x7fbcf41088f0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.221+0000 7fbcf9258700 1 --2- 172.21.15.70:0/2449479300 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbcf4102430 0x7fbcf41028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.221+0000 7fbcf9258700 1 -- 172.21.15.70:0/2449479300 >> 172.21.15.70:0/2449479300 conn(0x7fbcf40793b0 msgr2=0x7fbcf40797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:18.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.221+0000 7fbcf9258700 1 -- 172.21.15.70:0/2449479300 shutdown_connections 2024-06-04T05:18:18.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.221+0000 7fbcf9258700 1 -- 172.21.15.70:0/2449479300 wait complete. 2024-06-04T05:18:18.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.222+0000 7fbcf9258700 1 Processor -- start 2024-06-04T05:18:18.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.222+0000 7fbcf9258700 1 -- start start 2024-06-04T05:18:18.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf9258700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf4102430 0x7fbcf419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf9258700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcf4102df0 0x7fbcf419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf9258700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbcf4108510 0x7fbcf41998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf9258700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf4113590 con 0x7fbcf4102430 2024-06-04T05:18:18.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf9258700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf4113410 con 0x7fbcf4102df0 2024-06-04T05:18:18.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf4102430 0x7fbcf419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf9258700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf4113710 con 0x7fbcf4108510 2024-06-04T05:18:18.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf4102430 0x7fbcf419f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41814/0 (socket says 172.21.15.70:41814) 2024-06-04T05:18:18.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf2d9d700 1 -- 172.21.15.70:0/901403019 learned_addr learned my addr 172.21.15.70:0/901403019 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:18.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcf4102df0 0x7fbcf419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf359e700 1 --2- 172.21.15.70:0/901403019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbcf4108510 0x7fbcf41998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf259c700 1 -- 172.21.15.70:0/901403019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbcf4108510 msgr2=0x7fbcf41998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:18.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf259c700 1 --2- 172.21.15.70:0/901403019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbcf4108510 0x7fbcf41998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf259c700 1 -- 172.21.15.70:0/901403019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf4102430 msgr2=0x7fbcf419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:18.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf259c700 1 --2- 172.21.15.70:0/901403019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf4102430 0x7fbcf419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf259c700 1 -- 172.21.15.70:0/901403019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf419a050 con 0x7fbcf4102df0 2024-06-04T05:18:18.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf359e700 1 --2- 172.21.15.70:0/901403019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbcf4108510 0x7fbcf41998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:18.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.223+0000 7fbcf2d9d700 1 --2- 172.21.15.70:0/901403019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf4102430 0x7fbcf419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:18.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.224+0000 7fbcf259c700 1 --2- 172.21.15.70:0/901403019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcf4102df0 0x7fbcf419fc70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fbce800d950 tx=0x7fbce800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:18.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.224+0000 7fbcdbfff700 1 -- 172.21.15.70:0/901403019 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce8016070 con 0x7fbcf4102df0 2024-06-04T05:18:18.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.224+0000 7fbcdbfff700 1 -- 172.21.15.70:0/901403019 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbce800fd20 con 0x7fbcf4102df0 2024-06-04T05:18:18.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.224+0000 7fbcf9258700 1 -- 172.21.15.70:0/901403019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcf419a340 con 0x7fbcf4102df0 2024-06-04T05:18:18.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.224+0000 7fbcdbfff700 1 -- 172.21.15.70:0/901403019 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce8010d60 con 0x7fbcf4102df0 2024-06-04T05:18:18.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.224+0000 7fbcf9258700 1 -- 172.21.15.70:0/901403019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcf410fca0 con 0x7fbcf4102df0 2024-06-04T05:18:18.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.227+0000 7fbcdbfff700 1 -- 172.21.15.70:0/901403019 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fbce8019070 con 0x7fbcf4102df0 2024-06-04T05:18:18.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.228+0000 7fbcdbfff700 1 --2- 172.21.15.70:0/901403019 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbcdc0b2a70 0x7fbcdc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.228+0000 7fbcf2d9d700 1 --2- 172.21.15.70:0/901403019 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbcdc0b2a70 0x7fbcdc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.229+0000 7fbcdbfff700 1 -- 172.21.15.70:0/901403019 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7fbce80e1f50 con 0x7fbcf4102df0 2024-06-04T05:18:18.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.229+0000 7fbcf2d9d700 1 --2- 172.21.15.70:0/901403019 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbcdc0b2a70 0x7fbcdc0b4f30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009750 tx=0x7fbce4005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:18.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.229+0000 7fbd03aedb80 1 Processor -- start 2024-06-04T05:18:18.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.229+0000 7fbd03aedb80 1 -- start start 2024-06-04T05:18:18.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.229+0000 7fbd03aedb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 0x55b51de54780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.229+0000 7fbd03aedb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 0x55b51de4eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.229+0000 7fbd03aedb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4f2e0 0x55b51de5b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.229+0000 7fbd03aedb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b51ddacdf0 con 0x55b51de4e920 2024-06-04T05:18:18.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.229+0000 7fbd03aedb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b51dd9c990 con 0x55b51de543a0 2024-06-04T05:18:18.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.229+0000 7fbd03aedb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b51dda09d0 con 0x55b51de4f2e0 2024-06-04T05:18:18.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.230+0000 7fbcf2d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 0x55b51de54780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.230+0000 7fbcf259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 0x55b51de4eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.230+0000 7fbcf2d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 0x55b51de54780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54492/0 (socket says 172.21.15.70:54492) 2024-06-04T05:18:18.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.230+0000 7fbcf359e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4f2e0 0x55b51de5b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.230+0000 7fbcf2d9d700 1 -- 172.21.15.70:0/3526175733 learned_addr learned my addr 172.21.15.70:0/3526175733 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:18.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.230+0000 7fbcf2d9d700 1 -- 172.21.15.70:0/3526175733 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4f2e0 msgr2=0x55b51de5b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:18.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.230+0000 7fbcf2d9d700 1 --2- 172.21.15.70:0/3526175733 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4f2e0 0x55b51de5b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.230+0000 7fbcf2d9d700 1 -- 172.21.15.70:0/3526175733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 msgr2=0x55b51de4eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:18.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.230+0000 7fbcf2d9d700 1 --2- 172.21.15.70:0/3526175733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 0x55b51de4eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.230+0000 7fbcf2d9d700 1 -- 172.21.15.70:0/3526175733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b51dd46030 con 0x55b51de543a0 2024-06-04T05:18:18.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.230+0000 7fbcf259c700 1 --2- 172.21.15.70:0/3526175733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 0x55b51de4eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:18.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.230+0000 7fbcf2d9d700 1 --2- 172.21.15.70:0/3526175733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 0x55b51de54780 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbce4000c00 tx=0x7fbce401f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ee3c989ddfb3769 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:18.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.230+0000 7fbcd97fa700 1 -- 172.21.15.70:0/3526175733 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce4003a90 con 0x55b51de543a0 2024-06-04T05:18:18.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.230+0000 7fbcd97fa700 1 -- 172.21.15.70:0/3526175733 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbce4003c30 con 0x55b51de543a0 2024-06-04T05:18:18.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.231+0000 7fbcd97fa700 1 -- 172.21.15.70:0/3526175733 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce4022410 con 0x55b51de543a0 2024-06-04T05:18:18.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.231+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3526175733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 msgr2=0x55b51de54780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:18.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.231+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/3526175733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 0x55b51de54780 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbce4000c00 tx=0x7fbce401f000 comp rx=0 tx=0).stop 2024-06-04T05:18:18.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.231+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3526175733 shutdown_connections 2024-06-04T05:18:18.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.231+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/3526175733 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4f2e0 0x55b51de5b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.231+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/3526175733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 0x55b51de54780 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.231+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/3526175733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 0x55b51de4eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.231+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3526175733 >> 172.21.15.70:0/3526175733 conn(0x55b51dd7a2c0 msgr2=0x55b51de49460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:18.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.232+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3526175733 shutdown_connections 2024-06-04T05:18:18.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.232+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3526175733 wait complete. 2024-06-04T05:18:18.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.232+0000 7fbd03aedb80 1 -- start start 2024-06-04T05:18:18.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.232+0000 7fbd03aedb80 1 Processor -- start 2024-06-04T05:18:18.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.232+0000 7fbd03aedb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 0x55b51de5ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.233+0000 7fbcf2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 0x55b51de5ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.233+0000 7fbcf2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 0x55b51de5ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41832/0 (socket says 172.21.15.70:41832) 2024-06-04T05:18:18.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.233+0000 7fbcf2d9d700 1 -- 172.21.15.70:0/3779758708 learned_addr learned my addr 172.21.15.70:0/3779758708 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:18.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.233+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/3779758708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4f2e0 0x55b51de57f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.233+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/3779758708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 0x55b51de584d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.233+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3779758708 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b51dca3020 con 0x55b51de4e920 2024-06-04T05:18:18.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.233+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3779758708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b51dd04e20 con 0x55b51de543a0 2024-06-04T05:18:18.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.233+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3779758708 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b51de5b740 con 0x55b51de4f2e0 2024-06-04T05:18:18.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.233+0000 7fbcf359e700 1 --2- 172.21.15.70:0/3779758708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 0x55b51de584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.233+0000 7fbcf259c700 1 --2- 172.21.15.70:0/3779758708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4f2e0 0x55b51de57f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.233+0000 7fbcf2d9d700 1 -- 172.21.15.70:0/3779758708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4f2e0 msgr2=0x55b51de57f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:18.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.233+0000 7fbcf2d9d700 1 --2- 172.21.15.70:0/3779758708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4f2e0 0x55b51de57f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.233+0000 7fbcf2d9d700 1 -- 172.21.15.70:0/3779758708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 msgr2=0x55b51de584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:18.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.233+0000 7fbcf2d9d700 1 --2- 172.21.15.70:0/3779758708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 0x55b51de584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.233+0000 7fbcf2d9d700 1 -- 172.21.15.70:0/3779758708 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b51dcec950 con 0x55b51de4e920 2024-06-04T05:18:18.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.234+0000 7fbd03aedb80 10 client.0 _ll_register_callbacks cb 0x55b51dd379d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:18.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.234+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3779758708 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbce4003c30 con 0x55b51de4e920 2024-06-04T05:18:18.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.234+0000 7fbcf2d9d700 1 --2- 172.21.15.70:0/3779758708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 0x55b51de5ac90 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fbce402f040 tx=0x7fbce40057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7624baf7f24cb51e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:18.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.234+0000 7fbcd6ff5700 10 client.20856 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:18:18.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.234+0000 7fbcd6ff5700 1 -- 172.21.15.70:0/3779758708 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce4003a90 con 0x55b51de4e920 2024-06-04T05:18:18.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.234+0000 7fbcd6ff5700 1 -- 172.21.15.70:0/3779758708 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7fbce400f460 con 0x55b51de4e920 2024-06-04T05:18:18.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.234+0000 7fbcd6ff5700 1 -- 172.21.15.70:0/3779758708 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fbce400f660 con 0x55b51de4e920 2024-06-04T05:18:18.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.234+0000 7fbd03aedb80 10 client.20856 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:18.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.234+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3779758708 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({fsmap=59}) -- 0x55b51dd567e0 con 0x55b51de4e920 2024-06-04T05:18:18.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.235+0000 7fbcd6ff5700 1 -- 172.21.15.70:0/3779758708 <== mon.0 v2:172.21.15.70:3300/0 4 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7fbce400f800 con 0x55b51de4e920 2024-06-04T05:18:18.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.235+0000 7fbd03aedb80 10 client.20856 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:18.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.235+0000 7fbd03aedb80 10 client.20856 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:18:18.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.235+0000 7fbd03aedb80 20 client.20856 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:18.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.235+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/3779758708 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b51de707c0 0x55b51de72c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.235+0000 7fbd03aedb80 4 client.20856 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:18:18.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.235+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3779758708 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b51ddaed40 con 0x55b51de707c0 2024-06-04T05:18:18.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.235+0000 7fbcf259c700 1 --2- 172.21.15.70:0/3779758708 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b51de707c0 0x55b51de72c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.235+0000 7fbcf259c700 1 --2- 172.21.15.70:0/3779758708 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b51de707c0 0x55b51de72c80 crc :-1 s=READY pgs=445 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8240b715c4673ddd server_cookie=d2740b35b74b47f9 in_seq=0 out_seq=0 2024-06-04T05:18:18.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.235+0000 7fbcd6ff5700 10 client.20856 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:18.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.236+0000 7fbcd6ff5700 1 -- 172.21.15.70:0/3779758708 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b51ddaed40 con 0x55b51de707c0 2024-06-04T05:18:18.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.236+0000 7fbcd6ff5700 10 client.20856 handle_command_reply: tid=0 2024-06-04T05:18:18.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.236+0000 7fbd03aedb80 1 client.20856 shutdown 2024-06-04T05:18:18.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.236+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3779758708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 msgr2=0x55b51de5ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:18.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.236+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/3779758708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 0x55b51de5ac90 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fbce402f040 tx=0x7fbce40057c0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.237+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3779758708 shutdown_connections 2024-06-04T05:18:18.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.237+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/3779758708 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b51de707c0 0x55b51de72c80 crc :-1 s=READY pgs=445 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.237+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/3779758708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 0x55b51de584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.237+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/3779758708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 0x55b51de5ac90 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.237+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/3779758708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4f2e0 0x55b51de57f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.237+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3779758708 >> 172.21.15.70:0/3779758708 conn(0x55b51dd7a2c0 msgr2=0x55b51de48b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:18.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.237+0000 7fbcd6ff5700 0 client.20856 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:18.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.237+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3779758708 shutdown_connections 2024-06-04T05:18:18.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.237+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3779758708 wait complete. 2024-06-04T05:18:18.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.237+0000 7fbd03aedb80 20 client.20856 trim_cache size 0 max 16384 2024-06-04T05:18:18.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.273+0000 7fbd03aedb80 1 Processor -- start 2024-06-04T05:18:18.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.273+0000 7fbd03aedb80 1 -- start start 2024-06-04T05:18:18.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.273+0000 7fbd03aedb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 0x55b51de53980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.273+0000 7fbd03aedb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4f2e0 0x55b51de525f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.273+0000 7fbd03aedb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 0x55b51de492f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.273+0000 7fbd03aedb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf41abc00 con 0x55b51de4e920 2024-06-04T05:18:18.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.273+0000 7fbd03aedb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf41ad7a0 con 0x55b51de543a0 2024-06-04T05:18:18.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.273+0000 7fbd03aedb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf41ac330 con 0x55b51de4f2e0 2024-06-04T05:18:18.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.274+0000 7fbcf2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 0x55b51de53980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.274+0000 7fbcf359e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 0x55b51de492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.274+0000 7fbcf259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4f2e0 0x55b51de525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.274+0000 7fbcf2d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 0x55b51de53980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41840/0 (socket says 172.21.15.70:41840) 2024-06-04T05:18:18.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.274+0000 7fbcf2d9d700 1 -- 172.21.15.70:0/3960369122 learned_addr learned my addr 172.21.15.70:0/3960369122 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:18.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.274+0000 7fbcf359e700 1 --2- 172.21.15.70:0/3960369122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 0x55b51de492f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54512/0 (socket says 172.21.15.70:54512) 2024-06-04T05:18:18.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.274+0000 7fbcf259c700 1 --2- 172.21.15.70:0/3960369122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4f2e0 0x55b51de525f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39048/0 (socket says 172.21.15.70:39048) 2024-06-04T05:18:18.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.274+0000 7fbcf2d9d700 1 -- 172.21.15.70:0/3960369122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4f2e0 msgr2=0x55b51de525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:18.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.274+0000 7fbcf2d9d700 1 --2- 172.21.15.70:0/3960369122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4f2e0 0x55b51de525f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.274+0000 7fbcf2d9d700 1 -- 172.21.15.70:0/3960369122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 msgr2=0x55b51de492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:18.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.274+0000 7fbcf2d9d700 1 --2- 172.21.15.70:0/3960369122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 0x55b51de492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.274+0000 7fbcf2d9d700 1 -- 172.21.15.70:0/3960369122 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b51de5a8d0 con 0x55b51de4e920 2024-06-04T05:18:18.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.274+0000 7fbcf359e700 1 --2- 172.21.15.70:0/3960369122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 0x55b51de492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:18.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.274+0000 7fbcf259c700 1 --2- 172.21.15.70:0/3960369122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4f2e0 0x55b51de525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:18.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.274+0000 7fbcf2d9d700 1 --2- 172.21.15.70:0/3960369122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 0x55b51de53980 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fbce4003960 tx=0x7fbce4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a8c13f5dad1bca server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:18.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.274+0000 7fbcd8ff9700 1 -- 172.21.15.70:0/3960369122 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce400f460 con 0x55b51de4e920 2024-06-04T05:18:18.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.275+0000 7fbcd8ff9700 1 -- 172.21.15.70:0/3960369122 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbce4003c30 con 0x55b51de4e920 2024-06-04T05:18:18.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.275+0000 7fbcd8ff9700 1 -- 172.21.15.70:0/3960369122 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce403f690 con 0x55b51de4e920 2024-06-04T05:18:18.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.275+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3960369122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 msgr2=0x55b51de53980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:18.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.275+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/3960369122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 0x55b51de53980 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fbce4003960 tx=0x7fbce4032000 comp rx=0 tx=0).stop 2024-06-04T05:18:18.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.275+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3960369122 shutdown_connections 2024-06-04T05:18:18.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.276+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/3960369122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de543a0 0x55b51de492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.276+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/3960369122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de4e920 0x55b51de53980 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.028 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:17 smithi070 ceph-mon[32631]: pgmap v664: 97 pgs: 97 active+clean; 850 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.3 MiB/s wr, 299 op/s 2024-06-04T05:18:18.028 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:17 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:18:18.028 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:17 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:18.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.276+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/3960369122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4f2e0 0x55b51de525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.276+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3960369122 >> 172.21.15.70:0/3960369122 conn(0x55b51dd7a2c0 msgr2=0x55b51de55090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:18.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.276+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3960369122 shutdown_connections 2024-06-04T05:18:18.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.276+0000 7fbd03aedb80 1 -- 172.21.15.70:0/3960369122 wait complete. 2024-06-04T05:18:18.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.276+0000 7fbd03aedb80 1 -- start start 2024-06-04T05:18:18.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.276+0000 7fbd03aedb80 1 Processor -- start 2024-06-04T05:18:18.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbd03aedb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4e920 0x55b51de69270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbd03aedb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de4f2e0 0x55b51de697b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbd03aedb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de543a0 0x55b51de66240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbd03aedb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf41ad0d0 con 0x55b51de543a0 2024-06-04T05:18:18.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbd03aedb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf41aca00 con 0x55b51de4f2e0 2024-06-04T05:18:18.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbd03aedb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf41a9c10 con 0x55b51de4e920 2024-06-04T05:18:18.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbd03aedb80 10 client.0 _ll_register_callbacks cb 0x55b51dd379d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:18.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbcf2d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4e920 0x55b51de69270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbcf359e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de543a0 0x55b51de66240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbcf2d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4e920 0x55b51de69270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39054/0 (socket says 172.21.15.70:39054) 2024-06-04T05:18:18.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbcf2d9d700 1 -- 172.21.15.70:0/1946562767 learned_addr learned my addr 172.21.15.70:0/1946562767 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:18.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbcf259c700 1 --2- 172.21.15.70:0/1946562767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de4f2e0 0x55b51de697b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbcf2d9d700 1 -- 172.21.15.70:0/1946562767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de4f2e0 msgr2=0x55b51de697b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:18.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbcf2d9d700 1 --2- 172.21.15.70:0/1946562767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de4f2e0 0x55b51de697b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbcf2d9d700 1 -- 172.21.15.70:0/1946562767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de543a0 msgr2=0x55b51de66240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:18.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbcf2d9d700 1 --2- 172.21.15.70:0/1946562767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de543a0 0x55b51de66240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbcf2d9d700 1 -- 172.21.15.70:0/1946562767 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b51de79050 con 0x55b51de4e920 2024-06-04T05:18:18.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbcf359e700 1 --2- 172.21.15.70:0/1946562767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de543a0 0x55b51de66240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:18.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbcf2d9d700 1 --2- 172.21.15.70:0/1946562767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4e920 0x55b51de69270 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009720 tx=0x7fbce400f460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=903768cb0aa68b27 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:18.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.277+0000 7fbcd6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:18:18.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.278+0000 7fbcd6ff5700 1 -- 172.21.15.70:0/1946562767 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce4022ca0 con 0x55b51de4e920 2024-06-04T05:18:18.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.278+0000 7fbcd6ff5700 1 -- 172.21.15.70:0/1946562767 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7fbce403f450 con 0x55b51de4e920 2024-06-04T05:18:18.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.278+0000 7fbd03aedb80 1 -- 172.21.15.70:0/1946562767 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbce4003c30 con 0x55b51de4e920 2024-06-04T05:18:18.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.278+0000 7fbcd6ff5700 1 -- 172.21.15.70:0/1946562767 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fbce4022e40 con 0x55b51de4e920 2024-06-04T05:18:18.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.278+0000 7fbd03aedb80 10 client.29722 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:18.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.278+0000 7fbd03aedb80 1 -- 172.21.15.70:0/1946562767 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55b51dedb940 con 0x55b51de4e920 2024-06-04T05:18:18.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.279+0000 7fbcd6ff5700 1 -- 172.21.15.70:0/1946562767 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbce403a070 con 0x55b51de4e920 2024-06-04T05:18:18.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.279+0000 7fbcd6ff5700 1 -- 172.21.15.70:0/1946562767 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7fbce4050680 con 0x55b51de4e920 2024-06-04T05:18:18.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.279+0000 7fbd03aedb80 10 client.29722 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:18.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.279+0000 7fbd03aedb80 10 client.29722 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:18:18.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.279+0000 7fbd03aedb80 20 client.29722 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:18.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.279+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/1946562767 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b51dedbee0 0x55b51dede300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:18.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.279+0000 7fbd03aedb80 4 client.29722 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:18:18.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.279+0000 7fbd03aedb80 1 -- 172.21.15.70:0/1946562767 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fbce4003c30 con 0x55b51dedbee0 2024-06-04T05:18:18.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.280+0000 7fbcf259c700 1 --2- 172.21.15.70:0/1946562767 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b51dedbee0 0x55b51dede300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:18.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.280+0000 7fbcd6ff5700 10 client.29722 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:18.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.280+0000 7fbcf259c700 1 --2- 172.21.15.70:0/1946562767 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b51dedbee0 0x55b51dede300 crc :-1 s=READY pgs=446 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5295312f374ce18a server_cookie=621b2f86452efb9c in_seq=0 out_seq=0 2024-06-04T05:18:18.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.282+0000 7fbcd6ff5700 1 -- 172.21.15.70:0/1946562767 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fbce4003c30 con 0x55b51dedbee0 2024-06-04T05:18:18.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.282+0000 7fbcd6ff5700 10 client.29722 handle_command_reply: tid=0 2024-06-04T05:18:18.032 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:18:18.032 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:18:18.032 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "0e077767-01b8-486a-bba7-ea8a77044aff", 2024-06-04T05:18:18.032 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:18:18.032 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:18:18.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.282+0000 7fbd03aedb80 1 client.29722 shutdown 2024-06-04T05:18:18.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.282+0000 7fbd03aedb80 1 -- 172.21.15.70:0/1946562767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4e920 msgr2=0x55b51de69270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:18.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.282+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/1946562767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4e920 0x55b51de69270 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009720 tx=0x7fbce400f460 comp rx=0 tx=0).stop 2024-06-04T05:18:18.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.282+0000 7fbd03aedb80 1 -- 172.21.15.70:0/1946562767 shutdown_connections 2024-06-04T05:18:18.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.282+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/1946562767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b51de543a0 0x55b51de66240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.282+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/1946562767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b51de4e920 0x55b51de69270 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.282+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/1946562767 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b51dedbee0 0x55b51dede300 crc :-1 s=READY pgs=446 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.282+0000 7fbd03aedb80 1 --2- 172.21.15.70:0/1946562767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b51de4f2e0 0x55b51de697b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.282+0000 7fbcd6ff5700 0 client.29722 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:18.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.282+0000 7fbd03aedb80 1 -- 172.21.15.70:0/1946562767 >> 172.21.15.70:0/1946562767 conn(0x55b51dd7a2c0 msgr2=0x55b51de4e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:18.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.282+0000 7fbd03aedb80 1 -- 172.21.15.70:0/1946562767 shutdown_connections 2024-06-04T05:18:18.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.283+0000 7fbd03aedb80 1 -- 172.21.15.70:0/1946562767 wait complete. 2024-06-04T05:18:18.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.283+0000 7fbd03aedb80 20 client.29722 trim_cache size 0 max 16384 2024-06-04T05:18:18.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.283+0000 7fbcf9258700 1 -- 172.21.15.70:0/901403019 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbcdc0b2a70 msgr2=0x7fbcdc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:18.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.283+0000 7fbcf9258700 1 --2- 172.21.15.70:0/901403019 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbcdc0b2a70 0x7fbcdc0b4f30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009750 tx=0x7fbce4005960 comp rx=0 tx=0).stop 2024-06-04T05:18:18.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.283+0000 7fbcf9258700 1 -- 172.21.15.70:0/901403019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcf4102df0 msgr2=0x7fbcf419fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:18.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.283+0000 7fbcf9258700 1 --2- 172.21.15.70:0/901403019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcf4102df0 0x7fbcf419fc70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fbce800d950 tx=0x7fbce800dc60 comp rx=0 tx=0).stop 2024-06-04T05:18:18.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.283+0000 7fbcf9258700 1 -- 172.21.15.70:0/901403019 shutdown_connections 2024-06-04T05:18:18.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.283+0000 7fbcf9258700 1 --2- 172.21.15.70:0/901403019 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbcdc0b2a70 0x7fbcdc0b4f30 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.283+0000 7fbcf9258700 1 --2- 172.21.15.70:0/901403019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbcf4108510 0x7fbcf41998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.283+0000 7fbcf9258700 1 --2- 172.21.15.70:0/901403019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf4102430 0x7fbcf419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.283+0000 7fbcf9258700 1 --2- 172.21.15.70:0/901403019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcf4102df0 0x7fbcf419fc70 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:18.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.283+0000 7fbcf9258700 1 -- 172.21.15.70:0/901403019 >> 172.21.15.70:0/901403019 conn(0x7fbcf40793b0 msgr2=0x7fbcf41012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:18.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.284+0000 7fbcf9258700 1 -- 172.21.15.70:0/901403019 shutdown_connections 2024-06-04T05:18:18.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:17.284+0000 7fbcf9258700 1 -- 172.21.15.70:0/901403019 wait complete. 2024-06-04T05:18:18.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:17 smithi112 ceph-mon[36453]: pgmap v664: 97 pgs: 97 active+clean; 850 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.3 MiB/s wr, 299 op/s 2024-06-04T05:18:18.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:17 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:18:18.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:17 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:18.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 57726 7.18 MB/sec execute 310 sec latency 123.173 ms 2024-06-04T05:18:18.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:18 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:18:18.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:18 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:18.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:18 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:18:18.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:18 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:18:18.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:18 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:18:18.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:18 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:18:18.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:18 smithi098 ceph-mon[35569]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT) see mds.i log and `damage ls` output for details 2024-06-04T05:18:18.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:18 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:18.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:18 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:18.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:18 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:18.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:18 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:19.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:18 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:18:19.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:18 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:19.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:18 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:18:19.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:18 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:18:19.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:18 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:18:19.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:18 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:18:19.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:18 smithi070 ceph-mon[32631]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT) see mds.i log and `damage ls` output for details 2024-06-04T05:18:19.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:18 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:19.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:18 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:19.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:18 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:19.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:18 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:19.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:18 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:18:19.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:18 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:19.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:18 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:18:19.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:18 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:18:19.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:18 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:18:19.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:18 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:18:19.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:18 smithi112 ceph-mon[36453]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT) see mds.i log and `damage ls` output for details 2024-06-04T05:18:19.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:18 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:19.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:18 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:19.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:18 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:19.187 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:18 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:18:19.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 57872 7.17 MB/sec execute 311 sec latency 68.024 ms 2024-06-04T05:18:19.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:19 smithi098 ceph-mon[35569]: pgmap v665: 97 pgs: 97 active+clean; 850 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.2 MiB/s wr, 292 op/s 2024-06-04T05:18:20.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:19 smithi070 ceph-mon[32631]: pgmap v665: 97 pgs: 97 active+clean; 850 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.2 MiB/s wr, 292 op/s 2024-06-04T05:18:20.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:19 smithi112 ceph-mon[36453]: pgmap v665: 97 pgs: 97 active+clean; 850 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.2 MiB/s wr, 292 op/s 2024-06-04T05:18:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.694+0000 7f38401a4640 1 -- 172.21.15.70:0/3224843147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3838101980 msgr2=0x7f3838101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.694+0000 7f38401a4640 1 --2- 172.21.15.70:0/3224843147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3838101980 0x7f3838101e00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3828009860 tx=0x7f382802a340 comp rx=0 tx=0).stop 2024-06-04T05:18:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.694+0000 7f38401a4640 1 -- 172.21.15.70:0/3224843147 shutdown_connections 2024-06-04T05:18:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.694+0000 7f38401a4640 1 --2- 172.21.15.70:0/3224843147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3838101980 0x7f3838101e00 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.694+0000 7f38401a4640 1 --2- 172.21.15.70:0/3224843147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3838102340 0x7f383810e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.694+0000 7f38401a4640 1 --2- 172.21.15.70:0/3224843147 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3838107940 0x7f3838107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.694+0000 7f38401a4640 1 -- 172.21.15.70:0/3224843147 >> 172.21.15.70:0/3224843147 conn(0x7f3838078ad0 msgr2=0x7f3838078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.694+0000 7f38401a4640 1 -- 172.21.15.70:0/3224843147 shutdown_connections 2024-06-04T05:18:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.694+0000 7f38401a4640 1 -- 172.21.15.70:0/3224843147 wait complete. 2024-06-04T05:18:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.695+0000 7f38401a4640 1 Processor -- start 2024-06-04T05:18:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.695+0000 7f38401a4640 1 -- start start 2024-06-04T05:18:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.695+0000 7f38401a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3838101980 0x7f3838199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.695+0000 7f38401a4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3838102340 0x7f38381996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.695+0000 7f38401a4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3838107940 0x7f383819da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.695+0000 7f38401a4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3838111ab0 con 0x7f3838102340 2024-06-04T05:18:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.695+0000 7f38401a4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3838111930 con 0x7f3838101980 2024-06-04T05:18:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.695+0000 7f38401a4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3838111c30 con 0x7f3838107940 2024-06-04T05:18:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.696+0000 7f383df40640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3838101980 0x7f3838199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.696+0000 7f383df40640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3838101980 0x7f3838199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54542/0 (socket says 172.21.15.70:54542) 2024-06-04T05:18:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.696+0000 7f383e741640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3838107940 0x7f383819da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.696+0000 7f383d73f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3838102340 0x7f38381996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.696+0000 7f383df40640 1 -- 172.21.15.70:0/1778422793 learned_addr learned my addr 172.21.15.70:0/1778422793 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.696+0000 7f383df40640 1 -- 172.21.15.70:0/1778422793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3838107940 msgr2=0x7f383819da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.696+0000 7f383df40640 1 --2- 172.21.15.70:0/1778422793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3838107940 0x7f383819da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.696+0000 7f383df40640 1 -- 172.21.15.70:0/1778422793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3838102340 msgr2=0x7f38381996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.696+0000 7f383df40640 1 --2- 172.21.15.70:0/1778422793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3838102340 0x7f38381996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.696+0000 7f383df40640 1 -- 172.21.15.70:0/1778422793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383819e170 con 0x7f3838101980 2024-06-04T05:18:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.696+0000 7f383d73f640 1 --2- 172.21.15.70:0/1778422793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3838102340 0x7f38381996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.696+0000 7f383df40640 1 --2- 172.21.15.70:0/1778422793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3838101980 0x7f3838199180 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f383400e9f0 tx=0x7f383400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.696+0000 7f382effd640 1 -- 172.21.15.70:0/1778422793 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383400cd80 con 0x7f3838101980 2024-06-04T05:18:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.696+0000 7f382effd640 1 -- 172.21.15.70:0/1778422793 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f38340054b0 con 0x7f3838101980 2024-06-04T05:18:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.696+0000 7f382effd640 1 -- 172.21.15.70:0/1778422793 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3834010430 con 0x7f3838101980 2024-06-04T05:18:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.696+0000 7f38401a4640 1 -- 172.21.15.70:0/1778422793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f383819e460 con 0x7f3838101980 2024-06-04T05:18:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.697+0000 7f38401a4640 1 -- 172.21.15.70:0/1778422793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38381a5cf0 con 0x7f3838101980 2024-06-04T05:18:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.700+0000 7f3842f97740 1 Processor -- start 2024-06-04T05:18:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.700+0000 7f382effd640 1 -- 172.21.15.70:0/1778422793 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f38340046c0 con 0x7f3838101980 2024-06-04T05:18:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.700+0000 7f3842f97740 1 -- start start 2024-06-04T05:18:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.700+0000 7f3842f97740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092cac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.700+0000 7f3842f97740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 0x55e2092d54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.700+0000 7f3842f97740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092d5a00 0x55e2092d7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.700+0000 7f3842f97740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e20923cc30 con 0x55e2092cb140 2024-06-04T05:18:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.700+0000 7f3842f97740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e20922c7d0 con 0x55e2092ca820 2024-06-04T05:18:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.700+0000 7f3842f97740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e209230810 con 0x55e2092d5a00 2024-06-04T05:18:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.701+0000 7f383e741640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092cac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.701+0000 7f383e741640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092cac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54556/0 (socket says 172.21.15.70:54556) 2024-06-04T05:18:20.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.701+0000 7f383e741640 1 -- 172.21.15.70:0/4172893096 learned_addr learned my addr 172.21.15.70:0/4172893096 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:20.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.701+0000 7f383d73f640 1 --2- 172.21.15.70:0/4172893096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092d5a00 0x55e2092d7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.701+0000 7f383df40640 1 --2- 172.21.15.70:0/4172893096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 0x55e2092d54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.701+0000 7f383e741640 1 -- 172.21.15.70:0/4172893096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092d5a00 msgr2=0x55e2092d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:20.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.701+0000 7f383e741640 1 --2- 172.21.15.70:0/4172893096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092d5a00 0x55e2092d7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.701+0000 7f383e741640 1 -- 172.21.15.70:0/4172893096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 msgr2=0x55e2092d54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:20.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.701+0000 7f383e741640 1 --2- 172.21.15.70:0/4172893096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 0x55e2092d54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.701+0000 7f383e741640 1 -- 172.21.15.70:0/4172893096 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e208f41210 con 0x55e2092ca820 2024-06-04T05:18:20.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.701+0000 7f383df40640 1 --2- 172.21.15.70:0/4172893096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 0x55e2092d54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:20.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.702+0000 7f383e741640 1 --2- 172.21.15.70:0/4172893096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092cac00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f383000e450 tx=0x7f38300095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=973146e86b89df56 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:20.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.702+0000 7f3823fff640 1 -- 172.21.15.70:0/4172893096 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383000d910 con 0x55e2092ca820 2024-06-04T05:18:20.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.702+0000 7f382effd640 1 --2- 172.21.15.70:0/1778422793 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38240b2a10 0x7f38240b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.702+0000 7f3823fff640 1 -- 172.21.15.70:0/4172893096 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f383001c070 con 0x55e2092ca820 2024-06-04T05:18:20.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.702+0000 7f3823fff640 1 -- 172.21.15.70:0/4172893096 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3830017da0 con 0x55e2092ca820 2024-06-04T05:18:20.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.702+0000 7f382effd640 1 -- 172.21.15.70:0/1778422793 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f38340e2900 con 0x7f3838101980 2024-06-04T05:18:20.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.703+0000 7f383df40640 1 --2- 172.21.15.70:0/1778422793 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38240b2a10 0x7f38240b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.703+0000 7f3842f97740 1 -- 172.21.15.70:0/4172893096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 msgr2=0x55e2092cac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:20.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.703+0000 7f3842f97740 1 --2- 172.21.15.70:0/4172893096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092cac00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f383000e450 tx=0x7f38300095c0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.703+0000 7f3842f97740 1 -- 172.21.15.70:0/4172893096 shutdown_connections 2024-06-04T05:18:20.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.703+0000 7f3842f97740 1 --2- 172.21.15.70:0/4172893096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092d5a00 0x55e2092d7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.703+0000 7f3842f97740 1 --2- 172.21.15.70:0/4172893096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 0x55e2092d54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.703+0000 7f3842f97740 1 --2- 172.21.15.70:0/4172893096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092cac00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.703+0000 7f3842f97740 1 -- 172.21.15.70:0/4172893096 >> 172.21.15.70:0/4172893096 conn(0x55e2092c6d30 msgr2=0x55e2092c7110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.703+0000 7f3842f97740 1 -- 172.21.15.70:0/4172893096 shutdown_connections 2024-06-04T05:18:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.703+0000 7f383df40640 1 --2- 172.21.15.70:0/1778422793 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38240b2a10 0x7f38240b4ed0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f38340e00d0 tx=0x7f38340e8550 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.704+0000 7f3842f97740 1 -- 172.21.15.70:0/4172893096 wait complete. 2024-06-04T05:18:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.704+0000 7f3842f97740 1 -- start start 2024-06-04T05:18:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.704+0000 7f3842f97740 1 Processor -- start 2024-06-04T05:18:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.704+0000 7f3842f97740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092d2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.704+0000 7f3842f97740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092cb140 0x55e2092d2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.704+0000 7f3842f97740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092d5a00 0x55e2092d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.704+0000 7f3842f97740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e208f12790 con 0x55e2092d5a00 2024-06-04T05:18:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.704+0000 7f3842f97740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e2091e8890 con 0x55e2092ca820 2024-06-04T05:18:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.704+0000 7f3842f97740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e2092dca40 con 0x55e2092cb140 2024-06-04T05:18:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.705+0000 7f3842f97740 10 client.0 _ll_register_callbacks cb 0x55e208f40a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.705+0000 7f383d73f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092cb140 0x55e2092d2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.705+0000 7f383e741640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092d5a00 0x55e2092d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.705+0000 7f383e741640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092d5a00 0x55e2092d34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41868/0 (socket says 172.21.15.70:41868) 2024-06-04T05:18:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.705+0000 7f383e741640 1 -- 172.21.15.70:0/4197106236 learned_addr learned my addr 172.21.15.70:0/4197106236 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.705+0000 7f383e741640 1 --2- 172.21.15.70:0/4197106236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092d2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.705+0000 7f383e741640 1 -- 172.21.15.70:0/4197106236 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092cb140 msgr2=0x55e2092d2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.705+0000 7f383e741640 1 --2- 172.21.15.70:0/4197106236 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092cb140 0x55e2092d2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.705+0000 7f383e741640 1 -- 172.21.15.70:0/4197106236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 msgr2=0x55e2092d2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.705+0000 7f383e741640 1 --2- 172.21.15.70:0/4197106236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092d2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.705+0000 7f383e741640 1 -- 172.21.15.70:0/4197106236 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e20900d620 con 0x55e2092d5a00 2024-06-04T05:18:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.705+0000 7f383d73f640 1 --2- 172.21.15.70:0/4197106236 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092cb140 0x55e2092d2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.705+0000 7f383e741640 1 --2- 172.21.15.70:0/4197106236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092d5a00 0x55e2092d34c0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f3830005c40 tx=0x7f3830017980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea33ec664522725 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.705+0000 7f38217fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:18:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.705+0000 7f38217fa640 1 -- 172.21.15.70:0/4197106236 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3830018d90 con 0x55e2092d5a00 2024-06-04T05:18:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.705+0000 7f38217fa640 1 -- 172.21.15.70:0/4197106236 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f3830004a40 con 0x55e2092d5a00 2024-06-04T05:18:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.705+0000 7f3842f97740 1 -- 172.21.15.70:0/4197106236 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f383001c070 con 0x55e2092d5a00 2024-06-04T05:18:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.706+0000 7f38217fa640 1 -- 172.21.15.70:0/4197106236 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f3830003970 con 0x55e2092d5a00 2024-06-04T05:18:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.706+0000 7f3842f97740 10 client.20874 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.706+0000 7f3842f97740 1 -- 172.21.15.70:0/4197106236 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55e2092e8850 con 0x55e2092d5a00 2024-06-04T05:18:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.706+0000 7f38217fa640 1 -- 172.21.15.70:0/4197106236 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f383001c070 con 0x55e2092d5a00 2024-06-04T05:18:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.706+0000 7f38217fa640 1 -- 172.21.15.70:0/4197106236 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f383000b040 con 0x55e2092d5a00 2024-06-04T05:18:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.706+0000 7f3842f97740 10 client.20874 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.706+0000 7f3842f97740 10 client.20874 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:18:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.706+0000 7f3842f97740 20 client.20874 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.706+0000 7f3842f97740 1 --2- 172.21.15.70:0/4197106236 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e2092e9d10 0x55e2092ec100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.706+0000 7f3842f97740 4 client.20874 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:18:20.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.706+0000 7f3842f97740 1 -- 172.21.15.70:0/4197106236 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e20923eb80 con 0x55e2092e9d10 2024-06-04T05:18:20.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.707+0000 7f383d73f640 1 --2- 172.21.15.70:0/4197106236 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e2092e9d10 0x55e2092ec100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.707+0000 7f383d73f640 1 --2- 172.21.15.70:0/4197106236 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e2092e9d10 0x55e2092ec100 crc :-1 s=READY pgs=447 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d6c52aaef9dc643 server_cookie=61afabb121a4e09c in_seq=0 out_seq=0 2024-06-04T05:18:20.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.707+0000 7f38217fa640 10 client.20874 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.708+0000 7f38217fa640 1 -- 172.21.15.70:0/4197106236 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e20923eb80 con 0x55e2092e9d10 2024-06-04T05:18:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.708+0000 7f38217fa640 10 client.20874 handle_command_reply: tid=0 2024-06-04T05:18:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.708+0000 7f3842f97740 1 client.20874 shutdown 2024-06-04T05:18:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.709+0000 7f3842f97740 1 -- 172.21.15.70:0/4197106236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092d5a00 msgr2=0x55e2092d34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.709+0000 7f3842f97740 1 --2- 172.21.15.70:0/4197106236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092d5a00 0x55e2092d34c0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f3830005c40 tx=0x7f3830017980 comp rx=0 tx=0).stop 2024-06-04T05:18:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.709+0000 7f3842f97740 1 -- 172.21.15.70:0/4197106236 shutdown_connections 2024-06-04T05:18:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.709+0000 7f3842f97740 1 --2- 172.21.15.70:0/4197106236 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e2092e9d10 0x55e2092ec100 crc :-1 s=READY pgs=447 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.709+0000 7f3842f97740 1 --2- 172.21.15.70:0/4197106236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092d5a00 0x55e2092d34c0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.709+0000 7f3842f97740 1 --2- 172.21.15.70:0/4197106236 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092cb140 0x55e2092d2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.709+0000 7f3842f97740 1 --2- 172.21.15.70:0/4197106236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092d2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.709+0000 7f3842f97740 1 -- 172.21.15.70:0/4197106236 >> 172.21.15.70:0/4197106236 conn(0x55e2092c6d30 msgr2=0x55e2092c5fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:20.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.709+0000 7f38217fa640 0 client.20874 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:20.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.709+0000 7f3842f97740 1 -- 172.21.15.70:0/4197106236 shutdown_connections 2024-06-04T05:18:20.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.709+0000 7f3842f97740 1 -- 172.21.15.70:0/4197106236 wait complete. 2024-06-04T05:18:20.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.709+0000 7f3842f97740 20 client.20874 trim_cache size 0 max 16384 2024-06-04T05:18:20.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.733+0000 7f3842f97740 1 Processor -- start 2024-06-04T05:18:20.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.733+0000 7f3842f97740 1 -- start start 2024-06-04T05:18:20.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f3842f97740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f3842f97740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 0x55e2092d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f3842f97740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092d5a00 0x55e2092f1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f3842f97740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f38381ac4d0 con 0x55e2092cb140 2024-06-04T05:18:20.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f3842f97740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f38381ae070 con 0x55e2092ca820 2024-06-04T05:18:20.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f3842f97740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f38381acc00 con 0x55e2092d5a00 2024-06-04T05:18:20.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f383d73f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 0x55e2092d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f383e741640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f383d73f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 0x55e2092d18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41872/0 (socket says 172.21.15.70:41872) 2024-06-04T05:18:20.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f383d73f640 1 -- 172.21.15.70:0/3543086059 learned_addr learned my addr 172.21.15.70:0/3543086059 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:20.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f383e741640 1 --2- 172.21.15.70:0/3543086059 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092d5a00 0x55e2092f1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f383d73f640 1 -- 172.21.15.70:0/3543086059 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092d5a00 msgr2=0x55e2092f1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:20.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f383d73f640 1 --2- 172.21.15.70:0/3543086059 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092d5a00 0x55e2092f1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f383d73f640 1 -- 172.21.15.70:0/3543086059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 msgr2=0x55e2092d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:20.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f383d73f640 1 --2- 172.21.15.70:0/3543086059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092d32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f383d73f640 1 -- 172.21.15.70:0/3543086059 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2092d0f90 con 0x55e2092cb140 2024-06-04T05:18:20.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f383e741640 1 --2- 172.21.15.70:0/3543086059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092d32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:20.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.734+0000 7f383d73f640 1 --2- 172.21.15.70:0/3543086059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 0x55e2092d18f0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f3828000c00 tx=0x7f38280058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=737ce66c4c05df92 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:20.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.735+0000 7f38237fe640 1 -- 172.21.15.70:0/3543086059 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3828037040 con 0x55e2092cb140 2024-06-04T05:18:20.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.735+0000 7f38237fe640 1 -- 172.21.15.70:0/3543086059 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3828048030 con 0x55e2092cb140 2024-06-04T05:18:20.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.735+0000 7f38237fe640 1 -- 172.21.15.70:0/3543086059 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3828032a50 con 0x55e2092cb140 2024-06-04T05:18:20.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.736+0000 7f3842f97740 1 -- 172.21.15.70:0/3543086059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 msgr2=0x55e2092d18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:20.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.736+0000 7f3842f97740 1 --2- 172.21.15.70:0/3543086059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 0x55e2092d18f0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f3828000c00 tx=0x7f38280058b0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.736+0000 7f3842f97740 1 -- 172.21.15.70:0/3543086059 shutdown_connections 2024-06-04T05:18:20.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.736+0000 7f3842f97740 1 --2- 172.21.15.70:0/3543086059 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092d5a00 0x55e2092f1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.736+0000 7f3842f97740 1 --2- 172.21.15.70:0/3543086059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 0x55e2092d18f0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.736+0000 7f3842f97740 1 --2- 172.21.15.70:0/3543086059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092d32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.736+0000 7f3842f97740 1 -- 172.21.15.70:0/3543086059 >> 172.21.15.70:0/3543086059 conn(0x55e2092c6d30 msgr2=0x55e2092d11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.736+0000 7f3842f97740 1 -- 172.21.15.70:0/3543086059 shutdown_connections 2024-06-04T05:18:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.736+0000 7f3842f97740 1 -- 172.21.15.70:0/3543086059 wait complete. 2024-06-04T05:18:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.736+0000 7f3842f97740 1 -- start start 2024-06-04T05:18:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.736+0000 7f3842f97740 1 Processor -- start 2024-06-04T05:18:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.736+0000 7f3842f97740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092e2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.736+0000 7f3842f97740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 0x55e2092e2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.736+0000 7f3842f97740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092d5a00 0x55e2092dce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.736+0000 7f3842f97740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f38381ad9a0 con 0x55e2092cb140 2024-06-04T05:18:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.736+0000 7f3842f97740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f38381ad2d0 con 0x55e2092ca820 2024-06-04T05:18:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.736+0000 7f3842f97740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f38381aa4d0 con 0x55e2092d5a00 2024-06-04T05:18:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.737+0000 7f3842f97740 10 client.0 _ll_register_callbacks cb 0x7f38340e8420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.737+0000 7f383d73f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 0x55e2092e2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.737+0000 7f383e741640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092e2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.737+0000 7f383e741640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092e2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54588/0 (socket says 172.21.15.70:54588) 2024-06-04T05:18:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.737+0000 7f383e741640 1 -- 172.21.15.70:0/1854925432 learned_addr learned my addr 172.21.15.70:0/1854925432 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.737+0000 7f383e741640 1 --2- 172.21.15.70:0/1854925432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092d5a00 0x55e2092dce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.737+0000 7f383e741640 1 -- 172.21.15.70:0/1854925432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092d5a00 msgr2=0x55e2092dce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.737+0000 7f383e741640 1 --2- 172.21.15.70:0/1854925432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092d5a00 0x55e2092dce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.737+0000 7f383e741640 1 -- 172.21.15.70:0/1854925432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 msgr2=0x55e2092e2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.737+0000 7f383e741640 1 --2- 172.21.15.70:0/1854925432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 0x55e2092e2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.737+0000 7f383e741640 1 -- 172.21.15.70:0/1854925432 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2092dd650 con 0x55e2092ca820 2024-06-04T05:18:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.737+0000 7f383e741640 1 --2- 172.21.15.70:0/1854925432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092e2000 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3830017700 tx=0x7f38300179f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c4553fa2605e1c0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.737+0000 7f38217fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:18:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.737+0000 7f38217fa640 1 -- 172.21.15.70:0/1854925432 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3830029020 con 0x55e2092ca820 2024-06-04T05:18:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.737+0000 7f3842f97740 1 -- 172.21.15.70:0/1854925432 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3828048030 con 0x55e2092ca820 2024-06-04T05:18:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.737+0000 7f38217fa640 1 -- 172.21.15.70:0/1854925432 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f38300051e0 con 0x55e2092ca820 2024-06-04T05:18:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.738+0000 7f38217fa640 1 -- 172.21.15.70:0/1854925432 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f3830005040 con 0x55e2092ca820 2024-06-04T05:18:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.738+0000 7f3842f97740 10 client.29594 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.738+0000 7f3842f97740 1 -- 172.21.15.70:0/1854925432 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55e209353450 con 0x55e2092ca820 2024-06-04T05:18:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.738+0000 7f38217fa640 1 -- 172.21.15.70:0/1854925432 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3828048030 con 0x55e2092ca820 2024-06-04T05:18:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.738+0000 7f38217fa640 1 -- 172.21.15.70:0/1854925432 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f383000b040 con 0x55e2092ca820 2024-06-04T05:18:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.738+0000 7f3842f97740 10 client.29594 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.738+0000 7f3842f97740 10 client.29594 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:18:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.738+0000 7f3842f97740 20 client.29594 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.738+0000 7f3842f97740 1 --2- 172.21.15.70:0/1854925432 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e209353a50 0x55e209355e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.738+0000 7f3842f97740 4 client.29594 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:18:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.738+0000 7f3842f97740 1 -- 172.21.15.70:0/1854925432 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e209238360 con 0x55e209353a50 2024-06-04T05:18:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.738+0000 7f383d73f640 1 --2- 172.21.15.70:0/1854925432 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e209353a50 0x55e209355e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.739+0000 7f38217fa640 10 client.29594 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.739+0000 7f383d73f640 1 --2- 172.21.15.70:0/1854925432 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e209353a50 0x55e209355e40 crc :-1 s=READY pgs=448 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39b20e5ecad382f1 server_cookie=6c6592fc4f9569ec in_seq=0 out_seq=0 2024-06-04T05:18:20.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 58013 7.18 MB/sec execute 312 sec latency 134.431 ms 2024-06-04T05:18:20.836 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:18:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.846+0000 7f38217fa640 1 -- 172.21.15.70:0/1854925432 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55e209238360 con 0x55e209353a50 2024-06-04T05:18:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.846+0000 7f38217fa640 10 client.29594 handle_command_reply: tid=0 2024-06-04T05:18:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.846+0000 7f3842f97740 1 client.29594 shutdown 2024-06-04T05:18:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.847+0000 7f3842f97740 1 -- 172.21.15.70:0/1854925432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 msgr2=0x55e2092e2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.847+0000 7f3842f97740 1 --2- 172.21.15.70:0/1854925432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092e2000 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3830017700 tx=0x7f38300179f0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.847+0000 7f3842f97740 1 -- 172.21.15.70:0/1854925432 shutdown_connections 2024-06-04T05:18:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.847+0000 7f3842f97740 1 --2- 172.21.15.70:0/1854925432 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e209353a50 0x55e209355e40 crc :-1 s=READY pgs=448 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.847+0000 7f3842f97740 1 --2- 172.21.15.70:0/1854925432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2092d5a00 0x55e2092dce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.847+0000 7f3842f97740 1 --2- 172.21.15.70:0/1854925432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2092cb140 0x55e2092e2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.847+0000 7f38217fa640 0 client.29594 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:20.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.847+0000 7f3842f97740 1 --2- 172.21.15.70:0/1854925432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2092ca820 0x55e2092e2000 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.847+0000 7f3842f97740 1 -- 172.21.15.70:0/1854925432 >> 172.21.15.70:0/1854925432 conn(0x55e2092c6d30 msgr2=0x55e2092cca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:20.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.847+0000 7f3842f97740 1 -- 172.21.15.70:0/1854925432 shutdown_connections 2024-06-04T05:18:20.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.847+0000 7f3842f97740 1 -- 172.21.15.70:0/1854925432 wait complete. 2024-06-04T05:18:20.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.847+0000 7f3842f97740 20 client.29594 trim_cache size 0 max 16384 2024-06-04T05:18:20.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:18:20.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:18:20.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:18:20.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.848+0000 7f38401a4640 1 -- 172.21.15.70:0/1778422793 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38240b2a10 msgr2=0x7f38240b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:20.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.848+0000 7f38401a4640 1 --2- 172.21.15.70:0/1778422793 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38240b2a10 0x7f38240b4ed0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f38340e00d0 tx=0x7f38340e8550 comp rx=0 tx=0).stop 2024-06-04T05:18:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.848+0000 7f38401a4640 1 -- 172.21.15.70:0/1778422793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3838101980 msgr2=0x7f3838199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.848+0000 7f38401a4640 1 --2- 172.21.15.70:0/1778422793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3838101980 0x7f3838199180 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f383400e9f0 tx=0x7f383400edf0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.848+0000 7f38401a4640 1 -- 172.21.15.70:0/1778422793 shutdown_connections 2024-06-04T05:18:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.848+0000 7f38401a4640 1 --2- 172.21.15.70:0/1778422793 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38240b2a10 0x7f38240b4ed0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.848+0000 7f38401a4640 1 --2- 172.21.15.70:0/1778422793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3838107940 0x7f383819da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.848+0000 7f38401a4640 1 --2- 172.21.15.70:0/1778422793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3838102340 0x7f38381996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.848+0000 7f38401a4640 1 --2- 172.21.15.70:0/1778422793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3838101980 0x7f3838199180 secure :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f383400e9f0 tx=0x7f383400edf0 comp rx=0 tx=0).stop 2024-06-04T05:18:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.848+0000 7f38401a4640 1 -- 172.21.15.70:0/1778422793 >> 172.21.15.70:0/1778422793 conn(0x7f3838078ad0 msgr2=0x7f383806a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.848+0000 7f38401a4640 1 -- 172.21.15.70:0/1778422793 shutdown_connections 2024-06-04T05:18:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:20.848+0000 7f38401a4640 1 -- 172.21.15.70:0/1778422793 wait complete. 2024-06-04T05:18:21.179 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:18:21.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 58177 7.18 MB/sec execute 313 sec latency 97.959 ms 2024-06-04T05:18:21.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:21 smithi098 ceph-mon[35569]: pgmap v666: 97 pgs: 97 active+clean; 850 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 249 op/s 2024-06-04T05:18:22.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:21 smithi070 ceph-mon[32631]: pgmap v666: 97 pgs: 97 active+clean; 850 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 249 op/s 2024-06-04T05:18:22.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:21 smithi112 ceph-mon[36453]: pgmap v666: 97 pgs: 97 active+clean; 850 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 249 op/s 2024-06-04T05:18:22.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 58329 7.17 MB/sec execute 314 sec latency 86.433 ms 2024-06-04T05:18:23.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 58522 7.18 MB/sec execute 315 sec latency 87.405 ms 2024-06-04T05:18:23.833 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:23 smithi070 ceph-mon[32631]: pgmap v667: 97 pgs: 97 active+clean; 854 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.7 MiB/s wr, 449 op/s 2024-06-04T05:18:23.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.839+0000 7f934d75a700 1 -- 172.21.15.70:0/1992426890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934806bab0 msgr2=0x7f934806bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.839+0000 7f934d75a700 1 --2- 172.21.15.70:0/1992426890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934806bab0 0x7f934806bf30 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f933c009990 tx=0x7f933c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.839+0000 7f934d75a700 1 -- 172.21.15.70:0/1992426890 shutdown_connections 2024-06-04T05:18:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.839+0000 7f934d75a700 1 --2- 172.21.15.70:0/1992426890 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f934806c470 0x7f93481103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.839+0000 7f934d75a700 1 --2- 172.21.15.70:0/1992426890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93481095a0 0x7f9348109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.839+0000 7f934d75a700 1 --2- 172.21.15.70:0/1992426890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934806bab0 0x7f934806bf30 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.839+0000 7f934d75a700 1 -- 172.21.15.70:0/1992426890 >> 172.21.15.70:0/1992426890 conn(0x7f93480793b0 msgr2=0x7f93480797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.840+0000 7f934d75a700 1 -- 172.21.15.70:0/1992426890 shutdown_connections 2024-06-04T05:18:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.840+0000 7f934d75a700 1 -- 172.21.15.70:0/1992426890 wait complete. 2024-06-04T05:18:23.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.840+0000 7f934d75a700 1 Processor -- start 2024-06-04T05:18:23.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f934d75a700 1 -- start start 2024-06-04T05:18:23.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f934d75a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f934806bab0 0x7f934819f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f934d75a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934806c470 0x7f934819fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f934d75a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93481095a0 0x7f9348199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f934d75a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9348113560 con 0x7f934806bab0 2024-06-04T05:18:23.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f934d75a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f93481133e0 con 0x7f934806c470 2024-06-04T05:18:23.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f934d75a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f93481136e0 con 0x7f93481095a0 2024-06-04T05:18:23.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f9346ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f934806bab0 0x7f934819f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f9346ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f934806bab0 0x7f934819f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41894/0 (socket says 172.21.15.70:41894) 2024-06-04T05:18:23.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f9346ffd700 1 -- 172.21.15.70:0/3510848203 learned_addr learned my addr 172.21.15.70:0/3510848203 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:23.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f93467fc700 1 --2- 172.21.15.70:0/3510848203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934806c470 0x7f934819fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f93477fe700 1 --2- 172.21.15.70:0/3510848203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93481095a0 0x7f9348199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f93467fc700 1 -- 172.21.15.70:0/3510848203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93481095a0 msgr2=0x7f9348199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:23.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f93467fc700 1 --2- 172.21.15.70:0/3510848203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93481095a0 0x7f9348199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f93467fc700 1 -- 172.21.15.70:0/3510848203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f934806bab0 msgr2=0x7f934819f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:23.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f93467fc700 1 --2- 172.21.15.70:0/3510848203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f934806bab0 0x7f934819f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f93467fc700 1 -- 172.21.15.70:0/3510848203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f934819a0c0 con 0x7f934806c470 2024-06-04T05:18:23.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.841+0000 7f9346ffd700 1 --2- 172.21.15.70:0/3510848203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f934806bab0 0x7f934819f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:23.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.842+0000 7f93467fc700 1 --2- 172.21.15.70:0/3510848203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934806c470 0x7f934819fce0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f933c000c00 tx=0x7f933c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:23.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.842+0000 7f932ffff700 1 -- 172.21.15.70:0/3510848203 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c026070 con 0x7f934806c470 2024-06-04T05:18:23.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.842+0000 7f934d75a700 1 -- 172.21.15.70:0/3510848203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f934819a350 con 0x7f934806c470 2024-06-04T05:18:23.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.842+0000 7f932ffff700 1 -- 172.21.15.70:0/3510848203 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f933c021410 con 0x7f934806c470 2024-06-04T05:18:23.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.842+0000 7f932ffff700 1 -- 172.21.15.70:0/3510848203 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c017650 con 0x7f934806c470 2024-06-04T05:18:23.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.842+0000 7f934d75a700 1 -- 172.21.15.70:0/3510848203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93481a6670 con 0x7f934806c470 2024-06-04T05:18:23.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.845+0000 7f932ffff700 1 -- 172.21.15.70:0/3510848203 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f933c029070 con 0x7f934806c470 2024-06-04T05:18:23.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.846+0000 7f932ffff700 1 --2- 172.21.15.70:0/3510848203 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f93300b2a70 0x7f93300b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.847+0000 7f932ffff700 1 -- 172.21.15.70:0/3510848203 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f933c0e3420 con 0x7f934806c470 2024-06-04T05:18:23.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.847+0000 7f9346ffd700 1 --2- 172.21.15.70:0/3510848203 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f93300b2a70 0x7f93300b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.847+0000 7f9357fefb80 1 Processor -- start 2024-06-04T05:18:23.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.847+0000 7f9357fefb80 1 -- start start 2024-06-04T05:18:23.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.847+0000 7f9357fefb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc98820320 0x55cc98820700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.847+0000 7f9357fefb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881a8a0 0x55cc9881ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.847+0000 7f9346ffd700 1 --2- 172.21.15.70:0/3510848203 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f93300b2a70 0x7f93300b4f30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f934819aaa0 tx=0x7f9338009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:23.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.847+0000 7f9357fefb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc9881b260 0x55cc98827080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.847+0000 7f9357fefb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cc98778d70 con 0x55cc98820320 2024-06-04T05:18:23.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.847+0000 7f9357fefb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cc98768910 con 0x55cc9881a8a0 2024-06-04T05:18:23.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.847+0000 7f9357fefb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cc9876c950 con 0x55cc9881b260 2024-06-04T05:18:23.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.847+0000 7f93467fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881a8a0 0x55cc9881ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.847+0000 7f9346ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc98820320 0x55cc98820700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.847+0000 7f93467fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881a8a0 0x55cc9881ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54612/0 (socket says 172.21.15.70:54612) 2024-06-04T05:18:23.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.847+0000 7f93467fc700 1 -- 172.21.15.70:0/38484446 learned_addr learned my addr 172.21.15.70:0/38484446 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:23.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.847+0000 7f93477fe700 1 --2- 172.21.15.70:0/38484446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc9881b260 0x55cc98827080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.848+0000 7f93467fc700 1 -- 172.21.15.70:0/38484446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc9881b260 msgr2=0x55cc98827080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:23.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.848+0000 7f93467fc700 1 --2- 172.21.15.70:0/38484446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc9881b260 0x55cc98827080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.848+0000 7f93467fc700 1 -- 172.21.15.70:0/38484446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc98820320 msgr2=0x55cc98820700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:23.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.848+0000 7f93467fc700 1 --2- 172.21.15.70:0/38484446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc98820320 0x55cc98820700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.848+0000 7f93467fc700 1 -- 172.21.15.70:0/38484446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc98711fb0 con 0x55cc9881a8a0 2024-06-04T05:18:23.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.848+0000 7f93467fc700 1 --2- 172.21.15.70:0/38484446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881a8a0 0x55cc9881ad20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f933c0aa690 tx=0x7f933c0aa0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5ddc00e6b63a705 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:23.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.848+0000 7f932d7fa700 1 -- 172.21.15.70:0/38484446 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c0aae00 con 0x55cc9881a8a0 2024-06-04T05:18:23.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.848+0000 7f932d7fa700 1 -- 172.21.15.70:0/38484446 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f933c0aa700 con 0x55cc9881a8a0 2024-06-04T05:18:23.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.848+0000 7f932d7fa700 1 -- 172.21.15.70:0/38484446 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c0e0d60 con 0x55cc9881a8a0 2024-06-04T05:18:23.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.849+0000 7f9357fefb80 1 -- 172.21.15.70:0/38484446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881a8a0 msgr2=0x55cc9881ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.849+0000 7f9357fefb80 1 --2- 172.21.15.70:0/38484446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881a8a0 0x55cc9881ad20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f933c0aa690 tx=0x7f933c0aa0d0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.849+0000 7f9357fefb80 1 -- 172.21.15.70:0/38484446 shutdown_connections 2024-06-04T05:18:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.849+0000 7f9357fefb80 1 --2- 172.21.15.70:0/38484446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc9881b260 0x55cc98827080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.849+0000 7f9357fefb80 1 --2- 172.21.15.70:0/38484446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc98820320 0x55cc98820700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.849+0000 7f9357fefb80 1 --2- 172.21.15.70:0/38484446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881a8a0 0x55cc9881ad20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.849+0000 7f9357fefb80 1 -- 172.21.15.70:0/38484446 >> 172.21.15.70:0/38484446 conn(0x55cc98746240 msgr2=0x55cc988153e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.849+0000 7f9357fefb80 1 -- 172.21.15.70:0/38484446 shutdown_connections 2024-06-04T05:18:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.849+0000 7f9357fefb80 1 -- 172.21.15.70:0/38484446 wait complete. 2024-06-04T05:18:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.849+0000 7f9357fefb80 1 -- start start 2024-06-04T05:18:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.850+0000 7f9357fefb80 1 Processor -- start 2024-06-04T05:18:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.850+0000 7f9357fefb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc9881a8a0 0x55cc9882e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.850+0000 7f9357fefb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881b260 0x55cc9882e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.850+0000 7f9357fefb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc98820320 0x55cc98828500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.850+0000 7f9357fefb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cc9866efa0 con 0x55cc98820320 2024-06-04T05:18:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.850+0000 7f9357fefb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cc986d0da0 con 0x55cc9881b260 2024-06-04T05:18:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.850+0000 7f9357fefb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cc98768910 con 0x55cc9881a8a0 2024-06-04T05:18:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.850+0000 7f93467fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881b260 0x55cc9882e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.850+0000 7f93467fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881b260 0x55cc9882e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54614/0 (socket says 172.21.15.70:54614) 2024-06-04T05:18:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.850+0000 7f93467fc700 1 -- 172.21.15.70:0/1376221023 learned_addr learned my addr 172.21.15.70:0/1376221023 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.850+0000 7f9357fefb80 10 client.0 _ll_register_callbacks cb 0x55cc98703950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.850+0000 7f9346ffd700 1 --2- 172.21.15.70:0/1376221023 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc9881a8a0 0x55cc9882e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.850+0000 7f93477fe700 1 --2- 172.21.15.70:0/1376221023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc98820320 0x55cc98828500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.850+0000 7f93467fc700 1 -- 172.21.15.70:0/1376221023 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc9881a8a0 msgr2=0x55cc9882e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.850+0000 7f93467fc700 1 --2- 172.21.15.70:0/1376221023 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc9881a8a0 0x55cc9882e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.850+0000 7f93467fc700 1 -- 172.21.15.70:0/1376221023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc98820320 msgr2=0x55cc98828500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.851+0000 7f93467fc700 1 --2- 172.21.15.70:0/1376221023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc98820320 0x55cc98828500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.851+0000 7f93467fc700 1 -- 172.21.15.70:0/1376221023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc986b88d0 con 0x55cc9881b260 2024-06-04T05:18:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.851+0000 7f93477fe700 1 --2- 172.21.15.70:0/1376221023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc98820320 0x55cc98828500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:18:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.851+0000 7f93467fc700 1 --2- 172.21.15.70:0/1376221023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881b260 0x55cc9882e6c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f933c0aa400 tx=0x7f933c005c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=158e335d7f64094a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.851+0000 7f932aff5700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:18:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.851+0000 7f932aff5700 1 -- 172.21.15.70:0/1376221023 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c0c3070 con 0x55cc9881b260 2024-06-04T05:18:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.851+0000 7f932aff5700 1 -- 172.21.15.70:0/1376221023 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f933c0aab70 con 0x55cc9881b260 2024-06-04T05:18:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.851+0000 7f9357fefb80 1 -- 172.21.15.70:0/1376221023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f933c0aa700 con 0x55cc9881b260 2024-06-04T05:18:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.851+0000 7f932aff5700 1 -- 172.21.15.70:0/1376221023 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f933c0aeab0 con 0x55cc9881b260 2024-06-04T05:18:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.851+0000 7f9357fefb80 10 client.29606 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.851+0000 7f9357fefb80 1 -- 172.21.15.70:0/1376221023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55cc98722760 con 0x55cc9881b260 2024-06-04T05:18:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.852+0000 7f932aff5700 1 -- 172.21.15.70:0/1376221023 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f933c0aa700 con 0x55cc9881b260 2024-06-04T05:18:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.852+0000 7f932aff5700 1 -- 172.21.15.70:0/1376221023 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f933c00fb10 con 0x55cc9881b260 2024-06-04T05:18:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.852+0000 7f9357fefb80 10 client.29606 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.852+0000 7f9357fefb80 10 client.29606 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:18:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.852+0000 7f9357fefb80 20 client.29606 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.852+0000 7f9357fefb80 1 --2- 172.21.15.70:0/1376221023 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cc988381c0 0x55cc9883a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.852+0000 7f9357fefb80 4 client.29606 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:18:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.852+0000 7f9357fefb80 1 -- 172.21.15.70:0/1376221023 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc9877acc0 con 0x55cc988381c0 2024-06-04T05:18:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.853+0000 7f9346ffd700 1 --2- 172.21.15.70:0/1376221023 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cc988381c0 0x55cc9883a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.853+0000 7f9346ffd700 1 --2- 172.21.15.70:0/1376221023 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cc988381c0 0x55cc9883a5b0 crc :-1 s=READY pgs=449 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcf0b53f8a5755f9 server_cookie=e70da7f846ea34d3 in_seq=0 out_seq=0 2024-06-04T05:18:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.853+0000 7f932aff5700 10 client.29606 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:23.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.854+0000 7f932aff5700 1 -- 172.21.15.70:0/1376221023 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cc9877acc0 con 0x55cc988381c0 2024-06-04T05:18:23.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.854+0000 7f932aff5700 10 client.29606 handle_command_reply: tid=0 2024-06-04T05:18:23.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.854+0000 7f9357fefb80 1 client.29606 shutdown 2024-06-04T05:18:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.854+0000 7f9357fefb80 1 -- 172.21.15.70:0/1376221023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881b260 msgr2=0x55cc9882e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.854+0000 7f9357fefb80 1 --2- 172.21.15.70:0/1376221023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881b260 0x55cc9882e6c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f933c0aa400 tx=0x7f933c005c70 comp rx=0 tx=0).stop 2024-06-04T05:18:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.854+0000 7f9357fefb80 1 -- 172.21.15.70:0/1376221023 shutdown_connections 2024-06-04T05:18:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.854+0000 7f9357fefb80 1 --2- 172.21.15.70:0/1376221023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc98820320 0x55cc98828500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.854+0000 7f9357fefb80 1 --2- 172.21.15.70:0/1376221023 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc9881a8a0 0x55cc9882e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.854+0000 7f9357fefb80 1 --2- 172.21.15.70:0/1376221023 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cc988381c0 0x55cc9883a5b0 crc :-1 s=READY pgs=449 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.854+0000 7f9357fefb80 1 --2- 172.21.15.70:0/1376221023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881b260 0x55cc9882e6c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.854+0000 7f932aff5700 0 client.29606 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.854+0000 7f9357fefb80 1 -- 172.21.15.70:0/1376221023 >> 172.21.15.70:0/1376221023 conn(0x55cc98746240 msgr2=0x55cc98814af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.855+0000 7f9357fefb80 1 -- 172.21.15.70:0/1376221023 shutdown_connections 2024-06-04T05:18:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.855+0000 7f9357fefb80 1 -- 172.21.15.70:0/1376221023 wait complete. 2024-06-04T05:18:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.855+0000 7f9357fefb80 20 client.29606 trim_cache size 0 max 16384 2024-06-04T05:18:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.891+0000 7f9357fefb80 1 Processor -- start 2024-06-04T05:18:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.891+0000 7f9357fefb80 1 -- start start 2024-06-04T05:18:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.891+0000 7f9357fefb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881a8a0 0x55cc988284c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.891+0000 7f9357fefb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc9881b260 0x55cc9881bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.891+0000 7f9357fefb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc98820320 0x55cc9881c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.891+0000 7f9357fefb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f93481abb70 con 0x55cc9881b260 2024-06-04T05:18:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.891+0000 7f9357fefb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f93481ad710 con 0x55cc9881a8a0 2024-06-04T05:18:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.891+0000 7f9357fefb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f93481ac2a0 con 0x55cc98820320 2024-06-04T05:18:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.892+0000 7f93467fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc9881b260 0x55cc9881bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.892+0000 7f93477fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc98820320 0x55cc9881c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.892+0000 7f93467fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc9881b260 0x55cc9881bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41918/0 (socket says 172.21.15.70:41918) 2024-06-04T05:18:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.892+0000 7f93467fc700 1 -- 172.21.15.70:0/71979529 learned_addr learned my addr 172.21.15.70:0/71979529 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.892+0000 7f9346ffd700 1 --2- 172.21.15.70:0/71979529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881a8a0 0x55cc988284c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.892+0000 7f93477fe700 1 -- 172.21.15.70:0/71979529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881a8a0 msgr2=0x55cc988284c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.892+0000 7f93477fe700 1 --2- 172.21.15.70:0/71979529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881a8a0 0x55cc988284c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.892+0000 7f93477fe700 1 -- 172.21.15.70:0/71979529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc9881b260 msgr2=0x55cc9881bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:23.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.892+0000 7f93477fe700 1 --2- 172.21.15.70:0/71979529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc9881b260 0x55cc9881bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.892+0000 7f93477fe700 1 -- 172.21.15.70:0/71979529 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc9882ea10 con 0x55cc98820320 2024-06-04T05:18:23.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.892+0000 7f9346ffd700 1 --2- 172.21.15.70:0/71979529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881a8a0 0x55cc988284c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:23.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.892+0000 7f93477fe700 1 --2- 172.21.15.70:0/71979529 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc98820320 0x55cc9881c280 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9340009480 tx=0x7f9340009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e90584252e43dc9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:23.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.892+0000 7f932cff9700 1 -- 172.21.15.70:0/71979529 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9340003cc0 con 0x55cc98820320 2024-06-04T05:18:23.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.892+0000 7f932cff9700 1 -- 172.21.15.70:0/71979529 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f934001b030 con 0x55cc98820320 2024-06-04T05:18:23.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.892+0000 7f932cff9700 1 -- 172.21.15.70:0/71979529 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9340017740 con 0x55cc98820320 2024-06-04T05:18:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.893+0000 7f9357fefb80 1 -- 172.21.15.70:0/71979529 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc98820320 msgr2=0x55cc9881c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.893+0000 7f9357fefb80 1 --2- 172.21.15.70:0/71979529 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc98820320 0x55cc9881c280 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9340009480 tx=0x7f9340009920 comp rx=0 tx=0).stop 2024-06-04T05:18:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.893+0000 7f9357fefb80 1 -- 172.21.15.70:0/71979529 shutdown_connections 2024-06-04T05:18:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.893+0000 7f9357fefb80 1 --2- 172.21.15.70:0/71979529 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc98820320 0x55cc9881c280 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.893+0000 7f9357fefb80 1 --2- 172.21.15.70:0/71979529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881a8a0 0x55cc988284c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.893+0000 7f9357fefb80 1 --2- 172.21.15.70:0/71979529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc9881b260 0x55cc9881bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.893+0000 7f9357fefb80 1 -- 172.21.15.70:0/71979529 >> 172.21.15.70:0/71979529 conn(0x55cc98746240 msgr2=0x55cc98827db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.893+0000 7f9357fefb80 1 -- 172.21.15.70:0/71979529 shutdown_connections 2024-06-04T05:18:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.893+0000 7f9357fefb80 1 -- 172.21.15.70:0/71979529 wait complete. 2024-06-04T05:18:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.894+0000 7f9357fefb80 1 -- start start 2024-06-04T05:18:23.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.894+0000 7f9357fefb80 1 Processor -- start 2024-06-04T05:18:23.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.894+0000 7f9357fefb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc9881a8a0 0x55cc98834160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.894+0000 7f9357fefb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881b260 0x55cc988346a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.894+0000 7f9357fefb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc98820320 0x55cc98832ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.894+0000 7f9357fefb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f93481ad040 con 0x55cc98820320 2024-06-04T05:18:23.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.894+0000 7f9357fefb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f93481ac970 con 0x55cc9881b260 2024-06-04T05:18:23.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.894+0000 7f9357fefb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f93481a9b50 con 0x55cc9881a8a0 2024-06-04T05:18:23.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.894+0000 7f9357fefb80 10 client.0 _ll_register_callbacks cb 0x55cc98703950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:23.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.894+0000 7f9346ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc9881a8a0 0x55cc98834160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.894+0000 7f9346ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc9881a8a0 0x55cc98834160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39154/0 (socket says 172.21.15.70:39154) 2024-06-04T05:18:23.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.894+0000 7f9346ffd700 1 -- 172.21.15.70:0/1337919321 learned_addr learned my addr 172.21.15.70:0/1337919321 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:23.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.894+0000 7f93467fc700 1 --2- 172.21.15.70:0/1337919321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881b260 0x55cc988346a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.894+0000 7f93477fe700 1 --2- 172.21.15.70:0/1337919321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc98820320 0x55cc98832ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.895+0000 7f9346ffd700 1 -- 172.21.15.70:0/1337919321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881b260 msgr2=0x55cc988346a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:23.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.895+0000 7f9346ffd700 1 --2- 172.21.15.70:0/1337919321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881b260 0x55cc988346a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.895+0000 7f9346ffd700 1 -- 172.21.15.70:0/1337919321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc98820320 msgr2=0x55cc98832ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:23.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.895+0000 7f9346ffd700 1 --2- 172.21.15.70:0/1337919321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc98820320 0x55cc98832ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.895+0000 7f9346ffd700 1 -- 172.21.15.70:0/1337919321 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc98833420 con 0x55cc9881a8a0 2024-06-04T05:18:23.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.895+0000 7f9346ffd700 1 --2- 172.21.15.70:0/1337919321 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc9881a8a0 0x55cc98834160 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x55cc98828b50 tx=0x7f93380047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eedc6f73d8c45aae server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:23.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.895+0000 7f932aff5700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:18:23.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.895+0000 7f932aff5700 1 -- 172.21.15.70:0/1337919321 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9338011e60 con 0x55cc9881a8a0 2024-06-04T05:18:23.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.895+0000 7f9357fefb80 1 -- 172.21.15.70:0/1337919321 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f934001b030 con 0x55cc9881a8a0 2024-06-04T05:18:23.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.895+0000 7f932aff5700 1 -- 172.21.15.70:0/1337919321 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f9338029310 con 0x55cc9881a8a0 2024-06-04T05:18:23.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.895+0000 7f932aff5700 1 -- 172.21.15.70:0/1337919321 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f9338003f50 con 0x55cc9881a8a0 2024-06-04T05:18:23.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.895+0000 7f9357fefb80 10 client.29758 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:23.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.895+0000 7f9357fefb80 1 -- 172.21.15.70:0/1337919321 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55cc988a7c00 con 0x55cc9881a8a0 2024-06-04T05:18:23.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.895+0000 7f932aff5700 1 -- 172.21.15.70:0/1337919321 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f934001b030 con 0x55cc9881a8a0 2024-06-04T05:18:23.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.896+0000 7f932aff5700 1 -- 172.21.15.70:0/1337919321 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f93380327c0 con 0x55cc9881a8a0 2024-06-04T05:18:23.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.896+0000 7f9357fefb80 10 client.29758 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:23.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.896+0000 7f9357fefb80 10 client.29758 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:18:23.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.896+0000 7f9357fefb80 20 client.29758 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:23.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.896+0000 7f9357fefb80 1 --2- 172.21.15.70:0/1337919321 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cc988a8260 0x55cc988aa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:23.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.896+0000 7f9357fefb80 4 client.29758 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:18:23.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.896+0000 7f9357fefb80 1 -- 172.21.15.70:0/1337919321 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cc987744a0 con 0x55cc988a8260 2024-06-04T05:18:23.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.896+0000 7f93467fc700 1 --2- 172.21.15.70:0/1337919321 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cc988a8260 0x55cc988aa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:23.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.896+0000 7f93467fc700 1 --2- 172.21.15.70:0/1337919321 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cc988a8260 0x55cc988aa6c0 crc :-1 s=READY pgs=450 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab468d981cdc72d0 server_cookie=1c3e036865abd454 in_seq=0 out_seq=0 2024-06-04T05:18:23.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.896+0000 7f932aff5700 10 client.29758 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:23.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.897+0000 7f932aff5700 1 -- 172.21.15.70:0/1337919321 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55cc987744a0 con 0x55cc988a8260 2024-06-04T05:18:23.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.897+0000 7f932aff5700 10 client.29758 handle_command_reply: tid=0 2024-06-04T05:18:23.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.898+0000 7f9357fefb80 1 client.29758 shutdown 2024-06-04T05:18:23.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.898+0000 7f9357fefb80 1 -- 172.21.15.70:0/1337919321 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc9881a8a0 msgr2=0x55cc98834160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:23.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.898+0000 7f9357fefb80 1 --2- 172.21.15.70:0/1337919321 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc9881a8a0 0x55cc98834160 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x55cc98828b50 tx=0x7f93380047c0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.898+0000 7f9357fefb80 1 -- 172.21.15.70:0/1337919321 shutdown_connections 2024-06-04T05:18:23.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.898+0000 7f9357fefb80 1 --2- 172.21.15.70:0/1337919321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc98820320 0x55cc98832ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.898+0000 7f9357fefb80 1 --2- 172.21.15.70:0/1337919321 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc9881a8a0 0x55cc98834160 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.898+0000 7f9357fefb80 1 --2- 172.21.15.70:0/1337919321 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cc988a8260 0x55cc988aa6c0 crc :-1 s=READY pgs=450 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.898+0000 7f932aff5700 0 client.29758 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:23.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.898+0000 7f9357fefb80 1 --2- 172.21.15.70:0/1337919321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc9881b260 0x55cc988346a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.898+0000 7f9357fefb80 1 -- 172.21.15.70:0/1337919321 >> 172.21.15.70:0/1337919321 conn(0x55cc98746240 msgr2=0x55cc988279c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:23.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.898+0000 7f9357fefb80 1 -- 172.21.15.70:0/1337919321 shutdown_connections 2024-06-04T05:18:23.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.898+0000 7f9357fefb80 1 -- 172.21.15.70:0/1337919321 wait complete. 2024-06-04T05:18:23.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.898+0000 7f9357fefb80 20 client.29758 trim_cache size 0 max 16384 2024-06-04T05:18:23.904 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:18:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:18:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:18:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "0e077767-01b8-486a-bba7-ea8a77044aff": { 2024-06-04T05:18:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:18:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "0e077767-01b8-486a-bba7-ea8a77044aff", 2024-06-04T05:18:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:18:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:18:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:18:23.905 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:18:23.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.899+0000 7f934d75a700 1 -- 172.21.15.70:0/3510848203 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f93300b2a70 msgr2=0x7f93300b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:23.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.899+0000 7f934d75a700 1 --2- 172.21.15.70:0/3510848203 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f93300b2a70 0x7f93300b4f30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f934819aaa0 tx=0x7f9338009260 comp rx=0 tx=0).stop 2024-06-04T05:18:23.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.899+0000 7f934d75a700 1 -- 172.21.15.70:0/3510848203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934806c470 msgr2=0x7f934819fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:23.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.899+0000 7f934d75a700 1 --2- 172.21.15.70:0/3510848203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934806c470 0x7f934819fce0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f933c000c00 tx=0x7f933c0047b0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.899+0000 7f934d75a700 1 -- 172.21.15.70:0/3510848203 shutdown_connections 2024-06-04T05:18:23.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.899+0000 7f934d75a700 1 --2- 172.21.15.70:0/3510848203 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f93300b2a70 0x7f93300b4f30 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.899+0000 7f934d75a700 1 --2- 172.21.15.70:0/3510848203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93481095a0 0x7f9348199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.899+0000 7f934d75a700 1 --2- 172.21.15.70:0/3510848203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f934806bab0 0x7f934819f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.899+0000 7f934d75a700 1 --2- 172.21.15.70:0/3510848203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f934806c470 0x7f934819fce0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:23.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.899+0000 7f934d75a700 1 -- 172.21.15.70:0/3510848203 >> 172.21.15.70:0/3510848203 conn(0x7f93480793b0 msgr2=0x7f9348101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:23.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.900+0000 7f934d75a700 1 -- 172.21.15.70:0/3510848203 shutdown_connections 2024-06-04T05:18:23.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:23.900+0000 7f934d75a700 1 -- 172.21.15.70:0/3510848203 wait complete. 2024-06-04T05:18:23.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:23 smithi098 ceph-mon[35569]: pgmap v667: 97 pgs: 97 active+clean; 854 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.7 MiB/s wr, 449 op/s 2024-06-04T05:18:24.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:23 smithi112 ceph-mon[36453]: pgmap v667: 97 pgs: 97 active+clean; 854 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.7 MiB/s wr, 449 op/s 2024-06-04T05:18:24.694 INFO:tasks.cephfs.filesystem:scrub status for tag:0e077767-01b8-486a-bba7-ea8a77044aff - {'path': '/', 'tag': '0e077767-01b8-486a-bba7-ea8a77044aff', 'options': 'recursive,force'} 2024-06-04T05:18:24.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 58706 7.19 MB/sec execute 316 sec latency 62.495 ms 2024-06-04T05:18:25.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 58874 7.18 MB/sec execute 317 sec latency 59.504 ms 2024-06-04T05:18:25.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:25 smithi098 ceph-mon[35569]: pgmap v668: 97 pgs: 97 active+clean; 854 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 306 op/s 2024-06-04T05:18:26.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:25 smithi070 ceph-mon[32631]: pgmap v668: 97 pgs: 97 active+clean; 854 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 306 op/s 2024-06-04T05:18:26.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:25 smithi112 ceph-mon[36453]: pgmap v668: 97 pgs: 97 active+clean; 854 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 306 op/s 2024-06-04T05:18:26.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 59051 7.19 MB/sec execute 318 sec latency 57.357 ms 2024-06-04T05:18:27.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 59240 7.20 MB/sec execute 319 sec latency 56.776 ms 2024-06-04T05:18:27.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:27 smithi098 ceph-mon[35569]: pgmap v669: 97 pgs: 97 active+clean; 860 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.3 MiB/s wr, 446 op/s 2024-06-04T05:18:28.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:27 smithi070 ceph-mon[32631]: pgmap v669: 97 pgs: 97 active+clean; 860 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.3 MiB/s wr, 446 op/s 2024-06-04T05:18:28.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:27 smithi112 ceph-mon[36453]: pgmap v669: 97 pgs: 97 active+clean; 860 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.3 MiB/s wr, 446 op/s 2024-06-04T05:18:28.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 59450 7.22 MB/sec execute 320 sec latency 61.694 ms 2024-06-04T05:18:28.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:28 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:18:28.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:28 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:18:28.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:28 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:18:28.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:28 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:18:29.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:28 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:18:29.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:28 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:18:29.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:28 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:18:29.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:28 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:18:29.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:28 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:18:29.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:28 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:18:29.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:28 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:18:29.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:28 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:18:29.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 59640 7.22 MB/sec execute 321 sec latency 59.736 ms 2024-06-04T05:18:29.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:29 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:18:29.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:29 smithi098 ceph-mon[35569]: pgmap v670: 97 pgs: 97 active+clean; 860 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.6 MiB/s wr, 340 op/s 2024-06-04T05:18:30.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:29 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:18:30.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:29 smithi070 ceph-mon[32631]: pgmap v670: 97 pgs: 97 active+clean; 860 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.6 MiB/s wr, 340 op/s 2024-06-04T05:18:30.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:29 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:18:30.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:29 smithi112 ceph-mon[36453]: pgmap v670: 97 pgs: 97 active+clean; 860 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.6 MiB/s wr, 340 op/s 2024-06-04T05:18:30.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 59777 7.21 MB/sec execute 322 sec latency 227.884 ms 2024-06-04T05:18:31.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 59931 7.22 MB/sec execute 323 sec latency 409.344 ms 2024-06-04T05:18:31.988 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:31 smithi112 ceph-mon[36453]: pgmap v671: 97 pgs: 97 active+clean; 860 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.6 MiB/s wr, 340 op/s 2024-06-04T05:18:31.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:31 smithi098 ceph-mon[35569]: pgmap v671: 97 pgs: 97 active+clean; 860 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.6 MiB/s wr, 340 op/s 2024-06-04T05:18:32.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:31 smithi070 ceph-mon[32631]: pgmap v671: 97 pgs: 97 active+clean; 860 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.6 MiB/s wr, 340 op/s 2024-06-04T05:18:32.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 60161 7.25 MB/sec execute 324 sec latency 55.270 ms 2024-06-04T05:18:33.824 INFO:tasks.workunit.client.0.smithi070.stdout: 10 60324 7.25 MB/sec execute 325 sec latency 49.696 ms 2024-06-04T05:18:33.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.972+0000 7f7b52daf640 1 -- 172.21.15.70:0/1002293190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b4c101bc0 msgr2=0x7f7b4c102020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:33.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.972+0000 7f7b52daf640 1 --2- 172.21.15.70:0/1002293190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b4c101bc0 0x7f7b4c102020 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009900 tx=0x7f7b3802a410 comp rx=0 tx=0).stop 2024-06-04T05:18:33.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.972+0000 7f7b52daf640 1 -- 172.21.15.70:0/1002293190 shutdown_connections 2024-06-04T05:18:33.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.972+0000 7f7b52daf640 1 --2- 172.21.15.70:0/1002293190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b4c102560 0x7f7b4c10e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:33.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.972+0000 7f7b52daf640 1 --2- 172.21.15.70:0/1002293190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b4c101bc0 0x7f7b4c102020 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:33.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.972+0000 7f7b52daf640 1 --2- 172.21.15.70:0/1002293190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b4c1079f0 0x7f7b4c107dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:33.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.972+0000 7f7b52daf640 1 -- 172.21.15.70:0/1002293190 >> 172.21.15.70:0/1002293190 conn(0x7f7b4c078830 msgr2=0x7f7b4c078c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:33.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.973+0000 7f7b52daf640 1 -- 172.21.15.70:0/1002293190 shutdown_connections 2024-06-04T05:18:33.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.973+0000 7f7b52daf640 1 -- 172.21.15.70:0/1002293190 wait complete. 2024-06-04T05:18:33.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.973+0000 7f7b52daf640 1 Processor -- start 2024-06-04T05:18:33.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.973+0000 7f7b52daf640 1 -- start start 2024-06-04T05:18:33.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.974+0000 7f7b52daf640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b4c101bc0 0x7f7b4c19b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:33.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.974+0000 7f7b52daf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b4c102560 0x7f7b4c19b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:33.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.974+0000 7f7b52daf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b4c1079f0 0x7f7b4c19fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:33.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.974+0000 7f7b52daf640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c111b20 con 0x7f7b4c1079f0 2024-06-04T05:18:33.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.974+0000 7f7b52daf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c1119a0 con 0x7f7b4c102560 2024-06-04T05:18:33.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.974+0000 7f7b52daf640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c111ca0 con 0x7f7b4c101bc0 2024-06-04T05:18:33.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.974+0000 7f7b4bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b4c102560 0x7f7b4c19b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:33.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.974+0000 7f7b5134c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b4c1079f0 0x7f7b4c19fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:33.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.974+0000 7f7b5134c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b4c1079f0 0x7f7b4c19fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58174/0 (socket says 172.21.15.70:58174) 2024-06-04T05:18:33.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.974+0000 7f7b4bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b4c102560 0x7f7b4c19b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:35014/0 (socket says 172.21.15.70:35014) 2024-06-04T05:18:33.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.974+0000 7f7b5134c640 1 -- 172.21.15.70:0/2117013107 learned_addr learned my addr 172.21.15.70:0/2117013107 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:33.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.974+0000 7f7b50b4b640 1 --2- 172.21.15.70:0/2117013107 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b4c101bc0 0x7f7b4c19b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:33.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.974+0000 7f7b4bfff640 1 -- 172.21.15.70:0/2117013107 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b4c101bc0 msgr2=0x7f7b4c19b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:33.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.974+0000 7f7b4bfff640 1 --2- 172.21.15.70:0/2117013107 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b4c101bc0 0x7f7b4c19b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:33.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.974+0000 7f7b4bfff640 1 -- 172.21.15.70:0/2117013107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b4c1079f0 msgr2=0x7f7b4c19fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:33.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.974+0000 7f7b4bfff640 1 --2- 172.21.15.70:0/2117013107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b4c1079f0 0x7f7b4c19fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:33.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.974+0000 7f7b4bfff640 1 -- 172.21.15.70:0/2117013107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b4c1a0290 con 0x7f7b4c102560 2024-06-04T05:18:33.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.975+0000 7f7b4bfff640 1 --2- 172.21.15.70:0/2117013107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b4c102560 0x7f7b4c19b7e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7b3802a930 tx=0x7f7b38003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:33.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.975+0000 7f7b49ffb640 1 -- 172.21.15.70:0/2117013107 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3803f070 con 0x7f7b4c102560 2024-06-04T05:18:33.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.975+0000 7f7b49ffb640 1 -- 172.21.15.70:0/2117013107 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7b3803baf0 con 0x7f7b4c102560 2024-06-04T05:18:33.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.975+0000 7f7b52daf640 1 -- 172.21.15.70:0/2117013107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b4c1a0520 con 0x7f7b4c102560 2024-06-04T05:18:33.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.975+0000 7f7b49ffb640 1 -- 172.21.15.70:0/2117013107 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38035030 con 0x7f7b4c102560 2024-06-04T05:18:33.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.975+0000 7f7b52daf640 1 -- 172.21.15.70:0/2117013107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b4c1a7e10 con 0x7f7b4c102560 2024-06-04T05:18:33.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.978+0000 7f7b49ffb640 1 -- 172.21.15.70:0/2117013107 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7b38042070 con 0x7f7b4c102560 2024-06-04T05:18:33.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.979+0000 7f7b55b9e740 1 Processor -- start 2024-06-04T05:18:33.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.979+0000 7f7b55b9e740 1 -- start start 2024-06-04T05:18:33.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.979+0000 7f7b49ffb640 1 --2- 172.21.15.70:0/2117013107 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7b3c0b2a10 0x7f7b3c0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:33.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.979+0000 7f7b55b9e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55967040f820 0x55967040fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:33.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.979+0000 7f7b55b9e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559670410140 0x55967041a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:33.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.979+0000 7f7b55b9e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967041aa00 0x55967041cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:33.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.979+0000 7f7b55b9e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559670381c30 con 0x55967040f820 2024-06-04T05:18:33.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.979+0000 7f7b55b9e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5596703717d0 con 0x55967041aa00 2024-06-04T05:18:33.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.979+0000 7f7b55b9e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559670375810 con 0x559670410140 2024-06-04T05:18:33.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.979+0000 7f7b5134c640 1 --2- 172.21.15.70:0/2117013107 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7b3c0b2a10 0x7f7b3c0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:33.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.979+0000 7f7b4bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559670410140 0x55967041a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:33.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.979+0000 7f7b4bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559670410140 0x55967041a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39100/0 (socket says 172.21.15.70:39100) 2024-06-04T05:18:33.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.979+0000 7f7b50b4b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55967040f820 0x55967040fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:33.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.979+0000 7f7b4bfff640 1 -- 172.21.15.70:0/2846072157 learned_addr learned my addr 172.21.15.70:0/2846072157 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:33.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.979+0000 7f7b5134c640 1 --2- 172.21.15.70:0/2846072157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967041aa00 0x55967041cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:33.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.979+0000 7f7b49ffb640 1 -- 172.21.15.70:0/2117013107 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f7b38102ae0 con 0x7f7b4c102560 2024-06-04T05:18:33.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.980+0000 7f7b4bfff640 1 -- 172.21.15.70:0/2846072157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967041aa00 msgr2=0x55967041cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:33.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.980+0000 7f7b4bfff640 1 --2- 172.21.15.70:0/2846072157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967041aa00 0x55967041cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:33.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.980+0000 7f7b4bfff640 1 -- 172.21.15.70:0/2846072157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55967040f820 msgr2=0x55967040fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:33.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.980+0000 7f7b4bfff640 1 --2- 172.21.15.70:0/2846072157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55967040f820 0x55967040fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:33.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.980+0000 7f7b4bfff640 1 -- 172.21.15.70:0/2846072157 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559670086210 con 0x559670410140 2024-06-04T05:18:33.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.980+0000 7f7b5134c640 1 --2- 172.21.15.70:0/2846072157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967041aa00 0x55967041cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:18:33.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.980+0000 7f7b5134c640 1 --2- 172.21.15.70:0/2117013107 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7b3c0b2a10 0x7f7b3c0b4ed0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7b44009aa0 tx=0x7f7b44009c40 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:33.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.980+0000 7f7b4bfff640 1 --2- 172.21.15.70:0/2846072157 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559670410140 0x55967041a4c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7b3802a9d0 tx=0x7f7b380c95b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d46461a242fdcad3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:33.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.980+0000 7f7b36ffd640 1 -- 172.21.15.70:0/2846072157 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38049040 con 0x559670410140 2024-06-04T05:18:33.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.980+0000 7f7b36ffd640 1 -- 172.21.15.70:0/2846072157 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7b380c9e40 con 0x559670410140 2024-06-04T05:18:33.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.980+0000 7f7b36ffd640 1 -- 172.21.15.70:0/2846072157 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38100610 con 0x559670410140 2024-06-04T05:18:33.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.981+0000 7f7b55b9e740 1 -- 172.21.15.70:0/2846072157 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559670410140 msgr2=0x55967041a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:33.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.981+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/2846072157 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559670410140 0x55967041a4c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7b3802a9d0 tx=0x7f7b380c95b0 comp rx=0 tx=0).stop 2024-06-04T05:18:33.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.981+0000 7f7b55b9e740 1 -- 172.21.15.70:0/2846072157 shutdown_connections 2024-06-04T05:18:33.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.981+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/2846072157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967041aa00 0x55967041cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:33.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.981+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/2846072157 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559670410140 0x55967041a4c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:33.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.981+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/2846072157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55967040f820 0x55967040fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:33.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.981+0000 7f7b55b9e740 1 -- 172.21.15.70:0/2846072157 >> 172.21.15.70:0/2846072157 conn(0x55967040bd30 msgr2=0x55967040c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:33.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.981+0000 7f7b55b9e740 1 -- 172.21.15.70:0/2846072157 shutdown_connections 2024-06-04T05:18:33.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.981+0000 7f7b55b9e740 1 -- 172.21.15.70:0/2846072157 wait complete. 2024-06-04T05:18:33.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.982+0000 7f7b55b9e740 1 -- start start 2024-06-04T05:18:33.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.982+0000 7f7b55b9e740 1 Processor -- start 2024-06-04T05:18:33.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.982+0000 7f7b55b9e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55967040f820 0x559670417a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:33.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.982+0000 7f7b55b9e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559670410140 0x559670417fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:33.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.982+0000 7f7b55b9e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967041aa00 0x5596704184f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:33.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.982+0000 7f7b55b9e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559670057790 con 0x559670410140 2024-06-04T05:18:33.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.982+0000 7f7b55b9e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55967032d890 con 0x55967041aa00 2024-06-04T05:18:33.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.982+0000 7f7b55b9e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55967041d3f0 con 0x55967040f820 2024-06-04T05:18:33.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.982+0000 7f7b55b9e740 10 client.0 _ll_register_callbacks cb 0x559670085a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:33.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.982+0000 7f7b4bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559670410140 0x559670417fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:33.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.982+0000 7f7b4bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559670410140 0x559670417fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58200/0 (socket says 172.21.15.70:58200) 2024-06-04T05:18:33.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.982+0000 7f7b50b4b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55967040f820 0x559670417a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:33.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.982+0000 7f7b4bfff640 1 -- 172.21.15.70:0/3964723748 learned_addr learned my addr 172.21.15.70:0/3964723748 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:33.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.982+0000 7f7b5134c640 1 --2- 172.21.15.70:0/3964723748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967041aa00 0x5596704184f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:33.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.983+0000 7f7b4bfff640 1 -- 172.21.15.70:0/3964723748 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55967040f820 msgr2=0x559670417a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:33.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.983+0000 7f7b4bfff640 1 --2- 172.21.15.70:0/3964723748 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55967040f820 0x559670417a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:33.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.983+0000 7f7b4bfff640 1 -- 172.21.15.70:0/3964723748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967041aa00 msgr2=0x5596704184f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:33.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.983+0000 7f7b4bfff640 1 --2- 172.21.15.70:0/3964723748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967041aa00 0x5596704184f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:33.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.983+0000 7f7b4bfff640 1 -- 172.21.15.70:0/3964723748 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559670152620 con 0x559670410140 2024-06-04T05:18:33.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.983+0000 7f7b4bfff640 1 --2- 172.21.15.70:0/3964723748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559670410140 0x559670417fb0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f7b380c5ec0 tx=0x7f7b38100510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=330cc5784abe75ff server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:33.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.983+0000 7f7b2bfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:18:33.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.983+0000 7f7b2bfff640 1 -- 172.21.15.70:0/3964723748 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38049040 con 0x559670410140 2024-06-04T05:18:33.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.983+0000 7f7b2bfff640 1 -- 172.21.15.70:0/3964723748 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f7b380ca0a0 con 0x559670410140 2024-06-04T05:18:33.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.983+0000 7f7b55b9e740 1 -- 172.21.15.70:0/3964723748 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b380c9e40 con 0x559670410140 2024-06-04T05:18:33.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.983+0000 7f7b2bfff640 1 -- 172.21.15.70:0/3964723748 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f7b380d2bf0 con 0x559670410140 2024-06-04T05:18:33.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.983+0000 7f7b55b9e740 10 client.20910 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:33.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.983+0000 7f7b55b9e740 1 -- 172.21.15.70:0/3964723748 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x559670429190 con 0x559670410140 2024-06-04T05:18:33.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.984+0000 7f7b2bfff640 1 -- 172.21.15.70:0/3964723748 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7b380c9e40 con 0x559670410140 2024-06-04T05:18:33.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.984+0000 7f7b2bfff640 1 -- 172.21.15.70:0/3964723748 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f7b380c90a0 con 0x559670410140 2024-06-04T05:18:33.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.984+0000 7f7b55b9e740 10 client.20910 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:33.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.984+0000 7f7b55b9e740 10 client.20910 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:18:33.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.984+0000 7f7b55b9e740 20 client.20910 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:33.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.984+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/3964723748 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55967042a5a0 0x55967042c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:33.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.984+0000 7f7b55b9e740 4 client.20910 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:18:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.984+0000 7f7b55b9e740 1 -- 172.21.15.70:0/3964723748 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559670383b80 con 0x55967042a5a0 2024-06-04T05:18:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.984+0000 7f7b50b4b640 1 --2- 172.21.15.70:0/3964723748 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55967042a5a0 0x55967042c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.985+0000 7f7b2bfff640 10 client.20910 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.985+0000 7f7b50b4b640 1 --2- 172.21.15.70:0/3964723748 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55967042a5a0 0x55967042c990 crc :-1 s=READY pgs=451 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbfa5d8d1441937a server_cookie=40417959940728e1 in_seq=0 out_seq=0 2024-06-04T05:18:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.985+0000 7f7b2bfff640 1 -- 172.21.15.70:0/3964723748 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559670383b80 con 0x55967042a5a0 2024-06-04T05:18:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.986+0000 7f7b2bfff640 10 client.20910 handle_command_reply: tid=0 2024-06-04T05:18:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.986+0000 7f7b55b9e740 1 client.20910 shutdown 2024-06-04T05:18:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.986+0000 7f7b55b9e740 1 -- 172.21.15.70:0/3964723748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559670410140 msgr2=0x559670417fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.986+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/3964723748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559670410140 0x559670417fb0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f7b380c5ec0 tx=0x7f7b38100510 comp rx=0 tx=0).stop 2024-06-04T05:18:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.986+0000 7f7b55b9e740 1 -- 172.21.15.70:0/3964723748 shutdown_connections 2024-06-04T05:18:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.986+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/3964723748 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55967042a5a0 0x55967042c990 crc :-1 s=READY pgs=451 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.986+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/3964723748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967041aa00 0x5596704184f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.986+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/3964723748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559670410140 0x559670417fb0 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.986+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/3964723748 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55967040f820 0x559670417a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:33.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.986+0000 7f7b55b9e740 1 -- 172.21.15.70:0/3964723748 >> 172.21.15.70:0/3964723748 conn(0x55967040bd30 msgr2=0x55967040afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:33.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.986+0000 7f7b2bfff640 0 client.20910 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:33.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.986+0000 7f7b55b9e740 1 -- 172.21.15.70:0/3964723748 shutdown_connections 2024-06-04T05:18:33.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.986+0000 7f7b55b9e740 1 -- 172.21.15.70:0/3964723748 wait complete. 2024-06-04T05:18:33.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:33.986+0000 7f7b55b9e740 20 client.20910 trim_cache size 0 max 16384 2024-06-04T05:18:33.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:33 smithi098 ceph-mon[35569]: pgmap v672: 97 pgs: 97 active+clean; 872 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.6 MiB/s wr, 452 op/s 2024-06-04T05:18:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.010+0000 7f7b55b9e740 1 Processor -- start 2024-06-04T05:18:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.010+0000 7f7b55b9e740 1 -- start start 2024-06-04T05:18:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.011+0000 7f7b55b9e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967040f820 0x55967040b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.011+0000 7f7b55b9e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559670410140 0x559670418310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.011+0000 7f7b55b9e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55967041aa00 0x5596704321f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.011+0000 7f7b4bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559670410140 0x559670418310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.011+0000 7f7b4bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559670410140 0x559670418310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58208/0 (socket says 172.21.15.70:58208) 2024-06-04T05:18:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.011+0000 7f7b4bfff640 1 -- 172.21.15.70:0/952119546 learned_addr learned my addr 172.21.15.70:0/952119546 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.011+0000 7f7b55b9e740 1 -- 172.21.15.70:0/952119546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c1b08b0 con 0x559670410140 2024-06-04T05:18:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.011+0000 7f7b55b9e740 1 -- 172.21.15.70:0/952119546 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c1b2450 con 0x55967040f820 2024-06-04T05:18:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.011+0000 7f7b50b4b640 1 --2- 172.21.15.70:0/952119546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967040f820 0x55967040b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.011+0000 7f7b55b9e740 1 -- 172.21.15.70:0/952119546 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c1b0fe0 con 0x55967041aa00 2024-06-04T05:18:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.011+0000 7f7b5134c640 1 --2- 172.21.15.70:0/952119546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55967041aa00 0x5596704321f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.011+0000 7f7b4bfff640 1 -- 172.21.15.70:0/952119546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55967041aa00 msgr2=0x5596704321f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.011+0000 7f7b4bfff640 1 --2- 172.21.15.70:0/952119546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55967041aa00 0x5596704321f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.011+0000 7f7b4bfff640 1 -- 172.21.15.70:0/952119546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967040f820 msgr2=0x55967040b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.011+0000 7f7b4bfff640 1 --2- 172.21.15.70:0/952119546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967040f820 0x55967040b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.011+0000 7f7b4bfff640 1 -- 172.21.15.70:0/952119546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559670416a20 con 0x559670410140 2024-06-04T05:18:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.012+0000 7f7b4bfff640 1 --2- 172.21.15.70:0/952119546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559670410140 0x559670418310 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f7b380c9410 tx=0x7f7b38003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f4a18ae96397ef server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.012+0000 7f7b367fc640 1 -- 172.21.15.70:0/952119546 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38049070 con 0x559670410140 2024-06-04T05:18:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.012+0000 7f7b367fc640 1 -- 172.21.15.70:0/952119546 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7b38031400 con 0x559670410140 2024-06-04T05:18:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.012+0000 7f7b367fc640 1 -- 172.21.15.70:0/952119546 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b380d2760 con 0x559670410140 2024-06-04T05:18:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.013+0000 7f7b55b9e740 1 -- 172.21.15.70:0/952119546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559670410140 msgr2=0x559670418310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.013+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/952119546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559670410140 0x559670418310 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f7b380c9410 tx=0x7f7b38003f90 comp rx=0 tx=0).stop 2024-06-04T05:18:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.013+0000 7f7b55b9e740 1 -- 172.21.15.70:0/952119546 shutdown_connections 2024-06-04T05:18:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.013+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/952119546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55967041aa00 0x5596704321f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.013+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/952119546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559670410140 0x559670418310 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.013+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/952119546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967040f820 0x55967040b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.013+0000 7f7b55b9e740 1 -- 172.21.15.70:0/952119546 >> 172.21.15.70:0/952119546 conn(0x55967040bd30 msgr2=0x559670425c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.013+0000 7f7b55b9e740 1 -- 172.21.15.70:0/952119546 shutdown_connections 2024-06-04T05:18:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.013+0000 7f7b55b9e740 1 -- 172.21.15.70:0/952119546 wait complete. 2024-06-04T05:18:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.014+0000 7f7b55b9e740 1 -- start start 2024-06-04T05:18:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.014+0000 7f7b55b9e740 1 Processor -- start 2024-06-04T05:18:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.014+0000 7f7b55b9e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55967040f820 0x55967041dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.014+0000 7f7b55b9e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559670410140 0x55967041e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.014+0000 7f7b55b9e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967041aa00 0x5596704930b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.014+0000 7f7b55b9e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c1b1d80 con 0x55967040f820 2024-06-04T05:18:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.014+0000 7f7b55b9e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c1b16b0 con 0x55967041aa00 2024-06-04T05:18:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.014+0000 7f7b55b9e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c1ae8b0 con 0x559670410140 2024-06-04T05:18:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.014+0000 7f7b55b9e740 10 client.0 _ll_register_callbacks cb 0x7f7b400097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.014+0000 7f7b50b4b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55967040f820 0x55967041dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.014+0000 7f7b4bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559670410140 0x55967041e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.014+0000 7f7b4bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559670410140 0x55967041e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39132/0 (socket says 172.21.15.70:39132) 2024-06-04T05:18:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.014+0000 7f7b4bfff640 1 -- 172.21.15.70:0/1744418405 learned_addr learned my addr 172.21.15.70:0/1744418405 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.014+0000 7f7b5134c640 1 --2- 172.21.15.70:0/1744418405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967041aa00 0x5596704930b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.015+0000 7f7b4bfff640 1 -- 172.21.15.70:0/1744418405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967041aa00 msgr2=0x5596704930b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.015+0000 7f7b4bfff640 1 --2- 172.21.15.70:0/1744418405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967041aa00 0x5596704930b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.015+0000 7f7b4bfff640 1 -- 172.21.15.70:0/1744418405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55967040f820 msgr2=0x55967041dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.015+0000 7f7b4bfff640 1 --2- 172.21.15.70:0/1744418405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55967040f820 0x55967041dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.015+0000 7f7b4bfff640 1 -- 172.21.15.70:0/1744418405 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559670493670 con 0x559670410140 2024-06-04T05:18:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.015+0000 7f7b4bfff640 1 --2- 172.21.15.70:0/1744418405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559670410140 0x55967041e140 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7b380c5ec0 tx=0x7f7b380d2990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85fb91a4f641ddac server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.015+0000 7f7b2bfff640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:18:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.015+0000 7f7b2bfff640 1 -- 172.21.15.70:0/1744418405 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38049070 con 0x559670410140 2024-06-04T05:18:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.015+0000 7f7b2bfff640 1 -- 172.21.15.70:0/1744418405 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f7b380c6430 con 0x559670410140 2024-06-04T05:18:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.015+0000 7f7b55b9e740 1 -- 172.21.15.70:0/1744418405 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55967037d360 con 0x559670410140 2024-06-04T05:18:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.015+0000 7f7b2bfff640 1 -- 172.21.15.70:0/1744418405 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f7b380ce140 con 0x559670410140 2024-06-04T05:18:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.015+0000 7f7b55b9e740 10 client.29776 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.015+0000 7f7b55b9e740 1 -- 172.21.15.70:0/1744418405 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x559670493670 con 0x559670410140 2024-06-04T05:18:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.016+0000 7f7b2bfff640 1 -- 172.21.15.70:0/1744418405 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55967037d360 con 0x559670410140 2024-06-04T05:18:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.016+0000 7f7b2bfff640 1 -- 172.21.15.70:0/1744418405 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f7b380ca0a0 con 0x559670410140 2024-06-04T05:18:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.016+0000 7f7b55b9e740 10 client.29776 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.016+0000 7f7b55b9e740 10 client.29776 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:18:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.016+0000 7f7b55b9e740 20 client.29776 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.016+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/1744418405 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559670494c30 0x559670497050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.016+0000 7f7b55b9e740 4 client.29776 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:18:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.016+0000 7f7b55b9e740 1 -- 172.21.15.70:0/1744418405 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5596702f75f0 con 0x559670494c30 2024-06-04T05:18:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.016+0000 7f7b50b4b640 1 --2- 172.21.15.70:0/1744418405 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559670494c30 0x559670497050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.017+0000 7f7b2bfff640 10 client.29776 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.017+0000 7f7b50b4b640 1 --2- 172.21.15.70:0/1744418405 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559670494c30 0x559670497050 crc :-1 s=READY pgs=452 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4efc18b355845601 server_cookie=35ff27f32d33e193 in_seq=0 out_seq=0 2024-06-04T05:18:34.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:33 smithi070 ceph-mon[32631]: pgmap v672: 97 pgs: 97 active+clean; 872 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.6 MiB/s wr, 452 op/s 2024-06-04T05:18:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.122+0000 7f7b2bfff640 1 -- 172.21.15.70:0/1744418405 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5596702f75f0 con 0x559670494c30 2024-06-04T05:18:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.122+0000 7f7b2bfff640 10 client.29776 handle_command_reply: tid=0 2024-06-04T05:18:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.122+0000 7f7b55b9e740 1 client.29776 shutdown 2024-06-04T05:18:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.122+0000 7f7b55b9e740 1 -- 172.21.15.70:0/1744418405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559670410140 msgr2=0x55967041e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.122+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/1744418405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559670410140 0x55967041e140 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7b380c5ec0 tx=0x7f7b380d2990 comp rx=0 tx=0).stop 2024-06-04T05:18:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.122+0000 7f7b55b9e740 1 -- 172.21.15.70:0/1744418405 shutdown_connections 2024-06-04T05:18:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.122+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/1744418405 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559670494c30 0x559670497050 crc :-1 s=READY pgs=452 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:34.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.122+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/1744418405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55967041aa00 0x5596704930b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:34.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.122+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/1744418405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559670410140 0x55967041e140 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:34.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.122+0000 7f7b55b9e740 1 --2- 172.21.15.70:0/1744418405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55967040f820 0x55967041dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:34.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.123+0000 7f7b55b9e740 1 -- 172.21.15.70:0/1744418405 >> 172.21.15.70:0/1744418405 conn(0x55967040bd30 msgr2=0x5596704111d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:34.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.123+0000 7f7b2bfff640 0 client.29776 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:34.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.123+0000 7f7b55b9e740 1 -- 172.21.15.70:0/1744418405 shutdown_connections 2024-06-04T05:18:34.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.123+0000 7f7b55b9e740 1 -- 172.21.15.70:0/1744418405 wait complete. 2024-06-04T05:18:34.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.123+0000 7f7b55b9e740 20 client.29776 trim_cache size 0 max 16384 2024-06-04T05:18:34.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:18:34.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:18:34.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:18:34.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.123+0000 7f7b52daf640 1 -- 172.21.15.70:0/2117013107 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7b3c0b2a10 msgr2=0x7f7b3c0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:34.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.123+0000 7f7b52daf640 1 --2- 172.21.15.70:0/2117013107 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7b3c0b2a10 0x7f7b3c0b4ed0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7b44009aa0 tx=0x7f7b44009c40 comp rx=0 tx=0).stop 2024-06-04T05:18:34.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.123+0000 7f7b52daf640 1 -- 172.21.15.70:0/2117013107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b4c102560 msgr2=0x7f7b4c19b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:34.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.123+0000 7f7b52daf640 1 --2- 172.21.15.70:0/2117013107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b4c102560 0x7f7b4c19b7e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7b3802a930 tx=0x7f7b38003b90 comp rx=0 tx=0).stop 2024-06-04T05:18:34.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.124+0000 7f7b52daf640 1 -- 172.21.15.70:0/2117013107 shutdown_connections 2024-06-04T05:18:34.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.124+0000 7f7b52daf640 1 --2- 172.21.15.70:0/2117013107 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7b3c0b2a10 0x7f7b3c0b4ed0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:34.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.124+0000 7f7b52daf640 1 --2- 172.21.15.70:0/2117013107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b4c102560 0x7f7b4c19b7e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:34.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.124+0000 7f7b52daf640 1 --2- 172.21.15.70:0/2117013107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b4c1079f0 0x7f7b4c19fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:34.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.124+0000 7f7b52daf640 1 --2- 172.21.15.70:0/2117013107 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b4c101bc0 0x7f7b4c19b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:34.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.124+0000 7f7b52daf640 1 -- 172.21.15.70:0/2117013107 >> 172.21.15.70:0/2117013107 conn(0x7f7b4c078830 msgr2=0x7f7b4c06acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:34.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.124+0000 7f7b52daf640 1 -- 172.21.15.70:0/2117013107 shutdown_connections 2024-06-04T05:18:34.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:34.124+0000 7f7b52daf640 1 -- 172.21.15.70:0/2117013107 wait complete. 2024-06-04T05:18:34.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:33 smithi112 ceph-mon[36453]: pgmap v672: 97 pgs: 97 active+clean; 872 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.6 MiB/s wr, 452 op/s 2024-06-04T05:18:34.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 60522 7.29 MB/sec execute 326 sec latency 120.993 ms 2024-06-04T05:18:35.304 INFO:tasks.quiescer.fs.[cephfs]:Total duration (51.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-06-04T05:18:35.304 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.85 2024-06-04T05:18:35.305 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:18:35.655 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:18:35.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 60705 7.29 MB/sec execute 327 sec latency 56.793 ms 2024-06-04T05:18:35.981 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:35 smithi070 ceph-mon[32631]: pgmap v673: 97 pgs: 97 active+clean; 872 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 252 op/s 2024-06-04T05:18:35.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:35 smithi098 ceph-mon[35569]: pgmap v673: 97 pgs: 97 active+clean; 872 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 252 op/s 2024-06-04T05:18:36.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:35 smithi112 ceph-mon[36453]: pgmap v673: 97 pgs: 97 active+clean; 872 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 252 op/s 2024-06-04T05:18:36.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 60920 7.32 MB/sec execute 328 sec latency 65.802 ms 2024-06-04T05:18:37.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.620+0000 7fbd83dc7700 1 -- 172.21.15.70:0/389763423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd7c102e30 msgr2=0x7fbd7c10f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:37.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.620+0000 7fbd83dc7700 1 --2- 172.21.15.70:0/389763423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd7c102e30 0x7fbd7c10f320 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fbd7400b0d0 tx=0x7fbd7400b3e0 comp rx=0 tx=0).stop 2024-06-04T05:18:37.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.620+0000 7fbd83dc7700 1 -- 172.21.15.70:0/389763423 shutdown_connections 2024-06-04T05:18:37.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.620+0000 7fbd83dc7700 1 --2- 172.21.15.70:0/389763423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd7c102e30 0x7fbd7c10f320 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:37.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.620+0000 7fbd83dc7700 1 --2- 172.21.15.70:0/389763423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd7c108550 0x7fbd7c108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:37.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.620+0000 7fbd83dc7700 1 --2- 172.21.15.70:0/389763423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd7c102470 0x7fbd7c1028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:37.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.620+0000 7fbd83dc7700 1 -- 172.21.15.70:0/389763423 >> 172.21.15.70:0/389763423 conn(0x7fbd7c079480 msgr2=0x7fbd7c079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:37.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.620+0000 7fbd83dc7700 1 -- 172.21.15.70:0/389763423 shutdown_connections 2024-06-04T05:18:37.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.620+0000 7fbd83dc7700 1 -- 172.21.15.70:0/389763423 wait complete. 2024-06-04T05:18:37.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.621+0000 7fbd83dc7700 1 Processor -- start 2024-06-04T05:18:37.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.621+0000 7fbd83dc7700 1 -- start start 2024-06-04T05:18:37.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.621+0000 7fbd83dc7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd7c102470 0x7fbd7c1a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:37.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.621+0000 7fbd83dc7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd7c102e30 0x7fbd7c1a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.622+0000 7fbd83dc7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd7c108550 0x7fbd7c19bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.622+0000 7fbd83dc7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7c1124f0 con 0x7fbd7c102e30 2024-06-04T05:18:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.622+0000 7fbd83dc7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7c112370 con 0x7fbd7c102470 2024-06-04T05:18:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.622+0000 7fbd83dc7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7c112670 con 0x7fbd7c108550 2024-06-04T05:18:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.622+0000 7fbd81b63700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd7c102470 0x7fbd7c1a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.622+0000 7fbd81362700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd7c102e30 0x7fbd7c1a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.622+0000 7fbd81b63700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd7c102470 0x7fbd7c1a1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:50918/0 (socket says 172.21.15.70:50918) 2024-06-04T05:18:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.622+0000 7fbd81b63700 1 -- 172.21.15.70:0/4275328103 learned_addr learned my addr 172.21.15.70:0/4275328103 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.622+0000 7fbd82364700 1 --2- 172.21.15.70:0/4275328103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd7c108550 0x7fbd7c19bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.622+0000 7fbd82364700 1 -- 172.21.15.70:0/4275328103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd7c102470 msgr2=0x7fbd7c1a1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.622+0000 7fbd82364700 1 --2- 172.21.15.70:0/4275328103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd7c102470 0x7fbd7c1a1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:37.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.622+0000 7fbd82364700 1 -- 172.21.15.70:0/4275328103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd7c102e30 msgr2=0x7fbd7c1a1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.622+0000 7fbd82364700 1 --2- 172.21.15.70:0/4275328103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd7c102e30 0x7fbd7c1a1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.622+0000 7fbd82364700 1 -- 172.21.15.70:0/4275328103 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd7c19c400 con 0x7fbd7c108550 2024-06-04T05:18:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.622+0000 7fbd81b63700 1 --2- 172.21.15.70:0/4275328103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd7c102470 0x7fbd7c1a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.622+0000 7fbd82364700 1 --2- 172.21.15.70:0/4275328103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd7c108550 0x7fbd7c19bb20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbd7400b0d0 tx=0x7fbd74007a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.623+0000 7fbd72ffd700 1 -- 172.21.15.70:0/4275328103 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7401f070 con 0x7fbd7c108550 2024-06-04T05:18:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.623+0000 7fbd83dc7700 1 -- 172.21.15.70:0/4275328103 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd7c19c690 con 0x7fbd7c108550 2024-06-04T05:18:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.623+0000 7fbd83dc7700 1 -- 172.21.15.70:0/4275328103 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd7c1a89d0 con 0x7fbd7c108550 2024-06-04T05:18:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.623+0000 7fbd72ffd700 1 -- 172.21.15.70:0/4275328103 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd74004c50 con 0x7fbd7c108550 2024-06-04T05:18:37.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.623+0000 7fbd72ffd700 1 -- 172.21.15.70:0/4275328103 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd74005870 con 0x7fbd7c108550 2024-06-04T05:18:37.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.624+0000 7fbd83dc7700 1 -- 172.21.15.70:0/4275328103 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd60005330 con 0x7fbd7c108550 2024-06-04T05:18:37.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.626+0000 7fbd72ffd700 1 -- 172.21.15.70:0/4275328103 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fbd74022070 con 0x7fbd7c108550 2024-06-04T05:18:37.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.626+0000 7fbd72ffd700 1 --2- 172.21.15.70:0/4275328103 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd680b29f0 0x7fbd680b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:37.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.627+0000 7fbd81b63700 1 --2- 172.21.15.70:0/4275328103 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd680b29f0 0x7fbd680b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:37.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.627+0000 7fbd72ffd700 1 -- 172.21.15.70:0/4275328103 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7fbd7400f020 con 0x7fbd7c108550 2024-06-04T05:18:37.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.627+0000 7fbd81b63700 1 --2- 172.21.15.70:0/4275328103 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd680b29f0 0x7fbd680b4eb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbd78011550 tx=0x7fbd78009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:37.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:37.632+0000 7fbd72ffd700 1 -- 172.21.15.70:0/4275328103 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbd74029020 con 0x7fbd7c108550 2024-06-04T05:18:37.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61057 7.31 MB/sec execute 329 sec latency 98.098 ms 2024-06-04T05:18:37.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:37 smithi098 ceph-mon[35569]: pgmap v674: 97 pgs: 97 active+clean; 891 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.4 MiB/s wr, 407 op/s 2024-06-04T05:18:38.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:38.058+0000 7fbd83dc7700 1 -- 172.21.15.70:0/4275328103 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd60005620 con 0x7fbd7c108550 2024-06-04T05:18:38.061 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:37 smithi070 ceph-mon[32631]: pgmap v674: 97 pgs: 97 active+clean; 891 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.4 MiB/s wr, 407 op/s 2024-06-04T05:18:38.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:38.059+0000 7fbd72ffd700 1 -- 172.21.15.70:0/4275328103 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+11932 (secure 0 0 0) 0x7fbd740ab2e0 con 0x7fbd7c108550 2024-06-04T05:18:38.062 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:18:38.062 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":59,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:17:20.010341+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:18:38.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:38.063+0000 7fbd83dc7700 1 -- 172.21.15.70:0/4275328103 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd680b29f0 msgr2=0x7fbd680b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:38.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:38.063+0000 7fbd83dc7700 1 --2- 172.21.15.70:0/4275328103 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd680b29f0 0x7fbd680b4eb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbd78011550 tx=0x7fbd78009260 comp rx=0 tx=0).stop 2024-06-04T05:18:38.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:38.063+0000 7fbd83dc7700 1 -- 172.21.15.70:0/4275328103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd7c108550 msgr2=0x7fbd7c19bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:38.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:38.063+0000 7fbd83dc7700 1 --2- 172.21.15.70:0/4275328103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd7c108550 0x7fbd7c19bb20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbd7400b0d0 tx=0x7fbd74007a00 comp rx=0 tx=0).stop 2024-06-04T05:18:38.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:38.063+0000 7fbd83dc7700 1 -- 172.21.15.70:0/4275328103 shutdown_connections 2024-06-04T05:18:38.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:38.063+0000 7fbd83dc7700 1 --2- 172.21.15.70:0/4275328103 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd680b29f0 0x7fbd680b4eb0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:38.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:38.063+0000 7fbd83dc7700 1 --2- 172.21.15.70:0/4275328103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd7c108550 0x7fbd7c19bb20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:38.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:38.063+0000 7fbd83dc7700 1 --2- 172.21.15.70:0/4275328103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd7c102470 0x7fbd7c1a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:38.063+0000 7fbd83dc7700 1 --2- 172.21.15.70:0/4275328103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd7c102e30 0x7fbd7c1a1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:38.063+0000 7fbd83dc7700 1 -- 172.21.15.70:0/4275328103 >> 172.21.15.70:0/4275328103 conn(0x7fbd7c079480 msgr2=0x7fbd7c100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:38.063+0000 7fbd83dc7700 1 -- 172.21.15.70:0/4275328103 shutdown_connections 2024-06-04T05:18:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:38.063+0000 7fbd83dc7700 1 -- 172.21.15.70:0/4275328103 wait complete. 2024-06-04T05:18:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 59 2024-06-04T05:18:38.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:37 smithi112 ceph-mon[36453]: pgmap v674: 97 pgs: 97 active+clean; 891 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.4 MiB/s wr, 407 op/s 2024-06-04T05:18:38.806 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 145.85 --await-for 45.0' 2024-06-04T05:18:38.807 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 145.85 --await-for 45.0 2024-06-04T05:18:38.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61194 7.31 MB/sec execute 330 sec latency 64.611 ms 2024-06-04T05:18:38.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:38 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/4275328103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:18:39.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:38 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/4275328103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:18:39.146 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:18:39.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:38 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/4275328103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:18:39.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61394 7.33 MB/sec execute 331 sec latency 90.319 ms 2024-06-04T05:18:39.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:39 smithi070 ceph-mon[32631]: pgmap v675: 97 pgs: 97 active+clean; 891 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.2 MiB/s wr, 266 op/s 2024-06-04T05:18:39.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:39 smithi098 ceph-mon[35569]: pgmap v675: 97 pgs: 97 active+clean; 891 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.2 MiB/s wr, 266 op/s 2024-06-04T05:18:40.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:39 smithi112 ceph-mon[36453]: pgmap v675: 97 pgs: 97 active+clean; 891 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.2 MiB/s wr, 266 op/s 2024-06-04T05:18:40.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61558 7.33 MB/sec execute 332 sec latency 62.730 ms 2024-06-04T05:18:41.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.433+0000 7f45cd39c700 1 -- 172.21.15.70:0/4153038859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45c81073d0 msgr2=0x7f45c81077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:41.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.433+0000 7f45cd39c700 1 --2- 172.21.15.70:0/4153038859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45c81073d0 0x7f45c81077b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f45b80098f0 tx=0x7f45b8009c00 comp rx=0 tx=0).stop 2024-06-04T05:18:41.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.434+0000 7f45cd39c700 1 -- 172.21.15.70:0/4153038859 shutdown_connections 2024-06-04T05:18:41.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.434+0000 7f45cd39c700 1 --2- 172.21.15.70:0/4153038859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c8102e30 0x7f45c810f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.434+0000 7f45cd39c700 1 --2- 172.21.15.70:0/4153038859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45c81073d0 0x7f45c81077b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.434+0000 7f45cd39c700 1 --2- 172.21.15.70:0/4153038859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45c8102470 0x7f45c81028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.434+0000 7f45cd39c700 1 -- 172.21.15.70:0/4153038859 >> 172.21.15.70:0/4153038859 conn(0x7f45c8079480 msgr2=0x7f45c8079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:41.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.437+0000 7f45cd39c700 1 -- 172.21.15.70:0/4153038859 shutdown_connections 2024-06-04T05:18:41.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.437+0000 7f45cd39c700 1 -- 172.21.15.70:0/4153038859 wait complete. 2024-06-04T05:18:41.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.438+0000 7f45cd39c700 1 Processor -- start 2024-06-04T05:18:41.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.438+0000 7f45cd39c700 1 -- start start 2024-06-04T05:18:41.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.438+0000 7f45cd39c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45c8102470 0x7f45c806fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.438+0000 7f45cd39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c8102e30 0x7f45c8070040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.439+0000 7f45cd39c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45c81073d0 0x7f45c81bcc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.439+0000 7f45cd39c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f45c806e520 con 0x7f45c8102470 2024-06-04T05:18:41.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.439+0000 7f45cd39c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45c806e3a0 con 0x7f45c8102e30 2024-06-04T05:18:41.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.439+0000 7f45cd39c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f45c806e6a0 con 0x7f45c81073d0 2024-06-04T05:18:41.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.439+0000 7f45c6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45c8102470 0x7f45c806fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:41.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.439+0000 7f45c6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45c8102470 0x7f45c806fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42538/0 (socket says 172.21.15.70:42538) 2024-06-04T05:18:41.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.439+0000 7f45c6ffd700 1 -- 172.21.15.70:0/2310529853 learned_addr learned my addr 172.21.15.70:0/2310529853 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:41.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.439+0000 7f45c67fc700 1 --2- 172.21.15.70:0/2310529853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c8102e30 0x7f45c8070040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:41.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.440+0000 7f45c77fe700 1 --2- 172.21.15.70:0/2310529853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45c81073d0 0x7f45c81bcc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:41.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.440+0000 7f45c6ffd700 1 -- 172.21.15.70:0/2310529853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45c81073d0 msgr2=0x7f45c81bcc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:41.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.440+0000 7f45c6ffd700 1 --2- 172.21.15.70:0/2310529853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45c81073d0 0x7f45c81bcc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.440+0000 7f45c6ffd700 1 -- 172.21.15.70:0/2310529853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c8102e30 msgr2=0x7f45c8070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:41.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.440+0000 7f45c6ffd700 1 --2- 172.21.15.70:0/2310529853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c8102e30 0x7f45c8070040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.440+0000 7f45c6ffd700 1 -- 172.21.15.70:0/2310529853 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c81bd310 con 0x7f45c8102470 2024-06-04T05:18:41.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.441+0000 7f45c6ffd700 1 --2- 172.21.15.70:0/2310529853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45c8102470 0x7f45c806fb00 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f45b8005c50 tx=0x7f45b8005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:41.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.441+0000 7f45affff700 1 -- 172.21.15.70:0/2310529853 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8003680 con 0x7f45c8102470 2024-06-04T05:18:41.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.441+0000 7f45affff700 1 -- 172.21.15.70:0/2310529853 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f45b8003cb0 con 0x7f45c8102470 2024-06-04T05:18:41.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.441+0000 7f45affff700 1 -- 172.21.15.70:0/2310529853 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8017ae0 con 0x7f45c8102470 2024-06-04T05:18:41.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.442+0000 7f45cd39c700 1 -- 172.21.15.70:0/2310529853 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45c81bd5a0 con 0x7f45c8102470 2024-06-04T05:18:41.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.442+0000 7f45cd39c700 1 -- 172.21.15.70:0/2310529853 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45c81bda30 con 0x7f45c8102470 2024-06-04T05:18:41.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.448+0000 7f45affff700 1 -- 172.21.15.70:0/2310529853 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f45b800bcf0 con 0x7f45c8102470 2024-06-04T05:18:41.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.449+0000 7f45d7c31b80 1 Processor -- start 2024-06-04T05:18:41.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.449+0000 7f45affff700 1 --2- 172.21.15.70:0/2310529853 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f45b00b2c30 0x7f45b00b50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.450+0000 7f45affff700 1 -- 172.21.15.70:0/2310529853 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f45b80e3660 con 0x7f45c8102470 2024-06-04T05:18:41.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.450+0000 7f45c77fe700 1 --2- 172.21.15.70:0/2310529853 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f45b00b2c30 0x7f45b00b50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:41.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.450+0000 7f45d7c31b80 1 -- start start 2024-06-04T05:18:41.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.450+0000 7f45d7c31b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707dd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.450+0000 7f45d7c31b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 0x556f707d8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.450+0000 7f45d7c31b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 0x556f707e4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.450+0000 7f45d7c31b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556f7072df10 con 0x556f707dcfd0 2024-06-04T05:18:41.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.450+0000 7f45d7c31b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556f7071dab0 con 0x556f707d84a0 2024-06-04T05:18:41.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.450+0000 7f45d7c31b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556f70721af0 con 0x556f707d8e60 2024-06-04T05:18:41.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.451+0000 7f45c6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707dd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:41.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.451+0000 7f45c77fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 0x556f707e4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:41.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.451+0000 7f45c6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707dd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42546/0 (socket says 172.21.15.70:42546) 2024-06-04T05:18:41.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.451+0000 7f45c6ffd700 1 -- 172.21.15.70:0/2333033205 learned_addr learned my addr 172.21.15.70:0/2333033205 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:41.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.451+0000 7f45c77fe700 1 --2- 172.21.15.70:0/2310529853 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f45b00b2c30 0x7f45b00b50f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f45c001ad40 tx=0x7f45c001a1d0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:41.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.451+0000 7f45c67fc700 1 --2- 172.21.15.70:0/2333033205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 0x556f707d8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:41.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.451+0000 7f45c77fe700 1 -- 172.21.15.70:0/2333033205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 msgr2=0x556f707d8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:41.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.451+0000 7f45c77fe700 1 --2- 172.21.15.70:0/2333033205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 0x556f707d8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.451+0000 7f45c77fe700 1 -- 172.21.15.70:0/2333033205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 msgr2=0x556f707dd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:41.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.451+0000 7f45c77fe700 1 --2- 172.21.15.70:0/2333033205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707dd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.451+0000 7f45c77fe700 1 -- 172.21.15.70:0/2333033205 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f706c7150 con 0x556f707d8e60 2024-06-04T05:18:41.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.451+0000 7f45c77fe700 1 --2- 172.21.15.70:0/2333033205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 0x556f707e4ed0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f45c00181d0 tx=0x7f45c0012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f847cdda83276a0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:41.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.451+0000 7f45c67fc700 1 --2- 172.21.15.70:0/2333033205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 0x556f707d8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:41.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.452+0000 7f45ad7fa700 1 -- 172.21.15.70:0/2333033205 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c0007b40 con 0x556f707d8e60 2024-06-04T05:18:41.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.452+0000 7f45ad7fa700 1 -- 172.21.15.70:0/2333033205 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f45c0007ce0 con 0x556f707d8e60 2024-06-04T05:18:41.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.452+0000 7f45ad7fa700 1 -- 172.21.15.70:0/2333033205 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c0023660 con 0x556f707d8e60 2024-06-04T05:18:41.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.452+0000 7f45d7c31b80 1 -- 172.21.15.70:0/2333033205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 msgr2=0x556f707e4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:41.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.452+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/2333033205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 0x556f707e4ed0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f45c00181d0 tx=0x7f45c0012000 comp rx=0 tx=0).stop 2024-06-04T05:18:41.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.453+0000 7f45d7c31b80 1 -- 172.21.15.70:0/2333033205 shutdown_connections 2024-06-04T05:18:41.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.453+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/2333033205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 0x556f707e4ed0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.453+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/2333033205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707dd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.453+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/2333033205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 0x556f707d8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.453+0000 7f45d7c31b80 1 -- 172.21.15.70:0/2333033205 >> 172.21.15.70:0/2333033205 conn(0x556f706fb3e0 msgr2=0x556f707ca980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:41.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.453+0000 7f45d7c31b80 1 -- 172.21.15.70:0/2333033205 shutdown_connections 2024-06-04T05:18:41.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.453+0000 7f45d7c31b80 1 -- 172.21.15.70:0/2333033205 wait complete. 2024-06-04T05:18:41.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.453+0000 7f45d7c31b80 1 -- start start 2024-06-04T05:18:41.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.453+0000 7f45d7c31b80 1 Processor -- start 2024-06-04T05:18:41.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45d7c31b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d84a0 0x556f707e4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45d7c31b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d8e60 0x556f707e1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45d7c31b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707e23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45d7c31b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556f70624140 con 0x556f707dcfd0 2024-06-04T05:18:41.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45d7c31b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556f70685f40 con 0x556f707d8e60 2024-06-04T05:18:41.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45d7c31b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556f707e54d0 con 0x556f707d84a0 2024-06-04T05:18:41.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45c6ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d84a0 0x556f707e4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:41.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45c67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d8e60 0x556f707e1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:41.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45c6ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d84a0 0x556f707e4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:58174/0 (socket says 172.21.15.70:58174) 2024-06-04T05:18:41.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45c6ffd700 1 -- 172.21.15.70:0/3064889749 learned_addr learned my addr 172.21.15.70:0/3064889749 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:41.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45d7c31b80 10 client.0 _ll_register_callbacks cb 0x556f706b8af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:41.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45c6ffd700 1 -- 172.21.15.70:0/3064889749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d8e60 msgr2=0x556f707e1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:41.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45c6ffd700 1 --2- 172.21.15.70:0/3064889749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d8e60 0x556f707e1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45c6ffd700 1 -- 172.21.15.70:0/3064889749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 msgr2=0x556f707e23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:41.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45c6ffd700 1 --2- 172.21.15.70:0/3064889749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707e23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45c6ffd700 1 -- 172.21.15.70:0/3064889749 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f7066da70 con 0x556f707d84a0 2024-06-04T05:18:41.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45c6ffd700 1 --2- 172.21.15.70:0/3064889749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d84a0 0x556f707e4b80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f45b80e0e30 tx=0x7f45b80e12b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eb0243de17891d4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:41.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.454+0000 7f45aaff5700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:18:41.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.455+0000 7f45aaff5700 1 -- 172.21.15.70:0/3064889749 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b80c4060 con 0x556f707d84a0 2024-06-04T05:18:41.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.455+0000 7f45d7c31b80 1 -- 172.21.15.70:0/3064889749 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45c0007ce0 con 0x556f707d84a0 2024-06-04T05:18:41.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.455+0000 7f45aaff5700 1 -- 172.21.15.70:0/3064889749 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f45b80b10d0 con 0x556f707d84a0 2024-06-04T05:18:41.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.455+0000 7f45aaff5700 1 -- 172.21.15.70:0/3064889749 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f45b80b30b0 con 0x556f707d84a0 2024-06-04T05:18:41.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.455+0000 7f45d7c31b80 10 client.29794 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:41.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.455+0000 7f45d7c31b80 1 -- 172.21.15.70:0/3064889749 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x556f706d7900 con 0x556f707d84a0 2024-06-04T05:18:41.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.455+0000 7f45aaff5700 1 -- 172.21.15.70:0/3064889749 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f45b80aa4b0 con 0x556f707d84a0 2024-06-04T05:18:41.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.456+0000 7f45aaff5700 1 -- 172.21.15.70:0/3064889749 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f45b800d340 con 0x556f707d84a0 2024-06-04T05:18:41.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.456+0000 7f45d7c31b80 10 client.29794 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:41.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.456+0000 7f45d7c31b80 10 client.29794 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:18:41.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.456+0000 7f45d7c31b80 20 client.29794 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:41.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.456+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/3064889749 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556f707f1df0 0x556f707f4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.456+0000 7f45d7c31b80 4 client.29794 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:18:41.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.456+0000 7f45d7c31b80 1 -- 172.21.15.70:0/3064889749 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f45c0007ce0 con 0x556f707f1df0 2024-06-04T05:18:41.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.456+0000 7f45c67fc700 1 --2- 172.21.15.70:0/3064889749 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556f707f1df0 0x556f707f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:41.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.457+0000 7f45aaff5700 10 client.29794 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:41.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.457+0000 7f45c67fc700 1 --2- 172.21.15.70:0/3064889749 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556f707f1df0 0x556f707f4230 crc :-1 s=READY pgs=453 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12d139d181114003 server_cookie=9cb57de8abe70ca4 in_seq=0 out_seq=0 2024-06-04T05:18:41.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.457+0000 7f45aaff5700 1 -- 172.21.15.70:0/3064889749 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f45c0007ce0 con 0x556f707f1df0 2024-06-04T05:18:41.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.457+0000 7f45aaff5700 10 client.29794 handle_command_reply: tid=0 2024-06-04T05:18:41.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.457+0000 7f45d7c31b80 1 client.29794 shutdown 2024-06-04T05:18:41.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.458+0000 7f45d7c31b80 1 -- 172.21.15.70:0/3064889749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d84a0 msgr2=0x556f707e4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:41.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.458+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/3064889749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d84a0 0x556f707e4b80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f45b80e0e30 tx=0x7f45b80e12b0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.458+0000 7f45d7c31b80 1 -- 172.21.15.70:0/3064889749 shutdown_connections 2024-06-04T05:18:41.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.458+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/3064889749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707e23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.458+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/3064889749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d84a0 0x556f707e4b80 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.458+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/3064889749 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556f707f1df0 0x556f707f4230 crc :-1 s=READY pgs=453 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.458+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/3064889749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d8e60 0x556f707e1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.458+0000 7f45aaff5700 0 client.29794 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:41.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.458+0000 7f45d7c31b80 1 -- 172.21.15.70:0/3064889749 >> 172.21.15.70:0/3064889749 conn(0x556f706fb3e0 msgr2=0x556f707d7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:41.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.458+0000 7f45d7c31b80 1 -- 172.21.15.70:0/3064889749 shutdown_connections 2024-06-04T05:18:41.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.458+0000 7f45d7c31b80 1 -- 172.21.15.70:0/3064889749 wait complete. 2024-06-04T05:18:41.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.458+0000 7f45d7c31b80 20 client.29794 trim_cache size 0 max 16384 2024-06-04T05:18:41.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.491+0000 7f45d7c31b80 1 Processor -- start 2024-06-04T05:18:41.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.491+0000 7f45d7c31b80 1 -- start start 2024-06-04T05:18:41.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.491+0000 7f45d7c31b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 0x556f707e0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.491+0000 7f45d7c31b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 0x556f707c4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.491+0000 7f45d7c31b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707f9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.491+0000 7f45d7c31b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f45c81c51f0 con 0x556f707dcfd0 2024-06-04T05:18:41.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.491+0000 7f45d7c31b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45c81c6d90 con 0x556f707d84a0 2024-06-04T05:18:41.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.491+0000 7f45d7c31b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f45c81c5920 con 0x556f707d8e60 2024-06-04T05:18:41.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.492+0000 7f45c6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 0x556f707e0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:41.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.492+0000 7f45c77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707f9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:41.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.492+0000 7f45c6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 0x556f707e0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:50972/0 (socket says 172.21.15.70:50972) 2024-06-04T05:18:41.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.492+0000 7f45c6ffd700 1 -- 172.21.15.70:0/4238576465 learned_addr learned my addr 172.21.15.70:0/4238576465 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:41.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.492+0000 7f45c67fc700 1 --2- 172.21.15.70:0/4238576465 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 0x556f707c4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:41.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.492+0000 7f45c6ffd700 1 -- 172.21.15.70:0/4238576465 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 msgr2=0x556f707c4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:41.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.492+0000 7f45c6ffd700 1 --2- 172.21.15.70:0/4238576465 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 0x556f707c4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.492+0000 7f45c6ffd700 1 -- 172.21.15.70:0/4238576465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 msgr2=0x556f707f9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:41.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.492+0000 7f45c6ffd700 1 --2- 172.21.15.70:0/4238576465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707f9b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.492+0000 7f45c6ffd700 1 -- 172.21.15.70:0/4238576465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f707cc490 con 0x556f707d84a0 2024-06-04T05:18:41.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.492+0000 7f45c77fe700 1 --2- 172.21.15.70:0/4238576465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707f9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:41.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.492+0000 7f45c6ffd700 1 --2- 172.21.15.70:0/4238576465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 0x556f707e0de0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f45b80a6a20 tx=0x7f45b8005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=947f7119a3b921bc server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:41.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.492+0000 7f45acff9700 1 -- 172.21.15.70:0/4238576465 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b80b37c0 con 0x556f707d84a0 2024-06-04T05:18:41.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.492+0000 7f45acff9700 1 -- 172.21.15.70:0/4238576465 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f45b80aa4b0 con 0x556f707d84a0 2024-06-04T05:18:41.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.492+0000 7f45acff9700 1 -- 172.21.15.70:0/4238576465 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b80b32e0 con 0x556f707d84a0 2024-06-04T05:18:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.493+0000 7f45d7c31b80 1 -- 172.21.15.70:0/4238576465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 msgr2=0x556f707e0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.493+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/4238576465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 0x556f707e0de0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f45b80a6a20 tx=0x7f45b8005750 comp rx=0 tx=0).stop 2024-06-04T05:18:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.493+0000 7f45d7c31b80 1 -- 172.21.15.70:0/4238576465 shutdown_connections 2024-06-04T05:18:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.493+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/4238576465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707f9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.493+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/4238576465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 0x556f707e0de0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.493+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/4238576465 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 0x556f707c4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.493+0000 7f45d7c31b80 1 -- 172.21.15.70:0/4238576465 >> 172.21.15.70:0/4238576465 conn(0x556f706fb3e0 msgr2=0x556f707de500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.494+0000 7f45d7c31b80 1 -- 172.21.15.70:0/4238576465 shutdown_connections 2024-06-04T05:18:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.494+0000 7f45d7c31b80 1 -- 172.21.15.70:0/4238576465 wait complete. 2024-06-04T05:18:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.494+0000 7f45d7c31b80 1 -- start start 2024-06-04T05:18:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.494+0000 7f45d7c31b80 1 Processor -- start 2024-06-04T05:18:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.494+0000 7f45d7c31b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 0x556f707eaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.494+0000 7f45d7c31b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 0x556f707e7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45d7c31b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707e7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45d7c31b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f45c81c66c0 con 0x556f707dcfd0 2024-06-04T05:18:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45d7c31b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45c81c5ff0 con 0x556f707d84a0 2024-06-04T05:18:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45d7c31b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f45c81c31b0 con 0x556f707d8e60 2024-06-04T05:18:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45d7c31b80 10 client.0 _ll_register_callbacks cb 0x556f706b8af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45c67fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 0x556f707e7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45c67fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 0x556f707e7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:58190/0 (socket says 172.21.15.70:58190) 2024-06-04T05:18:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45c67fc700 1 -- 172.21.15.70:0/2615192726 learned_addr learned my addr 172.21.15.70:0/2615192726 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45c6ffd700 1 --2- 172.21.15.70:0/2615192726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 0x556f707eaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45c77fe700 1 --2- 172.21.15.70:0/2615192726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707e7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45c67fc700 1 -- 172.21.15.70:0/2615192726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 msgr2=0x556f707eaf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45c67fc700 1 --2- 172.21.15.70:0/2615192726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 0x556f707eaf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45c67fc700 1 -- 172.21.15.70:0/2615192726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 msgr2=0x556f707e7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45c67fc700 1 --2- 172.21.15.70:0/2615192726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707e7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45c67fc700 1 -- 172.21.15.70:0/2615192726 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f707e8550 con 0x556f707d8e60 2024-06-04T05:18:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45c6ffd700 1 --2- 172.21.15.70:0/2615192726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 0x556f707eaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:18:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45c77fe700 1 --2- 172.21.15.70:0/2615192726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707e7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:18:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45c67fc700 1 --2- 172.21.15.70:0/2615192726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 0x556f707e7730 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f45c80710f0 tx=0x7f45bc0196d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=517e02f309f08d6e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.495+0000 7f45aaff5700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:18:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.496+0000 7f45aaff5700 1 -- 172.21.15.70:0/2615192726 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc023660 con 0x556f707d8e60 2024-06-04T05:18:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.496+0000 7f45aaff5700 1 -- 172.21.15.70:0/2615192726 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f45bc023800 con 0x556f707d8e60 2024-06-04T05:18:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.496+0000 7f45d7c31b80 1 -- 172.21.15.70:0/2615192726 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45b80aa4b0 con 0x556f707d8e60 2024-06-04T05:18:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.496+0000 7f45aaff5700 1 -- 172.21.15.70:0/2615192726 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f45bc019e40 con 0x556f707d8e60 2024-06-04T05:18:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.496+0000 7f45d7c31b80 10 client.29800 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.496+0000 7f45d7c31b80 1 -- 172.21.15.70:0/2615192726 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x556f7085db70 con 0x556f707d8e60 2024-06-04T05:18:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.496+0000 7f45aaff5700 1 -- 172.21.15.70:0/2615192726 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f45bc028070 con 0x556f707d8e60 2024-06-04T05:18:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.496+0000 7f45aaff5700 1 -- 172.21.15.70:0/2615192726 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f45bc0060f0 con 0x556f707d8e60 2024-06-04T05:18:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.496+0000 7f45d7c31b80 10 client.29800 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.496+0000 7f45d7c31b80 10 client.29800 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:18:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.496+0000 7f45d7c31b80 20 client.29800 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.496+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/2615192726 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556f7085e370 0x556f70860790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.497+0000 7f45d7c31b80 4 client.29800 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.85, "await_for": 45.0}] 2024-06-04T05:18:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.497+0000 7f45d7c31b80 1 -- 172.21.15.70:0/2615192726 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.85, "await_for": 45.0}) -- 0x7f45b80aa4b0 con 0x556f7085e370 2024-06-04T05:18:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.497+0000 7f45c6ffd700 1 --2- 172.21.15.70:0/2615192726 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556f7085e370 0x556f70860790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.497+0000 7f45aaff5700 10 client.29800 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.497+0000 7f45c6ffd700 1 --2- 172.21.15.70:0/2615192726 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556f7085e370 0x556f70860790 crc :-1 s=READY pgs=454 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abf609211bcd441e server_cookie=30eb2100f5dbc2c4 in_seq=0 out_seq=0 2024-06-04T05:18:41.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61658 7.32 MB/sec execute 333 sec latency 318.673 ms 2024-06-04T05:18:41.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.864+0000 7f45aaff5700 1 -- 172.21.15.70:0/2615192726 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f45b80aa4b0 con 0x556f7085e370 2024-06-04T05:18:41.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.865+0000 7f45aaff5700 10 client.29800 handle_command_reply: tid=0 2024-06-04T05:18:41.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.865+0000 7f45d7c31b80 1 client.29800 shutdown 2024-06-04T05:18:41.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.865+0000 7f45d7c31b80 1 -- 172.21.15.70:0/2615192726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 msgr2=0x556f707e7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:41.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.865+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/2615192726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 0x556f707e7730 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f45c80710f0 tx=0x7f45bc0196d0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.865+0000 7f45d7c31b80 1 -- 172.21.15.70:0/2615192726 shutdown_connections 2024-06-04T05:18:41.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.865+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/2615192726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f707dcfd0 0x556f707e7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.865+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/2615192726 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556f7085e370 0x556f70860790 crc :-1 s=READY pgs=454 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.865+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/2615192726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f707d84a0 0x556f707eaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.865+0000 7f45d7c31b80 1 --2- 172.21.15.70:0/2615192726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556f707d8e60 0x556f707e7730 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.865+0000 7f45d7c31b80 1 -- 172.21.15.70:0/2615192726 >> 172.21.15.70:0/2615192726 conn(0x556f706fb3e0 msgr2=0x556f707e1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:41.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.865+0000 7f45aaff5700 0 client.29800 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:41.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.865+0000 7f45d7c31b80 1 -- 172.21.15.70:0/2615192726 shutdown_connections 2024-06-04T05:18:41.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.865+0000 7f45d7c31b80 1 -- 172.21.15.70:0/2615192726 wait complete. 2024-06-04T05:18:41.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.865+0000 7f45d7c31b80 20 client.29800 trim_cache size 0 max 16384 2024-06-04T05:18:41.870 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:18:41.870 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 59, 2024-06-04T05:18:41.870 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:18:41.870 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 66, 2024-06-04T05:18:41.870 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:18:41.870 INFO:teuthology.orchestra.run.smithi070.stdout: "549ed795": { 2024-06-04T05:18:41.870 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 66, 2024-06-04T05:18:41.870 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:18:41.870 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:18:41.870 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:18:41.870 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:18:41.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:18:41.871 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:18:41.871 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 145.8, 2024-06-04T05:18:41.871 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:18:41.871 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:18:41.871 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:18:41.871 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:18:41.871 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:18:41.871 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:18:41.871 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:18:41.871 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:18:41.871 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:18:41.871 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:18:41.871 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:18:41.872 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:18:41.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.866+0000 7f45cd39c700 1 -- 172.21.15.70:0/2310529853 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f45b00b2c30 msgr2=0x7f45b00b50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:41.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.866+0000 7f45cd39c700 1 --2- 172.21.15.70:0/2310529853 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f45b00b2c30 0x7f45b00b50f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f45c001ad40 tx=0x7f45c001a1d0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.866+0000 7f45cd39c700 1 -- 172.21.15.70:0/2310529853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45c8102470 msgr2=0x7f45c806fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:41.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.866+0000 7f45cd39c700 1 --2- 172.21.15.70:0/2310529853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45c8102470 0x7f45c806fb00 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f45b8005c50 tx=0x7f45b8005270 comp rx=0 tx=0).stop 2024-06-04T05:18:41.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.866+0000 7f45cd39c700 1 -- 172.21.15.70:0/2310529853 shutdown_connections 2024-06-04T05:18:41.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.866+0000 7f45cd39c700 1 --2- 172.21.15.70:0/2310529853 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f45b00b2c30 0x7f45b00b50f0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.866+0000 7f45cd39c700 1 --2- 172.21.15.70:0/2310529853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45c81073d0 0x7f45c81bcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.866+0000 7f45cd39c700 1 --2- 172.21.15.70:0/2310529853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45c8102470 0x7f45c806fb00 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.866+0000 7f45cd39c700 1 --2- 172.21.15.70:0/2310529853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c8102e30 0x7f45c8070040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:41.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.866+0000 7f45cd39c700 1 -- 172.21.15.70:0/2310529853 >> 172.21.15.70:0/2310529853 conn(0x7f45c8079480 msgr2=0x7f45c8100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:41.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.866+0000 7f45cd39c700 1 -- 172.21.15.70:0/2310529853 shutdown_connections 2024-06-04T05:18:41.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:41.866+0000 7f45cd39c700 1 -- 172.21.15.70:0/2310529853 wait complete. 2024-06-04T05:18:41.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:41 smithi098 ceph-mon[35569]: pgmap v676: 97 pgs: 97 active+clean; 891 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.2 MiB/s wr, 266 op/s 2024-06-04T05:18:42.033 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:41 smithi070 ceph-mon[32631]: pgmap v676: 97 pgs: 97 active+clean; 891 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.2 MiB/s wr, 266 op/s 2024-06-04T05:18:42.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:41 smithi112 ceph-mon[36453]: pgmap v676: 97 pgs: 97 active+clean; 891 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.2 MiB/s wr, 266 op/s 2024-06-04T05:18:42.698 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '549ed795', quiesce took 7.4 seconds. Will release after: 18.450000000000003 2024-06-04T05:18:42.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 7.31 MB/sec execute 334 sec latency 1318.703 ms 2024-06-04T05:18:43.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 7.29 MB/sec execute 335 sec latency 2318.755 ms 2024-06-04T05:18:43.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:43 smithi070 ceph-mon[32631]: pgmap v677: 97 pgs: 97 active+clean; 899 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.9 MiB/s wr, 399 op/s 2024-06-04T05:18:43.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:43 smithi098 ceph-mon[35569]: pgmap v677: 97 pgs: 97 active+clean; 899 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.9 MiB/s wr, 399 op/s 2024-06-04T05:18:44.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:43 smithi112 ceph-mon[36453]: pgmap v677: 97 pgs: 97 active+clean; 899 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.9 MiB/s wr, 399 op/s 2024-06-04T05:18:44.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 7.27 MB/sec execute 336 sec latency 3318.830 ms 2024-06-04T05:18:45.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 7.25 MB/sec execute 337 sec latency 4318.949 ms 2024-06-04T05:18:45.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:45 smithi098 ceph-mon[35569]: pgmap v678: 97 pgs: 97 active+clean; 899 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.9 MiB/s wr, 287 op/s 2024-06-04T05:18:46.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:45 smithi070 ceph-mon[32631]: pgmap v678: 97 pgs: 97 active+clean; 899 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.9 MiB/s wr, 287 op/s 2024-06-04T05:18:46.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:45 smithi112 ceph-mon[36453]: pgmap v678: 97 pgs: 97 active+clean; 899 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.9 MiB/s wr, 287 op/s 2024-06-04T05:18:46.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 7.22 MB/sec execute 338 sec latency 5319.007 ms 2024-06-04T05:18:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.226+0000 7f3436990640 1 -- 172.21.15.70:0/3846697296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3430104600 msgr2=0x7f343010cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.226+0000 7f3436990640 1 --2- 172.21.15.70:0/3846697296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3430104600 0x7f343010cae0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f3424009900 tx=0x7f342402a510 comp rx=0 tx=0).stop 2024-06-04T05:18:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.226+0000 7f3436990640 1 -- 172.21.15.70:0/3846697296 shutdown_connections 2024-06-04T05:18:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.226+0000 7f3436990640 1 --2- 172.21.15.70:0/3846697296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f343010d020 0x7f343010f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.226+0000 7f3436990640 1 --2- 172.21.15.70:0/3846697296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3430104600 0x7f343010cae0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.226+0000 7f3436990640 1 --2- 172.21.15.70:0/3846697296 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3430103ce0 0x7f34301040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.226+0000 7f3436990640 1 -- 172.21.15.70:0/3846697296 >> 172.21.15.70:0/3846697296 conn(0x7f3430078ad0 msgr2=0x7f3430078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:47.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.227+0000 7f3436990640 1 -- 172.21.15.70:0/3846697296 shutdown_connections 2024-06-04T05:18:47.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.227+0000 7f3436990640 1 -- 172.21.15.70:0/3846697296 wait complete. 2024-06-04T05:18:47.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.227+0000 7f3436990640 1 Processor -- start 2024-06-04T05:18:47.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.227+0000 7f3436990640 1 -- start start 2024-06-04T05:18:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f3436990640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3430103ce0 0x7f34301990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f3436990640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3430104600 0x7f3430199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f3436990640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f343010d020 0x7f343019d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f3436990640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3430112b80 con 0x7f3430103ce0 2024-06-04T05:18:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f3436990640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3430112a00 con 0x7f3430104600 2024-06-04T05:18:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f3436990640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3430112d00 con 0x7f343010d020 2024-06-04T05:18:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f342ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3430103ce0 0x7f34301990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f342ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3430103ce0 0x7f34301990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58058/0 (socket says 172.21.15.70:58058) 2024-06-04T05:18:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f342ffff640 1 -- 172.21.15.70:0/2485872386 learned_addr learned my addr 172.21.15.70:0/2485872386 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f342f7fe640 1 --2- 172.21.15.70:0/2485872386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3430104600 0x7f3430199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f3434f2d640 1 --2- 172.21.15.70:0/2485872386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f343010d020 0x7f343019d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f342ffff640 1 -- 172.21.15.70:0/2485872386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f343010d020 msgr2=0x7f343019d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f342ffff640 1 --2- 172.21.15.70:0/2485872386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f343010d020 0x7f343019d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f342ffff640 1 -- 172.21.15.70:0/2485872386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3430104600 msgr2=0x7f3430199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f342ffff640 1 --2- 172.21.15.70:0/2485872386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3430104600 0x7f3430199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f342ffff640 1 -- 172.21.15.70:0/2485872386 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343019e0d0 con 0x7f3430103ce0 2024-06-04T05:18:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f342ffff640 1 --2- 172.21.15.70:0/2485872386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3430103ce0 0x7f34301990e0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f342000d770 tx=0x7f342000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f342d7fa640 1 -- 172.21.15.70:0/2485872386 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3420017070 con 0x7f3430103ce0 2024-06-04T05:18:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f342d7fa640 1 -- 172.21.15.70:0/2485872386 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f34200054b0 con 0x7f3430103ce0 2024-06-04T05:18:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f3436990640 1 -- 172.21.15.70:0/2485872386 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343019e3c0 con 0x7f3430103ce0 2024-06-04T05:18:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f342d7fa640 1 -- 172.21.15.70:0/2485872386 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3420004c60 con 0x7f3430103ce0 2024-06-04T05:18:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.228+0000 7f3436990640 1 -- 172.21.15.70:0/2485872386 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34301a5c50 con 0x7f3430103ce0 2024-06-04T05:18:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.231+0000 7f3439783740 1 Processor -- start 2024-06-04T05:18:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.231+0000 7f342d7fa640 1 -- 172.21.15.70:0/2485872386 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3420004e00 con 0x7f3430103ce0 2024-06-04T05:18:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.231+0000 7f3439783740 1 -- start start 2024-06-04T05:18:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.231+0000 7f3439783740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccaf740 0x55b8eccafb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.231+0000 7f3439783740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccb0060 0x55b8eccbe8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.231+0000 7f3439783740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccbee10 0x55b8eccc1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.231+0000 7f3439783740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b8ecc21c30 con 0x55b8eccb0060 2024-06-04T05:18:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.231+0000 7f3439783740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b8ecc117d0 con 0x55b8eccbee10 2024-06-04T05:18:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.231+0000 7f3439783740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b8ecc15810 con 0x55b8eccaf740 2024-06-04T05:18:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.231+0000 7f342ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccb0060 0x55b8eccbe8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.231+0000 7f3434f2d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccaf740 0x55b8eccafb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.231+0000 7f342ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccb0060 0x55b8eccbe8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58072/0 (socket says 172.21.15.70:58072) 2024-06-04T05:18:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.231+0000 7f3434f2d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccaf740 0x55b8eccafb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48338/0 (socket says 172.21.15.70:48338) 2024-06-04T05:18:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.231+0000 7f342ffff640 1 -- 172.21.15.70:0/1887558603 learned_addr learned my addr 172.21.15.70:0/1887558603 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:47.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.232+0000 7f342f7fe640 1 --2- 172.21.15.70:0/1887558603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccbee10 0x55b8eccc1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.232+0000 7f342ffff640 1 -- 172.21.15.70:0/1887558603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccaf740 msgr2=0x55b8eccafb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:47.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.232+0000 7f342ffff640 1 --2- 172.21.15.70:0/1887558603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccaf740 0x55b8eccafb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.232+0000 7f342d7fa640 1 --2- 172.21.15.70:0/2485872386 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f34180b2a10 0x7f34180b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.232+0000 7f342ffff640 1 -- 172.21.15.70:0/1887558603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccbee10 msgr2=0x55b8eccc1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:47.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.232+0000 7f342ffff640 1 --2- 172.21.15.70:0/1887558603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccbee10 0x55b8eccc1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.232+0000 7f342ffff640 1 -- 172.21.15.70:0/1887558603 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8ec926210 con 0x55b8eccb0060 2024-06-04T05:18:47.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.232+0000 7f3434f2d640 1 --2- 172.21.15.70:0/2485872386 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f34180b2a10 0x7f34180b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.232+0000 7f342ffff640 1 --2- 172.21.15.70:0/1887558603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccb0060 0x55b8eccbe8d0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f34200df7e0 tx=0x7f34200a5b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438a0a9fa957fc7b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:47.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.232+0000 7f342d7fa640 1 -- 172.21.15.70:0/2485872386 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f34200e2150 con 0x7f3430103ce0 2024-06-04T05:18:47.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.232+0000 7f34167fc640 1 -- 172.21.15.70:0/1887558603 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3420020070 con 0x55b8eccb0060 2024-06-04T05:18:47.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.232+0000 7f34167fc640 1 -- 172.21.15.70:0/1887558603 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f34200a8f10 con 0x55b8eccb0060 2024-06-04T05:18:47.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.232+0000 7f34167fc640 1 -- 172.21.15.70:0/1887558603 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34200a91d0 con 0x55b8eccb0060 2024-06-04T05:18:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.233+0000 7f3439783740 1 -- 172.21.15.70:0/1887558603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccb0060 msgr2=0x55b8eccbe8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.233+0000 7f3434f2d640 1 --2- 172.21.15.70:0/2485872386 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f34180b2a10 0x7f34180b4ed0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3428005cc0 tx=0x7f342800b580 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.233+0000 7f3439783740 1 --2- 172.21.15.70:0/1887558603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccb0060 0x55b8eccbe8d0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f34200df7e0 tx=0x7f34200a5b10 comp rx=0 tx=0).stop 2024-06-04T05:18:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.233+0000 7f3439783740 1 -- 172.21.15.70:0/1887558603 shutdown_connections 2024-06-04T05:18:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.233+0000 7f3439783740 1 --2- 172.21.15.70:0/1887558603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccbee10 0x55b8eccc1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.233+0000 7f3439783740 1 --2- 172.21.15.70:0/1887558603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccb0060 0x55b8eccbe8d0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.233+0000 7f3439783740 1 --2- 172.21.15.70:0/1887558603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccaf740 0x55b8eccafb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.233+0000 7f3439783740 1 -- 172.21.15.70:0/1887558603 >> 172.21.15.70:0/1887558603 conn(0x55b8eccabd30 msgr2=0x55b8eccac110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.233+0000 7f3439783740 1 -- 172.21.15.70:0/1887558603 shutdown_connections 2024-06-04T05:18:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.233+0000 7f3439783740 1 -- 172.21.15.70:0/1887558603 wait complete. 2024-06-04T05:18:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.234+0000 7f3439783740 1 -- start start 2024-06-04T05:18:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.234+0000 7f3439783740 1 Processor -- start 2024-06-04T05:18:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.234+0000 7f3439783740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccaf740 0x55b8eccbbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.234+0000 7f3439783740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccb0060 0x55b8eccbc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.234+0000 7f3439783740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccbee10 0x55b8eccbc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.234+0000 7f3439783740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b8ec8f7790 con 0x55b8eccaf740 2024-06-04T05:18:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.234+0000 7f3439783740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b8ecbcd890 con 0x55b8eccbee10 2024-06-04T05:18:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.234+0000 7f3439783740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b8eccc1980 con 0x55b8eccb0060 2024-06-04T05:18:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.234+0000 7f3439783740 10 client.0 _ll_register_callbacks cb 0x55b8ec925a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.234+0000 7f342ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccaf740 0x55b8eccbbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.234+0000 7f342ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccaf740 0x55b8eccbbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58074/0 (socket says 172.21.15.70:58074) 2024-06-04T05:18:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.234+0000 7f342ffff640 1 -- 172.21.15.70:0/150674479 learned_addr learned my addr 172.21.15.70:0/150674479 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.234+0000 7f342f7fe640 1 --2- 172.21.15.70:0/150674479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccb0060 0x55b8eccbc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f342ffff640 1 -- 172.21.15.70:0/150674479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccb0060 msgr2=0x55b8eccbc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f342ffff640 1 --2- 172.21.15.70:0/150674479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccb0060 0x55b8eccbc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f3434f2d640 1 --2- 172.21.15.70:0/150674479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccbee10 0x55b8eccbc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f342ffff640 1 -- 172.21.15.70:0/150674479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccbee10 msgr2=0x55b8eccbc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f342ffff640 1 --2- 172.21.15.70:0/150674479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccbee10 0x55b8eccbc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f342ffff640 1 -- 172.21.15.70:0/150674479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8ec9f2620 con 0x55b8eccaf740 2024-06-04T05:18:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f342ffff640 1 --2- 172.21.15.70:0/150674479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccaf740 0x55b8eccbbea0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f34200df920 tx=0x7f3420004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=978a567f984536e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f340f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:18:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f340f7fe640 1 -- 172.21.15.70:0/150674479 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3420020070 con 0x55b8eccaf740 2024-06-04T05:18:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f3439783740 1 -- 172.21.15.70:0/150674479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34200a8f10 con 0x55b8eccaf740 2024-06-04T05:18:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f340f7fe640 1 -- 172.21.15.70:0/150674479 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f34200a9660 con 0x55b8eccaf740 2024-06-04T05:18:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f340f7fe640 1 -- 172.21.15.70:0/150674479 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f34200a8b70 con 0x55b8eccaf740 2024-06-04T05:18:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f3439783740 10 client.20946 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f3439783740 1 -- 172.21.15.70:0/150674479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55b8ecccd7e0 con 0x55b8eccaf740 2024-06-04T05:18:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f340f7fe640 1 -- 172.21.15.70:0/150674479 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f34200a8f10 con 0x55b8eccaf740 2024-06-04T05:18:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f340f7fe640 1 -- 172.21.15.70:0/150674479 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f34200b1dd0 con 0x55b8eccaf740 2024-06-04T05:18:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f3439783740 10 client.20946 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f3439783740 10 client.20946 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:18:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.235+0000 7f3439783740 20 client.20946 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.236+0000 7f3439783740 1 --2- 172.21.15.70:0/150674479 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b8eccceb30 0x55b8eccd0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.236+0000 7f3439783740 4 client.20946 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:18:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.236+0000 7f3439783740 1 -- 172.21.15.70:0/150674479 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8ecc23b80 con 0x55b8eccceb30 2024-06-04T05:18:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.236+0000 7f342f7fe640 1 --2- 172.21.15.70:0/150674479 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b8eccceb30 0x55b8eccd0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.236+0000 7f342f7fe640 1 --2- 172.21.15.70:0/150674479 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b8eccceb30 0x55b8eccd0f20 crc :-1 s=READY pgs=455 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adbb148773980510 server_cookie=6ee0dda89e6f062f in_seq=0 out_seq=0 2024-06-04T05:18:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.236+0000 7f340f7fe640 10 client.20946 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:47.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.237+0000 7f340f7fe640 1 -- 172.21.15.70:0/150674479 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b8ecc23b80 con 0x55b8eccceb30 2024-06-04T05:18:47.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.237+0000 7f340f7fe640 10 client.20946 handle_command_reply: tid=0 2024-06-04T05:18:47.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.237+0000 7f3439783740 1 client.20946 shutdown 2024-06-04T05:18:47.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.237+0000 7f3439783740 1 -- 172.21.15.70:0/150674479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccaf740 msgr2=0x55b8eccbbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:47.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.237+0000 7f3439783740 1 --2- 172.21.15.70:0/150674479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccaf740 0x55b8eccbbea0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f34200df920 tx=0x7f3420004730 comp rx=0 tx=0).stop 2024-06-04T05:18:47.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.237+0000 7f3439783740 1 -- 172.21.15.70:0/150674479 shutdown_connections 2024-06-04T05:18:47.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.237+0000 7f3439783740 1 --2- 172.21.15.70:0/150674479 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b8eccceb30 0x55b8eccd0f20 crc :-1 s=READY pgs=455 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.237+0000 7f3439783740 1 --2- 172.21.15.70:0/150674479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccbee10 0x55b8eccbc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.237+0000 7f3439783740 1 --2- 172.21.15.70:0/150674479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccb0060 0x55b8eccbc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.237+0000 7f3439783740 1 --2- 172.21.15.70:0/150674479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccaf740 0x55b8eccbbea0 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.237+0000 7f340f7fe640 0 client.20946 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:47.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.237+0000 7f3439783740 1 -- 172.21.15.70:0/150674479 >> 172.21.15.70:0/150674479 conn(0x55b8eccabd30 msgr2=0x55b8eccb1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:47.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.237+0000 7f3439783740 1 -- 172.21.15.70:0/150674479 shutdown_connections 2024-06-04T05:18:47.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.237+0000 7f3439783740 1 -- 172.21.15.70:0/150674479 wait complete. 2024-06-04T05:18:47.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.237+0000 7f3439783740 20 client.20946 trim_cache size 0 max 16384 2024-06-04T05:18:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.252+0000 7f3439783740 1 Processor -- start 2024-06-04T05:18:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3439783740 1 -- start start 2024-06-04T05:18:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3439783740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccaf740 0x55b8eccb1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3439783740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccb0060 0x55b8eccbad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3439783740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccbee10 0x55b8eccd64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3439783740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f34301ac430 con 0x55b8eccbee10 2024-06-04T05:18:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3439783740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f34301adfd0 con 0x55b8eccb0060 2024-06-04T05:18:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3439783740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f34301acb60 con 0x55b8eccaf740 2024-06-04T05:18:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3434f2d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccbee10 0x55b8eccd64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3434f2d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccbee10 0x55b8eccd64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58086/0 (socket says 172.21.15.70:58086) 2024-06-04T05:18:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3434f2d640 1 -- 172.21.15.70:0/3115818108 learned_addr learned my addr 172.21.15.70:0/3115818108 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f342f7fe640 1 --2- 172.21.15.70:0/3115818108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccb0060 0x55b8eccbad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f342ffff640 1 --2- 172.21.15.70:0/3115818108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccaf740 0x55b8eccb1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3434f2d640 1 -- 172.21.15.70:0/3115818108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccaf740 msgr2=0x55b8eccb1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3434f2d640 1 --2- 172.21.15.70:0/3115818108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccaf740 0x55b8eccb1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3434f2d640 1 -- 172.21.15.70:0/3115818108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccb0060 msgr2=0x55b8eccbad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3434f2d640 1 --2- 172.21.15.70:0/3115818108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccb0060 0x55b8eccbad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3434f2d640 1 -- 172.21.15.70:0/3115818108 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8eccc1920 con 0x55b8eccbee10 2024-06-04T05:18:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f342ffff640 1 --2- 172.21.15.70:0/3115818108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccaf740 0x55b8eccb1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:18:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3434f2d640 1 --2- 172.21.15.70:0/3115818108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccbee10 0x55b8eccd64f0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f342801c4e0 tx=0x7f3428013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=714d71e161ab0c26 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3415ffb640 1 -- 172.21.15.70:0/3115818108 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342801d830 con 0x55b8eccbee10 2024-06-04T05:18:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.253+0000 7f3415ffb640 1 -- 172.21.15.70:0/3115818108 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f342801a410 con 0x55b8eccbee10 2024-06-04T05:18:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.254+0000 7f3415ffb640 1 -- 172.21.15.70:0/3115818108 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342801de60 con 0x55b8eccbee10 2024-06-04T05:18:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.254+0000 7f3439783740 1 -- 172.21.15.70:0/3115818108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccbee10 msgr2=0x55b8eccd64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.254+0000 7f3439783740 1 --2- 172.21.15.70:0/3115818108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccbee10 0x55b8eccd64f0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f342801c4e0 tx=0x7f3428013000 comp rx=0 tx=0).stop 2024-06-04T05:18:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.254+0000 7f3439783740 1 -- 172.21.15.70:0/3115818108 shutdown_connections 2024-06-04T05:18:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.254+0000 7f3439783740 1 --2- 172.21.15.70:0/3115818108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccb0060 0x55b8eccbad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.254+0000 7f3439783740 1 --2- 172.21.15.70:0/3115818108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccbee10 0x55b8eccd64f0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.254+0000 7f3439783740 1 --2- 172.21.15.70:0/3115818108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccaf740 0x55b8eccb1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.254+0000 7f3439783740 1 -- 172.21.15.70:0/3115818108 >> 172.21.15.70:0/3115818108 conn(0x55b8eccabd30 msgr2=0x55b8eccba660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.254+0000 7f3439783740 1 -- 172.21.15.70:0/3115818108 shutdown_connections 2024-06-04T05:18:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.254+0000 7f3439783740 1 -- 172.21.15.70:0/3115818108 wait complete. 2024-06-04T05:18:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.254+0000 7f3439783740 1 -- start start 2024-06-04T05:18:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f3439783740 1 Processor -- start 2024-06-04T05:18:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f3439783740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccaf740 0x55b8eccb8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f3439783740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccb0060 0x55b8eccb9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f3439783740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccbee10 0x55b8eccb9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f3439783740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f34301ad900 con 0x55b8eccaf740 2024-06-04T05:18:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f3439783740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f34301ad230 con 0x55b8eccbee10 2024-06-04T05:18:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f3439783740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f34301aa430 con 0x55b8eccb0060 2024-06-04T05:18:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f3439783740 10 client.0 _ll_register_callbacks cb 0x7f342005f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f342ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccaf740 0x55b8eccb8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f342ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccaf740 0x55b8eccb8fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58088/0 (socket says 172.21.15.70:58088) 2024-06-04T05:18:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f342ffff640 1 -- 172.21.15.70:0/2902573821 learned_addr learned my addr 172.21.15.70:0/2902573821 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f3434f2d640 1 --2- 172.21.15.70:0/2902573821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccbee10 0x55b8eccb9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f342f7fe640 1 --2- 172.21.15.70:0/2902573821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccb0060 0x55b8eccb9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f342ffff640 1 -- 172.21.15.70:0/2902573821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccb0060 msgr2=0x55b8eccb9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f342ffff640 1 --2- 172.21.15.70:0/2902573821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccb0060 0x55b8eccb9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f342ffff640 1 -- 172.21.15.70:0/2902573821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccbee10 msgr2=0x55b8eccb9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f342ffff640 1 --2- 172.21.15.70:0/2902573821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccbee10 0x55b8eccb9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.255+0000 7f342ffff640 1 -- 172.21.15.70:0/2902573821 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8ecd38470 con 0x55b8eccaf740 2024-06-04T05:18:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.256+0000 7f342ffff640 1 --2- 172.21.15.70:0/2902573821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccaf740 0x55b8eccb8fd0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f34200a9390 tx=0x7f34200a5b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b6d4be29b6edf78 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.256+0000 7f340f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:18:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.256+0000 7f340f7fe640 1 -- 172.21.15.70:0/2902573821 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3420020070 con 0x55b8eccaf740 2024-06-04T05:18:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.256+0000 7f340f7fe640 1 -- 172.21.15.70:0/2902573821 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f34200e5d30 con 0x55b8eccaf740 2024-06-04T05:18:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.256+0000 7f3439783740 1 -- 172.21.15.70:0/2902573821 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f342801a410 con 0x55b8eccaf740 2024-06-04T05:18:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.256+0000 7f340f7fe640 1 -- 172.21.15.70:0/2902573821 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f34200a8d50 con 0x55b8eccaf740 2024-06-04T05:18:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.256+0000 7f3439783740 10 client.20952 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.256+0000 7f3439783740 1 -- 172.21.15.70:0/2902573821 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55b8ecd39740 con 0x55b8eccaf740 2024-06-04T05:18:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.256+0000 7f340f7fe640 1 -- 172.21.15.70:0/2902573821 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f342801a410 con 0x55b8eccaf740 2024-06-04T05:18:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.256+0000 7f340f7fe640 1 -- 172.21.15.70:0/2902573821 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f34200a9660 con 0x55b8eccaf740 2024-06-04T05:18:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.256+0000 7f3439783740 10 client.20952 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.256+0000 7f3439783740 10 client.20952 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:18:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.256+0000 7f3439783740 20 client.20952 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.256+0000 7f3439783740 1 --2- 172.21.15.70:0/2902573821 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b8ecd39c50 0x55b8ecd3c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.256+0000 7f3439783740 4 client.20952 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:18:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.256+0000 7f3439783740 1 -- 172.21.15.70:0/2902573821 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b8ecc1d360 con 0x55b8ecd39c50 2024-06-04T05:18:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.257+0000 7f342f7fe640 1 --2- 172.21.15.70:0/2902573821 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b8ecd39c50 0x55b8ecd3c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.257+0000 7f342f7fe640 1 --2- 172.21.15.70:0/2902573821 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b8ecd39c50 0x55b8ecd3c070 crc :-1 s=READY pgs=456 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8603fb30c6ed3a6 server_cookie=ba8f8b6561d60865 in_seq=0 out_seq=0 2024-06-04T05:18:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.257+0000 7f340f7fe640 10 client.20952 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:47.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.331+0000 7f340f7fe640 1 -- 172.21.15.70:0/2902573821 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55b8ecc1d360 con 0x55b8ecd39c50 2024-06-04T05:18:47.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.331+0000 7f340f7fe640 10 client.20952 handle_command_reply: tid=0 2024-06-04T05:18:47.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.331+0000 7f3439783740 1 client.20952 shutdown 2024-06-04T05:18:47.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.331+0000 7f3439783740 1 -- 172.21.15.70:0/2902573821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccaf740 msgr2=0x55b8eccb8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:47.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.331+0000 7f3439783740 1 --2- 172.21.15.70:0/2902573821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccaf740 0x55b8eccb8fd0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f34200a9390 tx=0x7f34200a5b80 comp rx=0 tx=0).stop 2024-06-04T05:18:47.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.331+0000 7f3439783740 1 -- 172.21.15.70:0/2902573821 shutdown_connections 2024-06-04T05:18:47.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.331+0000 7f3439783740 1 --2- 172.21.15.70:0/2902573821 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55b8ecd39c50 0x55b8ecd3c070 crc :-1 s=READY pgs=456 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.331+0000 7f3439783740 1 --2- 172.21.15.70:0/2902573821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8eccbee10 0x55b8eccb9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.331+0000 7f340f7fe640 0 client.20952 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:47.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.331+0000 7f3439783740 1 --2- 172.21.15.70:0/2902573821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b8eccb0060 0x55b8eccb9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.331+0000 7f3439783740 1 --2- 172.21.15.70:0/2902573821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8eccaf740 0x55b8eccb8fd0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.331+0000 7f3439783740 1 -- 172.21.15.70:0/2902573821 >> 172.21.15.70:0/2902573821 conn(0x55b8eccabd30 msgr2=0x55b8eccbaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:47.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.331+0000 7f3439783740 1 -- 172.21.15.70:0/2902573821 shutdown_connections 2024-06-04T05:18:47.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.331+0000 7f3439783740 1 -- 172.21.15.70:0/2902573821 wait complete. 2024-06-04T05:18:47.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.331+0000 7f3439783740 20 client.20952 trim_cache size 0 max 16384 2024-06-04T05:18:47.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:18:47.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:18:47.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:18:47.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.332+0000 7f3436990640 1 -- 172.21.15.70:0/2485872386 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f34180b2a10 msgr2=0x7f34180b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:47.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.332+0000 7f3436990640 1 --2- 172.21.15.70:0/2485872386 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f34180b2a10 0x7f34180b4ed0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3428005cc0 tx=0x7f342800b580 comp rx=0 tx=0).stop 2024-06-04T05:18:47.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.332+0000 7f3436990640 1 -- 172.21.15.70:0/2485872386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3430103ce0 msgr2=0x7f34301990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:47.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.332+0000 7f3436990640 1 --2- 172.21.15.70:0/2485872386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3430103ce0 0x7f34301990e0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f342000d770 tx=0x7f342000dc30 comp rx=0 tx=0).stop 2024-06-04T05:18:47.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.332+0000 7f3436990640 1 -- 172.21.15.70:0/2485872386 shutdown_connections 2024-06-04T05:18:47.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.332+0000 7f3436990640 1 --2- 172.21.15.70:0/2485872386 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f34180b2a10 0x7f34180b4ed0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.332+0000 7f3436990640 1 --2- 172.21.15.70:0/2485872386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3430104600 0x7f3430199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.332+0000 7f3436990640 1 --2- 172.21.15.70:0/2485872386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f343010d020 0x7f343019d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.332+0000 7f3436990640 1 --2- 172.21.15.70:0/2485872386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3430103ce0 0x7f34301990e0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:47.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.332+0000 7f3436990640 1 -- 172.21.15.70:0/2485872386 >> 172.21.15.70:0/2485872386 conn(0x7f3430078ad0 msgr2=0x7f34300feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:47.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.332+0000 7f3436990640 1 -- 172.21.15.70:0/2485872386 shutdown_connections 2024-06-04T05:18:47.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:18:47.332+0000 7f3436990640 1 -- 172.21.15.70:0/2485872386 wait complete. 2024-06-04T05:18:47.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 7.20 MB/sec execute 339 sec latency 6319.063 ms 2024-06-04T05:18:47.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:47 smithi098 ceph-mon[35569]: pgmap v679: 97 pgs: 97 active+clean; 896 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.3 MiB/s wr, 342 op/s 2024-06-04T05:18:48.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:47 smithi070 ceph-mon[32631]: pgmap v679: 97 pgs: 97 active+clean; 896 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.3 MiB/s wr, 342 op/s 2024-06-04T05:18:48.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:47 smithi112 ceph-mon[36453]: pgmap v679: 97 pgs: 97 active+clean; 896 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.3 MiB/s wr, 342 op/s 2024-06-04T05:18:48.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 7.18 MB/sec execute 340 sec latency 7319.120 ms 2024-06-04T05:18:49.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 7.16 MB/sec execute 341 sec latency 8319.191 ms 2024-06-04T05:18:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:49 smithi098 ceph-mon[35569]: pgmap v680: 97 pgs: 97 active+clean; 896 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 187 op/s 2024-06-04T05:18:50.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:49 smithi070 ceph-mon[32631]: pgmap v680: 97 pgs: 97 active+clean; 896 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 187 op/s 2024-06-04T05:18:50.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:49 smithi112 ceph-mon[36453]: pgmap v680: 97 pgs: 97 active+clean; 896 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 187 op/s 2024-06-04T05:18:50.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 7.14 MB/sec execute 342 sec latency 9319.299 ms 2024-06-04T05:18:51.825 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 7.12 MB/sec execute 343 sec latency 10319.346 ms 2024-06-04T05:18:52.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:51 smithi070 ceph-mon[32631]: pgmap v681: 97 pgs: 97 active+clean; 896 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 187 op/s 2024-06-04T05:18:52.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:51 smithi112 ceph-mon[36453]: pgmap v681: 97 pgs: 97 active+clean; 896 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 187 op/s 2024-06-04T05:18:52.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:51 smithi098 ceph-mon[35569]: pgmap v681: 97 pgs: 97 active+clean; 896 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 187 op/s 2024-06-04T05:18:52.826 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 7.10 MB/sec execute 344 sec latency 11319.476 ms 2024-06-04T05:18:53.826 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 7.08 MB/sec execute 345 sec latency 12319.535 ms 2024-06-04T05:18:54.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:53 smithi070 ceph-mon[32631]: pgmap v682: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 218 op/s 2024-06-04T05:18:54.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:53 smithi112 ceph-mon[36453]: pgmap v682: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 218 op/s 2024-06-04T05:18:54.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:53 smithi098 ceph-mon[35569]: pgmap v682: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 218 op/s 2024-06-04T05:18:54.695 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:18:54.826 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 7.06 MB/sec execute 346 sec latency 13319.571 ms 2024-06-04T05:18:54.951 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:18:55.826 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 7.04 MB/sec execute 347 sec latency 14319.605 ms 2024-06-04T05:18:56.098 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:55 smithi070 ceph-mon[32631]: pgmap v683: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 475 KiB/s wr, 86 op/s 2024-06-04T05:18:56.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:55 smithi112 ceph-mon[36453]: pgmap v683: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 475 KiB/s wr, 86 op/s 2024-06-04T05:18:56.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:55 smithi098 ceph-mon[35569]: pgmap v683: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 475 KiB/s wr, 86 op/s 2024-06-04T05:18:56.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.708+0000 7ff4e750f700 1 -- 172.21.15.70:0/2498024816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4e0102e00 msgr2=0x7ff4e010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:56.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.708+0000 7ff4e750f700 1 --2- 172.21.15.70:0/2498024816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4e0102e00 0x7ff4e010f2f0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800b0a0 tx=0x7ff4d800b3b0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.708+0000 7ff4e750f700 1 -- 172.21.15.70:0/2498024816 shutdown_connections 2024-06-04T05:18:56.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.708+0000 7ff4e750f700 1 --2- 172.21.15.70:0/2498024816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4e0102e00 0x7ff4e010f2f0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.708+0000 7ff4e750f700 1 --2- 172.21.15.70:0/2498024816 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e0108520 0x7ff4e0108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.708+0000 7ff4e750f700 1 --2- 172.21.15.70:0/2498024816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4e0102440 0x7ff4e01028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.708+0000 7ff4e750f700 1 -- 172.21.15.70:0/2498024816 >> 172.21.15.70:0/2498024816 conn(0x7ff4e00793f0 msgr2=0x7ff4e0079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:56.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.708+0000 7ff4e750f700 1 -- 172.21.15.70:0/2498024816 shutdown_connections 2024-06-04T05:18:56.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.709+0000 7ff4e750f700 1 -- 172.21.15.70:0/2498024816 wait complete. 2024-06-04T05:18:56.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.709+0000 7ff4e750f700 1 Processor -- start 2024-06-04T05:18:56.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.709+0000 7ff4e750f700 1 -- start start 2024-06-04T05:18:56.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.709+0000 7ff4e750f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4e0102440 0x7ff4e019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.709+0000 7ff4e750f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e0102e00 0x7ff4e019ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.709+0000 7ff4e750f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4e0108520 0x7ff4e0198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.709+0000 7ff4e750f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e0112500 con 0x7ff4e0102440 2024-06-04T05:18:56.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.709+0000 7ff4e750f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e0112380 con 0x7ff4e0108520 2024-06-04T05:18:56.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.709+0000 7ff4e750f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e0112680 con 0x7ff4e0102e00 2024-06-04T05:18:56.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.709+0000 7ff4e52ab700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4e0102440 0x7ff4e019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.709+0000 7ff4e52ab700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4e0102440 0x7ff4e019e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47712/0 (socket says 172.21.15.70:47712) 2024-06-04T05:18:56.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.709+0000 7ff4e52ab700 1 -- 172.21.15.70:0/3507291887 learned_addr learned my addr 172.21.15.70:0/3507291887 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:56.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.710+0000 7ff4e4aaa700 1 --2- 172.21.15.70:0/3507291887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e0102e00 0x7ff4e019ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.710+0000 7ff4e52ab700 1 -- 172.21.15.70:0/3507291887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e0102e00 msgr2=0x7ff4e019ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:56.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.710+0000 7ff4e52ab700 1 --2- 172.21.15.70:0/3507291887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e0102e00 0x7ff4e019ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.710+0000 7ff4e52ab700 1 -- 172.21.15.70:0/3507291887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4e0108520 msgr2=0x7ff4e0198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:56.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.710+0000 7ff4e5aac700 1 --2- 172.21.15.70:0/3507291887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4e0108520 0x7ff4e0198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.710+0000 7ff4e52ab700 1 --2- 172.21.15.70:0/3507291887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4e0108520 0x7ff4e0198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.710+0000 7ff4e52ab700 1 -- 172.21.15.70:0/3507291887 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4e0199020 con 0x7ff4e0102440 2024-06-04T05:18:56.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.710+0000 7ff4e4aaa700 1 --2- 172.21.15.70:0/3507291887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e0102e00 0x7ff4e019ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:18:56.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.710+0000 7ff4e52ab700 1 --2- 172.21.15.70:0/3507291887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4e0102440 0x7ff4e019e6c0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00b530 tx=0x7ff4dc00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:56.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.710+0000 7ff4d67fc700 1 -- 172.21.15.70:0/3507291887 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc015020 con 0x7ff4e0102440 2024-06-04T05:18:56.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.710+0000 7ff4e750f700 1 -- 172.21.15.70:0/3507291887 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4e01992b0 con 0x7ff4e0102440 2024-06-04T05:18:56.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.710+0000 7ff4d67fc700 1 -- 172.21.15.70:0/3507291887 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff4dc011cc0 con 0x7ff4e0102440 2024-06-04T05:18:56.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.710+0000 7ff4d67fc700 1 -- 172.21.15.70:0/3507291887 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc010d30 con 0x7ff4e0102440 2024-06-04T05:18:56.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.710+0000 7ff4e750f700 1 -- 172.21.15.70:0/3507291887 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4e01a5520 con 0x7ff4e0102440 2024-06-04T05:18:56.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.713+0000 7ff4d67fc700 1 -- 172.21.15.70:0/3507291887 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff4dc018020 con 0x7ff4e0102440 2024-06-04T05:18:56.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.713+0000 7ff4d67fc700 1 --2- 172.21.15.70:0/3507291887 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4cc0b29f0 0x7ff4cc0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.713+0000 7ff4e4aaa700 1 --2- 172.21.15.70:0/3507291887 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4cc0b29f0 0x7ff4cc0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.713+0000 7ff4d67fc700 1 -- 172.21.15.70:0/3507291887 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7ff4dc0e1dd0 con 0x7ff4e0102440 2024-06-04T05:18:56.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.714+0000 7ff4e4aaa700 1 --2- 172.21.15.70:0/3507291887 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4cc0b29f0 0x7ff4cc0b4eb0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0005fd0 tx=0x7ff4d0005dc0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:56.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.714+0000 7ff4f1da4b80 1 Processor -- start 2024-06-04T05:18:56.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.714+0000 7ff4f1da4b80 1 -- start start 2024-06-04T05:18:56.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.714+0000 7ff4f1da4b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036d490 0x56380036d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.714+0000 7ff4f1da4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56380036ddb0 0x563800378750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.714+0000 7ff4f1da4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563800378c90 0x56380037b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.714+0000 7ff4f1da4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5638002cad70 con 0x56380036ddb0 2024-06-04T05:18:56.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.714+0000 7ff4f1da4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5638002ba910 con 0x563800378c90 2024-06-04T05:18:56.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.714+0000 7ff4f1da4b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5638002be950 con 0x56380036d490 2024-06-04T05:18:56.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.714+0000 7ff4e4aaa700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56380036ddb0 0x563800378750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.714+0000 7ff4e4aaa700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56380036ddb0 0x563800378750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47716/0 (socket says 172.21.15.70:47716) 2024-06-04T05:18:56.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.714+0000 7ff4e4aaa700 1 -- 172.21.15.70:0/2978294859 learned_addr learned my addr 172.21.15.70:0/2978294859 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:56.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.715+0000 7ff4e52ab700 1 --2- 172.21.15.70:0/2978294859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036d490 0x56380036d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.715+0000 7ff4e5aac700 1 --2- 172.21.15.70:0/2978294859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563800378c90 0x56380037b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.715+0000 7ff4e4aaa700 1 -- 172.21.15.70:0/2978294859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036d490 msgr2=0x56380036d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:56.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.715+0000 7ff4e4aaa700 1 --2- 172.21.15.70:0/2978294859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036d490 0x56380036d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.715+0000 7ff4e4aaa700 1 -- 172.21.15.70:0/2978294859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563800378c90 msgr2=0x56380037b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:56.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.715+0000 7ff4e4aaa700 1 --2- 172.21.15.70:0/2978294859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563800378c90 0x56380037b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.715+0000 7ff4e4aaa700 1 -- 172.21.15.70:0/2978294859 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563800263fb0 con 0x56380036ddb0 2024-06-04T05:18:56.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.715+0000 7ff4e52ab700 1 --2- 172.21.15.70:0/2978294859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036d490 0x56380036d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:18:56.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.715+0000 7ff4e5aac700 1 --2- 172.21.15.70:0/2978294859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563800378c90 0x56380037b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:18:56.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.715+0000 7ff4e4aaa700 1 --2- 172.21.15.70:0/2978294859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56380036ddb0 0x563800378750 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000db50 tx=0x7ff4d0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa21617dc28c227a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:56.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.715+0000 7ff4cb7fe700 1 -- 172.21.15.70:0/2978294859 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0020070 con 0x56380036ddb0 2024-06-04T05:18:56.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.715+0000 7ff4cb7fe700 1 -- 172.21.15.70:0/2978294859 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff4d000fd40 con 0x56380036ddb0 2024-06-04T05:18:56.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.715+0000 7ff4cb7fe700 1 -- 172.21.15.70:0/2978294859 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d001ce60 con 0x56380036ddb0 2024-06-04T05:18:56.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.716+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/2978294859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56380036ddb0 msgr2=0x563800378750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:56.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.716+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/2978294859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56380036ddb0 0x563800378750 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000db50 tx=0x7ff4d0010000 comp rx=0 tx=0).stop 2024-06-04T05:18:56.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.716+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/2978294859 shutdown_connections 2024-06-04T05:18:56.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.716+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/2978294859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563800378c90 0x56380037b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.716+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/2978294859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036d490 0x56380036d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.716+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/2978294859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56380036ddb0 0x563800378750 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.716+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/2978294859 >> 172.21.15.70:0/2978294859 conn(0x563800298240 msgr2=0x563800369e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:56.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.716+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/2978294859 shutdown_connections 2024-06-04T05:18:56.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.716+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/2978294859 wait complete. 2024-06-04T05:18:56.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.716+0000 7ff4f1da4b80 1 -- start start 2024-06-04T05:18:56.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.716+0000 7ff4f1da4b80 1 Processor -- start 2024-06-04T05:18:56.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4f1da4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56380036d490 0x563800378440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4f1da4b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036ddb0 0x563800375740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4f1da4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 0x563800375c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4f1da4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5638001c0fa0 con 0x563800378c90 2024-06-04T05:18:56.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4f1da4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563800222da0 con 0x56380036d490 2024-06-04T05:18:56.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4f1da4b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56380037b9c0 con 0x56380036ddb0 2024-06-04T05:18:56.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4f1da4b80 10 client.0 _ll_register_callbacks cb 0x563800255950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:56.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4e5aac700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 0x563800375c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4e5aac700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 0x563800375c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47718/0 (socket says 172.21.15.70:47718) 2024-06-04T05:18:56.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4e5aac700 1 -- 172.21.15.70:0/413232173 learned_addr learned my addr 172.21.15.70:0/413232173 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:56.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4e52ab700 1 --2- 172.21.15.70:0/413232173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56380036d490 0x563800378440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4e4aaa700 1 --2- 172.21.15.70:0/413232173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036ddb0 0x563800375740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4e5aac700 1 -- 172.21.15.70:0/413232173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036ddb0 msgr2=0x563800375740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:56.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4e5aac700 1 --2- 172.21.15.70:0/413232173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036ddb0 0x563800375740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4e5aac700 1 -- 172.21.15.70:0/413232173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56380036d490 msgr2=0x563800378440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:56.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4e5aac700 1 --2- 172.21.15.70:0/413232173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56380036d490 0x563800378440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4e5aac700 1 -- 172.21.15.70:0/413232173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56380020a8d0 con 0x563800378c90 2024-06-04T05:18:56.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4e5aac700 1 --2- 172.21.15.70:0/413232173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 0x563800375c80 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800b830 tx=0x7ff4d80095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b048ddb03e83e6e3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:56.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4c8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:18:56.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.717+0000 7ff4c8ff9700 1 -- 172.21.15.70:0/413232173 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d80076e0 con 0x563800378c90 2024-06-04T05:18:56.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.718+0000 7ff4c8ff9700 1 -- 172.21.15.70:0/413232173 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7ff4d8007880 con 0x563800378c90 2024-06-04T05:18:56.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.718+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/413232173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4d000fd40 con 0x563800378c90 2024-06-04T05:18:56.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.718+0000 7ff4c8ff9700 1 -- 172.21.15.70:0/413232173 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7ff4d8007a80 con 0x563800378c90 2024-06-04T05:18:56.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.718+0000 7ff4f1da4b80 10 client.20964 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:56.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.718+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/413232173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x563800274760 con 0x563800378c90 2024-06-04T05:18:56.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.718+0000 7ff4c8ff9700 1 -- 172.21.15.70:0/413232173 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff4d000fd40 con 0x563800378c90 2024-06-04T05:18:56.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.718+0000 7ff4c8ff9700 1 -- 172.21.15.70:0/413232173 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7ff4d8011070 con 0x563800378c90 2024-06-04T05:18:56.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.718+0000 7ff4f1da4b80 10 client.20964 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:56.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.718+0000 7ff4f1da4b80 10 client.20964 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:18:56.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.718+0000 7ff4f1da4b80 20 client.20964 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:56.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.718+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/413232173 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563800388610 0x56380038aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.718+0000 7ff4f1da4b80 4 client.20964 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:18:56.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.718+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/413232173 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638002cccc0 con 0x563800388610 2024-06-04T05:18:56.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.718+0000 7ff4e52ab700 1 --2- 172.21.15.70:0/413232173 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563800388610 0x56380038aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.719+0000 7ff4c8ff9700 10 client.20964 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:56.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.719+0000 7ff4e52ab700 1 --2- 172.21.15.70:0/413232173 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563800388610 0x56380038aa20 crc :-1 s=READY pgs=457 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f33d355421ff31eb server_cookie=9ab9ec1f5d94c9dc in_seq=0 out_seq=0 2024-06-04T05:18:56.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.720+0000 7ff4c8ff9700 1 -- 172.21.15.70:0/413232173 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5638002cccc0 con 0x563800388610 2024-06-04T05:18:56.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.720+0000 7ff4c8ff9700 10 client.20964 handle_command_reply: tid=0 2024-06-04T05:18:56.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.720+0000 7ff4f1da4b80 1 client.20964 shutdown 2024-06-04T05:18:56.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.720+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/413232173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 msgr2=0x563800375c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:56.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.720+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/413232173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 0x563800375c80 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800b830 tx=0x7ff4d80095d0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.720+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/413232173 shutdown_connections 2024-06-04T05:18:56.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.720+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/413232173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 0x563800375c80 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.720+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/413232173 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563800388610 0x56380038aa20 crc :-1 s=READY pgs=457 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.720+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/413232173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56380036d490 0x563800378440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.720+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/413232173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036ddb0 0x563800375740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.720+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/413232173 >> 172.21.15.70:0/413232173 conn(0x563800298240 msgr2=0x563800366af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:56.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.720+0000 7ff4c8ff9700 0 client.20964 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:56.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.720+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/413232173 shutdown_connections 2024-06-04T05:18:56.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.720+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/413232173 wait complete. 2024-06-04T05:18:56.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.720+0000 7ff4f1da4b80 20 client.20964 trim_cache size 0 max 16384 2024-06-04T05:18:56.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.740+0000 7ff4f1da4b80 1 Processor -- start 2024-06-04T05:18:56.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.741+0000 7ff4f1da4b80 1 -- start start 2024-06-04T05:18:56.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.741+0000 7ff4f1da4b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036d490 0x563800369fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.741+0000 7ff4f1da4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56380036ddb0 0x5638003902f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.741+0000 7ff4f1da4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 0x563800390830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.741+0000 7ff4f1da4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e01abb70 con 0x563800378c90 2024-06-04T05:18:56.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.741+0000 7ff4f1da4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e01ad710 con 0x56380036ddb0 2024-06-04T05:18:56.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.741+0000 7ff4f1da4b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e01ac2a0 con 0x56380036d490 2024-06-04T05:18:56.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.741+0000 7ff4e52ab700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036d490 0x563800369fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.741+0000 7ff4e52ab700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036d490 0x563800369fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:41108/0 (socket says 172.21.15.70:41108) 2024-06-04T05:18:56.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.741+0000 7ff4e5aac700 1 --2- 172.21.15.70:0/48177000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 0x563800390830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.741+0000 7ff4e52ab700 1 -- 172.21.15.70:0/48177000 learned_addr learned my addr 172.21.15.70:0/48177000 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:56.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.741+0000 7ff4e4aaa700 1 --2- 172.21.15.70:0/48177000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56380036ddb0 0x5638003902f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.741+0000 7ff4e5aac700 1 -- 172.21.15.70:0/48177000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036d490 msgr2=0x563800369fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:56.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.741+0000 7ff4e5aac700 1 --2- 172.21.15.70:0/48177000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036d490 0x563800369fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.741+0000 7ff4e5aac700 1 -- 172.21.15.70:0/48177000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56380036ddb0 msgr2=0x5638003902f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:56.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.741+0000 7ff4e5aac700 1 --2- 172.21.15.70:0/48177000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56380036ddb0 0x5638003902f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.741+0000 7ff4e5aac700 1 -- 172.21.15.70:0/48177000 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638003692e0 con 0x563800378c90 2024-06-04T05:18:56.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.742+0000 7ff4e4aaa700 1 --2- 172.21.15.70:0/48177000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56380036ddb0 0x5638003902f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:56.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.742+0000 7ff4e52ab700 1 --2- 172.21.15.70:0/48177000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036d490 0x563800369fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:18:56.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.742+0000 7ff4e5aac700 1 --2- 172.21.15.70:0/48177000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 0x563800390830 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8005b80 tx=0x7ff4d8004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81772086e2168ab8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:56.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.742+0000 7ff4caffd700 1 -- 172.21.15.70:0/48177000 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8032070 con 0x563800378c90 2024-06-04T05:18:56.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.742+0000 7ff4caffd700 1 -- 172.21.15.70:0/48177000 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff4d8007ce0 con 0x563800378c90 2024-06-04T05:18:56.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.742+0000 7ff4caffd700 1 -- 172.21.15.70:0/48177000 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8026bb0 con 0x563800378c90 2024-06-04T05:18:56.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.743+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/48177000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 msgr2=0x563800390830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:56.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.743+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/48177000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 0x563800390830 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8005b80 tx=0x7ff4d8004db0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.743+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/48177000 shutdown_connections 2024-06-04T05:18:56.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.743+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/48177000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 0x563800390830 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.743+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/48177000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036d490 0x563800369fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.743+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/48177000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56380036ddb0 0x5638003902f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.743+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/48177000 >> 172.21.15.70:0/48177000 conn(0x563800298240 msgr2=0x5638003698d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:56.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.743+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/48177000 shutdown_connections 2024-06-04T05:18:56.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.743+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/48177000 wait complete. 2024-06-04T05:18:56.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.743+0000 7ff4f1da4b80 1 -- start start 2024-06-04T05:18:56.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.743+0000 7ff4f1da4b80 1 Processor -- start 2024-06-04T05:18:56.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.744+0000 7ff4f1da4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56380036d490 0x563800372490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.744+0000 7ff4f1da4b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036ddb0 0x5638003729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.744+0000 7ff4f1da4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 0x56380037fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.744+0000 7ff4f1da4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e01ad040 con 0x563800378c90 2024-06-04T05:18:56.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.744+0000 7ff4f1da4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e01ac970 con 0x56380036d490 2024-06-04T05:18:56.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.744+0000 7ff4f1da4b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e01a9b30 con 0x56380036ddb0 2024-06-04T05:18:56.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.744+0000 7ff4f1da4b80 10 client.0 _ll_register_callbacks cb 0x563800255950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:18:56.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.744+0000 7ff4e5aac700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 0x56380037fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.744+0000 7ff4e5aac700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 0x56380037fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47738/0 (socket says 172.21.15.70:47738) 2024-06-04T05:18:56.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.744+0000 7ff4e5aac700 1 -- 172.21.15.70:0/4013934067 learned_addr learned my addr 172.21.15.70:0/4013934067 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:56.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.744+0000 7ff4e52ab700 1 --2- 172.21.15.70:0/4013934067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56380036d490 0x563800372490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.744+0000 7ff4e4aaa700 1 --2- 172.21.15.70:0/4013934067 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036ddb0 0x5638003729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.744+0000 7ff4e5aac700 1 -- 172.21.15.70:0/4013934067 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036ddb0 msgr2=0x5638003729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:56.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.744+0000 7ff4e5aac700 1 --2- 172.21.15.70:0/4013934067 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036ddb0 0x5638003729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.744+0000 7ff4e5aac700 1 -- 172.21.15.70:0/4013934067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56380036d490 msgr2=0x563800372490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:18:56.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.744+0000 7ff4e5aac700 1 --2- 172.21.15.70:0/4013934067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56380036d490 0x563800372490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.744+0000 7ff4e5aac700 1 -- 172.21.15.70:0/4013934067 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563800380270 con 0x563800378c90 2024-06-04T05:18:56.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.745+0000 7ff4e5aac700 1 --2- 172.21.15.70:0/4013934067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 0x56380037fc30 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800d010 tx=0x7ff4d8005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5bedcab142b302 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:56.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.745+0000 7ff4c8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:18:56.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.745+0000 7ff4c8ff9700 1 -- 172.21.15.70:0/4013934067 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d803b020 con 0x563800378c90 2024-06-04T05:18:56.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.745+0000 7ff4c8ff9700 1 -- 172.21.15.70:0/4013934067 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7ff4d800f040 con 0x563800378c90 2024-06-04T05:18:56.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.745+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/4013934067 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4d8007ce0 con 0x563800378c90 2024-06-04T05:18:56.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.745+0000 7ff4c8ff9700 1 -- 172.21.15.70:0/4013934067 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7ff4d8046050 con 0x563800378c90 2024-06-04T05:18:56.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.745+0000 7ff4f1da4b80 10 client.20970 fetch_fsmap learned FSMap version 59 2024-06-04T05:18:56.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.745+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/4013934067 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x563800381560 con 0x563800378c90 2024-06-04T05:18:56.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.745+0000 7ff4c8ff9700 1 -- 172.21.15.70:0/4013934067 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff4d8007ce0 con 0x563800378c90 2024-06-04T05:18:56.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.745+0000 7ff4c8ff9700 1 -- 172.21.15.70:0/4013934067 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7ff4d8011040 con 0x563800378c90 2024-06-04T05:18:56.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.746+0000 7ff4f1da4b80 10 client.20970 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:18:56.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.746+0000 7ff4f1da4b80 10 client.20970 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:18:56.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.746+0000 7ff4f1da4b80 20 client.20970 populate_metadata read hostname 'smithi070' 2024-06-04T05:18:56.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.746+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/4013934067 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5638003f9ff0 0x5638003fc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:56.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.746+0000 7ff4f1da4b80 4 client.20970 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:18:56.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.746+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/4013934067 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5638002c64a0 con 0x5638003f9ff0 2024-06-04T05:18:56.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.746+0000 7ff4e52ab700 1 --2- 172.21.15.70:0/4013934067 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5638003f9ff0 0x5638003fc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:56.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.746+0000 7ff4c8ff9700 10 client.20970 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:56.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.746+0000 7ff4e52ab700 1 --2- 172.21.15.70:0/4013934067 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5638003f9ff0 0x5638003fc410 crc :-1 s=READY pgs=458 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c00024c292dd7d28 server_cookie=f42f257e35d45ead in_seq=0 out_seq=0 2024-06-04T05:18:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.746+0000 7ff4c8ff9700 1 -- 172.21.15.70:0/4013934067 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5638002c64a0 con 0x5638003f9ff0 2024-06-04T05:18:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.746+0000 7ff4c8ff9700 10 client.20970 handle_command_reply: tid=0 2024-06-04T05:18:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.747+0000 7ff4f1da4b80 1 client.20970 shutdown 2024-06-04T05:18:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.747+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/4013934067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 msgr2=0x56380037fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.747+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/4013934067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 0x56380037fc30 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800d010 tx=0x7ff4d8005270 comp rx=0 tx=0).stop 2024-06-04T05:18:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.747+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/4013934067 shutdown_connections 2024-06-04T05:18:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.747+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/4013934067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563800378c90 0x56380037fc30 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.747+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/4013934067 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5638003f9ff0 0x5638003fc410 crc :-1 s=READY pgs=458 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.747+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/4013934067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56380036d490 0x563800372490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.747+0000 7ff4f1da4b80 1 --2- 172.21.15.70:0/4013934067 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56380036ddb0 0x5638003729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.747+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/4013934067 >> 172.21.15.70:0/4013934067 conn(0x563800298240 msgr2=0x563800370640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.747+0000 7ff4c8ff9700 0 client.20970 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:18:56.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.747+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/4013934067 shutdown_connections 2024-06-04T05:18:56.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.747+0000 7ff4f1da4b80 1 -- 172.21.15.70:0/4013934067 wait complete. 2024-06-04T05:18:56.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.747+0000 7ff4f1da4b80 20 client.20970 trim_cache size 0 max 16384 2024-06-04T05:18:56.757 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:18:56.757 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:18:56.757 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:18:56.757 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:18:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.747+0000 7ff4e750f700 1 -- 172.21.15.70:0/3507291887 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4cc0b29f0 msgr2=0x7ff4cc0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.748+0000 7ff4e750f700 1 --2- 172.21.15.70:0/3507291887 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4cc0b29f0 0x7ff4cc0b4eb0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0005fd0 tx=0x7ff4d0005dc0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.748+0000 7ff4e750f700 1 -- 172.21.15.70:0/3507291887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4e0102440 msgr2=0x7ff4e019e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.748+0000 7ff4e750f700 1 --2- 172.21.15.70:0/3507291887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4e0102440 0x7ff4e019e6c0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00b530 tx=0x7ff4dc00b8f0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.748+0000 7ff4e750f700 1 -- 172.21.15.70:0/3507291887 shutdown_connections 2024-06-04T05:18:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.748+0000 7ff4e750f700 1 --2- 172.21.15.70:0/3507291887 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4cc0b29f0 0x7ff4cc0b4eb0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.748+0000 7ff4e750f700 1 --2- 172.21.15.70:0/3507291887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4e0108520 0x7ff4e0198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.748+0000 7ff4e750f700 1 --2- 172.21.15.70:0/3507291887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4e0102440 0x7ff4e019e6c0 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.748+0000 7ff4e750f700 1 --2- 172.21.15.70:0/3507291887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e0102e00 0x7ff4e019ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:56.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.748+0000 7ff4e750f700 1 -- 172.21.15.70:0/3507291887 >> 172.21.15.70:0/3507291887 conn(0x7ff4e00793f0 msgr2=0x7ff4e01005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:56.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.748+0000 7ff4e750f700 1 -- 172.21.15.70:0/3507291887 shutdown_connections 2024-06-04T05:18:56.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:56.748+0000 7ff4e750f700 1 -- 172.21.15.70:0/3507291887 wait complete. 2024-06-04T05:18:56.826 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 7.02 MB/sec execute 348 sec latency 15319.982 ms 2024-06-04T05:18:57.448 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:18:57.449 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:18:57.699 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:18:57.826 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 7.00 MB/sec execute 349 sec latency 16320.025 ms 2024-06-04T05:18:58.023 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:57 smithi070 ceph-mon[32631]: pgmap v684: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 482 KiB/s wr, 91 op/s 2024-06-04T05:18:58.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:57 smithi112 ceph-mon[36453]: pgmap v684: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 482 KiB/s wr, 91 op/s 2024-06-04T05:18:58.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:57 smithi098 ceph-mon[35569]: pgmap v684: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 482 KiB/s wr, 91 op/s 2024-06-04T05:18:58.826 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 6.98 MB/sec execute 350 sec latency 17320.053 ms 2024-06-04T05:18:59.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.306+0000 7fbd33d43700 1 -- 172.21.15.70:0/460835321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd2c1037f0 msgr2=0x7fbd2c103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.306+0000 7fbd33d43700 1 --2- 172.21.15.70:0/460835321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd2c1037f0 0x7fbd2c103bd0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fbd280099c0 tx=0x7fbd28009cd0 comp rx=0 tx=0).stop 2024-06-04T05:18:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.306+0000 7fbd33d43700 1 -- 172.21.15.70:0/460835321 shutdown_connections 2024-06-04T05:18:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.306+0000 7fbd33d43700 1 --2- 172.21.15.70:0/460835321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd2c1088f0 0x7fbd2c110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.306+0000 7fbd33d43700 1 --2- 172.21.15.70:0/460835321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd2c1037f0 0x7fbd2c103bd0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.306+0000 7fbd33d43700 1 --2- 172.21.15.70:0/460835321 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd2c1041a0 0x7fbd2c1081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.306+0000 7fbd33d43700 1 -- 172.21.15.70:0/460835321 >> 172.21.15.70:0/460835321 conn(0x7fbd2c07a6a0 msgr2=0x7fbd2c07aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.307+0000 7fbd33d43700 1 -- 172.21.15.70:0/460835321 shutdown_connections 2024-06-04T05:18:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.307+0000 7fbd33d43700 1 -- 172.21.15.70:0/460835321 wait complete. 2024-06-04T05:18:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.307+0000 7fbd33d43700 1 Processor -- start 2024-06-04T05:18:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.307+0000 7fbd33d43700 1 -- start start 2024-06-04T05:18:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.307+0000 7fbd33d43700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd2c1037f0 0x7fbd2c19bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.307+0000 7fbd33d43700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd2c1041a0 0x7fbd2c19c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.307+0000 7fbd33d43700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd2c1088f0 0x7fbd2c1a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.307+0000 7fbd33d43700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2c113570 con 0x7fbd2c1037f0 2024-06-04T05:18:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.307+0000 7fbd33d43700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2c1133f0 con 0x7fbd2c1088f0 2024-06-04T05:18:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.307+0000 7fbd33d43700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2c1136f0 con 0x7fbd2c1041a0 2024-06-04T05:18:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.307+0000 7fbd31adf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd2c1037f0 0x7fbd2c19bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.307+0000 7fbd31adf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd2c1037f0 0x7fbd2c19bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47752/0 (socket says 172.21.15.70:47752) 2024-06-04T05:18:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.308+0000 7fbd31adf700 1 -- 172.21.15.70:0/3536912750 learned_addr learned my addr 172.21.15.70:0/3536912750 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:18:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.308+0000 7fbd322e0700 1 --2- 172.21.15.70:0/3536912750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd2c1088f0 0x7fbd2c1a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.308+0000 7fbd312de700 1 --2- 172.21.15.70:0/3536912750 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd2c1041a0 0x7fbd2c19c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.308+0000 7fbd31adf700 1 -- 172.21.15.70:0/3536912750 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd2c1041a0 msgr2=0x7fbd2c19c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.308+0000 7fbd31adf700 1 --2- 172.21.15.70:0/3536912750 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd2c1041a0 0x7fbd2c19c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.308+0000 7fbd31adf700 1 -- 172.21.15.70:0/3536912750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd2c1088f0 msgr2=0x7fbd2c1a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.308+0000 7fbd31adf700 1 --2- 172.21.15.70:0/3536912750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd2c1088f0 0x7fbd2c1a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:59.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.308+0000 7fbd31adf700 1 -- 172.21.15.70:0/3536912750 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd2c1a0d30 con 0x7fbd2c1037f0 2024-06-04T05:18:59.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.308+0000 7fbd31adf700 1 --2- 172.21.15.70:0/3536912750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd2c1037f0 0x7fbd2c19bd20 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fbd28000c00 tx=0x7fbd280047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:59.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.308+0000 7fbd22ffd700 1 -- 172.21.15.70:0/3536912750 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2800f8a0 con 0x7fbd2c1037f0 2024-06-04T05:18:59.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.308+0000 7fbd22ffd700 1 -- 172.21.15.70:0/3536912750 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd28004830 con 0x7fbd2c1037f0 2024-06-04T05:18:59.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.308+0000 7fbd33d43700 1 -- 172.21.15.70:0/3536912750 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd2c1a0fc0 con 0x7fbd2c1037f0 2024-06-04T05:18:59.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.308+0000 7fbd22ffd700 1 -- 172.21.15.70:0/3536912750 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd28005410 con 0x7fbd2c1037f0 2024-06-04T05:18:59.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.308+0000 7fbd33d43700 1 -- 172.21.15.70:0/3536912750 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd2c1a88a0 con 0x7fbd2c1037f0 2024-06-04T05:18:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.309+0000 7fbd33d43700 1 -- 172.21.15.70:0/3536912750 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd2c104c80 con 0x7fbd2c1037f0 2024-06-04T05:18:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.310+0000 7fbd22ffd700 1 -- 172.21.15.70:0/3536912750 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fbd28029060 con 0x7fbd2c1037f0 2024-06-04T05:18:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.311+0000 7fbd22ffd700 1 --2- 172.21.15.70:0/3536912750 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd180b29f0 0x7fbd180b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:18:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.311+0000 7fbd312de700 1 --2- 172.21.15.70:0/3536912750 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd180b29f0 0x7fbd180b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:18:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.311+0000 7fbd22ffd700 1 -- 172.21.15.70:0/3536912750 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7fbd280e3120 con 0x7fbd2c1037f0 2024-06-04T05:18:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.312+0000 7fbd312de700 1 --2- 172.21.15.70:0/3536912750 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd180b29f0 0x7fbd180b4eb0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c005fd0 tx=0x7fbd1c00a560 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:18:59.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.316+0000 7fbd22ffd700 1 -- 172.21.15.70:0/3536912750 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbd28013070 con 0x7fbd2c1037f0 2024-06-04T05:18:59.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.570+0000 7fbd33d43700 1 -- 172.21.15.70:0/3536912750 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd2c1a8a70 con 0x7fbd2c1037f0 2024-06-04T05:18:59.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.570+0000 7fbd22ffd700 1 -- 172.21.15.70:0/3536912750 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+11932 (secure 0 0 0) 0x7fbd280aab60 con 0x7fbd2c1037f0 2024-06-04T05:18:59.573 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:18:59.573 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":59,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:17:20.010341+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:18:59.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.573+0000 7fbd33d43700 1 -- 172.21.15.70:0/3536912750 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd180b29f0 msgr2=0x7fbd180b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:59.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.573+0000 7fbd33d43700 1 --2- 172.21.15.70:0/3536912750 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd180b29f0 0x7fbd180b4eb0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c005fd0 tx=0x7fbd1c00a560 comp rx=0 tx=0).stop 2024-06-04T05:18:59.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.573+0000 7fbd33d43700 1 -- 172.21.15.70:0/3536912750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd2c1037f0 msgr2=0x7fbd2c19bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:18:59.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.573+0000 7fbd33d43700 1 --2- 172.21.15.70:0/3536912750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd2c1037f0 0x7fbd2c19bd20 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fbd28000c00 tx=0x7fbd280047b0 comp rx=0 tx=0).stop 2024-06-04T05:18:59.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.573+0000 7fbd33d43700 1 -- 172.21.15.70:0/3536912750 shutdown_connections 2024-06-04T05:18:59.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.573+0000 7fbd33d43700 1 --2- 172.21.15.70:0/3536912750 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd180b29f0 0x7fbd180b4eb0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:59.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.573+0000 7fbd33d43700 1 --2- 172.21.15.70:0/3536912750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd2c1088f0 0x7fbd2c1a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:59.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.573+0000 7fbd33d43700 1 --2- 172.21.15.70:0/3536912750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd2c1037f0 0x7fbd2c19bd20 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:59.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.573+0000 7fbd33d43700 1 --2- 172.21.15.70:0/3536912750 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd2c1041a0 0x7fbd2c19c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:18:59.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.573+0000 7fbd33d43700 1 -- 172.21.15.70:0/3536912750 >> 172.21.15.70:0/3536912750 conn(0x7fbd2c07a6a0 msgr2=0x7fbd2c101b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:18:59.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.573+0000 7fbd33d43700 1 -- 172.21.15.70:0/3536912750 shutdown_connections 2024-06-04T05:18:59.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:18:59.573+0000 7fbd33d43700 1 -- 172.21.15.70:0/3536912750 wait complete. 2024-06-04T05:18:59.577 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 59 2024-06-04T05:18:59.826 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 6.96 MB/sec execute 351 sec latency 18320.137 ms 2024-06-04T05:18:59.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:59 smithi070 ceph-mon[32631]: pgmap v685: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 129 KiB/s wr, 36 op/s 2024-06-04T05:18:59.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:18:59 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3536912750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:19:00.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:59 smithi112 ceph-mon[36453]: pgmap v685: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 129 KiB/s wr, 36 op/s 2024-06-04T05:19:00.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:18:59 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3536912750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:19:00.206 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:19:00.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:59 smithi098 ceph-mon[35569]: pgmap v685: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 129 KiB/s wr, 36 op/s 2024-06-04T05:19:00.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:18:59 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3536912750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:19:00.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.425+0000 7f0d144bd640 1 -- 172.21.15.70:0/2220742911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d0c100f50 msgr2=0x7f0d0c1013d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:00.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.425+0000 7f0d144bd640 1 --2- 172.21.15.70:0/2220742911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d0c100f50 0x7f0d0c1013d0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc009930 tx=0x7f0cfc02a420 comp rx=0 tx=0).stop 2024-06-04T05:19:00.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.425+0000 7f0d144bd640 1 -- 172.21.15.70:0/2220742911 shutdown_connections 2024-06-04T05:19:00.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.425+0000 7f0d144bd640 1 --2- 172.21.15.70:0/2220742911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d0c101910 0x7f0d0c10f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.425+0000 7f0d144bd640 1 --2- 172.21.15.70:0/2220742911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d0c100f50 0x7f0d0c1013d0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.425+0000 7f0d144bd640 1 --2- 172.21.15.70:0/2220742911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d0c107a30 0x7f0d0c107e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.425+0000 7f0d144bd640 1 -- 172.21.15.70:0/2220742911 >> 172.21.15.70:0/2220742911 conn(0x7f0d0c077ee0 msgr2=0x7f0d0c0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:00.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.425+0000 7f0d144bd640 1 -- 172.21.15.70:0/2220742911 shutdown_connections 2024-06-04T05:19:00.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.425+0000 7f0d144bd640 1 -- 172.21.15.70:0/2220742911 wait complete. 2024-06-04T05:19:00.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.426+0000 7f0d144bd640 1 Processor -- start 2024-06-04T05:19:00.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.426+0000 7f0d144bd640 1 -- start start 2024-06-04T05:19:00.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.426+0000 7f0d144bd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d0c100f50 0x7f0d0c102130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.426+0000 7f0d144bd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d0c101910 0x7f0d0c102670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.426+0000 7f0d144bd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d0c107a30 0x7f0d0c102bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.426+0000 7f0d144bd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0c112ca0 con 0x7f0d0c101910 2024-06-04T05:19:00.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.426+0000 7f0d144bd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0c112b20 con 0x7f0d0c100f50 2024-06-04T05:19:00.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.426+0000 7f0d144bd640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0c112e20 con 0x7f0d0c107a30 2024-06-04T05:19:00.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.426+0000 7f0d11a58640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d0c101910 0x7f0d0c102670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:00.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.426+0000 7f0d11a58640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d0c101910 0x7f0d0c102670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47770/0 (socket says 172.21.15.70:47770) 2024-06-04T05:19:00.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.426+0000 7f0d11a58640 1 -- 172.21.15.70:0/3149998719 learned_addr learned my addr 172.21.15.70:0/3149998719 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:00.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.426+0000 7f0d12a5a640 1 --2- 172.21.15.70:0/3149998719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d0c107a30 0x7f0d0c102bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:00.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.427+0000 7f0d12259640 1 --2- 172.21.15.70:0/3149998719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d0c100f50 0x7f0d0c102130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:00.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.427+0000 7f0d11a58640 1 -- 172.21.15.70:0/3149998719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d0c107a30 msgr2=0x7f0d0c102bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:00.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.427+0000 7f0d11a58640 1 --2- 172.21.15.70:0/3149998719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d0c107a30 0x7f0d0c102bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.427+0000 7f0d11a58640 1 -- 172.21.15.70:0/3149998719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d0c100f50 msgr2=0x7f0d0c102130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:00.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.427+0000 7f0d11a58640 1 --2- 172.21.15.70:0/3149998719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d0c100f50 0x7f0d0c102130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.427+0000 7f0d11a58640 1 -- 172.21.15.70:0/3149998719 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d0c1a76c0 con 0x7f0d0c101910 2024-06-04T05:19:00.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.427+0000 7f0d12a5a640 1 --2- 172.21.15.70:0/3149998719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d0c107a30 0x7f0d0c102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:19:00.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.427+0000 7f0d11a58640 1 --2- 172.21.15.70:0/3149998719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d0c101910 0x7f0d0c102670 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc005940 tx=0x7f0cfc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:00.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.427+0000 7f0d037fe640 1 -- 172.21.15.70:0/3149998719 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc03f030 con 0x7f0d0c101910 2024-06-04T05:19:00.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.427+0000 7f0d037fe640 1 -- 172.21.15.70:0/3149998719 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0cfc003c90 con 0x7f0d0c101910 2024-06-04T05:19:00.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.427+0000 7f0d144bd640 1 -- 172.21.15.70:0/3149998719 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d0c1a7950 con 0x7f0d0c101910 2024-06-04T05:19:00.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.427+0000 7f0d037fe640 1 -- 172.21.15.70:0/3149998719 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc035070 con 0x7f0d0c101910 2024-06-04T05:19:00.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.427+0000 7f0d144bd640 1 -- 172.21.15.70:0/3149998719 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d0c1a7e40 con 0x7f0d0c101910 2024-06-04T05:19:00.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.430+0000 7f0d037fe640 1 -- 172.21.15.70:0/3149998719 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f0cfc042030 con 0x7f0d0c101910 2024-06-04T05:19:00.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.430+0000 7f0d171d4740 1 Processor -- start 2024-06-04T05:19:00.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.430+0000 7f0d171d4740 1 -- start start 2024-06-04T05:19:00.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.430+0000 7f0d171d4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f1f820 0x558c22f1fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.430+0000 7f0d171d4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f20140 0x558c22f2a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.430+0000 7f0d171d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f2aa00 0x558c22f2cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.430+0000 7f0d171d4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558c22e91c30 con 0x558c22f20140 2024-06-04T05:19:00.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.430+0000 7f0d171d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558c22e817d0 con 0x558c22f2aa00 2024-06-04T05:19:00.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.430+0000 7f0d171d4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558c22e85810 con 0x558c22f1f820 2024-06-04T05:19:00.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.431+0000 7f0d12259640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f20140 0x558c22f2a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:00.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.431+0000 7f0d12259640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f20140 0x558c22f2a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47786/0 (socket says 172.21.15.70:47786) 2024-06-04T05:19:00.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.431+0000 7f0d12259640 1 -- 172.21.15.70:0/3016047694 learned_addr learned my addr 172.21.15.70:0/3016047694 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:00.434 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.431+0000 7f0d11a58640 1 --2- 172.21.15.70:0/3016047694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f2aa00 0x558c22f2cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.431+0000 7f0d12259640 1 -- 172.21.15.70:0/3016047694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f1f820 msgr2=0x558c22f1fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.431+0000 7f0d12259640 1 --2- 172.21.15.70:0/3016047694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f1f820 0x558c22f1fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.431+0000 7f0d037fe640 1 --2- 172.21.15.70:0/3149998719 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0cf80b2a40 0x7f0cf80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.431+0000 7f0d12259640 1 -- 172.21.15.70:0/3016047694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f2aa00 msgr2=0x558c22f2cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.431+0000 7f0d12259640 1 --2- 172.21.15.70:0/3016047694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f2aa00 0x558c22f2cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.431+0000 7f0d12259640 1 -- 172.21.15.70:0/3016047694 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c22b96210 con 0x558c22f20140 2024-06-04T05:19:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.431+0000 7f0d12259640 1 --2- 172.21.15.70:0/3016047694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f20140 0x558c22f2a4c0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f0d0800bc60 tx=0x7f0d0800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c7702d6a0db3600 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.431+0000 7f0d037fe640 1 -- 172.21.15.70:0/3149998719 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f0cfc102bc0 con 0x7f0d0c101910 2024-06-04T05:19:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.431+0000 7f0d12a5a640 1 --2- 172.21.15.70:0/3149998719 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0cf80b2a40 0x7f0cf80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.431+0000 7f0d00bfe640 1 -- 172.21.15.70:0/3016047694 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08015070 con 0x558c22f20140 2024-06-04T05:19:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.432+0000 7f0d00bfe640 1 -- 172.21.15.70:0/3016047694 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0d080118d0 con 0x558c22f20140 2024-06-04T05:19:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.432+0000 7f0d00bfe640 1 -- 172.21.15.70:0/3016047694 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d080106f0 con 0x558c22f20140 2024-06-04T05:19:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.432+0000 7f0d171d4740 1 -- 172.21.15.70:0/3016047694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f20140 msgr2=0x558c22f2a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.432+0000 7f0d12a5a640 1 --2- 172.21.15.70:0/3149998719 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0cf80b2a40 0x7f0cf80b4f00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0d04005c40 tx=0x7f0d04009510 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.432+0000 7f0d171d4740 1 --2- 172.21.15.70:0/3016047694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f20140 0x558c22f2a4c0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f0d0800bc60 tx=0x7f0d0800f730 comp rx=0 tx=0).stop 2024-06-04T05:19:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.432+0000 7f0d171d4740 1 -- 172.21.15.70:0/3016047694 shutdown_connections 2024-06-04T05:19:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.432+0000 7f0d171d4740 1 --2- 172.21.15.70:0/3016047694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f2aa00 0x558c22f2cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.432+0000 7f0d171d4740 1 --2- 172.21.15.70:0/3016047694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f20140 0x558c22f2a4c0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.432+0000 7f0d171d4740 1 --2- 172.21.15.70:0/3016047694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f1f820 0x558c22f1fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.432+0000 7f0d171d4740 1 -- 172.21.15.70:0/3016047694 >> 172.21.15.70:0/3016047694 conn(0x558c22f1bd30 msgr2=0x558c22f1c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.433+0000 7f0d171d4740 1 -- 172.21.15.70:0/3016047694 shutdown_connections 2024-06-04T05:19:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.433+0000 7f0d171d4740 1 -- 172.21.15.70:0/3016047694 wait complete. 2024-06-04T05:19:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.433+0000 7f0d171d4740 1 -- start start 2024-06-04T05:19:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.433+0000 7f0d171d4740 1 Processor -- start 2024-06-04T05:19:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.433+0000 7f0d171d4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f1f820 0x558c22f27a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.433+0000 7f0d171d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f20140 0x558c22f27f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.433+0000 7f0d171d4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f2aa00 0x558c22f284c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.433+0000 7f0d171d4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558c22b67790 con 0x558c22f1f820 2024-06-04T05:19:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.433+0000 7f0d171d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558c22e3d890 con 0x558c22f20140 2024-06-04T05:19:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.433+0000 7f0d171d4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558c22f2d630 con 0x558c22f2aa00 2024-06-04T05:19:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.433+0000 7f0d171d4740 10 client.0 _ll_register_callbacks cb 0x558c22b95a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.433+0000 7f0d12259640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f1f820 0x558c22f27a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.433+0000 7f0d12259640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f1f820 0x558c22f27a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47800/0 (socket says 172.21.15.70:47800) 2024-06-04T05:19:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.433+0000 7f0d12259640 1 -- 172.21.15.70:0/509859425 learned_addr learned my addr 172.21.15.70:0/509859425 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.433+0000 7f0d11a58640 1 --2- 172.21.15.70:0/509859425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f20140 0x558c22f27f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.433+0000 7f0d12a5a640 1 --2- 172.21.15.70:0/509859425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f2aa00 0x558c22f284c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.434+0000 7f0d12259640 1 -- 172.21.15.70:0/509859425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f2aa00 msgr2=0x558c22f284c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.434+0000 7f0d12259640 1 --2- 172.21.15.70:0/509859425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f2aa00 0x558c22f284c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.434+0000 7f0d12259640 1 -- 172.21.15.70:0/509859425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f20140 msgr2=0x558c22f27f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.434+0000 7f0d12259640 1 --2- 172.21.15.70:0/509859425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f20140 0x558c22f27f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.434+0000 7f0d12259640 1 -- 172.21.15.70:0/509859425 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c22c62620 con 0x558c22f1f820 2024-06-04T05:19:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.434+0000 7f0d12259640 1 --2- 172.21.15.70:0/509859425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f1f820 0x558c22f27a40 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f0d0800bf60 tx=0x7f0d08011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beae4a30ce5b55f2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.434+0000 7f0cf5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.434+0000 7f0cf5ffb640 1 -- 172.21.15.70:0/509859425 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08015070 con 0x558c22f1f820 2024-06-04T05:19:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.434+0000 7f0cf5ffb640 1 -- 172.21.15.70:0/509859425 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f0d08011a80 con 0x558c22f1f820 2024-06-04T05:19:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.434+0000 7f0d171d4740 1 -- 172.21.15.70:0/509859425 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d080118d0 con 0x558c22f1f820 2024-06-04T05:19:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.434+0000 7f0cf5ffb640 1 -- 172.21.15.70:0/509859425 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f0d08011c80 con 0x558c22f1f820 2024-06-04T05:19:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.434+0000 7f0d171d4740 10 client.20988 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.434+0000 7f0d171d4740 1 -- 172.21.15.70:0/509859425 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x558c22f39480 con 0x558c22f1f820 2024-06-04T05:19:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.434+0000 7f0cf5ffb640 1 -- 172.21.15.70:0/509859425 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0d080118d0 con 0x558c22f1f820 2024-06-04T05:19:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.435+0000 7f0cf5ffb640 1 -- 172.21.15.70:0/509859425 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f0d08005440 con 0x558c22f1f820 2024-06-04T05:19:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.435+0000 7f0d171d4740 10 client.20988 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.435+0000 7f0d171d4740 10 client.20988 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:19:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.435+0000 7f0d171d4740 20 client.20988 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.435+0000 7f0d171d4740 1 --2- 172.21.15.70:0/509859425 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558c22f3a880 0x558c22f3cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.435+0000 7f0d171d4740 4 client.20988 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:19:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.435+0000 7f0d171d4740 1 -- 172.21.15.70:0/509859425 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c22e93b80 con 0x558c22f3a880 2024-06-04T05:19:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.435+0000 7f0d11a58640 1 --2- 172.21.15.70:0/509859425 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558c22f3a880 0x558c22f3cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.435+0000 7f0d11a58640 1 --2- 172.21.15.70:0/509859425 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558c22f3a880 0x558c22f3cc70 crc :-1 s=READY pgs=459 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dac8956fe27ea4c server_cookie=33093e2bfb7d345e in_seq=0 out_seq=0 2024-06-04T05:19:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.435+0000 7f0cf5ffb640 10 client.20988 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.436+0000 7f0cf5ffb640 1 -- 172.21.15.70:0/509859425 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558c22e93b80 con 0x558c22f3a880 2024-06-04T05:19:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.436+0000 7f0cf5ffb640 10 client.20988 handle_command_reply: tid=0 2024-06-04T05:19:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.436+0000 7f0d171d4740 1 client.20988 shutdown 2024-06-04T05:19:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.437+0000 7f0d171d4740 1 -- 172.21.15.70:0/509859425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f1f820 msgr2=0x558c22f27a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.437+0000 7f0d171d4740 1 --2- 172.21.15.70:0/509859425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f1f820 0x558c22f27a40 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f0d0800bf60 tx=0x7f0d08011440 comp rx=0 tx=0).stop 2024-06-04T05:19:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.437+0000 7f0d171d4740 1 -- 172.21.15.70:0/509859425 shutdown_connections 2024-06-04T05:19:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.437+0000 7f0d171d4740 1 --2- 172.21.15.70:0/509859425 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558c22f3a880 0x558c22f3cc70 crc :-1 s=READY pgs=459 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.437+0000 7f0d171d4740 1 --2- 172.21.15.70:0/509859425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f20140 0x558c22f27f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.437+0000 7f0cf5ffb640 0 client.20988 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.437+0000 7f0d171d4740 1 --2- 172.21.15.70:0/509859425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f2aa00 0x558c22f284c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.437+0000 7f0d171d4740 1 --2- 172.21.15.70:0/509859425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f1f820 0x558c22f27a40 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.437+0000 7f0d171d4740 1 -- 172.21.15.70:0/509859425 >> 172.21.15.70:0/509859425 conn(0x558c22f1bd30 msgr2=0x558c22f1afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.437+0000 7f0d171d4740 1 -- 172.21.15.70:0/509859425 shutdown_connections 2024-06-04T05:19:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.437+0000 7f0d171d4740 1 -- 172.21.15.70:0/509859425 wait complete. 2024-06-04T05:19:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.437+0000 7f0d171d4740 20 client.20988 trim_cache size 0 max 16384 2024-06-04T05:19:00.452 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:19:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.453+0000 7f0d171d4740 1 Processor -- start 2024-06-04T05:19:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.453+0000 7f0d171d4740 1 -- start start 2024-06-04T05:19:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.453+0000 7f0d171d4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f1f820 0x558c22f1afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.453+0000 7f0d171d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f20140 0x558c22f282e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.453+0000 7f0d171d4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f2aa00 0x558c22f42210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.453+0000 7f0d171d4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0c1b0880 con 0x558c22f1f820 2024-06-04T05:19:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.453+0000 7f0d171d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0c1b2420 con 0x558c22f20140 2024-06-04T05:19:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.453+0000 7f0d171d4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0c1b0fb0 con 0x558c22f2aa00 2024-06-04T05:19:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.453+0000 7f0d12259640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f1f820 0x558c22f1afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.453+0000 7f0d12259640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f1f820 0x558c22f1afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47808/0 (socket says 172.21.15.70:47808) 2024-06-04T05:19:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.453+0000 7f0d12259640 1 -- 172.21.15.70:0/3891373793 learned_addr learned my addr 172.21.15.70:0/3891373793 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.453+0000 7f0d12a5a640 1 --2- 172.21.15.70:0/3891373793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f2aa00 0x558c22f42210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.453+0000 7f0d11a58640 1 --2- 172.21.15.70:0/3891373793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f20140 0x558c22f282e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.453+0000 7f0d12259640 1 -- 172.21.15.70:0/3891373793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f2aa00 msgr2=0x558c22f42210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.453+0000 7f0d12259640 1 --2- 172.21.15.70:0/3891373793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f2aa00 0x558c22f42210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.454+0000 7f0d12259640 1 -- 172.21.15.70:0/3891373793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f20140 msgr2=0x558c22f282e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.454+0000 7f0d12259640 1 --2- 172.21.15.70:0/3891373793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f20140 0x558c22f282e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.454+0000 7f0d12259640 1 -- 172.21.15.70:0/3891373793 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c22f269f0 con 0x558c22f1f820 2024-06-04T05:19:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.454+0000 7f0d12259640 1 --2- 172.21.15.70:0/3891373793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f1f820 0x558c22f1afb0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f0d08004750 tx=0x7f0d0800b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=565971b73b0d24b7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:00.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.454+0000 7f0cf7fff640 1 -- 172.21.15.70:0/3891373793 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08015030 con 0x558c22f1f820 2024-06-04T05:19:00.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.454+0000 7f0cf7fff640 1 -- 172.21.15.70:0/3891373793 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0d0800b3e0 con 0x558c22f1f820 2024-06-04T05:19:00.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.454+0000 7f0cf7fff640 1 -- 172.21.15.70:0/3891373793 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d080052b0 con 0x558c22f1f820 2024-06-04T05:19:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 1 -- 172.21.15.70:0/3891373793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f1f820 msgr2=0x558c22f1afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 1 --2- 172.21.15.70:0/3891373793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f1f820 0x558c22f1afb0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f0d08004750 tx=0x7f0d0800b770 comp rx=0 tx=0).stop 2024-06-04T05:19:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 1 -- 172.21.15.70:0/3891373793 shutdown_connections 2024-06-04T05:19:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 1 --2- 172.21.15.70:0/3891373793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f20140 0x558c22f282e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 1 --2- 172.21.15.70:0/3891373793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f2aa00 0x558c22f42210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 1 --2- 172.21.15.70:0/3891373793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f1f820 0x558c22f1afb0 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 1 -- 172.21.15.70:0/3891373793 >> 172.21.15.70:0/3891373793 conn(0x558c22f1bd30 msgr2=0x558c22f1c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 1 -- 172.21.15.70:0/3891373793 shutdown_connections 2024-06-04T05:19:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 1 -- 172.21.15.70:0/3891373793 wait complete. 2024-06-04T05:19:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 1 -- start start 2024-06-04T05:19:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 1 Processor -- start 2024-06-04T05:19:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f1f820 0x558c22f32e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f20140 0x558c22f2e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f2aa00 0x558c22f2e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0c1b1d50 con 0x558c22f2aa00 2024-06-04T05:19:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0c1b1680 con 0x558c22f1f820 2024-06-04T05:19:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0c1ae880 con 0x558c22f20140 2024-06-04T05:19:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.455+0000 7f0d171d4740 10 client.0 _ll_register_callbacks cb 0x7f0d0800bab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0d12a5a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f2aa00 0x558c22f2e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0d12a5a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f2aa00 0x558c22f2e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47816/0 (socket says 172.21.15.70:47816) 2024-06-04T05:19:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0d12a5a640 1 -- 172.21.15.70:0/659263965 learned_addr learned my addr 172.21.15.70:0/659263965 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0d12259640 1 --2- 172.21.15.70:0/659263965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f1f820 0x558c22f32e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0d11a58640 1 --2- 172.21.15.70:0/659263965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f20140 0x558c22f2e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0d12a5a640 1 -- 172.21.15.70:0/659263965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f20140 msgr2=0x558c22f2e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0d12a5a640 1 --2- 172.21.15.70:0/659263965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f20140 0x558c22f2e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0d12a5a640 1 -- 172.21.15.70:0/659263965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f1f820 msgr2=0x558c22f32e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0d12a5a640 1 --2- 172.21.15.70:0/659263965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f1f820 0x558c22f32e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0d12a5a640 1 -- 172.21.15.70:0/659263965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c22f3cfe0 con 0x558c22f2aa00 2024-06-04T05:19:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0d12a5a640 1 --2- 172.21.15.70:0/659263965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f2aa00 0x558c22f2e730 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f0d040127a0 tx=0x7f0d04004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc64b1907c1df3f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0cf5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0cf5ffb640 1 -- 172.21.15.70:0/659263965 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d04021dd0 con 0x558c22f2aa00 2024-06-04T05:19:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0cf5ffb640 1 -- 172.21.15.70:0/659263965 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f0d0401f450 con 0x558c22f2aa00 2024-06-04T05:19:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0d171d4740 1 -- 172.21.15.70:0/659263965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d0800b3e0 con 0x558c22f2aa00 2024-06-04T05:19:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0cf5ffb640 1 -- 172.21.15.70:0/659263965 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f0d0401f650 con 0x558c22f2aa00 2024-06-04T05:19:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0d171d4740 10 client.20994 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.456+0000 7f0d171d4740 1 -- 172.21.15.70:0/659263965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x558c22fa4e40 con 0x558c22f2aa00 2024-06-04T05:19:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.457+0000 7f0cf5ffb640 1 -- 172.21.15.70:0/659263965 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0d0800b3e0 con 0x558c22f2aa00 2024-06-04T05:19:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.457+0000 7f0cf5ffb640 1 -- 172.21.15.70:0/659263965 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f0d04028070 con 0x558c22f2aa00 2024-06-04T05:19:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.457+0000 7f0d171d4740 10 client.20994 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.457+0000 7f0d171d4740 10 client.20994 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:19:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.457+0000 7f0d171d4740 20 client.20994 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.457+0000 7f0d171d4740 1 --2- 172.21.15.70:0/659263965 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558c22fa5640 0x558c22fa7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.457+0000 7f0d171d4740 4 client.20994 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:19:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.457+0000 7f0d171d4740 1 -- 172.21.15.70:0/659263965 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558c22e8d360 con 0x558c22fa5640 2024-06-04T05:19:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.457+0000 7f0d12259640 1 --2- 172.21.15.70:0/659263965 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558c22fa5640 0x558c22fa7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.457+0000 7f0d12259640 1 --2- 172.21.15.70:0/659263965 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558c22fa5640 0x558c22fa7a60 crc :-1 s=READY pgs=460 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27fa0093fc909316 server_cookie=7bf1799365056684 in_seq=0 out_seq=0 2024-06-04T05:19:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.457+0000 7f0cf5ffb640 10 client.20994 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.493+0000 7f0cf5ffb640 1 -- 172.21.15.70:0/659263965 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x558c22e8d360 con 0x558c22fa5640 2024-06-04T05:19:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.493+0000 7f0cf5ffb640 10 client.20994 handle_command_reply: tid=0 2024-06-04T05:19:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.493+0000 7f0d171d4740 1 client.20994 shutdown 2024-06-04T05:19:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.493+0000 7f0d171d4740 1 -- 172.21.15.70:0/659263965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f2aa00 msgr2=0x558c22f2e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.493+0000 7f0d171d4740 1 --2- 172.21.15.70:0/659263965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f2aa00 0x558c22f2e730 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f0d040127a0 tx=0x7f0d04004ab0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.493+0000 7f0d171d4740 1 -- 172.21.15.70:0/659263965 shutdown_connections 2024-06-04T05:19:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.493+0000 7f0d171d4740 1 --2- 172.21.15.70:0/659263965 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558c22fa5640 0x558c22fa7a60 crc :-1 s=READY pgs=460 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.493+0000 7f0d171d4740 1 --2- 172.21.15.70:0/659263965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c22f2aa00 0x558c22f2e730 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.493+0000 7f0d171d4740 1 --2- 172.21.15.70:0/659263965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558c22f20140 0x558c22f2e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.493+0000 7f0cf5ffb640 0 client.20994 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.493+0000 7f0d171d4740 1 --2- 172.21.15.70:0/659263965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c22f1f820 0x558c22f32e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.493+0000 7f0d171d4740 1 -- 172.21.15.70:0/659263965 >> 172.21.15.70:0/659263965 conn(0x558c22f1bd30 msgr2=0x558c22f1afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.494+0000 7f0d171d4740 1 -- 172.21.15.70:0/659263965 shutdown_connections 2024-06-04T05:19:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.494+0000 7f0d171d4740 1 -- 172.21.15.70:0/659263965 wait complete. 2024-06-04T05:19:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.494+0000 7f0d171d4740 20 client.20994 trim_cache size 0 max 16384 2024-06-04T05:19:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:19:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:19:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:19:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.494+0000 7f0d144bd640 1 -- 172.21.15.70:0/3149998719 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0cf80b2a40 msgr2=0x7f0cf80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.494+0000 7f0d144bd640 1 --2- 172.21.15.70:0/3149998719 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0cf80b2a40 0x7f0cf80b4f00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0d04005c40 tx=0x7f0d04009510 comp rx=0 tx=0).stop 2024-06-04T05:19:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.494+0000 7f0d144bd640 1 -- 172.21.15.70:0/3149998719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d0c101910 msgr2=0x7f0d0c102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.494+0000 7f0d144bd640 1 --2- 172.21.15.70:0/3149998719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d0c101910 0x7f0d0c102670 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc005940 tx=0x7f0cfc0042f0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.494+0000 7f0d144bd640 1 -- 172.21.15.70:0/3149998719 shutdown_connections 2024-06-04T05:19:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.494+0000 7f0d144bd640 1 --2- 172.21.15.70:0/3149998719 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0cf80b2a40 0x7f0cf80b4f00 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.494+0000 7f0d144bd640 1 --2- 172.21.15.70:0/3149998719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d0c107a30 0x7f0d0c102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.494+0000 7f0d144bd640 1 --2- 172.21.15.70:0/3149998719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d0c101910 0x7f0d0c102670 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.494+0000 7f0d144bd640 1 --2- 172.21.15.70:0/3149998719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d0c100f50 0x7f0d0c102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.494+0000 7f0d144bd640 1 -- 172.21.15.70:0/3149998719 >> 172.21.15.70:0/3149998719 conn(0x7f0d0c077ee0 msgr2=0x7f0d0c0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.494+0000 7f0d144bd640 1 -- 172.21.15.70:0/3149998719 shutdown_connections 2024-06-04T05:19:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:00.494+0000 7f0d144bd640 1 -- 172.21.15.70:0/3149998719 wait complete. 2024-06-04T05:19:00.826 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 6.94 MB/sec execute 352 sec latency 19320.178 ms 2024-06-04T05:19:01.149 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:19:01.627 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:19:01.826 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 6.92 MB/sec execute 353 sec latency 20320.237 ms 2024-06-04T05:19:01.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:01 smithi070 ceph-mon[32631]: pgmap v686: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 129 KiB/s wr, 36 op/s 2024-06-04T05:19:02.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:01 smithi112 ceph-mon[36453]: pgmap v686: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 129 KiB/s wr, 36 op/s 2024-06-04T05:19:02.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:01 smithi098 ceph-mon[35569]: pgmap v686: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 129 KiB/s wr, 36 op/s 2024-06-04T05:19:02.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.448+0000 7f7cff9a4700 1 -- 172.21.15.70:0/254243060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cf8101ee0 msgr2=0x7f7cf8102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:02.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.448+0000 7f7cff9a4700 1 --2- 172.21.15.70:0/254243060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cf8101ee0 0x7f7cf8102360 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009990 tx=0x7f7ce8009ca0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.448+0000 7f7cff9a4700 1 -- 172.21.15.70:0/254243060 shutdown_connections 2024-06-04T05:19:02.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.448+0000 7f7cff9a4700 1 --2- 172.21.15.70:0/254243060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cf81028a0 0x7f7cf8110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.448+0000 7f7cff9a4700 1 --2- 172.21.15.70:0/254243060 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7cf8108600 0x7f7cf81089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.448+0000 7f7cff9a4700 1 --2- 172.21.15.70:0/254243060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cf8101ee0 0x7f7cf8102360 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.448+0000 7f7cff9a4700 1 -- 172.21.15.70:0/254243060 >> 172.21.15.70:0/254243060 conn(0x7f7cf8077760 msgr2=0x7f7cf8077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:02.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.448+0000 7f7cff9a4700 1 -- 172.21.15.70:0/254243060 shutdown_connections 2024-06-04T05:19:02.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.448+0000 7f7cff9a4700 1 -- 172.21.15.70:0/254243060 wait complete. 2024-06-04T05:19:02.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cff9a4700 1 Processor -- start 2024-06-04T05:19:02.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cff9a4700 1 -- start start 2024-06-04T05:19:02.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cff9a4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cf8101ee0 0x7f7cf819bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cff9a4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cf81028a0 0x7f7cf819c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cff9a4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7cf8108600 0x7f7cf81a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cff9a4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf8113600 con 0x7f7cf8101ee0 2024-06-04T05:19:02.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cff9a4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf8113480 con 0x7f7cf81028a0 2024-06-04T05:19:02.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cff9a4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf8113780 con 0x7f7cf8108600 2024-06-04T05:19:02.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cfd740700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cf8101ee0 0x7f7cf819bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:02.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cfd740700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cf8101ee0 0x7f7cf819bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47822/0 (socket says 172.21.15.70:47822) 2024-06-04T05:19:02.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cfd740700 1 -- 172.21.15.70:0/3525914565 learned_addr learned my addr 172.21.15.70:0/3525914565 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:02.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cfcf3f700 1 --2- 172.21.15.70:0/3525914565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cf81028a0 0x7f7cf819c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:02.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cfdf41700 1 --2- 172.21.15.70:0/3525914565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7cf8108600 0x7f7cf81a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:02.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cfd740700 1 -- 172.21.15.70:0/3525914565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7cf8108600 msgr2=0x7f7cf81a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:02.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cfd740700 1 --2- 172.21.15.70:0/3525914565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7cf8108600 0x7f7cf81a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cfd740700 1 -- 172.21.15.70:0/3525914565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cf81028a0 msgr2=0x7f7cf819c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:02.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cfd740700 1 --2- 172.21.15.70:0/3525914565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cf81028a0 0x7f7cf819c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.449+0000 7f7cfd740700 1 -- 172.21.15.70:0/3525914565 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf81a0da0 con 0x7f7cf8101ee0 2024-06-04T05:19:02.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.450+0000 7f7cfd740700 1 --2- 172.21.15.70:0/3525914565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cf8101ee0 0x7f7cf819bd90 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f7cf400d980 tx=0x7f7cf400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:02.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.450+0000 7f7cee7fc700 1 -- 172.21.15.70:0/3525914565 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4017070 con 0x7f7cf8101ee0 2024-06-04T05:19:02.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.450+0000 7f7cee7fc700 1 -- 172.21.15.70:0/3525914565 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cf400fd20 con 0x7f7cf8101ee0 2024-06-04T05:19:02.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.450+0000 7f7cff9a4700 1 -- 172.21.15.70:0/3525914565 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf81a1090 con 0x7f7cf8101ee0 2024-06-04T05:19:02.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.450+0000 7f7cee7fc700 1 -- 172.21.15.70:0/3525914565 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf400e3b0 con 0x7f7cf8101ee0 2024-06-04T05:19:02.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.450+0000 7f7cff9a4700 1 -- 172.21.15.70:0/3525914565 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf81a8910 con 0x7f7cf8101ee0 2024-06-04T05:19:02.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.452+0000 7f7cee7fc700 1 -- 172.21.15.70:0/3525914565 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7cf400f890 con 0x7f7cf8101ee0 2024-06-04T05:19:02.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.453+0000 7f7cee7fc700 1 --2- 172.21.15.70:0/3525914565 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ce40b29f0 0x7f7ce40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.453+0000 7f7cfcf3f700 1 --2- 172.21.15.70:0/3525914565 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ce40b29f0 0x7f7ce40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:02.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.453+0000 7f7cee7fc700 1 -- 172.21.15.70:0/3525914565 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f7cf40e20d0 con 0x7f7cf8101ee0 2024-06-04T05:19:02.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.453+0000 7f7d0a239b80 1 Processor -- start 2024-06-04T05:19:02.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7cfcf3f700 1 --2- 172.21.15.70:0/3525914565 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ce40b29f0 0x7f7ce40b4eb0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800b5c0 tx=0x7f7ce8005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:02.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7d0a239b80 1 -- start start 2024-06-04T05:19:02.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7d0a239b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c48860 0x563ef6c48c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7d0a239b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c49210 0x563ef6c4c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7d0a239b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c4cf60 0x563ef6c4d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7d0a239b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563ef6ba7c90 con 0x563ef6c4cf60 2024-06-04T05:19:02.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7d0a239b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563ef6b97830 con 0x563ef6c48860 2024-06-04T05:19:02.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7d0a239b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563ef6b9b870 con 0x563ef6c49210 2024-06-04T05:19:02.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7cfdf41700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c4cf60 0x563ef6c4d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:02.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7cfdf41700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c4cf60 0x563ef6c4d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47838/0 (socket says 172.21.15.70:47838) 2024-06-04T05:19:02.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7cfdf41700 1 -- 172.21.15.70:0/2135413178 learned_addr learned my addr 172.21.15.70:0/2135413178 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:02.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7cfcf3f700 1 --2- 172.21.15.70:0/2135413178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c49210 0x563ef6c4c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:02.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7cfd740700 1 --2- 172.21.15.70:0/2135413178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c48860 0x563ef6c48c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:02.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7cfdf41700 1 -- 172.21.15.70:0/2135413178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c49210 msgr2=0x563ef6c4c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:02.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7cfdf41700 1 --2- 172.21.15.70:0/2135413178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c49210 0x563ef6c4c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7cfdf41700 1 -- 172.21.15.70:0/2135413178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c48860 msgr2=0x563ef6c48c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:02.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7cfdf41700 1 --2- 172.21.15.70:0/2135413178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c48860 0x563ef6c48c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7cfdf41700 1 -- 172.21.15.70:0/2135413178 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ef6ae78d0 con 0x563ef6c4cf60 2024-06-04T05:19:02.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.454+0000 7f7cfdf41700 1 --2- 172.21.15.70:0/2135413178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c4cf60 0x563ef6c4d3a0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00098f0 tx=0x7f7cf0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e66c110de7fe2c0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:02.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.455+0000 7f7ce37fe700 1 -- 172.21.15.70:0/2135413178 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0004bb0 con 0x563ef6c4cf60 2024-06-04T05:19:02.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.455+0000 7f7ce37fe700 1 -- 172.21.15.70:0/2135413178 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cf0004d50 con 0x563ef6c4cf60 2024-06-04T05:19:02.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.455+0000 7f7ce37fe700 1 -- 172.21.15.70:0/2135413178 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0007d30 con 0x563ef6c4cf60 2024-06-04T05:19:02.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.455+0000 7f7d0a239b80 1 -- 172.21.15.70:0/2135413178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c4cf60 msgr2=0x563ef6c4d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:02.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.455+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/2135413178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c4cf60 0x563ef6c4d3a0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00098f0 tx=0x7f7cf0009c00 comp rx=0 tx=0).stop 2024-06-04T05:19:02.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.455+0000 7f7d0a239b80 1 -- 172.21.15.70:0/2135413178 shutdown_connections 2024-06-04T05:19:02.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.455+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/2135413178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c4cf60 0x563ef6c4d3a0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.455+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/2135413178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c48860 0x563ef6c48c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.455+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/2135413178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c49210 0x563ef6c4c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.455+0000 7f7d0a239b80 1 -- 172.21.15.70:0/2135413178 >> 172.21.15.70:0/2135413178 conn(0x563ef6b76080 msgr2=0x563ef6c42e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:02.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.455+0000 7f7d0a239b80 1 -- 172.21.15.70:0/2135413178 shutdown_connections 2024-06-04T05:19:02.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.455+0000 7f7d0a239b80 1 -- 172.21.15.70:0/2135413178 wait complete. 2024-06-04T05:19:02.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.456+0000 7f7d0a239b80 1 -- start start 2024-06-04T05:19:02.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.456+0000 7f7d0a239b80 1 Processor -- start 2024-06-04T05:19:02.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.456+0000 7f7d0a239b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c48860 0x563ef6c543e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.456+0000 7f7d0a239b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c49210 0x563ef6c516e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.456+0000 7f7d0a239b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c4cf60 0x563ef6c51c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.456+0000 7f7d0a239b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563ef6a9dfa0 con 0x563ef6c49210 2024-06-04T05:19:02.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.456+0000 7f7d0a239b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563ef6b397c0 con 0x563ef6c48860 2024-06-04T05:19:02.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.456+0000 7f7d0a239b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563ef6c55120 con 0x563ef6c4cf60 2024-06-04T05:19:02.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.456+0000 7f7d0a239b80 10 client.0 _ll_register_callbacks cb 0x563ef6b32920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:02.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.456+0000 7f7cfcf3f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c49210 0x563ef6c516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:02.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.456+0000 7f7cfcf3f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c49210 0x563ef6c516e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47850/0 (socket says 172.21.15.70:47850) 2024-06-04T05:19:02.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.456+0000 7f7cfcf3f700 1 -- 172.21.15.70:0/2439752855 learned_addr learned my addr 172.21.15.70:0/2439752855 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:02.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.456+0000 7f7cfd740700 1 --2- 172.21.15.70:0/2439752855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c48860 0x563ef6c543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:02.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.457+0000 7f7cfdf41700 1 --2- 172.21.15.70:0/2439752855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c4cf60 0x563ef6c51c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:02.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.457+0000 7f7cfcf3f700 1 -- 172.21.15.70:0/2439752855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c4cf60 msgr2=0x563ef6c51c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:02.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.457+0000 7f7cfcf3f700 1 --2- 172.21.15.70:0/2439752855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c4cf60 0x563ef6c51c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.457+0000 7f7cfcf3f700 1 -- 172.21.15.70:0/2439752855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c48860 msgr2=0x563ef6c543e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:02.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.457+0000 7f7cfcf3f700 1 --2- 172.21.15.70:0/2439752855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c48860 0x563ef6c543e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.457+0000 7f7cfcf3f700 1 -- 172.21.15.70:0/2439752855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ef6b36690 con 0x563ef6c49210 2024-06-04T05:19:02.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.457+0000 7f7cfcf3f700 1 --2- 172.21.15.70:0/2439752855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c49210 0x563ef6c516e0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009990 tx=0x7f7ce8009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b775d6c6551e177 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:02.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.457+0000 7f7ce0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:02.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.457+0000 7f7ce0ff9700 1 -- 172.21.15.70:0/2439752855 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce800f460 con 0x563ef6c49210 2024-06-04T05:19:02.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.457+0000 7f7ce0ff9700 1 -- 172.21.15.70:0/2439752855 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f7ce800f600 con 0x563ef6c49210 2024-06-04T05:19:02.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.457+0000 7f7d0a239b80 1 -- 172.21.15.70:0/2439752855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cf0004d50 con 0x563ef6c49210 2024-06-04T05:19:02.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.457+0000 7f7ce0ff9700 1 -- 172.21.15.70:0/2439752855 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f7ce803c050 con 0x563ef6c49210 2024-06-04T05:19:02.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.457+0000 7f7d0a239b80 10 client.21006 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:02.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.457+0000 7f7d0a239b80 1 -- 172.21.15.70:0/2439752855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x563ef6c5f850 con 0x563ef6c49210 2024-06-04T05:19:02.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.457+0000 7f7ce0ff9700 1 -- 172.21.15.70:0/2439752855 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cf0004d50 con 0x563ef6c49210 2024-06-04T05:19:02.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.458+0000 7f7ce0ff9700 1 -- 172.21.15.70:0/2439752855 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f7ce8022510 con 0x563ef6c49210 2024-06-04T05:19:02.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.458+0000 7f7d0a239b80 10 client.21006 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:02.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.458+0000 7f7d0a239b80 10 client.21006 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:19:02.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.458+0000 7f7d0a239b80 20 client.21006 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:02.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.458+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/2439752855 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x563ef6c61e90 0x563ef6c64310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.458+0000 7f7d0a239b80 4 client.21006 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:19:02.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.458+0000 7f7d0a239b80 1 -- 172.21.15.70:0/2439752855 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ef6c149e0 con 0x563ef6c61e90 2024-06-04T05:19:02.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.458+0000 7f7cfd740700 1 --2- 172.21.15.70:0/2439752855 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x563ef6c61e90 0x563ef6c64310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:02.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.459+0000 7f7ce0ff9700 10 client.21006 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:19:02.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.458+0000 7f7cfd740700 1 --2- 172.21.15.70:0/2439752855 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x563ef6c61e90 0x563ef6c64310 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e9ea9e662ce3bd8f server_cookie=8b7b172cce72d2fa in_seq=0 out_seq=0 2024-06-04T05:19:02.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.459+0000 7f7ce0ff9700 1 -- 172.21.15.70:0/2439752855 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563ef6c149e0 con 0x563ef6c61e90 2024-06-04T05:19:02.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.459+0000 7f7ce0ff9700 10 client.21006 handle_command_reply: tid=0 2024-06-04T05:19:02.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.459+0000 7f7d0a239b80 1 client.21006 shutdown 2024-06-04T05:19:02.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.459+0000 7f7d0a239b80 1 -- 172.21.15.70:0/2439752855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c49210 msgr2=0x563ef6c516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:02.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.459+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/2439752855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c49210 0x563ef6c516e0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009990 tx=0x7f7ce8009ca0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.459+0000 7f7d0a239b80 1 -- 172.21.15.70:0/2439752855 shutdown_connections 2024-06-04T05:19:02.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.459+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/2439752855 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x563ef6c61e90 0x563ef6c64310 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.459+0000 7f7ce0ff9700 0 client.21006 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:19:02.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.459+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/2439752855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c4cf60 0x563ef6c51c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.459+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/2439752855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c48860 0x563ef6c543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.459+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/2439752855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c49210 0x563ef6c516e0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.459+0000 7f7d0a239b80 1 -- 172.21.15.70:0/2439752855 >> 172.21.15.70:0/2439752855 conn(0x563ef6b76080 msgr2=0x563ef6c4e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:02.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.460+0000 7f7d0a239b80 1 -- 172.21.15.70:0/2439752855 shutdown_connections 2024-06-04T05:19:02.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.460+0000 7f7d0a239b80 1 -- 172.21.15.70:0/2439752855 wait complete. 2024-06-04T05:19:02.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.460+0000 7f7d0a239b80 20 client.21006 trim_cache size 0 max 16384 2024-06-04T05:19:02.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.474+0000 7f7d0a239b80 1 Processor -- start 2024-06-04T05:19:02.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7d0a239b80 1 -- start start 2024-06-04T05:19:02.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7d0a239b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c48860 0x563ef6c50680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7d0a239b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c49210 0x563ef6c42eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7d0a239b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c4cf60 0x563ef6c69be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7d0a239b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf81b00b0 con 0x563ef6c48860 2024-06-04T05:19:02.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7d0a239b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf81b1c50 con 0x563ef6c49210 2024-06-04T05:19:02.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7d0a239b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf81b07e0 con 0x563ef6c4cf60 2024-06-04T05:19:02.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7cfd740700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c48860 0x563ef6c50680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:02.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7cfd740700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c48860 0x563ef6c50680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47852/0 (socket says 172.21.15.70:47852) 2024-06-04T05:19:02.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7cfd740700 1 -- 172.21.15.70:0/1646837872 learned_addr learned my addr 172.21.15.70:0/1646837872 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:02.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7cfd740700 1 -- 172.21.15.70:0/1646837872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c4cf60 msgr2=0x563ef6c69be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:02.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7cfd740700 1 --2- 172.21.15.70:0/1646837872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c4cf60 0x563ef6c69be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7cfd740700 1 -- 172.21.15.70:0/1646837872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c49210 msgr2=0x563ef6c42eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:02.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7cfd740700 1 --2- 172.21.15.70:0/1646837872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c49210 0x563ef6c42eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7cfd740700 1 -- 172.21.15.70:0/1646837872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ef6c514b0 con 0x563ef6c48860 2024-06-04T05:19:02.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7cfd740700 1 --2- 172.21.15.70:0/1646837872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c48860 0x563ef6c50680 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f7cf40a5400 tx=0x7f7cf4005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f0286efc1f44e6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:02.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7ce2ffd700 1 -- 172.21.15.70:0/1646837872 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf40c3070 con 0x563ef6c48860 2024-06-04T05:19:02.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7ce2ffd700 1 -- 172.21.15.70:0/1646837872 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cf40a9a80 con 0x563ef6c48860 2024-06-04T05:19:02.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.475+0000 7f7ce2ffd700 1 -- 172.21.15.70:0/1646837872 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4020040 con 0x563ef6c48860 2024-06-04T05:19:02.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.476+0000 7f7d0a239b80 1 -- 172.21.15.70:0/1646837872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c48860 msgr2=0x563ef6c50680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:02.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.476+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/1646837872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c48860 0x563ef6c50680 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f7cf40a5400 tx=0x7f7cf4005da0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.476+0000 7f7d0a239b80 1 -- 172.21.15.70:0/1646837872 shutdown_connections 2024-06-04T05:19:02.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.476+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/1646837872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c4cf60 0x563ef6c69be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.476+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/1646837872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c48860 0x563ef6c50680 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.476+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/1646837872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c49210 0x563ef6c42eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.476+0000 7f7d0a239b80 1 -- 172.21.15.70:0/1646837872 >> 172.21.15.70:0/1646837872 conn(0x563ef6b76080 msgr2=0x563ef6c427a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:02.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.476+0000 7f7d0a239b80 1 -- 172.21.15.70:0/1646837872 shutdown_connections 2024-06-04T05:19:02.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.476+0000 7f7d0a239b80 1 -- 172.21.15.70:0/1646837872 wait complete. 2024-06-04T05:19:02.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.476+0000 7f7d0a239b80 1 -- start start 2024-06-04T05:19:02.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.476+0000 7f7d0a239b80 1 Processor -- start 2024-06-04T05:19:02.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7d0a239b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c48860 0x563ef6c4a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7d0a239b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c49210 0x563ef6c4acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7d0a239b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c4cf60 0x563ef6c4b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7d0a239b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf81b1580 con 0x563ef6c49210 2024-06-04T05:19:02.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7d0a239b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf81b0eb0 con 0x563ef6c4cf60 2024-06-04T05:19:02.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7d0a239b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf81ae070 con 0x563ef6c48860 2024-06-04T05:19:02.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7d0a239b80 10 client.0 _ll_register_callbacks cb 0x563ef6b32920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:02.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7cfcf3f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c49210 0x563ef6c4acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:02.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7cfcf3f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c49210 0x563ef6c4acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47868/0 (socket says 172.21.15.70:47868) 2024-06-04T05:19:02.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7cfcf3f700 1 -- 172.21.15.70:0/1440394460 learned_addr learned my addr 172.21.15.70:0/1440394460 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:02.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7cfd740700 1 --2- 172.21.15.70:0/1440394460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c48860 0x563ef6c4a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:02.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7cfdf41700 1 --2- 172.21.15.70:0/1440394460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c4cf60 0x563ef6c4b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:02.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7cfcf3f700 1 -- 172.21.15.70:0/1440394460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c48860 msgr2=0x563ef6c4a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:02.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7cfcf3f700 1 --2- 172.21.15.70:0/1440394460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c48860 0x563ef6c4a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7cfcf3f700 1 -- 172.21.15.70:0/1440394460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c4cf60 msgr2=0x563ef6c4b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:02.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7cfcf3f700 1 --2- 172.21.15.70:0/1440394460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c4cf60 0x563ef6c4b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7cfcf3f700 1 -- 172.21.15.70:0/1440394460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ef6ccbc70 con 0x563ef6c49210 2024-06-04T05:19:02.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7cfdf41700 1 --2- 172.21.15.70:0/1440394460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c4cf60 0x563ef6c4b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:19:02.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7cfd740700 1 --2- 172.21.15.70:0/1440394460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c48860 0x563ef6c4a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:19:02.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7cfcf3f700 1 --2- 172.21.15.70:0/1440394460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c49210 0x563ef6c4acf0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80036a0 tx=0x7f7ce8005d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69fbf3e710a56abb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:02.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.477+0000 7f7ce0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:02.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.478+0000 7f7ce0ff9700 1 -- 172.21.15.70:0/1440394460 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80225f0 con 0x563ef6c49210 2024-06-04T05:19:02.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.478+0000 7f7ce0ff9700 1 -- 172.21.15.70:0/1440394460 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f7ce800f460 con 0x563ef6c49210 2024-06-04T05:19:02.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.478+0000 7f7d0a239b80 1 -- 172.21.15.70:0/1440394460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cf40a9a80 con 0x563ef6c49210 2024-06-04T05:19:02.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.478+0000 7f7ce0ff9700 1 -- 172.21.15.70:0/1440394460 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f7ce804e050 con 0x563ef6c49210 2024-06-04T05:19:02.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.478+0000 7f7d0a239b80 10 client.21012 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:02.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.478+0000 7f7d0a239b80 1 -- 172.21.15.70:0/1440394460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x563ef6cccf10 con 0x563ef6c49210 2024-06-04T05:19:02.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.478+0000 7f7ce0ff9700 1 -- 172.21.15.70:0/1440394460 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cf40a9a80 con 0x563ef6c49210 2024-06-04T05:19:02.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.478+0000 7f7ce0ff9700 1 -- 172.21.15.70:0/1440394460 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f7ce8040430 con 0x563ef6c49210 2024-06-04T05:19:02.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.478+0000 7f7d0a239b80 10 client.21012 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:02.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.478+0000 7f7d0a239b80 10 client.21012 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:19:02.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.478+0000 7f7d0a239b80 20 client.21012 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:02.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.478+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/1440394460 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x563ef6ccd580 0x563ef6ccf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:02.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.478+0000 7f7d0a239b80 4 client.21012 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:19:02.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.478+0000 7f7d0a239b80 1 -- 172.21.15.70:0/1440394460 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563ef6ba9be0 con 0x563ef6ccd580 2024-06-04T05:19:02.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.478+0000 7f7cfd740700 1 --2- 172.21.15.70:0/1440394460 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x563ef6ccd580 0x563ef6ccf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:02.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.479+0000 7f7ce0ff9700 10 client.21012 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:19:02.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.479+0000 7f7cfd740700 1 --2- 172.21.15.70:0/1440394460 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x563ef6ccd580 0x563ef6ccf9a0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=276dbeda0e6364c2 server_cookie=eb9f5f62a9e4bdcc in_seq=0 out_seq=0 2024-06-04T05:19:02.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.479+0000 7f7ce0ff9700 1 -- 172.21.15.70:0/1440394460 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563ef6ba9be0 con 0x563ef6ccd580 2024-06-04T05:19:02.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.479+0000 7f7ce0ff9700 10 client.21012 handle_command_reply: tid=0 2024-06-04T05:19:02.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.479+0000 7f7d0a239b80 1 client.21012 shutdown 2024-06-04T05:19:02.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.479+0000 7f7d0a239b80 1 -- 172.21.15.70:0/1440394460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c49210 msgr2=0x563ef6c4acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:02.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.479+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/1440394460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c49210 0x563ef6c4acf0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80036a0 tx=0x7f7ce8005d20 comp rx=0 tx=0).stop 2024-06-04T05:19:02.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.479+0000 7f7d0a239b80 1 -- 172.21.15.70:0/1440394460 shutdown_connections 2024-06-04T05:19:02.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.479+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/1440394460 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x563ef6ccd580 0x563ef6ccf9a0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.479+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/1440394460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563ef6c4cf60 0x563ef6c4b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.479+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/1440394460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563ef6c48860 0x563ef6c4a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.479+0000 7f7d0a239b80 1 --2- 172.21.15.70:0/1440394460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ef6c49210 0x563ef6c4acf0 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.479+0000 7f7d0a239b80 1 -- 172.21.15.70:0/1440394460 >> 172.21.15.70:0/1440394460 conn(0x563ef6b76080 msgr2=0x563ef6c4c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:02.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.479+0000 7f7ce0ff9700 0 client.21012 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:19:02.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.479+0000 7f7d0a239b80 1 -- 172.21.15.70:0/1440394460 shutdown_connections 2024-06-04T05:19:02.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.479+0000 7f7d0a239b80 1 -- 172.21.15.70:0/1440394460 wait complete. 2024-06-04T05:19:02.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.479+0000 7f7d0a239b80 20 client.21012 trim_cache size 0 max 16384 2024-06-04T05:19:02.488 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:19:02.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.480+0000 7f7cff9a4700 1 -- 172.21.15.70:0/3525914565 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ce40b29f0 msgr2=0x7f7ce40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:02.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.480+0000 7f7cff9a4700 1 --2- 172.21.15.70:0/3525914565 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ce40b29f0 0x7f7ce40b4eb0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800b5c0 tx=0x7f7ce8005960 comp rx=0 tx=0).stop 2024-06-04T05:19:02.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.480+0000 7f7cff9a4700 1 -- 172.21.15.70:0/3525914565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cf8101ee0 msgr2=0x7f7cf819bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:02.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.480+0000 7f7cff9a4700 1 --2- 172.21.15.70:0/3525914565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cf8101ee0 0x7f7cf819bd90 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f7cf400d980 tx=0x7f7cf400dc90 comp rx=0 tx=0).stop 2024-06-04T05:19:02.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.480+0000 7f7cff9a4700 1 -- 172.21.15.70:0/3525914565 shutdown_connections 2024-06-04T05:19:02.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.480+0000 7f7cff9a4700 1 --2- 172.21.15.70:0/3525914565 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7ce40b29f0 0x7f7ce40b4eb0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.480+0000 7f7cff9a4700 1 --2- 172.21.15.70:0/3525914565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7cf8108600 0x7f7cf81a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.480+0000 7f7cff9a4700 1 --2- 172.21.15.70:0/3525914565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cf8101ee0 0x7f7cf819bd90 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.480+0000 7f7cff9a4700 1 --2- 172.21.15.70:0/3525914565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cf81028a0 0x7f7cf819c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:02.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.480+0000 7f7cff9a4700 1 -- 172.21.15.70:0/3525914565 >> 172.21.15.70:0/3525914565 conn(0x7f7cf8077760 msgr2=0x7f7cf8100930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:02.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.480+0000 7f7cff9a4700 1 -- 172.21.15.70:0/3525914565 shutdown_connections 2024-06-04T05:19:02.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:02.480+0000 7f7cff9a4700 1 -- 172.21.15.70:0/3525914565 wait complete. 2024-06-04T05:19:02.826 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 6.90 MB/sec execute 354 sec latency 21320.273 ms 2024-06-04T05:19:03.826 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 6.88 MB/sec execute 355 sec latency 22320.401 ms 2024-06-04T05:19:04.014 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:19:04.038 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:03 smithi070 ceph-mon[32631]: pgmap v687: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 194 KiB/s wr, 62 op/s 2024-06-04T05:19:04.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:03 smithi112 ceph-mon[36453]: pgmap v687: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 194 KiB/s wr, 62 op/s 2024-06-04T05:19:04.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:03 smithi098 ceph-mon[35569]: pgmap v687: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 194 KiB/s wr, 62 op/s 2024-06-04T05:19:04.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.483+0000 7fd089b4a700 1 -- 172.21.15.70:0/1340726887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd084102430 msgr2=0x7fd0841028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:04.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.483+0000 7fd089b4a700 1 --2- 172.21.15.70:0/1340726887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd084102430 0x7fd0841028b0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fd078009990 tx=0x7fd078009ca0 comp rx=0 tx=0).stop 2024-06-04T05:19:04.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.483+0000 7fd089b4a700 1 -- 172.21.15.70:0/1340726887 shutdown_connections 2024-06-04T05:19:04.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.483+0000 7fd089b4a700 1 --2- 172.21.15.70:0/1340726887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd084102df0 0x7fd08410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:04.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.483+0000 7fd089b4a700 1 --2- 172.21.15.70:0/1340726887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd084108510 0x7fd0841088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:04.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.483+0000 7fd089b4a700 1 --2- 172.21.15.70:0/1340726887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd084102430 0x7fd0841028b0 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:04.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.483+0000 7fd089b4a700 1 -- 172.21.15.70:0/1340726887 >> 172.21.15.70:0/1340726887 conn(0x7fd0840793b0 msgr2=0x7fd0840797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:04.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.483+0000 7fd089b4a700 1 -- 172.21.15.70:0/1340726887 shutdown_connections 2024-06-04T05:19:04.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.483+0000 7fd089b4a700 1 -- 172.21.15.70:0/1340726887 wait complete. 2024-06-04T05:19:04.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.484+0000 7fd089b4a700 1 Processor -- start 2024-06-04T05:19:04.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.484+0000 7fd089b4a700 1 -- start start 2024-06-04T05:19:04.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.484+0000 7fd089b4a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd084102430 0x7fd08419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:04.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.484+0000 7fd089b4a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd084102df0 0x7fd08419fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:04.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.484+0000 7fd089b4a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd084108510 0x7fd084199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:04.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.484+0000 7fd089b4a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd084112490 con 0x7fd084102430 2024-06-04T05:19:04.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.484+0000 7fd089b4a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd084112310 con 0x7fd084102df0 2024-06-04T05:19:04.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.484+0000 7fd089b4a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd084112610 con 0x7fd084108510 2024-06-04T05:19:04.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.484+0000 7fd0837fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd084102430 0x7fd08419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:04.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.484+0000 7fd0837fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd084102430 0x7fd08419f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47890/0 (socket says 172.21.15.70:47890) 2024-06-04T05:19:04.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.484+0000 7fd0837fe700 1 -- 172.21.15.70:0/3541527595 learned_addr learned my addr 172.21.15.70:0/3541527595 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:04.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.484+0000 7fd082ffd700 1 --2- 172.21.15.70:0/3541527595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd084102df0 0x7fd08419fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:04.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.485+0000 7fd083fff700 1 --2- 172.21.15.70:0/3541527595 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd084108510 0x7fd084199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:04.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.485+0000 7fd0837fe700 1 -- 172.21.15.70:0/3541527595 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd084108510 msgr2=0x7fd084199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:04.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.485+0000 7fd0837fe700 1 --2- 172.21.15.70:0/3541527595 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd084108510 0x7fd084199900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:04.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.485+0000 7fd0837fe700 1 -- 172.21.15.70:0/3541527595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd084102df0 msgr2=0x7fd08419fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:04.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.485+0000 7fd0837fe700 1 --2- 172.21.15.70:0/3541527595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd084102df0 0x7fd08419fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:04.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.485+0000 7fd0837fe700 1 -- 172.21.15.70:0/3541527595 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd08419a060 con 0x7fd084102430 2024-06-04T05:19:04.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.485+0000 7fd083fff700 1 --2- 172.21.15.70:0/3541527595 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd084108510 0x7fd084199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:19:04.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.485+0000 7fd0837fe700 1 --2- 172.21.15.70:0/3541527595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd084102430 0x7fd08419f740 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fd07400e9c0 tx=0x7fd07400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:04.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.485+0000 7fd080ff9700 1 -- 172.21.15.70:0/3541527595 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07400cbd0 con 0x7fd084102430 2024-06-04T05:19:04.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.485+0000 7fd080ff9700 1 -- 172.21.15.70:0/3541527595 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd07400cd70 con 0x7fd084102430 2024-06-04T05:19:04.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.485+0000 7fd089b4a700 1 -- 172.21.15.70:0/3541527595 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd08419a350 con 0x7fd084102430 2024-06-04T05:19:04.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.485+0000 7fd080ff9700 1 -- 172.21.15.70:0/3541527595 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd074005540 con 0x7fd084102430 2024-06-04T05:19:04.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.486+0000 7fd089b4a700 1 -- 172.21.15.70:0/3541527595 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd084103fa0 con 0x7fd084102430 2024-06-04T05:19:04.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.487+0000 7fd089b4a700 1 -- 172.21.15.70:0/3541527595 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd08406a230 con 0x7fd084102430 2024-06-04T05:19:04.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.488+0000 7fd080ff9700 1 -- 172.21.15.70:0/3541527595 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fd074021440 con 0x7fd084102430 2024-06-04T05:19:04.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.488+0000 7fd080ff9700 1 --2- 172.21.15.70:0/3541527595 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd06c0b2a70 0x7fd06c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:04.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.488+0000 7fd082ffd700 1 --2- 172.21.15.70:0/3541527595 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd06c0b2a70 0x7fd06c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:04.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.488+0000 7fd080ff9700 1 -- 172.21.15.70:0/3541527595 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7fd0740e2830 con 0x7fd084102430 2024-06-04T05:19:04.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.489+0000 7fd082ffd700 1 --2- 172.21.15.70:0/3541527595 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd06c0b2a70 0x7fd06c0b4f30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd084079ad0 tx=0x7fd078005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:04.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.497+0000 7fd080ff9700 1 -- 172.21.15.70:0/3541527595 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd0740aa1e0 con 0x7fd084102430 2024-06-04T05:19:04.505 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:19:04.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.771+0000 7fd089b4a700 1 -- 172.21.15.70:0/3541527595 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd08419ae30 con 0x7fd084102430 2024-06-04T05:19:04.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.772+0000 7fd080ff9700 1 -- 172.21.15.70:0/3541527595 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+11932 (secure 0 0 0) 0x7fd0740a9ae0 con 0x7fd084102430 2024-06-04T05:19:04.775 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:19:04.775 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":59,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:17:20.010341+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:19:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.774+0000 7fd089b4a700 1 -- 172.21.15.70:0/3541527595 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd06c0b2a70 msgr2=0x7fd06c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.774+0000 7fd089b4a700 1 --2- 172.21.15.70:0/3541527595 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd06c0b2a70 0x7fd06c0b4f30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd084079ad0 tx=0x7fd078005fb0 comp rx=0 tx=0).stop 2024-06-04T05:19:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.774+0000 7fd089b4a700 1 -- 172.21.15.70:0/3541527595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd084102430 msgr2=0x7fd08419f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.774+0000 7fd089b4a700 1 --2- 172.21.15.70:0/3541527595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd084102430 0x7fd08419f740 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fd07400e9c0 tx=0x7fd07400ecd0 comp rx=0 tx=0).stop 2024-06-04T05:19:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.774+0000 7fd089b4a700 1 -- 172.21.15.70:0/3541527595 shutdown_connections 2024-06-04T05:19:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.774+0000 7fd089b4a700 1 --2- 172.21.15.70:0/3541527595 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd06c0b2a70 0x7fd06c0b4f30 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.774+0000 7fd089b4a700 1 --2- 172.21.15.70:0/3541527595 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd084108510 0x7fd084199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.774+0000 7fd089b4a700 1 --2- 172.21.15.70:0/3541527595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd084102430 0x7fd08419f740 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.774+0000 7fd089b4a700 1 --2- 172.21.15.70:0/3541527595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd084102df0 0x7fd08419fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.774+0000 7fd089b4a700 1 -- 172.21.15.70:0/3541527595 >> 172.21.15.70:0/3541527595 conn(0x7fd0840793b0 msgr2=0x7fd08406bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.774+0000 7fd089b4a700 1 -- 172.21.15.70:0/3541527595 shutdown_connections 2024-06-04T05:19:04.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:04.774+0000 7fd089b4a700 1 -- 172.21.15.70:0/3541527595 wait complete. 2024-06-04T05:19:04.778 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 59 2024-06-04T05:19:04.826 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 6.86 MB/sec execute 356 sec latency 23320.474 ms 2024-06-04T05:19:05.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 6.84 MB/sec execute 357 sec latency 24320.578 ms 2024-06-04T05:19:06.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:05 smithi070 ceph-mon[32631]: pgmap v688: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 72 KiB/s wr, 31 op/s 2024-06-04T05:19:06.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:05 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3541527595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:19:06.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:05 smithi112 ceph-mon[36453]: pgmap v688: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 72 KiB/s wr, 31 op/s 2024-06-04T05:19:06.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:05 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3541527595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:19:06.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:05 smithi098 ceph-mon[35569]: pgmap v688: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 72 KiB/s wr, 31 op/s 2024-06-04T05:19:06.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:05 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3541527595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:19:06.675 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id 549ed795 --release --await' 2024-06-04T05:19:06.676 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id 549ed795 --release --await 2024-06-04T05:19:06.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 6.82 MB/sec execute 358 sec latency 25320.627 ms 2024-06-04T05:19:06.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.927+0000 7f0d35932700 1 -- 172.21.15.70:0/1759405413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d30075770 msgr2=0x7f0d30075b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:06.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.927+0000 7f0d35932700 1 --2- 172.21.15.70:0/1759405413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d30075770 0x7f0d30075b50 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f0d20009990 tx=0x7f0d20009ca0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.927+0000 7f0d35932700 1 -- 172.21.15.70:0/1759405413 shutdown_connections 2024-06-04T05:19:06.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.927+0000 7f0d35932700 1 --2- 172.21.15.70:0/1759405413 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d30114740 0x7f0d30116b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.927+0000 7f0d35932700 1 --2- 172.21.15.70:0/1759405413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d30075770 0x7f0d30075b50 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.927+0000 7f0d35932700 1 --2- 172.21.15.70:0/1759405413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d30076090 0x7f0d30114200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.927+0000 7f0d35932700 1 -- 172.21.15.70:0/1759405413 >> 172.21.15.70:0/1759405413 conn(0x7f0d3007a6b0 msgr2=0x7f0d3007aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:06.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.927+0000 7f0d35932700 1 -- 172.21.15.70:0/1759405413 shutdown_connections 2024-06-04T05:19:06.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.927+0000 7f0d35932700 1 -- 172.21.15.70:0/1759405413 wait complete. 2024-06-04T05:19:06.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.928+0000 7f0d35932700 1 Processor -- start 2024-06-04T05:19:06.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.928+0000 7f0d35932700 1 -- start start 2024-06-04T05:19:06.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.928+0000 7f0d35932700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d30075770 0x7f0d301a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.928+0000 7f0d35932700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d30076090 0x7f0d301a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.928+0000 7f0d35932700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d30114740 0x7f0d301a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.928+0000 7f0d35932700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0d30119c00 con 0x7f0d30076090 2024-06-04T05:19:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.928+0000 7f0d35932700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0d30119a80 con 0x7f0d30114740 2024-06-04T05:19:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.928+0000 7f0d35932700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0d30119d80 con 0x7f0d30075770 2024-06-04T05:19:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.928+0000 7f0d2e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d30076090 0x7f0d301a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.928+0000 7f0d2e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d30076090 0x7f0d301a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33230/0 (socket says 172.21.15.70:33230) 2024-06-04T05:19:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.928+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/1395632365 learned_addr learned my addr 172.21.15.70:0/1395632365 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.929+0000 7f0d2effd700 1 --2- 172.21.15.70:0/1395632365 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d30075770 0x7f0d301a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.929+0000 7f0d2f7fe700 1 --2- 172.21.15.70:0/1395632365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d30114740 0x7f0d301a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.929+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/1395632365 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d30075770 msgr2=0x7f0d301a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.929+0000 7f0d2e7fc700 1 --2- 172.21.15.70:0/1395632365 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d30075770 0x7f0d301a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.929+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/1395632365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d30114740 msgr2=0x7f0d301a4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:06.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.929+0000 7f0d2e7fc700 1 --2- 172.21.15.70:0/1395632365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d30114740 0x7f0d301a4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.929+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/1395632365 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d301a5160 con 0x7f0d30076090 2024-06-04T05:19:06.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.929+0000 7f0d2e7fc700 1 --2- 172.21.15.70:0/1395632365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d30076090 0x7f0d301a0690 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f0d2400d950 tx=0x7f0d2400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:06.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.929+0000 7f0d34930700 1 -- 172.21.15.70:0/1395632365 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d24016070 con 0x7f0d30076090 2024-06-04T05:19:06.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.929+0000 7f0d34930700 1 -- 172.21.15.70:0/1395632365 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0d24009dd0 con 0x7f0d30076090 2024-06-04T05:19:06.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.929+0000 7f0d35932700 1 -- 172.21.15.70:0/1395632365 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d301a5450 con 0x7f0d30076090 2024-06-04T05:19:06.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.929+0000 7f0d34930700 1 -- 172.21.15.70:0/1395632365 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2400f3c0 con 0x7f0d30076090 2024-06-04T05:19:06.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.929+0000 7f0d35932700 1 -- 172.21.15.70:0/1395632365 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d301accd0 con 0x7f0d30076090 2024-06-04T05:19:06.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.932+0000 7f0d34930700 1 -- 172.21.15.70:0/1395632365 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f0d24019070 con 0x7f0d30076090 2024-06-04T05:19:06.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.933+0000 7f0d34930700 1 --2- 172.21.15.70:0/1395632365 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0d180b29f0 0x7f0d180b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.933+0000 7f0d2effd700 1 --2- 172.21.15.70:0/1395632365 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0d180b29f0 0x7f0d180b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.933+0000 7f0d34930700 1 -- 172.21.15.70:0/1395632365 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f0d240e22c0 con 0x7f0d30076090 2024-06-04T05:19:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.934+0000 7f0d401c7b80 1 Processor -- start 2024-06-04T05:19:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.934+0000 7f0d2effd700 1 --2- 172.21.15.70:0/1395632365 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0d180b29f0 0x7f0d180b4eb0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0d20006010 tx=0x7f0d2000b560 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.934+0000 7f0d401c7b80 1 -- start start 2024-06-04T05:19:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.934+0000 7f0d401c7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 0x55bb9a36ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.934+0000 7f0d401c7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36b210 0x55bb9a36e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.934+0000 7f0d401c7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36ef60 0x55bb9a36f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.934+0000 7f0d401c7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bb9a2c9c90 con 0x55bb9a36ef60 2024-06-04T05:19:06.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.934+0000 7f0d401c7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bb9a2b9830 con 0x55bb9a36a860 2024-06-04T05:19:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.934+0000 7f0d401c7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bb9a2bd870 con 0x55bb9a36b210 2024-06-04T05:19:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.934+0000 7f0d2effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 0x55bb9a36ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.934+0000 7f0d2effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 0x55bb9a36ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:60280/0 (socket says 172.21.15.70:60280) 2024-06-04T05:19:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.934+0000 7f0d2e7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36b210 0x55bb9a36e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.934+0000 7f0d2effd700 1 -- 172.21.15.70:0/1790225152 learned_addr learned my addr 172.21.15.70:0/1790225152 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.934+0000 7f0d2f7fe700 1 --2- 172.21.15.70:0/1790225152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36ef60 0x55bb9a36f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.935+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/1790225152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 msgr2=0x55bb9a36ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.935+0000 7f0d2e7fc700 1 --2- 172.21.15.70:0/1790225152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 0x55bb9a36ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.935+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/1790225152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36ef60 msgr2=0x55bb9a36f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.935+0000 7f0d2e7fc700 1 --2- 172.21.15.70:0/1790225152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36ef60 0x55bb9a36f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.935+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/1790225152 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb9a2098d0 con 0x55bb9a36b210 2024-06-04T05:19:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.935+0000 7f0d2effd700 1 --2- 172.21.15.70:0/1790225152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 0x55bb9a36ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:19:06.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.935+0000 7f0d2e7fc700 1 --2- 172.21.15.70:0/1790225152 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36b210 0x55bb9a36e940 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0d240df9e0 tx=0x7f0d240a5560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d876e5af69eaa1c2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:06.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.935+0000 7f0d15ffb700 1 -- 172.21.15.70:0/1790225152 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d24020070 con 0x55bb9a36b210 2024-06-04T05:19:06.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.935+0000 7f0d15ffb700 1 -- 172.21.15.70:0/1790225152 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0d240ad920 con 0x55bb9a36b210 2024-06-04T05:19:06.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.935+0000 7f0d15ffb700 1 -- 172.21.15.70:0/1790225152 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d240b1f40 con 0x55bb9a36b210 2024-06-04T05:19:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.936+0000 7f0d401c7b80 1 -- 172.21.15.70:0/1790225152 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36b210 msgr2=0x55bb9a36e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.936+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/1790225152 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36b210 0x55bb9a36e940 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0d240df9e0 tx=0x7f0d240a5560 comp rx=0 tx=0).stop 2024-06-04T05:19:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.936+0000 7f0d401c7b80 1 -- 172.21.15.70:0/1790225152 shutdown_connections 2024-06-04T05:19:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.936+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/1790225152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36ef60 0x55bb9a36f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.936+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/1790225152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 0x55bb9a36ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.936+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/1790225152 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36b210 0x55bb9a36e940 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.936+0000 7f0d401c7b80 1 -- 172.21.15.70:0/1790225152 >> 172.21.15.70:0/1790225152 conn(0x55bb9a298080 msgr2=0x55bb9a364e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.936+0000 7f0d401c7b80 1 -- 172.21.15.70:0/1790225152 shutdown_connections 2024-06-04T05:19:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.936+0000 7f0d401c7b80 1 -- 172.21.15.70:0/1790225152 wait complete. 2024-06-04T05:19:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.936+0000 7f0d401c7b80 1 -- start start 2024-06-04T05:19:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d401c7b80 1 Processor -- start 2024-06-04T05:19:06.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d401c7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 0x55bb9a3764d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d401c7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a3737d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d401c7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36ef60 0x55bb9a373d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d401c7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bb9a1bffa0 con 0x55bb9a36b210 2024-06-04T05:19:06.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d401c7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bb9a25b7c0 con 0x55bb9a36a860 2024-06-04T05:19:06.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d401c7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bb9a377120 con 0x55bb9a36ef60 2024-06-04T05:19:06.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d401c7b80 10 client.0 _ll_register_callbacks cb 0x55bb9a254920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:06.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d2e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a3737d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d2e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a3737d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33254/0 (socket says 172.21.15.70:33254) 2024-06-04T05:19:06.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/3102109923 learned_addr learned my addr 172.21.15.70:0/3102109923 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:06.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d2effd700 1 --2- 172.21.15.70:0/3102109923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 0x55bb9a3764d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d2f7fe700 1 --2- 172.21.15.70:0/3102109923 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36ef60 0x55bb9a373d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/3102109923 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36ef60 msgr2=0x55bb9a373d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:06.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d2e7fc700 1 --2- 172.21.15.70:0/3102109923 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36ef60 0x55bb9a373d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/3102109923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 msgr2=0x55bb9a3764d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:06.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d2e7fc700 1 --2- 172.21.15.70:0/3102109923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 0x55bb9a3764d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.937+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/3102109923 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb9a258690 con 0x55bb9a36b210 2024-06-04T05:19:06.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.938+0000 7f0d2e7fc700 1 --2- 172.21.15.70:0/3102109923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a3737d0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f0d240dfa40 tx=0x7f0d240109e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c1fc3d20263c143 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:06.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.938+0000 7f0d0effd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:06.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.938+0000 7f0d0effd700 1 -- 172.21.15.70:0/3102109923 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d240c2070 con 0x55bb9a36b210 2024-06-04T05:19:06.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.938+0000 7f0d0effd700 1 -- 172.21.15.70:0/3102109923 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f0d240b2140 con 0x55bb9a36b210 2024-06-04T05:19:06.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.938+0000 7f0d401c7b80 1 -- 172.21.15.70:0/3102109923 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d240ad920 con 0x55bb9a36b210 2024-06-04T05:19:06.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.938+0000 7f0d0effd700 1 -- 172.21.15.70:0/3102109923 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f0d240c3e60 con 0x55bb9a36b210 2024-06-04T05:19:06.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.938+0000 7f0d401c7b80 10 client.21030 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:06.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.938+0000 7f0d401c7b80 1 -- 172.21.15.70:0/3102109923 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55bb9a381950 con 0x55bb9a36b210 2024-06-04T05:19:06.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.938+0000 7f0d0effd700 1 -- 172.21.15.70:0/3102109923 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0d240ad920 con 0x55bb9a36b210 2024-06-04T05:19:06.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.939+0000 7f0d0effd700 1 -- 172.21.15.70:0/3102109923 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f0d240a5000 con 0x55bb9a36b210 2024-06-04T05:19:06.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.939+0000 7f0d401c7b80 10 client.21030 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:06.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.939+0000 7f0d401c7b80 10 client.21030 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:19:06.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.939+0000 7f0d401c7b80 20 client.21030 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:06.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.939+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/3102109923 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55bb9a383e20 0x55bb9a386210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.939+0000 7f0d401c7b80 4 client.21030 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:19:06.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.939+0000 7f0d401c7b80 1 -- 172.21.15.70:0/3102109923 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb9a3369e0 con 0x55bb9a383e20 2024-06-04T05:19:06.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.939+0000 7f0d2effd700 1 --2- 172.21.15.70:0/3102109923 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55bb9a383e20 0x55bb9a386210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.939+0000 7f0d0effd700 10 client.21030 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:19:06.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.939+0000 7f0d2effd700 1 --2- 172.21.15.70:0/3102109923 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55bb9a383e20 0x55bb9a386210 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ba8984b94aae16a7 server_cookie=4673d9bca5e9b4da in_seq=0 out_seq=0 2024-06-04T05:19:06.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.940+0000 7f0d0effd700 1 -- 172.21.15.70:0/3102109923 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bb9a3369e0 con 0x55bb9a383e20 2024-06-04T05:19:06.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.940+0000 7f0d0effd700 10 client.21030 handle_command_reply: tid=0 2024-06-04T05:19:06.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.940+0000 7f0d401c7b80 1 client.21030 shutdown 2024-06-04T05:19:06.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.940+0000 7f0d401c7b80 1 -- 172.21.15.70:0/3102109923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 msgr2=0x55bb9a3737d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:06.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.940+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/3102109923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a3737d0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f0d240dfa40 tx=0x7f0d240109e0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.941+0000 7f0d401c7b80 1 -- 172.21.15.70:0/3102109923 shutdown_connections 2024-06-04T05:19:06.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.941+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/3102109923 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55bb9a383e20 0x55bb9a386210 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.941+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/3102109923 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36ef60 0x55bb9a373d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.941+0000 7f0d0effd700 0 client.21030 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:19:06.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.941+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/3102109923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 0x55bb9a3764d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.941+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/3102109923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a3737d0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.941+0000 7f0d401c7b80 1 -- 172.21.15.70:0/3102109923 >> 172.21.15.70:0/3102109923 conn(0x55bb9a298080 msgr2=0x55bb9a370350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:06.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.941+0000 7f0d401c7b80 1 -- 172.21.15.70:0/3102109923 shutdown_connections 2024-06-04T05:19:06.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.941+0000 7f0d401c7b80 1 -- 172.21.15.70:0/3102109923 wait complete. 2024-06-04T05:19:06.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.941+0000 7f0d401c7b80 20 client.21030 trim_cache size 0 max 16384 2024-06-04T05:19:06.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.956+0000 7f0d401c7b80 1 Processor -- start 2024-06-04T05:19:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.956+0000 7f0d401c7b80 1 -- start start 2024-06-04T05:19:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d401c7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 0x55bb9a364de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d401c7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a365320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d401c7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36ef60 0x55bb9a369670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d401c7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0d301b44b0 con 0x55bb9a36b210 2024-06-04T05:19:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d401c7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0d301b6050 con 0x55bb9a36a860 2024-06-04T05:19:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d401c7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0d301b4be0 con 0x55bb9a36ef60 2024-06-04T05:19:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d2e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a365320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d2e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a365320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33264/0 (socket says 172.21.15.70:33264) 2024-06-04T05:19:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/733610636 learned_addr learned my addr 172.21.15.70:0/733610636 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d2effd700 1 --2- 172.21.15.70:0/733610636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 0x55bb9a364de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/733610636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36ef60 msgr2=0x55bb9a369670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:06.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d2f7fe700 1 --2- 172.21.15.70:0/733610636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36ef60 0x55bb9a369670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d2e7fc700 1 --2- 172.21.15.70:0/733610636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36ef60 0x55bb9a369670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/733610636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 msgr2=0x55bb9a364de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:06.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d2e7fc700 1 --2- 172.21.15.70:0/733610636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 0x55bb9a364de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/733610636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb9a374020 con 0x55bb9a36b210 2024-06-04T05:19:06.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d2effd700 1 --2- 172.21.15.70:0/733610636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 0x55bb9a364de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:19:06.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d2f7fe700 1 --2- 172.21.15.70:0/733610636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36ef60 0x55bb9a369670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:19:06.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.957+0000 7f0d2e7fc700 1 --2- 172.21.15.70:0/733610636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a365320 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f0d240e2290 tx=0x7f0d240108d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=911a022d40a44a09 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:06.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.958+0000 7f0d157fa700 1 -- 172.21.15.70:0/733610636 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d240c2070 con 0x55bb9a36b210 2024-06-04T05:19:06.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.958+0000 7f0d157fa700 1 -- 172.21.15.70:0/733610636 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0d240b23d0 con 0x55bb9a36b210 2024-06-04T05:19:06.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.958+0000 7f0d157fa700 1 -- 172.21.15.70:0/733610636 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d24020040 con 0x55bb9a36b210 2024-06-04T05:19:06.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.958+0000 7f0d401c7b80 1 -- 172.21.15.70:0/733610636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 msgr2=0x55bb9a365320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:06.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.958+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/733610636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a365320 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f0d240e2290 tx=0x7f0d240108d0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.958+0000 7f0d401c7b80 1 -- 172.21.15.70:0/733610636 shutdown_connections 2024-06-04T05:19:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.959+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/733610636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36ef60 0x55bb9a369670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.959+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/733610636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36a860 0x55bb9a364de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.959+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/733610636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a365320 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.959+0000 7f0d401c7b80 1 -- 172.21.15.70:0/733610636 >> 172.21.15.70:0/733610636 conn(0x55bb9a298080 msgr2=0x55bb9a3720c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.959+0000 7f0d401c7b80 1 -- 172.21.15.70:0/733610636 shutdown_connections 2024-06-04T05:19:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.959+0000 7f0d401c7b80 1 -- 172.21.15.70:0/733610636 wait complete. 2024-06-04T05:19:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.959+0000 7f0d401c7b80 1 -- start start 2024-06-04T05:19:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.959+0000 7f0d401c7b80 1 Processor -- start 2024-06-04T05:19:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.959+0000 7f0d401c7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36a860 0x55bb9a3e6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.959+0000 7f0d401c7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a3e7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.959+0000 7f0d401c7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36ef60 0x55bb9a388aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.959+0000 7f0d401c7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0d301b5980 con 0x55bb9a36b210 2024-06-04T05:19:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.959+0000 7f0d401c7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0d301b52b0 con 0x55bb9a36ef60 2024-06-04T05:19:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.959+0000 7f0d401c7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0d301b2470 con 0x55bb9a36a860 2024-06-04T05:19:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.959+0000 7f0d401c7b80 10 client.0 _ll_register_callbacks cb 0x55bb9a254920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d2e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a3e7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d2e7fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a3e7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33274/0 (socket says 172.21.15.70:33274) 2024-06-04T05:19:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d2effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36a860 0x55bb9a3e6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/2582255965 learned_addr learned my addr 172.21.15.70:0/2582255965 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d2f7fe700 1 --2- 172.21.15.70:0/2582255965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36ef60 0x55bb9a388aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/2582255965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36a860 msgr2=0x55bb9a3e6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d2e7fc700 1 --2- 172.21.15.70:0/2582255965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36a860 0x55bb9a3e6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/2582255965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36ef60 msgr2=0x55bb9a388aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d2e7fc700 1 --2- 172.21.15.70:0/2582255965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36ef60 0x55bb9a388aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d2e7fc700 1 -- 172.21.15.70:0/2582255965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb9a3891a0 con 0x55bb9a36b210 2024-06-04T05:19:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d2e7fc700 1 --2- 172.21.15.70:0/2582255965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a3e7200 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f0d240dff60 tx=0x7f0d240a8ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=703802f019b364ba server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d0effd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d0effd700 1 -- 172.21.15.70:0/2582255965 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d24020070 con 0x55bb9a36b210 2024-06-04T05:19:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d0effd700 1 -- 172.21.15.70:0/2582255965 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8571+0+0 (secure 0 0 0) 0x7f0d240ca510 con 0x55bb9a36b210 2024-06-04T05:19:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d401c7b80 1 -- 172.21.15.70:0/2582255965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d240b23d0 con 0x55bb9a36b210 2024-06-04T05:19:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d0effd700 1 -- 172.21.15.70:0/2582255965 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f0d240c2070 con 0x55bb9a36b210 2024-06-04T05:19:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d401c7b80 10 client.21036 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.960+0000 7f0d401c7b80 1 -- 172.21.15.70:0/2582255965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55bb9a3891a0 con 0x55bb9a36b210 2024-06-04T05:19:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.961+0000 7f0d0effd700 1 -- 172.21.15.70:0/2582255965 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0d240b23d0 con 0x55bb9a36b210 2024-06-04T05:19:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.961+0000 7f0d0effd700 1 -- 172.21.15.70:0/2582255965 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f0d2408fa90 con 0x55bb9a36b210 2024-06-04T05:19:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.961+0000 7f0d401c7b80 10 client.21036 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.961+0000 7f0d401c7b80 10 client.21036 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:19:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.961+0000 7f0d401c7b80 20 client.21036 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.961+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/2582255965 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55bb9a3ee6e0 0x55bb9a3f0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.961+0000 7f0d401c7b80 4 client.21036 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:19:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.961+0000 7f0d401c7b80 1 -- 172.21.15.70:0/2582255965 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bb9a2cbbe0 con 0x55bb9a3ee6e0 2024-06-04T05:19:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.961+0000 7f0d2effd700 1 --2- 172.21.15.70:0/2582255965 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55bb9a3ee6e0 0x55bb9a3f0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.962+0000 7f0d0effd700 10 client.21036 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:19:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.962+0000 7f0d2effd700 1 --2- 172.21.15.70:0/2582255965 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55bb9a3ee6e0 0x55bb9a3f0ad0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d5c22557233dbbe6 server_cookie=1ae90c8d88ca66a8 in_seq=0 out_seq=0 2024-06-04T05:19:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.962+0000 7f0d0effd700 1 -- 172.21.15.70:0/2582255965 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bb9a2cbbe0 con 0x55bb9a3ee6e0 2024-06-04T05:19:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.962+0000 7f0d0effd700 10 client.21036 handle_command_reply: tid=0 2024-06-04T05:19:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.962+0000 7f0d401c7b80 1 client.21036 shutdown 2024-06-04T05:19:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.962+0000 7f0d401c7b80 1 -- 172.21.15.70:0/2582255965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 msgr2=0x55bb9a3e7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.962+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/2582255965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a3e7200 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f0d240dff60 tx=0x7f0d240a8ec0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.962+0000 7f0d401c7b80 1 -- 172.21.15.70:0/2582255965 shutdown_connections 2024-06-04T05:19:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.962+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/2582255965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb9a36ef60 0x55bb9a388aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.962+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/2582255965 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x55bb9a3ee6e0 0x55bb9a3f0ad0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.962+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/2582255965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bb9a36a860 0x55bb9a3e6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.962+0000 7f0d401c7b80 1 --2- 172.21.15.70:0/2582255965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb9a36b210 0x55bb9a3e7200 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.962+0000 7f0d401c7b80 1 -- 172.21.15.70:0/2582255965 >> 172.21.15.70:0/2582255965 conn(0x55bb9a298080 msgr2=0x55bb9a364cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:06.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.962+0000 7f0d0effd700 0 client.21036 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:19:06.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.962+0000 7f0d401c7b80 1 -- 172.21.15.70:0/2582255965 shutdown_connections 2024-06-04T05:19:06.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.962+0000 7f0d401c7b80 1 -- 172.21.15.70:0/2582255965 wait complete. 2024-06-04T05:19:06.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.962+0000 7f0d401c7b80 20 client.21036 trim_cache size 0 max 16384 2024-06-04T05:19:06.969 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:19:06.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.964+0000 7f0d35932700 1 -- 172.21.15.70:0/1395632365 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0d180b29f0 msgr2=0x7f0d180b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:06.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.964+0000 7f0d35932700 1 --2- 172.21.15.70:0/1395632365 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0d180b29f0 0x7f0d180b4eb0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0d20006010 tx=0x7f0d2000b560 comp rx=0 tx=0).stop 2024-06-04T05:19:06.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.964+0000 7f0d35932700 1 -- 172.21.15.70:0/1395632365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d30076090 msgr2=0x7f0d301a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:06.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.964+0000 7f0d35932700 1 --2- 172.21.15.70:0/1395632365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d30076090 0x7f0d301a0690 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f0d2400d950 tx=0x7f0d2400dc60 comp rx=0 tx=0).stop 2024-06-04T05:19:06.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.964+0000 7f0d35932700 1 -- 172.21.15.70:0/1395632365 shutdown_connections 2024-06-04T05:19:06.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.964+0000 7f0d35932700 1 --2- 172.21.15.70:0/1395632365 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0d180b29f0 0x7f0d180b4eb0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.964+0000 7f0d35932700 1 --2- 172.21.15.70:0/1395632365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d30114740 0x7f0d301a4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.964+0000 7f0d35932700 1 --2- 172.21.15.70:0/1395632365 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d30075770 0x7f0d301a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.964+0000 7f0d35932700 1 --2- 172.21.15.70:0/1395632365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d30076090 0x7f0d301a0690 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:06.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.964+0000 7f0d35932700 1 -- 172.21.15.70:0/1395632365 >> 172.21.15.70:0/1395632365 conn(0x7f0d3007a6b0 msgr2=0x7f0d30111640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:06.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.964+0000 7f0d35932700 1 -- 172.21.15.70:0/1395632365 shutdown_connections 2024-06-04T05:19:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:06.964+0000 7f0d35932700 1 -- 172.21.15.70:0/1395632365 wait complete. 2024-06-04T05:19:07.184 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:19:07.672 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:19:07.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 6.80 MB/sec execute 359 sec latency 26320.666 ms 2024-06-04T05:19:07.995 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:07 smithi070 ceph-mon[32631]: pgmap v689: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 72 KiB/s wr, 36 op/s 2024-06-04T05:19:08.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:07 smithi112 ceph-mon[36453]: pgmap v689: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 72 KiB/s wr, 36 op/s 2024-06-04T05:19:08.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:07 smithi098 ceph-mon[35569]: pgmap v689: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 72 KiB/s wr, 36 op/s 2024-06-04T05:19:08.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 6.78 MB/sec execute 360 sec latency 27320.703 ms 2024-06-04T05:19:08.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:08 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:19:08.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:08 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:19:08.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:08 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:19:08.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:08 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:19:09.013 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:19:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:08 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:19:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:08 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:19:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:08 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:19:09.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:08 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:19:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:08 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:19:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:08 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:19:09.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:08 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:19:09.246 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:08 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:19:09.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61673 6.76 MB/sec execute 361 sec latency 28320.769 ms 2024-06-04T05:19:10.054 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:09 smithi070 ceph-mon[32631]: pgmap v690: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 65 KiB/s wr, 30 op/s 2024-06-04T05:19:10.054 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:09 smithi070 ceph-mon[32631]: osdmap e111: 12 total, 12 up, 12 in 2024-06-04T05:19:10.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.160+0000 7f9b15549700 1 -- 172.21.15.70:0/2459639312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b101095a0 msgr2=0x7f9b10109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.160+0000 7f9b15549700 1 --2- 172.21.15.70:0/2459639312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b101095a0 0x7f9b10109980 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f9b00009990 tx=0x7f9b00009ca0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.160+0000 7f9b15549700 1 -- 172.21.15.70:0/2459639312 shutdown_connections 2024-06-04T05:19:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.160+0000 7f9b15549700 1 --2- 172.21.15.70:0/2459639312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b1006c470 0x7f9b101103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.160+0000 7f9b15549700 1 --2- 172.21.15.70:0/2459639312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b101095a0 0x7f9b10109980 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.160+0000 7f9b15549700 1 --2- 172.21.15.70:0/2459639312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b1006bab0 0x7f9b1006bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.160+0000 7f9b15549700 1 -- 172.21.15.70:0/2459639312 >> 172.21.15.70:0/2459639312 conn(0x7f9b100793b0 msgr2=0x7f9b100797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.161+0000 7f9b15549700 1 -- 172.21.15.70:0/2459639312 shutdown_connections 2024-06-04T05:19:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.161+0000 7f9b15549700 1 -- 172.21.15.70:0/2459639312 wait complete. 2024-06-04T05:19:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.161+0000 7f9b15549700 1 Processor -- start 2024-06-04T05:19:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.161+0000 7f9b15549700 1 -- start start 2024-06-04T05:19:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.161+0000 7f9b15549700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b1006bab0 0x7f9b1019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.161+0000 7f9b15549700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b1006c470 0x7f9b1019fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.161+0000 7f9b15549700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b101095a0 0x7f9b10199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b15549700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9b10113580 con 0x7f9b1006bab0 2024-06-04T05:19:10.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b15549700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b10113400 con 0x7f9b101095a0 2024-06-04T05:19:10.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b15549700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9b10113700 con 0x7f9b1006c470 2024-06-04T05:19:10.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b0effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b1006bab0 0x7f9b1019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b0effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b1006bab0 0x7f9b1019f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33284/0 (socket says 172.21.15.70:33284) 2024-06-04T05:19:10.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b0effd700 1 -- 172.21.15.70:0/2681896440 learned_addr learned my addr 172.21.15.70:0/2681896440 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:10.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b0f7fe700 1 --2- 172.21.15.70:0/2681896440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b101095a0 0x7f9b10199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b07fff700 1 --2- 172.21.15.70:0/2681896440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b1006c470 0x7f9b1019fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b0effd700 1 -- 172.21.15.70:0/2681896440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b1006c470 msgr2=0x7f9b1019fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:10.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b0effd700 1 --2- 172.21.15.70:0/2681896440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b1006c470 0x7f9b1019fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b0effd700 1 -- 172.21.15.70:0/2681896440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b101095a0 msgr2=0x7f9b10199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b0effd700 1 --2- 172.21.15.70:0/2681896440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b101095a0 0x7f9b10199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b0effd700 1 -- 172.21.15.70:0/2681896440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b1019a080 con 0x7f9b1006bab0 2024-06-04T05:19:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b0effd700 1 --2- 172.21.15.70:0/2681896440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b1006bab0 0x7f9b1019f760 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f9b00000c00 tx=0x7f9b000047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b0cff9700 1 -- 172.21.15.70:0/2681896440 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b00027070 con 0x7f9b1006bab0 2024-06-04T05:19:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b15549700 1 -- 172.21.15.70:0/2681896440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b1019a310 con 0x7f9b1006bab0 2024-06-04T05:19:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b0cff9700 1 -- 172.21.15.70:0/2681896440 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b0000fb80 con 0x7f9b1006bab0 2024-06-04T05:19:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b0cff9700 1 -- 172.21.15.70:0/2681896440 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b00022d00 con 0x7f9b1006bab0 2024-06-04T05:19:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.162+0000 7f9b15549700 1 -- 172.21.15.70:0/2681896440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b101a6630 con 0x7f9b1006bab0 2024-06-04T05:19:10.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.165+0000 7f9b0cff9700 1 -- 172.21.15.70:0/2681896440 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f9b0002a070 con 0x7f9b1006bab0 2024-06-04T05:19:10.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.165+0000 7f9b0cff9700 1 --2- 172.21.15.70:0/2681896440 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9afc0b29f0 0x7f9afc0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.165+0000 7f9b07fff700 1 --2- 172.21.15.70:0/2681896440 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9afc0b29f0 0x7f9afc0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.165+0000 7f9b0cff9700 1 -- 172.21.15.70:0/2681896440 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8615+0+0 (secure 0 0 0) 0x7f9b000e3500 con 0x7f9b1006bab0 2024-06-04T05:19:10.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.166+0000 7f9b07fff700 1 --2- 172.21.15.70:0/2681896440 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9afc0b29f0 0x7f9afc0b4eb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9af8005950 tx=0x7f9af80058e0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:10.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.166+0000 7f9b1fddeb80 1 Processor -- start 2024-06-04T05:19:10.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.166+0000 7f9b1fddeb80 1 -- start start 2024-06-04T05:19:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.166+0000 7f9b1fddeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe342b0 0x55ddcfe34690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.166+0000 7f9b1fddeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe2ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b1fddeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe3b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b1fddeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ddcfd8dd90 con 0x55ddcfe2f2a0 2024-06-04T05:19:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b1fddeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ddcfd7d930 con 0x55ddcfe342b0 2024-06-04T05:19:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b1fddeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ddcfd81970 con 0x55ddcfe2e8e0 2024-06-04T05:19:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b0f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe3b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b0f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe3b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33300/0 (socket says 172.21.15.70:33300) 2024-06-04T05:19:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b0f7fe700 1 -- 172.21.15.70:0/630188544 learned_addr learned my addr 172.21.15.70:0/630188544 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b07fff700 1 --2- 172.21.15.70:0/630188544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe2ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b0effd700 1 --2- 172.21.15.70:0/630188544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe342b0 0x55ddcfe34690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b0f7fe700 1 -- 172.21.15.70:0/630188544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe2e8e0 msgr2=0x55ddcfe2ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b0f7fe700 1 --2- 172.21.15.70:0/630188544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe2ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b0f7fe700 1 -- 172.21.15.70:0/630188544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe342b0 msgr2=0x55ddcfe34690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b0f7fe700 1 --2- 172.21.15.70:0/630188544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe342b0 0x55ddcfe34690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b0f7fe700 1 -- 172.21.15.70:0/630188544 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddcfccd9d0 con 0x55ddcfe2f2a0 2024-06-04T05:19:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b0effd700 1 --2- 172.21.15.70:0/630188544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe342b0 0x55ddcfe34690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:19:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b0f7fe700 1 --2- 172.21.15.70:0/630188544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe3b0e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f9b08003fa0 tx=0x7f9b08009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f931b87879c8ce14 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b057fa700 1 -- 172.21.15.70:0/630188544 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b08004bb0 con 0x55ddcfe2f2a0 2024-06-04T05:19:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b057fa700 1 -- 172.21.15.70:0/630188544 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b08004880 con 0x55ddcfe2f2a0 2024-06-04T05:19:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.167+0000 7f9b057fa700 1 -- 172.21.15.70:0/630188544 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b08007c90 con 0x55ddcfe2f2a0 2024-06-04T05:19:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.168+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/630188544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2f2a0 msgr2=0x55ddcfe3b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.168+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/630188544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe3b0e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f9b08003fa0 tx=0x7f9b08009ba0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.168+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/630188544 shutdown_connections 2024-06-04T05:19:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.168+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/630188544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe3b0e0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.168+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/630188544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe342b0 0x55ddcfe34690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.168+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/630188544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe2ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.168+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/630188544 >> 172.21.15.70:0/630188544 conn(0x55ddcfd5c180 msgr2=0x55ddcfe291f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.168+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/630188544 shutdown_connections 2024-06-04T05:19:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.168+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/630188544 wait complete. 2024-06-04T05:19:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b1fddeb80 1 -- start start 2024-06-04T05:19:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b1fddeb80 1 Processor -- start 2024-06-04T05:19:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b1fddeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe3ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b1fddeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe38080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b1fddeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe342b0 0x55ddcfe385c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b1fddeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ddcfc840a0 con 0x55ddcfe2e8e0 2024-06-04T05:19:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b1fddeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ddcfd1f8c0 con 0x55ddcfe2f2a0 2024-06-04T05:19:10.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b1fddeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ddcfe3faf0 con 0x55ddcfe342b0 2024-06-04T05:19:10.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b0effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe3ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b0effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe3ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33316/0 (socket says 172.21.15.70:33316) 2024-06-04T05:19:10.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b1fddeb80 10 client.0 _ll_register_callbacks cb 0x55ddcfd18a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:10.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b0effd700 1 -- 172.21.15.70:0/3362793847 learned_addr learned my addr 172.21.15.70:0/3362793847 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:10.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b07fff700 1 --2- 172.21.15.70:0/3362793847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe38080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b0f7fe700 1 --2- 172.21.15.70:0/3362793847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe342b0 0x55ddcfe385c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b0effd700 1 -- 172.21.15.70:0/3362793847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe342b0 msgr2=0x55ddcfe385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:10.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b0effd700 1 --2- 172.21.15.70:0/3362793847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe342b0 0x55ddcfe385c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b0effd700 1 -- 172.21.15.70:0/3362793847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe2f2a0 msgr2=0x55ddcfe38080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:10.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b0effd700 1 --2- 172.21.15.70:0/3362793847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe38080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b0effd700 1 -- 172.21.15.70:0/3362793847 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddcfd1c790 con 0x55ddcfe2e8e0 2024-06-04T05:19:10.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.169+0000 7f9b0f7fe700 1 --2- 172.21.15.70:0/3362793847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe342b0 0x55ddcfe385c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:19:10.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.170+0000 7f9b0effd700 1 --2- 172.21.15.70:0/3362793847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe3ad80 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f9b000aa7a0 tx=0x7f9b000aa410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e86cc784dd568b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:10.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.170+0000 7f9af67fc700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:10.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.170+0000 7f9af67fc700 1 -- 172.21.15.70:0/3362793847 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b000aa810 con 0x55ddcfe2e8e0 2024-06-04T05:19:10.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.170+0000 7f9af67fc700 1 -- 172.21.15.70:0/3362793847 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8615+0+0 (secure 0 0 0) 0x7f9b000aac80 con 0x55ddcfe2e8e0 2024-06-04T05:19:10.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.170+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/3362793847 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b08004880 con 0x55ddcfe2e8e0 2024-06-04T05:19:10.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.170+0000 7f9af67fc700 1 -- 172.21.15.70:0/3362793847 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f9b000aebc0 con 0x55ddcfe2e8e0 2024-06-04T05:19:10.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.170+0000 7f9b1fddeb80 10 client.21048 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:10.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.170+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/3362793847 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55ddcfe4a200 con 0x55ddcfe2e8e0 2024-06-04T05:19:10.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.170+0000 7f9af67fc700 1 -- 172.21.15.70:0/3362793847 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b08004880 con 0x55ddcfe2e8e0 2024-06-04T05:19:10.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.170+0000 7f9af67fc700 1 -- 172.21.15.70:0/3362793847 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f9b000920d0 con 0x55ddcfe2e8e0 2024-06-04T05:19:10.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.170+0000 7f9b1fddeb80 10 client.21048 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:10.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.170+0000 7f9b1fddeb80 10 client.21048 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:19:10.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.170+0000 7f9b1fddeb80 20 client.21048 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:10.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.170+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/3362793847 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ddcfe4c720 0x55ddcfe4eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.170+0000 7f9b1fddeb80 4 client.21048 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:19:10.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.170+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/3362793847 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddcfdfaae0 con 0x55ddcfe4c720 2024-06-04T05:19:10.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.171+0000 7f9b07fff700 1 --2- 172.21.15.70:0/3362793847 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ddcfe4c720 0x55ddcfe4eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.171+0000 7f9af67fc700 10 client.21048 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:10.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.171+0000 7f9b07fff700 1 --2- 172.21.15.70:0/3362793847 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ddcfe4c720 0x55ddcfe4eb70 crc :-1 s=READY pgs=461 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=169fe9fddef6f648 server_cookie=6e4864e26c8eca57 in_seq=0 out_seq=0 2024-06-04T05:19:10.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.172+0000 7f9af67fc700 1 -- 172.21.15.70:0/3362793847 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ddcfdfaae0 con 0x55ddcfe4c720 2024-06-04T05:19:10.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.172+0000 7f9af67fc700 10 client.21048 handle_command_reply: tid=0 2024-06-04T05:19:10.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.172+0000 7f9b1fddeb80 1 client.21048 shutdown 2024-06-04T05:19:10.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.172+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/3362793847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2e8e0 msgr2=0x55ddcfe3ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:10.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.172+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/3362793847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe3ad80 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f9b000aa7a0 tx=0x7f9b000aa410 comp rx=0 tx=0).stop 2024-06-04T05:19:10.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.172+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/3362793847 shutdown_connections 2024-06-04T05:19:10.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.172+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/3362793847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe342b0 0x55ddcfe385c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.172+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/3362793847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe3ad80 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.172+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/3362793847 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ddcfe4c720 0x55ddcfe4eb70 crc :-1 s=READY pgs=461 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.172+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/3362793847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe38080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.172+0000 7f9af67fc700 0 client.21048 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:10.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.172+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/3362793847 >> 172.21.15.70:0/3362793847 conn(0x55ddcfd5c180 msgr2=0x55ddcfe23ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:10.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.172+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/3362793847 shutdown_connections 2024-06-04T05:19:10.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.172+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/3362793847 wait complete. 2024-06-04T05:19:10.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.173+0000 7f9b1fddeb80 20 client.21048 trim_cache size 0 max 16384 2024-06-04T05:19:10.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:09 smithi112 ceph-mon[36453]: pgmap v690: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 65 KiB/s wr, 30 op/s 2024-06-04T05:19:10.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:09 smithi112 ceph-mon[36453]: osdmap e111: 12 total, 12 up, 12 in 2024-06-04T05:19:10.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.193+0000 7f9b1fddeb80 1 Processor -- start 2024-06-04T05:19:10.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.193+0000 7f9b1fddeb80 1 -- start start 2024-06-04T05:19:10.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.193+0000 7f9b1fddeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe32860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.193+0000 7f9b1fddeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe3a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.193+0000 7f9b1fddeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe342b0 0x55ddcfe54440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.193+0000 7f9b1fddeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9b101abaa0 con 0x55ddcfe2e8e0 2024-06-04T05:19:10.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.193+0000 7f9b1fddeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b101ad640 con 0x55ddcfe2f2a0 2024-06-04T05:19:10.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.193+0000 7f9b1fddeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9b101ac1d0 con 0x55ddcfe342b0 2024-06-04T05:19:10.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.193+0000 7f9b0effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe32860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.193+0000 7f9b0effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe32860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33318/0 (socket says 172.21.15.70:33318) 2024-06-04T05:19:10.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.193+0000 7f9b0effd700 1 -- 172.21.15.70:0/2513944011 learned_addr learned my addr 172.21.15.70:0/2513944011 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:10.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.193+0000 7f9b0f7fe700 1 --2- 172.21.15.70:0/2513944011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe342b0 0x55ddcfe54440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.194+0000 7f9b07fff700 1 --2- 172.21.15.70:0/2513944011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe3a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.194+0000 7f9b0effd700 1 -- 172.21.15.70:0/2513944011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe342b0 msgr2=0x55ddcfe54440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:10.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.194+0000 7f9b0effd700 1 --2- 172.21.15.70:0/2513944011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe342b0 0x55ddcfe54440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.194+0000 7f9b0effd700 1 -- 172.21.15.70:0/2513944011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe2f2a0 msgr2=0x55ddcfe3a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:10.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.194+0000 7f9b0effd700 1 --2- 172.21.15.70:0/2513944011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe3a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.194+0000 7f9b0effd700 1 -- 172.21.15.70:0/2513944011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddcfe259f0 con 0x55ddcfe2e8e0 2024-06-04T05:19:10.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.194+0000 7f9b0effd700 1 --2- 172.21.15.70:0/2513944011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe32860 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f9b000aa510 tx=0x7f9b000aa480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e19526e089d683 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:10.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.194+0000 7f9b04ff9700 1 -- 172.21.15.70:0/2513944011 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b000b35e0 con 0x55ddcfe2e8e0 2024-06-04T05:19:10.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.194+0000 7f9b04ff9700 1 -- 172.21.15.70:0/2513944011 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b000aac80 con 0x55ddcfe2e8e0 2024-06-04T05:19:10.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.194+0000 7f9b04ff9700 1 -- 172.21.15.70:0/2513944011 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b000bfe30 con 0x55ddcfe2e8e0 2024-06-04T05:19:10.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.195+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/2513944011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2e8e0 msgr2=0x55ddcfe32860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:10.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.195+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/2513944011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe32860 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f9b000aa510 tx=0x7f9b000aa480 comp rx=0 tx=0).stop 2024-06-04T05:19:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.195+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/2513944011 shutdown_connections 2024-06-04T05:19:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.195+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/2513944011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe342b0 0x55ddcfe54440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.195+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/2513944011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe32860 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.195+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/2513944011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe3a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.195+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/2513944011 >> 172.21.15.70:0/2513944011 conn(0x55ddcfd5c180 msgr2=0x55ddcfe23ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.195+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/2513944011 shutdown_connections 2024-06-04T05:19:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.195+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/2513944011 wait complete. 2024-06-04T05:19:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.195+0000 7f9b1fddeb80 1 -- start start 2024-06-04T05:19:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.195+0000 7f9b1fddeb80 1 Processor -- start 2024-06-04T05:19:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.195+0000 7f9b1fddeb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe45040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.195+0000 7f9b1fddeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe45580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.195+0000 7f9b1fddeb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe342b0 0x55ddcfe41e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.195+0000 7f9b1fddeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9b101acf70 con 0x55ddcfe2f2a0 2024-06-04T05:19:10.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.195+0000 7f9b1fddeb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b101ac8a0 con 0x55ddcfe2e8e0 2024-06-04T05:19:10.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.195+0000 7f9b1fddeb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9b101a9a60 con 0x55ddcfe342b0 2024-06-04T05:19:10.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9b1fddeb80 10 client.0 _ll_register_callbacks cb 0x55ddcfd18a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:10.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9b07fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe45580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9b07fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe45580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33326/0 (socket says 172.21.15.70:33326) 2024-06-04T05:19:10.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9b07fff700 1 -- 172.21.15.70:0/4081761936 learned_addr learned my addr 172.21.15.70:0/4081761936 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:10.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9b0effd700 1 --2- 172.21.15.70:0/4081761936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe45040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9b07fff700 1 -- 172.21.15.70:0/4081761936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe342b0 msgr2=0x55ddcfe41e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:10.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9b0f7fe700 1 --2- 172.21.15.70:0/4081761936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe342b0 0x55ddcfe41e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9b07fff700 1 --2- 172.21.15.70:0/4081761936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe342b0 0x55ddcfe41e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9b07fff700 1 -- 172.21.15.70:0/4081761936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe2e8e0 msgr2=0x55ddcfe45040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:10.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9b07fff700 1 --2- 172.21.15.70:0/4081761936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe45040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9b07fff700 1 -- 172.21.15.70:0/4081761936 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddcfe42680 con 0x55ddcfe2f2a0 2024-06-04T05:19:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9b0f7fe700 1 --2- 172.21.15.70:0/4081761936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe342b0 0x55ddcfe41e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:19:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9b07fff700 1 --2- 172.21.15.70:0/4081761936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe45580 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f9af800ba40 tx=0x7f9af80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e88a13a1e1a032 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9af67fc700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9af67fc700 1 -- 172.21.15.70:0/4081761936 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af802b3f0 con 0x55ddcfe2f2a0 2024-06-04T05:19:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9af67fc700 1 -- 172.21.15.70:0/4081761936 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8615+0+0 (secure 0 0 0) 0x7f9af802b590 con 0x55ddcfe2f2a0 2024-06-04T05:19:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/4081761936 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b000aac80 con 0x55ddcfe2f2a0 2024-06-04T05:19:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9af67fc700 1 -- 172.21.15.70:0/4081761936 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f9af802b790 con 0x55ddcfe2f2a0 2024-06-04T05:19:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9b1fddeb80 10 client.21054 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.196+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/4081761936 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55ddcfeb7ff0 con 0x55ddcfe2f2a0 2024-06-04T05:19:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.197+0000 7f9af67fc700 1 -- 172.21.15.70:0/4081761936 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b000aac80 con 0x55ddcfe2f2a0 2024-06-04T05:19:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.197+0000 7f9af67fc700 1 -- 172.21.15.70:0/4081761936 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f9af8032070 con 0x55ddcfe2f2a0 2024-06-04T05:19:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.197+0000 7f9b1fddeb80 10 client.21054 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:10.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.197+0000 7f9b1fddeb80 10 client.21054 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:19:10.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.197+0000 7f9b1fddeb80 20 client.21054 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:10.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.197+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/4081761936 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ddcfeb8760 0x55ddcfebab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:10.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.197+0000 7f9b1fddeb80 4 client.21054 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "549ed795", "release": true, "await": true}] 2024-06-04T05:19:10.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.197+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/4081761936 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "549ed795", "release": true, "await": true}) -- 0x55ddcfd8fce0 con 0x55ddcfeb8760 2024-06-04T05:19:10.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.197+0000 7f9b0effd700 1 --2- 172.21.15.70:0/4081761936 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ddcfeb8760 0x55ddcfebab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:10.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.198+0000 7f9af67fc700 10 client.21054 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:10.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.198+0000 7f9b0effd700 1 --2- 172.21.15.70:0/4081761936 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ddcfeb8760 0x55ddcfebab80 crc :-1 s=READY pgs=462 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3fd1e990dd7b4a server_cookie=f22eee2e85766f7f in_seq=0 out_seq=0 2024-06-04T05:19:10.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.229+0000 7f9af67fc700 1 -- 172.21.15.70:0/4081761936 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55ddcfd8fce0 con 0x55ddcfeb8760 2024-06-04T05:19:10.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.229+0000 7f9af67fc700 10 client.21054 handle_command_reply: tid=0 2024-06-04T05:19:10.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.229+0000 7f9b1fddeb80 1 client.21054 shutdown 2024-06-04T05:19:10.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.230+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/4081761936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2f2a0 msgr2=0x55ddcfe45580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:10.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.230+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/4081761936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe45580 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f9af800ba40 tx=0x7f9af80047c0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.230+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/4081761936 shutdown_connections 2024-06-04T05:19:10.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.230+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/4081761936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ddcfe342b0 0x55ddcfe41e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.230+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/4081761936 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ddcfeb8760 0x55ddcfebab80 crc :-1 s=READY pgs=462 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.230+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/4081761936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddcfe2e8e0 0x55ddcfe45040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.230+0000 7f9af67fc700 0 client.21054 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:10.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.230+0000 7f9b1fddeb80 1 --2- 172.21.15.70:0/4081761936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ddcfe2f2a0 0x55ddcfe45580 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.230+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/4081761936 >> 172.21.15.70:0/4081761936 conn(0x55ddcfd5c180 msgr2=0x55ddcfe36840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:10.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.230+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/4081761936 shutdown_connections 2024-06-04T05:19:10.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.230+0000 7f9b1fddeb80 1 -- 172.21.15.70:0/4081761936 wait complete. 2024-06-04T05:19:10.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.230+0000 7f9b1fddeb80 20 client.21054 trim_cache size 0 max 16384 2024-06-04T05:19:10.233 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:19:10.233 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 59, 2024-06-04T05:19:10.233 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:19:10.234 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 68, 2024-06-04T05:19:10.234 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:19:10.234 INFO:teuthology.orchestra.run.smithi070.stdout: "549ed795": { 2024-06-04T05:19:10.234 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 68, 2024-06-04T05:19:10.234 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:19:10.234 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:19:10.234 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:19:10.234 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:19:10.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:19:10.234 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:19:10.234 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 145.8, 2024-06-04T05:19:10.234 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:19:10.234 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:19:10.235 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:19:10.235 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:19:10.235 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:19:10.235 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:19:10.235 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:19:10.235 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:19:10.235 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:19:10.235 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:19:10.235 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:19:10.235 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:19:10.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.231+0000 7f9b15549700 1 -- 172.21.15.70:0/2681896440 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9afc0b29f0 msgr2=0x7f9afc0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:10.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.231+0000 7f9b15549700 1 --2- 172.21.15.70:0/2681896440 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9afc0b29f0 0x7f9afc0b4eb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9af8005950 tx=0x7f9af80058e0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.231+0000 7f9b15549700 1 -- 172.21.15.70:0/2681896440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b1006bab0 msgr2=0x7f9b1019f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:10.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.231+0000 7f9b15549700 1 --2- 172.21.15.70:0/2681896440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b1006bab0 0x7f9b1019f760 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f9b00000c00 tx=0x7f9b000047b0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.233+0000 7f9b15549700 1 -- 172.21.15.70:0/2681896440 shutdown_connections 2024-06-04T05:19:10.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.233+0000 7f9b15549700 1 --2- 172.21.15.70:0/2681896440 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f9afc0b29f0 0x7f9afc0b4eb0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.233+0000 7f9b15549700 1 --2- 172.21.15.70:0/2681896440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b101095a0 0x7f9b10199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.233+0000 7f9b15549700 1 --2- 172.21.15.70:0/2681896440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b1006bab0 0x7f9b1019f760 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.233+0000 7f9b15549700 1 --2- 172.21.15.70:0/2681896440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b1006c470 0x7f9b1019fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:10.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.233+0000 7f9b15549700 1 -- 172.21.15.70:0/2681896440 >> 172.21.15.70:0/2681896440 conn(0x7f9b100793b0 msgr2=0x7f9b10101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:10.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.234+0000 7f9b15549700 1 -- 172.21.15.70:0/2681896440 shutdown_connections 2024-06-04T05:19:10.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:10.234+0000 7f9b15549700 1 -- 172.21.15.70:0/2681896440 wait complete. 2024-06-04T05:19:10.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:09 smithi098 ceph-mon[35569]: pgmap v690: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 65 KiB/s wr, 30 op/s 2024-06-04T05:19:10.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:09 smithi098 ceph-mon[35569]: osdmap e111: 12 total, 12 up, 12 in 2024-06-04T05:19:10.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61778 6.75 MB/sec execute 362 sec latency 28710.822 ms 2024-06-04T05:19:10.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:10 smithi070 ceph-mon[32631]: pgmap v692: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 78 KiB/s wr, 36 op/s 2024-06-04T05:19:11.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:10 smithi112 ceph-mon[36453]: pgmap v692: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 78 KiB/s wr, 36 op/s 2024-06-04T05:19:11.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:10 smithi098 ceph-mon[35569]: pgmap v692: 97 pgs: 97 active+clean; 893 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 78 KiB/s wr, 36 op/s 2024-06-04T05:19:11.327 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '549ed795', total seconds elapsed: 36.0 2024-06-04T05:19:11.328 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.85 2024-06-04T05:19:11.328 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:19:11.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 61955 6.76 MB/sec execute 363 sec latency 149.452 ms 2024-06-04T05:19:11.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.851+0000 7f39cefa2700 1 -- 172.21.15.70:0/1808628229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c8108520 msgr2=0x7f39c8108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:11.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.851+0000 7f39cefa2700 1 --2- 172.21.15.70:0/1808628229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c8108520 0x7f39c8108900 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f39bc009990 tx=0x7f39bc009ca0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.851+0000 7f39cefa2700 1 -- 172.21.15.70:0/1808628229 shutdown_connections 2024-06-04T05:19:11.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.851+0000 7f39cefa2700 1 --2- 172.21.15.70:0/1808628229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c8102e00 0x7f39c810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.851+0000 7f39cefa2700 1 --2- 172.21.15.70:0/1808628229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c8108520 0x7f39c8108900 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.851+0000 7f39cefa2700 1 --2- 172.21.15.70:0/1808628229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39c8102440 0x7f39c81028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.851+0000 7f39cefa2700 1 -- 172.21.15.70:0/1808628229 >> 172.21.15.70:0/1808628229 conn(0x7f39c80793f0 msgr2=0x7f39c8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:11.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.851+0000 7f39cefa2700 1 -- 172.21.15.70:0/1808628229 shutdown_connections 2024-06-04T05:19:11.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.852+0000 7f39cefa2700 1 -- 172.21.15.70:0/1808628229 wait complete. 2024-06-04T05:19:11.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.853+0000 7f39cefa2700 1 Processor -- start 2024-06-04T05:19:11.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.853+0000 7f39cefa2700 1 -- start start 2024-06-04T05:19:11.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.853+0000 7f39cefa2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c8102440 0x7f39c8111380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.853+0000 7f39cefa2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39c8102e00 0x7f39c81118c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.853+0000 7f39cefa2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c8108520 0x7f39c8111e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.853+0000 7f39cefa2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39c8113640 con 0x7f39c8102e00 2024-06-04T05:19:11.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.853+0000 7f39ccd3e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c8102440 0x7f39c8111380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.853+0000 7f39ccd3e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c8102440 0x7f39c8111380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36076/0 (socket says 172.21.15.70:36076) 2024-06-04T05:19:11.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.853+0000 7f39ccd3e700 1 -- 172.21.15.70:0/318953113 learned_addr learned my addr 172.21.15.70:0/318953113 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:11.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.853+0000 7f39cd53f700 1 --2- 172.21.15.70:0/318953113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c8108520 0x7f39c8111e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.853+0000 7f39c7fff700 1 --2- 172.21.15.70:0/318953113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39c8102e00 0x7f39c81118c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.854+0000 7f39cefa2700 1 -- 172.21.15.70:0/318953113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81134c0 con 0x7f39c8108520 2024-06-04T05:19:11.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.854+0000 7f39cefa2700 1 -- 172.21.15.70:0/318953113 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81137c0 con 0x7f39c8102440 2024-06-04T05:19:11.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.854+0000 7f39ccd3e700 1 -- 172.21.15.70:0/318953113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c8108520 msgr2=0x7f39c8111e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:11.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.854+0000 7f39ccd3e700 1 --2- 172.21.15.70:0/318953113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c8108520 0x7f39c8111e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.854+0000 7f39ccd3e700 1 -- 172.21.15.70:0/318953113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39c8102e00 msgr2=0x7f39c81118c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:11.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.854+0000 7f39ccd3e700 1 --2- 172.21.15.70:0/318953113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39c8102e00 0x7f39c81118c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.854+0000 7f39ccd3e700 1 -- 172.21.15.70:0/318953113 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c81a8050 con 0x7f39c8102440 2024-06-04T05:19:11.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.855+0000 7f39ccd3e700 1 --2- 172.21.15.70:0/318953113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c8102440 0x7f39c8111380 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f39bc006010 tx=0x7f39bc0057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:11.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.855+0000 7f39c5ffb700 1 -- 172.21.15.70:0/318953113 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc027070 con 0x7f39c8102440 2024-06-04T05:19:11.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.855+0000 7f39c5ffb700 1 -- 172.21.15.70:0/318953113 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39bc022470 con 0x7f39c8102440 2024-06-04T05:19:11.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.855+0000 7f39cefa2700 1 -- 172.21.15.70:0/318953113 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39c81a82e0 con 0x7f39c8102440 2024-06-04T05:19:11.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.855+0000 7f39c5ffb700 1 -- 172.21.15.70:0/318953113 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc017650 con 0x7f39c8102440 2024-06-04T05:19:11.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.855+0000 7f39cefa2700 1 -- 172.21.15.70:0/318953113 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c81a8790 con 0x7f39c8102440 2024-06-04T05:19:11.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.859+0000 7f39c5ffb700 1 -- 172.21.15.70:0/318953113 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f39bc02a070 con 0x7f39c8102440 2024-06-04T05:19:11.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.860+0000 7f39c5ffb700 1 --2- 172.21.15.70:0/318953113 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39b80b2a70 0x7f39b80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.860+0000 7f39c5ffb700 1 -- 172.21.15.70:0/318953113 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8615+0+0 (secure 0 0 0) 0x7f39bc0e3460 con 0x7f39c8102440 2024-06-04T05:19:11.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.860+0000 7f39c7fff700 1 --2- 172.21.15.70:0/318953113 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39b80b2a70 0x7f39b80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.861+0000 7f39d9837b80 1 Processor -- start 2024-06-04T05:19:11.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.861+0000 7f39d9837b80 1 -- start start 2024-06-04T05:19:11.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.861+0000 7f39d9837b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 0x55eb10110700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.861+0000 7f39d9837b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010a8a0 0x55eb1010ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.861+0000 7f39c7fff700 1 --2- 172.21.15.70:0/318953113 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39b80b2a70 0x7f39b80b4f30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f39c8079b30 tx=0x7f39b4009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:11.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.861+0000 7f39d9837b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010b260 0x55eb10117080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.861+0000 7f39d9837b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55eb10068d70 con 0x55eb10110320 2024-06-04T05:19:11.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.861+0000 7f39d9837b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eb10058910 con 0x55eb1010a8a0 2024-06-04T05:19:11.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.861+0000 7f39d9837b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55eb1005c950 con 0x55eb1010b260 2024-06-04T05:19:11.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.861+0000 7f39cd53f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010b260 0x55eb10117080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.861+0000 7f39cd53f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010b260 0x55eb10117080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36084/0 (socket says 172.21.15.70:36084) 2024-06-04T05:19:11.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.861+0000 7f39cd53f700 1 -- 172.21.15.70:0/3089369961 learned_addr learned my addr 172.21.15.70:0/3089369961 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:11.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.861+0000 7f39c7fff700 1 --2- 172.21.15.70:0/3089369961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010a8a0 0x55eb1010ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.861+0000 7f39ccd3e700 1 --2- 172.21.15.70:0/3089369961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 0x55eb10110700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.862+0000 7f39cd53f700 1 -- 172.21.15.70:0/3089369961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010a8a0 msgr2=0x55eb1010ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:11.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.862+0000 7f39cd53f700 1 --2- 172.21.15.70:0/3089369961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010a8a0 0x55eb1010ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.862+0000 7f39cd53f700 1 -- 172.21.15.70:0/3089369961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 msgr2=0x55eb10110700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:11.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.862+0000 7f39cd53f700 1 --2- 172.21.15.70:0/3089369961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 0x55eb10110700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.862+0000 7f39cd53f700 1 -- 172.21.15.70:0/3089369961 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb10001fb0 con 0x55eb1010b260 2024-06-04T05:19:11.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.862+0000 7f39cd53f700 1 --2- 172.21.15.70:0/3089369961 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010b260 0x55eb10117080 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f39c0003f30 tx=0x7f39c0009c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a25ca13355466579 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:11.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.862+0000 7f39b2ffd700 1 -- 172.21.15.70:0/3089369961 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0004670 con 0x55eb1010b260 2024-06-04T05:19:11.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.862+0000 7f39b2ffd700 1 -- 172.21.15.70:0/3089369961 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39c001a070 con 0x55eb1010b260 2024-06-04T05:19:11.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.862+0000 7f39b2ffd700 1 -- 172.21.15.70:0/3089369961 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c00156e0 con 0x55eb1010b260 2024-06-04T05:19:11.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.863+0000 7f39d9837b80 1 -- 172.21.15.70:0/3089369961 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010b260 msgr2=0x55eb10117080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:11.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.863+0000 7f39d9837b80 1 --2- 172.21.15.70:0/3089369961 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010b260 0x55eb10117080 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f39c0003f30 tx=0x7f39c0009c20 comp rx=0 tx=0).stop 2024-06-04T05:19:11.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.863+0000 7f39d9837b80 1 -- 172.21.15.70:0/3089369961 shutdown_connections 2024-06-04T05:19:11.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.863+0000 7f39d9837b80 1 --2- 172.21.15.70:0/3089369961 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010b260 0x55eb10117080 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.863+0000 7f39d9837b80 1 --2- 172.21.15.70:0/3089369961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 0x55eb10110700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.863+0000 7f39d9837b80 1 --2- 172.21.15.70:0/3089369961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010a8a0 0x55eb1010ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.863+0000 7f39d9837b80 1 -- 172.21.15.70:0/3089369961 >> 172.21.15.70:0/3089369961 conn(0x55eb10036240 msgr2=0x55eb101053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:11.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.863+0000 7f39d9837b80 1 -- 172.21.15.70:0/3089369961 shutdown_connections 2024-06-04T05:19:11.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.863+0000 7f39d9837b80 1 -- 172.21.15.70:0/3089369961 wait complete. 2024-06-04T05:19:11.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.864+0000 7f39d9837b80 1 -- start start 2024-06-04T05:19:11.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.864+0000 7f39d9837b80 1 Processor -- start 2024-06-04T05:19:11.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.864+0000 7f39d9837b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 0x55eb10116d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.864+0000 7f39d9837b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010b260 0x55eb10114030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.864+0000 7f39d9837b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 0x55eb10114570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.864+0000 7f39ccd3e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 0x55eb10116d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.864+0000 7f39d9837b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55eb0ff5efa0 con 0x55eb10110320 2024-06-04T05:19:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.864+0000 7f39d9837b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eb0ffc0da0 con 0x55eb1010b260 2024-06-04T05:19:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.864+0000 7f39d9837b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55eb10117730 con 0x55eb1010a8a0 2024-06-04T05:19:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.864+0000 7f39ccd3e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 0x55eb10116d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36088/0 (socket says 172.21.15.70:36088) 2024-06-04T05:19:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.864+0000 7f39ccd3e700 1 -- 172.21.15.70:0/536552215 learned_addr learned my addr 172.21.15.70:0/536552215 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.864+0000 7f39cd53f700 1 --2- 172.21.15.70:0/536552215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 0x55eb10114570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.864+0000 7f39c7fff700 1 --2- 172.21.15.70:0/536552215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010b260 0x55eb10114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.865+0000 7f39d9837b80 10 client.0 _ll_register_callbacks cb 0x55eb0fff3950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.865+0000 7f39c7fff700 1 -- 172.21.15.70:0/536552215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 msgr2=0x55eb10116d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.865+0000 7f39c7fff700 1 --2- 172.21.15.70:0/536552215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 0x55eb10116d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.865+0000 7f39c7fff700 1 -- 172.21.15.70:0/536552215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 msgr2=0x55eb10114570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.865+0000 7f39c7fff700 1 --2- 172.21.15.70:0/536552215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 0x55eb10114570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.865+0000 7f39c7fff700 1 -- 172.21.15.70:0/536552215 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb0ffa88d0 con 0x55eb1010b260 2024-06-04T05:19:11.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.865+0000 7f39c7fff700 1 --2- 172.21.15.70:0/536552215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010b260 0x55eb10114030 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f39c8112e10 tx=0x7f39b40047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8705cee0008cc2cb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:11.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.865+0000 7f39a7fff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:19:11.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.865+0000 7f39a7fff700 1 -- 172.21.15.70:0/536552215 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b40044f0 con 0x55eb1010b260 2024-06-04T05:19:11.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.866+0000 7f39a7fff700 1 -- 172.21.15.70:0/536552215 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8615+0+0 (secure 0 0 0) 0x7f39b4003d70 con 0x55eb1010b260 2024-06-04T05:19:11.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.866+0000 7f39d9837b80 1 -- 172.21.15.70:0/536552215 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39c001a070 con 0x55eb1010b260 2024-06-04T05:19:11.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.866+0000 7f39a7fff700 1 -- 172.21.15.70:0/536552215 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f39b4027420 con 0x55eb1010b260 2024-06-04T05:19:11.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.866+0000 7f39d9837b80 10 client.29759 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:11.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.866+0000 7f39d9837b80 1 -- 172.21.15.70:0/536552215 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55eb10012760 con 0x55eb1010b260 2024-06-04T05:19:11.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.866+0000 7f39a7fff700 1 -- 172.21.15.70:0/536552215 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39b402f8a0 con 0x55eb1010b260 2024-06-04T05:19:11.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.867+0000 7f39a7fff700 1 -- 172.21.15.70:0/536552215 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f39b401d670 con 0x55eb1010b260 2024-06-04T05:19:11.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.867+0000 7f39d9837b80 10 client.29759 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:11.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.867+0000 7f39d9837b80 10 client.29759 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:19:11.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.867+0000 7f39d9837b80 20 client.29759 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:11.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.867+0000 7f39d9837b80 1 --2- 172.21.15.70:0/536552215 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55eb1012cc30 0x55eb1012f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.867+0000 7f39d9837b80 4 client.29759 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:19:11.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.867+0000 7f39d9837b80 1 -- 172.21.15.70:0/536552215 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f39c001a070 con 0x55eb1012cc30 2024-06-04T05:19:11.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.867+0000 7f39ccd3e700 1 --2- 172.21.15.70:0/536552215 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55eb1012cc30 0x55eb1012f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.868+0000 7f39a7fff700 10 client.29759 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:19:11.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.869+0000 7f39ccd3e700 1 --2- 172.21.15.70:0/536552215 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55eb1012cc30 0x55eb1012f070 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=aa0bbc91a8e5b22c server_cookie=8ce3a82bb8df3b22 in_seq=0 out_seq=0 2024-06-04T05:19:11.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.869+0000 7f39a7fff700 1 -- 172.21.15.70:0/536552215 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f39c001a070 con 0x55eb1012cc30 2024-06-04T05:19:11.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.869+0000 7f39a7fff700 10 client.29759 handle_command_reply: tid=0 2024-06-04T05:19:11.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.869+0000 7f39d9837b80 1 client.29759 shutdown 2024-06-04T05:19:11.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.870+0000 7f39d9837b80 1 -- 172.21.15.70:0/536552215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010b260 msgr2=0x55eb10114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:11.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.870+0000 7f39d9837b80 1 --2- 172.21.15.70:0/536552215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010b260 0x55eb10114030 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f39c8112e10 tx=0x7f39b40047c0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.870+0000 7f39d9837b80 1 -- 172.21.15.70:0/536552215 shutdown_connections 2024-06-04T05:19:11.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.870+0000 7f39d9837b80 1 --2- 172.21.15.70:0/536552215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 0x55eb10114570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.870+0000 7f39d9837b80 1 --2- 172.21.15.70:0/536552215 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55eb1012cc30 0x55eb1012f070 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.870+0000 7f39d9837b80 1 --2- 172.21.15.70:0/536552215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 0x55eb10116d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.870+0000 7f39d9837b80 1 --2- 172.21.15.70:0/536552215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010b260 0x55eb10114030 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.870+0000 7f39d9837b80 1 -- 172.21.15.70:0/536552215 >> 172.21.15.70:0/536552215 conn(0x55eb10036240 msgr2=0x55eb10104af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:11.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.870+0000 7f39a7fff700 0 client.29759 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:19:11.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.870+0000 7f39d9837b80 1 -- 172.21.15.70:0/536552215 shutdown_connections 2024-06-04T05:19:11.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.870+0000 7f39d9837b80 1 -- 172.21.15.70:0/536552215 wait complete. 2024-06-04T05:19:11.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.870+0000 7f39d9837b80 20 client.29759 trim_cache size 0 max 16384 2024-06-04T05:19:11.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.894+0000 7f39d9837b80 1 Processor -- start 2024-06-04T05:19:11.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.894+0000 7f39d9837b80 1 -- start start 2024-06-04T05:19:11.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.894+0000 7f39d9837b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 0x55eb10113930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.895+0000 7f39d9837b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010b260 0x55eb1010f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.895+0000 7f39d9837b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 0x55eb10105270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.895+0000 7f39d9837b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81abd10 con 0x55eb10110320 2024-06-04T05:19:11.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.895+0000 7f39ccd3e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 0x55eb10113930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.895+0000 7f39ccd3e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 0x55eb10113930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36100/0 (socket says 172.21.15.70:36100) 2024-06-04T05:19:11.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.895+0000 7f39d9837b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81ad8b0 con 0x55eb1010b260 2024-06-04T05:19:11.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.895+0000 7f39ccd3e700 1 -- 172.21.15.70:0/3259934137 learned_addr learned my addr 172.21.15.70:0/3259934137 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.895+0000 7f39d9837b80 1 -- 172.21.15.70:0/3259934137 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81ac440 con 0x55eb1010a8a0 2024-06-04T05:19:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.895+0000 7f39cd53f700 1 --2- 172.21.15.70:0/3259934137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 0x55eb10105270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.895+0000 7f39c7fff700 1 --2- 172.21.15.70:0/3259934137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010b260 0x55eb1010f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.895+0000 7f39c7fff700 1 -- 172.21.15.70:0/3259934137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 msgr2=0x55eb10113930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.895+0000 7f39c7fff700 1 --2- 172.21.15.70:0/3259934137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 0x55eb10113930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.895+0000 7f39c7fff700 1 -- 172.21.15.70:0/3259934137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 msgr2=0x55eb10105270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.895+0000 7f39c7fff700 1 --2- 172.21.15.70:0/3259934137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 0x55eb10105270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.895+0000 7f39c7fff700 1 -- 172.21.15.70:0/3259934137 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb101125a0 con 0x55eb1010b260 2024-06-04T05:19:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.895+0000 7f39c7fff700 1 --2- 172.21.15.70:0/3259934137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010b260 0x55eb1010f7c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f39b4003c40 tx=0x7f39b4009ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79cdd0bf87552ee server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.896+0000 7f39b27fc700 1 -- 172.21.15.70:0/3259934137 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b4004570 con 0x55eb1010b260 2024-06-04T05:19:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.896+0000 7f39b27fc700 1 -- 172.21.15.70:0/3259934137 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39b4003cf0 con 0x55eb1010b260 2024-06-04T05:19:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.896+0000 7f39b27fc700 1 -- 172.21.15.70:0/3259934137 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b4027920 con 0x55eb1010b260 2024-06-04T05:19:11.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.897+0000 7f39d9837b80 1 -- 172.21.15.70:0/3259934137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010b260 msgr2=0x55eb1010f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:11.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.897+0000 7f39d9837b80 1 --2- 172.21.15.70:0/3259934137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010b260 0x55eb1010f7c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f39b4003c40 tx=0x7f39b4009ae0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.897+0000 7f39d9837b80 1 -- 172.21.15.70:0/3259934137 shutdown_connections 2024-06-04T05:19:11.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.897+0000 7f39d9837b80 1 --2- 172.21.15.70:0/3259934137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 0x55eb10105270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.897+0000 7f39d9837b80 1 --2- 172.21.15.70:0/3259934137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 0x55eb10113930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.897+0000 7f39d9837b80 1 --2- 172.21.15.70:0/3259934137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010b260 0x55eb1010f7c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.897+0000 7f39d9837b80 1 -- 172.21.15.70:0/3259934137 >> 172.21.15.70:0/3259934137 conn(0x55eb10036240 msgr2=0x55eb1010a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.897+0000 7f39d9837b80 1 -- 172.21.15.70:0/3259934137 shutdown_connections 2024-06-04T05:19:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.897+0000 7f39d9837b80 1 -- 172.21.15.70:0/3259934137 wait complete. 2024-06-04T05:19:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.897+0000 7f39d9837b80 1 -- start start 2024-06-04T05:19:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.897+0000 7f39d9837b80 1 Processor -- start 2024-06-04T05:19:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.898+0000 7f39d9837b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 0x55eb10125870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.898+0000 7f39d9837b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010b260 0x55eb10134120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.898+0000 7f39d9837b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 0x55eb10134660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.898+0000 7f39d9837b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81ad1e0 con 0x55eb10110320 2024-06-04T05:19:11.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.898+0000 7f39d9837b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81acb10 con 0x55eb1010b260 2024-06-04T05:19:11.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.898+0000 7f39ccd3e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 0x55eb10125870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.898+0000 7f39d9837b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81a9d00 con 0x55eb1010a8a0 2024-06-04T05:19:11.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.898+0000 7f39ccd3e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 0x55eb10125870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36114/0 (socket says 172.21.15.70:36114) 2024-06-04T05:19:11.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.898+0000 7f39ccd3e700 1 -- 172.21.15.70:0/3030661955 learned_addr learned my addr 172.21.15.70:0/3030661955 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:11.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.898+0000 7f39d9837b80 10 client.0 _ll_register_callbacks cb 0x55eb0fff3950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:11.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.898+0000 7f39cd53f700 1 --2- 172.21.15.70:0/3030661955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 0x55eb10134660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.898+0000 7f39c7fff700 1 --2- 172.21.15.70:0/3030661955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010b260 0x55eb10134120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.899+0000 7f39ccd3e700 1 -- 172.21.15.70:0/3030661955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010b260 msgr2=0x55eb10134120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:11.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.899+0000 7f39ccd3e700 1 --2- 172.21.15.70:0/3030661955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010b260 0x55eb10134120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.899+0000 7f39ccd3e700 1 -- 172.21.15.70:0/3030661955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 msgr2=0x55eb10134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:11.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.899+0000 7f39ccd3e700 1 --2- 172.21.15.70:0/3030661955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 0x55eb10134660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.899+0000 7f39ccd3e700 1 -- 172.21.15.70:0/3030661955 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb10134db0 con 0x55eb1010a8a0 2024-06-04T05:19:11.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.899+0000 7f39ccd3e700 1 --2- 172.21.15.70:0/3030661955 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 0x55eb10125870 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0aa520 tx=0x7f39bc0cd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2803fb1d9aa8206 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:11.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.899+0000 7f39a7fff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:19:11.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.899+0000 7f39a7fff700 1 -- 172.21.15.70:0/3030661955 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc0aaf20 con 0x55eb1010a8a0 2024-06-04T05:19:11.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.899+0000 7f39a7fff700 1 -- 172.21.15.70:0/3030661955 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8615+0+0 (secure 0 0 0) 0x7f39bc0e0d70 con 0x55eb1010a8a0 2024-06-04T05:19:11.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.899+0000 7f39d9837b80 1 -- 172.21.15.70:0/3030661955 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39b4003cf0 con 0x55eb1010a8a0 2024-06-04T05:19:11.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.899+0000 7f39a7fff700 1 -- 172.21.15.70:0/3030661955 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f39bc0cd0b0 con 0x55eb1010a8a0 2024-06-04T05:19:11.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.899+0000 7f39d9837b80 10 client.29899 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:11.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.899+0000 7f39d9837b80 1 -- 172.21.15.70:0/3030661955 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55eb10197850 con 0x55eb1010a8a0 2024-06-04T05:19:11.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.900+0000 7f39a7fff700 1 -- 172.21.15.70:0/3030661955 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f39b4003cf0 con 0x55eb1010a8a0 2024-06-04T05:19:11.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.900+0000 7f39a7fff700 1 -- 172.21.15.70:0/3030661955 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f39bc022920 con 0x55eb1010a8a0 2024-06-04T05:19:11.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.900+0000 7f39d9837b80 10 client.29899 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:11.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.900+0000 7f39d9837b80 10 client.29899 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:19:11.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.900+0000 7f39d9837b80 20 client.29899 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:11.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.900+0000 7f39d9837b80 1 --2- 172.21.15.70:0/3030661955 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55eb10197f40 0x55eb1019a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:11.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.900+0000 7f39d9837b80 4 client.29899 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:19:11.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.900+0000 7f39d9837b80 1 -- 172.21.15.70:0/3030661955 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eb1006acc0 con 0x55eb10197f40 2024-06-04T05:19:11.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.900+0000 7f39c7fff700 1 --2- 172.21.15.70:0/3030661955 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55eb10197f40 0x55eb1019a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:11.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.901+0000 7f39c7fff700 1 --2- 172.21.15.70:0/3030661955 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55eb10197f40 0x55eb1019a360 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=122bd7ed302b05cc server_cookie=66070db31a0e3fa4 in_seq=0 out_seq=0 2024-06-04T05:19:11.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.901+0000 7f39a7fff700 10 client.29899 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:19:11.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.901+0000 7f39a7fff700 1 -- 172.21.15.70:0/3030661955 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eb1006acc0 con 0x55eb10197f40 2024-06-04T05:19:11.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.901+0000 7f39a7fff700 10 client.29899 handle_command_reply: tid=0 2024-06-04T05:19:11.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.901+0000 7f39d9837b80 1 client.29899 shutdown 2024-06-04T05:19:11.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.901+0000 7f39d9837b80 1 -- 172.21.15.70:0/3030661955 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 msgr2=0x55eb10125870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:11.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.901+0000 7f39d9837b80 1 --2- 172.21.15.70:0/3030661955 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 0x55eb10125870 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0aa520 tx=0x7f39bc0cd040 comp rx=0 tx=0).stop 2024-06-04T05:19:11.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.901+0000 7f39d9837b80 1 -- 172.21.15.70:0/3030661955 shutdown_connections 2024-06-04T05:19:11.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.901+0000 7f39d9837b80 1 --2- 172.21.15.70:0/3030661955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb10110320 0x55eb10134660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.901+0000 7f39d9837b80 1 --2- 172.21.15.70:0/3030661955 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55eb10197f40 0x55eb1019a360 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.901+0000 7f39d9837b80 1 --2- 172.21.15.70:0/3030661955 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55eb1010a8a0 0x55eb10125870 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.901+0000 7f39d9837b80 1 --2- 172.21.15.70:0/3030661955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb1010b260 0x55eb10134120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.901+0000 7f39d9837b80 1 -- 172.21.15.70:0/3030661955 >> 172.21.15.70:0/3030661955 conn(0x55eb10036240 msgr2=0x55eb1010dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:11.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.901+0000 7f39a7fff700 0 client.29899 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:19:11.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.901+0000 7f39d9837b80 1 -- 172.21.15.70:0/3030661955 shutdown_connections 2024-06-04T05:19:11.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.901+0000 7f39d9837b80 1 -- 172.21.15.70:0/3030661955 wait complete. 2024-06-04T05:19:11.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.902+0000 7f39d9837b80 20 client.29899 trim_cache size 0 max 16384 2024-06-04T05:19:11.907 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:19:11.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.902+0000 7f39cefa2700 1 -- 172.21.15.70:0/318953113 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39b80b2a70 msgr2=0x7f39b80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:11.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.902+0000 7f39cefa2700 1 --2- 172.21.15.70:0/318953113 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39b80b2a70 0x7f39b80b4f30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f39c8079b30 tx=0x7f39b4009260 comp rx=0 tx=0).stop 2024-06-04T05:19:11.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.902+0000 7f39cefa2700 1 -- 172.21.15.70:0/318953113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c8102440 msgr2=0x7f39c8111380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:11.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.902+0000 7f39cefa2700 1 --2- 172.21.15.70:0/318953113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c8102440 0x7f39c8111380 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f39bc006010 tx=0x7f39bc0057c0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.903+0000 7f39cefa2700 1 -- 172.21.15.70:0/318953113 shutdown_connections 2024-06-04T05:19:11.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.903+0000 7f39cefa2700 1 --2- 172.21.15.70:0/318953113 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f39b80b2a70 0x7f39b80b4f30 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.903+0000 7f39cefa2700 1 --2- 172.21.15.70:0/318953113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c8108520 0x7f39c8111e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.903+0000 7f39cefa2700 1 --2- 172.21.15.70:0/318953113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c8102440 0x7f39c8111380 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.903+0000 7f39cefa2700 1 --2- 172.21.15.70:0/318953113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39c8102e00 0x7f39c81118c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:11.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.903+0000 7f39cefa2700 1 -- 172.21.15.70:0/318953113 >> 172.21.15.70:0/318953113 conn(0x7f39c80793f0 msgr2=0x7f39c8101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:11.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.903+0000 7f39cefa2700 1 -- 172.21.15.70:0/318953113 shutdown_connections 2024-06-04T05:19:11.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:11.903+0000 7f39cefa2700 1 -- 172.21.15.70:0/318953113 wait complete. 2024-06-04T05:19:11.916 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:19:12.642 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:19:12.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 62087 6.74 MB/sec execute 364 sec latency 75.419 ms 2024-06-04T05:19:13.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:13 smithi070 ceph-mon[32631]: pgmap v693: 97 pgs: 97 active+clean; 878 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 454 KiB/s wr, 32 op/s 2024-06-04T05:19:13.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:13 smithi070 ceph-mon[32631]: osdmap e112: 12 total, 12 up, 12 in 2024-06-04T05:19:13.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.623+0000 7f3dc8254640 1 -- 172.21.15.70:0/3531975726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc0108a70 msgr2=0x7f3dc0108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:13.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.623+0000 7f3dc8254640 1 --2- 172.21.15.70:0/3531975726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc0108a70 0x7f3dc0108e50 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc009890 tx=0x7f3dbc02a360 comp rx=0 tx=0).stop 2024-06-04T05:19:13.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.623+0000 7f3dc8254640 1 -- 172.21.15.70:0/3531975726 shutdown_connections 2024-06-04T05:19:13.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.623+0000 7f3dc8254640 1 --2- 172.21.15.70:0/3531975726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc0102a60 0x7f3dc0102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.623+0000 7f3dc8254640 1 --2- 172.21.15.70:0/3531975726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3dc0103420 0x7f3dc010f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.623+0000 7f3dc8254640 1 --2- 172.21.15.70:0/3531975726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc0108a70 0x7f3dc0108e50 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.623+0000 7f3dc8254640 1 -- 172.21.15.70:0/3531975726 >> 172.21.15.70:0/3531975726 conn(0x7f3dc0079ee0 msgr2=0x7f3dc007a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:13.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.624+0000 7f3dc8254640 1 -- 172.21.15.70:0/3531975726 shutdown_connections 2024-06-04T05:19:13.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.624+0000 7f3dc8254640 1 -- 172.21.15.70:0/3531975726 wait complete. 2024-06-04T05:19:13.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.625+0000 7f3dc8254640 1 Processor -- start 2024-06-04T05:19:13.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.625+0000 7f3dc8254640 1 -- start start 2024-06-04T05:19:13.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.625+0000 7f3dc8254640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc0102a60 0x7f3dc01990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.625+0000 7f3dc8254640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3dc0103420 0x7f3dc0199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.625+0000 7f3dc8254640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc0108a70 0x7f3dc019d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.628 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.625+0000 7f3dc8254640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc0112bb0 con 0x7f3dc0102a60 2024-06-04T05:19:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.625+0000 7f3dc5ff0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc0102a60 0x7f3dc01990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.625+0000 7f3dc8254640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc0112a30 con 0x7f3dc0108a70 2024-06-04T05:19:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.625+0000 7f3dc8254640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc0112d30 con 0x7f3dc0103420 2024-06-04T05:19:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.625+0000 7f3dc5ff0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc0102a60 0x7f3dc01990d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33400/0 (socket says 172.21.15.70:33400) 2024-06-04T05:19:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.625+0000 7f3dc5ff0640 1 -- 172.21.15.70:0/1180169048 learned_addr learned my addr 172.21.15.70:0/1180169048 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.626+0000 7f3dc57ef640 1 --2- 172.21.15.70:0/1180169048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3dc0103420 0x7f3dc0199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.626+0000 7f3dc67f1640 1 --2- 172.21.15.70:0/1180169048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc0108a70 0x7f3dc019d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.626+0000 7f3dc5ff0640 1 -- 172.21.15.70:0/1180169048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3dc0103420 msgr2=0x7f3dc0199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.626+0000 7f3dc5ff0640 1 --2- 172.21.15.70:0/1180169048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3dc0103420 0x7f3dc0199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.626+0000 7f3dc5ff0640 1 -- 172.21.15.70:0/1180169048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc0108a70 msgr2=0x7f3dc019d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.626+0000 7f3dc5ff0640 1 --2- 172.21.15.70:0/1180169048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc0108a70 0x7f3dc019d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.626+0000 7f3dc5ff0640 1 -- 172.21.15.70:0/1180169048 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc019e0c0 con 0x7f3dc0102a60 2024-06-04T05:19:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.626+0000 7f3dc5ff0640 1 --2- 172.21.15.70:0/1180169048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc0102a60 0x7f3dc01990d0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc032880 tx=0x7f3dbc02adf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.626+0000 7f3db6ffd640 1 -- 172.21.15.70:0/1180169048 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc037040 con 0x7f3dc0102a60 2024-06-04T05:19:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.626+0000 7f3db6ffd640 1 -- 172.21.15.70:0/1180169048 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3dbc045030 con 0x7f3dc0102a60 2024-06-04T05:19:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.626+0000 7f3dc8254640 1 -- 172.21.15.70:0/1180169048 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc019e350 con 0x7f3dc0102a60 2024-06-04T05:19:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.627+0000 7f3db6ffd640 1 -- 172.21.15.70:0/1180169048 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc0044e0 con 0x7f3dc0102a60 2024-06-04T05:19:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.627+0000 7f3dc8254640 1 -- 172.21.15.70:0/1180169048 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc019e7d0 con 0x7f3dc0102a60 2024-06-04T05:19:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.630+0000 7f3dcaf6b740 1 Processor -- start 2024-06-04T05:19:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.630+0000 7f3db6ffd640 1 -- 172.21.15.70:0/1180169048 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3dbc0305a0 con 0x7f3dc0102a60 2024-06-04T05:19:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.630+0000 7f3dcaf6b740 1 -- start start 2024-06-04T05:19:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.630+0000 7f3dcaf6b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1b820 0x556da5f1bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.630+0000 7f3dcaf6b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1c140 0x556da5f264c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.630+0000 7f3dcaf6b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 0x556da5f28df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.630+0000 7f3dcaf6b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556da5e8dc30 con 0x556da5f1b820 2024-06-04T05:19:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.630+0000 7f3dcaf6b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556da5e7d7d0 con 0x556da5f26a00 2024-06-04T05:19:13.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.630+0000 7f3dcaf6b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556da5e81810 con 0x556da5f1c140 2024-06-04T05:19:13.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.631+0000 7f3dc5ff0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1c140 0x556da5f264c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.631+0000 7f3dc67f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1b820 0x556da5f1bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.631+0000 7f3dc67f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1b820 0x556da5f1bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33408/0 (socket says 172.21.15.70:33408) 2024-06-04T05:19:13.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.631+0000 7f3dc67f1640 1 -- 172.21.15.70:0/733993598 learned_addr learned my addr 172.21.15.70:0/733993598 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:13.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.631+0000 7f3dc57ef640 1 --2- 172.21.15.70:0/733993598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 0x556da5f28df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.631+0000 7f3dc57ef640 1 -- 172.21.15.70:0/733993598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1c140 msgr2=0x556da5f264c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:13.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.631+0000 7f3dc57ef640 1 --2- 172.21.15.70:0/733993598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1c140 0x556da5f264c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.631+0000 7f3dc57ef640 1 -- 172.21.15.70:0/733993598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1b820 msgr2=0x556da5f1bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:13.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.631+0000 7f3dc57ef640 1 --2- 172.21.15.70:0/733993598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1b820 0x556da5f1bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.631+0000 7f3dc57ef640 1 -- 172.21.15.70:0/733993598 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556da5b92210 con 0x556da5f26a00 2024-06-04T05:19:13.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.631+0000 7f3dc67f1640 1 --2- 172.21.15.70:0/733993598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1b820 0x556da5f1bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:19:13.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.631+0000 7f3dc57ef640 1 --2- 172.21.15.70:0/733993598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 0x556da5f28df0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3db0005fd0 tx=0x7f3db0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1710de7e4cbb6561 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:13.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.632+0000 7f3dabfff640 1 -- 172.21.15.70:0/733993598 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db001e070 con 0x556da5f26a00 2024-06-04T05:19:13.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.632+0000 7f3dabfff640 1 -- 172.21.15.70:0/733993598 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3db00035f0 con 0x556da5f26a00 2024-06-04T05:19:13.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.632+0000 7f3dabfff640 1 -- 172.21.15.70:0/733993598 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db00044e0 con 0x556da5f26a00 2024-06-04T05:19:13.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.632+0000 7f3db6ffd640 1 --2- 172.21.15.70:0/1180169048 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3dac0b2a90 0x7f3dac0b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.632+0000 7f3dc67f1640 1 --2- 172.21.15.70:0/1180169048 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3dac0b2a90 0x7f3dac0b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.633+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/733993598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 msgr2=0x556da5f28df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:13.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.633+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/733993598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 0x556da5f28df0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3db0005fd0 tx=0x7f3db0011810 comp rx=0 tx=0).stop 2024-06-04T05:19:13.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.633+0000 7f3dc67f1640 1 --2- 172.21.15.70:0/1180169048 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3dac0b2a90 0x7f3dac0b4f50 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3db8011790 tx=0x7f3db8012040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:13.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.633+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/733993598 shutdown_connections 2024-06-04T05:19:13.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.633+0000 7f3db6ffd640 1 -- 172.21.15.70:0/1180169048 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f3dbc10a230 con 0x7f3dc0102a60 2024-06-04T05:19:13.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.633+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/733993598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 0x556da5f28df0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.633+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/733993598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1c140 0x556da5f264c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.633+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/733993598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1b820 0x556da5f1bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.633+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/733993598 >> 172.21.15.70:0/733993598 conn(0x556da5f17d30 msgr2=0x556da5f18110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:13.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.633+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/733993598 shutdown_connections 2024-06-04T05:19:13.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.633+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/733993598 wait complete. 2024-06-04T05:19:13.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.634+0000 7f3dcaf6b740 1 -- start start 2024-06-04T05:19:13.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.634+0000 7f3dcaf6b740 1 Processor -- start 2024-06-04T05:19:13.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.634+0000 7f3dcaf6b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1b820 0x556da5f23980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.634+0000 7f3dcaf6b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1c140 0x556da5f23ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.634+0000 7f3dcaf6b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 0x556da5f24400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.634+0000 7f3dcaf6b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556da5b63790 con 0x556da5f1b820 2024-06-04T05:19:13.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.634+0000 7f3dcaf6b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556da5e39890 con 0x556da5f26a00 2024-06-04T05:19:13.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.634+0000 7f3dcaf6b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556da5f29630 con 0x556da5f1c140 2024-06-04T05:19:13.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.634+0000 7f3dcaf6b740 10 client.0 _ll_register_callbacks cb 0x556da5b91a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:13.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.635+0000 7f3dc57ef640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1c140 0x556da5f23ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.635+0000 7f3dc57ef640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1c140 0x556da5f23ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36156/0 (socket says 172.21.15.70:36156) 2024-06-04T05:19:13.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.635+0000 7f3dc57ef640 1 -- 172.21.15.70:0/2851547088 learned_addr learned my addr 172.21.15.70:0/2851547088 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:13.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.635+0000 7f3dc5ff0640 1 --2- 172.21.15.70:0/2851547088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1b820 0x556da5f23980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.635+0000 7f3dc67f1640 1 --2- 172.21.15.70:0/2851547088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 0x556da5f24400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.635+0000 7f3dc57ef640 1 -- 172.21.15.70:0/2851547088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 msgr2=0x556da5f24400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:13.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.635+0000 7f3dc57ef640 1 --2- 172.21.15.70:0/2851547088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 0x556da5f24400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.635+0000 7f3dc57ef640 1 -- 172.21.15.70:0/2851547088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1b820 msgr2=0x556da5f23980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:13.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.635+0000 7f3dc57ef640 1 --2- 172.21.15.70:0/2851547088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1b820 0x556da5f23980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.635+0000 7f3dc57ef640 1 -- 172.21.15.70:0/2851547088 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556da5c5e620 con 0x556da5f1c140 2024-06-04T05:19:13.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.635+0000 7f3dc67f1640 1 --2- 172.21.15.70:0/2851547088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 0x556da5f24400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:19:13.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.635+0000 7f3dc5ff0640 1 --2- 172.21.15.70:0/2851547088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1b820 0x556da5f23980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:19:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.635+0000 7f3dc57ef640 1 --2- 172.21.15.70:0/2851547088 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1c140 0x556da5f23ec0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3db001c040 tx=0x7f3db0003bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b78256debe38038 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.635+0000 7f3da97fa640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:19:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.636+0000 7f3da97fa640 1 -- 172.21.15.70:0/2851547088 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db0028020 con 0x556da5f1c140 2024-06-04T05:19:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.636+0000 7f3da97fa640 1 -- 172.21.15.70:0/2851547088 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f3db0004250 con 0x556da5f1c140 2024-06-04T05:19:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.636+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/2851547088 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3db00035f0 con 0x556da5f1c140 2024-06-04T05:19:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.636+0000 7f3da97fa640 1 -- 172.21.15.70:0/2851547088 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f3db001e040 con 0x556da5f1c140 2024-06-04T05:19:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.636+0000 7f3dcaf6b740 10 client.29911 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.636+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/2851547088 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x556da5f353d0 con 0x556da5f1c140 2024-06-04T05:19:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.636+0000 7f3da97fa640 1 -- 172.21.15.70:0/2851547088 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3db00035f0 con 0x556da5f1c140 2024-06-04T05:19:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.637+0000 7f3da97fa640 1 -- 172.21.15.70:0/2851547088 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f3db0021030 con 0x556da5f1c140 2024-06-04T05:19:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.637+0000 7f3dcaf6b740 10 client.29911 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.637+0000 7f3dcaf6b740 10 client.29911 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:19:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.637+0000 7f3dcaf6b740 20 client.29911 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.637+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/2851547088 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556da5f36780 0x556da5f38b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.637+0000 7f3dcaf6b740 4 client.29911 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:19:13.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.637+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/2851547088 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556da5e8fb80 con 0x556da5f36780 2024-06-04T05:19:13.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.637+0000 7f3dc5ff0640 1 --2- 172.21.15.70:0/2851547088 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556da5f36780 0x556da5f38b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.638+0000 7f3dc5ff0640 1 --2- 172.21.15.70:0/2851547088 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556da5f36780 0x556da5f38b70 crc :-1 s=READY pgs=463 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc9c0b369edbd5ae server_cookie=8ffd92d9e1746042 in_seq=0 out_seq=0 2024-06-04T05:19:13.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.638+0000 7f3da97fa640 10 client.29911 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:13.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.638+0000 7f3da97fa640 1 -- 172.21.15.70:0/2851547088 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556da5e8fb80 con 0x556da5f36780 2024-06-04T05:19:13.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.638+0000 7f3da97fa640 10 client.29911 handle_command_reply: tid=0 2024-06-04T05:19:13.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.638+0000 7f3dcaf6b740 1 client.29911 shutdown 2024-06-04T05:19:13.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.639+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/2851547088 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1c140 msgr2=0x556da5f23ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:13.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.639+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/2851547088 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1c140 0x556da5f23ec0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3db001c040 tx=0x7f3db0003bf0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.639+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/2851547088 shutdown_connections 2024-06-04T05:19:13.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.639+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/2851547088 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556da5f36780 0x556da5f38b70 crc :-1 s=READY pgs=463 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.639+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/2851547088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 0x556da5f24400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.639+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/2851547088 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1c140 0x556da5f23ec0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.639+0000 7f3da97fa640 0 client.29911 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:13.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.639+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/2851547088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1b820 0x556da5f23980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.639+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/2851547088 >> 172.21.15.70:0/2851547088 conn(0x556da5f17d30 msgr2=0x556da5f16fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:13.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.639+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/2851547088 shutdown_connections 2024-06-04T05:19:13.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.639+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/2851547088 wait complete. 2024-06-04T05:19:13.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.639+0000 7f3dcaf6b740 20 client.29911 trim_cache size 0 max 16384 2024-06-04T05:19:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.665+0000 7f3dcaf6b740 1 Processor -- start 2024-06-04T05:19:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.665+0000 7f3dcaf6b740 1 -- start start 2024-06-04T05:19:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.665+0000 7f3dcaf6b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1b820 0x556da5f17730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.665+0000 7f3dcaf6b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1c140 0x556da5f258c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.666+0000 7f3dcaf6b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 0x556da5f3e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.666+0000 7f3dcaf6b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc01ac420 con 0x556da5f1c140 2024-06-04T05:19:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.666+0000 7f3dcaf6b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc01adfc0 con 0x556da5f26a00 2024-06-04T05:19:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.666+0000 7f3dcaf6b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc01acb50 con 0x556da5f1b820 2024-06-04T05:19:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.666+0000 7f3dc57ef640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1c140 0x556da5f258c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.666+0000 7f3dc5ff0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1b820 0x556da5f17730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.666+0000 7f3dc5ff0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1b820 0x556da5f17730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36172/0 (socket says 172.21.15.70:36172) 2024-06-04T05:19:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.666+0000 7f3dc5ff0640 1 -- 172.21.15.70:0/4065855517 learned_addr learned my addr 172.21.15.70:0/4065855517 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.666+0000 7f3dc67f1640 1 --2- 172.21.15.70:0/4065855517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 0x556da5f3e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.666+0000 7f3dc5ff0640 1 -- 172.21.15.70:0/4065855517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 msgr2=0x556da5f3e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.666+0000 7f3dc5ff0640 1 --2- 172.21.15.70:0/4065855517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 0x556da5f3e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.666+0000 7f3dc5ff0640 1 -- 172.21.15.70:0/4065855517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1c140 msgr2=0x556da5f258c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.666+0000 7f3dc5ff0640 1 --2- 172.21.15.70:0/4065855517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1c140 0x556da5f258c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.666+0000 7f3dc5ff0640 1 -- 172.21.15.70:0/4065855517 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556da5f21ef0 con 0x556da5f1b820 2024-06-04T05:19:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.666+0000 7f3dc57ef640 1 --2- 172.21.15.70:0/4065855517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1c140 0x556da5f258c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:19:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.666+0000 7f3dc67f1640 1 --2- 172.21.15.70:0/4065855517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 0x556da5f3e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:19:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.666+0000 7f3dc5ff0640 1 --2- 172.21.15.70:0/4065855517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1b820 0x556da5f17730 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc005b60 tx=0x7f3dbc10dce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf2fd1b2bc24080b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.667+0000 7f3dab7fe640 1 -- 172.21.15.70:0/4065855517 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc04f040 con 0x556da5f1b820 2024-06-04T05:19:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.667+0000 7f3dab7fe640 1 -- 172.21.15.70:0/4065855517 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3dbc10de50 con 0x556da5f1b820 2024-06-04T05:19:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.667+0000 7f3dab7fe640 1 -- 172.21.15.70:0/4065855517 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc0d1590 con 0x556da5f1b820 2024-06-04T05:19:13.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.667+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/4065855517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1b820 msgr2=0x556da5f17730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:13.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.667+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/4065855517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1b820 0x556da5f17730 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc005b60 tx=0x7f3dbc10dce0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.668+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/4065855517 shutdown_connections 2024-06-04T05:19:13.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.668+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/4065855517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f26a00 0x556da5f3e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.668+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/4065855517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1c140 0x556da5f258c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.668+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/4065855517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f1b820 0x556da5f17730 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.668+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/4065855517 >> 172.21.15.70:0/4065855517 conn(0x556da5f17d30 msgr2=0x556da5f251b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:13.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.668+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/4065855517 shutdown_connections 2024-06-04T05:19:13.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.668+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/4065855517 wait complete. 2024-06-04T05:19:13.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.668+0000 7f3dcaf6b740 1 -- start start 2024-06-04T05:19:13.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.668+0000 7f3dcaf6b740 1 Processor -- start 2024-06-04T05:19:13.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dcaf6b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f1b820 0x556da5f2f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dcaf6b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1c140 0x556da5f20760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dcaf6b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f26a00 0x556da5f20cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dcaf6b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc01ad8f0 con 0x556da5f1c140 2024-06-04T05:19:13.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dcaf6b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc01ad220 con 0x556da5f1b820 2024-06-04T05:19:13.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dcaf6b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc01aa420 con 0x556da5f26a00 2024-06-04T05:19:13.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dc57ef640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1c140 0x556da5f20760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dc5ff0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f1b820 0x556da5f2f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dc5ff0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f1b820 0x556da5f2f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:60462/0 (socket says 172.21.15.70:60462) 2024-06-04T05:19:13.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dc57ef640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1c140 0x556da5f20760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33430/0 (socket says 172.21.15.70:33430) 2024-06-04T05:19:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dc5ff0640 1 -- 172.21.15.70:0/1357212127 learned_addr learned my addr 172.21.15.70:0/1357212127 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dc67f1640 1 --2- 172.21.15.70:0/1357212127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f26a00 0x556da5f20cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dcaf6b740 10 client.0 _ll_register_callbacks cb 0x7f3db001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dc5ff0640 1 -- 172.21.15.70:0/1357212127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f26a00 msgr2=0x556da5f20cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dc5ff0640 1 --2- 172.21.15.70:0/1357212127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f26a00 0x556da5f20cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dc5ff0640 1 -- 172.21.15.70:0/1357212127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1c140 msgr2=0x556da5f20760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dc5ff0640 1 --2- 172.21.15.70:0/1357212127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1c140 0x556da5f20760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dc5ff0640 1 -- 172.21.15.70:0/1357212127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556da5f214f0 con 0x556da5f1b820 2024-06-04T05:19:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dc57ef640 1 --2- 172.21.15.70:0/1357212127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1c140 0x556da5f20760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:19:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.669+0000 7f3dc67f1640 1 --2- 172.21.15.70:0/1357212127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f26a00 0x556da5f20cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:19:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.670+0000 7f3dc5ff0640 1 --2- 172.21.15.70:0/1357212127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f1b820 0x556da5f2f010 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc0d1290 tx=0x7f3dbc02a360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee185fd6cc1a7f83 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.670+0000 7f3da97fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:19:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.670+0000 7f3da97fa640 1 -- 172.21.15.70:0/1357212127 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc04f070 con 0x556da5f1b820 2024-06-04T05:19:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.670+0000 7f3da97fa640 1 -- 172.21.15.70:0/1357212127 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f3dbc10da50 con 0x556da5f1b820 2024-06-04T05:19:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.670+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/1357212127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3dbc10de50 con 0x556da5f1b820 2024-06-04T05:19:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.670+0000 7f3da97fa640 1 -- 172.21.15.70:0/1357212127 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f3dbc0d0d00 con 0x556da5f1b820 2024-06-04T05:19:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.670+0000 7f3dcaf6b740 10 client.29783 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.670+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/1357212127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x556da5fa12c0 con 0x556da5f1b820 2024-06-04T05:19:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.670+0000 7f3da97fa640 1 -- 172.21.15.70:0/1357212127 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3dbc10de50 con 0x556da5f1b820 2024-06-04T05:19:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.670+0000 7f3da97fa640 1 -- 172.21.15.70:0/1357212127 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f3dbc0cd820 con 0x556da5f1b820 2024-06-04T05:19:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.671+0000 7f3dcaf6b740 10 client.29783 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.671+0000 7f3dcaf6b740 10 client.29783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:19:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.671+0000 7f3dcaf6b740 20 client.29783 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.671+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/1357212127 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556da5f36780 0x556da5fa3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.671+0000 7f3dcaf6b740 4 client.29783 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:19:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.671+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/1357212127 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556da5e89360 con 0x556da5f36780 2024-06-04T05:19:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.671+0000 7f3dc57ef640 1 --2- 172.21.15.70:0/1357212127 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556da5f36780 0x556da5fa3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.671+0000 7f3da97fa640 10 client.29783 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.671+0000 7f3dc57ef640 1 --2- 172.21.15.70:0/1357212127 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556da5f36780 0x556da5fa3a90 crc :-1 s=READY pgs=464 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66c3dd516ee05fb7 server_cookie=e5c19cd9fd3ea2d3 in_seq=0 out_seq=0 2024-06-04T05:19:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:13 smithi112 ceph-mon[36453]: pgmap v693: 97 pgs: 97 active+clean; 878 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 454 KiB/s wr, 32 op/s 2024-06-04T05:19:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:13 smithi112 ceph-mon[36453]: osdmap e112: 12 total, 12 up, 12 in 2024-06-04T05:19:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.737+0000 7f3da97fa640 1 -- 172.21.15.70:0/1357212127 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x556da5e89360 con 0x556da5f36780 2024-06-04T05:19:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.737+0000 7f3da97fa640 10 client.29783 handle_command_reply: tid=0 2024-06-04T05:19:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.737+0000 7f3dcaf6b740 1 client.29783 shutdown 2024-06-04T05:19:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.737+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/1357212127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f1b820 msgr2=0x556da5f2f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.737+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/1357212127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f1b820 0x556da5f2f010 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc0d1290 tx=0x7f3dbc02a360 comp rx=0 tx=0).stop 2024-06-04T05:19:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.737+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/1357212127 shutdown_connections 2024-06-04T05:19:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.737+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/1357212127 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556da5f36780 0x556da5fa3a90 crc :-1 s=READY pgs=464 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.737+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/1357212127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556da5f26a00 0x556da5f20cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.737+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/1357212127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556da5f1c140 0x556da5f20760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.737+0000 7f3dcaf6b740 1 --2- 172.21.15.70:0/1357212127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556da5f1b820 0x556da5f2f010 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.737+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/1357212127 >> 172.21.15.70:0/1357212127 conn(0x556da5f17d30 msgr2=0x556da5f241d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.737+0000 7f3da97fa640 0 client.29783 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:13.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.737+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/1357212127 shutdown_connections 2024-06-04T05:19:13.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.737+0000 7f3dcaf6b740 1 -- 172.21.15.70:0/1357212127 wait complete. 2024-06-04T05:19:13.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.737+0000 7f3dcaf6b740 20 client.29783 trim_cache size 0 max 16384 2024-06-04T05:19:13.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:19:13.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:19:13.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:19:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.738+0000 7f3dc8254640 1 -- 172.21.15.70:0/1180169048 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3dac0b2a90 msgr2=0x7f3dac0b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.738+0000 7f3dc8254640 1 --2- 172.21.15.70:0/1180169048 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3dac0b2a90 0x7f3dac0b4f50 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3db8011790 tx=0x7f3db8012040 comp rx=0 tx=0).stop 2024-06-04T05:19:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.738+0000 7f3dc8254640 1 -- 172.21.15.70:0/1180169048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc0102a60 msgr2=0x7f3dc01990d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.738+0000 7f3dc8254640 1 --2- 172.21.15.70:0/1180169048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc0102a60 0x7f3dc01990d0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc032880 tx=0x7f3dbc02adf0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.738+0000 7f3dc8254640 1 -- 172.21.15.70:0/1180169048 shutdown_connections 2024-06-04T05:19:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.738+0000 7f3dc8254640 1 --2- 172.21.15.70:0/1180169048 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3dac0b2a90 0x7f3dac0b4f50 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.738+0000 7f3dc8254640 1 --2- 172.21.15.70:0/1180169048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc0108a70 0x7f3dc019d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.738+0000 7f3dc8254640 1 --2- 172.21.15.70:0/1180169048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3dc0103420 0x7f3dc0199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.738+0000 7f3dc8254640 1 --2- 172.21.15.70:0/1180169048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc0102a60 0x7f3dc01990d0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.738+0000 7f3dc8254640 1 -- 172.21.15.70:0/1180169048 >> 172.21.15.70:0/1180169048 conn(0x7f3dc0079ee0 msgr2=0x7f3dc01004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.738+0000 7f3dc8254640 1 -- 172.21.15.70:0/1180169048 shutdown_connections 2024-06-04T05:19:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:13.738+0000 7f3dc8254640 1 -- 172.21.15.70:0/1180169048 wait complete. 2024-06-04T05:19:13.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:13 smithi098 ceph-mon[35569]: pgmap v693: 97 pgs: 97 active+clean; 878 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 454 KiB/s wr, 32 op/s 2024-06-04T05:19:13.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:13 smithi098 ceph-mon[35569]: osdmap e112: 12 total, 12 up, 12 in 2024-06-04T05:19:13.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 62216 6.74 MB/sec execute 365 sec latency 84.075 ms 2024-06-04T05:19:13.830 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:19:14.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 62372 6.74 MB/sec execute 366 sec latency 158.509 ms 2024-06-04T05:19:15.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:15 smithi070 ceph-mon[32631]: pgmap v695: 97 pgs: 97 active+clean; 878 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 567 KiB/s wr, 34 op/s 2024-06-04T05:19:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:15 smithi112 ceph-mon[36453]: pgmap v695: 97 pgs: 97 active+clean; 878 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 567 KiB/s wr, 34 op/s 2024-06-04T05:19:15.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:15 smithi098 ceph-mon[35569]: pgmap v695: 97 pgs: 97 active+clean; 878 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 567 KiB/s wr, 34 op/s 2024-06-04T05:19:15.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 62568 6.76 MB/sec execute 367 sec latency 55.158 ms 2024-06-04T05:19:16.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.017+0000 7fe0b0d72700 1 -- 172.21.15.70:0/2684781709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0ac101b30 msgr2=0x7fe0ac101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:16.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.017+0000 7fe0b0d72700 1 --2- 172.21.15.70:0/2684781709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0ac101b30 0x7fe0ac101fb0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fe0a00099c0 tx=0x7fe0a0009cd0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.017+0000 7fe0b0d72700 1 -- 172.21.15.70:0/2684781709 shutdown_connections 2024-06-04T05:19:16.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.017+0000 7fe0b0d72700 1 --2- 172.21.15.70:0/2684781709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0ac1024f0 0x7fe0ac1103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.017+0000 7fe0b0d72700 1 --2- 172.21.15.70:0/2684781709 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe0ac109520 0x7fe0ac109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.017+0000 7fe0b0d72700 1 --2- 172.21.15.70:0/2684781709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0ac101b30 0x7fe0ac101fb0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.017+0000 7fe0b0d72700 1 -- 172.21.15.70:0/2684781709 >> 172.21.15.70:0/2684781709 conn(0x7fe0ac078ab0 msgr2=0x7fe0ac0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:16.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.017+0000 7fe0b0d72700 1 -- 172.21.15.70:0/2684781709 shutdown_connections 2024-06-04T05:19:16.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.017+0000 7fe0b0d72700 1 -- 172.21.15.70:0/2684781709 wait complete. 2024-06-04T05:19:16.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.018+0000 7fe0b0d72700 1 Processor -- start 2024-06-04T05:19:16.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.018+0000 7fe0b0d72700 1 -- start start 2024-06-04T05:19:16.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.018+0000 7fe0b0d72700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0ac101b30 0x7fe0ac102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.018+0000 7fe0b0d72700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0ac1024f0 0x7fe0ac103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.018+0000 7fe0b0d72700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe0ac109520 0x7fe0ac103790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.018+0000 7fe0b0d72700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe0ac113610 con 0x7fe0ac101b30 2024-06-04T05:19:16.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.018+0000 7fe0b0d72700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe0ac113490 con 0x7fe0ac1024f0 2024-06-04T05:19:16.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.018+0000 7fe0b0d72700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe0ac113790 con 0x7fe0ac109520 2024-06-04T05:19:16.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.019+0000 7fe0aad9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe0ac109520 0x7fe0ac103790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.019+0000 7fe0aad9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe0ac109520 0x7fe0ac103790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39590/0 (socket says 172.21.15.70:39590) 2024-06-04T05:19:16.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.019+0000 7fe0aad9d700 1 -- 172.21.15.70:0/2068111983 learned_addr learned my addr 172.21.15.70:0/2068111983 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:16.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.019+0000 7fe0a9d9b700 1 --2- 172.21.15.70:0/2068111983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0ac1024f0 0x7fe0ac103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.019+0000 7fe0aad9d700 1 -- 172.21.15.70:0/2068111983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0ac1024f0 msgr2=0x7fe0ac103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:16.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.019+0000 7fe0aad9d700 1 --2- 172.21.15.70:0/2068111983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0ac1024f0 0x7fe0ac103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.019+0000 7fe0aad9d700 1 -- 172.21.15.70:0/2068111983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0ac101b30 msgr2=0x7fe0ac102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:16.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.019+0000 7fe0aa59c700 1 --2- 172.21.15.70:0/2068111983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0ac101b30 0x7fe0ac102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.019+0000 7fe0aad9d700 1 --2- 172.21.15.70:0/2068111983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0ac101b30 0x7fe0ac102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.019+0000 7fe0aad9d700 1 -- 172.21.15.70:0/2068111983 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0ac19dce0 con 0x7fe0ac109520 2024-06-04T05:19:16.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.019+0000 7fe0aa59c700 1 --2- 172.21.15.70:0/2068111983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0ac101b30 0x7fe0ac102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:19:16.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.020+0000 7fe0aad9d700 1 --2- 172.21.15.70:0/2068111983 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe0ac109520 0x7fe0ac103790 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400ef30 tx=0x7fe0a400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:16.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.020+0000 7fe09b7fe700 1 -- 172.21.15.70:0/2068111983 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a40092e0 con 0x7fe0ac109520 2024-06-04T05:19:16.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.020+0000 7fe0b0d72700 1 -- 172.21.15.70:0/2068111983 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0ac19df70 con 0x7fe0ac109520 2024-06-04T05:19:16.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.020+0000 7fe09b7fe700 1 -- 172.21.15.70:0/2068111983 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0a4004bb0 con 0x7fe0ac109520 2024-06-04T05:19:16.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.020+0000 7fe09b7fe700 1 -- 172.21.15.70:0/2068111983 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4004020 con 0x7fe0ac109520 2024-06-04T05:19:16.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.020+0000 7fe0b0d72700 1 -- 172.21.15.70:0/2068111983 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0ac19e430 con 0x7fe0ac109520 2024-06-04T05:19:16.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.022+0000 7fe0b0d72700 1 -- 172.21.15.70:0/2068111983 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe08c005330 con 0x7fe0ac109520 2024-06-04T05:19:16.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.023+0000 7fe09b7fe700 1 -- 172.21.15.70:0/2068111983 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fe0a401b030 con 0x7fe0ac109520 2024-06-04T05:19:16.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.024+0000 7fe09b7fe700 1 --2- 172.21.15.70:0/2068111983 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe0940b29f0 0x7fe0940b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.025+0000 7fe0aa59c700 1 --2- 172.21.15.70:0/2068111983 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe0940b29f0 0x7fe0940b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.025+0000 7fe09b7fe700 1 -- 172.21.15.70:0/2068111983 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7fe0a40e20c0 con 0x7fe0ac109520 2024-06-04T05:19:16.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.025+0000 7fe0aa59c700 1 --2- 172.21.15.70:0/2068111983 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe0940b29f0 0x7fe0940b4eb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009bc0 tx=0x7fe09c009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:16.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.032+0000 7fe09b7fe700 1 -- 172.21.15.70:0/2068111983 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe0a4022020 con 0x7fe0ac109520 2024-06-04T05:19:16.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.458+0000 7fe0b0d72700 1 -- 172.21.15.70:0/2068111983 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe08c0058f0 con 0x7fe0ac109520 2024-06-04T05:19:16.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.459+0000 7fe09b7fe700 1 -- 172.21.15.70:0/2068111983 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+11932 (secure 0 0 0) 0x7fe0a4010040 con 0x7fe0ac109520 2024-06-04T05:19:16.463 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:19:16.463 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":59,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:17:20.010341+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:19:16.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.463+0000 7fe0b0d72700 1 -- 172.21.15.70:0/2068111983 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe0940b29f0 msgr2=0x7fe0940b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:16.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.463+0000 7fe0b0d72700 1 --2- 172.21.15.70:0/2068111983 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe0940b29f0 0x7fe0940b4eb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009bc0 tx=0x7fe09c009260 comp rx=0 tx=0).stop 2024-06-04T05:19:16.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.463+0000 7fe0b0d72700 1 -- 172.21.15.70:0/2068111983 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe0ac109520 msgr2=0x7fe0ac103790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:16.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.463+0000 7fe0b0d72700 1 --2- 172.21.15.70:0/2068111983 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe0ac109520 0x7fe0ac103790 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400ef30 tx=0x7fe0a400c3a0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.464+0000 7fe0b0d72700 1 -- 172.21.15.70:0/2068111983 shutdown_connections 2024-06-04T05:19:16.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.464+0000 7fe0b0d72700 1 --2- 172.21.15.70:0/2068111983 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe0940b29f0 0x7fe0940b4eb0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.464+0000 7fe0b0d72700 1 --2- 172.21.15.70:0/2068111983 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe0ac109520 0x7fe0ac103790 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.464+0000 7fe0b0d72700 1 --2- 172.21.15.70:0/2068111983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0ac101b30 0x7fe0ac102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.464+0000 7fe0b0d72700 1 --2- 172.21.15.70:0/2068111983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0ac1024f0 0x7fe0ac103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.464+0000 7fe0b0d72700 1 -- 172.21.15.70:0/2068111983 >> 172.21.15.70:0/2068111983 conn(0x7fe0ac078ab0 msgr2=0x7fe0ac1003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:16.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.464+0000 7fe0b0d72700 1 -- 172.21.15.70:0/2068111983 shutdown_connections 2024-06-04T05:19:16.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.464+0000 7fe0b0d72700 1 -- 172.21.15.70:0/2068111983 wait complete. 2024-06-04T05:19:16.468 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 59 2024-06-04T05:19:16.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.508+0000 7f6311ab9700 1 -- 172.21.15.70:0/1764817553 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f630c077a70 msgr2=0x7f630c077e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:16.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.508+0000 7f6311ab9700 1 --2- 172.21.15.70:0/1764817553 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f630c077a70 0x7f630c077e50 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009990 tx=0x7f62fc009ca0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.509+0000 7f6311ab9700 1 -- 172.21.15.70:0/1764817553 shutdown_connections 2024-06-04T05:19:16.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.509+0000 7f6311ab9700 1 --2- 172.21.15.70:0/1764817553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f630c114400 0x7f630c1167f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.509+0000 7f6311ab9700 1 --2- 172.21.15.70:0/1764817553 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f630c077a70 0x7f630c077e50 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.509+0000 7f6311ab9700 1 --2- 172.21.15.70:0/1764817553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f630c078390 0x7f630c113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.509+0000 7f6311ab9700 1 -- 172.21.15.70:0/1764817553 >> 172.21.15.70:0/1764817553 conn(0x7f630c101250 msgr2=0x7f630c103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:16.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.509+0000 7f6311ab9700 1 -- 172.21.15.70:0/1764817553 shutdown_connections 2024-06-04T05:19:16.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.509+0000 7f6311ab9700 1 -- 172.21.15.70:0/1764817553 wait complete. 2024-06-04T05:19:16.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.510+0000 7f6311ab9700 1 Processor -- start 2024-06-04T05:19:16.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.510+0000 7f6311ab9700 1 -- start start 2024-06-04T05:19:16.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.510+0000 7f6311ab9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f630c077a70 0x7f630c06ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.510+0000 7f6311ab9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f630c078390 0x7f630c06f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.510+0000 7f6311ab9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f630c114400 0x7f630c073500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.510+0000 7f6311ab9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f630c06d570 con 0x7f630c078390 2024-06-04T05:19:16.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.510+0000 7f6311ab9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f630c06d3f0 con 0x7f630c077a70 2024-06-04T05:19:16.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.510+0000 7f6311ab9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f630c06d6f0 con 0x7f630c114400 2024-06-04T05:19:16.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.510+0000 7f630affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f630c078390 0x7f630c06f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.510+0000 7f630bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f630c114400 0x7f630c073500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.510+0000 7f630bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f630c114400 0x7f630c073500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39594/0 (socket says 172.21.15.70:39594) 2024-06-04T05:19:16.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.510+0000 7f630bfff700 1 -- 172.21.15.70:0/2612163024 learned_addr learned my addr 172.21.15.70:0/2612163024 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:16.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.511+0000 7f630b7fe700 1 --2- 172.21.15.70:0/2612163024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f630c077a70 0x7f630c06ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.511+0000 7f630bfff700 1 -- 172.21.15.70:0/2612163024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f630c077a70 msgr2=0x7f630c06ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:16.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.511+0000 7f630bfff700 1 --2- 172.21.15.70:0/2612163024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f630c077a70 0x7f630c06ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.511+0000 7f630bfff700 1 -- 172.21.15.70:0/2612163024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f630c078390 msgr2=0x7f630c06f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:16.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.511+0000 7f630bfff700 1 --2- 172.21.15.70:0/2612163024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f630c078390 0x7f630c06f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.511+0000 7f630bfff700 1 -- 172.21.15.70:0/2612163024 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630c073c00 con 0x7f630c114400 2024-06-04T05:19:16.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.511+0000 7f630affd700 1 --2- 172.21.15.70:0/2612163024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f630c078390 0x7f630c06f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:19:16.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.511+0000 7f630bfff700 1 --2- 172.21.15.70:0/2612163024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f630c114400 0x7f630c073500 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6304009fd0 tx=0x7f630400ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:16.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.511+0000 7f6308ff9700 1 -- 172.21.15.70:0/2612163024 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63040092e0 con 0x7f630c114400 2024-06-04T05:19:16.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.511+0000 7f6308ff9700 1 -- 172.21.15.70:0/2612163024 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6304004bb0 con 0x7f630c114400 2024-06-04T05:19:16.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.512+0000 7f6308ff9700 1 -- 172.21.15.70:0/2612163024 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6304004020 con 0x7f630c114400 2024-06-04T05:19:16.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.512+0000 7f6311ab9700 1 -- 172.21.15.70:0/2612163024 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f630c073e90 con 0x7f630c114400 2024-06-04T05:19:16.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.512+0000 7f6311ab9700 1 -- 172.21.15.70:0/2612163024 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f630c0742c0 con 0x7f630c114400 2024-06-04T05:19:16.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.516+0000 7f6308ff9700 1 -- 172.21.15.70:0/2612163024 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f630401b030 con 0x7f630c114400 2024-06-04T05:19:16.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.517+0000 7f6308ff9700 1 --2- 172.21.15.70:0/2612163024 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62f40b2a70 0x7f62f40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.517+0000 7f630b7fe700 1 --2- 172.21.15.70:0/2612163024 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62f40b2a70 0x7f62f40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.517+0000 7f6308ff9700 1 -- 172.21.15.70:0/2612163024 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f63040e20b0 con 0x7f630c114400 2024-06-04T05:19:16.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.517+0000 7f630b7fe700 1 --2- 172.21.15.70:0/2612163024 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62f40b2a70 0x7f62f40b4f30 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00b2e0 tx=0x7f62fc01a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:16.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.518+0000 7f631c34eb80 1 Processor -- start 2024-06-04T05:19:16.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.518+0000 7f631c34eb80 1 -- start start 2024-06-04T05:19:16.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.518+0000 7f631c34eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890a860 0x563dc890ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.518+0000 7f631c34eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890b210 0x563dc890e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.518+0000 7f631c34eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890ef60 0x563dc890f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.518+0000 7f631c34eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563dc8869c90 con 0x563dc890ef60 2024-06-04T05:19:16.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.518+0000 7f631c34eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563dc8859830 con 0x563dc890a860 2024-06-04T05:19:16.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.518+0000 7f631c34eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563dc885d870 con 0x563dc890b210 2024-06-04T05:19:16.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.518+0000 7f630b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890a860 0x563dc890ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.518+0000 7f630b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890a860 0x563dc890ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33232/0 (socket says 172.21.15.70:33232) 2024-06-04T05:19:16.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.518+0000 7f630b7fe700 1 -- 172.21.15.70:0/1969206567 learned_addr learned my addr 172.21.15.70:0/1969206567 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:16.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.518+0000 7f630affd700 1 --2- 172.21.15.70:0/1969206567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890b210 0x563dc890e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.518+0000 7f630bfff700 1 --2- 172.21.15.70:0/1969206567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890ef60 0x563dc890f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.519+0000 7f630b7fe700 1 -- 172.21.15.70:0/1969206567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890b210 msgr2=0x563dc890e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:16.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.519+0000 7f630b7fe700 1 --2- 172.21.15.70:0/1969206567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890b210 0x563dc890e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.519+0000 7f630b7fe700 1 -- 172.21.15.70:0/1969206567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890ef60 msgr2=0x563dc890f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:16.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.519+0000 7f630b7fe700 1 --2- 172.21.15.70:0/1969206567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890ef60 0x563dc890f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.519+0000 7f630b7fe700 1 -- 172.21.15.70:0/1969206567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dc87a98d0 con 0x563dc890a860 2024-06-04T05:19:16.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.519+0000 7f630affd700 1 --2- 172.21.15.70:0/1969206567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890b210 0x563dc890e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:19:16.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.519+0000 7f630bfff700 1 --2- 172.21.15.70:0/1969206567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890ef60 0x563dc890f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:19:16.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.519+0000 7f630b7fe700 1 --2- 172.21.15.70:0/1969206567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890a860 0x563dc890ac40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00bda0 tx=0x7f62fc01e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a703e1a1c39795b0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:16.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.519+0000 7f62f1ffb700 1 -- 172.21.15.70:0/1969206567 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc009510 con 0x563dc890a860 2024-06-04T05:19:16.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.519+0000 7f62f1ffb700 1 -- 172.21.15.70:0/1969206567 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f62fc004b20 con 0x563dc890a860 2024-06-04T05:19:16.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.519+0000 7f62f1ffb700 1 -- 172.21.15.70:0/1969206567 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc02fb30 con 0x563dc890a860 2024-06-04T05:19:16.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.520+0000 7f631c34eb80 1 -- 172.21.15.70:0/1969206567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890a860 msgr2=0x563dc890ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:16.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.520+0000 7f631c34eb80 1 --2- 172.21.15.70:0/1969206567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890a860 0x563dc890ac40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00bda0 tx=0x7f62fc01e000 comp rx=0 tx=0).stop 2024-06-04T05:19:16.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.520+0000 7f631c34eb80 1 -- 172.21.15.70:0/1969206567 shutdown_connections 2024-06-04T05:19:16.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.520+0000 7f631c34eb80 1 --2- 172.21.15.70:0/1969206567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890ef60 0x563dc890f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.520+0000 7f631c34eb80 1 --2- 172.21.15.70:0/1969206567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890a860 0x563dc890ac40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.520+0000 7f631c34eb80 1 --2- 172.21.15.70:0/1969206567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890b210 0x563dc890e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.520+0000 7f631c34eb80 1 -- 172.21.15.70:0/1969206567 >> 172.21.15.70:0/1969206567 conn(0x563dc8838080 msgr2=0x563dc8904e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:16.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.520+0000 7f631c34eb80 1 -- 172.21.15.70:0/1969206567 shutdown_connections 2024-06-04T05:19:16.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.521+0000 7f631c34eb80 1 -- 172.21.15.70:0/1969206567 wait complete. 2024-06-04T05:19:16.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.521+0000 7f631c34eb80 1 -- start start 2024-06-04T05:19:16.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.521+0000 7f631c34eb80 1 Processor -- start 2024-06-04T05:19:16.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f631c34eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890a860 0x563dc8916440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f631c34eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 0x563dc8913740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f630b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890a860 0x563dc8916440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f631c34eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890ef60 0x563dc8913c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f630b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890a860 0x563dc8916440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35814/0 (socket says 172.21.15.70:35814) 2024-06-04T05:19:16.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f630b7fe700 1 -- 172.21.15.70:0/2036106078 learned_addr learned my addr 172.21.15.70:0/2036106078 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:16.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f631c34eb80 1 -- 172.21.15.70:0/2036106078 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563dc875ffa0 con 0x563dc890a860 2024-06-04T05:19:16.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f631c34eb80 1 -- 172.21.15.70:0/2036106078 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563dc87fb7c0 con 0x563dc890b210 2024-06-04T05:19:16.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f631c34eb80 1 -- 172.21.15.70:0/2036106078 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563dc89170c0 con 0x563dc890ef60 2024-06-04T05:19:16.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f631c34eb80 10 client.0 _ll_register_callbacks cb 0x563dc87f4920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:16.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f630affd700 1 --2- 172.21.15.70:0/2036106078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 0x563dc8913740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f630bfff700 1 --2- 172.21.15.70:0/2036106078 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890ef60 0x563dc8913c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f630bfff700 1 -- 172.21.15.70:0/2036106078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 msgr2=0x563dc8913740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:16.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f630bfff700 1 --2- 172.21.15.70:0/2036106078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 0x563dc8913740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f630bfff700 1 -- 172.21.15.70:0/2036106078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890a860 msgr2=0x563dc8916440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:16.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f630bfff700 1 --2- 172.21.15.70:0/2036106078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890a860 0x563dc8916440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f630bfff700 1 -- 172.21.15.70:0/2036106078 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dc87f8690 con 0x563dc890ef60 2024-06-04T05:19:16.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.522+0000 7f630b7fe700 1 --2- 172.21.15.70:0/2036106078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890a860 0x563dc8916440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:19:16.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.523+0000 7f630bfff700 1 --2- 172.21.15.70:0/2036106078 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890ef60 0x563dc8913c80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f63040df770 tx=0x7f63040b1fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef73ac72e8b279ff server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:16.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.523+0000 7f62eaffd700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:19:16.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.523+0000 7f62eaffd700 1 -- 172.21.15.70:0/2036106078 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63040c3060 con 0x563dc890ef60 2024-06-04T05:19:16.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.523+0000 7f631c34eb80 1 -- 172.21.15.70:0/2036106078 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62fc004b20 con 0x563dc890ef60 2024-06-04T05:19:16.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.523+0000 7f62eaffd700 1 -- 172.21.15.70:0/2036106078 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f63040a9820 con 0x563dc890ef60 2024-06-04T05:19:16.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.523+0000 7f62eaffd700 1 -- 172.21.15.70:0/2036106078 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f63040ca050 con 0x563dc890ef60 2024-06-04T05:19:16.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.523+0000 7f631c34eb80 10 client.29935 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:16.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.523+0000 7f631c34eb80 1 -- 172.21.15.70:0/2036106078 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x563dc8921700 con 0x563dc890ef60 2024-06-04T05:19:16.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.523+0000 7f62eaffd700 1 -- 172.21.15.70:0/2036106078 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f62fc004b20 con 0x563dc890ef60 2024-06-04T05:19:16.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.524+0000 7f62eaffd700 1 -- 172.21.15.70:0/2036106078 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f63040ca1f0 con 0x563dc890ef60 2024-06-04T05:19:16.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.524+0000 7f631c34eb80 10 client.29935 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:16.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.524+0000 7f631c34eb80 10 client.29935 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:19:16.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.524+0000 7f631c34eb80 20 client.29935 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:16.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.524+0000 7f631c34eb80 1 --2- 172.21.15.70:0/2036106078 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x563dc8924000 0x563dc8926420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.524+0000 7f631c34eb80 4 client.29935 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:19:16.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.524+0000 7f631c34eb80 1 -- 172.21.15.70:0/2036106078 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563dc88d69e0 con 0x563dc8924000 2024-06-04T05:19:16.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.524+0000 7f630b7fe700 1 --2- 172.21.15.70:0/2036106078 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x563dc8924000 0x563dc8926420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.524+0000 7f630b7fe700 1 --2- 172.21.15.70:0/2036106078 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x563dc8924000 0x563dc8926420 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=51a2034c0086324d server_cookie=f1cee0ef42eb5de7 in_seq=0 out_seq=0 2024-06-04T05:19:16.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.524+0000 7f62eaffd700 10 client.29935 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:19:16.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.527+0000 7f62eaffd700 1 -- 172.21.15.70:0/2036106078 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563dc88d69e0 con 0x563dc8924000 2024-06-04T05:19:16.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.527+0000 7f62eaffd700 10 client.29935 handle_command_reply: tid=0 2024-06-04T05:19:16.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.527+0000 7f631c34eb80 1 client.29935 shutdown 2024-06-04T05:19:16.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.527+0000 7f631c34eb80 1 -- 172.21.15.70:0/2036106078 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890ef60 msgr2=0x563dc8913c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:16.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.527+0000 7f631c34eb80 1 --2- 172.21.15.70:0/2036106078 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890ef60 0x563dc8913c80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f63040df770 tx=0x7f63040b1fb0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.527+0000 7f631c34eb80 1 -- 172.21.15.70:0/2036106078 shutdown_connections 2024-06-04T05:19:16.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.527+0000 7f631c34eb80 1 --2- 172.21.15.70:0/2036106078 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890ef60 0x563dc8913c80 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.527+0000 7f631c34eb80 1 --2- 172.21.15.70:0/2036106078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890a860 0x563dc8916440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.527+0000 7f631c34eb80 1 --2- 172.21.15.70:0/2036106078 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x563dc8924000 0x563dc8926420 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.527+0000 7f631c34eb80 1 --2- 172.21.15.70:0/2036106078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 0x563dc8913740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.527+0000 7f631c34eb80 1 -- 172.21.15.70:0/2036106078 >> 172.21.15.70:0/2036106078 conn(0x563dc8838080 msgr2=0x563dc8904510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:16.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.527+0000 7f62eaffd700 0 client.29935 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:19:16.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.528+0000 7f631c34eb80 1 -- 172.21.15.70:0/2036106078 shutdown_connections 2024-06-04T05:19:16.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.528+0000 7f631c34eb80 1 -- 172.21.15.70:0/2036106078 wait complete. 2024-06-04T05:19:16.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.528+0000 7f631c34eb80 20 client.29935 trim_cache size 0 max 16384 2024-06-04T05:19:16.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.551+0000 7f631c34eb80 1 Processor -- start 2024-06-04T05:19:16.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.551+0000 7f631c34eb80 1 -- start start 2024-06-04T05:19:16.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.552+0000 7f631c34eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890a860 0x563dc89123a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.552+0000 7f631c34eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 0x563dc892bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.552+0000 7f631c34eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890ef60 0x563dc892c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.552+0000 7f631c34eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f630c1bcb20 con 0x563dc890a860 2024-06-04T05:19:16.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.552+0000 7f631c34eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f630c1be6c0 con 0x563dc890b210 2024-06-04T05:19:16.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.552+0000 7f630affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 0x563dc892bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.552+0000 7f631c34eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f630c1bd250 con 0x563dc890ef60 2024-06-04T05:19:16.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.552+0000 7f630affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 0x563dc892bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33238/0 (socket says 172.21.15.70:33238) 2024-06-04T05:19:16.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.552+0000 7f630affd700 1 -- 172.21.15.70:0/123659233 learned_addr learned my addr 172.21.15.70:0/123659233 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:16.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.552+0000 7f630b7fe700 1 --2- 172.21.15.70:0/123659233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890a860 0x563dc89123a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.552+0000 7f630bfff700 1 --2- 172.21.15.70:0/123659233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890ef60 0x563dc892c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.552+0000 7f630affd700 1 -- 172.21.15.70:0/123659233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890ef60 msgr2=0x563dc892c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:16.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.552+0000 7f630affd700 1 --2- 172.21.15.70:0/123659233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890ef60 0x563dc892c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.552+0000 7f630affd700 1 -- 172.21.15.70:0/123659233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890a860 msgr2=0x563dc89123a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:16.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.552+0000 7f630affd700 1 --2- 172.21.15.70:0/123659233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890a860 0x563dc89123a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.552+0000 7f630affd700 1 -- 172.21.15.70:0/123659233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dc8911cb0 con 0x563dc890b210 2024-06-04T05:19:16.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.553+0000 7f630affd700 1 --2- 172.21.15.70:0/123659233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 0x563dc892bd30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6300012820 tx=0x7f6300009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d33a43075f279fd3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:16.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.553+0000 7f62f17fa700 1 -- 172.21.15.70:0/123659233 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6300004030 con 0x563dc890b210 2024-06-04T05:19:16.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.553+0000 7f62f17fa700 1 -- 172.21.15.70:0/123659233 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6300003680 con 0x563dc890b210 2024-06-04T05:19:16.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.553+0000 7f62f17fa700 1 -- 172.21.15.70:0/123659233 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630000f6c0 con 0x563dc890b210 2024-06-04T05:19:16.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.554+0000 7f631c34eb80 1 -- 172.21.15.70:0/123659233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 msgr2=0x563dc892bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:16.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.554+0000 7f631c34eb80 1 --2- 172.21.15.70:0/123659233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 0x563dc892bd30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6300012820 tx=0x7f6300009260 comp rx=0 tx=0).stop 2024-06-04T05:19:16.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.554+0000 7f631c34eb80 1 -- 172.21.15.70:0/123659233 shutdown_connections 2024-06-04T05:19:16.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.554+0000 7f631c34eb80 1 --2- 172.21.15.70:0/123659233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890ef60 0x563dc892c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.554+0000 7f631c34eb80 1 --2- 172.21.15.70:0/123659233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890a860 0x563dc89123a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.554+0000 7f631c34eb80 1 --2- 172.21.15.70:0/123659233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 0x563dc892bd30 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.554+0000 7f631c34eb80 1 -- 172.21.15.70:0/123659233 >> 172.21.15.70:0/123659233 conn(0x563dc8838080 msgr2=0x563dc8909f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:16.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.554+0000 7f631c34eb80 1 -- 172.21.15.70:0/123659233 shutdown_connections 2024-06-04T05:19:16.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.554+0000 7f631c34eb80 1 -- 172.21.15.70:0/123659233 wait complete. 2024-06-04T05:19:16.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.555+0000 7f631c34eb80 1 -- start start 2024-06-04T05:19:16.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.555+0000 7f631c34eb80 1 Processor -- start 2024-06-04T05:19:16.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.555+0000 7f631c34eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890a860 0x563dc8987000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.555+0000 7f631c34eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 0x563dc8987540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.555+0000 7f631c34eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890ef60 0x563dc8919a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.555+0000 7f631c34eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f630c1bdff0 con 0x563dc890ef60 2024-06-04T05:19:16.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.555+0000 7f631c34eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f630c1bd920 con 0x563dc890b210 2024-06-04T05:19:16.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.555+0000 7f631c34eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f630c1baae0 con 0x563dc890a860 2024-06-04T05:19:16.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.555+0000 7f631c34eb80 10 client.0 _ll_register_callbacks cb 0x563dc87f4920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:16.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.555+0000 7f630bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890ef60 0x563dc8919a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.555+0000 7f630bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890ef60 0x563dc8919a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35822/0 (socket says 172.21.15.70:35822) 2024-06-04T05:19:16.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.556+0000 7f630bfff700 1 -- 172.21.15.70:0/2323888793 learned_addr learned my addr 172.21.15.70:0/2323888793 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:16.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.556+0000 7f630affd700 1 --2- 172.21.15.70:0/2323888793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 0x563dc8987540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.556+0000 7f630b7fe700 1 --2- 172.21.15.70:0/2323888793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890a860 0x563dc8987000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.556+0000 7f630affd700 1 -- 172.21.15.70:0/2323888793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890a860 msgr2=0x563dc8987000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:16.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.556+0000 7f630affd700 1 --2- 172.21.15.70:0/2323888793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890a860 0x563dc8987000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.556+0000 7f630affd700 1 -- 172.21.15.70:0/2323888793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890ef60 msgr2=0x563dc8919a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:16.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.556+0000 7f630affd700 1 --2- 172.21.15.70:0/2323888793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890ef60 0x563dc8919a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.556+0000 7f630affd700 1 -- 172.21.15.70:0/2323888793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dc891a110 con 0x563dc890b210 2024-06-04T05:19:16.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.556+0000 7f630bfff700 1 --2- 172.21.15.70:0/2323888793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890ef60 0x563dc8919a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:19:16.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.556+0000 7f630affd700 1 --2- 172.21.15.70:0/2323888793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 0x563dc8987540 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x563dc89148d0 tx=0x7f63000123c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcbd92a6d0db4c30 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:16.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.556+0000 7f62eaffd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:19:16.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.556+0000 7f62eaffd700 1 -- 172.21.15.70:0/2323888793 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63000043b0 con 0x563dc890b210 2024-06-04T05:19:16.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.556+0000 7f631c34eb80 1 -- 172.21.15.70:0/2323888793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6300003680 con 0x563dc890b210 2024-06-04T05:19:16.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.556+0000 7f62eaffd700 1 -- 172.21.15.70:0/2323888793 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f630000f480 con 0x563dc890b210 2024-06-04T05:19:16.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.556+0000 7f62eaffd700 1 -- 172.21.15.70:0/2323888793 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f6300004550 con 0x563dc890b210 2024-06-04T05:19:16.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.557+0000 7f631c34eb80 10 client.29807 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:16.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.557+0000 7f631c34eb80 1 -- 172.21.15.70:0/2323888793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x563dc898e7f0 con 0x563dc890b210 2024-06-04T05:19:16.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.557+0000 7f62eaffd700 1 -- 172.21.15.70:0/2323888793 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6300003680 con 0x563dc890b210 2024-06-04T05:19:16.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.557+0000 7f62eaffd700 1 -- 172.21.15.70:0/2323888793 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f6300036070 con 0x563dc890b210 2024-06-04T05:19:16.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.557+0000 7f631c34eb80 10 client.29807 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:16.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.557+0000 7f631c34eb80 10 client.29807 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:19:16.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.557+0000 7f631c34eb80 20 client.29807 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:16.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.557+0000 7f631c34eb80 1 --2- 172.21.15.70:0/2323888793 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x563dc898edd0 0x563dc89911f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:16.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.557+0000 7f631c34eb80 4 client.29807 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:19:16.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.557+0000 7f631c34eb80 1 -- 172.21.15.70:0/2323888793 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563dc886bbe0 con 0x563dc898edd0 2024-06-04T05:19:16.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.557+0000 7f630b7fe700 1 --2- 172.21.15.70:0/2323888793 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x563dc898edd0 0x563dc89911f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:16.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.558+0000 7f62eaffd700 10 client.29807 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:19:16.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.558+0000 7f630b7fe700 1 --2- 172.21.15.70:0/2323888793 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x563dc898edd0 0x563dc89911f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2ab4bde6d57919 server_cookie=919632c5f168601d in_seq=0 out_seq=0 2024-06-04T05:19:16.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.558+0000 7f62eaffd700 1 -- 172.21.15.70:0/2323888793 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563dc886bbe0 con 0x563dc898edd0 2024-06-04T05:19:16.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.558+0000 7f62eaffd700 10 client.29807 handle_command_reply: tid=0 2024-06-04T05:19:16.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.558+0000 7f631c34eb80 1 client.29807 shutdown 2024-06-04T05:19:16.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.558+0000 7f631c34eb80 1 -- 172.21.15.70:0/2323888793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 msgr2=0x563dc8987540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:16.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.558+0000 7f631c34eb80 1 --2- 172.21.15.70:0/2323888793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 0x563dc8987540 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x563dc89148d0 tx=0x7f63000123c0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.558+0000 7f631c34eb80 1 -- 172.21.15.70:0/2323888793 shutdown_connections 2024-06-04T05:19:16.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.558+0000 7f631c34eb80 1 --2- 172.21.15.70:0/2323888793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563dc890ef60 0x563dc8919a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.558+0000 7f631c34eb80 1 --2- 172.21.15.70:0/2323888793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563dc890a860 0x563dc8987000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.558+0000 7f631c34eb80 1 --2- 172.21.15.70:0/2323888793 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x563dc898edd0 0x563dc89911f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.558+0000 7f631c34eb80 1 --2- 172.21.15.70:0/2323888793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dc890b210 0x563dc8987540 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.558+0000 7f631c34eb80 1 -- 172.21.15.70:0/2323888793 >> 172.21.15.70:0/2323888793 conn(0x563dc8838080 msgr2=0x563dc890d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:16.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.558+0000 7f62eaffd700 0 client.29807 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:19:16.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.558+0000 7f631c34eb80 1 -- 172.21.15.70:0/2323888793 shutdown_connections 2024-06-04T05:19:16.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.558+0000 7f631c34eb80 1 -- 172.21.15.70:0/2323888793 wait complete. 2024-06-04T05:19:16.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.559+0000 7f631c34eb80 20 client.29807 trim_cache size 0 max 16384 2024-06-04T05:19:16.571 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:19:16.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.559+0000 7f6311ab9700 1 -- 172.21.15.70:0/2612163024 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62f40b2a70 msgr2=0x7f62f40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:16.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.559+0000 7f6311ab9700 1 --2- 172.21.15.70:0/2612163024 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62f40b2a70 0x7f62f40b4f30 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00b2e0 tx=0x7f62fc01a040 comp rx=0 tx=0).stop 2024-06-04T05:19:16.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.559+0000 7f6311ab9700 1 -- 172.21.15.70:0/2612163024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f630c114400 msgr2=0x7f630c073500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:16.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.559+0000 7f6311ab9700 1 --2- 172.21.15.70:0/2612163024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f630c114400 0x7f630c073500 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6304009fd0 tx=0x7f630400ef90 comp rx=0 tx=0).stop 2024-06-04T05:19:16.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.559+0000 7f6311ab9700 1 -- 172.21.15.70:0/2612163024 shutdown_connections 2024-06-04T05:19:16.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.559+0000 7f6311ab9700 1 --2- 172.21.15.70:0/2612163024 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62f40b2a70 0x7f62f40b4f30 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.559+0000 7f6311ab9700 1 --2- 172.21.15.70:0/2612163024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f630c114400 0x7f630c073500 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.559+0000 7f6311ab9700 1 --2- 172.21.15.70:0/2612163024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f630c077a70 0x7f630c06ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.559+0000 7f6311ab9700 1 --2- 172.21.15.70:0/2612163024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f630c078390 0x7f630c06f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:16.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.559+0000 7f6311ab9700 1 -- 172.21.15.70:0/2612163024 >> 172.21.15.70:0/2612163024 conn(0x7f630c101250 msgr2=0x7f630c103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:16.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.560+0000 7f6311ab9700 1 -- 172.21.15.70:0/2612163024 shutdown_connections 2024-06-04T05:19:16.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:16.560+0000 7f6311ab9700 1 -- 172.21.15.70:0/2612163024 wait complete. 2024-06-04T05:19:16.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 62726 6.76 MB/sec execute 368 sec latency 107.996 ms 2024-06-04T05:19:17.396 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:19:17.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:17 smithi070 ceph-mon[32631]: pgmap v696: 97 pgs: 97 active+clean; 897 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 249 op/s 2024-06-04T05:19:17.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:17 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2068111983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:19:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:17 smithi112 ceph-mon[36453]: pgmap v696: 97 pgs: 97 active+clean; 897 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 249 op/s 2024-06-04T05:19:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:17 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2068111983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:19:17.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:17 smithi098 ceph-mon[35569]: pgmap v696: 97 pgs: 97 active+clean; 897 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 249 op/s 2024-06-04T05:19:17.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:17 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2068111983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:19:17.815 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 145.85 --await-for 45.0' 2024-06-04T05:19:17.816 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 145.85 --await-for 45.0 2024-06-04T05:19:17.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 62917 6.77 MB/sec execute 369 sec latency 62.511 ms 2024-06-04T05:19:17.866 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:19:18.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63060 6.76 MB/sec execute 370 sec latency 64.542 ms 2024-06-04T05:19:19.188 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:19:19.321 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:19 smithi070 ceph-mon[32631]: pgmap v697: 97 pgs: 97 active+clean; 897 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 210 op/s 2024-06-04T05:19:19.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:19 smithi112 ceph-mon[36453]: pgmap v697: 97 pgs: 97 active+clean; 897 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 210 op/s 2024-06-04T05:19:19.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:19 smithi098 ceph-mon[35569]: pgmap v697: 97 pgs: 97 active+clean; 897 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 210 op/s 2024-06-04T05:19:19.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63256 6.79 MB/sec execute 371 sec latency 59.325 ms 2024-06-04T05:19:20.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63448 6.80 MB/sec execute 372 sec latency 95.906 ms 2024-06-04T05:19:21.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.161+0000 7f621bbd6700 1 -- 172.21.15.70:0/2615314350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f621410aa80 msgr2=0x7f621410af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:21.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.161+0000 7f621bbd6700 1 --2- 172.21.15.70:0/2615314350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f621410aa80 0x7f621410af00 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f6200009990 tx=0x7f6200009ca0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.161+0000 7f621bbd6700 1 -- 172.21.15.70:0/2615314350 shutdown_connections 2024-06-04T05:19:21.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.161+0000 7f621bbd6700 1 --2- 172.21.15.70:0/2615314350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f621411cd20 0x7f621411f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.161+0000 7f621bbd6700 1 --2- 172.21.15.70:0/2615314350 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6214111d20 0x7f621410a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.161+0000 7f621bbd6700 1 --2- 172.21.15.70:0/2615314350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f621410aa80 0x7f621410af00 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.161+0000 7f621bbd6700 1 -- 172.21.15.70:0/2615314350 >> 172.21.15.70:0/2615314350 conn(0x7f621406e6b0 msgr2=0x7f621406eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:21.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.161+0000 7f621bbd6700 1 -- 172.21.15.70:0/2615314350 shutdown_connections 2024-06-04T05:19:21.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.161+0000 7f621bbd6700 1 -- 172.21.15.70:0/2615314350 wait complete. 2024-06-04T05:19:21.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.162+0000 7f621bbd6700 1 Processor -- start 2024-06-04T05:19:21.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.162+0000 7f621bbd6700 1 -- start start 2024-06-04T05:19:21.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.162+0000 7f621bbd6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f621410aa80 0x7f62141ae600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.162+0000 7f621bbd6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6214111d20 0x7f62141aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.162+0000 7f621bbd6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f621411cd20 0x7f62141a8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.162+0000 7f621bbd6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6214122440 con 0x7f6214111d20 2024-06-04T05:19:21.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.162+0000 7f621bbd6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62141222c0 con 0x7f621410aa80 2024-06-04T05:19:21.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.162+0000 7f621bbd6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f62141225c0 con 0x7f621411cd20 2024-06-04T05:19:21.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.163+0000 7f6219972700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f621410aa80 0x7f62141ae600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:21.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.163+0000 7f6219171700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6214111d20 0x7f62141aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:21.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.163+0000 7f6219972700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f621410aa80 0x7f62141ae600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33272/0 (socket says 172.21.15.70:33272) 2024-06-04T05:19:21.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.163+0000 7f6219972700 1 -- 172.21.15.70:0/759056907 learned_addr learned my addr 172.21.15.70:0/759056907 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:21.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.163+0000 7f621a173700 1 --2- 172.21.15.70:0/759056907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f621411cd20 0x7f62141a8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:21.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.163+0000 7f6219171700 1 -- 172.21.15.70:0/759056907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f621411cd20 msgr2=0x7f62141a8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:21.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.163+0000 7f6219171700 1 --2- 172.21.15.70:0/759056907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f621411cd20 0x7f62141a8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.163+0000 7f6219171700 1 -- 172.21.15.70:0/759056907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f621410aa80 msgr2=0x7f62141ae600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:21.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.163+0000 7f6219171700 1 --2- 172.21.15.70:0/759056907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f621410aa80 0x7f62141ae600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.163+0000 7f6219171700 1 -- 172.21.15.70:0/759056907 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62141a8f60 con 0x7f6214111d20 2024-06-04T05:19:21.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.163+0000 7f6219171700 1 --2- 172.21.15.70:0/759056907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6214111d20 0x7f62141aeb40 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f6200000c00 tx=0x7f62000047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:21.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.163+0000 7f6212ffd700 1 -- 172.21.15.70:0/759056907 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6200026070 con 0x7f6214111d20 2024-06-04T05:19:21.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.164+0000 7f6212ffd700 1 -- 172.21.15.70:0/759056907 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6200021410 con 0x7f6214111d20 2024-06-04T05:19:21.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.164+0000 7f6212ffd700 1 -- 172.21.15.70:0/759056907 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6200017650 con 0x7f6214111d20 2024-06-04T05:19:21.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.164+0000 7f621bbd6700 1 -- 172.21.15.70:0/759056907 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62141a91f0 con 0x7f6214111d20 2024-06-04T05:19:21.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.164+0000 7f621bbd6700 1 -- 172.21.15.70:0/759056907 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62141a95c0 con 0x7f6214111d20 2024-06-04T05:19:21.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.167+0000 7f6212ffd700 1 -- 172.21.15.70:0/759056907 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f6200029070 con 0x7f6214111d20 2024-06-04T05:19:21.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.168+0000 7f6212ffd700 1 --2- 172.21.15.70:0/759056907 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62040b2a70 0x7f62040b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.168+0000 7f6219972700 1 --2- 172.21.15.70:0/759056907 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62040b2a70 0x7f62040b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:21.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.168+0000 7f6212ffd700 1 -- 172.21.15.70:0/759056907 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f62000e3480 con 0x7f6214111d20 2024-06-04T05:19:21.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.168+0000 7f622646bb80 1 Processor -- start 2024-06-04T05:19:21.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.169+0000 7f622646bb80 1 -- start start 2024-06-04T05:19:21.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.169+0000 7f6219972700 1 --2- 172.21.15.70:0/759056907 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62040b2a70 0x7f62040b4f30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6208009c10 tx=0x7f620800b040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:21.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.169+0000 7f622646bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 0x55839f2e5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.169+0000 7f622646bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2df8a0 0x55839f2dfd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.169+0000 7f622646bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2e0260 0x55839f2ec080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.169+0000 7f622646bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55839f23dd70 con 0x55839f2e5320 2024-06-04T05:19:21.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.169+0000 7f622646bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55839f22d910 con 0x55839f2df8a0 2024-06-04T05:19:21.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.169+0000 7f622646bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55839f231950 con 0x55839f2e0260 2024-06-04T05:19:21.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.169+0000 7f6219972700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 0x55839f2e5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:21.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.169+0000 7f6219972700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 0x55839f2e5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35842/0 (socket says 172.21.15.70:35842) 2024-06-04T05:19:21.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.169+0000 7f6219972700 1 -- 172.21.15.70:0/3930101761 learned_addr learned my addr 172.21.15.70:0/3930101761 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:21.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.169+0000 7f6219972700 1 -- 172.21.15.70:0/3930101761 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2e0260 msgr2=0x55839f2ec080 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:19:21.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.169+0000 7f6219972700 1 --2- 172.21.15.70:0/3930101761 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2e0260 0x55839f2ec080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.169+0000 7f6219972700 1 -- 172.21.15.70:0/3930101761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2df8a0 msgr2=0x55839f2dfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:21.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.169+0000 7f6219171700 1 --2- 172.21.15.70:0/3930101761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2df8a0 0x55839f2dfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:21.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.170+0000 7f6219972700 1 --2- 172.21.15.70:0/3930101761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2df8a0 0x55839f2dfd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.170+0000 7f6219972700 1 -- 172.21.15.70:0/3930101761 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55839f1d6fb0 con 0x55839f2e5320 2024-06-04T05:19:21.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.170+0000 7f6219171700 1 --2- 172.21.15.70:0/3930101761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2df8a0 0x55839f2dfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:19:21.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.170+0000 7f6219972700 1 --2- 172.21.15.70:0/3930101761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 0x55839f2e5700 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f6208009fa0 tx=0x7f6208014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d7b7a4fd55e45f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:21.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.170+0000 7f61fffff700 1 -- 172.21.15.70:0/3930101761 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620801b070 con 0x55839f2e5320 2024-06-04T05:19:21.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.170+0000 7f61fffff700 1 -- 172.21.15.70:0/3930101761 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6208027070 con 0x55839f2e5320 2024-06-04T05:19:21.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.170+0000 7f61fffff700 1 -- 172.21.15.70:0/3930101761 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6208017940 con 0x55839f2e5320 2024-06-04T05:19:21.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.171+0000 7f622646bb80 1 -- 172.21.15.70:0/3930101761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 msgr2=0x55839f2e5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:21.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.171+0000 7f622646bb80 1 --2- 172.21.15.70:0/3930101761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 0x55839f2e5700 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f6208009fa0 tx=0x7f6208014000 comp rx=0 tx=0).stop 2024-06-04T05:19:21.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.171+0000 7f622646bb80 1 -- 172.21.15.70:0/3930101761 shutdown_connections 2024-06-04T05:19:21.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.171+0000 7f622646bb80 1 --2- 172.21.15.70:0/3930101761 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2e0260 0x55839f2ec080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.171+0000 7f622646bb80 1 --2- 172.21.15.70:0/3930101761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 0x55839f2e5700 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.171+0000 7f622646bb80 1 --2- 172.21.15.70:0/3930101761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2df8a0 0x55839f2dfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.171+0000 7f622646bb80 1 -- 172.21.15.70:0/3930101761 >> 172.21.15.70:0/3930101761 conn(0x55839f20b240 msgr2=0x55839f2da3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:21.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.171+0000 7f622646bb80 1 -- 172.21.15.70:0/3930101761 shutdown_connections 2024-06-04T05:19:21.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.171+0000 7f622646bb80 1 -- 172.21.15.70:0/3930101761 wait complete. 2024-06-04T05:19:21.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.171+0000 7f622646bb80 1 -- start start 2024-06-04T05:19:21.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.172+0000 7f622646bb80 1 Processor -- start 2024-06-04T05:19:21.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.172+0000 7f622646bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2df8a0 0x55839f2ebdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.172+0000 7f622646bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2e0260 0x55839f2e90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.172+0000 7f622646bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 0x55839f2e9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.172+0000 7f622646bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55839f133fa0 con 0x55839f2e5320 2024-06-04T05:19:21.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.172+0000 7f622646bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55839f195da0 con 0x55839f2e0260 2024-06-04T05:19:21.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.172+0000 7f622646bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55839f2ec6e0 con 0x55839f2df8a0 2024-06-04T05:19:21.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.172+0000 7f622646bb80 10 client.0 _ll_register_callbacks cb 0x55839f1c8950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:21.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.172+0000 7f6219972700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2df8a0 0x55839f2ebdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:21.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.172+0000 7f6219972700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2df8a0 0x55839f2ebdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39656/0 (socket says 172.21.15.70:39656) 2024-06-04T05:19:21.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.172+0000 7f6219972700 1 -- 172.21.15.70:0/146058811 learned_addr learned my addr 172.21.15.70:0/146058811 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:21.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.172+0000 7f6219171700 1 --2- 172.21.15.70:0/146058811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2e0260 0x55839f2e90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:21.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.172+0000 7f621a173700 1 --2- 172.21.15.70:0/146058811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 0x55839f2e9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:21.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.173+0000 7f621a173700 1 -- 172.21.15.70:0/146058811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2df8a0 msgr2=0x55839f2ebdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:21.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.173+0000 7f621a173700 1 --2- 172.21.15.70:0/146058811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2df8a0 0x55839f2ebdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.173+0000 7f621a173700 1 -- 172.21.15.70:0/146058811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2e0260 msgr2=0x55839f2e90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:21.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.173+0000 7f621a173700 1 --2- 172.21.15.70:0/146058811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2e0260 0x55839f2e90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.173+0000 7f621a173700 1 -- 172.21.15.70:0/146058811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55839f17d8d0 con 0x55839f2e5320 2024-06-04T05:19:21.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.173+0000 7f6219171700 1 --2- 172.21.15.70:0/146058811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2e0260 0x55839f2e90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:19:21.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.173+0000 7f621a173700 1 --2- 172.21.15.70:0/146058811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 0x55839f2e9600 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f620c00dcb0 tx=0x7f620c00f360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8bb3281d119d45 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:21.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.173+0000 7f61fd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:21.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.173+0000 7f61fd7fa700 1 -- 172.21.15.70:0/146058811 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620c00fdf0 con 0x55839f2e5320 2024-06-04T05:19:21.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.173+0000 7f622646bb80 1 -- 172.21.15.70:0/146058811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6208027070 con 0x55839f2e5320 2024-06-04T05:19:21.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.173+0000 7f61fd7fa700 1 -- 172.21.15.70:0/146058811 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f620c011440 con 0x55839f2e5320 2024-06-04T05:19:21.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.174+0000 7f61fd7fa700 1 -- 172.21.15.70:0/146058811 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f620c011640 con 0x55839f2e5320 2024-06-04T05:19:21.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.174+0000 7f622646bb80 10 client.21123 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:21.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.174+0000 7f622646bb80 1 -- 172.21.15.70:0/146058811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55839f1e7760 con 0x55839f2e5320 2024-06-04T05:19:21.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.174+0000 7f61fd7fa700 1 -- 172.21.15.70:0/146058811 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6208027070 con 0x55839f2e5320 2024-06-04T05:19:21.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.174+0000 7f61fd7fa700 1 -- 172.21.15.70:0/146058811 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f620c022070 con 0x55839f2e5320 2024-06-04T05:19:21.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.174+0000 7f622646bb80 10 client.21123 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:21.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.174+0000 7f622646bb80 10 client.21123 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:19:21.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.174+0000 7f622646bb80 20 client.21123 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:21.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.174+0000 7f622646bb80 1 --2- 172.21.15.70:0/146058811 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55839f2f9230 0x55839f2fb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.174+0000 7f622646bb80 4 client.21123 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:19:21.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.174+0000 7f622646bb80 1 -- 172.21.15.70:0/146058811 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55839f23fcc0 con 0x55839f2f9230 2024-06-04T05:19:21.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.174+0000 7f6219972700 1 --2- 172.21.15.70:0/146058811 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55839f2f9230 0x55839f2fb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:21.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.175+0000 7f61fd7fa700 10 client.21123 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:21.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.176+0000 7f6219972700 1 --2- 172.21.15.70:0/146058811 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55839f2f9230 0x55839f2fb620 crc :-1 s=READY pgs=465 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcd46ac6080da23d server_cookie=b7e38a27177aae16 in_seq=0 out_seq=0 2024-06-04T05:19:21.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.176+0000 7f61fd7fa700 1 -- 172.21.15.70:0/146058811 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55839f23fcc0 con 0x55839f2f9230 2024-06-04T05:19:21.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.176+0000 7f61fd7fa700 10 client.21123 handle_command_reply: tid=0 2024-06-04T05:19:21.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.176+0000 7f622646bb80 1 client.21123 shutdown 2024-06-04T05:19:21.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.177+0000 7f622646bb80 1 -- 172.21.15.70:0/146058811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 msgr2=0x55839f2e9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.177+0000 7f622646bb80 1 --2- 172.21.15.70:0/146058811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 0x55839f2e9600 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f620c00dcb0 tx=0x7f620c00f360 comp rx=0 tx=0).stop 2024-06-04T05:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.177+0000 7f622646bb80 1 -- 172.21.15.70:0/146058811 shutdown_connections 2024-06-04T05:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.177+0000 7f622646bb80 1 --2- 172.21.15.70:0/146058811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 0x55839f2e9600 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.177+0000 7f622646bb80 1 --2- 172.21.15.70:0/146058811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2df8a0 0x55839f2ebdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.177+0000 7f622646bb80 1 --2- 172.21.15.70:0/146058811 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55839f2f9230 0x55839f2fb620 crc :-1 s=READY pgs=465 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.177+0000 7f622646bb80 1 --2- 172.21.15.70:0/146058811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2e0260 0x55839f2e90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.177+0000 7f61fd7fa700 0 client.21123 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.177+0000 7f622646bb80 1 -- 172.21.15.70:0/146058811 >> 172.21.15.70:0/146058811 conn(0x55839f20b240 msgr2=0x55839f2d9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.177+0000 7f622646bb80 1 -- 172.21.15.70:0/146058811 shutdown_connections 2024-06-04T05:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.177+0000 7f622646bb80 1 -- 172.21.15.70:0/146058811 wait complete. 2024-06-04T05:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.177+0000 7f622646bb80 20 client.21123 trim_cache size 0 max 16384 2024-06-04T05:19:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.197+0000 7f622646bb80 1 Processor -- start 2024-06-04T05:19:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.197+0000 7f622646bb80 1 -- start start 2024-06-04T05:19:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.198+0000 7f622646bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2df8a0 0x55839f2f5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.198+0000 7f622646bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e0260 0x55839f2e4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.198+0000 7f622646bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2e5320 0x55839f2da270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.198+0000 7f622646bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62141bcb20 con 0x55839f2e0260 2024-06-04T05:19:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.198+0000 7f622646bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62141be6c0 con 0x55839f2df8a0 2024-06-04T05:19:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.198+0000 7f622646bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f62141bd250 con 0x55839f2e5320 2024-06-04T05:19:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.198+0000 7f6219171700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e0260 0x55839f2e4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.198+0000 7f6219171700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e0260 0x55839f2e4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35858/0 (socket says 172.21.15.70:35858) 2024-06-04T05:19:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.198+0000 7f6219972700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2df8a0 0x55839f2f5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.198+0000 7f6219171700 1 -- 172.21.15.70:0/787276002 learned_addr learned my addr 172.21.15.70:0/787276002 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.198+0000 7f6219171700 1 -- 172.21.15.70:0/787276002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2e5320 msgr2=0x55839f2da270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:19:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.198+0000 7f6219171700 1 --2- 172.21.15.70:0/787276002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2e5320 0x55839f2da270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.198+0000 7f6219171700 1 -- 172.21.15.70:0/787276002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2df8a0 msgr2=0x55839f2f5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.198+0000 7f6219171700 1 --2- 172.21.15.70:0/787276002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2df8a0 0x55839f2f5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.198+0000 7f6219171700 1 -- 172.21.15.70:0/787276002 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55839f2f6d00 con 0x55839f2e0260 2024-06-04T05:19:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.198+0000 7f6219972700 1 --2- 172.21.15.70:0/787276002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2df8a0 0x55839f2f5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:19:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.198+0000 7f6219171700 1 --2- 172.21.15.70:0/787276002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e0260 0x55839f2e4900 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f62000e11a0 tx=0x7f6200009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=835603617a9a30f2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.199+0000 7f61ff7fe700 1 -- 172.21.15.70:0/787276002 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62000b1e60 con 0x55839f2e0260 2024-06-04T05:19:21.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.199+0000 7f61ff7fe700 1 -- 172.21.15.70:0/787276002 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f62000aa7c0 con 0x55839f2e0260 2024-06-04T05:19:21.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.199+0000 7f61ff7fe700 1 -- 172.21.15.70:0/787276002 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62000b3100 con 0x55839f2e0260 2024-06-04T05:19:21.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.200+0000 7f622646bb80 1 -- 172.21.15.70:0/787276002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e0260 msgr2=0x55839f2e4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:21.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.200+0000 7f622646bb80 1 --2- 172.21.15.70:0/787276002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e0260 0x55839f2e4900 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f62000e11a0 tx=0x7f6200009670 comp rx=0 tx=0).stop 2024-06-04T05:19:21.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.200+0000 7f622646bb80 1 -- 172.21.15.70:0/787276002 shutdown_connections 2024-06-04T05:19:21.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.200+0000 7f622646bb80 1 --2- 172.21.15.70:0/787276002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2e5320 0x55839f2da270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.200+0000 7f622646bb80 1 --2- 172.21.15.70:0/787276002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2df8a0 0x55839f2f5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.200+0000 7f622646bb80 1 --2- 172.21.15.70:0/787276002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e0260 0x55839f2e4900 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.200+0000 7f622646bb80 1 -- 172.21.15.70:0/787276002 >> 172.21.15.70:0/787276002 conn(0x55839f20b240 msgr2=0x55839f2e41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:21.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.200+0000 7f622646bb80 1 -- 172.21.15.70:0/787276002 shutdown_connections 2024-06-04T05:19:21.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.200+0000 7f622646bb80 1 -- 172.21.15.70:0/787276002 wait complete. 2024-06-04T05:19:21.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.200+0000 7f622646bb80 1 -- start start 2024-06-04T05:19:21.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.200+0000 7f622646bb80 1 Processor -- start 2024-06-04T05:19:21.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.201+0000 7f622646bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2df8a0 0x55839f2f2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.201+0000 7f622646bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2e0260 0x55839f300700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.201+0000 7f622646bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 0x55839f300c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.201+0000 7f622646bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62141bdff0 con 0x55839f2e5320 2024-06-04T05:19:21.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.201+0000 7f622646bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62141bd920 con 0x55839f2e0260 2024-06-04T05:19:21.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.201+0000 7f622646bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f62141baae0 con 0x55839f2df8a0 2024-06-04T05:19:21.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.201+0000 7f6219171700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2e0260 0x55839f300700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:21.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.201+0000 7f6219972700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2df8a0 0x55839f2f2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:21.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.201+0000 7f622646bb80 10 client.0 _ll_register_callbacks cb 0x55839f1c8950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:21.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.201+0000 7f6219972700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2df8a0 0x55839f2f2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39674/0 (socket says 172.21.15.70:39674) 2024-06-04T05:19:21.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.201+0000 7f6219972700 1 -- 172.21.15.70:0/745562782 learned_addr learned my addr 172.21.15.70:0/745562782 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:21.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.201+0000 7f621a173700 1 --2- 172.21.15.70:0/745562782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 0x55839f300c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:21.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.201+0000 7f6219972700 1 -- 172.21.15.70:0/745562782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2e0260 msgr2=0x55839f300700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:21.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.201+0000 7f6219972700 1 --2- 172.21.15.70:0/745562782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2e0260 0x55839f300700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.202+0000 7f6219972700 1 -- 172.21.15.70:0/745562782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 msgr2=0x55839f300c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:21.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.202+0000 7f6219972700 1 --2- 172.21.15.70:0/745562782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 0x55839f300c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.202+0000 7f6219972700 1 -- 172.21.15.70:0/745562782 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55839f301480 con 0x55839f2df8a0 2024-06-04T05:19:21.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.202+0000 7f621a173700 1 --2- 172.21.15.70:0/745562782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 0x55839f300c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:19:21.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.202+0000 7f6219972700 1 --2- 172.21.15.70:0/745562782 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2df8a0 0x55839f2f2000 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6208010010 tx=0x7f620800f2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1e67ca95e6d892b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:21.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.202+0000 7f61fd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:19:21.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.202+0000 7f61fd7fa700 1 -- 172.21.15.70:0/745562782 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6208039070 con 0x55839f2df8a0 2024-06-04T05:19:21.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.202+0000 7f61fd7fa700 1 -- 172.21.15.70:0/745562782 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f6208022880 con 0x55839f2df8a0 2024-06-04T05:19:21.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.202+0000 7f622646bb80 1 -- 172.21.15.70:0/745562782 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62000aa7c0 con 0x55839f2df8a0 2024-06-04T05:19:21.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.202+0000 7f61fd7fa700 1 -- 172.21.15.70:0/745562782 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f6208041050 con 0x55839f2df8a0 2024-06-04T05:19:21.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.202+0000 7f622646bb80 10 client.29953 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:21.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.202+0000 7f622646bb80 1 -- 172.21.15.70:0/745562782 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55839f363d40 con 0x55839f2df8a0 2024-06-04T05:19:21.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.203+0000 7f61fd7fa700 1 -- 172.21.15.70:0/745562782 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6208027040 con 0x55839f2df8a0 2024-06-04T05:19:21.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.203+0000 7f61fd7fa700 1 -- 172.21.15.70:0/745562782 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f620803c030 con 0x55839f2df8a0 2024-06-04T05:19:21.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.203+0000 7f622646bb80 10 client.29953 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:21.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.203+0000 7f622646bb80 10 client.29953 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:19:21.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.203+0000 7f622646bb80 20 client.29953 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:21.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.203+0000 7f622646bb80 1 --2- 172.21.15.70:0/745562782 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55839f364120 0x55839f366540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:21.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.203+0000 7f622646bb80 4 client.29953 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:19:21.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.203+0000 7f622646bb80 1 -- 172.21.15.70:0/745562782 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f62000aa7c0 con 0x55839f364120 2024-06-04T05:19:21.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.203+0000 7f6219171700 1 --2- 172.21.15.70:0/745562782 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55839f364120 0x55839f366540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:21.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.204+0000 7f61fd7fa700 10 client.29953 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:21.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.204+0000 7f6219171700 1 --2- 172.21.15.70:0/745562782 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55839f364120 0x55839f366540 crc :-1 s=READY pgs=466 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ea0b579f7533a84 server_cookie=f06489adef879f04 in_seq=0 out_seq=0 2024-06-04T05:19:21.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.205+0000 7f61fd7fa700 1 -- 172.21.15.70:0/745562782 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f62000aa7c0 con 0x55839f364120 2024-06-04T05:19:21.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.205+0000 7f61fd7fa700 10 client.29953 handle_command_reply: tid=0 2024-06-04T05:19:21.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.205+0000 7f622646bb80 1 client.29953 shutdown 2024-06-04T05:19:21.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.205+0000 7f622646bb80 1 -- 172.21.15.70:0/745562782 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2df8a0 msgr2=0x55839f2f2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:21.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.205+0000 7f622646bb80 1 --2- 172.21.15.70:0/745562782 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2df8a0 0x55839f2f2000 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6208010010 tx=0x7f620800f2d0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.205+0000 7f622646bb80 1 -- 172.21.15.70:0/745562782 shutdown_connections 2024-06-04T05:19:21.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.205+0000 7f622646bb80 1 --2- 172.21.15.70:0/745562782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55839f2e5320 0x55839f300c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.205+0000 7f622646bb80 1 --2- 172.21.15.70:0/745562782 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55839f2df8a0 0x55839f2f2000 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.205+0000 7f622646bb80 1 --2- 172.21.15.70:0/745562782 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55839f364120 0x55839f366540 crc :-1 s=READY pgs=466 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.205+0000 7f622646bb80 1 --2- 172.21.15.70:0/745562782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55839f2e0260 0x55839f300700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.205+0000 7f622646bb80 1 -- 172.21.15.70:0/745562782 >> 172.21.15.70:0/745562782 conn(0x55839f20b240 msgr2=0x55839f2de890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:21.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.205+0000 7f61fd7fa700 0 client.29953 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:21.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.205+0000 7f622646bb80 1 -- 172.21.15.70:0/745562782 shutdown_connections 2024-06-04T05:19:21.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.205+0000 7f622646bb80 1 -- 172.21.15.70:0/745562782 wait complete. 2024-06-04T05:19:21.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.205+0000 7f622646bb80 20 client.29953 trim_cache size 0 max 16384 2024-06-04T05:19:21.212 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:19:21.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.206+0000 7f621bbd6700 1 -- 172.21.15.70:0/759056907 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62040b2a70 msgr2=0x7f62040b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:21.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.206+0000 7f621bbd6700 1 --2- 172.21.15.70:0/759056907 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62040b2a70 0x7f62040b4f30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6208009c10 tx=0x7f620800b040 comp rx=0 tx=0).stop 2024-06-04T05:19:21.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.206+0000 7f621bbd6700 1 -- 172.21.15.70:0/759056907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6214111d20 msgr2=0x7f62141aeb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:21.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.206+0000 7f621bbd6700 1 --2- 172.21.15.70:0/759056907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6214111d20 0x7f62141aeb40 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f6200000c00 tx=0x7f62000047b0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.206+0000 7f621bbd6700 1 -- 172.21.15.70:0/759056907 shutdown_connections 2024-06-04T05:19:21.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.206+0000 7f621bbd6700 1 --2- 172.21.15.70:0/759056907 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62040b2a70 0x7f62040b4f30 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.206+0000 7f621bbd6700 1 --2- 172.21.15.70:0/759056907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f621411cd20 0x7f62141a8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.206+0000 7f621bbd6700 1 --2- 172.21.15.70:0/759056907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f621410aa80 0x7f62141ae600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.206+0000 7f621bbd6700 1 --2- 172.21.15.70:0/759056907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6214111d20 0x7f62141aeb40 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:21.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.206+0000 7f621bbd6700 1 -- 172.21.15.70:0/759056907 >> 172.21.15.70:0/759056907 conn(0x7f621406e6b0 msgr2=0x7f621411ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:21.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.206+0000 7f621bbd6700 1 -- 172.21.15.70:0/759056907 shutdown_connections 2024-06-04T05:19:21.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:21.206+0000 7f621bbd6700 1 -- 172.21.15.70:0/759056907 wait complete. 2024-06-04T05:19:21.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:21 smithi070 ceph-mon[32631]: pgmap v698: 97 pgs: 97 active+clean; 897 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 199 op/s 2024-06-04T05:19:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:21 smithi112 ceph-mon[36453]: pgmap v698: 97 pgs: 97 active+clean; 897 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 199 op/s 2024-06-04T05:19:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:21 smithi098 ceph-mon[35569]: pgmap v698: 97 pgs: 97 active+clean; 897 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 199 op/s 2024-06-04T05:19:21.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63620 6.80 MB/sec execute 373 sec latency 92.128 ms 2024-06-04T05:19:22.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.381+0000 7fe54d4d0700 1 -- 172.21.15.70:0/291076249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe548108530 msgr2=0x7fe548108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:22.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.381+0000 7fe54d4d0700 1 --2- 172.21.15.70:0/291076249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe548108530 0x7fe548108910 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fe5380099c0 tx=0x7fe538009cd0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.381+0000 7fe54d4d0700 1 -- 172.21.15.70:0/291076249 shutdown_connections 2024-06-04T05:19:22.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.381+0000 7fe54d4d0700 1 --2- 172.21.15.70:0/291076249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe548103f50 0x7fe548110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.381+0000 7fe54d4d0700 1 --2- 172.21.15.70:0/291076249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe548108530 0x7fe548108910 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.381+0000 7fe54d4d0700 1 --2- 172.21.15.70:0/291076249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe548103590 0x7fe548103a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.381+0000 7fe54d4d0700 1 -- 172.21.15.70:0/291076249 >> 172.21.15.70:0/291076249 conn(0x7fe54807a6f0 msgr2=0x7fe54807ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:22.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.381+0000 7fe54d4d0700 1 -- 172.21.15.70:0/291076249 shutdown_connections 2024-06-04T05:19:22.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.381+0000 7fe54d4d0700 1 -- 172.21.15.70:0/291076249 wait complete. 2024-06-04T05:19:22.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.382+0000 7fe54d4d0700 1 Processor -- start 2024-06-04T05:19:22.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.382+0000 7fe54d4d0700 1 -- start start 2024-06-04T05:19:22.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.383+0000 7fe54d4d0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe548103590 0x7fe5481a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.383+0000 7fe54d4d0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe548103f50 0x7fe5481a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.383+0000 7fe54d4d0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe548108530 0x7fe54819bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.383+0000 7fe54d4d0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe548113620 con 0x7fe548108530 2024-06-04T05:19:22.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.383+0000 7fe54d4d0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5481134a0 con 0x7fe548103f50 2024-06-04T05:19:22.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.383+0000 7fe54d4d0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe5481137a0 con 0x7fe548103590 2024-06-04T05:19:22.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.383+0000 7fe546ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe548103590 0x7fe5481a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.383+0000 7fe5467fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe548103f50 0x7fe5481a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.383+0000 7fe546ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe548103590 0x7fe5481a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39698/0 (socket says 172.21.15.70:39698) 2024-06-04T05:19:22.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.383+0000 7fe546ffd700 1 -- 172.21.15.70:0/184893046 learned_addr learned my addr 172.21.15.70:0/184893046 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:22.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.383+0000 7fe5477fe700 1 --2- 172.21.15.70:0/184893046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe548108530 0x7fe54819bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.383+0000 7fe546ffd700 1 -- 172.21.15.70:0/184893046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe548103f50 msgr2=0x7fe5481a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:22.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.383+0000 7fe546ffd700 1 --2- 172.21.15.70:0/184893046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe548103f50 0x7fe5481a1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.383+0000 7fe546ffd700 1 -- 172.21.15.70:0/184893046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe548108530 msgr2=0x7fe54819bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:22.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.383+0000 7fe546ffd700 1 --2- 172.21.15.70:0/184893046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe548108530 0x7fe54819bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.383+0000 7fe546ffd700 1 -- 172.21.15.70:0/184893046 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54819c220 con 0x7fe548103590 2024-06-04T05:19:22.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.383+0000 7fe546ffd700 1 --2- 172.21.15.70:0/184893046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe548103590 0x7fe5481a1900 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe538005f50 tx=0x7fe538004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:22.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.384+0000 7fe52ffff700 1 -- 172.21.15.70:0/184893046 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe538026070 con 0x7fe548103590 2024-06-04T05:19:22.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.384+0000 7fe52ffff700 1 -- 172.21.15.70:0/184893046 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe538003c50 con 0x7fe548103590 2024-06-04T05:19:22.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.384+0000 7fe54d4d0700 1 -- 172.21.15.70:0/184893046 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe54819c4b0 con 0x7fe548103590 2024-06-04T05:19:22.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.384+0000 7fe52ffff700 1 -- 172.21.15.70:0/184893046 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53800f950 con 0x7fe548103590 2024-06-04T05:19:22.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.384+0000 7fe54d4d0700 1 -- 172.21.15.70:0/184893046 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5481a87d0 con 0x7fe548103590 2024-06-04T05:19:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.387+0000 7fe52ffff700 1 -- 172.21.15.70:0/184893046 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fe538029070 con 0x7fe548103590 2024-06-04T05:19:22.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.388+0000 7fe52ffff700 1 --2- 172.21.15.70:0/184893046 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5300b2aa0 0x7fe5300b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.388+0000 7fe5467fc700 1 --2- 172.21.15.70:0/184893046 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5300b2aa0 0x7fe5300b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.388+0000 7fe52ffff700 1 -- 172.21.15.70:0/184893046 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7fe5380e2e40 con 0x7fe548103590 2024-06-04T05:19:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.389+0000 7fe5467fc700 1 --2- 172.21.15.70:0/184893046 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5300b2aa0 0x7fe5300b4f60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe5481033d0 tx=0x7fe53c009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.389+0000 7fe557d65b80 1 Processor -- start 2024-06-04T05:19:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.389+0000 7fe557d65b80 1 -- start start 2024-06-04T05:19:22.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.390+0000 7fe557d65b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887c5350 0x5651887c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.390+0000 7fe557d65b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887bf980 0x5651887bfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.390+0000 7fe557d65b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c0340 0x5651887cc180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.390+0000 7fe557d65b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56518871ee30 con 0x5651887c0340 2024-06-04T05:19:22.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.390+0000 7fe557d65b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56518870e9d0 con 0x5651887bf980 2024-06-04T05:19:22.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.390+0000 7fe546ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887c5350 0x5651887c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.390+0000 7fe5477fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c0340 0x5651887cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.390+0000 7fe546ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887c5350 0x5651887c5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39706/0 (socket says 172.21.15.70:39706) 2024-06-04T05:19:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.390+0000 7fe5467fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887bf980 0x5651887bfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.390+0000 7fe546ffd700 1 -- 172.21.15.70:0/4029178624 learned_addr learned my addr 172.21.15.70:0/4029178624 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.390+0000 7fe557d65b80 1 -- 172.21.15.70:0/4029178624 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x565188712a10 con 0x5651887c5350 2024-06-04T05:19:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.390+0000 7fe5467fc700 1 -- 172.21.15.70:0/4029178624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887c5350 msgr2=0x5651887c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.390+0000 7fe5467fc700 1 --2- 172.21.15.70:0/4029178624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887c5350 0x5651887c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.390+0000 7fe5467fc700 1 -- 172.21.15.70:0/4029178624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c0340 msgr2=0x5651887cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.390+0000 7fe5467fc700 1 --2- 172.21.15.70:0/4029178624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c0340 0x5651887cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.391+0000 7fe5467fc700 1 -- 172.21.15.70:0/4029178624 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56518865ea70 con 0x5651887bf980 2024-06-04T05:19:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.391+0000 7fe5477fe700 1 --2- 172.21.15.70:0/4029178624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c0340 0x5651887cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:19:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.391+0000 7fe5467fc700 1 --2- 172.21.15.70:0/4029178624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887bf980 0x5651887bfe00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe53c011c90 tx=0x7fe53c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8236ad7d9ed3b1a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:22.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.391+0000 7fe52d7fa700 1 -- 172.21.15.70:0/4029178624 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53c004230 con 0x5651887bf980 2024-06-04T05:19:22.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.391+0000 7fe52d7fa700 1 -- 172.21.15.70:0/4029178624 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe53c0043d0 con 0x5651887bf980 2024-06-04T05:19:22.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.391+0000 7fe52d7fa700 1 -- 172.21.15.70:0/4029178624 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53c003d00 con 0x5651887bf980 2024-06-04T05:19:22.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.392+0000 7fe557d65b80 1 -- 172.21.15.70:0/4029178624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887bf980 msgr2=0x5651887bfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:22.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.392+0000 7fe557d65b80 1 --2- 172.21.15.70:0/4029178624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887bf980 0x5651887bfe00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe53c011c90 tx=0x7fe53c014000 comp rx=0 tx=0).stop 2024-06-04T05:19:22.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.392+0000 7fe557d65b80 1 -- 172.21.15.70:0/4029178624 shutdown_connections 2024-06-04T05:19:22.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.392+0000 7fe557d65b80 1 --2- 172.21.15.70:0/4029178624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c0340 0x5651887cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.392+0000 7fe557d65b80 1 --2- 172.21.15.70:0/4029178624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887c5350 0x5651887c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.392+0000 7fe557d65b80 1 --2- 172.21.15.70:0/4029178624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887bf980 0x5651887bfe00 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.392+0000 7fe557d65b80 1 -- 172.21.15.70:0/4029178624 >> 172.21.15.70:0/4029178624 conn(0x5651886ed220 msgr2=0x5651887ba290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:22.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.392+0000 7fe557d65b80 1 -- 172.21.15.70:0/4029178624 shutdown_connections 2024-06-04T05:19:22.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.392+0000 7fe557d65b80 1 -- 172.21.15.70:0/4029178624 wait complete. 2024-06-04T05:19:22.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.393+0000 7fe557d65b80 1 -- start start 2024-06-04T05:19:22.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.393+0000 7fe557d65b80 1 Processor -- start 2024-06-04T05:19:22.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.394+0000 7fe557d65b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887bf980 0x5651887cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.394+0000 7fe557d65b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887c0340 0x5651887c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.394+0000 7fe557d65b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c5350 0x5651887c9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.394+0000 7fe546ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887bf980 0x5651887cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.394+0000 7fe546ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887bf980 0x5651887cbec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39722/0 (socket says 172.21.15.70:39722) 2024-06-04T05:19:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.394+0000 7fe5467fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887c0340 0x5651887c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.394+0000 7fe5467fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887c0340 0x5651887c91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33346/0 (socket says 172.21.15.70:33346) 2024-06-04T05:19:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.394+0000 7fe557d65b80 1 -- 172.21.15.70:0/219139268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x565188615140 con 0x5651887c5350 2024-06-04T05:19:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.394+0000 7fe5477fe700 1 --2- 172.21.15.70:0/219139268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c5350 0x5651887c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.394+0000 7fe546ffd700 1 -- 172.21.15.70:0/219139268 learned_addr learned my addr 172.21.15.70:0/219139268 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.394+0000 7fe557d65b80 1 -- 172.21.15.70:0/219139268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5651886b0960 con 0x5651887c0340 2024-06-04T05:19:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.394+0000 7fe557d65b80 1 -- 172.21.15.70:0/219139268 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5651887cc720 con 0x5651887bf980 2024-06-04T05:19:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.394+0000 7fe557d65b80 10 client.0 _ll_register_callbacks cb 0x5651886a9ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.395+0000 7fe546ffd700 1 -- 172.21.15.70:0/219139268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887c0340 msgr2=0x5651887c91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.395+0000 7fe546ffd700 1 --2- 172.21.15.70:0/219139268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887c0340 0x5651887c91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.395+0000 7fe546ffd700 1 -- 172.21.15.70:0/219139268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c5350 msgr2=0x5651887c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.395+0000 7fe546ffd700 1 --2- 172.21.15.70:0/219139268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c5350 0x5651887c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.395+0000 7fe546ffd700 1 -- 172.21.15.70:0/219139268 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651886ad830 con 0x5651887bf980 2024-06-04T05:19:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.395+0000 7fe546ffd700 1 --2- 172.21.15.70:0/219139268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887bf980 0x5651887cbec0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe538009fd0 tx=0x7fe5380a9a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2325d61085b0923 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:22.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.395+0000 7fe52aff5700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:19:22.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.395+0000 7fe52aff5700 1 -- 172.21.15.70:0/219139268 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5380c4060 con 0x5651887bf980 2024-06-04T05:19:22.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.395+0000 7fe557d65b80 1 -- 172.21.15.70:0/219139268 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe53c0043d0 con 0x5651887bf980 2024-06-04T05:19:22.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.395+0000 7fe52aff5700 1 -- 172.21.15.70:0/219139268 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7fe5380b2cd0 con 0x5651887bf980 2024-06-04T05:19:22.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.395+0000 7fe52aff5700 1 -- 172.21.15.70:0/219139268 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fe5380a9c90 con 0x5651887bf980 2024-06-04T05:19:22.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.396+0000 7fe557d65b80 10 client.29965 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:22.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.396+0000 7fe557d65b80 1 -- 172.21.15.70:0/219139268 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x5651887d6eb0 con 0x5651887bf980 2024-06-04T05:19:22.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.396+0000 7fe52aff5700 1 -- 172.21.15.70:0/219139268 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe53c0043d0 con 0x5651887bf980 2024-06-04T05:19:22.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.396+0000 7fe52aff5700 1 -- 172.21.15.70:0/219139268 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7fe5380a93e0 con 0x5651887bf980 2024-06-04T05:19:22.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.396+0000 7fe557d65b80 10 client.29965 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:22.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.396+0000 7fe557d65b80 10 client.29965 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:19:22.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.396+0000 7fe557d65b80 20 client.29965 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:22.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.396+0000 7fe557d65b80 1 --2- 172.21.15.70:0/219139268 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5651887d9650 0x5651887dba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.396+0000 7fe557d65b80 4 client.29965 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:19:22.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.396+0000 7fe557d65b80 1 -- 172.21.15.70:0/219139268 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56518878bb80 con 0x5651887d9650 2024-06-04T05:19:22.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.397+0000 7fe5467fc700 1 --2- 172.21.15.70:0/219139268 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5651887d9650 0x5651887dba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.397+0000 7fe52aff5700 10 client.29965 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:22.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.397+0000 7fe5467fc700 1 --2- 172.21.15.70:0/219139268 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5651887d9650 0x5651887dba70 crc :-1 s=READY pgs=467 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b632253053895999 server_cookie=6b22eca9d736e8e in_seq=0 out_seq=0 2024-06-04T05:19:22.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.398+0000 7fe52aff5700 1 -- 172.21.15.70:0/219139268 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56518878bb80 con 0x5651887d9650 2024-06-04T05:19:22.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.398+0000 7fe52aff5700 10 client.29965 handle_command_reply: tid=0 2024-06-04T05:19:22.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.398+0000 7fe557d65b80 1 client.29965 shutdown 2024-06-04T05:19:22.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.398+0000 7fe557d65b80 1 -- 172.21.15.70:0/219139268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887bf980 msgr2=0x5651887cbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:22.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.398+0000 7fe557d65b80 1 --2- 172.21.15.70:0/219139268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887bf980 0x5651887cbec0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe538009fd0 tx=0x7fe5380a9a20 comp rx=0 tx=0).stop 2024-06-04T05:19:22.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.398+0000 7fe557d65b80 1 -- 172.21.15.70:0/219139268 shutdown_connections 2024-06-04T05:19:22.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.398+0000 7fe557d65b80 1 --2- 172.21.15.70:0/219139268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c5350 0x5651887c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.398+0000 7fe557d65b80 1 --2- 172.21.15.70:0/219139268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887bf980 0x5651887cbec0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.398+0000 7fe557d65b80 1 --2- 172.21.15.70:0/219139268 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5651887d9650 0x5651887dba70 crc :-1 s=READY pgs=467 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.398+0000 7fe557d65b80 1 --2- 172.21.15.70:0/219139268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887c0340 0x5651887c91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.398+0000 7fe52aff5700 0 client.29965 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:22.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.398+0000 7fe557d65b80 1 -- 172.21.15.70:0/219139268 >> 172.21.15.70:0/219139268 conn(0x5651886ed220 msgr2=0x5651887b4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:22.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.399+0000 7fe557d65b80 1 -- 172.21.15.70:0/219139268 shutdown_connections 2024-06-04T05:19:22.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.399+0000 7fe557d65b80 1 -- 172.21.15.70:0/219139268 wait complete. 2024-06-04T05:19:22.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.399+0000 7fe557d65b80 20 client.29965 trim_cache size 0 max 16384 2024-06-04T05:19:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.427+0000 7fe557d65b80 1 Processor -- start 2024-06-04T05:19:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.427+0000 7fe557d65b80 1 -- start start 2024-06-04T05:19:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe557d65b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887bf980 0x5651887d57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe557d65b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c0340 0x5651887c81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe557d65b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887c5350 0x5651887cba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe557d65b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe5481afef0 con 0x5651887c0340 2024-06-04T05:19:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe557d65b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5481b1a90 con 0x5651887c5350 2024-06-04T05:19:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe557d65b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe5481b0620 con 0x5651887bf980 2024-06-04T05:19:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe546ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887bf980 0x5651887d57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe5467fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c0340 0x5651887c81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe546ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887bf980 0x5651887d57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:39732/0 (socket says 172.21.15.70:39732) 2024-06-04T05:19:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe546ffd700 1 -- 172.21.15.70:0/3059513782 learned_addr learned my addr 172.21.15.70:0/3059513782 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe5477fe700 1 --2- 172.21.15.70:0/3059513782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887c5350 0x5651887cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe5467fc700 1 -- 172.21.15.70:0/3059513782 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887bf980 msgr2=0x5651887d57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:22.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe5467fc700 1 --2- 172.21.15.70:0/3059513782 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887bf980 0x5651887d57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe5467fc700 1 -- 172.21.15.70:0/3059513782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887c5350 msgr2=0x5651887cba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:22.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe5467fc700 1 --2- 172.21.15.70:0/3059513782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887c5350 0x5651887cba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe5467fc700 1 -- 172.21.15.70:0/3059513782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651887c7750 con 0x5651887c0340 2024-06-04T05:19:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe546ffd700 1 --2- 172.21.15.70:0/3059513782 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887bf980 0x5651887d57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:19:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe5467fc700 1 --2- 172.21.15.70:0/3059513782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c0340 0x5651887c81a0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x5651887ca790 tx=0x7fe53c005430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ffb7fe8d1288cfe server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.428+0000 7fe52cff9700 1 -- 172.21.15.70:0/3059513782 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53c005620 con 0x5651887c0340 2024-06-04T05:19:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.429+0000 7fe52cff9700 1 -- 172.21.15.70:0/3059513782 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe53c00ecd0 con 0x5651887c0340 2024-06-04T05:19:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.429+0000 7fe52cff9700 1 -- 172.21.15.70:0/3059513782 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53c003b60 con 0x5651887c0340 2024-06-04T05:19:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.429+0000 7fe557d65b80 1 -- 172.21.15.70:0/3059513782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c0340 msgr2=0x5651887c81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.429+0000 7fe557d65b80 1 --2- 172.21.15.70:0/3059513782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c0340 0x5651887c81a0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x5651887ca790 tx=0x7fe53c005430 comp rx=0 tx=0).stop 2024-06-04T05:19:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.429+0000 7fe557d65b80 1 -- 172.21.15.70:0/3059513782 shutdown_connections 2024-06-04T05:19:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.429+0000 7fe557d65b80 1 --2- 172.21.15.70:0/3059513782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887c5350 0x5651887cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.429+0000 7fe557d65b80 1 --2- 172.21.15.70:0/3059513782 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887bf980 0x5651887d57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.429+0000 7fe557d65b80 1 --2- 172.21.15.70:0/3059513782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c0340 0x5651887c81a0 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.429+0000 7fe557d65b80 1 -- 172.21.15.70:0/3059513782 >> 172.21.15.70:0/3059513782 conn(0x5651886ed220 msgr2=0x5651887b4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:22.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.430+0000 7fe557d65b80 1 -- 172.21.15.70:0/3059513782 shutdown_connections 2024-06-04T05:19:22.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.430+0000 7fe557d65b80 1 -- 172.21.15.70:0/3059513782 wait complete. 2024-06-04T05:19:22.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.430+0000 7fe557d65b80 1 -- start start 2024-06-04T05:19:22.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.430+0000 7fe557d65b80 1 Processor -- start 2024-06-04T05:19:22.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe557d65b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887bf980 0x5651887d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe557d65b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887c0340 0x5651887d21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe557d65b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c5350 0x5651887ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe546ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887bf980 0x5651887d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe557d65b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe5481b13c0 con 0x5651887c5350 2024-06-04T05:19:22.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe557d65b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5481b0cf0 con 0x5651887bf980 2024-06-04T05:19:22.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe546ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887bf980 0x5651887d1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33370/0 (socket says 172.21.15.70:33370) 2024-06-04T05:19:22.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe546ffd700 1 -- 172.21.15.70:0/1544078104 learned_addr learned my addr 172.21.15.70:0/1544078104 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe5477fe700 1 --2- 172.21.15.70:0/1544078104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c5350 0x5651887ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe557d65b80 1 -- 172.21.15.70:0/1544078104 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe5481adeb0 con 0x5651887c0340 2024-06-04T05:19:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe5467fc700 1 --2- 172.21.15.70:0/1544078104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887c0340 0x5651887d21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe557d65b80 10 client.0 _ll_register_callbacks cb 0x5651887c4f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe546ffd700 1 -- 172.21.15.70:0/1544078104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887c0340 msgr2=0x5651887d21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe546ffd700 1 --2- 172.21.15.70:0/1544078104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887c0340 0x5651887d21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe546ffd700 1 -- 172.21.15.70:0/1544078104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c5350 msgr2=0x5651887ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe546ffd700 1 --2- 172.21.15.70:0/1544078104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c5350 0x5651887ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe546ffd700 1 -- 172.21.15.70:0/1544078104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651887cd380 con 0x5651887bf980 2024-06-04T05:19:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.431+0000 7fe5477fe700 1 --2- 172.21.15.70:0/1544078104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c5350 0x5651887ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:19:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.432+0000 7fe546ffd700 1 --2- 172.21.15.70:0/1544078104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887bf980 0x5651887d1ca0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe5380a9e80 tx=0x7fe5380ae170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcef5ba5991a4592 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.432+0000 7fe52aff5700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:19:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.432+0000 7fe52aff5700 1 -- 172.21.15.70:0/1544078104 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5380aa880 con 0x5651887bf980 2024-06-04T05:19:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.432+0000 7fe557d65b80 1 -- 172.21.15.70:0/1544078104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe53c00ecd0 con 0x5651887bf980 2024-06-04T05:19:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.432+0000 7fe52aff5700 1 -- 172.21.15.70:0/1544078104 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7fe5380039d0 con 0x5651887bf980 2024-06-04T05:19:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.432+0000 7fe52aff5700 1 -- 172.21.15.70:0/1544078104 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fe5380aa180 con 0x5651887bf980 2024-06-04T05:19:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.432+0000 7fe557d65b80 10 client.29840 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.432+0000 7fe557d65b80 1 -- 172.21.15.70:0/1544078104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x565188844e40 con 0x5651887bf980 2024-06-04T05:19:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.432+0000 7fe52aff5700 1 -- 172.21.15.70:0/1544078104 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe53c00ecd0 con 0x5651887bf980 2024-06-04T05:19:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.432+0000 7fe52aff5700 1 -- 172.21.15.70:0/1544078104 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7fe538069ba0 con 0x5651887bf980 2024-06-04T05:19:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.433+0000 7fe557d65b80 10 client.29840 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.433+0000 7fe557d65b80 10 client.29840 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:19:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.433+0000 7fe557d65b80 20 client.29840 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.433+0000 7fe557d65b80 1 --2- 172.21.15.70:0/1544078104 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x565188845430 0x565188847850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.433+0000 7fe557d65b80 4 client.29840 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.85, "await_for": 45.0}] 2024-06-04T05:19:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.433+0000 7fe557d65b80 1 -- 172.21.15.70:0/1544078104 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.85, "await_for": 45.0}) -- 0x565188720d80 con 0x565188845430 2024-06-04T05:19:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.433+0000 7fe5467fc700 1 --2- 172.21.15.70:0/1544078104 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x565188845430 0x565188847850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.433+0000 7fe52aff5700 10 client.29840 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:22.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.433+0000 7fe5467fc700 1 --2- 172.21.15.70:0/1544078104 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x565188845430 0x565188847850 crc :-1 s=READY pgs=468 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88b464d3d3c03ac server_cookie=2a5b22bfe120fa3f in_seq=0 out_seq=0 2024-06-04T05:19:22.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63716 6.79 MB/sec execute 374 sec latency 390.745 ms 2024-06-04T05:19:22.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.884+0000 7fe52aff5700 1 -- 172.21.15.70:0/1544078104 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x565188720d80 con 0x565188845430 2024-06-04T05:19:22.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.884+0000 7fe52aff5700 10 client.29840 handle_command_reply: tid=0 2024-06-04T05:19:22.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.884+0000 7fe557d65b80 1 client.29840 shutdown 2024-06-04T05:19:22.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.884+0000 7fe557d65b80 1 -- 172.21.15.70:0/1544078104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887bf980 msgr2=0x5651887d1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:22.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.884+0000 7fe557d65b80 1 --2- 172.21.15.70:0/1544078104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887bf980 0x5651887d1ca0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe5380a9e80 tx=0x7fe5380ae170 comp rx=0 tx=0).stop 2024-06-04T05:19:22.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.884+0000 7fe557d65b80 1 -- 172.21.15.70:0/1544078104 shutdown_connections 2024-06-04T05:19:22.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.884+0000 7fe557d65b80 1 --2- 172.21.15.70:0/1544078104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651887c5350 0x5651887ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.884+0000 7fe557d65b80 1 --2- 172.21.15.70:0/1544078104 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x565188845430 0x565188847850 crc :-1 s=READY pgs=468 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.884+0000 7fe557d65b80 1 --2- 172.21.15.70:0/1544078104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5651887bf980 0x5651887d1ca0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.884+0000 7fe557d65b80 1 --2- 172.21.15.70:0/1544078104 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5651887c0340 0x5651887d21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.884+0000 7fe52aff5700 0 client.29840 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:22.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.884+0000 7fe557d65b80 1 -- 172.21.15.70:0/1544078104 >> 172.21.15.70:0/1544078104 conn(0x5651886ed220 msgr2=0x5651887c2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:22.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.884+0000 7fe557d65b80 1 -- 172.21.15.70:0/1544078104 shutdown_connections 2024-06-04T05:19:22.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.884+0000 7fe557d65b80 1 -- 172.21.15.70:0/1544078104 wait complete. 2024-06-04T05:19:22.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.884+0000 7fe557d65b80 20 client.29840 trim_cache size 0 max 16384 2024-06-04T05:19:22.888 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:19:22.888 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 59, 2024-06-04T05:19:22.888 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:19:22.888 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 70, 2024-06-04T05:19:22.888 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:19:22.889 INFO:teuthology.orchestra.run.smithi070.stdout: "da6cd7e7": { 2024-06-04T05:19:22.889 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 70, 2024-06-04T05:19:22.889 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:19:22.889 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:19:22.889 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:19:22.889 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:19:22.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:19:22.889 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:19:22.889 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 145.8, 2024-06-04T05:19:22.889 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:19:22.889 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:19:22.889 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:19:22.890 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:19:22.890 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:19:22.890 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:19:22.890 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:19:22.890 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:19:22.890 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:19:22.890 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:19:22.890 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:19:22.890 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:19:22.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.885+0000 7fe54d4d0700 1 -- 172.21.15.70:0/184893046 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5300b2aa0 msgr2=0x7fe5300b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:22.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.885+0000 7fe54d4d0700 1 --2- 172.21.15.70:0/184893046 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5300b2aa0 0x7fe5300b4f60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe5481033d0 tx=0x7fe53c009260 comp rx=0 tx=0).stop 2024-06-04T05:19:22.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.885+0000 7fe54d4d0700 1 -- 172.21.15.70:0/184893046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe548103590 msgr2=0x7fe5481a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:22.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.885+0000 7fe54d4d0700 1 --2- 172.21.15.70:0/184893046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe548103590 0x7fe5481a1900 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe538005f50 tx=0x7fe538004270 comp rx=0 tx=0).stop 2024-06-04T05:19:22.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.885+0000 7fe54d4d0700 1 -- 172.21.15.70:0/184893046 shutdown_connections 2024-06-04T05:19:22.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.885+0000 7fe54d4d0700 1 --2- 172.21.15.70:0/184893046 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5300b2aa0 0x7fe5300b4f60 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.885+0000 7fe54d4d0700 1 --2- 172.21.15.70:0/184893046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe548108530 0x7fe54819bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.885+0000 7fe54d4d0700 1 --2- 172.21.15.70:0/184893046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe548103590 0x7fe5481a1900 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.885+0000 7fe54d4d0700 1 --2- 172.21.15.70:0/184893046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe548103f50 0x7fe5481a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:22.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.885+0000 7fe54d4d0700 1 -- 172.21.15.70:0/184893046 >> 172.21.15.70:0/184893046 conn(0x7fe54807a6f0 msgr2=0x7fe548101400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:22.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.885+0000 7fe54d4d0700 1 -- 172.21.15.70:0/184893046 shutdown_connections 2024-06-04T05:19:22.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:22.885+0000 7fe54d4d0700 1 -- 172.21.15.70:0/184893046 wait complete. 2024-06-04T05:19:23.129 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:19:23.129 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:19:23.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:23 smithi070 ceph-mon[32631]: pgmap v699: 97 pgs: 97 active+clean; 915 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.2 MiB/s wr, 322 op/s 2024-06-04T05:19:23.645 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'da6cd7e7', quiesce took 12.3 seconds. Will release after: 13.55 2024-06-04T05:19:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:23 smithi112 ceph-mon[36453]: pgmap v699: 97 pgs: 97 active+clean; 915 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.2 MiB/s wr, 322 op/s 2024-06-04T05:19:23.705 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:19:23.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:23 smithi098 ceph-mon[35569]: pgmap v699: 97 pgs: 97 active+clean; 915 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.2 MiB/s wr, 322 op/s 2024-06-04T05:19:23.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.78 MB/sec execute 375 sec latency 1205.579 ms 2024-06-04T05:19:24.828 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.76 MB/sec execute 376 sec latency 2205.652 ms 2024-06-04T05:19:25.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:25 smithi070 ceph-mon[32631]: pgmap v700: 97 pgs: 97 active+clean; 915 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 286 op/s 2024-06-04T05:19:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:25 smithi112 ceph-mon[36453]: pgmap v700: 97 pgs: 97 active+clean; 915 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 286 op/s 2024-06-04T05:19:25.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:25 smithi098 ceph-mon[35569]: pgmap v700: 97 pgs: 97 active+clean; 915 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 286 op/s 2024-06-04T05:19:25.827 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.74 MB/sec execute 377 sec latency 3205.726 ms 2024-06-04T05:19:26.828 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.72 MB/sec execute 378 sec latency 4205.813 ms 2024-06-04T05:19:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.839+0000 7fb92f2a0640 1 -- 172.21.15.70:0/2378067973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb928102a60 msgr2=0x7fb928102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.839+0000 7fb92f2a0640 1 --2- 172.21.15.70:0/2378067973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb928102a60 0x7fb928102ee0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fb918009930 tx=0x7fb91802a420 comp rx=0 tx=0).stop 2024-06-04T05:19:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.839+0000 7fb92f2a0640 1 -- 172.21.15.70:0/2378067973 shutdown_connections 2024-06-04T05:19:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.839+0000 7fb92f2a0640 1 --2- 172.21.15.70:0/2378067973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb928103420 0x7fb92810f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.839+0000 7fb92f2a0640 1 --2- 172.21.15.70:0/2378067973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb928102a60 0x7fb928102ee0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.839+0000 7fb92f2a0640 1 --2- 172.21.15.70:0/2378067973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb928108a70 0x7fb928108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.839+0000 7fb92f2a0640 1 -- 172.21.15.70:0/2378067973 >> 172.21.15.70:0/2378067973 conn(0x7fb928079ee0 msgr2=0x7fb92807a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.839+0000 7fb92f2a0640 1 -- 172.21.15.70:0/2378067973 shutdown_connections 2024-06-04T05:19:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.839+0000 7fb92f2a0640 1 -- 172.21.15.70:0/2378067973 wait complete. 2024-06-04T05:19:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.840+0000 7fb92f2a0640 1 Processor -- start 2024-06-04T05:19:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.840+0000 7fb92f2a0640 1 -- start start 2024-06-04T05:19:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.840+0000 7fb92f2a0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb928102a60 0x7fb928199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.840+0000 7fb92f2a0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb928103420 0x7fb9281996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.840+0000 7fb92f2a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb928108a70 0x7fb92819d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.840+0000 7fb92f2a0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb928112be0 con 0x7fb928102a60 2024-06-04T05:19:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.840+0000 7fb92f2a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb928112a60 con 0x7fb928108a70 2024-06-04T05:19:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.840+0000 7fb92f2a0640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb928112d60 con 0x7fb928103420 2024-06-04T05:19:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.840+0000 7fb92d03c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb928102a60 0x7fb928199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.840+0000 7fb92d03c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb928102a60 0x7fb928199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60732/0 (socket says 172.21.15.70:60732) 2024-06-04T05:19:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.840+0000 7fb92d03c640 1 -- 172.21.15.70:0/1446413163 learned_addr learned my addr 172.21.15.70:0/1446413163 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.841+0000 7fb92c83b640 1 --2- 172.21.15.70:0/1446413163 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb928103420 0x7fb9281996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.841+0000 7fb92d03c640 1 -- 172.21.15.70:0/1446413163 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb928103420 msgr2=0x7fb9281996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.841+0000 7fb92d03c640 1 --2- 172.21.15.70:0/1446413163 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb928103420 0x7fb9281996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.841+0000 7fb92d03c640 1 -- 172.21.15.70:0/1446413163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb928108a70 msgr2=0x7fb92819d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.841+0000 7fb92d83d640 1 --2- 172.21.15.70:0/1446413163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb928108a70 0x7fb92819d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.841+0000 7fb92d03c640 1 --2- 172.21.15.70:0/1446413163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb928108a70 0x7fb92819d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.841+0000 7fb92d03c640 1 -- 172.21.15.70:0/1446413163 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92819e100 con 0x7fb928102a60 2024-06-04T05:19:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.841+0000 7fb92c83b640 1 --2- 172.21.15.70:0/1446413163 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb928103420 0x7fb9281996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:19:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.841+0000 7fb92d03c640 1 --2- 172.21.15.70:0/1446413163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb928102a60 0x7fb928199160 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fb92400d950 tx=0x7fb92400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.841+0000 7fb91e7fc640 1 -- 172.21.15.70:0/1446413163 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb924017070 con 0x7fb928102a60 2024-06-04T05:19:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.841+0000 7fb91e7fc640 1 -- 172.21.15.70:0/1446413163 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb924009c30 con 0x7fb928102a60 2024-06-04T05:19:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.841+0000 7fb92f2a0640 1 -- 172.21.15.70:0/1446413163 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb92819e3f0 con 0x7fb928102a60 2024-06-04T05:19:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.841+0000 7fb91e7fc640 1 -- 172.21.15.70:0/1446413163 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb92400f620 con 0x7fb928102a60 2024-06-04T05:19:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.841+0000 7fb92f2a0640 1 -- 172.21.15.70:0/1446413163 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9281a5c80 con 0x7fb928102a60 2024-06-04T05:19:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.843+0000 7fb91e7fc640 1 -- 172.21.15.70:0/1446413163 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fb924010460 con 0x7fb928102a60 2024-06-04T05:19:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.844+0000 7fb931fc0740 1 Processor -- start 2024-06-04T05:19:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.844+0000 7fb931fc0740 1 -- start start 2024-06-04T05:19:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.844+0000 7fb931fc0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebda820 0x55bd2ebdac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.844+0000 7fb931fc0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebdb140 0x55bd2ebe54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.844+0000 7fb931fc0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebe5a00 0x55bd2ebe7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.844+0000 7fb931fc0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bd2eb4cc30 con 0x55bd2ebdb140 2024-06-04T05:19:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.844+0000 7fb931fc0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bd2eb3c7d0 con 0x55bd2ebe5a00 2024-06-04T05:19:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.844+0000 7fb931fc0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bd2eb40810 con 0x55bd2ebda820 2024-06-04T05:19:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.844+0000 7fb91e7fc640 1 --2- 172.21.15.70:0/1446413163 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9140b2a90 0x7fb9140b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.844+0000 7fb92d03c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebdb140 0x55bd2ebe54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.844+0000 7fb92d03c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebdb140 0x55bd2ebe54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60734/0 (socket says 172.21.15.70:60734) 2024-06-04T05:19:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.844+0000 7fb92d03c640 1 -- 172.21.15.70:0/2672519867 learned_addr learned my addr 172.21.15.70:0/2672519867 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.845+0000 7fb92d83d640 1 --2- 172.21.15.70:0/1446413163 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9140b2a90 0x7fb9140b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.845+0000 7fb92c83b640 1 --2- 172.21.15.70:0/2672519867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebe5a00 0x55bd2ebe7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.845+0000 7fb92d83d640 1 --2- 172.21.15.70:0/2672519867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebda820 0x55bd2ebdac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.845+0000 7fb91e7fc640 1 -- 172.21.15.70:0/1446413163 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7fb9240e2240 con 0x7fb928102a60 2024-06-04T05:19:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.845+0000 7fb92d03c640 1 -- 172.21.15.70:0/2672519867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebda820 msgr2=0x55bd2ebdac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.845+0000 7fb92d03c640 1 --2- 172.21.15.70:0/2672519867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebda820 0x55bd2ebdac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.845+0000 7fb92d03c640 1 -- 172.21.15.70:0/2672519867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebe5a00 msgr2=0x55bd2ebe7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.845+0000 7fb92d03c640 1 --2- 172.21.15.70:0/2672519867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebe5a00 0x55bd2ebe7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.845+0000 7fb92d03c640 1 -- 172.21.15.70:0/2672519867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd2e851210 con 0x55bd2ebdb140 2024-06-04T05:19:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.845+0000 7fb92d03c640 1 --2- 172.21.15.70:0/2672519867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebdb140 0x55bd2ebe54c0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fb9240df900 tx=0x7fb924005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e16f6bef2e770ba server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.845+0000 7fb92d83d640 1 --2- 172.21.15.70:0/1446413163 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9140b2a90 0x7fb9140b4f50 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb920009fd0 tx=0x7fb920009570 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.845+0000 7fb9137fe640 1 -- 172.21.15.70:0/2672519867 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb924020070 con 0x55bd2ebdb140 2024-06-04T05:19:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.845+0000 7fb9137fe640 1 -- 172.21.15.70:0/2672519867 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb9240a9bf0 con 0x55bd2ebdb140 2024-06-04T05:19:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.845+0000 7fb9137fe640 1 -- 172.21.15.70:0/2672519867 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9240109b0 con 0x55bd2ebdb140 2024-06-04T05:19:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.846+0000 7fb931fc0740 1 -- 172.21.15.70:0/2672519867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebdb140 msgr2=0x55bd2ebe54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.846+0000 7fb931fc0740 1 --2- 172.21.15.70:0/2672519867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebdb140 0x55bd2ebe54c0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fb9240df900 tx=0x7fb924005880 comp rx=0 tx=0).stop 2024-06-04T05:19:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.846+0000 7fb931fc0740 1 -- 172.21.15.70:0/2672519867 shutdown_connections 2024-06-04T05:19:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.846+0000 7fb931fc0740 1 --2- 172.21.15.70:0/2672519867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebe5a00 0x55bd2ebe7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.846+0000 7fb931fc0740 1 --2- 172.21.15.70:0/2672519867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebdb140 0x55bd2ebe54c0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.846+0000 7fb931fc0740 1 --2- 172.21.15.70:0/2672519867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebda820 0x55bd2ebdac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.846+0000 7fb931fc0740 1 -- 172.21.15.70:0/2672519867 >> 172.21.15.70:0/2672519867 conn(0x55bd2ebd6d30 msgr2=0x55bd2ebd7110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.846+0000 7fb931fc0740 1 -- 172.21.15.70:0/2672519867 shutdown_connections 2024-06-04T05:19:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.846+0000 7fb931fc0740 1 -- 172.21.15.70:0/2672519867 wait complete. 2024-06-04T05:19:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.846+0000 7fb931fc0740 1 -- start start 2024-06-04T05:19:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.846+0000 7fb931fc0740 1 Processor -- start 2024-06-04T05:19:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.847+0000 7fb931fc0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebda820 0x55bd2ebe29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.847+0000 7fb931fc0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebdb140 0x55bd2ebe2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.847+0000 7fb931fc0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ebe3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.847+0000 7fb931fc0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bd2e822790 con 0x55bd2ebe5a00 2024-06-04T05:19:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.847+0000 7fb931fc0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bd2eaf8890 con 0x55bd2ebdb140 2024-06-04T05:19:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.847+0000 7fb931fc0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bd2ebe85d0 con 0x55bd2ebda820 2024-06-04T05:19:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.847+0000 7fb931fc0740 10 client.0 _ll_register_callbacks cb 0x55bd2e850a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.847+0000 7fb92d83d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ebe3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.847+0000 7fb92d83d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ebe3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60738/0 (socket says 172.21.15.70:60738) 2024-06-04T05:19:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.847+0000 7fb92d83d640 1 -- 172.21.15.70:0/1592052581 learned_addr learned my addr 172.21.15.70:0/1592052581 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.847+0000 7fb92c83b640 1 --2- 172.21.15.70:0/1592052581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebdb140 0x55bd2ebe2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.847+0000 7fb92d03c640 1 --2- 172.21.15.70:0/1592052581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebda820 0x55bd2ebe29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.847+0000 7fb92d83d640 1 -- 172.21.15.70:0/1592052581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebda820 msgr2=0x55bd2ebe29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.847+0000 7fb92d83d640 1 --2- 172.21.15.70:0/1592052581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebda820 0x55bd2ebe29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.847+0000 7fb92d83d640 1 -- 172.21.15.70:0/1592052581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebdb140 msgr2=0x55bd2ebe2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.847+0000 7fb92d83d640 1 --2- 172.21.15.70:0/1592052581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebdb140 0x55bd2ebe2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.847+0000 7fb92d83d640 1 -- 172.21.15.70:0/1592052581 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd2e91d620 con 0x55bd2ebe5a00 2024-06-04T05:19:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.848+0000 7fb92d83d640 1 --2- 172.21.15.70:0/1592052581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ebe3420 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fb92000fae0 tx=0x7fb920004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7983acc981b01d6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.848+0000 7fb910ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.848+0000 7fb910ff9640 1 -- 172.21.15.70:0/1592052581 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb920025070 con 0x55bd2ebe5a00 2024-06-04T05:19:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.848+0000 7fb910ff9640 1 -- 172.21.15.70:0/1592052581 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7fb920012d80 con 0x55bd2ebe5a00 2024-06-04T05:19:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.848+0000 7fb931fc0740 1 -- 172.21.15.70:0/1592052581 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9240a9bf0 con 0x55bd2ebe5a00 2024-06-04T05:19:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.848+0000 7fb910ff9640 1 -- 172.21.15.70:0/1592052581 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fb920021420 con 0x55bd2ebe5a00 2024-06-04T05:19:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.848+0000 7fb931fc0740 10 client.21159 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.848+0000 7fb931fc0740 1 -- 172.21.15.70:0/1592052581 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55bd2ebf43a0 con 0x55bd2ebe5a00 2024-06-04T05:19:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.848+0000 7fb910ff9640 1 -- 172.21.15.70:0/1592052581 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb9240a9bf0 con 0x55bd2ebe5a00 2024-06-04T05:19:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.848+0000 7fb910ff9640 1 -- 172.21.15.70:0/1592052581 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7fb92000c030 con 0x55bd2ebe5a00 2024-06-04T05:19:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.848+0000 7fb931fc0740 10 client.21159 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.848+0000 7fb931fc0740 10 client.21159 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:19:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.848+0000 7fb931fc0740 20 client.21159 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.848+0000 7fb931fc0740 1 --2- 172.21.15.70:0/1592052581 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bd2ebf58d0 0x55bd2ebf7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.848+0000 7fb931fc0740 4 client.21159 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:19:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.848+0000 7fb931fc0740 1 -- 172.21.15.70:0/1592052581 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd2eb4eb80 con 0x55bd2ebf58d0 2024-06-04T05:19:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.849+0000 7fb92d03c640 1 --2- 172.21.15.70:0/1592052581 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bd2ebf58d0 0x55bd2ebf7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.849+0000 7fb910ff9640 10 client.21159 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.849+0000 7fb92d03c640 1 --2- 172.21.15.70:0/1592052581 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bd2ebf58d0 0x55bd2ebf7cc0 crc :-1 s=READY pgs=469 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91237b131c3eace9 server_cookie=fa92432046a4dc0e in_seq=0 out_seq=0 2024-06-04T05:19:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.850+0000 7fb910ff9640 1 -- 172.21.15.70:0/1592052581 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bd2eb4eb80 con 0x55bd2ebf58d0 2024-06-04T05:19:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.850+0000 7fb910ff9640 10 client.21159 handle_command_reply: tid=0 2024-06-04T05:19:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.850+0000 7fb931fc0740 1 client.21159 shutdown 2024-06-04T05:19:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.850+0000 7fb931fc0740 1 -- 172.21.15.70:0/1592052581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 msgr2=0x55bd2ebe3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.850+0000 7fb931fc0740 1 --2- 172.21.15.70:0/1592052581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ebe3420 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fb92000fae0 tx=0x7fb920004a40 comp rx=0 tx=0).stop 2024-06-04T05:19:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.850+0000 7fb931fc0740 1 -- 172.21.15.70:0/1592052581 shutdown_connections 2024-06-04T05:19:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.850+0000 7fb931fc0740 1 --2- 172.21.15.70:0/1592052581 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bd2ebf58d0 0x55bd2ebf7cc0 crc :-1 s=READY pgs=469 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.850+0000 7fb931fc0740 1 --2- 172.21.15.70:0/1592052581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebdb140 0x55bd2ebe2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.850+0000 7fb931fc0740 1 --2- 172.21.15.70:0/1592052581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ebe3420 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.850+0000 7fb910ff9640 0 client.21159 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.850+0000 7fb931fc0740 1 --2- 172.21.15.70:0/1592052581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebda820 0x55bd2ebe29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.850+0000 7fb931fc0740 1 -- 172.21.15.70:0/1592052581 >> 172.21.15.70:0/1592052581 conn(0x55bd2ebd6d30 msgr2=0x55bd2ebd5fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.850+0000 7fb931fc0740 1 -- 172.21.15.70:0/1592052581 shutdown_connections 2024-06-04T05:19:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.850+0000 7fb931fc0740 1 -- 172.21.15.70:0/1592052581 wait complete. 2024-06-04T05:19:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.850+0000 7fb931fc0740 20 client.21159 trim_cache size 0 max 16384 2024-06-04T05:19:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.865+0000 7fb931fc0740 1 Processor -- start 2024-06-04T05:19:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.865+0000 7fb931fc0740 1 -- start start 2024-06-04T05:19:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.865+0000 7fb931fc0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebda820 0x55bd2ebedc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.865+0000 7fb931fc0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebdb140 0x55bd2ebe2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb931fc0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ebe2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb931fc0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9281ae7c0 con 0x55bd2ebe5a00 2024-06-04T05:19:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb931fc0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9281b0360 con 0x55bd2ebda820 2024-06-04T05:19:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb931fc0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb9281aeef0 con 0x55bd2ebdb140 2024-06-04T05:19:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb92d83d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ebe2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb92d83d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ebe2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60744/0 (socket says 172.21.15.70:60744) 2024-06-04T05:19:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb92d83d640 1 -- 172.21.15.70:0/180086705 learned_addr learned my addr 172.21.15.70:0/180086705 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb92d83d640 1 -- 172.21.15.70:0/180086705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebdb140 msgr2=0x55bd2ebe2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb92d83d640 1 --2- 172.21.15.70:0/180086705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebdb140 0x55bd2ebe2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb92d83d640 1 -- 172.21.15.70:0/180086705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebda820 msgr2=0x55bd2ebedc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb92d03c640 1 --2- 172.21.15.70:0/180086705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebda820 0x55bd2ebedc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb92d83d640 1 --2- 172.21.15.70:0/180086705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebda820 0x55bd2ebedc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb92d83d640 1 -- 172.21.15.70:0/180086705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd2ebe0ef0 con 0x55bd2ebe5a00 2024-06-04T05:19:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb92d03c640 1 --2- 172.21.15.70:0/180086705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebda820 0x55bd2ebedc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:19:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb92d83d640 1 --2- 172.21.15.70:0/180086705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ebe2ea0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fb92000f740 tx=0x7fb920005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e902a6c9b62113e4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb912ffd640 1 -- 172.21.15.70:0/180086705 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb920025040 con 0x55bd2ebe5a00 2024-06-04T05:19:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb912ffd640 1 -- 172.21.15.70:0/180086705 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb920012d80 con 0x55bd2ebe5a00 2024-06-04T05:19:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.866+0000 7fb912ffd640 1 -- 172.21.15.70:0/180086705 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb920011cb0 con 0x55bd2ebe5a00 2024-06-04T05:19:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.867+0000 7fb931fc0740 1 -- 172.21.15.70:0/180086705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 msgr2=0x55bd2ebe2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.867+0000 7fb931fc0740 1 --2- 172.21.15.70:0/180086705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ebe2ea0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fb92000f740 tx=0x7fb920005ce0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.867+0000 7fb931fc0740 1 -- 172.21.15.70:0/180086705 shutdown_connections 2024-06-04T05:19:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.867+0000 7fb931fc0740 1 --2- 172.21.15.70:0/180086705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ebe2ea0 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.867+0000 7fb931fc0740 1 --2- 172.21.15.70:0/180086705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebdb140 0x55bd2ebe2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.867+0000 7fb931fc0740 1 --2- 172.21.15.70:0/180086705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebda820 0x55bd2ebedc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.867+0000 7fb931fc0740 1 -- 172.21.15.70:0/180086705 >> 172.21.15.70:0/180086705 conn(0x55bd2ebd6d30 msgr2=0x55bd2ebd1e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.867+0000 7fb931fc0740 1 -- 172.21.15.70:0/180086705 shutdown_connections 2024-06-04T05:19:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.867+0000 7fb931fc0740 1 -- 172.21.15.70:0/180086705 wait complete. 2024-06-04T05:19:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.867+0000 7fb931fc0740 1 -- start start 2024-06-04T05:19:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.867+0000 7fb931fc0740 1 Processor -- start 2024-06-04T05:19:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb931fc0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebda820 0x55bd2ebfd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb931fc0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebdb140 0x55bd2ebfda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb931fc0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ec5bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb931fc0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9281afc90 con 0x55bd2ebe5a00 2024-06-04T05:19:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb931fc0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9281af5c0 con 0x55bd2ebda820 2024-06-04T05:19:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb931fc0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb9281ac7c0 con 0x55bd2ebdb140 2024-06-04T05:19:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb931fc0740 10 client.0 _ll_register_callbacks cb 0x7fb9200097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb92d83d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ec5bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb92d83d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ec5bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60750/0 (socket says 172.21.15.70:60750) 2024-06-04T05:19:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb92d83d640 1 -- 172.21.15.70:0/951138464 learned_addr learned my addr 172.21.15.70:0/951138464 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb92d03c640 1 --2- 172.21.15.70:0/951138464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebda820 0x55bd2ebfd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb92c83b640 1 --2- 172.21.15.70:0/951138464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebdb140 0x55bd2ebfda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb92d83d640 1 -- 172.21.15.70:0/951138464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebdb140 msgr2=0x55bd2ebfda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb92d83d640 1 --2- 172.21.15.70:0/951138464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebdb140 0x55bd2ebfda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb92d83d640 1 -- 172.21.15.70:0/951138464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebda820 msgr2=0x55bd2ebfd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb92d83d640 1 --2- 172.21.15.70:0/951138464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebda820 0x55bd2ebfd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb92d83d640 1 -- 172.21.15.70:0/951138464 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd2ec5c210 con 0x55bd2ebe5a00 2024-06-04T05:19:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb92d83d640 1 --2- 172.21.15.70:0/951138464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ec5bba0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fb920005fc0 tx=0x7fb9200118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc6c977048e5ca9c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb910ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb910ff9640 1 -- 172.21.15.70:0/951138464 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb920044060 con 0x55bd2ebe5a00 2024-06-04T05:19:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb910ff9640 1 -- 172.21.15.70:0/951138464 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7fb920012b80 con 0x55bd2ebe5a00 2024-06-04T05:19:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.868+0000 7fb931fc0740 1 -- 172.21.15.70:0/951138464 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb920012d80 con 0x55bd2ebe5a00 2024-06-04T05:19:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.869+0000 7fb910ff9640 1 -- 172.21.15.70:0/951138464 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fb920025070 con 0x55bd2ebe5a00 2024-06-04T05:19:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.869+0000 7fb931fc0740 10 client.21165 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.869+0000 7fb931fc0740 1 -- 172.21.15.70:0/951138464 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55bd2ec64730 con 0x55bd2ebe5a00 2024-06-04T05:19:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.869+0000 7fb910ff9640 1 -- 172.21.15.70:0/951138464 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb920012d80 con 0x55bd2ebe5a00 2024-06-04T05:19:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.869+0000 7fb910ff9640 1 -- 172.21.15.70:0/951138464 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7fb920021a50 con 0x55bd2ebe5a00 2024-06-04T05:19:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.869+0000 7fb931fc0740 10 client.21165 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.869+0000 7fb931fc0740 10 client.21165 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:19:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.869+0000 7fb931fc0740 20 client.21165 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:26.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.869+0000 7fb931fc0740 1 --2- 172.21.15.70:0/951138464 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bd2ebf58d0 0x55bd2ec66d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:26.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.869+0000 7fb931fc0740 4 client.21165 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:19:26.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.869+0000 7fb931fc0740 1 -- 172.21.15.70:0/951138464 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bd2eb48360 con 0x55bd2ebf58d0 2024-06-04T05:19:26.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.869+0000 7fb92d03c640 1 --2- 172.21.15.70:0/951138464 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bd2ebf58d0 0x55bd2ec66d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:26.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.870+0000 7fb910ff9640 10 client.21165 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:26.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.870+0000 7fb92d03c640 1 --2- 172.21.15.70:0/951138464 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bd2ebf58d0 0x55bd2ec66d30 crc :-1 s=READY pgs=470 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2d7db3767962d4e server_cookie=e897b5065d802673 in_seq=0 out_seq=0 2024-06-04T05:19:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.942+0000 7fb910ff9640 1 -- 172.21.15.70:0/951138464 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55bd2eb48360 con 0x55bd2ebf58d0 2024-06-04T05:19:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.942+0000 7fb910ff9640 10 client.21165 handle_command_reply: tid=0 2024-06-04T05:19:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.942+0000 7fb931fc0740 1 client.21165 shutdown 2024-06-04T05:19:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.942+0000 7fb931fc0740 1 -- 172.21.15.70:0/951138464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 msgr2=0x55bd2ec5bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.942+0000 7fb931fc0740 1 --2- 172.21.15.70:0/951138464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ec5bba0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fb920005fc0 tx=0x7fb9200118d0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.942+0000 7fb931fc0740 1 -- 172.21.15.70:0/951138464 shutdown_connections 2024-06-04T05:19:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.942+0000 7fb931fc0740 1 --2- 172.21.15.70:0/951138464 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bd2ebf58d0 0x55bd2ec66d30 crc :-1 s=READY pgs=470 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.942+0000 7fb931fc0740 1 --2- 172.21.15.70:0/951138464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd2ebe5a00 0x55bd2ec5bba0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.942+0000 7fb931fc0740 1 --2- 172.21.15.70:0/951138464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd2ebdb140 0x55bd2ebfda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.942+0000 7fb931fc0740 1 --2- 172.21.15.70:0/951138464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd2ebda820 0x55bd2ebfd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.942+0000 7fb910ff9640 0 client.21165 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.942+0000 7fb931fc0740 1 -- 172.21.15.70:0/951138464 >> 172.21.15.70:0/951138464 conn(0x55bd2ebd6d30 msgr2=0x55bd2ebe2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.942+0000 7fb931fc0740 1 -- 172.21.15.70:0/951138464 shutdown_connections 2024-06-04T05:19:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.942+0000 7fb931fc0740 1 -- 172.21.15.70:0/951138464 wait complete. 2024-06-04T05:19:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.942+0000 7fb931fc0740 20 client.21165 trim_cache size 0 max 16384 2024-06-04T05:19:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:19:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:19:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:19:26.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.943+0000 7fb92f2a0640 1 -- 172.21.15.70:0/1446413163 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9140b2a90 msgr2=0x7fb9140b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:26.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.943+0000 7fb92f2a0640 1 --2- 172.21.15.70:0/1446413163 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9140b2a90 0x7fb9140b4f50 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb920009fd0 tx=0x7fb920009570 comp rx=0 tx=0).stop 2024-06-04T05:19:26.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.943+0000 7fb92f2a0640 1 -- 172.21.15.70:0/1446413163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb928102a60 msgr2=0x7fb928199160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:26.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.943+0000 7fb92f2a0640 1 --2- 172.21.15.70:0/1446413163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb928102a60 0x7fb928199160 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fb92400d950 tx=0x7fb92400de10 comp rx=0 tx=0).stop 2024-06-04T05:19:26.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.943+0000 7fb92f2a0640 1 -- 172.21.15.70:0/1446413163 shutdown_connections 2024-06-04T05:19:26.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.943+0000 7fb92f2a0640 1 --2- 172.21.15.70:0/1446413163 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb9140b2a90 0x7fb9140b4f50 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.943+0000 7fb92f2a0640 1 --2- 172.21.15.70:0/1446413163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb928108a70 0x7fb92819d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.943+0000 7fb92f2a0640 1 --2- 172.21.15.70:0/1446413163 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb928103420 0x7fb9281996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.943+0000 7fb92f2a0640 1 --2- 172.21.15.70:0/1446413163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb928102a60 0x7fb928199160 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:26.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.943+0000 7fb92f2a0640 1 -- 172.21.15.70:0/1446413163 >> 172.21.15.70:0/1446413163 conn(0x7fb928079ee0 msgr2=0x7fb928100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:26.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.943+0000 7fb92f2a0640 1 -- 172.21.15.70:0/1446413163 shutdown_connections 2024-06-04T05:19:26.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:26.943+0000 7fb92f2a0640 1 -- 172.21.15.70:0/1446413163 wait complete. 2024-06-04T05:19:27.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:27 smithi070 ceph-mon[32631]: pgmap v701: 97 pgs: 97 active+clean; 921 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 333 op/s 2024-06-04T05:19:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:27 smithi112 ceph-mon[36453]: pgmap v701: 97 pgs: 97 active+clean; 921 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 333 op/s 2024-06-04T05:19:27.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:27 smithi098 ceph-mon[35569]: pgmap v701: 97 pgs: 97 active+clean; 921 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 333 op/s 2024-06-04T05:19:27.828 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.70 MB/sec execute 379 sec latency 5205.895 ms 2024-06-04T05:19:28.828 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.69 MB/sec execute 380 sec latency 6205.967 ms 2024-06-04T05:19:29.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:29 smithi070 ceph-mon[32631]: pgmap v702: 97 pgs: 97 active+clean; 921 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 190 op/s 2024-06-04T05:19:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:29 smithi112 ceph-mon[36453]: pgmap v702: 97 pgs: 97 active+clean; 921 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 190 op/s 2024-06-04T05:19:29.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:29 smithi098 ceph-mon[35569]: pgmap v702: 97 pgs: 97 active+clean; 921 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 190 op/s 2024-06-04T05:19:29.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.766+0000 7f3e381d6700 1 -- 172.21.15.70:0/1236409288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e30102440 msgr2=0x7f3e301028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:29.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.767+0000 7f3e381d6700 1 --2- 172.21.15.70:0/1236409288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e30102440 0x7f3e301028c0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f3e20009990 tx=0x7f3e20009ca0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.767+0000 7f3e381d6700 1 -- 172.21.15.70:0/1236409288 shutdown_connections 2024-06-04T05:19:29.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.767+0000 7f3e381d6700 1 --2- 172.21.15.70:0/1236409288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e30102e00 0x7f3e3010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.767+0000 7f3e381d6700 1 --2- 172.21.15.70:0/1236409288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e30108520 0x7f3e30108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.767+0000 7f3e381d6700 1 --2- 172.21.15.70:0/1236409288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e30102440 0x7f3e301028c0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.767+0000 7f3e381d6700 1 -- 172.21.15.70:0/1236409288 >> 172.21.15.70:0/1236409288 conn(0x7f3e300793f0 msgr2=0x7f3e30079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:29.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.767+0000 7f3e381d6700 1 -- 172.21.15.70:0/1236409288 shutdown_connections 2024-06-04T05:19:29.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.767+0000 7f3e381d6700 1 -- 172.21.15.70:0/1236409288 wait complete. 2024-06-04T05:19:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.767+0000 7f3e381d6700 1 Processor -- start 2024-06-04T05:19:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.767+0000 7f3e381d6700 1 -- start start 2024-06-04T05:19:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.767+0000 7f3e381d6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e30102440 0x7f3e3019bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.767+0000 7f3e381d6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e30102e00 0x7f3e3019c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.767+0000 7f3e381d6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e30108520 0x7f3e301a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e381d6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e30112580 con 0x7f3e30108520 2024-06-04T05:19:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e381d6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e30112400 con 0x7f3e30102e00 2024-06-04T05:19:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e381d6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3e30112700 con 0x7f3e30102440 2024-06-04T05:19:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e36773700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e30108520 0x7f3e301a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e36773700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e30108520 0x7f3e301a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60760/0 (socket says 172.21.15.70:60760) 2024-06-04T05:19:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e36773700 1 -- 172.21.15.70:0/193470669 learned_addr learned my addr 172.21.15.70:0/193470669 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e35f72700 1 --2- 172.21.15.70:0/193470669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e30102440 0x7f3e3019bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e35771700 1 --2- 172.21.15.70:0/193470669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e30102e00 0x7f3e3019c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e36773700 1 -- 172.21.15.70:0/193470669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e30102440 msgr2=0x7f3e3019bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e36773700 1 --2- 172.21.15.70:0/193470669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e30102440 0x7f3e3019bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e36773700 1 -- 172.21.15.70:0/193470669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e30102e00 msgr2=0x7f3e3019c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e36773700 1 --2- 172.21.15.70:0/193470669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e30102e00 0x7f3e3019c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e36773700 1 -- 172.21.15.70:0/193470669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e301a0ea0 con 0x7f3e30108520 2024-06-04T05:19:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e35f72700 1 --2- 172.21.15.70:0/193470669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e30102440 0x7f3e3019bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:19:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e35771700 1 --2- 172.21.15.70:0/193470669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e30102e00 0x7f3e3019c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:19:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e36773700 1 --2- 172.21.15.70:0/193470669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e30108520 0x7f3e301a0760 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800b970 tx=0x7f3e2800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e26ffd700 1 -- 172.21.15.70:0/193470669 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e28019070 con 0x7f3e30108520 2024-06-04T05:19:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e26ffd700 1 -- 172.21.15.70:0/193470669 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e2800ce40 con 0x7f3e30108520 2024-06-04T05:19:29.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e381d6700 1 -- 172.21.15.70:0/193470669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e301a1190 con 0x7f3e30108520 2024-06-04T05:19:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e26ffd700 1 -- 172.21.15.70:0/193470669 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e28007370 con 0x7f3e30108520 2024-06-04T05:19:29.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.768+0000 7f3e381d6700 1 -- 172.21.15.70:0/193470669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e301a8a10 con 0x7f3e30108520 2024-06-04T05:19:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.771+0000 7f3e26ffd700 1 -- 172.21.15.70:0/193470669 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3e2801c070 con 0x7f3e30108520 2024-06-04T05:19:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.771+0000 7f3e26ffd700 1 --2- 172.21.15.70:0/193470669 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e1c0b29f0 0x7f3e1c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.771+0000 7f3e35f72700 1 --2- 172.21.15.70:0/193470669 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e1c0b29f0 0x7f3e1c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.772+0000 7f3e26ffd700 1 -- 172.21.15.70:0/193470669 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f3e280e1b30 con 0x7f3e30108520 2024-06-04T05:19:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.772+0000 7f3e35f72700 1 --2- 172.21.15.70:0/193470669 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e1c0b29f0 0x7f3e1c0b4eb0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c0099d0 tx=0x7f3e2c008040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.772+0000 7f3e42a6bb80 1 Processor -- start 2024-06-04T05:19:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.772+0000 7f3e42a6bb80 1 -- start start 2024-06-04T05:19:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e42a6bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd978e0 0x55c84dd97cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e42a6bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd98290 0x55c84dd9b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e42a6bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd9bfe0 0x55c84dd9c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e42a6bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c84dcf6d10 con 0x55c84dd978e0 2024-06-04T05:19:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e42a6bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c84dce68b0 con 0x55c84dd9bfe0 2024-06-04T05:19:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e42a6bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c84dcea8f0 con 0x55c84dd98290 2024-06-04T05:19:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e35f72700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd978e0 0x55c84dd97cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e35f72700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd978e0 0x55c84dd97cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60762/0 (socket says 172.21.15.70:60762) 2024-06-04T05:19:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e35f72700 1 -- 172.21.15.70:0/3606564928 learned_addr learned my addr 172.21.15.70:0/3606564928 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e35771700 1 --2- 172.21.15.70:0/3606564928 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd98290 0x55c84dd9b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e36773700 1 --2- 172.21.15.70:0/3606564928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd9bfe0 0x55c84dd9c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e35f72700 1 -- 172.21.15.70:0/3606564928 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd98290 msgr2=0x55c84dd9b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e35f72700 1 --2- 172.21.15.70:0/3606564928 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd98290 0x55c84dd9b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e35f72700 1 -- 172.21.15.70:0/3606564928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd9bfe0 msgr2=0x55c84dd9c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e35f72700 1 --2- 172.21.15.70:0/3606564928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd9bfe0 0x55c84dd9c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e35f72700 1 -- 172.21.15.70:0/3606564928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c84dc36950 con 0x55c84dd978e0 2024-06-04T05:19:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e36773700 1 --2- 172.21.15.70:0/3606564928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd9bfe0 0x55c84dd9c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:19:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.773+0000 7f3e35f72700 1 --2- 172.21.15.70:0/3606564928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd978e0 0x55c84dd97cc0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00d7c0 tx=0x7f3e2c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a73f0f9f9d6003d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.774+0000 7f3e1bfff700 1 -- 172.21.15.70:0/3606564928 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c024070 con 0x55c84dd978e0 2024-06-04T05:19:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.774+0000 7f3e1bfff700 1 -- 172.21.15.70:0/3606564928 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e2c014ad0 con 0x55c84dd978e0 2024-06-04T05:19:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.774+0000 7f3e1bfff700 1 -- 172.21.15.70:0/3606564928 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c01fa30 con 0x55c84dd978e0 2024-06-04T05:19:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.774+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/3606564928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd978e0 msgr2=0x55c84dd97cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.774+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/3606564928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd978e0 0x55c84dd97cc0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00d7c0 tx=0x7f3e2c010000 comp rx=0 tx=0).stop 2024-06-04T05:19:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.774+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/3606564928 shutdown_connections 2024-06-04T05:19:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.774+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/3606564928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd9bfe0 0x55c84dd9c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.774+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/3606564928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd978e0 0x55c84dd97cc0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.774+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/3606564928 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd98290 0x55c84dd9b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.774+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/3606564928 >> 172.21.15.70:0/3606564928 conn(0x55c84dcc5100 msgr2=0x55c84dd91f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.775+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/3606564928 shutdown_connections 2024-06-04T05:19:29.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.775+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/3606564928 wait complete. 2024-06-04T05:19:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.775+0000 7f3e42a6bb80 1 -- start start 2024-06-04T05:19:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.775+0000 7f3e42a6bb80 1 Processor -- start 2024-06-04T05:19:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.775+0000 7f3e42a6bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd978e0 0x55c84dda3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.775+0000 7f3e42a6bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd98290 0x55c84dda0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.775+0000 7f3e42a6bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd9bfe0 0x55c84dda0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.775+0000 7f3e42a6bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c84dbed020 con 0x55c84dd978e0 2024-06-04T05:19:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.775+0000 7f3e42a6bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c84dc88840 con 0x55c84dd98290 2024-06-04T05:19:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.775+0000 7f3e42a6bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c84dda41a0 con 0x55c84dd9bfe0 2024-06-04T05:19:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e35f72700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd978e0 0x55c84dda3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e35f72700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd978e0 0x55c84dda3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60770/0 (socket says 172.21.15.70:60770) 2024-06-04T05:19:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e35f72700 1 -- 172.21.15.70:0/1847420704 learned_addr learned my addr 172.21.15.70:0/1847420704 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e42a6bb80 10 client.0 _ll_register_callbacks cb 0x55c84dc819a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e35771700 1 --2- 172.21.15.70:0/1847420704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd98290 0x55c84dda0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e36773700 1 --2- 172.21.15.70:0/1847420704 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd9bfe0 0x55c84dda0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e35f72700 1 -- 172.21.15.70:0/1847420704 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd9bfe0 msgr2=0x55c84dda0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e35f72700 1 --2- 172.21.15.70:0/1847420704 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd9bfe0 0x55c84dda0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e35f72700 1 -- 172.21.15.70:0/1847420704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd98290 msgr2=0x55c84dda0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e35f72700 1 --2- 172.21.15.70:0/1847420704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd98290 0x55c84dda0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e35f72700 1 -- 172.21.15.70:0/1847420704 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c84dc85710 con 0x55c84dd978e0 2024-06-04T05:19:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e35f72700 1 --2- 172.21.15.70:0/1847420704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd978e0 0x55c84dda3410 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c022040 tx=0x7f3e2c014640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678ec72e7f341169 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e197fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e197fa700 1 -- 172.21.15.70:0/1847420704 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c02f060 con 0x55c84dd978e0 2024-06-04T05:19:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e197fa700 1 -- 172.21.15.70:0/1847420704 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f3e2c01f6d0 con 0x55c84dd978e0 2024-06-04T05:19:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/1847420704 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e2c014ad0 con 0x55c84dd978e0 2024-06-04T05:19:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e197fa700 1 -- 172.21.15.70:0/1847420704 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f3e2c024040 con 0x55c84dd978e0 2024-06-04T05:19:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e42a6bb80 10 client.21177 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:29.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.776+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/1847420704 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55c84ddae860 con 0x55c84dd978e0 2024-06-04T05:19:29.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.777+0000 7f3e197fa700 1 -- 172.21.15.70:0/1847420704 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e2c014ad0 con 0x55c84dd978e0 2024-06-04T05:19:29.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.777+0000 7f3e197fa700 1 -- 172.21.15.70:0/1847420704 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f3e2c013070 con 0x55c84dd978e0 2024-06-04T05:19:29.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.777+0000 7f3e42a6bb80 10 client.21177 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:29.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.777+0000 7f3e42a6bb80 10 client.21177 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:19:29.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.777+0000 7f3e42a6bb80 20 client.21177 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:29.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.777+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/1847420704 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c84ddb0de0 0x55c84ddb31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.777+0000 7f3e42a6bb80 4 client.21177 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:19:29.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.777+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/1847420704 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c84dd63a60 con 0x55c84ddb0de0 2024-06-04T05:19:29.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.777+0000 7f3e35771700 1 --2- 172.21.15.70:0/1847420704 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c84ddb0de0 0x55c84ddb31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.778+0000 7f3e35771700 1 --2- 172.21.15.70:0/1847420704 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c84ddb0de0 0x55c84ddb31d0 crc :-1 s=READY pgs=471 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a5872e6605e8d66 server_cookie=1343b84138e030b in_seq=0 out_seq=0 2024-06-04T05:19:29.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.778+0000 7f3e197fa700 10 client.21177 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:29.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.778+0000 7f3e197fa700 1 -- 172.21.15.70:0/1847420704 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c84dd63a60 con 0x55c84ddb0de0 2024-06-04T05:19:29.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.778+0000 7f3e197fa700 10 client.21177 handle_command_reply: tid=0 2024-06-04T05:19:29.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.778+0000 7f3e42a6bb80 1 client.21177 shutdown 2024-06-04T05:19:29.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.779+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/1847420704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd978e0 msgr2=0x55c84dda3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:29.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.779+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/1847420704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd978e0 0x55c84dda3410 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c022040 tx=0x7f3e2c014640 comp rx=0 tx=0).stop 2024-06-04T05:19:29.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.779+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/1847420704 shutdown_connections 2024-06-04T05:19:29.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.779+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/1847420704 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd9bfe0 0x55c84dda0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.779+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/1847420704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd978e0 0x55c84dda3410 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.779+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/1847420704 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c84ddb0de0 0x55c84ddb31d0 crc :-1 s=READY pgs=471 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.779+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/1847420704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd98290 0x55c84dda0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.779+0000 7f3e197fa700 0 client.21177 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:29.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.779+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/1847420704 >> 172.21.15.70:0/1847420704 conn(0x55c84dcc5100 msgr2=0x55c84dd91590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:29.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.779+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/1847420704 shutdown_connections 2024-06-04T05:19:29.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.779+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/1847420704 wait complete. 2024-06-04T05:19:29.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.779+0000 7f3e42a6bb80 20 client.21177 trim_cache size 0 max 16384 2024-06-04T05:19:29.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.799+0000 7f3e42a6bb80 1 Processor -- start 2024-06-04T05:19:29.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.799+0000 7f3e42a6bb80 1 -- start start 2024-06-04T05:19:29.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.799+0000 7f3e42a6bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd978e0 0x55c84dda2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.799+0000 7f3e42a6bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd98290 0x55c84ddb8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.799+0000 7f3e42a6bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd9bfe0 0x55c84ddb8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.799+0000 7f3e42a6bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301b0160 con 0x55c84dd98290 2024-06-04T05:19:29.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.799+0000 7f3e42a6bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301b1d00 con 0x55c84dd978e0 2024-06-04T05:19:29.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.799+0000 7f3e42a6bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301b0890 con 0x55c84dd9bfe0 2024-06-04T05:19:29.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.800+0000 7f3e35771700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd98290 0x55c84ddb8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.800+0000 7f3e35771700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd98290 0x55c84ddb8890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60774/0 (socket says 172.21.15.70:60774) 2024-06-04T05:19:29.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.800+0000 7f3e35771700 1 -- 172.21.15.70:0/450591658 learned_addr learned my addr 172.21.15.70:0/450591658 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:29.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.800+0000 7f3e35f72700 1 --2- 172.21.15.70:0/450591658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd978e0 0x55c84dda2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.800+0000 7f3e36773700 1 --2- 172.21.15.70:0/450591658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd9bfe0 0x55c84ddb8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.800+0000 7f3e35771700 1 -- 172.21.15.70:0/450591658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd9bfe0 msgr2=0x55c84ddb8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:29.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.800+0000 7f3e35771700 1 --2- 172.21.15.70:0/450591658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd9bfe0 0x55c84ddb8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.800+0000 7f3e35771700 1 -- 172.21.15.70:0/450591658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd978e0 msgr2=0x55c84dda2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:29.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.800+0000 7f3e35771700 1 --2- 172.21.15.70:0/450591658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd978e0 0x55c84dda2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.800+0000 7f3e35771700 1 -- 172.21.15.70:0/450591658 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c84dda0fd0 con 0x55c84dd98290 2024-06-04T05:19:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.800+0000 7f3e35f72700 1 --2- 172.21.15.70:0/450591658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd978e0 0x55c84dda2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:19:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.800+0000 7f3e35771700 1 --2- 172.21.15.70:0/450591658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd98290 0x55c84ddb8890 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f3e20009990 tx=0x7f3e2001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221bc5ad922f92f0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.800+0000 7f3e1b7fe700 1 -- 172.21.15.70:0/450591658 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e200039c0 con 0x55c84dd98290 2024-06-04T05:19:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.800+0000 7f3e1b7fe700 1 -- 172.21.15.70:0/450591658 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e2000b7d0 con 0x55c84dd98290 2024-06-04T05:19:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.800+0000 7f3e1b7fe700 1 -- 172.21.15.70:0/450591658 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2000f6c0 con 0x55c84dd98290 2024-06-04T05:19:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.801+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/450591658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd98290 msgr2=0x55c84ddb8890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.801+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/450591658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd98290 0x55c84ddb8890 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f3e20009990 tx=0x7f3e2001a040 comp rx=0 tx=0).stop 2024-06-04T05:19:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.801+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/450591658 shutdown_connections 2024-06-04T05:19:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.801+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/450591658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd9bfe0 0x55c84ddb8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.801+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/450591658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd978e0 0x55c84dda2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.801+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/450591658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd98290 0x55c84ddb8890 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.801+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/450591658 >> 172.21.15.70:0/450591658 conn(0x55c84dcc5100 msgr2=0x55c84dda1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:29.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.801+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/450591658 shutdown_connections 2024-06-04T05:19:29.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.801+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/450591658 wait complete. 2024-06-04T05:19:29.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.801+0000 7f3e42a6bb80 1 -- start start 2024-06-04T05:19:29.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.801+0000 7f3e42a6bb80 1 Processor -- start 2024-06-04T05:19:29.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e42a6bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd978e0 0x55c84dda9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e42a6bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd98290 0x55c84dda7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e42a6bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd9bfe0 0x55c84dda8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e42a6bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301b1630 con 0x55c84dd98290 2024-06-04T05:19:29.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e42a6bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301b0f60 con 0x55c84dd978e0 2024-06-04T05:19:29.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e42a6bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301ae120 con 0x55c84dd9bfe0 2024-06-04T05:19:29.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e42a6bb80 10 client.0 _ll_register_callbacks cb 0x55c84dc819a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:29.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e35771700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd98290 0x55c84dda7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e35771700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd98290 0x55c84dda7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60776/0 (socket says 172.21.15.70:60776) 2024-06-04T05:19:29.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e35771700 1 -- 172.21.15.70:0/2230642431 learned_addr learned my addr 172.21.15.70:0/2230642431 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:29.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e35f72700 1 --2- 172.21.15.70:0/2230642431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd978e0 0x55c84dda9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e36773700 1 --2- 172.21.15.70:0/2230642431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd9bfe0 0x55c84dda8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e35771700 1 -- 172.21.15.70:0/2230642431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd9bfe0 msgr2=0x55c84dda8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:29.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e35771700 1 --2- 172.21.15.70:0/2230642431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd9bfe0 0x55c84dda8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e35771700 1 -- 172.21.15.70:0/2230642431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd978e0 msgr2=0x55c84dda9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:29.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e35771700 1 --2- 172.21.15.70:0/2230642431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd978e0 0x55c84dda9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e35771700 1 -- 172.21.15.70:0/2230642431 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c84dda8ba0 con 0x55c84dd98290 2024-06-04T05:19:29.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e35771700 1 --2- 172.21.15.70:0/2230642431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd98290 0x55c84dda7e20 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f3e200098e0 tx=0x7f3e20005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb97f02599d1fcaa server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:29.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.802+0000 7f3e197fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:29.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.803+0000 7f3e197fa700 1 -- 172.21.15.70:0/2230642431 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e200037c0 con 0x55c84dd98290 2024-06-04T05:19:29.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.803+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/2230642431 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e2000b7d0 con 0x55c84dd98290 2024-06-04T05:19:29.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.803+0000 7f3e197fa700 1 -- 172.21.15.70:0/2230642431 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f3e20003960 con 0x55c84dd98290 2024-06-04T05:19:29.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.803+0000 7f3e197fa700 1 -- 172.21.15.70:0/2230642431 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f3e20003b60 con 0x55c84dd98290 2024-06-04T05:19:29.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.803+0000 7f3e42a6bb80 10 client.21183 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:29.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.803+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/2230642431 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55c84de1c6c0 con 0x55c84dd98290 2024-06-04T05:19:29.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.803+0000 7f3e197fa700 1 -- 172.21.15.70:0/2230642431 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e2000b7d0 con 0x55c84dd98290 2024-06-04T05:19:29.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.803+0000 7f3e197fa700 1 -- 172.21.15.70:0/2230642431 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f3e20041070 con 0x55c84dd98290 2024-06-04T05:19:29.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.803+0000 7f3e42a6bb80 10 client.21183 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:29.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.803+0000 7f3e42a6bb80 10 client.21183 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:19:29.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.803+0000 7f3e42a6bb80 20 client.21183 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:29.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.803+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/2230642431 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c84de1ccc0 0x55c84de1f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:29.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.803+0000 7f3e42a6bb80 4 client.21183 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:19:29.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.803+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/2230642431 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c84dcf8c60 con 0x55c84de1ccc0 2024-06-04T05:19:29.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.803+0000 7f3e35f72700 1 --2- 172.21.15.70:0/2230642431 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c84de1ccc0 0x55c84de1f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:29.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.804+0000 7f3e197fa700 10 client.21183 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:29.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.804+0000 7f3e35f72700 1 --2- 172.21.15.70:0/2230642431 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c84de1ccc0 0x55c84de1f0e0 crc :-1 s=READY pgs=472 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afcc96f40265400c server_cookie=19e3e04097a0305a in_seq=0 out_seq=0 2024-06-04T05:19:29.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.805+0000 7f3e197fa700 1 -- 172.21.15.70:0/2230642431 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c84dcf8c60 con 0x55c84de1ccc0 2024-06-04T05:19:29.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.805+0000 7f3e197fa700 10 client.21183 handle_command_reply: tid=0 2024-06-04T05:19:29.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.805+0000 7f3e42a6bb80 1 client.21183 shutdown 2024-06-04T05:19:29.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.805+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/2230642431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd98290 msgr2=0x55c84dda7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:29.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.805+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/2230642431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd98290 0x55c84dda7e20 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f3e200098e0 tx=0x7f3e20005700 comp rx=0 tx=0).stop 2024-06-04T05:19:29.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.805+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/2230642431 shutdown_connections 2024-06-04T05:19:29.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.805+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/2230642431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c84dd9bfe0 0x55c84dda8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.805+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/2230642431 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c84de1ccc0 0x55c84de1f0e0 crc :-1 s=READY pgs=472 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.805+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/2230642431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c84dd978e0 0x55c84dda9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.805+0000 7f3e42a6bb80 1 --2- 172.21.15.70:0/2230642431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c84dd98290 0x55c84dda7e20 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.805+0000 7f3e197fa700 0 client.21183 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:29.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.805+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/2230642431 >> 172.21.15.70:0/2230642431 conn(0x55c84dcc5100 msgr2=0x55c84dda0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:29.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.805+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/2230642431 shutdown_connections 2024-06-04T05:19:29.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.805+0000 7f3e42a6bb80 1 -- 172.21.15.70:0/2230642431 wait complete. 2024-06-04T05:19:29.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.805+0000 7f3e42a6bb80 20 client.21183 trim_cache size 0 max 16384 2024-06-04T05:19:29.813 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:19:29.813 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:19:29.813 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "a7d31931-3769-409b-a19a-5a4856589aaa", 2024-06-04T05:19:29.813 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:19:29.813 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:19:29.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.806+0000 7f3e381d6700 1 -- 172.21.15.70:0/193470669 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e1c0b29f0 msgr2=0x7f3e1c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:29.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.806+0000 7f3e381d6700 1 --2- 172.21.15.70:0/193470669 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e1c0b29f0 0x7f3e1c0b4eb0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c0099d0 tx=0x7f3e2c008040 comp rx=0 tx=0).stop 2024-06-04T05:19:29.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.806+0000 7f3e381d6700 1 -- 172.21.15.70:0/193470669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e30108520 msgr2=0x7f3e301a0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:29.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.806+0000 7f3e381d6700 1 --2- 172.21.15.70:0/193470669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e30108520 0x7f3e301a0760 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800b970 tx=0x7f3e2800bd30 comp rx=0 tx=0).stop 2024-06-04T05:19:29.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.806+0000 7f3e381d6700 1 -- 172.21.15.70:0/193470669 shutdown_connections 2024-06-04T05:19:29.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.806+0000 7f3e381d6700 1 --2- 172.21.15.70:0/193470669 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3e1c0b29f0 0x7f3e1c0b4eb0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.806+0000 7f3e381d6700 1 --2- 172.21.15.70:0/193470669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e30108520 0x7f3e301a0760 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.806+0000 7f3e381d6700 1 --2- 172.21.15.70:0/193470669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e30102440 0x7f3e3019bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.806+0000 7f3e381d6700 1 --2- 172.21.15.70:0/193470669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e30102e00 0x7f3e3019c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:29.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.806+0000 7f3e381d6700 1 -- 172.21.15.70:0/193470669 >> 172.21.15.70:0/193470669 conn(0x7f3e300793f0 msgr2=0x7f3e3006b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:29.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.806+0000 7f3e381d6700 1 -- 172.21.15.70:0/193470669 shutdown_connections 2024-06-04T05:19:29.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:29.806+0000 7f3e381d6700 1 -- 172.21.15.70:0/193470669 wait complete. 2024-06-04T05:19:29.828 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.67 MB/sec execute 381 sec latency 7206.230 ms 2024-06-04T05:19:30.186 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:19:29 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:19:29.886+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000215 [...25,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL/ auth{1=1} fragtree_t(*^3) v54992 ap=4 f(v1480 m2024-06-04T05:18:41.288937+0000 11=11+0) n(v85 rc2024-06-04T05:18:41.288937+0000 b2861056 12=11+1) old_inodes=6 (inest sync->mix g=1) (ipolicy sync r=1) (iquiesce lock x=1 by request(mds.0:1920 nref=3)) caps={15231=pAsLsXsFs/-@42261} | dirtyscattered=0 request=1 lock=2 importing=0 dirfrag=8 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 0x55d6deff1b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000215:[<0x10000000009/COREL v50>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x10000000215:[<0x10000000009/COREL v54992>,<0x10000000008/~dmtmp v36064>,<0x10000000007/client0 v3256>,<0x10000000006/clients v2704>,<0x10000000005/tmp v1974>,<0x10000000003/client.0 v1920>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v1909>,<0x10000000001/sv_0 v1875>,<0x10000000000/qa v1850>,<0x1/volumes v1795>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:18:41.262513+0000 12=12+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:18:41.288937+0000 b2861056 12=11+1)","memory_value.dirstat":"f(v1480 m2024-06-04T05:18:41.288937+0000 11=11+0)","memory_value.rstat":"n(v85 rc2024-06-04T05:18:41.288937+0000 b2861056 12=11+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:19:30.186 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:19:29 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:19:29.894+0000 7f03b4150700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000020f [...27,head] /volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ auth{2=1,4=1} fragtree_t(*^2 00*^1 01*^1 10*^1) v55448 ap=5 f(v1952 m2024-06-04T05:19:12.967566+0000 7=7+0) n(v87 rc2024-06-04T05:19:12.967566+0000 b1462582 8=7+1) old_inodes=6 (inest sync->mix g=2,4) (ipolicy sync r=1) (iquiesce lock x=1 by request(mds.0:1927 nref=3)) caps={15231=pAsLsXsFs/-@40993} | dirtyscattered=0 request=1 lock=2 importing=0 dirfrag=7 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 0x55d6de801700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000020f:[<0x10000000009/PWRPNT v234>,<0x10000000008/~dmtmp v30>,<0x10000000007/client0 v14>,<0x10000000006/clients v4>,<0x10000000005/tmp v4>,<0x10000000003/client.0 v6>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v22>,<0x10000000001/sv_0 v46>,<0x10000000000/qa v37>,<0x1/volumes v38>]//[]","memoryvalue":"(2)0x1000000020f:[<0x10000000009/PWRPNT v55448>,<0x10000000008/~dmtmp v36064>,<0x10000000007/client0 v3256>,<0x10000000006/clients v2704>,<0x10000000005/tmp v1974>,<0x10000000003/client.0 v1920>,<0x10000000002/0cd839e3-1220-4eb9-8759-e3d92f3bc84b v1909>,<0x10000000001/sv_0 v1875>,<0x10000000000/qa v1850>,<0x1/volumes v1795>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-04T05:19:12.900825+0000 8=8+0)","ondisk_value.rstat":"n(v0 rc2024-06-04T05:19:12.967566+0000 b1462582 8=7+1)","memory_value.dirstat":"f(v1952 m2024-06-04T05:19:12.967566+0000 7=7+0)","memory_value.rstat":"n(v87 rc2024-06-04T05:19:12.967566+0000 b1462582 8=7+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-06-04T05:19:30.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:30 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:19:30.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:19:30.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:19:30.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:19:30.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:19:30.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:19:30.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:19:30.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:30 smithi070 ceph-mon[32631]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.i log and `damage ls` output for details 2024-06-04T05:19:30.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:30 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:19:30.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:30 smithi070 ceph-mon[32631]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT) see mds.i log and `damage ls` output for details 2024-06-04T05:19:30.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:30 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:30 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:19:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:19:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:19:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:19:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:19:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:19:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:19:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:30 smithi112 ceph-mon[36453]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.i log and `damage ls` output for details 2024-06-04T05:19:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:30 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:19:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:30 smithi112 ceph-mon[36453]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT) see mds.i log and `damage ls` output for details 2024-06-04T05:19:30.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:30 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:30 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:19:30.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:19:30.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:19:30.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:19:30.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:19:30.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:19:30.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:19:30.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:30 smithi098 ceph-mon[35569]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.i log and `damage ls` output for details 2024-06-04T05:19:30.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:30 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:19:30.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:30 smithi098 ceph-mon[35569]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/PWRPNT) see mds.i log and `damage ls` output for details 2024-06-04T05:19:30.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:30 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:30.828 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.65 MB/sec execute 382 sec latency 8206.258 ms 2024-06-04T05:19:30.926 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:19:31.172 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:19:31.553 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:31 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:19:31.553 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:31 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:31.553 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:31 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:31.553 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:31 smithi070 ceph-mon[32631]: pgmap v703: 97 pgs: 97 active+clean; 921 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 190 op/s 2024-06-04T05:19:31.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:31 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:19:31.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:31 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:31.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:31 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:31.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:31 smithi112 ceph-mon[36453]: pgmap v703: 97 pgs: 97 active+clean; 921 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 190 op/s 2024-06-04T05:19:31.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:31 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:19:31.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:31 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:31.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:31 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:19:31.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:31 smithi098 ceph-mon[35569]: pgmap v703: 97 pgs: 97 active+clean; 921 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 190 op/s 2024-06-04T05:19:31.828 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.63 MB/sec execute 383 sec latency 9206.351 ms 2024-06-04T05:19:32.828 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.62 MB/sec execute 384 sec latency 10206.381 ms 2024-06-04T05:19:32.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.931+0000 7f0a6b8f7700 1 -- 172.21.15.70:0/2704099947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a64102e20 msgr2=0x7f0a6410f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:32.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.931+0000 7f0a6b8f7700 1 --2- 172.21.15.70:0/2704099947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a64102e20 0x7f0a6410f300 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c00b0a0 tx=0x7f0a5c00b3b0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.931+0000 7f0a6b8f7700 1 -- 172.21.15.70:0/2704099947 shutdown_connections 2024-06-04T05:19:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.931+0000 7f0a6b8f7700 1 --2- 172.21.15.70:0/2704099947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a64102e20 0x7f0a6410f300 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.931+0000 7f0a6b8f7700 1 --2- 172.21.15.70:0/2704099947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a641084c0 0x7f0a641088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.931+0000 7f0a6b8f7700 1 --2- 172.21.15.70:0/2704099947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a64102480 0x7f0a641028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.931+0000 7f0a6b8f7700 1 -- 172.21.15.70:0/2704099947 >> 172.21.15.70:0/2704099947 conn(0x7f0a64079370 msgr2=0x7f0a64079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.931+0000 7f0a6b8f7700 1 -- 172.21.15.70:0/2704099947 shutdown_connections 2024-06-04T05:19:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.931+0000 7f0a6b8f7700 1 -- 172.21.15.70:0/2704099947 wait complete. 2024-06-04T05:19:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.931+0000 7f0a6b8f7700 1 Processor -- start 2024-06-04T05:19:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.931+0000 7f0a6b8f7700 1 -- start start 2024-06-04T05:19:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a6b8f7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a64102480 0x7f0a6419f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a6b8f7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a64102e20 0x7f0a6419fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a6b8f7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a641084c0 0x7f0a641998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a6b8f7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0a64112490 con 0x7f0a641084c0 2024-06-04T05:19:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a6b8f7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a64112310 con 0x7f0a64102480 2024-06-04T05:19:32.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a6b8f7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0a64112610 con 0x7f0a64102e20 2024-06-04T05:19:32.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a69e94700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a641084c0 0x7f0a641998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a69e94700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a641084c0 0x7f0a641998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60796/0 (socket says 172.21.15.70:60796) 2024-06-04T05:19:32.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a69e94700 1 -- 172.21.15.70:0/3555512874 learned_addr learned my addr 172.21.15.70:0/3555512874 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:32.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a68e92700 1 --2- 172.21.15.70:0/3555512874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a64102e20 0x7f0a6419fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a69693700 1 --2- 172.21.15.70:0/3555512874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a64102480 0x7f0a6419f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a69e94700 1 -- 172.21.15.70:0/3555512874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a64102e20 msgr2=0x7f0a6419fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:32.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a69e94700 1 --2- 172.21.15.70:0/3555512874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a64102e20 0x7f0a6419fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a69e94700 1 -- 172.21.15.70:0/3555512874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a64102480 msgr2=0x7f0a6419f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:32.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a69e94700 1 --2- 172.21.15.70:0/3555512874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a64102480 0x7f0a6419f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a69e94700 1 -- 172.21.15.70:0/3555512874 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6419a020 con 0x7f0a641084c0 2024-06-04T05:19:32.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a69693700 1 --2- 172.21.15.70:0/3555512874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a64102480 0x7f0a6419f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:19:32.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a68e92700 1 --2- 172.21.15.70:0/3555512874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a64102e20 0x7f0a6419fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:19:32.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a69e94700 1 --2- 172.21.15.70:0/3555512874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a641084c0 0x7f0a641998c0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c0062a0 tx=0x7f0a5c004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a5a7fc700 1 -- 172.21.15.70:0/3555512874 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c021070 con 0x7f0a641084c0 2024-06-04T05:19:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a5a7fc700 1 -- 172.21.15.70:0/3555512874 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a5c007d00 con 0x7f0a641084c0 2024-06-04T05:19:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a6b8f7700 1 -- 172.21.15.70:0/3555512874 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a6419a2b0 con 0x7f0a641084c0 2024-06-04T05:19:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a5a7fc700 1 -- 172.21.15.70:0/3555512874 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c005280 con 0x7f0a641084c0 2024-06-04T05:19:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.932+0000 7f0a6b8f7700 1 -- 172.21.15.70:0/3555512874 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a641a65d0 con 0x7f0a641084c0 2024-06-04T05:19:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.935+0000 7f0a5a7fc700 1 -- 172.21.15.70:0/3555512874 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f0a5c024070 con 0x7f0a641084c0 2024-06-04T05:19:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.936+0000 7f0a5a7fc700 1 --2- 172.21.15.70:0/3555512874 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0a500b29f0 0x7f0a500b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.936+0000 7f0a69693700 1 --2- 172.21.15.70:0/3555512874 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0a500b29f0 0x7f0a500b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.936+0000 7f0a5a7fc700 1 -- 172.21.15.70:0/3555512874 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f0a5c00f020 con 0x7f0a641084c0 2024-06-04T05:19:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.937+0000 7f0a69693700 1 --2- 172.21.15.70:0/3555512874 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0a500b29f0 0x7f0a500b4eb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0a60005fd0 tx=0x7f0a60005d60 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.937+0000 7f0a7618cb80 1 Processor -- start 2024-06-04T05:19:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.937+0000 7f0a7618cb80 1 -- start start 2024-06-04T05:19:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.937+0000 7f0a7618cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0e0320 0x55bdbe0e0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.937+0000 7f0a7618cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0dad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.937+0000 7f0a7618cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0db260 0x55bdbe0e7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.937+0000 7f0a7618cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bdbe038d70 con 0x55bdbe0da8a0 2024-06-04T05:19:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.937+0000 7f0a7618cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bdbe028910 con 0x55bdbe0e0320 2024-06-04T05:19:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.937+0000 7f0a7618cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bdbe02c950 con 0x55bdbe0db260 2024-06-04T05:19:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.937+0000 7f0a68e92700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0dad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.937+0000 7f0a68e92700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0dad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60808/0 (socket says 172.21.15.70:60808) 2024-06-04T05:19:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.937+0000 7f0a68e92700 1 -- 172.21.15.70:0/1291436223 learned_addr learned my addr 172.21.15.70:0/1291436223 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.937+0000 7f0a69693700 1 --2- 172.21.15.70:0/1291436223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0e0320 0x55bdbe0e0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.938+0000 7f0a69e94700 1 --2- 172.21.15.70:0/1291436223 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0db260 0x55bdbe0e7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.938+0000 7f0a68e92700 1 -- 172.21.15.70:0/1291436223 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0db260 msgr2=0x55bdbe0e7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:32.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.938+0000 7f0a68e92700 1 --2- 172.21.15.70:0/1291436223 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0db260 0x55bdbe0e7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.938+0000 7f0a68e92700 1 -- 172.21.15.70:0/1291436223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0e0320 msgr2=0x55bdbe0e0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:32.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.938+0000 7f0a68e92700 1 --2- 172.21.15.70:0/1291436223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0e0320 0x55bdbe0e0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.938+0000 7f0a68e92700 1 -- 172.21.15.70:0/1291436223 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdbdfd1fb0 con 0x55bdbe0da8a0 2024-06-04T05:19:32.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.938+0000 7f0a69693700 1 --2- 172.21.15.70:0/1291436223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0e0320 0x55bdbe0e0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:19:32.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.938+0000 7f0a68e92700 1 --2- 172.21.15.70:0/1291436223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0dad20 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f0a54008cd0 tx=0x7f0a5400e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=113ed21321a8c5d5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:32.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.938+0000 7f0a4f7fe700 1 -- 172.21.15.70:0/1291436223 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a54014070 con 0x55bdbe0da8a0 2024-06-04T05:19:32.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.938+0000 7f0a4f7fe700 1 -- 172.21.15.70:0/1291436223 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a5400ee90 con 0x55bdbe0da8a0 2024-06-04T05:19:32.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.938+0000 7f0a4f7fe700 1 -- 172.21.15.70:0/1291436223 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a54010430 con 0x55bdbe0da8a0 2024-06-04T05:19:32.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.939+0000 7f0a7618cb80 1 -- 172.21.15.70:0/1291436223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0da8a0 msgr2=0x55bdbe0dad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:32.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.939+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/1291436223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0dad20 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f0a54008cd0 tx=0x7f0a5400e350 comp rx=0 tx=0).stop 2024-06-04T05:19:32.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.939+0000 7f0a7618cb80 1 -- 172.21.15.70:0/1291436223 shutdown_connections 2024-06-04T05:19:32.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.939+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/1291436223 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0db260 0x55bdbe0e7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.939+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/1291436223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0e0320 0x55bdbe0e0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.939+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/1291436223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0dad20 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.939+0000 7f0a7618cb80 1 -- 172.21.15.70:0/1291436223 >> 172.21.15.70:0/1291436223 conn(0x55bdbe006240 msgr2=0x55bdbe0d53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:32.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.939+0000 7f0a7618cb80 1 -- 172.21.15.70:0/1291436223 shutdown_connections 2024-06-04T05:19:32.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.939+0000 7f0a7618cb80 1 -- 172.21.15.70:0/1291436223 wait complete. 2024-06-04T05:19:32.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.939+0000 7f0a7618cb80 1 -- start start 2024-06-04T05:19:32.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.939+0000 7f0a7618cb80 1 Processor -- start 2024-06-04T05:19:32.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a7618cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0e6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a7618cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0db260 0x55bdbe0e3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a7618cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0e0320 0x55bdbe0e4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a7618cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bdbdf2efa0 con 0x55bdbe0e0320 2024-06-04T05:19:32.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a7618cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bdbdf90da0 con 0x55bdbe0db260 2024-06-04T05:19:32.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a7618cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bdbe0e76e0 con 0x55bdbe0da8a0 2024-06-04T05:19:32.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a7618cb80 10 client.0 _ll_register_callbacks cb 0x55bdbdfc3950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:32.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a69e94700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0e0320 0x55bdbe0e4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a69e94700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0e0320 0x55bdbe0e4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60820/0 (socket says 172.21.15.70:60820) 2024-06-04T05:19:32.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a69e94700 1 -- 172.21.15.70:0/3904551683 learned_addr learned my addr 172.21.15.70:0/3904551683 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:32.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a68e92700 1 --2- 172.21.15.70:0/3904551683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0db260 0x55bdbe0e3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a69e94700 1 -- 172.21.15.70:0/3904551683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0da8a0 msgr2=0x55bdbe0e6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:32.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a69693700 1 --2- 172.21.15.70:0/3904551683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0e6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a69e94700 1 --2- 172.21.15.70:0/3904551683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0e6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a69e94700 1 -- 172.21.15.70:0/3904551683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0db260 msgr2=0x55bdbe0e3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:32.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a69e94700 1 --2- 172.21.15.70:0/3904551683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0db260 0x55bdbe0e3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a69e94700 1 -- 172.21.15.70:0/3904551683 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdbdf788d0 con 0x55bdbe0e0320 2024-06-04T05:19:32.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a69693700 1 --2- 172.21.15.70:0/3904551683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0e6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:19:32.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a69e94700 1 --2- 172.21.15.70:0/3904551683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0e0320 0x55bdbe0e4510 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c0ab0c0 tx=0x7f0a5c0aab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf93505322e4322 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:32.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a4cff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:32.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a4cff9700 1 -- 172.21.15.70:0/3904551683 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c0c5060 con 0x55bdbe0e0320 2024-06-04T05:19:32.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.940+0000 7f0a4cff9700 1 -- 172.21.15.70:0/3904551683 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f0a5c0af170 con 0x55bdbe0e0320 2024-06-04T05:19:32.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.941+0000 7f0a7618cb80 1 -- 172.21.15.70:0/3904551683 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a5400ee90 con 0x55bdbe0e0320 2024-06-04T05:19:32.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.941+0000 7f0a4cff9700 1 -- 172.21.15.70:0/3904551683 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f0a5c01d660 con 0x55bdbe0e0320 2024-06-04T05:19:32.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.941+0000 7f0a7618cb80 10 client.21195 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:32.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.941+0000 7f0a7618cb80 1 -- 172.21.15.70:0/3904551683 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55bdbdfe2760 con 0x55bdbe0e0320 2024-06-04T05:19:32.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.941+0000 7f0a4cff9700 1 -- 172.21.15.70:0/3904551683 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a5400ee90 con 0x55bdbe0e0320 2024-06-04T05:19:32.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.941+0000 7f0a4cff9700 1 -- 172.21.15.70:0/3904551683 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f0a5c06cce0 con 0x55bdbe0e0320 2024-06-04T05:19:32.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.941+0000 7f0a7618cb80 10 client.21195 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:32.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.941+0000 7f0a7618cb80 10 client.21195 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:19:32.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.941+0000 7f0a7618cb80 20 client.21195 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:32.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.941+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/3904551683 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bdbe0f40e0 0x55bdbe0f6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.941+0000 7f0a7618cb80 4 client.21195 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:19:32.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.941+0000 7f0a7618cb80 1 -- 172.21.15.70:0/3904551683 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdbe03acc0 con 0x55bdbe0f40e0 2024-06-04T05:19:32.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.941+0000 7f0a69693700 1 --2- 172.21.15.70:0/3904551683 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bdbe0f40e0 0x55bdbe0f6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.942+0000 7f0a4cff9700 10 client.21195 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:32.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.942+0000 7f0a69693700 1 --2- 172.21.15.70:0/3904551683 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bdbe0f40e0 0x55bdbe0f6540 crc :-1 s=READY pgs=473 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c9ad3061ebdc2c9 server_cookie=edbd209c80625c9 in_seq=0 out_seq=0 2024-06-04T05:19:32.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.942+0000 7f0a4cff9700 1 -- 172.21.15.70:0/3904551683 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bdbe03acc0 con 0x55bdbe0f40e0 2024-06-04T05:19:32.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.942+0000 7f0a4cff9700 10 client.21195 handle_command_reply: tid=0 2024-06-04T05:19:32.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.943+0000 7f0a7618cb80 1 client.21195 shutdown 2024-06-04T05:19:32.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.943+0000 7f0a7618cb80 1 -- 172.21.15.70:0/3904551683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0e0320 msgr2=0x55bdbe0e4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:32.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.943+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/3904551683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0e0320 0x55bdbe0e4510 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c0ab0c0 tx=0x7f0a5c0aab90 comp rx=0 tx=0).stop 2024-06-04T05:19:32.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.943+0000 7f0a7618cb80 1 -- 172.21.15.70:0/3904551683 shutdown_connections 2024-06-04T05:19:32.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.943+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/3904551683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0e0320 0x55bdbe0e4510 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.943+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/3904551683 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0e6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.943+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/3904551683 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bdbe0f40e0 0x55bdbe0f6540 crc :-1 s=READY pgs=473 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.943+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/3904551683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0db260 0x55bdbe0e3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.943+0000 7f0a7618cb80 1 -- 172.21.15.70:0/3904551683 >> 172.21.15.70:0/3904551683 conn(0x55bdbe006240 msgr2=0x55bdbe0d4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:32.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.943+0000 7f0a4cff9700 0 client.21195 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:32.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.943+0000 7f0a7618cb80 1 -- 172.21.15.70:0/3904551683 shutdown_connections 2024-06-04T05:19:32.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.943+0000 7f0a7618cb80 1 -- 172.21.15.70:0/3904551683 wait complete. 2024-06-04T05:19:32.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.943+0000 7f0a7618cb80 20 client.21195 trim_cache size 0 max 16384 2024-06-04T05:19:32.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.963+0000 7f0a7618cb80 1 Processor -- start 2024-06-04T05:19:32.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.963+0000 7f0a7618cb80 1 -- start start 2024-06-04T05:19:32.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a7618cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0df980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a7618cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0db260 0x55bdbe0d5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a7618cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0e0320 0x55bdbe0e2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a7618cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641add40 con 0x55bdbe0db260 2024-06-04T05:19:32.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a7618cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641af8e0 con 0x55bdbe0da8a0 2024-06-04T05:19:32.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a7618cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641ae470 con 0x55bdbe0e0320 2024-06-04T05:19:32.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a68e92700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0db260 0x55bdbe0d5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a68e92700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0db260 0x55bdbe0d5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60826/0 (socket says 172.21.15.70:60826) 2024-06-04T05:19:32.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a68e92700 1 -- 172.21.15.70:0/844198115 learned_addr learned my addr 172.21.15.70:0/844198115 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:32.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a68e92700 1 -- 172.21.15.70:0/844198115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0e0320 msgr2=0x55bdbe0e2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:32.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a69e94700 1 --2- 172.21.15.70:0/844198115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0e0320 0x55bdbe0e2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a69693700 1 --2- 172.21.15.70:0/844198115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0df980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a68e92700 1 --2- 172.21.15.70:0/844198115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0e0320 0x55bdbe0e2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a68e92700 1 -- 172.21.15.70:0/844198115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0da8a0 msgr2=0x55bdbe0df980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:32.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a68e92700 1 --2- 172.21.15.70:0/844198115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0df980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a68e92700 1 -- 172.21.15.70:0/844198115 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdbe0e0a70 con 0x55bdbe0db260 2024-06-04T05:19:32.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a69e94700 1 --2- 172.21.15.70:0/844198115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0e0320 0x55bdbe0e2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:19:32.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a68e92700 1 --2- 172.21.15.70:0/844198115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0db260 0x55bdbe0d5270 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400e750 tx=0x7f0a5400eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d3c085e0ba3dafe server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:32.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a4effd700 1 -- 172.21.15.70:0/844198115 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a54014050 con 0x55bdbe0db260 2024-06-04T05:19:32.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a4effd700 1 -- 172.21.15.70:0/844198115 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a54003680 con 0x55bdbe0db260 2024-06-04T05:19:32.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.964+0000 7f0a4effd700 1 -- 172.21.15.70:0/844198115 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a54010670 con 0x55bdbe0db260 2024-06-04T05:19:32.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.965+0000 7f0a7618cb80 1 -- 172.21.15.70:0/844198115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0db260 msgr2=0x55bdbe0d5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:32.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.965+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/844198115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0db260 0x55bdbe0d5270 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400e750 tx=0x7f0a5400eed0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.965+0000 7f0a7618cb80 1 -- 172.21.15.70:0/844198115 shutdown_connections 2024-06-04T05:19:32.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.965+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/844198115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0e0320 0x55bdbe0e2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.965+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/844198115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0df980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.965+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/844198115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0db260 0x55bdbe0d5270 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.965+0000 7f0a7618cb80 1 -- 172.21.15.70:0/844198115 >> 172.21.15.70:0/844198115 conn(0x55bdbe006240 msgr2=0x55bdbe0e28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:32.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.965+0000 7f0a7618cb80 1 -- 172.21.15.70:0/844198115 shutdown_connections 2024-06-04T05:19:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.965+0000 7f0a7618cb80 1 -- 172.21.15.70:0/844198115 wait complete. 2024-06-04T05:19:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.965+0000 7f0a7618cb80 1 -- start start 2024-06-04T05:19:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a7618cb80 1 Processor -- start 2024-06-04T05:19:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a7618cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0ed020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a7618cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0db260 0x55bdbe0fb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a7618cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0e0320 0x55bdbe0fbb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a7618cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641af210 con 0x55bdbe0da8a0 2024-06-04T05:19:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a7618cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641aeb40 con 0x55bdbe0e0320 2024-06-04T05:19:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a7618cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641abd00 con 0x55bdbe0db260 2024-06-04T05:19:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a7618cb80 10 client.0 _ll_register_callbacks cb 0x55bdbdfc3950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a69693700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0ed020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a69693700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0ed020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60842/0 (socket says 172.21.15.70:60842) 2024-06-04T05:19:32.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a69693700 1 -- 172.21.15.70:0/3990337130 learned_addr learned my addr 172.21.15.70:0/3990337130 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:32.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a69e94700 1 --2- 172.21.15.70:0/3990337130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0e0320 0x55bdbe0fbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a68e92700 1 --2- 172.21.15.70:0/3990337130 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0db260 0x55bdbe0fb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a69693700 1 -- 172.21.15.70:0/3990337130 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0db260 msgr2=0x55bdbe0fb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:32.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a69693700 1 --2- 172.21.15.70:0/3990337130 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0db260 0x55bdbe0fb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a69693700 1 -- 172.21.15.70:0/3990337130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0e0320 msgr2=0x55bdbe0fbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:32.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a69693700 1 --2- 172.21.15.70:0/3990337130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0e0320 0x55bdbe0fbb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a69693700 1 -- 172.21.15.70:0/3990337130 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdbe0fc340 con 0x55bdbe0da8a0 2024-06-04T05:19:32.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.966+0000 7f0a69e94700 1 --2- 172.21.15.70:0/3990337130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0e0320 0x55bdbe0fbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:19:32.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.967+0000 7f0a69693700 1 --2- 172.21.15.70:0/3990337130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0ed020 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f0a6419aa00 tx=0x7f0a600047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f09715195b7beecb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:32.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.967+0000 7f0a4cff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:32.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.967+0000 7f0a4cff9700 1 -- 172.21.15.70:0/3990337130 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6001e980 con 0x55bdbe0da8a0 2024-06-04T05:19:32.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.967+0000 7f0a4cff9700 1 -- 172.21.15.70:0/3990337130 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f0a6001eb20 con 0x55bdbe0da8a0 2024-06-04T05:19:32.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.967+0000 7f0a7618cb80 1 -- 172.21.15.70:0/3990337130 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a54003680 con 0x55bdbe0da8a0 2024-06-04T05:19:32.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.967+0000 7f0a4cff9700 1 -- 172.21.15.70:0/3990337130 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f0a6001ed20 con 0x55bdbe0da8a0 2024-06-04T05:19:32.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.967+0000 7f0a7618cb80 10 client.21201 fetch_fsmap learned FSMap version 59 2024-06-04T05:19:32.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.967+0000 7f0a7618cb80 1 -- 172.21.15.70:0/3990337130 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55bdbe15ece0 con 0x55bdbe0da8a0 2024-06-04T05:19:32.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.967+0000 7f0a4cff9700 1 -- 172.21.15.70:0/3990337130 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a54003680 con 0x55bdbe0da8a0 2024-06-04T05:19:32.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.967+0000 7f0a4cff9700 1 -- 172.21.15.70:0/3990337130 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8362+0+0 (secure 0 0 0) 0x7f0a60034070 con 0x55bdbe0da8a0 2024-06-04T05:19:32.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.967+0000 7f0a7618cb80 10 client.21201 fetch_fsmap finished waiting for FSMap version 59 2024-06-04T05:19:32.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.967+0000 7f0a7618cb80 10 client.21201 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:19:32.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.967+0000 7f0a7618cb80 20 client.21201 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:32.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.967+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/3990337130 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bdbe15f3a0 0x55bdbe1617c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:32.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.967+0000 7f0a7618cb80 4 client.21201 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:19:32.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.967+0000 7f0a7618cb80 1 -- 172.21.15.70:0/3990337130 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bdbe0344a0 con 0x55bdbe15f3a0 2024-06-04T05:19:32.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.968+0000 7f0a68e92700 1 --2- 172.21.15.70:0/3990337130 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bdbe15f3a0 0x55bdbe1617c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:32.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.968+0000 7f0a4cff9700 10 client.21201 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:32.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.968+0000 7f0a68e92700 1 --2- 172.21.15.70:0/3990337130 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bdbe15f3a0 0x55bdbe1617c0 crc :-1 s=READY pgs=474 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=847f73c38516d477 server_cookie=2a2fc8c4258d033a in_seq=0 out_seq=0 2024-06-04T05:19:32.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.968+0000 7f0a4cff9700 1 -- 172.21.15.70:0/3990337130 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55bdbe0344a0 con 0x55bdbe15f3a0 2024-06-04T05:19:32.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.968+0000 7f0a4cff9700 10 client.21201 handle_command_reply: tid=0 2024-06-04T05:19:32.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.968+0000 7f0a7618cb80 1 client.21201 shutdown 2024-06-04T05:19:32.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.968+0000 7f0a7618cb80 1 -- 172.21.15.70:0/3990337130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0da8a0 msgr2=0x55bdbe0ed020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:32.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.968+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/3990337130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0ed020 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f0a6419aa00 tx=0x7f0a600047c0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.968+0000 7f0a7618cb80 1 -- 172.21.15.70:0/3990337130 shutdown_connections 2024-06-04T05:19:32.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.968+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/3990337130 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55bdbe15f3a0 0x55bdbe1617c0 crc :-1 s=READY pgs=474 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.968+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/3990337130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdbe0e0320 0x55bdbe0fbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.968+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/3990337130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdbe0da8a0 0x55bdbe0ed020 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.968+0000 7f0a7618cb80 1 --2- 172.21.15.70:0/3990337130 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bdbe0db260 0x55bdbe0fb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.968+0000 7f0a7618cb80 1 -- 172.21.15.70:0/3990337130 >> 172.21.15.70:0/3990337130 conn(0x55bdbe006240 msgr2=0x55bdbe0e3150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:32.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.968+0000 7f0a4cff9700 0 client.21201 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:32.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.969+0000 7f0a7618cb80 1 -- 172.21.15.70:0/3990337130 shutdown_connections 2024-06-04T05:19:32.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.969+0000 7f0a7618cb80 1 -- 172.21.15.70:0/3990337130 wait complete. 2024-06-04T05:19:32.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.969+0000 7f0a7618cb80 20 client.21201 trim_cache size 0 max 16384 2024-06-04T05:19:32.984 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:19:32.984 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:19:32.984 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:19:32.984 INFO:teuthology.orchestra.run.smithi070.stdout: "a7d31931-3769-409b-a19a-5a4856589aaa": { 2024-06-04T05:19:32.984 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:19:32.984 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "a7d31931-3769-409b-a19a-5a4856589aaa", 2024-06-04T05:19:32.984 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:19:32.985 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:19:32.985 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:19:32.985 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:19:32.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.969+0000 7f0a6b8f7700 1 -- 172.21.15.70:0/3555512874 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0a500b29f0 msgr2=0x7f0a500b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:32.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.969+0000 7f0a6b8f7700 1 --2- 172.21.15.70:0/3555512874 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0a500b29f0 0x7f0a500b4eb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0a60005fd0 tx=0x7f0a60005d60 comp rx=0 tx=0).stop 2024-06-04T05:19:32.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.969+0000 7f0a6b8f7700 1 -- 172.21.15.70:0/3555512874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a641084c0 msgr2=0x7f0a641998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:32.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.969+0000 7f0a6b8f7700 1 --2- 172.21.15.70:0/3555512874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a641084c0 0x7f0a641998c0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c0062a0 tx=0x7f0a5c004ed0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.969+0000 7f0a6b8f7700 1 -- 172.21.15.70:0/3555512874 shutdown_connections 2024-06-04T05:19:32.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.969+0000 7f0a6b8f7700 1 --2- 172.21.15.70:0/3555512874 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f0a500b29f0 0x7f0a500b4eb0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.969+0000 7f0a6b8f7700 1 --2- 172.21.15.70:0/3555512874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a641084c0 0x7f0a641998c0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.969+0000 7f0a6b8f7700 1 --2- 172.21.15.70:0/3555512874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a64102480 0x7f0a6419f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.969+0000 7f0a6b8f7700 1 --2- 172.21.15.70:0/3555512874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a64102e20 0x7f0a6419fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:32.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.969+0000 7f0a6b8f7700 1 -- 172.21.15.70:0/3555512874 >> 172.21.15.70:0/3555512874 conn(0x7f0a64079370 msgr2=0x7f0a6406c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:32.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.969+0000 7f0a6b8f7700 1 -- 172.21.15.70:0/3555512874 shutdown_connections 2024-06-04T05:19:32.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:32.969+0000 7f0a6b8f7700 1 -- 172.21.15.70:0/3555512874 wait complete. 2024-06-04T05:19:33.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:33 smithi070 ceph-mon[32631]: pgmap v704: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 261 op/s 2024-06-04T05:19:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:33 smithi112 ceph-mon[36453]: pgmap v704: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 261 op/s 2024-06-04T05:19:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:33 smithi098 ceph-mon[35569]: pgmap v704: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 261 op/s 2024-06-04T05:19:33.828 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.60 MB/sec execute 385 sec latency 11206.412 ms 2024-06-04T05:19:33.840 INFO:tasks.cephfs.filesystem:scrub status for tag:a7d31931-3769-409b-a19a-5a4856589aaa - {'path': '/', 'tag': 'a7d31931-3769-409b-a19a-5a4856589aaa', 'options': 'recursive,force'} 2024-06-04T05:19:34.828 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.58 MB/sec execute 386 sec latency 12206.474 ms 2024-06-04T05:19:35.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:35 smithi070 ceph-mon[32631]: pgmap v705: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 986 KiB/s wr, 136 op/s 2024-06-04T05:19:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:35 smithi112 ceph-mon[36453]: pgmap v705: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 986 KiB/s wr, 136 op/s 2024-06-04T05:19:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:35 smithi098 ceph-mon[35569]: pgmap v705: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 986 KiB/s wr, 136 op/s 2024-06-04T05:19:35.828 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.57 MB/sec execute 387 sec latency 13206.568 ms 2024-06-04T05:19:36.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:36 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:19:36.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:36 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:19:36.828 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.55 MB/sec execute 388 sec latency 14206.675 ms 2024-06-04T05:19:36.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:36 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:19:37.196 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:19:37.450 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:19:37.571 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:37 smithi070 ceph-mon[32631]: pgmap v706: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 992 KiB/s wr, 141 op/s 2024-06-04T05:19:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:37 smithi112 ceph-mon[36453]: pgmap v706: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 992 KiB/s wr, 141 op/s 2024-06-04T05:19:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:37 smithi098 ceph-mon[35569]: pgmap v706: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 992 KiB/s wr, 141 op/s 2024-06-04T05:19:37.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.53 MB/sec execute 389 sec latency 15206.806 ms 2024-06-04T05:19:38.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.51 MB/sec execute 390 sec latency 16206.903 ms 2024-06-04T05:19:39.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.237+0000 7fc0614ce700 1 -- 172.21.15.70:0/1252649100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc05c102e90 msgr2=0x7fc05c10f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:39.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.237+0000 7fc0614ce700 1 --2- 172.21.15.70:0/1252649100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc05c102e90 0x7fc05c10f310 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fc05400b0a0 tx=0x7fc05400b3b0 comp rx=0 tx=0).stop 2024-06-04T05:19:39.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.237+0000 7fc0614ce700 1 -- 172.21.15.70:0/1252649100 shutdown_connections 2024-06-04T05:19:39.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.237+0000 7fc0614ce700 1 --2- 172.21.15.70:0/1252649100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc05c102e90 0x7fc05c10f310 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:39.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.237+0000 7fc0614ce700 1 --2- 172.21.15.70:0/1252649100 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc05c1084d0 0x7fc05c1088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:39.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.237+0000 7fc0614ce700 1 --2- 172.21.15.70:0/1252649100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc05c1024d0 0x7fc05c102950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:39.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.237+0000 7fc0614ce700 1 -- 172.21.15.70:0/1252649100 >> 172.21.15.70:0/1252649100 conn(0x7fc05c078490 msgr2=0x7fc05c0788a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:39.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.237+0000 7fc0614ce700 1 -- 172.21.15.70:0/1252649100 shutdown_connections 2024-06-04T05:19:39.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.237+0000 7fc0614ce700 1 -- 172.21.15.70:0/1252649100 wait complete. 2024-06-04T05:19:39.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.238+0000 7fc0614ce700 1 Processor -- start 2024-06-04T05:19:39.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.238+0000 7fc0614ce700 1 -- start start 2024-06-04T05:19:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.238+0000 7fc0614ce700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc05c1024d0 0x7fc05c19f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.238+0000 7fc0614ce700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc05c102e90 0x7fc05c19fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.238+0000 7fc0614ce700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc05c1084d0 0x7fc05c1999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.238+0000 7fc0614ce700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc05c113610 con 0x7fc05c1084d0 2024-06-04T05:19:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.238+0000 7fc0614ce700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc05c113490 con 0x7fc05c1024d0 2024-06-04T05:19:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.238+0000 7fc0614ce700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc05c113790 con 0x7fc05c102e90 2024-06-04T05:19:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.239+0000 7fc05b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc05c1084d0 0x7fc05c1999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.239+0000 7fc05b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc05c1084d0 0x7fc05c1999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50524/0 (socket says 172.21.15.70:50524) 2024-06-04T05:19:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.239+0000 7fc05b7fe700 1 -- 172.21.15.70:0/2677407831 learned_addr learned my addr 172.21.15.70:0/2677407831 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.239+0000 7fc05affd700 1 --2- 172.21.15.70:0/2677407831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc05c1024d0 0x7fc05c19f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.239+0000 7fc05a7fc700 1 --2- 172.21.15.70:0/2677407831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc05c102e90 0x7fc05c19fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.239+0000 7fc05b7fe700 1 -- 172.21.15.70:0/2677407831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc05c102e90 msgr2=0x7fc05c19fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:39.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.239+0000 7fc05b7fe700 1 --2- 172.21.15.70:0/2677407831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc05c102e90 0x7fc05c19fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:39.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.239+0000 7fc05b7fe700 1 -- 172.21.15.70:0/2677407831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc05c1024d0 msgr2=0x7fc05c19f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:39.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.239+0000 7fc05b7fe700 1 --2- 172.21.15.70:0/2677407831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc05c1024d0 0x7fc05c19f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:39.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.239+0000 7fc05b7fe700 1 -- 172.21.15.70:0/2677407831 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc05c19a110 con 0x7fc05c1084d0 2024-06-04T05:19:39.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.239+0000 7fc05b7fe700 1 --2- 172.21.15.70:0/2677407831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc05c1084d0 0x7fc05c1999b0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7fc0540062a0 tx=0x7fc054004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:39.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.239+0000 7fc043fff700 1 -- 172.21.15.70:0/2677407831 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc054021070 con 0x7fc05c1084d0 2024-06-04T05:19:39.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.239+0000 7fc0614ce700 1 -- 172.21.15.70:0/2677407831 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc05c19a3a0 con 0x7fc05c1084d0 2024-06-04T05:19:39.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.239+0000 7fc043fff700 1 -- 172.21.15.70:0/2677407831 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc054007c80 con 0x7fc05c1084d0 2024-06-04T05:19:39.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.239+0000 7fc043fff700 1 -- 172.21.15.70:0/2677407831 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0540052a0 con 0x7fc05c1084d0 2024-06-04T05:19:39.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.239+0000 7fc0614ce700 1 -- 172.21.15.70:0/2677407831 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc05c1a66c0 con 0x7fc05c1084d0 2024-06-04T05:19:39.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.240+0000 7fc0614ce700 1 -- 172.21.15.70:0/2677407831 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc05c103b80 con 0x7fc05c1084d0 2024-06-04T05:19:39.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.243+0000 7fc043fff700 1 -- 172.21.15.70:0/2677407831 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fc054024070 con 0x7fc05c1084d0 2024-06-04T05:19:39.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.243+0000 7fc043fff700 1 --2- 172.21.15.70:0/2677407831 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc0440b29f0 0x7fc0440b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:39.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.243+0000 7fc05affd700 1 --2- 172.21.15.70:0/2677407831 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc0440b29f0 0x7fc0440b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:39.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.243+0000 7fc043fff700 1 -- 172.21.15.70:0/2677407831 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7fc05400f020 con 0x7fc05c1084d0 2024-06-04T05:19:39.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.244+0000 7fc05affd700 1 --2- 172.21.15.70:0/2677407831 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc0440b29f0 0x7fc0440b4eb0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc05c19aaf0 tx=0x7fc04c009190 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:39.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.247+0000 7fc043fff700 1 -- 172.21.15.70:0/2677407831 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc05402b020 con 0x7fc05c1084d0 2024-06-04T05:19:39.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.495+0000 7fc0614ce700 1 -- 172.21.15.70:0/2677407831 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc05c19aee0 con 0x7fc05c1084d0 2024-06-04T05:19:39.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:39 smithi070 ceph-mon[32631]: pgmap v707: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 295 KiB/s wr, 76 op/s 2024-06-04T05:19:39.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.496+0000 7fc043fff700 1 -- 172.21.15.70:0/2677407831 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+11930 (secure 0 0 0) 0x7fc0540abb60 con 0x7fc05c1084d0 2024-06-04T05:19:39.499 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:19:39.499 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":60,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:19:35.014473+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:19:39.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.499+0000 7fc0614ce700 1 -- 172.21.15.70:0/2677407831 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc0440b29f0 msgr2=0x7fc0440b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:39.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.499+0000 7fc0614ce700 1 --2- 172.21.15.70:0/2677407831 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc0440b29f0 0x7fc0440b4eb0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc05c19aaf0 tx=0x7fc04c009190 comp rx=0 tx=0).stop 2024-06-04T05:19:39.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.499+0000 7fc0614ce700 1 -- 172.21.15.70:0/2677407831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc05c1084d0 msgr2=0x7fc05c1999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:39.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.499+0000 7fc0614ce700 1 --2- 172.21.15.70:0/2677407831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc05c1084d0 0x7fc05c1999b0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7fc0540062a0 tx=0x7fc054004e20 comp rx=0 tx=0).stop 2024-06-04T05:19:39.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.499+0000 7fc0614ce700 1 -- 172.21.15.70:0/2677407831 shutdown_connections 2024-06-04T05:19:39.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.499+0000 7fc0614ce700 1 --2- 172.21.15.70:0/2677407831 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc0440b29f0 0x7fc0440b4eb0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:39.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.499+0000 7fc0614ce700 1 --2- 172.21.15.70:0/2677407831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc05c1084d0 0x7fc05c1999b0 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:39.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.499+0000 7fc0614ce700 1 --2- 172.21.15.70:0/2677407831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc05c1024d0 0x7fc05c19f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:39.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.499+0000 7fc0614ce700 1 --2- 172.21.15.70:0/2677407831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc05c102e90 0x7fc05c19fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:39.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.499+0000 7fc0614ce700 1 -- 172.21.15.70:0/2677407831 >> 172.21.15.70:0/2677407831 conn(0x7fc05c078490 msgr2=0x7fc05c100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:39.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.499+0000 7fc0614ce700 1 -- 172.21.15.70:0/2677407831 shutdown_connections 2024-06-04T05:19:39.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:39.499+0000 7fc0614ce700 1 -- 172.21.15.70:0/2677407831 wait complete. 2024-06-04T05:19:39.502 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 60 2024-06-04T05:19:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:39 smithi112 ceph-mon[36453]: pgmap v707: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 295 KiB/s wr, 76 op/s 2024-06-04T05:19:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:39 smithi098 ceph-mon[35569]: pgmap v707: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 295 KiB/s wr, 76 op/s 2024-06-04T05:19:39.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.50 MB/sec execute 391 sec latency 17206.931 ms 2024-06-04T05:19:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.034+0000 7f97ec5e4640 1 -- 172.21.15.70:0/3685677009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97e4102a60 msgr2=0x7f97e4102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.034+0000 7f97ec5e4640 1 --2- 172.21.15.70:0/3685677009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97e4102a60 0x7f97e4102ee0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009930 tx=0x7f97d802a420 comp rx=0 tx=0).stop 2024-06-04T05:19:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.034+0000 7f97ec5e4640 1 -- 172.21.15.70:0/3685677009 shutdown_connections 2024-06-04T05:19:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.034+0000 7f97ec5e4640 1 --2- 172.21.15.70:0/3685677009 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97e4103420 0x7f97e410f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.034+0000 7f97ec5e4640 1 --2- 172.21.15.70:0/3685677009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97e4102a60 0x7f97e4102ee0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.034+0000 7f97ec5e4640 1 --2- 172.21.15.70:0/3685677009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e4108a70 0x7f97e4108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.034+0000 7f97ec5e4640 1 -- 172.21.15.70:0/3685677009 >> 172.21.15.70:0/3685677009 conn(0x7f97e40fe550 msgr2=0x7f97e4100970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.034+0000 7f97ec5e4640 1 -- 172.21.15.70:0/3685677009 shutdown_connections 2024-06-04T05:19:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.034+0000 7f97ec5e4640 1 -- 172.21.15.70:0/3685677009 wait complete. 2024-06-04T05:19:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97ec5e4640 1 Processor -- start 2024-06-04T05:19:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97ec5e4640 1 -- start start 2024-06-04T05:19:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97ec5e4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97e4102a60 0x7f97e41990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97ec5e4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97e4103420 0x7f97e4199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97ec5e4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e4108a70 0x7f97e419d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97ec5e4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f97e4112be0 con 0x7f97e4103420 2024-06-04T05:19:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97ec5e4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97e4112a60 con 0x7f97e4108a70 2024-06-04T05:19:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97ec5e4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f97e4112d60 con 0x7f97e4102a60 2024-06-04T05:19:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97e9b7f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97e4103420 0x7f97e4199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97e9b7f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97e4103420 0x7f97e4199600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50554/0 (socket says 172.21.15.70:50554) 2024-06-04T05:19:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97e9b7f640 1 -- 172.21.15.70:0/3442489574 learned_addr learned my addr 172.21.15.70:0/3442489574 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97eab81640 1 --2- 172.21.15.70:0/3442489574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e4108a70 0x7f97e419d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97ea380640 1 --2- 172.21.15.70:0/3442489574 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97e4102a60 0x7f97e41990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97e9b7f640 1 -- 172.21.15.70:0/3442489574 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97e4102a60 msgr2=0x7f97e41990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97e9b7f640 1 --2- 172.21.15.70:0/3442489574 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97e4102a60 0x7f97e41990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97e9b7f640 1 -- 172.21.15.70:0/3442489574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e4108a70 msgr2=0x7f97e419d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97e9b7f640 1 --2- 172.21.15.70:0/3442489574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e4108a70 0x7f97e419d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97e9b7f640 1 -- 172.21.15.70:0/3442489574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e419e060 con 0x7f97e4103420 2024-06-04T05:19:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97eab81640 1 --2- 172.21.15.70:0/3442489574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e4108a70 0x7f97e419d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:19:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.035+0000 7f97e9b7f640 1 --2- 172.21.15.70:0/3442489574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97e4103420 0x7f97e4199600 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f97d8005940 tx=0x7f97d80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.036+0000 7f97d77fe640 1 -- 172.21.15.70:0/3442489574 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d803f030 con 0x7f97e4103420 2024-06-04T05:19:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.036+0000 7f97d77fe640 1 -- 172.21.15.70:0/3442489574 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97d8003c90 con 0x7f97e4103420 2024-06-04T05:19:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.036+0000 7f97ec5e4640 1 -- 172.21.15.70:0/3442489574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97e419e2f0 con 0x7f97e4103420 2024-06-04T05:19:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.036+0000 7f97d77fe640 1 -- 172.21.15.70:0/3442489574 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d8035070 con 0x7f97e4103420 2024-06-04T05:19:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.036+0000 7f97ec5e4640 1 -- 172.21.15.70:0/3442489574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97e41a5be0 con 0x7f97e4103420 2024-06-04T05:19:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.038+0000 7f97d77fe640 1 -- 172.21.15.70:0/3442489574 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f97d8042030 con 0x7f97e4103420 2024-06-04T05:19:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.038+0000 7f97ef3db740 1 Processor -- start 2024-06-04T05:19:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.038+0000 7f97ef3db740 1 -- start start 2024-06-04T05:19:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97ef3db740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0e1820 0x55a8bb0e1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97ef3db740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e2140 0x55a8bb0ec4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97ef3db740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0eca00 0x55a8bb0eedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97ef3db740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a8bb053c30 con 0x55a8bb0e2140 2024-06-04T05:19:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97ef3db740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a8bb0437d0 con 0x55a8bb0e1820 2024-06-04T05:19:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97ef3db740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a8bb047810 con 0x55a8bb0eca00 2024-06-04T05:19:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97ea380640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e2140 0x55a8bb0ec4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97ea380640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e2140 0x55a8bb0ec4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50568/0 (socket says 172.21.15.70:50568) 2024-06-04T05:19:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97ea380640 1 -- 172.21.15.70:0/259987811 learned_addr learned my addr 172.21.15.70:0/259987811 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97eab81640 1 --2- 172.21.15.70:0/259987811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0e1820 0x55a8bb0e1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97e9b7f640 1 --2- 172.21.15.70:0/259987811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0eca00 0x55a8bb0eedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97d77fe640 1 --2- 172.21.15.70:0/3442489574 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97d00b2a90 0x7f97d00b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97ea380640 1 -- 172.21.15.70:0/259987811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0eca00 msgr2=0x55a8bb0eedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97ea380640 1 --2- 172.21.15.70:0/259987811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0eca00 0x55a8bb0eedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97ea380640 1 -- 172.21.15.70:0/259987811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0e1820 msgr2=0x55a8bb0e1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97ea380640 1 --2- 172.21.15.70:0/259987811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0e1820 0x55a8bb0e1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97ea380640 1 -- 172.21.15.70:0/259987811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8bad58210 con 0x55a8bb0e2140 2024-06-04T05:19:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97eab81640 1 --2- 172.21.15.70:0/259987811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0e1820 0x55a8bb0e1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:19:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97eab81640 1 --2- 172.21.15.70:0/3442489574 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97d00b2a90 0x7f97d00b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97d77fe640 1 -- 172.21.15.70:0/3442489574 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f97d8102b30 con 0x7f97e4103420 2024-06-04T05:19:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.039+0000 7f97ea380640 1 --2- 172.21.15.70:0/259987811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e2140 0x55a8bb0ec4c0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f97e000bac0 tx=0x7f97e000bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209134f7b8149abf server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.040+0000 7f97d4bfe640 1 -- 172.21.15.70:0/259987811 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e0017420 con 0x55a8bb0e2140 2024-06-04T05:19:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.040+0000 7f97d4bfe640 1 -- 172.21.15.70:0/259987811 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97e0017a50 con 0x55a8bb0e2140 2024-06-04T05:19:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.040+0000 7f97d4bfe640 1 -- 172.21.15.70:0/259987811 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e000f8e0 con 0x55a8bb0e2140 2024-06-04T05:19:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.040+0000 7f97ef3db740 1 -- 172.21.15.70:0/259987811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e2140 msgr2=0x55a8bb0ec4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.040+0000 7f97eab81640 1 --2- 172.21.15.70:0/3442489574 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97d00b2a90 0x7f97d00b4f50 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f97dc005fd0 tx=0x7f97dc004a40 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.040+0000 7f97ef3db740 1 --2- 172.21.15.70:0/259987811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e2140 0x55a8bb0ec4c0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f97e000bac0 tx=0x7f97e000bf80 comp rx=0 tx=0).stop 2024-06-04T05:19:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.040+0000 7f97ef3db740 1 -- 172.21.15.70:0/259987811 shutdown_connections 2024-06-04T05:19:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.040+0000 7f97ef3db740 1 --2- 172.21.15.70:0/259987811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0eca00 0x55a8bb0eedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.040+0000 7f97ef3db740 1 --2- 172.21.15.70:0/259987811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e2140 0x55a8bb0ec4c0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.040+0000 7f97ef3db740 1 --2- 172.21.15.70:0/259987811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0e1820 0x55a8bb0e1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.040+0000 7f97ef3db740 1 -- 172.21.15.70:0/259987811 >> 172.21.15.70:0/259987811 conn(0x55a8bb0ddd30 msgr2=0x55a8bb0de110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.040+0000 7f97ef3db740 1 -- 172.21.15.70:0/259987811 shutdown_connections 2024-06-04T05:19:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.040+0000 7f97ef3db740 1 -- 172.21.15.70:0/259987811 wait complete. 2024-06-04T05:19:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97ef3db740 1 -- start start 2024-06-04T05:19:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97ef3db740 1 Processor -- start 2024-06-04T05:19:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97ef3db740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e1820 0x55a8bb0e99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97ef3db740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0e2140 0x55a8bb0e9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97ef3db740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0eca00 0x55a8bb0ea450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97ef3db740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a8bad29790 con 0x55a8bb0e1820 2024-06-04T05:19:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97ef3db740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a8bafff890 con 0x55a8bb0e2140 2024-06-04T05:19:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97ef3db740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a8bb0ef630 con 0x55a8bb0eca00 2024-06-04T05:19:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97ef3db740 10 client.0 _ll_register_callbacks cb 0x55a8bad57a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97ea380640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e1820 0x55a8bb0e99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97ea380640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e1820 0x55a8bb0e99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50584/0 (socket says 172.21.15.70:50584) 2024-06-04T05:19:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97ea380640 1 -- 172.21.15.70:0/1072957586 learned_addr learned my addr 172.21.15.70:0/1072957586 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97e9b7f640 1 --2- 172.21.15.70:0/1072957586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0e2140 0x55a8bb0e9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97ea380640 1 -- 172.21.15.70:0/1072957586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0eca00 msgr2=0x55a8bb0ea450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97ea380640 1 --2- 172.21.15.70:0/1072957586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0eca00 0x55a8bb0ea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97ea380640 1 -- 172.21.15.70:0/1072957586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0e2140 msgr2=0x55a8bb0e9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97ea380640 1 --2- 172.21.15.70:0/1072957586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0e2140 0x55a8bb0e9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.041+0000 7f97ea380640 1 -- 172.21.15.70:0/1072957586 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8bae24620 con 0x55a8bb0e1820 2024-06-04T05:19:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.042+0000 7f97ea380640 1 --2- 172.21.15.70:0/1072957586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e1820 0x55a8bb0e99d0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f97e000df40 tx=0x7f97e000b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b818130988c7d91 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.042+0000 7f97cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.042+0000 7f97cdffb640 1 -- 172.21.15.70:0/1072957586 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e0027020 con 0x55a8bb0e1820 2024-06-04T05:19:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.042+0000 7f97cdffb640 1 -- 172.21.15.70:0/1072957586 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f97e0017830 con 0x55a8bb0e1820 2024-06-04T05:19:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.042+0000 7f97ef3db740 1 -- 172.21.15.70:0/1072957586 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97e0017a50 con 0x55a8bb0e1820 2024-06-04T05:19:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.042+0000 7f97cdffb640 1 -- 172.21.15.70:0/1072957586 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f97e0032050 con 0x55a8bb0e1820 2024-06-04T05:19:40.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.042+0000 7f97ef3db740 10 client.21219 fetch_fsmap learned FSMap version 60 2024-06-04T05:19:40.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.042+0000 7f97ef3db740 1 -- 172.21.15.70:0/1072957586 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x55a8bb0fb3d0 con 0x55a8bb0e1820 2024-06-04T05:19:40.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.042+0000 7f97cdffb640 1 -- 172.21.15.70:0/1072957586 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97e0017a50 con 0x55a8bb0e1820 2024-06-04T05:19:40.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.042+0000 7f97cdffb640 1 -- 172.21.15.70:0/1072957586 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 60) ==== 8358+0+0 (secure 0 0 0) 0x7f97e00237f0 con 0x55a8bb0e1820 2024-06-04T05:19:40.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.042+0000 7f97ef3db740 10 client.21219 fetch_fsmap finished waiting for FSMap version 60 2024-06-04T05:19:40.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.042+0000 7f97ef3db740 10 client.21219 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:19:40.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.042+0000 7f97ef3db740 20 client.21219 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:40.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.042+0000 7f97ef3db740 1 --2- 172.21.15.70:0/1072957586 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a8bb0fc890 0x55a8bb0fec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.042+0000 7f97ef3db740 4 client.21219 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:19:40.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.042+0000 7f97ef3db740 1 -- 172.21.15.70:0/1072957586 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8bb055b80 con 0x55a8bb0fc890 2024-06-04T05:19:40.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.043+0000 7f97e9b7f640 1 --2- 172.21.15.70:0/1072957586 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a8bb0fc890 0x55a8bb0fec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:40.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.043+0000 7f97cdffb640 10 client.21219 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:40.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.043+0000 7f97e9b7f640 1 --2- 172.21.15.70:0/1072957586 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a8bb0fc890 0x55a8bb0fec80 crc :-1 s=READY pgs=475 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25d835c89409251 server_cookie=6cf78c75ac70727b in_seq=0 out_seq=0 2024-06-04T05:19:40.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.044+0000 7f97cdffb640 1 -- 172.21.15.70:0/1072957586 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a8bb055b80 con 0x55a8bb0fc890 2024-06-04T05:19:40.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.044+0000 7f97cdffb640 10 client.21219 handle_command_reply: tid=0 2024-06-04T05:19:40.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.044+0000 7f97ef3db740 1 client.21219 shutdown 2024-06-04T05:19:40.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.044+0000 7f97ef3db740 1 -- 172.21.15.70:0/1072957586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e1820 msgr2=0x55a8bb0e99d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:40.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.044+0000 7f97ef3db740 1 --2- 172.21.15.70:0/1072957586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e1820 0x55a8bb0e99d0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f97e000df40 tx=0x7f97e000b610 comp rx=0 tx=0).stop 2024-06-04T05:19:40.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.044+0000 7f97ef3db740 1 -- 172.21.15.70:0/1072957586 shutdown_connections 2024-06-04T05:19:40.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.044+0000 7f97ef3db740 1 --2- 172.21.15.70:0/1072957586 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a8bb0fc890 0x55a8bb0fec80 crc :-1 s=READY pgs=475 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.044+0000 7f97ef3db740 1 --2- 172.21.15.70:0/1072957586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0e2140 0x55a8bb0e9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.044+0000 7f97ef3db740 1 --2- 172.21.15.70:0/1072957586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0eca00 0x55a8bb0ea450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.044+0000 7f97cdffb640 0 client.21219 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:40.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.044+0000 7f97ef3db740 1 --2- 172.21.15.70:0/1072957586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e1820 0x55a8bb0e99d0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.044+0000 7f97ef3db740 1 -- 172.21.15.70:0/1072957586 >> 172.21.15.70:0/1072957586 conn(0x55a8bb0ddd30 msgr2=0x55a8bb0dcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:40.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.044+0000 7f97ef3db740 1 -- 172.21.15.70:0/1072957586 shutdown_connections 2024-06-04T05:19:40.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.044+0000 7f97ef3db740 1 -- 172.21.15.70:0/1072957586 wait complete. 2024-06-04T05:19:40.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.044+0000 7f97ef3db740 20 client.21219 trim_cache size 0 max 16384 2024-06-04T05:19:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.059+0000 7f97ef3db740 1 Processor -- start 2024-06-04T05:19:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.059+0000 7f97ef3db740 1 -- start start 2024-06-04T05:19:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97ef3db740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0e1820 0x55a8bb0e9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97ef3db740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0e2140 0x55a8bb0ea460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97ef3db740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0eca00 0x55a8bb104310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97ef3db740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f97e41ae560 con 0x55a8bb0eca00 2024-06-04T05:19:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97ef3db740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97e41b00d0 con 0x55a8bb0e2140 2024-06-04T05:19:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97ef3db740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f97e41aec60 con 0x55a8bb0e1820 2024-06-04T05:19:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97eab81640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0eca00 0x55a8bb104310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97eab81640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0eca00 0x55a8bb104310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50598/0 (socket says 172.21.15.70:50598) 2024-06-04T05:19:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97eab81640 1 -- 172.21.15.70:0/3491760308 learned_addr learned my addr 172.21.15.70:0/3491760308 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97ea380640 1 --2- 172.21.15.70:0/3491760308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0e1820 0x55a8bb0e9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97eab81640 1 -- 172.21.15.70:0/3491760308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0e1820 msgr2=0x55a8bb0e9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97eab81640 1 --2- 172.21.15.70:0/3491760308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0e1820 0x55a8bb0e9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97eab81640 1 -- 172.21.15.70:0/3491760308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0e2140 msgr2=0x55a8bb0ea460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97eab81640 1 --2- 172.21.15.70:0/3491760308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0e2140 0x55a8bb0ea460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97eab81640 1 -- 172.21.15.70:0/3491760308 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8bb0ff0e0 con 0x55a8bb0eca00 2024-06-04T05:19:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97eab81640 1 --2- 172.21.15.70:0/3491760308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0eca00 0x55a8bb104310 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f97dc013540 tx=0x7f97dc014070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d9d5b67154866c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97cffff640 1 -- 172.21.15.70:0/3491760308 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc026680 con 0x55a8bb0eca00 2024-06-04T05:19:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97cffff640 1 -- 172.21.15.70:0/3491760308 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97dc01c8d0 con 0x55a8bb0eca00 2024-06-04T05:19:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.060+0000 7f97cffff640 1 -- 172.21.15.70:0/3491760308 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc01a900 con 0x55a8bb0eca00 2024-06-04T05:19:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.061+0000 7f97ef3db740 1 -- 172.21.15.70:0/3491760308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0eca00 msgr2=0x55a8bb104310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.061+0000 7f97ef3db740 1 --2- 172.21.15.70:0/3491760308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0eca00 0x55a8bb104310 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f97dc013540 tx=0x7f97dc014070 comp rx=0 tx=0).stop 2024-06-04T05:19:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.061+0000 7f97ef3db740 1 -- 172.21.15.70:0/3491760308 shutdown_connections 2024-06-04T05:19:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.061+0000 7f97ef3db740 1 --2- 172.21.15.70:0/3491760308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0e2140 0x55a8bb0ea460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.061+0000 7f97ef3db740 1 --2- 172.21.15.70:0/3491760308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0eca00 0x55a8bb104310 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.061+0000 7f97ef3db740 1 --2- 172.21.15.70:0/3491760308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0e1820 0x55a8bb0e9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.061+0000 7f97ef3db740 1 -- 172.21.15.70:0/3491760308 >> 172.21.15.70:0/3491760308 conn(0x55a8bb0ddd30 msgr2=0x55a8bb0dcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.061+0000 7f97ef3db740 1 -- 172.21.15.70:0/3491760308 shutdown_connections 2024-06-04T05:19:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.061+0000 7f97ef3db740 1 -- 172.21.15.70:0/3491760308 wait complete. 2024-06-04T05:19:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.061+0000 7f97ef3db740 1 -- start start 2024-06-04T05:19:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.061+0000 7f97ef3db740 1 Processor -- start 2024-06-04T05:19:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ef3db740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e1820 0x55a8bb0f4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ef3db740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0e2140 0x55a8bb0f0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ef3db740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0eca00 0x55a8bb0f0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ef3db740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f97e41afa00 con 0x55a8bb0e1820 2024-06-04T05:19:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ef3db740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97e41af330 con 0x55a8bb0eca00 2024-06-04T05:19:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ef3db740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f97e41ac560 con 0x55a8bb0e2140 2024-06-04T05:19:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ef3db740 10 client.0 _ll_register_callbacks cb 0x7f97e000b910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ea380640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e1820 0x55a8bb0f4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ea380640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e1820 0x55a8bb0f4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50600/0 (socket says 172.21.15.70:50600) 2024-06-04T05:19:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ea380640 1 -- 172.21.15.70:0/92251215 learned_addr learned my addr 172.21.15.70:0/92251215 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97e9b7f640 1 --2- 172.21.15.70:0/92251215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0e2140 0x55a8bb0f0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97eab81640 1 --2- 172.21.15.70:0/92251215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0eca00 0x55a8bb0f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ea380640 1 -- 172.21.15.70:0/92251215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0e2140 msgr2=0x55a8bb0f0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ea380640 1 --2- 172.21.15.70:0/92251215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0e2140 0x55a8bb0f0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ea380640 1 -- 172.21.15.70:0/92251215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0eca00 msgr2=0x55a8bb0f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ea380640 1 --2- 172.21.15.70:0/92251215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0eca00 0x55a8bb0f0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ea380640 1 -- 172.21.15.70:0/92251215 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8bb0f0af0 con 0x55a8bb0e1820 2024-06-04T05:19:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97eab81640 1 --2- 172.21.15.70:0/92251215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0eca00 0x55a8bb0f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:19:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ea380640 1 --2- 172.21.15.70:0/92251215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e1820 0x55a8bb0f4cd0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f97e000b9e0 tx=0x7f97e0021650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=347794fddbcfc48a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97cdffb640 1 -- 172.21.15.70:0/92251215 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e0027040 con 0x55a8bb0e1820 2024-06-04T05:19:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97cdffb640 1 -- 172.21.15.70:0/92251215 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f97e0017840 con 0x55a8bb0e1820 2024-06-04T05:19:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ef3db740 1 -- 172.21.15.70:0/92251215 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97dc01c8d0 con 0x55a8bb0e1820 2024-06-04T05:19:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97cdffb640 1 -- 172.21.15.70:0/92251215 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f97e0044050 con 0x55a8bb0e1820 2024-06-04T05:19:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.062+0000 7f97ef3db740 10 client.21225 fetch_fsmap learned FSMap version 60 2024-06-04T05:19:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.063+0000 7f97ef3db740 1 -- 172.21.15.70:0/92251215 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x55a8bb166f50 con 0x55a8bb0e1820 2024-06-04T05:19:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.063+0000 7f97cdffb640 1 -- 172.21.15.70:0/92251215 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97dc01c8d0 con 0x55a8bb0e1820 2024-06-04T05:19:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.063+0000 7f97cdffb640 1 -- 172.21.15.70:0/92251215 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 60) ==== 8358+0+0 (secure 0 0 0) 0x7f97e0021880 con 0x55a8bb0e1820 2024-06-04T05:19:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.063+0000 7f97ef3db740 10 client.21225 fetch_fsmap finished waiting for FSMap version 60 2024-06-04T05:19:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.063+0000 7f97ef3db740 10 client.21225 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:19:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.063+0000 7f97ef3db740 20 client.21225 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.063+0000 7f97ef3db740 1 --2- 172.21.15.70:0/92251215 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a8bb167630 0x55a8bb169a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.063+0000 7f97ef3db740 4 client.21225 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:19:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.063+0000 7f97ef3db740 1 -- 172.21.15.70:0/92251215 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a8bb04f360 con 0x55a8bb167630 2024-06-04T05:19:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.063+0000 7f97e9b7f640 1 --2- 172.21.15.70:0/92251215 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a8bb167630 0x55a8bb169a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.063+0000 7f97e9b7f640 1 --2- 172.21.15.70:0/92251215 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a8bb167630 0x55a8bb169a50 crc :-1 s=READY pgs=476 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb9e67c9caea32a0 server_cookie=ac54b81daab70efc in_seq=0 out_seq=0 2024-06-04T05:19:40.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.063+0000 7f97cdffb640 10 client.21225 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:40.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.104+0000 7f97cdffb640 1 -- 172.21.15.70:0/92251215 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55a8bb04f360 con 0x55a8bb167630 2024-06-04T05:19:40.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.104+0000 7f97cdffb640 10 client.21225 handle_command_reply: tid=0 2024-06-04T05:19:40.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.104+0000 7f97ef3db740 1 client.21225 shutdown 2024-06-04T05:19:40.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.104+0000 7f97ef3db740 1 -- 172.21.15.70:0/92251215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e1820 msgr2=0x55a8bb0f4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:40.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.104+0000 7f97ef3db740 1 --2- 172.21.15.70:0/92251215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e1820 0x55a8bb0f4cd0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f97e000b9e0 tx=0x7f97e0021650 comp rx=0 tx=0).stop 2024-06-04T05:19:40.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.104+0000 7f97ef3db740 1 -- 172.21.15.70:0/92251215 shutdown_connections 2024-06-04T05:19:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.104+0000 7f97ef3db740 1 --2- 172.21.15.70:0/92251215 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a8bb167630 0x55a8bb169a50 crc :-1 s=READY pgs=476 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.104+0000 7f97ef3db740 1 --2- 172.21.15.70:0/92251215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8bb0eca00 0x55a8bb0f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.104+0000 7f97cdffb640 0 client.21225 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.104+0000 7f97ef3db740 1 --2- 172.21.15.70:0/92251215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8bb0e2140 0x55a8bb0f0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.104+0000 7f97ef3db740 1 --2- 172.21.15.70:0/92251215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8bb0e1820 0x55a8bb0f4cd0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.104+0000 7f97ef3db740 1 -- 172.21.15.70:0/92251215 >> 172.21.15.70:0/92251215 conn(0x55a8bb0ddd30 msgr2=0x55a8bb0e31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.104+0000 7f97ef3db740 1 -- 172.21.15.70:0/92251215 shutdown_connections 2024-06-04T05:19:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.104+0000 7f97ef3db740 1 -- 172.21.15.70:0/92251215 wait complete. 2024-06-04T05:19:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.104+0000 7f97ef3db740 20 client.21225 trim_cache size 0 max 16384 2024-06-04T05:19:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:19:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:19:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:19:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.105+0000 7f97ec5e4640 1 -- 172.21.15.70:0/3442489574 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97d00b2a90 msgr2=0x7f97d00b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.105+0000 7f97ec5e4640 1 --2- 172.21.15.70:0/3442489574 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97d00b2a90 0x7f97d00b4f50 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f97dc005fd0 tx=0x7f97dc004a40 comp rx=0 tx=0).stop 2024-06-04T05:19:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.105+0000 7f97ec5e4640 1 -- 172.21.15.70:0/3442489574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97e4103420 msgr2=0x7f97e4199600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.105+0000 7f97ec5e4640 1 --2- 172.21.15.70:0/3442489574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97e4103420 0x7f97e4199600 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f97d8005940 tx=0x7f97d80042f0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.105+0000 7f97ec5e4640 1 -- 172.21.15.70:0/3442489574 shutdown_connections 2024-06-04T05:19:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.105+0000 7f97ec5e4640 1 --2- 172.21.15.70:0/3442489574 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f97d00b2a90 0x7f97d00b4f50 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.105+0000 7f97ec5e4640 1 --2- 172.21.15.70:0/3442489574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e4108a70 0x7f97e419d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.105+0000 7f97ec5e4640 1 --2- 172.21.15.70:0/3442489574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97e4103420 0x7f97e4199600 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.105+0000 7f97ec5e4640 1 --2- 172.21.15.70:0/3442489574 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97e4102a60 0x7f97e41990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.105+0000 7f97ec5e4640 1 -- 172.21.15.70:0/3442489574 >> 172.21.15.70:0/3442489574 conn(0x7f97e40fe550 msgr2=0x7f97e40feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.105+0000 7f97ec5e4640 1 -- 172.21.15.70:0/3442489574 shutdown_connections 2024-06-04T05:19:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:40.105+0000 7f97ec5e4640 1 -- 172.21.15.70:0/3442489574 wait complete. 2024-06-04T05:19:40.186 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id da6cd7e7 --release --await' 2024-06-04T05:19:40.187 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id da6cd7e7 --release --await 2024-06-04T05:19:40.469 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:19:40.591 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:40 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2677407831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:19:40.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:40 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2677407831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:19:40.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:40 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2677407831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:19:40.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.48 MB/sec execute 392 sec latency 18206.985 ms 2024-06-04T05:19:41.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:41 smithi070 ceph-mon[32631]: pgmap v708: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 295 KiB/s wr, 76 op/s 2024-06-04T05:19:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:41 smithi112 ceph-mon[36453]: pgmap v708: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 295 KiB/s wr, 76 op/s 2024-06-04T05:19:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:41 smithi098 ceph-mon[35569]: pgmap v708: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 295 KiB/s wr, 76 op/s 2024-06-04T05:19:41.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63722 6.47 MB/sec execute 393 sec latency 19207.059 ms 2024-06-04T05:19:42.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.241+0000 7fc8678fd700 1 -- 172.21.15.70:0/1044972727 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc86006c470 msgr2=0x7fc8601103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:42.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.241+0000 7fc8678fd700 1 --2- 172.21.15.70:0/1044972727 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc86006c470 0x7fc8601103c0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fc85800b0a0 tx=0x7fc85800b3b0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.241+0000 7fc8678fd700 1 -- 172.21.15.70:0/1044972727 shutdown_connections 2024-06-04T05:19:42.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.241+0000 7fc8678fd700 1 --2- 172.21.15.70:0/1044972727 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc86006c470 0x7fc8601103c0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.241+0000 7fc8678fd700 1 --2- 172.21.15.70:0/1044972727 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8601095a0 0x7fc860109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.241+0000 7fc8678fd700 1 --2- 172.21.15.70:0/1044972727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc86006bab0 0x7fc86006bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.241+0000 7fc8678fd700 1 -- 172.21.15.70:0/1044972727 >> 172.21.15.70:0/1044972727 conn(0x7fc8600793b0 msgr2=0x7fc8600797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:42.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.241+0000 7fc8678fd700 1 -- 172.21.15.70:0/1044972727 shutdown_connections 2024-06-04T05:19:42.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.241+0000 7fc8678fd700 1 -- 172.21.15.70:0/1044972727 wait complete. 2024-06-04T05:19:42.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc8678fd700 1 Processor -- start 2024-06-04T05:19:42.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc8678fd700 1 -- start start 2024-06-04T05:19:42.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc8678fd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc86006bab0 0x7fc86019f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc8678fd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc86006c470 0x7fc86019fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc8678fd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8601095a0 0x7fc860199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc8678fd700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc8601135e0 con 0x7fc86006c470 2024-06-04T05:19:42.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc8678fd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc860113460 con 0x7fc86006bab0 2024-06-04T05:19:42.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc8678fd700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc860113760 con 0x7fc8601095a0 2024-06-04T05:19:42.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc864e98700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc86006c470 0x7fc86019fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:42.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc864e98700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc86006c470 0x7fc86019fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50612/0 (socket says 172.21.15.70:50612) 2024-06-04T05:19:42.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc864e98700 1 -- 172.21.15.70:0/1199684983 learned_addr learned my addr 172.21.15.70:0/1199684983 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:42.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc865699700 1 --2- 172.21.15.70:0/1199684983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc86006bab0 0x7fc86019f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:42.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc864e98700 1 -- 172.21.15.70:0/1199684983 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8601095a0 msgr2=0x7fc860199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:42.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc864e98700 1 --2- 172.21.15.70:0/1199684983 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8601095a0 0x7fc860199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc864e98700 1 -- 172.21.15.70:0/1199684983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc86006bab0 msgr2=0x7fc86019f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:42.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc864e98700 1 --2- 172.21.15.70:0/1199684983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc86006bab0 0x7fc86019f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc864e98700 1 -- 172.21.15.70:0/1199684983 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86019a180 con 0x7fc86006c470 2024-06-04T05:19:42.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.242+0000 7fc865699700 1 --2- 172.21.15.70:0/1199684983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc86006bab0 0x7fc86019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:19:42.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.243+0000 7fc864e98700 1 --2- 172.21.15.70:0/1199684983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc86006c470 0x7fc86019fda0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fc85000b530 tx=0x7fc85000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:42.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.243+0000 7fc8567fc700 1 -- 172.21.15.70:0/1199684983 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc850015020 con 0x7fc86006c470 2024-06-04T05:19:42.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.243+0000 7fc8567fc700 1 -- 172.21.15.70:0/1199684983 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc850011cc0 con 0x7fc86006c470 2024-06-04T05:19:42.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.243+0000 7fc8678fd700 1 -- 172.21.15.70:0/1199684983 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc86019a470 con 0x7fc86006c470 2024-06-04T05:19:42.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.243+0000 7fc8567fc700 1 -- 172.21.15.70:0/1199684983 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc850010d30 con 0x7fc86006c470 2024-06-04T05:19:42.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.243+0000 7fc8678fd700 1 -- 172.21.15.70:0/1199684983 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8601a6730 con 0x7fc86006c470 2024-06-04T05:19:42.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.246+0000 7fc8567fc700 1 -- 172.21.15.70:0/1199684983 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fc850018020 con 0x7fc86006c470 2024-06-04T05:19:42.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.248+0000 7fc8567fc700 1 --2- 172.21.15.70:0/1199684983 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc84c0b28d0 0x7fc84c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.248+0000 7fc872192b80 1 Processor -- start 2024-06-04T05:19:42.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.248+0000 7fc865699700 1 --2- 172.21.15.70:0/1199684983 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc84c0b28d0 0x7fc84c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:42.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.248+0000 7fc872192b80 1 -- start start 2024-06-04T05:19:42.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.248+0000 7fc8567fc700 1 -- 172.21.15.70:0/1199684983 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7fc8500e1d20 con 0x7fc86006c470 2024-06-04T05:19:42.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.248+0000 7fc872192b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a53058ed0 0x564a530592b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.248+0000 7fc872192b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a53059880 0x564a5305d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.248+0000 7fc872192b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a5305d8c0 0x564a5305dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.248+0000 7fc872192b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564a52fb3d90 con 0x564a53059880 2024-06-04T05:19:42.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.248+0000 7fc872192b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564a52fa3930 con 0x564a53058ed0 2024-06-04T05:19:42.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.248+0000 7fc872192b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564a52fa7970 con 0x564a5305d8c0 2024-06-04T05:19:42.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.248+0000 7fc864e98700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a53059880 0x564a5305d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:42.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.248+0000 7fc864e98700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a53059880 0x564a5305d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50624/0 (socket says 172.21.15.70:50624) 2024-06-04T05:19:42.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.248+0000 7fc864e98700 1 -- 172.21.15.70:0/1047632288 learned_addr learned my addr 172.21.15.70:0/1047632288 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:42.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.248+0000 7fc865699700 1 --2- 172.21.15.70:0/1199684983 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc84c0b28d0 0x7fc84c0b4d90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc86019ab60 tx=0x7fc85c00a680 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:42.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.249+0000 7fc865699700 1 --2- 172.21.15.70:0/1047632288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a53058ed0 0x564a530592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:42.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.249+0000 7fc865e9a700 1 --2- 172.21.15.70:0/1047632288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a5305d8c0 0x564a5305dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:42.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.249+0000 7fc864e98700 1 -- 172.21.15.70:0/1047632288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a5305d8c0 msgr2=0x564a5305dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:42.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.249+0000 7fc864e98700 1 --2- 172.21.15.70:0/1047632288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a5305d8c0 0x564a5305dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.249+0000 7fc864e98700 1 -- 172.21.15.70:0/1047632288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a53058ed0 msgr2=0x564a530592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:42.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.249+0000 7fc864e98700 1 --2- 172.21.15.70:0/1047632288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a53058ed0 0x564a530592b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.249+0000 7fc864e98700 1 -- 172.21.15.70:0/1047632288 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a52ef39d0 con 0x564a53059880 2024-06-04T05:19:42.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.249+0000 7fc865699700 1 --2- 172.21.15.70:0/1047632288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a53058ed0 0x564a530592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:19:42.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.249+0000 7fc864e98700 1 --2- 172.21.15.70:0/1047632288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a53059880 0x564a5305d2a0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fc8500a9070 tx=0x7fc8500a8c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b39162f74dc0bfd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:42.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.249+0000 7fc84b7fe700 1 -- 172.21.15.70:0/1047632288 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc850010590 con 0x564a53059880 2024-06-04T05:19:42.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.249+0000 7fc84b7fe700 1 -- 172.21.15.70:0/1047632288 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc8500ad490 con 0x564a53059880 2024-06-04T05:19:42.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.249+0000 7fc84b7fe700 1 -- 172.21.15.70:0/1047632288 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8500a95d0 con 0x564a53059880 2024-06-04T05:19:42.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.250+0000 7fc872192b80 1 -- 172.21.15.70:0/1047632288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a53059880 msgr2=0x564a5305d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:42.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.250+0000 7fc872192b80 1 --2- 172.21.15.70:0/1047632288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a53059880 0x564a5305d2a0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fc8500a9070 tx=0x7fc8500a8c70 comp rx=0 tx=0).stop 2024-06-04T05:19:42.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.250+0000 7fc872192b80 1 -- 172.21.15.70:0/1047632288 shutdown_connections 2024-06-04T05:19:42.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.250+0000 7fc872192b80 1 --2- 172.21.15.70:0/1047632288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a5305d8c0 0x564a5305dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.250+0000 7fc872192b80 1 --2- 172.21.15.70:0/1047632288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a53058ed0 0x564a530592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.250+0000 7fc872192b80 1 --2- 172.21.15.70:0/1047632288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a53059880 0x564a5305d2a0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.250+0000 7fc872192b80 1 -- 172.21.15.70:0/1047632288 >> 172.21.15.70:0/1047632288 conn(0x564a52f82180 msgr2=0x564a5304f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:42.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.250+0000 7fc872192b80 1 -- 172.21.15.70:0/1047632288 shutdown_connections 2024-06-04T05:19:42.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.250+0000 7fc872192b80 1 -- 172.21.15.70:0/1047632288 wait complete. 2024-06-04T05:19:42.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.250+0000 7fc872192b80 1 -- start start 2024-06-04T05:19:42.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.250+0000 7fc872192b80 1 Processor -- start 2024-06-04T05:19:42.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc872192b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a53058ed0 0x564a53064e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc872192b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a53059880 0x564a53062100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc872192b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a5305d8c0 0x564a53062640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc872192b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564a52eaa0a0 con 0x564a5305d8c0 2024-06-04T05:19:42.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc872192b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564a52f458c0 con 0x564a53058ed0 2024-06-04T05:19:42.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc872192b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564a53065ad0 con 0x564a53059880 2024-06-04T05:19:42.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc872192b80 10 client.0 _ll_register_callbacks cb 0x564a52f3ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:42.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc865e9a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a5305d8c0 0x564a53062640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:42.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc865e9a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a5305d8c0 0x564a53062640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50638/0 (socket says 172.21.15.70:50638) 2024-06-04T05:19:42.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc865e9a700 1 -- 172.21.15.70:0/2419034271 learned_addr learned my addr 172.21.15.70:0/2419034271 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:42.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc865699700 1 --2- 172.21.15.70:0/2419034271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a53058ed0 0x564a53064e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:42.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc864e98700 1 --2- 172.21.15.70:0/2419034271 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a53059880 0x564a53062100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:42.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc865e9a700 1 -- 172.21.15.70:0/2419034271 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a53059880 msgr2=0x564a53062100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:42.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc865e9a700 1 --2- 172.21.15.70:0/2419034271 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a53059880 0x564a53062100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc865e9a700 1 -- 172.21.15.70:0/2419034271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a53058ed0 msgr2=0x564a53064e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:42.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc865e9a700 1 --2- 172.21.15.70:0/2419034271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a53058ed0 0x564a53064e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc865e9a700 1 -- 172.21.15.70:0/2419034271 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a52f42790 con 0x564a5305d8c0 2024-06-04T05:19:42.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc865e9a700 1 --2- 172.21.15.70:0/2419034271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a5305d8c0 0x564a53062640 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fc858003c30 tx=0x7fc858003c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe483f2c79431ab server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:42.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.251+0000 7fc848ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:42.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.252+0000 7fc848ff9700 1 -- 172.21.15.70:0/2419034271 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85801f030 con 0x564a5305d8c0 2024-06-04T05:19:42.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.252+0000 7fc848ff9700 1 -- 172.21.15.70:0/2419034271 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7fc8580074e0 con 0x564a5305d8c0 2024-06-04T05:19:42.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.252+0000 7fc872192b80 1 -- 172.21.15.70:0/2419034271 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8500ad490 con 0x564a5305d8c0 2024-06-04T05:19:42.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.252+0000 7fc848ff9700 1 -- 172.21.15.70:0/2419034271 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7fc858003e40 con 0x564a5305d8c0 2024-06-04T05:19:42.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.252+0000 7fc872192b80 10 client.21237 fetch_fsmap learned FSMap version 60 2024-06-04T05:19:42.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.252+0000 7fc872192b80 1 -- 172.21.15.70:0/2419034271 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x564a530702f0 con 0x564a5305d8c0 2024-06-04T05:19:42.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.252+0000 7fc848ff9700 1 -- 172.21.15.70:0/2419034271 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc8500ad490 con 0x564a5305d8c0 2024-06-04T05:19:42.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.252+0000 7fc848ff9700 1 -- 172.21.15.70:0/2419034271 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 60) ==== 8358+0+0 (secure 0 0 0) 0x7fc858011070 con 0x564a5305d8c0 2024-06-04T05:19:42.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.252+0000 7fc872192b80 10 client.21237 fetch_fsmap finished waiting for FSMap version 60 2024-06-04T05:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.252+0000 7fc872192b80 10 client.21237 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.252+0000 7fc872192b80 20 client.21237 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.252+0000 7fc872192b80 1 --2- 172.21.15.70:0/2419034271 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564a53072880 0x564a53074c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.252+0000 7fc872192b80 4 client.21237 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.252+0000 7fc872192b80 1 -- 172.21.15.70:0/2419034271 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a53020ae0 con 0x564a53072880 2024-06-04T05:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.253+0000 7fc865699700 1 --2- 172.21.15.70:0/2419034271 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564a53072880 0x564a53074c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.253+0000 7fc848ff9700 10 client.21237 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.253+0000 7fc865699700 1 --2- 172.21.15.70:0/2419034271 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564a53072880 0x564a53074c90 crc :-1 s=READY pgs=477 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab7e7cd04c714139 server_cookie=31afea5361c32ce7 in_seq=0 out_seq=0 2024-06-04T05:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.254+0000 7fc848ff9700 1 -- 172.21.15.70:0/2419034271 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564a53020ae0 con 0x564a53072880 2024-06-04T05:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.254+0000 7fc848ff9700 10 client.21237 handle_command_reply: tid=0 2024-06-04T05:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.254+0000 7fc872192b80 1 client.21237 shutdown 2024-06-04T05:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.254+0000 7fc872192b80 1 -- 172.21.15.70:0/2419034271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a5305d8c0 msgr2=0x564a53062640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.254+0000 7fc872192b80 1 --2- 172.21.15.70:0/2419034271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a5305d8c0 0x564a53062640 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fc858003c30 tx=0x7fc858003c60 comp rx=0 tx=0).stop 2024-06-04T05:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.254+0000 7fc872192b80 1 -- 172.21.15.70:0/2419034271 shutdown_connections 2024-06-04T05:19:42.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.254+0000 7fc872192b80 1 --2- 172.21.15.70:0/2419034271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a5305d8c0 0x564a53062640 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.254+0000 7fc872192b80 1 --2- 172.21.15.70:0/2419034271 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564a53072880 0x564a53074c90 crc :-1 s=READY pgs=477 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.254+0000 7fc872192b80 1 --2- 172.21.15.70:0/2419034271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a53058ed0 0x564a53064e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.254+0000 7fc872192b80 1 --2- 172.21.15.70:0/2419034271 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a53059880 0x564a53062100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.254+0000 7fc848ff9700 0 client.21237 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:42.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.254+0000 7fc872192b80 1 -- 172.21.15.70:0/2419034271 >> 172.21.15.70:0/2419034271 conn(0x564a52f82180 msgr2=0x564a5305ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:42.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.254+0000 7fc872192b80 1 -- 172.21.15.70:0/2419034271 shutdown_connections 2024-06-04T05:19:42.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.254+0000 7fc872192b80 1 -- 172.21.15.70:0/2419034271 wait complete. 2024-06-04T05:19:42.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.254+0000 7fc872192b80 20 client.21237 trim_cache size 0 max 16384 2024-06-04T05:19:42.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.274+0000 7fc872192b80 1 Processor -- start 2024-06-04T05:19:42.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.274+0000 7fc872192b80 1 -- start start 2024-06-04T05:19:42.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc872192b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a53058ed0 0x564a5306eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc872192b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a53059880 0x564a5307a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc872192b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a5305d8c0 0x564a5307aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc872192b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc8601ab880 con 0x564a53059880 2024-06-04T05:19:42.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc872192b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc8601ad3f0 con 0x564a5305d8c0 2024-06-04T05:19:42.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc872192b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc8601abf80 con 0x564a53058ed0 2024-06-04T05:19:42.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc864e98700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a53059880 0x564a5307a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:42.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc864e98700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a53059880 0x564a5307a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50646/0 (socket says 172.21.15.70:50646) 2024-06-04T05:19:42.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc864e98700 1 -- 172.21.15.70:0/394555253 learned_addr learned my addr 172.21.15.70:0/394555253 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:42.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc865699700 1 --2- 172.21.15.70:0/394555253 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a53058ed0 0x564a5306eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:42.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc865e9a700 1 --2- 172.21.15.70:0/394555253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a5305d8c0 0x564a5307aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:42.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc864e98700 1 -- 172.21.15.70:0/394555253 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a53058ed0 msgr2=0x564a5306eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:42.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc864e98700 1 --2- 172.21.15.70:0/394555253 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a53058ed0 0x564a5306eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc864e98700 1 -- 172.21.15.70:0/394555253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a5305d8c0 msgr2=0x564a5307aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:42.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc864e98700 1 --2- 172.21.15.70:0/394555253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a5305d8c0 0x564a5307aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc864e98700 1 -- 172.21.15.70:0/394555253 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a5304fcc0 con 0x564a53059880 2024-06-04T05:19:42.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc865e9a700 1 --2- 172.21.15.70:0/394555253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a5305d8c0 0x564a5307aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:19:42.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc864e98700 1 --2- 172.21.15.70:0/394555253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a53059880 0x564a5307a560 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fc8500a5110 tx=0x7fc8500093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5019aa81c9ad2857 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:42.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc84affd700 1 -- 172.21.15.70:0/394555253 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8500c7020 con 0x564a53059880 2024-06-04T05:19:42.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc84affd700 1 -- 172.21.15.70:0/394555253 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc850010590 con 0x564a53059880 2024-06-04T05:19:42.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.275+0000 7fc84affd700 1 -- 172.21.15.70:0/394555253 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8500b1ab0 con 0x564a53059880 2024-06-04T05:19:42.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.276+0000 7fc872192b80 1 -- 172.21.15.70:0/394555253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a53059880 msgr2=0x564a5307a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:42.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.276+0000 7fc872192b80 1 --2- 172.21.15.70:0/394555253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a53059880 0x564a5307a560 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fc8500a5110 tx=0x7fc8500093a0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.276+0000 7fc872192b80 1 -- 172.21.15.70:0/394555253 shutdown_connections 2024-06-04T05:19:42.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.276+0000 7fc872192b80 1 --2- 172.21.15.70:0/394555253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a5305d8c0 0x564a5307aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.276+0000 7fc872192b80 1 --2- 172.21.15.70:0/394555253 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a53058ed0 0x564a5306eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.276+0000 7fc872192b80 1 --2- 172.21.15.70:0/394555253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a53059880 0x564a5307a560 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.276+0000 7fc872192b80 1 -- 172.21.15.70:0/394555253 >> 172.21.15.70:0/394555253 conn(0x564a52f82180 msgr2=0x564a53054160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:42.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.276+0000 7fc872192b80 1 -- 172.21.15.70:0/394555253 shutdown_connections 2024-06-04T05:19:42.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.276+0000 7fc872192b80 1 -- 172.21.15.70:0/394555253 wait complete. 2024-06-04T05:19:42.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.277+0000 7fc872192b80 1 -- start start 2024-06-04T05:19:42.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.277+0000 7fc872192b80 1 Processor -- start 2024-06-04T05:19:42.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.277+0000 7fc872192b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a53058ed0 0x564a53067d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.277+0000 7fc872192b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a53059880 0x564a530682b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.277+0000 7fc872192b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a5305d8c0 0x564a53068830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.277+0000 7fc872192b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc8601acd20 con 0x564a5305d8c0 2024-06-04T05:19:42.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.277+0000 7fc872192b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc8601ac650 con 0x564a53058ed0 2024-06-04T05:19:42.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.277+0000 7fc872192b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc8601a9840 con 0x564a53059880 2024-06-04T05:19:42.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.277+0000 7fc872192b80 10 client.0 _ll_register_callbacks cb 0x564a52f3ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:42.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.277+0000 7fc865e9a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a5305d8c0 0x564a53068830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:42.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.277+0000 7fc865e9a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a5305d8c0 0x564a53068830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50658/0 (socket says 172.21.15.70:50658) 2024-06-04T05:19:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.277+0000 7fc865e9a700 1 -- 172.21.15.70:0/3196171522 learned_addr learned my addr 172.21.15.70:0/3196171522 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.278+0000 7fc865699700 1 --2- 172.21.15.70:0/3196171522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a53058ed0 0x564a53067d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.278+0000 7fc864e98700 1 --2- 172.21.15.70:0/3196171522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a53059880 0x564a530682b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.278+0000 7fc865e9a700 1 -- 172.21.15.70:0/3196171522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a53059880 msgr2=0x564a530682b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.278+0000 7fc865e9a700 1 --2- 172.21.15.70:0/3196171522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a53059880 0x564a530682b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.278+0000 7fc865e9a700 1 -- 172.21.15.70:0/3196171522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a53058ed0 msgr2=0x564a53067d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.278+0000 7fc865e9a700 1 --2- 172.21.15.70:0/3196171522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a53058ed0 0x564a53067d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.278+0000 7fc865e9a700 1 -- 172.21.15.70:0/3196171522 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a530dd790 con 0x564a5305d8c0 2024-06-04T05:19:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.278+0000 7fc865e9a700 1 --2- 172.21.15.70:0/3196171522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a5305d8c0 0x564a53068830 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fc858009fd0 tx=0x7fc8580051e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55196c0d1ba540a5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.278+0000 7fc848ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.278+0000 7fc848ff9700 1 -- 172.21.15.70:0/3196171522 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc858030020 con 0x564a5305d8c0 2024-06-04T05:19:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.278+0000 7fc848ff9700 1 -- 172.21.15.70:0/3196171522 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7fc85800f040 con 0x564a5305d8c0 2024-06-04T05:19:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.278+0000 7fc872192b80 1 -- 172.21.15.70:0/3196171522 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc850010590 con 0x564a5305d8c0 2024-06-04T05:19:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.278+0000 7fc848ff9700 1 -- 172.21.15.70:0/3196171522 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7fc85801f030 con 0x564a5305d8c0 2024-06-04T05:19:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.279+0000 7fc872192b80 10 client.21243 fetch_fsmap learned FSMap version 60 2024-06-04T05:19:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.279+0000 7fc872192b80 1 -- 172.21.15.70:0/3196171522 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x564a530de9e0 con 0x564a5305d8c0 2024-06-04T05:19:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.279+0000 7fc848ff9700 1 -- 172.21.15.70:0/3196171522 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc850010590 con 0x564a5305d8c0 2024-06-04T05:19:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.279+0000 7fc848ff9700 1 -- 172.21.15.70:0/3196171522 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 60) ==== 8358+0+0 (secure 0 0 0) 0x7fc858011070 con 0x564a5305d8c0 2024-06-04T05:19:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.279+0000 7fc872192b80 10 client.21243 fetch_fsmap finished waiting for FSMap version 60 2024-06-04T05:19:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.279+0000 7fc872192b80 10 client.21243 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:19:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.279+0000 7fc872192b80 20 client.21243 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.279+0000 7fc872192b80 1 --2- 172.21.15.70:0/3196171522 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564a53072880 0x564a530e10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.279+0000 7fc872192b80 4 client.21243 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "da6cd7e7", "release": true, "await": true}] 2024-06-04T05:19:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.279+0000 7fc872192b80 1 -- 172.21.15.70:0/3196171522 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "da6cd7e7", "release": true, "await": true}) -- 0x564a52fb5ce0 con 0x564a53072880 2024-06-04T05:19:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.279+0000 7fc865699700 1 --2- 172.21.15.70:0/3196171522 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564a53072880 0x564a530e10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.280+0000 7fc848ff9700 10 client.21243 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.280+0000 7fc865699700 1 --2- 172.21.15.70:0/3196171522 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564a53072880 0x564a530e10a0 crc :-1 s=READY pgs=478 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=173733be756fca08 server_cookie=678123f680bd3b2b in_seq=0 out_seq=0 2024-06-04T05:19:42.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.312+0000 7fc848ff9700 1 -- 172.21.15.70:0/3196171522 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x564a52fb5ce0 con 0x564a53072880 2024-06-04T05:19:42.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.312+0000 7fc848ff9700 10 client.21243 handle_command_reply: tid=0 2024-06-04T05:19:42.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.312+0000 7fc872192b80 1 client.21243 shutdown 2024-06-04T05:19:42.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.312+0000 7fc872192b80 1 -- 172.21.15.70:0/3196171522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a5305d8c0 msgr2=0x564a53068830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:42.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.312+0000 7fc872192b80 1 --2- 172.21.15.70:0/3196171522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a5305d8c0 0x564a53068830 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fc858009fd0 tx=0x7fc8580051e0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.312+0000 7fc872192b80 1 -- 172.21.15.70:0/3196171522 shutdown_connections 2024-06-04T05:19:42.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.312+0000 7fc872192b80 1 --2- 172.21.15.70:0/3196171522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564a5305d8c0 0x564a53068830 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.312+0000 7fc872192b80 1 --2- 172.21.15.70:0/3196171522 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x564a53072880 0x564a530e10a0 crc :-1 s=READY pgs=478 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.312+0000 7fc848ff9700 0 client.21243 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:42.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.313+0000 7fc872192b80 1 --2- 172.21.15.70:0/3196171522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564a53058ed0 0x564a53067d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.313+0000 7fc872192b80 1 --2- 172.21.15.70:0/3196171522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564a53059880 0x564a530682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.313+0000 7fc872192b80 1 -- 172.21.15.70:0/3196171522 >> 172.21.15.70:0/3196171522 conn(0x564a52f82180 msgr2=0x564a5305c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:42.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.313+0000 7fc872192b80 1 -- 172.21.15.70:0/3196171522 shutdown_connections 2024-06-04T05:19:42.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.313+0000 7fc872192b80 1 -- 172.21.15.70:0/3196171522 wait complete. 2024-06-04T05:19:42.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.313+0000 7fc872192b80 20 client.21243 trim_cache size 0 max 16384 2024-06-04T05:19:42.316 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:19:42.317 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 60, 2024-06-04T05:19:42.317 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:19:42.317 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 72, 2024-06-04T05:19:42.317 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:19:42.317 INFO:teuthology.orchestra.run.smithi070.stdout: "da6cd7e7": { 2024-06-04T05:19:42.317 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 72, 2024-06-04T05:19:42.317 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:19:42.317 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:19:42.317 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:19:42.317 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:19:42.317 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:19:42.317 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:19:42.317 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 145.8, 2024-06-04T05:19:42.318 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:19:42.318 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:19:42.318 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:19:42.318 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:19:42.318 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:19:42.318 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:19:42.318 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:19:42.318 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:19:42.318 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:19:42.318 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:19:42.318 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:19:42.318 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:19:42.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.314+0000 7fc8678fd700 1 -- 172.21.15.70:0/1199684983 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc84c0b28d0 msgr2=0x7fc84c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:42.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.314+0000 7fc8678fd700 1 --2- 172.21.15.70:0/1199684983 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc84c0b28d0 0x7fc84c0b4d90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc86019ab60 tx=0x7fc85c00a680 comp rx=0 tx=0).stop 2024-06-04T05:19:42.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.314+0000 7fc8678fd700 1 -- 172.21.15.70:0/1199684983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc86006c470 msgr2=0x7fc86019fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:42.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.314+0000 7fc8678fd700 1 --2- 172.21.15.70:0/1199684983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc86006c470 0x7fc86019fda0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fc85000b530 tx=0x7fc85000b8f0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.314+0000 7fc8678fd700 1 -- 172.21.15.70:0/1199684983 shutdown_connections 2024-06-04T05:19:42.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.314+0000 7fc8678fd700 1 --2- 172.21.15.70:0/1199684983 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc84c0b28d0 0x7fc84c0b4d90 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.314+0000 7fc8678fd700 1 --2- 172.21.15.70:0/1199684983 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8601095a0 0x7fc860199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.314+0000 7fc8678fd700 1 --2- 172.21.15.70:0/1199684983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc86006bab0 0x7fc86019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.314+0000 7fc8678fd700 1 --2- 172.21.15.70:0/1199684983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc86006c470 0x7fc86019fda0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:42.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.314+0000 7fc8678fd700 1 -- 172.21.15.70:0/1199684983 >> 172.21.15.70:0/1199684983 conn(0x7fc8600793b0 msgr2=0x7fc860101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:42.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.314+0000 7fc8678fd700 1 -- 172.21.15.70:0/1199684983 shutdown_connections 2024-06-04T05:19:42.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:19:42.314+0000 7fc8678fd700 1 -- 172.21.15.70:0/1199684983 wait complete. 2024-06-04T05:19:42.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63792 6.45 MB/sec execute 394 sec latency 19682.085 ms 2024-06-04T05:19:43.131 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'da6cd7e7', total seconds elapsed: 31.8 2024-06-04T05:19:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:43 smithi112 ceph-mon[36453]: pgmap v709: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 321 KiB/s wr, 87 op/s 2024-06-04T05:19:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:43 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:19:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:43 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:19:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:43 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:19:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:43 smithi098 ceph-mon[35569]: pgmap v709: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 321 KiB/s wr, 87 op/s 2024-06-04T05:19:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:43 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:19:43.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:43 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:19:43.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:43 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:19:43.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 63987 6.47 MB/sec execute 395 sec latency 54.243 ms 2024-06-04T05:19:43.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:43 smithi070 ceph-mon[32631]: pgmap v709: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 321 KiB/s wr, 87 op/s 2024-06-04T05:19:43.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:43 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:19:43.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:43 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:19:43.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:43 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:19:44.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:44 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:19:44.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:44 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:19:44.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:44 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:19:44.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:44 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:19:44.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:44 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:19:44.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:44 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:19:44.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 64207 6.49 MB/sec execute 396 sec latency 55.488 ms 2024-06-04T05:19:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:45 smithi112 ceph-mon[36453]: pgmap v710: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 31 KiB/s wr, 15 op/s 2024-06-04T05:19:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:45 smithi098 ceph-mon[35569]: pgmap v710: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 31 KiB/s wr, 15 op/s 2024-06-04T05:19:45.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 64402 6.49 MB/sec execute 397 sec latency 48.970 ms 2024-06-04T05:19:45.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:45 smithi070 ceph-mon[32631]: pgmap v710: 97 pgs: 97 active+clean; 918 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 31 KiB/s wr, 15 op/s 2024-06-04T05:19:46.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:46 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:19:46.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:46 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:19:46.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 64532 6.49 MB/sec execute 398 sec latency 53.288 ms 2024-06-04T05:19:46.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:46 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:19:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:47 smithi112 ceph-mon[36453]: pgmap v711: 97 pgs: 97 active+clean; 933 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.9 MiB/s wr, 103 op/s 2024-06-04T05:19:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:47 smithi098 ceph-mon[35569]: pgmap v711: 97 pgs: 97 active+clean; 933 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.9 MiB/s wr, 103 op/s 2024-06-04T05:19:47.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 64733 6.52 MB/sec execute 399 sec latency 55.225 ms 2024-06-04T05:19:47.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:47 smithi070 ceph-mon[32631]: pgmap v711: 97 pgs: 97 active+clean; 933 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.9 MiB/s wr, 103 op/s 2024-06-04T05:19:48.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 64955 6.54 MB/sec execute 400 sec latency 60.196 ms 2024-06-04T05:19:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:49 smithi112 ceph-mon[36453]: pgmap v712: 97 pgs: 97 active+clean; 933 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 1.9 MiB/s wr, 98 op/s 2024-06-04T05:19:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:49 smithi098 ceph-mon[35569]: pgmap v712: 97 pgs: 97 active+clean; 933 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 1.9 MiB/s wr, 98 op/s 2024-06-04T05:19:49.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 65147 6.55 MB/sec execute 401 sec latency 59.083 ms 2024-06-04T05:19:49.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:49 smithi070 ceph-mon[32631]: pgmap v712: 97 pgs: 97 active+clean; 933 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 1.9 MiB/s wr, 98 op/s 2024-06-04T05:19:50.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 65284 6.54 MB/sec execute 402 sec latency 54.311 ms 2024-06-04T05:19:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:51 smithi112 ceph-mon[36453]: pgmap v713: 97 pgs: 97 active+clean; 933 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 1.9 MiB/s wr, 98 op/s 2024-06-04T05:19:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:51 smithi098 ceph-mon[35569]: pgmap v713: 97 pgs: 97 active+clean; 933 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 1.9 MiB/s wr, 98 op/s 2024-06-04T05:19:51.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 65436 6.55 MB/sec execute 403 sec latency 63.411 ms 2024-06-04T05:19:51.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:51 smithi070 ceph-mon[32631]: pgmap v713: 97 pgs: 97 active+clean; 933 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 1.9 MiB/s wr, 98 op/s 2024-06-04T05:19:52.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 65619 6.56 MB/sec execute 404 sec latency 68.005 ms 2024-06-04T05:19:53.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.227+0000 7f906ff46640 1 -- 172.21.15.70:0/3857169822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9068102a30 msgr2=0x7f9068102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.227+0000 7f906ff46640 1 --2- 172.21.15.70:0/3857169822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9068102a30 0x7f9068102eb0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f9058009930 tx=0x7f905802a420 comp rx=0 tx=0).stop 2024-06-04T05:19:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.227+0000 7f906ff46640 1 -- 172.21.15.70:0/3857169822 shutdown_connections 2024-06-04T05:19:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.227+0000 7f906ff46640 1 --2- 172.21.15.70:0/3857169822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f90681033f0 0x7f906810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.227+0000 7f906ff46640 1 --2- 172.21.15.70:0/3857169822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9068102a30 0x7f9068102eb0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.227+0000 7f906ff46640 1 --2- 172.21.15.70:0/3857169822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9068108a40 0x7f9068108e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.227+0000 7f906ff46640 1 -- 172.21.15.70:0/3857169822 >> 172.21.15.70:0/3857169822 conn(0x7f9068079ee0 msgr2=0x7f906807a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.227+0000 7f906ff46640 1 -- 172.21.15.70:0/3857169822 shutdown_connections 2024-06-04T05:19:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.227+0000 7f906ff46640 1 -- 172.21.15.70:0/3857169822 wait complete. 2024-06-04T05:19:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.228+0000 7f906ff46640 1 Processor -- start 2024-06-04T05:19:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.228+0000 7f906ff46640 1 -- start start 2024-06-04T05:19:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.228+0000 7f906ff46640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9068102a30 0x7f9068074d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.228+0000 7f906ff46640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f90681033f0 0x7f9068075290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.228+0000 7f906ff46640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9068108a40 0x7f90680757d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.228+0000 7f906ff46640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9068112bb0 con 0x7f9068102a30 2024-06-04T05:19:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.228+0000 7f906ff46640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9068112a30 con 0x7f9068108a40 2024-06-04T05:19:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.228+0000 7f906ff46640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9068112d30 con 0x7f90681033f0 2024-06-04T05:19:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.228+0000 7f906dce2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9068102a30 0x7f9068074d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.228+0000 7f906dce2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9068102a30 0x7f9068074d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49704/0 (socket says 172.21.15.70:49704) 2024-06-04T05:19:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.228+0000 7f906dce2640 1 -- 172.21.15.70:0/3917078269 learned_addr learned my addr 172.21.15.70:0/3917078269 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.228+0000 7f906d4e1640 1 --2- 172.21.15.70:0/3917078269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f90681033f0 0x7f9068075290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.228+0000 7f906e4e3640 1 --2- 172.21.15.70:0/3917078269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9068108a40 0x7f90680757d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.229+0000 7f906dce2640 1 -- 172.21.15.70:0/3917078269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f90681033f0 msgr2=0x7f9068075290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.229+0000 7f906dce2640 1 --2- 172.21.15.70:0/3917078269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f90681033f0 0x7f9068075290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.229+0000 7f906dce2640 1 -- 172.21.15.70:0/3917078269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9068108a40 msgr2=0x7f90680757d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.229+0000 7f906dce2640 1 --2- 172.21.15.70:0/3917078269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9068108a40 0x7f90680757d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.229+0000 7f906dce2640 1 -- 172.21.15.70:0/3917078269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90681a7630 con 0x7f9068102a30 2024-06-04T05:19:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.229+0000 7f906e4e3640 1 --2- 172.21.15.70:0/3917078269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9068108a40 0x7f90680757d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:19:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.229+0000 7f906dce2640 1 --2- 172.21.15.70:0/3917078269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9068102a30 0x7f9068074d50 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f906400d950 tx=0x7f906400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.229+0000 7f905effd640 1 -- 172.21.15.70:0/3917078269 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9064017030 con 0x7f9068102a30 2024-06-04T05:19:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.229+0000 7f906ff46640 1 -- 172.21.15.70:0/3917078269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90681a7920 con 0x7f9068102a30 2024-06-04T05:19:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.229+0000 7f905effd640 1 -- 172.21.15.70:0/3917078269 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9064004020 con 0x7f9068102a30 2024-06-04T05:19:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.229+0000 7f905effd640 1 -- 172.21.15.70:0/3917078269 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90640055d0 con 0x7f9068102a30 2024-06-04T05:19:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.229+0000 7f906ff46640 1 -- 172.21.15.70:0/3917078269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90681a7e90 con 0x7f9068102a30 2024-06-04T05:19:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.232+0000 7f9072c67740 1 Processor -- start 2024-06-04T05:19:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.232+0000 7f9072c67740 1 -- start start 2024-06-04T05:19:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.233+0000 7f9072c67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd2f820 0x5635ebd2fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.233+0000 7f9072c67740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd30140 0x5635ebd3a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.233+0000 7f9072c67740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd3aa00 0x5635ebd3cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.233+0000 7f9072c67740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5635ebca1c30 con 0x5635ebd3aa00 2024-06-04T05:19:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.233+0000 7f9072c67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5635ebc917d0 con 0x5635ebd2f820 2024-06-04T05:19:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.233+0000 7f9072c67740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5635ebc95810 con 0x5635ebd30140 2024-06-04T05:19:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.233+0000 7f905effd640 1 -- 172.21.15.70:0/3917078269 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f9064009a30 con 0x7f9068102a30 2024-06-04T05:19:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.233+0000 7f906e4e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd2f820 0x5635ebd2fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.233+0000 7f906d4e1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd3aa00 0x5635ebd3cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.233+0000 7f906e4e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd2f820 0x5635ebd2fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:42960/0 (socket says 172.21.15.70:42960) 2024-06-04T05:19:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.233+0000 7f906e4e3640 1 -- 172.21.15.70:0/2404864492 learned_addr learned my addr 172.21.15.70:0/2404864492 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.233+0000 7f906e4e3640 1 -- 172.21.15.70:0/2404864492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd30140 msgr2=0x5635ebd3a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.233+0000 7f906dce2640 1 --2- 172.21.15.70:0/2404864492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd30140 0x5635ebd3a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.233+0000 7f906e4e3640 1 --2- 172.21.15.70:0/2404864492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd30140 0x5635ebd3a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.233+0000 7f906e4e3640 1 -- 172.21.15.70:0/2404864492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd3aa00 msgr2=0x5635ebd3cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.233+0000 7f906e4e3640 1 --2- 172.21.15.70:0/2404864492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd3aa00 0x5635ebd3cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.233+0000 7f906e4e3640 1 -- 172.21.15.70:0/2404864492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635eb9a6210 con 0x5635ebd2f820 2024-06-04T05:19:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.234+0000 7f906d4e1640 1 --2- 172.21.15.70:0/2404864492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd3aa00 0x5635ebd3cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:19:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.234+0000 7f906dce2640 1 --2- 172.21.15.70:0/2404864492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd30140 0x5635ebd3a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:19:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.234+0000 7f906e4e3640 1 --2- 172.21.15.70:0/2404864492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd2f820 0x5635ebd2fc00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f90600113e0 tx=0x7f90600118a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b26cddd949013195 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.234+0000 7f905effd640 1 --2- 172.21.15.70:0/3917078269 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f90540b2a10 0x7f90540b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.234+0000 7f906dce2640 1 --2- 172.21.15.70:0/3917078269 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f90540b2a10 0x7f90540b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.234+0000 7f9053fff640 1 -- 172.21.15.70:0/2404864492 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9060012770 con 0x5635ebd2f820 2024-06-04T05:19:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.234+0000 7f9053fff640 1 -- 172.21.15.70:0/2404864492 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9060012da0 con 0x5635ebd2f820 2024-06-04T05:19:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.234+0000 7f9053fff640 1 -- 172.21.15.70:0/2404864492 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906000fdd0 con 0x5635ebd2f820 2024-06-04T05:19:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.234+0000 7f905effd640 1 -- 172.21.15.70:0/3917078269 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f90640e1ff0 con 0x7f9068102a30 2024-06-04T05:19:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.235+0000 7f9072c67740 1 -- 172.21.15.70:0/2404864492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd2f820 msgr2=0x5635ebd2fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.235+0000 7f906dce2640 1 --2- 172.21.15.70:0/3917078269 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f90540b2a10 0x7f90540b4ed0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f90640df7c0 tx=0x7f9064000bc0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.235+0000 7f9072c67740 1 --2- 172.21.15.70:0/2404864492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd2f820 0x5635ebd2fc00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f90600113e0 tx=0x7f90600118a0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.235+0000 7f9072c67740 1 -- 172.21.15.70:0/2404864492 shutdown_connections 2024-06-04T05:19:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.235+0000 7f9072c67740 1 --2- 172.21.15.70:0/2404864492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd3aa00 0x5635ebd3cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.235+0000 7f9072c67740 1 --2- 172.21.15.70:0/2404864492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd30140 0x5635ebd3a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.235+0000 7f9072c67740 1 --2- 172.21.15.70:0/2404864492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd2f820 0x5635ebd2fc00 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.236+0000 7f9072c67740 1 -- 172.21.15.70:0/2404864492 >> 172.21.15.70:0/2404864492 conn(0x5635ebd2bd30 msgr2=0x5635ebd2c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.236+0000 7f9072c67740 1 -- 172.21.15.70:0/2404864492 shutdown_connections 2024-06-04T05:19:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.236+0000 7f9072c67740 1 -- 172.21.15.70:0/2404864492 wait complete. 2024-06-04T05:19:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.236+0000 7f9072c67740 1 -- start start 2024-06-04T05:19:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.236+0000 7f9072c67740 1 Processor -- start 2024-06-04T05:19:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.237+0000 7f9072c67740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd2f820 0x5635ebd379f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.237+0000 7f906e4e3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd2f820 0x5635ebd379f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.237+0000 7f906e4e3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd2f820 0x5635ebd379f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49730/0 (socket says 172.21.15.70:49730) 2024-06-04T05:19:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.237+0000 7f906e4e3640 1 -- 172.21.15.70:0/4249816612 learned_addr learned my addr 172.21.15.70:0/4249816612 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.237+0000 7f9072c67740 1 --2- 172.21.15.70:0/4249816612 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd30140 0x5635ebd37f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.237+0000 7f9072c67740 1 --2- 172.21.15.70:0/4249816612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd3aa00 0x5635ebd38470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.237+0000 7f9072c67740 1 -- 172.21.15.70:0/4249816612 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5635eb977790 con 0x5635ebd2f820 2024-06-04T05:19:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.237+0000 7f9072c67740 1 -- 172.21.15.70:0/4249816612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5635ebc4d890 con 0x5635ebd3aa00 2024-06-04T05:19:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.237+0000 7f9072c67740 1 -- 172.21.15.70:0/4249816612 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5635ebd3d630 con 0x5635ebd30140 2024-06-04T05:19:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.237+0000 7f906d4e1640 1 --2- 172.21.15.70:0/4249816612 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd30140 0x5635ebd37f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.238+0000 7f906e4e3640 1 -- 172.21.15.70:0/4249816612 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd30140 msgr2=0x5635ebd37f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.238+0000 7f906e4e3640 1 --2- 172.21.15.70:0/4249816612 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd30140 0x5635ebd37f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.238+0000 7f906e4e3640 1 -- 172.21.15.70:0/4249816612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd3aa00 msgr2=0x5635ebd38470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:19:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.238+0000 7f906e4e3640 1 --2- 172.21.15.70:0/4249816612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd3aa00 0x5635ebd38470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.238+0000 7f906e4e3640 1 -- 172.21.15.70:0/4249816612 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635eba72620 con 0x5635ebd2f820 2024-06-04T05:19:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.238+0000 7f9072c67740 10 client.0 _ll_register_callbacks cb 0x5635eb9a5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.238+0000 7f906e4e3640 1 --2- 172.21.15.70:0/4249816612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd2f820 0x5635ebd379f0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f9060006010 tx=0x7f906000fd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae3e52a2424f88bf server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.238+0000 7f9072c67740 1 -- 172.21.15.70:0/4249816612 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9060012da0 con 0x5635ebd2f820 2024-06-04T05:19:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.238+0000 7f90517fa640 10 client.21255 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.239+0000 7f90517fa640 1 -- 172.21.15.70:0/4249816612 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9060032050 con 0x5635ebd2f820 2024-06-04T05:19:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.239+0000 7f90517fa640 1 -- 172.21.15.70:0/4249816612 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f9060012570 con 0x5635ebd2f820 2024-06-04T05:19:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.239+0000 7f90517fa640 1 -- 172.21.15.70:0/4249816612 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f9060028070 con 0x5635ebd2f820 2024-06-04T05:19:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.239+0000 7f9072c67740 10 client.21255 fetch_fsmap learned FSMap version 61 2024-06-04T05:19:53.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.239+0000 7f9072c67740 1 -- 172.21.15.70:0/4249816612 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({fsmap=61}) -- 0x5635ebd519e0 con 0x5635ebd2f820 2024-06-04T05:19:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.240+0000 7f90517fa640 1 -- 172.21.15.70:0/4249816612 <== mon.0 v2:172.21.15.70:3300/0 4 ==== fsmap(e 61) ==== 8354+0+0 (secure 0 0 0) 0x7f9060035020 con 0x5635ebd2f820 2024-06-04T05:19:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.240+0000 7f9072c67740 10 client.21255 fetch_fsmap finished waiting for FSMap version 61 2024-06-04T05:19:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.240+0000 7f9072c67740 10 client.21255 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:19:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.240+0000 7f9072c67740 20 client.21255 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.240+0000 7f9072c67740 1 --2- 172.21.15.70:0/4249816612 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5635ebd52d90 0x5635ebd55180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.240+0000 7f9072c67740 4 client.21255 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:19:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.240+0000 7f9072c67740 1 -- 172.21.15.70:0/4249816612 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635ebca3b80 con 0x5635ebd52d90 2024-06-04T05:19:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.240+0000 7f906d4e1640 1 --2- 172.21.15.70:0/4249816612 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5635ebd52d90 0x5635ebd55180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.240+0000 7f90517fa640 10 client.21255 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.240+0000 7f906d4e1640 1 --2- 172.21.15.70:0/4249816612 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5635ebd52d90 0x5635ebd55180 crc :-1 s=READY pgs=479 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78d3720ea07e6985 server_cookie=a5d72b6d8b6e4d74 in_seq=0 out_seq=0 2024-06-04T05:19:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.241+0000 7f90517fa640 1 -- 172.21.15.70:0/4249816612 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5635ebca3b80 con 0x5635ebd52d90 2024-06-04T05:19:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.241+0000 7f90517fa640 10 client.21255 handle_command_reply: tid=0 2024-06-04T05:19:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.241+0000 7f9072c67740 1 client.21255 shutdown 2024-06-04T05:19:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.242+0000 7f9072c67740 1 -- 172.21.15.70:0/4249816612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd2f820 msgr2=0x5635ebd379f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.242+0000 7f9072c67740 1 --2- 172.21.15.70:0/4249816612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd2f820 0x5635ebd379f0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f9060006010 tx=0x7f906000fd80 comp rx=0 tx=0).stop 2024-06-04T05:19:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.242+0000 7f9072c67740 1 -- 172.21.15.70:0/4249816612 shutdown_connections 2024-06-04T05:19:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.242+0000 7f9072c67740 1 --2- 172.21.15.70:0/4249816612 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5635ebd52d90 0x5635ebd55180 crc :-1 s=READY pgs=479 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.242+0000 7f90517fa640 0 client.21255 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.242+0000 7f9072c67740 1 --2- 172.21.15.70:0/4249816612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd3aa00 0x5635ebd38470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.242+0000 7f9072c67740 1 --2- 172.21.15.70:0/4249816612 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd30140 0x5635ebd37f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.242+0000 7f9072c67740 1 --2- 172.21.15.70:0/4249816612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd2f820 0x5635ebd379f0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.242+0000 7f9072c67740 1 -- 172.21.15.70:0/4249816612 >> 172.21.15.70:0/4249816612 conn(0x5635ebd2bd30 msgr2=0x5635ebd2afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.242+0000 7f9072c67740 1 -- 172.21.15.70:0/4249816612 shutdown_connections 2024-06-04T05:19:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.242+0000 7f9072c67740 1 -- 172.21.15.70:0/4249816612 wait complete. 2024-06-04T05:19:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.243+0000 7f9072c67740 20 client.21255 trim_cache size 0 max 16384 2024-06-04T05:19:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.271+0000 7f9072c67740 1 Processor -- start 2024-06-04T05:19:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.271+0000 7f9072c67740 1 -- start start 2024-06-04T05:19:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.272+0000 7f9072c67740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd2f820 0x5635ebd318e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.272+0000 7f9072c67740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd30140 0x5635ebd37f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.272+0000 7f9072c67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd3aa00 0x5635ebd38480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.272+0000 7f9072c67740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f90681ac4f0 con 0x5635ebd30140 2024-06-04T05:19:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.272+0000 7f9072c67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f90681ae030 con 0x5635ebd3aa00 2024-06-04T05:19:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.272+0000 7f9072c67740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f90681acbc0 con 0x5635ebd2f820 2024-06-04T05:19:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.272+0000 7f906e4e3640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd2f820 0x5635ebd318e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.272+0000 7f906e4e3640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd2f820 0x5635ebd318e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:42510/0 (socket says 172.21.15.70:42510) 2024-06-04T05:19:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.272+0000 7f906e4e3640 1 -- 172.21.15.70:0/243199407 learned_addr learned my addr 172.21.15.70:0/243199407 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.272+0000 7f906e4e3640 1 --2- 172.21.15.70:0/243199407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd3aa00 0x5635ebd38480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.272+0000 7f906d4e1640 1 --2- 172.21.15.70:0/243199407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd30140 0x5635ebd37f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.272+0000 7f906e4e3640 1 -- 172.21.15.70:0/243199407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd3aa00 msgr2=0x5635ebd38480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.272+0000 7f906e4e3640 1 --2- 172.21.15.70:0/243199407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd3aa00 0x5635ebd38480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.272+0000 7f906e4e3640 1 -- 172.21.15.70:0/243199407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd30140 msgr2=0x5635ebd37f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.272+0000 7f906e4e3640 1 --2- 172.21.15.70:0/243199407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd30140 0x5635ebd37f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.272+0000 7f906e4e3640 1 -- 172.21.15.70:0/243199407 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635ebd327a0 con 0x5635ebd2f820 2024-06-04T05:19:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.273+0000 7f906e4e3640 1 --2- 172.21.15.70:0/243199407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd2f820 0x5635ebd318e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9060010010 tx=0x7f9060023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=992a409e654a31f8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.273+0000 7f90537fe640 1 -- 172.21.15.70:0/243199407 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9060032070 con 0x5635ebd2f820 2024-06-04T05:19:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.273+0000 7f90537fe640 1 -- 172.21.15.70:0/243199407 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9060012da0 con 0x5635ebd2f820 2024-06-04T05:19:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.273+0000 7f90537fe640 1 -- 172.21.15.70:0/243199407 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906002d750 con 0x5635ebd2f820 2024-06-04T05:19:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.274+0000 7f9072c67740 1 -- 172.21.15.70:0/243199407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd2f820 msgr2=0x5635ebd318e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.274+0000 7f9072c67740 1 --2- 172.21.15.70:0/243199407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd2f820 0x5635ebd318e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9060010010 tx=0x7f9060023000 comp rx=0 tx=0).stop 2024-06-04T05:19:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.274+0000 7f9072c67740 1 -- 172.21.15.70:0/243199407 shutdown_connections 2024-06-04T05:19:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.274+0000 7f9072c67740 1 --2- 172.21.15.70:0/243199407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd3aa00 0x5635ebd38480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.274+0000 7f9072c67740 1 --2- 172.21.15.70:0/243199407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd30140 0x5635ebd37f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.274+0000 7f9072c67740 1 --2- 172.21.15.70:0/243199407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd2f820 0x5635ebd318e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.274+0000 7f9072c67740 1 -- 172.21.15.70:0/243199407 >> 172.21.15.70:0/243199407 conn(0x5635ebd2bd30 msgr2=0x5635ebd311d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.275+0000 7f9072c67740 1 -- 172.21.15.70:0/243199407 shutdown_connections 2024-06-04T05:19:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.275+0000 7f9072c67740 1 -- 172.21.15.70:0/243199407 wait complete. 2024-06-04T05:19:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.276+0000 7f9072c67740 1 -- start start 2024-06-04T05:19:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.276+0000 7f9072c67740 1 Processor -- start 2024-06-04T05:19:53.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.277+0000 7f9072c67740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd2f820 0x5635ebd4b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.277+0000 7f906e4e3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd2f820 0x5635ebd4b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.277+0000 7f9072c67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd30140 0x5635ebd46790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.277+0000 7f9072c67740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd3aa00 0x5635ebd46cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.277+0000 7f9072c67740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f90681ad960 con 0x5635ebd2f820 2024-06-04T05:19:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.277+0000 7f9072c67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f90681ad290 con 0x5635ebd30140 2024-06-04T05:19:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.277+0000 7f9072c67740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f90681aa4f0 con 0x5635ebd3aa00 2024-06-04T05:19:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.278+0000 7f9072c67740 10 client.0 _ll_register_callbacks cb 0x7f9058009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:19:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.278+0000 7f906e4e3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd2f820 0x5635ebd4b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49744/0 (socket says 172.21.15.70:49744) 2024-06-04T05:19:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.278+0000 7f906e4e3640 1 -- 172.21.15.70:0/3758620564 learned_addr learned my addr 172.21.15.70:0/3758620564 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:19:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.278+0000 7f906d4e1640 1 --2- 172.21.15.70:0/3758620564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd30140 0x5635ebd46790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.278+0000 7f906e4e3640 1 -- 172.21.15.70:0/3758620564 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd3aa00 msgr2=0x5635ebd46cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-04T05:19:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.278+0000 7f906e4e3640 1 --2- 172.21.15.70:0/3758620564 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd3aa00 0x5635ebd46cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.278+0000 7f906e4e3640 1 -- 172.21.15.70:0/3758620564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd30140 msgr2=0x5635ebd46790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:19:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.278+0000 7f906e4e3640 1 --2- 172.21.15.70:0/3758620564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd30140 0x5635ebd46790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.278+0000 7f906e4e3640 1 -- 172.21.15.70:0/3758620564 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635ebd47210 con 0x5635ebd2f820 2024-06-04T05:19:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.279+0000 7f906e4e3640 1 --2- 172.21.15.70:0/3758620564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd2f820 0x5635ebd4b470 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f90600127a0 tx=0x7f90600127d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191647b52722f65a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:19:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.279+0000 7f90517fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:19:53.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.279+0000 7f90517fa640 1 -- 172.21.15.70:0/3758620564 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9060032040 con 0x5635ebd2f820 2024-06-04T05:19:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.279+0000 7f9072c67740 1 -- 172.21.15.70:0/3758620564 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9060012da0 con 0x5635ebd2f820 2024-06-04T05:19:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.279+0000 7f90517fa640 1 -- 172.21.15.70:0/3758620564 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8571+0+0 (secure 0 0 0) 0x7f906002d440 con 0x5635ebd2f820 2024-06-04T05:19:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.280+0000 7f90517fa640 1 -- 172.21.15.70:0/3758620564 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f906004b050 con 0x5635ebd2f820 2024-06-04T05:19:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.280+0000 7f9072c67740 10 client.21261 fetch_fsmap learned FSMap version 61 2024-06-04T05:19:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.280+0000 7f9072c67740 1 -- 172.21.15.70:0/3758620564 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x5635ebdbd370 con 0x5635ebd2f820 2024-06-04T05:19:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.280+0000 7f90517fa640 1 -- 172.21.15.70:0/3758620564 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9060012da0 con 0x5635ebd2f820 2024-06-04T05:19:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.281+0000 7f90517fa640 1 -- 172.21.15.70:0/3758620564 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 61) ==== 8354+0+0 (secure 0 0 0) 0x7f9060042070 con 0x5635ebd2f820 2024-06-04T05:19:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.281+0000 7f9072c67740 10 client.21261 fetch_fsmap finished waiting for FSMap version 61 2024-06-04T05:19:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.281+0000 7f9072c67740 10 client.21261 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:19:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.281+0000 7f9072c67740 20 client.21261 populate_metadata read hostname 'smithi070' 2024-06-04T05:19:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.281+0000 7f9072c67740 1 --2- 172.21.15.70:0/3758620564 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5635ebd52d90 0x5635ebdbfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:19:53.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.281+0000 7f9072c67740 4 client.21261 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:19:53.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.281+0000 7f9072c67740 1 -- 172.21.15.70:0/3758620564 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5635ebc9d360 con 0x5635ebd52d90 2024-06-04T05:19:53.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.281+0000 7f906d4e1640 1 --2- 172.21.15.70:0/3758620564 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5635ebd52d90 0x5635ebdbfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:19:53.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.282+0000 7f90517fa640 10 client.21261 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:53.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.282+0000 7f906d4e1640 1 --2- 172.21.15.70:0/3758620564 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5635ebd52d90 0x5635ebdbfaa0 crc :-1 s=READY pgs=480 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e9d647063e4fb13 server_cookie=8005ef598b344b3c in_seq=0 out_seq=0 2024-06-04T05:19:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.377+0000 7f90517fa640 1 -- 172.21.15.70:0/3758620564 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5635ebc9d360 con 0x5635ebd52d90 2024-06-04T05:19:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.377+0000 7f90517fa640 10 client.21261 handle_command_reply: tid=0 2024-06-04T05:19:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.377+0000 7f9072c67740 1 client.21261 shutdown 2024-06-04T05:19:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.377+0000 7f9072c67740 1 -- 172.21.15.70:0/3758620564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd2f820 msgr2=0x5635ebd4b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.377+0000 7f9072c67740 1 --2- 172.21.15.70:0/3758620564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd2f820 0x5635ebd4b470 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f90600127a0 tx=0x7f90600127d0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.377+0000 7f9072c67740 1 -- 172.21.15.70:0/3758620564 shutdown_connections 2024-06-04T05:19:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.377+0000 7f9072c67740 1 --2- 172.21.15.70:0/3758620564 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5635ebd52d90 0x5635ebdbfaa0 crc :-1 s=READY pgs=480 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.377+0000 7f9072c67740 1 --2- 172.21.15.70:0/3758620564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635ebd30140 0x5635ebd46790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.377+0000 7f90517fa640 0 client.21261 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:19:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.377+0000 7f9072c67740 1 --2- 172.21.15.70:0/3758620564 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5635ebd3aa00 0x5635ebd46cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.377+0000 7f9072c67740 1 --2- 172.21.15.70:0/3758620564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635ebd2f820 0x5635ebd4b470 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.377+0000 7f9072c67740 1 -- 172.21.15.70:0/3758620564 >> 172.21.15.70:0/3758620564 conn(0x5635ebd2bd30 msgr2=0x5635ebd39ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.378+0000 7f9072c67740 1 -- 172.21.15.70:0/3758620564 shutdown_connections 2024-06-04T05:19:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.378+0000 7f9072c67740 1 -- 172.21.15.70:0/3758620564 wait complete. 2024-06-04T05:19:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.378+0000 7f9072c67740 20 client.21261 trim_cache size 0 max 16384 2024-06-04T05:19:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:19:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:19:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:19:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.379+0000 7f906ff46640 1 -- 172.21.15.70:0/3917078269 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f90540b2a10 msgr2=0x7f90540b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.379+0000 7f906ff46640 1 --2- 172.21.15.70:0/3917078269 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f90540b2a10 0x7f90540b4ed0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f90640df7c0 tx=0x7f9064000bc0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.379+0000 7f906ff46640 1 -- 172.21.15.70:0/3917078269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9068102a30 msgr2=0x7f9068074d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:19:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.379+0000 7f906ff46640 1 --2- 172.21.15.70:0/3917078269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9068102a30 0x7f9068074d50 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f906400d950 tx=0x7f906400de10 comp rx=0 tx=0).stop 2024-06-04T05:19:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.379+0000 7f906ff46640 1 -- 172.21.15.70:0/3917078269 shutdown_connections 2024-06-04T05:19:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.379+0000 7f906ff46640 1 --2- 172.21.15.70:0/3917078269 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f90540b2a10 0x7f90540b4ed0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.379+0000 7f906ff46640 1 --2- 172.21.15.70:0/3917078269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9068108a40 0x7f90680757d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.379+0000 7f906ff46640 1 --2- 172.21.15.70:0/3917078269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f90681033f0 0x7f9068075290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.379+0000 7f906ff46640 1 --2- 172.21.15.70:0/3917078269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9068102a30 0x7f9068074d50 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:19:53.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.379+0000 7f906ff46640 1 -- 172.21.15.70:0/3917078269 >> 172.21.15.70:0/3917078269 conn(0x7f9068079ee0 msgr2=0x7f9068100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:19:53.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.379+0000 7f906ff46640 1 -- 172.21.15.70:0/3917078269 shutdown_connections 2024-06-04T05:19:53.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:19:53.380+0000 7f906ff46640 1 -- 172.21.15.70:0/3917078269 wait complete. 2024-06-04T05:19:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:53 smithi112 ceph-mon[36453]: pgmap v714: 97 pgs: 97 active+clean; 944 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 230 op/s 2024-06-04T05:19:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:53 smithi098 ceph-mon[35569]: pgmap v714: 97 pgs: 97 active+clean; 944 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 230 op/s 2024-06-04T05:19:53.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 65782 6.55 MB/sec execute 405 sec latency 111.381 ms 2024-06-04T05:19:53.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:53 smithi070 ceph-mon[32631]: pgmap v714: 97 pgs: 97 active+clean; 944 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 230 op/s 2024-06-04T05:19:54.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 65942 6.56 MB/sec execute 406 sec latency 129.518 ms 2024-06-04T05:19:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:55 smithi112 ceph-mon[36453]: pgmap v715: 97 pgs: 97 active+clean; 944 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.6 MiB/s wr, 220 op/s 2024-06-04T05:19:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:55 smithi098 ceph-mon[35569]: pgmap v715: 97 pgs: 97 active+clean; 944 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.6 MiB/s wr, 220 op/s 2024-06-04T05:19:55.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 66134 6.57 MB/sec execute 407 sec latency 53.776 ms 2024-06-04T05:19:55.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:55 smithi070 ceph-mon[32631]: pgmap v715: 97 pgs: 97 active+clean; 944 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.6 MiB/s wr, 220 op/s 2024-06-04T05:19:56.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:56 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:19:56.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:56 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:19:56.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 66310 6.58 MB/sec execute 408 sec latency 65.506 ms 2024-06-04T05:19:56.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:56 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:19:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:57 smithi112 ceph-mon[36453]: pgmap v716: 97 pgs: 97 active+clean; 950 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.2 MiB/s wr, 395 op/s 2024-06-04T05:19:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:57 smithi098 ceph-mon[35569]: pgmap v716: 97 pgs: 97 active+clean; 950 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.2 MiB/s wr, 395 op/s 2024-06-04T05:19:57.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 66439 6.57 MB/sec execute 409 sec latency 63.893 ms 2024-06-04T05:19:57.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:57 smithi070 ceph-mon[32631]: pgmap v716: 97 pgs: 97 active+clean; 950 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.2 MiB/s wr, 395 op/s 2024-06-04T05:19:58.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 66642 6.59 MB/sec execute 410 sec latency 67.562 ms 2024-06-04T05:19:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:19:59 smithi112 ceph-mon[36453]: pgmap v717: 97 pgs: 97 active+clean; 950 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 308 op/s 2024-06-04T05:19:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:19:59 smithi098 ceph-mon[35569]: pgmap v717: 97 pgs: 97 active+clean; 950 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 308 op/s 2024-06-04T05:19:59.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 66847 6.60 MB/sec execute 411 sec latency 82.747 ms 2024-06-04T05:19:59.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:19:59 smithi070 ceph-mon[32631]: pgmap v717: 97 pgs: 97 active+clean; 950 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 308 op/s 2024-06-04T05:20:00.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:00 smithi112 ceph-mon[36453]: overall HEALTH_OK 2024-06-04T05:20:00.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:00 smithi098 ceph-mon[35569]: overall HEALTH_OK 2024-06-04T05:20:00.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 67008 6.60 MB/sec execute 412 sec latency 62.272 ms 2024-06-04T05:20:00.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:00 smithi070 ceph-mon[32631]: overall HEALTH_OK 2024-06-04T05:20:01.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:01 smithi112 ceph-mon[36453]: pgmap v718: 97 pgs: 97 active+clean; 950 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 308 op/s 2024-06-04T05:20:01.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:01 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:20:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:01 smithi098 ceph-mon[35569]: pgmap v718: 97 pgs: 97 active+clean; 950 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 308 op/s 2024-06-04T05:20:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:01 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:20:01.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 67145 6.60 MB/sec execute 413 sec latency 84.871 ms 2024-06-04T05:20:01.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:01 smithi070 ceph-mon[32631]: pgmap v718: 97 pgs: 97 active+clean; 950 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 308 op/s 2024-06-04T05:20:01.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:01 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:20:02.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 67327 6.62 MB/sec execute 414 sec latency 56.809 ms 2024-06-04T05:20:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:03 smithi112 ceph-mon[36453]: pgmap v719: 97 pgs: 97 active+clean; 959 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.8 MiB/s wr, 459 op/s 2024-06-04T05:20:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:03 smithi098 ceph-mon[35569]: pgmap v719: 97 pgs: 97 active+clean; 959 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.8 MiB/s wr, 459 op/s 2024-06-04T05:20:03.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 67478 6.62 MB/sec execute 415 sec latency 118.611 ms 2024-06-04T05:20:03.841 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:20:03.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:03 smithi070 ceph-mon[32631]: pgmap v719: 97 pgs: 97 active+clean; 959 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.8 MiB/s wr, 459 op/s 2024-06-04T05:20:04.443 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:20:04.669 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:04 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:20:04.669 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:04 smithi070 ceph-mon[32631]: osdmap e113: 12 total, 12 up, 12 in 2024-06-04T05:20:04.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:04 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:20:04.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:04 smithi112 ceph-mon[36453]: osdmap e113: 12 total, 12 up, 12 in 2024-06-04T05:20:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:04 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:20:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:04 smithi098 ceph-mon[35569]: osdmap e113: 12 total, 12 up, 12 in 2024-06-04T05:20:04.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 67651 6.62 MB/sec execute 416 sec latency 75.778 ms 2024-06-04T05:20:05.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:05 smithi098 ceph-mon[35569]: pgmap v721: 97 pgs: 97 active+clean; 959 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.2 MiB/s wr, 392 op/s 2024-06-04T05:20:05.829 INFO:tasks.workunit.client.0.smithi070.stdout: 10 67808 6.63 MB/sec execute 417 sec latency 111.236 ms 2024-06-04T05:20:05.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:05 smithi070 ceph-mon[32631]: pgmap v721: 97 pgs: 97 active+clean; 959 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.2 MiB/s wr, 392 op/s 2024-06-04T05:20:05.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:05 smithi112 ceph-mon[36453]: pgmap v721: 97 pgs: 97 active+clean; 959 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.2 MiB/s wr, 392 op/s 2024-06-04T05:20:06.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.508+0000 7f2ea3732640 1 -- 172.21.15.70:0/1837835326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e9c101a90 msgr2=0x7f2e9c101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:06.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.508+0000 7f2ea3732640 1 --2- 172.21.15.70:0/1837835326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e9c101a90 0x7f2e9c101e70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009900 tx=0x7f2e9002a480 comp rx=0 tx=0).stop 2024-06-04T05:20:06.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.508+0000 7f2ea3732640 1 -- 172.21.15.70:0/1837835326 shutdown_connections 2024-06-04T05:20:06.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.508+0000 7f2ea3732640 1 --2- 172.21.15.70:0/1837835326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e9c106e90 0x7f2e9c107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.508+0000 7f2ea3732640 1 --2- 172.21.15.70:0/1837835326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e9c1023b0 0x7f2e9c106870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.508+0000 7f2ea3732640 1 --2- 172.21.15.70:0/1837835326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e9c101a90 0x7f2e9c101e70 secure :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009900 tx=0x7f2e9002a480 comp rx=0 tx=0).stop 2024-06-04T05:20:06.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.508+0000 7f2ea3732640 1 -- 172.21.15.70:0/1837835326 >> 172.21.15.70:0/1837835326 conn(0x7f2e9c078b60 msgr2=0x7f2e9c078f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:06.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.509+0000 7f2ea3732640 1 -- 172.21.15.70:0/1837835326 shutdown_connections 2024-06-04T05:20:06.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.509+0000 7f2ea3732640 1 -- 172.21.15.70:0/1837835326 wait complete. 2024-06-04T05:20:06.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.510+0000 7f2ea3732640 1 Processor -- start 2024-06-04T05:20:06.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.510+0000 7f2ea3732640 1 -- start start 2024-06-04T05:20:06.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.511+0000 7f2ea3732640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e9c1023b0 0x7f2e9c115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.511+0000 7f2ea3732640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e9c106e90 0x7f2e9c10fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.511+0000 7f2ea3732640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e9c110320 0x7f2e9c1107d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.511+0000 7f2ea3732640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c1085a0 con 0x7f2e9c110320 2024-06-04T05:20:06.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.511+0000 7f2ea3732640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c108420 con 0x7f2e9c106e90 2024-06-04T05:20:06.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.511+0000 7f2ea3732640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c108720 con 0x7f2e9c1023b0 2024-06-04T05:20:06.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.511+0000 7f2ea14ce640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e9c1023b0 0x7f2e9c115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.511+0000 7f2ea1ccf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e9c110320 0x7f2e9c1107d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.511+0000 7f2ea0ccd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e9c106e90 0x7f2e9c10fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.511+0000 7f2ea1ccf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e9c110320 0x7f2e9c1107d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57552/0 (socket says 172.21.15.70:57552) 2024-06-04T05:20:06.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.511+0000 7f2ea1ccf640 1 -- 172.21.15.70:0/2383732611 learned_addr learned my addr 172.21.15.70:0/2383732611 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:06.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.511+0000 7f2ea0ccd640 1 -- 172.21.15.70:0/2383732611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e9c1023b0 msgr2=0x7f2e9c115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:06.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.511+0000 7f2ea0ccd640 1 --2- 172.21.15.70:0/2383732611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e9c1023b0 0x7f2e9c115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.511+0000 7f2ea0ccd640 1 -- 172.21.15.70:0/2383732611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e9c110320 msgr2=0x7f2e9c1107d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:06.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.511+0000 7f2ea0ccd640 1 --2- 172.21.15.70:0/2383732611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e9c110320 0x7f2e9c1107d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.511+0000 7f2ea0ccd640 1 -- 172.21.15.70:0/2383732611 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e9c06c470 con 0x7f2e9c106e90 2024-06-04T05:20:06.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.511+0000 7f2ea1ccf640 1 --2- 172.21.15.70:0/2383732611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e9c110320 0x7f2e9c1107d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:06.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.512+0000 7f2ea0ccd640 1 --2- 172.21.15.70:0/2383732611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e9c106e90 0x7f2e9c10fde0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400b420 tx=0x7f2e9400b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:06.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.512+0000 7f2e9a7fc640 1 -- 172.21.15.70:0/2383732611 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e94011830 con 0x7f2e9c106e90 2024-06-04T05:20:06.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.512+0000 7f2ea3732640 1 -- 172.21.15.70:0/2383732611 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e9c06c700 con 0x7f2e9c106e90 2024-06-04T05:20:06.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.512+0000 7f2ea3732640 1 -- 172.21.15.70:0/2383732611 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e9c06cca0 con 0x7f2e9c106e90 2024-06-04T05:20:06.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.512+0000 7f2e9a7fc640 1 -- 172.21.15.70:0/2383732611 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2e94016070 con 0x7f2e9c106e90 2024-06-04T05:20:06.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.513+0000 7f2e9a7fc640 1 -- 172.21.15.70:0/2383732611 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e94010ce0 con 0x7f2e9c106e90 2024-06-04T05:20:06.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.516+0000 7f2e9a7fc640 1 -- 172.21.15.70:0/2383732611 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f2e9401b460 con 0x7f2e9c106e90 2024-06-04T05:20:06.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.516+0000 7f2ea6450740 1 Processor -- start 2024-06-04T05:20:06.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.516+0000 7f2ea6450740 1 -- start start 2024-06-04T05:20:06.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.516+0000 7f2ea6450740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d186f820 0x5640d186fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.516+0000 7f2ea6450740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d1870140 0x5640d187a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.516+0000 7f2ea6450740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d187aa00 0x5640d187cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.516+0000 7f2ea6450740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5640d17e1c30 con 0x5640d187aa00 2024-06-04T05:20:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.516+0000 7f2ea6450740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5640d17d17d0 con 0x5640d1870140 2024-06-04T05:20:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.516+0000 7f2ea6450740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5640d17d5810 con 0x5640d186f820 2024-06-04T05:20:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.516+0000 7f2ea14ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d1870140 0x5640d187a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.516+0000 7f2ea1ccf640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d186f820 0x5640d186fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.516+0000 7f2ea1ccf640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d186f820 0x5640d186fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:33882/0 (socket says 172.21.15.70:33882) 2024-06-04T05:20:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.516+0000 7f2ea1ccf640 1 -- 172.21.15.70:0/1765579017 learned_addr learned my addr 172.21.15.70:0/1765579017 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.516+0000 7f2ea0ccd640 1 --2- 172.21.15.70:0/1765579017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d187aa00 0x5640d187cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.517+0000 7f2ea1ccf640 1 -- 172.21.15.70:0/1765579017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d1870140 msgr2=0x5640d187a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.517+0000 7f2ea1ccf640 1 --2- 172.21.15.70:0/1765579017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d1870140 0x5640d187a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.517+0000 7f2ea1ccf640 1 -- 172.21.15.70:0/1765579017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d187aa00 msgr2=0x5640d187cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.517+0000 7f2ea1ccf640 1 --2- 172.21.15.70:0/1765579017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d187aa00 0x5640d187cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.517+0000 7f2ea1ccf640 1 -- 172.21.15.70:0/1765579017 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640d14e6210 con 0x5640d186f820 2024-06-04T05:20:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.517+0000 7f2ea14ce640 1 --2- 172.21.15.70:0/1765579017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d1870140 0x5640d187a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.517+0000 7f2ea0ccd640 1 --2- 172.21.15.70:0/1765579017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d187aa00 0x5640d187cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:06.521 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.517+0000 7f2e9a7fc640 1 --2- 172.21.15.70:0/2383732611 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2e840b2b60 0x7f2e840b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.521 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.517+0000 7f2ea1ccf640 1 --2- 172.21.15.70:0/1765579017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d186f820 0x5640d186fc00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c0117e0 tx=0x7f2e8c011ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=920280b17c97f31 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:06.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.517+0000 7f2ea14ce640 1 --2- 172.21.15.70:0/2383732611 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2e840b2b60 0x7f2e840b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.517+0000 7f2e837fe640 1 -- 172.21.15.70:0/1765579017 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c00fb40 con 0x5640d186f820 2024-06-04T05:20:06.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.517+0000 7f2e837fe640 1 -- 172.21.15.70:0/1765579017 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2e8c00fce0 con 0x5640d186f820 2024-06-04T05:20:06.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.517+0000 7f2e837fe640 1 -- 172.21.15.70:0/1765579017 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c0126b0 con 0x5640d186f820 2024-06-04T05:20:06.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.518+0000 7f2e9a7fc640 1 -- 172.21.15.70:0/2383732611 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8615+0+0 (secure 0 0 0) 0x7f2e940e1a40 con 0x7f2e9c106e90 2024-06-04T05:20:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.518+0000 7f2ea6450740 1 -- 172.21.15.70:0/1765579017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d186f820 msgr2=0x5640d186fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.518+0000 7f2ea14ce640 1 --2- 172.21.15.70:0/2383732611 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2e840b2b60 0x7f2e840b5020 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009850 tx=0x7f2e90005fd0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.518+0000 7f2ea6450740 1 --2- 172.21.15.70:0/1765579017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d186f820 0x5640d186fc00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c0117e0 tx=0x7f2e8c011ca0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.519+0000 7f2ea6450740 1 -- 172.21.15.70:0/1765579017 shutdown_connections 2024-06-04T05:20:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.519+0000 7f2ea6450740 1 --2- 172.21.15.70:0/1765579017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d1870140 0x5640d187a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.519+0000 7f2ea6450740 1 --2- 172.21.15.70:0/1765579017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d187aa00 0x5640d187cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.519+0000 7f2ea6450740 1 --2- 172.21.15.70:0/1765579017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d186f820 0x5640d186fc00 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.519+0000 7f2ea6450740 1 -- 172.21.15.70:0/1765579017 >> 172.21.15.70:0/1765579017 conn(0x5640d186bd30 msgr2=0x5640d186c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.519+0000 7f2ea6450740 1 -- 172.21.15.70:0/1765579017 shutdown_connections 2024-06-04T05:20:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.519+0000 7f2ea6450740 1 -- 172.21.15.70:0/1765579017 wait complete. 2024-06-04T05:20:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.519+0000 7f2ea6450740 1 -- start start 2024-06-04T05:20:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.519+0000 7f2ea6450740 1 Processor -- start 2024-06-04T05:20:06.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.520+0000 7f2ea6450740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d18779c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.520+0000 7f2ea6450740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d1870140 0x5640d1877f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.520+0000 7f2ea6450740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d187aa00 0x5640d1878440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.520+0000 7f2ea6450740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5640d14b7790 con 0x5640d1870140 2024-06-04T05:20:06.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.520+0000 7f2ea6450740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5640d178d890 con 0x5640d186f820 2024-06-04T05:20:06.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.520+0000 7f2ea6450740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5640d187d630 con 0x5640d187aa00 2024-06-04T05:20:06.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.520+0000 7f2ea14ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d18779c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.520+0000 7f2ea14ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d18779c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:50016/0 (socket says 172.21.15.70:50016) 2024-06-04T05:20:06.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.520+0000 7f2ea14ce640 1 -- 172.21.15.70:0/1051975805 learned_addr learned my addr 172.21.15.70:0/1051975805 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:06.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.520+0000 7f2ea1ccf640 1 --2- 172.21.15.70:0/1051975805 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d187aa00 0x5640d1878440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.520+0000 7f2ea6450740 10 client.0 _ll_register_callbacks cb 0x5640d14e5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:06.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.521+0000 7f2ea0ccd640 1 --2- 172.21.15.70:0/1051975805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d1870140 0x5640d1877f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.521+0000 7f2ea14ce640 1 -- 172.21.15.70:0/1051975805 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d187aa00 msgr2=0x5640d1878440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:06.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.521+0000 7f2ea14ce640 1 --2- 172.21.15.70:0/1051975805 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d187aa00 0x5640d1878440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.521+0000 7f2ea14ce640 1 -- 172.21.15.70:0/1051975805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d1870140 msgr2=0x5640d1877f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.521+0000 7f2ea14ce640 1 --2- 172.21.15.70:0/1051975805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d1870140 0x5640d1877f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.521+0000 7f2ea14ce640 1 -- 172.21.15.70:0/1051975805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640d15b2620 con 0x5640d186f820 2024-06-04T05:20:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.521+0000 7f2ea0ccd640 1 --2- 172.21.15.70:0/1051975805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d1870140 0x5640d1877f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:20:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.521+0000 7f2ea6450740 1 -- 172.21.15.70:0/1051975805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e8c00fce0 con 0x5640d186f820 2024-06-04T05:20:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.521+0000 7f2ea14ce640 1 --2- 172.21.15.70:0/1051975805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d18779c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2e90033000 tx=0x7f2e9002a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=413df91812629275 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.521+0000 7f2e80ff9640 10 client.29921 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:20:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.521+0000 7f2e80ff9640 1 -- 172.21.15.70:0/1051975805 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90035040 con 0x5640d186f820 2024-06-04T05:20:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.521+0000 7f2e80ff9640 1 -- 172.21.15.70:0/1051975805 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8615+0+0 (secure 0 0 0) 0x7f2e90051470 con 0x5640d186f820 2024-06-04T05:20:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.521+0000 7f2e80ff9640 1 -- 172.21.15.70:0/1051975805 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f2e900516b0 con 0x5640d186f820 2024-06-04T05:20:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.521+0000 7f2ea6450740 10 client.29921 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.521+0000 7f2ea6450740 1 -- 172.21.15.70:0/1051975805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({fsmap=64}) -- 0x5640d1889320 con 0x5640d186f820 2024-06-04T05:20:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.522+0000 7f2e80ff9640 1 -- 172.21.15.70:0/1051975805 <== mon.2 v2:172.21.15.98:3300/0 4 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f2e90044030 con 0x5640d186f820 2024-06-04T05:20:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.522+0000 7f2ea6450740 10 client.29921 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.522+0000 7f2ea6450740 10 client.29921 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.522+0000 7f2ea6450740 20 client.29921 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.522+0000 7f2ea6450740 1 --2- 172.21.15.70:0/1051975805 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5640d188a740 0x5640d188cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.522+0000 7f2ea6450740 4 client.29921 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:20:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.522+0000 7f2ea6450740 1 -- 172.21.15.70:0/1051975805 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640d17e3b80 con 0x5640d188a740 2024-06-04T05:20:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.522+0000 7f2ea0ccd640 1 --2- 172.21.15.70:0/1051975805 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5640d188a740 0x5640d188cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.523+0000 7f2e80ff9640 10 client.29921 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.523+0000 7f2ea0ccd640 1 --2- 172.21.15.70:0/1051975805 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5640d188a740 0x5640d188cb90 crc :-1 s=READY pgs=481 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=632359b23f1fb163 server_cookie=5e118ab5ab0ea1b3 in_seq=0 out_seq=0 2024-06-04T05:20:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.523+0000 7f2e80ff9640 1 -- 172.21.15.70:0/1051975805 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5640d17e3b80 con 0x5640d188a740 2024-06-04T05:20:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.524+0000 7f2e80ff9640 10 client.29921 handle_command_reply: tid=0 2024-06-04T05:20:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.524+0000 7f2ea6450740 1 client.29921 shutdown 2024-06-04T05:20:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.524+0000 7f2ea6450740 1 -- 172.21.15.70:0/1051975805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 msgr2=0x5640d18779c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.524+0000 7f2ea6450740 1 --2- 172.21.15.70:0/1051975805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d18779c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2e90033000 tx=0x7f2e9002a480 comp rx=0 tx=0).stop 2024-06-04T05:20:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.524+0000 7f2ea6450740 1 -- 172.21.15.70:0/1051975805 shutdown_connections 2024-06-04T05:20:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.524+0000 7f2ea6450740 1 --2- 172.21.15.70:0/1051975805 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5640d188a740 0x5640d188cb90 crc :-1 s=READY pgs=481 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.524+0000 7f2ea6450740 1 --2- 172.21.15.70:0/1051975805 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d187aa00 0x5640d1878440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.524+0000 7f2ea6450740 1 --2- 172.21.15.70:0/1051975805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d1870140 0x5640d1877f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.524+0000 7f2ea6450740 1 --2- 172.21.15.70:0/1051975805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d18779c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.524+0000 7f2ea6450740 1 -- 172.21.15.70:0/1051975805 >> 172.21.15.70:0/1051975805 conn(0x5640d186bd30 msgr2=0x5640d186afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.524+0000 7f2e80ff9640 0 client.29921 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.525+0000 7f2ea6450740 1 -- 172.21.15.70:0/1051975805 shutdown_connections 2024-06-04T05:20:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.525+0000 7f2ea6450740 1 -- 172.21.15.70:0/1051975805 wait complete. 2024-06-04T05:20:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.525+0000 7f2ea6450740 20 client.29921 trim_cache size 0 max 16384 2024-06-04T05:20:06.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.549+0000 7f2ea6450740 1 Processor -- start 2024-06-04T05:20:06.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.549+0000 7f2ea6450740 1 -- start start 2024-06-04T05:20:06.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.549+0000 7f2ea6450740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d18721e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.549+0000 7f2ea6450740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d1870140 0x5640d1878260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.549+0000 7f2ea6450740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d187aa00 0x5640d18924f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.549+0000 7f2ea6450740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8800e890 con 0x5640d1870140 2024-06-04T05:20:06.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.549+0000 7f2ea6450740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e88010460 con 0x5640d186f820 2024-06-04T05:20:06.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.549+0000 7f2ea6450740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8800efc0 con 0x5640d187aa00 2024-06-04T05:20:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.549+0000 7f2ea14ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d18721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.549+0000 7f2ea14ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d18721e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:50020/0 (socket says 172.21.15.70:50020) 2024-06-04T05:20:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.549+0000 7f2ea14ce640 1 -- 172.21.15.70:0/3269224370 learned_addr learned my addr 172.21.15.70:0/3269224370 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.549+0000 7f2ea0ccd640 1 --2- 172.21.15.70:0/3269224370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d1870140 0x5640d1878260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.549+0000 7f2ea1ccf640 1 --2- 172.21.15.70:0/3269224370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d187aa00 0x5640d18924f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.550+0000 7f2ea14ce640 1 -- 172.21.15.70:0/3269224370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d187aa00 msgr2=0x5640d18924f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.550+0000 7f2ea14ce640 1 --2- 172.21.15.70:0/3269224370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d187aa00 0x5640d18924f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.550+0000 7f2ea14ce640 1 -- 172.21.15.70:0/3269224370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d1870140 msgr2=0x5640d1878260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.550+0000 7f2ea14ce640 1 --2- 172.21.15.70:0/3269224370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d1870140 0x5640d1878260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.550+0000 7f2ea14ce640 1 -- 172.21.15.70:0/3269224370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640d1872720 con 0x5640d186f820 2024-06-04T05:20:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.550+0000 7f2ea0ccd640 1 --2- 172.21.15.70:0/3269224370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d1870140 0x5640d1878260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.550+0000 7f2ea14ce640 1 --2- 172.21.15.70:0/3269224370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d18721e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009630 tx=0x7f2e9004a070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb6fcd31645578db server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.550+0000 7f2e82ffd640 1 -- 172.21.15.70:0/3269224370 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90051470 con 0x5640d186f820 2024-06-04T05:20:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.551+0000 7f2e82ffd640 1 -- 172.21.15.70:0/3269224370 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2e8c00fce0 con 0x5640d186f820 2024-06-04T05:20:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.551+0000 7f2e82ffd640 1 -- 172.21.15.70:0/3269224370 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9003d990 con 0x5640d186f820 2024-06-04T05:20:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.551+0000 7f2ea6450740 1 -- 172.21.15.70:0/3269224370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 msgr2=0x5640d18721e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.551+0000 7f2ea6450740 1 --2- 172.21.15.70:0/3269224370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d18721e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009630 tx=0x7f2e9004a070 comp rx=0 tx=0).stop 2024-06-04T05:20:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.551+0000 7f2ea6450740 1 -- 172.21.15.70:0/3269224370 shutdown_connections 2024-06-04T05:20:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.551+0000 7f2ea6450740 1 --2- 172.21.15.70:0/3269224370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d187aa00 0x5640d18924f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.551+0000 7f2ea6450740 1 --2- 172.21.15.70:0/3269224370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d1870140 0x5640d1878260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.551+0000 7f2ea6450740 1 --2- 172.21.15.70:0/3269224370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d18721e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.551+0000 7f2ea6450740 1 -- 172.21.15.70:0/3269224370 >> 172.21.15.70:0/3269224370 conn(0x5640d186bd30 msgr2=0x5640d1885f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.552+0000 7f2ea6450740 1 -- 172.21.15.70:0/3269224370 shutdown_connections 2024-06-04T05:20:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.552+0000 7f2ea6450740 1 -- 172.21.15.70:0/3269224370 wait complete. 2024-06-04T05:20:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.552+0000 7f2ea6450740 1 -- start start 2024-06-04T05:20:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.552+0000 7f2ea6450740 1 Processor -- start 2024-06-04T05:20:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea6450740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d187e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea6450740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d1870140 0x5640d187ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea6450740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d187aa00 0x5640d187ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea6450740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8800fd60 con 0x5640d1870140 2024-06-04T05:20:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea6450740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8800f690 con 0x5640d186f820 2024-06-04T05:20:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea6450740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8800c890 con 0x5640d187aa00 2024-06-04T05:20:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea6450740 10 client.0 _ll_register_callbacks cb 0x7f2e90004230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea14ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d187e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea14ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d187e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:50034/0 (socket says 172.21.15.70:50034) 2024-06-04T05:20:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea14ce640 1 -- 172.21.15.70:0/2630666910 learned_addr learned my addr 172.21.15.70:0/2630666910 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea1ccf640 1 --2- 172.21.15.70:0/2630666910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d187aa00 0x5640d187ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea0ccd640 1 --2- 172.21.15.70:0/2630666910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d1870140 0x5640d187ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea14ce640 1 -- 172.21.15.70:0/2630666910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d187aa00 msgr2=0x5640d187ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea14ce640 1 --2- 172.21.15.70:0/2630666910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d187aa00 0x5640d187ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea14ce640 1 -- 172.21.15.70:0/2630666910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d1870140 msgr2=0x5640d187ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea14ce640 1 --2- 172.21.15.70:0/2630666910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d1870140 0x5640d187ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea14ce640 1 -- 172.21.15.70:0/2630666910 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640d18f43c0 con 0x5640d186f820 2024-06-04T05:20:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea14ce640 1 --2- 172.21.15.70:0/2630666910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d187e4e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2e90000c00 tx=0x7f2e900519c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e7114dbca5d4e88 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2e80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:20:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2e80ff9640 1 -- 172.21.15.70:0/2630666910 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90069060 con 0x5640d186f820 2024-06-04T05:20:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.553+0000 7f2ea6450740 1 -- 172.21.15.70:0/2630666910 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e8c00fce0 con 0x5640d186f820 2024-06-04T05:20:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.554+0000 7f2e80ff9640 1 -- 172.21.15.70:0/2630666910 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8615+0+0 (secure 0 0 0) 0x7f2e9003d710 con 0x5640d186f820 2024-06-04T05:20:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.554+0000 7f2e80ff9640 1 -- 172.21.15.70:0/2630666910 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f2e9003d910 con 0x5640d186f820 2024-06-04T05:20:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.554+0000 7f2ea6450740 10 client.29927 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.554+0000 7f2ea6450740 1 -- 172.21.15.70:0/2630666910 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x5640d18f55e0 con 0x5640d186f820 2024-06-04T05:20:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.554+0000 7f2e80ff9640 1 -- 172.21.15.70:0/2630666910 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2e8c00fce0 con 0x5640d186f820 2024-06-04T05:20:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.554+0000 7f2e80ff9640 1 -- 172.21.15.70:0/2630666910 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f2e90044040 con 0x5640d186f820 2024-06-04T05:20:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.554+0000 7f2ea6450740 10 client.29927 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.554+0000 7f2ea6450740 10 client.29927 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.554+0000 7f2ea6450740 20 client.29927 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.554+0000 7f2ea6450740 1 --2- 172.21.15.70:0/2630666910 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5640d18f5bd0 0x5640d18f7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.554+0000 7f2ea6450740 4 client.29927 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:20:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.554+0000 7f2ea6450740 1 -- 172.21.15.70:0/2630666910 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5640d17dd360 con 0x5640d18f5bd0 2024-06-04T05:20:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.555+0000 7f2ea0ccd640 1 --2- 172.21.15.70:0/2630666910 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5640d18f5bd0 0x5640d18f7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.555+0000 7f2ea0ccd640 1 --2- 172.21.15.70:0/2630666910 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5640d18f5bd0 0x5640d18f7ff0 crc :-1 s=READY pgs=482 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=824877c23e3d3542 server_cookie=36ae5a1c86835d20 in_seq=0 out_seq=0 2024-06-04T05:20:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.555+0000 7f2e80ff9640 10 client.29927 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.666+0000 7f2e80ff9640 1 -- 172.21.15.70:0/2630666910 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5640d17dd360 con 0x5640d18f5bd0 2024-06-04T05:20:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.666+0000 7f2e80ff9640 10 client.29927 handle_command_reply: tid=0 2024-06-04T05:20:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.666+0000 7f2ea6450740 1 client.29927 shutdown 2024-06-04T05:20:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.666+0000 7f2ea6450740 1 -- 172.21.15.70:0/2630666910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 msgr2=0x5640d187e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.666+0000 7f2ea6450740 1 --2- 172.21.15.70:0/2630666910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d187e4e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2e90000c00 tx=0x7f2e900519c0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.667+0000 7f2ea6450740 1 -- 172.21.15.70:0/2630666910 shutdown_connections 2024-06-04T05:20:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.667+0000 7f2ea6450740 1 --2- 172.21.15.70:0/2630666910 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5640d18f5bd0 0x5640d18f7ff0 crc :-1 s=READY pgs=482 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.667+0000 7f2ea6450740 1 --2- 172.21.15.70:0/2630666910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5640d187aa00 0x5640d187ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.667+0000 7f2ea6450740 1 --2- 172.21.15.70:0/2630666910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5640d1870140 0x5640d187ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.667+0000 7f2ea6450740 1 --2- 172.21.15.70:0/2630666910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640d186f820 0x5640d187e4e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.667+0000 7f2ea6450740 1 -- 172.21.15.70:0/2630666910 >> 172.21.15.70:0/2630666910 conn(0x5640d186bd30 msgr2=0x5640d1885680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.667+0000 7f2e80ff9640 0 client.29927 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.667+0000 7f2ea6450740 1 -- 172.21.15.70:0/2630666910 shutdown_connections 2024-06-04T05:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.667+0000 7f2ea6450740 1 -- 172.21.15.70:0/2630666910 wait complete. 2024-06-04T05:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.667+0000 7f2ea6450740 20 client.29927 trim_cache size 0 max 16384 2024-06-04T05:20:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:20:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:20:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.668+0000 7f2ea3732640 1 -- 172.21.15.70:0/2383732611 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2e840b2b60 msgr2=0x7f2e840b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.668+0000 7f2ea3732640 1 --2- 172.21.15.70:0/2383732611 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2e840b2b60 0x7f2e840b5020 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009850 tx=0x7f2e90005fd0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.668+0000 7f2ea3732640 1 -- 172.21.15.70:0/2383732611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e9c106e90 msgr2=0x7f2e9c10fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.668+0000 7f2ea3732640 1 --2- 172.21.15.70:0/2383732611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e9c106e90 0x7f2e9c10fde0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400b420 tx=0x7f2e9400b8e0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.668+0000 7f2ea3732640 1 -- 172.21.15.70:0/2383732611 shutdown_connections 2024-06-04T05:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.668+0000 7f2ea3732640 1 --2- 172.21.15.70:0/2383732611 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2e840b2b60 0x7f2e840b5020 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.668+0000 7f2ea3732640 1 --2- 172.21.15.70:0/2383732611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e9c106e90 0x7f2e9c10fde0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.668+0000 7f2ea3732640 1 --2- 172.21.15.70:0/2383732611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e9c110320 0x7f2e9c1107d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.668+0000 7f2ea3732640 1 --2- 172.21.15.70:0/2383732611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e9c1023b0 0x7f2e9c115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.668+0000 7f2ea3732640 1 -- 172.21.15.70:0/2383732611 >> 172.21.15.70:0/2383732611 conn(0x7f2e9c078b60 msgr2=0x7f2e9c104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.669+0000 7f2ea3732640 1 -- 172.21.15.70:0/2383732611 shutdown_connections 2024-06-04T05:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:06.669+0000 7f2ea3732640 1 -- 172.21.15.70:0/2383732611 wait complete. 2024-06-04T05:20:06.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 67934 6.62 MB/sec execute 418 sec latency 163.254 ms 2024-06-04T05:20:07.689 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:07 smithi070 ceph-mon[32631]: pgmap v722: 97 pgs: 97 active+clean; 972 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.0 MiB/s wr, 323 op/s 2024-06-04T05:20:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:07 smithi098 ceph-mon[35569]: pgmap v722: 97 pgs: 97 active+clean; 972 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.0 MiB/s wr, 323 op/s 2024-06-04T05:20:07.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 68103 6.64 MB/sec execute 419 sec latency 105.125 ms 2024-06-04T05:20:07.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:07 smithi112 ceph-mon[36453]: pgmap v722: 97 pgs: 97 active+clean; 972 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.0 MiB/s wr, 323 op/s 2024-06-04T05:20:08.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 68258 6.64 MB/sec execute 420 sec latency 117.792 ms 2024-06-04T05:20:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:09 smithi070 ceph-mon[32631]: osdmap e114: 12 total, 12 up, 12 in 2024-06-04T05:20:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:09 smithi070 ceph-mon[32631]: pgmap v724: 97 pgs: 97 active+clean; 972 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.5 MiB/s wr, 404 op/s 2024-06-04T05:20:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:09 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:20:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:09 smithi112 ceph-mon[36453]: osdmap e114: 12 total, 12 up, 12 in 2024-06-04T05:20:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:09 smithi112 ceph-mon[36453]: pgmap v724: 97 pgs: 97 active+clean; 972 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.5 MiB/s wr, 404 op/s 2024-06-04T05:20:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:09 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:20:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:09 smithi098 ceph-mon[35569]: osdmap e114: 12 total, 12 up, 12 in 2024-06-04T05:20:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:09 smithi098 ceph-mon[35569]: pgmap v724: 97 pgs: 97 active+clean; 972 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.5 MiB/s wr, 404 op/s 2024-06-04T05:20:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:09 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:20:09.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 68403 6.64 MB/sec execute 421 sec latency 118.467 ms 2024-06-04T05:20:10.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 68544 6.64 MB/sec execute 422 sec latency 517.103 ms 2024-06-04T05:20:11.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:11 smithi070 ceph-mon[32631]: pgmap v725: 97 pgs: 97 active+clean; 972 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 177 op/s 2024-06-04T05:20:11.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:11 smithi112 ceph-mon[36453]: pgmap v725: 97 pgs: 97 active+clean; 972 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 177 op/s 2024-06-04T05:20:11.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:11 smithi098 ceph-mon[35569]: pgmap v725: 97 pgs: 97 active+clean; 972 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 177 op/s 2024-06-04T05:20:11.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 68708 6.65 MB/sec execute 423 sec latency 645.135 ms 2024-06-04T05:20:12.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.371+0000 7f570c64e700 1 -- 172.21.15.70:0/588191514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5704102440 msgr2=0x7f57041028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.371+0000 7f570c64e700 1 --2- 172.21.15.70:0/588191514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5704102440 0x7f57041028c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f56f4009990 tx=0x7f56f4009ca0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.371+0000 7f570c64e700 1 -- 172.21.15.70:0/588191514 shutdown_connections 2024-06-04T05:20:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.371+0000 7f570c64e700 1 --2- 172.21.15.70:0/588191514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5704102e00 0x7f570410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.371+0000 7f570c64e700 1 --2- 172.21.15.70:0/588191514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5704108520 0x7f5704108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.371+0000 7f570c64e700 1 --2- 172.21.15.70:0/588191514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5704102440 0x7f57041028c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.371+0000 7f570c64e700 1 -- 172.21.15.70:0/588191514 >> 172.21.15.70:0/588191514 conn(0x7f57040793f0 msgr2=0x7f5704079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.371+0000 7f570c64e700 1 -- 172.21.15.70:0/588191514 shutdown_connections 2024-06-04T05:20:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.371+0000 7f570c64e700 1 -- 172.21.15.70:0/588191514 wait complete. 2024-06-04T05:20:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.372+0000 7f570c64e700 1 Processor -- start 2024-06-04T05:20:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.372+0000 7f570c64e700 1 -- start start 2024-06-04T05:20:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.372+0000 7f570c64e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5704102440 0x7f570410e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.372+0000 7f570c64e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5704102e00 0x7f570410ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.372+0000 7f570c64e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5704108520 0x7f57041a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.372+0000 7f570c64e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57041135a0 con 0x7f5704102e00 2024-06-04T05:20:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.372+0000 7f570c64e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5704113420 con 0x7f5704108520 2024-06-04T05:20:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.372+0000 7f570c64e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5704113720 con 0x7f5704102440 2024-06-04T05:20:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.372+0000 7f570abeb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5704108520 0x7f57041a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.372+0000 7f5709be9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5704102e00 0x7f570410ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.372+0000 7f570abeb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5704108520 0x7f57041a4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:50050/0 (socket says 172.21.15.70:50050) 2024-06-04T05:20:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.372+0000 7f570abeb700 1 -- 172.21.15.70:0/1964351542 learned_addr learned my addr 172.21.15.70:0/1964351542 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.372+0000 7f5709be9700 1 --2- 172.21.15.70:0/1964351542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5704102e00 0x7f570410ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57584/0 (socket says 172.21.15.70:57584) 2024-06-04T05:20:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.373+0000 7f570a3ea700 1 --2- 172.21.15.70:0/1964351542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5704102440 0x7f570410e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.373+0000 7f5709be9700 1 -- 172.21.15.70:0/1964351542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5704102440 msgr2=0x7f570410e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.373+0000 7f5709be9700 1 --2- 172.21.15.70:0/1964351542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5704102440 0x7f570410e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.373+0000 7f5709be9700 1 -- 172.21.15.70:0/1964351542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5704108520 msgr2=0x7f57041a4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.373+0000 7f5709be9700 1 --2- 172.21.15.70:0/1964351542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5704108520 0x7f57041a4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.373+0000 7f5709be9700 1 -- 172.21.15.70:0/1964351542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57041a51b0 con 0x7f5704102e00 2024-06-04T05:20:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.373+0000 7f5709be9700 1 --2- 172.21.15.70:0/1964351542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5704102e00 0x7f570410ec90 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f56f4000c00 tx=0x7f56f40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.373+0000 7f56fb7fe700 1 -- 172.21.15.70:0/1964351542 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4026070 con 0x7f5704102e00 2024-06-04T05:20:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.374+0000 7f570c64e700 1 -- 172.21.15.70:0/1964351542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57041a5440 con 0x7f5704102e00 2024-06-04T05:20:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.374+0000 7f570c64e700 1 -- 172.21.15.70:0/1964351542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57041acd00 con 0x7f5704102e00 2024-06-04T05:20:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.374+0000 7f56fb7fe700 1 -- 172.21.15.70:0/1964351542 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f56f4021410 con 0x7f5704102e00 2024-06-04T05:20:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.374+0000 7f56fb7fe700 1 -- 172.21.15.70:0/1964351542 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4017650 con 0x7f5704102e00 2024-06-04T05:20:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.377+0000 7f56fb7fe700 1 -- 172.21.15.70:0/1964351542 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f56f4029070 con 0x7f5704102e00 2024-06-04T05:20:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.378+0000 7f56fb7fe700 1 --2- 172.21.15.70:0/1964351542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56f00b2af0 0x7f56f00b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.379+0000 7f570a3ea700 1 --2- 172.21.15.70:0/1964351542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56f00b2af0 0x7f56f00b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.379+0000 7f5716ee3b80 1 Processor -- start 2024-06-04T05:20:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.379+0000 7f56fb7fe700 1 -- 172.21.15.70:0/1964351542 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f56f40e32a0 con 0x7f5704102e00 2024-06-04T05:20:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.379+0000 7f5716ee3b80 1 -- start start 2024-06-04T05:20:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.379+0000 7f570a3ea700 1 --2- 172.21.15.70:0/1964351542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56f00b2af0 0x7f56f00b4fb0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5700009be0 tx=0x7f5700009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.379+0000 7f5716ee3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 0x563823261700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.379+0000 7f5716ee3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325b8a0 0x56382325bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f5716ee3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325c260 0x563823268080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f570a3ea700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 0x563823261700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f5709be9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325b8a0 0x56382325bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f570a3ea700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 0x563823261700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:33918/0 (socket says 172.21.15.70:33918) 2024-06-04T05:20:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f570a3ea700 1 -- 172.21.15.70:0/592618112 learned_addr learned my addr 172.21.15.70:0/592618112 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f5709be9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325b8a0 0x56382325bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:50062/0 (socket says 172.21.15.70:50062) 2024-06-04T05:20:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f5716ee3b80 1 -- 172.21.15.70:0/592618112 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5638231b9d70 con 0x56382325c260 2024-06-04T05:20:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f5716ee3b80 1 -- 172.21.15.70:0/592618112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5638231a9910 con 0x56382325b8a0 2024-06-04T05:20:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f5716ee3b80 1 -- 172.21.15.70:0/592618112 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5638231ad950 con 0x563823261320 2024-06-04T05:20:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f570abeb700 1 --2- 172.21.15.70:0/592618112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325c260 0x563823268080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f570a3ea700 1 -- 172.21.15.70:0/592618112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325b8a0 msgr2=0x56382325bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f570a3ea700 1 --2- 172.21.15.70:0/592618112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325b8a0 0x56382325bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f570a3ea700 1 -- 172.21.15.70:0/592618112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325c260 msgr2=0x563823268080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f570a3ea700 1 --2- 172.21.15.70:0/592618112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325c260 0x563823268080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f570a3ea700 1 -- 172.21.15.70:0/592618112 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563823152fb0 con 0x563823261320 2024-06-04T05:20:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f5709be9700 1 --2- 172.21.15.70:0/592618112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325b8a0 0x56382325bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f570abeb700 1 --2- 172.21.15.70:0/592618112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325c260 0x563823268080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:20:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.380+0000 7f570a3ea700 1 --2- 172.21.15.70:0/592618112 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 0x563823261700 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5700011d70 tx=0x7f5700014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e902246af3450913 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.381+0000 7f56f8ff9700 1 -- 172.21.15.70:0/592618112 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5700011da0 con 0x563823261320 2024-06-04T05:20:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.381+0000 7f56f8ff9700 1 -- 172.21.15.70:0/592618112 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5700004020 con 0x563823261320 2024-06-04T05:20:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.381+0000 7f56f8ff9700 1 -- 172.21.15.70:0/592618112 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5700003c10 con 0x563823261320 2024-06-04T05:20:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.381+0000 7f5716ee3b80 1 -- 172.21.15.70:0/592618112 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 msgr2=0x563823261700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.381+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/592618112 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 0x563823261700 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5700011d70 tx=0x7f5700014000 comp rx=0 tx=0).stop 2024-06-04T05:20:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.382+0000 7f5716ee3b80 1 -- 172.21.15.70:0/592618112 shutdown_connections 2024-06-04T05:20:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.382+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/592618112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325c260 0x563823268080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.382+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/592618112 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 0x563823261700 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.382+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/592618112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325b8a0 0x56382325bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.382+0000 7f5716ee3b80 1 -- 172.21.15.70:0/592618112 >> 172.21.15.70:0/592618112 conn(0x563823187240 msgr2=0x5638232563e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.382+0000 7f5716ee3b80 1 -- 172.21.15.70:0/592618112 shutdown_connections 2024-06-04T05:20:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.382+0000 7f5716ee3b80 1 -- 172.21.15.70:0/592618112 wait complete. 2024-06-04T05:20:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.382+0000 7f5716ee3b80 1 -- start start 2024-06-04T05:20:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.382+0000 7f5716ee3b80 1 Processor -- start 2024-06-04T05:20:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.383+0000 7f5716ee3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325b8a0 0x563823267ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.383+0000 7f5716ee3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325c260 0x563823265030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.383+0000 7f5716ee3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 0x563823265570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.383+0000 7f5716ee3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5638230affa0 con 0x56382325b8a0 2024-06-04T05:20:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.383+0000 7f5716ee3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563823111da0 con 0x56382325c260 2024-06-04T05:20:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.383+0000 7f5716ee3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563823268730 con 0x563823261320 2024-06-04T05:20:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.383+0000 7f5716ee3b80 10 client.0 _ll_register_callbacks cb 0x563823144950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.383+0000 7f570abeb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 0x563823265570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.383+0000 7f570abeb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 0x563823265570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:33934/0 (socket says 172.21.15.70:33934) 2024-06-04T05:20:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.383+0000 7f570abeb700 1 -- 172.21.15.70:0/4177202834 learned_addr learned my addr 172.21.15.70:0/4177202834 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.383+0000 7f570a3ea700 1 --2- 172.21.15.70:0/4177202834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325b8a0 0x563823267ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.383+0000 7f570abeb700 1 -- 172.21.15.70:0/4177202834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325c260 msgr2=0x563823265030 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:20:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.383+0000 7f570abeb700 1 --2- 172.21.15.70:0/4177202834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325c260 0x563823265030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.383+0000 7f570abeb700 1 -- 172.21.15.70:0/4177202834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325b8a0 msgr2=0x563823267ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.383+0000 7f570abeb700 1 --2- 172.21.15.70:0/4177202834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325b8a0 0x563823267ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.383+0000 7f570abeb700 1 -- 172.21.15.70:0/4177202834 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638230f98d0 con 0x563823261320 2024-06-04T05:20:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.384+0000 7f570abeb700 1 --2- 172.21.15.70:0/4177202834 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 0x563823265570 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f56fc009fd0 tx=0x7f56fc012520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=448bdd6762bb01ae server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.384+0000 7f56edffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:20:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.384+0000 7f56edffb700 1 -- 172.21.15.70:0/4177202834 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc01c020 con 0x563823261320 2024-06-04T05:20:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.384+0000 7f56edffb700 1 -- 172.21.15.70:0/4177202834 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f56fc018470 con 0x563823261320 2024-06-04T05:20:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.384+0000 7f5716ee3b80 1 -- 172.21.15.70:0/4177202834 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5700004020 con 0x563823261320 2024-06-04T05:20:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.384+0000 7f56edffb700 1 -- 172.21.15.70:0/4177202834 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f56fc027050 con 0x563823261320 2024-06-04T05:20:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.384+0000 7f5716ee3b80 10 client.30073 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.384+0000 7f5716ee3b80 1 -- 172.21.15.70:0/4177202834 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x563823163760 con 0x563823261320 2024-06-04T05:20:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.385+0000 7f56edffb700 1 -- 172.21.15.70:0/4177202834 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f56fc018b00 con 0x563823261320 2024-06-04T05:20:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.385+0000 7f56edffb700 1 -- 172.21.15.70:0/4177202834 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f56fc016810 con 0x563823261320 2024-06-04T05:20:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.385+0000 7f5716ee3b80 10 client.30073 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.385+0000 7f5716ee3b80 10 client.30073 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.385+0000 7f5716ee3b80 20 client.30073 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:12.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.385+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/4177202834 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5638232752c0 0x5638232776b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.385+0000 7f5716ee3b80 4 client.30073 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:20:12.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.385+0000 7f5716ee3b80 1 -- 172.21.15.70:0/4177202834 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5700004020 con 0x5638232752c0 2024-06-04T05:20:12.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.385+0000 7f570a3ea700 1 --2- 172.21.15.70:0/4177202834 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5638232752c0 0x5638232776b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:12.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.386+0000 7f56edffb700 10 client.30073 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:12.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.386+0000 7f570a3ea700 1 --2- 172.21.15.70:0/4177202834 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5638232752c0 0x5638232776b0 crc :-1 s=READY pgs=483 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2051d18735df4a79 server_cookie=e2bdefacbc904814 in_seq=0 out_seq=0 2024-06-04T05:20:12.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.386+0000 7f56edffb700 1 -- 172.21.15.70:0/4177202834 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f5700004020 con 0x5638232752c0 2024-06-04T05:20:12.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.387+0000 7f56edffb700 10 client.30073 handle_command_reply: tid=0 2024-06-04T05:20:12.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.387+0000 7f5716ee3b80 1 client.30073 shutdown 2024-06-04T05:20:12.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.387+0000 7f5716ee3b80 1 -- 172.21.15.70:0/4177202834 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 msgr2=0x563823265570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:12.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.387+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/4177202834 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 0x563823265570 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f56fc009fd0 tx=0x7f56fc012520 comp rx=0 tx=0).stop 2024-06-04T05:20:12.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.389+0000 7f5716ee3b80 1 -- 172.21.15.70:0/4177202834 shutdown_connections 2024-06-04T05:20:12.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.389+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/4177202834 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 0x563823265570 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.389+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/4177202834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325b8a0 0x563823267ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.389+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/4177202834 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5638232752c0 0x5638232776b0 crc :-1 s=READY pgs=483 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.389+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/4177202834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325c260 0x563823265030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.389+0000 7f5716ee3b80 1 -- 172.21.15.70:0/4177202834 >> 172.21.15.70:0/4177202834 conn(0x563823187240 msgr2=0x563823255af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.389+0000 7f56edffb700 0 client.30073 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.389+0000 7f5716ee3b80 1 -- 172.21.15.70:0/4177202834 shutdown_connections 2024-06-04T05:20:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.389+0000 7f5716ee3b80 1 -- 172.21.15.70:0/4177202834 wait complete. 2024-06-04T05:20:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.389+0000 7f5716ee3b80 20 client.30073 trim_cache size 0 max 16384 2024-06-04T05:20:12.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.422+0000 7f5716ee3b80 1 Processor -- start 2024-06-04T05:20:12.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.422+0000 7f5716ee3b80 1 -- start start 2024-06-04T05:20:12.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.422+0000 7f5716ee3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325b8a0 0x563823260820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.422+0000 7f5716ee3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325c260 0x563823256270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.422+0000 7f5716ee3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 0x56382325a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.422+0000 7f5716ee3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57041b43c0 con 0x56382325b8a0 2024-06-04T05:20:12.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.422+0000 7f5716ee3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f57041b5f90 con 0x56382325c260 2024-06-04T05:20:12.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.422+0000 7f5716ee3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f57041b4af0 con 0x563823261320 2024-06-04T05:20:12.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.423+0000 7f5709be9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325c260 0x563823256270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:12.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.423+0000 7f5709be9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325c260 0x563823256270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:50082/0 (socket says 172.21.15.70:50082) 2024-06-04T05:20:12.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.423+0000 7f5709be9700 1 -- 172.21.15.70:0/362690121 learned_addr learned my addr 172.21.15.70:0/362690121 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:12.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.423+0000 7f570a3ea700 1 --2- 172.21.15.70:0/362690121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325b8a0 0x563823260820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:12.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.423+0000 7f570a3ea700 1 -- 172.21.15.70:0/362690121 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 msgr2=0x56382325a890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-04T05:20:12.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.423+0000 7f570a3ea700 1 --2- 172.21.15.70:0/362690121 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 0x56382325a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.423+0000 7f570a3ea700 1 -- 172.21.15.70:0/362690121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325c260 msgr2=0x563823256270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:12.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.423+0000 7f570a3ea700 1 --2- 172.21.15.70:0/362690121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325c260 0x563823256270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.423+0000 7f570a3ea700 1 -- 172.21.15.70:0/362690121 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563823268050 con 0x56382325b8a0 2024-06-04T05:20:12.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.423+0000 7f5709be9700 1 --2- 172.21.15.70:0/362690121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325c260 0x563823256270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:20:12.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.423+0000 7f570a3ea700 1 --2- 172.21.15.70:0/362690121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325b8a0 0x563823260820 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f57000117d0 tx=0x7f57000092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9d2321ee6a6315 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:12.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.424+0000 7f56effff700 1 -- 172.21.15.70:0/362690121 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5700036070 con 0x56382325b8a0 2024-06-04T05:20:12.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.424+0000 7f56effff700 1 -- 172.21.15.70:0/362690121 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f570000ee40 con 0x56382325b8a0 2024-06-04T05:20:12.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.424+0000 7f56effff700 1 -- 172.21.15.70:0/362690121 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5700031a50 con 0x56382325b8a0 2024-06-04T05:20:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.425+0000 7f5716ee3b80 1 -- 172.21.15.70:0/362690121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325b8a0 msgr2=0x563823260820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.425+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/362690121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325b8a0 0x563823260820 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f57000117d0 tx=0x7f57000092d0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.425+0000 7f5716ee3b80 1 -- 172.21.15.70:0/362690121 shutdown_connections 2024-06-04T05:20:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.425+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/362690121 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563823261320 0x56382325a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.425+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/362690121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325b8a0 0x563823260820 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.425+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/362690121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56382325c260 0x563823256270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.425+0000 7f5716ee3b80 1 -- 172.21.15.70:0/362690121 >> 172.21.15.70:0/362690121 conn(0x563823187240 msgr2=0x563823260110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.425+0000 7f5716ee3b80 1 -- 172.21.15.70:0/362690121 shutdown_connections 2024-06-04T05:20:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.425+0000 7f5716ee3b80 1 -- 172.21.15.70:0/362690121 wait complete. 2024-06-04T05:20:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.426+0000 7f5716ee3b80 1 -- start start 2024-06-04T05:20:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.426+0000 7f5716ee3b80 1 Processor -- start 2024-06-04T05:20:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.426+0000 7f5716ee3b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56382325b8a0 0x56382327cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.426+0000 7f5716ee3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325c260 0x56382327d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.426+0000 7f5716ee3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563823261320 0x56382327d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.426+0000 7f5716ee3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57041b5890 con 0x56382325c260 2024-06-04T05:20:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.426+0000 7f5716ee3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f57041b51c0 con 0x563823261320 2024-06-04T05:20:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.426+0000 7f5716ee3b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f57041b2380 con 0x56382325b8a0 2024-06-04T05:20:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.426+0000 7f5716ee3b80 10 client.0 _ll_register_callbacks cb 0x563823144950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.427+0000 7f570abeb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563823261320 0x56382327d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.427+0000 7f570abeb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563823261320 0x56382327d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:50088/0 (socket says 172.21.15.70:50088) 2024-06-04T05:20:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.427+0000 7f570abeb700 1 -- 172.21.15.70:0/736831842 learned_addr learned my addr 172.21.15.70:0/736831842 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.427+0000 7f570a3ea700 1 --2- 172.21.15.70:0/736831842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56382325b8a0 0x56382327cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.427+0000 7f5709be9700 1 --2- 172.21.15.70:0/736831842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325c260 0x56382327d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.427+0000 7f570abeb700 1 -- 172.21.15.70:0/736831842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56382325b8a0 msgr2=0x56382327cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.427+0000 7f570abeb700 1 --2- 172.21.15.70:0/736831842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56382325b8a0 0x56382327cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.427+0000 7f570abeb700 1 -- 172.21.15.70:0/736831842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325c260 msgr2=0x56382327d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.427+0000 7f570abeb700 1 --2- 172.21.15.70:0/736831842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325c260 0x56382327d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.427+0000 7f570abeb700 1 -- 172.21.15.70:0/736831842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56382326c200 con 0x563823261320 2024-06-04T05:20:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.427+0000 7f570abeb700 1 --2- 172.21.15.70:0/736831842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563823261320 0x56382327d6d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f56fc009b30 tx=0x7f56fc016e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbc012978f0cbaa9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.427+0000 7f56edffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:20:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.428+0000 7f56edffb700 1 -- 172.21.15.70:0/736831842 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc01c040 con 0x563823261320 2024-06-04T05:20:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.428+0000 7f56edffb700 1 -- 172.21.15.70:0/736831842 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f56fc005c60 con 0x563823261320 2024-06-04T05:20:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.428+0000 7f5716ee3b80 1 -- 172.21.15.70:0/736831842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5638231bbcc0 con 0x563823261320 2024-06-04T05:20:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.428+0000 7f56edffb700 1 -- 172.21.15.70:0/736831842 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f56fc039050 con 0x563823261320 2024-06-04T05:20:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.428+0000 7f5716ee3b80 10 client.29942 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.428+0000 7f5716ee3b80 1 -- 172.21.15.70:0/736831842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x5638232e0340 con 0x563823261320 2024-06-04T05:20:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.428+0000 7f56edffb700 1 -- 172.21.15.70:0/736831842 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f56fc030070 con 0x563823261320 2024-06-04T05:20:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.429+0000 7f56edffb700 1 -- 172.21.15.70:0/736831842 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f56fc016420 con 0x563823261320 2024-06-04T05:20:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.429+0000 7f5716ee3b80 10 client.29942 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.429+0000 7f5716ee3b80 10 client.29942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.429+0000 7f5716ee3b80 20 client.29942 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.429+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/736831842 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5638232e0a60 0x5638232e2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.429+0000 7f5716ee3b80 4 client.29942 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:20:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.429+0000 7f5716ee3b80 1 -- 172.21.15.70:0/736831842 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5638231bbcc0 con 0x5638232e0a60 2024-06-04T05:20:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.429+0000 7f570a3ea700 1 --2- 172.21.15.70:0/736831842 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5638232e0a60 0x5638232e2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.430+0000 7f56edffb700 10 client.29942 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.430+0000 7f570a3ea700 1 --2- 172.21.15.70:0/736831842 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5638232e0a60 0x5638232e2e80 crc :-1 s=READY pgs=484 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86f000acd1bacd51 server_cookie=939d97605006865b in_seq=0 out_seq=0 2024-06-04T05:20:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.430+0000 7f56edffb700 1 -- 172.21.15.70:0/736831842 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5638231bbcc0 con 0x5638232e0a60 2024-06-04T05:20:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.430+0000 7f56edffb700 10 client.29942 handle_command_reply: tid=0 2024-06-04T05:20:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.430+0000 7f5716ee3b80 1 client.29942 shutdown 2024-06-04T05:20:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.431+0000 7f5716ee3b80 1 -- 172.21.15.70:0/736831842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563823261320 msgr2=0x56382327d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.431+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/736831842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563823261320 0x56382327d6d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f56fc009b30 tx=0x7f56fc016e70 comp rx=0 tx=0).stop 2024-06-04T05:20:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.431+0000 7f5716ee3b80 1 -- 172.21.15.70:0/736831842 shutdown_connections 2024-06-04T05:20:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.431+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/736831842 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5638232e0a60 0x5638232e2e80 crc :-1 s=READY pgs=484 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.431+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/736831842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563823261320 0x56382327d6d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.431+0000 7f56edffb700 0 client.29942 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.431+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/736831842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56382325b8a0 0x56382327cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.431+0000 7f5716ee3b80 1 --2- 172.21.15.70:0/736831842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56382325c260 0x56382327d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.431+0000 7f5716ee3b80 1 -- 172.21.15.70:0/736831842 >> 172.21.15.70:0/736831842 conn(0x563823187240 msgr2=0x56382325ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.431+0000 7f5716ee3b80 1 -- 172.21.15.70:0/736831842 shutdown_connections 2024-06-04T05:20:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.431+0000 7f5716ee3b80 1 -- 172.21.15.70:0/736831842 wait complete. 2024-06-04T05:20:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.431+0000 7f5716ee3b80 20 client.29942 trim_cache size 0 max 16384 2024-06-04T05:20:12.437 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:20:12.437 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:20:12.437 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:20:12.437 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:20:12.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.432+0000 7f570c64e700 1 -- 172.21.15.70:0/1964351542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56f00b2af0 msgr2=0x7f56f00b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:12.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.432+0000 7f570c64e700 1 --2- 172.21.15.70:0/1964351542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56f00b2af0 0x7f56f00b4fb0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5700009be0 tx=0x7f5700009260 comp rx=0 tx=0).stop 2024-06-04T05:20:12.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.432+0000 7f570c64e700 1 -- 172.21.15.70:0/1964351542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5704102e00 msgr2=0x7f570410ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:12.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.432+0000 7f570c64e700 1 --2- 172.21.15.70:0/1964351542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5704102e00 0x7f570410ec90 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f56f4000c00 tx=0x7f56f40047b0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.432+0000 7f570c64e700 1 -- 172.21.15.70:0/1964351542 shutdown_connections 2024-06-04T05:20:12.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.432+0000 7f570c64e700 1 --2- 172.21.15.70:0/1964351542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f56f00b2af0 0x7f56f00b4fb0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.432+0000 7f570c64e700 1 --2- 172.21.15.70:0/1964351542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5704108520 0x7f57041a4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.432+0000 7f570c64e700 1 --2- 172.21.15.70:0/1964351542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5704102440 0x7f570410e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.432+0000 7f570c64e700 1 --2- 172.21.15.70:0/1964351542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5704102e00 0x7f570410ec90 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:12.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.432+0000 7f570c64e700 1 -- 172.21.15.70:0/1964351542 >> 172.21.15.70:0/1964351542 conn(0x7f57040793f0 msgr2=0x7f5704101040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:12.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.432+0000 7f570c64e700 1 -- 172.21.15.70:0/1964351542 shutdown_connections 2024-06-04T05:20:12.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:12.432+0000 7f570c64e700 1 -- 172.21.15.70:0/1964351542 wait complete. 2024-06-04T05:20:12.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:12 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:20:12.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:12 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:20:12.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:12 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:20:12.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:12 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:20:12.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:12 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:20:12.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:12 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:20:12.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:12 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:20:12.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:12 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:20:12.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:12 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:20:12.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 68870 6.66 MB/sec execute 424 sec latency 121.707 ms 2024-06-04T05:20:13.175 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:20:13.176 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:20:13.585 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:20:13.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 69028 6.66 MB/sec execute 425 sec latency 74.762 ms 2024-06-04T05:20:13.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:13 smithi070 ceph-mon[32631]: pgmap v726: 97 pgs: 97 active+clean; 985 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.3 MiB/s wr, 353 op/s 2024-06-04T05:20:14.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:13 smithi112 ceph-mon[36453]: pgmap v726: 97 pgs: 97 active+clean; 985 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.3 MiB/s wr, 353 op/s 2024-06-04T05:20:14.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:13 smithi098 ceph-mon[35569]: pgmap v726: 97 pgs: 97 active+clean; 985 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.3 MiB/s wr, 353 op/s 2024-06-04T05:20:14.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 69190 6.66 MB/sec execute 426 sec latency 55.570 ms 2024-06-04T05:20:15.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.549+0000 7f5311f74700 1 -- 172.21.15.70:0/3788260002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f530c103590 msgr2=0x7f530c103a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:15.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.549+0000 7f5311f74700 1 --2- 172.21.15.70:0/3788260002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f530c103590 0x7f530c103a10 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f5300009990 tx=0x7f5300009ca0 comp rx=0 tx=0).stop 2024-06-04T05:20:15.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.549+0000 7f5311f74700 1 -- 172.21.15.70:0/3788260002 shutdown_connections 2024-06-04T05:20:15.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.549+0000 7f5311f74700 1 --2- 172.21.15.70:0/3788260002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f530c103f50 0x7f530c110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:15.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.549+0000 7f5311f74700 1 --2- 172.21.15.70:0/3788260002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530c1096a0 0x7f530c109a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:15.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.549+0000 7f5311f74700 1 --2- 172.21.15.70:0/3788260002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f530c103590 0x7f530c103a10 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:15.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.549+0000 7f5311f74700 1 -- 172.21.15.70:0/3788260002 >> 172.21.15.70:0/3788260002 conn(0x7f530c07a6f0 msgr2=0x7f530c07ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:15.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.549+0000 7f5311f74700 1 -- 172.21.15.70:0/3788260002 shutdown_connections 2024-06-04T05:20:15.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.549+0000 7f5311f74700 1 -- 172.21.15.70:0/3788260002 wait complete. 2024-06-04T05:20:15.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.550+0000 7f5311f74700 1 Processor -- start 2024-06-04T05:20:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.551+0000 7f5311f74700 1 -- start start 2024-06-04T05:20:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.551+0000 7f5311f74700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530c103590 0x7f530c10f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.551+0000 7f5311f74700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f530c103f50 0x7f530c10fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.551+0000 7f5311f74700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f530c1096a0 0x7f530c1a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.551+0000 7f5311f74700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f530c113620 con 0x7f530c103f50 2024-06-04T05:20:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.551+0000 7f5311f74700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f530c1134a0 con 0x7f530c1096a0 2024-06-04T05:20:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.551+0000 7f5311f74700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f530c1137a0 con 0x7f530c103590 2024-06-04T05:20:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.551+0000 7f530b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530c103590 0x7f530c10f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.551+0000 7f530b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530c103590 0x7f530c10f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:33958/0 (socket says 172.21.15.70:33958) 2024-06-04T05:20:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.551+0000 7f530b7fe700 1 -- 172.21.15.70:0/3441172352 learned_addr learned my addr 172.21.15.70:0/3441172352 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.551+0000 7f530bfff700 1 --2- 172.21.15.70:0/3441172352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f530c1096a0 0x7f530c1a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:15.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.551+0000 7f530affd700 1 --2- 172.21.15.70:0/3441172352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f530c103f50 0x7f530c10fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:15.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.551+0000 7f530b7fe700 1 -- 172.21.15.70:0/3441172352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f530c1096a0 msgr2=0x7f530c1a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:15.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.551+0000 7f530b7fe700 1 --2- 172.21.15.70:0/3441172352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f530c1096a0 0x7f530c1a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:15.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.551+0000 7f530b7fe700 1 -- 172.21.15.70:0/3441172352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f530c103f50 msgr2=0x7f530c10fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:15.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.551+0000 7f530b7fe700 1 --2- 172.21.15.70:0/3441172352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f530c103f50 0x7f530c10fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:15.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.551+0000 7f530b7fe700 1 -- 172.21.15.70:0/3441172352 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530c1a50d0 con 0x7f530c103590 2024-06-04T05:20:15.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.552+0000 7f530b7fe700 1 --2- 172.21.15.70:0/3441172352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530c103590 0x7f530c10f790 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f52fc00e9a0 tx=0x7f52fc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:15.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.552+0000 7f5308ff9700 1 -- 172.21.15.70:0/3441172352 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc00cbd0 con 0x7f530c103590 2024-06-04T05:20:15.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.552+0000 7f5308ff9700 1 -- 172.21.15.70:0/3441172352 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f52fc00cd70 con 0x7f530c103590 2024-06-04T05:20:15.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.552+0000 7f5311f74700 1 -- 172.21.15.70:0/3441172352 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f530c1a53c0 con 0x7f530c103590 2024-06-04T05:20:15.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.552+0000 7f5308ff9700 1 -- 172.21.15.70:0/3441172352 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc018680 con 0x7f530c103590 2024-06-04T05:20:15.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.552+0000 7f5311f74700 1 -- 172.21.15.70:0/3441172352 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f530c1acc40 con 0x7f530c103590 2024-06-04T05:20:15.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.553+0000 7f5311f74700 1 -- 172.21.15.70:0/3441172352 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f530c06a230 con 0x7f530c103590 2024-06-04T05:20:15.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.555+0000 7f5308ff9700 1 -- 172.21.15.70:0/3441172352 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f52fc010430 con 0x7f530c103590 2024-06-04T05:20:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.556+0000 7f5308ff9700 1 --2- 172.21.15.70:0/3441172352 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52f40b2b40 0x7f52f40b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.556+0000 7f5308ff9700 1 -- 172.21.15.70:0/3441172352 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f52fc0e2840 con 0x7f530c103590 2024-06-04T05:20:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.556+0000 7f530affd700 1 --2- 172.21.15.70:0/3441172352 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52f40b2b40 0x7f52f40b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:15.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.557+0000 7f530affd700 1 --2- 172.21.15.70:0/3441172352 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52f40b2b40 0x7f52f40b5000 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f530000b2e0 tx=0x7f530001a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.561+0000 7f5308ff9700 1 -- 172.21.15.70:0/3441172352 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f52fc0aa190 con 0x7f530c103590 2024-06-04T05:20:15.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 69362 6.67 MB/sec execute 427 sec latency 90.878 ms 2024-06-04T05:20:15.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:15 smithi070 ceph-mon[32631]: pgmap v727: 97 pgs: 97 active+clean; 985 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.6 MiB/s wr, 312 op/s 2024-06-04T05:20:15.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.980+0000 7f5311f74700 1 -- 172.21.15.70:0/3441172352 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f530c1a1420 con 0x7f530c103590 2024-06-04T05:20:15.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.981+0000 7f5308ff9700 1 -- 172.21.15.70:0/3441172352 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+11933 (secure 0 0 0) 0x7f52fc0a5070 con 0x7f530c103590 2024-06-04T05:20:15.983 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:20:15.983 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":64,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:20:02.842346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:20:15.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.984+0000 7f5311f74700 1 -- 172.21.15.70:0/3441172352 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52f40b2b40 msgr2=0x7f52f40b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:15.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.984+0000 7f5311f74700 1 --2- 172.21.15.70:0/3441172352 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52f40b2b40 0x7f52f40b5000 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f530000b2e0 tx=0x7f530001a040 comp rx=0 tx=0).stop 2024-06-04T05:20:15.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.984+0000 7f5311f74700 1 -- 172.21.15.70:0/3441172352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530c103590 msgr2=0x7f530c10f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:15.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.984+0000 7f5311f74700 1 --2- 172.21.15.70:0/3441172352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530c103590 0x7f530c10f790 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f52fc00e9a0 tx=0x7f52fc00ecb0 comp rx=0 tx=0).stop 2024-06-04T05:20:15.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.984+0000 7f5311f74700 1 -- 172.21.15.70:0/3441172352 shutdown_connections 2024-06-04T05:20:15.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.984+0000 7f5311f74700 1 --2- 172.21.15.70:0/3441172352 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52f40b2b40 0x7f52f40b5000 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:15.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.984+0000 7f5311f74700 1 --2- 172.21.15.70:0/3441172352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f530c1096a0 0x7f530c1a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:15.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.984+0000 7f5311f74700 1 --2- 172.21.15.70:0/3441172352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530c103590 0x7f530c10f790 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:15.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.984+0000 7f5311f74700 1 --2- 172.21.15.70:0/3441172352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f530c103f50 0x7f530c10fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:15.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.984+0000 7f5311f74700 1 -- 172.21.15.70:0/3441172352 >> 172.21.15.70:0/3441172352 conn(0x7f530c07a6f0 msgr2=0x7f530c102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:15.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.984+0000 7f5311f74700 1 -- 172.21.15.70:0/3441172352 shutdown_connections 2024-06-04T05:20:15.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:15.984+0000 7f5311f74700 1 -- 172.21.15.70:0/3441172352 wait complete. 2024-06-04T05:20:15.987 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 64 2024-06-04T05:20:16.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:15 smithi112 ceph-mon[36453]: pgmap v727: 97 pgs: 97 active+clean; 985 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.6 MiB/s wr, 312 op/s 2024-06-04T05:20:16.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:15 smithi098 ceph-mon[35569]: pgmap v727: 97 pgs: 97 active+clean; 985 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.6 MiB/s wr, 312 op/s 2024-06-04T05:20:16.769 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:20:16.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 69558 6.67 MB/sec execute 428 sec latency 75.251 ms 2024-06-04T05:20:17.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:16 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3441172352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:20:17.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:16 smithi070 ceph-mon[32631]: pgmap v728: 97 pgs: 97 active+clean; 1003 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 304 op/s 2024-06-04T05:20:17.117 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:20:17.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:16 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3441172352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:20:17.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:16 smithi112 ceph-mon[36453]: pgmap v728: 97 pgs: 97 active+clean; 1003 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 304 op/s 2024-06-04T05:20:17.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:16 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3441172352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:20:17.245 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:16 smithi098 ceph-mon[35569]: pgmap v728: 97 pgs: 97 active+clean; 1003 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 304 op/s 2024-06-04T05:20:17.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 69725 6.68 MB/sec execute 429 sec latency 79.663 ms 2024-06-04T05:20:18.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 69877 6.69 MB/sec execute 430 sec latency 155.380 ms 2024-06-04T05:20:19.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.116+0000 7f96bba63700 1 -- 172.21.15.70:0/1538753524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96b41096d0 msgr2=0x7f96b4109ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.116+0000 7f96bba63700 1 --2- 172.21.15.70:0/1538753524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96b41096d0 0x7f96b4109ab0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f96a4009990 tx=0x7f96a4009ca0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.116+0000 7f96bba63700 1 -- 172.21.15.70:0/1538753524 shutdown_connections 2024-06-04T05:20:19.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.116+0000 7f96bba63700 1 --2- 172.21.15.70:0/1538753524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b4103f80 0x7f96b4110460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.116+0000 7f96bba63700 1 --2- 172.21.15.70:0/1538753524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96b41096d0 0x7f96b4109ab0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.116+0000 7f96bba63700 1 --2- 172.21.15.70:0/1538753524 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96b41035c0 0x7f96b4103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.116+0000 7f96bba63700 1 -- 172.21.15.70:0/1538753524 >> 172.21.15.70:0/1538753524 conn(0x7f96b407a6f0 msgr2=0x7f96b407ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:19.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.116+0000 7f96bba63700 1 -- 172.21.15.70:0/1538753524 shutdown_connections 2024-06-04T05:20:19.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.116+0000 7f96bba63700 1 -- 172.21.15.70:0/1538753524 wait complete. 2024-06-04T05:20:19.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.117+0000 7f96bba63700 1 Processor -- start 2024-06-04T05:20:19.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.117+0000 7f96bba63700 1 -- start start 2024-06-04T05:20:19.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.117+0000 7f96bba63700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96b41035c0 0x7f96b419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.117+0000 7f96bba63700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b4103f80 0x7f96b419fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.117+0000 7f96bba63700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96b41096d0 0x7f96b41999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.117+0000 7f96bba63700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f96b4113660 con 0x7f96b41035c0 2024-06-04T05:20:19.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.117+0000 7f96b97ff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96b41035c0 0x7f96b419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.118+0000 7f96bba63700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96b41134e0 con 0x7f96b4103f80 2024-06-04T05:20:19.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.118+0000 7f96b97ff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96b41035c0 0x7f96b419f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37656/0 (socket says 172.21.15.70:37656) 2024-06-04T05:20:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.118+0000 7f96b97ff700 1 -- 172.21.15.70:0/3958062782 learned_addr learned my addr 172.21.15.70:0/3958062782 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.118+0000 7f96b8ffe700 1 --2- 172.21.15.70:0/3958062782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b4103f80 0x7f96b419fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.118+0000 7f96ba000700 1 --2- 172.21.15.70:0/3958062782 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96b41096d0 0x7f96b41999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.118+0000 7f96bba63700 1 -- 172.21.15.70:0/3958062782 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f96b41137e0 con 0x7f96b41096d0 2024-06-04T05:20:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.118+0000 7f96b97ff700 1 -- 172.21.15.70:0/3958062782 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96b41096d0 msgr2=0x7f96b41999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.118+0000 7f96b97ff700 1 --2- 172.21.15.70:0/3958062782 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96b41096d0 0x7f96b41999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.118+0000 7f96b97ff700 1 -- 172.21.15.70:0/3958062782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b4103f80 msgr2=0x7f96b419fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.118+0000 7f96b97ff700 1 --2- 172.21.15.70:0/3958062782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b4103f80 0x7f96b419fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.118+0000 7f96b97ff700 1 -- 172.21.15.70:0/3958062782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b419a140 con 0x7f96b41035c0 2024-06-04T05:20:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.118+0000 7f96b97ff700 1 --2- 172.21.15.70:0/3958062782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96b41035c0 0x7f96b419f7d0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f96a400b650 tx=0x7f96a400bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.119+0000 7f96b27fc700 1 -- 172.21.15.70:0/3958062782 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a4027070 con 0x7f96b41035c0 2024-06-04T05:20:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.119+0000 7f96b27fc700 1 -- 172.21.15.70:0/3958062782 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96a4022470 con 0x7f96b41035c0 2024-06-04T05:20:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.119+0000 7f96b27fc700 1 -- 172.21.15.70:0/3958062782 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a4017650 con 0x7f96b41035c0 2024-06-04T05:20:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.119+0000 7f96bba63700 1 -- 172.21.15.70:0/3958062782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b419a3d0 con 0x7f96b41035c0 2024-06-04T05:20:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.119+0000 7f96bba63700 1 -- 172.21.15.70:0/3958062782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b419a7a0 con 0x7f96b41035c0 2024-06-04T05:20:19.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.123+0000 7f96b27fc700 1 -- 172.21.15.70:0/3958062782 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f96a4009d10 con 0x7f96b41035c0 2024-06-04T05:20:19.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.126+0000 7f96b27fc700 1 --2- 172.21.15.70:0/3958062782 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f969c0b6e80 0x7f969c0b9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.126+0000 7f96b27fc700 1 -- 172.21.15.70:0/3958062782 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f96a40e2ff0 con 0x7f96b41035c0 2024-06-04T05:20:19.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.126+0000 7f96b8ffe700 1 --2- 172.21.15.70:0/3958062782 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f969c0b6e80 0x7f969c0b9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.127+0000 7f96b8ffe700 1 --2- 172.21.15.70:0/3958062782 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f969c0b6e80 0x7f969c0b9340 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f96b407add0 tx=0x7f96a800b3c0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.127+0000 7f96c62f8b80 1 Processor -- start 2024-06-04T05:20:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.127+0000 7f96c62f8b80 1 -- start start 2024-06-04T05:20:19.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.127+0000 7f96c62f8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9ba70 0x560a3dc9be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.127+0000 7f96c62f8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9c420 0x560a3dc960b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.127+0000 7f96c62f8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc966d0 0x560a3dc96b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.128+0000 7f96c62f8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560a3dbebd70 con 0x560a3dc9c420 2024-06-04T05:20:19.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.128+0000 7f96c62f8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560a3dbdb910 con 0x560a3dc966d0 2024-06-04T05:20:19.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.128+0000 7f96c62f8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560a3dbdf950 con 0x560a3dc9ba70 2024-06-04T05:20:19.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.128+0000 7f96b97ff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9ba70 0x560a3dc9be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.128+0000 7f96b97ff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9ba70 0x560a3dc9be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:33330/0 (socket says 172.21.15.70:33330) 2024-06-04T05:20:19.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.128+0000 7f96b97ff700 1 -- 172.21.15.70:0/1491459928 learned_addr learned my addr 172.21.15.70:0/1491459928 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:19.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.128+0000 7f96b8ffe700 1 --2- 172.21.15.70:0/1491459928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9c420 0x560a3dc960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.128+0000 7f96ba000700 1 --2- 172.21.15.70:0/1491459928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc966d0 0x560a3dc96b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.128+0000 7f96b8ffe700 1 -- 172.21.15.70:0/1491459928 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9ba70 msgr2=0x560a3dc9be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:19.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.128+0000 7f96b8ffe700 1 --2- 172.21.15.70:0/1491459928 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9ba70 0x560a3dc9be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.128+0000 7f96b8ffe700 1 -- 172.21.15.70:0/1491459928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc966d0 msgr2=0x560a3dc96b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:19.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.128+0000 7f96b8ffe700 1 --2- 172.21.15.70:0/1491459928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc966d0 0x560a3dc96b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.128+0000 7f96b8ffe700 1 -- 172.21.15.70:0/1491459928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a3db84fb0 con 0x560a3dc9c420 2024-06-04T05:20:19.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.128+0000 7f96b97ff700 1 --2- 172.21.15.70:0/1491459928 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9ba70 0x560a3dc9be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:19.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.128+0000 7f96b8ffe700 1 --2- 172.21.15.70:0/1491459928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9c420 0x560a3dc960b0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f96a8003a20 tx=0x7f96a8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e564b022d55c00f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:19.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.129+0000 7f969b7fe700 1 -- 172.21.15.70:0/1491459928 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8004a30 con 0x560a3dc9c420 2024-06-04T05:20:19.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.129+0000 7f969b7fe700 1 -- 172.21.15.70:0/1491459928 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96a80054b0 con 0x560a3dc9c420 2024-06-04T05:20:19.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.129+0000 7f969b7fe700 1 -- 172.21.15.70:0/1491459928 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a80119b0 con 0x560a3dc9c420 2024-06-04T05:20:19.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.130+0000 7f96c62f8b80 1 -- 172.21.15.70:0/1491459928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9c420 msgr2=0x560a3dc960b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.130+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/1491459928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9c420 0x560a3dc960b0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f96a8003a20 tx=0x7f96a8013000 comp rx=0 tx=0).stop 2024-06-04T05:20:19.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.130+0000 7f96c62f8b80 1 -- 172.21.15.70:0/1491459928 shutdown_connections 2024-06-04T05:20:19.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.130+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/1491459928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc966d0 0x560a3dc96b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.130+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/1491459928 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9ba70 0x560a3dc9be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.130+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/1491459928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9c420 0x560a3dc960b0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.130+0000 7f96c62f8b80 1 -- 172.21.15.70:0/1491459928 >> 172.21.15.70:0/1491459928 conn(0x560a3dbb9240 msgr2=0x560a3dc883e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:19.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.130+0000 7f96c62f8b80 1 -- 172.21.15.70:0/1491459928 shutdown_connections 2024-06-04T05:20:19.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.130+0000 7f96c62f8b80 1 -- 172.21.15.70:0/1491459928 wait complete. 2024-06-04T05:20:19.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.130+0000 7f96c62f8b80 1 -- start start 2024-06-04T05:20:19.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.130+0000 7f96c62f8b80 1 Processor -- start 2024-06-04T05:20:19.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96c62f8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc966d0 0x560a3dca2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96c62f8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9ba70 0x560a3dc9f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96c62f8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc9c420 0x560a3dc9fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96c62f8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560a3dae1fa0 con 0x560a3dc966d0 2024-06-04T05:20:19.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96c62f8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560a3db43da0 con 0x560a3dc9c420 2024-06-04T05:20:19.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96c62f8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560a3dca2ea0 con 0x560a3dc9ba70 2024-06-04T05:20:19.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96c62f8b80 10 client.0 _ll_register_callbacks cb 0x560a3db76950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:19.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96ba000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc9c420 0x560a3dc9fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96b97ff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc966d0 0x560a3dca2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96b8ffe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9ba70 0x560a3dc9f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96ba000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc9c420 0x560a3dc9fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59860/0 (socket says 172.21.15.70:59860) 2024-06-04T05:20:19.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96b8ffe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9ba70 0x560a3dc9f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:33340/0 (socket says 172.21.15.70:33340) 2024-06-04T05:20:19.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96ba000700 1 -- 172.21.15.70:0/4136505381 learned_addr learned my addr 172.21.15.70:0/4136505381 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:19.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96b97ff700 1 -- 172.21.15.70:0/4136505381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9ba70 msgr2=0x560a3dc9f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:19.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96b97ff700 1 --2- 172.21.15.70:0/4136505381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9ba70 0x560a3dc9f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96b97ff700 1 -- 172.21.15.70:0/4136505381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc9c420 msgr2=0x560a3dc9fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:19.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96b97ff700 1 --2- 172.21.15.70:0/4136505381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc9c420 0x560a3dc9fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96b97ff700 1 -- 172.21.15.70:0/4136505381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a3db2b8d0 con 0x560a3dc966d0 2024-06-04T05:20:19.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96ba000700 1 --2- 172.21.15.70:0/4136505381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc9c420 0x560a3dc9fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:19.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f96b97ff700 1 --2- 172.21.15.70:0/4136505381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc966d0 0x560a3dca2210 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f96a40aa030 tx=0x7f96a4022c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87e3160bfbeeda66 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:19.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.131+0000 7f9698ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:20:19.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.132+0000 7f9698ff9700 1 -- 172.21.15.70:0/4136505381 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a40c4060 con 0x560a3dc966d0 2024-06-04T05:20:19.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.132+0000 7f96c62f8b80 1 -- 172.21.15.70:0/4136505381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96a80054b0 con 0x560a3dc966d0 2024-06-04T05:20:19.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.132+0000 7f9698ff9700 1 -- 172.21.15.70:0/4136505381 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f96a40b2c70 con 0x560a3dc966d0 2024-06-04T05:20:19.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.132+0000 7f9698ff9700 1 -- 172.21.15.70:0/4136505381 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f96a40e0e60 con 0x560a3dc966d0 2024-06-04T05:20:19.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.132+0000 7f96c62f8b80 10 client.21306 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:19.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.132+0000 7f96c62f8b80 1 -- 172.21.15.70:0/4136505381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x560a3db95760 con 0x560a3dc966d0 2024-06-04T05:20:19.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.132+0000 7f9698ff9700 1 -- 172.21.15.70:0/4136505381 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96a80054b0 con 0x560a3dc966d0 2024-06-04T05:20:19.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.132+0000 7f9698ff9700 1 -- 172.21.15.70:0/4136505381 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f96a40ae170 con 0x560a3dc966d0 2024-06-04T05:20:19.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.132+0000 7f96c62f8b80 10 client.21306 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:19.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.132+0000 7f96c62f8b80 10 client.21306 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:20:19.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.132+0000 7f96c62f8b80 20 client.21306 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:19.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.132+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/4136505381 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x560a3dcafa90 0x560a3dcb1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.132+0000 7f96c62f8b80 4 client.21306 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:20:19.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.132+0000 7f96c62f8b80 1 -- 172.21.15.70:0/4136505381 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a3dbedcc0 con 0x560a3dcafa90 2024-06-04T05:20:19.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.133+0000 7f96b8ffe700 1 --2- 172.21.15.70:0/4136505381 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x560a3dcafa90 0x560a3dcb1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.133+0000 7f9698ff9700 10 client.21306 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:20:19.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.133+0000 7f96b8ffe700 1 --2- 172.21.15.70:0/4136505381 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x560a3dcafa90 0x560a3dcb1ea0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1914824c3e9b2afe server_cookie=339362bdd4227d20 in_seq=0 out_seq=0 2024-06-04T05:20:19.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.134+0000 7f9698ff9700 1 -- 172.21.15.70:0/4136505381 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560a3dbedcc0 con 0x560a3dcafa90 2024-06-04T05:20:19.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.134+0000 7f9698ff9700 10 client.21306 handle_command_reply: tid=0 2024-06-04T05:20:19.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.134+0000 7f96c62f8b80 1 client.21306 shutdown 2024-06-04T05:20:19.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.134+0000 7f96c62f8b80 1 -- 172.21.15.70:0/4136505381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc966d0 msgr2=0x560a3dca2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.134+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/4136505381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc966d0 0x560a3dca2210 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f96a40aa030 tx=0x7f96a4022c20 comp rx=0 tx=0).stop 2024-06-04T05:20:19.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.134+0000 7f96c62f8b80 1 -- 172.21.15.70:0/4136505381 shutdown_connections 2024-06-04T05:20:19.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.134+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/4136505381 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x560a3dcafa90 0x560a3dcb1ea0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.134+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/4136505381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc9c420 0x560a3dc9fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.134+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/4136505381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc966d0 0x560a3dca2210 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.134+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/4136505381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9ba70 0x560a3dc9f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.134+0000 7f96c62f8b80 1 -- 172.21.15.70:0/4136505381 >> 172.21.15.70:0/4136505381 conn(0x560a3dbb9240 msgr2=0x560a3dc87af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:19.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.134+0000 7f9698ff9700 0 client.21306 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:20:19.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.134+0000 7f96c62f8b80 1 -- 172.21.15.70:0/4136505381 shutdown_connections 2024-06-04T05:20:19.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.134+0000 7f96c62f8b80 1 -- 172.21.15.70:0/4136505381 wait complete. 2024-06-04T05:20:19.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.134+0000 7f96c62f8b80 20 client.21306 trim_cache size 0 max 16384 2024-06-04T05:20:19.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.157+0000 7f96c62f8b80 1 Processor -- start 2024-06-04T05:20:19.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.157+0000 7f96c62f8b80 1 -- start start 2024-06-04T05:20:19.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.157+0000 7f96c62f8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc966d0 0x560a3dc99c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.157+0000 7f96c62f8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9ba70 0x560a3dcabf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.157+0000 7f96c62f8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9c420 0x560a3dc9cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.157+0000 7f96c62f8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f96b41aba50 con 0x560a3dc9ba70 2024-06-04T05:20:19.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.157+0000 7f96c62f8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96b41ad620 con 0x560a3dc966d0 2024-06-04T05:20:19.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.157+0000 7f96c62f8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f96b41ac180 con 0x560a3dc9c420 2024-06-04T05:20:19.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.157+0000 7f96b8ffe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9ba70 0x560a3dcabf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.158+0000 7f96b97ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc966d0 0x560a3dc99c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.158+0000 7f96b8ffe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9ba70 0x560a3dcabf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37672/0 (socket says 172.21.15.70:37672) 2024-06-04T05:20:19.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.158+0000 7f96b8ffe700 1 -- 172.21.15.70:0/3946889933 learned_addr learned my addr 172.21.15.70:0/3946889933 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:19.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.158+0000 7f96ba000700 1 --2- 172.21.15.70:0/3946889933 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9c420 0x560a3dc9cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.158+0000 7f96b8ffe700 1 -- 172.21.15.70:0/3946889933 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9c420 msgr2=0x560a3dc9cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:19.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.158+0000 7f96b8ffe700 1 --2- 172.21.15.70:0/3946889933 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9c420 0x560a3dc9cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.158+0000 7f96b8ffe700 1 -- 172.21.15.70:0/3946889933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc966d0 msgr2=0x560a3dc99c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:19.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.158+0000 7f96b8ffe700 1 --2- 172.21.15.70:0/3946889933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc966d0 0x560a3dc99c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.158+0000 7f96b8ffe700 1 -- 172.21.15.70:0/3946889933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a3dca0450 con 0x560a3dc9ba70 2024-06-04T05:20:19.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.158+0000 7f96b97ff700 1 --2- 172.21.15.70:0/3946889933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc966d0 0x560a3dc99c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:19.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.158+0000 7f96ba000700 1 --2- 172.21.15.70:0/3946889933 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9c420 0x560a3dc9cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:19.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.158+0000 7f96b8ffe700 1 --2- 172.21.15.70:0/3946889933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9ba70 0x560a3dcabf40 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f96a80039f0 tx=0x7f96a8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e47327a8f6a9dcd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:19.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.159+0000 7f969affd700 1 -- 172.21.15.70:0/3946889933 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8012bf0 con 0x560a3dc9ba70 2024-06-04T05:20:19.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.159+0000 7f969affd700 1 -- 172.21.15.70:0/3946889933 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96a8012d90 con 0x560a3dc9ba70 2024-06-04T05:20:19.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.159+0000 7f969affd700 1 -- 172.21.15.70:0/3946889933 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8033e20 con 0x560a3dc9ba70 2024-06-04T05:20:19.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.159+0000 7f96c62f8b80 1 -- 172.21.15.70:0/3946889933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9ba70 msgr2=0x560a3dcabf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.159+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/3946889933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9ba70 0x560a3dcabf40 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f96a80039f0 tx=0x7f96a8016000 comp rx=0 tx=0).stop 2024-06-04T05:20:19.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.160+0000 7f96c62f8b80 1 -- 172.21.15.70:0/3946889933 shutdown_connections 2024-06-04T05:20:19.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.160+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/3946889933 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc9c420 0x560a3dc9cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.160+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/3946889933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc966d0 0x560a3dc99c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.160+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/3946889933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9ba70 0x560a3dcabf40 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.160+0000 7f96c62f8b80 1 -- 172.21.15.70:0/3946889933 >> 172.21.15.70:0/3946889933 conn(0x560a3dbb9240 msgr2=0x560a3dc9c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:19.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.160+0000 7f96c62f8b80 1 -- 172.21.15.70:0/3946889933 shutdown_connections 2024-06-04T05:20:19.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.160+0000 7f96c62f8b80 1 -- 172.21.15.70:0/3946889933 wait complete. 2024-06-04T05:20:19.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.160+0000 7f96c62f8b80 1 -- start start 2024-06-04T05:20:19.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.160+0000 7f96c62f8b80 1 Processor -- start 2024-06-04T05:20:19.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96c62f8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc966d0 0x560a3dca8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96c62f8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9ba70 0x560a3dca8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96c62f8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc9c420 0x560a3dcb68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96c62f8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f96b41acf20 con 0x560a3dc9ba70 2024-06-04T05:20:19.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96c62f8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96b41ac850 con 0x560a3dc9c420 2024-06-04T05:20:19.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96c62f8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f96b41a9a30 con 0x560a3dc966d0 2024-06-04T05:20:19.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96c62f8b80 10 client.0 _ll_register_callbacks cb 0x560a3db76950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:19.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96b97ff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc966d0 0x560a3dca8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96b97ff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc966d0 0x560a3dca8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:33356/0 (socket says 172.21.15.70:33356) 2024-06-04T05:20:19.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96b97ff700 1 -- 172.21.15.70:0/2536477907 learned_addr learned my addr 172.21.15.70:0/2536477907 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:19.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96b8ffe700 1 --2- 172.21.15.70:0/2536477907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9ba70 0x560a3dca8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96ba000700 1 --2- 172.21.15.70:0/2536477907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc9c420 0x560a3dcb68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96b97ff700 1 -- 172.21.15.70:0/2536477907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc9c420 msgr2=0x560a3dcb68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:19.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96b97ff700 1 --2- 172.21.15.70:0/2536477907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc9c420 0x560a3dcb68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96b97ff700 1 -- 172.21.15.70:0/2536477907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9ba70 msgr2=0x560a3dca8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:19.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96b97ff700 1 --2- 172.21.15.70:0/2536477907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9ba70 0x560a3dca8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96b97ff700 1 -- 172.21.15.70:0/2536477907 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a3dcb6fb0 con 0x560a3dc966d0 2024-06-04T05:20:19.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.161+0000 7f96ba000700 1 --2- 172.21.15.70:0/2536477907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc9c420 0x560a3dcb68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:20:19.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.162+0000 7f96b97ff700 1 --2- 172.21.15.70:0/2536477907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc966d0 0x560a3dca8130 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f96a4006010 tx=0x7f96a40a60f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14e038ac45d0fe6e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:19.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.162+0000 7f9698ff9700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:20:19.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.162+0000 7f9698ff9700 1 -- 172.21.15.70:0/2536477907 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a40b3100 con 0x560a3dc966d0 2024-06-04T05:20:19.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.162+0000 7f9698ff9700 1 -- 172.21.15.70:0/2536477907 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f96a40b2e80 con 0x560a3dc966d0 2024-06-04T05:20:19.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.162+0000 7f96c62f8b80 1 -- 172.21.15.70:0/2536477907 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96a8012d90 con 0x560a3dc966d0 2024-06-04T05:20:19.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.162+0000 7f9698ff9700 1 -- 172.21.15.70:0/2536477907 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f96a40cb200 con 0x560a3dc966d0 2024-06-04T05:20:19.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.162+0000 7f96c62f8b80 10 client.30100 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:19.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.162+0000 7f96c62f8b80 1 -- 172.21.15.70:0/2536477907 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x560a3dd1a000 con 0x560a3dc966d0 2024-06-04T05:20:19.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.162+0000 7f9698ff9700 1 -- 172.21.15.70:0/2536477907 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96a40ae6e0 con 0x560a3dc966d0 2024-06-04T05:20:19.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.162+0000 7f9698ff9700 1 -- 172.21.15.70:0/2536477907 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f96a40a9120 con 0x560a3dc966d0 2024-06-04T05:20:19.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.162+0000 7f96c62f8b80 10 client.30100 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:19.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.162+0000 7f96c62f8b80 10 client.30100 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:20:19.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.162+0000 7f96c62f8b80 20 client.30100 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:19.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.162+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/2536477907 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x560a3dd1a370 0x560a3dd1c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.162+0000 7f96c62f8b80 4 client.30100 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:20:19.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.162+0000 7f96c62f8b80 1 -- 172.21.15.70:0/2536477907 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f96a8012d90 con 0x560a3dd1a370 2024-06-04T05:20:19.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.163+0000 7f96b8ffe700 1 --2- 172.21.15.70:0/2536477907 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x560a3dd1a370 0x560a3dd1c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.163+0000 7f9698ff9700 10 client.30100 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:20:19.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.163+0000 7f96b8ffe700 1 --2- 172.21.15.70:0/2536477907 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x560a3dd1a370 0x560a3dd1c810 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=726aaae1a848874a server_cookie=4a7802cdf0218c88 in_seq=0 out_seq=0 2024-06-04T05:20:19.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.164+0000 7f9698ff9700 1 -- 172.21.15.70:0/2536477907 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f96a8012d90 con 0x560a3dd1a370 2024-06-04T05:20:19.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.164+0000 7f9698ff9700 10 client.30100 handle_command_reply: tid=0 2024-06-04T05:20:19.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.164+0000 7f96c62f8b80 1 client.30100 shutdown 2024-06-04T05:20:19.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.164+0000 7f96c62f8b80 1 -- 172.21.15.70:0/2536477907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc966d0 msgr2=0x560a3dca8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.164+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/2536477907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc966d0 0x560a3dca8130 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f96a4006010 tx=0x7f96a40a60f0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.164+0000 7f96c62f8b80 1 -- 172.21.15.70:0/2536477907 shutdown_connections 2024-06-04T05:20:19.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.164+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/2536477907 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x560a3dd1a370 0x560a3dd1c810 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.164+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/2536477907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a3dc9c420 0x560a3dcb68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.164+0000 7f9698ff9700 0 client.30100 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:20:19.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.164+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/2536477907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560a3dc966d0 0x560a3dca8130 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.164+0000 7f96c62f8b80 1 --2- 172.21.15.70:0/2536477907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a3dc9ba70 0x560a3dca8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.164+0000 7f96c62f8b80 1 -- 172.21.15.70:0/2536477907 >> 172.21.15.70:0/2536477907 conn(0x560a3dbb9240 msgr2=0x560a3dc9a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:19.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.164+0000 7f96c62f8b80 1 -- 172.21.15.70:0/2536477907 shutdown_connections 2024-06-04T05:20:19.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.164+0000 7f96c62f8b80 1 -- 172.21.15.70:0/2536477907 wait complete. 2024-06-04T05:20:19.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.165+0000 7f96c62f8b80 20 client.30100 trim_cache size 0 max 16384 2024-06-04T05:20:19.170 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:20:19.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.165+0000 7f96bba63700 1 -- 172.21.15.70:0/3958062782 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f969c0b6e80 msgr2=0x7f969c0b9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.165+0000 7f96bba63700 1 --2- 172.21.15.70:0/3958062782 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f969c0b6e80 0x7f969c0b9340 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f96b407add0 tx=0x7f96a800b3c0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.165+0000 7f96bba63700 1 -- 172.21.15.70:0/3958062782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96b41035c0 msgr2=0x7f96b419f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.165+0000 7f96bba63700 1 --2- 172.21.15.70:0/3958062782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96b41035c0 0x7f96b419f7d0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f96a400b650 tx=0x7f96a400bfc0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.165+0000 7f96bba63700 1 -- 172.21.15.70:0/3958062782 shutdown_connections 2024-06-04T05:20:19.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.165+0000 7f96bba63700 1 --2- 172.21.15.70:0/3958062782 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f969c0b6e80 0x7f969c0b9340 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.165+0000 7f96bba63700 1 --2- 172.21.15.70:0/3958062782 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96b41096d0 0x7f96b41999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.165+0000 7f96bba63700 1 --2- 172.21.15.70:0/3958062782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96b41035c0 0x7f96b419f7d0 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.165+0000 7f96bba63700 1 --2- 172.21.15.70:0/3958062782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b4103f80 0x7f96b419fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.165+0000 7f96bba63700 1 -- 172.21.15.70:0/3958062782 >> 172.21.15.70:0/3958062782 conn(0x7f96b407a6f0 msgr2=0x7f96b4102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:19.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.166+0000 7f96bba63700 1 -- 172.21.15.70:0/3958062782 shutdown_connections 2024-06-04T05:20:19.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:19.166+0000 7f96bba63700 1 -- 172.21.15.70:0/3958062782 wait complete. 2024-06-04T05:20:19.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:19 smithi070 ceph-mon[32631]: pgmap v729: 97 pgs: 97 active+clean; 1003 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.4 MiB/s wr, 296 op/s 2024-06-04T05:20:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:19 smithi112 ceph-mon[36453]: pgmap v729: 97 pgs: 97 active+clean; 1003 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.4 MiB/s wr, 296 op/s 2024-06-04T05:20:19.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:19 smithi098 ceph-mon[35569]: pgmap v729: 97 pgs: 97 active+clean; 1003 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.4 MiB/s wr, 296 op/s 2024-06-04T05:20:19.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.795+0000 7f1e99e93640 1 -- 172.21.15.70:0/4029128102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e94102a70 msgr2=0x7f1e94102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.795+0000 7f1e99e93640 1 --2- 172.21.15.70:0/4029128102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e94102a70 0x7f1e94102ef0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c00fb40 tx=0x7f1e7c02e5b0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.795+0000 7f1e99e93640 1 -- 172.21.15.70:0/4029128102 shutdown_connections 2024-06-04T05:20:19.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.795+0000 7f1e99e93640 1 --2- 172.21.15.70:0/4029128102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e94103430 0x7f1e9410f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.795+0000 7f1e99e93640 1 --2- 172.21.15.70:0/4029128102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e94102a70 0x7f1e94102ef0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.795+0000 7f1e99e93640 1 --2- 172.21.15.70:0/4029128102 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e94108a80 0x7f1e94108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.795+0000 7f1e99e93640 1 -- 172.21.15.70:0/4029128102 >> 172.21.15.70:0/4029128102 conn(0x7f1e940fe500 msgr2=0x7f1e94100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.796+0000 7f1e99e93640 1 -- 172.21.15.70:0/4029128102 shutdown_connections 2024-06-04T05:20:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.796+0000 7f1e99e93640 1 -- 172.21.15.70:0/4029128102 wait complete. 2024-06-04T05:20:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e99e93640 1 Processor -- start 2024-06-04T05:20:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e99e93640 1 -- start start 2024-06-04T05:20:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e99e93640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e94102a70 0x7f1e94073f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e99e93640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e94103430 0x7f1e94074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e99e93640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e94108a80 0x7f1e9406dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e99e93640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9406c970 con 0x7f1e94103430 2024-06-04T05:20:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e99e93640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9406c7f0 con 0x7f1e94102a70 2024-06-04T05:20:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e99e93640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9406caf0 con 0x7f1e94108a80 2024-06-04T05:20:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e8b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e94102a70 0x7f1e94073f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e937fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e94103430 0x7f1e94074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e8b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e94102a70 0x7f1e94073f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59888/0 (socket says 172.21.15.70:59888) 2024-06-04T05:20:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e93fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e94108a80 0x7f1e9406dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e8b7fe640 1 -- 172.21.15.70:0/735927342 learned_addr learned my addr 172.21.15.70:0/735927342 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e8b7fe640 1 -- 172.21.15.70:0/735927342 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e94108a80 msgr2=0x7f1e9406dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e8b7fe640 1 --2- 172.21.15.70:0/735927342 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e94108a80 0x7f1e9406dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e8b7fe640 1 -- 172.21.15.70:0/735927342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e94103430 msgr2=0x7f1e94074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e8b7fe640 1 --2- 172.21.15.70:0/735927342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e94103430 0x7f1e94074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.797+0000 7f1e8b7fe640 1 -- 172.21.15.70:0/735927342 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9406e880 con 0x7f1e94102a70 2024-06-04T05:20:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.798+0000 7f1e8b7fe640 1 --2- 172.21.15.70:0/735927342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e94102a70 0x7f1e94073f50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400e9f0 tx=0x7f1e8400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.798+0000 7f1e917fa640 1 -- 172.21.15.70:0/735927342 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8400ccc0 con 0x7f1e94102a70 2024-06-04T05:20:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.798+0000 7f1e99e93640 1 -- 172.21.15.70:0/735927342 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e9406eb70 con 0x7f1e94102a70 2024-06-04T05:20:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.798+0000 7f1e917fa640 1 -- 172.21.15.70:0/735927342 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e8400ce60 con 0x7f1e94102a70 2024-06-04T05:20:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.798+0000 7f1e917fa640 1 -- 172.21.15.70:0/735927342 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e84004e20 con 0x7f1e94102a70 2024-06-04T05:20:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.798+0000 7f1e99e93640 1 -- 172.21.15.70:0/735927342 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e941b4c70 con 0x7f1e94102a70 2024-06-04T05:20:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.801+0000 7f1e917fa640 1 -- 172.21.15.70:0/735927342 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f1e84005000 con 0x7f1e94102a70 2024-06-04T05:20:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.801+0000 7f1e9cbb4740 1 Processor -- start 2024-06-04T05:20:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.801+0000 7f1e9cbb4740 1 -- start start 2024-06-04T05:20:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.801+0000 7f1e9cbb4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdcc2820 0x5613bdcc2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.801+0000 7f1e9cbb4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc3140 0x5613bdccd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.801+0000 7f1e9cbb4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdccda00 0x5613bdccfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.801+0000 7f1e9cbb4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5613bdc34c30 con 0x5613bdcc3140 2024-06-04T05:20:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.801+0000 7f1e9cbb4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5613bdc247d0 con 0x5613bdcc2820 2024-06-04T05:20:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.801+0000 7f1e9cbb4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5613bdc28810 con 0x5613bdccda00 2024-06-04T05:20:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.802+0000 7f1e93fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdcc2820 0x5613bdcc2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.802+0000 7f1e937fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdccda00 0x5613bdccfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.802+0000 7f1e937fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdccda00 0x5613bdccfdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:33384/0 (socket says 172.21.15.70:33384) 2024-06-04T05:20:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.802+0000 7f1e93fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdcc2820 0x5613bdcc2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59898/0 (socket says 172.21.15.70:59898) 2024-06-04T05:20:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.802+0000 7f1e937fe640 1 -- 172.21.15.70:0/888222659 learned_addr learned my addr 172.21.15.70:0/888222659 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.802+0000 7f1e8b7fe640 1 --2- 172.21.15.70:0/888222659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc3140 0x5613bdccd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.802+0000 7f1e937fe640 1 -- 172.21.15.70:0/888222659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdcc2820 msgr2=0x5613bdcc2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.802+0000 7f1e937fe640 1 --2- 172.21.15.70:0/888222659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdcc2820 0x5613bdcc2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.802+0000 7f1e937fe640 1 -- 172.21.15.70:0/888222659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc3140 msgr2=0x5613bdccd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.802+0000 7f1e937fe640 1 --2- 172.21.15.70:0/888222659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc3140 0x5613bdccd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.802+0000 7f1e937fe640 1 -- 172.21.15.70:0/888222659 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613bd939210 con 0x5613bdccda00 2024-06-04T05:20:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.802+0000 7f1e8b7fe640 1 --2- 172.21.15.70:0/888222659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc3140 0x5613bdccd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:20:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.802+0000 7f1e917fa640 1 --2- 172.21.15.70:0/735927342 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1e800b2a90 0x7f1e800b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.802+0000 7f1e937fe640 1 --2- 172.21.15.70:0/888222659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdccda00 0x5613bdccfdf0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c00fb10 tx=0x7f1e7c005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5073472d3ec034c6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.802+0000 7f1e917fa640 1 -- 172.21.15.70:0/735927342 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f1e840e2910 con 0x7f1e94102a70 2024-06-04T05:20:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.803+0000 7f1e89ffb640 1 -- 172.21.15.70:0/888222659 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c00f310 con 0x5613bdccda00 2024-06-04T05:20:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.803+0000 7f1e89ffb640 1 -- 172.21.15.70:0/888222659 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e7c0044d0 con 0x5613bdccda00 2024-06-04T05:20:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.803+0000 7f1e89ffb640 1 -- 172.21.15.70:0/888222659 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c005030 con 0x5613bdccda00 2024-06-04T05:20:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.802+0000 7f1e93fff640 1 --2- 172.21.15.70:0/735927342 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1e800b2a90 0x7f1e800b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/888222659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdccda00 msgr2=0x5613bdccfdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e93fff640 1 --2- 172.21.15.70:0/735927342 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1e800b2a90 0x7f1e800b4f50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c005fd0 tx=0x7f1e8c009500 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/888222659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdccda00 0x5613bdccfdf0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c00fb10 tx=0x7f1e7c005d50 comp rx=0 tx=0).stop 2024-06-04T05:20:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/888222659 shutdown_connections 2024-06-04T05:20:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/888222659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdccda00 0x5613bdccfdf0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/888222659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc3140 0x5613bdccd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/888222659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdcc2820 0x5613bdcc2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/888222659 >> 172.21.15.70:0/888222659 conn(0x5613bdcbed30 msgr2=0x5613bdcbf110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/888222659 shutdown_connections 2024-06-04T05:20:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/888222659 wait complete. 2024-06-04T05:20:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e9cbb4740 1 -- start start 2024-06-04T05:20:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e9cbb4740 1 Processor -- start 2024-06-04T05:20:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e9cbb4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdcc2820 0x5613bdcca900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e9cbb4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc3140 0x5613bdccae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e9cbb4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdccda00 0x5613bdccb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e9cbb4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5613bd90a790 con 0x5613bdcc3140 2024-06-04T05:20:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e9cbb4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5613bdbe0890 con 0x5613bdcc2820 2024-06-04T05:20:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.804+0000 7f1e9cbb4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5613bdcd05d0 con 0x5613bdccda00 2024-06-04T05:20:19.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.805+0000 7f1e9cbb4740 10 client.0 _ll_register_callbacks cb 0x5613bd938a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:19.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.805+0000 7f1e8b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdcc2820 0x5613bdcca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.805+0000 7f1e937fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc3140 0x5613bdccae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.805+0000 7f1e8b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdcc2820 0x5613bdcca900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59906/0 (socket says 172.21.15.70:59906) 2024-06-04T05:20:19.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.805+0000 7f1e8b7fe640 1 -- 172.21.15.70:0/3986050963 learned_addr learned my addr 172.21.15.70:0/3986050963 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:19.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.805+0000 7f1e93fff640 1 --2- 172.21.15.70:0/3986050963 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdccda00 0x5613bdccb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.805+0000 7f1e8b7fe640 1 -- 172.21.15.70:0/3986050963 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdccda00 msgr2=0x5613bdccb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:19.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.805+0000 7f1e8b7fe640 1 --2- 172.21.15.70:0/3986050963 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdccda00 0x5613bdccb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.805+0000 7f1e8b7fe640 1 -- 172.21.15.70:0/3986050963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc3140 msgr2=0x5613bdccae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:19.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.805+0000 7f1e8b7fe640 1 --2- 172.21.15.70:0/3986050963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc3140 0x5613bdccae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.805+0000 7f1e8b7fe640 1 -- 172.21.15.70:0/3986050963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613bda05620 con 0x5613bdcc2820 2024-06-04T05:20:19.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.805+0000 7f1e8b7fe640 1 --2- 172.21.15.70:0/3986050963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdcc2820 0x5613bdcca900 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400d010 tx=0x7f1e840a93f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=802eab816f42e550 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.805+0000 7f1e72ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:20:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.805+0000 7f1e72ffd640 1 -- 172.21.15.70:0/3986050963 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e84005660 con 0x5613bdcc2820 2024-06-04T05:20:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.805+0000 7f1e72ffd640 1 -- 172.21.15.70:0/3986050963 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f1e840a5e50 con 0x5613bdcc2820 2024-06-04T05:20:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.805+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/3986050963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e7c0044d0 con 0x5613bdcc2820 2024-06-04T05:20:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.806+0000 7f1e72ffd640 1 -- 172.21.15.70:0/3986050963 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f1e840a96d0 con 0x5613bdcc2820 2024-06-04T05:20:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.806+0000 7f1e9cbb4740 10 client.29978 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.806+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/3986050963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x5613bdcdc2c0 con 0x5613bdcc2820 2024-06-04T05:20:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.806+0000 7f1e72ffd640 1 -- 172.21.15.70:0/3986050963 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e7c0044d0 con 0x5613bdcc2820 2024-06-04T05:20:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.806+0000 7f1e72ffd640 1 -- 172.21.15.70:0/3986050963 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f1e840b2590 con 0x5613bdcc2820 2024-06-04T05:20:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.806+0000 7f1e9cbb4740 10 client.29978 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.806+0000 7f1e9cbb4740 10 client.29978 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.806+0000 7f1e9cbb4740 20 client.29978 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.806+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/3986050963 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5613bdcdd780 0x5613bdcdfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.807+0000 7f1e9cbb4740 4 client.29978 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:20:19.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.807+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/3986050963 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613bdc36b80 con 0x5613bdcdd780 2024-06-04T05:20:19.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.807+0000 7f1e937fe640 1 --2- 172.21.15.70:0/3986050963 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5613bdcdd780 0x5613bdcdfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.807+0000 7f1e937fe640 1 --2- 172.21.15.70:0/3986050963 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5613bdcdd780 0x5613bdcdfb70 crc :-1 s=READY pgs=485 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3689690614b4315 server_cookie=2a5287f2f76d53f4 in_seq=0 out_seq=0 2024-06-04T05:20:19.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.807+0000 7f1e72ffd640 10 client.29978 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:19.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.808+0000 7f1e72ffd640 1 -- 172.21.15.70:0/3986050963 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5613bdc36b80 con 0x5613bdcdd780 2024-06-04T05:20:19.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.808+0000 7f1e72ffd640 10 client.29978 handle_command_reply: tid=0 2024-06-04T05:20:19.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.808+0000 7f1e9cbb4740 1 client.29978 shutdown 2024-06-04T05:20:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.809+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/3986050963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdcc2820 msgr2=0x5613bdcca900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.809+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/3986050963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdcc2820 0x5613bdcca900 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400d010 tx=0x7f1e840a93f0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.809+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/3986050963 shutdown_connections 2024-06-04T05:20:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.809+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/3986050963 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5613bdcdd780 0x5613bdcdfb70 crc :-1 s=READY pgs=485 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.809+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/3986050963 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdccda00 0x5613bdccb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.809+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/3986050963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc3140 0x5613bdccae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.809+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/3986050963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdcc2820 0x5613bdcca900 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.809+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/3986050963 >> 172.21.15.70:0/3986050963 conn(0x5613bdcbed30 msgr2=0x5613bdcbdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.809+0000 7f1e72ffd640 0 client.29978 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.809+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/3986050963 shutdown_connections 2024-06-04T05:20:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.809+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/3986050963 wait complete. 2024-06-04T05:20:19.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.809+0000 7f1e9cbb4740 20 client.29978 trim_cache size 0 max 16384 2024-06-04T05:20:19.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 70058 6.71 MB/sec execute 431 sec latency 118.165 ms 2024-06-04T05:20:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.837+0000 7f1e9cbb4740 1 Processor -- start 2024-06-04T05:20:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.837+0000 7f1e9cbb4740 1 -- start start 2024-06-04T05:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.837+0000 7f1e9cbb4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc2820 0x5613bdccb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.837+0000 7f1e9cbb4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdcc3140 0x5613bdcbde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.837+0000 7f1e9cbb4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdccda00 0x5613bdcbe370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.837+0000 7f1e937fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdcc3140 0x5613bdcbde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.837+0000 7f1e8b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc2820 0x5613bdccb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.837+0000 7f1e937fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdcc3140 0x5613bdcbde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:33394/0 (socket says 172.21.15.70:33394) 2024-06-04T05:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.838+0000 7f1e937fe640 1 -- 172.21.15.70:0/3494781065 learned_addr learned my addr 172.21.15.70:0/3494781065 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.837+0000 7f1e8b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc2820 0x5613bdccb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37720/0 (socket says 172.21.15.70:37720) 2024-06-04T05:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.838+0000 7f1e93fff640 1 --2- 172.21.15.70:0/3494781065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdccda00 0x5613bdcbe370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.837+0000 7f1e9cbb4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1e941bd4a0 con 0x5613bdcc2820 2024-06-04T05:20:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.839+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/3494781065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e941bf070 con 0x5613bdccda00 2024-06-04T05:20:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.839+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/3494781065 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1e941bdbd0 con 0x5613bdcc3140 2024-06-04T05:20:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.839+0000 7f1e937fe640 1 -- 172.21.15.70:0/3494781065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdccda00 msgr2=0x5613bdcbe370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.839+0000 7f1e937fe640 1 --2- 172.21.15.70:0/3494781065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdccda00 0x5613bdcbe370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.839+0000 7f1e937fe640 1 -- 172.21.15.70:0/3494781065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc2820 msgr2=0x5613bdccb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.839+0000 7f1e937fe640 1 --2- 172.21.15.70:0/3494781065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc2820 0x5613bdccb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.839+0000 7f1e937fe640 1 -- 172.21.15.70:0/3494781065 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613bdcbeb60 con 0x5613bdcc3140 2024-06-04T05:20:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.840+0000 7f1e8b7fe640 1 --2- 172.21.15.70:0/3494781065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc2820 0x5613bdccb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.840+0000 7f1e937fe640 1 --2- 172.21.15.70:0/3494781065 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdcc3140 0x5613bdcbde30 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c02ead0 tx=0x7f1e7c02e5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae6ac5488dd3e08e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:19.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.840+0000 7f1e897fa640 1 -- 172.21.15.70:0/3494781065 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c00f110 con 0x5613bdcc3140 2024-06-04T05:20:19.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.840+0000 7f1e897fa640 1 -- 172.21.15.70:0/3494781065 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e7c00ee70 con 0x5613bdcc3140 2024-06-04T05:20:19.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.840+0000 7f1e897fa640 1 -- 172.21.15.70:0/3494781065 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c005530 con 0x5613bdcc3140 2024-06-04T05:20:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.841+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/3494781065 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdcc3140 msgr2=0x5613bdcbde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.841+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/3494781065 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdcc3140 0x5613bdcbde30 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c02ead0 tx=0x7f1e7c02e5b0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.841+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/3494781065 shutdown_connections 2024-06-04T05:20:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.841+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/3494781065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdccda00 0x5613bdcbe370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.841+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/3494781065 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdcc3140 0x5613bdcbde30 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.841+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/3494781065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc2820 0x5613bdccb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.841+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/3494781065 >> 172.21.15.70:0/3494781065 conn(0x5613bdcbed30 msgr2=0x5613bdcbd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.842+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/3494781065 shutdown_connections 2024-06-04T05:20:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.842+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/3494781065 wait complete. 2024-06-04T05:20:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.842+0000 7f1e9cbb4740 1 -- start start 2024-06-04T05:20:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.842+0000 7f1e9cbb4740 1 Processor -- start 2024-06-04T05:20:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.843+0000 7f1e9cbb4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdcc2820 0x5613bdcd5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.843+0000 7f1e9cbb4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc3140 0x5613bdcd1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.843+0000 7f1e9cbb4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdccda00 0x5613bdcd1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.843+0000 7f1e9cbb4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1e941be970 con 0x5613bdcc3140 2024-06-04T05:20:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.843+0000 7f1e9cbb4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e941be2a0 con 0x5613bdccda00 2024-06-04T05:20:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.843+0000 7f1e9cbb4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1e941bb4a0 con 0x5613bdcc2820 2024-06-04T05:20:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.843+0000 7f1e9cbb4740 10 client.0 _ll_register_callbacks cb 0x7f1e8400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.843+0000 7f1e93fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdccda00 0x5613bdcd1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.843+0000 7f1e93fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdccda00 0x5613bdcd1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59930/0 (socket says 172.21.15.70:59930) 2024-06-04T05:20:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.843+0000 7f1e93fff640 1 -- 172.21.15.70:0/711223200 learned_addr learned my addr 172.21.15.70:0/711223200 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.843+0000 7f1e8b7fe640 1 --2- 172.21.15.70:0/711223200 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdcc2820 0x5613bdcd5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.843+0000 7f1e937fe640 1 --2- 172.21.15.70:0/711223200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc3140 0x5613bdcd1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.843+0000 7f1e93fff640 1 -- 172.21.15.70:0/711223200 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdcc2820 msgr2=0x5613bdcd5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.843+0000 7f1e93fff640 1 --2- 172.21.15.70:0/711223200 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdcc2820 0x5613bdcd5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.843+0000 7f1e93fff640 1 -- 172.21.15.70:0/711223200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc3140 msgr2=0x5613bdcd1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.844+0000 7f1e93fff640 1 --2- 172.21.15.70:0/711223200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc3140 0x5613bdcd1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.844+0000 7f1e93fff640 1 -- 172.21.15.70:0/711223200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613bdcd1d10 con 0x5613bdccda00 2024-06-04T05:20:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.844+0000 7f1e937fe640 1 --2- 172.21.15.70:0/711223200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc3140 0x5613bdcd1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.844+0000 7f1e93fff640 1 --2- 172.21.15.70:0/711223200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdccda00 0x5613bdcd1670 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c004130 tx=0x7f1e8c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68ea55da3dad12e0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.844+0000 7f1e72ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:20:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.844+0000 7f1e72ffd640 1 -- 172.21.15.70:0/711223200 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c02f060 con 0x5613bdccda00 2024-06-04T05:20:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.844+0000 7f1e72ffd640 1 -- 172.21.15.70:0/711223200 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f1e8c00fa80 con 0x5613bdccda00 2024-06-04T05:20:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.844+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/711223200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e7c00ee70 con 0x5613bdccda00 2024-06-04T05:20:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.845+0000 7f1e72ffd640 1 -- 172.21.15.70:0/711223200 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f1e8c00fc80 con 0x5613bdccda00 2024-06-04T05:20:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.845+0000 7f1e9cbb4740 10 client.29984 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.845+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/711223200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x5613bdd48220 con 0x5613bdccda00 2024-06-04T05:20:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.845+0000 7f1e72ffd640 1 -- 172.21.15.70:0/711223200 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e7c00ee70 con 0x5613bdccda00 2024-06-04T05:20:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.845+0000 7f1e72ffd640 1 -- 172.21.15.70:0/711223200 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f1e8c01f400 con 0x5613bdccda00 2024-06-04T05:20:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.845+0000 7f1e9cbb4740 10 client.29984 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.845+0000 7f1e9cbb4740 10 client.29984 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.845+0000 7f1e9cbb4740 20 client.29984 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.845+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/711223200 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5613bdd48960 0x5613bdd4ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.845+0000 7f1e9cbb4740 4 client.29984 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:20:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.845+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/711223200 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5613bdc30360 con 0x5613bdd48960 2024-06-04T05:20:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.845+0000 7f1e8b7fe640 1 --2- 172.21.15.70:0/711223200 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5613bdd48960 0x5613bdd4ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.846+0000 7f1e8b7fe640 1 --2- 172.21.15.70:0/711223200 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5613bdd48960 0x5613bdd4ad80 crc :-1 s=READY pgs=486 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34cd7b12e96105f4 server_cookie=2958be655d15a78d in_seq=0 out_seq=0 2024-06-04T05:20:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.846+0000 7f1e72ffd640 10 client.29984 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.951+0000 7f1e72ffd640 1 -- 172.21.15.70:0/711223200 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5613bdc30360 con 0x5613bdd48960 2024-06-04T05:20:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.951+0000 7f1e72ffd640 10 client.29984 handle_command_reply: tid=0 2024-06-04T05:20:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.951+0000 7f1e9cbb4740 1 client.29984 shutdown 2024-06-04T05:20:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.951+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/711223200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdccda00 msgr2=0x5613bdcd1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.951+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/711223200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdccda00 0x5613bdcd1670 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c004130 tx=0x7f1e8c004a40 comp rx=0 tx=0).stop 2024-06-04T05:20:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.951+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/711223200 shutdown_connections 2024-06-04T05:20:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.951+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/711223200 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5613bdd48960 0x5613bdd4ad80 crc :-1 s=READY pgs=486 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.951+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/711223200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613bdccda00 0x5613bdcd1670 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.951+0000 7f1e72ffd640 0 client.29984 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.951+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/711223200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613bdcc3140 0x5613bdcd1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.951+0000 7f1e9cbb4740 1 --2- 172.21.15.70:0/711223200 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5613bdcc2820 0x5613bdcd5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.951+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/711223200 >> 172.21.15.70:0/711223200 conn(0x5613bdcbed30 msgr2=0x5613bdcbdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.951+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/711223200 shutdown_connections 2024-06-04T05:20:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.951+0000 7f1e9cbb4740 1 -- 172.21.15.70:0/711223200 wait complete. 2024-06-04T05:20:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.952+0000 7f1e9cbb4740 20 client.29984 trim_cache size 0 max 16384 2024-06-04T05:20:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:20:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:20:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:20:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.952+0000 7f1e99e93640 1 -- 172.21.15.70:0/735927342 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1e800b2a90 msgr2=0x7f1e800b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.952+0000 7f1e99e93640 1 --2- 172.21.15.70:0/735927342 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1e800b2a90 0x7f1e800b4f50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c005fd0 tx=0x7f1e8c009500 comp rx=0 tx=0).stop 2024-06-04T05:20:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.952+0000 7f1e99e93640 1 -- 172.21.15.70:0/735927342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e94102a70 msgr2=0x7f1e94073f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.952+0000 7f1e99e93640 1 --2- 172.21.15.70:0/735927342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e94102a70 0x7f1e94073f50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400e9f0 tx=0x7f1e8400edf0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.953+0000 7f1e99e93640 1 -- 172.21.15.70:0/735927342 shutdown_connections 2024-06-04T05:20:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.953+0000 7f1e99e93640 1 --2- 172.21.15.70:0/735927342 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1e800b2a90 0x7f1e800b4f50 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.953+0000 7f1e99e93640 1 --2- 172.21.15.70:0/735927342 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e94108a80 0x7f1e9406dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.953+0000 7f1e99e93640 1 --2- 172.21.15.70:0/735927342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e94103430 0x7f1e94074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.953+0000 7f1e99e93640 1 --2- 172.21.15.70:0/735927342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e94102a70 0x7f1e94073f50 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.953+0000 7f1e99e93640 1 -- 172.21.15.70:0/735927342 >> 172.21.15.70:0/735927342 conn(0x7f1e940fe500 msgr2=0x7f1e940fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.953+0000 7f1e99e93640 1 -- 172.21.15.70:0/735927342 shutdown_connections 2024-06-04T05:20:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:19.953+0000 7f1e99e93640 1 -- 172.21.15.70:0/735927342 wait complete. 2024-06-04T05:20:19.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:20:20.322 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:20:20.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 70226 6.72 MB/sec execute 432 sec latency 118.364 ms 2024-06-04T05:20:21.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:21 smithi070 ceph-mon[32631]: pgmap v730: 97 pgs: 97 active+clean; 1003 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 253 op/s 2024-06-04T05:20:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:21 smithi112 ceph-mon[36453]: pgmap v730: 97 pgs: 97 active+clean; 1003 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 253 op/s 2024-06-04T05:20:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:21 smithi098 ceph-mon[35569]: pgmap v730: 97 pgs: 97 active+clean; 1003 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 253 op/s 2024-06-04T05:20:21.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 70378 6.71 MB/sec execute 433 sec latency 66.608 ms 2024-06-04T05:20:22.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.410+0000 7f1b9a527700 1 -- 172.21.15.70:0/752057463 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b94102440 msgr2=0x7f1b941028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:22.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.410+0000 7f1b9a527700 1 --2- 172.21.15.70:0/752057463 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b94102440 0x7f1b941028c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c009990 tx=0x7f1b7c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.410+0000 7f1b9a527700 1 -- 172.21.15.70:0/752057463 shutdown_connections 2024-06-04T05:20:22.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.410+0000 7f1b9a527700 1 --2- 172.21.15.70:0/752057463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b94102e00 0x7f1b9410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.410+0000 7f1b9a527700 1 --2- 172.21.15.70:0/752057463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b94108520 0x7f1b94108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.410+0000 7f1b9a527700 1 --2- 172.21.15.70:0/752057463 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b94102440 0x7f1b941028c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.410+0000 7f1b9a527700 1 -- 172.21.15.70:0/752057463 >> 172.21.15.70:0/752057463 conn(0x7f1b940793f0 msgr2=0x7f1b94079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:22.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.410+0000 7f1b9a527700 1 -- 172.21.15.70:0/752057463 shutdown_connections 2024-06-04T05:20:22.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.410+0000 7f1b9a527700 1 -- 172.21.15.70:0/752057463 wait complete. 2024-06-04T05:20:22.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.411+0000 7f1b9a527700 1 Processor -- start 2024-06-04T05:20:22.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.411+0000 7f1b9a527700 1 -- start start 2024-06-04T05:20:22.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.412+0000 7f1b9a527700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b94102440 0x7f1b9419f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.412+0000 7f1b9a527700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b94102e00 0x7f1b9419fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.412+0000 7f1b9a527700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b94108520 0x7f1b941999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.412+0000 7f1b93fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b94102440 0x7f1b9419f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:22.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.412+0000 7f1b9a527700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1b94113620 con 0x7f1b94108520 2024-06-04T05:20:22.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.412+0000 7f1b9a527700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941134a0 con 0x7f1b94102e00 2024-06-04T05:20:22.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.412+0000 7f1b9a527700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941137a0 con 0x7f1b94102440 2024-06-04T05:20:22.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.412+0000 7f1b93fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b94102440 0x7f1b9419f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:33426/0 (socket says 172.21.15.70:33426) 2024-06-04T05:20:22.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.412+0000 7f1b93fff700 1 -- 172.21.15.70:0/2422541166 learned_addr learned my addr 172.21.15.70:0/2422541166 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:22.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.412+0000 7f1b98ac4700 1 --2- 172.21.15.70:0/2422541166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b94108520 0x7f1b941999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:22.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.413+0000 7f1b93fff700 1 -- 172.21.15.70:0/2422541166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b94102e00 msgr2=0x7f1b9419fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-04T05:20:22.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.413+0000 7f1b93fff700 1 --2- 172.21.15.70:0/2422541166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b94102e00 0x7f1b9419fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.413+0000 7f1b93fff700 1 -- 172.21.15.70:0/2422541166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b94108520 msgr2=0x7f1b941999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:22.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.413+0000 7f1b93fff700 1 --2- 172.21.15.70:0/2422541166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b94108520 0x7f1b941999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.413+0000 7f1b93fff700 1 -- 172.21.15.70:0/2422541166 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b9419a130 con 0x7f1b94102440 2024-06-04T05:20:22.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.413+0000 7f1b93fff700 1 --2- 172.21.15.70:0/2422541166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b94102440 0x7f1b9419f810 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400ef30 tx=0x7f1b8400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:22.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.413+0000 7f1b91ffb700 1 -- 172.21.15.70:0/2422541166 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b84004bb0 con 0x7f1b94102440 2024-06-04T05:20:22.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.413+0000 7f1b91ffb700 1 -- 172.21.15.70:0/2422541166 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b84004d50 con 0x7f1b94102440 2024-06-04T05:20:22.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.413+0000 7f1b9a527700 1 -- 172.21.15.70:0/2422541166 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b9419a420 con 0x7f1b94102440 2024-06-04T05:20:22.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.414+0000 7f1b91ffb700 1 -- 172.21.15.70:0/2422541166 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b84004260 con 0x7f1b94102440 2024-06-04T05:20:22.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.414+0000 7f1b9a527700 1 -- 172.21.15.70:0/2422541166 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b94110e90 con 0x7f1b94102440 2024-06-04T05:20:22.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.416+0000 7f1b91ffb700 1 -- 172.21.15.70:0/2422541166 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f1b84003a00 con 0x7f1b94102440 2024-06-04T05:20:22.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.417+0000 7f1b91ffb700 1 --2- 172.21.15.70:0/2422541166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1b800b2af0 0x7f1b800b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.417+0000 7f1b91ffb700 1 -- 172.21.15.70:0/2422541166 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f1b840e5f90 con 0x7f1b94102440 2024-06-04T05:20:22.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.417+0000 7f1b8b7fe700 1 --2- 172.21.15.70:0/2422541166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1b800b2af0 0x7f1b800b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:22.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.418+0000 7f1b8b7fe700 1 --2- 172.21.15.70:0/2422541166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1b800b2af0 0x7f1b800b4fb0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0098e0 tx=0x7f1b7c00b610 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:22.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.419+0000 7f1ba4dbcb80 1 Processor -- start 2024-06-04T05:20:22.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.419+0000 7f1ba4dbcb80 1 -- start start 2024-06-04T05:20:22.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.419+0000 7f1ba4dbcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45b860 0x5638fe45bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.419+0000 7f1ba4dbcb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45c210 0x5638fe45f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.419+0000 7f1ba4dbcb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45ff60 0x5638fe4603a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.419+0000 7f1ba4dbcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5638fe3bac90 con 0x5638fe45b860 2024-06-04T05:20:22.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.419+0000 7f1ba4dbcb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5638fe3aa830 con 0x5638fe45c210 2024-06-04T05:20:22.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.419+0000 7f1ba4dbcb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5638fe3ae870 con 0x5638fe45ff60 2024-06-04T05:20:22.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.419+0000 7f1b98ac4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45ff60 0x5638fe4603a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:22.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.419+0000 7f1b98ac4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45ff60 0x5638fe4603a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:33438/0 (socket says 172.21.15.70:33438) 2024-06-04T05:20:22.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.419+0000 7f1b98ac4700 1 -- 172.21.15.70:0/898688987 learned_addr learned my addr 172.21.15.70:0/898688987 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:22.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.419+0000 7f1b98ac4700 1 -- 172.21.15.70:0/898688987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45c210 msgr2=0x5638fe45f940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-04T05:20:22.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.419+0000 7f1b98ac4700 1 --2- 172.21.15.70:0/898688987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45c210 0x5638fe45f940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.419+0000 7f1b98ac4700 1 -- 172.21.15.70:0/898688987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45b860 msgr2=0x5638fe45bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:22.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.420+0000 7f1b93fff700 1 --2- 172.21.15.70:0/898688987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45b860 0x5638fe45bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:22.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.420+0000 7f1b98ac4700 1 --2- 172.21.15.70:0/898688987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45b860 0x5638fe45bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.420+0000 7f1b98ac4700 1 -- 172.21.15.70:0/898688987 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638fe2fa8d0 con 0x5638fe45ff60 2024-06-04T05:20:22.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.420+0000 7f1b93fff700 1 --2- 172.21.15.70:0/898688987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45b860 0x5638fe45bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:20:22.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.420+0000 7f1b98ac4700 1 --2- 172.21.15.70:0/898688987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45ff60 0x5638fe4603a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c009d30 tx=0x7f1b8c011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19bfd20019e951b6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:22.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.420+0000 7f1b8a7fc700 1 -- 172.21.15.70:0/898688987 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c01e070 con 0x5638fe45ff60 2024-06-04T05:20:22.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.420+0000 7f1b8a7fc700 1 -- 172.21.15.70:0/898688987 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b8c019ac0 con 0x5638fe45ff60 2024-06-04T05:20:22.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.420+0000 7f1b8a7fc700 1 -- 172.21.15.70:0/898688987 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c00e9d0 con 0x5638fe45ff60 2024-06-04T05:20:22.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.421+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/898688987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45ff60 msgr2=0x5638fe4603a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:22.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.421+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/898688987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45ff60 0x5638fe4603a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c009d30 tx=0x7f1b8c011720 comp rx=0 tx=0).stop 2024-06-04T05:20:22.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.421+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/898688987 shutdown_connections 2024-06-04T05:20:22.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.421+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/898688987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45ff60 0x5638fe4603a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.421+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/898688987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45b860 0x5638fe45bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.421+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/898688987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45c210 0x5638fe45f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.421+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/898688987 >> 172.21.15.70:0/898688987 conn(0x5638fe389080 msgr2=0x5638fe455e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:22.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.422+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/898688987 shutdown_connections 2024-06-04T05:20:22.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.422+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/898688987 wait complete. 2024-06-04T05:20:22.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.422+0000 7f1ba4dbcb80 1 -- start start 2024-06-04T05:20:22.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.422+0000 7f1ba4dbcb80 1 Processor -- start 2024-06-04T05:20:22.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.423+0000 7f1ba4dbcb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 0x5638fe4674f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.423+0000 7f1ba4dbcb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45c210 0x5638fe4647f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.423+0000 7f1ba4dbcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45ff60 0x5638fe464d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.423+0000 7f1ba4dbcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5638fe2b0fa0 con 0x5638fe45ff60 2024-06-04T05:20:22.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.423+0000 7f1ba4dbcb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5638fe34c7c0 con 0x5638fe45b860 2024-06-04T05:20:22.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.423+0000 7f1ba4dbcb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5638fe468120 con 0x5638fe45c210 2024-06-04T05:20:22.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.423+0000 7f1b93fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 0x5638fe4674f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:22.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.423+0000 7f1b8b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45c210 0x5638fe4647f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:22.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.423+0000 7f1ba4dbcb80 10 client.0 _ll_register_callbacks cb 0x5638fe345920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:22.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.423+0000 7f1b8b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45c210 0x5638fe4647f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:33446/0 (socket says 172.21.15.70:33446) 2024-06-04T05:20:22.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.423+0000 7f1b8b7fe700 1 -- 172.21.15.70:0/2236885743 learned_addr learned my addr 172.21.15.70:0/2236885743 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:22.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.423+0000 7f1b98ac4700 1 --2- 172.21.15.70:0/2236885743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45ff60 0x5638fe464d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:22.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.423+0000 7f1b93fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 0x5638fe4674f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59956/0 (socket says 172.21.15.70:59956) 2024-06-04T05:20:22.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.424+0000 7f1b8b7fe700 1 -- 172.21.15.70:0/2236885743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 msgr2=0x5638fe4674f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.424+0000 7f1b8b7fe700 1 --2- 172.21.15.70:0/2236885743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 0x5638fe4674f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.424+0000 7f1b8b7fe700 1 -- 172.21.15.70:0/2236885743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45ff60 msgr2=0x5638fe464d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.424+0000 7f1b8b7fe700 1 --2- 172.21.15.70:0/2236885743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45ff60 0x5638fe464d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.424+0000 7f1b8b7fe700 1 -- 172.21.15.70:0/2236885743 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638fe349690 con 0x5638fe45c210 2024-06-04T05:20:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.424+0000 7f1b8b7fe700 1 --2- 172.21.15.70:0/2236885743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45c210 0x5638fe4647f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c015000 tx=0x7f1b7c009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fc101c4b1b45ef server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.424+0000 7f1b737fe700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:20:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.424+0000 7f1b737fe700 1 -- 172.21.15.70:0/2236885743 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c003e10 con 0x5638fe45c210 2024-06-04T05:20:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.424+0000 7f1b737fe700 1 -- 172.21.15.70:0/2236885743 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f1b7c00bcd0 con 0x5638fe45c210 2024-06-04T05:20:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.424+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/2236885743 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b8c019ac0 con 0x5638fe45c210 2024-06-04T05:20:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.425+0000 7f1b737fe700 1 -- 172.21.15.70:0/2236885743 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f1b7c023440 con 0x5638fe45c210 2024-06-04T05:20:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.425+0000 7f1ba4dbcb80 10 client.30130 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.425+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/2236885743 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x5638fe4726d0 con 0x5638fe45c210 2024-06-04T05:20:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.425+0000 7f1b737fe700 1 -- 172.21.15.70:0/2236885743 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b8c019ac0 con 0x5638fe45c210 2024-06-04T05:20:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.425+0000 7f1b737fe700 1 -- 172.21.15.70:0/2236885743 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f1b7c0254c0 con 0x5638fe45c210 2024-06-04T05:20:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.425+0000 7f1ba4dbcb80 10 client.30130 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.425+0000 7f1ba4dbcb80 10 client.30130 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:20:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.426+0000 7f1ba4dbcb80 20 client.30130 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.426+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/2236885743 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5638fe474c60 0x5638fe477050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.426+0000 7f1ba4dbcb80 4 client.30130 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:20:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.426+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/2236885743 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638fe4279e0 con 0x5638fe474c60 2024-06-04T05:20:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.426+0000 7f1b93fff700 1 --2- 172.21.15.70:0/2236885743 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5638fe474c60 0x5638fe477050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.426+0000 7f1b737fe700 10 client.30130 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:20:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.426+0000 7f1b93fff700 1 --2- 172.21.15.70:0/2236885743 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5638fe474c60 0x5638fe477050 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bc437a1b73002cc0 server_cookie=31d2734e17d6f4f3 in_seq=0 out_seq=0 2024-06-04T05:20:22.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.427+0000 7f1b737fe700 1 -- 172.21.15.70:0/2236885743 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5638fe4279e0 con 0x5638fe474c60 2024-06-04T05:20:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.427+0000 7f1b737fe700 10 client.30130 handle_command_reply: tid=0 2024-06-04T05:20:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.427+0000 7f1ba4dbcb80 1 client.30130 shutdown 2024-06-04T05:20:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.427+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/2236885743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45c210 msgr2=0x5638fe4647f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.428+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/2236885743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45c210 0x5638fe4647f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c015000 tx=0x7f1b7c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.428+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/2236885743 shutdown_connections 2024-06-04T05:20:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.428+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/2236885743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45ff60 0x5638fe464d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.428+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/2236885743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 0x5638fe4674f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.428+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/2236885743 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5638fe474c60 0x5638fe477050 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.428+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/2236885743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45c210 0x5638fe4647f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.428+0000 7f1b737fe700 0 client.30130 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:20:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.428+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/2236885743 >> 172.21.15.70:0/2236885743 conn(0x5638fe389080 msgr2=0x5638fe461350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:22.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.428+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/2236885743 shutdown_connections 2024-06-04T05:20:22.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.428+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/2236885743 wait complete. 2024-06-04T05:20:22.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.428+0000 7f1ba4dbcb80 20 client.30130 trim_cache size 0 max 16384 2024-06-04T05:20:22.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.451+0000 7f1ba4dbcb80 1 Processor -- start 2024-06-04T05:20:22.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.451+0000 7f1ba4dbcb80 1 -- start start 2024-06-04T05:20:22.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.451+0000 7f1ba4dbcb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 0x5638fe4711c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.451+0000 7f1ba4dbcb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45c210 0x5638fe47c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.451+0000 7f1ba4dbcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45ff60 0x5638fe47cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.451+0000 7f1b8b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45c210 0x5638fe47c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:22.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.451+0000 7f1b98ac4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45ff60 0x5638fe47cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:22.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.451+0000 7f1b98ac4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45ff60 0x5638fe47cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37780/0 (socket says 172.21.15.70:37780) 2024-06-04T05:20:22.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.451+0000 7f1b98ac4700 1 -- 172.21.15.70:0/2237206909 learned_addr learned my addr 172.21.15.70:0/2237206909 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:22.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.451+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/2237206909 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941abd20 con 0x5638fe45ff60 2024-06-04T05:20:22.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.451+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/2237206909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941ad8f0 con 0x5638fe45b860 2024-06-04T05:20:22.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.451+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/2237206909 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941ac450 con 0x5638fe45c210 2024-06-04T05:20:22.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.451+0000 7f1b93fff700 1 --2- 172.21.15.70:0/2237206909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 0x5638fe4711c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:22.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.452+0000 7f1b8b7fe700 1 -- 172.21.15.70:0/2237206909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 msgr2=0x5638fe4711c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:22.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.452+0000 7f1b8b7fe700 1 --2- 172.21.15.70:0/2237206909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 0x5638fe4711c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.452+0000 7f1b8b7fe700 1 -- 172.21.15.70:0/2237206909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45ff60 msgr2=0x5638fe47cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:22.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.452+0000 7f1b8b7fe700 1 --2- 172.21.15.70:0/2237206909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45ff60 0x5638fe47cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.452+0000 7f1b8b7fe700 1 -- 172.21.15.70:0/2237206909 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638fe467860 con 0x5638fe45c210 2024-06-04T05:20:22.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.452+0000 7f1b8b7fe700 1 --2- 172.21.15.70:0/2237206909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45c210 0x5638fe47c960 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0258e0 tx=0x7f1b7c00ba50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e30f6c3986da702a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:22.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.452+0000 7f1b89ffb700 1 -- 172.21.15.70:0/2237206909 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c03c020 con 0x5638fe45c210 2024-06-04T05:20:22.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.452+0000 7f1b89ffb700 1 -- 172.21.15.70:0/2237206909 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b7c00bdd0 con 0x5638fe45c210 2024-06-04T05:20:22.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.452+0000 7f1b89ffb700 1 -- 172.21.15.70:0/2237206909 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c023a70 con 0x5638fe45c210 2024-06-04T05:20:22.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.453+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/2237206909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45c210 msgr2=0x5638fe47c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:22.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.453+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/2237206909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45c210 0x5638fe47c960 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0258e0 tx=0x7f1b7c00ba50 comp rx=0 tx=0).stop 2024-06-04T05:20:22.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.453+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/2237206909 shutdown_connections 2024-06-04T05:20:22.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.453+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/2237206909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45ff60 0x5638fe47cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.453+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/2237206909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 0x5638fe4711c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.453+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/2237206909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45c210 0x5638fe47c960 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.453+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/2237206909 >> 172.21.15.70:0/2237206909 conn(0x5638fe389080 msgr2=0x5638fe463140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:22.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.454+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/2237206909 shutdown_connections 2024-06-04T05:20:22.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.454+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/2237206909 wait complete. 2024-06-04T05:20:22.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.454+0000 7f1ba4dbcb80 1 -- start start 2024-06-04T05:20:22.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.454+0000 7f1ba4dbcb80 1 Processor -- start 2024-06-04T05:20:22.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1ba4dbcb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 0x5638fe46dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1ba4dbcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45c210 0x5638fe46ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1ba4dbcb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45ff60 0x5638fe46b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1ba4dbcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941ad1f0 con 0x5638fe45c210 2024-06-04T05:20:22.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1b93fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 0x5638fe46dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:22.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1ba4dbcb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941acb20 con 0x5638fe45b860 2024-06-04T05:20:22.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1ba4dbcb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941a9ce0 con 0x5638fe45ff60 2024-06-04T05:20:22.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1b93fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 0x5638fe46dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59976/0 (socket says 172.21.15.70:59976) 2024-06-04T05:20:22.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1b98ac4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45ff60 0x5638fe46b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:22.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1b93fff700 1 -- 172.21.15.70:0/3775850596 learned_addr learned my addr 172.21.15.70:0/3775850596 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:22.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1b8b7fe700 1 --2- 172.21.15.70:0/3775850596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45c210 0x5638fe46ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:22.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1ba4dbcb80 10 client.0 _ll_register_callbacks cb 0x5638fe345920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:22.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1b93fff700 1 -- 172.21.15.70:0/3775850596 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45ff60 msgr2=0x5638fe46b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:22.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1b93fff700 1 --2- 172.21.15.70:0/3775850596 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45ff60 0x5638fe46b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1b93fff700 1 -- 172.21.15.70:0/3775850596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45c210 msgr2=0x5638fe46ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:22.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1b93fff700 1 --2- 172.21.15.70:0/3775850596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45c210 0x5638fe46ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1b93fff700 1 -- 172.21.15.70:0/3775850596 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638fe46bc10 con 0x5638fe45b860 2024-06-04T05:20:22.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1b98ac4700 1 --2- 172.21.15.70:0/3775850596 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45ff60 0x5638fe46b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:22.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.455+0000 7f1b8b7fe700 1 --2- 172.21.15.70:0/3775850596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45c210 0x5638fe46ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:22.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.456+0000 7f1b93fff700 1 --2- 172.21.15.70:0/3775850596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 0x5638fe46dad0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1b84004b80 tx=0x7f1b840ad1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90a759aef346d982 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:22.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.456+0000 7f1b737fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:20:22.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.456+0000 7f1b737fe700 1 -- 172.21.15.70:0/3775850596 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b840c6070 con 0x5638fe45b860 2024-06-04T05:20:22.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.456+0000 7f1b737fe700 1 -- 172.21.15.70:0/3775850596 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f1b840e39d0 con 0x5638fe45b860 2024-06-04T05:20:22.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.456+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/3775850596 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b7c00bdd0 con 0x5638fe45b860 2024-06-04T05:20:22.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.456+0000 7f1b737fe700 1 -- 172.21.15.70:0/3775850596 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f1b8401f040 con 0x5638fe45b860 2024-06-04T05:20:22.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.456+0000 7f1ba4dbcb80 10 client.29996 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:22.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.456+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/3775850596 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x5638fe4e8120 con 0x5638fe45b860 2024-06-04T05:20:22.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.456+0000 7f1b737fe700 1 -- 172.21.15.70:0/3775850596 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b7c00bdd0 con 0x5638fe45b860 2024-06-04T05:20:22.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.457+0000 7f1b737fe700 1 -- 172.21.15.70:0/3775850596 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f1b840c7be0 con 0x5638fe45b860 2024-06-04T05:20:22.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.457+0000 7f1ba4dbcb80 10 client.29996 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:22.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.457+0000 7f1ba4dbcb80 10 client.29996 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:20:22.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.457+0000 7f1ba4dbcb80 20 client.29996 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:22.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.457+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/3775850596 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5638fe4e87b0 0x5638fe4eabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:22.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.457+0000 7f1ba4dbcb80 4 client.29996 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:20:22.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.457+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/3775850596 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5638fe3bcbe0 con 0x5638fe4e87b0 2024-06-04T05:20:22.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.457+0000 7f1b8b7fe700 1 --2- 172.21.15.70:0/3775850596 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5638fe4e87b0 0x5638fe4eabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:22.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.457+0000 7f1b8b7fe700 1 --2- 172.21.15.70:0/3775850596 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5638fe4e87b0 0x5638fe4eabf0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=256bbfb6f440b995 server_cookie=d3fb2693cb5a4cb6 in_seq=0 out_seq=0 2024-06-04T05:20:22.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.457+0000 7f1b737fe700 10 client.29996 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:20:22.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.458+0000 7f1b737fe700 1 -- 172.21.15.70:0/3775850596 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5638fe3bcbe0 con 0x5638fe4e87b0 2024-06-04T05:20:22.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.458+0000 7f1b737fe700 10 client.29996 handle_command_reply: tid=0 2024-06-04T05:20:22.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.458+0000 7f1ba4dbcb80 1 client.29996 shutdown 2024-06-04T05:20:22.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.458+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/3775850596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 msgr2=0x5638fe46dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:22.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.458+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/3775850596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 0x5638fe46dad0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1b84004b80 tx=0x7f1b840ad1d0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.458+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/3775850596 shutdown_connections 2024-06-04T05:20:22.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.458+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/3775850596 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5638fe4e87b0 0x5638fe4eabf0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.458+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/3775850596 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638fe45ff60 0x5638fe46b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.458+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/3775850596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638fe45b860 0x5638fe46dad0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.458+0000 7f1ba4dbcb80 1 --2- 172.21.15.70:0/3775850596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638fe45c210 0x5638fe46ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.458+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/3775850596 >> 172.21.15.70:0/3775850596 conn(0x5638fe389080 msgr2=0x5638fe466020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:22.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.458+0000 7f1b737fe700 0 client.29996 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:20:22.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.458+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/3775850596 shutdown_connections 2024-06-04T05:20:22.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.458+0000 7f1ba4dbcb80 1 -- 172.21.15.70:0/3775850596 wait complete. 2024-06-04T05:20:22.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.459+0000 7f1ba4dbcb80 20 client.29996 trim_cache size 0 max 16384 2024-06-04T05:20:22.464 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:20:22.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.459+0000 7f1b9a527700 1 -- 172.21.15.70:0/2422541166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1b800b2af0 msgr2=0x7f1b800b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:22.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.459+0000 7f1b9a527700 1 --2- 172.21.15.70:0/2422541166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1b800b2af0 0x7f1b800b4fb0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0098e0 tx=0x7f1b7c00b610 comp rx=0 tx=0).stop 2024-06-04T05:20:22.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.459+0000 7f1b9a527700 1 -- 172.21.15.70:0/2422541166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b94102440 msgr2=0x7f1b9419f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:22.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.459+0000 7f1b9a527700 1 --2- 172.21.15.70:0/2422541166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b94102440 0x7f1b9419f810 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400ef30 tx=0x7f1b8400c3a0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.460+0000 7f1b9a527700 1 -- 172.21.15.70:0/2422541166 shutdown_connections 2024-06-04T05:20:22.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.460+0000 7f1b9a527700 1 --2- 172.21.15.70:0/2422541166 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1b800b2af0 0x7f1b800b4fb0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.460+0000 7f1b9a527700 1 --2- 172.21.15.70:0/2422541166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b94108520 0x7f1b941999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.460+0000 7f1b9a527700 1 --2- 172.21.15.70:0/2422541166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b94102440 0x7f1b9419f810 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.460+0000 7f1b9a527700 1 --2- 172.21.15.70:0/2422541166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b94102e00 0x7f1b9419fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:22.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.460+0000 7f1b9a527700 1 -- 172.21.15.70:0/2422541166 >> 172.21.15.70:0/2422541166 conn(0x7f1b940793f0 msgr2=0x7f1b94100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:22.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.460+0000 7f1b9a527700 1 -- 172.21.15.70:0/2422541166 shutdown_connections 2024-06-04T05:20:22.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:22.460+0000 7f1b9a527700 1 -- 172.21.15.70:0/2422541166 wait complete. 2024-06-04T05:20:22.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 70502 6.70 MB/sec execute 434 sec latency 75.376 ms 2024-06-04T05:20:23.312 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:20:23.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:23 smithi070 ceph-mon[32631]: pgmap v731: 97 pgs: 97 active+clean; 1010 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.4 MiB/s wr, 421 op/s 2024-06-04T05:20:23.678 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:20:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:23 smithi112 ceph-mon[36453]: pgmap v731: 97 pgs: 97 active+clean; 1010 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.4 MiB/s wr, 421 op/s 2024-06-04T05:20:23.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:23 smithi098 ceph-mon[35569]: pgmap v731: 97 pgs: 97 active+clean; 1010 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.4 MiB/s wr, 421 op/s 2024-06-04T05:20:23.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 70658 6.70 MB/sec execute 435 sec latency 95.384 ms 2024-06-04T05:20:24.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 70820 6.70 MB/sec execute 436 sec latency 151.570 ms 2024-06-04T05:20:25.472 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:25 smithi070 ceph-mon[32631]: pgmap v732: 97 pgs: 97 active+clean; 1010 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.3 MiB/s wr, 279 op/s 2024-06-04T05:20:25.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.634+0000 7f539ad33700 1 -- 172.21.15.70:0/3610868401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5394108510 msgr2=0x7f53941088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:25.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.634+0000 7f539ad33700 1 --2- 172.21.15.70:0/3610868401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5394108510 0x7f53941088f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f537c0098f0 tx=0x7f537c009c00 comp rx=0 tx=0).stop 2024-06-04T05:20:25.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.635+0000 7f539ad33700 1 -- 172.21.15.70:0/3610868401 shutdown_connections 2024-06-04T05:20:25.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.635+0000 7f539ad33700 1 --2- 172.21.15.70:0/3610868401 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5394102df0 0x7f539410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.635+0000 7f539ad33700 1 --2- 172.21.15.70:0/3610868401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5394108510 0x7f53941088f0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.635+0000 7f539ad33700 1 --2- 172.21.15.70:0/3610868401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5394102430 0x7f53941028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.635+0000 7f539ad33700 1 -- 172.21.15.70:0/3610868401 >> 172.21.15.70:0/3610868401 conn(0x7f53940793b0 msgr2=0x7f53940797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:25.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.635+0000 7f539ad33700 1 -- 172.21.15.70:0/3610868401 shutdown_connections 2024-06-04T05:20:25.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.635+0000 7f539ad33700 1 -- 172.21.15.70:0/3610868401 wait complete. 2024-06-04T05:20:25.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.636+0000 7f539ad33700 1 Processor -- start 2024-06-04T05:20:25.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.636+0000 7f539ad33700 1 -- start start 2024-06-04T05:20:25.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.636+0000 7f539ad33700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5394102430 0x7f539419f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.636+0000 7f539ad33700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5394102df0 0x7f539419fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.636+0000 7f539ad33700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5394108510 0x7f53941999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.636+0000 7f539ad33700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f53941135f0 con 0x7f5394108510 2024-06-04T05:20:25.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.636+0000 7f539ad33700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5394113470 con 0x7f5394102430 2024-06-04T05:20:25.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.636+0000 7f539ad33700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5394113770 con 0x7f5394102df0 2024-06-04T05:20:25.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.636+0000 7f53992d0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5394108510 0x7f53941999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:25.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.636+0000 7f5398acf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5394102430 0x7f539419f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:25.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.636+0000 7f53992d0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5394108510 0x7f53941999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37816/0 (socket says 172.21.15.70:37816) 2024-06-04T05:20:25.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.636+0000 7f53992d0700 1 -- 172.21.15.70:0/2689148143 learned_addr learned my addr 172.21.15.70:0/2689148143 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:25.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.636+0000 7f5393fff700 1 --2- 172.21.15.70:0/2689148143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5394102df0 0x7f539419fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:25.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.637+0000 7f5398acf700 1 -- 172.21.15.70:0/2689148143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5394102df0 msgr2=0x7f539419fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:25.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.637+0000 7f5398acf700 1 --2- 172.21.15.70:0/2689148143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5394102df0 0x7f539419fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.637+0000 7f5398acf700 1 -- 172.21.15.70:0/2689148143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5394108510 msgr2=0x7f53941999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:25.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.637+0000 7f5398acf700 1 --2- 172.21.15.70:0/2689148143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5394108510 0x7f53941999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.637+0000 7f5398acf700 1 -- 172.21.15.70:0/2689148143 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f539419a140 con 0x7f5394102430 2024-06-04T05:20:25.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.637+0000 7f53992d0700 1 --2- 172.21.15.70:0/2689148143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5394108510 0x7f53941999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:25.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.637+0000 7f5393fff700 1 --2- 172.21.15.70:0/2689148143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5394102df0 0x7f539419fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:25.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.637+0000 7f5398acf700 1 --2- 172.21.15.70:0/2689148143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5394102430 0x7f539419f820 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f537c009fd0 tx=0x7f537c00f990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:25.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.637+0000 7f5391ffb700 1 -- 172.21.15.70:0/2689148143 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c00b560 con 0x7f5394102430 2024-06-04T05:20:25.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.637+0000 7f5391ffb700 1 -- 172.21.15.70:0/2689148143 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f537c00bb90 con 0x7f5394102430 2024-06-04T05:20:25.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.637+0000 7f5391ffb700 1 -- 172.21.15.70:0/2689148143 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c0038e0 con 0x7f5394102430 2024-06-04T05:20:25.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.637+0000 7f539ad33700 1 -- 172.21.15.70:0/2689148143 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f539419a3d0 con 0x7f5394102430 2024-06-04T05:20:25.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.638+0000 7f539ad33700 1 -- 172.21.15.70:0/2689148143 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f539410fb20 con 0x7f5394102430 2024-06-04T05:20:25.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.640+0000 7f5391ffb700 1 -- 172.21.15.70:0/2689148143 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f537c005240 con 0x7f5394102430 2024-06-04T05:20:25.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.641+0000 7f5391ffb700 1 --2- 172.21.15.70:0/2689148143 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f53800b2a70 0x7f53800b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.641+0000 7f5393fff700 1 --2- 172.21.15.70:0/2689148143 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f53800b2a70 0x7f53800b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:25.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.642+0000 7f5391ffb700 1 -- 172.21.15.70:0/2689148143 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f537c0e3540 con 0x7f5394102430 2024-06-04T05:20:25.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.642+0000 7f5393fff700 1 --2- 172.21.15.70:0/2689148143 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f53800b2a70 0x7f53800b4f30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5394079b90 tx=0x7f538400b040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:25.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.643+0000 7f53a55c8b80 1 Processor -- start 2024-06-04T05:20:25.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.643+0000 7f53a55c8b80 1 -- start start 2024-06-04T05:20:25.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.644+0000 7f53a55c8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 0x564232534700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.644+0000 7f53a55c8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56423252e8a0 0x56423252ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.644+0000 7f53a55c8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252f260 0x56423253b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.644+0000 7f53a55c8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56423248cd70 con 0x56423252f260 2024-06-04T05:20:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.644+0000 7f53a55c8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56423247c910 con 0x56423252e8a0 2024-06-04T05:20:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.644+0000 7f53a55c8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564232480950 con 0x564232534320 2024-06-04T05:20:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.644+0000 7f5398acf700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 0x564232534700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.644+0000 7f5398acf700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 0x564232534700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:33476/0 (socket says 172.21.15.70:33476) 2024-06-04T05:20:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.644+0000 7f5398acf700 1 -- 172.21.15.70:0/1613846471 learned_addr learned my addr 172.21.15.70:0/1613846471 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.644+0000 7f53992d0700 1 --2- 172.21.15.70:0/1613846471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252f260 0x56423253b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.644+0000 7f5393fff700 1 --2- 172.21.15.70:0/1613846471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56423252e8a0 0x56423252ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:25.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.644+0000 7f53992d0700 1 -- 172.21.15.70:0/1613846471 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 msgr2=0x564232534700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:25.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.644+0000 7f53992d0700 1 --2- 172.21.15.70:0/1613846471 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 0x564232534700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.644+0000 7f53992d0700 1 -- 172.21.15.70:0/1613846471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56423252e8a0 msgr2=0x56423252ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:25.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.644+0000 7f53992d0700 1 --2- 172.21.15.70:0/1613846471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56423252e8a0 0x56423252ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.644+0000 7f53992d0700 1 -- 172.21.15.70:0/1613846471 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564232425fb0 con 0x56423252f260 2024-06-04T05:20:25.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.645+0000 7f53992d0700 1 --2- 172.21.15.70:0/1613846471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252f260 0x56423253b080 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f538c011540 tx=0x7f538c011850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809340be84b481ff server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:25.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.645+0000 7f538affd700 1 -- 172.21.15.70:0/1613846471 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c004230 con 0x56423252f260 2024-06-04T05:20:25.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.645+0000 7f538affd700 1 -- 172.21.15.70:0/1613846471 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f538c0043d0 con 0x56423252f260 2024-06-04T05:20:25.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.645+0000 7f538affd700 1 -- 172.21.15.70:0/1613846471 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c00e870 con 0x56423252f260 2024-06-04T05:20:25.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.646+0000 7f53a55c8b80 1 -- 172.21.15.70:0/1613846471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252f260 msgr2=0x56423253b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:25.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.646+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/1613846471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252f260 0x56423253b080 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f538c011540 tx=0x7f538c011850 comp rx=0 tx=0).stop 2024-06-04T05:20:25.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.646+0000 7f53a55c8b80 1 -- 172.21.15.70:0/1613846471 shutdown_connections 2024-06-04T05:20:25.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.646+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/1613846471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252f260 0x56423253b080 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.646+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/1613846471 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 0x564232534700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.646+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/1613846471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56423252e8a0 0x56423252ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.646+0000 7f53a55c8b80 1 -- 172.21.15.70:0/1613846471 >> 172.21.15.70:0/1613846471 conn(0x56423245a240 msgr2=0x5642325293e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:25.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.647+0000 7f53a55c8b80 1 -- 172.21.15.70:0/1613846471 shutdown_connections 2024-06-04T05:20:25.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.647+0000 7f53a55c8b80 1 -- 172.21.15.70:0/1613846471 wait complete. 2024-06-04T05:20:25.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.647+0000 7f53a55c8b80 1 -- start start 2024-06-04T05:20:25.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.647+0000 7f53a55c8b80 1 Processor -- start 2024-06-04T05:20:25.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f53a55c8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252e8a0 0x56423253acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f53a55c8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56423252f260 0x564232537ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f53a55c8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 0x564232538530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f53a55c8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564232382fa0 con 0x56423252e8a0 2024-06-04T05:20:25.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f53a55c8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5642323e4da0 con 0x56423252f260 2024-06-04T05:20:25.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f5398acf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252e8a0 0x56423253acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f53a55c8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56423253b6e0 con 0x564232534320 2024-06-04T05:20:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f5393fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56423252f260 0x564232537ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f5398acf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252e8a0 0x56423253acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37834/0 (socket says 172.21.15.70:37834) 2024-06-04T05:20:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f5398acf700 1 -- 172.21.15.70:0/3015142415 learned_addr learned my addr 172.21.15.70:0/3015142415 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f5393fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56423252f260 0x564232537ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:60016/0 (socket says 172.21.15.70:60016) 2024-06-04T05:20:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f53a55c8b80 10 client.0 _ll_register_callbacks cb 0x564232417950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f53992d0700 1 --2- 172.21.15.70:0/3015142415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 0x564232538530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f53992d0700 1 -- 172.21.15.70:0/3015142415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56423252f260 msgr2=0x564232537ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f53992d0700 1 --2- 172.21.15.70:0/3015142415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56423252f260 0x564232537ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f53992d0700 1 -- 172.21.15.70:0/3015142415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252e8a0 msgr2=0x56423253acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f53992d0700 1 --2- 172.21.15.70:0/3015142415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252e8a0 0x56423253acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f53992d0700 1 -- 172.21.15.70:0/3015142415 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642323cc8d0 con 0x564232534320 2024-06-04T05:20:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f5393fff700 1 --2- 172.21.15.70:0/3015142415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56423252f260 0x564232537ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.648+0000 7f5398acf700 1 --2- 172.21.15.70:0/3015142415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252e8a0 0x56423253acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.649+0000 7f53992d0700 1 --2- 172.21.15.70:0/3015142415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 0x564232538530 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f538c009850 tx=0x7f538c0038b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbc3298564acc300 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.649+0000 7f5373fff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:20:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.649+0000 7f5373fff700 1 -- 172.21.15.70:0/3015142415 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c027070 con 0x564232534320 2024-06-04T05:20:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.649+0000 7f53a55c8b80 1 -- 172.21.15.70:0/3015142415 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f538c0043d0 con 0x564232534320 2024-06-04T05:20:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.649+0000 7f5373fff700 1 -- 172.21.15.70:0/3015142415 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f538c005240 con 0x564232534320 2024-06-04T05:20:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.649+0000 7f5373fff700 1 -- 172.21.15.70:0/3015142415 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f538c033050 con 0x564232534320 2024-06-04T05:20:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.649+0000 7f53a55c8b80 10 client.30148 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.649+0000 7f53a55c8b80 1 -- 172.21.15.70:0/3015142415 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x564232436760 con 0x564232534320 2024-06-04T05:20:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.649+0000 7f5373fff700 1 -- 172.21.15.70:0/3015142415 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f538c0043d0 con 0x564232534320 2024-06-04T05:20:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.650+0000 7f5373fff700 1 -- 172.21.15.70:0/3015142415 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f538c0234d0 con 0x564232534320 2024-06-04T05:20:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.650+0000 7f53a55c8b80 10 client.30148 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.650+0000 7f53a55c8b80 10 client.30148 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:20:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.650+0000 7f53a55c8b80 20 client.30148 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.650+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/3015142415 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5642325509b0 0x564232552da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.650+0000 7f53a55c8b80 4 client.30148 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:20:25.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.650+0000 7f53a55c8b80 1 -- 172.21.15.70:0/3015142415 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56423248ecc0 con 0x5642325509b0 2024-06-04T05:20:25.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.650+0000 7f5398acf700 1 --2- 172.21.15.70:0/3015142415 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5642325509b0 0x564232552da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:25.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.650+0000 7f5398acf700 1 --2- 172.21.15.70:0/3015142415 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5642325509b0 0x564232552da0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f9b74622f7e20166 server_cookie=2306c2fd15bc7b5c in_seq=0 out_seq=0 2024-06-04T05:20:25.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.650+0000 7f5373fff700 10 client.30148 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:20:25.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.652+0000 7f5373fff700 1 -- 172.21.15.70:0/3015142415 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56423248ecc0 con 0x5642325509b0 2024-06-04T05:20:25.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.652+0000 7f5373fff700 10 client.30148 handle_command_reply: tid=0 2024-06-04T05:20:25.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.652+0000 7f53a55c8b80 1 client.30148 shutdown 2024-06-04T05:20:25.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.652+0000 7f53a55c8b80 1 -- 172.21.15.70:0/3015142415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 msgr2=0x564232538530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.652+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/3015142415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 0x564232538530 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f538c009850 tx=0x7f538c0038b0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.652+0000 7f53a55c8b80 1 -- 172.21.15.70:0/3015142415 shutdown_connections 2024-06-04T05:20:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.652+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/3015142415 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x5642325509b0 0x564232552da0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.652+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/3015142415 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 0x564232538530 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.652+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/3015142415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252e8a0 0x56423253acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.652+0000 7f5373fff700 0 client.30148 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:20:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.652+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/3015142415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56423252f260 0x564232537ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.652+0000 7f53a55c8b80 1 -- 172.21.15.70:0/3015142415 >> 172.21.15.70:0/3015142415 conn(0x56423245a240 msgr2=0x564232528af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.652+0000 7f53a55c8b80 1 -- 172.21.15.70:0/3015142415 shutdown_connections 2024-06-04T05:20:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.652+0000 7f53a55c8b80 1 -- 172.21.15.70:0/3015142415 wait complete. 2024-06-04T05:20:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.653+0000 7f53a55c8b80 20 client.30148 trim_cache size 0 max 16384 2024-06-04T05:20:25.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.676+0000 7f53a55c8b80 1 Processor -- start 2024-06-04T05:20:25.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.676+0000 7f53a55c8b80 1 -- start start 2024-06-04T05:20:25.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.676+0000 7f53a55c8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56423252e8a0 0x564232552da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.676+0000 7f53a55c8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252f260 0x564232529270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.676+0000 7f53a55c8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 0x564232536ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.676+0000 7f53a55c8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f53941abb10 con 0x56423252f260 2024-06-04T05:20:25.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.676+0000 7f53a55c8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53941ad6e0 con 0x56423252e8a0 2024-06-04T05:20:25.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.676+0000 7f53a55c8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f53941ac240 con 0x564232534320 2024-06-04T05:20:25.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.676+0000 7f5393fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252f260 0x564232529270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:25.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.676+0000 7f5393fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252f260 0x564232529270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37838/0 (socket says 172.21.15.70:37838) 2024-06-04T05:20:25.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.676+0000 7f5393fff700 1 -- 172.21.15.70:0/1322647534 learned_addr learned my addr 172.21.15.70:0/1322647534 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:25.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.676+0000 7f53992d0700 1 --2- 172.21.15.70:0/1322647534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 0x564232536ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:25.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.677+0000 7f53992d0700 1 -- 172.21.15.70:0/1322647534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56423252e8a0 msgr2=0x564232552da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:25.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.677+0000 7f53992d0700 1 --2- 172.21.15.70:0/1322647534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56423252e8a0 0x564232552da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.677+0000 7f53992d0700 1 -- 172.21.15.70:0/1322647534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252f260 msgr2=0x564232529270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:25.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.677+0000 7f53992d0700 1 --2- 172.21.15.70:0/1322647534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252f260 0x564232529270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.677+0000 7f53992d0700 1 -- 172.21.15.70:0/1322647534 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564232553330 con 0x564232534320 2024-06-04T05:20:25.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.677+0000 7f53992d0700 1 --2- 172.21.15.70:0/1322647534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 0x564232536ff0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f538c0238f0 tx=0x7f538c005d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7135db0cc8aab521 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:25.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.677+0000 7f538a7fc700 1 -- 172.21.15.70:0/1322647534 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c000e60 con 0x564232534320 2024-06-04T05:20:25.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.678+0000 7f538a7fc700 1 -- 172.21.15.70:0/1322647534 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f538c003db0 con 0x564232534320 2024-06-04T05:20:25.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.678+0000 7f538a7fc700 1 -- 172.21.15.70:0/1322647534 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c030070 con 0x564232534320 2024-06-04T05:20:25.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.679+0000 7f53a55c8b80 1 -- 172.21.15.70:0/1322647534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 msgr2=0x564232536ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:25.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.679+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/1322647534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 0x564232536ff0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f538c0238f0 tx=0x7f538c005d00 comp rx=0 tx=0).stop 2024-06-04T05:20:25.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.679+0000 7f53a55c8b80 1 -- 172.21.15.70:0/1322647534 shutdown_connections 2024-06-04T05:20:25.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.679+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/1322647534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564232534320 0x564232536ff0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.679+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/1322647534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56423252e8a0 0x564232552da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.679+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/1322647534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252f260 0x564232529270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.679+0000 7f53a55c8b80 1 -- 172.21.15.70:0/1322647534 >> 172.21.15.70:0/1322647534 conn(0x56423245a240 msgr2=0x5642325368e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:25.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.679+0000 7f53a55c8b80 1 -- 172.21.15.70:0/1322647534 shutdown_connections 2024-06-04T05:20:25.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.679+0000 7f53a55c8b80 1 -- 172.21.15.70:0/1322647534 wait complete. 2024-06-04T05:20:25.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.680+0000 7f53a55c8b80 1 -- start start 2024-06-04T05:20:25.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.680+0000 7f53a55c8b80 1 Processor -- start 2024-06-04T05:20:25.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.681+0000 7f53a55c8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252e8a0 0x564232557570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.681+0000 7f53a55c8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56423252f260 0x564232557ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.681+0000 7f53a55c8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564232534320 0x564232557ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.681+0000 7f53a55c8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f53941acfe0 con 0x56423252e8a0 2024-06-04T05:20:25.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.681+0000 7f53a55c8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53941ac910 con 0x564232534320 2024-06-04T05:20:25.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.681+0000 7f53a55c8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f53941a9ad0 con 0x56423252f260 2024-06-04T05:20:25.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.681+0000 7f5398acf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252e8a0 0x564232557570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:25.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.681+0000 7f5398acf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252e8a0 0x564232557570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37844/0 (socket says 172.21.15.70:37844) 2024-06-04T05:20:25.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.681+0000 7f53a55c8b80 10 client.0 _ll_register_callbacks cb 0x564232417950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:25.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.681+0000 7f5398acf700 1 -- 172.21.15.70:0/3365805929 learned_addr learned my addr 172.21.15.70:0/3365805929 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:25.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.681+0000 7f5393fff700 1 --2- 172.21.15.70:0/3365805929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56423252f260 0x564232557ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:25.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.681+0000 7f53992d0700 1 --2- 172.21.15.70:0/3365805929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564232534320 0x564232557ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:25.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.681+0000 7f5398acf700 1 -- 172.21.15.70:0/3365805929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56423252f260 msgr2=0x564232557ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:25.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.681+0000 7f5398acf700 1 --2- 172.21.15.70:0/3365805929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56423252f260 0x564232557ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.681+0000 7f5398acf700 1 -- 172.21.15.70:0/3365805929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564232534320 msgr2=0x564232557ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:25.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.681+0000 7f5398acf700 1 --2- 172.21.15.70:0/3365805929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564232534320 0x564232557ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.681+0000 7f5398acf700 1 -- 172.21.15.70:0/3365805929 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56423253f960 con 0x56423252e8a0 2024-06-04T05:20:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:25 smithi112 ceph-mon[36453]: pgmap v732: 97 pgs: 97 active+clean; 1010 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.3 MiB/s wr, 279 op/s 2024-06-04T05:20:25.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.682+0000 7f5398acf700 1 --2- 172.21.15.70:0/3365805929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252e8a0 0x564232557570 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f537c0aa520 tx=0x7f537c0a6770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=328dc365a7f822fe server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:25.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.682+0000 7f5373fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:20:25.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.682+0000 7f5373fff700 1 -- 172.21.15.70:0/3365805929 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c0cd0c0 con 0x56423252e8a0 2024-06-04T05:20:25.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.682+0000 7f5373fff700 1 -- 172.21.15.70:0/3365805929 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f537c00bd40 con 0x56423252e8a0 2024-06-04T05:20:25.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.682+0000 7f53a55c8b80 1 -- 172.21.15.70:0/3365805929 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f538c003db0 con 0x56423252e8a0 2024-06-04T05:20:25.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.682+0000 7f5373fff700 1 -- 172.21.15.70:0/3365805929 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f537c005040 con 0x56423252e8a0 2024-06-04T05:20:25.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.682+0000 7f53a55c8b80 10 client.21363 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:25.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.682+0000 7f53a55c8b80 1 -- 172.21.15.70:0/3365805929 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x564232540c80 con 0x56423252e8a0 2024-06-04T05:20:25.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.683+0000 7f5373fff700 1 -- 172.21.15.70:0/3365805929 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f538c003db0 con 0x56423252e8a0 2024-06-04T05:20:25.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.683+0000 7f5373fff700 1 -- 172.21.15.70:0/3365805929 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f537c0e0e80 con 0x56423252e8a0 2024-06-04T05:20:25.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.683+0000 7f53a55c8b80 10 client.21363 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:25.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.683+0000 7f53a55c8b80 10 client.21363 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:20:25.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.683+0000 7f53a55c8b80 20 client.21363 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:25.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.683+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/3365805929 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x564232541340 0x5642325bd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:25.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.683+0000 7f53a55c8b80 4 client.21363 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:20:25.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.683+0000 7f53a55c8b80 1 -- 172.21.15.70:0/3365805929 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5642324884a0 con 0x564232541340 2024-06-04T05:20:25.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.683+0000 7f5393fff700 1 --2- 172.21.15.70:0/3365805929 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x564232541340 0x5642325bd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:25.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.684+0000 7f5373fff700 10 client.21363 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:20:25.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.684+0000 7f5393fff700 1 --2- 172.21.15.70:0/3365805929 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x564232541340 0x5642325bd460 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7f66cf6f6fdfcfa8 server_cookie=f9c7467c2cafe7d6 in_seq=0 out_seq=0 2024-06-04T05:20:25.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.684+0000 7f5373fff700 1 -- 172.21.15.70:0/3365805929 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5642324884a0 con 0x564232541340 2024-06-04T05:20:25.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.684+0000 7f5373fff700 10 client.21363 handle_command_reply: tid=0 2024-06-04T05:20:25.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.684+0000 7f53a55c8b80 1 client.21363 shutdown 2024-06-04T05:20:25.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.684+0000 7f53a55c8b80 1 -- 172.21.15.70:0/3365805929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252e8a0 msgr2=0x564232557570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:25.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.684+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/3365805929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252e8a0 0x564232557570 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f537c0aa520 tx=0x7f537c0a6770 comp rx=0 tx=0).stop 2024-06-04T05:20:25.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.684+0000 7f53a55c8b80 1 -- 172.21.15.70:0/3365805929 shutdown_connections 2024-06-04T05:20:25.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.684+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/3365805929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564232534320 0x564232557ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.684+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/3365805929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56423252e8a0 0x564232557570 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.684+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/3365805929 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x564232541340 0x5642325bd460 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.684+0000 7f5373fff700 0 client.21363 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:20:25.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.684+0000 7f53a55c8b80 1 --2- 172.21.15.70:0/3365805929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56423252f260 0x564232557ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.684+0000 7f53a55c8b80 1 -- 172.21.15.70:0/3365805929 >> 172.21.15.70:0/3365805929 conn(0x56423245a240 msgr2=0x564232533830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:25.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.685+0000 7f53a55c8b80 1 -- 172.21.15.70:0/3365805929 shutdown_connections 2024-06-04T05:20:25.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.685+0000 7f53a55c8b80 1 -- 172.21.15.70:0/3365805929 wait complete. 2024-06-04T05:20:25.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.685+0000 7f53a55c8b80 20 client.21363 trim_cache size 0 max 16384 2024-06-04T05:20:25.691 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:20:25.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.685+0000 7f539ad33700 1 -- 172.21.15.70:0/2689148143 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f53800b2a70 msgr2=0x7f53800b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:25.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.686+0000 7f539ad33700 1 --2- 172.21.15.70:0/2689148143 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f53800b2a70 0x7f53800b4f30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5394079b90 tx=0x7f538400b040 comp rx=0 tx=0).stop 2024-06-04T05:20:25.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.686+0000 7f539ad33700 1 -- 172.21.15.70:0/2689148143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5394102430 msgr2=0x7f539419f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:25.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.686+0000 7f539ad33700 1 --2- 172.21.15.70:0/2689148143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5394102430 0x7f539419f820 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f537c009fd0 tx=0x7f537c00f990 comp rx=0 tx=0).stop 2024-06-04T05:20:25.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.686+0000 7f539ad33700 1 -- 172.21.15.70:0/2689148143 shutdown_connections 2024-06-04T05:20:25.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.686+0000 7f539ad33700 1 --2- 172.21.15.70:0/2689148143 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f53800b2a70 0x7f53800b4f30 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.686+0000 7f539ad33700 1 --2- 172.21.15.70:0/2689148143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5394108510 0x7f53941999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.686+0000 7f539ad33700 1 --2- 172.21.15.70:0/2689148143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5394102430 0x7f539419f820 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.686+0000 7f539ad33700 1 --2- 172.21.15.70:0/2689148143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5394102df0 0x7f539419fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:25.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.686+0000 7f539ad33700 1 -- 172.21.15.70:0/2689148143 >> 172.21.15.70:0/2689148143 conn(0x7f53940793b0 msgr2=0x7f5394101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:25.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.686+0000 7f539ad33700 1 -- 172.21.15.70:0/2689148143 shutdown_connections 2024-06-04T05:20:25.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:25.686+0000 7f539ad33700 1 -- 172.21.15.70:0/2689148143 wait complete. 2024-06-04T05:20:25.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:25 smithi098 ceph-mon[35569]: pgmap v732: 97 pgs: 97 active+clean; 1010 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.3 MiB/s wr, 279 op/s 2024-06-04T05:20:25.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 70999 6.71 MB/sec execute 437 sec latency 71.629 ms 2024-06-04T05:20:26.494 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:20:26.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 71162 6.71 MB/sec execute 438 sec latency 69.252 ms 2024-06-04T05:20:26.842 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:20:27.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:27 smithi070 ceph-mon[32631]: pgmap v733: 97 pgs: 97 active+clean; 1018 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.5 MiB/s wr, 407 op/s 2024-06-04T05:20:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:27 smithi112 ceph-mon[36453]: pgmap v733: 97 pgs: 97 active+clean; 1018 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.5 MiB/s wr, 407 op/s 2024-06-04T05:20:27.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:27 smithi098 ceph-mon[35569]: pgmap v733: 97 pgs: 97 active+clean; 1018 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.5 MiB/s wr, 407 op/s 2024-06-04T05:20:27.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 71328 6.72 MB/sec execute 439 sec latency 76.037 ms 2024-06-04T05:20:28.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 71524 6.74 MB/sec execute 440 sec latency 101.517 ms 2024-06-04T05:20:29.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.227+0000 7f6164c4f700 1 -- 172.21.15.70:0/1888279910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6160108510 msgr2=0x7f61601088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:29.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.227+0000 7f6164c4f700 1 --2- 172.21.15.70:0/1888279910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6160108510 0x7f61601088f0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f6150009990 tx=0x7f6150009ca0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.227+0000 7f6164c4f700 1 -- 172.21.15.70:0/1888279910 shutdown_connections 2024-06-04T05:20:29.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.227+0000 7f6164c4f700 1 --2- 172.21.15.70:0/1888279910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6160102df0 0x7f616010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.227+0000 7f6164c4f700 1 --2- 172.21.15.70:0/1888279910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6160108510 0x7f61601088f0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.227+0000 7f6164c4f700 1 --2- 172.21.15.70:0/1888279910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6160102430 0x7f61601028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.227+0000 7f6164c4f700 1 -- 172.21.15.70:0/1888279910 >> 172.21.15.70:0/1888279910 conn(0x7f61600793b0 msgr2=0x7f61600797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:29.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.227+0000 7f6164c4f700 1 -- 172.21.15.70:0/1888279910 shutdown_connections 2024-06-04T05:20:29.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.227+0000 7f6164c4f700 1 -- 172.21.15.70:0/1888279910 wait complete. 2024-06-04T05:20:29.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.228+0000 7f6164c4f700 1 Processor -- start 2024-06-04T05:20:29.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.228+0000 7f6164c4f700 1 -- start start 2024-06-04T05:20:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.229+0000 7f6164c4f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6160102430 0x7f6160104070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.229+0000 7f6164c4f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6160102df0 0x7f61601045b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.229+0000 7f6164c4f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6160108510 0x7f6160104af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.229+0000 7f6164c4f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61601124e0 con 0x7f6160102430 2024-06-04T05:20:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.229+0000 7f6164c4f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6160112360 con 0x7f6160108510 2024-06-04T05:20:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.229+0000 7f6164c4f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6160112660 con 0x7f6160102df0 2024-06-04T05:20:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.229+0000 7f615ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6160108510 0x7f6160104af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.229+0000 7f615e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6160102430 0x7f6160104070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.229+0000 7f615ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6160108510 0x7f6160104af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:48114/0 (socket says 172.21.15.70:48114) 2024-06-04T05:20:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.229+0000 7f615ed9d700 1 -- 172.21.15.70:0/598606133 learned_addr learned my addr 172.21.15.70:0/598606133 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.230+0000 7f615e59c700 1 -- 172.21.15.70:0/598606133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6160102df0 msgr2=0x7f61601045b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-04T05:20:29.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.230+0000 7f615e59c700 1 --2- 172.21.15.70:0/598606133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6160102df0 0x7f61601045b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.230+0000 7f615e59c700 1 -- 172.21.15.70:0/598606133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6160108510 msgr2=0x7f6160104af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:29.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.230+0000 7f615e59c700 1 --2- 172.21.15.70:0/598606133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6160108510 0x7f6160104af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.230+0000 7f615e59c700 1 -- 172.21.15.70:0/598606133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61601ac5b0 con 0x7f6160102430 2024-06-04T05:20:29.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.230+0000 7f615e59c700 1 --2- 172.21.15.70:0/598606133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6160102430 0x7f6160104070 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f6150000c00 tx=0x7f61500047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:29.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.230+0000 7f614f7fe700 1 -- 172.21.15.70:0/598606133 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150026070 con 0x7f6160102430 2024-06-04T05:20:29.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.230+0000 7f614f7fe700 1 -- 172.21.15.70:0/598606133 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f615000fd90 con 0x7f6160102430 2024-06-04T05:20:29.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.230+0000 7f614f7fe700 1 -- 172.21.15.70:0/598606133 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150021e30 con 0x7f6160102430 2024-06-04T05:20:29.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.230+0000 7f6164c4f700 1 -- 172.21.15.70:0/598606133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61601ac7e0 con 0x7f6160102430 2024-06-04T05:20:29.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.230+0000 7f6164c4f700 1 -- 172.21.15.70:0/598606133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61601acc40 con 0x7f6160102430 2024-06-04T05:20:29.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.234+0000 7f614f7fe700 1 -- 172.21.15.70:0/598606133 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f6150029070 con 0x7f6160102430 2024-06-04T05:20:29.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.235+0000 7f614f7fe700 1 --2- 172.21.15.70:0/598606133 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f61480b2af0 0x7f61480b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.235+0000 7f615dd9b700 1 --2- 172.21.15.70:0/598606133 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f61480b2af0 0x7f61480b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:29.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.235+0000 7f614f7fe700 1 -- 172.21.15.70:0/598606133 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f61500e3200 con 0x7f6160102430 2024-06-04T05:20:29.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.235+0000 7f616f4e4b80 1 Processor -- start 2024-06-04T05:20:29.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.235+0000 7f616f4e4b80 1 -- start start 2024-06-04T05:20:29.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.235+0000 7f615dd9b700 1 --2- 172.21.15.70:0/598606133 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f61480b2af0 0x7f61480b4fb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6160105830 tx=0x7f6154009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:29.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f616f4e4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc701c320 0x55ecc701c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f616f4e4b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc70168a0 0x55ecc7016d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f616f4e4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc7017260 0x55ecc7023080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f616f4e4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ecc6f74d70 con 0x55ecc701c320 2024-06-04T05:20:29.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f616f4e4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ecc6f64910 con 0x55ecc7017260 2024-06-04T05:20:29.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f616f4e4b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ecc6f68950 con 0x55ecc70168a0 2024-06-04T05:20:29.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f615dd9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc70168a0 0x55ecc7016d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:29.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f615ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc7017260 0x55ecc7023080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:29.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f615ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc7017260 0x55ecc7023080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:48130/0 (socket says 172.21.15.70:48130) 2024-06-04T05:20:29.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f615dd9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc70168a0 0x55ecc7016d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51652/0 (socket says 172.21.15.70:51652) 2024-06-04T05:20:29.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f615ed9d700 1 -- 172.21.15.70:0/841295689 learned_addr learned my addr 172.21.15.70:0/841295689 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:29.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f615ed9d700 1 -- 172.21.15.70:0/841295689 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc70168a0 msgr2=0x55ecc7016d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:29.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f615ed9d700 1 --2- 172.21.15.70:0/841295689 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc70168a0 0x55ecc7016d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f615ed9d700 1 -- 172.21.15.70:0/841295689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc701c320 msgr2=0x55ecc701c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:29.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f615ed9d700 1 --2- 172.21.15.70:0/841295689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc701c320 0x55ecc701c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f615ed9d700 1 -- 172.21.15.70:0/841295689 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecc6f0dfb0 con 0x55ecc7017260 2024-06-04T05:20:29.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f615ed9d700 1 --2- 172.21.15.70:0/841295689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc7017260 0x55ecc7023080 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f61580094b0 tx=0x7f6158009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2c0167217062b35 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:29.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.236+0000 7f614cff9700 1 -- 172.21.15.70:0/841295689 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6158004db0 con 0x55ecc7017260 2024-06-04T05:20:29.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.237+0000 7f614cff9700 1 -- 172.21.15.70:0/841295689 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f61580054b0 con 0x55ecc7017260 2024-06-04T05:20:29.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.237+0000 7f614cff9700 1 -- 172.21.15.70:0/841295689 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61580158d0 con 0x55ecc7017260 2024-06-04T05:20:29.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.238+0000 7f616f4e4b80 1 -- 172.21.15.70:0/841295689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc7017260 msgr2=0x55ecc7023080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:29.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.238+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/841295689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc7017260 0x55ecc7023080 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f61580094b0 tx=0x7f6158009920 comp rx=0 tx=0).stop 2024-06-04T05:20:29.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.238+0000 7f616f4e4b80 1 -- 172.21.15.70:0/841295689 shutdown_connections 2024-06-04T05:20:29.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.238+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/841295689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc7017260 0x55ecc7023080 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.238+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/841295689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc701c320 0x55ecc701c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.238+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/841295689 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc70168a0 0x55ecc7016d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.238+0000 7f616f4e4b80 1 -- 172.21.15.70:0/841295689 >> 172.21.15.70:0/841295689 conn(0x55ecc6f42240 msgr2=0x55ecc70113e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.238+0000 7f616f4e4b80 1 -- 172.21.15.70:0/841295689 shutdown_connections 2024-06-04T05:20:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.238+0000 7f616f4e4b80 1 -- 172.21.15.70:0/841295689 wait complete. 2024-06-04T05:20:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.238+0000 7f616f4e4b80 1 -- start start 2024-06-04T05:20:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.239+0000 7f616f4e4b80 1 Processor -- start 2024-06-04T05:20:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.239+0000 7f616f4e4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc70168a0 0x55ecc7022ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.239+0000 7f616f4e4b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc7017260 0x55ecc7020030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.239+0000 7f616f4e4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc701c320 0x55ecc7020570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.239+0000 7f616f4e4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ecc6e6afa0 con 0x55ecc70168a0 2024-06-04T05:20:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.239+0000 7f615e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc70168a0 0x55ecc7022ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.239+0000 7f615ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc701c320 0x55ecc7020570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.239+0000 7f615dd9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc7017260 0x55ecc7020030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.239+0000 7f615e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc70168a0 0x55ecc7022ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56466/0 (socket says 172.21.15.70:56466) 2024-06-04T05:20:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.239+0000 7f615e59c700 1 -- 172.21.15.70:0/1319640596 learned_addr learned my addr 172.21.15.70:0/1319640596 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.239+0000 7f615ed9d700 1 --2- 172.21.15.70:0/1319640596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc701c320 0x55ecc7020570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:48138/0 (socket says 172.21.15.70:48138) 2024-06-04T05:20:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.239+0000 7f616f4e4b80 1 -- 172.21.15.70:0/1319640596 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ecc6eccda0 con 0x55ecc701c320 2024-06-04T05:20:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.239+0000 7f616f4e4b80 1 -- 172.21.15.70:0/1319640596 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ecc70236e0 con 0x55ecc7017260 2024-06-04T05:20:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.240+0000 7f616f4e4b80 10 client.0 _ll_register_callbacks cb 0x55ecc6eff950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.240+0000 7f615dd9b700 1 -- 172.21.15.70:0/1319640596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc701c320 msgr2=0x55ecc7020570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.240+0000 7f615dd9b700 1 --2- 172.21.15.70:0/1319640596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc701c320 0x55ecc7020570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.240+0000 7f615dd9b700 1 -- 172.21.15.70:0/1319640596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc70168a0 msgr2=0x55ecc7022ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.240+0000 7f615dd9b700 1 --2- 172.21.15.70:0/1319640596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc70168a0 0x55ecc7022ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.240+0000 7f615dd9b700 1 -- 172.21.15.70:0/1319640596 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecc6eb48d0 con 0x55ecc7017260 2024-06-04T05:20:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.240+0000 7f615dd9b700 1 --2- 172.21.15.70:0/1319640596 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc7017260 0x55ecc7020030 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6154009fd0 tx=0x7f61540047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e0b039b12086064 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.240+0000 7f6145ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:20:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.240+0000 7f6145ffb700 1 -- 172.21.15.70:0/1319640596 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615400ee40 con 0x55ecc7017260 2024-06-04T05:20:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.241+0000 7f616f4e4b80 1 -- 172.21.15.70:0/1319640596 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61580054b0 con 0x55ecc7017260 2024-06-04T05:20:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.241+0000 7f6145ffb700 1 -- 172.21.15.70:0/1319640596 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f6154004830 con 0x55ecc7017260 2024-06-04T05:20:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.241+0000 7f6145ffb700 1 -- 172.21.15.70:0/1319640596 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f6154004a30 con 0x55ecc7017260 2024-06-04T05:20:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.241+0000 7f616f4e4b80 10 client.30163 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.241+0000 7f616f4e4b80 1 -- 172.21.15.70:0/1319640596 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55ecc6f1e760 con 0x55ecc7017260 2024-06-04T05:20:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.242+0000 7f6145ffb700 1 -- 172.21.15.70:0/1319640596 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f61580054b0 con 0x55ecc7017260 2024-06-04T05:20:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.242+0000 7f6145ffb700 1 -- 172.21.15.70:0/1319640596 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f6154024070 con 0x55ecc7017260 2024-06-04T05:20:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.242+0000 7f616f4e4b80 10 client.30163 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.242+0000 7f616f4e4b80 10 client.30163 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:20:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.243+0000 7f616f4e4b80 20 client.30163 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.243+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/1319640596 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ecc7030680 0x55ecc7032ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.243+0000 7f616f4e4b80 4 client.30163 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:20:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.243+0000 7f616f4e4b80 1 -- 172.21.15.70:0/1319640596 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecc6f76cc0 con 0x55ecc7030680 2024-06-04T05:20:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.243+0000 7f615e59c700 1 --2- 172.21.15.70:0/1319640596 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ecc7030680 0x55ecc7032ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.244+0000 7f6145ffb700 10 client.30163 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:20:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.244+0000 7f615e59c700 1 --2- 172.21.15.70:0/1319640596 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ecc7030680 0x55ecc7032ac0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=25076c5480953735 server_cookie=85fbfe712e0db4a4 in_seq=0 out_seq=0 2024-06-04T05:20:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.245+0000 7f6145ffb700 1 -- 172.21.15.70:0/1319640596 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ecc6f76cc0 con 0x55ecc7030680 2024-06-04T05:20:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.245+0000 7f6145ffb700 10 client.30163 handle_command_reply: tid=0 2024-06-04T05:20:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.245+0000 7f616f4e4b80 1 client.30163 shutdown 2024-06-04T05:20:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.245+0000 7f616f4e4b80 1 -- 172.21.15.70:0/1319640596 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc7017260 msgr2=0x55ecc7020030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.245+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/1319640596 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc7017260 0x55ecc7020030 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6154009fd0 tx=0x7f61540047c0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.245+0000 7f616f4e4b80 1 -- 172.21.15.70:0/1319640596 shutdown_connections 2024-06-04T05:20:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.245+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/1319640596 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ecc7030680 0x55ecc7032ac0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.245+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/1319640596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc701c320 0x55ecc7020570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.245+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/1319640596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc70168a0 0x55ecc7022ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.245+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/1319640596 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc7017260 0x55ecc7020030 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.245+0000 7f616f4e4b80 1 -- 172.21.15.70:0/1319640596 >> 172.21.15.70:0/1319640596 conn(0x55ecc6f42240 msgr2=0x55ecc7010af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.245+0000 7f6145ffb700 0 client.30163 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:20:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.245+0000 7f616f4e4b80 1 -- 172.21.15.70:0/1319640596 shutdown_connections 2024-06-04T05:20:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.245+0000 7f616f4e4b80 1 -- 172.21.15.70:0/1319640596 wait complete. 2024-06-04T05:20:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.246+0000 7f616f4e4b80 20 client.30163 trim_cache size 0 max 16384 2024-06-04T05:20:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.274+0000 7f616f4e4b80 1 Processor -- start 2024-06-04T05:20:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.274+0000 7f616f4e4b80 1 -- start start 2024-06-04T05:20:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.274+0000 7f616f4e4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc70168a0 0x55ecc7011270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.274+0000 7f616f4e4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc7017260 0x55ecc701b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.274+0000 7f616f4e4b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc701c320 0x55ecc701b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.274+0000 7f616f4e4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61601b44e0 con 0x55ecc70168a0 2024-06-04T05:20:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.274+0000 7f616f4e4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61601b60b0 con 0x55ecc7017260 2024-06-04T05:20:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.274+0000 7f616f4e4b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f61601b4c10 con 0x55ecc701c320 2024-06-04T05:20:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.275+0000 7f615e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc70168a0 0x55ecc7011270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.275+0000 7f615dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc7017260 0x55ecc701b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.275+0000 7f615e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc70168a0 0x55ecc7011270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56470/0 (socket says 172.21.15.70:56470) 2024-06-04T05:20:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.275+0000 7f615e59c700 1 -- 172.21.15.70:0/2600634991 learned_addr learned my addr 172.21.15.70:0/2600634991 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.275+0000 7f615ed9d700 1 --2- 172.21.15.70:0/2600634991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc701c320 0x55ecc701b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.275+0000 7f615e59c700 1 -- 172.21.15.70:0/2600634991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc701c320 msgr2=0x55ecc701b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.275+0000 7f615e59c700 1 --2- 172.21.15.70:0/2600634991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc701c320 0x55ecc701b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.275+0000 7f615e59c700 1 -- 172.21.15.70:0/2600634991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc7017260 msgr2=0x55ecc701b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.275+0000 7f615e59c700 1 --2- 172.21.15.70:0/2600634991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc7017260 0x55ecc701b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.275+0000 7f615e59c700 1 -- 172.21.15.70:0/2600634991 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecc702dcf0 con 0x55ecc70168a0 2024-06-04T05:20:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.275+0000 7f615dd9b700 1 --2- 172.21.15.70:0/2600634991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc7017260 0x55ecc701b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.275+0000 7f615ed9d700 1 --2- 172.21.15.70:0/2600634991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc701c320 0x55ecc701b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:20:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.275+0000 7f615e59c700 1 --2- 172.21.15.70:0/2600634991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc70168a0 0x55ecc7011270 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f61500a7750 tx=0x7f6150029000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a080c6dfcb008cee server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.276+0000 7f6147fff700 1 -- 172.21.15.70:0/2600634991 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61500aabe0 con 0x55ecc70168a0 2024-06-04T05:20:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.276+0000 7f6147fff700 1 -- 172.21.15.70:0/2600634991 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f61500aa4e0 con 0x55ecc70168a0 2024-06-04T05:20:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.276+0000 7f6147fff700 1 -- 172.21.15.70:0/2600634991 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61500e0ba0 con 0x55ecc70168a0 2024-06-04T05:20:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.277+0000 7f616f4e4b80 1 -- 172.21.15.70:0/2600634991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc70168a0 msgr2=0x55ecc7011270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.277+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/2600634991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc70168a0 0x55ecc7011270 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f61500a7750 tx=0x7f6150029000 comp rx=0 tx=0).stop 2024-06-04T05:20:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.277+0000 7f616f4e4b80 1 -- 172.21.15.70:0/2600634991 shutdown_connections 2024-06-04T05:20:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.277+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/2600634991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc701c320 0x55ecc701b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.277+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/2600634991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc70168a0 0x55ecc7011270 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.277+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/2600634991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc7017260 0x55ecc701b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.277+0000 7f616f4e4b80 1 -- 172.21.15.70:0/2600634991 >> 172.21.15.70:0/2600634991 conn(0x55ecc6f42240 msgr2=0x55ecc701e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.277+0000 7f616f4e4b80 1 -- 172.21.15.70:0/2600634991 shutdown_connections 2024-06-04T05:20:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.277+0000 7f616f4e4b80 1 -- 172.21.15.70:0/2600634991 wait complete. 2024-06-04T05:20:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.278+0000 7f616f4e4b80 1 -- start start 2024-06-04T05:20:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.278+0000 7f616f4e4b80 1 Processor -- start 2024-06-04T05:20:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.278+0000 7f616f4e4b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc70168a0 0x55ecc7038a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.278+0000 7f616f4e4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc7017260 0x55ecc7028960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.278+0000 7f616f4e4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc701c320 0x55ecc7028ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.278+0000 7f616f4e4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61601b59b0 con 0x55ecc7017260 2024-06-04T05:20:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.278+0000 7f615e59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc70168a0 0x55ecc7038a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.278+0000 7f616f4e4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61601b52e0 con 0x55ecc701c320 2024-06-04T05:20:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.278+0000 7f616f4e4b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f61601b24a0 con 0x55ecc70168a0 2024-06-04T05:20:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.278+0000 7f615e59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc70168a0 0x55ecc7038a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51684/0 (socket says 172.21.15.70:51684) 2024-06-04T05:20:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.278+0000 7f616f4e4b80 10 client.0 _ll_register_callbacks cb 0x55ecc6eff950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.278+0000 7f615e59c700 1 -- 172.21.15.70:0/3667382017 learned_addr learned my addr 172.21.15.70:0/3667382017 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.278+0000 7f615ed9d700 1 --2- 172.21.15.70:0/3667382017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc701c320 0x55ecc7028ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.279+0000 7f615ed9d700 1 -- 172.21.15.70:0/3667382017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc70168a0 msgr2=0x55ecc7038a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.279+0000 7f615ed9d700 1 --2- 172.21.15.70:0/3667382017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc70168a0 0x55ecc7038a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.279+0000 7f615ed9d700 1 -- 172.21.15.70:0/3667382017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc7017260 msgr2=0x55ecc7028960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.279+0000 7f615ed9d700 1 --2- 172.21.15.70:0/3667382017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc7017260 0x55ecc7028960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.279+0000 7f615ed9d700 1 -- 172.21.15.70:0/3667382017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecc7037380 con 0x55ecc701c320 2024-06-04T05:20:29.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.279+0000 7f615ed9d700 1 --2- 172.21.15.70:0/3667382017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc701c320 0x55ecc7028ea0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6158009690 tx=0x7f6158015410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4da5c9cec832576f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:29.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.279+0000 7f6145ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:20:29.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.280+0000 7f6145ffb700 1 -- 172.21.15.70:0/3667382017 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615802b060 con 0x55ecc701c320 2024-06-04T05:20:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.280+0000 7f6145ffb700 1 -- 172.21.15.70:0/3667382017 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f6158015d90 con 0x55ecc701c320 2024-06-04T05:20:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.280+0000 7f616f4e4b80 1 -- 172.21.15.70:0/3667382017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61500aa4e0 con 0x55ecc701c320 2024-06-04T05:20:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.280+0000 7f6145ffb700 1 -- 172.21.15.70:0/3667382017 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f615802f440 con 0x55ecc701c320 2024-06-04T05:20:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.280+0000 7f616f4e4b80 10 client.30029 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.280+0000 7f616f4e4b80 1 -- 172.21.15.70:0/3667382017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55ecc709a420 con 0x55ecc701c320 2024-06-04T05:20:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.280+0000 7f6145ffb700 1 -- 172.21.15.70:0/3667382017 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f61500aa4e0 con 0x55ecc701c320 2024-06-04T05:20:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.281+0000 7f6145ffb700 1 -- 172.21.15.70:0/3667382017 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f615801d4a0 con 0x55ecc701c320 2024-06-04T05:20:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.281+0000 7f616f4e4b80 10 client.30029 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.281+0000 7f616f4e4b80 10 client.30029 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:20:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.281+0000 7f616f4e4b80 20 client.30029 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.281+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/3667382017 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ecc709a800 0x55ecc709cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.281+0000 7f616f4e4b80 4 client.30029 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:20:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.281+0000 7f616f4e4b80 1 -- 172.21.15.70:0/3667382017 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ecc6f704a0 con 0x55ecc709a800 2024-06-04T05:20:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.281+0000 7f615e59c700 1 --2- 172.21.15.70:0/3667382017 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ecc709a800 0x55ecc709cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:29.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.283+0000 7f6145ffb700 10 client.30029 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:20:29.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.283+0000 7f615e59c700 1 --2- 172.21.15.70:0/3667382017 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ecc709a800 0x55ecc709cc20 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bc72e097c225826e server_cookie=1c15b8cb3bd7ebc9 in_seq=0 out_seq=0 2024-06-04T05:20:29.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.283+0000 7f6145ffb700 1 -- 172.21.15.70:0/3667382017 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ecc6f704a0 con 0x55ecc709a800 2024-06-04T05:20:29.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.283+0000 7f6145ffb700 10 client.30029 handle_command_reply: tid=0 2024-06-04T05:20:29.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.283+0000 7f616f4e4b80 1 client.30029 shutdown 2024-06-04T05:20:29.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.283+0000 7f616f4e4b80 1 -- 172.21.15.70:0/3667382017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc701c320 msgr2=0x55ecc7028ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:29.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.283+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/3667382017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc701c320 0x55ecc7028ea0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6158009690 tx=0x7f6158015410 comp rx=0 tx=0).stop 2024-06-04T05:20:29.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.283+0000 7f616f4e4b80 1 -- 172.21.15.70:0/3667382017 shutdown_connections 2024-06-04T05:20:29.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.283+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/3667382017 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55ecc709a800 0x55ecc709cc20 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.283+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/3667382017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecc701c320 0x55ecc7028ea0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.283+0000 7f6145ffb700 0 client.30029 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:20:29.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.283+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/3667382017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ecc70168a0 0x55ecc7038a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.283+0000 7f616f4e4b80 1 --2- 172.21.15.70:0/3667382017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ecc7017260 0x55ecc7028960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.283+0000 7f616f4e4b80 1 -- 172.21.15.70:0/3667382017 >> 172.21.15.70:0/3667382017 conn(0x55ecc6f42240 msgr2=0x55ecc7019d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:29.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.283+0000 7f616f4e4b80 1 -- 172.21.15.70:0/3667382017 shutdown_connections 2024-06-04T05:20:29.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.284+0000 7f616f4e4b80 1 -- 172.21.15.70:0/3667382017 wait complete. 2024-06-04T05:20:29.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.284+0000 7f616f4e4b80 20 client.30029 trim_cache size 0 max 16384 2024-06-04T05:20:29.288 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:20:29.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.284+0000 7f6164c4f700 1 -- 172.21.15.70:0/598606133 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f61480b2af0 msgr2=0x7f61480b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:29.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.284+0000 7f6164c4f700 1 --2- 172.21.15.70:0/598606133 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f61480b2af0 0x7f61480b4fb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6160105830 tx=0x7f6154009260 comp rx=0 tx=0).stop 2024-06-04T05:20:29.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.285+0000 7f6164c4f700 1 -- 172.21.15.70:0/598606133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6160102430 msgr2=0x7f6160104070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:29.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.285+0000 7f6164c4f700 1 --2- 172.21.15.70:0/598606133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6160102430 0x7f6160104070 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f6150000c00 tx=0x7f61500047b0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.285+0000 7f6164c4f700 1 -- 172.21.15.70:0/598606133 shutdown_connections 2024-06-04T05:20:29.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.285+0000 7f6164c4f700 1 --2- 172.21.15.70:0/598606133 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f61480b2af0 0x7f61480b4fb0 secure :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6160105830 tx=0x7f6154009260 comp rx=0 tx=0).stop 2024-06-04T05:20:29.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.285+0000 7f6164c4f700 1 --2- 172.21.15.70:0/598606133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6160108510 0x7f6160104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.285+0000 7f6164c4f700 1 --2- 172.21.15.70:0/598606133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6160102430 0x7f6160104070 secure :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f6150000c00 tx=0x7f61500047b0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.285+0000 7f6164c4f700 1 --2- 172.21.15.70:0/598606133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6160102df0 0x7f61601045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:29.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.285+0000 7f6164c4f700 1 -- 172.21.15.70:0/598606133 >> 172.21.15.70:0/598606133 conn(0x7f61600793b0 msgr2=0x7f616006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:29.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.285+0000 7f6164c4f700 1 -- 172.21.15.70:0/598606133 shutdown_connections 2024-06-04T05:20:29.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:29.285+0000 7f6164c4f700 1 -- 172.21.15.70:0/598606133 wait complete. 2024-06-04T05:20:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:29 smithi112 ceph-mon[36453]: pgmap v734: 97 pgs: 97 active+clean; 1018 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 296 op/s 2024-06-04T05:20:29.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:29 smithi098 ceph-mon[35569]: pgmap v734: 97 pgs: 97 active+clean; 1018 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 296 op/s 2024-06-04T05:20:29.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 71707 6.75 MB/sec execute 441 sec latency 60.840 ms 2024-06-04T05:20:29.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:29 smithi070 ceph-mon[32631]: pgmap v734: 97 pgs: 97 active+clean; 1018 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 296 op/s 2024-06-04T05:20:30.334 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:20:30.687 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:20:30.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 71866 6.75 MB/sec execute 442 sec latency 70.620 ms 2024-06-04T05:20:31.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:31 smithi070 ceph-mon[32631]: pgmap v735: 97 pgs: 97 active+clean; 1018 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 296 op/s 2024-06-04T05:20:31.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:31 smithi112 ceph-mon[36453]: pgmap v735: 97 pgs: 97 active+clean; 1018 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 296 op/s 2024-06-04T05:20:31.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:31 smithi098 ceph-mon[35569]: pgmap v735: 97 pgs: 97 active+clean; 1018 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 296 op/s 2024-06-04T05:20:31.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 72036 6.76 MB/sec execute 443 sec latency 53.637 ms 2024-06-04T05:20:32.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 72208 6.76 MB/sec execute 444 sec latency 101.942 ms 2024-06-04T05:20:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.082+0000 7f0320932640 1 -- 172.21.15.70:0/2707256945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031c105870 msgr2=0x7f031c1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.082+0000 7f0320932640 1 --2- 172.21.15.70:0/2707256945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031c105870 0x7f031c1098b0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f0310009900 tx=0x7f031002a3f0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.082+0000 7f0320932640 1 -- 172.21.15.70:0/2707256945 shutdown_connections 2024-06-04T05:20:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.082+0000 7f0320932640 1 --2- 172.21.15.70:0/2707256945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f031c109f90 0x7f031c111b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.082+0000 7f0320932640 1 --2- 172.21.15.70:0/2707256945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031c105870 0x7f031c1098b0 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.082+0000 7f0320932640 1 --2- 172.21.15.70:0/2707256945 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f031c104ec0 0x7f031c1052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.082+0000 7f0320932640 1 -- 172.21.15.70:0/2707256945 >> 172.21.15.70:0/2707256945 conn(0x7f031c100700 msgr2=0x7f031c102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.082+0000 7f0320932640 1 -- 172.21.15.70:0/2707256945 shutdown_connections 2024-06-04T05:20:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.082+0000 7f0320932640 1 -- 172.21.15.70:0/2707256945 wait complete. 2024-06-04T05:20:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.083+0000 7f0320932640 1 Processor -- start 2024-06-04T05:20:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.083+0000 7f0320932640 1 -- start start 2024-06-04T05:20:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.083+0000 7f0320932640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f031c104ec0 0x7f031c111ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.083+0000 7f0320932640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031c109f90 0x7f031c10ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.083+0000 7f0320932640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f031c10f2b0 0x7f031c1abaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.083+0000 7f0320932640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f031c114df0 con 0x7f031c109f90 2024-06-04T05:20:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.083+0000 7f0320932640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f031c114c70 con 0x7f031c10f2b0 2024-06-04T05:20:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.083+0000 7f0320932640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f031c114f70 con 0x7f031c104ec0 2024-06-04T05:20:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.084+0000 7f031ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f031c10f2b0 0x7f031c1abaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.084+0000 7f0319d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031c109f90 0x7f031c10ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.084+0000 7f031ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f031c10f2b0 0x7f031c1abaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:48170/0 (socket says 172.21.15.70:48170) 2024-06-04T05:20:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.084+0000 7f031ad9d640 1 -- 172.21.15.70:0/1889372883 learned_addr learned my addr 172.21.15.70:0/1889372883 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.084+0000 7f031a59c640 1 --2- 172.21.15.70:0/1889372883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f031c104ec0 0x7f031c111ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.084+0000 7f031ad9d640 1 -- 172.21.15.70:0/1889372883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f031c104ec0 msgr2=0x7f031c111ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.084+0000 7f031ad9d640 1 --2- 172.21.15.70:0/1889372883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f031c104ec0 0x7f031c111ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.084+0000 7f031ad9d640 1 -- 172.21.15.70:0/1889372883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031c109f90 msgr2=0x7f031c10ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.084+0000 7f031ad9d640 1 --2- 172.21.15.70:0/1889372883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031c109f90 0x7f031c10ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.084+0000 7f031ad9d640 1 -- 172.21.15.70:0/1889372883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f031c10fa50 con 0x7f031c10f2b0 2024-06-04T05:20:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.084+0000 7f0319d9b640 1 --2- 172.21.15.70:0/1889372883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031c109f90 0x7f031c10ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.084+0000 7f031ad9d640 1 --2- 172.21.15.70:0/1889372883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f031c10f2b0 0x7f031c1abaf0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f031400eaa0 tx=0x7f031400ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.085+0000 7f030b7fe640 1 -- 172.21.15.70:0/1889372883 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031400cdb0 con 0x7f031c10f2b0 2024-06-04T05:20:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.085+0000 7f030b7fe640 1 -- 172.21.15.70:0/1889372883 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0314005020 con 0x7f031c10f2b0 2024-06-04T05:20:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.085+0000 7f030b7fe640 1 -- 172.21.15.70:0/1889372883 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03140116d0 con 0x7f031c10f2b0 2024-06-04T05:20:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.085+0000 7f0320932640 1 -- 172.21.15.70:0/1889372883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f031c1ac090 con 0x7f031c10f2b0 2024-06-04T05:20:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.085+0000 7f0320932640 1 -- 172.21.15.70:0/1889372883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f031c1ac560 con 0x7f031c10f2b0 2024-06-04T05:20:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.088+0000 7f030b7fe640 1 -- 172.21.15.70:0/1889372883 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f03140051d0 con 0x7f031c10f2b0 2024-06-04T05:20:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.089+0000 7f0323650740 1 Processor -- start 2024-06-04T05:20:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.089+0000 7f0323650740 1 -- start start 2024-06-04T05:20:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.089+0000 7f0323650740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6e820 0x556ebdd6ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.089+0000 7f0323650740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6f140 0x556ebdd794c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.089+0000 7f0323650740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 0x556ebdd7bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.089+0000 7f0323650740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556ebdce0c30 con 0x556ebdd6e820 2024-06-04T05:20:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.089+0000 7f0323650740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556ebdcd07d0 con 0x556ebdd79a00 2024-06-04T05:20:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.089+0000 7f0323650740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556ebdcd4810 con 0x556ebdd6f140 2024-06-04T05:20:33.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.090+0000 7f0319d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6e820 0x556ebdd6ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.090+0000 7f0319d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6e820 0x556ebdd6ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56504/0 (socket says 172.21.15.70:56504) 2024-06-04T05:20:33.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.090+0000 7f0319d9b640 1 -- 172.21.15.70:0/2744450080 learned_addr learned my addr 172.21.15.70:0/2744450080 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:33.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.090+0000 7f031a59c640 1 --2- 172.21.15.70:0/2744450080 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6f140 0x556ebdd794c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.090+0000 7f0319d9b640 1 --2- 172.21.15.70:0/2744450080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 0x556ebdd7bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.090+0000 7f0319d9b640 1 -- 172.21.15.70:0/2744450080 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6f140 msgr2=0x556ebdd794c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:33.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.090+0000 7f0319d9b640 1 --2- 172.21.15.70:0/2744450080 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6f140 0x556ebdd794c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.090+0000 7f0319d9b640 1 -- 172.21.15.70:0/2744450080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 msgr2=0x556ebdd7bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:33.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.090+0000 7f0319d9b640 1 --2- 172.21.15.70:0/2744450080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 0x556ebdd7bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.090+0000 7f0319d9b640 1 -- 172.21.15.70:0/2744450080 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ebd9e5210 con 0x556ebdd6e820 2024-06-04T05:20:33.093 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.090+0000 7f031a59c640 1 --2- 172.21.15.70:0/2744450080 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6f140 0x556ebdd794c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.090+0000 7f0319d9b640 1 --2- 172.21.15.70:0/2744450080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6e820 0x556ebdd6ec00 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f0310000c00 tx=0x7f0310009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d174d87b594a70 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.091+0000 7f0308ff9640 1 -- 172.21.15.70:0/2744450080 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310035040 con 0x556ebdd6e820 2024-06-04T05:20:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.091+0000 7f0308ff9640 1 -- 172.21.15.70:0/2744450080 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0310004b50 con 0x556ebdd6e820 2024-06-04T05:20:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.091+0000 7f0308ff9640 1 -- 172.21.15.70:0/2744450080 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310003c80 con 0x556ebdd6e820 2024-06-04T05:20:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.091+0000 7f030b7fe640 1 --2- 172.21.15.70:0/1889372883 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f03040b2ba0 0x7f03040b5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.092+0000 7f030b7fe640 1 -- 172.21.15.70:0/1889372883 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f03140e2660 con 0x7f031c10f2b0 2024-06-04T05:20:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.092+0000 7f031a59c640 1 --2- 172.21.15.70:0/1889372883 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f03040b2ba0 0x7f03040b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.092+0000 7f0323650740 1 -- 172.21.15.70:0/2744450080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6e820 msgr2=0x556ebdd6ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.092+0000 7f0323650740 1 --2- 172.21.15.70:0/2744450080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6e820 0x556ebdd6ec00 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f0310000c00 tx=0x7f0310009660 comp rx=0 tx=0).stop 2024-06-04T05:20:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.092+0000 7f0323650740 1 -- 172.21.15.70:0/2744450080 shutdown_connections 2024-06-04T05:20:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.092+0000 7f0323650740 1 --2- 172.21.15.70:0/2744450080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 0x556ebdd7bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.092+0000 7f0323650740 1 --2- 172.21.15.70:0/2744450080 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6f140 0x556ebdd794c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.092+0000 7f0323650740 1 --2- 172.21.15.70:0/2744450080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6e820 0x556ebdd6ec00 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.092+0000 7f0323650740 1 -- 172.21.15.70:0/2744450080 >> 172.21.15.70:0/2744450080 conn(0x556ebdd6ad30 msgr2=0x556ebdd6b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.092+0000 7f0323650740 1 -- 172.21.15.70:0/2744450080 shutdown_connections 2024-06-04T05:20:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.092+0000 7f0323650740 1 -- 172.21.15.70:0/2744450080 wait complete. 2024-06-04T05:20:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.092+0000 7f031a59c640 1 --2- 172.21.15.70:0/1889372883 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f03040b2ba0 0x7f03040b5060 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x556ebdd6fdc0 tx=0x7f030c00b410 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.092+0000 7f0323650740 1 -- start start 2024-06-04T05:20:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f0323650740 1 Processor -- start 2024-06-04T05:20:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f0323650740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 0x556ebdd769f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f0323650740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 0x556ebdd76f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f0323650740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 0x556ebdd77470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f0323650740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556ebd9b6790 con 0x556ebdd6f140 2024-06-04T05:20:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f0323650740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556ebdc8c890 con 0x556ebdd79a00 2024-06-04T05:20:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f0323650740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556ebdd7c5d0 con 0x556ebdd6e820 2024-06-04T05:20:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f0323650740 10 client.0 _ll_register_callbacks cb 0x556ebd9e4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f031a59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 0x556ebdd769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f031a59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 0x556ebdd769f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51726/0 (socket says 172.21.15.70:51726) 2024-06-04T05:20:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f031a59c640 1 -- 172.21.15.70:0/934958713 learned_addr learned my addr 172.21.15.70:0/934958713 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f0319d9b640 1 --2- 172.21.15.70:0/934958713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 0x556ebdd76f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f031ad9d640 1 --2- 172.21.15.70:0/934958713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 0x556ebdd77470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f031a59c640 1 -- 172.21.15.70:0/934958713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 msgr2=0x556ebdd77470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f031a59c640 1 --2- 172.21.15.70:0/934958713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 0x556ebdd77470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f031a59c640 1 -- 172.21.15.70:0/934958713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 msgr2=0x556ebdd76f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f031a59c640 1 --2- 172.21.15.70:0/934958713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 0x556ebdd76f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.093+0000 7f031a59c640 1 -- 172.21.15.70:0/934958713 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ebdab1620 con 0x556ebdd6e820 2024-06-04T05:20:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.094+0000 7f031a59c640 1 --2- 172.21.15.70:0/934958713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 0x556ebdd769f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f030c01f5a0 tx=0x7f030c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4248a7e6d6ecc6b7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.094+0000 7f0301ffb640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:20:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.094+0000 7f0301ffb640 1 -- 172.21.15.70:0/934958713 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030c02c060 con 0x556ebdd6e820 2024-06-04T05:20:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.094+0000 7f0301ffb640 1 -- 172.21.15.70:0/934958713 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f030c017030 con 0x556ebdd6e820 2024-06-04T05:20:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.094+0000 7f0323650740 1 -- 172.21.15.70:0/934958713 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0310004b50 con 0x556ebdd6e820 2024-06-04T05:20:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.094+0000 7f0301ffb640 1 -- 172.21.15.70:0/934958713 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f030c0107c0 con 0x556ebdd6e820 2024-06-04T05:20:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.094+0000 7f0323650740 10 client.30181 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.094+0000 7f0323650740 1 -- 172.21.15.70:0/934958713 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x556ebdd883e0 con 0x556ebdd6e820 2024-06-04T05:20:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.094+0000 7f0301ffb640 1 -- 172.21.15.70:0/934958713 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0310004b50 con 0x556ebdd6e820 2024-06-04T05:20:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.095+0000 7f0301ffb640 1 -- 172.21.15.70:0/934958713 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f030c0279f0 con 0x556ebdd6e820 2024-06-04T05:20:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.095+0000 7f0323650740 10 client.30181 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.095+0000 7f0323650740 10 client.30181 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.095+0000 7f0323650740 20 client.30181 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.095+0000 7f0323650740 1 --2- 172.21.15.70:0/934958713 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556ebdd89d40 0x556ebdd8c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.095+0000 7f0323650740 4 client.30181 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:20:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.095+0000 7f0323650740 1 -- 172.21.15.70:0/934958713 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ebdce2b80 con 0x556ebdd89d40 2024-06-04T05:20:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.095+0000 7f0319d9b640 1 --2- 172.21.15.70:0/934958713 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556ebdd89d40 0x556ebdd8c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.095+0000 7f0301ffb640 10 client.30181 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.095+0000 7f0319d9b640 1 --2- 172.21.15.70:0/934958713 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556ebdd89d40 0x556ebdd8c200 crc :-1 s=READY pgs=487 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abc701af7d66089f server_cookie=96a9a6605d2fdf7b in_seq=0 out_seq=0 2024-06-04T05:20:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.096+0000 7f0301ffb640 1 -- 172.21.15.70:0/934958713 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556ebdce2b80 con 0x556ebdd89d40 2024-06-04T05:20:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.096+0000 7f0301ffb640 10 client.30181 handle_command_reply: tid=0 2024-06-04T05:20:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.096+0000 7f0323650740 1 client.30181 shutdown 2024-06-04T05:20:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.096+0000 7f0323650740 1 -- 172.21.15.70:0/934958713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 msgr2=0x556ebdd769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.096+0000 7f0323650740 1 --2- 172.21.15.70:0/934958713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 0x556ebdd769f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f030c01f5a0 tx=0x7f030c0046c0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.096+0000 7f0323650740 1 -- 172.21.15.70:0/934958713 shutdown_connections 2024-06-04T05:20:33.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.096+0000 7f0323650740 1 --2- 172.21.15.70:0/934958713 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556ebdd89d40 0x556ebdd8c200 crc :-1 s=READY pgs=487 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.096+0000 7f0323650740 1 --2- 172.21.15.70:0/934958713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 0x556ebdd77470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.096+0000 7f0323650740 1 --2- 172.21.15.70:0/934958713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 0x556ebdd76f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.097+0000 7f0323650740 1 --2- 172.21.15.70:0/934958713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 0x556ebdd769f0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.097+0000 7f0323650740 1 -- 172.21.15.70:0/934958713 >> 172.21.15.70:0/934958713 conn(0x556ebdd6ad30 msgr2=0x556ebdd69fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:33.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.097+0000 7f0301ffb640 0 client.30181 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:33.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.097+0000 7f0323650740 1 -- 172.21.15.70:0/934958713 shutdown_connections 2024-06-04T05:20:33.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.097+0000 7f0323650740 1 -- 172.21.15.70:0/934958713 wait complete. 2024-06-04T05:20:33.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.097+0000 7f0323650740 20 client.30181 trim_cache size 0 max 16384 2024-06-04T05:20:33.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.118+0000 7f0323650740 1 Processor -- start 2024-06-04T05:20:33.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.118+0000 7f0323650740 1 -- start start 2024-06-04T05:20:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.118+0000 7f0323650740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 0x556ebdd708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.118+0000 7f0323650740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 0x556ebdd76f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.118+0000 7f0323650740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 0x556ebdd77480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.118+0000 7f0323650740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f031c1b4c10 con 0x556ebdd6f140 2024-06-04T05:20:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.118+0000 7f0323650740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f031c1b67e0 con 0x556ebdd79a00 2024-06-04T05:20:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.118+0000 7f0323650740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f031c1b5340 con 0x556ebdd6e820 2024-06-04T05:20:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.118+0000 7f0319d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 0x556ebdd76f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.118+0000 7f0319d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 0x556ebdd76f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56530/0 (socket says 172.21.15.70:56530) 2024-06-04T05:20:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.118+0000 7f0319d9b640 1 -- 172.21.15.70:0/4288404786 learned_addr learned my addr 172.21.15.70:0/4288404786 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.118+0000 7f031a59c640 1 --2- 172.21.15.70:0/4288404786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 0x556ebdd708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.119+0000 7f031ad9d640 1 --2- 172.21.15.70:0/4288404786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 0x556ebdd77480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.119+0000 7f0319d9b640 1 -- 172.21.15.70:0/4288404786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 msgr2=0x556ebdd708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.119+0000 7f0319d9b640 1 --2- 172.21.15.70:0/4288404786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 0x556ebdd708e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.119+0000 7f0319d9b640 1 -- 172.21.15.70:0/4288404786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 msgr2=0x556ebdd77480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.119+0000 7f0319d9b640 1 --2- 172.21.15.70:0/4288404786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 0x556ebdd77480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.119+0000 7f0319d9b640 1 -- 172.21.15.70:0/4288404786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ebdd664b0 con 0x556ebdd6f140 2024-06-04T05:20:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.119+0000 7f031a59c640 1 --2- 172.21.15.70:0/4288404786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 0x556ebdd708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.119+0000 7f031ad9d640 1 --2- 172.21.15.70:0/4288404786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 0x556ebdd77480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.119+0000 7f0319d9b640 1 --2- 172.21.15.70:0/4288404786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 0x556ebdd76f40 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f0310009900 tx=0x7f031002a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcbdea853deae9e2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.119+0000 7f0303fff640 1 -- 172.21.15.70:0/4288404786 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031005b070 con 0x556ebdd6f140 2024-06-04T05:20:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.119+0000 7f0303fff640 1 -- 172.21.15.70:0/4288404786 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0310040030 con 0x556ebdd6f140 2024-06-04T05:20:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.119+0000 7f0303fff640 1 -- 172.21.15.70:0/4288404786 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310035040 con 0x556ebdd6f140 2024-06-04T05:20:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.120+0000 7f0323650740 1 -- 172.21.15.70:0/4288404786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 msgr2=0x556ebdd76f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.120+0000 7f0323650740 1 --2- 172.21.15.70:0/4288404786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 0x556ebdd76f40 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f0310009900 tx=0x7f031002a3f0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.120+0000 7f0323650740 1 -- 172.21.15.70:0/4288404786 shutdown_connections 2024-06-04T05:20:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.120+0000 7f0323650740 1 --2- 172.21.15.70:0/4288404786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 0x556ebdd77480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.120+0000 7f0323650740 1 --2- 172.21.15.70:0/4288404786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 0x556ebdd76f40 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.120+0000 7f0323650740 1 --2- 172.21.15.70:0/4288404786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 0x556ebdd708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.120+0000 7f0323650740 1 -- 172.21.15.70:0/4288404786 >> 172.21.15.70:0/4288404786 conn(0x556ebdd6ad30 msgr2=0x556ebdd701d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.120+0000 7f0323650740 1 -- 172.21.15.70:0/4288404786 shutdown_connections 2024-06-04T05:20:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.120+0000 7f0323650740 1 -- 172.21.15.70:0/4288404786 wait complete. 2024-06-04T05:20:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.121+0000 7f0323650740 1 -- start start 2024-06-04T05:20:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.121+0000 7f0323650740 1 Processor -- start 2024-06-04T05:20:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.121+0000 7f0323650740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 0x556ebdd7c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.121+0000 7f0323650740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 0x556ebdd7cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.121+0000 7f0323650740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 0x556ebdd7d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.121+0000 7f0323650740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f031c1b60e0 con 0x556ebdd6f140 2024-06-04T05:20:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.121+0000 7f0323650740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f031c1b5a10 con 0x556ebdd79a00 2024-06-04T05:20:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.121+0000 7f0323650740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f031c1b2c10 con 0x556ebdd6e820 2024-06-04T05:20:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.121+0000 7f0323650740 10 client.0 _ll_register_callbacks cb 0x7f030c011e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.121+0000 7f031a59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 0x556ebdd7c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.121+0000 7f0319d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 0x556ebdd7cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.121+0000 7f0319d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 0x556ebdd7cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56544/0 (socket says 172.21.15.70:56544) 2024-06-04T05:20:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.121+0000 7f031a59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 0x556ebdd7c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51738/0 (socket says 172.21.15.70:51738) 2024-06-04T05:20:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.121+0000 7f0319d9b640 1 -- 172.21.15.70:0/4259340426 learned_addr learned my addr 172.21.15.70:0/4259340426 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.121+0000 7f031ad9d640 1 --2- 172.21.15.70:0/4259340426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 0x556ebdd7d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.122+0000 7f031a59c640 1 -- 172.21.15.70:0/4259340426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 msgr2=0x556ebdd7d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.122+0000 7f031a59c640 1 --2- 172.21.15.70:0/4259340426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 0x556ebdd7d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.122+0000 7f031a59c640 1 -- 172.21.15.70:0/4259340426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 msgr2=0x556ebdd7cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.122+0000 7f031a59c640 1 --2- 172.21.15.70:0/4259340426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 0x556ebdd7cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.122+0000 7f031a59c640 1 -- 172.21.15.70:0/4259340426 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ebddf2b00 con 0x556ebdd6e820 2024-06-04T05:20:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.122+0000 7f031a59c640 1 --2- 172.21.15.70:0/4259340426 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 0x556ebdd7c7e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f030c018040 tx=0x7f030c010b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=598fe08f8381ba72 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.122+0000 7f0301ffb640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:20:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.122+0000 7f0301ffb640 1 -- 172.21.15.70:0/4259340426 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030c01ca90 con 0x556ebdd6e820 2024-06-04T05:20:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.122+0000 7f0323650740 1 -- 172.21.15.70:0/4259340426 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0310040030 con 0x556ebdd6e820 2024-06-04T05:20:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.122+0000 7f0301ffb640 1 -- 172.21.15.70:0/4259340426 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f030c01cc30 con 0x556ebdd6e820 2024-06-04T05:20:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.122+0000 7f0301ffb640 1 -- 172.21.15.70:0/4259340426 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f030c01ce50 con 0x556ebdd6e820 2024-06-04T05:20:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.122+0000 7f0323650740 10 client.30187 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.122+0000 7f0323650740 1 -- 172.21.15.70:0/4259340426 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x556ebddf3e10 con 0x556ebdd6e820 2024-06-04T05:20:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.123+0000 7f0301ffb640 1 -- 172.21.15.70:0/4259340426 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0310040030 con 0x556ebdd6e820 2024-06-04T05:20:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.123+0000 7f0301ffb640 1 -- 172.21.15.70:0/4259340426 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f030c03f070 con 0x556ebdd6e820 2024-06-04T05:20:33.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.123+0000 7f0323650740 10 client.30187 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:33.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.123+0000 7f0323650740 10 client.30187 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:33.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.123+0000 7f0323650740 20 client.30187 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:33.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.123+0000 7f0323650740 1 --2- 172.21.15.70:0/4259340426 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556ebddf4590 0x556ebddf69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.123+0000 7f0323650740 4 client.30187 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:20:33.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.123+0000 7f0323650740 1 -- 172.21.15.70:0/4259340426 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556ebdcdc360 con 0x556ebddf4590 2024-06-04T05:20:33.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.123+0000 7f0319d9b640 1 --2- 172.21.15.70:0/4259340426 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556ebddf4590 0x556ebddf69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.124+0000 7f0301ffb640 10 client.30187 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:33.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.124+0000 7f0319d9b640 1 --2- 172.21.15.70:0/4259340426 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556ebddf4590 0x556ebddf69b0 crc :-1 s=READY pgs=488 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efce5dfc6bc3289b server_cookie=d4eca988a72d45da in_seq=0 out_seq=0 2024-06-04T05:20:33.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.231+0000 7f0301ffb640 1 -- 172.21.15.70:0/4259340426 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x556ebdcdc360 con 0x556ebddf4590 2024-06-04T05:20:33.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.231+0000 7f0301ffb640 10 client.30187 handle_command_reply: tid=0 2024-06-04T05:20:33.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.231+0000 7f0323650740 1 client.30187 shutdown 2024-06-04T05:20:33.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.232+0000 7f0323650740 1 -- 172.21.15.70:0/4259340426 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 msgr2=0x556ebdd7c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.232+0000 7f0323650740 1 --2- 172.21.15.70:0/4259340426 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 0x556ebdd7c7e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f030c018040 tx=0x7f030c010b90 comp rx=0 tx=0).stop 2024-06-04T05:20:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.232+0000 7f0323650740 1 -- 172.21.15.70:0/4259340426 shutdown_connections 2024-06-04T05:20:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.232+0000 7f0323650740 1 --2- 172.21.15.70:0/4259340426 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x556ebddf4590 0x556ebddf69b0 crc :-1 s=READY pgs=488 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.232+0000 7f0323650740 1 --2- 172.21.15.70:0/4259340426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556ebdd79a00 0x556ebdd7d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.232+0000 7f0323650740 1 --2- 172.21.15.70:0/4259340426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ebdd6f140 0x556ebdd7cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.232+0000 7f0323650740 1 --2- 172.21.15.70:0/4259340426 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556ebdd6e820 0x556ebdd7c7e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.232+0000 7f0323650740 1 -- 172.21.15.70:0/4259340426 >> 172.21.15.70:0/4259340426 conn(0x556ebdd6ad30 msgr2=0x556ebdd70a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.232+0000 7f0301ffb640 0 client.30187 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.232+0000 7f0323650740 1 -- 172.21.15.70:0/4259340426 shutdown_connections 2024-06-04T05:20:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.232+0000 7f0323650740 1 -- 172.21.15.70:0/4259340426 wait complete. 2024-06-04T05:20:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.232+0000 7f0323650740 20 client.30187 trim_cache size 0 max 16384 2024-06-04T05:20:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:20:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:20:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:20:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.233+0000 7f0320932640 1 -- 172.21.15.70:0/1889372883 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f03040b2ba0 msgr2=0x7f03040b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.233+0000 7f0320932640 1 --2- 172.21.15.70:0/1889372883 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f03040b2ba0 0x7f03040b5060 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x556ebdd6fdc0 tx=0x7f030c00b410 comp rx=0 tx=0).stop 2024-06-04T05:20:33.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.233+0000 7f0320932640 1 -- 172.21.15.70:0/1889372883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f031c10f2b0 msgr2=0x7f031c1abaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.233+0000 7f0320932640 1 --2- 172.21.15.70:0/1889372883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f031c10f2b0 0x7f031c1abaf0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f031400eaa0 tx=0x7f031400ef60 comp rx=0 tx=0).stop 2024-06-04T05:20:33.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.234+0000 7f0320932640 1 -- 172.21.15.70:0/1889372883 shutdown_connections 2024-06-04T05:20:33.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.234+0000 7f0320932640 1 --2- 172.21.15.70:0/1889372883 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f03040b2ba0 0x7f03040b5060 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.234+0000 7f0320932640 1 --2- 172.21.15.70:0/1889372883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f031c10f2b0 0x7f031c1abaf0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.234+0000 7f0320932640 1 --2- 172.21.15.70:0/1889372883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031c109f90 0x7f031c10ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.234+0000 7f0320932640 1 --2- 172.21.15.70:0/1889372883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f031c104ec0 0x7f031c111ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.234+0000 7f0320932640 1 -- 172.21.15.70:0/1889372883 >> 172.21.15.70:0/1889372883 conn(0x7f031c100700 msgr2=0x7f031c101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:33.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.234+0000 7f0320932640 1 -- 172.21.15.70:0/1889372883 shutdown_connections 2024-06-04T05:20:33.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:33.234+0000 7f0320932640 1 -- 172.21.15.70:0/1889372883 wait complete. 2024-06-04T05:20:33.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.329+0000 7fd0f1ed1700 1 -- 172.21.15.70:0/1209677233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0e4094ee0 msgr2=0x7fd0e4095360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.329+0000 7fd0f1ed1700 1 --2- 172.21.15.70:0/1209677233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0e4094ee0 0x7fd0e4095360 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8009990 tx=0x7fd0d8009ca0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.329+0000 7fd0f1ed1700 1 -- 172.21.15.70:0/1209677233 shutdown_connections 2024-06-04T05:20:33.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.329+0000 7fd0f1ed1700 1 --2- 172.21.15.70:0/1209677233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0e40958a0 0x7fd0e40a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.329+0000 7fd0f1ed1700 1 --2- 172.21.15.70:0/1209677233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0e409aca0 0x7fd0e409b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.329+0000 7fd0f1ed1700 1 --2- 172.21.15.70:0/1209677233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0e4094ee0 0x7fd0e4095360 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.329+0000 7fd0f1ed1700 1 -- 172.21.15.70:0/1209677233 >> 172.21.15.70:0/1209677233 conn(0x7fd0e400a7a0 msgr2=0x7fd0e400abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:33.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.329+0000 7fd0f1ed1700 1 -- 172.21.15.70:0/1209677233 shutdown_connections 2024-06-04T05:20:33.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.329+0000 7fd0f1ed1700 1 -- 172.21.15.70:0/1209677233 wait complete. 2024-06-04T05:20:33.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.330+0000 7fd0f1ed1700 1 Processor -- start 2024-06-04T05:20:33.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.330+0000 7fd0f1ed1700 1 -- start start 2024-06-04T05:20:33.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.331+0000 7fd0f1ed1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0e4094ee0 0x7fd0e4129f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.331+0000 7fd0f1ed1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0e40958a0 0x7fd0e412a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.331+0000 7fd0f1ed1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0e409aca0 0x7fd0e412e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.331+0000 7fd0f1ed1700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e40a3c40 con 0x7fd0e4094ee0 2024-06-04T05:20:33.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.331+0000 7fd0f1ed1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e40a3ac0 con 0x7fd0e40958a0 2024-06-04T05:20:33.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.331+0000 7fd0f1ed1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e40a3dc0 con 0x7fd0e409aca0 2024-06-04T05:20:33.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.331+0000 7fd0f0ecf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0e4094ee0 0x7fd0e4129f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.331+0000 7fd0f0ecf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0e4094ee0 0x7fd0e4129f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56566/0 (socket says 172.21.15.70:56566) 2024-06-04T05:20:33.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.331+0000 7fd0f0ecf700 1 -- 172.21.15.70:0/222745653 learned_addr learned my addr 172.21.15.70:0/222745653 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:33.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.331+0000 7fd0f16d0700 1 --2- 172.21.15.70:0/222745653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0e409aca0 0x7fd0e412e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.331+0000 7fd0ebfff700 1 --2- 172.21.15.70:0/222745653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0e40958a0 0x7fd0e412a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.331+0000 7fd0f16d0700 1 -- 172.21.15.70:0/222745653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0e40958a0 msgr2=0x7fd0e412a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.331+0000 7fd0f16d0700 1 --2- 172.21.15.70:0/222745653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0e40958a0 0x7fd0e412a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.331+0000 7fd0f16d0700 1 -- 172.21.15.70:0/222745653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0e4094ee0 msgr2=0x7fd0e4129f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.331+0000 7fd0f16d0700 1 --2- 172.21.15.70:0/222745653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0e4094ee0 0x7fd0e4129f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.331+0000 7fd0f16d0700 1 -- 172.21.15.70:0/222745653 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e412ef40 con 0x7fd0e409aca0 2024-06-04T05:20:33.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.332+0000 7fd0f16d0700 1 --2- 172.21.15.70:0/222745653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0e409aca0 0x7fd0e412e800 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec069fd0 tx=0x7fd0ec075c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:33.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.332+0000 7fd0e9ffb700 1 -- 172.21.15.70:0/222745653 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec084070 con 0x7fd0e409aca0 2024-06-04T05:20:33.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.332+0000 7fd0e9ffb700 1 -- 172.21.15.70:0/222745653 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd0ec06bd20 con 0x7fd0e409aca0 2024-06-04T05:20:33.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.332+0000 7fd0e9ffb700 1 -- 172.21.15.70:0/222745653 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec07f620 con 0x7fd0e409aca0 2024-06-04T05:20:33.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.332+0000 7fd0f1ed1700 1 -- 172.21.15.70:0/222745653 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0e412f230 con 0x7fd0e409aca0 2024-06-04T05:20:33.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.332+0000 7fd0f1ed1700 1 -- 172.21.15.70:0/222745653 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0e4136ab0 con 0x7fd0e409aca0 2024-06-04T05:20:33.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.335+0000 7fd0e9ffb700 1 -- 172.21.15.70:0/222745653 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fd0ec087070 con 0x7fd0e409aca0 2024-06-04T05:20:33.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.337+0000 7fd0e9ffb700 1 --2- 172.21.15.70:0/222745653 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd0dc0b2a70 0x7fd0dc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.337+0000 7fd0e9ffb700 1 -- 172.21.15.70:0/222745653 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7fd0ec1503a0 con 0x7fd0e409aca0 2024-06-04T05:20:33.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.337+0000 7fd0f0ecf700 1 --2- 172.21.15.70:0/222745653 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd0dc0b2a70 0x7fd0dc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.338+0000 7fd0f0ecf700 1 --2- 172.21.15.70:0/222745653 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd0dc0b2a70 0x7fd0dc0b4f30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0009ca0 tx=0x7fd0e0009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:33.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.338+0000 7fd0fd9c8b80 1 Processor -- start 2024-06-04T05:20:33.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.338+0000 7fd0fd9c8b80 1 -- start start 2024-06-04T05:20:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.338+0000 7fd0fd9c8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb5de0 0x55e426eb61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.338+0000 7fd0fd9c8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb1680 0x55e426eb1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.339+0000 7fd0fd9c8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb2040 0x55e426ebdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.339+0000 7fd0fd9c8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e426e10c90 con 0x55e426eb5de0 2024-06-04T05:20:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.339+0000 7fd0fd9c8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e426e00830 con 0x55e426eb2040 2024-06-04T05:20:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.339+0000 7fd0fd9c8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e426e04870 con 0x55e426eb1680 2024-06-04T05:20:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.339+0000 7fd0f0ecf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb5de0 0x55e426eb61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.339+0000 7fd0f0ecf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb5de0 0x55e426eb61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56576/0 (socket says 172.21.15.70:56576) 2024-06-04T05:20:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.339+0000 7fd0f0ecf700 1 -- 172.21.15.70:0/1566099631 learned_addr learned my addr 172.21.15.70:0/1566099631 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.339+0000 7fd0ebfff700 1 --2- 172.21.15.70:0/1566099631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb1680 0x55e426eb1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.339+0000 7fd0f16d0700 1 --2- 172.21.15.70:0/1566099631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb2040 0x55e426ebdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.339+0000 7fd0ebfff700 1 -- 172.21.15.70:0/1566099631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb2040 msgr2=0x55e426ebdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:33.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.339+0000 7fd0ebfff700 1 --2- 172.21.15.70:0/1566099631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb2040 0x55e426ebdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.339+0000 7fd0ebfff700 1 -- 172.21.15.70:0/1566099631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb5de0 msgr2=0x55e426eb61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:33.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.339+0000 7fd0ebfff700 1 --2- 172.21.15.70:0/1566099631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb5de0 0x55e426eb61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.339+0000 7fd0ebfff700 1 -- 172.21.15.70:0/1566099631 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e426d508d0 con 0x55e426eb1680 2024-06-04T05:20:33.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.339+0000 7fd0ebfff700 1 --2- 172.21.15.70:0/1566099631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb1680 0x55e426eb1b00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8006010 tx=0x7fd0d80095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=115c3d883826b4c5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:33.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.340+0000 7fd0d6ffd700 1 -- 172.21.15.70:0/1566099631 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8027070 con 0x55e426eb1680 2024-06-04T05:20:33.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.340+0000 7fd0d6ffd700 1 -- 172.21.15.70:0/1566099631 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd0d8005da0 con 0x55e426eb1680 2024-06-04T05:20:33.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.340+0000 7fd0d6ffd700 1 -- 172.21.15.70:0/1566099631 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8005580 con 0x55e426eb1680 2024-06-04T05:20:33.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.341+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/1566099631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb1680 msgr2=0x55e426eb1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.341+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/1566099631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb1680 0x55e426eb1b00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8006010 tx=0x7fd0d80095d0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.341+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/1566099631 shutdown_connections 2024-06-04T05:20:33.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.341+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/1566099631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb2040 0x55e426ebdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.341+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/1566099631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb5de0 0x55e426eb61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.341+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/1566099631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb1680 0x55e426eb1b00 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.341+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/1566099631 >> 172.21.15.70:0/1566099631 conn(0x55e426ddf080 msgr2=0x55e426eabe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:33.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.341+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/1566099631 shutdown_connections 2024-06-04T05:20:33.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.341+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/1566099631 wait complete. 2024-06-04T05:20:33.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.341+0000 7fd0fd9c8b80 1 -- start start 2024-06-04T05:20:33.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.341+0000 7fd0fd9c8b80 1 Processor -- start 2024-06-04T05:20:33.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.342+0000 7fd0fd9c8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb1680 0x55e426ebda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.342+0000 7fd0fd9c8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb2040 0x55e426ebad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.342+0000 7fd0fd9c8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb5de0 0x55e426ebb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.342+0000 7fd0f0ecf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb1680 0x55e426ebda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.342+0000 7fd0fd9c8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e426d06fa0 con 0x55e426eb2040 2024-06-04T05:20:33.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.342+0000 7fd0fd9c8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e426da27c0 con 0x55e426eb1680 2024-06-04T05:20:33.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.342+0000 7fd0fd9c8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e426ebe350 con 0x55e426eb5de0 2024-06-04T05:20:33.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.342+0000 7fd0f0ecf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb1680 0x55e426ebda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:48242/0 (socket says 172.21.15.70:48242) 2024-06-04T05:20:33.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.342+0000 7fd0ebfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb2040 0x55e426ebad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.342+0000 7fd0f0ecf700 1 -- 172.21.15.70:0/1584082111 learned_addr learned my addr 172.21.15.70:0/1584082111 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:33.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.342+0000 7fd0fd9c8b80 10 client.0 _ll_register_callbacks cb 0x55e426d9b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:33.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.342+0000 7fd0f16d0700 1 --2- 172.21.15.70:0/1584082111 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb5de0 0x55e426ebb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.342+0000 7fd0f0ecf700 1 -- 172.21.15.70:0/1584082111 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb5de0 msgr2=0x55e426ebb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:33.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.342+0000 7fd0f0ecf700 1 --2- 172.21.15.70:0/1584082111 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb5de0 0x55e426ebb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.342+0000 7fd0f0ecf700 1 -- 172.21.15.70:0/1584082111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb2040 msgr2=0x55e426ebad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:33.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.342+0000 7fd0f0ecf700 1 --2- 172.21.15.70:0/1584082111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb2040 0x55e426ebad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.342+0000 7fd0f0ecf700 1 -- 172.21.15.70:0/1584082111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e426d9f690 con 0x55e426eb1680 2024-06-04T05:20:33.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.343+0000 7fd0f0ecf700 1 --2- 172.21.15.70:0/1584082111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb1680 0x55e426ebda70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd0e00097f0 tx=0x7fd0e00047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6d48839fb28cdad server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:33.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.343+0000 7fd0cbfff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:20:33.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.343+0000 7fd0cbfff700 1 -- 172.21.15.70:0/1584082111 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e001dae0 con 0x55e426eb1680 2024-06-04T05:20:33.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.343+0000 7fd0cbfff700 1 -- 172.21.15.70:0/1584082111 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7fd0e0009040 con 0x55e426eb1680 2024-06-04T05:20:33.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.344+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/1584082111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0d8005da0 con 0x55e426eb1680 2024-06-04T05:20:33.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.344+0000 7fd0cbfff700 1 -- 172.21.15.70:0/1584082111 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7fd0e001dc80 con 0x55e426eb1680 2024-06-04T05:20:33.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.344+0000 7fd0fd9c8b80 10 client.30056 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:33.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.344+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/1584082111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55e426ed1330 con 0x55e426eb1680 2024-06-04T05:20:33.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.344+0000 7fd0cbfff700 1 -- 172.21.15.70:0/1584082111 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd0d8005da0 con 0x55e426eb1680 2024-06-04T05:20:33.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.344+0000 7fd0cbfff700 1 -- 172.21.15.70:0/1584082111 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7fd0e0027420 con 0x55e426eb1680 2024-06-04T05:20:33.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.344+0000 7fd0fd9c8b80 10 client.30056 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:33.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.344+0000 7fd0fd9c8b80 10 client.30056 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:33.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.344+0000 7fd0fd9c8b80 20 client.30056 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:33.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.344+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/1584082111 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e426ed3910 0x55e426ed5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.344+0000 7fd0fd9c8b80 4 client.30056 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:20:33.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.344+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/1584082111 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e426e7d9e0 con 0x55e426ed3910 2024-06-04T05:20:33.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.345+0000 7fd0ebfff700 1 --2- 172.21.15.70:0/1584082111 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e426ed3910 0x55e426ed5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.345+0000 7fd0cbfff700 10 client.30056 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:33.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.346+0000 7fd0ebfff700 1 --2- 172.21.15.70:0/1584082111 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e426ed3910 0x55e426ed5d00 crc :-1 s=READY pgs=489 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18e0a406a4df5a32 server_cookie=239053fd22b365e9 in_seq=0 out_seq=0 2024-06-04T05:20:33.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.346+0000 7fd0cbfff700 1 -- 172.21.15.70:0/1584082111 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e426e7d9e0 con 0x55e426ed3910 2024-06-04T05:20:33.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.346+0000 7fd0cbfff700 10 client.30056 handle_command_reply: tid=0 2024-06-04T05:20:33.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.346+0000 7fd0fd9c8b80 1 client.30056 shutdown 2024-06-04T05:20:33.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.346+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/1584082111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb1680 msgr2=0x55e426ebda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.347+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/1584082111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb1680 0x55e426ebda70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd0e00097f0 tx=0x7fd0e00047c0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.347+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/1584082111 shutdown_connections 2024-06-04T05:20:33.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.347+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/1584082111 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb5de0 0x55e426ebb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.347+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/1584082111 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e426ed3910 0x55e426ed5d00 crc :-1 s=READY pgs=489 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.347+0000 7fd0cbfff700 0 client.30056 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:33.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.347+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/1584082111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb1680 0x55e426ebda70 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.347+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/1584082111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb2040 0x55e426ebad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.347+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/1584082111 >> 172.21.15.70:0/1584082111 conn(0x55e426ddf080 msgr2=0x55e426eb74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:33.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.347+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/1584082111 shutdown_connections 2024-06-04T05:20:33.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.347+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/1584082111 wait complete. 2024-06-04T05:20:33.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.347+0000 7fd0fd9c8b80 20 client.30056 trim_cache size 0 max 16384 2024-06-04T05:20:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.374+0000 7fd0fd9c8b80 1 Processor -- start 2024-06-04T05:20:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.374+0000 7fd0fd9c8b80 1 -- start start 2024-06-04T05:20:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.375+0000 7fd0fd9c8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb1680 0x55e426eada60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.375+0000 7fd0fd9c8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb2040 0x55e426ebd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.375+0000 7fd0fd9c8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb5de0 0x55e426eb9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.375+0000 7fd0fd9c8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e413e160 con 0x55e426eb1680 2024-06-04T05:20:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.375+0000 7fd0fd9c8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e413fd30 con 0x55e426eb5de0 2024-06-04T05:20:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.375+0000 7fd0fd9c8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e413e890 con 0x55e426eb2040 2024-06-04T05:20:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.375+0000 7fd0ebfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb2040 0x55e426ebd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.375+0000 7fd0ebfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb2040 0x55e426ebd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:51788/0 (socket says 172.21.15.70:51788) 2024-06-04T05:20:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.375+0000 7fd0ebfff700 1 -- 172.21.15.70:0/3433240998 learned_addr learned my addr 172.21.15.70:0/3433240998 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.375+0000 7fd0f16d0700 1 --2- 172.21.15.70:0/3433240998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb5de0 0x55e426eb9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.375+0000 7fd0f0ecf700 1 --2- 172.21.15.70:0/3433240998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb1680 0x55e426eada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.375+0000 7fd0ebfff700 1 -- 172.21.15.70:0/3433240998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb5de0 msgr2=0x55e426eb9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.375+0000 7fd0ebfff700 1 --2- 172.21.15.70:0/3433240998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb5de0 0x55e426eb9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.375+0000 7fd0ebfff700 1 -- 172.21.15.70:0/3433240998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb1680 msgr2=0x55e426eada60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.375+0000 7fd0ebfff700 1 --2- 172.21.15.70:0/3433240998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb1680 0x55e426eada60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.375+0000 7fd0ebfff700 1 -- 172.21.15.70:0/3433240998 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e426eaba60 con 0x55e426eb2040 2024-06-04T05:20:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.375+0000 7fd0ebfff700 1 --2- 172.21.15.70:0/3433240998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb2040 0x55e426ebd5b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x55e426eac990 tx=0x7fd0d8005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eace01c3ca774fff server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.376+0000 7fd0d67fc700 1 -- 172.21.15.70:0/3433240998 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d803a070 con 0x55e426eb2040 2024-06-04T05:20:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.376+0000 7fd0d67fc700 1 -- 172.21.15.70:0/3433240998 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd0d8010450 con 0x55e426eb2040 2024-06-04T05:20:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.376+0000 7fd0d67fc700 1 -- 172.21.15.70:0/3433240998 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8027040 con 0x55e426eb2040 2024-06-04T05:20:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.377+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/3433240998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb2040 msgr2=0x55e426ebd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.377+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/3433240998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb2040 0x55e426ebd5b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x55e426eac990 tx=0x7fd0d8005040 comp rx=0 tx=0).stop 2024-06-04T05:20:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.377+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/3433240998 shutdown_connections 2024-06-04T05:20:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.377+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/3433240998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb5de0 0x55e426eb9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.377+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/3433240998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb1680 0x55e426eada60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.377+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/3433240998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb2040 0x55e426ebd5b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.377+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/3433240998 >> 172.21.15.70:0/3433240998 conn(0x55e426ddf080 msgr2=0x55e426eabde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.377+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/3433240998 shutdown_connections 2024-06-04T05:20:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.377+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/3433240998 wait complete. 2024-06-04T05:20:33.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.377+0000 7fd0fd9c8b80 1 -- start start 2024-06-04T05:20:33.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0fd9c8b80 1 Processor -- start 2024-06-04T05:20:33.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0fd9c8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb1680 0x55e426ec8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0fd9c8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb2040 0x55e426ec9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0fd9c8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb5de0 0x55e426f3c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0fd9c8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e413f630 con 0x55e426eb2040 2024-06-04T05:20:33.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0fd9c8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e413ef60 con 0x55e426eb1680 2024-06-04T05:20:33.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0fd9c8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e413c120 con 0x55e426eb5de0 2024-06-04T05:20:33.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0f0ecf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb1680 0x55e426ec8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0ebfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb2040 0x55e426ec9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0f0ecf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb1680 0x55e426ec8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:48258/0 (socket says 172.21.15.70:48258) 2024-06-04T05:20:33.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0f0ecf700 1 -- 172.21.15.70:0/3102182171 learned_addr learned my addr 172.21.15.70:0/3102182171 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:33.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0fd9c8b80 10 client.0 _ll_register_callbacks cb 0x55e426d9b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:33.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0f16d0700 1 --2- 172.21.15.70:0/3102182171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb5de0 0x55e426f3c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0f0ecf700 1 -- 172.21.15.70:0/3102182171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb5de0 msgr2=0x55e426f3c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:33.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0f0ecf700 1 --2- 172.21.15.70:0/3102182171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb5de0 0x55e426f3c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0f0ecf700 1 -- 172.21.15.70:0/3102182171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb2040 msgr2=0x55e426ec9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:33.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0f0ecf700 1 --2- 172.21.15.70:0/3102182171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb2040 0x55e426ec9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0f0ecf700 1 -- 172.21.15.70:0/3102182171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e426f3ca20 con 0x55e426eb1680 2024-06-04T05:20:33.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.378+0000 7fd0ebfff700 1 --2- 172.21.15.70:0/3102182171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb2040 0x55e426ec9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:33.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.379+0000 7fd0f0ecf700 1 --2- 172.21.15.70:0/3102182171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb1680 0x55e426ec8f40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0027a30 tx=0x7fd0e0031720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74b7295c771e0a34 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:33.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.379+0000 7fd0cbfff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:20:33.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.379+0000 7fd0cbfff700 1 -- 172.21.15.70:0/3102182171 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e001dae0 con 0x55e426eb1680 2024-06-04T05:20:33.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.379+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/3102182171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0d8010450 con 0x55e426eb1680 2024-06-04T05:20:33.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.379+0000 7fd0cbfff700 1 -- 172.21.15.70:0/3102182171 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7fd0e0023040 con 0x55e426eb1680 2024-06-04T05:20:33.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.379+0000 7fd0cbfff700 1 -- 172.21.15.70:0/3102182171 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7fd0e001c620 con 0x55e426eb1680 2024-06-04T05:20:33.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.379+0000 7fd0fd9c8b80 10 client.30062 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:33.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.379+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/3102182171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55e426f3db60 con 0x55e426eb1680 2024-06-04T05:20:33.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.379+0000 7fd0cbfff700 1 -- 172.21.15.70:0/3102182171 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd0e001cc50 con 0x55e426eb1680 2024-06-04T05:20:33.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.380+0000 7fd0cbfff700 1 -- 172.21.15.70:0/3102182171 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7fd0e003c420 con 0x55e426eb1680 2024-06-04T05:20:33.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.380+0000 7fd0fd9c8b80 10 client.30062 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:33.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.380+0000 7fd0fd9c8b80 10 client.30062 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:33.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.380+0000 7fd0fd9c8b80 20 client.30062 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:33.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.380+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/3102182171 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e426f3e2a0 0x55e426f406c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:33.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.380+0000 7fd0fd9c8b80 4 client.30062 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:20:33.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.380+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/3102182171 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fd0d8010450 con 0x55e426f3e2a0 2024-06-04T05:20:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.380+0000 7fd0ebfff700 1 --2- 172.21.15.70:0/3102182171 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e426f3e2a0 0x55e426f406c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.380+0000 7fd0ebfff700 1 --2- 172.21.15.70:0/3102182171 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e426f3e2a0 0x55e426f406c0 crc :-1 s=READY pgs=490 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4a9667e686e292a server_cookie=33721aaa42a3fbc in_seq=0 out_seq=0 2024-06-04T05:20:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.380+0000 7fd0cbfff700 10 client.30062 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.380+0000 7fd0cbfff700 1 -- 172.21.15.70:0/3102182171 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fd0d8010450 con 0x55e426f3e2a0 2024-06-04T05:20:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.380+0000 7fd0cbfff700 10 client.30062 handle_command_reply: tid=0 2024-06-04T05:20:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.380+0000 7fd0fd9c8b80 1 client.30062 shutdown 2024-06-04T05:20:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.381+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/3102182171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb1680 msgr2=0x55e426ec8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.381+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/3102182171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb1680 0x55e426ec8f40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0027a30 tx=0x7fd0e0031720 comp rx=0 tx=0).stop 2024-06-04T05:20:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.381+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/3102182171 shutdown_connections 2024-06-04T05:20:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.381+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/3102182171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e426eb5de0 0x55e426f3c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.381+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/3102182171 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55e426f3e2a0 0x55e426f406c0 crc :-1 s=READY pgs=490 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.381+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/3102182171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e426eb1680 0x55e426ec8f40 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.381+0000 7fd0fd9c8b80 1 --2- 172.21.15.70:0/3102182171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e426eb2040 0x55e426ec9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.381+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/3102182171 >> 172.21.15.70:0/3102182171 conn(0x55e426ddf080 msgr2=0x55e426eb95d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:33.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.381+0000 7fd0cbfff700 0 client.30062 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:33.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.381+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/3102182171 shutdown_connections 2024-06-04T05:20:33.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.381+0000 7fd0fd9c8b80 1 -- 172.21.15.70:0/3102182171 wait complete. 2024-06-04T05:20:33.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.381+0000 7fd0fd9c8b80 20 client.30062 trim_cache size 0 max 16384 2024-06-04T05:20:33.388 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:20:33.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.382+0000 7fd0f1ed1700 1 -- 172.21.15.70:0/222745653 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd0dc0b2a70 msgr2=0x7fd0dc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.382+0000 7fd0f1ed1700 1 --2- 172.21.15.70:0/222745653 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd0dc0b2a70 0x7fd0dc0b4f30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0009ca0 tx=0x7fd0e0009260 comp rx=0 tx=0).stop 2024-06-04T05:20:33.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.382+0000 7fd0f1ed1700 1 -- 172.21.15.70:0/222745653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0e409aca0 msgr2=0x7fd0e412e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:33.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.382+0000 7fd0f1ed1700 1 --2- 172.21.15.70:0/222745653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0e409aca0 0x7fd0e412e800 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec069fd0 tx=0x7fd0ec075c80 comp rx=0 tx=0).stop 2024-06-04T05:20:33.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.382+0000 7fd0f1ed1700 1 -- 172.21.15.70:0/222745653 shutdown_connections 2024-06-04T05:20:33.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.382+0000 7fd0f1ed1700 1 --2- 172.21.15.70:0/222745653 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fd0dc0b2a70 0x7fd0dc0b4f30 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.382+0000 7fd0f1ed1700 1 --2- 172.21.15.70:0/222745653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0e409aca0 0x7fd0e412e800 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.382+0000 7fd0f1ed1700 1 --2- 172.21.15.70:0/222745653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0e4094ee0 0x7fd0e4129f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.382+0000 7fd0f1ed1700 1 --2- 172.21.15.70:0/222745653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0e40958a0 0x7fd0e412a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:33.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.382+0000 7fd0f1ed1700 1 -- 172.21.15.70:0/222745653 >> 172.21.15.70:0/222745653 conn(0x7fd0e400a7a0 msgr2=0x7fd0e4093890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:33.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.382+0000 7fd0f1ed1700 1 -- 172.21.15.70:0/222745653 shutdown_connections 2024-06-04T05:20:33.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:33.382+0000 7fd0f1ed1700 1 -- 172.21.15.70:0/222745653 wait complete. 2024-06-04T05:20:33.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:33 smithi070 ceph-mon[32631]: pgmap v736: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.5 MiB/s wr, 421 op/s 2024-06-04T05:20:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:33 smithi112 ceph-mon[36453]: pgmap v736: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.5 MiB/s wr, 421 op/s 2024-06-04T05:20:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:33 smithi098 ceph-mon[35569]: pgmap v736: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.5 MiB/s wr, 421 op/s 2024-06-04T05:20:33.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 72341 6.76 MB/sec execute 445 sec latency 117.962 ms 2024-06-04T05:20:34.830 INFO:tasks.workunit.client.0.smithi070.stdout: 10 72501 6.75 MB/sec execute 446 sec latency 71.003 ms 2024-06-04T05:20:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:35 smithi112 ceph-mon[36453]: pgmap v737: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.7 MiB/s wr, 253 op/s 2024-06-04T05:20:35.725 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:20:35.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:20:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:35 smithi098 ceph-mon[35569]: pgmap v737: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.7 MiB/s wr, 253 op/s 2024-06-04T05:20:35.757 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:35 smithi070 ceph-mon[32631]: pgmap v737: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.7 MiB/s wr, 253 op/s 2024-06-04T05:20:35.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 72673 6.76 MB/sec execute 447 sec latency 55.555 ms 2024-06-04T05:20:36.087 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:20:36.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 72880 6.77 MB/sec execute 448 sec latency 68.866 ms 2024-06-04T05:20:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:37 smithi112 ceph-mon[36453]: pgmap v738: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 403 op/s 2024-06-04T05:20:37.717 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:37 smithi070 ceph-mon[32631]: pgmap v738: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 403 op/s 2024-06-04T05:20:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:37 smithi098 ceph-mon[35569]: pgmap v738: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 403 op/s 2024-06-04T05:20:37.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 73028 6.78 MB/sec execute 449 sec latency 116.125 ms 2024-06-04T05:20:38.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 73162 6.78 MB/sec execute 450 sec latency 141.877 ms 2024-06-04T05:20:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:39 smithi112 ceph-mon[36453]: pgmap v739: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 275 op/s 2024-06-04T05:20:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:39 smithi098 ceph-mon[35569]: pgmap v739: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 275 op/s 2024-06-04T05:20:39.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 73308 6.78 MB/sec execute 451 sec latency 511.943 ms 2024-06-04T05:20:39.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:39 smithi070 ceph-mon[32631]: pgmap v739: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 275 op/s 2024-06-04T05:20:40.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 73486 6.80 MB/sec execute 452 sec latency 167.520 ms 2024-06-04T05:20:41.660 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:41 smithi070 ceph-mon[32631]: pgmap v740: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 275 op/s 2024-06-04T05:20:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:41 smithi112 ceph-mon[36453]: pgmap v740: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 275 op/s 2024-06-04T05:20:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:41 smithi098 ceph-mon[35569]: pgmap v740: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 275 op/s 2024-06-04T05:20:41.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 73641 6.81 MB/sec execute 453 sec latency 185.788 ms 2024-06-04T05:20:42.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 73765 6.81 MB/sec execute 454 sec latency 139.446 ms 2024-06-04T05:20:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:43 smithi112 ceph-mon[36453]: pgmap v741: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.3 MiB/s wr, 390 op/s 2024-06-04T05:20:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:43 smithi098 ceph-mon[35569]: pgmap v741: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.3 MiB/s wr, 390 op/s 2024-06-04T05:20:43.760 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:43 smithi070 ceph-mon[32631]: pgmap v741: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.3 MiB/s wr, 390 op/s 2024-06-04T05:20:43.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 73886 6.81 MB/sec execute 455 sec latency 296.174 ms 2024-06-04T05:20:44.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.220+0000 7f02fd4e9700 1 -- 172.21.15.70:0/1478449971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02f8102a80 msgr2=0x7f02f8102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:44.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.220+0000 7f02fd4e9700 1 --2- 172.21.15.70:0/1478449971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02f8102a80 0x7f02f8102e60 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f02e80099c0 tx=0x7f02e8009cd0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.220+0000 7f02fd4e9700 1 -- 172.21.15.70:0/1478449971 shutdown_connections 2024-06-04T05:20:44.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.220+0000 7f02fd4e9700 1 --2- 172.21.15.70:0/1478449971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02f81077d0 0x7f02f810f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.220+0000 7f02fd4e9700 1 --2- 172.21.15.70:0/1478449971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02f8102a80 0x7f02f8102e60 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.220+0000 7f02fd4e9700 1 --2- 172.21.15.70:0/1478449971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02f8103430 0x7f02f81070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.220+0000 7f02fd4e9700 1 -- 172.21.15.70:0/1478449971 >> 172.21.15.70:0/1478449971 conn(0x7f02f80793f0 msgr2=0x7f02f8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:44.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.220+0000 7f02fd4e9700 1 -- 172.21.15.70:0/1478449971 shutdown_connections 2024-06-04T05:20:44.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.220+0000 7f02fd4e9700 1 -- 172.21.15.70:0/1478449971 wait complete. 2024-06-04T05:20:44.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.221+0000 7f02fd4e9700 1 Processor -- start 2024-06-04T05:20:44.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.221+0000 7f02fd4e9700 1 -- start start 2024-06-04T05:20:44.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.221+0000 7f02fd4e9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02f8102a80 0x7f02f819d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.221+0000 7f02fd4e9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02f8103430 0x7f02f819db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.221+0000 7f02fd4e9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02f81077d0 0x7f02f8197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.221+0000 7f02fd4e9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81114a0 con 0x7f02f8102a80 2024-06-04T05:20:44.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.221+0000 7f02fd4e9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02f8111320 con 0x7f02f81077d0 2024-06-04T05:20:44.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.221+0000 7f02fd4e9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f02f8111620 con 0x7f02f8103430 2024-06-04T05:20:44.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.222+0000 7f02f6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02f8102a80 0x7f02f819d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.222+0000 7f02f67fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02f8103430 0x7f02f819db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.222+0000 7f02f6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02f8102a80 0x7f02f819d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35656/0 (socket says 172.21.15.70:35656) 2024-06-04T05:20:44.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.222+0000 7f02f6ffd700 1 -- 172.21.15.70:0/1366309042 learned_addr learned my addr 172.21.15.70:0/1366309042 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:44.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.222+0000 7f02f77fe700 1 --2- 172.21.15.70:0/1366309042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02f81077d0 0x7f02f8197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.222+0000 7f02f77fe700 1 -- 172.21.15.70:0/1366309042 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02f8103430 msgr2=0x7f02f819db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:44.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.222+0000 7f02f77fe700 1 --2- 172.21.15.70:0/1366309042 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02f8103430 0x7f02f819db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.222+0000 7f02f77fe700 1 -- 172.21.15.70:0/1366309042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02f8102a80 msgr2=0x7f02f819d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:44.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.222+0000 7f02f77fe700 1 --2- 172.21.15.70:0/1366309042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02f8102a80 0x7f02f819d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.222+0000 7f02f77fe700 1 -- 172.21.15.70:0/1366309042 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f8198030 con 0x7f02f81077d0 2024-06-04T05:20:44.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.222+0000 7f02f77fe700 1 --2- 172.21.15.70:0/1366309042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02f81077d0 0x7f02f8197850 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f02f0007fd0 tx=0x7f02f000f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:44.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.222+0000 7f02dffff700 1 -- 172.21.15.70:0/1366309042 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f0019020 con 0x7f02f81077d0 2024-06-04T05:20:44.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.222+0000 7f02fd4e9700 1 -- 172.21.15.70:0/1366309042 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f8198320 con 0x7f02f81077d0 2024-06-04T05:20:44.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.222+0000 7f02dffff700 1 -- 172.21.15.70:0/1366309042 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02f0004db0 con 0x7f02f81077d0 2024-06-04T05:20:44.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.222+0000 7f02fd4e9700 1 -- 172.21.15.70:0/1366309042 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f81a4490 con 0x7f02f81077d0 2024-06-04T05:20:44.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.222+0000 7f02dffff700 1 -- 172.21.15.70:0/1366309042 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f0009970 con 0x7f02f81077d0 2024-06-04T05:20:44.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.225+0000 7f02dffff700 1 -- 172.21.15.70:0/1366309042 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f02f0009b10 con 0x7f02f81077d0 2024-06-04T05:20:44.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.227+0000 7f02dffff700 1 --2- 172.21.15.70:0/1366309042 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02e00b2a70 0x7f02e00b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.227+0000 7f02dffff700 1 -- 172.21.15.70:0/1366309042 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f02f00e2600 con 0x7f02f81077d0 2024-06-04T05:20:44.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.227+0000 7f02f6ffd700 1 --2- 172.21.15.70:0/1366309042 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02e00b2a70 0x7f02e00b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.227+0000 7f0307d7eb80 1 Processor -- start 2024-06-04T05:20:44.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.228+0000 7f0307d7eb80 1 -- start start 2024-06-04T05:20:44.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.228+0000 7f0307d7eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018bacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.228+0000 7f02f6ffd700 1 --2- 172.21.15.70:0/1366309042 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02e00b2a70 0x7f02e00b4f30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f02e8003760 tx=0x7f02e8005fd0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:44.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.228+0000 7f0307d7eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018bb290 0x557a018be9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.228+0000 7f02f6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018bacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.228+0000 7f02f6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018bacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35662/0 (socket says 172.21.15.70:35662) 2024-06-04T05:20:44.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.228+0000 7f02f67fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018bb290 0x557a018be9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.228+0000 7f02f6ffd700 1 -- 172.21.15.70:0/816194156 learned_addr learned my addr 172.21.15.70:0/816194156 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:44.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.228+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/816194156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018befe0 0x557a018bf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.228+0000 7f0307d7eb80 1 -- 172.21.15.70:0/816194156 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557a01819d10 con 0x557a018ba8e0 2024-06-04T05:20:44.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.228+0000 7f0307d7eb80 1 -- 172.21.15.70:0/816194156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557a018098b0 con 0x557a018befe0 2024-06-04T05:20:44.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.228+0000 7f0307d7eb80 1 -- 172.21.15.70:0/816194156 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557a0180d8f0 con 0x557a018bb290 2024-06-04T05:20:44.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.228+0000 7f02f77fe700 1 --2- 172.21.15.70:0/816194156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018befe0 0x557a018bf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.229+0000 7f02f77fe700 1 -- 172.21.15.70:0/816194156 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018bb290 msgr2=0x557a018be9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:44.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.229+0000 7f02f77fe700 1 --2- 172.21.15.70:0/816194156 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018bb290 0x557a018be9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.229+0000 7f02f77fe700 1 -- 172.21.15.70:0/816194156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 msgr2=0x557a018bacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:44.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.229+0000 7f02f77fe700 1 --2- 172.21.15.70:0/816194156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018bacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.229+0000 7f02f77fe700 1 -- 172.21.15.70:0/816194156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a01759950 con 0x557a018befe0 2024-06-04T05:20:44.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.229+0000 7f02f67fc700 1 --2- 172.21.15.70:0/816194156 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018bb290 0x557a018be9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:20:44.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.229+0000 7f02f6ffd700 1 --2- 172.21.15.70:0/816194156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018bacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:44.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.229+0000 7f02f77fe700 1 --2- 172.21.15.70:0/816194156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018befe0 0x557a018bf420 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f02f00dfd20 tx=0x7f02f00a91a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d34fd6aef0a59 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:44.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.229+0000 7f02dd7fa700 1 -- 172.21.15.70:0/816194156 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f008fe60 con 0x557a018befe0 2024-06-04T05:20:44.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.229+0000 7f02dd7fa700 1 -- 172.21.15.70:0/816194156 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02f00aa040 con 0x557a018befe0 2024-06-04T05:20:44.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.229+0000 7f02dd7fa700 1 -- 172.21.15.70:0/816194156 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f00dffc0 con 0x557a018befe0 2024-06-04T05:20:44.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.230+0000 7f0307d7eb80 1 -- 172.21.15.70:0/816194156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018befe0 msgr2=0x557a018bf420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:44.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.230+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/816194156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018befe0 0x557a018bf420 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f02f00dfd20 tx=0x7f02f00a91a0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.230+0000 7f0307d7eb80 1 -- 172.21.15.70:0/816194156 shutdown_connections 2024-06-04T05:20:44.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.230+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/816194156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018befe0 0x557a018bf420 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.230+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/816194156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018bacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.230+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/816194156 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018bb290 0x557a018be9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.230+0000 7f0307d7eb80 1 -- 172.21.15.70:0/816194156 >> 172.21.15.70:0/816194156 conn(0x557a017e8100 msgr2=0x557a018b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:44.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.230+0000 7f0307d7eb80 1 -- 172.21.15.70:0/816194156 shutdown_connections 2024-06-04T05:20:44.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.230+0000 7f0307d7eb80 1 -- 172.21.15.70:0/816194156 wait complete. 2024-06-04T05:20:44.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.231+0000 7f0307d7eb80 1 -- start start 2024-06-04T05:20:44.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.231+0000 7f0307d7eb80 1 Processor -- start 2024-06-04T05:20:44.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.231+0000 7f0307d7eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018ba8e0 0x557a018c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.231+0000 7f0307d7eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018bb290 0x557a018c3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.231+0000 7f0307d7eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018befe0 0x557a018c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.231+0000 7f0307d7eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557a01710020 con 0x557a018befe0 2024-06-04T05:20:44.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.231+0000 7f0307d7eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557a017ab840 con 0x557a018ba8e0 2024-06-04T05:20:44.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.231+0000 7f0307d7eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557a018c7240 con 0x557a018bb290 2024-06-04T05:20:44.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.231+0000 7f02f6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018ba8e0 0x557a018c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.231+0000 7f02f6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018ba8e0 0x557a018c63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:49026/0 (socket says 172.21.15.70:49026) 2024-06-04T05:20:44.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.231+0000 7f02f6ffd700 1 -- 172.21.15.70:0/1517997286 learned_addr learned my addr 172.21.15.70:0/1517997286 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:44.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.231+0000 7f02f77fe700 1 --2- 172.21.15.70:0/1517997286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018befe0 0x557a018c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.231+0000 7f0307d7eb80 10 client.0 _ll_register_callbacks cb 0x557a017a49a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:44.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.232+0000 7f02f67fc700 1 --2- 172.21.15.70:0/1517997286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018bb290 0x557a018c3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.232+0000 7f02f6ffd700 1 -- 172.21.15.70:0/1517997286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018bb290 msgr2=0x557a018c3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:44.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.232+0000 7f02f6ffd700 1 --2- 172.21.15.70:0/1517997286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018bb290 0x557a018c3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.232+0000 7f02f6ffd700 1 -- 172.21.15.70:0/1517997286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018befe0 msgr2=0x557a018c3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:44.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.232+0000 7f02f6ffd700 1 --2- 172.21.15.70:0/1517997286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018befe0 0x557a018c3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.232+0000 7f02f6ffd700 1 -- 172.21.15.70:0/1517997286 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a017a8710 con 0x557a018ba8e0 2024-06-04T05:20:44.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.232+0000 7f02f6ffd700 1 --2- 172.21.15.70:0/1517997286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018ba8e0 0x557a018c63c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f02e8005f50 tx=0x7f02e8009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad5e5de2e7550ba4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:44.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.232+0000 7f02daff5700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:20:44.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.232+0000 7f02daff5700 1 -- 172.21.15.70:0/1517997286 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e800be30 con 0x557a018ba8e0 2024-06-04T05:20:44.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.232+0000 7f0307d7eb80 1 -- 172.21.15.70:0/1517997286 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02f00aa040 con 0x557a018ba8e0 2024-06-04T05:20:44.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.232+0000 7f02daff5700 1 -- 172.21.15.70:0/1517997286 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f02e8023460 con 0x557a018ba8e0 2024-06-04T05:20:44.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.232+0000 7f02daff5700 1 -- 172.21.15.70:0/1517997286 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f02e803d050 con 0x557a018ba8e0 2024-06-04T05:20:44.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.232+0000 7f0307d7eb80 10 client.30074 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:44.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.232+0000 7f0307d7eb80 1 -- 172.21.15.70:0/1517997286 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x557a018d1930 con 0x557a018ba8e0 2024-06-04T05:20:44.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.233+0000 7f02daff5700 1 -- 172.21.15.70:0/1517997286 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02f00aa040 con 0x557a018ba8e0 2024-06-04T05:20:44.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.233+0000 7f02daff5700 1 -- 172.21.15.70:0/1517997286 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f02e8010600 con 0x557a018ba8e0 2024-06-04T05:20:44.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.233+0000 7f0307d7eb80 10 client.30074 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:44.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.233+0000 7f0307d7eb80 10 client.30074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:44.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.233+0000 7f0307d7eb80 20 client.30074 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:44.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.233+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/1517997286 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557a018d3f60 0x557a018d63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.233+0000 7f0307d7eb80 4 client.30074 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:20:44.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.233+0000 7f0307d7eb80 1 -- 172.21.15.70:0/1517997286 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a01886a60 con 0x557a018d3f60 2024-06-04T05:20:44.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.233+0000 7f02f67fc700 1 --2- 172.21.15.70:0/1517997286 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557a018d3f60 0x557a018d63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.234+0000 7f02f67fc700 1 --2- 172.21.15.70:0/1517997286 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557a018d3f60 0x557a018d63a0 crc :-1 s=READY pgs=491 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cda4ee00d780da5 server_cookie=b805645773ea91be in_seq=0 out_seq=0 2024-06-04T05:20:44.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.234+0000 7f02daff5700 10 client.30074 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:44.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.234+0000 7f02daff5700 1 -- 172.21.15.70:0/1517997286 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557a01886a60 con 0x557a018d3f60 2024-06-04T05:20:44.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.234+0000 7f02daff5700 10 client.30074 handle_command_reply: tid=0 2024-06-04T05:20:44.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.234+0000 7f0307d7eb80 1 client.30074 shutdown 2024-06-04T05:20:44.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.235+0000 7f0307d7eb80 1 -- 172.21.15.70:0/1517997286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018ba8e0 msgr2=0x557a018c63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:44.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.235+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/1517997286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018ba8e0 0x557a018c63c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f02e8005f50 tx=0x7f02e8009cd0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.235+0000 7f0307d7eb80 1 -- 172.21.15.70:0/1517997286 shutdown_connections 2024-06-04T05:20:44.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.235+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/1517997286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018befe0 0x557a018c3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.235+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/1517997286 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557a018d3f60 0x557a018d63a0 crc :-1 s=READY pgs=491 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.235+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/1517997286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018ba8e0 0x557a018c63c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.235+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/1517997286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018bb290 0x557a018c3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.235+0000 7f02daff5700 0 client.30074 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:44.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.235+0000 7f0307d7eb80 1 -- 172.21.15.70:0/1517997286 >> 172.21.15.70:0/1517997286 conn(0x557a017e8100 msgr2=0x557a018c03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:44.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.235+0000 7f0307d7eb80 1 -- 172.21.15.70:0/1517997286 shutdown_connections 2024-06-04T05:20:44.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.235+0000 7f0307d7eb80 1 -- 172.21.15.70:0/1517997286 wait complete. 2024-06-04T05:20:44.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.235+0000 7f0307d7eb80 20 client.30074 trim_cache size 0 max 16384 2024-06-04T05:20:44.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.260+0000 7f0307d7eb80 1 Processor -- start 2024-06-04T05:20:44.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.260+0000 7f0307d7eb80 1 -- start start 2024-06-04T05:20:44.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.260+0000 7f0307d7eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018b9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.260+0000 7f0307d7eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018bb290 0x557a018dbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.260+0000 7f0307d7eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018befe0 0x557a018dc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.260+0000 7f0307d7eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81abc80 con 0x557a018ba8e0 2024-06-04T05:20:44.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.260+0000 7f0307d7eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81ad850 con 0x557a018bb290 2024-06-04T05:20:44.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.260+0000 7f0307d7eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81ac3b0 con 0x557a018befe0 2024-06-04T05:20:44.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.260+0000 7f02f6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018b9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.260+0000 7f02f6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018b9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35688/0 (socket says 172.21.15.70:35688) 2024-06-04T05:20:44.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.260+0000 7f02f6ffd700 1 -- 172.21.15.70:0/3963361613 learned_addr learned my addr 172.21.15.70:0/3963361613 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:44.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.261+0000 7f02f67fc700 1 --2- 172.21.15.70:0/3963361613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018bb290 0x557a018dbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.261+0000 7f02f6ffd700 1 -- 172.21.15.70:0/3963361613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018befe0 msgr2=0x557a018dc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:44.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.261+0000 7f02f77fe700 1 --2- 172.21.15.70:0/3963361613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018befe0 0x557a018dc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.261+0000 7f02f6ffd700 1 --2- 172.21.15.70:0/3963361613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018befe0 0x557a018dc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.261+0000 7f02f6ffd700 1 -- 172.21.15.70:0/3963361613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018bb290 msgr2=0x557a018dbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:44.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.261+0000 7f02f6ffd700 1 --2- 172.21.15.70:0/3963361613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018bb290 0x557a018dbc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.261+0000 7f02f6ffd700 1 -- 172.21.15.70:0/3963361613 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a018c7080 con 0x557a018ba8e0 2024-06-04T05:20:44.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.261+0000 7f02f67fc700 1 --2- 172.21.15.70:0/3963361613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018bb290 0x557a018dbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:20:44.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.261+0000 7f02f6ffd700 1 --2- 172.21.15.70:0/3963361613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018b9e00 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009990 tx=0x7f02e8003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0fffbba280a1f3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:44.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.261+0000 7f02dcff9700 1 -- 172.21.15.70:0/3963361613 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8046070 con 0x557a018ba8e0 2024-06-04T05:20:44.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.261+0000 7f02dcff9700 1 -- 172.21.15.70:0/3963361613 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02e8034070 con 0x557a018ba8e0 2024-06-04T05:20:44.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.261+0000 7f02dcff9700 1 -- 172.21.15.70:0/3963361613 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8023990 con 0x557a018ba8e0 2024-06-04T05:20:44.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.262+0000 7f0307d7eb80 1 -- 172.21.15.70:0/3963361613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 msgr2=0x557a018b9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:44.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.262+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/3963361613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018b9e00 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009990 tx=0x7f02e8003f90 comp rx=0 tx=0).stop 2024-06-04T05:20:44.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.262+0000 7f0307d7eb80 1 -- 172.21.15.70:0/3963361613 shutdown_connections 2024-06-04T05:20:44.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.262+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/3963361613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018befe0 0x557a018dc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.262+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/3963361613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018b9e00 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.262+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/3963361613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018bb290 0x557a018dbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.262+0000 7f0307d7eb80 1 -- 172.21.15.70:0/3963361613 >> 172.21.15.70:0/3963361613 conn(0x557a017e8100 msgr2=0x557a018b96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:44.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.262+0000 7f0307d7eb80 1 -- 172.21.15.70:0/3963361613 shutdown_connections 2024-06-04T05:20:44.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.262+0000 7f0307d7eb80 1 -- 172.21.15.70:0/3963361613 wait complete. 2024-06-04T05:20:44.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.262+0000 7f0307d7eb80 1 -- start start 2024-06-04T05:20:44.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.262+0000 7f0307d7eb80 1 Processor -- start 2024-06-04T05:20:44.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.262+0000 7f0307d7eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018cc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.262+0000 7f0307d7eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018bb290 0x557a018cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.262+0000 7f0307d7eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018befe0 0x557a018cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.262+0000 7f0307d7eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81ad150 con 0x557a018ba8e0 2024-06-04T05:20:44.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.262+0000 7f0307d7eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81aca80 con 0x557a018befe0 2024-06-04T05:20:44.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.262+0000 7f0307d7eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81a9c40 con 0x557a018bb290 2024-06-04T05:20:44.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f0307d7eb80 10 client.0 _ll_register_callbacks cb 0x557a017a49a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:44.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f02f6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f02f6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018cc780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35696/0 (socket says 172.21.15.70:35696) 2024-06-04T05:20:44.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f02f6ffd700 1 -- 172.21.15.70:0/594969664 learned_addr learned my addr 172.21.15.70:0/594969664 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:44.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f02f77fe700 1 --2- 172.21.15.70:0/594969664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018befe0 0x557a018cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f02f67fc700 1 --2- 172.21.15.70:0/594969664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018bb290 0x557a018cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f02f6ffd700 1 -- 172.21.15.70:0/594969664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018bb290 msgr2=0x557a018cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:44.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f02f6ffd700 1 --2- 172.21.15.70:0/594969664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018bb290 0x557a018cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f02f6ffd700 1 -- 172.21.15.70:0/594969664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018befe0 msgr2=0x557a018cacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:44.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f02f6ffd700 1 --2- 172.21.15.70:0/594969664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018befe0 0x557a018cacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f02f6ffd700 1 -- 172.21.15.70:0/594969664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a018cb500 con 0x557a018ba8e0 2024-06-04T05:20:44.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f02f6ffd700 1 --2- 172.21.15.70:0/594969664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018cc780 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f02e8044040 tx=0x7f02e800bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12483f4041409e8d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:44.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f02daff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:20:44.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f02daff5700 1 -- 172.21.15.70:0/594969664 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8046070 con 0x557a018ba8e0 2024-06-04T05:20:44.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f0307d7eb80 1 -- 172.21.15.70:0/594969664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02e8034070 con 0x557a018ba8e0 2024-06-04T05:20:44.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f02daff5700 1 -- 172.21.15.70:0/594969664 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f02e8023460 con 0x557a018ba8e0 2024-06-04T05:20:44.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f02daff5700 1 -- 172.21.15.70:0/594969664 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f02e8023660 con 0x557a018ba8e0 2024-06-04T05:20:44.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f0307d7eb80 10 client.21429 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:44.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.263+0000 7f0307d7eb80 1 -- 172.21.15.70:0/594969664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x557a0193f820 con 0x557a018ba8e0 2024-06-04T05:20:44.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.264+0000 7f02daff5700 1 -- 172.21.15.70:0/594969664 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02e8034070 con 0x557a018ba8e0 2024-06-04T05:20:44.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.264+0000 7f02daff5700 1 -- 172.21.15.70:0/594969664 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f02e8039710 con 0x557a018ba8e0 2024-06-04T05:20:44.785 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:20:44.785 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:20:44.785 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "3bfba7ca-ffef-4b68-9d78-40be723b6e3c", 2024-06-04T05:20:44.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:20:44.785 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:20:44.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.264+0000 7f0307d7eb80 10 client.21429 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:44.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.264+0000 7f0307d7eb80 10 client.21429 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:44.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.264+0000 7f0307d7eb80 20 client.21429 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:44.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.264+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/594969664 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557a0193fe60 0x557a01942280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:44.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.264+0000 7f0307d7eb80 4 client.21429 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:20:44.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.264+0000 7f0307d7eb80 1 -- 172.21.15.70:0/594969664 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557a0181bc60 con 0x557a0193fe60 2024-06-04T05:20:44.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.264+0000 7f02f67fc700 1 --2- 172.21.15.70:0/594969664 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557a0193fe60 0x557a01942280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:44.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.264+0000 7f02daff5700 10 client.21429 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:44.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.264+0000 7f02f67fc700 1 --2- 172.21.15.70:0/594969664 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557a0193fe60 0x557a01942280 crc :-1 s=READY pgs=492 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bab02aef1ce7c46b server_cookie=f4742702998d5b1b in_seq=0 out_seq=0 2024-06-04T05:20:44.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.265+0000 7f02daff5700 1 -- 172.21.15.70:0/594969664 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557a0181bc60 con 0x557a0193fe60 2024-06-04T05:20:44.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.265+0000 7f02daff5700 10 client.21429 handle_command_reply: tid=0 2024-06-04T05:20:44.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.265+0000 7f0307d7eb80 1 client.21429 shutdown 2024-06-04T05:20:44.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.265+0000 7f0307d7eb80 1 -- 172.21.15.70:0/594969664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 msgr2=0x557a018cc780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:44.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.265+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/594969664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018cc780 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f02e8044040 tx=0x7f02e800bb00 comp rx=0 tx=0).stop 2024-06-04T05:20:44.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.265+0000 7f0307d7eb80 1 -- 172.21.15.70:0/594969664 shutdown_connections 2024-06-04T05:20:44.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.265+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/594969664 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x557a0193fe60 0x557a01942280 crc :-1 s=READY pgs=492 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.265+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/594969664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a018befe0 0x557a018cacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.265+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/594969664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557a018ba8e0 0x557a018cc780 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.265+0000 7f0307d7eb80 1 --2- 172.21.15.70:0/594969664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557a018bb290 0x557a018cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.265+0000 7f0307d7eb80 1 -- 172.21.15.70:0/594969664 >> 172.21.15.70:0/594969664 conn(0x557a017e8100 msgr2=0x557a018b9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:44.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.266+0000 7f02daff5700 0 client.21429 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:44.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.266+0000 7f0307d7eb80 1 -- 172.21.15.70:0/594969664 shutdown_connections 2024-06-04T05:20:44.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.266+0000 7f0307d7eb80 1 -- 172.21.15.70:0/594969664 wait complete. 2024-06-04T05:20:44.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.266+0000 7f0307d7eb80 20 client.21429 trim_cache size 0 max 16384 2024-06-04T05:20:44.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.266+0000 7f02fd4e9700 1 -- 172.21.15.70:0/1366309042 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02e00b2a70 msgr2=0x7f02e00b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:44.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.266+0000 7f02fd4e9700 1 --2- 172.21.15.70:0/1366309042 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02e00b2a70 0x7f02e00b4f30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f02e8003760 tx=0x7f02e8005fd0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.266+0000 7f02fd4e9700 1 -- 172.21.15.70:0/1366309042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02f81077d0 msgr2=0x7f02f8197850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:44.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.266+0000 7f02fd4e9700 1 --2- 172.21.15.70:0/1366309042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02f81077d0 0x7f02f8197850 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f02f0007fd0 tx=0x7f02f000f690 comp rx=0 tx=0).stop 2024-06-04T05:20:44.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.266+0000 7f02fd4e9700 1 -- 172.21.15.70:0/1366309042 shutdown_connections 2024-06-04T05:20:44.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.266+0000 7f02fd4e9700 1 --2- 172.21.15.70:0/1366309042 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f02e00b2a70 0x7f02e00b4f30 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.266+0000 7f02fd4e9700 1 --2- 172.21.15.70:0/1366309042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02f81077d0 0x7f02f8197850 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.266+0000 7f02fd4e9700 1 --2- 172.21.15.70:0/1366309042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02f8102a80 0x7f02f819d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.266+0000 7f02fd4e9700 1 --2- 172.21.15.70:0/1366309042 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02f8103430 0x7f02f819db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:44.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.266+0000 7f02fd4e9700 1 -- 172.21.15.70:0/1366309042 >> 172.21.15.70:0/1366309042 conn(0x7f02f80793f0 msgr2=0x7f02f8100cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:44.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.267+0000 7f02fd4e9700 1 -- 172.21.15.70:0/1366309042 shutdown_connections 2024-06-04T05:20:44.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:44.267+0000 7f02fd4e9700 1 -- 172.21.15.70:0/1366309042 wait complete. 2024-06-04T05:20:44.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 73998 6.80 MB/sec execute 456 sec latency 198.305 ms 2024-06-04T05:20:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:45 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:20:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:45 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:45 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:20:45.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:45 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:20:45.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:45 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:20:45.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:45 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:20:45.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:45 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:20:45.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:45 smithi112 ceph-mon[36453]: pgmap v742: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 265 op/s 2024-06-04T05:20:45.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:45 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:45 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:45 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:20:45.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:45 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:45 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:45 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:45 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:45 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:20:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:45 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:45 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:20:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:45 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:20:45.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:45 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:20:45.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:45 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:20:45.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:45 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:20:45.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:45 smithi098 ceph-mon[35569]: pgmap v742: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 265 op/s 2024-06-04T05:20:45.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:45 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:45 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:45 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:20:45.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:45 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:45 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:45 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:45 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 74140 6.80 MB/sec execute 457 sec latency 156.684 ms 2024-06-04T05:20:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:45 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:20:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:45 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:45 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:20:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:45 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:20:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:45 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:20:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:45 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:20:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:45 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:20:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:45 smithi070 ceph-mon[32631]: pgmap v742: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 265 op/s 2024-06-04T05:20:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:45 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:45 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:45 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:20:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:45 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:45 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:45 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:45.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:45 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:20:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.364+0000 7f7659d67640 1 -- 172.21.15.70:0/552473846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7654104600 msgr2=0x7f765410cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.364+0000 7f7659d67640 1 --2- 172.21.15.70:0/552473846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7654104600 0x7f765410cae0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f7648009900 tx=0x7f764802a480 comp rx=0 tx=0).stop 2024-06-04T05:20:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.364+0000 7f7659d67640 1 -- 172.21.15.70:0/552473846 shutdown_connections 2024-06-04T05:20:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.364+0000 7f7659d67640 1 --2- 172.21.15.70:0/552473846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f765410d0b0 0x7f765410f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.364+0000 7f7659d67640 1 --2- 172.21.15.70:0/552473846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7654104600 0x7f765410cae0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.364+0000 7f7659d67640 1 --2- 172.21.15.70:0/552473846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7654103ce0 0x7f76541040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.364+0000 7f7659d67640 1 -- 172.21.15.70:0/552473846 >> 172.21.15.70:0/552473846 conn(0x7f7654078a60 msgr2=0x7f7654078e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:46.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.364+0000 7f7659d67640 1 -- 172.21.15.70:0/552473846 shutdown_connections 2024-06-04T05:20:46.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.365+0000 7f7659d67640 1 -- 172.21.15.70:0/552473846 wait complete. 2024-06-04T05:20:46.369 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.365+0000 7f7659d67640 1 Processor -- start 2024-06-04T05:20:46.369 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.365+0000 7f7659d67640 1 -- start start 2024-06-04T05:20:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.366+0000 7f7659d67640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7654103ce0 0x7f765419b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.366+0000 7f7659d67640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f765410d0b0 0x7f765419ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.366+0000 7f7659d67640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f765419fe00 0x7f76541a02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.366+0000 7f7659d67640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7654114df0 con 0x7f765410d0b0 2024-06-04T05:20:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.366+0000 7f7659d67640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7654114c70 con 0x7f7654103ce0 2024-06-04T05:20:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.366+0000 7f7659d67640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7654114f70 con 0x7f765419fe00 2024-06-04T05:20:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.366+0000 7f7652ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f765410d0b0 0x7f765419ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.366+0000 7f7652ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f765410d0b0 0x7f765419ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52066/0 (socket says 172.21.15.70:52066) 2024-06-04T05:20:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.366+0000 7f7653fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f765419fe00 0x7f76541a02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.366+0000 7f7652ffd640 1 -- 172.21.15.70:0/1932624750 learned_addr learned my addr 172.21.15.70:0/1932624750 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.366+0000 7f76537fe640 1 --2- 172.21.15.70:0/1932624750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7654103ce0 0x7f765419b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.366+0000 7f7653fff640 1 -- 172.21.15.70:0/1932624750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7654103ce0 msgr2=0x7f765419b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.366+0000 7f7653fff640 1 --2- 172.21.15.70:0/1932624750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7654103ce0 0x7f765419b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.366+0000 7f7653fff640 1 -- 172.21.15.70:0/1932624750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f765410d0b0 msgr2=0x7f765419ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.366+0000 7f7653fff640 1 --2- 172.21.15.70:0/1932624750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f765410d0b0 0x7f765419ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.366+0000 7f7653fff640 1 -- 172.21.15.70:0/1932624750 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76541a08f0 con 0x7f765419fe00 2024-06-04T05:20:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.366+0000 7f76537fe640 1 --2- 172.21.15.70:0/1932624750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7654103ce0 0x7f765419b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:20:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.367+0000 7f7653fff640 1 --2- 172.21.15.70:0/1932624750 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f765419fe00 0x7f76541a02b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f764400e940 tx=0x7f764400ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.367+0000 7f7650ff9640 1 -- 172.21.15.70:0/1932624750 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764400ccc0 con 0x7f765419fe00 2024-06-04T05:20:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.367+0000 7f7650ff9640 1 -- 172.21.15.70:0/1932624750 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7644005020 con 0x7f765419fe00 2024-06-04T05:20:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.367+0000 7f7650ff9640 1 -- 172.21.15.70:0/1932624750 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764400ce60 con 0x7f765419fe00 2024-06-04T05:20:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.367+0000 7f7659d67640 1 -- 172.21.15.70:0/1932624750 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76541a0be0 con 0x7f765419fe00 2024-06-04T05:20:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.367+0000 7f7659d67640 1 -- 172.21.15.70:0/1932624750 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76541a8350 con 0x7f765419fe00 2024-06-04T05:20:46.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.371+0000 7f765ca7d740 1 Processor -- start 2024-06-04T05:20:46.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.372+0000 7f7650ff9640 1 -- 172.21.15.70:0/1932624750 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7644004a00 con 0x7f765419fe00 2024-06-04T05:20:46.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.372+0000 7f765ca7d740 1 -- start start 2024-06-04T05:20:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.372+0000 7f765ca7d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f462f820 0x55a6f462fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.372+0000 7f765ca7d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f4630140 0x55a6f463a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.372+0000 7f765ca7d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f463aa00 0x55a6f463cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.372+0000 7f7652ffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f462f820 0x55a6f462fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.372+0000 7f765ca7d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a6f45a1c30 con 0x55a6f4630140 2024-06-04T05:20:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.372+0000 7f765ca7d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a6f45917d0 con 0x55a6f463aa00 2024-06-04T05:20:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.372+0000 7f765ca7d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a6f4595810 con 0x55a6f462f820 2024-06-04T05:20:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.372+0000 7f7652ffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f462f820 0x55a6f462fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48096/0 (socket says 172.21.15.70:48096) 2024-06-04T05:20:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.372+0000 7f7652ffd640 1 -- 172.21.15.70:0/4031524907 learned_addr learned my addr 172.21.15.70:0/4031524907 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.372+0000 7f76537fe640 1 --2- 172.21.15.70:0/4031524907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f4630140 0x55a6f463a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.372+0000 7f7652ffd640 1 --2- 172.21.15.70:0/4031524907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f463aa00 0x55a6f463cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.373+0000 7f7652ffd640 1 -- 172.21.15.70:0/4031524907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f463aa00 msgr2=0x55a6f463cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.373+0000 7f7652ffd640 1 --2- 172.21.15.70:0/4031524907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f463aa00 0x55a6f463cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.373+0000 7f7652ffd640 1 -- 172.21.15.70:0/4031524907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f4630140 msgr2=0x55a6f463a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.373+0000 7f7652ffd640 1 --2- 172.21.15.70:0/4031524907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f4630140 0x55a6f463a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.373+0000 7f7652ffd640 1 -- 172.21.15.70:0/4031524907 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6f42a6210 con 0x55a6f462f820 2024-06-04T05:20:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.373+0000 7f76537fe640 1 --2- 172.21.15.70:0/4031524907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f4630140 0x55a6f463a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.373+0000 7f7650ff9640 1 --2- 172.21.15.70:0/1932624750 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f763c0b2a90 0x7f763c0b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.373+0000 7f7652ffd640 1 --2- 172.21.15.70:0/4031524907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f462f820 0x55a6f462fc00 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7648009230 tx=0x7f7648000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab7c49b32ba6a3d5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.373+0000 7f76537fe640 1 --2- 172.21.15.70:0/1932624750 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f763c0b2a90 0x7f763c0b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.373+0000 7f7650ff9640 1 -- 172.21.15.70:0/1932624750 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f76440e25b0 con 0x7f765419fe00 2024-06-04T05:20:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.373+0000 7f7639ffb640 1 -- 172.21.15.70:0/4031524907 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7648035070 con 0x55a6f462f820 2024-06-04T05:20:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.374+0000 7f7639ffb640 1 -- 172.21.15.70:0/4031524907 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7648003a80 con 0x55a6f462f820 2024-06-04T05:20:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.374+0000 7f7639ffb640 1 -- 172.21.15.70:0/4031524907 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764803d760 con 0x55a6f462f820 2024-06-04T05:20:46.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.374+0000 7f76537fe640 1 --2- 172.21.15.70:0/1932624750 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f763c0b2a90 0x7f763c0b4f50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x55a6f4630dc0 tx=0x7f764c009500 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:46.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.374+0000 7f765ca7d740 1 -- 172.21.15.70:0/4031524907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f462f820 msgr2=0x55a6f462fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:46.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.374+0000 7f765ca7d740 1 --2- 172.21.15.70:0/4031524907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f462f820 0x55a6f462fc00 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7648009230 tx=0x7f7648000bc0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.374+0000 7f765ca7d740 1 -- 172.21.15.70:0/4031524907 shutdown_connections 2024-06-04T05:20:46.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.374+0000 7f765ca7d740 1 --2- 172.21.15.70:0/4031524907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f463aa00 0x55a6f463cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.374+0000 7f765ca7d740 1 --2- 172.21.15.70:0/4031524907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f4630140 0x55a6f463a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.374+0000 7f765ca7d740 1 --2- 172.21.15.70:0/4031524907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f462f820 0x55a6f462fc00 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.374+0000 7f765ca7d740 1 -- 172.21.15.70:0/4031524907 >> 172.21.15.70:0/4031524907 conn(0x55a6f462bd30 msgr2=0x55a6f462c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:46.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.374+0000 7f765ca7d740 1 -- 172.21.15.70:0/4031524907 shutdown_connections 2024-06-04T05:20:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.374+0000 7f765ca7d740 1 -- 172.21.15.70:0/4031524907 wait complete. 2024-06-04T05:20:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.375+0000 7f765ca7d740 1 -- start start 2024-06-04T05:20:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.375+0000 7f765ca7d740 1 Processor -- start 2024-06-04T05:20:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.375+0000 7f765ca7d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f462f820 0x55a6f4637a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.375+0000 7f765ca7d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f4630140 0x55a6f4637f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f765ca7d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f463aa00 0x55a6f46384b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f765ca7d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a6f4277790 con 0x55a6f463aa00 2024-06-04T05:20:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f765ca7d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a6f454d890 con 0x55a6f462f820 2024-06-04T05:20:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f765ca7d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a6f463d5d0 con 0x55a6f4630140 2024-06-04T05:20:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f76537fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f462f820 0x55a6f4637a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f76537fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f462f820 0x55a6f4637a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55506/0 (socket says 172.21.15.70:55506) 2024-06-04T05:20:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f76537fe640 1 -- 172.21.15.70:0/2151347225 learned_addr learned my addr 172.21.15.70:0/2151347225 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f7652ffd640 1 --2- 172.21.15.70:0/2151347225 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f4630140 0x55a6f4637f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f7653fff640 1 --2- 172.21.15.70:0/2151347225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f463aa00 0x55a6f46384b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f765ca7d740 10 client.0 _ll_register_callbacks cb 0x55a6f42a5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f76537fe640 1 -- 172.21.15.70:0/2151347225 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f4630140 msgr2=0x55a6f4637f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f76537fe640 1 --2- 172.21.15.70:0/2151347225 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f4630140 0x55a6f4637f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f76537fe640 1 -- 172.21.15.70:0/2151347225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f463aa00 msgr2=0x55a6f46384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f76537fe640 1 --2- 172.21.15.70:0/2151347225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f463aa00 0x55a6f46384b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f76537fe640 1 -- 172.21.15.70:0/2151347225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6f4372620 con 0x55a6f462f820 2024-06-04T05:20:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f7653fff640 1 --2- 172.21.15.70:0/2151347225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f463aa00 0x55a6f46384b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:20:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f765ca7d740 1 -- 172.21.15.70:0/2151347225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7648003a80 con 0x55a6f462f820 2024-06-04T05:20:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f76537fe640 1 --2- 172.21.15.70:0/2151347225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f462f820 0x55a6f4637a30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f764c010bf0 tx=0x7f764c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a8c1fb360bb67dd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.376+0000 7f7632ffd640 10 client.30092 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:20:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.377+0000 7f7632ffd640 1 -- 172.21.15.70:0/2151347225 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764c02c060 con 0x55a6f462f820 2024-06-04T05:20:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.377+0000 7f7632ffd640 1 -- 172.21.15.70:0/2151347225 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f764c017030 con 0x55a6f462f820 2024-06-04T05:20:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.377+0000 7f7632ffd640 1 -- 172.21.15.70:0/2151347225 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f764c01fcc0 con 0x55a6f462f820 2024-06-04T05:20:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.377+0000 7f765ca7d740 10 client.30092 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.377+0000 7f765ca7d740 1 -- 172.21.15.70:0/2151347225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({fsmap=64}) -- 0x55a6f4651b80 con 0x55a6f462f820 2024-06-04T05:20:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.377+0000 7f7632ffd640 1 -- 172.21.15.70:0/2151347225 <== mon.2 v2:172.21.15.98:3300/0 4 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f764c01c3f0 con 0x55a6f462f820 2024-06-04T05:20:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.377+0000 7f765ca7d740 10 client.30092 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.377+0000 7f765ca7d740 10 client.30092 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.377+0000 7f765ca7d740 20 client.30092 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.377+0000 7f765ca7d740 1 --2- 172.21.15.70:0/2151347225 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a6f4652f30 0x55a6f4655320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.378+0000 7f765ca7d740 4 client.30092 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:20:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.378+0000 7f765ca7d740 1 -- 172.21.15.70:0/2151347225 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6f45a3b80 con 0x55a6f4652f30 2024-06-04T05:20:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.378+0000 7f7652ffd640 1 --2- 172.21.15.70:0/2151347225 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a6f4652f30 0x55a6f4655320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.378+0000 7f7632ffd640 10 client.30092 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.378+0000 7f7652ffd640 1 --2- 172.21.15.70:0/2151347225 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a6f4652f30 0x55a6f4655320 crc :-1 s=READY pgs=493 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a305edd3a86cae0c server_cookie=e01c5b32cae0ec1d in_seq=0 out_seq=0 2024-06-04T05:20:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.379+0000 7f7632ffd640 1 -- 172.21.15.70:0/2151347225 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a6f45a3b80 con 0x55a6f4652f30 2024-06-04T05:20:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.379+0000 7f7632ffd640 10 client.30092 handle_command_reply: tid=0 2024-06-04T05:20:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.380+0000 7f765ca7d740 1 client.30092 shutdown 2024-06-04T05:20:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.380+0000 7f765ca7d740 1 -- 172.21.15.70:0/2151347225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f462f820 msgr2=0x55a6f4637a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.380+0000 7f765ca7d740 1 --2- 172.21.15.70:0/2151347225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f462f820 0x55a6f4637a30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f764c010bf0 tx=0x7f764c004ab0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.380+0000 7f765ca7d740 1 -- 172.21.15.70:0/2151347225 shutdown_connections 2024-06-04T05:20:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.380+0000 7f765ca7d740 1 --2- 172.21.15.70:0/2151347225 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a6f4652f30 0x55a6f4655320 crc :-1 s=READY pgs=493 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.380+0000 7f765ca7d740 1 --2- 172.21.15.70:0/2151347225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f463aa00 0x55a6f46384b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.380+0000 7f7632ffd640 0 client.30092 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.380+0000 7f765ca7d740 1 --2- 172.21.15.70:0/2151347225 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f4630140 0x55a6f4637f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.380+0000 7f765ca7d740 1 --2- 172.21.15.70:0/2151347225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f462f820 0x55a6f4637a30 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.380+0000 7f765ca7d740 1 -- 172.21.15.70:0/2151347225 >> 172.21.15.70:0/2151347225 conn(0x55a6f462bd30 msgr2=0x55a6f462afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.380+0000 7f765ca7d740 1 -- 172.21.15.70:0/2151347225 shutdown_connections 2024-06-04T05:20:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.380+0000 7f765ca7d740 1 -- 172.21.15.70:0/2151347225 wait complete. 2024-06-04T05:20:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.380+0000 7f765ca7d740 20 client.30092 trim_cache size 0 max 16384 2024-06-04T05:20:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.397+0000 7f765ca7d740 1 Processor -- start 2024-06-04T05:20:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.397+0000 7f765ca7d740 1 -- start start 2024-06-04T05:20:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f765ca7d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f462f820 0x55a6f46321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f765ca7d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f4630140 0x55a6f46382d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f765ca7d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f463aa00 0x55a6f465abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f765ca7d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f76541acb20 con 0x55a6f462f820 2024-06-04T05:20:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f765ca7d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f76541ae6f0 con 0x55a6f463aa00 2024-06-04T05:20:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f765ca7d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f76541ad250 con 0x55a6f4630140 2024-06-04T05:20:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f76537fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f462f820 0x55a6f46321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f76537fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f462f820 0x55a6f46321e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52096/0 (socket says 172.21.15.70:52096) 2024-06-04T05:20:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f76537fe640 1 -- 172.21.15.70:0/3791244536 learned_addr learned my addr 172.21.15.70:0/3791244536 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f7652ffd640 1 --2- 172.21.15.70:0/3791244536 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f4630140 0x55a6f46382d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f7653fff640 1 --2- 172.21.15.70:0/3791244536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f463aa00 0x55a6f465abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f76537fe640 1 -- 172.21.15.70:0/3791244536 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f4630140 msgr2=0x55a6f46382d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f76537fe640 1 --2- 172.21.15.70:0/3791244536 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f4630140 0x55a6f46382d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f76537fe640 1 -- 172.21.15.70:0/3791244536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f463aa00 msgr2=0x55a6f465abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f76537fe640 1 --2- 172.21.15.70:0/3791244536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f463aa00 0x55a6f465abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f76537fe640 1 -- 172.21.15.70:0/3791244536 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6f46276f0 con 0x55a6f462f820 2024-06-04T05:20:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f76537fe640 1 --2- 172.21.15.70:0/3791244536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f462f820 0x55a6f46321e0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f764c009420 tx=0x7f764c022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc8dd50b95078812 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f76397fa640 1 -- 172.21.15.70:0/3791244536 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764c02c070 con 0x55a6f462f820 2024-06-04T05:20:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f76397fa640 1 -- 172.21.15.70:0/3791244536 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7648003a80 con 0x55a6f462f820 2024-06-04T05:20:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.398+0000 7f76397fa640 1 -- 172.21.15.70:0/3791244536 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764c01fe60 con 0x55a6f462f820 2024-06-04T05:20:46.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.399+0000 7f765ca7d740 1 -- 172.21.15.70:0/3791244536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f462f820 msgr2=0x55a6f46321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:46.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.399+0000 7f765ca7d740 1 --2- 172.21.15.70:0/3791244536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f462f820 0x55a6f46321e0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f764c009420 tx=0x7f764c022000 comp rx=0 tx=0).stop 2024-06-04T05:20:46.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.399+0000 7f765ca7d740 1 -- 172.21.15.70:0/3791244536 shutdown_connections 2024-06-04T05:20:46.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.399+0000 7f765ca7d740 1 --2- 172.21.15.70:0/3791244536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f463aa00 0x55a6f465abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.399+0000 7f765ca7d740 1 --2- 172.21.15.70:0/3791244536 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f4630140 0x55a6f46382d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.399+0000 7f765ca7d740 1 --2- 172.21.15.70:0/3791244536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f462f820 0x55a6f46321e0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.399+0000 7f765ca7d740 1 -- 172.21.15.70:0/3791244536 >> 172.21.15.70:0/3791244536 conn(0x55a6f462bd30 msgr2=0x55a6f464e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.399+0000 7f765ca7d740 1 -- 172.21.15.70:0/3791244536 shutdown_connections 2024-06-04T05:20:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.399+0000 7f765ca7d740 1 -- 172.21.15.70:0/3791244536 wait complete. 2024-06-04T05:20:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f765ca7d740 1 -- start start 2024-06-04T05:20:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f765ca7d740 1 Processor -- start 2024-06-04T05:20:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f765ca7d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f462f820 0x55a6f464b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f765ca7d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f4630140 0x55a6f464b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f765ca7d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f463aa00 0x55a6f4646900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f765ca7d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f76541adff0 con 0x55a6f462f820 2024-06-04T05:20:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f765ca7d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f76541ad920 con 0x55a6f463aa00 2024-06-04T05:20:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f765ca7d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f76541aab40 con 0x55a6f4630140 2024-06-04T05:20:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f765ca7d740 10 client.0 _ll_register_callbacks cb 0x7f764c009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f76537fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f462f820 0x55a6f464b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f76537fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f462f820 0x55a6f464b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52108/0 (socket says 172.21.15.70:52108) 2024-06-04T05:20:46.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f76537fe640 1 -- 172.21.15.70:0/1311698728 learned_addr learned my addr 172.21.15.70:0/1311698728 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:46.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f7653fff640 1 --2- 172.21.15.70:0/1311698728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f463aa00 0x55a6f4646900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f7652ffd640 1 --2- 172.21.15.70:0/1311698728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f4630140 0x55a6f464b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f76537fe640 1 -- 172.21.15.70:0/1311698728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f4630140 msgr2=0x55a6f464b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:46.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f76537fe640 1 --2- 172.21.15.70:0/1311698728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f4630140 0x55a6f464b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f76537fe640 1 -- 172.21.15.70:0/1311698728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f463aa00 msgr2=0x55a6f4646900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:46.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f76537fe640 1 --2- 172.21.15.70:0/1311698728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f463aa00 0x55a6f4646900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f76537fe640 1 -- 172.21.15.70:0/1311698728 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6f4647100 con 0x55a6f462f820 2024-06-04T05:20:46.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f76537fe640 1 --2- 172.21.15.70:0/1311698728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f462f820 0x55a6f464b390 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f764c018040 tx=0x7f764c0049c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de73406347626c5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:46.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.400+0000 7f7632ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:20:46.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.401+0000 7f7632ffd640 1 -- 172.21.15.70:0/1311698728 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764c01fe60 con 0x55a6f462f820 2024-06-04T05:20:46.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.401+0000 7f7632ffd640 1 -- 172.21.15.70:0/1311698728 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f764c028890 con 0x55a6f462f820 2024-06-04T05:20:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.401+0000 7f765ca7d740 1 -- 172.21.15.70:0/1311698728 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7648003a80 con 0x55a6f462f820 2024-06-04T05:20:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.401+0000 7f7632ffd640 1 -- 172.21.15.70:0/1311698728 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f764c02c070 con 0x55a6f462f820 2024-06-04T05:20:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.401+0000 7f765ca7d740 10 client.21447 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.401+0000 7f765ca7d740 1 -- 172.21.15.70:0/1311698728 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55a6f46bd700 con 0x55a6f462f820 2024-06-04T05:20:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.401+0000 7f7632ffd640 1 -- 172.21.15.70:0/1311698728 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7648003a80 con 0x55a6f462f820 2024-06-04T05:20:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.401+0000 7f7632ffd640 1 -- 172.21.15.70:0/1311698728 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f764c0318e0 con 0x55a6f462f820 2024-06-04T05:20:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.401+0000 7f765ca7d740 10 client.21447 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.401+0000 7f765ca7d740 10 client.21447 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.401+0000 7f765ca7d740 20 client.21447 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.401+0000 7f765ca7d740 1 --2- 172.21.15.70:0/1311698728 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a6f4652f30 0x55a6f46bfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.401+0000 7f765ca7d740 4 client.21447 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:20:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.401+0000 7f765ca7d740 1 -- 172.21.15.70:0/1311698728 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a6f459d360 con 0x55a6f4652f30 2024-06-04T05:20:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.401+0000 7f7652ffd640 1 --2- 172.21.15.70:0/1311698728 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a6f4652f30 0x55a6f46bfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:46.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.402+0000 7f7652ffd640 1 --2- 172.21.15.70:0/1311698728 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a6f4652f30 0x55a6f46bfd70 crc :-1 s=READY pgs=494 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc0e46a4aa4aaf61 server_cookie=23b1ce0ecdd7a39a in_seq=0 out_seq=0 2024-06-04T05:20:46.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.402+0000 7f7632ffd640 10 client.21447 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:46.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.504+0000 7f7632ffd640 1 -- 172.21.15.70:0/1311698728 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55a6f459d360 con 0x55a6f4652f30 2024-06-04T05:20:46.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.504+0000 7f7632ffd640 10 client.21447 handle_command_reply: tid=0 2024-06-04T05:20:46.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.504+0000 7f765ca7d740 1 client.21447 shutdown 2024-06-04T05:20:46.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.504+0000 7f765ca7d740 1 -- 172.21.15.70:0/1311698728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f462f820 msgr2=0x55a6f464b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:46.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.504+0000 7f765ca7d740 1 --2- 172.21.15.70:0/1311698728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f462f820 0x55a6f464b390 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f764c018040 tx=0x7f764c0049c0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.504+0000 7f765ca7d740 1 -- 172.21.15.70:0/1311698728 shutdown_connections 2024-06-04T05:20:46.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.504+0000 7f765ca7d740 1 --2- 172.21.15.70:0/1311698728 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a6f4652f30 0x55a6f46bfd70 crc :-1 s=READY pgs=494 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.504+0000 7f765ca7d740 1 --2- 172.21.15.70:0/1311698728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a6f463aa00 0x55a6f4646900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.504+0000 7f765ca7d740 1 --2- 172.21.15.70:0/1311698728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a6f4630140 0x55a6f464b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.504+0000 7f765ca7d740 1 --2- 172.21.15.70:0/1311698728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6f462f820 0x55a6f464b390 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.504+0000 7f765ca7d740 1 -- 172.21.15.70:0/1311698728 >> 172.21.15.70:0/1311698728 conn(0x55a6f462bd30 msgr2=0x55a6f4639260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:46.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.504+0000 7f7632ffd640 0 client.21447 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:46.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.504+0000 7f765ca7d740 1 -- 172.21.15.70:0/1311698728 shutdown_connections 2024-06-04T05:20:46.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.505+0000 7f765ca7d740 1 -- 172.21.15.70:0/1311698728 wait complete. 2024-06-04T05:20:46.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.505+0000 7f765ca7d740 20 client.21447 trim_cache size 0 max 16384 2024-06-04T05:20:46.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:20:46.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:20:46.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:20:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.505+0000 7f7659d67640 1 -- 172.21.15.70:0/1932624750 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f763c0b2a90 msgr2=0x7f763c0b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.505+0000 7f7659d67640 1 --2- 172.21.15.70:0/1932624750 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f763c0b2a90 0x7f763c0b4f50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x55a6f4630dc0 tx=0x7f764c009500 comp rx=0 tx=0).stop 2024-06-04T05:20:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.505+0000 7f7659d67640 1 -- 172.21.15.70:0/1932624750 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f765419fe00 msgr2=0x7f76541a02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.505+0000 7f7659d67640 1 --2- 172.21.15.70:0/1932624750 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f765419fe00 0x7f76541a02b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f764400e940 tx=0x7f764400ee00 comp rx=0 tx=0).stop 2024-06-04T05:20:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.505+0000 7f7659d67640 1 -- 172.21.15.70:0/1932624750 shutdown_connections 2024-06-04T05:20:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.505+0000 7f7659d67640 1 --2- 172.21.15.70:0/1932624750 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f763c0b2a90 0x7f763c0b4f50 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.505+0000 7f7659d67640 1 --2- 172.21.15.70:0/1932624750 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f765419fe00 0x7f76541a02b0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.505+0000 7f7659d67640 1 --2- 172.21.15.70:0/1932624750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f765410d0b0 0x7f765419ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.505+0000 7f7659d67640 1 --2- 172.21.15.70:0/1932624750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7654103ce0 0x7f765419b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.505+0000 7f7659d67640 1 -- 172.21.15.70:0/1932624750 >> 172.21.15.70:0/1932624750 conn(0x7f7654078a60 msgr2=0x7f76540feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.505+0000 7f7659d67640 1 -- 172.21.15.70:0/1932624750 shutdown_connections 2024-06-04T05:20:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:46.505+0000 7f7659d67640 1 -- 172.21.15.70:0/1932624750 wait complete. 2024-06-04T05:20:46.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 74292 6.80 MB/sec execute 458 sec latency 147.073 ms 2024-06-04T05:20:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:47 smithi112 ceph-mon[36453]: pgmap v743: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.8 MiB/s wr, 406 op/s 2024-06-04T05:20:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:47 smithi098 ceph-mon[35569]: pgmap v743: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.8 MiB/s wr, 406 op/s 2024-06-04T05:20:47.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 74410 6.80 MB/sec execute 459 sec latency 119.181 ms 2024-06-04T05:20:47.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:47 smithi070 ceph-mon[32631]: pgmap v743: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.8 MiB/s wr, 406 op/s 2024-06-04T05:20:48.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 74560 6.80 MB/sec execute 460 sec latency 148.359 ms 2024-06-04T05:20:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:49 smithi112 ceph-mon[36453]: pgmap v744: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 256 op/s 2024-06-04T05:20:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:49 smithi098 ceph-mon[35569]: pgmap v744: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 256 op/s 2024-06-04T05:20:49.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 74678 6.80 MB/sec execute 461 sec latency 123.593 ms 2024-06-04T05:20:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:49 smithi070 ceph-mon[32631]: pgmap v744: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 256 op/s 2024-06-04T05:20:50.344 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:20:50.745 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:20:50.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 74843 6.80 MB/sec execute 462 sec latency 94.523 ms 2024-06-04T05:20:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:51 smithi112 ceph-mon[36453]: pgmap v745: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 256 op/s 2024-06-04T05:20:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:51 smithi098 ceph-mon[35569]: pgmap v745: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 256 op/s 2024-06-04T05:20:51.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 74986 6.80 MB/sec execute 463 sec latency 130.248 ms 2024-06-04T05:20:51.832 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:51 smithi070 ceph-mon[32631]: pgmap v745: 97 pgs: 97 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 256 op/s 2024-06-04T05:20:52.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 75142 6.80 MB/sec execute 464 sec latency 107.012 ms 2024-06-04T05:20:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:53 smithi112 ceph-mon[36453]: pgmap v746: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.7 MiB/s wr, 403 op/s 2024-06-04T05:20:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:53 smithi098 ceph-mon[35569]: pgmap v746: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.7 MiB/s wr, 403 op/s 2024-06-04T05:20:53.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 75329 6.81 MB/sec execute 465 sec latency 82.834 ms 2024-06-04T05:20:53.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:53 smithi070 ceph-mon[32631]: pgmap v746: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.7 MiB/s wr, 403 op/s 2024-06-04T05:20:54.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 75501 6.82 MB/sec execute 466 sec latency 105.680 ms 2024-06-04T05:20:54.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.842+0000 7f7390ade700 1 -- 172.21.15.70:0/35647621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f738c105010 msgr2=0x7f738c10d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.842+0000 7f7390ade700 1 --2- 172.21.15.70:0/35647621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f738c105010 0x7f738c10d500 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7380009990 tx=0x7f7380009ca0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.843+0000 7f7390ade700 1 -- 172.21.15.70:0/35647621 shutdown_connections 2024-06-04T05:20:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.843+0000 7f7390ade700 1 --2- 172.21.15.70:0/35647621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f738c10dad0 0x7f738c10ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.843+0000 7f7390ade700 1 --2- 172.21.15.70:0/35647621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f738c1046f0 0x7f738c104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.843+0000 7f7390ade700 1 --2- 172.21.15.70:0/35647621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f738c105010 0x7f738c10d500 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.843+0000 7f7390ade700 1 -- 172.21.15.70:0/35647621 >> 172.21.15.70:0/35647621 conn(0x7f738c0fde60 msgr2=0x7f738c100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.843+0000 7f7390ade700 1 -- 172.21.15.70:0/35647621 shutdown_connections 2024-06-04T05:20:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.843+0000 7f7390ade700 1 -- 172.21.15.70:0/35647621 wait complete. 2024-06-04T05:20:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.844+0000 7f7390ade700 1 Processor -- start 2024-06-04T05:20:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.844+0000 7f7390ade700 1 -- start start 2024-06-04T05:20:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.844+0000 7f7390ade700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f738c1046f0 0x7f738c10d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.844+0000 7f7390ade700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f738c105010 0x7f738c10a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.844+0000 7f7390ade700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f738c10dad0 0x7f738c10a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.844+0000 7f7390ade700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f738c103c30 con 0x7f738c10dad0 2024-06-04T05:20:54.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.844+0000 7f7390ade700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f738c103ab0 con 0x7f738c105010 2024-06-04T05:20:54.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.844+0000 7f7390ade700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f738c103db0 con 0x7f738c1046f0 2024-06-04T05:20:54.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.844+0000 7f738bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f738c10dad0 0x7f738c10a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.844+0000 7f738affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f738c105010 0x7f738c10a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.844+0000 7f738bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f738c10dad0 0x7f738c10a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52124/0 (socket says 172.21.15.70:52124) 2024-06-04T05:20:54.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.844+0000 7f738b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f738c1046f0 0x7f738c10d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.844+0000 7f738bfff700 1 -- 172.21.15.70:0/3521831939 learned_addr learned my addr 172.21.15.70:0/3521831939 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:54.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.844+0000 7f738affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f738c105010 0x7f738c10a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55538/0 (socket says 172.21.15.70:55538) 2024-06-04T05:20:54.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.845+0000 7f738b7fe700 1 -- 172.21.15.70:0/3521831939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f738c105010 msgr2=0x7f738c10a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:54.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.845+0000 7f738b7fe700 1 --2- 172.21.15.70:0/3521831939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f738c105010 0x7f738c10a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.845+0000 7f738b7fe700 1 -- 172.21.15.70:0/3521831939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f738c10dad0 msgr2=0x7f738c10a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:54.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.845+0000 7f738b7fe700 1 --2- 172.21.15.70:0/3521831939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f738c10dad0 0x7f738c10a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.845+0000 7f738b7fe700 1 -- 172.21.15.70:0/3521831939 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f738c10af30 con 0x7f738c1046f0 2024-06-04T05:20:54.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.845+0000 7f738affd700 1 --2- 172.21.15.70:0/3521831939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f738c105010 0x7f738c10a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:54.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.845+0000 7f738b7fe700 1 --2- 172.21.15.70:0/3521831939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f738c1046f0 0x7f738c10d1b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f737c00e9f0 tx=0x7f737c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:54.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.845+0000 7f7388ff9700 1 -- 172.21.15.70:0/3521831939 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c00cb70 con 0x7f738c1046f0 2024-06-04T05:20:54.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.845+0000 7f7390ade700 1 -- 172.21.15.70:0/3521831939 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f738c1ac530 con 0x7f738c1046f0 2024-06-04T05:20:54.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.845+0000 7f7390ade700 1 -- 172.21.15.70:0/3521831939 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f738c1aca30 con 0x7f738c1046f0 2024-06-04T05:20:54.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.845+0000 7f7388ff9700 1 -- 172.21.15.70:0/3521831939 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f737c00ee90 con 0x7f738c1046f0 2024-06-04T05:20:54.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.845+0000 7f7388ff9700 1 -- 172.21.15.70:0/3521831939 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c003c30 con 0x7f738c1046f0 2024-06-04T05:20:54.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.848+0000 7f7388ff9700 1 -- 172.21.15.70:0/3521831939 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f737c018430 con 0x7f738c1046f0 2024-06-04T05:20:54.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.849+0000 7f7388ff9700 1 --2- 172.21.15.70:0/3521831939 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73740b29f0 0x7f73740b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.850+0000 7f738affd700 1 --2- 172.21.15.70:0/3521831939 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73740b29f0 0x7f73740b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.850+0000 7f7388ff9700 1 -- 172.21.15.70:0/3521831939 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f737c0e26d0 con 0x7f738c1046f0 2024-06-04T05:20:54.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.850+0000 7f739c5d5b80 1 Processor -- start 2024-06-04T05:20:54.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.850+0000 7f739c5d5b80 1 -- start start 2024-06-04T05:20:54.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.850+0000 7f738affd700 1 --2- 172.21.15.70:0/3521831939 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73740b29f0 0x7f73740b4eb0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f738c10bab0 tx=0x7f738001a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:54.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.850+0000 7f739c5d5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 0x561673e071c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.851+0000 7f739c5d5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e02680 0x561673e02b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.851+0000 7f739c5d5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e03040 0x561673e0ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.851+0000 7f739c5d5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561673d61c90 con 0x561673e03040 2024-06-04T05:20:54.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.851+0000 7f739c5d5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561673d51830 con 0x561673e02680 2024-06-04T05:20:54.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.851+0000 7f739c5d5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561673d55870 con 0x561673e06de0 2024-06-04T05:20:54.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.851+0000 7f738affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e02680 0x561673e02b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.851+0000 7f738b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 0x561673e071c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.851+0000 7f738b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 0x561673e071c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48154/0 (socket says 172.21.15.70:48154) 2024-06-04T05:20:54.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.851+0000 7f738b7fe700 1 -- 172.21.15.70:0/1472210512 learned_addr learned my addr 172.21.15.70:0/1472210512 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:54.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.851+0000 7f738bfff700 1 --2- 172.21.15.70:0/1472210512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e03040 0x561673e0ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.851+0000 7f738b7fe700 1 -- 172.21.15.70:0/1472210512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e02680 msgr2=0x561673e02b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:54.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.851+0000 7f738b7fe700 1 --2- 172.21.15.70:0/1472210512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e02680 0x561673e02b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.851+0000 7f738b7fe700 1 -- 172.21.15.70:0/1472210512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e03040 msgr2=0x561673e0ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:54.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.851+0000 7f738b7fe700 1 --2- 172.21.15.70:0/1472210512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e03040 0x561673e0ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.851+0000 7f738b7fe700 1 -- 172.21.15.70:0/1472210512 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561673ca18d0 con 0x561673e06de0 2024-06-04T05:20:54.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.851+0000 7f738affd700 1 --2- 172.21.15.70:0/1472210512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e02680 0x561673e02b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:20:54.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.851+0000 7f738b7fe700 1 --2- 172.21.15.70:0/1472210512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 0x561673e071c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f737c0a9680 tx=0x7f737c0aa100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e8e91d2c9d78e2d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:54.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.852+0000 7f7371ffb700 1 -- 172.21.15.70:0/1472210512 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c0b27e0 con 0x561673e06de0 2024-06-04T05:20:54.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.852+0000 7f7371ffb700 1 -- 172.21.15.70:0/1472210512 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f737c0a5c10 con 0x561673e06de0 2024-06-04T05:20:54.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.852+0000 7f7371ffb700 1 -- 172.21.15.70:0/1472210512 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c0a9be0 con 0x561673e06de0 2024-06-04T05:20:54.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.853+0000 7f739c5d5b80 1 -- 172.21.15.70:0/1472210512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 msgr2=0x561673e071c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:54.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.853+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/1472210512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 0x561673e071c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f737c0a9680 tx=0x7f737c0aa100 comp rx=0 tx=0).stop 2024-06-04T05:20:54.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.853+0000 7f739c5d5b80 1 -- 172.21.15.70:0/1472210512 shutdown_connections 2024-06-04T05:20:54.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.853+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/1472210512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e03040 0x561673e0ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.853+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/1472210512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 0x561673e071c0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.853+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/1472210512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e02680 0x561673e02b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.853+0000 7f739c5d5b80 1 -- 172.21.15.70:0/1472210512 >> 172.21.15.70:0/1472210512 conn(0x561673d30080 msgr2=0x561673dfce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:54.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.853+0000 7f739c5d5b80 1 -- 172.21.15.70:0/1472210512 shutdown_connections 2024-06-04T05:20:54.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.853+0000 7f739c5d5b80 1 -- 172.21.15.70:0/1472210512 wait complete. 2024-06-04T05:20:54.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.853+0000 7f739c5d5b80 1 -- start start 2024-06-04T05:20:54.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.854+0000 7f739c5d5b80 1 Processor -- start 2024-06-04T05:20:54.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.854+0000 7f739c5d5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e02680 0x561673e0ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.854+0000 7f739c5d5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 0x561673e0bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.854+0000 7f739c5d5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 0x561673e0c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.854+0000 7f739c5d5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561673c57fa0 con 0x561673e02680 2024-06-04T05:20:54.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.854+0000 7f739c5d5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561673cf37c0 con 0x561673e03040 2024-06-04T05:20:54.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.854+0000 7f739c5d5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561673e0f3b0 con 0x561673e06de0 2024-06-04T05:20:54.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.854+0000 7f738bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 0x561673e0c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.854+0000 7f738bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 0x561673e0c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48170/0 (socket says 172.21.15.70:48170) 2024-06-04T05:20:54.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.854+0000 7f738bfff700 1 -- 172.21.15.70:0/1508868430 learned_addr learned my addr 172.21.15.70:0/1508868430 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:54.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.855+0000 7f738affd700 1 --2- 172.21.15.70:0/1508868430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 0x561673e0bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.854+0000 7f738b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e02680 0x561673e0ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.855+0000 7f739c5d5b80 10 client.0 _ll_register_callbacks cb 0x561673cec920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:54.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.855+0000 7f738bfff700 1 -- 172.21.15.70:0/1508868430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 msgr2=0x561673e0bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:54.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.855+0000 7f738bfff700 1 --2- 172.21.15.70:0/1508868430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 0x561673e0bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.855+0000 7f738bfff700 1 -- 172.21.15.70:0/1508868430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e02680 msgr2=0x561673e0ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:54.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.855+0000 7f738bfff700 1 --2- 172.21.15.70:0/1508868430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e02680 0x561673e0ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.855+0000 7f738bfff700 1 -- 172.21.15.70:0/1508868430 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561673cf0690 con 0x561673e06de0 2024-06-04T05:20:54.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.855+0000 7f738bfff700 1 --2- 172.21.15.70:0/1508868430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 0x561673e0c2a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7384014e50 tx=0x7f738400e610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d75c2070e53ea10f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:54.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.856+0000 7f736affd700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:20:54.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.856+0000 7f736affd700 1 -- 172.21.15.70:0/1508868430 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738400ecf0 con 0x561673e06de0 2024-06-04T05:20:54.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.856+0000 7f739c5d5b80 1 -- 172.21.15.70:0/1508868430 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f737c0a5c10 con 0x561673e06de0 2024-06-04T05:20:54.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.856+0000 7f736affd700 1 -- 172.21.15.70:0/1508868430 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f7384003a00 con 0x561673e06de0 2024-06-04T05:20:54.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.856+0000 7f736affd700 1 -- 172.21.15.70:0/1508868430 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f738400ee90 con 0x561673e06de0 2024-06-04T05:20:54.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.856+0000 7f739c5d5b80 10 client.30250 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:54.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.856+0000 7f739c5d5b80 1 -- 172.21.15.70:0/1508868430 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x561673e19c50 con 0x561673e06de0 2024-06-04T05:20:54.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.856+0000 7f736affd700 1 -- 172.21.15.70:0/1508868430 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f737c0a5c10 con 0x561673e06de0 2024-06-04T05:20:54.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.856+0000 7f736affd700 1 -- 172.21.15.70:0/1508868430 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f738401e490 con 0x561673e06de0 2024-06-04T05:20:54.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.856+0000 7f739c5d5b80 10 client.30250 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:54.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.856+0000 7f739c5d5b80 10 client.30250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:54.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.857+0000 7f739c5d5b80 20 client.30250 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:54.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.857+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/1508868430 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561673e1c6c0 0x561673e1eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.857+0000 7f739c5d5b80 4 client.30250 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:20:54.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.857+0000 7f739c5d5b80 1 -- 172.21.15.70:0/1508868430 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561673dce9e0 con 0x561673e1c6c0 2024-06-04T05:20:54.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.857+0000 7f738b7fe700 1 --2- 172.21.15.70:0/1508868430 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561673e1c6c0 0x561673e1eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.857+0000 7f736affd700 10 client.30250 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:54.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.857+0000 7f738b7fe700 1 --2- 172.21.15.70:0/1508868430 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561673e1c6c0 0x561673e1eae0 crc :-1 s=READY pgs=495 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4193ec11a692ca5 server_cookie=b037281229758d25 in_seq=0 out_seq=0 2024-06-04T05:20:54.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.858+0000 7f736affd700 1 -- 172.21.15.70:0/1508868430 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561673dce9e0 con 0x561673e1c6c0 2024-06-04T05:20:54.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.858+0000 7f736affd700 10 client.30250 handle_command_reply: tid=0 2024-06-04T05:20:54.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.858+0000 7f739c5d5b80 1 client.30250 shutdown 2024-06-04T05:20:54.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.859+0000 7f739c5d5b80 1 -- 172.21.15.70:0/1508868430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 msgr2=0x561673e0c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:54.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.859+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/1508868430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 0x561673e0c2a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7384014e50 tx=0x7f738400e610 comp rx=0 tx=0).stop 2024-06-04T05:20:54.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.859+0000 7f739c5d5b80 1 -- 172.21.15.70:0/1508868430 shutdown_connections 2024-06-04T05:20:54.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.859+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/1508868430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 0x561673e0c2a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.859+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/1508868430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e02680 0x561673e0ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.859+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/1508868430 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561673e1c6c0 0x561673e1eae0 crc :-1 s=READY pgs=495 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.859+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/1508868430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 0x561673e0bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.859+0000 7f739c5d5b80 1 -- 172.21.15.70:0/1508868430 >> 172.21.15.70:0/1508868430 conn(0x561673d30080 msgr2=0x561673dfc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:54.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.859+0000 7f736affd700 0 client.30250 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:54.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.859+0000 7f739c5d5b80 1 -- 172.21.15.70:0/1508868430 shutdown_connections 2024-06-04T05:20:54.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.859+0000 7f739c5d5b80 1 -- 172.21.15.70:0/1508868430 wait complete. 2024-06-04T05:20:54.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.859+0000 7f739c5d5b80 20 client.30250 trim_cache size 0 max 16384 2024-06-04T05:20:54.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.890+0000 7f739c5d5b80 1 Processor -- start 2024-06-04T05:20:54.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.890+0000 7f739c5d5b80 1 -- start start 2024-06-04T05:20:54.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.891+0000 7f739c5d5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e02680 0x561673e0c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.891+0000 7f739c5d5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 0x561673e06350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.891+0000 7f739c5d5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e06de0 0x561673dfcc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.891+0000 7f739c5d5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f737800d690 con 0x561673e06de0 2024-06-04T05:20:54.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.891+0000 7f739c5d5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f737800f260 con 0x561673e03040 2024-06-04T05:20:54.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.891+0000 7f739c5d5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f737800ddc0 con 0x561673e02680 2024-06-04T05:20:54.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.891+0000 7f738b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e02680 0x561673e0c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.891+0000 7f738affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 0x561673e06350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.891+0000 7f738affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 0x561673e06350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55562/0 (socket says 172.21.15.70:55562) 2024-06-04T05:20:54.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.891+0000 7f738affd700 1 -- 172.21.15.70:0/3482336795 learned_addr learned my addr 172.21.15.70:0/3482336795 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:54.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.891+0000 7f738bfff700 1 --2- 172.21.15.70:0/3482336795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e06de0 0x561673dfcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.892+0000 7f738bfff700 1 -- 172.21.15.70:0/3482336795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e02680 msgr2=0x561673e0c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:54.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.892+0000 7f738bfff700 1 --2- 172.21.15.70:0/3482336795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e02680 0x561673e0c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.892+0000 7f738bfff700 1 -- 172.21.15.70:0/3482336795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 msgr2=0x561673e06350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:54.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.892+0000 7f738bfff700 1 --2- 172.21.15.70:0/3482336795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 0x561673e06350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.892+0000 7f738bfff700 1 -- 172.21.15.70:0/3482336795 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561673dfd730 con 0x561673e06de0 2024-06-04T05:20:54.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.892+0000 7f738b7fe700 1 --2- 172.21.15.70:0/3482336795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e02680 0x561673e0c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:54.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.892+0000 7f738bfff700 1 --2- 172.21.15.70:0/3482336795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e06de0 0x561673dfcc90 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f7384000f80 tx=0x7f73840095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661b36eff0fc625b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:54.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.892+0000 7f73717fa700 1 -- 172.21.15.70:0/3482336795 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7384034070 con 0x561673e06de0 2024-06-04T05:20:54.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.892+0000 7f73717fa700 1 -- 172.21.15.70:0/3482336795 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f738401eb40 con 0x561673e06de0 2024-06-04T05:20:54.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.892+0000 7f73717fa700 1 -- 172.21.15.70:0/3482336795 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738402a040 con 0x561673e06de0 2024-06-04T05:20:54.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.893+0000 7f739c5d5b80 1 -- 172.21.15.70:0/3482336795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e06de0 msgr2=0x561673dfcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:54.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.893+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/3482336795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e06de0 0x561673dfcc90 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f7384000f80 tx=0x7f73840095e0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.893+0000 7f739c5d5b80 1 -- 172.21.15.70:0/3482336795 shutdown_connections 2024-06-04T05:20:54.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.893+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/3482336795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e06de0 0x561673dfcc90 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.893+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/3482336795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e02680 0x561673e0c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.893+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/3482336795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 0x561673e06350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.893+0000 7f739c5d5b80 1 -- 172.21.15.70:0/3482336795 >> 172.21.15.70:0/3482336795 conn(0x561673d30080 msgr2=0x561673e0d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:54.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.894+0000 7f739c5d5b80 1 -- 172.21.15.70:0/3482336795 shutdown_connections 2024-06-04T05:20:54.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.894+0000 7f739c5d5b80 1 -- 172.21.15.70:0/3482336795 wait complete. 2024-06-04T05:20:54.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.894+0000 7f739c5d5b80 1 -- start start 2024-06-04T05:20:54.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.894+0000 7f739c5d5b80 1 Processor -- start 2024-06-04T05:20:54.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.894+0000 7f739c5d5b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e02680 0x561673e08fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.894+0000 7f739c5d5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 0x561673e09500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.894+0000 7f739c5d5b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 0x561673e09a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.895+0000 7f739c5d5b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f737800eb60 con 0x561673e02680 2024-06-04T05:20:54.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.895+0000 7f739c5d5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f737800e490 con 0x561673e03040 2024-06-04T05:20:54.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.895+0000 7f739c5d5b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f737800b650 con 0x561673e06de0 2024-06-04T05:20:54.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.895+0000 7f739c5d5b80 10 client.0 _ll_register_callbacks cb 0x561673cec920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:54.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.895+0000 7f738affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 0x561673e09500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.895+0000 7f738affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 0x561673e09500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55578/0 (socket says 172.21.15.70:55578) 2024-06-04T05:20:54.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.895+0000 7f738affd700 1 -- 172.21.15.70:0/3529420611 learned_addr learned my addr 172.21.15.70:0/3529420611 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:54.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.895+0000 7f738b7fe700 1 --2- 172.21.15.70:0/3529420611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e02680 0x561673e08fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.895+0000 7f738bfff700 1 --2- 172.21.15.70:0/3529420611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 0x561673e09a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.895+0000 7f738affd700 1 -- 172.21.15.70:0/3529420611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 msgr2=0x561673e09a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:54.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.895+0000 7f738affd700 1 --2- 172.21.15.70:0/3529420611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 0x561673e09a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.895+0000 7f738affd700 1 -- 172.21.15.70:0/3529420611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e02680 msgr2=0x561673e08fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:54.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.895+0000 7f738affd700 1 --2- 172.21.15.70:0/3529420611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e02680 0x561673e08fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.895+0000 7f738affd700 1 -- 172.21.15.70:0/3529420611 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561673e86480 con 0x561673e03040 2024-06-04T05:20:54.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.895+0000 7f738affd700 1 --2- 172.21.15.70:0/3529420611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 0x561673e09500 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f738000b2e0 tx=0x7f7380000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7a4b01b3bbb71f4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:54.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.895+0000 7f736affd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:20:54.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.896+0000 7f736affd700 1 -- 172.21.15.70:0/3529420611 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7380038060 con 0x561673e03040 2024-06-04T05:20:54.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.896+0000 7f736affd700 1 -- 172.21.15.70:0/3529420611 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f738000f670 con 0x561673e03040 2024-06-04T05:20:54.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.896+0000 7f739c5d5b80 1 -- 172.21.15.70:0/3529420611 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f738401eb40 con 0x561673e03040 2024-06-04T05:20:54.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.896+0000 7f736affd700 1 -- 172.21.15.70:0/3529420611 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f7380003e30 con 0x561673e03040 2024-06-04T05:20:54.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.896+0000 7f739c5d5b80 10 client.30116 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:54.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.896+0000 7f739c5d5b80 1 -- 172.21.15.70:0/3529420611 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x561673e87780 con 0x561673e03040 2024-06-04T05:20:54.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.896+0000 7f736affd700 1 -- 172.21.15.70:0/3529420611 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f738401eb40 con 0x561673e03040 2024-06-04T05:20:54.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.896+0000 7f736affd700 1 -- 172.21.15.70:0/3529420611 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f7380033930 con 0x561673e03040 2024-06-04T05:20:54.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.896+0000 7f739c5d5b80 10 client.30116 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:54.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.897+0000 7f739c5d5b80 10 client.30116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:54.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.897+0000 7f739c5d5b80 20 client.30116 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:54.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.897+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/3529420611 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561673e87ea0 0x561673e8a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:54.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.897+0000 7f739c5d5b80 4 client.30116 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:20:54.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.897+0000 7f739c5d5b80 1 -- 172.21.15.70:0/3529420611 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561673d63be0 con 0x561673e87ea0 2024-06-04T05:20:54.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.897+0000 7f738b7fe700 1 --2- 172.21.15.70:0/3529420611 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561673e87ea0 0x561673e8a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:54.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.897+0000 7f736affd700 10 client.30116 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:54.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.897+0000 7f738b7fe700 1 --2- 172.21.15.70:0/3529420611 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561673e87ea0 0x561673e8a2c0 crc :-1 s=READY pgs=496 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a9632ba6b62731c server_cookie=9a8ffd82dc5bbaf4 in_seq=0 out_seq=0 2024-06-04T05:20:54.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.898+0000 7f736affd700 1 -- 172.21.15.70:0/3529420611 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561673d63be0 con 0x561673e87ea0 2024-06-04T05:20:54.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.898+0000 7f736affd700 10 client.30116 handle_command_reply: tid=0 2024-06-04T05:20:54.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.898+0000 7f739c5d5b80 1 client.30116 shutdown 2024-06-04T05:20:54.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.898+0000 7f739c5d5b80 1 -- 172.21.15.70:0/3529420611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 msgr2=0x561673e09500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:54.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.898+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/3529420611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 0x561673e09500 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f738000b2e0 tx=0x7f7380000bc0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.898+0000 7f739c5d5b80 1 -- 172.21.15.70:0/3529420611 shutdown_connections 2024-06-04T05:20:54.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.898+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/3529420611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561673e06de0 0x561673e09a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.898+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/3529420611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561673e02680 0x561673e08fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.898+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/3529420611 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x561673e87ea0 0x561673e8a2c0 crc :-1 s=READY pgs=496 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.898+0000 7f739c5d5b80 1 --2- 172.21.15.70:0/3529420611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561673e03040 0x561673e09500 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.898+0000 7f736affd700 0 client.30116 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:54.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.898+0000 7f739c5d5b80 1 -- 172.21.15.70:0/3529420611 >> 172.21.15.70:0/3529420611 conn(0x561673d30080 msgr2=0x561673e01f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:54.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.898+0000 7f739c5d5b80 1 -- 172.21.15.70:0/3529420611 shutdown_connections 2024-06-04T05:20:54.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.898+0000 7f739c5d5b80 1 -- 172.21.15.70:0/3529420611 wait complete. 2024-06-04T05:20:54.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.898+0000 7f739c5d5b80 20 client.30116 trim_cache size 0 max 16384 2024-06-04T05:20:54.904 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:20:54.904 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:20:54.904 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:20:54.904 INFO:teuthology.orchestra.run.smithi070.stdout: "3bfba7ca-ffef-4b68-9d78-40be723b6e3c": { 2024-06-04T05:20:54.905 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:20:54.905 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "3bfba7ca-ffef-4b68-9d78-40be723b6e3c", 2024-06-04T05:20:54.905 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:20:54.905 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:20:54.905 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:20:54.905 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:20:54.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.899+0000 7f7390ade700 1 -- 172.21.15.70:0/3521831939 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73740b29f0 msgr2=0x7f73740b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:54.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.899+0000 7f7390ade700 1 --2- 172.21.15.70:0/3521831939 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73740b29f0 0x7f73740b4eb0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f738c10bab0 tx=0x7f738001a040 comp rx=0 tx=0).stop 2024-06-04T05:20:54.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.899+0000 7f7390ade700 1 -- 172.21.15.70:0/3521831939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f738c1046f0 msgr2=0x7f738c10d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:54.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.899+0000 7f7390ade700 1 --2- 172.21.15.70:0/3521831939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f738c1046f0 0x7f738c10d1b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f737c00e9f0 tx=0x7f737c00ed00 comp rx=0 tx=0).stop 2024-06-04T05:20:54.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.899+0000 7f7390ade700 1 -- 172.21.15.70:0/3521831939 shutdown_connections 2024-06-04T05:20:54.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.899+0000 7f7390ade700 1 --2- 172.21.15.70:0/3521831939 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73740b29f0 0x7f73740b4eb0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.899+0000 7f7390ade700 1 --2- 172.21.15.70:0/3521831939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f738c10dad0 0x7f738c10a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.899+0000 7f7390ade700 1 --2- 172.21.15.70:0/3521831939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f738c1046f0 0x7f738c10d1b0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.899+0000 7f7390ade700 1 --2- 172.21.15.70:0/3521831939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f738c105010 0x7f738c10a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:54.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.899+0000 7f7390ade700 1 -- 172.21.15.70:0/3521831939 >> 172.21.15.70:0/3521831939 conn(0x7f738c0fde60 msgr2=0x7f738c10f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:54.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.900+0000 7f7390ade700 1 -- 172.21.15.70:0/3521831939 shutdown_connections 2024-06-04T05:20:54.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:20:54.900+0000 7f7390ade700 1 -- 172.21.15.70:0/3521831939 wait complete. 2024-06-04T05:20:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:55 smithi112 ceph-mon[36453]: pgmap v747: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.3 MiB/s wr, 288 op/s 2024-06-04T05:20:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:55 smithi098 ceph-mon[35569]: pgmap v747: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.3 MiB/s wr, 288 op/s 2024-06-04T05:20:55.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 75667 6.83 MB/sec execute 467 sec latency 69.750 ms 2024-06-04T05:20:55.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:55 smithi070 ceph-mon[32631]: pgmap v747: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.3 MiB/s wr, 288 op/s 2024-06-04T05:20:55.939 INFO:tasks.cephfs.filesystem:scrub status for tag:3bfba7ca-ffef-4b68-9d78-40be723b6e3c - {'path': '/', 'tag': '3bfba7ca-ffef-4b68-9d78-40be723b6e3c', 'options': 'recursive,force'} 2024-06-04T05:20:56.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 75824 6.83 MB/sec execute 468 sec latency 90.982 ms 2024-06-04T05:20:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:57 smithi112 ceph-mon[36453]: pgmap v748: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.7 MiB/s wr, 417 op/s 2024-06-04T05:20:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:57 smithi098 ceph-mon[35569]: pgmap v748: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.7 MiB/s wr, 417 op/s 2024-06-04T05:20:57.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 76034 6.84 MB/sec execute 469 sec latency 398.343 ms 2024-06-04T05:20:57.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:57 smithi070 ceph-mon[32631]: pgmap v748: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.7 MiB/s wr, 417 op/s 2024-06-04T05:20:58.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:58 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:20:58.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:58 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:20:58.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:58 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:20:58.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:58 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:20:58.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:58 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:20:58.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:58 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:20:58.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 76247 6.84 MB/sec execute 470 sec latency 53.731 ms 2024-06-04T05:20:58.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:58 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:20:58.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:58 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:20:58.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:58 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:20:59.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.628+0000 7f52aead9640 1 -- 172.21.15.70:0/341743903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52a8102180 msgr2=0x7f52a810eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:59.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.628+0000 7f52aead9640 1 --2- 172.21.15.70:0/341743903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52a8102180 0x7f52a810eed0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f5294009900 tx=0x7f529402a410 comp rx=0 tx=0).stop 2024-06-04T05:20:59.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.628+0000 7f52aead9640 1 -- 172.21.15.70:0/341743903 shutdown_connections 2024-06-04T05:20:59.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.628+0000 7f52aead9640 1 --2- 172.21.15.70:0/341743903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52a810f410 0x7f52a8111800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.628+0000 7f52aead9640 1 --2- 172.21.15.70:0/341743903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52a8102180 0x7f52a810eed0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.628+0000 7f52aead9640 1 --2- 172.21.15.70:0/341743903 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52a8101860 0x7f52a8101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.628+0000 7f52aead9640 1 -- 172.21.15.70:0/341743903 >> 172.21.15.70:0/341743903 conn(0x7f52a8078ad0 msgr2=0x7f52a8078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:59.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.628+0000 7f52aead9640 1 -- 172.21.15.70:0/341743903 shutdown_connections 2024-06-04T05:20:59.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.628+0000 7f52aead9640 1 -- 172.21.15.70:0/341743903 wait complete. 2024-06-04T05:20:59.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.629+0000 7f52aead9640 1 Processor -- start 2024-06-04T05:20:59.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.629+0000 7f52aead9640 1 -- start start 2024-06-04T05:20:59.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.629+0000 7f52aead9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52a8101860 0x7f52a819b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.629+0000 7f52aead9640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52a8102180 0x7f52a819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.629+0000 7f52aead9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52a810f410 0x7f52a819fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.629+0000 7f52aead9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f52a8114da0 con 0x7f52a8101860 2024-06-04T05:20:59.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.629+0000 7f52aead9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f52a8114c20 con 0x7f52a810f410 2024-06-04T05:20:59.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.629+0000 7f52aead9640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f52a8114f20 con 0x7f52a8102180 2024-06-04T05:20:59.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.629+0000 7f52ac875640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52a8101860 0x7f52a819b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.629+0000 7f52a7fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52a8102180 0x7f52a819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.629+0000 7f52ac875640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52a8101860 0x7f52a819b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38502/0 (socket says 172.21.15.70:38502) 2024-06-04T05:20:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.629+0000 7f52ac875640 1 -- 172.21.15.70:0/287881533 learned_addr learned my addr 172.21.15.70:0/287881533 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.629+0000 7f52ad076640 1 --2- 172.21.15.70:0/287881533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52a810f410 0x7f52a819fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.630+0000 7f52ac875640 1 -- 172.21.15.70:0/287881533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52a8102180 msgr2=0x7f52a819b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.630+0000 7f52ac875640 1 --2- 172.21.15.70:0/287881533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52a8102180 0x7f52a819b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.630+0000 7f52ac875640 1 -- 172.21.15.70:0/287881533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52a810f410 msgr2=0x7f52a819fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.630+0000 7f52ac875640 1 --2- 172.21.15.70:0/287881533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52a810f410 0x7f52a819fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.630+0000 7f52ac875640 1 -- 172.21.15.70:0/287881533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a81a0350 con 0x7f52a8101860 2024-06-04T05:20:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.630+0000 7f52a7fff640 1 --2- 172.21.15.70:0/287881533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52a8102180 0x7f52a819b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.630+0000 7f52ac875640 1 --2- 172.21.15.70:0/287881533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52a8101860 0x7f52a819b360 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f529c00e9f0 tx=0x7f529c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.630+0000 7f52a5ffb640 1 -- 172.21.15.70:0/287881533 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c00cd80 con 0x7f52a8101860 2024-06-04T05:20:59.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.630+0000 7f52a5ffb640 1 -- 172.21.15.70:0/287881533 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f529c0054b0 con 0x7f52a8101860 2024-06-04T05:20:59.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.630+0000 7f52a5ffb640 1 -- 172.21.15.70:0/287881533 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c010430 con 0x7f52a8101860 2024-06-04T05:20:59.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.630+0000 7f52aead9640 1 -- 172.21.15.70:0/287881533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52a81a0640 con 0x7f52a8101860 2024-06-04T05:20:59.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.631+0000 7f52aead9640 1 -- 172.21.15.70:0/287881533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52a81a7ed0 con 0x7f52a8101860 2024-06-04T05:20:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.633+0000 7f52a5ffb640 1 -- 172.21.15.70:0/287881533 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f529c0046c0 con 0x7f52a8101860 2024-06-04T05:20:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.634+0000 7f52b17f0740 1 Processor -- start 2024-06-04T05:20:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.634+0000 7f52b17f0740 1 -- start start 2024-06-04T05:20:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.634+0000 7f52b17f0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 0x5615fc158c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.634+0000 7f52b17f0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 0x5615fc1634c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.634+0000 7f52b17f0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 0x5615fc165df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.634+0000 7f52b17f0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5615fc0cac30 con 0x5615fc159140 2024-06-04T05:20:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.634+0000 7f52b17f0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5615fc0ba7d0 con 0x5615fc163a00 2024-06-04T05:20:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.634+0000 7f52b17f0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5615fc0be810 con 0x5615fc158820 2024-06-04T05:20:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.634+0000 7f52ac875640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 0x5615fc1634c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.634+0000 7f52ad076640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 0x5615fc158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.634+0000 7f52ad076640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 0x5615fc158c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:45064/0 (socket says 172.21.15.70:45064) 2024-06-04T05:20:59.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.634+0000 7f52ad076640 1 -- 172.21.15.70:0/3249015373 learned_addr learned my addr 172.21.15.70:0/3249015373 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:59.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.634+0000 7f52a7fff640 1 --2- 172.21.15.70:0/3249015373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 0x5615fc165df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.634+0000 7f52a5ffb640 1 --2- 172.21.15.70:0/287881533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52980b2a10 0x7f52980b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.635+0000 7f52ad076640 1 -- 172.21.15.70:0/3249015373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 msgr2=0x5615fc165df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:59.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.635+0000 7f52ad076640 1 --2- 172.21.15.70:0/3249015373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 0x5615fc165df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.635+0000 7f52ad076640 1 -- 172.21.15.70:0/3249015373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 msgr2=0x5615fc1634c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:59.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.635+0000 7f52ad076640 1 --2- 172.21.15.70:0/3249015373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 0x5615fc1634c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.635+0000 7f52ad076640 1 -- 172.21.15.70:0/3249015373 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615fbdcf210 con 0x5615fc158820 2024-06-04T05:20:59.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.635+0000 7f52a5ffb640 1 -- 172.21.15.70:0/287881533 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f529c0e2990 con 0x7f52a8101860 2024-06-04T05:20:59.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.635+0000 7f52a7fff640 1 --2- 172.21.15.70:0/3249015373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 0x5615fc165df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:59.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.635+0000 7f52ad076640 1 --2- 172.21.15.70:0/287881533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52980b2a10 0x7f52980b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.635+0000 7f52ac875640 1 --2- 172.21.15.70:0/3249015373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 0x5615fc1634c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:20:59.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.635+0000 7f52ad076640 1 --2- 172.21.15.70:0/3249015373 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 0x5615fc158c00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f52a00114c0 tx=0x7f52a00118c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c17c0a366720a4a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:59.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.635+0000 7f5292ffd640 1 -- 172.21.15.70:0/3249015373 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a0017070 con 0x5615fc158820 2024-06-04T05:20:59.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.635+0000 7f52ad076640 1 --2- 172.21.15.70:0/287881533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52980b2a10 0x7f52980b4ed0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f52a0022000 tx=0x7f52a0023000 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:59.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.635+0000 7f5292ffd640 1 -- 172.21.15.70:0/3249015373 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f52a000fb70 con 0x5615fc158820 2024-06-04T05:20:59.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.635+0000 7f5292ffd640 1 -- 172.21.15.70:0/3249015373 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a001b840 con 0x5615fc158820 2024-06-04T05:20:59.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.636+0000 7f52b17f0740 1 -- 172.21.15.70:0/3249015373 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 msgr2=0x5615fc158c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:59.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.636+0000 7f52b17f0740 1 --2- 172.21.15.70:0/3249015373 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 0x5615fc158c00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f52a00114c0 tx=0x7f52a00118c0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.636+0000 7f52b17f0740 1 -- 172.21.15.70:0/3249015373 shutdown_connections 2024-06-04T05:20:59.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.636+0000 7f52b17f0740 1 --2- 172.21.15.70:0/3249015373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 0x5615fc165df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.636+0000 7f52b17f0740 1 --2- 172.21.15.70:0/3249015373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 0x5615fc1634c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.636+0000 7f52b17f0740 1 --2- 172.21.15.70:0/3249015373 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 0x5615fc158c00 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.636+0000 7f52b17f0740 1 -- 172.21.15.70:0/3249015373 >> 172.21.15.70:0/3249015373 conn(0x5615fc154d30 msgr2=0x5615fc155110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:59.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.636+0000 7f52b17f0740 1 -- 172.21.15.70:0/3249015373 shutdown_connections 2024-06-04T05:20:59.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.636+0000 7f52b17f0740 1 -- 172.21.15.70:0/3249015373 wait complete. 2024-06-04T05:20:59.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.637+0000 7f52b17f0740 1 -- start start 2024-06-04T05:20:59.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.637+0000 7f52b17f0740 1 Processor -- start 2024-06-04T05:20:59.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.637+0000 7f52b17f0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 0x5615fc1609f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.637+0000 7f52b17f0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 0x5615fc160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.637+0000 7f52b17f0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 0x5615fc161470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.637+0000 7f52b17f0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5615fbda0790 con 0x5615fc159140 2024-06-04T05:20:59.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.637+0000 7f52b17f0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5615fc076890 con 0x5615fc163a00 2024-06-04T05:20:59.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.637+0000 7f52b17f0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5615fc166630 con 0x5615fc158820 2024-06-04T05:20:59.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.637+0000 7f52b17f0740 10 client.0 _ll_register_callbacks cb 0x5615fbdcea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:59.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.638+0000 7f52ad076640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 0x5615fc161470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.638+0000 7f52a7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 0x5615fc160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.638+0000 7f52ac875640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 0x5615fc1609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.638+0000 7f52ad076640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 0x5615fc161470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:34934/0 (socket says 172.21.15.70:34934) 2024-06-04T05:20:59.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.638+0000 7f52a7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 0x5615fc160f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38514/0 (socket says 172.21.15.70:38514) 2024-06-04T05:20:59.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.638+0000 7f52ad076640 1 -- 172.21.15.70:0/3527447854 learned_addr learned my addr 172.21.15.70:0/3527447854 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:59.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.638+0000 7f52ad076640 1 -- 172.21.15.70:0/3527447854 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 msgr2=0x5615fc1609f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:59.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.638+0000 7f52ad076640 1 --2- 172.21.15.70:0/3527447854 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 0x5615fc1609f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.638+0000 7f52ad076640 1 -- 172.21.15.70:0/3527447854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 msgr2=0x5615fc160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:59.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.638+0000 7f52ad076640 1 --2- 172.21.15.70:0/3527447854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 0x5615fc160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.638+0000 7f52ad076640 1 -- 172.21.15.70:0/3527447854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615fbe9b620 con 0x5615fc163a00 2024-06-04T05:20:59.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.638+0000 7f52ad076640 1 --2- 172.21.15.70:0/3527447854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 0x5615fc161470 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f52a001b460 tx=0x7f52a000ff50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96c471521ac9663 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:59.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.638+0000 7f5287fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:20:59.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.638+0000 7f5287fff640 1 -- 172.21.15.70:0/3527447854 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a0017050 con 0x5615fc163a00 2024-06-04T05:20:59.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.639+0000 7f5287fff640 1 -- 172.21.15.70:0/3527447854 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f52a001bb20 con 0x5615fc163a00 2024-06-04T05:20:59.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.639+0000 7f52b17f0740 1 -- 172.21.15.70:0/3527447854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52a000fb70 con 0x5615fc163a00 2024-06-04T05:20:59.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.639+0000 7f5287fff640 1 -- 172.21.15.70:0/3527447854 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f52a001b890 con 0x5615fc163a00 2024-06-04T05:20:59.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.639+0000 7f52b17f0740 10 client.30128 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:59.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.639+0000 7f52b17f0740 1 -- 172.21.15.70:0/3527447854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x5615fc1723e0 con 0x5615fc163a00 2024-06-04T05:20:59.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.639+0000 7f5287fff640 1 -- 172.21.15.70:0/3527447854 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f52a000fb70 con 0x5615fc163a00 2024-06-04T05:20:59.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.639+0000 7f5287fff640 1 -- 172.21.15.70:0/3527447854 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f52a0026540 con 0x5615fc163a00 2024-06-04T05:20:59.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.639+0000 7f52b17f0740 10 client.30128 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:59.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.639+0000 7f52b17f0740 10 client.30128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:59.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.639+0000 7f52b17f0740 20 client.30128 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:59.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.639+0000 7f52b17f0740 1 --2- 172.21.15.70:0/3527447854 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5615fc1737f0 0x5615fc175be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.639+0000 7f52b17f0740 4 client.30128 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:20:59.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.639+0000 7f52b17f0740 1 -- 172.21.15.70:0/3527447854 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615fc0ccb80 con 0x5615fc1737f0 2024-06-04T05:20:59.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.640+0000 7f52ac875640 1 --2- 172.21.15.70:0/3527447854 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5615fc1737f0 0x5615fc175be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.640+0000 7f52ac875640 1 --2- 172.21.15.70:0/3527447854 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5615fc1737f0 0x5615fc175be0 crc :-1 s=READY pgs=497 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=893b4aec027cc487 server_cookie=22538dbb2dae4f90 in_seq=0 out_seq=0 2024-06-04T05:20:59.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.640+0000 7f5287fff640 10 client.30128 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:59.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.641+0000 7f5287fff640 1 -- 172.21.15.70:0/3527447854 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5615fc0ccb80 con 0x5615fc1737f0 2024-06-04T05:20:59.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.641+0000 7f5287fff640 10 client.30128 handle_command_reply: tid=0 2024-06-04T05:20:59.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.641+0000 7f52b17f0740 1 client.30128 shutdown 2024-06-04T05:20:59.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.641+0000 7f52b17f0740 1 -- 172.21.15.70:0/3527447854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 msgr2=0x5615fc161470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:59.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.641+0000 7f52b17f0740 1 --2- 172.21.15.70:0/3527447854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 0x5615fc161470 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f52a001b460 tx=0x7f52a000ff50 comp rx=0 tx=0).stop 2024-06-04T05:20:59.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.641+0000 7f52b17f0740 1 -- 172.21.15.70:0/3527447854 shutdown_connections 2024-06-04T05:20:59.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.641+0000 7f52b17f0740 1 --2- 172.21.15.70:0/3527447854 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5615fc1737f0 0x5615fc175be0 crc :-1 s=READY pgs=497 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.641+0000 7f52b17f0740 1 --2- 172.21.15.70:0/3527447854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 0x5615fc161470 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.641+0000 7f5287fff640 0 client.30128 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:59.650 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.641+0000 7f52b17f0740 1 --2- 172.21.15.70:0/3527447854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 0x5615fc160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.650 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.641+0000 7f52b17f0740 1 --2- 172.21.15.70:0/3527447854 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 0x5615fc1609f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.650 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.641+0000 7f52b17f0740 1 -- 172.21.15.70:0/3527447854 >> 172.21.15.70:0/3527447854 conn(0x5615fc154d30 msgr2=0x5615fc153fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:59.650 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.641+0000 7f52b17f0740 1 -- 172.21.15.70:0/3527447854 shutdown_connections 2024-06-04T05:20:59.650 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.641+0000 7f52b17f0740 1 -- 172.21.15.70:0/3527447854 wait complete. 2024-06-04T05:20:59.650 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.641+0000 7f52b17f0740 20 client.30128 trim_cache size 0 max 16384 2024-06-04T05:20:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.663+0000 7f52b17f0740 1 Processor -- start 2024-06-04T05:20:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.663+0000 7f52b17f0740 1 -- start start 2024-06-04T05:20:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.663+0000 7f52b17f0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 0x5615fc17b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.663+0000 7f52b17f0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 0x5615fc17b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.663+0000 7f52b17f0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 0x5615fc15dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.663+0000 7f52b17f0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f52a81b0850 con 0x5615fc159140 2024-06-04T05:20:59.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.663+0000 7f52b17f0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f52a81b2420 con 0x5615fc163a00 2024-06-04T05:20:59.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.663+0000 7f52b17f0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f52a81b0f80 con 0x5615fc158820 2024-06-04T05:20:59.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.663+0000 7f52a7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 0x5615fc17b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.663+0000 7f52a7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 0x5615fc17b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38520/0 (socket says 172.21.15.70:38520) 2024-06-04T05:20:59.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.663+0000 7f52a7fff640 1 -- 172.21.15.70:0/924760621 learned_addr learned my addr 172.21.15.70:0/924760621 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:59.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.663+0000 7f52ac875640 1 --2- 172.21.15.70:0/924760621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 0x5615fc17b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.664+0000 7f52ad076640 1 --2- 172.21.15.70:0/924760621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 0x5615fc15dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.664+0000 7f52a7fff640 1 -- 172.21.15.70:0/924760621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 msgr2=0x5615fc17b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:59.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.664+0000 7f52a7fff640 1 --2- 172.21.15.70:0/924760621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 0x5615fc17b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.664+0000 7f52a7fff640 1 -- 172.21.15.70:0/924760621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 msgr2=0x5615fc15dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:59.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.664+0000 7f52a7fff640 1 --2- 172.21.15.70:0/924760621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 0x5615fc15dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.664+0000 7f52a7fff640 1 -- 172.21.15.70:0/924760621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615fc176170 con 0x5615fc159140 2024-06-04T05:20:59.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.664+0000 7f52ac875640 1 --2- 172.21.15.70:0/924760621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 0x5615fc17b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:59.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.664+0000 7f52a7fff640 1 --2- 172.21.15.70:0/924760621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 0x5615fc17b920 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f529402a930 tx=0x7f52940314b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c992975cc7d7757d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:59.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.664+0000 7f52927fc640 1 -- 172.21.15.70:0/924760621 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5294035040 con 0x5615fc159140 2024-06-04T05:20:59.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.664+0000 7f52927fc640 1 -- 172.21.15.70:0/924760621 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f52940044c0 con 0x5615fc159140 2024-06-04T05:20:59.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.664+0000 7f52927fc640 1 -- 172.21.15.70:0/924760621 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5294003bf0 con 0x5615fc159140 2024-06-04T05:20:59.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.665+0000 7f52b17f0740 1 -- 172.21.15.70:0/924760621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 msgr2=0x5615fc17b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:59.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.665+0000 7f52b17f0740 1 --2- 172.21.15.70:0/924760621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 0x5615fc17b920 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f529402a930 tx=0x7f52940314b0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.665+0000 7f52b17f0740 1 -- 172.21.15.70:0/924760621 shutdown_connections 2024-06-04T05:20:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.665+0000 7f52b17f0740 1 --2- 172.21.15.70:0/924760621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc163a00 0x5615fc15dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.665+0000 7f52b17f0740 1 --2- 172.21.15.70:0/924760621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc159140 0x5615fc17b920 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.665+0000 7f52b17f0740 1 --2- 172.21.15.70:0/924760621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc158820 0x5615fc17b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.665+0000 7f52b17f0740 1 -- 172.21.15.70:0/924760621 >> 172.21.15.70:0/924760621 conn(0x5615fc154d30 msgr2=0x5615fc15aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.665+0000 7f52b17f0740 1 -- 172.21.15.70:0/924760621 shutdown_connections 2024-06-04T05:20:59.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.666+0000 7f52b17f0740 1 -- 172.21.15.70:0/924760621 wait complete. 2024-06-04T05:20:59.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.666+0000 7f52b17f0740 1 -- start start 2024-06-04T05:20:59.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.666+0000 7f52b17f0740 1 Processor -- start 2024-06-04T05:20:59.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.666+0000 7f52b17f0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc158820 0x5615fc167d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.666+0000 7f52b17f0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc159140 0x5615fc1682a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f52b17f0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc163a00 0x5615fc1dd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f52ac875640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc158820 0x5615fc167d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f52ac875640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc158820 0x5615fc167d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38530/0 (socket says 172.21.15.70:38530) 2024-06-04T05:20:59.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f52ac875640 1 -- 172.21.15.70:0/2589773809 learned_addr learned my addr 172.21.15.70:0/2589773809 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:20:59.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f52a7fff640 1 --2- 172.21.15.70:0/2589773809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc159140 0x5615fc1682a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f52b17f0740 1 -- 172.21.15.70:0/2589773809 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f52a81b1d20 con 0x5615fc158820 2024-06-04T05:20:59.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f52b17f0740 1 -- 172.21.15.70:0/2589773809 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f52a81b1650 con 0x5615fc159140 2024-06-04T05:20:59.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f52b17f0740 1 -- 172.21.15.70:0/2589773809 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f52a81ae850 con 0x5615fc163a00 2024-06-04T05:20:59.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f52b17f0740 10 client.0 _ll_register_callbacks cb 0x7f529c0a9a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:20:59.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f52ad076640 1 --2- 172.21.15.70:0/2589773809 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc163a00 0x5615fc1dd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f52a7fff640 1 -- 172.21.15.70:0/2589773809 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc163a00 msgr2=0x5615fc1dd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:59.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f52a7fff640 1 --2- 172.21.15.70:0/2589773809 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc163a00 0x5615fc1dd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f52a7fff640 1 -- 172.21.15.70:0/2589773809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc158820 msgr2=0x5615fc167d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:20:59.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f52a7fff640 1 --2- 172.21.15.70:0/2589773809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc158820 0x5615fc167d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f52a7fff640 1 -- 172.21.15.70:0/2589773809 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615fc1dd870 con 0x5615fc159140 2024-06-04T05:20:59.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f52ac875640 1 --2- 172.21.15.70:0/2589773809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc158820 0x5615fc167d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:20:59.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f52a7fff640 1 --2- 172.21.15.70:0/2589773809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc159140 0x5615fc1682a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5294005940 tx=0x7f5294032df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dad34debe1f7bb2b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:20:59.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f5287fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:20:59.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.667+0000 7f5287fff640 1 -- 172.21.15.70:0/2589773809 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5294055060 con 0x5615fc159140 2024-06-04T05:20:59.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.668+0000 7f5287fff640 1 -- 172.21.15.70:0/2589773809 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8571+0+0 (secure 0 0 0) 0x7f5294043040 con 0x5615fc159140 2024-06-04T05:20:59.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.668+0000 7f52b17f0740 1 -- 172.21.15.70:0/2589773809 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5615fc0c6360 con 0x5615fc159140 2024-06-04T05:20:59.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.668+0000 7f5287fff640 1 -- 172.21.15.70:0/2589773809 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f5294035030 con 0x5615fc159140 2024-06-04T05:20:59.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.668+0000 7f52b17f0740 10 client.30134 fetch_fsmap learned FSMap version 64 2024-06-04T05:20:59.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.668+0000 7f52b17f0740 1 -- 172.21.15.70:0/2589773809 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x5615fc1deb40 con 0x5615fc159140 2024-06-04T05:20:59.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.668+0000 7f5287fff640 1 -- 172.21.15.70:0/2589773809 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x5615fc0c6360 con 0x5615fc159140 2024-06-04T05:20:59.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.668+0000 7f5287fff640 1 -- 172.21.15.70:0/2589773809 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f529404f070 con 0x5615fc159140 2024-06-04T05:20:59.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.668+0000 7f52b17f0740 10 client.30134 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:20:59.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.668+0000 7f52b17f0740 10 client.30134 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:20:59.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.668+0000 7f52b17f0740 20 client.30134 populate_metadata read hostname 'smithi070' 2024-06-04T05:20:59.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.668+0000 7f52b17f0740 1 --2- 172.21.15.70:0/2589773809 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5615fc1df320 0x5615fc1e1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:20:59.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.668+0000 7f52b17f0740 4 client.30134 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:20:59.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.668+0000 7f52b17f0740 1 -- 172.21.15.70:0/2589773809 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5615fc0405f0 con 0x5615fc1df320 2024-06-04T05:20:59.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.668+0000 7f52ac875640 1 --2- 172.21.15.70:0/2589773809 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5615fc1df320 0x5615fc1e1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:20:59.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.669+0000 7f5287fff640 10 client.30134 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:59.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.669+0000 7f52ac875640 1 --2- 172.21.15.70:0/2589773809 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5615fc1df320 0x5615fc1e1740 crc :-1 s=READY pgs=498 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40d5a4c19111ed89 server_cookie=cc0515f013b58923 in_seq=0 out_seq=0 2024-06-04T05:20:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:59 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:20:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:59 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:20:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:20:59 smithi112 ceph-mon[36453]: pgmap v749: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.7 MiB/s wr, 275 op/s 2024-06-04T05:20:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:59 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:20:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:59 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:20:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:20:59 smithi098 ceph-mon[35569]: pgmap v749: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.7 MiB/s wr, 275 op/s 2024-06-04T05:20:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.789+0000 7f5287fff640 1 -- 172.21.15.70:0/2589773809 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5615fc0405f0 con 0x5615fc1df320 2024-06-04T05:20:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.789+0000 7f5287fff640 10 client.30134 handle_command_reply: tid=0 2024-06-04T05:20:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.789+0000 7f52b17f0740 1 client.30134 shutdown 2024-06-04T05:20:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.790+0000 7f52b17f0740 1 -- 172.21.15.70:0/2589773809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc159140 msgr2=0x5615fc1682a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.790+0000 7f52b17f0740 1 --2- 172.21.15.70:0/2589773809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc159140 0x5615fc1682a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5294005940 tx=0x7f5294032df0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.790+0000 7f52b17f0740 1 -- 172.21.15.70:0/2589773809 shutdown_connections 2024-06-04T05:20:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.790+0000 7f52b17f0740 1 --2- 172.21.15.70:0/2589773809 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5615fc1df320 0x5615fc1e1740 crc :-1 s=READY pgs=498 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.790+0000 7f52b17f0740 1 --2- 172.21.15.70:0/2589773809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615fc159140 0x5615fc1682a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.790+0000 7f5287fff640 0 client.30134 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:20:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.790+0000 7f52b17f0740 1 --2- 172.21.15.70:0/2589773809 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5615fc163a00 0x5615fc1dd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.790+0000 7f52b17f0740 1 --2- 172.21.15.70:0/2589773809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615fc158820 0x5615fc167d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.790+0000 7f52b17f0740 1 -- 172.21.15.70:0/2589773809 >> 172.21.15.70:0/2589773809 conn(0x5615fc154d30 msgr2=0x5615fc160990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.790+0000 7f52b17f0740 1 -- 172.21.15.70:0/2589773809 shutdown_connections 2024-06-04T05:20:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:20:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:20:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:20:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.790+0000 7f52b17f0740 1 -- 172.21.15.70:0/2589773809 wait complete. 2024-06-04T05:20:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.790+0000 7f52b17f0740 20 client.30134 trim_cache size 0 max 16384 2024-06-04T05:20:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.791+0000 7f52aead9640 1 -- 172.21.15.70:0/287881533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52980b2a10 msgr2=0x7f52980b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.791+0000 7f52aead9640 1 --2- 172.21.15.70:0/287881533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52980b2a10 0x7f52980b4ed0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f52a0022000 tx=0x7f52a0023000 comp rx=0 tx=0).stop 2024-06-04T05:20:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.791+0000 7f52aead9640 1 -- 172.21.15.70:0/287881533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52a8101860 msgr2=0x7f52a819b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:20:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.791+0000 7f52aead9640 1 --2- 172.21.15.70:0/287881533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52a8101860 0x7f52a819b360 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f529c00e9f0 tx=0x7f529c00edf0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.791+0000 7f52aead9640 1 -- 172.21.15.70:0/287881533 shutdown_connections 2024-06-04T05:20:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.791+0000 7f52aead9640 1 --2- 172.21.15.70:0/287881533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f52980b2a10 0x7f52980b4ed0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.791+0000 7f52aead9640 1 --2- 172.21.15.70:0/287881533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52a810f410 0x7f52a819fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.791+0000 7f52aead9640 1 --2- 172.21.15.70:0/287881533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52a8102180 0x7f52a819b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.791+0000 7f52aead9640 1 --2- 172.21.15.70:0/287881533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52a8101860 0x7f52a819b360 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:20:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.791+0000 7f52aead9640 1 -- 172.21.15.70:0/287881533 >> 172.21.15.70:0/287881533 conn(0x7f52a8078ad0 msgr2=0x7f52a8110b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:20:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.791+0000 7f52aead9640 1 -- 172.21.15.70:0/287881533 shutdown_connections 2024-06-04T05:20:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:20:59.791+0000 7f52aead9640 1 -- 172.21.15.70:0/287881533 wait complete. 2024-06-04T05:20:59.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 76384 6.84 MB/sec execute 471 sec latency 92.136 ms 2024-06-04T05:20:59.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:59 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:20:59.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:59 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:20:59.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:20:59 smithi070 ceph-mon[32631]: pgmap v749: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.7 MiB/s wr, 275 op/s 2024-06-04T05:21:00.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 76518 6.84 MB/sec execute 472 sec latency 94.890 ms 2024-06-04T05:21:01.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:01 smithi112 ceph-mon[36453]: pgmap v750: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.7 MiB/s wr, 275 op/s 2024-06-04T05:21:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:01 smithi098 ceph-mon[35569]: pgmap v750: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.7 MiB/s wr, 275 op/s 2024-06-04T05:21:01.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 76701 6.86 MB/sec execute 473 sec latency 77.844 ms 2024-06-04T05:21:01.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:01 smithi070 ceph-mon[32631]: pgmap v750: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.7 MiB/s wr, 275 op/s 2024-06-04T05:21:02.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 76898 6.86 MB/sec execute 474 sec latency 55.542 ms 2024-06-04T05:21:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:03 smithi112 ceph-mon[36453]: pgmap v751: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.3 MiB/s wr, 450 op/s 2024-06-04T05:21:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:03 smithi098 ceph-mon[35569]: pgmap v751: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.3 MiB/s wr, 450 op/s 2024-06-04T05:21:03.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 77081 6.87 MB/sec execute 475 sec latency 163.299 ms 2024-06-04T05:21:03.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:03 smithi070 ceph-mon[32631]: pgmap v751: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.3 MiB/s wr, 450 op/s 2024-06-04T05:21:04.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:04 smithi112 ceph-mon[36453]: osdmap e115: 12 total, 12 up, 12 in 2024-06-04T05:21:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:04 smithi098 ceph-mon[35569]: osdmap e115: 12 total, 12 up, 12 in 2024-06-04T05:21:04.831 INFO:tasks.workunit.client.0.smithi070.stdout: 10 77257 6.87 MB/sec execute 476 sec latency 67.069 ms 2024-06-04T05:21:04.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:04 smithi070 ceph-mon[32631]: osdmap e115: 12 total, 12 up, 12 in 2024-06-04T05:21:05.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:05 smithi112 ceph-mon[36453]: pgmap v753: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.9 MiB/s wr, 364 op/s 2024-06-04T05:21:05.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:05 smithi098 ceph-mon[35569]: pgmap v753: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.9 MiB/s wr, 364 op/s 2024-06-04T05:21:05.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 77414 6.88 MB/sec execute 477 sec latency 70.382 ms 2024-06-04T05:21:05.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:05 smithi070 ceph-mon[32631]: pgmap v753: 97 pgs: 97 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.9 MiB/s wr, 364 op/s 2024-06-04T05:21:06.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 77599 6.88 MB/sec execute 478 sec latency 53.874 ms 2024-06-04T05:21:07.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:07 smithi112 ceph-mon[36453]: pgmap v754: 97 pgs: 97 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.4 MiB/s wr, 361 op/s 2024-06-04T05:21:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:07 smithi098 ceph-mon[35569]: pgmap v754: 97 pgs: 97 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.4 MiB/s wr, 361 op/s 2024-06-04T05:21:07.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 77781 6.90 MB/sec execute 479 sec latency 74.523 ms 2024-06-04T05:21:07.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:07 smithi070 ceph-mon[32631]: pgmap v754: 97 pgs: 97 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.4 MiB/s wr, 361 op/s 2024-06-04T05:21:08.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 77964 6.90 MB/sec execute 480 sec latency 73.464 ms 2024-06-04T05:21:09.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:09 smithi070 ceph-mon[32631]: osdmap e116: 12 total, 12 up, 12 in 2024-06-04T05:21:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:09 smithi070 ceph-mon[32631]: pgmap v756: 97 pgs: 97 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.0 MiB/s wr, 451 op/s 2024-06-04T05:21:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:09 smithi112 ceph-mon[36453]: osdmap e116: 12 total, 12 up, 12 in 2024-06-04T05:21:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:09 smithi112 ceph-mon[36453]: pgmap v756: 97 pgs: 97 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.0 MiB/s wr, 451 op/s 2024-06-04T05:21:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:09 smithi098 ceph-mon[35569]: osdmap e116: 12 total, 12 up, 12 in 2024-06-04T05:21:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:09 smithi098 ceph-mon[35569]: pgmap v756: 97 pgs: 97 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.0 MiB/s wr, 451 op/s 2024-06-04T05:21:09.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 78144 6.92 MB/sec execute 481 sec latency 83.208 ms 2024-06-04T05:21:10.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 78294 6.91 MB/sec execute 482 sec latency 73.650 ms 2024-06-04T05:21:11.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:11 smithi070 ceph-mon[32631]: pgmap v757: 97 pgs: 97 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 MiB/s wr, 189 op/s 2024-06-04T05:21:11.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:11 smithi112 ceph-mon[36453]: pgmap v757: 97 pgs: 97 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 MiB/s wr, 189 op/s 2024-06-04T05:21:11.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:11 smithi098 ceph-mon[35569]: pgmap v757: 97 pgs: 97 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 MiB/s wr, 189 op/s 2024-06-04T05:21:11.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 78465 6.91 MB/sec execute 483 sec latency 59.408 ms 2024-06-04T05:21:12.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:12 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:21:12.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:12 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:21:12.789 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:12 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:21:12.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 78609 6.91 MB/sec execute 484 sec latency 76.827 ms 2024-06-04T05:21:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.934+0000 7f2422ca0640 1 -- 172.21.15.70:0/4218256355 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f241c10f410 msgr2=0x7f241c111800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.934+0000 7f2422ca0640 1 --2- 172.21.15.70:0/4218256355 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f241c10f410 0x7f241c111800 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2414009ca0 tx=0x7f241402a8b0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.934+0000 7f2422ca0640 1 -- 172.21.15.70:0/4218256355 shutdown_connections 2024-06-04T05:21:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.934+0000 7f2422ca0640 1 --2- 172.21.15.70:0/4218256355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f241c102180 0x7f241c10eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.934+0000 7f2422ca0640 1 --2- 172.21.15.70:0/4218256355 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f241c10f410 0x7f241c111800 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.934+0000 7f2422ca0640 1 --2- 172.21.15.70:0/4218256355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c101860 0x7f241c101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.934+0000 7f2422ca0640 1 -- 172.21.15.70:0/4218256355 >> 172.21.15.70:0/4218256355 conn(0x7f241c078ad0 msgr2=0x7f241c078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.934+0000 7f2422ca0640 1 -- 172.21.15.70:0/4218256355 shutdown_connections 2024-06-04T05:21:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.934+0000 7f2422ca0640 1 -- 172.21.15.70:0/4218256355 wait complete. 2024-06-04T05:21:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.935+0000 7f2422ca0640 1 Processor -- start 2024-06-04T05:21:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.935+0000 7f2422ca0640 1 -- start start 2024-06-04T05:21:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.935+0000 7f2422ca0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c101860 0x7f241c19b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.936+0000 7f2422ca0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f241c102180 0x7f241c19b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.936+0000 7f2420a3c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c101860 0x7f241c19b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.936+0000 7f2420a3c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c101860 0x7f241c19b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58028/0 (socket says 172.21.15.70:58028) 2024-06-04T05:21:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.936+0000 7f2420a3c640 1 -- 172.21.15.70:0/4196624351 learned_addr learned my addr 172.21.15.70:0/4196624351 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.936+0000 7f2422ca0640 1 --2- 172.21.15.70:0/4196624351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f241c10f410 0x7f241c19fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.936+0000 7f241bfff640 1 --2- 172.21.15.70:0/4196624351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f241c102180 0x7f241c19b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.936+0000 7f242123d640 1 --2- 172.21.15.70:0/4196624351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f241c10f410 0x7f241c19fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.936+0000 7f2422ca0640 1 -- 172.21.15.70:0/4196624351 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f241c114da0 con 0x7f241c101860 2024-06-04T05:21:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.936+0000 7f2422ca0640 1 -- 172.21.15.70:0/4196624351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f241c114c20 con 0x7f241c102180 2024-06-04T05:21:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.936+0000 7f2422ca0640 1 -- 172.21.15.70:0/4196624351 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f241c114f20 con 0x7f241c10f410 2024-06-04T05:21:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.937+0000 7f242123d640 1 -- 172.21.15.70:0/4196624351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f241c102180 msgr2=0x7f241c19b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.937+0000 7f242123d640 1 --2- 172.21.15.70:0/4196624351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f241c102180 0x7f241c19b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.937+0000 7f242123d640 1 -- 172.21.15.70:0/4196624351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c101860 msgr2=0x7f241c19b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.937+0000 7f242123d640 1 --2- 172.21.15.70:0/4196624351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c101860 0x7f241c19b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.937+0000 7f242123d640 1 -- 172.21.15.70:0/4196624351 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f241c1a0210 con 0x7f241c10f410 2024-06-04T05:21:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.937+0000 7f2420a3c640 1 --2- 172.21.15.70:0/4196624351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c101860 0x7f241c19b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:21:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.937+0000 7f241bfff640 1 --2- 172.21.15.70:0/4196624351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f241c102180 0x7f241c19b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:21:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.937+0000 7f242123d640 1 --2- 172.21.15.70:0/4196624351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f241c10f410 0x7f241c19fae0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2414005e30 tx=0x7f2414030f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.938+0000 7f2419ffb640 1 -- 172.21.15.70:0/4196624351 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2414003d80 con 0x7f241c10f410 2024-06-04T05:21:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.938+0000 7f2419ffb640 1 -- 172.21.15.70:0/4196624351 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f241403f070 con 0x7f241c10f410 2024-06-04T05:21:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.938+0000 7f2419ffb640 1 -- 172.21.15.70:0/4196624351 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24140319b0 con 0x7f241c10f410 2024-06-04T05:21:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.938+0000 7f2422ca0640 1 -- 172.21.15.70:0/4196624351 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f241c1a04a0 con 0x7f241c10f410 2024-06-04T05:21:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.938+0000 7f2422ca0640 1 -- 172.21.15.70:0/4196624351 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f241c1a7d90 con 0x7f241c10f410 2024-06-04T05:21:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.941+0000 7f2419ffb640 1 -- 172.21.15.70:0/4196624351 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f24140045c0 con 0x7f241c10f410 2024-06-04T05:21:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.942+0000 7f24259c2740 1 Processor -- start 2024-06-04T05:21:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.942+0000 7f24259c2740 1 -- start start 2024-06-04T05:21:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.942+0000 7f24259c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b321820 0x55655b321c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.942+0000 7f24259c2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b322140 0x55655b32c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.942+0000 7f24259c2740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 0x55655b32edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.942+0000 7f24259c2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55655b293c30 con 0x55655b322140 2024-06-04T05:21:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.942+0000 7f24259c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55655b2837d0 con 0x55655b321820 2024-06-04T05:21:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.942+0000 7f24259c2740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55655b287810 con 0x55655b32ca00 2024-06-04T05:21:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.943+0000 7f2420a3c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b322140 0x55655b32c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.943+0000 7f2420a3c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b322140 0x55655b32c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58036/0 (socket says 172.21.15.70:58036) 2024-06-04T05:21:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.943+0000 7f2420a3c640 1 -- 172.21.15.70:0/1016188071 learned_addr learned my addr 172.21.15.70:0/1016188071 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.943+0000 7f241bfff640 1 --2- 172.21.15.70:0/1016188071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b321820 0x55655b321c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.943+0000 7f2419ffb640 1 --2- 172.21.15.70:0/4196624351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f24040b2a90 0x7f24040b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.943+0000 7f2420a3c640 1 -- 172.21.15.70:0/1016188071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 msgr2=0x55655b32edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.943+0000 7f241bfff640 1 --2- 172.21.15.70:0/1016188071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 0x55655b32edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.943+0000 7f2420a3c640 1 --2- 172.21.15.70:0/1016188071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 0x55655b32edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.943+0000 7f2420a3c640 1 -- 172.21.15.70:0/1016188071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b321820 msgr2=0x55655b321c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.943+0000 7f2420a3c640 1 --2- 172.21.15.70:0/1016188071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b321820 0x55655b321c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.943+0000 7f2420a3c640 1 -- 172.21.15.70:0/1016188071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55655af98210 con 0x55655b322140 2024-06-04T05:21:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.943+0000 7f2419ffb640 1 -- 172.21.15.70:0/4196624351 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f2414103250 con 0x7f241c10f410 2024-06-04T05:21:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.944+0000 7f2420a3c640 1 --2- 172.21.15.70:0/1016188071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b322140 0x55655b32c4c0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f2410009fd0 tx=0x7f2410011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d428b0d4286b63 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.944+0000 7f241bfff640 1 --2- 172.21.15.70:0/1016188071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 0x55655b32edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:21:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.944+0000 7f242123d640 1 --2- 172.21.15.70:0/4196624351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f24040b2a90 0x7f24040b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.945+0000 7f242123d640 1 --2- 172.21.15.70:0/4196624351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f24040b2a90 0x7f24040b4f50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f24140c6690 tx=0x7f24140095c0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.946+0000 7f2402ffd640 1 -- 172.21.15.70:0/1016188071 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241001e070 con 0x55655b322140 2024-06-04T05:21:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.946+0000 7f2402ffd640 1 -- 172.21.15.70:0/1016188071 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f241000fb80 con 0x55655b322140 2024-06-04T05:21:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.946+0000 7f2402ffd640 1 -- 172.21.15.70:0/1016188071 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2410019c70 con 0x55655b322140 2024-06-04T05:21:12.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.947+0000 7f24259c2740 1 -- 172.21.15.70:0/1016188071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b322140 msgr2=0x55655b32c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:12.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.947+0000 7f24259c2740 1 --2- 172.21.15.70:0/1016188071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b322140 0x55655b32c4c0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f2410009fd0 tx=0x7f2410011ae0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.947+0000 7f24259c2740 1 -- 172.21.15.70:0/1016188071 shutdown_connections 2024-06-04T05:21:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.947+0000 7f24259c2740 1 --2- 172.21.15.70:0/1016188071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 0x55655b32edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.947+0000 7f24259c2740 1 --2- 172.21.15.70:0/1016188071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b322140 0x55655b32c4c0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.947+0000 7f24259c2740 1 --2- 172.21.15.70:0/1016188071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b321820 0x55655b321c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.947+0000 7f24259c2740 1 -- 172.21.15.70:0/1016188071 >> 172.21.15.70:0/1016188071 conn(0x55655b31dd30 msgr2=0x55655b31e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.947+0000 7f24259c2740 1 -- 172.21.15.70:0/1016188071 shutdown_connections 2024-06-04T05:21:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.947+0000 7f24259c2740 1 -- 172.21.15.70:0/1016188071 wait complete. 2024-06-04T05:21:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.947+0000 7f24259c2740 1 -- start start 2024-06-04T05:21:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.947+0000 7f24259c2740 1 Processor -- start 2024-06-04T05:21:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f24259c2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b321820 0x55655b329a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f24259c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b322140 0x55655b329f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f24259c2740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 0x55655b32a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f24259c2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55655af69790 con 0x55655b321820 2024-06-04T05:21:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f24259c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55655b23f890 con 0x55655b322140 2024-06-04T05:21:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f24259c2740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55655b32f630 con 0x55655b32ca00 2024-06-04T05:21:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f24259c2740 10 client.0 _ll_register_callbacks cb 0x55655af97a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f2420a3c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b322140 0x55655b329f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f2420a3c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b322140 0x55655b329f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55918/0 (socket says 172.21.15.70:55918) 2024-06-04T05:21:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f2420a3c640 1 -- 172.21.15.70:0/1520085028 learned_addr learned my addr 172.21.15.70:0/1520085028 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f241bfff640 1 --2- 172.21.15.70:0/1520085028 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 0x55655b32a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f241bfff640 1 --2- 172.21.15.70:0/1520085028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b321820 0x55655b329a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f2420a3c640 1 -- 172.21.15.70:0/1520085028 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 msgr2=0x55655b32a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f2420a3c640 1 --2- 172.21.15.70:0/1520085028 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 0x55655b32a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f2420a3c640 1 -- 172.21.15.70:0/1520085028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b321820 msgr2=0x55655b329a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f2420a3c640 1 --2- 172.21.15.70:0/1520085028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b321820 0x55655b329a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f2420a3c640 1 -- 172.21.15.70:0/1520085028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55655b064620 con 0x55655b322140 2024-06-04T05:21:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.948+0000 7f241bfff640 1 --2- 172.21.15.70:0/1520085028 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 0x55655b32a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:21:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.949+0000 7f2420a3c640 1 --2- 172.21.15.70:0/1520085028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b322140 0x55655b329f80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2410019850 tx=0x7f241000f6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5ec3a27d8100c3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.949+0000 7f23fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:21:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.949+0000 7f23fbfff640 1 -- 172.21.15.70:0/1520085028 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241001e040 con 0x55655b322140 2024-06-04T05:21:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.949+0000 7f24259c2740 1 -- 172.21.15.70:0/1520085028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f241000fb80 con 0x55655b322140 2024-06-04T05:21:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.949+0000 7f23fbfff640 1 -- 172.21.15.70:0/1520085028 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f241000f810 con 0x55655b322140 2024-06-04T05:21:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.949+0000 7f23fbfff640 1 -- 172.21.15.70:0/1520085028 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f24100199b0 con 0x55655b322140 2024-06-04T05:21:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.949+0000 7f24259c2740 10 client.30143 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.949+0000 7f24259c2740 1 -- 172.21.15.70:0/1520085028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55655b343c00 con 0x55655b322140 2024-06-04T05:21:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.949+0000 7f23fbfff640 1 -- 172.21.15.70:0/1520085028 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f241000fb80 con 0x55655b322140 2024-06-04T05:21:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.950+0000 7f23fbfff640 1 -- 172.21.15.70:0/1520085028 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f2410021070 con 0x55655b322140 2024-06-04T05:21:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.950+0000 7f24259c2740 10 client.30143 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.950+0000 7f24259c2740 10 client.30143 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:21:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.950+0000 7f24259c2740 20 client.30143 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.950+0000 7f24259c2740 1 --2- 172.21.15.70:0/1520085028 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55655b344fb0 0x55655b3473a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.950+0000 7f24259c2740 4 client.30143 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:21:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.950+0000 7f24259c2740 1 -- 172.21.15.70:0/1520085028 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55655b295b80 con 0x55655b344fb0 2024-06-04T05:21:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.950+0000 7f241bfff640 1 --2- 172.21.15.70:0/1520085028 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55655b344fb0 0x55655b3473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.950+0000 7f23fbfff640 10 client.30143 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.950+0000 7f241bfff640 1 --2- 172.21.15.70:0/1520085028 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55655b344fb0 0x55655b3473a0 crc :-1 s=READY pgs=499 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44fc17c1f1087170 server_cookie=b1de3b80a2b2b848 in_seq=0 out_seq=0 2024-06-04T05:21:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.951+0000 7f23fbfff640 1 -- 172.21.15.70:0/1520085028 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55655b295b80 con 0x55655b344fb0 2024-06-04T05:21:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.951+0000 7f23fbfff640 10 client.30143 handle_command_reply: tid=0 2024-06-04T05:21:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.951+0000 7f24259c2740 1 client.30143 shutdown 2024-06-04T05:21:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.951+0000 7f24259c2740 1 -- 172.21.15.70:0/1520085028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b322140 msgr2=0x55655b329f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.951+0000 7f24259c2740 1 --2- 172.21.15.70:0/1520085028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b322140 0x55655b329f80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2410019850 tx=0x7f241000f6f0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.951+0000 7f24259c2740 1 -- 172.21.15.70:0/1520085028 shutdown_connections 2024-06-04T05:21:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.951+0000 7f24259c2740 1 --2- 172.21.15.70:0/1520085028 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55655b344fb0 0x55655b3473a0 crc :-1 s=READY pgs=499 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.951+0000 7f24259c2740 1 --2- 172.21.15.70:0/1520085028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b322140 0x55655b329f80 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.951+0000 7f24259c2740 1 --2- 172.21.15.70:0/1520085028 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 0x55655b32a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.951+0000 7f24259c2740 1 --2- 172.21.15.70:0/1520085028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b321820 0x55655b329a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.951+0000 7f24259c2740 1 -- 172.21.15.70:0/1520085028 >> 172.21.15.70:0/1520085028 conn(0x55655b31dd30 msgr2=0x55655b31cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.951+0000 7f23fbfff640 0 client.30143 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.952+0000 7f24259c2740 1 -- 172.21.15.70:0/1520085028 shutdown_connections 2024-06-04T05:21:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.952+0000 7f24259c2740 1 -- 172.21.15.70:0/1520085028 wait complete. 2024-06-04T05:21:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.953+0000 7f24259c2740 20 client.30143 trim_cache size 0 max 16384 2024-06-04T05:21:12.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.981+0000 7f24259c2740 1 Processor -- start 2024-06-04T05:21:12.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.981+0000 7f24259c2740 1 -- start start 2024-06-04T05:21:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.981+0000 7f24259c2740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b321820 0x55655b31ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.981+0000 7f24259c2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b322140 0x55655b34cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.981+0000 7f24259c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b32ca00 0x55655b34d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.981+0000 7f241bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b321820 0x55655b31ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.982+0000 7f241bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b321820 0x55655b31ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:37638/0 (socket says 172.21.15.70:37638) 2024-06-04T05:21:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.982+0000 7f241bfff640 1 -- 172.21.15.70:0/2445383198 learned_addr learned my addr 172.21.15.70:0/2445383198 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.982+0000 7f2420a3c640 1 --2- 172.21.15.70:0/2445383198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b322140 0x55655b34cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.982+0000 7f24259c2740 1 -- 172.21.15.70:0/2445383198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f241c1ac3d0 con 0x55655b322140 2024-06-04T05:21:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.982+0000 7f24259c2740 1 -- 172.21.15.70:0/2445383198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f241c1adfa0 con 0x55655b32ca00 2024-06-04T05:21:12.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.982+0000 7f24259c2740 1 -- 172.21.15.70:0/2445383198 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f241c1acb00 con 0x55655b321820 2024-06-04T05:21:12.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.982+0000 7f241bfff640 1 --2- 172.21.15.70:0/2445383198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b32ca00 0x55655b34d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.982+0000 7f2420a3c640 1 -- 172.21.15.70:0/2445383198 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b321820 msgr2=0x55655b31ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:12.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.982+0000 7f2420a3c640 1 --2- 172.21.15.70:0/2445383198 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b321820 0x55655b31ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.982+0000 7f2420a3c640 1 -- 172.21.15.70:0/2445383198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b32ca00 msgr2=0x55655b34d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:12.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.982+0000 7f2420a3c640 1 --2- 172.21.15.70:0/2445383198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b32ca00 0x55655b34d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.982+0000 7f2420a3c640 1 -- 172.21.15.70:0/2445383198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55655b3196f0 con 0x55655b322140 2024-06-04T05:21:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.983+0000 7f2420a3c640 1 --2- 172.21.15.70:0/2445383198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b322140 0x55655b34cb50 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f2410004790 tx=0x7f2410005990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cacc7a8f41ef9fb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.983+0000 7f24027fc640 1 -- 172.21.15.70:0/2445383198 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2410015020 con 0x55655b322140 2024-06-04T05:21:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.983+0000 7f24027fc640 1 -- 172.21.15.70:0/2445383198 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2410004980 con 0x55655b322140 2024-06-04T05:21:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.983+0000 7f24027fc640 1 -- 172.21.15.70:0/2445383198 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241001e030 con 0x55655b322140 2024-06-04T05:21:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.984+0000 7f24259c2740 1 -- 172.21.15.70:0/2445383198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b322140 msgr2=0x55655b34cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.984+0000 7f24259c2740 1 --2- 172.21.15.70:0/2445383198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b322140 0x55655b34cb50 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f2410004790 tx=0x7f2410005990 comp rx=0 tx=0).stop 2024-06-04T05:21:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.984+0000 7f24259c2740 1 -- 172.21.15.70:0/2445383198 shutdown_connections 2024-06-04T05:21:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.984+0000 7f24259c2740 1 --2- 172.21.15.70:0/2445383198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b32ca00 0x55655b34d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.984+0000 7f24259c2740 1 --2- 172.21.15.70:0/2445383198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b322140 0x55655b34cb50 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.984+0000 7f24259c2740 1 --2- 172.21.15.70:0/2445383198 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b321820 0x55655b31ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.984+0000 7f24259c2740 1 -- 172.21.15.70:0/2445383198 >> 172.21.15.70:0/2445383198 conn(0x55655b31dd30 msgr2=0x55655b31c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.984+0000 7f24259c2740 1 -- 172.21.15.70:0/2445383198 shutdown_connections 2024-06-04T05:21:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.984+0000 7f24259c2740 1 -- 172.21.15.70:0/2445383198 wait complete. 2024-06-04T05:21:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.985+0000 7f24259c2740 1 -- start start 2024-06-04T05:21:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.985+0000 7f24259c2740 1 Processor -- start 2024-06-04T05:21:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.985+0000 7f24259c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b321820 0x55655b33d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.985+0000 7f24259c2740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 0x55655b33d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.985+0000 7f24259c2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b33a0e0 0x55655b33a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.985+0000 7f24259c2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f241c1ad8a0 con 0x55655b33a0e0 2024-06-04T05:21:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.985+0000 7f24259c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f241c1ad1d0 con 0x55655b321820 2024-06-04T05:21:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.985+0000 7f24259c2740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f241c1aa3f0 con 0x55655b32ca00 2024-06-04T05:21:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.985+0000 7f2420a3c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 0x55655b33d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.985+0000 7f241bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b321820 0x55655b33d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.985+0000 7f2420a3c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 0x55655b33d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:37642/0 (socket says 172.21.15.70:37642) 2024-06-04T05:21:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.985+0000 7f2420a3c640 1 -- 172.21.15.70:0/1463464593 learned_addr learned my addr 172.21.15.70:0/1463464593 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.985+0000 7f241bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b321820 0x55655b33d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55934/0 (socket says 172.21.15.70:55934) 2024-06-04T05:21:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.985+0000 7f24259c2740 10 client.0 _ll_register_callbacks cb 0x7f241001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.985+0000 7f241bfff640 1 --2- 172.21.15.70:0/1463464593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b33a0e0 0x55655b33a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.986+0000 7f2420a3c640 1 -- 172.21.15.70:0/1463464593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b321820 msgr2=0x55655b33d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.986+0000 7f2420a3c640 1 --2- 172.21.15.70:0/1463464593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b321820 0x55655b33d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.986+0000 7f2420a3c640 1 -- 172.21.15.70:0/1463464593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b33a0e0 msgr2=0x55655b33a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.986+0000 7f2420a3c640 1 --2- 172.21.15.70:0/1463464593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b33a0e0 0x55655b33a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.986+0000 7f2420a3c640 1 -- 172.21.15.70:0/1463464593 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55655b33aaa0 con 0x55655b32ca00 2024-06-04T05:21:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.986+0000 7f241bfff640 1 --2- 172.21.15.70:0/1463464593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b33a0e0 0x55655b33a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:21:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.986+0000 7f2420a3c640 1 --2- 172.21.15.70:0/1463464593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 0x55655b33d650 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2410004880 tx=0x7f2410005d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49fd1fc56a6e9215 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.986+0000 7f23fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:21:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.986+0000 7f23fbfff640 1 -- 172.21.15.70:0/1463464593 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241001e070 con 0x55655b32ca00 2024-06-04T05:21:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.986+0000 7f24259c2740 1 -- 172.21.15.70:0/1463464593 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2410004980 con 0x55655b32ca00 2024-06-04T05:21:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.986+0000 7f23fbfff640 1 -- 172.21.15.70:0/1463464593 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f2410004b30 con 0x55655b32ca00 2024-06-04T05:21:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.986+0000 7f23fbfff640 1 -- 172.21.15.70:0/1463464593 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f2410019b50 con 0x55655b32ca00 2024-06-04T05:21:12.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.986+0000 7f24259c2740 10 client.30286 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:12.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.986+0000 7f24259c2740 1 -- 172.21.15.70:0/1463464593 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55655b3b05c0 con 0x55655b32ca00 2024-06-04T05:21:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.987+0000 7f23fbfff640 1 -- 172.21.15.70:0/1463464593 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2410004980 con 0x55655b32ca00 2024-06-04T05:21:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.987+0000 7f23fbfff640 1 -- 172.21.15.70:0/1463464593 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f2410021040 con 0x55655b32ca00 2024-06-04T05:21:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.987+0000 7f24259c2740 10 client.30286 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.987+0000 7f24259c2740 10 client.30286 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:21:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.987+0000 7f24259c2740 20 client.30286 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.987+0000 7f24259c2740 1 --2- 172.21.15.70:0/1463464593 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55655b3b0bc0 0x55655b3b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.987+0000 7f24259c2740 4 client.30286 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:21:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.987+0000 7f24259c2740 1 -- 172.21.15.70:0/1463464593 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55655b28f360 con 0x55655b3b0bc0 2024-06-04T05:21:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.987+0000 7f241bfff640 1 --2- 172.21.15.70:0/1463464593 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55655b3b0bc0 0x55655b3b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.987+0000 7f241bfff640 1 --2- 172.21.15.70:0/1463464593 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55655b3b0bc0 0x55655b3b2fe0 crc :-1 s=READY pgs=500 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4384415331a9a6d8 server_cookie=d77ae1d72b19fc50 in_seq=0 out_seq=0 2024-06-04T05:21:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:12.987+0000 7f23fbfff640 10 client.30286 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.099+0000 7f23fbfff640 1 -- 172.21.15.70:0/1463464593 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55655b28f360 con 0x55655b3b0bc0 2024-06-04T05:21:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.099+0000 7f23fbfff640 10 client.30286 handle_command_reply: tid=0 2024-06-04T05:21:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.099+0000 7f24259c2740 1 client.30286 shutdown 2024-06-04T05:21:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.099+0000 7f24259c2740 1 -- 172.21.15.70:0/1463464593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 msgr2=0x55655b33d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.099+0000 7f24259c2740 1 --2- 172.21.15.70:0/1463464593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 0x55655b33d650 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2410004880 tx=0x7f2410005d80 comp rx=0 tx=0).stop 2024-06-04T05:21:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.100+0000 7f24259c2740 1 -- 172.21.15.70:0/1463464593 shutdown_connections 2024-06-04T05:21:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.100+0000 7f24259c2740 1 --2- 172.21.15.70:0/1463464593 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55655b3b0bc0 0x55655b3b2fe0 crc :-1 s=READY pgs=500 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.100+0000 7f24259c2740 1 --2- 172.21.15.70:0/1463464593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55655b33a0e0 0x55655b33a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.100+0000 7f24259c2740 1 --2- 172.21.15.70:0/1463464593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55655b32ca00 0x55655b33d650 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.100+0000 7f24259c2740 1 --2- 172.21.15.70:0/1463464593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55655b321820 0x55655b33d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.100+0000 7f24259c2740 1 -- 172.21.15.70:0/1463464593 >> 172.21.15.70:0/1463464593 conn(0x55655b31dd30 msgr2=0x55655b31cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.100+0000 7f23fbfff640 0 client.30286 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.100+0000 7f24259c2740 1 -- 172.21.15.70:0/1463464593 shutdown_connections 2024-06-04T05:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.100+0000 7f24259c2740 1 -- 172.21.15.70:0/1463464593 wait complete. 2024-06-04T05:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.100+0000 7f24259c2740 20 client.30286 trim_cache size 0 max 16384 2024-06-04T05:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:21:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:21:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:21:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.101+0000 7f2422ca0640 1 -- 172.21.15.70:0/4196624351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f24040b2a90 msgr2=0x7f24040b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.101+0000 7f2422ca0640 1 --2- 172.21.15.70:0/4196624351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f24040b2a90 0x7f24040b4f50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f24140c6690 tx=0x7f24140095c0 comp rx=0 tx=0).stop 2024-06-04T05:21:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.101+0000 7f2422ca0640 1 -- 172.21.15.70:0/4196624351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f241c10f410 msgr2=0x7f241c19fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.101+0000 7f2422ca0640 1 --2- 172.21.15.70:0/4196624351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f241c10f410 0x7f241c19fae0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2414005e30 tx=0x7f2414030f90 comp rx=0 tx=0).stop 2024-06-04T05:21:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.101+0000 7f2422ca0640 1 -- 172.21.15.70:0/4196624351 shutdown_connections 2024-06-04T05:21:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.101+0000 7f2422ca0640 1 --2- 172.21.15.70:0/4196624351 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f24040b2a90 0x7f24040b4f50 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.101+0000 7f2422ca0640 1 --2- 172.21.15.70:0/4196624351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f241c102180 0x7f241c19b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.101+0000 7f2422ca0640 1 --2- 172.21.15.70:0/4196624351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f241c10f410 0x7f241c19fae0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.101+0000 7f2422ca0640 1 --2- 172.21.15.70:0/4196624351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c101860 0x7f241c19b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.101+0000 7f2422ca0640 1 -- 172.21.15.70:0/4196624351 >> 172.21.15.70:0/4196624351 conn(0x7f241c078ad0 msgr2=0x7f241c110840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.101+0000 7f2422ca0640 1 -- 172.21.15.70:0/4196624351 shutdown_connections 2024-06-04T05:21:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:13.101+0000 7f2422ca0640 1 -- 172.21.15.70:0/4196624351 wait complete. 2024-06-04T05:21:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:13 smithi112 ceph-mon[36453]: pgmap v758: 97 pgs: 97 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.5 MiB/s wr, 343 op/s 2024-06-04T05:21:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:13 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:21:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:13 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:21:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:13 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:21:13.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:13 smithi098 ceph-mon[35569]: pgmap v758: 97 pgs: 97 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.5 MiB/s wr, 343 op/s 2024-06-04T05:21:13.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:13 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:21:13.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:13 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:21:13.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:13 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:21:13.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 78753 6.92 MB/sec execute 485 sec latency 103.942 ms 2024-06-04T05:21:13.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:13 smithi070 ceph-mon[32631]: pgmap v758: 97 pgs: 97 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.5 MiB/s wr, 343 op/s 2024-06-04T05:21:13.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:13 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:21:13.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:13 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:21:13.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:13 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:21:14.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 78927 6.92 MB/sec execute 486 sec latency 53.755 ms 2024-06-04T05:21:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:15 smithi112 ceph-mon[36453]: pgmap v759: 97 pgs: 97 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.7 MiB/s wr, 304 op/s 2024-06-04T05:21:15.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:15 smithi098 ceph-mon[35569]: pgmap v759: 97 pgs: 97 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.7 MiB/s wr, 304 op/s 2024-06-04T05:21:15.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 79083 6.92 MB/sec execute 487 sec latency 91.882 ms 2024-06-04T05:21:15.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:15 smithi070 ceph-mon[32631]: pgmap v759: 97 pgs: 97 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.7 MiB/s wr, 304 op/s 2024-06-04T05:21:16.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 79280 6.92 MB/sec execute 488 sec latency 82.793 ms 2024-06-04T05:21:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:17 smithi112 ceph-mon[36453]: pgmap v760: 97 pgs: 97 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.3 MiB/s wr, 357 op/s 2024-06-04T05:21:17.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:17 smithi098 ceph-mon[35569]: pgmap v760: 97 pgs: 97 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.3 MiB/s wr, 357 op/s 2024-06-04T05:21:17.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 79507 6.94 MB/sec execute 489 sec latency 79.225 ms 2024-06-04T05:21:17.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:17 smithi070 ceph-mon[32631]: pgmap v760: 97 pgs: 97 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.3 MiB/s wr, 357 op/s 2024-06-04T05:21:18.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 79683 6.94 MB/sec execute 490 sec latency 109.319 ms 2024-06-04T05:21:19.146 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-06-04T05:21:19.146 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-06-04T05:21:19.147 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:21:19.505 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:21:19.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:19 smithi070 ceph-mon[32631]: pgmap v761: 97 pgs: 97 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.2 MiB/s wr, 348 op/s 2024-06-04T05:21:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:19 smithi112 ceph-mon[36453]: pgmap v761: 97 pgs: 97 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.2 MiB/s wr, 348 op/s 2024-06-04T05:21:19.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:19 smithi098 ceph-mon[35569]: pgmap v761: 97 pgs: 97 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.2 MiB/s wr, 348 op/s 2024-06-04T05:21:19.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 79836 6.95 MB/sec execute 491 sec latency 58.265 ms 2024-06-04T05:21:20.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 80009 6.97 MB/sec execute 492 sec latency 85.785 ms 2024-06-04T05:21:21.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.512+0000 7f58fca10700 1 -- 172.21.15.70:0/3433345886 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58f8108520 msgr2=0x7f58f8108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:21.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.512+0000 7f58fca10700 1 --2- 172.21.15.70:0/3433345886 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58f8108520 0x7f58f8108900 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f58e80098f0 tx=0x7f58e8009c00 comp rx=0 tx=0).stop 2024-06-04T05:21:21.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.513+0000 7f58fca10700 1 -- 172.21.15.70:0/3433345886 shutdown_connections 2024-06-04T05:21:21.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.513+0000 7f58fca10700 1 --2- 172.21.15.70:0/3433345886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f8102e00 0x7f58f810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:21.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.513+0000 7f58fca10700 1 --2- 172.21.15.70:0/3433345886 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58f8108520 0x7f58f8108900 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:21.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.513+0000 7f58fca10700 1 --2- 172.21.15.70:0/3433345886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58f8102440 0x7f58f81028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:21.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.513+0000 7f58fca10700 1 -- 172.21.15.70:0/3433345886 >> 172.21.15.70:0/3433345886 conn(0x7f58f80793f0 msgr2=0x7f58f8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:21.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.513+0000 7f58fca10700 1 -- 172.21.15.70:0/3433345886 shutdown_connections 2024-06-04T05:21:21.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.513+0000 7f58fca10700 1 -- 172.21.15.70:0/3433345886 wait complete. 2024-06-04T05:21:21.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.514+0000 7f58fca10700 1 Processor -- start 2024-06-04T05:21:21.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.514+0000 7f58fca10700 1 -- start start 2024-06-04T05:21:21.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.514+0000 7f58fca10700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f8102440 0x7f58f810f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:21.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.514+0000 7f58fca10700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58f8102e00 0x7f58f810c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:21.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.514+0000 7f58fca10700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58f8108520 0x7f58f810c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:21.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.514+0000 7f58fca10700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f58f8111400 con 0x7f58f8108520 2024-06-04T05:21:21.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.514+0000 7f58fca10700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f58f8111280 con 0x7f58f8102440 2024-06-04T05:21:21.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.514+0000 7f58fca10700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f58f8111580 con 0x7f58f8102e00 2024-06-04T05:21:21.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.514+0000 7f58f6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58f8108520 0x7f58f810c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:21.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.514+0000 7f58f6d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58f8108520 0x7f58f810c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50510/0 (socket says 172.21.15.70:50510) 2024-06-04T05:21:21.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.514+0000 7f58f6d9d700 1 -- 172.21.15.70:0/1856263154 learned_addr learned my addr 172.21.15.70:0/1856263154 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:21.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.514+0000 7f58f659c700 1 --2- 172.21.15.70:0/1856263154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f8102440 0x7f58f810f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.514+0000 7f58f5d9b700 1 --2- 172.21.15.70:0/1856263154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58f8102e00 0x7f58f810c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.515+0000 7f58f6d9d700 1 -- 172.21.15.70:0/1856263154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58f8102e00 msgr2=0x7f58f810c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.515+0000 7f58f6d9d700 1 --2- 172.21.15.70:0/1856263154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58f8102e00 0x7f58f810c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.515+0000 7f58f6d9d700 1 -- 172.21.15.70:0/1856263154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f8102440 msgr2=0x7f58f810f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.515+0000 7f58f6d9d700 1 --2- 172.21.15.70:0/1856263154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f8102440 0x7f58f810f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.515+0000 7f58f6d9d700 1 -- 172.21.15.70:0/1856263154 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f810cdb0 con 0x7f58f8108520 2024-06-04T05:21:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.515+0000 7f58f659c700 1 --2- 172.21.15.70:0/1856263154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f8102440 0x7f58f810f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:21:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.515+0000 7f58f6d9d700 1 --2- 172.21.15.70:0/1856263154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58f8108520 0x7f58f810c5f0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f58f0007ee0 tx=0x7f58f000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.515+0000 7f58e77fe700 1 -- 172.21.15.70:0/1856263154 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0009e20 con 0x7f58f8108520 2024-06-04T05:21:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.515+0000 7f58fca10700 1 -- 172.21.15.70:0/1856263154 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58f81ac850 con 0x7f58f8108520 2024-06-04T05:21:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.515+0000 7f58e77fe700 1 -- 172.21.15.70:0/1856263154 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f58f000dbd0 con 0x7f58f8108520 2024-06-04T05:21:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.515+0000 7f58e77fe700 1 -- 172.21.15.70:0/1856263154 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0005810 con 0x7f58f8108520 2024-06-04T05:21:21.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.515+0000 7f58fca10700 1 -- 172.21.15.70:0/1856263154 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58f81acde0 con 0x7f58f8108520 2024-06-04T05:21:21.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.516+0000 7f58fca10700 1 -- 172.21.15.70:0/1856263154 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58d8005330 con 0x7f58f8108520 2024-06-04T05:21:21.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.518+0000 7f58e77fe700 1 -- 172.21.15.70:0/1856263154 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f58f001a020 con 0x7f58f8108520 2024-06-04T05:21:21.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.519+0000 7f58e77fe700 1 --2- 172.21.15.70:0/1856263154 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f58e00b2af0 0x7f58e00b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:21.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.519+0000 7f58e77fe700 1 -- 172.21.15.70:0/1856263154 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f58f00e1ba0 con 0x7f58f8108520 2024-06-04T05:21:21.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.519+0000 7f58f659c700 1 --2- 172.21.15.70:0/1856263154 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f58e00b2af0 0x7f58e00b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:21.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.520+0000 7f58f659c700 1 --2- 172.21.15.70:0/1856263154 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f58e00b2af0 0x7f58e00b4fb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f58e80098c0 tx=0x7f58e8019040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:21.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.525+0000 7f58e77fe700 1 -- 172.21.15.70:0/1856263154 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f58f0010070 con 0x7f58f8108520 2024-06-04T05:21:21.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:21 smithi070 ceph-mon[32631]: pgmap v762: 97 pgs: 97 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.3 MiB/s wr, 297 op/s 2024-06-04T05:21:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:21 smithi112 ceph-mon[36453]: pgmap v762: 97 pgs: 97 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.3 MiB/s wr, 297 op/s 2024-06-04T05:21:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:21 smithi098 ceph-mon[35569]: pgmap v762: 97 pgs: 97 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.3 MiB/s wr, 297 op/s 2024-06-04T05:21:21.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 80189 6.97 MB/sec execute 493 sec latency 94.017 ms 2024-06-04T05:21:21.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.965+0000 7f58fca10700 1 -- 172.21.15.70:0/1856263154 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58d80058f0 con 0x7f58f8108520 2024-06-04T05:21:21.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.966+0000 7f58e77fe700 1 -- 172.21.15.70:0/1856263154 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+11933 (secure 0 0 0) 0x7f58f00a94f0 con 0x7f58f8108520 2024-06-04T05:21:21.968 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:21:21.969 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":64,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:20:02.842346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:21:21.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.969+0000 7f58fca10700 1 -- 172.21.15.70:0/1856263154 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f58e00b2af0 msgr2=0x7f58e00b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:21.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.969+0000 7f58fca10700 1 --2- 172.21.15.70:0/1856263154 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f58e00b2af0 0x7f58e00b4fb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f58e80098c0 tx=0x7f58e8019040 comp rx=0 tx=0).stop 2024-06-04T05:21:21.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.969+0000 7f58fca10700 1 -- 172.21.15.70:0/1856263154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58f8108520 msgr2=0x7f58f810c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:21.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.969+0000 7f58fca10700 1 --2- 172.21.15.70:0/1856263154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58f8108520 0x7f58f810c5f0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f58f0007ee0 tx=0x7f58f000d3b0 comp rx=0 tx=0).stop 2024-06-04T05:21:21.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.970+0000 7f58fca10700 1 -- 172.21.15.70:0/1856263154 shutdown_connections 2024-06-04T05:21:21.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.970+0000 7f58fca10700 1 --2- 172.21.15.70:0/1856263154 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f58e00b2af0 0x7f58e00b4fb0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:21.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.970+0000 7f58fca10700 1 --2- 172.21.15.70:0/1856263154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58f8108520 0x7f58f810c5f0 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:21.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.970+0000 7f58fca10700 1 --2- 172.21.15.70:0/1856263154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f8102440 0x7f58f810f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:21.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.970+0000 7f58fca10700 1 --2- 172.21.15.70:0/1856263154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58f8102e00 0x7f58f810c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:21.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.970+0000 7f58fca10700 1 -- 172.21.15.70:0/1856263154 >> 172.21.15.70:0/1856263154 conn(0x7f58f80793f0 msgr2=0x7f58f8100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:21.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.970+0000 7f58fca10700 1 -- 172.21.15.70:0/1856263154 shutdown_connections 2024-06-04T05:21:21.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:21.970+0000 7f58fca10700 1 -- 172.21.15.70:0/1856263154 wait complete. 2024-06-04T05:21:21.974 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 64 2024-06-04T05:21:22.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:22 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1856263154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:21:22.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:22 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1856263154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:21:22.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:22 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1856263154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:21:22.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 80369 6.97 MB/sec execute 494 sec latency 68.156 ms 2024-06-04T05:21:23.062 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-06-04T05:21:23.062 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-06-04T05:21:23.394 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:21:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:23 smithi112 ceph-mon[36453]: pgmap v763: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.9 MiB/s wr, 431 op/s 2024-06-04T05:21:23.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:23 smithi098 ceph-mon[35569]: pgmap v763: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.9 MiB/s wr, 431 op/s 2024-06-04T05:21:23.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 80519 6.96 MB/sec execute 495 sec latency 84.182 ms 2024-06-04T05:21:23.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:23 smithi070 ceph-mon[32631]: pgmap v763: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.9 MiB/s wr, 431 op/s 2024-06-04T05:21:24.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 80663 6.96 MB/sec execute 496 sec latency 115.744 ms 2024-06-04T05:21:25.657 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:25 smithi070 ceph-mon[32631]: pgmap v764: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.2 MiB/s wr, 304 op/s 2024-06-04T05:21:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:25 smithi112 ceph-mon[36453]: pgmap v764: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.2 MiB/s wr, 304 op/s 2024-06-04T05:21:25.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:25 smithi098 ceph-mon[35569]: pgmap v764: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.2 MiB/s wr, 304 op/s 2024-06-04T05:21:25.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 80798 6.96 MB/sec execute 497 sec latency 135.888 ms 2024-06-04T05:21:25.940 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:21:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.227+0000 7f2ca9351640 1 -- 172.21.15.70:0/340856822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca41033f0 msgr2=0x7f2ca410f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.227+0000 7f2ca9351640 1 --2- 172.21.15.70:0/340856822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca41033f0 0x7f2ca410f900 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009cd0 tx=0x7f2c9c02a8f0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.227+0000 7f2ca9351640 1 -- 172.21.15.70:0/340856822 shutdown_connections 2024-06-04T05:21:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.227+0000 7f2ca9351640 1 --2- 172.21.15.70:0/340856822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ca4102a30 0x7f2ca4102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.227+0000 7f2ca9351640 1 --2- 172.21.15.70:0/340856822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca41033f0 0x7f2ca410f900 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.227+0000 7f2ca9351640 1 --2- 172.21.15.70:0/340856822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ca4108a40 0x7f2ca4108e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.227+0000 7f2ca9351640 1 -- 172.21.15.70:0/340856822 >> 172.21.15.70:0/340856822 conn(0x7f2ca40fe500 msgr2=0x7f2ca4100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.227+0000 7f2ca9351640 1 -- 172.21.15.70:0/340856822 shutdown_connections 2024-06-04T05:21:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.227+0000 7f2ca9351640 1 -- 172.21.15.70:0/340856822 wait complete. 2024-06-04T05:21:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.228+0000 7f2ca9351640 1 Processor -- start 2024-06-04T05:21:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.228+0000 7f2ca9351640 1 -- start start 2024-06-04T05:21:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca9351640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ca4102a30 0x7f2ca41990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca9351640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ca41033f0 0x7f2ca4199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca9351640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca4108a40 0x7f2ca419d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca9351640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca4112bb0 con 0x7f2ca4108a40 2024-06-04T05:21:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca9351640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca4112a30 con 0x7f2ca4102a30 2024-06-04T05:21:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca9351640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca4112d30 con 0x7f2ca41033f0 2024-06-04T05:21:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca4108a40 0x7f2ca419d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca4108a40 0x7f2ca419d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57044/0 (socket says 172.21.15.70:57044) 2024-06-04T05:21:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca37fe640 1 -- 172.21.15.70:0/2245767755 learned_addr learned my addr 172.21.15.70:0/2245767755 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca27fc640 1 --2- 172.21.15.70:0/2245767755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ca41033f0 0x7f2ca4199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca2ffd640 1 --2- 172.21.15.70:0/2245767755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ca4102a30 0x7f2ca41990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca37fe640 1 -- 172.21.15.70:0/2245767755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ca41033f0 msgr2=0x7f2ca4199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca37fe640 1 --2- 172.21.15.70:0/2245767755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ca41033f0 0x7f2ca4199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca37fe640 1 -- 172.21.15.70:0/2245767755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ca4102a30 msgr2=0x7f2ca41990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca37fe640 1 --2- 172.21.15.70:0/2245767755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ca4102a30 0x7f2ca41990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca37fe640 1 -- 172.21.15.70:0/2245767755 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca419e090 con 0x7f2ca4108a40 2024-06-04T05:21:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca2ffd640 1 --2- 172.21.15.70:0/2245767755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ca4102a30 0x7f2ca41990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:21:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.229+0000 7f2ca37fe640 1 --2- 172.21.15.70:0/2245767755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca4108a40 0x7f2ca419d960 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c005cc0 tx=0x7f2c9c030f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.230+0000 7f2c8bfff640 1 -- 172.21.15.70:0/2245767755 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c004040 con 0x7f2ca4108a40 2024-06-04T05:21:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.230+0000 7f2c8bfff640 1 -- 172.21.15.70:0/2245767755 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2c9c040070 con 0x7f2ca4108a40 2024-06-04T05:21:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.230+0000 7f2ca9351640 1 -- 172.21.15.70:0/2245767755 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ca419e320 con 0x7f2ca4108a40 2024-06-04T05:21:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.230+0000 7f2c8bfff640 1 -- 172.21.15.70:0/2245767755 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c045980 con 0x7f2ca4108a40 2024-06-04T05:21:26.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.230+0000 7f2ca9351640 1 -- 172.21.15.70:0/2245767755 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ca41a5c10 con 0x7f2ca4108a40 2024-06-04T05:21:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.233+0000 7f2cac06c740 1 Processor -- start 2024-06-04T05:21:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.233+0000 7f2c8bfff640 1 -- 172.21.15.70:0/2245767755 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f2c9c045b40 con 0x7f2ca4108a40 2024-06-04T05:21:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.233+0000 7f2cac06c740 1 -- start start 2024-06-04T05:21:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.233+0000 7f2cac06c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 0x55906dfb16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.233+0000 7f2cac06c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfb1c20 0x55906dfb20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.233+0000 7f2cac06c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfc50a0 0x55906dfc7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.233+0000 7f2cac06c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55906df23c30 con 0x55906dfc50a0 2024-06-04T05:21:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.233+0000 7f2cac06c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55906df137d0 con 0x55906dfb1c20 2024-06-04T05:21:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.233+0000 7f2cac06c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55906df17810 con 0x55906dfb2d60 2024-06-04T05:21:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.233+0000 7f2ca27fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfc50a0 0x55906dfc7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.233+0000 7f2ca27fc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 0x55906dfb16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.233+0000 7f2ca27fc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 0x55906dfb16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:53332/0 (socket says 172.21.15.70:53332) 2024-06-04T05:21:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.233+0000 7f2ca27fc640 1 -- 172.21.15.70:0/1419691558 learned_addr learned my addr 172.21.15.70:0/1419691558 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.234+0000 7f2ca2ffd640 1 --2- 172.21.15.70:0/1419691558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfb1c20 0x55906dfb20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.234+0000 7f2ca27fc640 1 -- 172.21.15.70:0/1419691558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfb1c20 msgr2=0x55906dfb20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.234+0000 7f2ca27fc640 1 --2- 172.21.15.70:0/1419691558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfb1c20 0x55906dfb20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.234+0000 7f2ca27fc640 1 -- 172.21.15.70:0/1419691558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfc50a0 msgr2=0x55906dfc7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.234+0000 7f2ca27fc640 1 --2- 172.21.15.70:0/1419691558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfc50a0 0x55906dfc7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.234+0000 7f2ca27fc640 1 -- 172.21.15.70:0/1419691558 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55906dc28210 con 0x55906dfb2d60 2024-06-04T05:21:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.234+0000 7f2c8bfff640 1 --2- 172.21.15.70:0/2245767755 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2c8c0b2a90 0x7f2c8c0b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.234+0000 7f2ca27fc640 1 --2- 172.21.15.70:0/1419691558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 0x55906dfb16e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2c98010960 tx=0x7f2c98005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21555434ccc1da2a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.234+0000 7f2ca2ffd640 1 --2- 172.21.15.70:0/2245767755 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2c8c0b2a90 0x7f2c8c0b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.234+0000 7f2c8bfff640 1 -- 172.21.15.70:0/2245767755 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f2c9c103680 con 0x7f2ca4108a40 2024-06-04T05:21:26.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.234+0000 7f2c897fa640 1 -- 172.21.15.70:0/1419691558 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98016070 con 0x55906dfb2d60 2024-06-04T05:21:26.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.234+0000 7f2c897fa640 1 -- 172.21.15.70:0/1419691558 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2c9800f7c0 con 0x55906dfb2d60 2024-06-04T05:21:26.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.234+0000 7f2c897fa640 1 -- 172.21.15.70:0/1419691558 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98019040 con 0x55906dfb2d60 2024-06-04T05:21:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.235+0000 7f2cac06c740 1 -- 172.21.15.70:0/1419691558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 msgr2=0x55906dfb16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.235+0000 7f2ca2ffd640 1 --2- 172.21.15.70:0/2245767755 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2c8c0b2a90 0x7f2c8c0b4f50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x55906dfb4410 tx=0x7f2c94005f90 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.235+0000 7f2cac06c740 1 --2- 172.21.15.70:0/1419691558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 0x55906dfb16e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2c98010960 tx=0x7f2c98005f90 comp rx=0 tx=0).stop 2024-06-04T05:21:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.235+0000 7f2cac06c740 1 -- 172.21.15.70:0/1419691558 shutdown_connections 2024-06-04T05:21:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.235+0000 7f2cac06c740 1 --2- 172.21.15.70:0/1419691558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfb1c20 0x55906dfb20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.235+0000 7f2cac06c740 1 --2- 172.21.15.70:0/1419691558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfc50a0 0x55906dfc7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.235+0000 7f2cac06c740 1 --2- 172.21.15.70:0/1419691558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 0x55906dfb16e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.235+0000 7f2cac06c740 1 -- 172.21.15.70:0/1419691558 >> 172.21.15.70:0/1419691558 conn(0x55906dfadd30 msgr2=0x55906dfae110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.235+0000 7f2cac06c740 1 -- 172.21.15.70:0/1419691558 shutdown_connections 2024-06-04T05:21:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.235+0000 7f2cac06c740 1 -- 172.21.15.70:0/1419691558 wait complete. 2024-06-04T05:21:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.236+0000 7f2cac06c740 1 -- start start 2024-06-04T05:21:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.236+0000 7f2cac06c740 1 Processor -- start 2024-06-04T05:21:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.236+0000 7f2cac06c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfb1c20 0x55906dfb5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.236+0000 7f2cac06c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 0x55906dfb5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.236+0000 7f2cac06c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfc50a0 0x55906dfb5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.236+0000 7f2cac06c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55906dbf9790 con 0x55906dfc50a0 2024-06-04T05:21:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.236+0000 7f2cac06c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55906decf890 con 0x55906dfb1c20 2024-06-04T05:21:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.236+0000 7f2cac06c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55906dfc7da0 con 0x55906dfb2d60 2024-06-04T05:21:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.236+0000 7f2ca2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfb1c20 0x55906dfb5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.236+0000 7f2ca27fc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 0x55906dfb5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.236+0000 7f2ca27fc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 0x55906dfb5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:53348/0 (socket says 172.21.15.70:53348) 2024-06-04T05:21:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.236+0000 7f2ca27fc640 1 -- 172.21.15.70:0/3937073556 learned_addr learned my addr 172.21.15.70:0/3937073556 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.236+0000 7f2cac06c740 10 client.0 _ll_register_callbacks cb 0x55906dc27a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.236+0000 7f2ca37fe640 1 --2- 172.21.15.70:0/3937073556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfc50a0 0x55906dfb5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.236+0000 7f2ca2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfb1c20 0x55906dfb5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:35628/0 (socket says 172.21.15.70:35628) 2024-06-04T05:21:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.237+0000 7f2ca2ffd640 1 -- 172.21.15.70:0/3937073556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 msgr2=0x55906dfb5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.237+0000 7f2ca2ffd640 1 --2- 172.21.15.70:0/3937073556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 0x55906dfb5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.237+0000 7f2ca2ffd640 1 -- 172.21.15.70:0/3937073556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfc50a0 msgr2=0x55906dfb5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.237+0000 7f2ca2ffd640 1 --2- 172.21.15.70:0/3937073556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfc50a0 0x55906dfb5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.237+0000 7f2ca2ffd640 1 -- 172.21.15.70:0/3937073556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55906dcf4620 con 0x55906dfb1c20 2024-06-04T05:21:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.237+0000 7f2ca37fe640 1 --2- 172.21.15.70:0/3937073556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfc50a0 0x55906dfb5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:21:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.237+0000 7f2ca2ffd640 1 --2- 172.21.15.70:0/3937073556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfb1c20 0x55906dfb5520 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2c94012dc0 tx=0x7f2c940046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f73bc53d0d5db62 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.237+0000 7f2c86ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:21:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.237+0000 7f2c86ff5640 1 -- 172.21.15.70:0/3937073556 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9400f810 con 0x55906dfb1c20 2024-06-04T05:21:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.237+0000 7f2cac06c740 1 -- 172.21.15.70:0/3937073556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c9800f7c0 con 0x55906dfb1c20 2024-06-04T05:21:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.237+0000 7f2c86ff5640 1 -- 172.21.15.70:0/3937073556 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f2c9400f9b0 con 0x55906dfb1c20 2024-06-04T05:21:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.237+0000 7f2c86ff5640 1 -- 172.21.15.70:0/3937073556 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f2c9400fbf0 con 0x55906dfb1c20 2024-06-04T05:21:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.237+0000 7f2cac06c740 10 client.30167 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.237+0000 7f2cac06c740 1 -- 172.21.15.70:0/3937073556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55906dfd3cb0 con 0x55906dfb1c20 2024-06-04T05:21:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.238+0000 7f2c86ff5640 1 -- 172.21.15.70:0/3937073556 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2c9800f7c0 con 0x55906dfb1c20 2024-06-04T05:21:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.238+0000 7f2c86ff5640 1 -- 172.21.15.70:0/3937073556 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f2c94027070 con 0x55906dfb1c20 2024-06-04T05:21:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.238+0000 7f2cac06c740 10 client.30167 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.238+0000 7f2cac06c740 10 client.30167 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:21:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.238+0000 7f2cac06c740 20 client.30167 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.238+0000 7f2cac06c740 1 --2- 172.21.15.70:0/3937073556 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55906dfd5190 0x55906dfd75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.238+0000 7f2cac06c740 4 client.30167 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:21:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.238+0000 7f2cac06c740 1 -- 172.21.15.70:0/3937073556 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55906df25b80 con 0x55906dfd5190 2024-06-04T05:21:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.238+0000 7f2ca27fc640 1 --2- 172.21.15.70:0/3937073556 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55906dfd5190 0x55906dfd75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.239+0000 7f2c86ff5640 10 client.30167 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.239+0000 7f2ca27fc640 1 --2- 172.21.15.70:0/3937073556 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55906dfd5190 0x55906dfd75c0 crc :-1 s=READY pgs=501 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b61be8666c00691e server_cookie=be5a77f2d18ad5b0 in_seq=0 out_seq=0 2024-06-04T05:21:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.240+0000 7f2c86ff5640 1 -- 172.21.15.70:0/3937073556 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55906df25b80 con 0x55906dfd5190 2024-06-04T05:21:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.240+0000 7f2c86ff5640 10 client.30167 handle_command_reply: tid=0 2024-06-04T05:21:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.240+0000 7f2cac06c740 1 client.30167 shutdown 2024-06-04T05:21:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.240+0000 7f2cac06c740 1 -- 172.21.15.70:0/3937073556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfb1c20 msgr2=0x55906dfb5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.240+0000 7f2cac06c740 1 --2- 172.21.15.70:0/3937073556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfb1c20 0x55906dfb5520 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2c94012dc0 tx=0x7f2c940046c0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.240+0000 7f2cac06c740 1 -- 172.21.15.70:0/3937073556 shutdown_connections 2024-06-04T05:21:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.240+0000 7f2cac06c740 1 --2- 172.21.15.70:0/3937073556 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55906dfd5190 0x55906dfd75c0 crc :-1 s=READY pgs=501 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.240+0000 7f2c86ff5640 0 client.30167 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.240+0000 7f2cac06c740 1 --2- 172.21.15.70:0/3937073556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfc50a0 0x55906dfb5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.240+0000 7f2cac06c740 1 --2- 172.21.15.70:0/3937073556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 0x55906dfb5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.240+0000 7f2cac06c740 1 --2- 172.21.15.70:0/3937073556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfb1c20 0x55906dfb5520 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.240+0000 7f2cac06c740 1 -- 172.21.15.70:0/3937073556 >> 172.21.15.70:0/3937073556 conn(0x55906dfadd30 msgr2=0x55906dfb3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.240+0000 7f2cac06c740 1 -- 172.21.15.70:0/3937073556 shutdown_connections 2024-06-04T05:21:26.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.241+0000 7f2cac06c740 1 -- 172.21.15.70:0/3937073556 wait complete. 2024-06-04T05:21:26.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.241+0000 7f2cac06c740 20 client.30167 trim_cache size 0 max 16384 2024-06-04T05:21:26.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.263+0000 7f2cac06c740 1 Processor -- start 2024-06-04T05:21:26.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.263+0000 7f2cac06c740 1 -- start start 2024-06-04T05:21:26.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.263+0000 7f2cac06c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfb1c20 0x55906dfad730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.263+0000 7f2cac06c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 0x55906dfdcf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.263+0000 7f2cac06c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfc50a0 0x55906dfdd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.263+0000 7f2cac06c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca41ac410 con 0x55906dfb1c20 2024-06-04T05:21:26.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.263+0000 7f2cac06c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca41adfe0 con 0x55906dfc50a0 2024-06-04T05:21:26.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.263+0000 7f2cac06c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca41acb40 con 0x55906dfb2d60 2024-06-04T05:21:26.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.264+0000 7f2ca2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfb1c20 0x55906dfad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.264+0000 7f2ca2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfb1c20 0x55906dfad730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57072/0 (socket says 172.21.15.70:57072) 2024-06-04T05:21:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.264+0000 7f2ca2ffd640 1 -- 172.21.15.70:0/1545460934 learned_addr learned my addr 172.21.15.70:0/1545460934 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.264+0000 7f2ca27fc640 1 --2- 172.21.15.70:0/1545460934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 0x55906dfdcf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.264+0000 7f2ca37fe640 1 --2- 172.21.15.70:0/1545460934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfc50a0 0x55906dfdd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.264+0000 7f2ca2ffd640 1 -- 172.21.15.70:0/1545460934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 msgr2=0x55906dfdcf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.264+0000 7f2ca2ffd640 1 --2- 172.21.15.70:0/1545460934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 0x55906dfdcf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.264+0000 7f2ca2ffd640 1 -- 172.21.15.70:0/1545460934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfc50a0 msgr2=0x55906dfdd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.264+0000 7f2ca2ffd640 1 --2- 172.21.15.70:0/1545460934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfc50a0 0x55906dfdd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.264+0000 7f2ca2ffd640 1 -- 172.21.15.70:0/1545460934 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55906dfd7c40 con 0x55906dfb1c20 2024-06-04T05:21:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.264+0000 7f2ca37fe640 1 --2- 172.21.15.70:0/1545460934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfc50a0 0x55906dfdd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:21:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.264+0000 7f2ca2ffd640 1 --2- 172.21.15.70:0/1545460934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfb1c20 0x55906dfad730 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f2c940092e0 tx=0x7f2c94009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc48282fd988e2b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.264+0000 7f2c88ff9640 1 -- 172.21.15.70:0/1545460934 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9400fe50 con 0x55906dfb1c20 2024-06-04T05:21:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.264+0000 7f2c88ff9640 1 -- 172.21.15.70:0/1545460934 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2c940318a0 con 0x55906dfb1c20 2024-06-04T05:21:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.264+0000 7f2c88ff9640 1 -- 172.21.15.70:0/1545460934 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c94005420 con 0x55906dfb1c20 2024-06-04T05:21:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.265+0000 7f2cac06c740 1 -- 172.21.15.70:0/1545460934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfb1c20 msgr2=0x55906dfad730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.265+0000 7f2cac06c740 1 --2- 172.21.15.70:0/1545460934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfb1c20 0x55906dfad730 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f2c940092e0 tx=0x7f2c94009fd0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.265+0000 7f2cac06c740 1 -- 172.21.15.70:0/1545460934 shutdown_connections 2024-06-04T05:21:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.265+0000 7f2cac06c740 1 --2- 172.21.15.70:0/1545460934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfc50a0 0x55906dfdd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.265+0000 7f2cac06c740 1 --2- 172.21.15.70:0/1545460934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfb2d60 0x55906dfdcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.265+0000 7f2cac06c740 1 --2- 172.21.15.70:0/1545460934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfb1c20 0x55906dfad730 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.265+0000 7f2cac06c740 1 -- 172.21.15.70:0/1545460934 >> 172.21.15.70:0/1545460934 conn(0x55906dfadd30 msgr2=0x55906dfb3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.265+0000 7f2cac06c740 1 -- 172.21.15.70:0/1545460934 shutdown_connections 2024-06-04T05:21:26.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.265+0000 7f2cac06c740 1 -- 172.21.15.70:0/1545460934 wait complete. 2024-06-04T05:21:26.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.266+0000 7f2cac06c740 1 -- start start 2024-06-04T05:21:26.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.266+0000 7f2cac06c740 1 Processor -- start 2024-06-04T05:21:26.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.266+0000 7f2cac06c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfb1c20 0x55906dfcd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.266+0000 7f2ca2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfb1c20 0x55906dfcd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.266+0000 7f2ca2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfb1c20 0x55906dfcd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57084/0 (socket says 172.21.15.70:57084) 2024-06-04T05:21:26.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.266+0000 7f2ca2ffd640 1 -- 172.21.15.70:0/1520613882 learned_addr learned my addr 172.21.15.70:0/1520613882 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:26.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.266+0000 7f2cac06c740 1 --2- 172.21.15.70:0/1520613882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfb2d60 0x55906dfcd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.266+0000 7f2cac06c740 1 --2- 172.21.15.70:0/1520613882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfc50a0 0x55906dfca000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.266+0000 7f2cac06c740 1 -- 172.21.15.70:0/1520613882 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca41ad8e0 con 0x55906dfb1c20 2024-06-04T05:21:26.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.266+0000 7f2cac06c740 1 -- 172.21.15.70:0/1520613882 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca41ad210 con 0x55906dfb2d60 2024-06-04T05:21:26.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.266+0000 7f2cac06c740 1 -- 172.21.15.70:0/1520613882 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca41aa410 con 0x55906dfc50a0 2024-06-04T05:21:26.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.266+0000 7f2cac06c740 10 client.0 _ll_register_callbacks cb 0x7f2c94009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:26.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.267+0000 7f2ca37fe640 1 --2- 172.21.15.70:0/1520613882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfc50a0 0x55906dfca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.267+0000 7f2ca27fc640 1 --2- 172.21.15.70:0/1520613882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfb2d60 0x55906dfcd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.267+0000 7f2ca2ffd640 1 -- 172.21.15.70:0/1520613882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfc50a0 msgr2=0x55906dfca000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:26.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.267+0000 7f2ca2ffd640 1 --2- 172.21.15.70:0/1520613882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfc50a0 0x55906dfca000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.267+0000 7f2ca2ffd640 1 -- 172.21.15.70:0/1520613882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfb2d60 msgr2=0x55906dfcd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:26.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.267+0000 7f2ca2ffd640 1 --2- 172.21.15.70:0/1520613882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfb2d60 0x55906dfcd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.267+0000 7f2ca2ffd640 1 -- 172.21.15.70:0/1520613882 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55906dfc4e90 con 0x55906dfb1c20 2024-06-04T05:21:26.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.267+0000 7f2ca2ffd640 1 --2- 172.21.15.70:0/1520613882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfb1c20 0x55906dfcd030 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009f20 tx=0x7f2c9400ff40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc67b6e4c8af24a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:26.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.267+0000 7f2c86ff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:21:26.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.267+0000 7f2c86ff5640 1 -- 172.21.15.70:0/1520613882 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9403f070 con 0x55906dfb1c20 2024-06-04T05:21:26.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.267+0000 7f2cac06c740 1 -- 172.21.15.70:0/1520613882 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c940318a0 con 0x55906dfb1c20 2024-06-04T05:21:26.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.267+0000 7f2c86ff5640 1 -- 172.21.15.70:0/1520613882 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f2c94005040 con 0x55906dfb1c20 2024-06-04T05:21:26.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.267+0000 7f2c86ff5640 1 -- 172.21.15.70:0/1520613882 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f2c94048050 con 0x55906dfb1c20 2024-06-04T05:21:26.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.267+0000 7f2cac06c740 10 client.21525 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:26.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.267+0000 7f2cac06c740 1 -- 172.21.15.70:0/1520613882 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55906e040350 con 0x55906dfb1c20 2024-06-04T05:21:26.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.267+0000 7f2c86ff5640 1 -- 172.21.15.70:0/1520613882 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2c940318a0 con 0x55906dfb1c20 2024-06-04T05:21:26.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.268+0000 7f2c86ff5640 1 -- 172.21.15.70:0/1520613882 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f2c94042020 con 0x55906dfb1c20 2024-06-04T05:21:26.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.268+0000 7f2cac06c740 10 client.21525 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:26.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.268+0000 7f2cac06c740 10 client.21525 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:21:26.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.268+0000 7f2cac06c740 20 client.21525 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:26.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.268+0000 7f2cac06c740 1 --2- 172.21.15.70:0/1520613882 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55906e0409a0 0x55906e042d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:26.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.268+0000 7f2cac06c740 4 client.21525 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:21:26.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.268+0000 7f2cac06c740 1 -- 172.21.15.70:0/1520613882 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55906df1f360 con 0x55906e0409a0 2024-06-04T05:21:26.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.268+0000 7f2ca27fc640 1 --2- 172.21.15.70:0/1520613882 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55906e0409a0 0x55906e042d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:26.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.268+0000 7f2c86ff5640 10 client.21525 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:26.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.268+0000 7f2ca27fc640 1 --2- 172.21.15.70:0/1520613882 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55906e0409a0 0x55906e042d90 crc :-1 s=READY pgs=502 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a171f9c3fdd49b0 server_cookie=14afa4bc5c126adc in_seq=0 out_seq=0 2024-06-04T05:21:26.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.412+0000 7f2c86ff5640 1 -- 172.21.15.70:0/1520613882 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55906df1f360 con 0x55906e0409a0 2024-06-04T05:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.412+0000 7f2c86ff5640 10 client.21525 handle_command_reply: tid=0 2024-06-04T05:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.412+0000 7f2cac06c740 1 client.21525 shutdown 2024-06-04T05:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.412+0000 7f2cac06c740 1 -- 172.21.15.70:0/1520613882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfb1c20 msgr2=0x55906dfcd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.412+0000 7f2cac06c740 1 --2- 172.21.15.70:0/1520613882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfb1c20 0x55906dfcd030 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009f20 tx=0x7f2c9400ff40 comp rx=0 tx=0).stop 2024-06-04T05:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.412+0000 7f2cac06c740 1 -- 172.21.15.70:0/1520613882 shutdown_connections 2024-06-04T05:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.412+0000 7f2cac06c740 1 --2- 172.21.15.70:0/1520613882 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55906e0409a0 0x55906e042d90 crc :-1 s=READY pgs=502 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.412+0000 7f2cac06c740 1 --2- 172.21.15.70:0/1520613882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906dfb2d60 0x55906dfcd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.412+0000 7f2c86ff5640 0 client.21525 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.412+0000 7f2cac06c740 1 --2- 172.21.15.70:0/1520613882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55906dfc50a0 0x55906dfca000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.412+0000 7f2cac06c740 1 --2- 172.21.15.70:0/1520613882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55906dfb1c20 0x55906dfcd030 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.412+0000 7f2cac06c740 1 -- 172.21.15.70:0/1520613882 >> 172.21.15.70:0/1520613882 conn(0x55906dfadd30 msgr2=0x55906dfb54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.412+0000 7f2cac06c740 1 -- 172.21.15.70:0/1520613882 shutdown_connections 2024-06-04T05:21:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.412+0000 7f2cac06c740 1 -- 172.21.15.70:0/1520613882 wait complete. 2024-06-04T05:21:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.413+0000 7f2cac06c740 20 client.21525 trim_cache size 0 max 16384 2024-06-04T05:21:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:21:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:21:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:21:26.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.413+0000 7f2ca9351640 1 -- 172.21.15.70:0/2245767755 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2c8c0b2a90 msgr2=0x7f2c8c0b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:26.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.413+0000 7f2ca9351640 1 --2- 172.21.15.70:0/2245767755 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2c8c0b2a90 0x7f2c8c0b4f50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x55906dfb4410 tx=0x7f2c94005f90 comp rx=0 tx=0).stop 2024-06-04T05:21:26.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.413+0000 7f2ca9351640 1 -- 172.21.15.70:0/2245767755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca4108a40 msgr2=0x7f2ca419d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:26.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.413+0000 7f2ca9351640 1 --2- 172.21.15.70:0/2245767755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca4108a40 0x7f2ca419d960 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c005cc0 tx=0x7f2c9c030f70 comp rx=0 tx=0).stop 2024-06-04T05:21:26.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.413+0000 7f2ca9351640 1 -- 172.21.15.70:0/2245767755 shutdown_connections 2024-06-04T05:21:26.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.413+0000 7f2ca9351640 1 --2- 172.21.15.70:0/2245767755 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2c8c0b2a90 0x7f2c8c0b4f50 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.414+0000 7f2ca9351640 1 --2- 172.21.15.70:0/2245767755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ca4108a40 0x7f2ca419d960 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.414+0000 7f2ca9351640 1 --2- 172.21.15.70:0/2245767755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ca41033f0 0x7f2ca4199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.414+0000 7f2ca9351640 1 --2- 172.21.15.70:0/2245767755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ca4102a30 0x7f2ca41990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.414+0000 7f2ca9351640 1 -- 172.21.15.70:0/2245767755 >> 172.21.15.70:0/2245767755 conn(0x7f2ca40fe500 msgr2=0x7f2ca40fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.414+0000 7f2ca9351640 1 -- 172.21.15.70:0/2245767755 shutdown_connections 2024-06-04T05:21:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:26.414+0000 7f2ca9351640 1 -- 172.21.15.70:0/2245767755 wait complete. 2024-06-04T05:21:26.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 80953 6.97 MB/sec execute 498 sec latency 161.019 ms 2024-06-04T05:21:27.663 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:27 smithi070 ceph-mon[32631]: pgmap v765: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.6 MiB/s wr, 412 op/s 2024-06-04T05:21:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:27 smithi112 ceph-mon[36453]: pgmap v765: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.6 MiB/s wr, 412 op/s 2024-06-04T05:21:27.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:27 smithi098 ceph-mon[35569]: pgmap v765: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.6 MiB/s wr, 412 op/s 2024-06-04T05:21:27.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 81105 6.97 MB/sec execute 499 sec latency 133.928 ms 2024-06-04T05:21:28.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 81290 6.98 MB/sec execute 500 sec latency 195.013 ms 2024-06-04T05:21:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:29 smithi112 ceph-mon[36453]: pgmap v766: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 242 op/s 2024-06-04T05:21:29.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:29 smithi098 ceph-mon[35569]: pgmap v766: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 242 op/s 2024-06-04T05:21:29.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 81409 6.98 MB/sec execute 501 sec latency 156.636 ms 2024-06-04T05:21:29.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:29 smithi070 ceph-mon[32631]: pgmap v766: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 242 op/s 2024-06-04T05:21:30.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 81556 6.99 MB/sec execute 502 sec latency 146.071 ms 2024-06-04T05:21:31.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:31 smithi112 ceph-mon[36453]: pgmap v767: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 242 op/s 2024-06-04T05:21:31.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:31 smithi098 ceph-mon[35569]: pgmap v767: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 242 op/s 2024-06-04T05:21:31.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 81695 6.99 MB/sec execute 503 sec latency 180.980 ms 2024-06-04T05:21:31.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:31 smithi070 ceph-mon[32631]: pgmap v767: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 242 op/s 2024-06-04T05:21:32.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 81829 6.98 MB/sec execute 504 sec latency 123.256 ms 2024-06-04T05:21:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:33 smithi112 ceph-mon[36453]: pgmap v768: 97 pgs: 97 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.6 MiB/s wr, 392 op/s 2024-06-04T05:21:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:33 smithi098 ceph-mon[35569]: pgmap v768: 97 pgs: 97 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.6 MiB/s wr, 392 op/s 2024-06-04T05:21:33.832 INFO:tasks.workunit.client.0.smithi070.stdout: 10 81942 6.98 MB/sec execute 505 sec latency 192.809 ms 2024-06-04T05:21:33.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:33 smithi070 ceph-mon[32631]: pgmap v768: 97 pgs: 97 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.6 MiB/s wr, 392 op/s 2024-06-04T05:21:34.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82076 6.98 MB/sec execute 506 sec latency 163.601 ms 2024-06-04T05:21:34.937 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:21:35.644 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:35 smithi070 ceph-mon[32631]: pgmap v769: 97 pgs: 97 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 258 op/s 2024-06-04T05:21:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:35 smithi112 ceph-mon[36453]: pgmap v769: 97 pgs: 97 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 258 op/s 2024-06-04T05:21:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:35 smithi098 ceph-mon[35569]: pgmap v769: 97 pgs: 97 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 258 op/s 2024-06-04T05:21:35.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82237 6.98 MB/sec execute 507 sec latency 200.923 ms 2024-06-04T05:21:36.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82407 6.99 MB/sec execute 508 sec latency 88.356 ms 2024-06-04T05:21:37.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:37 smithi070 ceph-mon[32631]: pgmap v770: 97 pgs: 97 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.7 MiB/s wr, 361 op/s 2024-06-04T05:21:37.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.628+0000 7f29914d5700 1 -- 172.21.15.70:0/2885124823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f298c107450 msgr2=0x7f298c107830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:37.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.628+0000 7f29914d5700 1 --2- 172.21.15.70:0/2885124823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f298c107450 0x7f298c107830 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2980009990 tx=0x7f2980009ca0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.628+0000 7f29914d5700 1 -- 172.21.15.70:0/2885124823 shutdown_connections 2024-06-04T05:21:37.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.628+0000 7f29914d5700 1 --2- 172.21.15.70:0/2885124823 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f298c108b50 0x7f298c10f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.628+0000 7f29914d5700 1 --2- 172.21.15.70:0/2885124823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f298c107450 0x7f298c107830 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.628+0000 7f29914d5700 1 --2- 172.21.15.70:0/2885124823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f298c108190 0x7f298c108610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.628+0000 7f29914d5700 1 -- 172.21.15.70:0/2885124823 >> 172.21.15.70:0/2885124823 conn(0x7f298c078670 msgr2=0x7f298c0ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:37.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.628+0000 7f29914d5700 1 -- 172.21.15.70:0/2885124823 shutdown_connections 2024-06-04T05:21:37.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.628+0000 7f29914d5700 1 -- 172.21.15.70:0/2885124823 wait complete. 2024-06-04T05:21:37.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.629+0000 7f29914d5700 1 Processor -- start 2024-06-04T05:21:37.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.629+0000 7f29914d5700 1 -- start start 2024-06-04T05:21:37.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.629+0000 7f29914d5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f298c107450 0x7f298c075060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.630+0000 7f29914d5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f298c108190 0x7f298c06d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.630+0000 7f298affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f298c107450 0x7f298c075060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.630+0000 7f29914d5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f298c108b50 0x7f298c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.630+0000 7f29914d5700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f298c111520 con 0x7f298c108b50 2024-06-04T05:21:37.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.630+0000 7f29914d5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f298c1113a0 con 0x7f298c108190 2024-06-04T05:21:37.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.630+0000 7f29914d5700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f298c1116a0 con 0x7f298c107450 2024-06-04T05:21:37.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.630+0000 7f298affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f298c107450 0x7f298c075060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36874/0 (socket says 172.21.15.70:36874) 2024-06-04T05:21:37.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.630+0000 7f298affd700 1 -- 172.21.15.70:0/1932198398 learned_addr learned my addr 172.21.15.70:0/1932198398 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:37.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.630+0000 7f298b7fe700 1 --2- 172.21.15.70:0/1932198398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f298c108b50 0x7f298c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.630+0000 7f298affd700 1 -- 172.21.15.70:0/1932198398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f298c108190 msgr2=0x7f298c06d030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-04T05:21:37.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.630+0000 7f298affd700 1 --2- 172.21.15.70:0/1932198398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f298c108190 0x7f298c06d030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.630+0000 7f298affd700 1 -- 172.21.15.70:0/1932198398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f298c108b50 msgr2=0x7f298c06d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:37.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.630+0000 7f298affd700 1 --2- 172.21.15.70:0/1932198398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f298c108b50 0x7f298c06d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.630+0000 7f298affd700 1 -- 172.21.15.70:0/1932198398 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f298c06dde0 con 0x7f298c107450 2024-06-04T05:21:37.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.631+0000 7f298affd700 1 --2- 172.21.15.70:0/1932198398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f298c107450 0x7f298c075060 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f29800047b0 tx=0x7f298000bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:37.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.632+0000 7f2973fff700 1 -- 172.21.15.70:0/1932198398 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298000f930 con 0x7f298c107450 2024-06-04T05:21:37.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.632+0000 7f2973fff700 1 -- 172.21.15.70:0/1932198398 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f29800047e0 con 0x7f298c107450 2024-06-04T05:21:37.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.632+0000 7f2973fff700 1 -- 172.21.15.70:0/1932198398 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980005360 con 0x7f298c107450 2024-06-04T05:21:37.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.632+0000 7f29914d5700 1 -- 172.21.15.70:0/1932198398 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f298c1ac780 con 0x7f298c107450 2024-06-04T05:21:37.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.632+0000 7f29914d5700 1 -- 172.21.15.70:0/1932198398 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f298c1acb20 con 0x7f298c107450 2024-06-04T05:21:37.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.639+0000 7f2973fff700 1 -- 172.21.15.70:0/1932198398 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f2980032060 con 0x7f298c107450 2024-06-04T05:21:37.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.640+0000 7f2973fff700 1 --2- 172.21.15.70:0/1932198398 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f29740b2be0 0x7f29740b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.640+0000 7f2973fff700 1 -- 172.21.15.70:0/1932198398 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f29800ebde0 con 0x7f298c107450 2024-06-04T05:21:37.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.640+0000 7f298a7fc700 1 --2- 172.21.15.70:0/1932198398 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f29740b2be0 0x7f29740b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.642+0000 7f299bd6ab80 1 Processor -- start 2024-06-04T05:21:37.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.642+0000 7f299bd6ab80 1 -- start start 2024-06-04T05:21:37.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.642+0000 7f299bd6ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0ac7c0 0x5581cb0acba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.642+0000 7f299bd6ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0a7c90 0x5581cb0a8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.642+0000 7f299bd6ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a8650 0x5581cb0b46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.642+0000 7f299bd6ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5581caff4f00 con 0x5581cb0a7c90 2024-06-04T05:21:37.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.642+0000 7f299bd6ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5581cafe4aa0 con 0x5581cb0ac7c0 2024-06-04T05:21:37.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.642+0000 7f299bd6ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5581cafe8ae0 con 0x5581cb0a8650 2024-06-04T05:21:37.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.642+0000 7f298affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0ac7c0 0x5581cb0acba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.642+0000 7f298affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0ac7c0 0x5581cb0acba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59876/0 (socket says 172.21.15.70:59876) 2024-06-04T05:21:37.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.642+0000 7f298affd700 1 -- 172.21.15.70:0/895298067 learned_addr learned my addr 172.21.15.70:0/895298067 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:37.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.642+0000 7f298a7fc700 1 --2- 172.21.15.70:0/1932198398 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f29740b2be0 0x7f29740b50a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f298c06e7f0 tx=0x7f298400b040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:37.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.643+0000 7f298affd700 1 -- 172.21.15.70:0/895298067 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a8650 msgr2=0x5581cb0b46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:37.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.643+0000 7f298affd700 1 --2- 172.21.15.70:0/895298067 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a8650 0x5581cb0b46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.643+0000 7f298affd700 1 -- 172.21.15.70:0/895298067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0a7c90 msgr2=0x5581cb0a8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:37.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.643+0000 7f298affd700 1 --2- 172.21.15.70:0/895298067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0a7c90 0x5581cb0a8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.643+0000 7f298affd700 1 -- 172.21.15.70:0/895298067 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581caf8e140 con 0x5581cb0ac7c0 2024-06-04T05:21:37.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.644+0000 7f298affd700 1 --2- 172.21.15.70:0/895298067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0ac7c0 0x5581cb0acba0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2980005000 tx=0x7f29800b2d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8b5ea849b9af62b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:37.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.644+0000 7f29717fa700 1 -- 172.21.15.70:0/895298067 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29800bbef0 con 0x5581cb0ac7c0 2024-06-04T05:21:37.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.644+0000 7f29717fa700 1 -- 172.21.15.70:0/895298067 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f29800b71b0 con 0x5581cb0ac7c0 2024-06-04T05:21:37.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.644+0000 7f29717fa700 1 -- 172.21.15.70:0/895298067 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29800b35e0 con 0x5581cb0ac7c0 2024-06-04T05:21:37.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.647+0000 7f299bd6ab80 1 -- 172.21.15.70:0/895298067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0ac7c0 msgr2=0x5581cb0acba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:37.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.647+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/895298067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0ac7c0 0x5581cb0acba0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2980005000 tx=0x7f29800b2d80 comp rx=0 tx=0).stop 2024-06-04T05:21:37.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.647+0000 7f299bd6ab80 1 -- 172.21.15.70:0/895298067 shutdown_connections 2024-06-04T05:21:37.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.647+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/895298067 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a8650 0x5581cb0b46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.647+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/895298067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0ac7c0 0x5581cb0acba0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.647+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/895298067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0a7c90 0x5581cb0a8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.647+0000 7f299bd6ab80 1 -- 172.21.15.70:0/895298067 >> 172.21.15.70:0/895298067 conn(0x5581cafc23d0 msgr2=0x5581cb091970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:37.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.648+0000 7f299bd6ab80 1 -- 172.21.15.70:0/895298067 shutdown_connections 2024-06-04T05:21:37.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.648+0000 7f299bd6ab80 1 -- 172.21.15.70:0/895298067 wait complete. 2024-06-04T05:21:37.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.648+0000 7f299bd6ab80 1 -- start start 2024-06-04T05:21:37.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.648+0000 7f299bd6ab80 1 Processor -- start 2024-06-04T05:21:37.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.649+0000 7f299bd6ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a7c90 0x5581cb0b4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.649+0000 7f299bd6ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0a8650 0x5581cb0b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.649+0000 7f299bd6ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0ac7c0 0x5581cb0b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.649+0000 7f299bd6ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5581caeeb130 con 0x5581cb0ac7c0 2024-06-04T05:21:37.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.649+0000 7f298affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a7c90 0x5581cb0b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.649+0000 7f298a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0a8650 0x5581cb0b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.649+0000 7f298affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a7c90 0x5581cb0b4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36906/0 (socket says 172.21.15.70:36906) 2024-06-04T05:21:37.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.649+0000 7f298a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0a8650 0x5581cb0b16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59890/0 (socket says 172.21.15.70:59890) 2024-06-04T05:21:37.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.649+0000 7f298affd700 1 -- 172.21.15.70:0/3480040480 learned_addr learned my addr 172.21.15.70:0/3480040480 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:37.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.649+0000 7f298b7fe700 1 --2- 172.21.15.70:0/3480040480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0ac7c0 0x5581cb0b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.649+0000 7f299bd6ab80 1 -- 172.21.15.70:0/3480040480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5581caf4cf30 con 0x5581cb0a8650 2024-06-04T05:21:37.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.649+0000 7f299bd6ab80 1 -- 172.21.15.70:0/3480040480 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5581cafe4aa0 con 0x5581cb0a7c90 2024-06-04T05:21:37.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.649+0000 7f299bd6ab80 10 client.0 _ll_register_callbacks cb 0x5581caf7fae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:37.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.650+0000 7f298a7fc700 1 -- 172.21.15.70:0/3480040480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a7c90 msgr2=0x5581cb0b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:37.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.650+0000 7f298a7fc700 1 --2- 172.21.15.70:0/3480040480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a7c90 0x5581cb0b4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.650+0000 7f298a7fc700 1 -- 172.21.15.70:0/3480040480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0ac7c0 msgr2=0x5581cb0b1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:37.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.650+0000 7f298a7fc700 1 --2- 172.21.15.70:0/3480040480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0ac7c0 0x5581cb0b1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.650+0000 7f298a7fc700 1 -- 172.21.15.70:0/3480040480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581caf34a60 con 0x5581cb0a8650 2024-06-04T05:21:37.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.650+0000 7f298a7fc700 1 --2- 172.21.15.70:0/3480040480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0a8650 0x5581cb0b16b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2984015a90 tx=0x7f2984009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f0db9adfffe24fd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:37.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.651+0000 7f296eff5700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:21:37.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.651+0000 7f296eff5700 1 -- 172.21.15.70:0/3480040480 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2984004bb0 con 0x5581cb0a8650 2024-06-04T05:21:37.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.651+0000 7f296eff5700 1 -- 172.21.15.70:0/3480040480 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f2984004d50 con 0x5581cb0a8650 2024-06-04T05:21:37.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.651+0000 7f299bd6ab80 1 -- 172.21.15.70:0/3480040480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29800b71b0 con 0x5581cb0a8650 2024-06-04T05:21:37.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.652+0000 7f296eff5700 1 -- 172.21.15.70:0/3480040480 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f2984013750 con 0x5581cb0a8650 2024-06-04T05:21:37.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.652+0000 7f299bd6ab80 10 client.30182 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:37.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.652+0000 7f299bd6ab80 1 -- 172.21.15.70:0/3480040480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x5581caf9e8f0 con 0x5581cb0a8650 2024-06-04T05:21:37.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.653+0000 7f296eff5700 1 -- 172.21.15.70:0/3480040480 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2984013d80 con 0x5581cb0a8650 2024-06-04T05:21:37.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.653+0000 7f296eff5700 1 -- 172.21.15.70:0/3480040480 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f298401fbf0 con 0x5581cb0a8650 2024-06-04T05:21:37.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.653+0000 7f299bd6ab80 10 client.30182 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:37.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.653+0000 7f299bd6ab80 10 client.30182 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:21:37.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.653+0000 7f299bd6ab80 20 client.30182 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:37.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.653+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/3480040480 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5581cb0a3600 0x5581cb0a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.653+0000 7f299bd6ab80 4 client.30182 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:21:37.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.653+0000 7f299bd6ab80 1 -- 172.21.15.70:0/3480040480 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f29800b71b0 con 0x5581cb0a3600 2024-06-04T05:21:37.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.653+0000 7f298affd700 1 --2- 172.21.15.70:0/3480040480 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5581cb0a3600 0x5581cb0a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.654+0000 7f296eff5700 10 client.30182 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:37.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.655+0000 7f298affd700 1 --2- 172.21.15.70:0/3480040480 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5581cb0a3600 0x5581cb0a5a20 crc :-1 s=READY pgs=503 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d03dd4f7d5e58f4 server_cookie=7e4e0acbf2174cc5 in_seq=0 out_seq=0 2024-06-04T05:21:37.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.656+0000 7f296eff5700 1 -- 172.21.15.70:0/3480040480 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f29800b71b0 con 0x5581cb0a3600 2024-06-04T05:21:37.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.656+0000 7f296eff5700 10 client.30182 handle_command_reply: tid=0 2024-06-04T05:21:37.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.657+0000 7f299bd6ab80 1 client.30182 shutdown 2024-06-04T05:21:37.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.658+0000 7f299bd6ab80 1 -- 172.21.15.70:0/3480040480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0a8650 msgr2=0x5581cb0b16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:37.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.658+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/3480040480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0a8650 0x5581cb0b16b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2984015a90 tx=0x7f2984009920 comp rx=0 tx=0).stop 2024-06-04T05:21:37.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.658+0000 7f299bd6ab80 1 -- 172.21.15.70:0/3480040480 shutdown_connections 2024-06-04T05:21:37.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.658+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/3480040480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0ac7c0 0x5581cb0b1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.658+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/3480040480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a7c90 0x5581cb0b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.658+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/3480040480 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5581cb0a3600 0x5581cb0a5a20 crc :-1 s=READY pgs=503 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.658+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/3480040480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0a8650 0x5581cb0b16b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.658+0000 7f299bd6ab80 1 -- 172.21.15.70:0/3480040480 >> 172.21.15.70:0/3480040480 conn(0x5581cafc23d0 msgr2=0x5581cb095c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:37.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.658+0000 7f296eff5700 0 client.30182 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:37.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.658+0000 7f299bd6ab80 1 -- 172.21.15.70:0/3480040480 shutdown_connections 2024-06-04T05:21:37.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.659+0000 7f299bd6ab80 1 -- 172.21.15.70:0/3480040480 wait complete. 2024-06-04T05:21:37.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.659+0000 7f299bd6ab80 20 client.30182 trim_cache size 0 max 16384 2024-06-04T05:21:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:37 smithi112 ceph-mon[36453]: pgmap v770: 97 pgs: 97 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.7 MiB/s wr, 361 op/s 2024-06-04T05:21:37.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.689+0000 7f299bd6ab80 1 Processor -- start 2024-06-04T05:21:37.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.689+0000 7f299bd6ab80 1 -- start start 2024-06-04T05:21:37.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.690+0000 7f299bd6ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a7c90 0x5581cb08bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.690+0000 7f299bd6ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0a8650 0x5581cb0a7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.690+0000 7f299bd6ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0ac7c0 0x5581cb0aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.690+0000 7f299bd6ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f298c1b4360 con 0x5581cb0ac7c0 2024-06-04T05:21:37.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.690+0000 7f299bd6ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f298c1b5f30 con 0x5581cb0a8650 2024-06-04T05:21:37.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.690+0000 7f299bd6ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f298c1b4a90 con 0x5581cb0a7c90 2024-06-04T05:21:37.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.690+0000 7f298b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0ac7c0 0x5581cb0aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.690+0000 7f298affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a7c90 0x5581cb08bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.690+0000 7f298affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a7c90 0x5581cb08bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36908/0 (socket says 172.21.15.70:36908) 2024-06-04T05:21:37.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.690+0000 7f298affd700 1 -- 172.21.15.70:0/2260168410 learned_addr learned my addr 172.21.15.70:0/2260168410 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:37.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.690+0000 7f298b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0ac7c0 0x5581cb0aeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60688/0 (socket says 172.21.15.70:60688) 2024-06-04T05:21:37.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.691+0000 7f298a7fc700 1 --2- 172.21.15.70:0/2260168410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0a8650 0x5581cb0a7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.691+0000 7f298affd700 1 -- 172.21.15.70:0/2260168410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0a8650 msgr2=0x5581cb0a7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:37.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.691+0000 7f298affd700 1 --2- 172.21.15.70:0/2260168410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0a8650 0x5581cb0a7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.691+0000 7f298affd700 1 -- 172.21.15.70:0/2260168410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0ac7c0 msgr2=0x5581cb0aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:37.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.691+0000 7f298affd700 1 --2- 172.21.15.70:0/2260168410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0ac7c0 0x5581cb0aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.691+0000 7f298affd700 1 -- 172.21.15.70:0/2260168410 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581cb0a0f80 con 0x5581cb0a7c90 2024-06-04T05:21:37.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.691+0000 7f298a7fc700 1 --2- 172.21.15.70:0/2260168410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0a8650 0x5581cb0a7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:21:37.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.691+0000 7f298b7fe700 1 --2- 172.21.15.70:0/2260168410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0ac7c0 0x5581cb0aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:21:37.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.691+0000 7f298affd700 1 --2- 172.21.15.70:0/2260168410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a7c90 0x5581cb08bf00 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f29800e95b0 tx=0x7f29800b7650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f2f0f7bb888667d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:37.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.691+0000 7f2970ff9700 1 -- 172.21.15.70:0/2260168410 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29800af3b0 con 0x5581cb0a7c90 2024-06-04T05:21:37.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.691+0000 7f2970ff9700 1 -- 172.21.15.70:0/2260168410 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2980005500 con 0x5581cb0a7c90 2024-06-04T05:21:37.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.691+0000 7f2970ff9700 1 -- 172.21.15.70:0/2260168410 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29800bba60 con 0x5581cb0a7c90 2024-06-04T05:21:37.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.692+0000 7f299bd6ab80 1 -- 172.21.15.70:0/2260168410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a7c90 msgr2=0x5581cb08bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:37.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.692+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/2260168410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a7c90 0x5581cb08bf00 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f29800e95b0 tx=0x7f29800b7650 comp rx=0 tx=0).stop 2024-06-04T05:21:37.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.692+0000 7f299bd6ab80 1 -- 172.21.15.70:0/2260168410 shutdown_connections 2024-06-04T05:21:37.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.693+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/2260168410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0ac7c0 0x5581cb0aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.693+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/2260168410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a7c90 0x5581cb08bf00 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.693+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/2260168410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0a8650 0x5581cb0a7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.693+0000 7f299bd6ab80 1 -- 172.21.15.70:0/2260168410 >> 172.21.15.70:0/2260168410 conn(0x5581cafc23d0 msgr2=0x5581cb0ae690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:37.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.693+0000 7f299bd6ab80 1 -- 172.21.15.70:0/2260168410 shutdown_connections 2024-06-04T05:21:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.693+0000 7f299bd6ab80 1 -- 172.21.15.70:0/2260168410 wait complete. 2024-06-04T05:21:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.693+0000 7f299bd6ab80 1 -- start start 2024-06-04T05:21:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.693+0000 7f299bd6ab80 1 Processor -- start 2024-06-04T05:21:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.694+0000 7f299bd6ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0a7c90 0x5581cb099300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.694+0000 7f299bd6ab80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a8650 0x5581cb099840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.694+0000 7f299bd6ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0ac7c0 0x5581cb13cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.694+0000 7f299bd6ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f298c1b5830 con 0x5581cb0a7c90 2024-06-04T05:21:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.694+0000 7f299bd6ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f298c1b5160 con 0x5581cb0ac7c0 2024-06-04T05:21:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.694+0000 7f299bd6ab80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f298c1b2320 con 0x5581cb0a8650 2024-06-04T05:21:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.694+0000 7f299bd6ab80 10 client.0 _ll_register_callbacks cb 0x5581caf7fae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.694+0000 7f298affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0a7c90 0x5581cb099300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.694+0000 7f298affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0a7c90 0x5581cb099300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60698/0 (socket says 172.21.15.70:60698) 2024-06-04T05:21:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.694+0000 7f298affd700 1 -- 172.21.15.70:0/829508498 learned_addr learned my addr 172.21.15.70:0/829508498 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:37.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.694+0000 7f298b7fe700 1 --2- 172.21.15.70:0/829508498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0ac7c0 0x5581cb13cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.694+0000 7f298a7fc700 1 --2- 172.21.15.70:0/829508498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a8650 0x5581cb099840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.694+0000 7f298affd700 1 -- 172.21.15.70:0/829508498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a8650 msgr2=0x5581cb099840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:37.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.694+0000 7f298affd700 1 --2- 172.21.15.70:0/829508498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a8650 0x5581cb099840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.695+0000 7f298affd700 1 -- 172.21.15.70:0/829508498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0ac7c0 msgr2=0x5581cb13cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:37.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.695+0000 7f298affd700 1 --2- 172.21.15.70:0/829508498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0ac7c0 0x5581cb13cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.695+0000 7f298affd700 1 -- 172.21.15.70:0/829508498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581cb13d070 con 0x5581cb0a7c90 2024-06-04T05:21:37.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.695+0000 7f298b7fe700 1 --2- 172.21.15.70:0/829508498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0ac7c0 0x5581cb13cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:21:37.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.695+0000 7f298affd700 1 --2- 172.21.15.70:0/829508498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0a7c90 0x5581cb099300 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f298000b2e0 tx=0x7f2980009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36d5e4554de9c6c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:37.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.695+0000 7f296eff5700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:21:37.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.695+0000 7f296eff5700 1 -- 172.21.15.70:0/829508498 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29800b3580 con 0x5581cb0a7c90 2024-06-04T05:21:37.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.695+0000 7f296eff5700 1 -- 172.21.15.70:0/829508498 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f29800bba60 con 0x5581cb0a7c90 2024-06-04T05:21:37.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.695+0000 7f299bd6ab80 1 -- 172.21.15.70:0/829508498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2980005500 con 0x5581cb0a7c90 2024-06-04T05:21:37.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.695+0000 7f296eff5700 1 -- 172.21.15.70:0/829508498 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f29800bbc60 con 0x5581cb0a7c90 2024-06-04T05:21:37.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.695+0000 7f299bd6ab80 10 client.21540 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:37.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.695+0000 7f299bd6ab80 1 -- 172.21.15.70:0/829508498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x5581cb13e270 con 0x5581cb0a7c90 2024-06-04T05:21:37.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.696+0000 7f296eff5700 1 -- 172.21.15.70:0/829508498 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2980005500 con 0x5581cb0a7c90 2024-06-04T05:21:37.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.696+0000 7f296eff5700 1 -- 172.21.15.70:0/829508498 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f298001d030 con 0x5581cb0a7c90 2024-06-04T05:21:37.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.696+0000 7f299bd6ab80 10 client.21540 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:37.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.696+0000 7f299bd6ab80 10 client.21540 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:21:37.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.696+0000 7f299bd6ab80 20 client.21540 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:37.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.696+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/829508498 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5581cb13e420 0x5581cb140840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.696+0000 7f299bd6ab80 4 client.21540 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-06-04T05:21:37.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.696+0000 7f299bd6ab80 1 -- 172.21.15.70:0/829508498 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x5581caff6e50 con 0x5581cb13e420 2024-06-04T05:21:37.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.696+0000 7f298a7fc700 1 --2- 172.21.15.70:0/829508498 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5581cb13e420 0x5581cb140840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.697+0000 7f296eff5700 10 client.21540 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:37.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.697+0000 7f298a7fc700 1 --2- 172.21.15.70:0/829508498 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5581cb13e420 0x5581cb140840 crc :-1 s=READY pgs=504 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a6cc09149fa7b51 server_cookie=48772d45ee9f392a in_seq=0 out_seq=0 2024-06-04T05:21:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:37 smithi098 ceph-mon[35569]: pgmap v770: 97 pgs: 97 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.7 MiB/s wr, 361 op/s 2024-06-04T05:21:37.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82547 6.98 MB/sec execute 509 sec latency 136.449 ms 2024-06-04T05:21:37.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.907+0000 7f386cf22700 1 -- 172.21.15.70:0/330168186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f386806b220 msgr2=0x7f386806b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:37.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.907+0000 7f386cf22700 1 --2- 172.21.15.70:0/330168186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f386806b220 0x7f386806b6a0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f385c0098f0 tx=0x7f385c009c00 comp rx=0 tx=0).stop 2024-06-04T05:21:37.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.907+0000 7f386cf22700 1 -- 172.21.15.70:0/330168186 shutdown_connections 2024-06-04T05:21:37.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.907+0000 7f386cf22700 1 --2- 172.21.15.70:0/330168186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f386806bbe0 0x7f38681103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.907+0000 7f386cf22700 1 --2- 172.21.15.70:0/330168186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f38681095b0 0x7f3868109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.907+0000 7f386cf22700 1 --2- 172.21.15.70:0/330168186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f386806b220 0x7f386806b6a0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.907+0000 7f386cf22700 1 -- 172.21.15.70:0/330168186 >> 172.21.15.70:0/330168186 conn(0x7f38680793f0 msgr2=0x7f3868079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:37.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.907+0000 7f386cf22700 1 -- 172.21.15.70:0/330168186 shutdown_connections 2024-06-04T05:21:37.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.907+0000 7f386cf22700 1 -- 172.21.15.70:0/330168186 wait complete. 2024-06-04T05:21:37.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.909+0000 7f386cf22700 1 Processor -- start 2024-06-04T05:21:37.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.909+0000 7f386cf22700 1 -- start start 2024-06-04T05:21:37.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.909+0000 7f386cf22700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f386806b220 0x7f386819d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.909+0000 7f386cf22700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f386806bbe0 0x7f386819db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.909+0000 7f386cf22700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38681095b0 0x7f3868197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.909+0000 7f386cf22700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3868111400 con 0x7f386806bbe0 2024-06-04T05:21:37.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.909+0000 7f386cf22700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3868111280 con 0x7f38681095b0 2024-06-04T05:21:37.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.909+0000 7f386cf22700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3868111580 con 0x7f386806b220 2024-06-04T05:21:37.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.910+0000 7f3865d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f386806bbe0 0x7f386819db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.910+0000 7f3865d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f386806bbe0 0x7f386819db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60708/0 (socket says 172.21.15.70:60708) 2024-06-04T05:21:37.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.910+0000 7f3865d9b700 1 -- 172.21.15.70:0/2731260770 learned_addr learned my addr 172.21.15.70:0/2731260770 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:37.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.910+0000 7f386659c700 1 --2- 172.21.15.70:0/2731260770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f386806b220 0x7f386819d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.910+0000 7f3866d9d700 1 --2- 172.21.15.70:0/2731260770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38681095b0 0x7f3868197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.911+0000 7f3865d9b700 1 -- 172.21.15.70:0/2731260770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f386806b220 msgr2=0x7f386819d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:37.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.911+0000 7f3865d9b700 1 --2- 172.21.15.70:0/2731260770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f386806b220 0x7f386819d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.911+0000 7f3865d9b700 1 -- 172.21.15.70:0/2731260770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38681095b0 msgr2=0x7f3868197790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:37.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.911+0000 7f3865d9b700 1 --2- 172.21.15.70:0/2731260770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38681095b0 0x7f3868197790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.911+0000 7f3865d9b700 1 -- 172.21.15.70:0/2731260770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3868197f70 con 0x7f386806bbe0 2024-06-04T05:21:37.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.911+0000 7f3865d9b700 1 --2- 172.21.15.70:0/2731260770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f386806bbe0 0x7f386819db50 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f385c000c00 tx=0x7f385c005960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:37.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.912+0000 7f38577fe700 1 -- 172.21.15.70:0/2731260770 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c025070 con 0x7f386806bbe0 2024-06-04T05:21:37.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.912+0000 7f38577fe700 1 -- 172.21.15.70:0/2731260770 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f385c00bbe0 con 0x7f386806bbe0 2024-06-04T05:21:37.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.912+0000 7f38577fe700 1 -- 172.21.15.70:0/2731260770 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c003da0 con 0x7f386806bbe0 2024-06-04T05:21:37.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.912+0000 7f386cf22700 1 -- 172.21.15.70:0/2731260770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3868198200 con 0x7f386806bbe0 2024-06-04T05:21:37.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.912+0000 7f386cf22700 1 -- 172.21.15.70:0/2731260770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38681a4470 con 0x7f386806bbe0 2024-06-04T05:21:37.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.919+0000 7f38577fe700 1 -- 172.21.15.70:0/2731260770 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f385c028070 con 0x7f386806bbe0 2024-06-04T05:21:37.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.919+0000 7f38777b7b80 1 Processor -- start 2024-06-04T05:21:37.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.919+0000 7f38777b7b80 1 -- start start 2024-06-04T05:21:37.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.920+0000 7f38777b7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55834682d070 0x55834682d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.920+0000 7f38777b7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682da20 0x558346831150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.920+0000 7f38777b7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558346831770 0x558346831bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.920+0000 7f38777b7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558346783c90 con 0x558346831770 2024-06-04T05:21:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.920+0000 7f38777b7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558346773830 con 0x55834682da20 2024-06-04T05:21:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.920+0000 7f38777b7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558346777870 con 0x55834682d070 2024-06-04T05:21:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.921+0000 7f386659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682da20 0x558346831150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.921+0000 7f386659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682da20 0x558346831150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59918/0 (socket says 172.21.15.70:59918) 2024-06-04T05:21:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.921+0000 7f386659c700 1 -- 172.21.15.70:0/983016756 learned_addr learned my addr 172.21.15.70:0/983016756 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.921+0000 7f3865d9b700 1 --2- 172.21.15.70:0/983016756 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558346831770 0x558346831bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.922+0000 7f38577fe700 1 --2- 172.21.15.70:0/2731260770 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38500b2b40 0x7f38500b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.922+0000 7f386659c700 1 -- 172.21.15.70:0/983016756 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55834682d070 msgr2=0x55834682d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.922+0000 7f3866d9d700 1 --2- 172.21.15.70:0/983016756 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55834682d070 0x55834682d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.922+0000 7f38577fe700 1 -- 172.21.15.70:0/2731260770 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f385c0e2fb0 con 0x7f386806bbe0 2024-06-04T05:21:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.922+0000 7f386659c700 1 --2- 172.21.15.70:0/983016756 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55834682d070 0x55834682d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.922+0000 7f386659c700 1 -- 172.21.15.70:0/983016756 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558346831770 msgr2=0x558346831bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.922+0000 7f386659c700 1 --2- 172.21.15.70:0/983016756 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558346831770 0x558346831bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.922+0000 7f386659c700 1 -- 172.21.15.70:0/983016756 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583466c38d0 con 0x55834682da20 2024-06-04T05:21:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.922+0000 7f3865d9b700 1 --2- 172.21.15.70:0/983016756 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558346831770 0x558346831bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:21:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.923+0000 7f386659c700 1 --2- 172.21.15.70:0/983016756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682da20 0x558346831150 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f385800f660 tx=0x7f3858009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bae7902785b6aed3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.923+0000 7f3866d9d700 1 --2- 172.21.15.70:0/2731260770 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38500b2b40 0x7f38500b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.923+0000 7f3854ff9700 1 -- 172.21.15.70:0/983016756 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385801d070 con 0x55834682da20 2024-06-04T05:21:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.923+0000 7f3854ff9700 1 -- 172.21.15.70:0/983016756 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3858003680 con 0x55834682da20 2024-06-04T05:21:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.923+0000 7f3866d9d700 1 --2- 172.21.15.70:0/2731260770 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38500b2b40 0x7f38500b5000 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3860009e30 tx=0x7f3860004b40 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.923+0000 7f3854ff9700 1 -- 172.21.15.70:0/983016756 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858009830 con 0x55834682da20 2024-06-04T05:21:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.924+0000 7f38777b7b80 1 -- 172.21.15.70:0/983016756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682da20 msgr2=0x558346831150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.924+0000 7f38777b7b80 1 --2- 172.21.15.70:0/983016756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682da20 0x558346831150 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f385800f660 tx=0x7f3858009310 comp rx=0 tx=0).stop 2024-06-04T05:21:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.925+0000 7f38777b7b80 1 -- 172.21.15.70:0/983016756 shutdown_connections 2024-06-04T05:21:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.925+0000 7f38777b7b80 1 --2- 172.21.15.70:0/983016756 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558346831770 0x558346831bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.925+0000 7f38777b7b80 1 --2- 172.21.15.70:0/983016756 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55834682d070 0x55834682d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.925+0000 7f38777b7b80 1 --2- 172.21.15.70:0/983016756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682da20 0x558346831150 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.925+0000 7f38777b7b80 1 -- 172.21.15.70:0/983016756 >> 172.21.15.70:0/983016756 conn(0x558346752080 msgr2=0x55834681ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.925+0000 7f38777b7b80 1 -- 172.21.15.70:0/983016756 shutdown_connections 2024-06-04T05:21:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.925+0000 7f38777b7b80 1 -- 172.21.15.70:0/983016756 wait complete. 2024-06-04T05:21:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.926+0000 7f38777b7b80 1 -- start start 2024-06-04T05:21:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.926+0000 7f38777b7b80 1 Processor -- start 2024-06-04T05:21:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.927+0000 7f38777b7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682d070 0x558346838b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.927+0000 7f38777b7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55834682da20 0x558346835e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.927+0000 7f38777b7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558346831770 0x5583468363b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.927+0000 7f3865d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55834682da20 0x558346835e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.927+0000 7f3865d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55834682da20 0x558346835e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36948/0 (socket says 172.21.15.70:36948) 2024-06-04T05:21:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.927+0000 7f3865d9b700 1 -- 172.21.15.70:0/133190918 learned_addr learned my addr 172.21.15.70:0/133190918 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.927+0000 7f386659c700 1 --2- 172.21.15.70:0/133190918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682d070 0x558346838b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.927+0000 7f38777b7b80 1 -- 172.21.15.70:0/133190918 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558346679fa0 con 0x558346831770 2024-06-04T05:21:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.927+0000 7f38777b7b80 1 -- 172.21.15.70:0/133190918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5583467157c0 con 0x55834682d070 2024-06-04T05:21:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.927+0000 7f38777b7b80 1 -- 172.21.15.70:0/133190918 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558346839980 con 0x55834682da20 2024-06-04T05:21:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.927+0000 7f38777b7b80 10 client.0 _ll_register_callbacks cb 0x55834670e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.927+0000 7f3866d9d700 1 --2- 172.21.15.70:0/133190918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558346831770 0x5583468363b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.928+0000 7f386659c700 1 -- 172.21.15.70:0/133190918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55834682da20 msgr2=0x558346835e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.928+0000 7f386659c700 1 --2- 172.21.15.70:0/133190918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55834682da20 0x558346835e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.928+0000 7f386659c700 1 -- 172.21.15.70:0/133190918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558346831770 msgr2=0x5583468363b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.928+0000 7f386659c700 1 --2- 172.21.15.70:0/133190918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558346831770 0x5583468363b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.928+0000 7f386659c700 1 -- 172.21.15.70:0/133190918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558346712690 con 0x55834682d070 2024-06-04T05:21:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.928+0000 7f386659c700 1 --2- 172.21.15.70:0/133190918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682d070 0x558346838b70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3858006010 tx=0x7f38580043f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=289e1865242f6928 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.928+0000 7f384dffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:21:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.928+0000 7f384dffb700 1 -- 172.21.15.70:0/133190918 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858029060 con 0x55834682d070 2024-06-04T05:21:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.928+0000 7f384dffb700 1 -- 172.21.15.70:0/133190918 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f3858005240 con 0x55834682d070 2024-06-04T05:21:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.929+0000 7f38777b7b80 1 -- 172.21.15.70:0/133190918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3858003680 con 0x55834682d070 2024-06-04T05:21:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.929+0000 7f384dffb700 1 -- 172.21.15.70:0/133190918 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f385801d070 con 0x55834682d070 2024-06-04T05:21:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.929+0000 7f38777b7b80 10 client.30197 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.929+0000 7f38777b7b80 1 -- 172.21.15.70:0/133190918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x558346844090 con 0x55834682d070 2024-06-04T05:21:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.929+0000 7f384dffb700 1 -- 172.21.15.70:0/133190918 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3858003680 con 0x55834682d070 2024-06-04T05:21:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.930+0000 7f384dffb700 1 -- 172.21.15.70:0/133190918 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f38580095d0 con 0x55834682d070 2024-06-04T05:21:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.930+0000 7f38777b7b80 10 client.30197 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.930+0000 7f38777b7b80 10 client.30197 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:21:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.930+0000 7f38777b7b80 20 client.30197 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.930+0000 7f38777b7b80 1 --2- 172.21.15.70:0/133190918 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558346846670 0x558346848a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.930+0000 7f38777b7b80 4 client.30197 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:21:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.930+0000 7f38777b7b80 1 -- 172.21.15.70:0/133190918 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583467f09e0 con 0x558346846670 2024-06-04T05:21:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.930+0000 7f3865d9b700 1 --2- 172.21.15.70:0/133190918 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558346846670 0x558346848a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.931+0000 7f3865d9b700 1 --2- 172.21.15.70:0/133190918 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558346846670 0x558346848a60 crc :-1 s=READY pgs=505 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=233f294f826646ef server_cookie=c1cab976c1849706 in_seq=0 out_seq=0 2024-06-04T05:21:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.931+0000 7f384dffb700 10 client.30197 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:37.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.936+0000 7f384dffb700 1 -- 172.21.15.70:0/133190918 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5583467f09e0 con 0x558346846670 2024-06-04T05:21:37.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.936+0000 7f384dffb700 10 client.30197 handle_command_reply: tid=0 2024-06-04T05:21:37.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.936+0000 7f38777b7b80 1 client.30197 shutdown 2024-06-04T05:21:37.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.937+0000 7f38777b7b80 1 -- 172.21.15.70:0/133190918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682d070 msgr2=0x558346838b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:37.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.937+0000 7f38777b7b80 1 --2- 172.21.15.70:0/133190918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682d070 0x558346838b70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3858006010 tx=0x7f38580043f0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.937+0000 7f38777b7b80 1 -- 172.21.15.70:0/133190918 shutdown_connections 2024-06-04T05:21:37.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.937+0000 7f38777b7b80 1 --2- 172.21.15.70:0/133190918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558346831770 0x5583468363b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.937+0000 7f38777b7b80 1 --2- 172.21.15.70:0/133190918 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x558346846670 0x558346848a60 crc :-1 s=READY pgs=505 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.937+0000 7f38777b7b80 1 --2- 172.21.15.70:0/133190918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682d070 0x558346838b70 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.937+0000 7f38777b7b80 1 --2- 172.21.15.70:0/133190918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55834682da20 0x558346835e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.937+0000 7f384dffb700 0 client.30197 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:37.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.937+0000 7f38777b7b80 1 -- 172.21.15.70:0/133190918 >> 172.21.15.70:0/133190918 conn(0x558346752080 msgr2=0x558346832b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:37.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.937+0000 7f38777b7b80 1 -- 172.21.15.70:0/133190918 shutdown_connections 2024-06-04T05:21:37.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.937+0000 7f38777b7b80 1 -- 172.21.15.70:0/133190918 wait complete. 2024-06-04T05:21:37.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.938+0000 7f38777b7b80 20 client.30197 trim_cache size 0 max 16384 2024-06-04T05:21:37.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.973+0000 7f38777b7b80 1 Processor -- start 2024-06-04T05:21:37.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.973+0000 7f38777b7b80 1 -- start start 2024-06-04T05:21:37.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.973+0000 7f38777b7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55834682d070 0x558346834dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.973+0000 7f38777b7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55834682da20 0x558346820890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.974+0000 7f38777b7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558346831770 0x55834684ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.974+0000 7f38777b7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f38681abaa0 con 0x55834682da20 2024-06-04T05:21:37.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.974+0000 7f38777b7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f38681ad670 con 0x558346831770 2024-06-04T05:21:37.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.974+0000 7f38777b7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f38681ac1d0 con 0x55834682d070 2024-06-04T05:21:37.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.974+0000 7f3865d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55834682da20 0x558346820890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.974+0000 7f3865d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55834682da20 0x558346820890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60744/0 (socket says 172.21.15.70:60744) 2024-06-04T05:21:37.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.974+0000 7f3865d9b700 1 -- 172.21.15.70:0/503901336 learned_addr learned my addr 172.21.15.70:0/503901336 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:37.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.974+0000 7f3866d9d700 1 --2- 172.21.15.70:0/503901336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558346831770 0x55834684ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.974+0000 7f386659c700 1 --2- 172.21.15.70:0/503901336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55834682d070 0x558346834dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.975+0000 7f3865d9b700 1 -- 172.21.15.70:0/503901336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55834682d070 msgr2=0x558346834dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:37.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.975+0000 7f3865d9b700 1 --2- 172.21.15.70:0/503901336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55834682d070 0x558346834dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.975+0000 7f3865d9b700 1 -- 172.21.15.70:0/503901336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558346831770 msgr2=0x55834684ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:37.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.975+0000 7f3865d9b700 1 --2- 172.21.15.70:0/503901336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558346831770 0x55834684ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.975+0000 7f3865d9b700 1 -- 172.21.15.70:0/503901336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558346836730 con 0x55834682da20 2024-06-04T05:21:37.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.975+0000 7f3865d9b700 1 --2- 172.21.15.70:0/503901336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55834682da20 0x558346820890 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f385c0a7500 tx=0x7f385c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e021881b725ae0d7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:37.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.976+0000 7f384ffff700 1 -- 172.21.15.70:0/503901336 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c0a9da0 con 0x55834682da20 2024-06-04T05:21:37.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.976+0000 7f384ffff700 1 -- 172.21.15.70:0/503901336 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f385c0aa990 con 0x55834682da20 2024-06-04T05:21:37.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.976+0000 7f384ffff700 1 -- 172.21.15.70:0/503901336 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c0e08f0 con 0x55834682da20 2024-06-04T05:21:37.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.977+0000 7f38777b7b80 1 -- 172.21.15.70:0/503901336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55834682da20 msgr2=0x558346820890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:37.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.977+0000 7f38777b7b80 1 --2- 172.21.15.70:0/503901336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55834682da20 0x558346820890 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f385c0a7500 tx=0x7f385c005880 comp rx=0 tx=0).stop 2024-06-04T05:21:37.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.978+0000 7f38777b7b80 1 -- 172.21.15.70:0/503901336 shutdown_connections 2024-06-04T05:21:37.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.978+0000 7f38777b7b80 1 --2- 172.21.15.70:0/503901336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558346831770 0x55834684ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.978+0000 7f38777b7b80 1 --2- 172.21.15.70:0/503901336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55834682d070 0x558346834dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.978+0000 7f38777b7b80 1 --2- 172.21.15.70:0/503901336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55834682da20 0x558346820890 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.978+0000 7f38777b7b80 1 -- 172.21.15.70:0/503901336 >> 172.21.15.70:0/503901336 conn(0x558346752080 msgr2=0x558346823f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:37.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.978+0000 7f38777b7b80 1 -- 172.21.15.70:0/503901336 shutdown_connections 2024-06-04T05:21:37.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.978+0000 7f38777b7b80 1 -- 172.21.15.70:0/503901336 wait complete. 2024-06-04T05:21:37.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.979+0000 7f38777b7b80 1 -- start start 2024-06-04T05:21:37.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.979+0000 7f38777b7b80 1 Processor -- start 2024-06-04T05:21:37.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.980+0000 7f38777b7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682d070 0x55834684bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.980+0000 7f38777b7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55834682da20 0x558346847240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.980+0000 7f38777b7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558346831770 0x558346847780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.980+0000 7f38777b7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f38681acf70 con 0x55834682da20 2024-06-04T05:21:37.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.980+0000 7f38777b7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f38681ac8a0 con 0x55834682d070 2024-06-04T05:21:37.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.980+0000 7f38777b7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f38681a9a20 con 0x558346831770 2024-06-04T05:21:37.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.980+0000 7f3866d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558346831770 0x558346847780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.980+0000 7f3865d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55834682da20 0x558346847240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.980+0000 7f3866d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558346831770 0x558346847780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:36966/0 (socket says 172.21.15.70:36966) 2024-06-04T05:21:37.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.980+0000 7f3865d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55834682da20 0x558346847240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60760/0 (socket says 172.21.15.70:60760) 2024-06-04T05:21:37.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.980+0000 7f3866d9d700 1 -- 172.21.15.70:0/1641350076 learned_addr learned my addr 172.21.15.70:0/1641350076 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:37.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.980+0000 7f38777b7b80 10 client.0 _ll_register_callbacks cb 0x55834670e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:37.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.980+0000 7f386659c700 1 --2- 172.21.15.70:0/1641350076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682d070 0x55834684bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.981+0000 7f3866d9d700 1 -- 172.21.15.70:0/1641350076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682d070 msgr2=0x55834684bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:37.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.981+0000 7f3866d9d700 1 --2- 172.21.15.70:0/1641350076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682d070 0x55834684bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.981+0000 7f3866d9d700 1 -- 172.21.15.70:0/1641350076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55834682da20 msgr2=0x558346847240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:37.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.981+0000 7f3866d9d700 1 --2- 172.21.15.70:0/1641350076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55834682da20 0x558346847240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.981+0000 7f3866d9d700 1 -- 172.21.15.70:0/1641350076 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558346847cf0 con 0x558346831770 2024-06-04T05:21:37.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.981+0000 7f3866d9d700 1 --2- 172.21.15.70:0/1641350076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558346831770 0x558346847780 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f3860016920 tx=0x7f3860004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d872b75d4501480a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:37.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.981+0000 7f384dffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:21:37.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.982+0000 7f384dffb700 1 -- 172.21.15.70:0/1641350076 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3860012040 con 0x558346831770 2024-06-04T05:21:37.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.982+0000 7f38777b7b80 1 -- 172.21.15.70:0/1641350076 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f385c0aa990 con 0x558346831770 2024-06-04T05:21:37.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.982+0000 7f384dffb700 1 -- 172.21.15.70:0/1641350076 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f3860015a00 con 0x558346831770 2024-06-04T05:21:37.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.982+0000 7f384dffb700 1 -- 172.21.15.70:0/1641350076 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f3860032050 con 0x558346831770 2024-06-04T05:21:37.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.982+0000 7f38777b7b80 10 client.30340 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:37.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.982+0000 7f38777b7b80 1 -- 172.21.15.70:0/1641350076 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55834682b4d0 con 0x558346831770 2024-06-04T05:21:37.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.983+0000 7f384dffb700 1 -- 172.21.15.70:0/1641350076 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f385c0aa990 con 0x558346831770 2024-06-04T05:21:37.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.983+0000 7f384dffb700 1 -- 172.21.15.70:0/1641350076 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f3860024ca0 con 0x558346831770 2024-06-04T05:21:37.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.983+0000 7f38777b7b80 10 client.30340 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:37.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.983+0000 7f38777b7b80 10 client.30340 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:21:37.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.983+0000 7f38777b7b80 20 client.30340 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:37.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.984+0000 7f38777b7b80 1 --2- 172.21.15.70:0/1641350076 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55834682bc40 0x55834682c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:37.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.984+0000 7f38777b7b80 4 client.30340 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:21:37.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.984+0000 7f38777b7b80 1 -- 172.21.15.70:0/1641350076 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558346785be0 con 0x55834682bc40 2024-06-04T05:21:37.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.984+0000 7f386659c700 1 --2- 172.21.15.70:0/1641350076 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55834682bc40 0x55834682c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:37.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.985+0000 7f386659c700 1 --2- 172.21.15.70:0/1641350076 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55834682bc40 0x55834682c050 crc :-1 s=READY pgs=506 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e88b199a6663a06 server_cookie=96bf130759fd4d6 in_seq=0 out_seq=0 2024-06-04T05:21:37.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.986+0000 7f384dffb700 10 client.30340 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:37.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.991+0000 7f384dffb700 1 -- 172.21.15.70:0/1641350076 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558346785be0 con 0x55834682bc40 2024-06-04T05:21:37.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.991+0000 7f384dffb700 10 client.30340 handle_command_reply: tid=0 2024-06-04T05:21:37.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.992+0000 7f38777b7b80 1 client.30340 shutdown 2024-06-04T05:21:37.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.992+0000 7f38777b7b80 1 -- 172.21.15.70:0/1641350076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558346831770 msgr2=0x558346847780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:37.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.992+0000 7f38777b7b80 1 --2- 172.21.15.70:0/1641350076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558346831770 0x558346847780 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f3860016920 tx=0x7f3860004ad0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.993+0000 7f38777b7b80 1 -- 172.21.15.70:0/1641350076 shutdown_connections 2024-06-04T05:21:37.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.993+0000 7f38777b7b80 1 --2- 172.21.15.70:0/1641350076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558346831770 0x558346847780 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.993+0000 7f38777b7b80 1 --2- 172.21.15.70:0/1641350076 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55834682bc40 0x55834682c050 crc :-1 s=READY pgs=506 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.993+0000 7f38777b7b80 1 --2- 172.21.15.70:0/1641350076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55834682d070 0x55834684bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.993+0000 7f38777b7b80 1 --2- 172.21.15.70:0/1641350076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55834682da20 0x558346847240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.993+0000 7f38777b7b80 1 -- 172.21.15.70:0/1641350076 >> 172.21.15.70:0/1641350076 conn(0x558346752080 msgr2=0x5583468376a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:37.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.993+0000 7f384dffb700 0 client.30340 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:37.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.993+0000 7f38777b7b80 1 -- 172.21.15.70:0/1641350076 shutdown_connections 2024-06-04T05:21:37.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.993+0000 7f38777b7b80 1 -- 172.21.15.70:0/1641350076 wait complete. 2024-06-04T05:21:37.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.993+0000 7f38777b7b80 20 client.30340 trim_cache size 0 max 16384 2024-06-04T05:21:37.998 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:21:37.998 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:21:37.998 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:21:37.998 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:21:37.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.996+0000 7f386cf22700 1 -- 172.21.15.70:0/2731260770 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38500b2b40 msgr2=0x7f38500b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:37.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.996+0000 7f386cf22700 1 --2- 172.21.15.70:0/2731260770 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38500b2b40 0x7f38500b5000 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3860009e30 tx=0x7f3860004b40 comp rx=0 tx=0).stop 2024-06-04T05:21:37.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.996+0000 7f386cf22700 1 -- 172.21.15.70:0/2731260770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f386806bbe0 msgr2=0x7f386819db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:37.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.996+0000 7f386cf22700 1 --2- 172.21.15.70:0/2731260770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f386806bbe0 0x7f386819db50 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f385c000c00 tx=0x7f385c005960 comp rx=0 tx=0).stop 2024-06-04T05:21:37.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.996+0000 7f386cf22700 1 -- 172.21.15.70:0/2731260770 shutdown_connections 2024-06-04T05:21:37.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.996+0000 7f386cf22700 1 --2- 172.21.15.70:0/2731260770 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f38500b2b40 0x7f38500b5000 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.996+0000 7f386cf22700 1 --2- 172.21.15.70:0/2731260770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38681095b0 0x7f3868197790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:37.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.996+0000 7f386cf22700 1 --2- 172.21.15.70:0/2731260770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f386806b220 0x7f386819d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:38.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.996+0000 7f386cf22700 1 --2- 172.21.15.70:0/2731260770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f386806bbe0 0x7f386819db50 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:38.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.996+0000 7f386cf22700 1 -- 172.21.15.70:0/2731260770 >> 172.21.15.70:0/2731260770 conn(0x7f38680793f0 msgr2=0x7f3868101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:38.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.997+0000 7f386cf22700 1 -- 172.21.15.70:0/2731260770 shutdown_connections 2024-06-04T05:21:38.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:37.997+0000 7f386cf22700 1 -- 172.21.15.70:0/2731260770 wait complete. 2024-06-04T05:21:38.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.058+0000 7f296eff5700 1 -- 172.21.15.70:0/829508498 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5581caff6e50 con 0x5581cb13e420 2024-06-04T05:21:38.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.058+0000 7f296eff5700 10 client.21540 handle_command_reply: tid=0 2024-06-04T05:21:38.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.058+0000 7f299bd6ab80 1 client.21540 shutdown 2024-06-04T05:21:38.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.059+0000 7f299bd6ab80 1 -- 172.21.15.70:0/829508498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0a7c90 msgr2=0x5581cb099300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:38.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.059+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/829508498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0a7c90 0x5581cb099300 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f298000b2e0 tx=0x7f2980009ca0 comp rx=0 tx=0).stop 2024-06-04T05:21:38.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.059+0000 7f299bd6ab80 1 -- 172.21.15.70:0/829508498 shutdown_connections 2024-06-04T05:21:38.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.059+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/829508498 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5581cb13e420 0x5581cb140840 crc :-1 s=READY pgs=504 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:38.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.059+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/829508498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581cb0ac7c0 0x5581cb13cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:38.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.059+0000 7f296eff5700 0 client.21540 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:38.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.059+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/829508498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581cb0a7c90 0x5581cb099300 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:38.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.059+0000 7f299bd6ab80 1 --2- 172.21.15.70:0/829508498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5581cb0a8650 0x5581cb099840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:38.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.059+0000 7f299bd6ab80 1 -- 172.21.15.70:0/829508498 >> 172.21.15.70:0/829508498 conn(0x5581cafc23d0 msgr2=0x5581cb0aef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:38.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.059+0000 7f299bd6ab80 1 -- 172.21.15.70:0/829508498 shutdown_connections 2024-06-04T05:21:38.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.059+0000 7f299bd6ab80 1 -- 172.21.15.70:0/829508498 wait complete. 2024-06-04T05:21:38.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.059+0000 7f299bd6ab80 20 client.21540 trim_cache size 0 max 16384 2024-06-04T05:21:38.064 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:21:38.064 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 64, 2024-06-04T05:21:38.064 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:21:38.064 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 74, 2024-06-04T05:21:38.064 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:21:38.064 INFO:teuthology.orchestra.run.smithi070.stdout: "c80d4a61": { 2024-06-04T05:21:38.064 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 74, 2024-06-04T05:21:38.064 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:21:38.065 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:21:38.065 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:21:38.065 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:21:38.065 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:21:38.065 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:21:38.065 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 150.0, 2024-06-04T05:21:38.065 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:21:38.065 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:21:38.065 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:21:38.065 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:21:38.065 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:21:38.065 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:21:38.065 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:21:38.065 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:21:38.066 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:21:38.066 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:21:38.066 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:21:38.066 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:21:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.060+0000 7f29914d5700 1 -- 172.21.15.70:0/1932198398 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f29740b2be0 msgr2=0x7f29740b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.060+0000 7f29914d5700 1 --2- 172.21.15.70:0/1932198398 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f29740b2be0 0x7f29740b50a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f298c06e7f0 tx=0x7f298400b040 comp rx=0 tx=0).stop 2024-06-04T05:21:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.060+0000 7f29914d5700 1 -- 172.21.15.70:0/1932198398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f298c107450 msgr2=0x7f298c075060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.060+0000 7f29914d5700 1 --2- 172.21.15.70:0/1932198398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f298c107450 0x7f298c075060 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f29800047b0 tx=0x7f298000bf80 comp rx=0 tx=0).stop 2024-06-04T05:21:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.060+0000 7f29914d5700 1 -- 172.21.15.70:0/1932198398 shutdown_connections 2024-06-04T05:21:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.060+0000 7f29914d5700 1 --2- 172.21.15.70:0/1932198398 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f29740b2be0 0x7f29740b50a0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.060+0000 7f29914d5700 1 --2- 172.21.15.70:0/1932198398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f298c108b50 0x7f298c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.060+0000 7f29914d5700 1 --2- 172.21.15.70:0/1932198398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f298c107450 0x7f298c075060 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.060+0000 7f29914d5700 1 --2- 172.21.15.70:0/1932198398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f298c108190 0x7f298c06d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.060+0000 7f29914d5700 1 -- 172.21.15.70:0/1932198398 >> 172.21.15.70:0/1932198398 conn(0x7f298c078670 msgr2=0x7f298c0ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.061+0000 7f29914d5700 1 -- 172.21.15.70:0/1932198398 shutdown_connections 2024-06-04T05:21:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:38.061+0000 7f29914d5700 1 -- 172.21.15.70:0/1932198398 wait complete. 2024-06-04T05:21:38.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.97 MB/sec execute 510 sec latency 1120.001 ms 2024-06-04T05:21:39.075 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c80d4a61', quiesce took 19.9 seconds. Will release after: 10.100000000000001 2024-06-04T05:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.512+0000 7f5f40262640 1 -- 172.21.15.70:0/208712542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f3806ef70 msgr2=0x7f5f3811bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.512+0000 7f5f40262640 1 --2- 172.21.15.70:0/208712542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f3806ef70 0x7f5f3811bbb0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009900 tx=0x7f5f2c02a410 comp rx=0 tx=0).stop 2024-06-04T05:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.512+0000 7f5f40262640 1 -- 172.21.15.70:0/208712542 shutdown_connections 2024-06-04T05:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.512+0000 7f5f40262640 1 --2- 172.21.15.70:0/208712542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f3811c0f0 0x7f5f3811e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.512+0000 7f5f40262640 1 --2- 172.21.15.70:0/208712542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f3806ef70 0x7f5f3811bbb0 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.512+0000 7f5f40262640 1 --2- 172.21.15.70:0/208712542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3806e650 0x7f5f3806ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.512+0000 7f5f40262640 1 -- 172.21.15.70:0/208712542 >> 172.21.15.70:0/208712542 conn(0x7f5f3806dcb0 msgr2=0x7f5f3810ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.512+0000 7f5f40262640 1 -- 172.21.15.70:0/208712542 shutdown_connections 2024-06-04T05:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.512+0000 7f5f40262640 1 -- 172.21.15.70:0/208712542 wait complete. 2024-06-04T05:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.513+0000 7f5f40262640 1 Processor -- start 2024-06-04T05:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.513+0000 7f5f40262640 1 -- start start 2024-06-04T05:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.513+0000 7f5f40262640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f3806e650 0x7f5f381a7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.513+0000 7f5f40262640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3806ef70 0x7f5f381a83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.513+0000 7f5f40262640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f3811c0f0 0x7f5f381ac730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.513+0000 7f5f40262640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5f38121950 con 0x7f5f3811c0f0 2024-06-04T05:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.513+0000 7f5f40262640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5f381217d0 con 0x7f5f3806ef70 2024-06-04T05:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.513+0000 7f5f40262640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5f38121ad0 con 0x7f5f3806e650 2024-06-04T05:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.513+0000 7f5f3e7ff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f3811c0f0 0x7f5f381ac730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.513+0000 7f5f3e7ff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f3811c0f0 0x7f5f381ac730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60766/0 (socket says 172.21.15.70:60766) 2024-06-04T05:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.513+0000 7f5f3e7ff640 1 -- 172.21.15.70:0/2615652082 learned_addr learned my addr 172.21.15.70:0/2615652082 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.513+0000 7f5f3d7fd640 1 --2- 172.21.15.70:0/2615652082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3806ef70 0x7f5f381a83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.514+0000 7f5f3dffe640 1 --2- 172.21.15.70:0/2615652082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f3806e650 0x7f5f381a7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.514+0000 7f5f3e7ff640 1 -- 172.21.15.70:0/2615652082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f3806e650 msgr2=0x7f5f381a7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.514+0000 7f5f3e7ff640 1 --2- 172.21.15.70:0/2615652082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f3806e650 0x7f5f381a7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.514+0000 7f5f3e7ff640 1 -- 172.21.15.70:0/2615652082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3806ef70 msgr2=0x7f5f381a83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.514+0000 7f5f3e7ff640 1 --2- 172.21.15.70:0/2615652082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3806ef70 0x7f5f381a83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.514+0000 7f5f3e7ff640 1 -- 172.21.15.70:0/2615652082 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f381ace60 con 0x7f5f3811c0f0 2024-06-04T05:21:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.514+0000 7f5f3dffe640 1 --2- 172.21.15.70:0/2615652082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f3806e650 0x7f5f381a7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:21:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.514+0000 7f5f3d7fd640 1 --2- 172.21.15.70:0/2615652082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3806ef70 0x7f5f381a83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:21:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.514+0000 7f5f3e7ff640 1 --2- 172.21.15.70:0/2615652082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f3811c0f0 0x7f5f381ac730 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f5f3000d750 tx=0x7f5f3000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.514+0000 7f5f36ffd640 1 -- 172.21.15.70:0/2615652082 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f30017070 con 0x7f5f3811c0f0 2024-06-04T05:21:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.514+0000 7f5f36ffd640 1 -- 172.21.15.70:0/2615652082 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5f300054b0 con 0x7f5f3811c0f0 2024-06-04T05:21:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.514+0000 7f5f40262640 1 -- 172.21.15.70:0/2615652082 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f381ad150 con 0x7f5f3811c0f0 2024-06-04T05:21:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.514+0000 7f5f36ffd640 1 -- 172.21.15.70:0/2615652082 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f30004b90 con 0x7f5f3811c0f0 2024-06-04T05:21:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.514+0000 7f5f40262640 1 -- 172.21.15.70:0/2615652082 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f381b49e0 con 0x7f5f3811c0f0 2024-06-04T05:21:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.516+0000 7f5f36ffd640 1 -- 172.21.15.70:0/2615652082 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f5f30004d30 con 0x7f5f3811c0f0 2024-06-04T05:21:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.516+0000 7f5f42f77740 1 Processor -- start 2024-06-04T05:21:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f42f77740 1 -- start start 2024-06-04T05:21:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f42f77740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac8f820 0x56392ac8fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f42f77740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac90140 0x56392ac9a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f42f77740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac9aa00 0x56392ac9cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f42f77740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56392ac01c30 con 0x56392ac90140 2024-06-04T05:21:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f42f77740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56392abf17d0 con 0x56392ac8f820 2024-06-04T05:21:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f42f77740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56392abf5810 con 0x56392ac9aa00 2024-06-04T05:21:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f3dffe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac90140 0x56392ac9a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f3dffe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac90140 0x56392ac9a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60782/0 (socket says 172.21.15.70:60782) 2024-06-04T05:21:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f3dffe640 1 -- 172.21.15.70:0/2930197741 learned_addr learned my addr 172.21.15.70:0/2930197741 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f3d7fd640 1 --2- 172.21.15.70:0/2930197741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac8f820 0x56392ac8fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f3d7fd640 1 --2- 172.21.15.70:0/2930197741 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac9aa00 0x56392ac9cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f36ffd640 1 --2- 172.21.15.70:0/2615652082 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5f200b2a90 0x7f5f200b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f3dffe640 1 -- 172.21.15.70:0/2930197741 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac9aa00 msgr2=0x56392ac9cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f3dffe640 1 --2- 172.21.15.70:0/2930197741 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac9aa00 0x56392ac9cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f3dffe640 1 -- 172.21.15.70:0/2930197741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac8f820 msgr2=0x56392ac8fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f3dffe640 1 --2- 172.21.15.70:0/2930197741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac8f820 0x56392ac8fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f3dffe640 1 -- 172.21.15.70:0/2930197741 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56392a906210 con 0x56392ac90140 2024-06-04T05:21:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.517+0000 7f5f3d7fd640 1 --2- 172.21.15.70:0/2930197741 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac9aa00 0x56392ac9cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:21:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.518+0000 7f5f3e7ff640 1 --2- 172.21.15.70:0/2615652082 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5f200b2a90 0x7f5f200b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.518+0000 7f5f3dffe640 1 --2- 172.21.15.70:0/2930197741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac90140 0x56392ac9a4c0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f5f28008e70 tx=0x7f5f2800d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eb211901e777d6a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.518+0000 7f5f36ffd640 1 -- 172.21.15.70:0/2615652082 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f5f300e2010 con 0x7f5f3811c0f0 2024-06-04T05:21:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.518+0000 7f5f1ffff640 1 -- 172.21.15.70:0/2930197741 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2800f830 con 0x56392ac90140 2024-06-04T05:21:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.518+0000 7f5f1ffff640 1 -- 172.21.15.70:0/2930197741 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5f28018480 con 0x56392ac90140 2024-06-04T05:21:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.518+0000 7f5f1ffff640 1 -- 172.21.15.70:0/2930197741 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2800fe60 con 0x56392ac90140 2024-06-04T05:21:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.519+0000 7f5f42f77740 1 -- 172.21.15.70:0/2930197741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac90140 msgr2=0x56392ac9a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.519+0000 7f5f42f77740 1 --2- 172.21.15.70:0/2930197741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac90140 0x56392ac9a4c0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f5f28008e70 tx=0x7f5f2800d710 comp rx=0 tx=0).stop 2024-06-04T05:21:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.519+0000 7f5f3e7ff640 1 --2- 172.21.15.70:0/2615652082 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5f200b2a90 0x7f5f200b4f50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5f300a5340 tx=0x7f5f30076e90 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.519+0000 7f5f42f77740 1 -- 172.21.15.70:0/2930197741 shutdown_connections 2024-06-04T05:21:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.519+0000 7f5f42f77740 1 --2- 172.21.15.70:0/2930197741 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac9aa00 0x56392ac9cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.519+0000 7f5f42f77740 1 --2- 172.21.15.70:0/2930197741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac90140 0x56392ac9a4c0 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.519+0000 7f5f42f77740 1 --2- 172.21.15.70:0/2930197741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac8f820 0x56392ac8fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.519+0000 7f5f42f77740 1 -- 172.21.15.70:0/2930197741 >> 172.21.15.70:0/2930197741 conn(0x56392ac8bd30 msgr2=0x56392ac8c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.519+0000 7f5f42f77740 1 -- 172.21.15.70:0/2930197741 shutdown_connections 2024-06-04T05:21:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.519+0000 7f5f42f77740 1 -- 172.21.15.70:0/2930197741 wait complete. 2024-06-04T05:21:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.519+0000 7f5f42f77740 1 -- start start 2024-06-04T05:21:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.519+0000 7f5f42f77740 1 Processor -- start 2024-06-04T05:21:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.519+0000 7f5f42f77740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac8f820 0x56392ac979a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f42f77740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac90140 0x56392ac97ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f42f77740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac9aa00 0x56392ac98420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f42f77740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56392a8d7790 con 0x56392ac90140 2024-06-04T05:21:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f42f77740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56392abad890 con 0x56392ac9aa00 2024-06-04T05:21:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f42f77740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56392ac9d630 con 0x56392ac8f820 2024-06-04T05:21:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f42f77740 10 client.0 _ll_register_callbacks cb 0x56392a905a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f3dffe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac90140 0x56392ac97ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f3dffe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac90140 0x56392ac97ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60794/0 (socket says 172.21.15.70:60794) 2024-06-04T05:21:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f3dffe640 1 -- 172.21.15.70:0/1824037592 learned_addr learned my addr 172.21.15.70:0/1824037592 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f3d7fd640 1 --2- 172.21.15.70:0/1824037592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac8f820 0x56392ac979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f3d7fd640 1 --2- 172.21.15.70:0/1824037592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac9aa00 0x56392ac98420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f3dffe640 1 -- 172.21.15.70:0/1824037592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac8f820 msgr2=0x56392ac979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f3dffe640 1 --2- 172.21.15.70:0/1824037592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac8f820 0x56392ac979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f3dffe640 1 -- 172.21.15.70:0/1824037592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac9aa00 msgr2=0x56392ac98420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f3dffe640 1 --2- 172.21.15.70:0/1824037592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac9aa00 0x56392ac98420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f3dffe640 1 -- 172.21.15.70:0/1824037592 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56392a9d2620 con 0x56392ac90140 2024-06-04T05:21:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f3dffe640 1 --2- 172.21.15.70:0/1824037592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac90140 0x56392ac97ee0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f5f28000f80 tx=0x7f5f28008bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a6f7417ccc6382 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:21:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f1d7fa640 1 -- 172.21.15.70:0/1824037592 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f28028060 con 0x56392ac90140 2024-06-04T05:21:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f1d7fa640 1 -- 172.21.15.70:0/1824037592 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f5f2800fa40 con 0x56392ac90140 2024-06-04T05:21:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f42f77740 1 -- 172.21.15.70:0/1824037592 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f28018480 con 0x56392ac90140 2024-06-04T05:21:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f1d7fa640 1 -- 172.21.15.70:0/1824037592 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f5f2800fc40 con 0x56392ac90140 2024-06-04T05:21:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f42f77740 10 client.21570 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.520+0000 7f5f42f77740 1 -- 172.21.15.70:0/1824037592 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x56392aca93a0 con 0x56392ac90140 2024-06-04T05:21:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.521+0000 7f5f1d7fa640 1 -- 172.21.15.70:0/1824037592 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5f28018480 con 0x56392ac90140 2024-06-04T05:21:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.521+0000 7f5f1d7fa640 1 -- 172.21.15.70:0/1824037592 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f5f2802bc50 con 0x56392ac90140 2024-06-04T05:21:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.521+0000 7f5f42f77740 10 client.21570 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.521+0000 7f5f42f77740 10 client.21570 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.521+0000 7f5f42f77740 20 client.21570 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.521+0000 7f5f42f77740 1 --2- 172.21.15.70:0/1824037592 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56392acaa800 0x56392acacbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.521+0000 7f5f42f77740 4 client.21570 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.521+0000 7f5f42f77740 1 -- 172.21.15.70:0/1824037592 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56392ac03b80 con 0x56392acaa800 2024-06-04T05:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.521+0000 7f5f3d7fd640 1 --2- 172.21.15.70:0/1824037592 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56392acaa800 0x56392acacbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.521+0000 7f5f1d7fa640 10 client.21570 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.521+0000 7f5f3d7fd640 1 --2- 172.21.15.70:0/1824037592 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56392acaa800 0x56392acacbf0 crc :-1 s=READY pgs=507 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=625b4db8a72b8df7 server_cookie=94ab20378af61a95 in_seq=0 out_seq=0 2024-06-04T05:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.522+0000 7f5f1d7fa640 1 -- 172.21.15.70:0/1824037592 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56392ac03b80 con 0x56392acaa800 2024-06-04T05:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.522+0000 7f5f1d7fa640 10 client.21570 handle_command_reply: tid=0 2024-06-04T05:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.522+0000 7f5f42f77740 1 client.21570 shutdown 2024-06-04T05:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.522+0000 7f5f42f77740 1 -- 172.21.15.70:0/1824037592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac90140 msgr2=0x56392ac97ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.522+0000 7f5f42f77740 1 --2- 172.21.15.70:0/1824037592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac90140 0x56392ac97ee0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f5f28000f80 tx=0x7f5f28008bb0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.522+0000 7f5f42f77740 1 -- 172.21.15.70:0/1824037592 shutdown_connections 2024-06-04T05:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.522+0000 7f5f42f77740 1 --2- 172.21.15.70:0/1824037592 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56392acaa800 0x56392acacbf0 crc :-1 s=READY pgs=507 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.522+0000 7f5f42f77740 1 --2- 172.21.15.70:0/1824037592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac9aa00 0x56392ac98420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.522+0000 7f5f42f77740 1 --2- 172.21.15.70:0/1824037592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac90140 0x56392ac97ee0 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.522+0000 7f5f1d7fa640 0 client.21570 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.522+0000 7f5f42f77740 1 --2- 172.21.15.70:0/1824037592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac8f820 0x56392ac979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.522+0000 7f5f42f77740 1 -- 172.21.15.70:0/1824037592 >> 172.21.15.70:0/1824037592 conn(0x56392ac8bd30 msgr2=0x56392ac8afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.522+0000 7f5f42f77740 1 -- 172.21.15.70:0/1824037592 shutdown_connections 2024-06-04T05:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.522+0000 7f5f42f77740 1 -- 172.21.15.70:0/1824037592 wait complete. 2024-06-04T05:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.522+0000 7f5f42f77740 20 client.21570 trim_cache size 0 max 16384 2024-06-04T05:21:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.537+0000 7f5f42f77740 1 Processor -- start 2024-06-04T05:21:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.537+0000 7f5f42f77740 1 -- start start 2024-06-04T05:21:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f42f77740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac8f820 0x56392ac98240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f42f77740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac90140 0x56392acb2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f42f77740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac9aa00 0x56392acb2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f42f77740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5f381bd4a0 con 0x56392ac8f820 2024-06-04T05:21:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f42f77740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5f381bf070 con 0x56392ac90140 2024-06-04T05:21:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f42f77740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5f381bdbd0 con 0x56392ac9aa00 2024-06-04T05:21:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f3d7fd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac8f820 0x56392ac98240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f3d7fd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac8f820 0x56392ac98240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60800/0 (socket says 172.21.15.70:60800) 2024-06-04T05:21:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f3d7fd640 1 -- 172.21.15.70:0/1858390752 learned_addr learned my addr 172.21.15.70:0/1858390752 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f3d7fd640 1 --2- 172.21.15.70:0/1858390752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac9aa00 0x56392acb2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f3dffe640 1 --2- 172.21.15.70:0/1858390752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac90140 0x56392acb2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f3d7fd640 1 -- 172.21.15.70:0/1858390752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac9aa00 msgr2=0x56392acb2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f3d7fd640 1 --2- 172.21.15.70:0/1858390752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac9aa00 0x56392acb2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f3d7fd640 1 -- 172.21.15.70:0/1858390752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac90140 msgr2=0x56392acb2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f3d7fd640 1 --2- 172.21.15.70:0/1858390752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac90140 0x56392acb2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f3d7fd640 1 -- 172.21.15.70:0/1858390752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56392aca6690 con 0x56392ac8f820 2024-06-04T05:21:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f3d7fd640 1 --2- 172.21.15.70:0/1858390752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac8f820 0x56392ac98240 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c000c00 tx=0x7f5f2c032b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695f09c7977b9e7f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f1f7fe640 1 -- 172.21.15.70:0/1858390752 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c035040 con 0x56392ac8f820 2024-06-04T05:21:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f1f7fe640 1 -- 172.21.15.70:0/1858390752 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5f2c052070 con 0x56392ac8f820 2024-06-04T05:21:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.538+0000 7f5f1f7fe640 1 -- 172.21.15.70:0/1858390752 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c0325b0 con 0x56392ac8f820 2024-06-04T05:21:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.539+0000 7f5f42f77740 1 -- 172.21.15.70:0/1858390752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac8f820 msgr2=0x56392ac98240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.539+0000 7f5f42f77740 1 --2- 172.21.15.70:0/1858390752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac8f820 0x56392ac98240 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c000c00 tx=0x7f5f2c032b80 comp rx=0 tx=0).stop 2024-06-04T05:21:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.539+0000 7f5f42f77740 1 -- 172.21.15.70:0/1858390752 shutdown_connections 2024-06-04T05:21:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.539+0000 7f5f42f77740 1 --2- 172.21.15.70:0/1858390752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac90140 0x56392acb2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.539+0000 7f5f42f77740 1 --2- 172.21.15.70:0/1858390752 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac9aa00 0x56392acb2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.539+0000 7f5f42f77740 1 --2- 172.21.15.70:0/1858390752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac8f820 0x56392ac98240 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.539+0000 7f5f42f77740 1 -- 172.21.15.70:0/1858390752 >> 172.21.15.70:0/1858390752 conn(0x56392ac8bd30 msgr2=0x56392ac86e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.539+0000 7f5f42f77740 1 -- 172.21.15.70:0/1858390752 shutdown_connections 2024-06-04T05:21:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.539+0000 7f5f42f77740 1 -- 172.21.15.70:0/1858390752 wait complete. 2024-06-04T05:21:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.539+0000 7f5f42f77740 1 -- start start 2024-06-04T05:21:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.539+0000 7f5f42f77740 1 Processor -- start 2024-06-04T05:21:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f42f77740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac8f820 0x56392ad0d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f42f77740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac90140 0x56392ad0d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f42f77740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac9aa00 0x56392ac9de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f42f77740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5f381be970 con 0x56392ac8f820 2024-06-04T05:21:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f42f77740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5f381be2a0 con 0x56392ac90140 2024-06-04T05:21:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f42f77740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5f381bb4a0 con 0x56392ac9aa00 2024-06-04T05:21:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f42f77740 10 client.0 _ll_register_callbacks cb 0x7f5f2c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f3d7fd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac8f820 0x56392ad0d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f3d7fd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac8f820 0x56392ad0d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60808/0 (socket says 172.21.15.70:60808) 2024-06-04T05:21:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f3d7fd640 1 -- 172.21.15.70:0/809734441 learned_addr learned my addr 172.21.15.70:0/809734441 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f3dffe640 1 --2- 172.21.15.70:0/809734441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac90140 0x56392ad0d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f3d7fd640 1 --2- 172.21.15.70:0/809734441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac9aa00 0x56392ac9de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f3d7fd640 1 -- 172.21.15.70:0/809734441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac9aa00 msgr2=0x56392ac9de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f3d7fd640 1 --2- 172.21.15.70:0/809734441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac9aa00 0x56392ac9de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f3d7fd640 1 -- 172.21.15.70:0/809734441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac90140 msgr2=0x56392ad0d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f3d7fd640 1 --2- 172.21.15.70:0/809734441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac90140 0x56392ad0d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f3d7fd640 1 -- 172.21.15.70:0/809734441 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56392ac9e530 con 0x56392ac8f820 2024-06-04T05:21:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f3d7fd640 1 --2- 172.21.15.70:0/809734441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac8f820 0x56392ad0d3f0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c0092a0 tx=0x7f5f2c009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea64187bf97a4a68 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:21:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f1d7fa640 1 -- 172.21.15.70:0/809734441 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c035050 con 0x56392ac8f820 2024-06-04T05:21:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f1d7fa640 1 -- 172.21.15.70:0/809734441 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f5f2c005240 con 0x56392ac8f820 2024-06-04T05:21:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f42f77740 1 -- 172.21.15.70:0/809734441 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f2c052070 con 0x56392ac8f820 2024-06-04T05:21:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f1d7fa640 1 -- 172.21.15.70:0/809734441 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f5f2c069050 con 0x56392ac8f820 2024-06-04T05:21:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.540+0000 7f5f42f77740 10 client.21576 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.541+0000 7f5f42f77740 1 -- 172.21.15.70:0/809734441 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x56392ad14b80 con 0x56392ac8f820 2024-06-04T05:21:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.541+0000 7f5f1d7fa640 1 -- 172.21.15.70:0/809734441 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5f2c052070 con 0x56392ac8f820 2024-06-04T05:21:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.541+0000 7f5f1d7fa640 1 -- 172.21.15.70:0/809734441 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f5f2c03a040 con 0x56392ac8f820 2024-06-04T05:21:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.541+0000 7f5f42f77740 10 client.21576 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.541+0000 7f5f42f77740 10 client.21576 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:21:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.541+0000 7f5f42f77740 20 client.21576 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.541+0000 7f5f42f77740 1 --2- 172.21.15.70:0/809734441 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56392ad152b0 0x56392ad176d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.541+0000 7f5f42f77740 4 client.21576 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:21:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.541+0000 7f5f42f77740 1 -- 172.21.15.70:0/809734441 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56392abfd360 con 0x56392ad152b0 2024-06-04T05:21:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.541+0000 7f5f3dffe640 1 --2- 172.21.15.70:0/809734441 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56392ad152b0 0x56392ad176d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.542+0000 7f5f1d7fa640 10 client.21576 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.542+0000 7f5f3dffe640 1 --2- 172.21.15.70:0/809734441 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56392ad152b0 0x56392ad176d0 crc :-1 s=READY pgs=508 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b419b89b38e9b561 server_cookie=d938a39346173271 in_seq=0 out_seq=0 2024-06-04T05:21:39.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.619+0000 7f5f1d7fa640 1 -- 172.21.15.70:0/809734441 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56392abfd360 con 0x56392ad152b0 2024-06-04T05:21:39.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.619+0000 7f5f1d7fa640 10 client.21576 handle_command_reply: tid=0 2024-06-04T05:21:39.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.620+0000 7f5f42f77740 1 client.21576 shutdown 2024-06-04T05:21:39.623 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.620+0000 7f5f42f77740 1 -- 172.21.15.70:0/809734441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac8f820 msgr2=0x56392ad0d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.620+0000 7f5f42f77740 1 --2- 172.21.15.70:0/809734441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac8f820 0x56392ad0d3f0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c0092a0 tx=0x7f5f2c009230 comp rx=0 tx=0).stop 2024-06-04T05:21:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.620+0000 7f5f42f77740 1 -- 172.21.15.70:0/809734441 shutdown_connections 2024-06-04T05:21:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.620+0000 7f5f42f77740 1 --2- 172.21.15.70:0/809734441 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56392ad152b0 0x56392ad176d0 crc :-1 s=READY pgs=508 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.620+0000 7f5f42f77740 1 --2- 172.21.15.70:0/809734441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56392ac90140 0x56392ad0d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.620+0000 7f5f42f77740 1 --2- 172.21.15.70:0/809734441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56392ac9aa00 0x56392ac9de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.620+0000 7f5f42f77740 1 --2- 172.21.15.70:0/809734441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56392ac8f820 0x56392ad0d3f0 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.620+0000 7f5f42f77740 1 -- 172.21.15.70:0/809734441 >> 172.21.15.70:0/809734441 conn(0x56392ac8bd30 msgr2=0x56392ac991d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.620+0000 7f5f1d7fa640 0 client.21576 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.620+0000 7f5f42f77740 1 -- 172.21.15.70:0/809734441 shutdown_connections 2024-06-04T05:21:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.620+0000 7f5f42f77740 1 -- 172.21.15.70:0/809734441 wait complete. 2024-06-04T05:21:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.620+0000 7f5f42f77740 20 client.21576 trim_cache size 0 max 16384 2024-06-04T05:21:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:21:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:21:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:21:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.620+0000 7f5f40262640 1 -- 172.21.15.70:0/2615652082 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5f200b2a90 msgr2=0x7f5f200b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.620+0000 7f5f40262640 1 --2- 172.21.15.70:0/2615652082 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5f200b2a90 0x7f5f200b4f50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5f300a5340 tx=0x7f5f30076e90 comp rx=0 tx=0).stop 2024-06-04T05:21:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.620+0000 7f5f40262640 1 -- 172.21.15.70:0/2615652082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f3811c0f0 msgr2=0x7f5f381ac730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.620+0000 7f5f40262640 1 --2- 172.21.15.70:0/2615652082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f3811c0f0 0x7f5f381ac730 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f5f3000d750 tx=0x7f5f3000dc10 comp rx=0 tx=0).stop 2024-06-04T05:21:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.621+0000 7f5f40262640 1 -- 172.21.15.70:0/2615652082 shutdown_connections 2024-06-04T05:21:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.621+0000 7f5f40262640 1 --2- 172.21.15.70:0/2615652082 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5f200b2a90 0x7f5f200b4f50 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.621+0000 7f5f40262640 1 --2- 172.21.15.70:0/2615652082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3806ef70 0x7f5f381a83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.621+0000 7f5f40262640 1 --2- 172.21.15.70:0/2615652082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f3811c0f0 0x7f5f381ac730 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.621+0000 7f5f40262640 1 --2- 172.21.15.70:0/2615652082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f3806e650 0x7f5f381a7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.621+0000 7f5f40262640 1 -- 172.21.15.70:0/2615652082 >> 172.21.15.70:0/2615652082 conn(0x7f5f3806dcb0 msgr2=0x7f5f3811d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.621+0000 7f5f40262640 1 -- 172.21.15.70:0/2615652082 shutdown_connections 2024-06-04T05:21:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:39.621+0000 7f5f40262640 1 -- 172.21.15.70:0/2615652082 wait complete. 2024-06-04T05:21:39.640 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:21:39.640 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:21:39.662 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:39 smithi070 ceph-mon[32631]: pgmap v771: 97 pgs: 97 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 252 op/s 2024-06-04T05:21:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:39 smithi112 ceph-mon[36453]: pgmap v771: 97 pgs: 97 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 252 op/s 2024-06-04T05:21:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:39 smithi098 ceph-mon[35569]: pgmap v771: 97 pgs: 97 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 252 op/s 2024-06-04T05:21:39.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.96 MB/sec execute 511 sec latency 2120.028 ms 2024-06-04T05:21:39.880 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:21:40.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.94 MB/sec execute 512 sec latency 3120.141 ms 2024-06-04T05:21:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:41 smithi112 ceph-mon[36453]: pgmap v772: 97 pgs: 97 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 252 op/s 2024-06-04T05:21:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:41 smithi098 ceph-mon[35569]: pgmap v772: 97 pgs: 97 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 252 op/s 2024-06-04T05:21:41.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.93 MB/sec execute 513 sec latency 4120.195 ms 2024-06-04T05:21:41.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:41 smithi070 ceph-mon[32631]: pgmap v772: 97 pgs: 97 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 252 op/s 2024-06-04T05:21:42.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.92 MB/sec execute 514 sec latency 5120.283 ms 2024-06-04T05:21:42.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.982+0000 7f32a8221700 1 -- 172.21.15.70:0/1790381405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32a01095a0 msgr2=0x7f32a0109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:42.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.982+0000 7f32a8221700 1 --2- 172.21.15.70:0/1790381405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32a01095a0 0x7f32a0109980 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f329c009990 tx=0x7f329c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:21:42.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.982+0000 7f32a8221700 1 -- 172.21.15.70:0/1790381405 shutdown_connections 2024-06-04T05:21:42.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.982+0000 7f32a8221700 1 --2- 172.21.15.70:0/1790381405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a006c470 0x7f32a01103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:42.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.982+0000 7f32a8221700 1 --2- 172.21.15.70:0/1790381405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32a01095a0 0x7f32a0109980 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:42.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.982+0000 7f32a8221700 1 --2- 172.21.15.70:0/1790381405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f32a006bab0 0x7f32a006bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:42.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.982+0000 7f32a8221700 1 -- 172.21.15.70:0/1790381405 >> 172.21.15.70:0/1790381405 conn(0x7f32a00793b0 msgr2=0x7f32a00797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:42.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.982+0000 7f32a8221700 1 -- 172.21.15.70:0/1790381405 shutdown_connections 2024-06-04T05:21:42.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.982+0000 7f32a8221700 1 -- 172.21.15.70:0/1790381405 wait complete. 2024-06-04T05:21:42.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.983+0000 7f32a8221700 1 Processor -- start 2024-06-04T05:21:42.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.983+0000 7f32a8221700 1 -- start start 2024-06-04T05:21:42.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.983+0000 7f32a8221700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a006bab0 0x7f32a019f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:42.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.983+0000 7f32a8221700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32a006c470 0x7f32a019fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:42.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.983+0000 7f32a8221700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f32a01095a0 0x7f32a01999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:42.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.983+0000 7f32a8221700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f32a01135e0 con 0x7f32a006c470 2024-06-04T05:21:42.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.983+0000 7f32a57bc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32a006c470 0x7f32a019fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:42.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.983+0000 7f32a57bc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32a006c470 0x7f32a019fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60824/0 (socket says 172.21.15.70:60824) 2024-06-04T05:21:42.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.983+0000 7f32a57bc700 1 -- 172.21.15.70:0/1190032542 learned_addr learned my addr 172.21.15.70:0/1190032542 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:42.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.983+0000 7f32a8221700 1 -- 172.21.15.70:0/1190032542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f32a0113460 con 0x7f32a006bab0 2024-06-04T05:21:42.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.983+0000 7f32a8221700 1 -- 172.21.15.70:0/1190032542 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f32a0113760 con 0x7f32a01095a0 2024-06-04T05:21:42.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.983+0000 7f32a5fbd700 1 --2- 172.21.15.70:0/1190032542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a006bab0 0x7f32a019f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:42.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.983+0000 7f32a67be700 1 --2- 172.21.15.70:0/1190032542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f32a01095a0 0x7f32a01999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:42.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.983+0000 7f32a57bc700 1 -- 172.21.15.70:0/1190032542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f32a01095a0 msgr2=0x7f32a01999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:42.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.983+0000 7f32a57bc700 1 --2- 172.21.15.70:0/1190032542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f32a01095a0 0x7f32a01999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:42.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.983+0000 7f32a57bc700 1 -- 172.21.15.70:0/1190032542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a006bab0 msgr2=0x7f32a019f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:42.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.983+0000 7f32a57bc700 1 --2- 172.21.15.70:0/1190032542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a006bab0 0x7f32a019f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:42.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.984+0000 7f32a57bc700 1 -- 172.21.15.70:0/1190032542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32a019a100 con 0x7f32a006c470 2024-06-04T05:21:42.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.984+0000 7f32a67be700 1 --2- 172.21.15.70:0/1190032542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f32a01095a0 0x7f32a01999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:21:42.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.984+0000 7f32a57bc700 1 --2- 172.21.15.70:0/1190032542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32a006c470 0x7f32a019fd20 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f329000b530 tx=0x7f329000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:42.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.984+0000 7f3296ffd700 1 -- 172.21.15.70:0/1190032542 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3290015020 con 0x7f32a006c470 2024-06-04T05:21:42.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.984+0000 7f32a8221700 1 -- 172.21.15.70:0/1190032542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32a019a3f0 con 0x7f32a006c470 2024-06-04T05:21:42.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.984+0000 7f3296ffd700 1 -- 172.21.15.70:0/1190032542 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3290011cc0 con 0x7f32a006c470 2024-06-04T05:21:42.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.984+0000 7f3296ffd700 1 -- 172.21.15.70:0/1190032542 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3290010dc0 con 0x7f32a006c470 2024-06-04T05:21:42.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.984+0000 7f32a8221700 1 -- 172.21.15.70:0/1190032542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32a01a66b0 con 0x7f32a006c470 2024-06-04T05:21:42.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.985+0000 7f32a8221700 1 -- 172.21.15.70:0/1190032542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32a010ba80 con 0x7f32a006c470 2024-06-04T05:21:42.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.987+0000 7f3296ffd700 1 -- 172.21.15.70:0/1190032542 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3290018020 con 0x7f32a006c470 2024-06-04T05:21:42.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.988+0000 7f3296ffd700 1 --2- 172.21.15.70:0/1190032542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f328c0b29f0 0x7f328c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:42.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.988+0000 7f32a5fbd700 1 --2- 172.21.15.70:0/1190032542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f328c0b29f0 0x7f328c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:42.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.988+0000 7f3296ffd700 1 -- 172.21.15.70:0/1190032542 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f32900e2480 con 0x7f32a006c470 2024-06-04T05:21:42.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.989+0000 7f32a5fbd700 1 --2- 172.21.15.70:0/1190032542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f328c0b29f0 0x7f328c0b4eb0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f329c00b5c0 tx=0x7f329c01a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:42.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:42.991+0000 7f3296ffd700 1 -- 172.21.15.70:0/1190032542 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f329001f020 con 0x7f32a006c470 2024-06-04T05:21:43.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:43.257+0000 7f32a8221700 1 -- 172.21.15.70:0/1190032542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32a006a230 con 0x7f32a006c470 2024-06-04T05:21:43.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:43.258+0000 7f3296ffd700 1 -- 172.21.15.70:0/1190032542 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+11933 (secure 0 0 0) 0x7f32900a9ee0 con 0x7f32a006c470 2024-06-04T05:21:43.260 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:21:43.260 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":64,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:20:02.842346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:21:43.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:43.260+0000 7f32a8221700 1 -- 172.21.15.70:0/1190032542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f328c0b29f0 msgr2=0x7f328c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:43.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:43.260+0000 7f32a8221700 1 --2- 172.21.15.70:0/1190032542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f328c0b29f0 0x7f328c0b4eb0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f329c00b5c0 tx=0x7f329c01a040 comp rx=0 tx=0).stop 2024-06-04T05:21:43.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:43.260+0000 7f32a8221700 1 -- 172.21.15.70:0/1190032542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32a006c470 msgr2=0x7f32a019fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:43.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:43.260+0000 7f32a8221700 1 --2- 172.21.15.70:0/1190032542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32a006c470 0x7f32a019fd20 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f329000b530 tx=0x7f329000b840 comp rx=0 tx=0).stop 2024-06-04T05:21:43.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:43.260+0000 7f32a8221700 1 -- 172.21.15.70:0/1190032542 shutdown_connections 2024-06-04T05:21:43.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:43.260+0000 7f32a8221700 1 --2- 172.21.15.70:0/1190032542 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f328c0b29f0 0x7f328c0b4eb0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:43.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:43.260+0000 7f32a8221700 1 --2- 172.21.15.70:0/1190032542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f32a01095a0 0x7f32a01999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:43.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:43.260+0000 7f32a8221700 1 --2- 172.21.15.70:0/1190032542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a006bab0 0x7f32a019f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:43.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:43.260+0000 7f32a8221700 1 --2- 172.21.15.70:0/1190032542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32a006c470 0x7f32a019fd20 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:43.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:43.260+0000 7f32a8221700 1 -- 172.21.15.70:0/1190032542 >> 172.21.15.70:0/1190032542 conn(0x7f32a00793b0 msgr2=0x7f32a0101330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:43.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:43.260+0000 7f32a8221700 1 -- 172.21.15.70:0/1190032542 shutdown_connections 2024-06-04T05:21:43.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:43.260+0000 7f32a8221700 1 -- 172.21.15.70:0/1190032542 wait complete. 2024-06-04T05:21:43.264 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 64 2024-06-04T05:21:43.581 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:43 smithi070 ceph-mon[32631]: pgmap v773: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.6 MiB/s wr, 370 op/s 2024-06-04T05:21:43.581 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:43 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1190032542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:21:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:43 smithi112 ceph-mon[36453]: pgmap v773: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.6 MiB/s wr, 370 op/s 2024-06-04T05:21:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:43 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1190032542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:21:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:43 smithi098 ceph-mon[35569]: pgmap v773: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.6 MiB/s wr, 370 op/s 2024-06-04T05:21:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:43 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1190032542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:21:43.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.90 MB/sec execute 515 sec latency 6120.364 ms 2024-06-04T05:21:44.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:21:44.312 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:21:44.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.89 MB/sec execute 516 sec latency 7120.435 ms 2024-06-04T05:21:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:45 smithi112 ceph-mon[36453]: pgmap v774: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 221 op/s 2024-06-04T05:21:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:45 smithi098 ceph-mon[35569]: pgmap v774: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 221 op/s 2024-06-04T05:21:45.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:45 smithi070 ceph-mon[32631]: pgmap v774: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 221 op/s 2024-06-04T05:21:45.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.88 MB/sec execute 517 sec latency 8120.523 ms 2024-06-04T05:21:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.149+0000 7f5dcf352700 1 -- 172.21.15.70:0/716721616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dc8102e00 msgr2=0x7f5dc810f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.149+0000 7f5dcf352700 1 --2- 172.21.15.70:0/716721616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dc8102e00 0x7f5dc810f2f0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f5dc000b0a0 tx=0x7f5dc000b3b0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.149+0000 7f5dcf352700 1 -- 172.21.15.70:0/716721616 shutdown_connections 2024-06-04T05:21:46.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.149+0000 7f5dcf352700 1 --2- 172.21.15.70:0/716721616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dc8102e00 0x7f5dc810f2f0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.149+0000 7f5dcf352700 1 --2- 172.21.15.70:0/716721616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5dc8108520 0x7f5dc8108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.149+0000 7f5dcf352700 1 --2- 172.21.15.70:0/716721616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc8102440 0x7f5dc81028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.149+0000 7f5dcf352700 1 -- 172.21.15.70:0/716721616 >> 172.21.15.70:0/716721616 conn(0x7f5dc80793f0 msgr2=0x7f5dc8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.149+0000 7f5dcf352700 1 -- 172.21.15.70:0/716721616 shutdown_connections 2024-06-04T05:21:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.149+0000 7f5dcf352700 1 -- 172.21.15.70:0/716721616 wait complete. 2024-06-04T05:21:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcf352700 1 Processor -- start 2024-06-04T05:21:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcf352700 1 -- start start 2024-06-04T05:21:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcf352700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dc8102440 0x7f5dc819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcf352700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5dc8102e00 0x7f5dc819fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcf352700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc8108520 0x7f5dc8199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcf352700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc8112480 con 0x7f5dc8102440 2024-06-04T05:21:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcf352700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc8112300 con 0x7f5dc8108520 2024-06-04T05:21:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcf352700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc8112600 con 0x7f5dc8102e00 2024-06-04T05:21:46.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcd0ee700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dc8102440 0x7f5dc819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcd0ee700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dc8102440 0x7f5dc819f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40526/0 (socket says 172.21.15.70:40526) 2024-06-04T05:21:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcd0ee700 1 -- 172.21.15.70:0/2250977226 learned_addr learned my addr 172.21.15.70:0/2250977226 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcd8ef700 1 --2- 172.21.15.70:0/2250977226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc8108520 0x7f5dc8199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcc8ed700 1 --2- 172.21.15.70:0/2250977226 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5dc8102e00 0x7f5dc819fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcd0ee700 1 -- 172.21.15.70:0/2250977226 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5dc8102e00 msgr2=0x7f5dc819fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcd0ee700 1 --2- 172.21.15.70:0/2250977226 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5dc8102e00 0x7f5dc819fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcd0ee700 1 -- 172.21.15.70:0/2250977226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc8108520 msgr2=0x7f5dc8199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcd0ee700 1 --2- 172.21.15.70:0/2250977226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc8108520 0x7f5dc8199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcd0ee700 1 -- 172.21.15.70:0/2250977226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc819a0d0 con 0x7f5dc8102440 2024-06-04T05:21:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.150+0000 7f5dcd8ef700 1 --2- 172.21.15.70:0/2250977226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc8108520 0x7f5dc8199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:21:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.151+0000 7f5dcc8ed700 1 --2- 172.21.15.70:0/2250977226 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5dc8102e00 0x7f5dc819fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:21:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.151+0000 7f5dcd0ee700 1 --2- 172.21.15.70:0/2250977226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dc8102440 0x7f5dc819f7b0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400c9b0 tx=0x7f5dc400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.151+0000 7f5dba7fc700 1 -- 172.21.15.70:0/2250977226 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4007b00 con 0x7f5dc8102440 2024-06-04T05:21:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.151+0000 7f5dcf352700 1 -- 172.21.15.70:0/2250977226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dc819a3c0 con 0x7f5dc8102440 2024-06-04T05:21:46.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.151+0000 7f5dba7fc700 1 -- 172.21.15.70:0/2250977226 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5dc4007ca0 con 0x7f5dc8102440 2024-06-04T05:21:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.151+0000 7f5dba7fc700 1 -- 172.21.15.70:0/2250977226 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4018660 con 0x7f5dc8102440 2024-06-04T05:21:46.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.151+0000 7f5dcf352700 1 -- 172.21.15.70:0/2250977226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dc81a6680 con 0x7f5dc8102440 2024-06-04T05:21:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.153+0000 7f5dba7fc700 1 -- 172.21.15.70:0/2250977226 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f5dc400f450 con 0x7f5dc8102440 2024-06-04T05:21:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.154+0000 7f5dba7fc700 1 --2- 172.21.15.70:0/2250977226 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5db40b29f0 0x7f5db40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.154+0000 7f5dcc8ed700 1 --2- 172.21.15.70:0/2250977226 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5db40b29f0 0x7f5db40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.154+0000 7f5dba7fc700 1 -- 172.21.15.70:0/2250977226 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f5dc40e2690 con 0x7f5dc8102440 2024-06-04T05:21:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dcc8ed700 1 --2- 172.21.15.70:0/2250977226 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5db40b29f0 0x7f5db40b4eb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc005fd0 tx=0x7f5dbc005ee0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dd9be7b80 1 Processor -- start 2024-06-04T05:21:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dd9be7b80 1 -- start start 2024-06-04T05:21:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dd9be7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9bb320 0x55b5da9bb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dd9be7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9b58a0 0x55b5da9b5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dd9be7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9c2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dd9be7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b5da913d70 con 0x55b5da9b6260 2024-06-04T05:21:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dd9be7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b5da903910 con 0x55b5da9bb320 2024-06-04T05:21:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dd9be7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b5da907950 con 0x55b5da9b58a0 2024-06-04T05:21:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dcd8ef700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9c2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dcd8ef700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9c2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40536/0 (socket says 172.21.15.70:40536) 2024-06-04T05:21:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dcd8ef700 1 -- 172.21.15.70:0/27651422 learned_addr learned my addr 172.21.15.70:0/27651422 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dcd0ee700 1 --2- 172.21.15.70:0/27651422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9bb320 0x55b5da9bb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dcc8ed700 1 --2- 172.21.15.70:0/27651422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9b58a0 0x55b5da9b5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dcd8ef700 1 -- 172.21.15.70:0/27651422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9b58a0 msgr2=0x55b5da9b5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dcd8ef700 1 --2- 172.21.15.70:0/27651422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9b58a0 0x55b5da9b5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dcd8ef700 1 -- 172.21.15.70:0/27651422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9bb320 msgr2=0x55b5da9bb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dcd8ef700 1 --2- 172.21.15.70:0/27651422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9bb320 0x55b5da9bb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.155+0000 7f5dcd8ef700 1 -- 172.21.15.70:0/27651422 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5da8acfb0 con 0x55b5da9b6260 2024-06-04T05:21:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.156+0000 7f5dcd8ef700 1 --2- 172.21.15.70:0/27651422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9c2080 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f5dc000b830 tx=0x7f5dc0009170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30bce8e6c1fb79a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.156+0000 7f5db37fe700 1 -- 172.21.15.70:0/27651422 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0022070 con 0x55b5da9b6260 2024-06-04T05:21:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.156+0000 7f5db37fe700 1 -- 172.21.15.70:0/27651422 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5dc00074e0 con 0x55b5da9b6260 2024-06-04T05:21:46.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.156+0000 7f5db37fe700 1 -- 172.21.15.70:0/27651422 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc001c640 con 0x55b5da9b6260 2024-06-04T05:21:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.156+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/27651422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 msgr2=0x55b5da9c2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.156+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/27651422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9c2080 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f5dc000b830 tx=0x7f5dc0009170 comp rx=0 tx=0).stop 2024-06-04T05:21:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.157+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/27651422 shutdown_connections 2024-06-04T05:21:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.157+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/27651422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9c2080 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.157+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/27651422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9bb320 0x55b5da9bb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.157+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/27651422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9b58a0 0x55b5da9b5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.157+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/27651422 >> 172.21.15.70:0/27651422 conn(0x55b5da8e1240 msgr2=0x55b5da9b03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.157+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/27651422 shutdown_connections 2024-06-04T05:21:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.157+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/27651422 wait complete. 2024-06-04T05:21:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.157+0000 7f5dd9be7b80 1 -- start start 2024-06-04T05:21:46.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.157+0000 7f5dd9be7b80 1 Processor -- start 2024-06-04T05:21:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.157+0000 7f5dd9be7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9b58a0 0x55b5da9c1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.157+0000 7f5dd9be7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9befa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.157+0000 7f5dd9be7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9bb320 0x55b5da9bf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.157+0000 7f5dd9be7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b5da809fa0 con 0x55b5da9b6260 2024-06-04T05:21:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.157+0000 7f5dd9be7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b5da86bda0 con 0x55b5da9b58a0 2024-06-04T05:21:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.157+0000 7f5dd9be7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b5da9c2680 con 0x55b5da9bb320 2024-06-04T05:21:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.157+0000 7f5dd9be7b80 10 client.0 _ll_register_callbacks cb 0x55b5da89e950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5dcc8ed700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9befa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5dcc8ed700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9befa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40552/0 (socket says 172.21.15.70:40552) 2024-06-04T05:21:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5dcc8ed700 1 -- 172.21.15.70:0/2353274638 learned_addr learned my addr 172.21.15.70:0/2353274638 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:46.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5dcd0ee700 1 --2- 172.21.15.70:0/2353274638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9b58a0 0x55b5da9c1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5dcd8ef700 1 --2- 172.21.15.70:0/2353274638 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9bb320 0x55b5da9bf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5dcc8ed700 1 -- 172.21.15.70:0/2353274638 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9bb320 msgr2=0x55b5da9bf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5dcc8ed700 1 --2- 172.21.15.70:0/2353274638 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9bb320 0x55b5da9bf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5dcc8ed700 1 -- 172.21.15.70:0/2353274638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9b58a0 msgr2=0x55b5da9c1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5dcc8ed700 1 --2- 172.21.15.70:0/2353274638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9b58a0 0x55b5da9c1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5dcc8ed700 1 -- 172.21.15.70:0/2353274638 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5da8538d0 con 0x55b5da9b6260 2024-06-04T05:21:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5dcd8ef700 1 --2- 172.21.15.70:0/2353274638 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9bb320 0x55b5da9bf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:21:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5dcd0ee700 1 --2- 172.21.15.70:0/2353274638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9b58a0 0x55b5da9c1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:21:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5dcc8ed700 1 --2- 172.21.15.70:0/2353274638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9befa0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc011720 tx=0x7f5dbc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b030b9cfdd2de80b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5db0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:21:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5db0ff9700 1 -- 172.21.15.70:0/2353274638 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc023070 con 0x55b5da9b6260 2024-06-04T05:21:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5db0ff9700 1 -- 172.21.15.70:0/2353274638 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f5dbc004220 con 0x55b5da9b6260 2024-06-04T05:21:46.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/2353274638 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5dc00074e0 con 0x55b5da9b6260 2024-06-04T05:21:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5db0ff9700 1 -- 172.21.15.70:0/2353274638 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f5dbc02e050 con 0x55b5da9b6260 2024-06-04T05:21:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5dd9be7b80 10 client.21594 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.158+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/2353274638 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55b5da8bd760 con 0x55b5da9b6260 2024-06-04T05:21:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.159+0000 7f5db0ff9700 1 -- 172.21.15.70:0/2353274638 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5dc00074e0 con 0x55b5da9b6260 2024-06-04T05:21:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.159+0000 7f5db0ff9700 1 -- 172.21.15.70:0/2353274638 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f5dbc00f040 con 0x55b5da9b6260 2024-06-04T05:21:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.159+0000 7f5dd9be7b80 10 client.21594 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.159+0000 7f5dd9be7b80 10 client.21594 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:21:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.159+0000 7f5dd9be7b80 20 client.21594 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.159+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/2353274638 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55b5da9cf130 0x55b5da9d1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.159+0000 7f5dd9be7b80 4 client.21594 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:21:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.159+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/2353274638 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5da915cc0 con 0x55b5da9cf130 2024-06-04T05:21:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.159+0000 7f5dcd0ee700 1 --2- 172.21.15.70:0/2353274638 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55b5da9cf130 0x55b5da9d1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.159+0000 7f5db0ff9700 10 client.21594 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:21:46.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.159+0000 7f5dcd0ee700 1 --2- 172.21.15.70:0/2353274638 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55b5da9cf130 0x55b5da9d1560 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=838068f1b8d43cbd server_cookie=fc9300883bc7d62c in_seq=0 out_seq=0 2024-06-04T05:21:46.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.160+0000 7f5db0ff9700 1 -- 172.21.15.70:0/2353274638 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b5da915cc0 con 0x55b5da9cf130 2024-06-04T05:21:46.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.160+0000 7f5db0ff9700 10 client.21594 handle_command_reply: tid=0 2024-06-04T05:21:46.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.160+0000 7f5dd9be7b80 1 client.21594 shutdown 2024-06-04T05:21:46.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.160+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/2353274638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 msgr2=0x55b5da9befa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:46.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.160+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/2353274638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9befa0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc011720 tx=0x7f5dbc0047c0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.160+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/2353274638 shutdown_connections 2024-06-04T05:21:46.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.160+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/2353274638 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55b5da9cf130 0x55b5da9d1560 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.160+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/2353274638 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9bb320 0x55b5da9bf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.160+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/2353274638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9b58a0 0x55b5da9c1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.160+0000 7f5db0ff9700 0 client.21594 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:21:46.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.160+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/2353274638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9befa0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.160+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/2353274638 >> 172.21.15.70:0/2353274638 conn(0x55b5da8e1240 msgr2=0x55b5da9afaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:46.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.161+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/2353274638 shutdown_connections 2024-06-04T05:21:46.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.161+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/2353274638 wait complete. 2024-06-04T05:21:46.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.161+0000 7f5dd9be7b80 20 client.21594 trim_cache size 0 max 16384 2024-06-04T05:21:46.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.175+0000 7f5dd9be7b80 1 Processor -- start 2024-06-04T05:21:46.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5dd9be7b80 1 -- start start 2024-06-04T05:21:46.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5dd9be7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9b58a0 0x55b5da9be120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5dd9be7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9b6260 0x55b5da9b0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5dd9be7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9bb320 0x55b5da9c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5dd9be7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc81adcf0 con 0x55b5da9bb320 2024-06-04T05:21:46.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5dd9be7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc81af8c0 con 0x55b5da9b58a0 2024-06-04T05:21:46.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5dd9be7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc81ae420 con 0x55b5da9b6260 2024-06-04T05:21:46.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5dcd8ef700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9bb320 0x55b5da9c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:46.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5dcd8ef700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9bb320 0x55b5da9c07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40562/0 (socket says 172.21.15.70:40562) 2024-06-04T05:21:46.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5dcd8ef700 1 -- 172.21.15.70:0/3324690250 learned_addr learned my addr 172.21.15.70:0/3324690250 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:46.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5dcd8ef700 1 -- 172.21.15.70:0/3324690250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9b6260 msgr2=0x55b5da9b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:46.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5dcd8ef700 1 --2- 172.21.15.70:0/3324690250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9b6260 0x55b5da9b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5dcd8ef700 1 -- 172.21.15.70:0/3324690250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9b58a0 msgr2=0x55b5da9be120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:46.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5dcd8ef700 1 --2- 172.21.15.70:0/3324690250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9b58a0 0x55b5da9be120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5dcd8ef700 1 -- 172.21.15.70:0/3324690250 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5da9c18e0 con 0x55b5da9bb320 2024-06-04T05:21:46.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5dcd8ef700 1 --2- 172.21.15.70:0/3324690250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9bb320 0x55b5da9c07d0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0009140 tx=0x7f5dc0009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fe211fa668ed174 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:46.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5db2ffd700 1 -- 172.21.15.70:0/3324690250 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc002a070 con 0x55b5da9bb320 2024-06-04T05:21:46.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.176+0000 7f5db2ffd700 1 -- 172.21.15.70:0/3324690250 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5dc0004c50 con 0x55b5da9bb320 2024-06-04T05:21:46.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.177+0000 7f5db2ffd700 1 -- 172.21.15.70:0/3324690250 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0022040 con 0x55b5da9bb320 2024-06-04T05:21:46.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.177+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/3324690250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9bb320 msgr2=0x55b5da9c07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:46.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.177+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/3324690250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9bb320 0x55b5da9c07d0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0009140 tx=0x7f5dc0009520 comp rx=0 tx=0).stop 2024-06-04T05:21:46.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.177+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/3324690250 shutdown_connections 2024-06-04T05:21:46.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.177+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/3324690250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9bb320 0x55b5da9c07d0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.177+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/3324690250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9b58a0 0x55b5da9be120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.177+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/3324690250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9b6260 0x55b5da9b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.177+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/3324690250 >> 172.21.15.70:0/3324690250 conn(0x55b5da8e1240 msgr2=0x55b5da9ba750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:46.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.177+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/3324690250 shutdown_connections 2024-06-04T05:21:46.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.177+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/3324690250 wait complete. 2024-06-04T05:21:46.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.177+0000 7f5dd9be7b80 1 -- start start 2024-06-04T05:21:46.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dd9be7b80 1 Processor -- start 2024-06-04T05:21:46.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dd9be7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9b58a0 0x55b5da9c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dd9be7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dd9be7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9bb320 0x55b5da9d6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dd9be7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc81af1c0 con 0x55b5da9b6260 2024-06-04T05:21:46.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dd9be7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc81aeaf0 con 0x55b5da9bb320 2024-06-04T05:21:46.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dd9be7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc81abcb0 con 0x55b5da9b58a0 2024-06-04T05:21:46.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dd9be7b80 10 client.0 _ll_register_callbacks cb 0x55b5da89e950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:46.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dcc8ed700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:46.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dcc8ed700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40572/0 (socket says 172.21.15.70:40572) 2024-06-04T05:21:46.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dcc8ed700 1 -- 172.21.15.70:0/1084678940 learned_addr learned my addr 172.21.15.70:0/1084678940 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:46.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dcd0ee700 1 --2- 172.21.15.70:0/1084678940 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9b58a0 0x55b5da9c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:46.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dcd8ef700 1 --2- 172.21.15.70:0/1084678940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9bb320 0x55b5da9d6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:46.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dcc8ed700 1 -- 172.21.15.70:0/1084678940 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9b58a0 msgr2=0x55b5da9c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:46.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dcc8ed700 1 --2- 172.21.15.70:0/1084678940 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9b58a0 0x55b5da9c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dcc8ed700 1 -- 172.21.15.70:0/1084678940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9bb320 msgr2=0x55b5da9d6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:46.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dcc8ed700 1 --2- 172.21.15.70:0/1084678940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9bb320 0x55b5da9d6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.178+0000 7f5dcc8ed700 1 -- 172.21.15.70:0/1084678940 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5da9d6e90 con 0x55b5da9b6260 2024-06-04T05:21:46.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.179+0000 7f5dcc8ed700 1 --2- 172.21.15.70:0/1084678940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9c8170 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0106b0 tx=0x7f5dbc011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55dbcdb2cbcd6d69 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:46.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.179+0000 7f5db0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:21:46.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.179+0000 7f5db0ff9700 1 -- 172.21.15.70:0/1084678940 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc023070 con 0x55b5da9b6260 2024-06-04T05:21:46.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.179+0000 7f5db0ff9700 1 -- 172.21.15.70:0/1084678940 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f5dbc0041e0 con 0x55b5da9b6260 2024-06-04T05:21:46.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.179+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/1084678940 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5dc0004c50 con 0x55b5da9b6260 2024-06-04T05:21:46.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.179+0000 7f5db0ff9700 1 -- 172.21.15.70:0/1084678940 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f5dbc010e60 con 0x55b5da9b6260 2024-06-04T05:21:46.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.179+0000 7f5dd9be7b80 10 client.21600 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:46.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.179+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/1084678940 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55b5da9c7590 con 0x55b5da9b6260 2024-06-04T05:21:46.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.179+0000 7f5db0ff9700 1 -- 172.21.15.70:0/1084678940 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5dc0004c50 con 0x55b5da9b6260 2024-06-04T05:21:46.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.179+0000 7f5db0ff9700 1 -- 172.21.15.70:0/1084678940 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f5dbc038030 con 0x55b5da9b6260 2024-06-04T05:21:46.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.179+0000 7f5dd9be7b80 10 client.21600 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:46.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.179+0000 7f5dd9be7b80 10 client.21600 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:21:46.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.179+0000 7f5dd9be7b80 20 client.21600 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:46.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.179+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/1084678940 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55b5daa3a020 0x55b5daa3c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:46.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.179+0000 7f5dd9be7b80 4 client.21600 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:21:46.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.179+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/1084678940 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b5da90f4a0 con 0x55b5daa3a020 2024-06-04T05:21:46.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.180+0000 7f5dcd0ee700 1 --2- 172.21.15.70:0/1084678940 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55b5daa3a020 0x55b5daa3c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:46.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.180+0000 7f5dcd0ee700 1 --2- 172.21.15.70:0/1084678940 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55b5daa3a020 0x55b5daa3c440 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=343bc325ae235237 server_cookie=15590062b6ef6c1 in_seq=0 out_seq=0 2024-06-04T05:21:46.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.180+0000 7f5db0ff9700 10 client.21600 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:21:46.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.180+0000 7f5db0ff9700 1 -- 172.21.15.70:0/1084678940 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b5da90f4a0 con 0x55b5daa3a020 2024-06-04T05:21:46.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.180+0000 7f5db0ff9700 10 client.21600 handle_command_reply: tid=0 2024-06-04T05:21:46.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.180+0000 7f5dd9be7b80 1 client.21600 shutdown 2024-06-04T05:21:46.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.180+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/1084678940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 msgr2=0x55b5da9c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:46.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.180+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/1084678940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9c8170 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0106b0 tx=0x7f5dbc011f90 comp rx=0 tx=0).stop 2024-06-04T05:21:46.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.180+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/1084678940 shutdown_connections 2024-06-04T05:21:46.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.180+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/1084678940 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55b5daa3a020 0x55b5daa3c440 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.180+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/1084678940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5da9bb320 0x55b5da9d6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.180+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/1084678940 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b5da9b58a0 0x55b5da9c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.180+0000 7f5db0ff9700 0 client.21600 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:21:46.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.180+0000 7f5dd9be7b80 1 --2- 172.21.15.70:0/1084678940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5da9b6260 0x55b5da9c8170 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.180+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/1084678940 >> 172.21.15.70:0/1084678940 conn(0x55b5da8e1240 msgr2=0x55b5da9b8ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:46.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.180+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/1084678940 shutdown_connections 2024-06-04T05:21:46.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.181+0000 7f5dd9be7b80 1 -- 172.21.15.70:0/1084678940 wait complete. 2024-06-04T05:21:46.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.181+0000 7f5dd9be7b80 20 client.21600 trim_cache size 0 max 16384 2024-06-04T05:21:46.188 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:21:46.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.181+0000 7f5dcf352700 1 -- 172.21.15.70:0/2250977226 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5db40b29f0 msgr2=0x7f5db40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:46.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.181+0000 7f5dcf352700 1 --2- 172.21.15.70:0/2250977226 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5db40b29f0 0x7f5db40b4eb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc005fd0 tx=0x7f5dbc005ee0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.181+0000 7f5dcf352700 1 -- 172.21.15.70:0/2250977226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dc8102440 msgr2=0x7f5dc819f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:46.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.181+0000 7f5dcf352700 1 --2- 172.21.15.70:0/2250977226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dc8102440 0x7f5dc819f7b0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400c9b0 tx=0x7f5dc400cd70 comp rx=0 tx=0).stop 2024-06-04T05:21:46.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.181+0000 7f5dcf352700 1 -- 172.21.15.70:0/2250977226 shutdown_connections 2024-06-04T05:21:46.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.181+0000 7f5dcf352700 1 --2- 172.21.15.70:0/2250977226 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f5db40b29f0 0x7f5db40b4eb0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.181+0000 7f5dcf352700 1 --2- 172.21.15.70:0/2250977226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc8108520 0x7f5dc8199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.181+0000 7f5dcf352700 1 --2- 172.21.15.70:0/2250977226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dc8102440 0x7f5dc819f7b0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.181+0000 7f5dcf352700 1 --2- 172.21.15.70:0/2250977226 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5dc8102e00 0x7f5dc819fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:46.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.181+0000 7f5dcf352700 1 -- 172.21.15.70:0/2250977226 >> 172.21.15.70:0/2250977226 conn(0x7f5dc80793f0 msgr2=0x7f5dc806b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:46.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.181+0000 7f5dcf352700 1 -- 172.21.15.70:0/2250977226 shutdown_connections 2024-06-04T05:21:46.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:46.181+0000 7f5dcf352700 1 -- 172.21.15.70:0/2250977226 wait complete. 2024-06-04T05:21:46.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.86 MB/sec execute 518 sec latency 9120.597 ms 2024-06-04T05:21:47.016 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:21:47.258 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:21:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:47 smithi112 ceph-mon[36453]: pgmap v775: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 230 op/s 2024-06-04T05:21:47.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:47 smithi070 ceph-mon[32631]: pgmap v775: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 230 op/s 2024-06-04T05:21:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:47 smithi098 ceph-mon[35569]: pgmap v775: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 230 op/s 2024-06-04T05:21:47.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.85 MB/sec execute 519 sec latency 10120.632 ms 2024-06-04T05:21:48.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.84 MB/sec execute 520 sec latency 11120.671 ms 2024-06-04T05:21:48.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.926+0000 7f48ec18c700 1 -- 172.21.15.70:0/778499307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48e41037f0 msgr2=0x7f48e4103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:48.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.926+0000 7f48ec18c700 1 --2- 172.21.15.70:0/778499307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48e41037f0 0x7f48e4103bd0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f48e00099c0 tx=0x7f48e0009cd0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.926+0000 7f48ec18c700 1 -- 172.21.15.70:0/778499307 shutdown_connections 2024-06-04T05:21:48.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.926+0000 7f48ec18c700 1 --2- 172.21.15.70:0/778499307 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48e41088f0 0x7f48e4110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.926+0000 7f48ec18c700 1 --2- 172.21.15.70:0/778499307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48e41037f0 0x7f48e4103bd0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.926+0000 7f48ec18c700 1 --2- 172.21.15.70:0/778499307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48e41041a0 0x7f48e41081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.926+0000 7f48ec18c700 1 -- 172.21.15.70:0/778499307 >> 172.21.15.70:0/778499307 conn(0x7f48e407a6a0 msgr2=0x7f48e407aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:48.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.927+0000 7f48ec18c700 1 -- 172.21.15.70:0/778499307 shutdown_connections 2024-06-04T05:21:48.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.927+0000 7f48ec18c700 1 -- 172.21.15.70:0/778499307 wait complete. 2024-06-04T05:21:48.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.927+0000 7f48ec18c700 1 Processor -- start 2024-06-04T05:21:48.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.927+0000 7f48ec18c700 1 -- start start 2024-06-04T05:21:48.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.927+0000 7f48ec18c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48e41037f0 0x7f48e419f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.927+0000 7f48ec18c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48e41041a0 0x7f48e419fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.927+0000 7f48ec18c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48e41088f0 0x7f48e4199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.927+0000 7f48ec18c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f48e4113600 con 0x7f48e41041a0 2024-06-04T05:21:48.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.927+0000 7f48ec18c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48e4113480 con 0x7f48e41037f0 2024-06-04T05:21:48.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.927+0000 7f48ec18c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f48e4113780 con 0x7f48e41088f0 2024-06-04T05:21:48.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.927+0000 7f48e9727700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48e41041a0 0x7f48e419fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.928+0000 7f48e9727700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48e41041a0 0x7f48e419fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40580/0 (socket says 172.21.15.70:40580) 2024-06-04T05:21:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.928+0000 7f48e9727700 1 -- 172.21.15.70:0/296979526 learned_addr learned my addr 172.21.15.70:0/296979526 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.928+0000 7f48ea729700 1 --2- 172.21.15.70:0/296979526 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48e41088f0 0x7f48e4199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.928+0000 7f48e9f28700 1 --2- 172.21.15.70:0/296979526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48e41037f0 0x7f48e419f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.928+0000 7f48e9727700 1 -- 172.21.15.70:0/296979526 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48e41088f0 msgr2=0x7f48e4199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.928+0000 7f48e9727700 1 --2- 172.21.15.70:0/296979526 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48e41088f0 0x7f48e4199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.928+0000 7f48e9727700 1 -- 172.21.15.70:0/296979526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48e41037f0 msgr2=0x7f48e419f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.928+0000 7f48e9727700 1 --2- 172.21.15.70:0/296979526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48e41037f0 0x7f48e419f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.928+0000 7f48e9727700 1 -- 172.21.15.70:0/296979526 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48e419a0e0 con 0x7f48e41041a0 2024-06-04T05:21:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.928+0000 7f48e9f28700 1 --2- 172.21.15.70:0/296979526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48e41037f0 0x7f48e419f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:21:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.928+0000 7f48e9727700 1 --2- 172.21.15.70:0/296979526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48e41041a0 0x7f48e419fd00 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f48d400b530 tx=0x7f48d400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.928+0000 7f48daffd700 1 -- 172.21.15.70:0/296979526 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4015020 con 0x7f48e41041a0 2024-06-04T05:21:48.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.928+0000 7f48ec18c700 1 -- 172.21.15.70:0/296979526 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48e419a3d0 con 0x7f48e41041a0 2024-06-04T05:21:48.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.928+0000 7f48daffd700 1 -- 172.21.15.70:0/296979526 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f48d4011cc0 con 0x7f48e41041a0 2024-06-04T05:21:48.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.928+0000 7f48daffd700 1 -- 172.21.15.70:0/296979526 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4010dc0 con 0x7f48e41041a0 2024-06-04T05:21:48.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.928+0000 7f48ec18c700 1 -- 172.21.15.70:0/296979526 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48e41a6690 con 0x7f48e41041a0 2024-06-04T05:21:48.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.930+0000 7f48daffd700 1 -- 172.21.15.70:0/296979526 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f48d4018020 con 0x7f48e41041a0 2024-06-04T05:21:48.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.931+0000 7f48daffd700 1 --2- 172.21.15.70:0/296979526 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48d00b2a70 0x7f48d00b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.931+0000 7f48e9f28700 1 --2- 172.21.15.70:0/296979526 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48d00b2a70 0x7f48d00b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.931+0000 7f48daffd700 1 -- 172.21.15.70:0/296979526 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f48d40e1cf0 con 0x7f48e41041a0 2024-06-04T05:21:48.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.932+0000 7f48e9f28700 1 --2- 172.21.15.70:0/296979526 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48d00b2a70 0x7f48d00b4f30 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f48e0006010 tx=0x7f48e0005e80 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:48.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.932+0000 7f48f6a21b80 1 Processor -- start 2024-06-04T05:21:48.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.932+0000 7f48f6a21b80 1 -- start start 2024-06-04T05:21:48.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.932+0000 7f48f6a21b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9bfc70 0x56227e9c0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.932+0000 7f48f6a21b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c0620 0x56227e9c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.932+0000 7f48f6a21b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c4370 0x56227e9c47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.932+0000 7f48f6a21b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56227e91ac90 con 0x56227e9c4370 2024-06-04T05:21:48.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.932+0000 7f48f6a21b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56227e90a830 con 0x56227e9bfc70 2024-06-04T05:21:48.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.932+0000 7f48f6a21b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56227e90e870 con 0x56227e9c0620 2024-06-04T05:21:48.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.932+0000 7f48ea729700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c4370 0x56227e9c47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.932+0000 7f48ea729700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c4370 0x56227e9c47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40592/0 (socket says 172.21.15.70:40592) 2024-06-04T05:21:48.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.932+0000 7f48ea729700 1 -- 172.21.15.70:0/2459625561 learned_addr learned my addr 172.21.15.70:0/2459625561 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:48.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.932+0000 7f48e9727700 1 --2- 172.21.15.70:0/2459625561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c0620 0x56227e9c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.932+0000 7f48e9f28700 1 --2- 172.21.15.70:0/2459625561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9bfc70 0x56227e9c0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.933+0000 7f48ea729700 1 -- 172.21.15.70:0/2459625561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c0620 msgr2=0x56227e9c3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:48.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.933+0000 7f48ea729700 1 --2- 172.21.15.70:0/2459625561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c0620 0x56227e9c3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.933+0000 7f48ea729700 1 -- 172.21.15.70:0/2459625561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9bfc70 msgr2=0x56227e9c0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:48.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.933+0000 7f48ea729700 1 --2- 172.21.15.70:0/2459625561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9bfc70 0x56227e9c0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.933+0000 7f48ea729700 1 -- 172.21.15.70:0/2459625561 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56227e85a8d0 con 0x56227e9c4370 2024-06-04T05:21:48.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.933+0000 7f48ea729700 1 --2- 172.21.15.70:0/2459625561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c4370 0x56227e9c47b0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f48dc0098f0 tx=0x7f48dc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=525ba259b2921572 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:48.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.933+0000 7f48cffff700 1 -- 172.21.15.70:0/2459625561 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc004bb0 con 0x56227e9c4370 2024-06-04T05:21:48.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.933+0000 7f48cffff700 1 -- 172.21.15.70:0/2459625561 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f48dc004d50 con 0x56227e9c4370 2024-06-04T05:21:48.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.933+0000 7f48cffff700 1 -- 172.21.15.70:0/2459625561 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc007cc0 con 0x56227e9c4370 2024-06-04T05:21:48.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.934+0000 7f48f6a21b80 1 -- 172.21.15.70:0/2459625561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c4370 msgr2=0x56227e9c47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:48.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.934+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/2459625561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c4370 0x56227e9c47b0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f48dc0098f0 tx=0x7f48dc009c00 comp rx=0 tx=0).stop 2024-06-04T05:21:48.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.934+0000 7f48f6a21b80 1 -- 172.21.15.70:0/2459625561 shutdown_connections 2024-06-04T05:21:48.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.934+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/2459625561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c4370 0x56227e9c47b0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.934+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/2459625561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9bfc70 0x56227e9c0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.934+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/2459625561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c0620 0x56227e9c3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.934+0000 7f48f6a21b80 1 -- 172.21.15.70:0/2459625561 >> 172.21.15.70:0/2459625561 conn(0x56227e8e9080 msgr2=0x56227e9b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:48.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.934+0000 7f48f6a21b80 1 -- 172.21.15.70:0/2459625561 shutdown_connections 2024-06-04T05:21:48.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.934+0000 7f48f6a21b80 1 -- 172.21.15.70:0/2459625561 wait complete. 2024-06-04T05:21:48.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.934+0000 7f48f6a21b80 1 -- start start 2024-06-04T05:21:48.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.934+0000 7f48f6a21b80 1 Processor -- start 2024-06-04T05:21:48.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48f6a21b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9bfc70 0x56227e9cb750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48f6a21b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9c0620 0x56227e9c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48f6a21b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c4370 0x56227e9c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48f6a21b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56227e810fa0 con 0x56227e9bfc70 2024-06-04T05:21:48.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48f6a21b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56227e8ac7c0 con 0x56227e9c0620 2024-06-04T05:21:48.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48f6a21b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56227e9cc530 con 0x56227e9c4370 2024-06-04T05:21:48.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48f6a21b80 10 client.0 _ll_register_callbacks cb 0x56227e8a5920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:48.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48e9f28700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9bfc70 0x56227e9cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48e9f28700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9bfc70 0x56227e9cb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40602/0 (socket says 172.21.15.70:40602) 2024-06-04T05:21:48.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48e9f28700 1 -- 172.21.15.70:0/3744321543 learned_addr learned my addr 172.21.15.70:0/3744321543 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:48.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48ea729700 1 --2- 172.21.15.70:0/3744321543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c4370 0x56227e9c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48e9727700 1 --2- 172.21.15.70:0/3744321543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9c0620 0x56227e9c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48e9f28700 1 -- 172.21.15.70:0/3744321543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c4370 msgr2=0x56227e9c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:48.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48e9f28700 1 --2- 172.21.15.70:0/3744321543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c4370 0x56227e9c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48e9f28700 1 -- 172.21.15.70:0/3744321543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9c0620 msgr2=0x56227e9c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:48.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48e9f28700 1 --2- 172.21.15.70:0/3744321543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9c0620 0x56227e9c8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48e9f28700 1 -- 172.21.15.70:0/3744321543 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56227e8a9690 con 0x56227e9bfc70 2024-06-04T05:21:48.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48e9727700 1 --2- 172.21.15.70:0/3744321543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9c0620 0x56227e9c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:21:48.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48e9f28700 1 --2- 172.21.15.70:0/3744321543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9bfc70 0x56227e9cb750 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f48e419aac0 tx=0x7f48e0009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a764e0d10236a84f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:48.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48cd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:21:48.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.935+0000 7f48cd7fa700 1 -- 172.21.15.70:0/3744321543 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e0034060 con 0x56227e9bfc70 2024-06-04T05:21:48.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.936+0000 7f48cd7fa700 1 -- 172.21.15.70:0/3744321543 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f48e0003cf0 con 0x56227e9bfc70 2024-06-04T05:21:48.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.936+0000 7f48f6a21b80 1 -- 172.21.15.70:0/3744321543 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48dc004d50 con 0x56227e9bfc70 2024-06-04T05:21:48.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.936+0000 7f48cd7fa700 1 -- 172.21.15.70:0/3744321543 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f48e002f460 con 0x56227e9bfc70 2024-06-04T05:21:48.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.936+0000 7f48f6a21b80 10 client.21612 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:48.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.936+0000 7f48f6a21b80 1 -- 172.21.15.70:0/3744321543 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x56227e9d6bd0 con 0x56227e9bfc70 2024-06-04T05:21:48.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.936+0000 7f48cd7fa700 1 -- 172.21.15.70:0/3744321543 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f48dc004d50 con 0x56227e9bfc70 2024-06-04T05:21:48.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.936+0000 7f48cd7fa700 1 -- 172.21.15.70:0/3744321543 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f48e000f580 con 0x56227e9bfc70 2024-06-04T05:21:48.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.936+0000 7f48f6a21b80 10 client.21612 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:48.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.936+0000 7f48f6a21b80 10 client.21612 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:21:48.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.936+0000 7f48f6a21b80 20 client.21612 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:48.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.936+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/3744321543 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x56227e9d9210 0x56227e9db620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.936+0000 7f48f6a21b80 4 client.21612 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:21:48.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.936+0000 7f48f6a21b80 1 -- 172.21.15.70:0/3744321543 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56227e9879e0 con 0x56227e9d9210 2024-06-04T05:21:48.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.936+0000 7f48e9727700 1 --2- 172.21.15.70:0/3744321543 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x56227e9d9210 0x56227e9db620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.937+0000 7f48e9727700 1 --2- 172.21.15.70:0/3744321543 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x56227e9d9210 0x56227e9db620 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=24fb7c0e612daf47 server_cookie=855e0ff2af5c07f1 in_seq=0 out_seq=0 2024-06-04T05:21:48.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.937+0000 7f48cd7fa700 10 client.21612 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:21:48.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.937+0000 7f48cd7fa700 1 -- 172.21.15.70:0/3744321543 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56227e9879e0 con 0x56227e9d9210 2024-06-04T05:21:48.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.937+0000 7f48cd7fa700 10 client.21612 handle_command_reply: tid=0 2024-06-04T05:21:48.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.938+0000 7f48f6a21b80 1 client.21612 shutdown 2024-06-04T05:21:48.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.938+0000 7f48f6a21b80 1 -- 172.21.15.70:0/3744321543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9bfc70 msgr2=0x56227e9cb750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:48.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.938+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/3744321543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9bfc70 0x56227e9cb750 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f48e419aac0 tx=0x7f48e0009cd0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.938+0000 7f48f6a21b80 1 -- 172.21.15.70:0/3744321543 shutdown_connections 2024-06-04T05:21:48.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.938+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/3744321543 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x56227e9d9210 0x56227e9db620 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.938+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/3744321543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c4370 0x56227e9c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.938+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/3744321543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9bfc70 0x56227e9cb750 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.938+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/3744321543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9c0620 0x56227e9c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.938+0000 7f48cd7fa700 0 client.21612 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:21:48.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.938+0000 7f48f6a21b80 1 -- 172.21.15.70:0/3744321543 >> 172.21.15.70:0/3744321543 conn(0x56227e8e9080 msgr2=0x56227e9c5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:48.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.938+0000 7f48f6a21b80 1 -- 172.21.15.70:0/3744321543 shutdown_connections 2024-06-04T05:21:48.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.938+0000 7f48f6a21b80 1 -- 172.21.15.70:0/3744321543 wait complete. 2024-06-04T05:21:48.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.938+0000 7f48f6a21b80 20 client.21612 trim_cache size 0 max 16384 2024-06-04T05:21:48.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.953+0000 7f48f6a21b80 1 Processor -- start 2024-06-04T05:21:48.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.953+0000 7f48f6a21b80 1 -- start start 2024-06-04T05:21:48.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.953+0000 7f48f6a21b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9bfc70 0x56227e9b6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.953+0000 7f48f6a21b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c0620 0x56227e9e0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.953+0000 7f48f6a21b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c4370 0x56227e9e1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.953+0000 7f48f6a21b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f48e41abbd0 con 0x56227e9c4370 2024-06-04T05:21:48.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.953+0000 7f48f6a21b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48e41ad7a0 con 0x56227e9bfc70 2024-06-04T05:21:48.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.953+0000 7f48f6a21b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f48e41ac300 con 0x56227e9c0620 2024-06-04T05:21:48.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.953+0000 7f48ea729700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c4370 0x56227e9e1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.953+0000 7f48ea729700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c4370 0x56227e9e1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40614/0 (socket says 172.21.15.70:40614) 2024-06-04T05:21:48.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.953+0000 7f48ea729700 1 -- 172.21.15.70:0/1215989279 learned_addr learned my addr 172.21.15.70:0/1215989279 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:48.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.953+0000 7f48e9727700 1 --2- 172.21.15.70:0/1215989279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c0620 0x56227e9e0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.953+0000 7f48e9f28700 1 --2- 172.21.15.70:0/1215989279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9bfc70 0x56227e9b6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.953+0000 7f48ea729700 1 -- 172.21.15.70:0/1215989279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c0620 msgr2=0x56227e9e0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:48.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.953+0000 7f48ea729700 1 --2- 172.21.15.70:0/1215989279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c0620 0x56227e9e0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.953+0000 7f48ea729700 1 -- 172.21.15.70:0/1215989279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9bfc70 msgr2=0x56227e9b6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:48.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.953+0000 7f48ea729700 1 --2- 172.21.15.70:0/1215989279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9bfc70 0x56227e9b6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.954+0000 7f48ea729700 1 -- 172.21.15.70:0/1215989279 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56227e9c40a0 con 0x56227e9c4370 2024-06-04T05:21:48.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.954+0000 7f48e9f28700 1 --2- 172.21.15.70:0/1215989279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9bfc70 0x56227e9b6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:21:48.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.954+0000 7f48ea729700 1 --2- 172.21.15.70:0/1215989279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c4370 0x56227e9e1430 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f48dc018000 tx=0x7f48dc005670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b296d3e1ce08a9b5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:48.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.954+0000 7f48cf7fe700 1 -- 172.21.15.70:0/1215989279 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc0094a0 con 0x56227e9c4370 2024-06-04T05:21:48.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.954+0000 7f48cf7fe700 1 -- 172.21.15.70:0/1215989279 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f48dc023030 con 0x56227e9c4370 2024-06-04T05:21:48.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.954+0000 7f48cf7fe700 1 -- 172.21.15.70:0/1215989279 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc015480 con 0x56227e9c4370 2024-06-04T05:21:48.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 1 -- 172.21.15.70:0/1215989279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c4370 msgr2=0x56227e9e1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:48.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/1215989279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c4370 0x56227e9e1430 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f48dc018000 tx=0x7f48dc005670 comp rx=0 tx=0).stop 2024-06-04T05:21:48.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 1 -- 172.21.15.70:0/1215989279 shutdown_connections 2024-06-04T05:21:48.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/1215989279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c4370 0x56227e9e1430 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/1215989279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9bfc70 0x56227e9b6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/1215989279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c0620 0x56227e9e0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 1 -- 172.21.15.70:0/1215989279 >> 172.21.15.70:0/1215989279 conn(0x56227e8e9080 msgr2=0x56227e9baf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 1 -- 172.21.15.70:0/1215989279 shutdown_connections 2024-06-04T05:21:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 1 -- 172.21.15.70:0/1215989279 wait complete. 2024-06-04T05:21:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 1 -- start start 2024-06-04T05:21:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 1 Processor -- start 2024-06-04T05:21:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9bfc70 0x56227e9c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c0620 0x56227e9c1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c4370 0x56227e9c2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f48e41ad0a0 con 0x56227e9c0620 2024-06-04T05:21:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48e41ac9d0 con 0x56227e9bfc70 2024-06-04T05:21:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f48e41a9bb0 con 0x56227e9c4370 2024-06-04T05:21:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.955+0000 7f48f6a21b80 10 client.0 _ll_register_callbacks cb 0x56227e8a5920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48e9727700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c0620 0x56227e9c1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48e9727700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c0620 0x56227e9c1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40630/0 (socket says 172.21.15.70:40630) 2024-06-04T05:21:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48e9727700 1 -- 172.21.15.70:0/2408686309 learned_addr learned my addr 172.21.15.70:0/2408686309 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48ea729700 1 --2- 172.21.15.70:0/2408686309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c4370 0x56227e9c2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48e9f28700 1 --2- 172.21.15.70:0/2408686309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9bfc70 0x56227e9c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48e9727700 1 -- 172.21.15.70:0/2408686309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c4370 msgr2=0x56227e9c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48e9727700 1 --2- 172.21.15.70:0/2408686309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c4370 0x56227e9c2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48e9727700 1 -- 172.21.15.70:0/2408686309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9bfc70 msgr2=0x56227e9c18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48e9727700 1 --2- 172.21.15.70:0/2408686309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9bfc70 0x56227e9c18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48e9727700 1 -- 172.21.15.70:0/2408686309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56227ea42dc0 con 0x56227e9c0620 2024-06-04T05:21:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48e9f28700 1 --2- 172.21.15.70:0/2408686309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9bfc70 0x56227e9c18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:21:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48e9727700 1 --2- 172.21.15.70:0/2408686309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c0620 0x56227e9c1e10 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f48d400b500 tx=0x7f48d4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2750271763dd06 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48cd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:21:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48cd7fa700 1 -- 172.21.15.70:0/2408686309 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d40b1f10 con 0x56227e9c0620 2024-06-04T05:21:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48cd7fa700 1 -- 172.21.15.70:0/2408686309 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f48d4010530 con 0x56227e9c0620 2024-06-04T05:21:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48f6a21b80 1 -- 172.21.15.70:0/2408686309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48dc023030 con 0x56227e9c0620 2024-06-04T05:21:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48cd7fa700 1 -- 172.21.15.70:0/2408686309 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f48d40ad0b0 con 0x56227e9c0620 2024-06-04T05:21:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48f6a21b80 10 client.21618 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.956+0000 7f48f6a21b80 1 -- 172.21.15.70:0/2408686309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x56227ea44040 con 0x56227e9c0620 2024-06-04T05:21:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.957+0000 7f48cd7fa700 1 -- 172.21.15.70:0/2408686309 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f48dc023030 con 0x56227e9c0620 2024-06-04T05:21:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.957+0000 7f48cd7fa700 1 -- 172.21.15.70:0/2408686309 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f48d40b1a80 con 0x56227e9c0620 2024-06-04T05:21:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.957+0000 7f48f6a21b80 10 client.21618 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.957+0000 7f48f6a21b80 10 client.21618 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:21:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.957+0000 7f48f6a21b80 20 client.21618 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.957+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/2408686309 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x56227ea44650 0x56227ea46a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.957+0000 7f48f6a21b80 4 client.21618 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:21:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.957+0000 7f48f6a21b80 1 -- 172.21.15.70:0/2408686309 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56227e91cbe0 con 0x56227ea44650 2024-06-04T05:21:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.957+0000 7f48e9f28700 1 --2- 172.21.15.70:0/2408686309 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x56227ea44650 0x56227ea46a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.957+0000 7f48cd7fa700 10 client.21618 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:21:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.957+0000 7f48e9f28700 1 --2- 172.21.15.70:0/2408686309 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x56227ea44650 0x56227ea46a40 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2508d7263aa7cfa server_cookie=8d14bc0de43be24e in_seq=0 out_seq=0 2024-06-04T05:21:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.958+0000 7f48cd7fa700 1 -- 172.21.15.70:0/2408686309 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56227e91cbe0 con 0x56227ea44650 2024-06-04T05:21:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.958+0000 7f48cd7fa700 10 client.21618 handle_command_reply: tid=0 2024-06-04T05:21:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.958+0000 7f48f6a21b80 1 client.21618 shutdown 2024-06-04T05:21:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.958+0000 7f48f6a21b80 1 -- 172.21.15.70:0/2408686309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c0620 msgr2=0x56227e9c1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.958+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/2408686309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c0620 0x56227e9c1e10 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f48d400b500 tx=0x7f48d4000bc0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.958+0000 7f48f6a21b80 1 -- 172.21.15.70:0/2408686309 shutdown_connections 2024-06-04T05:21:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.958+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/2408686309 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x56227ea44650 0x56227ea46a40 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.958+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/2408686309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56227e9c4370 0x56227e9c2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.958+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/2408686309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56227e9bfc70 0x56227e9c18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.958+0000 7f48cd7fa700 0 client.21618 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:21:48.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.958+0000 7f48f6a21b80 1 --2- 172.21.15.70:0/2408686309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56227e9c0620 0x56227e9c1e10 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.958+0000 7f48f6a21b80 1 -- 172.21.15.70:0/2408686309 >> 172.21.15.70:0/2408686309 conn(0x56227e8e9080 msgr2=0x56227e9c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:48.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.958+0000 7f48f6a21b80 1 -- 172.21.15.70:0/2408686309 shutdown_connections 2024-06-04T05:21:48.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.958+0000 7f48f6a21b80 1 -- 172.21.15.70:0/2408686309 wait complete. 2024-06-04T05:21:48.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.958+0000 7f48f6a21b80 20 client.21618 trim_cache size 0 max 16384 2024-06-04T05:21:48.966 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:21:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.959+0000 7f48ec18c700 1 -- 172.21.15.70:0/296979526 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48d00b2a70 msgr2=0x7f48d00b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.959+0000 7f48ec18c700 1 --2- 172.21.15.70:0/296979526 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48d00b2a70 0x7f48d00b4f30 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f48e0006010 tx=0x7f48e0005e80 comp rx=0 tx=0).stop 2024-06-04T05:21:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.959+0000 7f48ec18c700 1 -- 172.21.15.70:0/296979526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48e41041a0 msgr2=0x7f48e419fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.959+0000 7f48ec18c700 1 --2- 172.21.15.70:0/296979526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48e41041a0 0x7f48e419fd00 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f48d400b530 tx=0x7f48d400b840 comp rx=0 tx=0).stop 2024-06-04T05:21:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.959+0000 7f48ec18c700 1 -- 172.21.15.70:0/296979526 shutdown_connections 2024-06-04T05:21:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.959+0000 7f48ec18c700 1 --2- 172.21.15.70:0/296979526 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f48d00b2a70 0x7f48d00b4f30 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.959+0000 7f48ec18c700 1 --2- 172.21.15.70:0/296979526 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48e41088f0 0x7f48e4199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.959+0000 7f48ec18c700 1 --2- 172.21.15.70:0/296979526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48e41037f0 0x7f48e419f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.959+0000 7f48ec18c700 1 --2- 172.21.15.70:0/296979526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48e41041a0 0x7f48e419fd00 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:48.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.959+0000 7f48ec18c700 1 -- 172.21.15.70:0/296979526 >> 172.21.15.70:0/296979526 conn(0x7f48e407a6a0 msgr2=0x7f48e4101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.959+0000 7f48ec18c700 1 -- 172.21.15.70:0/296979526 shutdown_connections 2024-06-04T05:21:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:48.959+0000 7f48ec18c700 1 -- 172.21.15.70:0/296979526 wait complete. 2024-06-04T05:21:49.176 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:21:49.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:49 smithi070 ceph-mon[32631]: pgmap v776: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-06-04T05:21:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:49 smithi112 ceph-mon[36453]: pgmap v776: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-06-04T05:21:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:49 smithi098 ceph-mon[35569]: pgmap v776: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-06-04T05:21:49.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.82 MB/sec execute 521 sec latency 12120.742 ms 2024-06-04T05:21:49.962 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:21:50.059 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:21:50.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.81 MB/sec execute 522 sec latency 13120.832 ms 2024-06-04T05:21:51.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:51 smithi070 ceph-mon[32631]: pgmap v777: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-06-04T05:21:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:51 smithi112 ceph-mon[36453]: pgmap v777: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-06-04T05:21:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:51 smithi098 ceph-mon[35569]: pgmap v777: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-06-04T05:21:51.833 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.80 MB/sec execute 523 sec latency 14120.901 ms 2024-06-04T05:21:51.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.871+0000 7f88c3867700 1 -- 172.21.15.70:0/1104741754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88bc102810 msgr2=0x7f88bc106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:51.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.871+0000 7f88c3867700 1 --2- 172.21.15.70:0/1104741754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88bc102810 0x7f88bc106810 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f88ac0099c0 tx=0x7f88ac009cd0 comp rx=0 tx=0).stop 2024-06-04T05:21:51.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.871+0000 7f88c3867700 1 -- 172.21.15.70:0/1104741754 shutdown_connections 2024-06-04T05:21:51.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.871+0000 7f88c3867700 1 --2- 172.21.15.70:0/1104741754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88bc106ef0 0x7f88bc10eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:51.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.871+0000 7f88c3867700 1 --2- 172.21.15.70:0/1104741754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88bc101e60 0x7f88bc102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:51.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.871+0000 7f88c3867700 1 --2- 172.21.15.70:0/1104741754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88bc102810 0x7f88bc106810 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:51.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.871+0000 7f88c3867700 1 -- 172.21.15.70:0/1104741754 >> 172.21.15.70:0/1104741754 conn(0x7f88bc078ab0 msgr2=0x7f88bc0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:51.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.871+0000 7f88c3867700 1 -- 172.21.15.70:0/1104741754 shutdown_connections 2024-06-04T05:21:51.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.871+0000 7f88c3867700 1 -- 172.21.15.70:0/1104741754 wait complete. 2024-06-04T05:21:51.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.872+0000 7f88c3867700 1 Processor -- start 2024-06-04T05:21:51.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.872+0000 7f88c3867700 1 -- start start 2024-06-04T05:21:51.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.872+0000 7f88c3867700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88bc101e60 0x7f88bc0fdf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:51.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.872+0000 7f88c3867700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88bc102810 0x7f88bc0fe460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:51.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.872+0000 7f88c3867700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88bc106ef0 0x7f88bc0fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:51.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.872+0000 7f88c3867700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc1113c0 con 0x7f88bc101e60 2024-06-04T05:21:51.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.872+0000 7f88c3867700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc111240 con 0x7f88bc106ef0 2024-06-04T05:21:51.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.872+0000 7f88c3867700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc111540 con 0x7f88bc102810 2024-06-04T05:21:51.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.872+0000 7f88c1603700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88bc101e60 0x7f88bc0fdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:51.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.872+0000 7f88c1603700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88bc101e60 0x7f88bc0fdf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40648/0 (socket says 172.21.15.70:40648) 2024-06-04T05:21:51.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.872+0000 7f88c1603700 1 -- 172.21.15.70:0/1435846361 learned_addr learned my addr 172.21.15.70:0/1435846361 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:51.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.872+0000 7f88c0e02700 1 --2- 172.21.15.70:0/1435846361 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88bc102810 0x7f88bc0fe460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:51.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.872+0000 7f88c1e04700 1 --2- 172.21.15.70:0/1435846361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88bc106ef0 0x7f88bc0fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:51.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.873+0000 7f88c1603700 1 -- 172.21.15.70:0/1435846361 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88bc102810 msgr2=0x7f88bc0fe460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:51.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.873+0000 7f88c1603700 1 --2- 172.21.15.70:0/1435846361 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88bc102810 0x7f88bc0fe460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:51.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.873+0000 7f88c1603700 1 -- 172.21.15.70:0/1435846361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88bc106ef0 msgr2=0x7f88bc0fe9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:51.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.873+0000 7f88c1603700 1 --2- 172.21.15.70:0/1435846361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88bc106ef0 0x7f88bc0fe9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:51.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.873+0000 7f88c1603700 1 -- 172.21.15.70:0/1435846361 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88bc1a8100 con 0x7f88bc101e60 2024-06-04T05:21:51.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.873+0000 7f88c1603700 1 --2- 172.21.15.70:0/1435846361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88bc101e60 0x7f88bc0fdf20 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f88b800dbe0 tx=0x7f88b800def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.873+0000 7f88b27fc700 1 -- 172.21.15.70:0/1435846361 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b8007c60 con 0x7f88bc101e60 2024-06-04T05:21:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.873+0000 7f88c3867700 1 -- 172.21.15.70:0/1435846361 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88bc1a83f0 con 0x7f88bc101e60 2024-06-04T05:21:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.873+0000 7f88b27fc700 1 -- 172.21.15.70:0/1435846361 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f88b8007e00 con 0x7f88bc101e60 2024-06-04T05:21:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.873+0000 7f88b27fc700 1 -- 172.21.15.70:0/1435846361 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b8004e60 con 0x7f88bc101e60 2024-06-04T05:21:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.873+0000 7f88c3867700 1 -- 172.21.15.70:0/1435846361 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88bc1a8950 con 0x7f88bc101e60 2024-06-04T05:21:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.874+0000 7f88c3867700 1 -- 172.21.15.70:0/1435846361 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88bc1032f0 con 0x7f88bc101e60 2024-06-04T05:21:51.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.876+0000 7f88b27fc700 1 -- 172.21.15.70:0/1435846361 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f88b8004750 con 0x7f88bc101e60 2024-06-04T05:21:51.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.876+0000 7f88b27fc700 1 --2- 172.21.15.70:0/1435846361 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f88a80b29f0 0x7f88a80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:51.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.876+0000 7f88b27fc700 1 -- 172.21.15.70:0/1435846361 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f88b80e2360 con 0x7f88bc101e60 2024-06-04T05:21:51.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.876+0000 7f88c0e02700 1 --2- 172.21.15.70:0/1435846361 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f88a80b29f0 0x7f88a80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:51.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.877+0000 7f88c0e02700 1 --2- 172.21.15.70:0/1435846361 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f88a80b29f0 0x7f88a80b4eb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009990 tx=0x7f88ac005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:51.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:51.880+0000 7f88b27fc700 1 -- 172.21.15.70:0/1435846361 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f88b8051df0 con 0x7f88bc101e60 2024-06-04T05:21:51.903 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:21:52.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:52.140+0000 7f88c3867700 1 -- 172.21.15.70:0/1435846361 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88bc02e560 con 0x7f88bc101e60 2024-06-04T05:21:52.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:52.141+0000 7f88b27fc700 1 -- 172.21.15.70:0/1435846361 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+11933 (secure 0 0 0) 0x7f88b80a9e20 con 0x7f88bc101e60 2024-06-04T05:21:52.143 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:21:52.144 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":64,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:20:02.842346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:21:52.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:52.143+0000 7f88c3867700 1 -- 172.21.15.70:0/1435846361 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f88a80b29f0 msgr2=0x7f88a80b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:52.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:52.143+0000 7f88c3867700 1 --2- 172.21.15.70:0/1435846361 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f88a80b29f0 0x7f88a80b4eb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009990 tx=0x7f88ac005fb0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:52.143+0000 7f88c3867700 1 -- 172.21.15.70:0/1435846361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88bc101e60 msgr2=0x7f88bc0fdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:52.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:52.143+0000 7f88c3867700 1 --2- 172.21.15.70:0/1435846361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88bc101e60 0x7f88bc0fdf20 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f88b800dbe0 tx=0x7f88b800def0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:52.143+0000 7f88c3867700 1 -- 172.21.15.70:0/1435846361 shutdown_connections 2024-06-04T05:21:52.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:52.143+0000 7f88c3867700 1 --2- 172.21.15.70:0/1435846361 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f88a80b29f0 0x7f88a80b4eb0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:52.143+0000 7f88c3867700 1 --2- 172.21.15.70:0/1435846361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88bc106ef0 0x7f88bc0fe9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:52.143+0000 7f88c3867700 1 --2- 172.21.15.70:0/1435846361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88bc101e60 0x7f88bc0fdf20 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:52.143+0000 7f88c3867700 1 --2- 172.21.15.70:0/1435846361 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88bc102810 0x7f88bc0fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:52.143+0000 7f88c3867700 1 -- 172.21.15.70:0/1435846361 >> 172.21.15.70:0/1435846361 conn(0x7f88bc078ab0 msgr2=0x7f88bc1056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:52.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:52.143+0000 7f88c3867700 1 -- 172.21.15.70:0/1435846361 shutdown_connections 2024-06-04T05:21:52.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:52.143+0000 7f88c3867700 1 -- 172.21.15.70:0/1435846361 wait complete. 2024-06-04T05:21:52.146 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 64 2024-06-04T05:21:52.620 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:52 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1435846361' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:21:52.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:52 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1435846361' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:21:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.712+0000 7f1cd2b4d640 1 -- 172.21.15.70:0/2151945151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ccc10d040 msgr2=0x7f1ccc10f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.712+0000 7f1cd2b4d640 1 --2- 172.21.15.70:0/2151945151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ccc10d040 0x7f1ccc10f4b0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4009ca0 tx=0x7f1cc402a950 comp rx=0 tx=0).stop 2024-06-04T05:21:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1cd2b4d640 1 -- 172.21.15.70:0/2151945151 shutdown_connections 2024-06-04T05:21:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1cd2b4d640 1 --2- 172.21.15.70:0/2151945151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ccc104620 0x7f1ccc10cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1cd2b4d640 1 --2- 172.21.15.70:0/2151945151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ccc10d040 0x7f1ccc10f4b0 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1cd2b4d640 1 --2- 172.21.15.70:0/2151945151 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ccc103d00 0x7f1ccc1040e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1cd2b4d640 1 -- 172.21.15.70:0/2151945151 >> 172.21.15.70:0/2151945151 conn(0x7f1ccc078ad0 msgr2=0x7f1ccc078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1cd2b4d640 1 -- 172.21.15.70:0/2151945151 shutdown_connections 2024-06-04T05:21:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1cd2b4d640 1 -- 172.21.15.70:0/2151945151 wait complete. 2024-06-04T05:21:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1cd2b4d640 1 Processor -- start 2024-06-04T05:21:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1cd2b4d640 1 -- start start 2024-06-04T05:21:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1cd2b4d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ccc103d00 0x7f1ccc101950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1cd2b4d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ccc104620 0x7f1ccc101e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1cd2b4d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ccc10d040 0x7f1ccc1023d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1cd2b4d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1ccc112c20 con 0x7f1ccc104620 2024-06-04T05:21:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1cd2b4d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ccc112aa0 con 0x7f1ccc10d040 2024-06-04T05:21:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1cd2b4d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1ccc112da0 con 0x7f1ccc103d00 2024-06-04T05:21:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1ccbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ccc104620 0x7f1ccc101e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1ccbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ccc104620 0x7f1ccc101e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40666/0 (socket says 172.21.15.70:40666) 2024-06-04T05:21:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.713+0000 7f1ccbfff640 1 -- 172.21.15.70:0/2352029044 learned_addr learned my addr 172.21.15.70:0/2352029044 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.714+0000 7f1cd08e9640 1 --2- 172.21.15.70:0/2352029044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ccc103d00 0x7f1ccc101950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.714+0000 7f1ccbfff640 1 -- 172.21.15.70:0/2352029044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ccc103d00 msgr2=0x7f1ccc101950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.714+0000 7f1ccbfff640 1 --2- 172.21.15.70:0/2352029044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ccc103d00 0x7f1ccc101950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.714+0000 7f1ccbfff640 1 -- 172.21.15.70:0/2352029044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ccc10d040 msgr2=0x7f1ccc1023d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.714+0000 7f1ccbfff640 1 --2- 172.21.15.70:0/2352029044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ccc10d040 0x7f1ccc1023d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.714+0000 7f1ccbfff640 1 -- 172.21.15.70:0/2352029044 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ccc1a76a0 con 0x7f1ccc104620 2024-06-04T05:21:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.714+0000 7f1ccbfff640 1 --2- 172.21.15.70:0/2352029044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ccc104620 0x7f1ccc101e90 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f1cc000b4a0 tx=0x7f1cc000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.714+0000 7f1cc9ffb640 1 -- 172.21.15.70:0/2352029044 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc0014070 con 0x7f1ccc104620 2024-06-04T05:21:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.714+0000 7f1cc9ffb640 1 -- 172.21.15.70:0/2352029044 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1cc0010aa0 con 0x7f1ccc104620 2024-06-04T05:21:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.714+0000 7f1cd2b4d640 1 -- 172.21.15.70:0/2352029044 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ccc1a7990 con 0x7f1ccc104620 2024-06-04T05:21:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.714+0000 7f1cc9ffb640 1 -- 172.21.15.70:0/2352029044 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc0017b70 con 0x7f1ccc104620 2024-06-04T05:21:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.714+0000 7f1cd2b4d640 1 -- 172.21.15.70:0/2352029044 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ccc1a7ee0 con 0x7f1ccc104620 2024-06-04T05:21:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.716+0000 7f1cc9ffb640 1 -- 172.21.15.70:0/2352029044 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f1cc0017070 con 0x7f1ccc104620 2024-06-04T05:21:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.717+0000 7f1cd5861740 1 Processor -- start 2024-06-04T05:21:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.717+0000 7f1cd5861740 1 -- start start 2024-06-04T05:21:52.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.717+0000 7f1cd5861740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c5f7820 0x56015c5f7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.717+0000 7f1cd5861740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f8140 0x56015c6024c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.717+0000 7f1cd5861740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c602a00 0x56015c604df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.717+0000 7f1cd5861740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56015c569c30 con 0x56015c602a00 2024-06-04T05:21:52.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.717+0000 7f1cd5861740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56015c5597d0 con 0x56015c5f8140 2024-06-04T05:21:52.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.717+0000 7f1cd5861740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56015c55d810 con 0x56015c5f7820 2024-06-04T05:21:52.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.717+0000 7f1ccbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c602a00 0x56015c604df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:52.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.717+0000 7f1cd10ea640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c5f7820 0x56015c5f7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:52.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.717+0000 7f1cd10ea640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c5f7820 0x56015c5f7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:40434/0 (socket says 172.21.15.70:40434) 2024-06-04T05:21:52.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.717+0000 7f1ccbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c602a00 0x56015c604df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40678/0 (socket says 172.21.15.70:40678) 2024-06-04T05:21:52.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.717+0000 7f1cd10ea640 1 -- 172.21.15.70:0/3317737305 learned_addr learned my addr 172.21.15.70:0/3317737305 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:52.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.717+0000 7f1cc9ffb640 1 --2- 172.21.15.70:0/2352029044 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1cb40b2a10 0x7f1cb40b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.717+0000 7f1cd08e9640 1 --2- 172.21.15.70:0/3317737305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f8140 0x56015c6024c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.717+0000 7f1cd10ea640 1 --2- 172.21.15.70:0/2352029044 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1cb40b2a10 0x7f1cb40b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.718+0000 7f1cc9ffb640 1 -- 172.21.15.70:0/2352029044 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f1cc00e1cc0 con 0x7f1ccc104620 2024-06-04T05:21:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.718+0000 7f1ccbfff640 1 -- 172.21.15.70:0/3317737305 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c5f7820 msgr2=0x56015c5f7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.718+0000 7f1ccbfff640 1 --2- 172.21.15.70:0/3317737305 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c5f7820 0x56015c5f7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.718+0000 7f1ccbfff640 1 -- 172.21.15.70:0/3317737305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f8140 msgr2=0x56015c6024c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.718+0000 7f1ccbfff640 1 --2- 172.21.15.70:0/3317737305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f8140 0x56015c6024c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.718+0000 7f1ccbfff640 1 -- 172.21.15.70:0/3317737305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56015c26e210 con 0x56015c602a00 2024-06-04T05:21:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.718+0000 7f1cd10ea640 1 --2- 172.21.15.70:0/3317737305 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c5f7820 0x56015c5f7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:21:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.718+0000 7f1ccbfff640 1 --2- 172.21.15.70:0/3317737305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c602a00 0x56015c604df0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f1cc00df4f0 tx=0x7f1cc00a5070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf6a4ac0b905e86 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.718+0000 7f1cd10ea640 1 --2- 172.21.15.70:0/2352029044 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1cb40b2a10 0x7f1cb40b4ed0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1cc402f800 tx=0x7f1cc40095c0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.718+0000 7f1cb2ffd640 1 -- 172.21.15.70:0/3317737305 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc00e3020 con 0x56015c602a00 2024-06-04T05:21:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.718+0000 7f1cb2ffd640 1 -- 172.21.15.70:0/3317737305 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1cc00a8b90 con 0x56015c602a00 2024-06-04T05:21:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.718+0000 7f1cb2ffd640 1 -- 172.21.15.70:0/3317737305 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc001e070 con 0x56015c602a00 2024-06-04T05:21:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.719+0000 7f1cd5861740 1 -- 172.21.15.70:0/3317737305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c602a00 msgr2=0x56015c604df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.719+0000 7f1cd5861740 1 --2- 172.21.15.70:0/3317737305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c602a00 0x56015c604df0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f1cc00df4f0 tx=0x7f1cc00a5070 comp rx=0 tx=0).stop 2024-06-04T05:21:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.719+0000 7f1cd5861740 1 -- 172.21.15.70:0/3317737305 shutdown_connections 2024-06-04T05:21:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.719+0000 7f1cd5861740 1 --2- 172.21.15.70:0/3317737305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f8140 0x56015c6024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.719+0000 7f1cd5861740 1 --2- 172.21.15.70:0/3317737305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c602a00 0x56015c604df0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.719+0000 7f1cd5861740 1 --2- 172.21.15.70:0/3317737305 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c5f7820 0x56015c5f7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.719+0000 7f1cd5861740 1 -- 172.21.15.70:0/3317737305 >> 172.21.15.70:0/3317737305 conn(0x56015c5f3d30 msgr2=0x56015c5f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.719+0000 7f1cd5861740 1 -- 172.21.15.70:0/3317737305 shutdown_connections 2024-06-04T05:21:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.719+0000 7f1cd5861740 1 -- 172.21.15.70:0/3317737305 wait complete. 2024-06-04T05:21:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.719+0000 7f1cd5861740 1 -- start start 2024-06-04T05:21:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.719+0000 7f1cd5861740 1 Processor -- start 2024-06-04T05:21:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.719+0000 7f1cd5861740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f7820 0x56015c5ff9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.719+0000 7f1cd5861740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c5f8140 0x56015c5ffee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.719+0000 7f1cd5861740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c602a00 0x56015c600420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1cd5861740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56015c23f790 con 0x56015c5f8140 2024-06-04T05:21:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1cd5861740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56015c515890 con 0x56015c5f7820 2024-06-04T05:21:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1cd5861740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56015c605570 con 0x56015c602a00 2024-06-04T05:21:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1cd5861740 10 client.0 _ll_register_callbacks cb 0x56015c26da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1ccbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c5f8140 0x56015c5ffee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1ccbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c5f8140 0x56015c5ffee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40692/0 (socket says 172.21.15.70:40692) 2024-06-04T05:21:52.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1ccbfff640 1 -- 172.21.15.70:0/506252866 learned_addr learned my addr 172.21.15.70:0/506252866 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:52.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1cd08e9640 1 --2- 172.21.15.70:0/506252866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f7820 0x56015c5ff9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:52.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1ccbfff640 1 -- 172.21.15.70:0/506252866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c602a00 msgr2=0x56015c600420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:52.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1ccbfff640 1 --2- 172.21.15.70:0/506252866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c602a00 0x56015c600420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1ccbfff640 1 -- 172.21.15.70:0/506252866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f7820 msgr2=0x56015c5ff9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:52.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1ccbfff640 1 --2- 172.21.15.70:0/506252866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f7820 0x56015c5ff9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1ccbfff640 1 -- 172.21.15.70:0/506252866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56015c33a620 con 0x56015c5f8140 2024-06-04T05:21:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1cd08e9640 1 --2- 172.21.15.70:0/506252866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f7820 0x56015c5ff9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:21:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1ccbfff640 1 --2- 172.21.15.70:0/506252866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c5f8140 0x56015c5ffee0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f1cc00df5a0 tx=0x7f1cc00ad510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61b233606df9798 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1cabfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:21:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1cabfff640 1 -- 172.21.15.70:0/506252866 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc001e070 con 0x56015c5f8140 2024-06-04T05:21:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1cabfff640 1 -- 172.21.15.70:0/506252866 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f1cc00a5310 con 0x56015c5f8140 2024-06-04T05:21:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1cd5861740 1 -- 172.21.15.70:0/506252866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cc00a8b90 con 0x56015c5f8140 2024-06-04T05:21:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1cabfff640 1 -- 172.21.15.70:0/506252866 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f1cc00b1ee0 con 0x56015c5f8140 2024-06-04T05:21:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1cd5861740 10 client.21636 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.720+0000 7f1cd5861740 1 -- 172.21.15.70:0/506252866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x56015c6112e0 con 0x56015c5f8140 2024-06-04T05:21:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.721+0000 7f1cabfff640 1 -- 172.21.15.70:0/506252866 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1cc00a8b90 con 0x56015c5f8140 2024-06-04T05:21:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.721+0000 7f1cabfff640 1 -- 172.21.15.70:0/506252866 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f1cc00dfb30 con 0x56015c5f8140 2024-06-04T05:21:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.721+0000 7f1cd5861740 10 client.21636 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.721+0000 7f1cd5861740 10 client.21636 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:21:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.721+0000 7f1cd5861740 20 client.21636 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.721+0000 7f1cd5861740 1 --2- 172.21.15.70:0/506252866 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56015c612630 0x56015c614a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.721+0000 7f1cd5861740 4 client.21636 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:21:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.721+0000 7f1cd5861740 1 -- 172.21.15.70:0/506252866 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56015c56bb80 con 0x56015c612630 2024-06-04T05:21:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.721+0000 7f1cd08e9640 1 --2- 172.21.15.70:0/506252866 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56015c612630 0x56015c614a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.722+0000 7f1cd08e9640 1 --2- 172.21.15.70:0/506252866 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56015c612630 0x56015c614a20 crc :-1 s=READY pgs=509 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd5bde44bf4e5e5f server_cookie=d1a00cee7ee52ff4 in_seq=0 out_seq=0 2024-06-04T05:21:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.722+0000 7f1cabfff640 10 client.21636 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.722+0000 7f1cabfff640 1 -- 172.21.15.70:0/506252866 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56015c56bb80 con 0x56015c612630 2024-06-04T05:21:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.722+0000 7f1cabfff640 10 client.21636 handle_command_reply: tid=0 2024-06-04T05:21:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.722+0000 7f1cd5861740 1 client.21636 shutdown 2024-06-04T05:21:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.722+0000 7f1cd5861740 1 -- 172.21.15.70:0/506252866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c5f8140 msgr2=0x56015c5ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.722+0000 7f1cd5861740 1 --2- 172.21.15.70:0/506252866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c5f8140 0x56015c5ffee0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f1cc00df5a0 tx=0x7f1cc00ad510 comp rx=0 tx=0).stop 2024-06-04T05:21:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.722+0000 7f1cd5861740 1 -- 172.21.15.70:0/506252866 shutdown_connections 2024-06-04T05:21:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.722+0000 7f1cd5861740 1 --2- 172.21.15.70:0/506252866 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56015c612630 0x56015c614a20 crc :-1 s=READY pgs=509 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.722+0000 7f1cd5861740 1 --2- 172.21.15.70:0/506252866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c602a00 0x56015c600420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.722+0000 7f1cabfff640 0 client.21636 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.722+0000 7f1cd5861740 1 --2- 172.21.15.70:0/506252866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c5f8140 0x56015c5ffee0 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.722+0000 7f1cd5861740 1 --2- 172.21.15.70:0/506252866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f7820 0x56015c5ff9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.722+0000 7f1cd5861740 1 -- 172.21.15.70:0/506252866 >> 172.21.15.70:0/506252866 conn(0x56015c5f3d30 msgr2=0x56015c5f2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.723+0000 7f1cd5861740 1 -- 172.21.15.70:0/506252866 shutdown_connections 2024-06-04T05:21:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.723+0000 7f1cd5861740 1 -- 172.21.15.70:0/506252866 wait complete. 2024-06-04T05:21:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.723+0000 7f1cd5861740 20 client.21636 trim_cache size 0 max 16384 2024-06-04T05:21:52.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd5861740 1 Processor -- start 2024-06-04T05:21:52.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd5861740 1 -- start start 2024-06-04T05:21:52.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd5861740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c5f7820 0x56015c5f98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd5861740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f8140 0x56015c61a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd5861740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c602a00 0x56015c61a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd5861740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1ccc1b0940 con 0x56015c5f7820 2024-06-04T05:21:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd5861740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ccc1b2510 con 0x56015c5f8140 2024-06-04T05:21:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd5861740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1ccc1b1070 con 0x56015c602a00 2024-06-04T05:21:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd08e9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c5f7820 0x56015c5f98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd08e9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c5f7820 0x56015c5f98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40700/0 (socket says 172.21.15.70:40700) 2024-06-04T05:21:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd08e9640 1 -- 172.21.15.70:0/120061431 learned_addr learned my addr 172.21.15.70:0/120061431 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd10ea640 1 --2- 172.21.15.70:0/120061431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c602a00 0x56015c61a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1ccbfff640 1 --2- 172.21.15.70:0/120061431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f8140 0x56015c61a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd08e9640 1 -- 172.21.15.70:0/120061431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c602a00 msgr2=0x56015c61a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd08e9640 1 --2- 172.21.15.70:0/120061431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c602a00 0x56015c61a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd08e9640 1 -- 172.21.15.70:0/120061431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f8140 msgr2=0x56015c61a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd08e9640 1 --2- 172.21.15.70:0/120061431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f8140 0x56015c61a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd08e9640 1 -- 172.21.15.70:0/120061431 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56015c5f4ea0 con 0x56015c5f7820 2024-06-04T05:21:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.738+0000 7f1cd10ea640 1 --2- 172.21.15.70:0/120061431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c602a00 0x56015c61a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:21:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.739+0000 7f1cd08e9640 1 --2- 172.21.15.70:0/120061431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c5f7820 0x56015c5f98e0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x56015c614ff0 tx=0x7f1cbc013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c63c0f98895ae769 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.739+0000 7f1cb27fc640 1 -- 172.21.15.70:0/120061431 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc01d430 con 0x56015c5f7820 2024-06-04T05:21:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.739+0000 7f1cb27fc640 1 -- 172.21.15.70:0/120061431 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1cbc01da60 con 0x56015c5f7820 2024-06-04T05:21:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.739+0000 7f1cb27fc640 1 -- 172.21.15.70:0/120061431 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc011c40 con 0x56015c5f7820 2024-06-04T05:21:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.739+0000 7f1cd5861740 1 -- 172.21.15.70:0/120061431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c5f7820 msgr2=0x56015c5f98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.739+0000 7f1cd5861740 1 --2- 172.21.15.70:0/120061431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c5f7820 0x56015c5f98e0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x56015c614ff0 tx=0x7f1cbc013830 comp rx=0 tx=0).stop 2024-06-04T05:21:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.739+0000 7f1cd5861740 1 -- 172.21.15.70:0/120061431 shutdown_connections 2024-06-04T05:21:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.739+0000 7f1cd5861740 1 --2- 172.21.15.70:0/120061431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f8140 0x56015c61a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.739+0000 7f1cd5861740 1 --2- 172.21.15.70:0/120061431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c602a00 0x56015c61a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.752 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.739+0000 7f1cd5861740 1 --2- 172.21.15.70:0/120061431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c5f7820 0x56015c5f98e0 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.752 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.739+0000 7f1cd5861740 1 -- 172.21.15.70:0/120061431 >> 172.21.15.70:0/120061431 conn(0x56015c5f3d30 msgr2=0x56015c5f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.739+0000 7f1cd5861740 1 -- 172.21.15.70:0/120061431 shutdown_connections 2024-06-04T05:21:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.739+0000 7f1cd5861740 1 -- 172.21.15.70:0/120061431 wait complete. 2024-06-04T05:21:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.740+0000 7f1cd5861740 1 -- start start 2024-06-04T05:21:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.740+0000 7f1cd5861740 1 Processor -- start 2024-06-04T05:21:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.740+0000 7f1cd5861740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c5f7820 0x56015c606800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.740+0000 7f1cd5861740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f8140 0x56015c606d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.740+0000 7f1cd5861740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c602a00 0x56015c6072a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.740+0000 7f1cd5861740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1ccc1b1e10 con 0x56015c602a00 2024-06-04T05:21:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.740+0000 7f1cd5861740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ccc1b1740 con 0x56015c5f8140 2024-06-04T05:21:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.740+0000 7f1cd5861740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1ccc1ae940 con 0x56015c5f7820 2024-06-04T05:21:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.740+0000 7f1cd5861740 10 client.0 _ll_register_callbacks cb 0x7f1cbc00b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.740+0000 7f1cd10ea640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c602a00 0x56015c6072a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.740+0000 7f1cd10ea640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c602a00 0x56015c6072a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40706/0 (socket says 172.21.15.70:40706) 2024-06-04T05:21:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.740+0000 7f1cd10ea640 1 -- 172.21.15.70:0/3073226405 learned_addr learned my addr 172.21.15.70:0/3073226405 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.740+0000 7f1cd10ea640 1 -- 172.21.15.70:0/3073226405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c5f7820 msgr2=0x56015c606800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cd08e9640 1 --2- 172.21.15.70:0/3073226405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c5f7820 0x56015c606800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1ccbfff640 1 --2- 172.21.15.70:0/3073226405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f8140 0x56015c606d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cd10ea640 1 --2- 172.21.15.70:0/3073226405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c5f7820 0x56015c606800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cd10ea640 1 -- 172.21.15.70:0/3073226405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f8140 msgr2=0x56015c606d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cd10ea640 1 --2- 172.21.15.70:0/3073226405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f8140 0x56015c606d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cd10ea640 1 -- 172.21.15.70:0/3073226405 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56015c67c270 con 0x56015c602a00 2024-06-04T05:21:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cd08e9640 1 --2- 172.21.15.70:0/3073226405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c5f7820 0x56015c606800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:21:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cd10ea640 1 --2- 172.21.15.70:0/3073226405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c602a00 0x56015c6072a0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f1cc402af40 tx=0x7f1cc40099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d965f41a03654d00 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cabfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:21:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cabfff640 1 -- 172.21.15.70:0/3073226405 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc404f030 con 0x56015c602a00 2024-06-04T05:21:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cabfff640 1 -- 172.21.15.70:0/3073226405 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f1cc402f3a0 con 0x56015c602a00 2024-06-04T05:21:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cd5861740 1 -- 172.21.15.70:0/3073226405 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cbc01da60 con 0x56015c602a00 2024-06-04T05:21:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cabfff640 1 -- 172.21.15.70:0/3073226405 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f1cc4034040 con 0x56015c602a00 2024-06-04T05:21:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cd5861740 10 client.21642 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cd5861740 1 -- 172.21.15.70:0/3073226405 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x56015c67d490 con 0x56015c602a00 2024-06-04T05:21:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cabfff640 1 -- 172.21.15.70:0/3073226405 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1cbc01da60 con 0x56015c602a00 2024-06-04T05:21:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cabfff640 1 -- 172.21.15.70:0/3073226405 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f1cc4049040 con 0x56015c602a00 2024-06-04T05:21:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cd5861740 10 client.21642 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cd5861740 10 client.21642 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:21:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cd5861740 20 client.21642 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cd5861740 1 --2- 172.21.15.70:0/3073226405 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56015c67dc90 0x56015c6800b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cd5861740 4 client.21642 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:21:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.741+0000 7f1cd5861740 1 -- 172.21.15.70:0/3073226405 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56015c5f8c10 con 0x56015c67dc90 2024-06-04T05:21:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.742+0000 7f1cd08e9640 1 --2- 172.21.15.70:0/3073226405 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56015c67dc90 0x56015c6800b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.742+0000 7f1cabfff640 10 client.21642 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.742+0000 7f1cd08e9640 1 --2- 172.21.15.70:0/3073226405 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56015c67dc90 0x56015c6800b0 crc :-1 s=READY pgs=510 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d66bebf1b4f27663 server_cookie=29d3ec0309087012 in_seq=0 out_seq=0 2024-06-04T05:21:52.757 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:52 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1435846361' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:21:52.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.782+0000 7f1cabfff640 1 -- 172.21.15.70:0/3073226405 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56015c5f8c10 con 0x56015c67dc90 2024-06-04T05:21:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.782+0000 7f1cabfff640 10 client.21642 handle_command_reply: tid=0 2024-06-04T05:21:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.782+0000 7f1cd5861740 1 client.21642 shutdown 2024-06-04T05:21:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.782+0000 7f1cd5861740 1 -- 172.21.15.70:0/3073226405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c602a00 msgr2=0x56015c6072a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.782+0000 7f1cd5861740 1 --2- 172.21.15.70:0/3073226405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c602a00 0x56015c6072a0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f1cc402af40 tx=0x7f1cc40099b0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.782+0000 7f1cd5861740 1 -- 172.21.15.70:0/3073226405 shutdown_connections 2024-06-04T05:21:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.782+0000 7f1cd5861740 1 --2- 172.21.15.70:0/3073226405 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x56015c67dc90 0x56015c6800b0 crc :-1 s=READY pgs=510 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.782+0000 7f1cd5861740 1 --2- 172.21.15.70:0/3073226405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56015c5f8140 0x56015c606d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.782+0000 7f1cd5861740 1 --2- 172.21.15.70:0/3073226405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56015c602a00 0x56015c6072a0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.782+0000 7f1cabfff640 0 client.21642 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.782+0000 7f1cd5861740 1 --2- 172.21.15.70:0/3073226405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56015c5f7820 0x56015c606800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.782+0000 7f1cd5861740 1 -- 172.21.15.70:0/3073226405 >> 172.21.15.70:0/3073226405 conn(0x56015c5f3d30 msgr2=0x56015c5f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.782+0000 7f1cd5861740 1 -- 172.21.15.70:0/3073226405 shutdown_connections 2024-06-04T05:21:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.782+0000 7f1cd5861740 1 -- 172.21.15.70:0/3073226405 wait complete. 2024-06-04T05:21:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.782+0000 7f1cd5861740 20 client.21642 trim_cache size 0 max 16384 2024-06-04T05:21:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:21:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:21:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:21:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.783+0000 7f1cd2b4d640 1 -- 172.21.15.70:0/2352029044 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1cb40b2a10 msgr2=0x7f1cb40b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.783+0000 7f1cd2b4d640 1 --2- 172.21.15.70:0/2352029044 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1cb40b2a10 0x7f1cb40b4ed0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1cc402f800 tx=0x7f1cc40095c0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.783+0000 7f1cd2b4d640 1 -- 172.21.15.70:0/2352029044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ccc104620 msgr2=0x7f1ccc101e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.783+0000 7f1cd2b4d640 1 --2- 172.21.15.70:0/2352029044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ccc104620 0x7f1ccc101e90 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f1cc000b4a0 tx=0x7f1cc000b960 comp rx=0 tx=0).stop 2024-06-04T05:21:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.783+0000 7f1cd2b4d640 1 -- 172.21.15.70:0/2352029044 shutdown_connections 2024-06-04T05:21:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.783+0000 7f1cd2b4d640 1 --2- 172.21.15.70:0/2352029044 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1cb40b2a10 0x7f1cb40b4ed0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.783+0000 7f1cd2b4d640 1 --2- 172.21.15.70:0/2352029044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ccc10d040 0x7f1ccc1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.783+0000 7f1cd2b4d640 1 --2- 172.21.15.70:0/2352029044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ccc104620 0x7f1ccc101e90 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.783+0000 7f1cd2b4d640 1 --2- 172.21.15.70:0/2352029044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1ccc103d00 0x7f1ccc101950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.783+0000 7f1cd2b4d640 1 -- 172.21.15.70:0/2352029044 >> 172.21.15.70:0/2352029044 conn(0x7f1ccc078ad0 msgr2=0x7f1ccc0fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.783+0000 7f1cd2b4d640 1 -- 172.21.15.70:0/2352029044 shutdown_connections 2024-06-04T05:21:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:21:52.783+0000 7f1cd2b4d640 1 -- 172.21.15.70:0/2352029044 wait complete. 2024-06-04T05:21:52.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.79 MB/sec execute 524 sec latency 15121.010 ms 2024-06-04T05:21:53.380 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id c80d4a61 --release --await' 2024-06-04T05:21:53.380 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id c80d4a61 --release --await 2024-06-04T05:21:53.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:53 smithi070 ceph-mon[32631]: pgmap v778: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-06-04T05:21:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:53 smithi112 ceph-mon[36453]: pgmap v778: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-06-04T05:21:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:53 smithi098 ceph-mon[35569]: pgmap v778: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-06-04T05:21:53.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.77 MB/sec execute 525 sec latency 16121.053 ms 2024-06-04T05:21:54.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.459+0000 7f2bbe30c700 1 -- 172.21.15.70:0/1036864360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bb8101b30 msgr2=0x7f2bb8101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:54.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.459+0000 7f2bbe30c700 1 --2- 172.21.15.70:0/1036864360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bb8101b30 0x7f2bb8101fb0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f2bac0099c0 tx=0x7f2bac009cd0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.459+0000 7f2bbe30c700 1 -- 172.21.15.70:0/1036864360 shutdown_connections 2024-06-04T05:21:54.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.459+0000 7f2bbe30c700 1 --2- 172.21.15.70:0/1036864360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2bb81024f0 0x7f2bb81103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.459+0000 7f2bbe30c700 1 --2- 172.21.15.70:0/1036864360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bb8109520 0x7f2bb8109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.459+0000 7f2bbe30c700 1 --2- 172.21.15.70:0/1036864360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bb8101b30 0x7f2bb8101fb0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.459+0000 7f2bbe30c700 1 -- 172.21.15.70:0/1036864360 >> 172.21.15.70:0/1036864360 conn(0x7f2bb8078ab0 msgr2=0x7f2bb80ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:54.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.459+0000 7f2bbe30c700 1 -- 172.21.15.70:0/1036864360 shutdown_connections 2024-06-04T05:21:54.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.459+0000 7f2bbe30c700 1 -- 172.21.15.70:0/1036864360 wait complete. 2024-06-04T05:21:54.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.460+0000 7f2bbe30c700 1 Processor -- start 2024-06-04T05:21:54.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.460+0000 7f2bbe30c700 1 -- start start 2024-06-04T05:21:54.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.460+0000 7f2bbe30c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bb8101b30 0x7f2bb819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.460+0000 7f2bbe30c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2bb81024f0 0x7f2bb819fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.460+0000 7f2bbe30c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bb8109520 0x7f2bb8199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.460+0000 7f2bbe30c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb81135b0 con 0x7f2bb8101b30 2024-06-04T05:21:54.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.460+0000 7f2bbe30c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb8113430 con 0x7f2bb8109520 2024-06-04T05:21:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.460+0000 7f2bbe30c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb8113730 con 0x7f2bb81024f0 2024-06-04T05:21:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.460+0000 7f2bb7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bb8101b30 0x7f2bb819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.461+0000 7f2bb7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bb8101b30 0x7f2bb819f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40738/0 (socket says 172.21.15.70:40738) 2024-06-04T05:21:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.461+0000 7f2bb7fff700 1 -- 172.21.15.70:0/642390280 learned_addr learned my addr 172.21.15.70:0/642390280 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.461+0000 7f2bb77fe700 1 --2- 172.21.15.70:0/642390280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2bb81024f0 0x7f2bb819fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.461+0000 7f2bbc8a9700 1 --2- 172.21.15.70:0/642390280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bb8109520 0x7f2bb8199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.461+0000 7f2bb7fff700 1 -- 172.21.15.70:0/642390280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2bb81024f0 msgr2=0x7f2bb819fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.461+0000 7f2bb7fff700 1 --2- 172.21.15.70:0/642390280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2bb81024f0 0x7f2bb819fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.461+0000 7f2bb7fff700 1 -- 172.21.15.70:0/642390280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bb8109520 msgr2=0x7f2bb8199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.461+0000 7f2bb7fff700 1 --2- 172.21.15.70:0/642390280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bb8109520 0x7f2bb8199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.461+0000 7f2bb7fff700 1 -- 172.21.15.70:0/642390280 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb819a0b0 con 0x7f2bb8101b30 2024-06-04T05:21:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.461+0000 7f2bb7fff700 1 --2- 172.21.15.70:0/642390280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bb8101b30 0x7f2bb819f790 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800c9b0 tx=0x7f2ba800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.461+0000 7f2bb57fa700 1 -- 172.21.15.70:0/642390280 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8007b00 con 0x7f2bb8101b30 2024-06-04T05:21:54.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.461+0000 7f2bbe30c700 1 -- 172.21.15.70:0/642390280 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb819a3a0 con 0x7f2bb8101b30 2024-06-04T05:21:54.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.461+0000 7f2bbe30c700 1 -- 172.21.15.70:0/642390280 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb81a6660 con 0x7f2bb8101b30 2024-06-04T05:21:54.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.461+0000 7f2bb57fa700 1 -- 172.21.15.70:0/642390280 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2ba8007ca0 con 0x7f2bb8101b30 2024-06-04T05:21:54.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.461+0000 7f2bb57fa700 1 -- 172.21.15.70:0/642390280 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8018660 con 0x7f2bb8101b30 2024-06-04T05:21:54.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.464+0000 7f2bb57fa700 1 -- 172.21.15.70:0/642390280 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f2ba8004830 con 0x7f2bb8101b30 2024-06-04T05:21:54.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.465+0000 7f2bb57fa700 1 --2- 172.21.15.70:0/642390280 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2ba00b2a70 0x7f2ba00b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.465+0000 7f2bb77fe700 1 --2- 172.21.15.70:0/642390280 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2ba00b2a70 0x7f2ba00b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.465+0000 7f2bb57fa700 1 -- 172.21.15.70:0/642390280 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f2ba80e2530 con 0x7f2bb8101b30 2024-06-04T05:21:54.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.465+0000 7f2bc8ba1b80 1 Processor -- start 2024-06-04T05:21:54.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bc8ba1b80 1 -- start start 2024-06-04T05:21:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bb77fe700 1 --2- 172.21.15.70:0/642390280 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2ba00b2a70 0x7f2ba00b4f30 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2bac000c00 tx=0x7f2bac005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bc8ba1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745f320 0x559c6745f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bc8ba1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c674598a0 0x559c67459d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bc8ba1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c6745a260 0x559c67466080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bc8ba1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559c673b7d70 con 0x559c6745f320 2024-06-04T05:21:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bc8ba1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559c673a7910 con 0x559c6745a260 2024-06-04T05:21:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bc8ba1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559c673ab950 con 0x559c674598a0 2024-06-04T05:21:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bb7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745f320 0x559c6745f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bb7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745f320 0x559c6745f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40750/0 (socket says 172.21.15.70:40750) 2024-06-04T05:21:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bb7fff700 1 -- 172.21.15.70:0/1705258052 learned_addr learned my addr 172.21.15.70:0/1705258052 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bb77fe700 1 --2- 172.21.15.70:0/1705258052 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c674598a0 0x559c67459d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bbc8a9700 1 --2- 172.21.15.70:0/1705258052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c6745a260 0x559c67466080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bb7fff700 1 -- 172.21.15.70:0/1705258052 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c674598a0 msgr2=0x559c67459d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bb7fff700 1 --2- 172.21.15.70:0/1705258052 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c674598a0 0x559c67459d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bb7fff700 1 -- 172.21.15.70:0/1705258052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c6745a260 msgr2=0x559c67466080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bb7fff700 1 --2- 172.21.15.70:0/1705258052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c6745a260 0x559c67466080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.466+0000 7f2bb7fff700 1 -- 172.21.15.70:0/1705258052 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c67350fb0 con 0x559c6745f320 2024-06-04T05:21:54.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.467+0000 7f2bb7fff700 1 --2- 172.21.15.70:0/1705258052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745f320 0x559c6745f700 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80a94e0 tx=0x7f2ba80a9130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52b72b23a439830 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:54.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.467+0000 7f2b9e7fc700 1 -- 172.21.15.70:0/1705258052 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba80a9ee0 con 0x559c6745f320 2024-06-04T05:21:54.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.467+0000 7f2b9e7fc700 1 -- 172.21.15.70:0/1705258052 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2ba80a97e0 con 0x559c6745f320 2024-06-04T05:21:54.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.467+0000 7f2b9e7fc700 1 -- 172.21.15.70:0/1705258052 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba80b21b0 con 0x559c6745f320 2024-06-04T05:21:54.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.468+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/1705258052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745f320 msgr2=0x559c6745f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:54.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.468+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/1705258052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745f320 0x559c6745f700 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80a94e0 tx=0x7f2ba80a9130 comp rx=0 tx=0).stop 2024-06-04T05:21:54.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.468+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/1705258052 shutdown_connections 2024-06-04T05:21:54.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.468+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/1705258052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c6745a260 0x559c67466080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.468+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/1705258052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745f320 0x559c6745f700 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.468+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/1705258052 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c674598a0 0x559c67459d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.468+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/1705258052 >> 172.21.15.70:0/1705258052 conn(0x559c67385240 msgr2=0x559c674543e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:54.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.468+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/1705258052 shutdown_connections 2024-06-04T05:21:54.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.468+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/1705258052 wait complete. 2024-06-04T05:21:54.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.468+0000 7f2bc8ba1b80 1 -- start start 2024-06-04T05:21:54.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.469+0000 7f2bc8ba1b80 1 Processor -- start 2024-06-04T05:21:54.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.469+0000 7f2bc8ba1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c674598a0 0x559c67465d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.469+0000 7f2bc8ba1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745a260 0x559c67463010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.469+0000 7f2bc8ba1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c6745f320 0x559c67463550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.469+0000 7f2bc8ba1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559c672adfa0 con 0x559c6745a260 2024-06-04T05:21:54.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.469+0000 7f2bc8ba1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559c6730fda0 con 0x559c674598a0 2024-06-04T05:21:54.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.469+0000 7f2bc8ba1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559c6746aaf0 con 0x559c6745f320 2024-06-04T05:21:54.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.469+0000 7f2bc8ba1b80 10 client.0 _ll_register_callbacks cb 0x559c67342950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:54.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.469+0000 7f2bb77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745a260 0x559c67463010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.469+0000 7f2bb77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745a260 0x559c67463010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40756/0 (socket says 172.21.15.70:40756) 2024-06-04T05:21:54.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.469+0000 7f2bb77fe700 1 -- 172.21.15.70:0/3259660002 learned_addr learned my addr 172.21.15.70:0/3259660002 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:54.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.469+0000 7f2bb7fff700 1 --2- 172.21.15.70:0/3259660002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c674598a0 0x559c67465d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.469+0000 7f2bbc8a9700 1 --2- 172.21.15.70:0/3259660002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c6745f320 0x559c67463550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.470+0000 7f2bb77fe700 1 -- 172.21.15.70:0/3259660002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c6745f320 msgr2=0x559c67463550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:54.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.470+0000 7f2bb77fe700 1 --2- 172.21.15.70:0/3259660002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c6745f320 0x559c67463550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.470+0000 7f2bb77fe700 1 -- 172.21.15.70:0/3259660002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c674598a0 msgr2=0x559c67465d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:54.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.470+0000 7f2bb77fe700 1 --2- 172.21.15.70:0/3259660002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c674598a0 0x559c67465d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.470+0000 7f2bb77fe700 1 -- 172.21.15.70:0/3259660002 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c672f78d0 con 0x559c6745a260 2024-06-04T05:21:54.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.470+0000 7f2bb77fe700 1 --2- 172.21.15.70:0/3259660002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745a260 0x559c67463010 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f2bac005130 tx=0x7f2bac009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600038355cd50909 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:54.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.470+0000 7f2b977fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:21:54.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.470+0000 7f2b977fe700 1 -- 172.21.15.70:0/3259660002 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac005690 con 0x559c6745a260 2024-06-04T05:21:54.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.470+0000 7f2b977fe700 1 -- 172.21.15.70:0/3259660002 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f2bac00f460 con 0x559c6745a260 2024-06-04T05:21:54.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.470+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/3259660002 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ba80a97e0 con 0x559c6745a260 2024-06-04T05:21:54.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.470+0000 7f2b977fe700 1 -- 172.21.15.70:0/3259660002 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f2bac00f660 con 0x559c6745a260 2024-06-04T05:21:54.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.470+0000 7f2bc8ba1b80 10 client.21654 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:54.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.470+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/3259660002 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x559c67361760 con 0x559c6745a260 2024-06-04T05:21:54.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.470+0000 7f2b977fe700 1 -- 172.21.15.70:0/3259660002 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2ba80a97e0 con 0x559c6745a260 2024-06-04T05:21:54.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.471+0000 7f2b977fe700 1 -- 172.21.15.70:0/3259660002 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f2bac02e850 con 0x559c6745a260 2024-06-04T05:21:54.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.471+0000 7f2bc8ba1b80 10 client.21654 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:54.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.471+0000 7f2bc8ba1b80 10 client.21654 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:21:54.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.471+0000 7f2bc8ba1b80 20 client.21654 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:54.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.471+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/3259660002 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x559c674776b0 0x559c67479aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.471+0000 7f2bc8ba1b80 4 client.21654 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:21:54.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.471+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/3259660002 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c673b9cc0 con 0x559c674776b0 2024-06-04T05:21:54.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.471+0000 7f2bb7fff700 1 --2- 172.21.15.70:0/3259660002 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x559c674776b0 0x559c67479aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.471+0000 7f2b977fe700 10 client.21654 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:21:54.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.472+0000 7f2bb7fff700 1 --2- 172.21.15.70:0/3259660002 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x559c674776b0 0x559c67479aa0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=88b3b58379d71a47 server_cookie=568d9dd8fcb70b1b in_seq=0 out_seq=0 2024-06-04T05:21:54.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.473+0000 7f2b977fe700 1 -- 172.21.15.70:0/3259660002 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559c673b9cc0 con 0x559c674776b0 2024-06-04T05:21:54.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.473+0000 7f2b977fe700 10 client.21654 handle_command_reply: tid=0 2024-06-04T05:21:54.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.473+0000 7f2bc8ba1b80 1 client.21654 shutdown 2024-06-04T05:21:54.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.473+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/3259660002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745a260 msgr2=0x559c67463010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:54.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.473+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/3259660002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745a260 0x559c67463010 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f2bac005130 tx=0x7f2bac009cd0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.473+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/3259660002 shutdown_connections 2024-06-04T05:21:54.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.473+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/3259660002 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x559c674776b0 0x559c67479aa0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.473+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/3259660002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c6745f320 0x559c67463550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.473+0000 7f2b977fe700 0 client.21654 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:21:54.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.473+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/3259660002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c674598a0 0x559c67465d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.473+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/3259660002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745a260 0x559c67463010 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.473+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/3259660002 >> 172.21.15.70:0/3259660002 conn(0x559c67385240 msgr2=0x559c67453af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:54.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.473+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/3259660002 shutdown_connections 2024-06-04T05:21:54.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.473+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/3259660002 wait complete. 2024-06-04T05:21:54.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.474+0000 7f2bc8ba1b80 20 client.21654 trim_cache size 0 max 16384 2024-06-04T05:21:54.485 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:21:54.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.493+0000 7f2bc8ba1b80 1 Processor -- start 2024-06-04T05:21:54.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.493+0000 7f2bc8ba1b80 1 -- start start 2024-06-04T05:21:54.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.493+0000 7f2bc8ba1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c674598a0 0x559c6745e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.493+0000 7f2bc8ba1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745f320 0x559c67454270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.493+0000 7f2bc8ba1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c674776b0 0x559c67458890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.493+0000 7f2bc8ba1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba400d6a0 con 0x559c6745f320 2024-06-04T05:21:54.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.493+0000 7f2bc8ba1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba400f270 con 0x559c674776b0 2024-06-04T05:21:54.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.493+0000 7f2bc8ba1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba400ddd0 con 0x559c674598a0 2024-06-04T05:21:54.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.493+0000 7f2bb77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745f320 0x559c67454270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.493+0000 7f2bb77fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745f320 0x559c67454270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40758/0 (socket says 172.21.15.70:40758) 2024-06-04T05:21:54.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.493+0000 7f2bb77fe700 1 -- 172.21.15.70:0/2174606535 learned_addr learned my addr 172.21.15.70:0/2174606535 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:54.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.493+0000 7f2bbc8a9700 1 --2- 172.21.15.70:0/2174606535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c674776b0 0x559c67458890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.493+0000 7f2bb7fff700 1 --2- 172.21.15.70:0/2174606535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c674598a0 0x559c6745e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.493+0000 7f2bb77fe700 1 -- 172.21.15.70:0/2174606535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c674598a0 msgr2=0x559c6745e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:54.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.493+0000 7f2bb77fe700 1 --2- 172.21.15.70:0/2174606535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c674598a0 0x559c6745e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.494+0000 7f2bb77fe700 1 -- 172.21.15.70:0/2174606535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c674776b0 msgr2=0x559c67458890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:54.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.494+0000 7f2bb77fe700 1 --2- 172.21.15.70:0/2174606535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c674776b0 0x559c67458890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.494+0000 7f2bb77fe700 1 -- 172.21.15.70:0/2174606535 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c67463ec0 con 0x559c6745f320 2024-06-04T05:21:54.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.494+0000 7f2bbc8a9700 1 --2- 172.21.15.70:0/2174606535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c674776b0 0x559c67458890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:21:54.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.494+0000 7f2bb77fe700 1 --2- 172.21.15.70:0/2174606535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745f320 0x559c67454270 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f2bac005850 tx=0x7f2bac00b580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85d5ebbf27a503c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:54.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.494+0000 7f2b9dffb700 1 -- 172.21.15.70:0/2174606535 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac045070 con 0x559c6745f320 2024-06-04T05:21:54.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.494+0000 7f2b9dffb700 1 -- 172.21.15.70:0/2174606535 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2bac033070 con 0x559c6745f320 2024-06-04T05:21:54.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.494+0000 7f2b9dffb700 1 -- 172.21.15.70:0/2174606535 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac02e400 con 0x559c6745f320 2024-06-04T05:21:54.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.495+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/2174606535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745f320 msgr2=0x559c67454270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:54.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.495+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/2174606535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745f320 0x559c67454270 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f2bac005850 tx=0x7f2bac00b580 comp rx=0 tx=0).stop 2024-06-04T05:21:54.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.495+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/2174606535 shutdown_connections 2024-06-04T05:21:54.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.495+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/2174606535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c674776b0 0x559c67458890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.495+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/2174606535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c674598a0 0x559c6745e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.495+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/2174606535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c6745f320 0x559c67454270 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.495+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/2174606535 >> 172.21.15.70:0/2174606535 conn(0x559c67385240 msgr2=0x559c6745e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:54.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.495+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/2174606535 shutdown_connections 2024-06-04T05:21:54.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.495+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/2174606535 wait complete. 2024-06-04T05:21:54.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.495+0000 7f2bc8ba1b80 1 -- start start 2024-06-04T05:21:54.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.496+0000 7f2bc8ba1b80 1 Processor -- start 2024-06-04T05:21:54.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.496+0000 7f2bc8ba1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c674598a0 0x559c6747f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.496+0000 7f2bc8ba1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c6745f320 0x559c6747f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.496+0000 7f2bc8ba1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c674776b0 0x559c6747fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.496+0000 7f2bc8ba1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba400eb70 con 0x559c674598a0 2024-06-04T05:21:54.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.496+0000 7f2bc8ba1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba400e4a0 con 0x559c6745f320 2024-06-04T05:21:54.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.496+0000 7f2bc8ba1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba400b660 con 0x559c674776b0 2024-06-04T05:21:54.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.496+0000 7f2bc8ba1b80 10 client.0 _ll_register_callbacks cb 0x559c67342950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:54.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.496+0000 7f2bb7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c674598a0 0x559c6747f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.496+0000 7f2bb7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c674598a0 0x559c6747f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40768/0 (socket says 172.21.15.70:40768) 2024-06-04T05:21:54.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.496+0000 7f2bb7fff700 1 -- 172.21.15.70:0/4278792177 learned_addr learned my addr 172.21.15.70:0/4278792177 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:54.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2bb77fe700 1 --2- 172.21.15.70:0/4278792177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c6745f320 0x559c6747f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2bbc8a9700 1 --2- 172.21.15.70:0/4278792177 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c674776b0 0x559c6747fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2bb7fff700 1 -- 172.21.15.70:0/4278792177 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c674776b0 msgr2=0x559c6747fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:54.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2bb7fff700 1 --2- 172.21.15.70:0/4278792177 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c674776b0 0x559c6747fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2bb7fff700 1 -- 172.21.15.70:0/4278792177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c6745f320 msgr2=0x559c6747f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:54.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2bb7fff700 1 --2- 172.21.15.70:0/4278792177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c6745f320 0x559c6747f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2bb7fff700 1 -- 172.21.15.70:0/4278792177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c6746e550 con 0x559c674598a0 2024-06-04T05:21:54.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2bb77fe700 1 --2- 172.21.15.70:0/4278792177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c6745f320 0x559c6747f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:21:54.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2bbc8a9700 1 --2- 172.21.15.70:0/4278792177 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c674776b0 0x559c6747fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:21:54.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2bb7fff700 1 --2- 172.21.15.70:0/4278792177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c674598a0 0x559c6747f000 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80a9490 tx=0x7f2ba8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5dc0f042d4e2af server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:54.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2b977fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:21:54.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2b977fe700 1 -- 172.21.15.70:0/4278792177 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba80c6070 con 0x559c674598a0 2024-06-04T05:21:54.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2b977fe700 1 -- 172.21.15.70:0/4278792177 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f2ba80a9a40 con 0x559c674598a0 2024-06-04T05:21:54.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/4278792177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bac033070 con 0x559c674598a0 2024-06-04T05:21:54.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2b977fe700 1 -- 172.21.15.70:0/4278792177 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f2ba80a9ee0 con 0x559c674598a0 2024-06-04T05:21:54.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2bc8ba1b80 10 client.21660 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:54.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/4278792177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x559c674e2650 con 0x559c674598a0 2024-06-04T05:21:54.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.497+0000 7f2b977fe700 1 -- 172.21.15.70:0/4278792177 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2bac033070 con 0x559c674598a0 2024-06-04T05:21:54.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.498+0000 7f2b977fe700 1 -- 172.21.15.70:0/4278792177 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f2ba808a070 con 0x559c674598a0 2024-06-04T05:21:54.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.498+0000 7f2bc8ba1b80 10 client.21660 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:54.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.499+0000 7f2bc8ba1b80 10 client.21660 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:21:54.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.499+0000 7f2bc8ba1b80 20 client.21660 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:54.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.499+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/4278792177 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x559c674e2990 0x559c674e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:54.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.499+0000 7f2bc8ba1b80 4 client.21660 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:21:54.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.499+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/4278792177 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559c673b34a0 con 0x559c674e2990 2024-06-04T05:21:54.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.499+0000 7f2bb77fe700 1 --2- 172.21.15.70:0/4278792177 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x559c674e2990 0x559c674e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:54.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.499+0000 7f2b977fe700 10 client.21660 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:21:54.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.499+0000 7f2bb77fe700 1 --2- 172.21.15.70:0/4278792177 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x559c674e2990 0x559c674e4d80 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ed45f0a1e05e3a59 server_cookie=3d0d5c5efc221262 in_seq=0 out_seq=0 2024-06-04T05:21:54.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.499+0000 7f2b977fe700 1 -- 172.21.15.70:0/4278792177 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559c673b34a0 con 0x559c674e2990 2024-06-04T05:21:54.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.499+0000 7f2b977fe700 10 client.21660 handle_command_reply: tid=0 2024-06-04T05:21:54.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.499+0000 7f2bc8ba1b80 1 client.21660 shutdown 2024-06-04T05:21:54.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.500+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/4278792177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c674598a0 msgr2=0x559c6747f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:54.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.500+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/4278792177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c674598a0 0x559c6747f000 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80a9490 tx=0x7f2ba8004750 comp rx=0 tx=0).stop 2024-06-04T05:21:54.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.500+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/4278792177 shutdown_connections 2024-06-04T05:21:54.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.500+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/4278792177 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x559c674e2990 0x559c674e4d80 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.500+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/4278792177 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559c674776b0 0x559c6747fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.500+0000 7f2b977fe700 0 client.21660 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:21:54.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.500+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/4278792177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c674598a0 0x559c6747f000 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.500+0000 7f2bc8ba1b80 1 --2- 172.21.15.70:0/4278792177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c6745f320 0x559c6747f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.500+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/4278792177 >> 172.21.15.70:0/4278792177 conn(0x559c67385240 msgr2=0x559c6745cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:54.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.500+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/4278792177 shutdown_connections 2024-06-04T05:21:54.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.500+0000 7f2bc8ba1b80 1 -- 172.21.15.70:0/4278792177 wait complete. 2024-06-04T05:21:54.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.500+0000 7f2bc8ba1b80 20 client.21660 trim_cache size 0 max 16384 2024-06-04T05:21:54.506 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:21:54.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.501+0000 7f2bbe30c700 1 -- 172.21.15.70:0/642390280 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2ba00b2a70 msgr2=0x7f2ba00b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:54.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.501+0000 7f2bbe30c700 1 --2- 172.21.15.70:0/642390280 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2ba00b2a70 0x7f2ba00b4f30 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2bac000c00 tx=0x7f2bac005fb0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.501+0000 7f2bbe30c700 1 -- 172.21.15.70:0/642390280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bb8101b30 msgr2=0x7f2bb819f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:54.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.501+0000 7f2bbe30c700 1 --2- 172.21.15.70:0/642390280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bb8101b30 0x7f2bb819f790 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800c9b0 tx=0x7f2ba800ccc0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.501+0000 7f2bbe30c700 1 -- 172.21.15.70:0/642390280 shutdown_connections 2024-06-04T05:21:54.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.501+0000 7f2bbe30c700 1 --2- 172.21.15.70:0/642390280 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2ba00b2a70 0x7f2ba00b4f30 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.501+0000 7f2bbe30c700 1 --2- 172.21.15.70:0/642390280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bb8109520 0x7f2bb8199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.501+0000 7f2bbe30c700 1 --2- 172.21.15.70:0/642390280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bb8101b30 0x7f2bb819f790 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.501+0000 7f2bbe30c700 1 --2- 172.21.15.70:0/642390280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2bb81024f0 0x7f2bb819fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:54.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.501+0000 7f2bbe30c700 1 -- 172.21.15.70:0/642390280 >> 172.21.15.70:0/642390280 conn(0x7f2bb8078ab0 msgr2=0x7f2bb8100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:54.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.501+0000 7f2bbe30c700 1 -- 172.21.15.70:0/642390280 shutdown_connections 2024-06-04T05:21:54.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:54.501+0000 7f2bbe30c700 1 -- 172.21.15.70:0/642390280 wait complete. 2024-06-04T05:21:54.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.76 MB/sec execute 526 sec latency 17121.139 ms 2024-06-04T05:21:55.650 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:21:55.672 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:55 smithi070 ceph-mon[32631]: pgmap v779: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 14 op/s 2024-06-04T05:21:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:55 smithi112 ceph-mon[36453]: pgmap v779: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 14 op/s 2024-06-04T05:21:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:55 smithi098 ceph-mon[35569]: pgmap v779: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 14 op/s 2024-06-04T05:21:55.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.75 MB/sec execute 527 sec latency 18121.183 ms 2024-06-04T05:21:56.458 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:21:56.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82558 6.73 MB/sec execute 528 sec latency 19121.239 ms 2024-06-04T05:21:57.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.092+0000 7f625c1c9700 1 -- 172.21.15.70:0/975408050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254109520 msgr2=0x7f6254109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:57.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.092+0000 7f625c1c9700 1 --2- 172.21.15.70:0/975408050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254109520 0x7f6254109900 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f625000b0d0 tx=0x7f625000b3e0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.092+0000 7f625c1c9700 1 -- 172.21.15.70:0/975408050 shutdown_connections 2024-06-04T05:21:57.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.092+0000 7f625c1c9700 1 --2- 172.21.15.70:0/975408050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62541024f0 0x7f62541103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.092+0000 7f625c1c9700 1 --2- 172.21.15.70:0/975408050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254109520 0x7f6254109900 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.092+0000 7f625c1c9700 1 --2- 172.21.15.70:0/975408050 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6254101b30 0x7f6254101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.092+0000 7f625c1c9700 1 -- 172.21.15.70:0/975408050 >> 172.21.15.70:0/975408050 conn(0x7f6254078ab0 msgr2=0x7f62540ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:57.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.092+0000 7f625c1c9700 1 -- 172.21.15.70:0/975408050 shutdown_connections 2024-06-04T05:21:57.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.092+0000 7f625c1c9700 1 -- 172.21.15.70:0/975408050 wait complete. 2024-06-04T05:21:57.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.093+0000 7f625c1c9700 1 Processor -- start 2024-06-04T05:21:57.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.093+0000 7f625c1c9700 1 -- start start 2024-06-04T05:21:57.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.093+0000 7f625c1c9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254101b30 0x7f625419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.093+0000 7f625c1c9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62541024f0 0x7f625419fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.093+0000 7f625c1c9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6254109520 0x7f6254199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.093+0000 7f625c1c9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6254113610 con 0x7f6254101b30 2024-06-04T05:21:57.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.093+0000 7f625c1c9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6254113490 con 0x7f62541024f0 2024-06-04T05:21:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.093+0000 7f625c1c9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6254113790 con 0x7f6254109520 2024-06-04T05:21:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.093+0000 7f6259f65700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254101b30 0x7f625419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.093+0000 7f6259f65700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254101b30 0x7f625419f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54304/0 (socket says 172.21.15.70:54304) 2024-06-04T05:21:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.093+0000 7f6259f65700 1 -- 172.21.15.70:0/673031190 learned_addr learned my addr 172.21.15.70:0/673031190 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.093+0000 7f625a766700 1 --2- 172.21.15.70:0/673031190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6254109520 0x7f6254199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.094+0000 7f6259f65700 1 -- 172.21.15.70:0/673031190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6254109520 msgr2=0x7f6254199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.094+0000 7f6259f65700 1 --2- 172.21.15.70:0/673031190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6254109520 0x7f6254199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.094+0000 7f6259f65700 1 -- 172.21.15.70:0/673031190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62541024f0 msgr2=0x7f625419fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.094+0000 7f6259f65700 1 --2- 172.21.15.70:0/673031190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62541024f0 0x7f625419fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.094+0000 7f6259f65700 1 -- 172.21.15.70:0/673031190 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f625419a0d0 con 0x7f6254101b30 2024-06-04T05:21:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.094+0000 7f6259f65700 1 --2- 172.21.15.70:0/673031190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254101b30 0x7f625419f7b0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f6250007d90 tx=0x7f6250007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.094+0000 7f624affd700 1 -- 172.21.15.70:0/673031190 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625001f070 con 0x7f6254101b30 2024-06-04T05:21:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.094+0000 7f624affd700 1 -- 172.21.15.70:0/673031190 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6250005500 con 0x7f6254101b30 2024-06-04T05:21:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.094+0000 7f625c1c9700 1 -- 172.21.15.70:0/673031190 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f625419a360 con 0x7f6254101b30 2024-06-04T05:21:57.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.094+0000 7f624affd700 1 -- 172.21.15.70:0/673031190 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625001b8c0 con 0x7f6254101b30 2024-06-04T05:21:57.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.094+0000 7f625c1c9700 1 -- 172.21.15.70:0/673031190 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62541a6680 con 0x7f6254101b30 2024-06-04T05:21:57.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.096+0000 7f624affd700 1 -- 172.21.15.70:0/673031190 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f6250022070 con 0x7f6254101b30 2024-06-04T05:21:57.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.097+0000 7f624affd700 1 --2- 172.21.15.70:0/673031190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62400b29f0 0x7f62400b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.097+0000 7f6259764700 1 --2- 172.21.15.70:0/673031190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62400b29f0 0x7f62400b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.097+0000 7f624affd700 1 -- 172.21.15.70:0/673031190 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f625000f020 con 0x7f6254101b30 2024-06-04T05:21:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.097+0000 7f6259764700 1 --2- 172.21.15.70:0/673031190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62400b29f0 0x7f62400b4eb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6244005950 tx=0x7f62440058e0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f6266a5eb80 1 Processor -- start 2024-06-04T05:21:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f6266a5eb80 1 -- start start 2024-06-04T05:21:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f6266a5eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cbb2b0 0x55d513cbb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f6266a5eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb58e0 0x55d513cb5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f6266a5eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cb62a0 0x55d513cc20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f6266a5eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d513c14d90 con 0x55d513cb58e0 2024-06-04T05:21:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f6266a5eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d513c04930 con 0x55d513cbb2b0 2024-06-04T05:21:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f6266a5eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d513c08970 con 0x55d513cb62a0 2024-06-04T05:21:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f6259764700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb58e0 0x55d513cb5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f6259764700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb58e0 0x55d513cb5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54314/0 (socket says 172.21.15.70:54314) 2024-06-04T05:21:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f6259764700 1 -- 172.21.15.70:0/890056316 learned_addr learned my addr 172.21.15.70:0/890056316 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f6259f65700 1 --2- 172.21.15.70:0/890056316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cbb2b0 0x55d513cbb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f625a766700 1 --2- 172.21.15.70:0/890056316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cb62a0 0x55d513cc20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f6259764700 1 -- 172.21.15.70:0/890056316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cb62a0 msgr2=0x55d513cc20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f6259764700 1 --2- 172.21.15.70:0/890056316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cb62a0 0x55d513cc20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f6259764700 1 -- 172.21.15.70:0/890056316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cbb2b0 msgr2=0x55d513cbb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f6259764700 1 --2- 172.21.15.70:0/890056316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cbb2b0 0x55d513cbb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f6259764700 1 -- 172.21.15.70:0/890056316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d513b549d0 con 0x55d513cb58e0 2024-06-04T05:21:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.098+0000 7f625a766700 1 --2- 172.21.15.70:0/890056316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cb62a0 0x55d513cc20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:21:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.099+0000 7f6259764700 1 --2- 172.21.15.70:0/890056316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb58e0 0x55d513cb5d60 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f6244009270 tx=0x7f6244010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f2cff9669e86b9 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.099+0000 7f623ffff700 1 -- 172.21.15.70:0/890056316 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6244010af0 con 0x55d513cb58e0 2024-06-04T05:21:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.099+0000 7f623ffff700 1 -- 172.21.15.70:0/890056316 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6244022070 con 0x55d513cb58e0 2024-06-04T05:21:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.099+0000 7f623ffff700 1 -- 172.21.15.70:0/890056316 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624400f3c0 con 0x55d513cb58e0 2024-06-04T05:21:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.100+0000 7f6266a5eb80 1 -- 172.21.15.70:0/890056316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb58e0 msgr2=0x55d513cb5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.100+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/890056316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb58e0 0x55d513cb5d60 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f6244009270 tx=0x7f6244010000 comp rx=0 tx=0).stop 2024-06-04T05:21:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.100+0000 7f6266a5eb80 1 -- 172.21.15.70:0/890056316 shutdown_connections 2024-06-04T05:21:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.100+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/890056316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cb62a0 0x55d513cc20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.100+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/890056316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cbb2b0 0x55d513cbb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.100+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/890056316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb58e0 0x55d513cb5d60 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.100+0000 7f6266a5eb80 1 -- 172.21.15.70:0/890056316 >> 172.21.15.70:0/890056316 conn(0x55d513be3180 msgr2=0x55d513cb01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.100+0000 7f6266a5eb80 1 -- 172.21.15.70:0/890056316 shutdown_connections 2024-06-04T05:21:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.100+0000 7f6266a5eb80 1 -- 172.21.15.70:0/890056316 wait complete. 2024-06-04T05:21:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.100+0000 7f6266a5eb80 1 -- start start 2024-06-04T05:21:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.100+0000 7f6266a5eb80 1 Processor -- start 2024-06-04T05:21:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6266a5eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cb58e0 0x55d513cc1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6266a5eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb62a0 0x55d513cbf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6266a5eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cbb2b0 0x55d513cbf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6266a5eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d513b0b0a0 con 0x55d513cb62a0 2024-06-04T05:21:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6266a5eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d513ba68c0 con 0x55d513cb58e0 2024-06-04T05:21:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6266a5eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d513cc2680 con 0x55d513cbb2b0 2024-06-04T05:21:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6266a5eb80 10 client.0 _ll_register_callbacks cb 0x55d513b9fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6259764700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb62a0 0x55d513cbf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6259764700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb62a0 0x55d513cbf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54318/0 (socket says 172.21.15.70:54318) 2024-06-04T05:21:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6259764700 1 -- 172.21.15.70:0/1861548534 learned_addr learned my addr 172.21.15.70:0/1861548534 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f625a766700 1 --2- 172.21.15.70:0/1861548534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cbb2b0 0x55d513cbf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6259f65700 1 --2- 172.21.15.70:0/1861548534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cb58e0 0x55d513cc1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6259764700 1 -- 172.21.15.70:0/1861548534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cbb2b0 msgr2=0x55d513cbf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6259764700 1 --2- 172.21.15.70:0/1861548534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cbb2b0 0x55d513cbf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6259764700 1 -- 172.21.15.70:0/1861548534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cb58e0 msgr2=0x55d513cc1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6259764700 1 --2- 172.21.15.70:0/1861548534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cb58e0 0x55d513cc1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6259764700 1 -- 172.21.15.70:0/1861548534 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d513ba3790 con 0x55d513cb62a0 2024-06-04T05:21:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6259764700 1 --2- 172.21.15.70:0/1861548534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb62a0 0x55d513cbf0b0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f62440058b0 tx=0x7f624401db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb723ec8c52a20a1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f623d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:21:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f623d7fa700 1 -- 172.21.15.70:0/1861548534 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624401d440 con 0x55d513cb62a0 2024-06-04T05:21:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f623d7fa700 1 -- 172.21.15.70:0/1861548534 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f624401d5e0 con 0x55d513cb62a0 2024-06-04T05:21:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.101+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1861548534 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6244022070 con 0x55d513cb62a0 2024-06-04T05:21:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.102+0000 7f623d7fa700 1 -- 172.21.15.70:0/1861548534 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f624401d7e0 con 0x55d513cb62a0 2024-06-04T05:21:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.102+0000 7f6266a5eb80 10 client.21672 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.102+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1861548534 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55d513cccd40 con 0x55d513cb62a0 2024-06-04T05:21:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.102+0000 7f623d7fa700 1 -- 172.21.15.70:0/1861548534 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6244022070 con 0x55d513cb62a0 2024-06-04T05:21:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.102+0000 7f623d7fa700 1 -- 172.21.15.70:0/1861548534 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f6244038880 con 0x55d513cb62a0 2024-06-04T05:21:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.102+0000 7f6266a5eb80 10 client.21672 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.102+0000 7f6266a5eb80 10 client.21672 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:21:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.102+0000 7f6266a5eb80 20 client.21672 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.102+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/1861548534 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d513ccf320 0x55d513cd1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.102+0000 7f6266a5eb80 4 client.21672 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:21:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.102+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1861548534 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d513c81ae0 con 0x55d513ccf320 2024-06-04T05:21:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.102+0000 7f6259f65700 1 --2- 172.21.15.70:0/1861548534 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d513ccf320 0x55d513cd1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.103+0000 7f6259f65700 1 --2- 172.21.15.70:0/1861548534 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d513ccf320 0x55d513cd1710 crc :-1 s=READY pgs=511 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8d0d293c8226c24 server_cookie=8e618a7d92129e5e in_seq=0 out_seq=0 2024-06-04T05:21:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.103+0000 7f623d7fa700 10 client.21672 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.104+0000 7f623d7fa700 1 -- 172.21.15.70:0/1861548534 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d513c81ae0 con 0x55d513ccf320 2024-06-04T05:21:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.104+0000 7f623d7fa700 10 client.21672 handle_command_reply: tid=0 2024-06-04T05:21:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.104+0000 7f6266a5eb80 1 client.21672 shutdown 2024-06-04T05:21:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.104+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1861548534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb62a0 msgr2=0x55d513cbf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.104+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/1861548534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb62a0 0x55d513cbf0b0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f62440058b0 tx=0x7f624401db10 comp rx=0 tx=0).stop 2024-06-04T05:21:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.104+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1861548534 shutdown_connections 2024-06-04T05:21:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.104+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/1861548534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cbb2b0 0x55d513cbf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.104+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/1861548534 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d513ccf320 0x55d513cd1710 crc :-1 s=READY pgs=511 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.104+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/1861548534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cb58e0 0x55d513cc1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.104+0000 7f623d7fa700 0 client.21672 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.104+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/1861548534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb62a0 0x55d513cbf0b0 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.104+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1861548534 >> 172.21.15.70:0/1861548534 conn(0x55d513be3180 msgr2=0x55d513caaca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.104+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1861548534 shutdown_connections 2024-06-04T05:21:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.104+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1861548534 wait complete. 2024-06-04T05:21:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.104+0000 7f6266a5eb80 20 client.21672 trim_cache size 0 max 16384 2024-06-04T05:21:57.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.124+0000 7f6266a5eb80 1 Processor -- start 2024-06-04T05:21:57.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.124+0000 7f6266a5eb80 1 -- start start 2024-06-04T05:21:57.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.124+0000 7f6266a5eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb58e0 0x55d513cba930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.124+0000 7f6266a5eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cb62a0 0x55d513cb7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.124+0000 7f6266a5eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cbb2b0 0x55d513cb82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.124+0000 7f6266a5eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62541add50 con 0x55d513cb58e0 2024-06-04T05:21:57.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.124+0000 7f6266a5eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62541af920 con 0x55d513cb62a0 2024-06-04T05:21:57.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.124+0000 7f6266a5eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f62541ae480 con 0x55d513cbb2b0 2024-06-04T05:21:57.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.125+0000 7f6259f65700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb58e0 0x55d513cba930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:57.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.125+0000 7f6259f65700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb58e0 0x55d513cba930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54326/0 (socket says 172.21.15.70:54326) 2024-06-04T05:21:57.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.125+0000 7f6259f65700 1 -- 172.21.15.70:0/1176704658 learned_addr learned my addr 172.21.15.70:0/1176704658 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:57.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.125+0000 7f6259764700 1 --2- 172.21.15.70:0/1176704658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cb62a0 0x55d513cb7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:57.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.125+0000 7f6259f65700 1 -- 172.21.15.70:0/1176704658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cbb2b0 msgr2=0x55d513cb82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:57.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.125+0000 7f625a766700 1 --2- 172.21.15.70:0/1176704658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cbb2b0 0x55d513cb82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:57.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.125+0000 7f6259f65700 1 --2- 172.21.15.70:0/1176704658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cbb2b0 0x55d513cb82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.125+0000 7f6259f65700 1 -- 172.21.15.70:0/1176704658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cb62a0 msgr2=0x55d513cb7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:57.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.125+0000 7f6259f65700 1 --2- 172.21.15.70:0/1176704658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cb62a0 0x55d513cb7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.125+0000 7f6259f65700 1 -- 172.21.15.70:0/1176704658 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d513ca88a0 con 0x55d513cb58e0 2024-06-04T05:21:57.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.125+0000 7f6259764700 1 --2- 172.21.15.70:0/1176704658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cb62a0 0x55d513cb7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:21:57.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.125+0000 7f6259f65700 1 --2- 172.21.15.70:0/1176704658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb58e0 0x55d513cba930 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f6250009850 tx=0x7f62500cd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d16051ded023af server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:57.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.125+0000 7f623f7fe700 1 -- 172.21.15.70:0/1176704658 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62500059d0 con 0x55d513cb58e0 2024-06-04T05:21:57.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.125+0000 7f623f7fe700 1 -- 172.21.15.70:0/1176704658 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f62500ae040 con 0x55d513cb58e0 2024-06-04T05:21:57.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.125+0000 7f623f7fe700 1 -- 172.21.15.70:0/1176704658 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62500233b0 con 0x55d513cb58e0 2024-06-04T05:21:57.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.126+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1176704658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb58e0 msgr2=0x55d513cba930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:57.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.126+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/1176704658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb58e0 0x55d513cba930 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f6250009850 tx=0x7f62500cd040 comp rx=0 tx=0).stop 2024-06-04T05:21:57.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.126+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1176704658 shutdown_connections 2024-06-04T05:21:57.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.126+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/1176704658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cbb2b0 0x55d513cb82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.126+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/1176704658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cb58e0 0x55d513cba930 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.126+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/1176704658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cb62a0 0x55d513cb7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.126+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1176704658 >> 172.21.15.70:0/1176704658 conn(0x55d513be3180 msgr2=0x55d513caaca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:57.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.126+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1176704658 shutdown_connections 2024-06-04T05:21:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.126+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1176704658 wait complete. 2024-06-04T05:21:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.126+0000 7f6266a5eb80 1 -- start start 2024-06-04T05:21:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.126+0000 7f6266a5eb80 1 Processor -- start 2024-06-04T05:21:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f6266a5eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cb58e0 0x55d513cc7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f6266a5eb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cb62a0 0x55d513cc6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f6266a5eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cbb2b0 0x55d513cc6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f6266a5eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62541af220 con 0x55d513cbb2b0 2024-06-04T05:21:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f6266a5eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62541aeb50 con 0x55d513cb58e0 2024-06-04T05:21:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f6266a5eb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f62541abd10 con 0x55d513cb62a0 2024-06-04T05:21:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f6266a5eb80 10 client.0 _ll_register_callbacks cb 0x55d513b9fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f625a766700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cbb2b0 0x55d513cc6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f625a766700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cbb2b0 0x55d513cc6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54330/0 (socket says 172.21.15.70:54330) 2024-06-04T05:21:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f6259764700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cb62a0 0x55d513cc6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f625a766700 1 -- 172.21.15.70:0/1413069744 learned_addr learned my addr 172.21.15.70:0/1413069744 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f6259f65700 1 --2- 172.21.15.70:0/1413069744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cb58e0 0x55d513cc7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f625a766700 1 -- 172.21.15.70:0/1413069744 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cb62a0 msgr2=0x55d513cc6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f625a766700 1 --2- 172.21.15.70:0/1413069744 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cb62a0 0x55d513cc6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f625a766700 1 -- 172.21.15.70:0/1413069744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cb58e0 msgr2=0x55d513cc7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f625a766700 1 --2- 172.21.15.70:0/1413069744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cb58e0 0x55d513cc7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f625a766700 1 -- 172.21.15.70:0/1413069744 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d513cc74e0 con 0x55d513cbb2b0 2024-06-04T05:21:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f625a766700 1 --2- 172.21.15.70:0/1413069744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cbb2b0 0x55d513cc6e70 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f624c0142f0 tx=0x7f624c014600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8a87e93597d7fdd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.127+0000 7f623d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:21:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.128+0000 7f623d7fa700 1 -- 172.21.15.70:0/1413069744 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c01b030 con 0x55d513cbb2b0 2024-06-04T05:21:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.128+0000 7f623d7fa700 1 -- 172.21.15.70:0/1413069744 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f624c014e10 con 0x55d513cbb2b0 2024-06-04T05:21:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.128+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1413069744 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62500ae040 con 0x55d513cbb2b0 2024-06-04T05:21:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.128+0000 7f623d7fa700 1 -- 172.21.15.70:0/1413069744 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f624c00f040 con 0x55d513cbb2b0 2024-06-04T05:21:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.128+0000 7f6266a5eb80 10 client.21678 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.128+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1413069744 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55d513d3f020 con 0x55d513cbb2b0 2024-06-04T05:21:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.128+0000 7f623d7fa700 1 -- 172.21.15.70:0/1413069744 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f62500ae040 con 0x55d513cbb2b0 2024-06-04T05:21:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.128+0000 7f623d7fa700 1 -- 172.21.15.70:0/1413069744 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f624c015040 con 0x55d513cbb2b0 2024-06-04T05:21:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.128+0000 7f6266a5eb80 10 client.21678 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.128+0000 7f6266a5eb80 10 client.21678 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:21:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.128+0000 7f6266a5eb80 20 client.21678 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.128+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/1413069744 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d513d3f780 0x55d513d41ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.128+0000 7f6266a5eb80 4 client.21678 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c80d4a61", "release": true, "await": true}] 2024-06-04T05:21:57.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.128+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1413069744 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c80d4a61", "release": true, "await": true}) -- 0x55d513c16ce0 con 0x55d513d3f780 2024-06-04T05:21:57.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.128+0000 7f6259f65700 1 --2- 172.21.15.70:0/1413069744 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d513d3f780 0x55d513d41ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:57.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.129+0000 7f623d7fa700 10 client.21678 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:57.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.129+0000 7f6259f65700 1 --2- 172.21.15.70:0/1413069744 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d513d3f780 0x55d513d41ba0 crc :-1 s=READY pgs=512 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93df5e318bc6f208 server_cookie=47898af08fdc8aff in_seq=0 out_seq=0 2024-06-04T05:21:57.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.153+0000 7f623d7fa700 1 -- 172.21.15.70:0/1413069744 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55d513c16ce0 con 0x55d513d3f780 2024-06-04T05:21:57.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.153+0000 7f623d7fa700 10 client.21678 handle_command_reply: tid=0 2024-06-04T05:21:57.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.153+0000 7f6266a5eb80 1 client.21678 shutdown 2024-06-04T05:21:57.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.153+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1413069744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cbb2b0 msgr2=0x55d513cc6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.153+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/1413069744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cbb2b0 0x55d513cc6e70 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f624c0142f0 tx=0x7f624c014600 comp rx=0 tx=0).stop 2024-06-04T05:21:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.153+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1413069744 shutdown_connections 2024-06-04T05:21:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.153+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/1413069744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d513cbb2b0 0x55d513cc6e70 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.153+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/1413069744 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d513d3f780 0x55d513d41ba0 crc :-1 s=READY pgs=512 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.153+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/1413069744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d513cb58e0 0x55d513cc7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.153+0000 7f623d7fa700 0 client.21678 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:21:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.153+0000 7f6266a5eb80 1 --2- 172.21.15.70:0/1413069744 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d513cb62a0 0x55d513cc6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.153+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1413069744 >> 172.21.15.70:0/1413069744 conn(0x55d513be3180 msgr2=0x55d513cc08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.153+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1413069744 shutdown_connections 2024-06-04T05:21:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.154+0000 7f6266a5eb80 1 -- 172.21.15.70:0/1413069744 wait complete. 2024-06-04T05:21:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.154+0000 7f6266a5eb80 20 client.21678 trim_cache size 0 max 16384 2024-06-04T05:21:57.160 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:21:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 64, 2024-06-04T05:21:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:21:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 76, 2024-06-04T05:21:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:21:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "c80d4a61": { 2024-06-04T05:21:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 76, 2024-06-04T05:21:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:21:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:21:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:21:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:21:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:21:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:21:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 150.0, 2024-06-04T05:21:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:21:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:21:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:21:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:21:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:21:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:21:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:21:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:21:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:21:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:21:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:21:57.161 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:21:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.154+0000 7f625c1c9700 1 -- 172.21.15.70:0/673031190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62400b29f0 msgr2=0x7f62400b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.154+0000 7f625c1c9700 1 --2- 172.21.15.70:0/673031190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62400b29f0 0x7f62400b4eb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6244005950 tx=0x7f62440058e0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.154+0000 7f625c1c9700 1 -- 172.21.15.70:0/673031190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254101b30 msgr2=0x7f625419f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.154+0000 7f625c1c9700 1 --2- 172.21.15.70:0/673031190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254101b30 0x7f625419f7b0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f6250007d90 tx=0x7f6250007dc0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.155+0000 7f625c1c9700 1 -- 172.21.15.70:0/673031190 shutdown_connections 2024-06-04T05:21:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.155+0000 7f625c1c9700 1 --2- 172.21.15.70:0/673031190 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62400b29f0 0x7f62400b4eb0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.155+0000 7f625c1c9700 1 --2- 172.21.15.70:0/673031190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6254109520 0x7f6254199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.155+0000 7f625c1c9700 1 --2- 172.21.15.70:0/673031190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254101b30 0x7f625419f7b0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.155+0000 7f625c1c9700 1 --2- 172.21.15.70:0/673031190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62541024f0 0x7f625419fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:57.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.155+0000 7f625c1c9700 1 -- 172.21.15.70:0/673031190 >> 172.21.15.70:0/673031190 conn(0x7f6254078ab0 msgr2=0x7f6254100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:57.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.155+0000 7f625c1c9700 1 -- 172.21.15.70:0/673031190 shutdown_connections 2024-06-04T05:21:57.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:57.155+0000 7f625c1c9700 1 -- 172.21.15.70:0/673031190 wait complete. 2024-06-04T05:21:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:57 smithi112 ceph-mon[36453]: pgmap v780: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 62 KiB/s wr, 27 op/s 2024-06-04T05:21:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:57 smithi098 ceph-mon[35569]: pgmap v780: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 62 KiB/s wr, 27 op/s 2024-06-04T05:21:57.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82665 6.73 MB/sec execute 529 sec latency 19448.604 ms 2024-06-04T05:21:57.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:57 smithi070 ceph-mon[32631]: pgmap v780: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 62 KiB/s wr, 27 op/s 2024-06-04T05:21:58.192 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c80d4a61', total seconds elapsed: 39.0 2024-06-04T05:21:58.193 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-06-04T05:21:58.193 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:21:58.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 82835 6.74 MB/sec execute 530 sec latency 65.550 ms 2024-06-04T05:21:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:21:59 smithi112 ceph-mon[36453]: pgmap v781: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 38 KiB/s wr, 18 op/s 2024-06-04T05:21:59.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:21:59 smithi070 ceph-mon[32631]: pgmap v781: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 38 KiB/s wr, 18 op/s 2024-06-04T05:21:59.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.731+0000 7f1b05da4700 1 -- 172.21.15.70:0/1116074118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b001035c0 msgr2=0x7f1b00103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:59.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.731+0000 7f1b05da4700 1 --2- 172.21.15.70:0/1116074118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b001035c0 0x7f1b00103a40 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f1af4009990 tx=0x7f1af4009ca0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.731+0000 7f1b05da4700 1 -- 172.21.15.70:0/1116074118 shutdown_connections 2024-06-04T05:21:59.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.731+0000 7f1b05da4700 1 --2- 172.21.15.70:0/1116074118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b00103f80 0x7f1b00110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.731+0000 7f1b05da4700 1 --2- 172.21.15.70:0/1116074118 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b001095d0 0x7f1b001099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.731+0000 7f1b05da4700 1 --2- 172.21.15.70:0/1116074118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b001035c0 0x7f1b00103a40 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.731+0000 7f1b05da4700 1 -- 172.21.15.70:0/1116074118 >> 172.21.15.70:0/1116074118 conn(0x7f1b0007a610 msgr2=0x7f1b0007aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:59.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.732+0000 7f1b05da4700 1 -- 172.21.15.70:0/1116074118 shutdown_connections 2024-06-04T05:21:59.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.732+0000 7f1b05da4700 1 -- 172.21.15.70:0/1116074118 wait complete. 2024-06-04T05:21:59.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.732+0000 7f1b05da4700 1 Processor -- start 2024-06-04T05:21:59.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.733+0000 7f1b05da4700 1 -- start start 2024-06-04T05:21:59.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.733+0000 7f1b05da4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b001035c0 0x7f1b0019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.733+0000 7f1b05da4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b00103f80 0x7f1b0019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.733+0000 7f1b05da4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b001095d0 0x7f1b00199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.733+0000 7f1b05da4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1b00113630 con 0x7f1b001035c0 2024-06-04T05:21:59.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.733+0000 7f1b05da4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001134b0 con 0x7f1b001095d0 2024-06-04T05:21:59.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.733+0000 7f1b05da4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001137b0 con 0x7f1b00103f80 2024-06-04T05:21:59.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.733+0000 7f1aff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b001035c0 0x7f1b0019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.733+0000 7f1aff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b001035c0 0x7f1b0019f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54342/0 (socket says 172.21.15.70:54342) 2024-06-04T05:21:59.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.733+0000 7f1aff7fe700 1 -- 172.21.15.70:0/4228125392 learned_addr learned my addr 172.21.15.70:0/4228125392 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:59.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.733+0000 7f1afffff700 1 --2- 172.21.15.70:0/4228125392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b001095d0 0x7f1b00199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.733+0000 7f1afeffd700 1 --2- 172.21.15.70:0/4228125392 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b00103f80 0x7f1b0019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.734+0000 7f1aff7fe700 1 -- 172.21.15.70:0/4228125392 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b00103f80 msgr2=0x7f1b0019fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:59.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.734+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/4228125392 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b00103f80 0x7f1b0019fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.734+0000 7f1aff7fe700 1 -- 172.21.15.70:0/4228125392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b001095d0 msgr2=0x7f1b00199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:59.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.734+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/4228125392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b001095d0 0x7f1b00199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.734+0000 7f1aff7fe700 1 -- 172.21.15.70:0/4228125392 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b0019a070 con 0x7f1b001035c0 2024-06-04T05:21:59.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.734+0000 7f1afeffd700 1 --2- 172.21.15.70:0/4228125392 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b00103f80 0x7f1b0019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:21:59.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.734+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/4228125392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b001035c0 0x7f1b0019f750 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f1af000e9c0 tx=0x7f1af000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:59.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.734+0000 7f1afcff9700 1 -- 172.21.15.70:0/4228125392 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af000cbd0 con 0x7f1b001035c0 2024-06-04T05:21:59.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.735+0000 7f1afcff9700 1 -- 172.21.15.70:0/4228125392 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1af000cd70 con 0x7f1b001035c0 2024-06-04T05:21:59.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.735+0000 7f1b05da4700 1 -- 172.21.15.70:0/4228125392 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b0019a360 con 0x7f1b001035c0 2024-06-04T05:21:59.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.735+0000 7f1afcff9700 1 -- 172.21.15.70:0/4228125392 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0005540 con 0x7f1b001035c0 2024-06-04T05:21:59.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.735+0000 7f1b05da4700 1 -- 172.21.15.70:0/4228125392 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b00075770 con 0x7f1b001035c0 2024-06-04T05:21:59.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.737+0000 7f1afcff9700 1 -- 172.21.15.70:0/4228125392 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f1af0004750 con 0x7f1b001035c0 2024-06-04T05:21:59.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.738+0000 7f1afcff9700 1 --2- 172.21.15.70:0/4228125392 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1ae80b2a70 0x7f1ae80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.739+0000 7f1afcff9700 1 -- 172.21.15.70:0/4228125392 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f1af00e2900 con 0x7f1b001035c0 2024-06-04T05:21:59.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.739+0000 7f1afeffd700 1 --2- 172.21.15.70:0/4228125392 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1ae80b2a70 0x7f1ae80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.739+0000 7f1b10639b80 1 Processor -- start 2024-06-04T05:21:59.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.739+0000 7f1b10639b80 1 -- start start 2024-06-04T05:21:59.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.739+0000 7f1b10639b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431f070 0x55649431f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.739+0000 7f1b10639b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55649431fa20 0x556494323150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.739+0000 7f1b10639b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556494323770 0x556494323bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.740+0000 7f1b10639b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556494275c90 con 0x556494323770 2024-06-04T05:21:59.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.740+0000 7f1b10639b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556494265830 con 0x55649431f070 2024-06-04T05:21:59.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.740+0000 7f1b10639b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556494269870 con 0x55649431fa20 2024-06-04T05:21:59.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.740+0000 7f1afeffd700 1 --2- 172.21.15.70:0/4228125392 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1ae80b2a70 0x7f1ae80b4f30 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f1af400b2e0 tx=0x7f1af400b610 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:59.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.740+0000 7f1aff7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431f070 0x55649431f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.740+0000 7f1aff7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431f070 0x55649431f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55054/0 (socket says 172.21.15.70:55054) 2024-06-04T05:21:59.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.740+0000 7f1aff7fe700 1 -- 172.21.15.70:0/3965804305 learned_addr learned my addr 172.21.15.70:0/3965804305 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:59.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.740+0000 7f1afffff700 1 --2- 172.21.15.70:0/3965804305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556494323770 0x556494323bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.740+0000 7f1afeffd700 1 --2- 172.21.15.70:0/3965804305 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55649431fa20 0x556494323150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.740+0000 7f1aff7fe700 1 -- 172.21.15.70:0/3965804305 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55649431fa20 msgr2=0x556494323150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:59.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.740+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/3965804305 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55649431fa20 0x556494323150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.740+0000 7f1aff7fe700 1 -- 172.21.15.70:0/3965804305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556494323770 msgr2=0x556494323bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:59.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.740+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/3965804305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556494323770 0x556494323bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.740+0000 7f1aff7fe700 1 -- 172.21.15.70:0/3965804305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564941b58d0 con 0x55649431f070 2024-06-04T05:21:59.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.740+0000 7f1afffff700 1 --2- 172.21.15.70:0/3965804305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556494323770 0x556494323bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:21:59.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.742+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/3965804305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431f070 0x55649431f450 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1af0005770 tx=0x7f1af00a92c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f72d7983b856b6f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:59.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.742+0000 7f1ae5ffb700 1 -- 172.21.15.70:0/3965804305 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af00c3070 con 0x55649431f070 2024-06-04T05:21:59.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.742+0000 7f1ae5ffb700 1 -- 172.21.15.70:0/3965804305 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1af00b2300 con 0x55649431f070 2024-06-04T05:21:59.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.742+0000 7f1ae5ffb700 1 -- 172.21.15.70:0/3965804305 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af00a9db0 con 0x55649431f070 2024-06-04T05:21:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:21:59 smithi098 ceph-mon[35569]: pgmap v781: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 38 KiB/s wr, 18 op/s 2024-06-04T05:21:59.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.743+0000 7f1b10639b80 1 -- 172.21.15.70:0/3965804305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431f070 msgr2=0x55649431f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:59.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.743+0000 7f1b10639b80 1 --2- 172.21.15.70:0/3965804305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431f070 0x55649431f450 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1af0005770 tx=0x7f1af00a92c0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.744+0000 7f1b10639b80 1 -- 172.21.15.70:0/3965804305 shutdown_connections 2024-06-04T05:21:59.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.744+0000 7f1b10639b80 1 --2- 172.21.15.70:0/3965804305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556494323770 0x556494323bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.744+0000 7f1b10639b80 1 --2- 172.21.15.70:0/3965804305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431f070 0x55649431f450 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.744+0000 7f1b10639b80 1 --2- 172.21.15.70:0/3965804305 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55649431fa20 0x556494323150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.744+0000 7f1b10639b80 1 -- 172.21.15.70:0/3965804305 >> 172.21.15.70:0/3965804305 conn(0x556494244080 msgr2=0x556494310e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:59.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.744+0000 7f1b10639b80 1 -- 172.21.15.70:0/3965804305 shutdown_connections 2024-06-04T05:21:59.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.744+0000 7f1b10639b80 1 -- 172.21.15.70:0/3965804305 wait complete. 2024-06-04T05:21:59.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.744+0000 7f1b10639b80 1 -- start start 2024-06-04T05:21:59.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.744+0000 7f1b10639b80 1 Processor -- start 2024-06-04T05:21:59.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.745+0000 7f1b10639b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431f070 0x55649432ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.745+0000 7f1b10639b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55649431fa20 0x556494327fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.745+0000 7f1aff7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431f070 0x55649432ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.745+0000 7f1b10639b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556494323770 0x556494328520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.745+0000 7f1aff7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431f070 0x55649432ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55062/0 (socket says 172.21.15.70:55062) 2024-06-04T05:21:59.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.745+0000 7f1afeffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55649431fa20 0x556494327fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.745+0000 7f1afeffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55649431fa20 0x556494327fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54370/0 (socket says 172.21.15.70:54370) 2024-06-04T05:21:59.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.745+0000 7f1aff7fe700 1 -- 172.21.15.70:0/1282331735 learned_addr learned my addr 172.21.15.70:0/1282331735 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:59.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.745+0000 7f1b10639b80 1 -- 172.21.15.70:0/1282331735 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55649416bfa0 con 0x55649431fa20 2024-06-04T05:21:59.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.745+0000 7f1b10639b80 1 -- 172.21.15.70:0/1282331735 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5564942077c0 con 0x55649431f070 2024-06-04T05:21:59.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.745+0000 7f1b10639b80 1 -- 172.21.15.70:0/1282331735 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55649432b930 con 0x556494323770 2024-06-04T05:21:59.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.745+0000 7f1b10639b80 10 client.0 _ll_register_callbacks cb 0x556494200920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:59.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.745+0000 7f1afffff700 1 --2- 172.21.15.70:0/1282331735 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556494323770 0x556494328520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.746+0000 7f1aff7fe700 1 -- 172.21.15.70:0/1282331735 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556494323770 msgr2=0x556494328520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:59.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.746+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/1282331735 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556494323770 0x556494328520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.746+0000 7f1aff7fe700 1 -- 172.21.15.70:0/1282331735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55649431fa20 msgr2=0x556494327fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:59.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.746+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/1282331735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55649431fa20 0x556494327fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.746+0000 7f1aff7fe700 1 -- 172.21.15.70:0/1282331735 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556494204690 con 0x55649431f070 2024-06-04T05:21:59.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.746+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/1282331735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431f070 0x55649432ace0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f1af00a6dc0 tx=0x7f1af00a8fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e6d93521e1090af server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:59.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.746+0000 7f1adeffd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:21:59.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.746+0000 7f1adeffd700 1 -- 172.21.15.70:0/1282331735 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af00c3040 con 0x55649431f070 2024-06-04T05:21:59.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.746+0000 7f1adeffd700 1 -- 172.21.15.70:0/1282331735 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f1af00a6140 con 0x55649431f070 2024-06-04T05:21:59.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.746+0000 7f1b10639b80 1 -- 172.21.15.70:0/1282331735 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1af00b2300 con 0x55649431f070 2024-06-04T05:21:59.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.746+0000 7f1adeffd700 1 -- 172.21.15.70:0/1282331735 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f1af008f050 con 0x55649431f070 2024-06-04T05:21:59.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.746+0000 7f1b10639b80 10 client.30296 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:59.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.746+0000 7f1b10639b80 1 -- 172.21.15.70:0/1282331735 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x556494335f50 con 0x55649431f070 2024-06-04T05:21:59.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.747+0000 7f1adeffd700 1 -- 172.21.15.70:0/1282331735 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1af00a9b50 con 0x55649431f070 2024-06-04T05:21:59.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.747+0000 7f1adeffd700 1 -- 172.21.15.70:0/1282331735 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f1af00a54c0 con 0x55649431f070 2024-06-04T05:21:59.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.747+0000 7f1b10639b80 10 client.30296 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:59.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.747+0000 7f1b10639b80 10 client.30296 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:21:59.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.747+0000 7f1b10639b80 20 client.30296 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:59.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.747+0000 7f1b10639b80 1 --2- 172.21.15.70:0/1282331735 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x556494338420 0x55649433a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.747+0000 7f1b10639b80 4 client.30296 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:21:59.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.747+0000 7f1b10639b80 1 -- 172.21.15.70:0/1282331735 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1af00b2300 con 0x556494338420 2024-06-04T05:21:59.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.748+0000 7f1afeffd700 1 --2- 172.21.15.70:0/1282331735 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x556494338420 0x55649433a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.748+0000 7f1afeffd700 1 --2- 172.21.15.70:0/1282331735 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x556494338420 0x55649433a810 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7e6aa0621ae68de4 server_cookie=959f4c7b4431b338 in_seq=0 out_seq=0 2024-06-04T05:21:59.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.748+0000 7f1adeffd700 10 client.30296 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:21:59.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.749+0000 7f1adeffd700 1 -- 172.21.15.70:0/1282331735 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f1af00b2300 con 0x556494338420 2024-06-04T05:21:59.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.749+0000 7f1adeffd700 10 client.30296 handle_command_reply: tid=0 2024-06-04T05:21:59.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.749+0000 7f1b10639b80 1 client.30296 shutdown 2024-06-04T05:21:59.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.750+0000 7f1b10639b80 1 -- 172.21.15.70:0/1282331735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431f070 msgr2=0x55649432ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:59.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.750+0000 7f1b10639b80 1 --2- 172.21.15.70:0/1282331735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431f070 0x55649432ace0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f1af00a6dc0 tx=0x7f1af00a8fe0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.750+0000 7f1b10639b80 1 -- 172.21.15.70:0/1282331735 shutdown_connections 2024-06-04T05:21:59.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.750+0000 7f1b10639b80 1 --2- 172.21.15.70:0/1282331735 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556494323770 0x556494328520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.750+0000 7f1b10639b80 1 --2- 172.21.15.70:0/1282331735 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x556494338420 0x55649433a810 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.750+0000 7f1b10639b80 1 --2- 172.21.15.70:0/1282331735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431f070 0x55649432ace0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.750+0000 7f1b10639b80 1 --2- 172.21.15.70:0/1282331735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55649431fa20 0x556494327fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.750+0000 7f1b10639b80 1 -- 172.21.15.70:0/1282331735 >> 172.21.15.70:0/1282331735 conn(0x556494244080 msgr2=0x556494310510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:59.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.750+0000 7f1adeffd700 0 client.30296 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:21:59.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.750+0000 7f1b10639b80 1 -- 172.21.15.70:0/1282331735 shutdown_connections 2024-06-04T05:21:59.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.750+0000 7f1b10639b80 1 -- 172.21.15.70:0/1282331735 wait complete. 2024-06-04T05:21:59.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.751+0000 7f1b10639b80 20 client.30296 trim_cache size 0 max 16384 2024-06-04T05:21:59.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.776+0000 7f1b10639b80 1 Processor -- start 2024-06-04T05:21:59.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.776+0000 7f1b10639b80 1 -- start start 2024-06-04T05:21:59.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.777+0000 7f1b10639b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55649431f070 0x55649432a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.777+0000 7f1b10639b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431fa20 0x556494327fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.777+0000 7f1b10639b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556494323770 0x5564943284e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.777+0000 7f1afeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431fa20 0x556494327fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.777+0000 7f1afffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556494323770 0x5564943284e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.777+0000 7f1afffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556494323770 0x5564943284e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54372/0 (socket says 172.21.15.70:54372) 2024-06-04T05:21:59.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.777+0000 7f1afffff700 1 -- 172.21.15.70:0/2163727987 learned_addr learned my addr 172.21.15.70:0/2163727987 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:59.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.777+0000 7f1b10639b80 1 -- 172.21.15.70:0/2163727987 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001abc80 con 0x556494323770 2024-06-04T05:21:59.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.777+0000 7f1b10639b80 1 -- 172.21.15.70:0/2163727987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001ad850 con 0x55649431fa20 2024-06-04T05:21:59.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.777+0000 7f1b10639b80 1 -- 172.21.15.70:0/2163727987 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001ac3b0 con 0x55649431f070 2024-06-04T05:21:59.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.778+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/2163727987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55649431f070 0x55649432a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.778+0000 7f1afeffd700 1 -- 172.21.15.70:0/2163727987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55649431f070 msgr2=0x55649432a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.778+0000 7f1afeffd700 1 --2- 172.21.15.70:0/2163727987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55649431f070 0x55649432a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.778+0000 7f1afeffd700 1 -- 172.21.15.70:0/2163727987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556494323770 msgr2=0x5564943284e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.778+0000 7f1afeffd700 1 --2- 172.21.15.70:0/2163727987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556494323770 0x5564943284e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.778+0000 7f1afeffd700 1 -- 172.21.15.70:0/2163727987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556494311d40 con 0x55649431fa20 2024-06-04T05:21:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.779+0000 7f1afeffd700 1 --2- 172.21.15.70:0/2163727987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431fa20 0x556494327fa0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1af4006010 tx=0x7f1af4020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b998998abcc8632 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.779+0000 7f1ae57fa700 1 -- 172.21.15.70:0/2163727987 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af400f840 con 0x55649431fa20 2024-06-04T05:21:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.779+0000 7f1ae57fa700 1 -- 172.21.15.70:0/2163727987 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1af4037070 con 0x55649431fa20 2024-06-04T05:21:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.779+0000 7f1ae57fa700 1 -- 172.21.15.70:0/2163727987 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af4030b80 con 0x55649431fa20 2024-06-04T05:21:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.780+0000 7f1b10639b80 1 -- 172.21.15.70:0/2163727987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431fa20 msgr2=0x556494327fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.780+0000 7f1b10639b80 1 --2- 172.21.15.70:0/2163727987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431fa20 0x556494327fa0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1af4006010 tx=0x7f1af4020000 comp rx=0 tx=0).stop 2024-06-04T05:21:59.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.781+0000 7f1b10639b80 1 -- 172.21.15.70:0/2163727987 shutdown_connections 2024-06-04T05:21:59.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.781+0000 7f1b10639b80 1 --2- 172.21.15.70:0/2163727987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556494323770 0x5564943284e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.781+0000 7f1b10639b80 1 --2- 172.21.15.70:0/2163727987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55649431f070 0x55649432a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.781+0000 7f1b10639b80 1 --2- 172.21.15.70:0/2163727987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55649431fa20 0x556494327fa0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.781+0000 7f1b10639b80 1 -- 172.21.15.70:0/2163727987 >> 172.21.15.70:0/2163727987 conn(0x556494244080 msgr2=0x556494327890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:59.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.781+0000 7f1b10639b80 1 -- 172.21.15.70:0/2163727987 shutdown_connections 2024-06-04T05:21:59.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.781+0000 7f1b10639b80 1 -- 172.21.15.70:0/2163727987 wait complete. 2024-06-04T05:21:59.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.781+0000 7f1b10639b80 1 -- start start 2024-06-04T05:21:59.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.781+0000 7f1b10639b80 1 Processor -- start 2024-06-04T05:21:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.782+0000 7f1b10639b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55649431f070 0x556494396bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.782+0000 7f1b10639b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55649431fa20 0x556494397130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.782+0000 7f1b10639b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556494323770 0x5564943a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.782+0000 7f1b10639b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001ad150 con 0x55649431f070 2024-06-04T05:21:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.782+0000 7f1b10639b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001aca80 con 0x556494323770 2024-06-04T05:21:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.782+0000 7f1b10639b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001a9c40 con 0x55649431fa20 2024-06-04T05:21:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.782+0000 7f1aff7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55649431f070 0x556494396bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.782+0000 7f1b10639b80 10 client.0 _ll_register_callbacks cb 0x556494200920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:21:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.782+0000 7f1afffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556494323770 0x5564943a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.782+0000 7f1afeffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55649431fa20 0x556494397130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.782+0000 7f1afffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556494323770 0x5564943a1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55092/0 (socket says 172.21.15.70:55092) 2024-06-04T05:21:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.782+0000 7f1afffff700 1 -- 172.21.15.70:0/2584480353 learned_addr learned my addr 172.21.15.70:0/2584480353 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:21:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.782+0000 7f1afffff700 1 -- 172.21.15.70:0/2584480353 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55649431fa20 msgr2=0x556494397130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:59.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.783+0000 7f1afffff700 1 --2- 172.21.15.70:0/2584480353 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55649431fa20 0x556494397130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.783+0000 7f1afffff700 1 -- 172.21.15.70:0/2584480353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55649431f070 msgr2=0x556494396bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:21:59.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.783+0000 7f1afffff700 1 --2- 172.21.15.70:0/2584480353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55649431f070 0x556494396bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.783+0000 7f1afffff700 1 -- 172.21.15.70:0/2584480353 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556494397a00 con 0x556494323770 2024-06-04T05:21:59.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.783+0000 7f1afffff700 1 --2- 172.21.15.70:0/2584480353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556494323770 0x5564943a1c70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1af8004660 tx=0x7f1af8004970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3e8318412784602 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:21:59.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.783+0000 7f1adeffd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:21:59.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.783+0000 7f1adeffd700 1 -- 172.21.15.70:0/2584480353 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af8007d80 con 0x556494323770 2024-06-04T05:21:59.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.783+0000 7f1b10639b80 1 -- 172.21.15.70:0/2584480353 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1af4037070 con 0x556494323770 2024-06-04T05:21:59.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.783+0000 7f1adeffd700 1 -- 172.21.15.70:0/2584480353 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f1af8017410 con 0x556494323770 2024-06-04T05:21:59.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.784+0000 7f1adeffd700 1 -- 172.21.15.70:0/2584480353 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f1af8003e50 con 0x556494323770 2024-06-04T05:21:59.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.784+0000 7f1b10639b80 10 client.30302 fetch_fsmap learned FSMap version 64 2024-06-04T05:21:59.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.784+0000 7f1b10639b80 1 -- 172.21.15.70:0/2584480353 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x5564943a3360 con 0x556494323770 2024-06-04T05:21:59.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.784+0000 7f1adeffd700 1 -- 172.21.15.70:0/2584480353 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1af4037070 con 0x556494323770 2024-06-04T05:21:59.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.784+0000 7f1adeffd700 1 -- 172.21.15.70:0/2584480353 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f1af801f070 con 0x556494323770 2024-06-04T05:21:59.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.784+0000 7f1b10639b80 10 client.30302 fetch_fsmap finished waiting for FSMap version 64 2024-06-04T05:21:59.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.784+0000 7f1b10639b80 10 client.30302 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:21:59.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.784+0000 7f1b10639b80 20 client.30302 populate_metadata read hostname 'smithi070' 2024-06-04T05:21:59.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.784+0000 7f1b10639b80 1 --2- 172.21.15.70:0/2584480353 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5564943a3aa0 0x5564943a5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:21:59.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.784+0000 7f1b10639b80 4 client.30302 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:21:59.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.784+0000 7f1b10639b80 1 -- 172.21.15.70:0/2584480353 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5564942e29e0 con 0x5564943a3aa0 2024-06-04T05:21:59.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.785+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/2584480353 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5564943a3aa0 0x5564943a5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:21:59.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.786+0000 7f1aff7fe700 1 --2- 172.21.15.70:0/2584480353 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5564943a3aa0 0x5564943a5ec0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=19ba3f3b3e3215be server_cookie=db4458cb5b353eba in_seq=0 out_seq=0 2024-06-04T05:21:59.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.786+0000 7f1adeffd700 10 client.30302 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:21:59.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.786+0000 7f1adeffd700 1 -- 172.21.15.70:0/2584480353 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5564942e29e0 con 0x5564943a3aa0 2024-06-04T05:21:59.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.786+0000 7f1adeffd700 10 client.30302 handle_command_reply: tid=0 2024-06-04T05:21:59.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.786+0000 7f1b10639b80 1 client.30302 shutdown 2024-06-04T05:21:59.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.786+0000 7f1b10639b80 1 -- 172.21.15.70:0/2584480353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556494323770 msgr2=0x5564943a1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:59.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.786+0000 7f1b10639b80 1 --2- 172.21.15.70:0/2584480353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556494323770 0x5564943a1c70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1af8004660 tx=0x7f1af8004970 comp rx=0 tx=0).stop 2024-06-04T05:21:59.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.786+0000 7f1b10639b80 1 -- 172.21.15.70:0/2584480353 shutdown_connections 2024-06-04T05:21:59.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.787+0000 7f1b10639b80 1 --2- 172.21.15.70:0/2584480353 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x5564943a3aa0 0x5564943a5ec0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.787+0000 7f1b10639b80 1 --2- 172.21.15.70:0/2584480353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556494323770 0x5564943a1c70 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.787+0000 7f1adeffd700 0 client.30302 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:21:59.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.787+0000 7f1b10639b80 1 --2- 172.21.15.70:0/2584480353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55649431f070 0x556494396bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.787+0000 7f1b10639b80 1 --2- 172.21.15.70:0/2584480353 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55649431fa20 0x556494397130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.787+0000 7f1b10639b80 1 -- 172.21.15.70:0/2584480353 >> 172.21.15.70:0/2584480353 conn(0x556494244080 msgr2=0x5564943280a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:59.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.787+0000 7f1b10639b80 1 -- 172.21.15.70:0/2584480353 shutdown_connections 2024-06-04T05:21:59.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.787+0000 7f1b10639b80 1 -- 172.21.15.70:0/2584480353 wait complete. 2024-06-04T05:21:59.791 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:21:59.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.787+0000 7f1b10639b80 20 client.30302 trim_cache size 0 max 16384 2024-06-04T05:21:59.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.788+0000 7f1b05da4700 1 -- 172.21.15.70:0/4228125392 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1ae80b2a70 msgr2=0x7f1ae80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:59.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.788+0000 7f1b05da4700 1 --2- 172.21.15.70:0/4228125392 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1ae80b2a70 0x7f1ae80b4f30 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f1af400b2e0 tx=0x7f1af400b610 comp rx=0 tx=0).stop 2024-06-04T05:21:59.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.788+0000 7f1b05da4700 1 -- 172.21.15.70:0/4228125392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b001035c0 msgr2=0x7f1b0019f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:21:59.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.788+0000 7f1b05da4700 1 --2- 172.21.15.70:0/4228125392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b001035c0 0x7f1b0019f750 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f1af000e9c0 tx=0x7f1af000ecd0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.788+0000 7f1b05da4700 1 -- 172.21.15.70:0/4228125392 shutdown_connections 2024-06-04T05:21:59.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.788+0000 7f1b05da4700 1 --2- 172.21.15.70:0/4228125392 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f1ae80b2a70 0x7f1ae80b4f30 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.788+0000 7f1b05da4700 1 --2- 172.21.15.70:0/4228125392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b001095d0 0x7f1b00199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.788+0000 7f1b05da4700 1 --2- 172.21.15.70:0/4228125392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b001035c0 0x7f1b0019f750 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.788+0000 7f1b05da4700 1 --2- 172.21.15.70:0/4228125392 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b00103f80 0x7f1b0019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:21:59.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.788+0000 7f1b05da4700 1 -- 172.21.15.70:0/4228125392 >> 172.21.15.70:0/4228125392 conn(0x7f1b0007a610 msgr2=0x7f1b001024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:21:59.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.789+0000 7f1b05da4700 1 -- 172.21.15.70:0/4228125392 shutdown_connections 2024-06-04T05:21:59.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:21:59.789+0000 7f1b05da4700 1 -- 172.21.15.70:0/4228125392 wait complete. 2024-06-04T05:21:59.802 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:21:59.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 83032 6.75 MB/sec execute 531 sec latency 92.782 ms 2024-06-04T05:22:00.522 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:22:00.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 83170 6.75 MB/sec execute 532 sec latency 408.028 ms 2024-06-04T05:22:01.319 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:22:01.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:01 smithi112 ceph-mon[36453]: pgmap v782: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 38 KiB/s wr, 18 op/s 2024-06-04T05:22:01.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:01 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:22:01.688 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:01 smithi070 ceph-mon[32631]: pgmap v782: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 38 KiB/s wr, 18 op/s 2024-06-04T05:22:01.688 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:01 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:22:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:01 smithi098 ceph-mon[35569]: pgmap v782: 97 pgs: 97 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 38 KiB/s wr, 18 op/s 2024-06-04T05:22:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:01 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:22:01.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 83319 6.76 MB/sec execute 533 sec latency 1408.055 ms 2024-06-04T05:22:02.835 INFO:tasks.workunit.client.0.smithi070.stdout: 10 83487 6.76 MB/sec execute 534 sec latency 2408.089 ms 2024-06-04T05:22:03.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.970+0000 7f93cc42f700 1 -- 172.21.15.70:0/1983739509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c4101ee0 msgr2=0x7f93c4102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.970+0000 7f93cc42f700 1 --2- 172.21.15.70:0/1983739509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c4101ee0 0x7f93c4102360 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f93b40099c0 tx=0x7f93b4009cd0 comp rx=0 tx=0).stop 2024-06-04T05:22:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.970+0000 7f93cc42f700 1 -- 172.21.15.70:0/1983739509 shutdown_connections 2024-06-04T05:22:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.970+0000 7f93cc42f700 1 --2- 172.21.15.70:0/1983739509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93c41028a0 0x7f93c4110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.970+0000 7f93cc42f700 1 --2- 172.21.15.70:0/1983739509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93c41085b0 0x7f93c4108990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.970+0000 7f93cc42f700 1 --2- 172.21.15.70:0/1983739509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c4101ee0 0x7f93c4102360 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.970+0000 7f93cc42f700 1 -- 172.21.15.70:0/1983739509 >> 172.21.15.70:0/1983739509 conn(0x7f93c40777e0 msgr2=0x7f93c4077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.970+0000 7f93cc42f700 1 -- 172.21.15.70:0/1983739509 shutdown_connections 2024-06-04T05:22:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.970+0000 7f93cc42f700 1 -- 172.21.15.70:0/1983739509 wait complete. 2024-06-04T05:22:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.971+0000 7f93cc42f700 1 Processor -- start 2024-06-04T05:22:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.971+0000 7f93cc42f700 1 -- start start 2024-06-04T05:22:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.971+0000 7f93cc42f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93c4101ee0 0x7f93c41030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.971+0000 7f93cc42f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93c41028a0 0x7f93c4103600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.971+0000 7f93cc42f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c41085b0 0x7f93c4103b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.971+0000 7f93cc42f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f93c41135e0 con 0x7f93c41085b0 2024-06-04T05:22:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.971+0000 7f93cc42f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f93c4113460 con 0x7f93c41028a0 2024-06-04T05:22:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.971+0000 7f93cc42f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f93c4113760 con 0x7f93c4101ee0 2024-06-04T05:22:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.971+0000 7f93ca1cb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93c4101ee0 0x7f93c41030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.971+0000 7f93ca1cb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93c4101ee0 0x7f93c41030c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:54124/0 (socket says 172.21.15.70:54124) 2024-06-04T05:22:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.971+0000 7f93ca1cb700 1 -- 172.21.15.70:0/2632669664 learned_addr learned my addr 172.21.15.70:0/2632669664 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.971+0000 7f93ca9cc700 1 --2- 172.21.15.70:0/2632669664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c41085b0 0x7f93c4103b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.971+0000 7f93c99ca700 1 --2- 172.21.15.70:0/2632669664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93c41028a0 0x7f93c4103600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.972+0000 7f93ca9cc700 1 -- 172.21.15.70:0/2632669664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93c4101ee0 msgr2=0x7f93c41030c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.972+0000 7f93ca9cc700 1 --2- 172.21.15.70:0/2632669664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93c4101ee0 0x7f93c41030c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.972+0000 7f93ca9cc700 1 -- 172.21.15.70:0/2632669664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93c41028a0 msgr2=0x7f93c4103600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.972+0000 7f93ca9cc700 1 --2- 172.21.15.70:0/2632669664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93c41028a0 0x7f93c4103600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.972+0000 7f93ca9cc700 1 -- 172.21.15.70:0/2632669664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c41a8000 con 0x7f93c41085b0 2024-06-04T05:22:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.972+0000 7f93c99ca700 1 --2- 172.21.15.70:0/2632669664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93c41028a0 0x7f93c4103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:22:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.972+0000 7f93ca9cc700 1 --2- 172.21.15.70:0/2632669664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c41085b0 0x7f93c4103b40 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f93bc00efc0 tx=0x7f93bc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.972+0000 7f93bb7fe700 1 -- 172.21.15.70:0/2632669664 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc0092e0 con 0x7f93c41085b0 2024-06-04T05:22:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.972+0000 7f93cc42f700 1 -- 172.21.15.70:0/2632669664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93c41a8290 con 0x7f93c41085b0 2024-06-04T05:22:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.972+0000 7f93bb7fe700 1 -- 172.21.15.70:0/2632669664 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f93bc004bb0 con 0x7f93c41085b0 2024-06-04T05:22:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.972+0000 7f93bb7fe700 1 -- 172.21.15.70:0/2632669664 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc004020 con 0x7f93c41085b0 2024-06-04T05:22:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.972+0000 7f93cc42f700 1 -- 172.21.15.70:0/2632669664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93c41a87a0 con 0x7f93c41085b0 2024-06-04T05:22:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.973+0000 7f93cc42f700 1 -- 172.21.15.70:0/2632669664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93c406b6b0 con 0x7f93c41085b0 2024-06-04T05:22:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.976+0000 7f93bb7fe700 1 -- 172.21.15.70:0/2632669664 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f93bc01b030 con 0x7f93c41085b0 2024-06-04T05:22:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.976+0000 7f93bb7fe700 1 --2- 172.21.15.70:0/2632669664 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f93b00b29f0 0x7f93b00b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.977+0000 7f93ca1cb700 1 --2- 172.21.15.70:0/2632669664 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f93b00b29f0 0x7f93b00b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.977+0000 7f93bb7fe700 1 -- 172.21.15.70:0/2632669664 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8571+0+0 (secure 0 0 0) 0x7f93bc0e2bd0 con 0x7f93c41085b0 2024-06-04T05:22:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.977+0000 7f93ca1cb700 1 --2- 172.21.15.70:0/2632669664 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f93b00b29f0 0x7f93b00b4eb0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f93c0009ca0 tx=0x7f93c0009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:03.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:02.982+0000 7f93bb7fe700 1 -- 172.21.15.70:0/2632669664 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f93bc022020 con 0x7f93c41085b0 2024-06-04T05:22:03.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:03.414+0000 7f93cc42f700 1 -- 172.21.15.70:0/2632669664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93c402e560 con 0x7f93c41085b0 2024-06-04T05:22:03.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:03.415+0000 7f93bb7fe700 1 -- 172.21.15.70:0/2632669664 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+11931 (secure 0 0 0) 0x7f93bc010040 con 0x7f93c41085b0 2024-06-04T05:22:03.417 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:22:03.417 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":65,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:22:00.018729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24448,24442,24440,24410,24436]},"id":1}]} 2024-06-04T05:22:03.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:03.419+0000 7f93cc42f700 1 -- 172.21.15.70:0/2632669664 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f93b00b29f0 msgr2=0x7f93b00b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:03.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:03.419+0000 7f93cc42f700 1 --2- 172.21.15.70:0/2632669664 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f93b00b29f0 0x7f93b00b4eb0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f93c0009ca0 tx=0x7f93c0009260 comp rx=0 tx=0).stop 2024-06-04T05:22:03.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:03.419+0000 7f93cc42f700 1 -- 172.21.15.70:0/2632669664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c41085b0 msgr2=0x7f93c4103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:03.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:03.419+0000 7f93cc42f700 1 --2- 172.21.15.70:0/2632669664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c41085b0 0x7f93c4103b40 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f93bc00efc0 tx=0x7f93bc00c3a0 comp rx=0 tx=0).stop 2024-06-04T05:22:03.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:03.419+0000 7f93cc42f700 1 -- 172.21.15.70:0/2632669664 shutdown_connections 2024-06-04T05:22:03.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:03.419+0000 7f93cc42f700 1 --2- 172.21.15.70:0/2632669664 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f93b00b29f0 0x7f93b00b4eb0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:03.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:03.419+0000 7f93cc42f700 1 --2- 172.21.15.70:0/2632669664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c41085b0 0x7f93c4103b40 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:03.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:03.419+0000 7f93cc42f700 1 --2- 172.21.15.70:0/2632669664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93c4101ee0 0x7f93c41030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:03.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:03.419+0000 7f93cc42f700 1 --2- 172.21.15.70:0/2632669664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93c41028a0 0x7f93c4103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:03.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:03.419+0000 7f93cc42f700 1 -- 172.21.15.70:0/2632669664 >> 172.21.15.70:0/2632669664 conn(0x7f93c40777e0 msgr2=0x7f93c40fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:03.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:03.419+0000 7f93cc42f700 1 -- 172.21.15.70:0/2632669664 shutdown_connections 2024-06-04T05:22:03.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:03.419+0000 7f93cc42f700 1 -- 172.21.15.70:0/2632669664 wait complete. 2024-06-04T05:22:03.422 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 65 2024-06-04T05:22:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:03 smithi098 ceph-mon[35569]: pgmap v783: 97 pgs: 97 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-06-04T05:22:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:03 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2632669664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:22:03.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 83658 6.77 MB/sec execute 535 sec latency 3408.117 ms 2024-06-04T05:22:03.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:03 smithi070 ceph-mon[32631]: pgmap v783: 97 pgs: 97 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-06-04T05:22:03.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:03 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2632669664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:22:03.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:03 smithi112 ceph-mon[36453]: pgmap v783: 97 pgs: 97 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-06-04T05:22:03.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:03 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2632669664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:22:04.529 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:04 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:22:04.529 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:04 smithi070 ceph-mon[32631]: osdmap e117: 12 total, 12 up, 12 in 2024-06-04T05:22:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:04 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:22:04.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:04 smithi098 ceph-mon[35569]: osdmap e117: 12 total, 12 up, 12 in 2024-06-04T05:22:04.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 83815 6.77 MB/sec execute 536 sec latency 4408.142 ms 2024-06-04T05:22:04.874 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-06-04T05:22:04.875 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-06-04T05:22:04.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:04 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:22:04.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:04 smithi112 ceph-mon[36453]: osdmap e117: 12 total, 12 up, 12 in 2024-06-04T05:22:05.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.123+0000 7f8cbaa57700 1 -- 172.21.15.70:0/205196152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb410a780 msgr2=0x7f8cb410e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:05.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.123+0000 7f8cbaa57700 1 --2- 172.21.15.70:0/205196152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb410a780 0x7f8cb410e7d0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80099c0 tx=0x7f8ca8009cd0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.123+0000 7f8cbaa57700 1 -- 172.21.15.70:0/205196152 shutdown_connections 2024-06-04T05:22:05.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.123+0000 7f8cbaa57700 1 --2- 172.21.15.70:0/205196152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb410eeb0 0x7f8cb4116a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.123+0000 7f8cbaa57700 1 --2- 172.21.15.70:0/205196152 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8cb4109dd0 0x7f8cb410a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.123+0000 7f8cbaa57700 1 --2- 172.21.15.70:0/205196152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb410a780 0x7f8cb410e7d0 secure :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80099c0 tx=0x7f8ca8009cd0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.123+0000 7f8cbaa57700 1 -- 172.21.15.70:0/205196152 >> 172.21.15.70:0/205196152 conn(0x7f8cb4076d10 msgr2=0x7f8cb4079130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:05.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.124+0000 7f8cbaa57700 1 -- 172.21.15.70:0/205196152 shutdown_connections 2024-06-04T05:22:05.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.124+0000 7f8cbaa57700 1 -- 172.21.15.70:0/205196152 wait complete. 2024-06-04T05:22:05.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.125+0000 7f8cbaa57700 1 Processor -- start 2024-06-04T05:22:05.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.125+0000 7f8cbaa57700 1 -- start start 2024-06-04T05:22:05.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.125+0000 7f8cbaa57700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb4109dd0 0x7f8cb41a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.126+0000 7f8cbaa57700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb410eeb0 0x7f8cb41a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.126+0000 7f8cbaa57700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8cb41a4cd0 0x7f8cb41a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.126+0000 7f8cbaa57700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb4119c30 con 0x7f8cb4109dd0 2024-06-04T05:22:05.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.126+0000 7f8cb3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb4109dd0 0x7f8cb41a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.126+0000 7f8cb3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb4109dd0 0x7f8cb41a0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54414/0 (socket says 172.21.15.70:54414) 2024-06-04T05:22:05.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.126+0000 7f8cb3fff700 1 -- 172.21.15.70:0/2172199215 learned_addr learned my addr 172.21.15.70:0/2172199215 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:05.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.126+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/2172199215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb410eeb0 0x7f8cb41a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.126+0000 7f8cbaa57700 1 -- 172.21.15.70:0/2172199215 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb4119ab0 con 0x7f8cb410eeb0 2024-06-04T05:22:05.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.126+0000 7f8cbaa57700 1 -- 172.21.15.70:0/2172199215 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb4119db0 con 0x7f8cb41a4cd0 2024-06-04T05:22:05.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.126+0000 7f8cb8ff4700 1 --2- 172.21.15.70:0/2172199215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8cb41a4cd0 0x7f8cb41a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.127+0000 7f8cb37fe700 1 -- 172.21.15.70:0/2172199215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8cb41a4cd0 msgr2=0x7f8cb41a5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:05.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.127+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/2172199215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8cb41a4cd0 0x7f8cb41a5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.127+0000 7f8cb37fe700 1 -- 172.21.15.70:0/2172199215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb4109dd0 msgr2=0x7f8cb41a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:05.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.127+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/2172199215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb4109dd0 0x7f8cb41a0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.127+0000 7f8cb37fe700 1 -- 172.21.15.70:0/2172199215 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb41a57f0 con 0x7f8cb410eeb0 2024-06-04T05:22:05.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.127+0000 7f8cb3fff700 1 --2- 172.21.15.70:0/2172199215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb4109dd0 0x7f8cb41a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:05.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.127+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/2172199215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb410eeb0 0x7f8cb41a0940 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8cb410ace0 tx=0x7f8ca8005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:05.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.127+0000 7f8cb17fa700 1 -- 172.21.15.70:0/2172199215 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca8009e60 con 0x7f8cb410eeb0 2024-06-04T05:22:05.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.127+0000 7f8cbaa57700 1 -- 172.21.15.70:0/2172199215 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cb41a5a80 con 0x7f8cb410eeb0 2024-06-04T05:22:05.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.127+0000 7f8cbaa57700 1 -- 172.21.15.70:0/2172199215 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cb41ad150 con 0x7f8cb410eeb0 2024-06-04T05:22:05.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.127+0000 7f8cb17fa700 1 -- 172.21.15.70:0/2172199215 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8ca80056c0 con 0x7f8cb410eeb0 2024-06-04T05:22:05.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.127+0000 7f8cb17fa700 1 -- 172.21.15.70:0/2172199215 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca800faa0 con 0x7f8cb410eeb0 2024-06-04T05:22:05.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.131+0000 7f8cb17fa700 1 -- 172.21.15.70:0/2172199215 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f8ca800fc40 con 0x7f8cb410eeb0 2024-06-04T05:22:05.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.132+0000 7f8cc52ecb80 1 Processor -- start 2024-06-04T05:22:05.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.132+0000 7f8cc52ecb80 1 -- start start 2024-06-04T05:22:05.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.132+0000 7f8cb17fa700 1 --2- 172.21.15.70:0/2172199215 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8c9c0b2a70 0x7f8c9c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.132+0000 7f8cc52ecb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 0x5572bba48c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.132+0000 7f8cc52ecb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 0x5572bba4c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.132+0000 7f8cc52ecb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba4cf60 0x5572bba4d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.132+0000 7f8cc52ecb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5572bb9a7c90 con 0x5572bba48860 2024-06-04T05:22:05.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.132+0000 7f8cc52ecb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5572bb997830 con 0x5572bba49210 2024-06-04T05:22:05.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.132+0000 7f8cc52ecb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5572bb99b870 con 0x5572bba4cf60 2024-06-04T05:22:05.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.132+0000 7f8cb17fa700 1 -- 172.21.15.70:0/2172199215 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8615+0+0 (secure 0 0 0) 0x7f8ca80e3180 con 0x7f8cb410eeb0 2024-06-04T05:22:05.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.132+0000 7f8cb37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 0x5572bba4c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.133+0000 7f8cb37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 0x5572bba4c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55134/0 (socket says 172.21.15.70:55134) 2024-06-04T05:22:05.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.133+0000 7f8cb37fe700 1 -- 172.21.15.70:0/1815172927 learned_addr learned my addr 172.21.15.70:0/1815172927 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:05.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.133+0000 7f8cb8ff4700 1 --2- 172.21.15.70:0/2172199215 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8c9c0b2a70 0x7f8c9c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.133+0000 7f8cb3fff700 1 --2- 172.21.15.70:0/1815172927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 0x5572bba48c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.133+0000 7f8cb37fe700 1 -- 172.21.15.70:0/1815172927 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba4cf60 msgr2=0x5572bba4d3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:22:05.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.133+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/1815172927 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba4cf60 0x5572bba4d3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.133+0000 7f8cb37fe700 1 -- 172.21.15.70:0/1815172927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 msgr2=0x5572bba48c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:05.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.133+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/1815172927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 0x5572bba48c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.133+0000 7f8cb37fe700 1 -- 172.21.15.70:0/1815172927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572bb8e78d0 con 0x5572bba49210 2024-06-04T05:22:05.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.133+0000 7f8cb3fff700 1 --2- 172.21.15.70:0/1815172927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 0x5572bba48c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:22:05.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.133+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/1815172927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 0x5572bba4c940 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80a6510 tx=0x7f8ca80a6480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93b5254ce4759250 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:05.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.134+0000 7f8c9a7fc700 1 -- 172.21.15.70:0/1815172927 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca8003e00 con 0x5572bba49210 2024-06-04T05:22:05.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.134+0000 7f8c9a7fc700 1 -- 172.21.15.70:0/1815172927 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8ca800f460 con 0x5572bba49210 2024-06-04T05:22:05.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.134+0000 7f8c9a7fc700 1 -- 172.21.15.70:0/1815172927 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca80ae840 con 0x5572bba49210 2024-06-04T05:22:05.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.135+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/1815172927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 msgr2=0x5572bba4c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:05.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.135+0000 7f8cb8ff4700 1 --2- 172.21.15.70:0/2172199215 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8c9c0b2a70 0x7f8c9c0b4f30 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8cac009ec0 tx=0x7f8cac00b040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:05.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.135+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/1815172927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 0x5572bba4c940 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80a6510 tx=0x7f8ca80a6480 comp rx=0 tx=0).stop 2024-06-04T05:22:05.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.135+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/1815172927 shutdown_connections 2024-06-04T05:22:05.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.135+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/1815172927 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba4cf60 0x5572bba4d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.135+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/1815172927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 0x5572bba48c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.135+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/1815172927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 0x5572bba4c940 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.135+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/1815172927 >> 172.21.15.70:0/1815172927 conn(0x5572bb976080 msgr2=0x5572bba42e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:05.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.135+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/1815172927 shutdown_connections 2024-06-04T05:22:05.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.135+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/1815172927 wait complete. 2024-06-04T05:22:05.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.135+0000 7f8cc52ecb80 1 -- start start 2024-06-04T05:22:05.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.136+0000 7f8cc52ecb80 1 Processor -- start 2024-06-04T05:22:05.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.136+0000 7f8cc52ecb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba48860 0x5572bba54390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.136+0000 7f8cc52ecb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 0x5572bba51690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.136+0000 7f8cc52ecb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba4cf60 0x5572bba51bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.136+0000 7f8cc52ecb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5572bb89dfa0 con 0x5572bba4cf60 2024-06-04T05:22:05.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.136+0000 7f8cc52ecb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5572bb9397c0 con 0x5572bba49210 2024-06-04T05:22:05.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.136+0000 7f8cc52ecb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5572bba65ef0 con 0x5572bba48860 2024-06-04T05:22:05.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.136+0000 7f8cc52ecb80 10 client.0 _ll_register_callbacks cb 0x5572bb932920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.136+0000 7f8cb8ff4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba4cf60 0x5572bba51bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.136+0000 7f8cb37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 0x5572bba51690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.136+0000 7f8cb3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba48860 0x5572bba54390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.136+0000 7f8cb8ff4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba4cf60 0x5572bba51bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54436/0 (socket says 172.21.15.70:54436) 2024-06-04T05:22:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.136+0000 7f8cb8ff4700 1 -- 172.21.15.70:0/1743551842 learned_addr learned my addr 172.21.15.70:0/1743551842 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.137+0000 7f8cb37fe700 1 -- 172.21.15.70:0/1743551842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba48860 msgr2=0x5572bba54390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.137+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/1743551842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba48860 0x5572bba54390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.137+0000 7f8cb37fe700 1 -- 172.21.15.70:0/1743551842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba4cf60 msgr2=0x5572bba51bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.137+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/1743551842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba4cf60 0x5572bba51bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.137+0000 7f8cb37fe700 1 -- 172.21.15.70:0/1743551842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572bb936690 con 0x5572bba49210 2024-06-04T05:22:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.137+0000 7f8cb3fff700 1 --2- 172.21.15.70:0/1743551842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba48860 0x5572bba54390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:22:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.137+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/1743551842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 0x5572bba51690 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80a6510 tx=0x7f8ca80a69c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75863cff2a062127 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.137+0000 7f8c937fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:22:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.137+0000 7f8c937fe700 1 -- 172.21.15.70:0/1743551842 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca80b3260 con 0x5572bba49210 2024-06-04T05:22:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.137+0000 7f8c937fe700 1 -- 172.21.15.70:0/1743551842 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8615+0+0 (secure 0 0 0) 0x7f8ca80aa900 con 0x5572bba49210 2024-06-04T05:22:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.137+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/1743551842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5572bba149e0 con 0x5572bba49210 2024-06-04T05:22:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.138+0000 7f8c937fe700 1 -- 172.21.15.70:0/1743551842 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f8ca80ae840 con 0x5572bba49210 2024-06-04T05:22:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.138+0000 7f8cc52ecb80 10 client.30320 fetch_fsmap learned FSMap version 66 2024-06-04T05:22:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.138+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/1743551842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x5572bba70430 con 0x5572bba49210 2024-06-04T05:22:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.138+0000 7f8c937fe700 1 -- 172.21.15.70:0/1743551842 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x5572bba149e0 con 0x5572bba49210 2024-06-04T05:22:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.138+0000 7f8c937fe700 1 -- 172.21.15.70:0/1743551842 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 66) ==== 8358+0+0 (secure 0 0 0) 0x7f8ca80a4bb0 con 0x5572bba49210 2024-06-04T05:22:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.138+0000 7f8cc52ecb80 10 client.30320 fetch_fsmap finished waiting for FSMap version 66 2024-06-04T05:22:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.138+0000 7f8cc52ecb80 10 client.30320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.138+0000 7f8cc52ecb80 20 client.30320 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.138+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/1743551842 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5572bba72900 0x5572bba74cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.138+0000 7f8cc52ecb80 4 client.30320 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:22:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.138+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/1743551842 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572bb9a9be0 con 0x5572bba72900 2024-06-04T05:22:05.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.138+0000 7f8cb3fff700 1 --2- 172.21.15.70:0/1743551842 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5572bba72900 0x5572bba74cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.139+0000 7f8c937fe700 10 client.30320 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.139+0000 7f8cb3fff700 1 --2- 172.21.15.70:0/1743551842 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5572bba72900 0x5572bba74cf0 crc :-1 s=READY pgs=513 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e24c59a37bf44e06 server_cookie=865904ff634c440f in_seq=0 out_seq=0 2024-06-04T05:22:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.139+0000 7f8c937fe700 1 -- 172.21.15.70:0/1743551842 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5572bb9a9be0 con 0x5572bba72900 2024-06-04T05:22:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.139+0000 7f8c937fe700 10 client.30320 handle_command_reply: tid=0 2024-06-04T05:22:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.140+0000 7f8cc52ecb80 1 client.30320 shutdown 2024-06-04T05:22:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.140+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/1743551842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 msgr2=0x5572bba51690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.140+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/1743551842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 0x5572bba51690 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80a6510 tx=0x7f8ca80a69c0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.140+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/1743551842 shutdown_connections 2024-06-04T05:22:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.140+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/1743551842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba4cf60 0x5572bba51bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.140+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/1743551842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba48860 0x5572bba54390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.140+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/1743551842 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5572bba72900 0x5572bba74cf0 crc :-1 s=READY pgs=513 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.140+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/1743551842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 0x5572bba51690 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.140+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/1743551842 >> 172.21.15.70:0/1743551842 conn(0x5572bb976080 msgr2=0x5572bba4e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.140+0000 7f8c937fe700 0 client.30320 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:05.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.140+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/1743551842 shutdown_connections 2024-06-04T05:22:05.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.141+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/1743551842 wait complete. 2024-06-04T05:22:05.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.141+0000 7f8cc52ecb80 20 client.30320 trim_cache size 0 max 16384 2024-06-04T05:22:05.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.166+0000 7f8cc52ecb80 1 Processor -- start 2024-06-04T05:22:05.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.166+0000 7f8cc52ecb80 1 -- start start 2024-06-04T05:22:05.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.166+0000 7f8cc52ecb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 0x5572bba50690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.167+0000 7f8cc52ecb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba49210 0x5572bba7a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.167+0000 7f8cc52ecb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba4cf60 0x5572bba7aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.167+0000 7f8cc52ecb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb40832a0 con 0x5572bba48860 2024-06-04T05:22:05.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.167+0000 7f8cc52ecb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb41adcb0 con 0x5572bba4cf60 2024-06-04T05:22:05.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.167+0000 7f8cc52ecb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb40839a0 con 0x5572bba49210 2024-06-04T05:22:05.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.167+0000 7f8cb3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 0x5572bba50690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.167+0000 7f8cb3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 0x5572bba50690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54438/0 (socket says 172.21.15.70:54438) 2024-06-04T05:22:05.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.167+0000 7f8cb3fff700 1 -- 172.21.15.70:0/3923330275 learned_addr learned my addr 172.21.15.70:0/3923330275 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:05.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.167+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/3923330275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba49210 0x5572bba7a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.167+0000 7f8cb8ff4700 1 --2- 172.21.15.70:0/3923330275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba4cf60 0x5572bba7aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.167+0000 7f8cb37fe700 1 -- 172.21.15.70:0/3923330275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba4cf60 msgr2=0x5572bba7aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:05.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.167+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/3923330275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba4cf60 0x5572bba7aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.167+0000 7f8cb37fe700 1 -- 172.21.15.70:0/3923330275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 msgr2=0x5572bba50690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:05.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.167+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/3923330275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 0x5572bba50690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.167+0000 7f8cb37fe700 1 -- 172.21.15.70:0/3923330275 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572bba43d40 con 0x5572bba49210 2024-06-04T05:22:05.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.167+0000 7f8cb3fff700 1 --2- 172.21.15.70:0/3923330275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 0x5572bba50690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:05.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.167+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/3923330275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba49210 0x5572bba7a500 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80e0e70 tx=0x7f8ca80b3190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1e8164a9081444d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:05.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.168+0000 7f8c99ffb700 1 -- 172.21.15.70:0/3923330275 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca80b3260 con 0x5572bba49210 2024-06-04T05:22:05.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.168+0000 7f8c99ffb700 1 -- 172.21.15.70:0/3923330275 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8ca80aa490 con 0x5572bba49210 2024-06-04T05:22:05.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.168+0000 7f8c99ffb700 1 -- 172.21.15.70:0/3923330275 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca80a5e00 con 0x5572bba49210 2024-06-04T05:22:05.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.168+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/3923330275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba49210 msgr2=0x5572bba7a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:05.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.168+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/3923330275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba49210 0x5572bba7a500 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80e0e70 tx=0x7f8ca80b3190 comp rx=0 tx=0).stop 2024-06-04T05:22:05.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.168+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/3923330275 shutdown_connections 2024-06-04T05:22:05.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.168+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/3923330275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba4cf60 0x5572bba7aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.168+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/3923330275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 0x5572bba50690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.168+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/3923330275 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba49210 0x5572bba7a500 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.169+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/3923330275 >> 172.21.15.70:0/3923330275 conn(0x5572bb976080 msgr2=0x5572bba4ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:05.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.169+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/3923330275 shutdown_connections 2024-06-04T05:22:05.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.169+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/3923330275 wait complete. 2024-06-04T05:22:05.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.169+0000 7f8cc52ecb80 1 -- start start 2024-06-04T05:22:05.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.169+0000 7f8cc52ecb80 1 Processor -- start 2024-06-04T05:22:05.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.170+0000 7f8cc52ecb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 0x5572bba66050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.170+0000 7f8cc52ecb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 0x5572bba66590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.170+0000 7f8cc52ecb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba4cf60 0x5572bba66ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.170+0000 7f8cb3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 0x5572bba66050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.170+0000 7f8cb3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 0x5572bba66050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54442/0 (socket says 172.21.15.70:54442) 2024-06-04T05:22:05.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.170+0000 7f8cb3fff700 1 -- 172.21.15.70:0/2592538410 learned_addr learned my addr 172.21.15.70:0/2592538410 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:05.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.170+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/2592538410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 0x5572bba66590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.170+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/2592538410 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb41ad5e0 con 0x5572bba48860 2024-06-04T05:22:05.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.170+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/2592538410 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb4084070 con 0x5572bba49210 2024-06-04T05:22:05.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.170+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/2592538410 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb4081260 con 0x5572bba4cf60 2024-06-04T05:22:05.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.170+0000 7f8cb8ff4700 1 --2- 172.21.15.70:0/2592538410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba4cf60 0x5572bba66ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.171+0000 7f8cc52ecb80 10 client.0 _ll_register_callbacks cb 0x5572bb932920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:05.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.171+0000 7f8cb37fe700 1 -- 172.21.15.70:0/2592538410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba4cf60 msgr2=0x5572bba66ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:05.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.171+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/2592538410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba4cf60 0x5572bba66ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.171+0000 7f8cb37fe700 1 -- 172.21.15.70:0/2592538410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 msgr2=0x5572bba66050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:05.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.171+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/2592538410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 0x5572bba66050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.171+0000 7f8cb37fe700 1 -- 172.21.15.70:0/2592538410 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572bbadb500 con 0x5572bba49210 2024-06-04T05:22:05.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.171+0000 7f8cb37fe700 1 --2- 172.21.15.70:0/2592538410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 0x5572bba66590 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x5572bba4cdd0 tx=0x7f8ca800b580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f9ab6cfce46b0ca server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:05.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.171+0000 7f8c937fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:22:05.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.172+0000 7f8c937fe700 1 -- 172.21.15.70:0/2592538410 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca80c3060 con 0x5572bba49210 2024-06-04T05:22:05.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.172+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/2592538410 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ca80aa490 con 0x5572bba49210 2024-06-04T05:22:05.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.172+0000 7f8c937fe700 1 -- 172.21.15.70:0/2592538410 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8615+0+0 (secure 0 0 0) 0x7f8ca80b2dd0 con 0x5572bba49210 2024-06-04T05:22:05.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.172+0000 7f8c937fe700 1 -- 172.21.15.70:0/2592538410 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f8ca80a96f0 con 0x5572bba49210 2024-06-04T05:22:05.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.172+0000 7f8cc52ecb80 10 client.30326 fetch_fsmap learned FSMap version 66 2024-06-04T05:22:05.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.172+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/2592538410 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x5572bbadb500 con 0x5572bba49210 2024-06-04T05:22:05.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.172+0000 7f8c937fe700 1 -- 172.21.15.70:0/2592538410 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8ca80aa490 con 0x5572bba49210 2024-06-04T05:22:05.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.172+0000 7f8c937fe700 1 -- 172.21.15.70:0/2592538410 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 66) ==== 8358+0+0 (secure 0 0 0) 0x7f8ca8016a60 con 0x5572bba49210 2024-06-04T05:22:05.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.173+0000 7f8cc52ecb80 10 client.30326 fetch_fsmap finished waiting for FSMap version 66 2024-06-04T05:22:05.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.173+0000 7f8cc52ecb80 10 client.30326 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:05.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.173+0000 7f8cc52ecb80 20 client.30326 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:05.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.173+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/2592538410 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5572bbadc9b0 0x5572bbadeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.173+0000 7f8cc52ecb80 4 client.30326 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:22:05.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.173+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/2592538410 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5572bb9a33c0 con 0x5572bbadc9b0 2024-06-04T05:22:05.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.173+0000 7f8cb3fff700 1 --2- 172.21.15.70:0/2592538410 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5572bbadc9b0 0x5572bbadeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.173+0000 7f8cb3fff700 1 --2- 172.21.15.70:0/2592538410 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5572bbadc9b0 0x5572bbadeda0 crc :-1 s=READY pgs=514 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab5212f4856f138 server_cookie=5b1ff98eb3f3b468 in_seq=0 out_seq=0 2024-06-04T05:22:05.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.173+0000 7f8c937fe700 10 client.30326 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:05.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.173+0000 7f8c937fe700 1 -- 172.21.15.70:0/2592538410 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5572bb9a33c0 con 0x5572bbadc9b0 2024-06-04T05:22:05.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.174+0000 7f8c937fe700 10 client.30326 handle_command_reply: tid=0 2024-06-04T05:22:05.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.174+0000 7f8cc52ecb80 1 client.30326 shutdown 2024-06-04T05:22:05.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.174+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/2592538410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 msgr2=0x5572bba66590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:05.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.174+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/2592538410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 0x5572bba66590 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x5572bba4cdd0 tx=0x7f8ca800b580 comp rx=0 tx=0).stop 2024-06-04T05:22:05.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.174+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/2592538410 shutdown_connections 2024-06-04T05:22:05.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.174+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/2592538410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5572bba4cf60 0x5572bba66ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.174+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/2592538410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572bba48860 0x5572bba66050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.174+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/2592538410 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5572bbadc9b0 0x5572bbadeda0 crc :-1 s=READY pgs=514 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.174+0000 7f8cc52ecb80 1 --2- 172.21.15.70:0/2592538410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572bba49210 0x5572bba66590 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.174+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/2592538410 >> 172.21.15.70:0/2592538410 conn(0x5572bb976080 msgr2=0x5572bba47f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:05.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.174+0000 7f8c937fe700 0 client.30326 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:05.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.174+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/2592538410 shutdown_connections 2024-06-04T05:22:05.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.174+0000 7f8cc52ecb80 1 -- 172.21.15.70:0/2592538410 wait complete. 2024-06-04T05:22:05.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.175+0000 7f8cc52ecb80 20 client.30326 trim_cache size 0 max 16384 2024-06-04T05:22:05.180 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:22:05.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.175+0000 7f8cbaa57700 1 -- 172.21.15.70:0/2172199215 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8c9c0b2a70 msgr2=0x7f8c9c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:05.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.175+0000 7f8cbaa57700 1 --2- 172.21.15.70:0/2172199215 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8c9c0b2a70 0x7f8c9c0b4f30 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8cac009ec0 tx=0x7f8cac00b040 comp rx=0 tx=0).stop 2024-06-04T05:22:05.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.175+0000 7f8cbaa57700 1 -- 172.21.15.70:0/2172199215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb410eeb0 msgr2=0x7f8cb41a0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:05.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.175+0000 7f8cbaa57700 1 --2- 172.21.15.70:0/2172199215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb410eeb0 0x7f8cb41a0940 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8cb410ace0 tx=0x7f8ca8005280 comp rx=0 tx=0).stop 2024-06-04T05:22:05.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.176+0000 7f8cbaa57700 1 -- 172.21.15.70:0/2172199215 shutdown_connections 2024-06-04T05:22:05.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.176+0000 7f8cbaa57700 1 --2- 172.21.15.70:0/2172199215 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f8c9c0b2a70 0x7f8c9c0b4f30 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.176+0000 7f8cbaa57700 1 --2- 172.21.15.70:0/2172199215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8cb41a4cd0 0x7f8cb41a5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.176+0000 7f8cbaa57700 1 --2- 172.21.15.70:0/2172199215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb4109dd0 0x7f8cb41a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.176+0000 7f8cbaa57700 1 --2- 172.21.15.70:0/2172199215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb410eeb0 0x7f8cb41a0940 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.176+0000 7f8cbaa57700 1 -- 172.21.15.70:0/2172199215 >> 172.21.15.70:0/2172199215 conn(0x7f8cb4076d10 msgr2=0x7f8cb4077820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:05.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.176+0000 7f8cbaa57700 1 -- 172.21.15.70:0/2172199215 shutdown_connections 2024-06-04T05:22:05.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:05.176+0000 7f8cbaa57700 1 -- 172.21.15.70:0/2172199215 wait complete. 2024-06-04T05:22:05.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:05 smithi098 ceph-mon[35569]: pgmap v785: 97 pgs: 97 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.4 MiB/s wr, 140 op/s 2024-06-04T05:22:05.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 83950 6.78 MB/sec execute 537 sec latency 5267.027 ms 2024-06-04T05:22:05.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:05 smithi070 ceph-mon[32631]: pgmap v785: 97 pgs: 97 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.4 MiB/s wr, 140 op/s 2024-06-04T05:22:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.909+0000 7f0912f9d640 1 -- 172.21.15.70:0/3888898097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f090c102370 msgr2=0x7f090c10e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.909+0000 7f0912f9d640 1 --2- 172.21.15.70:0/3888898097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f090c102370 0x7f090c10e830 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f0904009cd0 tx=0x7f090402a8e0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.909+0000 7f0912f9d640 1 -- 172.21.15.70:0/3888898097 shutdown_connections 2024-06-04T05:22:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.909+0000 7f0912f9d640 1 --2- 172.21.15.70:0/3888898097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c1019b0 0x7f090c101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.909+0000 7f0912f9d640 1 --2- 172.21.15.70:0/3888898097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f090c102370 0x7f090c10e830 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.909+0000 7f0912f9d640 1 --2- 172.21.15.70:0/3888898097 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f090c107970 0x7f090c107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.909+0000 7f0912f9d640 1 -- 172.21.15.70:0/3888898097 >> 172.21.15.70:0/3888898097 conn(0x7f090c078aa0 msgr2=0x7f090c078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.909+0000 7f0912f9d640 1 -- 172.21.15.70:0/3888898097 shutdown_connections 2024-06-04T05:22:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.909+0000 7f0912f9d640 1 -- 172.21.15.70:0/3888898097 wait complete. 2024-06-04T05:22:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.910+0000 7f0912f9d640 1 Processor -- start 2024-06-04T05:22:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.910+0000 7f0912f9d640 1 -- start start 2024-06-04T05:22:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.910+0000 7f0912f9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f090c1019b0 0x7f090c10dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.910+0000 7f0912f9d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f090c102370 0x7f090c10e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.910+0000 7f0912f9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c107970 0x7f090c19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.910+0000 7f0912f9d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f090c06b670 con 0x7f090c1019b0 2024-06-04T05:22:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.910+0000 7f0912f9d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f090c06b4f0 con 0x7f090c107970 2024-06-04T05:22:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.910+0000 7f0912f9d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f090c06b7f0 con 0x7f090c102370 2024-06-04T05:22:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.910+0000 7f0910d39640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f090c1019b0 0x7f090c10dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.910+0000 7f090bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f090c102370 0x7f090c10e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.910+0000 7f090bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f090c102370 0x7f090c10e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:54234/0 (socket says 172.21.15.70:54234) 2024-06-04T05:22:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.910+0000 7f0910d39640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f090c1019b0 0x7f090c10dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54464/0 (socket says 172.21.15.70:54464) 2024-06-04T05:22:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.910+0000 7f090bfff640 1 -- 172.21.15.70:0/549064416 learned_addr learned my addr 172.21.15.70:0/549064416 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.910+0000 7f091153a640 1 --2- 172.21.15.70:0/549064416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c107970 0x7f090c19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.911+0000 7f091153a640 1 -- 172.21.15.70:0/549064416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f090c102370 msgr2=0x7f090c10e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.911+0000 7f091153a640 1 --2- 172.21.15.70:0/549064416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f090c102370 0x7f090c10e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.911+0000 7f091153a640 1 -- 172.21.15.70:0/549064416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f090c1019b0 msgr2=0x7f090c10dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.911+0000 7f091153a640 1 --2- 172.21.15.70:0/549064416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f090c1019b0 0x7f090c10dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.911+0000 7f091153a640 1 -- 172.21.15.70:0/549064416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f090c1a0390 con 0x7f090c107970 2024-06-04T05:22:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.911+0000 7f090bfff640 1 --2- 172.21.15.70:0/549064416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f090c102370 0x7f090c10e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.911+0000 7f0910d39640 1 --2- 172.21.15.70:0/549064416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f090c1019b0 0x7f090c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.911+0000 7f091153a640 1 --2- 172.21.15.70:0/549064416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c107970 0x7f090c19fc60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0904009ca0 tx=0x7f0904003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:05.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.911+0000 7f0909ffb640 1 -- 172.21.15.70:0/549064416 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0904004040 con 0x7f090c107970 2024-06-04T05:22:05.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.911+0000 7f0912f9d640 1 -- 172.21.15.70:0/549064416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f090c1a0620 con 0x7f090c107970 2024-06-04T05:22:05.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.912+0000 7f0912f9d640 1 -- 172.21.15.70:0/549064416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f090c1a7f10 con 0x7f090c107970 2024-06-04T05:22:05.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.912+0000 7f0909ffb640 1 -- 172.21.15.70:0/549064416 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0904004670 con 0x7f090c107970 2024-06-04T05:22:05.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.912+0000 7f0909ffb640 1 -- 172.21.15.70:0/549064416 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09040309c0 con 0x7f090c107970 2024-06-04T05:22:05.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.914+0000 7f0909ffb640 1 -- 172.21.15.70:0/549064416 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f0904030b80 con 0x7f090c107970 2024-06-04T05:22:05.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.915+0000 7f0915d90740 1 Processor -- start 2024-06-04T05:22:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.915+0000 7f0915d90740 1 -- start start 2024-06-04T05:22:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.915+0000 7f0915d90740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e85820 0x560315e85c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.915+0000 7f0915d90740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e86140 0x560315e904c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.915+0000 7f0915d90740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e90a00 0x560315e92df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.915+0000 7f0915d90740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560315df7c30 con 0x560315e90a00 2024-06-04T05:22:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.915+0000 7f0915d90740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560315de77d0 con 0x560315e85820 2024-06-04T05:22:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.915+0000 7f0915d90740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560315deb810 con 0x560315e86140 2024-06-04T05:22:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.915+0000 7f090bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e85820 0x560315e85c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.915+0000 7f0910d39640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e86140 0x560315e904c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.915+0000 7f090bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e85820 0x560315e85c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55202/0 (socket says 172.21.15.70:55202) 2024-06-04T05:22:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.915+0000 7f090bfff640 1 -- 172.21.15.70:0/1882365529 learned_addr learned my addr 172.21.15.70:0/1882365529 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:05.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.915+0000 7f0910d39640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e86140 0x560315e904c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:54246/0 (socket says 172.21.15.70:54246) 2024-06-04T05:22:05.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.915+0000 7f090bfff640 1 --2- 172.21.15.70:0/1882365529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e90a00 0x560315e92df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.915+0000 7f0909ffb640 1 --2- 172.21.15.70:0/549064416 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08fc0b2a10 0x7f08fc0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.916+0000 7f091153a640 1 --2- 172.21.15.70:0/549064416 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08fc0b2a10 0x7f08fc0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.916+0000 7f0910d39640 1 -- 172.21.15.70:0/1882365529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e85820 msgr2=0x560315e85c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:05.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.916+0000 7f0910d39640 1 --2- 172.21.15.70:0/1882365529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e85820 0x560315e85c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.916+0000 7f0910d39640 1 -- 172.21.15.70:0/1882365529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e90a00 msgr2=0x560315e92df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:05.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.916+0000 7f0910d39640 1 --2- 172.21.15.70:0/1882365529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e90a00 0x560315e92df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.916+0000 7f0909ffb640 1 -- 172.21.15.70:0/549064416 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8615+0+0 (secure 0 0 0) 0x7f090402e070 con 0x7f090c107970 2024-06-04T05:22:05.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.916+0000 7f0910d39640 1 -- 172.21.15.70:0/1882365529 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560315afc210 con 0x560315e86140 2024-06-04T05:22:05.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.916+0000 7f090bfff640 1 --2- 172.21.15.70:0/1882365529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e85820 0x560315e85c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:05.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.916+0000 7f091153a640 1 --2- 172.21.15.70:0/549064416 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08fc0b2a10 0x7f08fc0b4ed0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0904100dc0 tx=0x7f0904106b70 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:05.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.917+0000 7f0910d39640 1 --2- 172.21.15.70:0/1882365529 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e86140 0x560315e904c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f090000bf70 tx=0x7f0900010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9087e982d3c9e8aa server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.917+0000 7f08f6ffd640 1 -- 172.21.15.70:0/1882365529 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0900011700 con 0x560315e86140 2024-06-04T05:22:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.917+0000 7f08f6ffd640 1 -- 172.21.15.70:0/1882365529 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0900011d30 con 0x560315e86140 2024-06-04T05:22:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.917+0000 7f08f6ffd640 1 -- 172.21.15.70:0/1882365529 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090001cda0 con 0x560315e86140 2024-06-04T05:22:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.918+0000 7f0915d90740 1 -- 172.21.15.70:0/1882365529 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e86140 msgr2=0x560315e904c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.918+0000 7f0915d90740 1 --2- 172.21.15.70:0/1882365529 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e86140 0x560315e904c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f090000bf70 tx=0x7f0900010860 comp rx=0 tx=0).stop 2024-06-04T05:22:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.918+0000 7f0915d90740 1 -- 172.21.15.70:0/1882365529 shutdown_connections 2024-06-04T05:22:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.918+0000 7f0915d90740 1 --2- 172.21.15.70:0/1882365529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e90a00 0x560315e92df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.918+0000 7f0915d90740 1 --2- 172.21.15.70:0/1882365529 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e86140 0x560315e904c0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.918+0000 7f0915d90740 1 --2- 172.21.15.70:0/1882365529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e85820 0x560315e85c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.918+0000 7f0915d90740 1 -- 172.21.15.70:0/1882365529 >> 172.21.15.70:0/1882365529 conn(0x560315e81d30 msgr2=0x560315e82110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.918+0000 7f0915d90740 1 -- 172.21.15.70:0/1882365529 shutdown_connections 2024-06-04T05:22:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.918+0000 7f0915d90740 1 -- 172.21.15.70:0/1882365529 wait complete. 2024-06-04T05:22:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.918+0000 7f0915d90740 1 -- start start 2024-06-04T05:22:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.918+0000 7f0915d90740 1 Processor -- start 2024-06-04T05:22:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f0915d90740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e85820 0x560315e8d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f0915d90740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e86140 0x560315e8dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f0915d90740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e90a00 0x560315e8e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f0915d90740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560315acd790 con 0x560315e86140 2024-06-04T05:22:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f0915d90740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560315da3890 con 0x560315e90a00 2024-06-04T05:22:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f0915d90740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560315e93630 con 0x560315e85820 2024-06-04T05:22:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f0915d90740 10 client.0 _ll_register_callbacks cb 0x560315afba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f0910d39640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e86140 0x560315e8dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f0910d39640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e86140 0x560315e8dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54490/0 (socket says 172.21.15.70:54490) 2024-06-04T05:22:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f0910d39640 1 -- 172.21.15.70:0/486373238 learned_addr learned my addr 172.21.15.70:0/486373238 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f090bfff640 1 --2- 172.21.15.70:0/486373238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e85820 0x560315e8d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f0910d39640 1 -- 172.21.15.70:0/486373238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e85820 msgr2=0x560315e8d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f0910d39640 1 --2- 172.21.15.70:0/486373238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e85820 0x560315e8d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f0910d39640 1 -- 172.21.15.70:0/486373238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e90a00 msgr2=0x560315e8e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f090bfff640 1 --2- 172.21.15.70:0/486373238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e90a00 0x560315e8e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f0910d39640 1 --2- 172.21.15.70:0/486373238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e90a00 0x560315e8e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f0910d39640 1 -- 172.21.15.70:0/486373238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560315bc8620 con 0x560315e86140 2024-06-04T05:22:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f0910d39640 1 --2- 172.21.15.70:0/486373238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e86140 0x560315e8dea0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f090000bd20 tx=0x7f090001c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35d772904fe7c9cc server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.919+0000 7f08ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:22:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.920+0000 7f08ebfff640 1 -- 172.21.15.70:0/486373238 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090002a060 con 0x560315e86140 2024-06-04T05:22:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.920+0000 7f08ebfff640 1 -- 172.21.15.70:0/486373238 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8615+0+0 (secure 0 0 0) 0x7f090001cd30 con 0x560315e86140 2024-06-04T05:22:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.920+0000 7f0915d90740 1 -- 172.21.15.70:0/486373238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0900011d30 con 0x560315e86140 2024-06-04T05:22:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.920+0000 7f08ebfff640 1 -- 172.21.15.70:0/486373238 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f0900004040 con 0x560315e86140 2024-06-04T05:22:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.920+0000 7f0915d90740 10 client.21729 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.920+0000 7f0915d90740 1 -- 172.21.15.70:0/486373238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x560315ea7bf0 con 0x560315e86140 2024-06-04T05:22:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.920+0000 7f08ebfff640 1 -- 172.21.15.70:0/486373238 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0900011d30 con 0x560315e86140 2024-06-04T05:22:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.920+0000 7f08ebfff640 1 -- 172.21.15.70:0/486373238 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f0900011500 con 0x560315e86140 2024-06-04T05:22:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.920+0000 7f0915d90740 10 client.21729 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.920+0000 7f0915d90740 10 client.21729 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.920+0000 7f0915d90740 20 client.21729 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.920+0000 7f0915d90740 1 --2- 172.21.15.70:0/486373238 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560315ea8fa0 0x560315eab390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.920+0000 7f0915d90740 4 client.21729 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:22:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.920+0000 7f0915d90740 1 -- 172.21.15.70:0/486373238 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560315df9b80 con 0x560315ea8fa0 2024-06-04T05:22:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.921+0000 7f090bfff640 1 --2- 172.21.15.70:0/486373238 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560315ea8fa0 0x560315eab390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.921+0000 7f090bfff640 1 --2- 172.21.15.70:0/486373238 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560315ea8fa0 0x560315eab390 crc :-1 s=READY pgs=515 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f69b609acb5d0ce6 server_cookie=549b5da3f5d6617 in_seq=0 out_seq=0 2024-06-04T05:22:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.921+0000 7f08ebfff640 10 client.21729 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:05.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.922+0000 7f08ebfff640 1 -- 172.21.15.70:0/486373238 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560315df9b80 con 0x560315ea8fa0 2024-06-04T05:22:05.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.922+0000 7f08ebfff640 10 client.21729 handle_command_reply: tid=0 2024-06-04T05:22:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.922+0000 7f0915d90740 1 client.21729 shutdown 2024-06-04T05:22:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.922+0000 7f0915d90740 1 -- 172.21.15.70:0/486373238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e86140 msgr2=0x560315e8dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.922+0000 7f0915d90740 1 --2- 172.21.15.70:0/486373238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e86140 0x560315e8dea0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f090000bd20 tx=0x7f090001c7e0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.922+0000 7f0915d90740 1 -- 172.21.15.70:0/486373238 shutdown_connections 2024-06-04T05:22:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.922+0000 7f0915d90740 1 --2- 172.21.15.70:0/486373238 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560315ea8fa0 0x560315eab390 crc :-1 s=READY pgs=515 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.922+0000 7f0915d90740 1 --2- 172.21.15.70:0/486373238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e90a00 0x560315e8e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.922+0000 7f0915d90740 1 --2- 172.21.15.70:0/486373238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e86140 0x560315e8dea0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.922+0000 7f0915d90740 1 --2- 172.21.15.70:0/486373238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e85820 0x560315e8d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.922+0000 7f08ebfff640 0 client.21729 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.922+0000 7f0915d90740 1 -- 172.21.15.70:0/486373238 >> 172.21.15.70:0/486373238 conn(0x560315e81d30 msgr2=0x560315e80fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.923+0000 7f0915d90740 1 -- 172.21.15.70:0/486373238 shutdown_connections 2024-06-04T05:22:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.923+0000 7f0915d90740 1 -- 172.21.15.70:0/486373238 wait complete. 2024-06-04T05:22:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.923+0000 7f0915d90740 20 client.21729 trim_cache size 0 max 16384 2024-06-04T05:22:05.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:05 smithi112 ceph-mon[36453]: pgmap v785: 97 pgs: 97 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.4 MiB/s wr, 140 op/s 2024-06-04T05:22:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.950+0000 7f0915d90740 1 Processor -- start 2024-06-04T05:22:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.950+0000 7f0915d90740 1 -- start start 2024-06-04T05:22:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.950+0000 7f0915d90740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e85820 0x560315e881e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.950+0000 7f0915d90740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e86140 0x560315e8e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.950+0000 7f0915d90740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e90a00 0x560315eb0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.950+0000 7f0915d90740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08f000e920 con 0x560315e90a00 2024-06-04T05:22:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.950+0000 7f0915d90740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08f0010460 con 0x560315e86140 2024-06-04T05:22:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.950+0000 7f0915d90740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f08f000eff0 con 0x560315e85820 2024-06-04T05:22:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.950+0000 7f090bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e85820 0x560315e881e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.951+0000 7f0910d39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e86140 0x560315e8e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.951+0000 7f090bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e85820 0x560315e881e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:54260/0 (socket says 172.21.15.70:54260) 2024-06-04T05:22:05.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.951+0000 7f0910d39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e86140 0x560315e8e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55226/0 (socket says 172.21.15.70:55226) 2024-06-04T05:22:05.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.951+0000 7f0910d39640 1 -- 172.21.15.70:0/3080559706 learned_addr learned my addr 172.21.15.70:0/3080559706 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:05.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.951+0000 7f090bfff640 1 --2- 172.21.15.70:0/3080559706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e90a00 0x560315eb0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.951+0000 7f0910d39640 1 -- 172.21.15.70:0/3080559706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e85820 msgr2=0x560315e881e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:05.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.951+0000 7f0910d39640 1 --2- 172.21.15.70:0/3080559706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e85820 0x560315e881e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.951+0000 7f0910d39640 1 -- 172.21.15.70:0/3080559706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e90a00 msgr2=0x560315eb0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:05.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.951+0000 7f0910d39640 1 --2- 172.21.15.70:0/3080559706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e90a00 0x560315eb0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.951+0000 7f0910d39640 1 -- 172.21.15.70:0/3080559706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560315e8beb0 con 0x560315e86140 2024-06-04T05:22:05.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.951+0000 7f090bfff640 1 --2- 172.21.15.70:0/3080559706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e85820 0x560315e881e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:05.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.951+0000 7f0910d39640 1 --2- 172.21.15.70:0/3080559706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e86140 0x560315e8e030 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0900010ff0 tx=0x7f0900010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d146bdafb18efd8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:05.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.951+0000 7f08f67fc640 1 -- 172.21.15.70:0/3080559706 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090002a070 con 0x560315e86140 2024-06-04T05:22:05.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.951+0000 7f08f67fc640 1 -- 172.21.15.70:0/3080559706 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0900011c60 con 0x560315e86140 2024-06-04T05:22:05.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.951+0000 7f08f67fc640 1 -- 172.21.15.70:0/3080559706 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09000035f0 con 0x560315e86140 2024-06-04T05:22:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.952+0000 7f0915d90740 1 -- 172.21.15.70:0/3080559706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e86140 msgr2=0x560315e8e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.952+0000 7f0915d90740 1 --2- 172.21.15.70:0/3080559706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e86140 0x560315e8e030 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0900010ff0 tx=0x7f0900010860 comp rx=0 tx=0).stop 2024-06-04T05:22:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.952+0000 7f0915d90740 1 -- 172.21.15.70:0/3080559706 shutdown_connections 2024-06-04T05:22:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.952+0000 7f0915d90740 1 --2- 172.21.15.70:0/3080559706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e86140 0x560315e8e030 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.952+0000 7f0915d90740 1 --2- 172.21.15.70:0/3080559706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e90a00 0x560315eb0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.952+0000 7f0915d90740 1 --2- 172.21.15.70:0/3080559706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e85820 0x560315e881e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.952+0000 7f0915d90740 1 -- 172.21.15.70:0/3080559706 >> 172.21.15.70:0/3080559706 conn(0x560315e81d30 msgr2=0x560315e8d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.953+0000 7f0915d90740 1 -- 172.21.15.70:0/3080559706 shutdown_connections 2024-06-04T05:22:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.953+0000 7f0915d90740 1 -- 172.21.15.70:0/3080559706 wait complete. 2024-06-04T05:22:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.953+0000 7f0915d90740 1 -- start start 2024-06-04T05:22:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.953+0000 7f0915d90740 1 Processor -- start 2024-06-04T05:22:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.953+0000 7f0915d90740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e85820 0x560315e9d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.953+0000 7f0915d90740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e86140 0x560315e9dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.953+0000 7f0915d90740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e90a00 0x560315f0c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.953+0000 7f0915d90740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08f000fd90 con 0x560315e85820 2024-06-04T05:22:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.953+0000 7f0915d90740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08f000f6c0 con 0x560315e90a00 2024-06-04T05:22:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.953+0000 7f0915d90740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f08f000c920 con 0x560315e86140 2024-06-04T05:22:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.954+0000 7f0915d90740 10 client.0 _ll_register_callbacks cb 0x7f08f80117f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.954+0000 7f090bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e85820 0x560315e9d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.954+0000 7f0910d39640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e86140 0x560315e9dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.954+0000 7f0910d39640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e86140 0x560315e9dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:54266/0 (socket says 172.21.15.70:54266) 2024-06-04T05:22:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.954+0000 7f0910d39640 1 -- 172.21.15.70:0/393697621 learned_addr learned my addr 172.21.15.70:0/393697621 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.954+0000 7f090bfff640 1 --2- 172.21.15.70:0/393697621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e90a00 0x560315f0c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.954+0000 7f0910d39640 1 -- 172.21.15.70:0/393697621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e90a00 msgr2=0x560315f0c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.954+0000 7f0910d39640 1 --2- 172.21.15.70:0/393697621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e90a00 0x560315f0c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.954+0000 7f0910d39640 1 -- 172.21.15.70:0/393697621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e85820 msgr2=0x560315e9d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.954+0000 7f0910d39640 1 --2- 172.21.15.70:0/393697621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e85820 0x560315e9d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.954+0000 7f0910d39640 1 -- 172.21.15.70:0/393697621 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560315f0ceb0 con 0x560315e86140 2024-06-04T05:22:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.954+0000 7f090bfff640 1 --2- 172.21.15.70:0/393697621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e85820 0x560315e9d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.954+0000 7f0910d39640 1 --2- 172.21.15.70:0/393697621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e86140 0x560315e9dc00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f090000bec0 tx=0x7f0900004410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcd47d72d1a1f1b8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.954+0000 7f08ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:22:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.955+0000 7f08ebfff640 1 -- 172.21.15.70:0/393697621 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090002a070 con 0x560315e86140 2024-06-04T05:22:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.955+0000 7f08ebfff640 1 -- 172.21.15.70:0/393697621 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8615+0+0 (secure 0 0 0) 0x7f09000035f0 con 0x560315e86140 2024-06-04T05:22:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.955+0000 7f0915d90740 1 -- 172.21.15.70:0/393697621 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0900011c60 con 0x560315e86140 2024-06-04T05:22:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.955+0000 7f08ebfff640 1 -- 172.21.15.70:0/393697621 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f0900010d60 con 0x560315e86140 2024-06-04T05:22:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.955+0000 7f0915d90740 10 client.30472 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.955+0000 7f0915d90740 1 -- 172.21.15.70:0/393697621 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x560315f14250 con 0x560315e86140 2024-06-04T05:22:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.955+0000 7f08ebfff640 1 -- 172.21.15.70:0/393697621 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0900011c60 con 0x560315e86140 2024-06-04T05:22:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.955+0000 7f08ebfff640 1 -- 172.21.15.70:0/393697621 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f09000414e0 con 0x560315e86140 2024-06-04T05:22:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.955+0000 7f0915d90740 10 client.30472 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.955+0000 7f0915d90740 10 client.30472 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.955+0000 7f0915d90740 20 client.30472 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.955+0000 7f0915d90740 1 --2- 172.21.15.70:0/393697621 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560315f14960 0x560315f16d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.955+0000 7f0915d90740 4 client.30472 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:22:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.955+0000 7f0915d90740 1 -- 172.21.15.70:0/393697621 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560315df3360 con 0x560315f14960 2024-06-04T05:22:05.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.956+0000 7f090bfff640 1 --2- 172.21.15.70:0/393697621 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560315f14960 0x560315f16d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:05.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.956+0000 7f090bfff640 1 --2- 172.21.15.70:0/393697621 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560315f14960 0x560315f16d80 crc :-1 s=READY pgs=516 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b122bdcdfb48b5e server_cookie=b4da531affa59d81 in_seq=0 out_seq=0 2024-06-04T05:22:05.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:05.956+0000 7f08ebfff640 10 client.30472 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.059+0000 7f08ebfff640 1 -- 172.21.15.70:0/393697621 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x560315df3360 con 0x560315f14960 2024-06-04T05:22:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.059+0000 7f08ebfff640 10 client.30472 handle_command_reply: tid=0 2024-06-04T05:22:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.059+0000 7f0915d90740 1 client.30472 shutdown 2024-06-04T05:22:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.059+0000 7f0915d90740 1 -- 172.21.15.70:0/393697621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e86140 msgr2=0x560315e9dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.059+0000 7f0915d90740 1 --2- 172.21.15.70:0/393697621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e86140 0x560315e9dc00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f090000bec0 tx=0x7f0900004410 comp rx=0 tx=0).stop 2024-06-04T05:22:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.059+0000 7f0915d90740 1 -- 172.21.15.70:0/393697621 shutdown_connections 2024-06-04T05:22:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.059+0000 7f0915d90740 1 --2- 172.21.15.70:0/393697621 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560315f14960 0x560315f16d80 crc :-1 s=READY pgs=516 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.059+0000 7f0915d90740 1 --2- 172.21.15.70:0/393697621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560315e90a00 0x560315f0c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.060+0000 7f0915d90740 1 --2- 172.21.15.70:0/393697621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560315e86140 0x560315e9dc00 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.060+0000 7f0915d90740 1 --2- 172.21.15.70:0/393697621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560315e85820 0x560315e9d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.060+0000 7f0915d90740 1 -- 172.21.15.70:0/393697621 >> 172.21.15.70:0/393697621 conn(0x560315e81d30 msgr2=0x560315e8e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.060+0000 7f08ebfff640 0 client.30472 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:06.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.060+0000 7f0915d90740 1 -- 172.21.15.70:0/393697621 shutdown_connections 2024-06-04T05:22:06.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.060+0000 7f0915d90740 1 -- 172.21.15.70:0/393697621 wait complete. 2024-06-04T05:22:06.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.060+0000 7f0915d90740 20 client.30472 trim_cache size 0 max 16384 2024-06-04T05:22:06.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:22:06.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:22:06.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:22:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.061+0000 7f0912f9d640 1 -- 172.21.15.70:0/549064416 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08fc0b2a10 msgr2=0x7f08fc0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.061+0000 7f0912f9d640 1 --2- 172.21.15.70:0/549064416 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08fc0b2a10 0x7f08fc0b4ed0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0904100dc0 tx=0x7f0904106b70 comp rx=0 tx=0).stop 2024-06-04T05:22:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.061+0000 7f0912f9d640 1 -- 172.21.15.70:0/549064416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c107970 msgr2=0x7f090c19fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.061+0000 7f0912f9d640 1 --2- 172.21.15.70:0/549064416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c107970 0x7f090c19fc60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0904009ca0 tx=0x7f0904003ba0 comp rx=0 tx=0).stop 2024-06-04T05:22:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.061+0000 7f0912f9d640 1 -- 172.21.15.70:0/549064416 shutdown_connections 2024-06-04T05:22:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.062+0000 7f0912f9d640 1 --2- 172.21.15.70:0/549064416 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f08fc0b2a10 0x7f08fc0b4ed0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.062+0000 7f0912f9d640 1 --2- 172.21.15.70:0/549064416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c107970 0x7f090c19fc60 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.062+0000 7f0912f9d640 1 --2- 172.21.15.70:0/549064416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f090c102370 0x7f090c10e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.062+0000 7f0912f9d640 1 --2- 172.21.15.70:0/549064416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f090c1019b0 0x7f090c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:06.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.062+0000 7f0912f9d640 1 -- 172.21.15.70:0/549064416 >> 172.21.15.70:0/549064416 conn(0x7f090c078aa0 msgr2=0x7f090c112100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:06.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.062+0000 7f0912f9d640 1 -- 172.21.15.70:0/549064416 shutdown_connections 2024-06-04T05:22:06.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:06.062+0000 7f0912f9d640 1 -- 172.21.15.70:0/549064416 wait complete. 2024-06-04T05:22:06.084 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:22:06.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:06 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:22:06.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84114 6.78 MB/sec execute 538 sec latency 99.089 ms 2024-06-04T05:22:06.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:06 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:22:06.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:06 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:22:07.040 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:22:07.040 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:22:07.470 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:22:07.707 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:07 smithi070 ceph-mon[32631]: pgmap v786: 97 pgs: 97 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.4 MiB/s wr, 294 op/s 2024-06-04T05:22:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:07 smithi098 ceph-mon[35569]: pgmap v786: 97 pgs: 97 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.4 MiB/s wr, 294 op/s 2024-06-04T05:22:07.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84284 6.78 MB/sec execute 539 sec latency 106.336 ms 2024-06-04T05:22:07.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:07 smithi112 ceph-mon[36453]: pgmap v786: 97 pgs: 97 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.4 MiB/s wr, 294 op/s 2024-06-04T05:22:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.327+0000 7f6d842d3700 1 -- 172.21.15.70:0/3187109347 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d7c103890 msgr2=0x7f6d7c10fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.327+0000 7f6d842d3700 1 --2- 172.21.15.70:0/3187109347 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d7c103890 0x7f6d7c10fab0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c009990 tx=0x7f6d6c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.327+0000 7f6d842d3700 1 -- 172.21.15.70:0/3187109347 shutdown_connections 2024-06-04T05:22:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.327+0000 7f6d842d3700 1 --2- 172.21.15.70:0/3187109347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d7c10fff0 0x7f6d7c1123e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.327+0000 7f6d842d3700 1 --2- 172.21.15.70:0/3187109347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d7c102f70 0x7f6d7c103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.327+0000 7f6d842d3700 1 --2- 172.21.15.70:0/3187109347 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d7c103890 0x7f6d7c10fab0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.327+0000 7f6d842d3700 1 -- 172.21.15.70:0/3187109347 >> 172.21.15.70:0/3187109347 conn(0x7f6d7c0793e0 msgr2=0x7f6d7c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.327+0000 7f6d842d3700 1 -- 172.21.15.70:0/3187109347 shutdown_connections 2024-06-04T05:22:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.328+0000 7f6d842d3700 1 -- 172.21.15.70:0/3187109347 wait complete. 2024-06-04T05:22:08.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.329+0000 7f6d842d3700 1 Processor -- start 2024-06-04T05:22:08.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.329+0000 7f6d842d3700 1 -- start start 2024-06-04T05:22:08.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.329+0000 7f6d842d3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d7c102f70 0x7f6d7c19bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.329+0000 7f6d842d3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d7c103890 0x7f6d7c19c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.329+0000 7f6d842d3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d7c10fff0 0x7f6d7c1a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.329+0000 7f6d842d3700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7c115830 con 0x7f6d7c103890 2024-06-04T05:22:08.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.329+0000 7f6d842d3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7c1156b0 con 0x7f6d7c102f70 2024-06-04T05:22:08.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.329+0000 7f6d842d3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7c1159b0 con 0x7f6d7c10fff0 2024-06-04T05:22:08.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.329+0000 7f6d8186e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d7c103890 0x7f6d7c19c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.329+0000 7f6d8206f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d7c102f70 0x7f6d7c19bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.329+0000 7f6d8206f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d7c102f70 0x7f6d7c19bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:50746/0 (socket says 172.21.15.70:50746) 2024-06-04T05:22:08.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.329+0000 7f6d8186e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d7c103890 0x7f6d7c19c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33440/0 (socket says 172.21.15.70:33440) 2024-06-04T05:22:08.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.329+0000 7f6d8206f700 1 -- 172.21.15.70:0/3075760194 learned_addr learned my addr 172.21.15.70:0/3075760194 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:08.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.329+0000 7f6d82870700 1 --2- 172.21.15.70:0/3075760194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d7c10fff0 0x7f6d7c1a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.330+0000 7f6d8186e700 1 -- 172.21.15.70:0/3075760194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d7c10fff0 msgr2=0x7f6d7c1a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:08.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.330+0000 7f6d8186e700 1 --2- 172.21.15.70:0/3075760194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d7c10fff0 0x7f6d7c1a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.330+0000 7f6d8186e700 1 -- 172.21.15.70:0/3075760194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d7c102f70 msgr2=0x7f6d7c19bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:08.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.330+0000 7f6d8186e700 1 --2- 172.21.15.70:0/3075760194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d7c102f70 0x7f6d7c19bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.330+0000 7f6d8186e700 1 -- 172.21.15.70:0/3075760194 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d7c1a0df0 con 0x7f6d7c103890 2024-06-04T05:22:08.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.330+0000 7f6d8206f700 1 --2- 172.21.15.70:0/3075760194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d7c102f70 0x7f6d7c19bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:08.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.330+0000 7f6d82870700 1 --2- 172.21.15.70:0/3075760194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d7c10fff0 0x7f6d7c1a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:08.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.330+0000 7f6d8186e700 1 --2- 172.21.15.70:0/3075760194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d7c103890 0x7f6d7c19c320 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c009990 tx=0x7f6d6c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:08.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.330+0000 7f6d737fe700 1 -- 172.21.15.70:0/3075760194 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c027070 con 0x7f6d7c103890 2024-06-04T05:22:08.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.330+0000 7f6d737fe700 1 -- 172.21.15.70:0/3075760194 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6d6c022470 con 0x7f6d7c103890 2024-06-04T05:22:08.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.330+0000 7f6d842d3700 1 -- 172.21.15.70:0/3075760194 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d7c1a1080 con 0x7f6d7c103890 2024-06-04T05:22:08.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.330+0000 7f6d737fe700 1 -- 172.21.15.70:0/3075760194 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c017650 con 0x7f6d7c103890 2024-06-04T05:22:08.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.331+0000 7f6d842d3700 1 -- 172.21.15.70:0/3075760194 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d7c1a8960 con 0x7f6d7c103890 2024-06-04T05:22:08.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.333+0000 7f6d737fe700 1 -- 172.21.15.70:0/3075760194 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f6d6c02a070 con 0x7f6d7c103890 2024-06-04T05:22:08.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.334+0000 7f6d737fe700 1 --2- 172.21.15.70:0/3075760194 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6d680b2b20 0x7f6d680b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.334+0000 7f6d737fe700 1 -- 172.21.15.70:0/3075760194 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f6d6c0e3330 con 0x7f6d7c103890 2024-06-04T05:22:08.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.334+0000 7f6d8206f700 1 --2- 172.21.15.70:0/3075760194 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6d680b2b20 0x7f6d680b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.335+0000 7f6d8206f700 1 --2- 172.21.15.70:0/3075760194 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6d680b2b20 0x7f6d680b4fe0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6d78011550 tx=0x7f6d78009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:08.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.335+0000 7f6d8eb68b80 1 Processor -- start 2024-06-04T05:22:08.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d8eb68b80 1 -- start start 2024-06-04T05:22:08.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d8eb68b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d877482340 0x55d877482720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d8eb68b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d87747c970 0x55d87747cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d8eb68b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 0x55d877489170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d8eb68b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d8773dbe20 con 0x55d87747c970 2024-06-04T05:22:08.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d8206f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d877482340 0x55d877482720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d8eb68b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d8773cb9c0 con 0x55d87747d330 2024-06-04T05:22:08.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d8eb68b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d8773cfa00 con 0x55d877482340 2024-06-04T05:22:08.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d8186e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d87747c970 0x55d87747cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d8206f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d877482340 0x55d877482720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:54966/0 (socket says 172.21.15.70:54966) 2024-06-04T05:22:08.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d8206f700 1 -- 172.21.15.70:0/1202930391 learned_addr learned my addr 172.21.15.70:0/1202930391 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:08.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d82870700 1 --2- 172.21.15.70:0/1202930391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 0x55d877489170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d8206f700 1 -- 172.21.15.70:0/1202930391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 msgr2=0x55d877489170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:08.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d8206f700 1 --2- 172.21.15.70:0/1202930391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 0x55d877489170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d8206f700 1 -- 172.21.15.70:0/1202930391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d87747c970 msgr2=0x55d87747cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:08.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d8206f700 1 --2- 172.21.15.70:0/1202930391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d87747c970 0x55d87747cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d8206f700 1 -- 172.21.15.70:0/1202930391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d87731ba60 con 0x55d877482340 2024-06-04T05:22:08.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.336+0000 7f6d8186e700 1 --2- 172.21.15.70:0/1202930391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d87747c970 0x55d87747cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:08.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.337+0000 7f6d82870700 1 --2- 172.21.15.70:0/1202930391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 0x55d877489170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:22:08.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.337+0000 7f6d8206f700 1 --2- 172.21.15.70:0/1202930391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d877482340 0x55d877482720 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009880 tx=0x7f6d78014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a4b0a41bff282c4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:08.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.337+0000 7f6d70d70700 1 -- 172.21.15.70:0/1202930391 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78026630 con 0x55d877482340 2024-06-04T05:22:08.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.337+0000 7f6d70d70700 1 -- 172.21.15.70:0/1202930391 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6d7800ee40 con 0x55d877482340 2024-06-04T05:22:08.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.337+0000 7f6d70d70700 1 -- 172.21.15.70:0/1202930391 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7801d7d0 con 0x55d877482340 2024-06-04T05:22:08.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.338+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1202930391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d877482340 msgr2=0x55d877482720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:08.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.338+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/1202930391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d877482340 0x55d877482720 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009880 tx=0x7f6d78014000 comp rx=0 tx=0).stop 2024-06-04T05:22:08.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.338+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1202930391 shutdown_connections 2024-06-04T05:22:08.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.338+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/1202930391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 0x55d877489170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.338+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/1202930391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d877482340 0x55d877482720 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.338+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/1202930391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d87747c970 0x55d87747cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.338+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1202930391 >> 172.21.15.70:0/1202930391 conn(0x55d8773aa210 msgr2=0x55d877477280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:08.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.338+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1202930391 shutdown_connections 2024-06-04T05:22:08.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.338+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1202930391 wait complete. 2024-06-04T05:22:08.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.339+0000 7f6d8eb68b80 1 -- start start 2024-06-04T05:22:08.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.339+0000 7f6d8eb68b80 1 Processor -- start 2024-06-04T05:22:08.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.339+0000 7f6d8eb68b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d87747c970 0x55d877488eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.339+0000 7f6d8eb68b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747d330 0x55d8774861b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.340+0000 7f6d8206f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d87747c970 0x55d877488eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.340+0000 7f6d8206f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d87747c970 0x55d877488eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33452/0 (socket says 172.21.15.70:33452) 2024-06-04T05:22:08.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.340+0000 7f6d8eb68b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d877482340 0x55d8774866f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.340+0000 7f6d8206f700 1 -- 172.21.15.70:0/1818667636 learned_addr learned my addr 172.21.15.70:0/1818667636 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:08.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.340+0000 7f6d8186e700 1 --2- 172.21.15.70:0/1818667636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747d330 0x55d8774861b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.340+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1818667636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d8772d2130 con 0x55d87747c970 2024-06-04T05:22:08.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.340+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1818667636 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d87736d950 con 0x55d877482340 2024-06-04T05:22:08.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.340+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1818667636 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d877489770 con 0x55d87747d330 2024-06-04T05:22:08.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.340+0000 7f6d82870700 1 --2- 172.21.15.70:0/1818667636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d877482340 0x55d8774866f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.340+0000 7f6d8eb68b80 10 client.0 _ll_register_callbacks cb 0x55d877366ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:08.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.340+0000 7f6d8206f700 1 -- 172.21.15.70:0/1818667636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747d330 msgr2=0x55d8774861b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:08.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.340+0000 7f6d8206f700 1 --2- 172.21.15.70:0/1818667636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747d330 0x55d8774861b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.340+0000 7f6d8206f700 1 -- 172.21.15.70:0/1818667636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d877482340 msgr2=0x55d8774866f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:08.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.340+0000 7f6d8206f700 1 --2- 172.21.15.70:0/1818667636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d877482340 0x55d8774866f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.340+0000 7f6d8206f700 1 -- 172.21.15.70:0/1818667636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d87736a820 con 0x55d87747c970 2024-06-04T05:22:08.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.340+0000 7f6d8206f700 1 --2- 172.21.15.70:0/1818667636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d87747c970 0x55d877488eb0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f6d7801d010 tx=0x7f6d78005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c051482732b432 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:08.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.341+0000 7f6d65ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:22:08.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.341+0000 7f6d65ffb700 1 -- 172.21.15.70:0/1818667636 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78034060 con 0x55d87747c970 2024-06-04T05:22:08.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.341+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1818667636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d877448b70 con 0x55d87747c970 2024-06-04T05:22:08.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.341+0000 7f6d65ffb700 1 -- 172.21.15.70:0/1818667636 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f6d78005530 con 0x55d87747c970 2024-06-04T05:22:08.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.341+0000 7f6d65ffb700 1 -- 172.21.15.70:0/1818667636 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f6d7801c410 con 0x55d87747c970 2024-06-04T05:22:08.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.341+0000 7f6d8eb68b80 10 client.21747 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:08.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.341+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1818667636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55d877493f00 con 0x55d87747c970 2024-06-04T05:22:08.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.341+0000 7f6d65ffb700 1 -- 172.21.15.70:0/1818667636 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6d7801ca40 con 0x55d87747c970 2024-06-04T05:22:08.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.342+0000 7f6d65ffb700 1 -- 172.21.15.70:0/1818667636 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f6d7802f480 con 0x55d87747c970 2024-06-04T05:22:08.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.342+0000 7f6d8eb68b80 10 client.21747 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:08.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.342+0000 7f6d8eb68b80 10 client.21747 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:22:08.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.342+0000 7f6d8eb68b80 20 client.21747 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:08.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.342+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/1818667636 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d8774963d0 0x55d8774987c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.342+0000 7f6d8eb68b80 4 client.21747 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:22:08.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.342+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1818667636 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d877448b70 con 0x55d8774963d0 2024-06-04T05:22:08.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.342+0000 7f6d8186e700 1 --2- 172.21.15.70:0/1818667636 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d8774963d0 0x55d8774987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.342+0000 7f6d65ffb700 10 client.21747 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:08.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.342+0000 7f6d8186e700 1 --2- 172.21.15.70:0/1818667636 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d8774963d0 0x55d8774987c0 crc :-1 s=READY pgs=517 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7021d061b769dbf2 server_cookie=2d9299c1b10f287a in_seq=0 out_seq=0 2024-06-04T05:22:08.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.343+0000 7f6d65ffb700 1 -- 172.21.15.70:0/1818667636 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d877448b70 con 0x55d8774963d0 2024-06-04T05:22:08.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.343+0000 7f6d65ffb700 10 client.21747 handle_command_reply: tid=0 2024-06-04T05:22:08.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.343+0000 7f6d8eb68b80 1 client.21747 shutdown 2024-06-04T05:22:08.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.344+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1818667636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d87747c970 msgr2=0x55d877488eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:08.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.344+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/1818667636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d87747c970 0x55d877488eb0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f6d7801d010 tx=0x7f6d78005040 comp rx=0 tx=0).stop 2024-06-04T05:22:08.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.344+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1818667636 shutdown_connections 2024-06-04T05:22:08.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.344+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/1818667636 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d8774963d0 0x55d8774987c0 crc :-1 s=READY pgs=517 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.344+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/1818667636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d877482340 0x55d8774866f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.344+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/1818667636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d87747c970 0x55d877488eb0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.344+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/1818667636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747d330 0x55d8774861b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.344+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1818667636 >> 172.21.15.70:0/1818667636 conn(0x55d8773aa210 msgr2=0x55d877471d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:08.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.344+0000 7f6d65ffb700 0 client.21747 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:08.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.344+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1818667636 shutdown_connections 2024-06-04T05:22:08.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.344+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1818667636 wait complete. 2024-06-04T05:22:08.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.344+0000 7f6d8eb68b80 20 client.21747 trim_cache size 0 max 16384 2024-06-04T05:22:08.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.375+0000 7f6d8eb68b80 1 Processor -- start 2024-06-04T05:22:08.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.375+0000 7f6d8eb68b80 1 -- start start 2024-06-04T05:22:08.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.375+0000 7f6d8eb68b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747c970 0x55d877485170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.375+0000 7f6d8eb68b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 0x55d87747bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.376+0000 7f6d8eb68b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d877482340 0x55d8774889f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.376+0000 7f6d8eb68b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7c1abb60 con 0x55d877482340 2024-06-04T05:22:08.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.376+0000 7f6d8eb68b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7c1ad6a0 con 0x55d87747d330 2024-06-04T05:22:08.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.376+0000 7f6d8eb68b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7c1ac230 con 0x55d87747c970 2024-06-04T05:22:08.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.376+0000 7f6d8206f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747c970 0x55d877485170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.376+0000 7f6d8186e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 0x55d87747bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.376+0000 7f6d8186e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 0x55d87747bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:50780/0 (socket says 172.21.15.70:50780) 2024-06-04T05:22:08.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.376+0000 7f6d8186e700 1 -- 172.21.15.70:0/3163823003 learned_addr learned my addr 172.21.15.70:0/3163823003 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:08.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.376+0000 7f6d82870700 1 --2- 172.21.15.70:0/3163823003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d877482340 0x55d8774889f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.376+0000 7f6d8186e700 1 -- 172.21.15.70:0/3163823003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747c970 msgr2=0x55d877485170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:08.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.376+0000 7f6d8186e700 1 --2- 172.21.15.70:0/3163823003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747c970 0x55d877485170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.376+0000 7f6d8186e700 1 -- 172.21.15.70:0/3163823003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d877482340 msgr2=0x55d8774889f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:08.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.376+0000 7f6d8186e700 1 --2- 172.21.15.70:0/3163823003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d877482340 0x55d8774889f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.376+0000 7f6d8186e700 1 -- 172.21.15.70:0/3163823003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d877484720 con 0x55d87747d330 2024-06-04T05:22:08.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.376+0000 7f6d8206f700 1 --2- 172.21.15.70:0/3163823003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747c970 0x55d877485170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:08.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.376+0000 7f6d8186e700 1 --2- 172.21.15.70:0/3163823003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 0x55d87747bfd0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c0a6800 tx=0x7f6d6c005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f73a4778498bdab server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:08.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.377+0000 7f6d67fff700 1 -- 172.21.15.70:0/3163823003 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c0aae80 con 0x55d87747d330 2024-06-04T05:22:08.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.377+0000 7f6d67fff700 1 -- 172.21.15.70:0/3163823003 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6d6c0aa780 con 0x55d87747d330 2024-06-04T05:22:08.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.377+0000 7f6d67fff700 1 -- 172.21.15.70:0/3163823003 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c0aa9e0 con 0x55d87747d330 2024-06-04T05:22:08.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.378+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/3163823003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 msgr2=0x55d87747bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:08.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.378+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/3163823003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 0x55d87747bfd0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c0a6800 tx=0x7f6d6c005040 comp rx=0 tx=0).stop 2024-06-04T05:22:08.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.378+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/3163823003 shutdown_connections 2024-06-04T05:22:08.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.378+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/3163823003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d877482340 0x55d8774889f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.378+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/3163823003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747c970 0x55d877485170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.378+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/3163823003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 0x55d87747bfd0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.378+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/3163823003 >> 172.21.15.70:0/3163823003 conn(0x55d8773aa210 msgr2=0x55d877471d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:08.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.378+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/3163823003 shutdown_connections 2024-06-04T05:22:08.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.378+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/3163823003 wait complete. 2024-06-04T05:22:08.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.378+0000 7f6d8eb68b80 1 -- start start 2024-06-04T05:22:08.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.378+0000 7f6d8eb68b80 1 Processor -- start 2024-06-04T05:22:08.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.379+0000 7f6d8eb68b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747c970 0x55d877499540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.379+0000 7f6d8eb68b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 0x55d877499a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.379+0000 7f6d8eb68b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d877482340 0x55d8774ffc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.379+0000 7f6d8206f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747c970 0x55d877499540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.379+0000 7f6d8206f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747c970 0x55d877499540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:54996/0 (socket says 172.21.15.70:54996) 2024-06-04T05:22:08.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.379+0000 7f6d8eb68b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7c1acfd0 con 0x55d877482340 2024-06-04T05:22:08.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.379+0000 7f6d8eb68b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7c1ac900 con 0x55d87747d330 2024-06-04T05:22:08.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.379+0000 7f6d8eb68b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7c108830 con 0x55d87747c970 2024-06-04T05:22:08.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.379+0000 7f6d8eb68b80 10 client.0 _ll_register_callbacks cb 0x55d877366ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:08.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.379+0000 7f6d8186e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 0x55d877499a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.379+0000 7f6d8206f700 1 -- 172.21.15.70:0/1941000311 learned_addr learned my addr 172.21.15.70:0/1941000311 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:08.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.379+0000 7f6d82870700 1 --2- 172.21.15.70:0/1941000311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d877482340 0x55d8774ffc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.380+0000 7f6d8206f700 1 -- 172.21.15.70:0/1941000311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 msgr2=0x55d877499a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:08.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.380+0000 7f6d8206f700 1 --2- 172.21.15.70:0/1941000311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 0x55d877499a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.380+0000 7f6d8206f700 1 -- 172.21.15.70:0/1941000311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d877482340 msgr2=0x55d8774ffc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:08.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.380+0000 7f6d8206f700 1 --2- 172.21.15.70:0/1941000311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d877482340 0x55d8774ffc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.380+0000 7f6d8206f700 1 -- 172.21.15.70:0/1941000311 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d877500150 con 0x55d87747c970 2024-06-04T05:22:08.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.380+0000 7f6d8206f700 1 --2- 172.21.15.70:0/1941000311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747c970 0x55d877499540 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6d78006010 tx=0x7f6d78011de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0aee000cf30a9d6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:08.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.380+0000 7f6d65ffb700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:22:08.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.380+0000 7f6d65ffb700 1 -- 172.21.15.70:0/1941000311 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78034040 con 0x55d87747c970 2024-06-04T05:22:08.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.380+0000 7f6d65ffb700 1 -- 172.21.15.70:0/1941000311 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f6d7802fac0 con 0x55d87747c970 2024-06-04T05:22:08.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.380+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1941000311 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d6c0aa780 con 0x55d87747c970 2024-06-04T05:22:08.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.381+0000 7f6d8186e700 1 --2- 172.21.15.70:0/1941000311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 0x55d877499a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:08.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.381+0000 7f6d65ffb700 1 -- 172.21.15.70:0/1941000311 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f6d7802f690 con 0x55d87747c970 2024-06-04T05:22:08.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.381+0000 7f6d8eb68b80 10 client.30490 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:08.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.381+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1941000311 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55d8775012e0 con 0x55d87747c970 2024-06-04T05:22:08.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.381+0000 7f6d65ffb700 1 -- 172.21.15.70:0/1941000311 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6d7802fd00 con 0x55d87747c970 2024-06-04T05:22:08.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.382+0000 7f6d65ffb700 1 -- 172.21.15.70:0/1941000311 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f6d78004920 con 0x55d87747c970 2024-06-04T05:22:08.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.382+0000 7f6d8eb68b80 10 client.30490 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:08.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.382+0000 7f6d8eb68b80 10 client.30490 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:22:08.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.382+0000 7f6d8eb68b80 20 client.30490 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:08.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.382+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/1941000311 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d877501940 0x55d877503d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:08.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.382+0000 7f6d8eb68b80 4 client.30490 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-06-04T05:22:08.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.382+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1941000311 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x7f6d6c0aa780 con 0x55d877501940 2024-06-04T05:22:08.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.382+0000 7f6d8186e700 1 --2- 172.21.15.70:0/1941000311 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d877501940 0x55d877503d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:08.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.382+0000 7f6d65ffb700 10 client.30490 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:08.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.382+0000 7f6d8186e700 1 --2- 172.21.15.70:0/1941000311 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d877501940 0x55d877503d30 crc :-1 s=READY pgs=518 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e92fefeb6fa49de7 server_cookie=5d0bdc4ff72c052a in_seq=0 out_seq=0 2024-06-04T05:22:08.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84383 6.78 MB/sec execute 540 sec latency 436.127 ms 2024-06-04T05:22:08.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.968+0000 7f6d65ffb700 1 -- 172.21.15.70:0/1941000311 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f6d6c0aa780 con 0x55d877501940 2024-06-04T05:22:08.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.968+0000 7f6d65ffb700 10 client.30490 handle_command_reply: tid=0 2024-06-04T05:22:08.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.968+0000 7f6d8eb68b80 1 client.30490 shutdown 2024-06-04T05:22:08.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.968+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1941000311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747c970 msgr2=0x55d877499540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:08.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.968+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/1941000311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747c970 0x55d877499540 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6d78006010 tx=0x7f6d78011de0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.968+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1941000311 shutdown_connections 2024-06-04T05:22:08.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.968+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/1941000311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d877482340 0x55d8774ffc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.968+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/1941000311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87747c970 0x55d877499540 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.968+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/1941000311 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55d877501940 0x55d877503d30 crc :-1 s=READY pgs=518 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.968+0000 7f6d8eb68b80 1 --2- 172.21.15.70:0/1941000311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d87747d330 0x55d877499a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.968+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1941000311 >> 172.21.15.70:0/1941000311 conn(0x55d8773aa210 msgr2=0x55d87747fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:08.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.968+0000 7f6d65ffb700 0 client.30490 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:08.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.968+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1941000311 shutdown_connections 2024-06-04T05:22:08.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.968+0000 7f6d8eb68b80 1 -- 172.21.15.70:0/1941000311 wait complete. 2024-06-04T05:22:08.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.968+0000 7f6d8eb68b80 20 client.30490 trim_cache size 0 max 16384 2024-06-04T05:22:08.972 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:22:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 67, 2024-06-04T05:22:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:22:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 78, 2024-06-04T05:22:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:22:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "582ae688": { 2024-06-04T05:22:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 78, 2024-06-04T05:22:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:22:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:22:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:22:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:22:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:22:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:22:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 150.0, 2024-06-04T05:22:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:22:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:22:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:22:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:22:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "QUIESCED", 2024-06-04T05:22:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:22:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:22:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:22:08.974 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:22:08.974 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:22:08.974 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:22:08.974 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:22:08.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.969+0000 7f6d842d3700 1 -- 172.21.15.70:0/3075760194 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6d680b2b20 msgr2=0x7f6d680b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:08.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.969+0000 7f6d842d3700 1 --2- 172.21.15.70:0/3075760194 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6d680b2b20 0x7f6d680b4fe0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6d78011550 tx=0x7f6d78009260 comp rx=0 tx=0).stop 2024-06-04T05:22:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.969+0000 7f6d842d3700 1 -- 172.21.15.70:0/3075760194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d7c103890 msgr2=0x7f6d7c19c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.969+0000 7f6d842d3700 1 --2- 172.21.15.70:0/3075760194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d7c103890 0x7f6d7c19c320 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c009990 tx=0x7f6d6c0057c0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.969+0000 7f6d842d3700 1 -- 172.21.15.70:0/3075760194 shutdown_connections 2024-06-04T05:22:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.969+0000 7f6d842d3700 1 --2- 172.21.15.70:0/3075760194 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6d680b2b20 0x7f6d680b4fe0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.969+0000 7f6d842d3700 1 --2- 172.21.15.70:0/3075760194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d7c10fff0 0x7f6d7c1a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.969+0000 7f6d842d3700 1 --2- 172.21.15.70:0/3075760194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d7c102f70 0x7f6d7c19bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.969+0000 7f6d842d3700 1 --2- 172.21.15.70:0/3075760194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d7c103890 0x7f6d7c19c320 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.969+0000 7f6d842d3700 1 -- 172.21.15.70:0/3075760194 >> 172.21.15.70:0/3075760194 conn(0x7f6d7c0793e0 msgr2=0x7f6d7c111680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.969+0000 7f6d842d3700 1 -- 172.21.15.70:0/3075760194 shutdown_connections 2024-06-04T05:22:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:08.969+0000 7f6d842d3700 1 -- 172.21.15.70:0/3075760194 wait complete. 2024-06-04T05:22:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:09 smithi070 ceph-mon[32631]: osdmap e118: 12 total, 12 up, 12 in 2024-06-04T05:22:09.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:09 smithi070 ceph-mon[32631]: pgmap v788: 97 pgs: 97 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.7 MiB/s wr, 368 op/s 2024-06-04T05:22:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:09 smithi112 ceph-mon[36453]: osdmap e118: 12 total, 12 up, 12 in 2024-06-04T05:22:09.436 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:09 smithi112 ceph-mon[36453]: pgmap v788: 97 pgs: 97 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.7 MiB/s wr, 368 op/s 2024-06-04T05:22:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:09 smithi098 ceph-mon[35569]: osdmap e118: 12 total, 12 up, 12 in 2024-06-04T05:22:09.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:09 smithi098 ceph-mon[35569]: pgmap v788: 97 pgs: 97 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.7 MiB/s wr, 368 op/s 2024-06-04T05:22:09.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.77 MB/sec execute 541 sec latency 1431.765 ms 2024-06-04T05:22:10.835 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.76 MB/sec execute 542 sec latency 2431.837 ms 2024-06-04T05:22:11.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:11 smithi070 ceph-mon[32631]: pgmap v789: 97 pgs: 97 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 212 op/s 2024-06-04T05:22:11.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:11 smithi112 ceph-mon[36453]: pgmap v789: 97 pgs: 97 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 212 op/s 2024-06-04T05:22:11.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:11 smithi098 ceph-mon[35569]: pgmap v789: 97 pgs: 97 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 212 op/s 2024-06-04T05:22:11.835 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.74 MB/sec execute 543 sec latency 3431.877 ms 2024-06-04T05:22:12.834 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.73 MB/sec execute 544 sec latency 4431.914 ms 2024-06-04T05:22:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:13 smithi112 ceph-mon[36453]: pgmap v790: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.6 MiB/s wr, 324 op/s 2024-06-04T05:22:13.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:13 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:22:13.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:13 smithi098 ceph-mon[35569]: pgmap v790: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.6 MiB/s wr, 324 op/s 2024-06-04T05:22:13.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:13 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:22:13.835 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.72 MB/sec execute 545 sec latency 5431.965 ms 2024-06-04T05:22:13.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:13 smithi070 ceph-mon[32631]: pgmap v790: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.6 MiB/s wr, 324 op/s 2024-06-04T05:22:13.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:13 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:22:14.399 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '582ae688', quiesce took 16.2 seconds. Will release after: 13.8 2024-06-04T05:22:14.835 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.71 MB/sec execute 546 sec latency 6432.001 ms 2024-06-04T05:22:14.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.904+0000 7fa761afd700 1 -- 172.21.15.70:0/1515434747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa75c102430 msgr2=0x7fa75c1028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:14.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.904+0000 7fa761afd700 1 --2- 172.21.15.70:0/1515434747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa75c102430 0x7fa75c1028b0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fa750009990 tx=0x7fa750009ca0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.904+0000 7fa761afd700 1 -- 172.21.15.70:0/1515434747 shutdown_connections 2024-06-04T05:22:14.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.904+0000 7fa761afd700 1 --2- 172.21.15.70:0/1515434747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa75c102df0 0x7fa75c10f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.904+0000 7fa761afd700 1 --2- 172.21.15.70:0/1515434747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa75c108510 0x7fa75c1088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.904+0000 7fa761afd700 1 --2- 172.21.15.70:0/1515434747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa75c102430 0x7fa75c1028b0 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.904+0000 7fa761afd700 1 -- 172.21.15.70:0/1515434747 >> 172.21.15.70:0/1515434747 conn(0x7fa75c0793b0 msgr2=0x7fa75c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:14.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.904+0000 7fa761afd700 1 -- 172.21.15.70:0/1515434747 shutdown_connections 2024-06-04T05:22:14.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.904+0000 7fa761afd700 1 -- 172.21.15.70:0/1515434747 wait complete. 2024-06-04T05:22:14.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.905+0000 7fa761afd700 1 Processor -- start 2024-06-04T05:22:14.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.905+0000 7fa761afd700 1 -- start start 2024-06-04T05:22:14.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.905+0000 7fa761afd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa75c102430 0x7fa75c19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.905+0000 7fa761afd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa75c102df0 0x7fa75c19fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.905+0000 7fa761afd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa75c108510 0x7fa75c199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.905+0000 7fa761afd700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c112490 con 0x7fa75c108510 2024-06-04T05:22:14.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.905+0000 7fa761afd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c112310 con 0x7fa75c102df0 2024-06-04T05:22:14.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.905+0000 7fa761afd700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c112610 con 0x7fa75c102430 2024-06-04T05:22:14.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.905+0000 7fa75bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa75c108510 0x7fa75c199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:14.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.905+0000 7fa75bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa75c108510 0x7fa75c199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33484/0 (socket says 172.21.15.70:33484) 2024-06-04T05:22:14.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.905+0000 7fa75bfff700 1 -- 172.21.15.70:0/3782305022 learned_addr learned my addr 172.21.15.70:0/3782305022 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:14.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.906+0000 7fa75b7fe700 1 --2- 172.21.15.70:0/3782305022 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa75c102430 0x7fa75c19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:14.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.906+0000 7fa75bfff700 1 -- 172.21.15.70:0/3782305022 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa75c102430 msgr2=0x7fa75c19f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:14.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.906+0000 7fa75bfff700 1 --2- 172.21.15.70:0/3782305022 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa75c102430 0x7fa75c19f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.906+0000 7fa75bfff700 1 -- 172.21.15.70:0/3782305022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa75c102df0 msgr2=0x7fa75c19fcb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-04T05:22:14.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.906+0000 7fa75bfff700 1 --2- 172.21.15.70:0/3782305022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa75c102df0 0x7fa75c19fcb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.906+0000 7fa75bfff700 1 -- 172.21.15.70:0/3782305022 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa75c19a090 con 0x7fa75c108510 2024-06-04T05:22:14.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.906+0000 7fa75bfff700 1 --2- 172.21.15.70:0/3782305022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa75c108510 0x7fa75c199930 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fa74c00d980 tx=0x7fa74c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:14.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.906+0000 7fa758ff9700 1 -- 172.21.15.70:0/3782305022 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74c017070 con 0x7fa75c108510 2024-06-04T05:22:14.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.906+0000 7fa758ff9700 1 -- 172.21.15.70:0/3782305022 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa74c009dd0 con 0x7fa75c108510 2024-06-04T05:22:14.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.906+0000 7fa761afd700 1 -- 172.21.15.70:0/3782305022 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa75c19a380 con 0x7fa75c108510 2024-06-04T05:22:14.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.906+0000 7fa758ff9700 1 -- 172.21.15.70:0/3782305022 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74c005260 con 0x7fa75c108510 2024-06-04T05:22:14.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.906+0000 7fa761afd700 1 -- 172.21.15.70:0/3782305022 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa75c1a6640 con 0x7fa75c108510 2024-06-04T05:22:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.910+0000 7fa76c392b80 1 Processor -- start 2024-06-04T05:22:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.910+0000 7fa758ff9700 1 -- 172.21.15.70:0/3782305022 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fa74c009940 con 0x7fa75c108510 2024-06-04T05:22:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.910+0000 7fa76c392b80 1 -- start start 2024-06-04T05:22:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.910+0000 7fa76c392b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98cdcf0 0x55cec98ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.910+0000 7fa76c392b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98ce6a0 0x55cec98d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.911+0000 7fa76c392b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98d23f0 0x55cec98d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.911+0000 7fa76c392b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cec9828d10 con 0x55cec98d23f0 2024-06-04T05:22:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.911+0000 7fa76c392b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cec98188b0 con 0x55cec98cdcf0 2024-06-04T05:22:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.911+0000 7fa76c392b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cec981c8f0 con 0x55cec98ce6a0 2024-06-04T05:22:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.911+0000 7fa75affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98d23f0 0x55cec98d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.911+0000 7fa75b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98ce6a0 0x55cec98d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.911+0000 7fa75b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98ce6a0 0x55cec98d1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:55028/0 (socket says 172.21.15.70:55028) 2024-06-04T05:22:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.911+0000 7fa75b7fe700 1 -- 172.21.15.70:0/2100391812 learned_addr learned my addr 172.21.15.70:0/2100391812 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.911+0000 7fa75affd700 1 --2- 172.21.15.70:0/2100391812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98cdcf0 0x55cec98ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.911+0000 7fa75b7fe700 1 -- 172.21.15.70:0/2100391812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98cdcf0 msgr2=0x55cec98ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.911+0000 7fa75b7fe700 1 --2- 172.21.15.70:0/2100391812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98cdcf0 0x55cec98ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.911+0000 7fa75b7fe700 1 -- 172.21.15.70:0/2100391812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98d23f0 msgr2=0x55cec98d2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.911+0000 7fa75b7fe700 1 --2- 172.21.15.70:0/2100391812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98d23f0 0x55cec98d2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.911+0000 7fa75b7fe700 1 -- 172.21.15.70:0/2100391812 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cec9768950 con 0x55cec98ce6a0 2024-06-04T05:22:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.911+0000 7fa75affd700 1 --2- 172.21.15.70:0/2100391812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98d23f0 0x55cec98d2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.911+0000 7fa758ff9700 1 --2- 172.21.15.70:0/3782305022 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa7440b2a70 0x7fa7440b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.911+0000 7fa75b7fe700 1 --2- 172.21.15.70:0/2100391812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98ce6a0 0x55cec98d1dd0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa754009d60 tx=0x7fa7540077e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67e0168db252a04 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.912+0000 7fa75affd700 1 --2- 172.21.15.70:0/3782305022 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa7440b2a70 0x7fa7440b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.912+0000 7fa741ffb700 1 -- 172.21.15.70:0/2100391812 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754014610 con 0x55cec98ce6a0 2024-06-04T05:22:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.912+0000 7fa758ff9700 1 -- 172.21.15.70:0/3782305022 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7fa74c0e25d0 con 0x7fa75c108510 2024-06-04T05:22:14.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.912+0000 7fa741ffb700 1 -- 172.21.15.70:0/2100391812 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa754019070 con 0x55cec98ce6a0 2024-06-04T05:22:14.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.912+0000 7fa741ffb700 1 -- 172.21.15.70:0/2100391812 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754013830 con 0x55cec98ce6a0 2024-06-04T05:22:14.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.913+0000 7fa76c392b80 1 -- 172.21.15.70:0/2100391812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98ce6a0 msgr2=0x55cec98d1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:14.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.913+0000 7fa75affd700 1 --2- 172.21.15.70:0/3782305022 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa7440b2a70 0x7fa7440b4f30 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa750005cb0 tx=0x7fa750005c00 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:14.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.913+0000 7fa76c392b80 1 --2- 172.21.15.70:0/2100391812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98ce6a0 0x55cec98d1dd0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa754009d60 tx=0x7fa7540077e0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.913+0000 7fa76c392b80 1 -- 172.21.15.70:0/2100391812 shutdown_connections 2024-06-04T05:22:14.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.913+0000 7fa76c392b80 1 --2- 172.21.15.70:0/2100391812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98d23f0 0x55cec98d2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.913+0000 7fa76c392b80 1 --2- 172.21.15.70:0/2100391812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98cdcf0 0x55cec98ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.913+0000 7fa76c392b80 1 --2- 172.21.15.70:0/2100391812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98ce6a0 0x55cec98d1dd0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.913+0000 7fa76c392b80 1 -- 172.21.15.70:0/2100391812 >> 172.21.15.70:0/2100391812 conn(0x55cec97f7100 msgr2=0x55cec98c3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:14.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.913+0000 7fa76c392b80 1 -- 172.21.15.70:0/2100391812 shutdown_connections 2024-06-04T05:22:14.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.913+0000 7fa76c392b80 1 -- 172.21.15.70:0/2100391812 wait complete. 2024-06-04T05:22:14.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.913+0000 7fa76c392b80 1 -- start start 2024-06-04T05:22:14.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.913+0000 7fa76c392b80 1 Processor -- start 2024-06-04T05:22:14.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa76c392b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98cdcf0 0x55cec98d97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa76c392b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98ce6a0 0x55cec98d6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa76c392b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98d23f0 0x55cec98d7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa76c392b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cec971f020 con 0x55cec98ce6a0 2024-06-04T05:22:14.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa76c392b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cec97ba840 con 0x55cec98d23f0 2024-06-04T05:22:14.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa76c392b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cec98da5b0 con 0x55cec98cdcf0 2024-06-04T05:22:14.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa76c392b80 10 client.0 _ll_register_callbacks cb 0x55cec97b39a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:14.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa75b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98cdcf0 0x55cec98d97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:14.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa75affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98ce6a0 0x55cec98d6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:14.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa75b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98cdcf0 0x55cec98d97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:55036/0 (socket says 172.21.15.70:55036) 2024-06-04T05:22:14.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa75b7fe700 1 -- 172.21.15.70:0/614215631 learned_addr learned my addr 172.21.15.70:0/614215631 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:14.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa75bfff700 1 --2- 172.21.15.70:0/614215631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98d23f0 0x55cec98d7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:14.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa75b7fe700 1 -- 172.21.15.70:0/614215631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98d23f0 msgr2=0x55cec98d7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:14.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa75b7fe700 1 --2- 172.21.15.70:0/614215631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98d23f0 0x55cec98d7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa75b7fe700 1 -- 172.21.15.70:0/614215631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98ce6a0 msgr2=0x55cec98d6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:14.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa75b7fe700 1 --2- 172.21.15.70:0/614215631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98ce6a0 0x55cec98d6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa75b7fe700 1 -- 172.21.15.70:0/614215631 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cec97b7710 con 0x55cec98cdcf0 2024-06-04T05:22:14.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.914+0000 7fa75affd700 1 --2- 172.21.15.70:0/614215631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98ce6a0 0x55cec98d6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:14.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.915+0000 7fa75b7fe700 1 --2- 172.21.15.70:0/614215631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98cdcf0 0x55cec98d97e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa754014a70 tx=0x7fa754014aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25866d56a2cf884b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:14.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.915+0000 7fa73affd700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:22:14.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.915+0000 7fa73affd700 1 -- 172.21.15.70:0/614215631 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754014cc0 con 0x55cec98cdcf0 2024-06-04T05:22:14.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.915+0000 7fa73affd700 1 -- 172.21.15.70:0/614215631 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7fa754003a00 con 0x55cec98cdcf0 2024-06-04T05:22:14.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.915+0000 7fa76c392b80 1 -- 172.21.15.70:0/614215631 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa754019070 con 0x55cec98cdcf0 2024-06-04T05:22:14.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.915+0000 7fa73affd700 1 -- 172.21.15.70:0/614215631 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fa754009620 con 0x55cec98cdcf0 2024-06-04T05:22:14.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.915+0000 7fa76c392b80 10 client.30502 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:14.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.915+0000 7fa76c392b80 1 -- 172.21.15.70:0/614215631 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55cec98e4c80 con 0x55cec98cdcf0 2024-06-04T05:22:14.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.915+0000 7fa73affd700 1 -- 172.21.15.70:0/614215631 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa754019070 con 0x55cec98cdcf0 2024-06-04T05:22:14.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.915+0000 7fa73affd700 1 -- 172.21.15.70:0/614215631 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7fa75402d070 con 0x55cec98cdcf0 2024-06-04T05:22:14.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.915+0000 7fa76c392b80 10 client.30502 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:14.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.915+0000 7fa76c392b80 10 client.30502 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:14.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.915+0000 7fa76c392b80 20 client.30502 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:14.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.916+0000 7fa76c392b80 1 --2- 172.21.15.70:0/614215631 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cec98e7260 0x55cec98e9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.916+0000 7fa76c392b80 4 client.30502 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:22:14.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.916+0000 7fa76c392b80 1 -- 172.21.15.70:0/614215631 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cec9895a60 con 0x55cec98e7260 2024-06-04T05:22:14.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.916+0000 7fa75affd700 1 --2- 172.21.15.70:0/614215631 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cec98e7260 0x55cec98e9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:14.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.916+0000 7fa75affd700 1 --2- 172.21.15.70:0/614215631 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cec98e7260 0x55cec98e9650 crc :-1 s=READY pgs=519 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60aed42f691f0e00 server_cookie=9c014355bdd4e46d in_seq=0 out_seq=0 2024-06-04T05:22:14.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.916+0000 7fa73affd700 10 client.30502 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.917+0000 7fa73affd700 1 -- 172.21.15.70:0/614215631 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cec9895a60 con 0x55cec98e7260 2024-06-04T05:22:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.917+0000 7fa73affd700 10 client.30502 handle_command_reply: tid=0 2024-06-04T05:22:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.917+0000 7fa76c392b80 1 client.30502 shutdown 2024-06-04T05:22:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.917+0000 7fa76c392b80 1 -- 172.21.15.70:0/614215631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98cdcf0 msgr2=0x55cec98d97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.917+0000 7fa76c392b80 1 --2- 172.21.15.70:0/614215631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98cdcf0 0x55cec98d97e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa754014a70 tx=0x7fa754014aa0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.917+0000 7fa76c392b80 1 -- 172.21.15.70:0/614215631 shutdown_connections 2024-06-04T05:22:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.917+0000 7fa76c392b80 1 --2- 172.21.15.70:0/614215631 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cec98e7260 0x55cec98e9650 crc :-1 s=READY pgs=519 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.917+0000 7fa76c392b80 1 --2- 172.21.15.70:0/614215631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98d23f0 0x55cec98d7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.917+0000 7fa76c392b80 1 --2- 172.21.15.70:0/614215631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98cdcf0 0x55cec98d97e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.917+0000 7fa76c392b80 1 --2- 172.21.15.70:0/614215631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98ce6a0 0x55cec98d6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.917+0000 7fa73affd700 0 client.30502 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.917+0000 7fa76c392b80 1 -- 172.21.15.70:0/614215631 >> 172.21.15.70:0/614215631 conn(0x55cec97f7100 msgr2=0x55cec98c3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:14.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.917+0000 7fa76c392b80 1 -- 172.21.15.70:0/614215631 shutdown_connections 2024-06-04T05:22:14.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.917+0000 7fa76c392b80 1 -- 172.21.15.70:0/614215631 wait complete. 2024-06-04T05:22:14.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.918+0000 7fa76c392b80 20 client.30502 trim_cache size 0 max 16384 2024-06-04T05:22:14.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.938+0000 7fa76c392b80 1 Processor -- start 2024-06-04T05:22:14.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.938+0000 7fa76c392b80 1 -- start start 2024-06-04T05:22:14.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.938+0000 7fa76c392b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98cdcf0 0x55cec98d8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.938+0000 7fa76c392b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98ce6a0 0x55cec98ec2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.938+0000 7fa76c392b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98d23f0 0x55cec98ec7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.939+0000 7fa76c392b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c1abca0 con 0x55cec98cdcf0 2024-06-04T05:22:14.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.939+0000 7fa76c392b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c1ad7e0 con 0x55cec98ce6a0 2024-06-04T05:22:14.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.939+0000 7fa76c392b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c1ac370 con 0x55cec98d23f0 2024-06-04T05:22:14.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.939+0000 7fa75b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98cdcf0 0x55cec98d8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:14.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.939+0000 7fa75b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98cdcf0 0x55cec98d8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33518/0 (socket says 172.21.15.70:33518) 2024-06-04T05:22:14.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.939+0000 7fa75b7fe700 1 -- 172.21.15.70:0/4157767690 learned_addr learned my addr 172.21.15.70:0/4157767690 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:14.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.939+0000 7fa75affd700 1 --2- 172.21.15.70:0/4157767690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98ce6a0 0x55cec98ec2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:14.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.939+0000 7fa75bfff700 1 --2- 172.21.15.70:0/4157767690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98d23f0 0x55cec98ec7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:14.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.939+0000 7fa75b7fe700 1 -- 172.21.15.70:0/4157767690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98d23f0 msgr2=0x55cec98ec7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:14.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.939+0000 7fa75b7fe700 1 --2- 172.21.15.70:0/4157767690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98d23f0 0x55cec98ec7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.939+0000 7fa75b7fe700 1 -- 172.21.15.70:0/4157767690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98ce6a0 msgr2=0x55cec98ec2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:14.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.939+0000 7fa75b7fe700 1 --2- 172.21.15.70:0/4157767690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98ce6a0 0x55cec98ec2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.939+0000 7fa75b7fe700 1 -- 172.21.15.70:0/4157767690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cec98c50d0 con 0x55cec98cdcf0 2024-06-04T05:22:14.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.939+0000 7fa75bfff700 1 --2- 172.21.15.70:0/4157767690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98d23f0 0x55cec98ec7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:22:14.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.939+0000 7fa75b7fe700 1 --2- 172.21.15.70:0/4157767690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98cdcf0 0x55cec98d8a20 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fa754004950 tx=0x7fa754004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46659b2857f0b55 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:14.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.939+0000 7fa7417fa700 1 -- 172.21.15.70:0/4157767690 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754033070 con 0x55cec98cdcf0 2024-06-04T05:22:14.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.940+0000 7fa7417fa700 1 -- 172.21.15.70:0/4157767690 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa754019070 con 0x55cec98cdcf0 2024-06-04T05:22:14.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.940+0000 7fa7417fa700 1 -- 172.21.15.70:0/4157767690 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75402f460 con 0x55cec98cdcf0 2024-06-04T05:22:14.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.941+0000 7fa76c392b80 1 -- 172.21.15.70:0/4157767690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98cdcf0 msgr2=0x55cec98d8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:14.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.941+0000 7fa76c392b80 1 --2- 172.21.15.70:0/4157767690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98cdcf0 0x55cec98d8a20 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fa754004950 tx=0x7fa754004ef0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.941+0000 7fa76c392b80 1 -- 172.21.15.70:0/4157767690 shutdown_connections 2024-06-04T05:22:14.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.941+0000 7fa76c392b80 1 --2- 172.21.15.70:0/4157767690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98d23f0 0x55cec98ec7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.941+0000 7fa76c392b80 1 --2- 172.21.15.70:0/4157767690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98cdcf0 0x55cec98d8a20 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.941+0000 7fa76c392b80 1 --2- 172.21.15.70:0/4157767690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98ce6a0 0x55cec98ec2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.941+0000 7fa76c392b80 1 -- 172.21.15.70:0/4157767690 >> 172.21.15.70:0/4157767690 conn(0x55cec97f7100 msgr2=0x55cec98d8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:14.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.941+0000 7fa76c392b80 1 -- 172.21.15.70:0/4157767690 shutdown_connections 2024-06-04T05:22:14.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.941+0000 7fa76c392b80 1 -- 172.21.15.70:0/4157767690 wait complete. 2024-06-04T05:22:14.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.941+0000 7fa76c392b80 1 -- start start 2024-06-04T05:22:14.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa76c392b80 1 Processor -- start 2024-06-04T05:22:14.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa76c392b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98cdcf0 0x55cec9945a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa76c392b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98ce6a0 0x55cec9945f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa76c392b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98d23f0 0x55cec99464b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa76c392b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c1ad110 con 0x55cec98cdcf0 2024-06-04T05:22:14.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa76c392b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c1aca40 con 0x55cec98ce6a0 2024-06-04T05:22:14.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa76c392b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c1a9c60 con 0x55cec98d23f0 2024-06-04T05:22:14.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa75b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98cdcf0 0x55cec9945a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:14.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa76c392b80 10 client.0 _ll_register_callbacks cb 0x55cec97b39a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:14.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa75b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98cdcf0 0x55cec9945a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33530/0 (socket says 172.21.15.70:33530) 2024-06-04T05:22:14.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa75b7fe700 1 -- 172.21.15.70:0/2034279401 learned_addr learned my addr 172.21.15.70:0/2034279401 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:14.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa75bfff700 1 --2- 172.21.15.70:0/2034279401 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98d23f0 0x55cec99464b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:14.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa75affd700 1 --2- 172.21.15.70:0/2034279401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98ce6a0 0x55cec9945f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:14.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa75b7fe700 1 -- 172.21.15.70:0/2034279401 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98d23f0 msgr2=0x55cec99464b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:14.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa75b7fe700 1 --2- 172.21.15.70:0/2034279401 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98d23f0 0x55cec99464b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa75b7fe700 1 -- 172.21.15.70:0/2034279401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98ce6a0 msgr2=0x55cec9945f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:14.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa75b7fe700 1 --2- 172.21.15.70:0/2034279401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98ce6a0 0x55cec9945f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.942+0000 7fa75b7fe700 1 -- 172.21.15.70:0/2034279401 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cec9950b60 con 0x55cec98cdcf0 2024-06-04T05:22:14.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.943+0000 7fa75b7fe700 1 --2- 172.21.15.70:0/2034279401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98cdcf0 0x55cec9945a30 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fa754009080 tx=0x7fa7540053e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10228c57d7cdeba4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:14.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.943+0000 7fa73affd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:22:14.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.943+0000 7fa73affd700 1 -- 172.21.15.70:0/2034279401 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75403b020 con 0x55cec98cdcf0 2024-06-04T05:22:14.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.943+0000 7fa73affd700 1 -- 172.21.15.70:0/2034279401 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7fa75402f460 con 0x55cec98cdcf0 2024-06-04T05:22:14.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.943+0000 7fa76c392b80 1 -- 172.21.15.70:0/2034279401 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa754019070 con 0x55cec98cdcf0 2024-06-04T05:22:14.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.943+0000 7fa73affd700 1 -- 172.21.15.70:0/2034279401 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fa754046050 con 0x55cec98cdcf0 2024-06-04T05:22:14.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.943+0000 7fa76c392b80 10 client.21771 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:14.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.943+0000 7fa76c392b80 1 -- 172.21.15.70:0/2034279401 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55cec9951ea0 con 0x55cec98cdcf0 2024-06-04T05:22:14.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.943+0000 7fa73affd700 1 -- 172.21.15.70:0/2034279401 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa754019070 con 0x55cec98cdcf0 2024-06-04T05:22:14.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.944+0000 7fa73affd700 1 -- 172.21.15.70:0/2034279401 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7fa75402d040 con 0x55cec98cdcf0 2024-06-04T05:22:14.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.944+0000 7fa76c392b80 10 client.21771 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:14.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.944+0000 7fa76c392b80 10 client.21771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:14.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.944+0000 7fa76c392b80 20 client.21771 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:14.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.944+0000 7fa76c392b80 1 --2- 172.21.15.70:0/2034279401 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cec9952490 0x55cec99548b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:14.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.944+0000 7fa76c392b80 4 client.21771 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:22:14.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.944+0000 7fa76c392b80 1 -- 172.21.15.70:0/2034279401 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cec982ac60 con 0x55cec9952490 2024-06-04T05:22:14.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.944+0000 7fa75affd700 1 --2- 172.21.15.70:0/2034279401 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cec9952490 0x55cec99548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:14.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.945+0000 7fa73affd700 10 client.21771 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:14.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.945+0000 7fa75affd700 1 --2- 172.21.15.70:0/2034279401 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cec9952490 0x55cec99548b0 crc :-1 s=READY pgs=520 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3460459e6774f457 server_cookie=812d87d5a3add443 in_seq=0 out_seq=0 2024-06-04T05:22:14.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.946+0000 7fa73affd700 1 -- 172.21.15.70:0/2034279401 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cec982ac60 con 0x55cec9952490 2024-06-04T05:22:14.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.946+0000 7fa73affd700 10 client.21771 handle_command_reply: tid=0 2024-06-04T05:22:14.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.946+0000 7fa76c392b80 1 client.21771 shutdown 2024-06-04T05:22:14.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.946+0000 7fa76c392b80 1 -- 172.21.15.70:0/2034279401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98cdcf0 msgr2=0x55cec9945a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:14.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.946+0000 7fa76c392b80 1 --2- 172.21.15.70:0/2034279401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98cdcf0 0x55cec9945a30 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fa754009080 tx=0x7fa7540053e0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.946+0000 7fa76c392b80 1 -- 172.21.15.70:0/2034279401 shutdown_connections 2024-06-04T05:22:14.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.946+0000 7fa76c392b80 1 --2- 172.21.15.70:0/2034279401 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cec98d23f0 0x55cec99464b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.946+0000 7fa76c392b80 1 --2- 172.21.15.70:0/2034279401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cec98cdcf0 0x55cec9945a30 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.946+0000 7fa76c392b80 1 --2- 172.21.15.70:0/2034279401 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cec9952490 0x55cec99548b0 crc :-1 s=READY pgs=520 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.946+0000 7fa76c392b80 1 --2- 172.21.15.70:0/2034279401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec98ce6a0 0x55cec9945f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.946+0000 7fa76c392b80 1 -- 172.21.15.70:0/2034279401 >> 172.21.15.70:0/2034279401 conn(0x55cec97f7100 msgr2=0x55cec98d8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:14.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.946+0000 7fa73affd700 0 client.21771 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:14.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.946+0000 7fa76c392b80 1 -- 172.21.15.70:0/2034279401 shutdown_connections 2024-06-04T05:22:14.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.946+0000 7fa76c392b80 1 -- 172.21.15.70:0/2034279401 wait complete. 2024-06-04T05:22:14.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.947+0000 7fa76c392b80 20 client.21771 trim_cache size 0 max 16384 2024-06-04T05:22:14.953 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:22:14.953 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:22:14.953 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "33d24556-4936-460c-b618-9f2c9e54e2f7", 2024-06-04T05:22:14.953 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:22:14.953 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:22:14.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.947+0000 7fa761afd700 1 -- 172.21.15.70:0/3782305022 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa7440b2a70 msgr2=0x7fa7440b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:14.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.947+0000 7fa761afd700 1 --2- 172.21.15.70:0/3782305022 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa7440b2a70 0x7fa7440b4f30 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa750005cb0 tx=0x7fa750005c00 comp rx=0 tx=0).stop 2024-06-04T05:22:14.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.947+0000 7fa761afd700 1 -- 172.21.15.70:0/3782305022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa75c108510 msgr2=0x7fa75c199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:14.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.947+0000 7fa761afd700 1 --2- 172.21.15.70:0/3782305022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa75c108510 0x7fa75c199930 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fa74c00d980 tx=0x7fa74c00dd40 comp rx=0 tx=0).stop 2024-06-04T05:22:14.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.947+0000 7fa761afd700 1 -- 172.21.15.70:0/3782305022 shutdown_connections 2024-06-04T05:22:14.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.947+0000 7fa761afd700 1 --2- 172.21.15.70:0/3782305022 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa7440b2a70 0x7fa7440b4f30 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.947+0000 7fa761afd700 1 --2- 172.21.15.70:0/3782305022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa75c108510 0x7fa75c199930 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.947+0000 7fa761afd700 1 --2- 172.21.15.70:0/3782305022 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa75c102430 0x7fa75c19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.947+0000 7fa761afd700 1 --2- 172.21.15.70:0/3782305022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa75c102df0 0x7fa75c19fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:14.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.947+0000 7fa761afd700 1 -- 172.21.15.70:0/3782305022 >> 172.21.15.70:0/3782305022 conn(0x7fa75c0793b0 msgr2=0x7fa75c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:14.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.948+0000 7fa761afd700 1 -- 172.21.15.70:0/3782305022 shutdown_connections 2024-06-04T05:22:14.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:14.948+0000 7fa761afd700 1 -- 172.21.15.70:0/3782305022 wait complete. 2024-06-04T05:22:15.683 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:15 smithi098 ceph-mon[35569]: pgmap v791: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.0 MiB/s wr, 288 op/s 2024-06-04T05:22:15.683 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:15 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:22:15.683 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:15 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:15.683 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:15 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:22:15.683 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:15 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:22:15.683 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:15 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:22:15.683 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:15 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:15.683 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:15 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:15.684 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:15 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:22:15.684 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:15 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:15.684 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:15 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:15 smithi112 ceph-mon[36453]: pgmap v791: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.0 MiB/s wr, 288 op/s 2024-06-04T05:22:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:15 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:22:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:15 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:15 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:22:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:15 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:22:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:15 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:22:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:15 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:15.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:15 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:15.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:15 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:22:15.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:15 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:15.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:15 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:15.835 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.69 MB/sec execute 547 sec latency 7432.043 ms 2024-06-04T05:22:15.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:15 smithi070 ceph-mon[32631]: pgmap v791: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.0 MiB/s wr, 288 op/s 2024-06-04T05:22:15.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:15 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:22:15.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:15 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:15.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:15 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:22:15.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:15 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:22:15.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:15 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:22:15.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:15 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:15.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:15 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:15.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:15 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:22:15.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:15 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:15.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:15 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:15.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:22:16.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:16 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:22:16.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:16 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:22:16.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:16 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:16.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:16 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:16.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:16 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:22:16.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:16 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:22:16.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:16 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:16.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:16 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:16.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:16 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:22:16.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:16 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:22:16.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:16 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:16.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:16 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:22:16.764 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:22:16.835 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.68 MB/sec execute 548 sec latency 8432.149 ms 2024-06-04T05:22:17.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:17 smithi112 ceph-mon[36453]: pgmap v792: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 175 op/s 2024-06-04T05:22:17.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:17 smithi070 ceph-mon[32631]: pgmap v792: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 175 op/s 2024-06-04T05:22:17.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:17 smithi098 ceph-mon[35569]: pgmap v792: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 175 op/s 2024-06-04T05:22:17.835 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.67 MB/sec execute 549 sec latency 9432.232 ms 2024-06-04T05:22:18.835 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.66 MB/sec execute 550 sec latency 10432.287 ms 2024-06-04T05:22:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.177+0000 7f0206d86640 1 -- 172.21.15.70:0/1639426508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0200108a40 msgr2=0x7f0200108e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.177+0000 7f0206d86640 1 --2- 172.21.15.70:0/1639426508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0200108a40 0x7f0200108e20 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009930 tx=0x7f01f402a420 comp rx=0 tx=0).stop 2024-06-04T05:22:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.177+0000 7f0206d86640 1 -- 172.21.15.70:0/1639426508 shutdown_connections 2024-06-04T05:22:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.177+0000 7f0206d86640 1 --2- 172.21.15.70:0/1639426508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02001033f0 0x7f020010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.177+0000 7f0206d86640 1 --2- 172.21.15.70:0/1639426508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0200102a30 0x7f0200102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.177+0000 7f0206d86640 1 --2- 172.21.15.70:0/1639426508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0200108a40 0x7f0200108e20 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.177+0000 7f0206d86640 1 -- 172.21.15.70:0/1639426508 >> 172.21.15.70:0/1639426508 conn(0x7f0200079ee0 msgr2=0x7f020007a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.177+0000 7f0206d86640 1 -- 172.21.15.70:0/1639426508 shutdown_connections 2024-06-04T05:22:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.177+0000 7f0206d86640 1 -- 172.21.15.70:0/1639426508 wait complete. 2024-06-04T05:22:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.178+0000 7f0206d86640 1 Processor -- start 2024-06-04T05:22:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.178+0000 7f0206d86640 1 -- start start 2024-06-04T05:22:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.178+0000 7f0206d86640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0200102a30 0x7f02001990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.178+0000 7f0206d86640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02001033f0 0x7f0200199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.178+0000 7f0206d86640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0200108a40 0x7f020019d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.178+0000 7f0206d86640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0200112bb0 con 0x7f0200108a40 2024-06-04T05:22:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.178+0000 7f0206d86640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0200112a30 con 0x7f0200102a30 2024-06-04T05:22:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.178+0000 7f0206d86640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0200112d30 con 0x7f02001033f0 2024-06-04T05:22:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.178+0000 7f0205323640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0200108a40 0x7f020019d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.178+0000 7f0205323640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0200108a40 0x7f020019d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49532/0 (socket says 172.21.15.70:49532) 2024-06-04T05:22:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.178+0000 7f0205323640 1 -- 172.21.15.70:0/2751372325 learned_addr learned my addr 172.21.15.70:0/2751372325 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.178+0000 7f0204b22640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0200102a30 0x7f02001990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.179+0000 7f01fffff640 1 --2- 172.21.15.70:0/2751372325 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02001033f0 0x7f0200199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.179+0000 7f0205323640 1 -- 172.21.15.70:0/2751372325 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02001033f0 msgr2=0x7f0200199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.179+0000 7f0205323640 1 --2- 172.21.15.70:0/2751372325 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02001033f0 0x7f0200199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.179+0000 7f0205323640 1 -- 172.21.15.70:0/2751372325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0200102a30 msgr2=0x7f02001990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.179+0000 7f0205323640 1 --2- 172.21.15.70:0/2751372325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0200102a30 0x7f02001990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.179+0000 7f0205323640 1 -- 172.21.15.70:0/2751372325 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020019e090 con 0x7f0200108a40 2024-06-04T05:22:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.179+0000 7f01fffff640 1 --2- 172.21.15.70:0/2751372325 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02001033f0 0x7f0200199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:22:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.179+0000 7f0204b22640 1 --2- 172.21.15.70:0/2751372325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0200102a30 0x7f02001990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:19.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.179+0000 7f0205323640 1 --2- 172.21.15.70:0/2751372325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0200108a40 0x7f020019d960 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f01f800cab0 tx=0x7f01f800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:19.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.179+0000 7f01fdffb640 1 -- 172.21.15.70:0/2751372325 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f800ae20 con 0x7f0200108a40 2024-06-04T05:22:19.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.179+0000 7f01fdffb640 1 -- 172.21.15.70:0/2751372325 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f01f800f8e0 con 0x7f0200108a40 2024-06-04T05:22:19.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.179+0000 7f0206d86640 1 -- 172.21.15.70:0/2751372325 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f020019e380 con 0x7f0200108a40 2024-06-04T05:22:19.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.179+0000 7f01fdffb640 1 -- 172.21.15.70:0/2751372325 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f8018930 con 0x7f0200108a40 2024-06-04T05:22:19.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.179+0000 7f0206d86640 1 -- 172.21.15.70:0/2751372325 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02001a5c10 con 0x7f0200108a40 2024-06-04T05:22:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.182+0000 7f01fdffb640 1 -- 172.21.15.70:0/2751372325 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f01f800fa90 con 0x7f0200108a40 2024-06-04T05:22:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.182+0000 7f0209b78740 1 Processor -- start 2024-06-04T05:22:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.182+0000 7f0209b78740 1 -- start start 2024-06-04T05:22:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.182+0000 7f0209b78740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936312f820 0x55936312fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.182+0000 7f0209b78740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559363130140 0x55936313a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.182+0000 7f0209b78740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936313aa00 0x55936313cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.182+0000 7f0209b78740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5593630a1c30 con 0x55936313aa00 2024-06-04T05:22:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.182+0000 7f0209b78740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5593630917d0 con 0x55936312f820 2024-06-04T05:22:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.182+0000 7f0209b78740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559363095810 con 0x559363130140 2024-06-04T05:22:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.182+0000 7f01fffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936313aa00 0x55936313cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.182+0000 7f01fffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936313aa00 0x55936313cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49548/0 (socket says 172.21.15.70:49548) 2024-06-04T05:22:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.182+0000 7f01fffff640 1 -- 172.21.15.70:0/1490346624 learned_addr learned my addr 172.21.15.70:0/1490346624 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.183+0000 7f01fdffb640 1 --2- 172.21.15.70:0/2751372325 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01f00b2a90 0x7f01f00b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.183+0000 7f0204b22640 1 --2- 172.21.15.70:0/1490346624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559363130140 0x55936313a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.183+0000 7f01fffff640 1 --2- 172.21.15.70:0/1490346624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936312f820 0x55936312fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.183+0000 7f0205323640 1 --2- 172.21.15.70:0/2751372325 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01f00b2a90 0x7f01f00b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.183+0000 7f01fffff640 1 -- 172.21.15.70:0/1490346624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559363130140 msgr2=0x55936313a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.183+0000 7f01fffff640 1 --2- 172.21.15.70:0/1490346624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559363130140 0x55936313a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.183+0000 7f01fffff640 1 -- 172.21.15.70:0/1490346624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936312f820 msgr2=0x55936312fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.183+0000 7f01fffff640 1 --2- 172.21.15.70:0/1490346624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936312f820 0x55936312fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.183+0000 7f01fffff640 1 -- 172.21.15.70:0/1490346624 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559362da6210 con 0x55936313aa00 2024-06-04T05:22:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.183+0000 7f0204b22640 1 --2- 172.21.15.70:0/1490346624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559363130140 0x55936313a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:22:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.183+0000 7f01fdffb640 1 -- 172.21.15.70:0/2751372325 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f01f80e2970 con 0x7f0200108a40 2024-06-04T05:22:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.183+0000 7f01fffff640 1 --2- 172.21.15.70:0/1490346624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936313aa00 0x55936313cdf0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f01ec0124f0 tx=0x7f01ec00b600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e52a8a60831f1e99 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.183+0000 7f01eaffd640 1 -- 172.21.15.70:0/1490346624 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec018070 con 0x55936313aa00 2024-06-04T05:22:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.183+0000 7f0205323640 1 --2- 172.21.15.70:0/2751372325 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01f00b2a90 0x7f01f00b4f50 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f01f80e0140 tx=0x7f01f8076550 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.183+0000 7f01eaffd640 1 -- 172.21.15.70:0/1490346624 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f01ec01daf0 con 0x55936313aa00 2024-06-04T05:22:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.183+0000 7f01eaffd640 1 -- 172.21.15.70:0/1490346624 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec010a40 con 0x55936313aa00 2024-06-04T05:22:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.184+0000 7f0209b78740 1 -- 172.21.15.70:0/1490346624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936313aa00 msgr2=0x55936313cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.184+0000 7f0209b78740 1 --2- 172.21.15.70:0/1490346624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936313aa00 0x55936313cdf0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f01ec0124f0 tx=0x7f01ec00b600 comp rx=0 tx=0).stop 2024-06-04T05:22:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.184+0000 7f0209b78740 1 -- 172.21.15.70:0/1490346624 shutdown_connections 2024-06-04T05:22:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.184+0000 7f0209b78740 1 --2- 172.21.15.70:0/1490346624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936313aa00 0x55936313cdf0 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.184+0000 7f0209b78740 1 --2- 172.21.15.70:0/1490346624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559363130140 0x55936313a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.184+0000 7f0209b78740 1 --2- 172.21.15.70:0/1490346624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936312f820 0x55936312fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.184+0000 7f0209b78740 1 -- 172.21.15.70:0/1490346624 >> 172.21.15.70:0/1490346624 conn(0x55936312bd30 msgr2=0x55936312c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.184+0000 7f0209b78740 1 -- 172.21.15.70:0/1490346624 shutdown_connections 2024-06-04T05:22:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.184+0000 7f0209b78740 1 -- 172.21.15.70:0/1490346624 wait complete. 2024-06-04T05:22:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f0209b78740 1 -- start start 2024-06-04T05:22:19.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f0209b78740 1 Processor -- start 2024-06-04T05:22:19.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f0209b78740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936312f820 0x559363137a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f0209b78740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559363130140 0x559363137fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f0209b78740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936313aa00 0x559363138510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f0209b78740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559362d77790 con 0x55936312f820 2024-06-04T05:22:19.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f0209b78740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55936304d890 con 0x55936313aa00 2024-06-04T05:22:19.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f0209b78740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55936313d630 con 0x559363130140 2024-06-04T05:22:19.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f0209b78740 10 client.0 _ll_register_callbacks cb 0x559362da5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:19.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f01fffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936312f820 0x559363137a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f01fffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936312f820 0x559363137a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49560/0 (socket says 172.21.15.70:49560) 2024-06-04T05:22:19.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f01fffff640 1 -- 172.21.15.70:0/1987844518 learned_addr learned my addr 172.21.15.70:0/1987844518 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:19.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f0204b22640 1 --2- 172.21.15.70:0/1987844518 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559363130140 0x559363137fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f01fffff640 1 --2- 172.21.15.70:0/1987844518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936313aa00 0x559363138510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f01fffff640 1 -- 172.21.15.70:0/1987844518 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559363130140 msgr2=0x559363137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f01fffff640 1 --2- 172.21.15.70:0/1987844518 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559363130140 0x559363137fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f01fffff640 1 -- 172.21.15.70:0/1987844518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936313aa00 msgr2=0x559363138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f01fffff640 1 --2- 172.21.15.70:0/1987844518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936313aa00 0x559363138510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.185+0000 7f01fffff640 1 -- 172.21.15.70:0/1987844518 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559362e72620 con 0x55936312f820 2024-06-04T05:22:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.186+0000 7f01fffff640 1 --2- 172.21.15.70:0/1987844518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936312f820 0x559363137a90 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f01ec015020 tx=0x7f01ec005ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba88fa4c03c5950 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.186+0000 7f01dffff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:22:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.186+0000 7f01dffff640 1 -- 172.21.15.70:0/1987844518 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec018040 con 0x55936312f820 2024-06-04T05:22:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.186+0000 7f01dffff640 1 -- 172.21.15.70:0/1987844518 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f01ec010bd0 con 0x55936312f820 2024-06-04T05:22:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.186+0000 7f0209b78740 1 -- 172.21.15.70:0/1987844518 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01ec01daf0 con 0x55936312f820 2024-06-04T05:22:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.186+0000 7f01dffff640 1 -- 172.21.15.70:0/1987844518 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f01ec010430 con 0x55936312f820 2024-06-04T05:22:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.186+0000 7f0209b78740 10 client.21783 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.186+0000 7f0209b78740 1 -- 172.21.15.70:0/1987844518 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x559363149490 con 0x55936312f820 2024-06-04T05:22:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.186+0000 7f01dffff640 1 -- 172.21.15.70:0/1987844518 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f01ec01daf0 con 0x55936312f820 2024-06-04T05:22:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.186+0000 7f01dffff640 1 -- 172.21.15.70:0/1987844518 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f01ec028070 con 0x55936312f820 2024-06-04T05:22:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.186+0000 7f0209b78740 10 client.21783 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.186+0000 7f0209b78740 10 client.21783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.186+0000 7f0209b78740 20 client.21783 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.186+0000 7f0209b78740 1 --2- 172.21.15.70:0/1987844518 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55936314a840 0x55936314cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.186+0000 7f0209b78740 4 client.21783 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:22:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.186+0000 7f0209b78740 1 -- 172.21.15.70:0/1987844518 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593630a3b80 con 0x55936314a840 2024-06-04T05:22:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.187+0000 7f0204b22640 1 --2- 172.21.15.70:0/1987844518 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55936314a840 0x55936314cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.187+0000 7f0204b22640 1 --2- 172.21.15.70:0/1987844518 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55936314a840 0x55936314cc30 crc :-1 s=READY pgs=521 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fdb0071068a00b4 server_cookie=b498765738a3c23b in_seq=0 out_seq=0 2024-06-04T05:22:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.187+0000 7f01dffff640 10 client.21783 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.188+0000 7f01dffff640 1 -- 172.21.15.70:0/1987844518 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5593630a3b80 con 0x55936314a840 2024-06-04T05:22:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.188+0000 7f01dffff640 10 client.21783 handle_command_reply: tid=0 2024-06-04T05:22:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.188+0000 7f0209b78740 1 client.21783 shutdown 2024-06-04T05:22:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.188+0000 7f0209b78740 1 -- 172.21.15.70:0/1987844518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936312f820 msgr2=0x559363137a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.188+0000 7f0209b78740 1 --2- 172.21.15.70:0/1987844518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936312f820 0x559363137a90 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f01ec015020 tx=0x7f01ec005ee0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.188+0000 7f0209b78740 1 -- 172.21.15.70:0/1987844518 shutdown_connections 2024-06-04T05:22:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.188+0000 7f0209b78740 1 --2- 172.21.15.70:0/1987844518 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55936314a840 0x55936314cc30 crc :-1 s=READY pgs=521 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.188+0000 7f0209b78740 1 --2- 172.21.15.70:0/1987844518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936313aa00 0x559363138510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.188+0000 7f0209b78740 1 --2- 172.21.15.70:0/1987844518 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559363130140 0x559363137fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.188+0000 7f0209b78740 1 --2- 172.21.15.70:0/1987844518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936312f820 0x559363137a90 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.188+0000 7f01dffff640 0 client.21783 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.188+0000 7f0209b78740 1 -- 172.21.15.70:0/1987844518 >> 172.21.15.70:0/1987844518 conn(0x55936312bd30 msgr2=0x55936312afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.188+0000 7f0209b78740 1 -- 172.21.15.70:0/1987844518 shutdown_connections 2024-06-04T05:22:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.188+0000 7f0209b78740 1 -- 172.21.15.70:0/1987844518 wait complete. 2024-06-04T05:22:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.188+0000 7f0209b78740 20 client.21783 trim_cache size 0 max 16384 2024-06-04T05:22:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.206+0000 7f0209b78740 1 Processor -- start 2024-06-04T05:22:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.206+0000 7f0209b78740 1 -- start start 2024-06-04T05:22:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.207+0000 7f0209b78740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936312f820 0x55936312b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.207+0000 7f0209b78740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559363130140 0x559363139b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.207+0000 7f0209b78740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936313aa00 0x55936313a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.207+0000 7f0209b78740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f02001ae5f0 con 0x55936312f820 2024-06-04T05:22:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.207+0000 7f0209b78740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02001b0130 con 0x55936313aa00 2024-06-04T05:22:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.207+0000 7f0209b78740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f02001aecc0 con 0x559363130140 2024-06-04T05:22:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.207+0000 7f01fffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936312f820 0x55936312b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.207+0000 7f01fffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936312f820 0x55936312b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49566/0 (socket says 172.21.15.70:49566) 2024-06-04T05:22:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.207+0000 7f01fffff640 1 -- 172.21.15.70:0/398061830 learned_addr learned my addr 172.21.15.70:0/398061830 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.207+0000 7f0204b22640 1 --2- 172.21.15.70:0/398061830 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559363130140 0x559363139b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.207+0000 7f01fffff640 1 --2- 172.21.15.70:0/398061830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936313aa00 0x55936313a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.207+0000 7f01fffff640 1 -- 172.21.15.70:0/398061830 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559363130140 msgr2=0x559363139b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.207+0000 7f01fffff640 1 --2- 172.21.15.70:0/398061830 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559363130140 0x559363139b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.207+0000 7f01fffff640 1 -- 172.21.15.70:0/398061830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936313aa00 msgr2=0x55936313a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.207+0000 7f01fffff640 1 --2- 172.21.15.70:0/398061830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936313aa00 0x55936313a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.207+0000 7f01fffff640 1 -- 172.21.15.70:0/398061830 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55936313a5d0 con 0x55936312f820 2024-06-04T05:22:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.207+0000 7f01fffff640 1 --2- 172.21.15.70:0/398061830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936312f820 0x55936312b730 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f01ec005cc0 tx=0x7f01ec0105d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c003872b4f48f36 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.208+0000 7f01ea7fc640 1 -- 172.21.15.70:0/398061830 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec018070 con 0x55936312f820 2024-06-04T05:22:19.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.208+0000 7f01ea7fc640 1 -- 172.21.15.70:0/398061830 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f01ec032430 con 0x55936312f820 2024-06-04T05:22:19.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.208+0000 7f01ea7fc640 1 -- 172.21.15.70:0/398061830 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec0052a0 con 0x55936312f820 2024-06-04T05:22:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.208+0000 7f0209b78740 1 -- 172.21.15.70:0/398061830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936312f820 msgr2=0x55936312b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.208+0000 7f0209b78740 1 --2- 172.21.15.70:0/398061830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936312f820 0x55936312b730 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f01ec005cc0 tx=0x7f01ec0105d0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.209+0000 7f0209b78740 1 -- 172.21.15.70:0/398061830 shutdown_connections 2024-06-04T05:22:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.209+0000 7f0209b78740 1 --2- 172.21.15.70:0/398061830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936313aa00 0x55936313a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.209+0000 7f0209b78740 1 --2- 172.21.15.70:0/398061830 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559363130140 0x559363139b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.209+0000 7f0209b78740 1 --2- 172.21.15.70:0/398061830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55936312f820 0x55936312b730 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.209+0000 7f0209b78740 1 -- 172.21.15.70:0/398061830 >> 172.21.15.70:0/398061830 conn(0x55936312bd30 msgr2=0x559363146080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.209+0000 7f0209b78740 1 -- 172.21.15.70:0/398061830 shutdown_connections 2024-06-04T05:22:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.209+0000 7f0209b78740 1 -- 172.21.15.70:0/398061830 wait complete. 2024-06-04T05:22:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.209+0000 7f0209b78740 1 -- start start 2024-06-04T05:22:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.209+0000 7f0209b78740 1 Processor -- start 2024-06-04T05:22:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f0209b78740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55936312f820 0x559363142e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f0209b78740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559363130140 0x5593631401d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f0209b78740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936313aa00 0x559363140710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f0209b78740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f02001afa60 con 0x559363130140 2024-06-04T05:22:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f0209b78740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02001af390 con 0x55936313aa00 2024-06-04T05:22:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f0209b78740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f02001ac5f0 con 0x55936312f820 2024-06-04T05:22:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f0209b78740 10 client.0 _ll_register_callbacks cb 0x7f01ec00bf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f0204b22640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559363130140 0x5593631401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f01fffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55936312f820 0x559363142e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f0204b22640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559363130140 0x5593631401d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49582/0 (socket says 172.21.15.70:49582) 2024-06-04T05:22:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f0204b22640 1 -- 172.21.15.70:0/3055435599 learned_addr learned my addr 172.21.15.70:0/3055435599 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f01fffff640 1 --2- 172.21.15.70:0/3055435599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936313aa00 0x559363140710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f0204b22640 1 -- 172.21.15.70:0/3055435599 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55936312f820 msgr2=0x559363142e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f0204b22640 1 --2- 172.21.15.70:0/3055435599 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55936312f820 0x559363142e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f0204b22640 1 -- 172.21.15.70:0/3055435599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936313aa00 msgr2=0x559363140710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f0204b22640 1 --2- 172.21.15.70:0/3055435599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936313aa00 0x559363140710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f0204b22640 1 -- 172.21.15.70:0/3055435599 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559363140dd0 con 0x559363130140 2024-06-04T05:22:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f01fffff640 1 --2- 172.21.15.70:0/3055435599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936313aa00 0x559363140710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f0204b22640 1 --2- 172.21.15.70:0/3055435599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559363130140 0x5593631401d0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f01f40057c0 tx=0x7f01f40043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=285c77a17bf2e5e2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.210+0000 7f01dffff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:22:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.211+0000 7f01dffff640 1 -- 172.21.15.70:0/3055435599 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f404b070 con 0x559363130140 2024-06-04T05:22:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.211+0000 7f01dffff640 1 -- 172.21.15.70:0/3055435599 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f01f4031bb0 con 0x559363130140 2024-06-04T05:22:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.211+0000 7f0209b78740 1 -- 172.21.15.70:0/3055435599 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01ec032430 con 0x559363130140 2024-06-04T05:22:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.211+0000 7f01dffff640 1 -- 172.21.15.70:0/3055435599 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f01f4056050 con 0x559363130140 2024-06-04T05:22:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.211+0000 7f0209b78740 10 client.21789 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.211+0000 7f0209b78740 1 -- 172.21.15.70:0/3055435599 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5593631b5190 con 0x559363130140 2024-06-04T05:22:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.211+0000 7f01dffff640 1 -- 172.21.15.70:0/3055435599 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f01ec032430 con 0x559363130140 2024-06-04T05:22:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.211+0000 7f01dffff640 1 -- 172.21.15.70:0/3055435599 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f01f40035f0 con 0x559363130140 2024-06-04T05:22:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.211+0000 7f0209b78740 10 client.21789 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.211+0000 7f0209b78740 10 client.21789 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.211+0000 7f0209b78740 20 client.21789 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.211+0000 7f0209b78740 1 --2- 172.21.15.70:0/3055435599 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5593631b58c0 0x5593631b7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.211+0000 7f0209b78740 4 client.21789 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:22:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.211+0000 7f0209b78740 1 -- 172.21.15.70:0/3055435599 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55936309d360 con 0x5593631b58c0 2024-06-04T05:22:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.212+0000 7f01fffff640 1 --2- 172.21.15.70:0/3055435599 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5593631b58c0 0x5593631b7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.212+0000 7f01dffff640 10 client.21789 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:19.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.212+0000 7f01fffff640 1 --2- 172.21.15.70:0/3055435599 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5593631b58c0 0x5593631b7ce0 crc :-1 s=READY pgs=522 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63fd50c90791e3e9 server_cookie=789fcc29a3ac461e in_seq=0 out_seq=0 2024-06-04T05:22:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.279+0000 7f01dffff640 1 -- 172.21.15.70:0/3055435599 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55936309d360 con 0x5593631b58c0 2024-06-04T05:22:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.279+0000 7f01dffff640 10 client.21789 handle_command_reply: tid=0 2024-06-04T05:22:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.279+0000 7f0209b78740 1 client.21789 shutdown 2024-06-04T05:22:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.279+0000 7f0209b78740 1 -- 172.21.15.70:0/3055435599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559363130140 msgr2=0x5593631401d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.279+0000 7f0209b78740 1 --2- 172.21.15.70:0/3055435599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559363130140 0x5593631401d0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f01f40057c0 tx=0x7f01f40043b0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.279+0000 7f0209b78740 1 -- 172.21.15.70:0/3055435599 shutdown_connections 2024-06-04T05:22:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.279+0000 7f0209b78740 1 --2- 172.21.15.70:0/3055435599 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5593631b58c0 0x5593631b7ce0 crc :-1 s=READY pgs=522 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.279+0000 7f0209b78740 1 --2- 172.21.15.70:0/3055435599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55936313aa00 0x559363140710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.279+0000 7f0209b78740 1 --2- 172.21.15.70:0/3055435599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559363130140 0x5593631401d0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.279+0000 7f0209b78740 1 --2- 172.21.15.70:0/3055435599 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55936312f820 0x559363142e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.279+0000 7f01dffff640 0 client.21789 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.279+0000 7f0209b78740 1 -- 172.21.15.70:0/3055435599 >> 172.21.15.70:0/3055435599 conn(0x55936312bd30 msgr2=0x559363139b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.279+0000 7f0209b78740 1 -- 172.21.15.70:0/3055435599 shutdown_connections 2024-06-04T05:22:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.279+0000 7f0209b78740 1 -- 172.21.15.70:0/3055435599 wait complete. 2024-06-04T05:22:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.279+0000 7f0209b78740 20 client.21789 trim_cache size 0 max 16384 2024-06-04T05:22:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:22:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:22:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:22:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.280+0000 7f0206d86640 1 -- 172.21.15.70:0/2751372325 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01f00b2a90 msgr2=0x7f01f00b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.280+0000 7f0206d86640 1 --2- 172.21.15.70:0/2751372325 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01f00b2a90 0x7f01f00b4f50 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f01f80e0140 tx=0x7f01f8076550 comp rx=0 tx=0).stop 2024-06-04T05:22:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.280+0000 7f0206d86640 1 -- 172.21.15.70:0/2751372325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0200108a40 msgr2=0x7f020019d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.280+0000 7f0206d86640 1 --2- 172.21.15.70:0/2751372325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0200108a40 0x7f020019d960 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f01f800cab0 tx=0x7f01f800cf70 comp rx=0 tx=0).stop 2024-06-04T05:22:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.280+0000 7f0206d86640 1 -- 172.21.15.70:0/2751372325 shutdown_connections 2024-06-04T05:22:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.280+0000 7f0206d86640 1 --2- 172.21.15.70:0/2751372325 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f01f00b2a90 0x7f01f00b4f50 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.280+0000 7f0206d86640 1 --2- 172.21.15.70:0/2751372325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0200108a40 0x7f020019d960 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.280+0000 7f0206d86640 1 --2- 172.21.15.70:0/2751372325 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02001033f0 0x7f0200199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.280+0000 7f0206d86640 1 --2- 172.21.15.70:0/2751372325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0200102a30 0x7f02001990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.280+0000 7f0206d86640 1 -- 172.21.15.70:0/2751372325 >> 172.21.15.70:0/2751372325 conn(0x7f0200079ee0 msgr2=0x7f0200100430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.280+0000 7f0206d86640 1 -- 172.21.15.70:0/2751372325 shutdown_connections 2024-06-04T05:22:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:19.280+0000 7f0206d86640 1 -- 172.21.15.70:0/2751372325 wait complete. 2024-06-04T05:22:19.525 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:19 smithi070 ceph-mon[32631]: pgmap v793: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 170 op/s 2024-06-04T05:22:19.653 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:19 smithi098 ceph-mon[35569]: pgmap v793: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 170 op/s 2024-06-04T05:22:19.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:19 smithi112 ceph-mon[36453]: pgmap v793: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 170 op/s 2024-06-04T05:22:19.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.765+0000 7f47d8606700 1 -- 172.21.15.70:0/2644684665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47d006c470 msgr2=0x7f47d01103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.765+0000 7f47d8606700 1 --2- 172.21.15.70:0/2644684665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47d006c470 0x7f47d01103c0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f47c800b0a0 tx=0x7f47c800b3b0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.765+0000 7f47d8606700 1 -- 172.21.15.70:0/2644684665 shutdown_connections 2024-06-04T05:22:19.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.765+0000 7f47d8606700 1 --2- 172.21.15.70:0/2644684665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47d006c470 0x7f47d01103c0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.765+0000 7f47d8606700 1 --2- 172.21.15.70:0/2644684665 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47d01095a0 0x7f47d0109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.765+0000 7f47d8606700 1 --2- 172.21.15.70:0/2644684665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d006bab0 0x7f47d006bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.765+0000 7f47d8606700 1 -- 172.21.15.70:0/2644684665 >> 172.21.15.70:0/2644684665 conn(0x7f47d00793b0 msgr2=0x7f47d00797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.765+0000 7f47d8606700 1 -- 172.21.15.70:0/2644684665 shutdown_connections 2024-06-04T05:22:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.765+0000 7f47d8606700 1 -- 172.21.15.70:0/2644684665 wait complete. 2024-06-04T05:22:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d8606700 1 Processor -- start 2024-06-04T05:22:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d8606700 1 -- start start 2024-06-04T05:22:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d8606700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47d006bab0 0x7f47d019f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d8606700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47d006c470 0x7f47d019fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d8606700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d01095a0 0x7f47d01998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d8606700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f47d0113580 con 0x7f47d006bab0 2024-06-04T05:22:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d8606700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47d0113400 con 0x7f47d01095a0 2024-06-04T05:22:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d8606700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f47d0113700 con 0x7f47d006c470 2024-06-04T05:22:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d63a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47d006bab0 0x7f47d019f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d63a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47d006bab0 0x7f47d019f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49604/0 (socket says 172.21.15.70:49604) 2024-06-04T05:22:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d6ba3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d01095a0 0x7f47d01998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d63a2700 1 -- 172.21.15.70:0/2436905495 learned_addr learned my addr 172.21.15.70:0/2436905495 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d5ba1700 1 --2- 172.21.15.70:0/2436905495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47d006c470 0x7f47d019fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d63a2700 1 -- 172.21.15.70:0/2436905495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47d006c470 msgr2=0x7f47d019fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d63a2700 1 --2- 172.21.15.70:0/2436905495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47d006c470 0x7f47d019fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d63a2700 1 -- 172.21.15.70:0/2436905495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d01095a0 msgr2=0x7f47d01998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d63a2700 1 --2- 172.21.15.70:0/2436905495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d01095a0 0x7f47d01998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d63a2700 1 -- 172.21.15.70:0/2436905495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d019a020 con 0x7f47d006bab0 2024-06-04T05:22:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.766+0000 7f47d6ba3700 1 --2- 172.21.15.70:0/2436905495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d01095a0 0x7f47d01998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.767+0000 7f47d63a2700 1 --2- 172.21.15.70:0/2436905495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47d006bab0 0x7f47d019f700 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f47cc00c970 tx=0x7f47cc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.767+0000 7f47c77fe700 1 -- 172.21.15.70:0/2436905495 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc007aa0 con 0x7f47d006bab0 2024-06-04T05:22:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.767+0000 7f47c77fe700 1 -- 172.21.15.70:0/2436905495 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47cc004cc0 con 0x7f47d006bab0 2024-06-04T05:22:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.767+0000 7f47d8606700 1 -- 172.21.15.70:0/2436905495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47d019a2b0 con 0x7f47d006bab0 2024-06-04T05:22:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.767+0000 7f47c77fe700 1 -- 172.21.15.70:0/2436905495 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc005450 con 0x7f47d006bab0 2024-06-04T05:22:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.767+0000 7f47d8606700 1 -- 172.21.15.70:0/2436905495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47d01a65d0 con 0x7f47d006bab0 2024-06-04T05:22:19.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.769+0000 7f47c77fe700 1 -- 172.21.15.70:0/2436905495 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f47cc007c40 con 0x7f47d006bab0 2024-06-04T05:22:19.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.770+0000 7f47c77fe700 1 --2- 172.21.15.70:0/2436905495 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f47bc0b29f0 0x7f47bc0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.770+0000 7f47d5ba1700 1 --2- 172.21.15.70:0/2436905495 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f47bc0b29f0 0x7f47bc0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.771+0000 7f47c77fe700 1 -- 172.21.15.70:0/2436905495 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f47cc0e2360 con 0x7f47d006bab0 2024-06-04T05:22:19.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.771+0000 7f47d5ba1700 1 --2- 172.21.15.70:0/2436905495 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f47bc0b29f0 0x7f47bc0b4eb0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f47d0079a90 tx=0x7f47c0009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:19.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.771+0000 7f47e2e9bb80 1 Processor -- start 2024-06-04T05:22:19.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.771+0000 7f47e2e9bb80 1 -- start start 2024-06-04T05:22:19.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.771+0000 7f47e2e9bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd40320 0x55ef2fd40700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.771+0000 7f47e2e9bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd3ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.771+0000 7f47e2e9bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd3b260 0x55ef2fd47080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.771+0000 7f47e2e9bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ef2fc98d70 con 0x55ef2fd40320 2024-06-04T05:22:19.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.771+0000 7f47e2e9bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ef2fc88910 con 0x55ef2fd3a8a0 2024-06-04T05:22:19.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.771+0000 7f47e2e9bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ef2fc8c950 con 0x55ef2fd3b260 2024-06-04T05:22:19.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.771+0000 7f47d63a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd40320 0x55ef2fd40700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.771+0000 7f47d63a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd40320 0x55ef2fd40700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49618/0 (socket says 172.21.15.70:49618) 2024-06-04T05:22:19.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.771+0000 7f47d63a2700 1 -- 172.21.15.70:0/705027362 learned_addr learned my addr 172.21.15.70:0/705027362 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:19.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.771+0000 7f47d5ba1700 1 --2- 172.21.15.70:0/705027362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd3ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.772+0000 7f47d6ba3700 1 --2- 172.21.15.70:0/705027362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd3b260 0x55ef2fd47080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.772+0000 7f47d63a2700 1 -- 172.21.15.70:0/705027362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd3b260 msgr2=0x55ef2fd47080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:19.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.772+0000 7f47d63a2700 1 --2- 172.21.15.70:0/705027362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd3b260 0x55ef2fd47080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.772+0000 7f47d63a2700 1 -- 172.21.15.70:0/705027362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd3a8a0 msgr2=0x55ef2fd3ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:19.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.772+0000 7f47d63a2700 1 --2- 172.21.15.70:0/705027362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd3ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.772+0000 7f47d63a2700 1 -- 172.21.15.70:0/705027362 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef2fc31fb0 con 0x55ef2fd40320 2024-06-04T05:22:19.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.772+0000 7f47d63a2700 1 --2- 172.21.15.70:0/705027362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd40320 0x55ef2fd40700 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0a5720 tx=0x7f47cc0a8f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=130694955defc8e2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:19.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.772+0000 7f47c4ff9700 1 -- 172.21.15.70:0/705027362 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc0a96a0 con 0x55ef2fd40320 2024-06-04T05:22:19.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.772+0000 7f47c4ff9700 1 -- 172.21.15.70:0/705027362 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47cc0ada50 con 0x55ef2fd40320 2024-06-04T05:22:19.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.772+0000 7f47c4ff9700 1 -- 172.21.15.70:0/705027362 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc0a9b20 con 0x55ef2fd40320 2024-06-04T05:22:19.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.773+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/705027362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd40320 msgr2=0x55ef2fd40700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.773+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/705027362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd40320 0x55ef2fd40700 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0a5720 tx=0x7f47cc0a8f10 comp rx=0 tx=0).stop 2024-06-04T05:22:19.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.773+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/705027362 shutdown_connections 2024-06-04T05:22:19.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.773+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/705027362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd3b260 0x55ef2fd47080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.773+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/705027362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd40320 0x55ef2fd40700 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.773+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/705027362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd3ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.773+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/705027362 >> 172.21.15.70:0/705027362 conn(0x55ef2fc66240 msgr2=0x55ef2fd353e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:19.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.773+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/705027362 shutdown_connections 2024-06-04T05:22:19.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.773+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/705027362 wait complete. 2024-06-04T05:22:19.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.773+0000 7f47e2e9bb80 1 -- start start 2024-06-04T05:22:19.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.773+0000 7f47e2e9bb80 1 Processor -- start 2024-06-04T05:22:19.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47e2e9bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd46dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47e2e9bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd3b260 0x55ef2fd440c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47e2e9bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd40320 0x55ef2fd44600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47e2e9bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ef2fb8efa0 con 0x55ef2fd3a8a0 2024-06-04T05:22:19.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47e2e9bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ef2fbf0da0 con 0x55ef2fd40320 2024-06-04T05:22:19.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47e2e9bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ef2fd476e0 con 0x55ef2fd3b260 2024-06-04T05:22:19.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47e2e9bb80 10 client.0 _ll_register_callbacks cb 0x55ef2fc23950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:19.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47d63a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd46dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47d6ba3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd40320 0x55ef2fd44600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47d63a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd46dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49622/0 (socket says 172.21.15.70:49622) 2024-06-04T05:22:19.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47d63a2700 1 -- 172.21.15.70:0/997014605 learned_addr learned my addr 172.21.15.70:0/997014605 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:19.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47d5ba1700 1 --2- 172.21.15.70:0/997014605 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd3b260 0x55ef2fd440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47d6ba3700 1 -- 172.21.15.70:0/997014605 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd3b260 msgr2=0x55ef2fd440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:19.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47d6ba3700 1 --2- 172.21.15.70:0/997014605 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd3b260 0x55ef2fd440c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47d6ba3700 1 -- 172.21.15.70:0/997014605 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd3a8a0 msgr2=0x55ef2fd46dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:19.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47d6ba3700 1 --2- 172.21.15.70:0/997014605 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd46dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47d6ba3700 1 -- 172.21.15.70:0/997014605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef2fbd88d0 con 0x55ef2fd40320 2024-06-04T05:22:19.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47d5ba1700 1 --2- 172.21.15.70:0/997014605 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd3b260 0x55ef2fd440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:22:19.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47d63a2700 1 --2- 172.21.15.70:0/997014605 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd46dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:19.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47d6ba3700 1 --2- 172.21.15.70:0/997014605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd40320 0x55ef2fd44600 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f47c800b070 tx=0x7f47c8004c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6d9ae489f1e32cb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:19.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.774+0000 7f47b9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:22:19.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.775+0000 7f47b9ffb700 1 -- 172.21.15.70:0/997014605 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c8021070 con 0x55ef2fd40320 2024-06-04T05:22:19.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.775+0000 7f47b9ffb700 1 -- 172.21.15.70:0/997014605 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f47c80078f0 con 0x55ef2fd40320 2024-06-04T05:22:19.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.775+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/997014605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47cc0ada50 con 0x55ef2fd40320 2024-06-04T05:22:19.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.775+0000 7f47b9ffb700 1 -- 172.21.15.70:0/997014605 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f47c800ce70 con 0x55ef2fd40320 2024-06-04T05:22:19.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.775+0000 7f47e2e9bb80 10 client.30395 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:19.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.775+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/997014605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55ef2fc42760 con 0x55ef2fd40320 2024-06-04T05:22:19.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.775+0000 7f47b9ffb700 1 -- 172.21.15.70:0/997014605 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47cc0ada50 con 0x55ef2fd40320 2024-06-04T05:22:19.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.775+0000 7f47b9ffb700 1 -- 172.21.15.70:0/997014605 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f47c8011070 con 0x55ef2fd40320 2024-06-04T05:22:19.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.775+0000 7f47e2e9bb80 10 client.30395 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:19.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.775+0000 7f47e2e9bb80 10 client.30395 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.775+0000 7f47e2e9bb80 20 client.30395 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.775+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/997014605 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ef2fd54360 0x55ef2fd56750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.775+0000 7f47e2e9bb80 4 client.30395 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:22:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.775+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/997014605 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef2fc9acc0 con 0x55ef2fd54360 2024-06-04T05:22:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.776+0000 7f47d63a2700 1 --2- 172.21.15.70:0/997014605 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ef2fd54360 0x55ef2fd56750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.776+0000 7f47d63a2700 1 --2- 172.21.15.70:0/997014605 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ef2fd54360 0x55ef2fd56750 crc :-1 s=READY pgs=523 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6d929e57b395596 server_cookie=1b966832f0cc95b1 in_seq=0 out_seq=0 2024-06-04T05:22:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.776+0000 7f47b9ffb700 10 client.30395 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.777+0000 7f47b9ffb700 1 -- 172.21.15.70:0/997014605 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ef2fc9acc0 con 0x55ef2fd54360 2024-06-04T05:22:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.777+0000 7f47b9ffb700 10 client.30395 handle_command_reply: tid=0 2024-06-04T05:22:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.777+0000 7f47e2e9bb80 1 client.30395 shutdown 2024-06-04T05:22:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.777+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/997014605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd40320 msgr2=0x55ef2fd44600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.777+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/997014605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd40320 0x55ef2fd44600 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f47c800b070 tx=0x7f47c8004c60 comp rx=0 tx=0).stop 2024-06-04T05:22:19.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.777+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/997014605 shutdown_connections 2024-06-04T05:22:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.777+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/997014605 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ef2fd54360 0x55ef2fd56750 crc :-1 s=READY pgs=523 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.777+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/997014605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd40320 0x55ef2fd44600 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.777+0000 7f47b9ffb700 0 client.30395 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.777+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/997014605 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd46dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.777+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/997014605 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd3b260 0x55ef2fd440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.777+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/997014605 >> 172.21.15.70:0/997014605 conn(0x55ef2fc66240 msgr2=0x55ef2fd34af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.777+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/997014605 shutdown_connections 2024-06-04T05:22:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.777+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/997014605 wait complete. 2024-06-04T05:22:19.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.777+0000 7f47e2e9bb80 20 client.30395 trim_cache size 0 max 16384 2024-06-04T05:22:19.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47e2e9bb80 1 Processor -- start 2024-06-04T05:22:19.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47e2e9bb80 1 -- start start 2024-06-04T05:22:19.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47e2e9bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd50740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47e2e9bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd3b260 0x55ef2fd43240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47e2e9bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd40320 0x55ef2fd35270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47e2e9bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f47d01add20 con 0x55ef2fd3a8a0 2024-06-04T05:22:19.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47e2e9bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47d01af860 con 0x55ef2fd3b260 2024-06-04T05:22:19.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47e2e9bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f47d01ae3f0 con 0x55ef2fd40320 2024-06-04T05:22:19.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47d63a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd50740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47d63a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd50740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49628/0 (socket says 172.21.15.70:49628) 2024-06-04T05:22:19.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47d63a2700 1 -- 172.21.15.70:0/715043644 learned_addr learned my addr 172.21.15.70:0/715043644 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:19.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47d5ba1700 1 --2- 172.21.15.70:0/715043644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd3b260 0x55ef2fd43240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47d63a2700 1 -- 172.21.15.70:0/715043644 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd40320 msgr2=0x55ef2fd35270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:19.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47d63a2700 1 --2- 172.21.15.70:0/715043644 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd40320 0x55ef2fd35270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47d63a2700 1 -- 172.21.15.70:0/715043644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd3b260 msgr2=0x55ef2fd43240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:19.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47d63a2700 1 --2- 172.21.15.70:0/715043644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd3b260 0x55ef2fd43240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47d63a2700 1 -- 172.21.15.70:0/715043644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef2fd56ac0 con 0x55ef2fd3a8a0 2024-06-04T05:22:19.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.798+0000 7f47d63a2700 1 --2- 172.21.15.70:0/715043644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd50740 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0e0080 tx=0x7f47cc0cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34dbe14b4bd97bd8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:19.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.799+0000 7f47bbfff700 1 -- 172.21.15.70:0/715043644 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc0a5c90 con 0x55ef2fd3a8a0 2024-06-04T05:22:19.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.799+0000 7f47bbfff700 1 -- 172.21.15.70:0/715043644 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47cc0a9900 con 0x55ef2fd3a8a0 2024-06-04T05:22:19.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.799+0000 7f47bbfff700 1 -- 172.21.15.70:0/715043644 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc08fbc0 con 0x55ef2fd3a8a0 2024-06-04T05:22:19.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.799+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/715043644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd3a8a0 msgr2=0x55ef2fd50740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.799+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/715043644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd50740 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0e0080 tx=0x7f47cc0cc040 comp rx=0 tx=0).stop 2024-06-04T05:22:19.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.799+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/715043644 shutdown_connections 2024-06-04T05:22:19.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.799+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/715043644 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd40320 0x55ef2fd35270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.799+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/715043644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd50740 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.799+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/715043644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd3b260 0x55ef2fd43240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.799+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/715043644 >> 172.21.15.70:0/715043644 conn(0x55ef2fc66240 msgr2=0x55ef2fd458f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:19.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.799+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/715043644 shutdown_connections 2024-06-04T05:22:19.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.799+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/715043644 wait complete. 2024-06-04T05:22:19.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47e2e9bb80 1 -- start start 2024-06-04T05:22:19.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47e2e9bb80 1 Processor -- start 2024-06-04T05:22:19.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47e2e9bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd5bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47e2e9bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd3b260 0x55ef2fd5c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47e2e9bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd40320 0x55ef2fd5c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47e2e9bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f47d01af190 con 0x55ef2fd40320 2024-06-04T05:22:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47e2e9bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47d01aeac0 con 0x55ef2fd3b260 2024-06-04T05:22:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47e2e9bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f47d01abd30 con 0x55ef2fd3a8a0 2024-06-04T05:22:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47e2e9bb80 10 client.0 _ll_register_callbacks cb 0x55ef2fc23950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47d6ba3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd40320 0x55ef2fd5c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47d6ba3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd40320 0x55ef2fd5c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49632/0 (socket says 172.21.15.70:49632) 2024-06-04T05:22:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47d6ba3700 1 -- 172.21.15.70:0/2641628284 learned_addr learned my addr 172.21.15.70:0/2641628284 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47d5ba1700 1 --2- 172.21.15.70:0/2641628284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd3b260 0x55ef2fd5c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47d63a2700 1 --2- 172.21.15.70:0/2641628284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd5bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47d6ba3700 1 -- 172.21.15.70:0/2641628284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd3a8a0 msgr2=0x55ef2fd5bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47d6ba3700 1 --2- 172.21.15.70:0/2641628284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd5bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47d6ba3700 1 -- 172.21.15.70:0/2641628284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd3b260 msgr2=0x55ef2fd5c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47d6ba3700 1 --2- 172.21.15.70:0/2641628284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd3b260 0x55ef2fd5c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47d6ba3700 1 -- 172.21.15.70:0/2641628284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef2fdc24a0 con 0x55ef2fd40320 2024-06-04T05:22:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.800+0000 7f47d63a2700 1 --2- 172.21.15.70:0/2641628284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd5bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:22:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.801+0000 7f47d6ba3700 1 --2- 172.21.15.70:0/2641628284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd40320 0x55ef2fd5c7b0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f47c8009530 tx=0x7f47c8005ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ceeffb6a9f130f server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.801+0000 7f47b9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:22:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.801+0000 7f47b9ffb700 1 -- 172.21.15.70:0/2641628284 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c8021070 con 0x55ef2fd40320 2024-06-04T05:22:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.801+0000 7f47b9ffb700 1 -- 172.21.15.70:0/2641628284 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f47c8005250 con 0x55ef2fd40320 2024-06-04T05:22:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.801+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/2641628284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47cc0a9900 con 0x55ef2fd40320 2024-06-04T05:22:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.801+0000 7f47b9ffb700 1 -- 172.21.15.70:0/2641628284 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f47c803e050 con 0x55ef2fd40320 2024-06-04T05:22:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.801+0000 7f47e2e9bb80 10 client.21807 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.801+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/2641628284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55ef2fdc3790 con 0x55ef2fd40320 2024-06-04T05:22:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.801+0000 7f47b9ffb700 1 -- 172.21.15.70:0/2641628284 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47cc0a9900 con 0x55ef2fd40320 2024-06-04T05:22:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.801+0000 7f47b9ffb700 1 -- 172.21.15.70:0/2641628284 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f47c8035070 con 0x55ef2fd40320 2024-06-04T05:22:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.801+0000 7f47e2e9bb80 10 client.21807 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.801+0000 7f47e2e9bb80 10 client.21807 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.801+0000 7f47e2e9bb80 20 client.21807 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.801+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/2641628284 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ef2fdc3e40 0x55ef2fdc6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.801+0000 7f47e2e9bb80 4 client.21807 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:22:19.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.801+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/2641628284 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ef2fc944a0 con 0x55ef2fdc3e40 2024-06-04T05:22:19.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.802+0000 7f47d63a2700 1 --2- 172.21.15.70:0/2641628284 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ef2fdc3e40 0x55ef2fdc6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:19.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.802+0000 7f47d63a2700 1 --2- 172.21.15.70:0/2641628284 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ef2fdc3e40 0x55ef2fdc6260 crc :-1 s=READY pgs=524 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d383eff7a7f26d7e server_cookie=9229d213a699159b in_seq=0 out_seq=0 2024-06-04T05:22:19.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.802+0000 7f47b9ffb700 10 client.21807 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:19.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.802+0000 7f47b9ffb700 1 -- 172.21.15.70:0/2641628284 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ef2fc944a0 con 0x55ef2fdc3e40 2024-06-04T05:22:19.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.802+0000 7f47b9ffb700 10 client.21807 handle_command_reply: tid=0 2024-06-04T05:22:19.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.802+0000 7f47e2e9bb80 1 client.21807 shutdown 2024-06-04T05:22:19.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.802+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/2641628284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd40320 msgr2=0x55ef2fd5c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.802+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/2641628284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd40320 0x55ef2fd5c7b0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f47c8009530 tx=0x7f47c8005ac0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.802+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/2641628284 shutdown_connections 2024-06-04T05:22:19.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.802+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/2641628284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef2fd40320 0x55ef2fd5c7b0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.802+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/2641628284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ef2fd3a8a0 0x55ef2fd5bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.802+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/2641628284 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55ef2fdc3e40 0x55ef2fdc6260 crc :-1 s=READY pgs=524 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.802+0000 7f47e2e9bb80 1 --2- 172.21.15.70:0/2641628284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef2fd3b260 0x55ef2fd5c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.802+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/2641628284 >> 172.21.15.70:0/2641628284 conn(0x55ef2fc66240 msgr2=0x55ef2fd3a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.803+0000 7f47b9ffb700 0 client.21807 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.803+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/2641628284 shutdown_connections 2024-06-04T05:22:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.803+0000 7f47e2e9bb80 1 -- 172.21.15.70:0/2641628284 wait complete. 2024-06-04T05:22:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.803+0000 7f47e2e9bb80 20 client.21807 trim_cache size 0 max 16384 2024-06-04T05:22:19.810 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:22:19.810 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-04T05:22:19.810 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-06-04T05:22:19.810 INFO:teuthology.orchestra.run.smithi070.stdout: "33d24556-4936-460c-b618-9f2c9e54e2f7": { 2024-06-04T05:22:19.811 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-06-04T05:22:19.811 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "33d24556-4936-460c-b618-9f2c9e54e2f7", 2024-06-04T05:22:19.811 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-06-04T05:22:19.811 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:22:19.811 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:22:19.811 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:22:19.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.803+0000 7f47d8606700 1 -- 172.21.15.70:0/2436905495 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f47bc0b29f0 msgr2=0x7f47bc0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.803+0000 7f47d8606700 1 --2- 172.21.15.70:0/2436905495 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f47bc0b29f0 0x7f47bc0b4eb0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f47d0079a90 tx=0x7f47c0009260 comp rx=0 tx=0).stop 2024-06-04T05:22:19.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.803+0000 7f47d8606700 1 -- 172.21.15.70:0/2436905495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47d006bab0 msgr2=0x7f47d019f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:19.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.803+0000 7f47d8606700 1 --2- 172.21.15.70:0/2436905495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47d006bab0 0x7f47d019f700 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f47cc00c970 tx=0x7f47cc00cc80 comp rx=0 tx=0).stop 2024-06-04T05:22:19.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.803+0000 7f47d8606700 1 -- 172.21.15.70:0/2436905495 shutdown_connections 2024-06-04T05:22:19.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.803+0000 7f47d8606700 1 --2- 172.21.15.70:0/2436905495 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f47bc0b29f0 0x7f47bc0b4eb0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.803+0000 7f47d8606700 1 --2- 172.21.15.70:0/2436905495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d01095a0 0x7f47d01998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.803+0000 7f47d8606700 1 --2- 172.21.15.70:0/2436905495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47d006bab0 0x7f47d019f700 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.803+0000 7f47d8606700 1 --2- 172.21.15.70:0/2436905495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47d006c470 0x7f47d019fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:19.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.803+0000 7f47d8606700 1 -- 172.21.15.70:0/2436905495 >> 172.21.15.70:0/2436905495 conn(0x7f47d00793b0 msgr2=0x7f47d0101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:19.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.803+0000 7f47d8606700 1 -- 172.21.15.70:0/2436905495 shutdown_connections 2024-06-04T05:22:19.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:19.803+0000 7f47d8606700 1 -- 172.21.15.70:0/2436905495 wait complete. 2024-06-04T05:22:19.835 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.65 MB/sec execute 551 sec latency 11432.310 ms 2024-06-04T05:22:20.838 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.63 MB/sec execute 552 sec latency 12432.356 ms 2024-06-04T05:22:20.943 INFO:tasks.cephfs.filesystem:scrub status for tag:33d24556-4936-460c-b618-9f2c9e54e2f7 - {'path': '/', 'tag': '33d24556-4936-460c-b618-9f2c9e54e2f7', 'options': 'recursive,force'} 2024-06-04T05:22:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:21 smithi112 ceph-mon[36453]: pgmap v794: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 146 op/s 2024-06-04T05:22:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:21 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:22:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:21 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:22:21.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:21 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:22:21.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:21 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:22:21.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:21 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:22:21.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:21 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:22:21.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:21 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:22:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:21 smithi098 ceph-mon[35569]: pgmap v794: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 146 op/s 2024-06-04T05:22:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:21 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:22:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:21 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:22:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:21 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:22:21.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:21 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:22:21.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:21 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:22:21.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:21 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:22:21.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:21 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:22:21.835 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.62 MB/sec execute 553 sec latency 13432.450 ms 2024-06-04T05:22:21.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:21 smithi070 ceph-mon[32631]: pgmap v794: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 146 op/s 2024-06-04T05:22:21.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:21 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:22:21.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:21 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:22:21.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:21 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:22:21.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:21 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:22:21.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:21 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:22:21.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:21 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:22:21.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:21 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:22:22.835 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.61 MB/sec execute 554 sec latency 14432.521 ms 2024-06-04T05:22:23.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:23 smithi112 ceph-mon[36453]: pgmap v795: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 173 op/s 2024-06-04T05:22:23.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:23 smithi098 ceph-mon[35569]: pgmap v795: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 173 op/s 2024-06-04T05:22:23.835 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.60 MB/sec execute 555 sec latency 15432.630 ms 2024-06-04T05:22:23.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:23 smithi070 ceph-mon[32631]: pgmap v795: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 173 op/s 2024-06-04T05:22:24.835 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.59 MB/sec execute 556 sec latency 16432.693 ms 2024-06-04T05:22:25.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:25 smithi112 ceph-mon[36453]: pgmap v796: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 154 KiB/s wr, 75 op/s 2024-06-04T05:22:25.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:25 smithi098 ceph-mon[35569]: pgmap v796: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 154 KiB/s wr, 75 op/s 2024-06-04T05:22:25.835 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.57 MB/sec execute 557 sec latency 17432.783 ms 2024-06-04T05:22:25.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:25 smithi070 ceph-mon[32631]: pgmap v796: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 154 KiB/s wr, 75 op/s 2024-06-04T05:22:26.835 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.56 MB/sec execute 558 sec latency 18432.922 ms 2024-06-04T05:22:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:27 smithi112 ceph-mon[36453]: pgmap v797: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 162 KiB/s wr, 80 op/s 2024-06-04T05:22:27.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:27 smithi098 ceph-mon[35569]: pgmap v797: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 162 KiB/s wr, 80 op/s 2024-06-04T05:22:27.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.55 MB/sec execute 559 sec latency 19432.996 ms 2024-06-04T05:22:27.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:27 smithi070 ceph-mon[32631]: pgmap v797: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 162 KiB/s wr, 80 op/s 2024-06-04T05:22:28.201 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:22:28.467 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:22:28.589 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:28 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:22:28.589 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:28 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:22:28.589 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:28 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:22:28.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:28 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:22:28.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:28 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:22:28.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:28 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:22:28.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:28 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:22:28.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:28 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:22:28.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:28 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:22:28.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.54 MB/sec execute 560 sec latency 20433.042 ms 2024-06-04T05:22:29.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:29 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:22:29.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:29 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:22:29.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:29 smithi070 ceph-mon[32631]: pgmap v798: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 43 KiB/s wr, 33 op/s 2024-06-04T05:22:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:29 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:22:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:29 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:22:29.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:29 smithi112 ceph-mon[36453]: pgmap v798: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 43 KiB/s wr, 33 op/s 2024-06-04T05:22:29.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:29 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:22:29.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:29 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:22:29.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:29 smithi098 ceph-mon[35569]: pgmap v798: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 43 KiB/s wr, 33 op/s 2024-06-04T05:22:29.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.53 MB/sec execute 561 sec latency 21433.134 ms 2024-06-04T05:22:30.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.092+0000 7f065946b700 1 -- 172.21.15.70:0/2470885063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06541077d0 msgr2=0x7f065410f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:30.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.092+0000 7f065946b700 1 --2- 172.21.15.70:0/2470885063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06541077d0 0x7f065410f370 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f064c00b0d0 tx=0x7f064c00b3e0 comp rx=0 tx=0).stop 2024-06-04T05:22:30.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.092+0000 7f065946b700 1 -- 172.21.15.70:0/2470885063 shutdown_connections 2024-06-04T05:22:30.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.092+0000 7f065946b700 1 --2- 172.21.15.70:0/2470885063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06541077d0 0x7f065410f370 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:30.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.092+0000 7f065946b700 1 --2- 172.21.15.70:0/2470885063 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0654102a80 0x7f0654102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:30.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.092+0000 7f065946b700 1 --2- 172.21.15.70:0/2470885063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0654103430 0x7f06541070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:30.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.092+0000 7f065946b700 1 -- 172.21.15.70:0/2470885063 >> 172.21.15.70:0/2470885063 conn(0x7f06540793f0 msgr2=0x7f0654079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:30.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.092+0000 7f065946b700 1 -- 172.21.15.70:0/2470885063 shutdown_connections 2024-06-04T05:22:30.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.092+0000 7f065946b700 1 -- 172.21.15.70:0/2470885063 wait complete. 2024-06-04T05:22:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.093+0000 7f065946b700 1 Processor -- start 2024-06-04T05:22:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.093+0000 7f065946b700 1 -- start start 2024-06-04T05:22:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.093+0000 7f065946b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0654102a80 0x7f065419e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f065946b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0654103430 0x7f065419ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f065946b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06541077d0 0x7f06541988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f065946b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0654112570 con 0x7f06541077d0 2024-06-04T05:22:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f065946b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f06541123f0 con 0x7f0654103430 2024-06-04T05:22:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f065946b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f06541126f0 con 0x7f0654102a80 2024-06-04T05:22:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f06537fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06541077d0 0x7f06541988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f06537fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06541077d0 0x7f06541988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42632/0 (socket says 172.21.15.70:42632) 2024-06-04T05:22:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f06537fe700 1 -- 172.21.15.70:0/532861646 learned_addr learned my addr 172.21.15.70:0/532861646 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f06527fc700 1 --2- 172.21.15.70:0/532861646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0654103430 0x7f065419ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f0652ffd700 1 --2- 172.21.15.70:0/532861646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0654102a80 0x7f065419e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f06537fe700 1 -- 172.21.15.70:0/532861646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0654102a80 msgr2=0x7f065419e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f06537fe700 1 --2- 172.21.15.70:0/532861646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0654102a80 0x7f065419e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f06537fe700 1 -- 172.21.15.70:0/532861646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0654103430 msgr2=0x7f065419ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f06537fe700 1 --2- 172.21.15.70:0/532861646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0654103430 0x7f065419ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f06537fe700 1 -- 172.21.15.70:0/532861646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0654199010 con 0x7f06541077d0 2024-06-04T05:22:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f0652ffd700 1 --2- 172.21.15.70:0/532861646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0654102a80 0x7f065419e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:22:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f06537fe700 1 --2- 172.21.15.70:0/532861646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06541077d0 0x7f06541988b0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f064c009850 tx=0x7f064c007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f063bfff700 1 -- 172.21.15.70:0/532861646 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064c01f070 con 0x7f06541077d0 2024-06-04T05:22:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f065946b700 1 -- 172.21.15.70:0/532861646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06541992a0 con 0x7f06541077d0 2024-06-04T05:22:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f063bfff700 1 -- 172.21.15.70:0/532861646 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f064c005500 con 0x7f06541077d0 2024-06-04T05:22:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.094+0000 7f063bfff700 1 -- 172.21.15.70:0/532861646 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064c01b9a0 con 0x7f06541077d0 2024-06-04T05:22:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.095+0000 7f065946b700 1 -- 172.21.15.70:0/532861646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06541a55c0 con 0x7f06541077d0 2024-06-04T05:22:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.095+0000 7f065946b700 1 -- 172.21.15.70:0/532861646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f065406b4c0 con 0x7f06541077d0 2024-06-04T05:22:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.097+0000 7f063bfff700 1 -- 172.21.15.70:0/532861646 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f064c022070 con 0x7f06541077d0 2024-06-04T05:22:30.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.098+0000 7f063bfff700 1 --2- 172.21.15.70:0/532861646 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f063c0b29f0 0x7f063c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:30.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.098+0000 7f0652ffd700 1 --2- 172.21.15.70:0/532861646 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f063c0b29f0 0x7f063c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:30.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.098+0000 7f063bfff700 1 -- 172.21.15.70:0/532861646 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f064c00f020 con 0x7f06541077d0 2024-06-04T05:22:30.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.098+0000 7f0652ffd700 1 --2- 172.21.15.70:0/532861646 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f063c0b29f0 0x7f063c0b4eb0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f06541999f0 tx=0x7f0648008040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:30.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.101+0000 7f063bfff700 1 -- 172.21.15.70:0/532861646 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f064c029020 con 0x7f06541077d0 2024-06-04T05:22:30.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.349+0000 7f065946b700 1 -- 172.21.15.70:0/532861646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0654103b80 con 0x7f06541077d0 2024-06-04T05:22:30.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.350+0000 7f063bfff700 1 -- 172.21.15.70:0/532861646 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+11929 (secure 0 0 0) 0x7f064c0ab8e0 con 0x7f06541077d0 2024-06-04T05:22:30.353 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:22:30.353 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":67,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:22:05.021574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:22:30.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.353+0000 7f065946b700 1 -- 172.21.15.70:0/532861646 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f063c0b29f0 msgr2=0x7f063c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:30.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.353+0000 7f065946b700 1 --2- 172.21.15.70:0/532861646 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f063c0b29f0 0x7f063c0b4eb0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f06541999f0 tx=0x7f0648008040 comp rx=0 tx=0).stop 2024-06-04T05:22:30.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.353+0000 7f065946b700 1 -- 172.21.15.70:0/532861646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06541077d0 msgr2=0x7f06541988b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:30.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.353+0000 7f065946b700 1 --2- 172.21.15.70:0/532861646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06541077d0 0x7f06541988b0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f064c009850 tx=0x7f064c007d90 comp rx=0 tx=0).stop 2024-06-04T05:22:30.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.353+0000 7f065946b700 1 -- 172.21.15.70:0/532861646 shutdown_connections 2024-06-04T05:22:30.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.353+0000 7f065946b700 1 --2- 172.21.15.70:0/532861646 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f063c0b29f0 0x7f063c0b4eb0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:30.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.353+0000 7f065946b700 1 --2- 172.21.15.70:0/532861646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06541077d0 0x7f06541988b0 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:30.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.353+0000 7f065946b700 1 --2- 172.21.15.70:0/532861646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0654102a80 0x7f065419e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:30.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.353+0000 7f065946b700 1 --2- 172.21.15.70:0/532861646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0654103430 0x7f065419ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:30.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.353+0000 7f065946b700 1 -- 172.21.15.70:0/532861646 >> 172.21.15.70:0/532861646 conn(0x7f06540793f0 msgr2=0x7f0654100b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:30.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.353+0000 7f065946b700 1 -- 172.21.15.70:0/532861646 shutdown_connections 2024-06-04T05:22:30.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:30.354+0000 7f065946b700 1 -- 172.21.15.70:0/532861646 wait complete. 2024-06-04T05:22:30.357 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 67 2024-06-04T05:22:30.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:30 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/532861646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:22:30.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:30 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/532861646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:22:30.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:30 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/532861646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:22:30.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.52 MB/sec execute 562 sec latency 22433.167 ms 2024-06-04T05:22:31.027 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --set-id 582ae688 --release --await' 2024-06-04T05:22:31.028 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --set-id 582ae688 --release --await 2024-06-04T05:22:31.275 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:22:31.677 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:31 smithi070 ceph-mon[32631]: pgmap v799: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 43 KiB/s wr, 33 op/s 2024-06-04T05:22:31.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:31 smithi112 ceph-mon[36453]: pgmap v799: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 43 KiB/s wr, 33 op/s 2024-06-04T05:22:31.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:31 smithi098 ceph-mon[35569]: pgmap v799: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 43 KiB/s wr, 33 op/s 2024-06-04T05:22:31.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.50 MB/sec execute 563 sec latency 23433.253 ms 2024-06-04T05:22:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.376+0000 7fcc1e4bd640 1 -- 172.21.15.70:0/3054524335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc18101350 msgr2=0x7fcc18101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.376+0000 7fcc1e4bd640 1 --2- 172.21.15.70:0/3054524335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc18101350 0x7fcc18101730 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800aa10 tx=0x7fcc0802b720 comp rx=0 tx=0).stop 2024-06-04T05:22:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.376+0000 7fcc1e4bd640 1 -- 172.21.15.70:0/3054524335 shutdown_connections 2024-06-04T05:22:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.376+0000 7fcc1e4bd640 1 --2- 172.21.15.70:0/3054524335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc18101d00 0x7fcc18105d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.376+0000 7fcc1e4bd640 1 --2- 172.21.15.70:0/3054524335 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc18106420 0x7fcc1810dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.376+0000 7fcc1e4bd640 1 --2- 172.21.15.70:0/3054524335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc18101350 0x7fcc18101730 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.376+0000 7fcc1e4bd640 1 -- 172.21.15.70:0/3054524335 >> 172.21.15.70:0/3054524335 conn(0x7fcc18077eb0 msgr2=0x7fcc180ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:32.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.376+0000 7fcc1e4bd640 1 -- 172.21.15.70:0/3054524335 shutdown_connections 2024-06-04T05:22:32.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.376+0000 7fcc1e4bd640 1 -- 172.21.15.70:0/3054524335 wait complete. 2024-06-04T05:22:32.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc1e4bd640 1 Processor -- start 2024-06-04T05:22:32.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc1e4bd640 1 -- start start 2024-06-04T05:22:32.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc1e4bd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc18101350 0x7fcc18196d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc1e4bd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc18106420 0x7fcc18197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc1e4bd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc181977d0 0x7fcc181a7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc1e4bd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcc181109b0 con 0x7fcc18101350 2024-06-04T05:22:32.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc1e4bd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc18110830 con 0x7fcc18106420 2024-06-04T05:22:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc1e4bd640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcc18110b30 con 0x7fcc181977d0 2024-06-04T05:22:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc17fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc18101350 0x7fcc18196d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc17fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc18101350 0x7fcc18196d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42650/0 (socket says 172.21.15.70:42650) 2024-06-04T05:22:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc17fff640 1 -- 172.21.15.70:0/2219271124 learned_addr learned my addr 172.21.15.70:0/2219271124 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc1ca5a640 1 --2- 172.21.15.70:0/2219271124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc181977d0 0x7fcc181a7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc0ffff640 1 --2- 172.21.15.70:0/2219271124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc18106420 0x7fcc18197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc17fff640 1 -- 172.21.15.70:0/2219271124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc181977d0 msgr2=0x7fcc181a7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc17fff640 1 --2- 172.21.15.70:0/2219271124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc181977d0 0x7fcc181a7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc17fff640 1 -- 172.21.15.70:0/2219271124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc18106420 msgr2=0x7fcc18197290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc17fff640 1 --2- 172.21.15.70:0/2219271124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc18106420 0x7fcc18197290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc17fff640 1 -- 172.21.15.70:0/2219271124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc181a7a10 con 0x7fcc18101350 2024-06-04T05:22:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.377+0000 7fcc17fff640 1 --2- 172.21.15.70:0/2219271124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc18101350 0x7fcc18196d50 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800a340 tx=0x7fcc08004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.378+0000 7fcc15ffb640 1 -- 172.21.15.70:0/2219271124 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0803f070 con 0x7fcc18101350 2024-06-04T05:22:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.378+0000 7fcc15ffb640 1 -- 172.21.15.70:0/2219271124 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcc08003c80 con 0x7fcc18101350 2024-06-04T05:22:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.378+0000 7fcc1e4bd640 1 -- 172.21.15.70:0/2219271124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc181a7ca0 con 0x7fcc18101350 2024-06-04T05:22:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.378+0000 7fcc15ffb640 1 -- 172.21.15.70:0/2219271124 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc08036040 con 0x7fcc18101350 2024-06-04T05:22:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.378+0000 7fcc1e4bd640 1 -- 172.21.15.70:0/2219271124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc181a8190 con 0x7fcc18101350 2024-06-04T05:22:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.380+0000 7fcc15ffb640 1 -- 172.21.15.70:0/2219271124 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fcc08042070 con 0x7fcc18101350 2024-06-04T05:22:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.380+0000 7fcc211d1740 1 Processor -- start 2024-06-04T05:22:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.380+0000 7fcc211d1740 1 -- start start 2024-06-04T05:22:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.380+0000 7fcc211d1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e1820 0x562e6c2e1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.380+0000 7fcc211d1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2e2140 0x562e6c2ec4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.380+0000 7fcc211d1740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2eca00 0x562e6c2eedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.380+0000 7fcc211d1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562e6c253c30 con 0x562e6c2e1820 2024-06-04T05:22:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.380+0000 7fcc211d1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562e6c2437d0 con 0x562e6c2e2140 2024-06-04T05:22:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.380+0000 7fcc211d1740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562e6c247810 con 0x562e6c2eca00 2024-06-04T05:22:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.380+0000 7fcc1ca5a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e1820 0x562e6c2e1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.381+0000 7fcc1ca5a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e1820 0x562e6c2e1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42656/0 (socket says 172.21.15.70:42656) 2024-06-04T05:22:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.381+0000 7fcc1ca5a640 1 -- 172.21.15.70:0/1412525024 learned_addr learned my addr 172.21.15.70:0/1412525024 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.381+0000 7fcc17fff640 1 --2- 172.21.15.70:0/1412525024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2e2140 0x562e6c2ec4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.381+0000 7fcc0ffff640 1 --2- 172.21.15.70:0/1412525024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2eca00 0x562e6c2eedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.381+0000 7fcc1ca5a640 1 -- 172.21.15.70:0/1412525024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2eca00 msgr2=0x562e6c2eedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.381+0000 7fcc1ca5a640 1 --2- 172.21.15.70:0/1412525024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2eca00 0x562e6c2eedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.381+0000 7fcc1ca5a640 1 -- 172.21.15.70:0/1412525024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2e2140 msgr2=0x562e6c2ec4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.381+0000 7fcc1ca5a640 1 --2- 172.21.15.70:0/1412525024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2e2140 0x562e6c2ec4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.381+0000 7fcc1ca5a640 1 -- 172.21.15.70:0/1412525024 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e6bf58210 con 0x562e6c2e1820 2024-06-04T05:22:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.381+0000 7fcc0ffff640 1 --2- 172.21.15.70:0/1412525024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2eca00 0x562e6c2eedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:22:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.381+0000 7fcc15ffb640 1 --2- 172.21.15.70:0/2219271124 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcc040b2ac0 0x7fcc040b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.381+0000 7fcc1ca5a640 1 --2- 172.21.15.70:0/1412525024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e1820 0x562e6c2e1c00 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009f70 tx=0x7fcc10010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7003fa196a231304 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.381+0000 7fcc17fff640 1 --2- 172.21.15.70:0/2219271124 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcc040b2ac0 0x7fcc040b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.381+0000 7fcc0e7fc640 1 -- 172.21.15.70:0/1412525024 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10016070 con 0x562e6c2e1820 2024-06-04T05:22:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.381+0000 7fcc15ffb640 1 -- 172.21.15.70:0/2219271124 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7fcc08102fb0 con 0x7fcc18101350 2024-06-04T05:22:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.381+0000 7fcc0e7fc640 1 -- 172.21.15.70:0/1412525024 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcc10004d20 con 0x562e6c2e1820 2024-06-04T05:22:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.381+0000 7fcc0e7fc640 1 -- 172.21.15.70:0/1412525024 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10012760 con 0x562e6c2e1820 2024-06-04T05:22:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.382+0000 7fcc211d1740 1 -- 172.21.15.70:0/1412525024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e1820 msgr2=0x562e6c2e1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.382+0000 7fcc17fff640 1 --2- 172.21.15.70:0/2219271124 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcc040b2ac0 0x7fcc040b4f80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fcc08100780 tx=0x7fcc08106b00 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.382+0000 7fcc211d1740 1 --2- 172.21.15.70:0/1412525024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e1820 0x562e6c2e1c00 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009f70 tx=0x7fcc10010800 comp rx=0 tx=0).stop 2024-06-04T05:22:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.382+0000 7fcc211d1740 1 -- 172.21.15.70:0/1412525024 shutdown_connections 2024-06-04T05:22:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.382+0000 7fcc211d1740 1 --2- 172.21.15.70:0/1412525024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2e2140 0x562e6c2ec4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.382+0000 7fcc211d1740 1 --2- 172.21.15.70:0/1412525024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2eca00 0x562e6c2eedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.382+0000 7fcc211d1740 1 --2- 172.21.15.70:0/1412525024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e1820 0x562e6c2e1c00 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.382+0000 7fcc211d1740 1 -- 172.21.15.70:0/1412525024 >> 172.21.15.70:0/1412525024 conn(0x562e6c2ddd30 msgr2=0x562e6c2de110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.382+0000 7fcc211d1740 1 -- 172.21.15.70:0/1412525024 shutdown_connections 2024-06-04T05:22:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.382+0000 7fcc211d1740 1 -- 172.21.15.70:0/1412525024 wait complete. 2024-06-04T05:22:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.382+0000 7fcc211d1740 1 -- start start 2024-06-04T05:22:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc211d1740 1 Processor -- start 2024-06-04T05:22:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc211d1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2e1820 0x562e6c2e9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc211d1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e2140 0x562e6c2e9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc211d1740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2eca00 0x562e6c2ea4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc211d1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562e6bf29790 con 0x562e6c2e2140 2024-06-04T05:22:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc211d1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562e6c1ff890 con 0x562e6c2e1820 2024-06-04T05:22:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc211d1740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562e6c2ef630 con 0x562e6c2eca00 2024-06-04T05:22:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc211d1740 10 client.0 _ll_register_callbacks cb 0x562e6bf57a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc0ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e2140 0x562e6c2e9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc0ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e2140 0x562e6c2e9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42660/0 (socket says 172.21.15.70:42660) 2024-06-04T05:22:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc1ca5a640 1 --2- 172.21.15.70:0/1194878151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2e1820 0x562e6c2e9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc0ffff640 1 -- 172.21.15.70:0/1194878151 learned_addr learned my addr 172.21.15.70:0/1194878151 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc1ca5a640 1 --2- 172.21.15.70:0/1194878151 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2eca00 0x562e6c2ea4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc0ffff640 1 -- 172.21.15.70:0/1194878151 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2eca00 msgr2=0x562e6c2ea4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc0ffff640 1 --2- 172.21.15.70:0/1194878151 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2eca00 0x562e6c2ea4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc0ffff640 1 -- 172.21.15.70:0/1194878151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2e1820 msgr2=0x562e6c2e9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc0ffff640 1 --2- 172.21.15.70:0/1194878151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2e1820 0x562e6c2e9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc0ffff640 1 -- 172.21.15.70:0/1194878151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e6c024620 con 0x562e6c2e2140 2024-06-04T05:22:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcc0ffff640 1 --2- 172.21.15.70:0/1194878151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e2140 0x562e6c2e9f60 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fcc180751d0 tx=0x7fcc0000f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72d44bfdd478cb5 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.383+0000 7fcbf77fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:22:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.384+0000 7fcbf77fe640 1 -- 172.21.15.70:0/1194878151 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc00011850 con 0x562e6c2e2140 2024-06-04T05:22:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.384+0000 7fcbf77fe640 1 -- 172.21.15.70:0/1194878151 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7fcc000119f0 con 0x562e6c2e2140 2024-06-04T05:22:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.384+0000 7fcc211d1740 1 -- 172.21.15.70:0/1194878151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc10004d20 con 0x562e6c2e2140 2024-06-04T05:22:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.384+0000 7fcbf77fe640 1 -- 172.21.15.70:0/1194878151 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fcc00011bf0 con 0x562e6c2e2140 2024-06-04T05:22:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.384+0000 7fcc211d1740 10 client.21825 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.384+0000 7fcc211d1740 1 -- 172.21.15.70:0/1194878151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x562e6c2fb3e0 con 0x562e6c2e2140 2024-06-04T05:22:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.384+0000 7fcbf77fe640 1 -- 172.21.15.70:0/1194878151 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcc10004d20 con 0x562e6c2e2140 2024-06-04T05:22:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.384+0000 7fcbf77fe640 1 -- 172.21.15.70:0/1194878151 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7fcc000103d0 con 0x562e6c2e2140 2024-06-04T05:22:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.384+0000 7fcc211d1740 10 client.21825 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.384+0000 7fcc211d1740 10 client.21825 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.384+0000 7fcc211d1740 20 client.21825 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.384+0000 7fcc211d1740 1 --2- 172.21.15.70:0/1194878151 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562e6c2fc8a0 0x562e6c2fec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.384+0000 7fcc211d1740 4 client.21825 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:22:32.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.384+0000 7fcc211d1740 1 -- 172.21.15.70:0/1194878151 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e6c255b80 con 0x562e6c2fc8a0 2024-06-04T05:22:32.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.385+0000 7fcc1ca5a640 1 --2- 172.21.15.70:0/1194878151 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562e6c2fc8a0 0x562e6c2fec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.385+0000 7fcbf77fe640 10 client.21825 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:32.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.385+0000 7fcc1ca5a640 1 --2- 172.21.15.70:0/1194878151 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562e6c2fc8a0 0x562e6c2fec90 crc :-1 s=READY pgs=525 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e43a6c90de750f82 server_cookie=b12553facec67a74 in_seq=0 out_seq=0 2024-06-04T05:22:32.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.386+0000 7fcbf77fe640 1 -- 172.21.15.70:0/1194878151 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562e6c255b80 con 0x562e6c2fc8a0 2024-06-04T05:22:32.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.386+0000 7fcbf77fe640 10 client.21825 handle_command_reply: tid=0 2024-06-04T05:22:32.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.386+0000 7fcc211d1740 1 client.21825 shutdown 2024-06-04T05:22:32.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.386+0000 7fcc211d1740 1 -- 172.21.15.70:0/1194878151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e2140 msgr2=0x562e6c2e9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.386+0000 7fcc211d1740 1 --2- 172.21.15.70:0/1194878151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e2140 0x562e6c2e9f60 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fcc180751d0 tx=0x7fcc0000f7f0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.386+0000 7fcc211d1740 1 -- 172.21.15.70:0/1194878151 shutdown_connections 2024-06-04T05:22:32.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.386+0000 7fcc211d1740 1 --2- 172.21.15.70:0/1194878151 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562e6c2fc8a0 0x562e6c2fec90 crc :-1 s=READY pgs=525 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.386+0000 7fcc211d1740 1 --2- 172.21.15.70:0/1194878151 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2eca00 0x562e6c2ea4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.386+0000 7fcc211d1740 1 --2- 172.21.15.70:0/1194878151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e2140 0x562e6c2e9f60 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.386+0000 7fcbf77fe640 0 client.21825 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.386+0000 7fcc211d1740 1 --2- 172.21.15.70:0/1194878151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2e1820 0x562e6c2e9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.386+0000 7fcc211d1740 1 -- 172.21.15.70:0/1194878151 >> 172.21.15.70:0/1194878151 conn(0x562e6c2ddd30 msgr2=0x562e6c2dcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.386+0000 7fcc211d1740 1 -- 172.21.15.70:0/1194878151 shutdown_connections 2024-06-04T05:22:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.386+0000 7fcc211d1740 1 -- 172.21.15.70:0/1194878151 wait complete. 2024-06-04T05:22:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.386+0000 7fcc211d1740 20 client.21825 trim_cache size 0 max 16384 2024-06-04T05:22:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc211d1740 1 Processor -- start 2024-06-04T05:22:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc211d1740 1 -- start start 2024-06-04T05:22:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc211d1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e1820 0x562e6c2ea0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc211d1740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2e2140 0x562e6c3045a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc211d1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2eca00 0x562e6c304ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc211d1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcc181b0860 con 0x562e6c2e1820 2024-06-04T05:22:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc211d1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc181b23a0 con 0x562e6c2eca00 2024-06-04T05:22:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc211d1740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcc181b0f30 con 0x562e6c2e2140 2024-06-04T05:22:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc1ca5a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e1820 0x562e6c2ea0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc1ca5a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e1820 0x562e6c2ea0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42666/0 (socket says 172.21.15.70:42666) 2024-06-04T05:22:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc1ca5a640 1 -- 172.21.15.70:0/1104998968 learned_addr learned my addr 172.21.15.70:0/1104998968 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc0ffff640 1 --2- 172.21.15.70:0/1104998968 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2e2140 0x562e6c3045a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc1ca5a640 1 -- 172.21.15.70:0/1104998968 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2e2140 msgr2=0x562e6c3045a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc1ca5a640 1 --2- 172.21.15.70:0/1104998968 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2e2140 0x562e6c3045a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc1ca5a640 1 -- 172.21.15.70:0/1104998968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2eca00 msgr2=0x562e6c304ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc1ca5a640 1 --2- 172.21.15.70:0/1104998968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2eca00 0x562e6c304ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc1ca5a640 1 -- 172.21.15.70:0/1104998968 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e6c2e1f70 con 0x562e6c2e1820 2024-06-04T05:22:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.401+0000 7fcc1ca5a640 1 --2- 172.21.15.70:0/1104998968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e1820 0x562e6c2ea0f0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fcc10005a90 tx=0x7fcc10009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=339991c4fffcaa2c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.402+0000 7fcc0dffb640 1 -- 172.21.15.70:0/1104998968 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1002d070 con 0x562e6c2e1820 2024-06-04T05:22:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.402+0000 7fcc0dffb640 1 -- 172.21.15.70:0/1104998968 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcc10005680 con 0x562e6c2e1820 2024-06-04T05:22:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.402+0000 7fcc0dffb640 1 -- 172.21.15.70:0/1104998968 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10016040 con 0x562e6c2e1820 2024-06-04T05:22:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.402+0000 7fcc211d1740 1 -- 172.21.15.70:0/1104998968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e1820 msgr2=0x562e6c2ea0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.402+0000 7fcc211d1740 1 --2- 172.21.15.70:0/1104998968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e1820 0x562e6c2ea0f0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fcc10005a90 tx=0x7fcc10009cb0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.402+0000 7fcc211d1740 1 -- 172.21.15.70:0/1104998968 shutdown_connections 2024-06-04T05:22:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.402+0000 7fcc211d1740 1 --2- 172.21.15.70:0/1104998968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2eca00 0x562e6c304ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.402+0000 7fcc211d1740 1 --2- 172.21.15.70:0/1104998968 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2e2140 0x562e6c3045a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.402+0000 7fcc211d1740 1 --2- 172.21.15.70:0/1104998968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e1820 0x562e6c2ea0f0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.402+0000 7fcc211d1740 1 -- 172.21.15.70:0/1104998968 >> 172.21.15.70:0/1104998968 conn(0x562e6c2ddd30 msgr2=0x562e6c2e99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.402+0000 7fcc211d1740 1 -- 172.21.15.70:0/1104998968 shutdown_connections 2024-06-04T05:22:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.402+0000 7fcc211d1740 1 -- 172.21.15.70:0/1104998968 wait complete. 2024-06-04T05:22:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.403+0000 7fcc211d1740 1 -- start start 2024-06-04T05:22:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.403+0000 7fcc211d1740 1 Processor -- start 2024-06-04T05:22:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.403+0000 7fcc211d1740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2e1820 0x562e6c2f0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.403+0000 7fcc211d1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e2140 0x562e6c2f12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.403+0000 7fcc211d1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2eca00 0x562e6c366470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.403+0000 7fcc211d1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcc181b1cd0 con 0x562e6c2e2140 2024-06-04T05:22:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.403+0000 7fcc211d1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc181b1600 con 0x562e6c2eca00 2024-06-04T05:22:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.403+0000 7fcc211d1740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcc181ae860 con 0x562e6c2e1820 2024-06-04T05:22:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.403+0000 7fcc211d1740 10 client.0 _ll_register_callbacks cb 0x7fcc0000a310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.403+0000 7fcc0ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e2140 0x562e6c2f12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.403+0000 7fcc0ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e2140 0x562e6c2f12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42678/0 (socket says 172.21.15.70:42678) 2024-06-04T05:22:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.403+0000 7fcc0ffff640 1 -- 172.21.15.70:0/2130069157 learned_addr learned my addr 172.21.15.70:0/2130069157 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.403+0000 7fcc1ca5a640 1 --2- 172.21.15.70:0/2130069157 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2e1820 0x562e6c2f0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcc0ffff640 1 -- 172.21.15.70:0/2130069157 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2e1820 msgr2=0x562e6c2f0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcc0ffff640 1 --2- 172.21.15.70:0/2130069157 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2e1820 0x562e6c2f0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcc0ffff640 1 -- 172.21.15.70:0/2130069157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2eca00 msgr2=0x562e6c366470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcc0ffff640 1 --2- 172.21.15.70:0/2130069157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2eca00 0x562e6c366470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcc0ffff640 1 -- 172.21.15.70:0/2130069157 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e6c3669b0 con 0x562e6c2e2140 2024-06-04T05:22:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcc1ca5a640 1 --2- 172.21.15.70:0/2130069157 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2e1820 0x562e6c2f0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:22:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcc0ffff640 1 --2- 172.21.15.70:0/2130069157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e2140 0x562e6c2f12b0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7fcc00006010 tx=0x7fcc00010d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa19a6de4c05e8c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcbf77fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:22:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcbf77fe640 1 -- 172.21.15.70:0/2130069157 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0002f020 con 0x562e6c2e2140 2024-06-04T05:22:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcbf77fe640 1 -- 172.21.15.70:0/2130069157 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7fcc00011850 con 0x562e6c2e2140 2024-06-04T05:22:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcc211d1740 1 -- 172.21.15.70:0/2130069157 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc10005680 con 0x562e6c2e2140 2024-06-04T05:22:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcbf77fe640 1 -- 172.21.15.70:0/2130069157 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fcc00026040 con 0x562e6c2e2140 2024-06-04T05:22:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcc211d1740 10 client.21831 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcc211d1740 1 -- 172.21.15.70:0/2130069157 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x562e6c367cf0 con 0x562e6c2e2140 2024-06-04T05:22:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcbf77fe640 1 -- 172.21.15.70:0/2130069157 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcc10005680 con 0x562e6c2e2140 2024-06-04T05:22:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcbf77fe640 1 -- 172.21.15.70:0/2130069157 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7fcc00004660 con 0x562e6c2e2140 2024-06-04T05:22:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcc211d1740 10 client.21831 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcc211d1740 10 client.21831 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:32.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.404+0000 7fcc211d1740 20 client.21831 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:32.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.405+0000 7fcc211d1740 1 --2- 172.21.15.70:0/2130069157 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562e6c3684f0 0x562e6c36a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.405+0000 7fcc211d1740 4 client.21831 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:22:32.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.405+0000 7fcc211d1740 1 -- 172.21.15.70:0/2130069157 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562e6c24f360 con 0x562e6c3684f0 2024-06-04T05:22:32.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.405+0000 7fcc1ca5a640 1 --2- 172.21.15.70:0/2130069157 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562e6c3684f0 0x562e6c36a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.405+0000 7fcbf77fe640 10 client.21831 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:32.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.405+0000 7fcc1ca5a640 1 --2- 172.21.15.70:0/2130069157 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562e6c3684f0 0x562e6c36a910 crc :-1 s=READY pgs=526 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86a4d162c2cc43a0 server_cookie=122f935f1a2a8ac4 in_seq=0 out_seq=0 2024-06-04T05:22:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.436+0000 7fcbf77fe640 1 -- 172.21.15.70:0/2130069157 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x562e6c24f360 con 0x562e6c3684f0 2024-06-04T05:22:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.436+0000 7fcbf77fe640 10 client.21831 handle_command_reply: tid=0 2024-06-04T05:22:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.436+0000 7fcc211d1740 1 client.21831 shutdown 2024-06-04T05:22:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.436+0000 7fcc211d1740 1 -- 172.21.15.70:0/2130069157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e2140 msgr2=0x562e6c2f12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.436+0000 7fcc211d1740 1 --2- 172.21.15.70:0/2130069157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e2140 0x562e6c2f12b0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7fcc00006010 tx=0x7fcc00010d10 comp rx=0 tx=0).stop 2024-06-04T05:22:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.436+0000 7fcc211d1740 1 -- 172.21.15.70:0/2130069157 shutdown_connections 2024-06-04T05:22:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.436+0000 7fcc211d1740 1 --2- 172.21.15.70:0/2130069157 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562e6c3684f0 0x562e6c36a910 crc :-1 s=READY pgs=526 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.436+0000 7fcc211d1740 1 --2- 172.21.15.70:0/2130069157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e6c2eca00 0x562e6c366470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.436+0000 7fcbf77fe640 0 client.21831 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.436+0000 7fcc211d1740 1 --2- 172.21.15.70:0/2130069157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e6c2e2140 0x562e6c2f12b0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.436+0000 7fcc211d1740 1 --2- 172.21.15.70:0/2130069157 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562e6c2e1820 0x562e6c2f0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.436+0000 7fcc211d1740 1 -- 172.21.15.70:0/2130069157 >> 172.21.15.70:0/2130069157 conn(0x562e6c2ddd30 msgr2=0x562e6c2e99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.436+0000 7fcc211d1740 1 -- 172.21.15.70:0/2130069157 shutdown_connections 2024-06-04T05:22:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.436+0000 7fcc211d1740 1 -- 172.21.15.70:0/2130069157 wait complete. 2024-06-04T05:22:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.437+0000 7fcc211d1740 20 client.21831 trim_cache size 0 max 16384 2024-06-04T05:22:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:22:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:22:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:22:32.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.437+0000 7fcc1e4bd640 1 -- 172.21.15.70:0/2219271124 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcc040b2ac0 msgr2=0x7fcc040b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.437+0000 7fcc1e4bd640 1 --2- 172.21.15.70:0/2219271124 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcc040b2ac0 0x7fcc040b4f80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fcc08100780 tx=0x7fcc08106b00 comp rx=0 tx=0).stop 2024-06-04T05:22:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.437+0000 7fcc1e4bd640 1 -- 172.21.15.70:0/2219271124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc18101350 msgr2=0x7fcc18196d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.437+0000 7fcc1e4bd640 1 --2- 172.21.15.70:0/2219271124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc18101350 0x7fcc18196d50 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800a340 tx=0x7fcc08004270 comp rx=0 tx=0).stop 2024-06-04T05:22:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.437+0000 7fcc1e4bd640 1 -- 172.21.15.70:0/2219271124 shutdown_connections 2024-06-04T05:22:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.437+0000 7fcc1e4bd640 1 --2- 172.21.15.70:0/2219271124 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcc040b2ac0 0x7fcc040b4f80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.437+0000 7fcc1e4bd640 1 --2- 172.21.15.70:0/2219271124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc18106420 0x7fcc18197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.437+0000 7fcc1e4bd640 1 --2- 172.21.15.70:0/2219271124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc181977d0 0x7fcc181a7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.437+0000 7fcc1e4bd640 1 --2- 172.21.15.70:0/2219271124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc18101350 0x7fcc18196d50 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.437+0000 7fcc1e4bd640 1 -- 172.21.15.70:0/2219271124 >> 172.21.15.70:0/2219271124 conn(0x7fcc18077eb0 msgr2=0x7fcc180ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.437+0000 7fcc1e4bd640 1 -- 172.21.15.70:0/2219271124 shutdown_connections 2024-06-04T05:22:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:32.437+0000 7fcc1e4bd640 1 -- 172.21.15.70:0/2219271124 wait complete. 2024-06-04T05:22:32.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84390 6.49 MB/sec execute 564 sec latency 24433.302 ms 2024-06-04T05:22:32.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.867+0000 7ff49e98f700 1 -- 172.21.15.70:0/315228379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff498102440 msgr2=0x7ff4981028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.867+0000 7ff49e98f700 1 --2- 172.21.15.70:0/315228379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff498102440 0x7ff4981028c0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7ff48c009990 tx=0x7ff48c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.867+0000 7ff49e98f700 1 -- 172.21.15.70:0/315228379 shutdown_connections 2024-06-04T05:22:32.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.867+0000 7ff49e98f700 1 --2- 172.21.15.70:0/315228379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff498102e00 0x7ff49810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.867+0000 7ff49e98f700 1 --2- 172.21.15.70:0/315228379 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff498108520 0x7ff498108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.867+0000 7ff49e98f700 1 --2- 172.21.15.70:0/315228379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff498102440 0x7ff4981028c0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.867+0000 7ff49e98f700 1 -- 172.21.15.70:0/315228379 >> 172.21.15.70:0/315228379 conn(0x7ff4980793f0 msgr2=0x7ff498079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:32.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.867+0000 7ff49e98f700 1 -- 172.21.15.70:0/315228379 shutdown_connections 2024-06-04T05:22:32.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.867+0000 7ff49e98f700 1 -- 172.21.15.70:0/315228379 wait complete. 2024-06-04T05:22:32.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff49e98f700 1 Processor -- start 2024-06-04T05:22:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff49e98f700 1 -- start start 2024-06-04T05:22:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff49e98f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff498102440 0x7ff49819a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff49e98f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff498102e00 0x7ff49819ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff49e98f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff498108520 0x7ff4981a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff49e98f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff498112480 con 0x7ff498108520 2024-06-04T05:22:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff49e98f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff498112300 con 0x7ff498102440 2024-06-04T05:22:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff49e98f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff498112600 con 0x7ff498102e00 2024-06-04T05:22:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff49cf2c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff498108520 0x7ff4981a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff49cf2c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff498108520 0x7ff4981a5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42706/0 (socket says 172.21.15.70:42706) 2024-06-04T05:22:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff49cf2c700 1 -- 172.21.15.70:0/3367709167 learned_addr learned my addr 172.21.15.70:0/3367709167 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:32.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff497fff700 1 --2- 172.21.15.70:0/3367709167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff498102440 0x7ff49819a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff4977fe700 1 --2- 172.21.15.70:0/3367709167 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff498102e00 0x7ff49819ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff49cf2c700 1 -- 172.21.15.70:0/3367709167 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff498102e00 msgr2=0x7ff49819ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff49cf2c700 1 --2- 172.21.15.70:0/3367709167 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff498102e00 0x7ff49819ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff49cf2c700 1 -- 172.21.15.70:0/3367709167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff498102440 msgr2=0x7ff49819a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff49cf2c700 1 --2- 172.21.15.70:0/3367709167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff498102440 0x7ff49819a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff49cf2c700 1 -- 172.21.15.70:0/3367709167 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4981a5ea0 con 0x7ff498108520 2024-06-04T05:22:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff4977fe700 1 --2- 172.21.15.70:0/3367709167 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff498102e00 0x7ff49819ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:22:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.868+0000 7ff497fff700 1 --2- 172.21.15.70:0/3367709167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff498102440 0x7ff49819a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:22:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.869+0000 7ff49cf2c700 1 --2- 172.21.15.70:0/3367709167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff498108520 0x7ff4981a5960 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7ff49000d980 tx=0x7ff49000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.869+0000 7ff4957fa700 1 -- 172.21.15.70:0/3367709167 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff490017070 con 0x7ff498108520 2024-06-04T05:22:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.869+0000 7ff4957fa700 1 -- 172.21.15.70:0/3367709167 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff490009dd0 con 0x7ff498108520 2024-06-04T05:22:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.869+0000 7ff49e98f700 1 -- 172.21.15.70:0/3367709167 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4981a60d0 con 0x7ff498108520 2024-06-04T05:22:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.869+0000 7ff4957fa700 1 -- 172.21.15.70:0/3367709167 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff490005260 con 0x7ff498108520 2024-06-04T05:22:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.869+0000 7ff49e98f700 1 -- 172.21.15.70:0/3367709167 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4981a65c0 con 0x7ff498108520 2024-06-04T05:22:32.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.872+0000 7ff4957fa700 1 -- 172.21.15.70:0/3367709167 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff490009940 con 0x7ff498108520 2024-06-04T05:22:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.873+0000 7ff4957fa700 1 --2- 172.21.15.70:0/3367709167 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4800b29f0 0x7ff4800b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.873+0000 7ff4a9224b80 1 Processor -- start 2024-06-04T05:22:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.873+0000 7ff4a9224b80 1 -- start start 2024-06-04T05:22:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.873+0000 7ff497fff700 1 --2- 172.21.15.70:0/3367709167 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4800b29f0 0x7ff4800b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.873+0000 7ff4957fa700 1 -- 172.21.15.70:0/3367709167 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7ff4900e2680 con 0x7ff498108520 2024-06-04T05:22:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.873+0000 7ff4a9224b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf95a720 0x5588cf95ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.873+0000 7ff4a9224b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf956070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.873+0000 7ff4a9224b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf9565b0 0x5588cf962620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.873+0000 7ff4a9224b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5588cf8b3e70 con 0x5588cf955bf0 2024-06-04T05:22:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.873+0000 7ff4a9224b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5588cf8a3a10 con 0x5588cf95a720 2024-06-04T05:22:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.873+0000 7ff4a9224b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5588cf8a7a50 con 0x5588cf9565b0 2024-06-04T05:22:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.873+0000 7ff4977fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf956070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.873+0000 7ff4977fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf956070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42710/0 (socket says 172.21.15.70:42710) 2024-06-04T05:22:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.873+0000 7ff4977fe700 1 -- 172.21.15.70:0/2308356452 learned_addr learned my addr 172.21.15.70:0/2308356452 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.873+0000 7ff497fff700 1 --2- 172.21.15.70:0/3367709167 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4800b29f0 0x7ff4800b4eb0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff48800e350 tx=0x7ff488009040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.873+0000 7ff497fff700 1 --2- 172.21.15.70:0/2308356452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf95a720 0x5588cf95ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.874+0000 7ff4977fe700 1 -- 172.21.15.70:0/2308356452 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf9565b0 msgr2=0x5588cf962620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.874+0000 7ff4977fe700 1 --2- 172.21.15.70:0/2308356452 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf9565b0 0x5588cf962620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.874+0000 7ff4977fe700 1 -- 172.21.15.70:0/2308356452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf95a720 msgr2=0x5588cf95ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.874+0000 7ff4977fe700 1 --2- 172.21.15.70:0/2308356452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf95a720 0x5588cf95ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.874+0000 7ff4977fe700 1 -- 172.21.15.70:0/2308356452 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588cf84d0b0 con 0x5588cf955bf0 2024-06-04T05:22:32.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.874+0000 7ff497fff700 1 --2- 172.21.15.70:0/2308356452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf95a720 0x5588cf95ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:22:32.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.874+0000 7ff4977fe700 1 --2- 172.21.15.70:0/2308356452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf956070 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7ff48c009990 tx=0x7ff48c003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2769b5167c3987bb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:32.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.874+0000 7ff47e7fc700 1 -- 172.21.15.70:0/2308356452 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48c027070 con 0x5588cf955bf0 2024-06-04T05:22:32.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.874+0000 7ff47e7fc700 1 -- 172.21.15.70:0/2308356452 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff48c005070 con 0x5588cf955bf0 2024-06-04T05:22:32.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.874+0000 7ff47e7fc700 1 -- 172.21.15.70:0/2308356452 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48c003a70 con 0x5588cf955bf0 2024-06-04T05:22:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.875+0000 7ff4a9224b80 1 -- 172.21.15.70:0/2308356452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 msgr2=0x5588cf956070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.875+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/2308356452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf956070 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7ff48c009990 tx=0x7ff48c003680 comp rx=0 tx=0).stop 2024-06-04T05:22:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.875+0000 7ff4a9224b80 1 -- 172.21.15.70:0/2308356452 shutdown_connections 2024-06-04T05:22:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.875+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/2308356452 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf9565b0 0x5588cf962620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.875+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/2308356452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf95a720 0x5588cf95ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.875+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/2308356452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf956070 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.875+0000 7ff4a9224b80 1 -- 172.21.15.70:0/2308356452 >> 172.21.15.70:0/2308356452 conn(0x5588cf881340 msgr2=0x5588cf9508e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.875+0000 7ff4a9224b80 1 -- 172.21.15.70:0/2308356452 shutdown_connections 2024-06-04T05:22:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.875+0000 7ff4a9224b80 1 -- 172.21.15.70:0/2308356452 wait complete. 2024-06-04T05:22:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.875+0000 7ff4a9224b80 1 -- start start 2024-06-04T05:22:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.875+0000 7ff4a9224b80 1 Processor -- start 2024-06-04T05:22:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.875+0000 7ff4a9224b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf962360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.875+0000 7ff4a9224b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf9565b0 0x5588cf95f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.875+0000 7ff4a9224b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf95a720 0x5588cf95fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.875+0000 7ff4a9224b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5588cf7aa0a0 con 0x5588cf955bf0 2024-06-04T05:22:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff4a9224b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5588cf80bea0 con 0x5588cf95a720 2024-06-04T05:22:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff4a9224b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5588cf962c20 con 0x5588cf9565b0 2024-06-04T05:22:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff497fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf962360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff4a9224b80 10 client.0 _ll_register_callbacks cb 0x5588cf83ea50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff497fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf962360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42722/0 (socket says 172.21.15.70:42722) 2024-06-04T05:22:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff497fff700 1 -- 172.21.15.70:0/2310626366 learned_addr learned my addr 172.21.15.70:0/2310626366 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff4977fe700 1 --2- 172.21.15.70:0/2310626366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf9565b0 0x5588cf95f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff497fff700 1 -- 172.21.15.70:0/2310626366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf9565b0 msgr2=0x5588cf95f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff497fff700 1 --2- 172.21.15.70:0/2310626366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf9565b0 0x5588cf95f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff497fff700 1 -- 172.21.15.70:0/2310626366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf95a720 msgr2=0x5588cf95fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff497fff700 1 --2- 172.21.15.70:0/2310626366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf95a720 0x5588cf95fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff497fff700 1 -- 172.21.15.70:0/2310626366 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588cf7f39d0 con 0x5588cf955bf0 2024-06-04T05:22:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff4977fe700 1 --2- 172.21.15.70:0/2310626366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf9565b0 0x5588cf95f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:22:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff497fff700 1 --2- 172.21.15.70:0/2310626366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf962360 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7ff488015af0 tx=0x7ff488004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71813fba72fa9815 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff4777fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:22:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff4777fe700 1 -- 172.21.15.70:0/2310626366 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488013a80 con 0x5588cf955bf0 2024-06-04T05:22:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff4777fe700 1 -- 172.21.15.70:0/2310626366 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7ff488013c20 con 0x5588cf955bf0 2024-06-04T05:22:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff4a9224b80 1 -- 172.21.15.70:0/2310626366 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff48c005070 con 0x5588cf955bf0 2024-06-04T05:22:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff4777fe700 1 -- 172.21.15.70:0/2310626366 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7ff488013e20 con 0x5588cf955bf0 2024-06-04T05:22:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff4a9224b80 10 client.21843 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.876+0000 7ff4a9224b80 1 -- 172.21.15.70:0/2310626366 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5588cf85d860 con 0x5588cf955bf0 2024-06-04T05:22:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.877+0000 7ff4777fe700 1 -- 172.21.15.70:0/2310626366 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff48c005070 con 0x5588cf955bf0 2024-06-04T05:22:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.877+0000 7ff4777fe700 1 -- 172.21.15.70:0/2310626366 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7ff488014070 con 0x5588cf955bf0 2024-06-04T05:22:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.877+0000 7ff4a9224b80 10 client.21843 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.877+0000 7ff4a9224b80 10 client.21843 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:22:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.877+0000 7ff4a9224b80 20 client.21843 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.877+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/2310626366 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5588cf96f8b0 0x5588cf971d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.877+0000 7ff4a9224b80 4 client.21843 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:22:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.877+0000 7ff4a9224b80 1 -- 172.21.15.70:0/2310626366 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588cf8b5dc0 con 0x5588cf96f8b0 2024-06-04T05:22:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.877+0000 7ff4977fe700 1 --2- 172.21.15.70:0/2310626366 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5588cf96f8b0 0x5588cf971d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.878+0000 7ff4977fe700 1 --2- 172.21.15.70:0/2310626366 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5588cf96f8b0 0x5588cf971d30 crc :-1 s=READY pgs=527 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfdaa11075a165f7 server_cookie=585a5d0bc7bd6ab2 in_seq=0 out_seq=0 2024-06-04T05:22:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.878+0000 7ff4777fe700 10 client.21843 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.878+0000 7ff4777fe700 1 -- 172.21.15.70:0/2310626366 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5588cf8b5dc0 con 0x5588cf96f8b0 2024-06-04T05:22:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.878+0000 7ff4777fe700 10 client.21843 handle_command_reply: tid=0 2024-06-04T05:22:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.878+0000 7ff4a9224b80 1 client.21843 shutdown 2024-06-04T05:22:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.879+0000 7ff4a9224b80 1 -- 172.21.15.70:0/2310626366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 msgr2=0x5588cf962360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.879+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/2310626366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf962360 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7ff488015af0 tx=0x7ff488004b40 comp rx=0 tx=0).stop 2024-06-04T05:22:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.879+0000 7ff4a9224b80 1 -- 172.21.15.70:0/2310626366 shutdown_connections 2024-06-04T05:22:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.879+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/2310626366 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5588cf96f8b0 0x5588cf971d30 crc :-1 s=READY pgs=527 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.879+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/2310626366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf95a720 0x5588cf95fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.879+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/2310626366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf962360 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.879+0000 7ff4777fe700 0 client.21843 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:32.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.879+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/2310626366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf9565b0 0x5588cf95f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.879+0000 7ff4a9224b80 1 -- 172.21.15.70:0/2310626366 >> 172.21.15.70:0/2310626366 conn(0x5588cf881340 msgr2=0x5588cf954be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:32.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.879+0000 7ff4a9224b80 1 -- 172.21.15.70:0/2310626366 shutdown_connections 2024-06-04T05:22:32.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.879+0000 7ff4a9224b80 1 -- 172.21.15.70:0/2310626366 wait complete. 2024-06-04T05:22:32.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.879+0000 7ff4a9224b80 20 client.21843 trim_cache size 0 max 16384 2024-06-04T05:22:32.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.899+0000 7ff4a9224b80 1 Processor -- start 2024-06-04T05:22:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.899+0000 7ff4a9224b80 1 -- start start 2024-06-04T05:22:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.899+0000 7ff4a9224b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf94ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.899+0000 7ff4a9224b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf9565b0 0x5588cf96bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.899+0000 7ff4a9224b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf95a720 0x5588cf968250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.899+0000 7ff4a9224b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff4981add40 con 0x5588cf955bf0 2024-06-04T05:22:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.899+0000 7ff4a9224b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff4981af880 con 0x5588cf9565b0 2024-06-04T05:22:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.899+0000 7ff4a9224b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff4981ae410 con 0x5588cf95a720 2024-06-04T05:22:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.899+0000 7ff497fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf94ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.899+0000 7ff497fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf94ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42724/0 (socket says 172.21.15.70:42724) 2024-06-04T05:22:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.899+0000 7ff497fff700 1 -- 172.21.15.70:0/901275136 learned_addr learned my addr 172.21.15.70:0/901275136 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.899+0000 7ff497fff700 1 -- 172.21.15.70:0/901275136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf95a720 msgr2=0x5588cf968250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:32.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.899+0000 7ff497fff700 1 --2- 172.21.15.70:0/901275136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf95a720 0x5588cf968250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.899+0000 7ff497fff700 1 -- 172.21.15.70:0/901275136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf9565b0 msgr2=0x5588cf96bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:32.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.899+0000 7ff497fff700 1 --2- 172.21.15.70:0/901275136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf9565b0 0x5588cf96bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.899+0000 7ff497fff700 1 -- 172.21.15.70:0/901275136 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588cf95dbf0 con 0x5588cf955bf0 2024-06-04T05:22:32.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.899+0000 7ff497fff700 1 --2- 172.21.15.70:0/901275136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf94ae70 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7ff488008fd0 tx=0x7ff4880137c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08917a3cebcd5be server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:32.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.900+0000 7ff47dffb700 1 -- 172.21.15.70:0/901275136 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488037070 con 0x5588cf955bf0 2024-06-04T05:22:32.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.900+0000 7ff47dffb700 1 -- 172.21.15.70:0/901275136 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff488025070 con 0x5588cf955bf0 2024-06-04T05:22:32.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.900+0000 7ff47dffb700 1 -- 172.21.15.70:0/901275136 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488013c80 con 0x5588cf955bf0 2024-06-04T05:22:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.900+0000 7ff4a9224b80 1 -- 172.21.15.70:0/901275136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 msgr2=0x5588cf94ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.900+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/901275136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf94ae70 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7ff488008fd0 tx=0x7ff4880137c0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.900+0000 7ff4a9224b80 1 -- 172.21.15.70:0/901275136 shutdown_connections 2024-06-04T05:22:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.900+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/901275136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf95a720 0x5588cf968250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.900+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/901275136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf955bf0 0x5588cf94ae70 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.900+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/901275136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf9565b0 0x5588cf96bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.900+0000 7ff4a9224b80 1 -- 172.21.15.70:0/901275136 >> 172.21.15.70:0/901275136 conn(0x5588cf881340 msgr2=0x5588cf95b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.900+0000 7ff4a9224b80 1 -- 172.21.15.70:0/901275136 shutdown_connections 2024-06-04T05:22:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.900+0000 7ff4a9224b80 1 -- 172.21.15.70:0/901275136 wait complete. 2024-06-04T05:22:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff4a9224b80 1 -- start start 2024-06-04T05:22:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff4a9224b80 1 Processor -- start 2024-06-04T05:22:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff4a9224b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf955bf0 0x5588cf977af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff4a9224b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf9565b0 0x5588cf9765f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff4a9224b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf95a720 0x5588cf976b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff4a9224b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff4981af1b0 con 0x5588cf95a720 2024-06-04T05:22:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff4a9224b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff4981aeae0 con 0x5588cf955bf0 2024-06-04T05:22:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff4a9224b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff4981abd00 con 0x5588cf9565b0 2024-06-04T05:22:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff4a9224b80 10 client.0 _ll_register_callbacks cb 0x5588cf83ea50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff49cf2c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf95a720 0x5588cf976b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff49cf2c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf95a720 0x5588cf976b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42728/0 (socket says 172.21.15.70:42728) 2024-06-04T05:22:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff49cf2c700 1 -- 172.21.15.70:0/3700553480 learned_addr learned my addr 172.21.15.70:0/3700553480 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff497fff700 1 --2- 172.21.15.70:0/3700553480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf955bf0 0x5588cf977af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff4977fe700 1 --2- 172.21.15.70:0/3700553480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf9565b0 0x5588cf9765f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff49cf2c700 1 -- 172.21.15.70:0/3700553480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf9565b0 msgr2=0x5588cf9765f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff49cf2c700 1 --2- 172.21.15.70:0/3700553480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf9565b0 0x5588cf9765f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff49cf2c700 1 -- 172.21.15.70:0/3700553480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf955bf0 msgr2=0x5588cf977af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff49cf2c700 1 --2- 172.21.15.70:0/3700553480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf955bf0 0x5588cf977af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff49cf2c700 1 -- 172.21.15.70:0/3700553480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588cf977170 con 0x5588cf95a720 2024-06-04T05:22:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff4977fe700 1 --2- 172.21.15.70:0/3700553480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf9565b0 0x5588cf9765f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:22:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff497fff700 1 --2- 172.21.15.70:0/3700553480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf955bf0 0x5588cf977af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:22:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff49cf2c700 1 --2- 172.21.15.70:0/3700553480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf95a720 0x5588cf976b30 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7ff4900dfe00 tx=0x7ff490004bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10270f36a850a3d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.901+0000 7ff4777fe700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:22:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.902+0000 7ff4777fe700 1 -- 172.21.15.70:0/3700553480 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4900aa030 con 0x5588cf95a720 2024-06-04T05:22:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.902+0000 7ff4777fe700 1 -- 172.21.15.70:0/3700553480 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7ff4900cc070 con 0x5588cf95a720 2024-06-04T05:22:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.902+0000 7ff4a9224b80 1 -- 172.21.15.70:0/3700553480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff488025070 con 0x5588cf95a720 2024-06-04T05:22:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.902+0000 7ff4777fe700 1 -- 172.21.15.70:0/3700553480 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7ff4900cc270 con 0x5588cf95a720 2024-06-04T05:22:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.902+0000 7ff4a9224b80 10 client.21849 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:32.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.902+0000 7ff4a9224b80 1 -- 172.21.15.70:0/3700553480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5588cf9df400 con 0x5588cf95a720 2024-06-04T05:22:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.902+0000 7ff4777fe700 1 -- 172.21.15.70:0/3700553480 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff488025070 con 0x5588cf95a720 2024-06-04T05:22:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.902+0000 7ff4777fe700 1 -- 172.21.15.70:0/3700553480 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7ff4900cac70 con 0x5588cf95a720 2024-06-04T05:22:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.902+0000 7ff4a9224b80 10 client.21849 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.902+0000 7ff4a9224b80 10 client.21849 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:22:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.902+0000 7ff4a9224b80 20 client.21849 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.902+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/3700553480 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5588cf9dfb10 0x5588cf9e1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.902+0000 7ff4a9224b80 4 client.21849 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "582ae688", "release": true, "await": true}] 2024-06-04T05:22:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.902+0000 7ff4a9224b80 1 -- 172.21.15.70:0/3700553480 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "set_id": "582ae688", "release": true, "await": true}) -- 0x5588cf8af5a0 con 0x5588cf9dfb10 2024-06-04T05:22:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.902+0000 7ff497fff700 1 --2- 172.21.15.70:0/3700553480 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5588cf9dfb10 0x5588cf9e1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.903+0000 7ff497fff700 1 --2- 172.21.15.70:0/3700553480 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5588cf9dfb10 0x5588cf9e1f30 crc :-1 s=READY pgs=528 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=355b942449483a15 server_cookie=a2e2d72429ac50ad in_seq=0 out_seq=0 2024-06-04T05:22:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.903+0000 7ff4777fe700 10 client.21849 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:32.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.933+0000 7ff4777fe700 1 -- 172.21.15.70:0/3700553480 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5588cf8af5a0 con 0x5588cf9dfb10 2024-06-04T05:22:32.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.933+0000 7ff4777fe700 10 client.21849 handle_command_reply: tid=0 2024-06-04T05:22:32.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.933+0000 7ff4a9224b80 1 client.21849 shutdown 2024-06-04T05:22:32.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.934+0000 7ff4a9224b80 1 -- 172.21.15.70:0/3700553480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf95a720 msgr2=0x5588cf976b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.934+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/3700553480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf95a720 0x5588cf976b30 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7ff4900dfe00 tx=0x7ff490004bf0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.935+0000 7ff4a9224b80 1 -- 172.21.15.70:0/3700553480 shutdown_connections 2024-06-04T05:22:32.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.935+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/3700553480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5588cf95a720 0x5588cf976b30 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.935+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/3700553480 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5588cf9dfb10 0x5588cf9e1f30 crc :-1 s=READY pgs=528 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.935+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/3700553480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588cf955bf0 0x5588cf977af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.935+0000 7ff4a9224b80 1 --2- 172.21.15.70:0/3700553480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5588cf9565b0 0x5588cf9765f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.935+0000 7ff4a9224b80 1 -- 172.21.15.70:0/3700553480 >> 172.21.15.70:0/3700553480 conn(0x5588cf881340 msgr2=0x5588cf95bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:32.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.936+0000 7ff4777fe700 0 client.21849 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:32.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.936+0000 7ff4a9224b80 1 -- 172.21.15.70:0/3700553480 shutdown_connections 2024-06-04T05:22:32.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.937+0000 7ff4a9224b80 1 -- 172.21.15.70:0/3700553480 wait complete. 2024-06-04T05:22:32.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.938+0000 7ff4a9224b80 20 client.21849 trim_cache size 0 max 16384 2024-06-04T05:22:32.941 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:22:32.941 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 67, 2024-06-04T05:22:32.941 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:22:32.941 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 80, 2024-06-04T05:22:32.942 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": { 2024-06-04T05:22:32.942 INFO:teuthology.orchestra.run.smithi070.stdout: "582ae688": { 2024-06-04T05:22:32.942 INFO:teuthology.orchestra.run.smithi070.stdout: "version": 80, 2024-06-04T05:22:32.942 INFO:teuthology.orchestra.run.smithi070.stdout: "age_ref": 0.0, 2024-06-04T05:22:32.942 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:22:32.942 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:22:32.942 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:22:32.942 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:22:32.942 INFO:teuthology.orchestra.run.smithi070.stdout: "timeout": 90.0, 2024-06-04T05:22:32.942 INFO:teuthology.orchestra.run.smithi070.stdout: "expiration": 150.0, 2024-06-04T05:22:32.942 INFO:teuthology.orchestra.run.smithi070.stdout: "members": { 2024-06-04T05:22:32.942 INFO:teuthology.orchestra.run.smithi070.stdout: "file:/": { 2024-06-04T05:22:32.942 INFO:teuthology.orchestra.run.smithi070.stdout: "excluded": false, 2024-06-04T05:22:32.943 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-06-04T05:22:32.943 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RELEASED", 2024-06-04T05:22:32.943 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0 2024-06-04T05:22:32.943 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:22:32.943 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:22:32.943 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:22:32.943 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:22:32.943 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:22:32.943 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:22:32.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.940+0000 7ff49e98f700 1 -- 172.21.15.70:0/3367709167 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4800b29f0 msgr2=0x7ff4800b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.940+0000 7ff49e98f700 1 --2- 172.21.15.70:0/3367709167 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4800b29f0 0x7ff4800b4eb0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff48800e350 tx=0x7ff488009040 comp rx=0 tx=0).stop 2024-06-04T05:22:32.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.940+0000 7ff49e98f700 1 -- 172.21.15.70:0/3367709167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff498108520 msgr2=0x7ff4981a5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:32.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.940+0000 7ff49e98f700 1 --2- 172.21.15.70:0/3367709167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff498108520 0x7ff4981a5960 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7ff49000d980 tx=0x7ff49000dd40 comp rx=0 tx=0).stop 2024-06-04T05:22:32.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.941+0000 7ff49e98f700 1 -- 172.21.15.70:0/3367709167 shutdown_connections 2024-06-04T05:22:32.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.941+0000 7ff49e98f700 1 --2- 172.21.15.70:0/3367709167 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4800b29f0 0x7ff4800b4eb0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.941+0000 7ff49e98f700 1 --2- 172.21.15.70:0/3367709167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff498108520 0x7ff4981a5960 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.941+0000 7ff49e98f700 1 --2- 172.21.15.70:0/3367709167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff498102440 0x7ff49819a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.941+0000 7ff49e98f700 1 --2- 172.21.15.70:0/3367709167 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff498102e00 0x7ff49819ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:32.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.941+0000 7ff49e98f700 1 -- 172.21.15.70:0/3367709167 >> 172.21.15.70:0/3367709167 conn(0x7ff4980793f0 msgr2=0x7ff4981005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:32.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.941+0000 7ff49e98f700 1 -- 172.21.15.70:0/3367709167 shutdown_connections 2024-06-04T05:22:32.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:32.941+0000 7ff49e98f700 1 -- 172.21.15.70:0/3367709167 wait complete. 2024-06-04T05:22:33.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:33 smithi070 ceph-mon[32631]: pgmap v800: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 54 KiB/s wr, 38 op/s 2024-06-04T05:22:33.673 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '582ae688', total seconds elapsed: 35.5 2024-06-04T05:22:33.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:33 smithi112 ceph-mon[36453]: pgmap v800: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 54 KiB/s wr, 38 op/s 2024-06-04T05:22:33.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:33 smithi098 ceph-mon[35569]: pgmap v800: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 54 KiB/s wr, 38 op/s 2024-06-04T05:22:33.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84544 6.49 MB/sec execute 565 sec latency 24520.066 ms 2024-06-04T05:22:34.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84718 6.50 MB/sec execute 566 sec latency 95.705 ms 2024-06-04T05:22:35.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:35 smithi112 ceph-mon[36453]: pgmap v801: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 10 op/s 2024-06-04T05:22:35.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:35 smithi098 ceph-mon[35569]: pgmap v801: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 10 op/s 2024-06-04T05:22:35.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 84875 6.49 MB/sec execute 567 sec latency 66.080 ms 2024-06-04T05:22:35.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:35 smithi070 ceph-mon[32631]: pgmap v801: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 10 op/s 2024-06-04T05:22:36.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 85053 6.50 MB/sec execute 568 sec latency 66.438 ms 2024-06-04T05:22:37.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:37 smithi112 ceph-mon[36453]: pgmap v802: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 1.5 MiB/s wr, 92 op/s 2024-06-04T05:22:37.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:37 smithi098 ceph-mon[35569]: pgmap v802: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 1.5 MiB/s wr, 92 op/s 2024-06-04T05:22:37.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 85230 6.50 MB/sec execute 569 sec latency 63.391 ms 2024-06-04T05:22:37.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:37 smithi070 ceph-mon[32631]: pgmap v802: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 1.5 MiB/s wr, 92 op/s 2024-06-04T05:22:38.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 85422 6.51 MB/sec execute 570 sec latency 53.241 ms 2024-06-04T05:22:39.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:39 smithi112 ceph-mon[36453]: pgmap v803: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 1.5 MiB/s wr, 86 op/s 2024-06-04T05:22:39.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:39 smithi098 ceph-mon[35569]: pgmap v803: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 1.5 MiB/s wr, 86 op/s 2024-06-04T05:22:39.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 85620 6.52 MB/sec execute 571 sec latency 63.025 ms 2024-06-04T05:22:39.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:39 smithi070 ceph-mon[32631]: pgmap v803: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 1.5 MiB/s wr, 86 op/s 2024-06-04T05:22:40.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 85785 6.52 MB/sec execute 572 sec latency 72.073 ms 2024-06-04T05:22:41.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:41 smithi112 ceph-mon[36453]: pgmap v804: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 1.5 MiB/s wr, 86 op/s 2024-06-04T05:22:41.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:41 smithi070 ceph-mon[32631]: pgmap v804: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 1.5 MiB/s wr, 86 op/s 2024-06-04T05:22:41.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:41 smithi098 ceph-mon[35569]: pgmap v804: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 1.5 MiB/s wr, 86 op/s 2024-06-04T05:22:41.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 85952 6.52 MB/sec execute 573 sec latency 81.890 ms 2024-06-04T05:22:42.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 86147 6.53 MB/sec execute 574 sec latency 62.506 ms 2024-06-04T05:22:43.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:43 smithi112 ceph-mon[36453]: pgmap v805: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 217 op/s 2024-06-04T05:22:43.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:43 smithi098 ceph-mon[35569]: pgmap v805: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 217 op/s 2024-06-04T05:22:43.777 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:43 smithi070 ceph-mon[32631]: pgmap v805: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 217 op/s 2024-06-04T05:22:43.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 86333 6.54 MB/sec execute 575 sec latency 103.828 ms 2024-06-04T05:22:44.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 86481 6.54 MB/sec execute 576 sec latency 97.596 ms 2024-06-04T05:22:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.563+0000 7f2c9bd0a640 1 -- 172.21.15.70:0/1185101249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c94102c80 msgr2=0x7f2c941067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.563+0000 7f2c9bd0a640 1 --2- 172.21.15.70:0/1185101249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c94102c80 0x7f2c941067a0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f2c84009900 tx=0x7f2c8402a410 comp rx=0 tx=0).stop 2024-06-04T05:22:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.563+0000 7f2c9bd0a640 1 -- 172.21.15.70:0/1185101249 shutdown_connections 2024-06-04T05:22:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.563+0000 7f2c9bd0a640 1 --2- 172.21.15.70:0/1185101249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c94106e80 0x7f2c9410ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.563+0000 7f2c9bd0a640 1 --2- 172.21.15.70:0/1185101249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c94102c80 0x7f2c941067a0 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.563+0000 7f2c9bd0a640 1 --2- 172.21.15.70:0/1185101249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c941022d0 0x7f2c941026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.563+0000 7f2c9bd0a640 1 -- 172.21.15.70:0/1185101249 >> 172.21.15.70:0/1185101249 conn(0x7f2c94077ee0 msgr2=0x7f2c940ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.563+0000 7f2c9bd0a640 1 -- 172.21.15.70:0/1185101249 shutdown_connections 2024-06-04T05:22:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.563+0000 7f2c9bd0a640 1 -- 172.21.15.70:0/1185101249 wait complete. 2024-06-04T05:22:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.564+0000 7f2c9bd0a640 1 Processor -- start 2024-06-04T05:22:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.564+0000 7f2c9bd0a640 1 -- start start 2024-06-04T05:22:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.564+0000 7f2c9bd0a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c941022d0 0x7f2c9410f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.564+0000 7f2c9bd0a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c94106e80 0x7f2c9410f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.564+0000 7f2c9bd0a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c9410fe60 0x7f2c941a74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.564+0000 7f2c9bd0a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2c94112da0 con 0x7f2c941022d0 2024-06-04T05:22:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.564+0000 7f2c9bd0a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2c94112c20 con 0x7f2c94106e80 2024-06-04T05:22:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.564+0000 7f2c9bd0a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2c94112f20 con 0x7f2c9410fe60 2024-06-04T05:22:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.564+0000 7f2c99aa6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c941022d0 0x7f2c9410f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.564+0000 7f2c992a5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c94106e80 0x7f2c9410f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.564+0000 7f2c99aa6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c941022d0 0x7f2c9410f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59692/0 (socket says 172.21.15.70:59692) 2024-06-04T05:22:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.564+0000 7f2c9a2a7640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c9410fe60 0x7f2c941a74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.564+0000 7f2c99aa6640 1 -- 172.21.15.70:0/3330208911 learned_addr learned my addr 172.21.15.70:0/3330208911 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.565+0000 7f2c9a2a7640 1 -- 172.21.15.70:0/3330208911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c94106e80 msgr2=0x7f2c9410f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.565+0000 7f2c9a2a7640 1 --2- 172.21.15.70:0/3330208911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c94106e80 0x7f2c9410f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.565+0000 7f2c9a2a7640 1 -- 172.21.15.70:0/3330208911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c941022d0 msgr2=0x7f2c9410f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.565+0000 7f2c9a2a7640 1 --2- 172.21.15.70:0/3330208911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c941022d0 0x7f2c9410f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.565+0000 7f2c9a2a7640 1 -- 172.21.15.70:0/3330208911 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c941a7bd0 con 0x7f2c9410fe60 2024-06-04T05:22:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.565+0000 7f2c9a2a7640 1 --2- 172.21.15.70:0/3330208911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c9410fe60 0x7f2c941a74d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009fd0 tx=0x7f2c8c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.565+0000 7f2c8affd640 1 -- 172.21.15.70:0/3330208911 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c00ce20 con 0x7f2c9410fe60 2024-06-04T05:22:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.565+0000 7f2c8affd640 1 -- 172.21.15.70:0/3330208911 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2c8c005020 con 0x7f2c9410fe60 2024-06-04T05:22:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.565+0000 7f2c9bd0a640 1 -- 172.21.15.70:0/3330208911 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c941a7ec0 con 0x7f2c9410fe60 2024-06-04T05:22:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.565+0000 7f2c8affd640 1 -- 172.21.15.70:0/3330208911 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c010690 con 0x7f2c9410fe60 2024-06-04T05:22:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.565+0000 7f2c9bd0a640 1 -- 172.21.15.70:0/3330208911 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c941a8330 con 0x7f2c9410fe60 2024-06-04T05:22:45.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.568+0000 7f2c8affd640 1 -- 172.21.15.70:0/3330208911 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f2c8c004a40 con 0x7f2c9410fe60 2024-06-04T05:22:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.569+0000 7f2c9ea28740 1 Processor -- start 2024-06-04T05:22:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.569+0000 7f2c9ea28740 1 -- start start 2024-06-04T05:22:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c8affd640 1 --2- 172.21.15.70:0/3330208911 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2c800b2ac0 0x7f2c800b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c9ea28740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227c820 0x55fbe227cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c9ea28740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227d140 0x55fbe22874c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c9ea28740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 0x55fbe2289df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c9ea28740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fbe21eec30 con 0x55fbe227c820 2024-06-04T05:22:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c9ea28740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fbe21de7d0 con 0x55fbe2287a00 2024-06-04T05:22:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c9ea28740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fbe21e2810 con 0x55fbe227d140 2024-06-04T05:22:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c992a5640 1 --2- 172.21.15.70:0/3330208911 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2c800b2ac0 0x7f2c800b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c9a2a7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 0x55fbe2289df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c99aa6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227c820 0x55fbe227cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c9a2a7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 0x55fbe2289df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:60546/0 (socket says 172.21.15.70:60546) 2024-06-04T05:22:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c9a2a7640 1 -- 172.21.15.70:0/2722246107 learned_addr learned my addr 172.21.15.70:0/2722246107 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c8affd640 1 -- 172.21.15.70:0/3330208911 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f2c8c0e2a00 con 0x7f2c9410fe60 2024-06-04T05:22:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c992a5640 1 --2- 172.21.15.70:0/2722246107 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227d140 0x55fbe22874c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c9a2a7640 1 -- 172.21.15.70:0/2722246107 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227d140 msgr2=0x55fbe22874c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c9a2a7640 1 --2- 172.21.15.70:0/2722246107 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227d140 0x55fbe22874c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c9a2a7640 1 -- 172.21.15.70:0/2722246107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227c820 msgr2=0x55fbe227cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c9a2a7640 1 --2- 172.21.15.70:0/2722246107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227c820 0x55fbe227cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c9a2a7640 1 -- 172.21.15.70:0/2722246107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbe1ef3210 con 0x55fbe2287a00 2024-06-04T05:22:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.570+0000 7f2c992a5640 1 --2- 172.21.15.70:0/2722246107 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227d140 0x55fbe22874c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:22:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.571+0000 7f2c9a2a7640 1 --2- 172.21.15.70:0/2722246107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 0x55fbe2289df0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c0a9be0 tx=0x7f2c8c0a5c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a58b6f8e347a1d6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.571+0000 7f2c992a5640 1 --2- 172.21.15.70:0/3330208911 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2c800b2ac0 0x7f2c800b4f80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2c840057c0 tx=0x7f2c84005fd0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.571+0000 7f2c7ffff640 1 -- 172.21.15.70:0/2722246107 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c091070 con 0x55fbe2287a00 2024-06-04T05:22:45.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.571+0000 7f2c7ffff640 1 -- 172.21.15.70:0/2722246107 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2c8c0b23f0 con 0x55fbe2287a00 2024-06-04T05:22:45.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.571+0000 7f2c7ffff640 1 -- 172.21.15.70:0/2722246107 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c084070 con 0x55fbe2287a00 2024-06-04T05:22:45.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.572+0000 7f2c9ea28740 1 -- 172.21.15.70:0/2722246107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 msgr2=0x55fbe2289df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:45.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.572+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/2722246107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 0x55fbe2289df0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c0a9be0 tx=0x7f2c8c0a5c40 comp rx=0 tx=0).stop 2024-06-04T05:22:45.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.572+0000 7f2c9ea28740 1 -- 172.21.15.70:0/2722246107 shutdown_connections 2024-06-04T05:22:45.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.572+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/2722246107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 0x55fbe2289df0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.572+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/2722246107 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227d140 0x55fbe22874c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.572+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/2722246107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227c820 0x55fbe227cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.572+0000 7f2c9ea28740 1 -- 172.21.15.70:0/2722246107 >> 172.21.15.70:0/2722246107 conn(0x55fbe2278d30 msgr2=0x55fbe2279110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:45.578 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.572+0000 7f2c9ea28740 1 -- 172.21.15.70:0/2722246107 shutdown_connections 2024-06-04T05:22:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.572+0000 7f2c9ea28740 1 -- 172.21.15.70:0/2722246107 wait complete. 2024-06-04T05:22:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.573+0000 7f2c9ea28740 1 -- start start 2024-06-04T05:22:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.573+0000 7f2c9ea28740 1 Processor -- start 2024-06-04T05:22:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.573+0000 7f2c9ea28740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227c820 0x55fbe22849e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.573+0000 7f2c9ea28740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227d140 0x55fbe2284f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.573+0000 7f2c9ea28740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 0x55fbe2285460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.573+0000 7f2c99aa6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227c820 0x55fbe22849e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.573+0000 7f2c9ea28740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fbe1ec4790 con 0x55fbe227c820 2024-06-04T05:22:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.573+0000 7f2c992a5640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227d140 0x55fbe2284f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.573+0000 7f2c99aa6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227c820 0x55fbe22849e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59712/0 (socket says 172.21.15.70:59712) 2024-06-04T05:22:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.573+0000 7f2c99aa6640 1 -- 172.21.15.70:0/2409439734 learned_addr learned my addr 172.21.15.70:0/2409439734 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.573+0000 7f2c9ea28740 1 -- 172.21.15.70:0/2409439734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fbe219a890 con 0x55fbe2287a00 2024-06-04T05:22:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.573+0000 7f2c9ea28740 1 -- 172.21.15.70:0/2409439734 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fbe228a630 con 0x55fbe227d140 2024-06-04T05:22:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.574+0000 7f2c9a2a7640 1 --2- 172.21.15.70:0/2409439734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 0x55fbe2285460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.574+0000 7f2c9ea28740 10 client.0 _ll_register_callbacks cb 0x55fbe1ef2a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.574+0000 7f2c9a2a7640 1 -- 172.21.15.70:0/2409439734 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227d140 msgr2=0x55fbe2284f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.574+0000 7f2c9a2a7640 1 --2- 172.21.15.70:0/2409439734 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227d140 0x55fbe2284f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.574+0000 7f2c9a2a7640 1 -- 172.21.15.70:0/2409439734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227c820 msgr2=0x55fbe22849e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.574+0000 7f2c9a2a7640 1 --2- 172.21.15.70:0/2409439734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227c820 0x55fbe22849e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.574+0000 7f2c9a2a7640 1 -- 172.21.15.70:0/2409439734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbe1fbf620 con 0x55fbe2287a00 2024-06-04T05:22:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.574+0000 7f2c99aa6640 1 --2- 172.21.15.70:0/2409439734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227c820 0x55fbe22849e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.574+0000 7f2c9a2a7640 1 --2- 172.21.15.70:0/2409439734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 0x55fbe2285460 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c0a9be0 tx=0x7f2c8c0a9810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f4a15599457ac2d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.574+0000 7f2c7d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:22:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.574+0000 7f2c7d7fa640 1 -- 172.21.15.70:0/2409439734 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c095020 con 0x55fbe2287a00 2024-06-04T05:22:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.574+0000 7f2c9ea28740 1 -- 172.21.15.70:0/2409439734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fbe21f0b80 con 0x55fbe2287a00 2024-06-04T05:22:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.574+0000 7f2c7d7fa640 1 -- 172.21.15.70:0/2409439734 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f2c8c0051d0 con 0x55fbe2287a00 2024-06-04T05:22:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.574+0000 7f2c7d7fa640 1 -- 172.21.15.70:0/2409439734 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f2c8c081050 con 0x55fbe2287a00 2024-06-04T05:22:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.575+0000 7f2c9ea28740 10 client.30437 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.575+0000 7f2c9ea28740 1 -- 172.21.15.70:0/2409439734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55fbe2296340 con 0x55fbe2287a00 2024-06-04T05:22:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.575+0000 7f2c7d7fa640 1 -- 172.21.15.70:0/2409439734 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55fbe21f0b80 con 0x55fbe2287a00 2024-06-04T05:22:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.575+0000 7f2c7d7fa640 1 -- 172.21.15.70:0/2409439734 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f2c8c082ce0 con 0x55fbe2287a00 2024-06-04T05:22:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.575+0000 7f2c9ea28740 10 client.30437 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.575+0000 7f2c9ea28740 10 client.30437 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.575+0000 7f2c9ea28740 20 client.30437 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.575+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/2409439734 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fbe2297980 0x55fbe2299d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.575+0000 7f2c9ea28740 4 client.30437 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:22:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.575+0000 7f2c9ea28740 1 -- 172.21.15.70:0/2409439734 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbe21ea360 con 0x55fbe2297980 2024-06-04T05:22:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.575+0000 7f2c99aa6640 1 --2- 172.21.15.70:0/2409439734 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fbe2297980 0x55fbe2299d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.576+0000 7f2c99aa6640 1 --2- 172.21.15.70:0/2409439734 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fbe2297980 0x55fbe2299d70 crc :-1 s=READY pgs=529 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=420f09784a326f0b server_cookie=c3831e851c883a7b in_seq=0 out_seq=0 2024-06-04T05:22:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.576+0000 7f2c7d7fa640 10 client.30437 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:45.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.577+0000 7f2c7d7fa640 1 -- 172.21.15.70:0/2409439734 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fbe21ea360 con 0x55fbe2297980 2024-06-04T05:22:45.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.577+0000 7f2c7d7fa640 10 client.30437 handle_command_reply: tid=0 2024-06-04T05:22:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.577+0000 7f2c9ea28740 1 client.30437 shutdown 2024-06-04T05:22:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.577+0000 7f2c9ea28740 1 -- 172.21.15.70:0/2409439734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 msgr2=0x55fbe2285460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.577+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/2409439734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 0x55fbe2285460 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c0a9be0 tx=0x7f2c8c0a9810 comp rx=0 tx=0).stop 2024-06-04T05:22:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.577+0000 7f2c9ea28740 1 -- 172.21.15.70:0/2409439734 shutdown_connections 2024-06-04T05:22:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.577+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/2409439734 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fbe2297980 0x55fbe2299d70 crc :-1 s=READY pgs=529 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.577+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/2409439734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 0x55fbe2285460 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.577+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/2409439734 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227d140 0x55fbe2284f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.577+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/2409439734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227c820 0x55fbe22849e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.577+0000 7f2c9ea28740 1 -- 172.21.15.70:0/2409439734 >> 172.21.15.70:0/2409439734 conn(0x55fbe2278d30 msgr2=0x55fbe2277fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.577+0000 7f2c7d7fa640 0 client.30437 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.577+0000 7f2c9ea28740 1 -- 172.21.15.70:0/2409439734 shutdown_connections 2024-06-04T05:22:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.577+0000 7f2c9ea28740 1 -- 172.21.15.70:0/2409439734 wait complete. 2024-06-04T05:22:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.577+0000 7f2c9ea28740 20 client.30437 trim_cache size 0 max 16384 2024-06-04T05:22:45.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.605+0000 7f2c9ea28740 1 Processor -- start 2024-06-04T05:22:45.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.606+0000 7f2c9ea28740 1 -- start start 2024-06-04T05:22:45.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.606+0000 7f2c9ea28740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227c820 0x55fbe227f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.606+0000 7f2c9ea28740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe227d140 0x55fbe2292d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.606+0000 7f2c9ea28740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe2287a00 0x55fbe22932d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.606+0000 7f2c9ea28740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2c941ac480 con 0x55fbe2287a00 2024-06-04T05:22:45.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.606+0000 7f2c9ea28740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2c941adfc0 con 0x55fbe227d140 2024-06-04T05:22:45.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.606+0000 7f2c9ea28740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2c941acb50 con 0x55fbe227c820 2024-06-04T05:22:45.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.606+0000 7f2c9a2a7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe2287a00 0x55fbe22932d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.606+0000 7f2c99aa6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227c820 0x55fbe227f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.606+0000 7f2c9a2a7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe2287a00 0x55fbe22932d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59718/0 (socket says 172.21.15.70:59718) 2024-06-04T05:22:45.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.606+0000 7f2c9a2a7640 1 -- 172.21.15.70:0/444371515 learned_addr learned my addr 172.21.15.70:0/444371515 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:45.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.606+0000 7f2c992a5640 1 --2- 172.21.15.70:0/444371515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe227d140 0x55fbe2292d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.606+0000 7f2c9a2a7640 1 -- 172.21.15.70:0/444371515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227c820 msgr2=0x55fbe227f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:45.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.606+0000 7f2c9a2a7640 1 --2- 172.21.15.70:0/444371515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227c820 0x55fbe227f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.606+0000 7f2c9a2a7640 1 -- 172.21.15.70:0/444371515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe227d140 msgr2=0x55fbe2292d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:45.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.606+0000 7f2c9a2a7640 1 --2- 172.21.15.70:0/444371515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe227d140 0x55fbe2292d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.606+0000 7f2c9a2a7640 1 -- 172.21.15.70:0/444371515 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbe2282f30 con 0x55fbe2287a00 2024-06-04T05:22:45.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.607+0000 7f2c9a2a7640 1 --2- 172.21.15.70:0/444371515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe2287a00 0x55fbe22932d0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c0a9880 tx=0x7f2c8c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8897ca5328a896a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:45.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.607+0000 7f2c7f7fe640 1 -- 172.21.15.70:0/444371515 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c0c8070 con 0x55fbe2287a00 2024-06-04T05:22:45.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.607+0000 7f2c7f7fe640 1 -- 172.21.15.70:0/444371515 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2c8c0aa350 con 0x55fbe2287a00 2024-06-04T05:22:45.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.607+0000 7f2c7f7fe640 1 -- 172.21.15.70:0/444371515 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c095070 con 0x55fbe2287a00 2024-06-04T05:22:45.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.608+0000 7f2c9ea28740 1 -- 172.21.15.70:0/444371515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe2287a00 msgr2=0x55fbe22932d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:45.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.608+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/444371515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe2287a00 0x55fbe22932d0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c0a9880 tx=0x7f2c8c00c810 comp rx=0 tx=0).stop 2024-06-04T05:22:45.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.608+0000 7f2c9ea28740 1 -- 172.21.15.70:0/444371515 shutdown_connections 2024-06-04T05:22:45.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.608+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/444371515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe227d140 0x55fbe2292d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.608+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/444371515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe2287a00 0x55fbe22932d0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.608+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/444371515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227c820 0x55fbe227f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.608+0000 7f2c9ea28740 1 -- 172.21.15.70:0/444371515 >> 172.21.15.70:0/444371515 conn(0x55fbe2278d30 msgr2=0x55fbe2292680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:45.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.608+0000 7f2c9ea28740 1 -- 172.21.15.70:0/444371515 shutdown_connections 2024-06-04T05:22:45.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.608+0000 7f2c9ea28740 1 -- 172.21.15.70:0/444371515 wait complete. 2024-06-04T05:22:45.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.608+0000 7f2c9ea28740 1 -- start start 2024-06-04T05:22:45.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.608+0000 7f2c9ea28740 1 Processor -- start 2024-06-04T05:22:45.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c9ea28740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227c820 0x55fbe228f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c9ea28740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227d140 0x55fbe228fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c9ea28740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 0x55fbe2281060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c9ea28740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2c941ad8f0 con 0x55fbe227d140 2024-06-04T05:22:45.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c9ea28740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2c941ad220 con 0x55fbe2287a00 2024-06-04T05:22:45.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c9ea28740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2c941aa4a0 con 0x55fbe227c820 2024-06-04T05:22:45.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c9ea28740 10 client.0 _ll_register_callbacks cb 0x7f2c90009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:45.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c992a5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227d140 0x55fbe228fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c99aa6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227c820 0x55fbe228f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c99aa6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227c820 0x55fbe228f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:60080/0 (socket says 172.21.15.70:60080) 2024-06-04T05:22:45.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c992a5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227d140 0x55fbe228fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59724/0 (socket says 172.21.15.70:59724) 2024-06-04T05:22:45.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c99aa6640 1 -- 172.21.15.70:0/546876903 learned_addr learned my addr 172.21.15.70:0/546876903 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:45.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c9a2a7640 1 --2- 172.21.15.70:0/546876903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 0x55fbe2281060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c99aa6640 1 -- 172.21.15.70:0/546876903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 msgr2=0x55fbe2281060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:45.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c99aa6640 1 --2- 172.21.15.70:0/546876903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 0x55fbe2281060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c99aa6640 1 -- 172.21.15.70:0/546876903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227d140 msgr2=0x55fbe228fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:45.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c99aa6640 1 --2- 172.21.15.70:0/546876903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227d140 0x55fbe228fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c99aa6640 1 -- 172.21.15.70:0/546876903 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbe22815a0 con 0x55fbe227c820 2024-06-04T05:22:45.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c992a5640 1 --2- 172.21.15.70:0/546876903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227d140 0x55fbe228fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:45.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.609+0000 7f2c9a2a7640 1 --2- 172.21.15.70:0/546876903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 0x55fbe2281060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:45.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.610+0000 7f2c7d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:22:45.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.610+0000 7f2c99aa6640 1 --2- 172.21.15.70:0/546876903 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227c820 0x55fbe228f880 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2c90004700 tx=0x7f2c9001cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa5f7b619bbf4aa2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:45.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.610+0000 7f2c7d7fa640 1 -- 172.21.15.70:0/546876903 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c90027380 con 0x55fbe227c820 2024-06-04T05:22:45.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.610+0000 7f2c7d7fa640 1 -- 172.21.15.70:0/546876903 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f2c90027520 con 0x55fbe227c820 2024-06-04T05:22:45.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.610+0000 7f2c9ea28740 1 -- 172.21.15.70:0/546876903 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c8c0aa350 con 0x55fbe227c820 2024-06-04T05:22:45.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.610+0000 7f2c7d7fa640 1 -- 172.21.15.70:0/546876903 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f2c90037050 con 0x55fbe227c820 2024-06-04T05:22:45.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.610+0000 7f2c9ea28740 10 client.30571 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:45.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.610+0000 7f2c9ea28740 1 -- 172.21.15.70:0/546876903 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55fbe2301da0 con 0x55fbe227c820 2024-06-04T05:22:45.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.611+0000 7f2c7d7fa640 1 -- 172.21.15.70:0/546876903 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2c8c0aa350 con 0x55fbe227c820 2024-06-04T05:22:45.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.611+0000 7f2c7d7fa640 1 -- 172.21.15.70:0/546876903 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f2c90010750 con 0x55fbe227c820 2024-06-04T05:22:45.616 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.611+0000 7f2c9ea28740 10 client.30571 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.611+0000 7f2c9ea28740 10 client.30571 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.611+0000 7f2c9ea28740 20 client.30571 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.611+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/546876903 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fbe23024f0 0x55fbe2304910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.611+0000 7f2c9ea28740 4 client.30571 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:22:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.611+0000 7f2c9ea28740 1 -- 172.21.15.70:0/546876903 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fbe21645f0 con 0x55fbe23024f0 2024-06-04T05:22:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.611+0000 7f2c992a5640 1 --2- 172.21.15.70:0/546876903 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fbe23024f0 0x55fbe2304910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.612+0000 7f2c992a5640 1 --2- 172.21.15.70:0/546876903 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fbe23024f0 0x55fbe2304910 crc :-1 s=READY pgs=530 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa00ae377d5dcd0 server_cookie=3b0a7f5a7034df85 in_seq=0 out_seq=0 2024-06-04T05:22:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.612+0000 7f2c7d7fa640 10 client.30571 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:45.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:45 smithi112 ceph-mon[36453]: pgmap v806: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 211 op/s 2024-06-04T05:22:45.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.730+0000 7f2c7d7fa640 1 -- 172.21.15.70:0/546876903 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55fbe21645f0 con 0x55fbe23024f0 2024-06-04T05:22:45.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.730+0000 7f2c7d7fa640 10 client.30571 handle_command_reply: tid=0 2024-06-04T05:22:45.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.730+0000 7f2c9ea28740 1 client.30571 shutdown 2024-06-04T05:22:45.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.730+0000 7f2c9ea28740 1 -- 172.21.15.70:0/546876903 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227c820 msgr2=0x55fbe228f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:45.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.730+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/546876903 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227c820 0x55fbe228f880 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2c90004700 tx=0x7f2c9001cc30 comp rx=0 tx=0).stop 2024-06-04T05:22:45.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.730+0000 7f2c9ea28740 1 -- 172.21.15.70:0/546876903 shutdown_connections 2024-06-04T05:22:45.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.730+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/546876903 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55fbe23024f0 0x55fbe2304910 crc :-1 s=READY pgs=530 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.730+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/546876903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fbe2287a00 0x55fbe2281060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.730+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/546876903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbe227d140 0x55fbe228fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.730+0000 7f2c7d7fa640 0 client.30571 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:45.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.730+0000 7f2c9ea28740 1 --2- 172.21.15.70:0/546876903 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fbe227c820 0x55fbe228f880 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.731+0000 7f2c9ea28740 1 -- 172.21.15.70:0/546876903 >> 172.21.15.70:0/546876903 conn(0x55fbe2278d30 msgr2=0x55fbe2292f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:45.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.731+0000 7f2c9ea28740 1 -- 172.21.15.70:0/546876903 shutdown_connections 2024-06-04T05:22:45.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.731+0000 7f2c9ea28740 1 -- 172.21.15.70:0/546876903 wait complete. 2024-06-04T05:22:45.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.731+0000 7f2c9ea28740 20 client.30571 trim_cache size 0 max 16384 2024-06-04T05:22:45.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:22:45.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:22:45.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:22:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.731+0000 7f2c9bd0a640 1 -- 172.21.15.70:0/3330208911 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2c800b2ac0 msgr2=0x7f2c800b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.731+0000 7f2c9bd0a640 1 --2- 172.21.15.70:0/3330208911 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2c800b2ac0 0x7f2c800b4f80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2c840057c0 tx=0x7f2c84005fd0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.731+0000 7f2c9bd0a640 1 -- 172.21.15.70:0/3330208911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c9410fe60 msgr2=0x7f2c941a74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.731+0000 7f2c9bd0a640 1 --2- 172.21.15.70:0/3330208911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c9410fe60 0x7f2c941a74d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009fd0 tx=0x7f2c8c00ef10 comp rx=0 tx=0).stop 2024-06-04T05:22:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.732+0000 7f2c9bd0a640 1 -- 172.21.15.70:0/3330208911 shutdown_connections 2024-06-04T05:22:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.732+0000 7f2c9bd0a640 1 --2- 172.21.15.70:0/3330208911 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2c800b2ac0 0x7f2c800b4f80 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.732+0000 7f2c9bd0a640 1 --2- 172.21.15.70:0/3330208911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c94106e80 0x7f2c9410f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.732+0000 7f2c9bd0a640 1 --2- 172.21.15.70:0/3330208911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c9410fe60 0x7f2c941a74d0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.732+0000 7f2c9bd0a640 1 --2- 172.21.15.70:0/3330208911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c941022d0 0x7f2c9410f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.732+0000 7f2c9bd0a640 1 -- 172.21.15.70:0/3330208911 >> 172.21.15.70:0/3330208911 conn(0x7f2c94077ee0 msgr2=0x7f2c940ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.732+0000 7f2c9bd0a640 1 -- 172.21.15.70:0/3330208911 shutdown_connections 2024-06-04T05:22:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:45.732+0000 7f2c9bd0a640 1 -- 172.21.15.70:0/3330208911 wait complete. 2024-06-04T05:22:45.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:45 smithi098 ceph-mon[35569]: pgmap v806: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 211 op/s 2024-06-04T05:22:45.836 INFO:tasks.workunit.client.0.smithi070.stdout: 10 86659 6.56 MB/sec execute 577 sec latency 90.675 ms 2024-06-04T05:22:45.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:45 smithi070 ceph-mon[32631]: pgmap v806: 97 pgs: 97 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 211 op/s 2024-06-04T05:22:46.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 86854 6.56 MB/sec execute 578 sec latency 70.118 ms 2024-06-04T05:22:47.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:47 smithi112 ceph-mon[36453]: pgmap v807: 97 pgs: 97 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.8 MiB/s wr, 368 op/s 2024-06-04T05:22:47.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:47 smithi098 ceph-mon[35569]: pgmap v807: 97 pgs: 97 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.8 MiB/s wr, 368 op/s 2024-06-04T05:22:47.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 87047 6.57 MB/sec execute 579 sec latency 58.626 ms 2024-06-04T05:22:47.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:47 smithi070 ceph-mon[32631]: pgmap v807: 97 pgs: 97 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.8 MiB/s wr, 368 op/s 2024-06-04T05:22:48.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 87173 6.57 MB/sec execute 580 sec latency 140.480 ms 2024-06-04T05:22:49.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:49 smithi112 ceph-mon[36453]: pgmap v808: 97 pgs: 97 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 286 op/s 2024-06-04T05:22:49.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:49 smithi098 ceph-mon[35569]: pgmap v808: 97 pgs: 97 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 286 op/s 2024-06-04T05:22:49.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 87333 6.58 MB/sec execute 581 sec latency 63.077 ms 2024-06-04T05:22:49.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:49 smithi070 ceph-mon[32631]: pgmap v808: 97 pgs: 97 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 286 op/s 2024-06-04T05:22:50.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 87535 6.59 MB/sec execute 582 sec latency 73.241 ms 2024-06-04T05:22:50.944 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:22:51.288 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:22:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:51 smithi112 ceph-mon[36453]: pgmap v809: 97 pgs: 97 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 286 op/s 2024-06-04T05:22:51.743 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:51 smithi070 ceph-mon[32631]: pgmap v809: 97 pgs: 97 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 286 op/s 2024-06-04T05:22:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:51 smithi098 ceph-mon[35569]: pgmap v809: 97 pgs: 97 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 286 op/s 2024-06-04T05:22:51.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 87673 6.59 MB/sec execute 583 sec latency 271.174 ms 2024-06-04T05:22:52.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 87839 6.59 MB/sec execute 584 sec latency 142.722 ms 2024-06-04T05:22:53.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.439+0000 7f62b6936700 1 -- 172.21.15.70:0/2425739371 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b0109520 msgr2=0x7f62b0109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:53.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.439+0000 7f62b6936700 1 --2- 172.21.15.70:0/2425739371 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b0109520 0x7f62b0109900 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f62a0009990 tx=0x7f62a0009ca0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.439+0000 7f62b6936700 1 -- 172.21.15.70:0/2425739371 shutdown_connections 2024-06-04T05:22:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.439+0000 7f62b6936700 1 --2- 172.21.15.70:0/2425739371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b01024f0 0x7f62b01103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.439+0000 7f62b6936700 1 --2- 172.21.15.70:0/2425739371 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b0109520 0x7f62b0109900 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.439+0000 7f62b6936700 1 --2- 172.21.15.70:0/2425739371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62b0101b30 0x7f62b0101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.439+0000 7f62b6936700 1 -- 172.21.15.70:0/2425739371 >> 172.21.15.70:0/2425739371 conn(0x7f62b0078ab0 msgr2=0x7f62b00ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.439+0000 7f62b6936700 1 -- 172.21.15.70:0/2425739371 shutdown_connections 2024-06-04T05:22:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.439+0000 7f62b6936700 1 -- 172.21.15.70:0/2425739371 wait complete. 2024-06-04T05:22:53.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.440+0000 7f62b6936700 1 Processor -- start 2024-06-04T05:22:53.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.440+0000 7f62b6936700 1 -- start start 2024-06-04T05:22:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.440+0000 7f62b6936700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b0101b30 0x7f62b019f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.440+0000 7f62b6936700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62b01024f0 0x7f62b019fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.440+0000 7f62b6936700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b0109520 0x7f62b0199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.440+0000 7f62b6936700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62b0113610 con 0x7f62b0109520 2024-06-04T05:22:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.440+0000 7f62b6936700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62b0113490 con 0x7f62b01024f0 2024-06-04T05:22:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.440+0000 7f62b6936700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f62b0113790 con 0x7f62b0101b30 2024-06-04T05:22:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.440+0000 7f62af7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62b01024f0 0x7f62b019fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.440+0000 7f62b4ed3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b0109520 0x7f62b0199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.440+0000 7f62affff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b0101b30 0x7f62b019f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.440+0000 7f62affff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b0101b30 0x7f62b019f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:52698/0 (socket says 172.21.15.70:52698) 2024-06-04T05:22:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.440+0000 7f62b4ed3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b0109520 0x7f62b0199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52480/0 (socket says 172.21.15.70:52480) 2024-06-04T05:22:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.440+0000 7f62affff700 1 -- 172.21.15.70:0/488709730 learned_addr learned my addr 172.21.15.70:0/488709730 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.441+0000 7f62affff700 1 -- 172.21.15.70:0/488709730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62b01024f0 msgr2=0x7f62b019fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.441+0000 7f62affff700 1 --2- 172.21.15.70:0/488709730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62b01024f0 0x7f62b019fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.441+0000 7f62affff700 1 -- 172.21.15.70:0/488709730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b0109520 msgr2=0x7f62b0199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.441+0000 7f62affff700 1 --2- 172.21.15.70:0/488709730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b0109520 0x7f62b0199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.441+0000 7f62affff700 1 -- 172.21.15.70:0/488709730 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b019a0d0 con 0x7f62b0101b30 2024-06-04T05:22:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.441+0000 7f62b4ed3700 1 --2- 172.21.15.70:0/488709730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b0109520 0x7f62b0199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.441+0000 7f62affff700 1 --2- 172.21.15.70:0/488709730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b0101b30 0x7f62b019f7b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f62a0006010 tx=0x7f62a0003a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.441+0000 7f62ad7fa700 1 -- 172.21.15.70:0/488709730 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a0027070 con 0x7f62b0101b30 2024-06-04T05:22:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.441+0000 7f62ad7fa700 1 -- 172.21.15.70:0/488709730 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f62a000bd50 con 0x7f62b0101b30 2024-06-04T05:22:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.441+0000 7f62ad7fa700 1 -- 172.21.15.70:0/488709730 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a0022ce0 con 0x7f62b0101b30 2024-06-04T05:22:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.441+0000 7f62b6936700 1 -- 172.21.15.70:0/488709730 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62b019a360 con 0x7f62b0101b30 2024-06-04T05:22:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.441+0000 7f62b6936700 1 -- 172.21.15.70:0/488709730 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62b0102c40 con 0x7f62b0101b30 2024-06-04T05:22:53.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.444+0000 7f62ad7fa700 1 -- 172.21.15.70:0/488709730 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f62a002a070 con 0x7f62b0101b30 2024-06-04T05:22:53.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.445+0000 7f62ad7fa700 1 --2- 172.21.15.70:0/488709730 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62980b29f0 0x7f62980b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.446+0000 7f62ad7fa700 1 -- 172.21.15.70:0/488709730 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f62a00e3590 con 0x7f62b0101b30 2024-06-04T05:22:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.446+0000 7f62af7fe700 1 --2- 172.21.15.70:0/488709730 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62980b29f0 0x7f62980b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.446+0000 7f62c11cbb80 1 Processor -- start 2024-06-04T05:22:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.446+0000 7f62c11cbb80 1 -- start start 2024-06-04T05:22:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.446+0000 7f62c11cbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d89854320 0x563d89854700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.446+0000 7f62c11cbb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d8984e8a0 0x563d8984ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.446+0000 7f62c11cbb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984f260 0x563d8985b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.446+0000 7f62c11cbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563d897acd70 con 0x563d89854320 2024-06-04T05:22:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.446+0000 7f62c11cbb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563d8979c910 con 0x563d8984e8a0 2024-06-04T05:22:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.446+0000 7f62c11cbb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563d897a0950 con 0x563d8984f260 2024-06-04T05:22:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.446+0000 7f62af7fe700 1 --2- 172.21.15.70:0/488709730 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62980b29f0 0x7f62980b4eb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f62b00ffc40 tx=0x7f62a4009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.447+0000 7f62b4ed3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984f260 0x563d8985b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.447+0000 7f62af7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d8984e8a0 0x563d8984ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.447+0000 7f62b4ed3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984f260 0x563d8985b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:52702/0 (socket says 172.21.15.70:52702) 2024-06-04T05:22:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.447+0000 7f62b4ed3700 1 -- 172.21.15.70:0/1293518527 learned_addr learned my addr 172.21.15.70:0/1293518527 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.447+0000 7f62b4ed3700 1 -- 172.21.15.70:0/1293518527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d8984e8a0 msgr2=0x563d8984ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.447+0000 7f62b4ed3700 1 --2- 172.21.15.70:0/1293518527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d8984e8a0 0x563d8984ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.447+0000 7f62b4ed3700 1 -- 172.21.15.70:0/1293518527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d89854320 msgr2=0x563d89854700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.447+0000 7f62b4ed3700 1 --2- 172.21.15.70:0/1293518527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d89854320 0x563d89854700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.447+0000 7f62b4ed3700 1 -- 172.21.15.70:0/1293518527 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d89745fb0 con 0x563d8984f260 2024-06-04T05:22:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.447+0000 7f62b4ed3700 1 --2- 172.21.15.70:0/1293518527 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984f260 0x563d8985b080 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f62a8010cc0 tx=0x7f62a8012040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfe9ba9f3412b4b7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.447+0000 7f62967fc700 1 -- 172.21.15.70:0/1293518527 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a801e070 con 0x563d8984f260 2024-06-04T05:22:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.447+0000 7f62967fc700 1 -- 172.21.15.70:0/1293518527 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f62a80090d0 con 0x563d8984f260 2024-06-04T05:22:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.447+0000 7f62967fc700 1 -- 172.21.15.70:0/1293518527 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a800e760 con 0x563d8984f260 2024-06-04T05:22:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.448+0000 7f62c11cbb80 1 -- 172.21.15.70:0/1293518527 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984f260 msgr2=0x563d8985b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.448+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/1293518527 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984f260 0x563d8985b080 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f62a8010cc0 tx=0x7f62a8012040 comp rx=0 tx=0).stop 2024-06-04T05:22:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.448+0000 7f62c11cbb80 1 -- 172.21.15.70:0/1293518527 shutdown_connections 2024-06-04T05:22:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.449+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/1293518527 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984f260 0x563d8985b080 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.449+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/1293518527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d89854320 0x563d89854700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.449+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/1293518527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d8984e8a0 0x563d8984ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.449+0000 7f62c11cbb80 1 -- 172.21.15.70:0/1293518527 >> 172.21.15.70:0/1293518527 conn(0x563d8977a240 msgr2=0x563d898493e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.449+0000 7f62c11cbb80 1 -- 172.21.15.70:0/1293518527 shutdown_connections 2024-06-04T05:22:53.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.449+0000 7f62c11cbb80 1 -- 172.21.15.70:0/1293518527 wait complete. 2024-06-04T05:22:53.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.449+0000 7f62c11cbb80 1 -- start start 2024-06-04T05:22:53.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.450+0000 7f62c11cbb80 1 Processor -- start 2024-06-04T05:22:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.450+0000 7f62c11cbb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984e8a0 0x563d8985ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.450+0000 7f62c11cbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d8984f260 0x563d89858090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.450+0000 7f62c11cbb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d89854320 0x563d898585d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.450+0000 7f62c11cbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563d896a2fa0 con 0x563d8984f260 2024-06-04T05:22:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.450+0000 7f62c11cbb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563d89704da0 con 0x563d89854320 2024-06-04T05:22:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.450+0000 7f62c11cbb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563d8985b730 con 0x563d8984e8a0 2024-06-04T05:22:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.450+0000 7f62c11cbb80 10 client.0 _ll_register_callbacks cb 0x563d89737950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.450+0000 7f62b4ed3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d89854320 0x563d898585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.450+0000 7f62b4ed3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d89854320 0x563d898585d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:60010/0 (socket says 172.21.15.70:60010) 2024-06-04T05:22:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.450+0000 7f62b4ed3700 1 -- 172.21.15.70:0/3968268769 learned_addr learned my addr 172.21.15.70:0/3968268769 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.450+0000 7f62affff700 1 --2- 172.21.15.70:0/3968268769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984e8a0 0x563d8985ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.450+0000 7f62af7fe700 1 --2- 172.21.15.70:0/3968268769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d8984f260 0x563d89858090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.451+0000 7f62b4ed3700 1 -- 172.21.15.70:0/3968268769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984e8a0 msgr2=0x563d8985ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.451+0000 7f62b4ed3700 1 --2- 172.21.15.70:0/3968268769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984e8a0 0x563d8985ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.451+0000 7f62b4ed3700 1 -- 172.21.15.70:0/3968268769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d8984f260 msgr2=0x563d89858090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.451+0000 7f62b4ed3700 1 --2- 172.21.15.70:0/3968268769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d8984f260 0x563d89858090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.451+0000 7f62b4ed3700 1 -- 172.21.15.70:0/3968268769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d896ec8d0 con 0x563d89854320 2024-06-04T05:22:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.451+0000 7f62b4ed3700 1 --2- 172.21.15.70:0/3968268769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d89854320 0x563d898585d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f62a8003fb0 tx=0x7f62a8009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4e7bb425fd1ef7a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.451+0000 7f628f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:22:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.451+0000 7f628f7fe700 1 -- 172.21.15.70:0/3968268769 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8027060 con 0x563d89854320 2024-06-04T05:22:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.451+0000 7f62c11cbb80 1 -- 172.21.15.70:0/3968268769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62a80090d0 con 0x563d89854320 2024-06-04T05:22:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.451+0000 7f628f7fe700 1 -- 172.21.15.70:0/3968268769 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f62a8004ad0 con 0x563d89854320 2024-06-04T05:22:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.451+0000 7f628f7fe700 1 -- 172.21.15.70:0/3968268769 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f62a801e040 con 0x563d89854320 2024-06-04T05:22:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.451+0000 7f62c11cbb80 10 client.30455 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.451+0000 7f62c11cbb80 1 -- 172.21.15.70:0/3968268769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x563d89756760 con 0x563d89854320 2024-06-04T05:22:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.452+0000 7f628f7fe700 1 -- 172.21.15.70:0/3968268769 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f62a80090d0 con 0x563d89854320 2024-06-04T05:22:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.452+0000 7f628f7fe700 1 -- 172.21.15.70:0/3968268769 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f62a8004250 con 0x563d89854320 2024-06-04T05:22:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.452+0000 7f62c11cbb80 10 client.30455 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.452+0000 7f62c11cbb80 10 client.30455 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.452+0000 7f62c11cbb80 20 client.30455 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.452+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/3968268769 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563d898681a0 0x563d8986a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.452+0000 7f62c11cbb80 4 client.30455 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:22:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.452+0000 7f62c11cbb80 1 -- 172.21.15.70:0/3968268769 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d897aecc0 con 0x563d898681a0 2024-06-04T05:22:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.452+0000 7f62affff700 1 --2- 172.21.15.70:0/3968268769 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563d898681a0 0x563d8986a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.453+0000 7f628f7fe700 10 client.30455 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.453+0000 7f62affff700 1 --2- 172.21.15.70:0/3968268769 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563d898681a0 0x563d8986a5c0 crc :-1 s=READY pgs=531 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfee970c1245e12e server_cookie=e07c86c99cfbcb26 in_seq=0 out_seq=0 2024-06-04T05:22:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.453+0000 7f628f7fe700 1 -- 172.21.15.70:0/3968268769 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563d897aecc0 con 0x563d898681a0 2024-06-04T05:22:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.453+0000 7f628f7fe700 10 client.30455 handle_command_reply: tid=0 2024-06-04T05:22:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.453+0000 7f62c11cbb80 1 client.30455 shutdown 2024-06-04T05:22:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.454+0000 7f62c11cbb80 1 -- 172.21.15.70:0/3968268769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d89854320 msgr2=0x563d898585d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.454+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/3968268769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d89854320 0x563d898585d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f62a8003fb0 tx=0x7f62a8009920 comp rx=0 tx=0).stop 2024-06-04T05:22:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.454+0000 7f62c11cbb80 1 -- 172.21.15.70:0/3968268769 shutdown_connections 2024-06-04T05:22:53.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.454+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/3968268769 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563d898681a0 0x563d8986a5c0 crc :-1 s=READY pgs=531 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.454+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/3968268769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d89854320 0x563d898585d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.454+0000 7f628f7fe700 0 client.30455 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:53.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.454+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/3968268769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984e8a0 0x563d8985ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.454+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/3968268769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d8984f260 0x563d89858090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.454+0000 7f62c11cbb80 1 -- 172.21.15.70:0/3968268769 >> 172.21.15.70:0/3968268769 conn(0x563d8977a240 msgr2=0x563d89848af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:53.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.454+0000 7f62c11cbb80 1 -- 172.21.15.70:0/3968268769 shutdown_connections 2024-06-04T05:22:53.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.454+0000 7f62c11cbb80 1 -- 172.21.15.70:0/3968268769 wait complete. 2024-06-04T05:22:53.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.454+0000 7f62c11cbb80 20 client.30455 trim_cache size 0 max 16384 2024-06-04T05:22:53.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.490+0000 7f62c11cbb80 1 Processor -- start 2024-06-04T05:22:53.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.490+0000 7f62c11cbb80 1 -- start start 2024-06-04T05:22:53.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.490+0000 7f62c11cbb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984e8a0 0x563d89857210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.490+0000 7f62c11cbb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d8984f260 0x563d89849270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.490+0000 7f62c11cbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d89854320 0x563d89852460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.490+0000 7f62c11cbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62b01abb60 con 0x563d89854320 2024-06-04T05:22:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.490+0000 7f62b4ed3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d89854320 0x563d89852460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.490+0000 7f62c11cbb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62b01ad6a0 con 0x563d8984f260 2024-06-04T05:22:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.490+0000 7f62c11cbb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f62b01ac230 con 0x563d8984e8a0 2024-06-04T05:22:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.490+0000 7f62b4ed3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d89854320 0x563d89852460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52496/0 (socket says 172.21.15.70:52496) 2024-06-04T05:22:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.490+0000 7f62b4ed3700 1 -- 172.21.15.70:0/2658693319 learned_addr learned my addr 172.21.15.70:0/2658693319 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.490+0000 7f62affff700 1 --2- 172.21.15.70:0/2658693319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984e8a0 0x563d89857210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.490+0000 7f62af7fe700 1 --2- 172.21.15.70:0/2658693319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d8984f260 0x563d89849270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.491+0000 7f62af7fe700 1 -- 172.21.15.70:0/2658693319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984e8a0 msgr2=0x563d89857210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.491+0000 7f62af7fe700 1 --2- 172.21.15.70:0/2658693319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984e8a0 0x563d89857210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.491+0000 7f62af7fe700 1 -- 172.21.15.70:0/2658693319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d89854320 msgr2=0x563d89852460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:53.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.491+0000 7f62af7fe700 1 --2- 172.21.15.70:0/2658693319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d89854320 0x563d89852460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.491+0000 7f62af7fe700 1 -- 172.21.15.70:0/2658693319 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d89856600 con 0x563d8984f260 2024-06-04T05:22:53.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.491+0000 7f62affff700 1 --2- 172.21.15.70:0/2658693319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984e8a0 0x563d89857210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:53.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.491+0000 7f62af7fe700 1 --2- 172.21.15.70:0/2658693319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d8984f260 0x563d89849270 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x563d89859660 tx=0x7f62a4014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2efeb4af6c16bcbc server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:53.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.491+0000 7f6295ffb700 1 -- 172.21.15.70:0/2658693319 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a4003880 con 0x563d8984f260 2024-06-04T05:22:53.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.491+0000 7f6295ffb700 1 -- 172.21.15.70:0/2658693319 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f62a402c070 con 0x563d8984f260 2024-06-04T05:22:53.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.491+0000 7f6295ffb700 1 -- 172.21.15.70:0/2658693319 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a401db40 con 0x563d8984f260 2024-06-04T05:22:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.492+0000 7f62c11cbb80 1 -- 172.21.15.70:0/2658693319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d8984f260 msgr2=0x563d89849270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.492+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/2658693319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d8984f260 0x563d89849270 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x563d89859660 tx=0x7f62a4014000 comp rx=0 tx=0).stop 2024-06-04T05:22:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.492+0000 7f62c11cbb80 1 -- 172.21.15.70:0/2658693319 shutdown_connections 2024-06-04T05:22:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.492+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/2658693319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d89854320 0x563d89852460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.492+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/2658693319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d8984e8a0 0x563d89857210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.492+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/2658693319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d8984f260 0x563d89849270 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.492+0000 7f62c11cbb80 1 -- 172.21.15.70:0/2658693319 >> 172.21.15.70:0/2658693319 conn(0x563d8977a240 msgr2=0x563d89851d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.492+0000 7f62c11cbb80 1 -- 172.21.15.70:0/2658693319 shutdown_connections 2024-06-04T05:22:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.492+0000 7f62c11cbb80 1 -- 172.21.15.70:0/2658693319 wait complete. 2024-06-04T05:22:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.492+0000 7f62c11cbb80 1 -- start start 2024-06-04T05:22:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.493+0000 7f62c11cbb80 1 Processor -- start 2024-06-04T05:22:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.493+0000 7f62c11cbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d8984e8a0 0x563d89870450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.493+0000 7f62c11cbb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d8984f260 0x563d8986d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.493+0000 7f62c11cbb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d89854320 0x563d8986df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.493+0000 7f62c11cbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62b01acfd0 con 0x563d8984e8a0 2024-06-04T05:22:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.493+0000 7f62c11cbb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62b01ac900 con 0x563d8984f260 2024-06-04T05:22:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.493+0000 7f62c11cbb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f62b01a9b20 con 0x563d89854320 2024-06-04T05:22:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.493+0000 7f62c11cbb80 10 client.0 _ll_register_callbacks cb 0x563d89737950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:53.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.493+0000 7f62affff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d8984e8a0 0x563d89870450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:53.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.493+0000 7f62af7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d8984f260 0x563d8986d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:53.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.493+0000 7f62affff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d8984e8a0 0x563d89870450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52506/0 (socket says 172.21.15.70:52506) 2024-06-04T05:22:53.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.493+0000 7f62affff700 1 -- 172.21.15.70:0/1308072474 learned_addr learned my addr 172.21.15.70:0/1308072474 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:53.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.494+0000 7f62b4ed3700 1 --2- 172.21.15.70:0/1308072474 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d89854320 0x563d8986df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:53.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.494+0000 7f62b4ed3700 1 -- 172.21.15.70:0/1308072474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d8984f260 msgr2=0x563d8986d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:53.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.494+0000 7f62b4ed3700 1 --2- 172.21.15.70:0/1308072474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d8984f260 0x563d8986d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.494+0000 7f62b4ed3700 1 -- 172.21.15.70:0/1308072474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d8984e8a0 msgr2=0x563d89870450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:53.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.494+0000 7f62b4ed3700 1 --2- 172.21.15.70:0/1308072474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d8984e8a0 0x563d89870450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.494+0000 7f62b4ed3700 1 -- 172.21.15.70:0/1308072474 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d898d1b30 con 0x563d89854320 2024-06-04T05:22:53.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.494+0000 7f62affff700 1 --2- 172.21.15.70:0/1308072474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d8984e8a0 0x563d89870450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:53.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.494+0000 7f62b4ed3700 1 --2- 172.21.15.70:0/1308072474 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d89854320 0x563d8986df30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f62a8004620 tx=0x7f62a8009380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=136130fad628ef16 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:53.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.494+0000 7f628f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:22:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.494+0000 7f628f7fe700 1 -- 172.21.15.70:0/1308072474 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8027050 con 0x563d89854320 2024-06-04T05:22:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.494+0000 7f628f7fe700 1 -- 172.21.15.70:0/1308072474 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f62a8003fb0 con 0x563d89854320 2024-06-04T05:22:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.494+0000 7f62c11cbb80 1 -- 172.21.15.70:0/1308072474 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62a402c070 con 0x563d89854320 2024-06-04T05:22:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.494+0000 7f628f7fe700 1 -- 172.21.15.70:0/1308072474 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f62a8006070 con 0x563d89854320 2024-06-04T05:22:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.494+0000 7f62c11cbb80 10 client.30589 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.495+0000 7f62c11cbb80 1 -- 172.21.15.70:0/1308072474 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x563d898d2cc0 con 0x563d89854320 2024-06-04T05:22:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.495+0000 7f628f7fe700 1 -- 172.21.15.70:0/1308072474 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f62a402c070 con 0x563d89854320 2024-06-04T05:22:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.495+0000 7f628f7fe700 1 -- 172.21.15.70:0/1308072474 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f62a800e7a0 con 0x563d89854320 2024-06-04T05:22:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.495+0000 7f62c11cbb80 10 client.30589 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.495+0000 7f62c11cbb80 10 client.30589 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.495+0000 7f62c11cbb80 20 client.30589 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.495+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/1308072474 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563d898681a0 0x563d898d52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:53.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.495+0000 7f62c11cbb80 4 client.30589 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:22:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.495+0000 7f62c11cbb80 1 -- 172.21.15.70:0/1308072474 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563d897a84a0 con 0x563d898681a0 2024-06-04T05:22:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.495+0000 7f62affff700 1 --2- 172.21.15.70:0/1308072474 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563d898681a0 0x563d898d52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.496+0000 7f628f7fe700 10 client.30589 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.496+0000 7f62affff700 1 --2- 172.21.15.70:0/1308072474 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563d898681a0 0x563d898d52e0 crc :-1 s=READY pgs=532 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ce1b56db917d634 server_cookie=1bb9dfbf213a09f3 in_seq=0 out_seq=0 2024-06-04T05:22:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.496+0000 7f628f7fe700 1 -- 172.21.15.70:0/1308072474 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563d897a84a0 con 0x563d898681a0 2024-06-04T05:22:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.496+0000 7f628f7fe700 10 client.30589 handle_command_reply: tid=0 2024-06-04T05:22:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.496+0000 7f62c11cbb80 1 client.30589 shutdown 2024-06-04T05:22:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.496+0000 7f62c11cbb80 1 -- 172.21.15.70:0/1308072474 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d89854320 msgr2=0x563d8986df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.496+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/1308072474 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d89854320 0x563d8986df30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f62a8004620 tx=0x7f62a8009380 comp rx=0 tx=0).stop 2024-06-04T05:22:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.496+0000 7f62c11cbb80 1 -- 172.21.15.70:0/1308072474 shutdown_connections 2024-06-04T05:22:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.496+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/1308072474 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563d89854320 0x563d8986df30 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.496+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/1308072474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d8984e8a0 0x563d89870450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.496+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/1308072474 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x563d898681a0 0x563d898d52e0 crc :-1 s=READY pgs=532 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.496+0000 7f62c11cbb80 1 --2- 172.21.15.70:0/1308072474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d8984f260 0x563d8986d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.496+0000 7f62c11cbb80 1 -- 172.21.15.70:0/1308072474 >> 172.21.15.70:0/1308072474 conn(0x563d8977a240 msgr2=0x563d898525e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.497+0000 7f628f7fe700 0 client.30589 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.497+0000 7f62c11cbb80 1 -- 172.21.15.70:0/1308072474 shutdown_connections 2024-06-04T05:22:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.497+0000 7f62c11cbb80 1 -- 172.21.15.70:0/1308072474 wait complete. 2024-06-04T05:22:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.497+0000 7f62c11cbb80 20 client.30589 trim_cache size 0 max 16384 2024-06-04T05:22:53.504 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:22:53.504 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:22:53.504 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:22:53.504 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:22:53.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.497+0000 7f62b6936700 1 -- 172.21.15.70:0/488709730 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62980b29f0 msgr2=0x7f62980b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:53.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.497+0000 7f62b6936700 1 --2- 172.21.15.70:0/488709730 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62980b29f0 0x7f62980b4eb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f62b00ffc40 tx=0x7f62a4009260 comp rx=0 tx=0).stop 2024-06-04T05:22:53.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.497+0000 7f62b6936700 1 -- 172.21.15.70:0/488709730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b0101b30 msgr2=0x7f62b019f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:53.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.498+0000 7f62b6936700 1 --2- 172.21.15.70:0/488709730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b0101b30 0x7f62b019f7b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f62a0006010 tx=0x7f62a0003a10 comp rx=0 tx=0).stop 2024-06-04T05:22:53.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.498+0000 7f62b6936700 1 -- 172.21.15.70:0/488709730 shutdown_connections 2024-06-04T05:22:53.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.498+0000 7f62b6936700 1 --2- 172.21.15.70:0/488709730 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f62980b29f0 0x7f62980b4eb0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.498+0000 7f62b6936700 1 --2- 172.21.15.70:0/488709730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62b0109520 0x7f62b0199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.498+0000 7f62b6936700 1 --2- 172.21.15.70:0/488709730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b0101b30 0x7f62b019f7b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.498+0000 7f62b6936700 1 --2- 172.21.15.70:0/488709730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62b01024f0 0x7f62b019fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:53.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.498+0000 7f62b6936700 1 -- 172.21.15.70:0/488709730 >> 172.21.15.70:0/488709730 conn(0x7f62b0078ab0 msgr2=0x7f62b01003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:53.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.498+0000 7f62b6936700 1 -- 172.21.15.70:0/488709730 shutdown_connections 2024-06-04T05:22:53.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:53.498+0000 7f62b6936700 1 -- 172.21.15.70:0/488709730 wait complete. 2024-06-04T05:22:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:53 smithi098 ceph-mon[35569]: pgmap v810: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.4 MiB/s wr, 416 op/s 2024-06-04T05:22:53.815 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:53 smithi070 ceph-mon[32631]: pgmap v810: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.4 MiB/s wr, 416 op/s 2024-06-04T05:22:53.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 88004 6.60 MB/sec execute 585 sec latency 76.450 ms 2024-06-04T05:22:53.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:53 smithi112 ceph-mon[36453]: pgmap v810: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.4 MiB/s wr, 416 op/s 2024-06-04T05:22:54.221 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:22:54.222 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:22:54.563 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:22:54.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 88136 6.60 MB/sec execute 586 sec latency 93.483 ms 2024-06-04T05:22:55.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:55 smithi070 ceph-mon[32631]: pgmap v811: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.9 MiB/s wr, 286 op/s 2024-06-04T05:22:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:55 smithi098 ceph-mon[35569]: pgmap v811: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.9 MiB/s wr, 286 op/s 2024-06-04T05:22:55.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 88283 6.60 MB/sec execute 587 sec latency 105.743 ms 2024-06-04T05:22:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:55 smithi112 ceph-mon[36453]: pgmap v811: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.9 MiB/s wr, 286 op/s 2024-06-04T05:22:56.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 88432 6.59 MB/sec execute 588 sec latency 130.932 ms 2024-06-04T05:22:56.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.865+0000 7fa3d761a700 1 -- 172.21.15.70:0/2219657172 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3d0108500 msgr2=0x7fa3d01088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:56.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.865+0000 7fa3d761a700 1 --2- 172.21.15.70:0/2219657172 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3d0108500 0x7fa3d01088e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0098f0 tx=0x7fa3cc009c00 comp rx=0 tx=0).stop 2024-06-04T05:22:56.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.865+0000 7fa3d761a700 1 -- 172.21.15.70:0/2219657172 shutdown_connections 2024-06-04T05:22:56.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.865+0000 7fa3d761a700 1 --2- 172.21.15.70:0/2219657172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d0103f90 0x7fa3d0110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:56.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.865+0000 7fa3d761a700 1 --2- 172.21.15.70:0/2219657172 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3d0108500 0x7fa3d01088e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:56.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.865+0000 7fa3d761a700 1 --2- 172.21.15.70:0/2219657172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3d01035f0 0x7fa3d0103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:56.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.865+0000 7fa3d761a700 1 -- 172.21.15.70:0/2219657172 >> 172.21.15.70:0/2219657172 conn(0x7fa3d007a6a0 msgr2=0x7fa3d007aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:56.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.865+0000 7fa3d761a700 1 -- 172.21.15.70:0/2219657172 shutdown_connections 2024-06-04T05:22:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.865+0000 7fa3d761a700 1 -- 172.21.15.70:0/2219657172 wait complete. 2024-06-04T05:22:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.866+0000 7fa3d761a700 1 Processor -- start 2024-06-04T05:22:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.866+0000 7fa3d761a700 1 -- start start 2024-06-04T05:22:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.867+0000 7fa3d761a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d01035f0 0x7fa3d019f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.867+0000 7fa3d761a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3d0103f90 0x7fa3d019fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.867+0000 7fa3d761a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3d0108500 0x7fa3d0199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.867+0000 7fa3d761a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d0113690 con 0x7fa3d0103f90 2024-06-04T05:22:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.867+0000 7fa3d4bb5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3d0103f90 0x7fa3d019fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.867+0000 7fa3d5bb7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3d0108500 0x7fa3d0199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.867+0000 7fa3d5bb7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3d0108500 0x7fa3d0199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46252/0 (socket says 172.21.15.70:46252) 2024-06-04T05:22:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.867+0000 7fa3d5bb7700 1 -- 172.21.15.70:0/2363017228 learned_addr learned my addr 172.21.15.70:0/2363017228 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.867+0000 7fa3d761a700 1 -- 172.21.15.70:0/2363017228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d0113510 con 0x7fa3d01035f0 2024-06-04T05:22:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.868+0000 7fa3d761a700 1 -- 172.21.15.70:0/2363017228 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d0113810 con 0x7fa3d0108500 2024-06-04T05:22:56.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.868+0000 7fa3d53b6700 1 --2- 172.21.15.70:0/2363017228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d01035f0 0x7fa3d019f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:56.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.868+0000 7fa3d5bb7700 1 -- 172.21.15.70:0/2363017228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d01035f0 msgr2=0x7fa3d019f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:56.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.868+0000 7fa3d5bb7700 1 --2- 172.21.15.70:0/2363017228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d01035f0 0x7fa3d019f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:56.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.868+0000 7fa3d5bb7700 1 -- 172.21.15.70:0/2363017228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3d0103f90 msgr2=0x7fa3d019fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:56.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.868+0000 7fa3d5bb7700 1 --2- 172.21.15.70:0/2363017228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3d0103f90 0x7fa3d019fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:56.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.868+0000 7fa3d5bb7700 1 -- 172.21.15.70:0/2363017228 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d019a0f0 con 0x7fa3d0108500 2024-06-04T05:22:56.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.869+0000 7fa3d53b6700 1 --2- 172.21.15.70:0/2363017228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d01035f0 0x7fa3d019f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:22:56.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.869+0000 7fa3d4bb5700 1 --2- 172.21.15.70:0/2363017228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3d0103f90 0x7fa3d019fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:22:56.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.869+0000 7fa3d5bb7700 1 --2- 172.21.15.70:0/2363017228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3d0108500 0x7fa3d0199990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa3c800efc0 tx=0x7fa3c800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:56.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.869+0000 7fa3c67fc700 1 -- 172.21.15.70:0/2363017228 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c80092e0 con 0x7fa3d0108500 2024-06-04T05:22:56.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.869+0000 7fa3c67fc700 1 -- 172.21.15.70:0/2363017228 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa3c8004bb0 con 0x7fa3d0108500 2024-06-04T05:22:56.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.869+0000 7fa3c67fc700 1 -- 172.21.15.70:0/2363017228 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8004020 con 0x7fa3d0108500 2024-06-04T05:22:56.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.869+0000 7fa3d761a700 1 -- 172.21.15.70:0/2363017228 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3d019a380 con 0x7fa3d0108500 2024-06-04T05:22:56.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.869+0000 7fa3d761a700 1 -- 172.21.15.70:0/2363017228 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3d019a7b0 con 0x7fa3d0108500 2024-06-04T05:22:56.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.871+0000 7fa3d761a700 1 -- 172.21.15.70:0/2363017228 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3d010bb30 con 0x7fa3d0108500 2024-06-04T05:22:56.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.873+0000 7fa3c67fc700 1 -- 172.21.15.70:0/2363017228 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fa3c801b030 con 0x7fa3d0108500 2024-06-04T05:22:56.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.874+0000 7fa3c67fc700 1 --2- 172.21.15.70:0/2363017228 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa3bc0b2a70 0x7fa3bc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:56.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.874+0000 7fa3c67fc700 1 -- 172.21.15.70:0/2363017228 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7fa3c80e20b0 con 0x7fa3d0108500 2024-06-04T05:22:56.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.874+0000 7fa3d53b6700 1 --2- 172.21.15.70:0/2363017228 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa3bc0b2a70 0x7fa3bc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:56.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.875+0000 7fa3d53b6700 1 --2- 172.21.15.70:0/2363017228 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa3bc0b2a70 0x7fa3bc0b4f30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0098c0 tx=0x7fa3cc00fb00 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:56.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:56.881+0000 7fa3c67fc700 1 -- 172.21.15.70:0/2363017228 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa3c8022020 con 0x7fa3d0108500 2024-06-04T05:22:57.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:57.315+0000 7fa3d761a700 1 -- 172.21.15.70:0/2363017228 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3d006a230 con 0x7fa3d0108500 2024-06-04T05:22:57.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:57.315+0000 7fa3c67fc700 1 -- 172.21.15.70:0/2363017228 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+11929 (secure 0 0 0) 0x7fa3c8010040 con 0x7fa3d0108500 2024-06-04T05:22:57.318 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:22:57.318 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":67,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:22:05.021574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:22:57.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:57.318+0000 7fa3d761a700 1 -- 172.21.15.70:0/2363017228 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa3bc0b2a70 msgr2=0x7fa3bc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:57.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:57.318+0000 7fa3d761a700 1 --2- 172.21.15.70:0/2363017228 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa3bc0b2a70 0x7fa3bc0b4f30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0098c0 tx=0x7fa3cc00fb00 comp rx=0 tx=0).stop 2024-06-04T05:22:57.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:57.318+0000 7fa3d761a700 1 -- 172.21.15.70:0/2363017228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3d0108500 msgr2=0x7fa3d0199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:57.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:57.318+0000 7fa3d761a700 1 --2- 172.21.15.70:0/2363017228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3d0108500 0x7fa3d0199990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa3c800efc0 tx=0x7fa3c800c3a0 comp rx=0 tx=0).stop 2024-06-04T05:22:57.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:57.318+0000 7fa3d761a700 1 -- 172.21.15.70:0/2363017228 shutdown_connections 2024-06-04T05:22:57.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:57.318+0000 7fa3d761a700 1 --2- 172.21.15.70:0/2363017228 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fa3bc0b2a70 0x7fa3bc0b4f30 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:57.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:57.318+0000 7fa3d761a700 1 --2- 172.21.15.70:0/2363017228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3d0108500 0x7fa3d0199990 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:57.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:57.318+0000 7fa3d761a700 1 --2- 172.21.15.70:0/2363017228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d01035f0 0x7fa3d019f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:57.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:57.318+0000 7fa3d761a700 1 --2- 172.21.15.70:0/2363017228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3d0103f90 0x7fa3d019fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:57.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:57.318+0000 7fa3d761a700 1 -- 172.21.15.70:0/2363017228 >> 172.21.15.70:0/2363017228 conn(0x7fa3d007a6a0 msgr2=0x7fa3d01016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:57.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:57.319+0000 7fa3d761a700 1 -- 172.21.15.70:0/2363017228 shutdown_connections 2024-06-04T05:22:57.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:22:57.319+0000 7fa3d761a700 1 -- 172.21.15.70:0/2363017228 wait complete. 2024-06-04T05:22:57.321 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 67 2024-06-04T05:22:57.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:57 smithi070 ceph-mon[32631]: pgmap v812: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 8.4 MiB/s wr, 410 op/s 2024-06-04T05:22:57.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:57 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2363017228' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:22:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:57 smithi098 ceph-mon[35569]: pgmap v812: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 8.4 MiB/s wr, 410 op/s 2024-06-04T05:22:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:57 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2363017228' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:22:57.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 88594 6.59 MB/sec execute 589 sec latency 79.459 ms 2024-06-04T05:22:57.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:57 smithi112 ceph-mon[36453]: pgmap v812: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 8.4 MiB/s wr, 410 op/s 2024-06-04T05:22:57.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:57 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2363017228' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:22:58.176 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:22:58.525 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:22:58.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 88758 6.59 MB/sec execute 590 sec latency 69.802 ms 2024-06-04T05:22:58.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.858+0000 7fe97e7e7640 1 -- 172.21.15.70:0/2996519254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe978108b40 msgr2=0x7fe978108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:58.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.858+0000 7fe97e7e7640 1 --2- 172.21.15.70:0/2996519254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe978108b40 0x7fe978108f20 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fe968009900 tx=0x7fe96802a480 comp rx=0 tx=0).stop 2024-06-04T05:22:58.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.858+0000 7fe97e7e7640 1 -- 172.21.15.70:0/2996519254 shutdown_connections 2024-06-04T05:22:58.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.858+0000 7fe97e7e7640 1 --2- 172.21.15.70:0/2996519254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe97806b5f0 0x7fe97810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.858+0000 7fe97e7e7640 1 --2- 172.21.15.70:0/2996519254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe97806ac50 0x7fe97806b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.858+0000 7fe97e7e7640 1 --2- 172.21.15.70:0/2996519254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe978108b40 0x7fe978108f20 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.858+0000 7fe97e7e7640 1 -- 172.21.15.70:0/2996519254 >> 172.21.15.70:0/2996519254 conn(0x7fe978078830 msgr2=0x7fe978078c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:58.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.859+0000 7fe97e7e7640 1 -- 172.21.15.70:0/2996519254 shutdown_connections 2024-06-04T05:22:58.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.859+0000 7fe97e7e7640 1 -- 172.21.15.70:0/2996519254 wait complete. 2024-06-04T05:22:58.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.859+0000 7fe97e7e7640 1 Processor -- start 2024-06-04T05:22:58.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.859+0000 7fe97e7e7640 1 -- start start 2024-06-04T05:22:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe97e7e7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe97806ac50 0x7fe97819b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe97e7e7640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe97806b5f0 0x7fe97819b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe97e7e7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe978108b40 0x7fe97819fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe97e7e7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe978112c30 con 0x7fe97806ac50 2024-06-04T05:22:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe97e7e7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe978112ab0 con 0x7fe978108b40 2024-06-04T05:22:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe97e7e7640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe978112db0 con 0x7fe97806b5f0 2024-06-04T05:22:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe97cd84640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe978108b40 0x7fe97819fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe9777fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe97806b5f0 0x7fe97819b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe977fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe97806ac50 0x7fe97819b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe9777fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe97806b5f0 0x7fe97819b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46266/0 (socket says 172.21.15.70:46266) 2024-06-04T05:22:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe9777fe640 1 -- 172.21.15.70:0/4018113629 learned_addr learned my addr 172.21.15.70:0/4018113629 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe977fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe97806ac50 0x7fe97819b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38346/0 (socket says 172.21.15.70:38346) 2024-06-04T05:22:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe97cd84640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe978108b40 0x7fe97819fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59096/0 (socket says 172.21.15.70:59096) 2024-06-04T05:22:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe97cd84640 1 -- 172.21.15.70:0/4018113629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe97806b5f0 msgr2=0x7fe97819b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe97cd84640 1 --2- 172.21.15.70:0/4018113629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe97806b5f0 0x7fe97819b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe97cd84640 1 -- 172.21.15.70:0/4018113629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe97806ac50 msgr2=0x7fe97819b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe97cd84640 1 --2- 172.21.15.70:0/4018113629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe97806ac50 0x7fe97819b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe97cd84640 1 -- 172.21.15.70:0/4018113629 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9781a0240 con 0x7fe978108b40 2024-06-04T05:22:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.860+0000 7fe977fff640 1 --2- 172.21.15.70:0/4018113629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe97806ac50 0x7fe97819b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.861+0000 7fe97cd84640 1 --2- 172.21.15.70:0/4018113629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe978108b40 0x7fe97819fb10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe970009fd0 tx=0x7fe97000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:58.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.861+0000 7fe9757fa640 1 -- 172.21.15.70:0/4018113629 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97000ce20 con 0x7fe978108b40 2024-06-04T05:22:58.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.861+0000 7fe9757fa640 1 -- 172.21.15.70:0/4018113629 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe970005020 con 0x7fe978108b40 2024-06-04T05:22:58.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.861+0000 7fe97e7e7640 1 -- 172.21.15.70:0/4018113629 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9781a0530 con 0x7fe978108b40 2024-06-04T05:22:58.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.861+0000 7fe97e7e7640 1 -- 172.21.15.70:0/4018113629 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe978109840 con 0x7fe978108b40 2024-06-04T05:22:58.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.861+0000 7fe9757fa640 1 -- 172.21.15.70:0/4018113629 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9700216a0 con 0x7fe978108b40 2024-06-04T05:22:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.864+0000 7fe9815de740 1 Processor -- start 2024-06-04T05:22:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.864+0000 7fe9757fa640 1 -- 172.21.15.70:0/4018113629 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fe970004a40 con 0x7fe978108b40 2024-06-04T05:22:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.864+0000 7fe9815de740 1 -- start start 2024-06-04T05:22:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe9815de740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9e9c820 0x5622c9e9cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe9815de740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 0x5622c9ea74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe9815de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9ea7a00 0x5622c9ea9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe9815de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5622c9e0ec30 con 0x5622c9ea7a00 2024-06-04T05:22:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe9815de740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5622c9dfe7d0 con 0x5622c9e9d140 2024-06-04T05:22:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe9815de740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5622c9e02810 con 0x5622c9e9c820 2024-06-04T05:22:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe9777fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9ea7a00 0x5622c9ea9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe9777fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9ea7a00 0x5622c9ea9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38350/0 (socket says 172.21.15.70:38350) 2024-06-04T05:22:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe9777fe640 1 -- 172.21.15.70:0/4061749407 learned_addr learned my addr 172.21.15.70:0/4061749407 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe977fff640 1 --2- 172.21.15.70:0/4061749407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 0x5622c9ea74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe9757fa640 1 --2- 172.21.15.70:0/4018113629 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9600b2a10 0x7fe9600b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe9777fe640 1 -- 172.21.15.70:0/4061749407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9e9c820 msgr2=0x5622c9e9cc00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:22:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe9777fe640 1 --2- 172.21.15.70:0/4061749407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9e9c820 0x5622c9e9cc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe9777fe640 1 -- 172.21.15.70:0/4061749407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 msgr2=0x5622c9ea74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe9777fe640 1 --2- 172.21.15.70:0/4061749407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 0x5622c9ea74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe97cd84640 1 --2- 172.21.15.70:0/4018113629 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9600b2a10 0x7fe9600b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe9777fe640 1 -- 172.21.15.70:0/4061749407 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622c9b13210 con 0x5622c9ea7a00 2024-06-04T05:22:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe977fff640 1 --2- 172.21.15.70:0/4061749407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 0x5622c9ea74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:22:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.865+0000 7fe9757fa640 1 -- 172.21.15.70:0/4018113629 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7fe9700f3d40 con 0x7fe978108b40 2024-06-04T05:22:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.866+0000 7fe9777fe640 1 --2- 172.21.15.70:0/4061749407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9ea7a00 0x5622c9ea9df0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fe96c009dd0 tx=0x7fe96c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed00acdbdaaa50d6 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.866+0000 7fe97cd84640 1 --2- 172.21.15.70:0/4018113629 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9600b2a10 0x7fe9600b4ed0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe9700f1510 tx=0x7fe9700857c0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.866+0000 7fe95e7fc640 1 -- 172.21.15.70:0/4061749407 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c005020 con 0x5622c9ea7a00 2024-06-04T05:22:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.866+0000 7fe95e7fc640 1 -- 172.21.15.70:0/4061749407 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe96c0047a0 con 0x5622c9ea7a00 2024-06-04T05:22:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.866+0000 7fe95e7fc640 1 -- 172.21.15.70:0/4061749407 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c019670 con 0x5622c9ea7a00 2024-06-04T05:22:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.867+0000 7fe9815de740 1 -- 172.21.15.70:0/4061749407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9ea7a00 msgr2=0x5622c9ea9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.867+0000 7fe9815de740 1 --2- 172.21.15.70:0/4061749407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9ea7a00 0x5622c9ea9df0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fe96c009dd0 tx=0x7fe96c011810 comp rx=0 tx=0).stop 2024-06-04T05:22:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.867+0000 7fe9815de740 1 -- 172.21.15.70:0/4061749407 shutdown_connections 2024-06-04T05:22:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.867+0000 7fe9815de740 1 --2- 172.21.15.70:0/4061749407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 0x5622c9ea74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.867+0000 7fe9815de740 1 --2- 172.21.15.70:0/4061749407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9ea7a00 0x5622c9ea9df0 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.867+0000 7fe9815de740 1 --2- 172.21.15.70:0/4061749407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9e9c820 0x5622c9e9cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.867+0000 7fe9815de740 1 -- 172.21.15.70:0/4061749407 >> 172.21.15.70:0/4061749407 conn(0x5622c9e98d30 msgr2=0x5622c9e99110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.867+0000 7fe9815de740 1 -- 172.21.15.70:0/4061749407 shutdown_connections 2024-06-04T05:22:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.867+0000 7fe9815de740 1 -- 172.21.15.70:0/4061749407 wait complete. 2024-06-04T05:22:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.867+0000 7fe9815de740 1 -- start start 2024-06-04T05:22:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.868+0000 7fe9815de740 1 Processor -- start 2024-06-04T05:22:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.868+0000 7fe9815de740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9e9c820 0x5622c9ea49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.868+0000 7fe9815de740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 0x5622c9ea4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.868+0000 7fe9815de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9ea7a00 0x5622c9ea5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.868+0000 7fe9815de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5622c9ae4790 con 0x5622c9ea7a00 2024-06-04T05:22:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.868+0000 7fe9815de740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5622c9dba890 con 0x5622c9e9d140 2024-06-04T05:22:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.868+0000 7fe9815de740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5622c9eaa630 con 0x5622c9e9c820 2024-06-04T05:22:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.868+0000 7fe9815de740 10 client.0 _ll_register_callbacks cb 0x5622c9b12a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.868+0000 7fe9777fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9e9c820 0x5622c9ea49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.868+0000 7fe9777fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9e9c820 0x5622c9ea49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46284/0 (socket says 172.21.15.70:46284) 2024-06-04T05:22:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.868+0000 7fe9777fe640 1 -- 172.21.15.70:0/3514137927 learned_addr learned my addr 172.21.15.70:0/3514137927 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.868+0000 7fe977fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 0x5622c9ea4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.868+0000 7fe9777fe640 1 --2- 172.21.15.70:0/3514137927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9ea7a00 0x5622c9ea5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.869+0000 7fe977fff640 1 -- 172.21.15.70:0/3514137927 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9e9c820 msgr2=0x5622c9ea49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.869+0000 7fe977fff640 1 --2- 172.21.15.70:0/3514137927 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9e9c820 0x5622c9ea49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.869+0000 7fe977fff640 1 -- 172.21.15.70:0/3514137927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9ea7a00 msgr2=0x5622c9ea5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.869+0000 7fe977fff640 1 --2- 172.21.15.70:0/3514137927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9ea7a00 0x5622c9ea5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.869+0000 7fe977fff640 1 -- 172.21.15.70:0/3514137927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622c9bdf620 con 0x5622c9e9d140 2024-06-04T05:22:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.869+0000 7fe977fff640 1 --2- 172.21.15.70:0/3514137927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 0x5622c9ea4f20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe96802a9a0 tx=0x7fe968003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9679dcb7db57a48 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.869+0000 7fe9577fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:22:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.869+0000 7fe9577fe640 1 -- 172.21.15.70:0/3514137927 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe968035040 con 0x5622c9e9d140 2024-06-04T05:22:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.869+0000 7fe9577fe640 1 -- 172.21.15.70:0/3514137927 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7fe968003da0 con 0x5622c9e9d140 2024-06-04T05:22:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.869+0000 7fe9815de740 1 -- 172.21.15.70:0/3514137927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe96c0047a0 con 0x5622c9e9d140 2024-06-04T05:22:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.870+0000 7fe9577fe640 1 -- 172.21.15.70:0/3514137927 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fe968005030 con 0x5622c9e9d140 2024-06-04T05:22:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.870+0000 7fe9815de740 10 client.30479 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.870+0000 7fe9815de740 1 -- 172.21.15.70:0/3514137927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5622c9eb6410 con 0x5622c9e9d140 2024-06-04T05:22:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.870+0000 7fe9577fe640 1 -- 172.21.15.70:0/3514137927 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe96c0047a0 con 0x5622c9e9d140 2024-06-04T05:22:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.870+0000 7fe9577fe640 1 -- 172.21.15.70:0/3514137927 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7fe96803c440 con 0x5622c9e9d140 2024-06-04T05:22:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.870+0000 7fe9815de740 10 client.30479 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.870+0000 7fe9815de740 10 client.30479 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.870+0000 7fe9815de740 20 client.30479 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.871+0000 7fe9815de740 1 --2- 172.21.15.70:0/3514137927 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5622c9eb7920 0x5622c9eb9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.871+0000 7fe9815de740 4 client.30479 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:22:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.871+0000 7fe9815de740 1 -- 172.21.15.70:0/3514137927 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622c9e10b80 con 0x5622c9eb7920 2024-06-04T05:22:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.871+0000 7fe9777fe640 1 --2- 172.21.15.70:0/3514137927 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5622c9eb7920 0x5622c9eb9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.871+0000 7fe9577fe640 10 client.30479 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.871+0000 7fe9777fe640 1 --2- 172.21.15.70:0/3514137927 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5622c9eb7920 0x5622c9eb9d10 crc :-1 s=READY pgs=533 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66ecee19fe55d356 server_cookie=2a16dabd5c8ae1c0 in_seq=0 out_seq=0 2024-06-04T05:22:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.872+0000 7fe9577fe640 1 -- 172.21.15.70:0/3514137927 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5622c9e10b80 con 0x5622c9eb7920 2024-06-04T05:22:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.872+0000 7fe9577fe640 10 client.30479 handle_command_reply: tid=0 2024-06-04T05:22:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.872+0000 7fe9815de740 1 client.30479 shutdown 2024-06-04T05:22:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.872+0000 7fe9815de740 1 -- 172.21.15.70:0/3514137927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 msgr2=0x5622c9ea4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.872+0000 7fe9815de740 1 --2- 172.21.15.70:0/3514137927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 0x5622c9ea4f20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe96802a9a0 tx=0x7fe968003820 comp rx=0 tx=0).stop 2024-06-04T05:22:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.873+0000 7fe9815de740 1 -- 172.21.15.70:0/3514137927 shutdown_connections 2024-06-04T05:22:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.873+0000 7fe9815de740 1 --2- 172.21.15.70:0/3514137927 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5622c9eb7920 0x5622c9eb9d10 crc :-1 s=READY pgs=533 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.873+0000 7fe9815de740 1 --2- 172.21.15.70:0/3514137927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 0x5622c9ea4f20 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.873+0000 7fe9815de740 1 --2- 172.21.15.70:0/3514137927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9ea7a00 0x5622c9ea5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.873+0000 7fe9815de740 1 --2- 172.21.15.70:0/3514137927 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9e9c820 0x5622c9ea49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.873+0000 7fe9815de740 1 -- 172.21.15.70:0/3514137927 >> 172.21.15.70:0/3514137927 conn(0x5622c9e98d30 msgr2=0x5622c9e97fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.873+0000 7fe9577fe640 0 client.30479 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.873+0000 7fe9815de740 1 -- 172.21.15.70:0/3514137927 shutdown_connections 2024-06-04T05:22:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.873+0000 7fe9815de740 1 -- 172.21.15.70:0/3514137927 wait complete. 2024-06-04T05:22:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.873+0000 7fe9815de740 20 client.30479 trim_cache size 0 max 16384 2024-06-04T05:22:58.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.894+0000 7fe9815de740 1 Processor -- start 2024-06-04T05:22:58.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.894+0000 7fe9815de740 1 -- start start 2024-06-04T05:22:58.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.895+0000 7fe9815de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9e9c820 0x5622c9ea3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.895+0000 7fe9815de740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 0x5622c9ea5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.895+0000 7fe9815de740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9ea7a00 0x5622c9e9e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.895+0000 7fe9815de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe978070b00 con 0x5622c9e9c820 2024-06-04T05:22:58.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.895+0000 7fe9815de740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe978072640 con 0x5622c9e9d140 2024-06-04T05:22:58.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.895+0000 7fe9815de740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe9780711d0 con 0x5622c9ea7a00 2024-06-04T05:22:58.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.895+0000 7fe977fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 0x5622c9ea5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:58.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.895+0000 7fe977fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 0x5622c9ea5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59116/0 (socket says 172.21.15.70:59116) 2024-06-04T05:22:58.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.895+0000 7fe977fff640 1 -- 172.21.15.70:0/2126323732 learned_addr learned my addr 172.21.15.70:0/2126323732 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:58.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.895+0000 7fe9777fe640 1 --2- 172.21.15.70:0/2126323732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9e9c820 0x5622c9ea3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:58.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.895+0000 7fe9777fe640 1 --2- 172.21.15.70:0/2126323732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9ea7a00 0x5622c9e9e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:58.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.895+0000 7fe977fff640 1 -- 172.21.15.70:0/2126323732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9ea7a00 msgr2=0x5622c9e9e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:58.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.895+0000 7fe977fff640 1 --2- 172.21.15.70:0/2126323732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9ea7a00 0x5622c9e9e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.895+0000 7fe977fff640 1 -- 172.21.15.70:0/2126323732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9e9c820 msgr2=0x5622c9ea3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:58.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.896+0000 7fe977fff640 1 --2- 172.21.15.70:0/2126323732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9e9c820 0x5622c9ea3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.896+0000 7fe977fff640 1 -- 172.21.15.70:0/2126323732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622c9e99670 con 0x5622c9e9d140 2024-06-04T05:22:58.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.896+0000 7fe977fff640 1 --2- 172.21.15.70:0/2126323732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 0x5622c9ea5280 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe968009850 tx=0x7fe96802a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f4aba0948cd04ff server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.896+0000 7fe95dffb640 1 -- 172.21.15.70:0/2126323732 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe968052020 con 0x5622c9e9d140 2024-06-04T05:22:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.896+0000 7fe95dffb640 1 -- 172.21.15.70:0/2126323732 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe968005850 con 0x5622c9e9d140 2024-06-04T05:22:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.896+0000 7fe95dffb640 1 -- 172.21.15.70:0/2126323732 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe968035070 con 0x5622c9e9d140 2024-06-04T05:22:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.897+0000 7fe9815de740 1 -- 172.21.15.70:0/2126323732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 msgr2=0x5622c9ea5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.897+0000 7fe9815de740 1 --2- 172.21.15.70:0/2126323732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 0x5622c9ea5280 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe968009850 tx=0x7fe96802a480 comp rx=0 tx=0).stop 2024-06-04T05:22:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.897+0000 7fe9815de740 1 -- 172.21.15.70:0/2126323732 shutdown_connections 2024-06-04T05:22:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.897+0000 7fe9815de740 1 --2- 172.21.15.70:0/2126323732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9d140 0x5622c9ea5280 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.897+0000 7fe9815de740 1 --2- 172.21.15.70:0/2126323732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9ea7a00 0x5622c9e9e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.897+0000 7fe9815de740 1 --2- 172.21.15.70:0/2126323732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9e9c820 0x5622c9ea3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.897+0000 7fe9815de740 1 -- 172.21.15.70:0/2126323732 >> 172.21.15.70:0/2126323732 conn(0x5622c9e98d30 msgr2=0x5622c9e9e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.897+0000 7fe9815de740 1 -- 172.21.15.70:0/2126323732 shutdown_connections 2024-06-04T05:22:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.897+0000 7fe9815de740 1 -- 172.21.15.70:0/2126323732 wait complete. 2024-06-04T05:22:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.897+0000 7fe9815de740 1 -- start start 2024-06-04T05:22:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.898+0000 7fe9815de740 1 Processor -- start 2024-06-04T05:22:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.898+0000 7fe9815de740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9c820 0x5622c9eaa980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.898+0000 7fe9815de740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9e9d140 0x5622c9eaaec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.898+0000 7fe9815de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9ea7a00 0x5622c9f20400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.898+0000 7fe9815de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe978071f70 con 0x5622c9ea7a00 2024-06-04T05:22:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.898+0000 7fe9815de740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe9780718a0 con 0x5622c9e9c820 2024-06-04T05:22:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.898+0000 7fe9815de740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe97806eb20 con 0x5622c9e9d140 2024-06-04T05:22:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.898+0000 7fe9815de740 10 client.0 _ll_register_callbacks cb 0x7fe968004480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:22:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.898+0000 7fe977fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9e9d140 0x5622c9eaaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.899+0000 7fe977fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9e9d140 0x5622c9eaaec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46308/0 (socket says 172.21.15.70:46308) 2024-06-04T05:22:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.899+0000 7fe977fff640 1 -- 172.21.15.70:0/161433983 learned_addr learned my addr 172.21.15.70:0/161433983 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:22:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.899+0000 7fe9777fe640 1 --2- 172.21.15.70:0/161433983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9c820 0x5622c9eaa980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:58.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.899+0000 7fe9777fe640 1 --2- 172.21.15.70:0/161433983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9ea7a00 0x5622c9f20400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:58.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.899+0000 7fe977fff640 1 -- 172.21.15.70:0/161433983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9c820 msgr2=0x5622c9eaa980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:58.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.899+0000 7fe977fff640 1 --2- 172.21.15.70:0/161433983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9c820 0x5622c9eaa980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.899+0000 7fe977fff640 1 -- 172.21.15.70:0/161433983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9ea7a00 msgr2=0x5622c9f20400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:22:58.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.899+0000 7fe977fff640 1 --2- 172.21.15.70:0/161433983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9ea7a00 0x5622c9f20400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:58.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.899+0000 7fe977fff640 1 -- 172.21.15.70:0/161433983 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622c9f20940 con 0x5622c9e9d140 2024-06-04T05:22:58.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.899+0000 7fe9777fe640 1 --2- 172.21.15.70:0/161433983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9c820 0x5622c9eaa980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:22:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.899+0000 7fe977fff640 1 --2- 172.21.15.70:0/161433983 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9e9d140 0x5622c9eaaec0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fe968004590 tx=0x7fe96803cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fe3e17ab4433c0c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:22:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.899+0000 7fe9577fe640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:22:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.900+0000 7fe9577fe640 1 -- 172.21.15.70:0/161433983 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe968052030 con 0x5622c9e9d140 2024-06-04T05:22:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.900+0000 7fe9577fe640 1 -- 172.21.15.70:0/161433983 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7fe968003ba0 con 0x5622c9e9d140 2024-06-04T05:22:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.900+0000 7fe9815de740 1 -- 172.21.15.70:0/161433983 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe968005850 con 0x5622c9e9d140 2024-06-04T05:22:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.900+0000 7fe9577fe640 1 -- 172.21.15.70:0/161433983 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fe968066050 con 0x5622c9e9d140 2024-06-04T05:22:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.900+0000 7fe9815de740 10 client.30610 fetch_fsmap learned FSMap version 67 2024-06-04T05:22:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.900+0000 7fe9815de740 1 -- 172.21.15.70:0/161433983 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5622c9f21ad0 con 0x5622c9e9d140 2024-06-04T05:22:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.901+0000 7fe9577fe640 1 -- 172.21.15.70:0/161433983 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe968005850 con 0x5622c9e9d140 2024-06-04T05:22:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.901+0000 7fe9577fe640 1 -- 172.21.15.70:0/161433983 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7fe9680624f0 con 0x5622c9e9d140 2024-06-04T05:22:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.901+0000 7fe9815de740 10 client.30610 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:22:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.901+0000 7fe9815de740 10 client.30610 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:22:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.901+0000 7fe9815de740 20 client.30610 populate_metadata read hostname 'smithi070' 2024-06-04T05:22:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.901+0000 7fe9815de740 1 --2- 172.21.15.70:0/161433983 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5622c9f22090 0x5622c9f244b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:22:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.901+0000 7fe9815de740 4 client.30610 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:22:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.901+0000 7fe9815de740 1 -- 172.21.15.70:0/161433983 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5622c9e0a360 con 0x5622c9f22090 2024-06-04T05:22:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.901+0000 7fe9777fe640 1 --2- 172.21.15.70:0/161433983 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5622c9f22090 0x5622c9f244b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:22:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.901+0000 7fe9577fe640 10 client.30610 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:58.902+0000 7fe9777fe640 1 --2- 172.21.15.70:0/161433983 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5622c9f22090 0x5622c9f244b0 crc :-1 s=READY pgs=534 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98bdce42c76c5096 server_cookie=76527308e4b49d37 in_seq=0 out_seq=0 2024-06-04T05:22:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.004+0000 7fe9577fe640 1 -- 172.21.15.70:0/161433983 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5622c9e0a360 con 0x5622c9f22090 2024-06-04T05:22:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.004+0000 7fe9577fe640 10 client.30610 handle_command_reply: tid=0 2024-06-04T05:22:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.004+0000 7fe9815de740 1 client.30610 shutdown 2024-06-04T05:22:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.005+0000 7fe9815de740 1 -- 172.21.15.70:0/161433983 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9e9d140 msgr2=0x5622c9eaaec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.005+0000 7fe9815de740 1 --2- 172.21.15.70:0/161433983 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9e9d140 0x5622c9eaaec0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fe968004590 tx=0x7fe96803cc80 comp rx=0 tx=0).stop 2024-06-04T05:22:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.005+0000 7fe9815de740 1 -- 172.21.15.70:0/161433983 shutdown_connections 2024-06-04T05:22:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.005+0000 7fe9815de740 1 --2- 172.21.15.70:0/161433983 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x5622c9f22090 0x5622c9f244b0 crc :-1 s=READY pgs=534 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.005+0000 7fe9815de740 1 --2- 172.21.15.70:0/161433983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622c9ea7a00 0x5622c9f20400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.005+0000 7fe9815de740 1 --2- 172.21.15.70:0/161433983 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5622c9e9d140 0x5622c9eaaec0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.005+0000 7fe9815de740 1 --2- 172.21.15.70:0/161433983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622c9e9c820 0x5622c9eaa980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.005+0000 7fe9815de740 1 -- 172.21.15.70:0/161433983 >> 172.21.15.70:0/161433983 conn(0x5622c9e98d30 msgr2=0x5622c9ea52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.005+0000 7fe9577fe640 0 client.30610 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:22:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.005+0000 7fe9815de740 1 -- 172.21.15.70:0/161433983 shutdown_connections 2024-06-04T05:22:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.005+0000 7fe9815de740 1 -- 172.21.15.70:0/161433983 wait complete. 2024-06-04T05:22:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.005+0000 7fe9815de740 20 client.30610 trim_cache size 0 max 16384 2024-06-04T05:22:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:22:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:22:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:22:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.006+0000 7fe97e7e7640 1 -- 172.21.15.70:0/4018113629 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9600b2a10 msgr2=0x7fe9600b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.006+0000 7fe97e7e7640 1 --2- 172.21.15.70:0/4018113629 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9600b2a10 0x7fe9600b4ed0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe9700f1510 tx=0x7fe9700857c0 comp rx=0 tx=0).stop 2024-06-04T05:22:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.006+0000 7fe97e7e7640 1 -- 172.21.15.70:0/4018113629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe978108b40 msgr2=0x7fe97819fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:22:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.006+0000 7fe97e7e7640 1 --2- 172.21.15.70:0/4018113629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe978108b40 0x7fe97819fb10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe970009fd0 tx=0x7fe97000ef10 comp rx=0 tx=0).stop 2024-06-04T05:22:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.006+0000 7fe97e7e7640 1 -- 172.21.15.70:0/4018113629 shutdown_connections 2024-06-04T05:22:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.006+0000 7fe97e7e7640 1 --2- 172.21.15.70:0/4018113629 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe9600b2a10 0x7fe9600b4ed0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.006+0000 7fe97e7e7640 1 --2- 172.21.15.70:0/4018113629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe978108b40 0x7fe97819fb10 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.006+0000 7fe97e7e7640 1 --2- 172.21.15.70:0/4018113629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe97806b5f0 0x7fe97819b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.006+0000 7fe97e7e7640 1 --2- 172.21.15.70:0/4018113629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe97806ac50 0x7fe97819b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:22:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.006+0000 7fe97e7e7640 1 -- 172.21.15.70:0/4018113629 >> 172.21.15.70:0/4018113629 conn(0x7fe978078830 msgr2=0x7fe9780ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:22:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.006+0000 7fe97e7e7640 1 -- 172.21.15.70:0/4018113629 shutdown_connections 2024-06-04T05:22:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:22:59.006+0000 7fe97e7e7640 1 -- 172.21.15.70:0/4018113629 wait complete. 2024-06-04T05:22:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:22:59 smithi098 ceph-mon[35569]: pgmap v813: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.5 MiB/s wr, 254 op/s 2024-06-04T05:22:59.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 88932 6.59 MB/sec execute 591 sec latency 93.185 ms 2024-06-04T05:22:59.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:22:59 smithi070 ceph-mon[32631]: pgmap v813: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.5 MiB/s wr, 254 op/s 2024-06-04T05:22:59.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:22:59 smithi112 ceph-mon[36453]: pgmap v813: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.5 MiB/s wr, 254 op/s 2024-06-04T05:23:00.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.324+0000 7f983ded2700 1 -- 172.21.15.70:0/2204488057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9838102c40 msgr2=0x7f98381030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.324+0000 7f983ded2700 1 --2- 172.21.15.70:0/2204488057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9838102c40 0x7f98381030c0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f9820009990 tx=0x7f9820009ca0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.324+0000 7f983ded2700 1 -- 172.21.15.70:0/2204488057 shutdown_connections 2024-06-04T05:23:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.324+0000 7f983ded2700 1 --2- 172.21.15.70:0/2204488057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838103600 0x7f983810f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.325+0000 7f983ded2700 1 --2- 172.21.15.70:0/2204488057 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98381084c0 0x7f98381088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.325+0000 7f983ded2700 1 --2- 172.21.15.70:0/2204488057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9838102c40 0x7f98381030c0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.325+0000 7f983ded2700 1 -- 172.21.15.70:0/2204488057 >> 172.21.15.70:0/2204488057 conn(0x7f9838078ab0 msgr2=0x7f98380ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.325+0000 7f983ded2700 1 -- 172.21.15.70:0/2204488057 shutdown_connections 2024-06-04T05:23:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.325+0000 7f983ded2700 1 -- 172.21.15.70:0/2204488057 wait complete. 2024-06-04T05:23:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.325+0000 7f983ded2700 1 Processor -- start 2024-06-04T05:23:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.325+0000 7f983ded2700 1 -- start start 2024-06-04T05:23:00.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f983ded2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838102c40 0x7f983819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f983ded2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9838103600 0x7f983819ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f983ded2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98381084c0 0x7f98381a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f983ded2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f98381124a0 con 0x7f9838103600 2024-06-04T05:23:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f983ded2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9838112320 con 0x7f9838102c40 2024-06-04T05:23:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f983ded2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9838112620 con 0x7f98381084c0 2024-06-04T05:23:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f9836ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9838103600 0x7f983819ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f98377fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838102c40 0x7f983819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f9836ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9838103600 0x7f983819ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38388/0 (socket says 172.21.15.70:38388) 2024-06-04T05:23:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f9836ffd700 1 -- 172.21.15.70:0/662961532 learned_addr learned my addr 172.21.15.70:0/662961532 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f9837fff700 1 --2- 172.21.15.70:0/662961532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98381084c0 0x7f98381a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f98377fe700 1 -- 172.21.15.70:0/662961532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98381084c0 msgr2=0x7f98381a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f98377fe700 1 --2- 172.21.15.70:0/662961532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98381084c0 0x7f98381a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f98377fe700 1 -- 172.21.15.70:0/662961532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9838103600 msgr2=0x7f983819ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f98377fe700 1 --2- 172.21.15.70:0/662961532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9838103600 0x7f983819ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f98377fe700 1 -- 172.21.15.70:0/662961532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98381a5ee0 con 0x7f9838102c40 2024-06-04T05:23:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f9837fff700 1 --2- 172.21.15.70:0/662961532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98381084c0 0x7f98381a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:23:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.326+0000 7f98377fe700 1 --2- 172.21.15.70:0/662961532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838102c40 0x7f983819a920 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f982800ef90 tx=0x7f982800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.327+0000 7f9834ff9700 1 -- 172.21.15.70:0/662961532 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9828003a00 con 0x7f9838102c40 2024-06-04T05:23:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.327+0000 7f9834ff9700 1 -- 172.21.15.70:0/662961532 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f982800ce50 con 0x7f9838102c40 2024-06-04T05:23:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.327+0000 7f983ded2700 1 -- 172.21.15.70:0/662961532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98381a6110 con 0x7f9838102c40 2024-06-04T05:23:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.327+0000 7f9834ff9700 1 -- 172.21.15.70:0/662961532 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98280046a0 con 0x7f9838102c40 2024-06-04T05:23:00.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.327+0000 7f983ded2700 1 -- 172.21.15.70:0/662961532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98381a6620 con 0x7f9838102c40 2024-06-04T05:23:00.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.329+0000 7f9834ff9700 1 -- 172.21.15.70:0/662961532 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f9828012050 con 0x7f9838102c40 2024-06-04T05:23:00.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.330+0000 7f9834ff9700 1 --2- 172.21.15.70:0/662961532 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f98240b2a70 0x7f98240b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.330+0000 7f9836ffd700 1 --2- 172.21.15.70:0/662961532 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f98240b2a70 0x7f98240b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.330+0000 7f9834ff9700 1 -- 172.21.15.70:0/662961532 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f98280e2340 con 0x7f9838102c40 2024-06-04T05:23:00.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.330+0000 7f9836ffd700 1 --2- 172.21.15.70:0/662961532 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f98240b2a70 0x7f98240b4f30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f98381026a0 tx=0x7f9820005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:00.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.330+0000 7f9848767b80 1 Processor -- start 2024-06-04T05:23:00.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.330+0000 7f9848767b80 1 -- start start 2024-06-04T05:23:00.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.331+0000 7f9848767b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9c5860 0x55b08e9c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.331+0000 7f9848767b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c6210 0x55b08e9c9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.331+0000 7f9848767b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9c9f60 0x55b08e9ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.331+0000 7f9848767b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b08e924c90 con 0x55b08e9c5860 2024-06-04T05:23:00.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.331+0000 7f9848767b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b08e914830 con 0x55b08e9c6210 2024-06-04T05:23:00.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.331+0000 7f9848767b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b08e918870 con 0x55b08e9c9f60 2024-06-04T05:23:00.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.331+0000 7f98377fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9c5860 0x55b08e9c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.331+0000 7f98377fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9c5860 0x55b08e9c5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38390/0 (socket says 172.21.15.70:38390) 2024-06-04T05:23:00.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.331+0000 7f98377fe700 1 -- 172.21.15.70:0/2322272914 learned_addr learned my addr 172.21.15.70:0/2322272914 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:00.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.331+0000 7f9836ffd700 1 --2- 172.21.15.70:0/2322272914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c6210 0x55b08e9c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.331+0000 7f9837fff700 1 --2- 172.21.15.70:0/2322272914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9c9f60 0x55b08e9ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.331+0000 7f98377fe700 1 -- 172.21.15.70:0/2322272914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9c9f60 msgr2=0x55b08e9ca3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:00.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.331+0000 7f98377fe700 1 --2- 172.21.15.70:0/2322272914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9c9f60 0x55b08e9ca3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.331+0000 7f98377fe700 1 -- 172.21.15.70:0/2322272914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c6210 msgr2=0x55b08e9c9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:00.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.331+0000 7f98377fe700 1 --2- 172.21.15.70:0/2322272914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c6210 0x55b08e9c9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.331+0000 7f98377fe700 1 -- 172.21.15.70:0/2322272914 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b08e8648d0 con 0x55b08e9c5860 2024-06-04T05:23:00.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.331+0000 7f98377fe700 1 --2- 172.21.15.70:0/2322272914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9c5860 0x55b08e9c5c40 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f98280a5670 tx=0x7f98280a8d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f71c7d85827b68b3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:00.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.332+0000 7f982dffb700 1 -- 172.21.15.70:0/2322272914 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98280a9cf0 con 0x55b08e9c5860 2024-06-04T05:23:00.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.332+0000 7f982dffb700 1 -- 172.21.15.70:0/2322272914 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98280a95f0 con 0x55b08e9c5860 2024-06-04T05:23:00.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.332+0000 7f982dffb700 1 -- 172.21.15.70:0/2322272914 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98280a5880 con 0x55b08e9c5860 2024-06-04T05:23:00.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.332+0000 7f9848767b80 1 -- 172.21.15.70:0/2322272914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9c5860 msgr2=0x55b08e9c5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:00.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.332+0000 7f9848767b80 1 --2- 172.21.15.70:0/2322272914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9c5860 0x55b08e9c5c40 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f98280a5670 tx=0x7f98280a8d60 comp rx=0 tx=0).stop 2024-06-04T05:23:00.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.332+0000 7f9848767b80 1 -- 172.21.15.70:0/2322272914 shutdown_connections 2024-06-04T05:23:00.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.332+0000 7f9848767b80 1 --2- 172.21.15.70:0/2322272914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9c9f60 0x55b08e9ca3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.332+0000 7f9848767b80 1 --2- 172.21.15.70:0/2322272914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9c5860 0x55b08e9c5c40 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.332+0000 7f9848767b80 1 --2- 172.21.15.70:0/2322272914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c6210 0x55b08e9c9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.332+0000 7f9848767b80 1 -- 172.21.15.70:0/2322272914 >> 172.21.15.70:0/2322272914 conn(0x55b08e8f3080 msgr2=0x55b08e9bfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:00.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.333+0000 7f9848767b80 1 -- 172.21.15.70:0/2322272914 shutdown_connections 2024-06-04T05:23:00.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.333+0000 7f9848767b80 1 -- 172.21.15.70:0/2322272914 wait complete. 2024-06-04T05:23:00.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.333+0000 7f9848767b80 1 -- start start 2024-06-04T05:23:00.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.333+0000 7f9848767b80 1 Processor -- start 2024-06-04T05:23:00.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.333+0000 7f9848767b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c6210 0x55b08e9cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.333+0000 7f9848767b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9c9f60 0x55b08e9cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.333+0000 7f9848767b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9cf500 0x55b08e9df730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.333+0000 7f9848767b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b08e81afa0 con 0x55b08e9cf500 2024-06-04T05:23:00.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.333+0000 7f9848767b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b08e8b67c0 con 0x55b08e9c6210 2024-06-04T05:23:00.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.333+0000 7f9848767b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b08e9d6410 con 0x55b08e9c9f60 2024-06-04T05:23:00.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.333+0000 7f9848767b80 10 client.0 _ll_register_callbacks cb 0x55b08e8af920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:00.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.334+0000 7f9837fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9cf500 0x55b08e9df730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.334+0000 7f9837fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9cf500 0x55b08e9df730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38396/0 (socket says 172.21.15.70:38396) 2024-06-04T05:23:00.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.334+0000 7f9837fff700 1 -- 172.21.15.70:0/3928379340 learned_addr learned my addr 172.21.15.70:0/3928379340 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:00.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.334+0000 7f98377fe700 1 --2- 172.21.15.70:0/3928379340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c6210 0x55b08e9cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.334+0000 7f9836ffd700 1 --2- 172.21.15.70:0/3928379340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9c9f60 0x55b08e9cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.334+0000 7f9837fff700 1 -- 172.21.15.70:0/3928379340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9c9f60 msgr2=0x55b08e9cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:00.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.334+0000 7f9837fff700 1 --2- 172.21.15.70:0/3928379340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9c9f60 0x55b08e9cefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.334+0000 7f9837fff700 1 -- 172.21.15.70:0/3928379340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c6210 msgr2=0x55b08e9cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:00.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.334+0000 7f9837fff700 1 --2- 172.21.15.70:0/3928379340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c6210 0x55b08e9cea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.334+0000 7f9837fff700 1 -- 172.21.15.70:0/3928379340 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b08e8b3690 con 0x55b08e9cf500 2024-06-04T05:23:00.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.334+0000 7f98377fe700 1 --2- 172.21.15.70:0/3928379340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c6210 0x55b08e9cea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:00.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.334+0000 7f9837fff700 1 --2- 172.21.15.70:0/3928379340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9cf500 0x55b08e9df730 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f9830004400 tx=0x7f9830003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7376e36136ba5d5e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:00.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.334+0000 7f9816ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:00.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.334+0000 7f9816ffd700 1 -- 172.21.15.70:0/3928379340 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9830004a30 con 0x55b08e9cf500 2024-06-04T05:23:00.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.334+0000 7f9816ffd700 1 -- 172.21.15.70:0/3928379340 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f9830004bd0 con 0x55b08e9cf500 2024-06-04T05:23:00.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.334+0000 7f9848767b80 1 -- 172.21.15.70:0/3928379340 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98280a95f0 con 0x55b08e9cf500 2024-06-04T05:23:00.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.335+0000 7f9816ffd700 1 -- 172.21.15.70:0/3928379340 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f9830004dd0 con 0x55b08e9cf500 2024-06-04T05:23:00.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.335+0000 7f9848767b80 10 client.21915 fetch_fsmap learned FSMap version 67 2024-06-04T05:23:00.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.335+0000 7f9848767b80 1 -- 172.21.15.70:0/3928379340 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55b08e9e1310 con 0x55b08e9cf500 2024-06-04T05:23:00.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.335+0000 7f9816ffd700 1 -- 172.21.15.70:0/3928379340 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98280a95f0 con 0x55b08e9cf500 2024-06-04T05:23:00.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.335+0000 7f9816ffd700 1 -- 172.21.15.70:0/3928379340 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f9830019030 con 0x55b08e9cf500 2024-06-04T05:23:00.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.335+0000 7f9848767b80 10 client.21915 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:23:00.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.335+0000 7f9848767b80 10 client.21915 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:23:00.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.335+0000 7f9848767b80 20 client.21915 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:00.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.335+0000 7f9848767b80 1 --2- 172.21.15.70:0/3928379340 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55b08e9e3950 0x55b08e9e5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.335+0000 7f9848767b80 4 client.21915 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:23:00.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.335+0000 7f9848767b80 1 -- 172.21.15.70:0/3928379340 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b08e9919e0 con 0x55b08e9e3950 2024-06-04T05:23:00.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.335+0000 7f98377fe700 1 --2- 172.21.15.70:0/3928379340 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55b08e9e3950 0x55b08e9e5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.336+0000 7f98377fe700 1 --2- 172.21.15.70:0/3928379340 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55b08e9e3950 0x55b08e9e5d40 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7494c565d4060150 server_cookie=c3f309a202320850 in_seq=0 out_seq=0 2024-06-04T05:23:00.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.336+0000 7f9816ffd700 10 client.21915 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:23:00.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.336+0000 7f9816ffd700 1 -- 172.21.15.70:0/3928379340 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b08e9919e0 con 0x55b08e9e3950 2024-06-04T05:23:00.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.336+0000 7f9816ffd700 10 client.21915 handle_command_reply: tid=0 2024-06-04T05:23:00.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.336+0000 7f9848767b80 1 client.21915 shutdown 2024-06-04T05:23:00.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.337+0000 7f9848767b80 1 -- 172.21.15.70:0/3928379340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9cf500 msgr2=0x55b08e9df730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:00.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.337+0000 7f9848767b80 1 --2- 172.21.15.70:0/3928379340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9cf500 0x55b08e9df730 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f9830004400 tx=0x7f9830003960 comp rx=0 tx=0).stop 2024-06-04T05:23:00.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.337+0000 7f9848767b80 1 -- 172.21.15.70:0/3928379340 shutdown_connections 2024-06-04T05:23:00.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.337+0000 7f9848767b80 1 --2- 172.21.15.70:0/3928379340 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55b08e9e3950 0x55b08e9e5d40 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.337+0000 7f9848767b80 1 --2- 172.21.15.70:0/3928379340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9cf500 0x55b08e9df730 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.337+0000 7f9848767b80 1 --2- 172.21.15.70:0/3928379340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c6210 0x55b08e9cea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.337+0000 7f9816ffd700 0 client.21915 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:23:00.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.337+0000 7f9848767b80 1 --2- 172.21.15.70:0/3928379340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9c9f60 0x55b08e9cefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.337+0000 7f9848767b80 1 -- 172.21.15.70:0/3928379340 >> 172.21.15.70:0/3928379340 conn(0x55b08e8f3080 msgr2=0x55b08e9bf510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:00.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.337+0000 7f9848767b80 1 -- 172.21.15.70:0/3928379340 shutdown_connections 2024-06-04T05:23:00.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.337+0000 7f9848767b80 1 -- 172.21.15.70:0/3928379340 wait complete. 2024-06-04T05:23:00.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.337+0000 7f9848767b80 20 client.21915 trim_cache size 0 max 16384 2024-06-04T05:23:00.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.352+0000 7f9848767b80 1 Processor -- start 2024-06-04T05:23:00.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.352+0000 7f9848767b80 1 -- start start 2024-06-04T05:23:00.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.352+0000 7f9848767b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9c6210 0x55b08e9ce380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.352+0000 7f9848767b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c9f60 0x55b08e9d09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.352+0000 7f9848767b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9cf500 0x55b08e9df4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.352+0000 7f9848767b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f98381abc50 con 0x55b08e9cf500 2024-06-04T05:23:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.352+0000 7f9848767b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f98381ad790 con 0x55b08e9c9f60 2024-06-04T05:23:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.352+0000 7f9848767b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f98381ac320 con 0x55b08e9c6210 2024-06-04T05:23:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.352+0000 7f9837fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9cf500 0x55b08e9df4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.352+0000 7f9837fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9cf500 0x55b08e9df4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38406/0 (socket says 172.21.15.70:38406) 2024-06-04T05:23:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.352+0000 7f9837fff700 1 -- 172.21.15.70:0/1322366162 learned_addr learned my addr 172.21.15.70:0/1322366162 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.353+0000 7f98377fe700 1 --2- 172.21.15.70:0/1322366162 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9c6210 0x55b08e9ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.353+0000 7f9836ffd700 1 --2- 172.21.15.70:0/1322366162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c9f60 0x55b08e9d09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.353+0000 7f98377fe700 1 -- 172.21.15.70:0/1322366162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c9f60 msgr2=0x55b08e9d09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.353+0000 7f98377fe700 1 --2- 172.21.15.70:0/1322366162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c9f60 0x55b08e9d09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.353+0000 7f98377fe700 1 -- 172.21.15.70:0/1322366162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9cf500 msgr2=0x55b08e9df4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.353+0000 7f98377fe700 1 --2- 172.21.15.70:0/1322366162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9cf500 0x55b08e9df4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.353+0000 7f98377fe700 1 -- 172.21.15.70:0/1322366162 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b08e9e63c0 con 0x55b08e9c6210 2024-06-04T05:23:00.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.353+0000 7f9837fff700 1 --2- 172.21.15.70:0/1322366162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9cf500 0x55b08e9df4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:23:00.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.353+0000 7f98377fe700 1 --2- 172.21.15.70:0/1322366162 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9c6210 0x55b08e9ce380 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f98280090a0 tx=0x7f98280a55e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c4f000cbbde2074 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:00.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.353+0000 7f982d7fa700 1 -- 172.21.15.70:0/1322366162 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9828004840 con 0x55b08e9c6210 2024-06-04T05:23:00.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.353+0000 7f982d7fa700 1 -- 172.21.15.70:0/1322366162 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98280910e0 con 0x55b08e9c6210 2024-06-04T05:23:00.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.353+0000 7f982d7fa700 1 -- 172.21.15.70:0/1322366162 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98280a9ab0 con 0x55b08e9c6210 2024-06-04T05:23:00.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.354+0000 7f9848767b80 1 -- 172.21.15.70:0/1322366162 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9c6210 msgr2=0x55b08e9ce380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:00.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.354+0000 7f9848767b80 1 --2- 172.21.15.70:0/1322366162 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9c6210 0x55b08e9ce380 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f98280090a0 tx=0x7f98280a55e0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.354+0000 7f9848767b80 1 -- 172.21.15.70:0/1322366162 shutdown_connections 2024-06-04T05:23:00.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.354+0000 7f9848767b80 1 --2- 172.21.15.70:0/1322366162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9cf500 0x55b08e9df4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.354+0000 7f9848767b80 1 --2- 172.21.15.70:0/1322366162 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9c6210 0x55b08e9ce380 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.354+0000 7f9848767b80 1 --2- 172.21.15.70:0/1322366162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c9f60 0x55b08e9d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.354+0000 7f9848767b80 1 -- 172.21.15.70:0/1322366162 >> 172.21.15.70:0/1322366162 conn(0x55b08e8f3080 msgr2=0x55b08e9d02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:00.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.354+0000 7f9848767b80 1 -- 172.21.15.70:0/1322366162 shutdown_connections 2024-06-04T05:23:00.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.354+0000 7f9848767b80 1 -- 172.21.15.70:0/1322366162 wait complete. 2024-06-04T05:23:00.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.354+0000 7f9848767b80 1 -- start start 2024-06-04T05:23:00.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f9848767b80 1 Processor -- start 2024-06-04T05:23:00.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f9848767b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9c6210 0x55b08e9db930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f9848767b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c9f60 0x55b08e9dbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f9848767b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9cf500 0x55b08e9cb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f9848767b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f98381ad0c0 con 0x55b08e9c6210 2024-06-04T05:23:00.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f9848767b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f98381ac9f0 con 0x55b08e9c9f60 2024-06-04T05:23:00.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f9848767b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f98381a9c10 con 0x55b08e9cf500 2024-06-04T05:23:00.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f9848767b80 10 client.0 _ll_register_callbacks cb 0x55b08e8af920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:00.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f9836ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c9f60 0x55b08e9dbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f9836ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c9f60 0x55b08e9dbe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59192/0 (socket says 172.21.15.70:59192) 2024-06-04T05:23:00.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f98377fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9c6210 0x55b08e9db930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f9836ffd700 1 -- 172.21.15.70:0/733937517 learned_addr learned my addr 172.21.15.70:0/733937517 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:00.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f9837fff700 1 --2- 172.21.15.70:0/733937517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9cf500 0x55b08e9cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f9836ffd700 1 -- 172.21.15.70:0/733937517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9cf500 msgr2=0x55b08e9cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:00.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f9836ffd700 1 --2- 172.21.15.70:0/733937517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9cf500 0x55b08e9cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f9836ffd700 1 -- 172.21.15.70:0/733937517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9c6210 msgr2=0x55b08e9db930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:00.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f9836ffd700 1 --2- 172.21.15.70:0/733937517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9c6210 0x55b08e9db930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.355+0000 7f9836ffd700 1 -- 172.21.15.70:0/733937517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b08e9cbba0 con 0x55b08e9c9f60 2024-06-04T05:23:00.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.356+0000 7f9836ffd700 1 --2- 172.21.15.70:0/733937517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c9f60 0x55b08e9dbe70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f982000b650 tx=0x7f9820009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11828b465d01a97e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:00.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.356+0000 7f9816ffd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:23:00.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.356+0000 7f9816ffd700 1 -- 172.21.15.70:0/733937517 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9820036060 con 0x55b08e9c9f60 2024-06-04T05:23:00.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.356+0000 7f9848767b80 1 -- 172.21.15.70:0/733937517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98280910e0 con 0x55b08e9c9f60 2024-06-04T05:23:00.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.356+0000 7f9816ffd700 1 -- 172.21.15.70:0/733937517 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f9820031460 con 0x55b08e9c9f60 2024-06-04T05:23:00.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.356+0000 7f9816ffd700 1 -- 172.21.15.70:0/733937517 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f98200316a0 con 0x55b08e9c9f60 2024-06-04T05:23:00.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.356+0000 7f9848767b80 10 client.30503 fetch_fsmap learned FSMap version 67 2024-06-04T05:23:00.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.356+0000 7f9848767b80 1 -- 172.21.15.70:0/733937517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55b08e9db240 con 0x55b08e9c9f60 2024-06-04T05:23:00.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.356+0000 7f9816ffd700 1 -- 172.21.15.70:0/733937517 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98280910e0 con 0x55b08e9c9f60 2024-06-04T05:23:00.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.356+0000 7f9816ffd700 1 -- 172.21.15.70:0/733937517 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f9820039050 con 0x55b08e9c9f60 2024-06-04T05:23:00.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.356+0000 7f9848767b80 10 client.30503 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:23:00.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.356+0000 7f9848767b80 10 client.30503 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:23:00.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.357+0000 7f9848767b80 20 client.30503 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:00.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.357+0000 7f9848767b80 1 --2- 172.21.15.70:0/733937517 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55b08ea4e760 0x55b08ea50b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:00.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.357+0000 7f9848767b80 4 client.30503 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:23:00.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.357+0000 7f9848767b80 1 -- 172.21.15.70:0/733937517 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b08e926be0 con 0x55b08ea4e760 2024-06-04T05:23:00.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.357+0000 7f98377fe700 1 --2- 172.21.15.70:0/733937517 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55b08ea4e760 0x55b08ea50b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:00.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.357+0000 7f98377fe700 1 --2- 172.21.15.70:0/733937517 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55b08ea4e760 0x55b08ea50b50 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dfd5e68828cff30c server_cookie=bdefd7da22cc04ae in_seq=0 out_seq=0 2024-06-04T05:23:00.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.357+0000 7f9816ffd700 10 client.30503 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:23:00.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.357+0000 7f9816ffd700 1 -- 172.21.15.70:0/733937517 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b08e926be0 con 0x55b08ea4e760 2024-06-04T05:23:00.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.357+0000 7f9816ffd700 10 client.30503 handle_command_reply: tid=0 2024-06-04T05:23:00.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.357+0000 7f9848767b80 1 client.30503 shutdown 2024-06-04T05:23:00.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.357+0000 7f9848767b80 1 -- 172.21.15.70:0/733937517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c9f60 msgr2=0x55b08e9dbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:00.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.357+0000 7f9848767b80 1 --2- 172.21.15.70:0/733937517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c9f60 0x55b08e9dbe70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f982000b650 tx=0x7f9820009ca0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.358+0000 7f9848767b80 1 -- 172.21.15.70:0/733937517 shutdown_connections 2024-06-04T05:23:00.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.358+0000 7f9848767b80 1 --2- 172.21.15.70:0/733937517 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55b08ea4e760 0x55b08ea50b50 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.358+0000 7f9848767b80 1 --2- 172.21.15.70:0/733937517 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b08e9cf500 0x55b08e9cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.358+0000 7f9848767b80 1 --2- 172.21.15.70:0/733937517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b08e9c6210 0x55b08e9db930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.358+0000 7f9848767b80 1 --2- 172.21.15.70:0/733937517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b08e9c9f60 0x55b08e9dbe70 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.358+0000 7f9848767b80 1 -- 172.21.15.70:0/733937517 >> 172.21.15.70:0/733937517 conn(0x55b08e8f3080 msgr2=0x55b08e9c4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:00.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.358+0000 7f9816ffd700 0 client.30503 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:23:00.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.358+0000 7f9848767b80 1 -- 172.21.15.70:0/733937517 shutdown_connections 2024-06-04T05:23:00.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.358+0000 7f9848767b80 1 -- 172.21.15.70:0/733937517 wait complete. 2024-06-04T05:23:00.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.358+0000 7f9848767b80 20 client.30503 trim_cache size 0 max 16384 2024-06-04T05:23:00.366 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:23:00.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.358+0000 7f983ded2700 1 -- 172.21.15.70:0/662961532 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f98240b2a70 msgr2=0x7f98240b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:00.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.358+0000 7f983ded2700 1 --2- 172.21.15.70:0/662961532 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f98240b2a70 0x7f98240b4f30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f98381026a0 tx=0x7f9820005960 comp rx=0 tx=0).stop 2024-06-04T05:23:00.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.358+0000 7f983ded2700 1 -- 172.21.15.70:0/662961532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838102c40 msgr2=0x7f983819a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.358+0000 7f983ded2700 1 --2- 172.21.15.70:0/662961532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838102c40 0x7f983819a920 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f982800ef90 tx=0x7f982800c3a0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.359+0000 7f983ded2700 1 -- 172.21.15.70:0/662961532 shutdown_connections 2024-06-04T05:23:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.359+0000 7f983ded2700 1 --2- 172.21.15.70:0/662961532 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f98240b2a70 0x7f98240b4f30 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.359+0000 7f983ded2700 1 --2- 172.21.15.70:0/662961532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98381084c0 0x7f98381a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.359+0000 7f983ded2700 1 --2- 172.21.15.70:0/662961532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838102c40 0x7f983819a920 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.359+0000 7f983ded2700 1 --2- 172.21.15.70:0/662961532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9838103600 0x7f983819ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.359+0000 7f983ded2700 1 -- 172.21.15.70:0/662961532 >> 172.21.15.70:0/662961532 conn(0x7f9838078ab0 msgr2=0x7f98381002e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.359+0000 7f983ded2700 1 -- 172.21.15.70:0/662961532 shutdown_connections 2024-06-04T05:23:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:00.359+0000 7f983ded2700 1 -- 172.21.15.70:0/662961532 wait complete. 2024-06-04T05:23:00.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 88998 6.59 MB/sec execute 592 sec latency 568.332 ms 2024-06-04T05:23:01.084 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:23:01.330 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:01.734 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:01 smithi070 ceph-mon[32631]: pgmap v814: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.5 MiB/s wr, 254 op/s 2024-06-04T05:23:01.737 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:01 smithi112 ceph-mon[36453]: pgmap v814: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.5 MiB/s wr, 254 op/s 2024-06-04T05:23:01.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:01 smithi098 ceph-mon[35569]: pgmap v814: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.5 MiB/s wr, 254 op/s 2024-06-04T05:23:01.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 88998 6.58 MB/sec execute 593 sec latency 1568.390 ms 2024-06-04T05:23:02.838 INFO:tasks.workunit.client.0.smithi070.stdout: 10 88998 6.57 MB/sec execute 594 sec latency 2568.470 ms 2024-06-04T05:23:03.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.127+0000 7f4928bd6700 1 -- 172.21.15.70:0/1530399052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49241084b0 msgr2=0x7f4924108890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:03.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.127+0000 7f4928bd6700 1 --2- 172.21.15.70:0/1530399052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49241084b0 0x7f4924108890 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f4914009990 tx=0x7f4914009ca0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.127+0000 7f4928bd6700 1 -- 172.21.15.70:0/1530399052 shutdown_connections 2024-06-04T05:23:03.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.127+0000 7f4928bd6700 1 --2- 172.21.15.70:0/1530399052 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4924102e30 0x7f492410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.127+0000 7f4928bd6700 1 --2- 172.21.15.70:0/1530399052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49241084b0 0x7f4924108890 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.127+0000 7f4928bd6700 1 --2- 172.21.15.70:0/1530399052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4924102470 0x7f49241028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.127+0000 7f4928bd6700 1 -- 172.21.15.70:0/1530399052 >> 172.21.15.70:0/1530399052 conn(0x7f49240793f0 msgr2=0x7f4924079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:03.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.127+0000 7f4928bd6700 1 -- 172.21.15.70:0/1530399052 shutdown_connections 2024-06-04T05:23:03.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.127+0000 7f4928bd6700 1 -- 172.21.15.70:0/1530399052 wait complete. 2024-06-04T05:23:03.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.127+0000 7f4928bd6700 1 Processor -- start 2024-06-04T05:23:03.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f4928bd6700 1 -- start start 2024-06-04T05:23:03.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f4928bd6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4924102470 0x7f492419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f4928bd6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4924102e30 0x7f492419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f4928bd6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49241084b0 0x7f49241987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f4928bd6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4924112470 con 0x7f4924102e30 2024-06-04T05:23:03.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f4928bd6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f49241122f0 con 0x7f4924102470 2024-06-04T05:23:03.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f4928bd6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f49241125f0 con 0x7f49241084b0 2024-06-04T05:23:03.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f4921d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4924102e30 0x7f492419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:03.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f4921d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4924102e30 0x7f492419eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38432/0 (socket says 172.21.15.70:38432) 2024-06-04T05:23:03.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f4921d9b700 1 -- 172.21.15.70:0/2614304053 learned_addr learned my addr 172.21.15.70:0/2614304053 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:03.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f492259c700 1 --2- 172.21.15.70:0/2614304053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4924102470 0x7f492419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:03.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f4922d9d700 1 --2- 172.21.15.70:0/2614304053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49241084b0 0x7f49241987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:03.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f4921d9b700 1 -- 172.21.15.70:0/2614304053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49241084b0 msgr2=0x7f49241987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f4921d9b700 1 --2- 172.21.15.70:0/2614304053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49241084b0 0x7f49241987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f4921d9b700 1 -- 172.21.15.70:0/2614304053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4924102470 msgr2=0x7f492419e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f4921d9b700 1 --2- 172.21.15.70:0/2614304053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4924102470 0x7f492419e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f4921d9b700 1 -- 172.21.15.70:0/2614304053 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4924198fd0 con 0x7f4924102e30 2024-06-04T05:23:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.128+0000 7f4921d9b700 1 --2- 172.21.15.70:0/2614304053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4924102e30 0x7f492419eb60 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f491800c9b0 tx=0x7f491800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.129+0000 7f49137fe700 1 -- 172.21.15.70:0/2614304053 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4918007b00 con 0x7f4924102e30 2024-06-04T05:23:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.129+0000 7f49137fe700 1 -- 172.21.15.70:0/2614304053 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f491800ce50 con 0x7f4924102e30 2024-06-04T05:23:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.129+0000 7f4928bd6700 1 -- 172.21.15.70:0/2614304053 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49241992c0 con 0x7f4924102e30 2024-06-04T05:23:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.129+0000 7f49137fe700 1 -- 172.21.15.70:0/2614304053 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4918018660 con 0x7f4924102e30 2024-06-04T05:23:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.129+0000 7f4928bd6700 1 -- 172.21.15.70:0/2614304053 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49241a5480 con 0x7f4924102e30 2024-06-04T05:23:03.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.131+0000 7f49137fe700 1 -- 172.21.15.70:0/2614304053 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f491800f450 con 0x7f4924102e30 2024-06-04T05:23:03.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.131+0000 7f49137fe700 1 --2- 172.21.15.70:0/2614304053 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f490c0b2a70 0x7f490c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.132+0000 7f49137fe700 1 -- 172.21.15.70:0/2614304053 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f49180e2510 con 0x7f4924102e30 2024-06-04T05:23:03.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.132+0000 7f492259c700 1 --2- 172.21.15.70:0/2614304053 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f490c0b2a70 0x7f490c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:03.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.132+0000 7f492259c700 1 --2- 172.21.15.70:0/2614304053 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f490c0b2a70 0x7f490c0b4f30 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f491400b5c0 tx=0x7f4914005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:03.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.132+0000 7f493346bb80 1 Processor -- start 2024-06-04T05:23:03.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f493346bb80 1 -- start start 2024-06-04T05:23:03.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f493346bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568fb470 0x5638568fb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f493346bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fbe20 0x5638568f5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f493346bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x5638568f6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f493346bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563856851d70 con 0x5638568f60d0 2024-06-04T05:23:03.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f493346bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563856841910 con 0x5638568fb470 2024-06-04T05:23:03.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f493346bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563856845950 con 0x5638568fbe20 2024-06-04T05:23:03.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f4922d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x5638568f6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:03.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f4922d9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x5638568f6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38440/0 (socket says 172.21.15.70:38440) 2024-06-04T05:23:03.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f492259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568fb470 0x5638568fb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:03.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f4922d9d700 1 -- 172.21.15.70:0/180312 learned_addr learned my addr 172.21.15.70:0/180312 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:03.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f4921d9b700 1 --2- 172.21.15.70:0/180312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fbe20 0x5638568f5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:03.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f4922d9d700 1 -- 172.21.15.70:0/180312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fbe20 msgr2=0x5638568f5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:03.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f4922d9d700 1 --2- 172.21.15.70:0/180312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fbe20 0x5638568f5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f4922d9d700 1 -- 172.21.15.70:0/180312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568fb470 msgr2=0x5638568fb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:03.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f4922d9d700 1 --2- 172.21.15.70:0/180312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568fb470 0x5638568fb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f4922d9d700 1 -- 172.21.15.70:0/180312 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638567eafb0 con 0x5638568f60d0 2024-06-04T05:23:03.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f4921d9b700 1 --2- 172.21.15.70:0/180312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fbe20 0x5638568f5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:03.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f492259c700 1 --2- 172.21.15.70:0/180312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568fb470 0x5638568fb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:23:03.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.133+0000 7f4922d9d700 1 --2- 172.21.15.70:0/180312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x5638568f6530 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f491c00a730 tx=0x7f491c0078d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3277366f975181cb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:03.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.134+0000 7f4910ff9700 1 -- 172.21.15.70:0/180312 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491c003a00 con 0x5638568f60d0 2024-06-04T05:23:03.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.134+0000 7f4910ff9700 1 -- 172.21.15.70:0/180312 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f491c003ba0 con 0x5638568f60d0 2024-06-04T05:23:03.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.134+0000 7f4910ff9700 1 -- 172.21.15.70:0/180312 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491c014400 con 0x5638568f60d0 2024-06-04T05:23:03.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.134+0000 7f493346bb80 1 -- 172.21.15.70:0/180312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 msgr2=0x5638568f6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:03.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.134+0000 7f493346bb80 1 --2- 172.21.15.70:0/180312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x5638568f6530 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f491c00a730 tx=0x7f491c0078d0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.134+0000 7f493346bb80 1 -- 172.21.15.70:0/180312 shutdown_connections 2024-06-04T05:23:03.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.134+0000 7f493346bb80 1 --2- 172.21.15.70:0/180312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x5638568f6530 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.134+0000 7f493346bb80 1 --2- 172.21.15.70:0/180312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568fb470 0x5638568fb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.134+0000 7f493346bb80 1 --2- 172.21.15.70:0/180312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fbe20 0x5638568f5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.134+0000 7f493346bb80 1 -- 172.21.15.70:0/180312 >> 172.21.15.70:0/180312 conn(0x56385681f240 msgr2=0x5638568ee3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:03.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.135+0000 7f493346bb80 1 -- 172.21.15.70:0/180312 shutdown_connections 2024-06-04T05:23:03.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.135+0000 7f493346bb80 1 -- 172.21.15.70:0/180312 wait complete. 2024-06-04T05:23:03.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.135+0000 7f493346bb80 1 -- start start 2024-06-04T05:23:03.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.135+0000 7f493346bb80 1 Processor -- start 2024-06-04T05:23:03.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.135+0000 7f493346bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x563856901bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.135+0000 7f493346bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fb470 0x5638568fef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.135+0000 7f493346bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568fbe20 0x5638568ff450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.135+0000 7f493346bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563856747fa0 con 0x5638568f60d0 2024-06-04T05:23:03.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.135+0000 7f493346bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5638567a9da0 con 0x5638568fbe20 2024-06-04T05:23:03.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.135+0000 7f493346bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563856902840 con 0x5638568fb470 2024-06-04T05:23:03.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.135+0000 7f493346bb80 10 client.0 _ll_register_callbacks cb 0x5638567dc950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:03.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.135+0000 7f492259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x563856901bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:03.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.135+0000 7f492259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x563856901bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38450/0 (socket says 172.21.15.70:38450) 2024-06-04T05:23:03.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.135+0000 7f492259c700 1 -- 172.21.15.70:0/2423180005 learned_addr learned my addr 172.21.15.70:0/2423180005 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:03.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.136+0000 7f4921d9b700 1 --2- 172.21.15.70:0/2423180005 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fb470 0x5638568fef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:03.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.136+0000 7f492259c700 1 -- 172.21.15.70:0/2423180005 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fb470 msgr2=0x5638568fef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:03.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.136+0000 7f492259c700 1 --2- 172.21.15.70:0/2423180005 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fb470 0x5638568fef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.136+0000 7f492259c700 1 -- 172.21.15.70:0/2423180005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568fbe20 msgr2=0x5638568ff450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:03.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.136+0000 7f492259c700 1 --2- 172.21.15.70:0/2423180005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568fbe20 0x5638568ff450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.136+0000 7f492259c700 1 -- 172.21.15.70:0/2423180005 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638567918d0 con 0x5638568f60d0 2024-06-04T05:23:03.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.136+0000 7f4921d9b700 1 --2- 172.21.15.70:0/2423180005 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fb470 0x5638568fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:23:03.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.136+0000 7f492259c700 1 --2- 172.21.15.70:0/2423180005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x563856901bc0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f4914015000 tx=0x7f4914009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74853d33dd0e2f4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:03.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.136+0000 7f4909ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:03.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.136+0000 7f4909ffb700 1 -- 172.21.15.70:0/2423180005 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491400fd70 con 0x5638568f60d0 2024-06-04T05:23:03.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.136+0000 7f4909ffb700 1 -- 172.21.15.70:0/2423180005 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f4914030460 con 0x5638568f60d0 2024-06-04T05:23:03.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.136+0000 7f493346bb80 1 -- 172.21.15.70:0/2423180005 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f491c003ba0 con 0x5638568f60d0 2024-06-04T05:23:03.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.136+0000 7f4909ffb700 1 -- 172.21.15.70:0/2423180005 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f491403e050 con 0x5638568f60d0 2024-06-04T05:23:03.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.136+0000 7f493346bb80 10 client.21933 fetch_fsmap learned FSMap version 67 2024-06-04T05:23:03.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.136+0000 7f493346bb80 1 -- 172.21.15.70:0/2423180005 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5638567fb760 con 0x5638568f60d0 2024-06-04T05:23:03.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.136+0000 7f4909ffb700 1 -- 172.21.15.70:0/2423180005 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f491c003ba0 con 0x5638568f60d0 2024-06-04T05:23:03.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.137+0000 7f4909ffb700 1 -- 172.21.15.70:0/2423180005 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f49140244b0 con 0x5638568f60d0 2024-06-04T05:23:03.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.137+0000 7f493346bb80 10 client.21933 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:23:03.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.137+0000 7f493346bb80 10 client.21933 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:23:03.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.137+0000 7f493346bb80 20 client.21933 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:03.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.137+0000 7f493346bb80 1 --2- 172.21.15.70:0/2423180005 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x56385690f460 0x5638569118d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.137+0000 7f493346bb80 4 client.21933 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:23:03.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.137+0000 7f493346bb80 1 -- 172.21.15.70:0/2423180005 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563856853cc0 con 0x56385690f460 2024-06-04T05:23:03.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.137+0000 7f4921d9b700 1 --2- 172.21.15.70:0/2423180005 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x56385690f460 0x5638569118d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:03.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.137+0000 7f4921d9b700 1 --2- 172.21.15.70:0/2423180005 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x56385690f460 0x5638569118d0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6800851bd4453f5f server_cookie=b00bac05dc698d81 in_seq=0 out_seq=0 2024-06-04T05:23:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.137+0000 7f4909ffb700 10 client.21933 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:23:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.138+0000 7f4909ffb700 1 -- 172.21.15.70:0/2423180005 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563856853cc0 con 0x56385690f460 2024-06-04T05:23:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.138+0000 7f4909ffb700 10 client.21933 handle_command_reply: tid=0 2024-06-04T05:23:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.138+0000 7f493346bb80 1 client.21933 shutdown 2024-06-04T05:23:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.138+0000 7f493346bb80 1 -- 172.21.15.70:0/2423180005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 msgr2=0x563856901bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.138+0000 7f493346bb80 1 --2- 172.21.15.70:0/2423180005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x563856901bc0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f4914015000 tx=0x7f4914009310 comp rx=0 tx=0).stop 2024-06-04T05:23:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.138+0000 7f493346bb80 1 -- 172.21.15.70:0/2423180005 shutdown_connections 2024-06-04T05:23:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.138+0000 7f493346bb80 1 --2- 172.21.15.70:0/2423180005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568fbe20 0x5638568ff450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.138+0000 7f493346bb80 1 --2- 172.21.15.70:0/2423180005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x563856901bc0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.138+0000 7f493346bb80 1 --2- 172.21.15.70:0/2423180005 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x56385690f460 0x5638569118d0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.138+0000 7f493346bb80 1 --2- 172.21.15.70:0/2423180005 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fb470 0x5638568fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.138+0000 7f4909ffb700 0 client.21933 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:23:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.138+0000 7f493346bb80 1 -- 172.21.15.70:0/2423180005 >> 172.21.15.70:0/2423180005 conn(0x56385681f240 msgr2=0x5638568edaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:03.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.138+0000 7f493346bb80 1 -- 172.21.15.70:0/2423180005 shutdown_connections 2024-06-04T05:23:03.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.138+0000 7f493346bb80 1 -- 172.21.15.70:0/2423180005 wait complete. 2024-06-04T05:23:03.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.138+0000 7f493346bb80 20 client.21933 trim_cache size 0 max 16384 2024-06-04T05:23:03.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.153+0000 7f493346bb80 1 Processor -- start 2024-06-04T05:23:03.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.153+0000 7f493346bb80 1 -- start start 2024-06-04T05:23:03.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f493346bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568f60d0 0x5638568fc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f493346bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568fb470 0x5638568fe090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f493346bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fbe20 0x5638568ee270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f493346bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f49241abb60 con 0x5638568fb470 2024-06-04T05:23:03.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f493346bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f49241ad6a0 con 0x5638568f60d0 2024-06-04T05:23:03.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f493346bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f49241ac230 con 0x5638568fbe20 2024-06-04T05:23:03.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f4921d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568fb470 0x5638568fe090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:03.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f4921d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568fb470 0x5638568fe090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38456/0 (socket says 172.21.15.70:38456) 2024-06-04T05:23:03.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f4921d9b700 1 -- 172.21.15.70:0/4087470091 learned_addr learned my addr 172.21.15.70:0/4087470091 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:03.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f4922d9d700 1 --2- 172.21.15.70:0/4087470091 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fbe20 0x5638568ee270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:03.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f492259c700 1 --2- 172.21.15.70:0/4087470091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568f60d0 0x5638568fc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:03.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f4921d9b700 1 -- 172.21.15.70:0/4087470091 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fbe20 msgr2=0x5638568ee270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:03.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f4921d9b700 1 --2- 172.21.15.70:0/4087470091 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fbe20 0x5638568ee270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f4921d9b700 1 -- 172.21.15.70:0/4087470091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568f60d0 msgr2=0x5638568fc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:03.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f4921d9b700 1 --2- 172.21.15.70:0/4087470091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568f60d0 0x5638568fc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f4921d9b700 1 -- 172.21.15.70:0/4087470091 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638568f0380 con 0x5638568fb470 2024-06-04T05:23:03.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f4921d9b700 1 --2- 172.21.15.70:0/4087470091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568fb470 0x5638568fe090 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f49180b20f0 tx=0x7f49180b2040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3a2943bfedc4262 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:03.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.154+0000 7f490bfff700 1 -- 172.21.15.70:0/4087470091 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49180c2070 con 0x5638568fb470 2024-06-04T05:23:03.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.155+0000 7f490bfff700 1 -- 172.21.15.70:0/4087470091 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f49180a9ec0 con 0x5638568fb470 2024-06-04T05:23:03.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.155+0000 7f490bfff700 1 -- 172.21.15.70:0/4087470091 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49180a5a50 con 0x5638568fb470 2024-06-04T05:23:03.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.155+0000 7f493346bb80 1 -- 172.21.15.70:0/4087470091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568fb470 msgr2=0x5638568fe090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:03.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.155+0000 7f493346bb80 1 --2- 172.21.15.70:0/4087470091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568fb470 0x5638568fe090 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f49180b20f0 tx=0x7f49180b2040 comp rx=0 tx=0).stop 2024-06-04T05:23:03.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.155+0000 7f493346bb80 1 -- 172.21.15.70:0/4087470091 shutdown_connections 2024-06-04T05:23:03.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.155+0000 7f493346bb80 1 --2- 172.21.15.70:0/4087470091 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fbe20 0x5638568ee270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.155+0000 7f493346bb80 1 --2- 172.21.15.70:0/4087470091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568f60d0 0x5638568fc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.155+0000 7f493346bb80 1 --2- 172.21.15.70:0/4087470091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568fb470 0x5638568fe090 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.155+0000 7f493346bb80 1 -- 172.21.15.70:0/4087470091 >> 172.21.15.70:0/4087470091 conn(0x56385681f240 msgr2=0x5638568f2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:03.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.155+0000 7f493346bb80 1 -- 172.21.15.70:0/4087470091 shutdown_connections 2024-06-04T05:23:03.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.155+0000 7f493346bb80 1 -- 172.21.15.70:0/4087470091 wait complete. 2024-06-04T05:23:03.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.156+0000 7f493346bb80 1 -- start start 2024-06-04T05:23:03.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.156+0000 7f493346bb80 1 Processor -- start 2024-06-04T05:23:03.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.156+0000 7f493346bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x563856907e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.156+0000 7f493346bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fb470 0x5638569031e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.156+0000 7f493346bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568fbe20 0x563856903720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.156+0000 7f493346bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f49241acfd0 con 0x5638568f60d0 2024-06-04T05:23:03.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.156+0000 7f493346bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f49241ac900 con 0x5638568fbe20 2024-06-04T05:23:03.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.156+0000 7f493346bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f49241a9b20 con 0x5638568fb470 2024-06-04T05:23:03.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.156+0000 7f493346bb80 10 client.0 _ll_register_callbacks cb 0x5638567dc950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:03.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.156+0000 7f492259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x563856907e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:03.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.156+0000 7f492259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x563856907e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38472/0 (socket says 172.21.15.70:38472) 2024-06-04T05:23:03.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.156+0000 7f492259c700 1 -- 172.21.15.70:0/1682294735 learned_addr learned my addr 172.21.15.70:0/1682294735 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:03.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.156+0000 7f4921d9b700 1 --2- 172.21.15.70:0/1682294735 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fb470 0x5638569031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:03.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.157+0000 7f492259c700 1 -- 172.21.15.70:0/1682294735 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fb470 msgr2=0x5638569031e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:03.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.157+0000 7f492259c700 1 --2- 172.21.15.70:0/1682294735 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fb470 0x5638569031e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.157+0000 7f492259c700 1 -- 172.21.15.70:0/1682294735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568fbe20 msgr2=0x563856903720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:03.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.157+0000 7f492259c700 1 --2- 172.21.15.70:0/1682294735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568fbe20 0x563856903720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.157+0000 7f492259c700 1 -- 172.21.15.70:0/1682294735 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563856903c60 con 0x5638568f60d0 2024-06-04T05:23:03.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.157+0000 7f492259c700 1 --2- 172.21.15.70:0/1682294735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x563856907e60 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f4914009ce0 tx=0x7f4914024dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c311cc45feb036d1 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:03.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.157+0000 7f4909ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:03.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.157+0000 7f4909ffb700 1 -- 172.21.15.70:0/1682294735 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4914046070 con 0x5638568f60d0 2024-06-04T05:23:03.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.157+0000 7f4909ffb700 1 -- 172.21.15.70:0/1682294735 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f491400fd70 con 0x5638568f60d0 2024-06-04T05:23:03.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.157+0000 7f493346bb80 1 -- 172.21.15.70:0/1682294735 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49180a9ec0 con 0x5638568f60d0 2024-06-04T05:23:03.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.157+0000 7f4909ffb700 1 -- 172.21.15.70:0/1682294735 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f4914052050 con 0x5638568f60d0 2024-06-04T05:23:03.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.157+0000 7f493346bb80 10 client.21939 fetch_fsmap learned FSMap version 67 2024-06-04T05:23:03.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.157+0000 7f493346bb80 1 -- 172.21.15.70:0/1682294735 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x563856979d60 con 0x5638568f60d0 2024-06-04T05:23:03.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.157+0000 7f4909ffb700 1 -- 172.21.15.70:0/1682294735 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f49180a9ec0 con 0x5638568f60d0 2024-06-04T05:23:03.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.158+0000 7f4909ffb700 1 -- 172.21.15.70:0/1682294735 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f4914042430 con 0x5638568f60d0 2024-06-04T05:23:03.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.158+0000 7f493346bb80 10 client.21939 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:23:03.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.158+0000 7f493346bb80 10 client.21939 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:23:03.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.158+0000 7f493346bb80 20 client.21939 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:03.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.158+0000 7f493346bb80 1 --2- 172.21.15.70:0/1682294735 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x56385697a3b0 0x56385697c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:03.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.158+0000 7f493346bb80 4 client.21939 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:23:03.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.158+0000 7f493346bb80 1 -- 172.21.15.70:0/1682294735 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56385684d4a0 con 0x56385697a3b0 2024-06-04T05:23:03.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.158+0000 7f4921d9b700 1 --2- 172.21.15.70:0/1682294735 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x56385697a3b0 0x56385697c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:03.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.158+0000 7f4921d9b700 1 --2- 172.21.15.70:0/1682294735 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x56385697a3b0 0x56385697c810 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2696a5bcbe5c08de server_cookie=d1a6ebe604fa4a06 in_seq=0 out_seq=0 2024-06-04T05:23:03.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.158+0000 7f4909ffb700 10 client.21939 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:23:03.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.158+0000 7f4909ffb700 1 -- 172.21.15.70:0/1682294735 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56385684d4a0 con 0x56385697a3b0 2024-06-04T05:23:03.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.159+0000 7f4909ffb700 10 client.21939 handle_command_reply: tid=0 2024-06-04T05:23:03.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.159+0000 7f493346bb80 1 client.21939 shutdown 2024-06-04T05:23:03.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.159+0000 7f493346bb80 1 -- 172.21.15.70:0/1682294735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 msgr2=0x563856907e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:03.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.159+0000 7f493346bb80 1 --2- 172.21.15.70:0/1682294735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x563856907e60 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f4914009ce0 tx=0x7f4914024dc0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.159+0000 7f493346bb80 1 -- 172.21.15.70:0/1682294735 shutdown_connections 2024-06-04T05:23:03.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.159+0000 7f493346bb80 1 --2- 172.21.15.70:0/1682294735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638568fbe20 0x563856903720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.159+0000 7f493346bb80 1 --2- 172.21.15.70:0/1682294735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638568f60d0 0x563856907e60 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.159+0000 7f493346bb80 1 --2- 172.21.15.70:0/1682294735 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x56385697a3b0 0x56385697c810 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.159+0000 7f493346bb80 1 --2- 172.21.15.70:0/1682294735 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5638568fb470 0x5638569031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.159+0000 7f493346bb80 1 -- 172.21.15.70:0/1682294735 >> 172.21.15.70:0/1682294735 conn(0x56385681f240 msgr2=0x5638568fa020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:03.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.159+0000 7f4909ffb700 0 client.21939 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:23:03.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.159+0000 7f493346bb80 1 -- 172.21.15.70:0/1682294735 shutdown_connections 2024-06-04T05:23:03.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.159+0000 7f493346bb80 1 -- 172.21.15.70:0/1682294735 wait complete. 2024-06-04T05:23:03.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.159+0000 7f493346bb80 20 client.21939 trim_cache size 0 max 16384 2024-06-04T05:23:03.169 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:23:03.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.160+0000 7f4928bd6700 1 -- 172.21.15.70:0/2614304053 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f490c0b2a70 msgr2=0x7f490c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:03.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.160+0000 7f4928bd6700 1 --2- 172.21.15.70:0/2614304053 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f490c0b2a70 0x7f490c0b4f30 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f491400b5c0 tx=0x7f4914005960 comp rx=0 tx=0).stop 2024-06-04T05:23:03.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.160+0000 7f4928bd6700 1 -- 172.21.15.70:0/2614304053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4924102e30 msgr2=0x7f492419eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:03.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.160+0000 7f4928bd6700 1 --2- 172.21.15.70:0/2614304053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4924102e30 0x7f492419eb60 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f491800c9b0 tx=0x7f491800ccc0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.160+0000 7f4928bd6700 1 -- 172.21.15.70:0/2614304053 shutdown_connections 2024-06-04T05:23:03.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.160+0000 7f4928bd6700 1 --2- 172.21.15.70:0/2614304053 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f490c0b2a70 0x7f490c0b4f30 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.160+0000 7f4928bd6700 1 --2- 172.21.15.70:0/2614304053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49241084b0 0x7f49241987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.160+0000 7f4928bd6700 1 --2- 172.21.15.70:0/2614304053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4924102470 0x7f492419e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.160+0000 7f4928bd6700 1 --2- 172.21.15.70:0/2614304053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4924102e30 0x7f492419eb60 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:03.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.160+0000 7f4928bd6700 1 -- 172.21.15.70:0/2614304053 >> 172.21.15.70:0/2614304053 conn(0x7f49240793f0 msgr2=0x7f4924100560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:03.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.160+0000 7f4928bd6700 1 -- 172.21.15.70:0/2614304053 shutdown_connections 2024-06-04T05:23:03.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:03.160+0000 7f4928bd6700 1 -- 172.21.15.70:0/2614304053 wait complete. 2024-06-04T05:23:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:03 smithi098 ceph-mon[35569]: pgmap v815: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 7.4 MiB/s wr, 398 op/s 2024-06-04T05:23:03.776 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:03 smithi070 ceph-mon[32631]: pgmap v815: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 7.4 MiB/s wr, 398 op/s 2024-06-04T05:23:03.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 88998 6.56 MB/sec execute 595 sec latency 3568.520 ms 2024-06-04T05:23:03.918 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:23:03.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:03 smithi112 ceph-mon[36453]: pgmap v815: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 7.4 MiB/s wr, 398 op/s 2024-06-04T05:23:04.159 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:04.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 88998 6.55 MB/sec execute 596 sec latency 4568.615 ms 2024-06-04T05:23:05.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:05 smithi070 ceph-mon[32631]: pgmap v816: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.4 MiB/s wr, 269 op/s 2024-06-04T05:23:05.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:05 smithi098 ceph-mon[35569]: pgmap v816: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.4 MiB/s wr, 269 op/s 2024-06-04T05:23:05.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 89127 6.54 MB/sec execute 597 sec latency 4790.703 ms 2024-06-04T05:23:05.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.913+0000 7f8051c07700 1 -- 172.21.15.70:0/2035119184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f804c106110 msgr2=0x7f804c112620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:05.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.913+0000 7f8051c07700 1 --2- 172.21.15.70:0/2035119184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f804c106110 0x7f804c112620 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8044009f90 tx=0x7f804400b330 comp rx=0 tx=0).stop 2024-06-04T05:23:05.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.913+0000 7f8051c07700 1 -- 172.21.15.70:0/2035119184 shutdown_connections 2024-06-04T05:23:05.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.913+0000 7f8051c07700 1 --2- 172.21.15.70:0/2035119184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f804c106110 0x7f804c112620 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.913+0000 7f8051c07700 1 --2- 172.21.15.70:0/2035119184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f804c10b750 0x7f804c10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.913+0000 7f8051c07700 1 --2- 172.21.15.70:0/2035119184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f804c105750 0x7f804c105bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.913+0000 7f8051c07700 1 -- 172.21.15.70:0/2035119184 >> 172.21.15.70:0/2035119184 conn(0x7f804c101220 msgr2=0x7f804c103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:05.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.913+0000 7f8051c07700 1 -- 172.21.15.70:0/2035119184 shutdown_connections 2024-06-04T05:23:05.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.913+0000 7f8051c07700 1 -- 172.21.15.70:0/2035119184 wait complete. 2024-06-04T05:23:05.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.914+0000 7f8051c07700 1 Processor -- start 2024-06-04T05:23:05.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.914+0000 7f8051c07700 1 -- start start 2024-06-04T05:23:05.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.915+0000 7f8051c07700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f804c105750 0x7f804c1a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.915+0000 7f8051c07700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f804c106110 0x7f804c1a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.915+0000 7f8051c07700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f804c10b750 0x7f804c19bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.915+0000 7f804b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f804c105750 0x7f804c1a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:05.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.915+0000 7f8051c07700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f804c115840 con 0x7f804c105750 2024-06-04T05:23:05.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.915+0000 7f8051c07700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f804c1156c0 con 0x7f804c106110 2024-06-04T05:23:05.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.915+0000 7f8051c07700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f804c1159c0 con 0x7f804c10b750 2024-06-04T05:23:05.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.915+0000 7f804b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f804c105750 0x7f804c1a1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38498/0 (socket says 172.21.15.70:38498) 2024-06-04T05:23:05.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.915+0000 7f804b7fe700 1 -- 172.21.15.70:0/1578008424 learned_addr learned my addr 172.21.15.70:0/1578008424 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:05.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.915+0000 7f804affd700 1 --2- 172.21.15.70:0/1578008424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f804c106110 0x7f804c1a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:05.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.915+0000 7f804bfff700 1 --2- 172.21.15.70:0/1578008424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f804c10b750 0x7f804c19bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:05.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.915+0000 7f804bfff700 1 -- 172.21.15.70:0/1578008424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f804c106110 msgr2=0x7f804c1a1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:05.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.915+0000 7f804bfff700 1 --2- 172.21.15.70:0/1578008424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f804c106110 0x7f804c1a1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.915+0000 7f804bfff700 1 -- 172.21.15.70:0/1578008424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f804c105750 msgr2=0x7f804c1a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:05.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.915+0000 7f804bfff700 1 --2- 172.21.15.70:0/1578008424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f804c105750 0x7f804c1a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.915+0000 7f804bfff700 1 -- 172.21.15.70:0/1578008424 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f804c19c260 con 0x7f804c10b750 2024-06-04T05:23:05.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.915+0000 7f804affd700 1 --2- 172.21.15.70:0/1578008424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f804c106110 0x7f804c1a1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:23:05.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.916+0000 7f804bfff700 1 --2- 172.21.15.70:0/1578008424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f804c10b750 0x7f804c19bb00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8044009790 tx=0x7f804400ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:05.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.916+0000 7f8048ff9700 1 -- 172.21.15.70:0/1578008424 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8044021070 con 0x7f804c10b750 2024-06-04T05:23:05.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.916+0000 7f8048ff9700 1 -- 172.21.15.70:0/1578008424 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8044007ac0 con 0x7f804c10b750 2024-06-04T05:23:05.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.916+0000 7f8048ff9700 1 -- 172.21.15.70:0/1578008424 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8044005040 con 0x7f804c10b750 2024-06-04T05:23:05.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.916+0000 7f8051c07700 1 -- 172.21.15.70:0/1578008424 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f804c19c4f0 con 0x7f804c10b750 2024-06-04T05:23:05.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.916+0000 7f8051c07700 1 -- 172.21.15.70:0/1578008424 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f804c1a8810 con 0x7f804c10b750 2024-06-04T05:23:05.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.919+0000 7f8048ff9700 1 -- 172.21.15.70:0/1578008424 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f8044024070 con 0x7f804c10b750 2024-06-04T05:23:05.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.920+0000 7f8048ff9700 1 --2- 172.21.15.70:0/1578008424 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f80340b2a70 0x7f80340b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.920+0000 7f805c49cb80 1 Processor -- start 2024-06-04T05:23:05.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.921+0000 7f804b7fe700 1 --2- 172.21.15.70:0/1578008424 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f80340b2a70 0x7f80340b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:05.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.921+0000 7f805c49cb80 1 -- start start 2024-06-04T05:23:05.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.921+0000 7f8048ff9700 1 -- 172.21.15.70:0/1578008424 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f804400f020 con 0x7f804c10b750 2024-06-04T05:23:05.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.921+0000 7f805c49cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc4d320 0x55b3bbc4d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.921+0000 7f805c49cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc478a0 0x55b3bbc47d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.921+0000 7f805c49cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc48260 0x55b3bbc54080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.921+0000 7f805c49cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b3bbba5d70 con 0x55b3bbc4d320 2024-06-04T05:23:05.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.921+0000 7f805c49cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b3bbb95910 con 0x55b3bbc48260 2024-06-04T05:23:05.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.921+0000 7f805c49cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b3bbb99950 con 0x55b3bbc478a0 2024-06-04T05:23:05.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.921+0000 7f804affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc478a0 0x55b3bbc47d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:05.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.921+0000 7f804b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc4d320 0x55b3bbc4d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:05.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.921+0000 7f804affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc478a0 0x55b3bbc47d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:46444/0 (socket says 172.21.15.70:46444) 2024-06-04T05:23:05.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.921+0000 7f804affd700 1 -- 172.21.15.70:0/2167921597 learned_addr learned my addr 172.21.15.70:0/2167921597 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:05.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.921+0000 7f804affd700 1 -- 172.21.15.70:0/2167921597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc48260 msgr2=0x55b3bbc54080 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:23:05.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.921+0000 7f804affd700 1 --2- 172.21.15.70:0/2167921597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc48260 0x55b3bbc54080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.922+0000 7f804affd700 1 -- 172.21.15.70:0/2167921597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc4d320 msgr2=0x55b3bbc4d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:05.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.922+0000 7f804affd700 1 --2- 172.21.15.70:0/2167921597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc4d320 0x55b3bbc4d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.922+0000 7f804affd700 1 -- 172.21.15.70:0/2167921597 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3bbb3efb0 con 0x55b3bbc478a0 2024-06-04T05:23:05.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.922+0000 7f804b7fe700 1 --2- 172.21.15.70:0/2167921597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc4d320 0x55b3bbc4d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:23:05.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.922+0000 7f804b7fe700 1 --2- 172.21.15.70:0/1578008424 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f80340b2a70 0x7f80340b4f30 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f803c005950 tx=0x7f803c009bc0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:05.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.922+0000 7f804affd700 1 --2- 172.21.15.70:0/2167921597 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc478a0 0x55b3bbc47d20 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f80400115f0 tx=0x7f8040011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c954f41eab9771d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:05.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.922+0000 7f8031ffb700 1 -- 172.21.15.70:0/2167921597 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8040004220 con 0x55b3bbc478a0 2024-06-04T05:23:05.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.922+0000 7f8031ffb700 1 -- 172.21.15.70:0/2167921597 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f80400043c0 con 0x55b3bbc478a0 2024-06-04T05:23:05.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.922+0000 7f8031ffb700 1 -- 172.21.15.70:0/2167921597 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80400197b0 con 0x55b3bbc478a0 2024-06-04T05:23:05.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.923+0000 7f805c49cb80 1 -- 172.21.15.70:0/2167921597 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc478a0 msgr2=0x55b3bbc47d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:05.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.923+0000 7f805c49cb80 1 --2- 172.21.15.70:0/2167921597 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc478a0 0x55b3bbc47d20 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f80400115f0 tx=0x7f8040011900 comp rx=0 tx=0).stop 2024-06-04T05:23:05.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.923+0000 7f805c49cb80 1 -- 172.21.15.70:0/2167921597 shutdown_connections 2024-06-04T05:23:05.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.923+0000 7f805c49cb80 1 --2- 172.21.15.70:0/2167921597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc48260 0x55b3bbc54080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.923+0000 7f805c49cb80 1 --2- 172.21.15.70:0/2167921597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc4d320 0x55b3bbc4d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.923+0000 7f805c49cb80 1 --2- 172.21.15.70:0/2167921597 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc478a0 0x55b3bbc47d20 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.923+0000 7f805c49cb80 1 -- 172.21.15.70:0/2167921597 >> 172.21.15.70:0/2167921597 conn(0x55b3bbb73240 msgr2=0x55b3bbc423e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:05.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.923+0000 7f805c49cb80 1 -- 172.21.15.70:0/2167921597 shutdown_connections 2024-06-04T05:23:05.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.923+0000 7f805c49cb80 1 -- 172.21.15.70:0/2167921597 wait complete. 2024-06-04T05:23:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.924+0000 7f805c49cb80 1 -- start start 2024-06-04T05:23:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.924+0000 7f805c49cb80 1 Processor -- start 2024-06-04T05:23:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.925+0000 7f805c49cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc478a0 0x55b3bbc53cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.925+0000 7f805c49cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc48260 0x55b3bbc50fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.925+0000 7f805c49cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc4d320 0x55b3bbc51510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.925+0000 7f804b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc478a0 0x55b3bbc53cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.925+0000 7f805c49cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b3bba9bfa0 con 0x55b3bbc478a0 2024-06-04T05:23:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.925+0000 7f805c49cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b3bbafdda0 con 0x55b3bbc48260 2024-06-04T05:23:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.925+0000 7f805c49cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b3bbc546e0 con 0x55b3bbc4d320 2024-06-04T05:23:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.925+0000 7f804b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc478a0 0x55b3bbc53cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38506/0 (socket says 172.21.15.70:38506) 2024-06-04T05:23:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.925+0000 7f804b7fe700 1 -- 172.21.15.70:0/464460918 learned_addr learned my addr 172.21.15.70:0/464460918 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.925+0000 7f804affd700 1 --2- 172.21.15.70:0/464460918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc48260 0x55b3bbc50fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.925+0000 7f805c49cb80 10 client.0 _ll_register_callbacks cb 0x55b3bbb30950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.925+0000 7f804b7fe700 1 -- 172.21.15.70:0/464460918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc4d320 msgr2=0x55b3bbc51510 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-04T05:23:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.925+0000 7f804b7fe700 1 --2- 172.21.15.70:0/464460918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc4d320 0x55b3bbc51510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.925+0000 7f804b7fe700 1 -- 172.21.15.70:0/464460918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc48260 msgr2=0x55b3bbc50fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.925+0000 7f804b7fe700 1 --2- 172.21.15.70:0/464460918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc48260 0x55b3bbc50fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.925+0000 7f804b7fe700 1 -- 172.21.15.70:0/464460918 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3bbae58d0 con 0x55b3bbc478a0 2024-06-04T05:23:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.926+0000 7f804b7fe700 1 --2- 172.21.15.70:0/464460918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc478a0 0x55b3bbc53cd0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f803c0134a0 tx=0x7f803c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e62845f3239901d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.926+0000 7f802affd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.926+0000 7f802affd700 1 -- 172.21.15.70:0/464460918 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803c027070 con 0x55b3bbc478a0 2024-06-04T05:23:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.926+0000 7f805c49cb80 1 -- 172.21.15.70:0/464460918 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80400043c0 con 0x55b3bbc478a0 2024-06-04T05:23:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.926+0000 7f802affd700 1 -- 172.21.15.70:0/464460918 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f803c010700 con 0x55b3bbc478a0 2024-06-04T05:23:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.926+0000 7f802affd700 1 -- 172.21.15.70:0/464460918 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f803c033050 con 0x55b3bbc478a0 2024-06-04T05:23:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.926+0000 7f805c49cb80 10 client.21951 fetch_fsmap learned FSMap version 67 2024-06-04T05:23:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.926+0000 7f805c49cb80 1 -- 172.21.15.70:0/464460918 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55b3bbb4f760 con 0x55b3bbc478a0 2024-06-04T05:23:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.927+0000 7f802affd700 1 -- 172.21.15.70:0/464460918 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f803c010d90 con 0x55b3bbc478a0 2024-06-04T05:23:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.927+0000 7f802affd700 1 -- 172.21.15.70:0/464460918 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f803c020050 con 0x55b3bbc478a0 2024-06-04T05:23:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.927+0000 7f805c49cb80 10 client.21951 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:23:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.927+0000 7f805c49cb80 10 client.21951 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:23:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.927+0000 7f805c49cb80 20 client.21951 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.927+0000 7f805c49cb80 1 --2- 172.21.15.70:0/464460918 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55b3bbc615d0 0x55b3bbc63a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.927+0000 7f805c49cb80 4 client.21951 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:23:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.927+0000 7f805c49cb80 1 -- 172.21.15.70:0/464460918 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f80400043c0 con 0x55b3bbc615d0 2024-06-04T05:23:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.927+0000 7f804affd700 1 --2- 172.21.15.70:0/464460918 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55b3bbc615d0 0x55b3bbc63a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:05.942 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:05 smithi112 ceph-mon[36453]: pgmap v816: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.4 MiB/s wr, 269 op/s 2024-06-04T05:23:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.928+0000 7f802affd700 10 client.21951 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:23:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.928+0000 7f804affd700 1 --2- 172.21.15.70:0/464460918 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55b3bbc615d0 0x55b3bbc63a10 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dd25c95942f873b4 server_cookie=31256e325c3667cf in_seq=0 out_seq=0 2024-06-04T05:23:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.929+0000 7f802affd700 1 -- 172.21.15.70:0/464460918 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f80400043c0 con 0x55b3bbc615d0 2024-06-04T05:23:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.929+0000 7f802affd700 10 client.21951 handle_command_reply: tid=0 2024-06-04T05:23:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.929+0000 7f805c49cb80 1 client.21951 shutdown 2024-06-04T05:23:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.929+0000 7f805c49cb80 1 -- 172.21.15.70:0/464460918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc478a0 msgr2=0x55b3bbc53cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.929+0000 7f805c49cb80 1 --2- 172.21.15.70:0/464460918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc478a0 0x55b3bbc53cd0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f803c0134a0 tx=0x7f803c0047c0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.930+0000 7f805c49cb80 1 -- 172.21.15.70:0/464460918 shutdown_connections 2024-06-04T05:23:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.930+0000 7f805c49cb80 1 --2- 172.21.15.70:0/464460918 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55b3bbc615d0 0x55b3bbc63a10 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.931+0000 7f805c49cb80 1 --2- 172.21.15.70:0/464460918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc4d320 0x55b3bbc51510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.931+0000 7f805c49cb80 1 --2- 172.21.15.70:0/464460918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc478a0 0x55b3bbc53cd0 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.931+0000 7f805c49cb80 1 --2- 172.21.15.70:0/464460918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc48260 0x55b3bbc50fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.931+0000 7f802affd700 0 client.21951 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:23:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.932+0000 7f805c49cb80 1 -- 172.21.15.70:0/464460918 >> 172.21.15.70:0/464460918 conn(0x55b3bbb73240 msgr2=0x55b3bbc41af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.932+0000 7f805c49cb80 1 -- 172.21.15.70:0/464460918 shutdown_connections 2024-06-04T05:23:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.932+0000 7f805c49cb80 1 -- 172.21.15.70:0/464460918 wait complete. 2024-06-04T05:23:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.932+0000 7f805c49cb80 20 client.21951 trim_cache size 0 max 16384 2024-06-04T05:23:05.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.952+0000 7f805c49cb80 1 Processor -- start 2024-06-04T05:23:05.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.952+0000 7f805c49cb80 1 -- start start 2024-06-04T05:23:05.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.952+0000 7f805c49cb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc478a0 0x55b3bbc4c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.952+0000 7f805c49cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc48260 0x55b3bbc53cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.952+0000 7f805c49cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc4d320 0x55b3bbc42270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.953+0000 7f805c49cb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f804c1abb10 con 0x55b3bbc4d320 2024-06-04T05:23:05.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.953+0000 7f805c49cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f804c1ad650 con 0x55b3bbc48260 2024-06-04T05:23:05.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.953+0000 7f805c49cb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f804c1ac1e0 con 0x55b3bbc478a0 2024-06-04T05:23:05.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.953+0000 7f804bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc4d320 0x55b3bbc42270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:05.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.953+0000 7f804b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc478a0 0x55b3bbc4c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:05.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.953+0000 7f804bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc4d320 0x55b3bbc42270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38510/0 (socket says 172.21.15.70:38510) 2024-06-04T05:23:05.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.953+0000 7f804bfff700 1 -- 172.21.15.70:0/1796983376 learned_addr learned my addr 172.21.15.70:0/1796983376 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:05.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.953+0000 7f804affd700 1 --2- 172.21.15.70:0/1796983376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc48260 0x55b3bbc53cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:05.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.953+0000 7f804b7fe700 1 -- 172.21.15.70:0/1796983376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc48260 msgr2=0x55b3bbc53cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:05.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.953+0000 7f804b7fe700 1 --2- 172.21.15.70:0/1796983376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc48260 0x55b3bbc53cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.953+0000 7f804b7fe700 1 -- 172.21.15.70:0/1796983376 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc4d320 msgr2=0x55b3bbc42270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:05.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.953+0000 7f804b7fe700 1 --2- 172.21.15.70:0/1796983376 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc4d320 0x55b3bbc42270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.953+0000 7f804b7fe700 1 -- 172.21.15.70:0/1796983376 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3bbc4f540 con 0x55b3bbc478a0 2024-06-04T05:23:05.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.953+0000 7f804bfff700 1 --2- 172.21.15.70:0/1796983376 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc4d320 0x55b3bbc42270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:23:05.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.953+0000 7f804b7fe700 1 --2- 172.21.15.70:0/1796983376 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc478a0 0x55b3bbc4c990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f803c00a010 tx=0x7f803c009df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72b3c3a025132a26 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:05.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.954+0000 7f80317fa700 1 -- 172.21.15.70:0/1796983376 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803c027040 con 0x55b3bbc478a0 2024-06-04T05:23:05.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.954+0000 7f80317fa700 1 -- 172.21.15.70:0/1796983376 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f803c010700 con 0x55b3bbc478a0 2024-06-04T05:23:05.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.954+0000 7f80317fa700 1 -- 172.21.15.70:0/1796983376 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803c01fe60 con 0x55b3bbc478a0 2024-06-04T05:23:05.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.955+0000 7f805c49cb80 1 -- 172.21.15.70:0/1796983376 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc478a0 msgr2=0x55b3bbc4c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:05.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.955+0000 7f805c49cb80 1 --2- 172.21.15.70:0/1796983376 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc478a0 0x55b3bbc4c990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f803c00a010 tx=0x7f803c009df0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.955+0000 7f805c49cb80 1 -- 172.21.15.70:0/1796983376 shutdown_connections 2024-06-04T05:23:05.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.955+0000 7f805c49cb80 1 --2- 172.21.15.70:0/1796983376 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc4d320 0x55b3bbc42270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.955+0000 7f805c49cb80 1 --2- 172.21.15.70:0/1796983376 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc478a0 0x55b3bbc4c990 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.955+0000 7f805c49cb80 1 --2- 172.21.15.70:0/1796983376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc48260 0x55b3bbc53cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.955+0000 7f805c49cb80 1 -- 172.21.15.70:0/1796983376 >> 172.21.15.70:0/1796983376 conn(0x55b3bbb73240 msgr2=0x55b3bbc46890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:05.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.955+0000 7f805c49cb80 1 -- 172.21.15.70:0/1796983376 shutdown_connections 2024-06-04T05:23:05.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.955+0000 7f805c49cb80 1 -- 172.21.15.70:0/1796983376 wait complete. 2024-06-04T05:23:05.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.955+0000 7f805c49cb80 1 -- start start 2024-06-04T05:23:05.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.956+0000 7f805c49cb80 1 Processor -- start 2024-06-04T05:23:05.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.956+0000 7f805c49cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc478a0 0x55b3bbc68310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.956+0000 7f805c49cb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc48260 0x55b3bbc68850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.956+0000 7f804b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc478a0 0x55b3bbc68310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:05.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.956+0000 7f804b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc478a0 0x55b3bbc68310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59310/0 (socket says 172.21.15.70:59310) 2024-06-04T05:23:05.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.956+0000 7f804affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc48260 0x55b3bbc68850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:05.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.957+0000 7f804affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc48260 0x55b3bbc68850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38518/0 (socket says 172.21.15.70:38518) 2024-06-04T05:23:05.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.957+0000 7f804b7fe700 1 -- 172.21.15.70:0/1003988786 learned_addr learned my addr 172.21.15.70:0/1003988786 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:05.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.957+0000 7f805c49cb80 1 --2- 172.21.15.70:0/1003988786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc4d320 0x55b3bbc68d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.957+0000 7f805c49cb80 1 -- 172.21.15.70:0/1003988786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f804c1acf80 con 0x55b3bbc48260 2024-06-04T05:23:05.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.957+0000 7f805c49cb80 1 -- 172.21.15.70:0/1003988786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f804c1ac8b0 con 0x55b3bbc478a0 2024-06-04T05:23:05.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.957+0000 7f805c49cb80 1 -- 172.21.15.70:0/1003988786 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f804c1a9bf0 con 0x55b3bbc4d320 2024-06-04T05:23:05.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.957+0000 7f804bfff700 1 --2- 172.21.15.70:0/1003988786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc4d320 0x55b3bbc68d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:05.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.957+0000 7f804b7fe700 1 -- 172.21.15.70:0/1003988786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc4d320 msgr2=0x55b3bbc68d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:05.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.957+0000 7f804b7fe700 1 --2- 172.21.15.70:0/1003988786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc4d320 0x55b3bbc68d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.957+0000 7f804b7fe700 1 -- 172.21.15.70:0/1003988786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc48260 msgr2=0x55b3bbc68850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:05.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.957+0000 7f804b7fe700 1 --2- 172.21.15.70:0/1003988786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc48260 0x55b3bbc68850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.957+0000 7f804b7fe700 1 -- 172.21.15.70:0/1003988786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3bbc58a30 con 0x55b3bbc478a0 2024-06-04T05:23:05.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.958+0000 7f804affd700 1 --2- 172.21.15.70:0/1003988786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc48260 0x55b3bbc68850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:23:05.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.958+0000 7f804b7fe700 1 --2- 172.21.15.70:0/1003988786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc478a0 0x55b3bbc68310 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f803c0108b0 tx=0x7f803c0108e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=857757b8a6651947 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:05.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.958+0000 7f802affd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-06-04T05:23:05.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.958+0000 7f805c49cb80 10 client.0 _ll_register_callbacks cb 0x55b3bbb30950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:05.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.958+0000 7f805c49cb80 1 -- 172.21.15.70:0/1003988786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f803c010700 con 0x55b3bbc478a0 2024-06-04T05:23:05.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.958+0000 7f802affd700 1 -- 172.21.15.70:0/1003988786 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803c027040 con 0x55b3bbc478a0 2024-06-04T05:23:05.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.958+0000 7f802affd700 1 -- 172.21.15.70:0/1003988786 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8571+0+0 (secure 0 0 0) 0x7f803c01f620 con 0x55b3bbc478a0 2024-06-04T05:23:05.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.958+0000 7f802affd700 1 -- 172.21.15.70:0/1003988786 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f803c044050 con 0x55b3bbc478a0 2024-06-04T05:23:05.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.958+0000 7f805c49cb80 10 client.30527 fetch_fsmap learned FSMap version 67 2024-06-04T05:23:05.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.958+0000 7f805c49cb80 1 -- 172.21.15.70:0/1003988786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({fsmap=67}) -- 0x55b3bbc599b0 con 0x55b3bbc478a0 2024-06-04T05:23:05.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.959+0000 7f802affd700 1 -- 172.21.15.70:0/1003988786 <== mon.2 v2:172.21.15.98:3300/0 4 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7f803c03f030 con 0x55b3bbc478a0 2024-06-04T05:23:05.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.959+0000 7f805c49cb80 10 client.30527 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:23:05.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.959+0000 7f805c49cb80 10 client.30527 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:23:05.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.959+0000 7f805c49cb80 20 client.30527 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:05.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.959+0000 7f805c49cb80 1 --2- 172.21.15.70:0/1003988786 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55b3bbc59f60 0x55b3bbcce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:05.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.959+0000 7f805c49cb80 4 client.30527 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:23:05.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.959+0000 7f805c49cb80 1 -- 172.21.15.70:0/1003988786 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b3bbba7cc0 con 0x55b3bbc59f60 2024-06-04T05:23:05.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.959+0000 7f804affd700 1 --2- 172.21.15.70:0/1003988786 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55b3bbc59f60 0x55b3bbcce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:05.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.959+0000 7f804affd700 1 --2- 172.21.15.70:0/1003988786 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55b3bbc59f60 0x55b3bbcce200 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=776e2d51670b90cf server_cookie=8ab81c2b4fa47f43 in_seq=0 out_seq=0 2024-06-04T05:23:05.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.959+0000 7f802affd700 10 client.30527 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:23:05.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.960+0000 7f802affd700 1 -- 172.21.15.70:0/1003988786 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b3bbba7cc0 con 0x55b3bbc59f60 2024-06-04T05:23:05.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.960+0000 7f802affd700 10 client.30527 handle_command_reply: tid=0 2024-06-04T05:23:05.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.960+0000 7f805c49cb80 1 client.30527 shutdown 2024-06-04T05:23:05.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.960+0000 7f805c49cb80 1 -- 172.21.15.70:0/1003988786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc478a0 msgr2=0x55b3bbc68310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:05.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.960+0000 7f805c49cb80 1 --2- 172.21.15.70:0/1003988786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc478a0 0x55b3bbc68310 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f803c0108b0 tx=0x7f803c0108e0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.960+0000 7f805c49cb80 1 -- 172.21.15.70:0/1003988786 shutdown_connections 2024-06-04T05:23:05.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.960+0000 7f805c49cb80 1 --2- 172.21.15.70:0/1003988786 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55b3bbc59f60 0x55b3bbcce200 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.960+0000 7f805c49cb80 1 --2- 172.21.15.70:0/1003988786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b3bbc4d320 0x55b3bbc68d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.960+0000 7f805c49cb80 1 --2- 172.21.15.70:0/1003988786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b3bbc478a0 0x55b3bbc68310 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.960+0000 7f805c49cb80 1 --2- 172.21.15.70:0/1003988786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3bbc48260 0x55b3bbc68850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.960+0000 7f805c49cb80 1 -- 172.21.15.70:0/1003988786 >> 172.21.15.70:0/1003988786 conn(0x55b3bbb73240 msgr2=0x55b3bbc4c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:05.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.960+0000 7f802affd700 0 client.30527 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:23:05.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.960+0000 7f805c49cb80 1 -- 172.21.15.70:0/1003988786 shutdown_connections 2024-06-04T05:23:05.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.960+0000 7f805c49cb80 1 -- 172.21.15.70:0/1003988786 wait complete. 2024-06-04T05:23:05.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.960+0000 7f805c49cb80 20 client.30527 trim_cache size 0 max 16384 2024-06-04T05:23:05.966 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:23:05.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.961+0000 7f8051c07700 1 -- 172.21.15.70:0/1578008424 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f80340b2a70 msgr2=0x7f80340b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:05.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.961+0000 7f8051c07700 1 --2- 172.21.15.70:0/1578008424 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f80340b2a70 0x7f80340b4f30 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f803c005950 tx=0x7f803c009bc0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.961+0000 7f8051c07700 1 -- 172.21.15.70:0/1578008424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f804c10b750 msgr2=0x7f804c19bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:05.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.961+0000 7f8051c07700 1 --2- 172.21.15.70:0/1578008424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f804c10b750 0x7f804c19bb00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8044009790 tx=0x7f804400ccb0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.961+0000 7f8051c07700 1 -- 172.21.15.70:0/1578008424 shutdown_connections 2024-06-04T05:23:05.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.961+0000 7f8051c07700 1 --2- 172.21.15.70:0/1578008424 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f80340b2a70 0x7f80340b4f30 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.961+0000 7f8051c07700 1 --2- 172.21.15.70:0/1578008424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f804c10b750 0x7f804c19bb00 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.961+0000 7f8051c07700 1 --2- 172.21.15.70:0/1578008424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f804c105750 0x7f804c1a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.961+0000 7f8051c07700 1 --2- 172.21.15.70:0/1578008424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f804c106110 0x7f804c1a1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:05.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.961+0000 7f8051c07700 1 -- 172.21.15.70:0/1578008424 >> 172.21.15.70:0/1578008424 conn(0x7f804c101220 msgr2=0x7f804c101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:05.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.961+0000 7f8051c07700 1 -- 172.21.15.70:0/1578008424 shutdown_connections 2024-06-04T05:23:05.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:05.961+0000 7f8051c07700 1 -- 172.21.15.70:0/1578008424 wait complete. 2024-06-04T05:23:06.657 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:23:06.838 INFO:tasks.workunit.client.0.smithi070.stdout: 10 89328 6.55 MB/sec execute 598 sec latency 67.196 ms 2024-06-04T05:23:06.991 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:07.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:07 smithi098 ceph-mon[35569]: pgmap v817: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.2 MiB/s wr, 320 op/s 2024-06-04T05:23:07.838 INFO:tasks.workunit.client.0.smithi070.stdout: 10 89535 6.56 MB/sec execute 599 sec latency 64.958 ms 2024-06-04T05:23:07.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:07 smithi070 ceph-mon[32631]: pgmap v817: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.2 MiB/s wr, 320 op/s 2024-06-04T05:23:07.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:07 smithi112 ceph-mon[36453]: pgmap v817: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.2 MiB/s wr, 320 op/s 2024-06-04T05:23:08.837 INFO:tasks.workunit.client.0.smithi070.stdout: 10 cleanup 600 sec 2024-06-04T05:23:08.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.873+0000 7efcb2d48700 1 -- 172.21.15.70:0/3626008767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcac1039b0 msgr2=0x7efcac10f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:08.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.873+0000 7efcb2d48700 1 --2- 172.21.15.70:0/3626008767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcac1039b0 0x7efcac10f310 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7efca400b0a0 tx=0x7efca400b3b0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.873+0000 7efcb2d48700 1 -- 172.21.15.70:0/3626008767 shutdown_connections 2024-06-04T05:23:08.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.873+0000 7efcb2d48700 1 --2- 172.21.15.70:0/3626008767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcac1039b0 0x7efcac10f310 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.873+0000 7efcb2d48700 1 --2- 172.21.15.70:0/3626008767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcac107470 0x7efcac107850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.873+0000 7efcb2d48700 1 --2- 172.21.15.70:0/3626008767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efcac102ff0 0x7efcac103470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.873+0000 7efcb2d48700 1 -- 172.21.15.70:0/3626008767 >> 172.21.15.70:0/3626008767 conn(0x7efcac0777e0 msgr2=0x7efcac077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:08.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.873+0000 7efcb2d48700 1 -- 172.21.15.70:0/3626008767 shutdown_connections 2024-06-04T05:23:08.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.873+0000 7efcb2d48700 1 -- 172.21.15.70:0/3626008767 wait complete. 2024-06-04T05:23:08.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.874+0000 7efcb2d48700 1 Processor -- start 2024-06-04T05:23:08.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.874+0000 7efcb2d48700 1 -- start start 2024-06-04T05:23:08.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.874+0000 7efcb2d48700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcac102ff0 0x7efcac109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.874+0000 7efcb2d48700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efcac1039b0 0x7efcac1098d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.874+0000 7efcb2d48700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcac107470 0x7efcac109e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.874+0000 7efcb2d48700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efcac1124b0 con 0x7efcac102ff0 2024-06-04T05:23:08.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.874+0000 7efcb2d48700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efcac112330 con 0x7efcac107470 2024-06-04T05:23:08.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.874+0000 7efcb2d48700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efcac112630 con 0x7efcac1039b0 2024-06-04T05:23:08.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.875+0000 7efcabfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efcac1039b0 0x7efcac1098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.875+0000 7efcb12e5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcac107470 0x7efcac109e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.875+0000 7efcabfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efcac1039b0 0x7efcac1098d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:43006/0 (socket says 172.21.15.70:43006) 2024-06-04T05:23:08.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.875+0000 7efcabfff700 1 -- 172.21.15.70:0/1645952546 learned_addr learned my addr 172.21.15.70:0/1645952546 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:08.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.875+0000 7efcb0ae4700 1 --2- 172.21.15.70:0/1645952546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcac102ff0 0x7efcac109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.875+0000 7efcb12e5700 1 -- 172.21.15.70:0/1645952546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efcac1039b0 msgr2=0x7efcac1098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:08.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.875+0000 7efcb12e5700 1 --2- 172.21.15.70:0/1645952546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efcac1039b0 0x7efcac1098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.875+0000 7efcb12e5700 1 -- 172.21.15.70:0/1645952546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcac102ff0 msgr2=0x7efcac109390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:08.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.875+0000 7efcb12e5700 1 --2- 172.21.15.70:0/1645952546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcac102ff0 0x7efcac109390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.875+0000 7efcb12e5700 1 -- 172.21.15.70:0/1645952546 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcac1ac2f0 con 0x7efcac107470 2024-06-04T05:23:08.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.875+0000 7efcabfff700 1 --2- 172.21.15.70:0/1645952546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efcac1039b0 0x7efcac1098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:23:08.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.875+0000 7efcb0ae4700 1 --2- 172.21.15.70:0/1645952546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcac102ff0 0x7efcac109390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:08.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.875+0000 7efcb12e5700 1 --2- 172.21.15.70:0/1645952546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcac107470 0x7efcac109e10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efca400b0a0 tx=0x7efca4004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:08.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.875+0000 7efca9ffb700 1 -- 172.21.15.70:0/1645952546 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca4021070 con 0x7efcac107470 2024-06-04T05:23:08.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.876+0000 7efcb2d48700 1 -- 172.21.15.70:0/1645952546 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcac1ac580 con 0x7efcac107470 2024-06-04T05:23:08.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.876+0000 7efca9ffb700 1 -- 172.21.15.70:0/1645952546 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efca4007b80 con 0x7efcac107470 2024-06-04T05:23:08.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.876+0000 7efca9ffb700 1 -- 172.21.15.70:0/1645952546 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca4005040 con 0x7efcac107470 2024-06-04T05:23:08.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.876+0000 7efcb2d48700 1 -- 172.21.15.70:0/1645952546 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcac1aca80 con 0x7efcac107470 2024-06-04T05:23:08.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.879+0000 7efca9ffb700 1 -- 172.21.15.70:0/1645952546 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7efca4024070 con 0x7efcac107470 2024-06-04T05:23:08.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.880+0000 7efca9ffb700 1 --2- 172.21.15.70:0/1645952546 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efc9c0b2af0 0x7efc9c0b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.880+0000 7efcb0ae4700 1 --2- 172.21.15.70:0/1645952546 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efc9c0b2af0 0x7efc9c0b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.880+0000 7efca9ffb700 1 -- 172.21.15.70:0/1645952546 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8615+0+0 (secure 0 0 0) 0x7efca400f020 con 0x7efcac107470 2024-06-04T05:23:08.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.880+0000 7efcbd5ddb80 1 Processor -- start 2024-06-04T05:23:08.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.880+0000 7efcbd5ddb80 1 -- start start 2024-06-04T05:23:08.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcbd5ddb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f07237c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcbd5ddb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f07238210 0x562f0723b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcbd5ddb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f0723bf60 0x562f0723c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcbd5ddb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562f07196c90 con 0x562f07237860 2024-06-04T05:23:08.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcbd5ddb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562f07186830 con 0x562f0723bf60 2024-06-04T05:23:08.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcbd5ddb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562f0718a870 con 0x562f07238210 2024-06-04T05:23:08.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcb0ae4700 1 --2- 172.21.15.70:0/1645952546 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efc9c0b2af0 0x7efc9c0b4fb0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7efca0009cc0 tx=0x7efca0009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:08.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcb0ae4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f07237c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcabfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f07238210 0x562f0723b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcb0ae4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f07237c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59768/0 (socket says 172.21.15.70:59768) 2024-06-04T05:23:08.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcb0ae4700 1 -- 172.21.15.70:0/2672003434 learned_addr learned my addr 172.21.15.70:0/2672003434 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:08.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcb12e5700 1 --2- 172.21.15.70:0/2672003434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f0723bf60 0x562f0723c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcabfff700 1 -- 172.21.15.70:0/2672003434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f0723bf60 msgr2=0x562f0723c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:08.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcabfff700 1 --2- 172.21.15.70:0/2672003434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f0723bf60 0x562f0723c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcabfff700 1 -- 172.21.15.70:0/2672003434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 msgr2=0x562f07237c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:08.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcabfff700 1 --2- 172.21.15.70:0/2672003434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f07237c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcabfff700 1 -- 172.21.15.70:0/2672003434 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f070d68d0 con 0x562f07238210 2024-06-04T05:23:08.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.881+0000 7efcb0ae4700 1 --2- 172.21.15.70:0/2672003434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f07237c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:23:08.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.882+0000 7efcabfff700 1 --2- 172.21.15.70:0/2672003434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f07238210 0x562f0723b940 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7efcac10abe0 tx=0x7efc98011820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ed2c244225751b3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:08.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.882+0000 7efc96ffd700 1 -- 172.21.15.70:0/2672003434 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc98004230 con 0x562f07238210 2024-06-04T05:23:08.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.882+0000 7efc96ffd700 1 -- 172.21.15.70:0/2672003434 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc980043d0 con 0x562f07238210 2024-06-04T05:23:08.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.882+0000 7efc96ffd700 1 -- 172.21.15.70:0/2672003434 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc98019860 con 0x562f07238210 2024-06-04T05:23:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.883+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/2672003434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f07238210 msgr2=0x562f0723b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.883+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/2672003434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f07238210 0x562f0723b940 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7efcac10abe0 tx=0x7efc98011820 comp rx=0 tx=0).stop 2024-06-04T05:23:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.883+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/2672003434 shutdown_connections 2024-06-04T05:23:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.883+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/2672003434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f0723bf60 0x562f0723c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.883+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/2672003434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f07237c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.883+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/2672003434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f07238210 0x562f0723b940 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.883+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/2672003434 >> 172.21.15.70:0/2672003434 conn(0x562f07165080 msgr2=0x562f07231e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:08.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.883+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/2672003434 shutdown_connections 2024-06-04T05:23:08.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.883+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/2672003434 wait complete. 2024-06-04T05:23:08.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.883+0000 7efcbd5ddb80 1 -- start start 2024-06-04T05:23:08.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.883+0000 7efcbd5ddb80 1 Processor -- start 2024-06-04T05:23:08.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.884+0000 7efcbd5ddb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f072433a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.884+0000 7efcbd5ddb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f07238210 0x562f072406a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.884+0000 7efcbd5ddb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f0723bf60 0x562f07240be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.884+0000 7efcbd5ddb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562f0708cfa0 con 0x562f07237860 2024-06-04T05:23:08.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.884+0000 7efcbd5ddb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562f071287c0 con 0x562f0723bf60 2024-06-04T05:23:08.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.884+0000 7efcbd5ddb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562f07244120 con 0x562f07238210 2024-06-04T05:23:08.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.884+0000 7efcbd5ddb80 10 client.0 _ll_register_callbacks cb 0x562f07121920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:08.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.884+0000 7efcb0ae4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f072433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.884+0000 7efcb0ae4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f072433a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59780/0 (socket says 172.21.15.70:59780) 2024-06-04T05:23:08.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.884+0000 7efcb0ae4700 1 -- 172.21.15.70:0/831053061 learned_addr learned my addr 172.21.15.70:0/831053061 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:08.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.884+0000 7efcabfff700 1 --2- 172.21.15.70:0/831053061 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f07238210 0x562f072406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.884+0000 7efcb12e5700 1 --2- 172.21.15.70:0/831053061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f0723bf60 0x562f07240be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.884+0000 7efcb0ae4700 1 -- 172.21.15.70:0/831053061 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f07238210 msgr2=0x562f072406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:08.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.884+0000 7efcb0ae4700 1 --2- 172.21.15.70:0/831053061 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f07238210 0x562f072406a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.884+0000 7efcb0ae4700 1 -- 172.21.15.70:0/831053061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f0723bf60 msgr2=0x562f07240be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:08.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.885+0000 7efcb0ae4700 1 --2- 172.21.15.70:0/831053061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f0723bf60 0x562f07240be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.885+0000 7efcb0ae4700 1 -- 172.21.15.70:0/831053061 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f07125690 con 0x562f07237860 2024-06-04T05:23:08.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.885+0000 7efcabfff700 1 --2- 172.21.15.70:0/831053061 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f07238210 0x562f072406a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:08.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.885+0000 7efcb0ae4700 1 --2- 172.21.15.70:0/831053061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f072433a0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7efca0012c40 tx=0x7efca00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=818ce28b0aa8ed9c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:08.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.885+0000 7efc8bfff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:08.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.885+0000 7efc8bfff700 1 -- 172.21.15.70:0/831053061 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca0020cd0 con 0x562f07237860 2024-06-04T05:23:08.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.885+0000 7efc8bfff700 1 -- 172.21.15.70:0/831053061 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8615+0+0 (secure 0 0 0) 0x7efca00283e0 con 0x562f07237860 2024-06-04T05:23:08.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.885+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/831053061 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc980043d0 con 0x562f07237860 2024-06-04T05:23:08.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.885+0000 7efc8bfff700 1 -- 172.21.15.70:0/831053061 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7efca00285e0 con 0x562f07237860 2024-06-04T05:23:08.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.885+0000 7efcbd5ddb80 10 client.21969 fetch_fsmap learned FSMap version 67 2024-06-04T05:23:08.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.885+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/831053061 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x562f0724e870 con 0x562f07237860 2024-06-04T05:23:08.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.886+0000 7efc8bfff700 1 -- 172.21.15.70:0/831053061 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc980043d0 con 0x562f07237860 2024-06-04T05:23:08.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.886+0000 7efc8bfff700 1 -- 172.21.15.70:0/831053061 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7efca0027070 con 0x562f07237860 2024-06-04T05:23:08.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.886+0000 7efcbd5ddb80 10 client.21969 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:23:08.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.886+0000 7efcbd5ddb80 10 client.21969 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:23:08.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.886+0000 7efcbd5ddb80 20 client.21969 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:08.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.886+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/831053061 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x562f07250f50 0x562f07253340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.886+0000 7efcbd5ddb80 4 client.21969 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:23:08.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.886+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/831053061 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f072039e0 con 0x562f07250f50 2024-06-04T05:23:08.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.886+0000 7efcabfff700 1 --2- 172.21.15.70:0/831053061 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x562f07250f50 0x562f07253340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.886+0000 7efc8bfff700 10 client.21969 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:23:08.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.886+0000 7efcabfff700 1 --2- 172.21.15.70:0/831053061 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x562f07250f50 0x562f07253340 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c8363c7aa0a388c4 server_cookie=1b54c527c764d340 in_seq=0 out_seq=0 2024-06-04T05:23:08.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.888+0000 7efc8bfff700 1 -- 172.21.15.70:0/831053061 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562f072039e0 con 0x562f07250f50 2024-06-04T05:23:08.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.888+0000 7efc8bfff700 10 client.21969 handle_command_reply: tid=0 2024-06-04T05:23:08.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.888+0000 7efcbd5ddb80 1 client.21969 shutdown 2024-06-04T05:23:08.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.888+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/831053061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 msgr2=0x562f072433a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:08.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.888+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/831053061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f072433a0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7efca0012c40 tx=0x7efca00047c0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.888+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/831053061 shutdown_connections 2024-06-04T05:23:08.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.888+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/831053061 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x562f07250f50 0x562f07253340 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.888+0000 7efc8bfff700 0 client.21969 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:23:08.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.888+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/831053061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f0723bf60 0x562f07240be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.888+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/831053061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f072433a0 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.888+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/831053061 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f07238210 0x562f072406a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.888+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/831053061 >> 172.21.15.70:0/831053061 conn(0x562f07165080 msgr2=0x562f0723d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:08.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.888+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/831053061 shutdown_connections 2024-06-04T05:23:08.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.888+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/831053061 wait complete. 2024-06-04T05:23:08.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.889+0000 7efcbd5ddb80 20 client.21969 trim_cache size 0 max 16384 2024-06-04T05:23:08.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.915+0000 7efcbd5ddb80 1 Processor -- start 2024-06-04T05:23:08.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.915+0000 7efcbd5ddb80 1 -- start start 2024-06-04T05:23:08.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.915+0000 7efcbd5ddb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f07237860 0x562f07232530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.915+0000 7efcbd5ddb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07238210 0x562f07258c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.915+0000 7efcbd5ddb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f0723bf60 0x562f07259190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.915+0000 7efcbd5ddb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efcac1b43c0 con 0x562f07238210 2024-06-04T05:23:08.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.915+0000 7efcbd5ddb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efcac1b5f00 con 0x562f0723bf60 2024-06-04T05:23:08.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.915+0000 7efcbd5ddb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efcac1b4a90 con 0x562f07237860 2024-06-04T05:23:08.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.916+0000 7efcabfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07238210 0x562f07258c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.916+0000 7efcabfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07238210 0x562f07258c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59796/0 (socket says 172.21.15.70:59796) 2024-06-04T05:23:08.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.916+0000 7efcb12e5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f0723bf60 0x562f07259190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.916+0000 7efcabfff700 1 -- 172.21.15.70:0/3558362836 learned_addr learned my addr 172.21.15.70:0/3558362836 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:08.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.916+0000 7efcb0ae4700 1 --2- 172.21.15.70:0/3558362836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f07237860 0x562f07232530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.916+0000 7efcabfff700 1 -- 172.21.15.70:0/3558362836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f07237860 msgr2=0x562f07232530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:08.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.916+0000 7efcabfff700 1 --2- 172.21.15.70:0/3558362836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f07237860 0x562f07232530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.916+0000 7efcabfff700 1 -- 172.21.15.70:0/3558362836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f0723bf60 msgr2=0x562f07259190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:08.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.916+0000 7efcabfff700 1 --2- 172.21.15.70:0/3558362836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f0723bf60 0x562f07259190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.916+0000 7efcabfff700 1 -- 172.21.15.70:0/3558362836 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f07231350 con 0x562f07238210 2024-06-04T05:23:08.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.916+0000 7efcb12e5700 1 --2- 172.21.15.70:0/3558362836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f0723bf60 0x562f07259190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:23:08.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.916+0000 7efcabfff700 1 --2- 172.21.15.70:0/3558362836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07238210 0x562f07258c50 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7efc98011b70 tx=0x7efc98011820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b76c9565974f92 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:08.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.916+0000 7efc967fc700 1 -- 172.21.15.70:0/3558362836 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc980054c0 con 0x562f07238210 2024-06-04T05:23:08.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.916+0000 7efc967fc700 1 -- 172.21.15.70:0/3558362836 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc98005660 con 0x562f07238210 2024-06-04T05:23:08.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.916+0000 7efc967fc700 1 -- 172.21.15.70:0/3558362836 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9802c6b0 con 0x562f07238210 2024-06-04T05:23:08.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.917+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/3558362836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07238210 msgr2=0x562f07258c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:08.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.917+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/3558362836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07238210 0x562f07258c50 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7efc98011b70 tx=0x7efc98011820 comp rx=0 tx=0).stop 2024-06-04T05:23:08.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.918+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/3558362836 shutdown_connections 2024-06-04T05:23:08.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.918+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/3558362836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f0723bf60 0x562f07259190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.918+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/3558362836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f07237860 0x562f07232530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.918+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/3558362836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07238210 0x562f07258c50 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.918+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/3558362836 >> 172.21.15.70:0/3558362836 conn(0x562f07165080 msgr2=0x562f07231e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:08.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.918+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/3558362836 shutdown_connections 2024-06-04T05:23:08.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.918+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/3558362836 wait complete. 2024-06-04T05:23:08.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.918+0000 7efcbd5ddb80 1 -- start start 2024-06-04T05:23:08.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.918+0000 7efcbd5ddb80 1 Processor -- start 2024-06-04T05:23:08.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.919+0000 7efcbd5ddb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f07249870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.919+0000 7efcbd5ddb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f07238210 0x562f072390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.919+0000 7efcbd5ddb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f0723bf60 0x562f07239600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.919+0000 7efcbd5ddb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efcac1b5830 con 0x562f07237860 2024-06-04T05:23:08.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.919+0000 7efcbd5ddb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efcac1b5160 con 0x562f07238210 2024-06-04T05:23:08.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.919+0000 7efcbd5ddb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efcac1b2380 con 0x562f0723bf60 2024-06-04T05:23:08.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.919+0000 7efcbd5ddb80 10 client.0 _ll_register_callbacks cb 0x562f07121920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:08.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.919+0000 7efcb0ae4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f07249870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.919+0000 7efcabfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f07238210 0x562f072390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.919+0000 7efcb0ae4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f07249870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59812/0 (socket says 172.21.15.70:59812) 2024-06-04T05:23:08.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.919+0000 7efcb0ae4700 1 -- 172.21.15.70:0/3969692997 learned_addr learned my addr 172.21.15.70:0/3969692997 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:08.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.919+0000 7efcb12e5700 1 --2- 172.21.15.70:0/3969692997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f0723bf60 0x562f07239600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.920+0000 7efcb0ae4700 1 -- 172.21.15.70:0/3969692997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f0723bf60 msgr2=0x562f07239600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:08.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.920+0000 7efcb0ae4700 1 --2- 172.21.15.70:0/3969692997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f0723bf60 0x562f07239600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.920+0000 7efcb0ae4700 1 -- 172.21.15.70:0/3969692997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f07238210 msgr2=0x562f072390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:08.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.920+0000 7efcb0ae4700 1 --2- 172.21.15.70:0/3969692997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f07238210 0x562f072390c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.920+0000 7efcb0ae4700 1 -- 172.21.15.70:0/3969692997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f07239c40 con 0x562f07237860 2024-06-04T05:23:08.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.920+0000 7efcb0ae4700 1 --2- 172.21.15.70:0/3969692997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f07249870 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7efca0012c40 tx=0x7efca00296f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6549ad7ffbd4e0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:08.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.920+0000 7efc8bfff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:08.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.920+0000 7efc8bfff700 1 -- 172.21.15.70:0/3969692997 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca0020ce0 con 0x562f07237860 2024-06-04T05:23:08.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.920+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/3969692997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc98005660 con 0x562f07237860 2024-06-04T05:23:08.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.920+0000 7efc8bfff700 1 -- 172.21.15.70:0/3969692997 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8615+0+0 (secure 0 0 0) 0x7efca0028780 con 0x562f07237860 2024-06-04T05:23:08.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.921+0000 7efc8bfff700 1 -- 172.21.15.70:0/3969692997 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7efca0041050 con 0x562f07237860 2024-06-04T05:23:08.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.921+0000 7efcbd5ddb80 10 client.21975 fetch_fsmap learned FSMap version 67 2024-06-04T05:23:08.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.921+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/3969692997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x562f072bb930 con 0x562f07237860 2024-06-04T05:23:08.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.921+0000 7efc8bfff700 1 -- 172.21.15.70:0/3969692997 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc98005660 con 0x562f07237860 2024-06-04T05:23:08.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.921+0000 7efc8bfff700 1 -- 172.21.15.70:0/3969692997 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 67) ==== 8354+0+0 (secure 0 0 0) 0x7efca0038070 con 0x562f07237860 2024-06-04T05:23:08.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.921+0000 7efcbd5ddb80 10 client.21975 fetch_fsmap finished waiting for FSMap version 67 2024-06-04T05:23:08.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.921+0000 7efcbd5ddb80 10 client.21975 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:23:08.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.921+0000 7efcbd5ddb80 20 client.21975 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:08.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.921+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/3969692997 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x562f07250f50 0x562f072be080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:08.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.921+0000 7efcbd5ddb80 4 client.21975 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:23:08.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.921+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/3969692997 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562f07198be0 con 0x562f07250f50 2024-06-04T05:23:08.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.921+0000 7efcabfff700 1 --2- 172.21.15.70:0/3969692997 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x562f07250f50 0x562f072be080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:08.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.923+0000 7efcabfff700 1 --2- 172.21.15.70:0/3969692997 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x562f07250f50 0x562f072be080 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=812727f75ec7426b server_cookie=13eca14c58f11bb4 in_seq=0 out_seq=0 2024-06-04T05:23:08.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.923+0000 7efc8bfff700 10 client.21975 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:23:08.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.927+0000 7efc8bfff700 1 -- 172.21.15.70:0/3969692997 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562f07198be0 con 0x562f07250f50 2024-06-04T05:23:08.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.927+0000 7efc8bfff700 10 client.21975 handle_command_reply: tid=0 2024-06-04T05:23:08.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.927+0000 7efcbd5ddb80 1 client.21975 shutdown 2024-06-04T05:23:08.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.927+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/3969692997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 msgr2=0x562f07249870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:08.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.927+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/3969692997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f07249870 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7efca0012c40 tx=0x7efca00296f0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.927+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/3969692997 shutdown_connections 2024-06-04T05:23:08.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.927+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/3969692997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562f0723bf60 0x562f07239600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.927+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/3969692997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f07237860 0x562f07249870 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.927+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/3969692997 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x562f07250f50 0x562f072be080 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.927+0000 7efcbd5ddb80 1 --2- 172.21.15.70:0/3969692997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f07238210 0x562f072390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.928+0000 7efc8bfff700 0 client.21975 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:23:08.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.928+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/3969692997 >> 172.21.15.70:0/3969692997 conn(0x562f07165080 msgr2=0x562f0723fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:08.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.928+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/3969692997 shutdown_connections 2024-06-04T05:23:08.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.928+0000 7efcbd5ddb80 1 -- 172.21.15.70:0/3969692997 wait complete. 2024-06-04T05:23:08.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.928+0000 7efcbd5ddb80 20 client.21975 trim_cache size 0 max 16384 2024-06-04T05:23:08.931 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:23:08.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.929+0000 7efcb2d48700 1 -- 172.21.15.70:0/1645952546 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efc9c0b2af0 msgr2=0x7efc9c0b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:08.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.929+0000 7efcb2d48700 1 --2- 172.21.15.70:0/1645952546 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efc9c0b2af0 0x7efc9c0b4fb0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7efca0009cc0 tx=0x7efca0009260 comp rx=0 tx=0).stop 2024-06-04T05:23:08.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.929+0000 7efcb2d48700 1 -- 172.21.15.70:0/1645952546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcac107470 msgr2=0x7efcac109e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:08.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.929+0000 7efcb2d48700 1 --2- 172.21.15.70:0/1645952546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcac107470 0x7efcac109e10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efca400b0a0 tx=0x7efca4004d00 comp rx=0 tx=0).stop 2024-06-04T05:23:08.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.929+0000 7efcb2d48700 1 -- 172.21.15.70:0/1645952546 shutdown_connections 2024-06-04T05:23:08.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.929+0000 7efcb2d48700 1 --2- 172.21.15.70:0/1645952546 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7efc9c0b2af0 0x7efc9c0b4fb0 secure :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7efca0009cc0 tx=0x7efca0009260 comp rx=0 tx=0).stop 2024-06-04T05:23:08.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.929+0000 7efcb2d48700 1 --2- 172.21.15.70:0/1645952546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcac107470 0x7efcac109e10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.929+0000 7efcb2d48700 1 --2- 172.21.15.70:0/1645952546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcac102ff0 0x7efcac109390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.929+0000 7efcb2d48700 1 --2- 172.21.15.70:0/1645952546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efcac1039b0 0x7efcac1098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:08.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.929+0000 7efcb2d48700 1 -- 172.21.15.70:0/1645952546 >> 172.21.15.70:0/1645952546 conn(0x7efcac0777e0 msgr2=0x7efcac10d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:08.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.930+0000 7efcb2d48700 1 -- 172.21.15.70:0/1645952546 shutdown_connections 2024-06-04T05:23:08.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:08.930+0000 7efcb2d48700 1 -- 172.21.15.70:0/1645952546 wait complete. 2024-06-04T05:23:09.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:09 smithi070 ceph-mon[32631]: pgmap v818: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 196 op/s 2024-06-04T05:23:09.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:09 smithi070 ceph-mon[32631]: osdmap e119: 12 total, 12 up, 12 in 2024-06-04T05:23:09.669 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:23:09.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:09 smithi098 ceph-mon[35569]: pgmap v818: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 196 op/s 2024-06-04T05:23:09.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:09 smithi098 ceph-mon[35569]: osdmap e119: 12 total, 12 up, 12 in 2024-06-04T05:23:09.838 INFO:tasks.workunit.client.0.smithi070.stdout: 5 cleanup 601 sec 2024-06-04T05:23:09.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:09 smithi112 ceph-mon[36453]: pgmap v818: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 196 op/s 2024-06-04T05:23:09.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:09 smithi112 ceph-mon[36453]: osdmap e119: 12 total, 12 up, 12 in 2024-06-04T05:23:09.990 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:10.838 INFO:tasks.workunit.client.0.smithi070.stdout: 1 cleanup 602 sec 2024-06-04T05:23:10.845 INFO:tasks.workunit.client.0.smithi070.stdout: 0 cleanup 602 sec 2024-06-04T05:23:10.845 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-06-04T05:23:10.846 INFO:tasks.workunit.client.0.smithi070.stdout: Operation Count AvgLat MaxLat 2024-06-04T05:23:10.846 INFO:tasks.workunit.client.0.smithi070.stdout: ---------------------------------------- 2024-06-04T05:23:10.846 INFO:tasks.workunit.client.0.smithi070.stdout: NTCreateX 126893 17.028 28710.819 2024-06-04T05:23:10.846 INFO:tasks.workunit.client.0.smithi070.stdout: Close 93227 0.187 86.578 2024-06-04T05:23:10.846 INFO:tasks.workunit.client.0.smithi070.stdout: Rename 5369 38.749 22175.142 2024-06-04T05:23:10.846 INFO:tasks.workunit.client.0.smithi070.stdout: Unlink 25573 16.466 28708.079 2024-06-04T05:23:10.846 INFO:tasks.workunit.client.0.smithi070.stdout: Qpathinfo 114875 6.716 489.504 2024-06-04T05:23:10.846 INFO:tasks.workunit.client.0.smithi070.stdout: Qfileinfo 20002 0.367 151.259 2024-06-04T05:23:10.846 INFO:tasks.workunit.client.0.smithi070.stdout: Qfsinfo 21032 0.953 164.865 2024-06-04T05:23:10.846 INFO:tasks.workunit.client.0.smithi070.stdout: Sfileinfo 10363 19.096 23950.918 2024-06-04T05:23:10.847 INFO:tasks.workunit.client.0.smithi070.stdout: Find 44325 41.485 4789.984 2024-06-04T05:23:10.847 INFO:tasks.workunit.client.0.smithi070.stdout: WriteX 62576 4.576 28417.071 2024-06-04T05:23:10.847 INFO:tasks.workunit.client.0.smithi070.stdout: ReadX 198496 0.008 8.747 2024-06-04T05:23:10.847 INFO:tasks.workunit.client.0.smithi070.stdout: LockX 408 1.908 10.896 2024-06-04T05:23:10.847 INFO:tasks.workunit.client.0.smithi070.stdout: UnlockX 408 1.689 6.416 2024-06-04T05:23:10.847 INFO:tasks.workunit.client.0.smithi070.stdout: Flush 8916 7.357 308.610 2024-06-04T05:23:10.847 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-06-04T05:23:10.847 INFO:tasks.workunit.client.0.smithi070.stdout:Throughput 6.557 MB/sec 10 clients 10 procs max_latency=28710.822 ms 2024-06-04T05:23:10.848 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-04T05:23:10.848 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-04T05:23:11.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:11 smithi070 ceph-mon[32631]: pgmap v820: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 235 op/s 2024-06-04T05:23:11.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:11 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:23:11.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:11 smithi098 ceph-mon[35569]: pgmap v820: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 235 op/s 2024-06-04T05:23:11.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:11 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:23:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:11 smithi112 ceph-mon[36453]: pgmap v820: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 235 op/s 2024-06-04T05:23:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:11 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:23:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.102+0000 7f91a2d2b640 1 -- 172.21.15.70:0/1054505535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f919c107940 msgr2=0x7f919c107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.102+0000 7f91a2d2b640 1 --2- 172.21.15.70:0/1054505535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f919c107940 0x7f919c107d20 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f918c009930 tx=0x7f918c02a420 comp rx=0 tx=0).stop 2024-06-04T05:23:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.102+0000 7f91a2d2b640 1 -- 172.21.15.70:0/1054505535 shutdown_connections 2024-06-04T05:23:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.102+0000 7f91a2d2b640 1 --2- 172.21.15.70:0/1054505535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f919c102340 0x7f919c10e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.102+0000 7f91a2d2b640 1 --2- 172.21.15.70:0/1054505535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f919c101980 0x7f919c101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.102+0000 7f91a2d2b640 1 --2- 172.21.15.70:0/1054505535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f919c107940 0x7f919c107d20 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.102+0000 7f91a2d2b640 1 -- 172.21.15.70:0/1054505535 >> 172.21.15.70:0/1054505535 conn(0x7f919c078ad0 msgr2=0x7f919c078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.102+0000 7f91a2d2b640 1 -- 172.21.15.70:0/1054505535 shutdown_connections 2024-06-04T05:23:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.102+0000 7f91a2d2b640 1 -- 172.21.15.70:0/1054505535 wait complete. 2024-06-04T05:23:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.103+0000 7f91a2d2b640 1 Processor -- start 2024-06-04T05:23:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.103+0000 7f91a2d2b640 1 -- start start 2024-06-04T05:23:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.103+0000 7f91a2d2b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f919c101980 0x7f919c1991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.103+0000 7f91a2d2b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f919c102340 0x7f919c199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.103+0000 7f91a2d2b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f919c107940 0x7f919c19da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.103+0000 7f91a2d2b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f919c111ab0 con 0x7f919c102340 2024-06-04T05:23:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.103+0000 7f91a2d2b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f919c111930 con 0x7f919c101980 2024-06-04T05:23:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.103+0000 7f91a2d2b640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f919c111c30 con 0x7f919c107940 2024-06-04T05:23:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.103+0000 7f919bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f919c102340 0x7f919c199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.103+0000 7f91a0ac7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f919c101980 0x7f919c1991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.103+0000 7f919bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f919c102340 0x7f919c199700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59836/0 (socket says 172.21.15.70:59836) 2024-06-04T05:23:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.103+0000 7f91a0ac7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f919c101980 0x7f919c1991c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55802/0 (socket says 172.21.15.70:55802) 2024-06-04T05:23:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.103+0000 7f919bfff640 1 -- 172.21.15.70:0/2404086064 learned_addr learned my addr 172.21.15.70:0/2404086064 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.103+0000 7f91a12c8640 1 --2- 172.21.15.70:0/2404086064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f919c107940 0x7f919c19da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.104+0000 7f919bfff640 1 -- 172.21.15.70:0/2404086064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f919c107940 msgr2=0x7f919c19da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.104+0000 7f919bfff640 1 --2- 172.21.15.70:0/2404086064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f919c107940 0x7f919c19da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.104+0000 7f919bfff640 1 -- 172.21.15.70:0/2404086064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f919c101980 msgr2=0x7f919c1991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.104+0000 7f919bfff640 1 --2- 172.21.15.70:0/2404086064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f919c101980 0x7f919c1991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.104+0000 7f919bfff640 1 -- 172.21.15.70:0/2404086064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f919c19e160 con 0x7f919c102340 2024-06-04T05:23:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.104+0000 7f919bfff640 1 --2- 172.21.15.70:0/2404086064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f919c102340 0x7f919c199700 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f919000ea80 tx=0x7f919000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.104+0000 7f9199ffb640 1 -- 172.21.15.70:0/2404086064 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919000ce40 con 0x7f919c102340 2024-06-04T05:23:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.104+0000 7f9199ffb640 1 -- 172.21.15.70:0/2404086064 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9190005020 con 0x7f919c102340 2024-06-04T05:23:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.104+0000 7f91a2d2b640 1 -- 172.21.15.70:0/2404086064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f919c19e450 con 0x7f919c102340 2024-06-04T05:23:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.104+0000 7f9199ffb640 1 -- 172.21.15.70:0/2404086064 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91900116d0 con 0x7f919c102340 2024-06-04T05:23:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.105+0000 7f91a12c8640 1 --2- 172.21.15.70:0/2404086064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f919c107940 0x7f919c19da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.105+0000 7f91a2d2b640 1 -- 172.21.15.70:0/2404086064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f919c1a5ce0 con 0x7f919c102340 2024-06-04T05:23:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.109+0000 7f9199ffb640 1 -- 172.21.15.70:0/2404086064 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f9190004a40 con 0x7f919c102340 2024-06-04T05:23:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.109+0000 7f91a5a49740 1 Processor -- start 2024-06-04T05:23:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.109+0000 7f91a5a49740 1 -- start start 2024-06-04T05:23:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.109+0000 7f91a5a49740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79b820 0x55f0dd79bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.109+0000 7f91a5a49740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd79c140 0x55f0dd7a64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.109+0000 7f91a5a49740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd7a6a00 0x55f0dd7a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.109+0000 7f91a5a49740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f0dd70dc30 con 0x55f0dd79c140 2024-06-04T05:23:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.109+0000 7f91a5a49740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f0dd6fd7d0 con 0x55f0dd79b820 2024-06-04T05:23:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.109+0000 7f91a5a49740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f0dd701810 con 0x55f0dd7a6a00 2024-06-04T05:23:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.110+0000 7f91a12c8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79b820 0x55f0dd79bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.110+0000 7f919bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd7a6a00 0x55f0dd7a8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.110+0000 7f91a12c8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79b820 0x55f0dd79bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55814/0 (socket says 172.21.15.70:55814) 2024-06-04T05:23:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.110+0000 7f91a12c8640 1 -- 172.21.15.70:0/4025100082 learned_addr learned my addr 172.21.15.70:0/4025100082 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.110+0000 7f91a0ac7640 1 --2- 172.21.15.70:0/4025100082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd79c140 0x55f0dd7a64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.110+0000 7f91a12c8640 1 -- 172.21.15.70:0/4025100082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd7a6a00 msgr2=0x55f0dd7a8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.110+0000 7f91a12c8640 1 --2- 172.21.15.70:0/4025100082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd7a6a00 0x55f0dd7a8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.110+0000 7f91a12c8640 1 -- 172.21.15.70:0/4025100082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd79c140 msgr2=0x55f0dd7a64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.110+0000 7f91a12c8640 1 --2- 172.21.15.70:0/4025100082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd79c140 0x55f0dd7a64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.110+0000 7f91a12c8640 1 -- 172.21.15.70:0/4025100082 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0dd412210 con 0x55f0dd79b820 2024-06-04T05:23:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.110+0000 7f91a12c8640 1 --2- 172.21.15.70:0/4025100082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79b820 0x55f0dd79bc00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f919400bca0 tx=0x7f919400f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18e80987eff3b483 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.110+0000 7f9199ffb640 1 --2- 172.21.15.70:0/2404086064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f91840b2a90 0x7f91840b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.110+0000 7f9182ffd640 1 -- 172.21.15.70:0/4025100082 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9194015070 con 0x55f0dd79b820 2024-06-04T05:23:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.111+0000 7f9182ffd640 1 -- 172.21.15.70:0/4025100082 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f91940054b0 con 0x55f0dd79b820 2024-06-04T05:23:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.111+0000 7f9182ffd640 1 -- 172.21.15.70:0/4025100082 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9194011440 con 0x55f0dd79b820 2024-06-04T05:23:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.111+0000 7f91a0ac7640 1 --2- 172.21.15.70:0/2404086064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f91840b2a90 0x7f91840b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.111+0000 7f9199ffb640 1 -- 172.21.15.70:0/2404086064 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8615+0+0 (secure 0 0 0) 0x7f91900e2600 con 0x7f919c102340 2024-06-04T05:23:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.111+0000 7f91a5a49740 1 -- 172.21.15.70:0/4025100082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79b820 msgr2=0x55f0dd79bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.111+0000 7f91a0ac7640 1 --2- 172.21.15.70:0/2404086064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f91840b2a90 0x7f91840b4f50 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f918c004770 tx=0x7f918c0058d0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.111+0000 7f91a5a49740 1 --2- 172.21.15.70:0/4025100082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79b820 0x55f0dd79bc00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f919400bca0 tx=0x7f919400f730 comp rx=0 tx=0).stop 2024-06-04T05:23:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.111+0000 7f91a5a49740 1 -- 172.21.15.70:0/4025100082 shutdown_connections 2024-06-04T05:23:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.111+0000 7f91a5a49740 1 --2- 172.21.15.70:0/4025100082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd7a6a00 0x55f0dd7a8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.111+0000 7f91a5a49740 1 --2- 172.21.15.70:0/4025100082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd79c140 0x55f0dd7a64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.111+0000 7f91a5a49740 1 --2- 172.21.15.70:0/4025100082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79b820 0x55f0dd79bc00 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.111+0000 7f91a5a49740 1 -- 172.21.15.70:0/4025100082 >> 172.21.15.70:0/4025100082 conn(0x55f0dd797d30 msgr2=0x55f0dd798110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.112+0000 7f91a5a49740 1 -- 172.21.15.70:0/4025100082 shutdown_connections 2024-06-04T05:23:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.112+0000 7f91a5a49740 1 -- 172.21.15.70:0/4025100082 wait complete. 2024-06-04T05:23:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.112+0000 7f91a5a49740 1 -- start start 2024-06-04T05:23:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.112+0000 7f91a5a49740 1 Processor -- start 2024-06-04T05:23:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.112+0000 7f91a5a49740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79b820 0x55f0dd7a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.112+0000 7f91a5a49740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd79c140 0x55f0dd7a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.112+0000 7f91a5a49740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd7a6a00 0x55f0dd7a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.112+0000 7f91a5a49740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f0dd3e3790 con 0x55f0dd79c140 2024-06-04T05:23:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.112+0000 7f91a5a49740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f0dd6b9890 con 0x55f0dd79b820 2024-06-04T05:23:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.112+0000 7f91a5a49740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f0dd7a9630 con 0x55f0dd7a6a00 2024-06-04T05:23:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.112+0000 7f91a5a49740 10 client.0 _ll_register_callbacks cb 0x55f0dd411a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.112+0000 7f919bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd79c140 0x55f0dd7a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.112+0000 7f91a0ac7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79b820 0x55f0dd7a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.113+0000 7f91a0ac7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79b820 0x55f0dd7a3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55816/0 (socket says 172.21.15.70:55816) 2024-06-04T05:23:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.113+0000 7f91a0ac7640 1 -- 172.21.15.70:0/557488467 learned_addr learned my addr 172.21.15.70:0/557488467 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.113+0000 7f91a12c8640 1 --2- 172.21.15.70:0/557488467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd7a6a00 0x55f0dd7a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.113+0000 7f919bfff640 1 -- 172.21.15.70:0/557488467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd7a6a00 msgr2=0x55f0dd7a44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.113+0000 7f919bfff640 1 --2- 172.21.15.70:0/557488467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd7a6a00 0x55f0dd7a44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.113+0000 7f919bfff640 1 -- 172.21.15.70:0/557488467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79b820 msgr2=0x55f0dd7a3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.113+0000 7f919bfff640 1 --2- 172.21.15.70:0/557488467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79b820 0x55f0dd7a3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.113+0000 7f919bfff640 1 -- 172.21.15.70:0/557488467 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0dd4de620 con 0x55f0dd79c140 2024-06-04T05:23:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.113+0000 7f919bfff640 1 --2- 172.21.15.70:0/557488467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd79c140 0x55f0dd7a3fb0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f91900dfc90 tx=0x7f9190005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=345c3b46d4ff3088 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.113+0000 7f917bfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.113+0000 7f917bfff640 1 -- 172.21.15.70:0/557488467 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919001f030 con 0x55f0dd79c140 2024-06-04T05:23:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.113+0000 7f917bfff640 1 -- 172.21.15.70:0/557488467 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8615+0+0 (secure 0 0 0) 0x7f9190004d30 con 0x55f0dd79c140 2024-06-04T05:23:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.113+0000 7f91a5a49740 1 -- 172.21.15.70:0/557488467 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91940054b0 con 0x55f0dd79c140 2024-06-04T05:23:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.113+0000 7f917bfff640 1 -- 172.21.15.70:0/557488467 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f91900a95f0 con 0x55f0dd79c140 2024-06-04T05:23:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.113+0000 7f91a5a49740 10 client.21987 fetch_fsmap learned FSMap version 68 2024-06-04T05:23:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.113+0000 7f91a5a49740 1 -- 172.21.15.70:0/557488467 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) -- 0x55f0dd7b5460 con 0x55f0dd79c140 2024-06-04T05:23:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.113+0000 7f917bfff640 1 -- 172.21.15.70:0/557488467 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f91940054b0 con 0x55f0dd79c140 2024-06-04T05:23:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.114+0000 7f917bfff640 1 -- 172.21.15.70:0/557488467 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 68) ==== 8366+0+0 (secure 0 0 0) 0x7f9190022060 con 0x55f0dd79c140 2024-06-04T05:23:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.114+0000 7f91a5a49740 10 client.21987 fetch_fsmap finished waiting for FSMap version 68 2024-06-04T05:23:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.114+0000 7f91a5a49740 10 client.21987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:23:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.114+0000 7f91a5a49740 20 client.21987 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.114+0000 7f91a5a49740 1 --2- 172.21.15.70:0/557488467 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f0dd7b68d0 0x55f0dd7b8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.114+0000 7f91a5a49740 4 client.21987 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:23:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.114+0000 7f91a5a49740 1 -- 172.21.15.70:0/557488467 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0dd70fb80 con 0x55f0dd7b68d0 2024-06-04T05:23:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.114+0000 7f91a0ac7640 1 --2- 172.21.15.70:0/557488467 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f0dd7b68d0 0x55f0dd7b8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.114+0000 7f917bfff640 10 client.21987 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.114+0000 7f91a0ac7640 1 --2- 172.21.15.70:0/557488467 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f0dd7b68d0 0x55f0dd7b8cc0 crc :-1 s=READY pgs=535 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8476b10106e500b5 server_cookie=17cf5eafff65c8f3 in_seq=0 out_seq=0 2024-06-04T05:23:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.115+0000 7f917bfff640 1 -- 172.21.15.70:0/557488467 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f0dd70fb80 con 0x55f0dd7b68d0 2024-06-04T05:23:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.115+0000 7f917bfff640 10 client.21987 handle_command_reply: tid=0 2024-06-04T05:23:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.115+0000 7f91a5a49740 1 client.21987 shutdown 2024-06-04T05:23:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.115+0000 7f91a5a49740 1 -- 172.21.15.70:0/557488467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd79c140 msgr2=0x55f0dd7a3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.115+0000 7f91a5a49740 1 --2- 172.21.15.70:0/557488467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd79c140 0x55f0dd7a3fb0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f91900dfc90 tx=0x7f9190005700 comp rx=0 tx=0).stop 2024-06-04T05:23:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.115+0000 7f91a5a49740 1 -- 172.21.15.70:0/557488467 shutdown_connections 2024-06-04T05:23:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.115+0000 7f91a5a49740 1 --2- 172.21.15.70:0/557488467 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f0dd7b68d0 0x55f0dd7b8cc0 crc :-1 s=READY pgs=535 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.115+0000 7f91a5a49740 1 --2- 172.21.15.70:0/557488467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd7a6a00 0x55f0dd7a44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.115+0000 7f917bfff640 0 client.21987 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.115+0000 7f91a5a49740 1 --2- 172.21.15.70:0/557488467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd79c140 0x55f0dd7a3fb0 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.115+0000 7f91a5a49740 1 --2- 172.21.15.70:0/557488467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79b820 0x55f0dd7a3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.115+0000 7f91a5a49740 1 -- 172.21.15.70:0/557488467 >> 172.21.15.70:0/557488467 conn(0x55f0dd797d30 msgr2=0x55f0dd796fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.115+0000 7f91a5a49740 1 -- 172.21.15.70:0/557488467 shutdown_connections 2024-06-04T05:23:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.115+0000 7f91a5a49740 1 -- 172.21.15.70:0/557488467 wait complete. 2024-06-04T05:23:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.115+0000 7f91a5a49740 20 client.21987 trim_cache size 0 max 16384 2024-06-04T05:23:12.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.132+0000 7f91a5a49740 1 Processor -- start 2024-06-04T05:23:12.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.132+0000 7f91a5a49740 1 -- start start 2024-06-04T05:23:12.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.132+0000 7f91a5a49740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd79b820 0x55f0dd7a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.132+0000 7f91a5a49740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd79c140 0x55f0dd79e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.132+0000 7f91a5a49740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd7a6a00 0x55f0dd7b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.132+0000 7f91a5a49740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f919c1ae710 con 0x55f0dd79c140 2024-06-04T05:23:12.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.132+0000 7f91a5a49740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f919c1b02e0 con 0x55f0dd7a6a00 2024-06-04T05:23:12.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.132+0000 7f91a5a49740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f919c1aee70 con 0x55f0dd79b820 2024-06-04T05:23:12.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.132+0000 7f919bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd79c140 0x55f0dd79e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.132+0000 7f91a0ac7640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd79b820 0x55f0dd7a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.132+0000 7f91a0ac7640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd79b820 0x55f0dd7a42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:43096/0 (socket says 172.21.15.70:43096) 2024-06-04T05:23:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.132+0000 7f91a0ac7640 1 -- 172.21.15.70:0/464585550 learned_addr learned my addr 172.21.15.70:0/464585550 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.132+0000 7f91a12c8640 1 --2- 172.21.15.70:0/464585550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd7a6a00 0x55f0dd7b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.133+0000 7f919bfff640 1 -- 172.21.15.70:0/464585550 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd79b820 msgr2=0x55f0dd7a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.133+0000 7f919bfff640 1 --2- 172.21.15.70:0/464585550 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd79b820 0x55f0dd7a42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.133+0000 7f919bfff640 1 -- 172.21.15.70:0/464585550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd7a6a00 msgr2=0x55f0dd7b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.133+0000 7f919bfff640 1 --2- 172.21.15.70:0/464585550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd7a6a00 0x55f0dd7b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.133+0000 7f919bfff640 1 -- 172.21.15.70:0/464585550 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0dd7a1fc0 con 0x55f0dd79c140 2024-06-04T05:23:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.133+0000 7f919bfff640 1 --2- 172.21.15.70:0/464585550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd79c140 0x55f0dd79e230 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f91900a6a90 tx=0x7f91900dfcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b09b055c62af04 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.133+0000 7f91827fc640 1 -- 172.21.15.70:0/464585550 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919001f070 con 0x55f0dd79c140 2024-06-04T05:23:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.133+0000 7f91827fc640 1 -- 172.21.15.70:0/464585550 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f91900a9330 con 0x55f0dd79c140 2024-06-04T05:23:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.133+0000 7f91827fc640 1 -- 172.21.15.70:0/464585550 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91900a8f90 con 0x55f0dd79c140 2024-06-04T05:23:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.134+0000 7f91a5a49740 1 -- 172.21.15.70:0/464585550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd79c140 msgr2=0x55f0dd79e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.134+0000 7f91a5a49740 1 --2- 172.21.15.70:0/464585550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd79c140 0x55f0dd79e230 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f91900a6a90 tx=0x7f91900dfcc0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.134+0000 7f91a5a49740 1 -- 172.21.15.70:0/464585550 shutdown_connections 2024-06-04T05:23:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.134+0000 7f91a5a49740 1 --2- 172.21.15.70:0/464585550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd7a6a00 0x55f0dd7b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.134+0000 7f91a5a49740 1 --2- 172.21.15.70:0/464585550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd79c140 0x55f0dd79e230 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.134+0000 7f91a5a49740 1 --2- 172.21.15.70:0/464585550 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd79b820 0x55f0dd7a42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.134+0000 7f91a5a49740 1 -- 172.21.15.70:0/464585550 >> 172.21.15.70:0/464585550 conn(0x55f0dd797d30 msgr2=0x55f0dd7b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.134+0000 7f91a5a49740 1 -- 172.21.15.70:0/464585550 shutdown_connections 2024-06-04T05:23:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.134+0000 7f91a5a49740 1 -- 172.21.15.70:0/464585550 wait complete. 2024-06-04T05:23:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.134+0000 7f91a5a49740 1 -- start start 2024-06-04T05:23:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.134+0000 7f91a5a49740 1 Processor -- start 2024-06-04T05:23:12.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a5a49740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd79b820 0x55f0dd7aa920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a5a49740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79c140 0x55f0dd7aae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a5a49740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd7a6a00 0x55f0dd820460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a5a49740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f919c1afc10 con 0x55f0dd7a6a00 2024-06-04T05:23:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a5a49740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f919c1af540 con 0x55f0dd79c140 2024-06-04T05:23:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a5a49740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f919c1ac710 con 0x55f0dd79b820 2024-06-04T05:23:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a5a49740 10 client.0 _ll_register_callbacks cb 0x7f91900b0f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a12c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd7a6a00 0x55f0dd820460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a12c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd7a6a00 0x55f0dd820460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59880/0 (socket says 172.21.15.70:59880) 2024-06-04T05:23:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a12c8640 1 -- 172.21.15.70:0/1468226154 learned_addr learned my addr 172.21.15.70:0/1468226154 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a0ac7640 1 --2- 172.21.15.70:0/1468226154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd79b820 0x55f0dd7aa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f919bfff640 1 --2- 172.21.15.70:0/1468226154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79c140 0x55f0dd7aae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a12c8640 1 -- 172.21.15.70:0/1468226154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd79b820 msgr2=0x55f0dd7aa920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a12c8640 1 --2- 172.21.15.70:0/1468226154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd79b820 0x55f0dd7aa920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a12c8640 1 -- 172.21.15.70:0/1468226154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79c140 msgr2=0x55f0dd7aae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a12c8640 1 --2- 172.21.15.70:0/1468226154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79c140 0x55f0dd7aae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a12c8640 1 -- 172.21.15.70:0/1468226154 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0dd7ab730 con 0x55f0dd7a6a00 2024-06-04T05:23:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a12c8640 1 --2- 172.21.15.70:0/1468226154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd7a6a00 0x55f0dd820460 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f919400bbf0 tx=0x7f919400bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa0cc724d19d5252 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f917bfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:12.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f917bfff640 1 -- 172.21.15.70:0/1468226154 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9194029060 con 0x55f0dd7a6a00 2024-06-04T05:23:12.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f917bfff640 1 -- 172.21.15.70:0/1468226154 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8615+0+0 (secure 0 0 0) 0x7f9194005240 con 0x55f0dd7a6a00 2024-06-04T05:23:12.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a5a49740 1 -- 172.21.15.70:0/1468226154 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91900a9330 con 0x55f0dd7a6a00 2024-06-04T05:23:12.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f917bfff640 1 -- 172.21.15.70:0/1468226154 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f9194015050 con 0x55f0dd7a6a00 2024-06-04T05:23:12.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a5a49740 10 client.21993 fetch_fsmap learned FSMap version 68 2024-06-04T05:23:12.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.135+0000 7f91a5a49740 1 -- 172.21.15.70:0/1468226154 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) -- 0x55f0dd821ab0 con 0x55f0dd7a6a00 2024-06-04T05:23:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.136+0000 7f917bfff640 1 -- 172.21.15.70:0/1468226154 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f91900a9330 con 0x55f0dd7a6a00 2024-06-04T05:23:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.136+0000 7f917bfff640 1 -- 172.21.15.70:0/1468226154 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 68) ==== 8366+0+0 (secure 0 0 0) 0x7f9194023070 con 0x55f0dd7a6a00 2024-06-04T05:23:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.136+0000 7f91a5a49740 10 client.21993 fetch_fsmap finished waiting for FSMap version 68 2024-06-04T05:23:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.136+0000 7f91a5a49740 10 client.21993 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:23:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.136+0000 7f91a5a49740 20 client.21993 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.136+0000 7f91a5a49740 1 --2- 172.21.15.70:0/1468226154 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f0dd821ea0 0x55f0dd824290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.136+0000 7f91a5a49740 4 client.21993 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:23:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.136+0000 7f91a5a49740 1 -- 172.21.15.70:0/1468226154 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f0dd709360 con 0x55f0dd821ea0 2024-06-04T05:23:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.136+0000 7f91a0ac7640 1 --2- 172.21.15.70:0/1468226154 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f0dd821ea0 0x55f0dd824290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.136+0000 7f917bfff640 10 client.21993 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.136+0000 7f91a0ac7640 1 --2- 172.21.15.70:0/1468226154 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f0dd821ea0 0x55f0dd824290 crc :-1 s=READY pgs=536 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e54398b9dcb4cf6c server_cookie=748f0b3de41bb9ba in_seq=0 out_seq=0 2024-06-04T05:23:12.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.240+0000 7f917bfff640 1 -- 172.21.15.70:0/1468226154 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f0dd709360 con 0x55f0dd821ea0 2024-06-04T05:23:12.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.240+0000 7f917bfff640 10 client.21993 handle_command_reply: tid=0 2024-06-04T05:23:12.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.240+0000 7f91a5a49740 1 client.21993 shutdown 2024-06-04T05:23:12.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.240+0000 7f91a5a49740 1 -- 172.21.15.70:0/1468226154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd7a6a00 msgr2=0x55f0dd820460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.240+0000 7f91a5a49740 1 --2- 172.21.15.70:0/1468226154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd7a6a00 0x55f0dd820460 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f919400bbf0 tx=0x7f919400bfc0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.240+0000 7f91a5a49740 1 -- 172.21.15.70:0/1468226154 shutdown_connections 2024-06-04T05:23:12.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.240+0000 7f91a5a49740 1 --2- 172.21.15.70:0/1468226154 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55f0dd821ea0 0x55f0dd824290 crc :-1 s=READY pgs=536 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.240+0000 7f91a5a49740 1 --2- 172.21.15.70:0/1468226154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f0dd79c140 0x55f0dd7aae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.240+0000 7f917bfff640 0 client.21993 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:12.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.240+0000 7f91a5a49740 1 --2- 172.21.15.70:0/1468226154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0dd7a6a00 0x55f0dd820460 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.240+0000 7f91a5a49740 1 --2- 172.21.15.70:0/1468226154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f0dd79b820 0x55f0dd7aa920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.240+0000 7f91a5a49740 1 -- 172.21.15.70:0/1468226154 >> 172.21.15.70:0/1468226154 conn(0x55f0dd797d30 msgr2=0x55f0dd7a4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.240+0000 7f91a5a49740 1 -- 172.21.15.70:0/1468226154 shutdown_connections 2024-06-04T05:23:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.240+0000 7f91a5a49740 1 -- 172.21.15.70:0/1468226154 wait complete. 2024-06-04T05:23:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.241+0000 7f91a5a49740 20 client.21993 trim_cache size 0 max 16384 2024-06-04T05:23:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:23:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:23:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:23:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.241+0000 7f91a2d2b640 1 -- 172.21.15.70:0/2404086064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f91840b2a90 msgr2=0x7f91840b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.241+0000 7f91a2d2b640 1 --2- 172.21.15.70:0/2404086064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f91840b2a90 0x7f91840b4f50 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f918c004770 tx=0x7f918c0058d0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.241+0000 7f91a2d2b640 1 -- 172.21.15.70:0/2404086064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f919c102340 msgr2=0x7f919c199700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.241+0000 7f91a2d2b640 1 --2- 172.21.15.70:0/2404086064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f919c102340 0x7f919c199700 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f919000ea80 tx=0x7f919000ef40 comp rx=0 tx=0).stop 2024-06-04T05:23:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.241+0000 7f91a2d2b640 1 -- 172.21.15.70:0/2404086064 shutdown_connections 2024-06-04T05:23:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.241+0000 7f91a2d2b640 1 --2- 172.21.15.70:0/2404086064 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f91840b2a90 0x7f91840b4f50 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.241+0000 7f91a2d2b640 1 --2- 172.21.15.70:0/2404086064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f919c107940 0x7f919c19da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.241+0000 7f91a2d2b640 1 --2- 172.21.15.70:0/2404086064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f919c102340 0x7f919c199700 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.241+0000 7f91a2d2b640 1 --2- 172.21.15.70:0/2404086064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f919c101980 0x7f919c1991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.241+0000 7f91a2d2b640 1 -- 172.21.15.70:0/2404086064 >> 172.21.15.70:0/2404086064 conn(0x7f919c078ad0 msgr2=0x7f919c06a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:12.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.241+0000 7f91a2d2b640 1 -- 172.21.15.70:0/2404086064 shutdown_connections 2024-06-04T05:23:12.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:12.241+0000 7f91a2d2b640 1 -- 172.21.15.70:0/2404086064 wait complete. 2024-06-04T05:23:12.428 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-06-04T05:23:12.428 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-06-04T05:23:12.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.528+0000 7f6f0437e700 1 -- 172.21.15.70:0/3916936926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6efc104200 msgr2=0x7f6efc1081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.528+0000 7f6f0437e700 1 --2- 172.21.15.70:0/3916936926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6efc104200 0x7f6efc1081f0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f6ef80099c0 tx=0x7f6ef8009cd0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.528+0000 7f6f0437e700 1 -- 172.21.15.70:0/3916936926 shutdown_connections 2024-06-04T05:23:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.528+0000 7f6f0437e700 1 --2- 172.21.15.70:0/3916936926 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6efc108920 0x7f6efc1104a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.528+0000 7f6f0437e700 1 --2- 172.21.15.70:0/3916936926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6efc103850 0x7f6efc103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.528+0000 7f6f0437e700 1 --2- 172.21.15.70:0/3916936926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6efc104200 0x7f6efc1081f0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.528+0000 7f6f0437e700 1 -- 172.21.15.70:0/3916936926 >> 172.21.15.70:0/3916936926 conn(0x7f6efc07a620 msgr2=0x7f6efc07aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.528+0000 7f6f0437e700 1 -- 172.21.15.70:0/3916936926 shutdown_connections 2024-06-04T05:23:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.528+0000 7f6f0437e700 1 -- 172.21.15.70:0/3916936926 wait complete. 2024-06-04T05:23:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.529+0000 7f6f0437e700 1 Processor -- start 2024-06-04T05:23:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.529+0000 7f6f0437e700 1 -- start start 2024-06-04T05:23:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.529+0000 7f6f0437e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6efc103850 0x7f6efc19f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.529+0000 7f6f0437e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6efc104200 0x7f6efc19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.529+0000 7f6f0437e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6efc108920 0x7f6efc199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.529+0000 7f6f0437e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc1135b0 con 0x7f6efc104200 2024-06-04T05:23:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.529+0000 7f6f0437e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc113430 con 0x7f6efc108920 2024-06-04T05:23:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.529+0000 7f6f0437e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc113730 con 0x7f6efc103850 2024-06-04T05:23:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.529+0000 7f6f01919700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6efc104200 0x7f6efc19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.529+0000 7f6f01919700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6efc104200 0x7f6efc19fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59898/0 (socket says 172.21.15.70:59898) 2024-06-04T05:23:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.529+0000 7f6f01919700 1 -- 172.21.15.70:0/4172583165 learned_addr learned my addr 172.21.15.70:0/4172583165 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.529+0000 7f6f0211a700 1 --2- 172.21.15.70:0/4172583165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6efc103850 0x7f6efc19f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.529+0000 7f6f0291b700 1 --2- 172.21.15.70:0/4172583165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6efc108920 0x7f6efc199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.529+0000 7f6f01919700 1 -- 172.21.15.70:0/4172583165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6efc103850 msgr2=0x7f6efc19f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.530+0000 7f6f01919700 1 --2- 172.21.15.70:0/4172583165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6efc103850 0x7f6efc19f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.530+0000 7f6f01919700 1 -- 172.21.15.70:0/4172583165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6efc108920 msgr2=0x7f6efc199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.530+0000 7f6f01919700 1 --2- 172.21.15.70:0/4172583165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6efc108920 0x7f6efc199890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.530+0000 7f6f01919700 1 -- 172.21.15.70:0/4172583165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6efc199ff0 con 0x7f6efc104200 2024-06-04T05:23:12.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.530+0000 7f6f0211a700 1 --2- 172.21.15.70:0/4172583165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6efc103850 0x7f6efc19f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:23:12.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.530+0000 7f6f01919700 1 --2- 172.21.15.70:0/4172583165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6efc104200 0x7f6efc19fc10 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8006010 tx=0x7f6ef8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:12.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.530+0000 7f6eef7fe700 1 -- 172.21.15.70:0/4172583165 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8026070 con 0x7f6efc104200 2024-06-04T05:23:12.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.530+0000 7f6f0437e700 1 -- 172.21.15.70:0/4172583165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6efc19a280 con 0x7f6efc104200 2024-06-04T05:23:12.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.530+0000 7f6eef7fe700 1 -- 172.21.15.70:0/4172583165 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ef8004630 con 0x7f6efc104200 2024-06-04T05:23:12.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.530+0000 7f6eef7fe700 1 -- 172.21.15.70:0/4172583165 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef800faa0 con 0x7f6efc104200 2024-06-04T05:23:12.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.530+0000 7f6f0437e700 1 -- 172.21.15.70:0/4172583165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6efc1a65a0 con 0x7f6efc104200 2024-06-04T05:23:12.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.533+0000 7f6eef7fe700 1 -- 172.21.15.70:0/4172583165 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f6ef8029070 con 0x7f6efc104200 2024-06-04T05:23:12.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.534+0000 7f6f0ec13b80 1 Processor -- start 2024-06-04T05:23:12.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.534+0000 7f6f0ec13b80 1 -- start start 2024-06-04T05:23:12.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.534+0000 7f6f0ec13b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb0766860 0x55cdb0766c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.534+0000 7f6f0ec13b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb0767210 0x55cdb076a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.534+0000 7f6f0ec13b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb076af60 0x55cdb076b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.534+0000 7f6f0ec13b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cdb06c5c90 con 0x55cdb076af60 2024-06-04T05:23:12.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.534+0000 7f6f0ec13b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cdb06b5830 con 0x55cdb0767210 2024-06-04T05:23:12.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.534+0000 7f6f0ec13b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cdb06b9870 con 0x55cdb0766860 2024-06-04T05:23:12.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.534+0000 7f6f0291b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb0766860 0x55cdb0766c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.534+0000 7f6f01919700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb076af60 0x55cdb076b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.534+0000 7f6f0291b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb0766860 0x55cdb0766c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:43134/0 (socket says 172.21.15.70:43134) 2024-06-04T05:23:12.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.534+0000 7f6f0291b700 1 -- 172.21.15.70:0/1690756313 learned_addr learned my addr 172.21.15.70:0/1690756313 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:12.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.534+0000 7f6eef7fe700 1 --2- 172.21.15.70:0/4172583165 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ee80b29f0 0x7f6ee80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.534+0000 7f6f0211a700 1 --2- 172.21.15.70:0/1690756313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb0767210 0x55cdb076a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.535+0000 7f6f0291b700 1 -- 172.21.15.70:0/1690756313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb0767210 msgr2=0x55cdb076a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:12.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.535+0000 7f6f0291b700 1 --2- 172.21.15.70:0/1690756313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb0767210 0x55cdb076a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.535+0000 7f6f0291b700 1 -- 172.21.15.70:0/1690756313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb076af60 msgr2=0x55cdb076b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:12.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.535+0000 7f6f0291b700 1 --2- 172.21.15.70:0/1690756313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb076af60 0x55cdb076b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.535+0000 7f6f0291b700 1 -- 172.21.15.70:0/1690756313 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdb06058d0 con 0x55cdb0766860 2024-06-04T05:23:12.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.535+0000 7f6f0211a700 1 --2- 172.21.15.70:0/1690756313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb0767210 0x55cdb076a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:23:12.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.535+0000 7f6f0291b700 1 --2- 172.21.15.70:0/4172583165 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ee80b29f0 0x7f6ee80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.535+0000 7f6eef7fe700 1 -- 172.21.15.70:0/4172583165 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8615+0+0 (secure 0 0 0) 0x7f6ef80e2d80 con 0x7f6efc104200 2024-06-04T05:23:12.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.535+0000 7f6f0291b700 1 --2- 172.21.15.70:0/1690756313 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb0766860 0x55cdb0766c40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4011790 tx=0x7f6ef4011aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5168a52414faee server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:12.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.535+0000 7f6f0291b700 1 --2- 172.21.15.70:0/4172583165 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ee80b29f0 0x7f6ee80b4eb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4022000 tx=0x7f6ef401f000 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:12.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.535+0000 7f6eecd70700 1 -- 172.21.15.70:0/1690756313 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef401c040 con 0x55cdb0766860 2024-06-04T05:23:12.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.535+0000 7f6eecd70700 1 -- 172.21.15.70:0/1690756313 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ef4018b10 con 0x55cdb0766860 2024-06-04T05:23:12.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.535+0000 7f6eecd70700 1 -- 172.21.15.70:0/1690756313 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4023980 con 0x55cdb0766860 2024-06-04T05:23:12.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.536+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/1690756313 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb0766860 msgr2=0x55cdb0766c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.536+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/1690756313 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb0766860 0x55cdb0766c40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4011790 tx=0x7f6ef4011aa0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.536+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/1690756313 shutdown_connections 2024-06-04T05:23:12.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.536+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/1690756313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb076af60 0x55cdb076b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.536+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/1690756313 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb0766860 0x55cdb0766c40 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.536+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/1690756313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb0767210 0x55cdb076a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.536+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/1690756313 >> 172.21.15.70:0/1690756313 conn(0x55cdb0694080 msgr2=0x55cdb0760e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:12.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.536+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/1690756313 shutdown_connections 2024-06-04T05:23:12.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.536+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/1690756313 wait complete. 2024-06-04T05:23:12.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.536+0000 7f6f0ec13b80 1 -- start start 2024-06-04T05:23:12.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.536+0000 7f6f0ec13b80 1 Processor -- start 2024-06-04T05:23:12.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0ec13b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb0766860 0x55cdb07723e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0ec13b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb0767210 0x55cdb076f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0ec13b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb076af60 0x55cdb076fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0ec13b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cdb05bbfa0 con 0x55cdb0766860 2024-06-04T05:23:12.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0ec13b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cdb06577c0 con 0x55cdb076af60 2024-06-04T05:23:12.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0ec13b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cdb0773120 con 0x55cdb0767210 2024-06-04T05:23:12.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0ec13b80 10 client.0 _ll_register_callbacks cb 0x55cdb0650920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:12.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0211a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb0766860 0x55cdb07723e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0211a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb0766860 0x55cdb07723e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59910/0 (socket says 172.21.15.70:59910) 2024-06-04T05:23:12.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0211a700 1 -- 172.21.15.70:0/3897215229 learned_addr learned my addr 172.21.15.70:0/3897215229 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:12.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f01919700 1 --2- 172.21.15.70:0/3897215229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb0767210 0x55cdb076f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0291b700 1 --2- 172.21.15.70:0/3897215229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb076af60 0x55cdb076fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0211a700 1 -- 172.21.15.70:0/3897215229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb0767210 msgr2=0x55cdb076f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:12.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0211a700 1 --2- 172.21.15.70:0/3897215229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb0767210 0x55cdb076f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0211a700 1 -- 172.21.15.70:0/3897215229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb076af60 msgr2=0x55cdb076fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:12.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0211a700 1 --2- 172.21.15.70:0/3897215229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb076af60 0x55cdb076fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0211a700 1 -- 172.21.15.70:0/3897215229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdb0654690 con 0x55cdb0766860 2024-06-04T05:23:12.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0211a700 1 --2- 172.21.15.70:0/3897215229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb0766860 0x55cdb07723e0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0012d20 tx=0x7f6ef00106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5687b2b673ae6b9c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:12.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6ee5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:12.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6ee5ffb700 1 -- 172.21.15.70:0/3897215229 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0012e60 con 0x55cdb0766860 2024-06-04T05:23:12.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6ee5ffb700 1 -- 172.21.15.70:0/3897215229 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8615+0+0 (secure 0 0 0) 0x7f6ef0004030 con 0x55cdb0766860 2024-06-04T05:23:12.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.537+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/3897215229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ef4018b10 con 0x55cdb0766860 2024-06-04T05:23:12.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.538+0000 7f6ee5ffb700 1 -- 172.21.15.70:0/3897215229 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) ==== 24+0+0 (secure 0 0 0) 0x7f6ef0004230 con 0x55cdb0766860 2024-06-04T05:23:12.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.538+0000 7f6f0ec13b80 10 client.22005 fetch_fsmap learned FSMap version 69 2024-06-04T05:23:12.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.538+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/3897215229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=4+}) -- 0x55cdb077d820 con 0x55cdb0766860 2024-06-04T05:23:12.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.538+0000 7f6ee5ffb700 1 -- 172.21.15.70:0/3897215229 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ef4018b10 con 0x55cdb0766860 2024-06-04T05:23:12.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.538+0000 7f6ee5ffb700 1 -- 172.21.15.70:0/3897215229 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 69) ==== 8374+0+0 (secure 0 0 0) 0x7f6ef001d650 con 0x55cdb0766860 2024-06-04T05:23:12.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.538+0000 7f6f0ec13b80 10 client.22005 fetch_fsmap finished waiting for FSMap version 69 2024-06-04T05:23:12.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.538+0000 7f6f0ec13b80 10 client.22005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:23:12.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.538+0000 7f6f0ec13b80 20 client.22005 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:12.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.538+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/3897215229 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cdb077fe60 0x55cdb0782250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.538+0000 7f6f0ec13b80 4 client.22005 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:23:12.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.538+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/3897215229 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cdb07329e0 con 0x55cdb077fe60 2024-06-04T05:23:12.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.538+0000 7f6f01919700 1 --2- 172.21.15.70:0/3897215229 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cdb077fe60 0x55cdb0782250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.539+0000 7f6ee5ffb700 10 client.22005 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:12.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.539+0000 7f6f01919700 1 --2- 172.21.15.70:0/3897215229 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cdb077fe60 0x55cdb0782250 crc :-1 s=READY pgs=537 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8966f43f66c401e server_cookie=bbfde9ffc6267206 in_seq=0 out_seq=0 2024-06-04T05:23:12.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.539+0000 7f6ee5ffb700 1 -- 172.21.15.70:0/3897215229 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cdb07329e0 con 0x55cdb077fe60 2024-06-04T05:23:12.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.539+0000 7f6ee5ffb700 10 client.22005 handle_command_reply: tid=0 2024-06-04T05:23:12.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.539+0000 7f6f0ec13b80 1 client.22005 shutdown 2024-06-04T05:23:12.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.539+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/3897215229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb0766860 msgr2=0x55cdb07723e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.539+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/3897215229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb0766860 0x55cdb07723e0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0012d20 tx=0x7f6ef00106b0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.539+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/3897215229 shutdown_connections 2024-06-04T05:23:12.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.539+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/3897215229 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cdb077fe60 0x55cdb0782250 crc :-1 s=READY pgs=537 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.539+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/3897215229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb076af60 0x55cdb076fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.539+0000 7f6ee5ffb700 0 client.22005 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:12.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.539+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/3897215229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb0766860 0x55cdb07723e0 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.539+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/3897215229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb0767210 0x55cdb076f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.539+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/3897215229 >> 172.21.15.70:0/3897215229 conn(0x55cdb0694080 msgr2=0x55cdb0760510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:12.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.539+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/3897215229 shutdown_connections 2024-06-04T05:23:12.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.539+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/3897215229 wait complete. 2024-06-04T05:23:12.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.540+0000 7f6f0ec13b80 20 client.22005 trim_cache size 0 max 16384 2024-06-04T05:23:12.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0ec13b80 1 Processor -- start 2024-06-04T05:23:12.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0ec13b80 1 -- start start 2024-06-04T05:23:12.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0ec13b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb0766860 0x55cdb076d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0ec13b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb0767210 0x55cdb0769f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0ec13b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb076af60 0x55cdb0760c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0ec13b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc1add50 con 0x55cdb076af60 2024-06-04T05:23:12.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0ec13b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc1af980 con 0x55cdb0766860 2024-06-04T05:23:12.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0ec13b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc1ae4b0 con 0x55cdb0767210 2024-06-04T05:23:12.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0291b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb076af60 0x55cdb0760c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0291b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb076af60 0x55cdb0760c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59918/0 (socket says 172.21.15.70:59918) 2024-06-04T05:23:12.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0291b700 1 -- 172.21.15.70:0/3708903169 learned_addr learned my addr 172.21.15.70:0/3708903169 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:12.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0211a700 1 --2- 172.21.15.70:0/3708903169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb0766860 0x55cdb076d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f01919700 1 --2- 172.21.15.70:0/3708903169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb0767210 0x55cdb0769f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0291b700 1 -- 172.21.15.70:0/3708903169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb0767210 msgr2=0x55cdb0769f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:12.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0291b700 1 --2- 172.21.15.70:0/3708903169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb0767210 0x55cdb0769f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0291b700 1 -- 172.21.15.70:0/3708903169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb0766860 msgr2=0x55cdb076d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:12.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0291b700 1 --2- 172.21.15.70:0/3708903169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb0766860 0x55cdb076d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0291b700 1 -- 172.21.15.70:0/3708903169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdb076dc50 con 0x55cdb076af60 2024-06-04T05:23:12.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.560+0000 7f6f0211a700 1 --2- 172.21.15.70:0/3708903169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb0766860 0x55cdb076d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:12.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.561+0000 7f6f0291b700 1 --2- 172.21.15.70:0/3708903169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb076af60 0x55cdb0760c90 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4000f80 tx=0x7f6ef4009050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15804e76df3deb1b server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:12.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.561+0000 7f6ee7fff700 1 -- 172.21.15.70:0/3708903169 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4030070 con 0x55cdb076af60 2024-06-04T05:23:12.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.561+0000 7f6ee7fff700 1 -- 172.21.15.70:0/3708903169 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ef4018bd0 con 0x55cdb076af60 2024-06-04T05:23:12.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.561+0000 7f6ee7fff700 1 -- 172.21.15.70:0/3708903169 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef401c040 con 0x55cdb076af60 2024-06-04T05:23:12.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/3708903169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb076af60 msgr2=0x55cdb0760c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/3708903169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb076af60 0x55cdb0760c90 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4000f80 tx=0x7f6ef4009050 comp rx=0 tx=0).stop 2024-06-04T05:23:12.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/3708903169 shutdown_connections 2024-06-04T05:23:12.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/3708903169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb076af60 0x55cdb0760c90 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/3708903169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb0766860 0x55cdb076d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/3708903169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb0767210 0x55cdb0769f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/3708903169 >> 172.21.15.70:0/3708903169 conn(0x55cdb0694080 msgr2=0x55cdb0765f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:12.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/3708903169 shutdown_connections 2024-06-04T05:23:12.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/3708903169 wait complete. 2024-06-04T05:23:12.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 1 -- start start 2024-06-04T05:23:12.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 1 Processor -- start 2024-06-04T05:23:12.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb0766860 0x55cdb07787d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb0767210 0x55cdb07e3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb076af60 0x55cdb07e3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc1af2b0 con 0x55cdb0767210 2024-06-04T05:23:12.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc1aeb80 con 0x55cdb0766860 2024-06-04T05:23:12.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc1abd10 con 0x55cdb076af60 2024-06-04T05:23:12.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.562+0000 7f6f0ec13b80 10 client.0 _ll_register_callbacks cb 0x55cdb0650920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:12.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6f01919700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb0767210 0x55cdb07e3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6f01919700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb0767210 0x55cdb07e3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59920/0 (socket says 172.21.15.70:59920) 2024-06-04T05:23:12.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6f01919700 1 -- 172.21.15.70:0/1579707892 learned_addr learned my addr 172.21.15.70:0/1579707892 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:12.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6f0211a700 1 --2- 172.21.15.70:0/1579707892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb0766860 0x55cdb07787d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6f0291b700 1 --2- 172.21.15.70:0/1579707892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb076af60 0x55cdb07e3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6f01919700 1 -- 172.21.15.70:0/1579707892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb076af60 msgr2=0x55cdb07e3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:12.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6f01919700 1 --2- 172.21.15.70:0/1579707892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb076af60 0x55cdb07e3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6f01919700 1 -- 172.21.15.70:0/1579707892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb0766860 msgr2=0x55cdb07787d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:12.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6f01919700 1 --2- 172.21.15.70:0/1579707892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb0766860 0x55cdb07787d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6f01919700 1 -- 172.21.15.70:0/1579707892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdb07e3bd0 con 0x55cdb0767210 2024-06-04T05:23:12.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6f01919700 1 --2- 172.21.15.70:0/1579707892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb0767210 0x55cdb07e3150 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f6ef80a9d90 tx=0x7f6ef8005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16656613dc8c1668 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:12.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6ee5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:12.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6ee5ffb700 1 -- 172.21.15.70:0/1579707892 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef80a99e0 con 0x55cdb0767210 2024-06-04T05:23:12.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6ee5ffb700 1 -- 172.21.15.70:0/1579707892 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8615+0+0 (secure 0 0 0) 0x7f6ef80aa2f0 con 0x55cdb0767210 2024-06-04T05:23:12.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/1579707892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ef4018bd0 con 0x55cdb0767210 2024-06-04T05:23:12.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6ee5ffb700 1 -- 172.21.15.70:0/1579707892 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) ==== 24+0+0 (secure 0 0 0) 0x7f6ef80a9800 con 0x55cdb0767210 2024-06-04T05:23:12.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6f0ec13b80 10 client.22011 fetch_fsmap learned FSMap version 69 2024-06-04T05:23:12.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.563+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/1579707892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=4+}) -- 0x55cdb0777170 con 0x55cdb0767210 2024-06-04T05:23:12.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.564+0000 7f6ee5ffb700 1 -- 172.21.15.70:0/1579707892 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ef4018bd0 con 0x55cdb0767210 2024-06-04T05:23:12.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.564+0000 7f6ee5ffb700 1 -- 172.21.15.70:0/1579707892 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 69) ==== 8374+0+0 (secure 0 0 0) 0x7f6ef80b29d0 con 0x55cdb0767210 2024-06-04T05:23:12.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.564+0000 7f6f0ec13b80 10 client.22011 fetch_fsmap finished waiting for FSMap version 69 2024-06-04T05:23:12.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.564+0000 7f6f0ec13b80 10 client.22011 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:23:12.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.564+0000 7f6f0ec13b80 20 client.22011 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:12.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.564+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/1579707892 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cdb07ef380 0x55cdb07f17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:12.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.564+0000 7f6f0ec13b80 4 client.22011 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:23:12.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.564+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/1579707892 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cdb06c7be0 con 0x55cdb07ef380 2024-06-04T05:23:12.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.564+0000 7f6f0211a700 1 --2- 172.21.15.70:0/1579707892 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cdb07ef380 0x55cdb07f17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:12.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.564+0000 7f6ee5ffb700 10 client.22011 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:12.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.564+0000 7f6f0211a700 1 --2- 172.21.15.70:0/1579707892 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cdb07ef380 0x55cdb07f17a0 crc :-1 s=READY pgs=538 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=597a0e7c52b347de server_cookie=f8259b0af93942da in_seq=0 out_seq=0 2024-06-04T05:23:12.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.565+0000 7f6ee5ffb700 1 -- 172.21.15.70:0/1579707892 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cdb06c7be0 con 0x55cdb07ef380 2024-06-04T05:23:12.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.565+0000 7f6ee5ffb700 10 client.22011 handle_command_reply: tid=0 2024-06-04T05:23:12.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.565+0000 7f6f0ec13b80 1 client.22011 shutdown 2024-06-04T05:23:12.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.565+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/1579707892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb0767210 msgr2=0x55cdb07e3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.565+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/1579707892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb0767210 0x55cdb07e3150 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f6ef80a9d90 tx=0x7f6ef8005d40 comp rx=0 tx=0).stop 2024-06-04T05:23:12.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.565+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/1579707892 shutdown_connections 2024-06-04T05:23:12.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.565+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/1579707892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cdb076af60 0x55cdb07e3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.565+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/1579707892 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55cdb07ef380 0x55cdb07f17a0 crc :-1 s=READY pgs=538 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.565+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/1579707892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdb0766860 0x55cdb07787d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.565+0000 7f6f0ec13b80 1 --2- 172.21.15.70:0/1579707892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdb0767210 0x55cdb07e3150 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.565+0000 7f6ee5ffb700 0 client.22011 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:12.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.565+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/1579707892 >> 172.21.15.70:0/1579707892 conn(0x55cdb0694080 msgr2=0x55cdb0770f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:12.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.565+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/1579707892 shutdown_connections 2024-06-04T05:23:12.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.565+0000 7f6f0ec13b80 1 -- 172.21.15.70:0/1579707892 wait complete. 2024-06-04T05:23:12.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.565+0000 7f6f0ec13b80 20 client.22011 trim_cache size 0 max 16384 2024-06-04T05:23:12.576 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:23:12.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.566+0000 7f6f0437e700 1 -- 172.21.15.70:0/4172583165 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ee80b29f0 msgr2=0x7f6ee80b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.566+0000 7f6f0437e700 1 --2- 172.21.15.70:0/4172583165 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ee80b29f0 0x7f6ee80b4eb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4022000 tx=0x7f6ef401f000 comp rx=0 tx=0).stop 2024-06-04T05:23:12.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.566+0000 7f6f0437e700 1 -- 172.21.15.70:0/4172583165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6efc104200 msgr2=0x7f6efc19fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:12.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.566+0000 7f6f0437e700 1 --2- 172.21.15.70:0/4172583165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6efc104200 0x7f6efc19fc10 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8006010 tx=0x7f6ef8004270 comp rx=0 tx=0).stop 2024-06-04T05:23:12.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.566+0000 7f6f0437e700 1 -- 172.21.15.70:0/4172583165 shutdown_connections 2024-06-04T05:23:12.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.566+0000 7f6f0437e700 1 --2- 172.21.15.70:0/4172583165 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f6ee80b29f0 0x7f6ee80b4eb0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.566+0000 7f6f0437e700 1 --2- 172.21.15.70:0/4172583165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6efc108920 0x7f6efc199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.566+0000 7f6f0437e700 1 --2- 172.21.15.70:0/4172583165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6efc103850 0x7f6efc19f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.566+0000 7f6f0437e700 1 --2- 172.21.15.70:0/4172583165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6efc104200 0x7f6efc19fc10 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:12.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.566+0000 7f6f0437e700 1 -- 172.21.15.70:0/4172583165 >> 172.21.15.70:0/4172583165 conn(0x7f6efc07a620 msgr2=0x7f6efc101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:12.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.566+0000 7f6f0437e700 1 -- 172.21.15.70:0/4172583165 shutdown_connections 2024-06-04T05:23:12.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:12.566+0000 7f6f0437e700 1 -- 172.21.15.70:0/4172583165 wait complete. 2024-06-04T05:23:12.905 DEBUG:teuthology.parallel:result is None 2024-06-04T05:23:12.905 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-06-04T05:23:12.939 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-06-04T05:23:12.940 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-06-04T05:23:12.952 INFO:tasks.quiescer:joining Quiescers 2024-06-04T05:23:12.952 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-06-04T05:23:12.953 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:23:13.571 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:13.721 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:13 smithi070 ceph-mon[32631]: pgmap v821: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.0 MiB/s wr, 221 op/s 2024-06-04T05:23:13.721 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:13 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:23:13.721 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:13 smithi070 ceph-mon[32631]: osdmap e120: 12 total, 12 up, 12 in 2024-06-04T05:23:13.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:13 smithi098 ceph-mon[35569]: pgmap v821: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.0 MiB/s wr, 221 op/s 2024-06-04T05:23:13.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:13 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:23:13.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:13 smithi098 ceph-mon[35569]: osdmap e120: 12 total, 12 up, 12 in 2024-06-04T05:23:13.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:13 smithi112 ceph-mon[36453]: pgmap v821: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.0 MiB/s wr, 221 op/s 2024-06-04T05:23:13.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:13 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:23:13.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:13 smithi112 ceph-mon[36453]: osdmap e120: 12 total, 12 up, 12 in 2024-06-04T05:23:14.493 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-04T05:23:14.493 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-04T05:23:14.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:14 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:23:14.807 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:14 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:23:14.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:14 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:23:15.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:15 smithi098 ceph-mon[35569]: pgmap v823: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.9 MiB/s wr, 199 op/s 2024-06-04T05:23:15.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:15 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:23:15.769 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:15 smithi070 ceph-mon[32631]: pgmap v823: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.9 MiB/s wr, 199 op/s 2024-06-04T05:23:15.769 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:15 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:23:15.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:15 smithi112 ceph-mon[36453]: pgmap v823: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.9 MiB/s wr, 199 op/s 2024-06-04T05:23:15.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:15 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:23:16.550 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:23:16.550 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":71,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:23:14.339988+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24442,24448,24440,24436,24410]},"id":1}]} 2024-06-04T05:23:16.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.504+0000 7f2b79c31700 1 -- 172.21.15.70:0/2031539408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b7406bc70 msgr2=0x7f2b7406c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:16.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.504+0000 7f2b79c31700 1 --2- 172.21.15.70:0/2031539408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b7406bc70 0x7f2b7406c0f0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c009990 tx=0x7f2b5c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:23:16.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.504+0000 7f2b79c31700 1 -- 172.21.15.70:0/2031539408 shutdown_connections 2024-06-04T05:23:16.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.504+0000 7f2b79c31700 1 --2- 172.21.15.70:0/2031539408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b7406c630 0x7f2b741103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:16.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.504+0000 7f2b79c31700 1 --2- 172.21.15.70:0/2031539408 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b741095b0 0x7f2b74109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:16.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.504+0000 7f2b79c31700 1 --2- 172.21.15.70:0/2031539408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b7406bc70 0x7f2b7406c0f0 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:16.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.504+0000 7f2b79c31700 1 -- 172.21.15.70:0/2031539408 >> 172.21.15.70:0/2031539408 conn(0x7f2b740793f0 msgr2=0x7f2b74079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:16.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.504+0000 7f2b79c31700 1 -- 172.21.15.70:0/2031539408 shutdown_connections 2024-06-04T05:23:16.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.504+0000 7f2b79c31700 1 -- 172.21.15.70:0/2031539408 wait complete. 2024-06-04T05:23:16.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.504+0000 7f2b79c31700 1 Processor -- start 2024-06-04T05:23:16.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.504+0000 7f2b79c31700 1 -- start start 2024-06-04T05:23:16.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b79c31700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b7406bc70 0x7f2b74102540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:16.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b79c31700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b7406c630 0x7f2b74102a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:16.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b79c31700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b741095b0 0x7f2b74102fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:16.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b79c31700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2b741135f0 con 0x7f2b7406bc70 2024-06-04T05:23:16.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b79c31700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2b74113470 con 0x7f2b741095b0 2024-06-04T05:23:16.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b79c31700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2b74113770 con 0x7f2b7406c630 2024-06-04T05:23:16.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b737fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b7406bc70 0x7f2b74102540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:16.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b737fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b7406bc70 0x7f2b74102540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59936/0 (socket says 172.21.15.70:59936) 2024-06-04T05:23:16.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b737fe700 1 -- 172.21.15.70:0/2958575001 learned_addr learned my addr 172.21.15.70:0/2958575001 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:16.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b6bfff700 1 --2- 172.21.15.70:0/2958575001 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b7406c630 0x7f2b74102a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:16.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b73fff700 1 --2- 172.21.15.70:0/2958575001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b741095b0 0x7f2b74102fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:16.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b737fe700 1 -- 172.21.15.70:0/2958575001 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b7406c630 msgr2=0x7f2b74102a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:16.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b737fe700 1 --2- 172.21.15.70:0/2958575001 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b7406c630 0x7f2b74102a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:16.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b737fe700 1 -- 172.21.15.70:0/2958575001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b741095b0 msgr2=0x7f2b74102fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:16.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b737fe700 1 --2- 172.21.15.70:0/2958575001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b741095b0 0x7f2b74102fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:16.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b737fe700 1 -- 172.21.15.70:0/2958575001 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b741a8120 con 0x7f2b7406bc70 2024-06-04T05:23:16.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b73fff700 1 --2- 172.21.15.70:0/2958575001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b741095b0 0x7f2b74102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:16.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b737fe700 1 --2- 172.21.15.70:0/2958575001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b7406bc70 0x7f2b74102540 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f2b6400b530 tx=0x7f2b6400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:16.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b717fa700 1 -- 172.21.15.70:0/2958575001 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b64015020 con 0x7f2b7406bc70 2024-06-04T05:23:16.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b717fa700 1 -- 172.21.15.70:0/2958575001 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2b64011cc0 con 0x7f2b7406bc70 2024-06-04T05:23:16.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.505+0000 7f2b79c31700 1 -- 172.21.15.70:0/2958575001 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b741a8410 con 0x7f2b7406bc70 2024-06-04T05:23:16.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.506+0000 7f2b717fa700 1 -- 172.21.15.70:0/2958575001 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b64010dc0 con 0x7f2b7406bc70 2024-06-04T05:23:16.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.506+0000 7f2b79c31700 1 -- 172.21.15.70:0/2958575001 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b741a8970 con 0x7f2b7406bc70 2024-06-04T05:23:16.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.507+0000 7f2b79c31700 1 -- 172.21.15.70:0/2958575001 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b7410ba90 con 0x7f2b7406bc70 2024-06-04T05:23:16.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.508+0000 7f2b717fa700 1 -- 172.21.15.70:0/2958575001 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f2b64018020 con 0x7f2b7406bc70 2024-06-04T05:23:16.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.508+0000 7f2b717fa700 1 --2- 172.21.15.70:0/2958575001 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2b600b2a70 0x7f2b600b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:16.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.508+0000 7f2b6bfff700 1 --2- 172.21.15.70:0/2958575001 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2b600b2a70 0x7f2b600b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:16.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.508+0000 7f2b717fa700 1 -- 172.21.15.70:0/2958575001 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7f2b640e1d20 con 0x7f2b7406bc70 2024-06-04T05:23:16.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.509+0000 7f2b6bfff700 1 --2- 172.21.15.70:0/2958575001 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2b600b2a70 0x7f2b600b4f30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00b5c0 tx=0x7f2b5c005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:16.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.513+0000 7f2b717fa700 1 -- 172.21.15.70:0/2958575001 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2b6401f020 con 0x7f2b7406bc70 2024-06-04T05:23:16.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.767+0000 7f2b79c31700 1 -- 172.21.15.70:0/2958575001 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b7406a230 con 0x7f2b7406bc70 2024-06-04T05:23:16.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.768+0000 7f2b717fa700 1 -- 172.21.15.70:0/2958575001 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+11940 (secure 0 0 0) 0x7f2b640a97e0 con 0x7f2b7406bc70 2024-06-04T05:23:16.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.770+0000 7f2b79c31700 1 -- 172.21.15.70:0/2958575001 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2b600b2a70 msgr2=0x7f2b600b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:16.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.770+0000 7f2b79c31700 1 --2- 172.21.15.70:0/2958575001 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2b600b2a70 0x7f2b600b4f30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00b5c0 tx=0x7f2b5c005960 comp rx=0 tx=0).stop 2024-06-04T05:23:16.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.770+0000 7f2b79c31700 1 -- 172.21.15.70:0/2958575001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b7406bc70 msgr2=0x7f2b74102540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:16.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.770+0000 7f2b79c31700 1 --2- 172.21.15.70:0/2958575001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b7406bc70 0x7f2b74102540 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f2b6400b530 tx=0x7f2b6400b840 comp rx=0 tx=0).stop 2024-06-04T05:23:16.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.770+0000 7f2b79c31700 1 -- 172.21.15.70:0/2958575001 shutdown_connections 2024-06-04T05:23:16.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.770+0000 7f2b79c31700 1 --2- 172.21.15.70:0/2958575001 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2b600b2a70 0x7f2b600b4f30 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:16.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.770+0000 7f2b79c31700 1 --2- 172.21.15.70:0/2958575001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b741095b0 0x7f2b74102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:16.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.770+0000 7f2b79c31700 1 --2- 172.21.15.70:0/2958575001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b7406bc70 0x7f2b74102540 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:16.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.770+0000 7f2b79c31700 1 --2- 172.21.15.70:0/2958575001 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b7406c630 0x7f2b74102a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:16.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.770+0000 7f2b79c31700 1 -- 172.21.15.70:0/2958575001 >> 172.21.15.70:0/2958575001 conn(0x7f2b740793f0 msgr2=0x7f2b74101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:16.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.770+0000 7f2b79c31700 1 -- 172.21.15.70:0/2958575001 shutdown_connections 2024-06-04T05:23:16.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:15.771+0000 7f2b79c31700 1 -- 172.21.15.70:0/2958575001 wait complete. 2024-06-04T05:23:16.555 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 71 2024-06-04T05:23:16.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:16 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2958575001' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:23:16.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:16 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2958575001' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:23:16.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:16 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2958575001' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:23:17.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:17 smithi098 ceph-mon[35569]: pgmap v824: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 387 op/s 2024-06-04T05:23:17.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:17 smithi070 ceph-mon[32631]: pgmap v824: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 387 op/s 2024-06-04T05:23:17.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:17 smithi112 ceph-mon[36453]: pgmap v824: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 387 op/s 2024-06-04T05:23:18.544 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:18.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:18 smithi070 ceph-mon[32631]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:23:18.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:18 smithi112 ceph-mon[36453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:23:18.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:18 smithi098 ceph-mon[35569]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-04T05:23:19.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:19 smithi070 ceph-mon[32631]: pgmap v825: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 314 op/s 2024-06-04T05:23:19.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:19 smithi112 ceph-mon[36453]: pgmap v825: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 314 op/s 2024-06-04T05:23:19.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:19 smithi098 ceph-mon[35569]: pgmap v825: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 314 op/s 2024-06-04T05:23:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:21 smithi070 ceph-mon[32631]: pgmap v826: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.6 MiB/s wr, 310 op/s 2024-06-04T05:23:21.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:21 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:23:21.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:21 smithi112 ceph-mon[36453]: pgmap v826: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.6 MiB/s wr, 310 op/s 2024-06-04T05:23:21.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:21 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:23:21.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:21 smithi098 ceph-mon[35569]: pgmap v826: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.6 MiB/s wr, 310 op/s 2024-06-04T05:23:21.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:21 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:23:22.603 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24448 quiesce db --cancel --all' 2024-06-04T05:23:22.603 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24448 quiesce db --cancel --all 2024-06-04T05:23:23.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:23 smithi070 ceph-mon[32631]: pgmap v827: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 550 KiB/s wr, 166 op/s 2024-06-04T05:23:23.892 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:23.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:23 smithi112 ceph-mon[36453]: pgmap v827: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 550 KiB/s wr, 166 op/s 2024-06-04T05:23:23.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:23 smithi098 ceph-mon[35569]: pgmap v827: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 550 KiB/s wr, 166 op/s 2024-06-04T05:23:25.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.340+0000 7fbd51893640 1 -- 172.21.15.70:0/3071562438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd4c108a70 msgr2=0x7fbd4c108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.340+0000 7fbd51893640 1 --2- 172.21.15.70:0/3071562438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd4c108a70 0x7fbd4c108e50 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009930 tx=0x7fbd3c02a420 comp rx=0 tx=0).stop 2024-06-04T05:23:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.340+0000 7fbd51893640 1 -- 172.21.15.70:0/3071562438 shutdown_connections 2024-06-04T05:23:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.340+0000 7fbd51893640 1 --2- 172.21.15.70:0/3071562438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd4c103420 0x7fbd4c10f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.340+0000 7fbd51893640 1 --2- 172.21.15.70:0/3071562438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd4c102a60 0x7fbd4c102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.340+0000 7fbd51893640 1 --2- 172.21.15.70:0/3071562438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd4c108a70 0x7fbd4c108e50 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.340+0000 7fbd51893640 1 -- 172.21.15.70:0/3071562438 >> 172.21.15.70:0/3071562438 conn(0x7fbd4c079ee0 msgr2=0x7fbd4c07a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.340+0000 7fbd51893640 1 -- 172.21.15.70:0/3071562438 shutdown_connections 2024-06-04T05:23:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.341+0000 7fbd51893640 1 -- 172.21.15.70:0/3071562438 wait complete. 2024-06-04T05:23:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.341+0000 7fbd51893640 1 Processor -- start 2024-06-04T05:23:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.341+0000 7fbd51893640 1 -- start start 2024-06-04T05:23:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.341+0000 7fbd51893640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd4c102a60 0x7fbd4c199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.341+0000 7fbd51893640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd4c103420 0x7fbd4c1996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.341+0000 7fbd51893640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd4c108a70 0x7fbd4c19d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.341+0000 7fbd51893640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c112be0 con 0x7fbd4c103420 2024-06-04T05:23:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.341+0000 7fbd51893640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c112a60 con 0x7fbd4c108a70 2024-06-04T05:23:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.341+0000 7fbd51893640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c112d60 con 0x7fbd4c102a60 2024-06-04T05:23:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.342+0000 7fbd4a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd4c103420 0x7fbd4c1996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.342+0000 7fbd4a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd4c103420 0x7fbd4c1996a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34402/0 (socket says 172.21.15.70:34402) 2024-06-04T05:23:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.342+0000 7fbd4a7fc640 1 -- 172.21.15.70:0/2782050533 learned_addr learned my addr 172.21.15.70:0/2782050533 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.342+0000 7fbd4affd640 1 --2- 172.21.15.70:0/2782050533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd4c102a60 0x7fbd4c199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.342+0000 7fbd4b7fe640 1 --2- 172.21.15.70:0/2782050533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd4c108a70 0x7fbd4c19d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.342+0000 7fbd4a7fc640 1 -- 172.21.15.70:0/2782050533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd4c102a60 msgr2=0x7fbd4c199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.342+0000 7fbd4a7fc640 1 --2- 172.21.15.70:0/2782050533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd4c102a60 0x7fbd4c199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.342+0000 7fbd4a7fc640 1 -- 172.21.15.70:0/2782050533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd4c108a70 msgr2=0x7fbd4c19d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.342+0000 7fbd4a7fc640 1 --2- 172.21.15.70:0/2782050533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd4c108a70 0x7fbd4c19d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.342+0000 7fbd4a7fc640 1 -- 172.21.15.70:0/2782050533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd4c19e100 con 0x7fbd4c103420 2024-06-04T05:23:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.342+0000 7fbd4b7fe640 1 --2- 172.21.15.70:0/2782050533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd4c108a70 0x7fbd4c19d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.342+0000 7fbd4a7fc640 1 --2- 172.21.15.70:0/2782050533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd4c103420 0x7fbd4c1996a0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000e9f0 tx=0x7fbd4000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.342+0000 7fbd50891640 1 -- 172.21.15.70:0/2782050533 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4000cd80 con 0x7fbd4c103420 2024-06-04T05:23:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.342+0000 7fbd50891640 1 -- 172.21.15.70:0/2782050533 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd400054b0 con 0x7fbd4c103420 2024-06-04T05:23:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.342+0000 7fbd51893640 1 -- 172.21.15.70:0/2782050533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd4c19e3f0 con 0x7fbd4c103420 2024-06-04T05:23:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.342+0000 7fbd50891640 1 -- 172.21.15.70:0/2782050533 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd40010430 con 0x7fbd4c103420 2024-06-04T05:23:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.342+0000 7fbd51893640 1 -- 172.21.15.70:0/2782050533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd4c1a5c80 con 0x7fbd4c103420 2024-06-04T05:23:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.345+0000 7fbd545b5740 1 Processor -- start 2024-06-04T05:23:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.345+0000 7fbd545b5740 1 -- start start 2024-06-04T05:23:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.345+0000 7fbd50891640 1 -- 172.21.15.70:0/2782050533 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fbd400046c0 con 0x7fbd4c103420 2024-06-04T05:23:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.345+0000 7fbd545b5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 0x55c38d790ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.345+0000 7fbd545b5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d79f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.345+0000 7fbd545b5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d7a2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.345+0000 7fbd545b5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c38d702c30 con 0x55c38d79fe10 2024-06-04T05:23:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.345+0000 7fbd545b5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c38d6f27d0 con 0x55c38d790700 2024-06-04T05:23:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.345+0000 7fbd545b5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c38d6f6810 con 0x55c38d791020 2024-06-04T05:23:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.346+0000 7fbd4a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d7a2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.346+0000 7fbd4a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d7a2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34406/0 (socket says 172.21.15.70:34406) 2024-06-04T05:23:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.346+0000 7fbd4a7fc640 1 -- 172.21.15.70:0/1303473731 learned_addr learned my addr 172.21.15.70:0/1303473731 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.346+0000 7fbd4affd640 1 --2- 172.21.15.70:0/1303473731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d79f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.346+0000 7fbd4b7fe640 1 --2- 172.21.15.70:0/1303473731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 0x55c38d790ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.346+0000 7fbd4a7fc640 1 -- 172.21.15.70:0/1303473731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 msgr2=0x55c38d79f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.346+0000 7fbd4a7fc640 1 --2- 172.21.15.70:0/1303473731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d79f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.346+0000 7fbd4a7fc640 1 -- 172.21.15.70:0/1303473731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 msgr2=0x55c38d790ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.346+0000 7fbd4a7fc640 1 --2- 172.21.15.70:0/1303473731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 0x55c38d790ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.346+0000 7fbd4a7fc640 1 -- 172.21.15.70:0/1303473731 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c38d407210 con 0x55c38d79fe10 2024-06-04T05:23:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.346+0000 7fbd4affd640 1 --2- 172.21.15.70:0/1303473731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d79f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:23:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.346+0000 7fbd50891640 1 --2- 172.21.15.70:0/2782050533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd340b2a10 0x7fbd340b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.346+0000 7fbd4a7fc640 1 --2- 172.21.15.70:0/1303473731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d7a2200 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7fbd400e00a0 tx=0x7fbd400e6b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b637bf5336e4e7 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.346+0000 7fbd4b7fe640 1 --2- 172.21.15.70:0/2782050533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd340b2a10 0x7fbd340b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.346+0000 7fbd50891640 1 -- 172.21.15.70:0/2782050533 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fbd400e2a40 con 0x7fbd4c103420 2024-06-04T05:23:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.347+0000 7fbd31ffb640 1 -- 172.21.15.70:0/1303473731 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd400a5bb0 con 0x55c38d79fe10 2024-06-04T05:23:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.347+0000 7fbd31ffb640 1 -- 172.21.15.70:0/1303473731 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd400a9800 con 0x55c38d79fe10 2024-06-04T05:23:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.347+0000 7fbd31ffb640 1 -- 172.21.15.70:0/1303473731 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd400a9460 con 0x55c38d79fe10 2024-06-04T05:23:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.347+0000 7fbd545b5740 1 -- 172.21.15.70:0/1303473731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 msgr2=0x55c38d7a2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.347+0000 7fbd4b7fe640 1 --2- 172.21.15.70:0/2782050533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd340b2a10 0x7fbd340b4ed0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fbd44005fd0 tx=0x7fbd44009770 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.347+0000 7fbd545b5740 1 --2- 172.21.15.70:0/1303473731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d7a2200 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7fbd400e00a0 tx=0x7fbd400e6b80 comp rx=0 tx=0).stop 2024-06-04T05:23:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.347+0000 7fbd545b5740 1 -- 172.21.15.70:0/1303473731 shutdown_connections 2024-06-04T05:23:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.347+0000 7fbd545b5740 1 --2- 172.21.15.70:0/1303473731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d7a2200 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.347+0000 7fbd545b5740 1 --2- 172.21.15.70:0/1303473731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d79f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.347+0000 7fbd545b5740 1 --2- 172.21.15.70:0/1303473731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 0x55c38d790ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.347+0000 7fbd545b5740 1 -- 172.21.15.70:0/1303473731 >> 172.21.15.70:0/1303473731 conn(0x55c38d78cd30 msgr2=0x55c38d78d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.347+0000 7fbd545b5740 1 -- 172.21.15.70:0/1303473731 shutdown_connections 2024-06-04T05:23:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.347+0000 7fbd545b5740 1 -- 172.21.15.70:0/1303473731 wait complete. 2024-06-04T05:23:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.348+0000 7fbd545b5740 1 -- start start 2024-06-04T05:23:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.348+0000 7fbd545b5740 1 Processor -- start 2024-06-04T05:23:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.348+0000 7fbd545b5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 0x55c38d79cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.348+0000 7fbd545b5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d79d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.348+0000 7fbd545b5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d79d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.348+0000 7fbd545b5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c38d3d8790 con 0x55c38d79fe10 2024-06-04T05:23:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.348+0000 7fbd545b5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c38d6ae890 con 0x55c38d790700 2024-06-04T05:23:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.348+0000 7fbd545b5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c38d7a29e0 con 0x55c38d791020 2024-06-04T05:23:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.348+0000 7fbd545b5740 10 client.0 _ll_register_callbacks cb 0x55c38d406a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.348+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d79d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.348+0000 7fbd4a7fc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d79d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.348+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d79d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34422/0 (socket says 172.21.15.70:34422) 2024-06-04T05:23:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.348+0000 7fbd4b7fe640 1 -- 172.21.15.70:0/4094268976 learned_addr learned my addr 172.21.15.70:0/4094268976 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.349+0000 7fbd4affd640 1 --2- 172.21.15.70:0/4094268976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 0x55c38d79cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.349+0000 7fbd4b7fe640 1 -- 172.21.15.70:0/4094268976 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 msgr2=0x55c38d79d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.349+0000 7fbd4b7fe640 1 --2- 172.21.15.70:0/4094268976 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d79d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.349+0000 7fbd4b7fe640 1 -- 172.21.15.70:0/4094268976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 msgr2=0x55c38d79cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.349+0000 7fbd4b7fe640 1 --2- 172.21.15.70:0/4094268976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 0x55c38d79cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.349+0000 7fbd4b7fe640 1 -- 172.21.15.70:0/4094268976 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c38d4d3620 con 0x55c38d79fe10 2024-06-04T05:23:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.349+0000 7fbd4a7fc640 1 --2- 172.21.15.70:0/4094268976 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d79d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:23:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.349+0000 7fbd4affd640 1 --2- 172.21.15.70:0/4094268976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 0x55c38d79cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.349+0000 7fbd4b7fe640 1 --2- 172.21.15.70:0/4094268976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d79d860 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009740 tx=0x7fbd44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c6e441064e9b33 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.349+0000 7fbd2affd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.349+0000 7fbd2affd640 1 -- 172.21.15.70:0/4094268976 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4401eb20 con 0x55c38d79fe10 2024-06-04T05:23:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.349+0000 7fbd545b5740 1 -- 172.21.15.70:0/4094268976 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd400a9800 con 0x55c38d79fe10 2024-06-04T05:23:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.349+0000 7fbd2affd640 1 -- 172.21.15.70:0/4094268976 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fbd4401ecc0 con 0x55c38d79fe10 2024-06-04T05:23:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.349+0000 7fbd2affd640 1 -- 172.21.15.70:0/4094268976 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fbd4402a480 con 0x55c38d79fe10 2024-06-04T05:23:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.349+0000 7fbd545b5740 10 client.22029 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.349+0000 7fbd545b5740 1 -- 172.21.15.70:0/4094268976 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x55c38d7ae900 con 0x55c38d79fe10 2024-06-04T05:23:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.349+0000 7fbd2affd640 1 -- 172.21.15.70:0/4094268976 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd400a9800 con 0x55c38d79fe10 2024-06-04T05:23:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.350+0000 7fbd2affd640 1 -- 172.21.15.70:0/4094268976 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7fbd4400f040 con 0x55c38d79fe10 2024-06-04T05:23:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.350+0000 7fbd545b5740 10 client.22029 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.350+0000 7fbd545b5740 10 client.22029 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:23:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.350+0000 7fbd545b5740 20 client.22029 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.350+0000 7fbd545b5740 1 --2- 172.21.15.70:0/4094268976 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c38d7afde0 0x55c38d7b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.350+0000 7fbd545b5740 4 client.22029 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:23:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.350+0000 7fbd545b5740 1 -- 172.21.15.70:0/4094268976 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c38d704b80 con 0x55c38d7afde0 2024-06-04T05:23:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.350+0000 7fbd4affd640 1 --2- 172.21.15.70:0/4094268976 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c38d7afde0 0x55c38d7b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.350+0000 7fbd4affd640 1 --2- 172.21.15.70:0/4094268976 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c38d7afde0 0x55c38d7b21d0 crc :-1 s=READY pgs=539 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4342c1ba1a6f795b server_cookie=e05b501e14683af1 in_seq=0 out_seq=0 2024-06-04T05:23:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.350+0000 7fbd2affd640 10 client.22029 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.351+0000 7fbd2affd640 1 -- 172.21.15.70:0/4094268976 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c38d704b80 con 0x55c38d7afde0 2024-06-04T05:23:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.351+0000 7fbd2affd640 10 client.22029 handle_command_reply: tid=0 2024-06-04T05:23:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.351+0000 7fbd545b5740 1 client.22029 shutdown 2024-06-04T05:23:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.351+0000 7fbd545b5740 1 -- 172.21.15.70:0/4094268976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 msgr2=0x55c38d79d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.351+0000 7fbd545b5740 1 --2- 172.21.15.70:0/4094268976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d79d860 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009740 tx=0x7fbd44004a40 comp rx=0 tx=0).stop 2024-06-04T05:23:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.351+0000 7fbd545b5740 1 -- 172.21.15.70:0/4094268976 shutdown_connections 2024-06-04T05:23:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.351+0000 7fbd545b5740 1 --2- 172.21.15.70:0/4094268976 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c38d7afde0 0x55c38d7b21d0 crc :-1 s=READY pgs=539 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.351+0000 7fbd545b5740 1 --2- 172.21.15.70:0/4094268976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d79d860 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.351+0000 7fbd545b5740 1 --2- 172.21.15.70:0/4094268976 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d79d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.351+0000 7fbd2affd640 0 client.22029 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.351+0000 7fbd545b5740 1 --2- 172.21.15.70:0/4094268976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 0x55c38d79cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.351+0000 7fbd545b5740 1 -- 172.21.15.70:0/4094268976 >> 172.21.15.70:0/4094268976 conn(0x55c38d78cd30 msgr2=0x55c38d792ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.351+0000 7fbd545b5740 1 -- 172.21.15.70:0/4094268976 shutdown_connections 2024-06-04T05:23:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.351+0000 7fbd545b5740 1 -- 172.21.15.70:0/4094268976 wait complete. 2024-06-04T05:23:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.351+0000 7fbd545b5740 20 client.22029 trim_cache size 0 max 16384 2024-06-04T05:23:25.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.366+0000 7fbd545b5740 1 Processor -- start 2024-06-04T05:23:25.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.366+0000 7fbd545b5740 1 -- start start 2024-06-04T05:23:25.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.366+0000 7fbd545b5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 0x55c38d79bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.366+0000 7fbd545b5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d7929c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.366+0000 7fbd545b5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d7b7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.366+0000 7fbd545b5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c1ac4a0 con 0x55c38d79fe10 2024-06-04T05:23:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.366+0000 7fbd545b5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c1ae190 con 0x55c38d790700 2024-06-04T05:23:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.367+0000 7fbd545b5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c1acc00 con 0x55c38d791020 2024-06-04T05:23:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.367+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d7b7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.367+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d7b7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34426/0 (socket says 172.21.15.70:34426) 2024-06-04T05:23:25.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.367+0000 7fbd4b7fe640 1 -- 172.21.15.70:0/1018705063 learned_addr learned my addr 172.21.15.70:0/1018705063 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:25.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.367+0000 7fbd4a7fc640 1 --2- 172.21.15.70:0/1018705063 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d7929c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.367+0000 7fbd4b7fe640 1 -- 172.21.15.70:0/1018705063 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 msgr2=0x55c38d7929c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:25.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.367+0000 7fbd4b7fe640 1 --2- 172.21.15.70:0/1018705063 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d7929c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.367+0000 7fbd4b7fe640 1 -- 172.21.15.70:0/1018705063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 msgr2=0x55c38d79bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:25.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.367+0000 7fbd4affd640 1 --2- 172.21.15.70:0/1018705063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 0x55c38d79bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.367+0000 7fbd4b7fe640 1 --2- 172.21.15.70:0/1018705063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 0x55c38d79bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.367+0000 7fbd4b7fe640 1 -- 172.21.15.70:0/1018705063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c38d7a2200 con 0x55c38d79fe10 2024-06-04T05:23:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.367+0000 7fbd4b7fe640 1 --2- 172.21.15.70:0/1018705063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d7b7aa0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fbd44018040 tx=0x7fbd440097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2827660dc63f56 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.367+0000 7fbd317fa640 1 -- 172.21.15.70:0/1018705063 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd44036070 con 0x55c38d79fe10 2024-06-04T05:23:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.367+0000 7fbd317fa640 1 -- 172.21.15.70:0/1018705063 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd44038070 con 0x55c38d79fe10 2024-06-04T05:23:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.367+0000 7fbd317fa640 1 -- 172.21.15.70:0/1018705063 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4401ed50 con 0x55c38d79fe10 2024-06-04T05:23:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.368+0000 7fbd545b5740 1 -- 172.21.15.70:0/1018705063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 msgr2=0x55c38d7b7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.368+0000 7fbd545b5740 1 --2- 172.21.15.70:0/1018705063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d7b7aa0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fbd44018040 tx=0x7fbd440097e0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.368+0000 7fbd545b5740 1 -- 172.21.15.70:0/1018705063 shutdown_connections 2024-06-04T05:23:25.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.368+0000 7fbd545b5740 1 --2- 172.21.15.70:0/1018705063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d7b7aa0 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.368+0000 7fbd545b5740 1 --2- 172.21.15.70:0/1018705063 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d7929c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.368+0000 7fbd545b5740 1 --2- 172.21.15.70:0/1018705063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 0x55c38d79bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.368+0000 7fbd545b5740 1 -- 172.21.15.70:0/1018705063 >> 172.21.15.70:0/1018705063 conn(0x55c38d78cd30 msgr2=0x55c38d7922b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:25.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.368+0000 7fbd545b5740 1 -- 172.21.15.70:0/1018705063 shutdown_connections 2024-06-04T05:23:25.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.368+0000 7fbd545b5740 1 -- 172.21.15.70:0/1018705063 wait complete. 2024-06-04T05:23:25.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.368+0000 7fbd545b5740 1 -- start start 2024-06-04T05:23:25.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.368+0000 7fbd545b5740 1 Processor -- start 2024-06-04T05:23:25.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd545b5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 0x55c38d7a3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd545b5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d7a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd545b5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d812930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd545b5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c1ada90 con 0x55c38d79fe10 2024-06-04T05:23:25.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd545b5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c1ad300 con 0x55c38d790700 2024-06-04T05:23:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd545b5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c1aa4a0 con 0x55c38d791020 2024-06-04T05:23:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd545b5740 10 client.0 _ll_register_callbacks cb 0x7fbd44009cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d812930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd4affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 0x55c38d7a3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d812930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34436/0 (socket says 172.21.15.70:34436) 2024-06-04T05:23:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd4b7fe640 1 -- 172.21.15.70:0/3055717076 learned_addr learned my addr 172.21.15.70:0/3055717076 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd4a7fc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d7a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd4b7fe640 1 -- 172.21.15.70:0/3055717076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 msgr2=0x55c38d7a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd4b7fe640 1 --2- 172.21.15.70:0/3055717076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d7a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd4b7fe640 1 -- 172.21.15.70:0/3055717076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 msgr2=0x55c38d7a3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd4b7fe640 1 --2- 172.21.15.70:0/3055717076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 0x55c38d7a3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd4b7fe640 1 -- 172.21.15.70:0/3055717076 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c38d812f70 con 0x55c38d79fe10 2024-06-04T05:23:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd4b7fe640 1 --2- 172.21.15.70:0/3055717076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d812930 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009740 tx=0x7fbd44004c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9ca3bf19152077 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd2affd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd2affd640 1 -- 172.21.15.70:0/3055717076 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd44042020 con 0x55c38d79fe10 2024-06-04T05:23:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd2affd640 1 -- 172.21.15.70:0/3055717076 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fbd4401eb80 con 0x55c38d79fe10 2024-06-04T05:23:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.369+0000 7fbd545b5740 1 -- 172.21.15.70:0/3055717076 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd44038070 con 0x55c38d79fe10 2024-06-04T05:23:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.370+0000 7fbd2affd640 1 -- 172.21.15.70:0/3055717076 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fbd4404d050 con 0x55c38d79fe10 2024-06-04T05:23:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.370+0000 7fbd545b5740 10 client.22035 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.370+0000 7fbd545b5740 1 -- 172.21.15.70:0/3055717076 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x55c38d81a330 con 0x55c38d79fe10 2024-06-04T05:23:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.370+0000 7fbd2affd640 1 -- 172.21.15.70:0/3055717076 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd44038070 con 0x55c38d79fe10 2024-06-04T05:23:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.370+0000 7fbd2affd640 1 -- 172.21.15.70:0/3055717076 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7fbd4403d470 con 0x55c38d79fe10 2024-06-04T05:23:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.370+0000 7fbd545b5740 10 client.22035 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.370+0000 7fbd545b5740 10 client.22035 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-04T05:23:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.370+0000 7fbd545b5740 20 client.22035 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.370+0000 7fbd545b5740 1 --2- 172.21.15.70:0/3055717076 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c38d81a980 0x55c38d81cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.370+0000 7fbd545b5740 4 client.22035 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-04T05:23:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.370+0000 7fbd545b5740 1 -- 172.21.15.70:0/3055717076 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c38d6fe360 con 0x55c38d81a980 2024-06-04T05:23:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.370+0000 7fbd4affd640 1 --2- 172.21.15.70:0/3055717076 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c38d81a980 0x55c38d81cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.370+0000 7fbd4affd640 1 --2- 172.21.15.70:0/3055717076 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c38d81a980 0x55c38d81cda0 crc :-1 s=READY pgs=540 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e43a758be30f95d server_cookie=f0fb39aeb726c143 in_seq=0 out_seq=0 2024-06-04T05:23:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.370+0000 7fbd2affd640 10 client.22035 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.422+0000 7fbd2affd640 1 -- 172.21.15.70:0/3055717076 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55c38d6fe360 con 0x55c38d81a980 2024-06-04T05:23:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.422+0000 7fbd2affd640 10 client.22035 handle_command_reply: tid=0 2024-06-04T05:23:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.422+0000 7fbd545b5740 1 client.22035 shutdown 2024-06-04T05:23:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.422+0000 7fbd545b5740 1 -- 172.21.15.70:0/3055717076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 msgr2=0x55c38d812930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.422+0000 7fbd545b5740 1 --2- 172.21.15.70:0/3055717076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d812930 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009740 tx=0x7fbd44004c30 comp rx=0 tx=0).stop 2024-06-04T05:23:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.422+0000 7fbd545b5740 1 -- 172.21.15.70:0/3055717076 shutdown_connections 2024-06-04T05:23:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.422+0000 7fbd545b5740 1 --2- 172.21.15.70:0/3055717076 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55c38d81a980 0x55c38d81cda0 crc :-1 s=READY pgs=540 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.422+0000 7fbd545b5740 1 --2- 172.21.15.70:0/3055717076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c38d79fe10 0x55c38d812930 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.422+0000 7fbd2affd640 0 client.22035 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.422+0000 7fbd545b5740 1 --2- 172.21.15.70:0/3055717076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c38d791020 0x55c38d7a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.422+0000 7fbd545b5740 1 --2- 172.21.15.70:0/3055717076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38d790700 0x55c38d7a3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.422+0000 7fbd545b5740 1 -- 172.21.15.70:0/3055717076 >> 172.21.15.70:0/3055717076 conn(0x55c38d78cd30 msgr2=0x55c38d79d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.422+0000 7fbd545b5740 1 -- 172.21.15.70:0/3055717076 shutdown_connections 2024-06-04T05:23:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.422+0000 7fbd545b5740 1 -- 172.21.15.70:0/3055717076 wait complete. 2024-06-04T05:23:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.422+0000 7fbd545b5740 20 client.22035 trim_cache size 0 max 16384 2024-06-04T05:23:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-06-04T05:23:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-06-04T05:23:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-06-04T05:23:25.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.423+0000 7fbd51893640 1 -- 172.21.15.70:0/2782050533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd340b2a10 msgr2=0x7fbd340b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.423+0000 7fbd51893640 1 --2- 172.21.15.70:0/2782050533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd340b2a10 0x7fbd340b4ed0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fbd44005fd0 tx=0x7fbd44009770 comp rx=0 tx=0).stop 2024-06-04T05:23:25.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.423+0000 7fbd51893640 1 -- 172.21.15.70:0/2782050533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd4c103420 msgr2=0x7fbd4c1996a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.423+0000 7fbd51893640 1 --2- 172.21.15.70:0/2782050533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd4c103420 0x7fbd4c1996a0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000e9f0 tx=0x7fbd4000edf0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.423+0000 7fbd51893640 1 -- 172.21.15.70:0/2782050533 shutdown_connections 2024-06-04T05:23:25.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.423+0000 7fbd51893640 1 --2- 172.21.15.70:0/2782050533 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbd340b2a10 0x7fbd340b4ed0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.423+0000 7fbd51893640 1 --2- 172.21.15.70:0/2782050533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd4c108a70 0x7fbd4c19d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.423+0000 7fbd51893640 1 --2- 172.21.15.70:0/2782050533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd4c103420 0x7fbd4c1996a0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.423+0000 7fbd51893640 1 --2- 172.21.15.70:0/2782050533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd4c102a60 0x7fbd4c199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.423+0000 7fbd51893640 1 -- 172.21.15.70:0/2782050533 >> 172.21.15.70:0/2782050533 conn(0x7fbd4c079ee0 msgr2=0x7fbd4c100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.423+0000 7fbd51893640 1 -- 172.21.15.70:0/2782050533 shutdown_connections 2024-06-04T05:23:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-06-04T05:23:25.423+0000 7fbd51893640 1 -- 172.21.15.70:0/2782050533 wait complete. 2024-06-04T05:23:25.548 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:25 smithi070 ceph-mon[32631]: pgmap v828: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 488 KiB/s wr, 147 op/s 2024-06-04T05:23:25.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.812+0000 7ff41b242700 1 -- 172.21.15.70:0/2917266699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4141084b0 msgr2=0x7ff414108890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.812+0000 7ff41b242700 1 --2- 172.21.15.70:0/2917266699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4141084b0 0x7ff414108890 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7ff408009990 tx=0x7ff408009ca0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.812+0000 7ff41b242700 1 -- 172.21.15.70:0/2917266699 shutdown_connections 2024-06-04T05:23:25.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.812+0000 7ff41b242700 1 --2- 172.21.15.70:0/2917266699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff414102e30 0x7ff41410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.812+0000 7ff41b242700 1 --2- 172.21.15.70:0/2917266699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4141084b0 0x7ff414108890 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.812+0000 7ff41b242700 1 --2- 172.21.15.70:0/2917266699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff414102470 0x7ff4141028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.813+0000 7ff41b242700 1 -- 172.21.15.70:0/2917266699 >> 172.21.15.70:0/2917266699 conn(0x7ff4140784b0 msgr2=0x7ff4140788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:25.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.813+0000 7ff41b242700 1 -- 172.21.15.70:0/2917266699 shutdown_connections 2024-06-04T05:23:25.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.813+0000 7ff41b242700 1 -- 172.21.15.70:0/2917266699 wait complete. 2024-06-04T05:23:25.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.813+0000 7ff41b242700 1 Processor -- start 2024-06-04T05:23:25.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.813+0000 7ff41b242700 1 -- start start 2024-06-04T05:23:25.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.813+0000 7ff41b242700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff414102470 0x7ff41419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.813+0000 7ff41b242700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff414102e30 0x7ff41419af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.813+0000 7ff41b242700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4141084b0 0x7ff4141a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.813+0000 7ff41b242700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff414112510 con 0x7ff414102e30 2024-06-04T05:23:25.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.813+0000 7ff41b242700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff414112390 con 0x7ff414102470 2024-06-04T05:23:25.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.813+0000 7ff41b242700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff414112690 con 0x7ff4141084b0 2024-06-04T05:23:25.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.814+0000 7ff413fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff414102e30 0x7ff41419af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.814+0000 7ff413fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff414102e30 0x7ff41419af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34454/0 (socket says 172.21.15.70:34454) 2024-06-04T05:23:25.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.814+0000 7ff413fff700 1 -- 172.21.15.70:0/9265822 learned_addr learned my addr 172.21.15.70:0/9265822 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:25.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.814+0000 7ff4197df700 1 --2- 172.21.15.70:0/9265822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4141084b0 0x7ff4141a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.814+0000 7ff418fde700 1 --2- 172.21.15.70:0/9265822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff414102470 0x7ff41419aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.814+0000 7ff413fff700 1 -- 172.21.15.70:0/9265822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4141084b0 msgr2=0x7ff4141a5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.814+0000 7ff413fff700 1 --2- 172.21.15.70:0/9265822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4141084b0 0x7ff4141a5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.814+0000 7ff413fff700 1 -- 172.21.15.70:0/9265822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff414102470 msgr2=0x7ff41419aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.814+0000 7ff413fff700 1 --2- 172.21.15.70:0/9265822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff414102470 0x7ff41419aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.814+0000 7ff413fff700 1 -- 172.21.15.70:0/9265822 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4141a5fe0 con 0x7ff414102e30 2024-06-04T05:23:25.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.814+0000 7ff418fde700 1 --2- 172.21.15.70:0/9265822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff414102470 0x7ff41419aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:25.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.814+0000 7ff413fff700 1 --2- 172.21.15.70:0/9265822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff414102e30 0x7ff41419af60 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7ff40000b530 tx=0x7ff40000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.814+0000 7ff411ffb700 1 -- 172.21.15.70:0/9265822 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff400015020 con 0x7ff414102e30 2024-06-04T05:23:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.814+0000 7ff41b242700 1 -- 172.21.15.70:0/9265822 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4141a6210 con 0x7ff414102e30 2024-06-04T05:23:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.814+0000 7ff411ffb700 1 -- 172.21.15.70:0/9265822 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff400011cc0 con 0x7ff414102e30 2024-06-04T05:23:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.814+0000 7ff411ffb700 1 -- 172.21.15.70:0/9265822 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff400010d30 con 0x7ff414102e30 2024-06-04T05:23:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.815+0000 7ff41b242700 1 -- 172.21.15.70:0/9265822 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4141a67a0 con 0x7ff414102e30 2024-06-04T05:23:25.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.817+0000 7ff411ffb700 1 -- 172.21.15.70:0/9265822 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff400018020 con 0x7ff414102e30 2024-06-04T05:23:25.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.818+0000 7ff411ffb700 1 --2- 172.21.15.70:0/9265822 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4040b29f0 0x7ff4040b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.818+0000 7ff418fde700 1 --2- 172.21.15.70:0/9265822 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4040b29f0 0x7ff4040b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.818+0000 7ff411ffb700 1 -- 172.21.15.70:0/9265822 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7ff4000e1dd0 con 0x7ff414102e30 2024-06-04T05:23:25.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.818+0000 7ff418fde700 1 --2- 172.21.15.70:0/9265822 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4040b29f0 0x7ff4040b4eb0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff408006010 tx=0x7ff40801a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:25.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff425ad7b80 1 Processor -- start 2024-06-04T05:23:25.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff425ad7b80 1 -- start start 2024-06-04T05:23:25.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff425ad7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdefcf0 0x562a4bdf00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff425ad7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdf06a0 0x562a4bdf3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff425ad7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf43f0 0x562a4bdf4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff425ad7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562a4bd4ad10 con 0x562a4bdefcf0 2024-06-04T05:23:25.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff425ad7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562a4bd3a8b0 con 0x562a4bdf06a0 2024-06-04T05:23:25.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff425ad7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562a4bd3e8f0 con 0x562a4bdf43f0 2024-06-04T05:23:25.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff418fde700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdefcf0 0x562a4bdf00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff418fde700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdefcf0 0x562a4bdf00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34460/0 (socket says 172.21.15.70:34460) 2024-06-04T05:23:25.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff418fde700 1 -- 172.21.15.70:0/2099049537 learned_addr learned my addr 172.21.15.70:0/2099049537 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:25.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff4197df700 1 --2- 172.21.15.70:0/2099049537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf43f0 0x562a4bdf4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff413fff700 1 --2- 172.21.15.70:0/2099049537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdf06a0 0x562a4bdf3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff418fde700 1 -- 172.21.15.70:0/2099049537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf43f0 msgr2=0x562a4bdf4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:25.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff418fde700 1 --2- 172.21.15.70:0/2099049537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf43f0 0x562a4bdf4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff418fde700 1 -- 172.21.15.70:0/2099049537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdf06a0 msgr2=0x562a4bdf3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:25.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff418fde700 1 --2- 172.21.15.70:0/2099049537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdf06a0 0x562a4bdf3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff418fde700 1 -- 172.21.15.70:0/2099049537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a4bc8a950 con 0x562a4bdefcf0 2024-06-04T05:23:25.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.819+0000 7ff418fde700 1 --2- 172.21.15.70:0/2099049537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdefcf0 0x562a4bdf00d0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7ff408009750 tx=0x7ff40801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0daed3aa6b14b9a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:25.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.820+0000 7ff3feffd700 1 -- 172.21.15.70:0/2099049537 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff408003ba0 con 0x562a4bdefcf0 2024-06-04T05:23:25.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.820+0000 7ff3feffd700 1 -- 172.21.15.70:0/2099049537 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff408003d40 con 0x562a4bdefcf0 2024-06-04T05:23:25.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.820+0000 7ff3feffd700 1 -- 172.21.15.70:0/2099049537 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff40802f920 con 0x562a4bdefcf0 2024-06-04T05:23:25.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.820+0000 7ff425ad7b80 1 -- 172.21.15.70:0/2099049537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdefcf0 msgr2=0x562a4bdf00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.820+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/2099049537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdefcf0 0x562a4bdf00d0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7ff408009750 tx=0x7ff40801e000 comp rx=0 tx=0).stop 2024-06-04T05:23:25.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.820+0000 7ff425ad7b80 1 -- 172.21.15.70:0/2099049537 shutdown_connections 2024-06-04T05:23:25.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.820+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/2099049537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf43f0 0x562a4bdf4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.820+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/2099049537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdefcf0 0x562a4bdf00d0 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.820+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/2099049537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdf06a0 0x562a4bdf3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.820+0000 7ff425ad7b80 1 -- 172.21.15.70:0/2099049537 >> 172.21.15.70:0/2099049537 conn(0x562a4bd19100 msgr2=0x562a4bde5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.820+0000 7ff425ad7b80 1 -- 172.21.15.70:0/2099049537 shutdown_connections 2024-06-04T05:23:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.820+0000 7ff425ad7b80 1 -- 172.21.15.70:0/2099049537 wait complete. 2024-06-04T05:23:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.821+0000 7ff425ad7b80 1 -- start start 2024-06-04T05:23:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.821+0000 7ff425ad7b80 1 Processor -- start 2024-06-04T05:23:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.821+0000 7ff425ad7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdefcf0 0x562a4bdfb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.821+0000 7ff425ad7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdf06a0 0x562a4bdf8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.821+0000 7ff425ad7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf43f0 0x562a4bdf9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.821+0000 7ff425ad7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562a4bc41020 con 0x562a4bdefcf0 2024-06-04T05:23:25.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.821+0000 7ff425ad7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562a4bcdc840 con 0x562a4bdf06a0 2024-06-04T05:23:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.821+0000 7ff425ad7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x562a4bdfc550 con 0x562a4bdf43f0 2024-06-04T05:23:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.821+0000 7ff425ad7b80 10 client.0 _ll_register_callbacks cb 0x562a4bcd59a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.821+0000 7ff418fde700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdefcf0 0x562a4bdfb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.821+0000 7ff418fde700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdefcf0 0x562a4bdfb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34464/0 (socket says 172.21.15.70:34464) 2024-06-04T05:23:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.821+0000 7ff418fde700 1 -- 172.21.15.70:0/278627007 learned_addr learned my addr 172.21.15.70:0/278627007 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.822+0000 7ff413fff700 1 --2- 172.21.15.70:0/278627007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdf06a0 0x562a4bdf8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.822+0000 7ff4197df700 1 --2- 172.21.15.70:0/278627007 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf43f0 0x562a4bdf9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.822+0000 7ff418fde700 1 -- 172.21.15.70:0/278627007 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf43f0 msgr2=0x562a4bdf9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.822+0000 7ff418fde700 1 --2- 172.21.15.70:0/278627007 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf43f0 0x562a4bdf9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.822+0000 7ff418fde700 1 -- 172.21.15.70:0/278627007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdf06a0 msgr2=0x562a4bdf8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.822+0000 7ff418fde700 1 --2- 172.21.15.70:0/278627007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdf06a0 0x562a4bdf8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.822+0000 7ff418fde700 1 -- 172.21.15.70:0/278627007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a4bcd9710 con 0x562a4bdefcf0 2024-06-04T05:23:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.822+0000 7ff418fde700 1 --2- 172.21.15.70:0/278627007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdefcf0 0x562a4bdfb8d0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7ff40800bdd0 tx=0x7ff40800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4950b75f7f2c31e8 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.822+0000 7ff3f3fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.822+0000 7ff3f3fff700 1 -- 172.21.15.70:0/278627007 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff408003ba0 con 0x562a4bdefcf0 2024-06-04T05:23:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.822+0000 7ff425ad7b80 1 -- 172.21.15.70:0/278627007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff408003d40 con 0x562a4bdefcf0 2024-06-04T05:23:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.822+0000 7ff3f3fff700 1 -- 172.21.15.70:0/278627007 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7ff40802f710 con 0x562a4bdefcf0 2024-06-04T05:23:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.822+0000 7ff3f3fff700 1 -- 172.21.15.70:0/278627007 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7ff40802f910 con 0x562a4bdefcf0 2024-06-04T05:23:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.822+0000 7ff425ad7b80 10 client.22047 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.822+0000 7ff425ad7b80 1 -- 172.21.15.70:0/278627007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x562a4be06d10 con 0x562a4bdefcf0 2024-06-04T05:23:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.822+0000 7ff3f3fff700 1 -- 172.21.15.70:0/278627007 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff408003d40 con 0x562a4bdefcf0 2024-06-04T05:23:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.823+0000 7ff3f3fff700 1 -- 172.21.15.70:0/278627007 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7ff408022070 con 0x562a4bdefcf0 2024-06-04T05:23:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.823+0000 7ff425ad7b80 10 client.22047 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.823+0000 7ff425ad7b80 10 client.22047 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:23:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.823+0000 7ff425ad7b80 20 client.22047 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.823+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/278627007 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562a4be091e0 0x562a4be0b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.823+0000 7ff425ad7b80 4 client.22047 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:23:25.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.823+0000 7ff425ad7b80 1 -- 172.21.15.70:0/278627007 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a4bdb7a60 con 0x562a4be091e0 2024-06-04T05:23:25.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.823+0000 7ff413fff700 1 --2- 172.21.15.70:0/278627007 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562a4be091e0 0x562a4be0b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.823+0000 7ff413fff700 1 --2- 172.21.15.70:0/278627007 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562a4be091e0 0x562a4be0b5d0 crc :-1 s=READY pgs=541 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47d8207add196333 server_cookie=857b116e8f80e9e7 in_seq=0 out_seq=0 2024-06-04T05:23:25.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.823+0000 7ff3f3fff700 10 client.22047 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:25.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.824+0000 7ff3f3fff700 1 -- 172.21.15.70:0/278627007 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562a4bdb7a60 con 0x562a4be091e0 2024-06-04T05:23:25.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.824+0000 7ff3f3fff700 10 client.22047 handle_command_reply: tid=0 2024-06-04T05:23:25.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.824+0000 7ff425ad7b80 1 client.22047 shutdown 2024-06-04T05:23:25.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.824+0000 7ff425ad7b80 1 -- 172.21.15.70:0/278627007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdefcf0 msgr2=0x562a4bdfb8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.824+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/278627007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdefcf0 0x562a4bdfb8d0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7ff40800bdd0 tx=0x7ff40800be00 comp rx=0 tx=0).stop 2024-06-04T05:23:25.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.824+0000 7ff425ad7b80 1 -- 172.21.15.70:0/278627007 shutdown_connections 2024-06-04T05:23:25.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.824+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/278627007 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf43f0 0x562a4bdf9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.824+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/278627007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdefcf0 0x562a4bdfb8d0 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.824+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/278627007 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562a4be091e0 0x562a4be0b5d0 crc :-1 s=READY pgs=541 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.824+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/278627007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdf06a0 0x562a4bdf8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.824+0000 7ff3f3fff700 0 client.22047 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:25.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.824+0000 7ff425ad7b80 1 -- 172.21.15.70:0/278627007 >> 172.21.15.70:0/278627007 conn(0x562a4bd19100 msgr2=0x562a4bde5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:25.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.824+0000 7ff425ad7b80 1 -- 172.21.15.70:0/278627007 shutdown_connections 2024-06-04T05:23:25.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.824+0000 7ff425ad7b80 1 -- 172.21.15.70:0/278627007 wait complete. 2024-06-04T05:23:25.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.824+0000 7ff425ad7b80 20 client.22047 trim_cache size 0 max 16384 2024-06-04T05:23:25.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.845+0000 7ff425ad7b80 1 Processor -- start 2024-06-04T05:23:25.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.845+0000 7ff425ad7b80 1 -- start start 2024-06-04T05:23:25.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.845+0000 7ff425ad7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdefcf0 0x562a4bde5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.845+0000 7ff425ad7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdf06a0 0x562a4be0e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.845+0000 7ff425ad7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf43f0 0x562a4be0e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.845+0000 7ff425ad7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff4141abcb0 con 0x562a4bdf06a0 2024-06-04T05:23:25.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.845+0000 7ff425ad7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff4141ad9a0 con 0x562a4bdefcf0 2024-06-04T05:23:25.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.845+0000 7ff425ad7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff4141ac410 con 0x562a4bdf43f0 2024-06-04T05:23:25.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.845+0000 7ff413fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdf06a0 0x562a4be0e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.845+0000 7ff413fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdf06a0 0x562a4be0e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34472/0 (socket says 172.21.15.70:34472) 2024-06-04T05:23:25.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.845+0000 7ff413fff700 1 -- 172.21.15.70:0/1328103044 learned_addr learned my addr 172.21.15.70:0/1328103044 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:25.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.845+0000 7ff418fde700 1 --2- 172.21.15.70:0/1328103044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdefcf0 0x562a4bde5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.845+0000 7ff4197df700 1 --2- 172.21.15.70:0/1328103044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf43f0 0x562a4be0e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.846+0000 7ff413fff700 1 -- 172.21.15.70:0/1328103044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf43f0 msgr2=0x562a4be0e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:25.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.846+0000 7ff413fff700 1 --2- 172.21.15.70:0/1328103044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf43f0 0x562a4be0e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.846+0000 7ff413fff700 1 -- 172.21.15.70:0/1328103044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdefcf0 msgr2=0x562a4bde5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:25.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.846+0000 7ff413fff700 1 --2- 172.21.15.70:0/1328103044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdefcf0 0x562a4bde5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.846+0000 7ff413fff700 1 -- 172.21.15.70:0/1328103044 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a4bde6770 con 0x562a4bdf06a0 2024-06-04T05:23:25.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.846+0000 7ff4197df700 1 --2- 172.21.15.70:0/1328103044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf43f0 0x562a4be0e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:23:25.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.846+0000 7ff413fff700 1 --2- 172.21.15.70:0/1328103044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdf06a0 0x562a4be0e220 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7ff4000a8e90 tx=0x7ff4000e1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd2986462dba573a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:25.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.846+0000 7ff3fe7fc700 1 -- 172.21.15.70:0/1328103044 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4000ad540 con 0x562a4bdf06a0 2024-06-04T05:23:25.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.846+0000 7ff3fe7fc700 1 -- 172.21.15.70:0/1328103044 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff4000b1ff0 con 0x562a4bdf06a0 2024-06-04T05:23:25.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.846+0000 7ff3fe7fc700 1 -- 172.21.15.70:0/1328103044 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4000a93f0 con 0x562a4bdf06a0 2024-06-04T05:23:25.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.846+0000 7ff425ad7b80 1 -- 172.21.15.70:0/1328103044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdf06a0 msgr2=0x562a4be0e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.847+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/1328103044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdf06a0 0x562a4be0e220 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7ff4000a8e90 tx=0x7ff4000e1040 comp rx=0 tx=0).stop 2024-06-04T05:23:25.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.847+0000 7ff425ad7b80 1 -- 172.21.15.70:0/1328103044 shutdown_connections 2024-06-04T05:23:25.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.847+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/1328103044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf43f0 0x562a4be0e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.847+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/1328103044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdefcf0 0x562a4bde5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.847+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/1328103044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdf06a0 0x562a4be0e220 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.847+0000 7ff425ad7b80 1 -- 172.21.15.70:0/1328103044 >> 172.21.15.70:0/1328103044 conn(0x562a4bd19100 msgr2=0x562a4bdf7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:25.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.847+0000 7ff425ad7b80 1 -- 172.21.15.70:0/1328103044 shutdown_connections 2024-06-04T05:23:25.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.847+0000 7ff425ad7b80 1 -- 172.21.15.70:0/1328103044 wait complete. 2024-06-04T05:23:25.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.847+0000 7ff425ad7b80 1 -- start start 2024-06-04T05:23:25.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.847+0000 7ff425ad7b80 1 Processor -- start 2024-06-04T05:23:25.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.847+0000 7ff425ad7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdefcf0 0x562a4be6d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.847+0000 7ff425ad7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf06a0 0x562a4be6d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.847+0000 7ff425ad7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdf43f0 0x562a4be679b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.847+0000 7ff425ad7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff4141ad2a0 con 0x562a4bdf43f0 2024-06-04T05:23:25.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.847+0000 7ff425ad7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff4141acb10 con 0x562a4bdefcf0 2024-06-04T05:23:25.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.847+0000 7ff425ad7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff4141a9c70 con 0x562a4bdf06a0 2024-06-04T05:23:25.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.847+0000 7ff425ad7b80 10 client.0 _ll_register_callbacks cb 0x562a4bcd59a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:25.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff4197df700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdf43f0 0x562a4be679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff4197df700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdf43f0 0x562a4be679b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34474/0 (socket says 172.21.15.70:34474) 2024-06-04T05:23:25.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff4197df700 1 -- 172.21.15.70:0/3918295530 learned_addr learned my addr 172.21.15.70:0/3918295530 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:25.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff413fff700 1 --2- 172.21.15.70:0/3918295530 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf06a0 0x562a4be6d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff418fde700 1 --2- 172.21.15.70:0/3918295530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdefcf0 0x562a4be6d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff4197df700 1 -- 172.21.15.70:0/3918295530 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf06a0 msgr2=0x562a4be6d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:25.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff4197df700 1 --2- 172.21.15.70:0/3918295530 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf06a0 0x562a4be6d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff4197df700 1 -- 172.21.15.70:0/3918295530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdefcf0 msgr2=0x562a4be6d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:25.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff4197df700 1 --2- 172.21.15.70:0/3918295530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdefcf0 0x562a4be6d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff4197df700 1 -- 172.21.15.70:0/3918295530 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a4be680b0 con 0x562a4bdf43f0 2024-06-04T05:23:25.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff413fff700 1 --2- 172.21.15.70:0/3918295530 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf06a0 0x562a4be6d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:25.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff4197df700 1 --2- 172.21.15.70:0/3918295530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdf43f0 0x562a4be679b0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7ff40c007ab0 tx=0x7ff40c007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524393465be700dc server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:25.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff3f3fff700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:25.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff3f3fff700 1 -- 172.21.15.70:0/3918295530 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff40c01d070 con 0x562a4bdf43f0 2024-06-04T05:23:25.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff3f3fff700 1 -- 172.21.15.70:0/3918295530 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7ff40c015b50 con 0x562a4bdf43f0 2024-06-04T05:23:25.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff425ad7b80 1 -- 172.21.15.70:0/3918295530 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4000b1ff0 con 0x562a4bdf43f0 2024-06-04T05:23:25.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff3f3fff700 1 -- 172.21.15.70:0/3918295530 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7ff40c029050 con 0x562a4bdf43f0 2024-06-04T05:23:25.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff425ad7b80 10 client.22053 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:25.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.848+0000 7ff425ad7b80 1 -- 172.21.15.70:0/3918295530 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x562a4be74ca0 con 0x562a4bdf43f0 2024-06-04T05:23:25.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.849+0000 7ff3f3fff700 1 -- 172.21.15.70:0/3918295530 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff4000b1ff0 con 0x562a4bdf43f0 2024-06-04T05:23:25.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.849+0000 7ff3f3fff700 1 -- 172.21.15.70:0/3918295530 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7ff40c0213b0 con 0x562a4bdf43f0 2024-06-04T05:23:25.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.849+0000 7ff425ad7b80 10 client.22053 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:25.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.849+0000 7ff425ad7b80 10 client.22053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:23:25.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.849+0000 7ff425ad7b80 20 client.22053 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:25.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.849+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/3918295530 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562a4be75440 0x562a4be77860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:25.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.849+0000 7ff425ad7b80 4 client.22053 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-04T05:23:25.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.849+0000 7ff425ad7b80 1 -- 172.21.15.70:0/3918295530 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562a4bd4cc60 con 0x562a4be75440 2024-06-04T05:23:25.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.849+0000 7ff418fde700 1 --2- 172.21.15.70:0/3918295530 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562a4be75440 0x562a4be77860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:25.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.849+0000 7ff418fde700 1 --2- 172.21.15.70:0/3918295530 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562a4be75440 0x562a4be77860 crc :-1 s=READY pgs=542 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11854b934634a6c3 server_cookie=726438e75a18a700 in_seq=0 out_seq=0 2024-06-04T05:23:25.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.849+0000 7ff3f3fff700 10 client.22053 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:25.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.850+0000 7ff3f3fff700 1 -- 172.21.15.70:0/3918295530 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562a4bd4cc60 con 0x562a4be75440 2024-06-04T05:23:25.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.850+0000 7ff3f3fff700 10 client.22053 handle_command_reply: tid=0 2024-06-04T05:23:25.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.850+0000 7ff425ad7b80 1 client.22053 shutdown 2024-06-04T05:23:25.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.850+0000 7ff425ad7b80 1 -- 172.21.15.70:0/3918295530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdf43f0 msgr2=0x562a4be679b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.850+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/3918295530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdf43f0 0x562a4be679b0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7ff40c007ab0 tx=0x7ff40c007dc0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.850+0000 7ff425ad7b80 1 -- 172.21.15.70:0/3918295530 shutdown_connections 2024-06-04T05:23:25.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.850+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/3918295530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a4bdf43f0 0x562a4be679b0 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.850+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/3918295530 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x562a4be75440 0x562a4be77860 crc :-1 s=READY pgs=542 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.850+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/3918295530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a4bdefcf0 0x562a4be6d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.850+0000 7ff425ad7b80 1 --2- 172.21.15.70:0/3918295530 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x562a4bdf06a0 0x562a4be6d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.850+0000 7ff425ad7b80 1 -- 172.21.15.70:0/3918295530 >> 172.21.15.70:0/3918295530 conn(0x562a4bd19100 msgr2=0x562a4bdf8430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:25.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.850+0000 7ff3f3fff700 0 client.22053 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:25.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.851+0000 7ff425ad7b80 1 -- 172.21.15.70:0/3918295530 shutdown_connections 2024-06-04T05:23:25.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.851+0000 7ff425ad7b80 1 -- 172.21.15.70:0/3918295530 wait complete. 2024-06-04T05:23:25.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.851+0000 7ff425ad7b80 20 client.22053 trim_cache size 0 max 16384 2024-06-04T05:23:25.859 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:23:25.859 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-06-04T05:23:25.859 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "aff7208c-f590-42fc-8e7c-1b8ef6de0de9", 2024-06-04T05:23:25.859 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-06-04T05:23:25.860 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:23:25.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.851+0000 7ff41b242700 1 -- 172.21.15.70:0/9265822 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4040b29f0 msgr2=0x7ff4040b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.851+0000 7ff41b242700 1 --2- 172.21.15.70:0/9265822 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4040b29f0 0x7ff4040b4eb0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff408006010 tx=0x7ff40801a040 comp rx=0 tx=0).stop 2024-06-04T05:23:25.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.851+0000 7ff41b242700 1 -- 172.21.15.70:0/9265822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff414102e30 msgr2=0x7ff41419af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:25.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.851+0000 7ff41b242700 1 --2- 172.21.15.70:0/9265822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff414102e30 0x7ff41419af60 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7ff40000b530 tx=0x7ff40000b8f0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.851+0000 7ff41b242700 1 -- 172.21.15.70:0/9265822 shutdown_connections 2024-06-04T05:23:25.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.851+0000 7ff41b242700 1 --2- 172.21.15.70:0/9265822 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff4040b29f0 0x7ff4040b4eb0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.851+0000 7ff41b242700 1 --2- 172.21.15.70:0/9265822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4141084b0 0x7ff4141a5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.851+0000 7ff41b242700 1 --2- 172.21.15.70:0/9265822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff414102470 0x7ff41419aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.851+0000 7ff41b242700 1 --2- 172.21.15.70:0/9265822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff414102e30 0x7ff41419af60 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:25.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.851+0000 7ff41b242700 1 -- 172.21.15.70:0/9265822 >> 172.21.15.70:0/9265822 conn(0x7ff4140784b0 msgr2=0x7ff414100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:25.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.851+0000 7ff41b242700 1 -- 172.21.15.70:0/9265822 shutdown_connections 2024-06-04T05:23:25.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:25.851+0000 7ff41b242700 1 -- 172.21.15.70:0/9265822 wait complete. 2024-06-04T05:23:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:25 smithi112 ceph-mon[36453]: pgmap v828: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 488 KiB/s wr, 147 op/s 2024-06-04T05:23:25.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:25 smithi098 ceph-mon[35569]: pgmap v828: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 488 KiB/s wr, 147 op/s 2024-06-04T05:23:26.566 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:26 smithi070 ceph-mon[32631]: scrub queued for path: / 2024-06-04T05:23:26.566 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:26 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:23:26.566 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:26 smithi070 ceph-mon[32631]: scrub summary: active paths [/] 2024-06-04T05:23:26.566 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:26 smithi070 ceph-mon[32631]: scrub summary: idle+waiting paths [/] 2024-06-04T05:23:26.567 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:26 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:23:26.567 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:26 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:23:26.567 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:26 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:23:26.567 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:26 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:23:26.567 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:26 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:23:26.595 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-04T05:23:26.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:26 smithi112 ceph-mon[36453]: scrub queued for path: / 2024-06-04T05:23:26.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:26 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:23:26.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:26 smithi112 ceph-mon[36453]: scrub summary: active paths [/] 2024-06-04T05:23:26.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:26 smithi112 ceph-mon[36453]: scrub summary: idle+waiting paths [/] 2024-06-04T05:23:26.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:26 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:23:26.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:26 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:23:26.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:26 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:23:26.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:26 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:23:26.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:26 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:23:26.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.002+0000 7f7fa61f6700 1 -- 172.21.15.70:0/2395787912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7fa0102440 msgr2=0x7f7fa01028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:26.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.002+0000 7f7fa61f6700 1 --2- 172.21.15.70:0/2395787912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7fa0102440 0x7f7fa01028c0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f7f94009990 tx=0x7f7f94009ca0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.002+0000 7f7fa61f6700 1 -- 172.21.15.70:0/2395787912 shutdown_connections 2024-06-04T05:23:26.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.002+0000 7f7fa61f6700 1 --2- 172.21.15.70:0/2395787912 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7fa0102e00 0x7f7fa010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.002+0000 7f7fa61f6700 1 --2- 172.21.15.70:0/2395787912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fa0108520 0x7f7fa0108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.002+0000 7f7fa61f6700 1 --2- 172.21.15.70:0/2395787912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7fa0102440 0x7f7fa01028c0 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.002+0000 7f7fa61f6700 1 -- 172.21.15.70:0/2395787912 >> 172.21.15.70:0/2395787912 conn(0x7f7fa00793f0 msgr2=0x7f7fa0079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:26.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.002+0000 7f7fa61f6700 1 -- 172.21.15.70:0/2395787912 shutdown_connections 2024-06-04T05:23:26.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.002+0000 7f7fa61f6700 1 -- 172.21.15.70:0/2395787912 wait complete. 2024-06-04T05:23:26.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.003+0000 7f7fa61f6700 1 Processor -- start 2024-06-04T05:23:26.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.003+0000 7f7fa61f6700 1 -- start start 2024-06-04T05:23:26.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.003+0000 7f7fa61f6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7fa0102440 0x7f7fa019a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.003+0000 7f7fa61f6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fa0102e00 0x7f7fa019aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.003+0000 7f7fa61f6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7fa0108520 0x7f7fa01a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.003+0000 7f7fa61f6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa0112480 con 0x7f7fa0102440 2024-06-04T05:23:26.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.003+0000 7f7fa61f6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa0112300 con 0x7f7fa0102e00 2024-06-04T05:23:26.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.003+0000 7f7fa61f6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa0112600 con 0x7f7fa0108520 2024-06-04T05:23:26.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.003+0000 7f7f9f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7fa0102440 0x7f7fa019a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.003+0000 7f7f9f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7fa0102440 0x7f7fa019a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50442/0 (socket says 172.21.15.70:50442) 2024-06-04T05:23:26.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.003+0000 7f7f9f7fe700 1 -- 172.21.15.70:0/1913214549 learned_addr learned my addr 172.21.15.70:0/1913214549 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:26.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.003+0000 7f7f9ffff700 1 --2- 172.21.15.70:0/1913214549 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7fa0108520 0x7f7fa01a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.004+0000 7f7f9effd700 1 --2- 172.21.15.70:0/1913214549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fa0102e00 0x7f7fa019aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.004+0000 7f7f9f7fe700 1 -- 172.21.15.70:0/1913214549 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7fa0108520 msgr2=0x7f7fa01a5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:26.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.004+0000 7f7f9f7fe700 1 --2- 172.21.15.70:0/1913214549 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7fa0108520 0x7f7fa01a5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.004+0000 7f7f9f7fe700 1 -- 172.21.15.70:0/1913214549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fa0102e00 msgr2=0x7f7fa019aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:26.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.004+0000 7f7f9f7fe700 1 --2- 172.21.15.70:0/1913214549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fa0102e00 0x7f7fa019aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.004+0000 7f7f9f7fe700 1 -- 172.21.15.70:0/1913214549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa01a5f60 con 0x7f7fa0102440 2024-06-04T05:23:26.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.004+0000 7f7f9f7fe700 1 --2- 172.21.15.70:0/1913214549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7fa0102440 0x7f7fa019a9a0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000d950 tx=0x7f7f9000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:26.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.004+0000 7f7f9cff9700 1 -- 172.21.15.70:0/1913214549 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f90016070 con 0x7f7fa0102440 2024-06-04T05:23:26.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.004+0000 7f7f9cff9700 1 -- 172.21.15.70:0/1913214549 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7f90009dd0 con 0x7f7fa0102440 2024-06-04T05:23:26.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.004+0000 7f7fa61f6700 1 -- 172.21.15.70:0/1913214549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fa01a6190 con 0x7f7fa0102440 2024-06-04T05:23:26.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.004+0000 7f7f9cff9700 1 -- 172.21.15.70:0/1913214549 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9000f3c0 con 0x7f7fa0102440 2024-06-04T05:23:26.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.004+0000 7f7fa61f6700 1 -- 172.21.15.70:0/1913214549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fa01a66f0 con 0x7f7fa0102440 2024-06-04T05:23:26.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.007+0000 7f7f9cff9700 1 -- 172.21.15.70:0/1913214549 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7f90019070 con 0x7f7fa0102440 2024-06-04T05:23:26.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.007+0000 7f7f9cff9700 1 --2- 172.21.15.70:0/1913214549 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7f880b2a70 0x7f7f880b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.007+0000 7f7f9effd700 1 --2- 172.21.15.70:0/1913214549 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7f880b2a70 0x7f7f880b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.007+0000 7f7fb0a8bb80 1 Processor -- start 2024-06-04T05:23:26.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.007+0000 7f7f9cff9700 1 -- 172.21.15.70:0/1913214549 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7f7f900e2120 con 0x7f7fa0102440 2024-06-04T05:23:26.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.007+0000 7f7fb0a8bb80 1 -- start start 2024-06-04T05:23:26.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7fb0a8bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce7530 0x55a8ecce7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7fb0a8bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce2c60 0x55a8ecce30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7fb0a8bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce3620 0x55a8eccef450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7fb0a8bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a8ecc3dd00 con 0x55a8ecce7530 2024-06-04T05:23:26.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7fb0a8bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a8ecc2d8a0 con 0x55a8ecce2c60 2024-06-04T05:23:26.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7fb0a8bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a8ecc318e0 con 0x55a8ecce3620 2024-06-04T05:23:26.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7f9f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce7530 0x55a8ecce7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7f9f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce7530 0x55a8ecce7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50450/0 (socket says 172.21.15.70:50450) 2024-06-04T05:23:26.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7f9f7fe700 1 -- 172.21.15.70:0/995002812 learned_addr learned my addr 172.21.15.70:0/995002812 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7f9effd700 1 --2- 172.21.15.70:0/1913214549 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7f880b2a70 0x7f7f880b4f30 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7f94006010 tx=0x7f7f94005c40 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7f9ffff700 1 --2- 172.21.15.70:0/995002812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce3620 0x55a8eccef450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7f9effd700 1 --2- 172.21.15.70:0/995002812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce2c60 0x55a8ecce30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7f9f7fe700 1 -- 172.21.15.70:0/995002812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce3620 msgr2=0x55a8eccef450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7f9f7fe700 1 --2- 172.21.15.70:0/995002812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce3620 0x55a8eccef450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7f9f7fe700 1 -- 172.21.15.70:0/995002812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce2c60 msgr2=0x55a8ecce30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7f9f7fe700 1 --2- 172.21.15.70:0/995002812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce2c60 0x55a8ecce30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7f9f7fe700 1 -- 172.21.15.70:0/995002812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8ecb7d940 con 0x55a8ecce7530 2024-06-04T05:23:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7f9ffff700 1 --2- 172.21.15.70:0/995002812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce3620 0x55a8eccef450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:23:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.008+0000 7f7f9f7fe700 1 --2- 172.21.15.70:0/995002812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce7530 0x55a8ecce7910 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f7f900a54f0 tx=0x7f7f900a8ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17a44745e5f5c623 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.009+0000 7f7f85ffb700 1 -- 172.21.15.70:0/995002812 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f90020070 con 0x55a8ecce7530 2024-06-04T05:23:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.009+0000 7f7f85ffb700 1 -- 172.21.15.70:0/995002812 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7f900ad820 con 0x55a8ecce7530 2024-06-04T05:23:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.009+0000 7f7f85ffb700 1 -- 172.21.15.70:0/995002812 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f900b1e40 con 0x55a8ecce7530 2024-06-04T05:23:26.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.009+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/995002812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce7530 msgr2=0x55a8ecce7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:26.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.009+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/995002812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce7530 0x55a8ecce7910 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f7f900a54f0 tx=0x7f7f900a8ed0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.009+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/995002812 shutdown_connections 2024-06-04T05:23:26.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.009+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/995002812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce3620 0x55a8eccef450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.009+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/995002812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce7530 0x55a8ecce7910 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.009+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/995002812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce2c60 0x55a8ecce30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.009+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/995002812 >> 172.21.15.70:0/995002812 conn(0x55a8ecc0c0f0 msgr2=0x55a8eccd9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:26.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.009+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/995002812 shutdown_connections 2024-06-04T05:23:26.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.009+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/995002812 wait complete. 2024-06-04T05:23:26.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7fb0a8bb80 1 -- start start 2024-06-04T05:23:26.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7fb0a8bb80 1 Processor -- start 2024-06-04T05:23:26.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7fb0a8bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce2c60 0x55a8eccef150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7fb0a8bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce3620 0x55a8eccec450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7fb0a8bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce7530 0x55a8eccec990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7fb0a8bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a8ecb34010 con 0x55a8ecce3620 2024-06-04T05:23:26.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7fb0a8bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a8ecbcf830 con 0x55a8ecce2c60 2024-06-04T05:23:26.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7fb0a8bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a8eccefa50 con 0x55a8ecce7530 2024-06-04T05:23:26.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7fb0a8bb80 10 client.0 _ll_register_callbacks cb 0x55a8ecbc8990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:26.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7f9effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce3620 0x55a8eccec450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7f9effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce3620 0x55a8eccec450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50454/0 (socket says 172.21.15.70:50454) 2024-06-04T05:23:26.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7f9effd700 1 -- 172.21.15.70:0/26079567 learned_addr learned my addr 172.21.15.70:0/26079567 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:26.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7f9ffff700 1 --2- 172.21.15.70:0/26079567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce7530 0x55a8eccec990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7f9f7fe700 1 --2- 172.21.15.70:0/26079567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce2c60 0x55a8eccef150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7f9effd700 1 -- 172.21.15.70:0/26079567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce7530 msgr2=0x55a8eccec990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:26.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7f9effd700 1 --2- 172.21.15.70:0/26079567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce7530 0x55a8eccec990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7f9effd700 1 -- 172.21.15.70:0/26079567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce2c60 msgr2=0x55a8eccef150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:26.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7f9effd700 1 --2- 172.21.15.70:0/26079567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce2c60 0x55a8eccef150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.010+0000 7f7f9effd700 1 -- 172.21.15.70:0/26079567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8ecbcc700 con 0x55a8ecce3620 2024-06-04T05:23:26.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.011+0000 7f7f9f7fe700 1 --2- 172.21.15.70:0/26079567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce2c60 0x55a8eccef150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:26.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.011+0000 7f7f9effd700 1 --2- 172.21.15.70:0/26079567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce3620 0x55a8eccec450 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f7f94005cd0 tx=0x7f7f94009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f2a91432aac0e0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:26.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.011+0000 7f7f7effd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:26.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.011+0000 7f7f7effd700 1 -- 172.21.15.70:0/26079567 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9400b9e0 con 0x55a8ecce3620 2024-06-04T05:23:26.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.011+0000 7f7f7effd700 1 -- 172.21.15.70:0/26079567 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7f7f9401e070 con 0x55a8ecce3620 2024-06-04T05:23:26.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.011+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/26079567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f900ad820 con 0x55a8ecce3620 2024-06-04T05:23:26.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.011+0000 7f7f7effd700 1 -- 172.21.15.70:0/26079567 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f7f9400bb80 con 0x55a8ecce3620 2024-06-04T05:23:26.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.011+0000 7f7fb0a8bb80 10 client.22065 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:26.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.011+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/26079567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x55a8eccfa1d0 con 0x55a8ecce3620 2024-06-04T05:23:26.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.011+0000 7f7f7effd700 1 -- 172.21.15.70:0/26079567 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7f900ad820 con 0x55a8ecce3620 2024-06-04T05:23:26.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.011+0000 7f7f7effd700 1 -- 172.21.15.70:0/26079567 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7f7f94022050 con 0x55a8ecce3620 2024-06-04T05:23:26.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.011+0000 7f7fb0a8bb80 10 client.22065 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:26.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.011+0000 7f7fb0a8bb80 10 client.22065 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:23:26.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.011+0000 7f7fb0a8bb80 20 client.22065 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:26.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.011+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/26079567 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a8eccfc810 0x55a8eccfec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.011+0000 7f7fb0a8bb80 4 client.22065 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:23:26.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.011+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/26079567 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8eccaaa50 con 0x55a8eccfc810 2024-06-04T05:23:26.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.012+0000 7f7f9f7fe700 1 --2- 172.21.15.70:0/26079567 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a8eccfc810 0x55a8eccfec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.012+0000 7f7f7effd700 10 client.22065 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:26.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.012+0000 7f7f9f7fe700 1 --2- 172.21.15.70:0/26079567 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a8eccfc810 0x55a8eccfec80 crc :-1 s=READY pgs=543 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ea2789f7eb0ddae server_cookie=d2e6c6c58e6fee69 in_seq=0 out_seq=0 2024-06-04T05:23:26.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.013+0000 7f7f7effd700 1 -- 172.21.15.70:0/26079567 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a8eccaaa50 con 0x55a8eccfc810 2024-06-04T05:23:26.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.013+0000 7f7f7effd700 10 client.22065 handle_command_reply: tid=0 2024-06-04T05:23:26.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.013+0000 7f7fb0a8bb80 1 client.22065 shutdown 2024-06-04T05:23:26.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.013+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/26079567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce3620 msgr2=0x55a8eccec450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:26.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.013+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/26079567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce3620 0x55a8eccec450 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f7f94005cd0 tx=0x7f7f94009ca0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.013+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/26079567 shutdown_connections 2024-06-04T05:23:26.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.013+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/26079567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce7530 0x55a8eccec990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.013+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/26079567 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a8eccfc810 0x55a8eccfec80 crc :-1 s=READY pgs=543 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.013+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/26079567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce2c60 0x55a8eccef150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.013+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/26079567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce3620 0x55a8eccec450 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.013+0000 7f7f7effd700 0 client.22065 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:26.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.013+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/26079567 >> 172.21.15.70:0/26079567 conn(0x55a8ecc0c0f0 msgr2=0x55a8ecce8bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:26.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.013+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/26079567 shutdown_connections 2024-06-04T05:23:26.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.013+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/26079567 wait complete. 2024-06-04T05:23:26.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.013+0000 7f7fb0a8bb80 20 client.22065 trim_cache size 0 max 16384 2024-06-04T05:23:26.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.033+0000 7f7fb0a8bb80 1 Processor -- start 2024-06-04T05:23:26.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.033+0000 7f7fb0a8bb80 1 -- start start 2024-06-04T05:23:26.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.033+0000 7f7fb0a8bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce2c60 0x55a8eccf8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.033+0000 7f7fb0a8bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce3620 0x55a8ecceec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.033+0000 7f7fb0a8bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce7530 0x55a8ecce6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.033+0000 7f7fb0a8bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa01abc70 con 0x55a8ecce3620 2024-06-04T05:23:26.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.033+0000 7f7fb0a8bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa01ad960 con 0x55a8ecce7530 2024-06-04T05:23:26.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.033+0000 7f7fb0a8bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa01ac3d0 con 0x55a8ecce2c60 2024-06-04T05:23:26.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.033+0000 7f7f9effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce3620 0x55a8ecceec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.033+0000 7f7f9effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce3620 0x55a8ecceec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50468/0 (socket says 172.21.15.70:50468) 2024-06-04T05:23:26.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.033+0000 7f7f9f7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce2c60 0x55a8eccf8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.033+0000 7f7f9effd700 1 -- 172.21.15.70:0/3919789153 learned_addr learned my addr 172.21.15.70:0/3919789153 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:26.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.034+0000 7f7f9ffff700 1 --2- 172.21.15.70:0/3919789153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce7530 0x55a8ecce6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.034+0000 7f7f9effd700 1 -- 172.21.15.70:0/3919789153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce2c60 msgr2=0x55a8eccf8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:26.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.034+0000 7f7f9effd700 1 --2- 172.21.15.70:0/3919789153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce2c60 0x55a8eccf8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.034+0000 7f7f9effd700 1 -- 172.21.15.70:0/3919789153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce7530 msgr2=0x55a8ecce6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:26.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.034+0000 7f7f9effd700 1 --2- 172.21.15.70:0/3919789153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce7530 0x55a8ecce6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.034+0000 7f7f9effd700 1 -- 172.21.15.70:0/3919789153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8eccea9c0 con 0x55a8ecce3620 2024-06-04T05:23:26.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.034+0000 7f7f9f7fe700 1 --2- 172.21.15.70:0/3919789153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce2c60 0x55a8eccf8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:23:26.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.034+0000 7f7f9effd700 1 --2- 172.21.15.70:0/3919789153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce3620 0x55a8ecceec90 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f7f94038480 tx=0x7f7f94005a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66e10083e04f2248 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:26.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.034+0000 7f7f857fa700 1 -- 172.21.15.70:0/3919789153 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9400bd20 con 0x55a8ecce3620 2024-06-04T05:23:26.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.034+0000 7f7f857fa700 1 -- 172.21.15.70:0/3919789153 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7f9400f8f0 con 0x55a8ecce3620 2024-06-04T05:23:26.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.034+0000 7f7f857fa700 1 -- 172.21.15.70:0/3919789153 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f94038910 con 0x55a8ecce3620 2024-06-04T05:23:26.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/3919789153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce3620 msgr2=0x55a8ecceec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:26.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/3919789153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce3620 0x55a8ecceec90 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f7f94038480 tx=0x7f7f94005a10 comp rx=0 tx=0).stop 2024-06-04T05:23:26.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/3919789153 shutdown_connections 2024-06-04T05:23:26.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/3919789153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce7530 0x55a8ecce6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/3919789153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce2c60 0x55a8eccf8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/3919789153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce3620 0x55a8ecceec90 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/3919789153 >> 172.21.15.70:0/3919789153 conn(0x55a8ecc0c0f0 msgr2=0x55a8eccec410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:26.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/3919789153 shutdown_connections 2024-06-04T05:23:26.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/3919789153 wait complete. 2024-06-04T05:23:26.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 1 -- start start 2024-06-04T05:23:26.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 1 Processor -- start 2024-06-04T05:23:26.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce2c60 0x55a8eccf1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce3620 0x55a8eccf22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce7530 0x55a8ecd67470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa01ad260 con 0x55a8ecce7530 2024-06-04T05:23:26.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa01acad0 con 0x55a8ecce2c60 2024-06-04T05:23:26.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa01a9c30 con 0x55a8ecce3620 2024-06-04T05:23:26.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.035+0000 7f7fb0a8bb80 10 client.0 _ll_register_callbacks cb 0x55a8ecbc8990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:26.746 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:23:26.746 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 72, 2024-06-04T05:23:26.746 INFO:teuthology.orchestra.run.smithi070.stdout: "leader": 24448, 2024-06-04T05:23:26.747 INFO:teuthology.orchestra.run.smithi070.stdout: "set_version": 80, 2024-06-04T05:23:26.747 INFO:teuthology.orchestra.run.smithi070.stdout: "sets": {} 2024-06-04T05:23:26.747 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:23:26.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7f9effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce3620 0x55a8eccf22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7f9effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce3620 0x55a8eccf22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:47864/0 (socket says 172.21.15.70:47864) 2024-06-04T05:23:26.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7f9effd700 1 -- 172.21.15.70:0/1716862159 learned_addr learned my addr 172.21.15.70:0/1716862159 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:26.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7f9ffff700 1 --2- 172.21.15.70:0/1716862159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce7530 0x55a8ecd67470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7f9f7fe700 1 --2- 172.21.15.70:0/1716862159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce2c60 0x55a8eccf1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7f9effd700 1 -- 172.21.15.70:0/1716862159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce2c60 msgr2=0x55a8eccf1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:26.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7f9effd700 1 --2- 172.21.15.70:0/1716862159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce2c60 0x55a8eccf1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7f9effd700 1 -- 172.21.15.70:0/1716862159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce7530 msgr2=0x55a8ecd67470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:26.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7f9effd700 1 --2- 172.21.15.70:0/1716862159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce7530 0x55a8ecd67470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7f9effd700 1 -- 172.21.15.70:0/1716862159 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8ecd679b0 con 0x55a8ecce3620 2024-06-04T05:23:26.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7f9ffff700 1 --2- 172.21.15.70:0/1716862159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce7530 0x55a8ecd67470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:23:26.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7f9effd700 1 --2- 172.21.15.70:0/1716862159 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce3620 0x55a8eccf22f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7f94009990 tx=0x7f7f9400bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3df87ddf923ec59e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:26.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7f7effd700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-06-04T05:23:26.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7f7effd700 1 -- 172.21.15.70:0/1716862159 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f94045070 con 0x55a8ecce3620 2024-06-04T05:23:26.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/1716862159 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f9400f8f0 con 0x55a8ecce3620 2024-06-04T05:23:26.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7f7effd700 1 -- 172.21.15.70:0/1716862159 <== mon.1 v2:172.21.15.112:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7f7f9402c040 con 0x55a8ecce3620 2024-06-04T05:23:26.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7f7effd700 1 -- 172.21.15.70:0/1716862159 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f7f9400f670 con 0x55a8ecce3620 2024-06-04T05:23:26.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7fb0a8bb80 10 client.30760 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:26.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.036+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/1716862159 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x55a8ecd68b40 con 0x55a8ecce3620 2024-06-04T05:23:26.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.037+0000 7f7f7effd700 1 -- 172.21.15.70:0/1716862159 <== mon.1 v2:172.21.15.112:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7f9400f8f0 con 0x55a8ecce3620 2024-06-04T05:23:26.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.037+0000 7f7f7effd700 1 -- 172.21.15.70:0/1716862159 <== mon.1 v2:172.21.15.112:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7f7f94022040 con 0x55a8ecce3620 2024-06-04T05:23:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.037+0000 7f7fb0a8bb80 10 client.30760 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.037+0000 7f7fb0a8bb80 10 client.30760 resolve_mds: validated gid 24448 aka daemon mds.i 2024-06-04T05:23:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.037+0000 7f7fb0a8bb80 20 client.30760 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.037+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/1716862159 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a8ecd69180 0x55a8ecd6b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.037+0000 7f7fb0a8bb80 4 client.30760 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-06-04T05:23:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.037+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/1716862159 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x55a8ecc3fc50 con 0x55a8ecd69180 2024-06-04T05:23:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.037+0000 7f7f9f7fe700 1 --2- 172.21.15.70:0/1716862159 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a8ecd69180 0x55a8ecd6b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.038+0000 7f7f7effd700 10 client.30760 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.038+0000 7f7f9f7fe700 1 --2- 172.21.15.70:0/1716862159 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a8ecd69180 0x55a8ecd6b5a0 crc :-1 s=READY pgs=544 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d5a4ea8d25cf044 server_cookie=3d4bc779fefbd970 in_seq=0 out_seq=0 2024-06-04T05:23:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.038+0000 7f7f7effd700 1 -- 172.21.15.70:0/1716862159 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x55a8ecc3fc50 con 0x55a8ecd69180 2024-06-04T05:23:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.038+0000 7f7f7effd700 10 client.30760 handle_command_reply: tid=0 2024-06-04T05:23:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.038+0000 7f7fb0a8bb80 1 client.30760 shutdown 2024-06-04T05:23:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.038+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/1716862159 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce3620 msgr2=0x55a8eccf22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.038+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/1716862159 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce3620 0x55a8eccf22f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7f94009990 tx=0x7f7f9400bf50 comp rx=0 tx=0).stop 2024-06-04T05:23:26.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.038+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/1716862159 shutdown_connections 2024-06-04T05:23:26.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.038+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/1716862159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8ecce7530 0x55a8ecd67470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.038+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/1716862159 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x55a8ecd69180 0x55a8ecd6b5a0 crc :-1 s=READY pgs=544 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.038+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/1716862159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a8ecce2c60 0x55a8eccf1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.038+0000 7f7fb0a8bb80 1 --2- 172.21.15.70:0/1716862159 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a8ecce3620 0x55a8eccf22f0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.038+0000 7f7f7effd700 0 client.30760 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:26.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.038+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/1716862159 >> 172.21.15.70:0/1716862159 conn(0x55a8ecc0c0f0 msgr2=0x55a8ecce6600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:26.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.038+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/1716862159 shutdown_connections 2024-06-04T05:23:26.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.038+0000 7f7fb0a8bb80 1 -- 172.21.15.70:0/1716862159 wait complete. 2024-06-04T05:23:26.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.038+0000 7f7fb0a8bb80 20 client.30760 trim_cache size 0 max 16384 2024-06-04T05:23:26.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.039+0000 7f7fa61f6700 1 -- 172.21.15.70:0/1913214549 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7f880b2a70 msgr2=0x7f7f880b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:26.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.039+0000 7f7fa61f6700 1 --2- 172.21.15.70:0/1913214549 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7f880b2a70 0x7f7f880b4f30 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7f94006010 tx=0x7f7f94005c40 comp rx=0 tx=0).stop 2024-06-04T05:23:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.039+0000 7f7fa61f6700 1 -- 172.21.15.70:0/1913214549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7fa0102440 msgr2=0x7f7fa019a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.039+0000 7f7fa61f6700 1 --2- 172.21.15.70:0/1913214549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7fa0102440 0x7f7fa019a9a0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000d950 tx=0x7f7f9000dc60 comp rx=0 tx=0).stop 2024-06-04T05:23:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.039+0000 7f7fa61f6700 1 -- 172.21.15.70:0/1913214549 shutdown_connections 2024-06-04T05:23:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.039+0000 7f7fa61f6700 1 --2- 172.21.15.70:0/1913214549 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f7f880b2a70 0x7f7f880b4f30 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.039+0000 7f7fa61f6700 1 --2- 172.21.15.70:0/1913214549 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7fa0108520 0x7f7fa01a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.039+0000 7f7fa61f6700 1 --2- 172.21.15.70:0/1913214549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7fa0102440 0x7f7fa019a9a0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.039+0000 7f7fa61f6700 1 --2- 172.21.15.70:0/1913214549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fa0102e00 0x7f7fa019aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.039+0000 7f7fa61f6700 1 -- 172.21.15.70:0/1913214549 >> 172.21.15.70:0/1913214549 conn(0x7f7fa00793f0 msgr2=0x7f7fa01005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.039+0000 7f7fa61f6700 1 -- 172.21.15.70:0/1913214549 shutdown_connections 2024-06-04T05:23:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:26.039+0000 7f7fa61f6700 1 -- 172.21.15.70:0/1913214549 wait complete. 2024-06-04T05:23:26.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:26 smithi098 ceph-mon[35569]: scrub queued for path: / 2024-06-04T05:23:26.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:26 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:23:26.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:26 smithi098 ceph-mon[35569]: scrub summary: active paths [/] 2024-06-04T05:23:26.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:26 smithi098 ceph-mon[35569]: scrub summary: idle+waiting paths [/] 2024-06-04T05:23:26.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:26 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:23:26.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:26 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:23:26.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:26 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:23:26.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:26 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:23:26.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:26 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:23:27.387 INFO:tasks.quiescer:done joining Quiescers 2024-06-04T05:23:27.387 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-06-04T05:23:27.398 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi070.front.sepia.ceph.com 2024-06-04T05:23:27.409 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-06-04T05:23:27.421 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-06-04T05:23:27.421 INFO:teuthology.task.exec:Executing custom commands... 2024-06-04T05:23:27.421 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi070.front.sepia.ceph.com 2024-06-04T05:23:27.422 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-06-04T05:23:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:27.494 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-06-04T05:23:27.612 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:27 smithi070 ceph-mon[32631]: pgmap v829: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 481 KiB/s wr, 158 op/s 2024-06-04T05:23:27.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-06-04T05:23:27.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:27 smithi112 ceph-mon[36453]: pgmap v829: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 481 KiB/s wr, 158 op/s 2024-06-04T05:23:27.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.718+0000 7f815bf66640 1 -- 172.21.15.70:0/2237341567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8154102a80 msgr2=0x7f815410e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:27.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.718+0000 7f815bf66640 1 --2- 172.21.15.70:0/2237341567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8154102a80 0x7f815410e7b0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f814c009ca0 tx=0x7f814c02a8b0 comp rx=0 tx=0).stop 2024-06-04T05:23:27.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.718+0000 7f815bf66640 1 -- 172.21.15.70:0/2237341567 shutdown_connections 2024-06-04T05:23:27.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.718+0000 7f815bf66640 1 --2- 172.21.15.70:0/2237341567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8154102a80 0x7f815410e7b0 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:27.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.718+0000 7f815bf66640 1 --2- 172.21.15.70:0/2237341567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81541020e0 0x7f8154102540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:27.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.718+0000 7f815bf66640 1 --2- 172.21.15.70:0/2237341567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8154106940 0x7f8154106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:27.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.718+0000 7f815bf66640 1 -- 172.21.15.70:0/2237341567 >> 172.21.15.70:0/2237341567 conn(0x7f8154077ee0 msgr2=0x7f81540ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:27.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.718+0000 7f815bf66640 1 -- 172.21.15.70:0/2237341567 shutdown_connections 2024-06-04T05:23:27.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.718+0000 7f815bf66640 1 -- 172.21.15.70:0/2237341567 wait complete. 2024-06-04T05:23:27.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.718+0000 7f815bf66640 1 Processor -- start 2024-06-04T05:23:27.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f815bf66640 1 -- start start 2024-06-04T05:23:27.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f815bf66640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81541020e0 0x7f8154198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:27.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f815bf66640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8154102a80 0x7f8154198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:27.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f815bf66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8154106940 0x7f815419c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:27.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f815bf66640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8154111ba0 con 0x7f81541020e0 2024-06-04T05:23:27.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f815bf66640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8154111a20 con 0x7f8154106940 2024-06-04T05:23:27.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f815bf66640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8154111d20 con 0x7f8154102a80 2024-06-04T05:23:27.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f8159d02640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81541020e0 0x7f8154198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:27.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f8159d02640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81541020e0 0x7f8154198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50500/0 (socket says 172.21.15.70:50500) 2024-06-04T05:23:27.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f8159d02640 1 -- 172.21.15.70:0/604982602 learned_addr learned my addr 172.21.15.70:0/604982602 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:27.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f8159501640 1 --2- 172.21.15.70:0/604982602 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8154102a80 0x7f8154198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:27.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f815a503640 1 --2- 172.21.15.70:0/604982602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8154106940 0x7f815419c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:27.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f8159d02640 1 -- 172.21.15.70:0/604982602 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8154102a80 msgr2=0x7f8154198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:27.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f8159d02640 1 --2- 172.21.15.70:0/604982602 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8154102a80 0x7f8154198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:27.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f8159d02640 1 -- 172.21.15.70:0/604982602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8154106940 msgr2=0x7f815419c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:27.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f8159d02640 1 --2- 172.21.15.70:0/604982602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8154106940 0x7f815419c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:27.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f8159d02640 1 -- 172.21.15.70:0/604982602 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f815419d010 con 0x7f81541020e0 2024-06-04T05:23:27.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f815a503640 1 --2- 172.21.15.70:0/604982602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8154106940 0x7f815419c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:27.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f8159d02640 1 --2- 172.21.15.70:0/604982602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81541020e0 0x7f8154198020 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f814800e9f0 tx=0x7f814800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:27.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f8146ffd640 1 -- 172.21.15.70:0/604982602 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f814800cd80 con 0x7f81541020e0 2024-06-04T05:23:27.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f8146ffd640 1 -- 172.21.15.70:0/604982602 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f81480108c0 con 0x7f81541020e0 2024-06-04T05:23:27.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f8146ffd640 1 -- 172.21.15.70:0/604982602 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81480187c0 con 0x7f81541020e0 2024-06-04T05:23:27.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f815bf66640 1 -- 172.21.15.70:0/604982602 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f815419d300 con 0x7f81541020e0 2024-06-04T05:23:27.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.719+0000 7f815bf66640 1 -- 172.21.15.70:0/604982602 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81541a4b90 con 0x7f81541020e0 2024-06-04T05:23:27.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.721+0000 7f815bf66640 1 -- 172.21.15.70:0/604982602 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81540507e0 con 0x7f81541020e0 2024-06-04T05:23:27.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.723+0000 7f8146ffd640 1 -- 172.21.15.70:0/604982602 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f8148010430 con 0x7f81541020e0 2024-06-04T05:23:27.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.724+0000 7f8146ffd640 1 --2- 172.21.15.70:0/604982602 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81400b2a90 0x7f81400b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:27.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.724+0000 7f8159501640 1 --2- 172.21.15.70:0/604982602 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81400b2a90 0x7f81400b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:27.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.724+0000 7f8146ffd640 1 -- 172.21.15.70:0/604982602 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7f81480e2e40 con 0x7f81541020e0 2024-06-04T05:23:27.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.724+0000 7f8159501640 1 --2- 172.21.15.70:0/604982602 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81400b2a90 0x7f81400b4f50 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8154106140 tx=0x7f815000b410 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:27.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.727+0000 7f8146ffd640 1 -- 172.21.15.70:0/604982602 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f81480aa7f0 con 0x7f81541020e0 2024-06-04T05:23:27.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.938+0000 7f815bf66640 1 -- 172.21.15.70:0/604982602 --> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f8154062e90 con 0x7f81400b2a90 2024-06-04T05:23:27.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.948+0000 7f8146ffd640 1 -- 172.21.15.70:0/604982602 <== mgr.24806 v2:172.21.15.70:6800/542036627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7f8154062e90 con 0x7f81400b2a90 2024-06-04T05:23:27.951 INFO:teuthology.orchestra.run.smithi070.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-06-04T00:00:00", "created": "2024-06-04T04:57:22", "first": "2024-06-04T04:58:00", "last": "2024-06-04T05:23:00", "last_pruned": "2024-06-04T05:23:05", "created_count": 26, "pruned_count": 19, "active": true} 2024-06-04T05:23:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.950+0000 7f815bf66640 1 -- 172.21.15.70:0/604982602 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81400b2a90 msgr2=0x7f81400b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.950+0000 7f815bf66640 1 --2- 172.21.15.70:0/604982602 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81400b2a90 0x7f81400b4f50 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8154106140 tx=0x7f815000b410 comp rx=0 tx=0).stop 2024-06-04T05:23:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.950+0000 7f815bf66640 1 -- 172.21.15.70:0/604982602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81541020e0 msgr2=0x7f8154198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.950+0000 7f815bf66640 1 --2- 172.21.15.70:0/604982602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81541020e0 0x7f8154198020 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f814800e9f0 tx=0x7f814800eeb0 comp rx=0 tx=0).stop 2024-06-04T05:23:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.950+0000 7f815bf66640 1 -- 172.21.15.70:0/604982602 shutdown_connections 2024-06-04T05:23:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.950+0000 7f815bf66640 1 --2- 172.21.15.70:0/604982602 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f81400b2a90 0x7f81400b4f50 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.950+0000 7f815bf66640 1 --2- 172.21.15.70:0/604982602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8154106940 0x7f815419c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.950+0000 7f815bf66640 1 --2- 172.21.15.70:0/604982602 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8154102a80 0x7f8154198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.950+0000 7f815bf66640 1 --2- 172.21.15.70:0/604982602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81541020e0 0x7f8154198020 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.950+0000 7f815bf66640 1 -- 172.21.15.70:0/604982602 >> 172.21.15.70:0/604982602 conn(0x7f8154077ee0 msgr2=0x7f8154104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:27.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.950+0000 7f815bf66640 1 -- 172.21.15.70:0/604982602 shutdown_connections 2024-06-04T05:23:27.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:27.950+0000 7f815bf66640 1 -- 172.21.15.70:0/604982602 wait complete. 2024-06-04T05:23:27.964 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-06-04T05:23:27.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:27 smithi098 ceph-mon[35569]: pgmap v829: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 481 KiB/s wr, 158 op/s 2024-06-04T05:23:28.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.066+0000 7ff52806d640 1 -- 172.21.15.70:0/1327813150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff520108a50 msgr2=0x7ff520108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:28.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.066+0000 7ff52806d640 1 --2- 172.21.15.70:0/1327813150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff520108a50 0x7ff520108e30 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7ff510009930 tx=0x7ff51002a420 comp rx=0 tx=0).stop 2024-06-04T05:23:28.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.066+0000 7ff52806d640 1 -- 172.21.15.70:0/1327813150 shutdown_connections 2024-06-04T05:23:28.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.066+0000 7ff52806d640 1 --2- 172.21.15.70:0/1327813150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52006bbf0 0x7ff52010f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:28.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.066+0000 7ff52806d640 1 --2- 172.21.15.70:0/1327813150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52006b230 0x7ff52006b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:28.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.066+0000 7ff52806d640 1 --2- 172.21.15.70:0/1327813150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff520108a50 0x7ff520108e30 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:28.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.066+0000 7ff52806d640 1 -- 172.21.15.70:0/1327813150 >> 172.21.15.70:0/1327813150 conn(0x7ff520078730 msgr2=0x7ff520078b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:28.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.066+0000 7ff52806d640 1 -- 172.21.15.70:0/1327813150 shutdown_connections 2024-06-04T05:23:28.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.066+0000 7ff52806d640 1 -- 172.21.15.70:0/1327813150 wait complete. 2024-06-04T05:23:28.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.067+0000 7ff52806d640 1 Processor -- start 2024-06-04T05:23:28.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.067+0000 7ff52806d640 1 -- start start 2024-06-04T05:23:28.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.067+0000 7ff52806d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52006b230 0x7ff520101d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.067+0000 7ff52806d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52006bbf0 0x7ff520102240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.067+0000 7ff52806d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff520108a50 0x7ff520102780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.067+0000 7ff52806d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff520112be0 con 0x7ff520108a50 2024-06-04T05:23:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.067+0000 7ff52806d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff520112a60 con 0x7ff52006bbf0 2024-06-04T05:23:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.067+0000 7ff52806d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff520112d60 con 0x7ff52006b230 2024-06-04T05:23:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.067+0000 7ff52660a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff520108a50 0x7ff520102780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.067+0000 7ff52660a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff520108a50 0x7ff520102780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50516/0 (socket says 172.21.15.70:50516) 2024-06-04T05:23:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.067+0000 7ff52660a640 1 -- 172.21.15.70:0/571894094 learned_addr learned my addr 172.21.15.70:0/571894094 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.067+0000 7ff525e09640 1 --2- 172.21.15.70:0/571894094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52006b230 0x7ff520101d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.067+0000 7ff52660a640 1 -- 172.21.15.70:0/571894094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52006b230 msgr2=0x7ff520101d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.067+0000 7ff52660a640 1 --2- 172.21.15.70:0/571894094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52006b230 0x7ff520101d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:28.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.068+0000 7ff52660a640 1 -- 172.21.15.70:0/571894094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52006bbf0 msgr2=0x7ff520102240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:28.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.068+0000 7ff525608640 1 --2- 172.21.15.70:0/571894094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52006bbf0 0x7ff520102240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:28.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.068+0000 7ff52660a640 1 --2- 172.21.15.70:0/571894094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52006bbf0 0x7ff520102240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:28.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.068+0000 7ff52660a640 1 -- 172.21.15.70:0/571894094 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5201a7600 con 0x7ff520108a50 2024-06-04T05:23:28.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.068+0000 7ff525608640 1 --2- 172.21.15.70:0/571894094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52006bbf0 0x7ff520102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:23:28.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.068+0000 7ff52660a640 1 --2- 172.21.15.70:0/571894094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff520108a50 0x7ff520102780 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7ff518009fd0 tx=0x7ff51800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:28.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.068+0000 7ff516ffd640 1 -- 172.21.15.70:0/571894094 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff518017070 con 0x7ff520108a50 2024-06-04T05:23:28.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.068+0000 7ff52806d640 1 -- 172.21.15.70:0/571894094 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5201a78f0 con 0x7ff520108a50 2024-06-04T05:23:28.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.068+0000 7ff52806d640 1 -- 172.21.15.70:0/571894094 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5201a7e40 con 0x7ff520108a50 2024-06-04T05:23:28.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.069+0000 7ff516ffd640 1 -- 172.21.15.70:0/571894094 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff518003e00 con 0x7ff520108a50 2024-06-04T05:23:28.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.069+0000 7ff52806d640 1 -- 172.21.15.70:0/571894094 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff520104180 con 0x7ff520108a50 2024-06-04T05:23:28.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.069+0000 7ff516ffd640 1 -- 172.21.15.70:0/571894094 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff518020060 con 0x7ff520108a50 2024-06-04T05:23:28.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.071+0000 7ff516ffd640 1 -- 172.21.15.70:0/571894094 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff518003fb0 con 0x7ff520108a50 2024-06-04T05:23:28.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.072+0000 7ff516ffd640 1 --2- 172.21.15.70:0/571894094 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff50c0b2a40 0x7ff50c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:28.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.072+0000 7ff525e09640 1 --2- 172.21.15.70:0/571894094 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff50c0b2a40 0x7ff50c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:28.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.072+0000 7ff516ffd640 1 -- 172.21.15.70:0/571894094 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7ff5180e2370 con 0x7ff520108a50 2024-06-04T05:23:28.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.072+0000 7ff525e09640 1 --2- 172.21.15.70:0/571894094 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff50c0b2a40 0x7ff50c0b4f00 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff5100057c0 tx=0x7ff510005a40 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:28.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.075+0000 7ff516ffd640 1 -- 172.21.15.70:0/571894094 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff5180a9d20 con 0x7ff520108a50 2024-06-04T05:23:28.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.281+0000 7ff52806d640 1 -- 172.21.15.70:0/571894094 --> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7ff520062e90 con 0x7ff50c0b2a40 2024-06-04T05:23:28.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.291+0000 7ff516ffd640 1 -- 172.21.15.70:0/571894094 <== mgr.24806 v2:172.21.15.70:6800/542036627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7ff520062e90 con 0x7ff50c0b2a40 2024-06-04T05:23:28.294 INFO:teuthology.orchestra.run.smithi070.stdout:/ 1m 6m3h 2024-06-04T05:23:28.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.293+0000 7ff52806d640 1 -- 172.21.15.70:0/571894094 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff50c0b2a40 msgr2=0x7ff50c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:28.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.293+0000 7ff52806d640 1 --2- 172.21.15.70:0/571894094 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff50c0b2a40 0x7ff50c0b4f00 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff5100057c0 tx=0x7ff510005a40 comp rx=0 tx=0).stop 2024-06-04T05:23:28.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.293+0000 7ff52806d640 1 -- 172.21.15.70:0/571894094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff520108a50 msgr2=0x7ff520102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:28.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.293+0000 7ff52806d640 1 --2- 172.21.15.70:0/571894094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff520108a50 0x7ff520102780 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7ff518009fd0 tx=0x7ff51800b260 comp rx=0 tx=0).stop 2024-06-04T05:23:28.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.293+0000 7ff52806d640 1 -- 172.21.15.70:0/571894094 shutdown_connections 2024-06-04T05:23:28.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.293+0000 7ff52806d640 1 --2- 172.21.15.70:0/571894094 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff50c0b2a40 0x7ff50c0b4f00 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:28.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.293+0000 7ff52806d640 1 --2- 172.21.15.70:0/571894094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52006bbf0 0x7ff520102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:28.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.293+0000 7ff52806d640 1 --2- 172.21.15.70:0/571894094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff520108a50 0x7ff520102780 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:28.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.293+0000 7ff52806d640 1 --2- 172.21.15.70:0/571894094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52006b230 0x7ff520101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:28.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.293+0000 7ff52806d640 1 -- 172.21.15.70:0/571894094 >> 172.21.15.70:0/571894094 conn(0x7ff520078730 msgr2=0x7ff5200ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:28.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.293+0000 7ff52806d640 1 -- 172.21.15.70:0/571894094 shutdown_connections 2024-06-04T05:23:28.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:28.293+0000 7ff52806d640 1 -- 172.21.15.70:0/571894094 wait complete. 2024-06-04T05:23:28.308 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-06-04T05:23:28.319 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-06-04T05:23:28.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:28 smithi070 ceph-mon[32631]: scrub summary: idle 2024-06-04T05:23:28.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:28 smithi112 ceph-mon[36453]: scrub summary: idle 2024-06-04T05:23:28.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:28 smithi098 ceph-mon[35569]: scrub summary: idle 2024-06-04T05:23:29.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.262+0000 7f2371d2c700 1 -- 172.21.15.70:0/4273250898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f236c1041a0 msgr2=0x7f236c1081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:29.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.262+0000 7f2371d2c700 1 --2- 172.21.15.70:0/4273250898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f236c1041a0 0x7f236c1081c0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f23600099c0 tx=0x7f2360009cd0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.262+0000 7f2371d2c700 1 -- 172.21.15.70:0/4273250898 shutdown_connections 2024-06-04T05:23:29.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.262+0000 7f2371d2c700 1 --2- 172.21.15.70:0/4273250898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f236c1088f0 0x7f236c110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.262+0000 7f2371d2c700 1 --2- 172.21.15.70:0/4273250898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f236c1037f0 0x7f236c103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.262+0000 7f2371d2c700 1 --2- 172.21.15.70:0/4273250898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f236c1041a0 0x7f236c1081c0 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.262+0000 7f2371d2c700 1 -- 172.21.15.70:0/4273250898 >> 172.21.15.70:0/4273250898 conn(0x7f236c07a6a0 msgr2=0x7f236c07aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.262+0000 7f2371d2c700 1 -- 172.21.15.70:0/4273250898 shutdown_connections 2024-06-04T05:23:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.262+0000 7f2371d2c700 1 -- 172.21.15.70:0/4273250898 wait complete. 2024-06-04T05:23:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f2371d2c700 1 Processor -- start 2024-06-04T05:23:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f2371d2c700 1 -- start start 2024-06-04T05:23:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f2371d2c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f236c1037f0 0x7f236c075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f2371d2c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f236c1041a0 0x7f236c075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f2371d2c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f236c1088f0 0x7f236c0762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f2371d2c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f236c113580 con 0x7f236c1037f0 2024-06-04T05:23:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f2371d2c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f236c113400 con 0x7f236c1041a0 2024-06-04T05:23:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f2371d2c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f236c113700 con 0x7f236c1088f0 2024-06-04T05:23:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f236b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f236c1037f0 0x7f236c075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f236b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f236c1037f0 0x7f236c075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50544/0 (socket says 172.21.15.70:50544) 2024-06-04T05:23:29.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f236b7fe700 1 -- 172.21.15.70:0/2726069938 learned_addr learned my addr 172.21.15.70:0/2726069938 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:29.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f236bfff700 1 --2- 172.21.15.70:0/2726069938 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f236c1088f0 0x7f236c0762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:29.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f236affd700 1 --2- 172.21.15.70:0/2726069938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f236c1041a0 0x7f236c075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:29.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f236b7fe700 1 -- 172.21.15.70:0/2726069938 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f236c1088f0 msgr2=0x7f236c0762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:29.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f236b7fe700 1 --2- 172.21.15.70:0/2726069938 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f236c1088f0 0x7f236c0762c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f236b7fe700 1 -- 172.21.15.70:0/2726069938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f236c1041a0 msgr2=0x7f236c075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:29.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f236b7fe700 1 --2- 172.21.15.70:0/2726069938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f236c1041a0 0x7f236c075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f236b7fe700 1 -- 172.21.15.70:0/2726069938 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f236c1a7f80 con 0x7f236c1037f0 2024-06-04T05:23:29.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.263+0000 7f236affd700 1 --2- 172.21.15.70:0/2726069938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f236c1041a0 0x7f236c075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:29.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.264+0000 7f236b7fe700 1 --2- 172.21.15.70:0/2726069938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f236c1037f0 0x7f236c075840 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f235c00d980 tx=0x7f235c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:29.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.264+0000 7f2368ff9700 1 -- 172.21.15.70:0/2726069938 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235c017070 con 0x7f236c1037f0 2024-06-04T05:23:29.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.264+0000 7f2368ff9700 1 -- 172.21.15.70:0/2726069938 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f235c005020 con 0x7f236c1037f0 2024-06-04T05:23:29.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.264+0000 7f2368ff9700 1 -- 172.21.15.70:0/2726069938 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235c009ab0 con 0x7f236c1037f0 2024-06-04T05:23:29.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.264+0000 7f2371d2c700 1 -- 172.21.15.70:0/2726069938 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f236c1a8270 con 0x7f236c1037f0 2024-06-04T05:23:29.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.264+0000 7f2371d2c700 1 -- 172.21.15.70:0/2726069938 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f236c1a8780 con 0x7f236c1037f0 2024-06-04T05:23:29.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.266+0000 7f2368ff9700 1 -- 172.21.15.70:0/2726069938 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f235c0051d0 con 0x7f236c1037f0 2024-06-04T05:23:29.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.267+0000 7f2368ff9700 1 --2- 172.21.15.70:0/2726069938 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f23540b2a70 0x7f23540b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.267+0000 7f236affd700 1 --2- 172.21.15.70:0/2726069938 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f23540b2a70 0x7f23540b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.267+0000 7f2368ff9700 1 -- 172.21.15.70:0/2726069938 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7f235c0e24d0 con 0x7f236c1037f0 2024-06-04T05:23:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.268+0000 7f236affd700 1 --2- 172.21.15.70:0/2726069938 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f23540b2a70 0x7f23540b4f30 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2360000c00 tx=0x7f236001a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.268+0000 7f237c5c1b80 1 Processor -- start 2024-06-04T05:23:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.268+0000 7f237c5c1b80 1 -- start start 2024-06-04T05:23:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.268+0000 7f237c5c1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c2375e320 0x560c2375e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.268+0000 7f237c5c1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c237588a0 0x560c23758d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.268+0000 7f237c5c1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c23759260 0x560c23765080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.268+0000 7f237c5c1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560c236b6d70 con 0x560c23759260 2024-06-04T05:23:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.268+0000 7f237c5c1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560c236a6910 con 0x560c2375e320 2024-06-04T05:23:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.268+0000 7f237c5c1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560c236aa950 con 0x560c237588a0 2024-06-04T05:23:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.269+0000 7f236bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c23759260 0x560c23765080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.269+0000 7f236bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c23759260 0x560c23765080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50548/0 (socket says 172.21.15.70:50548) 2024-06-04T05:23:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.269+0000 7f236bfff700 1 -- 172.21.15.70:0/71252191 learned_addr learned my addr 172.21.15.70:0/71252191 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.269+0000 7f236b7fe700 1 --2- 172.21.15.70:0/71252191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c2375e320 0x560c2375e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.269+0000 7f236affd700 1 --2- 172.21.15.70:0/71252191 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c237588a0 0x560c23758d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.269+0000 7f236bfff700 1 -- 172.21.15.70:0/71252191 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c237588a0 msgr2=0x560c23758d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.269+0000 7f236bfff700 1 --2- 172.21.15.70:0/71252191 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c237588a0 0x560c23758d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.269+0000 7f236bfff700 1 -- 172.21.15.70:0/71252191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c2375e320 msgr2=0x560c2375e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.269+0000 7f236bfff700 1 --2- 172.21.15.70:0/71252191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c2375e320 0x560c2375e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.269+0000 7f236bfff700 1 -- 172.21.15.70:0/71252191 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c2364ffb0 con 0x560c23759260 2024-06-04T05:23:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.269+0000 7f236bfff700 1 --2- 172.21.15.70:0/71252191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c23759260 0x560c23765080 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f236400f410 tx=0x7f236400f720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c314a0faf25a1080 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.269+0000 7f2351ffb700 1 -- 172.21.15.70:0/71252191 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2364018030 con 0x560c23759260 2024-06-04T05:23:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.269+0000 7f2351ffb700 1 -- 172.21.15.70:0/71252191 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2364004bb0 con 0x560c23759260 2024-06-04T05:23:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.269+0000 7f2351ffb700 1 -- 172.21.15.70:0/71252191 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2364005020 con 0x560c23759260 2024-06-04T05:23:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.270+0000 7f237c5c1b80 1 -- 172.21.15.70:0/71252191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c23759260 msgr2=0x560c23765080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.270+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/71252191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c23759260 0x560c23765080 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f236400f410 tx=0x7f236400f720 comp rx=0 tx=0).stop 2024-06-04T05:23:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.270+0000 7f237c5c1b80 1 -- 172.21.15.70:0/71252191 shutdown_connections 2024-06-04T05:23:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.270+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/71252191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c23759260 0x560c23765080 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.270+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/71252191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c2375e320 0x560c2375e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.270+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/71252191 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c237588a0 0x560c23758d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.270+0000 7f237c5c1b80 1 -- 172.21.15.70:0/71252191 >> 172.21.15.70:0/71252191 conn(0x560c23684240 msgr2=0x560c237533e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.270+0000 7f237c5c1b80 1 -- 172.21.15.70:0/71252191 shutdown_connections 2024-06-04T05:23:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.270+0000 7f237c5c1b80 1 -- 172.21.15.70:0/71252191 wait complete. 2024-06-04T05:23:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.271+0000 7f237c5c1b80 1 -- start start 2024-06-04T05:23:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.271+0000 7f237c5c1b80 1 Processor -- start 2024-06-04T05:23:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.271+0000 7f237c5c1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c237588a0 0x560c23764d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.271+0000 7f237c5c1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c23759260 0x560c23762000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.271+0000 7f237c5c1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c2375e320 0x560c23762540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.271+0000 7f237c5c1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560c235acfa0 con 0x560c237588a0 2024-06-04T05:23:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.271+0000 7f237c5c1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560c2360eda0 con 0x560c2375e320 2024-06-04T05:23:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.271+0000 7f237c5c1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560c237656e0 con 0x560c23759260 2024-06-04T05:23:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.271+0000 7f237c5c1b80 10 client.0 _ll_register_callbacks cb 0x560c23641950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.271+0000 7f236b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c237588a0 0x560c23764d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.271+0000 7f236b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c237588a0 0x560c23764d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50554/0 (socket says 172.21.15.70:50554) 2024-06-04T05:23:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.271+0000 7f236b7fe700 1 -- 172.21.15.70:0/2585624244 learned_addr learned my addr 172.21.15.70:0/2585624244 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f236bfff700 1 --2- 172.21.15.70:0/2585624244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c2375e320 0x560c23762540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f236affd700 1 --2- 172.21.15.70:0/2585624244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c23759260 0x560c23762000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f236b7fe700 1 -- 172.21.15.70:0/2585624244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c23759260 msgr2=0x560c23762000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f236b7fe700 1 --2- 172.21.15.70:0/2585624244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c23759260 0x560c23762000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f236b7fe700 1 -- 172.21.15.70:0/2585624244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c2375e320 msgr2=0x560c23762540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f236b7fe700 1 --2- 172.21.15.70:0/2585624244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c2375e320 0x560c23762540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f236b7fe700 1 -- 172.21.15.70:0/2585624244 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c235f68d0 con 0x560c237588a0 2024-06-04T05:23:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f236affd700 1 --2- 172.21.15.70:0/2585624244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c23759260 0x560c23762000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f236bfff700 1 --2- 172.21.15.70:0/2585624244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c2375e320 0x560c23762540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f236b7fe700 1 --2- 172.21.15.70:0/2585624244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c237588a0 0x560c23764d00 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f235c0a69f0 tx=0x7f235c009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd81ef06451d77c server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f234affd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f234affd700 1 -- 172.21.15.70:0/2585624244 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235c0a9780 con 0x560c237588a0 2024-06-04T05:23:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f234affd700 1 -- 172.21.15.70:0/2585624244 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7f235c0912e0 con 0x560c237588a0 2024-06-04T05:23:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f237c5c1b80 1 -- 172.21.15.70:0/2585624244 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2364004bb0 con 0x560c237588a0 2024-06-04T05:23:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f234affd700 1 -- 172.21.15.70:0/2585624244 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f235c0c7020 con 0x560c237588a0 2024-06-04T05:23:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f237c5c1b80 10 client.22095 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f237c5c1b80 1 -- 172.21.15.70:0/2585624244 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x560c23660760 con 0x560c237588a0 2024-06-04T05:23:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.272+0000 7f234affd700 1 -- 172.21.15.70:0/2585624244 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2364004bb0 con 0x560c237588a0 2024-06-04T05:23:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.273+0000 7f234affd700 1 -- 172.21.15.70:0/2585624244 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7f235c0bebd0 con 0x560c237588a0 2024-06-04T05:23:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.273+0000 7f237c5c1b80 10 client.22095 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.273+0000 7f237c5c1b80 10 client.22095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:23:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.273+0000 7f237c5c1b80 20 client.22095 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.273+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/2585624244 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560c237722b0 0x560c237746d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.273+0000 7f237c5c1b80 4 client.22095 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:23:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.273+0000 7f237c5c1b80 1 -- 172.21.15.70:0/2585624244 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c236b8cc0 con 0x560c237722b0 2024-06-04T05:23:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.273+0000 7f236affd700 1 --2- 172.21.15.70:0/2585624244 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560c237722b0 0x560c237746d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.274+0000 7f234affd700 10 client.22095 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.274+0000 7f236affd700 1 --2- 172.21.15.70:0/2585624244 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560c237722b0 0x560c237746d0 crc :-1 s=READY pgs=545 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d22f0e4255e98ecf server_cookie=468085fa25b6f72e in_seq=0 out_seq=0 2024-06-04T05:23:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.274+0000 7f234affd700 1 -- 172.21.15.70:0/2585624244 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560c236b8cc0 con 0x560c237722b0 2024-06-04T05:23:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.274+0000 7f234affd700 10 client.22095 handle_command_reply: tid=0 2024-06-04T05:23:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.274+0000 7f237c5c1b80 1 client.22095 shutdown 2024-06-04T05:23:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.275+0000 7f237c5c1b80 1 -- 172.21.15.70:0/2585624244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c237588a0 msgr2=0x560c23764d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.275+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/2585624244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c237588a0 0x560c23764d00 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f235c0a69f0 tx=0x7f235c009f30 comp rx=0 tx=0).stop 2024-06-04T05:23:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.275+0000 7f237c5c1b80 1 -- 172.21.15.70:0/2585624244 shutdown_connections 2024-06-04T05:23:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.275+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/2585624244 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560c237722b0 0x560c237746d0 crc :-1 s=READY pgs=545 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.275+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/2585624244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c2375e320 0x560c23762540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.275+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/2585624244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c237588a0 0x560c23764d00 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.275+0000 7f234affd700 0 client.22095 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.275+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/2585624244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c23759260 0x560c23762000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.275+0000 7f237c5c1b80 1 -- 172.21.15.70:0/2585624244 >> 172.21.15.70:0/2585624244 conn(0x560c23684240 msgr2=0x560c23752af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.275+0000 7f237c5c1b80 1 -- 172.21.15.70:0/2585624244 shutdown_connections 2024-06-04T05:23:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.275+0000 7f237c5c1b80 1 -- 172.21.15.70:0/2585624244 wait complete. 2024-06-04T05:23:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.275+0000 7f237c5c1b80 20 client.22095 trim_cache size 0 max 16384 2024-06-04T05:23:29.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.294+0000 7f237c5c1b80 1 Processor -- start 2024-06-04T05:23:29.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f237c5c1b80 1 -- start start 2024-06-04T05:23:29.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f237c5c1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c237588a0 0x560c23753270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f237c5c1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c23759260 0x560c2375f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f237c5c1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c2375e320 0x560c2375fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f237c5c1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f236c1affa0 con 0x560c23759260 2024-06-04T05:23:29.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f237c5c1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f236c1b1c90 con 0x560c2375e320 2024-06-04T05:23:29.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f237c5c1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f236c1b0700 con 0x560c237588a0 2024-06-04T05:23:29.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f236affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c23759260 0x560c2375f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:29.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f236affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c23759260 0x560c2375f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50558/0 (socket says 172.21.15.70:50558) 2024-06-04T05:23:29.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f236affd700 1 -- 172.21.15.70:0/970015286 learned_addr learned my addr 172.21.15.70:0/970015286 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:29.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f236affd700 1 -- 172.21.15.70:0/970015286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c237588a0 msgr2=0x560c23753270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:29.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f236affd700 1 --2- 172.21.15.70:0/970015286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c237588a0 0x560c23753270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f236affd700 1 -- 172.21.15.70:0/970015286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c2375e320 msgr2=0x560c2375fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:29.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f236affd700 1 --2- 172.21.15.70:0/970015286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c2375e320 0x560c2375fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f236affd700 1 -- 172.21.15.70:0/970015286 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c23755380 con 0x560c23759260 2024-06-04T05:23:29.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f236affd700 1 --2- 172.21.15.70:0/970015286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c23759260 0x560c2375f720 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f2360005850 tx=0x7f236001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c9e97706d56b707 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f23517fa700 1 -- 172.21.15.70:0/970015286 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23600054e0 con 0x560c23759260 2024-06-04T05:23:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.295+0000 7f23517fa700 1 -- 172.21.15.70:0/970015286 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f236000f460 con 0x560c23759260 2024-06-04T05:23:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.296+0000 7f23517fa700 1 -- 172.21.15.70:0/970015286 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2360023660 con 0x560c23759260 2024-06-04T05:23:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.296+0000 7f237c5c1b80 1 -- 172.21.15.70:0/970015286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c23759260 msgr2=0x560c2375f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.296+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/970015286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c23759260 0x560c2375f720 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f2360005850 tx=0x7f236001e000 comp rx=0 tx=0).stop 2024-06-04T05:23:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.296+0000 7f237c5c1b80 1 -- 172.21.15.70:0/970015286 shutdown_connections 2024-06-04T05:23:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.296+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/970015286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c2375e320 0x560c2375fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.296+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/970015286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c237588a0 0x560c23753270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.296+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/970015286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c23759260 0x560c2375f720 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.296+0000 7f237c5c1b80 1 -- 172.21.15.70:0/970015286 >> 172.21.15.70:0/970015286 conn(0x560c23684240 msgr2=0x560c2375f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.296+0000 7f237c5c1b80 1 -- 172.21.15.70:0/970015286 shutdown_connections 2024-06-04T05:23:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.296+0000 7f237c5c1b80 1 -- 172.21.15.70:0/970015286 wait complete. 2024-06-04T05:23:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.296+0000 7f237c5c1b80 1 -- start start 2024-06-04T05:23:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f237c5c1b80 1 Processor -- start 2024-06-04T05:23:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f237c5c1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c237588a0 0x560c2376a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f237c5c1b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c23759260 0x560c2376aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f237c5c1b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c2375e320 0x560c23765950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f237c5c1b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f236c1b1590 con 0x560c2375e320 2024-06-04T05:23:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f237c5c1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f236c1b0e00 con 0x560c237588a0 2024-06-04T05:23:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f237c5c1b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f236c1adf60 con 0x560c23759260 2024-06-04T05:23:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f237c5c1b80 10 client.0 _ll_register_callbacks cb 0x560c237543f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f236bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c2375e320 0x560c23765950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f236bfff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c2375e320 0x560c23765950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50564/0 (socket says 172.21.15.70:50564) 2024-06-04T05:23:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f236bfff700 1 -- 172.21.15.70:0/383337366 learned_addr learned my addr 172.21.15.70:0/383337366 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f236b7fe700 1 --2- 172.21.15.70:0/383337366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c237588a0 0x560c2376a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f236affd700 1 --2- 172.21.15.70:0/383337366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c23759260 0x560c2376aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f236bfff700 1 -- 172.21.15.70:0/383337366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c23759260 msgr2=0x560c2376aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f236bfff700 1 --2- 172.21.15.70:0/383337366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c23759260 0x560c2376aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f236bfff700 1 -- 172.21.15.70:0/383337366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c237588a0 msgr2=0x560c2376a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f236bfff700 1 --2- 172.21.15.70:0/383337366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c237588a0 0x560c2376a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.297+0000 7f236bfff700 1 -- 172.21.15.70:0/383337366 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c23765f90 con 0x560c2375e320 2024-06-04T05:23:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.298+0000 7f236bfff700 1 --2- 172.21.15.70:0/383337366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c2375e320 0x560c23765950 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f236400a040 tx=0x7f2364014750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3d9183bf42fcd4 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.298+0000 7f234affd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.298+0000 7f234affd700 1 -- 172.21.15.70:0/383337366 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2364022070 con 0x560c2375e320 2024-06-04T05:23:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.298+0000 7f234affd700 1 -- 172.21.15.70:0/383337366 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7f236401d670 con 0x560c2375e320 2024-06-04T05:23:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.298+0000 7f237c5c1b80 1 -- 172.21.15.70:0/383337366 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f236000f460 con 0x560c2375e320 2024-06-04T05:23:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.298+0000 7f234affd700 1 -- 172.21.15.70:0/383337366 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f236402e050 con 0x560c2375e320 2024-06-04T05:23:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.298+0000 7f237c5c1b80 10 client.22101 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.298+0000 7f237c5c1b80 1 -- 172.21.15.70:0/383337366 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x560c237dd7f0 con 0x560c2375e320 2024-06-04T05:23:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.298+0000 7f234affd700 1 -- 172.21.15.70:0/383337366 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f236000f460 con 0x560c2375e320 2024-06-04T05:23:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.298+0000 7f234affd700 1 -- 172.21.15.70:0/383337366 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7f236401c4a0 con 0x560c2375e320 2024-06-04T05:23:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.298+0000 7f237c5c1b80 10 client.22101 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.298+0000 7f237c5c1b80 10 client.22101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:23:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.298+0000 7f237c5c1b80 20 client.22101 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.298+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/383337366 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560c237ddb60 0x560c237dff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.298+0000 7f237c5c1b80 4 client.22101 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-04T05:23:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.298+0000 7f237c5c1b80 1 -- 172.21.15.70:0/383337366 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560c236b24a0 con 0x560c237ddb60 2024-06-04T05:23:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.299+0000 7f236b7fe700 1 --2- 172.21.15.70:0/383337366 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560c237ddb60 0x560c237dff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.299+0000 7f236b7fe700 1 --2- 172.21.15.70:0/383337366 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560c237ddb60 0x560c237dff50 crc :-1 s=READY pgs=546 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf2500cad01d80a6 server_cookie=1960720c303db8cc in_seq=0 out_seq=0 2024-06-04T05:23:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.299+0000 7f234affd700 10 client.22101 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.299+0000 7f234affd700 1 -- 172.21.15.70:0/383337366 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560c236b24a0 con 0x560c237ddb60 2024-06-04T05:23:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.299+0000 7f234affd700 10 client.22101 handle_command_reply: tid=0 2024-06-04T05:23:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.299+0000 7f237c5c1b80 1 client.22101 shutdown 2024-06-04T05:23:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.299+0000 7f237c5c1b80 1 -- 172.21.15.70:0/383337366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c2375e320 msgr2=0x560c23765950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.299+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/383337366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c2375e320 0x560c23765950 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f236400a040 tx=0x7f2364014750 comp rx=0 tx=0).stop 2024-06-04T05:23:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.299+0000 7f237c5c1b80 1 -- 172.21.15.70:0/383337366 shutdown_connections 2024-06-04T05:23:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.299+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/383337366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c2375e320 0x560c23765950 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.299+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/383337366 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x560c237ddb60 0x560c237dff50 crc :-1 s=READY pgs=546 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.299+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/383337366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c237588a0 0x560c2376a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.299+0000 7f234affd700 0 client.22101 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.299+0000 7f237c5c1b80 1 --2- 172.21.15.70:0/383337366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560c23759260 0x560c2376aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.299+0000 7f237c5c1b80 1 -- 172.21.15.70:0/383337366 >> 172.21.15.70:0/383337366 conn(0x560c23684240 msgr2=0x560c237620f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.299+0000 7f237c5c1b80 1 -- 172.21.15.70:0/383337366 shutdown_connections 2024-06-04T05:23:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.300+0000 7f237c5c1b80 1 -- 172.21.15.70:0/383337366 wait complete. 2024-06-04T05:23:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.300+0000 7f237c5c1b80 20 client.22101 trim_cache size 0 max 16384 2024-06-04T05:23:29.307 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:23:29.307 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-06-04T05:23:29.307 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-06-04T05:23:29.307 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:23:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.300+0000 7f2371d2c700 1 -- 172.21.15.70:0/2726069938 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f23540b2a70 msgr2=0x7f23540b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.300+0000 7f2371d2c700 1 --2- 172.21.15.70:0/2726069938 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f23540b2a70 0x7f23540b4f30 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2360000c00 tx=0x7f236001a040 comp rx=0 tx=0).stop 2024-06-04T05:23:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.300+0000 7f2371d2c700 1 -- 172.21.15.70:0/2726069938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f236c1037f0 msgr2=0x7f236c075840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.300+0000 7f2371d2c700 1 --2- 172.21.15.70:0/2726069938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f236c1037f0 0x7f236c075840 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f235c00d980 tx=0x7f235c00dc90 comp rx=0 tx=0).stop 2024-06-04T05:23:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.300+0000 7f2371d2c700 1 -- 172.21.15.70:0/2726069938 shutdown_connections 2024-06-04T05:23:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.300+0000 7f2371d2c700 1 --2- 172.21.15.70:0/2726069938 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f23540b2a70 0x7f23540b4f30 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.300+0000 7f2371d2c700 1 --2- 172.21.15.70:0/2726069938 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f236c1088f0 0x7f236c0762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.300+0000 7f2371d2c700 1 --2- 172.21.15.70:0/2726069938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f236c1037f0 0x7f236c075840 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.300+0000 7f2371d2c700 1 --2- 172.21.15.70:0/2726069938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f236c1041a0 0x7f236c075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.300+0000 7f2371d2c700 1 -- 172.21.15.70:0/2726069938 >> 172.21.15.70:0/2726069938 conn(0x7f236c07a6a0 msgr2=0x7f236c101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.300+0000 7f2371d2c700 1 -- 172.21.15.70:0/2726069938 shutdown_connections 2024-06-04T05:23:29.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:29.301+0000 7f2371d2c700 1 -- 172.21.15.70:0/2726069938 wait complete. 2024-06-04T05:23:29.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:29 smithi070 ceph-mon[32631]: from='client.22077 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T05:23:29.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:29 smithi070 ceph-mon[32631]: from='client.22083 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T05:23:29.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:29 smithi070 ceph-mon[32631]: pgmap v830: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 63 KiB/s wr, 33 op/s 2024-06-04T05:23:29.925 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-04T05:23:29.926 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:23:29.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:29 smithi112 ceph-mon[36453]: from='client.22077 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T05:23:29.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:29 smithi112 ceph-mon[36453]: from='client.22083 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T05:23:29.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:29 smithi112 ceph-mon[36453]: pgmap v830: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 63 KiB/s wr, 33 op/s 2024-06-04T05:23:29.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:29 smithi098 ceph-mon[35569]: from='client.22077 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T05:23:29.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:29 smithi098 ceph-mon[35569]: from='client.22083 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-04T05:23:29.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:29 smithi098 ceph-mon[35569]: pgmap v830: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 63 KiB/s wr, 33 op/s 2024-06-04T05:23:30.168 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:31.773 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:31 smithi070 ceph-mon[32631]: pgmap v831: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 63 KiB/s wr, 33 op/s 2024-06-04T05:23:31.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:31 smithi112 ceph-mon[36453]: pgmap v831: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 63 KiB/s wr, 33 op/s 2024-06-04T05:23:31.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:31 smithi098 ceph-mon[35569]: pgmap v831: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 63 KiB/s wr, 33 op/s 2024-06-04T05:23:32.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.045+0000 7f3dc3488700 1 -- 172.21.15.70:0/3460540283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dbc102440 msgr2=0x7f3dbc1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:32.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.045+0000 7f3dc3488700 1 --2- 172.21.15.70:0/3460540283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dbc102440 0x7f3dbc1028c0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009990 tx=0x7f3db8009ca0 comp rx=0 tx=0).stop 2024-06-04T05:23:32.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.045+0000 7f3dc3488700 1 -- 172.21.15.70:0/3460540283 shutdown_connections 2024-06-04T05:23:32.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.045+0000 7f3dc3488700 1 --2- 172.21.15.70:0/3460540283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dbc102e00 0x7f3dbc10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:32.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.045+0000 7f3dc3488700 1 --2- 172.21.15.70:0/3460540283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3dbc108520 0x7f3dbc108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:32.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.045+0000 7f3dc3488700 1 --2- 172.21.15.70:0/3460540283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dbc102440 0x7f3dbc1028c0 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:32.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.045+0000 7f3dc3488700 1 -- 172.21.15.70:0/3460540283 >> 172.21.15.70:0/3460540283 conn(0x7f3dbc0793f0 msgr2=0x7f3dbc079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:32.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.045+0000 7f3dc3488700 1 -- 172.21.15.70:0/3460540283 shutdown_connections 2024-06-04T05:23:32.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.045+0000 7f3dc3488700 1 -- 172.21.15.70:0/3460540283 wait complete. 2024-06-04T05:23:32.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.046+0000 7f3dc3488700 1 Processor -- start 2024-06-04T05:23:32.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.046+0000 7f3dc3488700 1 -- start start 2024-06-04T05:23:32.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.046+0000 7f3dc3488700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dbc102440 0x7f3dbc19e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:32.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.046+0000 7f3dc3488700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dbc102e00 0x7f3dbc19ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:32.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.046+0000 7f3dc3488700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3dbc108520 0x7f3dbc198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:32.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.046+0000 7f3dc3488700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc1124a0 con 0x7f3dbc102e00 2024-06-04T05:23:32.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.046+0000 7f3dc3488700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc112320 con 0x7f3dbc102440 2024-06-04T05:23:32.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.046+0000 7f3dc3488700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc112620 con 0x7f3dbc108520 2024-06-04T05:23:32.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.047+0000 7f3dc0a23700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dbc102e00 0x7f3dbc19ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:32.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.047+0000 7f3dc0a23700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dbc102e00 0x7f3dbc19ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50586/0 (socket says 172.21.15.70:50586) 2024-06-04T05:23:32.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.047+0000 7f3dc0a23700 1 -- 172.21.15.70:0/3139410193 learned_addr learned my addr 172.21.15.70:0/3139410193 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:32.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.047+0000 7f3dc1224700 1 --2- 172.21.15.70:0/3139410193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dbc102440 0x7f3dbc19e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:32.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.047+0000 7f3dc0a23700 1 -- 172.21.15.70:0/3139410193 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3dbc108520 msgr2=0x7f3dbc198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:32.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.047+0000 7f3dc0a23700 1 --2- 172.21.15.70:0/3139410193 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3dbc108520 0x7f3dbc198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:32.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.047+0000 7f3dc0a23700 1 -- 172.21.15.70:0/3139410193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dbc102440 msgr2=0x7f3dbc19e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:32.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.047+0000 7f3dc0a23700 1 --2- 172.21.15.70:0/3139410193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dbc102440 0x7f3dbc19e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:32.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.047+0000 7f3dc0a23700 1 -- 172.21.15.70:0/3139410193 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dbc198fe0 con 0x7f3dbc102e00 2024-06-04T05:23:32.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.047+0000 7f3dc1224700 1 --2- 172.21.15.70:0/3139410193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dbc102440 0x7f3dbc19e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:32.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.047+0000 7f3dc0a23700 1 --2- 172.21.15.70:0/3139410193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dbc102e00 0x7f3dbc19ec00 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f3db8000c00 tx=0x7f3db8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:32.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.047+0000 7f3dae7fc700 1 -- 172.21.15.70:0/3139410193 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8027070 con 0x7f3dbc102e00 2024-06-04T05:23:32.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.047+0000 7f3dae7fc700 1 -- 172.21.15.70:0/3139410193 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3db800fc70 con 0x7f3dbc102e00 2024-06-04T05:23:32.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.047+0000 7f3dae7fc700 1 -- 172.21.15.70:0/3139410193 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8022d40 con 0x7f3dbc102e00 2024-06-04T05:23:32.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.047+0000 7f3dc3488700 1 -- 172.21.15.70:0/3139410193 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dbc199270 con 0x7f3dbc102e00 2024-06-04T05:23:32.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.047+0000 7f3dc3488700 1 -- 172.21.15.70:0/3139410193 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dbc1a5590 con 0x7f3dbc102e00 2024-06-04T05:23:32.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.048+0000 7f3dc3488700 1 -- 172.21.15.70:0/3139410193 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dbc10a9b0 con 0x7f3dbc102e00 2024-06-04T05:23:32.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.051+0000 7f3dae7fc700 1 -- 172.21.15.70:0/3139410193 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3db802a070 con 0x7f3dbc102e00 2024-06-04T05:23:32.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.051+0000 7f3dae7fc700 1 --2- 172.21.15.70:0/3139410193 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3da80b29f0 0x7f3da80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:32.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.051+0000 7f3dc1224700 1 --2- 172.21.15.70:0/3139410193 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3da80b29f0 0x7f3da80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:32.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.051+0000 7f3dae7fc700 1 -- 172.21.15.70:0/3139410193 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7f3db80e3810 con 0x7f3dbc102e00 2024-06-04T05:23:32.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.052+0000 7f3dc1224700 1 --2- 172.21.15.70:0/3139410193 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3da80b29f0 0x7f3da80b4eb0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3db0005950 tx=0x7f3db000a5b0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:32.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.055+0000 7f3dae7fc700 1 -- 172.21.15.70:0/3139410193 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3db8031070 con 0x7f3dbc102e00 2024-06-04T05:23:32.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.302+0000 7f3dc3488700 1 -- 172.21.15.70:0/3139410193 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3dbc199db0 con 0x7f3dbc102e00 2024-06-04T05:23:32.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.303+0000 7f3dae7fc700 1 -- 172.21.15.70:0/3139410193 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+11944 (secure 0 0 0) 0x7f3db8013070 con 0x7f3dbc102e00 2024-06-04T05:23:32.305 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:23:32.306 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":72,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:23:17.104466+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:23:32.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.305+0000 7f3dc3488700 1 -- 172.21.15.70:0/3139410193 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3da80b29f0 msgr2=0x7f3da80b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:32.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.305+0000 7f3dc3488700 1 --2- 172.21.15.70:0/3139410193 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3da80b29f0 0x7f3da80b4eb0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3db0005950 tx=0x7f3db000a5b0 comp rx=0 tx=0).stop 2024-06-04T05:23:32.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.305+0000 7f3dc3488700 1 -- 172.21.15.70:0/3139410193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dbc102e00 msgr2=0x7f3dbc19ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:32.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.305+0000 7f3dc3488700 1 --2- 172.21.15.70:0/3139410193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dbc102e00 0x7f3dbc19ec00 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f3db8000c00 tx=0x7f3db8009f90 comp rx=0 tx=0).stop 2024-06-04T05:23:32.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.306+0000 7f3dc3488700 1 -- 172.21.15.70:0/3139410193 shutdown_connections 2024-06-04T05:23:32.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.306+0000 7f3dc3488700 1 --2- 172.21.15.70:0/3139410193 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f3da80b29f0 0x7f3da80b4eb0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:32.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.306+0000 7f3dc3488700 1 --2- 172.21.15.70:0/3139410193 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3dbc108520 0x7f3dbc198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:32.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.306+0000 7f3dc3488700 1 --2- 172.21.15.70:0/3139410193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dbc102440 0x7f3dbc19e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:32.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.306+0000 7f3dc3488700 1 --2- 172.21.15.70:0/3139410193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dbc102e00 0x7f3dbc19ec00 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:32.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.306+0000 7f3dc3488700 1 -- 172.21.15.70:0/3139410193 >> 172.21.15.70:0/3139410193 conn(0x7f3dbc0793f0 msgr2=0x7f3dbc100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:32.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.306+0000 7f3dc3488700 1 -- 172.21.15.70:0/3139410193 shutdown_connections 2024-06-04T05:23:32.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:32.306+0000 7f3dc3488700 1 -- 172.21.15.70:0/3139410193 wait complete. 2024-06-04T05:23:32.309 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 72 2024-06-04T05:23:32.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:32 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3139410193' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:23:32.934 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-04T05:23:32.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:32 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3139410193' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:23:32.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:32 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3139410193' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:23:33.178 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:33.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:33 smithi070 ceph-mon[32631]: pgmap v832: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 66 KiB/s wr, 40 op/s 2024-06-04T05:23:33.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:33 smithi112 ceph-mon[36453]: pgmap v832: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 66 KiB/s wr, 40 op/s 2024-06-04T05:23:33.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:33 smithi098 ceph-mon[35569]: pgmap v832: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 66 KiB/s wr, 40 op/s 2024-06-04T05:23:34.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.770+0000 7fdd294a3700 1 -- 172.21.15.70:0/3827456467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd241041a0 msgr2=0x7fdd241081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:34.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.770+0000 7fdd294a3700 1 --2- 172.21.15.70:0/3827456467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd241041a0 0x7fdd241081c0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0099c0 tx=0x7fdd0c009cd0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.771+0000 7fdd294a3700 1 -- 172.21.15.70:0/3827456467 shutdown_connections 2024-06-04T05:23:34.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.771+0000 7fdd294a3700 1 --2- 172.21.15.70:0/3827456467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd241088f0 0x7fdd24110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.771+0000 7fdd294a3700 1 --2- 172.21.15.70:0/3827456467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd241037f0 0x7fdd24103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.771+0000 7fdd294a3700 1 --2- 172.21.15.70:0/3827456467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd241041a0 0x7fdd241081c0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.771+0000 7fdd294a3700 1 -- 172.21.15.70:0/3827456467 >> 172.21.15.70:0/3827456467 conn(0x7fdd2407a6a0 msgr2=0x7fdd2407aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:34.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.771+0000 7fdd294a3700 1 -- 172.21.15.70:0/3827456467 shutdown_connections 2024-06-04T05:23:34.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.771+0000 7fdd294a3700 1 -- 172.21.15.70:0/3827456467 wait complete. 2024-06-04T05:23:34.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.771+0000 7fdd294a3700 1 Processor -- start 2024-06-04T05:23:34.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.771+0000 7fdd294a3700 1 -- start start 2024-06-04T05:23:34.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd294a3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd241037f0 0x7fdd2419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd294a3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd241041a0 0x7fdd2419fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd294a3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd241088f0 0x7fdd24199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd294a3700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241135e0 con 0x7fdd241037f0 2024-06-04T05:23:34.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd294a3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdd24113460 con 0x7fdd241041a0 2024-06-04T05:23:34.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd294a3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdd24113760 con 0x7fdd241088f0 2024-06-04T05:23:34.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd22ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd241037f0 0x7fdd2419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:34.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd22ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd241037f0 0x7fdd2419f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50604/0 (socket says 172.21.15.70:50604) 2024-06-04T05:23:34.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd22ffd700 1 -- 172.21.15.70:0/1959744783 learned_addr learned my addr 172.21.15.70:0/1959744783 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:34.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd227fc700 1 --2- 172.21.15.70:0/1959744783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd241041a0 0x7fdd2419fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:34.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd237fe700 1 --2- 172.21.15.70:0/1959744783 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd241088f0 0x7fdd24199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:34.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd22ffd700 1 -- 172.21.15.70:0/1959744783 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd241088f0 msgr2=0x7fdd24199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:34.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd22ffd700 1 --2- 172.21.15.70:0/1959744783 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd241088f0 0x7fdd24199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd22ffd700 1 -- 172.21.15.70:0/1959744783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd241041a0 msgr2=0x7fdd2419fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:34.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd22ffd700 1 --2- 172.21.15.70:0/1959744783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd241041a0 0x7fdd2419fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd22ffd700 1 -- 172.21.15.70:0/1959744783 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd2419a0a0 con 0x7fdd241037f0 2024-06-04T05:23:34.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd22ffd700 1 --2- 172.21.15.70:0/1959744783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd241037f0 0x7fdd2419f780 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00b530 tx=0x7fdd1c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:34.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd1bfff700 1 -- 172.21.15.70:0/1959744783 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c015020 con 0x7fdd241037f0 2024-06-04T05:23:34.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd1bfff700 1 -- 172.21.15.70:0/1959744783 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd1c011cc0 con 0x7fdd241037f0 2024-06-04T05:23:34.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd294a3700 1 -- 172.21.15.70:0/1959744783 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd2419a390 con 0x7fdd241037f0 2024-06-04T05:23:34.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd1bfff700 1 -- 172.21.15.70:0/1959744783 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c010dc0 con 0x7fdd241037f0 2024-06-04T05:23:34.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.772+0000 7fdd294a3700 1 -- 172.21.15.70:0/1959744783 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd241a6650 con 0x7fdd241037f0 2024-06-04T05:23:34.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.775+0000 7fdd1bfff700 1 -- 172.21.15.70:0/1959744783 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fdd1c018020 con 0x7fdd241037f0 2024-06-04T05:23:34.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.776+0000 7fdd1bfff700 1 --2- 172.21.15.70:0/1959744783 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdd100b29f0 0x7fdd100b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.776+0000 7fdd227fc700 1 --2- 172.21.15.70:0/1959744783 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdd100b29f0 0x7fdd100b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:34.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.776+0000 7fdd1bfff700 1 -- 172.21.15.70:0/1959744783 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fdd1c0e1e20 con 0x7fdd241037f0 2024-06-04T05:23:34.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.776+0000 7fdd33d38b80 1 Processor -- start 2024-06-04T05:23:34.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.776+0000 7fdd33d38b80 1 -- start start 2024-06-04T05:23:34.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.776+0000 7fdd227fc700 1 --2- 172.21.15.70:0/1959744783 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdd100b29f0 0x7fdd100b4eb0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c000c00 tx=0x7fdd0c005fb0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:34.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.776+0000 7fdd33d38b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65c003320 0x55c65c003700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd33d38b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65bffd8a0 0x55c65bffdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd33d38b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffe260 0x55c65c00a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd33d38b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c65bf5bd70 con 0x55c65bffe260 2024-06-04T05:23:34.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd33d38b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c65bf4b910 con 0x55c65c003320 2024-06-04T05:23:34.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd33d38b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c65bf4f950 con 0x55c65bffd8a0 2024-06-04T05:23:34.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd237fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffe260 0x55c65c00a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:34.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd237fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffe260 0x55c65c00a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50612/0 (socket says 172.21.15.70:50612) 2024-06-04T05:23:34.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd237fe700 1 -- 172.21.15.70:0/3613397280 learned_addr learned my addr 172.21.15.70:0/3613397280 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:34.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd22ffd700 1 --2- 172.21.15.70:0/3613397280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65c003320 0x55c65c003700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:34.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd227fc700 1 --2- 172.21.15.70:0/3613397280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65bffd8a0 0x55c65bffdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:34.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd237fe700 1 -- 172.21.15.70:0/3613397280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65bffd8a0 msgr2=0x55c65bffdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:34.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd237fe700 1 --2- 172.21.15.70:0/3613397280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65bffd8a0 0x55c65bffdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd237fe700 1 -- 172.21.15.70:0/3613397280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65c003320 msgr2=0x55c65c003700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:34.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd237fe700 1 --2- 172.21.15.70:0/3613397280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65c003320 0x55c65c003700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd237fe700 1 -- 172.21.15.70:0/3613397280 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c65bef4fb0 con 0x55c65bffe260 2024-06-04T05:23:34.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd227fc700 1 --2- 172.21.15.70:0/3613397280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65bffd8a0 0x55c65bffdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:23:34.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd237fe700 1 --2- 172.21.15.70:0/3613397280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffe260 0x55c65c00a080 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009dc0 tx=0x7fdd140077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5aa261df4b629a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:34.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd197fa700 1 -- 172.21.15.70:0/3613397280 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd14014610 con 0x55c65bffe260 2024-06-04T05:23:34.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd197fa700 1 -- 172.21.15.70:0/3613397280 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd14019070 con 0x55c65bffe260 2024-06-04T05:23:34.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.777+0000 7fdd197fa700 1 -- 172.21.15.70:0/3613397280 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd140137f0 con 0x55c65bffe260 2024-06-04T05:23:34.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.778+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3613397280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffe260 msgr2=0x55c65c00a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:34.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.778+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/3613397280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffe260 0x55c65c00a080 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009dc0 tx=0x7fdd140077e0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.778+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3613397280 shutdown_connections 2024-06-04T05:23:34.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.778+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/3613397280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffe260 0x55c65c00a080 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.778+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/3613397280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65c003320 0x55c65c003700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.778+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/3613397280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65bffd8a0 0x55c65bffdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.778+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3613397280 >> 172.21.15.70:0/3613397280 conn(0x55c65bf29240 msgr2=0x55c65bff83e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:34.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.778+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3613397280 shutdown_connections 2024-06-04T05:23:34.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.778+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3613397280 wait complete. 2024-06-04T05:23:34.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd33d38b80 1 -- start start 2024-06-04T05:23:34.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd33d38b80 1 Processor -- start 2024-06-04T05:23:34.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd33d38b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65bffd8a0 0x55c65c009cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd33d38b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffe260 0x55c65c006fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd33d38b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65c003320 0x55c65c0074f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd33d38b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c65be51fa0 con 0x55c65bffe260 2024-06-04T05:23:34.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd33d38b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c65beb3da0 con 0x55c65c003320 2024-06-04T05:23:34.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd33d38b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c65c00a6e0 con 0x55c65bffd8a0 2024-06-04T05:23:34.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd33d38b80 10 client.0 _ll_register_callbacks cb 0x55c65bee6950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:34.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd22ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65bffd8a0 0x55c65c009cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:34.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd227fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffe260 0x55c65c006fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:34.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd22ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65bffd8a0 0x55c65c009cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.70:48012/0 (socket says 172.21.15.70:48012) 2024-06-04T05:23:34.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd22ffd700 1 -- 172.21.15.70:0/4140808283 learned_addr learned my addr 172.21.15.70:0/4140808283 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:34.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd237fe700 1 --2- 172.21.15.70:0/4140808283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65c003320 0x55c65c0074f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:34.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd227fc700 1 -- 172.21.15.70:0/4140808283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65bffd8a0 msgr2=0x55c65c009cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:34.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd227fc700 1 --2- 172.21.15.70:0/4140808283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65bffd8a0 0x55c65c009cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd227fc700 1 -- 172.21.15.70:0/4140808283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65c003320 msgr2=0x55c65c0074f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:34.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd227fc700 1 --2- 172.21.15.70:0/4140808283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65c003320 0x55c65c0074f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd227fc700 1 -- 172.21.15.70:0/4140808283 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c65be9b8d0 con 0x55c65bffe260 2024-06-04T05:23:34.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.779+0000 7fdd237fe700 1 --2- 172.21.15.70:0/4140808283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65c003320 0x55c65c0074f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:23:34.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.780+0000 7fdd227fc700 1 --2- 172.21.15.70:0/4140808283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffe260 0x55c65c006fb0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0058e0 tx=0x7fdd0c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ddb9c2dc26d580 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:34.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.780+0000 7fdd0a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:34.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.780+0000 7fdd0a7fc700 1 -- 172.21.15.70:0/4140808283 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c034060 con 0x55c65bffe260 2024-06-04T05:23:34.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.780+0000 7fdd0a7fc700 1 -- 172.21.15.70:0/4140808283 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fdd0c003680 con 0x55c65bffe260 2024-06-04T05:23:34.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.780+0000 7fdd33d38b80 1 -- 172.21.15.70:0/4140808283 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd14019070 con 0x55c65bffe260 2024-06-04T05:23:34.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.780+0000 7fdd0a7fc700 1 -- 172.21.15.70:0/4140808283 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fdd0c003880 con 0x55c65bffe260 2024-06-04T05:23:34.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.780+0000 7fdd33d38b80 10 client.22119 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:34.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.780+0000 7fdd33d38b80 1 -- 172.21.15.70:0/4140808283 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x55c65bf05760 con 0x55c65bffe260 2024-06-04T05:23:34.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.780+0000 7fdd0a7fc700 1 -- 172.21.15.70:0/4140808283 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd14019070 con 0x55c65bffe260 2024-06-04T05:23:34.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.780+0000 7fdd0a7fc700 1 -- 172.21.15.70:0/4140808283 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7fdd0c00fb40 con 0x55c65bffe260 2024-06-04T05:23:34.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.780+0000 7fdd33d38b80 10 client.22119 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:34.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.780+0000 7fdd33d38b80 10 client.22119 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:23:34.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.780+0000 7fdd33d38b80 20 client.22119 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:34.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.780+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/4140808283 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55c65c0171c0 0x55c65c0195e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.780+0000 7fdd33d38b80 4 client.22119 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:23:34.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.780+0000 7fdd33d38b80 1 -- 172.21.15.70:0/4140808283 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c65bf5dcc0 con 0x55c65c0171c0 2024-06-04T05:23:34.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.781+0000 7fdd22ffd700 1 --2- 172.21.15.70:0/4140808283 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55c65c0171c0 0x55c65c0195e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:34.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.781+0000 7fdd0a7fc700 10 client.22119 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:23:34.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.781+0000 7fdd22ffd700 1 --2- 172.21.15.70:0/4140808283 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55c65c0171c0 0x55c65c0195e0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=49904b339a326971 server_cookie=b4e968dfbc25b783 in_seq=0 out_seq=0 2024-06-04T05:23:34.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.782+0000 7fdd0a7fc700 1 -- 172.21.15.70:0/4140808283 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c65bf5dcc0 con 0x55c65c0171c0 2024-06-04T05:23:34.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.782+0000 7fdd0a7fc700 10 client.22119 handle_command_reply: tid=0 2024-06-04T05:23:34.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.782+0000 7fdd33d38b80 1 client.22119 shutdown 2024-06-04T05:23:34.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.782+0000 7fdd33d38b80 1 -- 172.21.15.70:0/4140808283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffe260 msgr2=0x55c65c006fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:34.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.782+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/4140808283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffe260 0x55c65c006fb0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0058e0 tx=0x7fdd0c009310 comp rx=0 tx=0).stop 2024-06-04T05:23:34.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.782+0000 7fdd33d38b80 1 -- 172.21.15.70:0/4140808283 shutdown_connections 2024-06-04T05:23:34.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.782+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/4140808283 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55c65c0171c0 0x55c65c0195e0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.782+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/4140808283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65c003320 0x55c65c0074f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.782+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/4140808283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65bffd8a0 0x55c65c009cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.782+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/4140808283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffe260 0x55c65c006fb0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.782+0000 7fdd33d38b80 1 -- 172.21.15.70:0/4140808283 >> 172.21.15.70:0/4140808283 conn(0x55c65bf29240 msgr2=0x55c65bff7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:34.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.782+0000 7fdd0a7fc700 0 client.22119 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:23:34.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.782+0000 7fdd33d38b80 1 -- 172.21.15.70:0/4140808283 shutdown_connections 2024-06-04T05:23:34.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.782+0000 7fdd33d38b80 1 -- 172.21.15.70:0/4140808283 wait complete. 2024-06-04T05:23:34.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.782+0000 7fdd33d38b80 20 client.22119 trim_cache size 0 max 16384 2024-06-04T05:23:34.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.797+0000 7fdd33d38b80 1 Processor -- start 2024-06-04T05:23:34.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.797+0000 7fdd33d38b80 1 -- start start 2024-06-04T05:23:34.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.797+0000 7fdd33d38b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffd8a0 0x55c65c0195b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.797+0000 7fdd33d38b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65bffe260 0x55c65c0013f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.797+0000 7fdd33d38b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65c003320 0x55c65bff8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.797+0000 7fdd33d38b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241adea0 con 0x55c65bffd8a0 2024-06-04T05:23:34.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.797+0000 7fdd33d38b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241afb90 con 0x55c65bffe260 2024-06-04T05:23:34.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.797+0000 7fdd33d38b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241ae600 con 0x55c65c003320 2024-06-04T05:23:34.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.797+0000 7fdd22ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffd8a0 0x55c65c0195b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:34.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.797+0000 7fdd22ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffd8a0 0x55c65c0195b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50628/0 (socket says 172.21.15.70:50628) 2024-06-04T05:23:34.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.797+0000 7fdd22ffd700 1 -- 172.21.15.70:0/3344193320 learned_addr learned my addr 172.21.15.70:0/3344193320 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:34.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.798+0000 7fdd237fe700 1 --2- 172.21.15.70:0/3344193320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65c003320 0x55c65bff8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:34.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.798+0000 7fdd227fc700 1 --2- 172.21.15.70:0/3344193320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65bffe260 0x55c65c0013f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:34.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.798+0000 7fdd22ffd700 1 -- 172.21.15.70:0/3344193320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65c003320 msgr2=0x55c65bff8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:34.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.798+0000 7fdd22ffd700 1 --2- 172.21.15.70:0/3344193320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65c003320 0x55c65bff8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.798+0000 7fdd22ffd700 1 -- 172.21.15.70:0/3344193320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65bffe260 msgr2=0x55c65c0013f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:34.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.798+0000 7fdd22ffd700 1 --2- 172.21.15.70:0/3344193320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65bffe260 0x55c65c0013f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.798+0000 7fdd22ffd700 1 -- 172.21.15.70:0/3344193320 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c65c007870 con 0x55c65bffd8a0 2024-06-04T05:23:34.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.798+0000 7fdd22ffd700 1 --2- 172.21.15.70:0/3344193320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffd8a0 0x55c65c0195b0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c0a8ee0 tx=0x7fdd1c01c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622d1bf9765e7cd server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:34.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.798+0000 7fdd18ff9700 1 -- 172.21.15.70:0/3344193320 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c0a91e0 con 0x55c65bffd8a0 2024-06-04T05:23:34.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.798+0000 7fdd18ff9700 1 -- 172.21.15.70:0/3344193320 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd1c06d070 con 0x55c65bffd8a0 2024-06-04T05:23:34.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.798+0000 7fdd18ff9700 1 -- 172.21.15.70:0/3344193320 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c0ad380 con 0x55c65bffd8a0 2024-06-04T05:23:34.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.799+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3344193320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffd8a0 msgr2=0x55c65c0195b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:34.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.799+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/3344193320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffd8a0 0x55c65c0195b0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c0a8ee0 tx=0x7fdd1c01c040 comp rx=0 tx=0).stop 2024-06-04T05:23:34.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.799+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3344193320 shutdown_connections 2024-06-04T05:23:34.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.799+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/3344193320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65c003320 0x55c65bff8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.799+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/3344193320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffd8a0 0x55c65c0195b0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.799+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/3344193320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65bffe260 0x55c65c0013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.799+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3344193320 >> 172.21.15.70:0/3344193320 conn(0x55c65bf29240 msgr2=0x55c65c000ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:34.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.799+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3344193320 shutdown_connections 2024-06-04T05:23:34.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.799+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3344193320 wait complete. 2024-06-04T05:23:34.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.799+0000 7fdd33d38b80 1 -- start start 2024-06-04T05:23:34.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.799+0000 7fdd33d38b80 1 Processor -- start 2024-06-04T05:23:34.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.799+0000 7fdd33d38b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffd8a0 0x55c65c07d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd33d38b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65bffe260 0x55c65c07d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd33d38b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65c003320 0x55c65c00f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd33d38b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241af490 con 0x55c65bffd8a0 2024-06-04T05:23:34.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd33d38b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241aed00 con 0x55c65c003320 2024-06-04T05:23:34.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd33d38b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241abe60 con 0x55c65bffe260 2024-06-04T05:23:34.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd33d38b80 10 client.0 _ll_register_callbacks cb 0x55c65bee6950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:34.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd22ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffd8a0 0x55c65c07d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:34.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd22ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffd8a0 0x55c65c07d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50638/0 (socket says 172.21.15.70:50638) 2024-06-04T05:23:34.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd22ffd700 1 -- 172.21.15.70:0/3713899705 learned_addr learned my addr 172.21.15.70:0/3713899705 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:34.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd22ffd700 1 -- 172.21.15.70:0/3713899705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65bffe260 msgr2=0x55c65c07d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:34.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd22ffd700 1 --2- 172.21.15.70:0/3713899705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65bffe260 0x55c65c07d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd22ffd700 1 -- 172.21.15.70:0/3713899705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65c003320 msgr2=0x55c65c00f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:34.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd22ffd700 1 --2- 172.21.15.70:0/3713899705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65c003320 0x55c65c00f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd22ffd700 1 -- 172.21.15.70:0/3713899705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c65c010170 con 0x55c65bffd8a0 2024-06-04T05:23:34.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd22ffd700 1 --2- 172.21.15.70:0/3713899705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffd8a0 0x55c65c07d100 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c0a9170 tx=0x7fdd1c0a9290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda94ff9b6e18020 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:34.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd0a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:34.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd0a7fc700 1 -- 172.21.15.70:0/3713899705 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c0c7020 con 0x55c65bffd8a0 2024-06-04T05:23:34.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd0a7fc700 1 -- 172.21.15.70:0/3713899705 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fdd1c0ad170 con 0x55c65bffd8a0 2024-06-04T05:23:34.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3713899705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd1c06d070 con 0x55c65bffd8a0 2024-06-04T05:23:34.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd0a7fc700 1 -- 172.21.15.70:0/3713899705 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fdd1c0a9440 con 0x55c65bffd8a0 2024-06-04T05:23:34.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd33d38b80 10 client.22125 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:34.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.800+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3713899705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x55c65c01f150 con 0x55c65bffd8a0 2024-06-04T05:23:34.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.801+0000 7fdd0a7fc700 1 -- 172.21.15.70:0/3713899705 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd1c06d070 con 0x55c65bffd8a0 2024-06-04T05:23:34.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.801+0000 7fdd0a7fc700 1 -- 172.21.15.70:0/3713899705 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7fdd1c089020 con 0x55c65bffd8a0 2024-06-04T05:23:34.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.801+0000 7fdd33d38b80 10 client.22125 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:34.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.801+0000 7fdd33d38b80 10 client.22125 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-06-04T05:23:34.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.801+0000 7fdd33d38b80 20 client.22125 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:34.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.801+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/3713899705 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55c65c0865a0 0x55c65c088a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:34.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.801+0000 7fdd33d38b80 4 client.22125 mds_command: new command op to 24410 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:23:34.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.801+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3713899705 --> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c65bf574a0 con 0x55c65c0865a0 2024-06-04T05:23:34.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.801+0000 7fdd227fc700 1 --2- 172.21.15.70:0/3713899705 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55c65c0865a0 0x55c65c088a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:34.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.801+0000 7fdd0a7fc700 10 client.22125 ms_handle_connect on v2:172.21.15.98:6836/3160044154 2024-06-04T05:23:34.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.801+0000 7fdd227fc700 1 --2- 172.21.15.70:0/3713899705 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55c65c0865a0 0x55c65c088a20 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=85eb5334219bfdf1 server_cookie=3c888437db1fa194 in_seq=0 out_seq=0 2024-06-04T05:23:34.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.801+0000 7fdd0a7fc700 1 -- 172.21.15.70:0/3713899705 <== mds.4 v2:172.21.15.98:6836/3160044154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c65bf574a0 con 0x55c65c0865a0 2024-06-04T05:23:34.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.801+0000 7fdd0a7fc700 10 client.22125 handle_command_reply: tid=0 2024-06-04T05:23:34.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.801+0000 7fdd33d38b80 1 client.22125 shutdown 2024-06-04T05:23:34.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3713899705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffd8a0 msgr2=0x55c65c07d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:34.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/3713899705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffd8a0 0x55c65c07d100 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c0a9170 tx=0x7fdd1c0a9290 comp rx=0 tx=0).stop 2024-06-04T05:23:34.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3713899705 shutdown_connections 2024-06-04T05:23:34.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/3713899705 >> [v2:172.21.15.98:6836/3160044154,v1:172.21.15.98:6837/3160044154] conn(0x55c65c0865a0 0x55c65c088a20 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/3713899705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c65c003320 0x55c65c00f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/3713899705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65bffd8a0 0x55c65c07d100 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd33d38b80 1 --2- 172.21.15.70:0/3713899705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c65bffe260 0x55c65c07d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3713899705 >> 172.21.15.70:0/3713899705 conn(0x55c65bf29240 msgr2=0x55c65bffd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:34.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd0a7fc700 0 client.22125 ms_handle_reset on v2:172.21.15.98:6836/3160044154 2024-06-04T05:23:34.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3713899705 shutdown_connections 2024-06-04T05:23:34.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd33d38b80 1 -- 172.21.15.70:0/3713899705 wait complete. 2024-06-04T05:23:34.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd33d38b80 20 client.22125 trim_cache size 0 max 16384 2024-06-04T05:23:34.810 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:23:34.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd294a3700 1 -- 172.21.15.70:0/1959744783 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdd100b29f0 msgr2=0x7fdd100b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:34.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd294a3700 1 --2- 172.21.15.70:0/1959744783 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdd100b29f0 0x7fdd100b4eb0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c000c00 tx=0x7fdd0c005fb0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd294a3700 1 -- 172.21.15.70:0/1959744783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd241037f0 msgr2=0x7fdd2419f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:34.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd294a3700 1 --2- 172.21.15.70:0/1959744783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd241037f0 0x7fdd2419f780 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00b530 tx=0x7fdd1c00b840 comp rx=0 tx=0).stop 2024-06-04T05:23:34.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd294a3700 1 -- 172.21.15.70:0/1959744783 shutdown_connections 2024-06-04T05:23:34.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd294a3700 1 --2- 172.21.15.70:0/1959744783 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fdd100b29f0 0x7fdd100b4eb0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd294a3700 1 --2- 172.21.15.70:0/1959744783 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd241088f0 0x7fdd24199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd294a3700 1 --2- 172.21.15.70:0/1959744783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd241037f0 0x7fdd2419f780 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd294a3700 1 --2- 172.21.15.70:0/1959744783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd241041a0 0x7fdd2419fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:34.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.802+0000 7fdd294a3700 1 -- 172.21.15.70:0/1959744783 >> 172.21.15.70:0/1959744783 conn(0x7fdd2407a6a0 msgr2=0x7fdd241018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:34.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.803+0000 7fdd294a3700 1 -- 172.21.15.70:0/1959744783 shutdown_connections 2024-06-04T05:23:34.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:34.803+0000 7fdd294a3700 1 -- 172.21.15.70:0/1959744783 wait complete. 2024-06-04T05:23:35.507 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-04T05:23:35.750 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:35.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:35 smithi070 ceph-mon[32631]: pgmap v833: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 26 KiB/s wr, 27 op/s 2024-06-04T05:23:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:35 smithi112 ceph-mon[36453]: pgmap v833: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 26 KiB/s wr, 27 op/s 2024-06-04T05:23:35.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:35 smithi098 ceph-mon[35569]: pgmap v833: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 26 KiB/s wr, 27 op/s 2024-06-04T05:23:37.572 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:37 smithi070 ceph-mon[32631]: pgmap v834: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-06-04T05:23:37.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.674+0000 7fcb7de62700 1 -- 172.21.15.70:0/2284914277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb78108520 msgr2=0x7fcb78108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:37.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.674+0000 7fcb7de62700 1 --2- 172.21.15.70:0/2284914277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb78108520 0x7fcb78108900 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fcb68009990 tx=0x7fcb68009ca0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.674+0000 7fcb7de62700 1 -- 172.21.15.70:0/2284914277 shutdown_connections 2024-06-04T05:23:37.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.674+0000 7fcb7de62700 1 --2- 172.21.15.70:0/2284914277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb78102e00 0x7fcb7810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.674+0000 7fcb7de62700 1 --2- 172.21.15.70:0/2284914277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb78108520 0x7fcb78108900 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.674+0000 7fcb7de62700 1 --2- 172.21.15.70:0/2284914277 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb78102440 0x7fcb781028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.674+0000 7fcb7de62700 1 -- 172.21.15.70:0/2284914277 >> 172.21.15.70:0/2284914277 conn(0x7fcb780793f0 msgr2=0x7fcb78079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:37.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.674+0000 7fcb7de62700 1 -- 172.21.15.70:0/2284914277 shutdown_connections 2024-06-04T05:23:37.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.674+0000 7fcb7de62700 1 -- 172.21.15.70:0/2284914277 wait complete. 2024-06-04T05:23:37.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.674+0000 7fcb7de62700 1 Processor -- start 2024-06-04T05:23:37.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.675+0000 7fcb7de62700 1 -- start start 2024-06-04T05:23:37.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.675+0000 7fcb7de62700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb78102440 0x7fcb7819e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.675+0000 7fcb7de62700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb78102e00 0x7fcb7819ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.675+0000 7fcb7de62700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb78108520 0x7fcb78198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.675+0000 7fcb7de62700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcb78112580 con 0x7fcb78102e00 2024-06-04T05:23:37.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.675+0000 7fcb7de62700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb78112400 con 0x7fcb78102440 2024-06-04T05:23:37.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.675+0000 7fcb7de62700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcb78112700 con 0x7fcb78108520 2024-06-04T05:23:37.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.675+0000 7fcb76ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb78102e00 0x7fcb7819ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:37.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.675+0000 7fcb76ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb78102e00 0x7fcb7819ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42060/0 (socket says 172.21.15.70:42060) 2024-06-04T05:23:37.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.675+0000 7fcb76ffd700 1 -- 172.21.15.70:0/368033910 learned_addr learned my addr 172.21.15.70:0/368033910 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:37.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.675+0000 7fcb777fe700 1 --2- 172.21.15.70:0/368033910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb78102440 0x7fcb7819e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:37.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.675+0000 7fcb77fff700 1 --2- 172.21.15.70:0/368033910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb78108520 0x7fcb78198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:37.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.675+0000 7fcb76ffd700 1 -- 172.21.15.70:0/368033910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb78108520 msgr2=0x7fcb78198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:37.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.675+0000 7fcb76ffd700 1 --2- 172.21.15.70:0/368033910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb78108520 0x7fcb78198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.675+0000 7fcb76ffd700 1 -- 172.21.15.70:0/368033910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb78102440 msgr2=0x7fcb7819e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:37.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.675+0000 7fcb76ffd700 1 --2- 172.21.15.70:0/368033910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb78102440 0x7fcb7819e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.675+0000 7fcb76ffd700 1 -- 172.21.15.70:0/368033910 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb781990e0 con 0x7fcb78102e00 2024-06-04T05:23:37.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.676+0000 7fcb76ffd700 1 --2- 172.21.15.70:0/368033910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb78102e00 0x7fcb7819ed00 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7fcb6000b530 tx=0x7fcb6000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:37.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.676+0000 7fcb74ff9700 1 -- 172.21.15.70:0/368033910 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb60015020 con 0x7fcb78102e00 2024-06-04T05:23:37.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.676+0000 7fcb74ff9700 1 -- 172.21.15.70:0/368033910 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb60011cc0 con 0x7fcb78102e00 2024-06-04T05:23:37.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.676+0000 7fcb7de62700 1 -- 172.21.15.70:0/368033910 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb781993d0 con 0x7fcb78102e00 2024-06-04T05:23:37.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.676+0000 7fcb74ff9700 1 -- 172.21.15.70:0/368033910 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb60010dc0 con 0x7fcb78102e00 2024-06-04T05:23:37.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.676+0000 7fcb7de62700 1 -- 172.21.15.70:0/368033910 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb781a5690 con 0x7fcb78102e00 2024-06-04T05:23:37.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.678+0000 7fcb74ff9700 1 -- 172.21.15.70:0/368033910 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fcb60018020 con 0x7fcb78102e00 2024-06-04T05:23:37.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.679+0000 7fcb74ff9700 1 --2- 172.21.15.70:0/368033910 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcb640b29f0 0x7fcb640b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.679+0000 7fcb777fe700 1 --2- 172.21.15.70:0/368033910 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcb640b29f0 0x7fcb640b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:37.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.679+0000 7fcb74ff9700 1 -- 172.21.15.70:0/368033910 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fcb600e1db0 con 0x7fcb78102e00 2024-06-04T05:23:37.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.679+0000 7fcb777fe700 1 --2- 172.21.15.70:0/368033910 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcb640b29f0 0x7fcb640b4eb0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fcb6800b5c0 tx=0x7fcb6801a040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:37.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb886f7b80 1 Processor -- start 2024-06-04T05:23:37.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb886f7b80 1 -- start start 2024-06-04T05:23:37.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb886f7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9cb2e0 0x5566ad9c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb886f7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9ca240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb886f7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 0x5566ad9da310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb886f7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5566ad927d70 con 0x5566ad9c9de0 2024-06-04T05:23:37.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb886f7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5566ad917910 con 0x5566ad9d7f20 2024-06-04T05:23:37.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb886f7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5566ad91b950 con 0x5566ad9cb2e0 2024-06-04T05:23:37.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb76ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:37.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb76ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9ca240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42072/0 (socket says 172.21.15.70:42072) 2024-06-04T05:23:37.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb76ffd700 1 -- 172.21.15.70:0/3966232603 learned_addr learned my addr 172.21.15.70:0/3966232603 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb777fe700 1 --2- 172.21.15.70:0/3966232603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9cb2e0 0x5566ad9c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb77fff700 1 --2- 172.21.15.70:0/3966232603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 0x5566ad9da310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb76ffd700 1 -- 172.21.15.70:0/3966232603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9cb2e0 msgr2=0x5566ad9c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb76ffd700 1 --2- 172.21.15.70:0/3966232603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9cb2e0 0x5566ad9c98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb76ffd700 1 -- 172.21.15.70:0/3966232603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 msgr2=0x5566ad9da310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb76ffd700 1 --2- 172.21.15.70:0/3966232603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 0x5566ad9da310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb76ffd700 1 -- 172.21.15.70:0/3966232603 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566ad8c0fb0 con 0x5566ad9c9de0 2024-06-04T05:23:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb777fe700 1 --2- 172.21.15.70:0/3966232603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9cb2e0 0x5566ad9c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.680+0000 7fcb76ffd700 1 --2- 172.21.15.70:0/3966232603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9ca240 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fcb600a51f0 tx=0x7fcb600a8bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1aeea40be716b2e server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.681+0000 7fcb6dffb700 1 -- 172.21.15.70:0/3966232603 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb600c3070 con 0x5566ad9c9de0 2024-06-04T05:23:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.681+0000 7fcb6dffb700 1 -- 172.21.15.70:0/3966232603 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb600a9170 con 0x5566ad9c9de0 2024-06-04T05:23:37.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.681+0000 7fcb6dffb700 1 -- 172.21.15.70:0/3966232603 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb600a9630 con 0x5566ad9c9de0 2024-06-04T05:23:37.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.681+0000 7fcb886f7b80 1 -- 172.21.15.70:0/3966232603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 msgr2=0x5566ad9ca240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:37.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.681+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/3966232603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9ca240 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fcb600a51f0 tx=0x7fcb600a8bc0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.681+0000 7fcb886f7b80 1 -- 172.21.15.70:0/3966232603 shutdown_connections 2024-06-04T05:23:37.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.681+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/3966232603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 0x5566ad9da310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.681+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/3966232603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9cb2e0 0x5566ad9c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.681+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/3966232603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9ca240 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.681+0000 7fcb886f7b80 1 -- 172.21.15.70:0/3966232603 >> 172.21.15.70:0/3966232603 conn(0x5566ad8f5240 msgr2=0x5566ad9c43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:37.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.681+0000 7fcb886f7b80 1 -- 172.21.15.70:0/3966232603 shutdown_connections 2024-06-04T05:23:37.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.681+0000 7fcb886f7b80 1 -- 172.21.15.70:0/3966232603 wait complete. 2024-06-04T05:23:37.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.682+0000 7fcb886f7b80 1 -- start start 2024-06-04T05:23:37.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.682+0000 7fcb886f7b80 1 Processor -- start 2024-06-04T05:23:37.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.682+0000 7fcb886f7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9d7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.682+0000 7fcb886f7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9cb2e0 0x5566ad9d4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.682+0000 7fcb886f7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 0x5566ad9d4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.682+0000 7fcb886f7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5566ad81dfa0 con 0x5566ad9c9de0 2024-06-04T05:23:37.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.682+0000 7fcb886f7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5566ad87fda0 con 0x5566ad9d7f20 2024-06-04T05:23:37.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.682+0000 7fcb886f7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5566ad9daaf0 con 0x5566ad9cb2e0 2024-06-04T05:23:37.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.682+0000 7fcb886f7b80 10 client.0 _ll_register_callbacks cb 0x5566ad8b2950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:37.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.682+0000 7fcb777fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9d7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:37.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.682+0000 7fcb777fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9d7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42086/0 (socket says 172.21.15.70:42086) 2024-06-04T05:23:37.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.682+0000 7fcb777fe700 1 -- 172.21.15.70:0/2373004333 learned_addr learned my addr 172.21.15.70:0/2373004333 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:37.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.682+0000 7fcb76ffd700 1 --2- 172.21.15.70:0/2373004333 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9cb2e0 0x5566ad9d4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:37.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.683+0000 7fcb77fff700 1 --2- 172.21.15.70:0/2373004333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 0x5566ad9d4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:37.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.683+0000 7fcb777fe700 1 -- 172.21.15.70:0/2373004333 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9cb2e0 msgr2=0x5566ad9d4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.683+0000 7fcb777fe700 1 --2- 172.21.15.70:0/2373004333 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9cb2e0 0x5566ad9d4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.683+0000 7fcb777fe700 1 -- 172.21.15.70:0/2373004333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 msgr2=0x5566ad9d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.683+0000 7fcb777fe700 1 --2- 172.21.15.70:0/2373004333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 0x5566ad9d4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.683+0000 7fcb777fe700 1 -- 172.21.15.70:0/2373004333 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566ad8678d0 con 0x5566ad9c9de0 2024-06-04T05:23:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.683+0000 7fcb777fe700 1 --2- 172.21.15.70:0/2373004333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9d7780 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fcb68009990 tx=0x7fcb68009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359e31ed7347b173 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.683+0000 7fcb56ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.683+0000 7fcb56ffd700 1 -- 172.21.15.70:0/2373004333 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb68035060 con 0x5566ad9c9de0 2024-06-04T05:23:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.683+0000 7fcb56ffd700 1 -- 172.21.15.70:0/2373004333 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fcb68003bd0 con 0x5566ad9c9de0 2024-06-04T05:23:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.683+0000 7fcb886f7b80 1 -- 172.21.15.70:0/2373004333 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb600a9170 con 0x5566ad9c9de0 2024-06-04T05:23:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.683+0000 7fcb56ffd700 1 -- 172.21.15.70:0/2373004333 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fcb68003e10 con 0x5566ad9c9de0 2024-06-04T05:23:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.683+0000 7fcb886f7b80 10 client.22137 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.683+0000 7fcb886f7b80 1 -- 172.21.15.70:0/2373004333 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x5566ad8d1760 con 0x5566ad9c9de0 2024-06-04T05:23:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.683+0000 7fcb56ffd700 1 -- 172.21.15.70:0/2373004333 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb600a9170 con 0x5566ad9c9de0 2024-06-04T05:23:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.684+0000 7fcb56ffd700 1 -- 172.21.15.70:0/2373004333 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7fcb6802f070 con 0x5566ad9c9de0 2024-06-04T05:23:37.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.684+0000 7fcb886f7b80 10 client.22137 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:37.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.684+0000 7fcb886f7b80 10 client.22137 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:23:37.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.684+0000 7fcb886f7b80 20 client.22137 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:37.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.684+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/2373004333 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5566ad9e7790 0x5566ad9e9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.684+0000 7fcb886f7b80 4 client.22137 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:23:37.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.684+0000 7fcb886f7b80 1 -- 172.21.15.70:0/2373004333 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566ad929cc0 con 0x5566ad9e7790 2024-06-04T05:23:37.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.684+0000 7fcb76ffd700 1 --2- 172.21.15.70:0/2373004333 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5566ad9e7790 0x5566ad9e9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:37.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.684+0000 7fcb56ffd700 10 client.22137 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:23:37.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.684+0000 7fcb76ffd700 1 --2- 172.21.15.70:0/2373004333 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5566ad9e7790 0x5566ad9e9bc0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2327464fc255a8d6 server_cookie=c066e6df45e62f4e in_seq=0 out_seq=0 2024-06-04T05:23:37.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.685+0000 7fcb56ffd700 1 -- 172.21.15.70:0/2373004333 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5566ad929cc0 con 0x5566ad9e7790 2024-06-04T05:23:37.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.685+0000 7fcb56ffd700 10 client.22137 handle_command_reply: tid=0 2024-06-04T05:23:37.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.685+0000 7fcb886f7b80 1 client.22137 shutdown 2024-06-04T05:23:37.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.685+0000 7fcb886f7b80 1 -- 172.21.15.70:0/2373004333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 msgr2=0x5566ad9d7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:37.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.685+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/2373004333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9d7780 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fcb68009990 tx=0x7fcb68009310 comp rx=0 tx=0).stop 2024-06-04T05:23:37.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.685+0000 7fcb886f7b80 1 -- 172.21.15.70:0/2373004333 shutdown_connections 2024-06-04T05:23:37.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.685+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/2373004333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 0x5566ad9d4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.685+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/2373004333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9d7780 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.685+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/2373004333 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5566ad9e7790 0x5566ad9e9bc0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.685+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/2373004333 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9cb2e0 0x5566ad9d4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.685+0000 7fcb886f7b80 1 -- 172.21.15.70:0/2373004333 >> 172.21.15.70:0/2373004333 conn(0x5566ad8f5240 msgr2=0x5566ad9c3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:37.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.685+0000 7fcb56ffd700 0 client.22137 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:23:37.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.685+0000 7fcb886f7b80 1 -- 172.21.15.70:0/2373004333 shutdown_connections 2024-06-04T05:23:37.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.685+0000 7fcb886f7b80 1 -- 172.21.15.70:0/2373004333 wait complete. 2024-06-04T05:23:37.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.685+0000 7fcb886f7b80 20 client.22137 trim_cache size 0 max 16384 2024-06-04T05:23:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.700+0000 7fcb886f7b80 1 Processor -- start 2024-06-04T05:23:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.700+0000 7fcb886f7b80 1 -- start start 2024-06-04T05:23:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.700+0000 7fcb886f7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9c9de0 0x5566ad9e3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.700+0000 7fcb886f7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9cb2e0 0x5566ad9c4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.700+0000 7fcb886f7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 0x5566ad9d3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.700+0000 7fcb886f7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781abbf0 con 0x5566ad9cb2e0 2024-06-04T05:23:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.700+0000 7fcb886f7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781ad8e0 con 0x5566ad9d7f20 2024-06-04T05:23:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.700+0000 7fcb886f7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781ac350 con 0x5566ad9c9de0 2024-06-04T05:23:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.701+0000 7fcb76ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9cb2e0 0x5566ad9c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.701+0000 7fcb76ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9cb2e0 0x5566ad9c4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42100/0 (socket says 172.21.15.70:42100) 2024-06-04T05:23:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.701+0000 7fcb76ffd700 1 -- 172.21.15.70:0/2426971361 learned_addr learned my addr 172.21.15.70:0/2426971361 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.701+0000 7fcb777fe700 1 --2- 172.21.15.70:0/2426971361 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9c9de0 0x5566ad9e3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.701+0000 7fcb76ffd700 1 -- 172.21.15.70:0/2426971361 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9c9de0 msgr2=0x5566ad9e3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.701+0000 7fcb76ffd700 1 --2- 172.21.15.70:0/2426971361 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9c9de0 0x5566ad9e3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.701+0000 7fcb76ffd700 1 -- 172.21.15.70:0/2426971361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 msgr2=0x5566ad9d3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.701+0000 7fcb76ffd700 1 --2- 172.21.15.70:0/2426971361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 0x5566ad9d3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.701+0000 7fcb76ffd700 1 -- 172.21.15.70:0/2426971361 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566ad9d2ff0 con 0x5566ad9cb2e0 2024-06-04T05:23:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.701+0000 7fcb777fe700 1 --2- 172.21.15.70:0/2426971361 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9c9de0 0x5566ad9e3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.701+0000 7fcb76ffd700 1 --2- 172.21.15.70:0/2426971361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9cb2e0 0x5566ad9c4270 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fcb600df690 tx=0x7fcb600a5030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6424f52fdc27dd32 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.701+0000 7fcb6d7fa700 1 -- 172.21.15.70:0/2426971361 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb600c3040 con 0x5566ad9cb2e0 2024-06-04T05:23:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.701+0000 7fcb6d7fa700 1 -- 172.21.15.70:0/2426971361 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb600ad170 con 0x5566ad9cb2e0 2024-06-04T05:23:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.701+0000 7fcb6d7fa700 1 -- 172.21.15.70:0/2426971361 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb600a9630 con 0x5566ad9cb2e0 2024-06-04T05:23:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.702+0000 7fcb886f7b80 1 -- 172.21.15.70:0/2426971361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9cb2e0 msgr2=0x5566ad9c4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.702+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/2426971361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9cb2e0 0x5566ad9c4270 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fcb600df690 tx=0x7fcb600a5030 comp rx=0 tx=0).stop 2024-06-04T05:23:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.702+0000 7fcb886f7b80 1 -- 172.21.15.70:0/2426971361 shutdown_connections 2024-06-04T05:23:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.702+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/2426971361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 0x5566ad9d3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.702+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/2426971361 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9c9de0 0x5566ad9e3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.702+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/2426971361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9cb2e0 0x5566ad9c4270 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.702+0000 7fcb886f7b80 1 -- 172.21.15.70:0/2426971361 >> 172.21.15.70:0/2426971361 conn(0x5566ad8f5240 msgr2=0x5566ad9c8890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.702+0000 7fcb886f7b80 1 -- 172.21.15.70:0/2426971361 shutdown_connections 2024-06-04T05:23:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.702+0000 7fcb886f7b80 1 -- 172.21.15.70:0/2426971361 wait complete. 2024-06-04T05:23:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.702+0000 7fcb886f7b80 1 -- start start 2024-06-04T05:23:37.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb886f7b80 1 Processor -- start 2024-06-04T05:23:37.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb886f7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9dff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb886f7b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9cb2e0 0x5566ad9e04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb886f7b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 0x5566ad9d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb886f7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781ad1e0 con 0x5566ad9c9de0 2024-06-04T05:23:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb886f7b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781aca50 con 0x5566ad9d7f20 2024-06-04T05:23:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb886f7b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781a9bb0 con 0x5566ad9cb2e0 2024-06-04T05:23:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb886f7b80 10 client.0 _ll_register_callbacks cb 0x5566ad8b2950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb777fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9dff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb777fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9dff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42102/0 (socket says 172.21.15.70:42102) 2024-06-04T05:23:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb777fe700 1 -- 172.21.15.70:0/4074390764 learned_addr learned my addr 172.21.15.70:0/4074390764 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb76ffd700 1 --2- 172.21.15.70:0/4074390764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9cb2e0 0x5566ad9e04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb77fff700 1 --2- 172.21.15.70:0/4074390764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 0x5566ad9d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb777fe700 1 -- 172.21.15.70:0/4074390764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9cb2e0 msgr2=0x5566ad9e04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb777fe700 1 --2- 172.21.15.70:0/4074390764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9cb2e0 0x5566ad9e04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb777fe700 1 -- 172.21.15.70:0/4074390764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 msgr2=0x5566ad9d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb777fe700 1 --2- 172.21.15.70:0/4074390764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 0x5566ad9d12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb777fe700 1 -- 172.21.15.70:0/4074390764 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566ad9d1b30 con 0x5566ad9c9de0 2024-06-04T05:23:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb77fff700 1 --2- 172.21.15.70:0/4074390764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 0x5566ad9d12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:23:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb777fe700 1 --2- 172.21.15.70:0/4074390764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9dff60 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fcb68009ce0 tx=0x7fcb6800bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7075a7e136ec65a3 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb56ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.703+0000 7fcb56ffd700 1 -- 172.21.15.70:0/4074390764 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb68035070 con 0x5566ad9c9de0 2024-06-04T05:23:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.704+0000 7fcb56ffd700 1 -- 172.21.15.70:0/4074390764 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fcb68030d00 con 0x5566ad9c9de0 2024-06-04T05:23:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.704+0000 7fcb886f7b80 1 -- 172.21.15.70:0/4074390764 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb600ad170 con 0x5566ad9c9de0 2024-06-04T05:23:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.704+0000 7fcb56ffd700 1 -- 172.21.15.70:0/4074390764 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fcb68049050 con 0x5566ad9c9de0 2024-06-04T05:23:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.704+0000 7fcb886f7b80 10 client.22143 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.704+0000 7fcb886f7b80 1 -- 172.21.15.70:0/4074390764 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x5566ada51ec0 con 0x5566ad9c9de0 2024-06-04T05:23:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.704+0000 7fcb56ffd700 1 -- 172.21.15.70:0/4074390764 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb600ad170 con 0x5566ad9c9de0 2024-06-04T05:23:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.704+0000 7fcb56ffd700 1 -- 172.21.15.70:0/4074390764 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7fcb6802f040 con 0x5566ad9c9de0 2024-06-04T05:23:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.704+0000 7fcb886f7b80 10 client.22143 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.704+0000 7fcb886f7b80 10 client.22143 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-06-04T05:23:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.704+0000 7fcb886f7b80 20 client.22143 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.704+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/4074390764 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5566ada525a0 0x5566ada549c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.704+0000 7fcb886f7b80 4 client.22143 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:23:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.704+0000 7fcb886f7b80 1 -- 172.21.15.70:0/4074390764 --> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5566ad9234a0 con 0x5566ada525a0 2024-06-04T05:23:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.704+0000 7fcb76ffd700 1 --2- 172.21.15.70:0/4074390764 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5566ada525a0 0x5566ada549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.705+0000 7fcb76ffd700 1 --2- 172.21.15.70:0/4074390764 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5566ada525a0 0x5566ada549c0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a7adb5965c280c6 server_cookie=58b82b5be36dfe32 in_seq=0 out_seq=0 2024-06-04T05:23:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.705+0000 7fcb56ffd700 10 client.22143 ms_handle_connect on v2:172.21.15.112:6834/242926435 2024-06-04T05:23:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.705+0000 7fcb56ffd700 1 -- 172.21.15.70:0/4074390764 <== mds.3 v2:172.21.15.112:6834/242926435 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5566ad9234a0 con 0x5566ada525a0 2024-06-04T05:23:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.705+0000 7fcb56ffd700 10 client.22143 handle_command_reply: tid=0 2024-06-04T05:23:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.705+0000 7fcb886f7b80 1 client.22143 shutdown 2024-06-04T05:23:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.705+0000 7fcb886f7b80 1 -- 172.21.15.70:0/4074390764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 msgr2=0x5566ad9dff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.705+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/4074390764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9dff60 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fcb68009ce0 tx=0x7fcb6800bef0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.705+0000 7fcb886f7b80 1 -- 172.21.15.70:0/4074390764 shutdown_connections 2024-06-04T05:23:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.705+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/4074390764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566ad9d7f20 0x5566ad9d12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.705+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/4074390764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566ad9c9de0 0x5566ad9dff60 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.705+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/4074390764 >> [v2:172.21.15.112:6834/242926435,v1:172.21.15.112:6835/242926435] conn(0x5566ada525a0 0x5566ada549c0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.705+0000 7fcb886f7b80 1 --2- 172.21.15.70:0/4074390764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5566ad9cb2e0 0x5566ad9e04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.705+0000 7fcb886f7b80 1 -- 172.21.15.70:0/4074390764 >> 172.21.15.70:0/4074390764 conn(0x5566ad8f5240 msgr2=0x5566ad9d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.705+0000 7fcb56ffd700 0 client.22143 ms_handle_reset on v2:172.21.15.112:6834/242926435 2024-06-04T05:23:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.705+0000 7fcb886f7b80 1 -- 172.21.15.70:0/4074390764 shutdown_connections 2024-06-04T05:23:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.705+0000 7fcb886f7b80 1 -- 172.21.15.70:0/4074390764 wait complete. 2024-06-04T05:23:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.705+0000 7fcb886f7b80 20 client.22143 trim_cache size 0 max 16384 2024-06-04T05:23:37.714 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:23:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.706+0000 7fcb7de62700 1 -- 172.21.15.70:0/368033910 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcb640b29f0 msgr2=0x7fcb640b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.706+0000 7fcb7de62700 1 --2- 172.21.15.70:0/368033910 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcb640b29f0 0x7fcb640b4eb0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fcb6800b5c0 tx=0x7fcb6801a040 comp rx=0 tx=0).stop 2024-06-04T05:23:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.706+0000 7fcb7de62700 1 -- 172.21.15.70:0/368033910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb78102e00 msgr2=0x7fcb7819ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.706+0000 7fcb7de62700 1 --2- 172.21.15.70:0/368033910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb78102e00 0x7fcb7819ed00 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7fcb6000b530 tx=0x7fcb6000b840 comp rx=0 tx=0).stop 2024-06-04T05:23:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.706+0000 7fcb7de62700 1 -- 172.21.15.70:0/368033910 shutdown_connections 2024-06-04T05:23:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.706+0000 7fcb7de62700 1 --2- 172.21.15.70:0/368033910 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fcb640b29f0 0x7fcb640b4eb0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.706+0000 7fcb7de62700 1 --2- 172.21.15.70:0/368033910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb78108520 0x7fcb78198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.706+0000 7fcb7de62700 1 --2- 172.21.15.70:0/368033910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb78102440 0x7fcb7819e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.706+0000 7fcb7de62700 1 --2- 172.21.15.70:0/368033910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb78102e00 0x7fcb7819ed00 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.706+0000 7fcb7de62700 1 -- 172.21.15.70:0/368033910 >> 172.21.15.70:0/368033910 conn(0x7fcb780793f0 msgr2=0x7fcb78101420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.706+0000 7fcb7de62700 1 -- 172.21.15.70:0/368033910 shutdown_connections 2024-06-04T05:23:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:37.706+0000 7fcb7de62700 1 -- 172.21.15.70:0/368033910 wait complete. 2024-06-04T05:23:37.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:37 smithi112 ceph-mon[36453]: pgmap v834: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-06-04T05:23:37.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:37 smithi098 ceph-mon[35569]: pgmap v834: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-06-04T05:23:38.498 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-04T05:23:38.737 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:39.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:39 smithi070 ceph-mon[32631]: pgmap v835: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 KiB/s wr, 13 op/s 2024-06-04T05:23:39.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:39 smithi112 ceph-mon[36453]: pgmap v835: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 KiB/s wr, 13 op/s 2024-06-04T05:23:39.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:39 smithi098 ceph-mon[35569]: pgmap v835: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 KiB/s wr, 13 op/s 2024-06-04T05:23:40.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.584+0000 7f66e5c33700 1 -- 172.21.15.70:0/1032776276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66e0102430 msgr2=0x7f66e01028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:40.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.584+0000 7f66e5c33700 1 --2- 172.21.15.70:0/1032776276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66e0102430 0x7f66e01028b0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f66d0009990 tx=0x7f66d0009ca0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.584+0000 7f66e5c33700 1 -- 172.21.15.70:0/1032776276 shutdown_connections 2024-06-04T05:23:40.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.584+0000 7f66e5c33700 1 --2- 172.21.15.70:0/1032776276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e0102df0 0x7f66e010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.584+0000 7f66e5c33700 1 --2- 172.21.15.70:0/1032776276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66e0108510 0x7f66e01088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.584+0000 7f66e5c33700 1 --2- 172.21.15.70:0/1032776276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66e0102430 0x7f66e01028b0 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.584+0000 7f66e5c33700 1 -- 172.21.15.70:0/1032776276 >> 172.21.15.70:0/1032776276 conn(0x7f66e00793b0 msgr2=0x7f66e00797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:40.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.584+0000 7f66e5c33700 1 -- 172.21.15.70:0/1032776276 shutdown_connections 2024-06-04T05:23:40.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.584+0000 7f66e5c33700 1 -- 172.21.15.70:0/1032776276 wait complete. 2024-06-04T05:23:40.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.585+0000 7f66e5c33700 1 Processor -- start 2024-06-04T05:23:40.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.585+0000 7f66e5c33700 1 -- start start 2024-06-04T05:23:40.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.585+0000 7f66e5c33700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66e0102430 0x7f66e019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.585+0000 7f66e5c33700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66e0102df0 0x7f66e019aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.585+0000 7f66e5c33700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e0108510 0x7f66e01a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.585+0000 7f66e5c33700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f66e0112470 con 0x7f66e0102df0 2024-06-04T05:23:40.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.585+0000 7f66e5c33700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66e01122f0 con 0x7f66e0102430 2024-06-04T05:23:40.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.585+0000 7f66e5c33700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f66e01125f0 con 0x7f66e0108510 2024-06-04T05:23:40.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.585+0000 7f66deffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66e0102df0 0x7f66e019aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:40.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.585+0000 7f66deffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66e0102df0 0x7f66e019aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42120/0 (socket says 172.21.15.70:42120) 2024-06-04T05:23:40.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.585+0000 7f66deffd700 1 -- 172.21.15.70:0/2204952978 learned_addr learned my addr 172.21.15.70:0/2204952978 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.585+0000 7f66df7fe700 1 --2- 172.21.15.70:0/2204952978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66e0102430 0x7f66e019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.585+0000 7f66dffff700 1 --2- 172.21.15.70:0/2204952978 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e0108510 0x7f66e01a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.585+0000 7f66deffd700 1 -- 172.21.15.70:0/2204952978 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e0108510 msgr2=0x7f66e01a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.586+0000 7f66deffd700 1 --2- 172.21.15.70:0/2204952978 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e0108510 0x7f66e01a59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.586+0000 7f66deffd700 1 -- 172.21.15.70:0/2204952978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66e0102430 msgr2=0x7f66e019a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.586+0000 7f66deffd700 1 --2- 172.21.15.70:0/2204952978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66e0102430 0x7f66e019a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.586+0000 7f66deffd700 1 -- 172.21.15.70:0/2204952978 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66e01a5f30 con 0x7f66e0102df0 2024-06-04T05:23:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.586+0000 7f66dffff700 1 --2- 172.21.15.70:0/2204952978 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e0108510 0x7f66e01a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:23:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.586+0000 7f66deffd700 1 --2- 172.21.15.70:0/2204952978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66e0102df0 0x7f66e019aeb0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f66d0000c00 tx=0x7f66d00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.586+0000 7f66dcff9700 1 -- 172.21.15.70:0/2204952978 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d0027070 con 0x7f66e0102df0 2024-06-04T05:23:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.586+0000 7f66e5c33700 1 -- 172.21.15.70:0/2204952978 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66e01a6100 con 0x7f66e0102df0 2024-06-04T05:23:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.586+0000 7f66dcff9700 1 -- 172.21.15.70:0/2204952978 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f66d000fb80 con 0x7f66e0102df0 2024-06-04T05:23:40.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.586+0000 7f66dcff9700 1 -- 172.21.15.70:0/2204952978 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d0022d00 con 0x7f66e0102df0 2024-06-04T05:23:40.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.586+0000 7f66e5c33700 1 -- 172.21.15.70:0/2204952978 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66e01a6550 con 0x7f66e0102df0 2024-06-04T05:23:40.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.589+0000 7f66dcff9700 1 -- 172.21.15.70:0/2204952978 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f66d002a070 con 0x7f66e0102df0 2024-06-04T05:23:40.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.590+0000 7f66dcff9700 1 --2- 172.21.15.70:0/2204952978 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f66cc0b29f0 0x7f66cc0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.590+0000 7f66df7fe700 1 --2- 172.21.15.70:0/2204952978 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f66cc0b29f0 0x7f66cc0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:40.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.590+0000 7f66dcff9700 1 -- 172.21.15.70:0/2204952978 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7f66d00e3470 con 0x7f66e0102df0 2024-06-04T05:23:40.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.590+0000 7f66f04c8b80 1 Processor -- start 2024-06-04T05:23:40.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.590+0000 7f66f04c8b80 1 -- start start 2024-06-04T05:23:40.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.590+0000 7f66df7fe700 1 --2- 172.21.15.70:0/2204952978 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f66cc0b29f0 0x7f66cc0b4eb0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f66c8005950 tx=0x7f66c800b410 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:40.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.590+0000 7f66f04c8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab6c70 0x55bd13ab7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.590+0000 7f66f04c8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13ab7620 0x55bd13abad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.590+0000 7f66f04c8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13abb370 0x55bd13abb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.590+0000 7f66f04c8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bd13a11c90 con 0x55bd13ab6c70 2024-06-04T05:23:40.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.590+0000 7f66f04c8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bd13a01830 con 0x55bd13abb370 2024-06-04T05:23:40.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.590+0000 7f66f04c8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bd13a05870 con 0x55bd13ab7620 2024-06-04T05:23:40.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.591+0000 7f66df7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab6c70 0x55bd13ab7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:40.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.591+0000 7f66df7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab6c70 0x55bd13ab7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42126/0 (socket says 172.21.15.70:42126) 2024-06-04T05:23:40.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.591+0000 7f66df7fe700 1 -- 172.21.15.70:0/938209962 learned_addr learned my addr 172.21.15.70:0/938209962 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:40.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.591+0000 7f66df7fe700 1 -- 172.21.15.70:0/938209962 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13ab7620 msgr2=0x55bd13abad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:40.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.591+0000 7f66df7fe700 1 --2- 172.21.15.70:0/938209962 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13ab7620 0x55bd13abad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.591+0000 7f66df7fe700 1 -- 172.21.15.70:0/938209962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13abb370 msgr2=0x55bd13abb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:40.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.591+0000 7f66dffff700 1 --2- 172.21.15.70:0/938209962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13abb370 0x55bd13abb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:40.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.591+0000 7f66df7fe700 1 --2- 172.21.15.70:0/938209962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13abb370 0x55bd13abb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.591+0000 7f66df7fe700 1 -- 172.21.15.70:0/938209962 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd139518d0 con 0x55bd13ab6c70 2024-06-04T05:23:40.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.591+0000 7f66dffff700 1 --2- 172.21.15.70:0/938209962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13abb370 0x55bd13abb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:23:40.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.591+0000 7f66df7fe700 1 --2- 172.21.15.70:0/938209962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab6c70 0x55bd13ab7050 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f66c8010e60 tx=0x7f66c8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72bbdadd4b4e5a76 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:40.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.591+0000 7f66d5ffb700 1 -- 172.21.15.70:0/938209962 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c801e620 con 0x55bd13ab6c70 2024-06-04T05:23:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.591+0000 7f66d5ffb700 1 -- 172.21.15.70:0/938209962 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f66c8010e90 con 0x55bd13ab6c70 2024-06-04T05:23:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.591+0000 7f66d5ffb700 1 -- 172.21.15.70:0/938209962 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c801ca90 con 0x55bd13ab6c70 2024-06-04T05:23:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.592+0000 7f66f04c8b80 1 -- 172.21.15.70:0/938209962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab6c70 msgr2=0x55bd13ab7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.592+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/938209962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab6c70 0x55bd13ab7050 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f66c8010e60 tx=0x7f66c8013000 comp rx=0 tx=0).stop 2024-06-04T05:23:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.592+0000 7f66f04c8b80 1 -- 172.21.15.70:0/938209962 shutdown_connections 2024-06-04T05:23:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.592+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/938209962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13abb370 0x55bd13abb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.592+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/938209962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab6c70 0x55bd13ab7050 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.592+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/938209962 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13ab7620 0x55bd13abad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.592+0000 7f66f04c8b80 1 -- 172.21.15.70:0/938209962 >> 172.21.15.70:0/938209962 conn(0x55bd139e0080 msgr2=0x55bd13aace80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.592+0000 7f66f04c8b80 1 -- 172.21.15.70:0/938209962 shutdown_connections 2024-06-04T05:23:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.592+0000 7f66f04c8b80 1 -- 172.21.15.70:0/938209962 wait complete. 2024-06-04T05:23:40.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.592+0000 7f66f04c8b80 1 -- start start 2024-06-04T05:23:40.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.593+0000 7f66f04c8b80 1 Processor -- start 2024-06-04T05:23:40.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.593+0000 7f66f04c8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab6c70 0x55bd13ac2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.593+0000 7f66f04c8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13ab7620 0x55bd13abfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.593+0000 7f66f04c8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13abb370 0x55bd13ac0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.593+0000 7f66f04c8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bd13907fa0 con 0x55bd13ab6c70 2024-06-04T05:23:40.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.593+0000 7f66f04c8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bd139a37c0 con 0x55bd13abb370 2024-06-04T05:23:40.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.593+0000 7f66f04c8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bd13ac3530 con 0x55bd13ab7620 2024-06-04T05:23:40.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.593+0000 7f66f04c8b80 10 client.0 _ll_register_callbacks cb 0x55bd1399c920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:40.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.593+0000 7f66df7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab6c70 0x55bd13ac2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:40.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.593+0000 7f66df7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab6c70 0x55bd13ac2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42128/0 (socket says 172.21.15.70:42128) 2024-06-04T05:23:40.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.593+0000 7f66df7fe700 1 -- 172.21.15.70:0/157639989 learned_addr learned my addr 172.21.15.70:0/157639989 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:40.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.593+0000 7f66deffd700 1 --2- 172.21.15.70:0/157639989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13ab7620 0x55bd13abfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:40.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.593+0000 7f66dffff700 1 --2- 172.21.15.70:0/157639989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13abb370 0x55bd13ac0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.594+0000 7f66df7fe700 1 -- 172.21.15.70:0/157639989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13ab7620 msgr2=0x55bd13abfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.594+0000 7f66df7fe700 1 --2- 172.21.15.70:0/157639989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13ab7620 0x55bd13abfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.594+0000 7f66df7fe700 1 -- 172.21.15.70:0/157639989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13abb370 msgr2=0x55bd13ac0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.594+0000 7f66df7fe700 1 --2- 172.21.15.70:0/157639989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13abb370 0x55bd13ac0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.594+0000 7f66df7fe700 1 -- 172.21.15.70:0/157639989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd139a0690 con 0x55bd13ab6c70 2024-06-04T05:23:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.594+0000 7f66df7fe700 1 --2- 172.21.15.70:0/157639989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab6c70 0x55bd13ac2840 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f66c80052f0 tx=0x7f66c8005320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d584fbc3c5497e2a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.594+0000 7f66beffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.594+0000 7f66beffd700 1 -- 172.21.15.70:0/157639989 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c8004690 con 0x55bd13ab6c70 2024-06-04T05:23:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.594+0000 7f66beffd700 1 -- 172.21.15.70:0/157639989 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7f66c8004830 con 0x55bd13ab6c70 2024-06-04T05:23:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.594+0000 7f66f04c8b80 1 -- 172.21.15.70:0/157639989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66c8010e90 con 0x55bd13ab6c70 2024-06-04T05:23:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.594+0000 7f66beffd700 1 -- 172.21.15.70:0/157639989 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f66c8004a70 con 0x55bd13ab6c70 2024-06-04T05:23:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.594+0000 7f66f04c8b80 10 client.22155 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.594+0000 7f66f04c8b80 1 -- 172.21.15.70:0/157639989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x55bd13acdc80 con 0x55bd13ab6c70 2024-06-04T05:23:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.595+0000 7f66beffd700 1 -- 172.21.15.70:0/157639989 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f66c8010e90 con 0x55bd13ab6c70 2024-06-04T05:23:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.595+0000 7f66beffd700 1 -- 172.21.15.70:0/157639989 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7f66c801e420 con 0x55bd13ab6c70 2024-06-04T05:23:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.595+0000 7f66f04c8b80 10 client.22155 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.595+0000 7f66f04c8b80 10 client.22155 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:23:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.595+0000 7f66f04c8b80 20 client.22155 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.595+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/157639989 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55bd13ad0200 0x55bd13ad25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.595+0000 7f66f04c8b80 4 client.22155 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:23:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.595+0000 7f66f04c8b80 1 -- 172.21.15.70:0/157639989 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd13a7e9e0 con 0x55bd13ad0200 2024-06-04T05:23:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.595+0000 7f66deffd700 1 --2- 172.21.15.70:0/157639989 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55bd13ad0200 0x55bd13ad25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.595+0000 7f66deffd700 1 --2- 172.21.15.70:0/157639989 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55bd13ad0200 0x55bd13ad25f0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c4815d0600024fca server_cookie=470dc51695d6cb7c in_seq=0 out_seq=0 2024-06-04T05:23:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.595+0000 7f66beffd700 10 client.22155 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:23:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.596+0000 7f66beffd700 1 -- 172.21.15.70:0/157639989 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bd13a7e9e0 con 0x55bd13ad0200 2024-06-04T05:23:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.596+0000 7f66beffd700 10 client.22155 handle_command_reply: tid=0 2024-06-04T05:23:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.596+0000 7f66f04c8b80 1 client.22155 shutdown 2024-06-04T05:23:40.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.596+0000 7f66f04c8b80 1 -- 172.21.15.70:0/157639989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab6c70 msgr2=0x55bd13ac2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:40.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.596+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/157639989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab6c70 0x55bd13ac2840 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f66c80052f0 tx=0x7f66c8005320 comp rx=0 tx=0).stop 2024-06-04T05:23:40.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.597+0000 7f66f04c8b80 1 -- 172.21.15.70:0/157639989 shutdown_connections 2024-06-04T05:23:40.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.597+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/157639989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13abb370 0x55bd13ac0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.597+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/157639989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab6c70 0x55bd13ac2840 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.597+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/157639989 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55bd13ad0200 0x55bd13ad25f0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.597+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/157639989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13ab7620 0x55bd13abfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.597+0000 7f66beffd700 0 client.22155 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:23:40.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.597+0000 7f66f04c8b80 1 -- 172.21.15.70:0/157639989 >> 172.21.15.70:0/157639989 conn(0x55bd139e0080 msgr2=0x55bd13aac510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:40.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.597+0000 7f66f04c8b80 1 -- 172.21.15.70:0/157639989 shutdown_connections 2024-06-04T05:23:40.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.597+0000 7f66f04c8b80 1 -- 172.21.15.70:0/157639989 wait complete. 2024-06-04T05:23:40.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.597+0000 7f66f04c8b80 20 client.22155 trim_cache size 0 max 16384 2024-06-04T05:23:40.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66f04c8b80 1 Processor -- start 2024-06-04T05:23:40.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66f04c8b80 1 -- start start 2024-06-04T05:23:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66f04c8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13ab6c70 0x55bd13ab1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66f04c8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab7620 0x55bd13aacc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66f04c8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13abb370 0x55bd13aad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66f04c8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f66e01abbb0 con 0x55bd13ab7620 2024-06-04T05:23:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66f04c8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66e01ad8a0 con 0x55bd13ab6c70 2024-06-04T05:23:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66f04c8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f66e01ac310 con 0x55bd13abb370 2024-06-04T05:23:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66deffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab7620 0x55bd13aacc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66deffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab7620 0x55bd13aacc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42144/0 (socket says 172.21.15.70:42144) 2024-06-04T05:23:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66deffd700 1 -- 172.21.15.70:0/266150941 learned_addr learned my addr 172.21.15.70:0/266150941 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66df7fe700 1 --2- 172.21.15.70:0/266150941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13ab6c70 0x55bd13ab1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66dffff700 1 --2- 172.21.15.70:0/266150941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13abb370 0x55bd13aad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66deffd700 1 -- 172.21.15.70:0/266150941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13abb370 msgr2=0x55bd13aad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:40.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66deffd700 1 --2- 172.21.15.70:0/266150941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13abb370 0x55bd13aad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66deffd700 1 -- 172.21.15.70:0/266150941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13ab6c70 msgr2=0x55bd13ab1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:40.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66deffd700 1 --2- 172.21.15.70:0/266150941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13ab6c70 0x55bd13ab1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66deffd700 1 -- 172.21.15.70:0/266150941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd13ac09f0 con 0x55bd13ab7620 2024-06-04T05:23:40.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.612+0000 7f66dffff700 1 --2- 172.21.15.70:0/266150941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13abb370 0x55bd13aad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:23:40.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.613+0000 7f66deffd700 1 --2- 172.21.15.70:0/266150941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab7620 0x55bd13aacc90 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f66d00a60b0 tx=0x7f66d00cd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb94e0dae81ce480 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:40.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.613+0000 7f66d57fa700 1 -- 172.21.15.70:0/266150941 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d00aa7b0 con 0x55bd13ab7620 2024-06-04T05:23:40.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.613+0000 7f66d57fa700 1 -- 172.21.15.70:0/266150941 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f66d00b1e50 con 0x55bd13ab7620 2024-06-04T05:23:40.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.613+0000 7f66d57fa700 1 -- 172.21.15.70:0/266150941 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d00aaa10 con 0x55bd13ab7620 2024-06-04T05:23:40.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.613+0000 7f66f04c8b80 1 -- 172.21.15.70:0/266150941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab7620 msgr2=0x55bd13aacc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:40.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.613+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/266150941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab7620 0x55bd13aacc90 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f66d00a60b0 tx=0x7f66d00cd040 comp rx=0 tx=0).stop 2024-06-04T05:23:40.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.613+0000 7f66f04c8b80 1 -- 172.21.15.70:0/266150941 shutdown_connections 2024-06-04T05:23:40.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.613+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/266150941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13abb370 0x55bd13aad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.613+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/266150941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13ab6c70 0x55bd13ab1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.613+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/266150941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab7620 0x55bd13aacc90 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.613+0000 7f66f04c8b80 1 -- 172.21.15.70:0/266150941 >> 172.21.15.70:0/266150941 conn(0x55bd139e0080 msgr2=0x55bd13ab1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.614+0000 7f66f04c8b80 1 -- 172.21.15.70:0/266150941 shutdown_connections 2024-06-04T05:23:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.614+0000 7f66f04c8b80 1 -- 172.21.15.70:0/266150941 wait complete. 2024-06-04T05:23:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.614+0000 7f66f04c8b80 1 -- start start 2024-06-04T05:23:40.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.614+0000 7f66f04c8b80 1 Processor -- start 2024-06-04T05:23:40.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.614+0000 7f66f04c8b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13ab6c70 0x55bd13b32fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.614+0000 7f66f04c8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab7620 0x55bd13b33500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.614+0000 7f66f04c8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13abb370 0x55bd13ac5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.614+0000 7f66f04c8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f66e01ad1a0 con 0x55bd13ab7620 2024-06-04T05:23:40.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.614+0000 7f66f04c8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66e01aca10 con 0x55bd13abb370 2024-06-04T05:23:40.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.614+0000 7f66f04c8b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f66e01a9b70 con 0x55bd13ab6c70 2024-06-04T05:23:40.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.614+0000 7f66f04c8b80 10 client.0 _ll_register_callbacks cb 0x55bd1399c920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:40.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.614+0000 7f66deffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab7620 0x55bd13b33500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:40.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.614+0000 7f66deffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab7620 0x55bd13b33500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42146/0 (socket says 172.21.15.70:42146) 2024-06-04T05:23:40.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.614+0000 7f66deffd700 1 -- 172.21.15.70:0/3755497230 learned_addr learned my addr 172.21.15.70:0/3755497230 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:40.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.615+0000 7f66df7fe700 1 --2- 172.21.15.70:0/3755497230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13ab6c70 0x55bd13b32fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:40.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.615+0000 7f66dffff700 1 --2- 172.21.15.70:0/3755497230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13abb370 0x55bd13ac5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:40.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.615+0000 7f66deffd700 1 -- 172.21.15.70:0/3755497230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13ab6c70 msgr2=0x55bd13b32fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:40.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.615+0000 7f66deffd700 1 --2- 172.21.15.70:0/3755497230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13ab6c70 0x55bd13b32fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.615+0000 7f66deffd700 1 -- 172.21.15.70:0/3755497230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13abb370 msgr2=0x55bd13ac5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:40.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.615+0000 7f66deffd700 1 --2- 172.21.15.70:0/3755497230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13abb370 0x55bd13ac5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.615+0000 7f66deffd700 1 -- 172.21.15.70:0/3755497230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd13ac6560 con 0x55bd13ab7620 2024-06-04T05:23:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.615+0000 7f66deffd700 1 --2- 172.21.15.70:0/3755497230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab7620 0x55bd13b33500 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f66d00aa4b0 tx=0x7f66d000ff60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=583bb6528f904c73 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.615+0000 7f66beffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.615+0000 7f66beffd700 1 -- 172.21.15.70:0/3755497230 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d00aa7b0 con 0x55bd13ab7620 2024-06-04T05:23:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.615+0000 7f66beffd700 1 -- 172.21.15.70:0/3755497230 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7f66d00aaa10 con 0x55bd13ab7620 2024-06-04T05:23:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.615+0000 7f66f04c8b80 1 -- 172.21.15.70:0/3755497230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66d00b1e50 con 0x55bd13ab7620 2024-06-04T05:23:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.615+0000 7f66beffd700 1 -- 172.21.15.70:0/3755497230 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f66d00aac30 con 0x55bd13ab7620 2024-06-04T05:23:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.615+0000 7f66f04c8b80 10 client.22161 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.615+0000 7f66f04c8b80 1 -- 172.21.15.70:0/3755497230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x55bd13b3a760 con 0x55bd13ab7620 2024-06-04T05:23:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.615+0000 7f66beffd700 1 -- 172.21.15.70:0/3755497230 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f66d00b1e50 con 0x55bd13ab7620 2024-06-04T05:23:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.616+0000 7f66beffd700 1 -- 172.21.15.70:0/3755497230 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7f66d000f6f0 con 0x55bd13ab7620 2024-06-04T05:23:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.616+0000 7f66f04c8b80 10 client.22161 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.616+0000 7f66f04c8b80 10 client.22161 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.b 2024-06-04T05:23:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.616+0000 7f66f04c8b80 20 client.22161 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:40.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.616+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/3755497230 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55bd13b3ada0 0x55bd13b3d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.616+0000 7f66f04c8b80 4 client.22161 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:23:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.616+0000 7f66f04c8b80 1 -- 172.21.15.70:0/3755497230 --> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bd13a13be0 con 0x55bd13b3ada0 2024-06-04T05:23:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.616+0000 7f66df7fe700 1 --2- 172.21.15.70:0/3755497230 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55bd13b3ada0 0x55bd13b3d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.616+0000 7f66beffd700 10 client.22161 ms_handle_connect on v2:172.21.15.98:6838/1361276220 2024-06-04T05:23:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.616+0000 7f66df7fe700 1 --2- 172.21.15.70:0/3755497230 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55bd13b3ada0 0x55bd13b3d190 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1fc4654b2e9b4c90 server_cookie=454c513795a02e39 in_seq=0 out_seq=0 2024-06-04T05:23:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.616+0000 7f66beffd700 1 -- 172.21.15.70:0/3755497230 <== mds.2 v2:172.21.15.98:6838/1361276220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bd13a13be0 con 0x55bd13b3ada0 2024-06-04T05:23:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.616+0000 7f66beffd700 10 client.22161 handle_command_reply: tid=0 2024-06-04T05:23:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.616+0000 7f66f04c8b80 1 client.22161 shutdown 2024-06-04T05:23:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66f04c8b80 1 -- 172.21.15.70:0/3755497230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab7620 msgr2=0x55bd13b33500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/3755497230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab7620 0x55bd13b33500 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f66d00aa4b0 tx=0x7f66d000ff60 comp rx=0 tx=0).stop 2024-06-04T05:23:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66f04c8b80 1 -- 172.21.15.70:0/3755497230 shutdown_connections 2024-06-04T05:23:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/3755497230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd13abb370 0x55bd13ac5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/3755497230 >> [v2:172.21.15.98:6838/1361276220,v1:172.21.15.98:6839/1361276220] conn(0x55bd13b3ada0 0x55bd13b3d190 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/3755497230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bd13ab6c70 0x55bd13b32fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66f04c8b80 1 --2- 172.21.15.70:0/3755497230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd13ab7620 0x55bd13b33500 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66beffd700 0 client.22161 ms_handle_reset on v2:172.21.15.98:6838/1361276220 2024-06-04T05:23:40.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66f04c8b80 1 -- 172.21.15.70:0/3755497230 >> 172.21.15.70:0/3755497230 conn(0x55bd139e0080 msgr2=0x55bd13ab1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:40.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66f04c8b80 1 -- 172.21.15.70:0/3755497230 shutdown_connections 2024-06-04T05:23:40.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66f04c8b80 1 -- 172.21.15.70:0/3755497230 wait complete. 2024-06-04T05:23:40.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66f04c8b80 20 client.22161 trim_cache size 0 max 16384 2024-06-04T05:23:40.624 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:23:40.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66e5c33700 1 -- 172.21.15.70:0/2204952978 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f66cc0b29f0 msgr2=0x7f66cc0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:40.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66e5c33700 1 --2- 172.21.15.70:0/2204952978 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f66cc0b29f0 0x7f66cc0b4eb0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f66c8005950 tx=0x7f66c800b410 comp rx=0 tx=0).stop 2024-06-04T05:23:40.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66e5c33700 1 -- 172.21.15.70:0/2204952978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66e0102df0 msgr2=0x7f66e019aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:40.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66e5c33700 1 --2- 172.21.15.70:0/2204952978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66e0102df0 0x7f66e019aeb0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f66d0000c00 tx=0x7f66d00047b0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66e5c33700 1 -- 172.21.15.70:0/2204952978 shutdown_connections 2024-06-04T05:23:40.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66e5c33700 1 --2- 172.21.15.70:0/2204952978 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f66cc0b29f0 0x7f66cc0b4eb0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66e5c33700 1 --2- 172.21.15.70:0/2204952978 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e0108510 0x7f66e01a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66e5c33700 1 --2- 172.21.15.70:0/2204952978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66e0102430 0x7f66e019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66e5c33700 1 --2- 172.21.15.70:0/2204952978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66e0102df0 0x7f66e019aeb0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:40.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.617+0000 7f66e5c33700 1 -- 172.21.15.70:0/2204952978 >> 172.21.15.70:0/2204952978 conn(0x7f66e00793b0 msgr2=0x7f66e0100580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:40.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.618+0000 7f66e5c33700 1 -- 172.21.15.70:0/2204952978 shutdown_connections 2024-06-04T05:23:40.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:40.618+0000 7f66e5c33700 1 -- 172.21.15.70:0/2204952978 wait complete. 2024-06-04T05:23:41.433 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-04T05:23:41.673 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:41.795 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:41 smithi070 ceph-mon[32631]: pgmap v836: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 KiB/s wr, 13 op/s 2024-06-04T05:23:41.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:41 smithi112 ceph-mon[36453]: pgmap v836: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 KiB/s wr, 13 op/s 2024-06-04T05:23:41.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:41 smithi098 ceph-mon[35569]: pgmap v836: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 KiB/s wr, 13 op/s 2024-06-04T05:23:43.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.529+0000 7fb40a183700 1 -- 172.21.15.70:0/1748787416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb404103600 msgr2=0x7fb404103a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.529+0000 7fb40a183700 1 --2- 172.21.15.70:0/1748787416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb404103600 0x7fb404103a80 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80099c0 tx=0x7fb3f8009cd0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.529+0000 7fb40a183700 1 -- 172.21.15.70:0/1748787416 shutdown_connections 2024-06-04T05:23:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.529+0000 7fb40a183700 1 --2- 172.21.15.70:0/1748787416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb404103fc0 0x7fb4041103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.529+0000 7fb40a183700 1 --2- 172.21.15.70:0/1748787416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4041095e0 0x7fb4041099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.529+0000 7fb40a183700 1 --2- 172.21.15.70:0/1748787416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb404103600 0x7fb404103a80 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.529+0000 7fb40a183700 1 -- 172.21.15.70:0/1748787416 >> 172.21.15.70:0/1748787416 conn(0x7fb40407a700 msgr2=0x7fb40407ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.529+0000 7fb40a183700 1 -- 172.21.15.70:0/1748787416 shutdown_connections 2024-06-04T05:23:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.529+0000 7fb40a183700 1 -- 172.21.15.70:0/1748787416 wait complete. 2024-06-04T05:23:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.529+0000 7fb40a183700 1 Processor -- start 2024-06-04T05:23:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.530+0000 7fb40a183700 1 -- start start 2024-06-04T05:23:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.530+0000 7fb40a183700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb404103600 0x7fb404075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.530+0000 7fb40a183700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb404103fc0 0x7fb404075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.530+0000 7fb40a183700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4041095e0 0x7fb4040762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.530+0000 7fb40a183700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb4041135f0 con 0x7fb4041095e0 2024-06-04T05:23:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.530+0000 7fb40a183700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb404113470 con 0x7fb404103600 2024-06-04T05:23:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.530+0000 7fb40a183700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb404113770 con 0x7fb404103fc0 2024-06-04T05:23:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.530+0000 7fb403fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4041095e0 0x7fb4040762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.530+0000 7fb403fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4041095e0 0x7fb4040762c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42168/0 (socket says 172.21.15.70:42168) 2024-06-04T05:23:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.530+0000 7fb403fff700 1 -- 172.21.15.70:0/2875105011 learned_addr learned my addr 172.21.15.70:0/2875105011 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.530+0000 7fb402ffd700 1 --2- 172.21.15.70:0/2875105011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb404103fc0 0x7fb404075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.530+0000 7fb4037fe700 1 --2- 172.21.15.70:0/2875105011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb404103600 0x7fb404075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.530+0000 7fb403fff700 1 -- 172.21.15.70:0/2875105011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb404103fc0 msgr2=0x7fb404075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.530+0000 7fb403fff700 1 --2- 172.21.15.70:0/2875105011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb404103fc0 0x7fb404075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.530+0000 7fb403fff700 1 -- 172.21.15.70:0/2875105011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb404103600 msgr2=0x7fb404075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.530+0000 7fb403fff700 1 --2- 172.21.15.70:0/2875105011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb404103600 0x7fb404075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.530+0000 7fb403fff700 1 -- 172.21.15.70:0/2875105011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4041a7f40 con 0x7fb4041095e0 2024-06-04T05:23:43.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.531+0000 7fb403fff700 1 --2- 172.21.15.70:0/2875105011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4041095e0 0x7fb4040762c0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00c900 tx=0x7fb3fc00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:43.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.531+0000 7fb400ff9700 1 -- 172.21.15.70:0/2875105011 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc0078d0 con 0x7fb4041095e0 2024-06-04T05:23:43.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.531+0000 7fb400ff9700 1 -- 172.21.15.70:0/2875105011 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3fc00f450 con 0x7fb4041095e0 2024-06-04T05:23:43.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.531+0000 7fb40a183700 1 -- 172.21.15.70:0/2875105011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4041a8230 con 0x7fb4041095e0 2024-06-04T05:23:43.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.531+0000 7fb400ff9700 1 -- 172.21.15.70:0/2875105011 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc018660 con 0x7fb4041095e0 2024-06-04T05:23:43.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.531+0000 7fb40a183700 1 -- 172.21.15.70:0/2875105011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4041a8740 con 0x7fb4041095e0 2024-06-04T05:23:43.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.533+0000 7fb400ff9700 1 -- 172.21.15.70:0/2875105011 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fb3fc007a70 con 0x7fb4041095e0 2024-06-04T05:23:43.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.534+0000 7fb400ff9700 1 --2- 172.21.15.70:0/2875105011 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3ec0b2a70 0x7fb3ec0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.534+0000 7fb4037fe700 1 --2- 172.21.15.70:0/2875105011 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3ec0b2a70 0x7fb3ec0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.534+0000 7fb400ff9700 1 -- 172.21.15.70:0/2875105011 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fb3fc0e2450 con 0x7fb4041095e0 2024-06-04T05:23:43.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb4037fe700 1 --2- 172.21.15.70:0/2875105011 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3ec0b2a70 0x7fb3ec0b4f30 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4003f10 tx=0x7fb3f4009180 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:43.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb414a18b80 1 Processor -- start 2024-06-04T05:23:43.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb414a18b80 1 -- start start 2024-06-04T05:23:43.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb414a18b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e23320 0x55d718e23700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb414a18b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e1d8a0 0x55d718e1dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb414a18b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1e260 0x55d718e2a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb414a18b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d718d7bd70 con 0x55d718e1e260 2024-06-04T05:23:43.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb414a18b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d718d6b910 con 0x55d718e23320 2024-06-04T05:23:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb414a18b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d718d6f950 con 0x55d718e1d8a0 2024-06-04T05:23:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb403fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1e260 0x55d718e2a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb403fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1e260 0x55d718e2a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42182/0 (socket says 172.21.15.70:42182) 2024-06-04T05:23:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb403fff700 1 -- 172.21.15.70:0/290608835 learned_addr learned my addr 172.21.15.70:0/290608835 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb402ffd700 1 --2- 172.21.15.70:0/290608835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e1d8a0 0x55d718e1dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb4037fe700 1 --2- 172.21.15.70:0/290608835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e23320 0x55d718e23700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb403fff700 1 -- 172.21.15.70:0/290608835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e1d8a0 msgr2=0x55d718e1dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb403fff700 1 --2- 172.21.15.70:0/290608835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e1d8a0 0x55d718e1dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb403fff700 1 -- 172.21.15.70:0/290608835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e23320 msgr2=0x55d718e23700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb403fff700 1 --2- 172.21.15.70:0/290608835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e23320 0x55d718e23700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.535+0000 7fb403fff700 1 -- 172.21.15.70:0/290608835 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d718d14fb0 con 0x55d718e1e260 2024-06-04T05:23:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.536+0000 7fb403fff700 1 --2- 172.21.15.70:0/290608835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1e260 0x55d718e2a080 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0a9400 tx=0x7fb3fc0cc140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e385711f0d8b809 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.536+0000 7fb3e9ffb700 1 -- 172.21.15.70:0/290608835 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc0adab0 con 0x55d718e1e260 2024-06-04T05:23:43.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.536+0000 7fb3e9ffb700 1 -- 172.21.15.70:0/290608835 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3fc0b2560 con 0x55d718e1e260 2024-06-04T05:23:43.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.536+0000 7fb3e9ffb700 1 -- 172.21.15.70:0/290608835 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc0910b0 con 0x55d718e1e260 2024-06-04T05:23:43.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.536+0000 7fb414a18b80 1 -- 172.21.15.70:0/290608835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1e260 msgr2=0x55d718e2a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:43.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.536+0000 7fb414a18b80 1 --2- 172.21.15.70:0/290608835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1e260 0x55d718e2a080 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0a9400 tx=0x7fb3fc0cc140 comp rx=0 tx=0).stop 2024-06-04T05:23:43.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.537+0000 7fb414a18b80 1 -- 172.21.15.70:0/290608835 shutdown_connections 2024-06-04T05:23:43.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.537+0000 7fb414a18b80 1 --2- 172.21.15.70:0/290608835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1e260 0x55d718e2a080 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.537+0000 7fb414a18b80 1 --2- 172.21.15.70:0/290608835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e23320 0x55d718e23700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.537+0000 7fb414a18b80 1 --2- 172.21.15.70:0/290608835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e1d8a0 0x55d718e1dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.537+0000 7fb414a18b80 1 -- 172.21.15.70:0/290608835 >> 172.21.15.70:0/290608835 conn(0x55d718d49240 msgr2=0x55d718e183e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:43.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.537+0000 7fb414a18b80 1 -- 172.21.15.70:0/290608835 shutdown_connections 2024-06-04T05:23:43.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.537+0000 7fb414a18b80 1 -- 172.21.15.70:0/290608835 wait complete. 2024-06-04T05:23:43.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.537+0000 7fb414a18b80 1 -- start start 2024-06-04T05:23:43.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.537+0000 7fb414a18b80 1 Processor -- start 2024-06-04T05:23:43.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.537+0000 7fb414a18b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e1d8a0 0x55d718e29dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.537+0000 7fb414a18b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1e260 0x55d718e270c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.537+0000 7fb414a18b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e23320 0x55d718e27600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.537+0000 7fb414a18b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d718c71fa0 con 0x55d718e1e260 2024-06-04T05:23:43.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.537+0000 7fb414a18b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d718cd3da0 con 0x55d718e23320 2024-06-04T05:23:43.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.537+0000 7fb414a18b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d718e2a6e0 con 0x55d718e1d8a0 2024-06-04T05:23:43.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.538+0000 7fb414a18b80 10 client.0 _ll_register_callbacks cb 0x55d718d06950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:43.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.538+0000 7fb402ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1e260 0x55d718e270c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.538+0000 7fb402ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1e260 0x55d718e270c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42192/0 (socket says 172.21.15.70:42192) 2024-06-04T05:23:43.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.538+0000 7fb402ffd700 1 -- 172.21.15.70:0/1341901625 learned_addr learned my addr 172.21.15.70:0/1341901625 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:43.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.538+0000 7fb4037fe700 1 --2- 172.21.15.70:0/1341901625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e1d8a0 0x55d718e29dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.538+0000 7fb403fff700 1 --2- 172.21.15.70:0/1341901625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e23320 0x55d718e27600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.538+0000 7fb402ffd700 1 -- 172.21.15.70:0/1341901625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e1d8a0 msgr2=0x55d718e29dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:43.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.538+0000 7fb402ffd700 1 --2- 172.21.15.70:0/1341901625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e1d8a0 0x55d718e29dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.538+0000 7fb402ffd700 1 -- 172.21.15.70:0/1341901625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e23320 msgr2=0x55d718e27600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:43.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.538+0000 7fb402ffd700 1 --2- 172.21.15.70:0/1341901625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e23320 0x55d718e27600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.538+0000 7fb402ffd700 1 -- 172.21.15.70:0/1341901625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d718cbb8d0 con 0x55d718e1e260 2024-06-04T05:23:43.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.538+0000 7fb402ffd700 1 --2- 172.21.15.70:0/1341901625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1e260 0x55d718e270c0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80099c0 tx=0x7fb3f80057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3665ce5dd23a3c2 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:43.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.538+0000 7fb3e2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:43.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.538+0000 7fb3e2ffd700 1 -- 172.21.15.70:0/1341901625 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8029060 con 0x55d718e1e260 2024-06-04T05:23:43.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.538+0000 7fb3e2ffd700 1 -- 172.21.15.70:0/1341901625 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fb3f800bba0 con 0x55d718e1e260 2024-06-04T05:23:43.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.538+0000 7fb414a18b80 1 -- 172.21.15.70:0/1341901625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3fc0b2560 con 0x55d718e1e260 2024-06-04T05:23:43.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.539+0000 7fb3e2ffd700 1 -- 172.21.15.70:0/1341901625 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fb3f800bda0 con 0x55d718e1e260 2024-06-04T05:23:43.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.539+0000 7fb414a18b80 10 client.22173 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:43.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.539+0000 7fb414a18b80 1 -- 172.21.15.70:0/1341901625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x55d718d25760 con 0x55d718e1e260 2024-06-04T05:23:43.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.539+0000 7fb3e2ffd700 1 -- 172.21.15.70:0/1341901625 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3fc0b2560 con 0x55d718e1e260 2024-06-04T05:23:43.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.539+0000 7fb3e2ffd700 1 -- 172.21.15.70:0/1341901625 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7fb3f800f6f0 con 0x55d718e1e260 2024-06-04T05:23:43.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.539+0000 7fb414a18b80 10 client.22173 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:43.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.539+0000 7fb414a18b80 10 client.22173 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:23:43.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.539+0000 7fb414a18b80 20 client.22173 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:43.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.539+0000 7fb414a18b80 1 --2- 172.21.15.70:0/1341901625 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55d718e37310 0x55d718e39700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.539+0000 7fb414a18b80 4 client.22173 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:23:43.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.539+0000 7fb414a18b80 1 -- 172.21.15.70:0/1341901625 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d718d7dcc0 con 0x55d718e37310 2024-06-04T05:23:43.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.539+0000 7fb4037fe700 1 --2- 172.21.15.70:0/1341901625 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55d718e37310 0x55d718e39700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.540+0000 7fb3e2ffd700 10 client.22173 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:23:43.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.540+0000 7fb4037fe700 1 --2- 172.21.15.70:0/1341901625 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55d718e37310 0x55d718e39700 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4d23a5940a09ee81 server_cookie=1d0d867451b6daa1 in_seq=0 out_seq=0 2024-06-04T05:23:43.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.541+0000 7fb3e2ffd700 1 -- 172.21.15.70:0/1341901625 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d718d7dcc0 con 0x55d718e37310 2024-06-04T05:23:43.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.541+0000 7fb3e2ffd700 10 client.22173 handle_command_reply: tid=0 2024-06-04T05:23:43.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.541+0000 7fb414a18b80 1 client.22173 shutdown 2024-06-04T05:23:43.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.541+0000 7fb414a18b80 1 -- 172.21.15.70:0/1341901625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1e260 msgr2=0x55d718e270c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:43.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.541+0000 7fb414a18b80 1 --2- 172.21.15.70:0/1341901625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1e260 0x55d718e270c0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80099c0 tx=0x7fb3f80057e0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.541+0000 7fb414a18b80 1 -- 172.21.15.70:0/1341901625 shutdown_connections 2024-06-04T05:23:43.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.541+0000 7fb414a18b80 1 --2- 172.21.15.70:0/1341901625 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55d718e37310 0x55d718e39700 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.541+0000 7fb414a18b80 1 --2- 172.21.15.70:0/1341901625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e23320 0x55d718e27600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.541+0000 7fb414a18b80 1 --2- 172.21.15.70:0/1341901625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e1d8a0 0x55d718e29dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.541+0000 7fb414a18b80 1 --2- 172.21.15.70:0/1341901625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1e260 0x55d718e270c0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.541+0000 7fb3e2ffd700 0 client.22173 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:23:43.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.541+0000 7fb414a18b80 1 -- 172.21.15.70:0/1341901625 >> 172.21.15.70:0/1341901625 conn(0x55d718d49240 msgr2=0x55d718e17af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:43.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.541+0000 7fb414a18b80 1 -- 172.21.15.70:0/1341901625 shutdown_connections 2024-06-04T05:23:43.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.541+0000 7fb414a18b80 1 -- 172.21.15.70:0/1341901625 wait complete. 2024-06-04T05:23:43.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.541+0000 7fb414a18b80 20 client.22173 trim_cache size 0 max 16384 2024-06-04T05:23:43.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.556+0000 7fb414a18b80 1 Processor -- start 2024-06-04T05:23:43.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.556+0000 7fb414a18b80 1 -- start start 2024-06-04T05:23:43.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.556+0000 7fb414a18b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e1d8a0 0x55d718e33740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.556+0000 7fb414a18b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e23320 0x55d718e18270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.556+0000 7fb414a18b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e1cfa0 0x55d718e288f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.556+0000 7fb414a18b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb4041afe90 con 0x55d718e23320 2024-06-04T05:23:43.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.556+0000 7fb414a18b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4041b1b80 con 0x55d718e1d8a0 2024-06-04T05:23:43.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.556+0000 7fb414a18b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb4041b05f0 con 0x55d718e1cfa0 2024-06-04T05:23:43.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.556+0000 7fb402ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e23320 0x55d718e18270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.556+0000 7fb402ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e23320 0x55d718e18270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42202/0 (socket says 172.21.15.70:42202) 2024-06-04T05:23:43.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.556+0000 7fb402ffd700 1 -- 172.21.15.70:0/4201677035 learned_addr learned my addr 172.21.15.70:0/4201677035 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:43.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.557+0000 7fb403fff700 1 --2- 172.21.15.70:0/4201677035 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e1cfa0 0x55d718e288f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.557+0000 7fb4037fe700 1 --2- 172.21.15.70:0/4201677035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e1d8a0 0x55d718e33740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.557+0000 7fb402ffd700 1 -- 172.21.15.70:0/4201677035 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e1cfa0 msgr2=0x55d718e288f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:43.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.557+0000 7fb402ffd700 1 --2- 172.21.15.70:0/4201677035 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e1cfa0 0x55d718e288f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.557+0000 7fb402ffd700 1 -- 172.21.15.70:0/4201677035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e1d8a0 msgr2=0x55d718e33740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:43.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.557+0000 7fb402ffd700 1 --2- 172.21.15.70:0/4201677035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e1d8a0 0x55d718e33740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.557+0000 7fb402ffd700 1 -- 172.21.15.70:0/4201677035 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d718e25630 con 0x55d718e23320 2024-06-04T05:23:43.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.557+0000 7fb4037fe700 1 --2- 172.21.15.70:0/4201677035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e1d8a0 0x55d718e33740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:23:43.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.557+0000 7fb402ffd700 1 --2- 172.21.15.70:0/4201677035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e23320 0x55d718e18270 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80058e0 tx=0x7fb3f8009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9160234b37dbf557 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:43.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.557+0000 7fb3e97fa700 1 -- 172.21.15.70:0/4201677035 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8029040 con 0x55d718e23320 2024-06-04T05:23:43.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.557+0000 7fb3e97fa700 1 -- 172.21.15.70:0/4201677035 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3f800b9a0 con 0x55d718e23320 2024-06-04T05:23:43.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.557+0000 7fb3e97fa700 1 -- 172.21.15.70:0/4201677035 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8003af0 con 0x55d718e23320 2024-06-04T05:23:43.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.558+0000 7fb414a18b80 1 -- 172.21.15.70:0/4201677035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e23320 msgr2=0x55d718e18270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:43.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.558+0000 7fb414a18b80 1 --2- 172.21.15.70:0/4201677035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e23320 0x55d718e18270 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80058e0 tx=0x7fb3f8009cd0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.558+0000 7fb414a18b80 1 -- 172.21.15.70:0/4201677035 shutdown_connections 2024-06-04T05:23:43.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.558+0000 7fb414a18b80 1 --2- 172.21.15.70:0/4201677035 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e1cfa0 0x55d718e288f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.558+0000 7fb414a18b80 1 --2- 172.21.15.70:0/4201677035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e1d8a0 0x55d718e33740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.558+0000 7fb414a18b80 1 --2- 172.21.15.70:0/4201677035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e23320 0x55d718e18270 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.558+0000 7fb414a18b80 1 -- 172.21.15.70:0/4201677035 >> 172.21.15.70:0/4201677035 conn(0x55d718d49240 msgr2=0x55d718e1c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:43.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.558+0000 7fb414a18b80 1 -- 172.21.15.70:0/4201677035 shutdown_connections 2024-06-04T05:23:43.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.558+0000 7fb414a18b80 1 -- 172.21.15.70:0/4201677035 wait complete. 2024-06-04T05:23:43.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.558+0000 7fb414a18b80 1 -- start start 2024-06-04T05:23:43.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb414a18b80 1 Processor -- start 2024-06-04T05:23:43.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb414a18b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1cfa0 0x55d718e2fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb414a18b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e1d8a0 0x55d718e3e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb414a18b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e23320 0x55d718e3ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb414a18b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb4041b1480 con 0x55d718e1cfa0 2024-06-04T05:23:43.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb414a18b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4041b0cf0 con 0x55d718e1d8a0 2024-06-04T05:23:43.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb414a18b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb4041ade50 con 0x55d718e23320 2024-06-04T05:23:43.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb414a18b80 10 client.0 _ll_register_callbacks cb 0x55d718d06950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:43.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb4037fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1cfa0 0x55d718e2fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb4037fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1cfa0 0x55d718e2fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42214/0 (socket says 172.21.15.70:42214) 2024-06-04T05:23:43.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb4037fe700 1 -- 172.21.15.70:0/2237457627 learned_addr learned my addr 172.21.15.70:0/2237457627 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:43.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb403fff700 1 --2- 172.21.15.70:0/2237457627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e23320 0x55d718e3ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb402ffd700 1 --2- 172.21.15.70:0/2237457627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e1d8a0 0x55d718e3e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb4037fe700 1 -- 172.21.15.70:0/2237457627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e23320 msgr2=0x55d718e3ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:43.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb4037fe700 1 --2- 172.21.15.70:0/2237457627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e23320 0x55d718e3ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb4037fe700 1 -- 172.21.15.70:0/2237457627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e1d8a0 msgr2=0x55d718e3e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:43.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb4037fe700 1 --2- 172.21.15.70:0/2237457627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e1d8a0 0x55d718e3e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb4037fe700 1 -- 172.21.15.70:0/2237457627 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d718e3f200 con 0x55d718e1cfa0 2024-06-04T05:23:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb403fff700 1 --2- 172.21.15.70:0/2237457627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e23320 0x55d718e3ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb402ffd700 1 --2- 172.21.15.70:0/2237457627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e1d8a0 0x55d718e3e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb4037fe700 1 --2- 172.21.15.70:0/2237457627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1cfa0 0x55d718e2fec0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009890 tx=0x7fb3f4004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94306e859ebc3ee server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.559+0000 7fb3e2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.560+0000 7fb3e2ffd700 1 -- 172.21.15.70:0/2237457627 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f401c040 con 0x55d718e1cfa0 2024-06-04T05:23:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.560+0000 7fb3e2ffd700 1 -- 172.21.15.70:0/2237457627 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fb3f402b270 con 0x55d718e1cfa0 2024-06-04T05:23:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.560+0000 7fb414a18b80 1 -- 172.21.15.70:0/2237457627 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3f800b9a0 con 0x55d718e1cfa0 2024-06-04T05:23:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.560+0000 7fb3e2ffd700 1 -- 172.21.15.70:0/2237457627 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fb3f4039050 con 0x55d718e1cfa0 2024-06-04T05:23:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.560+0000 7fb414a18b80 10 client.22179 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.560+0000 7fb414a18b80 1 -- 172.21.15.70:0/2237457627 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x55d718e2e7b0 con 0x55d718e1cfa0 2024-06-04T05:23:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.560+0000 7fb3e2ffd700 1 -- 172.21.15.70:0/2237457627 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3f800b9a0 con 0x55d718e1cfa0 2024-06-04T05:23:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.560+0000 7fb3e2ffd700 1 -- 172.21.15.70:0/2237457627 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7fb3f4022040 con 0x55d718e1cfa0 2024-06-04T05:23:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.560+0000 7fb414a18b80 10 client.22179 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.560+0000 7fb414a18b80 10 client.22179 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-04T05:23:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.560+0000 7fb414a18b80 20 client.22179 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.560+0000 7fb414a18b80 1 --2- 172.21.15.70:0/2237457627 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55d718ea24e0 0x55d718ea4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.560+0000 7fb414a18b80 4 client.22179 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:23:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.560+0000 7fb414a18b80 1 -- 172.21.15.70:0/2237457627 --> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d718d774a0 con 0x55d718ea24e0 2024-06-04T05:23:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.560+0000 7fb402ffd700 1 --2- 172.21.15.70:0/2237457627 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55d718ea24e0 0x55d718ea4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.561+0000 7fb3e2ffd700 10 client.22179 ms_handle_connect on v2:172.21.15.112:6836/3596492666 2024-06-04T05:23:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.561+0000 7fb402ffd700 1 --2- 172.21.15.70:0/2237457627 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55d718ea24e0 0x55d718ea4900 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ad2ce1c30682ca31 server_cookie=8a646a91e391ddb4 in_seq=0 out_seq=0 2024-06-04T05:23:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.561+0000 7fb3e2ffd700 1 -- 172.21.15.70:0/2237457627 <== mds.1 v2:172.21.15.112:6836/3596492666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d718d774a0 con 0x55d718ea24e0 2024-06-04T05:23:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.561+0000 7fb3e2ffd700 10 client.22179 handle_command_reply: tid=0 2024-06-04T05:23:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.561+0000 7fb414a18b80 1 client.22179 shutdown 2024-06-04T05:23:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.561+0000 7fb414a18b80 1 -- 172.21.15.70:0/2237457627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1cfa0 msgr2=0x55d718e2fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.561+0000 7fb414a18b80 1 --2- 172.21.15.70:0/2237457627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1cfa0 0x55d718e2fec0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009890 tx=0x7fb3f4004b40 comp rx=0 tx=0).stop 2024-06-04T05:23:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.561+0000 7fb414a18b80 1 -- 172.21.15.70:0/2237457627 shutdown_connections 2024-06-04T05:23:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.561+0000 7fb414a18b80 1 --2- 172.21.15.70:0/2237457627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d718e23320 0x55d718e3ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.561+0000 7fb414a18b80 1 --2- 172.21.15.70:0/2237457627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d718e1cfa0 0x55d718e2fec0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.561+0000 7fb414a18b80 1 --2- 172.21.15.70:0/2237457627 >> [v2:172.21.15.112:6836/3596492666,v1:172.21.15.112:6837/3596492666] conn(0x55d718ea24e0 0x55d718ea4900 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.561+0000 7fb414a18b80 1 --2- 172.21.15.70:0/2237457627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d718e1d8a0 0x55d718e3e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.561+0000 7fb3e2ffd700 0 client.22179 ms_handle_reset on v2:172.21.15.112:6836/3596492666 2024-06-04T05:23:43.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.561+0000 7fb414a18b80 1 -- 172.21.15.70:0/2237457627 >> 172.21.15.70:0/2237457627 conn(0x55d718d49240 msgr2=0x55d718e20ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:43.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.561+0000 7fb414a18b80 1 -- 172.21.15.70:0/2237457627 shutdown_connections 2024-06-04T05:23:43.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.561+0000 7fb414a18b80 1 -- 172.21.15.70:0/2237457627 wait complete. 2024-06-04T05:23:43.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.561+0000 7fb414a18b80 20 client.22179 trim_cache size 0 max 16384 2024-06-04T05:23:43.570 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:23:43.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.562+0000 7fb40a183700 1 -- 172.21.15.70:0/2875105011 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3ec0b2a70 msgr2=0x7fb3ec0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:43.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.562+0000 7fb40a183700 1 --2- 172.21.15.70:0/2875105011 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3ec0b2a70 0x7fb3ec0b4f30 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4003f10 tx=0x7fb3f4009180 comp rx=0 tx=0).stop 2024-06-04T05:23:43.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.562+0000 7fb40a183700 1 -- 172.21.15.70:0/2875105011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4041095e0 msgr2=0x7fb4040762c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:43.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.562+0000 7fb40a183700 1 --2- 172.21.15.70:0/2875105011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4041095e0 0x7fb4040762c0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00c900 tx=0x7fb3fc00cc10 comp rx=0 tx=0).stop 2024-06-04T05:23:43.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.562+0000 7fb40a183700 1 -- 172.21.15.70:0/2875105011 shutdown_connections 2024-06-04T05:23:43.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.562+0000 7fb40a183700 1 --2- 172.21.15.70:0/2875105011 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb3ec0b2a70 0x7fb3ec0b4f30 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.562+0000 7fb40a183700 1 --2- 172.21.15.70:0/2875105011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4041095e0 0x7fb4040762c0 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.562+0000 7fb40a183700 1 --2- 172.21.15.70:0/2875105011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb404103600 0x7fb404075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.562+0000 7fb40a183700 1 --2- 172.21.15.70:0/2875105011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb404103fc0 0x7fb404075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:43.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.562+0000 7fb40a183700 1 -- 172.21.15.70:0/2875105011 >> 172.21.15.70:0/2875105011 conn(0x7fb40407a700 msgr2=0x7fb404102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:43.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.562+0000 7fb40a183700 1 -- 172.21.15.70:0/2875105011 shutdown_connections 2024-06-04T05:23:43.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:43.562+0000 7fb40a183700 1 -- 172.21.15.70:0/2875105011 wait complete. 2024-06-04T05:23:43.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:43 smithi070 ceph-mon[32631]: pgmap v837: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.2 KiB/s wr, 18 op/s 2024-06-04T05:23:43.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:43 smithi112 ceph-mon[36453]: pgmap v837: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.2 KiB/s wr, 18 op/s 2024-06-04T05:23:43.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:43 smithi098 ceph-mon[35569]: pgmap v837: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.2 KiB/s wr, 18 op/s 2024-06-04T05:23:44.336 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-04T05:23:44.581 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:45.797 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:45 smithi098 ceph-mon[35569]: pgmap v838: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-06-04T05:23:45.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:45 smithi070 ceph-mon[32631]: pgmap v838: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-06-04T05:23:45.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:45 smithi112 ceph-mon[36453]: pgmap v838: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-06-04T05:23:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.257+0000 7fe5fbed3700 1 -- 172.21.15.70:0/1933669020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f410b760 msgr2=0x7fe5f410bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.257+0000 7fe5fbed3700 1 --2- 172.21.15.70:0/1933669020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f410b760 0x7fe5f410bb40 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009990 tx=0x7fe5f0009ca0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.257+0000 7fe5fbed3700 1 -- 172.21.15.70:0/1933669020 shutdown_connections 2024-06-04T05:23:46.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.257+0000 7fe5fbed3700 1 --2- 172.21.15.70:0/1933669020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f4106120 0x7fe5f4112630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.257+0000 7fe5fbed3700 1 --2- 172.21.15.70:0/1933669020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f410b760 0x7fe5f410bb40 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.257+0000 7fe5fbed3700 1 --2- 172.21.15.70:0/1933669020 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe5f4105760 0x7fe5f4105be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.257+0000 7fe5fbed3700 1 -- 172.21.15.70:0/1933669020 >> 172.21.15.70:0/1933669020 conn(0x7fe5f4101250 msgr2=0x7fe5f4103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:46.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.257+0000 7fe5fbed3700 1 -- 172.21.15.70:0/1933669020 shutdown_connections 2024-06-04T05:23:46.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.257+0000 7fe5fbed3700 1 -- 172.21.15.70:0/1933669020 wait complete. 2024-06-04T05:23:46.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5fbed3700 1 Processor -- start 2024-06-04T05:23:46.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5fbed3700 1 -- start start 2024-06-04T05:23:46.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5fbed3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f4105760 0x7fe5f419bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5fbed3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f4106120 0x7fe5f419c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5fbed3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe5f410b760 0x7fe5f41a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5fbed3700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f41157f0 con 0x7fe5f4105760 2024-06-04T05:23:46.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5fbed3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f4115670 con 0x7fe5f4106120 2024-06-04T05:23:46.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5fbed3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f4115970 con 0x7fe5f410b760 2024-06-04T05:23:46.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5f9c6f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f4105760 0x7fe5f419bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:46.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5f9c6f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f4105760 0x7fe5f419bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55332/0 (socket says 172.21.15.70:55332) 2024-06-04T05:23:46.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5f9c6f700 1 -- 172.21.15.70:0/3358654989 learned_addr learned my addr 172.21.15.70:0/3358654989 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:46.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5f946e700 1 --2- 172.21.15.70:0/3358654989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f4106120 0x7fe5f419c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:46.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5fa470700 1 --2- 172.21.15.70:0/3358654989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe5f410b760 0x7fe5f41a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:46.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5f9c6f700 1 -- 172.21.15.70:0/3358654989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe5f410b760 msgr2=0x7fe5f41a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:46.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5f9c6f700 1 --2- 172.21.15.70:0/3358654989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe5f410b760 0x7fe5f41a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5f9c6f700 1 -- 172.21.15.70:0/3358654989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f4106120 msgr2=0x7fe5f419c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:46.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5f9c6f700 1 --2- 172.21.15.70:0/3358654989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f4106120 0x7fe5f419c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.258+0000 7fe5f9c6f700 1 -- 172.21.15.70:0/3358654989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f41a0d30 con 0x7fe5f4105760 2024-06-04T05:23:46.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.259+0000 7fe5f9c6f700 1 --2- 172.21.15.70:0/3358654989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f4105760 0x7fe5f419bd20 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00057c0 tx=0x7fe5f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:46.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.259+0000 7fe5eaffd700 1 -- 172.21.15.70:0/3358654989 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0027070 con 0x7fe5f4105760 2024-06-04T05:23:46.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.259+0000 7fe5eaffd700 1 -- 172.21.15.70:0/3358654989 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5f000fc00 con 0x7fe5f4105760 2024-06-04T05:23:46.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.259+0000 7fe5fbed3700 1 -- 172.21.15.70:0/3358654989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5f41a0fc0 con 0x7fe5f4105760 2024-06-04T05:23:46.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.259+0000 7fe5eaffd700 1 -- 172.21.15.70:0/3358654989 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0022b30 con 0x7fe5f4105760 2024-06-04T05:23:46.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.259+0000 7fe5fbed3700 1 -- 172.21.15.70:0/3358654989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5f41a88a0 con 0x7fe5f4105760 2024-06-04T05:23:46.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.261+0000 7fe5eaffd700 1 -- 172.21.15.70:0/3358654989 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fe5f002a070 con 0x7fe5f4105760 2024-06-04T05:23:46.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.262+0000 7fe5eaffd700 1 --2- 172.21.15.70:0/3358654989 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5e00b29f0 0x7fe5e00b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.262+0000 7fe5f946e700 1 --2- 172.21.15.70:0/3358654989 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5e00b29f0 0x7fe5e00b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:46.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.263+0000 7fe5eaffd700 1 -- 172.21.15.70:0/3358654989 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fe5f00e33c0 con 0x7fe5f4105760 2024-06-04T05:23:46.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.263+0000 7fe606768b80 1 Processor -- start 2024-06-04T05:23:46.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.263+0000 7fe5f946e700 1 --2- 172.21.15.70:0/3358654989 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5e00b29f0 0x7fe5e00b4eb0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe5f41039c0 tx=0x7fe5e40073d0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:46.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.263+0000 7fe606768b80 1 -- start start 2024-06-04T05:23:46.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.263+0000 7fe606768b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be5320 0x559951be5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.263+0000 7fe606768b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951bdf8a0 0x559951bdfd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.263+0000 7fe606768b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951be0260 0x559951bec080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.263+0000 7fe606768b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559951b3dd70 con 0x559951be0260 2024-06-04T05:23:46.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.263+0000 7fe606768b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559951b2d910 con 0x559951bdf8a0 2024-06-04T05:23:46.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.263+0000 7fe606768b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559951b31950 con 0x559951be5320 2024-06-04T05:23:46.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.263+0000 7fe5fa470700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951be0260 0x559951bec080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:46.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.263+0000 7fe5fa470700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951be0260 0x559951bec080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55334/0 (socket says 172.21.15.70:55334) 2024-06-04T05:23:46.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.263+0000 7fe5fa470700 1 -- 172.21.15.70:0/1449491796 learned_addr learned my addr 172.21.15.70:0/1449491796 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:46.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.264+0000 7fe5f9c6f700 1 --2- 172.21.15.70:0/1449491796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be5320 0x559951be5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:46.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.264+0000 7fe5f946e700 1 --2- 172.21.15.70:0/1449491796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951bdf8a0 0x559951bdfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:46.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.264+0000 7fe5fa470700 1 -- 172.21.15.70:0/1449491796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be5320 msgr2=0x559951be5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:46.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.264+0000 7fe5fa470700 1 --2- 172.21.15.70:0/1449491796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be5320 0x559951be5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.264+0000 7fe5fa470700 1 -- 172.21.15.70:0/1449491796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951bdf8a0 msgr2=0x559951bdfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:46.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.264+0000 7fe5fa470700 1 --2- 172.21.15.70:0/1449491796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951bdf8a0 0x559951bdfd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.264+0000 7fe5fa470700 1 -- 172.21.15.70:0/1449491796 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559951ad6fb0 con 0x559951be0260 2024-06-04T05:23:46.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.264+0000 7fe5f946e700 1 --2- 172.21.15.70:0/1449491796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951bdf8a0 0x559951bdfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:46.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.264+0000 7fe5fa470700 1 --2- 172.21.15.70:0/1449491796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951be0260 0x559951bec080 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009d20 tx=0x7fe5ec0077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8fe70850d27edb server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:46.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.264+0000 7fe5dffff700 1 -- 172.21.15.70:0/1449491796 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec004bb0 con 0x559951be0260 2024-06-04T05:23:46.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.264+0000 7fe5dffff700 1 -- 172.21.15.70:0/1449491796 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5ec004d50 con 0x559951be0260 2024-06-04T05:23:46.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.264+0000 7fe5dffff700 1 -- 172.21.15.70:0/1449491796 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec014670 con 0x559951be0260 2024-06-04T05:23:46.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.265+0000 7fe606768b80 1 -- 172.21.15.70:0/1449491796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951be0260 msgr2=0x559951bec080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:46.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.265+0000 7fe606768b80 1 --2- 172.21.15.70:0/1449491796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951be0260 0x559951bec080 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009d20 tx=0x7fe5ec0077e0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.265+0000 7fe606768b80 1 -- 172.21.15.70:0/1449491796 shutdown_connections 2024-06-04T05:23:46.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.265+0000 7fe606768b80 1 --2- 172.21.15.70:0/1449491796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951be0260 0x559951bec080 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.265+0000 7fe606768b80 1 --2- 172.21.15.70:0/1449491796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be5320 0x559951be5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.265+0000 7fe606768b80 1 --2- 172.21.15.70:0/1449491796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951bdf8a0 0x559951bdfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.265+0000 7fe606768b80 1 -- 172.21.15.70:0/1449491796 >> 172.21.15.70:0/1449491796 conn(0x559951b0b240 msgr2=0x559951bda3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:46.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.265+0000 7fe606768b80 1 -- 172.21.15.70:0/1449491796 shutdown_connections 2024-06-04T05:23:46.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.265+0000 7fe606768b80 1 -- 172.21.15.70:0/1449491796 wait complete. 2024-06-04T05:23:46.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.265+0000 7fe606768b80 1 -- start start 2024-06-04T05:23:46.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.265+0000 7fe606768b80 1 Processor -- start 2024-06-04T05:23:46.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe606768b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bebce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe606768b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951be0260 0x559951be9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe606768b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be5320 0x559951be9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe606768b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559951a33fa0 con 0x559951bdf8a0 2024-06-04T05:23:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe606768b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559951a95da0 con 0x559951be0260 2024-06-04T05:23:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe606768b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559951bec6e0 con 0x559951be5320 2024-06-04T05:23:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe606768b80 10 client.0 _ll_register_callbacks cb 0x559951ac8950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe5f9c6f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bebce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe5f9c6f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bebce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55342/0 (socket says 172.21.15.70:55342) 2024-06-04T05:23:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe5f9c6f700 1 -- 172.21.15.70:0/24950375 learned_addr learned my addr 172.21.15.70:0/24950375 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe5f946e700 1 --2- 172.21.15.70:0/24950375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951be0260 0x559951be9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe5fa470700 1 --2- 172.21.15.70:0/24950375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be5320 0x559951be9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe5f9c6f700 1 -- 172.21.15.70:0/24950375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be5320 msgr2=0x559951be9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe5f9c6f700 1 --2- 172.21.15.70:0/24950375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be5320 0x559951be9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe5f9c6f700 1 -- 172.21.15.70:0/24950375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951be0260 msgr2=0x559951be9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:46.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe5f9c6f700 1 --2- 172.21.15.70:0/24950375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951be0260 0x559951be9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe5f9c6f700 1 -- 172.21.15.70:0/24950375 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559951a7d8d0 con 0x559951bdf8a0 2024-06-04T05:23:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe5f9c6f700 1 --2- 172.21.15.70:0/24950375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bebce0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00aa3a0 tx=0x7fe5f00aa230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb6bd9bc5d9b336d server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe5dd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe5dd7fa700 1 -- 172.21.15.70:0/24950375 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00aa6a0 con 0x559951bdf8a0 2024-06-04T05:23:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.266+0000 7fe5dd7fa700 1 -- 172.21.15.70:0/24950375 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fe5f00a6c90 con 0x559951bdf8a0 2024-06-04T05:23:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.267+0000 7fe606768b80 1 -- 172.21.15.70:0/24950375 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5ec004d50 con 0x559951bdf8a0 2024-06-04T05:23:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.267+0000 7fe5dd7fa700 1 -- 172.21.15.70:0/24950375 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fe5f00aea50 con 0x559951bdf8a0 2024-06-04T05:23:46.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.267+0000 7fe606768b80 10 client.22191 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:46.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.267+0000 7fe606768b80 1 -- 172.21.15.70:0/24950375 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x559951ae7760 con 0x559951bdf8a0 2024-06-04T05:23:46.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.267+0000 7fe5dd7fa700 1 -- 172.21.15.70:0/24950375 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5ec004d50 con 0x559951bdf8a0 2024-06-04T05:23:46.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.267+0000 7fe5dd7fa700 1 -- 172.21.15.70:0/24950375 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7fe5f00b2fe0 con 0x559951bdf8a0 2024-06-04T05:23:46.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.267+0000 7fe606768b80 10 client.22191 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:46.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.267+0000 7fe606768b80 10 client.22191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:23:46.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.267+0000 7fe606768b80 20 client.22191 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:46.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.267+0000 7fe606768b80 1 --2- 172.21.15.70:0/24950375 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559951bf9240 0x559951bfb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.267+0000 7fe606768b80 4 client.22191 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-04T05:23:46.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.267+0000 7fe606768b80 1 -- 172.21.15.70:0/24950375 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559951b3fcc0 con 0x559951bf9240 2024-06-04T05:23:46.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.267+0000 7fe5f946e700 1 --2- 172.21.15.70:0/24950375 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559951bf9240 0x559951bfb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:46.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.268+0000 7fe5dd7fa700 10 client.22191 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:46.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.268+0000 7fe5f946e700 1 --2- 172.21.15.70:0/24950375 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559951bf9240 0x559951bfb6a0 crc :-1 s=READY pgs=547 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ca0042568622b6b server_cookie=c9d928aff89b3269 in_seq=0 out_seq=0 2024-06-04T05:23:46.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.268+0000 7fe5dd7fa700 1 -- 172.21.15.70:0/24950375 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559951b3fcc0 con 0x559951bf9240 2024-06-04T05:23:46.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.268+0000 7fe5dd7fa700 10 client.22191 handle_command_reply: tid=0 2024-06-04T05:23:46.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.268+0000 7fe606768b80 1 client.22191 shutdown 2024-06-04T05:23:46.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.269+0000 7fe606768b80 1 -- 172.21.15.70:0/24950375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 msgr2=0x559951bebce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:46.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.269+0000 7fe606768b80 1 --2- 172.21.15.70:0/24950375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bebce0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00aa3a0 tx=0x7fe5f00aa230 comp rx=0 tx=0).stop 2024-06-04T05:23:46.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.269+0000 7fe606768b80 1 -- 172.21.15.70:0/24950375 shutdown_connections 2024-06-04T05:23:46.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.269+0000 7fe606768b80 1 --2- 172.21.15.70:0/24950375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be5320 0x559951be9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.269+0000 7fe606768b80 1 --2- 172.21.15.70:0/24950375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bebce0 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.269+0000 7fe606768b80 1 --2- 172.21.15.70:0/24950375 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559951bf9240 0x559951bfb6a0 crc :-1 s=READY pgs=547 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.269+0000 7fe606768b80 1 --2- 172.21.15.70:0/24950375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951be0260 0x559951be9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.269+0000 7fe5dd7fa700 0 client.22191 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:46.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.269+0000 7fe606768b80 1 -- 172.21.15.70:0/24950375 >> 172.21.15.70:0/24950375 conn(0x559951b0b240 msgr2=0x559951bd9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:46.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.269+0000 7fe606768b80 1 -- 172.21.15.70:0/24950375 shutdown_connections 2024-06-04T05:23:46.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.269+0000 7fe606768b80 1 -- 172.21.15.70:0/24950375 wait complete. 2024-06-04T05:23:46.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.269+0000 7fe606768b80 20 client.22191 trim_cache size 0 max 16384 2024-06-04T05:23:46.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe606768b80 1 Processor -- start 2024-06-04T05:23:46.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe606768b80 1 -- start start 2024-06-04T05:23:46.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe606768b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bda270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe606768b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951be0260 0x559951be8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe606768b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be5320 0x559951bde890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe606768b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f41b0010 con 0x559951bdf8a0 2024-06-04T05:23:46.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe606768b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f41b1d00 con 0x559951be0260 2024-06-04T05:23:46.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe606768b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f41b0770 con 0x559951be5320 2024-06-04T05:23:46.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe5f9c6f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bda270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:46.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe5f9c6f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bda270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55358/0 (socket says 172.21.15.70:55358) 2024-06-04T05:23:46.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe5f9c6f700 1 -- 172.21.15.70:0/3403135983 learned_addr learned my addr 172.21.15.70:0/3403135983 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:46.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe5fa470700 1 --2- 172.21.15.70:0/3403135983 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be5320 0x559951bde890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:46.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe5f9c6f700 1 -- 172.21.15.70:0/3403135983 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be5320 msgr2=0x559951bde890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:46.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe5f9c6f700 1 --2- 172.21.15.70:0/3403135983 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be5320 0x559951bde890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe5f9c6f700 1 -- 172.21.15.70:0/3403135983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951be0260 msgr2=0x559951be8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:46.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe5f9c6f700 1 --2- 172.21.15.70:0/3403135983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951be0260 0x559951be8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe5f9c6f700 1 -- 172.21.15.70:0/3403135983 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559951bdc380 con 0x559951bdf8a0 2024-06-04T05:23:46.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.284+0000 7fe5fa470700 1 --2- 172.21.15.70:0/3403135983 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be5320 0x559951bde890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:23:46.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.285+0000 7fe5f9c6f700 1 --2- 172.21.15.70:0/3403135983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bda270 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00a7910 tx=0x7fe5f00096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33182561201c28bf server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:46.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.285+0000 7fe5df7fe700 1 -- 172.21.15.70:0/3403135983 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00b3470 con 0x559951bdf8a0 2024-06-04T05:23:46.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.285+0000 7fe5df7fe700 1 -- 172.21.15.70:0/3403135983 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5f00a6c90 con 0x559951bdf8a0 2024-06-04T05:23:46.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.285+0000 7fe5df7fe700 1 -- 172.21.15.70:0/3403135983 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00a9b60 con 0x559951bdf8a0 2024-06-04T05:23:46.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.285+0000 7fe606768b80 1 -- 172.21.15.70:0/3403135983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 msgr2=0x559951bda270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:46.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.285+0000 7fe606768b80 1 --2- 172.21.15.70:0/3403135983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bda270 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00a7910 tx=0x7fe5f00096c0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.285+0000 7fe606768b80 1 -- 172.21.15.70:0/3403135983 shutdown_connections 2024-06-04T05:23:46.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.285+0000 7fe606768b80 1 --2- 172.21.15.70:0/3403135983 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be5320 0x559951bde890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.285+0000 7fe606768b80 1 --2- 172.21.15.70:0/3403135983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bda270 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.285+0000 7fe606768b80 1 --2- 172.21.15.70:0/3403135983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951be0260 0x559951be8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.285+0000 7fe606768b80 1 -- 172.21.15.70:0/3403135983 >> 172.21.15.70:0/3403135983 conn(0x559951b0b240 msgr2=0x559951be7920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:46.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.286+0000 7fe606768b80 1 -- 172.21.15.70:0/3403135983 shutdown_connections 2024-06-04T05:23:46.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.286+0000 7fe606768b80 1 -- 172.21.15.70:0/3403135983 wait complete. 2024-06-04T05:23:46.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.286+0000 7fe606768b80 1 -- start start 2024-06-04T05:23:46.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.286+0000 7fe606768b80 1 Processor -- start 2024-06-04T05:23:46.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.286+0000 7fe606768b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bf1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.286+0000 7fe606768b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be0260 0x559951bed0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.286+0000 7fe606768b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951be5320 0x559951bed610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.286+0000 7fe606768b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f41b1600 con 0x559951bdf8a0 2024-06-04T05:23:46.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.286+0000 7fe606768b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f41b0e70 con 0x559951be5320 2024-06-04T05:23:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.286+0000 7fe606768b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f41adfd0 con 0x559951be0260 2024-06-04T05:23:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe606768b80 10 client.0 _ll_register_callbacks cb 0x559951ac8950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-04T05:23:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe5f9c6f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bf1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe5f9c6f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bf1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55364/0 (socket says 172.21.15.70:55364) 2024-06-04T05:23:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe5f9c6f700 1 -- 172.21.15.70:0/1329268477 learned_addr learned my addr 172.21.15.70:0/1329268477 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe5f946e700 1 --2- 172.21.15.70:0/1329268477 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be0260 0x559951bed0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe5fa470700 1 --2- 172.21.15.70:0/1329268477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951be5320 0x559951bed610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe5f9c6f700 1 -- 172.21.15.70:0/1329268477 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be0260 msgr2=0x559951bed0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe5f9c6f700 1 --2- 172.21.15.70:0/1329268477 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be0260 0x559951bed0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe5f9c6f700 1 -- 172.21.15.70:0/1329268477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951be5320 msgr2=0x559951bed610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-04T05:23:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe5f9c6f700 1 --2- 172.21.15.70:0/1329268477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951be5320 0x559951bed610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe5f9c6f700 1 -- 172.21.15.70:0/1329268477 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559951bedb80 con 0x559951bdf8a0 2024-06-04T05:23:46.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe5fa470700 1 --2- 172.21.15.70:0/1329268477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951be5320 0x559951bed610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:23:46.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe5f9c6f700 1 --2- 172.21.15.70:0/1329268477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bf1d50 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00aa3a0 tx=0x7fe5f000b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f62b1dfb38e839a server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:46.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe5dd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-06-04T05:23:46.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe5dd7fa700 1 -- 172.21.15.70:0/1329268477 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00b2fe0 con 0x559951bdf8a0 2024-06-04T05:23:46.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe5dd7fa700 1 -- 172.21.15.70:0/1329268477 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fe5f0092150 con 0x559951bdf8a0 2024-06-04T05:23:46.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe606768b80 1 -- 172.21.15.70:0/1329268477 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5f00a6c90 con 0x559951bdf8a0 2024-06-04T05:23:46.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe5dd7fa700 1 -- 172.21.15.70:0/1329268477 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fe5f00cb3e0 con 0x559951bdf8a0 2024-06-04T05:23:46.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe606768b80 10 client.22197 fetch_fsmap learned FSMap version 72 2024-06-04T05:23:46.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.287+0000 7fe606768b80 1 -- 172.21.15.70:0/1329268477 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x559951bedb80 con 0x559951bdf8a0 2024-06-04T05:23:46.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.288+0000 7fe5dd7fa700 1 -- 172.21.15.70:0/1329268477 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5f00a6c90 con 0x559951bdf8a0 2024-06-04T05:23:46.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.288+0000 7fe5dd7fa700 1 -- 172.21.15.70:0/1329268477 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 72) ==== 8394+0+0 (secure 0 0 0) 0x7fe5f00a9900 con 0x559951bdf8a0 2024-06-04T05:23:46.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.288+0000 7fe606768b80 10 client.22197 fetch_fsmap finished waiting for FSMap version 72 2024-06-04T05:23:46.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.288+0000 7fe606768b80 10 client.22197 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-04T05:23:46.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.288+0000 7fe606768b80 20 client.22197 populate_metadata read hostname 'smithi070' 2024-06-04T05:23:46.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.288+0000 7fe606768b80 1 --2- 172.21.15.70:0/1329268477 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559951c63df0 0x559951c66210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:46.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.288+0000 7fe606768b80 4 client.22197 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-04T05:23:46.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.288+0000 7fe606768b80 1 -- 172.21.15.70:0/1329268477 --> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559951b394a0 con 0x559951c63df0 2024-06-04T05:23:46.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.288+0000 7fe5f946e700 1 --2- 172.21.15.70:0/1329268477 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559951c63df0 0x559951c66210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:46.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.288+0000 7fe5dd7fa700 10 client.22197 ms_handle_connect on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:46.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.288+0000 7fe5f946e700 1 --2- 172.21.15.70:0/1329268477 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559951c63df0 0x559951c66210 crc :-1 s=READY pgs=548 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4c6fca714d143e5 server_cookie=d254a0f99ad99cdb in_seq=0 out_seq=0 2024-06-04T05:23:46.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.289+0000 7fe5dd7fa700 1 -- 172.21.15.70:0/1329268477 <== mds.0 v2:172.21.15.112:6838/2996116087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559951b394a0 con 0x559951c63df0 2024-06-04T05:23:46.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.289+0000 7fe5dd7fa700 10 client.22197 handle_command_reply: tid=0 2024-06-04T05:23:46.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.289+0000 7fe606768b80 1 client.22197 shutdown 2024-06-04T05:23:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.289+0000 7fe606768b80 1 -- 172.21.15.70:0/1329268477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 msgr2=0x559951bf1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.289+0000 7fe606768b80 1 --2- 172.21.15.70:0/1329268477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bf1d50 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00aa3a0 tx=0x7fe5f000b540 comp rx=0 tx=0).stop 2024-06-04T05:23:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.289+0000 7fe606768b80 1 -- 172.21.15.70:0/1329268477 shutdown_connections 2024-06-04T05:23:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.289+0000 7fe606768b80 1 --2- 172.21.15.70:0/1329268477 >> [v2:172.21.15.112:6838/2996116087,v1:172.21.15.112:6839/2996116087] conn(0x559951c63df0 0x559951c66210 crc :-1 s=READY pgs=548 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.289+0000 7fe606768b80 1 --2- 172.21.15.70:0/1329268477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559951be5320 0x559951bed610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.289+0000 7fe5dd7fa700 0 client.22197 ms_handle_reset on v2:172.21.15.112:6838/2996116087 2024-06-04T05:23:46.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.289+0000 7fe606768b80 1 --2- 172.21.15.70:0/1329268477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559951bdf8a0 0x559951bf1d50 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.289+0000 7fe606768b80 1 --2- 172.21.15.70:0/1329268477 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559951be0260 0x559951bed0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.289+0000 7fe606768b80 1 -- 172.21.15.70:0/1329268477 >> 172.21.15.70:0/1329268477 conn(0x559951b0b240 msgr2=0x559951be68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:46.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.289+0000 7fe606768b80 1 -- 172.21.15.70:0/1329268477 shutdown_connections 2024-06-04T05:23:46.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.289+0000 7fe606768b80 1 -- 172.21.15.70:0/1329268477 wait complete. 2024-06-04T05:23:46.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.289+0000 7fe606768b80 20 client.22197 trim_cache size 0 max 16384 2024-06-04T05:23:46.302 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-06-04T05:23:46.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.289+0000 7fe5fbed3700 1 -- 172.21.15.70:0/3358654989 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5e00b29f0 msgr2=0x7fe5e00b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.290+0000 7fe5fbed3700 1 --2- 172.21.15.70:0/3358654989 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5e00b29f0 0x7fe5e00b4eb0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe5f41039c0 tx=0x7fe5e40073d0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.290+0000 7fe5fbed3700 1 -- 172.21.15.70:0/3358654989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f4105760 msgr2=0x7fe5f419bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.290+0000 7fe5fbed3700 1 --2- 172.21.15.70:0/3358654989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f4105760 0x7fe5f419bd20 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00057c0 tx=0x7fe5f00057f0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.290+0000 7fe5fbed3700 1 -- 172.21.15.70:0/3358654989 shutdown_connections 2024-06-04T05:23:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.290+0000 7fe5fbed3700 1 --2- 172.21.15.70:0/3358654989 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fe5e00b29f0 0x7fe5e00b4eb0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.290+0000 7fe5fbed3700 1 --2- 172.21.15.70:0/3358654989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe5f410b760 0x7fe5f41a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.290+0000 7fe5fbed3700 1 --2- 172.21.15.70:0/3358654989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5f4105760 0x7fe5f419bd20 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.290+0000 7fe5fbed3700 1 --2- 172.21.15.70:0/3358654989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f4106120 0x7fe5f419c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.290+0000 7fe5fbed3700 1 -- 172.21.15.70:0/3358654989 >> 172.21.15.70:0/3358654989 conn(0x7fe5f4101250 msgr2=0x7fe5f4101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.290+0000 7fe5fbed3700 1 -- 172.21.15.70:0/3358654989 shutdown_connections 2024-06-04T05:23:46.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:46.290+0000 7fe5fbed3700 1 -- 172.21.15.70:0/3358654989 wait complete. 2024-06-04T05:23:47.038 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_7aec00165b4b073a25396fd0279151d2b2fa23fe/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph_7aec00165b4b073a25396fd0279151d2b2fa23fe/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/git.ceph.com_ceph_7aec00165b4b073a25396fd0279151d2b2fa23fe/qa/tasks/thrasher.py", line 47, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-06-04T05:23:47.039 INFO:tasks.fwd_scrub:done joining 2024-06-04T05:23:47.039 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-06-04T05:23:47.050 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-04T05:23:47.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.151+0000 7fb559f40640 1 -- 172.21.15.70:0/2702918774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb554108a40 msgr2=0x7fb554108e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:47.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.151+0000 7fb559f40640 1 --2- 172.21.15.70:0/2702918774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb554108a40 0x7fb554108e20 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fb544009930 tx=0x7fb54402a420 comp rx=0 tx=0).stop 2024-06-04T05:23:47.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.151+0000 7fb559f40640 1 -- 172.21.15.70:0/2702918774 shutdown_connections 2024-06-04T05:23:47.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.151+0000 7fb559f40640 1 --2- 172.21.15.70:0/2702918774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55406b150 0x7fb55406b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.151+0000 7fb559f40640 1 --2- 172.21.15.70:0/2702918774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb55406bb10 0x7fb55410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.151+0000 7fb559f40640 1 --2- 172.21.15.70:0/2702918774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb554108a40 0x7fb554108e20 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.151+0000 7fb559f40640 1 -- 172.21.15.70:0/2702918774 >> 172.21.15.70:0/2702918774 conn(0x7fb554078a20 msgr2=0x7fb554078e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:47.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.151+0000 7fb559f40640 1 -- 172.21.15.70:0/2702918774 shutdown_connections 2024-06-04T05:23:47.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.151+0000 7fb559f40640 1 -- 172.21.15.70:0/2702918774 wait complete. 2024-06-04T05:23:47.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.152+0000 7fb559f40640 1 Processor -- start 2024-06-04T05:23:47.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.152+0000 7fb559f40640 1 -- start start 2024-06-04T05:23:47.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.152+0000 7fb559f40640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55406b150 0x7fb554199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:47.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.152+0000 7fb559f40640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb55406bb10 0x7fb5541997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:47.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.152+0000 7fb559f40640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb554108a40 0x7fb55419db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:47.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.152+0000 7fb559f40640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb554112c20 con 0x7fb55406bb10 2024-06-04T05:23:47.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.152+0000 7fb559f40640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb554112aa0 con 0x7fb55406b150 2024-06-04T05:23:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.152+0000 7fb559f40640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb554112da0 con 0x7fb554108a40 2024-06-04T05:23:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.152+0000 7fb552ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb55406bb10 0x7fb5541997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.152+0000 7fb552ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb55406bb10 0x7fb5541997a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55374/0 (socket says 172.21.15.70:55374) 2024-06-04T05:23:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.152+0000 7fb552ffd640 1 -- 172.21.15.70:0/1534862486 learned_addr learned my addr 172.21.15.70:0/1534862486 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.153+0000 7fb5537fe640 1 --2- 172.21.15.70:0/1534862486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55406b150 0x7fb554199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.153+0000 7fb552ffd640 1 -- 172.21.15.70:0/1534862486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb554108a40 msgr2=0x7fb55419db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.153+0000 7fb553fff640 1 --2- 172.21.15.70:0/1534862486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb554108a40 0x7fb55419db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.153+0000 7fb552ffd640 1 --2- 172.21.15.70:0/1534862486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb554108a40 0x7fb55419db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.153+0000 7fb552ffd640 1 -- 172.21.15.70:0/1534862486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55406b150 msgr2=0x7fb554199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.153+0000 7fb552ffd640 1 --2- 172.21.15.70:0/1534862486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55406b150 0x7fb554199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.153+0000 7fb552ffd640 1 -- 172.21.15.70:0/1534862486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb55419e250 con 0x7fb55406bb10 2024-06-04T05:23:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.153+0000 7fb5537fe640 1 --2- 172.21.15.70:0/1534862486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55406b150 0x7fb554199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:23:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.153+0000 7fb553fff640 1 --2- 172.21.15.70:0/1534862486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb554108a40 0x7fb55419db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:23:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.153+0000 7fb552ffd640 1 --2- 172.21.15.70:0/1534862486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb55406bb10 0x7fb5541997a0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00b4a0 tx=0x7fb53c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:47.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.153+0000 7fb550ff9640 1 -- 172.21.15.70:0/1534862486 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53c014070 con 0x7fb55406bb10 2024-06-04T05:23:47.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.153+0000 7fb550ff9640 1 -- 172.21.15.70:0/1534862486 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb53c010aa0 con 0x7fb55406bb10 2024-06-04T05:23:47.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.153+0000 7fb559f40640 1 -- 172.21.15.70:0/1534862486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb55419e540 con 0x7fb55406bb10 2024-06-04T05:23:47.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.153+0000 7fb550ff9640 1 -- 172.21.15.70:0/1534862486 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53c017b70 con 0x7fb55406bb10 2024-06-04T05:23:47.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.153+0000 7fb559f40640 1 -- 172.21.15.70:0/1534862486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5541a5dd0 con 0x7fb55406bb10 2024-06-04T05:23:47.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.154+0000 7fb559f40640 1 -- 172.21.15.70:0/1534862486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb554069cc0 con 0x7fb55406bb10 2024-06-04T05:23:47.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.155+0000 7fb550ff9640 1 -- 172.21.15.70:0/1534862486 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fb53c017070 con 0x7fb55406bb10 2024-06-04T05:23:47.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.156+0000 7fb550ff9640 1 --2- 172.21.15.70:0/1534862486 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb5400b2960 0x7fb5400b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:47.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.156+0000 7fb5537fe640 1 --2- 172.21.15.70:0/1534862486 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb5400b2960 0x7fb5400b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:47.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.156+0000 7fb550ff9640 1 -- 172.21.15.70:0/1534862486 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fb53c0e1cc0 con 0x7fb55406bb10 2024-06-04T05:23:47.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.156+0000 7fb5537fe640 1 --2- 172.21.15.70:0/1534862486 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb5400b2960 0x7fb5400b4e20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb544005ab0 tx=0x7fb544005fd0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:47.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.160+0000 7fb550ff9640 1 -- 172.21.15.70:0/1534862486 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb53c0a9780 con 0x7fb55406bb10 2024-06-04T05:23:47.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.346+0000 7fb559f40640 1 -- 172.21.15.70:0/1534862486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb55410aa10 con 0x7fb55406bb10 2024-06-04T05:23:47.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.348+0000 7fb550ff9640 1 -- 172.21.15.70:0/1534862486 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) ==== 74+0+23179 (secure 0 0 0) 0x7fb53c0a9080 con 0x7fb55406bb10 2024-06-04T05:23:47.350 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:23:47.350 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":120,"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","created":"2024-06-04T04:45:22.297536+0000","modified":"2024-06-04T05:23:13.066609+0000","last_up_change":"2024-06-04T04:52:34.388708+0000","last_in_change":"2024-06-04T04:52:14.197780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-04T04:48:43.724356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-04T04:53:31.398455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-04T04:53:35.298617+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":119,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ff491ea2-1906-48e4-883e-bc78e778260e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6803","nonce":824119100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6805","nonce":824119100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6809","nonce":824119100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6807","nonce":824119100}]},"public_addr":"172.21.15.70:6803/824119100","cluster_addr":"172.21.15.70:6805/824119100","heartbeat_back_addr":"172.21.15.70:6809/824119100","heartbeat_front_addr":"172.21.15.70:6807/824119100","state":["exists","up"]},{"osd":1,"uuid":"bd5e4806-535b-49b5-ba6b-93228da35a84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6801","nonce":2744598742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6803","nonce":2744598742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6807","nonce":2744598742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6805","nonce":2744598742}]},"public_addr":"172.21.15.98:6801/2744598742","cluster_addr":"172.21.15.98:6803/2744598742","heartbeat_back_addr":"172.21.15.98:6807/2744598742","heartbeat_front_addr":"172.21.15.98:6805/2744598742","state":["exists","up"]},{"osd":2,"uuid":"8456fb55-910a-4a59-b84b-2c0f68346ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6801","nonce":1328098370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6803","nonce":1328098370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6807","nonce":1328098370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6805","nonce":1328098370}]},"public_addr":"172.21.15.112:6801/1328098370","cluster_addr":"172.21.15.112:6803/1328098370","heartbeat_back_addr":"172.21.15.112:6807/1328098370","heartbeat_front_addr":"172.21.15.112:6805/1328098370","state":["exists","up"]},{"osd":3,"uuid":"c8fbc440-bf70-42d3-b01e-777ccfa7900b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6811","nonce":4041566346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6813","nonce":4041566346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6817","nonce":4041566346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6815","nonce":4041566346}]},"public_addr":"172.21.15.70:6811/4041566346","cluster_addr":"172.21.15.70:6813/4041566346","heartbeat_back_addr":"172.21.15.70:6817/4041566346","heartbeat_front_addr":"172.21.15.70:6815/4041566346","state":["exists","up"]},{"osd":4,"uuid":"549c8c70-8be8-46ae-bcb6-0fde32b40cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6809","nonce":3652775814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6811","nonce":3652775814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6815","nonce":3652775814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6813","nonce":3652775814}]},"public_addr":"172.21.15.98:6809/3652775814","cluster_addr":"172.21.15.98:6811/3652775814","heartbeat_back_addr":"172.21.15.98:6815/3652775814","heartbeat_front_addr":"172.21.15.98:6813/3652775814","state":["exists","up"]},{"osd":5,"uuid":"51d86af1-0d6c-4ae0-b090-00414fc1bf7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6809","nonce":1857257899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6811","nonce":1857257899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6815","nonce":1857257899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6813","nonce":1857257899}]},"public_addr":"172.21.15.112:6809/1857257899","cluster_addr":"172.21.15.112:6811/1857257899","heartbeat_back_addr":"172.21.15.112:6815/1857257899","heartbeat_front_addr":"172.21.15.112:6813/1857257899","state":["exists","up"]},{"osd":6,"uuid":"526a8138-bfa8-419e-a5b5-e6b80f0ae004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6819","nonce":1264676496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6821","nonce":1264676496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6825","nonce":1264676496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6823","nonce":1264676496}]},"public_addr":"172.21.15.70:6819/1264676496","cluster_addr":"172.21.15.70:6821/1264676496","heartbeat_back_addr":"172.21.15.70:6825/1264676496","heartbeat_front_addr":"172.21.15.70:6823/1264676496","state":["exists","up"]},{"osd":7,"uuid":"1173f785-6ed1-47ef-9265-2e233ea9ee02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6817","nonce":709593020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6819","nonce":709593020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6823","nonce":709593020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6821","nonce":709593020}]},"public_addr":"172.21.15.98:6817/709593020","cluster_addr":"172.21.15.98:6819/709593020","heartbeat_back_addr":"172.21.15.98:6823/709593020","heartbeat_front_addr":"172.21.15.98:6821/709593020","state":["exists","up"]},{"osd":8,"uuid":"6424cc6e-24a7-4b81-9ce6-39203fecc7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6817","nonce":2250225413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6819","nonce":2250225413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6823","nonce":2250225413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6821","nonce":2250225413}]},"public_addr":"172.21.15.112:6817/2250225413","cluster_addr":"172.21.15.112:6819/2250225413","heartbeat_back_addr":"172.21.15.112:6823/2250225413","heartbeat_front_addr":"172.21.15.112:6821/2250225413","state":["exists","up"]},{"osd":9,"uuid":"8d666db8-8831-4d63-9e3c-a4bf2504149e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6827","nonce":3557688707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6829","nonce":3557688707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6833","nonce":3557688707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6831","nonce":3557688707}]},"public_addr":"172.21.15.70:6827/3557688707","cluster_addr":"172.21.15.70:6829/3557688707","heartbeat_back_addr":"172.21.15.70:6833/3557688707","heartbeat_front_addr":"172.21.15.70:6831/3557688707","state":["exists","up"]},{"osd":10,"uuid":"ecb925d2-0263-4dfc-a2fd-9e27856022d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6825","nonce":1896493718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6827","nonce":1896493718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6831","nonce":1896493718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6829","nonce":1896493718}]},"public_addr":"172.21.15.98:6825/1896493718","cluster_addr":"172.21.15.98:6827/1896493718","heartbeat_back_addr":"172.21.15.98:6831/1896493718","heartbeat_front_addr":"172.21.15.98:6829/1896493718","state":["exists","up"]},{"osd":11,"uuid":"c8e3230c-bc3a-40bd-8dd0-5fef56e4df97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6825","nonce":1400793137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6827","nonce":1400793137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6831","nonce":1400793137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6829","nonce":1400793137}]},"public_addr":"172.21.15.112:6825/1400793137","cluster_addr":"172.21.15.112:6827/1400793137","heartbeat_back_addr":"172.21.15.112:6831/1400793137","heartbeat_front_addr":"172.21.15.112:6829/1400793137","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:47:53.626608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:17.784505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:41.971969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:07.130699+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:31.615496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:56.898607+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:22.739991+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:48.319727+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:13.902063+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:40.389470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:06.142092+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:32.949383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/4291129442":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/3584175673":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/3720789940":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/145533746":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/3442074771":"2024-06-05T04:46:17.636907+0000","172.21.15.70:6801/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/1442095493":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/2453733268":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6801/2752966840":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/1369781447":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6801/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/2752966840":"2024-06-05T04:57:09.937704+0000","172.21.15.70:6800/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/915186456":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/860033006":"2024-06-05T04:45:49.501864+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":33,"length":1},{"begin":46,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-04T05:23:47.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.350+0000 7fb559f40640 1 -- 172.21.15.70:0/1534862486 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb5400b2960 msgr2=0x7fb5400b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:47.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.350+0000 7fb559f40640 1 --2- 172.21.15.70:0/1534862486 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb5400b2960 0x7fb5400b4e20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb544005ab0 tx=0x7fb544005fd0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.350+0000 7fb559f40640 1 -- 172.21.15.70:0/1534862486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb55406bb10 msgr2=0x7fb5541997a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:47.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.350+0000 7fb559f40640 1 --2- 172.21.15.70:0/1534862486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb55406bb10 0x7fb5541997a0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00b4a0 tx=0x7fb53c00b960 comp rx=0 tx=0).stop 2024-06-04T05:23:47.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.350+0000 7fb559f40640 1 -- 172.21.15.70:0/1534862486 shutdown_connections 2024-06-04T05:23:47.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.350+0000 7fb559f40640 1 --2- 172.21.15.70:0/1534862486 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb5400b2960 0x7fb5400b4e20 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.350+0000 7fb559f40640 1 --2- 172.21.15.70:0/1534862486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb554108a40 0x7fb55419db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.350+0000 7fb559f40640 1 --2- 172.21.15.70:0/1534862486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb55406bb10 0x7fb5541997a0 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.350+0000 7fb559f40640 1 --2- 172.21.15.70:0/1534862486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55406b150 0x7fb554199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.350+0000 7fb559f40640 1 -- 172.21.15.70:0/1534862486 >> 172.21.15.70:0/1534862486 conn(0x7fb554078a20 msgr2=0x7fb5540ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:47.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.350+0000 7fb559f40640 1 -- 172.21.15.70:0/1534862486 shutdown_connections 2024-06-04T05:23:47.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.350+0000 7fb559f40640 1 -- 172.21.15.70:0/1534862486 wait complete. 2024-06-04T05:23:47.364 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-04T04:48:43.724356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-04T04:53:31.398455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-04T04:53:35.298617+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 46, 'snap_epoch': 119, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-04T05:23:47.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-04T05:23:47.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.467+0000 7f2ff99c1640 1 -- 172.21.15.70:0/354668557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff4104470 msgr2=0x7f2ff410cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:47.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.467+0000 7f2ff99c1640 1 --2- 172.21.15.70:0/354668557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff4104470 0x7f2ff410cb30 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009900 tx=0x7f2fdc02a480 comp rx=0 tx=0).stop 2024-06-04T05:23:47.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.467+0000 7f2ff99c1640 1 -- 172.21.15.70:0/354668557 shutdown_connections 2024-06-04T05:23:47.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.467+0000 7f2ff99c1640 1 --2- 172.21.15.70:0/354668557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff410d070 0x7f2ff410f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.467+0000 7f2ff99c1640 1 --2- 172.21.15.70:0/354668557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff4104470 0x7f2ff410cb30 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.467+0000 7f2ff99c1640 1 --2- 172.21.15.70:0/354668557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff4103b50 0x7f2ff4103f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.467+0000 7f2ff99c1640 1 -- 172.21.15.70:0/354668557 >> 172.21.15.70:0/354668557 conn(0x7f2ff4078ab0 msgr2=0x7f2ff4078ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:47.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.467+0000 7f2ff99c1640 1 -- 172.21.15.70:0/354668557 shutdown_connections 2024-06-04T05:23:47.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.467+0000 7f2ff99c1640 1 -- 172.21.15.70:0/354668557 wait complete. 2024-06-04T05:23:47.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2ff99c1640 1 Processor -- start 2024-06-04T05:23:47.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2ff99c1640 1 -- start start 2024-06-04T05:23:47.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2ff99c1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff4103b50 0x7f2ff41991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:47.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2ff99c1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff4104470 0x7f2ff4199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:47.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2ff99c1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff410d070 0x7f2ff419da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:47.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2ff99c1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff4112bb0 con 0x7f2ff4104470 2024-06-04T05:23:47.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2ff99c1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff4112a30 con 0x7f2ff410d070 2024-06-04T05:23:47.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2ff99c1640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff4112d30 con 0x7f2ff4103b50 2024-06-04T05:23:47.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2febfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff4104470 0x7f2ff4199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:47.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2febfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff4104470 0x7f2ff4199700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55390/0 (socket says 172.21.15.70:55390) 2024-06-04T05:23:47.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2febfff640 1 -- 172.21.15.70:0/3759815614 learned_addr learned my addr 172.21.15.70:0/3759815614 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:47.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2ff2ffd640 1 --2- 172.21.15.70:0/3759815614 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff4103b50 0x7f2ff41991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:47.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2ff37fe640 1 --2- 172.21.15.70:0/3759815614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff410d070 0x7f2ff419da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:47.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2febfff640 1 -- 172.21.15.70:0/3759815614 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff4103b50 msgr2=0x7f2ff41991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:47.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2febfff640 1 --2- 172.21.15.70:0/3759815614 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff4103b50 0x7f2ff41991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2febfff640 1 -- 172.21.15.70:0/3759815614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff410d070 msgr2=0x7f2ff419da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:47.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2febfff640 1 --2- 172.21.15.70:0/3759815614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff410d070 0x7f2ff419da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.468+0000 7f2febfff640 1 -- 172.21.15.70:0/3759815614 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff419e1b0 con 0x7f2ff4104470 2024-06-04T05:23:47.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.469+0000 7f2febfff640 1 --2- 172.21.15.70:0/3759815614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff4104470 0x7f2ff4199700 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009230 tx=0x7f2fdc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:47.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.469+0000 7f2ff89bf640 1 -- 172.21.15.70:0/3759815614 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc03f030 con 0x7f2ff4104470 2024-06-04T05:23:47.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.469+0000 7f2ff89bf640 1 -- 172.21.15.70:0/3759815614 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2fdc003a80 con 0x7f2ff4104470 2024-06-04T05:23:47.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.469+0000 7f2ff99c1640 1 -- 172.21.15.70:0/3759815614 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ff419e440 con 0x7f2ff4104470 2024-06-04T05:23:47.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.469+0000 7f2ff89bf640 1 -- 172.21.15.70:0/3759815614 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc035070 con 0x7f2ff4104470 2024-06-04T05:23:47.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.469+0000 7f2ff99c1640 1 -- 172.21.15.70:0/3759815614 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ff41a5d30 con 0x7f2ff4104470 2024-06-04T05:23:47.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.470+0000 7f2ff99c1640 1 -- 172.21.15.70:0/3759815614 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ff4068fe0 con 0x7f2ff4104470 2024-06-04T05:23:47.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.471+0000 7f2ff89bf640 1 -- 172.21.15.70:0/3759815614 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f2fdc042030 con 0x7f2ff4104470 2024-06-04T05:23:47.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.472+0000 7f2ff89bf640 1 --2- 172.21.15.70:0/3759815614 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2fe00b2960 0x7f2fe00b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:47.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.472+0000 7f2ff2ffd640 1 --2- 172.21.15.70:0/3759815614 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2fe00b2960 0x7f2fe00b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:47.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.472+0000 7f2ff89bf640 1 -- 172.21.15.70:0/3759815614 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7f2fdc103130 con 0x7f2ff4104470 2024-06-04T05:23:47.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.472+0000 7f2ff2ffd640 1 --2- 172.21.15.70:0/3759815614 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2fe00b2960 0x7f2fe00b4e20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4009890 tx=0x7f2fe4008040 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:47.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.475+0000 7f2ff89bf640 1 -- 172.21.15.70:0/3759815614 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2fdc0cab80 con 0x7f2ff4104470 2024-06-04T05:23:47.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.660+0000 7f2ff99c1640 1 -- 172.21.15.70:0/3759815614 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2ff41081f0 con 0x7f2ff4104470 2024-06-04T05:23:47.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.660+0000 7f2ff89bf640 1 -- 172.21.15.70:0/3759815614 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) ==== 93+0+10 (secure 0 0 0) 0x7f2fdc0ca480 con 0x7f2ff4104470 2024-06-04T05:23:47.662 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-06-04T05:23:47.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.662+0000 7f2ff99c1640 1 -- 172.21.15.70:0/3759815614 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2fe00b2960 msgr2=0x7f2fe00b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:47.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.662+0000 7f2ff99c1640 1 --2- 172.21.15.70:0/3759815614 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2fe00b2960 0x7f2fe00b4e20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4009890 tx=0x7f2fe4008040 comp rx=0 tx=0).stop 2024-06-04T05:23:47.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.662+0000 7f2ff99c1640 1 -- 172.21.15.70:0/3759815614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff4104470 msgr2=0x7f2ff4199700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:47.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.662+0000 7f2ff99c1640 1 --2- 172.21.15.70:0/3759815614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff4104470 0x7f2ff4199700 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009230 tx=0x7f2fdc004270 comp rx=0 tx=0).stop 2024-06-04T05:23:47.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.662+0000 7f2ff99c1640 1 -- 172.21.15.70:0/3759815614 shutdown_connections 2024-06-04T05:23:47.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.662+0000 7f2ff99c1640 1 --2- 172.21.15.70:0/3759815614 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f2fe00b2960 0x7f2fe00b4e20 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.662+0000 7f2ff99c1640 1 --2- 172.21.15.70:0/3759815614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff410d070 0x7f2ff419da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.662+0000 7f2ff99c1640 1 --2- 172.21.15.70:0/3759815614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff4104470 0x7f2ff4199700 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.662+0000 7f2ff99c1640 1 --2- 172.21.15.70:0/3759815614 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff4103b50 0x7f2ff41991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.662+0000 7f2ff99c1640 1 -- 172.21.15.70:0/3759815614 >> 172.21.15.70:0/3759815614 conn(0x7f2ff4078ab0 msgr2=0x7f2ff40febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:47.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.662+0000 7f2ff99c1640 1 -- 172.21.15.70:0/3759815614 shutdown_connections 2024-06-04T05:23:47.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.662+0000 7f2ff99c1640 1 -- 172.21.15.70:0/3759815614 wait complete. 2024-06-04T05:23:47.674 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:47 smithi070 ceph-mon[32631]: pgmap v839: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-06-04T05:23:47.674 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:47 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1534862486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T05:23:47.675 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-04T05:23:47.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.777+0000 7fbcf2f03640 1 -- 172.21.15.70:0/3943783648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec1079a0 msgr2=0x7fbcec107d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:47.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.777+0000 7fbcf2f03640 1 --2- 172.21.15.70:0/3943783648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec1079a0 0x7fbcec107d80 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009900 tx=0x7fbce402a410 comp rx=0 tx=0).stop 2024-06-04T05:23:47.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.777+0000 7fbcf2f03640 1 -- 172.21.15.70:0/3943783648 shutdown_connections 2024-06-04T05:23:47.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.777+0000 7fbcf2f03640 1 --2- 172.21.15.70:0/3943783648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcec101ca0 0x7fbcec102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.777+0000 7fbcf2f03640 1 --2- 172.21.15.70:0/3943783648 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbcec102640 0x7fbcec10e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.777+0000 7fbcf2f03640 1 --2- 172.21.15.70:0/3943783648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec1079a0 0x7fbcec107d80 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.777+0000 7fbcf2f03640 1 -- 172.21.15.70:0/3943783648 >> 172.21.15.70:0/3943783648 conn(0x7fbcec077750 msgr2=0x7fbcec077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:47.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.777+0000 7fbcf2f03640 1 -- 172.21.15.70:0/3943783648 shutdown_connections 2024-06-04T05:23:47.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.777+0000 7fbcf2f03640 1 -- 172.21.15.70:0/3943783648 wait complete. 2024-06-04T05:23:47.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.777+0000 7fbcf2f03640 1 Processor -- start 2024-06-04T05:23:47.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcf2f03640 1 -- start start 2024-06-04T05:23:47.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcf2f03640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec101ca0 0x7fbcec1990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:47.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcf2f03640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcec102640 0x7fbcec1995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:47.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcf2f03640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbcec1079a0 0x7fbcec19d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:47.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcf2f03640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbcec111a50 con 0x7fbcec101ca0 2024-06-04T05:23:47.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcf2f03640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbcec1118d0 con 0x7fbcec102640 2024-06-04T05:23:47.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcf2f03640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbcec111bd0 con 0x7fbcec1079a0 2024-06-04T05:23:47.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcf0c9f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec101ca0 0x7fbcec1990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:47.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcf0c9f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec101ca0 0x7fbcec1990b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55420/0 (socket says 172.21.15.70:55420) 2024-06-04T05:23:47.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcf0c9f640 1 -- 172.21.15.70:0/4133601851 learned_addr learned my addr 172.21.15.70:0/4133601851 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:47.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcebfff640 1 --2- 172.21.15.70:0/4133601851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcec102640 0x7fbcec1995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:47.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcf14a0640 1 --2- 172.21.15.70:0/4133601851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbcec1079a0 0x7fbcec19d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:47.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcf0c9f640 1 -- 172.21.15.70:0/4133601851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbcec1079a0 msgr2=0x7fbcec19d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:47.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcf0c9f640 1 --2- 172.21.15.70:0/4133601851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbcec1079a0 0x7fbcec19d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcf0c9f640 1 -- 172.21.15.70:0/4133601851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcec102640 msgr2=0x7fbcec1995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:47.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcf0c9f640 1 --2- 172.21.15.70:0/4133601851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcec102640 0x7fbcec1995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcf0c9f640 1 -- 172.21.15.70:0/4133601851 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcec19e050 con 0x7fbcec101ca0 2024-06-04T05:23:47.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbcf0c9f640 1 --2- 172.21.15.70:0/4133601851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec101ca0 0x7fbcec1990b0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009230 tx=0x7fbce4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:47.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.778+0000 7fbce9ffb640 1 -- 172.21.15.70:0/4133601851 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce403f030 con 0x7fbcec101ca0 2024-06-04T05:23:47.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.779+0000 7fbce9ffb640 1 -- 172.21.15.70:0/4133601851 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbce4003a80 con 0x7fbcec101ca0 2024-06-04T05:23:47.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.779+0000 7fbcf2f03640 1 -- 172.21.15.70:0/4133601851 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcec19e2e0 con 0x7fbcec101ca0 2024-06-04T05:23:47.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.779+0000 7fbce9ffb640 1 -- 172.21.15.70:0/4133601851 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce4035070 con 0x7fbcec101ca0 2024-06-04T05:23:47.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.779+0000 7fbcf2f03640 1 -- 172.21.15.70:0/4133601851 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcec1a5bd0 con 0x7fbcec101ca0 2024-06-04T05:23:47.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.780+0000 7fbcf2f03640 1 -- 172.21.15.70:0/4133601851 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcec103090 con 0x7fbcec101ca0 2024-06-04T05:23:47.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.781+0000 7fbce9ffb640 1 -- 172.21.15.70:0/4133601851 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fbce4042030 con 0x7fbcec101ca0 2024-06-04T05:23:47.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.782+0000 7fbce9ffb640 1 --2- 172.21.15.70:0/4133601851 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbcd80b2960 0x7fbcd80b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:47.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.782+0000 7fbcebfff640 1 --2- 172.21.15.70:0/4133601851 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbcd80b2960 0x7fbcd80b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:47.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.782+0000 7fbce9ffb640 1 -- 172.21.15.70:0/4133601851 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fbce4102c60 con 0x7fbcec101ca0 2024-06-04T05:23:47.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.782+0000 7fbcebfff640 1 --2- 172.21.15.70:0/4133601851 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbcd80b2960 0x7fbcd80b4e20 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fbcec1005a0 tx=0x7fbcdc009680 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:47.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.786+0000 7fbce9ffb640 1 -- 172.21.15.70:0/4133601851 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbce40c9fc0 con 0x7fbcec101ca0 2024-06-04T05:23:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:47 smithi112 ceph-mon[36453]: pgmap v839: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-06-04T05:23:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:47 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1534862486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T05:23:47.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.969+0000 7fbcf2f03640 1 -- 172.21.15.70:0/4133601851 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbcec10ae40 con 0x7fbcec101ca0 2024-06-04T05:23:47.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.969+0000 7fbce9ffb640 1 -- 172.21.15.70:0/4133601851 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) ==== 104+0+11 (secure 0 0 0) 0x7fbce40ce2c0 con 0x7fbcec101ca0 2024-06-04T05:23:47.971 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-06-04T05:23:47.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.971+0000 7fbcf2f03640 1 -- 172.21.15.70:0/4133601851 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbcd80b2960 msgr2=0x7fbcd80b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:47.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.971+0000 7fbcf2f03640 1 --2- 172.21.15.70:0/4133601851 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbcd80b2960 0x7fbcd80b4e20 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fbcec1005a0 tx=0x7fbcdc009680 comp rx=0 tx=0).stop 2024-06-04T05:23:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.971+0000 7fbcf2f03640 1 -- 172.21.15.70:0/4133601851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec101ca0 msgr2=0x7fbcec1990b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.971+0000 7fbcf2f03640 1 --2- 172.21.15.70:0/4133601851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec101ca0 0x7fbcec1990b0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009230 tx=0x7fbce4004270 comp rx=0 tx=0).stop 2024-06-04T05:23:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.971+0000 7fbcf2f03640 1 -- 172.21.15.70:0/4133601851 shutdown_connections 2024-06-04T05:23:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.971+0000 7fbcf2f03640 1 --2- 172.21.15.70:0/4133601851 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fbcd80b2960 0x7fbcd80b4e20 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.971+0000 7fbcf2f03640 1 --2- 172.21.15.70:0/4133601851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcec102640 0x7fbcec1995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.971+0000 7fbcf2f03640 1 --2- 172.21.15.70:0/4133601851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbcec1079a0 0x7fbcec19d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.971+0000 7fbcf2f03640 1 --2- 172.21.15.70:0/4133601851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec101ca0 0x7fbcec1990b0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.971+0000 7fbcf2f03640 1 -- 172.21.15.70:0/4133601851 >> 172.21.15.70:0/4133601851 conn(0x7fbcec077750 msgr2=0x7fbcec06b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.971+0000 7fbcf2f03640 1 -- 172.21.15.70:0/4133601851 shutdown_connections 2024-06-04T05:23:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:47.971+0000 7fbcf2f03640 1 -- 172.21.15.70:0/4133601851 wait complete. 2024-06-04T05:23:47.985 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-04T05:23:47.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:47 smithi098 ceph-mon[35569]: pgmap v839: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-06-04T05:23:47.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:47 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1534862486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T05:23:48.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.086+0000 7fb1dbae8640 1 -- 172.21.15.70:0/331676892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1d4101920 msgr2=0x7fb1d4101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:48.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.086+0000 7fb1dbae8640 1 --2- 172.21.15.70:0/331676892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1d4101920 0x7fb1d4101da0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009930 tx=0x7fb1d002a420 comp rx=0 tx=0).stop 2024-06-04T05:23:48.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.087+0000 7fb1dbae8640 1 -- 172.21.15.70:0/331676892 shutdown_connections 2024-06-04T05:23:48.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.087+0000 7fb1dbae8640 1 --2- 172.21.15.70:0/331676892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1d41022e0 0x7fb1d410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:48.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.087+0000 7fb1dbae8640 1 --2- 172.21.15.70:0/331676892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1d4101920 0x7fb1d4101da0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:48.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.087+0000 7fb1dbae8640 1 --2- 172.21.15.70:0/331676892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1d4107930 0x7fb1d4107d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:48.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.087+0000 7fb1dbae8640 1 -- 172.21.15.70:0/331676892 >> 172.21.15.70:0/331676892 conn(0x7fb1d4078a50 msgr2=0x7fb1d4078e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:48.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.087+0000 7fb1dbae8640 1 -- 172.21.15.70:0/331676892 shutdown_connections 2024-06-04T05:23:48.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.087+0000 7fb1dbae8640 1 -- 172.21.15.70:0/331676892 wait complete. 2024-06-04T05:23:48.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.087+0000 7fb1dbae8640 1 Processor -- start 2024-06-04T05:23:48.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.087+0000 7fb1dbae8640 1 -- start start 2024-06-04T05:23:48.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.087+0000 7fb1dbae8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1d4101920 0x7fb1d4199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:48.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.087+0000 7fb1dbae8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1d41022e0 0x7fb1d4199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:48.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.087+0000 7fb1dbae8640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1d4107930 0x7fb1d419da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:48.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.087+0000 7fb1dbae8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d4111aa0 con 0x7fb1d4101920 2024-06-04T05:23:48.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.087+0000 7fb1dbae8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d4111920 con 0x7fb1d41022e0 2024-06-04T05:23:48.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.087+0000 7fb1dbae8640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d4111c20 con 0x7fb1d4107930 2024-06-04T05:23:48.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.087+0000 7fb1d9884640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1d4101920 0x7fb1d4199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:48.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.088+0000 7fb1d9884640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1d4101920 0x7fb1d4199150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55434/0 (socket says 172.21.15.70:55434) 2024-06-04T05:23:48.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.088+0000 7fb1d9884640 1 -- 172.21.15.70:0/4007188811 learned_addr learned my addr 172.21.15.70:0/4007188811 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:48.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.088+0000 7fb1d9083640 1 --2- 172.21.15.70:0/4007188811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1d41022e0 0x7fb1d4199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:48.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.088+0000 7fb1d9884640 1 -- 172.21.15.70:0/4007188811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1d4107930 msgr2=0x7fb1d419da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:48.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.088+0000 7fb1d9884640 1 --2- 172.21.15.70:0/4007188811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1d4107930 0x7fb1d419da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:48.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.088+0000 7fb1d9884640 1 -- 172.21.15.70:0/4007188811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1d41022e0 msgr2=0x7fb1d4199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:48.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.088+0000 7fb1d9884640 1 --2- 172.21.15.70:0/4007188811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1d41022e0 0x7fb1d4199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:48.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.088+0000 7fb1d9884640 1 -- 172.21.15.70:0/4007188811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d419e140 con 0x7fb1d4101920 2024-06-04T05:23:48.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.088+0000 7fb1d9884640 1 --2- 172.21.15.70:0/4007188811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1d4101920 0x7fb1d4199150 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400c920 tx=0x7fb1c400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:48.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.088+0000 7fb1caffd640 1 -- 172.21.15.70:0/4007188811 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4007c70 con 0x7fb1d4101920 2024-06-04T05:23:48.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.088+0000 7fb1caffd640 1 -- 172.21.15.70:0/4007188811 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb1c4007e10 con 0x7fb1d4101920 2024-06-04T05:23:48.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.088+0000 7fb1dbae8640 1 -- 172.21.15.70:0/4007188811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1d419e430 con 0x7fb1d4101920 2024-06-04T05:23:48.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.088+0000 7fb1caffd640 1 -- 172.21.15.70:0/4007188811 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4018710 con 0x7fb1d4101920 2024-06-04T05:23:48.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.088+0000 7fb1dbae8640 1 -- 172.21.15.70:0/4007188811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1d41a5cc0 con 0x7fb1d4101920 2024-06-04T05:23:48.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.089+0000 7fb1dbae8640 1 -- 172.21.15.70:0/4007188811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1d4069870 con 0x7fb1d4101920 2024-06-04T05:23:48.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.091+0000 7fb1caffd640 1 -- 172.21.15.70:0/4007188811 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fb1c400f450 con 0x7fb1d4101920 2024-06-04T05:23:48.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.092+0000 7fb1caffd640 1 --2- 172.21.15.70:0/4007188811 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1c00b2960 0x7fb1c00b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:48.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.092+0000 7fb1d9083640 1 --2- 172.21.15.70:0/4007188811 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1c00b2960 0x7fb1c00b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:48.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.092+0000 7fb1caffd640 1 -- 172.21.15.70:0/4007188811 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7fb1c40e2bb0 con 0x7fb1d4101920 2024-06-04T05:23:48.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.092+0000 7fb1d9083640 1 --2- 172.21.15.70:0/4007188811 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1c00b2960 0x7fb1c00b4e20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0006010 tx=0x7fb1d0009610 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:48.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.095+0000 7fb1caffd640 1 -- 172.21.15.70:0/4007188811 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb1c4014030 con 0x7fb1d4101920 2024-06-04T05:23:48.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.279+0000 7fb1dbae8640 1 -- 172.21.15.70:0/4007188811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb1d419a4a0 con 0x7fb1d4101920 2024-06-04T05:23:48.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.279+0000 7fb1caffd640 1 -- 172.21.15.70:0/4007188811 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v120) ==== 100+0+11 (secure 0 0 0) 0x7fb1c40aa590 con 0x7fb1d4101920 2024-06-04T05:23:48.281 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-06-04T05:23:48.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.281+0000 7fb1dbae8640 1 -- 172.21.15.70:0/4007188811 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1c00b2960 msgr2=0x7fb1c00b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:48.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.281+0000 7fb1dbae8640 1 --2- 172.21.15.70:0/4007188811 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1c00b2960 0x7fb1c00b4e20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0006010 tx=0x7fb1d0009610 comp rx=0 tx=0).stop 2024-06-04T05:23:48.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.281+0000 7fb1dbae8640 1 -- 172.21.15.70:0/4007188811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1d4101920 msgr2=0x7fb1d4199150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:48.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.281+0000 7fb1dbae8640 1 --2- 172.21.15.70:0/4007188811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1d4101920 0x7fb1d4199150 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400c920 tx=0x7fb1c400cde0 comp rx=0 tx=0).stop 2024-06-04T05:23:48.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.281+0000 7fb1dbae8640 1 -- 172.21.15.70:0/4007188811 shutdown_connections 2024-06-04T05:23:48.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.281+0000 7fb1dbae8640 1 --2- 172.21.15.70:0/4007188811 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fb1c00b2960 0x7fb1c00b4e20 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:48.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.281+0000 7fb1dbae8640 1 --2- 172.21.15.70:0/4007188811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1d41022e0 0x7fb1d4199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:48.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.281+0000 7fb1dbae8640 1 --2- 172.21.15.70:0/4007188811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1d4107930 0x7fb1d419da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:48.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.281+0000 7fb1dbae8640 1 --2- 172.21.15.70:0/4007188811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1d4101920 0x7fb1d4199150 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:48.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.281+0000 7fb1dbae8640 1 -- 172.21.15.70:0/4007188811 >> 172.21.15.70:0/4007188811 conn(0x7fb1d4078a50 msgr2=0x7fb1d4069f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:48.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.281+0000 7fb1dbae8640 1 -- 172.21.15.70:0/4007188811 shutdown_connections 2024-06-04T05:23:48.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.281+0000 7fb1dbae8640 1 -- 172.21.15.70:0/4007188811 wait complete. 2024-06-04T05:23:48.294 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-06-04T05:23:48.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.393+0000 7f346fc75640 1 -- 172.21.15.70:0/2725626637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3468107940 msgr2=0x7f3468107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:48.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.393+0000 7f346fc75640 1 --2- 172.21.15.70:0/2725626637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3468107940 0x7f3468107d20 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f3458009930 tx=0x7f345802a420 comp rx=0 tx=0).stop 2024-06-04T05:23:48.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.393+0000 7f346fc75640 1 -- 172.21.15.70:0/2725626637 shutdown_connections 2024-06-04T05:23:48.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.393+0000 7f346fc75640 1 --2- 172.21.15.70:0/2725626637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34681022f0 0x7f346810e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:48.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.393+0000 7f346fc75640 1 --2- 172.21.15.70:0/2725626637 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3468101930 0x7f3468101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:48.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.393+0000 7f346fc75640 1 --2- 172.21.15.70:0/2725626637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3468107940 0x7f3468107d20 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:48.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.393+0000 7f346fc75640 1 -- 172.21.15.70:0/2725626637 >> 172.21.15.70:0/2725626637 conn(0x7f3468078a50 msgr2=0x7f3468078e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.393+0000 7f346fc75640 1 -- 172.21.15.70:0/2725626637 shutdown_connections 2024-06-04T05:23:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.393+0000 7f346fc75640 1 -- 172.21.15.70:0/2725626637 wait complete. 2024-06-04T05:23:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.393+0000 7f346fc75640 1 Processor -- start 2024-06-04T05:23:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.393+0000 7f346fc75640 1 -- start start 2024-06-04T05:23:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346fc75640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3468101930 0x7f3468199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346fc75640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34681022f0 0x7f34681996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346fc75640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3468107940 0x7f346819da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346fc75640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3468112c00 con 0x7f3468101930 2024-06-04T05:23:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346fc75640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3468112a80 con 0x7f3468107940 2024-06-04T05:23:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346fc75640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3468112d80 con 0x7f34681022f0 2024-06-04T05:23:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346da11640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3468101930 0x7f3468199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346da11640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3468101930 0x7f3468199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55446/0 (socket says 172.21.15.70:55446) 2024-06-04T05:23:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346da11640 1 -- 172.21.15.70:0/4041866420 learned_addr learned my addr 172.21.15.70:0/4041866420 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:23:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346e212640 1 --2- 172.21.15.70:0/4041866420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3468107940 0x7f346819da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346da11640 1 -- 172.21.15.70:0/4041866420 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34681022f0 msgr2=0x7f34681996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346da11640 1 --2- 172.21.15.70:0/4041866420 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34681022f0 0x7f34681996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346da11640 1 -- 172.21.15.70:0/4041866420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3468107940 msgr2=0x7f346819da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346da11640 1 --2- 172.21.15.70:0/4041866420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3468107940 0x7f346819da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346da11640 1 -- 172.21.15.70:0/4041866420 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f346819e170 con 0x7f3468101930 2024-06-04T05:23:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346e212640 1 --2- 172.21.15.70:0/4041866420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3468107940 0x7f346819da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:23:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346da11640 1 --2- 172.21.15.70:0/4041866420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3468101930 0x7f3468199180 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f345802a940 tx=0x7f3458003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f345effd640 1 -- 172.21.15.70:0/4041866420 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345803f070 con 0x7f3468101930 2024-06-04T05:23:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f345effd640 1 -- 172.21.15.70:0/4041866420 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3458003e10 con 0x7f3468101930 2024-06-04T05:23:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346fc75640 1 -- 172.21.15.70:0/4041866420 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f346819e400 con 0x7f3468101930 2024-06-04T05:23:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f345effd640 1 -- 172.21.15.70:0/4041866420 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3458035030 con 0x7f3468101930 2024-06-04T05:23:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.394+0000 7f346fc75640 1 -- 172.21.15.70:0/4041866420 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34681a5cf0 con 0x7f3468101930 2024-06-04T05:23:48.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.395+0000 7f346fc75640 1 -- 172.21.15.70:0/4041866420 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3468069870 con 0x7f3468101930 2024-06-04T05:23:48.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.396+0000 7f345effd640 1 -- 172.21.15.70:0/4041866420 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3458042070 con 0x7f3468101930 2024-06-04T05:23:48.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.398+0000 7f345effd640 1 --2- 172.21.15.70:0/4041866420 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f34540b29e0 0x7f34540b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:23:48.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.398+0000 7f346d210640 1 --2- 172.21.15.70:0/4041866420 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f34540b29e0 0x7f34540b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:23:48.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.398+0000 7f345effd640 1 -- 172.21.15.70:0/4041866420 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8571+0+0 (secure 0 0 0) 0x7f3458103310 con 0x7f3468101930 2024-06-04T05:23:48.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.398+0000 7f346d210640 1 --2- 172.21.15.70:0/4041866420 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f34540b29e0 0x7f34540b4ea0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f3464005fd0 tx=0x7f3464005c70 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:23:48.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.401+0000 7f345effd640 1 -- 172.21.15.70:0/4041866420 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f34580cac60 con 0x7f3468101930 2024-06-04T05:23:48.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.637+0000 7f346fc75640 1 -- 172.21.15.70:0/4041866420 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f3468109ea0 con 0x7f3468101930 2024-06-04T05:23:48.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.646+0000 7f345effd640 1 -- 172.21.15.70:0/4041866420 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+816172 (secure 0 0 0) 0x7f34580ca560 con 0x7f3468101930 2024-06-04T05:23:48.658 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:23:48.659 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:23:48.659 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 25, 2024-06-04T05:23:48.659 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.659 INFO:teuthology.orchestra.run.smithi070.stdout: "active_gid": 24806, 2024-06-04T05:23:48.659 INFO:teuthology.orchestra.run.smithi070.stdout: "active_name": "x", 2024-06-04T05:23:48.659 INFO:teuthology.orchestra.run.smithi070.stdout: "active_addrs": { 2024-06-04T05:23:48.659 INFO:teuthology.orchestra.run.smithi070.stdout: "addrvec": [ 2024-06-04T05:23:48.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.659 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v2", 2024-06-04T05:23:48.659 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "172.21.15.70:6800", 2024-06-04T05:23:48.659 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 542036627 2024-06-04T05:23:48.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.660 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-06-04T05:23:48.660 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "172.21.15.70:6801", 2024-06-04T05:23:48.660 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 542036627 2024-06-04T05:23:48.660 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.660 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-06-04T05:23:48.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.660 INFO:teuthology.orchestra.run.smithi070.stdout: "active_addr": "172.21.15.70:6801/542036627", 2024-06-04T05:23:48.660 INFO:teuthology.orchestra.run.smithi070.stdout: "active_change": "2024-06-04T04:57:09.937932+0000", 2024-06-04T05:23:48.660 INFO:teuthology.orchestra.run.smithi070.stdout: "active_mgr_features": 4540701547738038271, 2024-06-04T05:23:48.660 INFO:teuthology.orchestra.run.smithi070.stdout: "available": true, 2024-06-04T05:23:48.660 INFO:teuthology.orchestra.run.smithi070.stdout: "standbys": [ 2024-06-04T05:23:48.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.660 INFO:teuthology.orchestra.run.smithi070.stdout: "gid": 15213, 2024-06-04T05:23:48.661 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "z", 2024-06-04T05:23:48.661 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_features": 4540701547738038271, 2024-06-04T05:23:48.661 INFO:teuthology.orchestra.run.smithi070.stdout: "available_modules": [ 2024-06-04T05:23:48.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.661 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "alerts", 2024-06-04T05:23:48.661 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.661 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.661 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.661 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-06-04T05:23:48.661 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-06-04T05:23:48.661 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.661 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.661 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.661 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-06-04T05:23:48.662 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.662 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.662 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.662 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How frequently to reexamine health status", 2024-06-04T05:23:48.662 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.662 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.662 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.662 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.662 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.662 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.662 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.662 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.665 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.665 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.665 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.665 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.665 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.665 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.665 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.665 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.665 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.665 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.665 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.665 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.665 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.666 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.666 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.666 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.666 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.666 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.666 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.666 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.666 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.666 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.666 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.666 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.666 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.667 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.667 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.667 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.667 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.667 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.667 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.667 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.667 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.667 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.667 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_destination": { 2024-06-04T05:23:48.667 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_destination", 2024-06-04T05:23:48.667 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.668 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.668 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.668 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.668 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.668 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.668 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.668 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Email address to send alerts to", 2024-06-04T05:23:48.668 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.668 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.668 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.668 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_from_name": { 2024-06-04T05:23:48.668 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_from_name", 2024-06-04T05:23:48.668 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "Ceph", 2024-06-04T05:23:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Email From: name", 2024-06-04T05:23:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_host": { 2024-06-04T05:23:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_host", 2024-06-04T05:23:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP server", 2024-06-04T05:23:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_password": { 2024-06-04T05:23:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_password", 2024-06-04T05:23:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Password to authenticate with", 2024-06-04T05:23:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_port": { 2024-06-04T05:23:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_port", 2024-06-04T05:23:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.672 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.672 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.672 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "465", 2024-06-04T05:23:48.672 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.672 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.672 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.672 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP port", 2024-06-04T05:23:48.672 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.672 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.672 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.672 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_sender": { 2024-06-04T05:23:48.672 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_sender", 2024-06-04T05:23:48.673 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.673 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.673 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.673 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.673 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.673 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.673 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.673 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP envelope sender", 2024-06-04T05:23:48.673 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.673 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.673 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.673 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_ssl": { 2024-06-04T05:23:48.674 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_ssl", 2024-06-04T05:23:48.674 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.674 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.674 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.674 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.674 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.674 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.674 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.674 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use SSL to connect to SMTP server", 2024-06-04T05:23:48.674 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.674 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.674 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.675 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_user": { 2024-06-04T05:23:48.675 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_user", 2024-06-04T05:23:48.675 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.675 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.675 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.675 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.675 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.675 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.675 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.675 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "User to authenticate as", 2024-06-04T05:23:48.675 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.675 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.675 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.676 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.676 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.676 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.676 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.676 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.676 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.676 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.676 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.676 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.676 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.676 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.676 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.676 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.677 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.677 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.677 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "balancer", 2024-06-04T05:23:48.677 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.677 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.677 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.677 INFO:teuthology.orchestra.run.smithi070.stdout: "active": { 2024-06-04T05:23:48.677 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "active", 2024-06-04T05:23:48.677 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.677 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.677 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.677 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.678 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.678 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.678 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.678 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "automatically balance PGs across cluster", 2024-06-04T05:23:48.678 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.678 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.678 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.678 INFO:teuthology.orchestra.run.smithi070.stdout: "begin_time": { 2024-06-04T05:23:48.678 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "begin_time", 2024-06-04T05:23:48.678 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.678 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.678 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.679 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0000", 2024-06-04T05:23:48.679 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.679 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.679 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.679 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "beginning time of day to automatically balance", 2024-06-04T05:23:48.679 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-04T05:23:48.679 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.679 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.679 INFO:teuthology.orchestra.run.smithi070.stdout: "begin_weekday": { 2024-06-04T05:23:48.679 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "begin_weekday", 2024-06-04T05:23:48.679 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-06-04T05:23:48.679 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.679 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.680 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.680 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0", 2024-06-04T05:23:48.680 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "6", 2024-06-04T05:23:48.680 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.680 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-06-04T05:23:48.680 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-04T05:23:48.680 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.680 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.680 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_max_iterations": { 2024-06-04T05:23:48.680 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_max_iterations", 2024-06-04T05:23:48.680 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-06-04T05:23:48.680 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.681 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.681 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "25", 2024-06-04T05:23:48.681 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-06-04T05:23:48.681 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "250", 2024-06-04T05:23:48.681 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.681 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-06-04T05:23:48.681 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.681 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.681 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.681 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_metrics": { 2024-06-04T05:23:48.681 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_metrics", 2024-06-04T05:23:48.681 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.681 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.682 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.682 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "pgs,objects,bytes", 2024-06-04T05:23:48.682 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.682 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.682 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.682 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-06-04T05:23:48.682 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-06-04T05:23:48.682 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.682 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.682 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.682 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_step": { 2024-06-04T05:23:48.682 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_step", 2024-06-04T05:23:48.682 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:48.683 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.683 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.683 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0.5", 2024-06-04T05:23:48.683 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0.001", 2024-06-04T05:23:48.683 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "0.999", 2024-06-04T05:23:48.683 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.683 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "aggressiveness of optimization", 2024-06-04T05:23:48.683 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-06-04T05:23:48.683 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.683 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.683 INFO:teuthology.orchestra.run.smithi070.stdout: "end_time": { 2024-06-04T05:23:48.683 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "end_time", 2024-06-04T05:23:48.683 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.683 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.684 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.684 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "2359", 2024-06-04T05:23:48.684 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.684 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.684 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.684 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "ending time of day to automatically balance", 2024-06-04T05:23:48.684 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-04T05:23:48.684 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.684 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.684 INFO:teuthology.orchestra.run.smithi070.stdout: "end_weekday": { 2024-06-04T05:23:48.684 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "end_weekday", 2024-06-04T05:23:48.684 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-06-04T05:23:48.685 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.685 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.685 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.685 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0", 2024-06-04T05:23:48.685 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "6", 2024-06-04T05:23:48.685 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.685 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-06-04T05:23:48.685 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-04T05:23:48.685 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.685 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.685 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.685 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.685 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.686 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.686 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.686 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.686 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.686 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.686 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.686 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.686 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.686 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.686 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.686 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.686 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.686 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.687 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.687 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.687 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.687 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.687 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.687 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.687 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.687 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.687 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.687 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.687 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.687 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.687 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.688 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.688 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.688 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.688 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.688 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.688 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.688 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.688 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.688 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.688 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.688 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.688 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.688 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.689 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.689 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.689 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.689 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.689 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.689 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.689 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.689 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.689 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.689 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.689 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.689 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.689 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.690 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.690 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.690 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.690 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.690 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.690 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.690 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.690 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.690 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.690 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.690 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.690 INFO:teuthology.orchestra.run.smithi070.stdout: "min_score": { 2024-06-04T05:23:48.691 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "min_score", 2024-06-04T05:23:48.691 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:48.691 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.691 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.691 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.691 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.691 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.691 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.691 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-06-04T05:23:48.691 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.691 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.691 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.691 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": { 2024-06-04T05:23:48.692 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mode", 2024-06-04T05:23:48.692 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.692 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.692 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.692 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "upmap", 2024-06-04T05:23:48.692 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.692 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.692 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.692 INFO:teuthology.orchestra.run.smithi070.stdout: "crush-compat", 2024-06-04T05:23:48.692 INFO:teuthology.orchestra.run.smithi070.stdout: "none", 2024-06-04T05:23:48.692 INFO:teuthology.orchestra.run.smithi070.stdout: "read", 2024-06-04T05:23:48.692 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap", 2024-06-04T05:23:48.692 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap-read" 2024-06-04T05:23:48.692 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.693 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Balancer mode", 2024-06-04T05:23:48.693 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.693 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.693 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.693 INFO:teuthology.orchestra.run.smithi070.stdout: "pool_ids": { 2024-06-04T05:23:48.693 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pool_ids", 2024-06-04T05:23:48.693 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.693 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.693 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.693 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.693 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.693 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.693 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.694 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-06-04T05:23:48.694 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.694 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.694 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.694 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-06-04T05:23:48.694 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-06-04T05:23:48.694 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.694 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.694 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.694 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-06-04T05:23:48.694 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.694 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.695 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.695 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-06-04T05:23:48.695 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.695 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.695 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.695 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.695 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.695 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.695 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.695 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.695 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.695 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.695 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.695 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.696 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.696 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.696 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.696 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.696 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap_max_deviation": { 2024-06-04T05:23:48.696 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "upmap_max_deviation", 2024-06-04T05:23:48.696 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.696 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.696 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.696 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-06-04T05:23:48.696 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-06-04T05:23:48.696 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.696 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.697 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "deviation below which no optimization is attempted", 2024-06-04T05:23:48.697 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-06-04T05:23:48.697 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.697 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.697 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap_max_optimizations": { 2024-06-04T05:23:48.697 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "upmap_max_optimizations", 2024-06-04T05:23:48.697 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-06-04T05:23:48.697 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.697 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.697 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:48.697 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.697 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.697 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.697 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-06-04T05:23:48.697 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.698 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.698 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.698 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.698 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.698 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cephadm", 2024-06-04T05:23:48.698 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.698 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.698 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.698 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_down_multiplier": { 2024-06-04T05:23:48.698 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_down_multiplier", 2024-06-04T05:23:48.698 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:48.699 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.699 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.699 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3.0", 2024-06-04T05:23:48.699 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.699 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.699 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.699 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-06-04T05:23:48.699 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.699 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.699 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.699 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_refresh_rate": { 2024-06-04T05:23:48.699 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_refresh_rate", 2024-06-04T05:23:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "20", 2024-06-04T05:23:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-06-04T05:23:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_starting_port": { 2024-06-04T05:23:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_starting_port", 2024-06-04T05:23:48.701 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.701 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.701 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.701 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "4721", 2024-06-04T05:23:48.701 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.701 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.701 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.701 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-06-04T05:23:48.701 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.701 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.701 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.701 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_ptrace": { 2024-06-04T05:23:48.701 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_ptrace", 2024-06-04T05:23:48.701 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.702 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.702 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.702 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.702 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.702 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.702 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.702 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-06-04T05:23:48.702 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-06-04T05:23:48.702 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.702 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.702 INFO:teuthology.orchestra.run.smithi070.stdout: "autotune_interval": { 2024-06-04T05:23:48.702 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "autotune_interval", 2024-06-04T05:23:48.703 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.703 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.703 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.703 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-06-04T05:23:48.703 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.703 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.703 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.703 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to autotune daemon memory", 2024-06-04T05:23:48.703 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.703 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.703 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.703 INFO:teuthology.orchestra.run.smithi070.stdout: "autotune_memory_target_ratio": { 2024-06-04T05:23:48.703 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "autotune_memory_target_ratio", 2024-06-04T05:23:48.704 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:48.704 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.704 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.704 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0.7", 2024-06-04T05:23:48.704 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.704 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.704 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.704 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-06-04T05:23:48.704 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.704 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.704 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.704 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm_log_destination": { 2024-06-04T05:23:48.704 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cephadm_log_destination", 2024-06-04T05:23:48.705 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.705 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.705 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.705 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.705 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.705 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.705 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.705 INFO:teuthology.orchestra.run.smithi070.stdout: "file", 2024-06-04T05:23:48.705 INFO:teuthology.orchestra.run.smithi070.stdout: "file,syslog", 2024-06-04T05:23:48.705 INFO:teuthology.orchestra.run.smithi070.stdout: "syslog" 2024-06-04T05:23:48.705 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.705 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-06-04T05:23:48.705 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.705 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.705 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.706 INFO:teuthology.orchestra.run.smithi070.stdout: "cgroups_split": { 2024-06-04T05:23:48.706 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cgroups_split", 2024-06-04T05:23:48.706 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.706 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.706 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.706 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.706 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.706 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.706 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.706 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-06-04T05:23:48.706 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.706 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.707 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.707 INFO:teuthology.orchestra.run.smithi070.stdout: "config_checks_enabled": { 2024-06-04T05:23:48.707 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_checks_enabled", 2024-06-04T05:23:48.707 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.707 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.707 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.707 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.707 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.707 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.707 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.707 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-06-04T05:23:48.707 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.707 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.708 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.708 INFO:teuthology.orchestra.run.smithi070.stdout: "config_dashboard": { 2024-06-04T05:23:48.708 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_dashboard", 2024-06-04T05:23:48.708 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.708 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.708 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.708 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.708 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.708 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.708 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.708 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-06-04T05:23:48.708 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.709 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.709 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.709 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_alertmanager": { 2024-06-04T05:23:48.709 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_alertmanager", 2024-06-04T05:23:48.709 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.709 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.709 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.709 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-06-04T05:23:48.709 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.710 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.710 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.710 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-06-04T05:23:48.710 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.710 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.710 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.710 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_base": { 2024-06-04T05:23:48.710 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_base", 2024-06-04T05:23:48.710 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.710 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.710 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.710 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/ceph", 2024-06-04T05:23:48.711 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.711 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.711 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.711 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Container image name, without the tag", 2024-06-04T05:23:48.711 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.711 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.711 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.711 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_elasticsearch": { 2024-06-04T05:23:48.711 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_elasticsearch", 2024-06-04T05:23:48.711 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.711 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.711 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.711 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-06-04T05:23:48.712 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.712 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.712 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.712 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "elasticsearch container image", 2024-06-04T05:23:48.712 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.712 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.712 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.712 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_grafana": { 2024-06-04T05:23:48.712 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_grafana", 2024-06-04T05:23:48.712 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.712 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.712 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.713 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-06-04T05:23:48.713 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.713 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.713 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.713 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-06-04T05:23:48.713 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.713 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.713 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.713 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_haproxy": { 2024-06-04T05:23:48.713 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_haproxy", 2024-06-04T05:23:48.713 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.713 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.714 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.714 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-06-04T05:23:48.714 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.714 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.714 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.714 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "HAproxy container image", 2024-06-04T05:23:48.714 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.714 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.714 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.714 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_agent": { 2024-06-04T05:23:48.714 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_agent", 2024-06-04T05:23:48.714 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.714 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.715 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.715 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-06-04T05:23:48.715 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.715 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.715 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.715 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger agent container image", 2024-06-04T05:23:48.715 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.715 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.715 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.715 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_collector": { 2024-06-04T05:23:48.715 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_collector", 2024-06-04T05:23:48.715 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.716 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.716 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.716 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-06-04T05:23:48.716 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.716 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.716 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.716 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger collector container image", 2024-06-04T05:23:48.716 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.716 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.716 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.716 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_query": { 2024-06-04T05:23:48.716 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_query", 2024-06-04T05:23:48.716 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.717 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.717 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.717 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-06-04T05:23:48.717 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.717 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.717 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.717 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger query container image", 2024-06-04T05:23:48.717 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.717 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.717 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.717 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_keepalived": { 2024-06-04T05:23:48.717 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_keepalived", 2024-06-04T05:23:48.717 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.718 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.718 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.718 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-06-04T05:23:48.718 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.718 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.718 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.718 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Keepalived container image", 2024-06-04T05:23:48.718 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.718 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.718 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.718 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_loki": { 2024-06-04T05:23:48.718 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_loki", 2024-06-04T05:23:48.718 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.719 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.719 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.719 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-06-04T05:23:48.719 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.719 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.719 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.719 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Loki container image", 2024-06-04T05:23:48.719 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.719 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.719 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.719 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_node_exporter": { 2024-06-04T05:23:48.719 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_node_exporter", 2024-06-04T05:23:48.720 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.720 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.720 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.720 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-06-04T05:23:48.720 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.720 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.720 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.720 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-06-04T05:23:48.720 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.720 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.720 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.720 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_nvmeof": { 2024-06-04T05:23:48.720 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_nvmeof", 2024-06-04T05:23:48.721 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.721 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.721 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.721 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-06-04T05:23:48.721 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.721 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.721 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.721 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Nvme-of container image", 2024-06-04T05:23:48.721 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.721 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.721 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.721 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_prometheus": { 2024-06-04T05:23:48.721 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_prometheus", 2024-06-04T05:23:48.722 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.722 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.722 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.722 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-06-04T05:23:48.722 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.722 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.722 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.722 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-06-04T05:23:48.722 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.722 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.722 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.722 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_promtail": { 2024-06-04T05:23:48.722 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_promtail", 2024-06-04T05:23:48.723 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.723 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.723 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.723 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-06-04T05:23:48.723 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.723 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.723 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.723 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Promtail container image", 2024-06-04T05:23:48.723 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.723 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.723 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.723 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_samba": { 2024-06-04T05:23:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_samba", 2024-06-04T05:23:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-06-04T05:23:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Samba/SMB container image", 2024-06-04T05:23:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_snmp_gateway": { 2024-06-04T05:23:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_snmp_gateway", 2024-06-04T05:23:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-06-04T05:23:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SNMP Gateway container image", 2024-06-04T05:23:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.726 INFO:teuthology.orchestra.run.smithi070.stdout: "container_init": { 2024-06-04T05:23:48.726 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_init", 2024-06-04T05:23:48.726 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.726 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.726 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.726 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.726 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.726 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.726 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.726 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Run podman/docker with `--init`", 2024-06-04T05:23:48.726 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.726 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.727 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.727 INFO:teuthology.orchestra.run.smithi070.stdout: "daemon_cache_timeout": { 2024-06-04T05:23:48.727 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "daemon_cache_timeout", 2024-06-04T05:23:48.727 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.727 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.727 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.727 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-06-04T05:23:48.727 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.727 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.727 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.727 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-06-04T05:23:48.727 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.727 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.728 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.728 INFO:teuthology.orchestra.run.smithi070.stdout: "default_cephadm_command_timeout": { 2024-06-04T05:23:48.728 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "default_cephadm_command_timeout", 2024-06-04T05:23:48.728 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.728 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.728 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.728 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "900", 2024-06-04T05:23:48.728 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.728 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.728 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.728 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-06-04T05:23:48.728 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.728 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.729 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.729 INFO:teuthology.orchestra.run.smithi070.stdout: "default_registry": { 2024-06-04T05:23:48.729 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "default_registry", 2024-06-04T05:23:48.729 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.729 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.729 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.729 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io", 2024-06-04T05:23:48.729 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.729 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.729 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.729 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-06-04T05:23:48.729 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.730 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.730 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.730 INFO:teuthology.orchestra.run.smithi070.stdout: "device_cache_timeout": { 2024-06-04T05:23:48.730 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_cache_timeout", 2024-06-04T05:23:48.730 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.730 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.730 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.730 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1800", 2024-06-04T05:23:48.730 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.730 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.730 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.730 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache device inventory", 2024-06-04T05:23:48.730 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.731 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.731 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.731 INFO:teuthology.orchestra.run.smithi070.stdout: "device_enhanced_scan": { 2024-06-04T05:23:48.731 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_enhanced_scan", 2024-06-04T05:23:48.731 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.731 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.731 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.731 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.731 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.731 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.731 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.731 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use libstoragemgmt during device scans", 2024-06-04T05:23:48.731 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.732 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.732 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.732 INFO:teuthology.orchestra.run.smithi070.stdout: "facts_cache_timeout": { 2024-06-04T05:23:48.732 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "facts_cache_timeout", 2024-06-04T05:23:48.732 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.732 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.732 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.732 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-06-04T05:23:48.732 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.732 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.732 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.732 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache host facts data", 2024-06-04T05:23:48.732 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.733 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.733 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.733 INFO:teuthology.orchestra.run.smithi070.stdout: "grafana_dashboards_path": { 2024-06-04T05:23:48.733 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "grafana_dashboards_path", 2024-06-04T05:23:48.733 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.733 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.733 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.733 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-06-04T05:23:48.733 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.733 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.733 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.733 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-06-04T05:23:48.733 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.734 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.734 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.734 INFO:teuthology.orchestra.run.smithi070.stdout: "host_check_interval": { 2024-06-04T05:23:48.734 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "host_check_interval", 2024-06-04T05:23:48.734 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.734 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.734 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.734 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-06-04T05:23:48.734 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.734 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.734 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.734 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to perform a host check", 2024-06-04T05:23:48.734 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.735 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.735 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.735 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.735 INFO:teuthology.orchestra.run.smithi070.stdout: "hw_monitoring": { 2024-06-04T05:23:48.735 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "hw_monitoring", 2024-06-04T05:23:48.735 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.735 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.735 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.735 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.735 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.735 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.735 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.735 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-06-04T05:23:48.735 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.736 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.736 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.736 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.736 INFO:teuthology.orchestra.run.smithi070.stdout: "inventory_list_all": { 2024-06-04T05:23:48.736 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "inventory_list_all", 2024-06-04T05:23:48.736 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.736 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.736 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.736 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.736 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.736 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.736 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.736 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-06-04T05:23:48.736 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.737 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.737 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.737 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.737 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.737 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.737 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.737 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.737 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.737 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.737 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.737 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.737 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.737 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.738 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.738 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.738 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.738 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.738 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.738 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.738 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.738 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.738 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.738 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.738 INFO:teuthology.orchestra.run.smithi070.stdout: "log_refresh_metadata": { 2024-06-04T05:23:48.738 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_refresh_metadata", 2024-06-04T05:23:48.739 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.739 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.739 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.739 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.739 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.739 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.739 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.739 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-06-04T05:23:48.739 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.739 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.739 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.739 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.739 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.740 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.740 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.740 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.740 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.740 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.740 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.740 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.740 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-06-04T05:23:48.740 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.740 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.740 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.740 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.740 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.740 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.741 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.741 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.741 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.741 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.741 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.741 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.741 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.741 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.741 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.741 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.741 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.741 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.741 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.742 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.742 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.742 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.742 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.742 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.742 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.742 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.742 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.742 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.742 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.742 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.742 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.742 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.742 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.742 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.743 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.743 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.743 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.743 INFO:teuthology.orchestra.run.smithi070.stdout: "manage_etc_ceph_ceph_conf": { 2024-06-04T05:23:48.743 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-06-04T05:23:48.743 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.743 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.743 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.743 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.743 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.743 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.743 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.743 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-06-04T05:23:48.744 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.744 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.744 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.744 INFO:teuthology.orchestra.run.smithi070.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-06-04T05:23:48.744 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-06-04T05:23:48.744 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.744 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.744 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.744 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "*", 2024-06-04T05:23:48.744 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.744 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.744 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.744 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-06-04T05:23:48.745 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.745 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.745 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.745 INFO:teuthology.orchestra.run.smithi070.stdout: "max_count_per_host": { 2024-06-04T05:23:48.745 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_count_per_host", 2024-06-04T05:23:48.745 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.745 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.745 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.745 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:48.745 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.745 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.745 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.745 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "max number of daemons per service per host", 2024-06-04T05:23:48.746 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.746 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.746 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.746 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.746 INFO:teuthology.orchestra.run.smithi070.stdout: "max_osd_draining_count": { 2024-06-04T05:23:48.746 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_osd_draining_count", 2024-06-04T05:23:48.746 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.746 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.746 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.746 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:48.746 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.746 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.746 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.746 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-06-04T05:23:48.747 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.747 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.747 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.747 INFO:teuthology.orchestra.run.smithi070.stdout: "migration_current": { 2024-06-04T05:23:48.747 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "migration_current", 2024-06-04T05:23:48.747 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.747 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.747 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.747 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.747 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.747 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.747 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.747 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "internal - do not modify", 2024-06-04T05:23:48.747 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.748 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.748 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.748 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": { 2024-06-04T05:23:48.748 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mode", 2024-06-04T05:23:48.748 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.748 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.748 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.748 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "root", 2024-06-04T05:23:48.748 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.748 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.748 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.748 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm-package", 2024-06-04T05:23:48.748 INFO:teuthology.orchestra.run.smithi070.stdout: "root" 2024-06-04T05:23:48.749 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.749 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "mode for remote execution of cephadm", 2024-06-04T05:23:48.749 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.749 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.749 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.749 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.749 INFO:teuthology.orchestra.run.smithi070.stdout: "oob_default_addr": { 2024-06-04T05:23:48.749 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "oob_default_addr", 2024-06-04T05:23:48.749 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.749 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.749 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.749 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "169.254.1.1", 2024-06-04T05:23:48.750 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.750 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.750 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.750 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Default address for RedFish API (oob management).", 2024-06-04T05:23:48.750 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.750 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.750 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.750 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.750 INFO:teuthology.orchestra.run.smithi070.stdout: "prometheus_alerts_path": { 2024-06-04T05:23:48.750 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prometheus_alerts_path", 2024-06-04T05:23:48.750 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.750 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.750 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.751 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-06-04T05:23:48.751 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.751 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.751 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.751 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-06-04T05:23:48.751 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.751 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.751 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.751 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_insecure": { 2024-06-04T05:23:48.751 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_insecure", 2024-06-04T05:23:48.751 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.751 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.751 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.752 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.752 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.752 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.752 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.752 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-06-04T05:23:48.752 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.752 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.752 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.752 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_password": { 2024-06-04T05:23:48.752 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_password", 2024-06-04T05:23:48.752 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.752 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.753 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.753 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.753 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.753 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.753 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.753 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-06-04T05:23:48.753 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.753 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.753 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.753 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_url": { 2024-06-04T05:23:48.753 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_url", 2024-06-04T05:23:48.753 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.753 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.754 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.754 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.754 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.754 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.754 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.754 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-06-04T05:23:48.754 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.754 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.754 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.754 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_username": { 2024-06-04T05:23:48.754 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_username", 2024-06-04T05:23:48.754 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.755 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.755 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.755 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.755 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.755 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.755 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.755 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-06-04T05:23:48.755 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.755 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.755 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.755 INFO:teuthology.orchestra.run.smithi070.stdout: "secure_monitoring_stack": { 2024-06-04T05:23:48.755 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "secure_monitoring_stack", 2024-06-04T05:23:48.755 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.755 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.756 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.756 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.756 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.756 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.756 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.756 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-06-04T05:23:48.756 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.756 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.756 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.756 INFO:teuthology.orchestra.run.smithi070.stdout: "service_discovery_port": { 2024-06-04T05:23:48.756 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "service_discovery_port", 2024-06-04T05:23:48.756 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.756 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.757 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.757 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8765", 2024-06-04T05:23:48.757 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.757 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.757 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.757 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "cephadm service discovery port", 2024-06-04T05:23:48.757 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.757 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.757 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.757 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.757 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.757 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.757 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.758 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.758 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.758 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.758 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.758 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.758 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.758 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.758 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.758 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.758 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.758 INFO:teuthology.orchestra.run.smithi070.stdout: "ssh_config_file": { 2024-06-04T05:23:48.758 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssh_config_file", 2024-06-04T05:23:48.758 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.758 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.759 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.759 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.759 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.759 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.759 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.759 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-06-04T05:23:48.759 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.759 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.759 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.759 INFO:teuthology.orchestra.run.smithi070.stdout: "use_agent": { 2024-06-04T05:23:48.759 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "use_agent", 2024-06-04T05:23:48.759 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.759 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.760 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.760 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.760 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.760 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.760 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.760 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-06-04T05:23:48.760 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.760 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.760 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.760 INFO:teuthology.orchestra.run.smithi070.stdout: "use_repo_digest": { 2024-06-04T05:23:48.760 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "use_repo_digest", 2024-06-04T05:23:48.760 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.760 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.761 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.761 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.761 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.761 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.761 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.761 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-06-04T05:23:48.761 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.761 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.761 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.761 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_failed_host_check": { 2024-06-04T05:23:48.761 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_failed_host_check", 2024-06-04T05:23:48.761 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.761 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.762 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.762 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.762 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.762 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.762 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.762 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if the host check fails", 2024-06-04T05:23:48.762 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.762 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.762 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.762 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_stray_daemons": { 2024-06-04T05:23:48.762 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_stray_daemons", 2024-06-04T05:23:48.763 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.763 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.763 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.763 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.763 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.763 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.763 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.763 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-06-04T05:23:48.763 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.763 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.763 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.763 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_stray_hosts": { 2024-06-04T05:23:48.763 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_stray_hosts", 2024-06-04T05:23:48.764 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.764 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.764 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.764 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.764 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.764 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.764 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.764 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-06-04T05:23:48.764 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.764 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.764 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.764 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.764 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.765 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crash", 2024-06-04T05:23:48.765 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.765 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.765 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.765 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.765 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.765 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.765 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.765 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.765 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.765 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.765 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.766 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.766 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.766 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.766 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.766 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.766 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.766 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.766 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.766 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.766 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.766 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.766 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.767 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.767 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.767 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.767 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.767 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.767 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.767 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.767 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.767 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.767 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.767 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.767 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.767 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.768 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.768 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.768 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.768 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.768 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.768 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.768 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.768 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.768 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.768 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.768 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.768 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.768 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.769 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.769 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.769 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.769 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.769 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.769 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.769 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.769 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.769 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.769 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.769 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.769 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.769 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.770 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.770 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.770 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.770 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.770 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.770 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.770 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.770 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.770 INFO:teuthology.orchestra.run.smithi070.stdout: "retain_interval": { 2024-06-04T05:23:48.770 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "retain_interval", 2024-06-04T05:23:48.770 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.770 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.770 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.771 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "31536000", 2024-06-04T05:23:48.771 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.771 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.771 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.771 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long to retain crashes before pruning them", 2024-06-04T05:23:48.771 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.771 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.771 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.771 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.771 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.771 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.771 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.771 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.771 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.772 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.772 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.772 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.772 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.772 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.772 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.772 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.772 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.772 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.772 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_recent_interval": { 2024-06-04T05:23:48.772 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_recent_interval", 2024-06-04T05:23:48.772 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.772 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.772 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.773 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1209600", 2024-06-04T05:23:48.773 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.773 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.773 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.773 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "time interval in which to warn about recent crashes", 2024-06-04T05:23:48.773 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.773 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.773 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.773 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.773 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.773 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.773 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "dashboard", 2024-06-04T05:23:48.773 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.774 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.774 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.774 INFO:teuthology.orchestra.run.smithi070.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-06-04T05:23:48.774 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-06-04T05:23:48.774 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.774 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.774 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.774 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:48.774 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.774 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.774 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.774 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.774 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.775 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.775 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.775 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.775 INFO:teuthology.orchestra.run.smithi070.stdout: "ALERTMANAGER_API_HOST": { 2024-06-04T05:23:48.775 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ALERTMANAGER_API_HOST", 2024-06-04T05:23:48.775 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.775 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.775 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.775 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.775 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.775 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.775 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.775 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.775 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.776 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.776 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.776 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.776 INFO:teuthology.orchestra.run.smithi070.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-06-04T05:23:48.776 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-06-04T05:23:48.776 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.776 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.776 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.776 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.776 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.776 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.776 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.777 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.777 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.777 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.777 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.777 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.777 INFO:teuthology.orchestra.run.smithi070.stdout: "AUDIT_API_ENABLED": { 2024-06-04T05:23:48.777 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "AUDIT_API_ENABLED", 2024-06-04T05:23:48.777 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.777 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.777 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.777 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.777 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.777 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.778 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.778 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.778 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.778 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.778 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.778 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.778 INFO:teuthology.orchestra.run.smithi070.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-06-04T05:23:48.778 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-06-04T05:23:48.778 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.778 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.778 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.778 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.778 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.779 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.779 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.779 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.779 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.779 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.779 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.779 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.779 INFO:teuthology.orchestra.run.smithi070.stdout: "ENABLE_BROWSABLE_API": { 2024-06-04T05:23:48.779 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ENABLE_BROWSABLE_API", 2024-06-04T05:23:48.779 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.779 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.779 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.779 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.779 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.780 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.780 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.780 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.780 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.780 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.780 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.780 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.780 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-06-04T05:23:48.780 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-06-04T05:23:48.780 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.780 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.780 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.780 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.781 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.781 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.781 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.781 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.781 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.781 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.781 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.781 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.781 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-06-04T05:23:48.781 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-06-04T05:23:48.781 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.781 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.781 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.781 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.782 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.782 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.782 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.782 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.782 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.782 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.782 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.782 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.782 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-06-04T05:23:48.782 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-06-04T05:23:48.782 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.782 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.782 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.782 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.783 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.783 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.783 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.783 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.783 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.783 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.783 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.783 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.783 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-06-04T05:23:48.783 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-06-04T05:23:48.783 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.783 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.783 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.783 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.784 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.784 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.784 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.784 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.784 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.784 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.784 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.784 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.784 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_NFS": { 2024-06-04T05:23:48.784 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-06-04T05:23:48.784 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.784 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.784 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.784 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.785 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.785 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.785 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.785 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.785 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.785 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.785 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.785 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.785 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_RBD": { 2024-06-04T05:23:48.785 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-06-04T05:23:48.785 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.785 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.785 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.785 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.786 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.786 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.786 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.786 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.786 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.786 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.786 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.786 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.786 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_RGW": { 2024-06-04T05:23:48.786 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-06-04T05:23:48.786 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.786 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.786 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.787 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.787 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.787 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.787 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.787 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.787 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.787 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.787 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.787 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.787 INFO:teuthology.orchestra.run.smithi070.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-06-04T05:23:48.787 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-06-04T05:23:48.787 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.787 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.788 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.788 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.788 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.788 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.788 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.788 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.788 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.788 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.788 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.788 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.788 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_PASSWORD": { 2024-06-04T05:23:48.788 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_PASSWORD", 2024-06-04T05:23:48.788 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.789 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.789 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.789 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-06-04T05:23:48.789 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.789 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.789 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.789 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.789 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.789 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.789 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.789 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.789 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-06-04T05:23:48.789 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-06-04T05:23:48.789 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.790 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.790 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.790 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.790 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.790 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.790 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.790 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.790 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.790 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.790 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.790 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.790 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_URL": { 2024-06-04T05:23:48.790 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_URL", 2024-06-04T05:23:48.791 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.791 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.791 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.791 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.791 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.791 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.791 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.791 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.791 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.791 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.791 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.791 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.791 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_USERNAME": { 2024-06-04T05:23:48.791 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_USERNAME", 2024-06-04T05:23:48.792 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.792 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.792 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.792 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-06-04T05:23:48.792 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.792 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.792 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.792 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.792 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.792 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.792 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.792 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.792 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-06-04T05:23:48.793 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-06-04T05:23:48.793 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.793 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.793 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.793 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.793 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.793 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.793 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.793 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.793 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.793 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.793 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.793 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.794 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-06-04T05:23:48.794 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-06-04T05:23:48.794 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.794 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.794 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.794 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.794 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.794 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.794 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.794 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.794 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.794 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.794 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.794 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.795 INFO:teuthology.orchestra.run.smithi070.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-06-04T05:23:48.795 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-06-04T05:23:48.795 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.795 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.795 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.795 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.795 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.795 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.795 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.795 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.795 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.795 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.795 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.796 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ISSUE_TRACKER_API_KEY": { 2024-06-04T05:23:48.796 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-06-04T05:23:48.796 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.796 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.796 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.796 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.796 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.796 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.796 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.796 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.796 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.796 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.797 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.797 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.797 INFO:teuthology.orchestra.run.smithi070.stdout: "MANAGED_BY_CLUSTERS": { 2024-06-04T05:23:48.797 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-06-04T05:23:48.797 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.797 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.797 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.797 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "[]", 2024-06-04T05:23:48.797 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.797 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.797 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.797 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.797 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.797 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.798 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.798 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.798 INFO:teuthology.orchestra.run.smithi070.stdout: "MULTICLUSTER_CONFIG": { 2024-06-04T05:23:48.798 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "MULTICLUSTER_CONFIG", 2024-06-04T05:23:48.798 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.798 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.798 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.798 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "{}", 2024-06-04T05:23:48.798 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.798 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.798 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.798 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.798 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.798 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.799 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.799 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.799 INFO:teuthology.orchestra.run.smithi070.stdout: "PROMETHEUS_API_HOST": { 2024-06-04T05:23:48.799 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PROMETHEUS_API_HOST", 2024-06-04T05:23:48.799 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.799 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.799 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.799 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.799 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.799 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.799 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.799 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.799 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.799 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.799 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.800 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.800 INFO:teuthology.orchestra.run.smithi070.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-06-04T05:23:48.800 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-06-04T05:23:48.800 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.800 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.800 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.800 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.800 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.800 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.800 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.800 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.800 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.800 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.801 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.801 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.801 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-06-04T05:23:48.801 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-06-04T05:23:48.801 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.801 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.801 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.801 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.801 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.801 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.801 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.801 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.801 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.801 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.801 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.801 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.802 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-06-04T05:23:48.802 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-06-04T05:23:48.802 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.802 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.802 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.802 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.802 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.802 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.802 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.802 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.802 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.802 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.802 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.802 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.802 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-06-04T05:23:48.803 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-06-04T05:23:48.803 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.803 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.803 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.803 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.803 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.803 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.803 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.803 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.803 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.803 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.803 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.803 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.803 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-06-04T05:23:48.804 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-06-04T05:23:48.804 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.804 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.804 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.804 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.804 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.804 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.804 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.804 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.804 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.804 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.804 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.804 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.805 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-06-04T05:23:48.805 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-06-04T05:23:48.805 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.805 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.805 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.805 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.805 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.805 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.805 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.805 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.805 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.805 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.805 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.805 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.806 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-06-04T05:23:48.806 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-06-04T05:23:48.806 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.806 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.806 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.806 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.806 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.806 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.806 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.806 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.806 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.806 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.806 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.807 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.807 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-06-04T05:23:48.807 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-06-04T05:23:48.807 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.807 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.807 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.807 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.807 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.807 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.807 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.807 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.807 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.807 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.807 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.807 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.807 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_ENABLED": { 2024-06-04T05:23:48.808 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_ENABLED", 2024-06-04T05:23:48.808 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.808 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.808 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.808 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.808 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.808 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.808 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.808 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.808 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.808 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.808 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.808 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.808 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-06-04T05:23:48.808 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-06-04T05:23:48.809 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.809 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.809 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.809 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-06-04T05:23:48.809 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.809 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.809 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.809 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.809 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.809 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.809 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.809 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.809 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-06-04T05:23:48.809 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-06-04T05:23:48.809 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.810 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.810 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.810 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:48.810 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.810 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.810 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.810 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.810 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.810 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.810 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.810 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.810 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-06-04T05:23:48.810 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-06-04T05:23:48.810 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.811 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.811 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.811 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8", 2024-06-04T05:23:48.811 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.811 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.811 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.811 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.811 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.811 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.811 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.811 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.811 INFO:teuthology.orchestra.run.smithi070.stdout: "REST_REQUESTS_TIMEOUT": { 2024-06-04T05:23:48.811 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-06-04T05:23:48.811 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.812 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.812 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.812 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "45", 2024-06-04T05:23:48.812 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.812 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.812 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.812 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.812 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.812 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.812 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.812 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.812 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_ACCESS_KEY": { 2024-06-04T05:23:48.812 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_ACCESS_KEY", 2024-06-04T05:23:48.812 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.813 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.813 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.813 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.813 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.813 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.813 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.813 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.813 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.813 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.813 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.813 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.813 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-06-04T05:23:48.813 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-06-04T05:23:48.814 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.814 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.814 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.814 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-06-04T05:23:48.814 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.814 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.814 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.814 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.814 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.814 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.814 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.814 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.814 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_SECRET_KEY": { 2024-06-04T05:23:48.814 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_SECRET_KEY", 2024-06-04T05:23:48.815 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.815 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.815 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.815 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.815 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.815 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.815 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.815 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.815 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.815 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.815 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.815 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.815 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_SSL_VERIFY": { 2024-06-04T05:23:48.815 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_SSL_VERIFY", 2024-06-04T05:23:48.816 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.816 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.816 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.816 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.816 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.816 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.816 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.816 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.816 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.816 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.816 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.816 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.817 INFO:teuthology.orchestra.run.smithi070.stdout: "UNSAFE_TLS_v1_2": { 2024-06-04T05:23:48.817 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "UNSAFE_TLS_v1_2", 2024-06-04T05:23:48.817 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.817 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.817 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.817 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.817 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.817 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.817 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.817 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.817 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.817 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.817 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.817 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.818 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-06-04T05:23:48.818 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-06-04T05:23:48.818 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.818 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.818 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.818 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.818 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.818 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.818 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.818 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.818 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.818 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.818 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.819 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.819 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-06-04T05:23:48.819 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-06-04T05:23:48.819 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.819 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.819 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.819 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:48.819 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.819 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.819 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.819 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.819 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.819 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.819 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.820 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.820 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-06-04T05:23:48.820 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-06-04T05:23:48.820 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.820 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.820 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.820 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-06-04T05:23:48.820 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.820 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.820 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.820 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.820 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.820 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.820 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.821 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.821 INFO:teuthology.orchestra.run.smithi070.stdout: "cross_origin_url": { 2024-06-04T05:23:48.821 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cross_origin_url", 2024-06-04T05:23:48.821 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.821 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.821 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.821 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.821 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.821 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.821 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.821 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.821 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.821 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.821 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.822 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.822 INFO:teuthology.orchestra.run.smithi070.stdout: "crt_file": { 2024-06-04T05:23:48.822 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crt_file", 2024-06-04T05:23:48.822 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.822 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.822 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.822 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.822 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.822 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.822 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.822 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.822 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.822 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.823 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.823 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.823 INFO:teuthology.orchestra.run.smithi070.stdout: "debug": { 2024-06-04T05:23:48.823 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "debug", 2024-06-04T05:23:48.823 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.823 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.823 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.823 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.823 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.823 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.823 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.823 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable/disable debug options", 2024-06-04T05:23:48.823 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.823 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.823 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.824 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.824 INFO:teuthology.orchestra.run.smithi070.stdout: "jwt_token_ttl": { 2024-06-04T05:23:48.824 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "jwt_token_ttl", 2024-06-04T05:23:48.824 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.824 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.824 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.824 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "28800", 2024-06-04T05:23:48.824 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.824 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.824 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.824 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.824 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.824 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.824 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.824 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.825 INFO:teuthology.orchestra.run.smithi070.stdout: "key_file": { 2024-06-04T05:23:48.825 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "key_file", 2024-06-04T05:23:48.825 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.825 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.825 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.825 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.825 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.825 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.825 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.825 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.825 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.825 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.825 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.825 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.826 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.826 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.826 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.826 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.826 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.826 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.826 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.826 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.826 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.826 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.826 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.826 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.826 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.827 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.827 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.827 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.827 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.827 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.827 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.827 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.827 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.827 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.827 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.827 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.827 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.827 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.827 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.827 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.828 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.828 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.828 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.828 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.828 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.828 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.828 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.828 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.828 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.828 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.828 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.828 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.828 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.828 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.829 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.829 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.829 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.829 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.829 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.829 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.829 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.829 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.829 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.829 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.829 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.829 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.829 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.830 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.830 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.830 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.830 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.830 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.830 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.830 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.830 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.830 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.830 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.830 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.830 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.830 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.831 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.831 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.831 INFO:teuthology.orchestra.run.smithi070.stdout: "motd": { 2024-06-04T05:23:48.831 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "motd", 2024-06-04T05:23:48.831 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.831 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.831 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.831 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.831 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.831 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.831 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.831 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "The message of the day", 2024-06-04T05:23:48.831 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.832 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.832 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.832 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.832 INFO:teuthology.orchestra.run.smithi070.stdout: "redirect_resolve_ip_addr": { 2024-06-04T05:23:48.832 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "redirect_resolve_ip_addr", 2024-06-04T05:23:48.832 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.832 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.832 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.832 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.832 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.832 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.832 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.832 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.832 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.833 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.833 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.833 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.833 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-06-04T05:23:48.833 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-06-04T05:23:48.833 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.833 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.833 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.833 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "::", 2024-06-04T05:23:48.833 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.833 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.833 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.833 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.833 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.834 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.834 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.834 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.834 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-06-04T05:23:48.834 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-06-04T05:23:48.834 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.834 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.834 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.834 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8080", 2024-06-04T05:23:48.834 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.834 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.834 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.834 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.835 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.835 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.835 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.835 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.835 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.835 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.835 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.835 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.835 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.835 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.835 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.835 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.835 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.835 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.836 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.836 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.836 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.836 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.836 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl": { 2024-06-04T05:23:48.836 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl", 2024-06-04T05:23:48.836 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.836 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.836 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.836 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.836 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.836 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.836 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.836 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.837 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.837 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.837 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.837 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.837 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl_server_port": { 2024-06-04T05:23:48.837 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl_server_port", 2024-06-04T05:23:48.837 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.837 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.837 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.837 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8443", 2024-06-04T05:23:48.837 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.837 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.837 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.837 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.837 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.838 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.838 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.838 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.838 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_behaviour": { 2024-06-04T05:23:48.838 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_behaviour", 2024-06-04T05:23:48.838 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.838 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.838 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.838 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "redirect", 2024-06-04T05:23:48.838 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.838 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.838 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.838 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.838 INFO:teuthology.orchestra.run.smithi070.stdout: "redirect" 2024-06-04T05:23:48.839 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.839 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.839 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.839 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.839 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.839 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_error_status_code": { 2024-06-04T05:23:48.839 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_error_status_code", 2024-06-04T05:23:48.839 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.839 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.839 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.839 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "500", 2024-06-04T05:23:48.839 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "400", 2024-06-04T05:23:48.839 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "599", 2024-06-04T05:23:48.839 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.840 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.840 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.840 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.840 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.840 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.840 INFO:teuthology.orchestra.run.smithi070.stdout: "url_prefix": { 2024-06-04T05:23:48.840 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "url_prefix", 2024-06-04T05:23:48.840 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.840 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.840 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.840 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.840 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.840 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.840 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.840 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.840 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.840 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.841 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.841 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.841 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.841 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.841 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.841 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "devicehealth", 2024-06-04T05:23:48.841 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.841 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.841 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.841 INFO:teuthology.orchestra.run.smithi070.stdout: "enable_monitoring": { 2024-06-04T05:23:48.841 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enable_monitoring", 2024-06-04T05:23:48.841 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.841 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.841 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.841 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.842 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.842 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.842 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.842 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "monitor device health metrics", 2024-06-04T05:23:48.842 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.842 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.842 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.842 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.842 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.842 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.842 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.842 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.842 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.843 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.843 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.843 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.843 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.843 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.843 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.843 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.843 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.843 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.843 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.843 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.843 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.843 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.843 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.844 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.844 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.844 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.844 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.844 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.844 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.844 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.844 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.844 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.844 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.844 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.844 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.844 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.844 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.845 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.845 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.845 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.845 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.845 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.845 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.845 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.845 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.845 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.845 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.845 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.845 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.845 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.845 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.845 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.846 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.846 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.846 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.846 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.846 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.846 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.846 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.846 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.846 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.846 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.846 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.846 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.846 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.846 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.847 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.847 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.847 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.847 INFO:teuthology.orchestra.run.smithi070.stdout: "mark_out_threshold": { 2024-06-04T05:23:48.847 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mark_out_threshold", 2024-06-04T05:23:48.847 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.847 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.847 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.847 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "2419200", 2024-06-04T05:23:48.847 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.847 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.847 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.847 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-06-04T05:23:48.847 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.848 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.848 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.848 INFO:teuthology.orchestra.run.smithi070.stdout: "pool_name": { 2024-06-04T05:23:48.848 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pool_name", 2024-06-04T05:23:48.848 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.848 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.848 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.848 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "device_health_metrics", 2024-06-04T05:23:48.848 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.848 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.848 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.848 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "name of pool in which to store device health metrics", 2024-06-04T05:23:48.848 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.848 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.849 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.849 INFO:teuthology.orchestra.run.smithi070.stdout: "retention_period": { 2024-06-04T05:23:48.849 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "retention_period", 2024-06-04T05:23:48.849 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.849 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.849 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.849 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15552000", 2024-06-04T05:23:48.849 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.849 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.849 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.849 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long to retain device health metrics", 2024-06-04T05:23:48.849 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.849 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.849 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.850 INFO:teuthology.orchestra.run.smithi070.stdout: "scrape_frequency": { 2024-06-04T05:23:48.850 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "scrape_frequency", 2024-06-04T05:23:48.850 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.850 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.850 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.850 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "86400", 2024-06-04T05:23:48.850 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.850 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.850 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.850 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to scrape device health metrics", 2024-06-04T05:23:48.850 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.850 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.850 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.851 INFO:teuthology.orchestra.run.smithi070.stdout: "self_heal": { 2024-06-04T05:23:48.851 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "self_heal", 2024-06-04T05:23:48.851 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.851 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.851 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.851 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.851 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.851 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.851 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.851 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-06-04T05:23:48.851 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.851 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.851 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.852 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-06-04T05:23:48.852 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-06-04T05:23:48.852 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.852 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.852 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.852 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-06-04T05:23:48.852 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.852 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.852 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.852 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to wake up and check device health", 2024-06-04T05:23:48.852 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.852 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.852 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.852 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.853 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.853 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.853 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.853 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.853 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.853 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.853 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.853 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.853 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.853 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.853 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.853 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.853 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_threshold": { 2024-06-04T05:23:48.853 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_threshold", 2024-06-04T05:23:48.853 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.854 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.854 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.854 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "7257600", 2024-06-04T05:23:48.854 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.854 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.854 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.854 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-06-04T05:23:48.854 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.854 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.854 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.854 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.854 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.855 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "diskprediction_local", 2024-06-04T05:23:48.855 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.855 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.855 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.855 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.855 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.855 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.855 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.855 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.855 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.855 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.855 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.855 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.857 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.857 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.857 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.857 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.857 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.857 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.857 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.857 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.857 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.857 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.857 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.857 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.857 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.858 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.858 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.858 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.858 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.858 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.858 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.858 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.858 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.858 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.858 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.858 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.858 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.858 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.859 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.859 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.859 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.859 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.859 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.859 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.859 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.859 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.859 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.859 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.859 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.859 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.859 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.860 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.860 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.860 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.860 INFO:teuthology.orchestra.run.smithi070.stdout: "predict_interval": { 2024-06-04T05:23:48.860 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "predict_interval", 2024-06-04T05:23:48.860 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.860 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.860 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.860 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "86400", 2024-06-04T05:23:48.860 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.860 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.860 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.860 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.861 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.861 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.861 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.861 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.861 INFO:teuthology.orchestra.run.smithi070.stdout: "predictor_model": { 2024-06-04T05:23:48.861 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "predictor_model", 2024-06-04T05:23:48.861 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.861 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.861 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.861 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "prophetstor", 2024-06-04T05:23:48.861 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.861 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.861 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.861 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.862 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.862 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.862 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.862 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.862 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-06-04T05:23:48.862 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-06-04T05:23:48.862 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.862 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.862 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.862 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-06-04T05:23:48.862 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.862 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.863 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.863 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.863 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.863 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.863 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.863 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.863 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.863 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.863 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.863 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.863 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.863 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.863 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.863 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.864 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.864 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.864 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.864 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.864 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.864 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.864 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "influx", 2024-06-04T05:23:48.864 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": false, 2024-06-04T05:23:48.864 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "influxdb python module not found", 2024-06-04T05:23:48.864 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.864 INFO:teuthology.orchestra.run.smithi070.stdout: "batch_size": { 2024-06-04T05:23:48.864 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "batch_size", 2024-06-04T05:23:48.865 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.865 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.865 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.865 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5000", 2024-06-04T05:23:48.865 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.865 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.865 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.865 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-06-04T05:23:48.865 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.865 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.865 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.865 INFO:teuthology.orchestra.run.smithi070.stdout: "database": { 2024-06-04T05:23:48.865 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "database", 2024-06-04T05:23:48.865 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.866 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.866 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.866 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "ceph", 2024-06-04T05:23:48.866 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.866 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.866 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.866 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-06-04T05:23:48.866 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.866 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.866 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.866 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.866 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": { 2024-06-04T05:23:48.866 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "hostname", 2024-06-04T05:23:48.866 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.867 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.867 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.867 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.867 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.867 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.867 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.867 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB server hostname", 2024-06-04T05:23:48.867 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.867 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.867 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.867 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-06-04T05:23:48.867 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-06-04T05:23:48.867 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.868 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.868 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.868 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "30", 2024-06-04T05:23:48.868 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "5", 2024-06-04T05:23:48.868 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.868 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.868 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-06-04T05:23:48.868 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.868 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.868 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.868 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.868 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.868 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.869 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.869 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.869 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.869 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.869 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.869 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.869 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.869 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.869 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.869 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.869 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.869 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.869 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.870 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.870 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.870 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.870 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.870 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.870 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.870 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.870 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.870 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.870 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.870 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.870 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.870 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.871 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.871 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.871 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.871 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.871 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.871 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.871 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.871 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.871 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.871 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.871 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.871 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.871 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.872 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.872 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.872 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.872 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.872 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.872 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.872 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.872 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.872 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.872 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.872 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.872 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.872 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.873 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.873 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.873 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.873 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.873 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.873 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.873 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.873 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.873 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.873 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.873 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.873 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.874 INFO:teuthology.orchestra.run.smithi070.stdout: "password": { 2024-06-04T05:23:48.874 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "password", 2024-06-04T05:23:48.874 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.874 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.874 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.874 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.874 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.874 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.874 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.874 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "password of InfluxDB server user", 2024-06-04T05:23:48.874 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.874 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.874 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.875 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.875 INFO:teuthology.orchestra.run.smithi070.stdout: "port": { 2024-06-04T05:23:48.875 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "port", 2024-06-04T05:23:48.875 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.875 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.875 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.875 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8086", 2024-06-04T05:23:48.875 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.875 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.875 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.875 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB server port", 2024-06-04T05:23:48.875 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.875 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.875 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.876 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.876 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.876 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.876 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.876 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.876 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.876 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.876 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.876 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.876 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.876 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.876 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.876 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.877 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl": { 2024-06-04T05:23:48.877 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl", 2024-06-04T05:23:48.877 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.877 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.877 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.877 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "false", 2024-06-04T05:23:48.877 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.877 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.877 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.877 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-06-04T05:23:48.877 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.877 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.878 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.878 INFO:teuthology.orchestra.run.smithi070.stdout: "threads": { 2024-06-04T05:23:48.878 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "threads", 2024-06-04T05:23:48.878 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.878 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.878 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.878 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-06-04T05:23:48.878 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-06-04T05:23:48.878 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "32", 2024-06-04T05:23:48.878 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.878 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-06-04T05:23:48.878 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.878 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.879 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.879 INFO:teuthology.orchestra.run.smithi070.stdout: "username": { 2024-06-04T05:23:48.879 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "username", 2024-06-04T05:23:48.879 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.879 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.879 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.879 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.879 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.879 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.879 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.879 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "username of InfluxDB server user", 2024-06-04T05:23:48.879 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.879 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.880 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.880 INFO:teuthology.orchestra.run.smithi070.stdout: "verify_ssl": { 2024-06-04T05:23:48.880 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "verify_ssl", 2024-06-04T05:23:48.880 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.880 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.880 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.880 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "true", 2024-06-04T05:23:48.880 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.880 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.880 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.880 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-06-04T05:23:48.880 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.880 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.881 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.881 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.881 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.881 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "insights", 2024-06-04T05:23:48.881 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.881 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.881 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.881 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.881 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.881 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.881 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.881 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.882 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.882 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.882 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.882 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.882 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.882 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.882 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.882 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.882 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.882 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.882 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.882 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.883 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.883 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.883 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.883 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.883 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.883 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.883 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.883 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.883 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.883 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.883 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.883 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.883 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.884 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.884 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.884 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.884 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.884 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.884 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.884 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.884 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.884 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.884 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.884 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.884 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.885 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.885 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.885 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.885 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.885 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.885 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.885 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.885 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.885 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.885 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.885 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.885 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.885 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.886 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.886 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.886 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.886 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.886 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.886 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.886 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.886 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.886 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.886 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.886 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.886 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.886 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.886 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.887 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.887 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.887 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.887 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.887 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.887 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.887 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.887 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.887 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.887 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.887 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.887 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.887 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.888 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "iostat", 2024-06-04T05:23:48.888 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.888 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.888 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.888 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.888 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.888 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.888 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.888 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.888 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.888 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.889 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.889 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.889 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.889 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.889 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.889 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.889 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.889 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.889 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.889 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.889 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.889 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.889 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.890 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.890 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.890 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.890 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.890 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.890 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.890 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.890 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.890 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.890 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.890 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.890 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.890 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.891 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.891 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.891 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.891 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.891 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.891 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.891 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.891 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.891 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.891 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.891 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.891 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.891 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.892 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.892 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.892 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.892 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.892 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.892 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.892 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.892 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.892 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.892 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.892 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.892 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.893 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.893 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.893 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.893 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.893 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.893 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.893 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.893 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.893 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.893 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.893 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.893 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.893 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.894 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.894 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.894 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.894 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.894 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.894 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.894 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.894 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.894 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.894 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.894 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.894 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "k8sevents", 2024-06-04T05:23:48.895 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.895 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.895 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.895 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_event_retention_days": { 2024-06-04T05:23:48.895 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ceph_event_retention_days", 2024-06-04T05:23:48.895 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.895 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.895 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.895 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "7", 2024-06-04T05:23:48.895 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.895 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.895 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.895 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Days to hold ceph event information within local cache", 2024-06-04T05:23:48.895 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.896 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.896 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.896 INFO:teuthology.orchestra.run.smithi070.stdout: "config_check_secs": { 2024-06-04T05:23:48.896 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_check_secs", 2024-06-04T05:23:48.896 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.896 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.896 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.896 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:48.896 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "10", 2024-06-04T05:23:48.896 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.896 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.896 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-06-04T05:23:48.896 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.897 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.897 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.897 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.897 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.897 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.897 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.897 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.897 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.897 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.897 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.897 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.898 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.898 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.898 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.898 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.898 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.898 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.898 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.898 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.898 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.898 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.898 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.898 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.898 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.899 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.899 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.899 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.899 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.899 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.899 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.899 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.899 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.899 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.899 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.899 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.899 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.899 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.900 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.900 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.900 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.900 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.900 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.900 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.900 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.900 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.900 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.900 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.900 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.900 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.900 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.901 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.901 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.901 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.901 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.901 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.901 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.901 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.901 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.901 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.901 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.901 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.901 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.901 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.902 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.902 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.902 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.902 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.902 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.902 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.902 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.902 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.902 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.902 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.902 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.902 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.903 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.903 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.903 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.903 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.903 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.903 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.903 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.903 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.903 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.903 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "localpool", 2024-06-04T05:23:48.903 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.904 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.904 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.904 INFO:teuthology.orchestra.run.smithi070.stdout: "failure_domain": { 2024-06-04T05:23:48.904 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "failure_domain", 2024-06-04T05:23:48.904 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.904 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.904 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.904 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "host", 2024-06-04T05:23:48.904 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.904 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.904 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.904 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "failure domain for any created local pool", 2024-06-04T05:23:48.904 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-06-04T05:23:48.904 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.904 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.905 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.905 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.905 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.905 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.905 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.905 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.905 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.905 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.905 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.905 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.905 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.905 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.905 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.906 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.906 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.906 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.906 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.906 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.906 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.906 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.906 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.906 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.906 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.906 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.906 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.906 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.907 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.907 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.907 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.907 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.907 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.907 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.907 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.907 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.907 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.907 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.907 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.907 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.907 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.908 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.908 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.908 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.908 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.908 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.908 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.908 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.908 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.908 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.908 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.908 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.908 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.908 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.909 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.909 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.909 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.909 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.909 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.909 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.909 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.909 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.909 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.909 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.909 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.909 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.909 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.909 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.910 INFO:teuthology.orchestra.run.smithi070.stdout: "min_size": { 2024-06-04T05:23:48.910 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "min_size", 2024-06-04T05:23:48.910 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.910 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.910 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.910 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.910 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.910 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.910 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.910 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default min_size for any created local pool", 2024-06-04T05:23:48.910 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-06-04T05:23:48.910 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.910 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.911 INFO:teuthology.orchestra.run.smithi070.stdout: "num_rep": { 2024-06-04T05:23:48.911 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "num_rep", 2024-06-04T05:23:48.911 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.911 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.911 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.911 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3", 2024-06-04T05:23:48.911 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.911 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.911 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.911 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default replica count for any created local pool", 2024-06-04T05:23:48.911 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.912 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.912 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.912 INFO:teuthology.orchestra.run.smithi070.stdout: "pg_num": { 2024-06-04T05:23:48.912 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pg_num", 2024-06-04T05:23:48.912 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.912 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.912 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.912 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "128", 2024-06-04T05:23:48.912 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.912 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.912 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.912 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default pg_num for any created local pool", 2024-06-04T05:23:48.913 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.913 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.913 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.913 INFO:teuthology.orchestra.run.smithi070.stdout: "prefix": { 2024-06-04T05:23:48.913 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prefix", 2024-06-04T05:23:48.913 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.913 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.913 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.913 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.913 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.913 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.913 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.913 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "name prefix for any created local pool", 2024-06-04T05:23:48.914 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.914 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.914 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.914 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.914 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.914 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.914 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.914 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.914 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.914 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.914 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.914 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.915 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.915 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.915 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.915 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.915 INFO:teuthology.orchestra.run.smithi070.stdout: "subtree": { 2024-06-04T05:23:48.915 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "subtree", 2024-06-04T05:23:48.915 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.915 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.915 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.915 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "rack", 2024-06-04T05:23:48.915 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.915 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.915 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.915 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "CRUSH level for which to create a local pool", 2024-06-04T05:23:48.916 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-06-04T05:23:48.916 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.916 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.916 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.916 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.916 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.916 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mds_autoscaler", 2024-06-04T05:23:48.917 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.918 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.918 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.918 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.918 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.918 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.918 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.918 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.918 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.918 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.918 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.918 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.918 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.919 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.919 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.919 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.919 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.919 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.919 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.919 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.919 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.919 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.919 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.919 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.919 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.919 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.919 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.920 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.920 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.920 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.920 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.920 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.920 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.920 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.920 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.920 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.920 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.920 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.920 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.921 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.921 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.921 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.921 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.921 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.921 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.921 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.921 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.921 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.921 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.921 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.921 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.921 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.922 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.922 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.922 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.922 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.922 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.922 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.922 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.922 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.922 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.922 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.922 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.922 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.922 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.923 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.923 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.923 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.923 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.923 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.923 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.923 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.923 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.923 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.923 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.923 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.923 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.923 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.924 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.924 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.924 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.924 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.924 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.924 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.924 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.924 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mirroring", 2024-06-04T05:23:48.924 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.924 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.924 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.924 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.924 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.925 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.925 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.925 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.925 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.925 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.925 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.925 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.925 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.925 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.925 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.925 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.925 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.925 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.925 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.926 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.926 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.926 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.926 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.926 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.926 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.926 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.926 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.926 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.926 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.926 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.927 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.927 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.927 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.927 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.927 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.927 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.927 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.927 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.927 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.927 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.927 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.927 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.927 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.928 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.928 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.928 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.928 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.928 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.928 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.928 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.928 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.928 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.928 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.928 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.928 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.928 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.929 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.929 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.929 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.929 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.929 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.929 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.929 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.929 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.929 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.929 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.929 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.929 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.929 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.929 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.930 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.930 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.930 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.930 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.930 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.930 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.930 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.930 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.930 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.930 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.930 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.930 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.930 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.931 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.931 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.931 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.931 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "nfs", 2024-06-04T05:23:48.931 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.931 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.931 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.931 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.931 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.931 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.931 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.931 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.931 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.932 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.932 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.932 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.932 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.932 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.932 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.932 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.932 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.932 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.932 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.932 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.932 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.932 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.932 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.933 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.933 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.933 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.933 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.933 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.933 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.933 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.933 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.933 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.933 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.933 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.933 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.934 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.934 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.934 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.934 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.934 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.934 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.934 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.934 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.934 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.934 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.934 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.934 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.934 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.935 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.935 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.935 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.935 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.935 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.935 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.935 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.935 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.935 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.935 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.935 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.935 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.935 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.936 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.936 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.936 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.936 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.936 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.936 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.936 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.936 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.936 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.936 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.936 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.936 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.936 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.937 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.937 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.937 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.937 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.937 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.937 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.937 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.937 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.937 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.937 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.937 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.937 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.938 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "orchestrator", 2024-06-04T05:23:48.938 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.938 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.938 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.938 INFO:teuthology.orchestra.run.smithi070.stdout: "fail_fs": { 2024-06-04T05:23:48.938 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "fail_fs", 2024-06-04T05:23:48.938 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.938 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.938 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.938 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.938 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.938 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.938 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.938 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-06-04T05:23:48.939 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.939 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.939 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.939 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.939 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.939 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.939 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.939 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.939 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.939 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.939 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.939 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.939 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.940 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.940 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.940 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.940 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.940 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.940 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.940 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.940 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.940 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.940 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.940 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.940 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.940 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.940 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.941 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.941 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.941 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.941 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.941 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.941 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.941 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.941 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.941 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.941 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.941 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.942 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.942 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.942 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.942 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.942 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.942 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.942 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.942 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.942 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.942 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.942 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.942 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.942 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.942 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.943 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.943 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.943 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.943 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.943 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.943 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.943 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.943 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.943 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.943 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.943 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.943 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.943 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.943 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.944 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.944 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.944 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.944 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.944 INFO:teuthology.orchestra.run.smithi070.stdout: "orchestrator": { 2024-06-04T05:23:48.944 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "orchestrator", 2024-06-04T05:23:48.944 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.944 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.944 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.944 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.944 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.944 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.944 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.945 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm", 2024-06-04T05:23:48.945 INFO:teuthology.orchestra.run.smithi070.stdout: "rook", 2024-06-04T05:23:48.945 INFO:teuthology.orchestra.run.smithi070.stdout: "test_orchestrator" 2024-06-04T05:23:48.945 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.945 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Orchestrator backend", 2024-06-04T05:23:48.945 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.945 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.945 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.945 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.945 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.945 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.945 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.945 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.946 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.946 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.946 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.946 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.946 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.946 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.946 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.946 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.946 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.946 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.946 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.946 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.946 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "osd_perf_query", 2024-06-04T05:23:48.946 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.947 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.947 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.947 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.947 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.947 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.947 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.947 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.947 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.947 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.947 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.947 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.947 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.947 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.948 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.948 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.948 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.948 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.948 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.948 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.948 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.948 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.948 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.948 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.948 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.948 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.948 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.949 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.949 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.949 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.949 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.949 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.949 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.949 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.949 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.949 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.949 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.949 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.949 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.949 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.950 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.950 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.950 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.950 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.950 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.950 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.950 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.950 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.950 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.950 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.950 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.950 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.950 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.951 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.951 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.951 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.951 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.951 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.951 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.951 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.951 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.951 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.951 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.951 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.951 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.951 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.951 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.951 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.952 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.952 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.952 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.952 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.952 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.952 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.952 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.952 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.952 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.952 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.952 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.952 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.952 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.953 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.953 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.953 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.953 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.953 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.953 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.953 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "osd_support", 2024-06-04T05:23:48.953 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.953 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.953 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.953 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.953 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.953 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.954 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.954 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.954 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.954 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.954 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.954 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.954 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.954 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.954 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.954 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.954 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.954 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.954 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.955 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.955 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.955 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.955 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.955 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.955 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.955 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.955 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.955 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.955 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.955 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.955 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.955 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.956 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.956 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.956 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.956 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.956 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.956 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.956 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.956 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.956 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.956 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.956 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.956 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.956 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.956 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.957 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.957 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.957 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.957 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.957 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.957 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.957 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.957 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.957 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.957 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.957 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.957 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.957 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.957 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.958 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.958 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.958 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.958 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.958 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.958 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.958 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.958 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.958 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.958 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.958 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.958 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.958 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.959 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.959 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.959 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.959 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.959 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.959 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.959 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.959 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.959 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.959 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.959 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.959 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.959 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.959 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.959 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.960 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.960 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.960 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pg_autoscaler", 2024-06-04T05:23:48.960 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.960 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.960 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.960 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.960 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.960 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.960 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.960 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.960 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.960 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.960 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.961 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.961 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.961 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.961 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.961 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.961 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.961 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.961 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.961 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.961 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.961 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.961 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.961 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.961 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.962 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.962 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.962 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.962 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.962 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.962 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.962 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.962 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.962 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.962 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.962 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.962 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.962 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.963 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.963 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.963 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.963 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.963 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.963 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.963 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.963 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.963 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.963 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.963 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.963 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.963 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.963 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.963 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.964 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.964 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.964 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.964 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.964 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.964 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.964 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.964 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.964 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.964 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.964 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.964 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.964 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.964 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.965 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.965 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.965 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.965 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.965 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.965 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.965 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-06-04T05:23:48.965 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-06-04T05:23:48.965 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.965 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.965 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.965 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-06-04T05:23:48.965 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.965 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.966 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.966 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.966 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.966 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.966 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.966 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.966 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.966 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.966 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.966 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.966 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.966 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.966 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.967 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.967 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.967 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.967 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.967 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.967 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.967 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.967 INFO:teuthology.orchestra.run.smithi070.stdout: "threshold": { 2024-06-04T05:23:48.967 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "threshold", 2024-06-04T05:23:48.967 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:48.967 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.967 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.967 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3.0", 2024-06-04T05:23:48.967 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1.0", 2024-06-04T05:23:48.968 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.968 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.968 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "scaling threshold", 2024-06-04T05:23:48.968 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-06-04T05:23:48.968 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.968 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.968 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.968 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.968 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.968 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.968 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "progress", 2024-06-04T05:23:48.968 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.968 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.968 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.969 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_pg_recovery_event": { 2024-06-04T05:23:48.969 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_pg_recovery_event", 2024-06-04T05:23:48.969 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.969 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.969 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.969 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.969 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.969 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.969 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.969 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow the module to show pg recovery progress", 2024-06-04T05:23:48.969 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.969 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.970 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.970 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.970 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": { 2024-06-04T05:23:48.970 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enabled", 2024-06-04T05:23:48.970 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.970 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.970 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.970 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.970 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.970 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.970 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.970 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.970 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.971 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.971 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.971 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.971 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.971 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.971 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.971 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.971 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.971 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.971 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.971 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.971 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.971 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.972 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.972 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.972 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.972 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.972 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.972 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.972 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.972 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.972 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.972 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.972 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.972 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.972 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.972 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.973 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.973 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.973 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.973 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.973 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.973 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.973 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.973 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.973 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.973 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.973 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.973 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.973 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.973 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.974 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.974 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.974 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.974 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.974 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.974 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.974 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.974 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.974 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.974 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.974 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.974 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.974 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.975 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.975 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.975 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.975 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.975 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.975 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.975 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.975 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.975 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.975 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.975 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.975 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.975 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.975 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.976 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.976 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.976 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.976 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.976 INFO:teuthology.orchestra.run.smithi070.stdout: "max_completed_events": { 2024-06-04T05:23:48.976 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_completed_events", 2024-06-04T05:23:48.976 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.976 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.976 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.976 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "50", 2024-06-04T05:23:48.976 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.976 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.976 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.976 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "number of past completed events to remember", 2024-06-04T05:23:48.977 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.977 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.977 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.977 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.977 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-06-04T05:23:48.977 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-06-04T05:23:48.977 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:48.977 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.977 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.977 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-06-04T05:23:48.977 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.977 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.977 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.977 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long the module is going to sleep", 2024-06-04T05:23:48.978 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.978 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.978 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.978 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.978 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.978 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.978 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.978 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.978 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.978 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.978 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.978 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.978 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.978 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.979 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.979 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.979 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.979 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.979 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.979 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.979 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.979 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prometheus", 2024-06-04T05:23:48.979 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.979 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.979 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.979 INFO:teuthology.orchestra.run.smithi070.stdout: "cache": { 2024-06-04T05:23:48.979 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cache", 2024-06-04T05:23:48.979 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.980 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.980 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.980 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.980 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.980 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.980 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.980 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.980 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.980 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.980 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.980 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.980 INFO:teuthology.orchestra.run.smithi070.stdout: "exclude_perf_counters": { 2024-06-04T05:23:48.980 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "exclude_perf_counters", 2024-06-04T05:23:48.980 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.980 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.981 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.981 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:48.981 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.981 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.981 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.981 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-06-04T05:23:48.981 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-06-04T05:23:48.981 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.981 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.981 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.981 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.981 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.981 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.981 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.982 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.982 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.982 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.982 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.982 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.982 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.982 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.982 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.982 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.982 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.983 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.983 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.983 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.983 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.983 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.983 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.983 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.983 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.983 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.983 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.983 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.983 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.983 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.983 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.984 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.984 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.984 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.984 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.984 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.984 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.984 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.984 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.984 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.984 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.984 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.984 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.984 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:48.984 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.985 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.985 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.985 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.985 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.985 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.985 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.985 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.985 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.985 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.985 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.985 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.985 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.985 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.985 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.986 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:48.986 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:48.986 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.986 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.986 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.986 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.986 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.986 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.986 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.986 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.986 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.986 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.986 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.986 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.987 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_stats_pools": { 2024-06-04T05:23:48.987 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_stats_pools", 2024-06-04T05:23:48.987 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.987 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.987 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.987 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.987 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.987 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.987 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.987 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.987 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.987 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.987 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.987 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.988 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_stats_pools_refresh_interval": { 2024-06-04T05:23:48.988 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-06-04T05:23:48.988 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.988 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.988 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.988 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "300", 2024-06-04T05:23:48.988 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.988 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.988 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.988 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.988 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.988 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.988 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.988 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.989 INFO:teuthology.orchestra.run.smithi070.stdout: "scrape_interval": { 2024-06-04T05:23:48.989 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "scrape_interval", 2024-06-04T05:23:48.989 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:48.989 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.989 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.989 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15.0", 2024-06-04T05:23:48.989 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.989 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.989 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.989 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.989 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.989 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.989 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.989 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.990 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-06-04T05:23:48.990 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-06-04T05:23:48.990 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.990 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.990 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.990 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "::", 2024-06-04T05:23:48.990 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.990 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.990 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.990 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-06-04T05:23:48.990 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.990 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.990 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.991 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.991 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-06-04T05:23:48.991 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-06-04T05:23:48.991 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.991 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.991 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.991 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "9283", 2024-06-04T05:23:48.991 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.991 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.991 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.991 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-06-04T05:23:48.991 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.991 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.991 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.992 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.992 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:48.992 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:48.992 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.992 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:48.992 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.992 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:48.992 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.992 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.992 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.992 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.992 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.992 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.992 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.993 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.993 INFO:teuthology.orchestra.run.smithi070.stdout: "stale_cache_strategy": { 2024-06-04T05:23:48.993 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "stale_cache_strategy", 2024-06-04T05:23:48.993 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.993 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.993 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:48.993 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "log", 2024-06-04T05:23:48.993 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.993 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.993 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.993 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.993 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.993 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.993 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.994 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.994 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_behaviour": { 2024-06-04T05:23:48.994 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_behaviour", 2024-06-04T05:23:48.994 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.994 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.994 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.994 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "default", 2024-06-04T05:23:48.994 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.994 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.994 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.994 INFO:teuthology.orchestra.run.smithi070.stdout: "default", 2024-06-04T05:23:48.994 INFO:teuthology.orchestra.run.smithi070.stdout: "error" 2024-06-04T05:23:48.994 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.995 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.995 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.995 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.995 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.995 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.995 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_error_status_code": { 2024-06-04T05:23:48.995 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_error_status_code", 2024-06-04T05:23:48.995 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:48.995 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.995 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.995 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "500", 2024-06-04T05:23:48.995 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "400", 2024-06-04T05:23:48.995 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "599", 2024-06-04T05:23:48.995 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_support", 2024-06-04T05:23:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:48.997 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:48.997 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.997 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.997 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.997 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:48.997 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.997 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.997 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:48.997 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:48.997 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:48.997 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:48.997 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:48.997 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:48.998 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:48.998 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:48.998 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.998 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.998 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.998 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.998 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.998 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:48.998 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:48.998 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:48.998 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.998 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:48.998 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:48.999 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:48.999 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:48.999 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:48.999 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:48.999 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:48.999 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:48.999 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:48.999 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:48.999 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:48.999 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:48.999 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:48.999 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:48.999 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.000 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.000 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.000 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.000 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.000 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.000 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.000 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.000 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.000 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.000 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.000 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.000 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.000 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.000 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.001 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.001 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.001 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.001 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.001 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.001 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.001 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.001 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.001 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.001 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.001 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.001 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.001 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.001 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.002 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.002 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.002 INFO:teuthology.orchestra.run.smithi070.stdout: "max_concurrent_snap_create": { 2024-06-04T05:23:49.002 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_concurrent_snap_create", 2024-06-04T05:23:49.002 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.002 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.002 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.002 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:49.002 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.002 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.002 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.002 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.002 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.003 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.003 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.003 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.003 INFO:teuthology.orchestra.run.smithi070.stdout: "mirror_snapshot_schedule": { 2024-06-04T05:23:49.003 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mirror_snapshot_schedule", 2024-06-04T05:23:49.003 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.003 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.003 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.003 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.003 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.003 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.003 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.003 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.003 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.004 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.004 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.004 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.004 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.004 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.004 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.004 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.004 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.004 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.004 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.004 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.004 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.004 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.004 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.005 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.005 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.005 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.005 INFO:teuthology.orchestra.run.smithi070.stdout: "trash_purge_schedule": { 2024-06-04T05:23:49.005 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "trash_purge_schedule", 2024-06-04T05:23:49.005 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.005 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.005 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.005 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.005 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.005 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.005 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.005 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.005 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.006 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.006 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.006 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.006 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.006 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.006 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.006 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "restful", 2024-06-04T05:23:49.006 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.006 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.006 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.006 INFO:teuthology.orchestra.run.smithi070.stdout: "enable_auth": { 2024-06-04T05:23:49.006 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enable_auth", 2024-06-04T05:23:49.006 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.006 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.007 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.007 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.007 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.007 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.007 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.007 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.007 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.007 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.007 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.007 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.007 INFO:teuthology.orchestra.run.smithi070.stdout: "key_file": { 2024-06-04T05:23:49.007 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "key_file", 2024-06-04T05:23:49.007 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.007 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.008 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.008 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.008 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.008 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.008 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.008 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.008 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.008 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.008 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.008 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.008 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.008 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.008 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.008 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.008 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.009 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.009 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.009 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.009 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.009 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.009 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.009 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.009 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.009 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.009 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.009 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.009 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.009 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.010 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.010 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.010 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.010 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.010 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.010 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.010 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.010 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.010 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.010 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.010 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.010 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.010 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.011 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.011 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.011 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.011 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.011 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.011 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.011 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.011 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.011 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.011 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.011 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.011 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.011 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.011 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.012 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.012 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.012 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.012 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.012 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.012 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.012 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.012 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.012 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.012 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.012 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.012 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.012 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.012 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.013 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.013 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.013 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.013 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.013 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.013 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.013 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.013 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.013 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.013 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.013 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.013 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-06-04T05:23:49.013 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-06-04T05:23:49.013 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.014 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.014 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.014 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.014 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.014 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.014 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.014 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.014 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.014 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.014 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.014 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.014 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-06-04T05:23:49.014 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-06-04T05:23:49.015 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.015 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.015 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.015 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.015 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.015 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.015 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.015 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.015 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.015 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.015 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.015 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.015 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.015 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.016 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.016 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.016 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.016 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.016 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.016 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.016 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.016 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.016 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.016 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.016 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.016 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.016 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.016 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.017 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.017 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rgw", 2024-06-04T05:23:49.017 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.017 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.017 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.017 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.017 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.017 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.017 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.017 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.017 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.017 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.017 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.017 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.018 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.018 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.018 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.018 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.018 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.018 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.018 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.018 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.018 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.018 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.018 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.018 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.018 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.018 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.019 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.019 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.019 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.019 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.019 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.019 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.019 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.019 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.019 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.019 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.019 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.019 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.019 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.019 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.020 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.020 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.020 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.020 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.020 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.020 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.020 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.020 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.020 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.020 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.020 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.020 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.020 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.020 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.021 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.021 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.021 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.021 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.021 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.021 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.021 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.021 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.021 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.021 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.021 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.021 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.021 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.021 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.022 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.022 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.022 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.022 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.022 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.022 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.022 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.022 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.022 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.022 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.022 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.022 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.022 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.022 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.023 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.023 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.023 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.023 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.023 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.023 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.023 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.023 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.023 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rook", 2024-06-04T05:23:49.023 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.023 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.023 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.023 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.024 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.024 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.024 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.024 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.024 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.024 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.024 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.024 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.024 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.024 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.024 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.024 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.024 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.024 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.024 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.025 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.025 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.025 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.025 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.025 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.025 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.025 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.025 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.025 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.025 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.025 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.025 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.025 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.026 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.026 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.026 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.026 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.026 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.026 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.026 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.026 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.026 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.026 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.026 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.026 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.026 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.027 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.027 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.027 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.027 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.027 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.027 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.027 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.027 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.027 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.027 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.027 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.027 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.027 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.027 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.028 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.028 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.028 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.028 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.028 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.028 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.028 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.028 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.028 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.028 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.028 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.028 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.028 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.028 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.029 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.029 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.029 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.029 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.029 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.029 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.029 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.029 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.029 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.029 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.029 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.029 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.029 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.029 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.030 INFO:teuthology.orchestra.run.smithi070.stdout: "storage_class": { 2024-06-04T05:23:49.030 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "storage_class", 2024-06-04T05:23:49.030 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.030 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.030 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.030 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "local", 2024-06-04T05:23:49.030 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.030 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.030 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.030 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-06-04T05:23:49.030 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.030 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.030 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.031 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.031 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.031 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.031 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.031 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "selftest", 2024-06-04T05:23:49.031 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.031 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.031 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.031 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.031 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.031 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.031 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.031 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.031 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.032 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.032 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.032 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.032 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.032 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.032 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.032 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.032 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.032 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.032 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.032 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.032 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.032 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.032 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.033 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.033 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.033 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.033 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.033 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.033 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.033 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.033 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.033 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.033 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.033 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.033 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.033 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.034 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.034 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.034 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.034 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.034 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.034 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.034 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.034 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.034 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.034 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.034 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.034 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.034 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.035 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.035 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.035 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.035 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.035 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.035 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.035 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.035 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.035 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.035 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.035 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.035 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.035 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.036 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.036 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.036 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.036 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.036 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.036 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.036 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.036 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.036 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.036 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.036 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.036 INFO:teuthology.orchestra.run.smithi070.stdout: "roption1": { 2024-06-04T05:23:49.036 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "roption1", 2024-06-04T05:23:49.036 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.037 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.037 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.037 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.037 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.037 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.037 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.037 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.037 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.037 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.037 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.037 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.037 INFO:teuthology.orchestra.run.smithi070.stdout: "roption2": { 2024-06-04T05:23:49.037 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "roption2", 2024-06-04T05:23:49.037 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.038 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.038 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.038 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "xyz", 2024-06-04T05:23:49.038 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.038 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.038 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.038 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.038 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.038 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.038 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.038 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.038 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption1": { 2024-06-04T05:23:49.038 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption1", 2024-06-04T05:23:49.039 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.039 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.039 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.039 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.039 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.039 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.039 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.039 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.039 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.039 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.039 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.039 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.039 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption2": { 2024-06-04T05:23:49.039 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption2", 2024-06-04T05:23:49.040 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.040 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.040 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.040 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.040 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.040 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.040 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.040 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.040 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.040 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.040 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.040 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.040 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption3": { 2024-06-04T05:23:49.041 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption3", 2024-06-04T05:23:49.041 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:49.041 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.041 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.041 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.041 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.041 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.041 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.041 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.041 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.041 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.041 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.041 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.042 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption4": { 2024-06-04T05:23:49.042 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption4", 2024-06-04T05:23:49.042 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.042 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.042 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.042 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.042 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.042 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.042 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.042 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.042 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.042 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.042 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.042 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.043 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption5": { 2024-06-04T05:23:49.043 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption5", 2024-06-04T05:23:49.043 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.043 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.043 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.043 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.043 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.043 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.043 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.043 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.043 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.043 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.043 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.043 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.044 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption6": { 2024-06-04T05:23:49.044 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption6", 2024-06-04T05:23:49.044 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.044 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.044 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.044 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.044 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.044 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.044 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.044 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.044 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.044 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.044 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.044 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.045 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption7": { 2024-06-04T05:23:49.045 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption7", 2024-06-04T05:23:49.045 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.045 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.045 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.045 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.045 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-06-04T05:23:49.045 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "42", 2024-06-04T05:23:49.045 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.045 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.045 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.045 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.045 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.045 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.046 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.046 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.046 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.046 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.046 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.046 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.046 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.046 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.046 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.046 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.046 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.046 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.046 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.047 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.047 INFO:teuthology.orchestra.run.smithi070.stdout: "testkey": { 2024-06-04T05:23:49.047 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testkey", 2024-06-04T05:23:49.047 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.047 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.047 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.047 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.047 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.047 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.047 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.047 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.047 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.047 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.048 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.048 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.048 INFO:teuthology.orchestra.run.smithi070.stdout: "testlkey": { 2024-06-04T05:23:49.048 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testlkey", 2024-06-04T05:23:49.048 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.048 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.048 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.048 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.048 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.048 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.048 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.048 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.048 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.048 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.049 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.049 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.049 INFO:teuthology.orchestra.run.smithi070.stdout: "testnewline": { 2024-06-04T05:23:49.049 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testnewline", 2024-06-04T05:23:49.049 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.049 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.049 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.049 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.049 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.049 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.049 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.049 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.049 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.049 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.049 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.050 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.050 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.050 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.050 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.050 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smb", 2024-06-04T05:23:49.050 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.050 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.050 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.050 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.050 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.050 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.050 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.050 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.051 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.051 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.051 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.051 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.051 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.051 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.051 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.051 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.051 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.051 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.051 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.051 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.051 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.051 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.051 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.052 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.052 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.052 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.052 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.052 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.052 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.052 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.052 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.052 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.052 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.052 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.052 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.052 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.052 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.053 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.053 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.053 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.053 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.053 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.053 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.053 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.053 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.053 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.053 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.053 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.053 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.053 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.054 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.054 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.054 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.054 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.054 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.054 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.054 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.054 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.054 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.054 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.054 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.054 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.054 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.054 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.055 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.055 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.055 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.055 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.055 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.055 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.055 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.055 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.055 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.055 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.055 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.055 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.055 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.056 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.056 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.056 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.056 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.056 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.056 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.056 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.056 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.056 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.056 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.056 INFO:teuthology.orchestra.run.smithi070.stdout: "update_orchestration": { 2024-06-04T05:23:49.056 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "update_orchestration", 2024-06-04T05:23:49.056 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.056 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.057 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.057 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.057 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.057 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.057 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.057 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-06-04T05:23:49.057 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.057 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.057 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.057 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.057 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.057 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.057 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.057 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snap_schedule", 2024-06-04T05:23:49.058 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.058 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.058 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.058 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_m_granularity": { 2024-06-04T05:23:49.058 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_m_granularity", 2024-06-04T05:23:49.058 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.058 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.058 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.058 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.058 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.058 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.058 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.058 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow minute scheduled snapshots", 2024-06-04T05:23:49.059 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.059 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.059 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.059 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.059 INFO:teuthology.orchestra.run.smithi070.stdout: "dump_on_update": { 2024-06-04T05:23:49.059 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "dump_on_update", 2024-06-04T05:23:49.059 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.059 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.059 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.059 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.059 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.059 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.059 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.059 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "dump database to debug log on update", 2024-06-04T05:23:49.060 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.060 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.060 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.060 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.060 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.060 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.060 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.060 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.060 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.060 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.060 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.060 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.060 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.060 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.061 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.061 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.061 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.061 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.061 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.061 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.061 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.061 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.061 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.061 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.061 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.061 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.061 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.061 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.061 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.062 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.062 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.062 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.062 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.062 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.062 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.062 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.062 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.062 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.062 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.062 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.062 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.062 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.062 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.063 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.063 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.063 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.063 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.063 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.063 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.063 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.063 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.063 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.063 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.063 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.063 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.063 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.063 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.064 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.064 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.064 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.064 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.064 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.064 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.064 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.064 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.064 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.064 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.064 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.064 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.064 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.064 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.065 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.065 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.065 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.065 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.065 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.065 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.065 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.065 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.065 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.065 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.065 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.065 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.065 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.065 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.066 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.066 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.066 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.066 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.066 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.066 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.066 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "stats", 2024-06-04T05:23:49.066 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.066 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.066 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.066 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.066 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.066 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.066 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.067 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.067 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.067 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.067 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.067 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.067 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.067 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.067 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.067 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.067 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.067 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.067 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.067 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.068 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.068 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.068 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.068 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.068 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.068 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.068 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.068 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.068 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.068 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.068 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.068 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.068 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.069 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.069 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.069 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.069 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.069 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.069 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.069 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.069 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.069 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.069 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.069 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.069 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.069 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.069 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.070 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.070 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.070 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.070 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.070 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.070 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.070 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.070 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.070 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.070 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.070 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.070 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.070 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.071 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.071 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.071 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.071 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.071 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.071 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.071 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.071 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.071 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.071 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.071 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.071 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.071 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.071 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.072 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.072 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.072 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.072 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.072 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.072 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.072 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.072 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.072 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.072 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.072 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.072 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.072 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.073 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.073 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.073 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.073 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "status", 2024-06-04T05:23:49.073 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.073 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.073 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.073 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.073 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.073 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.073 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.073 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.073 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.074 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.074 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.074 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.074 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.074 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.074 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.074 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.074 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.074 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.074 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.074 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.074 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.074 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.074 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.075 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.075 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.075 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.075 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.075 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.075 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.075 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.075 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.075 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.075 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.075 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.075 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.075 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.075 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.076 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.076 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.076 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.076 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.076 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.076 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.076 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.076 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.076 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.076 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.076 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.076 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.076 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.076 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.077 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.077 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.077 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.077 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.077 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.077 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.077 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.077 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.077 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.077 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.077 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.077 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.077 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.077 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.078 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.078 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.078 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.078 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.078 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.078 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.078 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.078 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.078 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.078 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.078 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.078 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.078 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.078 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.079 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.079 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.079 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.079 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.079 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.079 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.079 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.079 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.079 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.079 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.079 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.079 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "telegraf", 2024-06-04T05:23:49.079 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.079 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.080 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.080 INFO:teuthology.orchestra.run.smithi070.stdout: "address": { 2024-06-04T05:23:49.080 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "address", 2024-06-04T05:23:49.080 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.080 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.080 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.080 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-06-04T05:23:49.080 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.080 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.080 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.080 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.080 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.080 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.080 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.081 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.081 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-06-04T05:23:49.081 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-06-04T05:23:49.081 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.081 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.081 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.081 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15", 2024-06-04T05:23:49.081 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.081 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.081 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.081 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.081 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.081 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.084 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.084 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.084 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.084 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.084 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.084 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.084 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.084 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.084 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.084 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.084 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.084 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.084 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.084 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.085 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.085 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.085 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.085 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.085 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.085 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.085 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.085 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.085 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.085 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.085 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.085 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.085 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.085 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.086 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.086 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.086 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.086 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.086 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.086 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.086 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.086 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.086 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.086 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.086 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.086 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.086 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.087 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.087 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.087 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.087 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.087 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.087 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.087 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.087 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.087 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.087 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.087 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.087 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.087 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.087 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.087 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.088 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.088 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.088 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.088 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.088 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.088 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "telemetry", 2024-06-04T05:23:49.088 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.088 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.088 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.088 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_basic": { 2024-06-04T05:23:49.088 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_basic", 2024-06-04T05:23:49.088 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.088 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.088 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.089 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.089 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.089 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.089 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.089 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share basic cluster information (size, version)", 2024-06-04T05:23:49.089 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.089 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.089 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.089 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.089 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_crash": { 2024-06-04T05:23:49.089 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_crash", 2024-06-04T05:23:49.089 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.089 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.089 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.090 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.090 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.090 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.090 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.090 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-06-04T05:23:49.090 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.090 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.090 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.090 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.090 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_device": { 2024-06-04T05:23:49.090 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_device", 2024-06-04T05:23:49.090 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.090 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.091 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.091 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.091 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.091 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.091 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.091 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-06-04T05:23:49.091 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.091 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.091 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.091 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.091 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_ident": { 2024-06-04T05:23:49.091 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_ident", 2024-06-04T05:23:49.091 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.091 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.092 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.092 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.092 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.092 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.092 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.092 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-06-04T05:23:49.092 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.092 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.092 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.092 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.092 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_perf": { 2024-06-04T05:23:49.092 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_perf", 2024-06-04T05:23:49.092 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.092 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.093 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.093 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.093 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.093 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.093 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.093 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share various performance metrics of a cluster", 2024-06-04T05:23:49.093 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.093 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.093 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.093 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.093 INFO:teuthology.orchestra.run.smithi070.stdout: "contact": { 2024-06-04T05:23:49.093 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "contact", 2024-06-04T05:23:49.093 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.093 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.094 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.094 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.094 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.094 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.094 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.094 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.094 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.094 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.094 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.094 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.094 INFO:teuthology.orchestra.run.smithi070.stdout: "description": { 2024-06-04T05:23:49.094 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "description", 2024-06-04T05:23:49.095 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.095 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.095 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.095 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.095 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.095 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.095 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.095 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.095 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.095 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.095 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.095 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.095 INFO:teuthology.orchestra.run.smithi070.stdout: "device_url": { 2024-06-04T05:23:49.096 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_url", 2024-06-04T05:23:49.096 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.096 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.096 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.096 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-06-04T05:23:49.096 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.096 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.096 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.096 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.096 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.096 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.096 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.096 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.096 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": { 2024-06-04T05:23:49.096 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enabled", 2024-06-04T05:23:49.097 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.097 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.097 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.097 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.097 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.097 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.097 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.097 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.097 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.097 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.097 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.097 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.097 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-06-04T05:23:49.098 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-06-04T05:23:49.098 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.098 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.098 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.098 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "24", 2024-06-04T05:23:49.098 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "8", 2024-06-04T05:23:49.098 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.098 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.098 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.098 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.098 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.098 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.098 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.099 INFO:teuthology.orchestra.run.smithi070.stdout: "last_opt_revision": { 2024-06-04T05:23:49.099 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "last_opt_revision", 2024-06-04T05:23:49.099 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.099 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.099 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.099 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1", 2024-06-04T05:23:49.099 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.099 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.099 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.099 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.099 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.099 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.099 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.099 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.100 INFO:teuthology.orchestra.run.smithi070.stdout: "leaderboard": { 2024-06-04T05:23:49.100 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "leaderboard", 2024-06-04T05:23:49.100 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.100 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.100 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.100 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.100 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.100 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.100 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.100 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.100 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.100 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.100 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.100 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.100 INFO:teuthology.orchestra.run.smithi070.stdout: "leaderboard_description": { 2024-06-04T05:23:49.101 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "leaderboard_description", 2024-06-04T05:23:49.101 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.101 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.101 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.101 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.101 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.101 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.101 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.101 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.101 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.101 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.101 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.101 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.102 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.102 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.102 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.102 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.102 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.102 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.102 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.102 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.102 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.102 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.102 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.102 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.102 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.102 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.103 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.103 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.103 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.103 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.103 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.103 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.103 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.103 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.103 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.103 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.103 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.103 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.103 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.103 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.104 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.104 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.104 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.104 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.104 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.104 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.104 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.104 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.104 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.104 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.104 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.104 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.104 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.104 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.104 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.105 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.105 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.105 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.105 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.105 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.105 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.105 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.105 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.105 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.105 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.105 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.105 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.105 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.105 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.106 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.106 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.106 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.106 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.106 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.106 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.106 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.106 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.106 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.106 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.106 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.106 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.106 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.106 INFO:teuthology.orchestra.run.smithi070.stdout: "organization": { 2024-06-04T05:23:49.107 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "organization", 2024-06-04T05:23:49.107 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.107 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.107 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.107 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.107 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.107 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.107 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.107 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.107 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.107 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.107 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.107 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.107 INFO:teuthology.orchestra.run.smithi070.stdout: "proxy": { 2024-06-04T05:23:49.108 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "proxy", 2024-06-04T05:23:49.108 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.108 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.108 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.108 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.108 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.108 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.108 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.108 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.108 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.108 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.108 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.108 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.109 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.109 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.109 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.109 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.109 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.109 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.109 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.109 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.109 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.109 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.109 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.109 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.110 INFO:teuthology.orchestra.run.smithi070.stdout: "url": { 2024-06-04T05:23:49.110 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "url", 2024-06-04T05:23:49.110 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.110 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.110 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.110 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-06-04T05:23:49.110 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.110 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.110 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.110 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.110 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.110 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.110 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.110 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.111 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.111 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.111 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.111 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "test_orchestrator", 2024-06-04T05:23:49.111 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.111 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.111 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.111 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.111 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.111 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.111 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.111 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.111 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.111 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.112 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.112 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.112 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.112 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.112 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.112 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.112 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.112 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.112 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.112 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.112 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.112 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.112 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.112 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.113 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.113 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.113 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.113 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.113 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.113 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.113 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.113 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.113 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.113 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.113 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.113 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.113 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.114 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.114 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.114 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.114 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.114 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.114 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.114 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.114 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.114 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.114 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.114 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.114 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.114 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.114 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.116 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.116 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.116 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.116 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.116 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.116 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.116 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.116 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.116 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.116 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.116 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.116 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.116 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.117 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.117 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.117 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.117 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.117 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.117 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.117 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.117 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.117 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.117 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.117 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.117 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "volumes", 2024-06-04T05:23:49.117 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.117 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.118 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.118 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.118 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.118 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.118 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.118 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.118 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.118 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.118 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.118 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.118 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.118 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.118 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.118 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.119 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.119 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.119 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.119 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.119 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.119 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.119 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.119 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.119 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.119 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.119 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.119 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.119 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.120 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.120 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.120 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.120 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.120 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.120 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.120 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.120 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.120 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.122 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.122 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.122 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.122 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.122 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.122 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.122 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.122 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.122 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.122 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.122 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.122 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.123 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.123 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.123 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.123 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.123 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.123 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.123 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.123 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.123 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.123 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.123 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.123 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.123 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.124 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.124 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.124 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.124 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.124 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.124 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.124 INFO:teuthology.orchestra.run.smithi070.stdout: "max_concurrent_clones": { 2024-06-04T05:23:49.124 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_concurrent_clones", 2024-06-04T05:23:49.124 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.124 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.124 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.125 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "4", 2024-06-04T05:23:49.125 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.125 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.125 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.125 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Number of asynchronous cloner threads", 2024-06-04T05:23:49.125 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.125 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.125 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.125 INFO:teuthology.orchestra.run.smithi070.stdout: "periodic_async_work": { 2024-06-04T05:23:49.125 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "periodic_async_work", 2024-06-04T05:23:49.125 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.125 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.125 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.126 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.126 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.126 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.126 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.126 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Periodically check for async work", 2024-06-04T05:23:49.126 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.126 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.126 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.126 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.126 INFO:teuthology.orchestra.run.smithi070.stdout: "snapshot_clone_delay": { 2024-06-04T05:23:49.126 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snapshot_clone_delay", 2024-06-04T05:23:49.126 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.126 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.127 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.127 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.127 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.127 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.127 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.127 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-06-04T05:23:49.127 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.127 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.127 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.127 INFO:teuthology.orchestra.run.smithi070.stdout: "snapshot_clone_no_wait": { 2024-06-04T05:23:49.127 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snapshot_clone_no_wait", 2024-06-04T05:23:49.127 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.127 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.127 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.128 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.128 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.128 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.128 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.128 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-06-04T05:23:49.128 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.128 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.128 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.128 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.128 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.128 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.128 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.128 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.129 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.129 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.129 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.129 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.129 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.129 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.129 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.129 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.129 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.129 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.129 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.129 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.129 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix", 2024-06-04T05:23:49.129 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.129 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.130 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.130 INFO:teuthology.orchestra.run.smithi070.stdout: "discovery_interval": { 2024-06-04T05:23:49.130 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "discovery_interval", 2024-06-04T05:23:49.130 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-06-04T05:23:49.130 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.130 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.130 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "100", 2024-06-04T05:23:49.130 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.130 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.130 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.130 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.130 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.130 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.131 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.131 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.131 INFO:teuthology.orchestra.run.smithi070.stdout: "identifier": { 2024-06-04T05:23:49.131 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "identifier", 2024-06-04T05:23:49.131 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.131 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.131 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.131 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.131 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.131 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.131 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.131 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.131 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.131 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.131 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.132 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-06-04T05:23:49.132 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-06-04T05:23:49.132 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.132 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.132 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.132 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-06-04T05:23:49.132 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.132 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.132 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.132 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.132 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.132 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.132 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.133 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.133 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.133 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.133 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.133 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.133 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.133 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.133 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.133 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.133 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.133 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.133 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.133 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.134 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.134 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.134 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.134 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.134 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.134 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.134 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.134 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.134 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.134 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.134 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.134 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.134 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.135 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.135 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.135 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.135 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.135 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.135 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.135 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.135 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.135 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.135 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.135 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.135 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.135 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.135 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.136 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.136 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.136 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.136 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.136 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.136 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.136 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.136 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.136 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.136 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.136 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.136 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.136 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.136 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.137 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.137 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.137 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.137 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.137 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.137 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.137 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.137 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.137 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.137 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.137 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.137 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.137 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.138 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.138 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.138 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.138 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.138 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.138 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.138 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.138 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.138 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.138 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.138 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.139 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.139 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.139 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.139 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_host": { 2024-06-04T05:23:49.139 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_host", 2024-06-04T05:23:49.139 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.139 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.139 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.139 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.139 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.139 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.139 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.139 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.140 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.140 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.140 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.140 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_port": { 2024-06-04T05:23:49.140 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_port", 2024-06-04T05:23:49.140 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.140 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.140 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.140 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10051", 2024-06-04T05:23:49.140 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.140 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.140 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.140 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.141 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.141 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.141 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.141 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.141 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_sender": { 2024-06-04T05:23:49.141 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_sender", 2024-06-04T05:23:49.141 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.141 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.141 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.141 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-06-04T05:23:49.141 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.141 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.141 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.141 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.142 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.142 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.142 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.142 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.142 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.142 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.142 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-06-04T05:23:49.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.142 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.142 INFO:teuthology.orchestra.run.smithi070.stdout: "gid": 24803, 2024-06-04T05:23:49.142 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "y", 2024-06-04T05:23:49.142 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_features": 4540701547738038271, 2024-06-04T05:23:49.142 INFO:teuthology.orchestra.run.smithi070.stdout: "available_modules": [ 2024-06-04T05:23:49.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.143 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "alerts", 2024-06-04T05:23:49.143 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.143 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.143 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.143 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-06-04T05:23:49.143 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-06-04T05:23:49.143 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.143 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.143 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.143 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-06-04T05:23:49.143 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.143 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.143 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.144 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How frequently to reexamine health status", 2024-06-04T05:23:49.144 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.144 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.144 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.144 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.144 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.144 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.144 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.144 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.144 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.144 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.144 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.144 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.145 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.145 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.145 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.145 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.145 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.145 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.145 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.145 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.145 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.145 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.145 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.145 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.145 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.146 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.146 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.146 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.146 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.146 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.146 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.146 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.146 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.146 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.146 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.146 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.146 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.146 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.147 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.147 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.147 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.147 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.147 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.147 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.147 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.147 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.147 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.147 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.147 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.147 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.147 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.147 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.147 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.148 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.148 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.148 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.148 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.148 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.148 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.148 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.148 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.148 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.148 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.148 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.148 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.148 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.148 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.149 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.149 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.149 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.149 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_destination": { 2024-06-04T05:23:49.149 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_destination", 2024-06-04T05:23:49.149 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.149 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.149 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.149 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.149 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.149 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.149 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.150 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Email address to send alerts to", 2024-06-04T05:23:49.150 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.150 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.150 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.150 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_from_name": { 2024-06-04T05:23:49.150 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_from_name", 2024-06-04T05:23:49.150 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.150 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.150 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.150 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "Ceph", 2024-06-04T05:23:49.150 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.150 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.150 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.151 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Email From: name", 2024-06-04T05:23:49.151 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.151 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.151 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.151 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.151 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_host": { 2024-06-04T05:23:49.151 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_host", 2024-06-04T05:23:49.151 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.151 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.151 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.151 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.151 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.151 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.151 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.152 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP server", 2024-06-04T05:23:49.152 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.152 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.152 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.152 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.152 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_password": { 2024-06-04T05:23:49.152 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_password", 2024-06-04T05:23:49.152 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.152 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.152 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.152 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.152 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.153 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.153 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.153 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Password to authenticate with", 2024-06-04T05:23:49.153 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.153 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.153 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.153 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_port": { 2024-06-04T05:23:49.153 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_port", 2024-06-04T05:23:49.153 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.153 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.153 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.153 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "465", 2024-06-04T05:23:49.153 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.154 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.154 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.154 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP port", 2024-06-04T05:23:49.154 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.154 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.154 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.154 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.154 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_sender": { 2024-06-04T05:23:49.154 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_sender", 2024-06-04T05:23:49.154 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.154 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.154 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.154 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.155 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.155 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.155 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.155 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP envelope sender", 2024-06-04T05:23:49.155 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.155 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.155 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.155 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.155 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_ssl": { 2024-06-04T05:23:49.155 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_ssl", 2024-06-04T05:23:49.155 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.155 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.155 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.155 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.156 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.156 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.156 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.156 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use SSL to connect to SMTP server", 2024-06-04T05:23:49.156 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.156 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.156 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.156 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.156 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_user": { 2024-06-04T05:23:49.156 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_user", 2024-06-04T05:23:49.156 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.156 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.156 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.156 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.157 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.157 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.157 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.157 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "User to authenticate as", 2024-06-04T05:23:49.157 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.157 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.157 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.157 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.157 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.157 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.157 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.157 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.157 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.157 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.158 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.158 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.158 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.158 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.158 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.158 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.158 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.158 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.158 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.158 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "balancer", 2024-06-04T05:23:49.158 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.159 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.159 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.159 INFO:teuthology.orchestra.run.smithi070.stdout: "active": { 2024-06-04T05:23:49.159 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "active", 2024-06-04T05:23:49.159 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.159 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.159 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.159 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.159 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.159 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.159 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.159 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "automatically balance PGs across cluster", 2024-06-04T05:23:49.159 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.159 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.159 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.160 INFO:teuthology.orchestra.run.smithi070.stdout: "begin_time": { 2024-06-04T05:23:49.160 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "begin_time", 2024-06-04T05:23:49.160 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.160 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.160 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.160 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0000", 2024-06-04T05:23:49.160 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.160 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.160 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.160 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "beginning time of day to automatically balance", 2024-06-04T05:23:49.160 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-04T05:23:49.160 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.161 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.161 INFO:teuthology.orchestra.run.smithi070.stdout: "begin_weekday": { 2024-06-04T05:23:49.161 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "begin_weekday", 2024-06-04T05:23:49.161 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-06-04T05:23:49.161 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.161 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.161 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.161 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0", 2024-06-04T05:23:49.161 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "6", 2024-06-04T05:23:49.161 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.161 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-06-04T05:23:49.161 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-04T05:23:49.161 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.162 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.162 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_max_iterations": { 2024-06-04T05:23:49.162 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_max_iterations", 2024-06-04T05:23:49.162 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-06-04T05:23:49.162 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.162 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.162 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "25", 2024-06-04T05:23:49.162 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-06-04T05:23:49.162 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "250", 2024-06-04T05:23:49.162 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.162 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-06-04T05:23:49.162 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.162 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.163 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.163 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.163 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_metrics": { 2024-06-04T05:23:49.163 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_metrics", 2024-06-04T05:23:49.163 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.163 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.163 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.163 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "pgs,objects,bytes", 2024-06-04T05:23:49.163 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.163 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.163 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.163 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-06-04T05:23:49.163 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-06-04T05:23:49.163 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.164 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.164 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_step": { 2024-06-04T05:23:49.164 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_step", 2024-06-04T05:23:49.164 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:49.164 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.164 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.164 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0.5", 2024-06-04T05:23:49.164 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0.001", 2024-06-04T05:23:49.164 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "0.999", 2024-06-04T05:23:49.164 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.164 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "aggressiveness of optimization", 2024-06-04T05:23:49.164 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-06-04T05:23:49.164 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.165 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.165 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.165 INFO:teuthology.orchestra.run.smithi070.stdout: "end_time": { 2024-06-04T05:23:49.165 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "end_time", 2024-06-04T05:23:49.165 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.165 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.165 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.165 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "2359", 2024-06-04T05:23:49.165 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.165 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.165 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.165 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "ending time of day to automatically balance", 2024-06-04T05:23:49.165 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-04T05:23:49.166 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.166 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.166 INFO:teuthology.orchestra.run.smithi070.stdout: "end_weekday": { 2024-06-04T05:23:49.166 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "end_weekday", 2024-06-04T05:23:49.166 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-06-04T05:23:49.166 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.166 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.166 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.166 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0", 2024-06-04T05:23:49.166 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "6", 2024-06-04T05:23:49.166 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.166 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-06-04T05:23:49.167 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-04T05:23:49.167 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.167 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.167 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.167 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.167 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.167 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.167 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.167 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.167 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.167 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.167 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.167 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.168 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.168 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.168 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.168 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.168 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.168 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.168 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.168 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.168 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.168 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.168 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.168 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.168 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.169 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.169 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.169 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.169 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.169 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.169 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.169 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.169 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.169 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.169 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.169 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.169 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.169 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.169 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.170 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.170 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.170 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.170 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.170 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.170 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.170 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.170 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.170 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.170 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.170 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.170 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.170 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.170 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.171 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.171 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.171 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.171 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.171 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.171 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.171 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.171 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.171 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.171 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.171 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.171 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.171 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.172 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.172 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.172 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.172 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.172 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.172 INFO:teuthology.orchestra.run.smithi070.stdout: "min_score": { 2024-06-04T05:23:49.172 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "min_score", 2024-06-04T05:23:49.172 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:49.172 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.172 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.172 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.172 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.172 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.172 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.173 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-06-04T05:23:49.173 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.173 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.173 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.173 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.173 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": { 2024-06-04T05:23:49.173 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mode", 2024-06-04T05:23:49.173 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.173 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.173 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.173 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "upmap", 2024-06-04T05:23:49.173 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.173 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.173 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.174 INFO:teuthology.orchestra.run.smithi070.stdout: "crush-compat", 2024-06-04T05:23:49.174 INFO:teuthology.orchestra.run.smithi070.stdout: "none", 2024-06-04T05:23:49.174 INFO:teuthology.orchestra.run.smithi070.stdout: "read", 2024-06-04T05:23:49.174 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap", 2024-06-04T05:23:49.174 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap-read" 2024-06-04T05:23:49.174 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.174 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Balancer mode", 2024-06-04T05:23:49.174 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.174 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.174 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.174 INFO:teuthology.orchestra.run.smithi070.stdout: "pool_ids": { 2024-06-04T05:23:49.174 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pool_ids", 2024-06-04T05:23:49.175 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.175 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.175 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.175 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.175 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.175 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.175 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.175 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-06-04T05:23:49.175 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.175 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.175 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.175 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.175 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-06-04T05:23:49.175 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-06-04T05:23:49.175 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.176 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.176 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.176 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-06-04T05:23:49.176 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.176 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.176 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.176 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-06-04T05:23:49.176 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.176 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.176 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.176 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.176 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.177 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.177 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.177 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.177 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.177 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.177 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.177 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.177 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.177 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.177 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.177 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.177 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap_max_deviation": { 2024-06-04T05:23:49.177 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "upmap_max_deviation", 2024-06-04T05:23:49.178 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.178 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.178 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.178 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-06-04T05:23:49.178 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-06-04T05:23:49.178 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.178 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.178 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "deviation below which no optimization is attempted", 2024-06-04T05:23:49.178 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-06-04T05:23:49.178 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.178 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.178 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap_max_optimizations": { 2024-06-04T05:23:49.179 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "upmap_max_optimizations", 2024-06-04T05:23:49.179 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-06-04T05:23:49.179 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.179 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.179 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:49.179 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.179 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.179 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.179 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-06-04T05:23:49.179 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.179 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.179 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.179 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.179 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.180 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cephadm", 2024-06-04T05:23:49.180 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.180 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.180 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.180 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_down_multiplier": { 2024-06-04T05:23:49.180 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_down_multiplier", 2024-06-04T05:23:49.180 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:49.180 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.180 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.180 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3.0", 2024-06-04T05:23:49.180 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.180 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.181 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.181 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-06-04T05:23:49.181 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.181 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.181 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.181 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_refresh_rate": { 2024-06-04T05:23:49.181 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_refresh_rate", 2024-06-04T05:23:49.181 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.181 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.181 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.181 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "20", 2024-06-04T05:23:49.181 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.181 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.181 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.182 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-06-04T05:23:49.182 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.182 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.182 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.182 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_starting_port": { 2024-06-04T05:23:49.182 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_starting_port", 2024-06-04T05:23:49.182 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.182 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.182 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.182 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "4721", 2024-06-04T05:23:49.182 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.182 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-06-04T05:23:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_ptrace": { 2024-06-04T05:23:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_ptrace", 2024-06-04T05:23:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-06-04T05:23:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-06-04T05:23:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "autotune_interval": { 2024-06-04T05:23:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "autotune_interval", 2024-06-04T05:23:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-06-04T05:23:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to autotune daemon memory", 2024-06-04T05:23:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "autotune_memory_target_ratio": { 2024-06-04T05:23:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "autotune_memory_target_ratio", 2024-06-04T05:23:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0.7", 2024-06-04T05:23:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-06-04T05:23:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm_log_destination": { 2024-06-04T05:23:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cephadm_log_destination", 2024-06-04T05:23:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "file", 2024-06-04T05:23:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "file,syslog", 2024-06-04T05:23:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "syslog" 2024-06-04T05:23:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-06-04T05:23:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "cgroups_split": { 2024-06-04T05:23:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cgroups_split", 2024-06-04T05:23:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.188 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.188 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.188 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.188 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.188 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-06-04T05:23:49.188 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.188 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.188 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.188 INFO:teuthology.orchestra.run.smithi070.stdout: "config_checks_enabled": { 2024-06-04T05:23:49.188 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_checks_enabled", 2024-06-04T05:23:49.188 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.188 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.188 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.189 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.189 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.189 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.189 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.189 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-06-04T05:23:49.189 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.189 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.189 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.189 INFO:teuthology.orchestra.run.smithi070.stdout: "config_dashboard": { 2024-06-04T05:23:49.189 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_dashboard", 2024-06-04T05:23:49.189 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.189 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.190 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.190 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.190 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.190 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.190 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.190 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-06-04T05:23:49.190 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.190 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.190 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.190 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_alertmanager": { 2024-06-04T05:23:49.190 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_alertmanager", 2024-06-04T05:23:49.190 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.191 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.191 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.191 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-06-04T05:23:49.191 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.191 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.191 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.191 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-06-04T05:23:49.191 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.191 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.191 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.191 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_base": { 2024-06-04T05:23:49.191 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_base", 2024-06-04T05:23:49.191 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.192 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.192 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.192 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/ceph", 2024-06-04T05:23:49.192 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.192 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.192 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.192 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Container image name, without the tag", 2024-06-04T05:23:49.192 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.192 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.192 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.192 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_elasticsearch": { 2024-06-04T05:23:49.192 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_elasticsearch", 2024-06-04T05:23:49.193 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.193 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.193 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.193 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-06-04T05:23:49.193 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.193 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.193 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.193 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "elasticsearch container image", 2024-06-04T05:23:49.193 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.193 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.193 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.193 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_grafana": { 2024-06-04T05:23:49.193 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_grafana", 2024-06-04T05:23:49.194 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.194 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.194 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.194 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-06-04T05:23:49.194 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.194 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.194 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.194 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-06-04T05:23:49.194 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.194 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.194 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.194 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_haproxy": { 2024-06-04T05:23:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_haproxy", 2024-06-04T05:23:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-06-04T05:23:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "HAproxy container image", 2024-06-04T05:23:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_agent": { 2024-06-04T05:23:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_agent", 2024-06-04T05:23:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-06-04T05:23:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger agent container image", 2024-06-04T05:23:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_collector": { 2024-06-04T05:23:49.197 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_collector", 2024-06-04T05:23:49.197 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.197 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.197 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.197 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-06-04T05:23:49.197 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.197 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.197 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.197 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger collector container image", 2024-06-04T05:23:49.197 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.197 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.197 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.197 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.197 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_query": { 2024-06-04T05:23:49.198 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_query", 2024-06-04T05:23:49.198 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.198 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.198 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.198 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-06-04T05:23:49.198 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.198 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.198 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.198 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger query container image", 2024-06-04T05:23:49.198 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.198 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.198 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.199 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_keepalived": { 2024-06-04T05:23:49.199 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_keepalived", 2024-06-04T05:23:49.199 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.199 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.199 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.199 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-06-04T05:23:49.199 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.199 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.199 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.199 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Keepalived container image", 2024-06-04T05:23:49.199 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.199 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.199 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.199 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_loki": { 2024-06-04T05:23:49.200 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_loki", 2024-06-04T05:23:49.200 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.200 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.200 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.200 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-06-04T05:23:49.200 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.200 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.200 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.200 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Loki container image", 2024-06-04T05:23:49.200 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.200 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.200 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.201 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_node_exporter": { 2024-06-04T05:23:49.201 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_node_exporter", 2024-06-04T05:23:49.201 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.201 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.201 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.201 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-06-04T05:23:49.201 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.201 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.201 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.201 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-06-04T05:23:49.201 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.201 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.201 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.201 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.202 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_nvmeof": { 2024-06-04T05:23:49.202 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_nvmeof", 2024-06-04T05:23:49.202 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.202 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.202 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.202 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-06-04T05:23:49.202 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.202 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.202 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.202 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Nvme-of container image", 2024-06-04T05:23:49.202 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.202 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.202 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.202 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.203 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_prometheus": { 2024-06-04T05:23:49.203 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_prometheus", 2024-06-04T05:23:49.203 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.203 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.203 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.203 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-06-04T05:23:49.203 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.203 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.203 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.203 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-06-04T05:23:49.203 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.203 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.204 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.204 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_promtail": { 2024-06-04T05:23:49.204 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_promtail", 2024-06-04T05:23:49.204 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.204 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.204 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.204 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-06-04T05:23:49.204 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.204 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.204 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.204 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Promtail container image", 2024-06-04T05:23:49.204 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.205 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.205 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.205 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.205 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_samba": { 2024-06-04T05:23:49.205 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_samba", 2024-06-04T05:23:49.205 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.205 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.205 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.205 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-06-04T05:23:49.205 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.205 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.205 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.205 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Samba/SMB container image", 2024-06-04T05:23:49.205 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.206 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.206 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.206 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.206 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_snmp_gateway": { 2024-06-04T05:23:49.206 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_snmp_gateway", 2024-06-04T05:23:49.206 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.206 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.206 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.206 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-06-04T05:23:49.206 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.206 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.206 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.206 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SNMP Gateway container image", 2024-06-04T05:23:49.207 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.207 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.207 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.207 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.207 INFO:teuthology.orchestra.run.smithi070.stdout: "container_init": { 2024-06-04T05:23:49.207 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_init", 2024-06-04T05:23:49.207 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.207 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.207 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.207 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.207 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.207 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.207 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.207 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Run podman/docker with `--init`", 2024-06-04T05:23:49.208 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.208 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.208 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.208 INFO:teuthology.orchestra.run.smithi070.stdout: "daemon_cache_timeout": { 2024-06-04T05:23:49.208 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "daemon_cache_timeout", 2024-06-04T05:23:49.208 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.208 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.208 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.208 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-06-04T05:23:49.208 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.208 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.208 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.208 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-06-04T05:23:49.209 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.209 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.209 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.209 INFO:teuthology.orchestra.run.smithi070.stdout: "default_cephadm_command_timeout": { 2024-06-04T05:23:49.209 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "default_cephadm_command_timeout", 2024-06-04T05:23:49.209 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.209 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.209 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.209 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "900", 2024-06-04T05:23:49.209 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.209 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.209 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.209 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-06-04T05:23:49.210 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.210 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.210 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.210 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.210 INFO:teuthology.orchestra.run.smithi070.stdout: "default_registry": { 2024-06-04T05:23:49.210 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "default_registry", 2024-06-04T05:23:49.210 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.210 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.210 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.210 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io", 2024-06-04T05:23:49.210 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.210 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.210 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.210 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-06-04T05:23:49.211 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.211 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.211 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.211 INFO:teuthology.orchestra.run.smithi070.stdout: "device_cache_timeout": { 2024-06-04T05:23:49.211 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_cache_timeout", 2024-06-04T05:23:49.211 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.211 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.211 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.211 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1800", 2024-06-04T05:23:49.211 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.211 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.211 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.211 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache device inventory", 2024-06-04T05:23:49.212 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.212 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.212 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.212 INFO:teuthology.orchestra.run.smithi070.stdout: "device_enhanced_scan": { 2024-06-04T05:23:49.212 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_enhanced_scan", 2024-06-04T05:23:49.212 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.212 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.212 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.212 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.212 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.212 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.212 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.213 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use libstoragemgmt during device scans", 2024-06-04T05:23:49.213 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.213 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.213 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.213 INFO:teuthology.orchestra.run.smithi070.stdout: "facts_cache_timeout": { 2024-06-04T05:23:49.213 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "facts_cache_timeout", 2024-06-04T05:23:49.213 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.213 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.213 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.213 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-06-04T05:23:49.213 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.213 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.213 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.214 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache host facts data", 2024-06-04T05:23:49.214 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.214 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.214 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.214 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.214 INFO:teuthology.orchestra.run.smithi070.stdout: "grafana_dashboards_path": { 2024-06-04T05:23:49.214 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "grafana_dashboards_path", 2024-06-04T05:23:49.214 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.214 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.214 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.214 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-06-04T05:23:49.214 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.214 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.214 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.215 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-06-04T05:23:49.215 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.215 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.215 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.215 INFO:teuthology.orchestra.run.smithi070.stdout: "host_check_interval": { 2024-06-04T05:23:49.215 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "host_check_interval", 2024-06-04T05:23:49.215 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.215 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.215 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.215 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-06-04T05:23:49.215 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.215 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.216 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.216 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to perform a host check", 2024-06-04T05:23:49.216 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.216 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.216 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.216 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.216 INFO:teuthology.orchestra.run.smithi070.stdout: "hw_monitoring": { 2024-06-04T05:23:49.216 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "hw_monitoring", 2024-06-04T05:23:49.216 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.216 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.216 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.216 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.216 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.217 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.217 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.217 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-06-04T05:23:49.217 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.217 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.217 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.217 INFO:teuthology.orchestra.run.smithi070.stdout: "inventory_list_all": { 2024-06-04T05:23:49.217 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "inventory_list_all", 2024-06-04T05:23:49.217 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.217 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.217 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.217 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.217 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.217 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.218 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.218 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-06-04T05:23:49.218 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.218 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.218 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.218 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.218 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.218 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.218 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.218 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.218 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.218 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.218 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.219 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.219 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.219 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.219 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.219 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.219 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.219 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.219 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.219 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.219 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.219 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.219 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.219 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.220 INFO:teuthology.orchestra.run.smithi070.stdout: "log_refresh_metadata": { 2024-06-04T05:23:49.220 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_refresh_metadata", 2024-06-04T05:23:49.220 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.220 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.220 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.220 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.220 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.220 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.220 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.220 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-06-04T05:23:49.220 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.220 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.220 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.221 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.221 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.221 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.221 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.221 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.221 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.221 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.221 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.221 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.221 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-06-04T05:23:49.221 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.221 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.221 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.221 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.222 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.222 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.222 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.222 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.222 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.222 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.222 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.222 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.222 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.222 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.222 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.222 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.222 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.222 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.223 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.223 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.223 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.223 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.223 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.223 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.223 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.223 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.223 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.223 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.223 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.223 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.223 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.223 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.224 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.224 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.224 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.224 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.224 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.224 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.224 INFO:teuthology.orchestra.run.smithi070.stdout: "manage_etc_ceph_ceph_conf": { 2024-06-04T05:23:49.224 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-06-04T05:23:49.224 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.224 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.224 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.224 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.224 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.225 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.225 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.225 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-06-04T05:23:49.225 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.225 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.225 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.225 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.225 INFO:teuthology.orchestra.run.smithi070.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-06-04T05:23:49.225 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-06-04T05:23:49.225 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.225 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.225 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.225 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "*", 2024-06-04T05:23:49.225 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.226 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.226 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.226 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-06-04T05:23:49.226 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.226 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.226 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.226 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.226 INFO:teuthology.orchestra.run.smithi070.stdout: "max_count_per_host": { 2024-06-04T05:23:49.226 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_count_per_host", 2024-06-04T05:23:49.226 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.226 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.226 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.226 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:49.227 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.227 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.227 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.227 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "max number of daemons per service per host", 2024-06-04T05:23:49.227 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.227 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.227 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.227 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.227 INFO:teuthology.orchestra.run.smithi070.stdout: "max_osd_draining_count": { 2024-06-04T05:23:49.227 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_osd_draining_count", 2024-06-04T05:23:49.227 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.227 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.227 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.227 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:49.227 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.228 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.228 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.228 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-06-04T05:23:49.228 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.228 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.228 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.228 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.228 INFO:teuthology.orchestra.run.smithi070.stdout: "migration_current": { 2024-06-04T05:23:49.228 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "migration_current", 2024-06-04T05:23:49.228 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.228 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.228 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.229 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.229 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.229 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.229 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.229 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "internal - do not modify", 2024-06-04T05:23:49.229 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.229 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.229 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.229 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.229 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": { 2024-06-04T05:23:49.229 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mode", 2024-06-04T05:23:49.229 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.229 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.230 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.230 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "root", 2024-06-04T05:23:49.230 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.230 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.230 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.230 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm-package", 2024-06-04T05:23:49.230 INFO:teuthology.orchestra.run.smithi070.stdout: "root" 2024-06-04T05:23:49.230 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.230 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "mode for remote execution of cephadm", 2024-06-04T05:23:49.230 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.230 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.230 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.230 INFO:teuthology.orchestra.run.smithi070.stdout: "oob_default_addr": { 2024-06-04T05:23:49.231 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "oob_default_addr", 2024-06-04T05:23:49.231 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.231 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.231 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.231 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "169.254.1.1", 2024-06-04T05:23:49.231 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.231 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.231 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.231 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Default address for RedFish API (oob management).", 2024-06-04T05:23:49.231 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.231 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.231 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.231 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.231 INFO:teuthology.orchestra.run.smithi070.stdout: "prometheus_alerts_path": { 2024-06-04T05:23:49.232 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prometheus_alerts_path", 2024-06-04T05:23:49.232 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.232 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.232 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.232 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-06-04T05:23:49.232 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.232 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.232 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.232 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-06-04T05:23:49.232 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.232 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.232 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.233 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_insecure": { 2024-06-04T05:23:49.233 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_insecure", 2024-06-04T05:23:49.233 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.233 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.233 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.233 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.233 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.233 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.233 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.233 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-06-04T05:23:49.233 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.233 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.233 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.233 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_password": { 2024-06-04T05:23:49.234 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_password", 2024-06-04T05:23:49.234 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.234 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.234 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.234 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.234 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.234 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.234 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.234 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-06-04T05:23:49.234 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.234 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.234 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.234 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_url": { 2024-06-04T05:23:49.235 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_url", 2024-06-04T05:23:49.235 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.235 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.235 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.235 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.235 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.235 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.235 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.235 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-06-04T05:23:49.235 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.235 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.235 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.235 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_username": { 2024-06-04T05:23:49.236 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_username", 2024-06-04T05:23:49.236 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.236 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.236 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.236 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.236 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.236 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.236 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.236 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-06-04T05:23:49.236 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.236 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.236 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.237 INFO:teuthology.orchestra.run.smithi070.stdout: "secure_monitoring_stack": { 2024-06-04T05:23:49.237 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "secure_monitoring_stack", 2024-06-04T05:23:49.237 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.237 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.237 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.237 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.237 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.237 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.237 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.237 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-06-04T05:23:49.237 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.237 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.237 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.237 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.237 INFO:teuthology.orchestra.run.smithi070.stdout: "service_discovery_port": { 2024-06-04T05:23:49.238 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "service_discovery_port", 2024-06-04T05:23:49.238 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.238 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.238 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.238 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8765", 2024-06-04T05:23:49.238 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.238 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.238 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.238 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "cephadm service discovery port", 2024-06-04T05:23:49.238 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.238 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.238 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.238 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.239 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.239 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.239 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.239 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.239 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.239 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.239 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.239 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.239 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.239 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.239 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.239 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.239 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.239 INFO:teuthology.orchestra.run.smithi070.stdout: "ssh_config_file": { 2024-06-04T05:23:49.239 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssh_config_file", 2024-06-04T05:23:49.240 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.240 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.240 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.240 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.240 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.240 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.240 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.240 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-06-04T05:23:49.240 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.240 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.240 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.240 INFO:teuthology.orchestra.run.smithi070.stdout: "use_agent": { 2024-06-04T05:23:49.240 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "use_agent", 2024-06-04T05:23:49.241 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.241 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.241 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.241 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.241 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.241 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.241 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.241 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-06-04T05:23:49.241 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.241 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.241 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.241 INFO:teuthology.orchestra.run.smithi070.stdout: "use_repo_digest": { 2024-06-04T05:23:49.241 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "use_repo_digest", 2024-06-04T05:23:49.241 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.242 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.242 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.242 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.242 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.242 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.242 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.242 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-06-04T05:23:49.242 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.242 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.242 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.242 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.242 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_failed_host_check": { 2024-06-04T05:23:49.242 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_failed_host_check", 2024-06-04T05:23:49.243 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.243 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.243 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.243 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.243 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.243 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.243 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.243 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if the host check fails", 2024-06-04T05:23:49.243 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.243 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.243 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.243 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_stray_daemons": { 2024-06-04T05:23:49.243 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_stray_daemons", 2024-06-04T05:23:49.244 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.244 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.244 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.244 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.244 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.244 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.244 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.244 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-06-04T05:23:49.244 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.244 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.244 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.244 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_stray_hosts": { 2024-06-04T05:23:49.244 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_stray_hosts", 2024-06-04T05:23:49.245 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.245 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.245 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.245 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.245 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.245 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.245 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.245 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-06-04T05:23:49.245 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.245 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.245 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.245 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.245 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.246 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.246 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crash", 2024-06-04T05:23:49.246 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.246 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.246 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.246 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.246 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.246 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.246 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.246 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.246 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.246 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.247 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.247 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.247 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.247 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.247 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.247 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.247 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.247 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.247 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.247 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.247 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.247 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.247 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.248 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.248 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.248 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.248 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.248 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.248 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.248 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.248 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.248 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.248 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.248 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.248 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.248 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.248 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.249 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.249 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.249 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.249 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.249 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.249 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.249 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.249 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.249 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.249 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.249 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.249 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.249 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.249 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.250 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.250 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.250 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.250 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.250 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.250 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.250 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.250 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.250 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.250 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.250 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.250 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.251 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.251 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.251 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.251 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.251 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.251 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.251 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.251 INFO:teuthology.orchestra.run.smithi070.stdout: "retain_interval": { 2024-06-04T05:23:49.251 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "retain_interval", 2024-06-04T05:23:49.251 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.251 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.251 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.251 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "31536000", 2024-06-04T05:23:49.251 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.252 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.252 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.252 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long to retain crashes before pruning them", 2024-06-04T05:23:49.252 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.252 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.252 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.252 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.252 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.252 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.252 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.252 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.252 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.252 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.252 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.253 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.253 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.253 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.253 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.253 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.253 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.253 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_recent_interval": { 2024-06-04T05:23:49.253 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_recent_interval", 2024-06-04T05:23:49.253 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.253 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.253 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.253 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1209600", 2024-06-04T05:23:49.254 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.254 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.254 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.254 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "time interval in which to warn about recent crashes", 2024-06-04T05:23:49.254 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.254 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.254 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.254 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.254 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.254 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.254 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "dashboard", 2024-06-04T05:23:49.254 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.254 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.255 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.255 INFO:teuthology.orchestra.run.smithi070.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-06-04T05:23:49.255 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-06-04T05:23:49.255 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.255 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.255 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.255 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:49.255 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.255 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.255 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.255 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.255 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.255 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.255 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.256 INFO:teuthology.orchestra.run.smithi070.stdout: "ALERTMANAGER_API_HOST": { 2024-06-04T05:23:49.256 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ALERTMANAGER_API_HOST", 2024-06-04T05:23:49.256 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.256 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.256 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.256 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.256 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.256 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.256 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.256 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.256 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.256 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.256 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.257 INFO:teuthology.orchestra.run.smithi070.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-06-04T05:23:49.257 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-06-04T05:23:49.257 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.257 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.257 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.257 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.257 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.257 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.257 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.257 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.257 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.257 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.258 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.258 INFO:teuthology.orchestra.run.smithi070.stdout: "AUDIT_API_ENABLED": { 2024-06-04T05:23:49.258 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "AUDIT_API_ENABLED", 2024-06-04T05:23:49.258 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.258 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.258 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.258 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.258 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.258 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.258 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.258 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.258 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.259 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.259 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.259 INFO:teuthology.orchestra.run.smithi070.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-06-04T05:23:49.259 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-06-04T05:23:49.259 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.259 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.259 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.259 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.259 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.259 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.259 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.259 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.260 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.260 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.260 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.260 INFO:teuthology.orchestra.run.smithi070.stdout: "ENABLE_BROWSABLE_API": { 2024-06-04T05:23:49.260 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ENABLE_BROWSABLE_API", 2024-06-04T05:23:49.260 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.260 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.260 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.260 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.260 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.260 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.260 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.260 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.260 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.261 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.261 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.261 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-06-04T05:23:49.261 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-06-04T05:23:49.261 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.261 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.261 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.261 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.261 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.261 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.261 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.261 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.261 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.262 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.262 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.262 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-06-04T05:23:49.262 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-06-04T05:23:49.262 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.262 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.262 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.262 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.262 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.262 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.262 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.262 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.263 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.263 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.263 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.263 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-06-04T05:23:49.263 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-06-04T05:23:49.263 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.263 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.263 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.263 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.263 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.263 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.263 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.263 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.264 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.264 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.264 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.264 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-06-04T05:23:49.264 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-06-04T05:23:49.264 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.264 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.264 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.264 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.264 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.264 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.264 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.264 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.265 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.265 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.265 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.265 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_NFS": { 2024-06-04T05:23:49.265 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-06-04T05:23:49.265 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.265 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.265 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.265 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.265 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.265 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.265 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.265 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.266 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.266 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.266 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.266 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.266 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_RBD": { 2024-06-04T05:23:49.266 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-06-04T05:23:49.266 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.266 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.266 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.266 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.266 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.266 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.266 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.267 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.267 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.267 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.267 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.267 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.267 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_RGW": { 2024-06-04T05:23:49.267 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-06-04T05:23:49.267 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.267 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.267 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.267 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.267 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.267 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.267 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.268 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.268 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.268 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.268 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.268 INFO:teuthology.orchestra.run.smithi070.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-06-04T05:23:49.268 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-06-04T05:23:49.268 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.268 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.268 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.268 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.268 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.268 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.268 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.269 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.269 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.269 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.269 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.269 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.269 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_PASSWORD": { 2024-06-04T05:23:49.269 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_PASSWORD", 2024-06-04T05:23:49.269 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.269 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.269 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.269 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-06-04T05:23:49.269 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.269 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.270 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.270 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.270 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.270 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.270 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.270 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.270 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-06-04T05:23:49.270 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-06-04T05:23:49.270 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.270 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.270 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.270 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.270 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.271 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.271 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.271 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.271 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.271 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.271 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.271 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.271 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_URL": { 2024-06-04T05:23:49.271 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_URL", 2024-06-04T05:23:49.271 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.271 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.271 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.271 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.272 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.272 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.272 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.272 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.272 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.272 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.272 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.272 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_USERNAME": { 2024-06-04T05:23:49.272 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_USERNAME", 2024-06-04T05:23:49.272 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.272 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.273 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.273 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-06-04T05:23:49.273 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.273 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.273 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.273 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.273 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.273 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.273 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.273 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.273 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-06-04T05:23:49.273 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-06-04T05:23:49.273 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.274 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.274 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.274 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.274 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.274 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.274 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.274 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.274 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.274 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.274 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.274 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.274 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-06-04T05:23:49.274 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-06-04T05:23:49.274 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.275 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.275 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.275 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.275 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.275 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.275 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.275 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.275 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.275 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.275 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.275 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.275 INFO:teuthology.orchestra.run.smithi070.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-06-04T05:23:49.275 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-06-04T05:23:49.275 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.276 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.276 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.276 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.276 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.276 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.276 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.276 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.276 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.276 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.276 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.276 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.276 INFO:teuthology.orchestra.run.smithi070.stdout: "ISSUE_TRACKER_API_KEY": { 2024-06-04T05:23:49.276 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-06-04T05:23:49.276 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.277 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.277 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.277 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.277 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.277 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.277 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.277 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.277 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.277 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.277 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.277 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.277 INFO:teuthology.orchestra.run.smithi070.stdout: "MANAGED_BY_CLUSTERS": { 2024-06-04T05:23:49.277 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-06-04T05:23:49.278 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.278 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.278 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.278 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "[]", 2024-06-04T05:23:49.278 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.278 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.278 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.278 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.278 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.278 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.278 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.278 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.278 INFO:teuthology.orchestra.run.smithi070.stdout: "MULTICLUSTER_CONFIG": { 2024-06-04T05:23:49.279 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "MULTICLUSTER_CONFIG", 2024-06-04T05:23:49.279 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.279 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.279 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.279 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "{}", 2024-06-04T05:23:49.279 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.279 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.279 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.279 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.279 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.279 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.279 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.279 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.279 INFO:teuthology.orchestra.run.smithi070.stdout: "PROMETHEUS_API_HOST": { 2024-06-04T05:23:49.280 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PROMETHEUS_API_HOST", 2024-06-04T05:23:49.280 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.280 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.280 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.280 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.280 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.280 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.280 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.280 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.280 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.280 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.280 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.280 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.281 INFO:teuthology.orchestra.run.smithi070.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-06-04T05:23:49.281 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-06-04T05:23:49.281 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.281 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.281 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.281 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.281 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.281 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.281 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.281 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.281 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.281 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.281 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.282 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-06-04T05:23:49.282 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-06-04T05:23:49.282 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.282 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.282 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.282 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.282 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.282 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.282 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.282 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.282 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.282 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.283 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.283 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.283 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-06-04T05:23:49.283 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-06-04T05:23:49.283 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.283 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.283 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.283 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.283 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.283 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.283 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.283 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.283 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.284 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.284 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.284 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.284 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-06-04T05:23:49.284 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-06-04T05:23:49.284 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.284 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.284 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.284 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.284 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.284 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.284 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.284 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.284 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.285 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.285 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.285 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-06-04T05:23:49.285 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-06-04T05:23:49.285 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.285 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.285 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.285 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.285 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.285 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.285 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.285 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.286 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.286 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.286 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.286 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.286 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-06-04T05:23:49.286 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-06-04T05:23:49.286 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.286 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.286 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.286 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.286 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.286 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.286 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.287 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.287 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.287 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.287 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.287 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.287 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-06-04T05:23:49.287 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-06-04T05:23:49.287 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.287 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.287 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.287 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.287 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.287 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.288 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.288 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.288 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.288 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.288 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.288 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.288 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-06-04T05:23:49.288 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-06-04T05:23:49.288 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.288 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.288 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.288 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.288 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.288 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.289 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.289 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.289 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.289 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.289 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.289 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_ENABLED": { 2024-06-04T05:23:49.289 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_ENABLED", 2024-06-04T05:23:49.289 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.289 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.289 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.289 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.289 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.289 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.290 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.290 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.290 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.290 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.290 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.290 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-06-04T05:23:49.290 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-06-04T05:23:49.290 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.290 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.290 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.290 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-06-04T05:23:49.290 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.291 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.291 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.291 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.291 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.291 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.291 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.291 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.291 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-06-04T05:23:49.291 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-06-04T05:23:49.291 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.291 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.291 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.291 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:49.291 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.292 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.292 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.292 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.292 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.292 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.292 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.292 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.292 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-06-04T05:23:49.292 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-06-04T05:23:49.292 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.292 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.292 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.292 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8", 2024-06-04T05:23:49.292 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.293 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.293 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.293 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.293 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.293 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.293 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.293 INFO:teuthology.orchestra.run.smithi070.stdout: "REST_REQUESTS_TIMEOUT": { 2024-06-04T05:23:49.293 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-06-04T05:23:49.293 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.293 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.293 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.293 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "45", 2024-06-04T05:23:49.293 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.294 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.294 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.294 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.294 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.294 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.294 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.294 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_ACCESS_KEY": { 2024-06-04T05:23:49.294 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_ACCESS_KEY", 2024-06-04T05:23:49.294 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.294 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.294 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.294 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.294 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.295 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.295 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.295 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.295 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.295 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.295 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.295 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.295 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-06-04T05:23:49.295 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-06-04T05:23:49.295 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.295 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.295 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.296 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-06-04T05:23:49.296 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.296 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.296 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.296 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.296 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.296 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.296 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.296 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.296 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_SECRET_KEY": { 2024-06-04T05:23:49.296 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_SECRET_KEY", 2024-06-04T05:23:49.296 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.296 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.296 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.297 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.297 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.297 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.297 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.297 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.297 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.297 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.297 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.297 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_SSL_VERIFY": { 2024-06-04T05:23:49.297 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_SSL_VERIFY", 2024-06-04T05:23:49.297 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.297 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.298 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.298 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.298 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.298 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.298 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.298 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.298 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.298 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.298 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.298 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.298 INFO:teuthology.orchestra.run.smithi070.stdout: "UNSAFE_TLS_v1_2": { 2024-06-04T05:23:49.298 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "UNSAFE_TLS_v1_2", 2024-06-04T05:23:49.298 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.299 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.299 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.299 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.299 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.299 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.299 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.299 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.299 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.299 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.299 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.299 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.299 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-06-04T05:23:49.299 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-06-04T05:23:49.299 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.300 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.300 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.300 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.300 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.300 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.300 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.300 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.300 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.300 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.300 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.300 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.300 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-06-04T05:23:49.300 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-06-04T05:23:49.300 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.300 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.301 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.301 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:49.301 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.301 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.301 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.301 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.301 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.301 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.301 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.301 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.301 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-06-04T05:23:49.301 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-06-04T05:23:49.301 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.301 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.302 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.302 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-06-04T05:23:49.302 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.302 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.302 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.302 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.302 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.302 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.302 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.302 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.302 INFO:teuthology.orchestra.run.smithi070.stdout: "cross_origin_url": { 2024-06-04T05:23:49.302 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cross_origin_url", 2024-06-04T05:23:49.302 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.302 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.303 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.303 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.303 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.303 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.303 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.303 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.303 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.303 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.303 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.303 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.303 INFO:teuthology.orchestra.run.smithi070.stdout: "crt_file": { 2024-06-04T05:23:49.303 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crt_file", 2024-06-04T05:23:49.303 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.304 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.304 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.304 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.304 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.304 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.304 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.304 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.304 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.304 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.304 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.304 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.304 INFO:teuthology.orchestra.run.smithi070.stdout: "debug": { 2024-06-04T05:23:49.304 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "debug", 2024-06-04T05:23:49.304 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.305 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.305 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.305 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.305 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.305 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.305 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.305 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable/disable debug options", 2024-06-04T05:23:49.305 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.305 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.305 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.305 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.305 INFO:teuthology.orchestra.run.smithi070.stdout: "jwt_token_ttl": { 2024-06-04T05:23:49.305 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "jwt_token_ttl", 2024-06-04T05:23:49.305 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.306 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.306 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.306 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "28800", 2024-06-04T05:23:49.306 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.306 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.306 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.306 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.306 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.306 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.306 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.306 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.306 INFO:teuthology.orchestra.run.smithi070.stdout: "key_file": { 2024-06-04T05:23:49.306 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "key_file", 2024-06-04T05:23:49.307 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.307 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.307 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.307 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.307 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.307 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.307 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.307 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.307 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.307 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.307 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.307 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.307 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.307 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.308 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.308 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.308 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.308 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.308 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.308 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.308 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.308 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.308 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.308 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.308 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.308 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.308 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.309 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.309 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.309 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.309 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.309 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.309 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.309 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.309 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.309 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.309 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.309 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.309 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.309 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.309 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.309 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.310 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.310 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.310 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.310 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.310 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.310 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.310 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.310 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.310 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.310 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.310 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.310 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.310 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.311 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.311 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.311 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.311 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.311 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.311 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.311 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.311 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.311 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.311 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.311 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.311 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.311 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.311 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.312 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.312 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.312 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.312 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.312 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.312 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.312 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.312 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.312 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.312 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.312 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.312 INFO:teuthology.orchestra.run.smithi070.stdout: "motd": { 2024-06-04T05:23:49.312 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "motd", 2024-06-04T05:23:49.313 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.313 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.313 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.313 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.313 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.313 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.313 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.313 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "The message of the day", 2024-06-04T05:23:49.313 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.313 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.313 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.313 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.313 INFO:teuthology.orchestra.run.smithi070.stdout: "redirect_resolve_ip_addr": { 2024-06-04T05:23:49.313 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "redirect_resolve_ip_addr", 2024-06-04T05:23:49.314 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.314 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.314 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.314 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.314 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.314 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.314 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.314 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.314 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.314 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.314 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.315 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-06-04T05:23:49.315 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-06-04T05:23:49.315 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.315 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.315 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.315 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "::", 2024-06-04T05:23:49.315 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.315 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.315 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.315 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.315 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.315 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.315 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.315 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.315 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-06-04T05:23:49.316 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-06-04T05:23:49.316 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.316 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.316 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.316 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8080", 2024-06-04T05:23:49.316 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.316 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.316 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.316 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.316 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.316 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.316 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.316 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.316 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.317 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.317 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.317 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.317 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.317 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.317 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.317 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.317 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.317 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.317 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.317 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.317 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.317 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.318 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl": { 2024-06-04T05:23:49.318 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl", 2024-06-04T05:23:49.318 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.318 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.318 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.318 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.318 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.318 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.318 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.318 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.318 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.318 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.318 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.318 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.318 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl_server_port": { 2024-06-04T05:23:49.319 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl_server_port", 2024-06-04T05:23:49.319 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.319 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.319 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.319 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8443", 2024-06-04T05:23:49.319 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.319 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.319 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.319 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.319 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.319 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.319 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.319 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.319 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_behaviour": { 2024-06-04T05:23:49.319 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_behaviour", 2024-06-04T05:23:49.320 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.320 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.320 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.320 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "redirect", 2024-06-04T05:23:49.320 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.320 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.320 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.320 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.320 INFO:teuthology.orchestra.run.smithi070.stdout: "redirect" 2024-06-04T05:23:49.320 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.320 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.320 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.320 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.320 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.321 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_error_status_code": { 2024-06-04T05:23:49.321 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_error_status_code", 2024-06-04T05:23:49.321 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.321 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.321 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.321 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "500", 2024-06-04T05:23:49.321 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "400", 2024-06-04T05:23:49.321 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "599", 2024-06-04T05:23:49.321 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.321 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.321 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.321 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.321 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.322 INFO:teuthology.orchestra.run.smithi070.stdout: "url_prefix": { 2024-06-04T05:23:49.322 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "url_prefix", 2024-06-04T05:23:49.322 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.322 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.322 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.322 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.322 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.322 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.322 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.322 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.322 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.322 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.323 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.323 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.323 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.323 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "devicehealth", 2024-06-04T05:23:49.323 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.323 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.323 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.323 INFO:teuthology.orchestra.run.smithi070.stdout: "enable_monitoring": { 2024-06-04T05:23:49.323 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enable_monitoring", 2024-06-04T05:23:49.323 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.323 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.323 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.324 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.324 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.324 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.324 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.324 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "monitor device health metrics", 2024-06-04T05:23:49.324 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.324 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.324 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.324 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.324 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.324 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.324 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.324 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.325 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.325 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.325 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.325 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.325 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.325 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.325 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.325 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.325 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.325 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.325 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.325 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.325 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.325 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.325 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.326 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.326 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.326 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.326 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.326 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.326 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.326 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.326 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.326 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.326 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.326 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.326 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.326 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.327 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.327 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.327 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.327 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.327 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.327 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.327 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.327 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.327 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.327 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.327 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.327 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.328 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.328 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.328 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.328 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.328 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.328 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.328 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.328 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.328 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.328 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.328 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.328 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.328 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.329 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.329 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.329 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.329 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.329 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.329 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.329 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.329 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.329 INFO:teuthology.orchestra.run.smithi070.stdout: "mark_out_threshold": { 2024-06-04T05:23:49.329 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mark_out_threshold", 2024-06-04T05:23:49.329 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.329 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.329 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.330 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "2419200", 2024-06-04T05:23:49.330 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.330 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.330 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.330 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-06-04T05:23:49.330 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.330 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.330 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.330 INFO:teuthology.orchestra.run.smithi070.stdout: "pool_name": { 2024-06-04T05:23:49.330 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pool_name", 2024-06-04T05:23:49.330 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.330 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.331 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.331 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "device_health_metrics", 2024-06-04T05:23:49.331 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.331 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.331 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.331 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "name of pool in which to store device health metrics", 2024-06-04T05:23:49.331 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.331 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.331 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.331 INFO:teuthology.orchestra.run.smithi070.stdout: "retention_period": { 2024-06-04T05:23:49.331 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "retention_period", 2024-06-04T05:23:49.331 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.331 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.332 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.332 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15552000", 2024-06-04T05:23:49.332 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.332 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.332 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.332 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long to retain device health metrics", 2024-06-04T05:23:49.332 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.332 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.332 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.332 INFO:teuthology.orchestra.run.smithi070.stdout: "scrape_frequency": { 2024-06-04T05:23:49.332 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "scrape_frequency", 2024-06-04T05:23:49.332 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.332 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.332 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.333 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "86400", 2024-06-04T05:23:49.333 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.333 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.333 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.333 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to scrape device health metrics", 2024-06-04T05:23:49.333 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.333 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.333 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.333 INFO:teuthology.orchestra.run.smithi070.stdout: "self_heal": { 2024-06-04T05:23:49.333 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "self_heal", 2024-06-04T05:23:49.333 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.333 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.333 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.333 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.334 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.334 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.334 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.334 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-06-04T05:23:49.334 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.334 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.334 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.334 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-06-04T05:23:49.334 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-06-04T05:23:49.334 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.334 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.334 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.334 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-06-04T05:23:49.334 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.335 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.335 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.335 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to wake up and check device health", 2024-06-04T05:23:49.335 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.335 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.335 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.335 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.335 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.335 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.335 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.335 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.335 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.336 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.336 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.336 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.336 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.336 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.336 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.336 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.336 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_threshold": { 2024-06-04T05:23:49.336 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_threshold", 2024-06-04T05:23:49.336 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.336 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.336 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.336 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "7257600", 2024-06-04T05:23:49.336 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.337 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.337 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.337 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-06-04T05:23:49.337 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.337 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.337 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.337 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.337 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.337 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "diskprediction_local", 2024-06-04T05:23:49.337 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.337 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.337 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.337 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.338 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.338 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.338 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.338 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.338 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.338 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.338 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.338 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.338 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.338 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.338 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.338 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.338 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.338 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.338 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.339 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.339 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.339 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.339 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.339 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.339 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.339 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.339 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.339 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.339 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.339 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.339 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.339 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.340 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.340 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.340 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.340 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.340 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.340 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.340 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.340 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.340 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.340 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.340 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.340 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.341 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.341 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.341 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.341 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.341 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.341 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.341 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.341 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.341 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.341 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.341 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.341 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.341 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.342 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.342 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.342 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.342 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.342 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.342 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.342 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.342 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.342 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.342 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.342 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.342 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.342 INFO:teuthology.orchestra.run.smithi070.stdout: "predict_interval": { 2024-06-04T05:23:49.343 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "predict_interval", 2024-06-04T05:23:49.343 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.343 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.343 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.343 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "86400", 2024-06-04T05:23:49.343 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.343 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.343 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.343 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.343 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.343 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.343 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.343 INFO:teuthology.orchestra.run.smithi070.stdout: "predictor_model": { 2024-06-04T05:23:49.344 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "predictor_model", 2024-06-04T05:23:49.344 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.344 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.344 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.344 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "prophetstor", 2024-06-04T05:23:49.344 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.344 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.344 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.344 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.344 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.344 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.344 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.345 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-06-04T05:23:49.345 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-06-04T05:23:49.345 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.345 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.345 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.345 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-06-04T05:23:49.345 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.345 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.345 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.345 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.345 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.345 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.345 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.345 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.346 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.346 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.346 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.346 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.346 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.346 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.346 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.346 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.346 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.346 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.346 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.346 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.346 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.347 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.347 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "influx", 2024-06-04T05:23:49.347 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": false, 2024-06-04T05:23:49.347 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "influxdb python module not found", 2024-06-04T05:23:49.347 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.347 INFO:teuthology.orchestra.run.smithi070.stdout: "batch_size": { 2024-06-04T05:23:49.347 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "batch_size", 2024-06-04T05:23:49.347 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.347 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.347 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.347 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5000", 2024-06-04T05:23:49.347 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.348 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.348 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.348 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-06-04T05:23:49.348 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.348 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.348 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.348 INFO:teuthology.orchestra.run.smithi070.stdout: "database": { 2024-06-04T05:23:49.348 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "database", 2024-06-04T05:23:49.348 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.348 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.348 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.348 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "ceph", 2024-06-04T05:23:49.348 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.349 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.349 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.349 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-06-04T05:23:49.349 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.349 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.349 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.349 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": { 2024-06-04T05:23:49.349 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "hostname", 2024-06-04T05:23:49.349 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.349 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.349 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.350 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.350 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.350 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.350 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.350 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB server hostname", 2024-06-04T05:23:49.350 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.350 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.350 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.350 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-06-04T05:23:49.350 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-06-04T05:23:49.350 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.350 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.351 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.351 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "30", 2024-06-04T05:23:49.351 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "5", 2024-06-04T05:23:49.351 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.351 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.351 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-06-04T05:23:49.351 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.351 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.351 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.351 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.351 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.351 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.351 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.352 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.352 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.352 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.352 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.352 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.352 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.352 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.352 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.352 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.352 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.352 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.352 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.352 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.353 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.353 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.353 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.353 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.353 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.353 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.353 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.353 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.353 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.353 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.353 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.353 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.353 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.354 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.354 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.354 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.354 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.354 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.354 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.354 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.354 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.354 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.354 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.354 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.354 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.354 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.355 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.355 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.355 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.355 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.355 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.355 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.355 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.355 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.355 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.355 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.355 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.355 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.356 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.356 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.356 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.356 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.356 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.356 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.356 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.356 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.356 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.356 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.356 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.356 INFO:teuthology.orchestra.run.smithi070.stdout: "password": { 2024-06-04T05:23:49.356 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "password", 2024-06-04T05:23:49.357 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.357 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.357 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.357 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.357 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.357 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.357 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.357 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "password of InfluxDB server user", 2024-06-04T05:23:49.357 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.357 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.357 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.357 INFO:teuthology.orchestra.run.smithi070.stdout: "port": { 2024-06-04T05:23:49.358 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "port", 2024-06-04T05:23:49.358 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.358 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.358 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.358 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8086", 2024-06-04T05:23:49.358 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.358 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.358 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.358 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB server port", 2024-06-04T05:23:49.358 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.358 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.358 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.359 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.359 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.359 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.359 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.359 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.359 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.359 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.359 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.359 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.359 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.359 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.359 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.359 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.360 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl": { 2024-06-04T05:23:49.360 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl", 2024-06-04T05:23:49.360 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.360 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.360 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.360 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "false", 2024-06-04T05:23:49.360 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.360 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.360 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.360 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-06-04T05:23:49.360 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.360 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.360 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.361 INFO:teuthology.orchestra.run.smithi070.stdout: "threads": { 2024-06-04T05:23:49.361 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "threads", 2024-06-04T05:23:49.361 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.361 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.361 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.361 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-06-04T05:23:49.361 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-06-04T05:23:49.361 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "32", 2024-06-04T05:23:49.361 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.361 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-06-04T05:23:49.361 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.361 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.362 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.362 INFO:teuthology.orchestra.run.smithi070.stdout: "username": { 2024-06-04T05:23:49.362 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "username", 2024-06-04T05:23:49.362 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.362 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.362 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.362 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.362 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.362 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.362 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.362 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "username of InfluxDB server user", 2024-06-04T05:23:49.362 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.362 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.363 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.363 INFO:teuthology.orchestra.run.smithi070.stdout: "verify_ssl": { 2024-06-04T05:23:49.363 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "verify_ssl", 2024-06-04T05:23:49.363 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.363 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.363 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.363 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "true", 2024-06-04T05:23:49.363 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.363 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.363 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.363 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-06-04T05:23:49.364 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.364 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.364 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.364 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.364 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.364 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "insights", 2024-06-04T05:23:49.364 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.364 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.364 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.364 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.364 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.364 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.365 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.365 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.365 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.365 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.365 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.365 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.365 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.365 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.365 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.365 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.365 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.365 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.365 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.366 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.366 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.366 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.366 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.366 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.366 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.366 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.366 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.366 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.366 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.366 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.366 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.367 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.367 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.367 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.367 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.367 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.367 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.367 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.367 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.367 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.367 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.367 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.367 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.367 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.368 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.368 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.368 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.368 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.368 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.368 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.368 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.368 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.368 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.368 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.368 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.368 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.368 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.369 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.369 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.369 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.369 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.369 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.369 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.369 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.369 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.369 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.369 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.369 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.369 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.369 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.370 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.370 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.370 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.370 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.370 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.370 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.370 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.370 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.370 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.370 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.370 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.370 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.370 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.371 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.371 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "iostat", 2024-06-04T05:23:49.371 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.371 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.371 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.371 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.371 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.371 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.371 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.371 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.371 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.371 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.372 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.372 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.372 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.372 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.372 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.372 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.372 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.372 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.372 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.372 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.372 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.372 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.372 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.373 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.373 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.373 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.373 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.373 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.373 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.373 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.373 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.373 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.373 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.373 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.373 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.373 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.374 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.374 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.374 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.374 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.374 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.374 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.374 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.374 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.374 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.374 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.374 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.374 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.375 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.375 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.375 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.375 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.375 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.375 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.375 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.375 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.375 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.375 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.375 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.375 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.376 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.376 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.376 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.376 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.376 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.376 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.376 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.376 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.376 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.376 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.376 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.376 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.376 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.376 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.377 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.377 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.377 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.377 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.377 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.377 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.377 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.377 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.377 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.377 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.378 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "k8sevents", 2024-06-04T05:23:49.378 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.378 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.378 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.378 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_event_retention_days": { 2024-06-04T05:23:49.378 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ceph_event_retention_days", 2024-06-04T05:23:49.378 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.378 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.378 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.378 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "7", 2024-06-04T05:23:49.378 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.378 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.378 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.378 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Days to hold ceph event information within local cache", 2024-06-04T05:23:49.379 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.379 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.379 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.379 INFO:teuthology.orchestra.run.smithi070.stdout: "config_check_secs": { 2024-06-04T05:23:49.379 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_check_secs", 2024-06-04T05:23:49.379 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.379 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.379 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.379 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:49.379 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "10", 2024-06-04T05:23:49.379 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.380 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.380 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-06-04T05:23:49.380 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.380 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.380 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.380 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.380 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.380 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.380 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.380 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.380 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.380 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.380 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.380 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.381 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.381 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.381 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.381 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.381 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.381 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.381 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.381 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.381 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.381 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.381 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.381 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.381 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.381 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.382 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.382 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.382 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.382 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.382 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.382 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.382 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.382 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.382 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.382 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.382 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.382 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.383 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.383 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.383 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.383 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.383 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.383 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.383 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.383 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.383 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.383 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.383 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.383 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.383 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.383 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.384 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.384 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.384 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.384 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.384 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.384 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.384 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.384 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.384 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.384 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.384 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.384 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.384 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.385 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.385 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.385 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.385 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.385 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.385 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.385 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.385 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.385 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.385 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.385 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.385 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.385 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.386 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.386 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.386 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.386 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.386 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.386 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.386 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "localpool", 2024-06-04T05:23:49.386 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.386 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.386 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.386 INFO:teuthology.orchestra.run.smithi070.stdout: "failure_domain": { 2024-06-04T05:23:49.386 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "failure_domain", 2024-06-04T05:23:49.387 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.387 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.387 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.387 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "host", 2024-06-04T05:23:49.387 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.387 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.387 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.387 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "failure domain for any created local pool", 2024-06-04T05:23:49.387 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-06-04T05:23:49.387 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.387 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.387 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.388 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.388 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.388 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.388 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.388 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.388 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.388 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.388 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.388 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.388 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.388 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.388 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.388 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.388 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.389 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.389 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.389 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.389 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.389 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.389 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.389 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.389 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.389 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.389 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.389 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.389 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.389 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.390 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.390 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.390 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.390 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.390 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.390 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.390 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.390 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.390 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.390 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.390 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.390 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.390 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.391 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.391 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.391 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.391 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.391 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.391 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.391 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.391 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.391 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.391 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.391 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.391 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.392 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.392 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.392 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.392 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.392 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.392 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.392 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.392 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.392 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.392 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.392 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.392 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.392 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.393 INFO:teuthology.orchestra.run.smithi070.stdout: "min_size": { 2024-06-04T05:23:49.393 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "min_size", 2024-06-04T05:23:49.393 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.393 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.393 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.393 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.393 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.393 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.393 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.393 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default min_size for any created local pool", 2024-06-04T05:23:49.393 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-06-04T05:23:49.393 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.393 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.394 INFO:teuthology.orchestra.run.smithi070.stdout: "num_rep": { 2024-06-04T05:23:49.394 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "num_rep", 2024-06-04T05:23:49.394 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.394 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.394 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.394 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3", 2024-06-04T05:23:49.394 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.394 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.394 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.394 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default replica count for any created local pool", 2024-06-04T05:23:49.394 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.394 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.394 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.395 INFO:teuthology.orchestra.run.smithi070.stdout: "pg_num": { 2024-06-04T05:23:49.395 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pg_num", 2024-06-04T05:23:49.395 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.395 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.395 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.395 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "128", 2024-06-04T05:23:49.395 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.395 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.395 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.395 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default pg_num for any created local pool", 2024-06-04T05:23:49.395 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.395 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.395 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.396 INFO:teuthology.orchestra.run.smithi070.stdout: "prefix": { 2024-06-04T05:23:49.396 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prefix", 2024-06-04T05:23:49.396 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.396 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.396 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.396 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.396 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.396 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.396 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.396 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "name prefix for any created local pool", 2024-06-04T05:23:49.396 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.396 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.396 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.396 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.397 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.397 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.397 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.397 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.397 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.397 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.397 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.397 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.397 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.397 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.397 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.397 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.397 INFO:teuthology.orchestra.run.smithi070.stdout: "subtree": { 2024-06-04T05:23:49.397 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "subtree", 2024-06-04T05:23:49.398 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.398 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.398 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.398 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "rack", 2024-06-04T05:23:49.398 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.398 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.398 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.398 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "CRUSH level for which to create a local pool", 2024-06-04T05:23:49.398 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-06-04T05:23:49.398 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.398 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.398 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.398 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.399 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mds_autoscaler", 2024-06-04T05:23:49.399 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.399 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.399 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.399 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.399 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.399 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.399 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.399 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.399 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.399 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.399 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.399 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.399 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.400 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.400 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.400 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.400 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.400 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.400 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.400 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.400 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.400 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.400 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.400 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.400 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.400 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.401 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.401 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.401 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.401 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.401 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.401 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.401 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.401 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.401 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.401 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.401 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.401 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.401 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.402 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.402 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.402 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.402 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.402 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.402 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.402 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.402 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.402 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.402 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.402 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.402 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.403 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.403 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.403 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.403 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.403 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.403 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.403 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.403 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.403 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.403 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.403 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.403 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.403 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.404 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.404 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.404 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.404 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.404 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.404 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.404 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.404 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.404 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.404 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.404 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.404 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.404 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.405 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.405 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.405 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.405 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.405 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.405 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.405 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.405 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mirroring", 2024-06-04T05:23:49.405 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.405 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.405 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.405 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.406 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.406 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.406 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.406 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.406 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.406 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.406 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.406 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.406 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.406 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.406 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.406 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.406 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.406 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.407 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.407 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.407 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.407 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.407 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.407 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.407 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.407 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.407 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.407 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.407 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.407 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.407 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.408 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.408 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.408 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.408 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.408 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.408 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.408 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.408 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.408 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.408 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.408 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.408 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.408 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.409 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.409 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.409 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.409 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.409 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.409 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.409 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.409 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.409 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.409 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.409 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.409 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.409 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.410 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.410 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.410 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.410 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.410 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.410 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.410 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.410 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.410 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.410 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.410 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.410 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.410 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.411 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.411 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.411 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.411 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.411 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.411 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.411 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.411 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.411 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.411 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.411 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.411 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.411 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.411 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.412 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "nfs", 2024-06-04T05:23:49.412 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.412 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.412 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.412 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.412 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.412 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.412 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.412 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.412 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.412 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.413 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.413 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.413 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.413 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.413 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.413 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.413 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.413 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.413 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.413 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.413 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.413 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.413 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.413 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.413 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.414 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.414 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.414 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.414 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.414 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.414 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.414 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.414 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.414 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.414 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.414 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.414 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.414 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.415 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.415 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.415 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.415 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.415 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.415 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.415 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.415 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.415 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.415 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.415 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.415 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.415 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.415 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.415 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.416 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.416 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.416 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.416 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.416 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.416 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.416 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.416 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.416 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.416 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.416 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.416 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.416 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.416 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.417 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.417 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.417 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.417 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.417 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.417 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.417 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.417 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.417 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.417 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.417 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.417 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.417 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.418 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.418 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.418 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.418 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.418 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.418 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.418 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.418 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.418 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.418 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.418 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "orchestrator", 2024-06-04T05:23:49.418 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.418 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.418 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.419 INFO:teuthology.orchestra.run.smithi070.stdout: "fail_fs": { 2024-06-04T05:23:49.419 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "fail_fs", 2024-06-04T05:23:49.419 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.419 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.419 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.419 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.419 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.419 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.419 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.419 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-06-04T05:23:49.419 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.419 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.419 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.420 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.420 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.420 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.420 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.420 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.420 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.420 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.420 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.420 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.420 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.420 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.420 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.420 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.420 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.421 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.421 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.421 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.421 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.421 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.421 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.421 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.421 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.421 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.421 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.421 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.421 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.421 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.421 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.422 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.422 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.422 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.422 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.422 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.422 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.422 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.422 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.422 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.422 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.422 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.422 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.422 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.423 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.423 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.423 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.423 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.423 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.423 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.423 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.423 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.423 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.423 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.423 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.423 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.423 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.423 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.424 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.424 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.424 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.424 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.424 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.424 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.424 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.424 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.424 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.424 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.424 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.424 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.424 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.425 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.425 INFO:teuthology.orchestra.run.smithi070.stdout: "orchestrator": { 2024-06-04T05:23:49.425 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "orchestrator", 2024-06-04T05:23:49.425 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.425 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.425 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.425 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.425 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.425 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.425 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.425 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm", 2024-06-04T05:23:49.425 INFO:teuthology.orchestra.run.smithi070.stdout: "rook", 2024-06-04T05:23:49.425 INFO:teuthology.orchestra.run.smithi070.stdout: "test_orchestrator" 2024-06-04T05:23:49.426 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.426 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Orchestrator backend", 2024-06-04T05:23:49.426 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.426 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.426 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.426 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.426 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.426 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.426 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.426 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.426 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.426 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.426 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.427 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.427 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.427 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.427 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.427 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.427 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.427 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.427 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "osd_perf_query", 2024-06-04T05:23:49.427 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.427 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.427 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.427 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.428 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.428 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.428 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.428 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.428 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.428 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.428 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.428 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.428 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.428 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.428 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.428 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.428 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.428 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.429 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.429 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.429 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.429 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.429 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.429 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.429 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.429 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.429 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.429 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.429 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.429 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.429 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.430 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.430 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.430 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.430 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.430 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.430 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.430 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.430 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.430 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.430 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.430 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.430 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.430 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.431 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.431 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.431 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.431 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.431 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.431 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.431 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.431 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.431 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.431 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.431 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.431 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.431 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.432 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.432 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.432 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.432 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.432 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.432 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.432 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.432 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.432 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.432 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.432 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.432 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.432 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.433 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.433 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.433 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.433 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.433 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.433 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.433 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.433 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.433 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.433 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.433 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.433 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.433 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.434 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.434 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.434 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "osd_support", 2024-06-04T05:23:49.434 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.434 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.434 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.434 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.434 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.434 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.434 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.434 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.434 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.435 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.435 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.435 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.435 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.435 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.435 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.435 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.435 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.435 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.435 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.435 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.435 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.435 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.435 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.436 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.436 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.436 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.436 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.436 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.436 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.436 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.436 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.436 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.436 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.436 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.436 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.436 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.437 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.437 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.437 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.437 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.437 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.437 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.437 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.437 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.437 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.437 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.437 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.438 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.438 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.438 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.438 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.438 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.438 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.438 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.438 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.438 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.438 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.438 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.438 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.438 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.439 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.439 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.439 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.439 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.439 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.439 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.439 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.439 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.439 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.439 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.439 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.439 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.440 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.440 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.440 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.440 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.440 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.440 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.440 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.440 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.440 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.440 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.440 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.440 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.441 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pg_autoscaler", 2024-06-04T05:23:49.441 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.441 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.441 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.441 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.441 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.441 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.441 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.441 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.441 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.441 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.441 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.441 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.442 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.442 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.442 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.442 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.442 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.442 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.442 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.442 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.442 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.442 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.442 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.442 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.443 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.443 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.443 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.443 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.443 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.443 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.443 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.443 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.443 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.443 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.443 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.443 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.443 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.443 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.446 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.446 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.446 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.446 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.446 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.446 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-06-04T05:23:49.446 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-06-04T05:23:49.446 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.446 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.446 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.446 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-06-04T05:23:49.446 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.446 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.447 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.447 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.447 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.447 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.447 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.447 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.447 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.447 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.447 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.447 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.447 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.447 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.447 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.448 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.448 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.448 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.448 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.448 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.448 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.448 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.448 INFO:teuthology.orchestra.run.smithi070.stdout: "threshold": { 2024-06-04T05:23:49.448 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "threshold", 2024-06-04T05:23:49.448 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:49.448 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.448 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.448 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3.0", 2024-06-04T05:23:49.449 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1.0", 2024-06-04T05:23:49.449 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.449 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.449 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "scaling threshold", 2024-06-04T05:23:49.449 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-06-04T05:23:49.449 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.449 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.449 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.449 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.449 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.449 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "progress", 2024-06-04T05:23:49.449 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.450 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.450 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.450 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_pg_recovery_event": { 2024-06-04T05:23:49.450 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_pg_recovery_event", 2024-06-04T05:23:49.450 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.450 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.450 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.450 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.450 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.450 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.450 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.450 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow the module to show pg recovery progress", 2024-06-04T05:23:49.451 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.451 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.451 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.451 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": { 2024-06-04T05:23:49.451 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enabled", 2024-06-04T05:23:49.451 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.451 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.451 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.451 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.451 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.451 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.451 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.451 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.451 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.452 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.452 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.452 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.452 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.452 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.452 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.452 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.452 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.452 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.452 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.452 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.452 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.452 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.452 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.453 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.453 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.453 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.453 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.453 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.453 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.453 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.453 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.453 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.453 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.453 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.453 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.453 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.454 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.454 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.454 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.454 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.454 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.454 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.454 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.454 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.454 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.454 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.454 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.454 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.454 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.454 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.455 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.455 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.455 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.455 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.455 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.455 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.455 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.455 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.455 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.455 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.455 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.455 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.455 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.456 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.456 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.456 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.456 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.456 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.456 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.456 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.456 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.456 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.456 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.456 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.456 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.456 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.457 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.457 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.457 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.457 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.457 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.457 INFO:teuthology.orchestra.run.smithi070.stdout: "max_completed_events": { 2024-06-04T05:23:49.457 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_completed_events", 2024-06-04T05:23:49.457 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.457 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.457 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.457 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "50", 2024-06-04T05:23:49.457 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.457 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.457 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.458 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "number of past completed events to remember", 2024-06-04T05:23:49.458 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.458 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.458 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.458 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.458 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-06-04T05:23:49.458 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-06-04T05:23:49.458 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.458 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.458 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.458 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-06-04T05:23:49.458 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.458 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.459 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.459 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long the module is going to sleep", 2024-06-04T05:23:49.459 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.459 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.459 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.459 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.459 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.459 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.459 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.459 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.459 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.459 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.459 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.459 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.460 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.460 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.460 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.460 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.460 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.460 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.460 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.460 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.460 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.460 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prometheus", 2024-06-04T05:23:49.460 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.460 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.460 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.460 INFO:teuthology.orchestra.run.smithi070.stdout: "cache": { 2024-06-04T05:23:49.461 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cache", 2024-06-04T05:23:49.461 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.461 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.461 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.461 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.461 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.461 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.461 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.461 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.461 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.461 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.461 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.461 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.461 INFO:teuthology.orchestra.run.smithi070.stdout: "exclude_perf_counters": { 2024-06-04T05:23:49.461 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "exclude_perf_counters", 2024-06-04T05:23:49.462 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.462 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.462 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.462 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.462 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.462 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.462 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.462 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-06-04T05:23:49.462 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-06-04T05:23:49.462 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.462 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.462 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.463 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.463 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.463 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.463 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.463 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.463 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.463 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.463 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.463 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.463 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.463 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.463 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.463 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.463 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.463 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.464 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.464 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.464 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.464 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.464 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.464 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.464 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.464 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.464 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.464 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.464 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.464 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.464 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.464 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.465 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.465 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.465 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.465 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.465 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.465 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.465 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.465 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.465 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.465 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.465 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.465 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.465 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.465 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.466 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.466 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.466 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.466 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.466 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.466 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.466 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.466 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.466 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.466 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.466 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.466 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.466 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.466 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.467 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.467 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.467 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.467 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.467 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.467 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.467 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.467 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.467 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.467 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.467 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.467 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.467 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.468 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_stats_pools": { 2024-06-04T05:23:49.468 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_stats_pools", 2024-06-04T05:23:49.468 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.468 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.468 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.468 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.468 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.468 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.468 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.468 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.468 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.468 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.468 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.468 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.469 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_stats_pools_refresh_interval": { 2024-06-04T05:23:49.469 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-06-04T05:23:49.469 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.469 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.469 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.469 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "300", 2024-06-04T05:23:49.469 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.469 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.469 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.469 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.469 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.469 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.469 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.470 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.470 INFO:teuthology.orchestra.run.smithi070.stdout: "scrape_interval": { 2024-06-04T05:23:49.470 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "scrape_interval", 2024-06-04T05:23:49.470 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:49.470 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.470 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.470 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15.0", 2024-06-04T05:23:49.470 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.470 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.470 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.470 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.470 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.470 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.470 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.471 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.471 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-06-04T05:23:49.471 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-06-04T05:23:49.471 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.471 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.471 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.471 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "::", 2024-06-04T05:23:49.471 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.471 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.471 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.471 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-06-04T05:23:49.471 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.471 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.471 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.472 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.472 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-06-04T05:23:49.472 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-06-04T05:23:49.472 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.472 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.472 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.472 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "9283", 2024-06-04T05:23:49.472 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.472 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.472 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.472 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-06-04T05:23:49.472 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.472 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.472 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.473 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.473 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.473 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.473 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.473 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.473 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.473 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.473 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.473 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.473 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.473 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.473 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.473 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.473 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.474 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.474 INFO:teuthology.orchestra.run.smithi070.stdout: "stale_cache_strategy": { 2024-06-04T05:23:49.474 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "stale_cache_strategy", 2024-06-04T05:23:49.474 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.474 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.474 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.474 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "log", 2024-06-04T05:23:49.474 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.474 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.474 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.474 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.474 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.474 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.475 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.475 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.475 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_behaviour": { 2024-06-04T05:23:49.475 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_behaviour", 2024-06-04T05:23:49.475 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.475 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.475 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.475 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "default", 2024-06-04T05:23:49.475 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.475 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.475 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.475 INFO:teuthology.orchestra.run.smithi070.stdout: "default", 2024-06-04T05:23:49.475 INFO:teuthology.orchestra.run.smithi070.stdout: "error" 2024-06-04T05:23:49.476 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.476 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.476 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.476 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.476 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.476 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.476 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_error_status_code": { 2024-06-04T05:23:49.476 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_error_status_code", 2024-06-04T05:23:49.476 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.476 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.476 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.476 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "500", 2024-06-04T05:23:49.476 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "400", 2024-06-04T05:23:49.477 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "599", 2024-06-04T05:23:49.477 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.477 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.477 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.477 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.477 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.477 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.477 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.477 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.477 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.477 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_support", 2024-06-04T05:23:49.477 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.477 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.477 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.478 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.478 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.478 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.478 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.478 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.478 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.478 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.478 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.478 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.478 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.478 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.478 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.478 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.479 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.479 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.479 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.479 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.479 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.479 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.479 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.479 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.479 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.479 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.479 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.479 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.479 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.479 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.480 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.480 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.480 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.480 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.480 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.480 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.480 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.480 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.480 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.480 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.480 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.480 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.480 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.480 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.481 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.481 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.481 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.481 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.481 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.481 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.481 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.481 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.481 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.481 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.481 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.481 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.481 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.481 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.481 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.482 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.482 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.482 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.482 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.482 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.482 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.482 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.482 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.482 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.482 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.482 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.482 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.482 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.483 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.483 INFO:teuthology.orchestra.run.smithi070.stdout: "max_concurrent_snap_create": { 2024-06-04T05:23:49.483 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_concurrent_snap_create", 2024-06-04T05:23:49.483 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.483 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.483 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.483 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:49.483 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.483 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.483 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.483 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.483 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.483 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.483 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.484 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.484 INFO:teuthology.orchestra.run.smithi070.stdout: "mirror_snapshot_schedule": { 2024-06-04T05:23:49.484 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mirror_snapshot_schedule", 2024-06-04T05:23:49.484 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.484 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.484 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.484 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.484 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.484 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.484 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.484 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.484 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.484 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.484 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.485 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.485 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.485 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.485 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.485 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.485 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.485 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.485 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.485 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.485 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.485 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.485 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.485 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.485 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.486 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.486 INFO:teuthology.orchestra.run.smithi070.stdout: "trash_purge_schedule": { 2024-06-04T05:23:49.486 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "trash_purge_schedule", 2024-06-04T05:23:49.486 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.486 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.486 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.486 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.486 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.486 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.486 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.486 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.486 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.486 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.486 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.487 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.487 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.487 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.487 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.487 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "restful", 2024-06-04T05:23:49.487 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.487 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.487 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.487 INFO:teuthology.orchestra.run.smithi070.stdout: "enable_auth": { 2024-06-04T05:23:49.487 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enable_auth", 2024-06-04T05:23:49.487 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.487 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.487 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.488 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.488 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.488 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.488 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.488 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.488 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.488 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.488 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.488 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.488 INFO:teuthology.orchestra.run.smithi070.stdout: "key_file": { 2024-06-04T05:23:49.488 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "key_file", 2024-06-04T05:23:49.488 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.488 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.488 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.489 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.489 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.489 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.489 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.489 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.489 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.489 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.489 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.489 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.489 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.489 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.489 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.489 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.489 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.489 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.490 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.490 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.490 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.490 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.490 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.490 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.490 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.490 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.490 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.490 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.490 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.490 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.490 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.490 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.491 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.491 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.491 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.491 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.491 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.491 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.491 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.491 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.491 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.491 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.491 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.491 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.491 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.491 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.492 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.492 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.492 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.492 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.492 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.492 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.492 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.492 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.492 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.492 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.492 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.492 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.492 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.492 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.493 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.493 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.493 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.493 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.493 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.493 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.493 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.493 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.493 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.493 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.493 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.493 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.493 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.493 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.493 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.494 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.494 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.494 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.494 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.494 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.494 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.494 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.494 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-06-04T05:23:49.494 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-06-04T05:23:49.494 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.494 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.494 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.494 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.494 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.495 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.495 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.495 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.495 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.495 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.495 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.495 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.495 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-06-04T05:23:49.495 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-06-04T05:23:49.495 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.495 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.495 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.495 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.495 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.495 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.496 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.496 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.496 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.496 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.496 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.496 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.496 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.496 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.496 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.496 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.496 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.496 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.496 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.497 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.497 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.497 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.497 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.497 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.497 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.497 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.497 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.497 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.497 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.497 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rgw", 2024-06-04T05:23:49.497 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.497 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.497 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.497 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.498 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.498 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.498 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.498 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.498 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.498 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.498 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.498 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.498 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.498 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.498 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.498 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.498 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.499 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.499 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.499 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.499 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.499 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.499 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.499 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.499 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.499 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.499 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.499 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.499 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.499 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.499 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.500 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.500 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.500 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.500 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.500 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.500 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.500 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.500 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.500 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.500 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.500 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.500 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.500 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.500 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.501 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.501 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.501 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.501 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.501 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.501 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.501 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.501 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.501 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.501 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.501 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.501 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.501 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.501 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.501 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.502 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.502 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.502 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.502 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.502 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.502 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.502 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.502 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.502 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.502 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.502 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.502 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.502 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.503 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.503 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.503 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.503 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.503 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.503 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.503 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.503 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.503 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.503 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.503 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.503 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.503 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.503 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.504 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.504 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.504 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.504 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rook", 2024-06-04T05:23:49.504 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.504 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.504 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.504 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.504 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.504 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.504 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.504 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.504 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.504 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.505 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.505 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.505 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.505 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.505 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.505 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.505 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.505 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.505 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.505 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.505 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.505 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.505 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.505 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.506 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.506 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.506 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.506 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.506 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.506 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.506 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.506 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.506 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.506 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.506 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.506 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.506 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.506 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.507 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.507 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.507 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.507 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.507 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.507 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.507 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.507 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.507 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.507 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.507 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.507 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.507 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.508 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.508 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.508 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.508 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.508 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.508 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.508 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.508 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.508 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.508 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.508 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.508 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.508 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.508 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.509 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.509 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.509 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.509 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.509 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.509 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.509 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.509 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.509 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.509 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.509 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.509 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.509 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.510 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.510 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.510 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.510 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.510 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.510 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.510 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.510 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.510 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.510 INFO:teuthology.orchestra.run.smithi070.stdout: "storage_class": { 2024-06-04T05:23:49.510 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "storage_class", 2024-06-04T05:23:49.510 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.510 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.511 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.511 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "local", 2024-06-04T05:23:49.511 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.511 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.511 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.511 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-06-04T05:23:49.511 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.511 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.511 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.511 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.511 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.511 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.511 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.511 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "selftest", 2024-06-04T05:23:49.512 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.512 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.512 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.512 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.512 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.512 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.512 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.512 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.512 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.512 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.512 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.512 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.512 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.512 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.513 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.513 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.513 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.513 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.513 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.513 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.513 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.513 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.513 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.513 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.513 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.513 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.513 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.513 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.514 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.514 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.514 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.514 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.514 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.514 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.514 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.514 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.514 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.514 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.514 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.514 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.514 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.514 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.515 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.515 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.515 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.515 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.515 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.515 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.515 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.515 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.515 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.515 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.515 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.515 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.515 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.515 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.516 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.516 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.516 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.516 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.516 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.516 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.516 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.516 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.516 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.516 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.516 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.516 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.516 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.516 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.517 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.517 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.517 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.517 INFO:teuthology.orchestra.run.smithi070.stdout: "roption1": { 2024-06-04T05:23:49.517 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "roption1", 2024-06-04T05:23:49.517 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.517 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.517 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.517 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.517 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.517 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.517 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.517 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.518 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.518 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.518 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.518 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.518 INFO:teuthology.orchestra.run.smithi070.stdout: "roption2": { 2024-06-04T05:23:49.518 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "roption2", 2024-06-04T05:23:49.518 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.518 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.518 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.518 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "xyz", 2024-06-04T05:23:49.518 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.518 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.518 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.519 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.519 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.519 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.519 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.519 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.519 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption1": { 2024-06-04T05:23:49.519 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption1", 2024-06-04T05:23:49.519 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.519 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.519 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.519 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.519 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.519 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.519 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.520 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.520 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.520 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.520 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.520 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.520 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption2": { 2024-06-04T05:23:49.520 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption2", 2024-06-04T05:23:49.520 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.520 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.520 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.520 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.520 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.520 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.520 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.521 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.521 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.521 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.521 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.521 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.521 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption3": { 2024-06-04T05:23:49.521 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption3", 2024-06-04T05:23:49.521 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:49.521 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.521 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.521 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.521 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.521 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.521 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.522 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.522 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.522 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.522 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.522 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.522 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption4": { 2024-06-04T05:23:49.522 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption4", 2024-06-04T05:23:49.522 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.522 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.522 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.522 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.522 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.522 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.522 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.523 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.523 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.523 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.523 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.523 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.523 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption5": { 2024-06-04T05:23:49.523 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption5", 2024-06-04T05:23:49.523 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.523 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.523 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.523 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.523 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.523 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.524 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.524 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.524 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.524 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.524 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.524 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.524 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption6": { 2024-06-04T05:23:49.524 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption6", 2024-06-04T05:23:49.524 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.524 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.524 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.524 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.524 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.524 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.525 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.525 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.525 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.525 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.525 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.525 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.525 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption7": { 2024-06-04T05:23:49.525 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption7", 2024-06-04T05:23:49.525 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.525 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.525 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.525 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.525 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-06-04T05:23:49.525 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "42", 2024-06-04T05:23:49.526 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.526 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.526 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.526 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.526 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.526 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.526 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.526 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.526 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.526 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.526 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.526 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.526 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.527 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.527 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.527 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.527 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.527 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.527 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.527 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.527 INFO:teuthology.orchestra.run.smithi070.stdout: "testkey": { 2024-06-04T05:23:49.527 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testkey", 2024-06-04T05:23:49.527 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.527 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.527 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.527 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.528 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.528 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.528 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.528 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.528 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.528 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.528 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.528 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.528 INFO:teuthology.orchestra.run.smithi070.stdout: "testlkey": { 2024-06-04T05:23:49.528 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testlkey", 2024-06-04T05:23:49.528 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.528 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.528 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.528 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.529 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.529 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.529 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.529 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.529 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.529 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.529 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.529 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.529 INFO:teuthology.orchestra.run.smithi070.stdout: "testnewline": { 2024-06-04T05:23:49.529 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testnewline", 2024-06-04T05:23:49.529 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.529 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.529 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.529 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.530 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.530 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.530 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.530 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.530 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.530 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.530 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.530 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.530 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.530 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.530 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.530 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smb", 2024-06-04T05:23:49.530 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.530 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.531 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.531 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.531 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.531 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.531 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.531 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.531 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.531 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.531 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.531 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.531 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.531 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.531 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.531 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.531 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.532 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.532 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.532 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.532 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.532 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.532 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.532 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.532 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.532 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.532 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.532 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.532 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.532 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.532 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.533 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.533 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.533 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.533 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.533 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.533 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.533 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.533 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.533 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.533 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.533 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.533 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.533 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.534 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.534 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.534 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.534 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.534 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.534 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.534 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.534 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.534 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.534 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.534 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.534 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.534 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.534 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.534 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.535 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.535 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.535 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.535 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.535 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.535 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.535 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.535 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.535 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.535 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.535 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.535 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.535 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.535 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.536 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.536 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.536 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.536 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.536 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.536 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.536 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.536 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.536 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.536 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.536 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.536 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.536 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.537 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.537 INFO:teuthology.orchestra.run.smithi070.stdout: "update_orchestration": { 2024-06-04T05:23:49.537 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "update_orchestration", 2024-06-04T05:23:49.537 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.537 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.537 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.537 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.537 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.537 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.537 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.537 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-06-04T05:23:49.537 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.537 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.538 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.538 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.538 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.538 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.538 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.538 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snap_schedule", 2024-06-04T05:23:49.538 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.538 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.538 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.538 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_m_granularity": { 2024-06-04T05:23:49.538 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_m_granularity", 2024-06-04T05:23:49.538 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.538 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.539 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.539 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.539 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.539 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.539 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.539 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow minute scheduled snapshots", 2024-06-04T05:23:49.539 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.539 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.539 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.539 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.539 INFO:teuthology.orchestra.run.smithi070.stdout: "dump_on_update": { 2024-06-04T05:23:49.539 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "dump_on_update", 2024-06-04T05:23:49.539 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.539 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.540 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.540 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.540 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.540 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.540 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.540 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "dump database to debug log on update", 2024-06-04T05:23:49.540 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.540 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.540 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.540 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.540 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.540 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.540 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.541 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.541 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.541 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.541 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.541 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.541 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.541 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.541 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.541 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.541 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.541 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.541 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.541 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.541 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.542 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.542 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.542 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.542 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.542 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.542 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.542 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.542 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.542 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.542 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.542 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.542 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.542 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.542 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.543 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.543 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.543 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.543 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.543 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.543 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.543 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.543 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.543 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.543 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.543 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.543 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.543 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.543 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.544 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.544 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.544 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.544 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.544 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.544 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.544 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.544 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.544 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.544 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.544 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.544 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.544 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.544 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.544 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.545 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.545 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.545 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.545 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.545 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.545 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.545 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.545 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.545 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.545 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.545 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.545 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.545 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.546 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.546 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.546 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.546 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.546 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.546 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.546 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.546 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.546 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.546 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.546 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.546 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.547 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.547 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.547 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "stats", 2024-06-04T05:23:49.547 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.547 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.547 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.547 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.547 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.547 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.547 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.547 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.547 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.547 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.547 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.548 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.548 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.548 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.548 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.548 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.548 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.548 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.548 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.548 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.548 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.548 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.548 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.548 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.548 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.550 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.550 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.550 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.550 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.550 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.550 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.550 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.550 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.550 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.550 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.550 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.550 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.550 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.551 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.551 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.551 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.551 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.551 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.551 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.551 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.551 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.551 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.551 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.551 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.551 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.551 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.551 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.552 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.552 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.552 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.552 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.552 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.552 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.552 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.552 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.552 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.552 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.552 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.552 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.552 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.552 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.553 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.553 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.553 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.553 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.553 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.553 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.553 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.553 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.553 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.553 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "status", 2024-06-04T05:23:49.553 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.553 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.553 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.553 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.554 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.554 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.554 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.554 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.554 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.554 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.554 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.554 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.554 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.554 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.554 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.554 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.554 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.554 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.555 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.555 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.555 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.555 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.555 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.555 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.555 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.555 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.555 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.555 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.555 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.555 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.555 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.555 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.556 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.556 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.556 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.556 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.556 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.556 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.556 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.556 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.556 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.556 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.556 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.556 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.556 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.556 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.557 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.557 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.557 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.557 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.557 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.557 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.557 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.557 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.557 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.557 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.557 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.557 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.557 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.557 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.558 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.558 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.558 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.558 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.558 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.558 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.558 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.558 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.558 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.558 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.558 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.558 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.558 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.559 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.559 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.559 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.559 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.559 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.559 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.559 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.559 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.559 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.559 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.559 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.559 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.559 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.559 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.560 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.560 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.560 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.560 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "telegraf", 2024-06-04T05:23:49.560 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.560 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.560 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.560 INFO:teuthology.orchestra.run.smithi070.stdout: "address": { 2024-06-04T05:23:49.560 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "address", 2024-06-04T05:23:49.560 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.560 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.560 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.560 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-06-04T05:23:49.561 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.561 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.561 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.561 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.561 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.561 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.561 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.561 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.561 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-06-04T05:23:49.561 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-06-04T05:23:49.561 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.561 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.561 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.562 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15", 2024-06-04T05:23:49.562 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.562 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.562 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.562 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.562 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.562 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.562 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.562 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.562 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.562 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.562 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.562 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.562 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.563 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.563 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.563 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.563 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.563 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.563 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.563 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.563 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.563 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.563 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.563 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.563 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.563 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.563 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.563 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.564 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.564 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.564 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.564 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.564 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.564 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.564 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.564 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.564 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.564 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.564 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.564 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.564 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.564 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.565 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.565 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.565 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.565 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.565 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.565 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.565 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.565 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.565 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.565 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.565 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.565 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.565 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.566 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.566 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.566 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.566 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.566 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.566 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.566 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.566 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.566 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.566 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.566 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.566 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.566 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.567 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.567 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.567 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.567 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.567 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.567 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.567 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.567 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.567 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.567 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.567 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.567 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.567 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.567 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.568 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.568 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.568 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.568 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.568 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.568 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.568 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.568 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.568 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.568 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.568 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.568 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.568 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.568 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "telemetry", 2024-06-04T05:23:49.569 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.569 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.569 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.569 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_basic": { 2024-06-04T05:23:49.569 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_basic", 2024-06-04T05:23:49.569 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.569 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.569 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.569 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.569 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.569 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.569 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.569 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share basic cluster information (size, version)", 2024-06-04T05:23:49.570 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.570 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.570 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.570 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.570 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_crash": { 2024-06-04T05:23:49.570 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_crash", 2024-06-04T05:23:49.570 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.570 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.570 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.570 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.570 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.570 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.570 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.570 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-06-04T05:23:49.571 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.571 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.571 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.571 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.571 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_device": { 2024-06-04T05:23:49.571 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_device", 2024-06-04T05:23:49.571 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.571 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.571 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.571 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.571 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.571 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.571 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.571 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-06-04T05:23:49.572 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.572 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.572 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.572 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.572 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_ident": { 2024-06-04T05:23:49.572 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_ident", 2024-06-04T05:23:49.572 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.572 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.572 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.572 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.572 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.572 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.572 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.572 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-06-04T05:23:49.573 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.573 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.573 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.573 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.573 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_perf": { 2024-06-04T05:23:49.573 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_perf", 2024-06-04T05:23:49.573 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.573 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.573 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.573 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.573 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.573 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.573 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.574 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share various performance metrics of a cluster", 2024-06-04T05:23:49.574 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.574 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.574 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.574 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.574 INFO:teuthology.orchestra.run.smithi070.stdout: "contact": { 2024-06-04T05:23:49.574 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "contact", 2024-06-04T05:23:49.574 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.574 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.574 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.574 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.574 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.574 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.574 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.575 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.575 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.575 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.575 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.575 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.575 INFO:teuthology.orchestra.run.smithi070.stdout: "description": { 2024-06-04T05:23:49.575 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "description", 2024-06-04T05:23:49.575 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.575 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.575 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.575 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.575 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.575 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.575 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.576 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.576 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.576 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.576 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.576 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.576 INFO:teuthology.orchestra.run.smithi070.stdout: "device_url": { 2024-06-04T05:23:49.576 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_url", 2024-06-04T05:23:49.576 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.576 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.576 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.576 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-06-04T05:23:49.576 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.576 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.577 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.577 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.577 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.577 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.577 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.577 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.577 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": { 2024-06-04T05:23:49.577 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enabled", 2024-06-04T05:23:49.577 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.577 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.577 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.577 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.577 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.577 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.578 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.578 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.578 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.578 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.578 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.578 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-06-04T05:23:49.578 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-06-04T05:23:49.578 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.578 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.578 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.578 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "24", 2024-06-04T05:23:49.578 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "8", 2024-06-04T05:23:49.578 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.579 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.579 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.579 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.579 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.579 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.579 INFO:teuthology.orchestra.run.smithi070.stdout: "last_opt_revision": { 2024-06-04T05:23:49.579 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "last_opt_revision", 2024-06-04T05:23:49.579 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.579 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.579 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.579 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1", 2024-06-04T05:23:49.579 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.579 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.580 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.580 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.580 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.580 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.580 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.580 INFO:teuthology.orchestra.run.smithi070.stdout: "leaderboard": { 2024-06-04T05:23:49.580 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "leaderboard", 2024-06-04T05:23:49.580 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.580 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.580 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.580 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.580 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.580 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.581 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.581 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.581 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.581 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.581 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.581 INFO:teuthology.orchestra.run.smithi070.stdout: "leaderboard_description": { 2024-06-04T05:23:49.581 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "leaderboard_description", 2024-06-04T05:23:49.581 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.581 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.581 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.581 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.581 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.581 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.581 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.582 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.582 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.582 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.582 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.582 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.582 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.582 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.582 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.582 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.582 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.582 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.582 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.582 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.583 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.583 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.583 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.583 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.583 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.583 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.583 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.583 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.583 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.583 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.583 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.583 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.583 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.584 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.584 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.584 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.584 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.584 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.584 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.584 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.584 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.584 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.584 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.584 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.584 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.584 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.585 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.585 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.585 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.585 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.585 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.585 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.585 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.585 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.585 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.585 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.585 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.585 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.585 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.585 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.586 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.586 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.586 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.586 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.586 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.586 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.586 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.586 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.586 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.586 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.586 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.586 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.587 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.587 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.587 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.587 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.587 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.587 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.587 INFO:teuthology.orchestra.run.smithi070.stdout: "organization": { 2024-06-04T05:23:49.587 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "organization", 2024-06-04T05:23:49.587 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.587 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.587 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.587 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.587 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.588 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.588 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.588 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.588 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.588 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.588 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.588 INFO:teuthology.orchestra.run.smithi070.stdout: "proxy": { 2024-06-04T05:23:49.588 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "proxy", 2024-06-04T05:23:49.588 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.588 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.588 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.588 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.589 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.589 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.589 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.589 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.589 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.589 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.589 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.589 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.589 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.589 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.589 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.589 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.589 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.590 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.590 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.590 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.590 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.590 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.590 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.590 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.590 INFO:teuthology.orchestra.run.smithi070.stdout: "url": { 2024-06-04T05:23:49.590 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "url", 2024-06-04T05:23:49.590 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.590 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.590 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.590 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-06-04T05:23:49.591 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.591 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.591 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.591 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.591 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.591 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.591 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.591 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.591 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.591 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "test_orchestrator", 2024-06-04T05:23:49.591 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.591 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.592 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.592 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.592 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.592 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.592 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.592 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.592 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.592 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.592 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.592 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.592 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.592 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.592 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.592 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.593 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.593 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.593 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.593 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.593 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.593 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.593 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.593 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.593 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.593 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.593 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.593 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.593 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.594 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.594 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.594 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.594 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.594 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.594 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.594 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.594 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.594 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.594 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.594 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.595 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.595 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.595 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.595 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.595 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.595 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.595 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.595 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.595 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.595 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.595 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.595 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.595 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.595 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.596 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.596 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.596 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.596 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.596 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.596 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.596 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.596 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.596 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.596 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.596 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.596 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.596 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.597 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.597 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.597 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.597 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.597 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.597 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.597 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.597 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.597 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.597 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.597 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.597 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.597 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.598 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.598 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.598 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.598 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.598 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "volumes", 2024-06-04T05:23:49.598 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.598 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.598 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.598 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.598 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.598 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.598 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.599 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.599 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.599 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.599 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.599 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.599 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.599 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.599 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.599 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.599 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.599 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.599 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.599 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.599 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.599 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.600 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.600 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.600 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.600 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.600 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.600 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.600 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.600 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.600 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.600 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.600 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.600 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.600 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.600 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.601 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.601 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.601 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.601 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.601 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.601 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.601 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.601 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.601 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.601 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.601 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.601 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.601 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.601 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.601 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.602 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.602 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.602 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.602 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.602 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.602 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.602 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.602 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.602 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.602 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.602 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.602 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.603 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.603 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.603 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.603 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.603 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.603 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.603 INFO:teuthology.orchestra.run.smithi070.stdout: "max_concurrent_clones": { 2024-06-04T05:23:49.603 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_concurrent_clones", 2024-06-04T05:23:49.603 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.603 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.603 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.603 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "4", 2024-06-04T05:23:49.604 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.604 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.604 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.604 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Number of asynchronous cloner threads", 2024-06-04T05:23:49.604 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.604 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.604 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.604 INFO:teuthology.orchestra.run.smithi070.stdout: "periodic_async_work": { 2024-06-04T05:23:49.604 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "periodic_async_work", 2024-06-04T05:23:49.604 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.604 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.604 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.604 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.605 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.605 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.605 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.605 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Periodically check for async work", 2024-06-04T05:23:49.605 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.605 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.605 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.605 INFO:teuthology.orchestra.run.smithi070.stdout: "snapshot_clone_delay": { 2024-06-04T05:23:49.605 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snapshot_clone_delay", 2024-06-04T05:23:49.605 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.605 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.605 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.605 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.606 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.606 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.606 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.606 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-06-04T05:23:49.606 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.606 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.606 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.606 INFO:teuthology.orchestra.run.smithi070.stdout: "snapshot_clone_no_wait": { 2024-06-04T05:23:49.606 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snapshot_clone_no_wait", 2024-06-04T05:23:49.606 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.606 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.606 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.606 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.607 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.607 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.607 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.607 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-06-04T05:23:49.607 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.607 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.607 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.607 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.607 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.607 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.607 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.607 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.607 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.608 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.608 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.608 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.608 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.608 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.608 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.608 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.608 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.608 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.608 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.608 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix", 2024-06-04T05:23:49.608 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.608 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.609 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.609 INFO:teuthology.orchestra.run.smithi070.stdout: "discovery_interval": { 2024-06-04T05:23:49.609 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "discovery_interval", 2024-06-04T05:23:49.609 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-06-04T05:23:49.609 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.609 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.609 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "100", 2024-06-04T05:23:49.609 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.609 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.609 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.609 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.609 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.609 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.609 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.610 INFO:teuthology.orchestra.run.smithi070.stdout: "identifier": { 2024-06-04T05:23:49.610 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "identifier", 2024-06-04T05:23:49.610 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.610 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.610 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.610 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.610 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.610 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.610 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.610 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.610 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.610 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.610 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.611 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-06-04T05:23:49.611 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-06-04T05:23:49.611 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.611 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.611 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.611 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-06-04T05:23:49.611 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.611 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.611 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.611 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.611 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.611 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.611 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.612 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.612 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.612 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.612 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.612 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.612 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.612 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.612 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.612 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.612 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.612 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.612 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.612 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.613 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.613 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.613 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.613 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.613 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.613 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.613 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.613 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.613 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.613 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.613 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.613 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.613 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.614 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.614 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.614 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.614 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.614 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.614 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.614 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.614 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.614 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.614 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.614 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.614 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.615 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.615 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.615 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.615 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.615 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.615 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.615 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.615 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.615 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.615 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.615 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.615 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.615 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.615 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.616 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.616 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.616 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.616 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.616 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.616 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.616 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.616 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.616 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.616 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.616 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.616 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.616 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.617 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.617 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.617 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.617 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.617 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.617 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.617 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.617 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.617 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.617 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.617 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.617 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.617 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.618 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.618 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_host": { 2024-06-04T05:23:49.618 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_host", 2024-06-04T05:23:49.618 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.618 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.618 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.618 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.618 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.618 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.618 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.618 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.618 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.619 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.619 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.619 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_port": { 2024-06-04T05:23:49.619 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_port", 2024-06-04T05:23:49.619 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.619 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.619 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.619 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10051", 2024-06-04T05:23:49.619 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.619 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.619 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.619 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.619 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.620 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.620 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.620 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_sender": { 2024-06-04T05:23:49.620 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_sender", 2024-06-04T05:23:49.620 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.620 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.620 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.620 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-06-04T05:23:49.620 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.620 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.620 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.620 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.620 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.621 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.621 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.621 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.621 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.621 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.621 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-06-04T05:23:49.621 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.621 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.621 INFO:teuthology.orchestra.run.smithi070.stdout: "modules": [ 2024-06-04T05:23:49.621 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm", 2024-06-04T05:23:49.621 INFO:teuthology.orchestra.run.smithi070.stdout: "dashboard", 2024-06-04T05:23:49.621 INFO:teuthology.orchestra.run.smithi070.stdout: "iostat", 2024-06-04T05:23:49.621 INFO:teuthology.orchestra.run.smithi070.stdout: "nfs", 2024-06-04T05:23:49.621 INFO:teuthology.orchestra.run.smithi070.stdout: "restful", 2024-06-04T05:23:49.621 INFO:teuthology.orchestra.run.smithi070.stdout: "snap_schedule" 2024-06-04T05:23:49.622 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.622 INFO:teuthology.orchestra.run.smithi070.stdout: "available_modules": [ 2024-06-04T05:23:49.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.622 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "alerts", 2024-06-04T05:23:49.622 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.622 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.622 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.622 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-06-04T05:23:49.622 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-06-04T05:23:49.622 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.622 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.622 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.622 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-06-04T05:23:49.623 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.623 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.623 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.623 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How frequently to reexamine health status", 2024-06-04T05:23:49.623 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.623 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.623 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.623 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.623 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.623 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.623 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.623 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.623 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.624 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.624 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.624 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.624 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.624 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.624 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.624 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.624 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.624 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.624 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.624 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.624 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.624 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.624 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.625 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.625 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.625 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.625 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.625 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.625 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.625 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.625 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.625 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.625 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.625 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.625 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.625 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.626 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.626 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.626 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.626 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.626 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.626 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.626 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.626 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.626 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.626 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.626 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.626 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.627 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.627 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.627 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.627 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.627 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.627 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.627 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.627 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.627 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.627 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.627 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.627 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.627 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.627 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.628 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.628 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.628 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.628 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.628 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.628 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.628 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.628 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_destination": { 2024-06-04T05:23:49.628 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_destination", 2024-06-04T05:23:49.628 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.628 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.628 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Email address to send alerts to", 2024-06-04T05:23:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_from_name": { 2024-06-04T05:23:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_from_name", 2024-06-04T05:23:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.630 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.630 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "Ceph", 2024-06-04T05:23:49.630 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.630 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.630 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.630 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Email From: name", 2024-06-04T05:23:49.630 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.630 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.630 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.630 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_host": { 2024-06-04T05:23:49.630 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_host", 2024-06-04T05:23:49.630 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.631 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.631 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.631 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.631 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.631 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.631 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.631 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP server", 2024-06-04T05:23:49.631 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.631 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.631 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.631 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_password": { 2024-06-04T05:23:49.631 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_password", 2024-06-04T05:23:49.631 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.632 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.632 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.632 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.632 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.632 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.632 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.632 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Password to authenticate with", 2024-06-04T05:23:49.632 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.632 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.632 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.632 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_port": { 2024-06-04T05:23:49.632 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_port", 2024-06-04T05:23:49.632 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.632 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.633 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.633 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "465", 2024-06-04T05:23:49.633 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.633 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.633 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.633 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP port", 2024-06-04T05:23:49.633 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.633 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.633 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.633 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_sender": { 2024-06-04T05:23:49.633 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_sender", 2024-06-04T05:23:49.633 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.633 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.633 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.634 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.634 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.634 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.634 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.634 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP envelope sender", 2024-06-04T05:23:49.634 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.634 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.634 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.634 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_ssl": { 2024-06-04T05:23:49.634 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_ssl", 2024-06-04T05:23:49.634 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.634 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.635 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.635 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.635 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.635 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.635 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.635 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use SSL to connect to SMTP server", 2024-06-04T05:23:49.635 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.635 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.635 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.635 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_user": { 2024-06-04T05:23:49.635 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_user", 2024-06-04T05:23:49.635 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.635 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.635 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.636 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.636 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.636 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.636 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.636 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "User to authenticate as", 2024-06-04T05:23:49.636 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.636 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.636 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.636 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.636 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.636 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.636 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.636 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.636 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.637 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.637 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.637 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.637 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.637 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.637 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.637 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.637 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.637 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.637 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "balancer", 2024-06-04T05:23:49.637 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.638 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.638 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.638 INFO:teuthology.orchestra.run.smithi070.stdout: "active": { 2024-06-04T05:23:49.638 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "active", 2024-06-04T05:23:49.638 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.638 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.638 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.638 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.638 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.638 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.638 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.638 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "automatically balance PGs across cluster", 2024-06-04T05:23:49.638 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.638 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.639 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.639 INFO:teuthology.orchestra.run.smithi070.stdout: "begin_time": { 2024-06-04T05:23:49.639 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "begin_time", 2024-06-04T05:23:49.639 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.639 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.639 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.639 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0000", 2024-06-04T05:23:49.639 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.639 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.639 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.639 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "beginning time of day to automatically balance", 2024-06-04T05:23:49.639 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-04T05:23:49.639 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.640 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.640 INFO:teuthology.orchestra.run.smithi070.stdout: "begin_weekday": { 2024-06-04T05:23:49.640 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "begin_weekday", 2024-06-04T05:23:49.640 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-06-04T05:23:49.640 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.640 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.640 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.640 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0", 2024-06-04T05:23:49.640 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "6", 2024-06-04T05:23:49.640 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.640 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-06-04T05:23:49.640 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-04T05:23:49.640 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.640 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.641 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_max_iterations": { 2024-06-04T05:23:49.641 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_max_iterations", 2024-06-04T05:23:49.641 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-06-04T05:23:49.641 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.641 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.641 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "25", 2024-06-04T05:23:49.641 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-06-04T05:23:49.641 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "250", 2024-06-04T05:23:49.641 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.641 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-06-04T05:23:49.641 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.641 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.641 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.641 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_metrics": { 2024-06-04T05:23:49.641 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_metrics", 2024-06-04T05:23:49.642 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.642 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.642 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.642 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "pgs,objects,bytes", 2024-06-04T05:23:49.642 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.642 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.642 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.642 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-06-04T05:23:49.642 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-06-04T05:23:49.642 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.642 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.642 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_step": { 2024-06-04T05:23:49.643 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_step", 2024-06-04T05:23:49.643 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:49.643 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.643 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.643 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0.5", 2024-06-04T05:23:49.643 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0.001", 2024-06-04T05:23:49.643 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "0.999", 2024-06-04T05:23:49.643 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.643 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "aggressiveness of optimization", 2024-06-04T05:23:49.643 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-06-04T05:23:49.643 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.643 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.643 INFO:teuthology.orchestra.run.smithi070.stdout: "end_time": { 2024-06-04T05:23:49.644 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "end_time", 2024-06-04T05:23:49.644 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.644 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.644 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.644 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "2359", 2024-06-04T05:23:49.644 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.644 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.644 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.644 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "ending time of day to automatically balance", 2024-06-04T05:23:49.644 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-04T05:23:49.644 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.644 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.645 INFO:teuthology.orchestra.run.smithi070.stdout: "end_weekday": { 2024-06-04T05:23:49.645 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "end_weekday", 2024-06-04T05:23:49.645 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-06-04T05:23:49.645 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.645 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.645 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.645 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0", 2024-06-04T05:23:49.645 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "6", 2024-06-04T05:23:49.645 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.645 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-06-04T05:23:49.645 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-04T05:23:49.645 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.645 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.646 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.646 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.646 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.646 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.646 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.646 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.646 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.646 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.646 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.646 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.646 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.646 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.647 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.647 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.647 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.647 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.647 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.647 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.647 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.647 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.647 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.647 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.647 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.647 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.647 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.648 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.648 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.648 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.648 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.648 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.648 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.648 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.648 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.648 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.648 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.648 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.649 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.649 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.649 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.649 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.649 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.649 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.649 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.649 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.649 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.649 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.649 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.649 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.649 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.649 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.650 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.650 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.650 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.650 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.650 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.650 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.650 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.650 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.650 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.650 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.650 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.650 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.651 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.651 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.651 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.651 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.651 INFO:teuthology.orchestra.run.smithi070.stdout: "min_score": { 2024-06-04T05:23:49.651 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "min_score", 2024-06-04T05:23:49.651 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:49.651 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.651 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.651 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.651 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.651 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.652 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.652 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-06-04T05:23:49.652 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.652 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.652 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.652 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": { 2024-06-04T05:23:49.652 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mode", 2024-06-04T05:23:49.652 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.652 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.652 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.652 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "upmap", 2024-06-04T05:23:49.652 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.652 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.652 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.652 INFO:teuthology.orchestra.run.smithi070.stdout: "crush-compat", 2024-06-04T05:23:49.653 INFO:teuthology.orchestra.run.smithi070.stdout: "none", 2024-06-04T05:23:49.653 INFO:teuthology.orchestra.run.smithi070.stdout: "read", 2024-06-04T05:23:49.653 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap", 2024-06-04T05:23:49.653 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap-read" 2024-06-04T05:23:49.653 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.653 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Balancer mode", 2024-06-04T05:23:49.653 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.653 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.653 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.653 INFO:teuthology.orchestra.run.smithi070.stdout: "pool_ids": { 2024-06-04T05:23:49.653 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pool_ids", 2024-06-04T05:23:49.653 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.653 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.653 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.654 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.654 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.654 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.654 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.654 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-06-04T05:23:49.654 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.654 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.654 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.654 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-06-04T05:23:49.654 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-06-04T05:23:49.654 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.654 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.654 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.655 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-06-04T05:23:49.655 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.655 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.655 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.655 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-06-04T05:23:49.655 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.655 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.655 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.655 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.655 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.655 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.655 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.655 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.655 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.655 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.656 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.656 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.656 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.656 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.656 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.656 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.656 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap_max_deviation": { 2024-06-04T05:23:49.656 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "upmap_max_deviation", 2024-06-04T05:23:49.656 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.656 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.656 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.656 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-06-04T05:23:49.656 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-06-04T05:23:49.656 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.657 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.657 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "deviation below which no optimization is attempted", 2024-06-04T05:23:49.657 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-06-04T05:23:49.657 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.657 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.657 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap_max_optimizations": { 2024-06-04T05:23:49.657 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "upmap_max_optimizations", 2024-06-04T05:23:49.657 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-06-04T05:23:49.657 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.657 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.657 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:49.657 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.657 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.658 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.658 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-06-04T05:23:49.658 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.658 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.658 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.658 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.658 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.658 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cephadm", 2024-06-04T05:23:49.658 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.658 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.658 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.658 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_down_multiplier": { 2024-06-04T05:23:49.659 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_down_multiplier", 2024-06-04T05:23:49.659 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:49.659 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.659 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.659 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3.0", 2024-06-04T05:23:49.659 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.659 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.659 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.659 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-06-04T05:23:49.659 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.659 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.659 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.659 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_refresh_rate": { 2024-06-04T05:23:49.660 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_refresh_rate", 2024-06-04T05:23:49.660 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.660 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.660 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.660 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "20", 2024-06-04T05:23:49.660 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.660 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.660 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.660 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-06-04T05:23:49.660 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.660 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.660 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.661 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_starting_port": { 2024-06-04T05:23:49.661 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_starting_port", 2024-06-04T05:23:49.661 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.661 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.661 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.661 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "4721", 2024-06-04T05:23:49.661 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.661 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.661 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.661 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-06-04T05:23:49.661 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.661 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.661 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.662 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_ptrace": { 2024-06-04T05:23:49.662 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_ptrace", 2024-06-04T05:23:49.662 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.662 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.662 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.662 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.662 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.662 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.662 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.662 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-06-04T05:23:49.662 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-06-04T05:23:49.662 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.662 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.663 INFO:teuthology.orchestra.run.smithi070.stdout: "autotune_interval": { 2024-06-04T05:23:49.663 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "autotune_interval", 2024-06-04T05:23:49.663 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.663 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.663 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.663 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-06-04T05:23:49.663 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.663 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.663 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.663 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to autotune daemon memory", 2024-06-04T05:23:49.663 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.663 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.663 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.664 INFO:teuthology.orchestra.run.smithi070.stdout: "autotune_memory_target_ratio": { 2024-06-04T05:23:49.664 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "autotune_memory_target_ratio", 2024-06-04T05:23:49.664 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:49.664 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.664 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.664 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0.7", 2024-06-04T05:23:49.664 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.664 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.664 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.664 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-06-04T05:23:49.664 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.664 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.664 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.664 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm_log_destination": { 2024-06-04T05:23:49.665 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cephadm_log_destination", 2024-06-04T05:23:49.665 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.665 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.665 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.665 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.665 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.665 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.665 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.665 INFO:teuthology.orchestra.run.smithi070.stdout: "file", 2024-06-04T05:23:49.665 INFO:teuthology.orchestra.run.smithi070.stdout: "file,syslog", 2024-06-04T05:23:49.665 INFO:teuthology.orchestra.run.smithi070.stdout: "syslog" 2024-06-04T05:23:49.665 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.665 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-06-04T05:23:49.665 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.666 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.666 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.666 INFO:teuthology.orchestra.run.smithi070.stdout: "cgroups_split": { 2024-06-04T05:23:49.666 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cgroups_split", 2024-06-04T05:23:49.666 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.666 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.666 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.666 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.666 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.666 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.666 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.666 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-06-04T05:23:49.666 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.667 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.667 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.667 INFO:teuthology.orchestra.run.smithi070.stdout: "config_checks_enabled": { 2024-06-04T05:23:49.667 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_checks_enabled", 2024-06-04T05:23:49.667 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.667 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.667 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.667 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.667 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.667 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.667 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.667 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-06-04T05:23:49.667 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.668 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.668 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.668 INFO:teuthology.orchestra.run.smithi070.stdout: "config_dashboard": { 2024-06-04T05:23:49.668 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_dashboard", 2024-06-04T05:23:49.668 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.668 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.668 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.668 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.668 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.668 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.668 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.668 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-06-04T05:23:49.668 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.669 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.669 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.669 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_alertmanager": { 2024-06-04T05:23:49.669 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_alertmanager", 2024-06-04T05:23:49.669 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.669 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.669 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.669 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-06-04T05:23:49.669 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.669 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.669 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.669 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-06-04T05:23:49.670 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.670 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.670 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.670 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_base": { 2024-06-04T05:23:49.670 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_base", 2024-06-04T05:23:49.670 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.670 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.670 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.670 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/ceph", 2024-06-04T05:23:49.670 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.670 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.670 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.671 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Container image name, without the tag", 2024-06-04T05:23:49.671 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.671 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.671 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.671 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_elasticsearch": { 2024-06-04T05:23:49.671 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_elasticsearch", 2024-06-04T05:23:49.671 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.671 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.671 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.671 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-06-04T05:23:49.671 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.671 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.671 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.672 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "elasticsearch container image", 2024-06-04T05:23:49.672 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.672 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.672 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.672 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_grafana": { 2024-06-04T05:23:49.672 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_grafana", 2024-06-04T05:23:49.672 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.672 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.672 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.672 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-06-04T05:23:49.672 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.672 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.672 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.672 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-06-04T05:23:49.673 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.673 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.673 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.673 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_haproxy": { 2024-06-04T05:23:49.673 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_haproxy", 2024-06-04T05:23:49.673 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.673 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.673 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.673 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-06-04T05:23:49.673 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.673 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.673 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.673 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "HAproxy container image", 2024-06-04T05:23:49.673 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.674 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.674 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.674 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_agent": { 2024-06-04T05:23:49.674 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_agent", 2024-06-04T05:23:49.674 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.674 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.674 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.674 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-06-04T05:23:49.674 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.674 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.674 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.674 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger agent container image", 2024-06-04T05:23:49.674 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.675 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.675 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.675 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_collector": { 2024-06-04T05:23:49.675 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_collector", 2024-06-04T05:23:49.675 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.675 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.675 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.675 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-06-04T05:23:49.675 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.675 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.675 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.675 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger collector container image", 2024-06-04T05:23:49.675 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.675 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.676 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.676 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_query": { 2024-06-04T05:23:49.676 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_query", 2024-06-04T05:23:49.676 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.676 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.676 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.676 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-06-04T05:23:49.676 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.676 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.676 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.676 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger query container image", 2024-06-04T05:23:49.676 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.676 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.676 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.677 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_keepalived": { 2024-06-04T05:23:49.677 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_keepalived", 2024-06-04T05:23:49.677 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.677 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.677 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.677 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-06-04T05:23:49.677 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.677 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.677 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.677 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Keepalived container image", 2024-06-04T05:23:49.677 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.677 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.677 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.678 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_loki": { 2024-06-04T05:23:49.678 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_loki", 2024-06-04T05:23:49.678 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.678 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.678 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.678 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-06-04T05:23:49.678 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.678 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.678 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.678 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Loki container image", 2024-06-04T05:23:49.678 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.678 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.678 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.679 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_node_exporter": { 2024-06-04T05:23:49.679 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_node_exporter", 2024-06-04T05:23:49.679 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.679 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.679 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.679 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-06-04T05:23:49.679 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.679 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.679 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.679 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-06-04T05:23:49.679 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.679 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.679 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.680 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_nvmeof": { 2024-06-04T05:23:49.680 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_nvmeof", 2024-06-04T05:23:49.680 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.680 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.680 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.680 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-06-04T05:23:49.680 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.680 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.680 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.680 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Nvme-of container image", 2024-06-04T05:23:49.680 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.680 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.680 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.680 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_prometheus": { 2024-06-04T05:23:49.681 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_prometheus", 2024-06-04T05:23:49.681 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.681 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.681 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.681 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-06-04T05:23:49.681 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.681 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.681 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.681 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-06-04T05:23:49.681 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.681 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.681 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.681 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_promtail": { 2024-06-04T05:23:49.681 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_promtail", 2024-06-04T05:23:49.682 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.682 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.682 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.682 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-06-04T05:23:49.682 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.682 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.682 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.682 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Promtail container image", 2024-06-04T05:23:49.682 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.682 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.682 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.682 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.682 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_samba": { 2024-06-04T05:23:49.682 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_samba", 2024-06-04T05:23:49.683 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.683 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.683 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.683 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-06-04T05:23:49.683 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.683 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.683 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.683 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Samba/SMB container image", 2024-06-04T05:23:49.683 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.683 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.683 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.683 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_snmp_gateway": { 2024-06-04T05:23:49.684 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_snmp_gateway", 2024-06-04T05:23:49.684 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.684 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.684 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.684 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-06-04T05:23:49.684 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.684 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.684 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.684 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SNMP Gateway container image", 2024-06-04T05:23:49.684 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.684 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.684 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.684 INFO:teuthology.orchestra.run.smithi070.stdout: "container_init": { 2024-06-04T05:23:49.684 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_init", 2024-06-04T05:23:49.685 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.685 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.685 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.685 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.685 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.685 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.685 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.685 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Run podman/docker with `--init`", 2024-06-04T05:23:49.685 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.685 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.685 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.685 INFO:teuthology.orchestra.run.smithi070.stdout: "daemon_cache_timeout": { 2024-06-04T05:23:49.685 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "daemon_cache_timeout", 2024-06-04T05:23:49.686 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.686 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.686 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.686 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-06-04T05:23:49.686 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.686 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.686 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.686 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-06-04T05:23:49.686 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.686 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.686 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.686 INFO:teuthology.orchestra.run.smithi070.stdout: "default_cephadm_command_timeout": { 2024-06-04T05:23:49.686 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "default_cephadm_command_timeout", 2024-06-04T05:23:49.687 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.687 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.687 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.687 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "900", 2024-06-04T05:23:49.687 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.687 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.687 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.687 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-06-04T05:23:49.687 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.687 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.687 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.687 INFO:teuthology.orchestra.run.smithi070.stdout: "default_registry": { 2024-06-04T05:23:49.687 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "default_registry", 2024-06-04T05:23:49.688 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.688 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.688 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.688 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io", 2024-06-04T05:23:49.688 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.688 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.688 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.688 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-06-04T05:23:49.688 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.688 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.688 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.688 INFO:teuthology.orchestra.run.smithi070.stdout: "device_cache_timeout": { 2024-06-04T05:23:49.688 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_cache_timeout", 2024-06-04T05:23:49.688 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.688 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.689 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.689 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1800", 2024-06-04T05:23:49.689 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.689 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.689 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.689 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache device inventory", 2024-06-04T05:23:49.689 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.689 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.689 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.689 INFO:teuthology.orchestra.run.smithi070.stdout: "device_enhanced_scan": { 2024-06-04T05:23:49.689 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_enhanced_scan", 2024-06-04T05:23:49.689 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.689 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.689 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.690 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.690 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.690 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.690 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.690 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use libstoragemgmt during device scans", 2024-06-04T05:23:49.690 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.690 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.690 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.690 INFO:teuthology.orchestra.run.smithi070.stdout: "facts_cache_timeout": { 2024-06-04T05:23:49.690 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "facts_cache_timeout", 2024-06-04T05:23:49.690 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.690 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.690 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.690 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-06-04T05:23:49.691 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.691 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.691 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.691 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache host facts data", 2024-06-04T05:23:49.691 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.691 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.691 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.691 INFO:teuthology.orchestra.run.smithi070.stdout: "grafana_dashboards_path": { 2024-06-04T05:23:49.691 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "grafana_dashboards_path", 2024-06-04T05:23:49.691 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.691 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.691 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.692 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-06-04T05:23:49.692 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.692 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.692 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.692 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-06-04T05:23:49.692 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.692 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.692 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.692 INFO:teuthology.orchestra.run.smithi070.stdout: "host_check_interval": { 2024-06-04T05:23:49.692 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "host_check_interval", 2024-06-04T05:23:49.692 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.692 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.692 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.692 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-06-04T05:23:49.693 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.693 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.693 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.693 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to perform a host check", 2024-06-04T05:23:49.693 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.693 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.693 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.693 INFO:teuthology.orchestra.run.smithi070.stdout: "hw_monitoring": { 2024-06-04T05:23:49.693 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "hw_monitoring", 2024-06-04T05:23:49.693 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.693 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.693 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.693 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.694 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.694 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.694 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.694 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-06-04T05:23:49.694 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.694 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.694 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.694 INFO:teuthology.orchestra.run.smithi070.stdout: "inventory_list_all": { 2024-06-04T05:23:49.694 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "inventory_list_all", 2024-06-04T05:23:49.694 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.694 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.694 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.694 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.695 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.695 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.695 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.695 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-06-04T05:23:49.695 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.695 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.695 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.695 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.695 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.695 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.695 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.695 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.696 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.696 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.696 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.696 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.696 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.696 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.696 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.696 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.696 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.696 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.696 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.696 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.696 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.696 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.697 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.697 INFO:teuthology.orchestra.run.smithi070.stdout: "log_refresh_metadata": { 2024-06-04T05:23:49.697 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_refresh_metadata", 2024-06-04T05:23:49.697 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.697 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.697 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.697 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.697 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.697 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.697 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.697 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-06-04T05:23:49.697 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.697 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.698 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.698 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.698 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.698 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.698 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.698 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.698 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.698 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.698 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.698 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.698 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-06-04T05:23:49.698 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.698 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.698 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.699 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.699 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.699 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.699 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.699 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.699 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.699 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.699 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.699 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.699 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.699 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.699 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.699 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.700 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.700 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.700 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.700 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.700 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.700 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.700 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.700 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.700 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.700 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.700 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.700 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.700 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.700 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.701 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.701 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.701 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.701 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.701 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.701 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.701 INFO:teuthology.orchestra.run.smithi070.stdout: "manage_etc_ceph_ceph_conf": { 2024-06-04T05:23:49.701 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-06-04T05:23:49.701 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.701 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.701 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.701 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.701 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.702 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.702 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.702 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-06-04T05:23:49.702 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.702 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.702 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.702 INFO:teuthology.orchestra.run.smithi070.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-06-04T05:23:49.702 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-06-04T05:23:49.702 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.702 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.702 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.702 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "*", 2024-06-04T05:23:49.702 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.703 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.703 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.703 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-06-04T05:23:49.703 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.703 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.703 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.703 INFO:teuthology.orchestra.run.smithi070.stdout: "max_count_per_host": { 2024-06-04T05:23:49.703 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_count_per_host", 2024-06-04T05:23:49.703 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.703 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.703 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.703 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:49.703 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.704 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.704 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.704 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "max number of daemons per service per host", 2024-06-04T05:23:49.704 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.704 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.704 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.704 INFO:teuthology.orchestra.run.smithi070.stdout: "max_osd_draining_count": { 2024-06-04T05:23:49.704 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_osd_draining_count", 2024-06-04T05:23:49.704 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.704 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.705 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.705 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:49.705 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.705 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.705 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.705 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-06-04T05:23:49.705 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.705 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.705 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.705 INFO:teuthology.orchestra.run.smithi070.stdout: "migration_current": { 2024-06-04T05:23:49.705 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "migration_current", 2024-06-04T05:23:49.705 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.706 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.706 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.706 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.706 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.706 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.706 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.706 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "internal - do not modify", 2024-06-04T05:23:49.706 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.706 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.706 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.706 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": { 2024-06-04T05:23:49.706 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mode", 2024-06-04T05:23:49.706 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.707 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.707 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.707 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "root", 2024-06-04T05:23:49.707 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.707 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.707 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.707 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm-package", 2024-06-04T05:23:49.707 INFO:teuthology.orchestra.run.smithi070.stdout: "root" 2024-06-04T05:23:49.707 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.707 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "mode for remote execution of cephadm", 2024-06-04T05:23:49.707 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.707 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.707 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.708 INFO:teuthology.orchestra.run.smithi070.stdout: "oob_default_addr": { 2024-06-04T05:23:49.708 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "oob_default_addr", 2024-06-04T05:23:49.708 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.708 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.708 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.708 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "169.254.1.1", 2024-06-04T05:23:49.708 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.708 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.708 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.708 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Default address for RedFish API (oob management).", 2024-06-04T05:23:49.708 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.708 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.708 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.709 INFO:teuthology.orchestra.run.smithi070.stdout: "prometheus_alerts_path": { 2024-06-04T05:23:49.709 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prometheus_alerts_path", 2024-06-04T05:23:49.709 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.709 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.709 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.709 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-06-04T05:23:49.709 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.709 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.709 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.709 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-06-04T05:23:49.709 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.709 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.709 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.710 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_insecure": { 2024-06-04T05:23:49.710 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_insecure", 2024-06-04T05:23:49.710 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.710 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.710 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.710 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.710 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.710 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.710 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.710 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-06-04T05:23:49.710 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.710 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.710 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.711 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_password": { 2024-06-04T05:23:49.711 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_password", 2024-06-04T05:23:49.711 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.711 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.711 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.711 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.711 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.711 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.711 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.711 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-06-04T05:23:49.711 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.711 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.711 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.712 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_url": { 2024-06-04T05:23:49.712 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_url", 2024-06-04T05:23:49.712 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.712 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.712 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.712 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.712 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.712 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.712 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.712 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-06-04T05:23:49.712 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.712 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.712 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.712 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_username": { 2024-06-04T05:23:49.713 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_username", 2024-06-04T05:23:49.713 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.713 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.713 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.713 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.713 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.713 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.713 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.713 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-06-04T05:23:49.713 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.713 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.713 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.713 INFO:teuthology.orchestra.run.smithi070.stdout: "secure_monitoring_stack": { 2024-06-04T05:23:49.713 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "secure_monitoring_stack", 2024-06-04T05:23:49.714 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.714 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.714 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.714 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.714 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.714 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.714 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.714 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-06-04T05:23:49.714 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.714 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.714 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.714 INFO:teuthology.orchestra.run.smithi070.stdout: "service_discovery_port": { 2024-06-04T05:23:49.715 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "service_discovery_port", 2024-06-04T05:23:49.715 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.715 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.715 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.715 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8765", 2024-06-04T05:23:49.715 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.715 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.715 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.715 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "cephadm service discovery port", 2024-06-04T05:23:49.715 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.715 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.715 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.716 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.716 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.716 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.716 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.716 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.716 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.716 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.716 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.716 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.716 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.716 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.716 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.716 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.716 INFO:teuthology.orchestra.run.smithi070.stdout: "ssh_config_file": { 2024-06-04T05:23:49.716 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssh_config_file", 2024-06-04T05:23:49.717 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.717 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.717 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.717 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.717 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.717 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.717 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.717 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-06-04T05:23:49.717 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.717 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.717 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.717 INFO:teuthology.orchestra.run.smithi070.stdout: "use_agent": { 2024-06-04T05:23:49.717 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "use_agent", 2024-06-04T05:23:49.717 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.718 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.718 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.718 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.718 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.718 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.718 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.718 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-06-04T05:23:49.718 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.718 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.718 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.718 INFO:teuthology.orchestra.run.smithi070.stdout: "use_repo_digest": { 2024-06-04T05:23:49.718 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "use_repo_digest", 2024-06-04T05:23:49.718 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.718 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.719 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.719 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.719 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.719 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.719 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.719 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-06-04T05:23:49.719 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.719 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.719 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.719 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_failed_host_check": { 2024-06-04T05:23:49.719 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_failed_host_check", 2024-06-04T05:23:49.719 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.720 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.720 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.720 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.720 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.720 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.720 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.720 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if the host check fails", 2024-06-04T05:23:49.720 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.720 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.720 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.720 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_stray_daemons": { 2024-06-04T05:23:49.720 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_stray_daemons", 2024-06-04T05:23:49.720 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.721 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.721 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.721 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.721 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.721 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.721 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.721 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-06-04T05:23:49.721 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.721 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.721 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.721 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_stray_hosts": { 2024-06-04T05:23:49.721 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_stray_hosts", 2024-06-04T05:23:49.721 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.722 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.722 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.722 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.722 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.722 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.722 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.722 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-06-04T05:23:49.722 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.722 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.722 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.722 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.722 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.723 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crash", 2024-06-04T05:23:49.723 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.723 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.723 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.723 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.723 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.723 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.723 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.723 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.723 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.723 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.724 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.724 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.724 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.724 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.724 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.724 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.724 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.724 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.724 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.724 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.724 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.724 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.724 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.725 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.725 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.725 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.725 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.725 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.725 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.725 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.725 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.725 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.725 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.725 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.725 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.725 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.726 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.726 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.726 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.726 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.726 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.726 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.726 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.726 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.726 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.726 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.726 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.726 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.727 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.727 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.727 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.727 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.727 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.727 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.727 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.727 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.727 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.727 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.727 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.727 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.727 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.727 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.728 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.728 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.728 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.728 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.728 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.728 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.728 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.728 INFO:teuthology.orchestra.run.smithi070.stdout: "retain_interval": { 2024-06-04T05:23:49.728 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "retain_interval", 2024-06-04T05:23:49.728 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.728 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.728 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.728 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "31536000", 2024-06-04T05:23:49.728 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.729 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.729 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.729 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long to retain crashes before pruning them", 2024-06-04T05:23:49.729 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.729 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.729 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.729 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.729 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.729 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.729 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.729 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.729 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.729 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.730 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.730 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.730 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.730 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.730 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.730 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.730 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_recent_interval": { 2024-06-04T05:23:49.730 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_recent_interval", 2024-06-04T05:23:49.730 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.730 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.730 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.730 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1209600", 2024-06-04T05:23:49.731 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.733 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.734 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.734 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "time interval in which to warn about recent crashes", 2024-06-04T05:23:49.734 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.734 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.734 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.734 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.734 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.734 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "dashboard", 2024-06-04T05:23:49.734 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.734 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.734 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.735 INFO:teuthology.orchestra.run.smithi070.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-06-04T05:23:49.735 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-06-04T05:23:49.735 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.735 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.735 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.735 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:49.735 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.735 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.735 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.735 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.735 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.735 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.736 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.736 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.736 INFO:teuthology.orchestra.run.smithi070.stdout: "ALERTMANAGER_API_HOST": { 2024-06-04T05:23:49.736 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ALERTMANAGER_API_HOST", 2024-06-04T05:23:49.736 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.736 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.736 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.736 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.736 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.736 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.736 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.736 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.736 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.736 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.736 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.736 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.737 INFO:teuthology.orchestra.run.smithi070.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-06-04T05:23:49.737 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-06-04T05:23:49.737 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.737 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.737 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.737 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.737 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.737 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.737 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.737 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.737 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.737 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.737 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.738 INFO:teuthology.orchestra.run.smithi070.stdout: "AUDIT_API_ENABLED": { 2024-06-04T05:23:49.738 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "AUDIT_API_ENABLED", 2024-06-04T05:23:49.738 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.738 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.738 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.738 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.738 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.738 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.738 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.738 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.738 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.738 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.738 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.739 INFO:teuthology.orchestra.run.smithi070.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-06-04T05:23:49.739 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-06-04T05:23:49.739 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.739 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.739 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.739 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.739 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.739 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.739 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.739 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.739 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.739 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.739 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.740 INFO:teuthology.orchestra.run.smithi070.stdout: "ENABLE_BROWSABLE_API": { 2024-06-04T05:23:49.740 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ENABLE_BROWSABLE_API", 2024-06-04T05:23:49.740 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.740 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.740 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.740 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.740 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.740 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.740 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.740 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.740 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.740 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.740 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.741 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-06-04T05:23:49.741 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-06-04T05:23:49.741 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.741 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.741 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.741 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.741 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.741 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.741 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.741 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.741 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.741 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.741 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.741 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-06-04T05:23:49.742 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-06-04T05:23:49.742 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.742 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.742 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.742 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.742 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.742 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.742 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.742 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.742 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.742 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.742 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.742 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.742 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-06-04T05:23:49.742 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-06-04T05:23:49.743 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.743 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.743 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.743 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.743 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.743 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.743 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.743 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.743 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.743 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.743 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.743 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-06-04T05:23:49.743 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-06-04T05:23:49.744 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.744 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.744 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.744 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.744 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.744 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.744 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.744 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.744 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.744 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.744 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.744 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_NFS": { 2024-06-04T05:23:49.744 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-06-04T05:23:49.745 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.745 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.745 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.745 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.745 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.745 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.745 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.745 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.745 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.745 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.745 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.745 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_RBD": { 2024-06-04T05:23:49.745 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-06-04T05:23:49.745 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.746 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.746 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.746 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.746 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.746 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.746 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.746 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.746 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.746 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.746 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.746 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.746 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_RGW": { 2024-06-04T05:23:49.746 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-06-04T05:23:49.746 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.747 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.747 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.747 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.747 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.747 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.747 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.747 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.747 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.747 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.747 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.747 INFO:teuthology.orchestra.run.smithi070.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-06-04T05:23:49.747 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-06-04T05:23:49.748 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.748 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.748 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.748 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.748 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.748 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.748 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.748 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.748 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.748 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.748 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.748 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_PASSWORD": { 2024-06-04T05:23:49.749 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_PASSWORD", 2024-06-04T05:23:49.749 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.749 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.749 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.749 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-06-04T05:23:49.749 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.749 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.749 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.749 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.749 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.749 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.749 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.749 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.750 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-06-04T05:23:49.750 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-06-04T05:23:49.750 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.750 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.750 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.750 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.750 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.750 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.750 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.750 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.750 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.750 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.750 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.750 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.751 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_URL": { 2024-06-04T05:23:49.751 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_URL", 2024-06-04T05:23:49.751 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.751 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.751 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.751 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.751 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.751 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.751 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.751 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.751 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.751 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.751 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.752 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_USERNAME": { 2024-06-04T05:23:49.752 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_USERNAME", 2024-06-04T05:23:49.752 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.752 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.752 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.752 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-06-04T05:23:49.752 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.752 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.752 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.752 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.752 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.752 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.752 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.753 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-06-04T05:23:49.753 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-06-04T05:23:49.753 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.753 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.753 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.753 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.753 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.753 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.753 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.753 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.753 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.753 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.753 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.754 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-06-04T05:23:49.754 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-06-04T05:23:49.754 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.754 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.754 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.754 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.754 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.754 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.754 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.754 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.754 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.754 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.754 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.755 INFO:teuthology.orchestra.run.smithi070.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-06-04T05:23:49.755 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-06-04T05:23:49.755 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.755 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.755 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.755 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.755 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.755 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.755 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.755 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.755 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.755 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.756 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.756 INFO:teuthology.orchestra.run.smithi070.stdout: "ISSUE_TRACKER_API_KEY": { 2024-06-04T05:23:49.756 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-06-04T05:23:49.756 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.756 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.756 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.756 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.756 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.756 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.756 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.756 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.756 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.756 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.757 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.757 INFO:teuthology.orchestra.run.smithi070.stdout: "MANAGED_BY_CLUSTERS": { 2024-06-04T05:23:49.757 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-06-04T05:23:49.757 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.757 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.757 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.757 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "[]", 2024-06-04T05:23:49.757 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.757 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.757 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.757 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.757 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.757 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: "MULTICLUSTER_CONFIG": { 2024-06-04T05:23:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "MULTICLUSTER_CONFIG", 2024-06-04T05:23:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "{}", 2024-06-04T05:23:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: "PROMETHEUS_API_HOST": { 2024-06-04T05:23:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PROMETHEUS_API_HOST", 2024-06-04T05:23:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-06-04T05:23:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-06-04T05:23:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-06-04T05:23:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-06-04T05:23:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.762 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.762 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.762 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-06-04T05:23:49.762 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-06-04T05:23:49.762 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.762 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.762 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.762 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.762 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.762 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.762 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.762 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.762 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.763 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.763 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.763 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-06-04T05:23:49.763 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-06-04T05:23:49.763 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.763 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.763 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.763 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.763 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.763 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.763 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.763 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.763 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.764 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.764 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.764 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.764 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-06-04T05:23:49.764 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-06-04T05:23:49.764 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.764 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.764 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.764 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.764 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.764 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.764 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.764 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.765 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.765 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.765 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.765 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-06-04T05:23:49.765 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-06-04T05:23:49.765 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.765 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.765 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.765 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.765 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.765 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.765 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.765 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.765 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.766 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.766 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.766 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-06-04T05:23:49.766 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-06-04T05:23:49.766 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.766 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.766 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.766 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.766 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.766 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.766 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.766 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.766 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.767 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.767 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.767 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-06-04T05:23:49.767 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-06-04T05:23:49.767 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.767 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.767 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.767 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.767 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.767 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.767 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.767 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.767 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.768 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.768 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.768 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.768 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_ENABLED": { 2024-06-04T05:23:49.768 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_ENABLED", 2024-06-04T05:23:49.768 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.768 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.768 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.768 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.768 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.768 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.768 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.768 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.768 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.769 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.769 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.769 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-06-04T05:23:49.769 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-06-04T05:23:49.769 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.769 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.769 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.769 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-06-04T05:23:49.769 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.769 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.769 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.769 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.769 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.770 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.770 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.770 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-06-04T05:23:49.770 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-06-04T05:23:49.770 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.770 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.770 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.770 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:49.770 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.770 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.770 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.770 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.771 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.771 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.771 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.771 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.771 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-06-04T05:23:49.771 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-06-04T05:23:49.771 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.771 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.771 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.771 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8", 2024-06-04T05:23:49.771 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.771 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.771 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.771 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.772 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.772 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.772 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.772 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.772 INFO:teuthology.orchestra.run.smithi070.stdout: "REST_REQUESTS_TIMEOUT": { 2024-06-04T05:23:49.772 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-06-04T05:23:49.772 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.772 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.772 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.772 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "45", 2024-06-04T05:23:49.772 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.772 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.772 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.772 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.773 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.773 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.773 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.773 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_ACCESS_KEY": { 2024-06-04T05:23:49.773 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_ACCESS_KEY", 2024-06-04T05:23:49.773 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.773 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.773 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.773 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.773 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.773 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.773 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.773 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.774 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.774 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.774 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.774 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.774 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-06-04T05:23:49.774 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-06-04T05:23:49.774 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.774 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.774 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.774 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-06-04T05:23:49.774 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.774 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.774 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.774 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.775 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.775 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.775 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.775 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.775 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_SECRET_KEY": { 2024-06-04T05:23:49.775 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_SECRET_KEY", 2024-06-04T05:23:49.775 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.775 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.775 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.775 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.775 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.775 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.775 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.775 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.776 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.776 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.776 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.776 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.776 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_SSL_VERIFY": { 2024-06-04T05:23:49.776 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_SSL_VERIFY", 2024-06-04T05:23:49.776 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.776 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.776 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.776 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.776 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.776 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.776 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.777 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.777 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.777 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.777 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.777 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.777 INFO:teuthology.orchestra.run.smithi070.stdout: "UNSAFE_TLS_v1_2": { 2024-06-04T05:23:49.777 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "UNSAFE_TLS_v1_2", 2024-06-04T05:23:49.777 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.777 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.777 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.777 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.777 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.777 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.777 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.777 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.778 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.778 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.778 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.778 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.778 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-06-04T05:23:49.778 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-06-04T05:23:49.778 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.778 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.778 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.778 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.778 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.778 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.778 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.779 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.779 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.779 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.779 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.779 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.779 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-06-04T05:23:49.779 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-06-04T05:23:49.779 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.779 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.779 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.779 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:49.779 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.779 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.779 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.780 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.780 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.780 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.780 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.780 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.780 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-06-04T05:23:49.780 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-06-04T05:23:49.780 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.780 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.780 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.780 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-06-04T05:23:49.780 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.780 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.781 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.781 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.781 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.781 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.781 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.781 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.781 INFO:teuthology.orchestra.run.smithi070.stdout: "cross_origin_url": { 2024-06-04T05:23:49.781 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cross_origin_url", 2024-06-04T05:23:49.781 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.781 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.781 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.781 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.781 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.781 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.782 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.782 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.782 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.782 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.782 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.782 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.782 INFO:teuthology.orchestra.run.smithi070.stdout: "crt_file": { 2024-06-04T05:23:49.782 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crt_file", 2024-06-04T05:23:49.782 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.782 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.782 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.782 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.782 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.782 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.783 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.783 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.783 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.783 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.783 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.783 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.783 INFO:teuthology.orchestra.run.smithi070.stdout: "debug": { 2024-06-04T05:23:49.783 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "debug", 2024-06-04T05:23:49.783 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.783 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.783 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.783 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.783 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.783 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.784 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.784 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable/disable debug options", 2024-06-04T05:23:49.784 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.784 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.784 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.784 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.784 INFO:teuthology.orchestra.run.smithi070.stdout: "jwt_token_ttl": { 2024-06-04T05:23:49.784 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "jwt_token_ttl", 2024-06-04T05:23:49.784 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.784 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.784 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.784 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "28800", 2024-06-04T05:23:49.784 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.784 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.785 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.785 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.785 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.785 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.785 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.785 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.785 INFO:teuthology.orchestra.run.smithi070.stdout: "key_file": { 2024-06-04T05:23:49.785 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "key_file", 2024-06-04T05:23:49.785 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.785 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.785 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.785 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.785 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.785 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.785 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.786 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.786 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.786 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.786 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.786 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.786 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.786 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.786 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.786 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.786 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.786 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.786 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.786 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.787 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.787 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.787 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.787 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.787 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.787 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.787 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.787 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.787 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.787 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.787 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.787 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.787 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.787 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.788 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.788 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.788 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.788 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.788 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.788 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.788 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.788 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.788 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.788 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.788 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.788 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.788 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.788 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.789 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.789 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.789 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.789 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.789 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.789 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.789 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.789 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.789 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.789 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.789 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.789 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.789 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.789 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.789 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.790 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.790 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.790 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.790 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.790 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.790 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.790 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.790 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.790 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.790 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.790 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.790 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.790 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.791 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.791 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.791 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.791 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.791 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.791 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.791 INFO:teuthology.orchestra.run.smithi070.stdout: "motd": { 2024-06-04T05:23:49.791 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "motd", 2024-06-04T05:23:49.791 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.791 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.791 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.791 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.791 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.792 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.792 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.792 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "The message of the day", 2024-06-04T05:23:49.792 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.792 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.792 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.792 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.792 INFO:teuthology.orchestra.run.smithi070.stdout: "redirect_resolve_ip_addr": { 2024-06-04T05:23:49.792 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "redirect_resolve_ip_addr", 2024-06-04T05:23:49.792 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.792 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.792 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.792 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.792 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.793 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.793 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.793 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.793 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.793 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.793 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.793 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.793 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-06-04T05:23:49.793 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-06-04T05:23:49.793 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.793 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.793 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.793 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "::", 2024-06-04T05:23:49.793 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.794 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.794 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.794 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.794 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.794 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.794 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.794 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.794 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-06-04T05:23:49.794 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-06-04T05:23:49.794 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.794 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.794 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.794 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8080", 2024-06-04T05:23:49.795 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.795 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.795 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.795 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.795 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.795 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.795 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.795 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.795 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.795 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.795 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.795 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.795 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.795 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.796 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.796 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.796 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.796 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.796 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.796 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.796 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.796 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl": { 2024-06-04T05:23:49.796 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl", 2024-06-04T05:23:49.796 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.796 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.796 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.796 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.797 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.797 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.797 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.797 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.797 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.797 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.797 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.797 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.797 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl_server_port": { 2024-06-04T05:23:49.797 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl_server_port", 2024-06-04T05:23:49.797 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.797 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.798 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.798 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8443", 2024-06-04T05:23:49.798 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.798 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.798 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.798 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.798 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.798 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.798 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.798 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.798 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_behaviour": { 2024-06-04T05:23:49.798 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_behaviour", 2024-06-04T05:23:49.798 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.798 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.799 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.799 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "redirect", 2024-06-04T05:23:49.799 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.799 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.799 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.799 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.799 INFO:teuthology.orchestra.run.smithi070.stdout: "redirect" 2024-06-04T05:23:49.799 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.799 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.799 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.799 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.799 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.799 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.799 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_error_status_code": { 2024-06-04T05:23:49.800 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_error_status_code", 2024-06-04T05:23:49.800 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.800 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.800 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.800 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "500", 2024-06-04T05:23:49.800 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "400", 2024-06-04T05:23:49.800 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "599", 2024-06-04T05:23:49.800 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.800 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.800 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.800 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.800 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.800 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.800 INFO:teuthology.orchestra.run.smithi070.stdout: "url_prefix": { 2024-06-04T05:23:49.800 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "url_prefix", 2024-06-04T05:23:49.801 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.801 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.801 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.801 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.801 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.801 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.801 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.801 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.801 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.801 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.801 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.801 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.801 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.801 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.802 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.802 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "devicehealth", 2024-06-04T05:23:49.802 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.802 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.802 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.802 INFO:teuthology.orchestra.run.smithi070.stdout: "enable_monitoring": { 2024-06-04T05:23:49.802 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enable_monitoring", 2024-06-04T05:23:49.802 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.802 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.802 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.802 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.802 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.802 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.803 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.803 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "monitor device health metrics", 2024-06-04T05:23:49.803 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.803 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.803 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.803 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.803 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.803 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.803 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.803 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.803 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.803 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.803 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.803 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.804 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.804 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.804 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.804 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.804 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.804 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.804 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.804 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.804 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.804 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.804 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.804 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.804 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.804 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.805 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.805 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.805 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.805 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.805 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.805 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.805 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.805 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.805 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.805 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.805 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.805 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.805 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.805 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.805 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.806 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.806 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.806 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.806 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.806 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.806 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.806 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.806 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.806 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.806 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.806 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.806 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.806 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.807 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.807 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.807 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.807 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.807 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.807 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.807 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.807 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.807 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.807 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.807 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.807 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.807 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.807 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.808 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.808 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.808 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.808 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.808 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.808 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.808 INFO:teuthology.orchestra.run.smithi070.stdout: "mark_out_threshold": { 2024-06-04T05:23:49.808 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mark_out_threshold", 2024-06-04T05:23:49.808 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.808 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.808 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.808 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "2419200", 2024-06-04T05:23:49.808 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.808 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.809 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.809 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-06-04T05:23:49.809 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.809 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.809 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.809 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.809 INFO:teuthology.orchestra.run.smithi070.stdout: "pool_name": { 2024-06-04T05:23:49.809 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pool_name", 2024-06-04T05:23:49.809 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.809 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.809 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.809 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "device_health_metrics", 2024-06-04T05:23:49.809 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.809 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.810 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.810 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "name of pool in which to store device health metrics", 2024-06-04T05:23:49.810 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.810 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.810 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.810 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.810 INFO:teuthology.orchestra.run.smithi070.stdout: "retention_period": { 2024-06-04T05:23:49.810 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "retention_period", 2024-06-04T05:23:49.810 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.810 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.810 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.810 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15552000", 2024-06-04T05:23:49.810 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.811 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.811 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.811 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long to retain device health metrics", 2024-06-04T05:23:49.811 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.811 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.811 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.811 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.811 INFO:teuthology.orchestra.run.smithi070.stdout: "scrape_frequency": { 2024-06-04T05:23:49.811 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "scrape_frequency", 2024-06-04T05:23:49.811 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.811 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.811 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.811 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "86400", 2024-06-04T05:23:49.811 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.812 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.812 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.812 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to scrape device health metrics", 2024-06-04T05:23:49.812 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.812 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.812 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.812 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.812 INFO:teuthology.orchestra.run.smithi070.stdout: "self_heal": { 2024-06-04T05:23:49.812 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "self_heal", 2024-06-04T05:23:49.812 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.812 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.812 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.812 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.812 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.813 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.813 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.813 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-06-04T05:23:49.813 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.813 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.813 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.813 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.813 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-06-04T05:23:49.813 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-06-04T05:23:49.813 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.813 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.813 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.813 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-06-04T05:23:49.813 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.814 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.814 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.814 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to wake up and check device health", 2024-06-04T05:23:49.814 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.814 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.814 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.814 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.814 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.814 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.814 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.814 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.814 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.815 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.815 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.815 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.815 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.815 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.815 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.815 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.815 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.815 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.815 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_threshold": { 2024-06-04T05:23:49.815 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_threshold", 2024-06-04T05:23:49.815 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.815 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.815 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.816 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "7257600", 2024-06-04T05:23:49.816 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.816 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.816 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.816 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-06-04T05:23:49.816 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.816 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.816 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.816 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.816 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.816 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.816 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.816 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "diskprediction_local", 2024-06-04T05:23:49.816 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.817 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.817 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.817 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.817 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.817 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.817 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.817 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.817 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.817 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.817 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.817 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.817 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.817 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.817 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.817 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.818 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.818 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.818 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.818 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.818 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.818 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.818 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.818 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.818 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.818 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.818 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.818 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.818 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.818 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.819 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.819 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.819 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.819 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.819 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.819 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.819 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.819 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.819 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.819 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.819 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.819 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.819 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.819 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.820 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.820 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.820 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.820 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.820 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.820 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.820 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.820 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.820 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.820 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.820 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.820 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.820 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.820 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.820 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.821 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.821 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.821 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.821 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.821 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.821 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.821 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.821 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.821 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.821 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.821 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.821 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.821 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.822 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.822 INFO:teuthology.orchestra.run.smithi070.stdout: "predict_interval": { 2024-06-04T05:23:49.822 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "predict_interval", 2024-06-04T05:23:49.822 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.822 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.822 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.822 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "86400", 2024-06-04T05:23:49.822 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.822 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.822 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.822 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.822 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.822 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.822 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: "predictor_model": { 2024-06-04T05:23:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "predictor_model", 2024-06-04T05:23:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "prophetstor", 2024-06-04T05:23:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-06-04T05:23:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-06-04T05:23:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-06-04T05:23:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.825 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.825 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.825 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.825 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.825 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.825 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.825 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.825 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.825 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.825 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.825 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.825 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.825 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.826 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.826 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.826 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.826 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.826 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.826 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "influx", 2024-06-04T05:23:49.826 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": false, 2024-06-04T05:23:49.826 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "influxdb python module not found", 2024-06-04T05:23:49.826 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.826 INFO:teuthology.orchestra.run.smithi070.stdout: "batch_size": { 2024-06-04T05:23:49.826 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "batch_size", 2024-06-04T05:23:49.826 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.826 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.826 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.827 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5000", 2024-06-04T05:23:49.827 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.827 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.827 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.827 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-06-04T05:23:49.827 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.827 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.827 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.827 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.827 INFO:teuthology.orchestra.run.smithi070.stdout: "database": { 2024-06-04T05:23:49.827 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "database", 2024-06-04T05:23:49.827 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.827 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.827 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.828 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "ceph", 2024-06-04T05:23:49.828 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.828 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.828 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.828 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-06-04T05:23:49.828 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.828 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.828 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.828 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.828 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": { 2024-06-04T05:23:49.828 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "hostname", 2024-06-04T05:23:49.828 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.828 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.828 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.829 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.829 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.829 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.829 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.829 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB server hostname", 2024-06-04T05:23:49.829 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.829 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.829 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.829 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.829 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-06-04T05:23:49.829 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-06-04T05:23:49.829 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.829 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.829 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.830 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "30", 2024-06-04T05:23:49.830 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "5", 2024-06-04T05:23:49.830 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.830 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.830 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-06-04T05:23:49.830 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.830 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.830 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.830 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.830 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.830 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.830 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.830 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.831 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.831 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.831 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.831 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.831 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.831 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.831 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.831 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.831 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.831 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.831 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.831 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.831 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.831 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.832 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.832 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.832 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.832 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.832 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.832 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.832 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.832 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.832 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.832 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.832 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.832 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.832 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.832 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.833 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.833 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.833 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.833 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.833 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.833 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.833 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.833 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.833 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.833 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.833 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.833 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.833 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.834 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.834 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.834 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.834 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.834 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.834 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.834 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.834 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.834 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.834 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.834 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.834 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.834 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.834 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.835 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.835 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.835 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.835 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.835 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.835 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.835 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.835 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.835 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.835 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.835 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.835 INFO:teuthology.orchestra.run.smithi070.stdout: "password": { 2024-06-04T05:23:49.835 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "password", 2024-06-04T05:23:49.835 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.836 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.836 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.836 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.836 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.836 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.836 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.836 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "password of InfluxDB server user", 2024-06-04T05:23:49.836 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.836 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.836 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.836 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.836 INFO:teuthology.orchestra.run.smithi070.stdout: "port": { 2024-06-04T05:23:49.836 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "port", 2024-06-04T05:23:49.836 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.837 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.837 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.837 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8086", 2024-06-04T05:23:49.837 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.837 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.837 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.837 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB server port", 2024-06-04T05:23:49.837 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.837 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.837 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.837 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.837 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.837 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.838 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.838 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.838 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.838 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.838 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.838 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.838 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.838 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.838 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.838 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.838 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.838 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.838 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl": { 2024-06-04T05:23:49.838 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl", 2024-06-04T05:23:49.838 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.839 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.839 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.839 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "false", 2024-06-04T05:23:49.839 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.839 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.839 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.839 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-06-04T05:23:49.839 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.839 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.839 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.839 INFO:teuthology.orchestra.run.smithi070.stdout: "threads": { 2024-06-04T05:23:49.839 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "threads", 2024-06-04T05:23:49.839 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.840 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.840 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.840 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-06-04T05:23:49.840 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-06-04T05:23:49.840 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "32", 2024-06-04T05:23:49.840 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.840 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-06-04T05:23:49.840 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.840 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.840 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.840 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.840 INFO:teuthology.orchestra.run.smithi070.stdout: "username": { 2024-06-04T05:23:49.840 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "username", 2024-06-04T05:23:49.840 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.841 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.841 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.841 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.841 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.841 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.841 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.841 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "username of InfluxDB server user", 2024-06-04T05:23:49.841 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.841 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.841 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.841 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.841 INFO:teuthology.orchestra.run.smithi070.stdout: "verify_ssl": { 2024-06-04T05:23:49.841 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "verify_ssl", 2024-06-04T05:23:49.841 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.842 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.842 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.842 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "true", 2024-06-04T05:23:49.842 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.842 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.842 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.842 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-06-04T05:23:49.842 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.842 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.842 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.842 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.842 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.843 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "insights", 2024-06-04T05:23:49.843 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.843 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.843 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.843 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.843 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.843 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.843 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.843 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.843 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.843 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.843 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.843 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.844 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.844 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.844 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.844 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.844 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.844 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.844 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.844 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.844 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.844 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.844 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.844 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.844 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.845 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.845 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.845 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.845 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.845 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.845 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.845 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.845 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.845 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.845 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.845 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.845 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.846 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.846 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.846 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.846 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.846 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.846 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.846 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.846 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.846 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.846 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.846 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.846 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.846 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.846 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.847 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.847 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.847 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.847 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.847 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.847 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.847 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.847 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.847 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.847 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.847 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.847 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.848 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.848 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.848 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.848 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.848 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.848 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.848 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.848 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.848 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.848 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.848 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.848 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.848 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.849 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.849 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.849 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.849 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.849 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.849 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.849 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.849 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.849 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "iostat", 2024-06-04T05:23:49.849 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.849 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.850 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.850 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.850 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.850 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.850 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.850 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.850 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.850 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.850 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.850 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.850 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.850 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.850 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.850 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.851 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.851 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.851 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.851 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.851 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.851 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.851 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.851 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.851 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.851 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.851 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.851 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.851 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.852 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.852 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.852 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.852 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.852 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.852 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.852 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.852 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.852 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.852 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.852 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.852 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.852 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.853 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.853 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.853 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.853 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.853 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.853 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.853 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.853 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.853 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.853 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.853 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.853 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.853 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.853 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.854 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.854 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.854 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.854 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.854 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.854 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.854 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.854 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.854 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.854 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.854 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.854 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.855 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.855 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.855 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.855 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.855 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.855 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.855 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.855 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.855 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.855 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.855 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.855 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.855 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.855 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.856 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.856 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.856 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "k8sevents", 2024-06-04T05:23:49.856 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.856 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.856 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.856 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_event_retention_days": { 2024-06-04T05:23:49.856 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ceph_event_retention_days", 2024-06-04T05:23:49.856 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.856 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.856 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.856 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "7", 2024-06-04T05:23:49.857 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.857 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.857 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.857 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Days to hold ceph event information within local cache", 2024-06-04T05:23:49.857 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.857 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.857 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.857 INFO:teuthology.orchestra.run.smithi070.stdout: "config_check_secs": { 2024-06-04T05:23:49.857 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_check_secs", 2024-06-04T05:23:49.857 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.857 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.857 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.857 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:49.857 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "10", 2024-06-04T05:23:49.858 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.858 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.858 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-06-04T05:23:49.858 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.858 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.858 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.858 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.858 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.858 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.858 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.858 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.858 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.858 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.859 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.859 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.859 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.859 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.859 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.859 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.859 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.859 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.859 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.859 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.859 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.859 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.859 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.859 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.860 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.860 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.860 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.860 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.860 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.860 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.860 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.860 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.860 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.860 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.860 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.860 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.861 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.861 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.861 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.861 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.861 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.861 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.861 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.861 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.861 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.861 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.861 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.861 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.861 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.861 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.862 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.862 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.862 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.862 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.862 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.862 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.862 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.862 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.862 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.862 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.862 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.862 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.862 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.863 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.863 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.863 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.863 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.863 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.863 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.863 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.863 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.863 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.863 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.863 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.863 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.864 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.864 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.864 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.864 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.864 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.864 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.864 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.864 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.864 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.864 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.864 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.864 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.864 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "localpool", 2024-06-04T05:23:49.865 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.865 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.865 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.865 INFO:teuthology.orchestra.run.smithi070.stdout: "failure_domain": { 2024-06-04T05:23:49.865 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "failure_domain", 2024-06-04T05:23:49.865 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.865 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.865 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.865 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "host", 2024-06-04T05:23:49.865 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.865 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.865 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.865 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "failure domain for any created local pool", 2024-06-04T05:23:49.865 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-06-04T05:23:49.866 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.866 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.866 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.866 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.866 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.866 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.866 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.866 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.866 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.866 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.866 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.866 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.866 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.866 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.867 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.867 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.867 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.867 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.867 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.867 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.867 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.867 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.867 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.867 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.867 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.867 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.867 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.868 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.868 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.868 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.868 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.868 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.868 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.868 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.868 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.868 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.868 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.868 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.868 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.868 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.869 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.869 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.869 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.869 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.869 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.869 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.869 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.869 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.869 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.869 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.869 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.869 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.869 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.869 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.870 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.870 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.870 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.870 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.870 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.870 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.870 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.870 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.870 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.870 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.870 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.870 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.870 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.871 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.871 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.871 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.871 INFO:teuthology.orchestra.run.smithi070.stdout: "min_size": { 2024-06-04T05:23:49.871 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "min_size", 2024-06-04T05:23:49.871 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.871 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.871 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.871 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.871 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.871 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.871 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.871 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default min_size for any created local pool", 2024-06-04T05:23:49.871 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-06-04T05:23:49.872 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.872 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.872 INFO:teuthology.orchestra.run.smithi070.stdout: "num_rep": { 2024-06-04T05:23:49.872 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "num_rep", 2024-06-04T05:23:49.872 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.872 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.872 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.872 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3", 2024-06-04T05:23:49.872 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.872 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.872 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.872 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default replica count for any created local pool", 2024-06-04T05:23:49.872 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.873 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.873 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.873 INFO:teuthology.orchestra.run.smithi070.stdout: "pg_num": { 2024-06-04T05:23:49.873 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pg_num", 2024-06-04T05:23:49.873 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.873 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.873 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.873 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "128", 2024-06-04T05:23:49.873 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.873 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.873 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.873 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default pg_num for any created local pool", 2024-06-04T05:23:49.873 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.874 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.874 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.874 INFO:teuthology.orchestra.run.smithi070.stdout: "prefix": { 2024-06-04T05:23:49.874 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prefix", 2024-06-04T05:23:49.874 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.874 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.874 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.874 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.874 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.874 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.874 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.874 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "name prefix for any created local pool", 2024-06-04T05:23:49.874 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.875 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.875 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.875 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.875 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.875 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.875 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.875 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.875 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.875 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.875 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.875 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.875 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.875 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.875 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.876 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.876 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.876 INFO:teuthology.orchestra.run.smithi070.stdout: "subtree": { 2024-06-04T05:23:49.876 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "subtree", 2024-06-04T05:23:49.876 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.876 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.876 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.876 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "rack", 2024-06-04T05:23:49.876 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.876 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.876 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.876 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "CRUSH level for which to create a local pool", 2024-06-04T05:23:49.877 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-06-04T05:23:49.877 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.877 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.877 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.877 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.877 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mds_autoscaler", 2024-06-04T05:23:49.877 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.877 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.877 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.877 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.877 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.877 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.878 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.878 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.878 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.878 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.878 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.878 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.878 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.878 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.878 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.878 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.878 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.878 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.878 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.879 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.879 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.879 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.879 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.879 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.879 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.879 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.879 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.879 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.879 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.879 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.879 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.879 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.880 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.880 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.880 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.880 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.880 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.880 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.880 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.880 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.880 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.880 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.880 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.880 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.880 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.880 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.881 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.881 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.881 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.881 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.881 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.881 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.881 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.881 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.881 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.881 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.881 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.881 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.882 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.882 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.882 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.882 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.882 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.882 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.882 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.882 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.882 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.882 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.882 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.882 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.882 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.882 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.883 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.883 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.883 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.883 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.883 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.883 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.883 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.883 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.883 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.883 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.883 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.883 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.883 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.884 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mirroring", 2024-06-04T05:23:49.884 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.884 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.884 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.884 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.884 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.884 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.884 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.884 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.884 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.884 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.884 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.884 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.884 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.884 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.885 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.885 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.885 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.885 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.885 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.885 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.885 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.885 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.885 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.885 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.885 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.885 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.885 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.886 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.886 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.886 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.886 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.886 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.886 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.886 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.886 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.886 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.886 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.886 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.886 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.886 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.886 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.887 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.887 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.887 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.887 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.887 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.887 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.887 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.887 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.887 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.887 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.887 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.887 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.887 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.887 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.888 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.888 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.888 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.888 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.888 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.888 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.888 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.888 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.888 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.888 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.888 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.888 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.889 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.889 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.889 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.889 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.889 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.889 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.889 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.889 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.889 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.889 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.889 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.889 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.889 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.890 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.890 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.890 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.890 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.890 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.890 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "nfs", 2024-06-04T05:23:49.890 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.890 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.890 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.890 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.890 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.891 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.891 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.891 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.891 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.891 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.891 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.891 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.891 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.891 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.891 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.891 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.891 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.891 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.891 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.892 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.892 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.892 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.892 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.892 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.892 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.892 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.892 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.892 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.892 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.892 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.892 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.892 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.893 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.893 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.893 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.893 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.893 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.893 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.893 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.893 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.893 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.893 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.893 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.893 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.894 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.894 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.894 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.894 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.894 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.894 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.894 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.894 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.894 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.894 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.894 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.894 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.894 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.895 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.895 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.895 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.895 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.895 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.895 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.895 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.895 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.895 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.895 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.895 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.895 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.895 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.896 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.896 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.896 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.896 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.896 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.896 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.896 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.896 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.896 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.896 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.896 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.896 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.896 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.896 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.897 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "orchestrator", 2024-06-04T05:23:49.897 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.897 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.897 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.897 INFO:teuthology.orchestra.run.smithi070.stdout: "fail_fs": { 2024-06-04T05:23:49.897 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "fail_fs", 2024-06-04T05:23:49.897 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.897 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.897 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.897 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.897 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.897 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.897 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.898 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-06-04T05:23:49.898 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.898 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.898 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.898 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.898 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.898 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.898 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.898 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.898 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.898 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.898 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.898 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.898 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.899 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.899 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.899 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.899 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.899 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.899 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.899 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.899 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.899 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.899 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.899 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.899 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.899 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.900 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.900 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.900 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.900 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.900 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.900 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.900 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.900 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.900 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.900 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.900 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.900 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.900 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.901 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.901 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.901 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.901 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.901 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.901 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.901 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.901 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.901 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.901 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.901 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.901 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.901 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.901 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.901 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.902 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.902 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.902 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.902 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.902 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.902 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.902 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.902 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.902 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.902 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.902 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.902 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.902 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.903 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.903 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.903 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.903 INFO:teuthology.orchestra.run.smithi070.stdout: "orchestrator": { 2024-06-04T05:23:49.903 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "orchestrator", 2024-06-04T05:23:49.903 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.903 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.903 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.903 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.903 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.903 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.903 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.904 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm", 2024-06-04T05:23:49.904 INFO:teuthology.orchestra.run.smithi070.stdout: "rook", 2024-06-04T05:23:49.904 INFO:teuthology.orchestra.run.smithi070.stdout: "test_orchestrator" 2024-06-04T05:23:49.904 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.904 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Orchestrator backend", 2024-06-04T05:23:49.904 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.904 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.904 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.904 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.904 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.904 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.904 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.904 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.905 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.905 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.905 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.905 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.905 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.905 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.905 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.905 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.905 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.905 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.905 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "osd_perf_query", 2024-06-04T05:23:49.905 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.906 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.906 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.906 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.906 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.906 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.906 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.906 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.906 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.906 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.906 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.906 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.906 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.906 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.907 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.907 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.907 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.907 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.907 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.907 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.907 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.907 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.907 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.907 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.907 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.907 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.907 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.908 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.908 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.908 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.908 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.908 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.908 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.908 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.908 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.908 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.908 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.908 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.908 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.908 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.908 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.909 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.909 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.909 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.909 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.909 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.909 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.909 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.909 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.909 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.909 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.909 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.909 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.909 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.910 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.910 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.910 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.910 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.910 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.910 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.910 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.910 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.910 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.910 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.910 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.910 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.910 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.911 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.911 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.911 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.911 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.911 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.911 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.911 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.911 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.911 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.911 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.911 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.911 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.911 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.912 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.912 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.912 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.912 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.912 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.912 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.912 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "osd_support", 2024-06-04T05:23:49.912 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.912 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.912 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.912 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.912 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.913 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.913 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.913 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.913 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.913 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.913 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.913 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.913 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.913 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.913 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.913 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.913 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.913 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.913 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.914 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.914 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.914 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.914 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.914 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.914 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.914 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.914 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.914 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.914 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.914 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.914 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.914 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.915 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.915 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.915 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.915 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.915 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.915 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.915 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.915 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.915 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.915 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.915 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.915 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.915 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.916 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.916 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.916 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.916 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.916 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.916 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.916 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.916 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.916 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.916 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.916 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.916 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.916 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.917 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.917 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.917 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.917 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.917 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.917 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.917 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.917 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.917 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.917 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.917 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.917 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.917 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.918 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.918 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.918 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.918 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.918 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.918 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.918 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.918 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.918 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.918 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.918 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.918 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.918 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.919 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.919 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pg_autoscaler", 2024-06-04T05:23:49.919 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.919 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.919 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.919 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.919 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.919 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.919 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.919 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.919 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.919 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.919 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.920 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.920 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.920 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.920 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.920 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.920 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.920 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.920 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.920 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.920 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.920 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.920 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.921 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.921 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.921 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.921 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.921 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.921 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.921 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.921 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.921 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.921 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.921 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.921 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.921 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.922 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.922 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.922 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.922 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.922 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.922 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.922 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.922 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.922 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.922 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.922 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.922 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.922 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.923 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.923 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.923 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.923 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.923 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.923 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.923 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.923 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.923 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.923 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.923 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.923 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.924 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.924 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.924 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.924 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.924 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.924 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.924 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.924 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.924 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-06-04T05:23:49.924 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-06-04T05:23:49.924 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.924 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.924 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.925 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-06-04T05:23:49.925 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.925 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.925 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.925 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.925 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.925 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.925 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.925 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.925 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.925 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.925 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.925 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.925 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.925 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.926 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.926 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.926 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.926 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.926 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.926 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.926 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.926 INFO:teuthology.orchestra.run.smithi070.stdout: "threshold": { 2024-06-04T05:23:49.926 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "threshold", 2024-06-04T05:23:49.926 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:49.926 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.926 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.927 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3.0", 2024-06-04T05:23:49.927 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1.0", 2024-06-04T05:23:49.927 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.927 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.927 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "scaling threshold", 2024-06-04T05:23:49.927 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-06-04T05:23:49.927 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.927 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.927 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.927 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.927 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "progress", 2024-06-04T05:23:49.927 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.928 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.928 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.928 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_pg_recovery_event": { 2024-06-04T05:23:49.928 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_pg_recovery_event", 2024-06-04T05:23:49.928 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.928 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.928 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.928 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.928 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.928 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.928 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.928 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow the module to show pg recovery progress", 2024-06-04T05:23:49.928 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.928 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.929 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.929 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.929 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": { 2024-06-04T05:23:49.929 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enabled", 2024-06-04T05:23:49.929 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.929 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.929 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.929 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.929 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.929 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.929 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.929 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.929 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.930 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.930 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.930 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.930 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.930 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.930 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.930 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.930 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.930 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.930 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.930 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.930 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.930 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.931 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.931 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.931 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.931 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.931 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.931 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.931 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.931 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.931 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.931 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.931 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.931 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.932 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.932 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.932 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.932 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.932 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.932 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.932 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.932 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.932 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.932 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.932 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.932 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.933 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.933 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.933 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.933 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.933 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.933 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.933 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.933 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.933 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.933 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.933 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.933 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.933 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.933 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.933 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.934 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.934 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.934 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.934 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.934 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.934 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.934 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.934 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.934 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.934 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.934 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.934 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.934 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.935 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.935 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.935 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.935 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.935 INFO:teuthology.orchestra.run.smithi070.stdout: "max_completed_events": { 2024-06-04T05:23:49.935 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_completed_events", 2024-06-04T05:23:49.935 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.935 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.935 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.935 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "50", 2024-06-04T05:23:49.935 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.935 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.935 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.935 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "number of past completed events to remember", 2024-06-04T05:23:49.937 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.937 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.937 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.938 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-06-04T05:23:49.938 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-06-04T05:23:49.938 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:49.938 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.938 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.938 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-06-04T05:23:49.938 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.938 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.938 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.938 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long the module is going to sleep", 2024-06-04T05:23:49.938 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.938 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.938 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.938 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.939 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.939 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.939 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.939 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.939 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.939 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.939 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.939 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.939 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.939 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.939 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.939 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.939 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.940 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.940 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.940 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.940 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prometheus", 2024-06-04T05:23:49.940 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.940 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.940 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.940 INFO:teuthology.orchestra.run.smithi070.stdout: "cache": { 2024-06-04T05:23:49.940 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cache", 2024-06-04T05:23:49.940 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.940 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.940 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.940 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.941 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.941 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.941 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.941 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.941 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.941 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.941 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.941 INFO:teuthology.orchestra.run.smithi070.stdout: "exclude_perf_counters": { 2024-06-04T05:23:49.941 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "exclude_perf_counters", 2024-06-04T05:23:49.941 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.941 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.941 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.941 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.942 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.942 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.942 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.942 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-06-04T05:23:49.942 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-06-04T05:23:49.942 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.942 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.942 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.942 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.942 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.942 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.942 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.943 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.943 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.943 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.943 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.943 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.943 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.943 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.943 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.943 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.943 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.943 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.943 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.943 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.943 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.943 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.944 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.944 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.944 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.944 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.944 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.944 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.944 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.944 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.944 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.944 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.944 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.945 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.945 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.945 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.945 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.945 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.945 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.945 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.945 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.945 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.945 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.945 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.945 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.945 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.945 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.946 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.946 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.946 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.946 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.946 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.946 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.946 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.946 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.946 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.946 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.946 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.946 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.946 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.946 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.946 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.947 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.947 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.947 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.947 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.947 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.947 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.947 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.947 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.947 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.947 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_stats_pools": { 2024-06-04T05:23:49.947 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_stats_pools", 2024-06-04T05:23:49.947 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.947 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.947 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.947 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.948 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.948 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.948 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.948 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.948 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.948 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.948 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.948 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_stats_pools_refresh_interval": { 2024-06-04T05:23:49.948 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-06-04T05:23:49.948 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.948 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.948 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.948 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "300", 2024-06-04T05:23:49.949 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.949 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.949 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.949 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.949 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.949 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.949 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.949 INFO:teuthology.orchestra.run.smithi070.stdout: "scrape_interval": { 2024-06-04T05:23:49.949 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "scrape_interval", 2024-06-04T05:23:49.949 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:49.949 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.949 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.949 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15.0", 2024-06-04T05:23:49.949 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.950 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.950 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.950 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.950 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.950 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.950 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.950 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.950 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-06-04T05:23:49.950 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-06-04T05:23:49.950 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.950 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.950 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.950 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "::", 2024-06-04T05:23:49.950 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.951 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.951 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.951 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-06-04T05:23:49.951 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.951 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.951 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.951 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.951 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-06-04T05:23:49.951 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-06-04T05:23:49.951 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.951 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.951 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.951 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "9283", 2024-06-04T05:23:49.951 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.951 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.952 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.952 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-06-04T05:23:49.952 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.952 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.952 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.952 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.952 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.952 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.952 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.952 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.952 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.952 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.952 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.952 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.953 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.953 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.953 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.953 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.953 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.953 INFO:teuthology.orchestra.run.smithi070.stdout: "stale_cache_strategy": { 2024-06-04T05:23:49.953 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "stale_cache_strategy", 2024-06-04T05:23:49.953 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.953 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.953 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.953 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "log", 2024-06-04T05:23:49.953 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.953 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.954 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.954 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.954 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.954 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.954 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.954 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.954 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_behaviour": { 2024-06-04T05:23:49.954 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_behaviour", 2024-06-04T05:23:49.954 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.954 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.954 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.954 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "default", 2024-06-04T05:23:49.954 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.955 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.955 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.955 INFO:teuthology.orchestra.run.smithi070.stdout: "default", 2024-06-04T05:23:49.955 INFO:teuthology.orchestra.run.smithi070.stdout: "error" 2024-06-04T05:23:49.955 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.955 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.955 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.955 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.955 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.955 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.955 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_error_status_code": { 2024-06-04T05:23:49.955 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_error_status_code", 2024-06-04T05:23:49.955 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.955 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.955 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.956 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "500", 2024-06-04T05:23:49.956 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "400", 2024-06-04T05:23:49.956 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "599", 2024-06-04T05:23:49.956 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.956 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.956 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.956 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.956 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.956 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.956 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.956 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.956 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.956 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_support", 2024-06-04T05:23:49.956 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.957 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.957 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.957 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.957 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.957 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.957 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.957 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.957 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.957 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.957 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.957 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.957 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.957 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.957 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.957 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.958 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.958 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.958 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.958 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.958 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.958 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.958 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.958 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.958 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.958 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.958 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.958 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.959 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.959 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.959 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.959 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.959 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.959 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.959 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.959 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.959 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.959 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.959 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.959 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.959 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.959 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.960 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.960 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.960 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.960 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.960 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.960 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.960 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.960 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.960 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.960 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.960 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.960 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.960 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.960 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.960 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.961 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.961 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.961 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.961 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.961 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.961 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.961 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.961 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.961 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.961 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.961 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.961 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.961 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.961 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.962 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.962 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.962 INFO:teuthology.orchestra.run.smithi070.stdout: "max_concurrent_snap_create": { 2024-06-04T05:23:49.962 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_concurrent_snap_create", 2024-06-04T05:23:49.962 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.962 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.962 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.962 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-06-04T05:23:49.962 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.962 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.962 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.962 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.962 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.962 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.963 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.963 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.963 INFO:teuthology.orchestra.run.smithi070.stdout: "mirror_snapshot_schedule": { 2024-06-04T05:23:49.963 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mirror_snapshot_schedule", 2024-06-04T05:23:49.963 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.963 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.963 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.963 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.963 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.963 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.963 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.963 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.963 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.963 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.964 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.964 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.964 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.964 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.964 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.964 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.964 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.964 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.964 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.964 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.964 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.964 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.964 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.964 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.965 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.965 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.965 INFO:teuthology.orchestra.run.smithi070.stdout: "trash_purge_schedule": { 2024-06-04T05:23:49.965 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "trash_purge_schedule", 2024-06-04T05:23:49.965 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.965 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.965 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.965 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.965 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.965 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.965 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.965 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.965 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.965 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.965 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.966 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.966 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.966 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.966 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.966 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "restful", 2024-06-04T05:23:49.966 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.966 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.966 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.966 INFO:teuthology.orchestra.run.smithi070.stdout: "enable_auth": { 2024-06-04T05:23:49.966 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enable_auth", 2024-06-04T05:23:49.966 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.966 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.966 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.967 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:49.967 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.967 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.967 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.967 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.967 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.967 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.967 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.967 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.967 INFO:teuthology.orchestra.run.smithi070.stdout: "key_file": { 2024-06-04T05:23:49.967 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "key_file", 2024-06-04T05:23:49.967 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.967 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.967 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.967 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.968 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.968 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.968 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.968 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.968 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.968 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.968 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.968 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.968 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.968 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.968 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.968 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.968 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.968 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.968 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.969 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.969 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.969 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.969 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.969 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.969 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.969 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.969 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.969 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.969 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.969 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.969 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.969 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.969 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.970 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.970 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.970 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.970 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.970 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.970 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.970 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.970 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.970 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.970 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.970 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.970 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.970 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.970 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.971 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.971 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.971 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.971 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.971 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.971 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.971 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.971 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.971 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.971 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.971 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.971 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.971 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.971 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.972 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.972 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.972 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.972 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.972 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.972 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.972 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.972 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.972 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.972 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.972 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.972 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.972 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.973 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.973 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.973 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.973 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.973 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.973 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.973 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.973 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-06-04T05:23:49.973 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-06-04T05:23:49.973 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.973 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.973 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.973 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.974 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.974 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.974 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.974 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.974 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.974 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.974 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.974 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.974 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-06-04T05:23:49.974 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-06-04T05:23:49.974 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.974 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.974 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.974 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.975 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.975 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.975 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.975 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.975 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.975 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.975 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.975 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.975 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.975 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.975 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.975 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.975 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.976 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.976 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.976 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.976 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.976 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.976 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.976 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.976 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.976 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.976 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.976 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.976 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.976 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rgw", 2024-06-04T05:23:49.976 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.976 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.977 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.977 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.977 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.977 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.977 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.977 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.977 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.977 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.977 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.977 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.977 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.977 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.977 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.977 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.978 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.978 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.978 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.978 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.978 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.978 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.978 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.978 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.978 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.978 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.978 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.978 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.978 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.979 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.979 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.979 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.979 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.979 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.979 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.979 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.979 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.979 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.979 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.979 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.979 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.979 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.979 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.980 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.980 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.980 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.980 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.980 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.980 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.980 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.980 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.980 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.980 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.980 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.980 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.980 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.980 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.981 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.981 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.981 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.981 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.981 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.981 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.981 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.981 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.981 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.981 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.981 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.981 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.981 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.981 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.982 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.982 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.982 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.982 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.982 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.982 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.982 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.982 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.982 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.982 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.982 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.982 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.982 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.982 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.983 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.983 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.983 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.983 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.983 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.983 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rook", 2024-06-04T05:23:49.983 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.983 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.983 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.983 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.983 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.983 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.983 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.983 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.984 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.984 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.984 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.984 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.984 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.984 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.984 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.984 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.984 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.984 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.984 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.984 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.984 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.984 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.985 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.985 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.985 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.985 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.985 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.985 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.985 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.985 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.985 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.985 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.985 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.985 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.985 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.985 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.986 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.986 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.986 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.986 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.986 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.986 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.986 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.986 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.986 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.986 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.986 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.986 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.986 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:49.989 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.989 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:49.989 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.989 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.989 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.989 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.989 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.989 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.989 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.989 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.989 INFO:teuthology.orchestra.run.smithi070.stdout: "storage_class": { 2024-06-04T05:23:49.989 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "storage_class", 2024-06-04T05:23:49.989 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.989 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.990 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.990 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "local", 2024-06-04T05:23:49.990 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.990 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.990 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.990 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-06-04T05:23:49.990 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.990 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.990 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.990 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.990 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:49.990 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.990 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:49.990 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "selftest", 2024-06-04T05:23:49.991 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:49.991 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:49.991 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:49.991 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:49.991 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:49.991 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.991 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.991 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.991 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.991 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.991 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.991 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.991 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.991 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.992 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.992 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.992 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.992 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.992 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.992 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.992 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.992 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.992 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.992 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.992 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:49.992 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:49.992 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.992 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.993 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.993 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.993 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.993 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.993 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.993 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.993 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.993 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.993 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.993 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.993 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:49.993 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:49.993 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.994 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.994 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.994 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:49.994 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.994 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.994 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:49.994 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:49.994 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:49.994 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:49.994 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:49.994 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:49.994 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:49.994 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:49.994 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.995 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.995 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.995 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.995 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.995 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:49.995 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:49.995 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:49.995 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.995 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:49.995 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:49.995 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.995 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.995 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.995 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.996 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.996 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.996 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.996 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.996 INFO:teuthology.orchestra.run.smithi070.stdout: "roption1": { 2024-06-04T05:23:49.996 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "roption1", 2024-06-04T05:23:49.996 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.996 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.996 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.996 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.996 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.996 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.996 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.996 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.997 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.997 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.997 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.997 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.997 INFO:teuthology.orchestra.run.smithi070.stdout: "roption2": { 2024-06-04T05:23:49.997 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "roption2", 2024-06-04T05:23:49.997 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.997 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.997 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.997 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "xyz", 2024-06-04T05:23:49.997 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.997 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.997 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.997 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.997 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.998 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.998 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.998 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.998 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption1": { 2024-06-04T05:23:49.998 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption1", 2024-06-04T05:23:49.998 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:49.998 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.998 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.998 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.998 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.998 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.998 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.998 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.998 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:49.999 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:49.999 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:49.999 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:49.999 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption2": { 2024-06-04T05:23:49.999 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption2", 2024-06-04T05:23:49.999 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:49.999 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:49.999 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:49.999 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:49.999 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:49.999 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:49.999 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:49.999 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:49.999 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.000 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.000 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.000 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.000 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption3": { 2024-06-04T05:23:50.000 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption3", 2024-06-04T05:23:50.000 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-06-04T05:23:50.000 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.000 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.000 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.000 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.000 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.000 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.000 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.000 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.001 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.001 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.001 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.001 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption4": { 2024-06-04T05:23:50.001 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption4", 2024-06-04T05:23:50.001 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.001 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.001 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.001 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.001 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.001 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.001 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.001 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.001 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.002 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.002 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.002 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.002 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption5": { 2024-06-04T05:23:50.002 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption5", 2024-06-04T05:23:50.002 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.002 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.002 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.002 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.002 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.002 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.002 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.002 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.002 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.003 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.003 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.003 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.003 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption6": { 2024-06-04T05:23:50.003 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption6", 2024-06-04T05:23:50.003 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.003 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.003 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.003 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:50.003 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.003 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.003 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.003 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.003 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.004 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.004 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.004 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.004 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption7": { 2024-06-04T05:23:50.004 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption7", 2024-06-04T05:23:50.004 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:50.004 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.004 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.004 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.004 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-06-04T05:23:50.004 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "42", 2024-06-04T05:23:50.004 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.004 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.004 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.005 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.005 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.005 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.005 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:50.005 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:50.005 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:50.005 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:50.005 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.005 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:50.005 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.005 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.005 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.005 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.005 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.006 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.006 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.006 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.006 INFO:teuthology.orchestra.run.smithi070.stdout: "testkey": { 2024-06-04T05:23:50.006 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testkey", 2024-06-04T05:23:50.006 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.006 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.006 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.006 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.006 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.006 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.006 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.006 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.006 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.007 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.007 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.007 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.007 INFO:teuthology.orchestra.run.smithi070.stdout: "testlkey": { 2024-06-04T05:23:50.007 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testlkey", 2024-06-04T05:23:50.007 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.007 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.007 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.007 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.007 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.007 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.007 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.007 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.007 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.007 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.008 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.008 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.008 INFO:teuthology.orchestra.run.smithi070.stdout: "testnewline": { 2024-06-04T05:23:50.008 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testnewline", 2024-06-04T05:23:50.008 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.008 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.008 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.008 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.008 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.008 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.008 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.008 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.008 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.008 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.009 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.009 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.009 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.009 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.009 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.009 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smb", 2024-06-04T05:23:50.009 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:50.009 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:50.009 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:50.009 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:50.009 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:50.009 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.009 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.009 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.010 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.010 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.010 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.010 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.010 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.010 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.010 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.010 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.010 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.010 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.010 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.010 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.010 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.010 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.011 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.011 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.011 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:50.011 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:50.011 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.011 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.011 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.011 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.011 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.011 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.011 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.011 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.011 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.011 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.012 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.012 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.012 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:50.012 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:50.012 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.012 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.012 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.012 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:50.012 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.012 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.012 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.012 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.012 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.012 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.012 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.013 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.013 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.013 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.013 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.013 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.013 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.013 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.013 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.013 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:50.013 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:50.013 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.013 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.013 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.013 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.014 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.014 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.014 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.014 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.014 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.014 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.014 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.014 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.014 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:50.014 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:50.014 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:50.014 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:50.014 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.014 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:50.015 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.015 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.015 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.015 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.015 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.015 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.015 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.015 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.015 INFO:teuthology.orchestra.run.smithi070.stdout: "update_orchestration": { 2024-06-04T05:23:50.015 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "update_orchestration", 2024-06-04T05:23:50.015 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.015 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.015 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.015 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:50.016 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.016 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.016 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.016 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-06-04T05:23:50.016 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.016 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.016 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.016 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.016 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.016 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.016 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.016 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snap_schedule", 2024-06-04T05:23:50.016 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:50.016 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:50.017 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:50.017 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_m_granularity": { 2024-06-04T05:23:50.017 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_m_granularity", 2024-06-04T05:23:50.017 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.017 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.017 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.017 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.017 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.017 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.017 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.017 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow minute scheduled snapshots", 2024-06-04T05:23:50.017 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.017 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.017 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.018 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.018 INFO:teuthology.orchestra.run.smithi070.stdout: "dump_on_update": { 2024-06-04T05:23:50.018 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "dump_on_update", 2024-06-04T05:23:50.018 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.018 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.018 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.018 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.018 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.018 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.018 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.018 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "dump database to debug log on update", 2024-06-04T05:23:50.018 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.018 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.018 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.019 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.019 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:50.019 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:50.019 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.019 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.019 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.019 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.019 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.019 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.019 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.019 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.019 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.019 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.019 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.020 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.020 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.020 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.020 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.020 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.020 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.020 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.020 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.020 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:50.020 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:50.020 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.020 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.020 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.020 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.021 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.021 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.021 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.021 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.021 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.021 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.021 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.021 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.021 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:50.021 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:50.021 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.021 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.021 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.021 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:50.021 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.022 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.022 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.022 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.022 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.022 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.022 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.022 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.022 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.022 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.022 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.022 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.022 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.022 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.022 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.023 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:50.023 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:50.023 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.023 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.023 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.023 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.023 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.023 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.023 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.023 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.023 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.023 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.023 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.023 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.024 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:50.024 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:50.024 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:50.024 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:50.024 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.024 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:50.024 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.024 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.024 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.024 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.024 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.024 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.024 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.024 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.024 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.025 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.025 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.025 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "stats", 2024-06-04T05:23:50.025 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:50.025 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:50.025 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:50.025 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:50.025 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:50.025 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.025 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.025 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.025 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.025 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.025 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.026 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.026 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.026 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.026 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.026 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.026 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.026 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.026 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.026 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.026 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.026 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.026 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.026 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.027 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:50.027 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:50.027 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.027 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.027 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.027 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.027 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.027 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.027 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.027 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.027 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.027 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.027 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.027 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.028 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:50.028 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:50.028 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.028 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.028 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.028 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:50.028 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.028 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.028 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.028 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.028 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.028 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.028 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.028 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.029 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.029 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.029 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.029 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.029 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.029 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.029 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.029 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:50.029 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:50.029 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.029 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.029 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.029 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.029 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.030 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.030 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.030 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.030 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.030 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.030 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.030 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.030 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:50.030 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:50.030 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:50.030 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:50.030 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.030 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:50.031 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.031 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.031 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.031 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.031 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.031 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.031 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.031 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.031 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.031 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.031 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.031 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "status", 2024-06-04T05:23:50.031 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:50.032 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:50.032 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:50.032 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:50.032 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:50.032 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.032 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.032 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.032 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.032 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.032 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.032 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.032 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.032 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.032 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.033 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.033 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.033 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.033 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.033 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.033 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.033 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.033 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.033 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.033 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:50.033 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:50.033 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.034 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.034 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.034 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.034 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.034 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.034 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.034 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.034 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.034 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.034 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.034 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.034 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:50.034 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:50.035 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.035 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.035 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.035 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:50.035 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.035 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.035 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.035 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.035 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.035 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.035 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.035 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.035 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.035 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.035 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.036 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.036 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.036 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.036 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.036 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:50.036 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:50.036 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.036 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.036 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.036 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.036 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.036 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.036 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.036 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.037 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.037 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.037 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.037 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.037 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:50.037 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:50.037 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:50.037 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:50.037 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.037 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:50.037 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.037 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.037 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.037 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.038 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.038 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.038 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.038 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.038 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.038 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.038 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.038 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "telegraf", 2024-06-04T05:23:50.038 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:50.038 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:50.038 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:50.038 INFO:teuthology.orchestra.run.smithi070.stdout: "address": { 2024-06-04T05:23:50.038 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "address", 2024-06-04T05:23:50.039 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.039 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.039 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.039 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-06-04T05:23:50.039 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.039 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.039 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.039 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.039 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.039 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.039 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.039 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.039 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-06-04T05:23:50.039 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-06-04T05:23:50.040 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:50.040 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.040 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.040 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15", 2024-06-04T05:23:50.040 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.040 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.040 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.040 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.040 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.040 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.040 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.040 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.040 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:50.040 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:50.041 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.041 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.041 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.041 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.041 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.041 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.041 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.041 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.041 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.041 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.041 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.041 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.041 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.041 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.042 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.042 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.042 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.042 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.042 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.042 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:50.042 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:50.042 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.042 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.042 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.042 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.042 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.042 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.042 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.043 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.043 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.043 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.043 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.043 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.043 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:50.043 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:50.043 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.043 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.043 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.043 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:50.043 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.043 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.043 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.044 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.044 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.044 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.044 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.044 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.044 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.044 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.044 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.044 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.044 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.044 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.044 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.044 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:50.044 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:50.045 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.045 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.045 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.045 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.045 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.045 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.045 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.045 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.045 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.045 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.045 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.045 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.045 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:50.046 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:50.046 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:50.046 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:50.046 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.046 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:50.046 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.046 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.046 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.046 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.046 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.046 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.046 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.046 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.047 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.047 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.047 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.047 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "telemetry", 2024-06-04T05:23:50.047 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:50.047 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:50.047 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:50.047 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_basic": { 2024-06-04T05:23:50.047 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_basic", 2024-06-04T05:23:50.047 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.047 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.047 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.047 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:50.047 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.048 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.048 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.048 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share basic cluster information (size, version)", 2024-06-04T05:23:50.048 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.048 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.048 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.048 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.048 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_crash": { 2024-06-04T05:23:50.048 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_crash", 2024-06-04T05:23:50.048 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.048 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.048 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.048 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:50.048 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.049 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.049 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.049 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-06-04T05:23:50.049 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.049 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.049 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.049 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.049 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_device": { 2024-06-04T05:23:50.049 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_device", 2024-06-04T05:23:50.049 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.049 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.049 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.049 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:50.049 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.050 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.050 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.050 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-06-04T05:23:50.050 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.050 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.050 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.050 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.050 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_ident": { 2024-06-04T05:23:50.050 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_ident", 2024-06-04T05:23:50.050 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.050 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.050 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.050 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.051 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.051 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.051 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.051 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-06-04T05:23:50.051 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.051 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.051 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.051 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.051 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_perf": { 2024-06-04T05:23:50.051 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_perf", 2024-06-04T05:23:50.051 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.051 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.051 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.051 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.052 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.052 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.052 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.052 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share various performance metrics of a cluster", 2024-06-04T05:23:50.052 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.052 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.052 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.052 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.052 INFO:teuthology.orchestra.run.smithi070.stdout: "contact": { 2024-06-04T05:23:50.052 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "contact", 2024-06-04T05:23:50.052 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.052 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.052 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.052 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.053 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.053 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.053 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.053 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.053 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.053 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.053 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.053 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.053 INFO:teuthology.orchestra.run.smithi070.stdout: "description": { 2024-06-04T05:23:50.053 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "description", 2024-06-04T05:23:50.053 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.053 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.053 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.053 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.053 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.054 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.054 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.054 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.054 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.054 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.054 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.054 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.054 INFO:teuthology.orchestra.run.smithi070.stdout: "device_url": { 2024-06-04T05:23:50.054 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_url", 2024-06-04T05:23:50.054 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.054 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.054 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.054 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-06-04T05:23:50.055 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.055 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.055 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.055 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.055 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.055 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.055 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.055 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.055 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": { 2024-06-04T05:23:50.055 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enabled", 2024-06-04T05:23:50.055 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.055 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.055 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.055 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.055 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.056 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.056 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.056 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.056 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.056 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.056 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.056 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.056 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-06-04T05:23:50.056 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-06-04T05:23:50.056 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:50.056 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.056 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.056 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "24", 2024-06-04T05:23:50.057 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "8", 2024-06-04T05:23:50.057 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.057 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.057 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.057 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.057 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.057 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.057 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.057 INFO:teuthology.orchestra.run.smithi070.stdout: "last_opt_revision": { 2024-06-04T05:23:50.057 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "last_opt_revision", 2024-06-04T05:23:50.057 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:50.057 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.057 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.058 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1", 2024-06-04T05:23:50.058 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.058 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.058 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.058 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.058 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.058 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.058 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.058 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.058 INFO:teuthology.orchestra.run.smithi070.stdout: "leaderboard": { 2024-06-04T05:23:50.058 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "leaderboard", 2024-06-04T05:23:50.058 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.058 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.058 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.059 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.059 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.059 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.059 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.059 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.059 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.059 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.059 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.059 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.059 INFO:teuthology.orchestra.run.smithi070.stdout: "leaderboard_description": { 2024-06-04T05:23:50.059 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "leaderboard_description", 2024-06-04T05:23:50.059 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.059 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.059 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.060 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.060 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.060 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.060 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.060 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.060 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.060 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.060 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.060 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.060 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:50.060 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:50.060 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.060 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.060 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.061 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.061 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.061 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.061 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.061 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.061 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.061 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.061 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.061 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.061 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.061 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.061 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.061 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.061 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.062 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.062 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.062 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:50.062 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:50.062 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.062 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.062 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.062 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.062 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.062 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.062 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.062 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.062 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.062 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.063 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.063 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.063 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:50.063 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:50.063 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.063 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.063 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.063 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:50.063 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.063 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.063 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.063 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.063 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.063 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.064 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.064 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.064 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.064 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.064 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.064 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.064 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.064 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.064 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.064 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:50.064 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:50.064 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.064 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.065 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.065 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.065 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.065 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.065 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.065 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.065 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.065 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.065 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.065 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.065 INFO:teuthology.orchestra.run.smithi070.stdout: "organization": { 2024-06-04T05:23:50.065 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "organization", 2024-06-04T05:23:50.065 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.066 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.066 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.066 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.066 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.066 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.066 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.066 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.066 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.066 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.066 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.066 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.066 INFO:teuthology.orchestra.run.smithi070.stdout: "proxy": { 2024-06-04T05:23:50.066 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "proxy", 2024-06-04T05:23:50.067 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.067 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.067 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.067 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.067 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.067 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.067 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.067 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.067 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.067 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.067 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.067 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.067 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:50.068 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:50.068 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:50.068 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:50.068 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.068 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:50.068 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.068 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.068 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.068 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.068 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.068 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.068 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.068 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.068 INFO:teuthology.orchestra.run.smithi070.stdout: "url": { 2024-06-04T05:23:50.069 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "url", 2024-06-04T05:23:50.069 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.069 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.069 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.069 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-06-04T05:23:50.069 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.069 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.069 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.069 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.069 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.069 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.069 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.069 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.069 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.070 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.070 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.070 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "test_orchestrator", 2024-06-04T05:23:50.070 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:50.070 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:50.070 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:50.070 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:50.070 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:50.070 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.070 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.070 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.070 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.070 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.070 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.071 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.071 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.071 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.071 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.071 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.071 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.071 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.071 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.071 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.071 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.071 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.071 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.071 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.072 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:50.072 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:50.072 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.072 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.072 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.072 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.072 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.072 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.072 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.072 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.072 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.072 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.072 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.072 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.073 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:50.073 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:50.073 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.073 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.073 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.073 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:50.073 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.073 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.073 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.073 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.073 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.073 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.073 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.073 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.074 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.074 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.074 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.074 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.074 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.074 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.074 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.074 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:50.074 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:50.074 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.074 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.074 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.074 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.074 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.075 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.075 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.075 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.075 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.075 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.075 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.075 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.075 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:50.075 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:50.075 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:50.075 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:50.075 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.075 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:50.075 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.075 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.076 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.076 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.076 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.076 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.076 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.076 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.076 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.076 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.076 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.076 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "volumes", 2024-06-04T05:23:50.076 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:50.076 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:50.076 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:50.076 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:50.077 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:50.077 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.077 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.077 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.077 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.077 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.077 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.077 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.077 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.077 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.077 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.077 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.077 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.077 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.078 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.078 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.078 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.078 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.078 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.078 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.078 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:50.078 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:50.078 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.078 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.078 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.078 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.078 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.078 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.079 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.079 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.079 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.079 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.079 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.079 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.079 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:50.079 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:50.079 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.079 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.079 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.079 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:50.079 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.080 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.080 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.080 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.080 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.080 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.080 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.080 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.080 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.080 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.080 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.080 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.080 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.080 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.080 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.081 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:50.081 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:50.081 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.081 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.081 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.081 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.081 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.081 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.081 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.081 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.081 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.081 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.081 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.082 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.082 INFO:teuthology.orchestra.run.smithi070.stdout: "max_concurrent_clones": { 2024-06-04T05:23:50.082 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_concurrent_clones", 2024-06-04T05:23:50.082 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:50.082 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.082 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.082 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "4", 2024-06-04T05:23:50.082 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.082 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.082 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.082 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Number of asynchronous cloner threads", 2024-06-04T05:23:50.082 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.082 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.082 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.083 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.083 INFO:teuthology.orchestra.run.smithi070.stdout: "periodic_async_work": { 2024-06-04T05:23:50.083 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "periodic_async_work", 2024-06-04T05:23:50.083 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.083 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.083 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.083 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.083 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.083 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.083 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.083 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Periodically check for async work", 2024-06-04T05:23:50.083 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.083 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.083 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.084 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.084 INFO:teuthology.orchestra.run.smithi070.stdout: "snapshot_clone_delay": { 2024-06-04T05:23:50.084 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snapshot_clone_delay", 2024-06-04T05:23:50.084 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:50.084 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.084 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.084 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:50.084 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.084 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.084 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.084 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-06-04T05:23:50.084 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.084 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.084 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.084 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.085 INFO:teuthology.orchestra.run.smithi070.stdout: "snapshot_clone_no_wait": { 2024-06-04T05:23:50.085 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snapshot_clone_no_wait", 2024-06-04T05:23:50.085 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.085 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.085 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.085 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-06-04T05:23:50.085 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.085 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.085 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.085 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-06-04T05:23:50.085 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.085 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.085 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.085 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.086 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:50.086 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:50.086 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:50.086 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:50.086 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.086 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:50.086 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.086 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.086 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.086 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.086 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.086 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.086 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.086 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.086 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.087 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.087 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.087 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix", 2024-06-04T05:23:50.087 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-06-04T05:23:50.087 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-06-04T05:23:50.087 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-06-04T05:23:50.087 INFO:teuthology.orchestra.run.smithi070.stdout: "discovery_interval": { 2024-06-04T05:23:50.087 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "discovery_interval", 2024-06-04T05:23:50.087 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-06-04T05:23:50.087 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.087 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.087 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "100", 2024-06-04T05:23:50.087 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.087 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.088 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.088 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.088 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.088 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.088 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.088 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.088 INFO:teuthology.orchestra.run.smithi070.stdout: "identifier": { 2024-06-04T05:23:50.088 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "identifier", 2024-06-04T05:23:50.088 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.088 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.088 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.088 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.088 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.088 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.088 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.089 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.089 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.089 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.089 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.089 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.089 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-06-04T05:23:50.089 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-06-04T05:23:50.089 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-06-04T05:23:50.089 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.089 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.089 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-06-04T05:23:50.089 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.089 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.090 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.090 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.090 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.090 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.090 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.090 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.090 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-06-04T05:23:50.090 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-06-04T05:23:50.090 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.090 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.090 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.090 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.090 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.090 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.090 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.091 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.091 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.091 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.091 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.091 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.091 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.091 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.091 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.091 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.091 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.091 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.091 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.091 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-06-04T05:23:50.091 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-06-04T05:23:50.092 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.092 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.092 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.092 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.092 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.092 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.092 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.092 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.092 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.092 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.092 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.092 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.092 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-06-04T05:23:50.092 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-06-04T05:23:50.092 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.093 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.093 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.093 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-06-04T05:23:50.093 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.093 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.093 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-06-04T05:23:50.093 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-06-04T05:23:50.093 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-06-04T05:23:50.093 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-06-04T05:23:50.093 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-06-04T05:23:50.093 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-06-04T05:23:50.093 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-06-04T05:23:50.093 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.093 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.093 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.094 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.094 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.094 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.094 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-06-04T05:23:50.094 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-06-04T05:23:50.094 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-06-04T05:23:50.094 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.094 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.094 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-06-04T05:23:50.094 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.094 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.094 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.094 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.095 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.095 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.095 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.095 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.095 INFO:teuthology.orchestra.run.smithi070.stdout: "sqlite3_killpoint": { 2024-06-04T05:23:50.095 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sqlite3_killpoint", 2024-06-04T05:23:50.095 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:50.095 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "dev", 2024-06-04T05:23:50.095 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-06-04T05:23:50.095 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-06-04T05:23:50.095 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.095 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.095 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.095 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.095 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.096 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.096 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.096 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.096 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_host": { 2024-06-04T05:23:50.096 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_host", 2024-06-04T05:23:50.096 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.096 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.096 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.096 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-06-04T05:23:50.096 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.096 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.096 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.096 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.096 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.097 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.097 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.097 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.097 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_port": { 2024-06-04T05:23:50.097 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_port", 2024-06-04T05:23:50.097 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-06-04T05:23:50.097 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.097 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.097 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10051", 2024-06-04T05:23:50.097 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.097 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.097 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.097 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.097 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.098 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.098 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.098 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.098 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_sender": { 2024-06-04T05:23:50.098 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_sender", 2024-06-04T05:23:50.098 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-06-04T05:23:50.098 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-06-04T05:23:50.098 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-06-04T05:23:50.098 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-06-04T05:23:50.098 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-06-04T05:23:50.098 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-06-04T05:23:50.098 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-06-04T05:23:50.098 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-06-04T05:23:50.099 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-06-04T05:23:50.099 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-06-04T05:23:50.099 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-06-04T05:23:50.099 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.099 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.099 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.099 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.099 INFO:teuthology.orchestra.run.smithi070.stdout: "services": { 2024-06-04T05:23:50.099 INFO:teuthology.orchestra.run.smithi070.stdout: "dashboard": "https://172.21.15.70:8443/" 2024-06-04T05:23:50.099 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.099 INFO:teuthology.orchestra.run.smithi070.stdout: "always_on_modules": { 2024-06-04T05:23:50.099 INFO:teuthology.orchestra.run.smithi070.stdout: "octopus": [ 2024-06-04T05:23:50.099 INFO:teuthology.orchestra.run.smithi070.stdout: "balancer", 2024-06-04T05:23:50.099 INFO:teuthology.orchestra.run.smithi070.stdout: "crash", 2024-06-04T05:23:50.099 INFO:teuthology.orchestra.run.smithi070.stdout: "devicehealth", 2024-06-04T05:23:50.099 INFO:teuthology.orchestra.run.smithi070.stdout: "orchestrator", 2024-06-04T05:23:50.100 INFO:teuthology.orchestra.run.smithi070.stdout: "pg_autoscaler", 2024-06-04T05:23:50.100 INFO:teuthology.orchestra.run.smithi070.stdout: "progress", 2024-06-04T05:23:50.100 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_support", 2024-06-04T05:23:50.100 INFO:teuthology.orchestra.run.smithi070.stdout: "status", 2024-06-04T05:23:50.100 INFO:teuthology.orchestra.run.smithi070.stdout: "telemetry", 2024-06-04T05:23:50.100 INFO:teuthology.orchestra.run.smithi070.stdout: "volumes" 2024-06-04T05:23:50.100 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.100 INFO:teuthology.orchestra.run.smithi070.stdout: "pacific": [ 2024-06-04T05:23:50.100 INFO:teuthology.orchestra.run.smithi070.stdout: "balancer", 2024-06-04T05:23:50.100 INFO:teuthology.orchestra.run.smithi070.stdout: "crash", 2024-06-04T05:23:50.100 INFO:teuthology.orchestra.run.smithi070.stdout: "devicehealth", 2024-06-04T05:23:50.100 INFO:teuthology.orchestra.run.smithi070.stdout: "orchestrator", 2024-06-04T05:23:50.100 INFO:teuthology.orchestra.run.smithi070.stdout: "pg_autoscaler", 2024-06-04T05:23:50.100 INFO:teuthology.orchestra.run.smithi070.stdout: "progress", 2024-06-04T05:23:50.101 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_support", 2024-06-04T05:23:50.101 INFO:teuthology.orchestra.run.smithi070.stdout: "status", 2024-06-04T05:23:50.101 INFO:teuthology.orchestra.run.smithi070.stdout: "telemetry", 2024-06-04T05:23:50.101 INFO:teuthology.orchestra.run.smithi070.stdout: "volumes" 2024-06-04T05:23:50.101 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.101 INFO:teuthology.orchestra.run.smithi070.stdout: "quincy": [ 2024-06-04T05:23:50.101 INFO:teuthology.orchestra.run.smithi070.stdout: "balancer", 2024-06-04T05:23:50.101 INFO:teuthology.orchestra.run.smithi070.stdout: "crash", 2024-06-04T05:23:50.101 INFO:teuthology.orchestra.run.smithi070.stdout: "devicehealth", 2024-06-04T05:23:50.101 INFO:teuthology.orchestra.run.smithi070.stdout: "orchestrator", 2024-06-04T05:23:50.101 INFO:teuthology.orchestra.run.smithi070.stdout: "pg_autoscaler", 2024-06-04T05:23:50.101 INFO:teuthology.orchestra.run.smithi070.stdout: "progress", 2024-06-04T05:23:50.101 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_support", 2024-06-04T05:23:50.102 INFO:teuthology.orchestra.run.smithi070.stdout: "status", 2024-06-04T05:23:50.102 INFO:teuthology.orchestra.run.smithi070.stdout: "telemetry", 2024-06-04T05:23:50.102 INFO:teuthology.orchestra.run.smithi070.stdout: "volumes" 2024-06-04T05:23:50.102 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.102 INFO:teuthology.orchestra.run.smithi070.stdout: "reef": [ 2024-06-04T05:23:50.102 INFO:teuthology.orchestra.run.smithi070.stdout: "balancer", 2024-06-04T05:23:50.102 INFO:teuthology.orchestra.run.smithi070.stdout: "crash", 2024-06-04T05:23:50.102 INFO:teuthology.orchestra.run.smithi070.stdout: "devicehealth", 2024-06-04T05:23:50.102 INFO:teuthology.orchestra.run.smithi070.stdout: "orchestrator", 2024-06-04T05:23:50.102 INFO:teuthology.orchestra.run.smithi070.stdout: "pg_autoscaler", 2024-06-04T05:23:50.102 INFO:teuthology.orchestra.run.smithi070.stdout: "progress", 2024-06-04T05:23:50.102 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_support", 2024-06-04T05:23:50.102 INFO:teuthology.orchestra.run.smithi070.stdout: "status", 2024-06-04T05:23:50.102 INFO:teuthology.orchestra.run.smithi070.stdout: "telemetry", 2024-06-04T05:23:50.103 INFO:teuthology.orchestra.run.smithi070.stdout: "volumes" 2024-06-04T05:23:50.103 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-06-04T05:23:50.103 INFO:teuthology.orchestra.run.smithi070.stdout: "squid": [ 2024-06-04T05:23:50.103 INFO:teuthology.orchestra.run.smithi070.stdout: "balancer", 2024-06-04T05:23:50.103 INFO:teuthology.orchestra.run.smithi070.stdout: "crash", 2024-06-04T05:23:50.103 INFO:teuthology.orchestra.run.smithi070.stdout: "devicehealth", 2024-06-04T05:23:50.103 INFO:teuthology.orchestra.run.smithi070.stdout: "orchestrator", 2024-06-04T05:23:50.103 INFO:teuthology.orchestra.run.smithi070.stdout: "pg_autoscaler", 2024-06-04T05:23:50.103 INFO:teuthology.orchestra.run.smithi070.stdout: "progress", 2024-06-04T05:23:50.103 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_support", 2024-06-04T05:23:50.103 INFO:teuthology.orchestra.run.smithi070.stdout: "status", 2024-06-04T05:23:50.103 INFO:teuthology.orchestra.run.smithi070.stdout: "telemetry", 2024-06-04T05:23:50.103 INFO:teuthology.orchestra.run.smithi070.stdout: "volumes" 2024-06-04T05:23:50.103 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-06-04T05:23:50.103 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.104 INFO:teuthology.orchestra.run.smithi070.stdout: "last_failure_osd_epoch": 82, 2024-06-04T05:23:50.104 INFO:teuthology.orchestra.run.smithi070.stdout: "active_clients": [ 2024-06-04T05:23:50.104 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.104 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "devicehealth", 2024-06-04T05:23:50.104 INFO:teuthology.orchestra.run.smithi070.stdout: "addrvec": [ 2024-06-04T05:23:50.104 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.104 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v2", 2024-06-04T05:23:50.104 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "172.21.15.70:0", 2024-06-04T05:23:50.104 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3756934745 2024-06-04T05:23:50.104 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.104 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-06-04T05:23:50.104 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.104 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.105 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "libcephsqlite", 2024-06-04T05:23:50.105 INFO:teuthology.orchestra.run.smithi070.stdout: "addrvec": [ 2024-06-04T05:23:50.105 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.105 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v2", 2024-06-04T05:23:50.105 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "172.21.15.70:0", 2024-06-04T05:23:50.105 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 1061259567 2024-06-04T05:23:50.105 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.105 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-06-04T05:23:50.105 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.105 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.105 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_support", 2024-06-04T05:23:50.105 INFO:teuthology.orchestra.run.smithi070.stdout: "addrvec": [ 2024-06-04T05:23:50.105 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.105 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v2", 2024-06-04T05:23:50.105 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "172.21.15.70:0", 2024-06-04T05:23:50.106 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 2593715455 2024-06-04T05:23:50.106 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.106 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-06-04T05:23:50.106 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.106 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.106 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snap_schedule", 2024-06-04T05:23:50.106 INFO:teuthology.orchestra.run.smithi070.stdout: "addrvec": [ 2024-06-04T05:23:50.106 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.106 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v2", 2024-06-04T05:23:50.106 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "172.21.15.70:0", 2024-06-04T05:23:50.106 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3955061766 2024-06-04T05:23:50.106 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.106 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-06-04T05:23:50.107 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.107 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.107 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snap_schedule", 2024-06-04T05:23:50.107 INFO:teuthology.orchestra.run.smithi070.stdout: "addrvec": [ 2024-06-04T05:23:50.107 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.107 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v2", 2024-06-04T05:23:50.107 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "172.21.15.70:0", 2024-06-04T05:23:50.107 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3856147910 2024-06-04T05:23:50.107 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.107 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-06-04T05:23:50.107 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:50.107 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.107 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "volumes", 2024-06-04T05:23:50.107 INFO:teuthology.orchestra.run.smithi070.stdout: "addrvec": [ 2024-06-04T05:23:50.107 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-06-04T05:23:50.108 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v2", 2024-06-04T05:23:50.108 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "172.21.15.70:0", 2024-06-04T05:23:50.108 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 1411655011 2024-06-04T05:23:50.108 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.108 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-06-04T05:23:50.108 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:50.108 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-06-04T05:23:50.108 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:23:50.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:48 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3759815614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-04T05:23:50.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:48 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/4133601851' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-04T05:23:50.110 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:48 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/4007188811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-04T05:23:50.110 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:49 smithi098 ceph-mon[35569]: pgmap v840: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:23:50.110 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:49 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/4041866420' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-06-04T05:23:50.110 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:48 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3759815614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-04T05:23:50.110 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:48 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/4133601851' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-04T05:23:50.110 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:48 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/4007188811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-04T05:23:50.110 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:49 smithi112 ceph-mon[36453]: pgmap v840: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:23:50.110 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:49 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/4041866420' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-06-04T05:23:50.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.652+0000 7f346fc75640 1 -- 172.21.15.70:0/4041866420 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f34540b29e0 msgr2=0x7f34540b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:50.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.652+0000 7f346fc75640 1 --2- 172.21.15.70:0/4041866420 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f34540b29e0 0x7f34540b4ea0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f3464005fd0 tx=0x7f3464005c70 comp rx=0 tx=0).stop 2024-06-04T05:23:50.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.652+0000 7f346fc75640 1 -- 172.21.15.70:0/4041866420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3468101930 msgr2=0x7f3468199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:23:50.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.652+0000 7f346fc75640 1 --2- 172.21.15.70:0/4041866420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3468101930 0x7f3468199180 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f345802a940 tx=0x7f3458003c20 comp rx=0 tx=0).stop 2024-06-04T05:23:50.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.652+0000 7f346fc75640 1 -- 172.21.15.70:0/4041866420 shutdown_connections 2024-06-04T05:23:50.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.652+0000 7f346fc75640 1 --2- 172.21.15.70:0/4041866420 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f34540b29e0 0x7f34540b4ea0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:50.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.652+0000 7f346fc75640 1 --2- 172.21.15.70:0/4041866420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3468107940 0x7f346819da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:50.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.652+0000 7f346fc75640 1 --2- 172.21.15.70:0/4041866420 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34681022f0 0x7f34681996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:50.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.652+0000 7f346fc75640 1 --2- 172.21.15.70:0/4041866420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3468101930 0x7f3468199180 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:23:50.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.652+0000 7f346fc75640 1 -- 172.21.15.70:0/4041866420 >> 172.21.15.70:0/4041866420 conn(0x7f3468078a50 msgr2=0x7f34681043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:23:50.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.652+0000 7f346fc75640 1 -- 172.21.15.70:0/4041866420 shutdown_connections 2024-06-04T05:23:50.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:23:48.652+0000 7f346fc75640 1 -- 172.21.15.70:0/4041866420 wait complete. 2024-06-04T05:23:50.162 DEBUG:tasks.check_counter:Getting stats from a 2024-06-04T05:23:50.163 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump 2024-06-04T05:23:50.163 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:48 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3759815614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-04T05:23:50.163 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:48 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/4133601851' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-04T05:23:50.163 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:48 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/4007188811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-04T05:23:50.163 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:49 smithi070 ceph-mon[32631]: pgmap v840: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:23:50.164 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:49 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/4041866420' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-06-04T05:23:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:51.826 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:51 smithi070 ceph-mon[32631]: pgmap v841: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:23:51.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:51 smithi112 ceph-mon[36453]: pgmap v841: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:23:51.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:51 smithi098 ceph-mon[35569]: pgmap v841: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:23:52.102 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:23:52.103 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-0": { 2024-06-04T05:23:52.103 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 3389, 2024-06-04T05:23:52.103 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 3391, 2024-06-04T05:23:52.103 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 42703088, 2024-06-04T05:23:52.103 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 1077703, 2024-06-04T05:23:52.103 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 8, 2024-06-04T05:23:52.103 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 4, 2024-06-04T05:23:52.103 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.294439908, 2024-06-04T05:23:52.103 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.086866338, 2024-06-04T05:23:52.103 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.098815617, 2024-06-04T05:23:52.103 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.086683624, 2024-06-04T05:23:52.103 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:23:52.103 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 3389, 2024-06-04T05:23:52.104 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.092666054, 2024-06-04T05:23:52.104 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000027343 2024-06-04T05:23:52.104 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.104 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:23:52.104 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.104 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.104 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.104 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.104 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 0, 2024-06-04T05:23:52.104 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 0 2024-06-04T05:23:52.104 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.104 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-1": { 2024-06-04T05:23:52.104 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 1928, 2024-06-04T05:23:52.105 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 2295, 2024-06-04T05:23:52.105 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 84375259, 2024-06-04T05:23:52.105 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 1432912, 2024-06-04T05:23:52.105 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 10, 2024-06-04T05:23:52.105 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 7, 2024-06-04T05:23:52.105 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.231066460, 2024-06-04T05:23:52.105 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.057754922, 2024-06-04T05:23:52.105 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.104762591, 2024-06-04T05:23:52.105 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.053278922, 2024-06-04T05:23:52.105 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:23:52.105 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 2293, 2024-06-04T05:23:52.105 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.068293478, 2024-06-04T05:23:52.105 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000029783 2024-06-04T05:23:52.105 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.106 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:23:52.106 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.106 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.106 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.106 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.106 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-06-04T05:23:52.106 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 821648 2024-06-04T05:23:52.106 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.106 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-2": { 2024-06-04T05:23:52.106 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 1111, 2024-06-04T05:23:52.106 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 1056, 2024-06-04T05:23:52.106 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 60843144, 2024-06-04T05:23:52.106 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 475286, 2024-06-04T05:23:52.107 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 6, 2024-06-04T05:23:52.107 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 4, 2024-06-04T05:23:52.107 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.145631786, 2024-06-04T05:23:52.107 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.040753953, 2024-06-04T05:23:52.107 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.088974685, 2024-06-04T05:23:52.107 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.021572331, 2024-06-04T05:23:52.107 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:23:52.107 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 1055, 2024-06-04T05:23:52.107 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.035749344, 2024-06-04T05:23:52.107 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000033885 2024-06-04T05:23:52.107 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.107 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:23:52.107 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.107 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.108 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.108 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 1985952, 2024-06-04T05:23:52.108 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 308336 2024-06-04T05:23:52.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.108 INFO:teuthology.orchestra.run.smithi070.stdout: "cct": { 2024-06-04T05:23:52.108 INFO:teuthology.orchestra.run.smithi070.stdout: "total_workers": 1, 2024-06-04T05:23:52.108 INFO:teuthology.orchestra.run.smithi070.stdout: "unhealthy_workers": 0 2024-06-04T05:23:52.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.108 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-MDSRank": { 2024-06-04T05:23:52.108 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-06-04T05:23:52.108 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-06-04T05:23:52.108 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 9799, 2024-06-04T05:23:52.109 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.232154989, 2024-06-04T05:23:52.109 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000023691 2024-06-04T05:23:52.109 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:52.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.109 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-PurgeQueue": { 2024-06-04T05:23:52.109 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-06-04T05:23:52.109 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-06-04T05:23:52.109 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.109 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.109 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.109 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:52.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.109 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-06-04T05:23:52.109 INFO:teuthology.orchestra.run.smithi070.stdout: "request": 0, 2024-06-04T05:23:52.110 INFO:teuthology.orchestra.run.smithi070.stdout: "reply": 0, 2024-06-04T05:23:52.110 INFO:teuthology.orchestra.run.smithi070.stdout: "reply_latency": { 2024-06-04T05:23:52.110 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.110 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.110 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.110 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.110 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_reply": 0, 2024-06-04T05:23:52.110 INFO:teuthology.orchestra.run.smithi070.stdout: "forward": 0, 2024-06-04T05:23:52.110 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_complete": 0, 2024-06-04T05:23:52.110 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_keys": 0, 2024-06-04T05:23:52.110 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_commit": 0, 2024-06-04T05:23:52.110 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_split": 0, 2024-06-04T05:23:52.110 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_merge": 0, 2024-06-04T05:23:52.110 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes": 7445, 2024-06-04T05:23:52.111 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_top": 128, 2024-06-04T05:23:52.111 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_bottom": 7317, 2024-06-04T05:23:52.111 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pin_tail": 0, 2024-06-04T05:23:52.111 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pinned": 7264, 2024-06-04T05:23:52.111 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_expired": 0, 2024-06-04T05:23:52.111 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_with_caps": 0, 2024-06-04T05:23:52.111 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": 0, 2024-06-04T05:23:52.111 INFO:teuthology.orchestra.run.smithi070.stdout: "subtrees": 1, 2024-06-04T05:23:52.111 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse": 0, 2024-06-04T05:23:52.111 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_hit": 0, 2024-06-04T05:23:52.111 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_forward": 0, 2024-06-04T05:23:52.111 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_discover": 0, 2024-06-04T05:23:52.111 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_dir_fetch": 0, 2024-06-04T05:23:52.111 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_remote_ino": 0, 2024-06-04T05:23:52.112 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_lock": 0, 2024-06-04T05:23:52.112 INFO:teuthology.orchestra.run.smithi070.stdout: "load_cent": 0, 2024-06-04T05:23:52.112 INFO:teuthology.orchestra.run.smithi070.stdout: "q": 0, 2024-06-04T05:23:52.112 INFO:teuthology.orchestra.run.smithi070.stdout: "exported": 0, 2024-06-04T05:23:52.112 INFO:teuthology.orchestra.run.smithi070.stdout: "exported_inodes": 0, 2024-06-04T05:23:52.112 INFO:teuthology.orchestra.run.smithi070.stdout: "imported": 0, 2024-06-04T05:23:52.112 INFO:teuthology.orchestra.run.smithi070.stdout: "imported_inodes": 0, 2024-06-04T05:23:52.112 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_dir_fetch": 0, 2024-06-04T05:23:52.112 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_backtrace_fetch": 0, 2024-06-04T05:23:52.112 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_peer_discover": 0, 2024-06-04T05:23:52.112 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rfiles": 0, 2024-06-04T05:23:52.112 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rbytes": 0, 2024-06-04T05:23:52.112 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rsnaps": 0, 2024-06-04T05:23:52.112 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_fetch": 0, 2024-06-04T05:23:52.112 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_set_tag": 0, 2024-06-04T05:23:52.113 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_repaired": 0, 2024-06-04T05:23:52.113 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_inotable_repaired": 0, 2024-06-04T05:23:52.113 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_inodes": 0, 2024-06-04T05:23:52.113 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_base_inodes": 0, 2024-06-04T05:23:52.113 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dirfrag_rstats": 0, 2024-06-04T05:23:52.113 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_file_inodes": 0, 2024-06-04T05:23:52.113 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_inode_file_caps": 0, 2024-06-04T05:23:52.113 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_revoke": 0, 2024-06-04T05:23:52.113 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_grant": 0, 2024-06-04T05:23:52.113 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_trunc": 0, 2024-06-04T05:23:52.113 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-04T05:23:52.113 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-04T05:23:52.113 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps": 0, 2024-06-04T05:23:52.113 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps_dirty": 0, 2024-06-04T05:23:52.113 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_cap_release": 0, 2024-06-04T05:23:52.114 INFO:teuthology.orchestra.run.smithi070.stdout: "process_request_cap_release": 0 2024-06-04T05:23:52.114 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.114 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache": { 2024-06-04T05:23:52.114 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update": 0, 2024-06-04T05:23:52.114 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update_receipt": 0, 2024-06-04T05:23:52.114 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_try_discover": 0, 2024-06-04T05:23:52.114 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_send_discover": 0, 2024-06-04T05:23:52.114 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_handle_discover": 0, 2024-06-04T05:23:52.114 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays": 0, 2024-06-04T05:23:52.114 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_delayed": 0, 2024-06-04T05:23:52.114 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_enqueuing": 0, 2024-06-04T05:23:52.114 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_created": 0, 2024-06-04T05:23:52.114 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_enqueued": 0, 2024-06-04T05:23:52.114 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_reintegrated": 0, 2024-06-04T05:23:52.115 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_migrated": 0, 2024-06-04T05:23:52.115 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_processing": 0, 2024-06-04T05:23:52.115 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_enqueued": 0, 2024-06-04T05:23:52.115 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_prioritized": 0, 2024-06-04T05:23:52.115 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_started": 0, 2024-06-04T05:23:52.115 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_completed": 0, 2024-06-04T05:23:52.115 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_quiesce_path": 0, 2024-06-04T05:23:52.115 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_quiesce_inode": 0, 2024-06-04T05:23:52.115 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_enqueue_scrub": 0, 2024-06-04T05:23:52.115 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_exportdir": 0, 2024-06-04T05:23:52.115 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_flush": 0, 2024-06-04T05:23:52.115 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragmentdir": 0, 2024-06-04T05:23:52.116 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragstats": 0, 2024-06-04T05:23:52.116 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_inodestats": 0 2024-06-04T05:23:52.116 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.116 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log": { 2024-06-04T05:23:52.116 INFO:teuthology.orchestra.run.smithi070.stdout: "evlrg": 0, 2024-06-04T05:23:52.116 INFO:teuthology.orchestra.run.smithi070.stdout: "evadd": 0, 2024-06-04T05:23:52.116 INFO:teuthology.orchestra.run.smithi070.stdout: "evex": 0, 2024-06-04T05:23:52.116 INFO:teuthology.orchestra.run.smithi070.stdout: "evtrm": 0, 2024-06-04T05:23:52.116 INFO:teuthology.orchestra.run.smithi070.stdout: "ev": 36446, 2024-06-04T05:23:52.116 INFO:teuthology.orchestra.run.smithi070.stdout: "evexg": 0, 2024-06-04T05:23:52.116 INFO:teuthology.orchestra.run.smithi070.stdout: "evexd": 0, 2024-06-04T05:23:52.116 INFO:teuthology.orchestra.run.smithi070.stdout: "segadd": 0, 2024-06-04T05:23:52.116 INFO:teuthology.orchestra.run.smithi070.stdout: "segex": 0, 2024-06-04T05:23:52.117 INFO:teuthology.orchestra.run.smithi070.stdout: "segtrm": 0, 2024-06-04T05:23:52.117 INFO:teuthology.orchestra.run.smithi070.stdout: "seg": 56, 2024-06-04T05:23:52.117 INFO:teuthology.orchestra.run.smithi070.stdout: "segmjr": 4, 2024-06-04T05:23:52.117 INFO:teuthology.orchestra.run.smithi070.stdout: "segexg": 0, 2024-06-04T05:23:52.117 INFO:teuthology.orchestra.run.smithi070.stdout: "segexd": 0, 2024-06-04T05:23:52.117 INFO:teuthology.orchestra.run.smithi070.stdout: "expos": 4194304, 2024-06-04T05:23:52.117 INFO:teuthology.orchestra.run.smithi070.stdout: "wrpos": 188487597, 2024-06-04T05:23:52.117 INFO:teuthology.orchestra.run.smithi070.stdout: "rdpos": 188474113, 2024-06-04T05:23:52.117 INFO:teuthology.orchestra.run.smithi070.stdout: "jlat": { 2024-06-04T05:23:52.117 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.117 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.117 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.117 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.117 INFO:teuthology.orchestra.run.smithi070.stdout: "replayed": 36446 2024-06-04T05:23:52.117 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.117 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_mem": { 2024-06-04T05:23:52.118 INFO:teuthology.orchestra.run.smithi070.stdout: "ino": 1513, 2024-06-04T05:23:52.118 INFO:teuthology.orchestra.run.smithi070.stdout: "ino+": 7290, 2024-06-04T05:23:52.118 INFO:teuthology.orchestra.run.smithi070.stdout: "ino-": 5777, 2024-06-04T05:23:52.118 INFO:teuthology.orchestra.run.smithi070.stdout: "dir": 60, 2024-06-04T05:23:52.118 INFO:teuthology.orchestra.run.smithi070.stdout: "dir+": 87, 2024-06-04T05:23:52.118 INFO:teuthology.orchestra.run.smithi070.stdout: "dir-": 27, 2024-06-04T05:23:52.118 INFO:teuthology.orchestra.run.smithi070.stdout: "dn": 7445, 2024-06-04T05:23:52.118 INFO:teuthology.orchestra.run.smithi070.stdout: "dn+": 7700, 2024-06-04T05:23:52.118 INFO:teuthology.orchestra.run.smithi070.stdout: "dn-": 255, 2024-06-04T05:23:52.118 INFO:teuthology.orchestra.run.smithi070.stdout: "cap": 0, 2024-06-04T05:23:52.118 INFO:teuthology.orchestra.run.smithi070.stdout: "cap+": 0, 2024-06-04T05:23:52.118 INFO:teuthology.orchestra.run.smithi070.stdout: "cap-": 0, 2024-06-04T05:23:52.119 INFO:teuthology.orchestra.run.smithi070.stdout: "rss": 63912, 2024-06-04T05:23:52.119 INFO:teuthology.orchestra.run.smithi070.stdout: "heap": 215416 2024-06-04T05:23:52.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.119 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_server": { 2024-06-04T05:23:52.119 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_client_request": 0, 2024-06-04T05:23:52.119 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_server_request": 0, 2024-06-04T05:23:52.119 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_request": 0, 2024-06-04T05:23:52.119 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_session": 0, 2024-06-04T05:23:52.119 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_peer_request": 0, 2024-06-04T05:23:52.119 INFO:teuthology.orchestra.run.smithi070.stdout: "req_create_latency": { 2024-06-04T05:23:52.119 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.119 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.119 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.120 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getattr_latency": { 2024-06-04T05:23:52.120 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.120 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.120 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.120 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getfilelock_latency": { 2024-06-04T05:23:52.120 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.120 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.120 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.120 INFO:teuthology.orchestra.run.smithi070.stdout: "req_link_latency": { 2024-06-04T05:23:52.120 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.120 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.120 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.121 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.121 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookup_latency": { 2024-06-04T05:23:52.121 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.121 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.121 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.121 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.121 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookuphash_latency": { 2024-06-04T05:23:52.121 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.121 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.121 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.121 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.121 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupino_latency": { 2024-06-04T05:23:52.121 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.121 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.121 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.122 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.122 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupname_latency": { 2024-06-04T05:23:52.122 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.122 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.122 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.122 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.122 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupparent_latency": { 2024-06-04T05:23:52.122 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.122 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.122 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.122 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.122 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupsnap_latency": { 2024-06-04T05:23:52.123 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.123 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.123 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.123 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lssnap_latency": { 2024-06-04T05:23:52.123 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.123 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.123 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.124 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mkdir_latency": { 2024-06-04T05:23:52.124 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.124 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.124 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.124 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mknod_latency": { 2024-06-04T05:23:52.124 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.124 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.124 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.124 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mksnap_latency": { 2024-06-04T05:23:52.125 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.125 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.125 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.125 INFO:teuthology.orchestra.run.smithi070.stdout: "req_open_latency": { 2024-06-04T05:23:52.125 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.125 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.125 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.125 INFO:teuthology.orchestra.run.smithi070.stdout: "req_readdir_latency": { 2024-06-04T05:23:52.125 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.125 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.125 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.126 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.126 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rename_latency": { 2024-06-04T05:23:52.126 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.126 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.126 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.126 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.126 INFO:teuthology.orchestra.run.smithi070.stdout: "req_renamesnap_latency": { 2024-06-04T05:23:52.126 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.126 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.126 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.126 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.126 INFO:teuthology.orchestra.run.smithi070.stdout: "req_snapdiff_latency": { 2024-06-04T05:23:52.126 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.127 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.127 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.127 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmdir_latency": { 2024-06-04T05:23:52.127 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.127 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.127 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.127 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmsnap_latency": { 2024-06-04T05:23:52.127 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.127 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.127 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.128 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmxattr_latency": { 2024-06-04T05:23:52.128 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.128 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.128 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.128 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setattr_latency": { 2024-06-04T05:23:52.128 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.128 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.128 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.128 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setdirlayout_latency": { 2024-06-04T05:23:52.128 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.128 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.128 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.129 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.129 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setfilelock_latency": { 2024-06-04T05:23:52.129 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.129 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.129 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.129 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.129 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setlayout_latency": { 2024-06-04T05:23:52.129 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.129 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.129 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.129 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.129 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setxattr_latency": { 2024-06-04T05:23:52.129 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.129 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.130 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.130 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.130 INFO:teuthology.orchestra.run.smithi070.stdout: "req_symlink_latency": { 2024-06-04T05:23:52.130 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.130 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.130 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.130 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.130 INFO:teuthology.orchestra.run.smithi070.stdout: "req_unlink_latency": { 2024-06-04T05:23:52.130 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.130 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.130 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.130 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.130 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_revoke_eviction": 0, 2024-06-04T05:23:52.131 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_acquisition_throttle": 0, 2024-06-04T05:23:52.131 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getvxattr_latency": { 2024-06-04T05:23:52.131 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.131 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.131 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.131 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:52.131 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.131 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_sessions": { 2024-06-04T05:23:52.131 INFO:teuthology.orchestra.run.smithi070.stdout: "session_count": 1, 2024-06-04T05:23:52.131 INFO:teuthology.orchestra.run.smithi070.stdout: "session_add": 1, 2024-06-04T05:23:52.131 INFO:teuthology.orchestra.run.smithi070.stdout: "session_remove": 0, 2024-06-04T05:23:52.131 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_open": 1, 2024-06-04T05:23:52.131 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_stale": 0, 2024-06-04T05:23:52.131 INFO:teuthology.orchestra.run.smithi070.stdout: "total_load": 0, 2024-06-04T05:23:52.132 INFO:teuthology.orchestra.run.smithi070.stdout: "average_load": 0, 2024-06-04T05:23:52.132 INFO:teuthology.orchestra.run.smithi070.stdout: "avg_session_uptime": 3697, 2024-06-04T05:23:52.132 INFO:teuthology.orchestra.run.smithi070.stdout: "mdthresh_evicted": 0 2024-06-04T05:23:52.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.132 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool": { 2024-06-04T05:23:52.132 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_bytes": 0, 2024-06-04T05:23:52.132 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_items": 0, 2024-06-04T05:23:52.132 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_bytes": 0, 2024-06-04T05:23:52.132 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_items": 0, 2024-06-04T05:23:52.132 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_bytes": 0, 2024-06-04T05:23:52.132 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_items": 0, 2024-06-04T05:23:52.132 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-04T05:23:52.132 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_items": 0, 2024-06-04T05:23:52.133 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-04T05:23:52.133 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_items": 0, 2024-06-04T05:23:52.133 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_bytes": 0, 2024-06-04T05:23:52.133 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_items": 0, 2024-06-04T05:23:52.133 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-04T05:23:52.133 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_items": 0, 2024-06-04T05:23:52.133 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_bytes": 0, 2024-06-04T05:23:52.133 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_items": 0, 2024-06-04T05:23:52.133 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_bytes": 0, 2024-06-04T05:23:52.133 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_items": 0, 2024-06-04T05:23:52.133 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-04T05:23:52.133 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_items": 0, 2024-06-04T05:23:52.133 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-04T05:23:52.134 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_items": 0, 2024-06-04T05:23:52.134 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_bytes": 0, 2024-06-04T05:23:52.134 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_items": 0, 2024-06-04T05:23:52.134 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_bytes": 0, 2024-06-04T05:23:52.134 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_items": 0, 2024-06-04T05:23:52.134 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-04T05:23:52.134 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_items": 0, 2024-06-04T05:23:52.134 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_bytes": 0, 2024-06-04T05:23:52.134 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_items": 0, 2024-06-04T05:23:52.134 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_bytes": 0, 2024-06-04T05:23:52.134 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_items": 0, 2024-06-04T05:23:52.134 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_bytes": 0, 2024-06-04T05:23:52.134 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_items": 0, 2024-06-04T05:23:52.134 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_bytes": 0, 2024-06-04T05:23:52.135 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_items": 0, 2024-06-04T05:23:52.135 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_bytes": 242578, 2024-06-04T05:23:52.135 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_items": 87, 2024-06-04T05:23:52.135 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_bytes": 0, 2024-06-04T05:23:52.135 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_items": 0, 2024-06-04T05:23:52.135 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bytes": 0, 2024-06-04T05:23:52.135 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_items": 0, 2024-06-04T05:23:52.135 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_bytes": 0, 2024-06-04T05:23:52.135 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_items": 0, 2024-06-04T05:23:52.135 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_bytes": 0, 2024-06-04T05:23:52.135 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_items": 0, 2024-06-04T05:23:52.135 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_bytes": 8224, 2024-06-04T05:23:52.135 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_items": 171, 2024-06-04T05:23:52.135 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_bytes": 0, 2024-06-04T05:23:52.136 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_items": 0, 2024-06-04T05:23:52.136 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_bytes": 0, 2024-06-04T05:23:52.136 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_items": 0, 2024-06-04T05:23:52.136 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_bytes": 8724504, 2024-06-04T05:23:52.136 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_items": 36639, 2024-06-04T05:23:52.136 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_bytes": 0, 2024-06-04T05:23:52.136 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_items": 0, 2024-06-04T05:23:52.136 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_bytes": 0, 2024-06-04T05:23:52.136 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_items": 0 2024-06-04T05:23:52.136 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.136 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter": { 2024-06-04T05:23:52.136 INFO:teuthology.orchestra.run.smithi070.stdout: "op_active": 0, 2024-06-04T05:23:52.136 INFO:teuthology.orchestra.run.smithi070.stdout: "op_laggy": 0, 2024-06-04T05:23:52.136 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send": 5836, 2024-06-04T05:23:52.137 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send_bytes": 21, 2024-06-04T05:23:52.137 INFO:teuthology.orchestra.run.smithi070.stdout: "op_resend": 3, 2024-06-04T05:23:52.137 INFO:teuthology.orchestra.run.smithi070.stdout: "op_reply": 5836, 2024-06-04T05:23:52.137 INFO:teuthology.orchestra.run.smithi070.stdout: "op_latency": { 2024-06-04T05:23:52.137 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 5836, 2024-06-04T05:23:52.137 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 3.986884514, 2024-06-04T05:23:52.137 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000683153 2024-06-04T05:23:52.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.137 INFO:teuthology.orchestra.run.smithi070.stdout: "op_inflight": 0, 2024-06-04T05:23:52.137 INFO:teuthology.orchestra.run.smithi070.stdout: "oplen_avg": { 2024-06-04T05:23:52.137 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 5836, 2024-06-04T05:23:52.137 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 5837 2024-06-04T05:23:52.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.137 INFO:teuthology.orchestra.run.smithi070.stdout: "op": 5836, 2024-06-04T05:23:52.138 INFO:teuthology.orchestra.run.smithi070.stdout: "op_r": 5836, 2024-06-04T05:23:52.138 INFO:teuthology.orchestra.run.smithi070.stdout: "op_w": 0, 2024-06-04T05:23:52.138 INFO:teuthology.orchestra.run.smithi070.stdout: "op_rmw": 0, 2024-06-04T05:23:52.138 INFO:teuthology.orchestra.run.smithi070.stdout: "op_pg": 0, 2024-06-04T05:23:52.138 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_stat": 3418, 2024-06-04T05:23:52.138 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_create": 0, 2024-06-04T05:23:52.138 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_read": 2416, 2024-06-04T05:23:52.138 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_write": 0, 2024-06-04T05:23:52.138 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writefull": 0, 2024-06-04T05:23:52.138 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writesame": 0, 2024-06-04T05:23:52.138 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_append": 0, 2024-06-04T05:23:52.138 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_zero": 0, 2024-06-04T05:23:52.138 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_truncate": 0, 2024-06-04T05:23:52.139 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_delete": 0, 2024-06-04T05:23:52.139 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_mapext": 0, 2024-06-04T05:23:52.139 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_sparse_read": 0, 2024-06-04T05:23:52.139 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_clonerange": 0, 2024-06-04T05:23:52.139 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_getxattr": 1, 2024-06-04T05:23:52.139 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_setxattr": 0, 2024-06-04T05:23:52.139 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_cmpxattr": 0, 2024-06-04T05:23:52.139 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_rmxattr": 0, 2024-06-04T05:23:52.139 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_resetxattrs": 0, 2024-06-04T05:23:52.139 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_call": 0, 2024-06-04T05:23:52.139 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_watch": 0, 2024-06-04T05:23:52.139 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_notify": 0, 2024-06-04T05:23:52.139 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_src_cmpxattr": 0, 2024-06-04T05:23:52.139 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls": 0, 2024-06-04T05:23:52.140 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls_filter": 0, 2024-06-04T05:23:52.140 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_other": 0, 2024-06-04T05:23:52.140 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_active": 0, 2024-06-04T05:23:52.140 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_send": 0, 2024-06-04T05:23:52.140 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_resend": 0, 2024-06-04T05:23:52.140 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_ping": 0, 2024-06-04T05:23:52.140 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_active": 0, 2024-06-04T05:23:52.140 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_send": 0, 2024-06-04T05:23:52.140 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_resend": 0, 2024-06-04T05:23:52.140 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_active": 0, 2024-06-04T05:23:52.140 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_send": 0, 2024-06-04T05:23:52.140 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_resend": 0, 2024-06-04T05:23:52.140 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_active": 0, 2024-06-04T05:23:52.140 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_send": 0, 2024-06-04T05:23:52.141 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_resend": 0, 2024-06-04T05:23:52.141 INFO:teuthology.orchestra.run.smithi070.stdout: "command_active": 0, 2024-06-04T05:23:52.141 INFO:teuthology.orchestra.run.smithi070.stdout: "command_send": 0, 2024-06-04T05:23:52.141 INFO:teuthology.orchestra.run.smithi070.stdout: "command_resend": 0, 2024-06-04T05:23:52.141 INFO:teuthology.orchestra.run.smithi070.stdout: "map_epoch": 120, 2024-06-04T05:23:52.141 INFO:teuthology.orchestra.run.smithi070.stdout: "map_full": 0, 2024-06-04T05:23:52.141 INFO:teuthology.orchestra.run.smithi070.stdout: "map_inc": 41, 2024-06-04T05:23:52.141 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_sessions": 11, 2024-06-04T05:23:52.141 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_open": 13, 2024-06-04T05:23:52.141 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_close": 2, 2024-06-04T05:23:52.141 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_laggy": 0, 2024-06-04T05:23:52.141 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_wr": 0, 2024-06-04T05:23:52.141 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_rd": 2, 2024-06-04T05:23:52.141 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_del": 0 2024-06-04T05:23:52.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.142 INFO:teuthology.orchestra.run.smithi070.stdout: "oft": { 2024-06-04T05:23:52.142 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_objs": 0, 2024-06-04T05:23:52.142 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_kv_pairs": 0, 2024-06-04T05:23:52.142 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_updates": 0, 2024-06-04T05:23:52.142 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_removes": 0 2024-06-04T05:23:52.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.142 INFO:teuthology.orchestra.run.smithi070.stdout: "osd-slow-ops": { 2024-06-04T05:23:52.142 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_ops_count": 0 2024-06-04T05:23:52.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.142 INFO:teuthology.orchestra.run.smithi070.stdout: "purge_queue": { 2024-06-04T05:23:52.142 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops": 0, 2024-06-04T05:23:52.142 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops_high_water": 0, 2024-06-04T05:23:52.143 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing": 0, 2024-06-04T05:23:52.143 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_high_water": 0, 2024-06-04T05:23:52.143 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed_ops": 0, 2024-06-04T05:23:52.143 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed": 0, 2024-06-04T05:23:52.143 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_item_in_journal": 0 2024-06-04T05:23:52.143 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.143 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-04T05:23:52.143 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-06-04T05:23:52.143 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-06-04T05:23:52.143 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-06-04T05:23:52.143 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 6424, 2024-06-04T05:23:52.143 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 187310612, 2024-06-04T05:23:52.143 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-06-04T05:23:52.144 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 6424, 2024-06-04T05:23:52.144 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-06-04T05:23:52.144 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-06-04T05:23:52.144 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 6424, 2024-06-04T05:23:52.144 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 187310612, 2024-06-04T05:23:52.144 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-06-04T05:23:52.144 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.144 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.144 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.144 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:52.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.144 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_bytes": { 2024-06-04T05:23:52.144 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-06-04T05:23:52.144 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-06-04T05:23:52.145 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-06-04T05:23:52.145 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-06-04T05:23:52.145 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-06-04T05:23:52.145 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-06-04T05:23:52.145 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-06-04T05:23:52.145 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 5836, 2024-06-04T05:23:52.145 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 184300585, 2024-06-04T05:23:52.145 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 5836, 2024-06-04T05:23:52.145 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 184300585, 2024-06-04T05:23:52.145 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-06-04T05:23:52.145 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.145 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.145 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.145 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:52.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.146 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_ops": { 2024-06-04T05:23:52.146 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-06-04T05:23:52.146 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 1024, 2024-06-04T05:23:52.146 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-06-04T05:23:52.146 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-06-04T05:23:52.146 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-06-04T05:23:52.146 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-06-04T05:23:52.146 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-06-04T05:23:52.146 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 5836, 2024-06-04T05:23:52.146 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 5836, 2024-06-04T05:23:52.146 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 5836, 2024-06-04T05:23:52.147 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 5836, 2024-06-04T05:23:52.147 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-06-04T05:23:52.147 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.147 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.147 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.147 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:52.147 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.147 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle": { 2024-06-04T05:23:52.147 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-06-04T05:23:52.147 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-06-04T05:23:52.147 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-06-04T05:23:52.147 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-06-04T05:23:52.148 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-06-04T05:23:52.148 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-06-04T05:23:52.148 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-06-04T05:23:52.148 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-06-04T05:23:52.148 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-06-04T05:23:52.148 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 0, 2024-06-04T05:23:52.148 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 0, 2024-06-04T05:23:52.148 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-06-04T05:23:52.148 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.148 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.148 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.148 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:52.148 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:52.148 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle-0x5568ac2346e0": { 2024-06-04T05:23:52.149 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-06-04T05:23:52.149 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-06-04T05:23:52.149 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-06-04T05:23:52.149 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-06-04T05:23:52.149 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-06-04T05:23:52.149 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-06-04T05:23:52.149 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-06-04T05:23:52.149 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-06-04T05:23:52.149 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-06-04T05:23:52.149 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 0, 2024-06-04T05:23:52.149 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 0, 2024-06-04T05:23:52.149 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-06-04T05:23:52.149 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:52.150 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:52.150 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:52.150 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:52.150 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:52.150 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:23:52.952 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-06-04T05:23:52.952 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-06-04T05:23:52.952 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-06-04T05:23:52.952 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-06-04T05:23:52.952 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-06-04T05:23:52.952 DEBUG:tasks.check_counter:Getting stats from d 2024-06-04T05:23:52.953 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph daemon mds.d perf dump 2024-06-04T05:23:53.197 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:53.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:53 smithi070 ceph-mon[32631]: pgmap v842: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 14 op/s 2024-06-04T05:23:53.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:53 smithi112 ceph-mon[36453]: pgmap v842: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 14 op/s 2024-06-04T05:23:53.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:53 smithi098 ceph-mon[35569]: pgmap v842: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 14 op/s 2024-06-04T05:23:55.028 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:23:55.029 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-0": { 2024-06-04T05:23:55.029 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 4, 2024-06-04T05:23:55.029 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 4, 2024-06-04T05:23:55.029 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 2912, 2024-06-04T05:23:55.029 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 448, 2024-06-04T05:23:55.029 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 4, 2024-06-04T05:23:55.029 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 0, 2024-06-04T05:23:55.029 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.001487317, 2024-06-04T05:23:55.029 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.000032529, 2024-06-04T05:23:55.029 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.000606700, 2024-06-04T05:23:55.029 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-04T05:23:55.029 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:23:55.029 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 3, 2024-06-04T05:23:55.030 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000628593, 2024-06-04T05:23:55.030 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000209531 2024-06-04T05:23:55.030 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:55.030 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:23:55.030 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:55.030 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:55.030 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:55.030 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:55.030 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 2912, 2024-06-04T05:23:55.030 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 448 2024-06-04T05:23:55.030 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:55.030 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-1": { 2024-06-04T05:23:55.030 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 3, 2024-06-04T05:23:55.031 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 376, 2024-06-04T05:23:55.031 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 480, 2024-06-04T05:23:55.031 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 302240, 2024-06-04T05:23:55.031 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 6, 2024-06-04T05:23:55.031 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 3, 2024-06-04T05:23:55.031 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.025633826, 2024-06-04T05:23:55.031 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.020068439, 2024-06-04T05:23:55.031 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.002067497, 2024-06-04T05:23:55.031 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-04T05:23:55.031 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:23:55.031 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 373, 2024-06-04T05:23:55.031 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.018830824, 2024-06-04T05:23:55.031 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000050484 2024-06-04T05:23:55.032 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:55.032 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:23:55.032 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:55.032 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:55.032 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:55.032 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:55.032 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 480, 2024-06-04T05:23:55.032 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 302240 2024-06-04T05:23:55.032 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:55.032 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-2": { 2024-06-04T05:23:55.032 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 502, 2024-06-04T05:23:55.032 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 492, 2024-06-04T05:23:55.032 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 1641584, 2024-06-04T05:23:55.032 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 304768, 2024-06-04T05:23:55.033 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 2, 2024-06-04T05:23:55.033 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 1, 2024-06-04T05:23:55.033 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.070689070, 2024-06-04T05:23:55.033 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.026910604, 2024-06-04T05:23:55.033 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.041692938, 2024-06-04T05:23:55.033 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.009178861, 2024-06-04T05:23:55.033 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:23:55.033 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 491, 2024-06-04T05:23:55.033 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.016520333, 2024-06-04T05:23:55.033 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000033646 2024-06-04T05:23:55.033 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:55.033 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:23:55.033 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:55.033 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:55.034 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:55.034 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:55.034 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 1641584, 2024-06-04T05:23:55.034 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 304768 2024-06-04T05:23:55.034 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:55.034 INFO:teuthology.orchestra.run.smithi070.stdout: "cct": { 2024-06-04T05:23:55.034 INFO:teuthology.orchestra.run.smithi070.stdout: "total_workers": 0, 2024-06-04T05:23:55.034 INFO:teuthology.orchestra.run.smithi070.stdout: "unhealthy_workers": 0 2024-06-04T05:23:55.034 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:55.034 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool": { 2024-06-04T05:23:55.034 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_bytes": 0, 2024-06-04T05:23:55.034 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_items": 0, 2024-06-04T05:23:55.034 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_bytes": 0, 2024-06-04T05:23:55.034 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_items": 0, 2024-06-04T05:23:55.035 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_bytes": 0, 2024-06-04T05:23:55.035 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_items": 0, 2024-06-04T05:23:55.035 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-04T05:23:55.035 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_items": 0, 2024-06-04T05:23:55.035 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-04T05:23:55.035 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_items": 0, 2024-06-04T05:23:55.035 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_bytes": 0, 2024-06-04T05:23:55.035 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_items": 0, 2024-06-04T05:23:55.035 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-04T05:23:55.035 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_items": 0, 2024-06-04T05:23:55.035 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_bytes": 0, 2024-06-04T05:23:55.035 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_items": 0, 2024-06-04T05:23:55.035 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_bytes": 0, 2024-06-04T05:23:55.035 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_items": 0, 2024-06-04T05:23:55.036 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-04T05:23:55.036 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_items": 0, 2024-06-04T05:23:55.036 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-04T05:23:55.036 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_items": 0, 2024-06-04T05:23:55.036 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_bytes": 0, 2024-06-04T05:23:55.036 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_items": 0, 2024-06-04T05:23:55.036 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_bytes": 0, 2024-06-04T05:23:55.036 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_items": 0, 2024-06-04T05:23:55.036 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-04T05:23:55.036 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_items": 0, 2024-06-04T05:23:55.036 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_bytes": 0, 2024-06-04T05:23:55.036 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_items": 0, 2024-06-04T05:23:55.036 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_bytes": 0, 2024-06-04T05:23:55.036 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_items": 0, 2024-06-04T05:23:55.037 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_bytes": 0, 2024-06-04T05:23:55.037 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_items": 0, 2024-06-04T05:23:55.037 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_bytes": 0, 2024-06-04T05:23:55.037 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_items": 0, 2024-06-04T05:23:55.037 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_bytes": 148784, 2024-06-04T05:23:55.037 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_items": 31, 2024-06-04T05:23:55.037 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_bytes": 0, 2024-06-04T05:23:55.037 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_items": 0, 2024-06-04T05:23:55.037 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bytes": 0, 2024-06-04T05:23:55.037 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_items": 0, 2024-06-04T05:23:55.037 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_bytes": 0, 2024-06-04T05:23:55.037 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_items": 0, 2024-06-04T05:23:55.037 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_bytes": 0, 2024-06-04T05:23:55.037 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_items": 0, 2024-06-04T05:23:55.038 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_bytes": 0, 2024-06-04T05:23:55.038 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_items": 0, 2024-06-04T05:23:55.038 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_bytes": 0, 2024-06-04T05:23:55.038 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_items": 0, 2024-06-04T05:23:55.038 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_bytes": 0, 2024-06-04T05:23:55.038 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_items": 0, 2024-06-04T05:23:55.038 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_bytes": 928, 2024-06-04T05:23:55.038 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_items": 2, 2024-06-04T05:23:55.038 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_bytes": 0, 2024-06-04T05:23:55.038 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_items": 0, 2024-06-04T05:23:55.038 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_bytes": 0, 2024-06-04T05:23:55.038 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_items": 0 2024-06-04T05:23:55.038 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:55.039 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-04T05:23:55.039 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-06-04T05:23:55.039 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-06-04T05:23:55.039 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-06-04T05:23:55.039 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 505, 2024-06-04T05:23:55.039 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 1570624, 2024-06-04T05:23:55.039 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-06-04T05:23:55.039 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 505, 2024-06-04T05:23:55.039 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-06-04T05:23:55.039 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-06-04T05:23:55.039 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 505, 2024-06-04T05:23:55.039 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 1570624, 2024-06-04T05:23:55.039 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-06-04T05:23:55.039 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:55.040 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:55.040 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:55.040 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:55.040 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:55.040 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:23:55.781 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:55 smithi070 ceph-mon[32631]: pgmap v843: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:23:55.809 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-06-04T05:23:55.809 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-06-04T05:23:55.809 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-06-04T05:23:55.809 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-06-04T05:23:55.809 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-06-04T05:23:55.809 DEBUG:tasks.check_counter:Getting stats from g 2024-06-04T05:23:55.809 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph daemon mds.g perf dump 2024-06-04T05:23:55.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:55 smithi112 ceph-mon[36453]: pgmap v843: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:23:55.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:55 smithi098 ceph-mon[35569]: pgmap v843: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:23:56.051 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:57.795 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:57 smithi070 ceph-mon[32631]: pgmap v844: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 14 op/s 2024-06-04T05:23:57.926 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:23:57.926 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-0": { 2024-06-04T05:23:57.926 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 2854, 2024-06-04T05:23:57.926 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 2798, 2024-06-04T05:23:57.926 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 65624278, 2024-06-04T05:23:57.927 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 1030208, 2024-06-04T05:23:57.927 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 7, 2024-06-04T05:23:57.927 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 3, 2024-06-04T05:23:57.927 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.307092206, 2024-06-04T05:23:57.927 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.109172573, 2024-06-04T05:23:57.927 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.120910611, 2024-06-04T05:23:57.927 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.068303331, 2024-06-04T05:23:57.927 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:23:57.927 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 2797, 2024-06-04T05:23:57.927 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.094526420, 2024-06-04T05:23:57.927 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000033795 2024-06-04T05:23:57.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.927 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:23:57.928 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.928 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.928 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.928 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.928 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 1986768, 2024-06-04T05:23:57.928 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 310256 2024-06-04T05:23:57.928 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.928 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-1": { 2024-06-04T05:23:57.928 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 2712, 2024-06-04T05:23:57.928 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 3081, 2024-06-04T05:23:57.928 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 80383644, 2024-06-04T05:23:57.928 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 1684181, 2024-06-04T05:23:57.928 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 9, 2024-06-04T05:23:57.928 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 7, 2024-06-04T05:23:57.929 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.233517437, 2024-06-04T05:23:57.929 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.059299139, 2024-06-04T05:23:57.929 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.101755893, 2024-06-04T05:23:57.929 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.057662476, 2024-06-04T05:23:57.929 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:23:57.929 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 3079, 2024-06-04T05:23:57.929 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.082953134, 2024-06-04T05:23:57.929 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000026941 2024-06-04T05:23:57.929 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.929 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:23:57.929 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.929 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.930 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.930 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-06-04T05:23:57.930 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 823616 2024-06-04T05:23:57.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.930 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-2": { 2024-06-04T05:23:57.930 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 1083, 2024-06-04T05:23:57.930 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 1085, 2024-06-04T05:23:57.930 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 118939543, 2024-06-04T05:23:57.930 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 344406, 2024-06-04T05:23:57.930 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 8, 2024-06-04T05:23:57.930 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 5, 2024-06-04T05:23:57.930 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.155143244, 2024-06-04T05:23:57.931 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.021322787, 2024-06-04T05:23:57.931 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.102725215, 2024-06-04T05:23:57.931 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.024772389, 2024-06-04T05:23:57.931 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:23:57.931 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 1083, 2024-06-04T05:23:57.931 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.026305196, 2024-06-04T05:23:57.931 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000024289 2024-06-04T05:23:57.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.931 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:23:57.931 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.931 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.931 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.931 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 0, 2024-06-04T05:23:57.932 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 0 2024-06-04T05:23:57.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.932 INFO:teuthology.orchestra.run.smithi070.stdout: "cct": { 2024-06-04T05:23:57.932 INFO:teuthology.orchestra.run.smithi070.stdout: "total_workers": 1, 2024-06-04T05:23:57.932 INFO:teuthology.orchestra.run.smithi070.stdout: "unhealthy_workers": 0 2024-06-04T05:23:57.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.932 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-MDSRank": { 2024-06-04T05:23:57.932 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-06-04T05:23:57.932 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-06-04T05:23:57.932 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 10113, 2024-06-04T05:23:57.932 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.220022278, 2024-06-04T05:23:57.932 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000021756 2024-06-04T05:23:57.932 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:57.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.933 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-PurgeQueue": { 2024-06-04T05:23:57.933 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-06-04T05:23:57.933 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-06-04T05:23:57.933 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.933 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.933 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.933 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:57.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.933 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-06-04T05:23:57.933 INFO:teuthology.orchestra.run.smithi070.stdout: "request": 0, 2024-06-04T05:23:57.933 INFO:teuthology.orchestra.run.smithi070.stdout: "reply": 0, 2024-06-04T05:23:57.933 INFO:teuthology.orchestra.run.smithi070.stdout: "reply_latency": { 2024-06-04T05:23:57.933 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.933 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.934 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.934 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_reply": 0, 2024-06-04T05:23:57.934 INFO:teuthology.orchestra.run.smithi070.stdout: "forward": 0, 2024-06-04T05:23:57.934 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_complete": 0, 2024-06-04T05:23:57.934 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_keys": 0, 2024-06-04T05:23:57.934 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_commit": 0, 2024-06-04T05:23:57.934 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_split": 0, 2024-06-04T05:23:57.934 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_merge": 0, 2024-06-04T05:23:57.934 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes": 11798, 2024-06-04T05:23:57.934 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_top": 0, 2024-06-04T05:23:57.934 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_bottom": 11798, 2024-06-04T05:23:57.934 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pin_tail": 0, 2024-06-04T05:23:57.935 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pinned": 11798, 2024-06-04T05:23:57.935 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_expired": 0, 2024-06-04T05:23:57.935 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_with_caps": 0, 2024-06-04T05:23:57.935 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": 0, 2024-06-04T05:23:57.935 INFO:teuthology.orchestra.run.smithi070.stdout: "subtrees": 1, 2024-06-04T05:23:57.935 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse": 0, 2024-06-04T05:23:57.935 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_hit": 0, 2024-06-04T05:23:57.935 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_forward": 0, 2024-06-04T05:23:57.935 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_discover": 0, 2024-06-04T05:23:57.935 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_dir_fetch": 0, 2024-06-04T05:23:57.935 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_remote_ino": 0, 2024-06-04T05:23:57.935 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_lock": 0, 2024-06-04T05:23:57.935 INFO:teuthology.orchestra.run.smithi070.stdout: "load_cent": 0, 2024-06-04T05:23:57.935 INFO:teuthology.orchestra.run.smithi070.stdout: "q": 0, 2024-06-04T05:23:57.935 INFO:teuthology.orchestra.run.smithi070.stdout: "exported": 0, 2024-06-04T05:23:57.936 INFO:teuthology.orchestra.run.smithi070.stdout: "exported_inodes": 0, 2024-06-04T05:23:57.936 INFO:teuthology.orchestra.run.smithi070.stdout: "imported": 0, 2024-06-04T05:23:57.936 INFO:teuthology.orchestra.run.smithi070.stdout: "imported_inodes": 0, 2024-06-04T05:23:57.936 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_dir_fetch": 0, 2024-06-04T05:23:57.936 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_backtrace_fetch": 0, 2024-06-04T05:23:57.936 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_peer_discover": 0, 2024-06-04T05:23:57.936 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rfiles": 0, 2024-06-04T05:23:57.936 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rbytes": 0, 2024-06-04T05:23:57.936 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rsnaps": 0, 2024-06-04T05:23:57.936 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_fetch": 0, 2024-06-04T05:23:57.936 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_set_tag": 0, 2024-06-04T05:23:57.936 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_repaired": 0, 2024-06-04T05:23:57.936 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_inotable_repaired": 0, 2024-06-04T05:23:57.936 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_inodes": 0, 2024-06-04T05:23:57.937 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_base_inodes": 0, 2024-06-04T05:23:57.937 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dirfrag_rstats": 0, 2024-06-04T05:23:57.937 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_file_inodes": 0, 2024-06-04T05:23:57.937 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_inode_file_caps": 0, 2024-06-04T05:23:57.937 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_revoke": 0, 2024-06-04T05:23:57.937 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_grant": 0, 2024-06-04T05:23:57.937 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_trunc": 0, 2024-06-04T05:23:57.937 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-04T05:23:57.937 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-04T05:23:57.937 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps": 0, 2024-06-04T05:23:57.937 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps_dirty": 0, 2024-06-04T05:23:57.937 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_cap_release": 0, 2024-06-04T05:23:57.937 INFO:teuthology.orchestra.run.smithi070.stdout: "process_request_cap_release": 0 2024-06-04T05:23:57.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.938 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache": { 2024-06-04T05:23:57.938 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update": 0, 2024-06-04T05:23:57.938 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update_receipt": 0, 2024-06-04T05:23:57.938 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_try_discover": 0, 2024-06-04T05:23:57.938 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_send_discover": 0, 2024-06-04T05:23:57.938 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_handle_discover": 0, 2024-06-04T05:23:57.938 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays": 0, 2024-06-04T05:23:57.938 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_delayed": 0, 2024-06-04T05:23:57.938 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_enqueuing": 0, 2024-06-04T05:23:57.938 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_created": 0, 2024-06-04T05:23:57.938 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_enqueued": 0, 2024-06-04T05:23:57.938 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_reintegrated": 0, 2024-06-04T05:23:57.938 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_migrated": 0, 2024-06-04T05:23:57.938 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_processing": 0, 2024-06-04T05:23:57.939 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_enqueued": 0, 2024-06-04T05:23:57.939 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_prioritized": 0, 2024-06-04T05:23:57.939 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_started": 0, 2024-06-04T05:23:57.939 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_completed": 0, 2024-06-04T05:23:57.939 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_quiesce_path": 0, 2024-06-04T05:23:57.939 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_quiesce_inode": 0, 2024-06-04T05:23:57.939 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_enqueue_scrub": 0, 2024-06-04T05:23:57.939 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_exportdir": 0, 2024-06-04T05:23:57.939 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_flush": 0, 2024-06-04T05:23:57.939 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragmentdir": 0, 2024-06-04T05:23:57.939 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragstats": 0, 2024-06-04T05:23:57.939 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_inodestats": 0 2024-06-04T05:23:57.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.939 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log": { 2024-06-04T05:23:57.940 INFO:teuthology.orchestra.run.smithi070.stdout: "evlrg": 0, 2024-06-04T05:23:57.940 INFO:teuthology.orchestra.run.smithi070.stdout: "evadd": 0, 2024-06-04T05:23:57.940 INFO:teuthology.orchestra.run.smithi070.stdout: "evex": 0, 2024-06-04T05:23:57.940 INFO:teuthology.orchestra.run.smithi070.stdout: "evtrm": 0, 2024-06-04T05:23:57.940 INFO:teuthology.orchestra.run.smithi070.stdout: "ev": 78393, 2024-06-04T05:23:57.940 INFO:teuthology.orchestra.run.smithi070.stdout: "evexg": 0, 2024-06-04T05:23:57.940 INFO:teuthology.orchestra.run.smithi070.stdout: "evexd": 0, 2024-06-04T05:23:57.940 INFO:teuthology.orchestra.run.smithi070.stdout: "segadd": 0, 2024-06-04T05:23:57.940 INFO:teuthology.orchestra.run.smithi070.stdout: "segex": 0, 2024-06-04T05:23:57.940 INFO:teuthology.orchestra.run.smithi070.stdout: "segtrm": 0, 2024-06-04T05:23:57.940 INFO:teuthology.orchestra.run.smithi070.stdout: "seg": 116, 2024-06-04T05:23:57.940 INFO:teuthology.orchestra.run.smithi070.stdout: "segmjr": 8, 2024-06-04T05:23:57.940 INFO:teuthology.orchestra.run.smithi070.stdout: "segexg": 0, 2024-06-04T05:23:57.940 INFO:teuthology.orchestra.run.smithi070.stdout: "segexd": 0, 2024-06-04T05:23:57.941 INFO:teuthology.orchestra.run.smithi070.stdout: "expos": 4194304, 2024-06-04T05:23:57.941 INFO:teuthology.orchestra.run.smithi070.stdout: "wrpos": 265462912, 2024-06-04T05:23:57.941 INFO:teuthology.orchestra.run.smithi070.stdout: "rdpos": 265448619, 2024-06-04T05:23:57.941 INFO:teuthology.orchestra.run.smithi070.stdout: "jlat": { 2024-06-04T05:23:57.941 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.941 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.941 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.941 INFO:teuthology.orchestra.run.smithi070.stdout: "replayed": 78393 2024-06-04T05:23:57.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.941 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_mem": { 2024-06-04T05:23:57.941 INFO:teuthology.orchestra.run.smithi070.stdout: "ino": 1564, 2024-06-04T05:23:57.941 INFO:teuthology.orchestra.run.smithi070.stdout: "ino+": 12875, 2024-06-04T05:23:57.941 INFO:teuthology.orchestra.run.smithi070.stdout: "ino-": 11311, 2024-06-04T05:23:57.942 INFO:teuthology.orchestra.run.smithi070.stdout: "dir": 57, 2024-06-04T05:23:57.942 INFO:teuthology.orchestra.run.smithi070.stdout: "dir+": 139, 2024-06-04T05:23:57.942 INFO:teuthology.orchestra.run.smithi070.stdout: "dir-": 82, 2024-06-04T05:23:57.942 INFO:teuthology.orchestra.run.smithi070.stdout: "dn": 11798, 2024-06-04T05:23:57.942 INFO:teuthology.orchestra.run.smithi070.stdout: "dn+": 13300, 2024-06-04T05:23:57.942 INFO:teuthology.orchestra.run.smithi070.stdout: "dn-": 1502, 2024-06-04T05:23:57.942 INFO:teuthology.orchestra.run.smithi070.stdout: "cap": 0, 2024-06-04T05:23:57.942 INFO:teuthology.orchestra.run.smithi070.stdout: "cap+": 0, 2024-06-04T05:23:57.942 INFO:teuthology.orchestra.run.smithi070.stdout: "cap-": 0, 2024-06-04T05:23:57.942 INFO:teuthology.orchestra.run.smithi070.stdout: "rss": 72944, 2024-06-04T05:23:57.942 INFO:teuthology.orchestra.run.smithi070.stdout: "heap": 215416 2024-06-04T05:23:57.942 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.942 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_server": { 2024-06-04T05:23:57.942 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_client_request": 0, 2024-06-04T05:23:57.943 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_server_request": 0, 2024-06-04T05:23:57.943 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_request": 0, 2024-06-04T05:23:57.943 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_session": 0, 2024-06-04T05:23:57.943 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_peer_request": 0, 2024-06-04T05:23:57.943 INFO:teuthology.orchestra.run.smithi070.stdout: "req_create_latency": { 2024-06-04T05:23:57.943 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.943 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.943 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.943 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.943 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getattr_latency": { 2024-06-04T05:23:57.943 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.943 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.943 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.943 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.944 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getfilelock_latency": { 2024-06-04T05:23:57.944 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.944 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.944 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.944 INFO:teuthology.orchestra.run.smithi070.stdout: "req_link_latency": { 2024-06-04T05:23:57.944 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.944 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.944 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.944 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookup_latency": { 2024-06-04T05:23:57.944 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.944 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.944 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.945 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookuphash_latency": { 2024-06-04T05:23:57.945 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.945 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.945 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.945 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupino_latency": { 2024-06-04T05:23:57.945 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.945 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.945 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.945 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupname_latency": { 2024-06-04T05:23:57.945 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.945 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.946 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.946 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.946 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupparent_latency": { 2024-06-04T05:23:57.946 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.946 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.946 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.946 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.946 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupsnap_latency": { 2024-06-04T05:23:57.946 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.946 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.946 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.946 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.946 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lssnap_latency": { 2024-06-04T05:23:57.947 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.947 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.947 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.947 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.947 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:57 smithi112 ceph-mon[36453]: pgmap v844: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 14 op/s 2024-06-04T05:23:57.948 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mkdir_latency": { 2024-06-04T05:23:57.948 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.948 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.948 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.948 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mknod_latency": { 2024-06-04T05:23:57.948 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.948 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.948 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.949 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mksnap_latency": { 2024-06-04T05:23:57.949 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.949 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.949 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.949 INFO:teuthology.orchestra.run.smithi070.stdout: "req_open_latency": { 2024-06-04T05:23:57.949 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.949 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.949 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.949 INFO:teuthology.orchestra.run.smithi070.stdout: "req_readdir_latency": { 2024-06-04T05:23:57.949 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.949 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.950 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.950 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.950 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rename_latency": { 2024-06-04T05:23:57.950 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.950 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.950 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.950 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.950 INFO:teuthology.orchestra.run.smithi070.stdout: "req_renamesnap_latency": { 2024-06-04T05:23:57.950 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.950 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.950 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.950 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.950 INFO:teuthology.orchestra.run.smithi070.stdout: "req_snapdiff_latency": { 2024-06-04T05:23:57.950 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.951 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.951 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.951 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.951 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmdir_latency": { 2024-06-04T05:23:57.951 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.951 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.951 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.951 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.951 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmsnap_latency": { 2024-06-04T05:23:57.951 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.951 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.951 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.951 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.951 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmxattr_latency": { 2024-06-04T05:23:57.952 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.952 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.952 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.952 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.952 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setattr_latency": { 2024-06-04T05:23:57.952 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.952 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.952 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.952 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.952 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setdirlayout_latency": { 2024-06-04T05:23:57.952 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.952 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.952 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.953 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setfilelock_latency": { 2024-06-04T05:23:57.953 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.953 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.953 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.953 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setlayout_latency": { 2024-06-04T05:23:57.953 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.953 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.953 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.953 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setxattr_latency": { 2024-06-04T05:23:57.953 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.953 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.954 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.954 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.954 INFO:teuthology.orchestra.run.smithi070.stdout: "req_symlink_latency": { 2024-06-04T05:23:57.954 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.954 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.954 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.954 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.954 INFO:teuthology.orchestra.run.smithi070.stdout: "req_unlink_latency": { 2024-06-04T05:23:57.954 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.954 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.954 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.954 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.954 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_revoke_eviction": 0, 2024-06-04T05:23:57.954 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_acquisition_throttle": 0, 2024-06-04T05:23:57.955 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getvxattr_latency": { 2024-06-04T05:23:57.955 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.955 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.955 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.955 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:57.955 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.955 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_sessions": { 2024-06-04T05:23:57.955 INFO:teuthology.orchestra.run.smithi070.stdout: "session_count": 1, 2024-06-04T05:23:57.955 INFO:teuthology.orchestra.run.smithi070.stdout: "session_add": 4, 2024-06-04T05:23:57.955 INFO:teuthology.orchestra.run.smithi070.stdout: "session_remove": 3, 2024-06-04T05:23:57.955 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_open": 3, 2024-06-04T05:23:57.955 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_stale": 0, 2024-06-04T05:23:57.955 INFO:teuthology.orchestra.run.smithi070.stdout: "total_load": 0, 2024-06-04T05:23:57.955 INFO:teuthology.orchestra.run.smithi070.stdout: "average_load": 0, 2024-06-04T05:23:57.956 INFO:teuthology.orchestra.run.smithi070.stdout: "avg_session_uptime": 7898, 2024-06-04T05:23:57.956 INFO:teuthology.orchestra.run.smithi070.stdout: "mdthresh_evicted": 0 2024-06-04T05:23:57.956 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.956 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool": { 2024-06-04T05:23:57.956 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_bytes": 0, 2024-06-04T05:23:57.956 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_items": 0, 2024-06-04T05:23:57.956 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_bytes": 0, 2024-06-04T05:23:57.956 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_items": 0, 2024-06-04T05:23:57.956 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_bytes": 0, 2024-06-04T05:23:57.956 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_items": 0, 2024-06-04T05:23:57.956 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-04T05:23:57.956 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_items": 0, 2024-06-04T05:23:57.956 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-04T05:23:57.956 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_items": 0, 2024-06-04T05:23:57.957 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_bytes": 0, 2024-06-04T05:23:57.957 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_items": 0, 2024-06-04T05:23:57.957 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-04T05:23:57.957 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_items": 0, 2024-06-04T05:23:57.957 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_bytes": 0, 2024-06-04T05:23:57.957 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_items": 0, 2024-06-04T05:23:57.957 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_bytes": 0, 2024-06-04T05:23:57.957 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_items": 0, 2024-06-04T05:23:57.957 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-04T05:23:57.957 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_items": 0, 2024-06-04T05:23:57.957 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-04T05:23:57.957 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_items": 0, 2024-06-04T05:23:57.957 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_bytes": 0, 2024-06-04T05:23:57.957 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_items": 0, 2024-06-04T05:23:57.957 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_bytes": 0, 2024-06-04T05:23:57.958 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_items": 0, 2024-06-04T05:23:57.958 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-04T05:23:57.958 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_items": 0, 2024-06-04T05:23:57.958 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_bytes": 0, 2024-06-04T05:23:57.958 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_items": 0, 2024-06-04T05:23:57.958 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_bytes": 0, 2024-06-04T05:23:57.958 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_items": 0, 2024-06-04T05:23:57.958 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_bytes": 0, 2024-06-04T05:23:57.958 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_items": 0, 2024-06-04T05:23:57.958 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_bytes": 0, 2024-06-04T05:23:57.958 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_items": 0, 2024-06-04T05:23:57.958 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_bytes": 241857, 2024-06-04T05:23:57.958 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_items": 86, 2024-06-04T05:23:57.959 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_bytes": 0, 2024-06-04T05:23:57.959 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_items": 0, 2024-06-04T05:23:57.959 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bytes": 0, 2024-06-04T05:23:57.959 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_items": 0, 2024-06-04T05:23:57.959 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_bytes": 0, 2024-06-04T05:23:57.959 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_items": 0, 2024-06-04T05:23:57.959 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_bytes": 0, 2024-06-04T05:23:57.959 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_items": 0, 2024-06-04T05:23:57.959 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_bytes": 8224, 2024-06-04T05:23:57.959 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_items": 171, 2024-06-04T05:23:57.959 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_bytes": 0, 2024-06-04T05:23:57.959 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_items": 0, 2024-06-04T05:23:57.959 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_bytes": 0, 2024-06-04T05:23:57.959 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_items": 0, 2024-06-04T05:23:57.960 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_bytes": 11920994, 2024-06-04T05:23:57.960 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_items": 54595, 2024-06-04T05:23:57.960 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_bytes": 0, 2024-06-04T05:23:57.960 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_items": 0, 2024-06-04T05:23:57.960 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_bytes": 0, 2024-06-04T05:23:57.960 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_items": 0 2024-06-04T05:23:57.960 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.960 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter": { 2024-06-04T05:23:57.960 INFO:teuthology.orchestra.run.smithi070.stdout: "op_active": 0, 2024-06-04T05:23:57.960 INFO:teuthology.orchestra.run.smithi070.stdout: "op_laggy": 0, 2024-06-04T05:23:57.960 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send": 6053, 2024-06-04T05:23:57.960 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send_bytes": 21, 2024-06-04T05:23:57.960 INFO:teuthology.orchestra.run.smithi070.stdout: "op_resend": 3, 2024-06-04T05:23:57.960 INFO:teuthology.orchestra.run.smithi070.stdout: "op_reply": 6053, 2024-06-04T05:23:57.960 INFO:teuthology.orchestra.run.smithi070.stdout: "op_latency": { 2024-06-04T05:23:57.961 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 6053, 2024-06-04T05:23:57.961 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 4.181879003, 2024-06-04T05:23:57.961 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000690877 2024-06-04T05:23:57.961 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.961 INFO:teuthology.orchestra.run.smithi070.stdout: "op_inflight": 0, 2024-06-04T05:23:57.961 INFO:teuthology.orchestra.run.smithi070.stdout: "oplen_avg": { 2024-06-04T05:23:57.961 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 6053, 2024-06-04T05:23:57.961 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 6054 2024-06-04T05:23:57.961 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.961 INFO:teuthology.orchestra.run.smithi070.stdout: "op": 6053, 2024-06-04T05:23:57.961 INFO:teuthology.orchestra.run.smithi070.stdout: "op_r": 6053, 2024-06-04T05:23:57.961 INFO:teuthology.orchestra.run.smithi070.stdout: "op_w": 0, 2024-06-04T05:23:57.961 INFO:teuthology.orchestra.run.smithi070.stdout: "op_rmw": 0, 2024-06-04T05:23:57.961 INFO:teuthology.orchestra.run.smithi070.stdout: "op_pg": 0, 2024-06-04T05:23:57.962 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_stat": 3641, 2024-06-04T05:23:57.962 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_create": 0, 2024-06-04T05:23:57.962 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_read": 2410, 2024-06-04T05:23:57.962 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_write": 0, 2024-06-04T05:23:57.962 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writefull": 0, 2024-06-04T05:23:57.962 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writesame": 0, 2024-06-04T05:23:57.962 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_append": 0, 2024-06-04T05:23:57.962 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_zero": 0, 2024-06-04T05:23:57.962 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_truncate": 0, 2024-06-04T05:23:57.962 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_delete": 0, 2024-06-04T05:23:57.962 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_mapext": 0, 2024-06-04T05:23:57.962 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_sparse_read": 0, 2024-06-04T05:23:57.962 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_clonerange": 0, 2024-06-04T05:23:57.962 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_getxattr": 1, 2024-06-04T05:23:57.963 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_setxattr": 0, 2024-06-04T05:23:57.963 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_cmpxattr": 0, 2024-06-04T05:23:57.963 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_rmxattr": 0, 2024-06-04T05:23:57.963 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_resetxattrs": 0, 2024-06-04T05:23:57.963 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_call": 0, 2024-06-04T05:23:57.963 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_watch": 0, 2024-06-04T05:23:57.963 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_notify": 0, 2024-06-04T05:23:57.963 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_src_cmpxattr": 0, 2024-06-04T05:23:57.963 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls": 0, 2024-06-04T05:23:57.963 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls_filter": 0, 2024-06-04T05:23:57.963 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_other": 0, 2024-06-04T05:23:57.963 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_active": 0, 2024-06-04T05:23:57.963 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_send": 0, 2024-06-04T05:23:57.963 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_resend": 0, 2024-06-04T05:23:57.964 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_ping": 0, 2024-06-04T05:23:57.964 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_active": 0, 2024-06-04T05:23:57.964 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_send": 0, 2024-06-04T05:23:57.964 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_resend": 0, 2024-06-04T05:23:57.964 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_active": 0, 2024-06-04T05:23:57.964 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_send": 0, 2024-06-04T05:23:57.964 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_resend": 0, 2024-06-04T05:23:57.964 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_active": 0, 2024-06-04T05:23:57.964 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_send": 0, 2024-06-04T05:23:57.964 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_resend": 0, 2024-06-04T05:23:57.964 INFO:teuthology.orchestra.run.smithi070.stdout: "command_active": 0, 2024-06-04T05:23:57.964 INFO:teuthology.orchestra.run.smithi070.stdout: "command_send": 0, 2024-06-04T05:23:57.964 INFO:teuthology.orchestra.run.smithi070.stdout: "command_resend": 0, 2024-06-04T05:23:57.964 INFO:teuthology.orchestra.run.smithi070.stdout: "map_epoch": 120, 2024-06-04T05:23:57.965 INFO:teuthology.orchestra.run.smithi070.stdout: "map_full": 0, 2024-06-04T05:23:57.965 INFO:teuthology.orchestra.run.smithi070.stdout: "map_inc": 41, 2024-06-04T05:23:57.965 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_sessions": 11, 2024-06-04T05:23:57.965 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_open": 13, 2024-06-04T05:23:57.965 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_close": 2, 2024-06-04T05:23:57.965 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_laggy": 0, 2024-06-04T05:23:57.965 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_wr": 0, 2024-06-04T05:23:57.965 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_rd": 2, 2024-06-04T05:23:57.965 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_del": 0 2024-06-04T05:23:57.965 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.965 INFO:teuthology.orchestra.run.smithi070.stdout: "oft": { 2024-06-04T05:23:57.965 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_objs": 0, 2024-06-04T05:23:57.965 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_kv_pairs": 0, 2024-06-04T05:23:57.965 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_updates": 0, 2024-06-04T05:23:57.966 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_removes": 0 2024-06-04T05:23:57.966 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.966 INFO:teuthology.orchestra.run.smithi070.stdout: "osd-slow-ops": { 2024-06-04T05:23:57.966 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_ops_count": 0 2024-06-04T05:23:57.966 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.966 INFO:teuthology.orchestra.run.smithi070.stdout: "purge_queue": { 2024-06-04T05:23:57.966 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops": 0, 2024-06-04T05:23:57.966 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops_high_water": 0, 2024-06-04T05:23:57.966 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing": 0, 2024-06-04T05:23:57.966 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_high_water": 0, 2024-06-04T05:23:57.966 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed_ops": 0, 2024-06-04T05:23:57.966 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed": 0, 2024-06-04T05:23:57.966 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_item_in_journal": 0 2024-06-04T05:23:57.966 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.966 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-04T05:23:57.967 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-06-04T05:23:57.967 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-06-04T05:23:57.967 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-06-04T05:23:57.967 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 6645, 2024-06-04T05:23:57.967 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 264316540, 2024-06-04T05:23:57.967 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-06-04T05:23:57.967 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 6645, 2024-06-04T05:23:57.967 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-06-04T05:23:57.967 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-06-04T05:23:57.968 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 6645, 2024-06-04T05:23:57.968 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 264316540, 2024-06-04T05:23:57.968 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-06-04T05:23:57.968 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.968 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.968 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.968 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:57.968 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.968 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_bytes": { 2024-06-04T05:23:57.968 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-06-04T05:23:57.968 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-06-04T05:23:57.968 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-06-04T05:23:57.968 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-06-04T05:23:57.968 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-06-04T05:23:57.968 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-06-04T05:23:57.969 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-06-04T05:23:57.969 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 6053, 2024-06-04T05:23:57.969 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 261268613, 2024-06-04T05:23:57.969 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 6053, 2024-06-04T05:23:57.969 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 261268613, 2024-06-04T05:23:57.969 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-06-04T05:23:57.969 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.969 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.969 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.969 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:57.969 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.969 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_ops": { 2024-06-04T05:23:57.969 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-06-04T05:23:57.970 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 1024, 2024-06-04T05:23:57.970 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-06-04T05:23:57.970 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-06-04T05:23:57.970 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-06-04T05:23:57.970 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-06-04T05:23:57.970 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-06-04T05:23:57.970 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 6053, 2024-06-04T05:23:57.970 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 6053, 2024-06-04T05:23:57.970 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 6053, 2024-06-04T05:23:57.970 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 6053, 2024-06-04T05:23:57.970 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-06-04T05:23:57.970 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.970 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.970 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.971 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:57.971 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.971 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle": { 2024-06-04T05:23:57.971 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-06-04T05:23:57.971 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-06-04T05:23:57.971 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-06-04T05:23:57.971 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-06-04T05:23:57.971 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-06-04T05:23:57.971 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-06-04T05:23:57.971 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-06-04T05:23:57.971 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-06-04T05:23:57.971 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-06-04T05:23:57.971 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 0, 2024-06-04T05:23:57.971 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 0, 2024-06-04T05:23:57.972 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-06-04T05:23:57.972 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.972 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.972 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.972 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:57.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:23:57.972 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle-0x55b69bcbe6e0": { 2024-06-04T05:23:57.972 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-06-04T05:23:57.972 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-06-04T05:23:57.972 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-06-04T05:23:57.972 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-06-04T05:23:57.972 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-06-04T05:23:57.972 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-06-04T05:23:57.972 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-06-04T05:23:57.973 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-06-04T05:23:57.973 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-06-04T05:23:57.973 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 0, 2024-06-04T05:23:57.973 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 0, 2024-06-04T05:23:57.973 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-06-04T05:23:57.973 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:23:57.973 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:23:57.973 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:23:57.973 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:57.973 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:23:57.973 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:23:57.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:57 smithi098 ceph-mon[35569]: pgmap v844: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 14 op/s 2024-06-04T05:23:58.712 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-06-04T05:23:58.712 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-06-04T05:23:58.712 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-06-04T05:23:58.713 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-06-04T05:23:58.713 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-06-04T05:23:58.713 DEBUG:tasks.check_counter:Getting stats from j 2024-06-04T05:23:58.713 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph daemon mds.j perf dump 2024-06-04T05:23:58.955 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:23:59.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:23:59 smithi070 ceph-mon[32631]: pgmap v845: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:23:59.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:23:59 smithi112 ceph-mon[36453]: pgmap v845: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:23:59.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:23:59 smithi098 ceph-mon[35569]: pgmap v845: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:24:00.934 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-06-04T05:24:00.934 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-0": { 2024-06-04T05:24:00.934 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 0, 2024-06-04T05:24:00.935 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 2, 2024-06-04T05:24:00.935 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 0, 2024-06-04T05:24:00.935 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 0, 2024-06-04T05:24:00.935 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 4, 2024-06-04T05:24:00.935 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 0, 2024-06-04T05:24:00.935 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.001065013, 2024-06-04T05:24:00.935 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.000000000, 2024-06-04T05:24:00.935 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.000223007, 2024-06-04T05:24:00.935 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-04T05:24:00.935 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:00.935 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:24:00.935 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:24:00.935 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:24:00.935 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:24:00.936 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:00.936 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:24:00.936 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:24:00.936 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:24:00.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:24:00.936 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 0, 2024-06-04T05:24:00.936 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 0 2024-06-04T05:24:00.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:24:00.936 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-1": { 2024-06-04T05:24:00.936 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 7, 2024-06-04T05:24:00.936 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 380, 2024-06-04T05:24:00.936 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 3392, 2024-06-04T05:24:00.936 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 303040, 2024-06-04T05:24:00.937 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 6, 2024-06-04T05:24:00.937 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 3, 2024-06-04T05:24:00.937 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.023975948, 2024-06-04T05:24:00.937 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.018048567, 2024-06-04T05:24:00.937 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.002401780, 2024-06-04T05:24:00.937 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-04T05:24:00.937 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:00.937 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 378, 2024-06-04T05:24:00.937 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.020268420, 2024-06-04T05:24:00.937 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000053620 2024-06-04T05:24:00.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:24:00.937 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:00.937 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:24:00.937 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:24:00.938 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:24:00.938 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:24:00.938 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-06-04T05:24:00.938 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 303040 2024-06-04T05:24:00.938 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:24:00.938 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-2": { 2024-06-04T05:24:00.938 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 504, 2024-06-04T05:24:00.938 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 494, 2024-06-04T05:24:00.938 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 1642128, 2024-06-04T05:24:00.938 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 306048, 2024-06-04T05:24:00.938 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 2, 2024-06-04T05:24:00.938 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 1, 2024-06-04T05:24:00.938 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.069579785, 2024-06-04T05:24:00.939 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.026269547, 2024-06-04T05:24:00.939 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.039554456, 2024-06-04T05:24:00.939 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.009470775, 2024-06-04T05:24:00.939 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:00.939 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 493, 2024-06-04T05:24:00.939 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.016736507, 2024-06-04T05:24:00.939 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000033948 2024-06-04T05:24:00.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:24:00.939 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:00.939 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:24:00.939 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:24:00.939 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:24:00.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:24:00.940 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 1642128, 2024-06-04T05:24:00.940 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 306048 2024-06-04T05:24:00.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:24:00.940 INFO:teuthology.orchestra.run.smithi070.stdout: "cct": { 2024-06-04T05:24:00.940 INFO:teuthology.orchestra.run.smithi070.stdout: "total_workers": 0, 2024-06-04T05:24:00.940 INFO:teuthology.orchestra.run.smithi070.stdout: "unhealthy_workers": 0 2024-06-04T05:24:00.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:24:00.940 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool": { 2024-06-04T05:24:00.940 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_bytes": 0, 2024-06-04T05:24:00.940 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_items": 0, 2024-06-04T05:24:00.940 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_bytes": 0, 2024-06-04T05:24:00.940 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_items": 0, 2024-06-04T05:24:00.940 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_bytes": 0, 2024-06-04T05:24:00.940 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_items": 0, 2024-06-04T05:24:00.941 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-04T05:24:00.941 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_items": 0, 2024-06-04T05:24:00.941 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-04T05:24:00.941 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_items": 0, 2024-06-04T05:24:00.941 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_bytes": 0, 2024-06-04T05:24:00.941 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_items": 0, 2024-06-04T05:24:00.941 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-04T05:24:00.941 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_items": 0, 2024-06-04T05:24:00.941 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_bytes": 0, 2024-06-04T05:24:00.941 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_items": 0, 2024-06-04T05:24:00.941 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_bytes": 0, 2024-06-04T05:24:00.941 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_items": 0, 2024-06-04T05:24:00.941 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-04T05:24:00.941 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_items": 0, 2024-06-04T05:24:00.941 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-04T05:24:00.942 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_items": 0, 2024-06-04T05:24:00.942 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_bytes": 0, 2024-06-04T05:24:00.942 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_items": 0, 2024-06-04T05:24:00.942 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_bytes": 0, 2024-06-04T05:24:00.942 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_items": 0, 2024-06-04T05:24:00.942 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-04T05:24:00.942 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_items": 0, 2024-06-04T05:24:00.942 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_bytes": 0, 2024-06-04T05:24:00.942 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_items": 0, 2024-06-04T05:24:00.942 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_bytes": 0, 2024-06-04T05:24:00.942 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_items": 0, 2024-06-04T05:24:00.942 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_bytes": 0, 2024-06-04T05:24:00.943 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_items": 0, 2024-06-04T05:24:00.943 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_bytes": 0, 2024-06-04T05:24:00.943 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_items": 0, 2024-06-04T05:24:00.943 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_bytes": 148754, 2024-06-04T05:24:00.943 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_items": 31, 2024-06-04T05:24:00.943 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_bytes": 0, 2024-06-04T05:24:00.943 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_items": 0, 2024-06-04T05:24:00.943 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bytes": 0, 2024-06-04T05:24:00.943 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_items": 0, 2024-06-04T05:24:00.943 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_bytes": 0, 2024-06-04T05:24:00.943 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_items": 0, 2024-06-04T05:24:00.943 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_bytes": 0, 2024-06-04T05:24:00.943 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_items": 0, 2024-06-04T05:24:00.943 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_bytes": 0, 2024-06-04T05:24:00.943 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_items": 0, 2024-06-04T05:24:00.944 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_bytes": 0, 2024-06-04T05:24:00.944 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_items": 0, 2024-06-04T05:24:00.944 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_bytes": 0, 2024-06-04T05:24:00.944 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_items": 0, 2024-06-04T05:24:00.944 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_bytes": 928, 2024-06-04T05:24:00.944 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_items": 2, 2024-06-04T05:24:00.944 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_bytes": 0, 2024-06-04T05:24:00.944 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_items": 0, 2024-06-04T05:24:00.944 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_bytes": 0, 2024-06-04T05:24:00.944 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_items": 0 2024-06-04T05:24:00.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-06-04T05:24:00.944 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-04T05:24:00.944 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-06-04T05:24:00.945 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-06-04T05:24:00.945 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-06-04T05:24:00.945 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 507, 2024-06-04T05:24:00.945 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 1570884, 2024-06-04T05:24:00.945 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:00.945 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 507, 2024-06-04T05:24:00.945 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-06-04T05:24:00.945 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-06-04T05:24:00.945 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 507, 2024-06-04T05:24:00.945 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 1570884, 2024-06-04T05:24:00.945 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-06-04T05:24:00.945 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-06-04T05:24:00.946 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-06-04T05:24:00.946 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-06-04T05:24:00.946 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:24:00.946 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-06-04T05:24:00.946 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-06-04T05:24:01.687 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:01 smithi070 ceph-mon[32631]: pgmap v846: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:24:01.718 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-06-04T05:24:01.718 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-06-04T05:24:01.718 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-06-04T05:24:01.718 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-06-04T05:24:01.718 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-06-04T05:24:01.718 DEBUG:tasks.check_counter:Getting stats from b 2024-06-04T05:24:01.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph daemon mds.b perf dump 2024-06-04T05:24:01.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:01 smithi112 ceph-mon[36453]: pgmap v846: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:24:01.998 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:01 smithi098 ceph-mon[35569]: pgmap v846: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:24:02.056 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.b/config 2024-06-04T05:24:03.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:03 smithi098 ceph-mon[35569]: pgmap v847: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 32 op/s 2024-06-04T05:24:03.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:03 smithi070 ceph-mon[32631]: pgmap v847: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 32 op/s 2024-06-04T05:24:03.875 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-04T05:24:03.875 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-06-04T05:24:03.876 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 516637, 2024-06-04T05:24:03.876 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 558637, 2024-06-04T05:24:03.876 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 153154302, 2024-06-04T05:24:03.876 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 411832954, 2024-06-04T05:24:03.876 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 15, 2024-06-04T05:24:03.876 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 4, 2024-06-04T05:24:03.876 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 23.901748602, 2024-06-04T05:24:03.876 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 9.897548732, 2024-06-04T05:24:03.876 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 17.250823372, 2024-06-04T05:24:03.876 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.428488604, 2024-06-04T05:24:03.877 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:03.877 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 558635, 2024-06-04T05:24:03.877 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 8.152472372, 2024-06-04T05:24:03.877 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000014593 2024-06-04T05:24:03.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.877 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:03.877 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1029005, 2024-06-04T05:24:03.877 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.144171821, 2024-06-04T05:24:03.877 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000140 2024-06-04T05:24:03.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.877 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-06-04T05:24:03.877 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-06-04T05:24:03.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.878 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-06-04T05:24:03.878 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 22939, 2024-06-04T05:24:03.878 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 23288, 2024-06-04T05:24:03.878 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 6071890, 2024-06-04T05:24:03.878 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 143061624, 2024-06-04T05:24:03.878 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 20, 2024-06-04T05:24:03.878 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 6, 2024-06-04T05:24:03.878 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 1.705955856, 2024-06-04T05:24:03.878 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.527970979, 2024-06-04T05:24:03.878 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.371465046, 2024-06-04T05:24:03.878 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.707265379, 2024-06-04T05:24:03.878 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:03.878 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 23285, 2024-06-04T05:24:03.879 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.563623400, 2024-06-04T05:24:03.879 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000024205 2024-06-04T05:24:03.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.879 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:03.879 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2102, 2024-06-04T05:24:03.879 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000473879, 2024-06-04T05:24:03.879 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000225 2024-06-04T05:24:03.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.879 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 320, 2024-06-04T05:24:03.879 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 825952 2024-06-04T05:24:03.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.879 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-06-04T05:24:03.879 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 158972, 2024-06-04T05:24:03.879 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 131633, 2024-06-04T05:24:03.880 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 48458533, 2024-06-04T05:24:03.880 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 83513209, 2024-06-04T05:24:03.880 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 52, 2024-06-04T05:24:03.880 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 7, 2024-06-04T05:24:03.880 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 6.792786683, 2024-06-04T05:24:03.880 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 2.722319628, 2024-06-04T05:24:03.880 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 4.812960843, 2024-06-04T05:24:03.880 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.294726450, 2024-06-04T05:24:03.880 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:03.880 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 131633, 2024-06-04T05:24:03.880 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 2.236806616, 2024-06-04T05:24:03.880 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000016992 2024-06-04T05:24:03.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.881 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:03.881 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 262547, 2024-06-04T05:24:03.881 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.034774791, 2024-06-04T05:24:03.881 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000132 2024-06-04T05:24:03.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.881 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 2019808, 2024-06-04T05:24:03.881 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 342080 2024-06-04T05:24:03.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.881 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-06-04T05:24:03.881 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 1, 2024-06-04T05:24:03.881 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-06-04T05:24:03.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.881 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-MDSRank": { 2024-06-04T05:24:03.882 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-06-04T05:24:03.882 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-06-04T05:24:03.882 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 64586, 2024-06-04T05:24:03.882 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 29.426119555, 2024-06-04T05:24:03.882 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000455611 2024-06-04T05:24:03.882 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:03.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.882 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-PurgeQueue": { 2024-06-04T05:24:03.882 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-06-04T05:24:03.882 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-06-04T05:24:03.882 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 18143, 2024-06-04T05:24:03.882 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 1.030074094, 2024-06-04T05:24:03.882 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000056775 2024-06-04T05:24:03.883 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:03.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.883 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": { 2024-06-04T05:24:03.883 INFO:teuthology.orchestra.run.smithi098.stdout: "request": 428927, 2024-06-04T05:24:03.883 INFO:teuthology.orchestra.run.smithi098.stdout: "reply": 408210, 2024-06-04T05:24:03.883 INFO:teuthology.orchestra.run.smithi098.stdout: "reply_latency": { 2024-06-04T05:24:03.883 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 408210, 2024-06-04T05:24:03.883 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 409.371682305, 2024-06-04T05:24:03.883 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.001002845 2024-06-04T05:24:03.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.883 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_reply": 0, 2024-06-04T05:24:03.883 INFO:teuthology.orchestra.run.smithi098.stdout: "forward": 20717, 2024-06-04T05:24:03.883 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_complete": 0, 2024-06-04T05:24:03.883 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_keys": 0, 2024-06-04T05:24:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_commit": 20, 2024-06-04T05:24:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_split": 1, 2024-06-04T05:24:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_merge": 0, 2024-06-04T05:24:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes": 2774, 2024-06-04T05:24:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_top": 921, 2024-06-04T05:24:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_bottom": 1775, 2024-06-04T05:24:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pin_tail": 78, 2024-06-04T05:24:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pinned": 1332, 2024-06-04T05:24:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_expired": 949, 2024-06-04T05:24:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_with_caps": 1, 2024-06-04T05:24:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": 1, 2024-06-04T05:24:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "subtrees": 6, 2024-06-04T05:24:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse": 841103, 2024-06-04T05:24:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_hit": 800101, 2024-06-04T05:24:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_forward": 4660, 2024-06-04T05:24:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_discover": 92, 2024-06-04T05:24:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_dir_fetch": 0, 2024-06-04T05:24:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_remote_ino": 0, 2024-06-04T05:24:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_lock": 1, 2024-06-04T05:24:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "load_cent": 2000, 2024-06-04T05:24:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "q": 2, 2024-06-04T05:24:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "exported": 29, 2024-06-04T05:24:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "exported_inodes": 720, 2024-06-04T05:24:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "imported": 64, 2024-06-04T05:24:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "imported_inodes": 2360, 2024-06-04T05:24:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_dir_fetch": 0, 2024-06-04T05:24:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_backtrace_fetch": 1112, 2024-06-04T05:24:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_peer_discover": 0, 2024-06-04T05:24:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rfiles": 4, 2024-06-04T05:24:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rbytes": 234, 2024-06-04T05:24:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rsnaps": 7, 2024-06-04T05:24:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_fetch": 1112, 2024-06-04T05:24:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_set_tag": 0, 2024-06-04T05:24:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_repaired": 0, 2024-06-04T05:24:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_inotable_repaired": 0, 2024-06-04T05:24:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_inodes": 270, 2024-06-04T05:24:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_base_inodes": 0, 2024-06-04T05:24:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dirfrag_rstats": 270, 2024-06-04T05:24:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_file_inodes": 842, 2024-06-04T05:24:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_inode_file_caps": 20, 2024-06-04T05:24:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_revoke": 26293, 2024-06-04T05:24:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_grant": 26987, 2024-06-04T05:24:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_trunc": 0, 2024-06-04T05:24:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-04T05:24:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-04T05:24:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps": 37418, 2024-06-04T05:24:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps_dirty": 9416, 2024-06-04T05:24:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_cap_release": 793, 2024-06-04T05:24:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: "process_request_cap_release": 29170 2024-06-04T05:24:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache": { 2024-06-04T05:24:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update": 15, 2024-06-04T05:24:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update_receipt": 436, 2024-06-04T05:24:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_try_discover": 233, 2024-06-04T05:24:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_send_discover": 492, 2024-06-04T05:24:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_handle_discover": 78, 2024-06-04T05:24:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays": 30, 2024-06-04T05:24:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_delayed": 0, 2024-06-04T05:24:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_enqueuing": 0, 2024-06-04T05:24:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_created": 8052, 2024-06-04T05:24:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_enqueued": 8022, 2024-06-04T05:24:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_reintegrated": 0, 2024-06-04T05:24:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_migrated": 0, 2024-06-04T05:24:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_processing": 0, 2024-06-04T05:24:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_enqueued": 0, 2024-06-04T05:24:03.889 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_prioritized": 0, 2024-06-04T05:24:03.889 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_started": 0, 2024-06-04T05:24:03.889 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_completed": 0, 2024-06-04T05:24:03.889 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_path": 20, 2024-06-04T05:24:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_inode": 1547, 2024-06-04T05:24:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_enqueue_scrub": 0, 2024-06-04T05:24:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_exportdir": 30, 2024-06-04T05:24:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_flush": 0, 2024-06-04T05:24:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragmentdir": 1, 2024-06-04T05:24:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragstats": 0, 2024-06-04T05:24:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_inodestats": 0 2024-06-04T05:24:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log": { 2024-06-04T05:24:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "evlrg": 0, 2024-06-04T05:24:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "evadd": 72646, 2024-06-04T05:24:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "evex": 0, 2024-06-04T05:24:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "evtrm": 0, 2024-06-04T05:24:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "ev": 72646, 2024-06-04T05:24:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "evexg": 0, 2024-06-04T05:24:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "evexd": 0, 2024-06-04T05:24:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "segadd": 105, 2024-06-04T05:24:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "segex": 0, 2024-06-04T05:24:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "segtrm": 0, 2024-06-04T05:24:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "seg": 105, 2024-06-04T05:24:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "segmjr": 7, 2024-06-04T05:24:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "segexg": 0, 2024-06-04T05:24:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "segexd": 0, 2024-06-04T05:24:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "expos": 4194304, 2024-06-04T05:24:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "wrpos": 266999534, 2024-06-04T05:24:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "rdpos": 0, 2024-06-04T05:24:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "jlat": { 2024-06-04T05:24:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 32452, 2024-06-04T05:24:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 108.392348300, 2024-06-04T05:24:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.003340082 2024-06-04T05:24:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "replayed": 0 2024-06-04T05:24:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mem": { 2024-06-04T05:24:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "ino": 2340, 2024-06-04T05:24:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "ino+": 11001, 2024-06-04T05:24:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "ino-": 8661, 2024-06-04T05:24:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "dir": 107, 2024-06-04T05:24:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "dir+": 366, 2024-06-04T05:24:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "dir-": 259, 2024-06-04T05:24:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "dn": 2774, 2024-06-04T05:24:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "dn+": 14464, 2024-06-04T05:24:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "dn-": 11690, 2024-06-04T05:24:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "cap": 1, 2024-06-04T05:24:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "cap+": 8245, 2024-06-04T05:24:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "cap-": 8244, 2024-06-04T05:24:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "rss": 90952, 2024-06-04T05:24:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "heap": 223608 2024-06-04T05:24:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server": { 2024-06-04T05:24:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_client_request": 472847, 2024-06-04T05:24:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_server_request": 19013, 2024-06-04T05:24:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_request": 428927, 2024-06-04T05:24:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_session": 400, 2024-06-04T05:24:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_peer_request": 48946, 2024-06-04T05:24:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: "req_create_latency": { 2024-06-04T05:24:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 7964, 2024-06-04T05:24:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 98.167431102, 2024-06-04T05:24:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.012326397 2024-06-04T05:24:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getattr_latency": { 2024-06-04T05:24:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 364066, 2024-06-04T05:24:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 135.658655632, 2024-06-04T05:24:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000372621 2024-06-04T05:24:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getfilelock_latency": { 2024-06-04T05:24:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "req_link_latency": { 2024-06-04T05:24:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookup_latency": { 2024-06-04T05:24:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 10959, 2024-06-04T05:24:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 51.978994337, 2024-06-04T05:24:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.004743041 2024-06-04T05:24:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookuphash_latency": { 2024-06-04T05:24:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupino_latency": { 2024-06-04T05:24:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupname_latency": { 2024-06-04T05:24:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupparent_latency": { 2024-06-04T05:24:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupsnap_latency": { 2024-06-04T05:24:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lssnap_latency": { 2024-06-04T05:24:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mkdir_latency": { 2024-06-04T05:24:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mknod_latency": { 2024-06-04T05:24:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mksnap_latency": { 2024-06-04T05:24:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "req_open_latency": { 2024-06-04T05:24:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 3933, 2024-06-04T05:24:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 2.402884797, 2024-06-04T05:24:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000610954 2024-06-04T05:24:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "req_readdir_latency": { 2024-06-04T05:24:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 11428, 2024-06-04T05:24:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 20.786987910, 2024-06-04T05:24:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.001818952 2024-06-04T05:24:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rename_latency": { 2024-06-04T05:24:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1694, 2024-06-04T05:24:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 5.351593786, 2024-06-04T05:24:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.003159146 2024-06-04T05:24:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "req_renamesnap_latency": { 2024-06-04T05:24:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "req_snapdiff_latency": { 2024-06-04T05:24:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmdir_latency": { 2024-06-04T05:24:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 30, 2024-06-04T05:24:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.199880802, 2024-06-04T05:24:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.006662693 2024-06-04T05:24:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmsnap_latency": { 2024-06-04T05:24:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmxattr_latency": { 2024-06-04T05:24:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setattr_latency": { 2024-06-04T05:24:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2, 2024-06-04T05:24:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 31.131938012, 2024-06-04T05:24:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 15.565969006 2024-06-04T05:24:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setdirlayout_latency": { 2024-06-04T05:24:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setfilelock_latency": { 2024-06-04T05:24:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 112, 2024-06-04T05:24:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.046035453, 2024-06-04T05:24:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000411030 2024-06-04T05:24:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setlayout_latency": { 2024-06-04T05:24:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setxattr_latency": { 2024-06-04T05:24:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "req_symlink_latency": { 2024-06-04T05:24:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "req_unlink_latency": { 2024-06-04T05:24:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 8022, 2024-06-04T05:24:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 63.647280474, 2024-06-04T05:24:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.007934091 2024-06-04T05:24:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_revoke_eviction": 0, 2024-06-04T05:24:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_acquisition_throttle": 0, 2024-06-04T05:24:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getvxattr_latency": { 2024-06-04T05:24:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessions": { 2024-06-04T05:24:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "session_count": 1, 2024-06-04T05:24:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "session_add": 1, 2024-06-04T05:24:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "session_remove": 0, 2024-06-04T05:24:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_open": 1, 2024-06-04T05:24:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_stale": 0, 2024-06-04T05:24:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "total_load": 39274, 2024-06-04T05:24:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: "average_load": 39274, 2024-06-04T05:24:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: "avg_session_uptime": 1616, 2024-06-04T05:24:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: "mdthresh_evicted": 0 2024-06-04T05:24:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-06-04T05:24:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-06-04T05:24:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-06-04T05:24:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-06-04T05:24:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-06-04T05:24:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-06-04T05:24:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-06-04T05:24:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-04T05:24:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-06-04T05:24:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-04T05:24:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-06-04T05:24:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-06-04T05:24:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-06-04T05:24:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-04T05:24:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-06-04T05:24:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-06-04T05:24:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-06-04T05:24:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-06-04T05:24:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-06-04T05:24:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-04T05:24:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-06-04T05:24:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-04T05:24:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-06-04T05:24:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-06-04T05:24:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-06-04T05:24:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-06-04T05:24:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-06-04T05:24:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-04T05:24:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-06-04T05:24:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-06-04T05:24:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-06-04T05:24:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-06-04T05:24:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-06-04T05:24:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-06-04T05:24:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-06-04T05:24:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-06-04T05:24:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-06-04T05:24:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 561089, 2024-06-04T05:24:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 175, 2024-06-04T05:24:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-06-04T05:24:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-06-04T05:24:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-06-04T05:24:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-06-04T05:24:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-06-04T05:24:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-06-04T05:24:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-06-04T05:24:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-06-04T05:24:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 8224, 2024-06-04T05:24:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 171, 2024-06-04T05:24:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-06-04T05:24:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-06-04T05:24:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-06-04T05:24:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-06-04T05:24:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 7365854, 2024-06-04T05:24:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 26411, 2024-06-04T05:24:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-06-04T05:24:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-06-04T05:24:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-06-04T05:24:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-06-04T05:24:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter": { 2024-06-04T05:24:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "op_active": 0, 2024-06-04T05:24:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "op_laggy": 0, 2024-06-04T05:24:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send": 46573, 2024-06-04T05:24:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send_bytes": 264238219, 2024-06-04T05:24:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "op_resend": 30, 2024-06-04T05:24:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "op_reply": 46573, 2024-06-04T05:24:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "op_latency": { 2024-06-04T05:24:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 46573, 2024-06-04T05:24:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 136.871314475, 2024-06-04T05:24:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.002938855 2024-06-04T05:24:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: "op_inflight": 0, 2024-06-04T05:24:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: "oplen_avg": { 2024-06-04T05:24:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 46573, 2024-06-04T05:24:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 48325 2024-06-04T05:24:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: "op": 46573, 2024-06-04T05:24:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: "op_r": 2599, 2024-06-04T05:24:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: "op_w": 43974, 2024-06-04T05:24:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: "op_rmw": 0, 2024-06-04T05:24:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: "op_pg": 0, 2024-06-04T05:24:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_stat": 0, 2024-06-04T05:24:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_create": 10, 2024-06-04T05:24:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_read": 1487, 2024-06-04T05:24:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_write": 33939, 2024-06-04T05:24:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writefull": 881, 2024-06-04T05:24:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writesame": 0, 2024-06-04T05:24:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_append": 0, 2024-06-04T05:24:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_zero": 0, 2024-06-04T05:24:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_truncate": 1, 2024-06-04T05:24:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_delete": 8096, 2024-06-04T05:24:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_mapext": 0, 2024-06-04T05:24:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_sparse_read": 0, 2024-06-04T05:24:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_clonerange": 0, 2024-06-04T05:24:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_getxattr": 1112, 2024-06-04T05:24:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_setxattr": 20, 2024-06-04T05:24:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_cmpxattr": 0, 2024-06-04T05:24:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_rmxattr": 0, 2024-06-04T05:24:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_resetxattrs": 0, 2024-06-04T05:24:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_call": 0, 2024-06-04T05:24:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_watch": 0, 2024-06-04T05:24:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_notify": 0, 2024-06-04T05:24:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_src_cmpxattr": 0, 2024-06-04T05:24:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls": 0, 2024-06-04T05:24:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls_filter": 0, 2024-06-04T05:24:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_other": 0, 2024-06-04T05:24:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_active": 0, 2024-06-04T05:24:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_send": 0, 2024-06-04T05:24:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_resend": 0, 2024-06-04T05:24:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_ping": 0, 2024-06-04T05:24:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_active": 0, 2024-06-04T05:24:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_send": 0, 2024-06-04T05:24:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_resend": 0, 2024-06-04T05:24:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_active": 0, 2024-06-04T05:24:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_send": 0, 2024-06-04T05:24:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_resend": 0, 2024-06-04T05:24:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_active": 0, 2024-06-04T05:24:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_send": 0, 2024-06-04T05:24:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_resend": 0, 2024-06-04T05:24:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: "command_active": 0, 2024-06-04T05:24:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: "command_send": 0, 2024-06-04T05:24:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: "command_resend": 0, 2024-06-04T05:24:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: "map_epoch": 120, 2024-06-04T05:24:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: "map_full": 0, 2024-06-04T05:24:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: "map_inc": 41, 2024-06-04T05:24:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_sessions": 12, 2024-06-04T05:24:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_open": 12, 2024-06-04T05:24:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_close": 0, 2024-06-04T05:24:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_laggy": 0, 2024-06-04T05:24:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_wr": 1896, 2024-06-04T05:24:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_rd": 0, 2024-06-04T05:24:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_del": 883 2024-06-04T05:24:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: "oft": { 2024-06-04T05:24:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_objs": 1, 2024-06-04T05:24:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_kv_pairs": 0, 2024-06-04T05:24:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_updates": 0, 2024-06-04T05:24:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_removes": 0 2024-06-04T05:24:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: "osd-slow-ops": { 2024-06-04T05:24:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_ops_count": 1810 2024-06-04T05:24:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: "purge_queue": { 2024-06-04T05:24:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops": 0, 2024-06-04T05:24:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops_high_water": 6, 2024-06-04T05:24:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing": 0, 2024-06-04T05:24:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_high_water": 6, 2024-06-04T05:24:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed_ops": 8022, 2024-06-04T05:24:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed": 8022, 2024-06-04T05:24:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_item_in_journal": 0 2024-06-04T05:24:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-04T05:24:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-06-04T05:24:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 698544, 2024-06-04T05:24:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 144779620, 2024-06-04T05:24:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 698544, 2024-06-04T05:24:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-06-04T05:24:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-06-04T05:24:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 698544, 2024-06-04T05:24:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 144779620, 2024-06-04T05:24:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_bytes": { 2024-06-04T05:24:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-06-04T05:24:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-06-04T05:24:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-06-04T05:24:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 46573, 2024-06-04T05:24:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 265053329, 2024-06-04T05:24:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 46573, 2024-06-04T05:24:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 265053329, 2024-06-04T05:24:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_ops": { 2024-06-04T05:24:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 1024, 2024-06-04T05:24:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-06-04T05:24:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-06-04T05:24:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 46573, 2024-06-04T05:24:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 46573, 2024-06-04T05:24:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 46573, 2024-06-04T05:24:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 46573, 2024-06-04T05:24:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle": { 2024-06-04T05:24:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-06-04T05:24:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 8022, 2024-06-04T05:24:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 815110, 2024-06-04T05:24:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 8022, 2024-06-04T05:24:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-06-04T05:24:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-06-04T05:24:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 1487, 2024-06-04T05:24:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 815110, 2024-06-04T05:24:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle-0x55ff6c3da500": { 2024-06-04T05:24:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-06-04T05:24:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 72646, 2024-06-04T05:24:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 262805230, 2024-06-04T05:24:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 72646, 2024-06-04T05:24:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-06-04T05:24:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-06-04T05:24:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 32452, 2024-06-04T05:24:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 262805230, 2024-06-04T05:24:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:03.934 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-04T05:24:03.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:03 smithi112 ceph-mon[36453]: pgmap v847: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 32 op/s 2024-06-04T05:24:04.667 INFO:tasks.check_counter:Daemon mds.b mds.exported=29 2024-06-04T05:24:04.668 INFO:tasks.check_counter:Daemon mds.b mds.imported=64 2024-06-04T05:24:04.668 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=7 2024-06-04T05:24:04.668 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=0 2024-06-04T05:24:04.668 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=0 2024-06-04T05:24:04.668 DEBUG:tasks.check_counter:Getting stats from e 2024-06-04T05:24:04.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph daemon mds.e perf dump 2024-06-04T05:24:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.b/config 2024-06-04T05:24:05.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:05 smithi070 ceph-mon[32631]: pgmap v848: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-06-04T05:24:05.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:05 smithi112 ceph-mon[36453]: pgmap v848: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-06-04T05:24:05.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:05 smithi098 ceph-mon[35569]: pgmap v848: 97 pgs: 97 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-06-04T05:24:06.689 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-04T05:24:06.689 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-06-04T05:24:06.689 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 3365, 2024-06-04T05:24:06.690 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 3308, 2024-06-04T05:24:06.690 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 56770061, 2024-06-04T05:24:06.690 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 1194132, 2024-06-04T05:24:06.690 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 7, 2024-06-04T05:24:06.690 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-06-04T05:24:06.690 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.329552079, 2024-06-04T05:24:06.690 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.112479745, 2024-06-04T05:24:06.690 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.128289759, 2024-06-04T05:24:06.690 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.075460627, 2024-06-04T05:24:06.690 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:06.690 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 3308, 2024-06-04T05:24:06.691 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.118038041, 2024-06-04T05:24:06.691 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000035682 2024-06-04T05:24:06.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.691 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:06.691 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.691 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.691 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.691 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1991312, 2024-06-04T05:24:06.691 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 314544 2024-06-04T05:24:06.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.692 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-06-04T05:24:06.692 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 1153, 2024-06-04T05:24:06.692 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 1529, 2024-06-04T05:24:06.692 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 84176784, 2024-06-04T05:24:06.692 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 1193360, 2024-06-04T05:24:06.692 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 9, 2024-06-04T05:24:06.692 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 7, 2024-06-04T05:24:06.692 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.134412056, 2024-06-04T05:24:06.692 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.035802366, 2024-06-04T05:24:06.692 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.061835662, 2024-06-04T05:24:06.692 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.026777423, 2024-06-04T05:24:06.692 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:06.692 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1526, 2024-06-04T05:24:06.693 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.045269643, 2024-06-04T05:24:06.693 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000029665 2024-06-04T05:24:06.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.693 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:06.693 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.693 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.693 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.693 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 320, 2024-06-04T05:24:06.693 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 827280 2024-06-04T05:24:06.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.693 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-06-04T05:24:06.693 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 2469, 2024-06-04T05:24:06.694 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2471, 2024-06-04T05:24:06.694 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 155818211, 2024-06-04T05:24:06.694 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 785149, 2024-06-04T05:24:06.694 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 7, 2024-06-04T05:24:06.694 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 5, 2024-06-04T05:24:06.694 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.232448028, 2024-06-04T05:24:06.694 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.039687591, 2024-06-04T05:24:06.694 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.127927037, 2024-06-04T05:24:06.694 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.052921018, 2024-06-04T05:24:06.694 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:06.694 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2469, 2024-06-04T05:24:06.694 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.053385436, 2024-06-04T05:24:06.695 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000021622 2024-06-04T05:24:06.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.695 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:06.695 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.695 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.695 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.695 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-06-04T05:24:06.695 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-06-04T05:24:06.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.695 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-06-04T05:24:06.695 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 1, 2024-06-04T05:24:06.695 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-06-04T05:24:06.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.695 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-MDSRank": { 2024-06-04T05:24:06.696 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-06-04T05:24:06.696 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-06-04T05:24:06.696 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 10865, 2024-06-04T05:24:06.696 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.216328012, 2024-06-04T05:24:06.696 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000019910 2024-06-04T05:24:06.696 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:06.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.696 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-PurgeQueue": { 2024-06-04T05:24:06.696 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-06-04T05:24:06.696 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-06-04T05:24:06.696 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.696 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.696 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.697 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:06.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.697 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": { 2024-06-04T05:24:06.697 INFO:teuthology.orchestra.run.smithi098.stdout: "request": 0, 2024-06-04T05:24:06.697 INFO:teuthology.orchestra.run.smithi098.stdout: "reply": 0, 2024-06-04T05:24:06.697 INFO:teuthology.orchestra.run.smithi098.stdout: "reply_latency": { 2024-06-04T05:24:06.697 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.697 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.697 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.697 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_reply": 0, 2024-06-04T05:24:06.697 INFO:teuthology.orchestra.run.smithi098.stdout: "forward": 0, 2024-06-04T05:24:06.697 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_complete": 0, 2024-06-04T05:24:06.697 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_keys": 0, 2024-06-04T05:24:06.697 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_commit": 0, 2024-06-04T05:24:06.698 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_split": 0, 2024-06-04T05:24:06.698 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_merge": 0, 2024-06-04T05:24:06.698 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes": 11566, 2024-06-04T05:24:06.698 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_top": 0, 2024-06-04T05:24:06.698 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_bottom": 11566, 2024-06-04T05:24:06.698 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pin_tail": 0, 2024-06-04T05:24:06.698 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pinned": 11566, 2024-06-04T05:24:06.698 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_expired": 0, 2024-06-04T05:24:06.698 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_with_caps": 0, 2024-06-04T05:24:06.698 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": 0, 2024-06-04T05:24:06.698 INFO:teuthology.orchestra.run.smithi098.stdout: "subtrees": 1, 2024-06-04T05:24:06.698 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse": 0, 2024-06-04T05:24:06.698 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_hit": 0, 2024-06-04T05:24:06.698 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_forward": 0, 2024-06-04T05:24:06.699 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_discover": 0, 2024-06-04T05:24:06.699 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_dir_fetch": 0, 2024-06-04T05:24:06.699 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_remote_ino": 0, 2024-06-04T05:24:06.699 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_lock": 0, 2024-06-04T05:24:06.699 INFO:teuthology.orchestra.run.smithi098.stdout: "load_cent": 0, 2024-06-04T05:24:06.699 INFO:teuthology.orchestra.run.smithi098.stdout: "q": 0, 2024-06-04T05:24:06.699 INFO:teuthology.orchestra.run.smithi098.stdout: "exported": 0, 2024-06-04T05:24:06.699 INFO:teuthology.orchestra.run.smithi098.stdout: "exported_inodes": 0, 2024-06-04T05:24:06.699 INFO:teuthology.orchestra.run.smithi098.stdout: "imported": 0, 2024-06-04T05:24:06.699 INFO:teuthology.orchestra.run.smithi098.stdout: "imported_inodes": 0, 2024-06-04T05:24:06.699 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_dir_fetch": 0, 2024-06-04T05:24:06.699 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_backtrace_fetch": 0, 2024-06-04T05:24:06.699 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_peer_discover": 0, 2024-06-04T05:24:06.699 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rfiles": 0, 2024-06-04T05:24:06.700 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rbytes": 0, 2024-06-04T05:24:06.700 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rsnaps": 0, 2024-06-04T05:24:06.700 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_fetch": 0, 2024-06-04T05:24:06.700 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_set_tag": 0, 2024-06-04T05:24:06.700 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_repaired": 0, 2024-06-04T05:24:06.700 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_inotable_repaired": 0, 2024-06-04T05:24:06.700 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_inodes": 0, 2024-06-04T05:24:06.700 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_base_inodes": 0, 2024-06-04T05:24:06.700 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dirfrag_rstats": 0, 2024-06-04T05:24:06.700 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_file_inodes": 0, 2024-06-04T05:24:06.700 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_inode_file_caps": 0, 2024-06-04T05:24:06.700 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_revoke": 0, 2024-06-04T05:24:06.700 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_grant": 0, 2024-06-04T05:24:06.701 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_trunc": 0, 2024-06-04T05:24:06.701 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-04T05:24:06.701 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-04T05:24:06.701 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps": 0, 2024-06-04T05:24:06.701 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps_dirty": 0, 2024-06-04T05:24:06.701 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_cap_release": 0, 2024-06-04T05:24:06.701 INFO:teuthology.orchestra.run.smithi098.stdout: "process_request_cap_release": 0 2024-06-04T05:24:06.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.701 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache": { 2024-06-04T05:24:06.701 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update": 0, 2024-06-04T05:24:06.701 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update_receipt": 0, 2024-06-04T05:24:06.701 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_try_discover": 0, 2024-06-04T05:24:06.701 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_send_discover": 0, 2024-06-04T05:24:06.701 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_handle_discover": 0, 2024-06-04T05:24:06.701 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays": 0, 2024-06-04T05:24:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_delayed": 0, 2024-06-04T05:24:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_enqueuing": 0, 2024-06-04T05:24:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_created": 0, 2024-06-04T05:24:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_enqueued": 0, 2024-06-04T05:24:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_reintegrated": 0, 2024-06-04T05:24:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_migrated": 0, 2024-06-04T05:24:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_processing": 0, 2024-06-04T05:24:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_enqueued": 0, 2024-06-04T05:24:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_prioritized": 0, 2024-06-04T05:24:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_started": 0, 2024-06-04T05:24:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_completed": 0, 2024-06-04T05:24:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_path": 0, 2024-06-04T05:24:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_inode": 0, 2024-06-04T05:24:06.703 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_enqueue_scrub": 0, 2024-06-04T05:24:06.703 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_exportdir": 0, 2024-06-04T05:24:06.703 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_flush": 0, 2024-06-04T05:24:06.703 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragmentdir": 0, 2024-06-04T05:24:06.703 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragstats": 0, 2024-06-04T05:24:06.703 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_inodestats": 0 2024-06-04T05:24:06.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.703 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log": { 2024-06-04T05:24:06.703 INFO:teuthology.orchestra.run.smithi098.stdout: "evlrg": 0, 2024-06-04T05:24:06.703 INFO:teuthology.orchestra.run.smithi098.stdout: "evadd": 0, 2024-06-04T05:24:06.703 INFO:teuthology.orchestra.run.smithi098.stdout: "evex": 0, 2024-06-04T05:24:06.703 INFO:teuthology.orchestra.run.smithi098.stdout: "evtrm": 0, 2024-06-04T05:24:06.703 INFO:teuthology.orchestra.run.smithi098.stdout: "ev": 86003, 2024-06-04T05:24:06.703 INFO:teuthology.orchestra.run.smithi098.stdout: "evexg": 0, 2024-06-04T05:24:06.703 INFO:teuthology.orchestra.run.smithi098.stdout: "evexd": 0, 2024-06-04T05:24:06.703 INFO:teuthology.orchestra.run.smithi098.stdout: "segadd": 0, 2024-06-04T05:24:06.704 INFO:teuthology.orchestra.run.smithi098.stdout: "segex": 0, 2024-06-04T05:24:06.704 INFO:teuthology.orchestra.run.smithi098.stdout: "segtrm": 0, 2024-06-04T05:24:06.704 INFO:teuthology.orchestra.run.smithi098.stdout: "seg": 118, 2024-06-04T05:24:06.704 INFO:teuthology.orchestra.run.smithi098.stdout: "segmjr": 8, 2024-06-04T05:24:06.704 INFO:teuthology.orchestra.run.smithi098.stdout: "segexg": 0, 2024-06-04T05:24:06.704 INFO:teuthology.orchestra.run.smithi098.stdout: "segexd": 0, 2024-06-04T05:24:06.704 INFO:teuthology.orchestra.run.smithi098.stdout: "expos": 4194304, 2024-06-04T05:24:06.704 INFO:teuthology.orchestra.run.smithi098.stdout: "wrpos": 297196869, 2024-06-04T05:24:06.704 INFO:teuthology.orchestra.run.smithi098.stdout: "rdpos": 297183429, 2024-06-04T05:24:06.704 INFO:teuthology.orchestra.run.smithi098.stdout: "jlat": { 2024-06-04T05:24:06.704 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.704 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.704 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.704 INFO:teuthology.orchestra.run.smithi098.stdout: "replayed": 86003 2024-06-04T05:24:06.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.705 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mem": { 2024-06-04T05:24:06.705 INFO:teuthology.orchestra.run.smithi098.stdout: "ino": 1593, 2024-06-04T05:24:06.705 INFO:teuthology.orchestra.run.smithi098.stdout: "ino+": 12159, 2024-06-04T05:24:06.705 INFO:teuthology.orchestra.run.smithi098.stdout: "ino-": 10566, 2024-06-04T05:24:06.705 INFO:teuthology.orchestra.run.smithi098.stdout: "dir": 59, 2024-06-04T05:24:06.705 INFO:teuthology.orchestra.run.smithi098.stdout: "dir+": 120, 2024-06-04T05:24:06.705 INFO:teuthology.orchestra.run.smithi098.stdout: "dir-": 61, 2024-06-04T05:24:06.705 INFO:teuthology.orchestra.run.smithi098.stdout: "dn": 11566, 2024-06-04T05:24:06.705 INFO:teuthology.orchestra.run.smithi098.stdout: "dn+": 12557, 2024-06-04T05:24:06.705 INFO:teuthology.orchestra.run.smithi098.stdout: "dn-": 991, 2024-06-04T05:24:06.705 INFO:teuthology.orchestra.run.smithi098.stdout: "cap": 0, 2024-06-04T05:24:06.705 INFO:teuthology.orchestra.run.smithi098.stdout: "cap+": 0, 2024-06-04T05:24:06.705 INFO:teuthology.orchestra.run.smithi098.stdout: "cap-": 0, 2024-06-04T05:24:06.705 INFO:teuthology.orchestra.run.smithi098.stdout: "rss": 70412, 2024-06-04T05:24:06.705 INFO:teuthology.orchestra.run.smithi098.stdout: "heap": 215416 2024-06-04T05:24:06.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.706 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server": { 2024-06-04T05:24:06.706 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_client_request": 0, 2024-06-04T05:24:06.706 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_server_request": 0, 2024-06-04T05:24:06.706 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_request": 0, 2024-06-04T05:24:06.706 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_session": 0, 2024-06-04T05:24:06.706 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_peer_request": 0, 2024-06-04T05:24:06.706 INFO:teuthology.orchestra.run.smithi098.stdout: "req_create_latency": { 2024-06-04T05:24:06.706 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.706 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.706 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.706 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getattr_latency": { 2024-06-04T05:24:06.706 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.707 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.707 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.707 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getfilelock_latency": { 2024-06-04T05:24:06.707 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.707 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.707 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.707 INFO:teuthology.orchestra.run.smithi098.stdout: "req_link_latency": { 2024-06-04T05:24:06.707 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.707 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.707 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.708 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookup_latency": { 2024-06-04T05:24:06.708 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.708 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.708 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.708 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookuphash_latency": { 2024-06-04T05:24:06.708 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.708 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.708 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.708 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupino_latency": { 2024-06-04T05:24:06.708 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.709 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.709 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.709 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupname_latency": { 2024-06-04T05:24:06.709 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.709 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.709 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.709 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupparent_latency": { 2024-06-04T05:24:06.709 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.709 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.709 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.710 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupsnap_latency": { 2024-06-04T05:24:06.710 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.710 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.710 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.710 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lssnap_latency": { 2024-06-04T05:24:06.710 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.710 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.710 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.711 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mkdir_latency": { 2024-06-04T05:24:06.711 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.711 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.711 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.711 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mknod_latency": { 2024-06-04T05:24:06.711 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.711 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.711 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.711 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mksnap_latency": { 2024-06-04T05:24:06.712 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.712 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.712 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.712 INFO:teuthology.orchestra.run.smithi098.stdout: "req_open_latency": { 2024-06-04T05:24:06.712 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.712 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.712 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.712 INFO:teuthology.orchestra.run.smithi098.stdout: "req_readdir_latency": { 2024-06-04T05:24:06.712 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.712 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.712 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.713 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rename_latency": { 2024-06-04T05:24:06.713 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.713 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.713 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.713 INFO:teuthology.orchestra.run.smithi098.stdout: "req_renamesnap_latency": { 2024-06-04T05:24:06.713 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.713 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.713 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.713 INFO:teuthology.orchestra.run.smithi098.stdout: "req_snapdiff_latency": { 2024-06-04T05:24:06.713 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.713 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.713 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.714 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmdir_latency": { 2024-06-04T05:24:06.714 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.714 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.714 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.714 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmsnap_latency": { 2024-06-04T05:24:06.714 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.714 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.714 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.714 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmxattr_latency": { 2024-06-04T05:24:06.714 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.715 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.715 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.715 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setattr_latency": { 2024-06-04T05:24:06.715 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.715 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.715 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.715 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setdirlayout_latency": { 2024-06-04T05:24:06.715 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.715 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.715 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.715 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setfilelock_latency": { 2024-06-04T05:24:06.716 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.716 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.716 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.716 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setlayout_latency": { 2024-06-04T05:24:06.716 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.716 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.716 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.716 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setxattr_latency": { 2024-06-04T05:24:06.716 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.716 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.716 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.717 INFO:teuthology.orchestra.run.smithi098.stdout: "req_symlink_latency": { 2024-06-04T05:24:06.717 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.717 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.717 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.717 INFO:teuthology.orchestra.run.smithi098.stdout: "req_unlink_latency": { 2024-06-04T05:24:06.717 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.717 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.717 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.717 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_revoke_eviction": 0, 2024-06-04T05:24:06.717 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_acquisition_throttle": 0, 2024-06-04T05:24:06.718 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getvxattr_latency": { 2024-06-04T05:24:06.718 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.718 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.718 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.718 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:06.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.718 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessions": { 2024-06-04T05:24:06.718 INFO:teuthology.orchestra.run.smithi098.stdout: "session_count": 2, 2024-06-04T05:24:06.718 INFO:teuthology.orchestra.run.smithi098.stdout: "session_add": 2, 2024-06-04T05:24:06.718 INFO:teuthology.orchestra.run.smithi098.stdout: "session_remove": 0, 2024-06-04T05:24:06.718 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_open": 2, 2024-06-04T05:24:06.719 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_stale": 0, 2024-06-04T05:24:06.719 INFO:teuthology.orchestra.run.smithi098.stdout: "total_load": 0, 2024-06-04T05:24:06.719 INFO:teuthology.orchestra.run.smithi098.stdout: "average_load": 0, 2024-06-04T05:24:06.719 INFO:teuthology.orchestra.run.smithi098.stdout: "avg_session_uptime": 3711, 2024-06-04T05:24:06.719 INFO:teuthology.orchestra.run.smithi098.stdout: "mdthresh_evicted": 0 2024-06-04T05:24:06.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.719 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-06-04T05:24:06.719 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-06-04T05:24:06.719 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-06-04T05:24:06.719 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-06-04T05:24:06.719 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-06-04T05:24:06.719 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-06-04T05:24:06.719 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-06-04T05:24:06.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-04T05:24:06.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-06-04T05:24:06.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-04T05:24:06.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-06-04T05:24:06.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-06-04T05:24:06.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-06-04T05:24:06.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-04T05:24:06.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-06-04T05:24:06.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-06-04T05:24:06.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-06-04T05:24:06.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-06-04T05:24:06.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-06-04T05:24:06.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-04T05:24:06.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-06-04T05:24:06.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-04T05:24:06.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-06-04T05:24:06.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-06-04T05:24:06.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-06-04T05:24:06.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-06-04T05:24:06.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-06-04T05:24:06.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-04T05:24:06.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-06-04T05:24:06.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-06-04T05:24:06.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-06-04T05:24:06.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-06-04T05:24:06.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-06-04T05:24:06.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-06-04T05:24:06.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-06-04T05:24:06.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-06-04T05:24:06.722 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-06-04T05:24:06.722 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 242568, 2024-06-04T05:24:06.722 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 86, 2024-06-04T05:24:06.722 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-06-04T05:24:06.722 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-06-04T05:24:06.722 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-06-04T05:24:06.722 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-06-04T05:24:06.722 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-06-04T05:24:06.722 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-06-04T05:24:06.722 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-06-04T05:24:06.722 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-06-04T05:24:06.722 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 8224, 2024-06-04T05:24:06.722 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 171, 2024-06-04T05:24:06.722 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-06-04T05:24:06.723 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-06-04T05:24:06.723 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-06-04T05:24:06.723 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-06-04T05:24:06.723 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 11874846, 2024-06-04T05:24:06.723 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 54054, 2024-06-04T05:24:06.723 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-06-04T05:24:06.723 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-06-04T05:24:06.723 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-06-04T05:24:06.723 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-06-04T05:24:06.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.723 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter": { 2024-06-04T05:24:06.723 INFO:teuthology.orchestra.run.smithi098.stdout: "op_active": 0, 2024-06-04T05:24:06.723 INFO:teuthology.orchestra.run.smithi098.stdout: "op_laggy": 0, 2024-06-04T05:24:06.724 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send": 6386, 2024-06-04T05:24:06.724 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send_bytes": 21, 2024-06-04T05:24:06.724 INFO:teuthology.orchestra.run.smithi098.stdout: "op_resend": 3, 2024-06-04T05:24:06.724 INFO:teuthology.orchestra.run.smithi098.stdout: "op_reply": 6386, 2024-06-04T05:24:06.724 INFO:teuthology.orchestra.run.smithi098.stdout: "op_latency": { 2024-06-04T05:24:06.724 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 6386, 2024-06-04T05:24:06.724 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 3.796952654, 2024-06-04T05:24:06.724 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000594574 2024-06-04T05:24:06.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.724 INFO:teuthology.orchestra.run.smithi098.stdout: "op_inflight": 0, 2024-06-04T05:24:06.724 INFO:teuthology.orchestra.run.smithi098.stdout: "oplen_avg": { 2024-06-04T05:24:06.724 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 6386, 2024-06-04T05:24:06.724 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 6387 2024-06-04T05:24:06.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.725 INFO:teuthology.orchestra.run.smithi098.stdout: "op": 6386, 2024-06-04T05:24:06.725 INFO:teuthology.orchestra.run.smithi098.stdout: "op_r": 6386, 2024-06-04T05:24:06.725 INFO:teuthology.orchestra.run.smithi098.stdout: "op_w": 0, 2024-06-04T05:24:06.725 INFO:teuthology.orchestra.run.smithi098.stdout: "op_rmw": 0, 2024-06-04T05:24:06.725 INFO:teuthology.orchestra.run.smithi098.stdout: "op_pg": 0, 2024-06-04T05:24:06.725 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_stat": 3503, 2024-06-04T05:24:06.725 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_create": 0, 2024-06-04T05:24:06.725 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_read": 2881, 2024-06-04T05:24:06.725 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_write": 0, 2024-06-04T05:24:06.725 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writefull": 0, 2024-06-04T05:24:06.725 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writesame": 0, 2024-06-04T05:24:06.725 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_append": 0, 2024-06-04T05:24:06.725 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_zero": 0, 2024-06-04T05:24:06.725 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_truncate": 0, 2024-06-04T05:24:06.726 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_delete": 0, 2024-06-04T05:24:06.726 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_mapext": 0, 2024-06-04T05:24:06.726 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_sparse_read": 0, 2024-06-04T05:24:06.726 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_clonerange": 0, 2024-06-04T05:24:06.726 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_getxattr": 1, 2024-06-04T05:24:06.726 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_setxattr": 0, 2024-06-04T05:24:06.726 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_cmpxattr": 0, 2024-06-04T05:24:06.726 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_rmxattr": 0, 2024-06-04T05:24:06.726 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_resetxattrs": 0, 2024-06-04T05:24:06.726 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_call": 0, 2024-06-04T05:24:06.726 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_watch": 0, 2024-06-04T05:24:06.726 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_notify": 0, 2024-06-04T05:24:06.726 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_src_cmpxattr": 0, 2024-06-04T05:24:06.727 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls": 0, 2024-06-04T05:24:06.727 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls_filter": 0, 2024-06-04T05:24:06.727 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_other": 0, 2024-06-04T05:24:06.727 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_active": 0, 2024-06-04T05:24:06.727 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_send": 0, 2024-06-04T05:24:06.727 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_resend": 0, 2024-06-04T05:24:06.727 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_ping": 0, 2024-06-04T05:24:06.727 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_active": 0, 2024-06-04T05:24:06.727 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_send": 0, 2024-06-04T05:24:06.727 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_resend": 0, 2024-06-04T05:24:06.727 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_active": 0, 2024-06-04T05:24:06.727 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_send": 0, 2024-06-04T05:24:06.727 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_resend": 0, 2024-06-04T05:24:06.727 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_active": 0, 2024-06-04T05:24:06.728 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_send": 0, 2024-06-04T05:24:06.728 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_resend": 0, 2024-06-04T05:24:06.728 INFO:teuthology.orchestra.run.smithi098.stdout: "command_active": 0, 2024-06-04T05:24:06.728 INFO:teuthology.orchestra.run.smithi098.stdout: "command_send": 0, 2024-06-04T05:24:06.728 INFO:teuthology.orchestra.run.smithi098.stdout: "command_resend": 0, 2024-06-04T05:24:06.728 INFO:teuthology.orchestra.run.smithi098.stdout: "map_epoch": 120, 2024-06-04T05:24:06.728 INFO:teuthology.orchestra.run.smithi098.stdout: "map_full": 0, 2024-06-04T05:24:06.728 INFO:teuthology.orchestra.run.smithi098.stdout: "map_inc": 41, 2024-06-04T05:24:06.728 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_sessions": 11, 2024-06-04T05:24:06.728 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_open": 12, 2024-06-04T05:24:06.728 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_close": 1, 2024-06-04T05:24:06.728 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_laggy": 0, 2024-06-04T05:24:06.728 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_wr": 0, 2024-06-04T05:24:06.729 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_rd": 2, 2024-06-04T05:24:06.729 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_del": 0 2024-06-04T05:24:06.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.729 INFO:teuthology.orchestra.run.smithi098.stdout: "oft": { 2024-06-04T05:24:06.729 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_objs": 0, 2024-06-04T05:24:06.729 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_kv_pairs": 0, 2024-06-04T05:24:06.729 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_updates": 0, 2024-06-04T05:24:06.729 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_removes": 0 2024-06-04T05:24:06.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.729 INFO:teuthology.orchestra.run.smithi098.stdout: "osd-slow-ops": { 2024-06-04T05:24:06.729 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_ops_count": 0 2024-06-04T05:24:06.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.730 INFO:teuthology.orchestra.run.smithi098.stdout: "purge_queue": { 2024-06-04T05:24:06.730 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops": 0, 2024-06-04T05:24:06.730 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops_high_water": 0, 2024-06-04T05:24:06.730 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing": 0, 2024-06-04T05:24:06.730 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_high_water": 0, 2024-06-04T05:24:06.730 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed_ops": 0, 2024-06-04T05:24:06.730 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed": 0, 2024-06-04T05:24:06.730 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_item_in_journal": 0 2024-06-04T05:24:06.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.730 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-04T05:24:06.730 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:06.730 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-06-04T05:24:06.730 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:06.731 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 6983, 2024-06-04T05:24:06.731 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 296103399, 2024-06-04T05:24:06.731 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:06.731 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 6983, 2024-06-04T05:24:06.731 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-06-04T05:24:06.731 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-06-04T05:24:06.731 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 6983, 2024-06-04T05:24:06.731 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 296103399, 2024-06-04T05:24:06.731 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:06.731 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.731 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.731 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.731 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:06.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.732 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_bytes": { 2024-06-04T05:24:06.732 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:06.732 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-06-04T05:24:06.732 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:06.732 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-06-04T05:24:06.732 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-06-04T05:24:06.732 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:06.732 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:06.732 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 6386, 2024-06-04T05:24:06.732 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 293002784, 2024-06-04T05:24:06.732 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 6386, 2024-06-04T05:24:06.732 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 293002784, 2024-06-04T05:24:06.732 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:06.732 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.732 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.733 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.733 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:06.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.733 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_ops": { 2024-06-04T05:24:06.733 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:06.733 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 1024, 2024-06-04T05:24:06.733 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:06.733 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-06-04T05:24:06.733 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-06-04T05:24:06.733 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:06.733 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:06.733 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 6386, 2024-06-04T05:24:06.733 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 6386, 2024-06-04T05:24:06.733 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 6386, 2024-06-04T05:24:06.733 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 6386, 2024-06-04T05:24:06.734 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:06.734 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.734 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.734 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.734 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:06.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.734 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle": { 2024-06-04T05:24:06.734 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:06.734 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-06-04T05:24:06.734 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:06.734 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-06-04T05:24:06.734 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-06-04T05:24:06.734 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:06.734 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:06.735 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-06-04T05:24:06.735 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-06-04T05:24:06.735 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-06-04T05:24:06.735 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-06-04T05:24:06.735 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:06.735 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.735 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.735 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.735 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:06.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:06.735 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle-0x55e820e0a6e0": { 2024-06-04T05:24:06.735 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:06.735 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-06-04T05:24:06.735 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:06.735 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-06-04T05:24:06.736 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-06-04T05:24:06.736 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:06.736 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:06.736 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-06-04T05:24:06.736 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-06-04T05:24:06.736 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-06-04T05:24:06.736 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-06-04T05:24:06.736 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:06.736 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:06.736 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:06.736 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:06.736 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:06.736 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:06.736 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-04T05:24:07.629 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-06-04T05:24:07.630 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-06-04T05:24:07.630 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-06-04T05:24:07.630 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-06-04T05:24:07.630 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-06-04T05:24:07.630 DEBUG:tasks.check_counter:Getting stats from h 2024-06-04T05:24:07.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph daemon mds.h perf dump 2024-06-04T05:24:07.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:07 smithi070 ceph-mon[32631]: pgmap v849: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 32 op/s 2024-06-04T05:24:07.870 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.b/config 2024-06-04T05:24:07.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:07 smithi112 ceph-mon[36453]: pgmap v849: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 32 op/s 2024-06-04T05:24:07.992 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:07 smithi098 ceph-mon[35569]: pgmap v849: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 32 op/s 2024-06-04T05:24:09.649 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-04T05:24:09.650 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-06-04T05:24:09.650 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 47440, 2024-06-04T05:24:09.650 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 48118, 2024-06-04T05:24:09.650 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 15359560, 2024-06-04T05:24:09.650 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 99972153, 2024-06-04T05:24:09.650 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 20, 2024-06-04T05:24:09.650 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 6, 2024-06-04T05:24:09.650 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 3.033676832, 2024-06-04T05:24:09.650 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 1.371491980, 2024-06-04T05:24:09.650 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 1.580805231, 2024-06-04T05:24:09.650 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.342597117, 2024-06-04T05:24:09.651 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:09.651 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 48113, 2024-06-04T05:24:09.651 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.898919841, 2024-06-04T05:24:09.651 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000018683 2024-06-04T05:24:09.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.651 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:09.651 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 69219, 2024-06-04T05:24:09.651 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.011735202, 2024-06-04T05:24:09.651 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000169 2024-06-04T05:24:09.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.651 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3744, 2024-06-04T05:24:09.651 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 674096 2024-06-04T05:24:09.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.652 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-06-04T05:24:09.652 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 380697, 2024-06-04T05:24:09.652 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 430982, 2024-06-04T05:24:09.652 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 118430511, 2024-06-04T05:24:09.652 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 341406893, 2024-06-04T05:24:09.652 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 19, 2024-06-04T05:24:09.652 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 6, 2024-06-04T05:24:09.652 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 17.958418824, 2024-06-04T05:24:09.652 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 7.446038100, 2024-06-04T05:24:09.652 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 12.800792469, 2024-06-04T05:24:09.652 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.288288940, 2024-06-04T05:24:09.652 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:09.652 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 430983, 2024-06-04T05:24:09.652 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 6.692387744, 2024-06-04T05:24:09.652 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000015528 2024-06-04T05:24:09.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.653 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:09.653 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 775642, 2024-06-04T05:24:09.653 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.110794690, 2024-06-04T05:24:09.653 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000142 2024-06-04T05:24:09.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.653 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-06-04T05:24:09.653 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 158448 2024-06-04T05:24:09.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.653 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-06-04T05:24:09.653 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 25333, 2024-06-04T05:24:09.653 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 25551, 2024-06-04T05:24:09.653 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 9216071, 2024-06-04T05:24:09.653 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 118924726, 2024-06-04T05:24:09.654 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 57, 2024-06-04T05:24:09.654 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 7, 2024-06-04T05:24:09.654 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 1.702768874, 2024-06-04T05:24:09.654 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.626712199, 2024-06-04T05:24:09.654 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.644654488, 2024-06-04T05:24:09.654 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.451209198, 2024-06-04T05:24:09.654 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:09.654 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 25550, 2024-06-04T05:24:09.654 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.600011782, 2024-06-04T05:24:09.654 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000023483 2024-06-04T05:24:09.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.654 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:09.655 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 22966, 2024-06-04T05:24:09.655 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.003457495, 2024-06-04T05:24:09.655 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000150 2024-06-04T05:24:09.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.655 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 2006960, 2024-06-04T05:24:09.655 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 339760 2024-06-04T05:24:09.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.655 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-06-04T05:24:09.655 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 1, 2024-06-04T05:24:09.655 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-06-04T05:24:09.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.655 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-MDSRank": { 2024-06-04T05:24:09.655 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-06-04T05:24:09.655 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-06-04T05:24:09.655 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 40057, 2024-06-04T05:24:09.656 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 34.804333322, 2024-06-04T05:24:09.656 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000868870 2024-06-04T05:24:09.656 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:09.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.656 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-PurgeQueue": { 2024-06-04T05:24:09.656 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-06-04T05:24:09.656 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-06-04T05:24:09.656 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 19821, 2024-06-04T05:24:09.656 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 1.567047348, 2024-06-04T05:24:09.656 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000079059 2024-06-04T05:24:09.656 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:09.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.656 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": { 2024-06-04T05:24:09.657 INFO:teuthology.orchestra.run.smithi098.stdout: "request": 278939, 2024-06-04T05:24:09.657 INFO:teuthology.orchestra.run.smithi098.stdout: "reply": 278854, 2024-06-04T05:24:09.657 INFO:teuthology.orchestra.run.smithi098.stdout: "reply_latency": { 2024-06-04T05:24:09.657 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 278854, 2024-06-04T05:24:09.657 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 838.441227109, 2024-06-04T05:24:09.657 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.003006739 2024-06-04T05:24:09.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.657 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_reply": 0, 2024-06-04T05:24:09.657 INFO:teuthology.orchestra.run.smithi098.stdout: "forward": 85, 2024-06-04T05:24:09.657 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_complete": 3, 2024-06-04T05:24:09.657 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_keys": 0, 2024-06-04T05:24:09.657 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_commit": 86, 2024-06-04T05:24:09.657 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_split": 9, 2024-06-04T05:24:09.657 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_merge": 1, 2024-06-04T05:24:09.657 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes": 1726, 2024-06-04T05:24:09.658 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_top": 181, 2024-06-04T05:24:09.658 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_bottom": 1412, 2024-06-04T05:24:09.658 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pin_tail": 133, 2024-06-04T05:24:09.658 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pinned": 1464, 2024-06-04T05:24:09.658 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_expired": 1869, 2024-06-04T05:24:09.658 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_with_caps": 0, 2024-06-04T05:24:09.658 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": 0, 2024-06-04T05:24:09.658 INFO:teuthology.orchestra.run.smithi098.stdout: "subtrees": 6, 2024-06-04T05:24:09.658 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse": 532751, 2024-06-04T05:24:09.658 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_hit": 504131, 2024-06-04T05:24:09.658 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_forward": 82, 2024-06-04T05:24:09.658 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_discover": 133, 2024-06-04T05:24:09.658 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_dir_fetch": 0, 2024-06-04T05:24:09.659 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_remote_ino": 0, 2024-06-04T05:24:09.659 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_lock": 58, 2024-06-04T05:24:09.659 INFO:teuthology.orchestra.run.smithi098.stdout: "load_cent": 0, 2024-06-04T05:24:09.659 INFO:teuthology.orchestra.run.smithi098.stdout: "q": 0, 2024-06-04T05:24:09.659 INFO:teuthology.orchestra.run.smithi098.stdout: "exported": 49, 2024-06-04T05:24:09.659 INFO:teuthology.orchestra.run.smithi098.stdout: "exported_inodes": 1394, 2024-06-04T05:24:09.659 INFO:teuthology.orchestra.run.smithi098.stdout: "imported": 15, 2024-06-04T05:24:09.659 INFO:teuthology.orchestra.run.smithi098.stdout: "imported_inodes": 586, 2024-06-04T05:24:09.659 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_dir_fetch": 0, 2024-06-04T05:24:09.659 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_backtrace_fetch": 2193, 2024-06-04T05:24:09.659 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_peer_discover": 0, 2024-06-04T05:24:09.659 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rfiles": 4, 2024-06-04T05:24:09.659 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rbytes": 234, 2024-06-04T05:24:09.659 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rsnaps": 7, 2024-06-04T05:24:09.660 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_fetch": 2193, 2024-06-04T05:24:09.660 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_set_tag": 0, 2024-06-04T05:24:09.660 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_repaired": 0, 2024-06-04T05:24:09.660 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_inotable_repaired": 0, 2024-06-04T05:24:09.660 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_inodes": 162, 2024-06-04T05:24:09.660 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_base_inodes": 0, 2024-06-04T05:24:09.660 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dirfrag_rstats": 162, 2024-06-04T05:24:09.660 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_file_inodes": 2031, 2024-06-04T05:24:09.660 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_inode_file_caps": 139, 2024-06-04T05:24:09.660 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_revoke": 28587, 2024-06-04T05:24:09.660 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_grant": 41982, 2024-06-04T05:24:09.660 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_trunc": 0, 2024-06-04T05:24:09.660 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-04T05:24:09.660 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-04T05:24:09.661 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps": 45016, 2024-06-04T05:24:09.661 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps_dirty": 12678, 2024-06-04T05:24:09.661 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_cap_release": 540, 2024-06-04T05:24:09.661 INFO:teuthology.orchestra.run.smithi098.stdout: "process_request_cap_release": 36200 2024-06-04T05:24:09.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.661 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache": { 2024-06-04T05:24:09.661 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update": 470, 2024-06-04T05:24:09.661 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update_receipt": 51, 2024-06-04T05:24:09.661 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_try_discover": 37, 2024-06-04T05:24:09.661 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_send_discover": 156, 2024-06-04T05:24:09.661 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_handle_discover": 592, 2024-06-04T05:24:09.661 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays": 18, 2024-06-04T05:24:09.661 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_delayed": 0, 2024-06-04T05:24:09.662 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_enqueuing": 0, 2024-06-04T05:24:09.662 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_created": 9993, 2024-06-04T05:24:09.662 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_enqueued": 9975, 2024-06-04T05:24:09.662 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_reintegrated": 0, 2024-06-04T05:24:09.662 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_migrated": 0, 2024-06-04T05:24:09.662 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_processing": 0, 2024-06-04T05:24:09.662 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_enqueued": 0, 2024-06-04T05:24:09.662 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_prioritized": 0, 2024-06-04T05:24:09.662 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_started": 0, 2024-06-04T05:24:09.662 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_completed": 0, 2024-06-04T05:24:09.662 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_path": 20, 2024-06-04T05:24:09.662 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_inode": 2662, 2024-06-04T05:24:09.662 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_enqueue_scrub": 0, 2024-06-04T05:24:09.663 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_exportdir": 52, 2024-06-04T05:24:09.663 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_flush": 0, 2024-06-04T05:24:09.663 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragmentdir": 10, 2024-06-04T05:24:09.663 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragstats": 0, 2024-06-04T05:24:09.663 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_inodestats": 0 2024-06-04T05:24:09.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.663 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log": { 2024-06-04T05:24:09.663 INFO:teuthology.orchestra.run.smithi098.stdout: "evlrg": 0, 2024-06-04T05:24:09.663 INFO:teuthology.orchestra.run.smithi098.stdout: "evadd": 78393, 2024-06-04T05:24:09.663 INFO:teuthology.orchestra.run.smithi098.stdout: "evex": 0, 2024-06-04T05:24:09.663 INFO:teuthology.orchestra.run.smithi098.stdout: "evtrm": 0, 2024-06-04T05:24:09.663 INFO:teuthology.orchestra.run.smithi098.stdout: "ev": 78393, 2024-06-04T05:24:09.663 INFO:teuthology.orchestra.run.smithi098.stdout: "evexg": 0, 2024-06-04T05:24:09.663 INFO:teuthology.orchestra.run.smithi098.stdout: "evexd": 0, 2024-06-04T05:24:09.663 INFO:teuthology.orchestra.run.smithi098.stdout: "segadd": 116, 2024-06-04T05:24:09.664 INFO:teuthology.orchestra.run.smithi098.stdout: "segex": 0, 2024-06-04T05:24:09.664 INFO:teuthology.orchestra.run.smithi098.stdout: "segtrm": 0, 2024-06-04T05:24:09.664 INFO:teuthology.orchestra.run.smithi098.stdout: "seg": 116, 2024-06-04T05:24:09.664 INFO:teuthology.orchestra.run.smithi098.stdout: "segmjr": 8, 2024-06-04T05:24:09.664 INFO:teuthology.orchestra.run.smithi098.stdout: "segexg": 0, 2024-06-04T05:24:09.664 INFO:teuthology.orchestra.run.smithi098.stdout: "segexd": 0, 2024-06-04T05:24:09.664 INFO:teuthology.orchestra.run.smithi098.stdout: "expos": 4194304, 2024-06-04T05:24:09.664 INFO:teuthology.orchestra.run.smithi098.stdout: "wrpos": 265462912, 2024-06-04T05:24:09.664 INFO:teuthology.orchestra.run.smithi098.stdout: "rdpos": 0, 2024-06-04T05:24:09.664 INFO:teuthology.orchestra.run.smithi098.stdout: "jlat": { 2024-06-04T05:24:09.664 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 19880, 2024-06-04T05:24:09.664 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 106.103839490, 2024-06-04T05:24:09.664 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.005337215 2024-06-04T05:24:09.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: "replayed": 0 2024-06-04T05:24:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mem": { 2024-06-04T05:24:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: "ino": 1471, 2024-06-04T05:24:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: "ino+": 12975, 2024-06-04T05:24:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: "ino-": 11504, 2024-06-04T05:24:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: "dir": 88, 2024-06-04T05:24:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: "dir+": 174, 2024-06-04T05:24:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: "dir-": 86, 2024-06-04T05:24:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: "dn": 1726, 2024-06-04T05:24:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: "dn+": 15835, 2024-06-04T05:24:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: "dn-": 14109, 2024-06-04T05:24:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: "cap": 0, 2024-06-04T05:24:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: "cap+": 10082, 2024-06-04T05:24:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: "cap-": 10082, 2024-06-04T05:24:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: "rss": 86576, 2024-06-04T05:24:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: "heap": 223608 2024-06-04T05:24:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server": { 2024-06-04T05:24:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_client_request": 324796, 2024-06-04T05:24:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_server_request": 1001, 2024-06-04T05:24:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_request": 278939, 2024-06-04T05:24:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_session": 907, 2024-06-04T05:24:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_peer_request": 9486, 2024-06-04T05:24:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: "req_create_latency": { 2024-06-04T05:24:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 9766, 2024-06-04T05:24:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 410.698579773, 2024-06-04T05:24:09.667 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.042053919 2024-06-04T05:24:09.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.667 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getattr_latency": { 2024-06-04T05:24:09.667 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 217702, 2024-06-04T05:24:09.667 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 85.657110215, 2024-06-04T05:24:09.667 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000393460 2024-06-04T05:24:09.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.667 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getfilelock_latency": { 2024-06-04T05:24:09.667 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.667 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.667 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.667 INFO:teuthology.orchestra.run.smithi098.stdout: "req_link_latency": { 2024-06-04T05:24:09.668 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.668 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.668 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.668 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookup_latency": { 2024-06-04T05:24:09.668 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 6272, 2024-06-04T05:24:09.668 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 5.978253361, 2024-06-04T05:24:09.668 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000953165 2024-06-04T05:24:09.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.668 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookuphash_latency": { 2024-06-04T05:24:09.668 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.668 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.668 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.669 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupino_latency": { 2024-06-04T05:24:09.669 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.669 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.669 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.669 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupname_latency": { 2024-06-04T05:24:09.669 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.669 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.669 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.669 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupparent_latency": { 2024-06-04T05:24:09.669 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.669 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.669 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.670 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupsnap_latency": { 2024-06-04T05:24:09.670 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.670 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.670 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.671 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lssnap_latency": { 2024-06-04T05:24:09.671 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.671 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.671 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.671 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mkdir_latency": { 2024-06-04T05:24:09.671 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.671 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.671 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.671 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mknod_latency": { 2024-06-04T05:24:09.671 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.671 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mksnap_latency": { 2024-06-04T05:24:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: "req_open_latency": { 2024-06-04T05:24:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 6842, 2024-06-04T05:24:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 5.962868090, 2024-06-04T05:24:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000871509 2024-06-04T05:24:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: "req_readdir_latency": { 2024-06-04T05:24:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 25720, 2024-06-04T05:24:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 59.105849577, 2024-06-04T05:24:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.002298050 2024-06-04T05:24:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rename_latency": { 2024-06-04T05:24:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2148, 2024-06-04T05:24:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 58.903487838, 2024-06-04T05:24:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.027422480 2024-06-04T05:24:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: "req_renamesnap_latency": { 2024-06-04T05:24:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.674 INFO:teuthology.orchestra.run.smithi098.stdout: "req_snapdiff_latency": { 2024-06-04T05:24:09.674 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.674 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.674 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.674 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmdir_latency": { 2024-06-04T05:24:09.674 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 18, 2024-06-04T05:24:09.674 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.051944752, 2024-06-04T05:24:09.674 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.002885819 2024-06-04T05:24:09.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.674 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmsnap_latency": { 2024-06-04T05:24:09.674 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.674 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.675 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.675 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmxattr_latency": { 2024-06-04T05:24:09.675 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.675 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.675 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.675 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setattr_latency": { 2024-06-04T05:24:09.675 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 43, 2024-06-04T05:24:09.675 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 111.296692817, 2024-06-04T05:24:09.675 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 2.588295181 2024-06-04T05:24:09.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.675 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setdirlayout_latency": { 2024-06-04T05:24:09.675 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.676 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.676 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.676 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setfilelock_latency": { 2024-06-04T05:24:09.676 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 368, 2024-06-04T05:24:09.676 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.164402430, 2024-06-04T05:24:09.676 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000446745 2024-06-04T05:24:09.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.676 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setlayout_latency": { 2024-06-04T05:24:09.676 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.676 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.676 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.676 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setxattr_latency": { 2024-06-04T05:24:09.677 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.677 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.677 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.677 INFO:teuthology.orchestra.run.smithi098.stdout: "req_symlink_latency": { 2024-06-04T05:24:09.677 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.677 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.677 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.677 INFO:teuthology.orchestra.run.smithi098.stdout: "req_unlink_latency": { 2024-06-04T05:24:09.677 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 9975, 2024-06-04T05:24:09.677 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 100.622038256, 2024-06-04T05:24:09.677 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.010087422 2024-06-04T05:24:09.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.678 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_revoke_eviction": 0, 2024-06-04T05:24:09.678 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_acquisition_throttle": 0, 2024-06-04T05:24:09.678 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getvxattr_latency": { 2024-06-04T05:24:09.678 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.678 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.678 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.678 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:09.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.678 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessions": { 2024-06-04T05:24:09.678 INFO:teuthology.orchestra.run.smithi098.stdout: "session_count": 1, 2024-06-04T05:24:09.678 INFO:teuthology.orchestra.run.smithi098.stdout: "session_add": 4, 2024-06-04T05:24:09.678 INFO:teuthology.orchestra.run.smithi098.stdout: "session_remove": 3, 2024-06-04T05:24:09.678 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_open": 1, 2024-06-04T05:24:09.679 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_stale": 0, 2024-06-04T05:24:09.679 INFO:teuthology.orchestra.run.smithi098.stdout: "total_load": 31353, 2024-06-04T05:24:09.679 INFO:teuthology.orchestra.run.smithi098.stdout: "average_load": 31353, 2024-06-04T05:24:09.679 INFO:teuthology.orchestra.run.smithi098.stdout: "avg_session_uptime": 1618, 2024-06-04T05:24:09.679 INFO:teuthology.orchestra.run.smithi098.stdout: "mdthresh_evicted": 0 2024-06-04T05:24:09.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.679 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-06-04T05:24:09.679 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-06-04T05:24:09.679 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-06-04T05:24:09.679 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-06-04T05:24:09.679 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-06-04T05:24:09.679 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-06-04T05:24:09.679 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-06-04T05:24:09.679 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-04T05:24:09.680 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-06-04T05:24:09.680 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-04T05:24:09.680 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-06-04T05:24:09.680 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-06-04T05:24:09.680 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-06-04T05:24:09.680 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-04T05:24:09.680 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-06-04T05:24:09.680 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-06-04T05:24:09.680 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-06-04T05:24:09.680 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-06-04T05:24:09.680 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-06-04T05:24:09.680 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-04T05:24:09.680 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-06-04T05:24:09.680 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-04T05:24:09.681 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-06-04T05:24:09.681 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-06-04T05:24:09.681 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-06-04T05:24:09.681 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-06-04T05:24:09.681 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-06-04T05:24:09.681 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-04T05:24:09.681 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-06-04T05:24:09.681 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-06-04T05:24:09.681 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-06-04T05:24:09.681 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-06-04T05:24:09.681 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-06-04T05:24:09.681 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-06-04T05:24:09.681 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-06-04T05:24:09.681 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-06-04T05:24:09.682 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-06-04T05:24:09.682 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 425769, 2024-06-04T05:24:09.682 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 138, 2024-06-04T05:24:09.682 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-06-04T05:24:09.682 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-06-04T05:24:09.682 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-06-04T05:24:09.682 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-06-04T05:24:09.682 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-06-04T05:24:09.682 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-06-04T05:24:09.682 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-06-04T05:24:09.682 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-06-04T05:24:09.682 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 8224, 2024-06-04T05:24:09.682 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 171, 2024-06-04T05:24:09.683 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-06-04T05:24:09.683 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-06-04T05:24:09.683 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-06-04T05:24:09.683 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-06-04T05:24:09.683 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 4759719, 2024-06-04T05:24:09.683 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 19646, 2024-06-04T05:24:09.683 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-06-04T05:24:09.683 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-06-04T05:24:09.683 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-06-04T05:24:09.683 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-06-04T05:24:09.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.683 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter": { 2024-06-04T05:24:09.683 INFO:teuthology.orchestra.run.smithi098.stdout: "op_active": 0, 2024-06-04T05:24:09.684 INFO:teuthology.orchestra.run.smithi098.stdout: "op_laggy": 0, 2024-06-04T05:24:09.684 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send": 35328, 2024-06-04T05:24:09.684 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send_bytes": 263697343, 2024-06-04T05:24:09.684 INFO:teuthology.orchestra.run.smithi098.stdout: "op_resend": 33, 2024-06-04T05:24:09.684 INFO:teuthology.orchestra.run.smithi098.stdout: "op_reply": 35328, 2024-06-04T05:24:09.684 INFO:teuthology.orchestra.run.smithi098.stdout: "op_latency": { 2024-06-04T05:24:09.684 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 35328, 2024-06-04T05:24:09.684 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 150.047193641, 2024-06-04T05:24:09.684 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.004247259 2024-06-04T05:24:09.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.684 INFO:teuthology.orchestra.run.smithi098.stdout: "op_inflight": 0, 2024-06-04T05:24:09.684 INFO:teuthology.orchestra.run.smithi098.stdout: "oplen_avg": { 2024-06-04T05:24:09.684 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 35328, 2024-06-04T05:24:09.685 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 36502 2024-06-04T05:24:09.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.685 INFO:teuthology.orchestra.run.smithi098.stdout: "op": 35328, 2024-06-04T05:24:09.685 INFO:teuthology.orchestra.run.smithi098.stdout: "op_r": 3193, 2024-06-04T05:24:09.685 INFO:teuthology.orchestra.run.smithi098.stdout: "op_w": 32135, 2024-06-04T05:24:09.685 INFO:teuthology.orchestra.run.smithi098.stdout: "op_rmw": 0, 2024-06-04T05:24:09.685 INFO:teuthology.orchestra.run.smithi098.stdout: "op_pg": 0, 2024-06-04T05:24:09.685 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_stat": 0, 2024-06-04T05:24:09.685 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_create": 10, 2024-06-04T05:24:09.685 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_read": 988, 2024-06-04T05:24:09.685 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_write": 20868, 2024-06-04T05:24:09.685 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writefull": 553, 2024-06-04T05:24:09.685 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writesame": 0, 2024-06-04T05:24:09.685 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_append": 0, 2024-06-04T05:24:09.686 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_zero": 0, 2024-06-04T05:24:09.686 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_truncate": 9, 2024-06-04T05:24:09.686 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_delete": 10051, 2024-06-04T05:24:09.686 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_mapext": 0, 2024-06-04T05:24:09.686 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_sparse_read": 0, 2024-06-04T05:24:09.686 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_clonerange": 0, 2024-06-04T05:24:09.686 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_getxattr": 2194, 2024-06-04T05:24:09.686 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_setxattr": 20, 2024-06-04T05:24:09.686 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_cmpxattr": 0, 2024-06-04T05:24:09.686 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_rmxattr": 0, 2024-06-04T05:24:09.686 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_resetxattrs": 0, 2024-06-04T05:24:09.686 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_call": 0, 2024-06-04T05:24:09.686 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_watch": 0, 2024-06-04T05:24:09.686 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_notify": 0, 2024-06-04T05:24:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_src_cmpxattr": 0, 2024-06-04T05:24:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls": 0, 2024-06-04T05:24:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls_filter": 0, 2024-06-04T05:24:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_other": 0, 2024-06-04T05:24:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_active": 0, 2024-06-04T05:24:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_send": 0, 2024-06-04T05:24:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_resend": 0, 2024-06-04T05:24:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_ping": 0, 2024-06-04T05:24:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_active": 0, 2024-06-04T05:24:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_send": 0, 2024-06-04T05:24:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_resend": 0, 2024-06-04T05:24:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_active": 0, 2024-06-04T05:24:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_send": 0, 2024-06-04T05:24:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_resend": 0, 2024-06-04T05:24:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_active": 0, 2024-06-04T05:24:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_send": 0, 2024-06-04T05:24:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_resend": 0, 2024-06-04T05:24:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "command_active": 0, 2024-06-04T05:24:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "command_send": 0, 2024-06-04T05:24:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "command_resend": 0, 2024-06-04T05:24:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "map_epoch": 121, 2024-06-04T05:24:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "map_full": 0, 2024-06-04T05:24:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "map_inc": 42, 2024-06-04T05:24:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_sessions": 12, 2024-06-04T05:24:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_open": 17, 2024-06-04T05:24:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_close": 5, 2024-06-04T05:24:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_laggy": 0, 2024-06-04T05:24:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_wr": 1245, 2024-06-04T05:24:09.689 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_rd": 15, 2024-06-04T05:24:09.689 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_del": 549 2024-06-04T05:24:09.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.689 INFO:teuthology.orchestra.run.smithi098.stdout: "oft": { 2024-06-04T05:24:09.689 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_objs": 1, 2024-06-04T05:24:09.689 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_kv_pairs": 0, 2024-06-04T05:24:09.689 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_updates": 0, 2024-06-04T05:24:09.689 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_removes": 0 2024-06-04T05:24:09.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.689 INFO:teuthology.orchestra.run.smithi098.stdout: "osd-slow-ops": { 2024-06-04T05:24:09.689 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_ops_count": 2744 2024-06-04T05:24:09.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.689 INFO:teuthology.orchestra.run.smithi098.stdout: "purge_queue": { 2024-06-04T05:24:09.689 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops": 0, 2024-06-04T05:24:09.689 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops_high_water": 6, 2024-06-04T05:24:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing": 0, 2024-06-04T05:24:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_high_water": 6, 2024-06-04T05:24:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed_ops": 9975, 2024-06-04T05:24:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed": 9975, 2024-06-04T05:24:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_item_in_journal": 0 2024-06-04T05:24:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-04T05:24:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-06-04T05:24:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 453466, 2024-06-04T05:24:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 102157809, 2024-06-04T05:24:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 453466, 2024-06-04T05:24:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-06-04T05:24:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-06-04T05:24:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 453466, 2024-06-04T05:24:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 102157809, 2024-06-04T05:24:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_bytes": { 2024-06-04T05:24:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-06-04T05:24:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-06-04T05:24:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-06-04T05:24:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 35328, 2024-06-04T05:24:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 264717317, 2024-06-04T05:24:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 35328, 2024-06-04T05:24:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 264717317, 2024-06-04T05:24:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_ops": { 2024-06-04T05:24:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 1024, 2024-06-04T05:24:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-06-04T05:24:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-06-04T05:24:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 35328, 2024-06-04T05:24:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 35328, 2024-06-04T05:24:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 35328, 2024-06-04T05:24:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 35328, 2024-06-04T05:24:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle": { 2024-06-04T05:24:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-06-04T05:24:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 9975, 2024-06-04T05:24:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1020283, 2024-06-04T05:24:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 9975, 2024-06-04T05:24:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-06-04T05:24:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-06-04T05:24:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 988, 2024-06-04T05:24:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1020283, 2024-06-04T05:24:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle-0x56157b6d2500": { 2024-06-04T05:24:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-06-04T05:24:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 78393, 2024-06-04T05:24:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 261268608, 2024-06-04T05:24:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 78393, 2024-06-04T05:24:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-06-04T05:24:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-06-04T05:24:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 19880, 2024-06-04T05:24:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 261268608, 2024-06-04T05:24:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:09.697 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:09.697 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:09.697 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-04T05:24:09.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:09 smithi070 ceph-mon[32631]: pgmap v850: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-06-04T05:24:09.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:09 smithi070 ceph-mon[32631]: osdmap e121: 12 total, 12 up, 12 in 2024-06-04T05:24:09.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:09 smithi112 ceph-mon[36453]: pgmap v850: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-06-04T05:24:09.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:09 smithi112 ceph-mon[36453]: osdmap e121: 12 total, 12 up, 12 in 2024-06-04T05:24:09.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:09 smithi098 ceph-mon[35569]: pgmap v850: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-06-04T05:24:09.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:09 smithi098 ceph-mon[35569]: osdmap e121: 12 total, 12 up, 12 in 2024-06-04T05:24:10.470 INFO:tasks.check_counter:Daemon mds.h mds.exported=49 2024-06-04T05:24:10.470 INFO:tasks.check_counter:Daemon mds.h mds.imported=15 2024-06-04T05:24:10.470 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=7 2024-06-04T05:24:10.470 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-06-04T05:24:10.471 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-06-04T05:24:10.471 DEBUG:tasks.check_counter:Getting stats from k 2024-06-04T05:24:10.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph daemon mds.k perf dump 2024-06-04T05:24:10.709 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.b/config 2024-06-04T05:24:11.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:11 smithi070 ceph-mon[32631]: pgmap v852: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 32 op/s 2024-06-04T05:24:11.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:11 smithi112 ceph-mon[36453]: pgmap v852: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 32 op/s 2024-06-04T05:24:11.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:11 smithi098 ceph-mon[35569]: pgmap v852: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 32 op/s 2024-06-04T05:24:12.862 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-04T05:24:12.862 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-06-04T05:24:12.862 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 1602, 2024-06-04T05:24:12.863 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 1604, 2024-06-04T05:24:12.863 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 99638025, 2024-06-04T05:24:12.863 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 509500, 2024-06-04T05:24:12.863 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 9, 2024-06-04T05:24:12.863 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 4, 2024-06-04T05:24:12.863 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.174988321, 2024-06-04T05:24:12.863 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.038754261, 2024-06-04T05:24:12.863 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.087628388, 2024-06-04T05:24:12.863 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.037288445, 2024-06-04T05:24:12.863 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:12.863 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1602, 2024-06-04T05:24:12.863 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.044523812, 2024-06-04T05:24:12.863 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000027792 2024-06-04T05:24:12.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.864 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:12.864 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.864 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.864 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.864 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-06-04T05:24:12.864 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-06-04T05:24:12.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.864 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-06-04T05:24:12.864 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 2087, 2024-06-04T05:24:12.864 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2466, 2024-06-04T05:24:12.864 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 109587689, 2024-06-04T05:24:12.865 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 1492459, 2024-06-04T05:24:12.865 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 8, 2024-06-04T05:24:12.865 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 6, 2024-06-04T05:24:12.865 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.236224604, 2024-06-04T05:24:12.865 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.052701872, 2024-06-04T05:24:12.865 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.112701190, 2024-06-04T05:24:12.865 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.054468673, 2024-06-04T05:24:12.865 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:12.865 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2463, 2024-06-04T05:24:12.865 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.079163101, 2024-06-04T05:24:12.865 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000032140 2024-06-04T05:24:12.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.865 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:12.865 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.866 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.866 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.866 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 320, 2024-06-04T05:24:12.866 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 829424 2024-06-04T05:24:12.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.866 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-06-04T05:24:12.866 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 3286, 2024-06-04T05:24:12.866 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 3229, 2024-06-04T05:24:12.866 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 57347900, 2024-06-04T05:24:12.866 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 1170156, 2024-06-04T05:24:12.866 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-06-04T05:24:12.867 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 4, 2024-06-04T05:24:12.867 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.341578834, 2024-06-04T05:24:12.867 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.098280460, 2024-06-04T05:24:12.867 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.138850447, 2024-06-04T05:24:12.867 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.090313907, 2024-06-04T05:24:12.867 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:12.867 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 3229, 2024-06-04T05:24:12.867 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.118369699, 2024-06-04T05:24:12.867 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000036658 2024-06-04T05:24:12.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.867 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:12.867 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.867 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.868 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.868 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1993264, 2024-06-04T05:24:12.868 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 317280 2024-06-04T05:24:12.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.868 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-06-04T05:24:12.868 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 1, 2024-06-04T05:24:12.868 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-06-04T05:24:12.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.868 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-MDSRank": { 2024-06-04T05:24:12.868 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-06-04T05:24:12.868 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-06-04T05:24:12.868 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 10881, 2024-06-04T05:24:12.869 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.247503689, 2024-06-04T05:24:12.869 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000022746 2024-06-04T05:24:12.869 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:12.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.869 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-PurgeQueue": { 2024-06-04T05:24:12.869 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-06-04T05:24:12.869 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-06-04T05:24:12.869 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.869 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.869 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.869 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:12.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.869 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": { 2024-06-04T05:24:12.870 INFO:teuthology.orchestra.run.smithi098.stdout: "request": 0, 2024-06-04T05:24:12.870 INFO:teuthology.orchestra.run.smithi098.stdout: "reply": 0, 2024-06-04T05:24:12.870 INFO:teuthology.orchestra.run.smithi098.stdout: "reply_latency": { 2024-06-04T05:24:12.870 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.870 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.870 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.870 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_reply": 0, 2024-06-04T05:24:12.870 INFO:teuthology.orchestra.run.smithi098.stdout: "forward": 0, 2024-06-04T05:24:12.870 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_complete": 0, 2024-06-04T05:24:12.870 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_keys": 0, 2024-06-04T05:24:12.870 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_commit": 0, 2024-06-04T05:24:12.870 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_split": 0, 2024-06-04T05:24:12.870 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_merge": 0, 2024-06-04T05:24:12.871 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes": 10795, 2024-06-04T05:24:12.871 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_top": 958, 2024-06-04T05:24:12.871 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_bottom": 9837, 2024-06-04T05:24:12.871 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pin_tail": 0, 2024-06-04T05:24:12.871 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pinned": 9436, 2024-06-04T05:24:12.871 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_expired": 0, 2024-06-04T05:24:12.871 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_with_caps": 0, 2024-06-04T05:24:12.871 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": 0, 2024-06-04T05:24:12.871 INFO:teuthology.orchestra.run.smithi098.stdout: "subtrees": 1, 2024-06-04T05:24:12.871 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse": 0, 2024-06-04T05:24:12.871 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_hit": 0, 2024-06-04T05:24:12.871 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_forward": 0, 2024-06-04T05:24:12.871 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_discover": 0, 2024-06-04T05:24:12.871 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_dir_fetch": 0, 2024-06-04T05:24:12.872 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_remote_ino": 0, 2024-06-04T05:24:12.872 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_lock": 0, 2024-06-04T05:24:12.872 INFO:teuthology.orchestra.run.smithi098.stdout: "load_cent": 0, 2024-06-04T05:24:12.872 INFO:teuthology.orchestra.run.smithi098.stdout: "q": 0, 2024-06-04T05:24:12.872 INFO:teuthology.orchestra.run.smithi098.stdout: "exported": 0, 2024-06-04T05:24:12.872 INFO:teuthology.orchestra.run.smithi098.stdout: "exported_inodes": 0, 2024-06-04T05:24:12.872 INFO:teuthology.orchestra.run.smithi098.stdout: "imported": 0, 2024-06-04T05:24:12.872 INFO:teuthology.orchestra.run.smithi098.stdout: "imported_inodes": 0, 2024-06-04T05:24:12.872 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_dir_fetch": 0, 2024-06-04T05:24:12.872 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_backtrace_fetch": 0, 2024-06-04T05:24:12.872 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_peer_discover": 0, 2024-06-04T05:24:12.872 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rfiles": 0, 2024-06-04T05:24:12.872 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rbytes": 0, 2024-06-04T05:24:12.873 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rsnaps": 0, 2024-06-04T05:24:12.873 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_fetch": 0, 2024-06-04T05:24:12.873 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_set_tag": 0, 2024-06-04T05:24:12.873 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_repaired": 0, 2024-06-04T05:24:12.873 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_inotable_repaired": 0, 2024-06-04T05:24:12.873 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_inodes": 0, 2024-06-04T05:24:12.873 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_base_inodes": 0, 2024-06-04T05:24:12.873 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dirfrag_rstats": 0, 2024-06-04T05:24:12.873 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_file_inodes": 0, 2024-06-04T05:24:12.873 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_inode_file_caps": 0, 2024-06-04T05:24:12.873 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_revoke": 0, 2024-06-04T05:24:12.873 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_grant": 0, 2024-06-04T05:24:12.873 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_trunc": 0, 2024-06-04T05:24:12.874 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-04T05:24:12.874 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-04T05:24:12.874 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps": 0, 2024-06-04T05:24:12.874 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps_dirty": 0, 2024-06-04T05:24:12.874 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_cap_release": 0, 2024-06-04T05:24:12.874 INFO:teuthology.orchestra.run.smithi098.stdout: "process_request_cap_release": 0 2024-06-04T05:24:12.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.874 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache": { 2024-06-04T05:24:12.874 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update": 0, 2024-06-04T05:24:12.874 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update_receipt": 0, 2024-06-04T05:24:12.874 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_try_discover": 0, 2024-06-04T05:24:12.874 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_send_discover": 0, 2024-06-04T05:24:12.874 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_handle_discover": 0, 2024-06-04T05:24:12.875 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays": 0, 2024-06-04T05:24:12.875 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_delayed": 0, 2024-06-04T05:24:12.875 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_enqueuing": 0, 2024-06-04T05:24:12.875 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_created": 0, 2024-06-04T05:24:12.875 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_enqueued": 0, 2024-06-04T05:24:12.875 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_reintegrated": 0, 2024-06-04T05:24:12.875 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_migrated": 0, 2024-06-04T05:24:12.875 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_processing": 0, 2024-06-04T05:24:12.875 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_enqueued": 0, 2024-06-04T05:24:12.875 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_prioritized": 0, 2024-06-04T05:24:12.875 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_started": 0, 2024-06-04T05:24:12.875 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_completed": 0, 2024-06-04T05:24:12.875 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_path": 0, 2024-06-04T05:24:12.875 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_inode": 0, 2024-06-04T05:24:12.876 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_enqueue_scrub": 0, 2024-06-04T05:24:12.876 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_exportdir": 0, 2024-06-04T05:24:12.876 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_flush": 0, 2024-06-04T05:24:12.876 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragmentdir": 0, 2024-06-04T05:24:12.876 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragstats": 0, 2024-06-04T05:24:12.876 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_inodestats": 0 2024-06-04T05:24:12.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.876 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log": { 2024-06-04T05:24:12.876 INFO:teuthology.orchestra.run.smithi098.stdout: "evlrg": 0, 2024-06-04T05:24:12.876 INFO:teuthology.orchestra.run.smithi098.stdout: "evadd": 0, 2024-06-04T05:24:12.876 INFO:teuthology.orchestra.run.smithi098.stdout: "evex": 0, 2024-06-04T05:24:12.876 INFO:teuthology.orchestra.run.smithi098.stdout: "evtrm": 0, 2024-06-04T05:24:12.876 INFO:teuthology.orchestra.run.smithi098.stdout: "ev": 72646, 2024-06-04T05:24:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: "evexg": 0, 2024-06-04T05:24:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: "evexd": 0, 2024-06-04T05:24:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: "segadd": 0, 2024-06-04T05:24:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: "segex": 0, 2024-06-04T05:24:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: "segtrm": 0, 2024-06-04T05:24:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: "seg": 105, 2024-06-04T05:24:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: "segmjr": 7, 2024-06-04T05:24:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: "segexg": 0, 2024-06-04T05:24:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: "segexd": 0, 2024-06-04T05:24:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: "expos": 4194304, 2024-06-04T05:24:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: "wrpos": 266999534, 2024-06-04T05:24:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: "rdpos": 266979428, 2024-06-04T05:24:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: "jlat": { 2024-06-04T05:24:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.878 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.878 INFO:teuthology.orchestra.run.smithi098.stdout: "replayed": 72646 2024-06-04T05:24:12.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.878 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mem": { 2024-06-04T05:24:12.878 INFO:teuthology.orchestra.run.smithi098.stdout: "ino": 2335, 2024-06-04T05:24:12.878 INFO:teuthology.orchestra.run.smithi098.stdout: "ino+": 10880, 2024-06-04T05:24:12.878 INFO:teuthology.orchestra.run.smithi098.stdout: "ino-": 8545, 2024-06-04T05:24:12.878 INFO:teuthology.orchestra.run.smithi098.stdout: "dir": 83, 2024-06-04T05:24:12.878 INFO:teuthology.orchestra.run.smithi098.stdout: "dir+": 120, 2024-06-04T05:24:12.878 INFO:teuthology.orchestra.run.smithi098.stdout: "dir-": 37, 2024-06-04T05:24:12.878 INFO:teuthology.orchestra.run.smithi098.stdout: "dn": 10795, 2024-06-04T05:24:12.878 INFO:teuthology.orchestra.run.smithi098.stdout: "dn+": 11546, 2024-06-04T05:24:12.878 INFO:teuthology.orchestra.run.smithi098.stdout: "dn-": 751, 2024-06-04T05:24:12.879 INFO:teuthology.orchestra.run.smithi098.stdout: "cap": 0, 2024-06-04T05:24:12.879 INFO:teuthology.orchestra.run.smithi098.stdout: "cap+": 0, 2024-06-04T05:24:12.879 INFO:teuthology.orchestra.run.smithi098.stdout: "cap-": 0, 2024-06-04T05:24:12.879 INFO:teuthology.orchestra.run.smithi098.stdout: "rss": 72052, 2024-06-04T05:24:12.879 INFO:teuthology.orchestra.run.smithi098.stdout: "heap": 215416 2024-06-04T05:24:12.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.879 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server": { 2024-06-04T05:24:12.879 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_client_request": 0, 2024-06-04T05:24:12.879 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_server_request": 0, 2024-06-04T05:24:12.879 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_request": 0, 2024-06-04T05:24:12.879 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_session": 0, 2024-06-04T05:24:12.879 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_peer_request": 0, 2024-06-04T05:24:12.879 INFO:teuthology.orchestra.run.smithi098.stdout: "req_create_latency": { 2024-06-04T05:24:12.879 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.879 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.880 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.880 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getattr_latency": { 2024-06-04T05:24:12.880 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.880 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.880 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.880 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getfilelock_latency": { 2024-06-04T05:24:12.880 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.880 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.880 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.880 INFO:teuthology.orchestra.run.smithi098.stdout: "req_link_latency": { 2024-06-04T05:24:12.880 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.880 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.880 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.881 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookup_latency": { 2024-06-04T05:24:12.881 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.881 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.881 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.881 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookuphash_latency": { 2024-06-04T05:24:12.881 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.881 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.881 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.881 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupino_latency": { 2024-06-04T05:24:12.881 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.881 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.881 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.882 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupname_latency": { 2024-06-04T05:24:12.882 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.882 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.882 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.882 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupparent_latency": { 2024-06-04T05:24:12.882 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.882 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.882 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.882 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupsnap_latency": { 2024-06-04T05:24:12.882 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.882 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.883 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.883 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lssnap_latency": { 2024-06-04T05:24:12.883 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.883 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.883 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.884 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mkdir_latency": { 2024-06-04T05:24:12.884 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.884 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.884 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.884 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mknod_latency": { 2024-06-04T05:24:12.884 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.884 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.884 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.884 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mksnap_latency": { 2024-06-04T05:24:12.884 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.884 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.885 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.885 INFO:teuthology.orchestra.run.smithi098.stdout: "req_open_latency": { 2024-06-04T05:24:12.885 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.885 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.885 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.885 INFO:teuthology.orchestra.run.smithi098.stdout: "req_readdir_latency": { 2024-06-04T05:24:12.885 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.885 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.885 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.885 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rename_latency": { 2024-06-04T05:24:12.885 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: "req_renamesnap_latency": { 2024-06-04T05:24:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: "req_snapdiff_latency": { 2024-06-04T05:24:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmdir_latency": { 2024-06-04T05:24:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmsnap_latency": { 2024-06-04T05:24:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmxattr_latency": { 2024-06-04T05:24:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setattr_latency": { 2024-06-04T05:24:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setdirlayout_latency": { 2024-06-04T05:24:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setfilelock_latency": { 2024-06-04T05:24:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setlayout_latency": { 2024-06-04T05:24:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setxattr_latency": { 2024-06-04T05:24:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "req_symlink_latency": { 2024-06-04T05:24:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "req_unlink_latency": { 2024-06-04T05:24:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_revoke_eviction": 0, 2024-06-04T05:24:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_acquisition_throttle": 0, 2024-06-04T05:24:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getvxattr_latency": { 2024-06-04T05:24:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessions": { 2024-06-04T05:24:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "session_count": 1, 2024-06-04T05:24:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "session_add": 1, 2024-06-04T05:24:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "session_remove": 0, 2024-06-04T05:24:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_open": 1, 2024-06-04T05:24:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_stale": 0, 2024-06-04T05:24:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "total_load": 0, 2024-06-04T05:24:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "average_load": 0, 2024-06-04T05:24:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "avg_session_uptime": 3714, 2024-06-04T05:24:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "mdthresh_evicted": 0 2024-06-04T05:24:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-06-04T05:24:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-06-04T05:24:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-06-04T05:24:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-06-04T05:24:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-06-04T05:24:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-06-04T05:24:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-06-04T05:24:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-04T05:24:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-06-04T05:24:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-04T05:24:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-06-04T05:24:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-06-04T05:24:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-06-04T05:24:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-04T05:24:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-06-04T05:24:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-06-04T05:24:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-06-04T05:24:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-06-04T05:24:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-06-04T05:24:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-04T05:24:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-06-04T05:24:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-04T05:24:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-06-04T05:24:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-06-04T05:24:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-06-04T05:24:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-06-04T05:24:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-06-04T05:24:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-04T05:24:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-06-04T05:24:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-06-04T05:24:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-06-04T05:24:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-06-04T05:24:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-06-04T05:24:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-06-04T05:24:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-06-04T05:24:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-06-04T05:24:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-06-04T05:24:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 235288, 2024-06-04T05:24:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 85, 2024-06-04T05:24:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-06-04T05:24:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-06-04T05:24:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-06-04T05:24:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-06-04T05:24:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-06-04T05:24:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-06-04T05:24:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-06-04T05:24:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-06-04T05:24:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 8344, 2024-06-04T05:24:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 174, 2024-06-04T05:24:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-06-04T05:24:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-06-04T05:24:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-06-04T05:24:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-06-04T05:24:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 12856049, 2024-06-04T05:24:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 50263, 2024-06-04T05:24:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-06-04T05:24:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-06-04T05:24:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-06-04T05:24:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-06-04T05:24:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter": { 2024-06-04T05:24:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "op_active": 0, 2024-06-04T05:24:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "op_laggy": 0, 2024-06-04T05:24:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send": 6369, 2024-06-04T05:24:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send_bytes": 21, 2024-06-04T05:24:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "op_resend": 3, 2024-06-04T05:24:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "op_reply": 6369, 2024-06-04T05:24:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "op_latency": { 2024-06-04T05:24:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 6369, 2024-06-04T05:24:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 4.283946726, 2024-06-04T05:24:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000672624 2024-06-04T05:24:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "op_inflight": 0, 2024-06-04T05:24:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "oplen_avg": { 2024-06-04T05:24:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 6369, 2024-06-04T05:24:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 6370 2024-06-04T05:24:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "op": 6369, 2024-06-04T05:24:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "op_r": 6369, 2024-06-04T05:24:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "op_w": 0, 2024-06-04T05:24:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "op_rmw": 0, 2024-06-04T05:24:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "op_pg": 0, 2024-06-04T05:24:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_stat": 3474, 2024-06-04T05:24:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_create": 0, 2024-06-04T05:24:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_read": 2893, 2024-06-04T05:24:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_write": 0, 2024-06-04T05:24:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writefull": 0, 2024-06-04T05:24:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writesame": 0, 2024-06-04T05:24:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_append": 0, 2024-06-04T05:24:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_zero": 0, 2024-06-04T05:24:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_truncate": 0, 2024-06-04T05:24:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_delete": 0, 2024-06-04T05:24:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_mapext": 0, 2024-06-04T05:24:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_sparse_read": 0, 2024-06-04T05:24:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_clonerange": 0, 2024-06-04T05:24:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_getxattr": 1, 2024-06-04T05:24:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_setxattr": 0, 2024-06-04T05:24:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_cmpxattr": 0, 2024-06-04T05:24:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_rmxattr": 0, 2024-06-04T05:24:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_resetxattrs": 0, 2024-06-04T05:24:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_call": 0, 2024-06-04T05:24:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_watch": 0, 2024-06-04T05:24:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_notify": 0, 2024-06-04T05:24:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_src_cmpxattr": 0, 2024-06-04T05:24:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls": 0, 2024-06-04T05:24:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls_filter": 0, 2024-06-04T05:24:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_other": 0, 2024-06-04T05:24:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_active": 0, 2024-06-04T05:24:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_send": 0, 2024-06-04T05:24:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_resend": 0, 2024-06-04T05:24:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_ping": 0, 2024-06-04T05:24:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_active": 0, 2024-06-04T05:24:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_send": 0, 2024-06-04T05:24:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_resend": 0, 2024-06-04T05:24:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_active": 0, 2024-06-04T05:24:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_send": 0, 2024-06-04T05:24:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_resend": 0, 2024-06-04T05:24:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_active": 0, 2024-06-04T05:24:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_send": 0, 2024-06-04T05:24:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_resend": 0, 2024-06-04T05:24:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "command_active": 0, 2024-06-04T05:24:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "command_send": 0, 2024-06-04T05:24:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "command_resend": 0, 2024-06-04T05:24:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "map_epoch": 121, 2024-06-04T05:24:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "map_full": 0, 2024-06-04T05:24:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "map_inc": 42, 2024-06-04T05:24:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_sessions": 10, 2024-06-04T05:24:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_open": 11, 2024-06-04T05:24:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_close": 1, 2024-06-04T05:24:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_laggy": 0, 2024-06-04T05:24:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_wr": 0, 2024-06-04T05:24:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_rd": 2, 2024-06-04T05:24:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_del": 0 2024-06-04T05:24:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "oft": { 2024-06-04T05:24:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_objs": 0, 2024-06-04T05:24:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_kv_pairs": 0, 2024-06-04T05:24:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_updates": 0, 2024-06-04T05:24:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_removes": 0 2024-06-04T05:24:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "osd-slow-ops": { 2024-06-04T05:24:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_ops_count": 0 2024-06-04T05:24:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "purge_queue": { 2024-06-04T05:24:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops": 0, 2024-06-04T05:24:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops_high_water": 0, 2024-06-04T05:24:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing": 0, 2024-06-04T05:24:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_high_water": 0, 2024-06-04T05:24:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed_ops": 0, 2024-06-04T05:24:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed": 0, 2024-06-04T05:24:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_item_in_journal": 0 2024-06-04T05:24:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-04T05:24:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-06-04T05:24:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 6971, 2024-06-04T05:24:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 265912778, 2024-06-04T05:24:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 6971, 2024-06-04T05:24:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-06-04T05:24:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-06-04T05:24:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 6971, 2024-06-04T05:24:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 265912778, 2024-06-04T05:24:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_bytes": { 2024-06-04T05:24:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-06-04T05:24:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-06-04T05:24:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-06-04T05:24:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 6369, 2024-06-04T05:24:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 262813576, 2024-06-04T05:24:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 6369, 2024-06-04T05:24:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 262813576, 2024-06-04T05:24:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_ops": { 2024-06-04T05:24:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 1024, 2024-06-04T05:24:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-06-04T05:24:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-06-04T05:24:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 6369, 2024-06-04T05:24:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 6369, 2024-06-04T05:24:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 6369, 2024-06-04T05:24:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 6369, 2024-06-04T05:24:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle": { 2024-06-04T05:24:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-06-04T05:24:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-06-04T05:24:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-06-04T05:24:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-06-04T05:24:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-06-04T05:24:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-06-04T05:24:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-06-04T05:24:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-04T05:24:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle-0x5631665786e0": { 2024-06-04T05:24:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-06-04T05:24:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-06-04T05:24:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-06-04T05:24:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-06-04T05:24:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-06-04T05:24:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-06-04T05:24:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-06-04T05:24:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-06-04T05:24:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-06-04T05:24:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-06-04T05:24:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-06-04T05:24:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-06-04T05:24:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-06-04T05:24:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-04T05:24:12.908 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-04T05:24:13.662 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-06-04T05:24:13.662 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-06-04T05:24:13.662 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-06-04T05:24:13.663 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-06-04T05:24:13.663 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-06-04T05:24:13.663 DEBUG:tasks.check_counter:Getting stats from c 2024-06-04T05:24:13.663 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph daemon mds.c perf dump 2024-06-04T05:24:13.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:13 smithi098 ceph-mon[35569]: pgmap v853: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 12 op/s 2024-06-04T05:24:13.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:13 smithi098 ceph-mon[35569]: osdmap e122: 12 total, 12 up, 12 in 2024-06-04T05:24:13.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:13 smithi070 ceph-mon[32631]: pgmap v853: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 12 op/s 2024-06-04T05:24:13.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:13 smithi070 ceph-mon[32631]: osdmap e122: 12 total, 12 up, 12 in 2024-06-04T05:24:13.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:13 smithi112 ceph-mon[36453]: pgmap v853: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 12 op/s 2024-06-04T05:24:13.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:13 smithi112 ceph-mon[36453]: osdmap e122: 12 total, 12 up, 12 in 2024-06-04T05:24:13.975 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.c/config 2024-06-04T05:24:15.643 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:15 smithi112 ceph-mon[36453]: pgmap v855: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 7 op/s 2024-06-04T05:24:15.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:15 smithi070 ceph-mon[32631]: pgmap v855: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 7 op/s 2024-06-04T05:24:15.902 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-06-04T05:24:15.902 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-0": { 2024-06-04T05:24:15.903 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 18930, 2024-06-04T05:24:15.903 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 19140, 2024-06-04T05:24:15.903 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 6018927, 2024-06-04T05:24:15.903 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 102163765, 2024-06-04T05:24:15.903 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 63, 2024-06-04T05:24:15.903 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 9, 2024-06-04T05:24:15.903 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 1.453263573, 2024-06-04T05:24:15.903 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.616223563, 2024-06-04T05:24:15.903 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.486585780, 2024-06-04T05:24:15.903 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.342216514, 2024-06-04T05:24:15.903 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:15.903 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 19135, 2024-06-04T05:24:15.903 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.584632477, 2024-06-04T05:24:15.903 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000030553 2024-06-04T05:24:15.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.904 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:15.904 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 11457, 2024-06-04T05:24:15.904 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.002140815, 2024-06-04T05:24:15.904 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000186 2024-06-04T05:24:15.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.904 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 3328, 2024-06-04T05:24:15.904 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 688400 2024-06-04T05:24:15.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.904 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-1": { 2024-06-04T05:24:15.904 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 494967, 2024-06-04T05:24:15.904 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 533130, 2024-06-04T05:24:15.904 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 163257534, 2024-06-04T05:24:15.905 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 407822301, 2024-06-04T05:24:15.905 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 15, 2024-06-04T05:24:15.905 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 6, 2024-06-04T05:24:15.905 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 25.549940629, 2024-06-04T05:24:15.905 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 10.831957019, 2024-06-04T05:24:15.905 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 18.299446654, 2024-06-04T05:24:15.905 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.298125214, 2024-06-04T05:24:15.905 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:15.905 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 533130, 2024-06-04T05:24:15.905 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 10.541457863, 2024-06-04T05:24:15.905 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000019772 2024-06-04T05:24:15.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.905 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:15.906 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 990008, 2024-06-04T05:24:15.906 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.153189415, 2024-06-04T05:24:15.906 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000154 2024-06-04T05:24:15.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.906 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 160, 2024-06-04T05:24:15.906 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 159792 2024-06-04T05:24:15.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.906 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-2": { 2024-06-04T05:24:15.906 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 17786, 2024-06-04T05:24:15.906 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 17959, 2024-06-04T05:24:15.906 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 7731060, 2024-06-04T05:24:15.906 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 132749773, 2024-06-04T05:24:15.907 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 11, 2024-06-04T05:24:15.907 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 6, 2024-06-04T05:24:15.907 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 1.456908093, 2024-06-04T05:24:15.907 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.546640340, 2024-06-04T05:24:15.907 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.352406724, 2024-06-04T05:24:15.907 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.495034708, 2024-06-04T05:24:15.907 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:15.907 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 17959, 2024-06-04T05:24:15.907 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.648271708, 2024-06-04T05:24:15.907 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000036097 2024-06-04T05:24:15.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.907 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:15.907 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 1773, 2024-06-04T05:24:15.907 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000413691, 2024-06-04T05:24:15.907 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000233 2024-06-04T05:24:15.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.908 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 2027824, 2024-06-04T05:24:15.908 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 346208 2024-06-04T05:24:15.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.908 INFO:teuthology.orchestra.run.smithi112.stdout: "cct": { 2024-06-04T05:24:15.908 INFO:teuthology.orchestra.run.smithi112.stdout: "total_workers": 1, 2024-06-04T05:24:15.908 INFO:teuthology.orchestra.run.smithi112.stdout: "unhealthy_workers": 0 2024-06-04T05:24:15.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.908 INFO:teuthology.orchestra.run.smithi112.stdout: "finisher-MDSRank": { 2024-06-04T05:24:15.908 INFO:teuthology.orchestra.run.smithi112.stdout: "queue_len": 0, 2024-06-04T05:24:15.908 INFO:teuthology.orchestra.run.smithi112.stdout: "complete_latency": { 2024-06-04T05:24:15.908 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 43326, 2024-06-04T05:24:15.908 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 35.855622093, 2024-06-04T05:24:15.908 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000827577 2024-06-04T05:24:15.909 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:15.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.909 INFO:teuthology.orchestra.run.smithi112.stdout: "finisher-PurgeQueue": { 2024-06-04T05:24:15.909 INFO:teuthology.orchestra.run.smithi112.stdout: "queue_len": 0, 2024-06-04T05:24:15.909 INFO:teuthology.orchestra.run.smithi112.stdout: "complete_latency": { 2024-06-04T05:24:15.909 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 21774, 2024-06-04T05:24:15.909 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 1.417474890, 2024-06-04T05:24:15.909 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000065099 2024-06-04T05:24:15.909 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:15.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.909 INFO:teuthology.orchestra.run.smithi112.stdout: "mds": { 2024-06-04T05:24:15.910 INFO:teuthology.orchestra.run.smithi112.stdout: "request": 346886, 2024-06-04T05:24:15.910 INFO:teuthology.orchestra.run.smithi112.stdout: "reply": 346258, 2024-06-04T05:24:15.910 INFO:teuthology.orchestra.run.smithi112.stdout: "reply_latency": { 2024-06-04T05:24:15.910 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 346258, 2024-06-04T05:24:15.910 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 707.166420917, 2024-06-04T05:24:15.910 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.002042310 2024-06-04T05:24:15.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.910 INFO:teuthology.orchestra.run.smithi112.stdout: "slow_reply": 0, 2024-06-04T05:24:15.910 INFO:teuthology.orchestra.run.smithi112.stdout: "forward": 628, 2024-06-04T05:24:15.910 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_fetch_complete": 12, 2024-06-04T05:24:15.910 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_fetch_keys": 0, 2024-06-04T05:24:15.910 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_commit": 71, 2024-06-04T05:24:15.910 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_split": 7, 2024-06-04T05:24:15.911 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_merge": 0, 2024-06-04T05:24:15.911 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes": 1849, 2024-06-04T05:24:15.911 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_top": 233, 2024-06-04T05:24:15.911 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_bottom": 1513, 2024-06-04T05:24:15.911 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_pin_tail": 103, 2024-06-04T05:24:15.911 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_pinned": 1513, 2024-06-04T05:24:15.911 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_expired": 1233, 2024-06-04T05:24:15.911 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_with_caps": 2, 2024-06-04T05:24:15.911 INFO:teuthology.orchestra.run.smithi112.stdout: "caps": 2, 2024-06-04T05:24:15.911 INFO:teuthology.orchestra.run.smithi112.stdout: "subtrees": 5, 2024-06-04T05:24:15.911 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse": 656078, 2024-06-04T05:24:15.911 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_hit": 631130, 2024-06-04T05:24:15.911 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_forward": 612, 2024-06-04T05:24:15.911 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_discover": 60, 2024-06-04T05:24:15.911 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_dir_fetch": 0, 2024-06-04T05:24:15.912 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_remote_ino": 0, 2024-06-04T05:24:15.912 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_lock": 3, 2024-06-04T05:24:15.912 INFO:teuthology.orchestra.run.smithi112.stdout: "load_cent": 0, 2024-06-04T05:24:15.912 INFO:teuthology.orchestra.run.smithi112.stdout: "q": 0, 2024-06-04T05:24:15.912 INFO:teuthology.orchestra.run.smithi112.stdout: "exported": 36, 2024-06-04T05:24:15.912 INFO:teuthology.orchestra.run.smithi112.stdout: "exported_inodes": 923, 2024-06-04T05:24:15.912 INFO:teuthology.orchestra.run.smithi112.stdout: "imported": 15, 2024-06-04T05:24:15.912 INFO:teuthology.orchestra.run.smithi112.stdout: "imported_inodes": 253, 2024-06-04T05:24:15.912 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_dir_fetch": 0, 2024-06-04T05:24:15.912 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_backtrace_fetch": 2198, 2024-06-04T05:24:15.912 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_peer_discover": 0, 2024-06-04T05:24:15.912 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rfiles": 4, 2024-06-04T05:24:15.912 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rbytes": 234, 2024-06-04T05:24:15.912 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rsnaps": 7, 2024-06-04T05:24:15.912 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_backtrace_fetch": 2198, 2024-06-04T05:24:15.913 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_set_tag": 0, 2024-06-04T05:24:15.913 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_backtrace_repaired": 0, 2024-06-04T05:24:15.913 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_inotable_repaired": 0, 2024-06-04T05:24:15.913 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dir_inodes": 190, 2024-06-04T05:24:15.913 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dir_base_inodes": 0, 2024-06-04T05:24:15.913 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dirfrag_rstats": 190, 2024-06-04T05:24:15.913 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_file_inodes": 2008, 2024-06-04T05:24:15.913 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_inode_file_caps": 0, 2024-06-04T05:24:15.913 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_revoke": 29526, 2024-06-04T05:24:15.913 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_grant": 43615, 2024-06-04T05:24:15.913 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_trunc": 0, 2024-06-04T05:24:15.913 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-04T05:24:15.913 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-04T05:24:15.913 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_caps": 47344, 2024-06-04T05:24:15.913 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_caps_dirty": 13489, 2024-06-04T05:24:15.914 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_cap_release": 921, 2024-06-04T05:24:15.914 INFO:teuthology.orchestra.run.smithi112.stdout: "process_request_cap_release": 34859 2024-06-04T05:24:15.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.914 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache": { 2024-06-04T05:24:15.914 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_update": 22, 2024-06-04T05:24:15.914 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_update_receipt": 408, 2024-06-04T05:24:15.914 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_try_discover": 212, 2024-06-04T05:24:15.914 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_send_discover": 446, 2024-06-04T05:24:15.914 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_handle_discover": 117, 2024-06-04T05:24:15.914 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays": 20, 2024-06-04T05:24:15.914 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays_delayed": 0, 2024-06-04T05:24:15.914 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays_enqueuing": 0, 2024-06-04T05:24:15.915 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_created": 9698, 2024-06-04T05:24:15.915 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_enqueued": 9678, 2024-06-04T05:24:15.915 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_reintegrated": 0, 2024-06-04T05:24:15.915 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_migrated": 0, 2024-06-04T05:24:15.915 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_processing": 0, 2024-06-04T05:24:15.915 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_enqueued": 0, 2024-06-04T05:24:15.915 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_prioritized": 0, 2024-06-04T05:24:15.915 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_started": 0, 2024-06-04T05:24:15.915 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_completed": 0, 2024-06-04T05:24:15.915 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_quiesce_path": 20, 2024-06-04T05:24:15.915 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_quiesce_inode": 2796, 2024-06-04T05:24:15.915 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_enqueue_scrub": 0, 2024-06-04T05:24:15.915 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_exportdir": 40, 2024-06-04T05:24:15.915 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_flush": 0, 2024-06-04T05:24:15.916 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_fragmentdir": 7, 2024-06-04T05:24:15.916 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_fragstats": 0, 2024-06-04T05:24:15.916 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_inodestats": 0 2024-06-04T05:24:15.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.916 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log": { 2024-06-04T05:24:15.916 INFO:teuthology.orchestra.run.smithi112.stdout: "evlrg": 0, 2024-06-04T05:24:15.916 INFO:teuthology.orchestra.run.smithi112.stdout: "evadd": 86003, 2024-06-04T05:24:15.916 INFO:teuthology.orchestra.run.smithi112.stdout: "evex": 0, 2024-06-04T05:24:15.916 INFO:teuthology.orchestra.run.smithi112.stdout: "evtrm": 0, 2024-06-04T05:24:15.916 INFO:teuthology.orchestra.run.smithi112.stdout: "ev": 86003, 2024-06-04T05:24:15.916 INFO:teuthology.orchestra.run.smithi112.stdout: "evexg": 0, 2024-06-04T05:24:15.916 INFO:teuthology.orchestra.run.smithi112.stdout: "evexd": 0, 2024-06-04T05:24:15.916 INFO:teuthology.orchestra.run.smithi112.stdout: "segadd": 118, 2024-06-04T05:24:15.917 INFO:teuthology.orchestra.run.smithi112.stdout: "segex": 0, 2024-06-04T05:24:15.917 INFO:teuthology.orchestra.run.smithi112.stdout: "segtrm": 0, 2024-06-04T05:24:15.917 INFO:teuthology.orchestra.run.smithi112.stdout: "seg": 118, 2024-06-04T05:24:15.917 INFO:teuthology.orchestra.run.smithi112.stdout: "segmjr": 8, 2024-06-04T05:24:15.917 INFO:teuthology.orchestra.run.smithi112.stdout: "segexg": 0, 2024-06-04T05:24:15.917 INFO:teuthology.orchestra.run.smithi112.stdout: "segexd": 0, 2024-06-04T05:24:15.917 INFO:teuthology.orchestra.run.smithi112.stdout: "expos": 4194304, 2024-06-04T05:24:15.917 INFO:teuthology.orchestra.run.smithi112.stdout: "wrpos": 297196869, 2024-06-04T05:24:15.917 INFO:teuthology.orchestra.run.smithi112.stdout: "rdpos": 0, 2024-06-04T05:24:15.917 INFO:teuthology.orchestra.run.smithi112.stdout: "jlat": { 2024-06-04T05:24:15.917 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 21348, 2024-06-04T05:24:15.917 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 70.678221417, 2024-06-04T05:24:15.917 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.003310765 2024-06-04T05:24:15.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.917 INFO:teuthology.orchestra.run.smithi112.stdout: "replayed": 0 2024-06-04T05:24:15.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.918 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_mem": { 2024-06-04T05:24:15.918 INFO:teuthology.orchestra.run.smithi112.stdout: "ino": 1557, 2024-06-04T05:24:15.918 INFO:teuthology.orchestra.run.smithi112.stdout: "ino+": 12292, 2024-06-04T05:24:15.918 INFO:teuthology.orchestra.run.smithi112.stdout: "ino-": 10735, 2024-06-04T05:24:15.918 INFO:teuthology.orchestra.run.smithi112.stdout: "dir": 75, 2024-06-04T05:24:15.918 INFO:teuthology.orchestra.run.smithi112.stdout: "dir+": 359, 2024-06-04T05:24:15.918 INFO:teuthology.orchestra.run.smithi112.stdout: "dir-": 284, 2024-06-04T05:24:15.918 INFO:teuthology.orchestra.run.smithi112.stdout: "dn": 1849, 2024-06-04T05:24:15.918 INFO:teuthology.orchestra.run.smithi112.stdout: "dn+": 15352, 2024-06-04T05:24:15.918 INFO:teuthology.orchestra.run.smithi112.stdout: "dn-": 13503, 2024-06-04T05:24:15.918 INFO:teuthology.orchestra.run.smithi112.stdout: "cap": 2, 2024-06-04T05:24:15.918 INFO:teuthology.orchestra.run.smithi112.stdout: "cap+": 9754, 2024-06-04T05:24:15.918 INFO:teuthology.orchestra.run.smithi112.stdout: "cap-": 9752, 2024-06-04T05:24:15.919 INFO:teuthology.orchestra.run.smithi112.stdout: "rss": 95476, 2024-06-04T05:24:15.919 INFO:teuthology.orchestra.run.smithi112.stdout: "heap": 223608 2024-06-04T05:24:15.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.919 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_server": { 2024-06-04T05:24:15.919 INFO:teuthology.orchestra.run.smithi112.stdout: "dispatch_client_request": 387093, 2024-06-04T05:24:15.919 INFO:teuthology.orchestra.run.smithi112.stdout: "dispatch_server_request": 477, 2024-06-04T05:24:15.919 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_request": 346886, 2024-06-04T05:24:15.919 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_session": 1052, 2024-06-04T05:24:15.919 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_peer_request": 8102, 2024-06-04T05:24:15.919 INFO:teuthology.orchestra.run.smithi112.stdout: "req_create_latency": { 2024-06-04T05:24:15.919 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 9468, 2024-06-04T05:24:15.919 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 347.058018382, 2024-06-04T05:24:15.919 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.036655895 2024-06-04T05:24:15.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.920 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getattr_latency": { 2024-06-04T05:24:15.920 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 279255, 2024-06-04T05:24:15.920 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 113.420357938, 2024-06-04T05:24:15.920 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000406153 2024-06-04T05:24:15.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.920 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getfilelock_latency": { 2024-06-04T05:24:15.920 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.920 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.920 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.920 INFO:teuthology.orchestra.run.smithi112.stdout: "req_link_latency": { 2024-06-04T05:24:15.920 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.920 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.920 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.920 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookup_latency": { 2024-06-04T05:24:15.921 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 5745, 2024-06-04T05:24:15.921 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 10.333024639, 2024-06-04T05:24:15.921 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.001798611 2024-06-04T05:24:15.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.921 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookuphash_latency": { 2024-06-04T05:24:15.921 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.921 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.921 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.921 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupino_latency": { 2024-06-04T05:24:15.921 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.921 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.921 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.922 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupname_latency": { 2024-06-04T05:24:15.922 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.922 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.922 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.922 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupparent_latency": { 2024-06-04T05:24:15.922 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.922 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.922 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.922 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupsnap_latency": { 2024-06-04T05:24:15.922 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.922 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.923 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.923 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lssnap_latency": { 2024-06-04T05:24:15.923 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.923 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.923 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.923 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mkdir_latency": { 2024-06-04T05:24:15.924 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.924 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.924 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.924 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mknod_latency": { 2024-06-04T05:24:15.924 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.924 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.924 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.924 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mksnap_latency": { 2024-06-04T05:24:15.924 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.924 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.924 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.925 INFO:teuthology.orchestra.run.smithi112.stdout: "req_open_latency": { 2024-06-04T05:24:15.925 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 7600, 2024-06-04T05:24:15.925 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 6.055264015, 2024-06-04T05:24:15.925 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000796745 2024-06-04T05:24:15.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.925 INFO:teuthology.orchestra.run.smithi112.stdout: "req_readdir_latency": { 2024-06-04T05:24:15.925 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 32031, 2024-06-04T05:24:15.925 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 68.355514970, 2024-06-04T05:24:15.925 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.002134042 2024-06-04T05:24:15.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.925 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rename_latency": { 2024-06-04T05:24:15.925 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 2007, 2024-06-04T05:24:15.925 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 81.626552033, 2024-06-04T05:24:15.926 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.040670927 2024-06-04T05:24:15.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.926 INFO:teuthology.orchestra.run.smithi112.stdout: "req_renamesnap_latency": { 2024-06-04T05:24:15.926 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.926 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.926 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.926 INFO:teuthology.orchestra.run.smithi112.stdout: "req_snapdiff_latency": { 2024-06-04T05:24:15.926 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.926 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.926 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.926 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmdir_latency": { 2024-06-04T05:24:15.927 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 20, 2024-06-04T05:24:15.927 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.129364508, 2024-06-04T05:24:15.927 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.006468225 2024-06-04T05:24:15.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.927 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmsnap_latency": { 2024-06-04T05:24:15.927 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.927 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.927 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.927 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmxattr_latency": { 2024-06-04T05:24:15.927 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.927 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.927 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.927 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setattr_latency": { 2024-06-04T05:24:15.928 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 2, 2024-06-04T05:24:15.928 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 43.060619828, 2024-06-04T05:24:15.928 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 21.530309914 2024-06-04T05:24:15.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.928 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setdirlayout_latency": { 2024-06-04T05:24:15.928 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.928 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.928 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.928 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setfilelock_latency": { 2024-06-04T05:24:15.928 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 452, 2024-06-04T05:24:15.928 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.167381924, 2024-06-04T05:24:15.928 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000370313 2024-06-04T05:24:15.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.929 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setlayout_latency": { 2024-06-04T05:24:15.929 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.929 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.929 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.929 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setxattr_latency": { 2024-06-04T05:24:15.929 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.929 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.929 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.929 INFO:teuthology.orchestra.run.smithi112.stdout: "req_symlink_latency": { 2024-06-04T05:24:15.929 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.929 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.930 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.930 INFO:teuthology.orchestra.run.smithi112.stdout: "req_unlink_latency": { 2024-06-04T05:24:15.930 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 9678, 2024-06-04T05:24:15.930 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 36.960322680, 2024-06-04T05:24:15.930 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.003819004 2024-06-04T05:24:15.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.930 INFO:teuthology.orchestra.run.smithi112.stdout: "cap_revoke_eviction": 0, 2024-06-04T05:24:15.930 INFO:teuthology.orchestra.run.smithi112.stdout: "cap_acquisition_throttle": 0, 2024-06-04T05:24:15.930 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getvxattr_latency": { 2024-06-04T05:24:15.930 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.930 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.930 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:15.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.931 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_sessions": { 2024-06-04T05:24:15.931 INFO:teuthology.orchestra.run.smithi112.stdout: "session_count": 2, 2024-06-04T05:24:15.931 INFO:teuthology.orchestra.run.smithi112.stdout: "session_add": 2, 2024-06-04T05:24:15.931 INFO:teuthology.orchestra.run.smithi112.stdout: "session_remove": 0, 2024-06-04T05:24:15.931 INFO:teuthology.orchestra.run.smithi112.stdout: "sessions_open": 2, 2024-06-04T05:24:15.931 INFO:teuthology.orchestra.run.smithi112.stdout: "sessions_stale": 0, 2024-06-04T05:24:15.931 INFO:teuthology.orchestra.run.smithi112.stdout: "total_load": 34536, 2024-06-04T05:24:15.931 INFO:teuthology.orchestra.run.smithi112.stdout: "average_load": 17268, 2024-06-04T05:24:15.931 INFO:teuthology.orchestra.run.smithi112.stdout: "avg_session_uptime": 1598, 2024-06-04T05:24:15.931 INFO:teuthology.orchestra.run.smithi112.stdout: "mdthresh_evicted": 0 2024-06-04T05:24:15.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.932 INFO:teuthology.orchestra.run.smithi112.stdout: "mempool": { 2024-06-04T05:24:15.932 INFO:teuthology.orchestra.run.smithi112.stdout: "bloom_filter_bytes": 0, 2024-06-04T05:24:15.932 INFO:teuthology.orchestra.run.smithi112.stdout: "bloom_filter_items": 0, 2024-06-04T05:24:15.932 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_alloc_bytes": 0, 2024-06-04T05:24:15.932 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_alloc_items": 0, 2024-06-04T05:24:15.932 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_data_bytes": 0, 2024-06-04T05:24:15.932 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_data_items": 0, 2024-06-04T05:24:15.932 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-04T05:24:15.932 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_onode_items": 0, 2024-06-04T05:24:15.932 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-04T05:24:15.932 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_meta_items": 0, 2024-06-04T05:24:15.932 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_other_bytes": 0, 2024-06-04T05:24:15.932 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_other_items": 0, 2024-06-04T05:24:15.932 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-04T05:24:15.933 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_buffer_items": 0, 2024-06-04T05:24:15.933 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_bytes": 0, 2024-06-04T05:24:15.933 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_items": 0, 2024-06-04T05:24:15.933 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_blob_bytes": 0, 2024-06-04T05:24:15.933 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_blob_items": 0, 2024-06-04T05:24:15.933 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-04T05:24:15.933 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_shared_blob_items": 0, 2024-06-04T05:24:15.933 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-04T05:24:15.933 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_inline_bl_items": 0, 2024-06-04T05:24:15.933 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_bytes": 0, 2024-06-04T05:24:15.933 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_items": 0, 2024-06-04T05:24:15.933 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_txc_bytes": 0, 2024-06-04T05:24:15.933 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_txc_items": 0, 2024-06-04T05:24:15.934 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-04T05:24:15.934 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_deferred_items": 0, 2024-06-04T05:24:15.934 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_bytes": 0, 2024-06-04T05:24:15.934 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_items": 0, 2024-06-04T05:24:15.934 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_bytes": 0, 2024-06-04T05:24:15.934 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_items": 0, 2024-06-04T05:24:15.934 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_reader_bytes": 0, 2024-06-04T05:24:15.934 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_reader_items": 0, 2024-06-04T05:24:15.934 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_writer_bytes": 0, 2024-06-04T05:24:15.934 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_writer_items": 0, 2024-06-04T05:24:15.934 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_anon_bytes": 457099, 2024-06-04T05:24:15.934 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_anon_items": 151, 2024-06-04T05:24:15.934 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_meta_bytes": 0, 2024-06-04T05:24:15.934 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_meta_items": 0, 2024-06-04T05:24:15.935 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_bytes": 0, 2024-06-04T05:24:15.935 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_items": 0, 2024-06-04T05:24:15.935 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mapbl_bytes": 0, 2024-06-04T05:24:15.935 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mapbl_items": 0, 2024-06-04T05:24:15.935 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pglog_bytes": 0, 2024-06-04T05:24:15.935 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pglog_items": 0, 2024-06-04T05:24:15.935 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_bytes": 8344, 2024-06-04T05:24:15.935 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_items": 174, 2024-06-04T05:24:15.935 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_mapping_bytes": 0, 2024-06-04T05:24:15.935 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_mapping_items": 0, 2024-06-04T05:24:15.935 INFO:teuthology.orchestra.run.smithi112.stdout: "pgmap_bytes": 0, 2024-06-04T05:24:15.935 INFO:teuthology.orchestra.run.smithi112.stdout: "pgmap_items": 0, 2024-06-04T05:24:15.935 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_co_bytes": 4961171, 2024-06-04T05:24:15.936 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_co_items": 20923, 2024-06-04T05:24:15.936 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_1_bytes": 0, 2024-06-04T05:24:15.936 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_1_items": 0, 2024-06-04T05:24:15.936 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_2_bytes": 0, 2024-06-04T05:24:15.936 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_2_items": 0 2024-06-04T05:24:15.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.936 INFO:teuthology.orchestra.run.smithi112.stdout: "objecter": { 2024-06-04T05:24:15.936 INFO:teuthology.orchestra.run.smithi112.stdout: "op_active": 0, 2024-06-04T05:24:15.936 INFO:teuthology.orchestra.run.smithi112.stdout: "op_laggy": 0, 2024-06-04T05:24:15.936 INFO:teuthology.orchestra.run.smithi112.stdout: "op_send": 38913, 2024-06-04T05:24:15.936 INFO:teuthology.orchestra.run.smithi112.stdout: "op_send_bytes": 295559152, 2024-06-04T05:24:15.936 INFO:teuthology.orchestra.run.smithi112.stdout: "op_resend": 33, 2024-06-04T05:24:15.936 INFO:teuthology.orchestra.run.smithi112.stdout: "op_reply": 38913, 2024-06-04T05:24:15.937 INFO:teuthology.orchestra.run.smithi112.stdout: "op_latency": { 2024-06-04T05:24:15.937 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 38913, 2024-06-04T05:24:15.937 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 112.013078069, 2024-06-04T05:24:15.937 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.002878551 2024-06-04T05:24:15.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.937 INFO:teuthology.orchestra.run.smithi112.stdout: "op_inflight": 0, 2024-06-04T05:24:15.937 INFO:teuthology.orchestra.run.smithi112.stdout: "oplen_avg": { 2024-06-04T05:24:15.937 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 38913, 2024-06-04T05:24:15.937 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 40798 2024-06-04T05:24:15.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.937 INFO:teuthology.orchestra.run.smithi112.stdout: "op": 38913, 2024-06-04T05:24:15.937 INFO:teuthology.orchestra.run.smithi112.stdout: "op_r": 3950, 2024-06-04T05:24:15.937 INFO:teuthology.orchestra.run.smithi112.stdout: "op_w": 34963, 2024-06-04T05:24:15.937 INFO:teuthology.orchestra.run.smithi112.stdout: "op_rmw": 0, 2024-06-04T05:24:15.938 INFO:teuthology.orchestra.run.smithi112.stdout: "op_pg": 0, 2024-06-04T05:24:15.938 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_stat": 0, 2024-06-04T05:24:15.938 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_create": 10, 2024-06-04T05:24:15.938 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_read": 1733, 2024-06-04T05:24:15.938 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_write": 23081, 2024-06-04T05:24:15.938 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_writefull": 1005, 2024-06-04T05:24:15.938 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_writesame": 0, 2024-06-04T05:24:15.938 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_append": 0, 2024-06-04T05:24:15.938 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_zero": 0, 2024-06-04T05:24:15.938 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_truncate": 7, 2024-06-04T05:24:15.938 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_delete": 9759, 2024-06-04T05:24:15.938 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_mapext": 0, 2024-06-04T05:24:15.938 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_sparse_read": 0, 2024-06-04T05:24:15.938 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_clonerange": 0, 2024-06-04T05:24:15.939 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_getxattr": 2198, 2024-06-04T05:24:15.939 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_setxattr": 20, 2024-06-04T05:24:15.939 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_cmpxattr": 0, 2024-06-04T05:24:15.939 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_rmxattr": 0, 2024-06-04T05:24:15.939 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_resetxattrs": 0, 2024-06-04T05:24:15.939 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_call": 0, 2024-06-04T05:24:15.939 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_watch": 0, 2024-06-04T05:24:15.939 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_notify": 0, 2024-06-04T05:24:15.939 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_src_cmpxattr": 0, 2024-06-04T05:24:15.939 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_pgls": 0, 2024-06-04T05:24:15.939 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_pgls_filter": 0, 2024-06-04T05:24:15.939 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_other": 0, 2024-06-04T05:24:15.939 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_active": 0, 2024-06-04T05:24:15.939 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_send": 0, 2024-06-04T05:24:15.940 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_resend": 0, 2024-06-04T05:24:15.940 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_ping": 0, 2024-06-04T05:24:15.940 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_active": 0, 2024-06-04T05:24:15.940 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_send": 0, 2024-06-04T05:24:15.940 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_resend": 0, 2024-06-04T05:24:15.940 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_active": 0, 2024-06-04T05:24:15.940 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_send": 0, 2024-06-04T05:24:15.940 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_resend": 0, 2024-06-04T05:24:15.940 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_active": 0, 2024-06-04T05:24:15.940 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_send": 0, 2024-06-04T05:24:15.940 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_resend": 0, 2024-06-04T05:24:15.940 INFO:teuthology.orchestra.run.smithi112.stdout: "command_active": 0, 2024-06-04T05:24:15.940 INFO:teuthology.orchestra.run.smithi112.stdout: "command_send": 0, 2024-06-04T05:24:15.940 INFO:teuthology.orchestra.run.smithi112.stdout: "command_resend": 0, 2024-06-04T05:24:15.941 INFO:teuthology.orchestra.run.smithi112.stdout: "map_epoch": 122, 2024-06-04T05:24:15.941 INFO:teuthology.orchestra.run.smithi112.stdout: "map_full": 0, 2024-06-04T05:24:15.941 INFO:teuthology.orchestra.run.smithi112.stdout: "map_inc": 43, 2024-06-04T05:24:15.941 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_sessions": 12, 2024-06-04T05:24:15.941 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_session_open": 12, 2024-06-04T05:24:15.941 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_session_close": 0, 2024-06-04T05:24:15.941 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_laggy": 0, 2024-06-04T05:24:15.941 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_wr": 2035, 2024-06-04T05:24:15.941 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_rd": 31, 2024-06-04T05:24:15.941 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_del": 919 2024-06-04T05:24:15.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.941 INFO:teuthology.orchestra.run.smithi112.stdout: "oft": { 2024-06-04T05:24:15.942 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_objs": 1, 2024-06-04T05:24:15.942 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_kv_pairs": 0, 2024-06-04T05:24:15.942 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_updates": 0, 2024-06-04T05:24:15.942 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_removes": 0 2024-06-04T05:24:15.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.942 INFO:teuthology.orchestra.run.smithi112.stdout: "osd-slow-ops": { 2024-06-04T05:24:15.942 INFO:teuthology.orchestra.run.smithi112.stdout: "slow_ops_count": 2823 2024-06-04T05:24:15.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.942 INFO:teuthology.orchestra.run.smithi112.stdout: "purge_queue": { 2024-06-04T05:24:15.942 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_ops": 0, 2024-06-04T05:24:15.942 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_ops_high_water": 6, 2024-06-04T05:24:15.942 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing": 0, 2024-06-04T05:24:15.942 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_high_water": 6, 2024-06-04T05:24:15.943 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executed_ops": 9678, 2024-06-04T05:24:15.943 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executed": 9678, 2024-06-04T05:24:15.943 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_item_in_journal": 0 2024-06-04T05:24:15.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.943 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-04T05:24:15.943 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:15.943 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 104857600, 2024-06-04T05:24:15.943 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:15.943 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 531679, 2024-06-04T05:24:15.943 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 129118769, 2024-06-04T05:24:15.943 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:15.943 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 531679, 2024-06-04T05:24:15.943 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-06-04T05:24:15.944 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-06-04T05:24:15.944 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 531679, 2024-06-04T05:24:15.944 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 129118769, 2024-06-04T05:24:15.944 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:15.944 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.944 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.944 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.944 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:15.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.944 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-objecter_bytes": { 2024-06-04T05:24:15.944 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:15.944 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 104857600, 2024-06-04T05:24:15.944 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:15.944 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-06-04T05:24:15.945 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-06-04T05:24:15.945 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:15.945 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:15.945 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 38913, 2024-06-04T05:24:15.945 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 296548135, 2024-06-04T05:24:15.945 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 38913, 2024-06-04T05:24:15.945 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 296548135, 2024-06-04T05:24:15.945 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:15.945 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.945 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.945 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.945 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:15.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.945 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-objecter_ops": { 2024-06-04T05:24:15.946 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:15.946 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 1024, 2024-06-04T05:24:15.946 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:15.946 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-06-04T05:24:15.946 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-06-04T05:24:15.946 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:15.946 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:15.946 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 38913, 2024-06-04T05:24:15.946 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 38913, 2024-06-04T05:24:15.946 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 38913, 2024-06-04T05:24:15.946 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 38913, 2024-06-04T05:24:15.946 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:15.946 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.946 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.947 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.947 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:15.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.947 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-write_buf_throttle": { 2024-06-04T05:24:15.947 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:15.947 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 3758096384, 2024-06-04T05:24:15.947 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:15.947 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 9678, 2024-06-04T05:24:15.947 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 989382, 2024-06-04T05:24:15.947 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:15.947 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 9678, 2024-06-04T05:24:15.947 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-06-04T05:24:15.947 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-06-04T05:24:15.947 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 1733, 2024-06-04T05:24:15.948 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 989382, 2024-06-04T05:24:15.948 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:15.948 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.948 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.948 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.948 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:15.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:15.948 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-write_buf_throttle-0x55de64b8e500": { 2024-06-04T05:24:15.948 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:15.948 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 3758096384, 2024-06-04T05:24:15.948 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:15.948 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 86003, 2024-06-04T05:24:15.948 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 293002565, 2024-06-04T05:24:15.948 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:15.949 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 86003, 2024-06-04T05:24:15.949 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-06-04T05:24:15.949 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-06-04T05:24:15.949 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 21348, 2024-06-04T05:24:15.949 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 293002565, 2024-06-04T05:24:15.949 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:15.949 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:15.949 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:15.949 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:15.949 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:15.949 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:15.949 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-06-04T05:24:15.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:15 smithi098 ceph-mon[35569]: pgmap v855: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 7 op/s 2024-06-04T05:24:16.687 INFO:tasks.check_counter:Daemon mds.c mds.exported=36 2024-06-04T05:24:16.687 INFO:tasks.check_counter:Daemon mds.c mds.imported=15 2024-06-04T05:24:16.687 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=7 2024-06-04T05:24:16.687 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-06-04T05:24:16.687 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-06-04T05:24:16.687 DEBUG:tasks.check_counter:Getting stats from f 2024-06-04T05:24:16.688 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph daemon mds.f perf dump 2024-06-04T05:24:16.925 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.c/config 2024-06-04T05:24:17.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:17 smithi070 ceph-mon[32631]: pgmap v856: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 15 op/s 2024-06-04T05:24:17.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:17 smithi112 ceph-mon[36453]: pgmap v856: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 15 op/s 2024-06-04T05:24:17.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:17 smithi098 ceph-mon[35569]: pgmap v856: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 15 op/s 2024-06-04T05:24:18.692 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-06-04T05:24:18.692 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-0": { 2024-06-04T05:24:18.692 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 370490, 2024-06-04T05:24:18.693 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 406714, 2024-06-04T05:24:18.693 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 112723188, 2024-06-04T05:24:18.693 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 295688024, 2024-06-04T05:24:18.693 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 16, 2024-06-04T05:24:18.693 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 4, 2024-06-04T05:24:18.693 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 18.816656959, 2024-06-04T05:24:18.693 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 7.717591540, 2024-06-04T05:24:18.693 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 13.590715566, 2024-06-04T05:24:18.693 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.196972262, 2024-06-04T05:24:18.693 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:18.693 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 406713, 2024-06-04T05:24:18.694 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 8.207345497, 2024-06-04T05:24:18.694 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000020179 2024-06-04T05:24:18.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.694 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:18.694 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 751697, 2024-06-04T05:24:18.694 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.119425985, 2024-06-04T05:24:18.694 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000158 2024-06-04T05:24:18.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.694 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 2012768, 2024-06-04T05:24:18.694 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 330176 2024-06-04T05:24:18.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.694 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-1": { 2024-06-04T05:24:18.695 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 21794, 2024-06-04T05:24:18.695 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 21909, 2024-06-04T05:24:18.695 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 6359213, 2024-06-04T05:24:18.695 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 108567470, 2024-06-04T05:24:18.695 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 14, 2024-06-04T05:24:18.695 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 6, 2024-06-04T05:24:18.695 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 1.691197505, 2024-06-04T05:24:18.695 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.618618083, 2024-06-04T05:24:18.695 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.647191463, 2024-06-04T05:24:18.695 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.464820923, 2024-06-04T05:24:18.695 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:18.695 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 21907, 2024-06-04T05:24:18.696 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.743488348, 2024-06-04T05:24:18.696 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000033938 2024-06-04T05:24:18.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.696 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:18.696 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 22999, 2024-06-04T05:24:18.696 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.003753197, 2024-06-04T05:24:18.696 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000163 2024-06-04T05:24:18.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.696 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-06-04T05:24:18.696 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 831872 2024-06-04T05:24:18.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.697 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-2": { 2024-06-04T05:24:18.697 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 31539, 2024-06-04T05:24:18.697 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 23313, 2024-06-04T05:24:18.697 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 11821269, 2024-06-04T05:24:18.697 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 54163657, 2024-06-04T05:24:18.697 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 63, 2024-06-04T05:24:18.697 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 6, 2024-06-04T05:24:18.697 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 1.962790263, 2024-06-04T05:24:18.697 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.870679145, 2024-06-04T05:24:18.697 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 1.168996373, 2024-06-04T05:24:18.698 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.156404382, 2024-06-04T05:24:18.698 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:18.698 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 23311, 2024-06-04T05:24:18.698 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.845761156, 2024-06-04T05:24:18.698 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000036281 2024-06-04T05:24:18.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.698 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:18.698 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 41683, 2024-06-04T05:24:18.698 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.008325273, 2024-06-04T05:24:18.698 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000199 2024-06-04T05:24:18.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.699 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 0, 2024-06-04T05:24:18.699 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 0 2024-06-04T05:24:18.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.699 INFO:teuthology.orchestra.run.smithi112.stdout: "cct": { 2024-06-04T05:24:18.699 INFO:teuthology.orchestra.run.smithi112.stdout: "total_workers": 1, 2024-06-04T05:24:18.699 INFO:teuthology.orchestra.run.smithi112.stdout: "unhealthy_workers": 0 2024-06-04T05:24:18.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.699 INFO:teuthology.orchestra.run.smithi112.stdout: "finisher-MDSRank": { 2024-06-04T05:24:18.699 INFO:teuthology.orchestra.run.smithi112.stdout: "queue_len": 0, 2024-06-04T05:24:18.699 INFO:teuthology.orchestra.run.smithi112.stdout: "complete_latency": { 2024-06-04T05:24:18.699 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 24314, 2024-06-04T05:24:18.699 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 24.228656854, 2024-06-04T05:24:18.699 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000996489 2024-06-04T05:24:18.699 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:18.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.700 INFO:teuthology.orchestra.run.smithi112.stdout: "finisher-PurgeQueue": { 2024-06-04T05:24:18.700 INFO:teuthology.orchestra.run.smithi112.stdout: "queue_len": 0, 2024-06-04T05:24:18.700 INFO:teuthology.orchestra.run.smithi112.stdout: "complete_latency": { 2024-06-04T05:24:18.700 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 12203, 2024-06-04T05:24:18.700 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 1.005521521, 2024-06-04T05:24:18.700 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000082399 2024-06-04T05:24:18.700 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:18.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.700 INFO:teuthology.orchestra.run.smithi112.stdout: "mds": { 2024-06-04T05:24:18.700 INFO:teuthology.orchestra.run.smithi112.stdout: "request": 332451, 2024-06-04T05:24:18.700 INFO:teuthology.orchestra.run.smithi112.stdout: "reply": 326889, 2024-06-04T05:24:18.700 INFO:teuthology.orchestra.run.smithi112.stdout: "reply_latency": { 2024-06-04T05:24:18.701 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 326889, 2024-06-04T05:24:18.701 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 501.460702554, 2024-06-04T05:24:18.701 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.001534039 2024-06-04T05:24:18.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.701 INFO:teuthology.orchestra.run.smithi112.stdout: "slow_reply": 0, 2024-06-04T05:24:18.701 INFO:teuthology.orchestra.run.smithi112.stdout: "forward": 5562, 2024-06-04T05:24:18.701 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_fetch_complete": 0, 2024-06-04T05:24:18.701 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_fetch_keys": 0, 2024-06-04T05:24:18.701 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_commit": 27, 2024-06-04T05:24:18.701 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_split": 2, 2024-06-04T05:24:18.701 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_merge": 0, 2024-06-04T05:24:18.701 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes": 1823, 2024-06-04T05:24:18.702 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_top": 236, 2024-06-04T05:24:18.702 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_bottom": 1534, 2024-06-04T05:24:18.702 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_pin_tail": 53, 2024-06-04T05:24:18.702 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_pinned": 1483, 2024-06-04T05:24:18.702 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_expired": 365, 2024-06-04T05:24:18.702 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_with_caps": 1, 2024-06-04T05:24:18.702 INFO:teuthology.orchestra.run.smithi112.stdout: "caps": 1, 2024-06-04T05:24:18.702 INFO:teuthology.orchestra.run.smithi112.stdout: "subtrees": 6, 2024-06-04T05:24:18.702 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse": 633865, 2024-06-04T05:24:18.702 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_hit": 632030, 2024-06-04T05:24:18.702 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_forward": 828, 2024-06-04T05:24:18.703 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_discover": 75, 2024-06-04T05:24:18.703 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_dir_fetch": 0, 2024-06-04T05:24:18.703 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_remote_ino": 0, 2024-06-04T05:24:18.703 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_lock": 0, 2024-06-04T05:24:18.703 INFO:teuthology.orchestra.run.smithi112.stdout: "load_cent": 0, 2024-06-04T05:24:18.703 INFO:teuthology.orchestra.run.smithi112.stdout: "q": 0, 2024-06-04T05:24:18.703 INFO:teuthology.orchestra.run.smithi112.stdout: "exported": 7, 2024-06-04T05:24:18.703 INFO:teuthology.orchestra.run.smithi112.stdout: "exported_inodes": 240, 2024-06-04T05:24:18.703 INFO:teuthology.orchestra.run.smithi112.stdout: "imported": 21, 2024-06-04T05:24:18.703 INFO:teuthology.orchestra.run.smithi112.stdout: "imported_inodes": 840, 2024-06-04T05:24:18.703 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_dir_fetch": 0, 2024-06-04T05:24:18.703 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_backtrace_fetch": 1142, 2024-06-04T05:24:18.703 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_peer_discover": 0, 2024-06-04T05:24:18.703 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rfiles": 4, 2024-06-04T05:24:18.704 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rbytes": 234, 2024-06-04T05:24:18.704 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rsnaps": 7, 2024-06-04T05:24:18.704 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_backtrace_fetch": 1142, 2024-06-04T05:24:18.704 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_set_tag": 0, 2024-06-04T05:24:18.704 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_backtrace_repaired": 0, 2024-06-04T05:24:18.704 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_inotable_repaired": 0, 2024-06-04T05:24:18.704 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dir_inodes": 189, 2024-06-04T05:24:18.704 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dir_base_inodes": 0, 2024-06-04T05:24:18.704 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dirfrag_rstats": 189, 2024-06-04T05:24:18.704 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_file_inodes": 953, 2024-06-04T05:24:18.704 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_inode_file_caps": 2, 2024-06-04T05:24:18.704 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_revoke": 20494, 2024-06-04T05:24:18.704 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_grant": 25990, 2024-06-04T05:24:18.705 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_trunc": 0, 2024-06-04T05:24:18.705 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-04T05:24:18.705 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-04T05:24:18.705 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_caps": 28968, 2024-06-04T05:24:18.705 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_caps_dirty": 6934, 2024-06-04T05:24:18.705 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_cap_release": 517, 2024-06-04T05:24:18.705 INFO:teuthology.orchestra.run.smithi112.stdout: "process_request_cap_release": 20242 2024-06-04T05:24:18.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.705 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache": { 2024-06-04T05:24:18.705 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_update": 4, 2024-06-04T05:24:18.705 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_update_receipt": 292, 2024-06-04T05:24:18.705 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_try_discover": 170, 2024-06-04T05:24:18.706 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_send_discover": 313, 2024-06-04T05:24:18.706 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_handle_discover": 207, 2024-06-04T05:24:18.706 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays": 21, 2024-06-04T05:24:18.706 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays_delayed": 0, 2024-06-04T05:24:18.706 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays_enqueuing": 0, 2024-06-04T05:24:18.706 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_created": 5640, 2024-06-04T05:24:18.706 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_enqueued": 5619, 2024-06-04T05:24:18.706 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_reintegrated": 0, 2024-06-04T05:24:18.706 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_migrated": 0, 2024-06-04T05:24:18.706 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_processing": 0, 2024-06-04T05:24:18.706 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_enqueued": 0, 2024-06-04T05:24:18.706 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_prioritized": 0, 2024-06-04T05:24:18.706 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_started": 0, 2024-06-04T05:24:18.707 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_completed": 0, 2024-06-04T05:24:18.707 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_quiesce_path": 20, 2024-06-04T05:24:18.707 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_quiesce_inode": 1487, 2024-06-04T05:24:18.707 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_enqueue_scrub": 0, 2024-06-04T05:24:18.707 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_exportdir": 7, 2024-06-04T05:24:18.707 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_flush": 0, 2024-06-04T05:24:18.707 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_fragmentdir": 2, 2024-06-04T05:24:18.707 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_fragstats": 0, 2024-06-04T05:24:18.707 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_inodestats": 0 2024-06-04T05:24:18.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.707 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log": { 2024-06-04T05:24:18.707 INFO:teuthology.orchestra.run.smithi112.stdout: "evlrg": 0, 2024-06-04T05:24:18.707 INFO:teuthology.orchestra.run.smithi112.stdout: "evadd": 36446, 2024-06-04T05:24:18.707 INFO:teuthology.orchestra.run.smithi112.stdout: "evex": 0, 2024-06-04T05:24:18.708 INFO:teuthology.orchestra.run.smithi112.stdout: "evtrm": 0, 2024-06-04T05:24:18.708 INFO:teuthology.orchestra.run.smithi112.stdout: "ev": 36446, 2024-06-04T05:24:18.708 INFO:teuthology.orchestra.run.smithi112.stdout: "evexg": 0, 2024-06-04T05:24:18.708 INFO:teuthology.orchestra.run.smithi112.stdout: "evexd": 0, 2024-06-04T05:24:18.708 INFO:teuthology.orchestra.run.smithi112.stdout: "segadd": 56, 2024-06-04T05:24:18.708 INFO:teuthology.orchestra.run.smithi112.stdout: "segex": 0, 2024-06-04T05:24:18.708 INFO:teuthology.orchestra.run.smithi112.stdout: "segtrm": 0, 2024-06-04T05:24:18.708 INFO:teuthology.orchestra.run.smithi112.stdout: "seg": 56, 2024-06-04T05:24:18.708 INFO:teuthology.orchestra.run.smithi112.stdout: "segmjr": 4, 2024-06-04T05:24:18.708 INFO:teuthology.orchestra.run.smithi112.stdout: "segexg": 0, 2024-06-04T05:24:18.708 INFO:teuthology.orchestra.run.smithi112.stdout: "segexd": 0, 2024-06-04T05:24:18.708 INFO:teuthology.orchestra.run.smithi112.stdout: "expos": 4194304, 2024-06-04T05:24:18.708 INFO:teuthology.orchestra.run.smithi112.stdout: "wrpos": 188487597, 2024-06-04T05:24:18.708 INFO:teuthology.orchestra.run.smithi112.stdout: "rdpos": 0, 2024-06-04T05:24:18.709 INFO:teuthology.orchestra.run.smithi112.stdout: "jlat": { 2024-06-04T05:24:18.709 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 12001, 2024-06-04T05:24:18.709 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 45.112792306, 2024-06-04T05:24:18.709 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.003759086 2024-06-04T05:24:18.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.709 INFO:teuthology.orchestra.run.smithi112.stdout: "replayed": 0 2024-06-04T05:24:18.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.709 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_mem": { 2024-06-04T05:24:18.709 INFO:teuthology.orchestra.run.smithi112.stdout: "ino": 1514, 2024-06-04T05:24:18.709 INFO:teuthology.orchestra.run.smithi112.stdout: "ino+": 7346, 2024-06-04T05:24:18.709 INFO:teuthology.orchestra.run.smithi112.stdout: "ino-": 5832, 2024-06-04T05:24:18.709 INFO:teuthology.orchestra.run.smithi112.stdout: "dir": 79, 2024-06-04T05:24:18.709 INFO:teuthology.orchestra.run.smithi112.stdout: "dir+": 240, 2024-06-04T05:24:18.709 INFO:teuthology.orchestra.run.smithi112.stdout: "dir-": 161, 2024-06-04T05:24:18.710 INFO:teuthology.orchestra.run.smithi112.stdout: "dn": 1823, 2024-06-04T05:24:18.710 INFO:teuthology.orchestra.run.smithi112.stdout: "dn+": 8117, 2024-06-04T05:24:18.710 INFO:teuthology.orchestra.run.smithi112.stdout: "dn-": 6294, 2024-06-04T05:24:18.710 INFO:teuthology.orchestra.run.smithi112.stdout: "cap": 1, 2024-06-04T05:24:18.710 INFO:teuthology.orchestra.run.smithi112.stdout: "cap+": 5776, 2024-06-04T05:24:18.710 INFO:teuthology.orchestra.run.smithi112.stdout: "cap-": 5775, 2024-06-04T05:24:18.710 INFO:teuthology.orchestra.run.smithi112.stdout: "rss": 72416, 2024-06-04T05:24:18.710 INFO:teuthology.orchestra.run.smithi112.stdout: "heap": 223608 2024-06-04T05:24:18.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.710 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_server": { 2024-06-04T05:24:18.710 INFO:teuthology.orchestra.run.smithi112.stdout: "dispatch_client_request": 346879, 2024-06-04T05:24:18.710 INFO:teuthology.orchestra.run.smithi112.stdout: "dispatch_server_request": 2472, 2024-06-04T05:24:18.711 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_request": 332451, 2024-06-04T05:24:18.711 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_session": 639, 2024-06-04T05:24:18.711 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_peer_request": 5161, 2024-06-04T05:24:18.711 INFO:teuthology.orchestra.run.smithi112.stdout: "req_create_latency": { 2024-06-04T05:24:18.711 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 5530, 2024-06-04T05:24:18.711 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 217.943676325, 2024-06-04T05:24:18.711 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.039411153 2024-06-04T05:24:18.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.711 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getattr_latency": { 2024-06-04T05:24:18.711 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 297011, 2024-06-04T05:24:18.711 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 113.902242613, 2024-06-04T05:24:18.711 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000383495 2024-06-04T05:24:18.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.711 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getfilelock_latency": { 2024-06-04T05:24:18.712 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.712 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.712 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.712 INFO:teuthology.orchestra.run.smithi112.stdout: "req_link_latency": { 2024-06-04T05:24:18.712 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.712 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.712 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.712 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookup_latency": { 2024-06-04T05:24:18.712 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 2635, 2024-06-04T05:24:18.712 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 22.479604198, 2024-06-04T05:24:18.712 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.008531159 2024-06-04T05:24:18.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.713 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookuphash_latency": { 2024-06-04T05:24:18.713 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.713 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.713 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.713 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupino_latency": { 2024-06-04T05:24:18.713 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.713 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.713 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.713 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupname_latency": { 2024-06-04T05:24:18.713 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.714 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.714 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.714 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupparent_latency": { 2024-06-04T05:24:18.714 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.714 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.714 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.714 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupsnap_latency": { 2024-06-04T05:24:18.714 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.714 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.715 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.715 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lssnap_latency": { 2024-06-04T05:24:18.715 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.715 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.715 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.716 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mkdir_latency": { 2024-06-04T05:24:18.716 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.716 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.716 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.716 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mknod_latency": { 2024-06-04T05:24:18.716 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.716 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.716 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.716 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mksnap_latency": { 2024-06-04T05:24:18.716 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.717 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.717 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.717 INFO:teuthology.orchestra.run.smithi112.stdout: "req_open_latency": { 2024-06-04T05:24:18.717 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 3266, 2024-06-04T05:24:18.717 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 7.572807131, 2024-06-04T05:24:18.717 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.002318679 2024-06-04T05:24:18.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.717 INFO:teuthology.orchestra.run.smithi112.stdout: "req_readdir_latency": { 2024-06-04T05:24:18.717 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 11541, 2024-06-04T05:24:18.717 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 42.553036894, 2024-06-04T05:24:18.717 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.003687118 2024-06-04T05:24:18.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.718 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rename_latency": { 2024-06-04T05:24:18.718 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 1144, 2024-06-04T05:24:18.718 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 8.121012702, 2024-06-04T05:24:18.718 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.007098787 2024-06-04T05:24:18.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.718 INFO:teuthology.orchestra.run.smithi112.stdout: "req_renamesnap_latency": { 2024-06-04T05:24:18.718 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.718 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.718 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.718 INFO:teuthology.orchestra.run.smithi112.stdout: "req_snapdiff_latency": { 2024-06-04T05:24:18.718 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.719 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.719 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.719 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmdir_latency": { 2024-06-04T05:24:18.719 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 21, 2024-06-04T05:24:18.719 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.137519339, 2024-06-04T05:24:18.719 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.006548539 2024-06-04T05:24:18.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.719 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmsnap_latency": { 2024-06-04T05:24:18.719 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.719 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.719 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.720 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmxattr_latency": { 2024-06-04T05:24:18.720 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.720 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.720 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.720 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setattr_latency": { 2024-06-04T05:24:18.720 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 2, 2024-06-04T05:24:18.720 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 38.916279556, 2024-06-04T05:24:18.720 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 19.458139778 2024-06-04T05:24:18.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.720 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setdirlayout_latency": { 2024-06-04T05:24:18.720 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.720 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.720 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.721 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setfilelock_latency": { 2024-06-04T05:24:18.721 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 120, 2024-06-04T05:24:18.721 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.050463700, 2024-06-04T05:24:18.721 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000420530 2024-06-04T05:24:18.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.721 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setlayout_latency": { 2024-06-04T05:24:18.721 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.721 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.721 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.721 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setxattr_latency": { 2024-06-04T05:24:18.721 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.722 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.722 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.722 INFO:teuthology.orchestra.run.smithi112.stdout: "req_symlink_latency": { 2024-06-04T05:24:18.722 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.722 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.722 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.722 INFO:teuthology.orchestra.run.smithi112.stdout: "req_unlink_latency": { 2024-06-04T05:24:18.722 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 5619, 2024-06-04T05:24:18.722 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 49.784060096, 2024-06-04T05:24:18.722 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.008859950 2024-06-04T05:24:18.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.723 INFO:teuthology.orchestra.run.smithi112.stdout: "cap_revoke_eviction": 0, 2024-06-04T05:24:18.723 INFO:teuthology.orchestra.run.smithi112.stdout: "cap_acquisition_throttle": 0, 2024-06-04T05:24:18.723 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getvxattr_latency": { 2024-06-04T05:24:18.723 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.723 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.723 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.723 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:18.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.723 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_sessions": { 2024-06-04T05:24:18.723 INFO:teuthology.orchestra.run.smithi112.stdout: "session_count": 1, 2024-06-04T05:24:18.723 INFO:teuthology.orchestra.run.smithi112.stdout: "session_add": 1, 2024-06-04T05:24:18.723 INFO:teuthology.orchestra.run.smithi112.stdout: "session_remove": 0, 2024-06-04T05:24:18.723 INFO:teuthology.orchestra.run.smithi112.stdout: "sessions_open": 1, 2024-06-04T05:24:18.723 INFO:teuthology.orchestra.run.smithi112.stdout: "sessions_stale": 0, 2024-06-04T05:24:18.724 INFO:teuthology.orchestra.run.smithi112.stdout: "total_load": 33325, 2024-06-04T05:24:18.724 INFO:teuthology.orchestra.run.smithi112.stdout: "average_load": 33325, 2024-06-04T05:24:18.724 INFO:teuthology.orchestra.run.smithi112.stdout: "avg_session_uptime": 1627, 2024-06-04T05:24:18.724 INFO:teuthology.orchestra.run.smithi112.stdout: "mdthresh_evicted": 0 2024-06-04T05:24:18.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.724 INFO:teuthology.orchestra.run.smithi112.stdout: "mempool": { 2024-06-04T05:24:18.724 INFO:teuthology.orchestra.run.smithi112.stdout: "bloom_filter_bytes": 0, 2024-06-04T05:24:18.724 INFO:teuthology.orchestra.run.smithi112.stdout: "bloom_filter_items": 0, 2024-06-04T05:24:18.724 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_alloc_bytes": 0, 2024-06-04T05:24:18.724 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_alloc_items": 0, 2024-06-04T05:24:18.724 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_data_bytes": 0, 2024-06-04T05:24:18.724 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_data_items": 0, 2024-06-04T05:24:18.725 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-04T05:24:18.725 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_onode_items": 0, 2024-06-04T05:24:18.725 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-04T05:24:18.725 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_meta_items": 0, 2024-06-04T05:24:18.725 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_other_bytes": 0, 2024-06-04T05:24:18.725 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_other_items": 0, 2024-06-04T05:24:18.725 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-04T05:24:18.725 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_buffer_items": 0, 2024-06-04T05:24:18.725 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_bytes": 0, 2024-06-04T05:24:18.725 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_items": 0, 2024-06-04T05:24:18.725 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_blob_bytes": 0, 2024-06-04T05:24:18.725 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_blob_items": 0, 2024-06-04T05:24:18.725 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-04T05:24:18.726 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_shared_blob_items": 0, 2024-06-04T05:24:18.726 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-04T05:24:18.726 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_inline_bl_items": 0, 2024-06-04T05:24:18.726 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_bytes": 0, 2024-06-04T05:24:18.726 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_items": 0, 2024-06-04T05:24:18.726 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_txc_bytes": 0, 2024-06-04T05:24:18.726 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_txc_items": 0, 2024-06-04T05:24:18.726 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-04T05:24:18.726 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_deferred_items": 0, 2024-06-04T05:24:18.726 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_bytes": 0, 2024-06-04T05:24:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_items": 0, 2024-06-04T05:24:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_bytes": 0, 2024-06-04T05:24:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_items": 0, 2024-06-04T05:24:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_reader_bytes": 0, 2024-06-04T05:24:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_reader_items": 0, 2024-06-04T05:24:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_writer_bytes": 0, 2024-06-04T05:24:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_writer_items": 0, 2024-06-04T05:24:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_anon_bytes": 481525, 2024-06-04T05:24:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_anon_items": 156, 2024-06-04T05:24:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_meta_bytes": 0, 2024-06-04T05:24:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_meta_items": 0, 2024-06-04T05:24:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_bytes": 0, 2024-06-04T05:24:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_items": 0, 2024-06-04T05:24:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mapbl_bytes": 0, 2024-06-04T05:24:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mapbl_items": 0, 2024-06-04T05:24:18.728 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pglog_bytes": 0, 2024-06-04T05:24:18.728 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pglog_items": 0, 2024-06-04T05:24:18.728 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_bytes": 8224, 2024-06-04T05:24:18.728 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_items": 171, 2024-06-04T05:24:18.728 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_mapping_bytes": 0, 2024-06-04T05:24:18.728 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_mapping_items": 0, 2024-06-04T05:24:18.728 INFO:teuthology.orchestra.run.smithi112.stdout: "pgmap_bytes": 0, 2024-06-04T05:24:18.728 INFO:teuthology.orchestra.run.smithi112.stdout: "pgmap_items": 0, 2024-06-04T05:24:18.728 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_co_bytes": 4850933, 2024-06-04T05:24:18.728 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_co_items": 19743, 2024-06-04T05:24:18.728 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_1_bytes": 0, 2024-06-04T05:24:18.728 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_1_items": 0, 2024-06-04T05:24:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_2_bytes": 0, 2024-06-04T05:24:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_2_items": 0 2024-06-04T05:24:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: "objecter": { 2024-06-04T05:24:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: "op_active": 0, 2024-06-04T05:24:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: "op_laggy": 0, 2024-06-04T05:24:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: "op_send": 21894, 2024-06-04T05:24:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: "op_send_bytes": 185389614, 2024-06-04T05:24:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: "op_resend": 33, 2024-06-04T05:24:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: "op_reply": 21894, 2024-06-04T05:24:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: "op_latency": { 2024-06-04T05:24:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 21894, 2024-06-04T05:24:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 69.017203233, 2024-06-04T05:24:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.003152334 2024-06-04T05:24:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: "op_inflight": 0, 2024-06-04T05:24:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: "oplen_avg": { 2024-06-04T05:24:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 21894, 2024-06-04T05:24:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 22985 2024-06-04T05:24:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: "op": 21894, 2024-06-04T05:24:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: "op_r": 2109, 2024-06-04T05:24:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: "op_w": 19785, 2024-06-04T05:24:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: "op_rmw": 0, 2024-06-04T05:24:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: "op_pg": 0, 2024-06-04T05:24:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_stat": 0, 2024-06-04T05:24:18.731 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_create": 10, 2024-06-04T05:24:18.731 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_read": 967, 2024-06-04T05:24:18.731 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_write": 12968, 2024-06-04T05:24:18.731 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_writefull": 549, 2024-06-04T05:24:18.731 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_writesame": 0, 2024-06-04T05:24:18.731 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_append": 0, 2024-06-04T05:24:18.731 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_zero": 0, 2024-06-04T05:24:18.731 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_truncate": 2, 2024-06-04T05:24:18.731 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_delete": 5674, 2024-06-04T05:24:18.731 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_mapext": 0, 2024-06-04T05:24:18.731 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_sparse_read": 0, 2024-06-04T05:24:18.731 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_clonerange": 0, 2024-06-04T05:24:18.731 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_getxattr": 1142, 2024-06-04T05:24:18.731 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_setxattr": 20, 2024-06-04T05:24:18.731 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_cmpxattr": 0, 2024-06-04T05:24:18.732 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_rmxattr": 0, 2024-06-04T05:24:18.732 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_resetxattrs": 0, 2024-06-04T05:24:18.732 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_call": 0, 2024-06-04T05:24:18.732 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_watch": 0, 2024-06-04T05:24:18.732 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_notify": 0, 2024-06-04T05:24:18.732 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_src_cmpxattr": 0, 2024-06-04T05:24:18.732 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_pgls": 0, 2024-06-04T05:24:18.732 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_pgls_filter": 0, 2024-06-04T05:24:18.732 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_other": 0, 2024-06-04T05:24:18.732 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_active": 0, 2024-06-04T05:24:18.732 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_send": 0, 2024-06-04T05:24:18.732 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_resend": 0, 2024-06-04T05:24:18.732 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_ping": 0, 2024-06-04T05:24:18.732 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_active": 0, 2024-06-04T05:24:18.732 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_send": 0, 2024-06-04T05:24:18.733 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_resend": 0, 2024-06-04T05:24:18.733 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_active": 0, 2024-06-04T05:24:18.733 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_send": 0, 2024-06-04T05:24:18.733 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_resend": 0, 2024-06-04T05:24:18.733 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_active": 0, 2024-06-04T05:24:18.733 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_send": 0, 2024-06-04T05:24:18.733 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_resend": 0, 2024-06-04T05:24:18.733 INFO:teuthology.orchestra.run.smithi112.stdout: "command_active": 0, 2024-06-04T05:24:18.733 INFO:teuthology.orchestra.run.smithi112.stdout: "command_send": 0, 2024-06-04T05:24:18.733 INFO:teuthology.orchestra.run.smithi112.stdout: "command_resend": 0, 2024-06-04T05:24:18.733 INFO:teuthology.orchestra.run.smithi112.stdout: "map_epoch": 122, 2024-06-04T05:24:18.733 INFO:teuthology.orchestra.run.smithi112.stdout: "map_full": 0, 2024-06-04T05:24:18.733 INFO:teuthology.orchestra.run.smithi112.stdout: "map_inc": 43, 2024-06-04T05:24:18.733 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_sessions": 12, 2024-06-04T05:24:18.733 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_session_open": 17, 2024-06-04T05:24:18.734 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_session_close": 5, 2024-06-04T05:24:18.734 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_laggy": 0, 2024-06-04T05:24:18.734 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_wr": 1124, 2024-06-04T05:24:18.734 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_rd": 0, 2024-06-04T05:24:18.734 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_del": 529 2024-06-04T05:24:18.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.734 INFO:teuthology.orchestra.run.smithi112.stdout: "oft": { 2024-06-04T05:24:18.734 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_objs": 1, 2024-06-04T05:24:18.734 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_kv_pairs": 0, 2024-06-04T05:24:18.734 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_updates": 0, 2024-06-04T05:24:18.734 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_removes": 0 2024-06-04T05:24:18.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.734 INFO:teuthology.orchestra.run.smithi112.stdout: "osd-slow-ops": { 2024-06-04T05:24:18.734 INFO:teuthology.orchestra.run.smithi112.stdout: "slow_ops_count": 1615 2024-06-04T05:24:18.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.735 INFO:teuthology.orchestra.run.smithi112.stdout: "purge_queue": { 2024-06-04T05:24:18.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_ops": 0, 2024-06-04T05:24:18.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_ops_high_water": 6, 2024-06-04T05:24:18.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing": 0, 2024-06-04T05:24:18.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_high_water": 6, 2024-06-04T05:24:18.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executed_ops": 5619, 2024-06-04T05:24:18.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executed": 5619, 2024-06-04T05:24:18.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_item_in_journal": 0 2024-06-04T05:24:18.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.735 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-04T05:24:18.735 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:18.735 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 104857600, 2024-06-04T05:24:18.736 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:18.736 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 423819, 2024-06-04T05:24:18.736 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 92724084, 2024-06-04T05:24:18.736 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:18.736 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 423819, 2024-06-04T05:24:18.736 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-06-04T05:24:18.736 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-06-04T05:24:18.736 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 423819, 2024-06-04T05:24:18.736 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 92724084, 2024-06-04T05:24:18.736 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:18.736 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.736 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.736 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:18.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.737 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-objecter_bytes": { 2024-06-04T05:24:18.737 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:18.737 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 104857600, 2024-06-04T05:24:18.737 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:18.737 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-06-04T05:24:18.737 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-06-04T05:24:18.737 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:18.737 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:18.737 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 21894, 2024-06-04T05:24:18.737 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 185962653, 2024-06-04T05:24:18.737 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 21894, 2024-06-04T05:24:18.737 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 185962653, 2024-06-04T05:24:18.737 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:18.737 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.738 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.738 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.738 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:18.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.738 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-objecter_ops": { 2024-06-04T05:24:18.738 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:18.738 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 1024, 2024-06-04T05:24:18.738 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:18.738 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-06-04T05:24:18.738 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-06-04T05:24:18.738 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:18.739 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:18.739 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 21894, 2024-06-04T05:24:18.739 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 21894, 2024-06-04T05:24:18.739 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 21894, 2024-06-04T05:24:18.739 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 21894, 2024-06-04T05:24:18.739 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:18.739 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.739 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.739 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.739 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:18.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.739 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-write_buf_throttle": { 2024-06-04T05:24:18.739 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:18.740 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 3758096384, 2024-06-04T05:24:18.740 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:18.740 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 5619, 2024-06-04T05:24:18.740 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 573039, 2024-06-04T05:24:18.740 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:18.740 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 5619, 2024-06-04T05:24:18.740 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-06-04T05:24:18.740 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-06-04T05:24:18.740 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 967, 2024-06-04T05:24:18.740 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 573039, 2024-06-04T05:24:18.740 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:18.740 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.740 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.740 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.741 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:18.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:18.741 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-write_buf_throttle-0x55f03783c500": { 2024-06-04T05:24:18.741 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:18.741 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 3758096384, 2024-06-04T05:24:18.741 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:18.741 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 36446, 2024-06-04T05:24:18.741 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 184293293, 2024-06-04T05:24:18.741 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:18.741 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 36446, 2024-06-04T05:24:18.741 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-06-04T05:24:18.741 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-06-04T05:24:18.742 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 12001, 2024-06-04T05:24:18.742 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 184293293, 2024-06-04T05:24:18.742 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:18.742 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:18.742 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:18.742 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:18.742 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:18.742 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:18.742 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-06-04T05:24:19.724 INFO:tasks.check_counter:Daemon mds.f mds.exported=7 2024-06-04T05:24:19.725 INFO:tasks.check_counter:Daemon mds.f mds.imported=21 2024-06-04T05:24:19.725 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=7 2024-06-04T05:24:19.725 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-06-04T05:24:19.725 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-06-04T05:24:19.725 DEBUG:tasks.check_counter:Getting stats from i 2024-06-04T05:24:19.725 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph daemon mds.i perf dump 2024-06-04T05:24:19.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:19 smithi070 ceph-mon[32631]: pgmap v857: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 12 op/s 2024-06-04T05:24:19.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:19 smithi112 ceph-mon[36453]: pgmap v857: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 12 op/s 2024-06-04T05:24:19.965 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.c/config 2024-06-04T05:24:19.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:19 smithi098 ceph-mon[35569]: pgmap v857: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 12 op/s 2024-06-04T05:24:21.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:21 smithi112 ceph-mon[36453]: pgmap v858: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 12 op/s 2024-06-04T05:24:21.703 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-06-04T05:24:21.703 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-0": { 2024-06-04T05:24:21.703 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 909078, 2024-06-04T05:24:21.703 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 965283, 2024-06-04T05:24:21.703 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 265152838, 2024-06-04T05:24:21.703 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 667008376, 2024-06-04T05:24:21.704 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 26, 2024-06-04T05:24:21.704 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 4, 2024-06-04T05:24:21.704 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 40.097991204, 2024-06-04T05:24:21.704 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 16.321619370, 2024-06-04T05:24:21.704 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 28.298965894, 2024-06-04T05:24:21.704 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 1.135678100, 2024-06-04T05:24:21.704 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:21.704 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 965281, 2024-06-04T05:24:21.704 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 22.020276379, 2024-06-04T05:24:21.704 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000022812 2024-06-04T05:24:21.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.704 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:21.704 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 1818917, 2024-06-04T05:24:21.705 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.225603689, 2024-06-04T05:24:21.705 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000124 2024-06-04T05:24:21.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.705 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 913440, 2024-06-04T05:24:21.705 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 338608 2024-06-04T05:24:21.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.705 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-1": { 2024-06-04T05:24:21.705 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 60572, 2024-06-04T05:24:21.705 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 69256, 2024-06-04T05:24:21.705 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 19636224, 2024-06-04T05:24:21.705 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 148529420, 2024-06-04T05:24:21.705 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 29, 2024-06-04T05:24:21.705 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 8, 2024-06-04T05:24:21.705 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 4.409343369, 2024-06-04T05:24:21.706 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 1.833306735, 2024-06-04T05:24:21.706 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 1.897036542, 2024-06-04T05:24:21.706 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.771927382, 2024-06-04T05:24:21.706 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:21.706 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 69252, 2024-06-04T05:24:21.706 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 9.993991211, 2024-06-04T05:24:21.706 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000144313 2024-06-04T05:24:21.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.706 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:21.706 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 80044, 2024-06-04T05:24:21.706 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.014739315, 2024-06-04T05:24:21.706 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000184 2024-06-04T05:24:21.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.707 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 2035840, 2024-06-04T05:24:21.707 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 364752 2024-06-04T05:24:21.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.707 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-2": { 2024-06-04T05:24:21.707 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 91026, 2024-06-04T05:24:21.707 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 120515, 2024-06-04T05:24:21.707 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 24249907, 2024-06-04T05:24:21.707 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 120749245, 2024-06-04T05:24:21.707 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 495, 2024-06-04T05:24:21.707 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 9, 2024-06-04T05:24:21.707 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 6.727756150, 2024-06-04T05:24:21.707 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 3.050545005, 2024-06-04T05:24:21.707 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 3.739744147, 2024-06-04T05:24:21.708 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.456721036, 2024-06-04T05:24:21.708 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:21.708 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 120516, 2024-06-04T05:24:21.708 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 6.282924209, 2024-06-04T05:24:21.708 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000052133 2024-06-04T05:24:21.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.708 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:21.708 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 184435, 2024-06-04T05:24:21.708 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.030807014, 2024-06-04T05:24:21.708 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000167 2024-06-04T05:24:21.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.708 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 45776, 2024-06-04T05:24:21.708 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 845104 2024-06-04T05:24:21.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.709 INFO:teuthology.orchestra.run.smithi112.stdout: "cct": { 2024-06-04T05:24:21.709 INFO:teuthology.orchestra.run.smithi112.stdout: "total_workers": 1, 2024-06-04T05:24:21.709 INFO:teuthology.orchestra.run.smithi112.stdout: "unhealthy_workers": 0 2024-06-04T05:24:21.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.709 INFO:teuthology.orchestra.run.smithi112.stdout: "finisher-MDSRank": { 2024-06-04T05:24:21.709 INFO:teuthology.orchestra.run.smithi112.stdout: "queue_len": 0, 2024-06-04T05:24:21.709 INFO:teuthology.orchestra.run.smithi112.stdout: "complete_latency": { 2024-06-04T05:24:21.709 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 73087, 2024-06-04T05:24:21.709 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 46.179138673, 2024-06-04T05:24:21.709 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000631837 2024-06-04T05:24:21.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:21.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.709 INFO:teuthology.orchestra.run.smithi112.stdout: "finisher-PurgeQueue": { 2024-06-04T05:24:21.710 INFO:teuthology.orchestra.run.smithi112.stdout: "queue_len": 0, 2024-06-04T05:24:21.710 INFO:teuthology.orchestra.run.smithi112.stdout: "complete_latency": { 2024-06-04T05:24:21.710 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 12997, 2024-06-04T05:24:21.710 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.925916458, 2024-06-04T05:24:21.710 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000071240 2024-06-04T05:24:21.710 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:21.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.710 INFO:teuthology.orchestra.run.smithi112.stdout: "mds": { 2024-06-04T05:24:21.710 INFO:teuthology.orchestra.run.smithi112.stdout: "request": 725011, 2024-06-04T05:24:21.710 INFO:teuthology.orchestra.run.smithi112.stdout: "reply": 691815, 2024-06-04T05:24:21.710 INFO:teuthology.orchestra.run.smithi112.stdout: "reply_latency": { 2024-06-04T05:24:21.710 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 691815, 2024-06-04T05:24:21.710 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 831.874199254, 2024-06-04T05:24:21.711 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.001202451 2024-06-04T05:24:21.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.711 INFO:teuthology.orchestra.run.smithi112.stdout: "slow_reply": 0, 2024-06-04T05:24:21.711 INFO:teuthology.orchestra.run.smithi112.stdout: "forward": 33196, 2024-06-04T05:24:21.711 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_fetch_complete": 24, 2024-06-04T05:24:21.711 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_fetch_keys": 5, 2024-06-04T05:24:21.711 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_commit": 3861, 2024-06-04T05:24:21.711 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_split": 6, 2024-06-04T05:24:21.711 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_merge": 3, 2024-06-04T05:24:21.711 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes": 1580, 2024-06-04T05:24:21.711 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_top": 939, 2024-06-04T05:24:21.711 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_bottom": 606, 2024-06-04T05:24:21.711 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_pin_tail": 35, 2024-06-04T05:24:21.711 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_pinned": 39, 2024-06-04T05:24:21.711 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_expired": 2897, 2024-06-04T05:24:21.712 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_with_caps": 3, 2024-06-04T05:24:21.712 INFO:teuthology.orchestra.run.smithi112.stdout: "caps": 3, 2024-06-04T05:24:21.712 INFO:teuthology.orchestra.run.smithi112.stdout: "subtrees": 6, 2024-06-04T05:24:21.712 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse": 1399237, 2024-06-04T05:24:21.712 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_hit": 1354262, 2024-06-04T05:24:21.712 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_forward": 13136, 2024-06-04T05:24:21.712 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_discover": 157, 2024-06-04T05:24:21.712 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_dir_fetch": 2, 2024-06-04T05:24:21.712 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_remote_ino": 0, 2024-06-04T05:24:21.712 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_lock": 28, 2024-06-04T05:24:21.712 INFO:teuthology.orchestra.run.smithi112.stdout: "load_cent": 156, 2024-06-04T05:24:21.712 INFO:teuthology.orchestra.run.smithi112.stdout: "q": 0, 2024-06-04T05:24:21.712 INFO:teuthology.orchestra.run.smithi112.stdout: "exported": 61, 2024-06-04T05:24:21.712 INFO:teuthology.orchestra.run.smithi112.stdout: "exported_inodes": 2241, 2024-06-04T05:24:21.712 INFO:teuthology.orchestra.run.smithi112.stdout: "imported": 67, 2024-06-04T05:24:21.713 INFO:teuthology.orchestra.run.smithi112.stdout: "imported_inodes": 1479, 2024-06-04T05:24:21.713 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_dir_fetch": 0, 2024-06-04T05:24:21.713 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_backtrace_fetch": 2063, 2024-06-04T05:24:21.713 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_peer_discover": 0, 2024-06-04T05:24:21.713 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rfiles": 4, 2024-06-04T05:24:21.713 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rbytes": 234, 2024-06-04T05:24:21.713 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rsnaps": 7, 2024-06-04T05:24:21.713 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_backtrace_fetch": 2063, 2024-06-04T05:24:21.713 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_set_tag": 0, 2024-06-04T05:24:21.713 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_backtrace_repaired": 0, 2024-06-04T05:24:21.713 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_inotable_repaired": 0, 2024-06-04T05:24:21.713 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dir_inodes": 551, 2024-06-04T05:24:21.713 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dir_base_inodes": 25, 2024-06-04T05:24:21.713 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dirfrag_rstats": 526, 2024-06-04T05:24:21.713 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_file_inodes": 1512, 2024-06-04T05:24:21.714 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_inode_file_caps": 839, 2024-06-04T05:24:21.714 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_revoke": 30343, 2024-06-04T05:24:21.714 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_grant": 21018, 2024-06-04T05:24:21.714 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_trunc": 0, 2024-06-04T05:24:21.714 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-04T05:24:21.714 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-04T05:24:21.714 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_caps": 41246, 2024-06-04T05:24:21.714 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_caps_dirty": 8385, 2024-06-04T05:24:21.714 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_cap_release": 593, 2024-06-04T05:24:21.714 INFO:teuthology.orchestra.run.smithi112.stdout: "process_request_cap_release": 21894 2024-06-04T05:24:21.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.714 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache": { 2024-06-04T05:24:21.714 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_update": 26, 2024-06-04T05:24:21.714 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_update_receipt": 6, 2024-06-04T05:24:21.715 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_try_discover": 4, 2024-06-04T05:24:21.715 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_send_discover": 160, 2024-06-04T05:24:21.715 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_handle_discover": 573, 2024-06-04T05:24:21.715 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays": 24, 2024-06-04T05:24:21.715 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays_delayed": 0, 2024-06-04T05:24:21.715 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays_enqueuing": 0, 2024-06-04T05:24:21.715 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_created": 5927, 2024-06-04T05:24:21.715 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_enqueued": 5903, 2024-06-04T05:24:21.715 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_reintegrated": 0, 2024-06-04T05:24:21.715 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_migrated": 0, 2024-06-04T05:24:21.715 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_processing": 0, 2024-06-04T05:24:21.715 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_enqueued": 0, 2024-06-04T05:24:21.715 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_prioritized": 0, 2024-06-04T05:24:21.715 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_started": 0, 2024-06-04T05:24:21.715 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_completed": 0, 2024-06-04T05:24:21.716 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_quiesce_path": 20, 2024-06-04T05:24:21.716 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_quiesce_inode": 2132, 2024-06-04T05:24:21.716 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_enqueue_scrub": 25, 2024-06-04T05:24:21.716 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_exportdir": 65, 2024-06-04T05:24:21.716 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_flush": 0, 2024-06-04T05:24:21.716 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_fragmentdir": 9, 2024-06-04T05:24:21.716 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_fragstats": 0, 2024-06-04T05:24:21.716 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_inodestats": 0 2024-06-04T05:24:21.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.716 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log": { 2024-06-04T05:24:21.716 INFO:teuthology.orchestra.run.smithi112.stdout: "evlrg": 0, 2024-06-04T05:24:21.716 INFO:teuthology.orchestra.run.smithi112.stdout: "evadd": 76344, 2024-06-04T05:24:21.716 INFO:teuthology.orchestra.run.smithi112.stdout: "evex": 76326, 2024-06-04T05:24:21.716 INFO:teuthology.orchestra.run.smithi112.stdout: "evtrm": 76326, 2024-06-04T05:24:21.716 INFO:teuthology.orchestra.run.smithi112.stdout: "ev": 18, 2024-06-04T05:24:21.716 INFO:teuthology.orchestra.run.smithi112.stdout: "evexg": 0, 2024-06-04T05:24:21.717 INFO:teuthology.orchestra.run.smithi112.stdout: "evexd": 135, 2024-06-04T05:24:21.717 INFO:teuthology.orchestra.run.smithi112.stdout: "segadd": 207, 2024-06-04T05:24:21.717 INFO:teuthology.orchestra.run.smithi112.stdout: "segex": 206, 2024-06-04T05:24:21.717 INFO:teuthology.orchestra.run.smithi112.stdout: "segtrm": 206, 2024-06-04T05:24:21.717 INFO:teuthology.orchestra.run.smithi112.stdout: "seg": 1, 2024-06-04T05:24:21.717 INFO:teuthology.orchestra.run.smithi112.stdout: "segmjr": 1, 2024-06-04T05:24:21.717 INFO:teuthology.orchestra.run.smithi112.stdout: "segexg": 0, 2024-06-04T05:24:21.717 INFO:teuthology.orchestra.run.smithi112.stdout: "segexd": 1, 2024-06-04T05:24:21.717 INFO:teuthology.orchestra.run.smithi112.stdout: "expos": 282794411, 2024-06-04T05:24:21.717 INFO:teuthology.orchestra.run.smithi112.stdout: "wrpos": 282854109, 2024-06-04T05:24:21.717 INFO:teuthology.orchestra.run.smithi112.stdout: "rdpos": 0, 2024-06-04T05:24:21.717 INFO:teuthology.orchestra.run.smithi112.stdout: "jlat": { 2024-06-04T05:24:21.718 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 35880, 2024-06-04T05:24:21.718 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 120.463904605, 2024-06-04T05:24:21.718 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.003357410 2024-06-04T05:24:21.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.718 INFO:teuthology.orchestra.run.smithi112.stdout: "replayed": 0 2024-06-04T05:24:21.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.718 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_mem": { 2024-06-04T05:24:21.718 INFO:teuthology.orchestra.run.smithi112.stdout: "ino": 1364, 2024-06-04T05:24:21.718 INFO:teuthology.orchestra.run.smithi112.stdout: "ino+": 9676, 2024-06-04T05:24:21.718 INFO:teuthology.orchestra.run.smithi112.stdout: "ino-": 8312, 2024-06-04T05:24:21.718 INFO:teuthology.orchestra.run.smithi112.stdout: "dir": 117, 2024-06-04T05:24:21.718 INFO:teuthology.orchestra.run.smithi112.stdout: "dir+": 287, 2024-06-04T05:24:21.719 INFO:teuthology.orchestra.run.smithi112.stdout: "dir-": 170, 2024-06-04T05:24:21.719 INFO:teuthology.orchestra.run.smithi112.stdout: "dn": 1580, 2024-06-04T05:24:21.719 INFO:teuthology.orchestra.run.smithi112.stdout: "dn+": 16966, 2024-06-04T05:24:21.719 INFO:teuthology.orchestra.run.smithi112.stdout: "dn-": 15386, 2024-06-04T05:24:21.719 INFO:teuthology.orchestra.run.smithi112.stdout: "cap": 3, 2024-06-04T05:24:21.719 INFO:teuthology.orchestra.run.smithi112.stdout: "cap+": 6772, 2024-06-04T05:24:21.719 INFO:teuthology.orchestra.run.smithi112.stdout: "cap-": 6769, 2024-06-04T05:24:21.719 INFO:teuthology.orchestra.run.smithi112.stdout: "rss": 85352, 2024-06-04T05:24:21.719 INFO:teuthology.orchestra.run.smithi112.stdout: "heap": 231800 2024-06-04T05:24:21.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.719 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_server": { 2024-06-04T05:24:21.719 INFO:teuthology.orchestra.run.smithi112.stdout: "dispatch_client_request": 798987, 2024-06-04T05:24:21.719 INFO:teuthology.orchestra.run.smithi112.stdout: "dispatch_server_request": 20201, 2024-06-04T05:24:21.719 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_request": 725011, 2024-06-04T05:24:21.720 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_session": 447, 2024-06-04T05:24:21.720 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_peer_request": 52359, 2024-06-04T05:24:21.720 INFO:teuthology.orchestra.run.smithi112.stdout: "req_create_latency": { 2024-06-04T05:24:21.720 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 6473, 2024-06-04T05:24:21.720 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 187.511327018, 2024-06-04T05:24:21.720 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.028968226 2024-06-04T05:24:21.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.720 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getattr_latency": { 2024-06-04T05:24:21.720 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 613306, 2024-06-04T05:24:21.720 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 221.816374988, 2024-06-04T05:24:21.720 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000361673 2024-06-04T05:24:21.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.721 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getfilelock_latency": { 2024-06-04T05:24:21.721 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.721 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.721 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.721 INFO:teuthology.orchestra.run.smithi112.stdout: "req_link_latency": { 2024-06-04T05:24:21.721 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.721 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.721 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.721 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookup_latency": { 2024-06-04T05:24:21.721 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 33199, 2024-06-04T05:24:21.721 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 184.080673486, 2024-06-04T05:24:21.721 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.005544765 2024-06-04T05:24:21.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.722 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookuphash_latency": { 2024-06-04T05:24:21.722 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.722 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.722 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.722 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupino_latency": { 2024-06-04T05:24:21.722 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.722 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.722 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.723 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupname_latency": { 2024-06-04T05:24:21.723 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.723 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.723 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.723 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupparent_latency": { 2024-06-04T05:24:21.723 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.723 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.723 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.723 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupsnap_latency": { 2024-06-04T05:24:21.724 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.724 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.724 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.724 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lssnap_latency": { 2024-06-04T05:24:21.724 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 27, 2024-06-04T05:24:21.724 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.007616324, 2024-06-04T05:24:21.725 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000282086 2024-06-04T05:24:21.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.725 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mkdir_latency": { 2024-06-04T05:24:21.725 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 119, 2024-06-04T05:24:21.725 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 2.848411158, 2024-06-04T05:24:21.725 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.023936228 2024-06-04T05:24:21.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.725 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mknod_latency": { 2024-06-04T05:24:21.725 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.725 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.725 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.725 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mksnap_latency": { 2024-06-04T05:24:21.726 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 27, 2024-06-04T05:24:21.726 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 10.385645092, 2024-06-04T05:24:21.726 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.384653521 2024-06-04T05:24:21.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.726 INFO:teuthology.orchestra.run.smithi112.stdout: "req_open_latency": { 2024-06-04T05:24:21.726 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 4103, 2024-06-04T05:24:21.726 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 3.367843368, 2024-06-04T05:24:21.726 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000820824 2024-06-04T05:24:21.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.726 INFO:teuthology.orchestra.run.smithi112.stdout: "req_readdir_latency": { 2024-06-04T05:24:21.726 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 27251, 2024-06-04T05:24:21.726 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 38.233819985, 2024-06-04T05:24:21.727 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.001403024 2024-06-04T05:24:21.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.727 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rename_latency": { 2024-06-04T05:24:21.727 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 1153, 2024-06-04T05:24:21.727 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 69.734658764, 2024-06-04T05:24:21.727 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.060481057 2024-06-04T05:24:21.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.727 INFO:teuthology.orchestra.run.smithi112.stdout: "req_renamesnap_latency": { 2024-06-04T05:24:21.727 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.727 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.727 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.727 INFO:teuthology.orchestra.run.smithi112.stdout: "req_snapdiff_latency": { 2024-06-04T05:24:21.728 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.728 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.728 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.728 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmdir_latency": { 2024-06-04T05:24:21.728 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 35, 2024-06-04T05:24:21.728 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.306952760, 2024-06-04T05:24:21.728 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.008770078 2024-06-04T05:24:21.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.728 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmsnap_latency": { 2024-06-04T05:24:21.728 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 20, 2024-06-04T05:24:21.728 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.192437565, 2024-06-04T05:24:21.729 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.009621878 2024-06-04T05:24:21.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.729 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmxattr_latency": { 2024-06-04T05:24:21.729 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.729 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.729 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.729 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setattr_latency": { 2024-06-04T05:24:21.729 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 4, 2024-06-04T05:24:21.730 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.017192901, 2024-06-04T05:24:21.730 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.004298225 2024-06-04T05:24:21.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.730 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setdirlayout_latency": { 2024-06-04T05:24:21.730 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.730 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.730 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.730 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setfilelock_latency": { 2024-06-04T05:24:21.730 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 188, 2024-06-04T05:24:21.730 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.136791543, 2024-06-04T05:24:21.730 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000727614 2024-06-04T05:24:21.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.731 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setlayout_latency": { 2024-06-04T05:24:21.731 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.731 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.731 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.731 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setxattr_latency": { 2024-06-04T05:24:21.731 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 7, 2024-06-04T05:24:21.731 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.019410657, 2024-06-04T05:24:21.731 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.002772951 2024-06-04T05:24:21.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.732 INFO:teuthology.orchestra.run.smithi112.stdout: "req_symlink_latency": { 2024-06-04T05:24:21.732 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.732 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.732 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.732 INFO:teuthology.orchestra.run.smithi112.stdout: "req_unlink_latency": { 2024-06-04T05:24:21.732 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 5903, 2024-06-04T05:24:21.732 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 113.215043645, 2024-06-04T05:24:21.732 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.019179238 2024-06-04T05:24:21.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.733 INFO:teuthology.orchestra.run.smithi112.stdout: "cap_revoke_eviction": 0, 2024-06-04T05:24:21.733 INFO:teuthology.orchestra.run.smithi112.stdout: "cap_acquisition_throttle": 0, 2024-06-04T05:24:21.733 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getvxattr_latency": { 2024-06-04T05:24:21.733 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.733 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.733 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.733 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:21.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.733 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_sessions": { 2024-06-04T05:24:21.733 INFO:teuthology.orchestra.run.smithi112.stdout: "session_count": 2, 2024-06-04T05:24:21.734 INFO:teuthology.orchestra.run.smithi112.stdout: "session_add": 4, 2024-06-04T05:24:21.734 INFO:teuthology.orchestra.run.smithi112.stdout: "session_remove": 2, 2024-06-04T05:24:21.734 INFO:teuthology.orchestra.run.smithi112.stdout: "sessions_open": 2, 2024-06-04T05:24:21.734 INFO:teuthology.orchestra.run.smithi112.stdout: "sessions_stale": 0, 2024-06-04T05:24:21.734 INFO:teuthology.orchestra.run.smithi112.stdout: "total_load": 19474, 2024-06-04T05:24:21.734 INFO:teuthology.orchestra.run.smithi112.stdout: "average_load": 9737, 2024-06-04T05:24:21.734 INFO:teuthology.orchestra.run.smithi112.stdout: "avg_session_uptime": 1603, 2024-06-04T05:24:21.734 INFO:teuthology.orchestra.run.smithi112.stdout: "mdthresh_evicted": 0 2024-06-04T05:24:21.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.734 INFO:teuthology.orchestra.run.smithi112.stdout: "mempool": { 2024-06-04T05:24:21.734 INFO:teuthology.orchestra.run.smithi112.stdout: "bloom_filter_bytes": 0, 2024-06-04T05:24:21.734 INFO:teuthology.orchestra.run.smithi112.stdout: "bloom_filter_items": 0, 2024-06-04T05:24:21.735 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_alloc_bytes": 0, 2024-06-04T05:24:21.735 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_alloc_items": 0, 2024-06-04T05:24:21.735 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_data_bytes": 0, 2024-06-04T05:24:21.735 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_data_items": 0, 2024-06-04T05:24:21.735 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-04T05:24:21.735 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_onode_items": 0, 2024-06-04T05:24:21.735 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-04T05:24:21.735 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_meta_items": 0, 2024-06-04T05:24:21.735 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_other_bytes": 0, 2024-06-04T05:24:21.735 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_other_items": 0, 2024-06-04T05:24:21.735 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-04T05:24:21.735 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_buffer_items": 0, 2024-06-04T05:24:21.736 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_bytes": 0, 2024-06-04T05:24:21.736 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_items": 0, 2024-06-04T05:24:21.736 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_blob_bytes": 0, 2024-06-04T05:24:21.736 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_blob_items": 0, 2024-06-04T05:24:21.736 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-04T05:24:21.736 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_shared_blob_items": 0, 2024-06-04T05:24:21.736 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-04T05:24:21.736 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_inline_bl_items": 0, 2024-06-04T05:24:21.736 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_bytes": 0, 2024-06-04T05:24:21.737 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_items": 0, 2024-06-04T05:24:21.737 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_txc_bytes": 0, 2024-06-04T05:24:21.737 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_txc_items": 0, 2024-06-04T05:24:21.737 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-04T05:24:21.737 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_deferred_items": 0, 2024-06-04T05:24:21.737 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_bytes": 0, 2024-06-04T05:24:21.737 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_items": 0, 2024-06-04T05:24:21.737 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_bytes": 0, 2024-06-04T05:24:21.737 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_items": 0, 2024-06-04T05:24:21.737 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_reader_bytes": 0, 2024-06-04T05:24:21.737 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_reader_items": 0, 2024-06-04T05:24:21.737 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_writer_bytes": 0, 2024-06-04T05:24:21.737 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_writer_items": 0, 2024-06-04T05:24:21.737 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_anon_bytes": 512471, 2024-06-04T05:24:21.738 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_anon_items": 166, 2024-06-04T05:24:21.738 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_meta_bytes": 0, 2024-06-04T05:24:21.738 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_meta_items": 0, 2024-06-04T05:24:21.738 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_bytes": 0, 2024-06-04T05:24:21.738 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_items": 0, 2024-06-04T05:24:21.738 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mapbl_bytes": 0, 2024-06-04T05:24:21.738 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mapbl_items": 0, 2024-06-04T05:24:21.738 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pglog_bytes": 0, 2024-06-04T05:24:21.738 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pglog_items": 0, 2024-06-04T05:24:21.738 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_bytes": 8224, 2024-06-04T05:24:21.738 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_items": 171, 2024-06-04T05:24:21.738 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_mapping_bytes": 0, 2024-06-04T05:24:21.738 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_mapping_items": 0, 2024-06-04T05:24:21.738 INFO:teuthology.orchestra.run.smithi112.stdout: "pgmap_bytes": 0, 2024-06-04T05:24:21.739 INFO:teuthology.orchestra.run.smithi112.stdout: "pgmap_items": 0, 2024-06-04T05:24:21.739 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_co_bytes": 4547594, 2024-06-04T05:24:21.739 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_co_items": 18125, 2024-06-04T05:24:21.739 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_1_bytes": 0, 2024-06-04T05:24:21.739 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_1_items": 0, 2024-06-04T05:24:21.739 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_2_bytes": 0, 2024-06-04T05:24:21.739 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_2_items": 0 2024-06-04T05:24:21.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.739 INFO:teuthology.orchestra.run.smithi112.stdout: "objecter": { 2024-06-04T05:24:21.739 INFO:teuthology.orchestra.run.smithi112.stdout: "op_active": 0, 2024-06-04T05:24:21.739 INFO:teuthology.orchestra.run.smithi112.stdout: "op_laggy": 0, 2024-06-04T05:24:21.739 INFO:teuthology.orchestra.run.smithi112.stdout: "op_send": 54915, 2024-06-04T05:24:21.739 INFO:teuthology.orchestra.run.smithi112.stdout: "op_send_bytes": 294627862, 2024-06-04T05:24:21.739 INFO:teuthology.orchestra.run.smithi112.stdout: "op_resend": 37, 2024-06-04T05:24:21.740 INFO:teuthology.orchestra.run.smithi112.stdout: "op_reply": 54915, 2024-06-04T05:24:21.740 INFO:teuthology.orchestra.run.smithi112.stdout: "op_latency": { 2024-06-04T05:24:21.740 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 54915, 2024-06-04T05:24:21.740 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 321.594618966, 2024-06-04T05:24:21.740 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.005856225 2024-06-04T05:24:21.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.740 INFO:teuthology.orchestra.run.smithi112.stdout: "op_inflight": 0, 2024-06-04T05:24:21.740 INFO:teuthology.orchestra.run.smithi112.stdout: "oplen_avg": { 2024-06-04T05:24:21.740 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 54915, 2024-06-04T05:24:21.740 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 69443 2024-06-04T05:24:21.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.740 INFO:teuthology.orchestra.run.smithi112.stdout: "op": 54915, 2024-06-04T05:24:21.740 INFO:teuthology.orchestra.run.smithi112.stdout: "op_r": 3186, 2024-06-04T05:24:21.740 INFO:teuthology.orchestra.run.smithi112.stdout: "op_w": 51729, 2024-06-04T05:24:21.740 INFO:teuthology.orchestra.run.smithi112.stdout: "op_rmw": 0, 2024-06-04T05:24:21.741 INFO:teuthology.orchestra.run.smithi112.stdout: "op_pg": 0, 2024-06-04T05:24:21.741 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_stat": 3719, 2024-06-04T05:24:21.741 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_create": 2495, 2024-06-04T05:24:21.741 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_read": 1052, 2024-06-04T05:24:21.741 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_write": 36907, 2024-06-04T05:24:21.741 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_writefull": 1025, 2024-06-04T05:24:21.741 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_writesame": 0, 2024-06-04T05:24:21.741 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_append": 0, 2024-06-04T05:24:21.741 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_zero": 0, 2024-06-04T05:24:21.741 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_truncate": 4, 2024-06-04T05:24:21.741 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_delete": 6555, 2024-06-04T05:24:21.741 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_mapext": 0, 2024-06-04T05:24:21.741 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_sparse_read": 0, 2024-06-04T05:24:21.741 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_clonerange": 0, 2024-06-04T05:24:21.741 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_getxattr": 2112, 2024-06-04T05:24:21.742 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_setxattr": 4990, 2024-06-04T05:24:21.742 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_cmpxattr": 0, 2024-06-04T05:24:21.742 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_rmxattr": 0, 2024-06-04T05:24:21.742 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_resetxattrs": 0, 2024-06-04T05:24:21.742 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_call": 0, 2024-06-04T05:24:21.742 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_watch": 0, 2024-06-04T05:24:21.742 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_notify": 0, 2024-06-04T05:24:21.742 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_src_cmpxattr": 0, 2024-06-04T05:24:21.742 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_pgls": 0, 2024-06-04T05:24:21.742 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_pgls_filter": 0, 2024-06-04T05:24:21.742 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_other": 0, 2024-06-04T05:24:21.742 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_active": 0, 2024-06-04T05:24:21.742 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_send": 0, 2024-06-04T05:24:21.742 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_resend": 0, 2024-06-04T05:24:21.743 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_ping": 0, 2024-06-04T05:24:21.743 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_active": 0, 2024-06-04T05:24:21.743 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_send": 0, 2024-06-04T05:24:21.743 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_resend": 0, 2024-06-04T05:24:21.743 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_active": 0, 2024-06-04T05:24:21.743 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_send": 0, 2024-06-04T05:24:21.743 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_resend": 0, 2024-06-04T05:24:21.743 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_active": 0, 2024-06-04T05:24:21.743 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_send": 0, 2024-06-04T05:24:21.743 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_resend": 0, 2024-06-04T05:24:21.743 INFO:teuthology.orchestra.run.smithi112.stdout: "command_active": 0, 2024-06-04T05:24:21.743 INFO:teuthology.orchestra.run.smithi112.stdout: "command_send": 0, 2024-06-04T05:24:21.743 INFO:teuthology.orchestra.run.smithi112.stdout: "command_resend": 0, 2024-06-04T05:24:21.744 INFO:teuthology.orchestra.run.smithi112.stdout: "map_epoch": 122, 2024-06-04T05:24:21.744 INFO:teuthology.orchestra.run.smithi112.stdout: "map_full": 0, 2024-06-04T05:24:21.744 INFO:teuthology.orchestra.run.smithi112.stdout: "map_inc": 43, 2024-06-04T05:24:21.744 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_sessions": 12, 2024-06-04T05:24:21.744 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_session_open": 12, 2024-06-04T05:24:21.744 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_session_close": 0, 2024-06-04T05:24:21.744 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_laggy": 0, 2024-06-04T05:24:21.744 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_wr": 8594, 2024-06-04T05:24:21.744 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_rd": 75, 2024-06-04T05:24:21.744 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_del": 1915 2024-06-04T05:24:21.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.744 INFO:teuthology.orchestra.run.smithi112.stdout: "oft": { 2024-06-04T05:24:21.744 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_objs": 1, 2024-06-04T05:24:21.744 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_kv_pairs": 0, 2024-06-04T05:24:21.745 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_updates": 0, 2024-06-04T05:24:21.745 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_removes": 0 2024-06-04T05:24:21.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.745 INFO:teuthology.orchestra.run.smithi112.stdout: "osd-slow-ops": { 2024-06-04T05:24:21.745 INFO:teuthology.orchestra.run.smithi112.stdout: "slow_ops_count": 3566 2024-06-04T05:24:21.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.745 INFO:teuthology.orchestra.run.smithi112.stdout: "purge_queue": { 2024-06-04T05:24:21.745 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_ops": 0, 2024-06-04T05:24:21.745 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_ops_high_water": 6, 2024-06-04T05:24:21.745 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing": 0, 2024-06-04T05:24:21.745 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_high_water": 6, 2024-06-04T05:24:21.746 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executed_ops": 5903, 2024-06-04T05:24:21.746 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executed": 5903, 2024-06-04T05:24:21.746 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_item_in_journal": 0 2024-06-04T05:24:21.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.746 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-04T05:24:21.746 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:21.746 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 104857600, 2024-06-04T05:24:21.746 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:21.746 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 1060672, 2024-06-04T05:24:21.746 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 213439124, 2024-06-04T05:24:21.746 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:21.746 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 1060672, 2024-06-04T05:24:21.746 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-06-04T05:24:21.747 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-06-04T05:24:21.747 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 1060672, 2024-06-04T05:24:21.747 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 213439124, 2024-06-04T05:24:21.747 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:21.747 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.747 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.747 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.747 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:21.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.747 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-objecter_bytes": { 2024-06-04T05:24:21.747 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:21.747 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 104857600, 2024-06-04T05:24:21.747 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:21.748 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-06-04T05:24:21.748 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-06-04T05:24:21.748 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:21.748 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:21.748 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 54915, 2024-06-04T05:24:21.748 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 295231486, 2024-06-04T05:24:21.748 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 54915, 2024-06-04T05:24:21.748 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 295231486, 2024-06-04T05:24:21.748 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:21.749 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.749 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.749 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.749 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:21.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.749 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-objecter_ops": { 2024-06-04T05:24:21.749 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:21.749 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 1024, 2024-06-04T05:24:21.749 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:21.749 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-06-04T05:24:21.749 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-06-04T05:24:21.750 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:21.750 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:21.750 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 54915, 2024-06-04T05:24:21.750 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 54915, 2024-06-04T05:24:21.750 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 54915, 2024-06-04T05:24:21.750 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 54915, 2024-06-04T05:24:21.750 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:21.750 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.750 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.750 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.750 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:21.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.751 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-write_buf_throttle": { 2024-06-04T05:24:21.751 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:21.751 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 3758096384, 2024-06-04T05:24:21.751 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:21.751 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 5903, 2024-06-04T05:24:21.751 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 604515, 2024-06-04T05:24:21.751 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:21.751 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 5903, 2024-06-04T05:24:21.751 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-06-04T05:24:21.751 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-06-04T05:24:21.751 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 1027, 2024-06-04T05:24:21.751 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 604515, 2024-06-04T05:24:21.751 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:21.752 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.752 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.752 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.752 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:21.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:21.752 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-write_buf_throttle-0x55d6dea26500": { 2024-06-04T05:24:21.752 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:21.752 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 3758096384, 2024-06-04T05:24:21.752 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:21.752 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 76344, 2024-06-04T05:24:21.752 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 278659805, 2024-06-04T05:24:21.752 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:21.752 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 76344, 2024-06-04T05:24:21.752 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-06-04T05:24:21.753 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-06-04T05:24:21.753 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 35880, 2024-06-04T05:24:21.753 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 278659805, 2024-06-04T05:24:21.753 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:21.753 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:21.753 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:21.753 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:21.753 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:21.753 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:21.753 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-06-04T05:24:21.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:21 smithi070 ceph-mon[32631]: pgmap v858: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 12 op/s 2024-06-04T05:24:21.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:21 smithi098 ceph-mon[35569]: pgmap v858: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 12 op/s 2024-06-04T05:24:22.400 INFO:tasks.check_counter:Daemon mds.i mds.exported=61 2024-06-04T05:24:22.400 INFO:tasks.check_counter:Daemon mds.i mds.imported=67 2024-06-04T05:24:22.401 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=7 2024-06-04T05:24:22.401 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=27 2024-06-04T05:24:22.401 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=20 2024-06-04T05:24:22.401 DEBUG:tasks.check_counter:Getting stats from l 2024-06-04T05:24:22.401 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph daemon mds.l perf dump 2024-06-04T05:24:22.638 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.c/config 2024-06-04T05:24:23.772 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:23 smithi112 ceph-mon[36453]: pgmap v859: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 12 op/s 2024-06-04T05:24:23.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:23 smithi070 ceph-mon[32631]: pgmap v859: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 12 op/s 2024-06-04T05:24:23.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:23 smithi098 ceph-mon[35569]: pgmap v859: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 12 op/s 2024-06-04T05:24:24.335 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-06-04T05:24:24.335 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-0": { 2024-06-04T05:24:24.335 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 3617, 2024-06-04T05:24:24.336 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 3617, 2024-06-04T05:24:24.336 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 101734409, 2024-06-04T05:24:24.336 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 1149392, 2024-06-04T05:24:24.336 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 8, 2024-06-04T05:24:24.336 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 4, 2024-06-04T05:24:24.336 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 0.345922023, 2024-06-04T05:24:24.336 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.088344951, 2024-06-04T05:24:24.336 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.131481837, 2024-06-04T05:24:24.336 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.099974267, 2024-06-04T05:24:24.336 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:24.336 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 3616, 2024-06-04T05:24:24.337 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.111697936, 2024-06-04T05:24:24.337 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000030889 2024-06-04T05:24:24.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.337 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:24.337 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.337 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.337 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.337 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 2912, 2024-06-04T05:24:24.337 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 448 2024-06-04T05:24:24.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.338 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-1": { 2024-06-04T05:24:24.338 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 2046, 2024-06-04T05:24:24.338 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 1991, 2024-06-04T05:24:24.338 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 94631248, 2024-06-04T05:24:24.338 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 777714, 2024-06-04T05:24:24.338 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 6, 2024-06-04T05:24:24.338 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 5, 2024-06-04T05:24:24.338 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 0.218709406, 2024-06-04T05:24:24.338 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.059708189, 2024-06-04T05:24:24.338 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.117680545, 2024-06-04T05:24:24.338 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.042490944, 2024-06-04T05:24:24.339 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:24.339 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 1990, 2024-06-04T05:24:24.339 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.061046118, 2024-06-04T05:24:24.339 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000030676 2024-06-04T05:24:24.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.339 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:24.339 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.339 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.339 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.339 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 1992064, 2024-06-04T05:24:24.339 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 319728 2024-06-04T05:24:24.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.340 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-2": { 2024-06-04T05:24:24.340 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 1674, 2024-06-04T05:24:24.340 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 2056, 2024-06-04T05:24:24.340 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 86144020, 2024-06-04T05:24:24.340 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 1364881, 2024-06-04T05:24:24.340 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 8, 2024-06-04T05:24:24.340 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 6, 2024-06-04T05:24:24.340 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 0.188300418, 2024-06-04T05:24:24.340 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.056374755, 2024-06-04T05:24:24.341 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.073751673, 2024-06-04T05:24:24.341 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.040406076, 2024-06-04T05:24:24.341 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-06-04T05:24:24.341 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 2053, 2024-06-04T05:24:24.341 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.073202396, 2024-06-04T05:24:24.341 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000035656 2024-06-04T05:24:24.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.341 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-06-04T05:24:24.341 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.341 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.341 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.342 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 320, 2024-06-04T05:24:24.342 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 833184 2024-06-04T05:24:24.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.342 INFO:teuthology.orchestra.run.smithi112.stdout: "cct": { 2024-06-04T05:24:24.342 INFO:teuthology.orchestra.run.smithi112.stdout: "total_workers": 1, 2024-06-04T05:24:24.342 INFO:teuthology.orchestra.run.smithi112.stdout: "unhealthy_workers": 0 2024-06-04T05:24:24.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.342 INFO:teuthology.orchestra.run.smithi112.stdout: "finisher-MDSRank": { 2024-06-04T05:24:24.342 INFO:teuthology.orchestra.run.smithi112.stdout: "queue_len": 0, 2024-06-04T05:24:24.342 INFO:teuthology.orchestra.run.smithi112.stdout: "complete_latency": { 2024-06-04T05:24:24.343 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 11382, 2024-06-04T05:24:24.343 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.385514262, 2024-06-04T05:24:24.343 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000033870 2024-06-04T05:24:24.343 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:24.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.343 INFO:teuthology.orchestra.run.smithi112.stdout: "finisher-PurgeQueue": { 2024-06-04T05:24:24.343 INFO:teuthology.orchestra.run.smithi112.stdout: "queue_len": 0, 2024-06-04T05:24:24.343 INFO:teuthology.orchestra.run.smithi112.stdout: "complete_latency": { 2024-06-04T05:24:24.343 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.343 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.343 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.343 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:24.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.344 INFO:teuthology.orchestra.run.smithi112.stdout: "mds": { 2024-06-04T05:24:24.344 INFO:teuthology.orchestra.run.smithi112.stdout: "request": 0, 2024-06-04T05:24:24.344 INFO:teuthology.orchestra.run.smithi112.stdout: "reply": 0, 2024-06-04T05:24:24.344 INFO:teuthology.orchestra.run.smithi112.stdout: "reply_latency": { 2024-06-04T05:24:24.344 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.344 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.344 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.344 INFO:teuthology.orchestra.run.smithi112.stdout: "slow_reply": 0, 2024-06-04T05:24:24.344 INFO:teuthology.orchestra.run.smithi112.stdout: "forward": 0, 2024-06-04T05:24:24.345 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_fetch_complete": 0, 2024-06-04T05:24:24.345 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_fetch_keys": 0, 2024-06-04T05:24:24.345 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_commit": 0, 2024-06-04T05:24:24.345 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_split": 0, 2024-06-04T05:24:24.345 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_merge": 0, 2024-06-04T05:24:24.345 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes": 7719, 2024-06-04T05:24:24.345 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_top": 5241, 2024-06-04T05:24:24.345 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_bottom": 2478, 2024-06-04T05:24:24.345 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_pin_tail": 0, 2024-06-04T05:24:24.345 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_pinned": 38, 2024-06-04T05:24:24.346 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_expired": 0, 2024-06-04T05:24:24.346 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_with_caps": 0, 2024-06-04T05:24:24.346 INFO:teuthology.orchestra.run.smithi112.stdout: "caps": 0, 2024-06-04T05:24:24.346 INFO:teuthology.orchestra.run.smithi112.stdout: "subtrees": 2, 2024-06-04T05:24:24.346 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse": 0, 2024-06-04T05:24:24.346 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_hit": 0, 2024-06-04T05:24:24.346 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_forward": 0, 2024-06-04T05:24:24.346 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_discover": 0, 2024-06-04T05:24:24.346 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_dir_fetch": 0, 2024-06-04T05:24:24.346 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_remote_ino": 0, 2024-06-04T05:24:24.346 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_lock": 0, 2024-06-04T05:24:24.347 INFO:teuthology.orchestra.run.smithi112.stdout: "load_cent": 0, 2024-06-04T05:24:24.347 INFO:teuthology.orchestra.run.smithi112.stdout: "q": 0, 2024-06-04T05:24:24.347 INFO:teuthology.orchestra.run.smithi112.stdout: "exported": 0, 2024-06-04T05:24:24.347 INFO:teuthology.orchestra.run.smithi112.stdout: "exported_inodes": 0, 2024-06-04T05:24:24.347 INFO:teuthology.orchestra.run.smithi112.stdout: "imported": 0, 2024-06-04T05:24:24.347 INFO:teuthology.orchestra.run.smithi112.stdout: "imported_inodes": 0, 2024-06-04T05:24:24.347 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_dir_fetch": 0, 2024-06-04T05:24:24.347 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_backtrace_fetch": 0, 2024-06-04T05:24:24.347 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_peer_discover": 0, 2024-06-04T05:24:24.347 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rfiles": 4, 2024-06-04T05:24:24.347 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rbytes": 234, 2024-06-04T05:24:24.347 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rsnaps": 7, 2024-06-04T05:24:24.348 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_backtrace_fetch": 0, 2024-06-04T05:24:24.348 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_set_tag": 0, 2024-06-04T05:24:24.348 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_backtrace_repaired": 0, 2024-06-04T05:24:24.348 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_inotable_repaired": 0, 2024-06-04T05:24:24.348 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dir_inodes": 0, 2024-06-04T05:24:24.348 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dir_base_inodes": 0, 2024-06-04T05:24:24.348 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dirfrag_rstats": 0, 2024-06-04T05:24:24.348 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_file_inodes": 0, 2024-06-04T05:24:24.348 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_inode_file_caps": 0, 2024-06-04T05:24:24.348 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_revoke": 0, 2024-06-04T05:24:24.348 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_grant": 0, 2024-06-04T05:24:24.348 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_trunc": 0, 2024-06-04T05:24:24.348 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-04T05:24:24.349 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-04T05:24:24.349 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_caps": 0, 2024-06-04T05:24:24.349 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_caps_dirty": 0, 2024-06-04T05:24:24.349 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_cap_release": 0, 2024-06-04T05:24:24.349 INFO:teuthology.orchestra.run.smithi112.stdout: "process_request_cap_release": 0 2024-06-04T05:24:24.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.349 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache": { 2024-06-04T05:24:24.349 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_update": 0, 2024-06-04T05:24:24.349 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_update_receipt": 0, 2024-06-04T05:24:24.349 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_try_discover": 0, 2024-06-04T05:24:24.349 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_send_discover": 0, 2024-06-04T05:24:24.350 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_handle_discover": 0, 2024-06-04T05:24:24.350 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays": 0, 2024-06-04T05:24:24.350 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays_delayed": 0, 2024-06-04T05:24:24.350 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays_enqueuing": 0, 2024-06-04T05:24:24.350 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_created": 0, 2024-06-04T05:24:24.350 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_enqueued": 0, 2024-06-04T05:24:24.350 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_reintegrated": 0, 2024-06-04T05:24:24.350 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_migrated": 0, 2024-06-04T05:24:24.350 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_processing": 0, 2024-06-04T05:24:24.350 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_enqueued": 0, 2024-06-04T05:24:24.350 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_prioritized": 0, 2024-06-04T05:24:24.350 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_started": 0, 2024-06-04T05:24:24.351 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_completed": 0, 2024-06-04T05:24:24.351 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_quiesce_path": 0, 2024-06-04T05:24:24.351 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_quiesce_inode": 0, 2024-06-04T05:24:24.351 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_enqueue_scrub": 0, 2024-06-04T05:24:24.351 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_exportdir": 0, 2024-06-04T05:24:24.351 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_flush": 0, 2024-06-04T05:24:24.351 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_fragmentdir": 0, 2024-06-04T05:24:24.351 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_fragstats": 0, 2024-06-04T05:24:24.351 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_inodestats": 0 2024-06-04T05:24:24.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.351 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log": { 2024-06-04T05:24:24.351 INFO:teuthology.orchestra.run.smithi112.stdout: "evlrg": 0, 2024-06-04T05:24:24.352 INFO:teuthology.orchestra.run.smithi112.stdout: "evadd": 0, 2024-06-04T05:24:24.352 INFO:teuthology.orchestra.run.smithi112.stdout: "evex": 0, 2024-06-04T05:24:24.352 INFO:teuthology.orchestra.run.smithi112.stdout: "evtrm": 0, 2024-06-04T05:24:24.352 INFO:teuthology.orchestra.run.smithi112.stdout: "ev": 76344, 2024-06-04T05:24:24.352 INFO:teuthology.orchestra.run.smithi112.stdout: "evexg": 0, 2024-06-04T05:24:24.352 INFO:teuthology.orchestra.run.smithi112.stdout: "evexd": 0, 2024-06-04T05:24:24.352 INFO:teuthology.orchestra.run.smithi112.stdout: "segadd": 0, 2024-06-04T05:24:24.352 INFO:teuthology.orchestra.run.smithi112.stdout: "segex": 0, 2024-06-04T05:24:24.352 INFO:teuthology.orchestra.run.smithi112.stdout: "segtrm": 0, 2024-06-04T05:24:24.352 INFO:teuthology.orchestra.run.smithi112.stdout: "seg": 2, 2024-06-04T05:24:24.352 INFO:teuthology.orchestra.run.smithi112.stdout: "segmjr": 120, 2024-06-04T05:24:24.352 INFO:teuthology.orchestra.run.smithi112.stdout: "segexg": 0, 2024-06-04T05:24:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: "segexd": 0, 2024-06-04T05:24:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: "expos": 282794411, 2024-06-04T05:24:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: "wrpos": 282854109, 2024-06-04T05:24:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: "rdpos": 282853997, 2024-06-04T05:24:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: "jlat": { 2024-06-04T05:24:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: "replayed": 76344 2024-06-04T05:24:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_mem": { 2024-06-04T05:24:24.354 INFO:teuthology.orchestra.run.smithi112.stdout: "ino": 1555, 2024-06-04T05:24:24.354 INFO:teuthology.orchestra.run.smithi112.stdout: "ino+": 9454, 2024-06-04T05:24:24.354 INFO:teuthology.orchestra.run.smithi112.stdout: "ino-": 7899, 2024-06-04T05:24:24.354 INFO:teuthology.orchestra.run.smithi112.stdout: "dir": 86, 2024-06-04T05:24:24.354 INFO:teuthology.orchestra.run.smithi112.stdout: "dir+": 229, 2024-06-04T05:24:24.354 INFO:teuthology.orchestra.run.smithi112.stdout: "dir-": 143, 2024-06-04T05:24:24.354 INFO:teuthology.orchestra.run.smithi112.stdout: "dn": 7719, 2024-06-04T05:24:24.354 INFO:teuthology.orchestra.run.smithi112.stdout: "dn+": 10003, 2024-06-04T05:24:24.354 INFO:teuthology.orchestra.run.smithi112.stdout: "dn-": 2284, 2024-06-04T05:24:24.355 INFO:teuthology.orchestra.run.smithi112.stdout: "cap": 0, 2024-06-04T05:24:24.355 INFO:teuthology.orchestra.run.smithi112.stdout: "cap+": 0, 2024-06-04T05:24:24.355 INFO:teuthology.orchestra.run.smithi112.stdout: "cap-": 0, 2024-06-04T05:24:24.355 INFO:teuthology.orchestra.run.smithi112.stdout: "rss": 69012, 2024-06-04T05:24:24.355 INFO:teuthology.orchestra.run.smithi112.stdout: "heap": 215416 2024-06-04T05:24:24.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.355 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_server": { 2024-06-04T05:24:24.355 INFO:teuthology.orchestra.run.smithi112.stdout: "dispatch_client_request": 0, 2024-06-04T05:24:24.355 INFO:teuthology.orchestra.run.smithi112.stdout: "dispatch_server_request": 0, 2024-06-04T05:24:24.355 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_request": 0, 2024-06-04T05:24:24.355 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_session": 0, 2024-06-04T05:24:24.356 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_peer_request": 0, 2024-06-04T05:24:24.356 INFO:teuthology.orchestra.run.smithi112.stdout: "req_create_latency": { 2024-06-04T05:24:24.356 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.356 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.356 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.356 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getattr_latency": { 2024-06-04T05:24:24.356 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.356 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.356 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.357 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getfilelock_latency": { 2024-06-04T05:24:24.357 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.357 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.357 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.357 INFO:teuthology.orchestra.run.smithi112.stdout: "req_link_latency": { 2024-06-04T05:24:24.357 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.357 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.357 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.357 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookup_latency": { 2024-06-04T05:24:24.357 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.358 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.358 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.358 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookuphash_latency": { 2024-06-04T05:24:24.358 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.358 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.358 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.358 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupino_latency": { 2024-06-04T05:24:24.358 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.358 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.359 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.359 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupname_latency": { 2024-06-04T05:24:24.359 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.359 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.359 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.359 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupparent_latency": { 2024-06-04T05:24:24.359 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.359 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.359 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.360 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupsnap_latency": { 2024-06-04T05:24:24.360 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.360 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.360 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.360 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lssnap_latency": { 2024-06-04T05:24:24.360 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.360 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.361 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.361 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mkdir_latency": { 2024-06-04T05:24:24.361 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.362 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.362 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.362 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mknod_latency": { 2024-06-04T05:24:24.362 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.362 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.362 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.362 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mksnap_latency": { 2024-06-04T05:24:24.362 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.362 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.363 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.363 INFO:teuthology.orchestra.run.smithi112.stdout: "req_open_latency": { 2024-06-04T05:24:24.363 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.363 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.363 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.363 INFO:teuthology.orchestra.run.smithi112.stdout: "req_readdir_latency": { 2024-06-04T05:24:24.363 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.363 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.363 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.364 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rename_latency": { 2024-06-04T05:24:24.364 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.364 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.364 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.364 INFO:teuthology.orchestra.run.smithi112.stdout: "req_renamesnap_latency": { 2024-06-04T05:24:24.364 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.364 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.364 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.364 INFO:teuthology.orchestra.run.smithi112.stdout: "req_snapdiff_latency": { 2024-06-04T05:24:24.365 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.365 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.365 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.365 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmdir_latency": { 2024-06-04T05:24:24.365 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.365 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.365 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.365 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmsnap_latency": { 2024-06-04T05:24:24.365 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.365 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.366 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.366 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmxattr_latency": { 2024-06-04T05:24:24.366 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.366 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.366 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.366 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setattr_latency": { 2024-06-04T05:24:24.366 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.366 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.366 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.367 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setdirlayout_latency": { 2024-06-04T05:24:24.367 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.367 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.367 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.367 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setfilelock_latency": { 2024-06-04T05:24:24.367 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.367 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.367 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.367 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setlayout_latency": { 2024-06-04T05:24:24.367 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.368 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.368 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.368 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setxattr_latency": { 2024-06-04T05:24:24.368 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.368 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.368 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.368 INFO:teuthology.orchestra.run.smithi112.stdout: "req_symlink_latency": { 2024-06-04T05:24:24.368 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.368 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.368 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.369 INFO:teuthology.orchestra.run.smithi112.stdout: "req_unlink_latency": { 2024-06-04T05:24:24.369 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.369 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.369 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.369 INFO:teuthology.orchestra.run.smithi112.stdout: "cap_revoke_eviction": 0, 2024-06-04T05:24:24.369 INFO:teuthology.orchestra.run.smithi112.stdout: "cap_acquisition_throttle": 0, 2024-06-04T05:24:24.369 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getvxattr_latency": { 2024-06-04T05:24:24.369 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.369 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.369 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.369 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:24.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.370 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_sessions": { 2024-06-04T05:24:24.370 INFO:teuthology.orchestra.run.smithi112.stdout: "session_count": 2, 2024-06-04T05:24:24.370 INFO:teuthology.orchestra.run.smithi112.stdout: "session_add": 4, 2024-06-04T05:24:24.370 INFO:teuthology.orchestra.run.smithi112.stdout: "session_remove": 2, 2024-06-04T05:24:24.370 INFO:teuthology.orchestra.run.smithi112.stdout: "sessions_open": 3, 2024-06-04T05:24:24.370 INFO:teuthology.orchestra.run.smithi112.stdout: "sessions_stale": 0, 2024-06-04T05:24:24.370 INFO:teuthology.orchestra.run.smithi112.stdout: "total_load": 0, 2024-06-04T05:24:24.370 INFO:teuthology.orchestra.run.smithi112.stdout: "average_load": 0, 2024-06-04T05:24:24.370 INFO:teuthology.orchestra.run.smithi112.stdout: "avg_session_uptime": 4827, 2024-06-04T05:24:24.370 INFO:teuthology.orchestra.run.smithi112.stdout: "mdthresh_evicted": 0 2024-06-04T05:24:24.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.371 INFO:teuthology.orchestra.run.smithi112.stdout: "mempool": { 2024-06-04T05:24:24.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bloom_filter_bytes": 0, 2024-06-04T05:24:24.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bloom_filter_items": 0, 2024-06-04T05:24:24.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_alloc_bytes": 0, 2024-06-04T05:24:24.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_alloc_items": 0, 2024-06-04T05:24:24.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_data_bytes": 0, 2024-06-04T05:24:24.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_data_items": 0, 2024-06-04T05:24:24.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-04T05:24:24.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_onode_items": 0, 2024-06-04T05:24:24.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-04T05:24:24.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_meta_items": 0, 2024-06-04T05:24:24.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_other_bytes": 0, 2024-06-04T05:24:24.372 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_other_items": 0, 2024-06-04T05:24:24.372 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-04T05:24:24.372 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_buffer_items": 0, 2024-06-04T05:24:24.372 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_bytes": 0, 2024-06-04T05:24:24.372 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_items": 0, 2024-06-04T05:24:24.372 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_blob_bytes": 0, 2024-06-04T05:24:24.372 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_blob_items": 0, 2024-06-04T05:24:24.372 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-04T05:24:24.372 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_shared_blob_items": 0, 2024-06-04T05:24:24.372 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-04T05:24:24.372 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_inline_bl_items": 0, 2024-06-04T05:24:24.372 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_bytes": 0, 2024-06-04T05:24:24.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_items": 0, 2024-06-04T05:24:24.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_txc_bytes": 0, 2024-06-04T05:24:24.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_txc_items": 0, 2024-06-04T05:24:24.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-04T05:24:24.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_deferred_items": 0, 2024-06-04T05:24:24.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_bytes": 0, 2024-06-04T05:24:24.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_items": 0, 2024-06-04T05:24:24.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_bytes": 0, 2024-06-04T05:24:24.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_items": 0, 2024-06-04T05:24:24.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_reader_bytes": 0, 2024-06-04T05:24:24.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_reader_items": 0, 2024-06-04T05:24:24.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_writer_bytes": 0, 2024-06-04T05:24:24.374 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_writer_items": 0, 2024-06-04T05:24:24.374 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_anon_bytes": 241000, 2024-06-04T05:24:24.374 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_anon_items": 82, 2024-06-04T05:24:24.374 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_meta_bytes": 0, 2024-06-04T05:24:24.374 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_meta_items": 0, 2024-06-04T05:24:24.374 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_bytes": 0, 2024-06-04T05:24:24.374 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_items": 0, 2024-06-04T05:24:24.374 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mapbl_bytes": 0, 2024-06-04T05:24:24.374 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mapbl_items": 0, 2024-06-04T05:24:24.374 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pglog_bytes": 0, 2024-06-04T05:24:24.374 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pglog_items": 0, 2024-06-04T05:24:24.374 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_bytes": 8224, 2024-06-04T05:24:24.375 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_items": 171, 2024-06-04T05:24:24.375 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_mapping_bytes": 0, 2024-06-04T05:24:24.375 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_mapping_items": 0, 2024-06-04T05:24:24.375 INFO:teuthology.orchestra.run.smithi112.stdout: "pgmap_bytes": 0, 2024-06-04T05:24:24.375 INFO:teuthology.orchestra.run.smithi112.stdout: "pgmap_items": 0, 2024-06-04T05:24:24.375 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_co_bytes": 8917997, 2024-06-04T05:24:24.375 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_co_items": 37709, 2024-06-04T05:24:24.375 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_1_bytes": 0, 2024-06-04T05:24:24.375 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_1_items": 0, 2024-06-04T05:24:24.375 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_2_bytes": 0, 2024-06-04T05:24:24.375 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_2_items": 0 2024-06-04T05:24:24.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.376 INFO:teuthology.orchestra.run.smithi112.stdout: "objecter": { 2024-06-04T05:24:24.376 INFO:teuthology.orchestra.run.smithi112.stdout: "op_active": 0, 2024-06-04T05:24:24.376 INFO:teuthology.orchestra.run.smithi112.stdout: "op_laggy": 0, 2024-06-04T05:24:24.376 INFO:teuthology.orchestra.run.smithi112.stdout: "op_send": 6725, 2024-06-04T05:24:24.376 INFO:teuthology.orchestra.run.smithi112.stdout: "op_send_bytes": 26, 2024-06-04T05:24:24.376 INFO:teuthology.orchestra.run.smithi112.stdout: "op_resend": 4, 2024-06-04T05:24:24.376 INFO:teuthology.orchestra.run.smithi112.stdout: "op_reply": 6725, 2024-06-04T05:24:24.376 INFO:teuthology.orchestra.run.smithi112.stdout: "op_latency": { 2024-06-04T05:24:24.376 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 6725, 2024-06-04T05:24:24.376 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 4.317887096, 2024-06-04T05:24:24.376 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000642064 2024-06-04T05:24:24.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.377 INFO:teuthology.orchestra.run.smithi112.stdout: "op_inflight": 0, 2024-06-04T05:24:24.377 INFO:teuthology.orchestra.run.smithi112.stdout: "oplen_avg": { 2024-06-04T05:24:24.377 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 6725, 2024-06-04T05:24:24.377 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 6726 2024-06-04T05:24:24.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.377 INFO:teuthology.orchestra.run.smithi112.stdout: "op": 6725, 2024-06-04T05:24:24.377 INFO:teuthology.orchestra.run.smithi112.stdout: "op_r": 6725, 2024-06-04T05:24:24.377 INFO:teuthology.orchestra.run.smithi112.stdout: "op_w": 0, 2024-06-04T05:24:24.377 INFO:teuthology.orchestra.run.smithi112.stdout: "op_rmw": 0, 2024-06-04T05:24:24.378 INFO:teuthology.orchestra.run.smithi112.stdout: "op_pg": 0, 2024-06-04T05:24:24.378 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_stat": 3634, 2024-06-04T05:24:24.378 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_create": 0, 2024-06-04T05:24:24.378 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_read": 3088, 2024-06-04T05:24:24.378 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_write": 0, 2024-06-04T05:24:24.378 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_writefull": 0, 2024-06-04T05:24:24.378 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_writesame": 0, 2024-06-04T05:24:24.378 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_append": 0, 2024-06-04T05:24:24.378 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_zero": 0, 2024-06-04T05:24:24.378 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_truncate": 0, 2024-06-04T05:24:24.378 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_delete": 0, 2024-06-04T05:24:24.378 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_mapext": 0, 2024-06-04T05:24:24.379 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_sparse_read": 0, 2024-06-04T05:24:24.379 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_clonerange": 0, 2024-06-04T05:24:24.379 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_getxattr": 2, 2024-06-04T05:24:24.379 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_setxattr": 0, 2024-06-04T05:24:24.379 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_cmpxattr": 0, 2024-06-04T05:24:24.379 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_rmxattr": 0, 2024-06-04T05:24:24.379 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_resetxattrs": 0, 2024-06-04T05:24:24.379 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_call": 0, 2024-06-04T05:24:24.379 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_watch": 0, 2024-06-04T05:24:24.379 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_notify": 0, 2024-06-04T05:24:24.379 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_src_cmpxattr": 0, 2024-06-04T05:24:24.379 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_pgls": 0, 2024-06-04T05:24:24.379 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_pgls_filter": 0, 2024-06-04T05:24:24.380 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_other": 0, 2024-06-04T05:24:24.380 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_active": 0, 2024-06-04T05:24:24.380 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_send": 0, 2024-06-04T05:24:24.380 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_resend": 0, 2024-06-04T05:24:24.380 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_ping": 0, 2024-06-04T05:24:24.380 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_active": 0, 2024-06-04T05:24:24.380 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_send": 0, 2024-06-04T05:24:24.380 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_resend": 0, 2024-06-04T05:24:24.380 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_active": 0, 2024-06-04T05:24:24.380 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_send": 0, 2024-06-04T05:24:24.380 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_resend": 0, 2024-06-04T05:24:24.380 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_active": 0, 2024-06-04T05:24:24.380 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_send": 0, 2024-06-04T05:24:24.380 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_resend": 0, 2024-06-04T05:24:24.381 INFO:teuthology.orchestra.run.smithi112.stdout: "command_active": 0, 2024-06-04T05:24:24.381 INFO:teuthology.orchestra.run.smithi112.stdout: "command_send": 0, 2024-06-04T05:24:24.381 INFO:teuthology.orchestra.run.smithi112.stdout: "command_resend": 0, 2024-06-04T05:24:24.381 INFO:teuthology.orchestra.run.smithi112.stdout: "map_epoch": 122, 2024-06-04T05:24:24.381 INFO:teuthology.orchestra.run.smithi112.stdout: "map_full": 0, 2024-06-04T05:24:24.381 INFO:teuthology.orchestra.run.smithi112.stdout: "map_inc": 43, 2024-06-04T05:24:24.381 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_sessions": 11, 2024-06-04T05:24:24.381 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_session_open": 11, 2024-06-04T05:24:24.381 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_session_close": 0, 2024-06-04T05:24:24.381 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_laggy": 0, 2024-06-04T05:24:24.381 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_wr": 0, 2024-06-04T05:24:24.381 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_rd": 2, 2024-06-04T05:24:24.381 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_del": 0 2024-06-04T05:24:24.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.382 INFO:teuthology.orchestra.run.smithi112.stdout: "oft": { 2024-06-04T05:24:24.382 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_objs": 0, 2024-06-04T05:24:24.382 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_kv_pairs": 0, 2024-06-04T05:24:24.382 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_updates": 0, 2024-06-04T05:24:24.382 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_removes": 0 2024-06-04T05:24:24.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.382 INFO:teuthology.orchestra.run.smithi112.stdout: "osd-slow-ops": { 2024-06-04T05:24:24.382 INFO:teuthology.orchestra.run.smithi112.stdout: "slow_ops_count": 0 2024-06-04T05:24:24.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.382 INFO:teuthology.orchestra.run.smithi112.stdout: "purge_queue": { 2024-06-04T05:24:24.382 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_ops": 0, 2024-06-04T05:24:24.382 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_ops_high_water": 0, 2024-06-04T05:24:24.382 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing": 0, 2024-06-04T05:24:24.383 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_high_water": 0, 2024-06-04T05:24:24.383 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executed_ops": 0, 2024-06-04T05:24:24.383 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executed": 0, 2024-06-04T05:24:24.383 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_item_in_journal": 0 2024-06-04T05:24:24.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.383 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-04T05:24:24.383 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:24.383 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 104857600, 2024-06-04T05:24:24.383 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:24.383 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 7333, 2024-06-04T05:24:24.383 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 281815965, 2024-06-04T05:24:24.383 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:24.383 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 7333, 2024-06-04T05:24:24.383 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-06-04T05:24:24.383 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-06-04T05:24:24.384 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 7333, 2024-06-04T05:24:24.384 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 281815965, 2024-06-04T05:24:24.384 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:24.384 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.384 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.384 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.384 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:24.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.384 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-objecter_bytes": { 2024-06-04T05:24:24.384 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:24.384 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 104857600, 2024-06-04T05:24:24.385 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:24.385 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-06-04T05:24:24.385 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-06-04T05:24:24.385 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:24.385 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:24.385 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 6725, 2024-06-04T05:24:24.385 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 278659815, 2024-06-04T05:24:24.385 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 6725, 2024-06-04T05:24:24.385 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 278659815, 2024-06-04T05:24:24.385 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:24.385 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.385 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.385 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.386 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:24.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.386 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-objecter_ops": { 2024-06-04T05:24:24.386 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:24.386 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 1024, 2024-06-04T05:24:24.386 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:24.386 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-06-04T05:24:24.386 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-06-04T05:24:24.386 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:24.386 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:24.386 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 6725, 2024-06-04T05:24:24.386 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 6725, 2024-06-04T05:24:24.386 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 6725, 2024-06-04T05:24:24.387 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 6725, 2024-06-04T05:24:24.387 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:24.387 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.387 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.387 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.387 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:24.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.387 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-write_buf_throttle": { 2024-06-04T05:24:24.387 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:24.387 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 3758096384, 2024-06-04T05:24:24.387 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:24.387 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-06-04T05:24:24.387 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-06-04T05:24:24.388 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:24.388 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:24.388 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-06-04T05:24:24.388 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-06-04T05:24:24.388 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 0, 2024-06-04T05:24:24.388 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 0, 2024-06-04T05:24:24.388 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:24.388 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.388 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.388 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.388 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:24.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-06-04T05:24:24.388 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-write_buf_throttle-0x55f9be3306e0": { 2024-06-04T05:24:24.389 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-06-04T05:24:24.389 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 3758096384, 2024-06-04T05:24:24.389 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-06-04T05:24:24.389 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-06-04T05:24:24.389 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-06-04T05:24:24.389 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-06-04T05:24:24.389 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-06-04T05:24:24.389 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-06-04T05:24:24.389 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-06-04T05:24:24.389 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 0, 2024-06-04T05:24:24.389 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 0, 2024-06-04T05:24:24.389 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-06-04T05:24:24.389 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-06-04T05:24:24.389 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-06-04T05:24:24.390 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-06-04T05:24:24.390 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:24.390 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-06-04T05:24:24.390 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-06-04T05:24:25.316 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-06-04T05:24:25.316 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-06-04T05:24:25.316 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=7 2024-06-04T05:24:25.316 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-06-04T05:24:25.317 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-06-04T05:24:25.317 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-06-04T05:24:25.328 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-06-04T05:24:25.328 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T05:24:25.328 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-04T05:24:25.356 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T05:24:25.356 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-04T05:24:25.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-06-04T05:24:25.703 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:24:25.825 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:25 smithi070 ceph-mon[32631]: pgmap v860: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 90 B/s wr, 10 op/s 2024-06-04T05:24:25.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:25 smithi112 ceph-mon[36453]: pgmap v860: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 90 B/s wr, 10 op/s 2024-06-04T05:24:25.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:25 smithi098 ceph-mon[35569]: pgmap v860: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 90 B/s wr, 10 op/s 2024-06-04T05:24:27.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.633+0000 7f710ed75700 1 -- 172.21.15.70:0/4093715897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71081095a0 msgr2=0x7f7108109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:27.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.633+0000 7f710ed75700 1 --2- 172.21.15.70:0/4093715897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71081095a0 0x7f7108109980 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f7100009990 tx=0x7f7100009ca0 comp rx=0 tx=0).stop 2024-06-04T05:24:27.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.633+0000 7f710ed75700 1 -- 172.21.15.70:0/4093715897 shutdown_connections 2024-06-04T05:24:27.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.633+0000 7f710ed75700 1 --2- 172.21.15.70:0/4093715897 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f710806c470 0x7f71081103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:27.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.633+0000 7f710ed75700 1 --2- 172.21.15.70:0/4093715897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71081095a0 0x7f7108109980 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:27.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.633+0000 7f710ed75700 1 --2- 172.21.15.70:0/4093715897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f710806bab0 0x7f710806bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:27.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.633+0000 7f710ed75700 1 -- 172.21.15.70:0/4093715897 >> 172.21.15.70:0/4093715897 conn(0x7f71080793b0 msgr2=0x7f71080797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:27.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.633+0000 7f710ed75700 1 -- 172.21.15.70:0/4093715897 shutdown_connections 2024-06-04T05:24:27.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.633+0000 7f710ed75700 1 -- 172.21.15.70:0/4093715897 wait complete. 2024-06-04T05:24:27.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.634+0000 7f710ed75700 1 Processor -- start 2024-06-04T05:24:27.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.634+0000 7f710ed75700 1 -- start start 2024-06-04T05:24:27.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.634+0000 7f710ed75700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f710806bab0 0x7f7108102500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:27.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.634+0000 7f710ed75700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f710806c470 0x7f7108102a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:27.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.634+0000 7f710ed75700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71081095a0 0x7f7108102f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:27.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.634+0000 7f710ed75700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7108113580 con 0x7f710806bab0 2024-06-04T05:24:27.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.634+0000 7f710ed75700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7108113400 con 0x7f71081095a0 2024-06-04T05:24:27.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.634+0000 7f710ed75700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7108113700 con 0x7f710806c470 2024-06-04T05:24:27.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.634+0000 7f710cb11700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f710806bab0 0x7f7108102500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:27.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.634+0000 7f710cb11700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f710806bab0 0x7f7108102500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35982/0 (socket says 172.21.15.70:35982) 2024-06-04T05:24:27.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.634+0000 7f710cb11700 1 -- 172.21.15.70:0/2658432934 learned_addr learned my addr 172.21.15.70:0/2658432934 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:24:27.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.634+0000 7f7107fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f710806c470 0x7f7108102a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:27.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.634+0000 7f710d312700 1 --2- 172.21.15.70:0/2658432934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71081095a0 0x7f7108102f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:27.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.635+0000 7f710cb11700 1 -- 172.21.15.70:0/2658432934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f710806c470 msgr2=0x7f7108102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:27.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.635+0000 7f710cb11700 1 --2- 172.21.15.70:0/2658432934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f710806c470 0x7f7108102a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:27.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.635+0000 7f710cb11700 1 -- 172.21.15.70:0/2658432934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71081095a0 msgr2=0x7f7108102f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:27.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.635+0000 7f710cb11700 1 --2- 172.21.15.70:0/2658432934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71081095a0 0x7f7108102f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:27.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.635+0000 7f710cb11700 1 -- 172.21.15.70:0/2658432934 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71081a8080 con 0x7f710806bab0 2024-06-04T05:24:27.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.635+0000 7f7107fff700 1 --2- 172.21.15.70:0/2658432934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f710806c470 0x7f7108102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-04T05:24:27.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.635+0000 7f710cb11700 1 --2- 172.21.15.70:0/2658432934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f710806bab0 0x7f7108102500 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f7100006010 tx=0x7f71000047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:27.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.635+0000 7f7105ffb700 1 -- 172.21.15.70:0/2658432934 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7100027070 con 0x7f710806bab0 2024-06-04T05:24:27.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.635+0000 7f7105ffb700 1 -- 172.21.15.70:0/2658432934 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7100022470 con 0x7f710806bab0 2024-06-04T05:24:27.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.635+0000 7f710ed75700 1 -- 172.21.15.70:0/2658432934 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71081a8310 con 0x7f710806bab0 2024-06-04T05:24:27.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.635+0000 7f7105ffb700 1 -- 172.21.15.70:0/2658432934 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7100017650 con 0x7f710806bab0 2024-06-04T05:24:27.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.635+0000 7f710ed75700 1 -- 172.21.15.70:0/2658432934 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71081a8760 con 0x7f710806bab0 2024-06-04T05:24:27.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.636+0000 7f710ed75700 1 -- 172.21.15.70:0/2658432934 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f710810ba80 con 0x7f710806bab0 2024-06-04T05:24:27.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.637+0000 7f7105ffb700 1 -- 172.21.15.70:0/2658432934 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f7100009d10 con 0x7f710806bab0 2024-06-04T05:24:27.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.638+0000 7f7105ffb700 1 --2- 172.21.15.70:0/2658432934 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f70f40b29f0 0x7f70f40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:27.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.638+0000 7f7107fff700 1 --2- 172.21.15.70:0/2658432934 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f70f40b29f0 0x7f70f40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:27.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.638+0000 7f7105ffb700 1 -- 172.21.15.70:0/2658432934 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8571+0+0 (secure 0 0 0) 0x7f71000e33f0 con 0x7f710806bab0 2024-06-04T05:24:27.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.639+0000 7f7107fff700 1 --2- 172.21.15.70:0/2658432934 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f70f40b29f0 0x7f70f40b4eb0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7108079af0 tx=0x7f70f0009260 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:27.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.642+0000 7f7105ffb700 1 -- 172.21.15.70:0/2658432934 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7100013070 con 0x7f710806bab0 2024-06-04T05:24:27.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:27 smithi070 ceph-mon[32631]: pgmap v861: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 15 op/s 2024-06-04T05:24:27.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:27 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:24:27.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:27 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:24:27.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:27 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:24:27.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:27 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:24:27.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.861+0000 7f710ed75700 1 -- 172.21.15.70:0/2658432934 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f710806a230 con 0x7f710806bab0 2024-06-04T05:24:27.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.862+0000 7f7105ffb700 1 -- 172.21.15.70:0/2658432934 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v122) ==== 81+0+943 (secure 0 0 0) 0x7f71000aae30 con 0x7f710806bab0 2024-06-04T05:24:27.864 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/4291129442 2024-06-05T04:57:09.937704+0000 2024-06-04T05:24:27.864 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/3584175673 2024-06-05T04:57:09.937704+0000 2024-06-04T05:24:27.864 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/3720789940 2024-06-05T04:57:09.937704+0000 2024-06-04T05:24:27.864 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/145533746 2024-06-05T04:57:09.937704+0000 2024-06-04T05:24:27.864 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/3442074771 2024-06-05T04:46:17.636907+0000 2024-06-04T05:24:27.864 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6801/1353782413 2024-06-05T04:46:17.636907+0000 2024-06-04T05:24:27.864 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/1442095493 2024-06-05T04:46:17.636907+0000 2024-06-04T05:24:27.864 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/2453733268 2024-06-05T04:45:49.501864+0000 2024-06-04T05:24:27.865 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6801/2752966840 2024-06-05T04:57:09.937704+0000 2024-06-04T05:24:27.865 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/1369781447 2024-06-05T04:45:49.501864+0000 2024-06-04T05:24:27.865 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6800/2087909610 2024-06-05T04:45:49.501864+0000 2024-06-04T05:24:27.865 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6801/2087909610 2024-06-05T04:45:49.501864+0000 2024-06-04T05:24:27.865 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6800/2752966840 2024-06-05T04:57:09.937704+0000 2024-06-04T05:24:27.865 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6800/1353782413 2024-06-05T04:46:17.636907+0000 2024-06-04T05:24:27.865 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/915186456 2024-06-05T04:46:17.636907+0000 2024-06-04T05:24:27.865 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/860033006 2024-06-05T04:45:49.501864+0000 2024-06-04T05:24:27.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.864+0000 7f710ed75700 1 -- 172.21.15.70:0/2658432934 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f70f40b29f0 msgr2=0x7f70f40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:27.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.864+0000 7f710ed75700 1 --2- 172.21.15.70:0/2658432934 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f70f40b29f0 0x7f70f40b4eb0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7108079af0 tx=0x7f70f0009260 comp rx=0 tx=0).stop 2024-06-04T05:24:27.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.864+0000 7f710ed75700 1 -- 172.21.15.70:0/2658432934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f710806bab0 msgr2=0x7f7108102500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:27.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.864+0000 7f710ed75700 1 --2- 172.21.15.70:0/2658432934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f710806bab0 0x7f7108102500 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f7100006010 tx=0x7f71000047b0 comp rx=0 tx=0).stop 2024-06-04T05:24:27.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.864+0000 7f710ed75700 1 -- 172.21.15.70:0/2658432934 shutdown_connections 2024-06-04T05:24:27.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.864+0000 7f710ed75700 1 --2- 172.21.15.70:0/2658432934 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f70f40b29f0 0x7f70f40b4eb0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:27.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.864+0000 7f710ed75700 1 --2- 172.21.15.70:0/2658432934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71081095a0 0x7f7108102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:27.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.864+0000 7f710ed75700 1 --2- 172.21.15.70:0/2658432934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f710806bab0 0x7f7108102500 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:27.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.864+0000 7f710ed75700 1 --2- 172.21.15.70:0/2658432934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f710806c470 0x7f7108102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:27.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.864+0000 7f710ed75700 1 -- 172.21.15.70:0/2658432934 >> 172.21.15.70:0/2658432934 conn(0x7f71080793b0 msgr2=0x7f7108101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:27.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.864+0000 7f710ed75700 1 -- 172.21.15.70:0/2658432934 shutdown_connections 2024-06-04T05:24:27.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:27.864+0000 7f710ed75700 1 -- 172.21.15.70:0/2658432934 wait complete. 2024-06-04T05:24:27.867 INFO:teuthology.orchestra.run.smithi070.stderr:listed 16 entries 2024-06-04T05:24:27.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:27 smithi112 ceph-mon[36453]: pgmap v861: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 15 op/s 2024-06-04T05:24:27.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:27 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:24:27.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:27 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:24:27.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:27 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:24:27.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:27 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:24:27.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:27 smithi098 ceph-mon[35569]: pgmap v861: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 15 op/s 2024-06-04T05:24:27.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:27 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-04T05:24:27.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:27 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-04T05:24:27.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:27 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-04T05:24:27.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:27 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' 2024-06-04T05:24:28.606 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-06-04T05:24:28.606 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-04T05:24:28.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-06-04T05:24:28.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:28 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2658432934' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-04T05:24:28.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:28 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2658432934' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-04T05:24:28.903 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:24:28.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:28 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2658432934' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-04T05:24:29.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:29 smithi112 ceph-mon[36453]: pgmap v862: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-06-04T05:24:29.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:29 smithi098 ceph-mon[35569]: pgmap v862: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-06-04T05:24:30.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:29 smithi070 ceph-mon[32631]: pgmap v862: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-06-04T05:24:30.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.752+0000 7ff8b47f6700 1 -- 172.21.15.70:0/3160086699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8ac108520 msgr2=0x7ff8ac108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:30.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.752+0000 7ff8b47f6700 1 --2- 172.21.15.70:0/3160086699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8ac108520 0x7ff8ac108900 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7ff89c009990 tx=0x7ff89c009ca0 comp rx=0 tx=0).stop 2024-06-04T05:24:30.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.752+0000 7ff8b47f6700 1 -- 172.21.15.70:0/3160086699 shutdown_connections 2024-06-04T05:24:30.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.752+0000 7ff8b47f6700 1 --2- 172.21.15.70:0/3160086699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8ac102e00 0x7ff8ac10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:30.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.752+0000 7ff8b47f6700 1 --2- 172.21.15.70:0/3160086699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8ac108520 0x7ff8ac108900 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:30.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.752+0000 7ff8b47f6700 1 --2- 172.21.15.70:0/3160086699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8ac102440 0x7ff8ac1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:30.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.752+0000 7ff8b47f6700 1 -- 172.21.15.70:0/3160086699 >> 172.21.15.70:0/3160086699 conn(0x7ff8ac0793f0 msgr2=0x7ff8ac079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:30.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.752+0000 7ff8b47f6700 1 -- 172.21.15.70:0/3160086699 shutdown_connections 2024-06-04T05:24:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.752+0000 7ff8b47f6700 1 -- 172.21.15.70:0/3160086699 wait complete. 2024-06-04T05:24:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.753+0000 7ff8b47f6700 1 Processor -- start 2024-06-04T05:24:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.753+0000 7ff8b47f6700 1 -- start start 2024-06-04T05:24:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.753+0000 7ff8b47f6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8ac102440 0x7ff8ac19f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.753+0000 7ff8b47f6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8ac102e00 0x7ff8ac19fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.753+0000 7ff8b47f6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8ac108520 0x7ff8ac199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.753+0000 7ff8b47f6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac112500 con 0x7ff8ac102440 2024-06-04T05:24:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.753+0000 7ff8b47f6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac112380 con 0x7ff8ac102e00 2024-06-04T05:24:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.753+0000 7ff8b47f6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac112680 con 0x7ff8ac108520 2024-06-04T05:24:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.753+0000 7ff8b2592700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8ac102440 0x7ff8ac19f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.753+0000 7ff8b2592700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8ac102440 0x7ff8ac19f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36000/0 (socket says 172.21.15.70:36000) 2024-06-04T05:24:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.753+0000 7ff8b2592700 1 -- 172.21.15.70:0/204693892 learned_addr learned my addr 172.21.15.70:0/204693892 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:24:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.753+0000 7ff8b1d91700 1 --2- 172.21.15.70:0/204693892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8ac102e00 0x7ff8ac19fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.753+0000 7ff8b2d93700 1 --2- 172.21.15.70:0/204693892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8ac108520 0x7ff8ac199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:30.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.753+0000 7ff8b2592700 1 -- 172.21.15.70:0/204693892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8ac108520 msgr2=0x7ff8ac199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:30.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.753+0000 7ff8b2592700 1 --2- 172.21.15.70:0/204693892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8ac108520 0x7ff8ac199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:30.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.753+0000 7ff8b2592700 1 -- 172.21.15.70:0/204693892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8ac102e00 msgr2=0x7ff8ac19fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:30.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.754+0000 7ff8b2592700 1 --2- 172.21.15.70:0/204693892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8ac102e00 0x7ff8ac19fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:30.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.754+0000 7ff8b2592700 1 -- 172.21.15.70:0/204693892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8ac19a0f0 con 0x7ff8ac102440 2024-06-04T05:24:30.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.754+0000 7ff8b1d91700 1 --2- 172.21.15.70:0/204693892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8ac102e00 0x7ff8ac19fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:24:30.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.754+0000 7ff8b2d93700 1 --2- 172.21.15.70:0/204693892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8ac108520 0x7ff8ac199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-04T05:24:30.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.754+0000 7ff8b2592700 1 --2- 172.21.15.70:0/204693892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8ac102440 0x7ff8ac19f780 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7ff89c0057f0 tx=0x7ff89c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:30.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.754+0000 7ff8a37fe700 1 -- 172.21.15.70:0/204693892 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c027070 con 0x7ff8ac102440 2024-06-04T05:24:30.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.754+0000 7ff8a37fe700 1 -- 172.21.15.70:0/204693892 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff89c00fd60 con 0x7ff8ac102440 2024-06-04T05:24:30.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.754+0000 7ff8b47f6700 1 -- 172.21.15.70:0/204693892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8ac19a380 con 0x7ff8ac102440 2024-06-04T05:24:30.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.754+0000 7ff8a37fe700 1 -- 172.21.15.70:0/204693892 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c022d70 con 0x7ff8ac102440 2024-06-04T05:24:30.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.754+0000 7ff8b47f6700 1 -- 172.21.15.70:0/204693892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8ac1a6650 con 0x7ff8ac102440 2024-06-04T05:24:30.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.755+0000 7ff8b47f6700 1 -- 172.21.15.70:0/204693892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8ac06a230 con 0x7ff8ac102440 2024-06-04T05:24:30.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.756+0000 7ff8a37fe700 1 -- 172.21.15.70:0/204693892 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7ff89c02a070 con 0x7ff8ac102440 2024-06-04T05:24:30.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.757+0000 7ff8a37fe700 1 --2- 172.21.15.70:0/204693892 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8980b2a70 0x7ff8980b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:30.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.757+0000 7ff8b1d91700 1 --2- 172.21.15.70:0/204693892 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8980b2a70 0x7ff8980b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:30.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.757+0000 7ff8a37fe700 1 -- 172.21.15.70:0/204693892 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8571+0+0 (secure 0 0 0) 0x7ff89c0e3580 con 0x7ff8ac102440 2024-06-04T05:24:30.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.757+0000 7ff8b1d91700 1 --2- 172.21.15.70:0/204693892 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8980b2a70 0x7ff8980b4f30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8005fd0 tx=0x7ff8a8005e30 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:30.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.762+0000 7ff8a37fe700 1 -- 172.21.15.70:0/204693892 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff89c013070 con 0x7ff8ac102440 2024-06-04T05:24:30.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.979+0000 7ff8b47f6700 1 -- 172.21.15.70:0/204693892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff8ac19ae70 con 0x7ff8ac102440 2024-06-04T05:24:30.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.979+0000 7ff8a37fe700 1 -- 172.21.15.70:0/204693892 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v122) ==== 81+0+943 (secure 0 0 0) 0x7ff89c0aafc0 con 0x7ff8ac102440 2024-06-04T05:24:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/4291129442 2024-06-05T04:57:09.937704+0000 2024-06-04T05:24:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/3584175673 2024-06-05T04:57:09.937704+0000 2024-06-04T05:24:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/3720789940 2024-06-05T04:57:09.937704+0000 2024-06-04T05:24:30.981 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/145533746 2024-06-05T04:57:09.937704+0000 2024-06-04T05:24:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/3442074771 2024-06-05T04:46:17.636907+0000 2024-06-04T05:24:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6801/1353782413 2024-06-05T04:46:17.636907+0000 2024-06-04T05:24:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/1442095493 2024-06-05T04:46:17.636907+0000 2024-06-04T05:24:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/2453733268 2024-06-05T04:45:49.501864+0000 2024-06-04T05:24:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6801/2752966840 2024-06-05T04:57:09.937704+0000 2024-06-04T05:24:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/1369781447 2024-06-05T04:45:49.501864+0000 2024-06-04T05:24:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6800/2087909610 2024-06-05T04:45:49.501864+0000 2024-06-04T05:24:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6801/2087909610 2024-06-05T04:45:49.501864+0000 2024-06-04T05:24:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6800/2752966840 2024-06-05T04:57:09.937704+0000 2024-06-04T05:24:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6800/1353782413 2024-06-05T04:46:17.636907+0000 2024-06-04T05:24:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/915186456 2024-06-05T04:46:17.636907+0000 2024-06-04T05:24:30.982 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/860033006 2024-06-05T04:45:49.501864+0000 2024-06-04T05:24:30.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.981+0000 7ff8b47f6700 1 -- 172.21.15.70:0/204693892 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8980b2a70 msgr2=0x7ff8980b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:30.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.981+0000 7ff8b47f6700 1 --2- 172.21.15.70:0/204693892 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8980b2a70 0x7ff8980b4f30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8005fd0 tx=0x7ff8a8005e30 comp rx=0 tx=0).stop 2024-06-04T05:24:30.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.981+0000 7ff8b47f6700 1 -- 172.21.15.70:0/204693892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8ac102440 msgr2=0x7ff8ac19f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:30.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.981+0000 7ff8b47f6700 1 --2- 172.21.15.70:0/204693892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8ac102440 0x7ff8ac19f780 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7ff89c0057f0 tx=0x7ff89c009f90 comp rx=0 tx=0).stop 2024-06-04T05:24:30.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.982+0000 7ff8b47f6700 1 -- 172.21.15.70:0/204693892 shutdown_connections 2024-06-04T05:24:30.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.982+0000 7ff8b47f6700 1 --2- 172.21.15.70:0/204693892 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7ff8980b2a70 0x7ff8980b4f30 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:30.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.982+0000 7ff8b47f6700 1 --2- 172.21.15.70:0/204693892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8ac108520 0x7ff8ac199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:30.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.982+0000 7ff8b47f6700 1 --2- 172.21.15.70:0/204693892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8ac102440 0x7ff8ac19f780 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:30.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.982+0000 7ff8b47f6700 1 --2- 172.21.15.70:0/204693892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8ac102e00 0x7ff8ac19fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:30.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.982+0000 7ff8b47f6700 1 -- 172.21.15.70:0/204693892 >> 172.21.15.70:0/204693892 conn(0x7ff8ac0793f0 msgr2=0x7ff8ac06b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:30.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.982+0000 7ff8b47f6700 1 -- 172.21.15.70:0/204693892 shutdown_connections 2024-06-04T05:24:30.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:30.982+0000 7ff8b47f6700 1 -- 172.21.15.70:0/204693892 wait complete. 2024-06-04T05:24:30.985 INFO:teuthology.orchestra.run.smithi070.stderr:listed 16 entries 2024-06-04T05:24:31.793 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-06-04T05:24:31.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T05:24:31.793 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-06-04T05:24:31.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[89512]: fuse finished with error 0 and tester_r 0 2024-06-04T05:24:31.838 INFO:teuthology.orchestra.run:waiting for 300 2024-06-04T05:24:31.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:31 smithi112 ceph-mon[36453]: pgmap v863: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-06-04T05:24:31.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:31 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/204693892' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-04T05:24:31.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:31 smithi098 ceph-mon[35569]: pgmap v863: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-06-04T05:24:31.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:31 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/204693892' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-04T05:24:32.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:31 smithi070 ceph-mon[32631]: pgmap v863: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-06-04T05:24:32.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:31 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/204693892' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-04T05:24:33.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:33 smithi112 ceph-mon[36453]: pgmap v864: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 14 op/s 2024-06-04T05:24:33.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:33 smithi098 ceph-mon[35569]: pgmap v864: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 14 op/s 2024-06-04T05:24:34.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:33 smithi070 ceph-mon[32631]: pgmap v864: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 14 op/s 2024-06-04T05:24:35.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:35 smithi112 ceph-mon[36453]: pgmap v865: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:24:35.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:35 smithi098 ceph-mon[35569]: pgmap v865: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:24:36.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:35 smithi070 ceph-mon[32631]: pgmap v865: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-04T05:24:37.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:37 smithi112 ceph-mon[36453]: pgmap v866: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 16 op/s 2024-06-04T05:24:37.939 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-06-04T05:24:37.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T05:24:37.940 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-04T05:24:37.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T05:24:37.976 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-06-04T05:24:37.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:37 smithi098 ceph-mon[35569]: pgmap v866: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 16 op/s 2024-06-04T05:24:38.045 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-04T05:24:38.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T05:24:38.045 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-04T05:24:38.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:37 smithi070 ceph-mon[32631]: pgmap v866: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 16 op/s 2024-06-04T05:24:38.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-04T05:24:38.116 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link delete ceph-brx 2024-06-04T05:24:38.226 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-06-04T05:24:38.238 INFO:tasks.cephadm:Teardown begin 2024-06-04T05:24:38.238 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-04T05:24:38.239 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:24:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:24:39.909 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:39 smithi070 ceph-mon[32631]: pgmap v867: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-06-04T05:24:39.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:39 smithi112 ceph-mon[36453]: pgmap v867: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-06-04T05:24:39.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:39 smithi098 ceph-mon[35569]: pgmap v867: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-06-04T05:24:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.254+0000 7fc9d11f9700 1 -- 172.21.15.70:0/3183452039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9cc102df0 msgr2=0x7fc9cc10f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.254+0000 7fc9d11f9700 1 --2- 172.21.15.70:0/3183452039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9cc102df0 0x7fc9cc10f2e0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fc9c400b0a0 tx=0x7fc9c400b3b0 comp rx=0 tx=0).stop 2024-06-04T05:24:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.254+0000 7fc9d11f9700 1 -- 172.21.15.70:0/3183452039 shutdown_connections 2024-06-04T05:24:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.254+0000 7fc9d11f9700 1 --2- 172.21.15.70:0/3183452039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9cc102df0 0x7fc9cc10f2e0 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.254+0000 7fc9d11f9700 1 --2- 172.21.15.70:0/3183452039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9cc108510 0x7fc9cc1088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.254+0000 7fc9d11f9700 1 --2- 172.21.15.70:0/3183452039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9cc102430 0x7fc9cc1028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.254+0000 7fc9d11f9700 1 -- 172.21.15.70:0/3183452039 >> 172.21.15.70:0/3183452039 conn(0x7fc9cc0793b0 msgr2=0x7fc9cc0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.254+0000 7fc9d11f9700 1 -- 172.21.15.70:0/3183452039 shutdown_connections 2024-06-04T05:24:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.254+0000 7fc9d11f9700 1 -- 172.21.15.70:0/3183452039 wait complete. 2024-06-04T05:24:40.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.255+0000 7fc9d11f9700 1 Processor -- start 2024-06-04T05:24:40.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.255+0000 7fc9d11f9700 1 -- start start 2024-06-04T05:24:40.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.255+0000 7fc9d11f9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9cc102430 0x7fc9cc104070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:40.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.255+0000 7fc9d11f9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9cc102df0 0x7fc9cc1045b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:40.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.255+0000 7fc9d11f9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9cc108510 0x7fc9cc104af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:40.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9d11f9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc9cc1124f0 con 0x7fc9cc102df0 2024-06-04T05:24:40.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9d11f9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9cc112370 con 0x7fc9cc108510 2024-06-04T05:24:40.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9d11f9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc9cc112670 con 0x7fc9cc102430 2024-06-04T05:24:40.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9ca59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9cc102df0 0x7fc9cc1045b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:40.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9ca59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9cc102df0 0x7fc9cc1045b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53728/0 (socket says 172.21.15.70:53728) 2024-06-04T05:24:40.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9ca59c700 1 -- 172.21.15.70:0/1141055407 learned_addr learned my addr 172.21.15.70:0/1141055407 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:24:40.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9cb59e700 1 --2- 172.21.15.70:0/1141055407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9cc108510 0x7fc9cc104af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:40.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9cad9d700 1 --2- 172.21.15.70:0/1141055407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9cc102430 0x7fc9cc104070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:40.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9ca59c700 1 -- 172.21.15.70:0/1141055407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9cc102430 msgr2=0x7fc9cc104070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:40.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9ca59c700 1 --2- 172.21.15.70:0/1141055407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9cc102430 0x7fc9cc104070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:40.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9ca59c700 1 -- 172.21.15.70:0/1141055407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9cc108510 msgr2=0x7fc9cc104af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:40.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9ca59c700 1 --2- 172.21.15.70:0/1141055407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9cc108510 0x7fc9cc104af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:40.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9ca59c700 1 -- 172.21.15.70:0/1141055407 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9cc1a8000 con 0x7fc9cc102df0 2024-06-04T05:24:40.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9cb59e700 1 --2- 172.21.15.70:0/1141055407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9cc108510 0x7fc9cc104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:24:40.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9ca59c700 1 --2- 172.21.15.70:0/1141055407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9cc102df0 0x7fc9cc1045b0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fc9c000b530 tx=0x7fc9c000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:40.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9b3fff700 1 -- 172.21.15.70:0/1141055407 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c0015020 con 0x7fc9cc102df0 2024-06-04T05:24:40.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9d11f9700 1 -- 172.21.15.70:0/1141055407 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9cc1a82f0 con 0x7fc9cc102df0 2024-06-04T05:24:40.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9b3fff700 1 -- 172.21.15.70:0/1141055407 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9c0011cc0 con 0x7fc9cc102df0 2024-06-04T05:24:40.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9b3fff700 1 -- 172.21.15.70:0/1141055407 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c0010d30 con 0x7fc9cc102df0 2024-06-04T05:24:40.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.256+0000 7fc9d11f9700 1 -- 172.21.15.70:0/1141055407 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9cc1a8830 con 0x7fc9cc102df0 2024-06-04T05:24:40.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.258+0000 7fc9d11f9700 1 -- 172.21.15.70:0/1141055407 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9cc06a230 con 0x7fc9cc102df0 2024-06-04T05:24:40.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.259+0000 7fc9b3fff700 1 -- 172.21.15.70:0/1141055407 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7fc9c0018020 con 0x7fc9cc102df0 2024-06-04T05:24:40.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.260+0000 7fc9b3fff700 1 --2- 172.21.15.70:0/1141055407 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9b40b29f0 0x7fc9b40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:40.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.260+0000 7fc9cad9d700 1 --2- 172.21.15.70:0/1141055407 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9b40b29f0 0x7fc9b40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:40.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.260+0000 7fc9b3fff700 1 -- 172.21.15.70:0/1141055407 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8571+0+0 (secure 0 0 0) 0x7fc9c00e1dc0 con 0x7fc9cc102df0 2024-06-04T05:24:40.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.260+0000 7fc9cad9d700 1 --2- 172.21.15.70:0/1141055407 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9b40b29f0 0x7fc9b40b4eb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc005ea0 tx=0x7fc9bc005e30 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:40.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.264+0000 7fc9b3fff700 1 -- 172.21.15.70:0/1141055407 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc9c001f020 con 0x7fc9cc102df0 2024-06-04T05:24:40.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.519+0000 7fc9d11f9700 1 -- 172.21.15.70:0/1141055407 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9cc02e560 con 0x7fc9cc102df0 2024-06-04T05:24:40.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.520+0000 7fc9b3fff700 1 -- 172.21.15.70:0/1141055407 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+11944 (secure 0 0 0) 0x7fc9c00a9880 con 0x7fc9cc102df0 2024-06-04T05:24:40.522 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:24:40.522 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":72,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:23:17.104466+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:24:40.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.522+0000 7fc9d11f9700 1 -- 172.21.15.70:0/1141055407 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9b40b29f0 msgr2=0x7fc9b40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:40.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.522+0000 7fc9d11f9700 1 --2- 172.21.15.70:0/1141055407 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9b40b29f0 0x7fc9b40b4eb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc005ea0 tx=0x7fc9bc005e30 comp rx=0 tx=0).stop 2024-06-04T05:24:40.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.522+0000 7fc9d11f9700 1 -- 172.21.15.70:0/1141055407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9cc102df0 msgr2=0x7fc9cc1045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:40.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.522+0000 7fc9d11f9700 1 --2- 172.21.15.70:0/1141055407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9cc102df0 0x7fc9cc1045b0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fc9c000b530 tx=0x7fc9c000b8f0 comp rx=0 tx=0).stop 2024-06-04T05:24:40.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.522+0000 7fc9d11f9700 1 -- 172.21.15.70:0/1141055407 shutdown_connections 2024-06-04T05:24:40.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.522+0000 7fc9d11f9700 1 --2- 172.21.15.70:0/1141055407 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7fc9b40b29f0 0x7fc9b40b4eb0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:40.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.522+0000 7fc9d11f9700 1 --2- 172.21.15.70:0/1141055407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9cc108510 0x7fc9cc104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:40.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.522+0000 7fc9d11f9700 1 --2- 172.21.15.70:0/1141055407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9cc102430 0x7fc9cc104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:40.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.522+0000 7fc9d11f9700 1 --2- 172.21.15.70:0/1141055407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9cc102df0 0x7fc9cc1045b0 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:40.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.522+0000 7fc9d11f9700 1 -- 172.21.15.70:0/1141055407 >> 172.21.15.70:0/1141055407 conn(0x7fc9cc0793b0 msgr2=0x7fc9cc06be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:40.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.522+0000 7fc9d11f9700 1 -- 172.21.15.70:0/1141055407 shutdown_connections 2024-06-04T05:24:40.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:40.522+0000 7fc9d11f9700 1 -- 172.21.15.70:0/1141055407 wait complete. 2024-06-04T05:24:40.525 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 72 2024-06-04T05:24:40.817 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:40 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1141055407' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:24:40.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:40 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1141055407' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:24:40.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:40 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1141055407' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:24:41.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-04T05:24:41.495 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:24:41.871 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:41 smithi070 ceph-mon[32631]: pgmap v868: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-06-04T05:24:41.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:41 smithi112 ceph-mon[36453]: pgmap v868: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-06-04T05:24:41.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:41 smithi098 ceph-mon[35569]: pgmap v868: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-06-04T05:24:43.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.309+0000 7f73bbc27700 1 -- 172.21.15.70:0/3531198675 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73b4101ee0 msgr2=0x7f73b4102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:43.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.309+0000 7f73bbc27700 1 --2- 172.21.15.70:0/3531198675 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73b4101ee0 0x7f73b4102360 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f73a40099c0 tx=0x7f73a4009cd0 comp rx=0 tx=0).stop 2024-06-04T05:24:43.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.309+0000 7f73bbc27700 1 -- 172.21.15.70:0/3531198675 shutdown_connections 2024-06-04T05:24:43.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.309+0000 7f73bbc27700 1 --2- 172.21.15.70:0/3531198675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b41028a0 0x7f73b4110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:43.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.309+0000 7f73bbc27700 1 --2- 172.21.15.70:0/3531198675 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f73b41085b0 0x7f73b4108990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:43.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.309+0000 7f73bbc27700 1 --2- 172.21.15.70:0/3531198675 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73b4101ee0 0x7f73b4102360 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:43.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.309+0000 7f73bbc27700 1 -- 172.21.15.70:0/3531198675 >> 172.21.15.70:0/3531198675 conn(0x7f73b40777e0 msgr2=0x7f73b4077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:43.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.309+0000 7f73bbc27700 1 -- 172.21.15.70:0/3531198675 shutdown_connections 2024-06-04T05:24:43.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.309+0000 7f73bbc27700 1 -- 172.21.15.70:0/3531198675 wait complete. 2024-06-04T05:24:43.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.310+0000 7f73bbc27700 1 Processor -- start 2024-06-04T05:24:43.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.310+0000 7f73bbc27700 1 -- start start 2024-06-04T05:24:43.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.310+0000 7f73bbc27700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b4101ee0 0x7f73b419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:43.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.310+0000 7f73bbc27700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73b41028a0 0x7f73b419fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:43.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.310+0000 7f73bbc27700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f73b41085b0 0x7f73b4199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:43.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.310+0000 7f73bbc27700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f73b4113630 con 0x7f73b41028a0 2024-06-04T05:24:43.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.310+0000 7f73bbc27700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f73b41134b0 con 0x7f73b4101ee0 2024-06-04T05:24:43.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.310+0000 7f73bbc27700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f73b41137b0 con 0x7f73b41085b0 2024-06-04T05:24:43.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.311+0000 7f73b91c2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73b41028a0 0x7f73b419fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:43.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.311+0000 7f73b91c2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73b41028a0 0x7f73b419fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53758/0 (socket says 172.21.15.70:53758) 2024-06-04T05:24:43.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.311+0000 7f73b91c2700 1 -- 172.21.15.70:0/3249586959 learned_addr learned my addr 172.21.15.70:0/3249586959 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:24:43.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.311+0000 7f73b99c3700 1 --2- 172.21.15.70:0/3249586959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b4101ee0 0x7f73b419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:43.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.311+0000 7f73ba1c4700 1 --2- 172.21.15.70:0/3249586959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f73b41085b0 0x7f73b4199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:43.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.311+0000 7f73b91c2700 1 -- 172.21.15.70:0/3249586959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f73b41085b0 msgr2=0x7f73b4199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:43.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.311+0000 7f73b91c2700 1 --2- 172.21.15.70:0/3249586959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f73b41085b0 0x7f73b4199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:43.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.311+0000 7f73b91c2700 1 -- 172.21.15.70:0/3249586959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b4101ee0 msgr2=0x7f73b419f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:43.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.311+0000 7f73b91c2700 1 --2- 172.21.15.70:0/3249586959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b4101ee0 0x7f73b419f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:43.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.311+0000 7f73b91c2700 1 -- 172.21.15.70:0/3249586959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73b419a090 con 0x7f73b41028a0 2024-06-04T05:24:43.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.311+0000 7f73b91c2700 1 --2- 172.21.15.70:0/3249586959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73b41028a0 0x7f73b419fc60 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f73a400b580 tx=0x7f73a40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:43.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.311+0000 7f73aaffd700 1 -- 172.21.15.70:0/3249586959 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a4027070 con 0x7f73b41028a0 2024-06-04T05:24:43.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.311+0000 7f73aaffd700 1 -- 172.21.15.70:0/3249586959 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f73a400fd30 con 0x7f73b41028a0 2024-06-04T05:24:43.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.311+0000 7f73bbc27700 1 -- 172.21.15.70:0/3249586959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73b419a320 con 0x7f73b41028a0 2024-06-04T05:24:43.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.311+0000 7f73aaffd700 1 -- 172.21.15.70:0/3249586959 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a4022e50 con 0x7f73b41028a0 2024-06-04T05:24:43.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.311+0000 7f73bbc27700 1 -- 172.21.15.70:0/3249586959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73b41a65f0 con 0x7f73b41028a0 2024-06-04T05:24:43.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.312+0000 7f73bbc27700 1 -- 172.21.15.70:0/3249586959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73b406b6b0 con 0x7f73b41028a0 2024-06-04T05:24:43.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.314+0000 7f73aaffd700 1 -- 172.21.15.70:0/3249586959 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f73a402a070 con 0x7f73b41028a0 2024-06-04T05:24:43.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.314+0000 7f73aaffd700 1 --2- 172.21.15.70:0/3249586959 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73a00b2c10 0x7f73a00b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:43.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.314+0000 7f73b99c3700 1 --2- 172.21.15.70:0/3249586959 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73a00b2c10 0x7f73a00b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:43.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.315+0000 7f73aaffd700 1 -- 172.21.15.70:0/3249586959 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8571+0+0 (secure 0 0 0) 0x7f73a40e36e0 con 0x7f73b41028a0 2024-06-04T05:24:43.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.315+0000 7f73b99c3700 1 --2- 172.21.15.70:0/3249586959 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73a00b2c10 0x7f73a00b50d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f73b0009550 tx=0x7f73b00091b0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:43.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.318+0000 7f73aaffd700 1 -- 172.21.15.70:0/3249586959 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f73a4013070 con 0x7f73b41028a0 2024-06-04T05:24:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.567+0000 7f73bbc27700 1 -- 172.21.15.70:0/3249586959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73b419ae10 con 0x7f73b41028a0 2024-06-04T05:24:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.568+0000 7f73aaffd700 1 -- 172.21.15.70:0/3249586959 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+11944 (secure 0 0 0) 0x7f73a40ab120 con 0x7f73b41028a0 2024-06-04T05:24:43.570 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:24:43.570 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2694011970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2694011970},{"type":"v1","addr":"172.21.15.70:6835","nonce":2694011970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/729112700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":729112700},{"type":"v1","addr":"172.21.15.70:6837","nonce":729112700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":72,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-04T04:53:38.237993+0000","modified":"2024-06-04T05:23:17.104466+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24448,"mds_1":24442,"mds_2":24440,"mds_3":24436,"mds_4":24410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14580":{"gid":14580,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/3817259285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3817259285},{"type":"v1","addr":"172.21.15.70:6839","nonce":3817259285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14592":{"gid":14592,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2748767095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2748767095},{"type":"v1","addr":"172.21.15.70:6841","nonce":2748767095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6833/1896647585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1896647585},{"type":"v1","addr":"172.21.15.98:6833","nonce":1896647585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24386":{"gid":24386,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/3148471566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3148471566},{"type":"v1","addr":"172.21.15.98:6835","nonce":3148471566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1359739019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1359739019},{"type":"v1","addr":"172.21.15.112:6833","nonce":1359739019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.98:6837/3160044154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3160044154},{"type":"v1","addr":"172.21.15.98:6837","nonce":3160044154}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6835/242926435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":242926435},{"type":"v1","addr":"172.21.15.112:6835","nonce":242926435}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"b","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6839/1361276220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1361276220},{"type":"v1","addr":"172.21.15.98:6839","nonce":1361276220}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.112:6837/3596492666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3596492666},{"type":"v1","addr":"172.21.15.112:6837","nonce":3596492666}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.112:6839/2996116087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2996116087},{"type":"v1","addr":"172.21.15.112:6839","nonce":2996116087}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24448,"qdb_cluster":[24410,24436,24440,24442,24448]},"id":1}]} 2024-06-04T05:24:43.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.570+0000 7f73bbc27700 1 -- 172.21.15.70:0/3249586959 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73a00b2c10 msgr2=0x7f73a00b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:43.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.570+0000 7f73bbc27700 1 --2- 172.21.15.70:0/3249586959 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73a00b2c10 0x7f73a00b50d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f73b0009550 tx=0x7f73b00091b0 comp rx=0 tx=0).stop 2024-06-04T05:24:43.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.570+0000 7f73bbc27700 1 -- 172.21.15.70:0/3249586959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73b41028a0 msgr2=0x7f73b419fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:43.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.570+0000 7f73bbc27700 1 --2- 172.21.15.70:0/3249586959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73b41028a0 0x7f73b419fc60 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f73a400b580 tx=0x7f73a40047b0 comp rx=0 tx=0).stop 2024-06-04T05:24:43.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.570+0000 7f73bbc27700 1 -- 172.21.15.70:0/3249586959 shutdown_connections 2024-06-04T05:24:43.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.570+0000 7f73bbc27700 1 --2- 172.21.15.70:0/3249586959 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73a00b2c10 0x7f73a00b50d0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:43.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.570+0000 7f73bbc27700 1 --2- 172.21.15.70:0/3249586959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f73b41085b0 0x7f73b4199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:43.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.570+0000 7f73bbc27700 1 --2- 172.21.15.70:0/3249586959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b4101ee0 0x7f73b419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:43.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.570+0000 7f73bbc27700 1 --2- 172.21.15.70:0/3249586959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73b41028a0 0x7f73b419fc60 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:43.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.570+0000 7f73bbc27700 1 -- 172.21.15.70:0/3249586959 >> 172.21.15.70:0/3249586959 conn(0x7f73b40777e0 msgr2=0x7f73b40fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:43.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.570+0000 7f73bbc27700 1 -- 172.21.15.70:0/3249586959 shutdown_connections 2024-06-04T05:24:43.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:43.570+0000 7f73bbc27700 1 -- 172.21.15.70:0/3249586959 wait complete. 2024-06-04T05:24:43.573 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 72 2024-06-04T05:24:43.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:43 smithi070 ceph-mon[32631]: pgmap v869: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-06-04T05:24:43.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:43 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/3249586959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:24:43.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:43 smithi112 ceph-mon[36453]: pgmap v869: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-06-04T05:24:43.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:43 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/3249586959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:24:43.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:43 smithi098 ceph-mon[35569]: pgmap v869: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-06-04T05:24:43.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:43 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/3249586959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-04T05:24:44.275 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-04T05:24:44.515 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:24:45.914 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:45 smithi070 ceph-mon[32631]: pgmap v870: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-06-04T05:24:45.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:45 smithi112 ceph-mon[36453]: pgmap v870: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-06-04T05:24:45.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:45 smithi098 ceph-mon[35569]: pgmap v870: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-06-04T05:24:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.167+0000 7f312f1bb700 1 -- 172.21.15.70:0/3762490320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3128102e30 msgr2=0x7f312810f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.167+0000 7f312f1bb700 1 --2- 172.21.15.70:0/3762490320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3128102e30 0x7f312810f2f0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f312000b0a0 tx=0x7f312000b3b0 comp rx=0 tx=0).stop 2024-06-04T05:24:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.167+0000 7f312f1bb700 1 -- 172.21.15.70:0/3762490320 shutdown_connections 2024-06-04T05:24:46.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.167+0000 7f312f1bb700 1 --2- 172.21.15.70:0/3762490320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3128102e30 0x7f312810f2f0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:46.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.167+0000 7f312f1bb700 1 --2- 172.21.15.70:0/3762490320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31281084b0 0x7f3128108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:46.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.167+0000 7f312f1bb700 1 --2- 172.21.15.70:0/3762490320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3128102470 0x7f31281028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:46.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.167+0000 7f312f1bb700 1 -- 172.21.15.70:0/3762490320 >> 172.21.15.70:0/3762490320 conn(0x7f3128079410 msgr2=0x7f3128079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:46.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.167+0000 7f312f1bb700 1 -- 172.21.15.70:0/3762490320 shutdown_connections 2024-06-04T05:24:46.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.167+0000 7f312f1bb700 1 -- 172.21.15.70:0/3762490320 wait complete. 2024-06-04T05:24:46.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.168+0000 7f312f1bb700 1 Processor -- start 2024-06-04T05:24:46.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.168+0000 7f312f1bb700 1 -- start start 2024-06-04T05:24:46.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.168+0000 7f312f1bb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3128102470 0x7f312819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:46.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.168+0000 7f312f1bb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3128102e30 0x7f312819fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:46.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.168+0000 7f312f1bb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31281084b0 0x7f3128199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:46.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.168+0000 7f312f1bb700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f31281124f0 con 0x7f31281084b0 2024-06-04T05:24:46.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.168+0000 7f312f1bb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3128112370 con 0x7f3128102e30 2024-06-04T05:24:46.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.168+0000 7f312f1bb700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3128112670 con 0x7f3128102470 2024-06-04T05:24:46.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.168+0000 7f312d758700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31281084b0 0x7f3128199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:46.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.168+0000 7f312d758700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31281084b0 0x7f3128199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35258/0 (socket says 172.21.15.70:35258) 2024-06-04T05:24:46.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.169+0000 7f312d758700 1 -- 172.21.15.70:0/2073742793 learned_addr learned my addr 172.21.15.70:0/2073742793 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:24:46.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.169+0000 7f312cf57700 1 --2- 172.21.15.70:0/2073742793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3128102470 0x7f312819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:46.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.169+0000 7f3127fff700 1 --2- 172.21.15.70:0/2073742793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3128102e30 0x7f312819fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:46.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.169+0000 7f312d758700 1 -- 172.21.15.70:0/2073742793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3128102470 msgr2=0x7f312819f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:46.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.169+0000 7f312d758700 1 --2- 172.21.15.70:0/2073742793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3128102470 0x7f312819f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:46.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.169+0000 7f312d758700 1 -- 172.21.15.70:0/2073742793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3128102e30 msgr2=0x7f312819fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:46.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.169+0000 7f312d758700 1 --2- 172.21.15.70:0/2073742793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3128102e30 0x7f312819fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:46.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.169+0000 7f312d758700 1 -- 172.21.15.70:0/2073742793 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f312819a0d0 con 0x7f31281084b0 2024-06-04T05:24:46.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.169+0000 7f312d758700 1 --2- 172.21.15.70:0/2073742793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31281084b0 0x7f3128199970 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f312000b830 tx=0x7f3120004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:46.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.169+0000 7f3125ffb700 1 -- 172.21.15.70:0/2073742793 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3120021070 con 0x7f31281084b0 2024-06-04T05:24:46.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.169+0000 7f312f1bb700 1 -- 172.21.15.70:0/2073742793 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f312819a360 con 0x7f31281084b0 2024-06-04T05:24:46.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.169+0000 7f3125ffb700 1 -- 172.21.15.70:0/2073742793 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3120007b80 con 0x7f31281084b0 2024-06-04T05:24:46.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.169+0000 7f3125ffb700 1 -- 172.21.15.70:0/2073742793 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31200052a0 con 0x7f31281084b0 2024-06-04T05:24:46.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.169+0000 7f312f1bb700 1 -- 172.21.15.70:0/2073742793 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31281a6680 con 0x7f31281084b0 2024-06-04T05:24:46.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.170+0000 7f312f1bb700 1 -- 172.21.15.70:0/2073742793 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3128103b60 con 0x7f31281084b0 2024-06-04T05:24:46.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.172+0000 7f3125ffb700 1 -- 172.21.15.70:0/2073742793 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f3120024070 con 0x7f31281084b0 2024-06-04T05:24:46.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.172+0000 7f3125ffb700 1 --2- 172.21.15.70:0/2073742793 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f31180b29f0 0x7f31180b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:46.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.172+0000 7f312cf57700 1 --2- 172.21.15.70:0/2073742793 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f31180b29f0 0x7f31180b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:46.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.172+0000 7f3125ffb700 1 -- 172.21.15.70:0/2073742793 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8571+0+0 (secure 0 0 0) 0x7f312000f020 con 0x7f31281084b0 2024-06-04T05:24:46.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.173+0000 7f312cf57700 1 --2- 172.21.15.70:0/2073742793 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f31180b29f0 0x7f31180b4eb0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f312819aab0 tx=0x7f311c00b410 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:46.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.177+0000 7f3125ffb700 1 -- 172.21.15.70:0/2073742793 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f31200e5e20 con 0x7f31281084b0 2024-06-04T05:24:46.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.390+0000 7f312f1bb700 1 -- 172.21.15.70:0/2073742793 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f31281a6850 con 0x7f31281084b0 2024-06-04T05:24:46.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.392+0000 7f3125ffb700 1 -- 172.21.15.70:0/2073742793 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) ==== 74+0+23179 (secure 0 0 0) 0x7f312002b020 con 0x7f31281084b0 2024-06-04T05:24:46.394 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-06-04T05:24:46.395 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":122,"fsid":"164dbaba-222d-11ef-bc9b-c7b262605968","created":"2024-06-04T04:45:22.297536+0000","modified":"2024-06-04T05:24:13.080278+0000","last_up_change":"2024-06-04T04:52:34.388708+0000","last_in_change":"2024-06-04T04:52:14.197780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-04T04:48:43.724356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-04T04:53:31.398455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-04T04:53:35.298617+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":48,"snap_epoch":121,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ff491ea2-1906-48e4-883e-bc78e778260e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6803","nonce":824119100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6805","nonce":824119100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6809","nonce":824119100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":824119100},{"type":"v1","addr":"172.21.15.70:6807","nonce":824119100}]},"public_addr":"172.21.15.70:6803/824119100","cluster_addr":"172.21.15.70:6805/824119100","heartbeat_back_addr":"172.21.15.70:6809/824119100","heartbeat_front_addr":"172.21.15.70:6807/824119100","state":["exists","up"]},{"osd":1,"uuid":"bd5e4806-535b-49b5-ba6b-93228da35a84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6801","nonce":2744598742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6803","nonce":2744598742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6807","nonce":2744598742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2744598742},{"type":"v1","addr":"172.21.15.98:6805","nonce":2744598742}]},"public_addr":"172.21.15.98:6801/2744598742","cluster_addr":"172.21.15.98:6803/2744598742","heartbeat_back_addr":"172.21.15.98:6807/2744598742","heartbeat_front_addr":"172.21.15.98:6805/2744598742","state":["exists","up"]},{"osd":2,"uuid":"8456fb55-910a-4a59-b84b-2c0f68346ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6801","nonce":1328098370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6803","nonce":1328098370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6807","nonce":1328098370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1328098370},{"type":"v1","addr":"172.21.15.112:6805","nonce":1328098370}]},"public_addr":"172.21.15.112:6801/1328098370","cluster_addr":"172.21.15.112:6803/1328098370","heartbeat_back_addr":"172.21.15.112:6807/1328098370","heartbeat_front_addr":"172.21.15.112:6805/1328098370","state":["exists","up"]},{"osd":3,"uuid":"c8fbc440-bf70-42d3-b01e-777ccfa7900b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6811","nonce":4041566346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6813","nonce":4041566346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6817","nonce":4041566346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":4041566346},{"type":"v1","addr":"172.21.15.70:6815","nonce":4041566346}]},"public_addr":"172.21.15.70:6811/4041566346","cluster_addr":"172.21.15.70:6813/4041566346","heartbeat_back_addr":"172.21.15.70:6817/4041566346","heartbeat_front_addr":"172.21.15.70:6815/4041566346","state":["exists","up"]},{"osd":4,"uuid":"549c8c70-8be8-46ae-bcb6-0fde32b40cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6809","nonce":3652775814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6811","nonce":3652775814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6815","nonce":3652775814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3652775814},{"type":"v1","addr":"172.21.15.98:6813","nonce":3652775814}]},"public_addr":"172.21.15.98:6809/3652775814","cluster_addr":"172.21.15.98:6811/3652775814","heartbeat_back_addr":"172.21.15.98:6815/3652775814","heartbeat_front_addr":"172.21.15.98:6813/3652775814","state":["exists","up"]},{"osd":5,"uuid":"51d86af1-0d6c-4ae0-b090-00414fc1bf7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6809","nonce":1857257899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6811","nonce":1857257899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6815","nonce":1857257899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":1857257899},{"type":"v1","addr":"172.21.15.112:6813","nonce":1857257899}]},"public_addr":"172.21.15.112:6809/1857257899","cluster_addr":"172.21.15.112:6811/1857257899","heartbeat_back_addr":"172.21.15.112:6815/1857257899","heartbeat_front_addr":"172.21.15.112:6813/1857257899","state":["exists","up"]},{"osd":6,"uuid":"526a8138-bfa8-419e-a5b5-e6b80f0ae004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6819","nonce":1264676496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6821","nonce":1264676496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6825","nonce":1264676496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":1264676496},{"type":"v1","addr":"172.21.15.70:6823","nonce":1264676496}]},"public_addr":"172.21.15.70:6819/1264676496","cluster_addr":"172.21.15.70:6821/1264676496","heartbeat_back_addr":"172.21.15.70:6825/1264676496","heartbeat_front_addr":"172.21.15.70:6823/1264676496","state":["exists","up"]},{"osd":7,"uuid":"1173f785-6ed1-47ef-9265-2e233ea9ee02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6817","nonce":709593020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6819","nonce":709593020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6823","nonce":709593020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":709593020},{"type":"v1","addr":"172.21.15.98:6821","nonce":709593020}]},"public_addr":"172.21.15.98:6817/709593020","cluster_addr":"172.21.15.98:6819/709593020","heartbeat_back_addr":"172.21.15.98:6823/709593020","heartbeat_front_addr":"172.21.15.98:6821/709593020","state":["exists","up"]},{"osd":8,"uuid":"6424cc6e-24a7-4b81-9ce6-39203fecc7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6817","nonce":2250225413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6819","nonce":2250225413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6823","nonce":2250225413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2250225413},{"type":"v1","addr":"172.21.15.112:6821","nonce":2250225413}]},"public_addr":"172.21.15.112:6817/2250225413","cluster_addr":"172.21.15.112:6819/2250225413","heartbeat_back_addr":"172.21.15.112:6823/2250225413","heartbeat_front_addr":"172.21.15.112:6821/2250225413","state":["exists","up"]},{"osd":9,"uuid":"8d666db8-8831-4d63-9e3c-a4bf2504149e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6827","nonce":3557688707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6829","nonce":3557688707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6833","nonce":3557688707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3557688707},{"type":"v1","addr":"172.21.15.70:6831","nonce":3557688707}]},"public_addr":"172.21.15.70:6827/3557688707","cluster_addr":"172.21.15.70:6829/3557688707","heartbeat_back_addr":"172.21.15.70:6833/3557688707","heartbeat_front_addr":"172.21.15.70:6831/3557688707","state":["exists","up"]},{"osd":10,"uuid":"ecb925d2-0263-4dfc-a2fd-9e27856022d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6825","nonce":1896493718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6827","nonce":1896493718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6831","nonce":1896493718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1896493718},{"type":"v1","addr":"172.21.15.98:6829","nonce":1896493718}]},"public_addr":"172.21.15.98:6825/1896493718","cluster_addr":"172.21.15.98:6827/1896493718","heartbeat_back_addr":"172.21.15.98:6831/1896493718","heartbeat_front_addr":"172.21.15.98:6829/1896493718","state":["exists","up"]},{"osd":11,"uuid":"c8e3230c-bc3a-40bd-8dd0-5fef56e4df97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6825","nonce":1400793137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6827","nonce":1400793137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6831","nonce":1400793137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1400793137},{"type":"v1","addr":"172.21.15.112:6829","nonce":1400793137}]},"public_addr":"172.21.15.112:6825/1400793137","cluster_addr":"172.21.15.112:6827/1400793137","heartbeat_back_addr":"172.21.15.112:6831/1400793137","heartbeat_front_addr":"172.21.15.112:6829/1400793137","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:47:53.626608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:17.784505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:48:41.971969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:07.130699+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:31.615496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:49:56.898607+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:22.739991+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:50:48.319727+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:13.902063+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:51:40.389470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:06.142092+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-04T04:52:32.949383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/4291129442":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/3584175673":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/3720789940":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/145533746":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/3442074771":"2024-06-05T04:46:17.636907+0000","172.21.15.70:6801/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/1442095493":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/2453733268":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6801/2752966840":"2024-06-05T04:57:09.937704+0000","172.21.15.70:0/1369781447":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6801/2087909610":"2024-06-05T04:45:49.501864+0000","172.21.15.70:6800/2752966840":"2024-06-05T04:57:09.937704+0000","172.21.15.70:6800/1353782413":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/915186456":"2024-06-05T04:46:17.636907+0000","172.21.15.70:0/860033006":"2024-06-05T04:45:49.501864+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":35,"length":1},{"begin":48,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-04T05:24:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.394+0000 7f312f1bb700 1 -- 172.21.15.70:0/2073742793 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f31180b29f0 msgr2=0x7f31180b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.394+0000 7f312f1bb700 1 --2- 172.21.15.70:0/2073742793 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f31180b29f0 0x7f31180b4eb0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f312819aab0 tx=0x7f311c00b410 comp rx=0 tx=0).stop 2024-06-04T05:24:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.394+0000 7f312f1bb700 1 -- 172.21.15.70:0/2073742793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31281084b0 msgr2=0x7f3128199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.394+0000 7f312f1bb700 1 --2- 172.21.15.70:0/2073742793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31281084b0 0x7f3128199970 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f312000b830 tx=0x7f3120004d00 comp rx=0 tx=0).stop 2024-06-04T05:24:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.395+0000 7f312f1bb700 1 -- 172.21.15.70:0/2073742793 shutdown_connections 2024-06-04T05:24:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.395+0000 7f312f1bb700 1 --2- 172.21.15.70:0/2073742793 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f31180b29f0 0x7f31180b4eb0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.395+0000 7f312f1bb700 1 --2- 172.21.15.70:0/2073742793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31281084b0 0x7f3128199970 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.395+0000 7f312f1bb700 1 --2- 172.21.15.70:0/2073742793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3128102470 0x7f312819f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.395+0000 7f312f1bb700 1 --2- 172.21.15.70:0/2073742793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3128102e30 0x7f312819fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.395+0000 7f312f1bb700 1 -- 172.21.15.70:0/2073742793 >> 172.21.15.70:0/2073742793 conn(0x7f3128079410 msgr2=0x7f3128106320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.395+0000 7f312f1bb700 1 -- 172.21.15.70:0/2073742793 shutdown_connections 2024-06-04T05:24:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:46.395+0000 7f312f1bb700 1 -- 172.21.15.70:0/2073742793 wait complete. 2024-06-04T05:24:46.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:46 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2073742793' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T05:24:46.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:46 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2073742793' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T05:24:47.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:46 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2073742793' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-04T05:24:47.073 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs fail cephfs 2024-06-04T05:24:47.312 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:24:47.685 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:47 smithi070 ceph-mon[32631]: pgmap v871: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-06-04T05:24:47.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:47 smithi112 ceph-mon[36453]: pgmap v871: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-06-04T05:24:47.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:47 smithi098 ceph-mon[35569]: pgmap v871: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-06-04T05:24:48.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.986+0000 7f75432d0700 1 -- 172.21.15.70:0/2641546749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f753c1095b0 msgr2=0x7f753c109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:48.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.986+0000 7f75432d0700 1 --2- 172.21.15.70:0/2641546749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f753c1095b0 0x7f753c109990 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f7538009990 tx=0x7f7538009ca0 comp rx=0 tx=0).stop 2024-06-04T05:24:48.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.986+0000 7f75432d0700 1 -- 172.21.15.70:0/2641546749 shutdown_connections 2024-06-04T05:24:48.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.986+0000 7f75432d0700 1 --2- 172.21.15.70:0/2641546749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f753c06bbe0 0x7f753c1103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:48.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.986+0000 7f75432d0700 1 --2- 172.21.15.70:0/2641546749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f753c1095b0 0x7f753c109990 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:48.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.986+0000 7f75432d0700 1 --2- 172.21.15.70:0/2641546749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f753c06b220 0x7f753c06b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:48.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.986+0000 7f75432d0700 1 -- 172.21.15.70:0/2641546749 >> 172.21.15.70:0/2641546749 conn(0x7f753c0793f0 msgr2=0x7f753c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:48.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.986+0000 7f75432d0700 1 -- 172.21.15.70:0/2641546749 shutdown_connections 2024-06-04T05:24:48.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.986+0000 7f75432d0700 1 -- 172.21.15.70:0/2641546749 wait complete. 2024-06-04T05:24:48.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.986+0000 7f75432d0700 1 Processor -- start 2024-06-04T05:24:48.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.986+0000 7f75432d0700 1 -- start start 2024-06-04T05:24:48.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f75432d0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f753c06b220 0x7f753c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:48.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f75432d0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f753c06bbe0 0x7f753c19fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:48.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f75432d0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f753c1095b0 0x7f753c199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:48.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f75432d0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f753c113590 con 0x7f753c06b220 2024-06-04T05:24:48.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f75432d0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f753c113410 con 0x7f753c1095b0 2024-06-04T05:24:48.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f75432d0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f753c113710 con 0x7f753c06bbe0 2024-06-04T05:24:48.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f754106c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f753c06b220 0x7f753c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:48.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f754106c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f753c06b220 0x7f753c19f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35284/0 (socket says 172.21.15.70:35284) 2024-06-04T05:24:48.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f754106c700 1 -- 172.21.15.70:0/2642384393 learned_addr learned my addr 172.21.15.70:0/2642384393 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:24:48.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f754186d700 1 --2- 172.21.15.70:0/2642384393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f753c1095b0 0x7f753c199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:48.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f754086b700 1 --2- 172.21.15.70:0/2642384393 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f753c06bbe0 0x7f753c19fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:48.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f754106c700 1 -- 172.21.15.70:0/2642384393 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f753c06bbe0 msgr2=0x7f753c19fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:48.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f754106c700 1 --2- 172.21.15.70:0/2642384393 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f753c06bbe0 0x7f753c19fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:48.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f754106c700 1 -- 172.21.15.70:0/2642384393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f753c1095b0 msgr2=0x7f753c199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:48.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f754106c700 1 --2- 172.21.15.70:0/2642384393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f753c1095b0 0x7f753c199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:48.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f754106c700 1 -- 172.21.15.70:0/2642384393 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753c19a0d0 con 0x7f753c06b220 2024-06-04T05:24:48.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f754186d700 1 --2- 172.21.15.70:0/2642384393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f753c1095b0 0x7f753c199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:24:48.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f754106c700 1 --2- 172.21.15.70:0/2642384393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f753c06b220 0x7f753c19f7b0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f7538000c00 tx=0x7f7538009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:48.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f75327fc700 1 -- 172.21.15.70:0/2642384393 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7538027070 con 0x7f753c06b220 2024-06-04T05:24:48.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f75327fc700 1 -- 172.21.15.70:0/2642384393 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f753800fc70 con 0x7f753c06b220 2024-06-04T05:24:48.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f75432d0700 1 -- 172.21.15.70:0/2642384393 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f753c19a360 con 0x7f753c06b220 2024-06-04T05:24:48.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.987+0000 7f75327fc700 1 -- 172.21.15.70:0/2642384393 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7538022d40 con 0x7f753c06b220 2024-06-04T05:24:48.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.988+0000 7f75432d0700 1 -- 172.21.15.70:0/2642384393 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f753c1a6680 con 0x7f753c06b220 2024-06-04T05:24:48.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.988+0000 7f75432d0700 1 -- 172.21.15.70:0/2642384393 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f753c10ba90 con 0x7f753c06b220 2024-06-04T05:24:48.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.990+0000 7f75327fc700 1 -- 172.21.15.70:0/2642384393 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f753802a070 con 0x7f753c06b220 2024-06-04T05:24:48.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.991+0000 7f75327fc700 1 --2- 172.21.15.70:0/2642384393 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f75280b29f0 0x7f75280b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:48.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.991+0000 7f754086b700 1 --2- 172.21.15.70:0/2642384393 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f75280b29f0 0x7f75280b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:48.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.991+0000 7f75327fc700 1 -- 172.21.15.70:0/2642384393 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8571+0+0 (secure 0 0 0) 0x7f75380e3860 con 0x7f753c06b220 2024-06-04T05:24:48.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.992+0000 7f754086b700 1 --2- 172.21.15.70:0/2642384393 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f75280b29f0 0x7f75280b4eb0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f752c005950 tx=0x7f752c0058e0 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:48.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:48.995+0000 7f75327fc700 1 -- 172.21.15.70:0/2642384393 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7538031070 con 0x7f753c06b220 2024-06-04T05:24:49.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:49.244+0000 7f75432d0700 1 -- 172.21.15.70:0/2642384393 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f753c19aea0 con 0x7f753c06b220 2024-06-04T05:24:49.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:49.246+0000 7f75327fc700 1 -- 172.21.15.70:0/2642384393 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v73) ==== 161+0+0 (secure 0 0 0) 0x7f7538013070 con 0x7f753c06b220 2024-06-04T05:24:49.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:49.250+0000 7f75432d0700 1 -- 172.21.15.70:0/2642384393 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f75280b29f0 msgr2=0x7f75280b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:49.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:49.250+0000 7f75432d0700 1 --2- 172.21.15.70:0/2642384393 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f75280b29f0 0x7f75280b4eb0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f752c005950 tx=0x7f752c0058e0 comp rx=0 tx=0).stop 2024-06-04T05:24:49.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:49.250+0000 7f75432d0700 1 -- 172.21.15.70:0/2642384393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f753c06b220 msgr2=0x7f753c19f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:49.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:49.250+0000 7f75432d0700 1 --2- 172.21.15.70:0/2642384393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f753c06b220 0x7f753c19f7b0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f7538000c00 tx=0x7f7538009f90 comp rx=0 tx=0).stop 2024-06-04T05:24:49.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:49.250+0000 7f75432d0700 1 -- 172.21.15.70:0/2642384393 shutdown_connections 2024-06-04T05:24:49.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:49.250+0000 7f75432d0700 1 --2- 172.21.15.70:0/2642384393 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f75280b29f0 0x7f75280b4eb0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:49.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:49.250+0000 7f75432d0700 1 --2- 172.21.15.70:0/2642384393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f753c1095b0 0x7f753c199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:49.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:49.250+0000 7f75432d0700 1 --2- 172.21.15.70:0/2642384393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f753c06b220 0x7f753c19f7b0 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:49.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:49.250+0000 7f75432d0700 1 --2- 172.21.15.70:0/2642384393 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f753c06bbe0 0x7f753c19fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:49.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:49.250+0000 7f75432d0700 1 -- 172.21.15.70:0/2642384393 >> 172.21.15.70:0/2642384393 conn(0x7f753c0793f0 msgr2=0x7f753c1012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:49.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:49.250+0000 7f75432d0700 1 -- 172.21.15.70:0/2642384393 shutdown_connections 2024-06-04T05:24:49.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:49.250+0000 7f75432d0700 1 -- 172.21.15.70:0/2642384393 wait complete. 2024-06-04T05:24:49.253 INFO:teuthology.orchestra.run.smithi070.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-04T05:24:49.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:49 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-a[32627]: 2024-06-04T05:24:49.244+0000 7fdaa419c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-04T05:24:49.609 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 05:24:49 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-g[73644]: ignoring --setuser ceph since I am not root 2024-06-04T05:24:49.609 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 05:24:49 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-g[73644]: ignoring --setgroup ceph since I am not root 2024-06-04T05:24:49.609 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 05:24:49 smithi070 ceph-mds[73666]: -- 172.21.15.70:0/1843952271 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ae1c5f7a40 con 0x55ae1c5cb000 2024-06-04T05:24:49.609 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 05:24:49 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-g[73644]: starting mds.g at 2024-06-04T05:24:49.609 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 05:24:49 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-a[74103]: ignoring --setuser ceph since I am not root 2024-06-04T05:24:49.610 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 05:24:49 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-a[74103]: ignoring --setgroup ceph since I am not root 2024-06-04T05:24:49.610 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 05:24:49 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-a[74103]: starting mds.a at 2024-06-04T05:24:49.652 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-f[67196]: ignoring --setuser ceph since I am not root 2024-06-04T05:24:49.652 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-f[67196]: ignoring --setgroup ceph since I am not root 2024-06-04T05:24:49.652 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-mds[67201]: -- 172.21.15.112:0/2310291300 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5639353f5a40 con 0x5639353c8800 2024-06-04T05:24:49.652 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-f[67196]: starting mds.f at 2024-06-04T05:24:49.652 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: ignoring --setuser ceph since I am not root 2024-06-04T05:24:49.652 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: ignoring --setgroup ceph since I am not root 2024-06-04T05:24:49.652 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-mds[67808]: -- 172.21.15.112:0/1266685747 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55940e115a40 con 0x55940e0e8c00 2024-06-04T05:24:49.653 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: starting mds.i at 2024-06-04T05:24:49.653 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-l[66885]: ignoring --setuser ceph since I am not root 2024-06-04T05:24:49.653 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-l[66885]: ignoring --setgroup ceph since I am not root 2024-06-04T05:24:49.653 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-mds[66889]: -- 172.21.15.112:0/1004782106 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ee829bda40 con 0x55ee82990c00 2024-06-04T05:24:49.653 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-l[66885]: starting mds.l at 2024-06-04T05:24:49.653 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-c[67503]: ignoring --setuser ceph since I am not root 2024-06-04T05:24:49.653 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-c[67503]: ignoring --setgroup ceph since I am not root 2024-06-04T05:24:49.653 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-mds[67507]: -- 172.21.15.112:0/3767098880 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558cb47bfa40 con 0x558cb4792c00 2024-06-04T05:24:49.653 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-c[67503]: starting mds.c at 2024-06-04T05:24:49.654 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-e[64860]: ignoring --setuser ceph since I am not root 2024-06-04T05:24:49.654 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-e[64860]: ignoring --setgroup ceph since I am not root 2024-06-04T05:24:49.654 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-mds[64864]: -- 172.21.15.98:0/1854794054 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562fe5167a40 con 0x562fe513b000 2024-06-04T05:24:49.654 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-e[64860]: starting mds.e at 2024-06-04T05:24:49.654 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-b[65481]: ignoring --setuser ceph since I am not root 2024-06-04T05:24:49.655 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-b[65481]: ignoring --setgroup ceph since I am not root 2024-06-04T05:24:49.655 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-mds[65485]: -- 172.21.15.98:0/1427626970 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a288ff44e0 con 0x55a288fcac00 2024-06-04T05:24:49.655 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-mds[65485]: -- 172.21.15.98:0/1427626970 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a288ff7a40 con 0x55a288fcac00 2024-06-04T05:24:49.655 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-b[65481]: starting mds.b at 2024-06-04T05:24:49.655 INFO:journalctl@ceph.mds.k.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-k[64561]: ignoring --setuser ceph since I am not root 2024-06-04T05:24:49.655 INFO:journalctl@ceph.mds.k.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-k[64561]: ignoring --setgroup ceph since I am not root 2024-06-04T05:24:49.655 INFO:journalctl@ceph.mds.k.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-mds[64565]: -- 172.21.15.98:0/1336247060 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563785ee5a40 con 0x563785eb8c00 2024-06-04T05:24:49.655 INFO:journalctl@ceph.mds.k.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-k[64561]: starting mds.k at 2024-06-04T05:24:49.655 INFO:journalctl@ceph.mds.h.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-h[65161]: ignoring --setuser ceph since I am not root 2024-06-04T05:24:49.656 INFO:journalctl@ceph.mds.h.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-h[65161]: ignoring --setgroup ceph since I am not root 2024-06-04T05:24:49.656 INFO:journalctl@ceph.mds.h.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-mds[65165]: -- 172.21.15.98:0/3707386762 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557cd9881a40 con 0x557cd9855000 2024-06-04T05:24:49.656 INFO:journalctl@ceph.mds.h.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-h[65161]: starting mds.h at 2024-06-04T05:24:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-mon[36453]: pgmap v872: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-04T05:24:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2642384393' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-06-04T05:24:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-mon[36453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-06-04T05:24:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-mon[36453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-04T05:24:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2642384393' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-06-04T05:24:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-mon[36453]: osdmap e123: 12 total, 12 up, 12 in 2024-06-04T05:24:49.936 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:49 smithi112 ceph-mon[36453]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-06-04T05:24:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-mon[35569]: pgmap v872: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-04T05:24:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2642384393' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-06-04T05:24:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-mon[35569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-06-04T05:24:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-mon[35569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-04T05:24:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2642384393' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-06-04T05:24:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-mon[35569]: osdmap e123: 12 total, 12 up, 12 in 2024-06-04T05:24:49.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:49 smithi098 ceph-mon[35569]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-06-04T05:24:50.031 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:49 smithi070 ceph-mon[32631]: pgmap v872: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-04T05:24:50.032 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:49 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2642384393' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-06-04T05:24:50.032 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:49 smithi070 ceph-mon[32631]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-06-04T05:24:50.032 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:49 smithi070 ceph-mon[32631]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-04T05:24:50.032 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:49 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2642384393' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-06-04T05:24:50.032 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:49 smithi070 ceph-mon[32631]: osdmap e123: 12 total, 12 up, 12 in 2024-06-04T05:24:50.032 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:49 smithi070 ceph-mon[32631]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-06-04T05:24:50.061 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-06-04T05:24:50.307 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:24:51.465 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.98:6832/1097978097,v1:172.21.15.98:6835/1097978097] up:boot 2024-06-04T05:24:51.465 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.98:6833/1627295238,v1:172.21.15.98:6834/1627295238] up:boot 2024-06-04T05:24:51.465 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.98:6836/2378417829,v1:172.21.15.98:6838/2378417829] up:boot 2024-06-04T05:24:51.465 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.98:6837/1605056506,v1:172.21.15.98:6839/1605056506] up:boot 2024-06-04T05:24:51.465 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.112:6832/3021285279,v1:172.21.15.112:6835/3021285279] up:boot 2024-06-04T05:24:51.465 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.112:6834/3634732236,v1:172.21.15.112:6837/3634732236] up:boot 2024-06-04T05:24:51.465 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.112:6833/824377770,v1:172.21.15.112:6838/824377770] up:boot 2024-06-04T05:24:51.465 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.112:6836/3278907976,v1:172.21.15.112:6839/3278907976] up:boot 2024-06-04T05:24:51.465 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.70:6839/3576139470,v1:172.21.15.70:6841/3576139470] up:boot 2024-06-04T05:24:51.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: mds.? [v2:172.21.15.70:6838/2738108501,v1:172.21.15.70:6840/2738108501] up:boot 2024-06-04T05:24:51.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-06-04T05:24:51.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-04T05:24:51.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-04T05:24:51.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-04T05:24:51.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-04T05:24:51.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-04T05:24:51.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-04T05:24:51.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-04T05:24:51.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-04T05:24:51.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-04T05:24:51.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-04T05:24:51.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:51 smithi070 ceph-mon[32631]: pgmap v874: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-04T05:24:51.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.98:6832/1097978097,v1:172.21.15.98:6835/1097978097] up:boot 2024-06-04T05:24:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.98:6833/1627295238,v1:172.21.15.98:6834/1627295238] up:boot 2024-06-04T05:24:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.98:6836/2378417829,v1:172.21.15.98:6838/2378417829] up:boot 2024-06-04T05:24:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.98:6837/1605056506,v1:172.21.15.98:6839/1605056506] up:boot 2024-06-04T05:24:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.112:6832/3021285279,v1:172.21.15.112:6835/3021285279] up:boot 2024-06-04T05:24:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.112:6834/3634732236,v1:172.21.15.112:6837/3634732236] up:boot 2024-06-04T05:24:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.112:6833/824377770,v1:172.21.15.112:6838/824377770] up:boot 2024-06-04T05:24:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.112:6836/3278907976,v1:172.21.15.112:6839/3278907976] up:boot 2024-06-04T05:24:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.70:6839/3576139470,v1:172.21.15.70:6841/3576139470] up:boot 2024-06-04T05:24:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: mds.? [v2:172.21.15.70:6838/2738108501,v1:172.21.15.70:6840/2738108501] up:boot 2024-06-04T05:24:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-06-04T05:24:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-04T05:24:51.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-04T05:24:51.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-04T05:24:51.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-04T05:24:51.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-04T05:24:51.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-04T05:24:51.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-04T05:24:51.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-04T05:24:51.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-04T05:24:51.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-04T05:24:51.688 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:51 smithi112 ceph-mon[36453]: pgmap v874: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-04T05:24:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.98:6832/1097978097,v1:172.21.15.98:6835/1097978097] up:boot 2024-06-04T05:24:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.98:6833/1627295238,v1:172.21.15.98:6834/1627295238] up:boot 2024-06-04T05:24:51.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.98:6836/2378417829,v1:172.21.15.98:6838/2378417829] up:boot 2024-06-04T05:24:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.98:6837/1605056506,v1:172.21.15.98:6839/1605056506] up:boot 2024-06-04T05:24:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.112:6832/3021285279,v1:172.21.15.112:6835/3021285279] up:boot 2024-06-04T05:24:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.112:6834/3634732236,v1:172.21.15.112:6837/3634732236] up:boot 2024-06-04T05:24:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.112:6833/824377770,v1:172.21.15.112:6838/824377770] up:boot 2024-06-04T05:24:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.112:6836/3278907976,v1:172.21.15.112:6839/3278907976] up:boot 2024-06-04T05:24:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.70:6839/3576139470,v1:172.21.15.70:6841/3576139470] up:boot 2024-06-04T05:24:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: mds.? [v2:172.21.15.70:6838/2738108501,v1:172.21.15.70:6840/2738108501] up:boot 2024-06-04T05:24:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-06-04T05:24:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-04T05:24:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-04T05:24:51.746 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-04T05:24:51.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-04T05:24:51.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-04T05:24:51.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-04T05:24:51.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-04T05:24:51.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-04T05:24:51.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-04T05:24:51.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: from='mgr.24806 172.21.15.70:0/1569702779' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-04T05:24:51.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:51 smithi098 ceph-mon[35569]: pgmap v874: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-04T05:24:52.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.037+0000 7f078dd32700 1 -- 172.21.15.70:0/2981266340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078806bc90 msgr2=0x7f078806c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:52.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.037+0000 7f078dd32700 1 --2- 172.21.15.70:0/2981266340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078806bc90 0x7f078806c110 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f0770009990 tx=0x7f0770009ca0 comp rx=0 tx=0).stop 2024-06-04T05:24:52.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.037+0000 7f078dd32700 1 -- 172.21.15.70:0/2981266340 shutdown_connections 2024-06-04T05:24:52.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.037+0000 7f078dd32700 1 --2- 172.21.15.70:0/2981266340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f078806c650 0x7f07881103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:52.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.037+0000 7f078dd32700 1 --2- 172.21.15.70:0/2981266340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07881095b0 0x7f0788109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:52.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.037+0000 7f078dd32700 1 --2- 172.21.15.70:0/2981266340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078806bc90 0x7f078806c110 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:52.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.037+0000 7f078dd32700 1 -- 172.21.15.70:0/2981266340 >> 172.21.15.70:0/2981266340 conn(0x7f0788079410 msgr2=0x7f0788079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:52.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.037+0000 7f078dd32700 1 -- 172.21.15.70:0/2981266340 shutdown_connections 2024-06-04T05:24:52.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.037+0000 7f078dd32700 1 -- 172.21.15.70:0/2981266340 wait complete. 2024-06-04T05:24:52.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.038+0000 7f078dd32700 1 Processor -- start 2024-06-04T05:24:52.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.038+0000 7f078dd32700 1 -- start start 2024-06-04T05:24:52.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.038+0000 7f078dd32700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078806bc90 0x7f0788102540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:52.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.038+0000 7f078dd32700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f078806c650 0x7f0788102a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:52.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.038+0000 7f078dd32700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07881095b0 0x7f0788102fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:52.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.038+0000 7f078dd32700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0788113570 con 0x7f078806bc90 2024-06-04T05:24:52.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.038+0000 7f078dd32700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f07881133f0 con 0x7f07881095b0 2024-06-04T05:24:52.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.038+0000 7f078dd32700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f07881136f0 con 0x7f078806c650 2024-06-04T05:24:52.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.038+0000 7f07877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078806bc90 0x7f0788102540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:52.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.038+0000 7f07877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078806bc90 0x7f0788102540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35336/0 (socket says 172.21.15.70:35336) 2024-06-04T05:24:52.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.038+0000 7f07877fe700 1 -- 172.21.15.70:0/2304784532 learned_addr learned my addr 172.21.15.70:0/2304784532 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:24:52.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.038+0000 7f077ffff700 1 --2- 172.21.15.70:0/2304784532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f078806c650 0x7f0788102a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:52.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.039+0000 7f0787fff700 1 --2- 172.21.15.70:0/2304784532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07881095b0 0x7f0788102fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:52.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.039+0000 7f07877fe700 1 -- 172.21.15.70:0/2304784532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f078806c650 msgr2=0x7f0788102a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:52.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.039+0000 7f07877fe700 1 --2- 172.21.15.70:0/2304784532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f078806c650 0x7f0788102a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:52.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.039+0000 7f07877fe700 1 -- 172.21.15.70:0/2304784532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07881095b0 msgr2=0x7f0788102fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:52.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.039+0000 7f07877fe700 1 --2- 172.21.15.70:0/2304784532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07881095b0 0x7f0788102fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:52.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.039+0000 7f07877fe700 1 -- 172.21.15.70:0/2304784532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07881a8010 con 0x7f078806bc90 2024-06-04T05:24:52.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.039+0000 7f07877fe700 1 --2- 172.21.15.70:0/2304784532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078806bc90 0x7f0788102540 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f077800d950 tx=0x7f077800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:52.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.039+0000 7f07857fa700 1 -- 172.21.15.70:0/2304784532 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778016070 con 0x7f078806bc90 2024-06-04T05:24:52.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.039+0000 7f07857fa700 1 -- 172.21.15.70:0/2304784532 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f077800fd20 con 0x7f078806bc90 2024-06-04T05:24:52.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.039+0000 7f07857fa700 1 -- 172.21.15.70:0/2304784532 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077800e3b0 con 0x7f078806bc90 2024-06-04T05:24:52.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.039+0000 7f078dd32700 1 -- 172.21.15.70:0/2304784532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07881a8300 con 0x7f078806bc90 2024-06-04T05:24:52.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.039+0000 7f078dd32700 1 -- 172.21.15.70:0/2304784532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07881a8860 con 0x7f078806bc90 2024-06-04T05:24:52.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.040+0000 7f078dd32700 1 -- 172.21.15.70:0/2304784532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f078806a230 con 0x7f078806bc90 2024-06-04T05:24:52.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.042+0000 7f07857fa700 1 -- 172.21.15.70:0/2304784532 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f0778019070 con 0x7f078806bc90 2024-06-04T05:24:52.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.042+0000 7f07857fa700 1 --2- 172.21.15.70:0/2304784532 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f07740b2a70 0x7f07740b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:52.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.042+0000 7f077ffff700 1 --2- 172.21.15.70:0/2304784532 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f07740b2a70 0x7f07740b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:52.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.042+0000 7f07857fa700 1 -- 172.21.15.70:0/2304784532 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8957+0+0 (secure 0 0 0) 0x7f07780e21e0 con 0x7f078806bc90 2024-06-04T05:24:52.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.043+0000 7f077ffff700 1 --2- 172.21.15.70:0/2304784532 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f07740b2a70 0x7f07740b4f30 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f077000b5c0 tx=0x7f0770005960 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:52.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.047+0000 7f07857fa700 1 -- 172.21.15.70:0/2304784532 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f07780a9a10 con 0x7f078806bc90 2024-06-04T05:24:52.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.296+0000 7f078dd32700 1 -- 172.21.15.70:0/2304784532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f078802e560 con 0x7f078806bc90 2024-06-04T05:24:52.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.348+0000 7f07857fa700 1 -- 172.21.15.70:0/2304784532 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v75) ==== 104+0+0 (secure 0 0 0) 0x7f07780a9310 con 0x7f078806bc90 2024-06-04T05:24:52.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.350+0000 7f078dd32700 1 -- 172.21.15.70:0/2304784532 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f07740b2a70 msgr2=0x7f07740b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:52.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.350+0000 7f078dd32700 1 --2- 172.21.15.70:0/2304784532 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f07740b2a70 0x7f07740b4f30 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f077000b5c0 tx=0x7f0770005960 comp rx=0 tx=0).stop 2024-06-04T05:24:52.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.350+0000 7f078dd32700 1 -- 172.21.15.70:0/2304784532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078806bc90 msgr2=0x7f0788102540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:52.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.350+0000 7f078dd32700 1 --2- 172.21.15.70:0/2304784532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078806bc90 0x7f0788102540 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f077800d950 tx=0x7f077800dc60 comp rx=0 tx=0).stop 2024-06-04T05:24:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.350+0000 7f078dd32700 1 -- 172.21.15.70:0/2304784532 shutdown_connections 2024-06-04T05:24:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.350+0000 7f078dd32700 1 --2- 172.21.15.70:0/2304784532 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f07740b2a70 0x7f07740b4f30 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.350+0000 7f078dd32700 1 --2- 172.21.15.70:0/2304784532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07881095b0 0x7f0788102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.350+0000 7f078dd32700 1 --2- 172.21.15.70:0/2304784532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078806bc90 0x7f0788102540 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.350+0000 7f078dd32700 1 --2- 172.21.15.70:0/2304784532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f078806c650 0x7f0788102a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.350+0000 7f078dd32700 1 -- 172.21.15.70:0/2304784532 >> 172.21.15.70:0/2304784532 conn(0x7f0788079410 msgr2=0x7f0788101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.350+0000 7f078dd32700 1 -- 172.21.15.70:0/2304784532 shutdown_connections 2024-06-04T05:24:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:52.350+0000 7f078dd32700 1 -- 172.21.15.70:0/2304784532 wait complete. 2024-06-04T05:24:52.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:52 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2304784532' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-06-04T05:24:52.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:52 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2304784532' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-06-04T05:24:52.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:52 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2304784532' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-06-04T05:24:53.009 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-04T05:24:53.248 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:24:53.643 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:53 smithi070 ceph-mon[32631]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-06-04T05:24:53.643 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:53 smithi070 ceph-mon[32631]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-06-04T05:24:53.644 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:53 smithi070 ceph-mon[32631]: Cluster is now healthy 2024-06-04T05:24:53.644 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:53 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/2304784532' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-06-04T05:24:53.644 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:53 smithi070 ceph-mon[32631]: fsmap 12 up:standby 2024-06-04T05:24:53.644 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:53 smithi070 ceph-mon[32631]: pgmap v875: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-06-04T05:24:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:53 smithi112 ceph-mon[36453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-06-04T05:24:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:53 smithi112 ceph-mon[36453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-06-04T05:24:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:53 smithi112 ceph-mon[36453]: Cluster is now healthy 2024-06-04T05:24:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:53 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/2304784532' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-06-04T05:24:53.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:53 smithi112 ceph-mon[36453]: fsmap 12 up:standby 2024-06-04T05:24:53.687 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:53 smithi112 ceph-mon[36453]: pgmap v875: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-06-04T05:24:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:53 smithi098 ceph-mon[35569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-06-04T05:24:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:53 smithi098 ceph-mon[35569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-06-04T05:24:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:53 smithi098 ceph-mon[35569]: Cluster is now healthy 2024-06-04T05:24:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:53 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/2304784532' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-06-04T05:24:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:53 smithi098 ceph-mon[35569]: fsmap 12 up:standby 2024-06-04T05:24:53.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:53 smithi098 ceph-mon[35569]: pgmap v875: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-06-04T05:24:54.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.849+0000 7feed049f700 1 -- 172.21.15.70:0/2919253630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec81088f0 msgr2=0x7feec8110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:54.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.849+0000 7feed049f700 1 --2- 172.21.15.70:0/2919253630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec81088f0 0x7feec8110470 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7feec000b0d0 tx=0x7feec000b3e0 comp rx=0 tx=0).stop 2024-06-04T05:24:54.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.849+0000 7feed049f700 1 -- 172.21.15.70:0/2919253630 shutdown_connections 2024-06-04T05:24:54.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.849+0000 7feed049f700 1 --2- 172.21.15.70:0/2919253630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec81088f0 0x7feec8110470 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:54.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.849+0000 7feed049f700 1 --2- 172.21.15.70:0/2919253630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feec81037f0 0x7feec8103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:54.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.849+0000 7feed049f700 1 --2- 172.21.15.70:0/2919253630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feec81041a0 0x7feec81081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:54.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.849+0000 7feed049f700 1 -- 172.21.15.70:0/2919253630 >> 172.21.15.70:0/2919253630 conn(0x7feec807a6a0 msgr2=0x7feec807aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:54.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.849+0000 7feed049f700 1 -- 172.21.15.70:0/2919253630 shutdown_connections 2024-06-04T05:24:54.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.849+0000 7feed049f700 1 -- 172.21.15.70:0/2919253630 wait complete. 2024-06-04T05:24:54.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.850+0000 7feed049f700 1 Processor -- start 2024-06-04T05:24:54.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.850+0000 7feed049f700 1 -- start start 2024-06-04T05:24:54.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.850+0000 7feed049f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feec81037f0 0x7feec819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:54.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.850+0000 7feed049f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feec81041a0 0x7feec819fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:54.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.850+0000 7feed049f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec81088f0 0x7feec81999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:54.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.850+0000 7feed049f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7feec8113660 con 0x7feec81088f0 2024-06-04T05:24:54.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.850+0000 7feed049f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feec81134e0 con 0x7feec81037f0 2024-06-04T05:24:54.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.850+0000 7feed049f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7feec81137e0 con 0x7feec81041a0 2024-06-04T05:24:54.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.850+0000 7feecea3c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec81088f0 0x7feec81999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:54.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.850+0000 7feecea3c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec81088f0 0x7feec81999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35360/0 (socket says 172.21.15.70:35360) 2024-06-04T05:24:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.850+0000 7feece23b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feec81037f0 0x7feec819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.850+0000 7feecea3c700 1 -- 172.21.15.70:0/1235844980 learned_addr learned my addr 172.21.15.70:0/1235844980 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:24:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.851+0000 7feecda3a700 1 --2- 172.21.15.70:0/1235844980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feec81041a0 0x7feec819fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.851+0000 7feecea3c700 1 -- 172.21.15.70:0/1235844980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feec81041a0 msgr2=0x7feec819fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.851+0000 7feecea3c700 1 --2- 172.21.15.70:0/1235844980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feec81041a0 0x7feec819fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.851+0000 7feecea3c700 1 -- 172.21.15.70:0/1235844980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feec81037f0 msgr2=0x7feec819f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.851+0000 7feecea3c700 1 --2- 172.21.15.70:0/1235844980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feec81037f0 0x7feec819f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.851+0000 7feecea3c700 1 -- 172.21.15.70:0/1235844980 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec819a100 con 0x7feec81088f0 2024-06-04T05:24:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.851+0000 7feecda3a700 1 --2- 172.21.15.70:0/1235844980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feec81041a0 0x7feec819fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:24:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.851+0000 7feecea3c700 1 --2- 172.21.15.70:0/1235844980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec81088f0 0x7feec81999a0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7feec0007d90 tx=0x7feec0007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.851+0000 7feebb7fe700 1 -- 172.21.15.70:0/1235844980 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec001f030 con 0x7feec81088f0 2024-06-04T05:24:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.851+0000 7feebb7fe700 1 -- 172.21.15.70:0/1235844980 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feec001bb90 con 0x7feec81088f0 2024-06-04T05:24:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.851+0000 7feebb7fe700 1 -- 172.21.15.70:0/1235844980 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec0012bd0 con 0x7feec81088f0 2024-06-04T05:24:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.851+0000 7feed049f700 1 -- 172.21.15.70:0/1235844980 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feec819a390 con 0x7feec81088f0 2024-06-04T05:24:54.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.851+0000 7feed049f700 1 -- 172.21.15.70:0/1235844980 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feec81a66b0 con 0x7feec81088f0 2024-06-04T05:24:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.852+0000 7feed049f700 1 -- 172.21.15.70:0/1235844980 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feec806a230 con 0x7feec81088f0 2024-06-04T05:24:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.853+0000 7feebb7fe700 1 -- 172.21.15.70:0/1235844980 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7feec0022030 con 0x7feec81088f0 2024-06-04T05:24:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.854+0000 7feebb7fe700 1 --2- 172.21.15.70:0/1235844980 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feeb40b2a70 0x7feeb40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.854+0000 7feece23b700 1 --2- 172.21.15.70:0/1235844980 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feeb40b2a70 0x7feeb40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.855+0000 7feebb7fe700 1 -- 172.21.15.70:0/1235844980 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8957+0+0 (secure 0 0 0) 0x7feec000f020 con 0x7feec81088f0 2024-06-04T05:24:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.855+0000 7feece23b700 1 --2- 172.21.15.70:0/1235844980 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feeb40b2a70 0x7feeb40b4f30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7feec819aae0 tx=0x7feec4009190 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:54.858+0000 7feebb7fe700 1 -- 172.21.15.70:0/1235844980 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7feec0029020 con 0x7feec81088f0 2024-06-04T05:24:55.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:55.058+0000 7feed049f700 1 -- 172.21.15.70:0/1235844980 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7feec810bb30 con 0x7feec81088f0 2024-06-04T05:24:55.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:55.360+0000 7feebb7fe700 1 -- 172.21.15.70:0/1235844980 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v124) ==== 181+0+0 (secure 0 0 0) 0x7feec00ab920 con 0x7feec81088f0 2024-06-04T05:24:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:55.363+0000 7feed049f700 1 -- 172.21.15.70:0/1235844980 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feeb40b2a70 msgr2=0x7feeb40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:55.363+0000 7feed049f700 1 --2- 172.21.15.70:0/1235844980 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feeb40b2a70 0x7feeb40b4f30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7feec819aae0 tx=0x7feec4009190 comp rx=0 tx=0).stop 2024-06-04T05:24:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:55.363+0000 7feed049f700 1 -- 172.21.15.70:0/1235844980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec81088f0 msgr2=0x7feec81999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:55.363+0000 7feed049f700 1 --2- 172.21.15.70:0/1235844980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec81088f0 0x7feec81999a0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7feec0007d90 tx=0x7feec0007dc0 comp rx=0 tx=0).stop 2024-06-04T05:24:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:55.364+0000 7feed049f700 1 -- 172.21.15.70:0/1235844980 shutdown_connections 2024-06-04T05:24:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:55.364+0000 7feed049f700 1 --2- 172.21.15.70:0/1235844980 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7feeb40b2a70 0x7feeb40b4f30 secure :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7feec819aae0 tx=0x7feec4009190 comp rx=0 tx=0).stop 2024-06-04T05:24:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:55.364+0000 7feed049f700 1 --2- 172.21.15.70:0/1235844980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feec81088f0 0x7feec81999a0 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:55.364+0000 7feed049f700 1 --2- 172.21.15.70:0/1235844980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feec81037f0 0x7feec819f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:55.364+0000 7feed049f700 1 --2- 172.21.15.70:0/1235844980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feec81041a0 0x7feec819fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:55.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:55.364+0000 7feed049f700 1 -- 172.21.15.70:0/1235844980 >> 172.21.15.70:0/1235844980 conn(0x7feec807a6a0 msgr2=0x7feec8101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:55.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:55.364+0000 7feed049f700 1 -- 172.21.15.70:0/1235844980 shutdown_connections 2024-06-04T05:24:55.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:55.364+0000 7feed049f700 1 -- 172.21.15.70:0/1235844980 wait complete. 2024-06-04T05:24:55.369 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' removed 2024-06-04T05:24:55.681 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:55 smithi070 ceph-mon[32631]: pgmap v876: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-06-04T05:24:55.682 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:55 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1235844980' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-04T05:24:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:55 smithi112 ceph-mon[36453]: pgmap v876: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-06-04T05:24:55.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:55 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1235844980' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-04T05:24:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:55 smithi098 ceph-mon[35569]: pgmap v876: 97 pgs: 97 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-06-04T05:24:55.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:55 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1235844980' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-04T05:24:56.045 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504 shell --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-04T05:24:56.316 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/mon.a/config 2024-06-04T05:24:56.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:56 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/1235844980' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-06-04T05:24:56.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:56 smithi112 ceph-mon[36453]: osdmap e124: 12 total, 12 up, 12 in 2024-06-04T05:24:56.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:56 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/1235844980' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-06-04T05:24:56.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:56 smithi098 ceph-mon[35569]: osdmap e124: 12 total, 12 up, 12 in 2024-06-04T05:24:56.833 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:56 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/1235844980' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-06-04T05:24:56.834 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:56 smithi070 ceph-mon[32631]: osdmap e124: 12 total, 12 up, 12 in 2024-06-04T05:24:57.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:57 smithi112 ceph-mon[36453]: pgmap v878: 65 pgs: 65 active+clean; 291 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T05:24:57.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:57 smithi070 ceph-mon[32631]: pgmap v878: 65 pgs: 65 active+clean; 291 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T05:24:57.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:57 smithi098 ceph-mon[35569]: pgmap v878: 65 pgs: 65 active+clean; 291 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T05:24:58.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.260+0000 7f740b460700 1 -- 172.21.15.70:0/3935011370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74041088f0 msgr2=0x7f7404110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:58.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.260+0000 7f740b460700 1 --2- 172.21.15.70:0/3935011370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74041088f0 0x7f7404110470 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f73fc00b0d0 tx=0x7f73fc00b3e0 comp rx=0 tx=0).stop 2024-06-04T05:24:58.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.260+0000 7f740b460700 1 -- 172.21.15.70:0/3935011370 shutdown_connections 2024-06-04T05:24:58.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.260+0000 7f740b460700 1 --2- 172.21.15.70:0/3935011370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74041088f0 0x7f7404110470 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:58.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.260+0000 7f740b460700 1 --2- 172.21.15.70:0/3935011370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74041037f0 0x7f7404103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:58.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.260+0000 7f740b460700 1 --2- 172.21.15.70:0/3935011370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74041041a0 0x7f74041081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:58.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.260+0000 7f740b460700 1 -- 172.21.15.70:0/3935011370 >> 172.21.15.70:0/3935011370 conn(0x7f740407a6a0 msgr2=0x7f740407aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:58.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.260+0000 7f740b460700 1 -- 172.21.15.70:0/3935011370 shutdown_connections 2024-06-04T05:24:58.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.260+0000 7f740b460700 1 -- 172.21.15.70:0/3935011370 wait complete. 2024-06-04T05:24:58.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.261+0000 7f740b460700 1 Processor -- start 2024-06-04T05:24:58.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.261+0000 7f740b460700 1 -- start start 2024-06-04T05:24:58.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.261+0000 7f740b460700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74041037f0 0x7f7404075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:58.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.261+0000 7f740b460700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74041041a0 0x7f7404075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:58.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.261+0000 7f740b460700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74041088f0 0x7f74040762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:58.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.261+0000 7f740b460700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7404113590 con 0x7f74041041a0 2024-06-04T05:24:58.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.261+0000 7f740b460700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7404113410 con 0x7f74041037f0 2024-06-04T05:24:58.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.261+0000 7f740b460700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7404113710 con 0x7f74041088f0 2024-06-04T05:24:58.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.261+0000 7f74089fb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74041041a0 0x7f7404075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:58.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.261+0000 7f74091fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74041037f0 0x7f7404075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:58.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.261+0000 7f74091fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74041037f0 0x7f7404075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:55332/0 (socket says 172.21.15.70:55332) 2024-06-04T05:24:58.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.261+0000 7f74089fb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74041041a0 0x7f7404075d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39280/0 (socket says 172.21.15.70:39280) 2024-06-04T05:24:58.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.261+0000 7f74091fc700 1 -- 172.21.15.70:0/5809468 learned_addr learned my addr 172.21.15.70:0/5809468 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-06-04T05:24:58.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.262+0000 7f74099fd700 1 --2- 172.21.15.70:0/5809468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74041088f0 0x7f74040762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:58.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.262+0000 7f74089fb700 1 -- 172.21.15.70:0/5809468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74041088f0 msgr2=0x7f74040762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:58.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.262+0000 7f74089fb700 1 --2- 172.21.15.70:0/5809468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74041088f0 0x7f74040762c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:58.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.262+0000 7f74089fb700 1 -- 172.21.15.70:0/5809468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74041037f0 msgr2=0x7f7404075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.262+0000 7f74089fb700 1 --2- 172.21.15.70:0/5809468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74041037f0 0x7f7404075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.262+0000 7f74089fb700 1 -- 172.21.15.70:0/5809468 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74041a7f60 con 0x7f74041041a0 2024-06-04T05:24:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.262+0000 7f74099fd700 1 --2- 172.21.15.70:0/5809468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74041088f0 0x7f74040762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-04T05:24:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.262+0000 7f74089fb700 1 --2- 172.21.15.70:0/5809468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74041041a0 0x7f7404075d80 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f73f400d750 tx=0x7f73f400da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.262+0000 7f73fa7fc700 1 -- 172.21.15.70:0/5809468 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f4017070 con 0x7f74041041a0 2024-06-04T05:24:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.262+0000 7f73fa7fc700 1 -- 172.21.15.70:0/5809468 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f73f4005020 con 0x7f74041041a0 2024-06-04T05:24:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.262+0000 7f740b460700 1 -- 172.21.15.70:0/5809468 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74041a8250 con 0x7f74041041a0 2024-06-04T05:24:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.262+0000 7f73fa7fc700 1 -- 172.21.15.70:0/5809468 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f400f980 con 0x7f74041041a0 2024-06-04T05:24:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.262+0000 7f740b460700 1 -- 172.21.15.70:0/5809468 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74041a8790 con 0x7f74041041a0 2024-06-04T05:24:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.263+0000 7f740b460700 1 -- 172.21.15.70:0/5809468 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f740406a230 con 0x7f74041041a0 2024-06-04T05:24:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.264+0000 7f73fa7fc700 1 -- 172.21.15.70:0/5809468 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 149783+0+0 (secure 0 0 0) 0x7f73f4004a30 con 0x7f74041041a0 2024-06-04T05:24:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.265+0000 7f73fa7fc700 1 --2- 172.21.15.70:0/5809468 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73f00b2a70 0x7f73f00b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-04T05:24:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.265+0000 7f74091fc700 1 --2- 172.21.15.70:0/5809468 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73f00b2a70 0x7f73f00b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-04T05:24:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.265+0000 7f73fa7fc700 1 -- 172.21.15.70:0/5809468 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8403+0+0 (secure 0 0 0) 0x7f73f40e1c40 con 0x7f74041041a0 2024-06-04T05:24:58.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.265+0000 7f74091fc700 1 --2- 172.21.15.70:0/5809468 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73f00b2a70 0x7f73f00b4f30 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7400005950 tx=0x7f740000b500 comp rx=0 tx=0).ready entity=mgr.24806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-04T05:24:58.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.270+0000 7f73fa7fc700 1 -- 172.21.15.70:0/5809468 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f73f40a9690 con 0x7f74041041a0 2024-06-04T05:24:58.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:58.475+0000 7f740b460700 1 -- 172.21.15.70:0/5809468 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f740402e560 con 0x7f74041041a0 2024-06-04T05:24:59.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:59.381+0000 7f73fa7fc700 1 -- 172.21.15.70:0/5809468 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v125) ==== 169+0+0 (secure 0 0 0) 0x7f73f40a8f90 con 0x7f74041041a0 2024-06-04T05:24:59.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:59.385+0000 7f740b460700 1 -- 172.21.15.70:0/5809468 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73f00b2a70 msgr2=0x7f73f00b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:59.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:59.385+0000 7f740b460700 1 --2- 172.21.15.70:0/5809468 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73f00b2a70 0x7f73f00b4f30 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7400005950 tx=0x7f740000b500 comp rx=0 tx=0).stop 2024-06-04T05:24:59.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:59.386+0000 7f740b460700 1 -- 172.21.15.70:0/5809468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74041041a0 msgr2=0x7f7404075d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-04T05:24:59.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:59.386+0000 7f740b460700 1 --2- 172.21.15.70:0/5809468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74041041a0 0x7f7404075d80 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f73f400d750 tx=0x7f73f400da60 comp rx=0 tx=0).stop 2024-06-04T05:24:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:59.387+0000 7f740b460700 1 -- 172.21.15.70:0/5809468 shutdown_connections 2024-06-04T05:24:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:59.387+0000 7f740b460700 1 --2- 172.21.15.70:0/5809468 >> [v2:172.21.15.70:6800/542036627,v1:172.21.15.70:6801/542036627] conn(0x7f73f00b2a70 0x7f73f00b4f30 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:59.387+0000 7f740b460700 1 --2- 172.21.15.70:0/5809468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74041088f0 0x7f74040762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:59.387+0000 7f740b460700 1 --2- 172.21.15.70:0/5809468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74041037f0 0x7f7404075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:59.387+0000 7f740b460700 1 --2- 172.21.15.70:0/5809468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74041041a0 0x7f7404075d80 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-04T05:24:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:59.387+0000 7f740b460700 1 -- 172.21.15.70:0/5809468 >> 172.21.15.70:0/5809468 conn(0x7f740407a6a0 msgr2=0x7f74041018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-04T05:24:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:59.388+0000 7f740b460700 1 -- 172.21.15.70:0/5809468 shutdown_connections 2024-06-04T05:24:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-06-04T05:24:59.388+0000 7f740b460700 1 -- 172.21.15.70:0/5809468 wait complete. 2024-06-04T05:24:59.392 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' removed 2024-06-04T05:24:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:59 smithi112 ceph-mon[36453]: pgmap v879: 65 pgs: 65 active+clean; 291 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T05:24:59.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:24:59 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/5809468' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-04T05:24:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:59 smithi098 ceph-mon[35569]: pgmap v879: 65 pgs: 65 active+clean; 291 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T05:24:59.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:24:59 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/5809468' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-04T05:24:59.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:59 smithi070 ceph-mon[32631]: pgmap v879: 65 pgs: 65 active+clean; 291 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-04T05:24:59.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:24:59 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/5809468' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-04T05:25:00.220 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-04T05:25:00.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-04T05:25:00.307 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-04T05:25:00.343 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-06-04T05:25:00.343 DEBUG:teuthology.orchestra.run.smithi070:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-04T05:25:00.370 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-04T05:25:00.386 DEBUG:teuthology.orchestra.run.smithi112:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-04T05:25:00.404 INFO:tasks.cephadm:Stopping all daemons... 2024-06-04T05:25:00.404 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-06-04T05:25:00.404 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mon.a 2024-06-04T05:25:00.456 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:25:00 smithi070 ceph-mon[32631]: from='client.? 172.21.15.70:0/5809468' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-06-04T05:25:00.456 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:25:00 smithi070 ceph-mon[32631]: osdmap e125: 12 total, 12 up, 12 in 2024-06-04T05:25:00.456 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 05:25:00 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T05:25:00.260+0000 7fb96e9e9700 -1 client.15261: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2024-06-04T05:25:00.456 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 05:25:00 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T05:25:00.261+0000 7fb96e9e9700 -1 client.15261: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2024-06-04T05:25:00.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:25:00 smithi112 ceph-mon[36453]: from='client.? 172.21.15.70:0/5809468' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-06-04T05:25:00.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:25:00 smithi112 ceph-mon[36453]: osdmap e125: 12 total, 12 up, 12 in 2024-06-04T05:25:00.722 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:25:00 smithi070 systemd[1]: Stopping Ceph mon.a for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:25:00.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:25:00 smithi098 ceph-mon[35569]: from='client.? 172.21.15.70:0/5809468' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-06-04T05:25:00.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:25:00 smithi098 ceph-mon[35569]: osdmap e125: 12 total, 12 up, 12 in 2024-06-04T05:25:01.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:25:00 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-a[32627]: 2024-06-04T05:25:00.719+0000 7fdaa99a7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:25:01.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:25:00 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-a[32627]: 2024-06-04T05:25:00.719+0000 7fdaa99a7700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-06-04T05:25:01.109 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:25:00 smithi070 podman[166541]: 2024-06-04 05:25:00.781861099 +0000 UTC m=+0.174480156 container died e29249260accb47368f128a967c2d241c9c8843c9a20926d82ae02f15fb6bbe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-b7328fe) 2024-06-04T05:25:01.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:25:01 smithi070 podman[166541]: 2024-06-04 05:25:01.345880269 +0000 UTC m=+0.738499312 container cleanup e29249260accb47368f128a967c2d241c9c8843c9a20926d82ae02f15fb6bbe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-a, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-b7328fe, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-06-04T05:25:01.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Jun 04 05:25:01 smithi070 bash[166541]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-a 2024-06-04T05:25:01.801 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mon.a.service' 2024-06-04T05:25:01.863 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:25:01.863 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-06-04T05:25:01.864 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-06-04T05:25:01.864 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mon.b 2024-06-04T05:25:02.212 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:25:01 smithi098 systemd[1]: Stopping Ceph mon.b for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:25:02.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:25:02 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-b[35565]: 2024-06-04T05:25:02.210+0000 7f8b14598700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:25:02.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:25:02 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-b[35565]: 2024-06-04T05:25:02.210+0000 7f8b14598700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-06-04T05:25:02.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:25:02 smithi098 podman[78732]: 2024-06-04 05:25:02.394318618 +0000 UTC m=+0.293686902 container died f688c5292625b8c15e5207a27b04de0676ba1a58702cb1f3472c29579d6ebff4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=main-b7328fe, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0) 2024-06-04T05:25:03.181 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:25:02 smithi098 podman[78732]: 2024-06-04 05:25:02.88629406 +0000 UTC m=+0.785662351 container cleanup f688c5292625b8c15e5207a27b04de0676ba1a58702cb1f3472c29579d6ebff4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-b, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , RELEASE=main-b7328fe, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-06-04T05:25:03.181 INFO:journalctl@ceph.mon.b.smithi098.stdout:Jun 04 05:25:02 smithi098 bash[78732]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-b 2024-06-04T05:25:03.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mon.b.service' 2024-06-04T05:25:03.363 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:25:03.363 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-06-04T05:25:03.363 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-06-04T05:25:03.364 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mon.c 2024-06-04T05:25:03.686 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:25:03 smithi112 systemd[1]: Stopping Ceph mon.c for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:25:04.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:25:03 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-c[36436]: 2024-06-04T05:25:03.765+0000 7fc2ae57e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:25:04.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:25:03 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-c[36436]: 2024-06-04T05:25:03.765+0000 7fc2ae57e700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-06-04T05:25:04.186 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:25:03 smithi112 podman[80041]: 2024-06-04 05:25:03.847471493 +0000 UTC m=+0.209876070 container died ddd0d2b2cd3f9e471da04690bd0e0c608fc3cb988e25a4182ce74a8bfdeebd2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.build-date=20240521, RELEASE=main-b7328fe, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-04T05:25:04.543 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:25:04 smithi112 podman[80041]: 2024-06-04 05:25:04.307722323 +0000 UTC m=+0.670126901 container cleanup ddd0d2b2cd3f9e471da04690bd0e0c608fc3cb988e25a4182ce74a8bfdeebd2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-c, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-04T05:25:04.544 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:25:04 smithi112 bash[80041]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-c 2024-06-04T05:25:04.788 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mon.c.service' 2024-06-04T05:25:04.810 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:25:04 smithi112 podman[80054]: 2024-06-04 05:25:04.542631459 +0000 UTC m=+0.695184070 container remove ddd0d2b2cd3f9e471da04690bd0e0c608fc3cb988e25a4182ce74a8bfdeebd2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mon-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-06-04T05:25:04.810 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:25:04 smithi112 systemd[1]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mon.c.service: Deactivated successfully. 2024-06-04T05:25:04.810 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:25:04 smithi112 systemd[1]: Stopped Ceph mon.c for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T05:25:04.810 INFO:journalctl@ceph.mon.c.smithi112.stdout:Jun 04 05:25:04 smithi112 systemd[1]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mon.c.service: Consumed 19.500s CPU time. 2024-06-04T05:25:04.844 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:25:04.844 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-06-04T05:25:04.844 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-06-04T05:25:04.844 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mgr.x 2024-06-04T05:25:05.176 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 05:25:04 smithi070 systemd[1]: Stopping Ceph mgr.x for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:25:05.520 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 05:25:05 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x[32859]: 2024-06-04T05:25:05.173+0000 7fba2c8ff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:25:05.520 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 05:25:05 smithi070 podman[166663]: 2024-06-04 05:25:05.206844967 +0000 UTC m=+0.165635192 container died b162b5a386994b5216cbdedc938e3e215a665f9b393d5542265d174dad9867e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-06-04T05:25:05.858 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 05:25:05 smithi070 podman[166663]: 2024-06-04 05:25:05.618231377 +0000 UTC m=+0.577021593 container cleanup b162b5a386994b5216cbdedc938e3e215a665f9b393d5542265d174dad9867e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b7328fed0e2e9ab66b09f8134f3cebfc71214504, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, RELEASE=main-b7328fe, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True) 2024-06-04T05:25:05.859 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Jun 04 05:25:05 smithi070 bash[166663]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-x 2024-06-04T05:25:06.049 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mgr.x.service' 2024-06-04T05:25:06.085 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:25:06.085 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-06-04T05:25:06.085 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-06-04T05:25:06.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mgr.y 2024-06-04T05:25:06.431 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 05:25:06 smithi098 systemd[1]: Stopping Ceph mgr.y for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:25:06.745 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 05:25:06 smithi098 podman[78853]: 2024-06-04 05:25:06.429940174 +0000 UTC m=+0.137743036 container died 6d890c31030897bee872995330708a4ab7e5de8fad69dca0954ec6e89d6f28cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-06-04T05:25:07.090 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 05:25:06 smithi098 podman[78853]: 2024-06-04 05:25:06.870438165 +0000 UTC m=+0.578241059 container cleanup 6d890c31030897bee872995330708a4ab7e5de8fad69dca0954ec6e89d6f28cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-06-04T05:25:07.091 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 05:25:06 smithi098 bash[78853]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y 2024-06-04T05:25:07.342 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 05:25:07 smithi098 podman[78866]: 2024-06-04 05:25:07.088901267 +0000 UTC m=+0.662170293 container remove 6d890c31030897bee872995330708a4ab7e5de8fad69dca0954ec6e89d6f28cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-b7328fe, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-06-04T05:25:07.342 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Jun 04 05:25:07 smithi098 systemd[1]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-06-04T05:25:07.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mgr.y.service' 2024-06-04T05:25:07.380 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:25:07.381 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-06-04T05:25:07.381 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-06-04T05:25:07.381 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mgr.z 2024-06-04T05:25:07.686 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 05:25:07 smithi112 systemd[1]: Stopping Ceph mgr.z for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:25:08.172 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 05:25:07 smithi112 podman[80161]: 2024-06-04 05:25:07.874836666 +0000 UTC m=+0.303083253 container stop 0b79d87cdfcaf8c83847990679ce64829fa7767a65a4f3963748534a4f387fd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-b7328fe, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-04T05:25:08.172 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 05:25:07 smithi112 podman[80161]: 2024-06-04 05:25:07.900421619 +0000 UTC m=+0.328668215 container died 0b79d87cdfcaf8c83847990679ce64829fa7767a65a4f3963748534a4f387fd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, RELEASE=main-b7328fe, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-06-04T05:25:08.436 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 05:25:08 smithi112 podman[80161]: 2024-06-04 05:25:08.279356526 +0000 UTC m=+0.707603126 container cleanup 0b79d87cdfcaf8c83847990679ce64829fa7767a65a4f3963748534a4f387fd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-b7328fe, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-06-04T05:25:08.436 INFO:journalctl@ceph.mgr.z.smithi112.stdout:Jun 04 05:25:08 smithi112 bash[80161]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mgr-z 2024-06-04T05:25:08.703 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mgr.z.service' 2024-06-04T05:25:08.737 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:25:08.737 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-06-04T05:25:08.738 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-04T05:25:08.738 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.0 2024-06-04T05:25:09.057 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 05:25:08 smithi070 systemd[1]: Stopping Ceph osd.0 for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:25:09.358 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 05:25:09 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-0[47458]: 2024-06-04T05:25:09.055+0000 7f9f205cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:25:09.359 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 05:25:09 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-0[47458]: 2024-06-04T05:25:09.055+0000 7f9f205cd700 -1 osd.0 125 *** Got signal Terminated *** 2024-06-04T05:25:09.359 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 05:25:09 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-0[47458]: 2024-06-04T05:25:09.055+0000 7f9f205cd700 -1 osd.0 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-04T05:25:14.851 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 05:25:14 smithi070 podman[166785]: 2024-06-04 05:25:14.536286071 +0000 UTC m=+5.605973635 container died cb47a300e8afdb52a4320af60d3cf038e34a153e0fb52c9ef71d4db998ced7c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-b7328fe, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux ) 2024-06-04T05:25:15.108 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 05:25:15 smithi070 podman[166785]: 2024-06-04 05:25:15.065238712 +0000 UTC m=+6.134926319 container cleanup cb47a300e8afdb52a4320af60d3cf038e34a153e0fb52c9ef71d4db998ced7c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-06-04T05:25:15.109 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 05:25:15 smithi070 bash[166785]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-0 2024-06-04T05:25:15.608 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 05:25:15 smithi070 podman[166800]: 2024-06-04 05:25:15.292069697 +0000 UTC m=+0.757945252 container remove cb47a300e8afdb52a4320af60d3cf038e34a153e0fb52c9ef71d4db998ced7c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe) 2024-06-04T05:25:16.022 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 05:25:15 smithi070 podman[166893]: 2024-06-04 05:25:15.644200985 +0000 UTC m=+0.016866029 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T05:25:16.022 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 05:25:15 smithi070 podman[166893]: 2024-06-04 05:25:15.768039878 +0000 UTC m=+0.140704905 container create f1ca5bcaac0aeef22e244f69a06b91bb9e0a87e9fce9079cc2c639342f011240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-0-deactivate, RELEASE=main-b7328fe, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521) 2024-06-04T05:25:16.022 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 05:25:16 smithi070 podman[166893]: 2024-06-04 05:25:16.02070661 +0000 UTC m=+0.393371654 container init f1ca5bcaac0aeef22e244f69a06b91bb9e0a87e9fce9079cc2c639342f011240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, RELEASE=main-b7328fe, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-06-04T05:25:16.288 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 05:25:16 smithi070 podman[166893]: 2024-06-04 05:25:16.024319761 +0000 UTC m=+0.396984803 container start f1ca5bcaac0aeef22e244f69a06b91bb9e0a87e9fce9079cc2c639342f011240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-b7328fe, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-06-04T05:25:16.288 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 05:25:16 smithi070 podman[166893]: 2024-06-04 05:25:16.154722426 +0000 UTC m=+0.527387462 container attach f1ca5bcaac0aeef22e244f69a06b91bb9e0a87e9fce9079cc2c639342f011240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-06-04T05:25:16.608 INFO:journalctl@ceph.osd.0.smithi070.stdout:Jun 04 05:25:16 smithi070 podman[166893]: 2024-06-04 05:25:16.365882884 +0000 UTC m=+0.738547932 container died f1ca5bcaac0aeef22e244f69a06b91bb9e0a87e9fce9079cc2c639342f011240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-06-04T05:25:17.168 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.0.service' 2024-06-04T05:25:17.206 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:25:17.207 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-04T05:25:17.207 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-06-04T05:25:17.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.1 2024-06-04T05:25:17.495 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 05:25:17 smithi098 systemd[1]: Stopping Ceph osd.1 for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:25:17.995 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 05:25:17 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-1[40165]: 2024-06-04T05:25:17.500+0000 7f76f4d5b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:25:17.996 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 05:25:17 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-1[40165]: 2024-06-04T05:25:17.500+0000 7f76f4d5b700 -1 osd.1 125 *** Got signal Terminated *** 2024-06-04T05:25:17.996 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 05:25:17 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-1[40165]: 2024-06-04T05:25:17.500+0000 7f76f4d5b700 -1 osd.1 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-04T05:25:23.245 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 05:25:22 smithi098 podman[78987]: 2024-06-04 05:25:22.966677794 +0000 UTC m=+5.572821859 container died a5622784960a50c779d201fca3d815d818892b6b036a3f3620df30bcab10a1a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-1, io.buildah.version=1.33.5, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-06-04T05:25:23.670 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 05:25:23 smithi098 podman[78987]: 2024-06-04 05:25:23.384444196 +0000 UTC m=+5.990588266 container cleanup a5622784960a50c779d201fca3d815d818892b6b036a3f3620df30bcab10a1a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-1, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-06-04T05:25:23.670 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 05:25:23 smithi098 bash[78987]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-1 2024-06-04T05:25:23.995 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 05:25:23 smithi098 podman[79004]: 2024-06-04 05:25:23.669481677 +0000 UTC m=+0.665300826 container remove a5622784960a50c779d201fca3d815d818892b6b036a3f3620df30bcab10a1a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-1, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-b7328fe, org.label-schema.license=GPLv2) 2024-06-04T05:25:24.300 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 05:25:24 smithi098 podman[79091]: 2024-06-04 05:25:23.91122545 +0000 UTC m=+0.017274194 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T05:25:24.300 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 05:25:24 smithi098 podman[79091]: 2024-06-04 05:25:24.05435813 +0000 UTC m=+0.160406855 container create 82730396cc2f6228023e079e7a9d6d20091469a046fe32356394e10b5aaa5ba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=main-b7328fe, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-06-04T05:25:24.649 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 05:25:24 smithi098 podman[79091]: 2024-06-04 05:25:24.298716022 +0000 UTC m=+0.404764764 container init 82730396cc2f6228023e079e7a9d6d20091469a046fe32356394e10b5aaa5ba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-06-04T05:25:24.649 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 05:25:24 smithi098 podman[79091]: 2024-06-04 05:25:24.302131244 +0000 UTC m=+0.408179978 container start 82730396cc2f6228023e079e7a9d6d20091469a046fe32356394e10b5aaa5ba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, RELEASE=main-b7328fe, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-04T05:25:24.649 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 05:25:24 smithi098 podman[79091]: 2024-06-04 05:25:24.407189674 +0000 UTC m=+0.513238412 container attach 82730396cc2f6228023e079e7a9d6d20091469a046fe32356394e10b5aaa5ba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-1-deactivate, RELEASE=main-b7328fe, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-06-04T05:25:24.995 INFO:journalctl@ceph.osd.1.smithi098.stdout:Jun 04 05:25:24 smithi098 podman[79091]: 2024-06-04 05:25:24.648109132 +0000 UTC m=+0.754157873 container died 82730396cc2f6228023e079e7a9d6d20091469a046fe32356394e10b5aaa5ba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-b7328fe, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-04T05:25:25.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.1.service' 2024-06-04T05:25:25.337 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:25:25.337 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-06-04T05:25:25.337 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-04T05:25:25.337 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.2 2024-06-04T05:25:25.652 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 05:25:25 smithi112 systemd[1]: Stopping Ceph osd.2 for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:25:25.936 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 05:25:25 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-2[41553]: 2024-06-04T05:25:25.649+0000 7f3e89a68700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:25:25.936 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 05:25:25 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-2[41553]: 2024-06-04T05:25:25.649+0000 7f3e89a68700 -1 osd.2 125 *** Got signal Terminated *** 2024-06-04T05:25:25.936 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 05:25:25 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-2[41553]: 2024-06-04T05:25:25.649+0000 7f3e89a68700 -1 osd.2 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-04T05:25:31.319 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 05:25:31 smithi112 podman[80286]: 2024-06-04 05:25:31.030857133 +0000 UTC m=+5.499840733 container died 8d000e948eadab89dc3e3cfdb115be76c875ba3368b4d386b28aff032833901f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-b7328fe, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-06-04T05:25:31.686 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 05:25:31 smithi112 podman[80286]: 2024-06-04 05:25:31.416749329 +0000 UTC m=+5.885732926 container cleanup 8d000e948eadab89dc3e3cfdb115be76c875ba3368b4d386b28aff032833901f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-06-04T05:25:31.686 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 05:25:31 smithi112 bash[80286]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-2 2024-06-04T05:25:32.088 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 05:25:31 smithi112 podman[80303]: 2024-06-04 05:25:31.743587178 +0000 UTC m=+0.708457834 container remove 8d000e948eadab89dc3e3cfdb115be76c875ba3368b4d386b28aff032833901f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, RELEASE=main-b7328fe) 2024-06-04T05:25:32.431 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 05:25:32 smithi112 podman[80390]: 2024-06-04 05:25:31.989416623 +0000 UTC m=+0.017115599 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T05:25:32.432 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 05:25:32 smithi112 podman[80390]: 2024-06-04 05:25:32.103044284 +0000 UTC m=+0.130743247 container create 709ec79bab25439c42cb9cf265bd519501f0253bdbd84fad0e181a751b81dcd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-06-04T05:25:32.432 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 05:25:32 smithi112 podman[80390]: 2024-06-04 05:25:32.330600986 +0000 UTC m=+0.358299956 container init 709ec79bab25439c42cb9cf265bd519501f0253bdbd84fad0e181a751b81dcd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-06-04T05:25:32.432 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 05:25:32 smithi112 podman[80390]: 2024-06-04 05:25:32.333892749 +0000 UTC m=+0.361591727 container start 709ec79bab25439c42cb9cf265bd519501f0253bdbd84fad0e181a751b81dcd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-b7328fe, ceph=True, CEPH_POINT_RELEASE=) 2024-06-04T05:25:32.432 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 05:25:32 smithi112 podman[80390]: 2024-06-04 05:25:32.430447631 +0000 UTC m=+0.458146614 container attach 709ec79bab25439c42cb9cf265bd519501f0253bdbd84fad0e181a751b81dcd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-b7328fe, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True) 2024-06-04T05:25:32.686 INFO:journalctl@ceph.osd.2.smithi112.stdout:Jun 04 05:25:32 smithi112 podman[80390]: 2024-06-04 05:25:32.643392706 +0000 UTC m=+0.671091678 container died 709ec79bab25439c42cb9cf265bd519501f0253bdbd84fad0e181a751b81dcd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-b7328fe, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD) 2024-06-04T05:25:33.267 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.2.service' 2024-06-04T05:25:33.308 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:25:33.308 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-04T05:25:33.308 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-06-04T05:25:33.309 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.3 2024-06-04T05:25:33.608 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 05:25:33 smithi070 systemd[1]: Stopping Ceph osd.3 for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:25:34.108 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 05:25:33 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-3[53903]: 2024-06-04T05:25:33.643+0000 7f672d21f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:25:34.109 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 05:25:33 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-3[53903]: 2024-06-04T05:25:33.643+0000 7f672d21f700 -1 osd.3 125 *** Got signal Terminated *** 2024-06-04T05:25:34.109 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 05:25:33 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-3[53903]: 2024-06-04T05:25:33.643+0000 7f672d21f700 -1 osd.3 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-04T05:25:36.362 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:36 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:36.007+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:36.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:36 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:36.156+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:37.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:37 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:37.000+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:37.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:37 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:37.200+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:37.686 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:37 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:37.246+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:37.687 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:37 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:37.360+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:38.043 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 05:25:37 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-3[53903]: 2024-06-04T05:25:37.685+0000 7f672580b700 -1 osd.3 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.268732+0000 front 2024-06-04T05:25:12.268708+0000 (oldest deadline 2024-06-04T05:25:37.568905+0000) 2024-06-04T05:25:38.322 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:38 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:38.041+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:38.495 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:38 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:38.203+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:38.608 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:38 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:38.320+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:38.686 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:38 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:38.215+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:38.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:38 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:38.227+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:39.355 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:39 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:39.028+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:39.356 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 05:25:39 smithi070 podman[167012]: 2024-06-04 05:25:39.178377529 +0000 UTC m=+5.666092412 container died be0a6cb91d7d168c1f33e3cd5e3f54fc3f3737fdcd9bde494198e210830120f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-3, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-b7328fe, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-06-04T05:25:39.495 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:39 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:39.197+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:39.609 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:39 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:39.354+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:39.686 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:39 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:39.227+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:39.687 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:39 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:39.250+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:39.964 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 05:25:39 smithi070 podman[167012]: 2024-06-04 05:25:39.694375056 +0000 UTC m=+6.182089958 container cleanup be0a6cb91d7d168c1f33e3cd5e3f54fc3f3737fdcd9bde494198e210830120f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-b7328fe, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-06-04T05:25:39.964 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 05:25:39 smithi070 bash[167012]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-3 2024-06-04T05:25:40.280 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 05:25:39 smithi070 podman[167029]: 2024-06-04 05:25:39.962548005 +0000 UTC m=+0.777961235 container remove be0a6cb91d7d168c1f33e3cd5e3f54fc3f3737fdcd9bde494198e210830120f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-06-04T05:25:40.280 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:40 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:40.008+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:40.495 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:40 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:40.203+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:40.542 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 05:25:40 smithi070 podman[167121]: 2024-06-04 05:25:40.181270394 +0000 UTC m=+0.017162048 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T05:25:40.542 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 05:25:40 smithi070 podman[167121]: 2024-06-04 05:25:40.296853189 +0000 UTC m=+0.132744828 container create 1881639ff5952d532befdb309f9f9ed2ba3e973490ba2872cf356be3fdcb0402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-b7328fe, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-06-04T05:25:40.543 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:40 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:40.332+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:40.686 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:40 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:40.199+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:40.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:40 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:40.288+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:40.837 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 05:25:40 smithi070 podman[167121]: 2024-06-04 05:25:40.541558272 +0000 UTC m=+0.377449921 container init 1881639ff5952d532befdb309f9f9ed2ba3e973490ba2872cf356be3fdcb0402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-b7328fe, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0) 2024-06-04T05:25:40.838 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 05:25:40 smithi070 podman[167121]: 2024-06-04 05:25:40.545344577 +0000 UTC m=+0.381236211 container start 1881639ff5952d532befdb309f9f9ed2ba3e973490ba2872cf356be3fdcb0402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240521, RELEASE=main-b7328fe, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-06-04T05:25:40.838 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 05:25:40 smithi070 podman[167121]: 2024-06-04 05:25:40.641423039 +0000 UTC m=+0.477314674 container attach 1881639ff5952d532befdb309f9f9ed2ba3e973490ba2872cf356be3fdcb0402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-04T05:25:41.108 INFO:journalctl@ceph.osd.3.smithi070.stdout:Jun 04 05:25:40 smithi070 podman[167121]: 2024-06-04 05:25:40.836607867 +0000 UTC m=+0.672499505 container died 1881639ff5952d532befdb309f9f9ed2ba3e973490ba2872cf356be3fdcb0402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-06-04T05:25:41.109 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:41 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:41.036+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:41.495 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:41 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:41.158+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:41.496 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:41 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:41.326+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:41.519 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.3.service' 2024-06-04T05:25:41.560 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:25:41.560 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-06-04T05:25:41.561 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-04T05:25:41.561 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.4 2024-06-04T05:25:41.686 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:41 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:41.246+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:41.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:41 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:41.244+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:41.849 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 05:25:41 smithi098 systemd[1]: Stopping Ceph osd.4 for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:25:42.116 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 05:25:41 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-4[46658]: 2024-06-04T05:25:41.847+0000 7f6cd49e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:25:42.116 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 05:25:41 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-4[46658]: 2024-06-04T05:25:41.847+0000 7f6cd49e9700 -1 osd.4 125 *** Got signal Terminated *** 2024-06-04T05:25:42.116 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 05:25:41 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-4[46658]: 2024-06-04T05:25:41.847+0000 7f6cd49e9700 -1 osd.4 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-04T05:25:42.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:42 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:42.019+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:42.495 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:42 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:42.114+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:42.686 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:42.294+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:42.687 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:42.250+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:42.858 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:42 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:42.371+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:43.055 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:43 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:43.049+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:43.495 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:43 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:43.132+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:43.609 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:43 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:43.335+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:43.686 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:43 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:43.289+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:43.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:43 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:43.229+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:44.350 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:44 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:44.070+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:44.351 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:44 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:44.070+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:25:44.495 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:44 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:44.162+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:44.495 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:44 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:44.198+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:25:44.546 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:44 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:44.245+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:44.546 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:44 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:44.241+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:44.608 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:44 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:44.349+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:45.233 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:45 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:45.202+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:45.233 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:44 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:44.971+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:25:45.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:45 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:45.033+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:45.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:45 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:45.033+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:25:45.396 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 05:25:45 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-4[46658]: 2024-06-04T05:25:45.022+0000 7f6cccfd5700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.307706+0000 front 2024-06-04T05:25:19.307747+0000 (oldest deadline 2024-06-04T05:25:44.607868+0000) 2024-06-04T05:25:45.397 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:45 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:45.210+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:25:45.397 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:45 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:45.123+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:45.633 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:45 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:45.376+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:45.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:45 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:45.231+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:45.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:45 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:45.231+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:25:46.351 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:46 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:46.057+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:46.351 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:46 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:46.057+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:25:46.436 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:46 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:46.010+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:25:46.436 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:46 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:46.192+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:46.436 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:46 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:46.192+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.591671+0000 front 2024-06-04T05:25:19.591691+0000 (oldest deadline 2024-06-04T05:25:45.491717+0000) 2024-06-04T05:25:46.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:46 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:46.224+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:46.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:46 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:46.224+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:25:46.495 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 05:25:46 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-4[46658]: 2024-06-04T05:25:46.006+0000 7f6cccfd5700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.307706+0000 front 2024-06-04T05:25:19.307747+0000 (oldest deadline 2024-06-04T05:25:44.607868+0000) 2024-06-04T05:25:46.495 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:46 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:46.246+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:25:46.495 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:46 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:46.104+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:46.496 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:46 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:46.104+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:25:46.608 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:46 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:46.348+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:47.266 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:46 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:46.968+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:25:47.266 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:47 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:47.180+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:47.266 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:47 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:47.180+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.591671+0000 front 2024-06-04T05:25:19.591691+0000 (oldest deadline 2024-06-04T05:25:45.491717+0000) 2024-06-04T05:25:47.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:47 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:47.309+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:47.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:47 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:47.309+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:16.661003+0000 front 2024-06-04T05:25:16.660950+0000 (oldest deadline 2024-06-04T05:25:46.661539+0000) 2024-06-04T05:25:47.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:47 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:47.060+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:47.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:47 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:47.060+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:25:47.495 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:47 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:47.253+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:25:47.496 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 05:25:47 smithi098 podman[79194]: 2024-06-04 05:25:47.199414131 +0000 UTC m=+5.446044057 container died e421053b69e00b6e517275247013f9121c78dc6204ca188d3f7736b684569f7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-06-04T05:25:47.496 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:47 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:47.073+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:47.496 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:47 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:47.073+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:25:47.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:47 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:47.264+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:47.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:47 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:47.264+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:25:47.834 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 05:25:47 smithi098 podman[79194]: 2024-06-04 05:25:47.614823181 +0000 UTC m=+5.861453098 container cleanup e421053b69e00b6e517275247013f9121c78dc6204ca188d3f7736b684569f7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-4, RELEASE=main-b7328fe, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-04T05:25:47.834 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 05:25:47 smithi098 bash[79194]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-4 2024-06-04T05:25:48.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:47 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:47.927+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:25:48.186 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:48 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:48.134+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:48.186 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:48 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:48.134+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.591671+0000 front 2024-06-04T05:25:19.591691+0000 (oldest deadline 2024-06-04T05:25:45.491717+0000) 2024-06-04T05:25:48.245 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 05:25:47 smithi098 podman[79221]: 2024-06-04 05:25:47.832875918 +0000 UTC m=+0.634385525 container remove e421053b69e00b6e517275247013f9121c78dc6204ca188d3f7736b684569f7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-4, org.label-schema.build-date=20240521, ceph=True, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-b7328fe) 2024-06-04T05:25:48.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:48 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:48.032+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:48.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:48 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:48.032+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:25:48.314 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:48 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:48.016+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:48.314 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:48 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:48.016+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:25:48.571 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:48 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:48.263+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:25:48.571 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 05:25:48 smithi098 podman[79311]: 2024-06-04 05:25:48.155765081 +0000 UTC m=+0.017524853 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T05:25:48.571 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 05:25:48 smithi098 podman[79311]: 2024-06-04 05:25:48.25882937 +0000 UTC m=+0.120589128 container create 5a06a180ffd12a7876b925d0af35de8db34b17200f4268e442c30245c089f546 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-b7328fe, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-06-04T05:25:48.571 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 05:25:48 smithi098 podman[79311]: 2024-06-04 05:25:48.4866978 +0000 UTC m=+0.348457546 container init 5a06a180ffd12a7876b925d0af35de8db34b17200f4268e442c30245c089f546 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-b7328fe, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-06-04T05:25:48.571 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 05:25:48 smithi098 podman[79311]: 2024-06-04 05:25:48.490129104 +0000 UTC m=+0.351888859 container start 5a06a180ffd12a7876b925d0af35de8db34b17200f4268e442c30245c089f546 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, RELEASE=main-b7328fe, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True) 2024-06-04T05:25:48.608 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:48 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:48.312+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:48.609 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:48 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:48.312+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:16.661003+0000 front 2024-06-04T05:25:16.660950+0000 (oldest deadline 2024-06-04T05:25:46.661539+0000) 2024-06-04T05:25:48.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:48 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:48.301+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:48.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:48 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:48.301+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:25:48.995 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 05:25:48 smithi098 podman[79311]: 2024-06-04 05:25:48.569967576 +0000 UTC m=+0.431727340 container attach 5a06a180ffd12a7876b925d0af35de8db34b17200f4268e442c30245c089f546 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, RELEASE=main-b7328fe, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-04T05:25:48.995 INFO:journalctl@ceph.osd.4.smithi098.stdout:Jun 04 05:25:48 smithi098 podman[79311]: 2024-06-04 05:25:48.78870753 +0000 UTC m=+0.650467283 container died 5a06a180ffd12a7876b925d0af35de8db34b17200f4268e442c30245c089f546 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=main-b7328fe, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-04T05:25:49.303 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:48 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:48.976+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:25:49.303 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:49.152+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:49.303 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:49.152+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.591671+0000 front 2024-06-04T05:25:19.591691+0000 (oldest deadline 2024-06-04T05:25:45.491717+0000) 2024-06-04T05:25:49.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:49 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:49.061+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:49.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:49 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:49.061+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:25:49.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:49 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:49.283+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:49.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:49 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:49.283+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:16.661003+0000 front 2024-06-04T05:25:16.660950+0000 (oldest deadline 2024-06-04T05:25:46.661539+0000) 2024-06-04T05:25:49.433 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:49 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:49.242+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:25:49.434 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:49 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:49.068+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:49.434 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:49 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:49.068+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:25:49.455 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.4.service' 2024-06-04T05:25:49.496 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:25:49.496 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-04T05:25:49.496 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-04T05:25:49.496 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.5 2024-06-04T05:25:49.686 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:49 smithi112 systemd[1]: Stopping Ceph osd.5 for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:25:49.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:49.301+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:49.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:49.301+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:25:50.163 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:49.947+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:25:50.163 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:49.837+0000 7fbeb44c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:25:50.164 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:49.837+0000 7fbeb44c2700 -1 osd.5 125 *** Got signal Terminated *** 2024-06-04T05:25:50.164 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:49 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:49.837+0000 7fbeb44c2700 -1 osd.5 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-04T05:25:50.334 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:50 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:50.080+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:50.335 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:50 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:50.080+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:25:50.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:50 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:50.268+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:50.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:50 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:50.268+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:25:50.436 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:50 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:50.161+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:50.436 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:50 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:50.161+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.591671+0000 front 2024-06-04T05:25:19.591691+0000 (oldest deadline 2024-06-04T05:25:45.491717+0000) 2024-06-04T05:25:50.495 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:50 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:50.278+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:25:50.495 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:50 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:50.042+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:50.495 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:50 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:50.042+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:25:50.608 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:50 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:50.332+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:50.609 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:50 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:50.332+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:16.661003+0000 front 2024-06-04T05:25:16.660950+0000 (oldest deadline 2024-06-04T05:25:46.661539+0000) 2024-06-04T05:25:51.194 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:50 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:50.941+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:25:51.327 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:51 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:51.063+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:51.327 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:51 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:51.063+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:25:51.608 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:51 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:51.115+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:51.609 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:51 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:51.115+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:25:51.609 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:51 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:51.344+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:51.609 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:51 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:51.344+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:16.661003+0000 front 2024-06-04T05:25:16.660950+0000 (oldest deadline 2024-06-04T05:25:46.661539+0000) 2024-06-04T05:25:51.686 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:51 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:51.191+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:51.686 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:51 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:51.191+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.591671+0000 front 2024-06-04T05:25:19.591691+0000 (oldest deadline 2024-06-04T05:25:45.491717+0000) 2024-06-04T05:25:51.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:51 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:51.309+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:51.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:51 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:51.309+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:25:51.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:51 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:51.325+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:25:52.212 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:51 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:51.944+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:25:52.212 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:51 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:51.944+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:25:52.340 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:52 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:52.018+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:52.340 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:52 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:52.018+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:25:52.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:52 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:52.105+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:52.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:52 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:52.105+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:25:52.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:52 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:52.352+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:52.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:52 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:52.352+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:16.661003+0000 front 2024-06-04T05:25:16.660950+0000 (oldest deadline 2024-06-04T05:25:46.661539+0000) 2024-06-04T05:25:52.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:52 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:52.276+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:52.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:52 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:52.276+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:25:52.686 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:52 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:52.210+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:52.686 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:52 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:52.210+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.591671+0000 front 2024-06-04T05:25:19.591691+0000 (oldest deadline 2024-06-04T05:25:45.491717+0000) 2024-06-04T05:25:52.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:52 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:52.338+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:25:53.248 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:53 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:53.195+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:53.248 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:53 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:53.195+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.591671+0000 front 2024-06-04T05:25:19.591691+0000 (oldest deadline 2024-06-04T05:25:45.491717+0000) 2024-06-04T05:25:53.249 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:52 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:52.963+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:25:53.249 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:52 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:52.963+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:25:53.348 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:53 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:53.012+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:53.348 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:53 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:53.012+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:25:53.348 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:53 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:53.012+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:25:53.609 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:53 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:53.119+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:53.609 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:53 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:53.119+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:25:53.609 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:53 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:53.306+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:53.609 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:53 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:53.306+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:16.661003+0000 front 2024-06-04T05:25:16.660950+0000 (oldest deadline 2024-06-04T05:25:46.661539+0000) 2024-06-04T05:25:53.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:53 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:53.246+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:53.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:53 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:53.246+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:25:53.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:53 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:53.346+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:25:54.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:53 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:53.982+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:54.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:53 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:53.982+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:25:54.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:53 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:53.982+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:25:54.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:54.087+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:54.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:54.087+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:25:54.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:54.286+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:54.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:54.286+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:16.661003+0000 front 2024-06-04T05:25:16.660950+0000 (oldest deadline 2024-06-04T05:25:46.661539+0000) 2024-06-04T05:25:54.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:54 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:54.286+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:26.662112+0000 front 2024-06-04T05:25:26.662200+0000 (oldest deadline 2024-06-04T05:25:54.262911+0000) 2024-06-04T05:25:54.436 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:54 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:54.192+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.189561+0000 front 2024-06-04T05:25:12.189644+0000 (oldest deadline 2024-06-04T05:25:36.889456+0000) 2024-06-04T05:25:54.436 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:54 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5[48470]: 2024-06-04T05:25:54.192+0000 7fbeb0ab6700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.591671+0000 front 2024-06-04T05:25:19.591691+0000 (oldest deadline 2024-06-04T05:25:45.491717+0000) 2024-06-04T05:25:54.436 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:53 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:53.981+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:25:54.436 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:53 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:53.981+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:25:54.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:54 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:54.202+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:54.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:54 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:54.202+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:25:54.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:54 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:54.322+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:25:55.298 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:54 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:54.972+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:25:55.298 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:54 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:54.972+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:25:55.298 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:55 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:55.165+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:55.298 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:55 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:55.165+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:25:55.298 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:55 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:55.165+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:25:55.319 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:55 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:55.018+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:55.319 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:55 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:55.018+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:25:55.319 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:55 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:55.018+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:25:55.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:55 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:55.064+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:55.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:55 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:55.064+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:25:55.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:55 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:55.280+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:55.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:55 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:55.280+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:16.661003+0000 front 2024-06-04T05:25:16.660950+0000 (oldest deadline 2024-06-04T05:25:46.661539+0000) 2024-06-04T05:25:55.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:55 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:55.280+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:26.662112+0000 front 2024-06-04T05:25:26.662200+0000 (oldest deadline 2024-06-04T05:25:54.262911+0000) 2024-06-04T05:25:55.613 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:55 smithi112 podman[80507]: 2024-06-04 05:25:55.296905645 +0000 UTC m=+5.598368236 container died 5bc6749a82321adf2882314b35ba87186a07c72ab3f2a1142ad4f6425b3b0ca9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-b7328fe, org.label-schema.build-date=20240521, GIT_CLEAN=True) 2024-06-04T05:25:55.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:55 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:55.317+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:25:55.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:55 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:55.317+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.272336+0000 front 2024-06-04T05:25:29.272362+0000 (oldest deadline 2024-06-04T05:25:54.572446+0000) 2024-06-04T05:25:55.934 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:55 smithi112 podman[80507]: 2024-06-04 05:25:55.612657231 +0000 UTC m=+5.914119844 container cleanup 5bc6749a82321adf2882314b35ba87186a07c72ab3f2a1142ad4f6425b3b0ca9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-b7328fe, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-06-04T05:25:55.934 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:55 smithi112 bash[80507]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5 2024-06-04T05:25:55.934 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:55 smithi112 podman[80526]: 2024-06-04 05:25:55.855608728 +0000 UTC m=+0.553128913 container remove 5bc6749a82321adf2882314b35ba87186a07c72ab3f2a1142ad4f6425b3b0ca9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-b7328fe, org.label-schema.build-date=20240521, GIT_CLEAN=True) 2024-06-04T05:25:56.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:55 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:55.933+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:25:56.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:55 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:55.933+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:25:56.349 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:56 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:56.030+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:56.349 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:56 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:56.030+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:25:56.349 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:56 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:56.030+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:25:56.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:56 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:56.055+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:56.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:56 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:56.055+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:25:56.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:56 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:56.055+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:25:56.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:56 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:56.239+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:56.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:56 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:56.239+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:16.661003+0000 front 2024-06-04T05:25:16.660950+0000 (oldest deadline 2024-06-04T05:25:46.661539+0000) 2024-06-04T05:25:56.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:56 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:56.239+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:26.662112+0000 front 2024-06-04T05:25:26.662200+0000 (oldest deadline 2024-06-04T05:25:54.262911+0000) 2024-06-04T05:25:56.474 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:56 smithi112 podman[80614]: 2024-06-04 05:25:56.183925961 +0000 UTC m=+0.017667263 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T05:25:56.474 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:56 smithi112 podman[80614]: 2024-06-04 05:25:56.314853058 +0000 UTC m=+0.148594346 container create 4dd9b9bdabe56e63b2466e6c25887b9c60876d9fb49072d18eaed94534680e83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-b7328fe, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-06-04T05:25:56.492 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:56 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:56.206+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:56.492 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:56 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:56.206+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:25:56.492 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:56 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:56.206+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:25:56.681 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:56 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:56.347+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:25:56.681 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:56 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:56.347+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.272336+0000 front 2024-06-04T05:25:29.272362+0000 (oldest deadline 2024-06-04T05:25:54.572446+0000) 2024-06-04T05:25:56.888 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:56 smithi112 podman[80614]: 2024-06-04 05:25:56.59265147 +0000 UTC m=+0.426392763 container init 4dd9b9bdabe56e63b2466e6c25887b9c60876d9fb49072d18eaed94534680e83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , RELEASE=main-b7328fe, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-04T05:25:56.888 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:56 smithi112 podman[80614]: 2024-06-04 05:25:56.596545004 +0000 UTC m=+0.430286305 container start 4dd9b9bdabe56e63b2466e6c25887b9c60876d9fb49072d18eaed94534680e83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521) 2024-06-04T05:25:56.888 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:56 smithi112 podman[80614]: 2024-06-04 05:25:56.675932633 +0000 UTC m=+0.509673929 container attach 4dd9b9bdabe56e63b2466e6c25887b9c60876d9fb49072d18eaed94534680e83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5-deactivate, RELEASE=main-b7328fe, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-06-04T05:25:57.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:56 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:56.912+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:25:57.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:56 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:56.912+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:25:57.186 INFO:journalctl@ceph.osd.5.smithi112.stdout:Jun 04 05:25:56 smithi112 podman[80614]: 2024-06-04 05:25:56.887396596 +0000 UTC m=+0.721137893 container died 4dd9b9bdabe56e63b2466e6c25887b9c60876d9fb49072d18eaed94534680e83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , RELEASE=main-b7328fe, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-06-04T05:25:57.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:56 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:56.986+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:57.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:56 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:56.986+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:25:57.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:56 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:56.986+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:25:57.358 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:57 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:57.240+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:57.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:57 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:57.240+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:16.661003+0000 front 2024-06-04T05:25:16.660950+0000 (oldest deadline 2024-06-04T05:25:46.661539+0000) 2024-06-04T05:25:57.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:57 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:57.240+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:26.662112+0000 front 2024-06-04T05:25:26.662200+0000 (oldest deadline 2024-06-04T05:25:54.262911+0000) 2024-06-04T05:25:57.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:57 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:57.098+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:57.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:57 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:57.098+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:25:57.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:57 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:57.098+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:25:57.506 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:57 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:57.206+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:57.506 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:57 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:57.206+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:25:57.506 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:57 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:57.206+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:25:57.531 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.5.service' 2024-06-04T05:25:57.570 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:25:57.570 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-04T05:25:57.570 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-06-04T05:25:57.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.6 2024-06-04T05:25:57.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:57 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:57.312+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:25:57.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:57 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:57.312+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.272336+0000 front 2024-06-04T05:25:29.272362+0000 (oldest deadline 2024-06-04T05:25:54.572446+0000) 2024-06-04T05:25:57.858 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:57 smithi070 systemd[1]: Stopping Ceph osd.6 for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:25:58.217 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:57 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:57.952+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:25:58.217 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:57 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:57.952+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:25:58.226 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:58 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:58.082+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:58.226 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:58 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:58.082+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:25:58.227 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:58 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:58.082+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:25:58.227 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:57 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:57.889+0000 7ff1dad20700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:25:58.227 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:57 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:57.889+0000 7ff1dad20700 -1 osd.6 125 *** Got signal Terminated *** 2024-06-04T05:25:58.227 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:57 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:57.889+0000 7ff1dad20700 -1 osd.6 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-04T05:25:58.227 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:58 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:58.225+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:58.227 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:58 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:58.225+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:16.661003+0000 front 2024-06-04T05:25:16.660950+0000 (oldest deadline 2024-06-04T05:25:46.661539+0000) 2024-06-04T05:25:58.227 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:58 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:58.225+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:26.662112+0000 front 2024-06-04T05:25:26.662200+0000 (oldest deadline 2024-06-04T05:25:54.262911+0000) 2024-06-04T05:25:58.279 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:58 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:58.024+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:58.279 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:58 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:58.024+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:25:58.279 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:58 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:58.024+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:25:58.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:58 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:58.216+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:58.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:58 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:58.216+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:25:58.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:58 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:58.216+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:25:58.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:58 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:58.277+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:25:58.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:58 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:58.277+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.272336+0000 front 2024-06-04T05:25:29.272362+0000 (oldest deadline 2024-06-04T05:25:54.572446+0000) 2024-06-04T05:25:59.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:59 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:59.033+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:25:59.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:59 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:59.033+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:25:59.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:59 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:59.033+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:25:59.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:59 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:59.198+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:25:59.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:59 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:59.198+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:16.661003+0000 front 2024-06-04T05:25:16.660950+0000 (oldest deadline 2024-06-04T05:25:46.661539+0000) 2024-06-04T05:25:59.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:25:59 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:25:59.198+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:26.662112+0000 front 2024-06-04T05:25:26.662200+0000 (oldest deadline 2024-06-04T05:25:54.262911+0000) 2024-06-04T05:25:59.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:59 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:59.168+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:25:59.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:59 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:59.168+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:25:59.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:25:59 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:25:59.168+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:25:59.436 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:58 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:58.959+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:25:59.436 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:58 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:58.959+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:25:59.495 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:59 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:59.292+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:25:59.495 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:25:59 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:25:59.292+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.272336+0000 front 2024-06-04T05:25:29.272362+0000 (oldest deadline 2024-06-04T05:25:54.572446+0000) 2024-06-04T05:25:59.495 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:59 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:59.053+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:25:59.496 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:59 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:59.053+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:25:59.496 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:25:59 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:25:59.053+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:00.186 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:00 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:00.138+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:00.186 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:00 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:00.138+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:00.186 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:00 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:00.138+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:00.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:59 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:59.915+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:00.187 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:25:59 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:25:59.915+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:00.249 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:59 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:59.989+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:00.249 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:59 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:59.989+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:00.249 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:25:59 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:25:59.989+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:00.495 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:00 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:00.252+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:26:00.495 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:00 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:00.252+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.272336+0000 front 2024-06-04T05:25:29.272362+0000 (oldest deadline 2024-06-04T05:25:54.572446+0000) 2024-06-04T05:26:00.495 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:00 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:00.050+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:00.495 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:00 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:00.050+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:00.495 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:00 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:00.050+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:00.608 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:00 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:26:00.247+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:26:00.608 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:00 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:26:00.247+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:16.661003+0000 front 2024-06-04T05:25:16.660950+0000 (oldest deadline 2024-06-04T05:25:46.661539+0000) 2024-06-04T05:26:00.609 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:00 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:26:00.247+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:26.662112+0000 front 2024-06-04T05:25:26.662200+0000 (oldest deadline 2024-06-04T05:25:54.262911+0000) 2024-06-04T05:26:00.609 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:00 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:26:00.247+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:34.263400+0000 front 2024-06-04T05:25:34.263438+0000 (oldest deadline 2024-06-04T05:26:00.163560+0000) 2024-06-04T05:26:01.151 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:00 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:00.868+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:01.151 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:00 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:00.868+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:01.301 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:01 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:01.002+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:01.301 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:01 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:01.002+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:01.301 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:01 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:01.002+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:01.301 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:01 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:01.002+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:01.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:01 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:01.024+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:01.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:01 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:01.024+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:01.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:01 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:01.024+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:01.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:01 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:01.024+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:01.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:01 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:26:01.272+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:26:01.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:01 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:26:01.272+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:16.661003+0000 front 2024-06-04T05:25:16.660950+0000 (oldest deadline 2024-06-04T05:25:46.661539+0000) 2024-06-04T05:26:01.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:01 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:26:01.272+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:26.662112+0000 front 2024-06-04T05:25:26.662200+0000 (oldest deadline 2024-06-04T05:25:54.262911+0000) 2024-06-04T05:26:01.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:01 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:26:01.272+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:34.263400+0000 front 2024-06-04T05:25:34.263438+0000 (oldest deadline 2024-06-04T05:26:00.163560+0000) 2024-06-04T05:26:01.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:01 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:01.149+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:01.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:01 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:01.149+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:01.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:01 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:01.149+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:01.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:01 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:01.149+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:01.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:01 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:01.300+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:26:01.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:01 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:01.300+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.272336+0000 front 2024-06-04T05:25:29.272362+0000 (oldest deadline 2024-06-04T05:25:54.572446+0000) 2024-06-04T05:26:02.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:01 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:01.827+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:02.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:01 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:01.827+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:02.262 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:01 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:01.979+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:02.263 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:01 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:01.979+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:02.263 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:01 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:01.979+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:02.263 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:01 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:01.979+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:02.337 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:02 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:02.036+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:02.337 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:02 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:02.036+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:02.337 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:02 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:02.036+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:02.337 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:02 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:02.036+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:02.608 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:02 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:26:02.260+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:11.360244+0000 front 2024-06-04T05:25:11.360346+0000 (oldest deadline 2024-06-04T05:25:36.660408+0000) 2024-06-04T05:26:02.609 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:02 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:26:02.260+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:16.661003+0000 front 2024-06-04T05:25:16.660950+0000 (oldest deadline 2024-06-04T05:25:46.661539+0000) 2024-06-04T05:26:02.609 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:02 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:26:02.260+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:26.662112+0000 front 2024-06-04T05:25:26.662200+0000 (oldest deadline 2024-06-04T05:25:54.262911+0000) 2024-06-04T05:26:02.609 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:02 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6[60430]: 2024-06-04T05:26:02.260+0000 7ff1d7314700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:34.263400+0000 front 2024-06-04T05:25:34.263438+0000 (oldest deadline 2024-06-04T05:26:00.163560+0000) 2024-06-04T05:26:02.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:02 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:02.187+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:02.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:02 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:02.187+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:02.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:02 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:02.187+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:02.687 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:02 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:02.187+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:02.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:02 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:02.335+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:26:02.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:02 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:02.335+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.272336+0000 front 2024-06-04T05:25:29.272362+0000 (oldest deadline 2024-06-04T05:25:54.572446+0000) 2024-06-04T05:26:03.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:02 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:02.822+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:03.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:02 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:02.822+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:03.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:02 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:02.822+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:36.360040+0000 front 2024-06-04T05:25:36.359900+0000 (oldest deadline 2024-06-04T05:26:02.259878+0000) 2024-06-04T05:26:03.294 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:02 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:02.964+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:03.294 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:02 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:02.964+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:03.294 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:02 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:02.964+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:03.294 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:02 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:02.964+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:03.340 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:03 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:03.056+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:03.341 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:03 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:03.056+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:03.341 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:03 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:03.056+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:03.341 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:03 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:03.056+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:03.461 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:03 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:03.186+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:03.461 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:03 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:03.186+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:03.461 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:03 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:03.187+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:03.461 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:03 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:03.187+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:03.608 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:03 smithi070 podman[167240]: 2024-06-04 05:26:03.292953569 +0000 UTC m=+5.518805547 container died 0fca30c8f3f18f9335b8aa14ade13914535a96b6ba832957b9e052569a5bcb12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, RELEASE=main-b7328fe, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-06-04T05:26:03.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:03 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:03.339+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:26:03.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:03 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:03.339+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.272336+0000 front 2024-06-04T05:25:29.272362+0000 (oldest deadline 2024-06-04T05:25:54.572446+0000) 2024-06-04T05:26:03.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:03 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:03.339+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:37.874465+0000 front 2024-06-04T05:25:37.874486+0000 (oldest deadline 2024-06-04T05:26:02.574620+0000) 2024-06-04T05:26:03.858 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:03 smithi070 podman[167240]: 2024-06-04 05:26:03.815036914 +0000 UTC m=+6.040888892 container cleanup 0fca30c8f3f18f9335b8aa14ade13914535a96b6ba832957b9e052569a5bcb12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-b7328fe, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521) 2024-06-04T05:26:03.859 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:03 smithi070 bash[167240]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6 2024-06-04T05:26:04.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:03 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:03.794+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:04.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:03 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:03.794+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:04.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:03 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:03.794+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:36.360040+0000 front 2024-06-04T05:25:36.359900+0000 (oldest deadline 2024-06-04T05:26:02.259878+0000) 2024-06-04T05:26:04.359 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:04 smithi070 podman[167255]: 2024-06-04 05:26:04.07476452 +0000 UTC m=+0.779481081 container remove 0fca30c8f3f18f9335b8aa14ade13914535a96b6ba832957b9e052569a5bcb12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-b7328fe, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-06-04T05:26:04.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:03 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:03.981+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:04.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:03 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:03.981+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:04.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:03 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:03.981+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:04.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:03 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:03.981+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:04.370 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:04 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:04.094+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:04.371 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:04 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:04.094+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:04.371 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:04 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:04.094+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:04.371 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:04 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:04.094+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:04.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:04 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:04.191+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:04.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:04 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:04.191+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:04.687 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:04 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:04.191+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:04.687 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:04 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:04.191+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:04.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:04 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:04.368+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:26:04.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:04 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:04.368+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.272336+0000 front 2024-06-04T05:25:29.272362+0000 (oldest deadline 2024-06-04T05:25:54.572446+0000) 2024-06-04T05:26:04.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:04 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:04.368+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:37.874465+0000 front 2024-06-04T05:25:37.874486+0000 (oldest deadline 2024-06-04T05:26:02.574620+0000) 2024-06-04T05:26:04.779 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:04 smithi070 podman[167347]: 2024-06-04 05:26:04.398453113 +0000 UTC m=+0.017852210 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T05:26:04.779 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:04 smithi070 podman[167347]: 2024-06-04 05:26:04.517268064 +0000 UTC m=+0.136667138 container create 2ba5e407f47b85f15f76de2c97f2fbf8efce289bba84f1f8ca32fde5fbd8dda8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6-deactivate, RELEASE=main-b7328fe, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-06-04T05:26:05.080 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:05 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:05.023+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:05.080 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:05 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:05.023+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:05.080 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:05 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:05.023+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:05.080 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:05 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:05.023+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:05.081 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:04 smithi070 podman[167347]: 2024-06-04 05:26:04.777908113 +0000 UTC m=+0.397307198 container init 2ba5e407f47b85f15f76de2c97f2fbf8efce289bba84f1f8ca32fde5fbd8dda8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-04T05:26:05.081 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:04 smithi070 podman[167347]: 2024-06-04 05:26:04.781076407 +0000 UTC m=+0.400475492 container start 2ba5e407f47b85f15f76de2c97f2fbf8efce289bba84f1f8ca32fde5fbd8dda8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-b7328fe, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-06-04T05:26:05.081 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:04 smithi070 podman[167347]: 2024-06-04 05:26:04.844564624 +0000 UTC m=+0.463963712 container attach 2ba5e407f47b85f15f76de2c97f2fbf8efce289bba84f1f8ca32fde5fbd8dda8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, RELEASE=main-b7328fe, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-06-04T05:26:05.081 INFO:journalctl@ceph.osd.6.smithi070.stdout:Jun 04 05:26:05 smithi070 podman[167347]: 2024-06-04 05:26:05.079386863 +0000 UTC m=+0.698785964 container died 2ba5e407f47b85f15f76de2c97f2fbf8efce289bba84f1f8ca32fde5fbd8dda8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-06-04T05:26:05.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:04 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:04.781+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:05.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:04 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:04.781+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:05.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:04 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:04.781+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:36.360040+0000 front 2024-06-04T05:25:36.359900+0000 (oldest deadline 2024-06-04T05:26:02.259878+0000) 2024-06-04T05:26:05.354 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:05 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:05.058+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:05.354 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:05 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:05.058+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:05.354 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:05 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:05.058+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:05.354 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:05 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:05.058+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:05.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.6.service' 2024-06-04T05:26:05.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:05 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:05.201+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:05.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:05 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:05.201+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:05.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:05 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:05.201+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:05.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:05 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:05.201+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:05.718 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:26:05.719 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-06-04T05:26:05.719 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-06-04T05:26:05.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.7 2024-06-04T05:26:05.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:05 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:05.352+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:26:05.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:05 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:05.352+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.272336+0000 front 2024-06-04T05:25:29.272362+0000 (oldest deadline 2024-06-04T05:25:54.572446+0000) 2024-06-04T05:26:05.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:05 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:05.352+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:37.874465+0000 front 2024-06-04T05:25:37.874486+0000 (oldest deadline 2024-06-04T05:26:02.574620+0000) 2024-06-04T05:26:06.036 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:05 smithi098 systemd[1]: Stopping Ceph osd.7 for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:26:06.174 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:05 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:05.771+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:06.174 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:05 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:05.771+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:06.174 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:05 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:05.771+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:36.360040+0000 front 2024-06-04T05:25:36.359900+0000 (oldest deadline 2024-06-04T05:26:02.259878+0000) 2024-06-04T05:26:06.354 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:06.035+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:06.354 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:06.035+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:06.354 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:06.035+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:06.354 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:06.035+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:06.355 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:06.043+0000 7fae56e43700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:26:06.355 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:06.043+0000 7fae56e43700 -1 osd.7 125 *** Got signal Terminated *** 2024-06-04T05:26:06.355 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:06.043+0000 7fae56e43700 -1 osd.7 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-04T05:26:06.355 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:06.352+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:26:06.355 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:06.352+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.272336+0000 front 2024-06-04T05:25:29.272362+0000 (oldest deadline 2024-06-04T05:25:54.572446+0000) 2024-06-04T05:26:06.355 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:06.352+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:37.874465+0000 front 2024-06-04T05:25:37.874486+0000 (oldest deadline 2024-06-04T05:26:02.574620+0000) 2024-06-04T05:26:06.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:06 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:06.052+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:06.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:06 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:06.052+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:06.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:06 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:06.052+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:06.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:06 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:06.052+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:06.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:06 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:06.172+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:06.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:06 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:06.172+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:06.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:06 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:06.172+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:06.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:06 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:06.172+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:07.183 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:06 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:06.743+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:07.183 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:06 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:06.743+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:07.183 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:06 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:06.743+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:36.360040+0000 front 2024-06-04T05:25:36.359900+0000 (oldest deadline 2024-06-04T05:26:02.259878+0000) 2024-06-04T05:26:07.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:06.988+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:07.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:06.988+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:07.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:06.988+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:07.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:06.988+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:07.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:07 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:07.093+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:07.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:07 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:07.093+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:07.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:07 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:07.093+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:07.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:07 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:07.093+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:07.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:07 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:07.181+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:07.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:07 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:07.181+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:07.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:07 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:07.181+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:07.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:07 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:07.181+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:07.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:07 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:07.323+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:26:07.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:07 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:07.323+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.272336+0000 front 2024-06-04T05:25:29.272362+0000 (oldest deadline 2024-06-04T05:25:54.572446+0000) 2024-06-04T05:26:07.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:07 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:07.323+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:37.874465+0000 front 2024-06-04T05:25:37.874486+0000 (oldest deadline 2024-06-04T05:26:02.574620+0000) 2024-06-04T05:26:08.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:07 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:07.783+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:08.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:07 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:07.783+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:08.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:07 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:07.783+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:36.360040+0000 front 2024-06-04T05:25:36.359900+0000 (oldest deadline 2024-06-04T05:26:02.259878+0000) 2024-06-04T05:26:08.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:07 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:07.990+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:08.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:07 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:07.990+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:08.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:07 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:07.990+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:08.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:07 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:07.990+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:08.609 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:08 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:08.140+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:08.609 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:08 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:08.140+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:08.609 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:08 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:08.140+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:08.609 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:08 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:08.140+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:08.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:08 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:08.192+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:08.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:08 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:08.192+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:08.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:08 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:08.192+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:08.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:08 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:08.192+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:08.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:08 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:08.322+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:26:08.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:08 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:08.322+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.272336+0000 front 2024-06-04T05:25:29.272362+0000 (oldest deadline 2024-06-04T05:25:54.572446+0000) 2024-06-04T05:26:08.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:08 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:08.322+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:37.874465+0000 front 2024-06-04T05:25:37.874486+0000 (oldest deadline 2024-06-04T05:26:02.574620+0000) 2024-06-04T05:26:08.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:08 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:08.322+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:42.575023+0000 front 2024-06-04T05:25:42.575083+0000 (oldest deadline 2024-06-04T05:26:07.875155+0000) 2024-06-04T05:26:09.167 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:08 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:08.749+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:09.167 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:08 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:08.749+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:09.167 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:08 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:08.749+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:36.360040+0000 front 2024-06-04T05:25:36.359900+0000 (oldest deadline 2024-06-04T05:26:02.259878+0000) 2024-06-04T05:26:09.167 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:08 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:08.749+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.261387+0000 front 2024-06-04T05:25:46.261364+0000 (oldest deadline 2024-06-04T05:26:08.561461+0000) 2024-06-04T05:26:09.369 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:08 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:08.996+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:09.369 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:08 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:08.996+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:09.369 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:08 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:08.996+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:09.369 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:08 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:08.996+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:09.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:09 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:09.165+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:09.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:09 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:09.165+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:09.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:09 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:09.165+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:09.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:09 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:09.165+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:09.609 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:09 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:09.132+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:09.609 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:09 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:09.132+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:09.609 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:09 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:09.132+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:09.609 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:09 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:09.132+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:09.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:09 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:09.367+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:26:09.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:09 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:09.367+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.272336+0000 front 2024-06-04T05:25:29.272362+0000 (oldest deadline 2024-06-04T05:25:54.572446+0000) 2024-06-04T05:26:09.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:09 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:09.367+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:37.874465+0000 front 2024-06-04T05:25:37.874486+0000 (oldest deadline 2024-06-04T05:26:02.574620+0000) 2024-06-04T05:26:09.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:09 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:09.367+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:42.575023+0000 front 2024-06-04T05:25:42.575083+0000 (oldest deadline 2024-06-04T05:26:07.875155+0000) 2024-06-04T05:26:10.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:09 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:09.731+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:10.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:09 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:09.731+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:10.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:09 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:09.731+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:36.360040+0000 front 2024-06-04T05:25:36.359900+0000 (oldest deadline 2024-06-04T05:26:02.259878+0000) 2024-06-04T05:26:10.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:09 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:09.731+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.261387+0000 front 2024-06-04T05:25:46.261364+0000 (oldest deadline 2024-06-04T05:26:08.561461+0000) 2024-06-04T05:26:10.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:09 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:09.971+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:10.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:09 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:09.971+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:10.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:09 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:09.971+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:10.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:09 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:09.971+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:10.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:10.098+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:10.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:10.098+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:10.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:10.098+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:10.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:10 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:10.098+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:10.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:10 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:10.187+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:10.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:10 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:10.187+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:10.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:10 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:10.187+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:10.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:10 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:10.187+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:10.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:10 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:10.390+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.070923+0000 front 2024-06-04T05:25:21.070948+0000 (oldest deadline 2024-06-04T05:25:43.371134+0000) 2024-06-04T05:26:10.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:10 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:10.390+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.272336+0000 front 2024-06-04T05:25:29.272362+0000 (oldest deadline 2024-06-04T05:25:54.572446+0000) 2024-06-04T05:26:10.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:10 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:10.390+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:37.874465+0000 front 2024-06-04T05:25:37.874486+0000 (oldest deadline 2024-06-04T05:26:02.574620+0000) 2024-06-04T05:26:10.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:10 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7[53609]: 2024-06-04T05:26:10.390+0000 7fae53437700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:42.575023+0000 front 2024-06-04T05:25:42.575083+0000 (oldest deadline 2024-06-04T05:26:07.875155+0000) 2024-06-04T05:26:11.175 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:10 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:10.686+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:11.176 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:10 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:10.686+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:11.176 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:10 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:10.686+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:36.360040+0000 front 2024-06-04T05:25:36.359900+0000 (oldest deadline 2024-06-04T05:26:02.259878+0000) 2024-06-04T05:26:11.176 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:10 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:10.686+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.261387+0000 front 2024-06-04T05:25:46.261364+0000 (oldest deadline 2024-06-04T05:26:08.561461+0000) 2024-06-04T05:26:11.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:11 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:11.064+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:11.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:11 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:11.064+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:11.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:11 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:11.064+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:11.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:11 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:11.064+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:11.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:11 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:11.174+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:11.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:11 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:11.174+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:11.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:11 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:11.174+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:11.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:11 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:11.174+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:11.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:11 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:11.174+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:11.466 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:11 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:11.006+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:11.466 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:11 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:11.006+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:11.466 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:11 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:11.006+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:11.466 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:11 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:11.006+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:11.745 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:11 smithi098 podman[79414]: 2024-06-04 05:26:11.465243786 +0000 UTC m=+5.551993867 container died 02ab71a6d6d6b0d58c8361730010213eedbeef52ea09e856e134453961ab6273 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=main-b7328fe, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-06-04T05:26:11.936 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:11 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:11.676+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:11.936 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:11 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:11.676+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:11.936 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:11 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:11.676+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:36.360040+0000 front 2024-06-04T05:25:36.359900+0000 (oldest deadline 2024-06-04T05:26:02.259878+0000) 2024-06-04T05:26:11.936 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:11 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:11.676+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.261387+0000 front 2024-06-04T05:25:46.261364+0000 (oldest deadline 2024-06-04T05:26:08.561461+0000) 2024-06-04T05:26:12.121 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:11 smithi098 podman[79414]: 2024-06-04 05:26:11.86052527 +0000 UTC m=+5.947275349 container cleanup 02ab71a6d6d6b0d58c8361730010213eedbeef52ea09e856e134453961ab6273 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=main-b7328fe, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-06-04T05:26:12.121 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:11 smithi098 bash[79414]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7 2024-06-04T05:26:12.122 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:11 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:11.993+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:12.122 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:11 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:11.993+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:12.122 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:11 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:11.993+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:12.122 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:11 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:11.993+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:12.122 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:11 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:11.993+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:12.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:12 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:12.045+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:12.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:12 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:12.045+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:12.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:12 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:12.045+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:12.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:12 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:12.045+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:12.495 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:12 smithi098 podman[79430]: 2024-06-04 05:26:12.120314354 +0000 UTC m=+0.650710676 container remove 02ab71a6d6d6b0d58c8361730010213eedbeef52ea09e856e134453961ab6273 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-b7328fe, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-06-04T05:26:12.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:12 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:12.209+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:12.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:12 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:12.209+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:12.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:12 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:12.209+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:12.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:12 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:12.209+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:12.687 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:12 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:12.209+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:12.841 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:12 smithi098 podman[79517]: 2024-06-04 05:26:12.466867341 +0000 UTC m=+0.016811504 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T05:26:12.841 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:12 smithi098 podman[79517]: 2024-06-04 05:26:12.596221115 +0000 UTC m=+0.146165263 container create 756235bea84533387d18839393da4827c1198587a4dbf11a33eadbfb04f79de8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7-deactivate, RELEASE=main-b7328fe, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=) 2024-06-04T05:26:12.841 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:12 smithi098 podman[79517]: 2024-06-04 05:26:12.840296754 +0000 UTC m=+0.390240921 container init 756235bea84533387d18839393da4827c1198587a4dbf11a33eadbfb04f79de8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-06-04T05:26:13.137 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:12 smithi098 podman[79517]: 2024-06-04 05:26:12.843828436 +0000 UTC m=+0.393772593 container start 756235bea84533387d18839393da4827c1198587a4dbf11a33eadbfb04f79de8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-06-04T05:26:13.137 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:12 smithi098 podman[79517]: 2024-06-04 05:26:12.90710951 +0000 UTC m=+0.457053681 container attach 756235bea84533387d18839393da4827c1198587a4dbf11a33eadbfb04f79de8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-b7328fe, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.build-date=20240521) 2024-06-04T05:26:13.137 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:12 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:12.994+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:13.137 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:12 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:12.994+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:13.137 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:12 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:12.994+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:13.138 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:12 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:12.994+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:13.138 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:12 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:12.994+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:13.183 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:12 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:12.722+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:13.183 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:12 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:12.722+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:13.183 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:12 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:12.722+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:36.360040+0000 front 2024-06-04T05:25:36.359900+0000 (oldest deadline 2024-06-04T05:26:02.259878+0000) 2024-06-04T05:26:13.184 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:12 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:12.722+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.261387+0000 front 2024-06-04T05:25:46.261364+0000 (oldest deadline 2024-06-04T05:26:08.561461+0000) 2024-06-04T05:26:13.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:13 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:13.034+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:13.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:13 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:13.034+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:13.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:13 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:13.034+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:13.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:13 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:13.034+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:13.404 INFO:journalctl@ceph.osd.7.smithi098.stdout:Jun 04 05:26:13 smithi098 podman[79517]: 2024-06-04 05:26:13.136501608 +0000 UTC m=+0.686445765 container died 756235bea84533387d18839393da4827c1198587a4dbf11a33eadbfb04f79de8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-06-04T05:26:13.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:13 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:13.181+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:13.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:13 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:13.181+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:13.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:13 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:13.181+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:13.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:13 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:13.181+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:13.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:13 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:13.181+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:13.816 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.7.service' 2024-06-04T05:26:13.894 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:26:13.894 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-06-04T05:26:13.894 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-06-04T05:26:13.894 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.8 2024-06-04T05:26:14.167 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:13 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:13.703+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:14.167 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:13 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:13.703+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:14.167 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:13 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:13.703+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:36.360040+0000 front 2024-06-04T05:25:36.359900+0000 (oldest deadline 2024-06-04T05:26:02.259878+0000) 2024-06-04T05:26:14.167 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:13 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:13.703+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.261387+0000 front 2024-06-04T05:25:46.261364+0000 (oldest deadline 2024-06-04T05:26:08.561461+0000) 2024-06-04T05:26:14.167 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:13 smithi112 systemd[1]: Stopping Ceph osd.8 for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:26:14.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:14 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:14.003+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:14.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:14 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:14.003+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:14.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:14 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:14.003+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:14.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:14 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:14.003+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:14.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:14 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:14.003+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:14.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:14 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:14.076+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:14.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:14 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:14.076+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:14.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:14 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:14.076+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:14.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:14 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:14.076+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:14.436 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:14 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:14.207+0000 7f0994298700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:26:14.436 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:14 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:14.207+0000 7f0994298700 -1 osd.8 125 *** Got signal Terminated *** 2024-06-04T05:26:14.436 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:14 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:14.207+0000 7f0994298700 -1 osd.8 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-04T05:26:14.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:14 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:14.164+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:14.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:14 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:14.164+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:14.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:14 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:14.164+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:14.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:14 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:14.164+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:14.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:14 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:14.164+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:14.936 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:14 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:14.686+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:14.936 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:14 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:14.686+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:14.936 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:14 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:14.686+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:36.360040+0000 front 2024-06-04T05:25:36.359900+0000 (oldest deadline 2024-06-04T05:26:02.259878+0000) 2024-06-04T05:26:14.936 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:14 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:14.686+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.261387+0000 front 2024-06-04T05:25:46.261364+0000 (oldest deadline 2024-06-04T05:26:08.561461+0000) 2024-06-04T05:26:15.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:14 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:14.966+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:15.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:14 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:14.966+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:15.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:14 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:14.966+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:15.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:14 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:14.966+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:15.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:14 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:14.966+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:15.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:15 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:15.028+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:15.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:15 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:15.028+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:15.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:15 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:15.028+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:15.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:15 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:15.028+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:15.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:15 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:15.160+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:15.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:15 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:15.160+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:15.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:15 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:15.160+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:15.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:15 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:15.160+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:15.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:15 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:15.160+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:15.936 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:15 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:15.673+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:15.936 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:15 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:15.673+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:15.936 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:15 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:15.673+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:36.360040+0000 front 2024-06-04T05:25:36.359900+0000 (oldest deadline 2024-06-04T05:26:02.259878+0000) 2024-06-04T05:26:15.936 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:15 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:15.673+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.261387+0000 front 2024-06-04T05:25:46.261364+0000 (oldest deadline 2024-06-04T05:26:08.561461+0000) 2024-06-04T05:26:16.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:15 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:15.939+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:16.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:15 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:15.939+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:16.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:15 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:15.939+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:16.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:15 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:15.939+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:16.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:15 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:15.939+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:16.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:15 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:15.998+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:16.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:15 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:15.998+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:16.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:15 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:15.998+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:16.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:15 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:15.998+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:16.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:16 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:16.143+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:16.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:16 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:16.143+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:16.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:16 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:16.143+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:16.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:16 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:16.143+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:16.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:16 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:16.143+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:17.144 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:16 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:16.686+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:17.144 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:16 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:16.686+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:17.144 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:16 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:16.686+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:36.360040+0000 front 2024-06-04T05:25:36.359900+0000 (oldest deadline 2024-06-04T05:26:02.259878+0000) 2024-06-04T05:26:17.144 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:16 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:16.686+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.261387+0000 front 2024-06-04T05:25:46.261364+0000 (oldest deadline 2024-06-04T05:26:08.561461+0000) 2024-06-04T05:26:17.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:16 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:16.926+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:17.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:16 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:16.926+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:17.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:16 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:16.926+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:17.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:16 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:16.926+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:17.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:16 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:16.926+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:17.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:17 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:17.027+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:17.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:17 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:17.027+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:17.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:17 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:17.027+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:17.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:17 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:17.027+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:17.438 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:17 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:17.141+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:17.438 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:17 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:17.141+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:17.438 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:17 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:17.141+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:17.439 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:17 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:17.141+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:17.439 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:17 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:17.141+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:17.439 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:17 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:17.141+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:18.127 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:17 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:17.727+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:18.127 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:17 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:17.727+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:18.127 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:17 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:17.727+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:36.360040+0000 front 2024-06-04T05:25:36.359900+0000 (oldest deadline 2024-06-04T05:26:02.259878+0000) 2024-06-04T05:26:18.127 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:17 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:17.727+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.261387+0000 front 2024-06-04T05:25:46.261364+0000 (oldest deadline 2024-06-04T05:26:08.561461+0000) 2024-06-04T05:26:18.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:17 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:17.950+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:18.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:17 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:17.950+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:18.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:17 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:17.950+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:18.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:17 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:17.950+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:18.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:17 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:17.950+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:18.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:17 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:17.950+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:18.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:18 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:18.049+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:18.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:18 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:18.049+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:18.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:18 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:18.049+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:18.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:18 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:18.049+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:18.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:18 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:18.125+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:18.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:18 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:18.125+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:18.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:18 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:18.125+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:18.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:18 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:18.125+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:18.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:18 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:18.125+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:18.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:18 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:18.125+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:19.159 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:18 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:18.685+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:20.156633+0000 front 2024-06-04T05:25:20.156615+0000 (oldest deadline 2024-06-04T05:25:44.256627+0000) 2024-06-04T05:26:19.159 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:18 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:18.685+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:25.957700+0000 front 2024-06-04T05:25:25.957683+0000 (oldest deadline 2024-06-04T05:25:51.857806+0000) 2024-06-04T05:26:19.159 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:18 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:18.685+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:36.360040+0000 front 2024-06-04T05:25:36.359900+0000 (oldest deadline 2024-06-04T05:26:02.259878+0000) 2024-06-04T05:26:19.159 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:18 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8[55104]: 2024-06-04T05:26:18.685+0000 7f099088c700 -1 osd.8 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.261387+0000 front 2024-06-04T05:25:46.261364+0000 (oldest deadline 2024-06-04T05:26:08.561461+0000) 2024-06-04T05:26:19.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:18 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:18.942+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:19.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:18 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:18.942+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:19.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:18 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:18.942+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:19.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:18 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:18.942+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:19.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:18 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:18.942+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:19.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:18 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:18.942+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:19.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:19 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:19.057+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:19.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:19 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:19.057+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:19.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:19 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:19.057+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:19.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:19 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:19.057+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:19.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:19 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:19.057+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:52.597571+0000 front 2024-06-04T05:25:52.597416+0000 (oldest deadline 2024-06-04T05:26:18.497402+0000) 2024-06-04T05:26:19.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:19 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:19.157+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:19.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:19 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:19.157+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:19.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:19 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:19.157+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:19.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:19 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:19.157+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:19.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:19 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:19.157+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:19.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:19 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:19.157+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:19.909 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:19 smithi112 podman[80733]: 2024-06-04 05:26:19.632689555 +0000 UTC m=+5.551357974 container died 562b972b6f5915ec9330c590826287e381e0d6eb3f7399ba1955fa1f8be4170e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8, RELEASE=main-b7328fe, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-06-04T05:26:20.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:20 smithi112 podman[80733]: 2024-06-04 05:26:20.007650042 +0000 UTC m=+5.926318450 container cleanup 562b972b6f5915ec9330c590826287e381e0d6eb3f7399ba1955fa1f8be4170e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD) 2024-06-04T05:26:20.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:20 smithi112 bash[80733]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8 2024-06-04T05:26:20.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:19 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:19.895+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:20.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:19 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:19.896+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:20.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:19 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:19.896+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:20.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:19 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:19.896+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:20.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:19 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:19.896+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:20.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:19 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:19.896+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:20.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:20 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:20.080+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:20.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:20 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:20.080+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:20.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:20 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:20.080+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:20.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:20 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:20.080+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:20.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:20 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:20.080+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:52.597571+0000 front 2024-06-04T05:25:52.597416+0000 (oldest deadline 2024-06-04T05:26:18.497402+0000) 2024-06-04T05:26:20.654 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:20 smithi112 podman[80752]: 2024-06-04 05:26:20.342257349 +0000 UTC m=+0.710225846 container remove 562b972b6f5915ec9330c590826287e381e0d6eb3f7399ba1955fa1f8be4170e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-b7328fe, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, ceph=True) 2024-06-04T05:26:20.655 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:20 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:20.185+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:20.655 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:20 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:20.185+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:20.655 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:20 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:20.185+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:20.655 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:20 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:20.185+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:20.655 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:20 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:20.185+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:20.655 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:20 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:20.185+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:20.922 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:20 smithi112 podman[80838]: 2024-06-04 05:26:20.555993098 +0000 UTC m=+0.017000949 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T05:26:20.922 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:20 smithi112 podman[80838]: 2024-06-04 05:26:20.684931669 +0000 UTC m=+0.145939515 container create 39f888b36127ce81dc7ee65c605d439c9859039987b3d95f10c9673e3bf2a39d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-b7328fe, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-06-04T05:26:20.922 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:20 smithi112 podman[80838]: 2024-06-04 05:26:20.920653646 +0000 UTC m=+0.381661498 container init 39f888b36127ce81dc7ee65c605d439c9859039987b3d95f10c9673e3bf2a39d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-04T05:26:21.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:20 smithi112 podman[80838]: 2024-06-04 05:26:20.924030854 +0000 UTC m=+0.385038714 container start 39f888b36127ce81dc7ee65c605d439c9859039987b3d95f10c9673e3bf2a39d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-b7328fe, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-04T05:26:21.186 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:20 smithi112 podman[80838]: 2024-06-04 05:26:20.995437743 +0000 UTC m=+0.456445597 container attach 39f888b36127ce81dc7ee65c605d439c9859039987b3d95f10c9673e3bf2a39d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=main-b7328fe, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-04T05:26:21.187 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:21 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:21.144+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:21.187 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:21 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:21.144+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:21.187 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:21 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:21.144+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:21.187 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:21 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:21.144+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:21.187 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:21 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:21.144+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:21.187 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:21 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:21.144+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:21.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:20 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:20.940+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:21.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:20 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:20.940+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:21.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:20 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:20.940+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:21.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:20 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:20.940+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:21.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:20 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:20.940+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:21.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:20 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:20.940+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:21.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:21 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:21.039+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:21.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:21 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:21.039+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:21.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:21 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:21.039+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:21.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:21 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:21.039+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:21.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:21 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:21.039+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:52.597571+0000 front 2024-06-04T05:25:52.597416+0000 (oldest deadline 2024-06-04T05:26:18.497402+0000) 2024-06-04T05:26:21.491 INFO:journalctl@ceph.osd.8.smithi112.stdout:Jun 04 05:26:21 smithi112 podman[80838]: 2024-06-04 05:26:21.221517776 +0000 UTC m=+0.682525628 container died 39f888b36127ce81dc7ee65c605d439c9859039987b3d95f10c9673e3bf2a39d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-b7328fe, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-04T05:26:21.830 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.8.service' 2024-06-04T05:26:21.869 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:26:21.869 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-06-04T05:26:21.869 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-06-04T05:26:21.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.9 2024-06-04T05:26:22.184 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:21 smithi070 systemd[1]: Stopping Ceph osd.9 for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:26:22.185 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:22 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:22.005+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:22.185 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:22 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:22.005+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:22.185 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:22 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:22.005+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:22.185 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:22 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:22.005+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:22.185 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:22 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:22.005+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:52.597571+0000 front 2024-06-04T05:25:52.597416+0000 (oldest deadline 2024-06-04T05:26:18.497402+0000) 2024-06-04T05:26:22.185 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:22 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:22.181+0000 7f1fb2123700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:26:22.185 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:22 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:22.181+0000 7f1fb2123700 -1 osd.9 125 *** Got signal Terminated *** 2024-06-04T05:26:22.185 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:22 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:22.181+0000 7f1fb2123700 -1 osd.9 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-04T05:26:22.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:21 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:21.945+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:22.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:21 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:21.945+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:22.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:21 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:21.945+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:22.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:21 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:21.945+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:22.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:21 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:21.945+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:22.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:21 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:21.945+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:22.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:22 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:22.182+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:22.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:22 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:22.182+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:22.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:22 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:22.182+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:22.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:22 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:22.182+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:22.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:22 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:22.182+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:22.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:22 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:22.182+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:23.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:22 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:22.975+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:23.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:22 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:22.975+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:23.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:22 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:22.975+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:23.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:22 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:22.975+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:23.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:22 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:22.975+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:23.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:22 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:22.975+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:23.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:22 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:22.977+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:23.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:22 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:22.977+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:23.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:22 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:22.977+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:23.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:22 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:22.977+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:23.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:22 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:22.977+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:52.597571+0000 front 2024-06-04T05:25:52.597416+0000 (oldest deadline 2024-06-04T05:26:18.497402+0000) 2024-06-04T05:26:23.687 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:23 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:23.231+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:23.687 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:23 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:23.231+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:23.687 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:23 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:23.231+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:23.687 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:23 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:23.231+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:23.687 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:23 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:23.231+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:23.687 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:23 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:23.231+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:24.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:23 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:23.933+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:24.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:23 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:23.933+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:24.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:23 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:23.933+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:24.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:23 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:23.933+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:24.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:23 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:23.933+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:24.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:23 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:23.933+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:24.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:23 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:23.972+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:24.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:23 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:23.972+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:24.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:23 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:23.972+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:24.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:23 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:23.972+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:24.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:23 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:23.972+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:52.597571+0000 front 2024-06-04T05:25:52.597416+0000 (oldest deadline 2024-06-04T05:26:18.497402+0000) 2024-06-04T05:26:24.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:24 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:24.189+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:24.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:24 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:24.189+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:24.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:24 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:24.189+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:24.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:24 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:24.189+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:24.687 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:24 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:24.189+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:24.687 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:24 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:24.189+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:25.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:24 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:24.907+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:25.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:24 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:24.907+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:25.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:24 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:24.907+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:25.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:24 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:24.907+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:25.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:24 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:24.907+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:25.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:24 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:24.907+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:25.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:24 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:24.974+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:25.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:24 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:24.974+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:25.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:24 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:24.974+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:25.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:24 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:24.974+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:25.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:24 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:24.974+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:52.597571+0000 front 2024-06-04T05:25:52.597416+0000 (oldest deadline 2024-06-04T05:26:18.497402+0000) 2024-06-04T05:26:25.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:25 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:25.237+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:25.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:25 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:25.237+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:25.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:25 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:25.237+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:25.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:25 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:25.237+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:25.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:25 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:25.237+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:25.687 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:25 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:25.237+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:26.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:25 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:25.921+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:26.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:25 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:25.921+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:26.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:25 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:25.921+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:26.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:25 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:25.921+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:26.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:25 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:25.921+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:26.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:25 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:25.921+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:26.358 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:26 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:26.003+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:26.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:26 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:26.003+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:26.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:26 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:26.003+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:26.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:26 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:26.003+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:26.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:26 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:26.003+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:52.597571+0000 front 2024-06-04T05:25:52.597416+0000 (oldest deadline 2024-06-04T05:26:18.497402+0000) 2024-06-04T05:26:26.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:26 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:26.003+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-06-04T05:26:02.399491+0000 front 2024-06-04T05:26:02.399527+0000 (oldest deadline 2024-06-04T05:26:25.299585+0000) 2024-06-04T05:26:26.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:26 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:26.198+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:26.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:26 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:26.198+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:26.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:26 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:26.198+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:26.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:26 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:26.198+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:26.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:26 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:26.198+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:26.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:26 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:26.198+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:27.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:26 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:26.887+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:27.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:26 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:26.887+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:27.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:26 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:26.887+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:27.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:26 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:26.887+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:27.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:26 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:26.887+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:27.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:26 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:26.887+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:27.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:26 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:26.887+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-06-04T05:26:00.614754+0000 front 2024-06-04T05:26:00.614652+0000 (oldest deadline 2024-06-04T05:26:26.514528+0000) 2024-06-04T05:26:27.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:27 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:27.053+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:12.489602+0000 front 2024-06-04T05:25:12.489819+0000 (oldest deadline 2024-06-04T05:25:35.890505+0000) 2024-06-04T05:26:27.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:27 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:27.054+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:18.791764+0000 front 2024-06-04T05:25:18.791736+0000 (oldest deadline 2024-06-04T05:25:43.491897+0000) 2024-06-04T05:26:27.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:27 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:27.054+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.893820+0000 front 2024-06-04T05:25:29.893772+0000 (oldest deadline 2024-06-04T05:25:55.793738+0000) 2024-06-04T05:26:27.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:27 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:27.054+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:35.794198+0000 front 2024-06-04T05:25:35.794185+0000 (oldest deadline 2024-06-04T05:26:00.394706+0000) 2024-06-04T05:26:27.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:27 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:27.054+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:52.597571+0000 front 2024-06-04T05:25:52.597416+0000 (oldest deadline 2024-06-04T05:26:18.497402+0000) 2024-06-04T05:26:27.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:27 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9[66978]: 2024-06-04T05:26:27.054+0000 7f1fae717700 -1 osd.9 125 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-06-04T05:26:02.399491+0000 front 2024-06-04T05:26:02.399527+0000 (oldest deadline 2024-06-04T05:26:25.299585+0000) 2024-06-04T05:26:27.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:27 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:27.150+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:27.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:27 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:27.150+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:27.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:27 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:27.150+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:27.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:27 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:27.150+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:27.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:27 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:27.150+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:27.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:27 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:27.150+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:28.017 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:27 smithi070 podman[167463]: 2024-06-04 05:26:27.741867085 +0000 UTC m=+5.671352754 container died 8ac04e70817f9a1c4bf77dfd406eb10959435efeda6c525d4f2ae7b31435e687 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-b7328fe, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-06-04T05:26:28.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:27 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:27.855+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:28.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:27 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:27.855+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:28.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:27 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:27.855+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:28.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:27 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:27.855+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:28.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:27 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:27.855+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:28.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:27 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:27.855+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:28.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:27 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:27.855+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-06-04T05:26:00.614754+0000 front 2024-06-04T05:26:00.614652+0000 (oldest deadline 2024-06-04T05:26:26.514528+0000) 2024-06-04T05:26:28.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:28 smithi070 podman[167463]: 2024-06-04 05:26:28.206582531 +0000 UTC m=+6.136068194 container cleanup 8ac04e70817f9a1c4bf77dfd406eb10959435efeda6c525d4f2ae7b31435e687 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, RELEASE=main-b7328fe, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-04T05:26:28.359 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:28 smithi070 bash[167463]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9 2024-06-04T05:26:28.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:28 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:28.115+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:28.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:28 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:28.116+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:28.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:28 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:28.116+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:28.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:28 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:28.116+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:28.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:28 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:28.116+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:28.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:28 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:28.116+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:28.827 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:28 smithi070 podman[167478]: 2024-06-04 05:26:28.491492891 +0000 UTC m=+0.747660209 container remove 8ac04e70817f9a1c4bf77dfd406eb10959435efeda6c525d4f2ae7b31435e687 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , RELEASE=main-b7328fe, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521) 2024-06-04T05:26:29.109 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:28 smithi070 podman[167566]: 2024-06-04 05:26:28.729765243 +0000 UTC m=+0.017070560 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T05:26:29.109 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:28 smithi070 podman[167566]: 2024-06-04 05:26:28.850446227 +0000 UTC m=+0.137751534 container create bc2d30e3f54b68a389dff1fdd04db2ba321c6bd76a63cb6112472c9702772ee1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-b7328fe, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-04T05:26:29.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:28 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:28.861+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:29.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:28 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:28.861+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:29.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:28 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:28.861+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:29.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:28 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:28.861+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:29.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:28 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:28.861+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:29.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:28 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:28.861+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:29.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:28 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:28.861+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-06-04T05:26:00.614754+0000 front 2024-06-04T05:26:00.614652+0000 (oldest deadline 2024-06-04T05:26:26.514528+0000) 2024-06-04T05:26:29.409 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:29 smithi070 podman[167566]: 2024-06-04 05:26:29.111010584 +0000 UTC m=+0.398315895 container init bc2d30e3f54b68a389dff1fdd04db2ba321c6bd76a63cb6112472c9702772ee1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-06-04T05:26:29.409 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:29 smithi070 podman[167566]: 2024-06-04 05:26:29.114300638 +0000 UTC m=+0.401605955 container start bc2d30e3f54b68a389dff1fdd04db2ba321c6bd76a63cb6112472c9702772ee1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-b7328fe, CEPH_POINT_RELEASE=, ceph=True) 2024-06-04T05:26:29.409 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:29 smithi070 podman[167566]: 2024-06-04 05:26:29.202738572 +0000 UTC m=+0.490043894 container attach bc2d30e3f54b68a389dff1fdd04db2ba321c6bd76a63cb6112472c9702772ee1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, RELEASE=main-b7328fe, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-06-04T05:26:29.409 INFO:journalctl@ceph.osd.9.smithi070.stdout:Jun 04 05:26:29 smithi070 podman[167566]: 2024-06-04 05:26:29.407962573 +0000 UTC m=+0.695267886 container died bc2d30e3f54b68a389dff1fdd04db2ba321c6bd76a63cb6112472c9702772ee1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-9-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-b7328fe) 2024-06-04T05:26:29.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:29 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:29.159+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:29.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:29 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:29.159+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:29.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:29 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:29.159+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:29.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:29 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:29.159+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:29.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:29 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:29.159+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:29.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:29 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:29.159+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:29.970 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.9.service' 2024-06-04T05:26:30.010 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:26:30.010 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-06-04T05:26:30.010 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-06-04T05:26:30.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.10 2024-06-04T05:26:30.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:29 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:29.835+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:30.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:29 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:29.835+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:30.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:29 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:29.835+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:30.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:29 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:29.835+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:30.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:29 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:29.835+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:30.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:29 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:29.835+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:30.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:29 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:29.835+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-06-04T05:26:00.614754+0000 front 2024-06-04T05:26:00.614652+0000 (oldest deadline 2024-06-04T05:26:26.514528+0000) 2024-06-04T05:26:30.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:30 smithi098 systemd[1]: Stopping Ceph osd.10 for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:26:30.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:30 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:30.169+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:30.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:30 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:30.169+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:30.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:30 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:30.169+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:30.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:30 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:30.169+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:30.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:30 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:30.169+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:30.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:30 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:30.169+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:30.745 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:30 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:30.330+0000 7efd4e5bf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:26:30.745 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:30 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:30.330+0000 7efd4e5bf700 -1 osd.10 125 *** Got signal Terminated *** 2024-06-04T05:26:30.745 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:30 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:30.330+0000 7efd4e5bf700 -1 osd.10 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-04T05:26:31.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:30 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:30.882+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:31.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:30 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:30.882+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:31.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:30 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:30.882+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:31.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:30 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:30.882+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:31.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:30 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:30.882+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:31.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:30 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:30.882+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:31.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:30 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:30.882+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-06-04T05:26:00.614754+0000 front 2024-06-04T05:26:00.614652+0000 (oldest deadline 2024-06-04T05:26:26.514528+0000) 2024-06-04T05:26:31.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:31 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:31.141+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:31.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:31 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:31.141+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:31.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:31 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:31.141+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:31.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:31 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:31.141+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:31.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:31 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:31.141+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:31.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:31 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:31.141+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:32.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:31 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:31.847+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:32.396 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:31 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:31.847+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:32.396 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:31 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:31.847+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:32.396 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:31 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:31.847+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:32.396 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:31 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:31.847+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:32.396 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:31 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:31.847+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:32.396 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:31 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:31.847+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-06-04T05:26:00.614754+0000 front 2024-06-04T05:26:00.614652+0000 (oldest deadline 2024-06-04T05:26:26.514528+0000) 2024-06-04T05:26:32.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:32 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:32.154+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:32.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:32 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:32.154+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:32.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:32 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:32.154+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:32.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:32 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:32.154+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:32.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:32 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:32.154+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:32.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:32 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:32.154+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:33.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:32 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:32.806+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:33.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:32 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:32.806+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:33.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:32 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:32.806+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:33.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:32 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:32.806+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:33.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:32 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:32.806+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:33.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:32 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:32.806+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:33.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:32 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:32.806+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-06-04T05:26:00.614754+0000 front 2024-06-04T05:26:00.614652+0000 (oldest deadline 2024-06-04T05:26:26.514528+0000) 2024-06-04T05:26:33.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:33 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:33.114+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:33.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:33 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:33.114+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:33.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:33 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:33.114+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:33.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:33 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:33.114+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:33.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:33 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:33.114+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:33.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:33 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:33.114+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:33.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:33 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:33.114+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-06-04T05:26:09.937446+0000 front 2024-06-04T05:26:09.937363+0000 (oldest deadline 2024-06-04T05:26:32.837375+0000) 2024-06-04T05:26:34.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:33 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:33.808+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:34.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:33 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:33.808+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:34.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:33 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:33.808+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:34.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:33 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:33.808+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:34.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:33 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:33.808+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:34.245 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:33 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:33.808+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:34.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:33 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:33.808+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-06-04T05:26:00.614754+0000 front 2024-06-04T05:26:00.614652+0000 (oldest deadline 2024-06-04T05:26:26.514528+0000) 2024-06-04T05:26:34.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:34 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:34.123+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:34.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:34 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:34.123+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:34.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:34 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:34.123+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:34.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:34 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:34.123+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:34.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:34 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:34.123+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:34.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:34 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:34.123+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:34.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:34 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:34.123+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-06-04T05:26:09.937446+0000 front 2024-06-04T05:26:09.937363+0000 (oldest deadline 2024-06-04T05:26:32.837375+0000) 2024-06-04T05:26:35.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:34 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:34.843+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:13.507216+0000 front 2024-06-04T05:25:13.507249+0000 (oldest deadline 2024-06-04T05:25:37.607222+0000) 2024-06-04T05:26:35.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:34 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:34.843+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:19.308060+0000 front 2024-06-04T05:25:19.308198+0000 (oldest deadline 2024-06-04T05:25:45.208136+0000) 2024-06-04T05:26:35.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:34 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:34.843+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:28.009558+0000 front 2024-06-04T05:25:28.009434+0000 (oldest deadline 2024-06-04T05:25:52.709582+0000) 2024-06-04T05:26:35.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:34 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:34.843+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.511099+0000 front 2024-06-04T05:25:38.511191+0000 (oldest deadline 2024-06-04T05:26:00.811189+0000) 2024-06-04T05:26:35.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:34 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:34.843+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:46.512927+0000 front 2024-06-04T05:25:46.512972+0000 (oldest deadline 2024-06-04T05:26:11.213134+0000) 2024-06-04T05:26:35.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:34 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:34.843+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:51.213839+0000 front 2024-06-04T05:25:51.213963+0000 (oldest deadline 2024-06-04T05:26:17.113675+0000) 2024-06-04T05:26:35.246 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:34 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10[60164]: 2024-06-04T05:26:34.843+0000 7efd46bab700 -1 osd.10 125 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-06-04T05:26:00.614754+0000 front 2024-06-04T05:26:00.614652+0000 (oldest deadline 2024-06-04T05:26:26.514528+0000) 2024-06-04T05:26:35.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:35 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:35.130+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:35.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:35 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:35.130+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:35.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:35 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:35.130+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:35.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:35 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:35.130+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:35.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:35 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:35.130+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:35.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:35 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:35.130+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:35.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:35 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:35.130+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-06-04T05:26:09.937446+0000 front 2024-06-04T05:26:09.937363+0000 (oldest deadline 2024-06-04T05:26:32.837375+0000) 2024-06-04T05:26:36.074 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:35 smithi098 podman[79637]: 2024-06-04 05:26:35.808829853 +0000 UTC m=+5.601385342 container died 1432d7446edbb739c532355125253520458b9f8e6a07af8b06050010a6b28884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10, RELEASE=main-b7328fe, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-06-04T05:26:36.434 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:36 smithi098 podman[79637]: 2024-06-04 05:26:36.164361105 +0000 UTC m=+5.956916585 container cleanup 1432d7446edbb739c532355125253520458b9f8e6a07af8b06050010a6b28884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-b7328fe, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-06-04T05:26:36.434 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:36 smithi098 bash[79637]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10 2024-06-04T05:26:36.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:36 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:36.116+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:36.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:36 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:36.116+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:36.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:36 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:36.116+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:36.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:36 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:36.116+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:36.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:36 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:36.116+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:36.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:36 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:36.116+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:36.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:36 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:36.116+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-06-04T05:26:09.937446+0000 front 2024-06-04T05:26:09.937363+0000 (oldest deadline 2024-06-04T05:26:32.837375+0000) 2024-06-04T05:26:36.745 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:36 smithi098 podman[79654]: 2024-06-04 05:26:36.432339459 +0000 UTC m=+0.613512437 container remove 1432d7446edbb739c532355125253520458b9f8e6a07af8b06050010a6b28884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521) 2024-06-04T05:26:37.003 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:36 smithi098 podman[79744]: 2024-06-04 05:26:36.648985588 +0000 UTC m=+0.016743466 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T05:26:37.003 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:36 smithi098 podman[79744]: 2024-06-04 05:26:36.766372993 +0000 UTC m=+0.134130913 container create 668f6705af1a42f66a579ed1bdd6e63cca541f7af19bc54c0be2834d7c9cd633 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240521, RELEASE=main-b7328fe, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True) 2024-06-04T05:26:37.003 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:37 smithi098 podman[79744]: 2024-06-04 05:26:37.002129769 +0000 UTC m=+0.369887646 container init 668f6705af1a42f66a579ed1bdd6e63cca541f7af19bc54c0be2834d7c9cd633 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-b7328fe, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-06-04T05:26:37.299 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:37 smithi098 podman[79744]: 2024-06-04 05:26:37.005395968 +0000 UTC m=+0.373153846 container start 668f6705af1a42f66a579ed1bdd6e63cca541f7af19bc54c0be2834d7c9cd633 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-b7328fe, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-06-04T05:26:37.300 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:37 smithi098 podman[79744]: 2024-06-04 05:26:37.093704459 +0000 UTC m=+0.461462347 container attach 668f6705af1a42f66a579ed1bdd6e63cca541f7af19bc54c0be2834d7c9cd633 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-b7328fe) 2024-06-04T05:26:37.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:37 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:37.110+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:37.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:37 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:37.110+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:37.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:37 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:37.110+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:37.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:37 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:37.110+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:37.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:37 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:37.110+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:37.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:37 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:37.110+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:37.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:37 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:37.110+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-06-04T05:26:09.937446+0000 front 2024-06-04T05:26:09.937363+0000 (oldest deadline 2024-06-04T05:26:32.837375+0000) 2024-06-04T05:26:37.589 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:37 smithi098 podman[79744]: 2024-06-04 05:26:37.298557537 +0000 UTC m=+0.666315423 container died 668f6705af1a42f66a579ed1bdd6e63cca541f7af19bc54c0be2834d7c9cd633 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10-deactivate, RELEASE=main-b7328fe, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-06-04T05:26:37.848 INFO:journalctl@ceph.osd.10.smithi098.stdout:Jun 04 05:26:37 smithi098 podman[79770]: 2024-06-04 05:26:37.822353197 +0000 UTC m=+0.517830540 container remove 668f6705af1a42f66a579ed1bdd6e63cca541f7af19bc54c0be2834d7c9cd633 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-b7328fe, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-06-04T05:26:37.855 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.10.service' 2024-06-04T05:26:37.898 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:26:37.898 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-06-04T05:26:37.899 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-06-04T05:26:37.899 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.11 2024-06-04T05:26:38.186 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:37 smithi112 systemd[1]: Stopping Ceph osd.11 for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:26:38.187 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:38 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:38.072+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:38.187 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:38 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:38.072+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:38.187 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:38 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:38.072+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:38.187 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:38 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:38.072+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:38.187 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:38 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:38.072+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:38.187 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:38 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:38.072+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:38.187 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:38 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:38.072+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-06-04T05:26:09.937446+0000 front 2024-06-04T05:26:09.937363+0000 (oldest deadline 2024-06-04T05:26:32.837375+0000) 2024-06-04T05:26:38.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:38 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:38.227+0000 7f0ee62de700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:26:38.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:38 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:38.227+0000 7f0ee62de700 -1 osd.11 125 *** Got signal Terminated *** 2024-06-04T05:26:38.686 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:38 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:38.227+0000 7f0ee62de700 -1 osd.11 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-04T05:26:39.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:39 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:39.048+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:39.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:39 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:39.048+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:39.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:39 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:39.048+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:39.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:39 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:39.048+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:39.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:39 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:39.048+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:39.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:39 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:39.048+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:39.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:39 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:39.048+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-06-04T05:26:09.937446+0000 front 2024-06-04T05:26:09.937363+0000 (oldest deadline 2024-06-04T05:26:32.837375+0000) 2024-06-04T05:26:40.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:40 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:40.011+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:40.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:40 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:40.011+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:40.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:40 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:40.011+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:40.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:40 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:40.011+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:40.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:40 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:40.011+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:40.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:40 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:40.011+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:40.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:40 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:40.011+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-06-04T05:26:09.937446+0000 front 2024-06-04T05:26:09.937363+0000 (oldest deadline 2024-06-04T05:26:32.837375+0000) 2024-06-04T05:26:41.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:40 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:40.990+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:41.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:40 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:40.990+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:41.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:40 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:40.990+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:41.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:40 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:40.990+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:41.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:40 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:40.990+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:41.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:40 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:40.990+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:41.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:40 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:40.990+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-06-04T05:26:09.937446+0000 front 2024-06-04T05:26:09.937363+0000 (oldest deadline 2024-06-04T05:26:32.837375+0000) 2024-06-04T05:26:42.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:42.032+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:42.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:42.032+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:42.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:42.032+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:42.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:42.032+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:42.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:42.032+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:42.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:42.032+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:42.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:42.032+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-06-04T05:26:09.937446+0000 front 2024-06-04T05:26:09.937363+0000 (oldest deadline 2024-06-04T05:26:32.837375+0000) 2024-06-04T05:26:42.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:42.032+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6820 osd.8 since back 2024-06-04T05:26:18.638771+0000 front 2024-06-04T05:26:18.638747+0000 (oldest deadline 2024-06-04T05:26:41.538736+0000) 2024-06-04T05:26:43.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:42.994+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-06-04T05:25:10.528164+0000 front 2024-06-04T05:25:10.528041+0000 (oldest deadline 2024-06-04T05:25:35.228068+0000) 2024-06-04T05:26:43.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:42.994+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-06-04T05:25:21.629790+0000 front 2024-06-04T05:25:21.630030+0000 (oldest deadline 2024-06-04T05:25:45.129851+0000) 2024-06-04T05:26:43.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:42.994+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6804 osd.2 since back 2024-06-04T05:25:29.830682+0000 front 2024-06-04T05:25:29.830662+0000 (oldest deadline 2024-06-04T05:25:54.530822+0000) 2024-06-04T05:26:43.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:42.994+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-06-04T05:25:38.532543+0000 front 2024-06-04T05:25:38.532461+0000 (oldest deadline 2024-06-04T05:26:00.832468+0000) 2024-06-04T05:26:43.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:42.994+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-06-04T05:25:45.533454+0000 front 2024-06-04T05:25:45.533500+0000 (oldest deadline 2024-06-04T05:26:10.833444+0000) 2024-06-04T05:26:43.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:42.994+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6812 osd.5 since back 2024-06-04T05:25:50.833981+0000 front 2024-06-04T05:25:50.833997+0000 (oldest deadline 2024-06-04T05:26:16.734025+0000) 2024-06-04T05:26:43.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:42.994+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-06-04T05:26:09.937446+0000 front 2024-06-04T05:26:09.937363+0000 (oldest deadline 2024-06-04T05:26:32.837375+0000) 2024-06-04T05:26:43.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:42 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11[62098]: 2024-06-04T05:26:42.994+0000 7f0ee28d2700 -1 osd.11 125 heartbeat_check: no reply from 172.21.15.112:6820 osd.8 since back 2024-06-04T05:26:18.638771+0000 front 2024-06-04T05:26:18.638747+0000 (oldest deadline 2024-06-04T05:26:41.538736+0000) 2024-06-04T05:26:43.986 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:43 smithi112 podman[80965]: 2024-06-04 05:26:43.691170819 +0000 UTC m=+5.586025359 container died ce1ba889925807d3103f2ddd0dcb4f71ad53a30814b31299c3523a6ac066058a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-b7328fe, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521) 2024-06-04T05:26:44.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:43 smithi112 podman[80965]: 2024-06-04 05:26:43.984933986 +0000 UTC m=+5.879788521 container cleanup ce1ba889925807d3103f2ddd0dcb4f71ad53a30814b31299c3523a6ac066058a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-06-04T05:26:44.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:43 smithi112 bash[80965]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11 2024-06-04T05:26:44.437 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:44 smithi112 podman[80980]: 2024-06-04 05:26:44.177877955 +0000 UTC m=+0.488133136 container remove ce1ba889925807d3103f2ddd0dcb4f71ad53a30814b31299c3523a6ac066058a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-b7328fe, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-06-04T05:26:44.860 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:44 smithi112 podman[81070]: 2024-06-04 05:26:44.510868819 +0000 UTC m=+0.017059652 image pull 57e49aa75a3f375d3f00c41cb0b86c82a3db74c6589d24b01ff82e31f6c2f00d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a 2024-06-04T05:26:44.860 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:44 smithi112 podman[81070]: 2024-06-04 05:26:44.628575678 +0000 UTC m=+0.134766485 container create 4fa806b0aa973a815308a53b9d62f9a7645b81c2d085255bb7f6ecac6bb24801 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , RELEASE=main-b7328fe, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-06-04T05:26:44.860 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:44 smithi112 podman[81070]: 2024-06-04 05:26:44.855993157 +0000 UTC m=+0.362183978 container init 4fa806b0aa973a815308a53b9d62f9a7645b81c2d085255bb7f6ecac6bb24801 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=main-b7328fe, org.label-schema.build-date=20240521, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-06-04T05:26:44.860 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:44 smithi112 podman[81070]: 2024-06-04 05:26:44.859355386 +0000 UTC m=+0.365546195 container start 4fa806b0aa973a815308a53b9d62f9a7645b81c2d085255bb7f6ecac6bb24801 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, RELEASE=main-b7328fe, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_CLEAN=True) 2024-06-04T05:26:45.154 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:44 smithi112 podman[81070]: 2024-06-04 05:26:44.955958959 +0000 UTC m=+0.462149776 container attach 4fa806b0aa973a815308a53b9d62f9a7645b81c2d085255bb7f6ecac6bb24801 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-b7328fe, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521) 2024-06-04T05:26:45.436 INFO:journalctl@ceph.osd.11.smithi112.stdout:Jun 04 05:26:45 smithi112 podman[81070]: 2024-06-04 05:26:45.153154913 +0000 UTC m=+0.659345738 container died 4fa806b0aa973a815308a53b9d62f9a7645b81c2d085255bb7f6ecac6bb24801 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-osd-11-deactivate, RELEASE=main-b7328fe, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-06-04T05:26:45.750 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@osd.11.service' 2024-06-04T05:26:45.790 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:26:45.790 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-06-04T05:26:45.790 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-06-04T05:26:45.790 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.a 2024-06-04T05:26:46.100 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 05:26:45 smithi070 systemd[1]: Stopping Ceph mds.a for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:26:46.357 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 05:26:46 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-a[74103]: 2024-06-04T05:26:46.097+0000 7fa065fdd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:26:46.357 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 05:26:46 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-a[74103]: 2024-06-04T05:26:46.097+0000 7fa065fdd700 -1 mds.a *** got signal Terminated *** 2024-06-04T05:26:49.741 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 05:26:49 smithi070 podman[167671]: 2024-06-04 05:26:49.470790428 +0000 UTC m=+3.489354916 container died 2227476cb2266f698108b64582e84028ea20f8a8cecf3c3ee28bb1c1c6092be6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-b7328fe, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-06-04T05:26:49.992 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 05:26:49 smithi070 podman[167671]: 2024-06-04 05:26:49.972352331 +0000 UTC m=+3.990916812 container cleanup 2227476cb2266f698108b64582e84028ea20f8a8cecf3c3ee28bb1c1c6092be6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-b7328fe, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-06-04T05:26:49.992 INFO:journalctl@ceph.mds.a.smithi070.stdout:Jun 04 05:26:49 smithi070 bash[167671]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-a 2024-06-04T05:26:50.462 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.a.service' 2024-06-04T05:26:50.499 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:26:50.499 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-06-04T05:26:50.499 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-06-04T05:26:50.499 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.d 2024-06-04T05:26:50.859 INFO:journalctl@ceph.mds.d.smithi070.stdout:Jun 04 05:26:50 smithi070 systemd[1]: Stopping Ceph mds.d for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:26:51.359 INFO:journalctl@ceph.mds.d.smithi070.stdout:Jun 04 05:26:50 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-d[72834]: 2024-06-04T05:26:50.893+0000 7f62f19ce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:26:51.359 INFO:journalctl@ceph.mds.d.smithi070.stdout:Jun 04 05:26:50 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-d[72834]: 2024-06-04T05:26:50.893+0000 7f62f19ce700 -1 mds.d *** got signal Terminated *** 2024-06-04T05:26:55.109 INFO:journalctl@ceph.mds.d.smithi070.stdout:Jun 04 05:26:54 smithi070 podman[167803]: 2024-06-04 05:26:54.778516679 +0000 UTC m=+4.061198534 container died 26ade5db2dbacf6ef91745e6b17345494cb4ac14b3cd7899c221a663ca4e7b95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, ceph=True, org.label-schema.schema-version=1.0) 2024-06-04T05:26:55.575 INFO:journalctl@ceph.mds.d.smithi070.stdout:Jun 04 05:26:55 smithi070 podman[167803]: 2024-06-04 05:26:55.160001824 +0000 UTC m=+4.442683699 container cleanup 26ade5db2dbacf6ef91745e6b17345494cb4ac14b3cd7899c221a663ca4e7b95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-04T05:26:55.575 INFO:journalctl@ceph.mds.d.smithi070.stdout:Jun 04 05:26:55 smithi070 bash[167803]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-d 2024-06-04T05:26:55.575 INFO:journalctl@ceph.mds.d.smithi070.stdout:Jun 04 05:26:55 smithi070 podman[167815]: 2024-06-04 05:26:55.336353623 +0000 UTC m=+0.554199319 container remove 26ade5db2dbacf6ef91745e6b17345494cb4ac14b3cd7899c221a663ca4e7b95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-d, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=main-b7328fe, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-04T05:26:55.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.d.service' 2024-06-04T05:26:55.627 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:26:55.627 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-06-04T05:26:55.628 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-06-04T05:26:55.628 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.g 2024-06-04T05:26:55.858 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 05:26:55 smithi070 systemd[1]: Stopping Ceph mds.g for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:26:56.358 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 05:26:55 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-g[73644]: 2024-06-04T05:26:55.953+0000 7f5ea9396700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:26:56.359 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 05:26:55 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-g[73644]: 2024-06-04T05:26:55.953+0000 7f5ea9396700 -1 mds.g *** got signal Terminated *** 2024-06-04T05:26:57.749 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 05:26:57 smithi070 podman[167922]: 2024-06-04 05:26:57.455837933 +0000 UTC m=+1.603310879 container died ee01603f0be4382400ef24bb04a4ed8c3e3cd036a36119d0fcd9e630c8849192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-g, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=main-b7328fe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-06-04T05:26:58.109 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 05:26:57 smithi070 podman[167922]: 2024-06-04 05:26:57.938684293 +0000 UTC m=+2.086157239 container cleanup ee01603f0be4382400ef24bb04a4ed8c3e3cd036a36119d0fcd9e630c8849192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-b7328fe) 2024-06-04T05:26:58.109 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 05:26:57 smithi070 bash[167922]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-g 2024-06-04T05:26:58.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.g.service' 2024-06-04T05:26:58.386 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 05:26:58 smithi070 podman[167937]: 2024-06-04 05:26:58.123538393 +0000 UTC m=+0.670720593 container remove ee01603f0be4382400ef24bb04a4ed8c3e3cd036a36119d0fcd9e630c8849192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-g, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, RELEASE=main-b7328fe, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5) 2024-06-04T05:26:58.386 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 05:26:58 smithi070 systemd[1]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.g.service: Deactivated successfully. 2024-06-04T05:26:58.386 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 05:26:58 smithi070 systemd[1]: Stopped Ceph mds.g for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T05:26:58.386 INFO:journalctl@ceph.mds.g.smithi070.stdout:Jun 04 05:26:58 smithi070 systemd[1]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.g.service: Consumed 18.229s CPU time. 2024-06-04T05:26:58.398 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:26:58.398 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-06-04T05:26:58.398 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-06-04T05:26:58.399 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.j 2024-06-04T05:26:58.719 INFO:journalctl@ceph.mds.j.smithi070.stdout:Jun 04 05:26:58 smithi070 systemd[1]: Stopping Ceph mds.j for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:26:59.108 INFO:journalctl@ceph.mds.j.smithi070.stdout:Jun 04 05:26:58 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-j[72390]: 2024-06-04T05:26:58.716+0000 7fea304af700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:26:59.109 INFO:journalctl@ceph.mds.j.smithi070.stdout:Jun 04 05:26:58 smithi070 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-j[72390]: 2024-06-04T05:26:58.716+0000 7fea304af700 -1 mds.j *** got signal Terminated *** 2024-06-04T05:27:02.859 INFO:journalctl@ceph.mds.j.smithi070.stdout:Jun 04 05:27:02 smithi070 podman[168043]: 2024-06-04 05:27:02.58923327 +0000 UTC m=+3.979819988 container died 92fa0d8e8b6b560c03c378d2769e8e2bc3b98ffb39fed5eb4fd8f55d15f0d493 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-j, ceph=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-b7328fe, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-06-04T05:27:03.358 INFO:journalctl@ceph.mds.j.smithi070.stdout:Jun 04 05:27:03 smithi070 podman[168043]: 2024-06-04 05:27:03.02569903 +0000 UTC m=+4.416285769 container cleanup 92fa0d8e8b6b560c03c378d2769e8e2bc3b98ffb39fed5eb4fd8f55d15f0d493 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-j, RELEASE=main-b7328fe, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-06-04T05:27:03.359 INFO:journalctl@ceph.mds.j.smithi070.stdout:Jun 04 05:27:03 smithi070 bash[168043]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-j 2024-06-04T05:27:03.359 INFO:journalctl@ceph.mds.j.smithi070.stdout:Jun 04 05:27:03 smithi070 podman[168056]: 2024-06-04 05:27:03.268756731 +0000 UTC m=+0.679313329 container remove 92fa0d8e8b6b560c03c378d2769e8e2bc3b98ffb39fed5eb4fd8f55d15f0d493 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-j, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=main-b7328fe, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-04T05:27:03.497 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.j.service' 2024-06-04T05:27:03.529 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:27:03.529 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-06-04T05:27:03.529 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-06-04T05:27:03.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.b 2024-06-04T05:27:03.833 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 05:27:03 smithi098 systemd[1]: Stopping Ceph mds.b for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:27:04.092 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 05:27:03 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-b[65481]: 2024-06-04T05:27:03.830+0000 7f0a49493700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:27:04.093 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 05:27:03 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-b[65481]: 2024-06-04T05:27:03.830+0000 7f0a49493700 -1 mds.b *** got signal Terminated *** 2024-06-04T05:27:05.633 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 05:27:05 smithi098 podman[79849]: 2024-06-04 05:27:05.326520711 +0000 UTC m=+1.602455035 container died 5d68ed213d1f94a119bf9ca1d2c65f897bf33a49879c1b061f4d258ed145656c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-b, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-b7328fe, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-04T05:27:05.967 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 05:27:05 smithi098 podman[79849]: 2024-06-04 05:27:05.714876385 +0000 UTC m=+1.990810745 container cleanup 5d68ed213d1f94a119bf9ca1d2c65f897bf33a49879c1b061f4d258ed145656c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240521, GIT_CLEAN=True) 2024-06-04T05:27:05.967 INFO:journalctl@ceph.mds.b.smithi098.stdout:Jun 04 05:27:05 smithi098 bash[79849]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-b 2024-06-04T05:27:06.095 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.b.service' 2024-06-04T05:27:06.131 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:27:06.131 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-06-04T05:27:06.131 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-06-04T05:27:06.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.e 2024-06-04T05:27:06.245 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 05:27:06 smithi098 systemd[1]: Stopping Ceph mds.e for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:27:06.745 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 05:27:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-e[64860]: 2024-06-04T05:27:06.451+0000 7f29a438e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:27:06.745 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 05:27:06 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-e[64860]: 2024-06-04T05:27:06.451+0000 7f29a438e700 -1 mds.e *** got signal Terminated *** 2024-06-04T05:27:09.587 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 05:27:09 smithi098 podman[79983]: 2024-06-04 05:27:09.31339581 +0000 UTC m=+2.963219547 container died 14db21803650c377a6f8d81dd01b9c85f758bcecc06c994077d317e01f497cec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, RELEASE=main-b7328fe, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-06-04T05:27:09.946 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 05:27:09 smithi098 podman[79983]: 2024-06-04 05:27:09.66901028 +0000 UTC m=+3.318834035 container cleanup 14db21803650c377a6f8d81dd01b9c85f758bcecc06c994077d317e01f497cec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-e, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-04T05:27:09.946 INFO:journalctl@ceph.mds.e.smithi098.stdout:Jun 04 05:27:09 smithi098 bash[79983]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-e 2024-06-04T05:27:10.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.e.service' 2024-06-04T05:27:10.134 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:27:10.134 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-06-04T05:27:10.135 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-06-04T05:27:10.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.h 2024-06-04T05:27:10.466 INFO:journalctl@ceph.mds.h.smithi098.stdout:Jun 04 05:27:10 smithi098 systemd[1]: Stopping Ceph mds.h for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:27:10.745 INFO:journalctl@ceph.mds.h.smithi098.stdout:Jun 04 05:27:10 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-h[65161]: 2024-06-04T05:27:10.464+0000 7f4ff1494700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:27:10.745 INFO:journalctl@ceph.mds.h.smithi098.stdout:Jun 04 05:27:10 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-h[65161]: 2024-06-04T05:27:10.464+0000 7f4ff1494700 -1 mds.h *** got signal Terminated *** 2024-06-04T05:27:13.608 INFO:journalctl@ceph.mds.h.smithi098.stdout:Jun 04 05:27:13 smithi098 podman[80105]: 2024-06-04 05:27:13.313971313 +0000 UTC m=+2.955786140 container died 87dd7687abc53b63010ae0008fa928e43cbd2a708b6341f6b96a33a120e3d820 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-h, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=main-b7328fe, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-06-04T05:27:13.933 INFO:journalctl@ceph.mds.h.smithi098.stdout:Jun 04 05:27:13 smithi098 podman[80105]: 2024-06-04 05:27:13.689629153 +0000 UTC m=+3.331443980 container cleanup 87dd7687abc53b63010ae0008fa928e43cbd2a708b6341f6b96a33a120e3d820 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-h, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-06-04T05:27:13.934 INFO:journalctl@ceph.mds.h.smithi098.stdout:Jun 04 05:27:13 smithi098 bash[80105]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-h 2024-06-04T05:27:14.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.h.service' 2024-06-04T05:27:14.096 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:27:14.096 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-06-04T05:27:14.097 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-06-04T05:27:14.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.k 2024-06-04T05:27:14.245 INFO:journalctl@ceph.mds.k.smithi098.stdout:Jun 04 05:27:14 smithi098 systemd[1]: Stopping Ceph mds.k for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:27:14.745 INFO:journalctl@ceph.mds.k.smithi098.stdout:Jun 04 05:27:14 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-k[64561]: 2024-06-04T05:27:14.409+0000 7ffb1516a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:27:14.745 INFO:journalctl@ceph.mds.k.smithi098.stdout:Jun 04 05:27:14 smithi098 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-k[64561]: 2024-06-04T05:27:14.409+0000 7ffb1516a700 -1 mds.k *** got signal Terminated *** 2024-06-04T05:27:17.670 INFO:journalctl@ceph.mds.k.smithi098.stdout:Jun 04 05:27:17 smithi098 podman[80225]: 2024-06-04 05:27:17.327067018 +0000 UTC m=+3.014283476 container died 99a968947a488078f74723bc143dda25c688d6b659bd636aa9e99dced5c26cf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-k, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, RELEASE=main-b7328fe, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-04T05:27:17.995 INFO:journalctl@ceph.mds.k.smithi098.stdout:Jun 04 05:27:17 smithi098 podman[80225]: 2024-06-04 05:27:17.760633091 +0000 UTC m=+3.447849543 container cleanup 99a968947a488078f74723bc143dda25c688d6b659bd636aa9e99dced5c26cf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-k, ceph=True, RELEASE=main-b7328fe, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521) 2024-06-04T05:27:17.995 INFO:journalctl@ceph.mds.k.smithi098.stdout:Jun 04 05:27:17 smithi098 bash[80225]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-k 2024-06-04T05:27:18.237 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.k.service' 2024-06-04T05:27:18.258 INFO:journalctl@ceph.mds.k.smithi098.stdout:Jun 04 05:27:17 smithi098 podman[80240]: 2024-06-04 05:27:17.994811868 +0000 UTC m=+0.645374020 container remove 99a968947a488078f74723bc143dda25c688d6b659bd636aa9e99dced5c26cf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-b7328fe, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-06-04T05:27:18.258 INFO:journalctl@ceph.mds.k.smithi098.stdout:Jun 04 05:27:18 smithi098 systemd[1]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.k.service: Deactivated successfully. 2024-06-04T05:27:18.258 INFO:journalctl@ceph.mds.k.smithi098.stdout:Jun 04 05:27:18 smithi098 systemd[1]: Stopped Ceph mds.k for 164dbaba-222d-11ef-bc9b-c7b262605968. 2024-06-04T05:27:18.258 INFO:journalctl@ceph.mds.k.smithi098.stdout:Jun 04 05:27:18 smithi098 systemd[1]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.k.service: Consumed 14.676s CPU time. 2024-06-04T05:27:18.268 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:27:18.268 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-06-04T05:27:18.268 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-06-04T05:27:18.268 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.c 2024-06-04T05:27:18.577 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 05:27:18 smithi112 systemd[1]: Stopping Ceph mds.c for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:27:18.936 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 05:27:18 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-c[67503]: 2024-06-04T05:27:18.576+0000 7f3cf872f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:27:18.936 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 05:27:18 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-c[67503]: 2024-06-04T05:27:18.576+0000 7f3cf872f700 -1 mds.c *** got signal Terminated *** 2024-06-04T05:27:21.602 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 05:27:21 smithi112 podman[81174]: 2024-06-04 05:27:21.331641651 +0000 UTC m=+2.865106388 container died abd1e979f2cc5bf11f1cfb25d6be62126dcd964a5ec4880fd096317802a28291 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-c, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-06-04T05:27:21.936 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 05:27:21 smithi112 podman[81174]: 2024-06-04 05:27:21.692238827 +0000 UTC m=+3.225703552 container cleanup abd1e979f2cc5bf11f1cfb25d6be62126dcd964a5ec4880fd096317802a28291 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-b7328fe, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0) 2024-06-04T05:27:21.936 INFO:journalctl@ceph.mds.c.smithi112.stdout:Jun 04 05:27:21 smithi112 bash[81174]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-c 2024-06-04T05:27:22.130 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.c.service' 2024-06-04T05:27:22.165 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:27:22.165 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-06-04T05:27:22.165 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-06-04T05:27:22.166 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.f 2024-06-04T05:27:22.686 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 05:27:22 smithi112 systemd[1]: Stopping Ceph mds.f for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:27:22.686 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 05:27:22 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-f[67196]: 2024-06-04T05:27:22.504+0000 7fee60614700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:27:22.686 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 05:27:22 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-f[67196]: 2024-06-04T05:27:22.504+0000 7fee60614700 -1 mds.f *** got signal Terminated *** 2024-06-04T05:27:25.615 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 05:27:25 smithi112 podman[81307]: 2024-06-04 05:27:25.340796681 +0000 UTC m=+2.942683284 container died 76244bb7a9e717c7950bc3562dbd85ac644cd274b55f676b0dd82add97cc6a34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=main-b7328fe, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-06-04T05:27:25.936 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 05:27:25 smithi112 podman[81307]: 2024-06-04 05:27:25.695551548 +0000 UTC m=+3.297438143 container cleanup 76244bb7a9e717c7950bc3562dbd85ac644cd274b55f676b0dd82add97cc6a34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-f, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-b7328fe, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-06-04T05:27:25.936 INFO:journalctl@ceph.mds.f.smithi112.stdout:Jun 04 05:27:25 smithi112 bash[81307]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-f 2024-06-04T05:27:26.135 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.f.service' 2024-06-04T05:27:26.169 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:27:26.169 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-06-04T05:27:26.169 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-06-04T05:27:26.169 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.i 2024-06-04T05:27:26.686 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:27:26 smithi112 systemd[1]: Stopping Ceph mds.i for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:27:26.686 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:27:26 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:27:26.473+0000 7fa451a4e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:27:26.686 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:27:26 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i[67804]: 2024-06-04T05:27:26.473+0000 7fa451a4e700 -1 mds.i *** got signal Terminated *** 2024-06-04T05:27:29.610 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:27:29 smithi112 podman[81428]: 2024-06-04 05:27:29.332408198 +0000 UTC m=+2.949914917 container died ac055dfba103d64848825cd2a86f4a38a2700256fe7404114c0d0ee379b7ac87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, RELEASE=main-b7328fe, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521) 2024-06-04T05:27:29.885 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:27:29 smithi112 podman[81428]: 2024-06-04 05:27:29.682486456 +0000 UTC m=+3.299993174 container cleanup ac055dfba103d64848825cd2a86f4a38a2700256fe7404114c0d0ee379b7ac87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=main-b7328fe, ceph=True) 2024-06-04T05:27:29.885 INFO:journalctl@ceph.mds.i.smithi112.stdout:Jun 04 05:27:29 smithi112 bash[81428]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-i 2024-06-04T05:27:30.110 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.i.service' 2024-06-04T05:27:30.142 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:27:30.143 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-06-04T05:27:30.143 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-06-04T05:27:30.143 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.l 2024-06-04T05:27:30.436 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 05:27:30 smithi112 systemd[1]: Stopping Ceph mds.l for 164dbaba-222d-11ef-bc9b-c7b262605968... 2024-06-04T05:27:30.844 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 05:27:30 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-l[66885]: 2024-06-04T05:27:30.452+0000 7fdde277a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-04T05:27:30.844 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 05:27:30 smithi112 ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-l[66885]: 2024-06-04T05:27:30.452+0000 7fdde277a700 -1 mds.l *** got signal Terminated *** 2024-06-04T05:27:33.654 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 05:27:33 smithi112 podman[81550]: 2024-06-04 05:27:33.347601505 +0000 UTC m=+2.986662498 container died 4730b993ef36ac73daa6386462d16f572151a5963ddd90ab6c598e4522be1ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-l, RELEASE=main-b7328fe, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-06-04T05:27:33.654 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 05:27:33 smithi112 podman[81550]: 2024-06-04 05:27:33.652696387 +0000 UTC m=+3.291757379 container cleanup 4730b993ef36ac73daa6386462d16f572151a5963ddd90ab6c598e4522be1ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-l, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-b7328fe, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521) 2024-06-04T05:27:33.936 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 05:27:33 smithi112 bash[81550]: ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-l 2024-06-04T05:27:33.936 INFO:journalctl@ceph.mds.l.smithi112.stdout:Jun 04 05:27:33 smithi112 podman[81565]: 2024-06-04 05:27:33.853566946 +0000 UTC m=+0.502603392 container remove 4730b993ef36ac73daa6386462d16f572151a5963ddd90ab6c598e4522be1ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d69fb745e0d748b72f6cab1f010a6c2bbc9d2c58cb065d7f63d3528692b7c29a, name=ceph-164dbaba-222d-11ef-bc9b-c7b262605968-mds-l, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=main-b7328fe, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2) 2024-06-04T05:27:33.974 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-164dbaba-222d-11ef-bc9b-c7b262605968@mds.l.service' 2024-06-04T05:27:34.006 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-04T05:27:34.007 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-06-04T05:27:34.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-04T05:27:34.197 INFO:teuthology.orchestra.run.smithi070.stdout:Deleting cluster with fsid: 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T05:27:36.759 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-04T05:27:36.949 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T05:27:39.598 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-04T05:27:39.789 INFO:teuthology.orchestra.run.smithi112.stdout:Deleting cluster with fsid: 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T05:27:42.201 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-04T05:27:42.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-04T05:27:42.268 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-04T05:27:42.295 INFO:tasks.cephadm:Archiving crash dumps... 2024-06-04T05:27:42.297 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-22_20:24:16-fs-main-distro-default-smithi/7721102/remote/smithi070/crash 2024-06-04T05:27:42.297 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/crash -- . 2024-06-04T05:27:42.334 INFO:teuthology.orchestra.run.smithi070.stderr:tar: /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-04T05:27:42.334 INFO:teuthology.orchestra.run.smithi070.stderr:tar: Error is not recoverable: exiting now 2024-06-04T05:27:42.336 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-22_20:24:16-fs-main-distro-default-smithi/7721102/remote/smithi098/crash 2024-06-04T05:27:42.337 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/crash -- . 2024-06-04T05:27:42.374 INFO:teuthology.orchestra.run.smithi098.stderr:tar: /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-04T05:27:42.374 INFO:teuthology.orchestra.run.smithi098.stderr:tar: Error is not recoverable: exiting now 2024-06-04T05:27:42.376 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-22_20:24:16-fs-main-distro-default-smithi/7721102/remote/smithi112/crash 2024-06-04T05:27:42.377 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/crash -- . 2024-06-04T05:27:42.413 INFO:teuthology.orchestra.run.smithi112.stderr:tar: /var/lib/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-04T05:27:42.413 INFO:teuthology.orchestra.run.smithi112.stderr:tar: Error is not recoverable: exiting now 2024-06-04T05:27:42.414 INFO:tasks.cephadm:Checking cluster log for badness... 2024-06-04T05:27:42.414 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-06-04T05:27:42.502 INFO:teuthology.orchestra.run.smithi070.stdout:2024-06-04T05:00:34.358695+0000 mds.i (mds.0) 24 : cluster [WRN] Scrub error on inode 0x10000000215 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.i log and `damage ls` output for details 2024-06-04T05:27:42.502 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-06-04T05:27:42.503 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[SEC\]' /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-06-04T05:27:42.537 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[ERR\]' /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-06-04T05:27:42.613 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[WRN\]' /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-06-04T05:27:42.687 INFO:teuthology.orchestra.run.smithi070.stdout:2024-06-04T05:00:34.358695+0000 mds.i (mds.0) 24 : cluster [WRN] Scrub error on inode 0x10000000215 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.i log and `damage ls` output for details 2024-06-04T05:27:42.688 INFO:tasks.cephadm:Compressing logs... 2024-06-04T05:27:42.688 DEBUG:teuthology.orchestra.run.smithi070:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-04T05:27:42.731 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-04T05:27:42.733 DEBUG:teuthology.orchestra.run.smithi112:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-04T05:27:42.757 INFO:teuthology.orchestra.run.smithi070.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-06-04T05:27:42.758 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.log 2024-06-04T05:27:42.760 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-04T05:27:42.760 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.g.log 2024-06-04T05:27:42.760 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.j.log 2024-06-04T05:27:42.760 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.3.log 2024-06-04T05:27:42.760 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.0.log 2024-06-04T05:27:42.760 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mgr.x.log 2024-06-04T05:27:42.764 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mon.a.log 2024-06-04T05:27:42.764 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.6.log 2024-06-04T05:27:42.764 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-04T05:27:42.764 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.a.log 2024-06-04T05:27:42.764 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-04T05:27:42.764 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.cephadm.log: /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.d.log 2024-06-04T05:27:42.764 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.9.log 2024-06-04T05:27:42.764 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-06-04T05:27:42.765 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-04T05:27:42.765 INFO:teuthology.orchestra.run.smithi070.stderr: 87.6% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.log.gz 2024-06-04T05:27:42.765 INFO:teuthology.orchestra.run.smithi098.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-06-04T05:27:42.767 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.log 2024-06-04T05:27:42.767 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-04T05:27:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.7.log 2024-06-04T05:27:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.b.log 2024-06-04T05:27:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.audit.log: /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.4.log 2024-06-04T05:27:42.769 INFO:teuthology.orchestra.run.smithi112.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-06-04T05:27:42.772 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.1.log 2024-06-04T05:27:42.772 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mgr.y.log 2024-06-04T05:27:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.h.log 2024-06-04T05:27:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.10.log 2024-06-04T05:27:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-04T05:27:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.e.log 2024-06-04T05:27:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-04T05:27:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mon.b.log 2024-06-04T05:27:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.k.log 2024-06-04T05:27:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-04T05:27:42.774 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.log 2024-06-04T05:27:42.774 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.2.log 2024-06-04T05:27:42.774 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.f.log 2024-06-04T05:27:42.775 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-04T05:27:42.775 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.5.log 2024-06-04T05:27:42.777 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.c.log 2024-06-04T05:27:42.778 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mon.c.log 2024-06-04T05:27:42.778 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mgr.z.log 2024-06-04T05:27:42.778 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.i.log 2024-06-04T05:27:42.778 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-04T05:27:42.778 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.11.log 2024-06-04T05:27:42.778 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.8.log 2024-06-04T05:27:42.778 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-04T05:27:42.778 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.l.log 2024-06-04T05:27:42.778 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-04T05:27:42.854 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 84.3% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-04T05:27:42.857 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 84.3% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-04T05:27:42.876 INFO:teuthology.orchestra.run.smithi112.stderr: 90.5% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-04T05:27:42.897 INFO:teuthology.orchestra.run.smithi112.stderr: 87.6% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.log.gz 2024-06-04T05:27:42.910 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.0.log: /var/log/ceph/cephadm.log: 94.7% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.j.log.gz 2024-06-04T05:27:42.930 INFO:teuthology.orchestra.run.smithi098.stderr: 92.7% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mgr.y.log.gz 2024-06-04T05:27:42.974 INFO:teuthology.orchestra.run.smithi098.stderr: 90.5% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-04T05:27:42.977 INFO:teuthology.orchestra.run.smithi112.stderr: 92.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-04T05:27:42.978 INFO:teuthology.orchestra.run.smithi098.stderr: 91.1% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-04T05:27:42.990 INFO:teuthology.orchestra.run.smithi070.stderr: 94.6% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.d.log.gz 2024-06-04T05:27:42.994 INFO:teuthology.orchestra.run.smithi112.stderr: 92.7% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mgr.z.log.gz 2024-06-04T05:27:42.995 INFO:teuthology.orchestra.run.smithi070.stderr: 91.1% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-04T05:27:43.006 INFO:teuthology.orchestra.run.smithi070.stderr: 90.3% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-04T05:27:43.013 INFO:teuthology.orchestra.run.smithi098.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-04T05:27:43.246 INFO:teuthology.orchestra.run.smithi112.stderr: 91.1% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-04T05:27:43.446 INFO:teuthology.orchestra.run.smithi070.stderr: 84.5% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-04T05:27:43.457 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph.log.gz 2024-06-04T05:27:43.743 INFO:teuthology.orchestra.run.smithi070.stderr: 95.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-04T05:27:44.640 INFO:teuthology.orchestra.run.smithi070.stderr: 90.2% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mgr.x.log.gz 2024-06-04T05:27:49.204 INFO:teuthology.orchestra.run.smithi098.stderr: 91.6% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mon.b.log.gz 2024-06-04T05:27:55.554 INFO:teuthology.orchestra.run.smithi112.stderr: 91.7% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mon.c.log.gz 2024-06-04T05:27:59.858 INFO:teuthology.orchestra.run.smithi098.stderr: 95.2% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.k.log.gz 2024-06-04T05:28:05.140 INFO:teuthology.orchestra.run.smithi070.stderr: 90.8% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mon.a.log.gz 2024-06-04T05:28:05.630 INFO:teuthology.orchestra.run.smithi098.stderr: 95.1% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.e.log.gz 2024-06-04T05:28:15.338 INFO:teuthology.orchestra.run.smithi098.stderr: 91.1% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.4.log.gz 2024-06-04T05:28:22.063 INFO:teuthology.orchestra.run.smithi112.stderr: 94.8% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.l.log.gz 2024-06-04T05:28:27.805 INFO:teuthology.orchestra.run.smithi098.stderr: 91.4% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.1.log.gz 2024-06-04T05:28:29.389 INFO:teuthology.orchestra.run.smithi070.stderr: 94.8% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.a.log.gz 2024-06-04T05:28:57.355 INFO:teuthology.orchestra.run.smithi098.stderr: 90.9% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.10.log.gz 2024-06-04T05:29:02.570 INFO:teuthology.orchestra.run.smithi070.stderr: 94.7% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.g.log.gz 2024-06-04T05:29:37.584 INFO:teuthology.orchestra.run.smithi098.stderr: 90.9% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.7.log.gz 2024-06-04T05:30:46.267 INFO:teuthology.orchestra.run.smithi070.stderr: 91.3% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.0.log.gz 2024-06-04T05:30:54.348 INFO:teuthology.orchestra.run.smithi070.stderr: 90.7% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.9.log.gz 2024-06-04T05:31:11.382 INFO:teuthology.orchestra.run.smithi070.stderr: 91.0% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.3.log.gz 2024-06-04T05:31:23.616 INFO:teuthology.orchestra.run.smithi070.stderr: 90.8% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.6.log.gz 2024-06-04T05:31:47.748 INFO:teuthology.orchestra.run.smithi112.stderr: 90.6% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.5.log.gz 2024-06-04T05:32:09.762 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.h.log.gz 2024-06-04T05:32:19.062 INFO:teuthology.orchestra.run.smithi112.stderr: 90.8% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.11.log.gz 2024-06-04T05:32:22.628 INFO:teuthology.orchestra.run.smithi098.stderr: 94.5% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.b.log.gz 2024-06-04T05:32:22.630 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-04T05:32:22.630 INFO:teuthology.orchestra.run.smithi098.stderr:real 4m39.881s 2024-06-04T05:32:22.630 INFO:teuthology.orchestra.run.smithi098.stderr:user 4m52.502s 2024-06-04T05:32:22.630 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m11.708s 2024-06-04T05:32:41.583 INFO:teuthology.orchestra.run.smithi112.stderr: 90.6% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.8.log.gz 2024-06-04T05:32:50.532 INFO:teuthology.orchestra.run.smithi112.stderr: 91.2% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-osd.2.log.gz 2024-06-04T05:36:25.577 INFO:teuthology.orchestra.run.smithi070.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-06-04T05:36:25.578 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-06-04T05:36:25.578 INFO:teuthology.orchestra.run.smithi070.stderr:real 8m42.834s 2024-06-04T05:36:25.578 INFO:teuthology.orchestra.run.smithi070.stderr:user 6m1.612s 2024-06-04T05:36:25.579 INFO:teuthology.orchestra.run.smithi070.stderr:sys 0m15.898s 2024-06-04T05:45:05.847 INFO:teuthology.orchestra.run.smithi112.stderr: 94.5% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.f.log.gz 2024-06-04T05:47:47.146 INFO:teuthology.orchestra.run.smithi112.stderr: 94.3% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.c.log.gz 2024-06-04T05:49:29.577 INFO:teuthology.orchestra.run.smithi112.stderr: 95.1% -- replaced with /var/log/ceph/164dbaba-222d-11ef-bc9b-c7b262605968/ceph-mds.i.log.gz 2024-06-04T05:49:29.579 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-06-04T05:49:29.579 INFO:teuthology.orchestra.run.smithi112.stderr:real 21m46.830s 2024-06-04T05:49:29.579 INFO:teuthology.orchestra.run.smithi112.stderr:user 7m43.953s 2024-06-04T05:49:29.579 INFO:teuthology.orchestra.run.smithi112.stderr:sys 0m23.023s 2024-06-04T05:49:29.580 INFO:tasks.cephadm:Archiving logs... 2024-06-04T05:49:29.580 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-22_20:24:16-fs-main-distro-default-smithi/7721102/remote/smithi070/log 2024-06-04T05:49:29.581 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-04T05:50:30.371 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-22_20:24:16-fs-main-distro-default-smithi/7721102/remote/smithi098/log 2024-06-04T05:50:30.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-04T05:51:11.902 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-22_20:24:16-fs-main-distro-default-smithi/7721102/remote/smithi112/log 2024-06-04T05:51:11.903 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-04T05:52:17.545 INFO:tasks.cephadm:Removing cluster... 2024-06-04T05:52:17.546 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 --force 2024-06-04T05:52:17.842 INFO:teuthology.orchestra.run.smithi070.stdout:Deleting cluster with fsid: 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T05:52:18.154 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 --force 2024-06-04T05:52:18.345 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T05:52:18.905 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 164dbaba-222d-11ef-bc9b-c7b262605968 --force 2024-06-04T05:52:19.295 INFO:teuthology.orchestra.run.smithi112.stdout:Deleting cluster with fsid: 164dbaba-222d-11ef-bc9b-c7b262605968 2024-06-04T05:52:19.912 INFO:tasks.cephadm:Removing cephadm ... 2024-06-04T05:52:19.913 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-04T05:52:19.935 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-04T05:52:19.955 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-04T05:52:19.973 INFO:tasks.cephadm:Teardown complete 2024-06-04T05:52:19.973 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-04T05:52:19.986 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-04T05:52:19.986 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-04T05:52:19.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-04T05:52:19.997 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-04T05:52:20.129 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-04T05:52:20.143 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-04T05:52:20.143 DEBUG:teuthology.orchestra.run.smithi070:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-04T05:52:20.146 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-04T05:52:20.149 DEBUG:teuthology.orchestra.run.smithi112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-04T05:52:20.160 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-06-04T05:52:20.163 INFO:teuthology.orchestra.run.smithi112.stderr:bash: line 1: ntpq: command not found 2024-06-04T05:52:20.163 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-06-04T05:52:20.186 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-04T05:52:20.186 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-06-04T05:52:20.186 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv01.front.sepia.ceph.com 3 6 377 12 -1551us[-1544us] +/- 51ms 2024-06-04T05:52:20.186 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 8 -1752us[-1752us] +/- 58ms 2024-06-04T05:52:20.186 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 +5056us[+5056us] +/- 75ms 2024-06-04T05:52:20.186 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-06-04T05:52:20.190 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-04T05:52:20.190 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2024-06-04T05:52:20.190 INFO:teuthology.orchestra.run.smithi112.stdout:^* hv01.front.sepia.ceph.com 3 7 377 75 -1532us[-1535us] +/- 50ms 2024-06-04T05:52:20.190 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 78 -1740us[-1744us] +/- 57ms 2024-06-04T05:52:20.190 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +5052us[+5052us] +/- 75ms 2024-06-04T05:52:20.190 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-06-04T05:52:20.275 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-04T05:52:20.276 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-06-04T05:52:20.276 INFO:teuthology.orchestra.run.smithi070.stdout:^* hv01.front.sepia.ceph.com 3 6 377 11 -1627us[-1636us] +/- 51ms 2024-06-04T05:52:20.276 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 11 -1829us[-1838us] +/- 58ms 2024-06-04T05:52:20.276 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 +4983us[+4974us] +/- 75ms 2024-06-04T05:52:20.276 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-06-04T05:52:20.276 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-04T05:52:20.287 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-04T05:52:20.287 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-04T05:52:20.380 DEBUG:teuthology.orchestra.run.smithi070:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-04T05:52:20.539 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-04T05:52:20.662 DEBUG:teuthology.orchestra.run.smithi112:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-04T05:52:20.766 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-04T05:52:20.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-04T05:52:20.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-04T05:52:20.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-04T05:52:20.828 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-04T05:52:20.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-04T05:52:20.856 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-04T05:52:20.870 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-04T05:52:20.889 INFO:teuthology.task.internal:Duration was 5349.720475 seconds 2024-06-04T05:52:20.890 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-04T05:52:20.901 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-04T05:52:20.901 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-04T05:52:20.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-04T05:52:20.907 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-04T05:52:20.961 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-04T05:52:20.968 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-04T05:52:20.974 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-04T05:52:21.479 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-04T05:52:21.480 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi070.front.sepia.ceph.com 2024-06-04T05:52:21.480 DEBUG:teuthology.orchestra.run.smithi070:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-04T05:52:21.504 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-06-04T05:52:21.504 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-04T05:52:21.532 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi112.front.sepia.ceph.com 2024-06-04T05:52:21.532 DEBUG:teuthology.orchestra.run.smithi112:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-04T05:52:21.562 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-04T05:52:21.562 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-04T05:52:21.565 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-04T05:52:21.575 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-04T05:52:21.632 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-04T05:52:21.632 DEBUG:teuthology.orchestra.run.smithi070:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-04T05:52:21.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-04T05:52:21.643 DEBUG:teuthology.orchestra.run.smithi112:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-04T05:52:23.645 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-04T05:52:23.658 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-04T05:52:23.658 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-04T05:52:23.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-04T05:52:23.723 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-04T05:52:23.754 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-04T05:52:23.765 DEBUG:teuthology.orchestra.run.smithi070:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-04T05:52:23.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-04T05:52:23.770 DEBUG:teuthology.orchestra.run.smithi112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-04T05:52:23.802 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-06-04T05:52:23.812 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = core 2024-06-04T05:52:23.838 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = core 2024-06-04T05:52:23.880 DEBUG:teuthology.orchestra.run.smithi070:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-04T05:52:23.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-04T05:52:23.895 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-04T05:52:23.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-04T05:52:23.912 DEBUG:teuthology.orchestra.run.smithi112:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-04T05:52:23.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-04T05:52:23.927 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-04T05:52:23.939 INFO:teuthology.task.internal:Transferring archived files... 2024-06-04T05:52:23.940 DEBUG:teuthology.misc:Transferring archived files from smithi070:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-22_20:24:16-fs-main-distro-default-smithi/7721102/remote/smithi070 2024-06-04T05:52:23.940 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-04T05:52:24.022 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-22_20:24:16-fs-main-distro-default-smithi/7721102/remote/smithi098 2024-06-04T05:52:24.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-04T05:52:24.062 DEBUG:teuthology.misc:Transferring archived files from smithi112:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-22_20:24:16-fs-main-distro-default-smithi/7721102/remote/smithi112 2024-06-04T05:52:24.062 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-04T05:52:24.101 INFO:teuthology.task.internal:Removing archive directory... 2024-06-04T05:52:24.102 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-04T05:52:24.104 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-04T05:52:24.107 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-04T05:52:24.153 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-04T05:52:24.166 INFO:teuthology.task.internal:Not uploading archives. 2024-06-04T05:52:24.166 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-04T05:52:24.189 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-04T05:52:24.189 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-04T05:52:24.192 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-04T05:52:24.194 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-04T05:52:24.204 INFO:teuthology.orchestra.run.smithi070.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 4 05:52 /home/ubuntu/cephtest 2024-06-04T05:52:24.208 INFO:teuthology.orchestra.run.smithi098.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 4 05:52 /home/ubuntu/cephtest 2024-06-04T05:52:24.210 INFO:teuthology.orchestra.run.smithi112.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 4 05:52 /home/ubuntu/cephtest 2024-06-04T05:52:24.211 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-04T05:52:24.224 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-04T05:52:24.253 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} duration: 5349.72047495842 failure_reason: '"2024-06-04T05:00:34.358695+0000 mds.i (mds.0) 24 : cluster [WRN] Scrub error on inode 0x10000000215 (/volumes/qa/sv_0/0cd839e3-1220-4eb9-8759-e3d92f3bc84b/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.i log and `damage ls` output for details" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-06-04T05:52:24.254 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-04T05:52:24.342 INFO:teuthology.run:FAIL